> Training Environment: | > Backend: Torch | > Mixed precision: True | > Precision: bf16 | > Current device: 0 | > Num. of GPUs: 1 | > Num. of CPUs: 24 | > Num. of Torch Threads: 1 | > Torch seed: 54321 | > Torch CUDNN: True | > Torch CUDNN deterministic: False | > Torch CUDNN benchmark: False | > Torch TF32 MatMul: False > Start Tensorboard: tensorboard --logdir=xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3 > Model has 521734790 parameters  > EPOCH: 0/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.02072405343008513 (+0) | > avg_loss_text_ce: 0.24675691976110534 (+0) | > avg_loss_mel_ce: 5.665910275855867 (+0) | > avg_loss: 5.912667175330738 (+0)  > EPOCH: 1/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-23 19:53:18)   --> TIME: 2025-02-23 19:53:19 -- STEP: 0/36938 -- GLOBAL_STEP: 0 | > loss_text_ce: 0.25331079959869385 (0.25331079959869385) | > loss_mel_ce: 5.85863733291626 (5.85863733291626) | > loss: 0.024253763258457184 (0.024253763258457184) | > current_lr: 5e-06 | > step_time: 0.1917 (0.19173932075500488) | > loader_time: 0.709 (0.7090170383453369)  --> TIME: 2025-02-23 19:53:28 -- STEP: 50/36938 -- GLOBAL_STEP: 50 | > loss_text_ce: 0.24186012148857117 (0.2466050860285759) | > loss_mel_ce: 5.769166469573975 (6.048273658752441) | > loss: 0.023853281512856483 (0.024979678876698018) | > current_lr: 5e-06 | > step_time: 0.123 (0.13104568481445308) | > loader_time: 0.0043 (0.006132435798645019)  --> TIME: 2025-02-23 19:53:37 -- STEP: 100/36938 -- GLOBAL_STEP: 100 | > loss_text_ce: 0.2606712877750397 (0.2444060169160366) | > loss_mel_ce: 4.674097537994385 (5.825288619995117) | > loss: 0.019582416862249374 (0.02408609114587307) | > current_lr: 5e-06 | > step_time: 0.1123 (0.12868310928344723) | > loader_time: 0.0064 (0.006127362251281738)  --> TIME: 2025-02-23 19:53:46 -- STEP: 150/36938 -- GLOBAL_STEP: 150 | > loss_text_ce: 0.24726562201976776 (0.24529533604780832) | > loss_mel_ce: 7.006048202514648 (5.701662795146306) | > loss: 0.02878299355506897 (0.023599041517203052) | > current_lr: 5e-06 | > step_time: 0.1118 (0.12959246476491296) | > loader_time: 0.006 (0.006033104260762531)  --> TIME: 2025-02-23 19:53:54 -- STEP: 200/36938 -- GLOBAL_STEP: 200 | > loss_text_ce: 0.2342241257429123 (0.24586484149098398) | > loss_mel_ce: 6.094424724578857 (5.71299804240465) | > loss: 0.02511368878185749 (0.02364628263749182) | > current_lr: 5e-06 | > step_time: 0.1344 (0.1287778580188752) | > loader_time: 0.0049 (0.006034066677093506)  --> TIME: 2025-02-23 19:54:03 -- STEP: 250/36938 -- GLOBAL_STEP: 250 | > loss_text_ce: 0.2679101526737213 (0.24483399617671966) | > loss_mel_ce: 6.149561405181885 (5.736919766664504) | > loss: 0.025466157123446465 (0.023737119473516942) | > current_lr: 5e-06 | > step_time: 0.114 (0.1296315431594849) | > loader_time: 0.0048 (0.005968722343444824)  --> TIME: 2025-02-23 19:54:13 -- STEP: 300/36938 -- GLOBAL_STEP: 300 | > loss_text_ce: 0.22184374928474426 (0.24423698708415031) | > loss_mel_ce: 6.279453754425049 (nan) | > loss: 0.025798801332712173 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13044204711914065) | > loader_time: 0.0075 (0.005868601004282632)  --> TIME: 2025-02-23 19:54:21 -- STEP: 350/36938 -- GLOBAL_STEP: 350 | > loss_text_ce: 0.22740057110786438 (0.24428967880351202) | > loss_mel_ce: 5.1323957443237305 (nan) | > loss: 0.02126903459429741 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13012508051736021) | > loader_time: 0.0063 (0.0058034447261265335)  --> TIME: 2025-02-23 19:54:30 -- STEP: 400/36938 -- GLOBAL_STEP: 400 | > loss_text_ce: 0.22581817209720612 (0.24355372071266174) | > loss_mel_ce: 5.975616455078125 (nan) | > loss: 0.02460886910557747 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.1302898162603379) | > loader_time: 0.0045 (0.005702230334281921)  --> TIME: 2025-02-23 19:54:39 -- STEP: 450/36938 -- GLOBAL_STEP: 450 | > loss_text_ce: 0.2414136826992035 (0.24399172481563355) | > loss_mel_ce: 5.987605571746826 (nan) | > loss: 0.024718331173062325 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.12984177059597443) | > loader_time: 0.0043 (0.005601560274759928)  --> TIME: 2025-02-23 19:54:48 -- STEP: 500/36938 -- GLOBAL_STEP: 500 | > loss_text_ce: 0.2984134554862976 (0.24359726244211197) | > loss_mel_ce: 4.4764885902404785 (nan) | > loss: 0.01894802413880825 (nan) | > current_lr: 5e-06 | > step_time: 0.1071 (0.12997542572021492) | > loader_time: 0.0042 (0.00553192377090454)  --> TIME: 2025-02-23 19:54:57 -- STEP: 550/36938 -- GLOBAL_STEP: 550 | > loss_text_ce: 0.2604687511920929 (0.24363397982987492) | > loss_mel_ce: 2.467822313308716 (nan) | > loss: 0.010826552286744118 (nan) | > current_lr: 5e-06 | > step_time: 0.1081 (0.13022099234841092) | > loader_time: 0.0042 (0.005468038212169299)  --> TIME: 2025-02-23 19:55:07 -- STEP: 600/36938 -- GLOBAL_STEP: 600 | > loss_text_ce: 0.234386146068573 (0.24306906128923098) | > loss_mel_ce: 5.98486852645874 (nan) | > loss: 0.024679582566022873 (nan) | > current_lr: 5e-06 | > step_time: 0.1523 (0.13096991022427884) | > loader_time: 0.006 (0.005418458382288614)  --> TIME: 2025-02-23 19:55:16 -- STEP: 650/36938 -- GLOBAL_STEP: 650 | > loss_text_ce: 0.23565340042114258 (0.24295926261406678) | > loss_mel_ce: 5.984106063842773 (nan) | > loss: 0.024681586772203445 (nan) | > current_lr: 5e-06 | > step_time: 0.1591 (0.13076183905968303) | > loader_time: 0.0045 (0.005358713223383972)  --> TIME: 2025-02-23 19:55:24 -- STEP: 700/36938 -- GLOBAL_STEP: 700 | > loss_text_ce: 0.23754766583442688 (0.24266595565846988) | > loss_mel_ce: 6.3994140625 (nan) | > loss: 0.02633715234696865 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13061535903385707) | > loader_time: 0.0044 (0.005309779984610417)  --> TIME: 2025-02-23 19:55:33 -- STEP: 750/36938 -- GLOBAL_STEP: 750 | > loss_text_ce: 0.2192031294107437 (0.24267987205584843) | > loss_mel_ce: 6.107813835144043 (nan) | > loss: 0.025107210502028465 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13042682234446198) | > loader_time: 0.0045 (0.005281146367390944)  --> TIME: 2025-02-23 19:55:43 -- STEP: 800/36938 -- GLOBAL_STEP: 800 | > loss_text_ce: 0.22583603858947754 (0.24239359831437468) | > loss_mel_ce: 5.282857418060303 (nan) | > loss: 0.021859897300601006 (nan) | > current_lr: 5e-06 | > step_time: 0.1828 (0.13120710521936402) | > loader_time: 0.0047 (0.005263389348983758)  --> TIME: 2025-02-23 19:55:52 -- STEP: 850/36938 -- GLOBAL_STEP: 850 | > loss_text_ce: 0.2377403825521469 (0.24197104746804518) | > loss_mel_ce: 5.165714263916016 (nan) | > loss: 0.02144228294491768 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.13152955475975472) | > loader_time: 0.0047 (0.005244121271021219)  --> TIME: 2025-02-23 19:56:01 -- STEP: 900/36938 -- GLOBAL_STEP: 900 | > loss_text_ce: 0.24252499639987946 (0.2419161834153864) | > loss_mel_ce: 5.90188455581665 (nan) | > loss: 0.0243825800716877 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13159638722737618) | > loader_time: 0.0043 (0.005213407940334738)  --> TIME: 2025-02-23 19:56:11 -- STEP: 950/36938 -- GLOBAL_STEP: 950 | > loss_text_ce: 0.21697761118412018 (0.24211569767249258) | > loss_mel_ce: 5.754200458526611 (nan) | > loss: 0.02369515225291252 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.13185570691761211) | > loader_time: 0.0049 (0.0051948196009585675)  --> TIME: 2025-02-23 19:56:20 -- STEP: 1000/36938 -- GLOBAL_STEP: 1000 | > loss_text_ce: 0.23077474534511566 (0.24221608307957648) | > loss_mel_ce: 5.455347537994385 (nan) | > loss: 0.022563979029655457 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.1317103896141052) | > loader_time: 0.0043 (0.005164421081542963)  --> TIME: 2025-02-23 19:56:29 -- STEP: 1050/36938 -- GLOBAL_STEP: 1050 | > loss_text_ce: 0.24072915315628052 (0.2421281394078618) | > loss_mel_ce: 6.1908955574035645 (nan) | > loss: 0.025522323325276375 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13193071092878067) | > loader_time: 0.0042 (0.0051346640359787666)  --> TIME: 2025-02-23 19:56:38 -- STEP: 1100/36938 -- GLOBAL_STEP: 1100 | > loss_text_ce: 0.2527749836444855 (0.2417579992521893) | > loss_mel_ce: 6.444157123565674 (nan) | > loss: 0.026575129479169846 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13212513251738114) | > loader_time: 0.0043 (0.005105084722692313)  --> TIME: 2025-02-23 19:56:47 -- STEP: 1150/36938 -- GLOBAL_STEP: 1150 | > loss_text_ce: 0.2573557496070862 (0.2417410402064738) | > loss_mel_ce: 1.7146693468093872 (nan) | > loss: 0.007825497537851334 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.1320660518563312) | > loader_time: 0.0041 (0.005073960138403849)  --> TIME: 2025-02-23 19:56:56 -- STEP: 1200/36938 -- GLOBAL_STEP: 1200 | > loss_text_ce: 0.25822368264198303 (0.2415178820490837) | > loss_mel_ce: 4.612242698669434 (nan) | > loss: 0.019327247515320778 (nan) | > current_lr: 5e-06 | > step_time: 0.108 (0.13223091304302223) | > loader_time: 0.0041 (0.0050499292214711495)  --> TIME: 2025-02-23 19:57:06 -- STEP: 1250/36938 -- GLOBAL_STEP: 1250 | > loss_text_ce: 0.26769229769706726 (0.2412845852136612) | > loss_mel_ce: 4.36726713180542 (nan) | > loss: 0.018392696976661682 (nan) | > current_lr: 5e-06 | > step_time: 0.1078 (0.13260733261108404) | > loader_time: 0.0043 (0.005031859588623045)  --> TIME: 2025-02-23 19:57:15 -- STEP: 1300/36938 -- GLOBAL_STEP: 1300 | > loss_text_ce: 0.2300138771533966 (0.24119697034358978) | > loss_mel_ce: 5.189810276031494 (nan) | > loss: 0.02150723896920681 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.1326439459507283) | > loader_time: 0.0043 (0.005008874122913067)  --> TIME: 2025-02-23 19:57:25 -- STEP: 1350/36938 -- GLOBAL_STEP: 1350 | > loss_text_ce: 0.21901676058769226 (0.2410650469417925) | > loss_mel_ce: 6.596798896789551 (nan) | > loss: 0.027046889066696167 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13274314491837116) | > loader_time: 0.0044 (0.004992255281519007)  --> TIME: 2025-02-23 19:57:34 -- STEP: 1400/36938 -- GLOBAL_STEP: 1400 | > loss_text_ce: 0.22454999387264252 (0.24079358393592493) | > loss_mel_ce: 5.462039470672607 (nan) | > loss: 0.02256583236157894 (nan) | > current_lr: 5e-06 | > step_time: 0.1702 (0.13270577345575604) | > loader_time: 0.0045 (0.004974461623600554)  --> TIME: 2025-02-23 19:57:43 -- STEP: 1450/36938 -- GLOBAL_STEP: 1450 | > loss_text_ce: 0.2551250159740448 (0.24064550918751748) | > loss_mel_ce: 3.1194121837615967 (nan) | > loss: 0.013391021639108658 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13291429848506545) | > loader_time: 0.0045 (0.004959989745041421)  --> TIME: 2025-02-23 19:57:52 -- STEP: 1500/36938 -- GLOBAL_STEP: 1500 | > loss_text_ce: 0.24823863804340363 (0.24050281230608622) | > loss_mel_ce: 5.504461288452148 (nan) | > loss: 0.022828174754977226 (nan) | > current_lr: 5e-06 | > step_time: 0.1075 (0.13296172746022555) | > loader_time: 0.0042 (0.004946496486663821)  --> TIME: 2025-02-23 19:58:02 -- STEP: 1550/36938 -- GLOBAL_STEP: 1550 | > loss_text_ce: 0.250394731760025 (0.2403880096827784) | > loss_mel_ce: 4.691243648529053 (nan) | > loss: 0.01960967667400837 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13304330041331647) | > loader_time: 0.0045 (0.004937077799151026)  --> TIME: 2025-02-23 19:58:11 -- STEP: 1600/36938 -- GLOBAL_STEP: 1600 | > loss_text_ce: 0.23332878947257996 (0.2401656518317759) | > loss_mel_ce: 6.019513130187988 (nan) | > loss: 0.024812866002321243 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13310970798134827) | > loader_time: 0.0043 (0.00492298752069474)  --> TIME: 2025-02-23 19:58:20 -- STEP: 1650/36938 -- GLOBAL_STEP: 1650 | > loss_text_ce: 0.2462499886751175 (0.24009595196355474) | > loss_mel_ce: 4.771093845367432 (nan) | > loss: 0.019910097122192383 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.133181181387468) | > loader_time: 0.0044 (0.004913723974516903)  --> TIME: 2025-02-23 19:58:29 -- STEP: 1700/36938 -- GLOBAL_STEP: 1700 | > loss_text_ce: 0.2862187623977661 (0.2400575254068655) | > loss_mel_ce: 6.968006134033203 (nan) | > loss: 0.028786607086658478 (nan) | > current_lr: 5e-06 | > step_time: 0.1059 (0.1331968921773574) | > loader_time: 0.004 (0.004905688482172354)  --> TIME: 2025-02-23 19:58:39 -- STEP: 1750/36938 -- GLOBAL_STEP: 1750 | > loss_text_ce: 0.2887239456176758 (0.23996807592255728) | > loss_mel_ce: 5.960760116577148 (nan) | > loss: 0.024799540638923645 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13330433409554612) | > loader_time: 0.0042 (0.00489559023720878)  --> TIME: 2025-02-23 19:58:48 -- STEP: 1800/36938 -- GLOBAL_STEP: 1800 | > loss_text_ce: 0.2316335290670395 (0.23981269212232695) | > loss_mel_ce: 5.697699546813965 (nan) | > loss: 0.02352910116314888 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13335331612163115) | > loader_time: 0.0045 (0.004884523020850293)  --> TIME: 2025-02-23 19:58:57 -- STEP: 1850/36938 -- GLOBAL_STEP: 1850 | > loss_text_ce: 0.23628123104572296 (0.23973421468928052) | > loss_mel_ce: 5.9849138259887695 (nan) | > loss: 0.024687284603714943 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1333353737238291) | > loader_time: 0.0043 (0.004875521917600894)  --> TIME: 2025-02-23 19:59:06 -- STEP: 1900/36938 -- GLOBAL_STEP: 1900 | > loss_text_ce: 0.23327302932739258 (0.23955639900345552) | > loss_mel_ce: 4.4306135177612305 (nan) | > loss: 0.018507488071918488 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1335208865215905) | > loader_time: 0.0043 (0.004864625679819212)  --> TIME: 2025-02-23 19:59:15 -- STEP: 1950/36938 -- GLOBAL_STEP: 1950 | > loss_text_ce: 0.2263602912425995 (0.2395116471632933) | > loss_mel_ce: 5.822764873504639 (nan) | > loss: 0.024004466831684113 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.13347364535698542) | > loader_time: 0.0044 (0.004852458758231926)  --> TIME: 2025-02-23 19:59:24 -- STEP: 2000/36938 -- GLOBAL_STEP: 2000 | > loss_text_ce: 0.22934028506278992 (0.23953493656218053) | > loss_mel_ce: 4.390299320220947 (nan) | > loss: 0.018331903964281082 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13337129843235032) | > loader_time: 0.0041 (0.004843762755393987)  --> TIME: 2025-02-23 19:59:34 -- STEP: 2050/36938 -- GLOBAL_STEP: 2050 | > loss_text_ce: 0.22521550953388214 (0.23936408692743721) | > loss_mel_ce: 1.2244466543197632 (nan) | > loss: 0.005752628203481436 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.1334982324228056) | > loader_time: 0.0042 (0.004832810541478603)  --> TIME: 2025-02-23 19:59:43 -- STEP: 2100/36938 -- GLOBAL_STEP: 2100 | > loss_text_ce: 0.2341766655445099 (0.23923760257306553) | > loss_mel_ce: 5.803768157958984 (nan) | > loss: 0.02396010048687458 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13345641522180482) | > loader_time: 0.0042 (0.004824324221838093)  --> TIME: 2025-02-23 19:59:52 -- STEP: 2150/36938 -- GLOBAL_STEP: 2150 | > loss_text_ce: 0.2479807585477829 (0.2391196659692498) | > loss_mel_ce: 6.011303424835205 (nan) | > loss: 0.024838430806994438 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.1335040537146636) | > loader_time: 0.0042 (0.004817742192467982)  --> TIME: 2025-02-23 20:00:01 -- STEP: 2200/36938 -- GLOBAL_STEP: 2200 | > loss_text_ce: 0.22765624523162842 (0.23911809035322884) | > loss_mel_ce: 5.928956508636475 (nan) | > loss: 0.024431005120277405 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.13353218447078372) | > loader_time: 0.0044 (0.004807589270851834)  --> TIME: 2025-02-23 20:00:11 -- STEP: 2250/36938 -- GLOBAL_STEP: 2250 | > loss_text_ce: 0.2239999920129776 (0.23901541105243895) | > loss_mel_ce: 5.964926242828369 (nan) | > loss: 0.024559233337640762 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.13361979113684772) | > loader_time: 0.0047 (0.004799069722493497)  --> TIME: 2025-02-23 20:00:20 -- STEP: 2300/36938 -- GLOBAL_STEP: 2300 | > loss_text_ce: 0.221811443567276 (0.23888468738483346) | > loss_mel_ce: 5.663643836975098 (nan) | > loss: 0.02335498295724392 (nan) | > current_lr: 5e-06 | > step_time: 0.1527 (0.13368731654208654) | > loader_time: 0.0045 (0.00479037222654924)  --> TIME: 2025-02-23 20:00:29 -- STEP: 2350/36938 -- GLOBAL_STEP: 2350 | > loss_text_ce: 0.21835756301879883 (0.23868369711840406) | > loss_mel_ce: 5.7313385009765625 (nan) | > loss: 0.023609906435012817 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.13377489130547715) | > loader_time: 0.0045 (0.004781314362870893)  --> TIME: 2025-02-23 20:00:39 -- STEP: 2400/36938 -- GLOBAL_STEP: 2400 | > loss_text_ce: 0.24101974070072174 (0.23852917531505227) | > loss_mel_ce: 4.832592487335205 (nan) | > loss: 0.020133383572101593 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13388539542754507) | > loader_time: 0.0044 (0.004775633613268538)  --> TIME: 2025-02-23 20:00:48 -- STEP: 2450/36938 -- GLOBAL_STEP: 2450 | > loss_text_ce: 0.23779296875 (0.2384358816122522) | > loss_mel_ce: 4.17003059387207 (nan) | > loss: 0.01749136485159397 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.1339147930729148) | > loader_time: 0.0044 (0.004770611256969222)  --> TIME: 2025-02-23 20:00:58 -- STEP: 2500/36938 -- GLOBAL_STEP: 2500 | > loss_text_ce: 0.21472014486789703 (0.2382677674770355) | > loss_mel_ce: 5.705721378326416 (nan) | > loss: 0.023493817076086998 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.1340970146179201) | > loader_time: 0.0067 (0.004765549659729003)  --> TIME: 2025-02-23 20:01:07 -- STEP: 2550/36938 -- GLOBAL_STEP: 2550 | > loss_text_ce: 0.3176785707473755 (0.23813308438834022) | > loss_mel_ce: 0.8060302734375 (nan) | > loss: 0.00445916224271059 (nan) | > current_lr: 5e-06 | > step_time: 0.1063 (0.1341232987946157) | > loader_time: 0.0042 (0.004760051241108014)  --> TIME: 2025-02-23 20:01:17 -- STEP: 2600/36938 -- GLOBAL_STEP: 2600 | > loss_text_ce: 0.2220161259174347 (0.23798556931316853) | > loss_mel_ce: 5.754416465759277 (nan) | > loss: 0.023716004565358162 (nan) | > current_lr: 5e-06 | > step_time: 0.1891 (0.13412709382864169) | > loader_time: 0.0053 (0.004755563919360821)  --> TIME: 2025-02-23 20:01:25 -- STEP: 2650/36938 -- GLOBAL_STEP: 2650 | > loss_text_ce: 0.24388021230697632 (0.2379593570502299) | > loss_mel_ce: 5.9921875 (nan) | > loss: 0.024746302515268326 (nan) | > current_lr: 5e-06 | > step_time: 0.1079 (0.13404527646190742) | > loader_time: 0.0045 (0.004749496028108418)  --> TIME: 2025-02-23 20:01:34 -- STEP: 2700/36938 -- GLOBAL_STEP: 2700 | > loss_text_ce: 0.21727271378040314 (0.2379447581756998) | > loss_mel_ce: 5.622198581695557 (nan) | > loss: 0.02317250706255436 (nan) | > current_lr: 5e-06 | > step_time: 0.1828 (0.13397978411780492) | > loader_time: 0.0102 (0.004746448198954271)  --> TIME: 2025-02-23 20:01:44 -- STEP: 2750/36938 -- GLOBAL_STEP: 2750 | > loss_text_ce: 0.23819443583488464 (0.23781163729862734) | > loss_mel_ce: 5.7187957763671875 (nan) | > loss: 0.023638851940631866 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1340890274914831) | > loader_time: 0.005 (0.004742579893632377)  --> TIME: 2025-02-23 20:01:53 -- STEP: 2800/36938 -- GLOBAL_STEP: 2800 | > loss_text_ce: 0.23881249129772186 (0.23772547369024583) | > loss_mel_ce: 5.689094066619873 (nan) | > loss: 0.023523440584540367 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1341376670769286) | > loader_time: 0.0045 (0.004738367966243206)  --> TIME: 2025-02-23 20:02:03 -- STEP: 2850/36938 -- GLOBAL_STEP: 2850 | > loss_text_ce: 0.22265087068080902 (0.23758894111503634) | > loss_mel_ce: 4.823034763336182 (nan) | > loss: 0.020022563636302948 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.13420970004901575) | > loader_time: 0.0046 (0.004734311103820808)  --> TIME: 2025-02-23 20:02:12 -- STEP: 2900/36938 -- GLOBAL_STEP: 2900 | > loss_text_ce: 0.2307845652103424 (0.2375651829612666) | > loss_mel_ce: 5.353255748748779 (nan) | > loss: 0.022158890962600708 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.13410865389067586) | > loader_time: 0.0045 (0.00473004456224113)  --> TIME: 2025-02-23 20:02:21 -- STEP: 2950/36938 -- GLOBAL_STEP: 2950 | > loss_text_ce: 0.2306249886751175 (0.2375789414219937) | > loss_mel_ce: 4.880085468292236 (nan) | > loss: 0.02028059959411621 (nan) | > current_lr: 5e-06 | > step_time: 0.151 (0.13406519170534845) | > loader_time: 0.0043 (0.004725994982961883)  --> TIME: 2025-02-23 20:02:30 -- STEP: 3000/36938 -- GLOBAL_STEP: 3000 | > loss_text_ce: 0.22631578147411346 (0.23748440001408258) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13404578344027238) | > loader_time: 0.0044 (0.004720748424530033)  --> TIME: 2025-02-23 20:02:39 -- STEP: 3050/36938 -- GLOBAL_STEP: 3050 | > loss_text_ce: 0.21945512294769287 (0.23737039264108314) | > loss_mel_ce: 5.607132911682129 (nan) | > loss: 0.023121383041143417 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13403739444545099) | > loader_time: 0.0045 (0.004715584848747882)  --> TIME: 2025-02-23 20:02:48 -- STEP: 3100/36938 -- GLOBAL_STEP: 3100 | > loss_text_ce: 0.21457473933696747 (0.23728379797070256) | > loss_mel_ce: 5.936711311340332 (nan) | > loss: 0.024409867823123932 (nan) | > current_lr: 5e-06 | > step_time: 0.1893 (0.13409090895806638) | > loader_time: 0.0045 (0.004711283329994451)  --> TIME: 2025-02-23 20:02:58 -- STEP: 3150/36938 -- GLOBAL_STEP: 3150 | > loss_text_ce: 0.22073112428188324 (0.23718312967391242) | > loss_mel_ce: 5.183116436004639 (nan) | > loss: 0.021443841978907585 (nan) | > current_lr: 5e-06 | > step_time: 0.1514 (0.13421671912783717) | > loader_time: 0.0045 (0.004706997644333618)  --> TIME: 2025-02-23 20:03:07 -- STEP: 3200/36938 -- GLOBAL_STEP: 3200 | > loss_text_ce: 0.21423913538455963 (0.23709365169517696) | > loss_mel_ce: 4.634726524353027 (nan) | > loss: 0.019241929054260254 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.1341515596956021) | > loader_time: 0.0042 (0.004705699086189273)  --> TIME: 2025-02-23 20:03:16 -- STEP: 3250/36938 -- GLOBAL_STEP: 3250 | > loss_text_ce: 0.21685738861560822 (0.23705290759068268) | > loss_mel_ce: 5.711526870727539 (nan) | > loss: 0.023525336757302284 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.13414669931851958) | > loader_time: 0.0044 (0.004703088026780356)  --> TIME: 2025-02-23 20:03:25 -- STEP: 3300/36938 -- GLOBAL_STEP: 3300 | > loss_text_ce: 0.27611109614372253 (0.23700659913547112) | > loss_mel_ce: 2.9010987281799316 (nan) | > loss: 0.012607976794242859 (nan) | > current_lr: 5e-06 | > step_time: 0.1082 (0.13413295536330316) | > loader_time: 0.0043 (0.004699263572692879)  --> TIME: 2025-02-23 20:03:34 -- STEP: 3350/36938 -- GLOBAL_STEP: 3350 | > loss_text_ce: 0.2392897754907608 (0.23698915095026812) | > loss_mel_ce: 5.875289440155029 (nan) | > loss: 0.024264205247163773 (nan) | > current_lr: 5e-06 | > step_time: 0.1017 (0.13405392540035027) | > loader_time: 0.0043 (0.00469400000216357)  --> TIME: 2025-02-23 20:03:43 -- STEP: 3400/36938 -- GLOBAL_STEP: 3400 | > loss_text_ce: 0.22221697866916656 (0.23689145121942548) | > loss_mel_ce: 5.489360809326172 (nan) | > loss: 0.0226649921387434 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13403117299079972) | > loader_time: 0.0044 (0.0046885651700637)  --> TIME: 2025-02-23 20:03:52 -- STEP: 3450/36938 -- GLOBAL_STEP: 3450 | > loss_text_ce: 0.25407811999320984 (0.2367752715513326) | > loss_mel_ce: 4.78593111038208 (nan) | > loss: 0.02000003680586815 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13407507385032844) | > loader_time: 0.0043 (0.0046859510394110255)  --> TIME: 2025-02-23 20:04:02 -- STEP: 3500/36938 -- GLOBAL_STEP: 3500 | > loss_text_ce: 0.2321944385766983 (0.2366458986571857) | > loss_mel_ce: 5.931196689605713 (nan) | > loss: 0.02445790357887745 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.1341028897421708) | > loader_time: 0.0044 (0.0046823784283229345)  --> TIME: 2025-02-23 20:04:11 -- STEP: 3550/36938 -- GLOBAL_STEP: 3550 | > loss_text_ce: 0.2226874828338623 (0.2365650557002551) | > loss_mel_ce: 2.680712938308716 (nan) | > loss: 0.011521430686116219 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13413057347418622) | > loader_time: 0.0043 (0.004679446690519097)  --> TIME: 2025-02-23 20:04:20 -- STEP: 3600/36938 -- GLOBAL_STEP: 3600 | > loss_text_ce: 0.24259614944458008 (0.2364777000165648) | > loss_mel_ce: 5.3372578620910645 (nan) | > loss: 0.022142279893159866 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13412835386064384) | > loader_time: 0.0045 (0.00467802180184259)  --> TIME: 2025-02-23 20:04:29 -- STEP: 3650/36938 -- GLOBAL_STEP: 3650 | > loss_text_ce: 0.22046296298503876 (0.2363760157970533) | > loss_mel_ce: 6.1258225440979 (nan) | > loss: 0.025183673948049545 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.1340969297983882) | > loader_time: 0.0046 (0.004675153510211271)  --> TIME: 2025-02-23 20:04:38 -- STEP: 3700/36938 -- GLOBAL_STEP: 3700 | > loss_text_ce: 0.24523809552192688 (0.2363259992607542) | > loss_mel_ce: 5.60122537612915 (nan) | > loss: 0.023200254887342453 (nan) | > current_lr: 5e-06 | > step_time: 0.1123 (0.1340226580645593) | > loader_time: 0.0043 (0.004670922111820536)  --> TIME: 2025-02-23 20:04:48 -- STEP: 3750/36938 -- GLOBAL_STEP: 3750 | > loss_text_ce: 0.2409375011920929 (0.23620567741791407) | > loss_mel_ce: 5.1081671714782715 (nan) | > loss: 0.021226607263088226 (nan) | > current_lr: 5e-06 | > step_time: 0.1268 (0.1340991809209195) | > loader_time: 0.0042 (0.0046676925023396865)  --> TIME: 2025-02-23 20:04:57 -- STEP: 3800/36938 -- GLOBAL_STEP: 3800 | > loss_text_ce: 0.22955356538295746 (0.23613161525443982) | > loss_mel_ce: 5.611776828765869 (nan) | > loss: 0.023179884999990463 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.13411712238663184) | > loader_time: 0.0044 (0.004665661611055079)  --> TIME: 2025-02-23 20:05:06 -- STEP: 3850/36938 -- GLOBAL_STEP: 3850 | > loss_text_ce: 0.21719695627689362 (0.2360489322612812) | > loss_mel_ce: 5.209881782531738 (nan) | > loss: 0.021536028012633324 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.13419395793568065) | > loader_time: 0.0045 (0.004662304296121974)  --> TIME: 2025-02-23 20:05:15 -- STEP: 3900/36938 -- GLOBAL_STEP: 3900 | > loss_text_ce: 0.2149648368358612 (0.2360456873018008) | > loss_mel_ce: 4.720776557922363 (nan) | > loss: 0.01958627626299858 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.1341650459705261) | > loader_time: 0.0045 (0.004659584179902694)  --> TIME: 2025-02-23 20:05:25 -- STEP: 3950/36938 -- GLOBAL_STEP: 3950 | > loss_text_ce: 0.22802557051181793 (0.23595015371902078) | > loss_mel_ce: 5.39652681350708 (nan) | > loss: 0.022319654002785683 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13417275851285967) | > loader_time: 0.0043 (0.0046560195126111046)  --> TIME: 2025-02-23 20:05:34 -- STEP: 4000/36938 -- GLOBAL_STEP: 4000 | > loss_text_ce: 0.22053711116313934 (0.23588655112311244) | > loss_mel_ce: 5.577641010284424 (nan) | > loss: 0.023008644580841064 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.13416424489021359) | > loader_time: 0.0044 (0.004653364658355718)  --> TIME: 2025-02-23 20:05:43 -- STEP: 4050/36938 -- GLOBAL_STEP: 4050 | > loss_text_ce: 0.21136994659900665 (0.23575320771079006) | > loss_mel_ce: 5.233156204223633 (nan) | > loss: 0.021605264395475388 (nan) | > current_lr: 5e-06 | > step_time: 0.1923 (0.13423606448703398) | > loader_time: 0.005 (0.004650304229171193)  --> TIME: 2025-02-23 20:05:52 -- STEP: 4100/36938 -- GLOBAL_STEP: 4100 | > loss_text_ce: 0.22684209048748016 (0.2357546095368339) | > loss_mel_ce: 4.943838596343994 (nan) | > loss: 0.020518574863672256 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13415808613707436) | > loader_time: 0.0041 (0.0046466544779335634)  --> TIME: 2025-02-23 20:06:01 -- STEP: 4150/36938 -- GLOBAL_STEP: 4150 | > loss_text_ce: 0.23208928108215332 (0.23575818379600366) | > loss_mel_ce: 5.408832550048828 (nan) | > loss: 0.022384610027074814 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13405677732214924) | > loader_time: 0.0042 (0.004643657293664409)  --> TIME: 2025-02-23 20:06:10 -- STEP: 4200/36938 -- GLOBAL_STEP: 4200 | > loss_text_ce: 0.21307413280010223 (0.23566012669177286) | > loss_mel_ce: 4.175939083099365 (nan) | > loss: 0.01741672120988369 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13410670450755582) | > loader_time: 0.0042 (0.0046410163811275)  --> TIME: 2025-02-23 20:06:20 -- STEP: 4250/36938 -- GLOBAL_STEP: 4250 | > loss_text_ce: 0.3089583218097687 (0.23555000355313804) | > loss_mel_ce: 2.966519355773926 (nan) | > loss: 0.012997928075492382 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.1341826864691347) | > loader_time: 0.0042 (0.00463790904774386)  --> TIME: 2025-02-23 20:06:29 -- STEP: 4300/36938 -- GLOBAL_STEP: 4300 | > loss_text_ce: 0.22499999403953552 (0.23546605340616653) | > loss_mel_ce: 5.384457111358643 (nan) | > loss: 0.022259751334786415 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13414618985597532) | > loader_time: 0.0043 (0.0046348703739255035)  --> TIME: 2025-02-23 20:06:38 -- STEP: 4350/36938 -- GLOBAL_STEP: 4350 | > loss_text_ce: 0.23696427047252655 (0.23534321569163227) | > loss_mel_ce: 4.926888942718506 (nan) | > loss: 0.020491482689976692 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13417225815784026) | > loader_time: 0.0042 (0.004632011446459545)  --> TIME: 2025-02-23 20:06:47 -- STEP: 4400/36938 -- GLOBAL_STEP: 4400 | > loss_text_ce: 0.2438858598470688 (0.2352441104670817) | > loss_mel_ce: 5.478302478790283 (nan) | > loss: 0.022707099094986916 (nan) | > current_lr: 5e-06 | > step_time: 0.1067 (0.13415244958617525) | > loader_time: 0.0043 (0.0046296469189904006)  --> TIME: 2025-02-23 20:06:57 -- STEP: 4450/36938 -- GLOBAL_STEP: 4450 | > loss_text_ce: 0.22024038434028625 (0.23513412631964414) | > loss_mel_ce: 4.653341293334961 (nan) | > loss: 0.019339611753821373 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.13422047422173364) | > loader_time: 0.0045 (0.004627820829327196)  --> TIME: 2025-02-23 20:07:06 -- STEP: 4500/36938 -- GLOBAL_STEP: 4500 | > loss_text_ce: 0.21265242993831635 (0.23502716001868248) | > loss_mel_ce: 5.5426201820373535 (nan) | > loss: 0.02283838391304016 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13425703387790294) | > loader_time: 0.0044 (0.004625589317745642)  --> TIME: 2025-02-23 20:07:15 -- STEP: 4550/36938 -- GLOBAL_STEP: 4550 | > loss_text_ce: 0.24389705061912537 (0.23493501836454475) | > loss_mel_ce: 4.865106582641602 (nan) | > loss: 0.020273825153708458 (nan) | > current_lr: 5e-06 | > step_time: 0.1037 (0.1342399831918573) | > loader_time: 0.0043 (0.004622735453175984)  --> TIME: 2025-02-23 20:07:24 -- STEP: 4600/36938 -- GLOBAL_STEP: 4600 | > loss_text_ce: 0.23036058247089386 (0.23483854245880376) | > loss_mel_ce: 5.487511157989502 (nan) | > loss: 0.02268996834754944 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13426147046296522) | > loader_time: 0.0043 (0.004620489607686591)  --> TIME: 2025-02-23 20:07:33 -- STEP: 4650/36938 -- GLOBAL_STEP: 4650 | > loss_text_ce: 0.22437499463558197 (0.23476614639323246) | > loss_mel_ce: 4.581720352172852 (nan) | > loss: 0.019071806222200394 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.1342512678843675) | > loader_time: 0.0045 (0.004618060973382775)  --> TIME: 2025-02-23 20:07:43 -- STEP: 4700/36938 -- GLOBAL_STEP: 4700 | > loss_text_ce: 0.22085192799568176 (0.23465108484980907) | > loss_mel_ce: 5.5188889503479 (nan) | > loss: 0.022776750847697258 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13430183537462959) | > loader_time: 0.0044 (0.004616237143252768)  --> TIME: 2025-02-23 20:07:52 -- STEP: 4750/36938 -- GLOBAL_STEP: 4750 | > loss_text_ce: 0.22045138478279114 (0.23454804470350868) | > loss_mel_ce: 4.0587897300720215 (nan) | > loss: 0.01698111556470394 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13434021126596576) | > loader_time: 0.0042 (0.004614363068028507)  --> TIME: 2025-02-23 20:08:02 -- STEP: 4800/36938 -- GLOBAL_STEP: 4800 | > loss_text_ce: 0.23113176226615906 (0.23440533409826458) | > loss_mel_ce: 5.634006500244141 (nan) | > loss: 0.023274358361959457 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1344327688713871) | > loader_time: 0.0043 (0.0046129447718461425)  --> TIME: 2025-02-23 20:08:11 -- STEP: 4850/36938 -- GLOBAL_STEP: 4850 | > loss_text_ce: 0.23426339030265808 (0.2343155984810947) | > loss_mel_ce: 4.656211853027344 (nan) | > loss: 0.019406648352742195 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13445406820356257) | > loader_time: 0.0042 (0.004611665243955006)  --> TIME: 2025-02-23 20:08:20 -- STEP: 4900/36938 -- GLOBAL_STEP: 4900 | > loss_text_ce: 0.20913098752498627 (0.23423621465965194) | > loss_mel_ce: 5.364877700805664 (nan) | > loss: 0.022119082510471344 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.1344191430539504) | > loader_time: 0.0043 (0.004609727129644286)  --> TIME: 2025-02-23 20:08:30 -- STEP: 4950/36938 -- GLOBAL_STEP: 4950 | > loss_text_ce: 0.2159469574689865 (0.23413212597069114) | > loss_mel_ce: 5.460475921630859 (nan) | > loss: 0.022525489330291748 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.134427291166903) | > loader_time: 0.0046 (0.004608150010157125)  --> TIME: 2025-02-23 20:08:39 -- STEP: 5000/36938 -- GLOBAL_STEP: 5000 | > loss_text_ce: 0.23002499341964722 (0.23403403981029988) | > loss_mel_ce: 5.246058464050293 (nan) | > loss: 0.021730490028858185 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13444998116493254) | > loader_time: 0.0048 (0.004607376050949104)  --> TIME: 2025-02-23 20:08:48 -- STEP: 5050/36938 -- GLOBAL_STEP: 5050 | > loss_text_ce: 0.24573864042758942 (0.23395920169825601) | > loss_mel_ce: 3.2694904804229736 (nan) | > loss: 0.013949323445558548 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.1344566536893942) | > loader_time: 0.0045 (0.004605765059442812)  --> TIME: 2025-02-23 20:08:58 -- STEP: 5100/36938 -- GLOBAL_STEP: 5100 | > loss_text_ce: 0.21613968908786774 (0.23382795656136438) | > loss_mel_ce: 5.411184310913086 (nan) | > loss: 0.02233065292239189 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.1345438601456441) | > loader_time: 0.0046 (0.004603783009099034)  --> TIME: 2025-02-23 20:09:07 -- STEP: 5150/36938 -- GLOBAL_STEP: 5150 | > loss_text_ce: 0.208217591047287 (0.23372655944048779) | > loss_mel_ce: 5.453091621398926 (nan) | > loss: 0.02246551401913166 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13452531448845767) | > loader_time: 0.0044 (0.004601918007563627)  --> TIME: 2025-02-23 20:09:16 -- STEP: 5200/36938 -- GLOBAL_STEP: 5200 | > loss_text_ce: 0.21973039209842682 (0.23367738555830259) | > loss_mel_ce: 4.282372951507568 (nan) | > loss: 0.017865490168333054 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13450240827523788) | > loader_time: 0.0046 (0.004600273370742807)  --> TIME: 2025-02-23 20:09:25 -- STEP: 5250/36938 -- GLOBAL_STEP: 5250 | > loss_text_ce: 0.21622352302074432 (0.23358963922943388) | > loss_mel_ce: 5.7945122718811035 (nan) | > loss: 0.02385212853550911 (nan) | > current_lr: 5e-06 | > step_time: 0.1443 (0.13452115685599225) | > loader_time: 0.0045 (0.004599606377737872)  --> TIME: 2025-02-23 20:09:35 -- STEP: 5300/36938 -- GLOBAL_STEP: 5300 | > loss_text_ce: 0.23026441037654877 (0.23346297075444797) | > loss_mel_ce: 0.2607421875 (nan) | > loss: 0.0019484390504658222 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13451635261751604) | > loader_time: 0.0042 (0.004598768882031719)  --> TIME: 2025-02-23 20:09:44 -- STEP: 5350/36938 -- GLOBAL_STEP: 5350 | > loss_text_ce: 0.2179405689239502 (0.23338024724588216) | > loss_mel_ce: 5.559760093688965 (nan) | > loss: 0.022927384823560715 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.1345263694602756) | > loader_time: 0.0044 (0.0045974059862511265)  --> TIME: 2025-02-23 20:09:53 -- STEP: 5400/36938 -- GLOBAL_STEP: 5400 | > loss_text_ce: 0.2154531180858612 (0.23329501999197183) | > loss_mel_ce: 4.640499591827393 (nan) | > loss: 0.019269654527306557 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13455541058822945) | > loader_time: 0.0044 (0.004596305953131795)  --> TIME: 2025-02-23 20:10:03 -- STEP: 5450/36938 -- GLOBAL_STEP: 5450 | > loss_text_ce: 0.19899122416973114 (0.23318138178062) | > loss_mel_ce: 4.022547245025635 (nan) | > loss: 0.01675213873386383 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.13459335340272435) | > loader_time: 0.0043 (0.004594262061862781)  --> TIME: 2025-02-23 20:10:12 -- STEP: 5500/36938 -- GLOBAL_STEP: 5500 | > loss_text_ce: 0.2684375047683716 (0.23309192486513744) | > loss_mel_ce: 3.964456081390381 (nan) | > loss: 0.016797197982668877 (nan) | > current_lr: 5e-06 | > step_time: 0.1033 (0.13460887986963488) | > loader_time: 0.0044 (0.0045931126421148265)  --> TIME: 2025-02-23 20:10:21 -- STEP: 5550/36938 -- GLOBAL_STEP: 5550 | > loss_text_ce: 0.21682870388031006 (0.23297648950471533) | > loss_mel_ce: 3.08251953125 (nan) | > loss: 0.013092652894556522 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.13464064688295965) | > loader_time: 0.0043 (0.004591948792741119)  --> TIME: 2025-02-23 20:10:31 -- STEP: 5600/36938 -- GLOBAL_STEP: 5600 | > loss_text_ce: 0.21056383848190308 (0.232863606632288) | > loss_mel_ce: 5.357861518859863 (nan) | > loss: 0.022096926346421242 (nan) | > current_lr: 5e-06 | > step_time: 0.1955 (0.13467376836708647) | > loader_time: 0.0046 (0.004590993140425014)  --> TIME: 2025-02-23 20:10:40 -- STEP: 5650/36938 -- GLOBAL_STEP: 5650 | > loss_text_ce: 0.22187499701976776 (0.2327760136232967) | > loss_mel_ce: 4.651859760284424 (nan) | > loss: 0.019340218976140022 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13471206648159897) | > loader_time: 0.0045 (0.004590378524982834)  --> TIME: 2025-02-23 20:10:50 -- STEP: 5700/36938 -- GLOBAL_STEP: 5700 | > loss_text_ce: 0.2108064442873001 (0.23267933246597908) | > loss_mel_ce: 5.038337707519531 (nan) | > loss: 0.020829938352108 (nan) | > current_lr: 5e-06 | > step_time: 0.1483 (0.13473073160439228) | > loader_time: 0.0043 (0.004588743427343547)  --> TIME: 2025-02-23 20:10:59 -- STEP: 5750/36938 -- GLOBAL_STEP: 5750 | > loss_text_ce: 0.21082191169261932 (0.2325678973249767) | > loss_mel_ce: 5.446585178375244 (nan) | > loss: 0.022450029850006104 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.13474361958711067) | > loader_time: 0.0046 (0.004587161271468464)  --> TIME: 2025-02-23 20:11:08 -- STEP: 5800/36938 -- GLOBAL_STEP: 5800 | > loss_text_ce: 0.21035538613796234 (0.2324663349703468) | > loss_mel_ce: 0.76171875 (nan) | > loss: 0.00385743728838861 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13473310807655625) | > loader_time: 0.0043 (0.004585436335925412)  --> TIME: 2025-02-23 20:11:17 -- STEP: 5850/36938 -- GLOBAL_STEP: 5850 | > loss_text_ce: 0.20131979882717133 (0.23239896268161953) | > loss_mel_ce: 5.5119853019714355 (nan) | > loss: 0.02267184667289257 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.13471211971380792) | > loader_time: 0.0049 (0.004584578489645946)  --> TIME: 2025-02-23 20:11:26 -- STEP: 5900/36938 -- GLOBAL_STEP: 5900 | > loss_text_ce: 0.2129812389612198 (0.23231284492601784) | > loss_mel_ce: 5.578317642211914 (nan) | > loss: 0.022981345653533936 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13468174057491786) | > loader_time: 0.0042 (0.004583468800884189)  --> TIME: 2025-02-23 20:11:35 -- STEP: 5950/36938 -- GLOBAL_STEP: 5950 | > loss_text_ce: 0.20546874403953552 (0.2322300416406463) | > loss_mel_ce: 5.3254594802856445 (nan) | > loss: 0.021948128938674927 (nan) | > current_lr: 5e-06 | > step_time: 0.1508 (0.1346702275957382) | > loader_time: 0.0043 (0.004581933021545419)  --> TIME: 2025-02-23 20:11:45 -- STEP: 6000/36938 -- GLOBAL_STEP: 6000 | > loss_text_ce: 0.20638392865657806 (0.23213483784844477) | > loss_mel_ce: 5.24171257019043 (nan) | > loss: 0.021619431674480438 (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.13469812115033497) | > loader_time: 0.0045 (0.004579793334007273)  --> TIME: 2025-02-23 20:11:54 -- STEP: 6050/36938 -- GLOBAL_STEP: 6050 | > loss_text_ce: 0.21301135420799255 (0.2320184045167994) | > loss_mel_ce: 0.2832728922367096 (nan) | > loss: 0.001969381934031844 (nan) | > current_lr: 5e-06 | > step_time: 0.1355 (0.13473455385728358) | > loader_time: 0.0044 (0.004578037616635164)  --> TIME: 2025-02-23 20:12:04 -- STEP: 6100/36938 -- GLOBAL_STEP: 6100 | > loss_text_ce: 0.21315158903598785 (0.23193274361188296) | > loss_mel_ce: 4.800703525543213 (nan) | > loss: 0.019896250218153 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13473351169805098) | > loader_time: 0.0044 (0.0045771827854094015)  --> TIME: 2025-02-23 20:12:13 -- STEP: 6150/36938 -- GLOBAL_STEP: 6150 | > loss_text_ce: 0.2205803394317627 (0.23182869116465252) | > loss_mel_ce: 0.2557896077632904 (nan) | > loss: 0.001890357001684606 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13475709275501574) | > loader_time: 0.0044 (0.0045777395682606355)  --> TIME: 2025-02-23 20:12:22 -- STEP: 6200/36938 -- GLOBAL_STEP: 6200 | > loss_text_ce: 0.2358088195323944 (0.23170786643941557) | > loss_mel_ce: 4.305419921875 (nan) | > loss: 0.0180207509547472 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13478264743281984) | > loader_time: 0.0043 (0.004577334042518377)  --> TIME: 2025-02-23 20:12:31 -- STEP: 6250/36938 -- GLOBAL_STEP: 6250 | > loss_text_ce: 0.23032326996326447 (0.23163106897592545) | > loss_mel_ce: 5.175735950469971 (nan) | > loss: 0.021452616900205612 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13475785842895543) | > loader_time: 0.0046 (0.004576190948486338)  --> TIME: 2025-02-23 20:12:41 -- STEP: 6300/36938 -- GLOBAL_STEP: 6300 | > loss_text_ce: 0.21100807189941406 (0.23155352881976538) | > loss_mel_ce: 4.93110990524292 (nan) | > loss: 0.020405231043696404 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13475582974297687) | > loader_time: 0.0047 (0.004574698493594225)  --> TIME: 2025-02-23 20:12:50 -- STEP: 6350/36938 -- GLOBAL_STEP: 6350 | > loss_text_ce: 0.20181621611118317 (0.2314587229067885) | > loss_mel_ce: 5.589817523956299 (nan) | > loss: 0.022982673719525337 (nan) | > current_lr: 5e-06 | > step_time: 0.1955 (0.13474319645739005) | > loader_time: 0.0046 (0.004573492290466798)  --> TIME: 2025-02-23 20:12:59 -- STEP: 6400/36938 -- GLOBAL_STEP: 6400 | > loss_text_ce: 0.22056248784065247 (0.2313661963515915) | > loss_mel_ce: 4.903750896453857 (nan) | > loss: 0.020334577187895775 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1347250574454668) | > loader_time: 0.0048 (0.004572701193392284)  --> TIME: 2025-02-23 20:13:08 -- STEP: 6450/36938 -- GLOBAL_STEP: 6450 | > loss_text_ce: 0.2070961594581604 (0.2312736310122549) | > loss_mel_ce: 5.620392322540283 (nan) | > loss: 0.023124955594539642 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.13471676534460533) | > loader_time: 0.0049 (0.00457194539003595)  --> TIME: 2025-02-23 20:13:17 -- STEP: 6500/36938 -- GLOBAL_STEP: 6500 | > loss_text_ce: 0.24330727756023407 (0.23118470497314747) | > loss_mel_ce: 0.3094075620174408 (nan) | > loss: 0.0021933128591626883 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.1347179015966566) | > loader_time: 0.0043 (0.00457112246293289)  --> TIME: 2025-02-23 20:13:26 -- STEP: 6550/36938 -- GLOBAL_STEP: 6550 | > loss_text_ce: 0.2049388438463211 (0.231090487578443) | > loss_mel_ce: 5.444094657897949 (nan) | > loss: 0.022416800260543823 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13469540377609643) | > loader_time: 0.0044 (0.00456992400511532)  --> TIME: 2025-02-23 20:13:36 -- STEP: 6600/36938 -- GLOBAL_STEP: 6600 | > loss_text_ce: 0.21953123807907104 (0.230983257740736) | > loss_mel_ce: 4.8458380699157715 (nan) | > loss: 0.0201006717979908 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13471273592024177) | > loader_time: 0.0044 (0.004569029482928204)  --> TIME: 2025-02-23 20:13:44 -- STEP: 6650/36938 -- GLOBAL_STEP: 6650 | > loss_text_ce: 0.21230603754520416 (0.23090517464885138) | > loss_mel_ce: 5.545603275299072 (nan) | > loss: 0.02284884825348854 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.13464265464840122) | > loader_time: 0.0047 (0.0045676731525507355)  --> TIME: 2025-02-23 20:13:54 -- STEP: 6700/36938 -- GLOBAL_STEP: 6700 | > loss_text_ce: 0.20768292248249054 (0.23079866407967325) | > loss_mel_ce: 4.19384765625 (nan) | > loss: 0.017466392368078232 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13464662149770976) | > loader_time: 0.0044 (0.004566399339419712)  --> TIME: 2025-02-23 20:14:03 -- STEP: 6750/36938 -- GLOBAL_STEP: 6750 | > loss_text_ce: 0.21332499384880066 (0.23069394005669489) | > loss_mel_ce: 3.848360061645508 (nan) | > loss: 0.016117798164486885 (nan) | > current_lr: 5e-06 | > step_time: 0.1762 (0.13466143357312282) | > loader_time: 0.0049 (0.0045655663101761635)  --> TIME: 2025-02-23 20:14:12 -- STEP: 6800/36938 -- GLOBAL_STEP: 6800 | > loss_text_ce: 0.22664843499660492 (0.23057985449319376) | > loss_mel_ce: 5.325075626373291 (nan) | > loss: 0.022030651569366455 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13470117277958857) | > loader_time: 0.0044 (0.0045658648364684355)  --> TIME: 2025-02-23 20:14:22 -- STEP: 6850/36938 -- GLOBAL_STEP: 6850 | > loss_text_ce: 0.21129359304904938 (0.23050607218577043) | > loss_mel_ce: 5.929976940155029 (nan) | > loss: 0.02437012270092964 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.1347081788612982) | > loader_time: 0.0045 (0.004565516144689869)  --> TIME: 2025-02-23 20:14:31 -- STEP: 6900/36938 -- GLOBAL_STEP: 6900 | > loss_text_ce: 0.22677884995937347 (0.23041534421452578) | > loss_mel_ce: 4.501142978668213 (nan) | > loss: 0.018761595711112022 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13470115761826038) | > loader_time: 0.0042 (0.004566009459288239)  --> TIME: 2025-02-23 20:14:40 -- STEP: 6950/36938 -- GLOBAL_STEP: 6950 | > loss_text_ce: 0.2273828089237213 (0.23033795190372056) | > loss_mel_ce: 4.737224102020264 (nan) | > loss: 0.019700821489095688 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1346948278207577) | > loader_time: 0.0044 (0.0045655477990349585)  --> TIME: 2025-02-23 20:14:50 -- STEP: 7000/36938 -- GLOBAL_STEP: 7000 | > loss_text_ce: 0.2186249941587448 (0.23023972666689327) | > loss_mel_ce: 5.339871883392334 (nan) | > loss: 0.022057529538869858 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.1347466547489169) | > loader_time: 0.0045 (0.00456639266014101)  --> TIME: 2025-02-23 20:14:59 -- STEP: 7050/36938 -- GLOBAL_STEP: 7050 | > loss_text_ce: 0.21780449151992798 (0.230137703997869) | > loss_mel_ce: 5.149691104888916 (nan) | > loss: 0.021299587562680244 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13478464194223405) | > loader_time: 0.0044 (0.004566098916615173)  --> TIME: 2025-02-23 20:15:09 -- STEP: 7100/36938 -- GLOBAL_STEP: 7100 | > loss_text_ce: 0.19820696115493774 (0.23001841168588316) | > loss_mel_ce: 5.315129280090332 (nan) | > loss: 0.0218783188611269 (nan) | > current_lr: 5e-06 | > step_time: 0.1578 (0.13480046853213257) | > loader_time: 0.0048 (0.004565361452774281)  --> TIME: 2025-02-23 20:15:18 -- STEP: 7150/36938 -- GLOBAL_STEP: 7150 | > loss_text_ce: 0.2003009170293808 (0.22992254748627855) | > loss_mel_ce: 5.070016384124756 (nan) | > loss: 0.02091395854949951 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.13480239784800938) | > loader_time: 0.0045 (0.004564011913913132)  --> TIME: 2025-02-23 20:15:27 -- STEP: 7200/36938 -- GLOBAL_STEP: 7200 | > loss_text_ce: 0.2148125022649765 (0.229835161452906) | > loss_mel_ce: 5.427486419677734 (nan) | > loss: 0.022390075027942657 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.1348066967063481) | > loader_time: 0.0046 (0.004562482105361108)  --> TIME: 2025-02-23 20:15:37 -- STEP: 7250/36938 -- GLOBAL_STEP: 7250 | > loss_text_ce: 0.20937499403953552 (0.22972963041889258) | > loss_mel_ce: 3.6334147453308105 (nan) | > loss: 0.015249165706336498 (nan) | > current_lr: 5e-06 | > step_time: 0.1355 (0.1348251003725778) | > loader_time: 0.0044 (0.004562074759910864)  --> TIME: 2025-02-23 20:15:46 -- STEP: 7300/36938 -- GLOBAL_STEP: 7300 | > loss_text_ce: 0.22300271689891815 (0.22965636107203077) | > loss_mel_ce: 5.030625820159912 (nan) | > loss: 0.020847734063863754 (nan) | > current_lr: 5e-06 | > step_time: 0.1353 (0.13481321769217947) | > loader_time: 0.0045 (0.004561105166396067)  --> TIME: 2025-02-23 20:15:55 -- STEP: 7350/36938 -- GLOBAL_STEP: 7350 | > loss_text_ce: 0.21566489338874817 (0.22956711785322956) | > loss_mel_ce: 2.61932373046875 (nan) | > loss: 0.011249955743551254 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13480808481878187) | > loader_time: 0.0044 (0.004559981060676852)  --> TIME: 2025-02-23 20:16:04 -- STEP: 7400/36938 -- GLOBAL_STEP: 7400 | > loss_text_ce: 0.22150735557079315 (0.22946725399510282) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13480159231134384) | > loader_time: 0.0042 (0.004558508267273791)  --> TIME: 2025-02-23 20:16:13 -- STEP: 7450/36938 -- GLOBAL_STEP: 7450 | > loss_text_ce: 0.2471710592508316 (0.2293712406170448) | > loss_mel_ce: 5.294322490692139 (nan) | > loss: 0.021990055218338966 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.13481531130387495) | > loader_time: 0.0043 (0.004557794692532345)  --> TIME: 2025-02-23 20:16:23 -- STEP: 7500/36938 -- GLOBAL_STEP: 7500 | > loss_text_ce: 0.2065144181251526 (0.22924645447731018) | > loss_mel_ce: 5.210543632507324 (nan) | > loss: 0.021496262401342392 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.1348552529652914) | > loader_time: 0.0043 (0.004557073561350522)  --> TIME: 2025-02-23 20:16:32 -- STEP: 7550/36938 -- GLOBAL_STEP: 7550 | > loss_text_ce: 0.22077703475952148 (0.22915940970379786) | > loss_mel_ce: 5.572817802429199 (nan) | > loss: 0.022990457713603973 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13485431630090383) | > loader_time: 0.0044 (0.004556362644726094)  --> TIME: 2025-02-23 20:16:42 -- STEP: 7600/36938 -- GLOBAL_STEP: 7600 | > loss_text_ce: 0.1941145807504654 (0.22906926837210592) | > loss_mel_ce: 4.582503318786621 (nan) | > loss: 0.01895483396947384 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.134891684870971) | > loader_time: 0.0043 (0.004556302271391233)  --> TIME: 2025-02-23 20:16:51 -- STEP: 7650/36938 -- GLOBAL_STEP: 7650 | > loss_text_ce: 0.2611607015132904 (0.2289974759080831) | > loss_mel_ce: 3.5463144779205322 (nan) | > loss: 0.015109028667211533 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13490357143427045) | > loader_time: 0.0045 (0.004555505428438889)  --> TIME: 2025-02-23 20:17:01 -- STEP: 7700/36938 -- GLOBAL_STEP: 7700 | > loss_text_ce: 0.20654861629009247 (0.22888057347628976) | > loss_mel_ce: 4.916359901428223 (nan) | > loss: 0.020329004153609276 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13493444414881928) | > loader_time: 0.0045 (0.00455491985593525)  --> TIME: 2025-02-23 20:17:10 -- STEP: 7750/36938 -- GLOBAL_STEP: 7750 | > loss_text_ce: 0.216779887676239 (0.22878722804977047) | > loss_mel_ce: 5.142169952392578 (nan) | > loss: 0.021265674382448196 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13496913085445292) | > loader_time: 0.0041 (0.00455429947760799)  --> TIME: 2025-02-23 20:17:20 -- STEP: 7800/36938 -- GLOBAL_STEP: 7800 | > loss_text_ce: 0.21290276944637299 (0.22868452612024087) | > loss_mel_ce: 5.074448585510254 (nan) | > loss: 0.020981553941965103 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13501862201935216) | > loader_time: 0.0044 (0.004553181330362972)  --> TIME: 2025-02-23 20:17:29 -- STEP: 7850/36938 -- GLOBAL_STEP: 7850 | > loss_text_ce: 0.23017045855522156 (0.2286089131881477) | > loss_mel_ce: 4.5228962898254395 (nan) | > loss: 0.018861375749111176 (nan) | > current_lr: 5e-06 | > step_time: 0.1078 (0.13498332649279562) | > loader_time: 0.0045 (0.0045523805679029885)  --> TIME: 2025-02-23 20:17:38 -- STEP: 7900/36938 -- GLOBAL_STEP: 7900 | > loss_text_ce: 0.234416663646698 (0.2285246857717822) | > loss_mel_ce: 3.27254056930542 (nan) | > loss: 0.01391649805009365 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13498966189879416) | > loader_time: 0.0044 (0.004551818340639541)  --> TIME: 2025-02-23 20:17:47 -- STEP: 7950/36938 -- GLOBAL_STEP: 7950 | > loss_text_ce: 0.22022321820259094 (0.2284405942064411) | > loss_mel_ce: 3.3549978733062744 (nan) | > loss: 0.014187386259436607 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13497407490352425) | > loader_time: 0.0043 (0.0045514080959296235)  --> TIME: 2025-02-23 20:17:56 -- STEP: 8000/36938 -- GLOBAL_STEP: 8000 | > loss_text_ce: 0.2008088231086731 (0.2283372886981815) | > loss_mel_ce: 4.775088787078857 (nan) | > loss: 0.01974562741816044 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13497930562496196) | > loader_time: 0.0044 (0.004551404237747208)  --> TIME: 2025-02-23 20:18:06 -- STEP: 8050/36938 -- GLOBAL_STEP: 8050 | > loss_text_ce: 0.22064338624477386 (0.2282356020030768) | > loss_mel_ce: 4.376096248626709 (nan) | > loss: 0.01824103109538555 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13497182203375785) | > loader_time: 0.0052 (0.004551093770850535)  --> TIME: 2025-02-23 20:18:15 -- STEP: 8100/36938 -- GLOBAL_STEP: 8100 | > loss_text_ce: 0.23601561784744263 (0.22812150283544153) | > loss_mel_ce: 4.145665645599365 (nan) | > loss: 0.017387626692652702 (nan) | > current_lr: 5e-06 | > step_time: 0.108 (0.13497376368369596) | > loader_time: 0.0045 (0.0045507896093674805)  --> TIME: 2025-02-23 20:18:24 -- STEP: 8150/36938 -- GLOBAL_STEP: 8150 | > loss_text_ce: 0.21976974606513977 (0.2280064388433117) | > loss_mel_ce: 4.760575294494629 (nan) | > loss: 0.019763275980949402 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13498982561146566) | > loader_time: 0.0042 (0.004550587180202009)  --> TIME: 2025-02-23 20:18:33 -- STEP: 8200/36938 -- GLOBAL_STEP: 8200 | > loss_text_ce: 0.20629733800888062 (0.22790685050189496) | > loss_mel_ce: 3.45068359375 (nan) | > loss: 0.014511830173432827 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13497072728668782) | > loader_time: 0.005 (0.004549314016249137)  --> TIME: 2025-02-23 20:18:43 -- STEP: 8250/36938 -- GLOBAL_STEP: 8250 | > loss_text_ce: 0.2828125059604645 (0.22782105191548666) | > loss_mel_ce: 2.8657073974609375 (nan) | > loss: 0.01249412726610899 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13501572542479556) | > loader_time: 0.0042 (0.00454856803200463)  --> TIME: 2025-02-23 20:18:52 -- STEP: 8300/36938 -- GLOBAL_STEP: 8300 | > loss_text_ce: 0.20935183763504028 (0.2277284992459308) | > loss_mel_ce: 5.425952911376953 (nan) | > loss: 0.022362321615219116 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13502856995686002) | > loader_time: 0.0043 (0.004547768989241281)  --> TIME: 2025-02-23 20:19:02 -- STEP: 8350/36938 -- GLOBAL_STEP: 8350 | > loss_text_ce: 0.2192559391260147 (0.2276503379455583) | > loss_mel_ce: 4.323768138885498 (nan) | > loss: 0.01802787370979786 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13503150465959568) | > loader_time: 0.0044 (0.004546719339793327)  --> TIME: 2025-02-23 20:19:11 -- STEP: 8400/36938 -- GLOBAL_STEP: 8400 | > loss_text_ce: 0.19542571902275085 (0.22756779085489962) | > loss_mel_ce: 5.0385918617248535 (nan) | > loss: 0.020769910886883736 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.13504266824041106) | > loader_time: 0.0045 (0.004546072114081623)  --> TIME: 2025-02-23 20:19:20 -- STEP: 8450/36938 -- GLOBAL_STEP: 8450 | > loss_text_ce: 0.20480681955814362 (0.22748357783407852) | > loss_mel_ce: 4.907830715179443 (nan) | > loss: 0.020288245752453804 (nan) | > current_lr: 5e-06 | > step_time: 0.1353 (0.13503908591862973) | > loader_time: 0.0045 (0.004545983957821108)  --> TIME: 2025-02-23 20:19:29 -- STEP: 8500/36938 -- GLOBAL_STEP: 8500 | > loss_text_ce: 0.1993703991174698 (0.22740354097064805) | > loss_mel_ce: 5.4523539543151855 (nan) | > loss: 0.022427478805184364 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.13502065678203815) | > loader_time: 0.0043 (0.004545219982371622)  --> TIME: 2025-02-23 20:19:39 -- STEP: 8550/36938 -- GLOBAL_STEP: 8550 | > loss_text_ce: 0.19394530355930328 (0.22730250174016287) | > loss_mel_ce: 4.000535011291504 (nan) | > loss: 0.016644764691591263 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13504222510153793) | > loader_time: 0.0044 (0.004544957590382013)  --> TIME: 2025-02-23 20:19:48 -- STEP: 8600/36938 -- GLOBAL_STEP: 8600 | > loss_text_ce: 0.1993749886751175 (0.22721718182917253) | > loss_mel_ce: 5.036435604095459 (nan) | > loss: 0.020777028053998947 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.1350384983073834) | > loader_time: 0.0043 (0.004544467149778867)  --> TIME: 2025-02-23 20:19:57 -- STEP: 8650/36938 -- GLOBAL_STEP: 8650 | > loss_text_ce: 0.20266275107860565 (0.22711962950367456) | > loss_mel_ce: 5.361237525939941 (nan) | > loss: 0.022078972309827805 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13504468305951609) | > loader_time: 0.0044 (0.004544415308560953)  --> TIME: 2025-02-23 20:20:06 -- STEP: 8700/36938 -- GLOBAL_STEP: 8700 | > loss_text_ce: 0.25737497210502625 (0.22703363080305614) | > loss_mel_ce: 2.8253424167633057 (nan) | > loss: 0.0122330067679286 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.13502772851922057) | > loader_time: 0.0044 (0.004543919618102339)  --> TIME: 2025-02-23 20:20:16 -- STEP: 8750/36938 -- GLOBAL_STEP: 8750 | > loss_text_ce: 0.19474025070667267 (0.22692423928294864) | > loss_mel_ce: 5.174986839294434 (nan) | > loss: 0.02130844257771969 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.1350359518051149) | > loader_time: 0.0046 (0.004543715340750574)  --> TIME: 2025-02-23 20:20:24 -- STEP: 8800/36938 -- GLOBAL_STEP: 8800 | > loss_text_ce: 0.21975544095039368 (0.2268401051634414) | > loss_mel_ce: 4.089441776275635 (nan) | > loss: 0.01709999144077301 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.13499710150740377) | > loader_time: 0.0045 (0.00454307068477979)  --> TIME: 2025-02-23 20:20:34 -- STEP: 8850/36938 -- GLOBAL_STEP: 8850 | > loss_text_ce: 0.22085227072238922 (0.22673249749982424) | > loss_mel_ce: 4.447971820831299 (nan) | > loss: 0.018527081236243248 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.1349971540634242) | > loader_time: 0.0043 (0.004542141607252236)  --> TIME: 2025-02-23 20:20:43 -- STEP: 8900/36938 -- GLOBAL_STEP: 8900 | > loss_text_ce: 0.20458807051181793 (0.22664963095877944) | > loss_mel_ce: 4.670029163360596 (nan) | > loss: 0.019343718886375427 (nan) | > current_lr: 5e-06 | > step_time: 0.1775 (0.1350085192316035) | > loader_time: 0.0049 (0.00454202568932867)  --> TIME: 2025-02-23 20:20:53 -- STEP: 8950/36938 -- GLOBAL_STEP: 8950 | > loss_text_ce: 0.21025943756103516 (0.22655498433878973) | > loss_mel_ce: 5.0581512451171875 (nan) | > loss: 0.020906392484903336 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13502426880031998) | > loader_time: 0.0045 (0.004541525414536134)  --> TIME: 2025-02-23 20:21:02 -- STEP: 9000/36938 -- GLOBAL_STEP: 9000 | > loss_text_ce: 0.19518494606018066 (0.22644321412179214) | > loss_mel_ce: 4.7752861976623535 (nan) | > loss: 0.01972409337759018 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13503928216298433) | > loader_time: 0.0044 (0.00454097308052912)  --> TIME: 2025-02-23 20:21:11 -- STEP: 9050/36938 -- GLOBAL_STEP: 9050 | > loss_text_ce: 0.19056451320648193 (0.2263693401629095) | > loss_mel_ce: 5.59885835647583 (nan) | > loss: 0.02297390252351761 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13505441602422402) | > loader_time: 0.0044 (0.004540163034892227)  --> TIME: 2025-02-23 20:21:20 -- STEP: 9100/36938 -- GLOBAL_STEP: 9100 | > loss_text_ce: 0.2217881977558136 (0.22626584123124133) | > loss_mel_ce: 4.584878444671631 (nan) | > loss: 0.019074076786637306 (nan) | > current_lr: 5e-06 | > step_time: 0.1073 (0.1350344631436108) | > loader_time: 0.0043 (0.004539487702505939)  --> TIME: 2025-02-23 20:21:30 -- STEP: 9150/36938 -- GLOBAL_STEP: 9150 | > loss_text_ce: 0.20033088326454163 (0.22620913090113082) | > loss_mel_ce: 5.359682083129883 (nan) | > loss: 0.022063544020056725 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.13503010028046986) | > loader_time: 0.0042 (0.004538501103719085)  --> TIME: 2025-02-23 20:21:39 -- STEP: 9200/36938 -- GLOBAL_STEP: 9200 | > loss_text_ce: 0.1981355845928192 (0.22612159154337386) | > loss_mel_ce: 5.540372848510742 (nan) | > loss: 0.022771859541535378 (nan) | > current_lr: 5e-06 | > step_time: 0.1486 (0.13503938685292777) | > loader_time: 0.0042 (0.004537790858227285)  --> TIME: 2025-02-23 20:21:48 -- STEP: 9250/36938 -- GLOBAL_STEP: 9250 | > loss_text_ce: 0.19259469211101532 (0.22601640866092734) | > loss_mel_ce: 5.546473026275635 (nan) | > loss: 0.02277407981455326 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13506366000304348) | > loader_time: 0.0047 (0.004537378311157241)  --> TIME: 2025-02-23 20:21:58 -- STEP: 9300/36938 -- GLOBAL_STEP: 9300 | > loss_text_ce: 0.20867368578910828 (0.2259445209865288) | > loss_mel_ce: 4.814980983734131 (nan) | > loss: 0.019935138523578644 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.1350579750666054) | > loader_time: 0.0042 (0.004536873038097109)  --> TIME: 2025-02-23 20:22:07 -- STEP: 9350/36938 -- GLOBAL_STEP: 9350 | > loss_text_ce: 0.20621709525585175 (0.2258636346475326) | > loss_mel_ce: 4.640588283538818 (nan) | > loss: 0.019233357161283493 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.1350479518410994) | > loader_time: 0.0044 (0.004536251919792313)  --> TIME: 2025-02-23 20:22:16 -- STEP: 9400/36938 -- GLOBAL_STEP: 9400 | > loss_text_ce: 0.2039158195257187 (0.22577977798562102) | > loss_mel_ce: 3.95947265625 (nan) | > loss: 0.016521383076906204 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13506174145860875) | > loader_time: 0.0049 (0.004535951031015289)  --> TIME: 2025-02-23 20:22:25 -- STEP: 9450/36938 -- GLOBAL_STEP: 9450 | > loss_text_ce: 0.21470588445663452 (0.22568772624251704) | > loss_mel_ce: 5.171761989593506 (nan) | > loss: 0.02137487381696701 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13505675747281026) | > loader_time: 0.0047 (0.0045355387844106)  --> TIME: 2025-02-23 20:22:34 -- STEP: 9500/36938 -- GLOBAL_STEP: 9500 | > loss_text_ce: 0.21588940918445587 (0.22560228690818737) | > loss_mel_ce: 5.248224258422852 (nan) | > loss: 0.021682992577552795 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13503606871554727) | > loader_time: 0.0046 (0.004534678936004652)  --> TIME: 2025-02-23 20:22:43 -- STEP: 9550/36938 -- GLOBAL_STEP: 9550 | > loss_text_ce: 0.20771445333957672 (0.22552804489254327) | > loss_mel_ce: 4.866404056549072 (nan) | > loss: 0.020135393366217613 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.13502438824838361) | > loader_time: 0.0044 (0.004534333488704035)  --> TIME: 2025-02-23 20:22:53 -- STEP: 9600/36938 -- GLOBAL_STEP: 9600 | > loss_text_ce: 0.1882680058479309 (0.22544162350706756) | > loss_mel_ce: 5.00868034362793 (nan) | > loss: 0.02062281221151352 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.1350154706090687) | > loader_time: 0.0043 (0.004534227301677077)  --> TIME: 2025-02-23 20:23:02 -- STEP: 9650/36938 -- GLOBAL_STEP: 9650 | > loss_text_ce: 0.21800675988197327 (0.22534680695731404) | > loss_mel_ce: 5.24287748336792 (nan) | > loss: 0.021670175716280937 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13500972641564393) | > loader_time: 0.0045 (0.004533563179055652)  --> TIME: 2025-02-23 20:23:11 -- STEP: 9700/36938 -- GLOBAL_STEP: 9700 | > loss_text_ce: 0.22073863446712494 (0.22525968742278432) | > loss_mel_ce: 4.637784004211426 (nan) | > loss: 0.019279852509498596 (nan) | > current_lr: 5e-06 | > step_time: 0.1031 (0.13500940961936064) | > loader_time: 0.0042 (0.004534285068511971)  --> TIME: 2025-02-23 20:23:20 -- STEP: 9750/36938 -- GLOBAL_STEP: 9750 | > loss_text_ce: 0.21665440499782562 (0.22518976168907606) | > loss_mel_ce: 5.361852645874023 (nan) | > loss: 0.022136934101581573 (nan) | > current_lr: 5e-06 | > step_time: 0.1067 (0.13500391796307656) | > loader_time: 0.0042 (0.004533913294474291)  --> TIME: 2025-02-23 20:23:29 -- STEP: 9800/36938 -- GLOBAL_STEP: 9800 | > loss_text_ce: 0.1940954029560089 (0.22512297662545225) | > loss_mel_ce: 4.796512603759766 (nan) | > loss: 0.019804002717137337 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13499676227569557) | > loader_time: 0.005 (0.004533852387447753)  --> TIME: 2025-02-23 20:23:39 -- STEP: 9850/36938 -- GLOBAL_STEP: 9850 | > loss_text_ce: 0.21820944547653198 (0.22504556983558055) | > loss_mel_ce: 4.9808430671691895 (nan) | > loss: 0.020631160587072372 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13500261616585804) | > loader_time: 0.0044 (0.004533579506849886)  --> TIME: 2025-02-23 20:23:48 -- STEP: 9900/36938 -- GLOBAL_STEP: 9900 | > loss_text_ce: 0.2033928483724594 (0.22495167210999162) | > loss_mel_ce: 4.934470176696777 (nan) | > loss: 0.020388346165418625 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.1350013244272482) | > loader_time: 0.0047 (0.004532941158371748)  --> TIME: 2025-02-23 20:23:57 -- STEP: 9950/36938 -- GLOBAL_STEP: 9950 | > loss_text_ce: 0.22042229771614075 (0.2248672830594245) | > loss_mel_ce: 4.240435600280762 (nan) | > loss: 0.01770181767642498 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13496992485008033) | > loader_time: 0.0043 (0.004532329928335836)  --> TIME: 2025-02-23 20:24:06 -- STEP: 10000/36938 -- GLOBAL_STEP: 10000 | > loss_text_ce: 0.21796874701976776 (0.22477351848185062) | > loss_mel_ce: 0.303955078125 (nan) | > loss: 0.002071126364171505 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.1349752522945403) | > loader_time: 0.0043 (0.00453172292709351) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_10000.pth  --> TIME: 2025-02-23 20:24:21 -- STEP: 10050/36938 -- GLOBAL_STEP: 10050 | > loss_text_ce: 0.20230206847190857 (0.22466872472668167) | > loss_mel_ce: 5.234742164611816 (nan) | > loss: 0.021575573831796646 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13498957187975213) | > loader_time: 0.0046 (0.004531524928648085)  --> TIME: 2025-02-23 20:24:29 -- STEP: 10100/36938 -- GLOBAL_STEP: 10100 | > loss_text_ce: 0.1903780847787857 (0.22458375504848982) | > loss_mel_ce: 3.7786741256713867 (nan) | > loss: 0.015750208869576454 (nan) | > current_lr: 5e-06 | > step_time: 0.1793 (0.1349589703342702) | > loader_time: 0.0046 (0.004531080888049443)  --> TIME: 2025-02-23 20:24:38 -- STEP: 10150/36938 -- GLOBAL_STEP: 10150 | > loss_text_ce: 0.20819078385829926 (0.22449105498091929) | > loss_mel_ce: 4.859453201293945 (nan) | > loss: 0.02010970003902912 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13494300121157043) | > loader_time: 0.0045 (0.004530571904675722)  --> TIME: 2025-02-23 20:24:48 -- STEP: 10200/36938 -- GLOBAL_STEP: 10200 | > loss_text_ce: 0.20641477406024933 (0.22439121795343417) | > loss_mel_ce: 5.10027551651001 (nan) | > loss: 0.02105829492211342 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13493364261645904) | > loader_time: 0.0047 (0.004529903706382311)  --> TIME: 2025-02-23 20:24:56 -- STEP: 10250/36938 -- GLOBAL_STEP: 10250 | > loss_text_ce: 0.1952470988035202 (0.22430700151513264) | > loss_mel_ce: 5.157543182373047 (nan) | > loss: 0.021241232752799988 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.1348979850862085) | > loader_time: 0.0044 (0.004529336091948723)  --> TIME: 2025-02-23 20:25:06 -- STEP: 10300/36938 -- GLOBAL_STEP: 10300 | > loss_text_ce: 0.23152777552604675 (0.2242017827783395) | > loss_mel_ce: 4.126877307891846 (nan) | > loss: 0.01729525998234749 (nan) | > current_lr: 5e-06 | > step_time: 0.1076 (0.1349077739993345) | > loader_time: 0.0043 (0.004528767345021075)  --> TIME: 2025-02-23 20:25:14 -- STEP: 10350/36938 -- GLOBAL_STEP: 10350 | > loss_text_ce: 0.2006250023841858 (0.22415058858440695) | > loss_mel_ce: 4.862847328186035 (nan) | > loss: 0.020093144848942757 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1348737323687273) | > loader_time: 0.0045 (0.004527974819791492)  --> TIME: 2025-02-23 20:25:23 -- STEP: 10400/36938 -- GLOBAL_STEP: 10400 | > loss_text_ce: 0.1961607187986374 (0.22405822082924154) | > loss_mel_ce: 3.926023006439209 (nan) | > loss: 0.016357872635126114 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.1348573322938039) | > loader_time: 0.0042 (0.004527477553257576)  --> TIME: 2025-02-23 20:25:32 -- STEP: 10450/36938 -- GLOBAL_STEP: 10450 | > loss_text_ce: 0.20482420921325684 (0.22398002238650072) | > loss_mel_ce: 5.149769306182861 (nan) | > loss: 0.021248387172818184 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13483920022061013) | > loader_time: 0.0043 (0.0045266482704564145)  --> TIME: 2025-02-23 20:25:42 -- STEP: 10500/36938 -- GLOBAL_STEP: 10500 | > loss_text_ce: 0.2291875034570694 (0.22388817081281118) | > loss_mel_ce: 0.2503204345703125 (nan) | > loss: 0.0019028093665838242 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13483766667048153) | > loader_time: 0.0043 (0.004525883901686895)  --> TIME: 2025-02-23 20:25:51 -- STEP: 10550/36938 -- GLOBAL_STEP: 10550 | > loss_text_ce: 0.2067100703716278 (0.2238013502757696) | > loss_mel_ce: 5.421922206878662 (nan) | > loss: 0.02233584225177765 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13483926813749358) | > loader_time: 0.0043 (0.004525113580351191)  --> TIME: 2025-02-23 20:26:00 -- STEP: 10600/36938 -- GLOBAL_STEP: 10600 | > loss_text_ce: 0.21926134824752808 (0.22370091864522898) | > loss_mel_ce: 4.085911273956299 (nan) | > loss: 0.017084019258618355 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13485468628271594) | > loader_time: 0.0066 (0.004524876801472789)  --> TIME: 2025-02-23 20:26:10 -- STEP: 10650/36938 -- GLOBAL_STEP: 10650 | > loss_text_ce: 0.20539633929729462 (0.22359129293004112) | > loss_mel_ce: 4.392816543579102 (nan) | > loss: 0.018246876075863838 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.1348746160722117) | > loader_time: 0.0045 (0.004524614934070568)  --> TIME: 2025-02-23 20:26:19 -- STEP: 10700/36938 -- GLOBAL_STEP: 10700 | > loss_text_ce: 0.21182066202163696 (0.22349499641038548) | > loss_mel_ce: 4.497835636138916 (nan) | > loss: 0.01868911273777485 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.1348799932114435) | > loader_time: 0.0043 (0.004524243323602406)  --> TIME: 2025-02-23 20:26:28 -- STEP: 10750/36938 -- GLOBAL_STEP: 10750 | > loss_text_ce: 0.20065177977085114 (0.22341500874591427) | > loss_mel_ce: 4.3402605056762695 (nan) | > loss: 0.01801949366927147 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13488478849100544) | > loader_time: 0.0043 (0.004523701689964114)  --> TIME: 2025-02-23 20:26:37 -- STEP: 10800/36938 -- GLOBAL_STEP: 10800 | > loss_text_ce: 0.22889423370361328 (0.22333217030598057) | > loss_mel_ce: 5.758333206176758 (nan) | > loss: 0.023758839815855026 (nan) | > current_lr: 5e-06 | > step_time: 0.1027 (0.13487016351134726) | > loader_time: 0.0045 (0.004523305341049474)  --> TIME: 2025-02-23 20:26:47 -- STEP: 10850/36938 -- GLOBAL_STEP: 10850 | > loss_text_ce: 0.1894480437040329 (0.22324124237359394) | > loss_mel_ce: 5.130814075469971 (nan) | > loss: 0.021112151443958282 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.13486888125195523) | > loader_time: 0.0047 (0.004523225138264313)  --> TIME: 2025-02-23 20:26:56 -- STEP: 10900/36938 -- GLOBAL_STEP: 10900 | > loss_text_ce: 0.20652343332767487 (0.22314050806771724) | > loss_mel_ce: 5.186241626739502 (nan) | > loss: 0.021399863064289093 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1348754647893648) | > loader_time: 0.0043 (0.004522730013646111)  --> TIME: 2025-02-23 20:27:05 -- STEP: 10950/36938 -- GLOBAL_STEP: 10950 | > loss_text_ce: 0.20278409123420715 (0.22303336861607145) | > loss_mel_ce: 4.025111198425293 (nan) | > loss: 0.016777362674474716 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13487977833508408) | > loader_time: 0.0043 (0.004522156628299513)  --> TIME: 2025-02-23 20:27:15 -- STEP: 11000/36938 -- GLOBAL_STEP: 11000 | > loss_text_ce: 0.20217548310756683 (0.22295368840613147) | > loss_mel_ce: 4.427175998687744 (nan) | > loss: 0.01837044395506382 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13487728951194133) | > loader_time: 0.0044 (0.004522178194739595)  --> TIME: 2025-02-23 20:27:24 -- STEP: 11050/36938 -- GLOBAL_STEP: 11050 | > loss_text_ce: 0.18955355882644653 (0.22287153707775056) | > loss_mel_ce: 5.2518391609191895 (nan) | > loss: 0.02159282937645912 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13488317066727773) | > loader_time: 0.0047 (0.004521849511435663)  --> TIME: 2025-02-23 20:27:33 -- STEP: 11100/36938 -- GLOBAL_STEP: 11100 | > loss_text_ce: 0.1829449087381363 (0.2227879656918414) | > loss_mel_ce: 4.574159145355225 (nan) | > loss: 0.01887739822268486 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13486579890723743) | > loader_time: 0.0047 (0.004522508651286625)  --> TIME: 2025-02-23 20:27:42 -- STEP: 11150/36938 -- GLOBAL_STEP: 11150 | > loss_text_ce: 0.19679686427116394 (0.22270456745485553) | > loss_mel_ce: 4.8455491065979 (nan) | > loss: 0.02000931091606617 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13485159531837138) | > loader_time: 0.0044 (0.004522142175067161)  --> TIME: 2025-02-23 20:27:51 -- STEP: 11200/36938 -- GLOBAL_STEP: 11200 | > loss_text_ce: 0.19754260778427124 (0.22261141311909471) | > loss_mel_ce: 4.217721939086914 (nan) | > loss: 0.01752089150249958 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13487064284937783) | > loader_time: 0.0051 (0.00452214326177324)  --> TIME: 2025-02-23 20:28:01 -- STEP: 11250/36938 -- GLOBAL_STEP: 11250 | > loss_text_ce: 0.1967289000749588 (0.2225396951622433) | > loss_mel_ce: 4.2254815101623535 (nan) | > loss: 0.0175484549254179 (nan) | > current_lr: 5e-06 | > step_time: 0.1892 (0.13486508036719502) | > loader_time: 0.005 (0.004522537167867018)  --> TIME: 2025-02-23 20:28:10 -- STEP: 11300/36938 -- GLOBAL_STEP: 11300 | > loss_text_ce: 0.19632665812969208 (0.22248336601046334) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13484935967268205) | > loader_time: 0.0044 (0.004522667289835155)  --> TIME: 2025-02-23 20:28:19 -- STEP: 11350/36938 -- GLOBAL_STEP: 11350 | > loss_text_ce: 0.2191796898841858 (0.2223859828539882) | > loss_mel_ce: 3.74853515625 (nan) | > loss: 0.015744900330901146 (nan) | > current_lr: 5e-06 | > step_time: 0.1075 (0.13485360063645352) | > loader_time: 0.0042 (0.004522681110230832)  --> TIME: 2025-02-23 20:28:28 -- STEP: 11400/36938 -- GLOBAL_STEP: 11400 | > loss_text_ce: 0.1985757201910019 (0.22229672560958486) | > loss_mel_ce: 0.034912109375 (nan) | > loss: 0.0009265390690416098 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13484671069864712) | > loader_time: 0.0048 (0.004522425312744936)  --> TIME: 2025-02-23 20:28:37 -- STEP: 11450/36938 -- GLOBAL_STEP: 11450 | > loss_text_ce: 0.1988145411014557 (0.22223302893763547) | > loss_mel_ce: 4.729434967041016 (nan) | > loss: 0.019556546583771706 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13485731564234513) | > loader_time: 0.0042 (0.004521929853347705)  --> TIME: 2025-02-23 20:28:47 -- STEP: 11500/36938 -- GLOBAL_STEP: 11500 | > loss_text_ce: 0.20087836682796478 (0.2221431043692257) | > loss_mel_ce: 4.810124397277832 (nan) | > loss: 0.019884932786226273 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13487140130996758) | > loader_time: 0.0043 (0.0045214883348216155)  --> TIME: 2025-02-23 20:28:56 -- STEP: 11550/36938 -- GLOBAL_STEP: 11550 | > loss_text_ce: 0.1825687438249588 (0.22204667100400635) | > loss_mel_ce: 4.839137077331543 (nan) | > loss: 0.019927404820919037 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13487043535554602) | > loader_time: 0.0044 (0.004521625093567419)  --> TIME: 2025-02-23 20:29:05 -- STEP: 11600/36938 -- GLOBAL_STEP: 11600 | > loss_text_ce: 0.22989583015441895 (0.2219576425966004) | > loss_mel_ce: 0.1646815687417984 (nan) | > loss: 0.0015657834010198712 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1348660575521407) | > loader_time: 0.0043 (0.004521374291387096)  --> TIME: 2025-02-23 20:29:14 -- STEP: 11650/36938 -- GLOBAL_STEP: 11650 | > loss_text_ce: 0.20205818116664886 (0.2218696231660413) | > loss_mel_ce: 4.736812114715576 (nan) | > loss: 0.019598692655563354 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13484741192518934) | > loader_time: 0.0043 (0.004521241617816708)  --> TIME: 2025-02-23 20:29:23 -- STEP: 11700/36938 -- GLOBAL_STEP: 11700 | > loss_text_ce: 0.19129464030265808 (0.22177313591425235) | > loss_mel_ce: 4.929542064666748 (nan) | > loss: 0.02032078243792057 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.1348355919479309) | > loader_time: 0.0046 (0.004521056077419181)  --> TIME: 2025-02-23 20:29:32 -- STEP: 11750/36938 -- GLOBAL_STEP: 11750 | > loss_text_ce: 0.19690971076488495 (0.2217014390204815) | > loss_mel_ce: 5.057257652282715 (nan) | > loss: 0.02084987238049507 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.1348256377565105) | > loader_time: 0.0044 (0.0045208424913122285)  --> TIME: 2025-02-23 20:29:41 -- STEP: 11800/36938 -- GLOBAL_STEP: 11800 | > loss_text_ce: 0.20649999380111694 (0.22162046670661134) | > loss_mel_ce: 4.6403279304504395 (nan) | > loss: 0.019233444705605507 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13479372107376536) | > loader_time: 0.0043 (0.00452052482103897)  --> TIME: 2025-02-23 20:29:50 -- STEP: 11850/36938 -- GLOBAL_STEP: 11850 | > loss_text_ce: 0.2083854079246521 (0.22152790488573065) | > loss_mel_ce: 4.943361282348633 (nan) | > loss: 0.020443441346287727 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13479980070379757) | > loader_time: 0.0045 (0.004520510641331409)  --> TIME: 2025-02-23 20:29:59 -- STEP: 11900/36938 -- GLOBAL_STEP: 11900 | > loss_text_ce: 0.20889802277088165 (0.22143097114537943) | > loss_mel_ce: 4.98954439163208 (nan) | > loss: 0.02062874101102352 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.1347830028093165) | > loader_time: 0.0044 (0.004520689860111522)  --> TIME: 2025-02-23 20:30:09 -- STEP: 11950/36938 -- GLOBAL_STEP: 11950 | > loss_text_ce: 0.19112597405910492 (0.2213285410017648) | > loss_mel_ce: 0.2119598388671875 (nan) | > loss: 0.0015995469875633717 (nan) | > current_lr: 5e-06 | > step_time: 0.1533 (0.13477700875892837) | > loader_time: 0.0044 (0.004520969151452993)  --> TIME: 2025-02-23 20:30:18 -- STEP: 12000/36938 -- GLOBAL_STEP: 12000 | > loss_text_ce: 0.1989406794309616 (0.22123498657221596) | > loss_mel_ce: 5.253280162811279 (nan) | > loss: 0.02163579873740673 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.13477729986111356) | > loader_time: 0.0045 (0.004521494229634599)  --> TIME: 2025-02-23 20:30:27 -- STEP: 12050/36938 -- GLOBAL_STEP: 12050 | > loss_text_ce: 0.17822784185409546 (0.22114711957353775) | > loss_mel_ce: 5.2199201583862305 (nan) | > loss: 0.021421223878860474 (nan) | > current_lr: 5e-06 | > step_time: 0.1903 (0.13478212669182646) | > loader_time: 0.0048 (0.004521419616161042)  --> TIME: 2025-02-23 20:30:36 -- STEP: 12100/36938 -- GLOBAL_STEP: 12100 | > loss_text_ce: 0.18711425364017487 (0.22105843831192362) | > loss_mel_ce: 4.703726768493652 (nan) | > loss: 0.019408101215958595 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.13477712540587156) | > loader_time: 0.0045 (0.004521906375885005)  --> TIME: 2025-02-23 20:30:45 -- STEP: 12150/36938 -- GLOBAL_STEP: 12150 | > loss_text_ce: 0.17189174890518188 (0.22096462618421625) | > loss_mel_ce: 5.022057056427002 (nan) | > loss: 0.020610908046364784 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13477093623989447) | > loader_time: 0.0044 (0.004521391264205107)  --> TIME: 2025-02-23 20:30:55 -- STEP: 12200/36938 -- GLOBAL_STEP: 12200 | > loss_text_ce: 0.19346408545970917 (0.22087924878616802) | > loss_mel_ce: 5.2989959716796875 (nan) | > loss: 0.021795477718114853 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.1347596496636756) | > loader_time: 0.0053 (0.004521072340793286)  --> TIME: 2025-02-23 20:31:04 -- STEP: 12250/36938 -- GLOBAL_STEP: 12250 | > loss_text_ce: 0.1908423900604248 (0.22078671005915623) | > loss_mel_ce: 0.18095703423023224 (nan) | > loss: 0.001475394587032497 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13477095675955078) | > loader_time: 0.0044 (0.004520942279270707)  --> TIME: 2025-02-23 20:31:13 -- STEP: 12300/36938 -- GLOBAL_STEP: 12300 | > loss_text_ce: 0.20733553171157837 (0.22069590041065604) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1077 (0.13476238320513537) | > loader_time: 0.0043 (0.004520561811400613)  --> TIME: 2025-02-23 20:31:22 -- STEP: 12350/36938 -- GLOBAL_STEP: 12350 | > loss_text_ce: 0.19452328979969025 (0.22061517354568488) | > loss_mel_ce: 4.027350902557373 (nan) | > loss: 0.01675347052514553 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13474818677554795) | > loader_time: 0.0045 (0.0045202335365387945)  --> TIME: 2025-02-23 20:31:31 -- STEP: 12400/36938 -- GLOBAL_STEP: 12400 | > loss_text_ce: 0.34468749165534973 (0.22054146230821647) | > loss_mel_ce: 0.296875 (nan) | > loss: 0.002545882947742939 (nan) | > current_lr: 5e-06 | > step_time: 0.102 (0.13472757722100967) | > loader_time: 0.0044 (0.004519854822466438)  --> TIME: 2025-02-23 20:31:40 -- STEP: 12450/36938 -- GLOBAL_STEP: 12450 | > loss_text_ce: 0.17903052270412445 (0.22044546670104606) | > loss_mel_ce: 4.994385242462158 (nan) | > loss: 0.020529428496956825 (nan) | > current_lr: 5e-06 | > step_time: 0.1842 (0.1347278654910481) | > loader_time: 0.0046 (0.004519714370788809)  --> TIME: 2025-02-23 20:31:50 -- STEP: 12500/36938 -- GLOBAL_STEP: 12500 | > loss_text_ce: 0.19557291269302368 (0.22034488598585128) | > loss_mel_ce: 4.003336429595947 (nan) | > loss: 0.016662338748574257 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13473910257339544) | > loader_time: 0.0045 (0.004519807357788075)  --> TIME: 2025-02-23 20:31:58 -- STEP: 12550/36938 -- GLOBAL_STEP: 12550 | > loss_text_ce: 0.2057386338710785 (0.22027750194785128) | > loss_mel_ce: 5.086200714111328 (nan) | > loss: 0.020999759435653687 (nan) | > current_lr: 5e-06 | > step_time: 0.1034 (0.13471702003859023) | > loader_time: 0.0043 (0.004519603755844531)  --> TIME: 2025-02-23 20:32:08 -- STEP: 12600/36938 -- GLOBAL_STEP: 12600 | > loss_text_ce: 0.19505415856838226 (0.2201909722955454) | > loss_mel_ce: 4.615477085113525 (nan) | > loss: 0.019089410081505775 (nan) | > current_lr: 5e-06 | > step_time: 0.1515 (0.1347121663888302) | > loader_time: 0.0049 (0.004519380614871059)  --> TIME: 2025-02-23 20:32:17 -- STEP: 12650/36938 -- GLOBAL_STEP: 12650 | > loss_text_ce: 0.18715588748455048 (0.22009805617714118) | > loss_mel_ce: 4.593578338623047 (nan) | > loss: 0.018971167504787445 (nan) | > current_lr: 5e-06 | > step_time: 0.1848 (0.13470379499578722) | > loader_time: 0.0048 (0.004519293939643213)  --> TIME: 2025-02-23 20:32:26 -- STEP: 12700/36938 -- GLOBAL_STEP: 12700 | > loss_text_ce: 0.20657050609588623 (0.2200190915660126) | > loss_mel_ce: 4.935606002807617 (nan) | > loss: 0.020405463874340057 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13470856769817138) | > loader_time: 0.0043 (0.004519244948710033)  --> TIME: 2025-02-23 20:32:35 -- STEP: 12750/36938 -- GLOBAL_STEP: 12750 | > loss_text_ce: 0.19270089268684387 (0.21992980000902626) | > loss_mel_ce: 5.9228997230529785 (nan) | > loss: 0.0242682583630085 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.13471251680336796) | > loader_time: 0.0044 (0.004519046315959841)  --> TIME: 2025-02-23 20:32:44 -- STEP: 12800/36938 -- GLOBAL_STEP: 12800 | > loss_text_ce: 0.2448863536119461 (0.2198479062539991) | > loss_mel_ce: 5.283518314361572 (nan) | > loss: 0.02193811535835266 (nan) | > current_lr: 5e-06 | > step_time: 0.1064 (0.13470711329951934) | > loader_time: 0.0042 (0.0045193321257829535)  --> TIME: 2025-02-23 20:32:54 -- STEP: 12850/36938 -- GLOBAL_STEP: 12850 | > loss_text_ce: 0.21231770515441895 (0.21975691379392193) | > loss_mel_ce: 5.266888618469238 (nan) | > loss: 0.02174288220703602 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13470510746254588) | > loader_time: 0.0046 (0.004519067886738452)  --> TIME: 2025-02-23 20:33:03 -- STEP: 12900/36938 -- GLOBAL_STEP: 12900 | > loss_text_ce: 0.1870703101158142 (0.21967424223589344) | > loss_mel_ce: 4.744101047515869 (nan) | > loss: 0.01956814154982567 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13469221078148236) | > loader_time: 0.0044 (0.0045189159230668305)  --> TIME: 2025-02-23 20:33:12 -- STEP: 12950/36938 -- GLOBAL_STEP: 12950 | > loss_text_ce: 0.19624999165534973 (0.21957607599644127) | > loss_mel_ce: 4.955802917480469 (nan) | > loss: 0.020444655790925026 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.1347147636524044) | > loader_time: 0.0041 (0.004518399201765011)  --> TIME: 2025-02-23 20:33:22 -- STEP: 13000/36938 -- GLOBAL_STEP: 13000 | > loss_text_ce: 0.1834183633327484 (0.21947194135991427) | > loss_mel_ce: 4.817134380340576 (nan) | > loss: 0.019843464717268944 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.13471765905160232) | > loader_time: 0.0043 (0.004517722038122314)  --> TIME: 2025-02-23 20:33:31 -- STEP: 13050/36938 -- GLOBAL_STEP: 13050 | > loss_text_ce: 0.19510795176029205 (0.21939767280986025) | > loss_mel_ce: 5.179508209228516 (nan) | > loss: 0.021327843889594078 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13470931756542173) | > loader_time: 0.0046 (0.0045170661955500815)  --> TIME: 2025-02-23 20:33:40 -- STEP: 13100/36938 -- GLOBAL_STEP: 13100 | > loss_text_ce: 0.19828124344348907 (0.2193140909109862) | > loss_mel_ce: 4.6139678955078125 (nan) | > loss: 0.01909622736275196 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13470585712039804) | > loader_time: 0.0043 (0.0045165469628253825)  --> TIME: 2025-02-23 20:33:49 -- STEP: 13150/36938 -- GLOBAL_STEP: 13150 | > loss_text_ce: 0.20097500085830688 (0.21922513824678647) | > loss_mel_ce: 4.444937705993652 (nan) | > loss: 0.018436161801218987 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13468627527186236) | > loader_time: 0.0043 (0.004516144542186423)  --> TIME: 2025-02-23 20:33:58 -- STEP: 13200/36938 -- GLOBAL_STEP: 13200 | > loss_text_ce: 0.18211804330348969 (0.21914078777486629) | > loss_mel_ce: 5.571573257446289 (nan) | > loss: 0.022832108661532402 (nan) | > current_lr: 5e-06 | > step_time: 0.1261 (0.1346814406279368) | > loader_time: 0.0042 (0.004515851970874892)  --> TIME: 2025-02-23 20:34:07 -- STEP: 13250/36938 -- GLOBAL_STEP: 13250 | > loss_text_ce: 0.21512500941753387 (0.21904141300241903) | > loss_mel_ce: 4.852941513061523 (nan) | > loss: 0.02011137641966343 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13469724806299802) | > loader_time: 0.0047 (0.004515902843115456)  --> TIME: 2025-02-23 20:34:17 -- STEP: 13300/36938 -- GLOBAL_STEP: 13300 | > loss_text_ce: 0.1884782612323761 (0.21895081116068632) | > loss_mel_ce: 3.5329158306121826 (nan) | > loss: 0.014767437241971493 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13470463078721176) | > loader_time: 0.0045 (0.004515711060143942)  --> TIME: 2025-02-23 20:34:26 -- STEP: 13350/36938 -- GLOBAL_STEP: 13350 | > loss_text_ce: 0.18694745004177094 (0.2188599912928285) | > loss_mel_ce: 3.788062572479248 (nan) | > loss: 0.015773849561810493 (nan) | > current_lr: 5e-06 | > step_time: 0.1626 (0.13473238562823275) | > loader_time: 0.0044 (0.00451572009240196)  --> TIME: 2025-02-23 20:34:36 -- STEP: 13400/36938 -- GLOBAL_STEP: 13400 | > loss_text_ce: 0.17532691359519958 (0.2187628281049764) | > loss_mel_ce: 4.889030933380127 (nan) | > loss: 0.020096659660339355 (nan) | > current_lr: 5e-06 | > step_time: 0.1662 (0.1347290313421794) | > loader_time: 0.0043 (0.004515394239283313)  --> TIME: 2025-02-23 20:34:45 -- STEP: 13450/36938 -- GLOBAL_STEP: 13450 | > loss_text_ce: 0.22374999523162842 (0.21868952424109647) | > loss_mel_ce: 4.971153736114502 (nan) | > loss: 0.02061469852924347 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.13472575637930873) | > loader_time: 0.0046 (0.004515303023242591)  --> TIME: 2025-02-23 20:34:54 -- STEP: 13500/36938 -- GLOBAL_STEP: 13500 | > loss_text_ce: 0.24070312082767487 (0.21860310204382297) | > loss_mel_ce: 3.3144099712371826 (nan) | > loss: 0.014107592403888702 (nan) | > current_lr: 5e-06 | > step_time: 0.1037 (0.1347288410045486) | > loader_time: 0.0044 (0.004514997923815689)  --> TIME: 2025-02-23 20:35:03 -- STEP: 13550/36938 -- GLOBAL_STEP: 13550 | > loss_text_ce: 0.1866264045238495 (0.21852560907932225) | > loss_mel_ce: 5.285628795623779 (nan) | > loss: 0.021715300157666206 (nan) | > current_lr: 5e-06 | > step_time: 0.1797 (0.13471711435001316) | > loader_time: 0.0046 (0.0045151848810625235)  --> TIME: 2025-02-23 20:35:12 -- STEP: 13600/36938 -- GLOBAL_STEP: 13600 | > loss_text_ce: 0.17503905296325684 (0.21844672908677773) | > loss_mel_ce: 5.373517036437988 (nan) | > loss: 0.022018082439899445 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13470860004425095) | > loader_time: 0.0045 (0.004515450491624718)  --> TIME: 2025-02-23 20:35:21 -- STEP: 13650/36938 -- GLOBAL_STEP: 13650 | > loss_text_ce: 0.20011363923549652 (0.21836057582508514) | > loss_mel_ce: 5.008767604827881 (nan) | > loss: 0.020670166239142418 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.1347030059933228) | > loader_time: 0.0047 (0.004515389976920659)  --> TIME: 2025-02-23 20:35:31 -- STEP: 13700/36938 -- GLOBAL_STEP: 13700 | > loss_text_ce: 0.23484374582767487 (0.21827481507276097) | > loss_mel_ce: 3.396768569946289 (nan) | > loss: 0.014411160722374916 (nan) | > current_lr: 5e-06 | > step_time: 0.1076 (0.13472869269169194) | > loader_time: 0.0043 (0.004515499859830751)  --> TIME: 2025-02-23 20:35:40 -- STEP: 13750/36938 -- GLOBAL_STEP: 13750 | > loss_text_ce: 0.2122812420129776 (0.21818796500075946) | > loss_mel_ce: 4.271552562713623 (nan) | > loss: 0.017792992293834686 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.134731610003385) | > loader_time: 0.0044 (0.004515375206687228)  --> TIME: 2025-02-23 20:35:49 -- STEP: 13800/36938 -- GLOBAL_STEP: 13800 | > loss_text_ce: 0.18599264323711395 (0.21809594740038332) | > loss_mel_ce: 4.9417724609375 (nan) | > loss: 0.02034827508032322 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13472119224244292) | > loader_time: 0.005 (0.004515142354412355)  --> TIME: 2025-02-23 20:35:58 -- STEP: 13850/36938 -- GLOBAL_STEP: 13850 | > loss_text_ce: 0.19418631494045258 (0.21801674136723853) | > loss_mel_ce: 6.078881740570068 (nan) | > loss: 0.024893127381801605 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13472275880269646) | > loader_time: 0.0048 (0.004515008909177266)  --> TIME: 2025-02-23 20:36:07 -- STEP: 13900/36938 -- GLOBAL_STEP: 13900 | > loss_text_ce: 0.2351406216621399 (0.21793407134550938) | > loss_mel_ce: 8.4375 (nan) | > loss: 0.034415245056152344 (nan) | > current_lr: 5e-06 | > step_time: 0.1072 (0.13470032542729576) | > loader_time: 0.0043 (0.004515008017313566)  --> TIME: 2025-02-23 20:36:16 -- STEP: 13950/36938 -- GLOBAL_STEP: 13950 | > loss_text_ce: 0.20143228769302368 (0.2178553818225006) | > loss_mel_ce: 2.8105428218841553 (nan) | > loss: 0.01195228286087513 (nan) | > current_lr: 5e-06 | > step_time: 0.1058 (0.13468972694916564) | > loader_time: 0.0044 (0.004514899527300216)  --> TIME: 2025-02-23 20:36:25 -- STEP: 14000/36938 -- GLOBAL_STEP: 14000 | > loss_text_ce: 0.18828701972961426 (0.21776115731256349) | > loss_mel_ce: 5.0515875816345215 (nan) | > loss: 0.02079315483570099 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13468391907215146) | > loader_time: 0.0044 (0.004514533604894361)  --> TIME: 2025-02-23 20:36:35 -- STEP: 14050/36938 -- GLOBAL_STEP: 14050 | > loss_text_ce: 0.1801411211490631 (0.21767487862886484) | > loss_mel_ce: 4.718903541564941 (nan) | > loss: 0.019440654665231705 (nan) | > current_lr: 5e-06 | > step_time: 0.1622 (0.13469080639903683) | > loader_time: 0.0048 (0.004514186390778342)  --> TIME: 2025-02-23 20:36:44 -- STEP: 14100/36938 -- GLOBAL_STEP: 14100 | > loss_text_ce: 0.18702499568462372 (0.21758520993886266) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1516 (0.13468824246251013) | > loader_time: 0.0045 (0.0045137709928742475)  --> TIME: 2025-02-23 20:36:53 -- STEP: 14150/36938 -- GLOBAL_STEP: 14150 | > loss_text_ce: 0.19112330675125122 (0.21749530297496714) | > loss_mel_ce: 5.25755500793457 (nan) | > loss: 0.021621741354465485 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13468895721772553) | > loader_time: 0.0043 (0.004513334082209173)  --> TIME: 2025-02-23 20:37:03 -- STEP: 14200/36938 -- GLOBAL_STEP: 14200 | > loss_text_ce: 0.18281862139701843 (0.21739366339231042) | > loss_mel_ce: 1.760035514831543 (nan) | > loss: 0.00770973926410079 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13469237644907478) | > loader_time: 0.0044 (0.004512863461400413)  --> TIME: 2025-02-23 20:37:12 -- STEP: 14250/36938 -- GLOBAL_STEP: 14250 | > loss_text_ce: 0.22164401412010193 (0.2172942152389309) | > loss_mel_ce: 4.578062534332275 (nan) | > loss: 0.01904645562171936 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13471857124462494) | > loader_time: 0.0045 (0.004512835686666916)  --> TIME: 2025-02-23 20:37:21 -- STEP: 14300/36938 -- GLOBAL_STEP: 14300 | > loss_text_ce: 0.21254464983940125 (0.21721928091315956) | > loss_mel_ce: 4.493260860443115 (nan) | > loss: 0.018673831596970558 (nan) | > current_lr: 5e-06 | > step_time: 0.1078 (0.13469523601598704) | > loader_time: 0.0042 (0.004512409346920619)  --> TIME: 2025-02-23 20:37:30 -- STEP: 14350/36938 -- GLOBAL_STEP: 14350 | > loss_text_ce: 0.21724431216716766 (0.21714564261951513) | > loss_mel_ce: 4.5506486892700195 (nan) | > loss: 0.018920211121439934 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13468720125404426) | > loader_time: 0.0044 (0.004512358176998968)  --> TIME: 2025-02-23 20:37:39 -- STEP: 14400/36938 -- GLOBAL_STEP: 14400 | > loss_text_ce: 0.20398648083209991 (0.21706532932817935) | > loss_mel_ce: 4.99661922454834 (nan) | > loss: 0.020637325942516327 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13466873157355555) | > loader_time: 0.0046 (0.004512290805578224)  --> TIME: 2025-02-23 20:37:48 -- STEP: 14450/36938 -- GLOBAL_STEP: 14450 | > loss_text_ce: 0.2002410590648651 (0.21697058133291125) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13467090484592775) | > loader_time: 0.0047 (0.0045122002235333325)  --> TIME: 2025-02-23 20:37:58 -- STEP: 14500/36938 -- GLOBAL_STEP: 14500 | > loss_text_ce: 0.18962208926677704 (0.21688001745425423) | > loss_mel_ce: 4.506144046783447 (nan) | > loss: 0.018633993342518806 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13467666190246094) | > loader_time: 0.0044 (0.004512163392428686)  --> TIME: 2025-02-23 20:38:07 -- STEP: 14550/36938 -- GLOBAL_STEP: 14550 | > loss_text_ce: 0.18763239681720734 (0.21678678965343232) | > loss_mel_ce: 3.9374711513519287 (nan) | > loss: 0.01636945828795433 (nan) | > current_lr: 5e-06 | > step_time: 0.1591 (0.13469452720327513) | > loader_time: 0.0043 (0.0045120950253149)  --> TIME: 2025-02-23 20:38:16 -- STEP: 14600/36938 -- GLOBAL_STEP: 14600 | > loss_text_ce: 0.1888561248779297 (0.2167032425564854) | > loss_mel_ce: 5.461781024932861 (nan) | > loss: 0.022423164919018745 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.1346718691963043) | > loader_time: 0.0044 (0.004511573657597573)  --> TIME: 2025-02-23 20:38:25 -- STEP: 14650/36938 -- GLOBAL_STEP: 14650 | > loss_text_ce: 0.20721353590488434 (0.21661342608033593) | > loss_mel_ce: 4.48621940612793 (nan) | > loss: 0.01862473413348198 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13467497599409756) | > loader_time: 0.0043 (0.0045110733598572325)  --> TIME: 2025-02-23 20:38:35 -- STEP: 14700/36938 -- GLOBAL_STEP: 14700 | > loss_text_ce: 0.18940217792987823 (0.21651719780803538) | > loss_mel_ce: 5.63297176361084 (nan) | > loss: 0.023104660212993622 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13468513154659195) | > loader_time: 0.0044 (0.004510637675823801)  --> TIME: 2025-02-23 20:38:44 -- STEP: 14750/36938 -- GLOBAL_STEP: 14750 | > loss_text_ce: 0.1792140156030655 (0.216440719994448) | > loss_mel_ce: 1.6155332326889038 (nan) | > loss: 0.0071220132522284985 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13467618096885098) | > loader_time: 0.0043 (0.004510011543661854)  --> TIME: 2025-02-23 20:38:53 -- STEP: 14800/36938 -- GLOBAL_STEP: 14800 | > loss_text_ce: 0.20197543501853943 (0.2163504743122974) | > loss_mel_ce: 3.479996681213379 (nan) | > loss: 0.014611001126468182 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.1346806454658514) | > loader_time: 0.0046 (0.004509511847753775)  --> TIME: 2025-02-23 20:39:03 -- STEP: 14850/36938 -- GLOBAL_STEP: 14850 | > loss_text_ce: 0.21953123807907104 (0.2162594565899685) | > loss_mel_ce: 2.0038411617279053 (nan) | > loss: 0.008822906762361526 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13470371795423144) | > loader_time: 0.005 (0.004509464658872038)  --> TIME: 2025-02-23 20:39:12 -- STEP: 14900/36938 -- GLOBAL_STEP: 14900 | > loss_text_ce: 0.18855157494544983 (0.21617460736492336) | > loss_mel_ce: 5.038934230804443 (nan) | > loss: 0.020743992179632187 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.13470028306013837) | > loader_time: 0.0047 (0.004509704881066437)  --> TIME: 2025-02-23 20:39:21 -- STEP: 14950/36938 -- GLOBAL_STEP: 14950 | > loss_text_ce: 0.18867996335029602 (0.21608501718594478) | > loss_mel_ce: 4.9214253425598145 (nan) | > loss: 0.020278196781873703 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1347127554807382) | > loader_time: 0.0044 (0.004509659435438066)  --> TIME: 2025-02-23 20:39:31 -- STEP: 15000/36938 -- GLOBAL_STEP: 15000 | > loss_text_ce: 0.2415144145488739 (0.21600084388256072) | > loss_mel_ce: 2.8169362545013428 (nan) | > loss: 0.012136709876358509 (nan) | > current_lr: 5e-06 | > step_time: 0.1086 (0.13472198003133196) | > loader_time: 0.0043 (0.004509811607996615)  --> TIME: 2025-02-23 20:39:40 -- STEP: 15050/36938 -- GLOBAL_STEP: 15050 | > loss_text_ce: 0.19483953714370728 (0.21592551473664287) | > loss_mel_ce: 5.318323612213135 (nan) | > loss: 0.021877633407711983 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13471842388774144) | > loader_time: 0.0045 (0.004510036544546329)  --> TIME: 2025-02-23 20:39:49 -- STEP: 15100/36938 -- GLOBAL_STEP: 15100 | > loss_text_ce: 0.16503123939037323 (0.21583512281246534) | > loss_mel_ce: 5.026065826416016 (nan) | > loss: 0.020599594339728355 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.134718129429597) | > loader_time: 0.0045 (0.004509599398303498)  --> TIME: 2025-02-23 20:39:58 -- STEP: 15150/36938 -- GLOBAL_STEP: 15150 | > loss_text_ce: 0.18118950724601746 (0.21574629976902857) | > loss_mel_ce: 4.025692462921143 (nan) | > loss: 0.016693977639079094 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.1347158955426115) | > loader_time: 0.0042 (0.004509353228528107)  --> TIME: 2025-02-23 20:40:07 -- STEP: 15200/36938 -- GLOBAL_STEP: 15200 | > loss_text_ce: 0.20510415732860565 (0.21566490349310793) | > loss_mel_ce: 4.825148582458496 (nan) | > loss: 0.019961321726441383 (nan) | > current_lr: 5e-06 | > step_time: 0.1066 (0.13471022723536832) | > loader_time: 0.0043 (0.004509647268997988)  --> TIME: 2025-02-23 20:40:17 -- STEP: 15250/36938 -- GLOBAL_STEP: 15250 | > loss_text_ce: 0.19903531670570374 (0.21557015990624662) | > loss_mel_ce: 0.1094970703125 (nan) | > loss: 0.0012243349337950349 (nan) | > current_lr: 5e-06 | > step_time: 0.1086 (0.1347263326175886) | > loader_time: 0.0045 (0.0045100141275124395)  --> TIME: 2025-02-23 20:40:26 -- STEP: 15300/36938 -- GLOBAL_STEP: 15300 | > loss_text_ce: 0.2134999930858612 (0.2154842226986402) | > loss_mel_ce: 0.15851236879825592 (nan) | > loss: 0.0014762396458536386 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.1347304475541217) | > loader_time: 0.0045 (0.00450998525993496)  --> TIME: 2025-02-23 20:40:35 -- STEP: 15350/36938 -- GLOBAL_STEP: 15350 | > loss_text_ce: 0.19103161990642548 (0.215412308714483) | > loss_mel_ce: 4.806957721710205 (nan) | > loss: 0.019833290949463844 (nan) | > current_lr: 5e-06 | > step_time: 0.1873 (0.1347298245787241) | > loader_time: 0.0047 (0.004509824386248752)  --> TIME: 2025-02-23 20:40:45 -- STEP: 15400/36938 -- GLOBAL_STEP: 15400 | > loss_text_ce: 0.19840145111083984 (0.2153338702622946) | > loss_mel_ce: 5.098487377166748 (nan) | > loss: 0.021019401028752327 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13473237622868017) | > loader_time: 0.0044 (0.004509822935252987)  --> TIME: 2025-02-23 20:40:54 -- STEP: 15450/36938 -- GLOBAL_STEP: 15450 | > loss_text_ce: 0.20351561903953552 (0.21524822138659777) | > loss_mel_ce: 3.259521484375 (nan) | > loss: 0.013742211274802685 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13472988307669023) | > loader_time: 0.0045 (0.00450995354204887)  --> TIME: 2025-02-23 20:41:03 -- STEP: 15500/36938 -- GLOBAL_STEP: 15500 | > loss_text_ce: 0.2032812386751175 (0.21518110122988302) | > loss_mel_ce: 4.513599872589111 (nan) | > loss: 0.01871778443455696 (nan) | > current_lr: 5e-06 | > step_time: 0.1046 (0.13470721506303443) | > loader_time: 0.0042 (0.00450960134690807)  --> TIME: 2025-02-23 20:41:12 -- STEP: 15550/36938 -- GLOBAL_STEP: 15550 | > loss_text_ce: 0.17142359912395477 (0.2151036004535255) | > loss_mel_ce: 4.777475833892822 (nan) | > loss: 0.019638489931821823 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13469954190146968) | > loader_time: 0.0043 (0.00450912030946786)  --> TIME: 2025-02-23 20:41:21 -- STEP: 15600/36938 -- GLOBAL_STEP: 15600 | > loss_text_ce: 0.18281249701976776 (0.2150184600523267) | > loss_mel_ce: 5.084797382354736 (nan) | > loss: 0.020903216674923897 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13470020900934254) | > loader_time: 0.0042 (0.004508742873485265)  --> TIME: 2025-02-23 20:41:30 -- STEP: 15650/36938 -- GLOBAL_STEP: 15650 | > loss_text_ce: 0.17681695520877838 (0.21493486608845738) | > loss_mel_ce: 5.943509578704834 (nan) | > loss: 0.024287011474370956 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.1347042140412263) | > loader_time: 0.0048 (0.004508391767264167)  --> TIME: 2025-02-23 20:41:40 -- STEP: 15700/36938 -- GLOBAL_STEP: 15700 | > loss_text_ce: 0.17701612412929535 (0.21483546026383235) | > loss_mel_ce: 4.9002885818481445 (nan) | > loss: 0.020148037001490593 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.1347207135911209) | > loader_time: 0.0044 (0.004508313282280204)  --> TIME: 2025-02-23 20:41:49 -- STEP: 15750/36938 -- GLOBAL_STEP: 15750 | > loss_text_ce: 0.19751837849617004 (0.21474611695229062) | > loss_mel_ce: 4.625079154968262 (nan) | > loss: 0.019137293100357056 (nan) | > current_lr: 5e-06 | > step_time: 0.1081 (0.13470757205902667) | > loader_time: 0.0045 (0.004508074669610879)  --> TIME: 2025-02-23 20:41:58 -- STEP: 15800/36938 -- GLOBAL_STEP: 15800 | > loss_text_ce: 0.1859321892261505 (0.21466261602843864) | > loss_mel_ce: 4.582737445831299 (nan) | > loss: 0.01892329379916191 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13469256518762407) | > loader_time: 0.0044 (0.004507812518107733)  --> TIME: 2025-02-23 20:42:07 -- STEP: 15850/36938 -- GLOBAL_STEP: 15850 | > loss_text_ce: 0.17774038016796112 (0.21458318711455313) | > loss_mel_ce: 4.649927616119385 (nan) | > loss: 0.019157415255904198 (nan) | > current_lr: 5e-06 | > step_time: 0.1764 (0.13468921456803365) | > loader_time: 0.0052 (0.004507927172567559)  --> TIME: 2025-02-23 20:42:16 -- STEP: 15900/36938 -- GLOBAL_STEP: 15900 | > loss_text_ce: 0.17884868383407593 (0.21449947961939955) | > loss_mel_ce: 4.194769859313965 (nan) | > loss: 0.017355630174279213 (nan) | > current_lr: 5e-06 | > step_time: 0.1266 (0.13467054644470777) | > loader_time: 0.0046 (0.004508280994007415)  --> TIME: 2025-02-23 20:42:25 -- STEP: 15950/36938 -- GLOBAL_STEP: 15950 | > loss_text_ce: 0.1807386428117752 (0.2144216733647738) | > loss_mel_ce: 4.707639694213867 (nan) | > loss: 0.01939832791686058 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13466372488434494) | > loader_time: 0.0043 (0.004508106118085609)  --> TIME: 2025-02-23 20:42:35 -- STEP: 16000/36938 -- GLOBAL_STEP: 16000 | > loss_text_ce: 0.20628677308559418 (0.21433443881291897) | > loss_mel_ce: 3.6087443828582764 (nan) | > loss: 0.015139012597501278 (nan) | > current_lr: 5e-06 | > step_time: 0.1121 (0.13468260335922325) | > loader_time: 0.0042 (0.004507853969931595)  --> TIME: 2025-02-23 20:42:44 -- STEP: 16050/36938 -- GLOBAL_STEP: 16050 | > loss_text_ce: 0.17000000178813934 (0.21424398122732513) | > loss_mel_ce: 4.9072394371032715 (nan) | > loss: 0.0201477762311697 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13470354246573368) | > loader_time: 0.0044 (0.004507589488757355)  --> TIME: 2025-02-23 20:42:53 -- STEP: 16100/36938 -- GLOBAL_STEP: 16100 | > loss_text_ce: 0.1797453612089157 (0.21417750049544418) | > loss_mel_ce: 4.860033988952637 (nan) | > loss: 0.019999124109745026 (nan) | > current_lr: 5e-06 | > step_time: 0.1769 (0.13468570288664147) | > loader_time: 0.0044 (0.004507120040632916)  --> TIME: 2025-02-23 20:43:03 -- STEP: 16150/36938 -- GLOBAL_STEP: 16150 | > loss_text_ce: 0.2551562488079071 (0.21409049768289176) | > loss_mel_ce: 3.069220781326294 (nan) | > loss: 0.013191972859203815 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13470931521141205) | > loader_time: 0.0043 (0.004506909249367722)  --> TIME: 2025-02-23 20:43:12 -- STEP: 16200/36938 -- GLOBAL_STEP: 16200 | > loss_text_ce: 0.18493205308914185 (0.21400441492413297) | > loss_mel_ce: 3.879025936126709 (nan) | > loss: 0.01612681895494461 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13470681627591533) | > loader_time: 0.0045 (0.004506632825474673)  --> TIME: 2025-02-23 20:43:21 -- STEP: 16250/36938 -- GLOBAL_STEP: 16250 | > loss_text_ce: 0.20478123426437378 (0.21392443933853736) | > loss_mel_ce: 5.111954689025879 (nan) | > loss: 0.02109815925359726 (nan) | > current_lr: 5e-06 | > step_time: 0.1066 (0.13469307438777126) | > loader_time: 0.0045 (0.004506296568650459)  --> TIME: 2025-02-23 20:43:30 -- STEP: 16300/36938 -- GLOBAL_STEP: 16300 | > loss_text_ce: 0.19337499141693115 (0.21384467779858712) | > loss_mel_ce: 8.203125 (nan) | > loss: 0.033319443464279175 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13468996750065235) | > loader_time: 0.0044 (0.0045057918835271285)  --> TIME: 2025-02-23 20:43:39 -- STEP: 16350/36938 -- GLOBAL_STEP: 16350 | > loss_text_ce: 0.17989465594291687 (0.2137629677259594) | > loss_mel_ce: 5.225035667419434 (nan) | > loss: 0.02144813723862171 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.13467470303223483) | > loader_time: 0.0046 (0.004505254544249361)  --> TIME: 2025-02-23 20:43:48 -- STEP: 16400/36938 -- GLOBAL_STEP: 16400 | > loss_text_ce: 0.19372498989105225 (0.21367973628385764) | > loss_mel_ce: 6.489583492279053 (nan) | > loss: 0.026521068066358566 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1346722832104063) | > loader_time: 0.0042 (0.004504872516887937)  --> TIME: 2025-02-23 20:43:57 -- STEP: 16450/36938 -- GLOBAL_STEP: 16450 | > loss_text_ce: 0.1957850605249405 (0.2136131747886765) | > loss_mel_ce: 1.2600795030593872 (nan) | > loss: 0.005777240730822086 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13465818503707455) | > loader_time: 0.0045 (0.004504392067349783)  --> TIME: 2025-02-23 20:44:06 -- STEP: 16500/36938 -- GLOBAL_STEP: 16500 | > loss_text_ce: 0.18143749237060547 (0.21352374133557983) | > loss_mel_ce: 5.0315842628479 (nan) | > loss: 0.020686594769358635 (nan) | > current_lr: 5e-06 | > step_time: 0.1486 (0.13467098253423598) | > loader_time: 0.0046 (0.004504215457222672)  --> TIME: 2025-02-23 20:44:16 -- STEP: 16550/36938 -- GLOBAL_STEP: 16550 | > loss_text_ce: 0.17026785016059875 (0.21342991154089433) | > loss_mel_ce: 5.229105472564697 (nan) | > loss: 0.021426087245345116 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1346732368353999) | > loader_time: 0.0044 (0.004504448323091345)  --> TIME: 2025-02-23 20:44:25 -- STEP: 16600/36938 -- GLOBAL_STEP: 16600 | > loss_text_ce: 0.18063507974147797 (0.21334269274968695) | > loss_mel_ce: 4.356679439544678 (nan) | > loss: 0.018005216494202614 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13467660616679447) | > loader_time: 0.0042 (0.004504365547593811)  --> TIME: 2025-02-23 20:44:35 -- STEP: 16650/36938 -- GLOBAL_STEP: 16650 | > loss_text_ce: 0.18476973474025726 (0.21324934729763692) | > loss_mel_ce: 5.480731010437012 (nan) | > loss: 0.022482147440314293 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13469118590827542) | > loader_time: 0.0047 (0.0045043266857708476)  --> TIME: 2025-02-23 20:44:44 -- STEP: 16700/36938 -- GLOBAL_STEP: 16700 | > loss_text_ce: 0.171762153506279 (0.2131641483869024) | > loss_mel_ce: 4.677968502044678 (nan) | > loss: 0.01924496330320835 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13467879863556353) | > loader_time: 0.0044 (0.0045042716671606685)  --> TIME: 2025-02-23 20:44:52 -- STEP: 16750/36938 -- GLOBAL_STEP: 16750 | > loss_text_ce: 0.18598558008670807 (0.21308513966425136) | > loss_mel_ce: 2.284252882003784 (nan) | > loss: 0.009802534244954586 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13465470197307608) | > loader_time: 0.0043 (0.004503960936816766)  --> TIME: 2025-02-23 20:45:01 -- STEP: 16800/36938 -- GLOBAL_STEP: 16800 | > loss_text_ce: 0.17735733091831207 (0.21301182334444346) | > loss_mel_ce: 4.368350028991699 (nan) | > loss: 0.018038522452116013 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13464618092491595) | > loader_time: 0.0044 (0.0045040875531378215)  --> TIME: 2025-02-23 20:45:11 -- STEP: 16850/36938 -- GLOBAL_STEP: 16850 | > loss_text_ce: 0.17933593690395355 (0.21292558975230338) | > loss_mel_ce: 4.305755138397217 (nan) | > loss: 0.017797982320189476 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.13464803416934218) | > loader_time: 0.0046 (0.004504227072265804)  --> TIME: 2025-02-23 20:45:20 -- STEP: 16900/36938 -- GLOBAL_STEP: 16900 | > loss_text_ce: 0.17085616290569305 (0.2128315820002697) | > loss_mel_ce: 5.183280944824219 (nan) | > loss: 0.02124657668173313 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13465833843107133) | > loader_time: 0.0047 (0.004504515574528618)  --> TIME: 2025-02-23 20:45:29 -- STEP: 16950/36938 -- GLOBAL_STEP: 16950 | > loss_text_ce: 0.19305337965488434 (0.21275203191864805) | > loss_mel_ce: 4.9135918617248535 (nan) | > loss: 0.020264465361833572 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13465088833046604) | > loader_time: 0.0043 (0.00450435539965784)  --> TIME: 2025-02-23 20:45:38 -- STEP: 17000/36938 -- GLOBAL_STEP: 17000 | > loss_text_ce: 0.1712304651737213 (0.2126694843006484) | > loss_mel_ce: 5.422181606292725 (nan) | > loss: 0.022196080535650253 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.1346481464470142) | > loader_time: 0.0044 (0.004504331153981824)  --> TIME: 2025-02-23 20:45:48 -- STEP: 17050/36938 -- GLOBAL_STEP: 17050 | > loss_text_ce: 0.18892499804496765 (0.21257838524593975) | > loss_mel_ce: 4.739117622375488 (nan) | > loss: 0.019555725157260895 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13465181422023906) | > loader_time: 0.0044 (0.004504187645450713)  --> TIME: 2025-02-23 20:45:57 -- STEP: 17100/36938 -- GLOBAL_STEP: 17100 | > loss_text_ce: 0.20109912753105164 (0.21249201552038288) | > loss_mel_ce: 4.442097187042236 (nan) | > loss: 0.018425382673740387 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1346485674311551) | > loader_time: 0.0045 (0.004504124797575651)  --> TIME: 2025-02-23 20:46:06 -- STEP: 17150/36938 -- GLOBAL_STEP: 17150 | > loss_text_ce: 0.19553977251052856 (0.21241189696959065) | > loss_mel_ce: 4.157376766204834 (nan) | > loss: 0.01727348007261753 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.1346420792031922) | > loader_time: 0.0042 (0.0045038381053824445)  --> TIME: 2025-02-23 20:46:15 -- STEP: 17200/36938 -- GLOBAL_STEP: 17200 | > loss_text_ce: 0.16917552053928375 (0.21232509870428654) | > loss_mel_ce: 4.314850807189941 (nan) | > loss: 0.01779375597834587 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.13462900599768082) | > loader_time: 0.0045 (0.004503854402275971)  --> TIME: 2025-02-23 20:46:24 -- STEP: 17250/36938 -- GLOBAL_STEP: 17250 | > loss_text_ce: 0.1837216019630432 (0.2122389031873231) | > loss_mel_ce: 4.091766357421875 (nan) | > loss: 0.01696622371673584 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.13462578776262893) | > loader_time: 0.0046 (0.0045040186729983995)  --> TIME: 2025-02-23 20:46:33 -- STEP: 17300/36938 -- GLOBAL_STEP: 17300 | > loss_text_ce: 0.1769791543483734 (0.21215552555860104) | > loss_mel_ce: 5.053199291229248 (nan) | > loss: 0.020754676312208176 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13462411859821116) | > loader_time: 0.0046 (0.004503942473086316)  --> TIME: 2025-02-23 20:46:43 -- STEP: 17350/36938 -- GLOBAL_STEP: 17350 | > loss_text_ce: 0.1743987798690796 (0.21206454574804118) | > loss_mel_ce: 5.390714168548584 (nan) | > loss: 0.022083783522248268 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.1346391678131287) | > loader_time: 0.0047 (0.004503995788200438)  --> TIME: 2025-02-23 20:46:52 -- STEP: 17400/36938 -- GLOBAL_STEP: 17400 | > loss_text_ce: 0.17486192286014557 (0.2119920280541494) | > loss_mel_ce: 4.175988674163818 (nan) | > loss: 0.017265280708670616 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13463372379884023) | > loader_time: 0.0042 (0.004503874367681039)  --> TIME: 2025-02-23 20:47:01 -- STEP: 17450/36938 -- GLOBAL_STEP: 17450 | > loss_text_ce: 0.17007102072238922 (0.21190874318680283) | > loss_mel_ce: 5.524075031280518 (nan) | > loss: 0.02259581908583641 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.1346193277391807) | > loader_time: 0.0044 (0.0045036525371081465)  --> TIME: 2025-02-23 20:47:10 -- STEP: 17500/36938 -- GLOBAL_STEP: 17500 | > loss_text_ce: 0.18032391369342804 (0.21182570104088078) | > loss_mel_ce: 4.900694847106934 (nan) | > loss: 0.02016277424991131 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.1346044227463865) | > loader_time: 0.0044 (0.004503859615325924)  --> TIME: 2025-02-23 20:47:19 -- STEP: 17550/36938 -- GLOBAL_STEP: 17550 | > loss_text_ce: 0.17691965401172638 (0.2117397818852351) | > loss_mel_ce: 1.4367904663085938 (nan) | > loss: 0.006403611972928047 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13460341680423552) | > loader_time: 0.0044 (0.004504067564961573)  --> TIME: 2025-02-23 20:47:28 -- STEP: 17600/36938 -- GLOBAL_STEP: 17600 | > loss_text_ce: 0.17070913314819336 (0.2116603446099903) | > loss_mel_ce: 5.140278339385986 (nan) | > loss: 0.021075347438454628 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13461516484618255) | > loader_time: 0.0045 (0.0045041978901082785)  --> TIME: 2025-02-23 20:47:37 -- STEP: 17650/36938 -- GLOBAL_STEP: 17650 | > loss_text_ce: 0.17205792665481567 (0.21157513581912812) | > loss_mel_ce: 5.126741409301758 (nan) | > loss: 0.021026983857154846 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13460991167819603) | > loader_time: 0.0047 (0.004504377119264249)  --> TIME: 2025-02-23 20:47:47 -- STEP: 17700/36938 -- GLOBAL_STEP: 17700 | > loss_text_ce: 0.19499999284744263 (0.2114885646904589) | > loss_mel_ce: 1.445898413658142 (nan) | > loss: 0.006511502433568239 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13460949123242555) | > loader_time: 0.0042 (0.0045043215913287615)  --> TIME: 2025-02-23 20:47:56 -- STEP: 17750/36938 -- GLOBAL_STEP: 17750 | > loss_text_ce: 0.17380554974079132 (0.21140109465239712) | > loss_mel_ce: 4.824436187744141 (nan) | > loss: 0.01983429491519928 (nan) | > current_lr: 5e-06 | > step_time: 0.1352 (0.13461295995578038) | > loader_time: 0.0044 (0.004504298411624526)  --> TIME: 2025-02-23 20:48:05 -- STEP: 17800/36938 -- GLOBAL_STEP: 17800 | > loss_text_ce: 0.1735050529241562 (0.21131011093684085) | > loss_mel_ce: 4.47037935256958 (nan) | > loss: 0.01842811331152916 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.134611269249006) | > loader_time: 0.0042 (0.004504310883832777)  --> TIME: 2025-02-23 20:48:15 -- STEP: 17850/36938 -- GLOBAL_STEP: 17850 | > loss_text_ce: 0.1873437464237213 (0.21121504688964127) | > loss_mel_ce: 4.387641906738281 (nan) | > loss: 0.018154704943299294 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13462017747200578) | > loader_time: 0.0043 (0.004504323339595847)  --> TIME: 2025-02-23 20:48:24 -- STEP: 17900/36938 -- GLOBAL_STEP: 17900 | > loss_text_ce: 0.1824888437986374 (0.21112191691078908) | > loss_mel_ce: 4.535833358764648 (nan) | > loss: 0.0187235027551651 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13462531144392625) | > loader_time: 0.0044 (0.00450424113087148)  --> TIME: 2025-02-23 20:48:33 -- STEP: 17950/36938 -- GLOBAL_STEP: 17950 | > loss_text_ce: 0.17476078867912292 (0.2110428670571703) | > loss_mel_ce: 4.367579936981201 (nan) | > loss: 0.018025163561105728 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.1346325665529731) | > loader_time: 0.0046 (0.004504517127544435)  --> TIME: 2025-02-23 20:48:43 -- STEP: 18000/36938 -- GLOBAL_STEP: 18000 | > loss_text_ce: 0.16687093675136566 (0.21095350242323305) | > loss_mel_ce: 4.315726280212402 (nan) | > loss: 0.017788086086511612 (nan) | > current_lr: 5e-06 | > step_time: 0.185 (0.13465150558948588) | > loader_time: 0.0046 (0.004504791895548501)  --> TIME: 2025-02-23 20:48:52 -- STEP: 18050/36938 -- GLOBAL_STEP: 18050 | > loss_text_ce: 0.17390625178813934 (0.21086893691085346) | > loss_mel_ce: 4.927670478820801 (nan) | > loss: 0.020244354382157326 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13465593670874165) | > loader_time: 0.0044 (0.0045047021110302165)  --> TIME: 2025-02-23 20:49:01 -- STEP: 18100/36938 -- GLOBAL_STEP: 18100 | > loss_text_ce: 0.18258680403232574 (0.21079714999014454) | > loss_mel_ce: 4.470627307891846 (nan) | > loss: 0.018465135246515274 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13465702117477404) | > loader_time: 0.0044 (0.004504492414590401)  --> TIME: 2025-02-23 20:49:11 -- STEP: 18150/36938 -- GLOBAL_STEP: 18150 | > loss_text_ce: 0.1854761838912964 (0.21070477585818762) | > loss_mel_ce: 4.985605239868164 (nan) | > loss: 0.0205201655626297 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13466090285088492) | > loader_time: 0.0043 (0.004504685769724778)  --> TIME: 2025-02-23 20:49:20 -- STEP: 18200/36938 -- GLOBAL_STEP: 18200 | > loss_text_ce: 0.1702321469783783 (0.21062074315990015) | > loss_mel_ce: 5.332563400268555 (nan) | > loss: 0.02183649130165577 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13465122359139667) | > loader_time: 0.0048 (0.004504681052742422)  --> TIME: 2025-02-23 20:49:29 -- STEP: 18250/36938 -- GLOBAL_STEP: 18250 | > loss_text_ce: 0.1689828485250473 (0.21054173334330675) | > loss_mel_ce: 4.997625350952148 (nan) | > loss: 0.02050241455435753 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13465236260139862) | > loader_time: 0.0045 (0.004504577832679226)  --> TIME: 2025-02-23 20:49:39 -- STEP: 18300/36938 -- GLOBAL_STEP: 18300 | > loss_text_ce: 0.17111486196517944 (0.210452435225085) | > loss_mel_ce: 5.277144432067871 (nan) | > loss: 0.02162007801234722 (nan) | > current_lr: 5e-06 | > step_time: 0.175 (0.1346629872738998) | > loader_time: 0.0052 (0.004504509623584855)  --> TIME: 2025-02-23 20:49:48 -- STEP: 18350/36938 -- GLOBAL_STEP: 18350 | > loss_text_ce: 0.17103123664855957 (0.2103623784026913) | > loss_mel_ce: 4.926590919494629 (nan) | > loss: 0.020228659734129906 (nan) | > current_lr: 5e-06 | > step_time: 0.1628 (0.13466914564127588) | > loader_time: 0.0045 (0.004504317210870482)  --> TIME: 2025-02-23 20:49:57 -- STEP: 18400/36938 -- GLOBAL_STEP: 18400 | > loss_text_ce: 0.16115233302116394 (0.21027743694493908) | > loss_mel_ce: 4.243271827697754 (nan) | > loss: 0.01747787557542324 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1346690718505704) | > loader_time: 0.0044 (0.004504155879435337)  --> TIME: 2025-02-23 20:50:06 -- STEP: 18450/36938 -- GLOBAL_STEP: 18450 | > loss_text_ce: 0.16996528208255768 (0.2101904332500482) | > loss_mel_ce: 5.088782787322998 (nan) | > loss: 0.020868049934506416 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13467248119313013) | > loader_time: 0.0045 (0.004504107976670514)  --> TIME: 2025-02-23 20:50:16 -- STEP: 18500/36938 -- GLOBAL_STEP: 18500 | > loss_text_ce: 0.17205439507961273 (0.21010797149909471) | > loss_mel_ce: 4.433815002441406 (nan) | > loss: 0.018277259543538094 (nan) | > current_lr: 5e-06 | > step_time: 0.1349 (0.13468521439062692) | > loader_time: 0.0043 (0.004503891648472968)  --> TIME: 2025-02-23 20:50:25 -- STEP: 18550/36938 -- GLOBAL_STEP: 18550 | > loss_text_ce: 0.2016145884990692 (0.21003565017103784) | > loss_mel_ce: 2.1276044845581055 (nan) | > loss: 0.009242933243513107 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13468057985897114) | > loader_time: 0.0042 (0.004503626296462394)  --> TIME: 2025-02-23 20:50:34 -- STEP: 18600/36938 -- GLOBAL_STEP: 18600 | > loss_text_ce: 0.18593749403953552 (0.20995098305485535) | > loss_mel_ce: 4.190509796142578 (nan) | > loss: 0.01736685447394848 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1346814172626834) | > loader_time: 0.0044 (0.004503454187864895)  --> TIME: 2025-02-23 20:50:44 -- STEP: 18650/36938 -- GLOBAL_STEP: 18650 | > loss_text_ce: 0.17533451318740845 (0.2098658894286074) | > loss_mel_ce: 4.172004699707031 (nan) | > loss: 0.01725134626030922 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13469204213600158) | > loader_time: 0.0046 (0.004503320855046083)  --> TIME: 2025-02-23 20:50:53 -- STEP: 18700/36938 -- GLOBAL_STEP: 18700 | > loss_text_ce: 0.2649218738079071 (0.20978202197800302) | > loss_mel_ce: 0.158172607421875 (nan) | > loss: 0.001678946428000927 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13467955828988082) | > loader_time: 0.0046 (0.004503127291878275)  --> TIME: 2025-02-23 20:51:02 -- STEP: 18750/36938 -- GLOBAL_STEP: 18750 | > loss_text_ce: 0.1719304472208023 (0.20969913842836968) | > loss_mel_ce: 3.7333412170410156 (nan) | > loss: 0.015497110784053802 (nan) | > current_lr: 5e-06 | > step_time: 0.1513 (0.1346851155090343) | > loader_time: 0.0068 (0.004503554331461588)  --> TIME: 2025-02-23 20:51:11 -- STEP: 18800/36938 -- GLOBAL_STEP: 18800 | > loss_text_ce: 0.1808522790670395 (0.20961500169194072) | > loss_mel_ce: 4.3105854988098145 (nan) | > loss: 0.017823167145252228 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1346790392728572) | > loader_time: 0.0043 (0.004503652656331978)  --> TIME: 2025-02-23 20:51:20 -- STEP: 18850/36938 -- GLOBAL_STEP: 18850 | > loss_text_ce: 0.19592104852199554 (0.20953283727327093) | > loss_mel_ce: 4.955774307250977 (nan) | > loss: 0.020443236455321312 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13466138361619798) | > loader_time: 0.0044 (0.004503554463070331)  --> TIME: 2025-02-23 20:51:29 -- STEP: 18900/36938 -- GLOBAL_STEP: 18900 | > loss_text_ce: 0.18872499465942383 (0.2094488025602521) | > loss_mel_ce: 4.062150478363037 (nan) | > loss: 0.016868554055690765 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13466723744831488) | > loader_time: 0.0046 (0.004503557354054124)  --> TIME: 2025-02-23 20:51:39 -- STEP: 18950/36938 -- GLOBAL_STEP: 18950 | > loss_text_ce: 0.16818222403526306 (0.20935475400931564) | > loss_mel_ce: 5.01667594909668 (nan) | > loss: 0.020574836060404778 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.13467899749021203) | > loader_time: 0.0045 (0.0045036311011201185)  --> TIME: 2025-02-23 20:51:48 -- STEP: 19000/36938 -- GLOBAL_STEP: 19000 | > loss_text_ce: 0.16942647099494934 (0.20926072331086545) | > loss_mel_ce: 5.333951473236084 (nan) | > loss: 0.021838802844285965 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.13467555204190798) | > loader_time: 0.005 (0.004503899097442627)  --> TIME: 2025-02-23 20:51:57 -- STEP: 19050/36938 -- GLOBAL_STEP: 19050 | > loss_text_ce: 0.18170230090618134 (0.2091940513034188) | > loss_mel_ce: 4.1224565505981445 (nan) | > loss: 0.017079995945096016 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.1346629546007782) | > loader_time: 0.0046 (0.004504111244922547)  --> TIME: 2025-02-23 20:52:06 -- STEP: 19100/36938 -- GLOBAL_STEP: 19100 | > loss_text_ce: 0.17097175121307373 (0.20910496085258012) | > loss_mel_ce: 4.496643543243408 (nan) | > loss: 0.01852228492498398 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.1346770424368504) | > loader_time: 0.0046 (0.004504210337294332)  --> TIME: 2025-02-23 20:52:15 -- STEP: 19150/36938 -- GLOBAL_STEP: 19150 | > loss_text_ce: 0.1859615296125412 (0.2090252078412091) | > loss_mel_ce: 4.579648494720459 (nan) | > loss: 0.018911153078079224 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13465997236515878) | > loader_time: 0.0042 (0.004504228933050489)  --> TIME: 2025-02-23 20:52:24 -- STEP: 19200/36938 -- GLOBAL_STEP: 19200 | > loss_text_ce: 0.19882352650165558 (0.2089450569342195) | > loss_mel_ce: 0.0423583984375 (nan) | > loss: 0.0009570711990818381 (nan) | > current_lr: 5e-06 | > step_time: 0.1074 (0.13465054322034223) | > loader_time: 0.0043 (0.004504218436777593)  --> TIME: 2025-02-23 20:52:34 -- STEP: 19250/36938 -- GLOBAL_STEP: 19250 | > loss_text_ce: 0.187687486410141 (0.2088614614892309) | > loss_mel_ce: 2.6209053993225098 (nan) | > loss: 0.011145209893584251 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1346643676757823) | > loader_time: 0.0046 (0.00450433550252543)  --> TIME: 2025-02-23 20:52:43 -- STEP: 19300/36938 -- GLOBAL_STEP: 19300 | > loss_text_ce: 0.15718749165534973 (0.2087832827086269) | > loss_mel_ce: 4.43518590927124 (nan) | > loss: 0.018223704770207405 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.13464904796274318) | > loader_time: 0.0047 (0.004504466612721972)  --> TIME: 2025-02-23 20:52:52 -- STEP: 19350/36938 -- GLOBAL_STEP: 19350 | > loss_text_ce: 0.17954044044017792 (0.20869543835020118) | > loss_mel_ce: 4.787096977233887 (nan) | > loss: 0.01970888115465641 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1346506542383249) | > loader_time: 0.0044 (0.004504740946668678)  --> TIME: 2025-02-23 20:53:01 -- STEP: 19400/36938 -- GLOBAL_STEP: 19400 | > loss_text_ce: 0.16589374840259552 (0.20861406694137488) | > loss_mel_ce: 5.25608491897583 (nan) | > loss: 0.02151578851044178 (nan) | > current_lr: 5e-06 | > step_time: 0.1566 (0.13464490429642378) | > loader_time: 0.0044 (0.004504860762468317)  --> TIME: 2025-02-23 20:53:11 -- STEP: 19450/36938 -- GLOBAL_STEP: 19450 | > loss_text_ce: 0.1686985194683075 (0.2085203533820745) | > loss_mel_ce: 5.245135307312012 (nan) | > loss: 0.021483467891812325 (nan) | > current_lr: 5e-06 | > step_time: 0.1798 (0.13465311684155018) | > loader_time: 0.0046 (0.0045047754983975515)  --> TIME: 2025-02-23 20:53:20 -- STEP: 19500/36938 -- GLOBAL_STEP: 19500 | > loss_text_ce: 0.17398637533187866 (0.2084301476493855) | > loss_mel_ce: 5.34708309173584 (nan) | > loss: 0.02190900780260563 (nan) | > current_lr: 5e-06 | > step_time: 0.1591 (0.1346750276761187) | > loader_time: 0.0045 (0.0045049971311520295)  --> TIME: 2025-02-23 20:53:29 -- STEP: 19550/36938 -- GLOBAL_STEP: 19550 | > loss_text_ce: 0.1679810881614685 (0.20834763708779208) | > loss_mel_ce: 4.5210065841674805 (nan) | > loss: 0.018607094883918762 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.1346664534810264) | > loader_time: 0.0047 (0.004504930308407831)  --> TIME: 2025-02-23 20:53:39 -- STEP: 19600/36938 -- GLOBAL_STEP: 19600 | > loss_text_ce: 0.17142045497894287 (0.20826582854727638) | > loss_mel_ce: 5.577554225921631 (nan) | > loss: 0.02281339280307293 (nan) | > current_lr: 5e-06 | > step_time: 0.146 (0.13467486446001198) | > loader_time: 0.0045 (0.004504916862565641)  --> TIME: 2025-02-23 20:53:48 -- STEP: 19650/36938 -- GLOBAL_STEP: 19650 | > loss_text_ce: 0.18124178051948547 (0.20818631337647467) | > loss_mel_ce: 4.290929317474365 (nan) | > loss: 0.017746713012456894 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13467220938539484) | > loader_time: 0.0045 (0.004504921357443617)  --> TIME: 2025-02-23 20:53:57 -- STEP: 19700/36938 -- GLOBAL_STEP: 19700 | > loss_text_ce: 0.1642225682735443 (0.20809360171256883) | > loss_mel_ce: 4.526039123535156 (nan) | > loss: 0.018612151965498924 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13467333934028836) | > loader_time: 0.0044 (0.00450478067252842)  --> TIME: 2025-02-23 20:54:06 -- STEP: 19750/36938 -- GLOBAL_STEP: 19750 | > loss_text_ce: 0.1793859601020813 (0.20801893746551023) | > loss_mel_ce: 4.23046875 (nan) | > loss: 0.017499424517154694 (nan) | > current_lr: 5e-06 | > step_time: 0.1505 (0.13467262032665875) | > loader_time: 0.0045 (0.0045047142294388845)  --> TIME: 2025-02-23 20:54:16 -- STEP: 19800/36938 -- GLOBAL_STEP: 19800 | > loss_text_ce: 0.16742612421512604 (0.2079313829300374) | > loss_mel_ce: 5.321759223937988 (nan) | > loss: 0.02178248204290867 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13467563795321014) | > loader_time: 0.0048 (0.004505297848672585)  --> TIME: 2025-02-23 20:54:25 -- STEP: 19850/36938 -- GLOBAL_STEP: 19850 | > loss_text_ce: 0.1762239634990692 (0.20784575257178162) | > loss_mel_ce: 3.9700851440429688 (nan) | > loss: 0.016453607007861137 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13467545140600415) | > loader_time: 0.0042 (0.0045051994732104896)  --> TIME: 2025-02-23 20:54:34 -- STEP: 19900/36938 -- GLOBAL_STEP: 19900 | > loss_text_ce: 0.18062499165534973 (0.20775978734819678) | > loss_mel_ce: 4.2418437004089355 (nan) | > loss: 0.017549479380249977 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13465645423486697) | > loader_time: 0.0043 (0.004504997850063462)  --> TIME: 2025-02-23 20:54:43 -- STEP: 19950/36938 -- GLOBAL_STEP: 19950 | > loss_text_ce: 0.16240178048610687 (0.20767106810832028) | > loss_mel_ce: 4.468311309814453 (nan) | > loss: 0.01837584562599659 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.13466999541548047) | > loader_time: 0.0045 (0.004504833639713765)  --> TIME: 2025-02-23 20:54:52 -- STEP: 20000/36938 -- GLOBAL_STEP: 20000 | > loss_text_ce: 0.1792769581079483 (0.2075897645629937) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13466402039527972) | > loader_time: 0.0043 (0.004504428935050972) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_20000.pth  --> TIME: 2025-02-23 20:55:06 -- STEP: 20050/36938 -- GLOBAL_STEP: 20050 | > loss_text_ce: 0.17205491662025452 (0.2075089859955026) | > loss_mel_ce: 3.735809803009033 (nan) | > loss: 0.015507400967180729 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13467003961453794) | > loader_time: 0.0045 (0.0045044141636227785)  --> TIME: 2025-02-23 20:55:15 -- STEP: 20100/36938 -- GLOBAL_STEP: 20100 | > loss_text_ce: 0.15470394492149353 (0.20742389966821734) | > loss_mel_ce: 4.760128498077393 (nan) | > loss: 0.019503304734826088 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.1346605714043583) | > loader_time: 0.0041 (0.0045042023255457385)  --> TIME: 2025-02-23 20:55:24 -- STEP: 20150/36938 -- GLOBAL_STEP: 20150 | > loss_text_ce: 0.1752474009990692 (0.20733811722573753) | > loss_mel_ce: 4.405074119567871 (nan) | > loss: 0.01817587949335575 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.13465913824644726) | > loader_time: 0.0044 (0.00450391253525803)  --> TIME: 2025-02-23 20:55:34 -- STEP: 20200/36938 -- GLOBAL_STEP: 20200 | > loss_text_ce: 0.16046874225139618 (0.20725190683092756) | > loss_mel_ce: 3.632939338684082 (nan) | > loss: 0.015053207986056805 (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.13466125239240156) | > loader_time: 0.0043 (0.004503820449999066)  --> TIME: 2025-02-23 20:55:43 -- STEP: 20250/36938 -- GLOBAL_STEP: 20250 | > loss_text_ce: 0.19242186844348907 (0.2071636918260722) | > loss_mel_ce: 2.223020315170288 (nan) | > loss: 0.009585089050233364 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.1346675691604623) | > loader_time: 0.0045 (0.00450386334642952)  --> TIME: 2025-02-23 20:55:52 -- STEP: 20300/36938 -- GLOBAL_STEP: 20300 | > loss_text_ce: 0.1639266312122345 (0.20707681142123996) | > loss_mel_ce: 4.741329193115234 (nan) | > loss: 0.01946530118584633 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13466504703014442) | > loader_time: 0.0043 (0.0045035625796012715)  --> TIME: 2025-02-23 20:56:02 -- STEP: 20350/36938 -- GLOBAL_STEP: 20350 | > loss_text_ce: 0.176815465092659 (0.2069888957793057) | > loss_mel_ce: 4.683703422546387 (nan) | > loss: 0.019287774339318275 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.13467271235416872) | > loader_time: 0.0043 (0.004503339268358687)  --> TIME: 2025-02-23 20:56:11 -- STEP: 20400/36938 -- GLOBAL_STEP: 20400 | > loss_text_ce: 0.16471467912197113 (0.20690098328844536) | > loss_mel_ce: 5.311197757720947 (nan) | > loss: 0.021729813888669014 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13467213354858762) | > loader_time: 0.0044 (0.004503326451077185)  --> TIME: 2025-02-23 20:56:20 -- STEP: 20450/36938 -- GLOBAL_STEP: 20450 | > loss_text_ce: 0.20100000500679016 (0.20681842884836632) | > loss_mel_ce: 2.2154083251953125 (nan) | > loss: 0.009588922373950481 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13466426747935395) | > loader_time: 0.0045 (0.0045030921245845495)  --> TIME: 2025-02-23 20:56:29 -- STEP: 20500/36938 -- GLOBAL_STEP: 20500 | > loss_text_ce: 0.17293749749660492 (0.20672958026644608) | > loss_mel_ce: 5.963541507720947 (nan) | > loss: 0.024351108819246292 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13466639674582193) | > loader_time: 0.0042 (0.004502978208588395)  --> TIME: 2025-02-23 20:56:38 -- STEP: 20550/36938 -- GLOBAL_STEP: 20550 | > loss_text_ce: 0.1548026204109192 (0.20664365929626147) | > loss_mel_ce: 4.929193496704102 (nan) | > loss: 0.02017458900809288 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13466302338017808) | > loader_time: 0.0044 (0.004502957163065891)  --> TIME: 2025-02-23 20:56:47 -- STEP: 20600/36938 -- GLOBAL_STEP: 20600 | > loss_text_ce: 0.16852563619613647 (0.20655768521202367) | > loss_mel_ce: 5.408557891845703 (nan) | > loss: 0.022131284698843956 (nan) | > current_lr: 5e-06 | > step_time: 0.1774 (0.1346557937779478) | > loader_time: 0.0045 (0.004502729848750587)  --> TIME: 2025-02-23 20:56:56 -- STEP: 20650/36938 -- GLOBAL_STEP: 20650 | > loss_text_ce: 0.1675819605588913 (0.20648209267032008) | > loss_mel_ce: 4.892146587371826 (nan) | > loss: 0.02007829025387764 (nan) | > current_lr: 5e-06 | > step_time: 0.1571 (0.1346454466110864) | > loader_time: 0.0043 (0.004502399770168648)  --> TIME: 2025-02-23 20:57:06 -- STEP: 20700/36938 -- GLOBAL_STEP: 20700 | > loss_text_ce: 0.15939956903457642 (0.20640411397685124) | > loss_mel_ce: 4.700222015380859 (nan) | > loss: 0.019284212961792946 (nan) | > current_lr: 5e-06 | > step_time: 0.184 (0.13464974608398367) | > loader_time: 0.0047 (0.004502244398789704)  --> TIME: 2025-02-23 20:57:15 -- STEP: 20750/36938 -- GLOBAL_STEP: 20750 | > loss_text_ce: 0.16645453870296478 (0.2063236542214825) | > loss_mel_ce: 3.9011423587799072 (nan) | > loss: 0.01614125818014145 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13464540835461056) | > loader_time: 0.0044 (0.004502110251461166)  --> TIME: 2025-02-23 20:57:24 -- STEP: 20800/36938 -- GLOBAL_STEP: 20800 | > loss_text_ce: 0.169631689786911 (0.2062333973475655) | > loss_mel_ce: 3.686962127685547 (nan) | > loss: 0.015303945168852806 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13464803594809377) | > loader_time: 0.0042 (0.004501974490972663)  --> TIME: 2025-02-23 20:57:33 -- STEP: 20850/36938 -- GLOBAL_STEP: 20850 | > loss_text_ce: 0.1683993935585022 (0.20614641770255884) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.1346487083480806) | > loader_time: 0.0045 (0.004501838958520682)  --> TIME: 2025-02-23 20:57:43 -- STEP: 20900/36938 -- GLOBAL_STEP: 20900 | > loss_text_ce: 0.1744726449251175 (0.20606027728989312) | > loss_mel_ce: 3.844661235809326 (nan) | > loss: 0.0159489456564188 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1346517760217483) | > loader_time: 0.0066 (0.004501856970444818)  --> TIME: 2025-02-23 20:57:52 -- STEP: 20950/36938 -- GLOBAL_STEP: 20950 | > loss_text_ce: 0.1628125011920929 (0.20597472870620115) | > loss_mel_ce: 3.8757071495056152 (nan) | > loss: 0.01602587290108204 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13466147741440745) | > loader_time: 0.0047 (0.004501829921203463)  --> TIME: 2025-02-23 20:58:01 -- STEP: 21000/36938 -- GLOBAL_STEP: 21000 | > loss_text_ce: 0.1637852042913437 (0.20588962854657802) | > loss_mel_ce: 4.344202995300293 (nan) | > loss: 0.017888842150568962 (nan) | > current_lr: 5e-06 | > step_time: 0.1697 (0.13465558755965545) | > loader_time: 0.005 (0.004501866522289454)  --> TIME: 2025-02-23 20:58:10 -- STEP: 21050/36938 -- GLOBAL_STEP: 21050 | > loss_text_ce: 0.15780091285705566 (0.20580638659679923) | > loss_mel_ce: 4.300456523895264 (nan) | > loss: 0.01769150048494339 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.1346486142129069) | > loader_time: 0.0045 (0.004501695859460537)  --> TIME: 2025-02-23 20:58:20 -- STEP: 21100/36938 -- GLOBAL_STEP: 21100 | > loss_text_ce: 0.16747768223285675 (0.20571869486174826) | > loss_mel_ce: 3.1130099296569824 (nan) | > loss: 0.013017808087170124 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13465833830042556) | > loader_time: 0.0044 (0.004501792955172571)  --> TIME: 2025-02-23 20:58:29 -- STEP: 21150/36938 -- GLOBAL_STEP: 21150 | > loss_text_ce: 0.17162945866584778 (0.20563804033729713) | > loss_mel_ce: 4.655158996582031 (nan) | > loss: 0.01915392279624939 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13464828512629232) | > loader_time: 0.0046 (0.0045020971456036195)  --> TIME: 2025-02-23 20:58:38 -- STEP: 21200/36938 -- GLOBAL_STEP: 21200 | > loss_text_ce: 0.16104166209697723 (0.20554660391132734) | > loss_mel_ce: 4.987752437591553 (nan) | > loss: 0.020431723445653915 (nan) | > current_lr: 5e-06 | > step_time: 0.1866 (0.1346461389649597) | > loader_time: 0.0046 (0.004502010649105283)  --> TIME: 2025-02-23 20:58:47 -- STEP: 21250/36938 -- GLOBAL_STEP: 21250 | > loss_text_ce: 0.16413764655590057 (0.20546225540076934) | > loss_mel_ce: 5.416530132293701 (nan) | > loss: 0.022145509719848633 (nan) | > current_lr: 5e-06 | > step_time: 0.168 (0.13464451867271957) | > loader_time: 0.0049 (0.004501919151754936)  --> TIME: 2025-02-23 20:58:56 -- STEP: 21300/36938 -- GLOBAL_STEP: 21300 | > loss_text_ce: 0.17049190402030945 (0.20538265244431847) | > loss_mel_ce: 4.5659027099609375 (nan) | > loss: 0.01879521645605564 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13464853451285638) | > loader_time: 0.0044 (0.004501900717686036)  --> TIME: 2025-02-23 20:59:06 -- STEP: 21350/36938 -- GLOBAL_STEP: 21350 | > loss_text_ce: 0.16303065419197083 (0.205299057446821) | > loss_mel_ce: 4.9672393798828125 (nan) | > loss: 0.020358216017484665 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.13464955052949862) | > loader_time: 0.0044 (0.004502220198477174)  --> TIME: 2025-02-23 20:59:15 -- STEP: 21400/36938 -- GLOBAL_STEP: 21400 | > loss_text_ce: 0.17112979292869568 (0.2052149476459088) | > loss_mel_ce: 4.873495578765869 (nan) | > loss: 0.020018355920910835 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.1346619768009015) | > loader_time: 0.0046 (0.0045023085803629044)  --> TIME: 2025-02-23 20:59:24 -- STEP: 21450/36938 -- GLOBAL_STEP: 21450 | > loss_text_ce: 0.15988145768642426 (0.2051314896991198) | > loss_mel_ce: 4.370623588562012 (nan) | > loss: 0.01797819696366787 (nan) | > current_lr: 5e-06 | > step_time: 0.1782 (0.13466264682494203) | > loader_time: 0.0047 (0.004502514392345928)  --> TIME: 2025-02-23 20:59:34 -- STEP: 21500/36938 -- GLOBAL_STEP: 21500 | > loss_text_ce: 0.16288460791110992 (0.20504879602928444) | > loss_mel_ce: 4.675710201263428 (nan) | > loss: 0.019200773909687996 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13466667115411177) | > loader_time: 0.0044 (0.004502678150354429)  --> TIME: 2025-02-23 20:59:43 -- STEP: 21550/36938 -- GLOBAL_STEP: 21550 | > loss_text_ce: 0.16130012273788452 (0.20496766492067472) | > loss_mel_ce: 5.185888767242432 (nan) | > loss: 0.021219005808234215 (nan) | > current_lr: 5e-06 | > step_time: 0.1944 (0.134684645073044) | > loader_time: 0.0045 (0.004502542533343463)  --> TIME: 2025-02-23 20:59:52 -- STEP: 21600/36938 -- GLOBAL_STEP: 21600 | > loss_text_ce: 0.17550402879714966 (0.20488685454552322) | > loss_mel_ce: 4.652495861053467 (nan) | > loss: 0.019158732146024704 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1346817325110796) | > loader_time: 0.0044 (0.0045023315151532465)  --> TIME: 2025-02-23 21:00:02 -- STEP: 21650/36938 -- GLOBAL_STEP: 21650 | > loss_text_ce: 0.16376614570617676 (0.20479692732076396) | > loss_mel_ce: 4.502573013305664 (nan) | > loss: 0.01851721853017807 (nan) | > current_lr: 5e-06 | > step_time: 0.152 (0.13468772153656208) | > loader_time: 0.0047 (0.004502156513117201)  --> TIME: 2025-02-23 21:00:11 -- STEP: 21700/36938 -- GLOBAL_STEP: 21700 | > loss_text_ce: 0.16296181082725525 (0.20471482848997438) | > loss_mel_ce: 4.720058917999268 (nan) | > loss: 0.01937706768512726 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13470136762214757) | > loader_time: 0.0046 (0.004502177139581072)  --> TIME: 2025-02-23 21:00:21 -- STEP: 21750/36938 -- GLOBAL_STEP: 21750 | > loss_text_ce: 0.21585936844348907 (0.20463761377403056) | > loss_mel_ce: 2.2728140354156494 (nan) | > loss: 0.009875688701868057 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13469540167403005) | > loader_time: 0.0043 (0.004502246944383643)  --> TIME: 2025-02-23 21:00:30 -- STEP: 21800/36938 -- GLOBAL_STEP: 21800 | > loss_text_ce: 0.15898776054382324 (0.2045515646081447) | > loss_mel_ce: 4.138760566711426 (nan) | > loss: 0.017054559662938118 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13468913907304875) | > loader_time: 0.005 (0.004502531552533493)  --> TIME: 2025-02-23 21:00:39 -- STEP: 21850/36938 -- GLOBAL_STEP: 21850 | > loss_text_ce: 0.17826509475708008 (0.20446147135316048) | > loss_mel_ce: 4.584194660186768 (nan) | > loss: 0.018898651003837585 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.134699797619399) | > loader_time: 0.0049 (0.004502477656785633)  --> TIME: 2025-02-23 21:00:49 -- STEP: 21900/36938 -- GLOBAL_STEP: 21900 | > loss_text_ce: 0.16116176545619965 (0.20437601902645422) | > loss_mel_ce: 4.688869953155518 (nan) | > loss: 0.019246159121394157 (nan) | > current_lr: 5e-06 | > step_time: 0.1794 (0.13471047358970165) | > loader_time: 0.0046 (0.004502596768070038)  --> TIME: 2025-02-23 21:00:58 -- STEP: 21950/36938 -- GLOBAL_STEP: 21950 | > loss_text_ce: 0.17092856764793396 (0.2042885319247056) | > loss_mel_ce: 4.768828868865967 (nan) | > loss: 0.0196022130548954 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13470875803050306) | > loader_time: 0.0043 (0.004502358871060254)  --> TIME: 2025-02-23 21:01:07 -- STEP: 22000/36938 -- GLOBAL_STEP: 22000 | > loss_text_ce: 0.16088750958442688 (0.20419487224790134) | > loss_mel_ce: 5.276988506317139 (nan) | > loss: 0.021578874439001083 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13472371625900317) | > loader_time: 0.0042 (0.0045022113865072185)  --> TIME: 2025-02-23 21:01:17 -- STEP: 22050/36938 -- GLOBAL_STEP: 22050 | > loss_text_ce: 0.16410155594348907 (0.20411606172720542) | > loss_mel_ce: 4.721901893615723 (nan) | > loss: 0.01938890479505062 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13472082484065862) | > loader_time: 0.0048 (0.004502240362621501)  --> TIME: 2025-02-23 21:01:26 -- STEP: 22100/36938 -- GLOBAL_STEP: 22100 | > loss_text_ce: 0.1589677929878235 (0.20403719652562072) | > loss_mel_ce: 4.089351177215576 (nan) | > loss: 0.016858410090208054 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13471937584121885) | > loader_time: 0.0043 (0.004502315758580008)  --> TIME: 2025-02-23 21:01:35 -- STEP: 22150/36938 -- GLOBAL_STEP: 22150 | > loss_text_ce: 0.16780681908130646 (0.2039573205741744) | > loss_mel_ce: 3.8486735820770264 (nan) | > loss: 0.015938416123390198 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13473480529225468) | > loader_time: 0.0044 (0.004502393967949248)  --> TIME: 2025-02-23 21:01:44 -- STEP: 22200/36938 -- GLOBAL_STEP: 22200 | > loss_text_ce: 0.21791015565395355 (0.20388292493315383) | > loss_mel_ce: 4.092580318450928 (nan) | > loss: 0.01710512302815914 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13472913585267665) | > loader_time: 0.0043 (0.004502581465351694)  --> TIME: 2025-02-23 21:01:54 -- STEP: 22250/36938 -- GLOBAL_STEP: 22250 | > loss_text_ce: 0.16151227056980133 (0.20379543936252545) | > loss_mel_ce: 4.467600345611572 (nan) | > loss: 0.01836949586868286 (nan) | > current_lr: 5e-06 | > step_time: 0.1519 (0.13473551873410733) | > loader_time: 0.0046 (0.004502614149886577)  --> TIME: 2025-02-23 21:02:03 -- STEP: 22300/36938 -- GLOBAL_STEP: 22300 | > loss_text_ce: 0.15873990952968597 (0.20372610813432696) | > loss_mel_ce: 3.981269121170044 (nan) | > loss: 0.01642860844731331 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13471912258943688) | > loader_time: 0.0043 (0.0045025188933573565)  --> TIME: 2025-02-23 21:02:12 -- STEP: 22350/36938 -- GLOBAL_STEP: 22350 | > loss_text_ce: 0.16673913598060608 (0.20364774554367784) | > loss_mel_ce: 4.991484642028809 (nan) | > loss: 0.020469142124056816 (nan) | > current_lr: 5e-06 | > step_time: 0.1943 (0.13471053606711814) | > loader_time: 0.0046 (0.004502506053421064)  --> TIME: 2025-02-23 21:02:21 -- STEP: 22400/36938 -- GLOBAL_STEP: 22400 | > loss_text_ce: 0.16696427762508392 (0.20356183872957265) | > loss_mel_ce: 5.131928443908691 (nan) | > loss: 0.021027352660894394 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.1347065576059482) | > loader_time: 0.0047 (0.00450244609798705)  --> TIME: 2025-02-23 21:02:30 -- STEP: 22450/36938 -- GLOBAL_STEP: 22450 | > loss_text_ce: 0.16538307070732117 (0.20347856067867698) | > loss_mel_ce: 3.8133933544158936 (nan) | > loss: 0.015788797289133072 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.1347019926423756) | > loader_time: 0.0043 (0.004502250824314458)  --> TIME: 2025-02-23 21:02:39 -- STEP: 22500/36938 -- GLOBAL_STEP: 22500 | > loss_text_ce: 0.1588306427001953 (0.2033992492185694) | > loss_mel_ce: 5.075239181518555 (nan) | > loss: 0.02077011950314045 (nan) | > current_lr: 5e-06 | > step_time: 0.152 (0.1346975666681931) | > loader_time: 0.0046 (0.004502037122514516)  --> TIME: 2025-02-23 21:02:48 -- STEP: 22550/36938 -- GLOBAL_STEP: 22550 | > loss_text_ce: 0.15202879905700684 (0.20331985006839848) | > loss_mel_ce: 5.334425449371338 (nan) | > loss: 0.021771643310785294 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.1346843357783998) | > loader_time: 0.0044 (0.004501823797458554)  --> TIME: 2025-02-23 21:02:57 -- STEP: 22600/36938 -- GLOBAL_STEP: 22600 | > loss_text_ce: 0.16029374301433563 (0.20323681516391465) | > loss_mel_ce: 4.146097183227539 (nan) | > loss: 0.01708885282278061 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13470189610413813) | > loader_time: 0.0044 (0.004501812732325189)  --> TIME: 2025-02-23 21:03:07 -- STEP: 22650/36938 -- GLOBAL_STEP: 22650 | > loss_text_ce: 0.167418971657753 (0.20315486709408365) | > loss_mel_ce: 1.5287017822265625 (nan) | > loss: 0.0067306384444236755 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13470300973382815) | > loader_time: 0.0045 (0.004501639001943155)  --> TIME: 2025-02-23 21:03:16 -- STEP: 22700/36938 -- GLOBAL_STEP: 22700 | > loss_text_ce: 0.15753810107707977 (0.20307015282073163) | > loss_mel_ce: 4.112156867980957 (nan) | > loss: 0.01694323495030403 (nan) | > current_lr: 5e-06 | > step_time: 0.181 (0.1347019238619033) | > loader_time: 0.0046 (0.004501413175187969)  --> TIME: 2025-02-23 21:03:25 -- STEP: 22750/36938 -- GLOBAL_STEP: 22750 | > loss_text_ce: 0.18618303537368774 (0.2029825779939741) | > loss_mel_ce: 0.137786865234375 (nan) | > loss: 0.0012855948880314827 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.1346955831024679) | > loader_time: 0.0042 (0.004501304259667027)  --> TIME: 2025-02-23 21:03:34 -- STEP: 22800/36938 -- GLOBAL_STEP: 22800 | > loss_text_ce: 0.1571197807788849 (0.202896292984093) | > loss_mel_ce: 4.380531311035156 (nan) | > loss: 0.018006552010774612 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.1346922080767789) | > loader_time: 0.0046 (0.004501069334515348)  --> TIME: 2025-02-23 21:03:43 -- STEP: 22850/36938 -- GLOBAL_STEP: 22850 | > loss_text_ce: 0.16585737466812134 (0.20281154688260178) | > loss_mel_ce: 5.210360050201416 (nan) | > loss: 0.021334197372198105 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.1346904868161399) | > loader_time: 0.0044 (0.004500583736327851)  --> TIME: 2025-02-23 21:03:53 -- STEP: 22900/36938 -- GLOBAL_STEP: 22900 | > loss_text_ce: 0.15674932301044464 (0.20271945921809864) | > loss_mel_ce: 3.6696059703826904 (nan) | > loss: 0.01518394984304905 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13470680985388367) | > loader_time: 0.0045 (0.004500291472439155)  --> TIME: 2025-02-23 21:04:02 -- STEP: 22950/36938 -- GLOBAL_STEP: 22950 | > loss_text_ce: 0.19365385174751282 (0.20263700202789134) | > loss_mel_ce: 4.803248405456543 (nan) | > loss: 0.019828978925943375 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.1347112034400833) | > loader_time: 0.0043 (0.004500249974867871)  --> TIME: 2025-02-23 21:04:11 -- STEP: 23000/36938 -- GLOBAL_STEP: 23000 | > loss_text_ce: 0.15430058538913727 (0.2025445978486015) | > loss_mel_ce: 4.64380407333374 (nan) | > loss: 0.019040100276470184 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13471193729276326) | > loader_time: 0.0044 (0.004500084555667378)  --> TIME: 2025-02-23 21:04:21 -- STEP: 23050/36938 -- GLOBAL_STEP: 23050 | > loss_text_ce: 0.14409419894218445 (0.20245814509802892) | > loss_mel_ce: 4.232010364532471 (nan) | > loss: 0.01736549474298954 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.1347214031736668) | > loader_time: 0.0046 (0.004500072907470567)  --> TIME: 2025-02-23 21:04:30 -- STEP: 23100/36938 -- GLOBAL_STEP: 23100 | > loss_text_ce: 0.16136717796325684 (0.20237944520977622) | > loss_mel_ce: 4.579137802124023 (nan) | > loss: 0.018811529502272606 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.13471063827539498) | > loader_time: 0.0046 (0.004499937600387639)  --> TIME: 2025-02-23 21:04:39 -- STEP: 23150/36938 -- GLOBAL_STEP: 23150 | > loss_text_ce: 0.1564978063106537 (0.2022925506985981) | > loss_mel_ce: 4.225643634796143 (nan) | > loss: 0.01738945208489895 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.1347112327312041) | > loader_time: 0.0048 (0.004500014086778941)  --> TIME: 2025-02-23 21:04:48 -- STEP: 23200/36938 -- GLOBAL_STEP: 23200 | > loss_text_ce: 0.15760467946529388 (0.2022153841742663) | > loss_mel_ce: 5.181539058685303 (nan) | > loss: 0.021187080070376396 (nan) | > current_lr: 5e-06 | > step_time: 0.1875 (0.13470378644507588) | > loader_time: 0.0046 (0.004499845504760738)  --> TIME: 2025-02-23 21:04:57 -- STEP: 23250/36938 -- GLOBAL_STEP: 23250 | > loss_text_ce: 0.1580716371536255 (0.2021297728598754) | > loss_mel_ce: 4.901390552520752 (nan) | > loss: 0.020077232271432877 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13470126474031924) | > loader_time: 0.0056 (0.004499626046867776)  --> TIME: 2025-02-23 21:05:07 -- STEP: 23300/36938 -- GLOBAL_STEP: 23300 | > loss_text_ce: 0.14695200324058533 (0.20203754620490622) | > loss_mel_ce: 3.8049845695495605 (nan) | > loss: 0.015682289376854897 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13472107812570489) | > loader_time: 0.0043 (0.004499528970841172)  --> TIME: 2025-02-23 21:05:16 -- STEP: 23350/36938 -- GLOBAL_STEP: 23350 | > loss_text_ce: 0.14496377110481262 (0.20195131263781363) | > loss_mel_ce: 4.708878517150879 (nan) | > loss: 0.019261280074715614 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.1347174181519544) | > loader_time: 0.0042 (0.004499334604888196)  --> TIME: 2025-02-23 21:05:25 -- STEP: 23400/36938 -- GLOBAL_STEP: 23400 | > loss_text_ce: 0.1613100916147232 (0.201873625876684) | > loss_mel_ce: 4.321810722351074 (nan) | > loss: 0.017790162935853004 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13471821445685248) | > loader_time: 0.0047 (0.004499447192901216)  --> TIME: 2025-02-23 21:05:34 -- STEP: 23450/36938 -- GLOBAL_STEP: 23450 | > loss_text_ce: 0.13565103709697723 (0.20178882204774554) | > loss_mel_ce: 5.0 (nan) | > loss: 0.02037956938147545 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13471563968577202) | > loader_time: 0.0043 (0.004499437976747681)  --> TIME: 2025-02-23 21:05:44 -- STEP: 23500/36938 -- GLOBAL_STEP: 23500 | > loss_text_ce: 0.1636546552181244 (0.20170300542230254) | > loss_mel_ce: 4.545502662658691 (nan) | > loss: 0.018687134608626366 (nan) | > current_lr: 5e-06 | > step_time: 0.1499 (0.13472409573007235) | > loader_time: 0.0047 (0.004499511779622824)  --> TIME: 2025-02-23 21:05:53 -- STEP: 23550/36938 -- GLOBAL_STEP: 23550 | > loss_text_ce: 0.16783782839775085 (0.20161583653870616) | > loss_mel_ce: 4.934903144836426 (nan) | > loss: 0.02024897187948227 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13473322230539606) | > loader_time: 0.0046 (0.004499670482238639)  --> TIME: 2025-02-23 21:06:02 -- STEP: 23600/36938 -- GLOBAL_STEP: 23600 | > loss_text_ce: 0.15307745337486267 (0.20153549164912413) | > loss_mel_ce: 4.627523422241211 (nan) | > loss: 0.018970640376210213 (nan) | > current_lr: 5e-06 | > step_time: 0.1895 (0.13472316212573404) | > loader_time: 0.0047 (0.004499971593840648)  --> TIME: 2025-02-23 21:06:12 -- STEP: 23650/36938 -- GLOBAL_STEP: 23650 | > loss_text_ce: 0.1509791612625122 (0.20144657104010263) | > loss_mel_ce: 5.167616844177246 (nan) | > loss: 0.021105540916323662 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13472361449711523) | > loader_time: 0.0045 (0.0045001106988047926)  --> TIME: 2025-02-23 21:06:21 -- STEP: 23700/36938 -- GLOBAL_STEP: 23700 | > loss_text_ce: 0.16508063673973083 (0.20136222660792985) | > loss_mel_ce: 4.844576835632324 (nan) | > loss: 0.019879594445228577 (nan) | > current_lr: 5e-06 | > step_time: 0.1482 (0.13472407450655938) | > loader_time: 0.0045 (0.004500305954414074)  --> TIME: 2025-02-23 21:06:30 -- STEP: 23750/36938 -- GLOBAL_STEP: 23750 | > loss_text_ce: 0.1575901359319687 (0.20128321460987347) | > loss_mel_ce: 4.861711502075195 (nan) | > loss: 0.019917864352464676 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13472062332755735) | > loader_time: 0.0044 (0.004500088069313449)  --> TIME: 2025-02-23 21:06:39 -- STEP: 23800/36938 -- GLOBAL_STEP: 23800 | > loss_text_ce: 0.16414955258369446 (0.20120176009270277) | > loss_mel_ce: 4.579808712005615 (nan) | > loss: 0.018825232982635498 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.13472376348591747) | > loader_time: 0.0043 (0.004499950148478271)  --> TIME: 2025-02-23 21:06:49 -- STEP: 23850/36938 -- GLOBAL_STEP: 23850 | > loss_text_ce: 0.1567378044128418 (0.20111479681649003) | > loss_mel_ce: 3.896017551422119 (nan) | > loss: 0.016082363203167915 (nan) | > current_lr: 5e-06 | > step_time: 0.1885 (0.13472109999546805) | > loader_time: 0.0049 (0.004500164355871806)  --> TIME: 2025-02-23 21:06:58 -- STEP: 23900/36938 -- GLOBAL_STEP: 23900 | > loss_text_ce: 0.15299062430858612 (0.2010292481113548) | > loss_mel_ce: 3.643126964569092 (nan) | > loss: 0.015063959173858166 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13473211844097197) | > loader_time: 0.0044 (0.004499998960534893)  --> TIME: 2025-02-23 21:07:07 -- STEP: 23950/36938 -- GLOBAL_STEP: 23950 | > loss_text_ce: 0.16435213387012482 (0.2009433461339957) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13473739202933505) | > loader_time: 0.0045 (0.00449984083593762)  --> TIME: 2025-02-23 21:07:17 -- STEP: 24000/36938 -- GLOBAL_STEP: 24000 | > loss_text_ce: 0.15500743687152863 (0.2008619401029617) | > loss_mel_ce: 3.952197551727295 (nan) | > loss: 0.016298433765769005 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13473286548256963) | > loader_time: 0.0045 (0.004499960849682478)  --> TIME: 2025-02-23 21:07:26 -- STEP: 24050/36938 -- GLOBAL_STEP: 24050 | > loss_text_ce: 0.15351562201976776 (0.20077695289390454) | > loss_mel_ce: 5.045814037322998 (nan) | > loss: 0.020632263273000717 (nan) | > current_lr: 5e-06 | > step_time: 0.1821 (0.13473823278708771) | > loader_time: 0.0054 (0.0044998348775375785)  --> TIME: 2025-02-23 21:07:35 -- STEP: 24100/36938 -- GLOBAL_STEP: 24100 | > loss_text_ce: 0.17911304533481598 (0.2006915438475075) | > loss_mel_ce: 4.987840175628662 (nan) | > loss: 0.020503783598542213 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1347423655264615) | > loader_time: 0.0048 (0.004499816666994837)  --> TIME: 2025-02-23 21:07:44 -- STEP: 24150/36938 -- GLOBAL_STEP: 24150 | > loss_text_ce: 0.16592708230018616 (0.20061858515345735) | > loss_mel_ce: 0.2060546875 (nan) | > loss: 0.0014761182246729732 (nan) | > current_lr: 5e-06 | > step_time: 0.1028 (0.13473421111610467) | > loader_time: 0.0042 (0.004499690152596726)  --> TIME: 2025-02-23 21:07:53 -- STEP: 24200/36938 -- GLOBAL_STEP: 24200 | > loss_text_ce: 0.16594532132148743 (0.20054202082636144) | > loss_mel_ce: 2.994027853012085 (nan) | > loss: 0.0125395767390728 (nan) | > current_lr: 5e-06 | > step_time: 0.1453 (0.1347321484305652) | > loader_time: 0.0045 (0.004499531540988876)  --> TIME: 2025-02-23 21:08:02 -- STEP: 24250/36938 -- GLOBAL_STEP: 24250 | > loss_text_ce: 0.1651953160762787 (0.20046555470283453) | > loss_mel_ce: 0.6752522587776184 (nan) | > loss: 0.003335109446197748 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13472108908781166) | > loader_time: 0.0044 (0.004499474171510664)  --> TIME: 2025-02-23 21:08:12 -- STEP: 24300/36938 -- GLOBAL_STEP: 24300 | > loss_text_ce: 0.1547321379184723 (0.20038495649188834) | > loss_mel_ce: 4.563117504119873 (nan) | > loss: 0.018721627071499825 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1347249439125699) | > loader_time: 0.0044 (0.004499280403670937)  --> TIME: 2025-02-23 21:08:21 -- STEP: 24350/36938 -- GLOBAL_STEP: 24350 | > loss_text_ce: 0.15074999630451202 (0.2002955611741636) | > loss_mel_ce: 3.6076297760009766 (nan) | > loss: 0.014914206229150295 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13473648096746582) | > loader_time: 0.0043 (0.0044990442812565005)  --> TIME: 2025-02-23 21:08:30 -- STEP: 24400/36938 -- GLOBAL_STEP: 24400 | > loss_text_ce: 0.15909573435783386 (0.20021114777010196) | > loss_mel_ce: 5.021325588226318 (nan) | > loss: 0.020557228475809097 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1347308031457379) | > loader_time: 0.0043 (0.004498602552492094)  --> TIME: 2025-02-23 21:08:39 -- STEP: 24450/36938 -- GLOBAL_STEP: 24450 | > loss_text_ce: 0.1518883854150772 (0.2001364300974425) | > loss_mel_ce: 4.3630805015563965 (nan) | > loss: 0.017916543409228325 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13471412457815318) | > loader_time: 0.0045 (0.00449852091159312)  --> TIME: 2025-02-23 21:08:48 -- STEP: 24500/36938 -- GLOBAL_STEP: 24500 | > loss_text_ce: 0.15978389978408813 (0.2000563694685693) | > loss_mel_ce: 3.0635743141174316 (nan) | > loss: 0.01279110461473465 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1347006664665388) | > loader_time: 0.0044 (0.004498343302279089)  --> TIME: 2025-02-23 21:08:57 -- STEP: 24550/36938 -- GLOBAL_STEP: 24550 | > loss_text_ce: 0.1593330204486847 (0.19997240293991012) | > loss_mel_ce: 4.6931023597717285 (nan) | > loss: 0.01925569772720337 (nan) | > current_lr: 5e-06 | > step_time: 0.1625 (0.13470527322617756) | > loader_time: 0.0044 (0.004498183712697064)  --> TIME: 2025-02-23 21:09:06 -- STEP: 24600/36938 -- GLOBAL_STEP: 24600 | > loss_text_ce: 0.1658593714237213 (0.19989462038941214) | > loss_mel_ce: 3.268197774887085 (nan) | > loss: 0.01362721249461174 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.1346953343763592) | > loader_time: 0.0044 (0.004497979414172276)  --> TIME: 2025-02-23 21:09:15 -- STEP: 24650/36938 -- GLOBAL_STEP: 24650 | > loss_text_ce: 0.1606670767068863 (0.19980800330487714) | > loss_mel_ce: 3.92405104637146 (nan) | > loss: 0.016209200024604797 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13469999077354028) | > loader_time: 0.0044 (0.004498034316676131)  --> TIME: 2025-02-23 21:09:25 -- STEP: 24700/36938 -- GLOBAL_STEP: 24700 | > loss_text_ce: 0.16903646290302277 (0.19972796335182888) | > loss_mel_ce: 2.26055908203125 (nan) | > loss: 0.00964125245809555 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.1346987109049137) | > loader_time: 0.0044 (0.004498149103481268)  --> TIME: 2025-02-23 21:09:34 -- STEP: 24750/36938 -- GLOBAL_STEP: 24750 | > loss_text_ce: 0.16374999284744263 (0.19964468449865885) | > loss_mel_ce: 4.0692267417907715 (nan) | > loss: 0.01679752767086029 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13470285910789498) | > loader_time: 0.0045 (0.0044983448934073295)  --> TIME: 2025-02-23 21:09:43 -- STEP: 24800/36938 -- GLOBAL_STEP: 24800 | > loss_text_ce: 0.14749999344348907 (0.19956064473687357) | > loss_mel_ce: 5.796875 (nan) | > loss: 0.023588791489601135 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13469817369215034) | > loader_time: 0.0043 (0.004498356292324669)  --> TIME: 2025-02-23 21:09:53 -- STEP: 24850/36938 -- GLOBAL_STEP: 24850 | > loss_text_ce: 0.15703749656677246 (0.19947173752385322) | > loss_mel_ce: 4.317816257476807 (nan) | > loss: 0.0177573561668396 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13471337545805626) | > loader_time: 0.0044 (0.004498117972667537)  --> TIME: 2025-02-23 21:10:02 -- STEP: 24900/36938 -- GLOBAL_STEP: 24900 | > loss_text_ce: 0.16927801072597504 (0.19938720192446005) | > loss_mel_ce: 4.318990230560303 (nan) | > loss: 0.01781059056520462 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13471879644087617) | > loader_time: 0.0041 (0.004497804201271629)  --> TIME: 2025-02-23 21:10:11 -- STEP: 24950/36938 -- GLOBAL_STEP: 24950 | > loss_text_ce: 0.1433497816324234 (0.19930501256145508) | > loss_mel_ce: 5.006875514984131 (nan) | > loss: 0.02043740265071392 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13471115767836428) | > loader_time: 0.0044 (0.004497420802145051)  --> TIME: 2025-02-23 21:10:20 -- STEP: 25000/36938 -- GLOBAL_STEP: 25000 | > loss_text_ce: 0.15435640513896942 (0.1992188292202349) | > loss_mel_ce: 4.5696024894714355 (nan) | > loss: 0.018745869398117065 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.13471567789077862) | > loader_time: 0.0044 (0.004497089738845812)  --> TIME: 2025-02-23 21:10:30 -- STEP: 25050/36938 -- GLOBAL_STEP: 25050 | > loss_text_ce: 0.15693359076976776 (0.19913783466595808) | > loss_mel_ce: 3.4871809482574463 (nan) | > loss: 0.014460772275924683 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13471774043199422) | > loader_time: 0.0043 (0.004496845884951271)  --> TIME: 2025-02-23 21:10:39 -- STEP: 25100/36938 -- GLOBAL_STEP: 25100 | > loss_text_ce: 0.24992187321186066 (0.19905969167224632) | > loss_mel_ce: 2.7305908203125 (nan) | > loss: 0.011827431619167328 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13471882703294902) | > loader_time: 0.0042 (0.0044964993997398915)  --> TIME: 2025-02-23 21:10:48 -- STEP: 25150/36938 -- GLOBAL_STEP: 25150 | > loss_text_ce: 0.15352964401245117 (0.19897246147096706) | > loss_mel_ce: 4.635422229766846 (nan) | > loss: 0.01900377869606018 (nan) | > current_lr: 5e-06 | > step_time: 0.1813 (0.13473107852708396) | > loader_time: 0.0047 (0.004496216006117818)  --> TIME: 2025-02-23 21:10:57 -- STEP: 25200/36938 -- GLOBAL_STEP: 25200 | > loss_text_ce: 0.15539062023162842 (0.19889139558912936) | > loss_mel_ce: 2.346712589263916 (nan) | > loss: 0.009928981773555279 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13472330465203367) | > loader_time: 0.0048 (0.004496004127320775)  --> TIME: 2025-02-23 21:11:07 -- STEP: 25250/36938 -- GLOBAL_STEP: 25250 | > loss_text_ce: 0.16106250882148743 (0.19880224205332184) | > loss_mel_ce: 0.03369140625 (nan) | > loss: 0.000772833067458123 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13473074998005916) | > loader_time: 0.0043 (0.004495847239352678)  --> TIME: 2025-02-23 21:11:16 -- STEP: 25300/36938 -- GLOBAL_STEP: 25300 | > loss_text_ce: 0.16869664192199707 (0.19871661302924348) | > loss_mel_ce: 4.351195335388184 (nan) | > loss: 0.01793607883155346 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13472908680618179) | > loader_time: 0.0045 (0.0044957510755938394)  --> TIME: 2025-02-23 21:11:25 -- STEP: 25350/36938 -- GLOBAL_STEP: 25350 | > loss_text_ce: 0.16102343797683716 (0.19863161853049827) | > loss_mel_ce: 4.34574556350708 (nan) | > loss: 0.01788400486111641 (nan) | > current_lr: 5e-06 | > step_time: 0.1268 (0.1347271217349968) | > loader_time: 0.0042 (0.004495658065675504)  --> TIME: 2025-02-23 21:11:34 -- STEP: 25400/36938 -- GLOBAL_STEP: 25400 | > loss_text_ce: 0.15571996569633484 (0.1985494188057973) | > loss_mel_ce: 3.9188833236694336 (nan) | > loss: 0.01616906002163887 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13472634965040683) | > loader_time: 0.0043 (0.004495433343677057)  --> TIME: 2025-02-23 21:11:44 -- STEP: 25450/36938 -- GLOBAL_STEP: 25450 | > loss_text_ce: 0.1548372358083725 (0.1984660248909222) | > loss_mel_ce: 4.195829391479492 (nan) | > loss: 0.017264550551772118 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13472273964310585) | > loader_time: 0.0045 (0.004495543112689233)  --> TIME: 2025-02-23 21:11:53 -- STEP: 25500/36938 -- GLOBAL_STEP: 25500 | > loss_text_ce: 0.1442936509847641 (0.1983790033000937) | > loss_mel_ce: 4.967759609222412 (nan) | > loss: 0.02028592675924301 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.1347311693640331) | > loader_time: 0.0068 (0.004495678695977892)  --> TIME: 2025-02-23 21:12:02 -- STEP: 25550/36938 -- GLOBAL_STEP: 25550 | > loss_text_ce: 0.16370831429958344 (0.1982960741087761) | > loss_mel_ce: 3.3880112171173096 (nan) | > loss: 0.014094125479459763 (nan) | > current_lr: 5e-06 | > step_time: 0.1082 (0.13472616731304168) | > loader_time: 0.0043 (0.004495875765432804)  --> TIME: 2025-02-23 21:12:11 -- STEP: 25600/36938 -- GLOBAL_STEP: 25600 | > loss_text_ce: 0.1491406261920929 (0.19821067000593715) | > loss_mel_ce: 4.661255836486816 (nan) | > loss: 0.019088877364993095 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13472323239781123) | > loader_time: 0.0044 (0.0044960190448909846)  --> TIME: 2025-02-23 21:12:20 -- STEP: 25650/36938 -- GLOBAL_STEP: 25650 | > loss_text_ce: 0.16570642590522766 (0.1981225432420679) | > loss_mel_ce: 5.117306232452393 (nan) | > loss: 0.02096433751285076 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.1347209401809233) | > loader_time: 0.0049 (0.004495943759152285)  --> TIME: 2025-02-23 21:12:29 -- STEP: 25700/36938 -- GLOBAL_STEP: 25700 | > loss_text_ce: 0.15732142329216003 (0.1980384801771734) | > loss_mel_ce: 2.498284101486206 (nan) | > loss: 0.010538117960095406 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13471026257318455) | > loader_time: 0.0043 (0.004495865983257946)  --> TIME: 2025-02-23 21:12:39 -- STEP: 25750/36938 -- GLOBAL_STEP: 25750 | > loss_text_ce: 0.15353365242481232 (0.19795117952030797) | > loss_mel_ce: 4.49795389175415 (nan) | > loss: 0.018458284437656403 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13471673969157855) | > loader_time: 0.0043 (0.0044957530753126276)  --> TIME: 2025-02-23 21:12:48 -- STEP: 25800/36938 -- GLOBAL_STEP: 25800 | > loss_text_ce: 0.14210937917232513 (0.1978626731121836) | > loss_mel_ce: 4.540685653686523 (nan) | > loss: 0.018582521006464958 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13472298566685328) | > loader_time: 0.0045 (0.004495658412460195)  --> TIME: 2025-02-23 21:12:57 -- STEP: 25850/36938 -- GLOBAL_STEP: 25850 | > loss_text_ce: 0.16370831429958344 (0.1977801344037859) | > loss_mel_ce: 4.506027698516846 (nan) | > loss: 0.018530698493123055 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13471207116051548) | > loader_time: 0.0046 (0.004495598306047138)  --> TIME: 2025-02-23 21:13:06 -- STEP: 25900/36938 -- GLOBAL_STEP: 25900 | > loss_text_ce: 0.15054486691951752 (0.19769585149887633) | > loss_mel_ce: 4.500751972198486 (nan) | > loss: 0.018457528203725815 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.134710771595651) | > loader_time: 0.0045 (0.004495672903465928)  --> TIME: 2025-02-23 21:13:16 -- STEP: 25950/36938 -- GLOBAL_STEP: 25950 | > loss_text_ce: 0.1586129367351532 (0.19761491169101428) | > loss_mel_ce: 4.455094814300537 (nan) | > loss: 0.0183083638548851 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.13471883178446514) | > loader_time: 0.0045 (0.004495598796704814)  --> TIME: 2025-02-23 21:13:25 -- STEP: 26000/36938 -- GLOBAL_STEP: 26000 | > loss_text_ce: 0.15965494513511658 (0.19753136177103064) | > loss_mel_ce: 3.297499418258667 (nan) | > loss: 0.013718866743147373 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13470987068689952) | > loader_time: 0.0042 (0.0044954421061735705)  --> TIME: 2025-02-23 21:13:34 -- STEP: 26050/36938 -- GLOBAL_STEP: 26050 | > loss_text_ce: 0.14254340529441833 (0.19744259883626367) | > loss_mel_ce: 3.973637580871582 (nan) | > loss: 0.0163340512663126 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13471761028048332) | > loader_time: 0.0045 (0.004495324832235309)  --> TIME: 2025-02-23 21:13:44 -- STEP: 26100/36938 -- GLOBAL_STEP: 26100 | > loss_text_ce: 0.15075097978115082 (0.19735775757823598) | > loss_mel_ce: 4.860324382781982 (nan) | > loss: 0.019885221496224403 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.1347232442797389) | > loader_time: 0.0047 (0.004495284822252037)  --> TIME: 2025-02-23 21:13:53 -- STEP: 26150/36938 -- GLOBAL_STEP: 26150 | > loss_text_ce: 0.15495064854621887 (0.19727023787212375) | > loss_mel_ce: 4.145191669464111 (nan) | > loss: 0.017064057290554047 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1347189611868938) | > loader_time: 0.0043 (0.004495147300267972)  --> TIME: 2025-02-23 21:14:02 -- STEP: 26200/36938 -- GLOBAL_STEP: 26200 | > loss_text_ce: 0.14763423800468445 (0.19718860888555034) | > loss_mel_ce: 4.374161720275879 (nan) | > loss: 0.01794363558292389 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.13471445186447709) | > loader_time: 0.0047 (0.00449503369913754)  --> TIME: 2025-02-23 21:14:11 -- STEP: 26250/36938 -- GLOBAL_STEP: 26250 | > loss_text_ce: 0.15219400823116302 (0.19710698513331826) | > loss_mel_ce: 4.830954551696777 (nan) | > loss: 0.01977439969778061 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13470965552557237) | > loader_time: 0.0043 (0.004494915099371024)  --> TIME: 2025-02-23 21:14:20 -- STEP: 26300/36938 -- GLOBAL_STEP: 26300 | > loss_text_ce: 0.15495486557483673 (0.1970223760432046) | > loss_mel_ce: 4.285420894622803 (nan) | > loss: 0.017620539292693138 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13471269249462975) | > loader_time: 0.0043 (0.0044948316254996325)  --> TIME: 2025-02-23 21:14:29 -- STEP: 26350/36938 -- GLOBAL_STEP: 26350 | > loss_text_ce: 0.1635781228542328 (0.1969413144360566) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13471057080227405) | > loader_time: 0.0043 (0.004494629014827701)  --> TIME: 2025-02-23 21:14:38 -- STEP: 26400/36938 -- GLOBAL_STEP: 26400 | > loss_text_ce: 0.14024925231933594 (0.19686572130797672) | > loss_mel_ce: 4.865896701812744 (nan) | > loss: 0.01986565999686718 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13469577810078126) | > loader_time: 0.0042 (0.00449445384018347)  --> TIME: 2025-02-23 21:14:47 -- STEP: 26450/36938 -- GLOBAL_STEP: 26450 | > loss_text_ce: 0.15590177476406097 (0.19678455100139466) | > loss_mel_ce: 3.8488824367523193 (nan) | > loss: 0.01589200086891651 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.1346821855281853) | > loader_time: 0.0044 (0.004494277723344807)  --> TIME: 2025-02-23 21:14:56 -- STEP: 26500/36938 -- GLOBAL_STEP: 26500 | > loss_text_ce: 0.1386767029762268 (0.19669961223877785) | > loss_mel_ce: 4.626668453216553 (nan) | > loss: 0.018910100683569908 (nan) | > current_lr: 5e-06 | > step_time: 0.1779 (0.1346788202051863) | > loader_time: 0.0046 (0.004494327968021587)  --> TIME: 2025-02-23 21:15:05 -- STEP: 26550/36938 -- GLOBAL_STEP: 26550 | > loss_text_ce: 0.15606026351451874 (0.19661689040996444) | > loss_mel_ce: 4.748345851898193 (nan) | > loss: 0.01946192979812622 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13467534373260207) | > loader_time: 0.0044 (0.004494356794770355)  --> TIME: 2025-02-23 21:15:14 -- STEP: 26600/36938 -- GLOBAL_STEP: 26600 | > loss_text_ce: 0.1493321806192398 (0.19653672851313958) | > loss_mel_ce: 5.126857280731201 (nan) | > loss: 0.02093726024031639 (nan) | > current_lr: 5e-06 | > step_time: 0.1749 (0.13467707637557574) | > loader_time: 0.0046 (0.004494096166209147)  --> TIME: 2025-02-23 21:15:23 -- STEP: 26650/36938 -- GLOBAL_STEP: 26650 | > loss_text_ce: 0.14953584969043732 (0.19645290800095905) | > loss_mel_ce: 3.5462191104888916 (nan) | > loss: 0.014665694907307625 (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.13467004630176646) | > loader_time: 0.0047 (0.00449423158370082)  --> TIME: 2025-02-23 21:15:32 -- STEP: 26700/36938 -- GLOBAL_STEP: 26700 | > loss_text_ce: 0.18747159838676453 (0.19636907061443307) | > loss_mel_ce: 3.484161376953125 (nan) | > loss: 0.014569972641766071 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13466982825418708) | > loader_time: 0.0042 (0.004494292423519731)  --> TIME: 2025-02-23 21:15:42 -- STEP: 26750/36938 -- GLOBAL_STEP: 26750 | > loss_text_ce: 0.15760156512260437 (0.19628703927380975) | > loss_mel_ce: 4.991666793823242 (nan) | > loss: 0.02043360471725464 (nan) | > current_lr: 5e-06 | > step_time: 0.1062 (0.13466918647623685) | > loader_time: 0.0043 (0.004494222034917794)  --> TIME: 2025-02-23 21:15:51 -- STEP: 26800/36938 -- GLOBAL_STEP: 26800 | > loss_text_ce: 0.15171875059604645 (0.19620062968068117) | > loss_mel_ce: 2.908046245574951 (nan) | > loss: 0.012141925282776356 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13467607783737637) | > loader_time: 0.0042 (0.004494147754427195)  --> TIME: 2025-02-23 21:16:00 -- STEP: 26850/36938 -- GLOBAL_STEP: 26850 | > loss_text_ce: 0.14116814732551575 (0.196116097828085) | > loss_mel_ce: 1.1502413749694824 (nan) | > loss: 0.005124641116708517 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.13467886865250167) | > loader_time: 0.0043 (0.004493918685273732)  --> TIME: 2025-02-23 21:16:10 -- STEP: 26900/36938 -- GLOBAL_STEP: 26900 | > loss_text_ce: 0.1599999964237213 (0.1960318097793476) | > loss_mel_ce: 5.491898059844971 (nan) | > loss: 0.02242816798388958 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13468087168874515) | > loader_time: 0.0043 (0.004493689173659408)  --> TIME: 2025-02-23 21:16:19 -- STEP: 26950/36938 -- GLOBAL_STEP: 26950 | > loss_text_ce: 0.1527063548564911 (0.19595102140987475) | > loss_mel_ce: 5.150626182556152 (nan) | > loss: 0.02104497142136097 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.1346777249178787) | > loader_time: 0.0045 (0.004493456461876334)  --> TIME: 2025-02-23 21:16:28 -- STEP: 27000/36938 -- GLOBAL_STEP: 27000 | > loss_text_ce: 0.1810416579246521 (0.1958704435499726) | > loss_mel_ce: 4.206811904907227 (nan) | > loss: 0.01741211861371994 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13466615936491377) | > loader_time: 0.0043 (0.00449323694794264)  --> TIME: 2025-02-23 21:16:37 -- STEP: 27050/36938 -- GLOBAL_STEP: 27050 | > loss_text_ce: 0.13595229387283325 (0.19578098935582902) | > loss_mel_ce: 4.534780025482178 (nan) | > loss: 0.01853465475142002 (nan) | > current_lr: 5e-06 | > step_time: 0.1864 (0.13467388734800037) | > loader_time: 0.0048 (0.004493095385609596)  --> TIME: 2025-02-23 21:16:46 -- STEP: 27100/36938 -- GLOBAL_STEP: 27100 | > loss_text_ce: 0.1496538370847702 (0.1956972747672744) | > loss_mel_ce: 4.115509986877441 (nan) | > loss: 0.016925254836678505 (nan) | > current_lr: 5e-06 | > step_time: 0.1579 (0.13466575979746734) | > loader_time: 0.0044 (0.004492818103945095)  --> TIME: 2025-02-23 21:16:55 -- STEP: 27150/36938 -- GLOBAL_STEP: 27150 | > loss_text_ce: 0.1469399631023407 (0.1956168875526324) | > loss_mel_ce: 4.725736618041992 (nan) | > loss: 0.019336018711328506 (nan) | > current_lr: 5e-06 | > step_time: 0.1962 (0.13466967806631866) | > loader_time: 0.0046 (0.004492638914624601)  --> TIME: 2025-02-23 21:17:04 -- STEP: 27200/36938 -- GLOBAL_STEP: 27200 | > loss_text_ce: 0.13877056539058685 (0.19553542124819628) | > loss_mel_ce: 3.9063658714294434 (nan) | > loss: 0.016052130609750748 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.13465996919309572) | > loader_time: 0.0044 (0.004492481543737269)  --> TIME: 2025-02-23 21:17:13 -- STEP: 27250/36938 -- GLOBAL_STEP: 27250 | > loss_text_ce: 0.148214653134346 (0.19545391370943888) | > loss_mel_ce: 5.087409973144531 (nan) | > loss: 0.02077629044651985 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13465757563354952) | > loader_time: 0.0044 (0.004492314338684054)  --> TIME: 2025-02-23 21:17:22 -- STEP: 27300/36938 -- GLOBAL_STEP: 27300 | > loss_text_ce: 0.1508958339691162 (0.19537799807640618) | > loss_mel_ce: 3.906381845474243 (nan) | > loss: 0.016100309789180756 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.1346513336569418) | > loader_time: 0.0044 (0.004492055409120524)  --> TIME: 2025-02-23 21:17:32 -- STEP: 27350/36938 -- GLOBAL_STEP: 27350 | > loss_text_ce: 0.14897915720939636 (0.19529026748384334) | > loss_mel_ce: 5.0059404373168945 (nan) | > loss: 0.02045603096485138 (nan) | > current_lr: 5e-06 | > step_time: 0.1735 (0.13466412882482534) | > loader_time: 0.0045 (0.00449210062323363)  --> TIME: 2025-02-23 21:17:41 -- STEP: 27400/36938 -- GLOBAL_STEP: 27400 | > loss_text_ce: 0.1547124981880188 (0.195203387394112) | > loss_mel_ce: 4.03387451171875 (nan) | > loss: 0.016621379181742668 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13467088829862137) | > loader_time: 0.0041 (0.004491982781974034)  --> TIME: 2025-02-23 21:17:51 -- STEP: 27450/36938 -- GLOBAL_STEP: 27450 | > loss_text_ce: 0.24818749725818634 (0.1951246072311215) | > loss_mel_ce: 1.4230753183364868 (nan) | > loss: 0.006631996016949415 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.1346718463480816) | > loader_time: 0.0044 (0.004491752396950057)  --> TIME: 2025-02-23 21:18:00 -- STEP: 27500/36938 -- GLOBAL_STEP: 27500 | > loss_text_ce: 0.1582338809967041 (0.19504020283601461) | > loss_mel_ce: 4.632566928863525 (nan) | > loss: 0.019011115655303 (nan) | > current_lr: 5e-06 | > step_time: 0.1073 (0.13467457068183386) | > loader_time: 0.0043 (0.004491642163016548)  --> TIME: 2025-02-23 21:18:09 -- STEP: 27550/36938 -- GLOBAL_STEP: 27550 | > loss_text_ce: 0.14923220872879028 (0.1949532266872544) | > loss_mel_ce: 1.5529378652572632 (nan) | > loss: 0.006754643749445677 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13467696478059835) | > loader_time: 0.0043 (0.004491658885768854)  --> TIME: 2025-02-23 21:18:19 -- STEP: 27600/36938 -- GLOBAL_STEP: 27600 | > loss_text_ce: 0.16314902901649475 (0.19486989182946435) | > loss_mel_ce: 4.675316333770752 (nan) | > loss: 0.019200259819626808 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13468202292055514) | > loader_time: 0.005 (0.0044920879602432)  --> TIME: 2025-02-23 21:18:28 -- STEP: 27650/36938 -- GLOBAL_STEP: 27650 | > loss_text_ce: 0.13859952986240387 (0.19478363309426036) | > loss_mel_ce: 2.3155109882354736 (nan) | > loss: 0.009738534688949585 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13468088001713602) | > loader_time: 0.0044 (0.004492014407249275)  --> TIME: 2025-02-23 21:18:37 -- STEP: 27700/36938 -- GLOBAL_STEP: 27700 | > loss_text_ce: 0.15032079815864563 (0.19470480667339726) | > loss_mel_ce: 4.2644476890563965 (nan) | > loss: 0.017518924549221992 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13467830050293075) | > loader_time: 0.0044 (0.004491880275712523)  --> TIME: 2025-02-23 21:18:46 -- STEP: 27750/36938 -- GLOBAL_STEP: 27750 | > loss_text_ce: 0.15409083664417267 (0.19462082811250303) | > loss_mel_ce: 4.5896806716918945 (nan) | > loss: 0.018824491649866104 (nan) | > current_lr: 5e-06 | > step_time: 0.1268 (0.13467940904428524) | > loader_time: 0.0046 (0.004491897548641151)  --> TIME: 2025-02-23 21:18:55 -- STEP: 27800/36938 -- GLOBAL_STEP: 27800 | > loss_text_ce: 0.15420454740524292 (0.1945381084784004) | > loss_mel_ce: 3.8499999046325684 (nan) | > loss: 0.01588970050215721 (nan) | > current_lr: 5e-06 | > step_time: 0.1079 (0.13467030407713698) | > loader_time: 0.0042 (0.004491999380880104)  --> TIME: 2025-02-23 21:19:04 -- STEP: 27850/36938 -- GLOBAL_STEP: 27850 | > loss_text_ce: 0.14401927590370178 (0.19445339118851113) | > loss_mel_ce: 4.478179931640625 (nan) | > loss: 0.01834206096827984 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13466660999524324) | > loader_time: 0.0046 (0.004492091875744014)  --> TIME: 2025-02-23 21:19:13 -- STEP: 27900/36938 -- GLOBAL_STEP: 27900 | > loss_text_ce: 0.13924665749073029 (0.19437417283360423) | > loss_mel_ce: 4.803733825683594 (nan) | > loss: 0.01961500197649002 (nan) | > current_lr: 5e-06 | > step_time: 0.1499 (0.1346617833346044) | > loader_time: 0.0048 (0.0044922226560585605)  --> TIME: 2025-02-23 21:19:23 -- STEP: 27950/36938 -- GLOBAL_STEP: 27950 | > loss_text_ce: 0.1404581516981125 (0.19428670984581206) | > loss_mel_ce: 5.133898735046387 (nan) | > loss: 0.020929988473653793 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.1346621364941559) | > loader_time: 0.0044 (0.004492445665949764)  --> TIME: 2025-02-23 21:19:32 -- STEP: 28000/36938 -- GLOBAL_STEP: 28000 | > loss_text_ce: 0.1429508924484253 (0.1941998883938147) | > loss_mel_ce: 4.015941619873047 (nan) | > loss: 0.01650354266166687 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1346572688221952) | > loader_time: 0.005 (0.004492592130388512)  --> TIME: 2025-02-23 21:19:41 -- STEP: 28050/36938 -- GLOBAL_STEP: 28050 | > loss_text_ce: 0.13523437082767487 (0.19412088087907622) | > loss_mel_ce: 3.703070878982544 (nan) | > loss: 0.01523137092590332 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13464994376993641) | > loader_time: 0.0046 (0.004492506785401241)  --> TIME: 2025-02-23 21:19:50 -- STEP: 28100/36938 -- GLOBAL_STEP: 28100 | > loss_text_ce: 0.13819444179534912 (0.19403141225598633) | > loss_mel_ce: 4.469865798950195 (nan) | > loss: 0.018285954371094704 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.1346542589689934) | > loader_time: 0.0047 (0.004492641572850434)  --> TIME: 2025-02-23 21:19:59 -- STEP: 28150/36938 -- GLOBAL_STEP: 28150 | > loss_text_ce: 0.14421163499355316 (0.19394664254085806) | > loss_mel_ce: 4.057992935180664 (nan) | > loss: 0.016675416380167007 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13465514632779155) | > loader_time: 0.0046 (0.004492655706659709)  --> TIME: 2025-02-23 21:20:08 -- STEP: 28200/36938 -- GLOBAL_STEP: 28200 | > loss_text_ce: 0.14298610389232635 (0.19386456075342373) | > loss_mel_ce: 3.813089370727539 (nan) | > loss: 0.015698712319135666 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.1346551512995532) | > loader_time: 0.0044 (0.004492598560684938)  --> TIME: 2025-02-23 21:20:18 -- STEP: 28250/36938 -- GLOBAL_STEP: 28250 | > loss_text_ce: 0.14107349514961243 (0.19378043412845708) | > loss_mel_ce: 4.476118087768555 (nan) | > loss: 0.018322190269827843 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.1346534460962368) | > loader_time: 0.0046 (0.004492515943746622)  --> TIME: 2025-02-23 21:20:27 -- STEP: 28300/36938 -- GLOBAL_STEP: 28300 | > loss_text_ce: 0.14158977568149567 (0.19369387075252292) | > loss_mel_ce: 5.4386162757873535 (nan) | > loss: 0.02214367501437664 (nan) | > current_lr: 5e-06 | > step_time: 0.1514 (0.13465629202738472) | > loader_time: 0.0045 (0.004492467536521857)  --> TIME: 2025-02-23 21:20:36 -- STEP: 28350/36938 -- GLOBAL_STEP: 28350 | > loss_text_ce: 0.15507377684116364 (0.19360928423715817) | > loss_mel_ce: 4.177261829376221 (nan) | > loss: 0.0171918086707592 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13465307951394237) | > loader_time: 0.0047 (0.004492550695173716)  --> TIME: 2025-02-23 21:20:45 -- STEP: 28400/36938 -- GLOBAL_STEP: 28400 | > loss_text_ce: 0.13057775795459747 (0.193526071461678) | > loss_mel_ce: 4.839868545532227 (nan) | > loss: 0.01972399279475212 (nan) | > current_lr: 5e-06 | > step_time: 0.1265 (0.13465220792193042) | > loader_time: 0.0043 (0.004492475306484033)  --> TIME: 2025-02-23 21:20:54 -- STEP: 28450/36938 -- GLOBAL_STEP: 28450 | > loss_text_ce: 0.14934863150119781 (0.19343909575304336) | > loss_mel_ce: 4.629831790924072 (nan) | > loss: 0.01896500401198864 (nan) | > current_lr: 5e-06 | > step_time: 0.177 (0.13465344427340287) | > loader_time: 0.0046 (0.004492355608143971)  --> TIME: 2025-02-23 21:21:04 -- STEP: 28500/36938 -- GLOBAL_STEP: 28500 | > loss_text_ce: 0.1501021683216095 (0.1933547326420479) | > loss_mel_ce: 3.41192626953125 (nan) | > loss: 0.014135033823549747 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13465793876480872) | > loader_time: 0.005 (0.0044923068013107284)  --> TIME: 2025-02-23 21:21:13 -- STEP: 28550/36938 -- GLOBAL_STEP: 28550 | > loss_text_ce: 0.1368083953857422 (0.1932631182059929) | > loss_mel_ce: 5.263938903808594 (nan) | > loss: 0.021431537345051765 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.13466843847218238) | > loader_time: 0.0044 (0.00449246890073062)  --> TIME: 2025-02-23 21:21:23 -- STEP: 28600/36938 -- GLOBAL_STEP: 28600 | > loss_text_ce: 0.14806677401065826 (0.1931787426540481) | > loss_mel_ce: 4.7115559577941895 (nan) | > loss: 0.019284218549728394 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.134668616598305) | > loader_time: 0.0044 (0.004492591702854689)  --> TIME: 2025-02-23 21:21:32 -- STEP: 28650/36938 -- GLOBAL_STEP: 28650 | > loss_text_ce: 0.14120234549045563 (0.19309201282694455) | > loss_mel_ce: 4.416366100311279 (nan) | > loss: 0.01808558963239193 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13467553532352666) | > loader_time: 0.0046 (0.004492419693898545)  --> TIME: 2025-02-23 21:21:42 -- STEP: 28700/36938 -- GLOBAL_STEP: 28700 | > loss_text_ce: 0.14427325129508972 (0.19300869232660142) | > loss_mel_ce: 3.902589797973633 (nan) | > loss: 0.01605898141860962 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13468268655318677) | > loader_time: 0.0045 (0.00449247307062977)  --> TIME: 2025-02-23 21:21:51 -- STEP: 28750/36938 -- GLOBAL_STEP: 28750 | > loss_text_ce: 0.1654798984527588 (0.1929286375457822) | > loss_mel_ce: 3.62847900390625 (nan) | > loss: 0.015055393800139427 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.134683520184395) | > loader_time: 0.0043 (0.004492490527940806)  --> TIME: 2025-02-23 21:22:00 -- STEP: 28800/36938 -- GLOBAL_STEP: 28800 | > loss_text_ce: 0.14373715221881866 (0.1928427059751828) | > loss_mel_ce: 4.895462989807129 (nan) | > loss: 0.019996827468276024 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13468582352830547) | > loader_time: 0.0047 (0.004492498776978889)  --> TIME: 2025-02-23 21:22:09 -- STEP: 28850/36938 -- GLOBAL_STEP: 28850 | > loss_text_ce: 0.14809103310108185 (0.19275696846186766) | > loss_mel_ce: 0.7648426294326782 (nan) | > loss: 0.003622752847149968 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13468440006262958) | > loader_time: 0.0044 (0.004492920862326768)  --> TIME: 2025-02-23 21:22:18 -- STEP: 28900/36938 -- GLOBAL_STEP: 28900 | > loss_text_ce: 0.1670560985803604 (0.1926777376435616) | > loss_mel_ce: 3.8860676288604736 (nan) | > loss: 0.01608382537961006 (nan) | > current_lr: 5e-06 | > step_time: 0.1069 (0.13467767896124488) | > loader_time: 0.0044 (0.004492972540607887)  --> TIME: 2025-02-23 21:22:28 -- STEP: 28950/36938 -- GLOBAL_STEP: 28950 | > loss_text_ce: 0.14045624434947968 (0.19258607573871542) | > loss_mel_ce: 3.3097519874572754 (nan) | > loss: 0.013691303320229053 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.1346819394198955) | > loader_time: 0.0045 (0.0044929987078915126)  --> TIME: 2025-02-23 21:22:37 -- STEP: 29000/36938 -- GLOBAL_STEP: 29000 | > loss_text_ce: 0.15125317871570587 (0.19250048859006336) | > loss_mel_ce: 3.87992787361145 (nan) | > loss: 0.015996750444173813 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1346902621039054) | > loader_time: 0.0048 (0.004493106800934336)  --> TIME: 2025-02-23 21:22:47 -- STEP: 29050/36938 -- GLOBAL_STEP: 29050 | > loss_text_ce: 0.13662974536418915 (0.1924145932739241) | > loss_mel_ce: 3.90805721282959 (nan) | > loss: 0.016050346195697784 (nan) | > current_lr: 5e-06 | > step_time: 0.1867 (0.13469518060733698) | > loader_time: 0.0044 (0.004493040168552076)  --> TIME: 2025-02-23 21:22:56 -- STEP: 29100/36938 -- GLOBAL_STEP: 29100 | > loss_text_ce: 0.14321325719356537 (0.1923324430070788) | > loss_mel_ce: 4.466412544250488 (nan) | > loss: 0.01829216629266739 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13470049677845775) | > loader_time: 0.0042 (0.004492755567085265)  --> TIME: 2025-02-23 21:23:05 -- STEP: 29150/36938 -- GLOBAL_STEP: 29150 | > loss_text_ce: 0.14330829679965973 (0.19224843071751246) | > loss_mel_ce: 4.014867305755615 (nan) | > loss: 0.016500696539878845 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1347046649272021) | > loader_time: 0.0046 (0.004492903091077303)  --> TIME: 2025-02-23 21:23:14 -- STEP: 29200/36938 -- GLOBAL_STEP: 29200 | > loss_text_ce: 0.17439453303813934 (0.19216644576763417) | > loss_mel_ce: 0.16013406217098236 (nan) | > loss: 0.001327494508586824 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.13470029866042443) | > loader_time: 0.005 (0.004492969774220061)  --> TIME: 2025-02-23 21:23:24 -- STEP: 29250/36938 -- GLOBAL_STEP: 29250 | > loss_text_ce: 0.1341019868850708 (0.19208150333446278) | > loss_mel_ce: 4.735454559326172 (nan) | > loss: 0.01932363770902157 (nan) | > current_lr: 5e-06 | > step_time: 0.184 (0.13470575877132815) | > loader_time: 0.0048 (0.004492732007279328)  --> TIME: 2025-02-23 21:23:33 -- STEP: 29300/36938 -- GLOBAL_STEP: 29300 | > loss_text_ce: 0.16823016107082367 (0.1919980419002495) | > loss_mel_ce: 2.731773853302002 (nan) | > loss: 0.011507952585816383 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.134704268531997) | > loader_time: 0.0043 (0.004492475189039704)  --> TIME: 2025-02-23 21:23:42 -- STEP: 29350/36938 -- GLOBAL_STEP: 29350 | > loss_text_ce: 0.14299903810024261 (0.19191197421243045) | > loss_mel_ce: 4.22139310836792 (nan) | > loss: 0.01731901802122593 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13470485569468382) | > loader_time: 0.0045 (0.004492274126770928)  --> TIME: 2025-02-23 21:23:52 -- STEP: 29400/36938 -- GLOBAL_STEP: 29400 | > loss_text_ce: 0.13932673633098602 (0.1918251566469767) | > loss_mel_ce: 4.5225300788879395 (nan) | > loss: 0.018499432131648064 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.1347069516555006) | > loader_time: 0.0049 (0.004492334362600909)  --> TIME: 2025-02-23 21:24:01 -- STEP: 29450/36938 -- GLOBAL_STEP: 29450 | > loss_text_ce: 0.13881036639213562 (0.19174280906168797) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13470672246353577) | > loader_time: 0.0042 (0.004492092650693598)  --> TIME: 2025-02-23 21:24:10 -- STEP: 29500/36938 -- GLOBAL_STEP: 29500 | > loss_text_ce: 0.14064687490463257 (0.1916582096959066) | > loss_mel_ce: 4.437966346740723 (nan) | > loss: 0.0181691013276577 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.134707686925341) | > loader_time: 0.0046 (0.004492055270631408)  --> TIME: 2025-02-23 21:24:19 -- STEP: 29550/36938 -- GLOBAL_STEP: 29550 | > loss_text_ce: 0.1515299528837204 (0.19157504606746137) | > loss_mel_ce: 4.401567459106445 (nan) | > loss: 0.018067847937345505 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13471197679361765) | > loader_time: 0.0043 (0.004492137775001563)  --> TIME: 2025-02-23 21:24:29 -- STEP: 29600/36938 -- GLOBAL_STEP: 29600 | > loss_text_ce: 0.14378204941749573 (0.19148878635133806) | > loss_mel_ce: 4.205636501312256 (nan) | > loss: 0.01725959964096546 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.1347111942155966) | > loader_time: 0.0044 (0.004492146960786844)  --> TIME: 2025-02-23 21:24:38 -- STEP: 29650/36938 -- GLOBAL_STEP: 29650 | > loss_text_ce: 0.13785937428474426 (0.19140124358952915) | > loss_mel_ce: 4.658118724822998 (nan) | > loss: 0.01903166063129902 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.13471378151120317) | > loader_time: 0.0045 (0.0044921875884520596)  --> TIME: 2025-02-23 21:24:47 -- STEP: 29700/36938 -- GLOBAL_STEP: 29700 | > loss_text_ce: 0.13388758897781372 (0.19131489775397523) | > loss_mel_ce: 4.661087989807129 (nan) | > loss: 0.01902768202126026 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13470695196980434) | > loader_time: 0.0048 (0.004492103989276795)  --> TIME: 2025-02-23 21:24:57 -- STEP: 29750/36938 -- GLOBAL_STEP: 29750 | > loss_text_ce: 0.13678710162639618 (0.19122591416725554) | > loss_mel_ce: 4.571965217590332 (nan) | > loss: 0.01868552528321743 (nan) | > current_lr: 5e-06 | > step_time: 0.1667 (0.13471693608741053) | > loader_time: 0.0045 (0.0044922285881362775)  --> TIME: 2025-02-23 21:25:06 -- STEP: 29800/36938 -- GLOBAL_STEP: 29800 | > loss_text_ce: 0.12886910140514374 (0.19113760587403) | > loss_mel_ce: 4.2680864334106445 (nan) | > loss: 0.017448237165808678 (nan) | > current_lr: 5e-06 | > step_time: 0.151 (0.13471687953744416) | > loader_time: 0.0044 (0.00449243067094941)  --> TIME: 2025-02-23 21:25:15 -- STEP: 29850/36938 -- GLOBAL_STEP: 29850 | > loss_text_ce: 0.13221904635429382 (0.19105217008784353) | > loss_mel_ce: 4.101651191711426 (nan) | > loss: 0.016801072284579277 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.1347108218737808) | > loader_time: 0.0044 (0.004492307803538949)  --> TIME: 2025-02-23 21:25:24 -- STEP: 29900/36938 -- GLOBAL_STEP: 29900 | > loss_text_ce: 0.14275521039962769 (0.19096552636187775) | > loss_mel_ce: 2.139575242996216 (nan) | > loss: 0.00905686803162098 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13470085427912828) | > loader_time: 0.0044 (0.004492243532352995)  --> TIME: 2025-02-23 21:25:33 -- STEP: 29950/36938 -- GLOBAL_STEP: 29950 | > loss_text_ce: 0.13602066040039062 (0.19087623486236224) | > loss_mel_ce: 4.9287333488464355 (nan) | > loss: 0.020098231732845306 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.1347025157852071) | > loader_time: 0.0043 (0.004492221483603118)  --> TIME: 2025-02-23 21:25:42 -- STEP: 30000/36938 -- GLOBAL_STEP: 30000 | > loss_text_ce: 0.14248605072498322 (0.19079102079470917) | > loss_mel_ce: 4.275279998779297 (nan) | > loss: 0.01753081940114498 (nan) | > current_lr: 5e-06 | > step_time: 0.1825 (0.1347053716262207) | > loader_time: 0.0044 (0.004491939322153701) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_30000.pth  --> TIME: 2025-02-23 21:25:57 -- STEP: 30050/36938 -- GLOBAL_STEP: 30050 | > loss_text_ce: 0.13142740726470947 (0.1907028774520419) | > loss_mel_ce: 0.3209894299507141 (nan) | > loss: 0.001795305055566132 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13470855168614174) | > loader_time: 0.0043 (0.004491996495378566)  --> TIME: 2025-02-23 21:26:06 -- STEP: 30100/36938 -- GLOBAL_STEP: 30100 | > loss_text_ce: 0.12988553941249847 (0.19062089248452038) | > loss_mel_ce: 4.527011871337891 (nan) | > loss: 0.01847975328564644 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13470876929768144) | > loader_time: 0.0044 (0.004492101091087036)  --> TIME: 2025-02-23 21:26:15 -- STEP: 30150/36938 -- GLOBAL_STEP: 30150 | > loss_text_ce: 0.18354909121990204 (0.19053878959533946) | > loss_mel_ce: 0.9970703125 (nan) | > loss: 0.004684997722506523 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13470792204191284) | > loader_time: 0.0043 (0.004491974069902131)  --> TIME: 2025-02-23 21:26:24 -- STEP: 30200/36938 -- GLOBAL_STEP: 30200 | > loss_text_ce: 0.1370875984430313 (0.19045593823497425) | > loss_mel_ce: 4.407339096069336 (nan) | > loss: 0.01803344115614891 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.13470057388804835) | > loader_time: 0.0044 (0.004491766083319429)  --> TIME: 2025-02-23 21:26:34 -- STEP: 30250/36938 -- GLOBAL_STEP: 30250 | > loss_text_ce: 0.14189113676548004 (0.19036638741724712) | > loss_mel_ce: 4.462556838989258 (nan) | > loss: 0.018271619454026222 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13470634031690085) | > loader_time: 0.0044 (0.004491577936598066)  --> TIME: 2025-02-23 21:26:43 -- STEP: 30300/36938 -- GLOBAL_STEP: 30300 | > loss_text_ce: 0.1284714639186859 (0.1902755700012438) | > loss_mel_ce: 3.6298868656158447 (nan) | > loss: 0.01491412054747343 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13471515694073943) | > loader_time: 0.0043 (0.00449141631425408)  --> TIME: 2025-02-23 21:26:53 -- STEP: 30350/36938 -- GLOBAL_STEP: 30350 | > loss_text_ce: 0.14736737310886383 (0.19019083500252096) | > loss_mel_ce: 3.445962429046631 (nan) | > loss: 0.014259246177971363 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13472512663883893) | > loader_time: 0.0042 (0.004491202882329924)  --> TIME: 2025-02-23 21:27:02 -- STEP: 30400/36938 -- GLOBAL_STEP: 30400 | > loss_text_ce: 0.13966834545135498 (0.19010667746616983) | > loss_mel_ce: 4.4425225257873535 (nan) | > loss: 0.018183298408985138 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13472535031406732) | > loader_time: 0.0044 (0.0044910614898330025)  --> TIME: 2025-02-23 21:27:11 -- STEP: 30450/36938 -- GLOBAL_STEP: 30450 | > loss_text_ce: 0.15046030282974243 (0.1900225139082649) | > loss_mel_ce: 4.1114726066589355 (nan) | > loss: 0.0169124323874712 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13472143902175954) | > loader_time: 0.0044 (0.0044908504298167605)  --> TIME: 2025-02-23 21:27:21 -- STEP: 30500/36938 -- GLOBAL_STEP: 30500 | > loss_text_ce: 0.12819045782089233 (0.18993868477935638) | > loss_mel_ce: 4.372324466705322 (nan) | > loss: 0.017859186977148056 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13472930445436798) | > loader_time: 0.0044 (0.004490755948864019)  --> TIME: 2025-02-23 21:27:30 -- STEP: 30550/36938 -- GLOBAL_STEP: 30550 | > loss_text_ce: 0.1448097825050354 (0.1898510613103444) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13472541632707138) | > loader_time: 0.0042 (0.004490467397749105)  --> TIME: 2025-02-23 21:27:39 -- STEP: 30600/36938 -- GLOBAL_STEP: 30600 | > loss_text_ce: 0.137376070022583 (0.18976979784448322) | > loss_mel_ce: 3.2608120441436768 (nan) | > loss: 0.013484873808920383 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.1347195433479531) | > loader_time: 0.0049 (0.004490310841915628)  --> TIME: 2025-02-23 21:27:48 -- STEP: 30650/36938 -- GLOBAL_STEP: 30650 | > loss_text_ce: 0.13951468467712402 (0.1896888221695797) | > loss_mel_ce: 3.769412040710449 (nan) | > loss: 0.015511615201830864 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.13472170268925554) | > loader_time: 0.0043 (0.004490143175420689)  --> TIME: 2025-02-23 21:27:58 -- STEP: 30700/36938 -- GLOBAL_STEP: 30700 | > loss_text_ce: 0.12935344874858856 (0.1896010306885262) | > loss_mel_ce: 2.9683024883270264 (nan) | > loss: 0.01229228638112545 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1347320881262687) | > loader_time: 0.0044 (0.004489923276808026)  --> TIME: 2025-02-23 21:28:07 -- STEP: 30750/36938 -- GLOBAL_STEP: 30750 | > loss_text_ce: 0.12261044979095459 (0.1895183750182629) | > loss_mel_ce: 4.065765857696533 (nan) | > loss: 0.016620542854070663 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.134727926672961) | > loader_time: 0.0044 (0.0044898970921834055)  --> TIME: 2025-02-23 21:28:16 -- STEP: 30800/36938 -- GLOBAL_STEP: 30800 | > loss_text_ce: 0.13513970375061035 (0.1894331520585364) | > loss_mel_ce: 4.650868892669678 (nan) | > loss: 0.018992099910974503 (nan) | > current_lr: 5e-06 | > step_time: 0.1691 (0.134733502802913) | > loader_time: 0.0046 (0.004489879314001479)  --> TIME: 2025-02-23 21:28:25 -- STEP: 30850/36938 -- GLOBAL_STEP: 30850 | > loss_text_ce: 0.13833880424499512 (0.1893477666042319) | > loss_mel_ce: 3.8710947036743164 (nan) | > loss: 0.015910452231764793 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.1347345237098234) | > loader_time: 0.0044 (0.0044897562487584974)  --> TIME: 2025-02-23 21:28:35 -- STEP: 30900/36938 -- GLOBAL_STEP: 30900 | > loss_text_ce: 0.12314862012863159 (0.1892581186785953) | > loss_mel_ce: 5.197556495666504 (nan) | > loss: 0.021113909780979156 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.134737968136194) | > loader_time: 0.0043 (0.004489583614188851)  --> TIME: 2025-02-23 21:28:44 -- STEP: 30950/36938 -- GLOBAL_STEP: 30950 | > loss_text_ce: 0.1362251341342926 (0.18917123472363154) | > loss_mel_ce: 3.789011240005493 (nan) | > loss: 0.015576336532831192 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13473828329601112) | > loader_time: 0.0041 (0.004489385931480271)  --> TIME: 2025-02-23 21:28:53 -- STEP: 31000/36938 -- GLOBAL_STEP: 31000 | > loss_text_ce: 0.15134385228157043 (0.18908366331914706) | > loss_mel_ce: 5.19921875 (nan) | > loss: 0.021232392638921738 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.13473953053259302) | > loader_time: 0.0041 (0.004489151039431146)  --> TIME: 2025-02-23 21:29:02 -- STEP: 31050/36938 -- GLOBAL_STEP: 31050 | > loss_text_ce: 0.1206977367401123 (0.18900235430293963) | > loss_mel_ce: 4.061291694641113 (nan) | > loss: 0.016595197841525078 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.134736022334938) | > loader_time: 0.0048 (0.004488989105163156)  --> TIME: 2025-02-23 21:29:12 -- STEP: 31100/36938 -- GLOBAL_STEP: 31100 | > loss_text_ce: 0.13209599256515503 (0.18891248858506812) | > loss_mel_ce: 4.447918891906738 (nan) | > loss: 0.018174663186073303 (nan) | > current_lr: 5e-06 | > step_time: 0.1749 (0.134737292102681) | > loader_time: 0.0046 (0.004488872638469314)  --> TIME: 2025-02-23 21:29:21 -- STEP: 31150/36938 -- GLOBAL_STEP: 31150 | > loss_text_ce: 0.13053035736083984 (0.1888269651892859) | > loss_mel_ce: 4.515241622924805 (nan) | > loss: 0.018435604870319366 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13473250806236184) | > loader_time: 0.0046 (0.0044886754068095775)  --> TIME: 2025-02-23 21:29:30 -- STEP: 31200/36938 -- GLOBAL_STEP: 31200 | > loss_text_ce: 0.1291370689868927 (0.18874463006806264) | > loss_mel_ce: 4.6969475746154785 (nan) | > loss: 0.019151130691170692 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13472671759434457) | > loader_time: 0.0044 (0.004488457907468819)  --> TIME: 2025-02-23 21:29:39 -- STEP: 31250/36938 -- GLOBAL_STEP: 31250 | > loss_text_ce: 0.1342153400182724 (0.18866397272896732) | > loss_mel_ce: 4.161180019378662 (nan) | > loss: 0.017045220360159874 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13471746368408424) | > loader_time: 0.0044 (0.004488232070922826)  --> TIME: 2025-02-23 21:29:48 -- STEP: 31300/36938 -- GLOBAL_STEP: 31300 | > loss_text_ce: 0.12459051609039307 (0.1885757279657895) | > loss_mel_ce: 3.384573221206665 (nan) | > loss: 0.013925254344940186 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13472105253999828) | > loader_time: 0.0042 (0.004488183119045634)  --> TIME: 2025-02-23 21:29:57 -- STEP: 31350/36938 -- GLOBAL_STEP: 31350 | > loss_text_ce: 0.13910701870918274 (0.18848978337512406) | > loss_mel_ce: 4.428601264953613 (nan) | > loss: 0.018125828355550766 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1347162035587704) | > loader_time: 0.0045 (0.004488098298153403)  --> TIME: 2025-02-23 21:30:06 -- STEP: 31400/36938 -- GLOBAL_STEP: 31400 | > loss_text_ce: 0.13498175144195557 (0.1884025474818076) | > loss_mel_ce: 3.731233835220337 (nan) | > loss: 0.015342126600444317 (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.13471248702638688) | > loader_time: 0.0044 (0.004488061385549532)  --> TIME: 2025-02-23 21:30:15 -- STEP: 31450/36938 -- GLOBAL_STEP: 31450 | > loss_text_ce: 0.12661641836166382 (0.18831976935385866) | > loss_mel_ce: 5.057621955871582 (nan) | > loss: 0.020572375506162643 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.13471745162017992) | > loader_time: 0.0048 (0.004488245823045957)  --> TIME: 2025-02-23 21:30:25 -- STEP: 31500/36938 -- GLOBAL_STEP: 31500 | > loss_text_ce: 0.13905395567417145 (0.18823660976096718) | > loss_mel_ce: 4.336331844329834 (nan) | > loss: 0.017759468406438828 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.1347128545821681) | > loader_time: 0.0047 (0.004488242467244441)  --> TIME: 2025-02-23 21:30:34 -- STEP: 31550/36938 -- GLOBAL_STEP: 31550 | > loss_text_ce: 0.12964434921741486 (0.18814699680463223) | > loss_mel_ce: 4.286193370819092 (nan) | > loss: 0.01752316765487194 (nan) | > current_lr: 5e-06 | > step_time: 0.1785 (0.13471875972868932) | > loader_time: 0.0048 (0.004488411675164489)  --> TIME: 2025-02-23 21:30:43 -- STEP: 31600/36938 -- GLOBAL_STEP: 31600 | > loss_text_ce: 0.13400976359844208 (0.1880590366371727) | > loss_mel_ce: 4.6141486167907715 (nan) | > loss: 0.018841899931430817 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13472422465493628) | > loader_time: 0.0047 (0.004488635040536686)  --> TIME: 2025-02-23 21:30:53 -- STEP: 31650/36938 -- GLOBAL_STEP: 31650 | > loss_text_ce: 0.14621922373771667 (0.18797012682303776) | > loss_mel_ce: 4.125542640686035 (nan) | > loss: 0.016951436176896095 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13473597530316997) | > loader_time: 0.0046 (0.004488595612994466)  --> TIME: 2025-02-23 21:31:02 -- STEP: 31700/36938 -- GLOBAL_STEP: 31700 | > loss_text_ce: 0.13005118072032928 (0.18788300083215645) | > loss_mel_ce: 1.6638915538787842 (nan) | > loss: 0.007118820678442717 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.1347343740252688) | > loader_time: 0.0043 (0.004488491583321724)  --> TIME: 2025-02-23 21:31:11 -- STEP: 31750/36938 -- GLOBAL_STEP: 31750 | > loss_text_ce: 0.1250661015510559 (0.18779990776481556) | > loss_mel_ce: 4.393706321716309 (nan) | > loss: 0.017931638285517693 (nan) | > current_lr: 5e-06 | > step_time: 0.179 (0.1347255633722162) | > loader_time: 0.0045 (0.004488215213685498)  --> TIME: 2025-02-23 21:31:20 -- STEP: 31800/36938 -- GLOBAL_STEP: 31800 | > loss_text_ce: 0.1587499976158142 (0.18771433362842696) | > loss_mel_ce: 4.443873405456543 (nan) | > loss: 0.01826437935233116 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13472626084801753) | > loader_time: 0.0042 (0.004488206172139365)  --> TIME: 2025-02-23 21:31:30 -- STEP: 31850/36938 -- GLOBAL_STEP: 31850 | > loss_text_ce: 0.14715810120105743 (0.1876281722500519) | > loss_mel_ce: 0.1131184920668602 (nan) | > loss: 0.0010328436037525535 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13472950920392648) | > loader_time: 0.0044 (0.004488177875896051)  --> TIME: 2025-02-23 21:31:39 -- STEP: 31900/36938 -- GLOBAL_STEP: 31900 | > loss_text_ce: 0.12926803529262543 (0.1875389364270671) | > loss_mel_ce: 3.5327882766723633 (nan) | > loss: 0.014531970024108887 (nan) | > current_lr: 5e-06 | > step_time: 0.1864 (0.13472745244406076) | > loader_time: 0.0047 (0.004488239946036478)  --> TIME: 2025-02-23 21:31:48 -- STEP: 31950/36938 -- GLOBAL_STEP: 31950 | > loss_text_ce: 0.1255742758512497 (0.1874556078086713) | > loss_mel_ce: 4.375950813293457 (nan) | > loss: 0.0178631953895092 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.13471656718724426) | > loader_time: 0.0047 (0.004488343252262574)  --> TIME: 2025-02-23 21:31:57 -- STEP: 32000/36938 -- GLOBAL_STEP: 32000 | > loss_text_ce: 0.14146488904953003 (0.18736447888612712) | > loss_mel_ce: 3.599810838699341 (nan) | > loss: 0.014846333302557468 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13472290930897213) | > loader_time: 0.0044 (0.004488431550562357)  --> TIME: 2025-02-23 21:32:06 -- STEP: 32050/36938 -- GLOBAL_STEP: 32050 | > loss_text_ce: 0.13032357394695282 (0.1872791144765573) | > loss_mel_ce: 4.001583576202393 (nan) | > loss: 0.016396457329392433 (nan) | > current_lr: 5e-06 | > step_time: 0.176 (0.13471955959213164) | > loader_time: 0.0047 (0.004488509567962955)  --> TIME: 2025-02-23 21:32:16 -- STEP: 32100/36938 -- GLOBAL_STEP: 32100 | > loss_text_ce: 0.1254410743713379 (0.18719287696448222) | > loss_mel_ce: 4.574502944946289 (nan) | > loss: 0.01865057274699211 (nan) | > current_lr: 5e-06 | > step_time: 0.1878 (0.13472190533843212) | > loader_time: 0.0049 (0.00448842270723382)  --> TIME: 2025-02-23 21:32:25 -- STEP: 32150/36938 -- GLOBAL_STEP: 32150 | > loss_text_ce: 0.121612548828125 (0.18710595132797506) | > loss_mel_ce: 4.4727253913879395 (nan) | > loss: 0.018231499940156937 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.1347324813617354) | > loader_time: 0.0071 (0.004488417097420912)  --> TIME: 2025-02-23 21:32:35 -- STEP: 32200/36938 -- GLOBAL_STEP: 32200 | > loss_text_ce: 0.1285054087638855 (0.18702311336693336) | > loss_mel_ce: 4.294308662414551 (nan) | > loss: 0.01755085028707981 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13473747459257826) | > loader_time: 0.0044 (0.004488497520825853)  --> TIME: 2025-02-23 21:32:44 -- STEP: 32250/36938 -- GLOBAL_STEP: 32250 | > loss_text_ce: 0.14771422743797302 (0.1869420459711733) | > loss_mel_ce: 4.632774829864502 (nan) | > loss: 0.018970195204019547 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13474123971776658) | > loader_time: 0.0043 (0.004488554311353082)  --> TIME: 2025-02-23 21:32:53 -- STEP: 32300/36938 -- GLOBAL_STEP: 32300 | > loss_text_ce: 0.12716221809387207 (0.18685365217573469) | > loss_mel_ce: 4.549668788909912 (nan) | > loss: 0.01855885609984398 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13474173170863504) | > loader_time: 0.0044 (0.00448847604609862)  --> TIME: 2025-02-23 21:33:03 -- STEP: 32350/36938 -- GLOBAL_STEP: 32350 | > loss_text_ce: 0.1179959625005722 (0.18676476793504723) | > loss_mel_ce: 4.349538803100586 (nan) | > loss: 0.01772831194102764 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.13475168029160398) | > loader_time: 0.0048 (0.004488439309357467)  --> TIME: 2025-02-23 21:33:12 -- STEP: 32400/36938 -- GLOBAL_STEP: 32400 | > loss_text_ce: 0.12276610732078552 (0.18668760808713608) | > loss_mel_ce: 4.088634014129639 (nan) | > loss: 0.016711905598640442 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.13475201673713944) | > loader_time: 0.0051 (0.004488495897363709)  --> TIME: 2025-02-23 21:33:21 -- STEP: 32450/36938 -- GLOBAL_STEP: 32450 | > loss_text_ce: 0.1622895449399948 (0.18660217598333165) | > loss_mel_ce: 1.9021694660186768 (nan) | > loss: 0.008192298002541065 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.1347511160722704) | > loader_time: 0.0043 (0.004488354574182895)  --> TIME: 2025-02-23 21:33:31 -- STEP: 32500/36938 -- GLOBAL_STEP: 32500 | > loss_text_ce: 0.1305803507566452 (0.18651617815586202) | > loss_mel_ce: 3.820148468017578 (nan) | > loss: 0.015677496790885925 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.13475883353306908) | > loader_time: 0.0046 (0.004488370484572166)  --> TIME: 2025-02-23 21:33:40 -- STEP: 32550/36938 -- GLOBAL_STEP: 32550 | > loss_text_ce: 0.12064583599567413 (0.18642923765651243) | > loss_mel_ce: 4.1160173416137695 (nan) | > loss: 0.01681215688586235 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13475868386240994) | > loader_time: 0.0044 (0.00448858195918677)  --> TIME: 2025-02-23 21:33:50 -- STEP: 32600/36938 -- GLOBAL_STEP: 32600 | > loss_text_ce: 0.1350593864917755 (0.18634343051745814) | > loss_mel_ce: 3.8267951011657715 (nan) | > loss: 0.015721645206212997 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13476102709770416) | > loader_time: 0.0045 (0.004488922730545301)  --> TIME: 2025-02-23 21:33:59 -- STEP: 32650/36938 -- GLOBAL_STEP: 32650 | > loss_text_ce: 0.14395274221897125 (0.18625417164009128) | > loss_mel_ce: 4.057735919952393 (nan) | > loss: 0.016673369333148003 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.1347613019271451) | > loader_time: 0.0044 (0.004489022622977404)  --> TIME: 2025-02-23 21:34:08 -- STEP: 32700/36938 -- GLOBAL_STEP: 32700 | > loss_text_ce: 0.13428884744644165 (0.18616903261831197) | > loss_mel_ce: 0.8526814579963684 (nan) | > loss: 0.003916549030691385 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13476302677703042) | > loader_time: 0.0045 (0.00448905341851235)  --> TIME: 2025-02-23 21:34:17 -- STEP: 32750/36938 -- GLOBAL_STEP: 32750 | > loss_text_ce: 0.13134965300559998 (0.1860864236921296) | > loss_mel_ce: 4.850002765655518 (nan) | > loss: 0.01976727321743965 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13475766269305492) | > loader_time: 0.0045 (0.00448910858066935)  --> TIME: 2025-02-23 21:34:27 -- STEP: 32800/36938 -- GLOBAL_STEP: 32800 | > loss_text_ce: 0.1333090364933014 (0.18599846974832948) | > loss_mel_ce: 0.1881646066904068 (nan) | > loss: 0.0012756892247125506 (nan) | > current_lr: 5e-06 | > step_time: 0.1501 (0.1347646960616133) | > loader_time: 0.0045 (0.004489013799806896)  --> TIME: 2025-02-23 21:34:36 -- STEP: 32850/36938 -- GLOBAL_STEP: 32850 | > loss_text_ce: 0.12115628272294998 (0.18591295307312167) | > loss_mel_ce: 4.604839324951172 (nan) | > loss: 0.01875395141541958 (nan) | > current_lr: 5e-06 | > step_time: 0.1582 (0.13477606351698762) | > loader_time: 0.0049 (0.004488993837771689)  --> TIME: 2025-02-23 21:34:45 -- STEP: 32900/36938 -- GLOBAL_STEP: 32900 | > loss_text_ce: 0.12872003018856049 (0.18582738368231258) | > loss_mel_ce: 5.01031494140625 (nan) | > loss: 0.020392995327711105 (nan) | > current_lr: 5e-06 | > step_time: 0.1612 (0.13477709338295635) | > loader_time: 0.0044 (0.00448895929794541)  --> TIME: 2025-02-23 21:34:55 -- STEP: 32950/36938 -- GLOBAL_STEP: 32950 | > loss_text_ce: 0.12694010138511658 (0.1857399559192842) | > loss_mel_ce: 4.41291618347168 (nan) | > loss: 0.01801530458033085 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13477520059200837) | > loader_time: 0.0043 (0.004488935456109493)  --> TIME: 2025-02-23 21:35:04 -- STEP: 33000/36938 -- GLOBAL_STEP: 33000 | > loss_text_ce: 0.12538671493530273 (0.18565707868427908) | > loss_mel_ce: 4.503186225891113 (nan) | > loss: 0.018367353826761246 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13477656402732602) | > loader_time: 0.0041 (0.004488915241125834)  --> TIME: 2025-02-23 21:35:13 -- STEP: 33050/36938 -- GLOBAL_STEP: 33050 | > loss_text_ce: 0.12168943881988525 (0.18557029346249254) | > loss_mel_ce: 4.578855037689209 (nan) | > loss: 0.018652955070137978 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.1347756209351854) | > loader_time: 0.0044 (0.004488812185451928)  --> TIME: 2025-02-23 21:35:23 -- STEP: 33100/36938 -- GLOBAL_STEP: 33100 | > loss_text_ce: 0.13711825013160706 (0.18548198512381084) | > loss_mel_ce: 2.7228305339813232 (nan) | > loss: 0.01134900376200676 (nan) | > current_lr: 5e-06 | > step_time: 0.1506 (0.13478526801861515) | > loader_time: 0.0047 (0.004488696605417281)  --> TIME: 2025-02-23 21:35:32 -- STEP: 33150/36938 -- GLOBAL_STEP: 33150 | > loss_text_ce: 0.13245201110839844 (0.18539731334147558) | > loss_mel_ce: 4.7458271980285645 (nan) | > loss: 0.019358251243829727 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13478012611423895) | > loader_time: 0.0043 (0.004488546981351206)  --> TIME: 2025-02-23 21:35:41 -- STEP: 33200/36938 -- GLOBAL_STEP: 33200 | > loss_text_ce: 0.14306151866912842 (0.18531072269721172) | > loss_mel_ce: 4.058847427368164 (nan) | > loss: 0.016674242913722992 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13478999867496935) | > loader_time: 0.0042 (0.004488291287996661)  --> TIME: 2025-02-23 21:35:50 -- STEP: 33250/36938 -- GLOBAL_STEP: 33250 | > loss_text_ce: 0.12427949905395508 (0.18522895237848247) | > loss_mel_ce: 4.717437267303467 (nan) | > loss: 0.01921316236257553 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13478218425664906) | > loader_time: 0.0043 (0.004487920051230503)  --> TIME: 2025-02-23 21:35:59 -- STEP: 33300/36938 -- GLOBAL_STEP: 33300 | > loss_text_ce: 0.12935175001621246 (0.185146035914709) | > loss_mel_ce: 4.4611639976501465 (nan) | > loss: 0.01821633242070675 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.1347779920652486) | > loader_time: 0.0044 (0.004487739081855273)  --> TIME: 2025-02-23 21:36:09 -- STEP: 33350/36938 -- GLOBAL_STEP: 33350 | > loss_text_ce: 0.12026416510343552 (0.18505545697983383) | > loss_mel_ce: 4.368582725524902 (nan) | > loss: 0.017812885344028473 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13477983650596648) | > loader_time: 0.0043 (0.004487574161260718)  --> TIME: 2025-02-23 21:36:18 -- STEP: 33400/36938 -- GLOBAL_STEP: 33400 | > loss_text_ce: 0.12398795783519745 (0.1849704013787162) | > loss_mel_ce: 4.241905212402344 (nan) | > loss: 0.01732497476041317 (nan) | > current_lr: 5e-06 | > step_time: 0.1761 (0.134777806604696) | > loader_time: 0.0046 (0.004487570145886793)  --> TIME: 2025-02-23 21:36:27 -- STEP: 33450/36938 -- GLOBAL_STEP: 33450 | > loss_text_ce: 0.13745801150798798 (0.18488666537943055) | > loss_mel_ce: 4.165799140930176 (nan) | > loss: 0.017076417803764343 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.1347730038162505) | > loader_time: 0.0043 (0.004487442236131951)  --> TIME: 2025-02-23 21:36:36 -- STEP: 33500/36938 -- GLOBAL_STEP: 33500 | > loss_text_ce: 0.12153070420026779 (0.1848070025010337) | > loss_mel_ce: 2.62803316116333 (nan) | > loss: 0.01091096829622984 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.1347699393656739) | > loader_time: 0.0043 (0.004487161878329583)  --> TIME: 2025-02-23 21:36:46 -- STEP: 33550/36938 -- GLOBAL_STEP: 33550 | > loss_text_ce: 0.12698039412498474 (0.1847200188902377) | > loss_mel_ce: 4.603384971618652 (nan) | > loss: 0.01877129264175892 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13477948581764007) | > loader_time: 0.0044 (0.004487023111960193)  --> TIME: 2025-02-23 21:36:55 -- STEP: 33600/36938 -- GLOBAL_STEP: 33600 | > loss_text_ce: 0.10453800112009048 (0.18463343189120088) | > loss_mel_ce: 4.019425868988037 (nan) | > loss: 0.016364937648177147 (nan) | > current_lr: 5e-06 | > step_time: 0.1441 (0.13477855216889387) | > loader_time: 0.0043 (0.004486875349567028)  --> TIME: 2025-02-23 21:37:04 -- STEP: 33650/36938 -- GLOBAL_STEP: 33650 | > loss_text_ce: 0.13877461850643158 (0.1845463751482305) | > loss_mel_ce: 4.702939987182617 (nan) | > loss: 0.019213153049349785 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.1347859493726099) | > loader_time: 0.0044 (0.004486713211036874)  --> TIME: 2025-02-23 21:37:13 -- STEP: 33700/36938 -- GLOBAL_STEP: 33700 | > loss_text_ce: 0.12378529459238052 (0.18446089173968325) | > loss_mel_ce: 4.238682270050049 (nan) | > loss: 0.017311381176114082 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13478945413753846) | > loader_time: 0.0045 (0.00448646855991033)  --> TIME: 2025-02-23 21:37:23 -- STEP: 33750/36938 -- GLOBAL_STEP: 33750 | > loss_text_ce: 0.12351903319358826 (0.18437577564694227) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13479461603518114) | > loader_time: 0.0043 (0.0044862305323282664)  --> TIME: 2025-02-23 21:37:32 -- STEP: 33800/36938 -- GLOBAL_STEP: 33800 | > loss_text_ce: 0.12006811797618866 (0.1842924343849688) | > loss_mel_ce: 3.1907002925872803 (nan) | > loss: 0.013137970119714737 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13478856592488941) | > loader_time: 0.0051 (0.004486115282103839)  --> TIME: 2025-02-23 21:37:41 -- STEP: 33850/36938 -- GLOBAL_STEP: 33850 | > loss_text_ce: 0.11923376470804214 (0.18421354670886406) | > loss_mel_ce: 4.392096519470215 (nan) | > loss: 0.017902104184031487 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13478250541574546) | > loader_time: 0.0046 (0.004486026186358432)  --> TIME: 2025-02-23 21:37:50 -- STEP: 33900/36938 -- GLOBAL_STEP: 33900 | > loss_text_ce: 0.1272578090429306 (0.18412714033000185) | > loss_mel_ce: 1.893316388130188 (nan) | > loss: 0.008018151856958866 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.1347833116145986) | > loader_time: 0.0042 (0.004485889423561638)  --> TIME: 2025-02-23 21:37:59 -- STEP: 33950/36938 -- GLOBAL_STEP: 33950 | > loss_text_ce: 0.11708656698465347 (0.18404146981520178) | > loss_mel_ce: 4.403074741363525 (nan) | > loss: 0.017937147989869118 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.1347853515004221) | > loader_time: 0.0043 (0.004485651740971629)  --> TIME: 2025-02-23 21:38:09 -- STEP: 34000/36938 -- GLOBAL_STEP: 34000 | > loss_text_ce: 0.11957822740077972 (0.18395428898163535) | > loss_mel_ce: 4.223203659057617 (nan) | > loss: 0.017233263701200485 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.13478675306544816) | > loader_time: 0.0045 (0.00448581705373874)  --> TIME: 2025-02-23 21:38:18 -- STEP: 34050/36938 -- GLOBAL_STEP: 34050 | > loss_text_ce: 0.12773482501506805 (0.18386897953049333) | > loss_mel_ce: 4.8078460693359375 (nan) | > loss: 0.019585639238357544 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.13478796826585504) | > loader_time: 0.0044 (0.004485874988259001)  --> TIME: 2025-02-23 21:38:27 -- STEP: 34100/36938 -- GLOBAL_STEP: 34100 | > loss_text_ce: 0.1327792853116989 (0.18378565561714538) | > loss_mel_ce: 4.616480350494385 (nan) | > loss: 0.018846269696950912 (nan) | > current_lr: 5e-06 | > step_time: 0.1517 (0.13478852969111116) | > loader_time: 0.0046 (0.004485707723500071)  --> TIME: 2025-02-23 21:38:37 -- STEP: 34150/36938 -- GLOBAL_STEP: 34150 | > loss_text_ce: 0.1177685484290123 (0.1836967372959734) | > loss_mel_ce: 4.268625736236572 (nan) | > loss: 0.01740632764995098 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.1347967267001553) | > loader_time: 0.0045 (0.004485765969596161)  --> TIME: 2025-02-23 21:38:46 -- STEP: 34200/36938 -- GLOBAL_STEP: 34200 | > loss_text_ce: 0.12286891788244247 (0.18361641395118108) | > loss_mel_ce: 4.775292873382568 (nan) | > loss: 0.019437151029706 (nan) | > current_lr: 5e-06 | > step_time: 0.1461 (0.13479407150146083) | > loader_time: 0.0052 (0.00448594263422557)  --> TIME: 2025-02-23 21:38:55 -- STEP: 34250/36938 -- GLOBAL_STEP: 34250 | > loss_text_ce: 0.14079296588897705 (0.18353129076805377) | > loss_mel_ce: 4.005819797515869 (nan) | > loss: 0.01645481213927269 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13479772296961426) | > loader_time: 0.0046 (0.004485933310794115)  --> TIME: 2025-02-23 21:39:05 -- STEP: 34300/36938 -- GLOBAL_STEP: 34300 | > loss_text_ce: 0.11818715929985046 (0.1834444797388748) | > loss_mel_ce: 4.086212158203125 (nan) | > loss: 0.016684124246239662 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13480611837987871) | > loader_time: 0.0044 (0.004485968730192471)  --> TIME: 2025-02-23 21:39:14 -- STEP: 34350/36938 -- GLOBAL_STEP: 34350 | > loss_text_ce: 0.1306612640619278 (0.18335537851854472) | > loss_mel_ce: 3.3988804817199707 (nan) | > loss: 0.014006119221448898 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13481394612980244) | > loader_time: 0.0044 (0.0044859579452780985)  --> TIME: 2025-02-23 21:39:23 -- STEP: 34400/36938 -- GLOBAL_STEP: 34400 | > loss_text_ce: 0.2435937374830246 (0.18327631753544416) | > loss_mel_ce: 2.2415928840637207 (nan) | > loss: 0.009861852042376995 (nan) | > current_lr: 5e-06 | > step_time: 0.1075 (0.13480821154838601) | > loader_time: 0.0043 (0.004486098753851494)  --> TIME: 2025-02-23 21:39:33 -- STEP: 34450/36938 -- GLOBAL_STEP: 34450 | > loss_text_ce: 0.12076660245656967 (0.18318953800659857) | > loss_mel_ce: 4.8099284172058105 (nan) | > loss: 0.0195662509649992 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.13480916489710704) | > loader_time: 0.0045 (0.004486096746170622)  --> TIME: 2025-02-23 21:39:42 -- STEP: 34500/36938 -- GLOBAL_STEP: 34500 | > loss_text_ce: 0.11980040371417999 (0.18310285652979505) | > loss_mel_ce: 4.105983734130859 (nan) | > loss: 0.016768986359238625 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.1348118573824587) | > loader_time: 0.0044 (0.004485991208449635)  --> TIME: 2025-02-23 21:39:51 -- STEP: 34550/36938 -- GLOBAL_STEP: 34550 | > loss_text_ce: 0.12849953770637512 (0.18301866025401037) | > loss_mel_ce: 3.1274633407592773 (nan) | > loss: 0.012920488603413105 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13480855149574344) | > loader_time: 0.0043 (0.004485861989420141)  --> TIME: 2025-02-23 21:40:00 -- STEP: 34600/36938 -- GLOBAL_STEP: 34600 | > loss_text_ce: 0.15387152135372162 (0.1829307468812584) | > loss_mel_ce: 0.1634347140789032 (nan) | > loss: 0.0012591517297551036 (nan) | > current_lr: 5e-06 | > step_time: 0.108 (0.13481125998359422) | > loader_time: 0.0044 (0.004485709743003604)  --> TIME: 2025-02-23 21:40:10 -- STEP: 34650/36938 -- GLOBAL_STEP: 34650 | > loss_text_ce: 0.1270202249288559 (0.18284792724389307) | > loss_mel_ce: 4.191997528076172 (nan) | > loss: 0.01713896170258522 (nan) | > current_lr: 5e-06 | > step_time: 0.1526 (0.1348068112694018) | > loader_time: 0.0046 (0.004485547415277811)  --> TIME: 2025-02-23 21:40:19 -- STEP: 34700/36938 -- GLOBAL_STEP: 34700 | > loss_text_ce: 0.12303348630666733 (0.18276105519051283) | > loss_mel_ce: 4.649162292480469 (nan) | > loss: 0.018937285989522934 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13480478770451396) | > loader_time: 0.0045 (0.004485349881889477)  --> TIME: 2025-02-23 21:40:28 -- STEP: 34750/36938 -- GLOBAL_STEP: 34750 | > loss_text_ce: 0.11996684968471527 (0.18267492847476877) | > loss_mel_ce: 4.824693202972412 (nan) | > loss: 0.01962166838347912 (nan) | > current_lr: 5e-06 | > step_time: 0.1531 (0.13480952072829863) | > loader_time: 0.0043 (0.0044851784671810675)  --> TIME: 2025-02-23 21:40:38 -- STEP: 34800/36938 -- GLOBAL_STEP: 34800 | > loss_text_ce: 0.1092827096581459 (0.18258857398711353) | > loss_mel_ce: 4.891664981842041 (nan) | > loss: 0.019845031201839447 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13481448931940881) | > loader_time: 0.0046 (0.0044849989743068076)  --> TIME: 2025-02-23 21:40:47 -- STEP: 34850/36938 -- GLOBAL_STEP: 34850 | > loss_text_ce: 0.11770697683095932 (0.18249804981234544) | > loss_mel_ce: 2.65280818939209 (nan) | > loss: 0.010994108393788338 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13482289594762675) | > loader_time: 0.0042 (0.004484968643790485)  --> TIME: 2025-02-23 21:40:57 -- STEP: 34900/36938 -- GLOBAL_STEP: 34900 | > loss_text_ce: 0.15469783544540405 (0.1824122284677269) | > loss_mel_ce: 4.0519795417785645 (nan) | > loss: 0.01669316552579403 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.1348266579223588) | > loader_time: 0.0044 (0.00448496080059716)  --> TIME: 2025-02-23 21:41:06 -- STEP: 34950/36938 -- GLOBAL_STEP: 34950 | > loss_text_ce: 0.12163105607032776 (0.1823315848682992) | > loss_mel_ce: 2.6417644023895264 (nan) | > loss: 0.010965855792164803 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.1348245728851579) | > loader_time: 0.0042 (0.004484778930871418)  --> TIME: 2025-02-23 21:41:15 -- STEP: 35000/36938 -- GLOBAL_STEP: 35000 | > loss_text_ce: 0.12359645217657089 (0.18224712585125602) | > loss_mel_ce: 4.36942720413208 (nan) | > loss: 0.01782946102321148 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.1348292950902688) | > loader_time: 0.0043 (0.00448471962383813)  --> TIME: 2025-02-23 21:41:24 -- STEP: 35050/36938 -- GLOBAL_STEP: 35050 | > loss_text_ce: 0.15299156308174133 (0.18216860246853833) | > loss_mel_ce: 2.2606353759765625 (nan) | > loss: 0.009577885270118713 (nan) | > current_lr: 5e-06 | > step_time: 0.1123 (0.13482297328671664) | > loader_time: 0.0042 (0.004484713550300957)  --> TIME: 2025-02-23 21:41:33 -- STEP: 35100/36938 -- GLOBAL_STEP: 35100 | > loss_text_ce: 0.13164062798023224 (0.182088158562609) | > loss_mel_ce: 3.2862443923950195 (nan) | > loss: 0.013563036918640137 (nan) | > current_lr: 5e-06 | > step_time: 0.1086 (0.13481536886970763) | > loader_time: 0.0045 (0.004484616344810532)  --> TIME: 2025-02-23 21:41:42 -- STEP: 35150/36938 -- GLOBAL_STEP: 35150 | > loss_text_ce: 0.12037423998117447 (0.18200539967519397) | > loss_mel_ce: 2.930337429046631 (nan) | > loss: 0.012105999514460564 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13481219883833512) | > loader_time: 0.0043 (0.004484541324599177)  --> TIME: 2025-02-23 21:41:51 -- STEP: 35200/36938 -- GLOBAL_STEP: 35200 | > loss_text_ce: 0.1519750952720642 (0.18192094600928713) | > loss_mel_ce: 0.8720612525939941 (nan) | > loss: 0.0040636369958519936 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13481154233217454) | > loader_time: 0.0041 (0.00448430721732701)  --> TIME: 2025-02-23 21:42:01 -- STEP: 35250/36938 -- GLOBAL_STEP: 35250 | > loss_text_ce: 0.14116571843624115 (0.18183713393579073) | > loss_mel_ce: 4.572624206542969 (nan) | > loss: 0.018705517053604126 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1348148893058744) | > loader_time: 0.0047 (0.004484211712018795)  --> TIME: 2025-02-23 21:42:10 -- STEP: 35300/36938 -- GLOBAL_STEP: 35300 | > loss_text_ce: 0.12735547125339508 (0.18175321733415403) | > loss_mel_ce: 0.707413911819458 (nan) | > loss: 0.0033125770278275013 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13481275352472838) | > loader_time: 0.0044 (0.004484348972525857)  --> TIME: 2025-02-23 21:42:19 -- STEP: 35350/36938 -- GLOBAL_STEP: 35350 | > loss_text_ce: 0.1203845962882042 (0.18166547943227976) | > loss_mel_ce: 4.267714023590088 (nan) | > loss: 0.01741309091448784 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13481227034389592) | > loader_time: 0.0044 (0.004484448372221712)  --> TIME: 2025-02-23 21:42:28 -- STEP: 35400/36938 -- GLOBAL_STEP: 35400 | > loss_text_ce: 0.12213154882192612 (0.18158132994073867) | > loss_mel_ce: 3.5743651390075684 (nan) | > loss: 0.014668638817965984 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13481216432011278) | > loader_time: 0.0041 (0.004484311328769382)  --> TIME: 2025-02-23 21:42:37 -- STEP: 35450/36938 -- GLOBAL_STEP: 35450 | > loss_text_ce: 0.12606488168239594 (0.1814961006512335) | > loss_mel_ce: 5.050863265991211 (nan) | > loss: 0.02054336667060852 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.1348099796593799) | > loader_time: 0.0045 (0.004484415982438727)  --> TIME: 2025-02-23 21:42:47 -- STEP: 35500/36938 -- GLOBAL_STEP: 35500 | > loss_text_ce: 0.11450287699699402 (0.18141394105406644) | > loss_mel_ce: 3.8634033203125 (nan) | > loss: 0.015785343945026398 (nan) | > current_lr: 5e-06 | > step_time: 0.1507 (0.13481144053499675) | > loader_time: 0.0045 (0.0044844733023307465)  --> TIME: 2025-02-23 21:42:56 -- STEP: 35550/36938 -- GLOBAL_STEP: 35550 | > loss_text_ce: 0.11265729367733002 (0.1813315408256647) | > loss_mel_ce: 4.417774677276611 (nan) | > loss: 0.01797790452837944 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13481088067576738) | > loader_time: 0.0047 (0.0044845484211810125)  --> TIME: 2025-02-23 21:43:05 -- STEP: 35600/36938 -- GLOBAL_STEP: 35600 | > loss_text_ce: 0.12264420092105865 (0.18124796866261314) | > loss_mel_ce: 4.194356918334961 (nan) | > loss: 0.017130957916378975 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1348090545113189) | > loader_time: 0.0044 (0.004484594350450468)  --> TIME: 2025-02-23 21:43:14 -- STEP: 35650/36938 -- GLOBAL_STEP: 35650 | > loss_text_ce: 0.11519896239042282 (0.18116186642943483) | > loss_mel_ce: 4.627413749694824 (nan) | > loss: 0.018819892778992653 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.1348124787984713) | > loader_time: 0.0044 (0.004484772300987699)  --> TIME: 2025-02-23 21:43:24 -- STEP: 35700/36938 -- GLOBAL_STEP: 35700 | > loss_text_ce: 0.11248674243688583 (0.18107739335056366) | > loss_mel_ce: 4.956679821014404 (nan) | > loss: 0.020115742459893227 (nan) | > current_lr: 5e-06 | > step_time: 0.1848 (0.13481506342313757) | > loader_time: 0.0047 (0.004484707768223839)  --> TIME: 2025-02-23 21:43:33 -- STEP: 35750/36938 -- GLOBAL_STEP: 35750 | > loss_text_ce: 0.11652562022209167 (0.18099145321329074) | > loss_mel_ce: 4.207149028778076 (nan) | > loss: 0.01715744100511074 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.1348136727459802) | > loader_time: 0.0043 (0.004484672539717636)  --> TIME: 2025-02-23 21:43:42 -- STEP: 35800/36938 -- GLOBAL_STEP: 35800 | > loss_text_ce: 0.11232583969831467 (0.18090814183793846) | > loss_mel_ce: 4.461544513702393 (nan) | > loss: 0.01815027929842472 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13481772978212797) | > loader_time: 0.0044 (0.0044846432035861436)  --> TIME: 2025-02-23 21:43:52 -- STEP: 35850/36938 -- GLOBAL_STEP: 35850 | > loss_text_ce: 0.11779492348432541 (0.1808193402096226) | > loss_mel_ce: 4.4081573486328125 (nan) | > loss: 0.01796012930572033 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13481870609347513) | > loader_time: 0.0042 (0.004484459466016413)  --> TIME: 2025-02-23 21:44:01 -- STEP: 35900/36938 -- GLOBAL_STEP: 35900 | > loss_text_ce: 0.12261329591274261 (0.180738928223801) | > loss_mel_ce: 2.184617280960083 (nan) | > loss: 0.009155676700174809 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13481979977785935) | > loader_time: 0.0043 (0.00448453019920495)  --> TIME: 2025-02-23 21:44:10 -- STEP: 35950/36938 -- GLOBAL_STEP: 35950 | > loss_text_ce: 0.13607673346996307 (0.18065369039504428) | > loss_mel_ce: 1.78887939453125 (nan) | > loss: 0.007638715207576752 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13482827823915036) | > loader_time: 0.0043 (0.004484460615813292)  --> TIME: 2025-02-23 21:44:20 -- STEP: 36000/36938 -- GLOBAL_STEP: 36000 | > loss_text_ce: 0.12294652312994003 (0.18056978808219143) | > loss_mel_ce: 4.320066452026367 (nan) | > loss: 0.017631005495786667 (nan) | > current_lr: 5e-06 | > step_time: 0.1343 (0.13483887853887586) | > loader_time: 0.0043 (0.004484409530957514)  --> TIME: 2025-02-23 21:44:29 -- STEP: 36050/36938 -- GLOBAL_STEP: 36050 | > loss_text_ce: 0.12016178667545319 (0.18048832105180665) | > loss_mel_ce: 0.14349976181983948 (nan) | > loss: 0.0010462760692462325 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13483302027772678) | > loader_time: 0.0044 (0.004484172647769813)  --> TIME: 2025-02-23 21:44:38 -- STEP: 36100/36938 -- GLOBAL_STEP: 36100 | > loss_text_ce: 0.13668040931224823 (0.18040867095900776) | > loss_mel_ce: 2.3212132453918457 (nan) | > loss: 0.009753546677529812 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13482701178072795) | > loader_time: 0.0042 (0.004483992710007834)  --> TIME: 2025-02-23 21:44:47 -- STEP: 36150/36938 -- GLOBAL_STEP: 36150 | > loss_text_ce: 0.1322965919971466 (0.18032822115869684) | > loss_mel_ce: 0.2698296308517456 (nan) | > loss: 0.0015957390423864126 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13482696704020544) | > loader_time: 0.0044 (0.00448378942814109)  --> TIME: 2025-02-23 21:44:56 -- STEP: 36200/36938 -- GLOBAL_STEP: 36200 | > loss_text_ce: 0.12621504068374634 (0.18024495071201438) | > loss_mel_ce: 4.622874736785889 (nan) | > loss: 0.01884559541940689 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.13482763824542157) | > loader_time: 0.0043 (0.004483534670666419)  --> TIME: 2025-02-23 21:45:05 -- STEP: 36250/36938 -- GLOBAL_STEP: 36250 | > loss_text_ce: 0.12443260103464127 (0.18016338085634867) | > loss_mel_ce: 4.67165470123291 (nan) | > loss: 0.01903209276497364 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13482380259941604) | > loader_time: 0.0044 (0.004483322478984945)  --> TIME: 2025-02-23 21:45:15 -- STEP: 36300/36938 -- GLOBAL_STEP: 36300 | > loss_text_ce: 0.12721271812915802 (0.18008143047040068) | > loss_mel_ce: 4.4679412841796875 (nan) | > loss: 0.01823473908007145 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13482850918756936) | > loader_time: 0.0044 (0.004483198311703248)  --> TIME: 2025-02-23 21:45:24 -- STEP: 36350/36938 -- GLOBAL_STEP: 36350 | > loss_text_ce: 0.11237835884094238 (0.18000032042374708) | > loss_mel_ce: 2.8308627605438232 (nan) | > loss: 0.011679529212415218 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13482839384302212) | > loader_time: 0.0041 (0.0044829807255258106)  --> TIME: 2025-02-23 21:45:34 -- STEP: 36400/36938 -- GLOBAL_STEP: 36400 | > loss_text_ce: 0.11296205967664719 (0.1799145736174832) | > loss_mel_ce: 3.9536170959472656 (nan) | > loss: 0.016137219965457916 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13483904421984724) | > loader_time: 0.0044 (0.004482806645906874)  --> TIME: 2025-02-23 21:45:43 -- STEP: 36450/36938 -- GLOBAL_STEP: 36450 | > loss_text_ce: 0.11614179611206055 (0.17983161682761145) | > loss_mel_ce: 3.370849609375 (nan) | > loss: 0.013837268576025963 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13483198867740512) | > loader_time: 0.0044 (0.004482563408640991)  --> TIME: 2025-02-23 21:45:52 -- STEP: 36500/36938 -- GLOBAL_STEP: 36500 | > loss_text_ce: 0.11370071768760681 (0.1797494575442506) | > loss_mel_ce: 2.0392019748687744 (nan) | > loss: 0.008543265052139759 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.13483443173317286) | > loader_time: 0.0042 (0.004482318871641801)  --> TIME: 2025-02-23 21:46:01 -- STEP: 36550/36938 -- GLOBAL_STEP: 36550 | > loss_text_ce: 0.16022370755672455 (0.17966783226331243) | > loss_mel_ce: 0.1166856586933136 (nan) | > loss: 0.001098846667446196 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13483357488864647) | > loader_time: 0.004 (0.004482023771419067)  --> TIME: 2025-02-23 21:46:11 -- STEP: 36600/36938 -- GLOBAL_STEP: 36600 | > loss_text_ce: 0.11441211402416229 (0.17958381403563192) | > loss_mel_ce: 4.276662826538086 (nan) | > loss: 0.017424901947379112 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13483604540590613) | > loader_time: 0.0045 (0.004481889836775136)  --> TIME: 2025-02-23 21:46:20 -- STEP: 36650/36938 -- GLOBAL_STEP: 36650 | > loss_text_ce: 0.12075398862361908 (0.17950484497038013) | > loss_mel_ce: 4.496744155883789 (nan) | > loss: 0.01832340471446514 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13483665210651255) | > loader_time: 0.0044 (0.004481764691913586)  --> TIME: 2025-02-23 21:46:29 -- STEP: 36700/36938 -- GLOBAL_STEP: 36700 | > loss_text_ce: 0.1163942813873291 (0.17942521319848875) | > loss_mel_ce: 3.5774729251861572 (nan) | > loss: 0.014658204279839993 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.13483824119905918) | > loader_time: 0.0044 (0.004481574897870042)  --> TIME: 2025-02-23 21:46:38 -- STEP: 36750/36938 -- GLOBAL_STEP: 36750 | > loss_text_ce: 0.19166797399520874 (0.17934133695582868) | > loss_mel_ce: 1.8580619096755981 (nan) | > loss: 0.008133849129080772 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13483900378999483) | > loader_time: 0.0042 (0.004481363951754394)  --> TIME: 2025-02-23 21:46:48 -- STEP: 36800/36938 -- GLOBAL_STEP: 36800 | > loss_text_ce: 0.11792024224996567 (0.17926262161066758) | > loss_mel_ce: 2.3388118743896484 (nan) | > loss: 0.00974893756210804 (nan) | > current_lr: 5e-06 | > step_time: 0.103 (0.13483819500259966) | > loader_time: 0.004 (0.004481071000513808)  --> TIME: 2025-02-23 21:46:57 -- STEP: 36850/36938 -- GLOBAL_STEP: 36850 | > loss_text_ce: 0.10640033334493637 (0.1791753944883992) | > loss_mel_ce: 3.796374559402466 (nan) | > loss: 0.015487202443182468 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.134833804270308) | > loader_time: 0.0042 (0.004480887999204625)  --> TIME: 2025-02-23 21:47:06 -- STEP: 36900/36938 -- GLOBAL_STEP: 36900 | > loss_text_ce: 0.1269889920949936 (0.17909179884647528) | > loss_mel_ce: 2.2693653106689453 (nan) | > loss: 0.009509342722594738 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.1348289837423726) | > loader_time: 0.0045 (0.004480706747308319)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.01845622534799103 (-0.0022678280820940984) | > avg_loss_text_ce: 0.11555741985540578 (-0.13119949990569957) | > avg_loss_mel_ce: 3.7890475569385114 (-1.876862718917356) | > avg_loss: 3.904604980261019 (-2.0080621950697193) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_36938.pth  > EPOCH: 2/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-23 21:47:35)   --> TIME: 2025-02-23 21:47:37 -- STEP: 12/36938 -- GLOBAL_STEP: 36950 | > loss_text_ce: 0.11872254312038422 (0.12153150948385398) | > loss_mel_ce: 4.769329071044922 (3.6726525922616324) | > loss: 0.019397031515836716 (0.015056287016098699) | > current_lr: 5e-06 | > step_time: 0.1939 (0.13660728931427002) | > loader_time: 0.0049 (0.004805227120717367)  --> TIME: 2025-02-23 21:47:46 -- STEP: 62/36938 -- GLOBAL_STEP: 37000 | > loss_text_ce: 0.12389707565307617 (0.12018048210490134) | > loss_mel_ce: 2.9213380813598633 (nan) | > loss: 0.012084267102181911 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13273090700949394) | > loader_time: 0.0044 (0.004558071013419858)  --> TIME: 2025-02-23 21:47:55 -- STEP: 112/36938 -- GLOBAL_STEP: 37050 | > loss_text_ce: 0.12485266476869583 (0.1202431199406939) | > loss_mel_ce: 4.515129089355469 (nan) | > loss: 0.018412627279758453 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13239081629684984) | > loader_time: 0.0043 (0.004484615155628748)  --> TIME: 2025-02-23 21:48:05 -- STEP: 162/36938 -- GLOBAL_STEP: 37100 | > loss_text_ce: 0.10780525952577591 (0.11926492777319601) | > loss_mel_ce: 3.994023084640503 (nan) | > loss: 0.016277099028229713 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13313378375253546) | > loader_time: 0.0048 (0.004561830449987338)  --> TIME: 2025-02-23 21:48:14 -- STEP: 212/36938 -- GLOBAL_STEP: 37150 | > loss_text_ce: 0.10629177838563919 (0.11885565849688817) | > loss_mel_ce: 4.851357936859131 (nan) | > loss: 0.019673213362693787 (nan) | > current_lr: 5e-06 | > step_time: 0.1267 (0.13408411111471782) | > loader_time: 0.0044 (0.004556402845202753)  --> TIME: 2025-02-23 21:48:23 -- STEP: 262/36938 -- GLOBAL_STEP: 37200 | > loss_text_ce: 0.11605730652809143 (0.11819314291231504) | > loss_mel_ce: 4.040859222412109 (nan) | > loss: 0.016495700925588608 (nan) | > current_lr: 5e-06 | > step_time: 0.1858 (0.1331916592503322) | > loader_time: 0.0046 (0.004557797016988274)  --> TIME: 2025-02-23 21:48:32 -- STEP: 312/36938 -- GLOBAL_STEP: 37250 | > loss_text_ce: 0.11157572269439697 (0.11778111920620386) | > loss_mel_ce: 4.0631103515625 (nan) | > loss: 0.016566215083003044 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.1340528031190237) | > loader_time: 0.0043 (0.0045402714839348395)  --> TIME: 2025-02-23 21:48:42 -- STEP: 362/36938 -- GLOBAL_STEP: 37300 | > loss_text_ce: 0.1278119832277298 (0.11810805496260605) | > loss_mel_ce: 4.629034042358398 (nan) | > loss: 0.018876373767852783 (nan) | > current_lr: 5e-06 | > step_time: 0.1483 (0.13425357697418394) | > loader_time: 0.0045 (0.004516288720441789)  --> TIME: 2025-02-23 21:48:51 -- STEP: 412/36938 -- GLOBAL_STEP: 37350 | > loss_text_ce: 0.12269055098295212 (0.11821376114577345) | > loss_mel_ce: 3.4542336463928223 (nan) | > loss: 0.014194143936038017 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13378557533893778) | > loader_time: 0.0041 (0.004498603853207191)  --> TIME: 2025-02-23 21:49:00 -- STEP: 462/36938 -- GLOBAL_STEP: 37400 | > loss_text_ce: 0.12166277319192886 (0.11825720511086575) | > loss_mel_ce: 3.8554635047912598 (nan) | > loss: 0.015782248228788376 (nan) | > current_lr: 5e-06 | > step_time: 0.162 (0.13378537526894438) | > loader_time: 0.0047 (0.00448720898979154)  --> TIME: 2025-02-23 21:49:09 -- STEP: 512/36938 -- GLOBAL_STEP: 37450 | > loss_text_ce: 0.10792548954486847 (0.11792977577715646) | > loss_mel_ce: 4.392729759216309 (nan) | > loss: 0.017859743908047676 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.134316811338067) | > loader_time: 0.0043 (0.0044883484952151775)  --> TIME: 2025-02-23 21:49:19 -- STEP: 562/36938 -- GLOBAL_STEP: 37500 | > loss_text_ce: 0.12432947009801865 (0.11767600654389086) | > loss_mel_ce: 4.519773483276367 (nan) | > loss: 0.0184289813041687 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13436139690494192) | > loader_time: 0.0042 (0.004475949497833794)  --> TIME: 2025-02-23 21:49:28 -- STEP: 612/36938 -- GLOBAL_STEP: 37550 | > loss_text_ce: 0.1267068088054657 (0.11772360108500603) | > loss_mel_ce: 4.6499481201171875 (nan) | > loss: 0.01895497925579548 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13431635557436472) | > loader_time: 0.0042 (0.004473474680208695)  --> TIME: 2025-02-23 21:49:37 -- STEP: 662/36938 -- GLOBAL_STEP: 37600 | > loss_text_ce: 0.11090198904275894 (0.1178409966142156) | > loss_mel_ce: 4.544858932495117 (nan) | > loss: 0.018475241959095 (nan) | > current_lr: 5e-06 | > step_time: 0.1882 (0.1342024169299538) | > loader_time: 0.0046 (0.0044756342637214805)  --> TIME: 2025-02-23 21:49:46 -- STEP: 712/36938 -- GLOBAL_STEP: 37650 | > loss_text_ce: 0.12310546636581421 (0.11756111140457097) | > loss_mel_ce: 2.5543923377990723 (nan) | > loss: 0.010624991729855537 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13462106364496643) | > loader_time: 0.0043 (0.004474373011106857)  --> TIME: 2025-02-23 21:49:56 -- STEP: 762/36938 -- GLOBAL_STEP: 37700 | > loss_text_ce: 0.11384475231170654 (0.11748112665777757) | > loss_mel_ce: 4.513329982757568 (nan) | > loss: 0.018361806869506836 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.13459067206996012) | > loader_time: 0.0044 (0.004469008896294542)  --> TIME: 2025-02-23 21:50:05 -- STEP: 812/36938 -- GLOBAL_STEP: 37750 | > loss_text_ce: 0.12276279181241989 (0.1175213358617328) | > loss_mel_ce: 4.6640625 (nan) | > loss: 0.018995339050889015 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13445818835291368) | > loader_time: 0.0044 (0.004470916217183834)  --> TIME: 2025-02-23 21:50:14 -- STEP: 862/36938 -- GLOBAL_STEP: 37800 | > loss_text_ce: 0.11061269789934158 (0.1173819668435442) | > loss_mel_ce: 4.394829273223877 (nan) | > loss: 0.017878739163279533 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.1345701734870326) | > loader_time: 0.0043 (0.004474113546225419)  --> TIME: 2025-02-23 21:50:23 -- STEP: 912/36938 -- GLOBAL_STEP: 37850 | > loss_text_ce: 0.13025324046611786 (0.11727696620418053) | > loss_mel_ce: 4.747640132904053 (nan) | > loss: 0.01935672201216221 (nan) | > current_lr: 5e-06 | > step_time: 0.145 (0.13444669518554425) | > loader_time: 0.0048 (0.0044856390409302276)  --> TIME: 2025-02-23 21:50:32 -- STEP: 962/36938 -- GLOBAL_STEP: 37900 | > loss_text_ce: 0.11900243163108826 (0.11718159169733153) | > loss_mel_ce: 3.937549591064453 (nan) | > loss: 0.016097430139780045 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13450582086916046) | > loader_time: 0.0043 (0.00449427522393621)  --> TIME: 2025-02-23 21:50:42 -- STEP: 1012/36938 -- GLOBAL_STEP: 37950 | > loss_text_ce: 0.12145407497882843 (0.11713787598955065) | > loss_mel_ce: 9.75 (nan) | > loss: 0.03917244076728821 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13446086951395286) | > loader_time: 0.0043 (0.004497417583767132)  --> TIME: 2025-02-23 21:50:51 -- STEP: 1062/36938 -- GLOBAL_STEP: 38000 | > loss_text_ce: 0.11909236013889313 (0.11704101853367299) | > loss_mel_ce: 4.746660232543945 (nan) | > loss: 0.0193085428327322 (nan) | > current_lr: 5e-06 | > step_time: 0.1628 (0.1347548588970064) | > loader_time: 0.0042 (0.004494171582597347)  --> TIME: 2025-02-23 21:51:01 -- STEP: 1112/36938 -- GLOBAL_STEP: 38050 | > loss_text_ce: 0.11085517704486847 (0.11703600020234962) | > loss_mel_ce: 4.28559684753418 (nan) | > loss: 0.017446238547563553 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13487083877590944) | > loader_time: 0.0044 (0.004493893908082152)  --> TIME: 2025-02-23 21:51:10 -- STEP: 1162/36938 -- GLOBAL_STEP: 38100 | > loss_text_ce: 0.12914858758449554 (0.11702447651970407) | > loss_mel_ce: 3.3439407348632812 (nan) | > loss: 0.01378210075199604 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.13493708673729968) | > loader_time: 0.0042 (0.0044924031026186894)  --> TIME: 2025-02-23 21:51:19 -- STEP: 1212/36938 -- GLOBAL_STEP: 38150 | > loss_text_ce: 0.11730434000492096 (0.11698043953147855) | > loss_mel_ce: 4.733491897583008 (nan) | > loss: 0.019249193370342255 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13492174844930685) | > loader_time: 0.0043 (0.0044900914623398945)  --> TIME: 2025-02-23 21:51:28 -- STEP: 1262/36938 -- GLOBAL_STEP: 38200 | > loss_text_ce: 0.12048576027154922 (0.11696581203614476) | > loss_mel_ce: 4.173052787780762 (nan) | > loss: 0.017037851735949516 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13494951864807828) | > loader_time: 0.0043 (0.0044900190244574895)  --> TIME: 2025-02-23 21:51:38 -- STEP: 1312/36938 -- GLOBAL_STEP: 38250 | > loss_text_ce: 0.10766871273517609 (0.11688085115605557) | > loss_mel_ce: 4.152525901794434 (nan) | > loss: 0.016905535012483597 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13508214528967716) | > loader_time: 0.0045 (0.0044863347600146)  --> TIME: 2025-02-23 21:51:47 -- STEP: 1362/36938 -- GLOBAL_STEP: 38300 | > loss_text_ce: 0.12519331276416779 (0.11680211410167109) | > loss_mel_ce: 4.057009220123291 (nan) | > loss: 0.01659604161977768 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13522050051612158) | > loader_time: 0.0046 (0.004484406126753343)  --> TIME: 2025-02-23 21:51:56 -- STEP: 1412/36938 -- GLOBAL_STEP: 38350 | > loss_text_ce: 0.1054399162530899 (0.11684360456977451) | > loss_mel_ce: 4.444355010986328 (nan) | > loss: 0.018054744228720665 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13512000821467526) | > loader_time: 0.0045 (0.004482107864898589)  --> TIME: 2025-02-23 21:52:05 -- STEP: 1462/36938 -- GLOBAL_STEP: 38400 | > loss_text_ce: 0.12240850180387497 (0.11675798318128416) | > loss_mel_ce: 4.807186126708984 (nan) | > loss: 0.0195618849247694 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.1350247908877938) | > loader_time: 0.0046 (0.004479493227278968)  --> TIME: 2025-02-23 21:52:14 -- STEP: 1512/36938 -- GLOBAL_STEP: 38450 | > loss_text_ce: 0.11289365589618683 (0.11667006688990764) | > loss_mel_ce: 3.8586785793304443 (nan) | > loss: 0.015760207548737526 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13488079426149863) | > loader_time: 0.0044 (0.004478242504533629)  --> TIME: 2025-02-23 21:52:24 -- STEP: 1562/36938 -- GLOBAL_STEP: 38500 | > loss_text_ce: 0.1222286969423294 (0.11662185586101725) | > loss_mel_ce: 2.972987651824951 (nan) | > loss: 0.012282605282962322 (nan) | > current_lr: 5e-06 | > step_time: 0.1031 (0.13498316852139775) | > loader_time: 0.0043 (0.0044747542174921676)  --> TIME: 2025-02-23 21:52:33 -- STEP: 1612/36938 -- GLOBAL_STEP: 38550 | > loss_text_ce: 0.12020228803157806 (0.116554503705787) | > loss_mel_ce: 1.693756103515625 (nan) | > loss: 0.00719824805855751 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1351270403518868) | > loader_time: 0.0043 (0.004471547402459983)  --> TIME: 2025-02-23 21:52:42 -- STEP: 1662/36938 -- GLOBAL_STEP: 38600 | > loss_text_ce: 0.10760625451803207 (0.1165029561162784) | > loss_mel_ce: 3.9557383060455322 (nan) | > loss: 0.016124384477734566 (nan) | > current_lr: 5e-06 | > step_time: 0.1781 (0.1348998870803707) | > loader_time: 0.0046 (0.004469569265914156)  --> TIME: 2025-02-23 21:52:51 -- STEP: 1712/36938 -- GLOBAL_STEP: 38650 | > loss_text_ce: 0.11559827625751495 (0.11641886476949553) | > loss_mel_ce: 4.212175369262695 (nan) | > loss: 0.017173705622553825 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13479749969789953) | > loader_time: 0.0046 (0.004471880392493485)  --> TIME: 2025-02-23 21:53:00 -- STEP: 1762/36938 -- GLOBAL_STEP: 38700 | > loss_text_ce: 0.11463847011327744 (0.1164418568583068) | > loss_mel_ce: 4.385288715362549 (nan) | > loss: 0.01785685494542122 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13460927261261643) | > loader_time: 0.0045 (0.004472632684177219)  --> TIME: 2025-02-23 21:53:10 -- STEP: 1812/36938 -- GLOBAL_STEP: 38750 | > loss_text_ce: 0.10468212515115738 (0.1163148017678303) | > loss_mel_ce: 3.730829954147339 (nan) | > loss: 0.015220287255942822 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1347410817019991) | > loader_time: 0.0046 (0.004475843827456036)  --> TIME: 2025-02-23 21:53:19 -- STEP: 1862/36938 -- GLOBAL_STEP: 38800 | > loss_text_ce: 0.10705345869064331 (0.1162672939277169) | > loss_mel_ce: 3.0344202518463135 (nan) | > loss: 0.01246616616845131 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.1347596382864311) | > loader_time: 0.0044 (0.004481272333802776)  --> TIME: 2025-02-23 21:53:28 -- STEP: 1912/36938 -- GLOBAL_STEP: 38850 | > loss_text_ce: 0.102264404296875 (0.11625377970229382) | > loss_mel_ce: 2.884303569793701 (nan) | > loss: 0.011851460672914982 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.1346679539361263) | > loader_time: 0.0044 (0.00447916186504284)  --> TIME: 2025-02-23 21:53:37 -- STEP: 1962/36938 -- GLOBAL_STEP: 38900 | > loss_text_ce: 0.12117182463407516 (0.1162554971364903) | > loss_mel_ce: 2.277322769165039 (nan) | > loss: 0.009517835453152657 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13468837592215382) | > loader_time: 0.0044 (0.004478080435508859)  --> TIME: 2025-02-23 21:53:46 -- STEP: 2012/36938 -- GLOBAL_STEP: 38950 | > loss_text_ce: 0.09840022027492523 (0.11619088753213584) | > loss_mel_ce: 4.568469047546387 (nan) | > loss: 0.0185193233191967 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13473506808043983) | > loader_time: 0.0043 (0.0044814991666592835)  --> TIME: 2025-02-23 21:53:55 -- STEP: 2062/36938 -- GLOBAL_STEP: 39000 | > loss_text_ce: 0.12097779661417007 (0.11624446965457744) | > loss_mel_ce: 3.8740220069885254 (nan) | > loss: 0.01585317589342594 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13462575874088145) | > loader_time: 0.0042 (0.004482428155745496)  --> TIME: 2025-02-23 21:54:05 -- STEP: 2112/36938 -- GLOBAL_STEP: 39050 | > loss_text_ce: 0.11952432245016098 (0.11617633669326706) | > loss_mel_ce: 4.071510314941406 (nan) | > loss: 0.01663109101355076 (nan) | > current_lr: 5e-06 | > step_time: 0.1796 (0.13465985813827258) | > loader_time: 0.0052 (0.004480029484539318)  --> TIME: 2025-02-23 21:54:14 -- STEP: 2162/36938 -- GLOBAL_STEP: 39100 | > loss_text_ce: 0.11015119403600693 (0.11610741819238687) | > loss_mel_ce: 4.0366010665893555 (nan) | > loss: 0.01645536720752716 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1346374109428755) | > loader_time: 0.0042 (0.0044803777080680545)  --> TIME: 2025-02-23 21:54:23 -- STEP: 2212/36938 -- GLOBAL_STEP: 39150 | > loss_text_ce: 0.11191890388727188 (0.116146039830733) | > loss_mel_ce: 4.324467182159424 (nan) | > loss: 0.017604708671569824 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.1346589837755479) | > loader_time: 0.0043 (0.004478633080425572)  --> TIME: 2025-02-23 21:54:32 -- STEP: 2262/36938 -- GLOBAL_STEP: 39200 | > loss_text_ce: 0.12165399640798569 (0.11607472176616646) | > loss_mel_ce: 0.030029296875 (nan) | > loss: 0.0006019179127179086 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13466872239302613) | > loader_time: 0.0046 (0.004477827133698129)  --> TIME: 2025-02-23 21:54:42 -- STEP: 2312/36938 -- GLOBAL_STEP: 39250 | > loss_text_ce: 0.10754436254501343 (0.11603907484135831) | > loss_mel_ce: 4.536566257476807 (nan) | > loss: 0.01842901110649109 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.13472113580439776) | > loader_time: 0.0045 (0.004476092987819526)  --> TIME: 2025-02-23 21:54:51 -- STEP: 2362/36938 -- GLOBAL_STEP: 39300 | > loss_text_ce: 0.09890728443861008 (0.11600899220989182) | > loss_mel_ce: 4.647367477416992 (nan) | > loss: 0.018834425136446953 (nan) | > current_lr: 5e-06 | > step_time: 0.1884 (0.13462226528116078) | > loader_time: 0.0045 (0.004473935538283048)  --> TIME: 2025-02-23 21:54:59 -- STEP: 2412/36938 -- GLOBAL_STEP: 39350 | > loss_text_ce: 0.10651053488254547 (0.11596403896783918) | > loss_mel_ce: 3.703914165496826 (nan) | > loss: 0.015120734460651875 (nan) | > current_lr: 5e-06 | > step_time: 0.1494 (0.13447975351245084) | > loader_time: 0.0042 (0.004473995608872241)  --> TIME: 2025-02-23 21:55:09 -- STEP: 2462/36938 -- GLOBAL_STEP: 39400 | > loss_text_ce: 0.1428098827600479 (0.1158930366929957) | > loss_mel_ce: 3.3031058311462402 (nan) | > loss: 0.013674269430339336 (nan) | > current_lr: 5e-06 | > step_time: 0.1045 (0.13447506644685164) | > loader_time: 0.0043 (0.004473710137591139)  --> TIME: 2025-02-23 21:55:18 -- STEP: 2512/36938 -- GLOBAL_STEP: 39450 | > loss_text_ce: 0.10585765540599823 (0.11586755637826912) | > loss_mel_ce: 4.482275485992432 (nan) | > loss: 0.018206879496574402 (nan) | > current_lr: 5e-06 | > step_time: 0.1037 (0.13444334847532294) | > loader_time: 0.0043 (0.004475108946964236)  --> TIME: 2025-02-23 21:55:27 -- STEP: 2562/36938 -- GLOBAL_STEP: 39500 | > loss_text_ce: 0.10577470809221268 (0.11581331567662852) | > loss_mel_ce: 4.659057140350342 (nan) | > loss: 0.0189080648124218 (nan) | > current_lr: 5e-06 | > step_time: 0.1751 (0.13446876706786715) | > loader_time: 0.0046 (0.0044736447137003225)  --> TIME: 2025-02-23 21:55:36 -- STEP: 2612/36938 -- GLOBAL_STEP: 39550 | > loss_text_ce: 0.11254163831472397 (0.11577872163509159) | > loss_mel_ce: 3.517155647277832 (nan) | > loss: 0.014403562061488628 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13447324094144367) | > loader_time: 0.0045 (0.0044718307156292585)  --> TIME: 2025-02-23 21:55:46 -- STEP: 2662/36938 -- GLOBAL_STEP: 39600 | > loss_text_ce: 0.1111101508140564 (0.11569792481739663) | > loss_mel_ce: 3.7370097637176514 (nan) | > loss: 0.015270317904651165 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13457984954768087) | > loader_time: 0.0047 (0.004471059225270121)  --> TIME: 2025-02-23 21:55:55 -- STEP: 2712/36938 -- GLOBAL_STEP: 39650 | > loss_text_ce: 0.108454130589962 (0.11566662825204459) | > loss_mel_ce: 3.910830020904541 (nan) | > loss: 0.015949541702866554 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13455206463470595) | > loader_time: 0.0046 (0.004472406652473062)  --> TIME: 2025-02-23 21:56:04 -- STEP: 2762/36938 -- GLOBAL_STEP: 39700 | > loss_text_ce: 0.10973475873470306 (0.11561998407977551) | > loss_mel_ce: 1.8084603548049927 (nan) | > loss: 0.007611885666847229 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13457077193484643) | > loader_time: 0.0043 (0.004471760742220645)  --> TIME: 2025-02-23 21:56:13 -- STEP: 2812/36938 -- GLOBAL_STEP: 39750 | > loss_text_ce: 0.10050924122333527 (0.11552923632412093) | > loss_mel_ce: 4.85856819152832 (nan) | > loss: 0.019678879529237747 (nan) | > current_lr: 5e-06 | > step_time: 0.1612 (0.13458138780607443) | > loader_time: 0.0044 (0.0044719831194002585)  --> TIME: 2025-02-23 21:56:23 -- STEP: 2862/36938 -- GLOBAL_STEP: 39800 | > loss_text_ce: 0.10665326565504074 (0.11545390311100119) | > loss_mel_ce: 3.9498162269592285 (nan) | > loss: 0.01609710231423378 (nan) | > current_lr: 5e-06 | > step_time: 0.1873 (0.1345981606897319) | > loader_time: 0.0045 (0.004471685984682817)  --> TIME: 2025-02-23 21:56:32 -- STEP: 2912/36938 -- GLOBAL_STEP: 39850 | > loss_text_ce: 0.12052928656339645 (0.11541208032528863) | > loss_mel_ce: 4.588630199432373 (nan) | > loss: 0.018687142059206963 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1346155739092567) | > loader_time: 0.0044 (0.0044707036116620955)  --> TIME: 2025-02-23 21:56:41 -- STEP: 2962/36938 -- GLOBAL_STEP: 39900 | > loss_text_ce: 0.12498153746128082 (0.11538191917456142) | > loss_mel_ce: 4.437192916870117 (nan) | > loss: 0.018103867769241333 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13463934356501434) | > loader_time: 0.0043 (0.0044697557728810506)  --> TIME: 2025-02-23 21:56:50 -- STEP: 3012/36938 -- GLOBAL_STEP: 39950 | > loss_text_ce: 0.12104743719100952 (0.1153334916023857) | > loss_mel_ce: 4.489358425140381 (nan) | > loss: 0.01829526200890541 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13466442145832666) | > loader_time: 0.0044 (0.00446959969848592)  --> TIME: 2025-02-23 21:57:00 -- STEP: 3062/36938 -- GLOBAL_STEP: 40000 | > loss_text_ce: 0.10033727437257767 (0.11527409918173168) | > loss_mel_ce: 4.857856273651123 (nan) | > loss: 0.019675374031066895 (nan) | > current_lr: 5e-06 | > step_time: 0.1909 (0.13464460133104694) | > loader_time: 0.0046 (0.004469514282445823) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_40000.pth  --> TIME: 2025-02-23 21:57:15 -- STEP: 3112/36938 -- GLOBAL_STEP: 40050 | > loss_text_ce: 0.09271901100873947 (0.11520749401387338) | > loss_mel_ce: 4.601951599121094 (nan) | > loss: 0.0186296459287405 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.13470884100330538) | > loader_time: 0.0047 (0.004470242785917147)  --> TIME: 2025-02-23 21:57:24 -- STEP: 3162/36938 -- GLOBAL_STEP: 40100 | > loss_text_ce: 0.11676643788814545 (0.11514830425532467) | > loss_mel_ce: 4.141197681427002 (nan) | > loss: 0.01689668372273445 (nan) | > current_lr: 5e-06 | > step_time: 0.1588 (0.13474592787038686) | > loader_time: 0.0047 (0.004470399932571788)  --> TIME: 2025-02-23 21:57:33 -- STEP: 3212/36938 -- GLOBAL_STEP: 40150 | > loss_text_ce: 0.11925426125526428 (0.11511189290838479) | > loss_mel_ce: 4.224198818206787 (nan) | > loss: 0.017235925421118736 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13462950714201596) | > loader_time: 0.0043 (0.004469470766977234)  --> TIME: 2025-02-23 21:57:42 -- STEP: 3262/36938 -- GLOBAL_STEP: 40200 | > loss_text_ce: 0.11670161783695221 (0.11502121496947908) | > loss_mel_ce: 3.0589041709899902 (nan) | > loss: 0.012601611204445362 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13471229430139622) | > loader_time: 0.0041 (0.00446910736825406)  --> TIME: 2025-02-23 21:57:51 -- STEP: 3312/36938 -- GLOBAL_STEP: 40250 | > loss_text_ce: 0.09252435714006424 (0.11497140567578774) | > loss_mel_ce: 4.272064685821533 (nan) | > loss: 0.017319800332188606 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13471048271310515) | > loader_time: 0.0043 (0.0044674774730839)  --> TIME: 2025-02-23 21:58:01 -- STEP: 3362/36938 -- GLOBAL_STEP: 40300 | > loss_text_ce: 0.10545322299003601 (0.11491622615567922) | > loss_mel_ce: 4.340855121612549 (nan) | > loss: 0.017644081264734268 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13472251002807006) | > loader_time: 0.0043 (0.004468355059694628)  --> TIME: 2025-02-23 21:58:10 -- STEP: 3412/36938 -- GLOBAL_STEP: 40350 | > loss_text_ce: 0.11131036281585693 (0.1148365559023647) | > loss_mel_ce: 5.24908447265625 (nan) | > loss: 0.02127140946686268 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13471102183477282) | > loader_time: 0.0044 (0.004468442407051255)  --> TIME: 2025-02-23 21:58:19 -- STEP: 3462/36938 -- GLOBAL_STEP: 40400 | > loss_text_ce: 0.10720672458410263 (0.11481395080375716) | > loss_mel_ce: 4.06289529800415 (nan) | > loss: 0.016548024490475655 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.1347720720119247) | > loader_time: 0.0045 (0.00446829445991538)  --> TIME: 2025-02-23 21:58:28 -- STEP: 3512/36938 -- GLOBAL_STEP: 40450 | > loss_text_ce: 0.10560039430856705 (0.11475351623605287) | > loss_mel_ce: 4.348018646240234 (nan) | > loss: 0.01767309196293354 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.1347656792415847) | > loader_time: 0.0047 (0.004467103638790192)  --> TIME: 2025-02-23 21:58:38 -- STEP: 3562/36938 -- GLOBAL_STEP: 40500 | > loss_text_ce: 0.10576772689819336 (0.11470592375436842) | > loss_mel_ce: 3.874342203140259 (nan) | > loss: 0.015794087201356888 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13478322614116142) | > loader_time: 0.0045 (0.004467016923702014)  --> TIME: 2025-02-23 21:58:47 -- STEP: 3612/36938 -- GLOBAL_STEP: 40550 | > loss_text_ce: 0.11556591093540192 (0.1146442916838335) | > loss_mel_ce: 3.6208088397979736 (nan) | > loss: 0.014826885424554348 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13478472449322765) | > loader_time: 0.0044 (0.004467995261829167)  --> TIME: 2025-02-23 21:58:56 -- STEP: 3662/36938 -- GLOBAL_STEP: 40600 | > loss_text_ce: 0.10691232234239578 (0.1145776411745309) | > loss_mel_ce: 4.9185309410095215 (nan) | > loss: 0.019942235201597214 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13478684855054354) | > loader_time: 0.0043 (0.004468121859364539)  --> TIME: 2025-02-23 21:59:06 -- STEP: 3712/36938 -- GLOBAL_STEP: 40650 | > loss_text_ce: 0.11076155304908752 (0.11449933601222165) | > loss_mel_ce: 5.010247230529785 (nan) | > loss: 0.02032146416604519 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.13482510497600889) | > loader_time: 0.0051 (0.004467481233436491)  --> TIME: 2025-02-23 21:59:15 -- STEP: 3762/36938 -- GLOBAL_STEP: 40700 | > loss_text_ce: 0.12583236396312714 (0.11442567524933425) | > loss_mel_ce: 4.287693023681641 (nan) | > loss: 0.01751399226486683 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13490175155171188) | > loader_time: 0.0043 (0.004466389861910479)  --> TIME: 2025-02-23 21:59:24 -- STEP: 3812/36938 -- GLOBAL_STEP: 40750 | > loss_text_ce: 0.12489306181669235 (0.1143686448527815) | > loss_mel_ce: 4.508368492126465 (nan) | > loss: 0.018385959789156914 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13487385746564606) | > loader_time: 0.0043 (0.0044655362056159826)  --> TIME: 2025-02-23 21:59:34 -- STEP: 3862/36938 -- GLOBAL_STEP: 40800 | > loss_text_ce: 0.10229694843292236 (0.1143120199766016) | > loss_mel_ce: 4.796600341796875 (nan) | > loss: 0.019440069794654846 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13488174935562644) | > loader_time: 0.0042 (0.00446426232573531)  --> TIME: 2025-02-23 21:59:43 -- STEP: 3912/36938 -- GLOBAL_STEP: 40850 | > loss_text_ce: 0.10813737660646439 (0.11428367415850639) | > loss_mel_ce: 3.6712801456451416 (nan) | > loss: 0.014997689053416252 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.134908818576965) | > loader_time: 0.0044 (0.004463406062564972)  --> TIME: 2025-02-23 21:59:52 -- STEP: 3962/36938 -- GLOBAL_STEP: 40900 | > loss_text_ce: 0.10756754875183105 (0.11423649024152506) | > loss_mel_ce: 4.0303874015808105 (nan) | > loss: 0.016420455649495125 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13495411384232064) | > loader_time: 0.0042 (0.004461969407383454)  --> TIME: 2025-02-23 22:00:01 -- STEP: 4012/36938 -- GLOBAL_STEP: 40950 | > loss_text_ce: 0.11252867430448532 (0.11419631091549119) | > loss_mel_ce: 3.9381611347198486 (nan) | > loss: 0.01607416570186615 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.1348802210207833) | > loader_time: 0.0043 (0.0044601970275163945)  --> TIME: 2025-02-23 22:00:11 -- STEP: 4062/36938 -- GLOBAL_STEP: 41000 | > loss_text_ce: 0.09733081609010696 (0.11412294055296249) | > loss_mel_ce: 2.533818244934082 (nan) | > loss: 0.010441068559885025 (nan) | > current_lr: 5e-06 | > step_time: 0.1447 (0.1349362760620832) | > loader_time: 0.0048 (0.004459767550567527)  --> TIME: 2025-02-23 22:00:20 -- STEP: 4112/36938 -- GLOBAL_STEP: 41050 | > loss_text_ce: 0.12568436563014984 (0.11407846671525493) | > loss_mel_ce: 1.4974582195281982 (nan) | > loss: 0.006441042292863131 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13494571672339395) | > loader_time: 0.0044 (0.004459213769853352)  --> TIME: 2025-02-23 22:00:29 -- STEP: 4162/36938 -- GLOBAL_STEP: 41100 | > loss_text_ce: 0.10595067590475082 (0.11405287561996352) | > loss_mel_ce: 2.9584786891937256 (nan) | > loss: 0.012160434387624264 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13488370538158875) | > loader_time: 0.0044 (0.004459289791834926)  --> TIME: 2025-02-23 22:00:38 -- STEP: 4212/36938 -- GLOBAL_STEP: 41150 | > loss_text_ce: 0.10738514363765717 (0.11401205747463487) | > loss_mel_ce: 0.07080078125 (nan) | > loss: 0.0007070870487950742 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.1349000384551735) | > loader_time: 0.0043 (0.004459708447922787)  --> TIME: 2025-02-23 22:00:47 -- STEP: 4262/36938 -- GLOBAL_STEP: 41200 | > loss_text_ce: 0.11758875846862793 (0.11396230296601108) | > loss_mel_ce: 4.122534275054932 (nan) | > loss: 0.016825884580612183 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13488709518239297) | > loader_time: 0.0044 (0.004460798524904011)  --> TIME: 2025-02-23 22:00:56 -- STEP: 4312/36938 -- GLOBAL_STEP: 41250 | > loss_text_ce: 0.10478300601243973 (0.11392023478304637) | > loss_mel_ce: 4.442733287811279 (nan) | > loss: 0.01804570108652115 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.1348450981730213) | > loader_time: 0.0045 (0.0044619742373146285)  --> TIME: 2025-02-23 22:01:05 -- STEP: 4362/36938 -- GLOBAL_STEP: 41300 | > loss_text_ce: 0.10174664109945297 (0.11387570241123703) | > loss_mel_ce: 4.9658203125 (nan) | > loss: 0.0201093927025795 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.13484261731170083) | > loader_time: 0.0045 (0.00446222895386568)  --> TIME: 2025-02-23 22:01:15 -- STEP: 4412/36938 -- GLOBAL_STEP: 41350 | > loss_text_ce: 0.11171146482229233 (0.11381235688330373) | > loss_mel_ce: 4.674210071563721 (nan) | > loss: 0.018991753458976746 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13484982966512105) | > loader_time: 0.0044 (0.004463468912181703)  --> TIME: 2025-02-23 22:01:24 -- STEP: 4462/36938 -- GLOBAL_STEP: 41400 | > loss_text_ce: 0.1053844541311264 (0.11374121039090598) | > loss_mel_ce: 4.779678821563721 (nan) | > loss: 0.019385172054171562 (nan) | > current_lr: 5e-06 | > step_time: 0.165 (0.1348388701583816) | > loader_time: 0.0072 (0.004465176459653988)  --> TIME: 2025-02-23 22:01:33 -- STEP: 4512/36938 -- GLOBAL_STEP: 41450 | > loss_text_ce: 0.10755712538957596 (0.11367787077283192) | > loss_mel_ce: 4.3944292068481445 (nan) | > loss: 0.017865026369690895 (nan) | > current_lr: 5e-06 | > step_time: 0.1505 (0.13483159418435822) | > loader_time: 0.0046 (0.00446545633863896)  --> TIME: 2025-02-23 22:01:42 -- STEP: 4562/36938 -- GLOBAL_STEP: 41500 | > loss_text_ce: 0.10660842061042786 (0.1136299285944593) | > loss_mel_ce: 4.5308966636657715 (nan) | > loss: 0.01840279810130596 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13483137396645628) | > loader_time: 0.0044 (0.0044659572648981264)  --> TIME: 2025-02-23 22:01:51 -- STEP: 4612/36938 -- GLOBAL_STEP: 41550 | > loss_text_ce: 0.1157521978020668 (0.11363353069099476) | > loss_mel_ce: 4.423874378204346 (nan) | > loss: 0.018014391884207726 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13480025648760574) | > loader_time: 0.0043 (0.0044661626956407135)  --> TIME: 2025-02-23 22:02:00 -- STEP: 4662/36938 -- GLOBAL_STEP: 41600 | > loss_text_ce: 0.10168680548667908 (0.11360025473805221) | > loss_mel_ce: 4.800573825836182 (nan) | > loss: 0.019453417509794235 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.1347620508180879) | > loader_time: 0.0041 (0.004466976233686527)  --> TIME: 2025-02-23 22:02:09 -- STEP: 4712/36938 -- GLOBAL_STEP: 41650 | > loss_text_ce: 0.10153601318597794 (0.1135308866678455) | > loss_mel_ce: 4.249593734741211 (nan) | > loss: 0.01726638711988926 (nan) | > current_lr: 5e-06 | > step_time: 0.1556 (0.1347530800944073) | > loader_time: 0.0042 (0.004465782935833901)  --> TIME: 2025-02-23 22:02:18 -- STEP: 4762/36938 -- GLOBAL_STEP: 41700 | > loss_text_ce: 0.12091705948114395 (0.1135276405094603) | > loss_mel_ce: 4.344992160797119 (nan) | > loss: 0.017721861600875854 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13468934106806785) | > loader_time: 0.0044 (0.004466462065221291)  --> TIME: 2025-02-23 22:02:27 -- STEP: 4812/36938 -- GLOBAL_STEP: 41750 | > loss_text_ce: 0.10098829120397568 (0.11348127466583896) | > loss_mel_ce: 4.497447490692139 (nan) | > loss: 0.018247762694954872 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13462718347658867) | > loader_time: 0.0043 (0.004465134859283281)  --> TIME: 2025-02-23 22:02:36 -- STEP: 4862/36938 -- GLOBAL_STEP: 41800 | > loss_text_ce: 0.11877163499593735 (0.11343677688262836) | > loss_mel_ce: 4.709801197052002 (nan) | > loss: 0.019161004573106766 (nan) | > current_lr: 5e-06 | > step_time: 0.1634 (0.13457244004501068) | > loader_time: 0.0051 (0.004465440917240658)  --> TIME: 2025-02-23 22:02:45 -- STEP: 4912/36938 -- GLOBAL_STEP: 41850 | > loss_text_ce: 0.10061395168304443 (0.11337247401390974) | > loss_mel_ce: 4.009536266326904 (nan) | > loss: 0.01631012186408043 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13458360265637098) | > loader_time: 0.0048 (0.004466109487443493)  --> TIME: 2025-02-23 22:02:54 -- STEP: 4962/36938 -- GLOBAL_STEP: 41900 | > loss_text_ce: 0.10203162580728531 (0.11332412347113456) | > loss_mel_ce: 4.392656326293945 (nan) | > loss: 0.017836064100265503 (nan) | > current_lr: 5e-06 | > step_time: 0.179 (0.13456860856152736) | > loader_time: 0.0045 (0.004468260489851666)  --> TIME: 2025-02-23 22:03:04 -- STEP: 5012/36938 -- GLOBAL_STEP: 41950 | > loss_text_ce: 0.12379087507724762 (0.11324430288002202) | > loss_mel_ce: 3.477261543273926 (nan) | > loss: 0.014289891347289085 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13461039095807853) | > loader_time: 0.0044 (0.004468776660258553)  --> TIME: 2025-02-23 22:03:13 -- STEP: 5062/36938 -- GLOBAL_STEP: 42000 | > loss_text_ce: 0.11194995790719986 (0.11319627142421165) | > loss_mel_ce: 3.713226795196533 (nan) | > loss: 0.015179273672401905 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13457486884766384) | > loader_time: 0.0043 (0.004469382296622118)  --> TIME: 2025-02-23 22:03:22 -- STEP: 5112/36938 -- GLOBAL_STEP: 42050 | > loss_text_ce: 0.10777919739484787 (0.11314084219669054) | > loss_mel_ce: 4.243018627166748 (nan) | > loss: 0.017265070229768753 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13452672981693561) | > loader_time: 0.0044 (0.004469243293636853)  --> TIME: 2025-02-23 22:03:31 -- STEP: 5162/36938 -- GLOBAL_STEP: 42100 | > loss_text_ce: 0.12715266644954681 (0.1130884730815311) | > loss_mel_ce: 0.1182352676987648 (nan) | > loss: 0.0009737617219798267 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.1345170364967387) | > loader_time: 0.0044 (0.004469383275587409)  --> TIME: 2025-02-23 22:03:40 -- STEP: 5212/36938 -- GLOBAL_STEP: 42150 | > loss_text_ce: 0.10441364347934723 (0.113041152738761) | > loss_mel_ce: 3.301945447921753 (nan) | > loss: 0.013517298735678196 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.1344700361890056) | > loader_time: 0.0045 (0.0044698342852471816)  --> TIME: 2025-02-23 22:03:49 -- STEP: 5262/36938 -- GLOBAL_STEP: 42200 | > loss_text_ce: 0.11100353300571442 (0.11300311191556173) | > loss_mel_ce: 4.04644250869751 (nan) | > loss: 0.01649780198931694 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1344971444297774) | > loader_time: 0.0045 (0.004471017670060527)  --> TIME: 2025-02-23 22:03:59 -- STEP: 5312/36938 -- GLOBAL_STEP: 42250 | > loss_text_ce: 0.12829101085662842 (0.11295689421801286) | > loss_mel_ce: 2.6010773181915283 (nan) | > loss: 0.010830827057361603 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13451543239405384) | > loader_time: 0.0043 (0.004471215677548619)  --> TIME: 2025-02-23 22:04:08 -- STEP: 5362/36938 -- GLOBAL_STEP: 42300 | > loss_text_ce: 0.11512324213981628 (0.1128932616198544) | > loss_mel_ce: 3.887286424636841 (nan) | > loss: 0.015882577747106552 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.13453102783230184) | > loader_time: 0.0046 (0.004472162687080396)  --> TIME: 2025-02-23 22:04:17 -- STEP: 5412/36938 -- GLOBAL_STEP: 42350 | > loss_text_ce: 0.1102125346660614 (0.11284916685083277) | > loss_mel_ce: 3.9435532093048096 (nan) | > loss: 0.01608637347817421 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13449662298250467) | > loader_time: 0.0045 (0.0044722351777668245)  --> TIME: 2025-02-23 22:04:26 -- STEP: 5462/36938 -- GLOBAL_STEP: 42400 | > loss_text_ce: 0.09312454611063004 (0.11280729850677079) | > loss_mel_ce: 3.7433652877807617 (nan) | > loss: 0.01522416714578867 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1344948980407549) | > loader_time: 0.0046 (0.004473110687510095)  --> TIME: 2025-02-23 22:04:35 -- STEP: 5512/36938 -- GLOBAL_STEP: 42450 | > loss_text_ce: 0.12246432155370712 (0.11277040288511926) | > loss_mel_ce: 4.282793998718262 (nan) | > loss: 0.017481183633208275 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13450814958753732) | > loader_time: 0.0044 (0.004474147570323546)  --> TIME: 2025-02-23 22:04:45 -- STEP: 5562/36938 -- GLOBAL_STEP: 42500 | > loss_text_ce: 0.10858611762523651 (0.1126982506226611) | > loss_mel_ce: 2.4908814430236816 (nan) | > loss: 0.010315347462892532 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13448353462706197) | > loader_time: 0.0043 (0.0044740327010691534)  --> TIME: 2025-02-23 22:04:54 -- STEP: 5612/36938 -- GLOBAL_STEP: 42550 | > loss_text_ce: 0.11604058742523193 (0.11264281589816666) | > loss_mel_ce: 4.635661602020264 (nan) | > loss: 0.01885596290230751 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.13449938027075325) | > loader_time: 0.0045 (0.004474112372354207)  --> TIME: 2025-02-23 22:05:03 -- STEP: 5662/36938 -- GLOBAL_STEP: 42600 | > loss_text_ce: 0.09848759323358536 (0.11259531010638409) | > loss_mel_ce: 4.724608421325684 (nan) | > loss: 0.01913926936686039 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13451764348621953) | > loader_time: 0.0044 (0.004474320036073929)  --> TIME: 2025-02-23 22:05:12 -- STEP: 5712/36938 -- GLOBAL_STEP: 42650 | > loss_text_ce: 0.1076580137014389 (0.1125244211669) | > loss_mel_ce: 4.999374866485596 (nan) | > loss: 0.020266003906726837 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13451471828779904) | > loader_time: 0.0044 (0.004474201205731789)  --> TIME: 2025-02-23 22:05:22 -- STEP: 5762/36938 -- GLOBAL_STEP: 42700 | > loss_text_ce: 0.10966434329748154 (0.11246819969645526) | > loss_mel_ce: 3.6057958602905273 (nan) | > loss: 0.01474389061331749 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13457538025613364) | > loader_time: 0.0046 (0.004474745612722122)  --> TIME: 2025-02-23 22:05:32 -- STEP: 5812/36938 -- GLOBAL_STEP: 42750 | > loss_text_ce: 0.10822321474552155 (0.11241253463619352) | > loss_mel_ce: 4.590144634246826 (nan) | > loss: 0.018644317984580994 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.1346120097189217) | > loader_time: 0.0047 (0.004475303296786886)  --> TIME: 2025-02-23 22:05:41 -- STEP: 5862/36938 -- GLOBAL_STEP: 42800 | > loss_text_ce: 0.11642400175333023 (0.11236759977804772) | > loss_mel_ce: 2.448256254196167 (nan) | > loss: 0.010177303105592728 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.1346155847147924) | > loader_time: 0.0046 (0.00447576910674228)  --> TIME: 2025-02-23 22:05:50 -- STEP: 5912/36938 -- GLOBAL_STEP: 42850 | > loss_text_ce: 0.11118059605360031 (0.11229383416287357) | > loss_mel_ce: 4.1038970947265625 (nan) | > loss: 0.016726501286029816 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13460794982471372) | > loader_time: 0.0046 (0.004476272850139881)  --> TIME: 2025-02-23 22:05:59 -- STEP: 5962/36938 -- GLOBAL_STEP: 42900 | > loss_text_ce: 0.11898212134838104 (0.11227272497037048) | > loss_mel_ce: 4.4142374992370605 (nan) | > loss: 0.017988968640565872 (nan) | > current_lr: 5e-06 | > step_time: 0.1064 (0.1346444106269945) | > loader_time: 0.0044 (0.004475806152128785)  --> TIME: 2025-02-23 22:06:09 -- STEP: 6012/36938 -- GLOBAL_STEP: 42950 | > loss_text_ce: 0.10979914665222168 (0.11221651750147668) | > loss_mel_ce: 4.348592281341553 (nan) | > loss: 0.01769202947616577 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.1346426302008206) | > loader_time: 0.0044 (0.004475170623756477)  --> TIME: 2025-02-23 22:06:17 -- STEP: 6062/36938 -- GLOBAL_STEP: 43000 | > loss_text_ce: 0.09967827796936035 (0.1122056914537074) | > loss_mel_ce: 4.148372173309326 (nan) | > loss: 0.016857344657182693 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13458903572890513) | > loader_time: 0.0042 (0.00447481105682365)  --> TIME: 2025-02-23 22:06:27 -- STEP: 6112/36938 -- GLOBAL_STEP: 43050 | > loss_text_ce: 0.10628586262464523 (0.11214870994383994) | > loss_mel_ce: 3.2528035640716553 (nan) | > loss: 0.013329720124602318 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13461573199607943) | > loader_time: 0.0048 (0.004474939203075111)  --> TIME: 2025-02-23 22:06:37 -- STEP: 6162/36938 -- GLOBAL_STEP: 43100 | > loss_text_ce: 0.12234371155500412 (0.11207503031640624) | > loss_mel_ce: 0.07403564453125 (nan) | > loss: 0.000779283232986927 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13467032171928503) | > loader_time: 0.0045 (0.004475334796453621)  --> TIME: 2025-02-23 22:06:46 -- STEP: 6212/36938 -- GLOBAL_STEP: 43150 | > loss_text_ce: 0.11695706099271774 (0.11201808256914526) | > loss_mel_ce: 4.434606075286865 (nan) | > loss: 0.018061760812997818 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13468011371258828) | > loader_time: 0.0044 (0.00447619448764359)  --> TIME: 2025-02-23 22:06:55 -- STEP: 6262/36938 -- GLOBAL_STEP: 43200 | > loss_text_ce: 0.17670312523841858 (0.11197333840617019) | > loss_mel_ce: 7.84375 (nan) | > loss: 0.03182719647884369 (nan) | > current_lr: 5e-06 | > step_time: 0.1076 (0.13464161333890046) | > loader_time: 0.0043 (0.004476320435162464)  --> TIME: 2025-02-23 22:07:04 -- STEP: 6312/36938 -- GLOBAL_STEP: 43250 | > loss_text_ce: 0.10745617747306824 (0.11192783667541888) | > loss_mel_ce: 2.203369140625 (nan) | > loss: 0.009169942699372768 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13464690561681333) | > loader_time: 0.0044 (0.0044777721582106005)  --> TIME: 2025-02-23 22:07:13 -- STEP: 6362/36938 -- GLOBAL_STEP: 43300 | > loss_text_ce: 0.11342738568782806 (0.11187723929666199) | > loss_mel_ce: 4.0231781005859375 (nan) | > loss: 0.016415100544691086 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13464251680592992) | > loader_time: 0.0044 (0.004478289997828783)  --> TIME: 2025-02-23 22:07:22 -- STEP: 6412/36938 -- GLOBAL_STEP: 43350 | > loss_text_ce: 0.09542731195688248 (0.11183517112054463) | > loss_mel_ce: 4.679311275482178 (nan) | > loss: 0.018947377800941467 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13465666428849757) | > loader_time: 0.0043 (0.004477886988532178)  --> TIME: 2025-02-23 22:07:31 -- STEP: 6462/36938 -- GLOBAL_STEP: 43400 | > loss_text_ce: 0.11377464234828949 (0.11178820236654932) | > loss_mel_ce: 4.814139366149902 (nan) | > loss: 0.019555216655135155 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13462771210128843) | > loader_time: 0.0044 (0.004477090453334166)  --> TIME: 2025-02-23 22:07:41 -- STEP: 6512/36938 -- GLOBAL_STEP: 43450 | > loss_text_ce: 0.09983813017606735 (0.11172756472721483) | > loss_mel_ce: 4.408026695251465 (nan) | > loss: 0.017888354137539864 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13464265281943202) | > loader_time: 0.0045 (0.004477117915411285)  --> TIME: 2025-02-23 22:07:50 -- STEP: 6562/36938 -- GLOBAL_STEP: 43500 | > loss_text_ce: 0.0920398086309433 (0.11165773169827592) | > loss_mel_ce: 4.43570613861084 (nan) | > loss: 0.01796724647283554 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.13467019634688646) | > loader_time: 0.0044 (0.004477748381077853)  --> TIME: 2025-02-23 22:08:00 -- STEP: 6612/36938 -- GLOBAL_STEP: 43550 | > loss_text_ce: 0.10160975903272629 (0.11161193780767802) | > loss_mel_ce: 3.7995870113372803 (nan) | > loss: 0.015480940230190754 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13468685391017052) | > loader_time: 0.0045 (0.004477427897133131)  --> TIME: 2025-02-23 22:08:09 -- STEP: 6662/36938 -- GLOBAL_STEP: 43600 | > loss_text_ce: 0.10926350206136703 (0.11156407148719592) | > loss_mel_ce: 4.291901588439941 (nan) | > loss: 0.01746494136750698 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13466170198457186) | > loader_time: 0.0045 (0.004476785373487356)  --> TIME: 2025-02-23 22:08:18 -- STEP: 6712/36938 -- GLOBAL_STEP: 43650 | > loss_text_ce: 0.09113587439060211 (0.11152012232930825) | > loss_mel_ce: 4.41063117980957 (nan) | > loss: 0.017864156514406204 (nan) | > current_lr: 5e-06 | > step_time: 0.1081 (0.13465829046178948) | > loader_time: 0.0043 (0.004475845270418319)  --> TIME: 2025-02-23 22:08:27 -- STEP: 6762/36938 -- GLOBAL_STEP: 43700 | > loss_text_ce: 0.09460911899805069 (0.11146659925315593) | > loss_mel_ce: 4.018539905548096 (nan) | > loss: 0.016322022303938866 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.1346897473528486) | > loader_time: 0.0043 (0.0044758338541788754)  --> TIME: 2025-02-23 22:08:36 -- STEP: 6812/36938 -- GLOBAL_STEP: 43750 | > loss_text_ce: 0.10278508812189102 (0.11142966665187556) | > loss_mel_ce: 4.18598747253418 (nan) | > loss: 0.017018940299749374 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.1346523197552632) | > loader_time: 0.0046 (0.004475147109555301)  --> TIME: 2025-02-23 22:08:46 -- STEP: 6862/36938 -- GLOBAL_STEP: 43800 | > loss_text_ce: 0.10173499584197998 (0.11137972546063307) | > loss_mel_ce: 4.804670333862305 (nan) | > loss: 0.019469864666461945 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.13465200986086695) | > loader_time: 0.0047 (0.004474477738758759)  --> TIME: 2025-02-23 22:08:55 -- STEP: 6912/36938 -- GLOBAL_STEP: 43850 | > loss_text_ce: 0.09865451604127884 (0.11132973531493924) | > loss_mel_ce: 3.817943811416626 (nan) | > loss: 0.01554205734282732 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13464635196659305) | > loader_time: 0.0044 (0.004474147395403324)  --> TIME: 2025-02-23 22:09:04 -- STEP: 6962/36938 -- GLOBAL_STEP: 43900 | > loss_text_ce: 0.09492019563913345 (0.11128542448119239) | > loss_mel_ce: 4.318737030029297 (nan) | > loss: 0.017514513805508614 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.134639712422444) | > loader_time: 0.0042 (0.004473069144952806)  --> TIME: 2025-02-23 22:09:13 -- STEP: 7012/36938 -- GLOBAL_STEP: 43950 | > loss_text_ce: 0.11158037930727005 (0.11121585568613966) | > loss_mel_ce: 2.407743453979492 (nan) | > loss: 0.00999731756746769 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.134640276874057) | > loader_time: 0.0042 (0.004472160774574807)  --> TIME: 2025-02-23 22:09:22 -- STEP: 7062/36938 -- GLOBAL_STEP: 44000 | > loss_text_ce: 0.11839386820793152 (0.11117939165384808) | > loss_mel_ce: 4.348763942718506 (nan) | > loss: 0.017726818099617958 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.13463061276524693) | > loader_time: 0.0046 (0.00447209027899931)  --> TIME: 2025-02-23 22:09:31 -- STEP: 7112/36938 -- GLOBAL_STEP: 44050 | > loss_text_ce: 0.10547219961881638 (0.1111388040677183) | > loss_mel_ce: 3.2291955947875977 (nan) | > loss: 0.013232809491455555 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13461594767994225) | > loader_time: 0.0043 (0.004472489264470005)  --> TIME: 2025-02-23 22:09:40 -- STEP: 7162/36938 -- GLOBAL_STEP: 44100 | > loss_text_ce: 0.1064809262752533 (0.11106782639490202) | > loss_mel_ce: 1.8106281757354736 (nan) | > loss: 0.007607576437294483 (nan) | > current_lr: 5e-06 | > step_time: 0.1082 (0.13459121132323068) | > loader_time: 0.0044 (0.004472227211073126)  --> TIME: 2025-02-23 22:09:49 -- STEP: 7212/36938 -- GLOBAL_STEP: 44150 | > loss_text_ce: 0.11929311603307724 (0.11103384829623733) | > loss_mel_ce: 4.077077865600586 (nan) | > loss: 0.016652267426252365 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13457058792436888) | > loader_time: 0.0045 (0.00447201950313382)  --> TIME: 2025-02-23 22:09:58 -- STEP: 7262/36938 -- GLOBAL_STEP: 44200 | > loss_text_ce: 0.09870314598083496 (0.11097824934247248) | > loss_mel_ce: 4.374246120452881 (nan) | > loss: 0.01774979941546917 (nan) | > current_lr: 5e-06 | > step_time: 0.1767 (0.1345733608368242) | > loader_time: 0.0044 (0.004472208627161348)  --> TIME: 2025-02-23 22:10:08 -- STEP: 7312/36938 -- GLOBAL_STEP: 44250 | > loss_text_ce: 0.11977328360080719 (0.11091284139623925) | > loss_mel_ce: 3.9787325859069824 (nan) | > loss: 0.016263913363218307 (nan) | > current_lr: 5e-06 | > step_time: 0.1074 (0.1346047111100126) | > loader_time: 0.0044 (0.004471963161451833)  --> TIME: 2025-02-23 22:10:17 -- STEP: 7362/36938 -- GLOBAL_STEP: 44300 | > loss_text_ce: 0.10411860048770905 (0.1108602749604274) | > loss_mel_ce: 4.3158135414123535 (nan) | > loss: 0.017539415508508682 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.1346210699889748) | > loader_time: 0.0042 (0.004472037723160951)  --> TIME: 2025-02-23 22:10:27 -- STEP: 7412/36938 -- GLOBAL_STEP: 44350 | > loss_text_ce: 0.11290895938873291 (0.11082443879807703) | > loss_mel_ce: 4.102719783782959 (nan) | > loss: 0.016728686168789864 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13462815169572684) | > loader_time: 0.0043 (0.004471555923167547)  --> TIME: 2025-02-23 22:10:36 -- STEP: 7462/36938 -- GLOBAL_STEP: 44400 | > loss_text_ce: 0.10460546612739563 (0.1107904589833571) | > loss_mel_ce: 2.5687255859375 (nan) | > loss: 0.010608457028865814 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1346506753896969) | > loader_time: 0.0044 (0.004472238836643769)  --> TIME: 2025-02-23 22:10:45 -- STEP: 7512/36938 -- GLOBAL_STEP: 44450 | > loss_text_ce: 0.08958422392606735 (0.1107292858828197) | > loss_mel_ce: 4.318150520324707 (nan) | > loss: 0.017491012811660767 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.1346407538404859) | > loader_time: 0.0043 (0.004472819062721509)  --> TIME: 2025-02-23 22:10:55 -- STEP: 7562/36938 -- GLOBAL_STEP: 44500 | > loss_text_ce: 0.09524407237768173 (0.11069338583096541) | > loss_mel_ce: 3.803969383239746 (nan) | > loss: 0.015473070554435253 (nan) | > current_lr: 5e-06 | > step_time: 0.108 (0.13465267134482053) | > loader_time: 0.0042 (0.004473423396617281)  --> TIME: 2025-02-23 22:11:03 -- STEP: 7612/36938 -- GLOBAL_STEP: 44550 | > loss_text_ce: 0.08506669849157333 (0.1106496486231427) | > loss_mel_ce: 4.833166599273682 (nan) | > loss: 0.019516799598932266 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13460381123121307) | > loader_time: 0.0044 (0.0044731712629465775)  --> TIME: 2025-02-23 22:11:12 -- STEP: 7662/36938 -- GLOBAL_STEP: 44600 | > loss_text_ce: 0.09732307493686676 (0.11059545994940709) | > loss_mel_ce: 3.466282844543457 (nan) | > loss: 0.014141294173896313 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13460501616942713) | > loader_time: 0.0046 (0.004473137376327754)  --> TIME: 2025-02-23 22:11:22 -- STEP: 7712/36938 -- GLOBAL_STEP: 44650 | > loss_text_ce: 0.16208983957767487 (0.11055281314191209) | > loss_mel_ce: 2.6134440898895264 (nan) | > loss: 0.011014023795723915 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13459547068073524) | > loader_time: 0.0043 (0.004472811872533763)  --> TIME: 2025-02-23 22:11:31 -- STEP: 7762/36938 -- GLOBAL_STEP: 44700 | > loss_text_ce: 0.09500255435705185 (0.11052061686965652) | > loss_mel_ce: 4.976334095001221 (nan) | > loss: 0.02012435346841812 (nan) | > current_lr: 5e-06 | > step_time: 0.1712 (0.13458013193596868) | > loader_time: 0.0071 (0.004473000357858137)  --> TIME: 2025-02-23 22:11:40 -- STEP: 7812/36938 -- GLOBAL_STEP: 44750 | > loss_text_ce: 0.08923399448394775 (0.11048477727784392) | > loss_mel_ce: 3.9468934535980225 (nan) | > loss: 0.016016380861401558 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.134603212170276) | > loader_time: 0.0043 (0.0044739347388057255)  --> TIME: 2025-02-23 22:11:49 -- STEP: 7862/36938 -- GLOBAL_STEP: 44800 | > loss_text_ce: 0.10426955670118332 (0.11045002543844366) | > loss_mel_ce: 3.974888324737549 (nan) | > loss: 0.016187135130167007 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13459311081113096) | > loader_time: 0.0043 (0.004474022315070716)  --> TIME: 2025-02-23 22:11:58 -- STEP: 7912/36938 -- GLOBAL_STEP: 44850 | > loss_text_ce: 0.10337706655263901 (0.11039022194490943) | > loss_mel_ce: 2.4360392093658447 (nan) | > loss: 0.010077049024403095 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.13459553968074692) | > loader_time: 0.0045 (0.004474022300467593)  --> TIME: 2025-02-23 22:12:08 -- STEP: 7962/36938 -- GLOBAL_STEP: 44900 | > loss_text_ce: 0.08939976245164871 (0.11033832641536948) | > loss_mel_ce: 4.580514430999756 (nan) | > loss: 0.018531406298279762 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.13460882974910648) | > loader_time: 0.0046 (0.0044738952911729775)  --> TIME: 2025-02-23 22:12:17 -- STEP: 8012/36938 -- GLOBAL_STEP: 44950 | > loss_text_ce: 0.10094432532787323 (0.11030416288731613) | > loss_mel_ce: 4.652393341064453 (nan) | > loss: 0.01886245235800743 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13465674754922158) | > loader_time: 0.0045 (0.0044744521215803745)  --> TIME: 2025-02-23 22:12:27 -- STEP: 8062/36938 -- GLOBAL_STEP: 45000 | > loss_text_ce: 0.09239067137241364 (0.11026201140633654) | > loss_mel_ce: 4.052743434906006 (nan) | > loss: 0.016448944807052612 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.13469465386332233) | > loader_time: 0.0046 (0.004475506681036774)  --> TIME: 2025-02-23 22:12:36 -- STEP: 8112/36938 -- GLOBAL_STEP: 45050 | > loss_text_ce: 0.10167116671800613 (0.1102051672000343) | > loss_mel_ce: 3.7966854572296143 (nan) | > loss: 0.015469670295715332 (nan) | > current_lr: 5e-06 | > step_time: 0.1508 (0.13466992656860108) | > loader_time: 0.0048 (0.004476045216093892)  --> TIME: 2025-02-23 22:12:45 -- STEP: 8162/36938 -- GLOBAL_STEP: 45100 | > loss_text_ce: 0.09655201435089111 (0.11017121673926546) | > loss_mel_ce: 4.382824897766113 (nan) | > loss: 0.01777530647814274 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.13468454433872062) | > loader_time: 0.0046 (0.004475911292805466)  --> TIME: 2025-02-23 22:12:54 -- STEP: 8212/36938 -- GLOBAL_STEP: 45150 | > loss_text_ce: 0.10255169868469238 (0.11014898514571976) | > loss_mel_ce: 3.3848862648010254 (nan) | > loss: 0.013839039951562881 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.1346679021542209) | > loader_time: 0.0043 (0.00447561815780255)  --> TIME: 2025-02-23 22:13:04 -- STEP: 8262/36938 -- GLOBAL_STEP: 45200 | > loss_text_ce: 0.09050297737121582 (0.11008850213647962) | > loss_mel_ce: 3.9030065536499023 (nan) | > loss: 0.015847260132431984 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.13469635808303362) | > loader_time: 0.0043 (0.004476012487556959)  --> TIME: 2025-02-23 22:13:13 -- STEP: 8312/36938 -- GLOBAL_STEP: 45250 | > loss_text_ce: 0.09917516261339188 (0.11005870757769723) | > loss_mel_ce: 4.305769920349121 (nan) | > loss: 0.017479941248893738 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13468789999822323) | > loader_time: 0.0046 (0.004476367566766841)  --> TIME: 2025-02-23 22:13:22 -- STEP: 8362/36938 -- GLOBAL_STEP: 45300 | > loss_text_ce: 0.09498543292284012 (0.11001955134104928) | > loss_mel_ce: 4.03314208984375 (nan) | > loss: 0.016381459310650826 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13467415335417296) | > loader_time: 0.0047 (0.0044760412076135046)  --> TIME: 2025-02-23 22:13:31 -- STEP: 8412/36938 -- GLOBAL_STEP: 45350 | > loss_text_ce: 0.09404400736093521 (0.10996634525096792) | > loss_mel_ce: 4.2908735275268555 (nan) | > loss: 0.0174004677683115 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.1346807925121816) | > loader_time: 0.0045 (0.004476764176042679)  --> TIME: 2025-02-23 22:13:41 -- STEP: 8462/36938 -- GLOBAL_STEP: 45400 | > loss_text_ce: 0.12684474885463715 (0.1099362486317011) | > loss_mel_ce: 0.4343428313732147 (nan) | > loss: 0.002226935001090169 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.13468416158089644) | > loader_time: 0.0043 (0.0044772766127256355)  --> TIME: 2025-02-23 22:13:49 -- STEP: 8512/36938 -- GLOBAL_STEP: 45450 | > loss_text_ce: 0.10603579878807068 (0.1098997557896393) | > loss_mel_ce: 5.51953125 (nan) | > loss: 0.02232367917895317 (nan) | > current_lr: 5e-06 | > step_time: 0.1079 (0.13464353171954468) | > loader_time: 0.0042 (0.004477072199036324)  --> TIME: 2025-02-23 22:13:59 -- STEP: 8562/36938 -- GLOBAL_STEP: 45500 | > loss_text_ce: 0.09881094843149185 (0.10984475175699775) | > loss_mel_ce: 3.0096938610076904 (nan) | > loss: 0.01233533676713705 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13465855158369436) | > loader_time: 0.0045 (0.004477149358115409)  --> TIME: 2025-02-23 22:14:08 -- STEP: 8612/36938 -- GLOBAL_STEP: 45550 | > loss_text_ce: 0.09719520062208176 (0.10978390657525955) | > loss_mel_ce: 3.3037667274475098 (nan) | > loss: 0.013495881110429764 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13467937954292697) | > loader_time: 0.0044 (0.004477495378635141)  --> TIME: 2025-02-23 22:14:18 -- STEP: 8662/36938 -- GLOBAL_STEP: 45600 | > loss_text_ce: 0.09695081412792206 (0.10973176974102204) | > loss_mel_ce: 4.052096843719482 (nan) | > loss: 0.01646447740495205 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.13469890977295346) | > loader_time: 0.0045 (0.004477390101258202)  --> TIME: 2025-02-23 22:14:27 -- STEP: 8712/36938 -- GLOBAL_STEP: 45650 | > loss_text_ce: 0.09463003277778625 (0.10968317730990282) | > loss_mel_ce: 3.947847604751587 (nan) | > loss: 0.016041578724980354 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13469258987980215) | > loader_time: 0.0045 (0.004477276618158869)  --> TIME: 2025-02-23 22:14:36 -- STEP: 8762/36938 -- GLOBAL_STEP: 45700 | > loss_text_ce: 0.09878449141979218 (0.10962419585924103) | > loss_mel_ce: 3.824232339859009 (nan) | > loss: 0.015567528083920479 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13467123551445978) | > loader_time: 0.0042 (0.004477305081412574)  --> TIME: 2025-02-23 22:14:45 -- STEP: 8812/36938 -- GLOBAL_STEP: 45750 | > loss_text_ce: 0.10088502615690231 (0.10958445886563463) | > loss_mel_ce: 4.383784770965576 (nan) | > loss: 0.017796309664845467 (nan) | > current_lr: 5e-06 | > step_time: 0.1501 (0.13466997234268271) | > loader_time: 0.0043 (0.0044772915552271945)  --> TIME: 2025-02-23 22:14:55 -- STEP: 8862/36938 -- GLOBAL_STEP: 45800 | > loss_text_ce: 0.09365533292293549 (0.10953158993611121) | > loss_mel_ce: 3.189805746078491 (nan) | > loss: 0.013029607944190502 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.1346998948110356) | > loader_time: 0.0045 (0.004476842183987156)  --> TIME: 2025-02-23 22:15:04 -- STEP: 8912/36938 -- GLOBAL_STEP: 45850 | > loss_text_ce: 0.11003348231315613 (0.10948352134891895) | > loss_mel_ce: 1.1608446836471558 (nan) | > loss: 0.005043167620897293 (nan) | > current_lr: 5e-06 | > step_time: 0.1067 (0.13470451953809154) | > loader_time: 0.0043 (0.004476548177549629)  --> TIME: 2025-02-23 22:15:13 -- STEP: 8962/36938 -- GLOBAL_STEP: 45900 | > loss_text_ce: 0.09510985761880875 (0.10943195203772704) | > loss_mel_ce: 4.4761810302734375 (nan) | > loss: 0.018140044063329697 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.1346940859384074) | > loader_time: 0.0042 (0.00447626282556387)  --> TIME: 2025-02-23 22:15:23 -- STEP: 9012/36938 -- GLOBAL_STEP: 45950 | > loss_text_ce: 0.111149363219738 (0.10938835136436971) | > loss_mel_ce: 0.14270630478858948 (nan) | > loss: 0.001007363898679614 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13472362536617868) | > loader_time: 0.0044 (0.0044758691821583175)  --> TIME: 2025-02-23 22:15:32 -- STEP: 9062/36938 -- GLOBAL_STEP: 46000 | > loss_text_ce: 0.09981605410575867 (0.10935240949431531) | > loss_mel_ce: 3.8918685913085938 (nan) | > loss: 0.015840020030736923 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.13473921094712127) | > loader_time: 0.0043 (0.004475470963652597)  --> TIME: 2025-02-23 22:15:41 -- STEP: 9112/36938 -- GLOBAL_STEP: 46050 | > loss_text_ce: 0.11253493279218674 (0.10930853952098525) | > loss_mel_ce: 2.4174935817718506 (nan) | > loss: 0.010039796121418476 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13473078914647346) | > loader_time: 0.0044 (0.004475412214922238)  --> TIME: 2025-02-23 22:15:50 -- STEP: 9162/36938 -- GLOBAL_STEP: 46100 | > loss_text_ce: 0.09134627878665924 (0.10926564267855435) | > loss_mel_ce: 3.958401679992676 (nan) | > loss: 0.016070429235696793 (nan) | > current_lr: 5e-06 | > step_time: 0.1651 (0.13474582041020305) | > loader_time: 0.0044 (0.004475300787213976)  --> TIME: 2025-02-23 22:16:00 -- STEP: 9212/36938 -- GLOBAL_STEP: 46150 | > loss_text_ce: 0.09609124064445496 (0.10923392077826245) | > loss_mel_ce: 4.124724388122559 (nan) | > loss: 0.016749270260334015 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13477122379912512) | > loader_time: 0.0042 (0.0044755730947825196)  --> TIME: 2025-02-23 22:16:09 -- STEP: 9262/36938 -- GLOBAL_STEP: 46200 | > loss_text_ce: 0.10041478276252747 (0.10918745806445147) | > loss_mel_ce: 4.195605278015137 (nan) | > loss: 0.0170476995408535 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13475676387052893) | > loader_time: 0.0044 (0.004475130604451763)  --> TIME: 2025-02-23 22:16:18 -- STEP: 9312/36938 -- GLOBAL_STEP: 46250 | > loss_text_ce: 0.09328585118055344 (0.10913992450409343) | > loss_mel_ce: 4.7943830490112305 (nan) | > loss: 0.019395513460040092 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.134742990187353) | > loader_time: 0.0045 (0.004474701340665542)  --> TIME: 2025-02-23 22:16:27 -- STEP: 9362/36938 -- GLOBAL_STEP: 46300 | > loss_text_ce: 0.09476141631603241 (0.10910476992474605) | > loss_mel_ce: 4.5201334953308105 (nan) | > loss: 0.018313076347112656 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.13474142902861563) | > loader_time: 0.0045 (0.004474179837928341)  --> TIME: 2025-02-23 22:16:37 -- STEP: 9412/36938 -- GLOBAL_STEP: 46350 | > loss_text_ce: 0.09551957994699478 (0.10905589250617355) | > loss_mel_ce: 4.6851630210876465 (nan) | > loss: 0.01897096447646618 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.13475384367206347) | > loader_time: 0.0045 (0.004474107453127007)  --> TIME: 2025-02-23 22:16:46 -- STEP: 9462/36938 -- GLOBAL_STEP: 46400 | > loss_text_ce: 0.08273850381374359 (0.10901478584914866) | > loss_mel_ce: 3.6156723499298096 (nan) | > loss: 0.014676233753561974 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13478675482767452) | > loader_time: 0.0045 (0.00447439734161192)  --> TIME: 2025-02-23 22:16:56 -- STEP: 9512/36938 -- GLOBAL_STEP: 46450 | > loss_text_ce: 0.07783012837171555 (0.10896763912127176) | > loss_mel_ce: 4.964099884033203 (nan) | > loss: 0.02000766061246395 (nan) | > current_lr: 5e-06 | > step_time: 0.1071 (0.13479637931734673) | > loader_time: 0.0043 (0.00447476434045927)  --> TIME: 2025-02-23 22:17:05 -- STEP: 9562/36938 -- GLOBAL_STEP: 46500 | > loss_text_ce: 0.09784866124391556 (0.1089190479957817) | > loss_mel_ce: 3.849238634109497 (nan) | > loss: 0.01566304638981819 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.13479174405205435) | > loader_time: 0.0044 (0.004475047338510075)  --> TIME: 2025-02-23 22:17:14 -- STEP: 9612/36938 -- GLOBAL_STEP: 46550 | > loss_text_ce: 0.10297192633152008 (0.10887532499845569) | > loss_mel_ce: 4.27529764175415 (nan) | > loss: 0.017374087125062943 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13480984494927176) | > loader_time: 0.0043 (0.004474798912710341)  --> TIME: 2025-02-23 22:17:23 -- STEP: 9662/36938 -- GLOBAL_STEP: 46600 | > loss_text_ce: 0.09923049807548523 (0.10882835412258036) | > loss_mel_ce: 3.458773612976074 (nan) | > loss: 0.014119064435362816 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1348095217789456) | > loader_time: 0.0043 (0.004474350789693368)  --> TIME: 2025-02-23 22:17:33 -- STEP: 9712/36938 -- GLOBAL_STEP: 46650 | > loss_text_ce: 0.10362010449171066 (0.10878388390951547) | > loss_mel_ce: 4.509125709533691 (nan) | > loss: 0.018304547294974327 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.13482162525763763) | > loader_time: 0.0044 (0.00447377292485373)  --> TIME: 2025-02-23 22:17:42 -- STEP: 9762/36938 -- GLOBAL_STEP: 46700 | > loss_text_ce: 0.12873779237270355 (0.10873948025718098) | > loss_mel_ce: 1.7538213729858398 (nan) | > loss: 0.0074704731814563274 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13479380714089767) | > loader_time: 0.0044 (0.004473480160327724)  --> TIME: 2025-02-23 22:17:51 -- STEP: 9812/36938 -- GLOBAL_STEP: 46750 | > loss_text_ce: 0.09888119250535965 (0.10869777339362184) | > loss_mel_ce: 3.2970008850097656 (nan) | > loss: 0.013475723564624786 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13483544526856275) | > loader_time: 0.0043 (0.004474447981659095)  --> TIME: 2025-02-23 22:18:01 -- STEP: 9862/36938 -- GLOBAL_STEP: 46800 | > loss_text_ce: 0.09987866878509521 (0.10865448582316856) | > loss_mel_ce: 4.169680595397949 (nan) | > loss: 0.016942696645855904 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13485643726361196) | > loader_time: 0.0043 (0.004474114535232534)  --> TIME: 2025-02-23 22:18:10 -- STEP: 9912/36938 -- GLOBAL_STEP: 46850 | > loss_text_ce: 0.08912826329469681 (0.10859782000082085) | > loss_mel_ce: 4.590346336364746 (nan) | > loss: 0.018569346517324448 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13487738813480263) | > loader_time: 0.0043 (0.0044737020454838)  --> TIME: 2025-02-23 22:18:19 -- STEP: 9962/36938 -- GLOBAL_STEP: 46900 | > loss_text_ce: 0.0959065780043602 (0.10854850234372207) | > loss_mel_ce: 4.51653528213501 (nan) | > loss: 0.018303342163562775 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13485680735032207) | > loader_time: 0.0044 (0.004473747486236295)  --> TIME: 2025-02-23 22:18:28 -- STEP: 10012/36938 -- GLOBAL_STEP: 46950 | > loss_text_ce: 0.11990274488925934 (0.10850638590002974) | > loss_mel_ce: 0.9945591688156128 (nan) | > loss: 0.0044224681332707405 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13484290819667166) | > loader_time: 0.0044 (0.004473493854569973)  --> TIME: 2025-02-23 22:18:38 -- STEP: 10062/36938 -- GLOBAL_STEP: 47000 | > loss_text_ce: 0.09961698204278946 (0.10846076402579963) | > loss_mel_ce: 4.6326141357421875 (nan) | > loss: 0.018778696656227112 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.1348556076596488) | > loader_time: 0.0044 (0.004473262434094855)  --> TIME: 2025-02-23 22:18:47 -- STEP: 10112/36938 -- GLOBAL_STEP: 47050 | > loss_text_ce: 0.109551340341568 (0.10841288666986532) | > loss_mel_ce: 3.3773345947265625 (nan) | > loss: 0.01383685041218996 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13486944898208417) | > loader_time: 0.0045 (0.004473165243486842)  --> TIME: 2025-02-23 22:18:56 -- STEP: 10162/36938 -- GLOBAL_STEP: 47100 | > loss_text_ce: 0.11738070845603943 (0.1083534132015446) | > loss_mel_ce: 4.574234485626221 (nan) | > loss: 0.018617521971464157 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13486133056277183) | > loader_time: 0.0043 (0.004472843119300909)  --> TIME: 2025-02-23 22:19:05 -- STEP: 10212/36938 -- GLOBAL_STEP: 47150 | > loss_text_ce: 0.12496386468410492 (0.10830494176627059) | > loss_mel_ce: 3.0357115268707275 (nan) | > loss: 0.01254236325621605 (nan) | > current_lr: 5e-06 | > step_time: 0.1062 (0.13484201137570304) | > loader_time: 0.0044 (0.0044732558647885175)  --> TIME: 2025-02-23 22:19:15 -- STEP: 10262/36938 -- GLOBAL_STEP: 47200 | > loss_text_ce: 0.09619052708148956 (0.10825844823187324) | > loss_mel_ce: 2.2304725646972656 (nan) | > loss: 0.00923279020935297 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.1348386081285903) | > loader_time: 0.0043 (0.00447327324903123)  --> TIME: 2025-02-23 22:19:24 -- STEP: 10312/36938 -- GLOBAL_STEP: 47250 | > loss_text_ce: 0.11561858654022217 (0.10820373125400341) | > loss_mel_ce: 4.746006965637207 (nan) | > loss: 0.019292166456580162 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1348107553804626) | > loader_time: 0.0045 (0.004473152943967041)  --> TIME: 2025-02-23 22:19:33 -- STEP: 10362/36938 -- GLOBAL_STEP: 47300 | > loss_text_ce: 0.09619666635990143 (0.1081669334731343) | > loss_mel_ce: 4.4035844802856445 (nan) | > loss: 0.017856275662779808 (nan) | > current_lr: 5e-06 | > step_time: 0.1965 (0.13482481410341796) | > loader_time: 0.0049 (0.004473922404974202)  --> TIME: 2025-02-23 22:19:42 -- STEP: 10412/36938 -- GLOBAL_STEP: 47350 | > loss_text_ce: 0.09334322065114975 (0.10813204492748232) | > loss_mel_ce: 4.261200428009033 (nan) | > loss: 0.017279936000704765 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.1348149382147561) | > loader_time: 0.0045 (0.004474516080701737)  --> TIME: 2025-02-23 22:19:51 -- STEP: 10462/36938 -- GLOBAL_STEP: 47400 | > loss_text_ce: 0.09951792657375336 (0.10808053246660652) | > loss_mel_ce: 4.562315464019775 (nan) | > loss: 0.018499338999390602 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.13481284942619587) | > loader_time: 0.0045 (0.0044748659139380725)  --> TIME: 2025-02-23 22:20:01 -- STEP: 10512/36938 -- GLOBAL_STEP: 47450 | > loss_text_ce: 0.08971050381660461 (0.10802926771524213) | > loss_mel_ce: 4.714879989624023 (nan) | > loss: 0.01906583644449711 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1348158609422549) | > loader_time: 0.0046 (0.004475527203427803)  --> TIME: 2025-02-23 22:20:10 -- STEP: 10562/36938 -- GLOBAL_STEP: 47500 | > loss_text_ce: 0.1056000143289566 (0.10798266589785488) | > loss_mel_ce: 4.012498378753662 (nan) | > loss: 0.016341660171747208 (nan) | > current_lr: 5e-06 | > step_time: 0.1485 (0.13483478844199584) | > loader_time: 0.0045 (0.004475997289322803)  --> TIME: 2025-02-23 22:20:19 -- STEP: 10612/36938 -- GLOBAL_STEP: 47550 | > loss_text_ce: 0.11128564924001694 (0.10793876957570984) | > loss_mel_ce: 4.822463512420654 (nan) | > loss: 0.019578371196985245 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.134841918698176) | > loader_time: 0.0045 (0.0044759682920623624)  --> TIME: 2025-02-23 22:20:29 -- STEP: 10662/36938 -- GLOBAL_STEP: 47600 | > loss_text_ce: 0.09137948602437973 (0.107893424879049) | > loss_mel_ce: 3.91931414604187 (nan) | > loss: 0.01591545157134533 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13485116586075077) | > loader_time: 0.0044 (0.00447626108077607)  --> TIME: 2025-02-23 22:20:38 -- STEP: 10712/36938 -- GLOBAL_STEP: 47650 | > loss_text_ce: 0.10263791680335999 (0.10783744005025937) | > loss_mel_ce: 4.648663520812988 (nan) | > loss: 0.018854372203350067 (nan) | > current_lr: 5e-06 | > step_time: 0.1495 (0.13485065581105835) | > loader_time: 0.0045 (0.0044759053450364504)  --> TIME: 2025-02-23 22:20:47 -- STEP: 10762/36938 -- GLOBAL_STEP: 47700 | > loss_text_ce: 0.1091795414686203 (0.10779567186500356) | > loss_mel_ce: 4.053823471069336 (nan) | > loss: 0.016519853845238686 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1348721965984032) | > loader_time: 0.0044 (0.004475459824749776)  --> TIME: 2025-02-23 22:20:57 -- STEP: 10812/36938 -- GLOBAL_STEP: 47750 | > loss_text_ce: 0.0965769812464714 (0.10775900961878773) | > loss_mel_ce: 1.18927001953125 (nan) | > loss: 0.005102567374706268 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13487273029076116) | > loader_time: 0.0047 (0.004475275653404449)  --> TIME: 2025-02-23 22:21:06 -- STEP: 10862/36938 -- GLOBAL_STEP: 47800 | > loss_text_ce: 0.09106533974409103 (0.1077169602875047) | > loss_mel_ce: 3.73565673828125 (nan) | > loss: 0.015185406431555748 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.1348717067808405) | > loader_time: 0.0044 (0.004475963903689479)  --> TIME: 2025-02-23 22:21:15 -- STEP: 10912/36938 -- GLOBAL_STEP: 47850 | > loss_text_ce: 0.09433260560035706 (0.10766020442340044) | > loss_mel_ce: 4.208828926086426 (nan) | > loss: 0.01707603968679905 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.13486717722195612) | > loader_time: 0.0044 (0.004476422809960214)  --> TIME: 2025-02-23 22:21:24 -- STEP: 10962/36938 -- GLOBAL_STEP: 47900 | > loss_text_ce: 0.09841477870941162 (0.10761873674308976) | > loss_mel_ce: 3.3166942596435547 (nan) | > loss: 0.013552021235227585 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13485829215962286) | > loader_time: 0.0043 (0.004476729415450912)  --> TIME: 2025-02-23 22:21:33 -- STEP: 11012/36938 -- GLOBAL_STEP: 47950 | > loss_text_ce: 0.08950556814670563 (0.10757100141393774) | > loss_mel_ce: 4.177773475646973 (nan) | > loss: 0.01693364791572094 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13482799126891382) | > loader_time: 0.0046 (0.0044767454326087915)  --> TIME: 2025-02-23 22:21:42 -- STEP: 11062/36938 -- GLOBAL_STEP: 48000 | > loss_text_ce: 0.10039932280778885 (0.10753137780758568) | > loss_mel_ce: 4.024576663970947 (nan) | > loss: 0.01636895351111889 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1348200404952213) | > loader_time: 0.0043 (0.004476513985030542)  --> TIME: 2025-02-23 22:21:51 -- STEP: 11112/36938 -- GLOBAL_STEP: 48050 | > loss_text_ce: 0.10369842499494553 (0.10749218735275326) | > loss_mel_ce: 2.959670066833496 (nan) | > loss: 0.012156224809587002 (nan) | > current_lr: 5e-06 | > step_time: 0.1076 (0.13479395763233756) | > loader_time: 0.0043 (0.004476008439252842)  --> TIME: 2025-02-23 22:22:00 -- STEP: 11162/36938 -- GLOBAL_STEP: 48100 | > loss_text_ce: 0.08743296563625336 (0.10744762605617209) | > loss_mel_ce: 3.7482869625091553 (nan) | > loss: 0.015221111476421356 (nan) | > current_lr: 5e-06 | > step_time: 0.1773 (0.13480393719104944) | > loader_time: 0.0046 (0.004475798386438971)  --> TIME: 2025-02-23 22:22:10 -- STEP: 11212/36938 -- GLOBAL_STEP: 48150 | > loss_text_ce: 0.09286598116159439 (0.10740574947357338) | > loss_mel_ce: 3.601233959197998 (nan) | > loss: 0.014659127220511436 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13480419326670595) | > loader_time: 0.0045 (0.004475422025449869)  --> TIME: 2025-02-23 22:22:19 -- STEP: 11262/36938 -- GLOBAL_STEP: 48200 | > loss_text_ce: 0.10687270760536194 (0.10735817820090456) | > loss_mel_ce: 0.03900146484375 (nan) | > loss: 0.0005788658163510263 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13481869460041335) | > loader_time: 0.0047 (0.004475201876204556)  --> TIME: 2025-02-23 22:22:28 -- STEP: 11312/36938 -- GLOBAL_STEP: 48250 | > loss_text_ce: 0.10800259560346603 (0.1073027152068851) | > loss_mel_ce: 2.639085292816162 (nan) | > loss: 0.01090114377439022 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13482669733538447) | > loader_time: 0.0042 (0.004474688937842766)  --> TIME: 2025-02-23 22:22:38 -- STEP: 11362/36938 -- GLOBAL_STEP: 48300 | > loss_text_ce: 0.0968046560883522 (0.10726093448720828) | > loss_mel_ce: 4.0796685218811035 (nan) | > loss: 0.016573306173086166 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.1348343635188754) | > loader_time: 0.0042 (0.004473912738994111)  --> TIME: 2025-02-23 22:22:47 -- STEP: 11412/36938 -- GLOBAL_STEP: 48350 | > loss_text_ce: 0.09809501469135284 (0.1072191496468139) | > loss_mel_ce: 4.511889934539795 (nan) | > loss: 0.01829359121620655 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.1348374687367639) | > loader_time: 0.0043 (0.004473251394752539)  --> TIME: 2025-02-23 22:22:56 -- STEP: 11462/36938 -- GLOBAL_STEP: 48400 | > loss_text_ce: 0.10653189569711685 (0.10718044033720958) | > loss_mel_ce: 3.284754753112793 (nan) | > loss: 0.01345748733729124 (nan) | > current_lr: 5e-06 | > step_time: 0.1757 (0.13482220148753135) | > loader_time: 0.0044 (0.004472465170675753)  --> TIME: 2025-02-23 22:23:05 -- STEP: 11512/36938 -- GLOBAL_STEP: 48450 | > loss_text_ce: 0.08910006284713745 (0.10712756264483654) | > loss_mel_ce: 3.7644565105438232 (nan) | > loss: 0.015291891992092133 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.1348275183761507) | > loader_time: 0.0042 (0.004471875794485604)  --> TIME: 2025-02-23 22:23:15 -- STEP: 11562/36938 -- GLOBAL_STEP: 48500 | > loss_text_ce: 0.08604012429714203 (0.1070964730208906) | > loss_mel_ce: 4.2956461906433105 (nan) | > loss: 0.017387645319104195 (nan) | > current_lr: 5e-06 | > step_time: 0.1878 (0.13482872892510722) | > loader_time: 0.0047 (0.004472184379173808)  --> TIME: 2025-02-23 22:23:24 -- STEP: 11612/36938 -- GLOBAL_STEP: 48550 | > loss_text_ce: 0.10738330334424973 (0.10705259692628376) | > loss_mel_ce: 4.059835433959961 (nan) | > loss: 0.01653658226132393 (nan) | > current_lr: 5e-06 | > step_time: 0.1266 (0.1348495268222176) | > loader_time: 0.0043 (0.004472145716404047)  --> TIME: 2025-02-23 22:23:33 -- STEP: 11662/36938 -- GLOBAL_STEP: 48600 | > loss_text_ce: 0.13145510852336884 (0.10700575982577774) | > loss_mel_ce: 0.2664794921875 (nan) | > loss: 0.001579105737619102 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.1348486492511561) | > loader_time: 0.0043 (0.0044723835639122874)  --> TIME: 2025-02-23 22:23:43 -- STEP: 11712/36938 -- GLOBAL_STEP: 48650 | > loss_text_ce: 0.09971268475055695 (0.10696463384907297) | > loss_mel_ce: 4.426716327667236 (nan) | > loss: 0.017962021753191948 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.13484765442668384) | > loader_time: 0.0049 (0.004472515866404682)  --> TIME: 2025-02-23 22:23:52 -- STEP: 11762/36938 -- GLOBAL_STEP: 48700 | > loss_text_ce: 0.10707879811525345 (0.10691531765252703) | > loss_mel_ce: 3.4605553150177 (nan) | > loss: 0.014157279394567013 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.13483960071888673) | > loader_time: 0.0044 (0.0044730461448497)  --> TIME: 2025-02-23 22:24:01 -- STEP: 11812/36938 -- GLOBAL_STEP: 48750 | > loss_text_ce: 0.1069587990641594 (0.10686836003704665) | > loss_mel_ce: 4.250938415527344 (nan) | > loss: 0.01729324460029602 (nan) | > current_lr: 5e-06 | > step_time: 0.1268 (0.13483857759329712) | > loader_time: 0.0046 (0.0044734715567659)  --> TIME: 2025-02-23 22:24:10 -- STEP: 11862/36938 -- GLOBAL_STEP: 48800 | > loss_text_ce: 0.08255597949028015 (0.1068287404661406) | > loss_mel_ce: 3.5174450874328613 (nan) | > loss: 0.014285719022154808 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.13482134176574875) | > loader_time: 0.0046 (0.004473427740523806)  --> TIME: 2025-02-23 22:24:19 -- STEP: 11912/36938 -- GLOBAL_STEP: 48850 | > loss_text_ce: 0.09696555882692337 (0.10677766659884376) | > loss_mel_ce: 4.210529327392578 (nan) | > loss: 0.01709323562681675 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13483892462251953) | > loader_time: 0.0045 (0.004473467674569213)  --> TIME: 2025-02-23 22:24:28 -- STEP: 11962/36938 -- GLOBAL_STEP: 48900 | > loss_text_ce: 0.09703036397695541 (0.10673529635806535) | > loss_mel_ce: 3.2735233306884766 (nan) | > loss: 0.01337521430104971 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13482896162701793) | > loader_time: 0.0044 (0.004473366001501001)  --> TIME: 2025-02-23 22:24:38 -- STEP: 12012/36938 -- GLOBAL_STEP: 48950 | > loss_text_ce: 0.11031214147806168 (0.10669032827979867) | > loss_mel_ce: 3.1213879585266113 (nan) | > loss: 0.012824207544326782 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.1348347221062334) | > loader_time: 0.0044 (0.004473523322717386)  --> TIME: 2025-02-23 22:24:47 -- STEP: 12062/36938 -- GLOBAL_STEP: 49000 | > loss_text_ce: 0.08914900571107864 (0.10665112024794239) | > loss_mel_ce: 2.993335008621216 (nan) | > loss: 0.012232080101966858 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13483409064461604) | > loader_time: 0.0047 (0.004473978064582442)  --> TIME: 2025-02-23 22:24:57 -- STEP: 12112/36938 -- GLOBAL_STEP: 49050 | > loss_text_ce: 0.09345924109220505 (0.10659297998463119) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13485161212114197) | > loader_time: 0.0045 (0.004474326554818713)  --> TIME: 2025-02-23 22:25:06 -- STEP: 12162/36938 -- GLOBAL_STEP: 49100 | > loss_text_ce: 0.11675002425909042 (0.10654090930378639) | > loss_mel_ce: 4.096020221710205 (nan) | > loss: 0.016717344522476196 (nan) | > current_lr: 5e-06 | > step_time: 0.1031 (0.13484600313281017) | > loader_time: 0.0044 (0.004474426488150235)  --> TIME: 2025-02-23 22:25:15 -- STEP: 12212/36938 -- GLOBAL_STEP: 49150 | > loss_text_ce: 0.10029010474681854 (0.10650938338835915) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1577 (0.13484381442377655) | > loader_time: 0.0045 (0.004474398389317122)  --> TIME: 2025-02-23 22:25:24 -- STEP: 12262/36938 -- GLOBAL_STEP: 49200 | > loss_text_ce: 0.09492550045251846 (0.1064600904191607) | > loss_mel_ce: 3.4333629608154297 (nan) | > loss: 0.014001145027577877 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13484712942761443) | > loader_time: 0.0044 (0.004474761668435329)  --> TIME: 2025-02-23 22:25:34 -- STEP: 12312/36938 -- GLOBAL_STEP: 49250 | > loss_text_ce: 0.11040583997964859 (0.10641778579503405) | > loss_mel_ce: 4.049229621887207 (nan) | > loss: 0.016506491228938103 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1348774372137077) | > loader_time: 0.0043 (0.00447499722997581)  --> TIME: 2025-02-23 22:25:43 -- STEP: 12362/36938 -- GLOBAL_STEP: 49300 | > loss_text_ce: 0.09688709676265717 (0.1063651822120449) | > loss_mel_ce: 3.4199650287628174 (nan) | > loss: 0.013955763541162014 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13488667930451168) | > loader_time: 0.0042 (0.004475345100096733)  --> TIME: 2025-02-23 22:25:52 -- STEP: 12412/36938 -- GLOBAL_STEP: 49350 | > loss_text_ce: 0.09705612063407898 (0.10632197734219422) | > loss_mel_ce: 2.333998680114746 (nan) | > loss: 0.00964704342186451 (nan) | > current_lr: 5e-06 | > step_time: 0.1435 (0.13488451402263374) | > loader_time: 0.0044 (0.004475343162076853)  --> TIME: 2025-02-23 22:26:01 -- STEP: 12462/36938 -- GLOBAL_STEP: 49400 | > loss_text_ce: 0.09621825069189072 (0.10628126835582968) | > loss_mel_ce: 3.7324652671813965 (nan) | > loss: 0.01519318949431181 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13486450641768563) | > loader_time: 0.0044 (0.004475487902810736)  --> TIME: 2025-02-23 22:26:10 -- STEP: 12512/36938 -- GLOBAL_STEP: 49450 | > loss_text_ce: 0.09095359593629837 (0.10623541590638584) | > loss_mel_ce: 4.649808883666992 (nan) | > loss: 0.018812552094459534 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.1348570011117874) | > loader_time: 0.0047 (0.0044758527556343864)  --> TIME: 2025-02-23 22:26:20 -- STEP: 12562/36938 -- GLOBAL_STEP: 49500 | > loss_text_ce: 0.1217653751373291 (0.1061901899060282) | > loss_mel_ce: 3.337339162826538 (nan) | > loss: 0.013726606033742428 (nan) | > current_lr: 5e-06 | > step_time: 0.1121 (0.13485612680987488) | > loader_time: 0.0042 (0.004476176688405794)  --> TIME: 2025-02-23 22:26:29 -- STEP: 12612/36938 -- GLOBAL_STEP: 49550 | > loss_text_ce: 0.08517628163099289 (0.10614501514767159) | > loss_mel_ce: 0.45682936906814575 (nan) | > loss: 0.0021508161444216967 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1348711519894652) | > loader_time: 0.0044 (0.0044764238919222845)  --> TIME: 2025-02-23 22:26:38 -- STEP: 12662/36938 -- GLOBAL_STEP: 49600 | > loss_text_ce: 0.09207253903150558 (0.10609524064925202) | > loss_mel_ce: 3.6199183464050293 (nan) | > loss: 0.014730122871696949 (nan) | > current_lr: 5e-06 | > step_time: 0.1357 (0.13487334095872436) | > loader_time: 0.0045 (0.004476481451089686)  --> TIME: 2025-02-23 22:26:47 -- STEP: 12712/36938 -- GLOBAL_STEP: 49650 | > loss_text_ce: 0.0814637839794159 (0.10604857323284501) | > loss_mel_ce: 3.775341510772705 (nan) | > loss: 0.015304784290492535 (nan) | > current_lr: 5e-06 | > step_time: 0.1844 (0.13486330146456404) | > loader_time: 0.0083 (0.00447648577978208)  --> TIME: 2025-02-23 22:26:57 -- STEP: 12762/36938 -- GLOBAL_STEP: 49700 | > loss_text_ce: 0.07963095605373383 (0.10599628248603876) | > loss_mel_ce: 3.2142374515533447 (nan) | > loss: 0.013070906512439251 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13486697587487118) | > loader_time: 0.0044 (0.004476665964770742)  --> TIME: 2025-02-23 22:27:06 -- STEP: 12812/36938 -- GLOBAL_STEP: 49750 | > loss_text_ce: 0.1339471936225891 (0.10595605559088823) | > loss_mel_ce: 1.5629204511642456 (nan) | > loss: 0.006733602378517389 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13486285673764306) | > loader_time: 0.0044 (0.00447673016773549)  --> TIME: 2025-02-23 22:27:15 -- STEP: 12862/36938 -- GLOBAL_STEP: 49800 | > loss_text_ce: 0.08358637988567352 (0.10590068381867677) | > loss_mel_ce: 4.3434882164001465 (nan) | > loss: 0.01756775751709938 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.1348690517099584) | > loader_time: 0.0045 (0.004476929040890132)  --> TIME: 2025-02-23 22:27:24 -- STEP: 12912/36938 -- GLOBAL_STEP: 49850 | > loss_text_ce: 0.0859375074505806 (0.10586335001942682) | > loss_mel_ce: 3.973985195159912 (nan) | > loss: 0.016110805794596672 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13486995692959156) | > loader_time: 0.0045 (0.004477397604502714)  --> TIME: 2025-02-23 22:27:34 -- STEP: 12962/36938 -- GLOBAL_STEP: 49900 | > loss_text_ce: 0.09143513441085815 (0.10581737274642888) | > loss_mel_ce: 4.38895845413208 (nan) | > loss: 0.017779339104890823 (nan) | > current_lr: 5e-06 | > step_time: 0.1842 (0.13488736819970737) | > loader_time: 0.0045 (0.004477535422027459)  --> TIME: 2025-02-23 22:27:43 -- STEP: 13012/36938 -- GLOBAL_STEP: 49950 | > loss_text_ce: 0.10406984388828278 (0.1057852418005898) | > loss_mel_ce: 4.2071452140808105 (nan) | > loss: 0.017107997089624405 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1348806340035527) | > loader_time: 0.0044 (0.004477364427670317)  --> TIME: 2025-02-23 22:27:52 -- STEP: 13062/36938 -- GLOBAL_STEP: 50000 | > loss_text_ce: 0.08790021389722824 (0.10574584618345417) | > loss_mel_ce: 4.037323951721191 (nan) | > loss: 0.016369936987757683 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13489601136056964) | > loader_time: 0.0049 (0.004477258353553869) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_50000.pth  --> TIME: 2025-02-23 22:28:07 -- STEP: 13112/36938 -- GLOBAL_STEP: 50050 | > loss_text_ce: 0.08814183622598648 (0.10569450330350597) | > loss_mel_ce: 3.25291109085083 (nan) | > loss: 0.013258147984743118 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13488699085760375) | > loader_time: 0.0045 (0.004477590613426281)  --> TIME: 2025-02-23 22:28:16 -- STEP: 13162/36938 -- GLOBAL_STEP: 50100 | > loss_text_ce: 0.10201200842857361 (0.10565142610427775) | > loss_mel_ce: 4.425596237182617 (nan) | > loss: 0.017966700717806816 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.1348826061029083) | > loader_time: 0.0045 (0.004477666279697587)  --> TIME: 2025-02-23 22:28:25 -- STEP: 13212/36938 -- GLOBAL_STEP: 50150 | > loss_text_ce: 0.0906548872590065 (0.1055963967861668) | > loss_mel_ce: 3.9237003326416016 (nan) | > loss: 0.01592998206615448 (nan) | > current_lr: 5e-06 | > step_time: 0.1659 (0.1348736403098867) | > loader_time: 0.005 (0.004477761692615347)  --> TIME: 2025-02-23 22:28:34 -- STEP: 13262/36938 -- GLOBAL_STEP: 50200 | > loss_text_ce: 0.08715296536684036 (0.1055482977956575) | > loss_mel_ce: 4.540329456329346 (nan) | > loss: 0.018363026902079582 (nan) | > current_lr: 5e-06 | > step_time: 0.1833 (0.13486470769494677) | > loader_time: 0.0047 (0.00447787898389701)  --> TIME: 2025-02-23 22:28:43 -- STEP: 13312/36938 -- GLOBAL_STEP: 50250 | > loss_text_ce: 0.09439931809902191 (0.10550393796046811) | > loss_mel_ce: 4.5322041511535645 (nan) | > loss: 0.018359540030360222 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.13484814739786014) | > loader_time: 0.0044 (0.004477831158930296)  --> TIME: 2025-02-23 22:28:52 -- STEP: 13362/36938 -- GLOBAL_STEP: 50300 | > loss_text_ce: 0.11006536334753036 (0.10546568575440786) | > loss_mel_ce: 2.679490804672241 (nan) | > loss: 0.01106966845691204 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.1348464859047608) | > loader_time: 0.0042 (0.004477829530627733)  --> TIME: 2025-02-23 22:29:02 -- STEP: 13412/36938 -- GLOBAL_STEP: 50350 | > loss_text_ce: 0.08813583105802536 (0.10542591629161566) | > loss_mel_ce: 4.458285808563232 (nan) | > loss: 0.018041355535387993 (nan) | > current_lr: 5e-06 | > step_time: 0.1821 (0.1348672900276677) | > loader_time: 0.0045 (0.004477677738492333)  --> TIME: 2025-02-23 22:29:11 -- STEP: 13462/36938 -- GLOBAL_STEP: 50400 | > loss_text_ce: 0.0887431874871254 (0.10537933188685257) | > loss_mel_ce: 2.56730580329895 (nan) | > loss: 0.010539877228438854 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13487876508268076) | > loader_time: 0.0044 (0.004478065702965145)  --> TIME: 2025-02-23 22:29:20 -- STEP: 13512/36938 -- GLOBAL_STEP: 50450 | > loss_text_ce: 0.09372857213020325 (0.10532811374771334) | > loss_mel_ce: 4.423691749572754 (nan) | > loss: 0.017926272004842758 (nan) | > current_lr: 5e-06 | > step_time: 0.15 (0.13486893860597804) | > loader_time: 0.0046 (0.004477829658611073)  --> TIME: 2025-02-23 22:29:29 -- STEP: 13562/36938 -- GLOBAL_STEP: 50500 | > loss_text_ce: 0.1066952645778656 (0.1052886288156554) | > loss_mel_ce: 4.142527103424072 (nan) | > loss: 0.016861993819475174 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13485073359415806) | > loader_time: 0.0046 (0.004477686383453137)  --> TIME: 2025-02-23 22:29:39 -- STEP: 13612/36938 -- GLOBAL_STEP: 50550 | > loss_text_ce: 0.0888301357626915 (0.10524195763976259) | > loss_mel_ce: 2.3379838466644287 (nan) | > loss: 0.009630215354263783 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13486585479044605) | > loader_time: 0.0046 (0.004477673896859058)  --> TIME: 2025-02-23 22:29:48 -- STEP: 13662/36938 -- GLOBAL_STEP: 50600 | > loss_text_ce: 0.08439712971448898 (0.10519590086568566) | > loss_mel_ce: 4.252808570861816 (nan) | > loss: 0.01721113547682762 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13486966237266496) | > loader_time: 0.0044 (0.00447755499686073)  --> TIME: 2025-02-23 22:29:57 -- STEP: 13712/36938 -- GLOBAL_STEP: 50650 | > loss_text_ce: 0.09944988787174225 (0.10515675976135655) | > loss_mel_ce: 3.9392781257629395 (nan) | > loss: 0.016026699915528297 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13486340883662779) | > loader_time: 0.0043 (0.004477325231358498)  --> TIME: 2025-02-23 22:30:06 -- STEP: 13762/36938 -- GLOBAL_STEP: 50700 | > loss_text_ce: 0.08130868524312973 (0.10511458059752557) | > loss_mel_ce: 4.617071628570557 (nan) | > loss: 0.018644368276000023 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1348695953022922) | > loader_time: 0.0043 (0.004477585597260715)  --> TIME: 2025-02-23 22:30:16 -- STEP: 13812/36938 -- GLOBAL_STEP: 50750 | > loss_text_ce: 0.08062689751386642 (0.10507246245981533) | > loss_mel_ce: 4.464709281921387 (nan) | > loss: 0.01803704909980297 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13487622622576292) | > loader_time: 0.0046 (0.004477765744505086)  --> TIME: 2025-02-23 22:30:25 -- STEP: 13862/36938 -- GLOBAL_STEP: 50800 | > loss_text_ce: 0.08942733705043793 (0.10504462094495118) | > loss_mel_ce: 4.4857611656188965 (nan) | > loss: 0.01815551146864891 (nan) | > current_lr: 5e-06 | > step_time: 0.1522 (0.1348844403364729) | > loader_time: 0.0044 (0.0044777832270666154)  --> TIME: 2025-02-23 22:30:34 -- STEP: 13912/36938 -- GLOBAL_STEP: 50850 | > loss_text_ce: 0.09057644009590149 (0.10500513747936449) | > loss_mel_ce: 1.4595540761947632 (nan) | > loss: 0.006151312030851841 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.1348882964830954) | > loader_time: 0.0045 (0.004477300353952356)  --> TIME: 2025-02-23 22:30:44 -- STEP: 13962/36938 -- GLOBAL_STEP: 50900 | > loss_text_ce: 0.0976901724934578 (0.10495986079039667) | > loss_mel_ce: 4.106328964233398 (nan) | > loss: 0.016682615503668785 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13489361436707442) | > loader_time: 0.0043 (0.00447669341389659)  --> TIME: 2025-02-23 22:30:53 -- STEP: 14012/36938 -- GLOBAL_STEP: 50950 | > loss_text_ce: 0.1015780121088028 (0.1049200595238747) | > loss_mel_ce: 4.581818103790283 (nan) | > loss: 0.018584907054901123 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.13488794475700153) | > loader_time: 0.0044 (0.004476158407256494)  --> TIME: 2025-02-23 22:31:02 -- STEP: 14062/36938 -- GLOBAL_STEP: 51000 | > loss_text_ce: 0.0833960771560669 (0.10487826657471566) | > loss_mel_ce: 1.5796765089035034 (nan) | > loss: 0.006599494721740484 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13487411238800873) | > loader_time: 0.0041 (0.004475682223611224)  --> TIME: 2025-02-23 22:31:11 -- STEP: 14112/36938 -- GLOBAL_STEP: 51050 | > loss_text_ce: 0.09502039849758148 (0.10482685462161993) | > loss_mel_ce: 4.4449992179870605 (nan) | > loss: 0.018015950918197632 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.13488256611994245) | > loader_time: 0.0044 (0.004476360317801126)  --> TIME: 2025-02-23 22:31:20 -- STEP: 14162/36938 -- GLOBAL_STEP: 51100 | > loss_text_ce: 0.09334629029035568 (0.10477941987996325) | > loss_mel_ce: 4.142796516418457 (nan) | > loss: 0.016810091212391853 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13487960133675692) | > loader_time: 0.0045 (0.004476731602308216)  --> TIME: 2025-02-23 22:31:30 -- STEP: 14212/36938 -- GLOBAL_STEP: 51150 | > loss_text_ce: 0.09450602531433105 (0.10473766415795416) | > loss_mel_ce: 4.271845817565918 (nan) | > loss: 0.01732679456472397 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.134890301392643) | > loader_time: 0.0045 (0.004477071235322828)  --> TIME: 2025-02-23 22:31:39 -- STEP: 14262/36938 -- GLOBAL_STEP: 51200 | > loss_text_ce: 0.0830598920583725 (0.10468698605683999) | > loss_mel_ce: 4.354853630065918 (nan) | > loss: 0.017610767856240273 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.13489241677603636) | > loader_time: 0.0046 (0.004477624354230756)  --> TIME: 2025-02-23 22:31:48 -- STEP: 14312/36938 -- GLOBAL_STEP: 51250 | > loss_text_ce: 0.09478183835744858 (0.1046398323515505) | > loss_mel_ce: 2.2882602214813232 (nan) | > loss: 0.009456517174839973 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13489562246371578) | > loader_time: 0.0046 (0.004477978918790947)  --> TIME: 2025-02-23 22:31:58 -- STEP: 14362/36938 -- GLOBAL_STEP: 51300 | > loss_text_ce: 0.09214487671852112 (0.10459107800056529) | > loss_mel_ce: 2.956552267074585 (nan) | > loss: 0.012098005972802639 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.1349124959577267) | > loader_time: 0.0045 (0.004478149337035466)  --> TIME: 2025-02-23 22:32:07 -- STEP: 14412/36938 -- GLOBAL_STEP: 51350 | > loss_text_ce: 0.11028559505939484 (0.1045411288171652) | > loss_mel_ce: 0.10291340947151184 (nan) | > loss: 0.0008460278622806072 (nan) | > current_lr: 5e-06 | > step_time: 0.1079 (0.13492771153512118) | > loader_time: 0.0044 (0.004478623841889457)  --> TIME: 2025-02-23 22:32:17 -- STEP: 14462/36938 -- GLOBAL_STEP: 51400 | > loss_text_ce: 0.08594468981027603 (0.10449338054546949) | > loss_mel_ce: 3.8294737339019775 (nan) | > loss: 0.015537375584244728 (nan) | > current_lr: 5e-06 | > step_time: 0.1082 (0.13492759946404412) | > loader_time: 0.0043 (0.004478705076076001)  --> TIME: 2025-02-23 22:32:26 -- STEP: 14512/36938 -- GLOBAL_STEP: 51450 | > loss_text_ce: 0.09396759420633316 (0.10444902412053268) | > loss_mel_ce: 3.8224854469299316 (nan) | > loss: 0.015541481785476208 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13495690875831398) | > loader_time: 0.0045 (0.004478869801582282)  --> TIME: 2025-02-23 22:32:36 -- STEP: 14562/36938 -- GLOBAL_STEP: 51500 | > loss_text_ce: 0.0846657007932663 (0.1043991531341217) | > loss_mel_ce: 4.018880367279053 (nan) | > loss: 0.01628391444683075 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13494868591875908) | > loader_time: 0.0044 (0.004478848777847485)  --> TIME: 2025-02-23 22:32:45 -- STEP: 14612/36938 -- GLOBAL_STEP: 51550 | > loss_text_ce: 0.08013585954904556 (0.10436079915191772) | > loss_mel_ce: 4.313587188720703 (nan) | > loss: 0.017435409128665924 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13495955203220972) | > loader_time: 0.0043 (0.004479481493189923)  --> TIME: 2025-02-23 22:32:54 -- STEP: 14662/36938 -- GLOBAL_STEP: 51600 | > loss_text_ce: 0.0819668173789978 (0.10432229984092813) | > loss_mel_ce: 4.929284572601318 (nan) | > loss: 0.019885919988155365 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13495244953515512) | > loader_time: 0.0045 (0.004479655951501366)  --> TIME: 2025-02-23 22:33:03 -- STEP: 14712/36938 -- GLOBAL_STEP: 51650 | > loss_text_ce: 0.07967442274093628 (0.10428050140758592) | > loss_mel_ce: 4.04703950881958 (nan) | > loss: 0.016375849023461342 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13495150511520115) | > loader_time: 0.0047 (0.004479944916637515)  --> TIME: 2025-02-23 22:33:13 -- STEP: 14762/36938 -- GLOBAL_STEP: 51700 | > loss_text_ce: 0.08771368116140366 (0.10424321466271293) | > loss_mel_ce: 3.9721758365631104 (nan) | > loss: 0.016110673546791077 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.134950864817323) | > loader_time: 0.0045 (0.004479939448867379)  --> TIME: 2025-02-23 22:33:22 -- STEP: 14812/36938 -- GLOBAL_STEP: 51750 | > loss_text_ce: 0.08435390889644623 (0.10420352562179518) | > loss_mel_ce: 4.339727401733398 (nan) | > loss: 0.017555879428982735 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13495021311617963) | > loader_time: 0.0049 (0.004480177136714024)  --> TIME: 2025-02-23 22:33:31 -- STEP: 14862/36938 -- GLOBAL_STEP: 51800 | > loss_text_ce: 0.10493004322052002 (0.10415860079554744) | > loss_mel_ce: 4.033969879150391 (nan) | > loss: 0.016424207016825676 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13495816669173996) | > loader_time: 0.0045 (0.00448028910707102)  --> TIME: 2025-02-23 22:33:40 -- STEP: 14912/36938 -- GLOBAL_STEP: 51850 | > loss_text_ce: 0.0721280500292778 (0.10410996596328924) | > loss_mel_ce: 4.335185527801514 (nan) | > loss: 0.017489340156316757 (nan) | > current_lr: 5e-06 | > step_time: 0.1571 (0.13495647581094256) | > loader_time: 0.0043 (0.004479945025730538)  --> TIME: 2025-02-23 22:33:50 -- STEP: 14962/36938 -- GLOBAL_STEP: 51900 | > loss_text_ce: 0.0894964188337326 (0.10406424797291577) | > loss_mel_ce: 3.8714847564697266 (nan) | > loss: 0.015718180686235428 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13495808034286677) | > loader_time: 0.0043 (0.0044795542282147285)  --> TIME: 2025-02-23 22:33:58 -- STEP: 15012/36938 -- GLOBAL_STEP: 51950 | > loss_text_ce: 0.09524652361869812 (0.1040283003913787) | > loss_mel_ce: 4.193565845489502 (nan) | > loss: 0.01701909676194191 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13493428252837836) | > loader_time: 0.0042 (0.004479480066078359)  --> TIME: 2025-02-23 22:34:07 -- STEP: 15062/36938 -- GLOBAL_STEP: 52000 | > loss_text_ce: 0.06599712371826172 (0.10398755847958602) | > loss_mel_ce: 3.23335862159729 (nan) | > loss: 0.01309268269687891 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13492334260992403) | > loader_time: 0.0042 (0.0044795697214371965)  --> TIME: 2025-02-23 22:34:17 -- STEP: 15112/36938 -- GLOBAL_STEP: 52050 | > loss_text_ce: 0.09777546674013138 (0.10394081084206591) | > loss_mel_ce: 1.990506887435913 (nan) | > loss: 0.008286835625767708 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.1349132325738867) | > loader_time: 0.0045 (0.004479415221716623)  --> TIME: 2025-02-23 22:34:26 -- STEP: 15162/36938 -- GLOBAL_STEP: 52100 | > loss_text_ce: 0.08545160293579102 (0.10389619299844889) | > loss_mel_ce: 4.043520450592041 (nan) | > loss: 0.016384810209274292 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.13492601442393581) | > loader_time: 0.0045 (0.004479424177601787)  --> TIME: 2025-02-23 22:34:35 -- STEP: 15212/36938 -- GLOBAL_STEP: 52150 | > loss_text_ce: 0.08420393615961075 (0.10385241008689623) | > loss_mel_ce: 4.914463520050049 (nan) | > loss: 0.019835982471704483 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13492500852541642) | > loader_time: 0.0043 (0.004479872656909217)  --> TIME: 2025-02-23 22:34:45 -- STEP: 15262/36938 -- GLOBAL_STEP: 52200 | > loss_text_ce: 0.09799589961767197 (0.103814409545116) | > loss_mel_ce: 4.279435634613037 (nan) | > loss: 0.017370760440826416 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.1349282271895454) | > loader_time: 0.0045 (0.0044803688588952815)  --> TIME: 2025-02-23 22:34:54 -- STEP: 15312/36938 -- GLOBAL_STEP: 52250 | > loss_text_ce: 0.09899304807186127 (0.10377514580227716) | > loss_mel_ce: 3.667795181274414 (nan) | > loss: 0.014947573654353619 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.1349322275650781) | > loader_time: 0.0045 (0.004480810203397781)  --> TIME: 2025-02-23 22:35:03 -- STEP: 15362/36938 -- GLOBAL_STEP: 52300 | > loss_text_ce: 0.0937620997428894 (0.10373030260133498) | > loss_mel_ce: 4.915129661560059 (nan) | > loss: 0.019876554608345032 (nan) | > current_lr: 5e-06 | > step_time: 0.1531 (0.13493334855935174) | > loader_time: 0.0045 (0.004480760368483244)  --> TIME: 2025-02-23 22:35:13 -- STEP: 15412/36938 -- GLOBAL_STEP: 52350 | > loss_text_ce: 0.1100430116057396 (0.10368851585472734) | > loss_mel_ce: 2.959625482559204 (nan) | > loss: 0.01218122523277998 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.1349383212206359) | > loader_time: 0.0043 (0.004481104142381836)  --> TIME: 2025-02-23 22:35:22 -- STEP: 15462/36938 -- GLOBAL_STEP: 52400 | > loss_text_ce: 0.09316657483577728 (0.10364912676018657) | > loss_mel_ce: 1.6882188320159912 (nan) | > loss: 0.007068990264087915 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.1349426169488575) | > loader_time: 0.0044 (0.004481369057288837)  --> TIME: 2025-02-23 22:35:31 -- STEP: 15512/36938 -- GLOBAL_STEP: 52450 | > loss_text_ce: 0.11549261957406998 (0.10361189447399075) | > loss_mel_ce: 3.242753267288208 (nan) | > loss: 0.01332637295126915 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.1349298665823544) | > loader_time: 0.0045 (0.004481372727506731)  --> TIME: 2025-02-23 22:35:40 -- STEP: 15562/36938 -- GLOBAL_STEP: 52500 | > loss_text_ce: 0.07487545162439346 (0.10356789787848487) | > loss_mel_ce: 4.511269569396973 (nan) | > loss: 0.01819898933172226 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.134926839255379) | > loader_time: 0.0045 (0.004481350620276145)  --> TIME: 2025-02-23 22:35:49 -- STEP: 15612/36938 -- GLOBAL_STEP: 52550 | > loss_text_ce: 0.09385962039232254 (0.10353261531077004) | > loss_mel_ce: 3.7750422954559326 (nan) | > loss: 0.015352786518633366 (nan) | > current_lr: 5e-06 | > step_time: 0.1504 (0.13492973193491842) | > loader_time: 0.0045 (0.004481481354327617)  --> TIME: 2025-02-23 22:35:58 -- STEP: 15662/36938 -- GLOBAL_STEP: 52600 | > loss_text_ce: 0.09062741696834564 (0.1034902182894449) | > loss_mel_ce: 4.095864772796631 (nan) | > loss: 0.0166130643337965 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13491825726491616) | > loader_time: 0.0044 (0.0044816182408961795)  --> TIME: 2025-02-23 22:36:07 -- STEP: 15712/36938 -- GLOBAL_STEP: 52650 | > loss_text_ce: 0.09847604483366013 (0.10344855886999746) | > loss_mel_ce: 4.447118759155273 (nan) | > loss: 0.018038075417280197 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13490872714827779) | > loader_time: 0.005 (0.004482093432163266)  --> TIME: 2025-02-23 22:36:17 -- STEP: 15762/36938 -- GLOBAL_STEP: 52700 | > loss_text_ce: 0.07334383577108383 (0.10340096309540955) | > loss_mel_ce: 4.664135932922363 (nan) | > loss: 0.018799522891640663 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.134907172758133) | > loader_time: 0.0049 (0.00448239138549603)  --> TIME: 2025-02-23 22:36:26 -- STEP: 15812/36938 -- GLOBAL_STEP: 52750 | > loss_text_ce: 0.08714167028665543 (0.10335775375716791) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13490272159368874) | > loader_time: 0.0044 (0.004482600723916052)  --> TIME: 2025-02-23 22:36:35 -- STEP: 15862/36938 -- GLOBAL_STEP: 52800 | > loss_text_ce: 0.09021804481744766 (0.10331438497167318) | > loss_mel_ce: 2.878044843673706 (nan) | > loss: 0.011778822168707848 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13490382546720728) | > loader_time: 0.0044 (0.004482538624047899)  --> TIME: 2025-02-23 22:36:44 -- STEP: 15912/36938 -- GLOBAL_STEP: 52850 | > loss_text_ce: 0.08798730373382568 (0.10327357454333123) | > loss_mel_ce: 3.495859384536743 (nan) | > loss: 0.014221614226698875 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.1348976193659859) | > loader_time: 0.0045 (0.004482514043738935)  --> TIME: 2025-02-23 22:36:54 -- STEP: 15962/36938 -- GLOBAL_STEP: 52900 | > loss_text_ce: 0.07831144332885742 (0.10322911725570327) | > loss_mel_ce: 4.055045127868652 (nan) | > loss: 0.01640220917761326 (nan) | > current_lr: 5e-06 | > step_time: 0.1979 (0.13492118719301877) | > loader_time: 0.0046 (0.004482589648071943)  --> TIME: 2025-02-23 22:37:03 -- STEP: 16012/36938 -- GLOBAL_STEP: 52950 | > loss_text_ce: 0.09237796813249588 (0.10319159085793249) | > loss_mel_ce: 1.578688383102417 (nan) | > loss: 0.006631216499954462 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13492368823910722) | > loader_time: 0.0045 (0.004482677540956599)  --> TIME: 2025-02-23 22:37:12 -- STEP: 16062/36938 -- GLOBAL_STEP: 53000 | > loss_text_ce: 0.1001104786992073 (0.10314880731053934) | > loss_mel_ce: 4.26763916015625 (nan) | > loss: 0.01733234152197838 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1349222291793931) | > loader_time: 0.0045 (0.004482859010165599)  --> TIME: 2025-02-23 22:37:21 -- STEP: 16112/36938 -- GLOBAL_STEP: 53050 | > loss_text_ce: 0.09485718607902527 (0.10310747359149618) | > loss_mel_ce: 4.369956970214844 (nan) | > loss: 0.017717517912387848 (nan) | > current_lr: 5e-06 | > step_time: 0.177 (0.13490367779191872) | > loader_time: 0.0046 (0.004483102301973569)  --> TIME: 2025-02-23 22:37:30 -- STEP: 16162/36938 -- GLOBAL_STEP: 53100 | > loss_text_ce: 0.08820684999227524 (0.10306577820249271) | > loss_mel_ce: 3.7785732746124268 (nan) | > loss: 0.015344366431236267 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13489093514690095) | > loader_time: 0.0044 (0.0044825042390156545)  --> TIME: 2025-02-23 22:37:40 -- STEP: 16212/36938 -- GLOBAL_STEP: 53150 | > loss_text_ce: 0.09400578588247299 (0.10302032444321865) | > loss_mel_ce: 4.80078125 (nan) | > loss: 0.01942375861108303 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13488889150845335) | > loader_time: 0.0044 (0.004482329362355833)  --> TIME: 2025-02-23 22:37:49 -- STEP: 16262/36938 -- GLOBAL_STEP: 53200 | > loss_text_ce: 0.10449731349945068 (0.10297894043188865) | > loss_mel_ce: 4.069505214691162 (nan) | > loss: 0.016563503071665764 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13490393152683747) | > loader_time: 0.0044 (0.004482914668111219)  --> TIME: 2025-02-23 22:37:58 -- STEP: 16312/36938 -- GLOBAL_STEP: 53250 | > loss_text_ce: 0.08565504103899002 (0.10293894353431858) | > loss_mel_ce: 3.621373414993286 (nan) | > loss: 0.01471043098717928 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1348994806913133) | > loader_time: 0.0046 (0.004483031080656623)  --> TIME: 2025-02-23 22:38:08 -- STEP: 16362/36938 -- GLOBAL_STEP: 53300 | > loss_text_ce: 0.09372242540121078 (0.10289659888297237) | > loss_mel_ce: 4.251789093017578 (nan) | > loss: 0.017244093120098114 (nan) | > current_lr: 5e-06 | > step_time: 0.1357 (0.13490657499812453) | > loader_time: 0.0045 (0.0044836858973907)  --> TIME: 2025-02-23 22:38:17 -- STEP: 16412/36938 -- GLOBAL_STEP: 53350 | > loss_text_ce: 0.08648703247308731 (0.10285679401310957) | > loss_mel_ce: 1.4808592796325684 (nan) | > loss: 0.006219628732651472 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13490661694716974) | > loader_time: 0.0044 (0.004483790447617224)  --> TIME: 2025-02-23 22:38:26 -- STEP: 16462/36938 -- GLOBAL_STEP: 53400 | > loss_text_ce: 0.08405376225709915 (0.1028193146422146) | > loss_mel_ce: 1.5165971517562866 (nan) | > loss: 0.0063517894595861435 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1349145354918723) | > loader_time: 0.0043 (0.004483979899137973)  --> TIME: 2025-02-23 22:38:36 -- STEP: 16512/36938 -- GLOBAL_STEP: 53450 | > loss_text_ce: 0.07603897154331207 (0.10277510664151455) | > loss_mel_ce: 3.3678243160247803 (nan) | > loss: 0.013666125014424324 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13494022033184472) | > loader_time: 0.0045 (0.004484319785075591)  --> TIME: 2025-02-23 22:38:45 -- STEP: 16562/36938 -- GLOBAL_STEP: 53500 | > loss_text_ce: 0.0780659094452858 (0.10273409701182919) | > loss_mel_ce: 4.604513168334961 (nan) | > loss: 0.018581664189696312 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13492338495573458) | > loader_time: 0.0043 (0.0044842768573426965)  --> TIME: 2025-02-23 22:38:54 -- STEP: 16612/36938 -- GLOBAL_STEP: 53550 | > loss_text_ce: 0.08405395597219467 (0.10269618848894432) | > loss_mel_ce: 0.103759765625 (nan) | > loss: 0.0007452926365658641 (nan) | > current_lr: 5e-06 | > step_time: 0.1049 (0.13491264662339542) | > loader_time: 0.0041 (0.004484381068335534)  --> TIME: 2025-02-23 22:39:03 -- STEP: 16662/36938 -- GLOBAL_STEP: 53600 | > loss_text_ce: 0.08371149003505707 (0.10264987780724517) | > loss_mel_ce: 4.490576267242432 (nan) | > loss: 0.018151937052607536 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.13492397415782428) | > loader_time: 0.0048 (0.004484732244613145)  --> TIME: 2025-02-23 22:39:13 -- STEP: 16712/36938 -- GLOBAL_STEP: 53650 | > loss_text_ce: 0.10258536040782928 (0.10260727059572246) | > loss_mel_ce: 4.256814002990723 (nan) | > loss: 0.01729920506477356 (nan) | > current_lr: 5e-06 | > step_time: 0.157 (0.13492442171729938) | > loader_time: 0.0045 (0.004485048492768549)  --> TIME: 2025-02-23 22:39:22 -- STEP: 16762/36938 -- GLOBAL_STEP: 53700 | > loss_text_ce: 0.08475780487060547 (0.10256758847463583) | > loss_mel_ce: 4.101884365081787 (nan) | > loss: 0.016613660380244255 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.13493519184370564) | > loader_time: 0.005 (0.004485365016237672)  --> TIME: 2025-02-23 22:39:31 -- STEP: 16812/36938 -- GLOBAL_STEP: 53750 | > loss_text_ce: 0.0885288342833519 (0.10252928039953815) | > loss_mel_ce: 4.435682773590088 (nan) | > loss: 0.01795322075486183 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.13492816910697134) | > loader_time: 0.0045 (0.004485432176456089)  --> TIME: 2025-02-23 22:39:40 -- STEP: 16862/36938 -- GLOBAL_STEP: 53800 | > loss_text_ce: 0.08321171253919601 (0.10249066789530811) | > loss_mel_ce: 3.3041179180145264 (nan) | > loss: 0.013441785238683224 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13491813945057574) | > loader_time: 0.0043 (0.004485430729910931)  --> TIME: 2025-02-23 22:39:49 -- STEP: 16912/36938 -- GLOBAL_STEP: 53850 | > loss_text_ce: 0.0912499949336052 (0.10245123554273126) | > loss_mel_ce: 1.590742826461792 (nan) | > loss: 0.006674574688076973 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13491519847171016) | > loader_time: 0.0044 (0.004485626235364624)  --> TIME: 2025-02-23 22:39:59 -- STEP: 16962/36938 -- GLOBAL_STEP: 53900 | > loss_text_ce: 0.09278829395771027 (0.10240419582323143) | > loss_mel_ce: 4.531037330627441 (nan) | > loss: 0.018348515033721924 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1349148543400235) | > loader_time: 0.0043 (0.004485575436786746)  --> TIME: 2025-02-23 22:40:08 -- STEP: 17012/36938 -- GLOBAL_STEP: 53950 | > loss_text_ce: 0.09490476548671722 (0.10236701492264852) | > loss_mel_ce: 4.167272567749023 (nan) | > loss: 0.01691340282559395 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.13491688269658383) | > loader_time: 0.0045 (0.004485570526728479)  --> TIME: 2025-02-23 22:40:17 -- STEP: 17062/36938 -- GLOBAL_STEP: 54000 | > loss_text_ce: 0.08653075248003006 (0.10231792821985898) | > loss_mel_ce: 4.728095054626465 (nan) | > loss: 0.019105657935142517 (nan) | > current_lr: 5e-06 | > step_time: 0.1846 (0.1349046107347951) | > loader_time: 0.0043 (0.0044854543452497045)  --> TIME: 2025-02-23 22:40:26 -- STEP: 17112/36938 -- GLOBAL_STEP: 54050 | > loss_text_ce: 0.08716781437397003 (0.10227892415832295) | > loss_mel_ce: 3.952141523361206 (nan) | > loss: 0.016029007732868195 (nan) | > current_lr: 5e-06 | > step_time: 0.1796 (0.13490139351319058) | > loader_time: 0.0048 (0.00448551689034703)  --> TIME: 2025-02-23 22:40:36 -- STEP: 17162/36938 -- GLOBAL_STEP: 54100 | > loss_text_ce: 0.08687388151884079 (0.10223902742964616) | > loss_mel_ce: 1.6622488498687744 (nan) | > loss: 0.006940963678061962 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13491486641732964) | > loader_time: 0.0043 (0.004485485062234093)  --> TIME: 2025-02-23 22:40:44 -- STEP: 17212/36938 -- GLOBAL_STEP: 54150 | > loss_text_ce: 0.1364152580499649 (0.10220618070759693) | > loss_mel_ce: 2.735825300216675 (nan) | > loss: 0.01139778085052967 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.1348997834613641) | > loader_time: 0.0066 (0.004485102398517986)  --> TIME: 2025-02-23 22:40:54 -- STEP: 17262/36938 -- GLOBAL_STEP: 54200 | > loss_text_ce: 0.0824187844991684 (0.10217155374102323) | > loss_mel_ce: 2.692538261413574 (nan) | > loss: 0.01101173460483551 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13490076300415768) | > loader_time: 0.0046 (0.004484830263412062)  --> TIME: 2025-02-23 22:41:03 -- STEP: 17312/36938 -- GLOBAL_STEP: 54250 | > loss_text_ce: 0.07972197979688644 (0.10212685985321991) | > loss_mel_ce: 4.161192893981934 (nan) | > loss: 0.016829028725624084 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13490412683121078) | > loader_time: 0.0042 (0.0044846404171695105)  --> TIME: 2025-02-23 22:41:12 -- STEP: 17362/36938 -- GLOBAL_STEP: 54300 | > loss_text_ce: 0.0838816836476326 (0.10208572124593111) | > loss_mel_ce: 4.310019493103027 (nan) | > loss: 0.017436116933822632 (nan) | > current_lr: 5e-06 | > step_time: 0.182 (0.13491630854934472) | > loader_time: 0.0047 (0.0044842647003861084)  --> TIME: 2025-02-23 22:41:22 -- STEP: 17412/36938 -- GLOBAL_STEP: 54350 | > loss_text_ce: 0.07241519540548325 (0.10204795214233953) | > loss_mel_ce: 4.307596683502197 (nan) | > loss: 0.017381001263856888 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13490828918966974) | > loader_time: 0.0043 (0.004483963521409246)  --> TIME: 2025-02-23 22:41:31 -- STEP: 17462/36938 -- GLOBAL_STEP: 54400 | > loss_text_ce: 0.10118218511343002 (0.10201075504993813) | > loss_mel_ce: 4.369256496429443 (nan) | > loss: 0.017739836126565933 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13490783775020598) | > loader_time: 0.0042 (0.00448439643085651)  --> TIME: 2025-02-23 22:41:40 -- STEP: 17512/36938 -- GLOBAL_STEP: 54450 | > loss_text_ce: 0.07178651541471481 (0.1019687754599411) | > loss_mel_ce: 4.414566993713379 (nan) | > loss: 0.017802990972995758 (nan) | > current_lr: 5e-06 | > step_time: 0.1837 (0.13489423633002876) | > loader_time: 0.0046 (0.00448439954731114)  --> TIME: 2025-02-23 22:41:49 -- STEP: 17562/36938 -- GLOBAL_STEP: 54500 | > loss_text_ce: 0.08397365361452103 (0.10192601391938516) | > loss_mel_ce: 3.3750343322753906 (nan) | > loss: 0.01372622326016426 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.1348924968090857) | > loader_time: 0.005 (0.004484609174994474)  --> TIME: 2025-02-23 22:41:58 -- STEP: 17612/36938 -- GLOBAL_STEP: 54550 | > loss_text_ce: 0.07754894345998764 (0.10188858032306117) | > loss_mel_ce: 2.5516135692596436 (nan) | > loss: 0.010433184914290905 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.1348915202033366) | > loader_time: 0.0042 (0.004484960552564724)  --> TIME: 2025-02-23 22:42:07 -- STEP: 17662/36938 -- GLOBAL_STEP: 54600 | > loss_text_ce: 0.09638412296772003 (0.10185105693008252) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13488122014628887) | > loader_time: 0.0044 (0.004484800679026821)  --> TIME: 2025-02-23 22:42:16 -- STEP: 17712/36938 -- GLOBAL_STEP: 54650 | > loss_text_ce: 0.0931410938501358 (0.10181056583626685) | > loss_mel_ce: 4.217292308807373 (nan) | > loss: 0.017104895785450935 (nan) | > current_lr: 5e-06 | > step_time: 0.1077 (0.13487170800876197) | > loader_time: 0.0044 (0.004485046664335319)  --> TIME: 2025-02-23 22:42:25 -- STEP: 17762/36938 -- GLOBAL_STEP: 54700 | > loss_text_ce: 0.088668592274189 (0.10177077461474948) | > loss_mel_ce: 3.544976234436035 (nan) | > loss: 0.014419225975871086 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.1348641475515978) | > loader_time: 0.0045 (0.004485249613308841)  --> TIME: 2025-02-23 22:42:34 -- STEP: 17812/36938 -- GLOBAL_STEP: 54750 | > loss_text_ce: 0.0835593193769455 (0.10173060271371154) | > loss_mel_ce: 4.290634632110596 (nan) | > loss: 0.01735791377723217 (nan) | > current_lr: 5e-06 | > step_time: 0.1872 (0.13486424913680428) | > loader_time: 0.0044 (0.00448550326407102)  --> TIME: 2025-02-23 22:42:43 -- STEP: 17862/36938 -- GLOBAL_STEP: 54800 | > loss_text_ce: 0.08546575158834457 (0.10168886311830297) | > loss_mel_ce: 5.003623008728027 (nan) | > loss: 0.020194798707962036 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.13485582289163345) | > loader_time: 0.0043 (0.004485173917353285)  --> TIME: 2025-02-23 22:42:53 -- STEP: 17912/36938 -- GLOBAL_STEP: 54850 | > loss_text_ce: 0.07708875834941864 (0.10165080013915806) | > loss_mel_ce: 4.497786998748779 (nan) | > loss: 0.018154270946979523 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.13486778489869247) | > loader_time: 0.0043 (0.004485015093934159)  --> TIME: 2025-02-23 22:43:02 -- STEP: 17962/36938 -- GLOBAL_STEP: 54900 | > loss_text_ce: 0.08785092830657959 (0.10160419955980747) | > loss_mel_ce: 4.388365268707275 (nan) | > loss: 0.017762763425707817 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.1348719308695134) | > loader_time: 0.0046 (0.004484896125725487)  --> TIME: 2025-02-23 22:43:12 -- STEP: 18012/36938 -- GLOBAL_STEP: 54950 | > loss_text_ce: 0.09463586658239365 (0.10156164886796004) | > loss_mel_ce: 4.515592098236084 (nan) | > loss: 0.018294556066393852 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13486959859844566) | > loader_time: 0.0045 (0.004485528415609605)  --> TIME: 2025-02-23 22:43:21 -- STEP: 18062/36938 -- GLOBAL_STEP: 55000 | > loss_text_ce: 0.09323421120643616 (0.10151589461691497) | > loss_mel_ce: 4.379295825958252 (nan) | > loss: 0.017748136073350906 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13487861690007955) | > loader_time: 0.0043 (0.0044854745531171785)  --> TIME: 2025-02-23 22:43:30 -- STEP: 18112/36938 -- GLOBAL_STEP: 55050 | > loss_text_ce: 0.08898035436868668 (0.10147319229355917) | > loss_mel_ce: 3.597630262374878 (nan) | > loss: 0.014629408717155457 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13488082614046032) | > loader_time: 0.0044 (0.004485299751531097)  --> TIME: 2025-02-23 22:43:40 -- STEP: 18162/36938 -- GLOBAL_STEP: 55100 | > loss_text_ce: 0.07247911393642426 (0.10142798904219211) | > loss_mel_ce: 4.33839225769043 (nan) | > loss: 0.017503459006547928 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.1348897288935435) | > loader_time: 0.0044 (0.004484857891693121)  --> TIME: 2025-02-23 22:43:49 -- STEP: 18212/36938 -- GLOBAL_STEP: 55150 | > loss_text_ce: 0.08832193911075592 (0.10139041757483169) | > loss_mel_ce: 3.991628646850586 (nan) | > loss: 0.01619028113782406 (nan) | > current_lr: 5e-06 | > step_time: 0.1271 (0.1348766791880738) | > loader_time: 0.0042 (0.004484293410051727)  --> TIME: 2025-02-23 22:43:58 -- STEP: 18262/36938 -- GLOBAL_STEP: 55200 | > loss_text_ce: 0.07223866879940033 (0.10135519193541911) | > loss_mel_ce: 3.4124979972839355 (nan) | > loss: 0.013828321360051632 (nan) | > current_lr: 5e-06 | > step_time: 0.1688 (0.13488787913319755) | > loader_time: 0.0044 (0.0044839307755580975)  --> TIME: 2025-02-23 22:44:07 -- STEP: 18312/36938 -- GLOBAL_STEP: 55250 | > loss_text_ce: 0.0922081470489502 (0.1013085586977039) | > loss_mel_ce: 4.410579204559326 (nan) | > loss: 0.017868205904960632 (nan) | > current_lr: 5e-06 | > step_time: 0.176 (0.1348907823331383) | > loader_time: 0.0048 (0.004483789062333334)  --> TIME: 2025-02-23 22:44:17 -- STEP: 18362/36938 -- GLOBAL_STEP: 55300 | > loss_text_ce: 0.09293476492166519 (0.10127034527734037) | > loss_mel_ce: 4.450265884399414 (nan) | > loss: 0.01802857406437397 (nan) | > current_lr: 5e-06 | > step_time: 0.1694 (0.1348891917844564) | > loader_time: 0.0051 (0.004484177570615158)  --> TIME: 2025-02-23 22:44:26 -- STEP: 18412/36938 -- GLOBAL_STEP: 55350 | > loss_text_ce: 0.0815253034234047 (0.10122714395663791) | > loss_mel_ce: 0.15113741159439087 (nan) | > loss: 0.0009232647716999054 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.1348943860925015) | > loader_time: 0.0041 (0.004484334912425444)  --> TIME: 2025-02-23 22:44:35 -- STEP: 18462/36938 -- GLOBAL_STEP: 55400 | > loss_text_ce: 0.08868996053934097 (0.10119039733650954) | > loss_mel_ce: 3.948014497756958 (nan) | > loss: 0.016018670052289963 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13489575780441354) | > loader_time: 0.0047 (0.004484690742753008)  --> TIME: 2025-02-23 22:44:44 -- STEP: 18512/36938 -- GLOBAL_STEP: 55450 | > loss_text_ce: 0.08728519082069397 (0.10115018162865134) | > loss_mel_ce: 3.5614242553710938 (nan) | > loss: 0.014479006640613079 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1348918335851958) | > loader_time: 0.0046 (0.004484637876265797)  --> TIME: 2025-02-23 22:44:53 -- STEP: 18562/36938 -- GLOBAL_STEP: 55500 | > loss_text_ce: 0.08258312195539474 (0.1011118612338177) | > loss_mel_ce: 0.02587890625 (nan) | > loss: 0.00043040490709245205 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13488522568546307) | > loader_time: 0.0044 (0.00448433472516527)  --> TIME: 2025-02-23 22:45:02 -- STEP: 18612/36938 -- GLOBAL_STEP: 55550 | > loss_text_ce: 0.08769010007381439 (0.10106860279121054) | > loss_mel_ce: 4.2764739990234375 (nan) | > loss: 0.017318110913038254 (nan) | > current_lr: 5e-06 | > step_time: 0.1086 (0.13487969189337756) | > loader_time: 0.0044 (0.004484178595662791)  --> TIME: 2025-02-23 22:45:12 -- STEP: 18662/36938 -- GLOBAL_STEP: 55600 | > loss_text_ce: 0.07459647208452225 (0.10102908908737983) | > loss_mel_ce: 4.09083366394043 (nan) | > loss: 0.01652948558330536 (nan) | > current_lr: 5e-06 | > step_time: 0.1041 (0.13488071951857372) | > loader_time: 0.0041 (0.004484450289475122)  --> TIME: 2025-02-23 22:45:21 -- STEP: 18712/36938 -- GLOBAL_STEP: 55650 | > loss_text_ce: 0.08887632936239243 (0.10098877365216363) | > loss_mel_ce: 1.8185628652572632 (nan) | > loss: 0.007569203618913889 (nan) | > current_lr: 5e-06 | > step_time: 0.1059 (0.13487231137661737) | > loader_time: 0.0045 (0.004484578514771856)  --> TIME: 2025-02-23 22:45:30 -- STEP: 18762/36938 -- GLOBAL_STEP: 55700 | > loss_text_ce: 0.07920991629362106 (0.10094440773934842) | > loss_mel_ce: 4.479841232299805 (nan) | > loss: 0.01809147372841835 (nan) | > current_lr: 5e-06 | > step_time: 0.1833 (0.13486460422785793) | > loader_time: 0.0049 (0.004484679015025092)  --> TIME: 2025-02-23 22:45:39 -- STEP: 18812/36938 -- GLOBAL_STEP: 55750 | > loss_text_ce: 0.09107444435358047 (0.10090743674845805) | > loss_mel_ce: 2.3026821613311768 (nan) | > loss: 0.009499034844338894 (nan) | > current_lr: 5e-06 | > step_time: 0.1039 (0.13487205102599262) | > loader_time: 0.0044 (0.004484779031738516)  --> TIME: 2025-02-23 22:45:48 -- STEP: 18862/36938 -- GLOBAL_STEP: 55800 | > loss_text_ce: 0.09270434826612473 (0.10087220621242088) | > loss_mel_ce: 4.9453125 (nan) | > loss: 0.019992131739854813 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13484622497799192) | > loader_time: 0.0043 (0.0044848099580031865)  --> TIME: 2025-02-23 22:45:57 -- STEP: 18912/36938 -- GLOBAL_STEP: 55850 | > loss_text_ce: 0.07526557892560959 (0.10083150713863726) | > loss_mel_ce: 4.143552303314209 (nan) | > loss: 0.01674134097993374 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13484494453658438) | > loader_time: 0.0047 (0.004484996577809897)  --> TIME: 2025-02-23 22:46:06 -- STEP: 18962/36938 -- GLOBAL_STEP: 55900 | > loss_text_ce: 0.07943172007799149 (0.10078883692881555) | > loss_mel_ce: 3.926377773284912 (nan) | > loss: 0.015896068885922432 (nan) | > current_lr: 5e-06 | > step_time: 0.1569 (0.13484988258101424) | > loader_time: 0.0046 (0.004485258333366906)  --> TIME: 2025-02-23 22:46:15 -- STEP: 19012/36938 -- GLOBAL_STEP: 55950 | > loss_text_ce: 0.08266092836856842 (0.10074841455542193) | > loss_mel_ce: 3.5781121253967285 (nan) | > loss: 0.01452687755227089 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13484230774867945) | > loader_time: 0.0044 (0.004485264918991263)  --> TIME: 2025-02-23 22:46:25 -- STEP: 19062/36938 -- GLOBAL_STEP: 56000 | > loss_text_ce: 0.09302743524312973 (0.10070763014756516) | > loss_mel_ce: 3.25315523147583 (nan) | > loss: 0.013278502970933914 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13484488086652324) | > loader_time: 0.0044 (0.0044854484016179)  --> TIME: 2025-02-23 22:46:34 -- STEP: 19112/36938 -- GLOBAL_STEP: 56050 | > loss_text_ce: 0.08696988970041275 (0.10066877368115652) | > loss_mel_ce: 4.0254974365234375 (nan) | > loss: 0.016319315880537033 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13485777220899578) | > loader_time: 0.0041 (0.004485358212372397)  --> TIME: 2025-02-23 22:46:43 -- STEP: 19162/36938 -- GLOBAL_STEP: 56100 | > loss_text_ce: 0.09182456135749817 (0.1006335079613189) | > loss_mel_ce: 0.66583251953125 (nan) | > loss: 0.0030065758619457483 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.1348546439055265) | > loader_time: 0.0046 (0.004485305497074224)  --> TIME: 2025-02-23 22:46:53 -- STEP: 19212/36938 -- GLOBAL_STEP: 56150 | > loss_text_ce: 0.09101227670907974 (0.1005968055508488) | > loss_mel_ce: 3.7174105644226074 (nan) | > loss: 0.01511278934776783 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13485159052180176) | > loader_time: 0.0043 (0.0044851133457551465)  --> TIME: 2025-02-23 22:47:02 -- STEP: 19262/36938 -- GLOBAL_STEP: 56200 | > loss_text_ce: 0.07716822624206543 (0.10055936036345085) | > loss_mel_ce: 3.2251641750335693 (nan) | > loss: 0.013104494661092758 (nan) | > current_lr: 5e-06 | > step_time: 0.1025 (0.13485722100669015) | > loader_time: 0.0042 (0.004484922068226231)  --> TIME: 2025-02-23 22:47:12 -- STEP: 19312/36938 -- GLOBAL_STEP: 56250 | > loss_text_ce: 0.08058037608861923 (0.10052312847169807) | > loss_mel_ce: 4.030190467834473 (nan) | > loss: 0.016312582418322563 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.13487633297338616) | > loader_time: 0.0044 (0.004485411296329315)  --> TIME: 2025-02-23 22:47:21 -- STEP: 19362/36938 -- GLOBAL_STEP: 56300 | > loss_text_ce: 0.07766919583082199 (0.10048417752696627) | > loss_mel_ce: 4.180888652801514 (nan) | > loss: 0.01689903996884823 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13488841163953164) | > loader_time: 0.0046 (0.00448544491372956)  --> TIME: 2025-02-23 22:47:31 -- STEP: 19412/36938 -- GLOBAL_STEP: 56350 | > loss_text_ce: 0.09022165089845657 (0.10044949626472972) | > loss_mel_ce: 4.113867282867432 (nan) | > loss: 0.01668289490044117 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13489472006316844) | > loader_time: 0.0044 (0.004485476270007899)  --> TIME: 2025-02-23 22:47:40 -- STEP: 19462/36938 -- GLOBAL_STEP: 56400 | > loss_text_ce: 0.09831054508686066 (0.10041574893040603) | > loss_mel_ce: 3.417996883392334 (nan) | > loss: 0.013953601010143757 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13490138883495545) | > loader_time: 0.0044 (0.004485888626185649)  --> TIME: 2025-02-23 22:47:49 -- STEP: 19512/36938 -- GLOBAL_STEP: 56450 | > loss_text_ce: 0.09490136802196503 (0.10037755865646499) | > loss_mel_ce: 2.1418941020965576 (nan) | > loss: 0.008876172825694084 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13490573844981912) | > loader_time: 0.0049 (0.004486356457514359)  --> TIME: 2025-02-23 22:47:59 -- STEP: 19562/36938 -- GLOBAL_STEP: 56500 | > loss_text_ce: 0.08892079442739487 (0.10033868442828221) | > loss_mel_ce: 3.315619707107544 (nan) | > loss: 0.0135100819170475 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13490668305735365) | > loader_time: 0.0046 (0.004486746710510212)  --> TIME: 2025-02-23 22:48:08 -- STEP: 19612/36938 -- GLOBAL_STEP: 56550 | > loss_text_ce: 0.08363127708435059 (0.10029428798303457) | > loss_mel_ce: 5.003443717956543 (nan) | > loss: 0.020186807960271835 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.13490634821104508) | > loader_time: 0.0046 (0.00448683111516402)  --> TIME: 2025-02-23 22:48:17 -- STEP: 19662/36938 -- GLOBAL_STEP: 56600 | > loss_text_ce: 0.07368108630180359 (0.10024904102153762) | > loss_mel_ce: 4.578289031982422 (nan) | > loss: 0.018460199236869812 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13490937252323953) | > loader_time: 0.0044 (0.004486726957885154)  --> TIME: 2025-02-23 22:48:26 -- STEP: 19712/36938 -- GLOBAL_STEP: 56650 | > loss_text_ce: 0.0971360057592392 (0.10021128957125847) | > loss_mel_ce: 0.7955754399299622 (nan) | > loss: 0.0035425059031695127 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.134908608039961) | > loader_time: 0.0045 (0.00448676516822018)  --> TIME: 2025-02-23 22:48:36 -- STEP: 19762/36938 -- GLOBAL_STEP: 56700 | > loss_text_ce: 0.09345239400863647 (0.1001721381266065) | > loss_mel_ce: 4.211467742919922 (nan) | > loss: 0.01708301715552807 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13490670672699268) | > loader_time: 0.0044 (0.004486931273961581)  --> TIME: 2025-02-23 22:48:45 -- STEP: 19812/36938 -- GLOBAL_STEP: 56750 | > loss_text_ce: 0.08651968836784363 (0.10013540253865305) | > loss_mel_ce: 2.9866535663604736 (nan) | > loss: 0.012195132672786713 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.1349123069524223) | > loader_time: 0.0049 (0.004487350158876237)  --> TIME: 2025-02-23 22:48:54 -- STEP: 19862/36938 -- GLOBAL_STEP: 56800 | > loss_text_ce: 0.08237631618976593 (0.10009178882456381) | > loss_mel_ce: 3.7423672676086426 (nan) | > loss: 0.015177554450929165 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.1349054514947642) | > loader_time: 0.0043 (0.004487496034074615)  --> TIME: 2025-02-23 22:49:03 -- STEP: 19912/36938 -- GLOBAL_STEP: 56850 | > loss_text_ce: 0.07393375039100647 (0.10005504734512632) | > loss_mel_ce: 4.464466094970703 (nan) | > loss: 0.018009522929787636 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13490227878979624) | > loader_time: 0.0048 (0.004487723962232821)  --> TIME: 2025-02-23 22:49:13 -- STEP: 19962/36938 -- GLOBAL_STEP: 56900 | > loss_text_ce: 0.07970546931028366 (0.10001980557133824) | > loss_mel_ce: 4.331143856048584 (nan) | > loss: 0.017503369599580765 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.13490410953948112) | > loader_time: 0.0043 (0.004487718684008513)  --> TIME: 2025-02-23 22:49:22 -- STEP: 20012/36938 -- GLOBAL_STEP: 56950 | > loss_text_ce: 0.09302809834480286 (0.09998104137811777) | > loss_mel_ce: 3.928793430328369 (nan) | > loss: 0.015959609299898148 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13490655141998426) | > loader_time: 0.0044 (0.004487630679038853)  --> TIME: 2025-02-23 22:49:31 -- STEP: 20062/36938 -- GLOBAL_STEP: 57000 | > loss_text_ce: 0.0821404904127121 (0.09993704308748041) | > loss_mel_ce: 4.460952281951904 (nan) | > loss: 0.018028147518634796 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.1349206450762471) | > loader_time: 0.0045 (0.0044876375912360185)  --> TIME: 2025-02-23 22:49:41 -- STEP: 20112/36938 -- GLOBAL_STEP: 57050 | > loss_text_ce: 0.08510858565568924 (0.09989285508045558) | > loss_mel_ce: 4.085358142852783 (nan) | > loss: 0.016549473628401756 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.1349365920444836) | > loader_time: 0.0045 (0.0044879560775953975)  --> TIME: 2025-02-23 22:49:50 -- STEP: 20162/36938 -- GLOBAL_STEP: 57100 | > loss_text_ce: 0.08643784373998642 (0.0998543736167082) | > loss_mel_ce: 2.6735568046569824 (nan) | > loss: 0.010952360928058624 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13492578715632744) | > loader_time: 0.0044 (0.004488791848042564)  --> TIME: 2025-02-23 22:49:59 -- STEP: 20212/36938 -- GLOBAL_STEP: 57150 | > loss_text_ce: 0.08533450216054916 (0.09981446167446431) | > loss_mel_ce: 4.1788153648376465 (nan) | > loss: 0.016921229660511017 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13491972396804378) | > loader_time: 0.0044 (0.004488913960535163)  --> TIME: 2025-02-23 22:50:08 -- STEP: 20262/36938 -- GLOBAL_STEP: 57200 | > loss_text_ce: 0.0868980884552002 (0.09977700723464449) | > loss_mel_ce: 3.1293511390686035 (nan) | > loss: 0.012762894853949547 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.1349133194969605) | > loader_time: 0.0044 (0.004489220656008153)  --> TIME: 2025-02-23 22:50:18 -- STEP: 20312/36938 -- GLOBAL_STEP: 57250 | > loss_text_ce: 0.07493068277835846 (0.0997322945989081) | > loss_mel_ce: 4.50303840637207 (nan) | > loss: 0.018166545778512955 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13491970752580315) | > loader_time: 0.0042 (0.004489290404385487)  --> TIME: 2025-02-23 22:50:27 -- STEP: 20362/36938 -- GLOBAL_STEP: 57300 | > loss_text_ce: 0.09238452464342117 (0.09969355906589057) | > loss_mel_ce: 3.663238763809204 (nan) | > loss: 0.014903267845511436 (nan) | > current_lr: 5e-06 | > step_time: 0.1574 (0.13493253521759394) | > loader_time: 0.0045 (0.0044897474482466625)  --> TIME: 2025-02-23 22:50:37 -- STEP: 20412/36938 -- GLOBAL_STEP: 57350 | > loss_text_ce: 0.07462499290704727 (0.09965588026480261) | > loss_mel_ce: 4.330458164215088 (nan) | > loss: 0.01748049072921276 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13493194547139503) | > loader_time: 0.0044 (0.004489810950798583)  --> TIME: 2025-02-23 22:50:46 -- STEP: 20462/36938 -- GLOBAL_STEP: 57400 | > loss_text_ce: 0.08666230738162994 (0.09961400215172969) | > loss_mel_ce: 4.436838626861572 (nan) | > loss: 0.017950400710105896 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.1349306673309608) | > loader_time: 0.0046 (0.0044899652715233085)  --> TIME: 2025-02-23 22:50:55 -- STEP: 20512/36938 -- GLOBAL_STEP: 57450 | > loss_text_ce: 0.07107307016849518 (0.09957503603427036) | > loss_mel_ce: 3.7361395359039307 (nan) | > loss: 0.015107987448573112 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13491755327596178) | > loader_time: 0.0054 (0.0044902022724590445)  --> TIME: 2025-02-23 22:51:03 -- STEP: 20562/36938 -- GLOBAL_STEP: 57500 | > loss_text_ce: 0.11159316450357437 (0.09953641553844539) | > loss_mel_ce: 1.6284600496292114 (nan) | > loss: 0.006904973182827234 (nan) | > current_lr: 5e-06 | > step_time: 0.1053 (0.13490139529023362) | > loader_time: 0.0042 (0.0044902165035087815)  --> TIME: 2025-02-23 22:51:13 -- STEP: 20612/36938 -- GLOBAL_STEP: 57550 | > loss_text_ce: 0.09275621175765991 (0.09950100553601414) | > loss_mel_ce: 2.1937062740325928 (nan) | > loss: 0.00907326489686966 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13490992022236062) | > loader_time: 0.0043 (0.00449025218009763)  --> TIME: 2025-02-23 22:51:22 -- STEP: 20662/36938 -- GLOBAL_STEP: 57600 | > loss_text_ce: 0.06581708043813705 (0.09945416132962002) | > loss_mel_ce: 3.682915210723877 (nan) | > loss: 0.014875922352075577 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13490904987221733) | > loader_time: 0.0046 (0.004490395135079169)  --> TIME: 2025-02-23 22:51:31 -- STEP: 20712/36938 -- GLOBAL_STEP: 57650 | > loss_text_ce: 0.08582627028226852 (0.09941118418707828) | > loss_mel_ce: 4.880742073059082 (nan) | > loss: 0.019708605483174324 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13491179393773348) | > loader_time: 0.0051 (0.004490604567315962)  --> TIME: 2025-02-23 22:51:41 -- STEP: 20762/36938 -- GLOBAL_STEP: 57700 | > loss_text_ce: 0.08393781632184982 (0.0993757151022764) | > loss_mel_ce: 4.4068145751953125 (nan) | > loss: 0.0178204458206892 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.13490641887632704) | > loader_time: 0.0045 (0.0044907720524639295)  --> TIME: 2025-02-23 22:51:50 -- STEP: 20812/36938 -- GLOBAL_STEP: 57750 | > loss_text_ce: 0.0873953327536583 (0.09933574742639596) | > loss_mel_ce: 3.316042184829712 (nan) | > loss: 0.013505705632269382 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13491537263708087) | > loader_time: 0.0044 (0.004490767651605024)  --> TIME: 2025-02-23 22:51:59 -- STEP: 20862/36938 -- GLOBAL_STEP: 57800 | > loss_text_ce: 0.08554892987012863 (0.09929510822830243) | > loss_mel_ce: 0.11892218142747879 (nan) | > loss: 0.0008113933727145195 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1349141362323444) | > loader_time: 0.0044 (0.0044905841893402776)  --> TIME: 2025-02-23 22:52:09 -- STEP: 20912/36938 -- GLOBAL_STEP: 57850 | > loss_text_ce: 0.0707039162516594 (0.09925613725663435) | > loss_mel_ce: 4.293811321258545 (nan) | > loss: 0.017319506034255028 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1349202469048275) | > loader_time: 0.0045 (0.004490586951112779)  --> TIME: 2025-02-23 22:52:18 -- STEP: 20962/36938 -- GLOBAL_STEP: 57900 | > loss_text_ce: 0.07486046850681305 (0.0992186974996027) | > loss_mel_ce: 4.736878395080566 (nan) | > loss: 0.019094202667474747 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.13493066490998723) | > loader_time: 0.0045 (0.004490850081187642)  --> TIME: 2025-02-23 22:52:27 -- STEP: 21012/36938 -- GLOBAL_STEP: 57950 | > loss_text_ce: 0.07725591212511063 (0.0991783194235403) | > loss_mel_ce: 4.571077823638916 (nan) | > loss: 0.01844576932489872 (nan) | > current_lr: 5e-06 | > step_time: 0.107 (0.13492659400944937) | > loader_time: 0.0043 (0.004491006138890706)  --> TIME: 2025-02-23 22:52:37 -- STEP: 21062/36938 -- GLOBAL_STEP: 58000 | > loss_text_ce: 0.07979027926921844 (0.099145644739258) | > loss_mel_ce: 4.6652655601501465 (nan) | > loss: 0.0188295878469944 (nan) | > current_lr: 5e-06 | > step_time: 0.1626 (0.13493531232047182) | > loader_time: 0.0046 (0.004491020274540369)  --> TIME: 2025-02-23 22:52:46 -- STEP: 21112/36938 -- GLOBAL_STEP: 58050 | > loss_text_ce: 0.09297042340040207 (0.0991090702689842) | > loss_mel_ce: 4.608336448669434 (nan) | > loss: 0.01865598000586033 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13493214291846772) | > loader_time: 0.0046 (0.004491006867277565)  --> TIME: 2025-02-23 22:52:55 -- STEP: 21162/36938 -- GLOBAL_STEP: 58100 | > loss_text_ce: 0.08233284950256348 (0.0990686924513211) | > loss_mel_ce: 4.17079496383667 (nan) | > loss: 0.016877492889761925 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.1349367449948531) | > loader_time: 0.0045 (0.00449140890256836)  --> TIME: 2025-02-23 22:53:04 -- STEP: 21212/36938 -- GLOBAL_STEP: 58150 | > loss_text_ce: 0.0895303338766098 (0.09903422517408043) | > loss_mel_ce: 4.196961879730225 (nan) | > loss: 0.01700989156961441 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13493435963490855) | > loader_time: 0.0045 (0.004491432745367227)  --> TIME: 2025-02-23 22:53:14 -- STEP: 21262/36938 -- GLOBAL_STEP: 58200 | > loss_text_ce: 0.0961158275604248 (0.09899463721496475) | > loss_mel_ce: 3.7426278591156006 (nan) | > loss: 0.015233110636472702 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.13495300955633727) | > loader_time: 0.0047 (0.004491704033500065)  --> TIME: 2025-02-23 22:53:23 -- STEP: 21312/36938 -- GLOBAL_STEP: 58250 | > loss_text_ce: 0.08724526315927505 (0.0989568542165396) | > loss_mel_ce: 2.2305474281311035 (nan) | > loss: 0.009197590872645378 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1349460886114524) | > loader_time: 0.0045 (0.004491705615241255)  --> TIME: 2025-02-23 22:53:32 -- STEP: 21362/36938 -- GLOBAL_STEP: 58300 | > loss_text_ce: 0.0881861001253128 (0.0989164531293014) | > loss_mel_ce: 3.3626174926757812 (nan) | > loss: 0.013693665154278278 (nan) | > current_lr: 5e-06 | > step_time: 0.1084 (0.13494046786206154) | > loader_time: 0.0044 (0.004491998544179958)  --> TIME: 2025-02-23 22:53:41 -- STEP: 21412/36938 -- GLOBAL_STEP: 58350 | > loss_text_ce: 0.08163995295763016 (0.09887912906476728) | > loss_mel_ce: 3.802267074584961 (nan) | > loss: 0.015412330627441406 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13493445341372742) | > loader_time: 0.0046 (0.004492087117449368)  --> TIME: 2025-02-23 22:53:51 -- STEP: 21462/36938 -- GLOBAL_STEP: 58400 | > loss_text_ce: 0.08240674436092377 (0.09883770446413134) | > loss_mel_ce: 4.027703285217285 (nan) | > loss: 0.016309959813952446 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.1349451912270369) | > loader_time: 0.0043 (0.0044920863737298875)  --> TIME: 2025-02-23 22:54:00 -- STEP: 21512/36938 -- GLOBAL_STEP: 58450 | > loss_text_ce: 0.0928378775715828 (0.0988006627796013) | > loss_mel_ce: 4.763848781585693 (nan) | > loss: 0.019272567704319954 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13493325470681006) | > loader_time: 0.0045 (0.004491930814337319)  --> TIME: 2025-02-23 22:54:09 -- STEP: 21562/36938 -- GLOBAL_STEP: 58500 | > loss_text_ce: 0.10912229120731354 (0.09875827937452375) | > loss_mel_ce: 3.8059298992156982 (nan) | > loss: 0.015535921789705753 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.1349378494724973) | > loader_time: 0.0043 (0.004491625375290352)  --> TIME: 2025-02-23 22:54:18 -- STEP: 21612/36938 -- GLOBAL_STEP: 58550 | > loss_text_ce: 0.07152450084686279 (0.09871331025632095) | > loss_mel_ce: 4.248836040496826 (nan) | > loss: 0.01714428886771202 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.13492705908921218) | > loader_time: 0.0048 (0.004491546640302573)  --> TIME: 2025-02-23 22:54:27 -- STEP: 21662/36938 -- GLOBAL_STEP: 58600 | > loss_text_ce: 0.078535296022892 (0.09867619251564654) | > loss_mel_ce: 4.649766445159912 (nan) | > loss: 0.01876310259103775 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13492606554805256) | > loader_time: 0.0047 (0.004492086768095301)  --> TIME: 2025-02-23 22:54:37 -- STEP: 21712/36938 -- GLOBAL_STEP: 58650 | > loss_text_ce: 0.08500032871961594 (0.09863795702021096) | > loss_mel_ce: 5.181507587432861 (nan) | > loss: 0.020898843184113503 (nan) | > current_lr: 5e-06 | > step_time: 0.1459 (0.1349315879071726) | > loader_time: 0.0048 (0.004492449316806275)  --> TIME: 2025-02-23 22:54:46 -- STEP: 21762/36938 -- GLOBAL_STEP: 58700 | > loss_text_ce: 0.09044170379638672 (0.0986031257174178) | > loss_mel_ce: 4.15234375 (nan) | > loss: 0.01683645136654377 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13493376349648997) | > loader_time: 0.0046 (0.004493122076289247)  --> TIME: 2025-02-23 22:54:55 -- STEP: 21812/36938 -- GLOBAL_STEP: 58750 | > loss_text_ce: 0.086004838347435 (0.098562590297269) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13492882283438204) | > loader_time: 0.0043 (0.004493250030127222)  --> TIME: 2025-02-23 22:55:04 -- STEP: 21862/36938 -- GLOBAL_STEP: 58800 | > loss_text_ce: 0.07388711720705032 (0.0985254963699574) | > loss_mel_ce: 3.803868293762207 (nan) | > loss: 0.015387918800115585 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1349346759977778) | > loader_time: 0.0047 (0.00449465591858862)  --> TIME: 2025-02-23 22:55:13 -- STEP: 21912/36938 -- GLOBAL_STEP: 58850 | > loss_text_ce: 0.08273623883724213 (0.09848874319051029) | > loss_mel_ce: 4.199104309082031 (nan) | > loss: 0.016991430893540382 (nan) | > current_lr: 5e-06 | > step_time: 0.1832 (0.1349239857060136) | > loader_time: 0.0055 (0.004496166428844173)  --> TIME: 2025-02-23 22:55:23 -- STEP: 21962/36938 -- GLOBAL_STEP: 58900 | > loss_text_ce: 0.07159864902496338 (0.09845142561578375) | > loss_mel_ce: 1.1188232898712158 (nan) | > loss: 0.004723896738141775 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1349255272452152) | > loader_time: 0.0051 (0.004497434899472825)  --> TIME: 2025-02-23 22:55:32 -- STEP: 22012/36938 -- GLOBAL_STEP: 58950 | > loss_text_ce: 0.08031100779771805 (0.09841050556873607) | > loss_mel_ce: 4.685645580291748 (nan) | > loss: 0.018912525847554207 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.13493719493911627) | > loader_time: 0.0046 (0.004498181561004368)  --> TIME: 2025-02-23 22:55:42 -- STEP: 22062/36938 -- GLOBAL_STEP: 59000 | > loss_text_ce: 0.06809844821691513 (0.09836962398683023) | > loss_mel_ce: 0.0835774764418602 (nan) | > loss: 0.0006018886342644691 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13496649191803295) | > loader_time: 0.0046 (0.004500816387317948)  --> TIME: 2025-02-23 22:55:52 -- STEP: 22112/36938 -- GLOBAL_STEP: 59050 | > loss_text_ce: 0.07516305148601532 (0.0983317062267486) | > loss_mel_ce: 4.602952480316162 (nan) | > loss: 0.018563950434327126 (nan) | > current_lr: 5e-06 | > step_time: 0.1944 (0.13498605929002322) | > loader_time: 0.0044 (0.004500684879623165)  --> TIME: 2025-02-23 22:56:01 -- STEP: 22162/36938 -- GLOBAL_STEP: 59100 | > loss_text_ce: 0.090707927942276 (0.09829852257961773) | > loss_mel_ce: 3.9146623611450195 (nan) | > loss: 0.0158943273127079 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13497974430220464) | > loader_time: 0.0043 (0.0045002594976129885)  --> TIME: 2025-02-23 22:56:10 -- STEP: 22212/36938 -- GLOBAL_STEP: 59150 | > loss_text_ce: 0.08154691755771637 (0.09826173099566635) | > loss_mel_ce: 1.6199618577957153 (nan) | > loss: 0.00675201928243041 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13498306187470402) | > loader_time: 0.0044 (0.004500157464286897)  --> TIME: 2025-02-23 22:56:20 -- STEP: 22262/36938 -- GLOBAL_STEP: 59200 | > loss_text_ce: 0.07822494953870773 (0.09822343618923406) | > loss_mel_ce: 3.785590887069702 (nan) | > loss: 0.015332602895796299 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13498086064408582) | > loader_time: 0.0045 (0.004500423723470485)  --> TIME: 2025-02-23 22:56:29 -- STEP: 22312/36938 -- GLOBAL_STEP: 59250 | > loss_text_ce: 0.0721052885055542 (0.0981840609585188) | > loss_mel_ce: 4.797913551330566 (nan) | > loss: 0.01932547241449356 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13497711767770684) | > loader_time: 0.0044 (0.004500302320827756)  --> TIME: 2025-02-23 22:56:38 -- STEP: 22362/36938 -- GLOBAL_STEP: 59300 | > loss_text_ce: 0.07264737784862518 (0.0981452906641382) | > loss_mel_ce: 3.5866539478302 (nan) | > loss: 0.014521037228405476 (nan) | > current_lr: 5e-06 | > step_time: 0.1037 (0.13497501246543767) | > loader_time: 0.0041 (0.004500100133073806)  --> TIME: 2025-02-23 22:56:47 -- STEP: 22412/36938 -- GLOBAL_STEP: 59350 | > loss_text_ce: 0.08810439705848694 (0.09810771405239484) | > loss_mel_ce: 3.7616159915924072 (nan) | > loss: 0.015276669524610043 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13497043674000145) | > loader_time: 0.0047 (0.004500123521759534)  --> TIME: 2025-02-23 22:56:56 -- STEP: 22462/36938 -- GLOBAL_STEP: 59400 | > loss_text_ce: 0.07334780693054199 (0.09807341217274293) | > loss_mel_ce: 4.15804386138916 (nan) | > loss: 0.016791239380836487 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1349654263819998) | > loader_time: 0.0045 (0.004500144768372612)  --> TIME: 2025-02-23 22:57:06 -- STEP: 22512/36938 -- GLOBAL_STEP: 59450 | > loss_text_ce: 0.07701455056667328 (0.09803823179677283) | > loss_mel_ce: 3.509838342666626 (nan) | > loss: 0.014233543537557125 (nan) | > current_lr: 5e-06 | > step_time: 0.1771 (0.13497397301462993) | > loader_time: 0.005 (0.004500101687811584)  --> TIME: 2025-02-23 22:57:15 -- STEP: 22562/36938 -- GLOBAL_STEP: 59500 | > loss_text_ce: 0.08768286556005478 (0.0980000984814961) | > loss_mel_ce: 4.604963779449463 (nan) | > loss: 0.018621614202857018 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.13499225387441255) | > loader_time: 0.0044 (0.004500554487348268)  --> TIME: 2025-02-23 22:57:25 -- STEP: 22612/36938 -- GLOBAL_STEP: 59550 | > loss_text_ce: 0.06628433614969254 (0.09795728138113084) | > loss_mel_ce: 4.3423991203308105 (nan) | > loss: 0.017494775354862213 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13499246507371573) | > loader_time: 0.0043 (0.004500563358092292)  --> TIME: 2025-02-23 22:57:33 -- STEP: 22662/36938 -- GLOBAL_STEP: 59600 | > loss_text_ce: 0.07153349369764328 (0.09791958186088036) | > loss_mel_ce: 3.806332588195801 (nan) | > loss: 0.015388358384370804 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1349777383676379) | > loader_time: 0.0046 (0.004500298200854233)  --> TIME: 2025-02-23 22:57:43 -- STEP: 22712/36938 -- GLOBAL_STEP: 59650 | > loss_text_ce: 0.08166848123073578 (0.09787515987382674) | > loss_mel_ce: 3.772980213165283 (nan) | > loss: 0.015296225436031818 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13497907321148134) | > loader_time: 0.0042 (0.004500426932009389)  --> TIME: 2025-02-23 22:57:52 -- STEP: 22762/36938 -- GLOBAL_STEP: 59700 | > loss_text_ce: 0.10395708680152893 (0.09784320198302629) | > loss_mel_ce: 3.7294232845306396 (nan) | > loss: 0.015211828052997589 (nan) | > current_lr: 5e-06 | > step_time: 0.1515 (0.13498302411450588) | > loader_time: 0.0044 (0.004500422145892774)  --> TIME: 2025-02-23 22:58:01 -- STEP: 22812/36938 -- GLOBAL_STEP: 59750 | > loss_text_ce: 0.08203422278165817 (0.0978076369350599) | > loss_mel_ce: 4.889993667602539 (nan) | > loss: 0.01973026990890503 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13498300783219774) | > loader_time: 0.0045 (0.00450031610616315)  --> TIME: 2025-02-23 22:58:11 -- STEP: 22862/36938 -- GLOBAL_STEP: 59800 | > loss_text_ce: 0.08771000802516937 (0.09777423301350799) | > loss_mel_ce: 3.2960903644561768 (nan) | > loss: 0.013427780009806156 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13498369977497252) | > loader_time: 0.0044 (0.004500581298159352)  --> TIME: 2025-02-23 22:58:20 -- STEP: 22912/36938 -- GLOBAL_STEP: 59850 | > loss_text_ce: 0.07665969431400299 (0.09773577513728031) | > loss_mel_ce: 2.76619029045105 (nan) | > loss: 0.011281151324510574 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.13497725095828703) | > loader_time: 0.0046 (0.0045004884956935205)  --> TIME: 2025-02-23 22:58:29 -- STEP: 22962/36938 -- GLOBAL_STEP: 59900 | > loss_text_ce: 0.07181944698095322 (0.09769661590121503) | > loss_mel_ce: 4.410649299621582 (nan) | > loss: 0.017787575721740723 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13496586611104452) | > loader_time: 0.0045 (0.004500402451890565)  --> TIME: 2025-02-23 22:58:38 -- STEP: 23012/36938 -- GLOBAL_STEP: 59950 | > loss_text_ce: 0.08755640685558319 (0.09765944574122194) | > loss_mel_ce: 3.813213586807251 (nan) | > loss: 0.015479247085750103 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13496659536807507) | > loader_time: 0.0043 (0.0045004277649535775)  --> TIME: 2025-02-23 22:58:47 -- STEP: 23062/36938 -- GLOBAL_STEP: 60000 | > loss_text_ce: 0.06591630727052689 (0.09762274493645029) | > loss_mel_ce: 3.11755108833313 (nan) | > loss: 0.012632807716727257 (nan) | > current_lr: 5e-06 | > step_time: 0.108 (0.13496046550730117) | > loader_time: 0.0042 (0.004500364132471687) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_60000.pth  --> TIME: 2025-02-23 22:59:02 -- STEP: 23112/36938 -- GLOBAL_STEP: 60050 | > loss_text_ce: 0.0726260393857956 (0.09758402799169476) | > loss_mel_ce: 4.064255714416504 (nan) | > loss: 0.016416199505329132 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13496041394880937) | > loader_time: 0.006 (0.004501210886039876)  --> TIME: 2025-02-23 22:59:10 -- STEP: 23162/36938 -- GLOBAL_STEP: 60100 | > loss_text_ce: 0.08283005654811859 (0.09754366867225633) | > loss_mel_ce: 3.8483927249908447 (nan) | > loss: 0.015600090846419334 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.1349499882960822) | > loader_time: 0.0045 (0.004501675913014994)  --> TIME: 2025-02-23 22:59:19 -- STEP: 23212/36938 -- GLOBAL_STEP: 60150 | > loss_text_ce: 0.08071977645158768 (0.09750589007017794) | > loss_mel_ce: 2.909623622894287 (nan) | > loss: 0.011866442859172821 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13494540355790088) | > loader_time: 0.0044 (0.00450170553698121)  --> TIME: 2025-02-23 22:59:29 -- STEP: 23262/36938 -- GLOBAL_STEP: 60200 | > loss_text_ce: 0.08721138536930084 (0.09746642344182976) | > loss_mel_ce: 1.9469022750854492 (nan) | > loss: 0.008071879856288433 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.1349411470012083) | > loader_time: 0.0047 (0.00450246954771876)  --> TIME: 2025-02-23 22:59:38 -- STEP: 23312/36938 -- GLOBAL_STEP: 60250 | > loss_text_ce: 0.08523447066545486 (0.09742882314145193) | > loss_mel_ce: 4.182022571563721 (nan) | > loss: 0.016933562234044075 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13493749301934205) | > loader_time: 0.0045 (0.004503091097941686)  --> TIME: 2025-02-23 22:59:48 -- STEP: 23362/36938 -- GLOBAL_STEP: 60300 | > loss_text_ce: 0.07932348549365997 (0.09739082283230813) | > loss_mel_ce: 4.499159812927246 (nan) | > loss: 0.018168585374951363 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13497707236763173) | > loader_time: 0.0044 (0.00450787072965068)  --> TIME: 2025-02-23 22:59:57 -- STEP: 23412/36938 -- GLOBAL_STEP: 60350 | > loss_text_ce: 0.08876722306013107 (0.09735656197090752) | > loss_mel_ce: 0.0843854621052742 (nan) | > loss: 0.0006871138466522098 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13497809284918244) | > loader_time: 0.0047 (0.004507870352738702)  --> TIME: 2025-02-23 23:00:06 -- STEP: 23462/36938 -- GLOBAL_STEP: 60400 | > loss_text_ce: 0.07558321952819824 (0.09731761946849996) | > loss_mel_ce: 4.724110126495361 (nan) | > loss: 0.019046401605010033 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13497813844831105) | > loader_time: 0.0045 (0.0045079123728944594)  --> TIME: 2025-02-23 23:00:16 -- STEP: 23512/36938 -- GLOBAL_STEP: 60450 | > loss_text_ce: 0.07722301781177521 (0.09727710331104142) | > loss_mel_ce: 2.0053699016571045 (nan) | > loss: 0.00826425850391388 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.13498924789723926) | > loader_time: 0.0054 (0.004508016404751899)  --> TIME: 2025-02-23 23:00:25 -- STEP: 23562/36938 -- GLOBAL_STEP: 60500 | > loss_text_ce: 0.08349783718585968 (0.09724131908892933) | > loss_mel_ce: 3.6977286338806152 (nan) | > loss: 0.015004867687821388 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13499227104198258) | > loader_time: 0.0044 (0.004508020143012685)  --> TIME: 2025-02-23 23:00:34 -- STEP: 23612/36938 -- GLOBAL_STEP: 60550 | > loss_text_ce: 0.0690251961350441 (0.09720774803310048) | > loss_mel_ce: 4.055870532989502 (nan) | > loss: 0.01636863499879837 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1349872266581832) | > loader_time: 0.0045 (0.004508022673954383)  --> TIME: 2025-02-23 23:00:43 -- STEP: 23662/36938 -- GLOBAL_STEP: 60600 | > loss_text_ce: 0.06660939007997513 (0.09717158756456863) | > loss_mel_ce: 4.404693603515625 (nan) | > loss: 0.017743267118930817 (nan) | > current_lr: 5e-06 | > step_time: 0.1529 (0.13498377753638052) | > loader_time: 0.0045 (0.0045080945473855456)  --> TIME: 2025-02-23 23:00:52 -- STEP: 23712/36938 -- GLOBAL_STEP: 60650 | > loss_text_ce: 0.08244767040014267 (0.09714014960189651) | > loss_mel_ce: 3.320462703704834 (nan) | > loss: 0.013503613881766796 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13498320275673734) | > loader_time: 0.0043 (0.004507953519763254)  --> TIME: 2025-02-23 23:01:02 -- STEP: 23762/36938 -- GLOBAL_STEP: 60700 | > loss_text_ce: 0.06707298755645752 (0.09710210042011502) | > loss_mel_ce: 3.1394851207733154 (nan) | > loss: 0.01272443775087595 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13499892146265607) | > loader_time: 0.0044 (0.004507927649370922)  --> TIME: 2025-02-23 23:01:11 -- STEP: 23812/36938 -- GLOBAL_STEP: 60750 | > loss_text_ce: 0.07764828205108643 (0.0970639475037275) | > loss_mel_ce: 4.0555419921875 (nan) | > loss: 0.01640154980123043 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13499874537399134) | > loader_time: 0.0043 (0.004507824911222965)  --> TIME: 2025-02-23 23:01:20 -- STEP: 23862/36938 -- GLOBAL_STEP: 60800 | > loss_text_ce: 0.0748799741268158 (0.09702449469466451) | > loss_mel_ce: 3.621563196182251 (nan) | > loss: 0.014668425545096397 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1350004069513981) | > loader_time: 0.0046 (0.0045077306068649955)  --> TIME: 2025-02-23 23:01:30 -- STEP: 23912/36938 -- GLOBAL_STEP: 60850 | > loss_text_ce: 0.07605036348104477 (0.09698962252764728) | > loss_mel_ce: 3.9524834156036377 (nan) | > loss: 0.0159862469881773 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.13501025988091162) | > loader_time: 0.0047 (0.004507635819469259)  --> TIME: 2025-02-23 23:01:39 -- STEP: 23962/36938 -- GLOBAL_STEP: 60900 | > loss_text_ce: 0.07821257412433624 (0.09695107788413462) | > loss_mel_ce: 4.343857765197754 (nan) | > loss: 0.01754789985716343 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.1350133596243885) | > loader_time: 0.0044 (0.0045076261407673715)  --> TIME: 2025-02-23 23:01:48 -- STEP: 24012/36938 -- GLOBAL_STEP: 60950 | > loss_text_ce: 0.08405213057994843 (0.09691453867427667) | > loss_mel_ce: 4.295674800872803 (nan) | > loss: 0.017379868775606155 (nan) | > current_lr: 5e-06 | > step_time: 0.1911 (0.13501501712881567) | > loader_time: 0.0047 (0.0045074835908665125)  --> TIME: 2025-02-23 23:01:57 -- STEP: 24062/36938 -- GLOBAL_STEP: 61000 | > loss_text_ce: 0.08291096240282059 (0.09687822204268724) | > loss_mel_ce: 3.6696505546569824 (nan) | > loss: 0.014891117811203003 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.1350100203430776) | > loader_time: 0.0044 (0.004507324838467877)  --> TIME: 2025-02-23 23:02:07 -- STEP: 24112/36938 -- GLOBAL_STEP: 61050 | > loss_text_ce: 0.0730479285120964 (0.09684279661467461) | > loss_mel_ce: 2.4514756202697754 (nan) | > loss: 0.010017951019108295 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.1350119290817777) | > loader_time: 0.0044 (0.004507184167056974)  --> TIME: 2025-02-23 23:02:16 -- STEP: 24162/36938 -- GLOBAL_STEP: 61100 | > loss_text_ce: 0.0970054641366005 (0.09680483400233998) | > loss_mel_ce: 3.8317995071411133 (nan) | > loss: 0.015590496361255646 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13502052439905898) | > loader_time: 0.0044 (0.0045071967774378404)  --> TIME: 2025-02-23 23:02:25 -- STEP: 24212/36938 -- GLOBAL_STEP: 61150 | > loss_text_ce: 0.08494826406240463 (0.0967711507697103) | > loss_mel_ce: 1.7272827625274658 (nan) | > loss: 0.0071913935244083405 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13502438840798237) | > loader_time: 0.0044 (0.0045070681673582665)  --> TIME: 2025-02-23 23:02:34 -- STEP: 24262/36938 -- GLOBAL_STEP: 61200 | > loss_text_ce: 0.07042315602302551 (0.096734871547361) | > loss_mel_ce: 4.247708320617676 (nan) | > loss: 0.017135443165898323 (nan) | > current_lr: 5e-06 | > step_time: 0.1886 (0.1350226053725001) | > loader_time: 0.0046 (0.004507081790273446)  --> TIME: 2025-02-23 23:02:44 -- STEP: 24312/36938 -- GLOBAL_STEP: 61250 | > loss_text_ce: 0.07769550383090973 (0.09669806941664648) | > loss_mel_ce: 4.104580879211426 (nan) | > loss: 0.016596335917711258 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13502543657655175) | > loader_time: 0.0047 (0.0045068883001510044)  --> TIME: 2025-02-23 23:02:53 -- STEP: 24362/36938 -- GLOBAL_STEP: 61300 | > loss_text_ce: 0.07111644744873047 (0.09665808194141041) | > loss_mel_ce: 3.6380605697631836 (nan) | > loss: 0.014718957245349884 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.1350405213085354) | > loader_time: 0.0062 (0.004506811750374554)  --> TIME: 2025-02-23 23:03:03 -- STEP: 24412/36938 -- GLOBAL_STEP: 61350 | > loss_text_ce: 0.08836928009986877 (0.09662329376643645) | > loss_mel_ce: 3.5343143939971924 (nan) | > loss: 0.01437573041766882 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1350427262610763) | > loader_time: 0.0043 (0.004506684337974111)  --> TIME: 2025-02-23 23:03:12 -- STEP: 24462/36938 -- GLOBAL_STEP: 61400 | > loss_text_ce: 0.0751805454492569 (0.09659003329064123) | > loss_mel_ce: 3.9380054473876953 (nan) | > loss: 0.015925342217087746 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13504651367454706) | > loader_time: 0.0044 (0.004506534649397703)  --> TIME: 2025-02-23 23:03:21 -- STEP: 24512/36938 -- GLOBAL_STEP: 61450 | > loss_text_ce: 0.09368982166051865 (0.09655434788568305) | > loss_mel_ce: 4.229912757873535 (nan) | > loss: 0.01715715415775776 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.1350383222803428) | > loader_time: 0.0044 (0.00450637869478516)  --> TIME: 2025-02-23 23:03:30 -- STEP: 24562/36938 -- GLOBAL_STEP: 61500 | > loss_text_ce: 0.08367767930030823 (0.09652114247433251) | > loss_mel_ce: 4.343313694000244 (nan) | > loss: 0.017567427828907967 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13503220506293265) | > loader_time: 0.0043 (0.004506192536591287)  --> TIME: 2025-02-23 23:03:39 -- STEP: 24612/36938 -- GLOBAL_STEP: 61550 | > loss_text_ce: 0.07618709653615952 (0.096488201093609) | > loss_mel_ce: 4.33251953125 (nan) | > loss: 0.017494868487119675 (nan) | > current_lr: 5e-06 | > step_time: 0.1876 (0.13504215737348504) | > loader_time: 0.0046 (0.004506028785407967)  --> TIME: 2025-02-23 23:03:48 -- STEP: 24662/36938 -- GLOBAL_STEP: 61600 | > loss_text_ce: 0.09650144726037979 (0.09645348947470192) | > loss_mel_ce: 3.2276809215545654 (nan) | > loss: 0.013191199861466885 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.1350381793249605) | > loader_time: 0.0043 (0.004505962614363075)  --> TIME: 2025-02-23 23:03:57 -- STEP: 24712/36938 -- GLOBAL_STEP: 61650 | > loss_text_ce: 0.08332648873329163 (0.09641894184046355) | > loss_mel_ce: 4.205792427062988 (nan) | > loss: 0.017020313069224358 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1350348239849586) | > loader_time: 0.0043 (0.0045058659439759854)  --> TIME: 2025-02-23 23:04:07 -- STEP: 24762/36938 -- GLOBAL_STEP: 61700 | > loss_text_ce: 0.07837768644094467 (0.09638172235993152) | > loss_mel_ce: 4.1470046043396 (nan) | > loss: 0.016767391934990883 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13504143839996954) | > loader_time: 0.0045 (0.004505824382215978)  --> TIME: 2025-02-23 23:04:16 -- STEP: 24812/36938 -- GLOBAL_STEP: 61750 | > loss_text_ce: 0.07108306139707565 (0.09634465990113969) | > loss_mel_ce: 4.680436134338379 (nan) | > loss: 0.01885523647069931 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.13505317583288515) | > loader_time: 0.0046 (0.004505681997342543)  --> TIME: 2025-02-23 23:04:25 -- STEP: 24862/36938 -- GLOBAL_STEP: 61800 | > loss_text_ce: 0.08437757939100266 (0.09630850697594535) | > loss_mel_ce: 4.140282154083252 (nan) | > loss: 0.016764525324106216 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.1350486001104538) | > loader_time: 0.0042 (0.004505559537140493)  --> TIME: 2025-02-23 23:04:35 -- STEP: 24912/36938 -- GLOBAL_STEP: 61850 | > loss_text_ce: 0.07249081134796143 (0.09627590303452264) | > loss_mel_ce: 4.265541076660156 (nan) | > loss: 0.01721441186964512 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.1350519905104871) | > loader_time: 0.0046 (0.004505342945657472)  --> TIME: 2025-02-23 23:04:44 -- STEP: 24962/36938 -- GLOBAL_STEP: 61900 | > loss_text_ce: 0.06879225373268127 (0.09624090918914624) | > loss_mel_ce: 3.8319270610809326 (nan) | > loss: 0.015479045920073986 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.1350486544345524) | > loader_time: 0.0044 (0.004505291551306167)  --> TIME: 2025-02-23 23:04:53 -- STEP: 25012/36938 -- GLOBAL_STEP: 61950 | > loss_text_ce: 0.0760573223233223 (0.09619851391816157) | > loss_mel_ce: 4.2809834480285645 (nan) | > loss: 0.017289845272898674 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.135059431059425) | > loader_time: 0.0045 (0.004505278214671666)  --> TIME: 2025-02-23 23:05:03 -- STEP: 25062/36938 -- GLOBAL_STEP: 62000 | > loss_text_ce: 0.07705801725387573 (0.09616883895671756) | > loss_mel_ce: 4.43992280960083 (nan) | > loss: 0.01792452670633793 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.135062416213801) | > loader_time: 0.0045 (0.004505226574228714)  --> TIME: 2025-02-23 23:05:12 -- STEP: 25112/36938 -- GLOBAL_STEP: 62050 | > loss_text_ce: 0.08089783787727356 (0.09613714322187322) | > loss_mel_ce: 3.7515869140625 (nan) | > loss: 0.015208273194730282 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13507298554117803) | > loader_time: 0.0045 (0.0045053479150435945)  --> TIME: 2025-02-23 23:05:22 -- STEP: 25162/36938 -- GLOBAL_STEP: 62100 | > loss_text_ce: 0.07609858363866806 (0.09610051802675582) | > loss_mel_ce: 3.5718953609466553 (nan) | > loss: 0.014476167969405651 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.1350803656946079) | > loader_time: 0.0044 (0.0045052741026993855)  --> TIME: 2025-02-23 23:05:31 -- STEP: 25212/36938 -- GLOBAL_STEP: 62150 | > loss_text_ce: 0.07221858203411102 (0.09606371679336381) | > loss_mel_ce: 4.1053876876831055 (nan) | > loss: 0.01657780259847641 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13508013593910276) | > loader_time: 0.0045 (0.004505164184854753)  --> TIME: 2025-02-23 23:05:40 -- STEP: 25262/36938 -- GLOBAL_STEP: 62200 | > loss_text_ce: 0.07853872328996658 (0.09602761972442285) | > loss_mel_ce: 3.2104837894439697 (nan) | > loss: 0.013051677495241165 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13508313487029205) | > loader_time: 0.0044 (0.004505147608165894)  --> TIME: 2025-02-23 23:05:49 -- STEP: 25312/36938 -- GLOBAL_STEP: 62250 | > loss_text_ce: 0.08195693045854568 (0.09599227219079481) | > loss_mel_ce: 3.5282866954803467 (nan) | > loss: 0.014326363801956177 (nan) | > current_lr: 5e-06 | > step_time: 0.184 (0.13508030783512112) | > loader_time: 0.0048 (0.004505092817371613)  --> TIME: 2025-02-23 23:05:58 -- STEP: 25362/36938 -- GLOBAL_STEP: 62300 | > loss_text_ce: 0.07669145613908768 (0.09595923882729009) | > loss_mel_ce: 4.421391487121582 (nan) | > loss: 0.01784953661262989 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13507252504336476) | > loader_time: 0.0043 (0.004504939771095244)  --> TIME: 2025-02-23 23:06:08 -- STEP: 25412/36938 -- GLOBAL_STEP: 62350 | > loss_text_ce: 0.07835987210273743 (0.09592533749234347) | > loss_mel_ce: 3.2747902870178223 (nan) | > loss: 0.013306152075529099 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.1350792782073609) | > loader_time: 0.0042 (0.004504899781237762)  --> TIME: 2025-02-23 23:06:17 -- STEP: 25462/36938 -- GLOBAL_STEP: 62400 | > loss_text_ce: 0.07602984458208084 (0.09588759178934408) | > loss_mel_ce: 4.010387897491455 (nan) | > loss: 0.016215944662690163 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13508016430529052) | > loader_time: 0.0044 (0.004504705756357248)  --> TIME: 2025-02-23 23:06:26 -- STEP: 25512/36938 -- GLOBAL_STEP: 62450 | > loss_text_ce: 0.07658052444458008 (0.09585034155163358) | > loss_mel_ce: 4.068751335144043 (nan) | > loss: 0.016449730843305588 (nan) | > current_lr: 5e-06 | > step_time: 0.1885 (0.13508489665359966) | > loader_time: 0.0046 (0.004504572077954807)  --> TIME: 2025-02-23 23:06:35 -- STEP: 25562/36938 -- GLOBAL_STEP: 62500 | > loss_text_ce: 0.08752497285604477 (0.09581157474688061) | > loss_mel_ce: 3.4669852256774902 (nan) | > loss: 0.014105199836194515 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13508486907234496) | > loader_time: 0.0044 (0.00450438554368475)  --> TIME: 2025-02-23 23:06:45 -- STEP: 25612/36938 -- GLOBAL_STEP: 62550 | > loss_text_ce: 0.07525002956390381 (0.09577542285453998) | > loss_mel_ce: 4.672966957092285 (nan) | > loss: 0.01884213276207447 (nan) | > current_lr: 5e-06 | > step_time: 0.1996 (0.1350936702726473) | > loader_time: 0.0047 (0.004504417797820216)  --> TIME: 2025-02-23 23:06:54 -- STEP: 25662/36938 -- GLOBAL_STEP: 62600 | > loss_text_ce: 0.08845961093902588 (0.09573704491324975) | > loss_mel_ce: 3.8216183185577393 (nan) | > loss: 0.015516183339059353 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.13509679233634753) | > loader_time: 0.0048 (0.004504283798817724)  --> TIME: 2025-02-23 23:07:03 -- STEP: 25712/36938 -- GLOBAL_STEP: 62650 | > loss_text_ce: 0.07963207364082336 (0.09569630806633403) | > loss_mel_ce: 4.230550765991211 (nan) | > loss: 0.017103901132941246 (nan) | > current_lr: 5e-06 | > step_time: 0.1751 (0.13509293980826814) | > loader_time: 0.0046 (0.0045043193986123555)  --> TIME: 2025-02-23 23:07:12 -- STEP: 25762/36938 -- GLOBAL_STEP: 62700 | > loss_text_ce: 0.07384050637483597 (0.09565698282266703) | > loss_mel_ce: 3.90562105178833 (nan) | > loss: 0.0157915148884058 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.1350828075471856) | > loader_time: 0.0044 (0.004504187878369632)  --> TIME: 2025-02-23 23:07:21 -- STEP: 25812/36938 -- GLOBAL_STEP: 62750 | > loss_text_ce: 0.07686297595500946 (0.09562224372727851) | > loss_mel_ce: 3.962815523147583 (nan) | > loss: 0.01603047177195549 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13508566054340082) | > loader_time: 0.0044 (0.004504061550682463)  --> TIME: 2025-02-23 23:07:31 -- STEP: 25862/36938 -- GLOBAL_STEP: 62800 | > loss_text_ce: 0.07118207961320877 (0.09558489287302244) | > loss_mel_ce: 3.82550048828125 (nan) | > loss: 0.015463026240468025 (nan) | > current_lr: 5e-06 | > step_time: 0.1457 (0.13510554849608167) | > loader_time: 0.0044 (0.004504097253833216)  --> TIME: 2025-02-23 23:07:40 -- STEP: 25912/36938 -- GLOBAL_STEP: 62850 | > loss_text_ce: 0.06960126012563705 (0.09554876423521137) | > loss_mel_ce: 3.7681891918182373 (nan) | > loss: 0.015229328535497189 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.1350925686956363) | > loader_time: 0.0044 (0.004503888741949893)  --> TIME: 2025-02-23 23:07:49 -- STEP: 25962/36938 -- GLOBAL_STEP: 62900 | > loss_text_ce: 0.06743557006120682 (0.09551211624896268) | > loss_mel_ce: 4.1303510665893555 (nan) | > loss: 0.016657885164022446 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13509549196287435) | > loader_time: 0.0043 (0.004503807414708508)  --> TIME: 2025-02-23 23:07:58 -- STEP: 26012/36938 -- GLOBAL_STEP: 62950 | > loss_text_ce: 0.08419525623321533 (0.09547715310471963) | > loss_mel_ce: 4.0293121337890625 (nan) | > loss: 0.01632344163954258 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.13509378745961653) | > loader_time: 0.0044 (0.0045037738693579875)  --> TIME: 2025-02-23 23:08:07 -- STEP: 26062/36938 -- GLOBAL_STEP: 63000 | > loss_text_ce: 0.0790126845240593 (0.09543959214850614) | > loss_mel_ce: 0.02392578125 (nan) | > loss: 0.0004084859974682331 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13509136053690785) | > loader_time: 0.0044 (0.004503749756369915)  --> TIME: 2025-02-23 23:08:17 -- STEP: 26112/36938 -- GLOBAL_STEP: 63050 | > loss_text_ce: 0.07609929144382477 (0.09540236391074079) | > loss_mel_ce: 3.5842702388763428 (nan) | > loss: 0.0145252775400877 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13509923813170585) | > loader_time: 0.0045 (0.004503728420126671)  --> TIME: 2025-02-23 23:08:26 -- STEP: 26162/36938 -- GLOBAL_STEP: 63100 | > loss_text_ce: 0.08478567749261856 (0.09536808150001787) | > loss_mel_ce: 4.399313449859619 (nan) | > loss: 0.017794044688344002 (nan) | > current_lr: 5e-06 | > step_time: 0.1123 (0.13509290097694912) | > loader_time: 0.0043 (0.004503669127094008)  --> TIME: 2025-02-23 23:08:35 -- STEP: 26212/36938 -- GLOBAL_STEP: 63150 | > loss_text_ce: 0.10299333184957504 (0.09533463463436825) | > loss_mel_ce: 2.937526226043701 (nan) | > loss: 0.012065554037690163 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1350997349478419) | > loader_time: 0.0044 (0.004503736764479097)  --> TIME: 2025-02-23 23:08:44 -- STEP: 26262/36938 -- GLOBAL_STEP: 63200 | > loss_text_ce: 0.06963012367486954 (0.09529589190293326) | > loss_mel_ce: 3.990814208984375 (nan) | > loss: 0.016112875193357468 (nan) | > current_lr: 5e-06 | > step_time: 0.1591 (0.13510602418942894) | > loader_time: 0.0047 (0.004504142208097913)  --> TIME: 2025-02-23 23:08:54 -- STEP: 26312/36938 -- GLOBAL_STEP: 63250 | > loss_text_ce: 0.06548132002353668 (0.09526002018489653) | > loss_mel_ce: 0.615612804889679 (nan) | > loss: 0.002702754456549883 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13510405726438512) | > loader_time: 0.0044 (0.0045041261599324135)  --> TIME: 2025-02-23 23:09:03 -- STEP: 26362/36938 -- GLOBAL_STEP: 63300 | > loss_text_ce: 0.06907990574836731 (0.09522203324189851) | > loss_mel_ce: 3.892681121826172 (nan) | > loss: 0.0157212745398283 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13511207669704223) | > loader_time: 0.0043 (0.0045041184207935715)  --> TIME: 2025-02-23 23:09:12 -- STEP: 26412/36938 -- GLOBAL_STEP: 63350 | > loss_text_ce: 0.07514459639787674 (0.09518624572008641) | > loss_mel_ce: 4.529469966888428 (nan) | > loss: 0.01827228255569935 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13510879746498644) | > loader_time: 0.0045 (0.004504277600350937)  --> TIME: 2025-02-23 23:09:22 -- STEP: 26462/36938 -- GLOBAL_STEP: 63400 | > loss_text_ce: 0.0696883574128151 (0.09515141359572031) | > loss_mel_ce: 4.008584976196289 (nan) | > loss: 0.016183625906705856 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.1351189099240388) | > loader_time: 0.0045 (0.004504403112228566)  --> TIME: 2025-02-23 23:09:31 -- STEP: 26512/36938 -- GLOBAL_STEP: 63450 | > loss_text_ce: 0.07335604727268219 (0.09511542618575429) | > loss_mel_ce: 3.7512075901031494 (nan) | > loss: 0.01517684105783701 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.1351223788350611) | > loader_time: 0.0044 (0.004504368788302484)  --> TIME: 2025-02-23 23:09:40 -- STEP: 26562/36938 -- GLOBAL_STEP: 63500 | > loss_text_ce: 0.06791533529758453 (0.0950827176219343) | > loss_mel_ce: 4.259264945983887 (nan) | > loss: 0.017171351239085197 (nan) | > current_lr: 5e-06 | > step_time: 0.1528 (0.13512778594527666) | > loader_time: 0.0047 (0.004504397460992648)  --> TIME: 2025-02-23 23:09:50 -- STEP: 26612/36938 -- GLOBAL_STEP: 63550 | > loss_text_ce: 0.07246825844049454 (0.09504756545467438) | > loss_mel_ce: 3.243211507797241 (nan) | > loss: 0.013157459907233715 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13513932036543486) | > loader_time: 0.0043 (0.004504395878693045)  --> TIME: 2025-02-23 23:09:59 -- STEP: 26662/36938 -- GLOBAL_STEP: 63600 | > loss_text_ce: 0.07686610519886017 (0.09501381539712114) | > loss_mel_ce: 3.03812837600708 (nan) | > loss: 0.012361089698970318 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13514619194265556) | > loader_time: 0.0045 (0.004504425027939006)  --> TIME: 2025-02-23 23:10:09 -- STEP: 26712/36938 -- GLOBAL_STEP: 63650 | > loss_text_ce: 0.0691755935549736 (0.09497910861512249) | > loss_mel_ce: 3.3994994163513184 (nan) | > loss: 0.013764583505690098 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.13516060237257443) | > loader_time: 0.0044 (0.004504407699967121)  --> TIME: 2025-02-23 23:10:18 -- STEP: 26762/36938 -- GLOBAL_STEP: 63700 | > loss_text_ce: 0.07556426525115967 (0.09494341135247229) | > loss_mel_ce: 0.1173502579331398 (nan) | > loss: 0.0007655338267795742 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13516471811976746) | > loader_time: 0.0044 (0.004504396753116594)  --> TIME: 2025-02-23 23:10:28 -- STEP: 26812/36938 -- GLOBAL_STEP: 63750 | > loss_text_ce: 0.07620040327310562 (0.0949081934822607) | > loss_mel_ce: 3.5782501697540283 (nan) | > loss: 0.014501789584755898 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.1351687321729755) | > loader_time: 0.0045 (0.004504413679786792)  --> TIME: 2025-02-23 23:10:37 -- STEP: 26862/36938 -- GLOBAL_STEP: 63800 | > loss_text_ce: 0.08130531013011932 (0.09487338456819747) | > loss_mel_ce: 4.07024621963501 (nan) | > loss: 0.016474412754178047 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13517754315424468) | > loader_time: 0.0043 (0.004504334153533301)  --> TIME: 2025-02-23 23:10:47 -- STEP: 26912/36938 -- GLOBAL_STEP: 63850 | > loss_text_ce: 0.07432708144187927 (0.09483477274397834) | > loss_mel_ce: 4.787294387817383 (nan) | > loss: 0.01929214969277382 (nan) | > current_lr: 5e-06 | > step_time: 0.1862 (0.13518298251662716) | > loader_time: 0.0047 (0.004504233146890314)  --> TIME: 2025-02-23 23:10:56 -- STEP: 26962/36938 -- GLOBAL_STEP: 63900 | > loss_text_ce: 0.0797419622540474 (0.09479851295661608) | > loss_mel_ce: 4.271958351135254 (nan) | > loss: 0.017268652096390724 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13518094120707405) | > loader_time: 0.0043 (0.004504173306539504)  --> TIME: 2025-02-23 23:11:05 -- STEP: 27012/36938 -- GLOBAL_STEP: 63950 | > loss_text_ce: 0.07209248840808868 (0.09476661758946164) | > loss_mel_ce: 3.7948052883148193 (nan) | > loss: 0.015344833955168724 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13517870006076702) | > loader_time: 0.0046 (0.004504089256260076)  --> TIME: 2025-02-23 23:11:14 -- STEP: 27062/36938 -- GLOBAL_STEP: 64000 | > loss_text_ce: 0.0717439278960228 (0.09473279663854808) | > loss_mel_ce: 2.595013380050659 (nan) | > loss: 0.010582370683550835 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.1351880533328174) | > loader_time: 0.0043 (0.004504081089492562)  --> TIME: 2025-02-23 23:11:24 -- STEP: 27112/36938 -- GLOBAL_STEP: 64050 | > loss_text_ce: 0.08192291855812073 (0.09469678504906436) | > loss_mel_ce: 2.962986469268799 (nan) | > loss: 0.012082974426448345 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13519526520778408) | > loader_time: 0.0043 (0.004503927625854236)  --> TIME: 2025-02-23 23:11:33 -- STEP: 27162/36938 -- GLOBAL_STEP: 64100 | > loss_text_ce: 0.07440263777971268 (0.09466250288627293) | > loss_mel_ce: 4.240793704986572 (nan) | > loss: 0.017123796045780182 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13519589110630495) | > loader_time: 0.0043 (0.004503769495729299)  --> TIME: 2025-02-23 23:11:43 -- STEP: 27212/36938 -- GLOBAL_STEP: 64150 | > loss_text_ce: 0.06322969496250153 (0.09462791895413135) | > loss_mel_ce: 2.9123218059539795 (nan) | > loss: 0.011807745322585106 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13520463325513474) | > loader_time: 0.0044 (0.004503732146050458)  --> TIME: 2025-02-23 23:11:52 -- STEP: 27262/36938 -- GLOBAL_STEP: 64200 | > loss_text_ce: 0.07904965430498123 (0.09459454913598869) | > loss_mel_ce: 3.765087604522705 (nan) | > loss: 0.01525451336055994 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13521654785496928) | > loader_time: 0.0044 (0.004503675544701825)  --> TIME: 2025-02-23 23:12:02 -- STEP: 27312/36938 -- GLOBAL_STEP: 64250 | > loss_text_ce: 0.07635106146335602 (0.09455959387813796) | > loss_mel_ce: 3.834547281265259 (nan) | > loss: 0.015519439242780209 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.1352267175406531) | > loader_time: 0.0043 (0.004503716553731067)  --> TIME: 2025-02-23 23:12:11 -- STEP: 27362/36938 -- GLOBAL_STEP: 64300 | > loss_text_ce: 0.07925447076559067 (0.09452675467899085) | > loss_mel_ce: 3.614525079727173 (nan) | > loss: 0.014657855965197086 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13522957932067947) | > loader_time: 0.0044 (0.004503583491576118)  --> TIME: 2025-02-23 23:12:20 -- STEP: 27412/36938 -- GLOBAL_STEP: 64350 | > loss_text_ce: 0.07543561607599258 (0.09448938973383401) | > loss_mel_ce: 3.9758875370025635 (nan) | > loss: 0.016076678410172462 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.1352275086026721) | > loader_time: 0.0045 (0.004503660344319082)  --> TIME: 2025-02-23 23:12:29 -- STEP: 27462/36938 -- GLOBAL_STEP: 64400 | > loss_text_ce: 0.06325482577085495 (0.0944515015066106) | > loss_mel_ce: 4.422889232635498 (nan) | > loss: 0.0178021602332592 (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.1352280602070693) | > loader_time: 0.0046 (0.004503536984887682)  --> TIME: 2025-02-23 23:12:38 -- STEP: 27512/36938 -- GLOBAL_STEP: 64450 | > loss_text_ce: 0.05963071808218956 (0.09441557682481425) | > loss_mel_ce: 4.685598373413086 (nan) | > loss: 0.018830275163054466 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.13521930688895006) | > loader_time: 0.0046 (0.004503323271649621)  --> TIME: 2025-02-23 23:12:48 -- STEP: 27562/36938 -- GLOBAL_STEP: 64500 | > loss_text_ce: 0.06992608308792114 (0.09438392191389547) | > loss_mel_ce: 1.7167266607284546 (nan) | > loss: 0.007089892402291298 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.1352206447388988) | > loader_time: 0.0043 (0.004503227951443416)  --> TIME: 2025-02-23 23:12:57 -- STEP: 27612/36938 -- GLOBAL_STEP: 64550 | > loss_text_ce: 0.07247396558523178 (0.09435334246023919) | > loss_mel_ce: 3.5714073181152344 (nan) | > loss: 0.01445984747260809 (nan) | > current_lr: 5e-06 | > step_time: 0.1825 (0.1352174435338109) | > loader_time: 0.0049 (0.004503100795295457)  --> TIME: 2025-02-23 23:13:06 -- STEP: 27662/36938 -- GLOBAL_STEP: 64600 | > loss_text_ce: 0.08409909158945084 (0.09431519726452002) | > loss_mel_ce: 3.8043148517608643 (nan) | > loss: 0.015430214814841747 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13521506196634767) | > loader_time: 0.0044 (0.004502998869810911)  --> TIME: 2025-02-23 23:13:15 -- STEP: 27712/36938 -- GLOBAL_STEP: 64650 | > loss_text_ce: 0.07181240618228912 (0.09428290171009973) | > loss_mel_ce: 3.763709545135498 (nan) | > loss: 0.015220326371490955 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.1352109112292855) | > loader_time: 0.0043 (0.004502962311131839)  --> TIME: 2025-02-23 23:13:24 -- STEP: 27762/36938 -- GLOBAL_STEP: 64700 | > loss_text_ce: 0.07957379519939423 (0.09424957403009164) | > loss_mel_ce: 2.7756619453430176 (nan) | > loss: 0.011330301873385906 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13521517996880267) | > loader_time: 0.0043 (0.004502817925059572)  --> TIME: 2025-02-23 23:13:34 -- STEP: 27812/36938 -- GLOBAL_STEP: 64750 | > loss_text_ce: 0.07939061522483826 (0.09421562019030526) | > loss_mel_ce: 3.9970340728759766 (nan) | > loss: 0.016176288947463036 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13521739252861956) | > loader_time: 0.0044 (0.004502655524376317)  --> TIME: 2025-02-23 23:13:43 -- STEP: 27862/36938 -- GLOBAL_STEP: 64800 | > loss_text_ce: 0.07892816513776779 (0.09418196710125191) | > loss_mel_ce: 3.738764524459839 (nan) | > loss: 0.015149575658142567 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.13522311893422173) | > loader_time: 0.0046 (0.004502510487087721)  --> TIME: 2025-02-23 23:13:53 -- STEP: 27912/36938 -- GLOBAL_STEP: 64850 | > loss_text_ce: 0.07831553369760513 (0.09414340092529128) | > loss_mel_ce: 4.161579132080078 (nan) | > loss: 0.016824981197714806 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.1352362976558693) | > loader_time: 0.0046 (0.004502494326974429)  --> TIME: 2025-02-23 23:14:02 -- STEP: 27962/36938 -- GLOBAL_STEP: 64900 | > loss_text_ce: 0.06457167118787766 (0.09410603831105006) | > loss_mel_ce: 4.054839134216309 (nan) | > loss: 0.016346869990229607 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13523231122415302) | > loader_time: 0.0044 (0.0045024036687549805)  --> TIME: 2025-02-23 23:14:11 -- STEP: 28012/36938 -- GLOBAL_STEP: 64950 | > loss_text_ce: 0.07912883907556534 (0.09406668240917047) | > loss_mel_ce: 0.7364941239356995 (nan) | > loss: 0.0032365992665290833 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13523584014429937) | > loader_time: 0.0044 (0.004502339514939759)  --> TIME: 2025-02-23 23:14:21 -- STEP: 28062/36938 -- GLOBAL_STEP: 65000 | > loss_text_ce: 0.0809861570596695 (0.0940323374987844) | > loss_mel_ce: 4.026147365570068 (nan) | > loss: 0.016298148781061172 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13523820614612392) | > loader_time: 0.0045 (0.004502267161571799)  --> TIME: 2025-02-23 23:14:30 -- STEP: 28112/36938 -- GLOBAL_STEP: 65050 | > loss_text_ce: 0.07633409649133682 (0.09400118282581647) | > loss_mel_ce: 1.9949411153793335 (nan) | > loss: 0.008219346404075623 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13523586323366954) | > loader_time: 0.0044 (0.004502197567481642)  --> TIME: 2025-02-23 23:14:39 -- STEP: 28162/36938 -- GLOBAL_STEP: 65100 | > loss_text_ce: 0.08089111000299454 (0.09396668630922352) | > loss_mel_ce: 1.9567333459854126 (nan) | > loss: 0.008085811510682106 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13523396219975778) | > loader_time: 0.0043 (0.004502140445369232)  --> TIME: 2025-02-23 23:14:48 -- STEP: 28212/36938 -- GLOBAL_STEP: 65150 | > loss_text_ce: 0.068169504404068 (0.09393189064368769) | > loss_mel_ce: 4.271905422210693 (nan) | > loss: 0.01722252182662487 (nan) | > current_lr: 5e-06 | > step_time: 0.1916 (0.13522718945040235) | > loader_time: 0.0046 (0.004502052713518829)  --> TIME: 2025-02-23 23:14:57 -- STEP: 28262/36938 -- GLOBAL_STEP: 65200 | > loss_text_ce: 0.08534453809261322 (0.09390014700468226) | > loss_mel_ce: 4.759888648986816 (nan) | > loss: 0.01922711543738842 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.1352235426544056) | > loader_time: 0.0045 (0.004502037605589065)  --> TIME: 2025-02-23 23:15:06 -- STEP: 28312/36938 -- GLOBAL_STEP: 65250 | > loss_text_ce: 0.08227941393852234 (0.09386532600797008) | > loss_mel_ce: 3.754152536392212 (nan) | > loss: 0.01522393710911274 (nan) | > current_lr: 5e-06 | > step_time: 0.1791 (0.13522323912973125) | > loader_time: 0.0046 (0.00450187968682286)  --> TIME: 2025-02-23 23:15:16 -- STEP: 28362/36938 -- GLOBAL_STEP: 65300 | > loss_text_ce: 0.08271688967943192 (0.09383457054749698) | > loss_mel_ce: 1.948106288909912 (nan) | > loss: 0.008058822713792324 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13521941023731554) | > loader_time: 0.0045 (0.004501781462950547)  --> TIME: 2025-02-23 23:15:25 -- STEP: 28412/36938 -- GLOBAL_STEP: 65350 | > loss_text_ce: 0.0760265439748764 (0.09380537064440034) | > loss_mel_ce: 4.309545993804932 (nan) | > loss: 0.017403066158294678 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.1352139128172498) | > loader_time: 0.0044 (0.004501611594329432)  --> TIME: 2025-02-23 23:15:34 -- STEP: 28462/36938 -- GLOBAL_STEP: 65400 | > loss_text_ce: 0.08136971294879913 (0.09377171669170319) | > loss_mel_ce: 3.6324195861816406 (nan) | > loss: 0.014737259596586227 (nan) | > current_lr: 5e-06 | > step_time: 0.1837 (0.13522081647868353) | > loader_time: 0.0046 (0.004501547810734976)  --> TIME: 2025-02-23 23:15:43 -- STEP: 28512/36938 -- GLOBAL_STEP: 65450 | > loss_text_ce: 0.08835683763027191 (0.09373950217348823) | > loss_mel_ce: 1.9638140201568604 (nan) | > loss: 0.008143534883856773 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.1352143250113102) | > loader_time: 0.0044 (0.0045013558265871246)  --> TIME: 2025-02-23 23:15:53 -- STEP: 28562/36938 -- GLOBAL_STEP: 65500 | > loss_text_ce: 0.07047253102064133 (0.09370943738253171) | > loss_mel_ce: 1.8514107465744019 (nan) | > loss: 0.007626521401107311 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1352223285013652) | > loader_time: 0.0044 (0.004501384927832311)  --> TIME: 2025-02-23 23:16:02 -- STEP: 28612/36938 -- GLOBAL_STEP: 65550 | > loss_text_ce: 0.07229609042406082 (0.09367657906257251) | > loss_mel_ce: 4.651171684265137 (nan) | > loss: 0.01874392107129097 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.1352266861844778) | > loader_time: 0.0045 (0.004501352631160804)  --> TIME: 2025-02-23 23:16:11 -- STEP: 28662/36938 -- GLOBAL_STEP: 65600 | > loss_text_ce: 0.06675854325294495 (0.09364313286105388) | > loss_mel_ce: 4.2868146896362305 (nan) | > loss: 0.017276085913181305 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.1352237883652908) | > loader_time: 0.0044 (0.004501269822110356)  --> TIME: 2025-02-23 23:16:21 -- STEP: 28712/36938 -- GLOBAL_STEP: 65650 | > loss_text_ce: 0.09211025387048721 (0.09361130445334122) | > loss_mel_ce: 3.9521148204803467 (nan) | > loss: 0.01604851335287094 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.1352270090520185) | > loader_time: 0.0045 (0.004501227533362248)  --> TIME: 2025-02-23 23:16:30 -- STEP: 28762/36938 -- GLOBAL_STEP: 65700 | > loss_text_ce: 0.06878262013196945 (0.09357474634757926) | > loss_mel_ce: 3.943356513977051 (nan) | > loss: 0.01592118851840496 (nan) | > current_lr: 5e-06 | > step_time: 0.1515 (0.13523102633918643) | > loader_time: 0.0044 (0.004501092525941346)  --> TIME: 2025-02-23 23:16:39 -- STEP: 28812/36938 -- GLOBAL_STEP: 65750 | > loss_text_ce: 0.07298096269369125 (0.0935405355300352) | > loss_mel_ce: 4.733078956604004 (nan) | > loss: 0.0190716665238142 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13523096940127707) | > loader_time: 0.0046 (0.004500922901207058)  --> TIME: 2025-02-23 23:16:48 -- STEP: 28862/36938 -- GLOBAL_STEP: 65800 | > loss_text_ce: 0.06810116767883301 (0.09350794588799934) | > loss_mel_ce: 4.107967853546143 (nan) | > loss: 0.016571704298257828 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.1352309110993831) | > loader_time: 0.0045 (0.004500848630239414)  --> TIME: 2025-02-23 23:16:58 -- STEP: 28912/36938 -- GLOBAL_STEP: 65850 | > loss_text_ce: 0.07207078486680984 (0.09347536960654369) | > loss_mel_ce: 4.478173732757568 (nan) | > loss: 0.01805652678012848 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.1352264719175632) | > loader_time: 0.0044 (0.004500720495338529)  --> TIME: 2025-02-23 23:17:07 -- STEP: 28962/36938 -- GLOBAL_STEP: 65900 | > loss_text_ce: 0.07276585698127747 (0.09344351512328725) | > loss_mel_ce: 4.074423789978027 (nan) | > loss: 0.016457103192806244 (nan) | > current_lr: 5e-06 | > step_time: 0.1664 (0.13522567617144793) | > loader_time: 0.0046 (0.004500597421079621)  --> TIME: 2025-02-23 23:17:16 -- STEP: 29012/36938 -- GLOBAL_STEP: 65950 | > loss_text_ce: 0.06181967630982399 (0.09341206989540925) | > loss_mel_ce: 3.820570707321167 (nan) | > loss: 0.015406312420964241 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13522876347736673) | > loader_time: 0.0043 (0.0045006531740770074)  --> TIME: 2025-02-23 23:17:25 -- STEP: 29062/36938 -- GLOBAL_STEP: 66000 | > loss_text_ce: 0.06494993716478348 (0.09337787063488974) | > loss_mel_ce: 4.361805438995361 (nan) | > loss: 0.017566490918397903 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.13521971543257935) | > loader_time: 0.0048 (0.004500524551917173)  --> TIME: 2025-02-23 23:17:35 -- STEP: 29112/36938 -- GLOBAL_STEP: 66050 | > loss_text_ce: 0.0674276128411293 (0.09334555769113513) | > loss_mel_ce: 3.119981527328491 (nan) | > loss: 0.012648450210690498 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13522655966682087) | > loader_time: 0.0056 (0.004500565349678972)  --> TIME: 2025-02-23 23:17:44 -- STEP: 29162/36938 -- GLOBAL_STEP: 66100 | > loss_text_ce: 0.06907331943511963 (0.09330802524776503) | > loss_mel_ce: 0.063873291015625 (nan) | > loss: 0.0005275659495964646 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13523534073373988) | > loader_time: 0.0043 (0.004500523449731837)  --> TIME: 2025-02-23 23:17:53 -- STEP: 29212/36938 -- GLOBAL_STEP: 66150 | > loss_text_ce: 0.07038800418376923 (0.09327639046088296) | > loss_mel_ce: 3.3125176429748535 (nan) | > loss: 0.013424229808151722 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13522978915623549) | > loader_time: 0.0044 (0.004500455575888412)  --> TIME: 2025-02-23 23:18:02 -- STEP: 29262/36938 -- GLOBAL_STEP: 66200 | > loss_text_ce: 0.078605517745018 (0.09324503168575775) | > loss_mel_ce: 3.949632406234741 (nan) | > loss: 0.015985071659088135 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13521787233960694) | > loader_time: 0.0045 (0.00450025084861019)  --> TIME: 2025-02-23 23:18:11 -- STEP: 29312/36938 -- GLOBAL_STEP: 66250 | > loss_text_ce: 0.07627687603235245 (0.09321034209958852) | > loss_mel_ce: 0.17539897561073303 (nan) | > loss: 0.0009987137746065855 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.1352145385859304) | > loader_time: 0.0044 (0.004500146044263677)  --> TIME: 2025-02-23 23:18:21 -- STEP: 29362/36938 -- GLOBAL_STEP: 66300 | > loss_text_ce: 0.06659591943025589 (0.0931753000330396) | > loss_mel_ce: 0.2875198423862457 (nan) | > loss: 0.0014052212936803699 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13522610338919153) | > loader_time: 0.0043 (0.004500082294148931)  --> TIME: 2025-02-23 23:18:30 -- STEP: 29412/36938 -- GLOBAL_STEP: 66350 | > loss_text_ce: 0.06976441293954849 (0.09314392888395455) | > loss_mel_ce: 2.296875 (nan) | > loss: 0.009391427040100098 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13522995616665545) | > loader_time: 0.0043 (0.00449992215148644)  --> TIME: 2025-02-23 23:18:40 -- STEP: 29462/36938 -- GLOBAL_STEP: 66400 | > loss_text_ce: 0.06533420085906982 (0.0931142747467187) | > loss_mel_ce: 1.2744026184082031 (nan) | > loss: 0.0053164162673056126 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13523366193762865) | > loader_time: 0.0043 (0.004499918922016204)  --> TIME: 2025-02-23 23:18:49 -- STEP: 29512/36938 -- GLOBAL_STEP: 66450 | > loss_text_ce: 0.09482644498348236 (0.09308116557833988) | > loss_mel_ce: 1.9868011474609375 (nan) | > loss: 0.008260427042841911 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13523695764744478) | > loader_time: 0.0044 (0.004499780247934665)  --> TIME: 2025-02-23 23:18:59 -- STEP: 29562/36938 -- GLOBAL_STEP: 66500 | > loss_text_ce: 0.0790746733546257 (0.0930461312167405) | > loss_mel_ce: 3.7714390754699707 (nan) | > loss: 0.015279817394912243 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13523789517055643) | > loader_time: 0.0043 (0.004499649672472435)  --> TIME: 2025-02-23 23:19:08 -- STEP: 29612/36938 -- GLOBAL_STEP: 66550 | > loss_text_ce: 0.07012356072664261 (0.09301226966784566) | > loss_mel_ce: 1.6273193359375 (nan) | > loss: 0.006735885050147772 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13524318194688778) | > loader_time: 0.0043 (0.004499603425685129)  --> TIME: 2025-02-23 23:19:17 -- STEP: 29662/36938 -- GLOBAL_STEP: 66600 | > loss_text_ce: 0.05474431812763214 (0.09298001980322028) | > loss_mel_ce: 2.3687191009521484 (nan) | > loss: 0.009616918861865997 (nan) | > current_lr: 5e-06 | > step_time: 0.1081 (0.13523703608989607) | > loader_time: 0.0043 (0.004499472334592138)  --> TIME: 2025-02-23 23:19:26 -- STEP: 29712/36938 -- GLOBAL_STEP: 66650 | > loss_text_ce: 0.08017800003290176 (0.09294407643835706) | > loss_mel_ce: 4.006796836853027 (nan) | > loss: 0.01621815375983715 (nan) | > current_lr: 5e-06 | > step_time: 0.1796 (0.135242251343282) | > loader_time: 0.0048 (0.004499395046429286)  --> TIME: 2025-02-23 23:19:36 -- STEP: 29762/36938 -- GLOBAL_STEP: 66700 | > loss_text_ce: 0.0803307592868805 (0.09291496546410212) | > loss_mel_ce: 3.6929664611816406 (nan) | > loss: 0.014973402954638004 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13523876888922845) | > loader_time: 0.0044 (0.004499307003050724)  --> TIME: 2025-02-23 23:19:45 -- STEP: 29812/36938 -- GLOBAL_STEP: 66750 | > loss_text_ce: 0.07822363078594208 (0.0928829009709045) | > loss_mel_ce: 4.455498695373535 (nan) | > loss: 0.0179909635335207 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13523270189562098) | > loader_time: 0.0044 (0.0044992429832956034)  --> TIME: 2025-02-23 23:19:54 -- STEP: 29862/36938 -- GLOBAL_STEP: 66800 | > loss_text_ce: 0.07357712835073471 (0.09284889691653866) | > loss_mel_ce: 4.240959167480469 (nan) | > loss: 0.017121175304055214 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.135231785505648) | > loader_time: 0.0044 (0.004499166946418802)  --> TIME: 2025-02-23 23:20:03 -- STEP: 29912/36938 -- GLOBAL_STEP: 66850 | > loss_text_ce: 0.08483029901981354 (0.09281568611740672) | > loss_mel_ce: 4.074936389923096 (nan) | > loss: 0.016507010906934738 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.1352326082303319) | > loader_time: 0.0045 (0.004499092765847899)  --> TIME: 2025-02-23 23:20:12 -- STEP: 29962/36938 -- GLOBAL_STEP: 66900 | > loss_text_ce: 0.0688619539141655 (0.09278088271310572) | > loss_mel_ce: 4.460457801818848 (nan) | > loss: 0.017973491922020912 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13523244849034907) | > loader_time: 0.0047 (0.004499010795920218)  --> TIME: 2025-02-23 23:20:22 -- STEP: 30012/36938 -- GLOBAL_STEP: 66950 | > loss_text_ce: 0.07813245058059692 (0.09274945040653639) | > loss_mel_ce: 0.06305486708879471 (nan) | > loss: 0.0005602671299129725 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13522784586478906) | > loader_time: 0.0045 (0.00449890839677008)  --> TIME: 2025-02-23 23:20:31 -- STEP: 30062/36938 -- GLOBAL_STEP: 67000 | > loss_text_ce: 0.08510307222604752 (0.09271788421920925) | > loss_mel_ce: 4.37435245513916 (nan) | > loss: 0.017696252092719078 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13524306105977255) | > loader_time: 0.0044 (0.00449891072469243)  --> TIME: 2025-02-23 23:20:41 -- STEP: 30112/36938 -- GLOBAL_STEP: 67050 | > loss_text_ce: 0.07480063289403915 (0.09268323359325639) | > loss_mel_ce: 4.23460054397583 (nan) | > loss: 0.017100797966122627 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.13525627221710163) | > loader_time: 0.0044 (0.004498910938768664)  --> TIME: 2025-02-23 23:20:50 -- STEP: 30162/36938 -- GLOBAL_STEP: 67100 | > loss_text_ce: 0.08220929652452469 (0.09265005841379959) | > loss_mel_ce: 3.797071933746338 (nan) | > loss: 0.015393974259495735 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13525776047817503) | > loader_time: 0.0046 (0.0044988422319179435)  --> TIME: 2025-02-23 23:21:00 -- STEP: 30212/36938 -- GLOBAL_STEP: 67150 | > loss_text_ce: 0.08486554026603699 (0.09261690078464291) | > loss_mel_ce: 4.37080717086792 (nan) | > loss: 0.017681242898106575 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13526013208259413) | > loader_time: 0.0045 (0.004498763359352155)  --> TIME: 2025-02-23 23:21:09 -- STEP: 30262/36938 -- GLOBAL_STEP: 67200 | > loss_text_ce: 0.07620401680469513 (0.09258408297322848) | > loss_mel_ce: 4.089136123657227 (nan) | > loss: 0.016529127955436707 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13525881121867744) | > loader_time: 0.0043 (0.004498645102904487)  --> TIME: 2025-02-23 23:21:19 -- STEP: 30312/36938 -- GLOBAL_STEP: 67250 | > loss_text_ce: 0.0754421055316925 (0.0925511750145594) | > loss_mel_ce: 3.079075336456299 (nan) | > loss: 0.012517927214503288 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13526930292101255) | > loader_time: 0.0044 (0.004498602312641244)  --> TIME: 2025-02-23 23:21:28 -- STEP: 30362/36938 -- GLOBAL_STEP: 67300 | > loss_text_ce: 0.05941895768046379 (0.09251925378118166) | > loss_mel_ce: 3.42578125 (nan) | > loss: 0.013830159790813923 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13527255893483814) | > loader_time: 0.0043 (0.004498526392133177)  --> TIME: 2025-02-23 23:21:38 -- STEP: 30412/36938 -- GLOBAL_STEP: 67350 | > loss_text_ce: 0.06374961137771606 (0.09248758155612993) | > loss_mel_ce: 4.3584818840026855 (nan) | > loss: 0.017548538744449615 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13528538819568486) | > loader_time: 0.0044 (0.004498519898088111)  --> TIME: 2025-02-23 23:21:47 -- STEP: 30462/36938 -- GLOBAL_STEP: 67400 | > loss_text_ce: 0.06155173107981682 (0.09245657112089169) | > loss_mel_ce: 0.0868021622300148 (nan) | > loss: 0.0005887059378437698 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13528348636301532) | > loader_time: 0.0042 (0.0044984330602519055)  --> TIME: 2025-02-23 23:21:56 -- STEP: 30512/36938 -- GLOBAL_STEP: 67450 | > loss_text_ce: 0.06292694061994553 (0.09242120146356836) | > loss_mel_ce: 3.774563789367676 (nan) | > loss: 0.015228139236569405 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.1352811390636594) | > loader_time: 0.0043 (0.00449834656171544)  --> TIME: 2025-02-23 23:22:05 -- STEP: 30562/36938 -- GLOBAL_STEP: 67500 | > loss_text_ce: 0.06402833014726639 (0.09238870871512918) | > loss_mel_ce: 0.08502197265625 (nan) | > loss: 0.0005914694629609585 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13528173086523165) | > loader_time: 0.0043 (0.004498339270384603)  --> TIME: 2025-02-23 23:22:15 -- STEP: 30612/36938 -- GLOBAL_STEP: 67550 | > loss_text_ce: 0.06362276524305344 (0.09235395270723612) | > loss_mel_ce: 1.2679781913757324 (nan) | > loss: 0.005284131038933992 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13528054239203716) | > loader_time: 0.0045 (0.004498274010426859)  --> TIME: 2025-02-23 23:22:24 -- STEP: 30662/36938 -- GLOBAL_STEP: 67600 | > loss_text_ce: 0.07558692991733551 (0.09232327813641907) | > loss_mel_ce: 4.323084354400635 (nan) | > loss: 0.017455045133829117 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.1352825118062945) | > loader_time: 0.0044 (0.004498196312238326)  --> TIME: 2025-02-23 23:22:33 -- STEP: 30712/36938 -- GLOBAL_STEP: 67650 | > loss_text_ce: 0.07590123265981674 (0.09229270633121861) | > loss_mel_ce: 3.927401065826416 (nan) | > loss: 0.01588612049818039 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13528051823338724) | > loader_time: 0.0044 (0.004498112035561556)  --> TIME: 2025-02-23 23:22:42 -- STEP: 30762/36938 -- GLOBAL_STEP: 67700 | > loss_text_ce: 0.07525991648435593 (0.09226167617479225) | > loss_mel_ce: 2.4165892601013184 (nan) | > loss: 0.009888291358947754 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13527715663421255) | > loader_time: 0.0045 (0.004498004789323279)  --> TIME: 2025-02-23 23:22:52 -- STEP: 30812/36938 -- GLOBAL_STEP: 67750 | > loss_text_ce: 0.055674489587545395 (0.09222688225009609) | > loss_mel_ce: 0.1135050430893898 (nan) | > loss: 0.0006713473703712225 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13527734860280627) | > loader_time: 0.0045 (0.0044981456725269025)  --> TIME: 2025-02-23 23:23:01 -- STEP: 30862/36938 -- GLOBAL_STEP: 67800 | > loss_text_ce: 0.08054089546203613 (0.09219631410683302) | > loss_mel_ce: 3.5543456077575684 (nan) | > loss: 0.014424153603613377 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.1352773702165474) | > loader_time: 0.0044 (0.004498055019703747)  --> TIME: 2025-02-23 23:23:10 -- STEP: 30912/36938 -- GLOBAL_STEP: 67850 | > loss_text_ce: 0.053824007511138916 (0.09216197498982005) | > loss_mel_ce: 3.760936975479126 (nan) | > loss: 0.015137940645217896 (nan) | > current_lr: 5e-06 | > step_time: 0.1897 (0.13528887693062508) | > loader_time: 0.0047 (0.004497989078918109)  --> TIME: 2025-02-23 23:23:20 -- STEP: 30962/36938 -- GLOBAL_STEP: 67900 | > loss_text_ce: 0.06860090792179108 (0.09212972114846751) | > loss_mel_ce: 4.46343469619751 (nan) | > loss: 0.01798427104949951 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.13529073438181702) | > loader_time: 0.0045 (0.004497929057073292)  --> TIME: 2025-02-23 23:23:29 -- STEP: 31012/36938 -- GLOBAL_STEP: 67950 | > loss_text_ce: 0.09535833448171616 (0.09210169481445851) | > loss_mel_ce: 1.3563319444656372 (nan) | > loss: 0.005760676227509975 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13528933969294896) | > loader_time: 0.0044 (0.0044978904090634386)  --> TIME: 2025-02-23 23:23:38 -- STEP: 31062/36938 -- GLOBAL_STEP: 68000 | > loss_text_ce: 0.07010951638221741 (0.09207364454180446) | > loss_mel_ce: 3.40240740776062 (nan) | > loss: 0.013779830187559128 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13528897262465345) | > loader_time: 0.0045 (0.0044978501814990305)  --> TIME: 2025-02-23 23:23:48 -- STEP: 31112/36938 -- GLOBAL_STEP: 68050 | > loss_text_ce: 0.06836839765310287 (0.09204230511462407) | > loss_mel_ce: 3.7216365337371826 (nan) | > loss: 0.015039702877402306 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13529116017985746) | > loader_time: 0.0045 (0.004497757467459704)  --> TIME: 2025-02-23 23:23:57 -- STEP: 31162/36938 -- GLOBAL_STEP: 68100 | > loss_text_ce: 0.07950121909379959 (0.0920097690610488) | > loss_mel_ce: 0.3563930094242096 (nan) | > loss: 0.0017297390149906278 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13528947991328913) | > loader_time: 0.0044 (0.004497648019951629)  --> TIME: 2025-02-23 23:24:06 -- STEP: 31212/36938 -- GLOBAL_STEP: 68150 | > loss_text_ce: 0.08027124404907227 (0.09197943976348798) | > loss_mel_ce: 4.30322790145874 (nan) | > loss: 0.017394838854670525 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.13529552386599253) | > loader_time: 0.0045 (0.004497601071428251)  --> TIME: 2025-02-23 23:24:16 -- STEP: 31262/36938 -- GLOBAL_STEP: 68200 | > loss_text_ce: 0.0766255259513855 (0.09195019406363128) | > loss_mel_ce: 3.5280439853668213 (nan) | > loss: 0.01430424489080906 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13529593375920188) | > loader_time: 0.0046 (0.004497634373854545)  --> TIME: 2025-02-23 23:24:25 -- STEP: 31312/36938 -- GLOBAL_STEP: 68250 | > loss_text_ce: 0.08716528862714767 (0.09191579670944192) | > loss_mel_ce: 4.9412617683410645 (nan) | > loss: 0.019954077899456024 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13530064261136207) | > loader_time: 0.0044 (0.004497615754695715)  --> TIME: 2025-02-23 23:24:34 -- STEP: 31362/36938 -- GLOBAL_STEP: 68300 | > loss_text_ce: 0.06744188815355301 (0.09188362847550181) | > loss_mel_ce: 0.5600934624671936 (nan) | > loss: 0.002490219660103321 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13530068701303755) | > loader_time: 0.0043 (0.00449756451326931)  --> TIME: 2025-02-23 23:24:44 -- STEP: 31412/36938 -- GLOBAL_STEP: 68350 | > loss_text_ce: 0.08388151973485947 (0.09185480544592436) | > loss_mel_ce: 0.0531412772834301 (nan) | > loss: 0.0005437412764877081 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13530521964049189) | > loader_time: 0.0043 (0.004497609206191999)  --> TIME: 2025-02-23 23:24:53 -- STEP: 31462/36938 -- GLOBAL_STEP: 68400 | > loss_text_ce: 0.06656721234321594 (0.09182566851301742) | > loss_mel_ce: 3.4961063861846924 (nan) | > loss: 0.014137594029307365 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.135303234009769) | > loader_time: 0.0044 (0.004497531895476559)  --> TIME: 2025-02-23 23:25:02 -- STEP: 31512/36938 -- GLOBAL_STEP: 68450 | > loss_text_ce: 0.07227631658315659 (0.09179511801669549) | > loss_mel_ce: 4.912193775177002 (nan) | > loss: 0.019779643043875694 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13529552147454363) | > loader_time: 0.0046 (0.004497505809547155)  --> TIME: 2025-02-23 23:25:11 -- STEP: 31562/36938 -- GLOBAL_STEP: 68500 | > loss_text_ce: 0.06778651475906372 (0.09176601761702206) | > loss_mel_ce: 4.012897491455078 (nan) | > loss: 0.016193192452192307 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.13529565791939843) | > loader_time: 0.0045 (0.004497503057404564)  --> TIME: 2025-02-23 23:25:21 -- STEP: 31612/36938 -- GLOBAL_STEP: 68550 | > loss_text_ce: 0.07580545544624329 (0.09173443264326918) | > loss_mel_ce: 3.3645315170288086 (nan) | > loss: 0.013652131892740726 (nan) | > current_lr: 5e-06 | > step_time: 0.1591 (0.13530589708961466) | > loader_time: 0.0046 (0.004497497591295986)  --> TIME: 2025-02-23 23:25:30 -- STEP: 31662/36938 -- GLOBAL_STEP: 68600 | > loss_text_ce: 0.09152180701494217 (0.09170261875026076) | > loss_mel_ce: 3.3409945964813232 (nan) | > loss: 0.013621097430586815 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13530629085248896) | > loader_time: 0.0043 (0.004497456073188701)  --> TIME: 2025-02-23 23:25:39 -- STEP: 31712/36938 -- GLOBAL_STEP: 68650 | > loss_text_ce: 0.06501137465238571 (0.09167067353274448) | > loss_mel_ce: 3.9671640396118164 (nan) | > loss: 0.016000697389245033 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13529736647902948) | > loader_time: 0.0044 (0.004497364855077268)  --> TIME: 2025-02-23 23:25:48 -- STEP: 31762/36938 -- GLOBAL_STEP: 68700 | > loss_text_ce: 0.08069732785224915 (0.09164007861343393) | > loss_mel_ce: 3.800240993499756 (nan) | > loss: 0.01540054939687252 (nan) | > current_lr: 5e-06 | > step_time: 0.1888 (0.13529976558553122) | > loader_time: 0.0047 (0.004497356457131075)  --> TIME: 2025-02-23 23:25:58 -- STEP: 31812/36938 -- GLOBAL_STEP: 68750 | > loss_text_ce: 0.06791742146015167 (0.09160288898212272) | > loss_mel_ce: 3.419351816177368 (nan) | > loss: 0.013838370330631733 (nan) | > current_lr: 5e-06 | > step_time: 0.1565 (0.13529952167790077) | > loader_time: 0.0045 (0.004497276691914988)  --> TIME: 2025-02-23 23:26:07 -- STEP: 31862/36938 -- GLOBAL_STEP: 68800 | > loss_text_ce: 0.05012551695108414 (0.09157038100826619) | > loss_mel_ce: 4.4367899894714355 (nan) | > loss: 0.017805222421884537 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13529757656894476) | > loader_time: 0.0043 (0.004497166235458457)  --> TIME: 2025-02-23 23:26:16 -- STEP: 31912/36938 -- GLOBAL_STEP: 68850 | > loss_text_ce: 0.05982637405395508 (0.09153748074650152) | > loss_mel_ce: 4.238417148590088 (nan) | > loss: 0.017056522890925407 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13529989397534684) | > loader_time: 0.0044 (0.0044971730631973945)  --> TIME: 2025-02-23 23:26:25 -- STEP: 31962/36938 -- GLOBAL_STEP: 68900 | > loss_text_ce: 0.0756751000881195 (0.09150415330735538) | > loss_mel_ce: 3.8716835975646973 (nan) | > loss: 0.015664122998714447 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13529970101962205) | > loader_time: 0.0048 (0.004497167278041787)  --> TIME: 2025-02-23 23:26:35 -- STEP: 32012/36938 -- GLOBAL_STEP: 68950 | > loss_text_ce: 0.08193683624267578 (0.09147347666055099) | > loss_mel_ce: 4.412232398986816 (nan) | > loss: 0.017834005877375603 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13530075099488123) | > loader_time: 0.0044 (0.0044970211722948255)  --> TIME: 2025-02-23 23:26:44 -- STEP: 32062/36938 -- GLOBAL_STEP: 69000 | > loss_text_ce: 0.060608815401792526 (0.09144207573816002) | > loss_mel_ce: 2.019784688949585 (nan) | > loss: 0.008255530148744583 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13530821559597306) | > loader_time: 0.0043 (0.004497106073353464)  --> TIME: 2025-02-23 23:26:53 -- STEP: 32112/36938 -- GLOBAL_STEP: 69050 | > loss_text_ce: 0.07458635419607162 (0.09141227772100256) | > loss_mel_ce: 3.5804426670074463 (nan) | > loss: 0.01450408436357975 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13530419440963157) | > loader_time: 0.0042 (0.004496913906288911)  --> TIME: 2025-02-23 23:27:02 -- STEP: 32162/36938 -- GLOBAL_STEP: 69100 | > loss_text_ce: 0.06554822623729706 (0.09138511280896934) | > loss_mel_ce: 4.598541259765625 (nan) | > loss: 0.018508292734622955 (nan) | > current_lr: 5e-06 | > step_time: 0.1853 (0.135303173630949) | > loader_time: 0.0046 (0.004496786111208142)  --> TIME: 2025-02-23 23:27:11 -- STEP: 32212/36938 -- GLOBAL_STEP: 69150 | > loss_text_ce: 0.06803981959819794 (0.09135695451998503) | > loss_mel_ce: 2.8342950344085693 (nan) | > loss: 0.011517202481627464 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13529631139831447) | > loader_time: 0.0044 (0.004496709591081453)  --> TIME: 2025-02-23 23:27:21 -- STEP: 32262/36938 -- GLOBAL_STEP: 69200 | > loss_text_ce: 0.07624314725399017 (0.09132725570710493) | > loss_mel_ce: 3.839329957962036 (nan) | > loss: 0.0155379893258214 (nan) | > current_lr: 5e-06 | > step_time: 0.1779 (0.13529897395339585) | > loader_time: 0.0046 (0.004496723600066788)  --> TIME: 2025-02-23 23:27:30 -- STEP: 32312/36938 -- GLOBAL_STEP: 69250 | > loss_text_ce: 0.045041196048259735 (0.09129476243518023) | > loss_mel_ce: 1.9253169298171997 (nan) | > loss: 0.00781888235360384 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1352995573943066) | > loader_time: 0.0044 (0.004496760195982168)  --> TIME: 2025-02-23 23:27:39 -- STEP: 32362/36938 -- GLOBAL_STEP: 69300 | > loss_text_ce: 0.059556953608989716 (0.0912621867774541) | > loss_mel_ce: 4.157032489776611 (nan) | > loss: 0.01673249900341034 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13529672195233405) | > loader_time: 0.0045 (0.004496750478860618)  --> TIME: 2025-02-23 23:27:49 -- STEP: 32412/36938 -- GLOBAL_STEP: 69350 | > loss_text_ce: 0.06821868568658829 (0.09123051942809006) | > loss_mel_ce: 4.098444938659668 (nan) | > loss: 0.016534380614757538 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13530237589679167) | > loader_time: 0.0045 (0.004496766934494291)  --> TIME: 2025-02-23 23:27:58 -- STEP: 32462/36938 -- GLOBAL_STEP: 69400 | > loss_text_ce: 0.050892431288957596 (0.09120070096822612) | > loss_mel_ce: 2.9991567134857178 (nan) | > loss: 0.012103370390832424 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.1352988295557752) | > loader_time: 0.0044 (0.004496708917174191)  --> TIME: 2025-02-23 23:28:07 -- STEP: 32512/36938 -- GLOBAL_STEP: 69450 | > loss_text_ce: 0.09068245440721512 (0.09117152464280937) | > loss_mel_ce: 3.972216844558716 (nan) | > loss: 0.016122616827487946 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.1353025497560654) | > loader_time: 0.0044 (0.004496598777573882)  --> TIME: 2025-02-23 23:28:17 -- STEP: 32562/36938 -- GLOBAL_STEP: 69500 | > loss_text_ce: 0.06855633854866028 (0.09114254485157047) | > loss_mel_ce: 4.486130714416504 (nan) | > loss: 0.01807415671646595 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.1353113295207343) | > loader_time: 0.0045 (0.004496495712448507)  --> TIME: 2025-02-23 23:28:26 -- STEP: 32612/36938 -- GLOBAL_STEP: 69550 | > loss_text_ce: 0.06833844631910324 (0.09111222774786484) | > loss_mel_ce: 3.754335641860962 (nan) | > loss: 0.015169342048466206 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13530929790718071) | > loader_time: 0.0044 (0.004496403863704034)  --> TIME: 2025-02-23 23:28:35 -- STEP: 32662/36938 -- GLOBAL_STEP: 69600 | > loss_text_ce: 0.05808829516172409 (0.09108069262564705) | > loss_mel_ce: 3.750697612762451 (nan) | > loss: 0.015114231035113335 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13530802096062627) | > loader_time: 0.0043 (0.004496235322196038)  --> TIME: 2025-02-23 23:28:44 -- STEP: 32712/36938 -- GLOBAL_STEP: 69650 | > loss_text_ce: 0.06384700536727905 (0.09104753022785288) | > loss_mel_ce: 4.29926872253418 (nan) | > loss: 0.01731395162642002 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13531025669836472) | > loader_time: 0.0043 (0.004496096303801751)  --> TIME: 2025-02-23 23:28:54 -- STEP: 32762/36938 -- GLOBAL_STEP: 69700 | > loss_text_ce: 0.07194656133651733 (0.09101694647610989) | > loss_mel_ce: 3.442420244216919 (nan) | > loss: 0.013945900835096836 (nan) | > current_lr: 5e-06 | > step_time: 0.1876 (0.13531213072031592) | > loader_time: 0.0047 (0.00449593913809123)  --> TIME: 2025-02-23 23:29:03 -- STEP: 32812/36938 -- GLOBAL_STEP: 69750 | > loss_text_ce: 0.06469137221574783 (0.09098734673147632) | > loss_mel_ce: 3.8423912525177 (nan) | > loss: 0.015504296869039536 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.13531505888618311) | > loader_time: 0.0046 (0.004495796664057894)  --> TIME: 2025-02-23 23:29:12 -- STEP: 32862/36938 -- GLOBAL_STEP: 69800 | > loss_text_ce: 0.08174166828393936 (0.09095830259108183) | > loss_mel_ce: 4.167700290679932 (nan) | > loss: 0.016862865537405014 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13531577984709572) | > loader_time: 0.0046 (0.004495720057730973)  --> TIME: 2025-02-23 23:29:22 -- STEP: 32912/36938 -- GLOBAL_STEP: 69850 | > loss_text_ce: 0.06458275765180588 (0.09092730448427827) | > loss_mel_ce: 1.559326171875 (nan) | > loss: 0.00644408306106925 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13531848404322824) | > loader_time: 0.0043 (0.00449562540724258)  --> TIME: 2025-02-23 23:29:31 -- STEP: 32962/36938 -- GLOBAL_STEP: 69900 | > loss_text_ce: 0.05896301195025444 (0.09089759380659691) | > loss_mel_ce: 0.080718994140625 (nan) | > loss: 0.0005542937433347106 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13532144473916988) | > loader_time: 0.0042 (0.004495469844353577)  --> TIME: 2025-02-23 23:29:40 -- STEP: 33012/36938 -- GLOBAL_STEP: 69950 | > loss_text_ce: 0.06307649612426758 (0.09086857238592852) | > loss_mel_ce: 4.558538913726807 (nan) | > loss: 0.018339743837714195 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13531909877425055) | > loader_time: 0.0043 (0.004495354373570894)  --> TIME: 2025-02-23 23:29:50 -- STEP: 33062/36938 -- GLOBAL_STEP: 70000 | > loss_text_ce: 0.05745428055524826 (0.09084034615956489) | > loss_mel_ce: 2.3439581394195557 (nan) | > loss: 0.009529415518045425 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.13532057599402286) | > loader_time: 0.0043 (0.004495303403385814) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_70000.pth  --> TIME: 2025-02-23 23:30:04 -- STEP: 33112/36938 -- GLOBAL_STEP: 70050 | > loss_text_ce: 0.06968741863965988 (0.09080592276900681) | > loss_mel_ce: 3.6895864009857178 (nan) | > loss: 0.014917753636837006 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13531475133093981) | > loader_time: 0.0044 (0.004495431523473982)  --> TIME: 2025-02-23 23:30:13 -- STEP: 33162/36938 -- GLOBAL_STEP: 70100 | > loss_text_ce: 0.08139915019273758 (0.09077595735779771) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.1353125574217647) | > loader_time: 0.0046 (0.004495460588360763)  --> TIME: 2025-02-23 23:30:23 -- STEP: 33212/36938 -- GLOBAL_STEP: 70150 | > loss_text_ce: 0.0792960450053215 (0.09074722052380563) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13531819549107024) | > loader_time: 0.0044 (0.004496206967899289)  --> TIME: 2025-02-23 23:30:32 -- STEP: 33262/36938 -- GLOBAL_STEP: 70200 | > loss_text_ce: 0.069854237139225 (0.09071663568772674) | > loss_mel_ce: 4.137514591217041 (nan) | > loss: 0.016695909202098846 (nan) | > current_lr: 5e-06 | > step_time: 0.186 (0.13531784897751958) | > loader_time: 0.0047 (0.0044966490268277225)  --> TIME: 2025-02-23 23:30:41 -- STEP: 33312/36938 -- GLOBAL_STEP: 70250 | > loss_text_ce: 0.08263639360666275 (0.09068580606527836) | > loss_mel_ce: 4.007956504821777 (nan) | > loss: 0.016232512891292572 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13531261050535032) | > loader_time: 0.0046 (0.004496618532768017)  --> TIME: 2025-02-23 23:30:50 -- STEP: 33362/36938 -- GLOBAL_STEP: 70300 | > loss_text_ce: 0.07760680466890335 (0.09065588102513818) | > loss_mel_ce: 3.2880184650421143 (nan) | > loss: 0.013355657458305359 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13530812305080817) | > loader_time: 0.0045 (0.004496916257594033)  --> TIME: 2025-02-23 23:30:59 -- STEP: 33412/36938 -- GLOBAL_STEP: 70350 | > loss_text_ce: 0.06050942838191986 (0.09062360519962324) | > loss_mel_ce: 3.4028701782226562 (nan) | > loss: 0.013743571005761623 (nan) | > current_lr: 5e-06 | > step_time: 0.1352 (0.13531287061401678) | > loader_time: 0.0045 (0.0044970578609591265)  --> TIME: 2025-02-23 23:31:09 -- STEP: 33462/36938 -- GLOBAL_STEP: 70400 | > loss_text_ce: 0.08115245401859283 (0.09059287999559004) | > loss_mel_ce: 4.730939865112305 (nan) | > loss: 0.019095605239272118 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.13531341552591808) | > loader_time: 0.0046 (0.004497136148290082)  --> TIME: 2025-02-23 23:31:17 -- STEP: 33512/36938 -- GLOBAL_STEP: 70450 | > loss_text_ce: 0.06296166032552719 (0.09055988329639936) | > loss_mel_ce: 3.0536751747131348 (nan) | > loss: 0.012367607094347477 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13530101516304222) | > loader_time: 0.0043 (0.004497080963077398)  --> TIME: 2025-02-23 23:31:27 -- STEP: 33562/36938 -- GLOBAL_STEP: 70500 | > loss_text_ce: 0.07488293945789337 (0.09052948088777409) | > loss_mel_ce: 3.4000651836395264 (nan) | > loss: 0.01378947775810957 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13530298665598284) | > loader_time: 0.0045 (0.004497117333002709)  --> TIME: 2025-02-23 23:31:36 -- STEP: 33612/36938 -- GLOBAL_STEP: 70550 | > loss_text_ce: 0.06512946635484695 (0.09049792482067608) | > loss_mel_ce: 3.7055227756500244 (nan) | > loss: 0.014962906949222088 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1353021794084445) | > loader_time: 0.0046 (0.004497183648855875)  --> TIME: 2025-02-23 23:31:45 -- STEP: 33662/36938 -- GLOBAL_STEP: 70600 | > loss_text_ce: 0.07701851427555084 (0.09046622408486157) | > loss_mel_ce: 3.608360767364502 (nan) | > loss: 0.014624522067606449 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.1353086231085042) | > loader_time: 0.0045 (0.004497307520216995)  --> TIME: 2025-02-23 23:31:55 -- STEP: 33712/36938 -- GLOBAL_STEP: 70650 | > loss_text_ce: 0.08361535519361496 (0.09043700247793088) | > loss_mel_ce: 4.021435260772705 (nan) | > loss: 0.016289884224534035 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13531202089888308) | > loader_time: 0.0043 (0.0044975762874750825)  --> TIME: 2025-02-23 23:32:04 -- STEP: 33762/36938 -- GLOBAL_STEP: 70700 | > loss_text_ce: 0.07461213320493698 (0.09040784968443835) | > loss_mel_ce: 4.142870903015137 (nan) | > loss: 0.016736045479774475 (nan) | > current_lr: 5e-06 | > step_time: 0.1265 (0.135307313198091) | > loader_time: 0.0046 (0.004497665518882764)  --> TIME: 2025-02-23 23:32:13 -- STEP: 33812/36938 -- GLOBAL_STEP: 70750 | > loss_text_ce: 0.06244784593582153 (0.09038011859075608) | > loss_mel_ce: 3.363293170928955 (nan) | > loss: 0.01359421107918024 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13530424142411224) | > loader_time: 0.0044 (0.00449770211637563)  --> TIME: 2025-02-23 23:32:22 -- STEP: 33862/36938 -- GLOBAL_STEP: 70800 | > loss_text_ce: 0.07718179374933243 (0.0903485175346478) | > loss_mel_ce: 4.335980415344238 (nan) | > loss: 0.017512548714876175 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.13530469067470796) | > loader_time: 0.0046 (0.004497739302838139)  --> TIME: 2025-02-23 23:32:32 -- STEP: 33912/36938 -- GLOBAL_STEP: 70850 | > loss_text_ce: 0.06597951054573059 (0.09031796176107404) | > loss_mel_ce: 4.005851745605469 (nan) | > loss: 0.01615806110203266 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.13530968626340162) | > loader_time: 0.0046 (0.00449767966798016)  --> TIME: 2025-02-23 23:32:41 -- STEP: 33962/36938 -- GLOBAL_STEP: 70900 | > loss_text_ce: 0.06896711885929108 (0.09028772373410826) | > loss_mel_ce: 4.540200710296631 (nan) | > loss: 0.01829035021364689 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1353114383829032) | > loader_time: 0.0044 (0.0044976282748757064)  --> TIME: 2025-02-23 23:32:50 -- STEP: 34012/36938 -- GLOBAL_STEP: 70950 | > loss_text_ce: 0.06777643412351608 (0.09025690826938677) | > loss_mel_ce: 3.8862321376800537 (nan) | > loss: 0.015690511092543602 (nan) | > current_lr: 5e-06 | > step_time: 0.1624 (0.13530775756397329) | > loader_time: 0.0046 (0.004497548797250312)  --> TIME: 2025-02-23 23:33:00 -- STEP: 34062/36938 -- GLOBAL_STEP: 71000 | > loss_text_ce: 0.057377975434064865 (0.0902270160865463) | > loss_mel_ce: 1.7280433177947998 (nan) | > loss: 0.007085005287081003 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.1353121555724368) | > loader_time: 0.0044 (0.004497600346497433)  --> TIME: 2025-02-23 23:33:09 -- STEP: 34112/36938 -- GLOBAL_STEP: 71050 | > loss_text_ce: 0.0703696683049202 (0.09019634228775937) | > loss_mel_ce: 3.6323678493499756 (nan) | > loss: 0.014693403616547585 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.1353142291139699) | > loader_time: 0.0044 (0.004497548317148804)  --> TIME: 2025-02-23 23:33:19 -- STEP: 34162/36938 -- GLOBAL_STEP: 71100 | > loss_text_ce: 0.07204475998878479 (0.09016797302839233) | > loss_mel_ce: 4.4538798332214355 (nan) | > loss: 0.017960019409656525 (nan) | > current_lr: 5e-06 | > step_time: 0.1582 (0.13532377901529993) | > loader_time: 0.0045 (0.004497533373276889)  --> TIME: 2025-02-23 23:33:28 -- STEP: 34212/36938 -- GLOBAL_STEP: 71150 | > loss_text_ce: 0.08083857595920563 (0.09014214004073003) | > loss_mel_ce: 4.0186944007873535 (nan) | > loss: 0.016267988830804825 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13531400127242446) | > loader_time: 0.0045 (0.0044974558439559165)  --> TIME: 2025-02-23 23:33:37 -- STEP: 34262/36938 -- GLOBAL_STEP: 71200 | > loss_text_ce: 0.06787166744470596 (0.09011266990760239) | > loss_mel_ce: 4.312187671661377 (nan) | > loss: 0.01738118939101696 (nan) | > current_lr: 5e-06 | > step_time: 0.1766 (0.13531583274463574) | > loader_time: 0.0047 (0.004497433438010771)  --> TIME: 2025-02-23 23:33:46 -- STEP: 34312/36938 -- GLOBAL_STEP: 71250 | > loss_text_ce: 0.07328318059444427 (0.09008199573874744) | > loss_mel_ce: 3.6577155590057373 (nan) | > loss: 0.014805551618337631 (nan) | > current_lr: 5e-06 | > step_time: 0.1867 (0.1353128942736763) | > loader_time: 0.0047 (0.004497415308186323)  --> TIME: 2025-02-23 23:33:56 -- STEP: 34362/36938 -- GLOBAL_STEP: 71300 | > loss_text_ce: 0.06639604270458221 (0.0900510034436798) | > loss_mel_ce: 4.306623935699463 (nan) | > loss: 0.017353255301713943 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13531959022161988) | > loader_time: 0.0045 (0.004497417373406331)  --> TIME: 2025-02-23 23:34:05 -- STEP: 34412/36938 -- GLOBAL_STEP: 71350 | > loss_text_ce: 0.06907331198453903 (0.09002162240877375) | > loss_mel_ce: 3.827548027038574 (nan) | > loss: 0.01546278316527605 (nan) | > current_lr: 5e-06 | > step_time: 0.152 (0.13532393172384963) | > loader_time: 0.0045 (0.004497533632713209)  --> TIME: 2025-02-23 23:34:14 -- STEP: 34462/36938 -- GLOBAL_STEP: 71400 | > loss_text_ce: 0.07544036954641342 (0.08999302106940354) | > loss_mel_ce: 3.7638378143310547 (nan) | > loss: 0.015235232189297676 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.13532528878780248) | > loader_time: 0.0046 (0.004497479405905958)  --> TIME: 2025-02-23 23:34:24 -- STEP: 34512/36938 -- GLOBAL_STEP: 71450 | > loss_text_ce: 0.08503863960504532 (0.08996328692835794) | > loss_mel_ce: 3.89961314201355 (nan) | > loss: 0.015812112018465996 (nan) | > current_lr: 5e-06 | > step_time: 0.1451 (0.1353274140719404) | > loader_time: 0.0045 (0.004497442406584501)  --> TIME: 2025-02-23 23:34:33 -- STEP: 34562/36938 -- GLOBAL_STEP: 71500 | > loss_text_ce: 0.07525822520256042 (0.08993616194819076) | > loss_mel_ce: 3.7876925468444824 (nan) | > loss: 0.015329170972108841 (nan) | > current_lr: 5e-06 | > step_time: 0.1574 (0.13532913683134112) | > loader_time: 0.0047 (0.004497458493155632)  --> TIME: 2025-02-23 23:34:42 -- STEP: 34612/36938 -- GLOBAL_STEP: 71550 | > loss_text_ce: 0.07489348948001862 (0.08990794859781823) | > loss_mel_ce: 3.737576723098755 (nan) | > loss: 0.015128850936889648 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13532875282122628) | > loader_time: 0.0044 (0.004497446800868084)  --> TIME: 2025-02-23 23:34:52 -- STEP: 34662/36938 -- GLOBAL_STEP: 71600 | > loss_text_ce: 0.0600086972117424 (0.08987927430132968) | > loss_mel_ce: 3.7157270908355713 (nan) | > loss: 0.014983079396188259 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.1353300736811772) | > loader_time: 0.0045 (0.004497493911233832)  --> TIME: 2025-02-23 23:35:01 -- STEP: 34712/36938 -- GLOBAL_STEP: 71650 | > loss_text_ce: 0.06905487179756165 (0.08984984168963164) | > loss_mel_ce: 1.16766357421875 (nan) | > loss: 0.004907613154500723 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.1353325748682617) | > loader_time: 0.0046 (0.0044974289297041005)  --> TIME: 2025-02-23 23:35:10 -- STEP: 34762/36938 -- GLOBAL_STEP: 71700 | > loss_text_ce: 0.06295230984687805 (0.08981904019649636) | > loss_mel_ce: 3.7984871864318848 (nan) | > loss: 0.015323173254728317 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13533765186985483) | > loader_time: 0.0043 (0.0044973318585454555)  --> TIME: 2025-02-23 23:35:20 -- STEP: 34812/36938 -- GLOBAL_STEP: 71750 | > loss_text_ce: 0.09665364027023315 (0.08979188487696523) | > loss_mel_ce: 0.09640225768089294 (nan) | > loss: 0.0007660948904231191 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13533761517612175) | > loader_time: 0.0043 (0.0044972425382137)  --> TIME: 2025-02-23 23:35:29 -- STEP: 34862/36938 -- GLOBAL_STEP: 71800 | > loss_text_ce: 0.07608841359615326 (0.08976167116324253) | > loss_mel_ce: 3.816406488418579 (nan) | > loss: 0.015446409583091736 (nan) | > current_lr: 5e-06 | > step_time: 0.1783 (0.13533538190463956) | > loader_time: 0.0046 (0.004497194856424064)  --> TIME: 2025-02-23 23:35:38 -- STEP: 34912/36938 -- GLOBAL_STEP: 71850 | > loss_text_ce: 0.06569027155637741 (0.08973152486948496) | > loss_mel_ce: 4.328125 (nan) | > loss: 0.017435774207115173 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13534183980509418) | > loader_time: 0.0044 (0.00449721711174705)  --> TIME: 2025-02-23 23:35:48 -- STEP: 34962/36938 -- GLOBAL_STEP: 71900 | > loss_text_ce: 0.06930338591337204 (0.08970482304767316) | > loss_mel_ce: 1.6875598430633545 (nan) | > loss: 0.0069716800935566425 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.1353546409978988) | > loader_time: 0.0043 (0.004497276516679726)  --> TIME: 2025-02-23 23:35:58 -- STEP: 35012/36938 -- GLOBAL_STEP: 71950 | > loss_text_ce: 0.07674416899681091 (0.08967706168843574) | > loss_mel_ce: 0.14065390825271606 (nan) | > loss: 0.0008626908529549837 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13536306437009932) | > loader_time: 0.0044 (0.004497274846659105)  --> TIME: 2025-02-23 23:36:07 -- STEP: 35062/36938 -- GLOBAL_STEP: 72000 | > loss_text_ce: 0.08501012623310089 (0.08965006840808085) | > loss_mel_ce: 4.5752177238464355 (nan) | > loss: 0.018492968752980232 (nan) | > current_lr: 5e-06 | > step_time: 0.1505 (0.13535543654555268) | > loader_time: 0.0045 (0.004497239576227331)  --> TIME: 2025-02-23 23:36:16 -- STEP: 35112/36938 -- GLOBAL_STEP: 72050 | > loss_text_ce: 0.06657934188842773 (0.08962484577112236) | > loss_mel_ce: 4.684488773345947 (nan) | > loss: 0.0188534464687109 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13535402090844348) | > loader_time: 0.0045 (0.004497188598589787)  --> TIME: 2025-02-23 23:36:25 -- STEP: 35162/36938 -- GLOBAL_STEP: 72100 | > loss_text_ce: 0.09596467018127441 (0.08959480887737171) | > loss_mel_ce: 4.086520671844482 (nan) | > loss: 0.016597166657447815 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13535360233971186) | > loader_time: 0.005 (0.004497521390524897)  --> TIME: 2025-02-23 23:36:35 -- STEP: 35212/36938 -- GLOBAL_STEP: 72150 | > loss_text_ce: 0.07669112831354141 (0.08956779518150876) | > loss_mel_ce: 3.77433443069458 (nan) | > loss: 0.01528184860944748 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.135353855913123) | > loader_time: 0.0047 (0.004498239966748613)  --> TIME: 2025-02-23 23:36:44 -- STEP: 35262/36938 -- GLOBAL_STEP: 72200 | > loss_text_ce: 0.07871624082326889 (0.08953799439065856) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13534921050362364) | > loader_time: 0.0046 (0.00449870747125978)  --> TIME: 2025-02-23 23:36:53 -- STEP: 35312/36938 -- GLOBAL_STEP: 72250 | > loss_text_ce: 0.07458887249231339 (0.08950897302838545) | > loss_mel_ce: 3.9265103340148926 (nan) | > loss: 0.01587737910449505 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.13535100721264973) | > loader_time: 0.0045 (0.004499067817816752)  --> TIME: 2025-02-23 23:37:02 -- STEP: 35362/36938 -- GLOBAL_STEP: 72300 | > loss_text_ce: 0.0715416669845581 (0.08947991027709228) | > loss_mel_ce: 4.120433807373047 (nan) | > loss: 0.016634825617074966 (nan) | > current_lr: 5e-06 | > step_time: 0.1842 (0.1353487102203681) | > loader_time: 0.0046 (0.004499106357003363)  --> TIME: 2025-02-23 23:37:12 -- STEP: 35412/36938 -- GLOBAL_STEP: 72350 | > loss_text_ce: 0.07281532883644104 (0.08945403226411333) | > loss_mel_ce: 3.5338807106018066 (nan) | > loss: 0.014312286861240864 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13535109808333678) | > loader_time: 0.0045 (0.004499118812579985)  --> TIME: 2025-02-23 23:37:21 -- STEP: 35462/36938 -- GLOBAL_STEP: 72400 | > loss_text_ce: 0.0676458477973938 (0.08942417803788127) | > loss_mel_ce: 3.866487741470337 (nan) | > loss: 0.01561164204031229 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13535481200902003) | > loader_time: 0.0044 (0.004499158831822213)  --> TIME: 2025-02-23 23:37:30 -- STEP: 35512/36938 -- GLOBAL_STEP: 72450 | > loss_text_ce: 0.09746643900871277 (0.08939813871717563) | > loss_mel_ce: 3.9142744541168213 (nan) | > loss: 0.015919607132673264 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.1353589523735542) | > loader_time: 0.0044 (0.004499225982757099)  --> TIME: 2025-02-23 23:37:40 -- STEP: 35562/36938 -- GLOBAL_STEP: 72500 | > loss_text_ce: 0.08353348821401596 (0.08936965308748707) | > loss_mel_ce: 2.872818946838379 (nan) | > loss: 0.011731558479368687 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.1353587107250489) | > loader_time: 0.0045 (0.004499186855904202)  --> TIME: 2025-02-23 23:37:49 -- STEP: 35612/36938 -- GLOBAL_STEP: 72550 | > loss_text_ce: 0.08389773964881897 (0.08934006708488286) | > loss_mel_ce: 4.397509574890137 (nan) | > loss: 0.017783362418413162 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.1353639417935599) | > loader_time: 0.0045 (0.004499163411245516)  --> TIME: 2025-02-23 23:37:59 -- STEP: 35662/36938 -- GLOBAL_STEP: 72600 | > loss_text_ce: 0.07290098816156387 (0.089311528043059) | > loss_mel_ce: 3.4486546516418457 (nan) | > loss: 0.013974428176879883 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.1353696138845015) | > loader_time: 0.0047 (0.004499158076513669)  --> TIME: 2025-02-23 23:38:08 -- STEP: 35712/36938 -- GLOBAL_STEP: 72650 | > loss_text_ce: 0.07943695038557053 (0.08928189130270929) | > loss_mel_ce: 3.9511425495147705 (nan) | > loss: 0.015994364395737648 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13537073851726514) | > loader_time: 0.0044 (0.004499127147018263)  --> TIME: 2025-02-23 23:38:17 -- STEP: 35762/36938 -- GLOBAL_STEP: 72700 | > loss_text_ce: 0.04266357421875 (0.08925036408354486) | > loss_mel_ce: 4.056376934051514 (nan) | > loss: 0.01626603491604328 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.135367866677396) | > loader_time: 0.0044 (0.0044990756568889425)  --> TIME: 2025-02-23 23:38:26 -- STEP: 35812/36938 -- GLOBAL_STEP: 72750 | > loss_text_ce: 0.07248424738645554 (0.08922376054566236) | > loss_mel_ce: 3.1871018409729004 (nan) | > loss: 0.012934866361320019 (nan) | > current_lr: 5e-06 | > step_time: 0.1618 (0.13536303123654916) | > loader_time: 0.0047 (0.004499085706052304)  --> TIME: 2025-02-23 23:38:35 -- STEP: 35862/36938 -- GLOBAL_STEP: 72800 | > loss_text_ce: 0.06921501457691193 (0.08919329938255106) | > loss_mel_ce: 4.48399019241333 (nan) | > loss: 0.01806827448308468 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13536086420720023) | > loader_time: 0.0045 (0.004499105972105799)  --> TIME: 2025-02-23 23:38:45 -- STEP: 35912/36938 -- GLOBAL_STEP: 72850 | > loss_text_ce: 0.06741713732481003 (0.08916290310254264) | > loss_mel_ce: 3.7132091522216797 (nan) | > loss: 0.015002486295998096 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13536458627952913) | > loader_time: 0.0044 (0.0044990883595323605)  --> TIME: 2025-02-23 23:38:54 -- STEP: 35962/36938 -- GLOBAL_STEP: 72900 | > loss_text_ce: 0.07124021649360657 (0.08913398317189493) | > loss_mel_ce: 4.062674045562744 (nan) | > loss: 0.016404423862695694 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.13537311099155105) | > loader_time: 0.0046 (0.004499160993497659)  --> TIME: 2025-02-23 23:39:03 -- STEP: 36012/36938 -- GLOBAL_STEP: 72950 | > loss_text_ce: 0.08184024691581726 (0.08910366671476852) | > loss_mel_ce: 3.0459134578704834 (nan) | > loss: 0.012411721982061863 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.1353650275672866) | > loader_time: 0.0047 (0.004499224951488896)  --> TIME: 2025-02-23 23:39:13 -- STEP: 36062/36938 -- GLOBAL_STEP: 73000 | > loss_text_ce: 0.057419851422309875 (0.08907437372006895) | > loss_mel_ce: 3.5186569690704346 (nan) | > loss: 0.014190781861543655 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13537342208150135) | > loader_time: 0.0045 (0.004499244984807709)  --> TIME: 2025-02-23 23:39:22 -- STEP: 36112/36938 -- GLOBAL_STEP: 73050 | > loss_text_ce: 0.08068957924842834 (0.08904631582764906) | > loss_mel_ce: 1.8499929904937744 (nan) | > loss: 0.007661439012736082 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13537211154988193) | > loader_time: 0.0044 (0.004499277335170312)  --> TIME: 2025-02-23 23:39:31 -- STEP: 36162/36938 -- GLOBAL_STEP: 73100 | > loss_text_ce: 0.06920802593231201 (0.08901954251027612) | > loss_mel_ce: 4.470242023468018 (nan) | > loss: 0.018013691529631615 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13537215313600995) | > loader_time: 0.0047 (0.004499219600674749)  --> TIME: 2025-02-23 23:39:40 -- STEP: 36212/36938 -- GLOBAL_STEP: 73150 | > loss_text_ce: 0.07544154673814774 (0.08899287221764837) | > loss_mel_ce: 3.805119752883911 (nan) | > loss: 0.015399053692817688 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13536918658413766) | > loader_time: 0.0044 (0.004499077638920852)  --> TIME: 2025-02-23 23:39:50 -- STEP: 36262/36938 -- GLOBAL_STEP: 73200 | > loss_text_ce: 0.06245098635554314 (0.08896546697402391) | > loss_mel_ce: 4.034478664398193 (nan) | > loss: 0.01625765860080719 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.1353755220996843) | > loader_time: 0.0043 (0.004498973144447935)  --> TIME: 2025-02-23 23:39:59 -- STEP: 36312/36938 -- GLOBAL_STEP: 73250 | > loss_text_ce: 0.07383429259061813 (0.08893759957843846) | > loss_mel_ce: 4.255798816680908 (nan) | > loss: 0.01718108542263508 (nan) | > current_lr: 5e-06 | > step_time: 0.1812 (0.13537431843980924) | > loader_time: 0.0046 (0.004498833127682576)  --> TIME: 2025-02-23 23:40:09 -- STEP: 36362/36938 -- GLOBAL_STEP: 73300 | > loss_text_ce: 0.07006428390741348 (0.08891120297234667) | > loss_mel_ce: 4.161728382110596 (nan) | > loss: 0.01679282821714878 (nan) | > current_lr: 5e-06 | > step_time: 0.1837 (0.13537583711890672) | > loader_time: 0.0047 (0.004498817694999554)  --> TIME: 2025-02-23 23:40:18 -- STEP: 36412/36938 -- GLOBAL_STEP: 73350 | > loss_text_ce: 0.07026288658380508 (0.08888268721284134) | > loss_mel_ce: 4.63905143737793 (nan) | > loss: 0.01868775673210621 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.13537315341858694) | > loader_time: 0.0044 (0.004498740565471697)  --> TIME: 2025-02-23 23:40:27 -- STEP: 36462/36938 -- GLOBAL_STEP: 73400 | > loss_text_ce: 0.05549677833914757 (0.08885352863418282) | > loss_mel_ce: 3.860342502593994 (nan) | > loss: 0.015539045445621014 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.1353725456483963) | > loader_time: 0.0048 (0.004498642925946468)  --> TIME: 2025-02-23 23:40:36 -- STEP: 36512/36938 -- GLOBAL_STEP: 73450 | > loss_text_ce: 0.06146949529647827 (0.08882609924245834) | > loss_mel_ce: 1.9877413511276245 (nan) | > loss: 0.008131789043545723 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13537013387727023) | > loader_time: 0.0045 (0.0044986797230584345)  --> TIME: 2025-02-23 23:40:46 -- STEP: 36562/36938 -- GLOBAL_STEP: 73500 | > loss_text_ce: 0.08582133799791336 (0.088799218258677) | > loss_mel_ce: 0.47459715604782104 (nan) | > loss: 0.002223883057013154 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13537676079673175) | > loader_time: 0.0044 (0.004498696374160243)  --> TIME: 2025-02-23 23:40:55 -- STEP: 36612/36938 -- GLOBAL_STEP: 73550 | > loss_text_ce: 0.06091516837477684 (0.08877082100521907) | > loss_mel_ce: 3.5427803993225098 (nan) | > loss: 0.014300379902124405 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13537458669913952) | > loader_time: 0.0044 (0.004498661345866659)  --> TIME: 2025-02-23 23:41:04 -- STEP: 36662/36938 -- GLOBAL_STEP: 73600 | > loss_text_ce: 0.06884932518005371 (0.0887419602630316) | > loss_mel_ce: 4.094974040985107 (nan) | > loss: 0.016523107886314392 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.1353775179137063) | > loader_time: 0.0047 (0.004498601701137226)  --> TIME: 2025-02-23 23:41:14 -- STEP: 36712/36938 -- GLOBAL_STEP: 73650 | > loss_text_ce: 0.07058649510145187 (0.08871414723232186) | > loss_mel_ce: 4.158151626586914 (nan) | > loss: 0.016780707985162735 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13537535696773598) | > loader_time: 0.0043 (0.004498539166555452)  --> TIME: 2025-02-23 23:41:23 -- STEP: 36762/36938 -- GLOBAL_STEP: 73700 | > loss_text_ce: 0.0664139986038208 (0.08868728121985339) | > loss_mel_ce: 4.230827331542969 (nan) | > loss: 0.0170525461435318 (nan) | > current_lr: 5e-06 | > step_time: 0.1073 (0.1353784399747027) | > loader_time: 0.0043 (0.004498479350867092)  --> TIME: 2025-02-23 23:41:32 -- STEP: 36812/36938 -- GLOBAL_STEP: 73750 | > loss_text_ce: 0.07251806557178497 (0.08866263343003901) | > loss_mel_ce: 3.7549569606781006 (nan) | > loss: 0.015188394114375114 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1353712462995483) | > loader_time: 0.0045 (0.004498435960546029)  --> TIME: 2025-02-23 23:41:41 -- STEP: 36862/36938 -- GLOBAL_STEP: 73800 | > loss_text_ce: 0.05807308852672577 (0.0886360694853934) | > loss_mel_ce: 4.022162437438965 (nan) | > loss: 0.01619141176342964 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13536540698477295) | > loader_time: 0.0044 (0.004498385560343732)  --> TIME: 2025-02-23 23:41:50 -- STEP: 36912/36938 -- GLOBAL_STEP: 73850 | > loss_text_ce: 0.06564133614301682 (0.08860967574705933) | > loss_mel_ce: 4.111513614654541 (nan) | > loss: 0.016576012596488 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13535985964869893) | > loader_time: 0.0047 (0.004498304357557621)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.021504794016923047 (+0.0030485686689320154) | > avg_loss_text_ce: 0.06445326864498087 (-0.051104151210424914) | > avg_loss_mel_ce: 3.45489369583602 (-0.33415386110249123) | > avg_loss: 3.519346958929949 (-0.3852580213310701) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_73876.pth  > EPOCH: 3/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-23 23:42:19)   --> TIME: 2025-02-23 23:42:24 -- STEP: 24/36938 -- GLOBAL_STEP: 73900 | > loss_text_ce: 0.07341787964105606 (0.06866548831264177) | > loss_mel_ce: 1.9308267831802368 (nan) | > loss: 0.00795335229486227 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1375661094983419) | > loader_time: 0.0045 (0.004531532526016236)  --> TIME: 2025-02-23 23:42:33 -- STEP: 74/36938 -- GLOBAL_STEP: 73950 | > loss_text_ce: 0.06216392293572426 (0.07008214415730657) | > loss_mel_ce: 4.2959418296813965 (nan) | > loss: 0.01729407161474228 (nan) | > current_lr: 5e-06 | > step_time: 0.1744 (0.13755950412234746) | > loader_time: 0.0051 (0.00463895217792408)  --> TIME: 2025-02-23 23:42:42 -- STEP: 124/36938 -- GLOBAL_STEP: 74000 | > loss_text_ce: 0.05624083802103996 (0.06917514564890051) | > loss_mel_ce: 3.6694886684417725 (nan) | > loss: 0.014784641563892365 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.1361960557199293) | > loader_time: 0.0045 (0.004657581929237612)  --> TIME: 2025-02-23 23:42:52 -- STEP: 174/36938 -- GLOBAL_STEP: 74050 | > loss_text_ce: 0.07770541310310364 (0.0691307688067699) | > loss_mel_ce: 3.3810155391693115 (nan) | > loss: 0.013725084252655506 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13699146933939266) | > loader_time: 0.0046 (0.004664751304977241)  --> TIME: 2025-02-23 23:43:01 -- STEP: 224/36938 -- GLOBAL_STEP: 74100 | > loss_text_ce: 0.07252131402492523 (0.06906026990951175) | > loss_mel_ce: 4.061644077301025 (nan) | > loss: 0.016405418515205383 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.13678352002586627) | > loader_time: 0.0047 (0.00467701788459505)  --> TIME: 2025-02-23 23:43:11 -- STEP: 274/36938 -- GLOBAL_STEP: 74150 | > loss_text_ce: 0.060289036482572556 (0.0687046859032699) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13670914277543106) | > loader_time: 0.0044 (0.004680536089152313)  --> TIME: 2025-02-23 23:43:20 -- STEP: 324/36938 -- GLOBAL_STEP: 74200 | > loss_text_ce: 0.08545515686273575 (0.06859264062878533) | > loss_mel_ce: 4.6183953285217285 (nan) | > loss: 0.01866607367992401 (nan) | > current_lr: 5e-06 | > step_time: 0.1565 (0.13720261314768845) | > loader_time: 0.0048 (0.004678035959785366)  --> TIME: 2025-02-23 23:43:29 -- STEP: 374/36938 -- GLOBAL_STEP: 74250 | > loss_text_ce: 0.0747331753373146 (0.06849591148130717) | > loss_mel_ce: 4.160239219665527 (nan) | > loss: 0.01680544763803482 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.1367250865793483) | > loader_time: 0.0045 (0.004671385581480628)  --> TIME: 2025-02-23 23:43:38 -- STEP: 424/36938 -- GLOBAL_STEP: 74300 | > loss_text_ce: 0.06133903190493584 (0.06815601886197363) | > loss_mel_ce: 4.466908931732178 (nan) | > loss: 0.01796923764050007 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.13679711267633268) | > loader_time: 0.0046 (0.004665060988012352)  --> TIME: 2025-02-23 23:43:48 -- STEP: 474/36938 -- GLOBAL_STEP: 74350 | > loss_text_ce: 0.07256032526493073 (0.06805924063682056) | > loss_mel_ce: 4.298882484436035 (nan) | > loss: 0.017346996814012527 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13685357268852533) | > loader_time: 0.0046 (0.004662585157885332)  --> TIME: 2025-02-23 23:43:57 -- STEP: 524/36938 -- GLOBAL_STEP: 74400 | > loss_text_ce: 0.060378298163414 (0.06809078425680866) | > loss_mel_ce: 3.240941047668457 (nan) | > loss: 0.013100474141538143 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.1370149136499595) | > loader_time: 0.0052 (0.00466795686547083)  --> TIME: 2025-02-23 23:44:06 -- STEP: 574/36938 -- GLOBAL_STEP: 74450 | > loss_text_ce: 0.07727359235286713 (0.06806312602675336) | > loss_mel_ce: 3.8112285137176514 (nan) | > loss: 0.015430564992129803 (nan) | > current_lr: 5e-06 | > step_time: 0.1262 (0.13699734377113373) | > loader_time: 0.0046 (0.004673963639794328)  --> TIME: 2025-02-23 23:44:15 -- STEP: 624/36938 -- GLOBAL_STEP: 74500 | > loss_text_ce: 0.06393210589885712 (0.06801885089431059) | > loss_mel_ce: 3.625023126602173 (nan) | > loss: 0.014638712629675865 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.1364571173221637) | > loader_time: 0.0045 (0.004673280012913242)  --> TIME: 2025-02-23 23:44:25 -- STEP: 674/36938 -- GLOBAL_STEP: 74550 | > loss_text_ce: 0.0807046890258789 (0.06800806895451134) | > loss_mel_ce: 4.371947288513184 (nan) | > loss: 0.017669254913926125 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13654904407987967) | > loader_time: 0.0046 (0.004671978667511082)  --> TIME: 2025-02-23 23:44:34 -- STEP: 724/36938 -- GLOBAL_STEP: 74600 | > loss_text_ce: 0.07592218369245529 (0.06815117761204591) | > loss_mel_ce: 3.9764668941497803 (nan) | > loss: 0.016080910339951515 (nan) | > current_lr: 5e-06 | > step_time: 0.1855 (0.13697501473663923) | > loader_time: 0.0048 (0.0046731164442241555)  --> TIME: 2025-02-23 23:44:44 -- STEP: 774/36938 -- GLOBAL_STEP: 74650 | > loss_text_ce: 0.06849443912506104 (0.06817190225406988) | > loss_mel_ce: 4.61079216003418 (nan) | > loss: 0.018568597733974457 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13713312487885637) | > loader_time: 0.0048 (0.0046727417051330095)  --> TIME: 2025-02-23 23:44:53 -- STEP: 824/36938 -- GLOBAL_STEP: 74700 | > loss_text_ce: 0.05941193923354149 (0.0680580885800561) | > loss_mel_ce: 3.4427177906036377 (nan) | > loss: 0.013897341676056385 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.1374539544860138) | > loader_time: 0.0046 (0.004675005535477572)  --> TIME: 2025-02-23 23:45:03 -- STEP: 874/36938 -- GLOBAL_STEP: 74750 | > loss_text_ce: 0.0606238879263401 (0.0679007558966189) | > loss_mel_ce: 4.52609920501709 (nan) | > loss: 0.018201284110546112 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.1376142818267616) | > loader_time: 0.0069 (0.0046877601872319765)  --> TIME: 2025-02-23 23:45:12 -- STEP: 924/36938 -- GLOBAL_STEP: 74800 | > loss_text_ce: 0.07547729462385178 (0.06780062948083473) | > loss_mel_ce: 2.601341724395752 (nan) | > loss: 0.01062229834496975 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13775320151151552) | > loader_time: 0.0064 (0.004698382311569148)  --> TIME: 2025-02-23 23:45:22 -- STEP: 974/36938 -- GLOBAL_STEP: 74850 | > loss_text_ce: 0.06408673524856567 (0.06787548489960443) | > loss_mel_ce: 4.082576274871826 (nan) | > loss: 0.016455013304948807 (nan) | > current_lr: 5e-06 | > step_time: 0.1818 (0.1378363375791048) | > loader_time: 0.0052 (0.004708539289125916)  --> TIME: 2025-02-23 23:45:31 -- STEP: 1024/36938 -- GLOBAL_STEP: 74900 | > loss_text_ce: 0.05932723358273506 (0.06777400835926536) | > loss_mel_ce: 3.5092806816101074 (nan) | > loss: 0.014161142520606518 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13771170005202327) | > loader_time: 0.0047 (0.004720860160887241)  --> TIME: 2025-02-23 23:45:40 -- STEP: 1074/36938 -- GLOBAL_STEP: 74950 | > loss_text_ce: 0.06979992985725403 (0.06769875383257765) | > loss_mel_ce: 3.0801632404327393 (nan) | > loss: 0.012499854899942875 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13753990687471557) | > loader_time: 0.0048 (0.004726825035706135)  --> TIME: 2025-02-23 23:45:49 -- STEP: 1124/36938 -- GLOBAL_STEP: 75000 | > loss_text_ce: 0.06652489304542542 (0.0676570542793046) | > loss_mel_ce: 4.2592082023620605 (nan) | > loss: 0.01716560870409012 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.13731723493528575) | > loader_time: 0.0049 (0.004735477666413656)  --> TIME: 2025-02-23 23:45:58 -- STEP: 1174/36938 -- GLOBAL_STEP: 75050 | > loss_text_ce: 0.07587049156427383 (0.06760553348320826) | > loss_mel_ce: 4.246718883514404 (nan) | > loss: 0.01715313270688057 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.1370327600628599) | > loader_time: 0.0046 (0.004736598549344061)  --> TIME: 2025-02-23 23:46:07 -- STEP: 1224/36938 -- GLOBAL_STEP: 75100 | > loss_text_ce: 0.07241110503673553 (0.06767096224673155) | > loss_mel_ce: 3.8847744464874268 (nan) | > loss: 0.015703117474913597 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13685082551700675) | > loader_time: 0.0047 (0.0047397159673030044)  --> TIME: 2025-02-23 23:46:17 -- STEP: 1274/36938 -- GLOBAL_STEP: 75150 | > loss_text_ce: 0.07744055241346359 (0.067699574086131) | > loss_mel_ce: 4.107268810272217 (nan) | > loss: 0.01660599187016487 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13684079205409763) | > loader_time: 0.0046 (0.004741328851768694)  --> TIME: 2025-02-23 23:46:26 -- STEP: 1324/36938 -- GLOBAL_STEP: 75200 | > loss_text_ce: 0.06715898215770721 (0.06766963450750203) | > loss_mel_ce: 4.404775619506836 (nan) | > loss: 0.01774577423930168 (nan) | > current_lr: 5e-06 | > step_time: 0.149 (0.13696244512079764) | > loader_time: 0.0047 (0.00474502708976722)  --> TIME: 2025-02-23 23:46:36 -- STEP: 1374/36938 -- GLOBAL_STEP: 75250 | > loss_text_ce: 0.06373970955610275 (0.0676569452202911) | > loss_mel_ce: 3.819512128829956 (nan) | > loss: 0.01540973037481308 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13706737123375595) | > loader_time: 0.0047 (0.004745620306922875)  --> TIME: 2025-02-23 23:46:45 -- STEP: 1424/36938 -- GLOBAL_STEP: 75300 | > loss_text_ce: 0.05894026160240173 (0.06755944799757413) | > loss_mel_ce: 3.369929075241089 (nan) | > loss: 0.01360662467777729 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13689424516109977) | > loader_time: 0.0049 (0.004749452130178386)  --> TIME: 2025-02-23 23:46:54 -- STEP: 1474/36938 -- GLOBAL_STEP: 75350 | > loss_text_ce: 0.06788193434476852 (0.0675435181798128) | > loss_mel_ce: 3.5104405879974365 (nan) | > loss: 0.014199692755937576 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13692585681154446) | > loader_time: 0.0051 (0.00475163375668157)  --> TIME: 2025-02-23 23:47:03 -- STEP: 1524/36938 -- GLOBAL_STEP: 75400 | > loss_text_ce: 0.0681387335062027 (0.06749588653464797) | > loss_mel_ce: 4.161941051483154 (nan) | > loss: 0.016786031424999237 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.1368998939909651) | > loader_time: 0.0048 (0.0047546263754837175)  --> TIME: 2025-02-23 23:47:12 -- STEP: 1574/36938 -- GLOBAL_STEP: 75450 | > loss_text_ce: 0.08335193246603012 (0.06751056568131432) | > loss_mel_ce: 0.8000078797340393 (nan) | > loss: 0.0035053961910307407 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13673961950227037) | > loader_time: 0.0045 (0.004756146495193812)  --> TIME: 2025-02-23 23:47:22 -- STEP: 1624/36938 -- GLOBAL_STEP: 75500 | > loss_text_ce: 0.06330091506242752 (0.06752588364795727) | > loss_mel_ce: 4.273695945739746 (nan) | > loss: 0.017210306599736214 (nan) | > current_lr: 5e-06 | > step_time: 0.1486 (0.13666877326706992) | > loader_time: 0.0071 (0.004756876253729383)  --> TIME: 2025-02-23 23:47:31 -- STEP: 1674/36938 -- GLOBAL_STEP: 75550 | > loss_text_ce: 0.06770388036966324 (0.06751235908513781) | > loss_mel_ce: 4.0225324630737305 (nan) | > loss: 0.016231097280979156 (nan) | > current_lr: 5e-06 | > step_time: 0.1605 (0.13658638951883853) | > loader_time: 0.0046 (0.004753922903409563)  --> TIME: 2025-02-23 23:47:40 -- STEP: 1724/36938 -- GLOBAL_STEP: 75600 | > loss_text_ce: 0.06719129532575607 (0.0674939899340168) | > loss_mel_ce: 4.194123268127441 (nan) | > loss: 0.016909979283809662 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13664918847537708) | > loader_time: 0.0049 (0.004756423409189808)  --> TIME: 2025-02-23 23:47:50 -- STEP: 1774/36938 -- GLOBAL_STEP: 75650 | > loss_text_ce: 0.06776341050863266 (0.06744159118659798) | > loss_mel_ce: 4.225481986999512 (nan) | > loss: 0.017036689445376396 (nan) | > current_lr: 5e-06 | > step_time: 0.1508 (0.1366633492634295) | > loader_time: 0.0048 (0.004772242053654439)  --> TIME: 2025-02-23 23:47:59 -- STEP: 1824/36938 -- GLOBAL_STEP: 75700 | > loss_text_ce: 0.06174955889582634 (0.06750357571760544) | > loss_mel_ce: 4.448537826538086 (nan) | > loss: 0.017897967249155045 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.1368257402066605) | > loader_time: 0.0048 (0.004772965322461039)  --> TIME: 2025-02-23 23:48:08 -- STEP: 1874/36938 -- GLOBAL_STEP: 75750 | > loss_text_ce: 0.07132405787706375 (0.06750525541905714) | > loss_mel_ce: 2.4573161602020264 (nan) | > loss: 0.010034287348389626 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13679333150450484) | > loader_time: 0.0048 (0.004772840976206186)  --> TIME: 2025-02-23 23:48:18 -- STEP: 1924/36938 -- GLOBAL_STEP: 75800 | > loss_text_ce: 0.0685979425907135 (0.06751847799030937) | > loss_mel_ce: 2.7472496032714844 (nan) | > loss: 0.011173998937010765 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13686397033520953) | > loader_time: 0.0045 (0.004771300759979688)  --> TIME: 2025-02-23 23:48:27 -- STEP: 1974/36938 -- GLOBAL_STEP: 75850 | > loss_text_ce: 0.07811301946640015 (0.06746612264460518) | > loss_mel_ce: 2.490966796875 (nan) | > loss: 0.01019476167857647 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.13695525350058632) | > loader_time: 0.0044 (0.0047702733748106445)  --> TIME: 2025-02-23 23:48:37 -- STEP: 2024/36938 -- GLOBAL_STEP: 75900 | > loss_text_ce: 0.08379606902599335 (0.06744449196214018) | > loss_mel_ce: 4.216836929321289 (nan) | > loss: 0.017066005617380142 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13699436470453974) | > loader_time: 0.0047 (0.004769309236126922)  --> TIME: 2025-02-23 23:48:46 -- STEP: 2074/36938 -- GLOBAL_STEP: 75950 | > loss_text_ce: 0.07253934442996979 (0.06743798057010737) | > loss_mel_ce: 2.1394641399383545 (nan) | > loss: 0.008777791634202003 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13693604772783) | > loader_time: 0.0047 (0.00477065288227342)  --> TIME: 2025-02-23 23:48:55 -- STEP: 2124/36938 -- GLOBAL_STEP: 76000 | > loss_text_ce: 0.08296363800764084 (0.06747628506378259) | > loss_mel_ce: 2.371166944503784 (nan) | > loss: 0.009738613851368427 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13694492336046943) | > loader_time: 0.0046 (0.0047682415967607176)  --> TIME: 2025-02-23 23:49:05 -- STEP: 2174/36938 -- GLOBAL_STEP: 76050 | > loss_text_ce: 0.0586618036031723 (0.06748115138699945) | > loss_mel_ce: 3.734375 (nan) | > loss: 0.015051733702421188 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13690534203888235) | > loader_time: 0.0045 (0.0047705659322378885)  --> TIME: 2025-02-23 23:49:14 -- STEP: 2224/36938 -- GLOBAL_STEP: 76100 | > loss_text_ce: 0.08156047016382217 (0.06746947134195036) | > loss_mel_ce: 3.7176811695098877 (nan) | > loss: 0.015076356008648872 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13691447653787642) | > loader_time: 0.0045 (0.004773167611883693)  --> TIME: 2025-02-23 23:49:23 -- STEP: 2274/36938 -- GLOBAL_STEP: 76150 | > loss_text_ce: 0.0646272748708725 (0.06744748045646938) | > loss_mel_ce: 3.9598917961120605 (nan) | > loss: 0.015970313921570778 (nan) | > current_lr: 5e-06 | > step_time: 0.1886 (0.13691098394368797) | > loader_time: 0.0049 (0.0047782022070234605)  --> TIME: 2025-02-23 23:49:33 -- STEP: 2324/36938 -- GLOBAL_STEP: 76200 | > loss_text_ce: 0.05363821983337402 (0.06746755455734389) | > loss_mel_ce: 4.523636817932129 (nan) | > loss: 0.018163790926337242 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.1370066817779348) | > loader_time: 0.0049 (0.004777814455574018)  --> TIME: 2025-02-23 23:49:42 -- STEP: 2374/36938 -- GLOBAL_STEP: 76250 | > loss_text_ce: 0.0730360671877861 (0.06745599620922973) | > loss_mel_ce: 4.091226100921631 (nan) | > loss: 0.016524851322174072 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13698502538981544) | > loader_time: 0.0046 (0.004775315548254842)  --> TIME: 2025-02-23 23:49:51 -- STEP: 2424/36938 -- GLOBAL_STEP: 76300 | > loss_text_ce: 0.06617417931556702 (0.06740866131195325) | > loss_mel_ce: 0.12075042724609375 (nan) | > loss: 0.0007417643791995943 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.1369338769330447) | > loader_time: 0.0046 (0.004776432077483371)  --> TIME: 2025-02-23 23:50:01 -- STEP: 2474/36938 -- GLOBAL_STEP: 76350 | > loss_text_ce: 0.067610964179039 (0.06739589676754831) | > loss_mel_ce: 3.876415252685547 (nan) | > loss: 0.015650898218154907 (nan) | > current_lr: 5e-06 | > step_time: 0.144 (0.13696117852133177) | > loader_time: 0.0047 (0.004774771106753889)  --> TIME: 2025-02-23 23:50:10 -- STEP: 2524/36938 -- GLOBAL_STEP: 76400 | > loss_text_ce: 0.06588473916053772 (0.067337080613428) | > loss_mel_ce: 4.26151704788208 (nan) | > loss: 0.017172230407595634 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13685407974829622) | > loader_time: 0.0046 (0.004774961059709353)  --> TIME: 2025-02-23 23:50:19 -- STEP: 2574/36938 -- GLOBAL_STEP: 76450 | > loss_text_ce: 0.05993213132023811 (0.06729004111375413) | > loss_mel_ce: 4.268004894256592 (nan) | > loss: 0.017174353823065758 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13684706393377516) | > loader_time: 0.0048 (0.004773579092107264)  --> TIME: 2025-02-23 23:50:28 -- STEP: 2624/36938 -- GLOBAL_STEP: 76500 | > loss_text_ce: 0.07145438343286514 (0.06729845383641772) | > loss_mel_ce: 4.292704105377197 (nan) | > loss: 0.017318090423941612 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.13676202060972803) | > loader_time: 0.0046 (0.004773613974088569)  --> TIME: 2025-02-23 23:50:37 -- STEP: 2674/36938 -- GLOBAL_STEP: 76550 | > loss_text_ce: 0.06360118836164474 (0.06730964075197175) | > loss_mel_ce: 4.342902660369873 (nan) | > loss: 0.017486127093434334 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.1367205425082703) | > loader_time: 0.0051 (0.004772424430390653)  --> TIME: 2025-02-23 23:50:47 -- STEP: 2724/36938 -- GLOBAL_STEP: 76600 | > loss_text_ce: 0.07011542469263077 (0.06734351661854936) | > loss_mel_ce: 3.818183183670044 (nan) | > loss: 0.015429756604135036 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13675017468904815) | > loader_time: 0.0045 (0.004772193631411643)  --> TIME: 2025-02-23 23:50:56 -- STEP: 2774/36938 -- GLOBAL_STEP: 76650 | > loss_text_ce: 0.06801606714725494 (0.06732727455078642) | > loss_mel_ce: 4.034067630767822 (nan) | > loss: 0.016278110444545746 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.13670272352717847) | > loader_time: 0.0049 (0.004770657873531849)  --> TIME: 2025-02-23 23:51:05 -- STEP: 2824/36938 -- GLOBAL_STEP: 76700 | > loss_text_ce: 0.07479196041822433 (0.06732853209974479) | > loss_mel_ce: 3.778825283050537 (nan) | > loss: 0.015292132273316383 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13676787427397055) | > loader_time: 0.0046 (0.00476977904525086)  --> TIME: 2025-02-23 23:51:15 -- STEP: 2874/36938 -- GLOBAL_STEP: 76750 | > loss_text_ce: 0.06733868271112442 (0.06731017931735825) | > loss_mel_ce: 3.6119422912597656 (nan) | > loss: 0.01460032258182764 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13678858250317383) | > loader_time: 0.0049 (0.004768389915540631)  --> TIME: 2025-02-23 23:51:24 -- STEP: 2924/36938 -- GLOBAL_STEP: 76800 | > loss_text_ce: 0.07795624434947968 (0.06727305590780244) | > loss_mel_ce: 4.440849781036377 (nan) | > loss: 0.017931770533323288 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.13683523704513187) | > loader_time: 0.0047 (0.004767110589101763)  --> TIME: 2025-02-23 23:51:34 -- STEP: 2974/36938 -- GLOBAL_STEP: 76850 | > loss_text_ce: 0.07319185882806778 (0.06728823187998365) | > loss_mel_ce: 3.2281126976013184 (nan) | > loss: 0.013100415468215942 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13682879348208493) | > loader_time: 0.0045 (0.004765277110896973)  --> TIME: 2025-02-23 23:51:43 -- STEP: 3024/36938 -- GLOBAL_STEP: 76900 | > loss_text_ce: 0.07396833598613739 (0.067298058521436) | > loss_mel_ce: 4.357409477233887 (nan) | > loss: 0.01758483424782753 (nan) | > current_lr: 5e-06 | > step_time: 0.1262 (0.13675311507371402) | > loader_time: 0.0048 (0.004763423056198795)  --> TIME: 2025-02-23 23:51:52 -- STEP: 3074/36938 -- GLOBAL_STEP: 76950 | > loss_text_ce: 0.07509071379899979 (0.06728220485494966) | > loss_mel_ce: 3.008437395095825 (nan) | > loss: 0.012236223556101322 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13677386153263985) | > loader_time: 0.0046 (0.004761898602924322)  --> TIME: 2025-02-23 23:52:01 -- STEP: 3124/36938 -- GLOBAL_STEP: 77000 | > loss_text_ce: 0.07250530272722244 (0.06726085217061425) | > loss_mel_ce: 3.7970969676971436 (nan) | > loss: 0.015355565585196018 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13674589858012418) | > loader_time: 0.0045 (0.004760707538961279)  --> TIME: 2025-02-23 23:52:11 -- STEP: 3174/36938 -- GLOBAL_STEP: 77050 | > loss_text_ce: 0.06945807486772537 (0.0672473850593711) | > loss_mel_ce: 4.410589694976807 (nan) | > loss: 0.01777796819806099 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13682411305920963) | > loader_time: 0.0045 (0.004759293122342627)  --> TIME: 2025-02-23 23:52:20 -- STEP: 3224/36938 -- GLOBAL_STEP: 77100 | > loss_text_ce: 0.06259450316429138 (0.06722286451150356) | > loss_mel_ce: 3.578847646713257 (nan) | > loss: 0.014450167305767536 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13684828205972427) | > loader_time: 0.0048 (0.004757331337585622)  --> TIME: 2025-02-23 23:52:30 -- STEP: 3274/36938 -- GLOBAL_STEP: 77150 | > loss_text_ce: 0.0679454654455185 (0.06723235222916618) | > loss_mel_ce: 3.93800687789917 (nan) | > loss: 0.015896636992692947 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.1369051272066618) | > loader_time: 0.0045 (0.004755243194780469)  --> TIME: 2025-02-23 23:52:39 -- STEP: 3324/36938 -- GLOBAL_STEP: 77200 | > loss_text_ce: 0.04821887984871864 (0.0671976425199713) | > loss_mel_ce: 4.762760639190674 (nan) | > loss: 0.01909118890762329 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13688941610132063) | > loader_time: 0.0044 (0.004753875603314328)  --> TIME: 2025-02-23 23:52:48 -- STEP: 3374/36938 -- GLOBAL_STEP: 77250 | > loss_text_ce: 0.07586168497800827 (0.06717697833292058) | > loss_mel_ce: 3.998746633529663 (nan) | > loss: 0.01616908237338066 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13689334405267067) | > loader_time: 0.0046 (0.0047512679815150905)  --> TIME: 2025-02-23 23:52:58 -- STEP: 3424/36938 -- GLOBAL_STEP: 77300 | > loss_text_ce: 0.054849497973918915 (0.06712947588492275) | > loss_mel_ce: 4.054009914398193 (nan) | > loss: 0.016304999589920044 (nan) | > current_lr: 5e-06 | > step_time: 0.1599 (0.13692739849614238) | > loader_time: 0.0046 (0.004749087460130162)  --> TIME: 2025-02-23 23:53:07 -- STEP: 3474/36938 -- GLOBAL_STEP: 77350 | > loss_text_ce: 0.06338489055633545 (0.06711395696088632) | > loss_mel_ce: 3.192443370819092 (nan) | > loss: 0.01291995495557785 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.13687200675287964) | > loader_time: 0.0046 (0.004747712001350316)  --> TIME: 2025-02-23 23:53:16 -- STEP: 3524/36938 -- GLOBAL_STEP: 77400 | > loss_text_ce: 0.07912082970142365 (0.0671347043150348) | > loss_mel_ce: 3.6366336345672607 (nan) | > loss: 0.014745058491826057 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1369008126919171) | > loader_time: 0.0045 (0.004745713021779562)  --> TIME: 2025-02-23 23:53:26 -- STEP: 3574/36938 -- GLOBAL_STEP: 77450 | > loss_text_ce: 0.07211467623710632 (0.06714685095206237) | > loss_mel_ce: 3.699284315109253 (nan) | > loss: 0.014965870417654514 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.13695259126258705) | > loader_time: 0.0047 (0.004744215190377247)  --> TIME: 2025-02-23 23:53:35 -- STEP: 3624/36938 -- GLOBAL_STEP: 77500 | > loss_text_ce: 0.05892011150717735 (0.06711785759567124) | > loss_mel_ce: 4.274378299713135 (nan) | > loss: 0.017195628955960274 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13693302488221798) | > loader_time: 0.0045 (0.004743734732368912)  --> TIME: 2025-02-23 23:53:44 -- STEP: 3674/36938 -- GLOBAL_STEP: 77550 | > loss_text_ce: 0.06300540268421173 (0.06709316753714899) | > loss_mel_ce: 3.776679277420044 (nan) | > loss: 0.01523684523999691 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.13687452662555713) | > loader_time: 0.0047 (0.004741346634920344)  --> TIME: 2025-02-23 23:53:53 -- STEP: 3724/36938 -- GLOBAL_STEP: 77600 | > loss_text_ce: 0.06963388621807098 (0.06709237530463392) | > loss_mel_ce: 3.8480987548828125 (nan) | > loss: 0.015546559356153011 (nan) | > current_lr: 5e-06 | > step_time: 0.1922 (0.1368189877265252) | > loader_time: 0.0049 (0.00473973010971773)  --> TIME: 2025-02-23 23:54:03 -- STEP: 3774/36938 -- GLOBAL_STEP: 77650 | > loss_text_ce: 0.05093332752585411 (0.06709019477895214) | > loss_mel_ce: 2.405071496963501 (nan) | > loss: 0.009746051393449306 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13681320911623804) | > loader_time: 0.0045 (0.0047384027962211975)  --> TIME: 2025-02-23 23:54:12 -- STEP: 3824/36938 -- GLOBAL_STEP: 77700 | > loss_text_ce: 0.06504760682582855 (0.06709187885621655) | > loss_mel_ce: 1.693603515625 (nan) | > loss: 0.006978774908930063 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.136833299839846) | > loader_time: 0.0046 (0.004737171356149287)  --> TIME: 2025-02-23 23:54:21 -- STEP: 3874/36938 -- GLOBAL_STEP: 77750 | > loss_text_ce: 0.07283961772918701 (0.06705076153023246) | > loss_mel_ce: 4.268750190734863 (nan) | > loss: 0.01722853258252144 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.1367476998867299) | > loader_time: 0.0051 (0.004736357518059715)  --> TIME: 2025-02-23 23:54:30 -- STEP: 3924/36938 -- GLOBAL_STEP: 77800 | > loss_text_ce: 0.07136281579732895 (0.06704860290647244) | > loss_mel_ce: 3.732646942138672 (nan) | > loss: 0.015095277689397335 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13677140423281864) | > loader_time: 0.0045 (0.004736477568001303)  --> TIME: 2025-02-23 23:54:40 -- STEP: 3974/36938 -- GLOBAL_STEP: 77850 | > loss_text_ce: 0.06424570828676224 (0.06704195458901298) | > loss_mel_ce: 4.287077903747559 (nan) | > loss: 0.01726715825498104 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13679194924335858) | > loader_time: 0.0046 (0.004734967243269442)  --> TIME: 2025-02-23 23:54:49 -- STEP: 4024/36938 -- GLOBAL_STEP: 77900 | > loss_text_ce: 0.07306905090808868 (0.06701500742894331) | > loss_mel_ce: 3.0514776706695557 (nan) | > loss: 0.012398995459079742 (nan) | > current_lr: 5e-06 | > step_time: 0.151 (0.13680522650421043) | > loader_time: 0.0046 (0.004734303735596511)  --> TIME: 2025-02-23 23:54:58 -- STEP: 4074/36938 -- GLOBAL_STEP: 77950 | > loss_text_ce: 0.06229739636182785 (0.06700580350224379) | > loss_mel_ce: 4.660561561584473 (nan) | > loss: 0.018741505220532417 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.13680445205714717) | > loader_time: 0.0046 (0.004733069012079617)  --> TIME: 2025-02-23 23:55:08 -- STEP: 4124/36938 -- GLOBAL_STEP: 78000 | > loss_text_ce: 0.05586531385779381 (0.06697403765658394) | > loss_mel_ce: 4.210859775543213 (nan) | > loss: 0.016931449994444847 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13678517709301027) | > loader_time: 0.0047 (0.004731898337862761)  --> TIME: 2025-02-23 23:55:17 -- STEP: 4174/36938 -- GLOBAL_STEP: 78050 | > loss_text_ce: 0.05626736208796501 (0.06697099492182321) | > loss_mel_ce: 4.423714637756348 (nan) | > loss: 0.017777707427740097 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.13679268055117547) | > loader_time: 0.0047 (0.00473099795607898)  --> TIME: 2025-02-23 23:55:26 -- STEP: 4224/36938 -- GLOBAL_STEP: 78100 | > loss_text_ce: 0.06250172108411789 (0.0669645566050541) | > loss_mel_ce: 3.459425449371338 (nan) | > loss: 0.013975902460515499 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.13672715792376008) | > loader_time: 0.0047 (0.00472918011699661)  --> TIME: 2025-02-23 23:55:36 -- STEP: 4274/36938 -- GLOBAL_STEP: 78150 | > loss_text_ce: 0.06666704267263412 (0.0669461649702665) | > loss_mel_ce: 4.083067417144775 (nan) | > loss: 0.016467200592160225 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.1367632599149179) | > loader_time: 0.0046 (0.00472786033828513)  --> TIME: 2025-02-23 23:55:45 -- STEP: 4324/36938 -- GLOBAL_STEP: 78200 | > loss_text_ce: 0.054828256368637085 (0.06692848921125395) | > loss_mel_ce: 3.4900782108306885 (nan) | > loss: 0.014067089185118675 (nan) | > current_lr: 5e-06 | > step_time: 0.1453 (0.1367860676843079) | > loader_time: 0.0046 (0.00472722386563959)  --> TIME: 2025-02-23 23:55:55 -- STEP: 4374/36938 -- GLOBAL_STEP: 78250 | > loss_text_ce: 0.05598565936088562 (0.0669285191923965) | > loss_mel_ce: 3.231985569000244 (nan) | > loss: 0.013047506101429462 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13687257255437352) | > loader_time: 0.0046 (0.004727468896141091)  --> TIME: 2025-02-23 23:56:04 -- STEP: 4424/36938 -- GLOBAL_STEP: 78300 | > loss_text_ce: 0.04087486118078232 (0.06690377996523751) | > loss_mel_ce: 1.0111205577850342 (nan) | > loss: 0.004174584988504648 (nan) | > current_lr: 5e-06 | > step_time: 0.1086 (0.13687507751621808) | > loader_time: 0.0045 (0.004726153655250493)  --> TIME: 2025-02-23 23:56:14 -- STEP: 4474/36938 -- GLOBAL_STEP: 78350 | > loss_text_ce: 0.06041046977043152 (0.06690520649317062) | > loss_mel_ce: 4.0361328125 (nan) | > loss: 0.016256125643849373 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13690444370119734) | > loader_time: 0.0045 (0.004727714126750605)  --> TIME: 2025-02-23 23:56:23 -- STEP: 4524/36938 -- GLOBAL_STEP: 78400 | > loss_text_ce: 0.07536632567644119 (0.06691628315664691) | > loss_mel_ce: 4.010348320007324 (nan) | > loss: 0.016213154420256615 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13691935709996342) | > loader_time: 0.0045 (0.004726741294962011)  --> TIME: 2025-02-23 23:56:32 -- STEP: 4574/36938 -- GLOBAL_STEP: 78450 | > loss_text_ce: 0.05929629132151604 (0.06688387373443576) | > loss_mel_ce: 2.81085205078125 (nan) | > loss: 0.011389478109776974 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13691602104555664) | > loader_time: 0.0046 (0.004725581649927043)  --> TIME: 2025-02-23 23:56:42 -- STEP: 4624/36938 -- GLOBAL_STEP: 78500 | > loss_text_ce: 0.06048668548464775 (0.06688509752190817) | > loss_mel_ce: 4.102465629577637 (nan) | > loss: 0.01651965267956257 (nan) | > current_lr: 5e-06 | > step_time: 0.1357 (0.13689014289824636) | > loader_time: 0.0046 (0.004724144935607896)  --> TIME: 2025-02-23 23:56:51 -- STEP: 4674/36938 -- GLOBAL_STEP: 78550 | > loss_text_ce: 0.055085744708776474 (0.06687076666914825) | > loss_mel_ce: 2.2364892959594727 (nan) | > loss: 0.009093551896512508 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13691049398496538) | > loader_time: 0.0045 (0.004723117552509546)  --> TIME: 2025-02-23 23:57:00 -- STEP: 4724/36938 -- GLOBAL_STEP: 78600 | > loss_text_ce: 0.06718128174543381 (0.06684042324819087) | > loss_mel_ce: 4.008304595947266 (nan) | > loss: 0.016172563657164574 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.13690833521535323) | > loader_time: 0.0046 (0.0047226709193036506)  --> TIME: 2025-02-23 23:57:09 -- STEP: 4774/36938 -- GLOBAL_STEP: 78650 | > loss_text_ce: 0.053075168281793594 (0.06681907818948037) | > loss_mel_ce: 4.643914699554443 (nan) | > loss: 0.018638851121068 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13685204965777636) | > loader_time: 0.0044 (0.004722007658374706)  --> TIME: 2025-02-23 23:57:19 -- STEP: 4824/36938 -- GLOBAL_STEP: 78700 | > loss_text_ce: 0.07524238526821136 (0.06682227845388651) | > loss_mel_ce: 4.664614677429199 (nan) | > loss: 0.018808959051966667 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13686638699835238) | > loader_time: 0.0045 (0.004720706696534022)  --> TIME: 2025-02-23 23:57:28 -- STEP: 4874/36938 -- GLOBAL_STEP: 78750 | > loss_text_ce: 0.06441784650087357 (0.06681967116617178) | > loss_mel_ce: 3.5853841304779053 (nan) | > loss: 0.014483341947197914 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1368723751630853) | > loader_time: 0.0044 (0.004719510448238213)  --> TIME: 2025-02-23 23:57:37 -- STEP: 4924/36938 -- GLOBAL_STEP: 78800 | > loss_text_ce: 0.0628264769911766 (0.06681558333872947) | > loss_mel_ce: 4.161018371582031 (nan) | > loss: 0.016761289909482002 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.1368065465765212) | > loader_time: 0.0045 (0.004719068033728129)  --> TIME: 2025-02-23 23:57:46 -- STEP: 4974/36938 -- GLOBAL_STEP: 78850 | > loss_text_ce: 0.07152801007032394 (0.06679169893576332) | > loss_mel_ce: 4.033353328704834 (nan) | > loss: 0.016289211809635162 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13678307422059705) | > loader_time: 0.0045 (0.004717421378291734)  --> TIME: 2025-02-23 23:57:55 -- STEP: 5024/36938 -- GLOBAL_STEP: 78900 | > loss_text_ce: 0.07506797462701797 (0.06677458651073175) | > loss_mel_ce: 0.18829345703125 (nan) | > loss: 0.0010450850240886211 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13677997730530012) | > loader_time: 0.0048 (0.0047169896257910635)  --> TIME: 2025-02-23 23:58:05 -- STEP: 5074/36938 -- GLOBAL_STEP: 78950 | > loss_text_ce: 0.06051425635814667 (0.06676322145693678) | > loss_mel_ce: 3.95939302444458 (nan) | > loss: 0.015952015295624733 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13675155398128655) | > loader_time: 0.0046 (0.004715266827292725)  --> TIME: 2025-02-23 23:58:14 -- STEP: 5124/36938 -- GLOBAL_STEP: 79000 | > loss_text_ce: 0.06365251541137695 (0.06672547361977063) | > loss_mel_ce: 4.020683288574219 (nan) | > loss: 0.016207681968808174 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.13675995323455148) | > loader_time: 0.0048 (0.004714211619151549)  --> TIME: 2025-02-23 23:58:23 -- STEP: 5174/36938 -- GLOBAL_STEP: 79050 | > loss_text_ce: 0.0662178322672844 (0.06670234639199024) | > loss_mel_ce: 4.0566229820251465 (nan) | > loss: 0.016360480338335037 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13675640691050178) | > loader_time: 0.0045 (0.004715628452544319)  --> TIME: 2025-02-23 23:58:33 -- STEP: 5224/36938 -- GLOBAL_STEP: 79100 | > loss_text_ce: 0.049784086644649506 (0.06667277116576634) | > loss_mel_ce: 0.13623046875 (nan) | > loss: 0.0007381530595012009 (nan) | > current_lr: 5e-06 | > step_time: 0.1084 (0.13675906555941086) | > loader_time: 0.0045 (0.004715130486130523)  --> TIME: 2025-02-23 23:58:42 -- STEP: 5274/36938 -- GLOBAL_STEP: 79150 | > loss_text_ce: 0.06884611397981644 (0.06664965324602509) | > loss_mel_ce: 4.38219690322876 (nan) | > loss: 0.01766286976635456 (nan) | > current_lr: 5e-06 | > step_time: 0.181 (0.1367301222588198) | > loader_time: 0.0051 (0.004713934571805387)  --> TIME: 2025-02-23 23:58:51 -- STEP: 5324/36938 -- GLOBAL_STEP: 79200 | > loss_text_ce: 0.06954994052648544 (0.06663404598133812) | > loss_mel_ce: 3.112868309020996 (nan) | > loss: 0.012628644704818726 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.13677977369209604) | > loader_time: 0.0046 (0.004713906236916526)  --> TIME: 2025-02-23 23:59:01 -- STEP: 5374/36938 -- GLOBAL_STEP: 79250 | > loss_text_ce: 0.04979146271944046 (0.06663642917313635) | > loss_mel_ce: 0.0677490234375 (nan) | > loss: 0.0004664305306505412 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13675735766088223) | > loader_time: 0.0045 (0.004713627056069292)  --> TIME: 2025-02-23 23:59:10 -- STEP: 5424/36938 -- GLOBAL_STEP: 79300 | > loss_text_ce: 0.05442187562584877 (0.06663592197625977) | > loss_mel_ce: 3.7219340801239014 (nan) | > loss: 0.014985540881752968 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13673632202950223) | > loader_time: 0.0045 (0.0047134353963323005)  --> TIME: 2025-02-23 23:59:19 -- STEP: 5474/36938 -- GLOBAL_STEP: 79350 | > loss_text_ce: 0.07594697922468185 (0.06662273721291026) | > loss_mel_ce: 3.9166250228881836 (nan) | > loss: 0.01584354043006897 (nan) | > current_lr: 5e-06 | > step_time: 0.1764 (0.136723079601256) | > loader_time: 0.005 (0.004712565170834431)  --> TIME: 2025-02-23 23:59:28 -- STEP: 5524/36938 -- GLOBAL_STEP: 79400 | > loss_text_ce: 0.05346275120973587 (0.06661463415339199) | > loss_mel_ce: 4.034271240234375 (nan) | > loss: 0.016221169382333755 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13673188486106178) | > loader_time: 0.0045 (0.004712159006946072)  --> TIME: 2025-02-23 23:59:37 -- STEP: 5574/36938 -- GLOBAL_STEP: 79450 | > loss_text_ce: 0.07304150611162186 (0.06660152088441713) | > loss_mel_ce: 2.8263604640960693 (nan) | > loss: 0.011505563743412495 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1366930445078074) | > loader_time: 0.0046 (0.004710961123884983)  --> TIME: 2025-02-23 23:59:46 -- STEP: 5624/36938 -- GLOBAL_STEP: 79500 | > loss_text_ce: 0.06181066483259201 (0.06658097026719956) | > loss_mel_ce: 4.228831768035889 (nan) | > loss: 0.01702635921537876 (nan) | > current_lr: 5e-06 | > step_time: 0.1867 (0.13667808059777875) | > loader_time: 0.0048 (0.004710041993373145)  --> TIME: 2025-02-23 23:59:56 -- STEP: 5674/36938 -- GLOBAL_STEP: 79550 | > loss_text_ce: 0.06762544810771942 (0.06656850543750313) | > loss_mel_ce: 3.4299638271331787 (nan) | > loss: 0.0138793233782053 (nan) | > current_lr: 5e-06 | > step_time: 0.1505 (0.136661489916638) | > loader_time: 0.0046 (0.004708866355366208)  --> TIME: 2025-02-24 00:00:05 -- STEP: 5724/36938 -- GLOBAL_STEP: 79600 | > loss_text_ce: 0.054204899817705154 (0.06654259066600535) | > loss_mel_ce: 3.4739203453063965 (nan) | > loss: 0.01400049775838852 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13663376031598407) | > loader_time: 0.0045 (0.004707468089151006)  --> TIME: 2025-02-24 00:00:14 -- STEP: 5774/36938 -- GLOBAL_STEP: 79650 | > loss_text_ce: 0.048156287521123886 (0.06652762628104625) | > loss_mel_ce: 2.281176805496216 (nan) | > loss: 0.00924338586628437 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13661547817075664) | > loader_time: 0.0045 (0.004706811235980828)  --> TIME: 2025-02-24 00:00:23 -- STEP: 5824/36938 -- GLOBAL_STEP: 79700 | > loss_text_ce: 0.06812292337417603 (0.06650817409895919) | > loss_mel_ce: 4.120316028594971 (nan) | > loss: 0.01662079058587551 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13661630513085146) | > loader_time: 0.0046 (0.0047059036627575546)  --> TIME: 2025-02-24 00:00:32 -- STEP: 5874/36938 -- GLOBAL_STEP: 79750 | > loss_text_ce: 0.06949327141046524 (0.06649139934436017) | > loss_mel_ce: 3.94891357421875 (nan) | > loss: 0.01594606041908264 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13660177983188881) | > loader_time: 0.0045 (0.004704629356129698)  --> TIME: 2025-02-24 00:00:42 -- STEP: 5924/36938 -- GLOBAL_STEP: 79800 | > loss_text_ce: 0.07024525105953217 (0.06648939885897111) | > loss_mel_ce: 4.524497985839844 (nan) | > loss: 0.018233109265565872 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13659498608651663) | > loader_time: 0.0047 (0.004703411413639632)  --> TIME: 2025-02-24 00:00:51 -- STEP: 5974/36938 -- GLOBAL_STEP: 79850 | > loss_text_ce: 0.061120957136154175 (0.06646719526070502) | > loss_mel_ce: 1.9946695566177368 (nan) | > loss: 0.008157898671925068 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13661328605636572) | > loader_time: 0.0045 (0.00470257208504562)  --> TIME: 2025-02-24 00:01:00 -- STEP: 6024/36938 -- GLOBAL_STEP: 79900 | > loss_text_ce: 0.05872485041618347 (0.06646324750957443) | > loss_mel_ce: 4.448721885681152 (nan) | > loss: 0.0178866945207119 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13658738029430625) | > loader_time: 0.0045 (0.004702157629121969)  --> TIME: 2025-02-24 00:01:09 -- STEP: 6074/36938 -- GLOBAL_STEP: 79950 | > loss_text_ce: 0.06760542094707489 (0.06644475668549778) | > loss_mel_ce: 3.4811582565307617 (nan) | > loss: 0.014082396402955055 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.1365731073752576) | > loader_time: 0.0051 (0.004702847177432002)  --> TIME: 2025-02-24 00:01:19 -- STEP: 6124/36938 -- GLOBAL_STEP: 80000 | > loss_text_ce: 0.06626442819833755 (0.06642504060363554) | > loss_mel_ce: 3.2397425174713135 (nan) | > loss: 0.013119075447320938 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13657203127869177) | > loader_time: 0.0047 (0.004702349374062237) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_80000.pth  --> TIME: 2025-02-24 00:01:34 -- STEP: 6174/36938 -- GLOBAL_STEP: 80050 | > loss_text_ce: 0.060504548251628876 (0.06641010138570339) | > loss_mel_ce: 4.3872222900390625 (nan) | > loss: 0.017649710178375244 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13655167816138702) | > loader_time: 0.0045 (0.004701549272379663)  --> TIME: 2025-02-24 00:01:43 -- STEP: 6224/36938 -- GLOBAL_STEP: 80100 | > loss_text_ce: 0.06616021692752838 (0.06639229964164445) | > loss_mel_ce: 4.042819023132324 (nan) | > loss: 0.016305474564433098 (nan) | > current_lr: 5e-06 | > step_time: 0.1582 (0.13652882822681592) | > loader_time: 0.0047 (0.004701079508639228)  --> TIME: 2025-02-24 00:01:52 -- STEP: 6274/36938 -- GLOBAL_STEP: 80150 | > loss_text_ce: 0.057431530207395554 (0.06637847566799615) | > loss_mel_ce: 4.274888038635254 (nan) | > loss: 0.017191745340824127 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13650485714568045) | > loader_time: 0.0053 (0.00470154814405504)  --> TIME: 2025-02-24 00:02:01 -- STEP: 6324/36938 -- GLOBAL_STEP: 80200 | > loss_text_ce: 0.06761480122804642 (0.06637364972393388) | > loss_mel_ce: 3.6971781253814697 (nan) | > loss: 0.014939655549824238 (nan) | > current_lr: 5e-06 | > step_time: 0.1515 (0.13648556475395116) | > loader_time: 0.0047 (0.004702084544035234)  --> TIME: 2025-02-24 00:02:10 -- STEP: 6374/36938 -- GLOBAL_STEP: 80250 | > loss_text_ce: 0.06045011803507805 (0.06635440515449548) | > loss_mel_ce: 3.536794424057007 (nan) | > loss: 0.01427478063851595 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13646387144955993) | > loader_time: 0.0047 (0.004701806566054664)  --> TIME: 2025-02-24 00:02:20 -- STEP: 6424/36938 -- GLOBAL_STEP: 80300 | > loss_text_ce: 0.06688671559095383 (0.06634877303906331) | > loss_mel_ce: 4.051513671875 (nan) | > loss: 0.016342859715223312 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13645358771494032) | > loader_time: 0.0046 (0.004701067286440317)  --> TIME: 2025-02-24 00:02:29 -- STEP: 6474/36938 -- GLOBAL_STEP: 80350 | > loss_text_ce: 0.07548096030950546 (0.06633487915672572) | > loss_mel_ce: 4.235124111175537 (nan) | > loss: 0.017105575650930405 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13649113431623597) | > loader_time: 0.0067 (0.004701131724044416)  --> TIME: 2025-02-24 00:02:39 -- STEP: 6524/36938 -- GLOBAL_STEP: 80400 | > loss_text_ce: 0.059289079159498215 (0.06633183737739394) | > loss_mel_ce: 3.7451040744781494 (nan) | > loss: 0.015096798539161682 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13650355980046947) | > loader_time: 0.0048 (0.004701330170610936)  --> TIME: 2025-02-24 00:02:48 -- STEP: 6574/36938 -- GLOBAL_STEP: 80450 | > loss_text_ce: 0.03898555412888527 (0.06631090975181522) | > loss_mel_ce: 4.488715171813965 (nan) | > loss: 0.017967067658901215 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.13648386026248452) | > loader_time: 0.0044 (0.0047010276540275136)  --> TIME: 2025-02-24 00:02:57 -- STEP: 6624/36938 -- GLOBAL_STEP: 80500 | > loss_text_ce: 0.06545194983482361 (0.06629038826086685) | > loss_mel_ce: 4.200930118560791 (nan) | > loss: 0.01693008840084076 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.13647980898042864) | > loader_time: 0.0048 (0.004701036833910545)  --> TIME: 2025-02-24 00:03:06 -- STEP: 6674/36938 -- GLOBAL_STEP: 80550 | > loss_text_ce: 0.05578838661313057 (0.06627476274501679) | > loss_mel_ce: 3.948606491088867 (nan) | > loss: 0.0158904567360878 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13642662308121115) | > loader_time: 0.0045 (0.004700003679119471)  --> TIME: 2025-02-24 00:03:15 -- STEP: 6724/36938 -- GLOBAL_STEP: 80600 | > loss_text_ce: 0.05751128867268562 (0.06624938708695145) | > loss_mel_ce: 3.6555018424987793 (nan) | > loss: 0.014734179712831974 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13640160879019447) | > loader_time: 0.0046 (0.004699709158720957)  --> TIME: 2025-02-24 00:03:24 -- STEP: 6774/36938 -- GLOBAL_STEP: 80650 | > loss_text_ce: 0.07759135961532593 (0.06623891048361051) | > loss_mel_ce: 2.8542277812957764 (nan) | > loss: 0.011634203605353832 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13636987389524305) | > loader_time: 0.0045 (0.004698919095505921)  --> TIME: 2025-02-24 00:03:33 -- STEP: 6824/36938 -- GLOBAL_STEP: 80700 | > loss_text_ce: 0.06269937753677368 (0.06623066335961283) | > loss_mel_ce: 3.0007946491241455 (nan) | > loss: 0.012156723067164421 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1363874903038272) | > loader_time: 0.0047 (0.004698679078226207)  --> TIME: 2025-02-24 00:03:43 -- STEP: 6874/36938 -- GLOBAL_STEP: 80750 | > loss_text_ce: 0.07059722393751144 (0.06621581083692006) | > loss_mel_ce: 3.7503435611724854 (nan) | > loss: 0.01516246423125267 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1363913466287352) | > loader_time: 0.0045 (0.004698322198749588)  --> TIME: 2025-02-24 00:03:52 -- STEP: 6924/36938 -- GLOBAL_STEP: 80800 | > loss_text_ce: 0.05960937216877937 (0.06620648656936778) | > loss_mel_ce: 3.4142885208129883 (nan) | > loss: 0.013785310089588165 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13637021271773456) | > loader_time: 0.0045 (0.004697856704738353)  --> TIME: 2025-02-24 00:04:01 -- STEP: 6974/36938 -- GLOBAL_STEP: 80850 | > loss_text_ce: 0.06619524955749512 (0.06619609623975407) | > loss_mel_ce: 2.8089261054992676 (nan) | > loss: 0.011409212835133076 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13639010759489895) | > loader_time: 0.0045 (0.004697496069855351)  --> TIME: 2025-02-24 00:04:10 -- STEP: 7024/36938 -- GLOBAL_STEP: 80900 | > loss_text_ce: 0.06818084418773651 (0.06619309469901556) | > loss_mel_ce: 3.490018129348755 (nan) | > loss: 0.014119837433099747 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13639347866211562) | > loader_time: 0.0045 (0.004696935686815295)  --> TIME: 2025-02-24 00:04:20 -- STEP: 7074/36938 -- GLOBAL_STEP: 80950 | > loss_text_ce: 0.07767605781555176 (0.06617472542632565) | > loss_mel_ce: 2.3066277503967285 (nan) | > loss: 0.009461523965001106 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.1364440326472281) | > loader_time: 0.0046 (0.00469628956346886)  --> TIME: 2025-02-24 00:04:29 -- STEP: 7124/36938 -- GLOBAL_STEP: 81000 | > loss_text_ce: 0.0664614886045456 (0.06615942068986808) | > loss_mel_ce: 4.288305759429932 (nan) | > loss: 0.01728082448244095 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13642537861577245) | > loader_time: 0.0045 (0.004695187553403614)  --> TIME: 2025-02-24 00:04:38 -- STEP: 7174/36938 -- GLOBAL_STEP: 81050 | > loss_text_ce: 0.052244629710912704 (0.06615814601533593) | > loss_mel_ce: 4.685869216918945 (nan) | > loss: 0.018802041187882423 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.1363966018003051) | > loader_time: 0.0044 (0.00469440043986192)  --> TIME: 2025-02-24 00:04:48 -- STEP: 7224/36938 -- GLOBAL_STEP: 81100 | > loss_text_ce: 0.05717870220541954 (0.06614688737601541) | > loss_mel_ce: 4.485713005065918 (nan) | > loss: 0.018027348443865776 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13639321890252304) | > loader_time: 0.0045 (0.00469353389106319)  --> TIME: 2025-02-24 00:04:57 -- STEP: 7274/36938 -- GLOBAL_STEP: 81150 | > loss_text_ce: 0.07532288879156113 (0.06613089620543641) | > loss_mel_ce: 3.930720567703247 (nan) | > loss: 0.015896998345851898 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13640842524274993) | > loader_time: 0.0047 (0.004692960447553157)  --> TIME: 2025-02-24 00:05:06 -- STEP: 7324/36938 -- GLOBAL_STEP: 81200 | > loss_text_ce: 0.06731556355953217 (0.06610991794169395) | > loss_mel_ce: 3.5712060928344727 (nan) | > loss: 0.014438578858971596 (nan) | > current_lr: 5e-06 | > step_time: 0.1878 (0.13641443641898074) | > loader_time: 0.0048 (0.004692652295808979)  --> TIME: 2025-02-24 00:05:16 -- STEP: 7374/36938 -- GLOBAL_STEP: 81250 | > loss_text_ce: 0.06600885093212128 (0.06609978782028718) | > loss_mel_ce: 3.5380570888519287 (nan) | > loss: 0.014301849529147148 (nan) | > current_lr: 5e-06 | > step_time: 0.1694 (0.13643315441730694) | > loader_time: 0.0049 (0.004692713969245907)  --> TIME: 2025-02-24 00:05:25 -- STEP: 7424/36938 -- GLOBAL_STEP: 81300 | > loss_text_ce: 0.05457961559295654 (0.06608984596271605) | > loss_mel_ce: 3.7046945095062256 (nan) | > loss: 0.01491775456815958 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13645291668844695) | > loader_time: 0.0046 (0.00469222680890354)  --> TIME: 2025-02-24 00:05:35 -- STEP: 7474/36938 -- GLOBAL_STEP: 81350 | > loss_text_ce: 0.05872328206896782 (0.06606184073856187) | > loss_mel_ce: 4.305945873260498 (nan) | > loss: 0.017320116981863976 (nan) | > current_lr: 5e-06 | > step_time: 0.1871 (0.13645858505622707) | > loader_time: 0.0048 (0.00469245137395085)  --> TIME: 2025-02-24 00:05:44 -- STEP: 7524/36938 -- GLOBAL_STEP: 81400 | > loss_text_ce: 0.057816632091999054 (0.06603737536185023) | > loss_mel_ce: 0.14361572265625 (nan) | > loss: 0.000799334782641381 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13646693675838786) | > loader_time: 0.0052 (0.004692792005199242)  --> TIME: 2025-02-24 00:05:53 -- STEP: 7574/36938 -- GLOBAL_STEP: 81450 | > loss_text_ce: 0.06409404426813126 (0.06602591027020978) | > loss_mel_ce: 3.8724000453948975 (nan) | > loss: 0.015621009282767773 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.1364732216114044) | > loader_time: 0.0046 (0.0046925438341252)  --> TIME: 2025-02-24 00:06:03 -- STEP: 7624/36938 -- GLOBAL_STEP: 81500 | > loss_text_ce: 0.06809594482183456 (0.06601411217636977) | > loss_mel_ce: 3.936147689819336 (nan) | > loss: 0.015889856964349747 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13648401324296242) | > loader_time: 0.0045 (0.004692364871064107)  --> TIME: 2025-02-24 00:06:12 -- STEP: 7674/36938 -- GLOBAL_STEP: 81550 | > loss_text_ce: 0.06535205990076065 (0.06599414997955255) | > loss_mel_ce: 4.668749809265137 (nan) | > loss: 0.018786119297146797 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13650967933748306) | > loader_time: 0.0048 (0.004691703853303945)  --> TIME: 2025-02-24 00:06:22 -- STEP: 7724/36938 -- GLOBAL_STEP: 81600 | > loss_text_ce: 0.05597592890262604 (0.06597889500010837) | > loss_mel_ce: 4.218916416168213 (nan) | > loss: 0.016963860020041466 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13653298004003284) | > loader_time: 0.0047 (0.004691356824877836)  --> TIME: 2025-02-24 00:06:31 -- STEP: 7774/36938 -- GLOBAL_STEP: 81650 | > loss_text_ce: 0.0523449070751667 (0.06596994380544995) | > loss_mel_ce: 4.289813041687012 (nan) | > loss: 0.017230786383152008 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13654250447842778) | > loader_time: 0.0046 (0.0046910271719423895)  --> TIME: 2025-02-24 00:06:40 -- STEP: 7824/36938 -- GLOBAL_STEP: 81700 | > loss_text_ce: 0.051863543689250946 (0.06595074040562887) | > loss_mel_ce: 3.983161449432373 (nan) | > loss: 0.016012005507946014 (nan) | > current_lr: 5e-06 | > step_time: 0.1436 (0.13652309994756145) | > loader_time: 0.0046 (0.004690245464291792)  --> TIME: 2025-02-24 00:06:50 -- STEP: 7874/36938 -- GLOBAL_STEP: 81750 | > loss_text_ce: 0.07225833088159561 (0.06593408230701617) | > loss_mel_ce: 3.9839766025543213 (nan) | > loss: 0.016096170991659164 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13650313839562453) | > loader_time: 0.0045 (0.004689758127341872)  --> TIME: 2025-02-24 00:06:59 -- STEP: 7924/36938 -- GLOBAL_STEP: 81800 | > loss_text_ce: 0.08578513562679291 (0.06591814941271078) | > loss_mel_ce: 3.3522326946258545 (nan) | > loss: 0.013642928563058376 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1365380776103015) | > loader_time: 0.0045 (0.00468928629794787)  --> TIME: 2025-02-24 00:07:08 -- STEP: 7974/36938 -- GLOBAL_STEP: 81850 | > loss_text_ce: 0.05227971076965332 (0.06589086560809494) | > loss_mel_ce: 3.4011027812957764 (nan) | > loss: 0.013703899458050728 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13651566147236235) | > loader_time: 0.0046 (0.0046888922042406505)  --> TIME: 2025-02-24 00:07:18 -- STEP: 8024/36938 -- GLOBAL_STEP: 81900 | > loss_text_ce: 0.07727605104446411 (0.06588362052257653) | > loss_mel_ce: 3.965674877166748 (nan) | > loss: 0.01604345813393593 (nan) | > current_lr: 5e-06 | > step_time: 0.1835 (0.1365149385135178) | > loader_time: 0.0048 (0.004688793557472262)  --> TIME: 2025-02-24 00:07:27 -- STEP: 8074/36938 -- GLOBAL_STEP: 81950 | > loss_text_ce: 0.058125484734773636 (0.06586795488424588) | > loss_mel_ce: 4.021816730499268 (nan) | > loss: 0.016190247610211372 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.13649927940822285) | > loader_time: 0.0046 (0.004688205475673713)  --> TIME: 2025-02-24 00:07:36 -- STEP: 8124/36938 -- GLOBAL_STEP: 82000 | > loss_text_ce: 0.07874451577663422 (0.06585545873210749) | > loss_mel_ce: 3.552457809448242 (nan) | > loss: 0.014409533701837063 (nan) | > current_lr: 5e-06 | > step_time: 0.1613 (0.13650262179719885) | > loader_time: 0.0046 (0.004688306491049794)  --> TIME: 2025-02-24 00:07:46 -- STEP: 8174/36938 -- GLOBAL_STEP: 82050 | > loss_text_ce: 0.06964360922574997 (0.0658448975652198) | > loss_mel_ce: 3.377277374267578 (nan) | > loss: 0.013678258284926414 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13654766810211663) | > loader_time: 0.0048 (0.004688334605032866)  --> TIME: 2025-02-24 00:07:55 -- STEP: 8224/36938 -- GLOBAL_STEP: 82100 | > loss_text_ce: 0.059435028582811356 (0.06583492005500288) | > loss_mel_ce: 3.4864985942840576 (nan) | > loss: 0.014071166515350342 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13653475492612924) | > loader_time: 0.0044 (0.004688221193009308)  --> TIME: 2025-02-24 00:08:04 -- STEP: 8274/36938 -- GLOBAL_STEP: 82150 | > loss_text_ce: 0.06334332376718521 (0.06582472752554606) | > loss_mel_ce: 0.0262451171875 (nan) | > loss: 0.00035550972097553313 (nan) | > current_lr: 5e-06 | > step_time: 0.1491 (0.13652152657249575) | > loader_time: 0.0046 (0.004687347128565882)  --> TIME: 2025-02-24 00:08:13 -- STEP: 8324/36938 -- GLOBAL_STEP: 82200 | > loss_text_ce: 0.066334567964077 (0.06581358786189914) | > loss_mel_ce: 3.663468360900879 (nan) | > loss: 0.01480080559849739 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13649019631899595) | > loader_time: 0.0077 (0.004686858091487265)  --> TIME: 2025-02-24 00:08:22 -- STEP: 8374/36938 -- GLOBAL_STEP: 82250 | > loss_text_ce: 0.0750788077712059 (0.06579313034194843) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13644671881429563) | > loader_time: 0.0045 (0.004686946170780458)  --> TIME: 2025-02-24 00:08:31 -- STEP: 8424/36938 -- GLOBAL_STEP: 82300 | > loss_text_ce: 0.05266031250357628 (0.0657779280247091) | > loss_mel_ce: 3.6683237552642822 (nan) | > loss: 0.01476581022143364 (nan) | > current_lr: 5e-06 | > step_time: 0.107 (0.13643162452031238) | > loader_time: 0.0044 (0.004686211050971844)  --> TIME: 2025-02-24 00:08:41 -- STEP: 8474/36938 -- GLOBAL_STEP: 82350 | > loss_text_ce: 0.06857758015394211 (0.0657644606755239) | > loss_mel_ce: 4.262150764465332 (nan) | > loss: 0.017185430973768234 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.136432689260925) | > loader_time: 0.0045 (0.0046855077615233255)  --> TIME: 2025-02-24 00:08:50 -- STEP: 8524/36938 -- GLOBAL_STEP: 82400 | > loss_text_ce: 0.06003554165363312 (0.0657582181026868) | > loss_mel_ce: 3.679759979248047 (nan) | > loss: 0.014840458519756794 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13642201069541457) | > loader_time: 0.0046 (0.004685398671952617)  --> TIME: 2025-02-24 00:08:59 -- STEP: 8574/36938 -- GLOBAL_STEP: 82450 | > loss_text_ce: 0.05157216265797615 (0.0657458426539443) | > loss_mel_ce: 4.2442708015441895 (nan) | > loss: 0.01704699732363224 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13641702558767999) | > loader_time: 0.0045 (0.004685364293418119)  --> TIME: 2025-02-24 00:09:08 -- STEP: 8624/36938 -- GLOBAL_STEP: 82500 | > loss_text_ce: 0.06961608678102493 (0.06572942312565802) | > loss_mel_ce: 3.2247965335845947 (nan) | > loss: 0.013073066249489784 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1363980091956202) | > loader_time: 0.0048 (0.004684985983349616)  --> TIME: 2025-02-24 00:09:18 -- STEP: 8674/36938 -- GLOBAL_STEP: 82550 | > loss_text_ce: 0.06465818732976913 (0.06572042573767337) | > loss_mel_ce: 3.6286606788635254 (nan) | > loss: 0.014656027778983116 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.1364153704116611) | > loader_time: 0.0046 (0.0046849942630629695)  --> TIME: 2025-02-24 00:09:27 -- STEP: 8724/36938 -- GLOBAL_STEP: 82600 | > loss_text_ce: 0.06699224561452866 (0.06570542855179708) | > loss_mel_ce: 3.397122859954834 (nan) | > loss: 0.013746489770710468 (nan) | > current_lr: 5e-06 | > step_time: 0.1634 (0.13641044020598156) | > loader_time: 0.005 (0.004684984383370992)  --> TIME: 2025-02-24 00:09:36 -- STEP: 8774/36938 -- GLOBAL_STEP: 82650 | > loss_text_ce: 0.05690556764602661 (0.06568372892740204) | > loss_mel_ce: 0.2702789306640625 (nan) | > loss: 0.001298351213335991 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.1364057215683443) | > loader_time: 0.0045 (0.004684631200050852)  --> TIME: 2025-02-24 00:09:45 -- STEP: 8824/36938 -- GLOBAL_STEP: 82700 | > loss_text_ce: 0.0635254830121994 (0.06566746004879072) | > loss_mel_ce: 3.508087158203125 (nan) | > loss: 0.01417306624352932 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13637380578900538) | > loader_time: 0.0045 (0.004684352215484172)  --> TIME: 2025-02-24 00:09:54 -- STEP: 8874/36938 -- GLOBAL_STEP: 82750 | > loss_text_ce: 0.05267523229122162 (0.06565683816637478) | > loss_mel_ce: 4.240677356719971 (nan) | > loss: 0.01703711412847042 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13635885680557433) | > loader_time: 0.0046 (0.0046838493209608425)  --> TIME: 2025-02-24 00:10:03 -- STEP: 8924/36938 -- GLOBAL_STEP: 82800 | > loss_text_ce: 0.06724219024181366 (0.06564463948094582) | > loss_mel_ce: 5.3203125 (nan) | > loss: 0.02137918584048748 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1363501855740365) | > loader_time: 0.0045 (0.00468354687889506)  --> TIME: 2025-02-24 00:10:13 -- STEP: 8974/36938 -- GLOBAL_STEP: 82850 | > loss_text_ce: 0.056826990097761154 (0.06562929421332016) | > loss_mel_ce: 3.421502113342285 (nan) | > loss: 0.01380289439111948 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13635604201432275) | > loader_time: 0.0044 (0.004682967995970293)  --> TIME: 2025-02-24 00:10:22 -- STEP: 9024/36938 -- GLOBAL_STEP: 82900 | > loss_text_ce: 0.06524879485368729 (0.06561610251413288) | > loss_mel_ce: 4.117503643035889 (nan) | > loss: 0.016598226502537727 (nan) | > current_lr: 5e-06 | > step_time: 0.1777 (0.13636660853281965) | > loader_time: 0.0055 (0.004682598067513596)  --> TIME: 2025-02-24 00:10:31 -- STEP: 9074/36938 -- GLOBAL_STEP: 82950 | > loss_text_ce: 0.057578496634960175 (0.06560566854708963) | > loss_mel_ce: 2.474578857421875 (nan) | > loss: 0.010048244148492813 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13634793249184507) | > loader_time: 0.0048 (0.004682160590434354)  --> TIME: 2025-02-24 00:10:40 -- STEP: 9124/36938 -- GLOBAL_STEP: 83000 | > loss_text_ce: 0.05841623619198799 (0.06557714489726724) | > loss_mel_ce: 2.994706630706787 (nan) | > loss: 0.012115566991269588 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13632846066743778) | > loader_time: 0.0046 (0.004681732768502376)  --> TIME: 2025-02-24 00:10:50 -- STEP: 9174/36938 -- GLOBAL_STEP: 83050 | > loss_text_ce: 0.06099836528301239 (0.06556728218320898) | > loss_mel_ce: 3.900698184967041 (nan) | > loss: 0.015721019357442856 (nan) | > current_lr: 5e-06 | > step_time: 0.186 (0.13634745352715763) | > loader_time: 0.0049 (0.004681388329180961)  --> TIME: 2025-02-24 00:10:59 -- STEP: 9224/36938 -- GLOBAL_STEP: 83100 | > loss_text_ce: 0.061860911548137665 (0.06554295534973971) | > loss_mel_ce: 4.051451206207275 (nan) | > loss: 0.01632266864180565 (nan) | > current_lr: 5e-06 | > step_time: 0.1924 (0.13638076625589923) | > loader_time: 0.0048 (0.004681072411905241)  --> TIME: 2025-02-24 00:11:09 -- STEP: 9274/36938 -- GLOBAL_STEP: 83150 | > loss_text_ce: 0.06436998397111893 (0.06553649518446826) | > loss_mel_ce: 4.160844326019287 (nan) | > loss: 0.016766725108027458 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13639546566696473) | > loader_time: 0.0046 (0.0046810329487438395)  --> TIME: 2025-02-24 00:11:18 -- STEP: 9324/36938 -- GLOBAL_STEP: 83200 | > loss_text_ce: 0.043631475418806076 (0.06552385270846448) | > loss_mel_ce: 3.065340518951416 (nan) | > loss: 0.01233719103038311 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13640035758556524) | > loader_time: 0.0045 (0.004680695655467629)  --> TIME: 2025-02-24 00:11:27 -- STEP: 9374/36938 -- GLOBAL_STEP: 83250 | > loss_text_ce: 0.06013099476695061 (0.06551820838695543) | > loss_mel_ce: 4.060191631317139 (nan) | > loss: 0.016350487247109413 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13638685442426235) | > loader_time: 0.0046 (0.004680268795525362)  --> TIME: 2025-02-24 00:11:37 -- STEP: 9424/36938 -- GLOBAL_STEP: 83300 | > loss_text_ce: 0.05550489202141762 (0.06550699773051641) | > loss_mel_ce: 3.3868308067321777 (nan) | > loss: 0.013660063035786152 (nan) | > current_lr: 5e-06 | > step_time: 0.1856 (0.13639694333380112) | > loader_time: 0.0048 (0.0046802870739498055)  --> TIME: 2025-02-24 00:11:46 -- STEP: 9474/36938 -- GLOBAL_STEP: 83350 | > loss_text_ce: 0.06128791347146034 (0.06549242259932567) | > loss_mel_ce: 9.0546875 (nan) | > loss: 0.036174505949020386 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13638273681143812) | > loader_time: 0.0045 (0.004680084532918323)  --> TIME: 2025-02-24 00:11:55 -- STEP: 9524/36938 -- GLOBAL_STEP: 83400 | > loss_text_ce: 0.06959746032953262 (0.06548165287493911) | > loss_mel_ce: 1.2235229015350342 (nan) | > loss: 0.005131430458277464 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13636222578106433) | > loader_time: 0.0054 (0.004680136605903214)  --> TIME: 2025-02-24 00:12:04 -- STEP: 9574/36938 -- GLOBAL_STEP: 83450 | > loss_text_ce: 0.06562483310699463 (0.06546707470448572) | > loss_mel_ce: 3.4027445316314697 (nan) | > loss: 0.013763371855020523 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.1363691707630613) | > loader_time: 0.0045 (0.004679979773984217)  --> TIME: 2025-02-24 00:12:14 -- STEP: 9624/36938 -- GLOBAL_STEP: 83500 | > loss_text_ce: 0.06816212087869644 (0.06545742413343376) | > loss_mel_ce: 3.18845534324646 (nan) | > loss: 0.012923086062073708 (nan) | > current_lr: 5e-06 | > step_time: 0.1069 (0.1363771313790574) | > loader_time: 0.0044 (0.004679692480233146)  --> TIME: 2025-02-24 00:12:23 -- STEP: 9674/36938 -- GLOBAL_STEP: 83550 | > loss_text_ce: 0.06347046047449112 (0.06543947661637094) | > loss_mel_ce: 4.990908622741699 (nan) | > loss: 0.020057059824466705 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.1363858790443981) | > loader_time: 0.0045 (0.0046794630413043575)  --> TIME: 2025-02-24 00:12:33 -- STEP: 9724/36938 -- GLOBAL_STEP: 83600 | > loss_text_ce: 0.07914762943983078 (0.0654271158915746) | > loss_mel_ce: 4.7940850257873535 (nan) | > loss: 0.019338225945830345 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13639263656099201) | > loader_time: 0.0045 (0.004679228998613372)  --> TIME: 2025-02-24 00:12:42 -- STEP: 9774/36938 -- GLOBAL_STEP: 83650 | > loss_text_ce: 0.07196691632270813 (0.06541188983717378) | > loss_mel_ce: 2.1253020763397217 (nan) | > loss: 0.008719321340322495 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13638929372334385) | > loader_time: 0.0045 (0.004679047210050159)  --> TIME: 2025-02-24 00:12:51 -- STEP: 9824/36938 -- GLOBAL_STEP: 83700 | > loss_text_ce: 0.06582958251237869 (0.06540499755305736) | > loss_mel_ce: 3.3664422035217285 (nan) | > loss: 0.013620126992464066 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13635410732477465) | > loader_time: 0.0045 (0.004678629241665331)  --> TIME: 2025-02-24 00:13:00 -- STEP: 9874/36938 -- GLOBAL_STEP: 83750 | > loss_text_ce: 0.061877377331256866 (0.06539157593118365) | > loss_mel_ce: 0.0812581405043602 (nan) | > loss: 0.0005679980968125165 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13637001509122715) | > loader_time: 0.0045 (0.0046784979432175355)  --> TIME: 2025-02-24 00:13:10 -- STEP: 9924/36938 -- GLOBAL_STEP: 83800 | > loss_text_ce: 0.06266546994447708 (0.06537558544496214) | > loss_mel_ce: 2.1372859477996826 (nan) | > loss: 0.00872996635735035 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13637911603125039) | > loader_time: 0.0045 (0.004678532439150764)  --> TIME: 2025-02-24 00:13:19 -- STEP: 9974/36938 -- GLOBAL_STEP: 83850 | > loss_text_ce: 0.07685817778110504 (0.06535600963513076) | > loss_mel_ce: 3.4489035606384277 (nan) | > loss: 0.013991119340062141 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13637577987802482) | > loader_time: 0.0048 (0.0046788278600364995)  --> TIME: 2025-02-24 00:13:28 -- STEP: 10024/36938 -- GLOBAL_STEP: 83900 | > loss_text_ce: 0.06618530303239822 (0.06534094859029065) | > loss_mel_ce: 4.245354652404785 (nan) | > loss: 0.017109287902712822 (nan) | > current_lr: 5e-06 | > step_time: 0.1824 (0.1363856825510789) | > loader_time: 0.0047 (0.0046790445554951)  --> TIME: 2025-02-24 00:13:37 -- STEP: 10074/36938 -- GLOBAL_STEP: 83950 | > loss_text_ce: 0.05578533187508583 (0.06533719323418698) | > loss_mel_ce: 3.8263978958129883 (nan) | > loss: 0.015405490063130856 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13637925361105452) | > loader_time: 0.0044 (0.004678420161879061)  --> TIME: 2025-02-24 00:13:47 -- STEP: 10124/36938 -- GLOBAL_STEP: 84000 | > loss_text_ce: 0.06656580418348312 (0.06533090540008106) | > loss_mel_ce: 4.352819442749023 (nan) | > loss: 0.017537245526909828 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13639219306006084) | > loader_time: 0.0048 (0.004678784788290726)  --> TIME: 2025-02-24 00:13:56 -- STEP: 10174/36938 -- GLOBAL_STEP: 84050 | > loss_text_ce: 0.06225767359137535 (0.06531735816034925) | > loss_mel_ce: 3.9472217559814453 (nan) | > loss: 0.015910634770989418 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.13639111035648635) | > loader_time: 0.0045 (0.004678301148501786)  --> TIME: 2025-02-24 00:14:06 -- STEP: 10224/36938 -- GLOBAL_STEP: 84100 | > loss_text_ce: 0.04537464678287506 (0.06530938717771269) | > loss_mel_ce: 3.828125 (nan) | > loss: 0.015371031127870083 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13640953623726154) | > loader_time: 0.0044 (0.004678087871585838)  --> TIME: 2025-02-24 00:14:15 -- STEP: 10274/36938 -- GLOBAL_STEP: 84150 | > loss_text_ce: 0.05955174192786217 (0.06529517649533864) | > loss_mel_ce: 3.8484020233154297 (nan) | > loss: 0.015507753938436508 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.1364213828797329) | > loader_time: 0.0047 (0.0046778820079426955)  --> TIME: 2025-02-24 00:14:24 -- STEP: 10324/36938 -- GLOBAL_STEP: 84200 | > loss_text_ce: 0.06686828285455704 (0.06527599887630116) | > loss_mel_ce: 3.8003177642822266 (nan) | > loss: 0.015345977619290352 (nan) | > current_lr: 5e-06 | > step_time: 0.1465 (0.1364102819916447) | > loader_time: 0.0045 (0.00467745902602038)  --> TIME: 2025-02-24 00:14:34 -- STEP: 10374/36938 -- GLOBAL_STEP: 84250 | > loss_text_ce: 0.04432747885584831 (0.06526715287456435) | > loss_mel_ce: 1.7165859937667847 (nan) | > loss: 0.006987752392888069 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13640560496168294) | > loader_time: 0.0052 (0.004677171097662215)  --> TIME: 2025-02-24 00:14:43 -- STEP: 10424/36938 -- GLOBAL_STEP: 84300 | > loss_text_ce: 0.06729808449745178 (0.06525893161803796) | > loss_mel_ce: 4.4453935623168945 (nan) | > loss: 0.017907507717609406 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13641960878877216) | > loader_time: 0.0048 (0.004676997318509344)  --> TIME: 2025-02-24 00:14:52 -- STEP: 10474/36938 -- GLOBAL_STEP: 84350 | > loss_text_ce: 0.0699312686920166 (0.06523838665614495) | > loss_mel_ce: 2.711388349533081 (nan) | > loss: 0.01103698369115591 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.1364259653608604) | > loader_time: 0.0045 (0.004676940720204703)  --> TIME: 2025-02-24 00:15:02 -- STEP: 10524/36938 -- GLOBAL_STEP: 84400 | > loss_text_ce: 0.06968221813440323 (0.06522914587372443) | > loss_mel_ce: 4.304917812347412 (nan) | > loss: 0.017359524965286255 (nan) | > current_lr: 5e-06 | > step_time: 0.1864 (0.1364473221375436) | > loader_time: 0.0048 (0.004676957086935011)  --> TIME: 2025-02-24 00:15:11 -- STEP: 10574/36938 -- GLOBAL_STEP: 84450 | > loss_text_ce: 0.05314517393708229 (0.06522143775350943) | > loss_mel_ce: 3.6669695377349854 (nan) | > loss: 0.014762360602617264 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13642980417908557) | > loader_time: 0.0045 (0.004676722682038708)  --> TIME: 2025-02-24 00:15:21 -- STEP: 10624/36938 -- GLOBAL_STEP: 84500 | > loss_text_ce: 0.053067486733198166 (0.06520493086448481) | > loss_mel_ce: 3.8456807136535645 (nan) | > loss: 0.01547122374176979 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.13643772778920407) | > loader_time: 0.0045 (0.004676201055387415)  --> TIME: 2025-02-24 00:15:30 -- STEP: 10674/36938 -- GLOBAL_STEP: 84550 | > loss_text_ce: 0.06688857823610306 (0.06518816775562923) | > loss_mel_ce: 3.8324294090270996 (nan) | > loss: 0.015473484992980957 (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.13642463311957873) | > loader_time: 0.0047 (0.0046755989236243745)  --> TIME: 2025-02-24 00:15:39 -- STEP: 10724/36938 -- GLOBAL_STEP: 84600 | > loss_text_ce: 0.05905235931277275 (0.06518240546663821) | > loss_mel_ce: 2.833524465560913 (nan) | > loss: 0.011478480882942677 (nan) | > current_lr: 5e-06 | > step_time: 0.1642 (0.13641785981716506) | > loader_time: 0.0046 (0.0046749730378733935)  --> TIME: 2025-02-24 00:15:48 -- STEP: 10774/36938 -- GLOBAL_STEP: 84650 | > loss_text_ce: 0.06637508422136307 (0.06517107807603247) | > loss_mel_ce: 4.421982765197754 (nan) | > loss: 0.017810946330428123 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13641252561073192) | > loader_time: 0.0045 (0.004674425721721742)  --> TIME: 2025-02-24 00:15:57 -- STEP: 10824/36938 -- GLOBAL_STEP: 84700 | > loss_text_ce: 0.06827342510223389 (0.06515512188190291) | > loss_mel_ce: 3.255784273147583 (nan) | > loss: 0.013190705329179764 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13639957021098642) | > loader_time: 0.0045 (0.004673810927672293)  --> TIME: 2025-02-24 00:16:06 -- STEP: 10874/36938 -- GLOBAL_STEP: 84750 | > loss_text_ce: 0.05681245028972626 (0.06514069439453764) | > loss_mel_ce: 4.174112319946289 (nan) | > loss: 0.016789384186267853 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13640591487389894) | > loader_time: 0.0047 (0.00467338140163874)  --> TIME: 2025-02-24 00:16:16 -- STEP: 10924/36938 -- GLOBAL_STEP: 84800 | > loss_text_ce: 0.06708209216594696 (0.0651359283902413) | > loss_mel_ce: 3.977996349334717 (nan) | > loss: 0.016051899641752243 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13643011059005053) | > loader_time: 0.0045 (0.004673298157776111)  --> TIME: 2025-02-24 00:16:25 -- STEP: 10974/36938 -- GLOBAL_STEP: 84850 | > loss_text_ce: 0.04732932895421982 (0.06512705137176114) | > loss_mel_ce: 3.5890300273895264 (nan) | > loss: 0.014429998584091663 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.13641065503591984) | > loader_time: 0.0044 (0.004672937778169082)  --> TIME: 2025-02-24 00:16:34 -- STEP: 11024/36938 -- GLOBAL_STEP: 84900 | > loss_text_ce: 0.07395424693822861 (0.06512230166527387) | > loss_mel_ce: 4.477571964263916 (nan) | > loss: 0.01806161180138588 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.1364154749278237) | > loader_time: 0.0045 (0.004672636119824529)  --> TIME: 2025-02-24 00:16:44 -- STEP: 11074/36938 -- GLOBAL_STEP: 84950 | > loss_text_ce: 0.07003401964902878 (0.0651096102951643) | > loss_mel_ce: 2.5949738025665283 (nan) | > loss: 0.010575428605079651 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.1364160070829238) | > loader_time: 0.0044 (0.004672378242414127)  --> TIME: 2025-02-24 00:16:53 -- STEP: 11124/36938 -- GLOBAL_STEP: 85000 | > loss_text_ce: 0.06013597175478935 (0.06510233131084275) | > loss_mel_ce: 0.10933685302734375 (nan) | > loss: 0.0006725112907588482 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13643938673781422) | > loader_time: 0.0052 (0.00467203103651549)  --> TIME: 2025-02-24 00:17:03 -- STEP: 11174/36938 -- GLOBAL_STEP: 85050 | > loss_text_ce: 0.07283234596252441 (0.06508358778221632) | > loss_mel_ce: 3.6229753494262695 (nan) | > loss: 0.014665904454886913 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13645499032534222) | > loader_time: 0.0047 (0.004671933400475189)  --> TIME: 2025-02-24 00:17:12 -- STEP: 11224/36938 -- GLOBAL_STEP: 85100 | > loss_text_ce: 0.07622988522052765 (0.06508323401000271) | > loss_mel_ce: 0.20459747314453125 (nan) | > loss: 0.0011143942829221487 (nan) | > current_lr: 5e-06 | > step_time: 0.1486 (0.1364556541420784) | > loader_time: 0.0046 (0.004671973304415462)  --> TIME: 2025-02-24 00:17:21 -- STEP: 11274/36938 -- GLOBAL_STEP: 85150 | > loss_text_ce: 0.04519088193774223 (0.06506956416498283) | > loss_mel_ce: 2.3416991233825684 (nan) | > loss: 0.009471786208450794 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13644419597018762) | > loader_time: 0.0051 (0.004671893750857816)  --> TIME: 2025-02-24 00:17:30 -- STEP: 11324/36938 -- GLOBAL_STEP: 85200 | > loss_text_ce: 0.05712353065609932 (0.06505865214641908) | > loss_mel_ce: 3.897650718688965 (nan) | > loss: 0.015693549066781998 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13642905727792115) | > loader_time: 0.0047 (0.004672110417587349)  --> TIME: 2025-02-24 00:17:40 -- STEP: 11374/36938 -- GLOBAL_STEP: 85250 | > loss_text_ce: 0.0608627013862133 (0.06503507548706368) | > loss_mel_ce: 3.8282265663146973 (nan) | > loss: 0.01543289516121149 (nan) | > current_lr: 5e-06 | > step_time: 0.1873 (0.13642823092439055) | > loader_time: 0.005 (0.004672201547174941)  --> TIME: 2025-02-24 00:17:49 -- STEP: 11424/36938 -- GLOBAL_STEP: 85300 | > loss_text_ce: 0.05715527758002281 (0.06502660386315201) | > loss_mel_ce: 3.245445966720581 (nan) | > loss: 0.013105561956763268 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.13644754329398956) | > loader_time: 0.0048 (0.00467226305762593)  --> TIME: 2025-02-24 00:17:58 -- STEP: 11474/36938 -- GLOBAL_STEP: 85350 | > loss_text_ce: 0.06697598099708557 (0.06502032717529169) | > loss_mel_ce: 1.1725608110427856 (nan) | > loss: 0.004918796941637993 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1364232955835876) | > loader_time: 0.0045 (0.004672030091846744)  --> TIME: 2025-02-24 00:18:08 -- STEP: 11524/36938 -- GLOBAL_STEP: 85400 | > loss_text_ce: 0.0774959847331047 (0.06500973453611429) | > loss_mel_ce: 3.870028495788574 (nan) | > loss: 0.015664780512452126 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13642969916819425) | > loader_time: 0.0046 (0.00467196457587113)  --> TIME: 2025-02-24 00:18:17 -- STEP: 11574/36938 -- GLOBAL_STEP: 85450 | > loss_text_ce: 0.08709507435560226 (0.06500192440876307) | > loss_mel_ce: 1.3184469938278198 (nan) | > loss: 0.005577547941356897 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13642714073852913) | > loader_time: 0.0047 (0.004671776873554119)  --> TIME: 2025-02-24 00:18:26 -- STEP: 11624/36938 -- GLOBAL_STEP: 85500 | > loss_text_ce: 0.060251202434301376 (0.06499223356633227) | > loss_mel_ce: 1.5898823738098145 (nan) | > loss: 0.006548149511218071 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13641691074565013) | > loader_time: 0.0045 (0.004671391830391826)  --> TIME: 2025-02-24 00:18:35 -- STEP: 11674/36938 -- GLOBAL_STEP: 85550 | > loss_text_ce: 0.06786272674798965 (0.06498220154857003) | > loss_mel_ce: 4.659604072570801 (nan) | > loss: 0.01875978894531727 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.1363995985480766) | > loader_time: 0.0067 (0.004671110076734377)  --> TIME: 2025-02-24 00:18:45 -- STEP: 11724/36938 -- GLOBAL_STEP: 85600 | > loss_text_ce: 0.06180094927549362 (0.06495701887078287) | > loss_mel_ce: 3.755382537841797 (nan) | > loss: 0.015147554688155651 (nan) | > current_lr: 5e-06 | > step_time: 0.1583 (0.13641267146563177) | > loader_time: 0.0047 (0.004671008665786995)  --> TIME: 2025-02-24 00:18:54 -- STEP: 11774/36938 -- GLOBAL_STEP: 85650 | > loss_text_ce: 0.05947720259428024 (0.06494867882506392) | > loss_mel_ce: 4.221001625061035 (nan) | > loss: 0.016986029222607613 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13641812332785797) | > loader_time: 0.0048 (0.004671287066830786)  --> TIME: 2025-02-24 00:19:03 -- STEP: 11824/36938 -- GLOBAL_STEP: 85700 | > loss_text_ce: 0.065781369805336 (0.06493898673412797) | > loss_mel_ce: 3.870563507080078 (nan) | > loss: 0.015620416961610317 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.1364050953975394) | > loader_time: 0.0045 (0.004670997191830479)  --> TIME: 2025-02-24 00:19:12 -- STEP: 11874/36938 -- GLOBAL_STEP: 85750 | > loss_text_ce: 0.055163539946079254 (0.06493623486406454) | > loss_mel_ce: 1.8548583984375 (nan) | > loss: 0.007579452358186245 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.1364104036572452) | > loader_time: 0.0045 (0.004670670965373643)  --> TIME: 2025-02-24 00:19:22 -- STEP: 11924/36938 -- GLOBAL_STEP: 85800 | > loss_text_ce: 0.06421592831611633 (0.06492658169563341) | > loss_mel_ce: 4.561259746551514 (nan) | > loss: 0.01835506409406662 (nan) | > current_lr: 5e-06 | > step_time: 0.1821 (0.13642913493003245) | > loader_time: 0.0047 (0.004670677809697674)  --> TIME: 2025-02-24 00:19:31 -- STEP: 11974/36938 -- GLOBAL_STEP: 85850 | > loss_text_ce: 0.05153360590338707 (0.06492148078225164) | > loss_mel_ce: 3.3694984912872314 (nan) | > loss: 0.013575525023043156 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13641101630637664) | > loader_time: 0.0051 (0.0046711705534211645)  --> TIME: 2025-02-24 00:19:40 -- STEP: 12024/36938 -- GLOBAL_STEP: 85900 | > loss_text_ce: 0.06857916712760925 (0.06490632271764682) | > loss_mel_ce: 3.4300851821899414 (nan) | > loss: 0.013883589766919613 (nan) | > current_lr: 5e-06 | > step_time: 0.1829 (0.13640238081941003) | > loader_time: 0.0048 (0.00467098538509148)  --> TIME: 2025-02-24 00:19:49 -- STEP: 12074/36938 -- GLOBAL_STEP: 85950 | > loss_text_ce: 0.048938553780317307 (0.06489816265011321) | > loss_mel_ce: 3.376410484313965 (nan) | > loss: 0.013592655770480633 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13639700112604333) | > loader_time: 0.0044 (0.004670749185336865)  --> TIME: 2025-02-24 00:19:59 -- STEP: 12124/36938 -- GLOBAL_STEP: 86000 | > loss_text_ce: 0.07131516933441162 (0.06489189857041198) | > loss_mel_ce: 4.7014360427856445 (nan) | > loss: 0.018939491361379623 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13640154902591334) | > loader_time: 0.0045 (0.00467041892920418)  --> TIME: 2025-02-24 00:20:08 -- STEP: 12174/36938 -- GLOBAL_STEP: 86050 | > loss_text_ce: 0.058596763759851456 (0.06488645141104302) | > loss_mel_ce: 4.313154220581055 (nan) | > loss: 0.017348218709230423 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13640244551160932) | > loader_time: 0.0045 (0.004670887465780149)  --> TIME: 2025-02-24 00:20:17 -- STEP: 12224/36938 -- GLOBAL_STEP: 86100 | > loss_text_ce: 0.06102773919701576 (0.06488137948418186) | > loss_mel_ce: 4.174602508544922 (nan) | > loss: 0.016808057203888893 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13637966863538012) | > loader_time: 0.0047 (0.004671022530001492)  --> TIME: 2025-02-24 00:20:26 -- STEP: 12274/36938 -- GLOBAL_STEP: 86150 | > loss_text_ce: 0.058318886905908585 (0.06486799931725767) | > loss_mel_ce: 3.4566261768341064 (nan) | > loss: 0.013948195613920689 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.1363645236655719) | > loader_time: 0.0045 (0.004670723109674975)  --> TIME: 2025-02-24 00:20:35 -- STEP: 12324/36938 -- GLOBAL_STEP: 86200 | > loss_text_ce: 0.052568137645721436 (0.06485728941391096) | > loss_mel_ce: 3.468013048171997 (nan) | > loss: 0.01397056132555008 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.1363532807253591) | > loader_time: 0.0051 (0.004671090030701119)  --> TIME: 2025-02-24 00:20:44 -- STEP: 12374/36938 -- GLOBAL_STEP: 86250 | > loss_text_ce: 0.072100430727005 (0.064850012372432) | > loss_mel_ce: 2.7022006511688232 (nan) | > loss: 0.01100913155823946 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13635184465442) | > loader_time: 0.0045 (0.004671087514724102)  --> TIME: 2025-02-24 00:20:53 -- STEP: 12424/36938 -- GLOBAL_STEP: 86300 | > loss_text_ce: 0.05700577050447464 (0.06483944352291629) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13633641383760056) | > loader_time: 0.0045 (0.00467060288535497)  --> TIME: 2025-02-24 00:21:03 -- STEP: 12474/36938 -- GLOBAL_STEP: 86350 | > loss_text_ce: 0.05498987436294556 (0.06483466334190628) | > loss_mel_ce: 3.88116455078125 (nan) | > loss: 0.01561966072767973 (nan) | > current_lr: 5e-06 | > step_time: 0.1556 (0.1363564877244786) | > loader_time: 0.0047 (0.0046705221048012275)  --> TIME: 2025-02-24 00:21:12 -- STEP: 12524/36938 -- GLOBAL_STEP: 86400 | > loss_text_ce: 0.06685440242290497 (0.064821294238193) | > loss_mel_ce: 3.873342514038086 (nan) | > loss: 0.01563570275902748 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.1363503302449588) | > loader_time: 0.0047 (0.004670401706135107)  --> TIME: 2025-02-24 00:21:21 -- STEP: 12574/36938 -- GLOBAL_STEP: 86450 | > loss_text_ce: 0.0637248158454895 (0.06481279889087577) | > loss_mel_ce: 0.016082763671875 (nan) | > loss: 0.0003166967653669417 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13633345641789568) | > loader_time: 0.0045 (0.004670019064075487)  --> TIME: 2025-02-24 00:21:30 -- STEP: 12624/36938 -- GLOBAL_STEP: 86500 | > loss_text_ce: 0.06589416414499283 (0.06480251234502815) | > loss_mel_ce: 3.516298532485962 (nan) | > loss: 0.01421505119651556 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.13632843874782127) | > loader_time: 0.0054 (0.004670171872140187)  --> TIME: 2025-02-24 00:21:39 -- STEP: 12674/36938 -- GLOBAL_STEP: 86550 | > loss_text_ce: 0.06689262390136719 (0.06479088387764949) | > loss_mel_ce: 2.908080577850342 (nan) | > loss: 0.011805449612438679 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13629835869435686) | > loader_time: 0.0048 (0.004670199599950046)  --> TIME: 2025-02-24 00:21:49 -- STEP: 12724/36938 -- GLOBAL_STEP: 86600 | > loss_text_ce: 0.06594552099704742 (0.06478679722250531) | > loss_mel_ce: 3.361111640930176 (nan) | > loss: 0.013599433936178684 (nan) | > current_lr: 5e-06 | > step_time: 0.1514 (0.13632990162689704) | > loader_time: 0.0046 (0.004670090586912028)  --> TIME: 2025-02-24 00:21:58 -- STEP: 12774/36938 -- GLOBAL_STEP: 86650 | > loss_text_ce: 0.07131677120923996 (0.06477840321818866) | > loss_mel_ce: 2.542487621307373 (nan) | > loss: 0.010372240096330643 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13634517772555774) | > loader_time: 0.0046 (0.004669799143222457)  --> TIME: 2025-02-24 00:22:07 -- STEP: 12824/36938 -- GLOBAL_STEP: 86700 | > loss_text_ce: 0.052496373653411865 (0.0647622444990822) | > loss_mel_ce: 4.555887222290039 (nan) | > loss: 0.018287237733602524 (nan) | > current_lr: 5e-06 | > step_time: 0.1783 (0.13633752871288551) | > loader_time: 0.0049 (0.004669621782011338)  --> TIME: 2025-02-24 00:22:17 -- STEP: 12874/36938 -- GLOBAL_STEP: 86750 | > loss_text_ce: 0.06087114289402962 (0.06475524499816208) | > loss_mel_ce: 4.019434928894043 (nan) | > loss: 0.016191691160202026 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13636722921380728) | > loader_time: 0.0046 (0.004670006639564445)  --> TIME: 2025-02-24 00:22:26 -- STEP: 12924/36938 -- GLOBAL_STEP: 86800 | > loss_text_ce: 0.06429094076156616 (0.06475568082571583) | > loss_mel_ce: 4.202944755554199 (nan) | > loss: 0.01693347655236721 (nan) | > current_lr: 5e-06 | > step_time: 0.1463 (0.136358153465842) | > loader_time: 0.0048 (0.004670161150882424)  --> TIME: 2025-02-24 00:22:36 -- STEP: 12974/36938 -- GLOBAL_STEP: 86850 | > loss_text_ce: 0.06293310225009918 (0.06474426447904716) | > loss_mel_ce: 3.141982316970825 (nan) | > loss: 0.012717919424176216 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.1363602394827609) | > loader_time: 0.0046 (0.004669996261302649)  --> TIME: 2025-02-24 00:22:45 -- STEP: 13024/36938 -- GLOBAL_STEP: 86900 | > loss_text_ce: 0.062422703951597214 (0.06473309423958112) | > loss_mel_ce: 3.645061731338501 (nan) | > loss: 0.014712240546941757 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.1363648742766879) | > loader_time: 0.0048 (0.0046697398258661454)  --> TIME: 2025-02-24 00:22:54 -- STEP: 13074/36938 -- GLOBAL_STEP: 86950 | > loss_text_ce: 0.05504751577973366 (0.06472380415128863) | > loss_mel_ce: 4.016939163208008 (nan) | > loss: 0.016158677637577057 (nan) | > current_lr: 5e-06 | > step_time: 0.1626 (0.13635734851998285) | > loader_time: 0.0046 (0.004669698166960292)  --> TIME: 2025-02-24 00:23:04 -- STEP: 13124/36938 -- GLOBAL_STEP: 87000 | > loss_text_ce: 0.06547633558511734 (0.06471403953439064) | > loss_mel_ce: 3.826219081878662 (nan) | > loss: 0.015443236567080021 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.13636946836211183) | > loader_time: 0.0049 (0.004669825193613847)  --> TIME: 2025-02-24 00:23:13 -- STEP: 13174/36938 -- GLOBAL_STEP: 87050 | > loss_text_ce: 0.06601661443710327 (0.0647059351405749) | > loss_mel_ce: 0.1454249769449234 (nan) | > loss: 0.0008390540024265647 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.13634850730189524) | > loader_time: 0.0046 (0.004669580217809982)  --> TIME: 2025-02-24 00:23:22 -- STEP: 13224/36938 -- GLOBAL_STEP: 87100 | > loss_text_ce: 0.061121292412281036 (0.06469476170862104) | > loss_mel_ce: 4.61181640625 (nan) | > loss: 0.018543405458331108 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13635344962892057) | > loader_time: 0.0045 (0.004669314125272192)  --> TIME: 2025-02-24 00:23:31 -- STEP: 13274/36938 -- GLOBAL_STEP: 87150 | > loss_text_ce: 0.05634467303752899 (0.06469138104415821) | > loss_mel_ce: 1.5902293920516968 (nan) | > loss: 0.006534024141728878 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1363508208286198) | > loader_time: 0.0045 (0.004669253467274158)  --> TIME: 2025-02-24 00:23:41 -- STEP: 13324/36938 -- GLOBAL_STEP: 87200 | > loss_text_ce: 0.0642445832490921 (0.06468627185450877) | > loss_mel_ce: 2.971989870071411 (nan) | > loss: 0.012048549950122833 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.13636170074911852) | > loader_time: 0.0046 (0.004669068543913128)  --> TIME: 2025-02-24 00:23:50 -- STEP: 13374/36938 -- GLOBAL_STEP: 87250 | > loss_text_ce: 0.0616939477622509 (0.06467271834671498) | > loss_mel_ce: 4.447993755340576 (nan) | > loss: 0.017895588651299477 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13633809790975687) | > loader_time: 0.0046 (0.0046687696089529945)  --> TIME: 2025-02-24 00:23:59 -- STEP: 13424/36938 -- GLOBAL_STEP: 87300 | > loss_text_ce: 0.06545280665159225 (0.06465770502775099) | > loss_mel_ce: 3.780125856399536 (nan) | > loss: 0.015260233543813229 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.1363350650089152) | > loader_time: 0.0048 (0.004668586853008022)  --> TIME: 2025-02-24 00:24:08 -- STEP: 13474/36938 -- GLOBAL_STEP: 87350 | > loss_text_ce: 0.0759614035487175 (0.06464275414699479) | > loss_mel_ce: 3.5795607566833496 (nan) | > loss: 0.014506041072309017 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.13634080261863546) | > loader_time: 0.0049 (0.00466862023186397)  --> TIME: 2025-02-24 00:24:18 -- STEP: 13524/36938 -- GLOBAL_STEP: 87400 | > loss_text_ce: 0.05919637903571129 (0.06463123901759009) | > loss_mel_ce: 3.7760865688323975 (nan) | > loss: 0.015219378285109997 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.1363282792492755) | > loader_time: 0.0046 (0.0046682865688061044)  --> TIME: 2025-02-24 00:24:27 -- STEP: 13574/36938 -- GLOBAL_STEP: 87450 | > loss_text_ce: 0.05644954741001129 (0.0646211544366826) | > loss_mel_ce: 4.260153293609619 (nan) | > loss: 0.017129376530647278 (nan) | > current_lr: 5e-06 | > step_time: 0.1445 (0.13633331378500063) | > loader_time: 0.0047 (0.00466788104905881)  --> TIME: 2025-02-24 00:24:36 -- STEP: 13624/36938 -- GLOBAL_STEP: 87500 | > loss_text_ce: 0.056313998997211456 (0.0646137566079018) | > loss_mel_ce: 3.3985934257507324 (nan) | > loss: 0.01370995119214058 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13632419453392441) | > loader_time: 0.0045 (0.00466763691487763)  --> TIME: 2025-02-24 00:24:45 -- STEP: 13674/36938 -- GLOBAL_STEP: 87550 | > loss_text_ce: 0.04373902082443237 (0.06459701279919555) | > loss_mel_ce: 3.58161997795105 (nan) | > loss: 0.014386346563696861 (nan) | > current_lr: 5e-06 | > step_time: 0.1068 (0.136330302580766) | > loader_time: 0.0045 (0.004667657813395395)  --> TIME: 2025-02-24 00:24:54 -- STEP: 13724/36938 -- GLOBAL_STEP: 87600 | > loss_text_ce: 0.05956166610121727 (0.06458088029509869) | > loss_mel_ce: 1.5443518161773682 (nan) | > loss: 0.006364736240357161 (nan) | > current_lr: 5e-06 | > step_time: 0.1083 (0.13631647239791775) | > loader_time: 0.0045 (0.0046675730919011026)  --> TIME: 2025-02-24 00:25:04 -- STEP: 13774/36938 -- GLOBAL_STEP: 87650 | > loss_text_ce: 0.07402963936328888 (0.06457002465730163) | > loss_mel_ce: 2.844682216644287 (nan) | > loss: 0.011582191102206707 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13629945312509398) | > loader_time: 0.0046 (0.004667526442858863)  --> TIME: 2025-02-24 00:25:13 -- STEP: 13824/36938 -- GLOBAL_STEP: 87700 | > loss_text_ce: 0.04841636121273041 (0.06455467633929445) | > loss_mel_ce: 0.38818359375 (nan) | > loss: 0.0017325396183878183 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.1363024423071354) | > loader_time: 0.0045 (0.0046674673514509645)  --> TIME: 2025-02-24 00:25:22 -- STEP: 13874/36938 -- GLOBAL_STEP: 87750 | > loss_text_ce: 0.06796307116746902 (0.0645382098932262) | > loss_mel_ce: 3.9535751342773438 (nan) | > loss: 0.01595848612487316 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13629918593708693) | > loader_time: 0.0046 (0.0046672621360339984)  --> TIME: 2025-02-24 00:25:32 -- STEP: 13924/36938 -- GLOBAL_STEP: 87800 | > loss_text_ce: 0.062092285603284836 (0.06452771155265324) | > loss_mel_ce: 3.914571523666382 (nan) | > loss: 0.015780413523316383 (nan) | > current_lr: 5e-06 | > step_time: 0.1482 (0.13631449186469702) | > loader_time: 0.0046 (0.0046668818407242275)  --> TIME: 2025-02-24 00:25:41 -- STEP: 13974/36938 -- GLOBAL_STEP: 87850 | > loss_text_ce: 0.0697418749332428 (0.0645238629652793) | > loss_mel_ce: 3.9782156944274902 (nan) | > loss: 0.016063323244452477 (nan) | > current_lr: 5e-06 | > step_time: 0.1605 (0.13630549930205593) | > loader_time: 0.0046 (0.004666835432215038)  --> TIME: 2025-02-24 00:25:50 -- STEP: 14024/36938 -- GLOBAL_STEP: 87900 | > loss_text_ce: 0.06402724236249924 (0.06451636557446046) | > loss_mel_ce: 3.6389689445495605 (nan) | > loss: 0.014694430865347385 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.13629712391362053) | > loader_time: 0.0048 (0.004666826518276931)  --> TIME: 2025-02-24 00:25:59 -- STEP: 14074/36938 -- GLOBAL_STEP: 87950 | > loss_text_ce: 0.057741425931453705 (0.06450956708308137) | > loss_mel_ce: 2.9633891582489014 (nan) | > loss: 0.01198861375451088 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.1362843271241668) | > loader_time: 0.0045 (0.0046668887663517756)  --> TIME: 2025-02-24 00:26:09 -- STEP: 14124/36938 -- GLOBAL_STEP: 88000 | > loss_text_ce: 0.05955394729971886 (0.06449896594785381) | > loss_mel_ce: 3.8209190368652344 (nan) | > loss: 0.015398702584207058 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13629690021132979) | > loader_time: 0.0046 (0.004666775102083287)  --> TIME: 2025-02-24 00:26:18 -- STEP: 14174/36938 -- GLOBAL_STEP: 88050 | > loss_text_ce: 0.05938812717795372 (0.06448454929654238) | > loss_mel_ce: 3.2015204429626465 (nan) | > loss: 0.012940114364027977 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.1362871303870543) | > loader_time: 0.0046 (0.004666494973307691)  --> TIME: 2025-02-24 00:26:27 -- STEP: 14224/36938 -- GLOBAL_STEP: 88100 | > loss_text_ce: 0.06557975709438324 (0.06447870043369332) | > loss_mel_ce: 3.4057042598724365 (nan) | > loss: 0.013774937018752098 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.1363086938790984) | > loader_time: 0.0046 (0.004666326938383491)  --> TIME: 2025-02-24 00:26:37 -- STEP: 14274/36938 -- GLOBAL_STEP: 88150 | > loss_text_ce: 0.04676542058587074 (0.0644702640788402) | > loss_mel_ce: 3.7968356609344482 (nan) | > loss: 0.015252386219799519 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.13633179713050506) | > loader_time: 0.0047 (0.004666258795381311)  --> TIME: 2025-02-24 00:26:46 -- STEP: 14324/36938 -- GLOBAL_STEP: 88200 | > loss_text_ce: 0.07178866863250732 (0.06445931767692695) | > loss_mel_ce: 3.730916738510132 (nan) | > loss: 0.015090101398527622 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13633155150308318) | > loader_time: 0.0046 (0.004666003142945163)  --> TIME: 2025-02-24 00:26:55 -- STEP: 14374/36938 -- GLOBAL_STEP: 88250 | > loss_text_ce: 0.062880739569664 (0.06445391027999901) | > loss_mel_ce: 1.2657538652420044 (nan) | > loss: 0.005272360052913427 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13631959150244527) | > loader_time: 0.0046 (0.004665700487322878)  --> TIME: 2025-02-24 00:27:05 -- STEP: 14424/36938 -- GLOBAL_STEP: 88300 | > loss_text_ce: 0.060995977371931076 (0.06444418398147915) | > loss_mel_ce: 4.131167888641357 (nan) | > loss: 0.016635572537779808 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13633877853850024) | > loader_time: 0.0047 (0.004665702763227906)  --> TIME: 2025-02-24 00:27:14 -- STEP: 14474/36938 -- GLOBAL_STEP: 88350 | > loss_text_ce: 0.05897749215364456 (0.06442287251151506) | > loss_mel_ce: 4.087687969207764 (nan) | > loss: 0.016455022618174553 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.1363308391008356) | > loader_time: 0.0048 (0.00466545820400928)  --> TIME: 2025-02-24 00:27:24 -- STEP: 14524/36938 -- GLOBAL_STEP: 88400 | > loss_text_ce: 0.04862584173679352 (0.06441414208235806) | > loss_mel_ce: 3.1563427448272705 (nan) | > loss: 0.012718130834400654 (nan) | > current_lr: 5e-06 | > step_time: 0.1739 (0.13633609817757567) | > loader_time: 0.0047 (0.0046652640662092175)  --> TIME: 2025-02-24 00:27:33 -- STEP: 14574/36938 -- GLOBAL_STEP: 88450 | > loss_text_ce: 0.05796490237116814 (0.06440331198758728) | > loss_mel_ce: 3.871633291244507 (nan) | > loss: 0.015593644231557846 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13633086523663554) | > loader_time: 0.0045 (0.0046649470943735186)  --> TIME: 2025-02-24 00:27:42 -- STEP: 14624/36938 -- GLOBAL_STEP: 88500 | > loss_text_ce: 0.05632364749908447 (0.06439012632714468) | > loss_mel_ce: 3.1235992908477783 (nan) | > loss: 0.012618742883205414 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13633443982517696) | > loader_time: 0.0046 (0.00466470611108471)  --> TIME: 2025-02-24 00:27:51 -- STEP: 14674/36938 -- GLOBAL_STEP: 88550 | > loss_text_ce: 0.0679248720407486 (0.06438270159481083) | > loss_mel_ce: 4.553020477294922 (nan) | > loss: 0.018337085843086243 (nan) | > current_lr: 5e-06 | > step_time: 0.1862 (0.1363205561865995) | > loader_time: 0.0048 (0.00466447401651167)  --> TIME: 2025-02-24 00:28:00 -- STEP: 14724/36938 -- GLOBAL_STEP: 88600 | > loss_text_ce: 0.05446662753820419 (0.06437649443394973) | > loss_mel_ce: 3.687023639678955 (nan) | > loss: 0.014847184531390667 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.1363081237894787) | > loader_time: 0.0049 (0.00466424931135232)  --> TIME: 2025-02-24 00:28:09 -- STEP: 14774/36938 -- GLOBAL_STEP: 88650 | > loss_text_ce: 0.05884191021323204 (0.06436555093529789) | > loss_mel_ce: 4.296629905700684 (nan) | > loss: 0.017283618450164795 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.1363027704347853) | > loader_time: 0.0046 (0.004664122275639987)  --> TIME: 2025-02-24 00:28:19 -- STEP: 14824/36938 -- GLOBAL_STEP: 88700 | > loss_text_ce: 0.049850109964609146 (0.06434880863788227) | > loss_mel_ce: 3.282109260559082 (nan) | > loss: 0.013222062028944492 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13632283156714556) | > loader_time: 0.0045 (0.004663921229207574)  --> TIME: 2025-02-24 00:28:28 -- STEP: 14874/36938 -- GLOBAL_STEP: 88750 | > loss_text_ce: 0.07868291437625885 (0.06433459553998841) | > loss_mel_ce: 4.166291236877441 (nan) | > loss: 0.016845136880874634 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.1363158895713885) | > loader_time: 0.0048 (0.004663636563557426)  --> TIME: 2025-02-24 00:28:38 -- STEP: 14924/36938 -- GLOBAL_STEP: 88800 | > loss_text_ce: 0.05625004693865776 (0.06432223165506645) | > loss_mel_ce: 3.949831962585449 (nan) | > loss: 0.015897151082754135 (nan) | > current_lr: 5e-06 | > step_time: 0.1797 (0.13632132289109938) | > loader_time: 0.005 (0.004663506355531029)  --> TIME: 2025-02-24 00:28:47 -- STEP: 14974/36938 -- GLOBAL_STEP: 88850 | > loss_text_ce: 0.05827789008617401 (0.06430775443701779) | > loss_mel_ce: 0.07861328125 (nan) | > loss: 0.0005432189791463315 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13632420993892574) | > loader_time: 0.0045 (0.004663438651786877)  --> TIME: 2025-02-24 00:28:56 -- STEP: 15024/36938 -- GLOBAL_STEP: 88900 | > loss_text_ce: 0.06442248821258545 (0.06429650915300163) | > loss_mel_ce: 3.3458454608917236 (nan) | > loss: 0.013532809913158417 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.1363144895107475) | > loader_time: 0.0046 (0.0046631273624107355)  --> TIME: 2025-02-24 00:29:06 -- STEP: 15074/36938 -- GLOBAL_STEP: 88950 | > loss_text_ce: 0.06645718961954117 (0.06428412584822982) | > loss_mel_ce: 3.049283504486084 (nan) | > loss: 0.012364051304757595 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13632909630343604) | > loader_time: 0.0046 (0.00466286868778293)  --> TIME: 2025-02-24 00:29:15 -- STEP: 15124/36938 -- GLOBAL_STEP: 89000 | > loss_text_ce: 0.06370194256305695 (0.06427188078803664) | > loss_mel_ce: 3.88558292388916 (nan) | > loss: 0.01567176543176174 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13632150998162207) | > loader_time: 0.0044 (0.00466258192907342)  --> TIME: 2025-02-24 00:29:24 -- STEP: 15174/36938 -- GLOBAL_STEP: 89050 | > loss_text_ce: 0.06361670792102814 (0.06425890516827577) | > loss_mel_ce: 4.114644527435303 (nan) | > loss: 0.016580402851104736 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.13633362970695337) | > loader_time: 0.0045 (0.0046622977200839084)  --> TIME: 2025-02-24 00:29:33 -- STEP: 15224/36938 -- GLOBAL_STEP: 89100 | > loss_text_ce: 0.051497071981430054 (0.06424899254822791) | > loss_mel_ce: 3.053847074508667 (nan) | > loss: 0.012322794646024704 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13632434013011396) | > loader_time: 0.0045 (0.004662011196534381)  --> TIME: 2025-02-24 00:29:43 -- STEP: 15274/36938 -- GLOBAL_STEP: 89150 | > loss_text_ce: 0.06810221076011658 (0.06423436997806396) | > loss_mel_ce: 4.333861351013184 (nan) | > loss: 0.017468111589550972 (nan) | > current_lr: 5e-06 | > step_time: 0.1797 (0.1363381045767193) | > loader_time: 0.0047 (0.004661915688454145)  --> TIME: 2025-02-24 00:29:52 -- STEP: 15324/36938 -- GLOBAL_STEP: 89200 | > loss_text_ce: 0.046249303966760635 (0.06422099011933916) | > loss_mel_ce: 3.552232265472412 (nan) | > loss: 0.014279689639806747 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13633080245744655) | > loader_time: 0.0044 (0.004661949254675467)  --> TIME: 2025-02-24 00:30:01 -- STEP: 15374/36938 -- GLOBAL_STEP: 89250 | > loss_text_ce: 0.07150869816541672 (0.06420863627080431) | > loss_mel_ce: 3.814852714538574 (nan) | > loss: 0.01542206946760416 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13632786636841354) | > loader_time: 0.0047 (0.00466161689507693)  --> TIME: 2025-02-24 00:30:11 -- STEP: 15424/36938 -- GLOBAL_STEP: 89300 | > loss_text_ce: 0.05637295916676521 (0.06419757823157902) | > loss_mel_ce: 4.125835418701172 (nan) | > loss: 0.01659606583416462 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.1363417748593691) | > loader_time: 0.0046 (0.004661343280705179)  --> TIME: 2025-02-24 00:30:20 -- STEP: 15474/36938 -- GLOBAL_STEP: 89350 | > loss_text_ce: 0.058414917439222336 (0.06418504466562582) | > loss_mel_ce: 3.5331554412841797 (nan) | > loss: 0.01425226405262947 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13634780221504966) | > loader_time: 0.0045 (0.004661285955806997)  --> TIME: 2025-02-24 00:30:30 -- STEP: 15524/36938 -- GLOBAL_STEP: 89400 | > loss_text_ce: 0.06077982857823372 (0.06417756312157452) | > loss_mel_ce: 3.80983567237854 (nan) | > loss: 0.015359586104750633 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.13634970832565224) | > loader_time: 0.0048 (0.0046609688038618344)  --> TIME: 2025-02-24 00:30:39 -- STEP: 15574/36938 -- GLOBAL_STEP: 89450 | > loss_text_ce: 0.054035089910030365 (0.06416644408274652) | > loss_mel_ce: 3.117997646331787 (nan) | > loss: 0.012587432749569416 (nan) | > current_lr: 5e-06 | > step_time: 0.1862 (0.1363546764098957) | > loader_time: 0.0049 (0.004660643094676258)  --> TIME: 2025-02-24 00:30:48 -- STEP: 15624/36938 -- GLOBAL_STEP: 89500 | > loss_text_ce: 0.06865561008453369 (0.06415838683374551) | > loss_mel_ce: 3.98146915435791 (nan) | > loss: 0.016071924939751625 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13633621056386286) | > loader_time: 0.0045 (0.00466032618445979)  --> TIME: 2025-02-24 00:30:57 -- STEP: 15674/36938 -- GLOBAL_STEP: 89550 | > loss_text_ce: 0.05931193381547928 (0.06414886022679898) | > loss_mel_ce: 4.060336112976074 (nan) | > loss: 0.016347810626029968 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.13633880249255625) | > loader_time: 0.0045 (0.004660138612927048)  --> TIME: 2025-02-24 00:31:07 -- STEP: 15724/36938 -- GLOBAL_STEP: 89600 | > loss_text_ce: 0.052507832646369934 (0.06413327836609493) | > loss_mel_ce: 3.0003559589385986 (nan) | > loss: 0.012114539742469788 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13634650104071955) | > loader_time: 0.0044 (0.004660024060089453)  --> TIME: 2025-02-24 00:31:16 -- STEP: 15774/36938 -- GLOBAL_STEP: 89650 | > loss_text_ce: 0.0410257987678051 (0.06412324204502953) | > loss_mel_ce: 3.2959625720977783 (nan) | > loss: 0.013242018409073353 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1363458439350673) | > loader_time: 0.0045 (0.004659676591116953)  --> TIME: 2025-02-24 00:31:25 -- STEP: 15824/36938 -- GLOBAL_STEP: 89700 | > loss_text_ce: 0.06228093430399895 (0.06410945709020234) | > loss_mel_ce: 3.9981086254119873 (nan) | > loss: 0.016112657263875008 (nan) | > current_lr: 5e-06 | > step_time: 0.1798 (0.13633245784182887) | > loader_time: 0.005 (0.004659327008847159)  --> TIME: 2025-02-24 00:31:34 -- STEP: 15874/36938 -- GLOBAL_STEP: 89750 | > loss_text_ce: 0.04756362736225128 (0.06409635126749849) | > loss_mel_ce: 4.2152910232543945 (nan) | > loss: 0.01691609062254429 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13632511640505215) | > loader_time: 0.0045 (0.0046589795386924905)  --> TIME: 2025-02-24 00:31:43 -- STEP: 15924/36938 -- GLOBAL_STEP: 89800 | > loss_text_ce: 0.0723615437746048 (0.06408470885896453) | > loss_mel_ce: 4.086061000823975 (nan) | > loss: 0.01650167815387249 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13631064855582276) | > loader_time: 0.0045 (0.004658606491984809)  --> TIME: 2025-02-24 00:31:52 -- STEP: 15974/36938 -- GLOBAL_STEP: 89850 | > loss_text_ce: 0.04485514014959335 (0.06407498393774033) | > loss_mel_ce: 1.4140815734863281 (nan) | > loss: 0.005789431743323803 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13630552302019405) | > loader_time: 0.0045 (0.004658241616451234)  --> TIME: 2025-02-24 00:32:02 -- STEP: 16024/36938 -- GLOBAL_STEP: 89900 | > loss_text_ce: 0.05429178476333618 (0.06406325040256343) | > loss_mel_ce: 4.16320276260376 (nan) | > loss: 0.016736090183258057 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13632113108977764) | > loader_time: 0.0046 (0.004658016096754069)  --> TIME: 2025-02-24 00:32:11 -- STEP: 16074/36938 -- GLOBAL_STEP: 89950 | > loss_text_ce: 0.057760853320360184 (0.06404962089965849) | > loss_mel_ce: 3.9157700538635254 (nan) | > loss: 0.015767980366945267 (nan) | > current_lr: 5e-06 | > step_time: 0.1827 (0.1363341970735631) | > loader_time: 0.0048 (0.004657784237469461)  --> TIME: 2025-02-24 00:32:21 -- STEP: 16124/36938 -- GLOBAL_STEP: 90000 | > loss_text_ce: 0.051393527537584305 (0.06403492803204122) | > loss_mel_ce: 3.6346476078033447 (nan) | > loss: 0.014627148397266865 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13634705179768566) | > loader_time: 0.0052 (0.004658005841462673) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_90000.pth  --> TIME: 2025-02-24 00:32:36 -- STEP: 16174/36938 -- GLOBAL_STEP: 90050 | > loss_text_ce: 0.057816702872514725 (0.06402787961412651) | > loss_mel_ce: 4.288331031799316 (nan) | > loss: 0.01724661886692047 (nan) | > current_lr: 5e-06 | > step_time: 0.1947 (0.1363449335923755) | > loader_time: 0.0049 (0.004658303243703831)  --> TIME: 2025-02-24 00:32:45 -- STEP: 16224/36938 -- GLOBAL_STEP: 90100 | > loss_text_ce: 0.05449915677309036 (0.06401863202456803) | > loss_mel_ce: 1.334448218345642 (nan) | > loss: 0.005511696450412273 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.1363227528903847) | > loader_time: 0.0045 (0.004658240714722131)  --> TIME: 2025-02-24 00:32:55 -- STEP: 16274/36938 -- GLOBAL_STEP: 90150 | > loss_text_ce: 0.055455002933740616 (0.06400731361839788) | > loss_mel_ce: 4.204043865203857 (nan) | > loss: 0.016902774572372437 (nan) | > current_lr: 5e-06 | > step_time: 0.1854 (0.13633087581771888) | > loader_time: 0.005 (0.004658334990955328)  --> TIME: 2025-02-24 00:33:04 -- STEP: 16324/36938 -- GLOBAL_STEP: 90200 | > loss_text_ce: 0.08053622394800186 (0.06400018760909837) | > loss_mel_ce: 4.1693315505981445 (nan) | > loss: 0.016864556819200516 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.1363253087075544) | > loader_time: 0.0044 (0.004658375511389099)  --> TIME: 2025-02-24 00:33:13 -- STEP: 16374/36938 -- GLOBAL_STEP: 90250 | > loss_text_ce: 0.06704559177160263 (0.06399369296355381) | > loss_mel_ce: 2.0831491947174072 (nan) | > loss: 0.008532519452273846 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13632303858554687) | > loader_time: 0.0044 (0.004658078934751152)  --> TIME: 2025-02-24 00:33:22 -- STEP: 16424/36938 -- GLOBAL_STEP: 90300 | > loss_text_ce: 0.0812581330537796 (0.06398388954524563) | > loss_mel_ce: 4.24609375 (nan) | > loss: 0.017172032967209816 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13632006747979283) | > loader_time: 0.0044 (0.004658048247570998)  --> TIME: 2025-02-24 00:33:32 -- STEP: 16474/36938 -- GLOBAL_STEP: 90350 | > loss_text_ce: 0.061768077313899994 (0.0639740577483404) | > loss_mel_ce: 3.982785701751709 (nan) | > loss: 0.016049817204475403 (nan) | > current_lr: 5e-06 | > step_time: 0.1872 (0.1363262259672174) | > loader_time: 0.0048 (0.004657813844800039)  --> TIME: 2025-02-24 00:33:41 -- STEP: 16524/36938 -- GLOBAL_STEP: 90400 | > loss_text_ce: 0.055852100253105164 (0.06396136693052257) | > loss_mel_ce: 3.520127534866333 (nan) | > loss: 0.014190396293997765 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13631928753431533) | > loader_time: 0.0045 (0.004657595851928667)  --> TIME: 2025-02-24 00:33:50 -- STEP: 16574/36938 -- GLOBAL_STEP: 90450 | > loss_text_ce: 0.056403748691082 (0.0639545766917015) | > loss_mel_ce: 1.0832570791244507 (nan) | > loss: 0.004522463772445917 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13632973649899696) | > loader_time: 0.0046 (0.004657390337161827)  --> TIME: 2025-02-24 00:34:00 -- STEP: 16624/36938 -- GLOBAL_STEP: 90500 | > loss_text_ce: 0.06259819120168686 (0.0639425959362792) | > loss_mel_ce: 3.550076484680176 (nan) | > loss: 0.014336011372506618 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13634980730246313) | > loader_time: 0.0052 (0.00465783297556656)  --> TIME: 2025-02-24 00:34:10 -- STEP: 16674/36938 -- GLOBAL_STEP: 90550 | > loss_text_ce: 0.06924375891685486 (0.06393751698824614) | > loss_mel_ce: 4.4266357421875 (nan) | > loss: 0.0178407933562994 (nan) | > current_lr: 5e-06 | > step_time: 0.1731 (0.13635731764344192) | > loader_time: 0.0047 (0.004658140480711375)  --> TIME: 2025-02-24 00:34:19 -- STEP: 16724/36938 -- GLOBAL_STEP: 90600 | > loss_text_ce: 0.07030679285526276 (0.06392880176594755) | > loss_mel_ce: 2.766404390335083 (nan) | > loss: 0.011256790719926357 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13638028704187746) | > loader_time: 0.0047 (0.004658038608767263)  --> TIME: 2025-02-24 00:34:28 -- STEP: 16774/36938 -- GLOBAL_STEP: 90650 | > loss_text_ce: 0.06280340254306793 (0.06392045511341235) | > loss_mel_ce: 4.120508670806885 (nan) | > loss: 0.01660044491291046 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.13636395323471254) | > loader_time: 0.0047 (0.00465760626724114)  --> TIME: 2025-02-24 00:34:37 -- STEP: 16824/36938 -- GLOBAL_STEP: 90700 | > loss_text_ce: 0.05890335887670517 (0.06391279285957875) | > loss_mel_ce: 4.145238876342773 (nan) | > loss: 0.016683105379343033 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.1363556082340519) | > loader_time: 0.0045 (0.004657266851952116)  --> TIME: 2025-02-24 00:34:46 -- STEP: 16874/36938 -- GLOBAL_STEP: 90750 | > loss_text_ce: 0.06596923619508743 (0.0639014407509759) | > loss_mel_ce: 4.097039222717285 (nan) | > loss: 0.016519876196980476 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.1363414227150058) | > loader_time: 0.0046 (0.004656907222671388)  --> TIME: 2025-02-24 00:34:55 -- STEP: 16924/36938 -- GLOBAL_STEP: 90800 | > loss_text_ce: 0.05886895954608917 (0.06388625356669636) | > loss_mel_ce: 3.4016478061676025 (nan) | > loss: 0.01373220980167389 (nan) | > current_lr: 5e-06 | > step_time: 0.1438 (0.13632517886821116) | > loader_time: 0.0046 (0.0046565582554437815)  --> TIME: 2025-02-24 00:35:05 -- STEP: 16974/36938 -- GLOBAL_STEP: 90850 | > loss_text_ce: 0.05689748004078865 (0.06387580529555886) | > loss_mel_ce: 3.858828544616699 (nan) | > loss: 0.015538595616817474 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13633279118055286) | > loader_time: 0.0046 (0.004656212369474516)  --> TIME: 2025-02-24 00:35:14 -- STEP: 17024/36938 -- GLOBAL_STEP: 90900 | > loss_text_ce: 0.0649171695113182 (0.0638717372896925) | > loss_mel_ce: 4.501965045928955 (nan) | > loss: 0.01812255010008812 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13632684205203582) | > loader_time: 0.0045 (0.004655890082730386)  --> TIME: 2025-02-24 00:35:24 -- STEP: 17074/36938 -- GLOBAL_STEP: 90950 | > loss_text_ce: 0.06743554025888443 (0.06385574508944943) | > loss_mel_ce: 3.329159736633301 (nan) | > loss: 0.013478552922606468 (nan) | > current_lr: 5e-06 | > step_time: 0.1823 (0.13635571977356864) | > loader_time: 0.0047 (0.004655712086799429)  --> TIME: 2025-02-24 00:35:33 -- STEP: 17124/36938 -- GLOBAL_STEP: 91000 | > loss_text_ce: 0.06394706666469574 (0.06384704729985514) | > loss_mel_ce: 3.5509839057922363 (nan) | > loss: 0.014344964176416397 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.1363554172247433) | > loader_time: 0.0045 (0.004655371784913264)  --> TIME: 2025-02-24 00:35:43 -- STEP: 17174/36938 -- GLOBAL_STEP: 91050 | > loss_text_ce: 0.055451590567827225 (0.06383300053383735) | > loss_mel_ce: 3.1165108680725098 (nan) | > loss: 0.012587153352797031 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13636286776238254) | > loader_time: 0.0047 (0.004655028897169182)  --> TIME: 2025-02-24 00:35:52 -- STEP: 17224/36938 -- GLOBAL_STEP: 91100 | > loss_text_ce: 0.06482425332069397 (0.06382040754397526) | > loss_mel_ce: 3.913832902908325 (nan) | > loss: 0.015788324177265167 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13638017366389824) | > loader_time: 0.0044 (0.004654907967066468)  --> TIME: 2025-02-24 00:36:02 -- STEP: 17274/36938 -- GLOBAL_STEP: 91150 | > loss_text_ce: 0.06169318035244942 (0.06381249720789495) | > loss_mel_ce: 4.348731994628906 (nan) | > loss: 0.01750168763101101 (nan) | > current_lr: 5e-06 | > step_time: 0.1634 (0.13638376662393567) | > loader_time: 0.0046 (0.004654619212612673)  --> TIME: 2025-02-24 00:36:11 -- STEP: 17324/36938 -- GLOBAL_STEP: 91200 | > loss_text_ce: 0.05090124532580376 (0.06380092861039253) | > loss_mel_ce: 3.910905599594116 (nan) | > loss: 0.015721457079052925 (nan) | > current_lr: 5e-06 | > step_time: 0.1879 (0.13638090150126733) | > loader_time: 0.0047 (0.004654525774671917)  --> TIME: 2025-02-24 00:36:20 -- STEP: 17374/36938 -- GLOBAL_STEP: 91250 | > loss_text_ce: 0.05057757347822189 (0.06379208411677069) | > loss_mel_ce: 3.948840618133545 (nan) | > loss: 0.015870708972215652 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.1363710405583127) | > loader_time: 0.0045 (0.004654183395441077)  --> TIME: 2025-02-24 00:36:30 -- STEP: 17424/36938 -- GLOBAL_STEP: 91300 | > loss_text_ce: 0.061351336538791656 (0.06377598362430938) | > loss_mel_ce: 3.524594306945801 (nan) | > loss: 0.014229943975806236 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.1363856174236487) | > loader_time: 0.0045 (0.004653996576728686)  --> TIME: 2025-02-24 00:36:39 -- STEP: 17474/36938 -- GLOBAL_STEP: 91350 | > loss_text_ce: 0.06094935163855553 (0.0637570369638624) | > loss_mel_ce: 3.6093719005584717 (nan) | > loss: 0.014564767479896545 (nan) | > current_lr: 5e-06 | > step_time: 0.1689 (0.1363760940103791) | > loader_time: 0.0046 (0.004653997807122353)  --> TIME: 2025-02-24 00:36:48 -- STEP: 17524/36938 -- GLOBAL_STEP: 91400 | > loss_text_ce: 0.058148495852947235 (0.06374793103688842) | > loss_mel_ce: 3.707554578781128 (nan) | > loss: 0.014943266287446022 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13637106037934435) | > loader_time: 0.0045 (0.004653803767800962)  --> TIME: 2025-02-24 00:36:57 -- STEP: 17574/36938 -- GLOBAL_STEP: 91450 | > loss_text_ce: 0.05698622390627861 (0.0637368937636131) | > loss_mel_ce: 3.56012225151062 (nan) | > loss: 0.014353605918586254 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.1363872788906694) | > loader_time: 0.0047 (0.004653688338316523)  --> TIME: 2025-02-24 00:37:07 -- STEP: 17624/36938 -- GLOBAL_STEP: 91500 | > loss_text_ce: 0.07930000126361847 (0.06372051266657444) | > loss_mel_ce: 3.3586037158966064 (nan) | > loss: 0.013642475940287113 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13639308008290513) | > loader_time: 0.0046 (0.004653465853352149)  --> TIME: 2025-02-24 00:37:16 -- STEP: 17674/36938 -- GLOBAL_STEP: 91550 | > loss_text_ce: 0.06153201311826706 (0.06371079799227534) | > loss_mel_ce: 3.7890939712524414 (nan) | > loss: 0.015280262567102909 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13639599629167234) | > loader_time: 0.0045 (0.004653190762475705)  --> TIME: 2025-02-24 00:37:25 -- STEP: 17724/36938 -- GLOBAL_STEP: 91600 | > loss_text_ce: 0.053970351815223694 (0.06370255617642194) | > loss_mel_ce: 3.6039631366729736 (nan) | > loss: 0.014515609480440617 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13639954342526805) | > loader_time: 0.0044 (0.004652956314426982)  --> TIME: 2025-02-24 00:37:35 -- STEP: 17774/36938 -- GLOBAL_STEP: 91650 | > loss_text_ce: 0.06176134571433067 (0.06370151845047285) | > loss_mel_ce: 3.1590728759765625 (nan) | > loss: 0.012781089171767235 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13639107092006075) | > loader_time: 0.0045 (0.004652644955590116)  --> TIME: 2025-02-24 00:37:44 -- STEP: 17824/36938 -- GLOBAL_STEP: 91700 | > loss_text_ce: 0.05789995938539505 (0.06368977469484102) | > loss_mel_ce: 3.98005747795105 (nan) | > loss: 0.01602364331483841 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13638794682986877) | > loader_time: 0.0044 (0.0046524393173265795)  --> TIME: 2025-02-24 00:37:53 -- STEP: 17874/36938 -- GLOBAL_STEP: 91750 | > loss_text_ce: 0.058374956250190735 (0.06367703705723465) | > loss_mel_ce: 3.3707275390625 (nan) | > loss: 0.013607550412416458 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1363827041081967) | > loader_time: 0.0045 (0.004652131760268462)  --> TIME: 2025-02-24 00:38:02 -- STEP: 17924/36938 -- GLOBAL_STEP: 91800 | > loss_text_ce: 0.0697656199336052 (0.06366932110749136) | > loss_mel_ce: 0.933837890625 (nan) | > loss: 0.0039825537241995335 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13637795175087583) | > loader_time: 0.0045 (0.004651922861451268)  --> TIME: 2025-02-24 00:38:12 -- STEP: 17974/36938 -- GLOBAL_STEP: 91850 | > loss_text_ce: 0.06139230728149414 (0.06366196003619928) | > loss_mel_ce: 4.211818218231201 (nan) | > loss: 0.01695718616247177 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13638189262896552) | > loader_time: 0.0045 (0.00465168216223976)  --> TIME: 2025-02-24 00:38:21 -- STEP: 18024/36938 -- GLOBAL_STEP: 91900 | > loss_text_ce: 0.060513779520988464 (0.06364726420567172) | > loss_mel_ce: 3.8753788471221924 (nan) | > loss: 0.015618622303009033 (nan) | > current_lr: 5e-06 | > step_time: 0.1481 (0.13637248596984256) | > loader_time: 0.0047 (0.004651405654692205)  --> TIME: 2025-02-24 00:38:30 -- STEP: 18074/36938 -- GLOBAL_STEP: 91950 | > loss_text_ce: 0.062262214720249176 (0.06363747850525463) | > loss_mel_ce: 3.445173978805542 (nan) | > loss: 0.013918398879468441 (nan) | > current_lr: 5e-06 | > step_time: 0.1758 (0.13637087559349143) | > loader_time: 0.0048 (0.0046511873466008056)  --> TIME: 2025-02-24 00:38:39 -- STEP: 18124/36938 -- GLOBAL_STEP: 92000 | > loss_text_ce: 0.06412389129400253 (0.0636278839218278) | > loss_mel_ce: 3.4323973655700684 (nan) | > loss: 0.013875084929168224 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13636308861105062) | > loader_time: 0.0043 (0.004650912611617664)  --> TIME: 2025-02-24 00:38:49 -- STEP: 18174/36938 -- GLOBAL_STEP: 92050 | > loss_text_ce: 0.06172960624098778 (0.06362146764372184) | > loss_mel_ce: 3.847377300262451 (nan) | > loss: 0.015512330457568169 (nan) | > current_lr: 5e-06 | > step_time: 0.1083 (0.13638434611335512) | > loader_time: 0.0043 (0.004650969585071537)  --> TIME: 2025-02-24 00:38:58 -- STEP: 18224/36938 -- GLOBAL_STEP: 92100 | > loss_text_ce: 0.04858620837330818 (0.06361009650356766) | > loss_mel_ce: 3.8348093032836914 (nan) | > loss: 0.015410300344228745 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13637598637324663) | > loader_time: 0.0046 (0.004650667427505088)  --> TIME: 2025-02-24 00:39:08 -- STEP: 18274/36938 -- GLOBAL_STEP: 92150 | > loss_text_ce: 0.06949996203184128 (0.06359901135919681) | > loss_mel_ce: 3.9365434646606445 (nan) | > loss: 0.015896998345851898 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13639372506965716) | > loader_time: 0.0044 (0.00465046499677544)  --> TIME: 2025-02-24 00:39:16 -- STEP: 18324/36938 -- GLOBAL_STEP: 92200 | > loss_text_ce: 0.05019655078649521 (0.0635904752633307) | > loss_mel_ce: 3.8104939460754395 (nan) | > loss: 0.01532020140439272 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13637640428865777) | > loader_time: 0.0044 (0.004650104009436365)  --> TIME: 2025-02-24 00:39:26 -- STEP: 18374/36938 -- GLOBAL_STEP: 92250 | > loss_text_ce: 0.07752401381731033 (0.06357849252622012) | > loss_mel_ce: 4.393750190734863 (nan) | > loss: 0.017743153497576714 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13638064892958734) | > loader_time: 0.0044 (0.004649845653536853)  --> TIME: 2025-02-24 00:39:36 -- STEP: 18424/36938 -- GLOBAL_STEP: 92300 | > loss_text_ce: 0.08673054724931717 (0.06356526683766085) | > loss_mel_ce: 1.7762625217437744 (nan) | > loss: 0.007392830215394497 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13640017780955335) | > loader_time: 0.0044 (0.004649624985505652)  --> TIME: 2025-02-24 00:39:45 -- STEP: 18474/36938 -- GLOBAL_STEP: 92350 | > loss_text_ce: 0.06501708924770355 (0.06355395310713562) | > loss_mel_ce: 3.3278579711914062 (nan) | > loss: 0.013463790528476238 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13640284618283938) | > loader_time: 0.0044 (0.004649291348764503)  --> TIME: 2025-02-24 00:39:54 -- STEP: 18524/36938 -- GLOBAL_STEP: 92400 | > loss_text_ce: 0.06630010157823563 (0.06354947781012919) | > loss_mel_ce: 1.5657399892807007 (nan) | > loss: 0.006476350128650665 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13640836148302177) | > loader_time: 0.0048 (0.00464904164166206)  --> TIME: 2025-02-24 00:40:03 -- STEP: 18574/36938 -- GLOBAL_STEP: 92450 | > loss_text_ce: 0.06501150131225586 (0.0635388768903325) | > loss_mel_ce: 3.471047878265381 (nan) | > loss: 0.01403198204934597 (nan) | > current_lr: 5e-06 | > step_time: 0.1343 (0.13640452035856895) | > loader_time: 0.0045 (0.004648745977751509)  --> TIME: 2025-02-24 00:40:12 -- STEP: 18624/36938 -- GLOBAL_STEP: 92500 | > loss_text_ce: 0.039990074932575226 (0.0635306535435113) | > loss_mel_ce: 3.138094902038574 (nan) | > loss: 0.0126114496961236 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.1363969828126972) | > loader_time: 0.0048 (0.004648448995400969)  --> TIME: 2025-02-24 00:40:22 -- STEP: 18674/36938 -- GLOBAL_STEP: 92550 | > loss_text_ce: 0.054373592138290405 (0.06352171012979144) | > loss_mel_ce: 2.9098174571990967 (nan) | > loss: 0.011762662790715694 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13640454723988968) | > loader_time: 0.0045 (0.00464822568818873)  --> TIME: 2025-02-24 00:40:31 -- STEP: 18724/36938 -- GLOBAL_STEP: 92600 | > loss_text_ce: 0.04960525780916214 (0.06350451841022495) | > loss_mel_ce: 3.790358304977417 (nan) | > loss: 0.015237951651215553 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13640084229226668) | > loader_time: 0.0045 (0.004647898778424429)  --> TIME: 2025-02-24 00:40:41 -- STEP: 18774/36938 -- GLOBAL_STEP: 92650 | > loss_text_ce: 0.060781367123126984 (0.06349182297500752) | > loss_mel_ce: 4.054039478302002 (nan) | > loss: 0.016328655183315277 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13641224250862571) | > loader_time: 0.0044 (0.004647677656153202)  --> TIME: 2025-02-24 00:40:50 -- STEP: 18824/36938 -- GLOBAL_STEP: 92700 | > loss_text_ce: 0.057187821716070175 (0.06348275866815055) | > loss_mel_ce: 3.8718435764312744 (nan) | > loss: 0.01559139508754015 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.13641903412326903) | > loader_time: 0.0045 (0.004647616042110008)  --> TIME: 2025-02-24 00:41:00 -- STEP: 18874/36938 -- GLOBAL_STEP: 92750 | > loss_text_ce: 0.0623907707631588 (0.06347163887940227) | > loss_mel_ce: 2.875288248062134 (nan) | > loss: 0.011657456867396832 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.1364212139463534) | > loader_time: 0.0045 (0.0046474711425327055)  --> TIME: 2025-02-24 00:41:09 -- STEP: 18924/36938 -- GLOBAL_STEP: 92800 | > loss_text_ce: 0.061205971986055374 (0.06346246362310466) | > loss_mel_ce: 3.887759208679199 (nan) | > loss: 0.015670496970415115 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13641434630740937) | > loader_time: 0.0044 (0.004647350190331398)  --> TIME: 2025-02-24 00:41:18 -- STEP: 18974/36938 -- GLOBAL_STEP: 92850 | > loss_text_ce: 0.054605815559625626 (0.06344992012104375) | > loss_mel_ce: 4.095398902893066 (nan) | > loss: 0.016468273475766182 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13642820098622066) | > loader_time: 0.0048 (0.004647281808974801)  --> TIME: 2025-02-24 00:41:28 -- STEP: 19024/36938 -- GLOBAL_STEP: 92900 | > loss_text_ce: 0.056628841906785965 (0.06344139224264543) | > loss_mel_ce: 4.189606666564941 (nan) | > loss: 0.016850141808390617 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.1364374302885335) | > loader_time: 0.0046 (0.004647130433303973)  --> TIME: 2025-02-24 00:41:37 -- STEP: 19074/36938 -- GLOBAL_STEP: 92950 | > loss_text_ce: 0.06465694308280945 (0.06343477678002724) | > loss_mel_ce: 4.053502082824707 (nan) | > loss: 0.01634190045297146 (nan) | > current_lr: 5e-06 | > step_time: 0.2006 (0.13644191601797181) | > loader_time: 0.0048 (0.0046468984034497346)  --> TIME: 2025-02-24 00:41:47 -- STEP: 19124/36938 -- GLOBAL_STEP: 93000 | > loss_text_ce: 0.06770879775285721 (0.06342333242982089) | > loss_mel_ce: 4.141113758087158 (nan) | > loss: 0.016701677814126015 (nan) | > current_lr: 5e-06 | > step_time: 0.1884 (0.13644564357469188) | > loader_time: 0.0048 (0.004646830642554998)  --> TIME: 2025-02-24 00:41:56 -- STEP: 19174/36938 -- GLOBAL_STEP: 93050 | > loss_text_ce: 0.0634637102484703 (0.0634094467265174) | > loss_mel_ce: 4.070892810821533 (nan) | > loss: 0.01640617661178112 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.136439513523512) | > loader_time: 0.0046 (0.0046467685570144605)  --> TIME: 2025-02-24 00:42:05 -- STEP: 19224/36938 -- GLOBAL_STEP: 93100 | > loss_text_ce: 0.057945460081100464 (0.06340079994107163) | > loss_mel_ce: 4.075835704803467 (nan) | > loss: 0.016403893008828163 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1364247761465248) | > loader_time: 0.0046 (0.004646553491682915)  --> TIME: 2025-02-24 00:42:14 -- STEP: 19274/36938 -- GLOBAL_STEP: 93150 | > loss_text_ce: 0.06233417987823486 (0.06339416670448421) | > loss_mel_ce: 4.447713851928711 (nan) | > loss: 0.017897017300128937 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.13643345797446707) | > loader_time: 0.0047 (0.004646434889822413)  --> TIME: 2025-02-24 00:42:24 -- STEP: 19324/36938 -- GLOBAL_STEP: 93200 | > loss_text_ce: 0.06630048900842667 (0.06338349438986292) | > loss_mel_ce: 2.920966625213623 (nan) | > loss: 0.011854235082864761 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13643248655909151) | > loader_time: 0.0045 (0.004646232843251018)  --> TIME: 2025-02-24 00:42:33 -- STEP: 19374/36938 -- GLOBAL_STEP: 93250 | > loss_text_ce: 0.050866179168224335 (0.06337693311186375) | > loss_mel_ce: 4.171154975891113 (nan) | > loss: 0.01675405353307724 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13643734610411837) | > loader_time: 0.0048 (0.00464631003148174)  --> TIME: 2025-02-24 00:42:43 -- STEP: 19424/36938 -- GLOBAL_STEP: 93300 | > loss_text_ce: 0.05011756718158722 (0.06336908680375335) | > loss_mel_ce: 3.178764820098877 (nan) | > loss: 0.012813026085495949 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13645023064568257) | > loader_time: 0.0047 (0.00464654046139963)  --> TIME: 2025-02-24 00:42:52 -- STEP: 19474/36938 -- GLOBAL_STEP: 93350 | > loss_text_ce: 0.06065346673130989 (0.0633577799293109) | > loss_mel_ce: 3.7340898513793945 (nan) | > loss: 0.015058506280183792 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.1364506939249125) | > loader_time: 0.0046 (0.004646487594495605)  --> TIME: 2025-02-24 00:43:01 -- STEP: 19524/36938 -- GLOBAL_STEP: 93400 | > loss_text_ce: 0.046021003276109695 (0.06334415432260337) | > loss_mel_ce: 2.479220151901245 (nan) | > loss: 0.01002079900354147 (nan) | > current_lr: 5e-06 | > step_time: 0.1079 (0.13645376677494656) | > loader_time: 0.0051 (0.004646569240095703)  --> TIME: 2025-02-24 00:43:11 -- STEP: 19574/36938 -- GLOBAL_STEP: 93450 | > loss_text_ce: 0.07057296484708786 (0.06333733422644948) | > loss_mel_ce: 3.4149601459503174 (nan) | > loss: 0.013831481337547302 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.13647454735255252) | > loader_time: 0.0046 (0.004646497678011304)  --> TIME: 2025-02-24 00:43:20 -- STEP: 19624/36938 -- GLOBAL_STEP: 93500 | > loss_text_ce: 0.061531707644462585 (0.06333330322908977) | > loss_mel_ce: 4.021692752838135 (nan) | > loss: 0.016203271225094795 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.13647202121399965) | > loader_time: 0.0046 (0.004646423054480159)  --> TIME: 2025-02-24 00:43:30 -- STEP: 19674/36938 -- GLOBAL_STEP: 93550 | > loss_text_ce: 0.05350491777062416 (0.06332894575516927) | > loss_mel_ce: 3.496009349822998 (nan) | > loss: 0.014085374772548676 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13647049634549066) | > loader_time: 0.0044 (0.0046463808151017355)  --> TIME: 2025-02-24 00:43:39 -- STEP: 19724/36938 -- GLOBAL_STEP: 93600 | > loss_text_ce: 0.05498591437935829 (0.063317500197056) | > loss_mel_ce: 2.992464303970337 (nan) | > loss: 0.012093057855963707 (nan) | > current_lr: 5e-06 | > step_time: 0.1504 (0.13647576510797524) | > loader_time: 0.0045 (0.004646440786223634)  --> TIME: 2025-02-24 00:43:48 -- STEP: 19774/36938 -- GLOBAL_STEP: 93650 | > loss_text_ce: 0.066628597676754 (0.06331211990878677) | > loss_mel_ce: 4.245184898376465 (nan) | > loss: 0.017110371962189674 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.13648334799346365) | > loader_time: 0.0046 (0.004646603386166081)  --> TIME: 2025-02-24 00:43:58 -- STEP: 19824/36938 -- GLOBAL_STEP: 93700 | > loss_text_ce: 0.0712137296795845 (0.06330088101446929) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.1364773113560166) | > loader_time: 0.0045 (0.004646602263365983)  --> TIME: 2025-02-24 00:44:07 -- STEP: 19874/36938 -- GLOBAL_STEP: 93750 | > loss_text_ce: 0.07910888642072678 (0.06329033467208203) | > loss_mel_ce: 0.1625126451253891 (nan) | > loss: 0.0009588156244717538 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13648431121409357) | > loader_time: 0.0047 (0.0046468479623494475)  --> TIME: 2025-02-24 00:44:16 -- STEP: 19924/36938 -- GLOBAL_STEP: 93800 | > loss_text_ce: 0.06582510471343994 (0.06327721906990057) | > loss_mel_ce: 4.263051509857178 (nan) | > loss: 0.017178082838654518 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.136478769171313) | > loader_time: 0.0045 (0.004646511363638555)  --> TIME: 2025-02-24 00:44:25 -- STEP: 19974/36938 -- GLOBAL_STEP: 93850 | > loss_text_ce: 0.05880100280046463 (0.06326717072807803) | > loss_mel_ce: 3.674215793609619 (nan) | > loss: 0.014813559129834175 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1364734842575057) | > loader_time: 0.0045 (0.004646477642462331)  --> TIME: 2025-02-24 00:44:35 -- STEP: 20024/36938 -- GLOBAL_STEP: 93900 | > loss_text_ce: 0.057867858558893204 (0.06325565181956096) | > loss_mel_ce: 3.268427848815918 (nan) | > loss: 0.013199586421251297 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13647437140887408) | > loader_time: 0.0046 (0.0046464429704656555)  --> TIME: 2025-02-24 00:44:44 -- STEP: 20074/36938 -- GLOBAL_STEP: 93950 | > loss_text_ce: 0.067242331802845 (0.06325048913581086) | > loss_mel_ce: 4.117037773132324 (nan) | > loss: 0.016604285687208176 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13647159034065498) | > loader_time: 0.0046 (0.004646424956443842)  --> TIME: 2025-02-24 00:44:53 -- STEP: 20124/36938 -- GLOBAL_STEP: 94000 | > loss_text_ce: 0.07061213254928589 (0.06324126967687109) | > loss_mel_ce: 3.5488312244415283 (nan) | > loss: 0.014362871646881104 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.1364738566930913) | > loader_time: 0.0044 (0.004646618912173113)  --> TIME: 2025-02-24 00:45:02 -- STEP: 20174/36938 -- GLOBAL_STEP: 94050 | > loss_text_ce: 0.06307496875524521 (0.06322931397364284) | > loss_mel_ce: 1.9763213396072388 (nan) | > loss: 0.008092842996120453 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13646667222503106) | > loader_time: 0.0045 (0.0046463688218559614)  --> TIME: 2025-02-24 00:45:12 -- STEP: 20224/36938 -- GLOBAL_STEP: 94100 | > loss_text_ce: 0.0449596531689167 (0.06322290546297987) | > loss_mel_ce: 3.6953125 (nan) | > loss: 0.014842350035905838 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1364868396091508) | > loader_time: 0.0045 (0.0046461726762826215)  --> TIME: 2025-02-24 00:45:22 -- STEP: 20274/36938 -- GLOBAL_STEP: 94150 | > loss_text_ce: 0.05905609577894211 (0.06321433093343687) | > loss_mel_ce: 3.9038705825805664 (nan) | > loss: 0.01572589948773384 (nan) | > current_lr: 5e-06 | > step_time: 0.1499 (0.13649398180769873) | > loader_time: 0.0045 (0.004646022526531353)  --> TIME: 2025-02-24 00:45:31 -- STEP: 20324/36938 -- GLOBAL_STEP: 94200 | > loss_text_ce: 0.0505218580365181 (0.06320294524414498) | > loss_mel_ce: 3.0 (nan) | > loss: 0.0121052460744977 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13648405458204152) | > loader_time: 0.0054 (0.0046461420461808515)  --> TIME: 2025-02-24 00:45:40 -- STEP: 20374/36938 -- GLOBAL_STEP: 94250 | > loss_text_ce: 0.053504232317209244 (0.06319218139734901) | > loss_mel_ce: 3.7481892108917236 (nan) | > loss: 0.01508608553558588 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13647963650341127) | > loader_time: 0.0046 (0.004645938995929053)  --> TIME: 2025-02-24 00:45:49 -- STEP: 20424/36938 -- GLOBAL_STEP: 94300 | > loss_text_ce: 0.06917021423578262 (0.06318227111841987) | > loss_mel_ce: 3.919468879699707 (nan) | > loss: 0.015827933326363564 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.1364762770443219) | > loader_time: 0.0045 (0.0046457817659068335)  --> TIME: 2025-02-24 00:45:58 -- STEP: 20474/36938 -- GLOBAL_STEP: 94350 | > loss_text_ce: 0.06232679635286331 (0.06317200975572108) | > loss_mel_ce: 3.9747910499572754 (nan) | > loss: 0.016020311042666435 (nan) | > current_lr: 5e-06 | > step_time: 0.1609 (0.13647536785676972) | > loader_time: 0.0045 (0.004645600360364722)  --> TIME: 2025-02-24 00:46:08 -- STEP: 20524/36938 -- GLOBAL_STEP: 94400 | > loss_text_ce: 0.062389809638261795 (0.06316316059901812) | > loss_mel_ce: 3.051220417022705 (nan) | > loss: 0.012355596758425236 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13648522770258553) | > loader_time: 0.0045 (0.004645550234743906)  --> TIME: 2025-02-24 00:46:17 -- STEP: 20574/36938 -- GLOBAL_STEP: 94450 | > loss_text_ce: 0.058333415538072586 (0.06315291311723677) | > loss_mel_ce: 1.452056884765625 (nan) | > loss: 0.005993612576276064 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13647545994782537) | > loader_time: 0.0047 (0.0046454737459221815)  --> TIME: 2025-02-24 00:46:26 -- STEP: 20624/36938 -- GLOBAL_STEP: 94500 | > loss_text_ce: 0.064332976937294 (0.0631403135652548) | > loss_mel_ce: 3.4125142097473145 (nan) | > loss: 0.013797013089060783 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13646534603173002) | > loader_time: 0.0045 (0.0046453629819065345)  --> TIME: 2025-02-24 00:46:35 -- STEP: 20674/36938 -- GLOBAL_STEP: 94550 | > loss_text_ce: 0.05003461241722107 (0.0631307991219023) | > loss_mel_ce: 3.36995005607605 (nan) | > loss: 0.013571368530392647 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13646608848826508) | > loader_time: 0.0045 (0.004645257032135565)  --> TIME: 2025-02-24 00:46:45 -- STEP: 20724/36938 -- GLOBAL_STEP: 94600 | > loss_text_ce: 0.06661097705364227 (0.06312220157423384) | > loss_mel_ce: 1.8791245222091675 (nan) | > loss: 0.007721173111349344 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13646764179795665) | > loader_time: 0.0051 (0.004645386468809684)  --> TIME: 2025-02-24 00:46:54 -- STEP: 20774/36938 -- GLOBAL_STEP: 94650 | > loss_text_ce: 0.06606568396091461 (0.06310966408226801) | > loss_mel_ce: 3.8216710090637207 (nan) | > loss: 0.015427527949213982 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13645897949580957) | > loader_time: 0.0054 (0.004645488323462544)  --> TIME: 2025-02-24 00:47:03 -- STEP: 20824/36938 -- GLOBAL_STEP: 94700 | > loss_text_ce: 0.054585326462984085 (0.06310167527150037) | > loss_mel_ce: 3.511744737625122 (nan) | > loss: 0.014152104035019875 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13646327694745702) | > loader_time: 0.0045 (0.004645387587343998)  --> TIME: 2025-02-24 00:47:12 -- STEP: 20874/36938 -- GLOBAL_STEP: 94750 | > loss_text_ce: 0.0580090656876564 (0.06309755529106836) | > loss_mel_ce: 4.2565598487854 (nan) | > loss: 0.01712130568921566 (nan) | > current_lr: 5e-06 | > step_time: 0.2004 (0.13646559449371665) | > loader_time: 0.0048 (0.004645250624163017)  --> TIME: 2025-02-24 00:47:22 -- STEP: 20924/36938 -- GLOBAL_STEP: 94800 | > loss_text_ce: 0.06059028580784798 (0.06308865156517834) | > loss_mel_ce: 0.15949706733226776 (nan) | > loss: 0.000873362529091537 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13646732690890004) | > loader_time: 0.0045 (0.00464539965669872)  --> TIME: 2025-02-24 00:47:31 -- STEP: 20974/36938 -- GLOBAL_STEP: 94850 | > loss_text_ce: 0.05590532347559929 (0.06307857299643216) | > loss_mel_ce: 4.055816650390625 (nan) | > loss: 0.016316358000040054 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.136475187154134) | > loader_time: 0.0045 (0.0046452118919156835)  --> TIME: 2025-02-24 00:47:41 -- STEP: 21024/36938 -- GLOBAL_STEP: 94900 | > loss_text_ce: 0.05836785212159157 (0.06306781510025429) | > loss_mel_ce: 4.249456405639648 (nan) | > loss: 0.01709454134106636 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13648152721073434) | > loader_time: 0.0045 (0.004645441935214861)  --> TIME: 2025-02-24 00:47:50 -- STEP: 21074/36938 -- GLOBAL_STEP: 94950 | > loss_text_ce: 0.05491918325424194 (0.06305735745075045) | > loss_mel_ce: 4.109492301940918 (nan) | > loss: 0.016525443643331528 (nan) | > current_lr: 5e-06 | > step_time: 0.1942 (0.13648120971403546) | > loader_time: 0.0047 (0.004645348047750861)  --> TIME: 2025-02-24 00:47:59 -- STEP: 21124/36938 -- GLOBAL_STEP: 95000 | > loss_text_ce: 0.06542184948921204 (0.06305253352041176) | > loss_mel_ce: 2.456835985183716 (nan) | > loss: 0.010008960030972958 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.13647859216391087) | > loader_time: 0.0051 (0.004645379908810416)  --> TIME: 2025-02-24 00:48:09 -- STEP: 21174/36938 -- GLOBAL_STEP: 95050 | > loss_text_ce: 0.05633595958352089 (0.06304188434434285) | > loss_mel_ce: 4.124913692474365 (nan) | > loss: 0.01659226045012474 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13647920486102197) | > loader_time: 0.0045 (0.004645309423920698)  --> TIME: 2025-02-24 00:48:18 -- STEP: 21224/36938 -- GLOBAL_STEP: 95100 | > loss_text_ce: 0.0591440387070179 (0.06302697836372959) | > loss_mel_ce: 3.6141889095306396 (nan) | > loss: 0.014576719142496586 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.13646819751711164) | > loader_time: 0.0047 (0.0046450916299989355)  --> TIME: 2025-02-24 00:48:27 -- STEP: 21274/36938 -- GLOBAL_STEP: 95150 | > loss_text_ce: 0.05346793681383133 (0.06301868968512143) | > loss_mel_ce: 4.040131568908691 (nan) | > loss: 0.016244443133473396 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.1364641759341139) | > loader_time: 0.0047 (0.004645077359847876)  --> TIME: 2025-02-24 00:48:36 -- STEP: 21324/36938 -- GLOBAL_STEP: 95200 | > loss_text_ce: 0.06743615865707397 (0.06300777864297025) | > loss_mel_ce: 4.147656440734863 (nan) | > loss: 0.016726559028029442 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.1364662669944267) | > loader_time: 0.0045 (0.004645038749012933)  --> TIME: 2025-02-24 00:48:45 -- STEP: 21374/36938 -- GLOBAL_STEP: 95250 | > loss_text_ce: 0.05400992929935455 (0.06299772264105226) | > loss_mel_ce: 4.054427623748779 (nan) | > loss: 0.016303325071930885 (nan) | > current_lr: 5e-06 | > step_time: 0.1435 (0.13646606835902297) | > loader_time: 0.0046 (0.004644800182855508)  --> TIME: 2025-02-24 00:48:55 -- STEP: 21424/36938 -- GLOBAL_STEP: 95300 | > loss_text_ce: 0.05252136290073395 (0.0629850807767518) | > loss_mel_ce: 2.4964091777801514 (nan) | > loss: 0.010114804841578007 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13646977195061416) | > loader_time: 0.0044 (0.004644577965262804)  --> TIME: 2025-02-24 00:49:04 -- STEP: 21474/36938 -- GLOBAL_STEP: 95350 | > loss_text_ce: 0.06438898295164108 (0.06297367399927381) | > loss_mel_ce: 4.415812969207764 (nan) | > loss: 0.017778579145669937 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13646541822441896) | > loader_time: 0.0045 (0.004644473515887474)  --> TIME: 2025-02-24 00:49:13 -- STEP: 21524/36938 -- GLOBAL_STEP: 95400 | > loss_text_ce: 0.06903555989265442 (0.06296816353376833) | > loss_mel_ce: 3.6315901279449463 (nan) | > loss: 0.014685023576021194 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.13646024103347298) | > loader_time: 0.0047 (0.00464444602650135)  --> TIME: 2025-02-24 00:49:22 -- STEP: 21574/36938 -- GLOBAL_STEP: 95450 | > loss_text_ce: 0.05859294906258583 (0.06295631455803673) | > loss_mel_ce: 1.3124858140945435 (nan) | > loss: 0.005440788809210062 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1364568130426512) | > loader_time: 0.0046 (0.004644377653532743)  --> TIME: 2025-02-24 00:49:32 -- STEP: 21624/36938 -- GLOBAL_STEP: 95500 | > loss_text_ce: 0.05419256165623665 (0.06294347921117921) | > loss_mel_ce: 4.060794353485107 (nan) | > loss: 0.016329314559698105 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13645994736007883) | > loader_time: 0.0045 (0.0046444378360483395)  --> TIME: 2025-02-24 00:49:41 -- STEP: 21674/36938 -- GLOBAL_STEP: 95550 | > loss_text_ce: 0.055460356175899506 (0.06293185087594762) | > loss_mel_ce: 3.0335793495178223 (nan) | > loss: 0.012258094735443592 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13647036922256392) | > loader_time: 0.0046 (0.004644702278820845)  --> TIME: 2025-02-24 00:49:51 -- STEP: 21724/36938 -- GLOBAL_STEP: 95600 | > loss_text_ce: 0.06470270454883575 (0.06291872488445861) | > loss_mel_ce: 3.9201548099517822 (nan) | > loss: 0.01581292785704136 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13647179584041438) | > loader_time: 0.0045 (0.004644503878353088)  --> TIME: 2025-02-24 00:49:59 -- STEP: 21774/36938 -- GLOBAL_STEP: 95650 | > loss_text_ce: 0.053105540573596954 (0.06290637760903761) | > loss_mel_ce: 3.759962320327759 (nan) | > loss: 0.015131223015487194 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13645730528214456) | > loader_time: 0.0045 (0.004644337661384399)  --> TIME: 2025-02-24 00:50:09 -- STEP: 21824/36938 -- GLOBAL_STEP: 95700 | > loss_text_ce: 0.0541498176753521 (0.06289701449319131) | > loss_mel_ce: 3.1001064777374268 (nan) | > loss: 0.012516890652477741 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.1364532431855238) | > loader_time: 0.0045 (0.004644229101383128)  --> TIME: 2025-02-24 00:50:18 -- STEP: 21874/36938 -- GLOBAL_STEP: 95750 | > loss_text_ce: 0.04523678496479988 (0.06288705015871923) | > loss_mel_ce: 0.06555730849504471 (nan) | > loss: 0.00043965913937427104 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13644940603299377) | > loader_time: 0.0044 (0.004643966524476982)  --> TIME: 2025-02-24 00:50:27 -- STEP: 21924/36938 -- GLOBAL_STEP: 95800 | > loss_text_ce: 0.06404447555541992 (0.06288021560369199) | > loss_mel_ce: 3.530359983444214 (nan) | > loss: 0.014263510704040527 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13644294791186012) | > loader_time: 0.0044 (0.004644030485899677)  --> TIME: 2025-02-24 00:50:36 -- STEP: 21974/36938 -- GLOBAL_STEP: 95850 | > loss_text_ce: 0.06849689036607742 (0.06287037216534094) | > loss_mel_ce: 2.70263671875 (nan) | > loss: 0.01099656242877245 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13643619107562158) | > loader_time: 0.0047 (0.004643882645742177)  --> TIME: 2025-02-24 00:50:45 -- STEP: 22024/36938 -- GLOBAL_STEP: 95900 | > loss_text_ce: 0.05578748136758804 (0.06285433674339723) | > loss_mel_ce: 4.108653545379639 (nan) | > loss: 0.01652556098997593 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13643330763220365) | > loader_time: 0.0046 (0.004643661506904363)  --> TIME: 2025-02-24 00:50:55 -- STEP: 22074/36938 -- GLOBAL_STEP: 95950 | > loss_text_ce: 0.04976018890738487 (0.06284228648704797) | > loss_mel_ce: 2.4733643531799316 (nan) | > loss: 0.010012399405241013 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13644031880179952) | > loader_time: 0.0045 (0.004643590043957873)  --> TIME: 2025-02-24 00:51:04 -- STEP: 22124/36938 -- GLOBAL_STEP: 96000 | > loss_text_ce: 0.046796057373285294 (0.06283027239376508) | > loss_mel_ce: 3.000805616378784 (nan) | > loss: 0.012093658559024334 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13644848942562712) | > loader_time: 0.0048 (0.004643996861933231)  --> TIME: 2025-02-24 00:51:13 -- STEP: 22174/36938 -- GLOBAL_STEP: 96050 | > loss_text_ce: 0.06891810148954391 (0.06281744708850785) | > loss_mel_ce: 4.173628330230713 (nan) | > loss: 0.01683550328016281 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.1364330992316325) | > loader_time: 0.0046 (0.004644037120946777)  --> TIME: 2025-02-24 00:51:22 -- STEP: 22224/36938 -- GLOBAL_STEP: 96100 | > loss_text_ce: 0.057961124926805496 (0.06280562554845107) | > loss_mel_ce: 4.042271137237549 (nan) | > loss: 0.01627076417207718 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.13643256224600706) | > loader_time: 0.0047 (0.004644024138230951)  --> TIME: 2025-02-24 00:51:31 -- STEP: 22274/36938 -- GLOBAL_STEP: 96150 | > loss_text_ce: 0.054247062653303146 (0.06279095935957094) | > loss_mel_ce: 4.044104099273682 (nan) | > loss: 0.01626329869031906 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13642620302845818) | > loader_time: 0.0052 (0.004644104947818651)  --> TIME: 2025-02-24 00:51:40 -- STEP: 22324/36938 -- GLOBAL_STEP: 96200 | > loss_text_ce: 0.055574800819158554 (0.06277805912491742) | > loss_mel_ce: 3.5444705486297607 (nan) | > loss: 0.01428589504212141 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13641046186519618) | > loader_time: 0.0045 (0.004644090717912425)  --> TIME: 2025-02-24 00:51:50 -- STEP: 22374/36938 -- GLOBAL_STEP: 96250 | > loss_text_ce: 0.03851976618170738 (0.06276605084454831) | > loss_mel_ce: 2.8600454330444336 (nan) | > loss: 0.01150224357843399 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13642550876614382) | > loader_time: 0.0044 (0.004644157164663991)  --> TIME: 2025-02-24 00:51:59 -- STEP: 22424/36938 -- GLOBAL_STEP: 96300 | > loss_text_ce: 0.05805417522788048 (0.06275851306700822) | > loss_mel_ce: 3.837718963623047 (nan) | > loss: 0.015459418296813965 (nan) | > current_lr: 5e-06 | > step_time: 0.1558 (0.1364234973395083) | > loader_time: 0.0045 (0.0046440036093555416)  --> TIME: 2025-02-24 00:52:09 -- STEP: 22474/36938 -- GLOBAL_STEP: 96350 | > loss_text_ce: 0.05256311595439911 (0.06275049566152062) | > loss_mel_ce: 3.988727331161499 (nan) | > loss: 0.0160368662327528 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.13642743426921558) | > loader_time: 0.0047 (0.0046438693448595265)  --> TIME: 2025-02-24 00:52:18 -- STEP: 22524/36938 -- GLOBAL_STEP: 96400 | > loss_text_ce: 0.05528917536139488 (0.06273627467263221) | > loss_mel_ce: 3.9467780590057373 (nan) | > loss: 0.015881219878792763 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13642228589644806) | > loader_time: 0.0047 (0.004643724212805724)  --> TIME: 2025-02-24 00:52:27 -- STEP: 22574/36938 -- GLOBAL_STEP: 96450 | > loss_text_ce: 0.07314803451299667 (0.06272443131186932) | > loss_mel_ce: 3.274592161178589 (nan) | > loss: 0.013284684158861637 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13641422164760214) | > loader_time: 0.0048 (0.00464379142128549)  --> TIME: 2025-02-24 00:52:36 -- STEP: 22624/36938 -- GLOBAL_STEP: 96500 | > loss_text_ce: 0.07183636724948883 (0.06271623494292852) | > loss_mel_ce: 1.562575101852417 (nan) | > loss: 0.00648576021194458 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13641176826967422) | > loader_time: 0.0045 (0.004643899421543548)  --> TIME: 2025-02-24 00:52:45 -- STEP: 22674/36938 -- GLOBAL_STEP: 96550 | > loss_text_ce: 0.045938920229673386 (0.06270589495408403) | > loss_mel_ce: 3.612483263015747 (nan) | > loss: 0.01451754942536354 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13641363947524257) | > loader_time: 0.0046 (0.0046439483555429616)  --> TIME: 2025-02-24 00:52:55 -- STEP: 22724/36938 -- GLOBAL_STEP: 96600 | > loss_text_ce: 0.06410425156354904 (0.06269358594750488) | > loss_mel_ce: 3.346555471420288 (nan) | > loss: 0.013534365221858025 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.1364209148963982) | > loader_time: 0.0045 (0.004643955410754419)  --> TIME: 2025-02-24 00:53:04 -- STEP: 22774/36938 -- GLOBAL_STEP: 96650 | > loss_text_ce: 0.044791169464588165 (0.06268152499503825) | > loss_mel_ce: 3.5495738983154297 (nan) | > loss: 0.014263354241847992 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13642825935432826) | > loader_time: 0.0044 (0.004643804605949205)  --> TIME: 2025-02-24 00:53:14 -- STEP: 22824/36938 -- GLOBAL_STEP: 96700 | > loss_text_ce: 0.06706630438566208 (0.06267254646399678) | > loss_mel_ce: 2.2584280967712402 (nan) | > loss: 0.009228152222931385 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.1364283442831528) | > loader_time: 0.0045 (0.004643744015083667)  --> TIME: 2025-02-24 00:53:23 -- STEP: 22874/36938 -- GLOBAL_STEP: 96750 | > loss_text_ce: 0.058003343641757965 (0.0626601407213524) | > loss_mel_ce: 3.245197296142578 (nan) | > loss: 0.013107940554618835 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13642009263315105) | > loader_time: 0.0044 (0.0046438827708102145)  --> TIME: 2025-02-24 00:53:32 -- STEP: 22924/36938 -- GLOBAL_STEP: 96800 | > loss_text_ce: 0.06425740569829941 (0.06264891435852525) | > loss_mel_ce: 4.445967197418213 (nan) | > loss: 0.017897717654705048 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.1364234096299627) | > loader_time: 0.0046 (0.004643635909150947)  --> TIME: 2025-02-24 00:53:41 -- STEP: 22974/36938 -- GLOBAL_STEP: 96850 | > loss_text_ce: 0.059401001781225204 (0.06264187031994396) | > loss_mel_ce: 3.8220832347869873 (nan) | > loss: 0.01540271658450365 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13640631187509086) | > loader_time: 0.0051 (0.004643508366176112)  --> TIME: 2025-02-24 00:53:50 -- STEP: 23024/36938 -- GLOBAL_STEP: 96900 | > loss_text_ce: 0.051867250353097916 (0.06263313907644472) | > loss_mel_ce: 3.3239946365356445 (nan) | > loss: 0.013396278023719788 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13639802757528996) | > loader_time: 0.0046 (0.004643605629796416)  --> TIME: 2025-02-24 00:54:00 -- STEP: 23074/36938 -- GLOBAL_STEP: 96950 | > loss_text_ce: 0.06181078776717186 (0.0626242629651876) | > loss_mel_ce: 3.757310152053833 (nan) | > loss: 0.015155242756009102 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13641068734318423) | > loader_time: 0.0045 (0.004643432083685425)  --> TIME: 2025-02-24 00:54:09 -- STEP: 23124/36938 -- GLOBAL_STEP: 97000 | > loss_text_ce: 0.057525962591171265 (0.0626145016342071) | > loss_mel_ce: 3.5542430877685547 (nan) | > loss: 0.014332417398691177 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13641696023891592) | > loader_time: 0.0045 (0.004643237997021123)  --> TIME: 2025-02-24 00:54:18 -- STEP: 23174/36938 -- GLOBAL_STEP: 97050 | > loss_text_ce: 0.05458657443523407 (0.06260848420016112) | > loss_mel_ce: 3.1709413528442383 (nan) | > loss: 0.01279971469193697 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.1364128780784753) | > loader_time: 0.0046 (0.004642987020826276)  --> TIME: 2025-02-24 00:54:27 -- STEP: 23224/36938 -- GLOBAL_STEP: 97100 | > loss_text_ce: 0.05097849667072296 (0.06259762948681072) | > loss_mel_ce: 4.5440993309021 (nan) | > loss: 0.018234437331557274 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13640616011340367) | > loader_time: 0.0045 (0.004642809932083307)  --> TIME: 2025-02-24 00:54:36 -- STEP: 23274/36938 -- GLOBAL_STEP: 97150 | > loss_text_ce: 0.05362917482852936 (0.06259061913923503) | > loss_mel_ce: 4.084761619567871 (nan) | > loss: 0.01642218790948391 (nan) | > current_lr: 5e-06 | > step_time: 0.1352 (0.13639654653879832) | > loader_time: 0.0045 (0.004642741903669142)  --> TIME: 2025-02-24 00:54:45 -- STEP: 23324/36938 -- GLOBAL_STEP: 97200 | > loss_text_ce: 0.08869673311710358 (0.06258112462374768) | > loss_mel_ce: 3.021204710006714 (nan) | > loss: 0.012340879067778587 (nan) | > current_lr: 5e-06 | > step_time: 0.107 (0.13638995851298588) | > loader_time: 0.0044 (0.004642623516773666)  --> TIME: 2025-02-24 00:54:55 -- STEP: 23374/36938 -- GLOBAL_STEP: 97250 | > loss_text_ce: 0.06103037670254707 (0.06257402256382598) | > loss_mel_ce: 4.563774585723877 (nan) | > loss: 0.01835240237414837 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.13638293506489832) | > loader_time: 0.0046 (0.004642515775328569)  --> TIME: 2025-02-24 00:55:04 -- STEP: 23424/36938 -- GLOBAL_STEP: 97300 | > loss_text_ce: 0.054632920771837234 (0.06255968592891041) | > loss_mel_ce: 3.597259044647217 (nan) | > loss: 0.014491635374724865 (nan) | > current_lr: 5e-06 | > step_time: 0.1766 (0.136374292182711) | > loader_time: 0.0054 (0.004642443823032749)  --> TIME: 2025-02-24 00:55:13 -- STEP: 23474/36938 -- GLOBAL_STEP: 97350 | > loss_text_ce: 0.05137414485216141 (0.06255173909645156) | > loss_mel_ce: 4.031772613525391 (nan) | > loss: 0.016202963888645172 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13637380464078858) | > loader_time: 0.0048 (0.004642368561467758)  --> TIME: 2025-02-24 00:55:22 -- STEP: 23524/36938 -- GLOBAL_STEP: 97400 | > loss_text_ce: 0.05612550303339958 (0.06253904496572733) | > loss_mel_ce: 0.10427152365446091 (nan) | > loss: 0.0006364961736835539 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13636185091747136) | > loader_time: 0.0046 (0.004642283393501226)  --> TIME: 2025-02-24 00:55:31 -- STEP: 23574/36938 -- GLOBAL_STEP: 97450 | > loss_text_ce: 0.0615120455622673 (0.06252478474762578) | > loss_mel_ce: 4.446801662445068 (nan) | > loss: 0.01789013482630253 (nan) | > current_lr: 5e-06 | > step_time: 0.1648 (0.13637131307429948) | > loader_time: 0.0046 (0.004642483983177869)  --> TIME: 2025-02-24 00:55:41 -- STEP: 23624/36938 -- GLOBAL_STEP: 97500 | > loss_text_ce: 0.051675304770469666 (0.06251585732308379) | > loss_mel_ce: 3.211214303970337 (nan) | > loss: 0.012947975657880306 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.1363769749786912) | > loader_time: 0.0045 (0.0046426114533822576)  --> TIME: 2025-02-24 00:55:50 -- STEP: 23674/36938 -- GLOBAL_STEP: 97550 | > loss_text_ce: 0.04325679689645767 (0.0625047103697838) | > loss_mel_ce: 3.392669200897217 (nan) | > loss: 0.013634627684950829 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.1363614750359628) | > loader_time: 0.0047 (0.004642851914446736)  --> TIME: 2025-02-24 00:55:59 -- STEP: 23724/36938 -- GLOBAL_STEP: 97600 | > loss_text_ce: 0.06383021920919418 (0.062494021613489895) | > loss_mel_ce: 0.3089272677898407 (nan) | > loss: 0.001479196478612721 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13635023799760249) | > loader_time: 0.0045 (0.0046429903124139554)  --> TIME: 2025-02-24 00:56:08 -- STEP: 23774/36938 -- GLOBAL_STEP: 97650 | > loss_text_ce: 0.05337885767221451 (0.062485035540366225) | > loss_mel_ce: 4.550734043121338 (nan) | > loss: 0.01827029138803482 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13636480916526258) | > loader_time: 0.0045 (0.004643345426708895)  --> TIME: 2025-02-24 00:56:18 -- STEP: 23824/36938 -- GLOBAL_STEP: 97700 | > loss_text_ce: 0.06020618602633476 (0.06247279493752955) | > loss_mel_ce: 3.453402519226074 (nan) | > loss: 0.013942892663180828 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13636474756685407) | > loader_time: 0.0045 (0.004643521677575129)  --> TIME: 2025-02-24 00:56:27 -- STEP: 23874/36938 -- GLOBAL_STEP: 97750 | > loss_text_ce: 0.04874911531805992 (0.06246412998628425) | > loss_mel_ce: 1.2717193365097046 (nan) | > loss: 0.0052399542182683945 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.1363595964182186) | > loader_time: 0.0048 (0.0046436958819506)  --> TIME: 2025-02-24 00:56:36 -- STEP: 23924/36938 -- GLOBAL_STEP: 97800 | > loss_text_ce: 0.052720896899700165 (0.06245681085252973) | > loss_mel_ce: 3.739347219467163 (nan) | > loss: 0.01504789013415575 (nan) | > current_lr: 5e-06 | > step_time: 0.1606 (0.13635613177456943) | > loader_time: 0.0049 (0.0046438991455425675)  --> TIME: 2025-02-24 00:56:45 -- STEP: 23974/36938 -- GLOBAL_STEP: 97850 | > loss_text_ce: 0.04886623099446297 (0.06244954419115338) | > loss_mel_ce: 3.625838279724121 (nan) | > loss: 0.014582161791622639 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13636406377744845) | > loader_time: 0.0045 (0.0046440891003484)  --> TIME: 2025-02-24 00:56:55 -- STEP: 24024/36938 -- GLOBAL_STEP: 97900 | > loss_text_ce: 0.055404141545295715 (0.06243644862569351) | > loss_mel_ce: 3.593440055847168 (nan) | > loss: 0.014479541219770908 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13637195831729837) | > loader_time: 0.0055 (0.004644098636749207)  --> TIME: 2025-02-24 00:57:04 -- STEP: 24074/36938 -- GLOBAL_STEP: 97950 | > loss_text_ce: 0.054014623165130615 (0.06242935652298677) | > loss_mel_ce: 2.9865951538085938 (nan) | > loss: 0.01206591259688139 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1363650371914068) | > loader_time: 0.0045 (0.004644054852322745)  --> TIME: 2025-02-24 00:57:13 -- STEP: 24124/36938 -- GLOBAL_STEP: 98000 | > loss_text_ce: 0.050151512026786804 (0.06241788363025134) | > loss_mel_ce: 3.0732977390289307 (nan) | > loss: 0.01239464059472084 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13636535540522138) | > loader_time: 0.0045 (0.00464400344178806)  --> TIME: 2025-02-24 00:57:23 -- STEP: 24174/36938 -- GLOBAL_STEP: 98050 | > loss_text_ce: 0.06428888440132141 (0.06240852803202607) | > loss_mel_ce: 4.390960216522217 (nan) | > loss: 0.01767956092953682 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.13637400333270142) | > loader_time: 0.0048 (0.004643864022933001)  --> TIME: 2025-02-24 00:57:32 -- STEP: 24224/36938 -- GLOBAL_STEP: 98100 | > loss_text_ce: 0.05308660864830017 (0.06240054267757503) | > loss_mel_ce: 2.1057629585266113 (nan) | > loss: 0.00856686383485794 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.13638300402397574) | > loader_time: 0.0046 (0.004643771280700721)  --> TIME: 2025-02-24 00:57:42 -- STEP: 24274/36938 -- GLOBAL_STEP: 98150 | > loss_text_ce: 0.05559384822845459 (0.06238980549722505) | > loss_mel_ce: 3.8306257724761963 (nan) | > loss: 0.015421506948769093 (nan) | > current_lr: 5e-06 | > step_time: 0.1809 (0.13638577242605607) | > loader_time: 0.0048 (0.00464374202671075)  --> TIME: 2025-02-24 00:57:51 -- STEP: 24324/36938 -- GLOBAL_STEP: 98200 | > loss_text_ce: 0.06958818435668945 (0.06238102882272186) | > loss_mel_ce: 1.6856404542922974 (nan) | > loss: 0.006965193431824446 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13640414948056512) | > loader_time: 0.0045 (0.004643823241466156)  --> TIME: 2025-02-24 00:58:00 -- STEP: 24374/36938 -- GLOBAL_STEP: 98250 | > loss_text_ce: 0.05487849563360214 (0.0623681437704973) | > loss_mel_ce: 3.3738036155700684 (nan) | > loss: 0.013605882413685322 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13639481566327374) | > loader_time: 0.0047 (0.004643701828170775)  --> TIME: 2025-02-24 00:58:10 -- STEP: 24424/36938 -- GLOBAL_STEP: 98300 | > loss_text_ce: 0.06571372598409653 (0.06235913334788284) | > loss_mel_ce: 3.655421495437622 (nan) | > loss: 0.014766409993171692 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.13640883605948012) | > loader_time: 0.0048 (0.004643965003688096)  --> TIME: 2025-02-24 00:58:19 -- STEP: 24474/36938 -- GLOBAL_STEP: 98350 | > loss_text_ce: 0.052236564457416534 (0.06235128814812479) | > loss_mel_ce: 3.720090866088867 (nan) | > loss: 0.014969553798437119 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13640808598360282) | > loader_time: 0.0053 (0.004643970419595521)  --> TIME: 2025-02-24 00:58:29 -- STEP: 24524/36938 -- GLOBAL_STEP: 98400 | > loss_text_ce: 0.056597232818603516 (0.06234162222678105) | > loss_mel_ce: 3.585139751434326 (nan) | > loss: 0.014451337978243828 (nan) | > current_lr: 5e-06 | > step_time: 0.1753 (0.13640832236026668) | > loader_time: 0.005 (0.004644094517172337)  --> TIME: 2025-02-24 00:58:38 -- STEP: 24574/36938 -- GLOBAL_STEP: 98450 | > loss_text_ce: 0.06742320209741592 (0.06233214859381371) | > loss_mel_ce: 3.9564907550811768 (nan) | > loss: 0.01596791297197342 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.13640338423545734) | > loader_time: 0.0045 (0.004643988054355838)  --> TIME: 2025-02-24 00:58:47 -- STEP: 24624/36938 -- GLOBAL_STEP: 98500 | > loss_text_ce: 0.05419723317027092 (0.0623237380160791) | > loss_mel_ce: 2.516211986541748 (nan) | > loss: 0.010200037620961666 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13639704305061395) | > loader_time: 0.0046 (0.0046439246069386415)  --> TIME: 2025-02-24 00:58:56 -- STEP: 24674/36938 -- GLOBAL_STEP: 98550 | > loss_text_ce: 0.0540757030248642 (0.06231414347638151) | > loss_mel_ce: 4.788473606109619 (nan) | > loss: 0.019216466695070267 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.13638663275504517) | > loader_time: 0.0047 (0.0046437156058367225)  --> TIME: 2025-02-24 00:59:05 -- STEP: 24724/36938 -- GLOBAL_STEP: 98600 | > loss_text_ce: 0.05535656958818436 (0.062307182108342914) | > loss_mel_ce: 4.032407760620117 (nan) | > loss: 0.016221288591623306 (nan) | > current_lr: 5e-06 | > step_time: 0.1908 (0.13639087926568108) | > loader_time: 0.005 (0.004643620140555279)  --> TIME: 2025-02-24 00:59:15 -- STEP: 24774/36938 -- GLOBAL_STEP: 98650 | > loss_text_ce: 0.04884914681315422 (0.062297003784588315) | > loss_mel_ce: 2.656183958053589 (nan) | > loss: 0.010734259150922298 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13639055153374127) | > loader_time: 0.0052 (0.004643574584393736)  --> TIME: 2025-02-24 00:59:24 -- STEP: 24824/36938 -- GLOBAL_STEP: 98700 | > loss_text_ce: 0.056376855820417404 (0.062288267730488116) | > loss_mel_ce: 4.202279090881348 (nan) | > loss: 0.016899429261684418 (nan) | > current_lr: 5e-06 | > step_time: 0.1648 (0.13639141613923103) | > loader_time: 0.0047 (0.004643629942253854)  --> TIME: 2025-02-24 00:59:33 -- STEP: 24874/36938 -- GLOBAL_STEP: 98750 | > loss_text_ce: 0.07715337723493576 (0.062276157551486515) | > loss_mel_ce: 2.9475011825561523 (nan) | > loss: 0.012002598494291306 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.1363913037629915) | > loader_time: 0.0044 (0.004643623848249807)  --> TIME: 2025-02-24 00:59:42 -- STEP: 24924/36938 -- GLOBAL_STEP: 98800 | > loss_text_ce: 0.054206036031246185 (0.06226706157627161) | > loss_mel_ce: 1.4064747095108032 (nan) | > loss: 0.005796352401375771 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.1363923670085818) | > loader_time: 0.0045 (0.004643658490839924)  --> TIME: 2025-02-24 00:59:52 -- STEP: 24974/36938 -- GLOBAL_STEP: 98850 | > loss_text_ce: 0.051403168588876724 (0.06225917580327624) | > loss_mel_ce: 3.5495362281799316 (nan) | > loss: 0.014289443381130695 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13638605023820446) | > loader_time: 0.0045 (0.0046437896070330486)  --> TIME: 2025-02-24 01:00:01 -- STEP: 25024/36938 -- GLOBAL_STEP: 98900 | > loss_text_ce: 0.044609878212213516 (0.06225012177753232) | > loss_mel_ce: 3.591421604156494 (nan) | > loss: 0.014428696595132351 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13638763704225257) | > loader_time: 0.0045 (0.00464367963697605)  --> TIME: 2025-02-24 01:00:10 -- STEP: 25074/36938 -- GLOBAL_STEP: 98950 | > loss_text_ce: 0.055759601294994354 (0.062242397404846904) | > loss_mel_ce: 3.3047471046447754 (nan) | > loss: 0.013335345312952995 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13638500326927516) | > loader_time: 0.0046 (0.004643653001455958)  --> TIME: 2025-02-24 01:00:19 -- STEP: 25124/36938 -- GLOBAL_STEP: 99000 | > loss_text_ce: 0.07644112408161163 (0.06223288462911655) | > loss_mel_ce: 2.9809978008270264 (nan) | > loss: 0.012132694944739342 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.136385969315872) | > loader_time: 0.0045 (0.004643806661889218)  --> TIME: 2025-02-24 01:00:28 -- STEP: 25174/36938 -- GLOBAL_STEP: 99050 | > loss_text_ce: 0.07322646677494049 (0.06222586355046169) | > loss_mel_ce: 7.390625 (nan) | > loss: 0.029618460685014725 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13636743982591928) | > loader_time: 0.0045 (0.004643895414489915)  --> TIME: 2025-02-24 01:00:37 -- STEP: 25224/36938 -- GLOBAL_STEP: 99100 | > loss_text_ce: 0.07244057953357697 (0.06221812638083962) | > loss_mel_ce: 3.857133150100708 (nan) | > loss: 0.015593547374010086 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13636452365103055) | > loader_time: 0.0045 (0.0046437677413065075)  --> TIME: 2025-02-24 01:00:47 -- STEP: 25274/36938 -- GLOBAL_STEP: 99150 | > loss_text_ce: 0.056061986833810806 (0.06221098736971441) | > loss_mel_ce: 3.422685146331787 (nan) | > loss: 0.01380455307662487 (nan) | > current_lr: 5e-06 | > step_time: 0.1873 (0.1363693079050748) | > loader_time: 0.0049 (0.004643694871662692)  --> TIME: 2025-02-24 01:00:56 -- STEP: 25324/36938 -- GLOBAL_STEP: 99200 | > loss_text_ce: 0.053490445017814636 (0.06220344071105921) | > loss_mel_ce: 3.4067330360412598 (nan) | > loss: 0.013731046579778194 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.13636335947562161) | > loader_time: 0.0048 (0.004643538639907401)  --> TIME: 2025-02-24 01:01:05 -- STEP: 25374/36938 -- GLOBAL_STEP: 99250 | > loss_text_ce: 0.05906571075320244 (0.06219512138677431) | > loss_mel_ce: 3.0257785320281982 (nan) | > loss: 0.012241446413099766 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13634866623109115) | > loader_time: 0.0045 (0.004643364109364997)  --> TIME: 2025-02-24 01:01:14 -- STEP: 25424/36938 -- GLOBAL_STEP: 99300 | > loss_text_ce: 0.0676041692495346 (0.06218738857184246) | > loss_mel_ce: 3.269179105758667 (nan) | > loss: 0.013241203501820564 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.13634919557577416) | > loader_time: 0.0047 (0.0046432347624763125)  --> TIME: 2025-02-24 01:01:23 -- STEP: 25474/36938 -- GLOBAL_STEP: 99350 | > loss_text_ce: 0.06382837146520615 (0.06218046543592747) | > loss_mel_ce: 4.179758071899414 (nan) | > loss: 0.016839630901813507 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.13634273365152838) | > loader_time: 0.0048 (0.0046432713207387125)  --> TIME: 2025-02-24 01:01:32 -- STEP: 25524/36938 -- GLOBAL_STEP: 99400 | > loss_text_ce: 0.05540960654616356 (0.06217174372105759) | > loss_mel_ce: 3.6893084049224854 (nan) | > loss: 0.014859993010759354 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13633824234527409) | > loader_time: 0.0047 (0.0046432279266360385)  --> TIME: 2025-02-24 01:01:41 -- STEP: 25574/36938 -- GLOBAL_STEP: 99450 | > loss_text_ce: 0.06578182429075241 (0.062162557395269526) | > loss_mel_ce: 0.25390625 (nan) | > loss: 0.0012686036061495543 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13633444950263074) | > loader_time: 0.0045 (0.0046432665461454805)  --> TIME: 2025-02-24 01:01:51 -- STEP: 25624/36938 -- GLOBAL_STEP: 99500 | > loss_text_ce: 0.061538126319646835 (0.06215458101416394) | > loss_mel_ce: 3.964852809906006 (nan) | > loss: 0.01597774401307106 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13633637429400217) | > loader_time: 0.0045 (0.004643223032964808)  --> TIME: 2025-02-24 01:02:00 -- STEP: 25674/36938 -- GLOBAL_STEP: 99550 | > loss_text_ce: 0.04642676189541817 (0.06214335898938971) | > loss_mel_ce: 3.06573486328125 (nan) | > loss: 0.012349848635494709 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13633511281323638) | > loader_time: 0.0044 (0.0046430155896060145)  --> TIME: 2025-02-24 01:02:10 -- STEP: 25724/36938 -- GLOBAL_STEP: 99600 | > loss_text_ce: 0.04062756523489952 (0.06213534595246329) | > loss_mel_ce: 3.1969852447509766 (nan) | > loss: 0.01284767035394907 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.13634116480758657) | > loader_time: 0.005 (0.004642963363082228)  --> TIME: 2025-02-24 01:02:19 -- STEP: 25774/36938 -- GLOBAL_STEP: 99650 | > loss_text_ce: 0.05814094841480255 (0.06212369849640945) | > loss_mel_ce: 3.612473249435425 (nan) | > loss: 0.01456593070179224 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.13633915296100027) | > loader_time: 0.0046 (0.004643035256909876)  --> TIME: 2025-02-24 01:02:28 -- STEP: 25824/36938 -- GLOBAL_STEP: 99700 | > loss_text_ce: 0.05826263129711151 (0.06211378019536134) | > loss_mel_ce: 2.2491109371185303 (nan) | > loss: 0.009156244806945324 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.13634489790583754) | > loader_time: 0.0046 (0.00464308154775018)  --> TIME: 2025-02-24 01:02:38 -- STEP: 25874/36938 -- GLOBAL_STEP: 99750 | > loss_text_ce: 0.0713227167725563 (0.062108466853930844) | > loss_mel_ce: 3.8627634048461914 (nan) | > loss: 0.015611453913152218 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13635099762566155) | > loader_time: 0.0045 (0.0046430334864623855)  --> TIME: 2025-02-24 01:02:47 -- STEP: 25924/36938 -- GLOBAL_STEP: 99800 | > loss_text_ce: 0.06718946993350983 (0.062101264254449014) | > loss_mel_ce: 3.2637596130371094 (nan) | > loss: 0.013218052685260773 (nan) | > current_lr: 5e-06 | > step_time: 0.1713 (0.13635268818907187) | > loader_time: 0.0047 (0.0046428976704945765)  --> TIME: 2025-02-24 01:02:56 -- STEP: 25974/36938 -- GLOBAL_STEP: 99850 | > loss_text_ce: 0.0410500094294548 (0.06209380663239999) | > loss_mel_ce: 3.5761327743530273 (nan) | > loss: 0.014353900216519833 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13635327989127763) | > loader_time: 0.0051 (0.004643035080027782)  --> TIME: 2025-02-24 01:03:05 -- STEP: 26024/36938 -- GLOBAL_STEP: 99900 | > loss_text_ce: 0.0516386441886425 (0.0620861228708889) | > loss_mel_ce: 2.6584529876708984 (nan) | > loss: 0.010754331946372986 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.1363403525328286) | > loader_time: 0.0045 (0.004643179739653993)  --> TIME: 2025-02-24 01:03:15 -- STEP: 26074/36938 -- GLOBAL_STEP: 99950 | > loss_text_ce: 0.0385281965136528 (0.062076981325075296) | > loss_mel_ce: 4.44409704208374 (nan) | > loss: 0.017788195982575417 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.1363421554368647) | > loader_time: 0.0045 (0.004643383298246176)  --> TIME: 2025-02-24 01:03:24 -- STEP: 26124/36938 -- GLOBAL_STEP: 100000 | > loss_text_ce: 0.06141091138124466 (0.06206996714468915) | > loss_mel_ce: 4.220106601715088 (nan) | > loss: 0.01699014939367771 (nan) | > current_lr: 5e-06 | > step_time: 0.1523 (0.13635302366624055) | > loader_time: 0.0046 (0.004643420989837258) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_100000.pth  --> TIME: 2025-02-24 01:03:39 -- STEP: 26174/36938 -- GLOBAL_STEP: 100050 | > loss_text_ce: 0.05568677932024002 (0.062061594741517484) | > loss_mel_ce: 4.235869407653809 (nan) | > loss: 0.017029985785484314 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.13635841490450445) | > loader_time: 0.0048 (0.004643532347531632)  --> TIME: 2025-02-24 01:03:48 -- STEP: 26224/36938 -- GLOBAL_STEP: 100100 | > loss_text_ce: 0.0530116967856884 (0.06205343059917516) | > loss_mel_ce: 1.4030303955078125 (nan) | > loss: 0.005777944810688496 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13635733421798701) | > loader_time: 0.0044 (0.0046433813510834854)  --> TIME: 2025-02-24 01:03:58 -- STEP: 26274/36938 -- GLOBAL_STEP: 100150 | > loss_text_ce: 0.053010955452919006 (0.06204229187654984) | > loss_mel_ce: 4.264078140258789 (nan) | > loss: 0.01713130623102188 (nan) | > current_lr: 5e-06 | > step_time: 0.1821 (0.13636410567571128) | > loader_time: 0.0051 (0.004643698837365702)  --> TIME: 2025-02-24 01:04:08 -- STEP: 26324/36938 -- GLOBAL_STEP: 100200 | > loss_text_ce: 0.060930464416742325 (0.06203243818587988) | > loss_mel_ce: 4.045430660247803 (nan) | > loss: 0.016295084729790688 (nan) | > current_lr: 5e-06 | > step_time: 0.1591 (0.13637417211591957) | > loader_time: 0.0048 (0.004643889994013423)  --> TIME: 2025-02-24 01:04:17 -- STEP: 26374/36938 -- GLOBAL_STEP: 100250 | > loss_text_ce: 0.057610996067523956 (0.062024780773057946) | > loss_mel_ce: 4.229506969451904 (nan) | > loss: 0.017012374475598335 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.1363771079551493) | > loader_time: 0.0046 (0.004643887026040455)  --> TIME: 2025-02-24 01:04:26 -- STEP: 26424/36938 -- GLOBAL_STEP: 100300 | > loss_text_ce: 0.06487129628658295 (0.06201657256979316) | > loss_mel_ce: 3.6920857429504395 (nan) | > loss: 0.014908560551702976 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.1363819433799697) | > loader_time: 0.0047 (0.004643839433487044)  --> TIME: 2025-02-24 01:04:36 -- STEP: 26474/36938 -- GLOBAL_STEP: 100350 | > loss_text_ce: 0.03981369733810425 (0.06200556819074314) | > loss_mel_ce: 3.943114757537842 (nan) | > loss: 0.01580527238547802 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.1363909943924932) | > loader_time: 0.0046 (0.004643980304199052)  --> TIME: 2025-02-24 01:04:45 -- STEP: 26524/36938 -- GLOBAL_STEP: 100400 | > loss_text_ce: 0.06496534496545792 (0.061999898447983544) | > loss_mel_ce: 3.8614048957824707 (nan) | > loss: 0.015580834820866585 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13637802203915741) | > loader_time: 0.0045 (0.0046439318792832395)  --> TIME: 2025-02-24 01:04:54 -- STEP: 26574/36938 -- GLOBAL_STEP: 100450 | > loss_text_ce: 0.06510768830776215 (0.061989863661783134) | > loss_mel_ce: 3.869868516921997 (nan) | > loss: 0.01561498548835516 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1363683540090581) | > loader_time: 0.0045 (0.004644211316347412)  --> TIME: 2025-02-24 01:05:03 -- STEP: 26624/36938 -- GLOBAL_STEP: 100500 | > loss_text_ce: 0.05960720032453537 (0.061979180867987343) | > loss_mel_ce: 1.8972656726837158 (nan) | > loss: 0.0077653685584664345 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13637382359817005) | > loader_time: 0.0047 (0.0046444780444010545)  --> TIME: 2025-02-24 01:05:12 -- STEP: 26674/36938 -- GLOBAL_STEP: 100550 | > loss_text_ce: 0.059475988149642944 (0.06197055877647379) | > loss_mel_ce: 3.281087875366211 (nan) | > loss: 0.013256206177175045 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13637069645086672) | > loader_time: 0.0045 (0.004644594396654311)  --> TIME: 2025-02-24 01:05:22 -- STEP: 26724/36938 -- GLOBAL_STEP: 100600 | > loss_text_ce: 0.06317138671875 (0.061960175812487424) | > loss_mel_ce: 3.713074207305908 (nan) | > loss: 0.014985102228820324 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13637507812042884) | > loader_time: 0.0046 (0.00464467723254338)  --> TIME: 2025-02-24 01:05:31 -- STEP: 26774/36938 -- GLOBAL_STEP: 100650 | > loss_text_ce: 0.06370226293802261 (0.061950821878420284) | > loss_mel_ce: 2.9598796367645264 (nan) | > loss: 0.011998342350125313 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13637887569559082) | > loader_time: 0.0045 (0.004644919066893315)  --> TIME: 2025-02-24 01:05:41 -- STEP: 26824/36938 -- GLOBAL_STEP: 100700 | > loss_text_ce: 0.053474362939596176 (0.06194040527818017) | > loss_mel_ce: 4.214419841766357 (nan) | > loss: 0.01693608984351158 (nan) | > current_lr: 5e-06 | > step_time: 0.1796 (0.1363831920354282) | > loader_time: 0.005 (0.004645016187701461)  --> TIME: 2025-02-24 01:05:50 -- STEP: 26874/36938 -- GLOBAL_STEP: 100750 | > loss_text_ce: 0.06145549938082695 (0.061928497315480474) | > loss_mel_ce: 4.07803201675415 (nan) | > loss: 0.016426539048552513 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13638994904370658) | > loader_time: 0.0053 (0.004645238552915436)  --> TIME: 2025-02-24 01:05:59 -- STEP: 26924/36938 -- GLOBAL_STEP: 100800 | > loss_text_ce: 0.05994943529367447 (0.061919049376599325) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.13638909266689572) | > loader_time: 0.0046 (0.004645240004004974)  --> TIME: 2025-02-24 01:06:08 -- STEP: 26974/36938 -- GLOBAL_STEP: 100850 | > loss_text_ce: 0.059898246079683304 (0.061908914310612684) | > loss_mel_ce: 3.3033254146575928 (nan) | > loss: 0.013346126303076744 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13638316596810304) | > loader_time: 0.0046 (0.004645241449714922)  --> TIME: 2025-02-24 01:06:17 -- STEP: 27024/36938 -- GLOBAL_STEP: 100900 | > loss_text_ce: 0.07358171045780182 (0.06190042488790802) | > loss_mel_ce: 3.261301040649414 (nan) | > loss: 0.013233662582933903 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.1363708725813364) | > loader_time: 0.0046 (0.004645187237899414)  --> TIME: 2025-02-24 01:06:26 -- STEP: 27074/36938 -- GLOBAL_STEP: 100950 | > loss_text_ce: 0.04938488453626633 (0.061888677828756816) | > loss_mel_ce: 3.8407065868377686 (nan) | > loss: 0.015436871908605099 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13636024553754433) | > loader_time: 0.0048 (0.004645138932725731)  --> TIME: 2025-02-24 01:06:35 -- STEP: 27124/36938 -- GLOBAL_STEP: 101000 | > loss_text_ce: 0.05505284667015076 (0.06187878093257637) | > loss_mel_ce: 0.02764892578125 (nan) | > loss: 0.00032818166073411703 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13635994345191396) | > loader_time: 0.0045 (0.004645095183036848)  --> TIME: 2025-02-24 01:06:45 -- STEP: 27174/36938 -- GLOBAL_STEP: 101050 | > loss_text_ce: 0.061141666024923325 (0.061872484142845276) | > loss_mel_ce: 3.3076589107513428 (nan) | > loss: 0.013368257321417332 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13636534267715236) | > loader_time: 0.0047 (0.004645156870880148)  --> TIME: 2025-02-24 01:06:54 -- STEP: 27224/36938 -- GLOBAL_STEP: 101100 | > loss_text_ce: 0.032681334763765335 (0.06186118761207181) | > loss_mel_ce: 0.0274658203125 (nan) | > loss: 0.00023867920390330255 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13636481074455892) | > loader_time: 0.0051 (0.004645334852817287)  --> TIME: 2025-02-24 01:07:04 -- STEP: 27274/36938 -- GLOBAL_STEP: 101150 | > loss_text_ce: 0.05795951187610626 (0.06185429405699319) | > loss_mel_ce: 4.1179094314575195 (nan) | > loss: 0.016570908948779106 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.13636855233962997) | > loader_time: 0.0046 (0.004645241271053516)  --> TIME: 2025-02-24 01:07:13 -- STEP: 27324/36938 -- GLOBAL_STEP: 101200 | > loss_text_ce: 0.05835149809718132 (0.061846400006717074) | > loss_mel_ce: 3.447237968444824 (nan) | > loss: 0.013911070302128792 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13637514701438927) | > loader_time: 0.0046 (0.004645293662216899)  --> TIME: 2025-02-24 01:07:23 -- STEP: 27374/36938 -- GLOBAL_STEP: 101250 | > loss_text_ce: 0.06844338029623032 (0.061841772342973916) | > loss_mel_ce: 4.178439140319824 (nan) | > loss: 0.016852708533406258 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13639357677550834) | > loader_time: 0.0052 (0.0046454196851754195)  --> TIME: 2025-02-24 01:07:33 -- STEP: 27424/36938 -- GLOBAL_STEP: 101300 | > loss_text_ce: 0.06401819735765457 (0.06183441684015092) | > loss_mel_ce: 4.244524955749512 (nan) | > loss: 0.017097394913434982 (nan) | > current_lr: 5e-06 | > step_time: 0.1972 (0.13640101846033634) | > loader_time: 0.0048 (0.004645753334533991)  --> TIME: 2025-02-24 01:07:42 -- STEP: 27474/36938 -- GLOBAL_STEP: 101350 | > loss_text_ce: 0.05015034228563309 (0.061827367277949664) | > loss_mel_ce: 3.8286588191986084 (nan) | > loss: 0.015392101369798183 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13640523261444898) | > loader_time: 0.0045 (0.00464565014400073)  --> TIME: 2025-02-24 01:07:51 -- STEP: 27524/36938 -- GLOBAL_STEP: 101400 | > loss_text_ce: 0.05553482472896576 (0.06181598493450063) | > loss_mel_ce: 3.99371337890625 (nan) | > loss: 0.01606844551861286 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13639985504165594) | > loader_time: 0.0048 (0.004645481556242583)  --> TIME: 2025-02-24 01:08:00 -- STEP: 27574/36938 -- GLOBAL_STEP: 101450 | > loss_text_ce: 0.05400052294135094 (0.06180710262473187) | > loss_mel_ce: 2.3861494064331055 (nan) | > loss: 0.009683135896921158 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.13639477549223825) | > loader_time: 0.0045 (0.00464533246384066)  --> TIME: 2025-02-24 01:08:10 -- STEP: 27624/36938 -- GLOBAL_STEP: 101500 | > loss_text_ce: 0.05035252869129181 (0.06179713386281087) | > loss_mel_ce: 3.309399366378784 (nan) | > loss: 0.013332350179553032 (nan) | > current_lr: 5e-06 | > step_time: 0.1504 (0.1364010613343693) | > loader_time: 0.0053 (0.004645344997329287)  --> TIME: 2025-02-24 01:08:19 -- STEP: 27674/36938 -- GLOBAL_STEP: 101550 | > loss_text_ce: 0.04499687999486923 (0.06178604270364949) | > loss_mel_ce: 4.131100177764893 (nan) | > loss: 0.016571814194321632 (nan) | > current_lr: 5e-06 | > step_time: 0.1807 (0.13640432023688442) | > loader_time: 0.0055 (0.00464538650170848)  --> TIME: 2025-02-24 01:08:28 -- STEP: 27724/36938 -- GLOBAL_STEP: 101600 | > loss_text_ce: 0.05885414779186249 (0.06177727917477676) | > loss_mel_ce: 4.274158000946045 (nan) | > loss: 0.017194492742419243 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13639559621608607) | > loader_time: 0.0048 (0.004645425371063605)  --> TIME: 2025-02-24 01:08:38 -- STEP: 27774/36938 -- GLOBAL_STEP: 101650 | > loss_text_ce: 0.06802228093147278 (0.06176804734726932) | > loss_mel_ce: 3.3909287452697754 (nan) | > loss: 0.01372599694877863 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13640790767852304) | > loader_time: 0.0045 (0.004645523117094608)  --> TIME: 2025-02-24 01:08:47 -- STEP: 27824/36938 -- GLOBAL_STEP: 101700 | > loss_text_ce: 0.05196063965559006 (0.06175983099949918) | > loss_mel_ce: 3.7572429180145264 (nan) | > loss: 0.015115887857973576 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13640991677629138) | > loader_time: 0.0046 (0.004645445339634535)  --> TIME: 2025-02-24 01:08:57 -- STEP: 27874/36938 -- GLOBAL_STEP: 101750 | > loss_text_ce: 0.06876018643379211 (0.0617483815980421) | > loss_mel_ce: 4.449403762817383 (nan) | > loss: 0.01792922429740429 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.1364177592371883) | > loader_time: 0.0047 (0.0046454424870752955)  --> TIME: 2025-02-24 01:09:06 -- STEP: 27924/36938 -- GLOBAL_STEP: 101800 | > loss_text_ce: 0.06434457749128342 (0.06173848918176267) | > loss_mel_ce: 2.1691629886627197 (nan) | > loss: 0.008863125927746296 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13641181340140354) | > loader_time: 0.0045 (0.004645598513607779)  --> TIME: 2025-02-24 01:09:15 -- STEP: 27974/36938 -- GLOBAL_STEP: 101850 | > loss_text_ce: 0.06695498526096344 (0.06173033116850972) | > loss_mel_ce: 4.550502300262451 (nan) | > loss: 0.018323244526982307 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13640659503346253) | > loader_time: 0.0047 (0.004645536990284068)  --> TIME: 2025-02-24 01:09:24 -- STEP: 28024/36938 -- GLOBAL_STEP: 101900 | > loss_text_ce: 0.06884870678186417 (0.06172159497316065) | > loss_mel_ce: 4.113092422485352 (nan) | > loss: 0.016595004126429558 (nan) | > current_lr: 5e-06 | > step_time: 0.169 (0.13640940852107986) | > loader_time: 0.0047 (0.004645410058432423)  --> TIME: 2025-02-24 01:09:34 -- STEP: 28074/36938 -- GLOBAL_STEP: 101950 | > loss_text_ce: 0.04893437772989273 (0.06171455193372483) | > loss_mel_ce: 3.236510753631592 (nan) | > loss: 0.013037481345236301 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.13641488322795808) | > loader_time: 0.0046 (0.004645558404660976)  --> TIME: 2025-02-24 01:09:43 -- STEP: 28124/36938 -- GLOBAL_STEP: 102000 | > loss_text_ce: 0.05728908255696297 (0.06170789290631696) | > loss_mel_ce: 4.088825225830078 (nan) | > loss: 0.016452835872769356 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.1364225265295123) | > loader_time: 0.0045 (0.004645615752535844)  --> TIME: 2025-02-24 01:09:53 -- STEP: 28174/36938 -- GLOBAL_STEP: 102050 | > loss_text_ce: 0.04490061476826668 (0.061700369503982846) | > loss_mel_ce: 4.0019145011901855 (nan) | > loss: 0.01605878956615925 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13641711507398957) | > loader_time: 0.0045 (0.004645537761407477)  --> TIME: 2025-02-24 01:10:02 -- STEP: 28224/36938 -- GLOBAL_STEP: 102100 | > loss_text_ce: 0.0681304931640625 (0.061691562934063145) | > loss_mel_ce: 4.201909065246582 (nan) | > loss: 0.016944602131843567 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13640538886898954) | > loader_time: 0.0046 (0.004645317201576595)  --> TIME: 2025-02-24 01:10:11 -- STEP: 28274/36938 -- GLOBAL_STEP: 102150 | > loss_text_ce: 0.047778837382793427 (0.0616828065807327) | > loss_mel_ce: 1.7805664539337158 (nan) | > loss: 0.007255339063704014 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.1363981470508868) | > loader_time: 0.0045 (0.004645289040407897)  --> TIME: 2025-02-24 01:10:20 -- STEP: 28324/36938 -- GLOBAL_STEP: 102200 | > loss_text_ce: 0.04870900139212608 (0.061672480885347054) | > loss_mel_ce: 3.882924795150757 (nan) | > loss: 0.015601721592247486 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13640160140403926) | > loader_time: 0.0049 (0.004645410053407847)  --> TIME: 2025-02-24 01:10:29 -- STEP: 28374/36938 -- GLOBAL_STEP: 102250 | > loss_text_ce: 0.059680547565221786 (0.0616632509352849) | > loss_mel_ce: 4.923551559448242 (nan) | > loss: 0.019774731248617172 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13639844278120525) | > loader_time: 0.0053 (0.004645528472015274)  --> TIME: 2025-02-24 01:10:38 -- STEP: 28424/36938 -- GLOBAL_STEP: 102300 | > loss_text_ce: 0.058255866169929504 (0.06165490440098685) | > loss_mel_ce: 0.7864183783531189 (nan) | > loss: 0.0033518821001052856 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1363945587852985) | > loader_time: 0.0047 (0.004645514422794404)  --> TIME: 2025-02-24 01:10:47 -- STEP: 28474/36938 -- GLOBAL_STEP: 102350 | > loss_text_ce: 0.04494641348719597 (0.061647484467506244) | > loss_mel_ce: 3.4178054332733154 (nan) | > loss: 0.013741079717874527 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13639144800499164) | > loader_time: 0.0046 (0.00464545294685367)  --> TIME: 2025-02-24 01:10:56 -- STEP: 28524/36938 -- GLOBAL_STEP: 102400 | > loss_text_ce: 0.05890534445643425 (0.06163908933145494) | > loss_mel_ce: 3.8687126636505127 (nan) | > loss: 0.015585786662995815 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.13638188404816842) | > loader_time: 0.0046 (0.0046454607027808965)  --> TIME: 2025-02-24 01:11:06 -- STEP: 28574/36938 -- GLOBAL_STEP: 102450 | > loss_text_ce: 0.05927973985671997 (0.06162790967274142) | > loss_mel_ce: 0.5705464482307434 (nan) | > loss: 0.0024993102997541428 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1363902332367712) | > loader_time: 0.0045 (0.00464536265595317)  --> TIME: 2025-02-24 01:11:15 -- STEP: 28624/36938 -- GLOBAL_STEP: 102500 | > loss_text_ce: 0.05321868881583214 (0.061621019243920105) | > loss_mel_ce: 2.8801512718200684 (nan) | > loss: 0.011640356853604317 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13638938302897752) | > loader_time: 0.0045 (0.004645325122697909)  --> TIME: 2025-02-24 01:11:24 -- STEP: 28674/36938 -- GLOBAL_STEP: 102550 | > loss_text_ce: 0.06076625734567642 (0.06161353822719264) | > loss_mel_ce: 3.693547487258911 (nan) | > loss: 0.014898071065545082 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13638299475356686) | > loader_time: 0.0046 (0.004645307484619046)  --> TIME: 2025-02-24 01:11:34 -- STEP: 28724/36938 -- GLOBAL_STEP: 102600 | > loss_text_ce: 0.056449469178915024 (0.061602094658771624) | > loss_mel_ce: 3.131098508834839 (nan) | > loss: 0.012649000622332096 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13638782357624998) | > loader_time: 0.0048 (0.004645280013956336)  --> TIME: 2025-02-24 01:11:43 -- STEP: 28774/36938 -- GLOBAL_STEP: 102650 | > loss_text_ce: 0.0504583902657032 (0.06159386775914816) | > loss_mel_ce: 4.165103912353516 (nan) | > loss: 0.016728423535823822 (nan) | > current_lr: 5e-06 | > step_time: 0.1707 (0.13639348689416161) | > loader_time: 0.0056 (0.0046454871878263615)  --> TIME: 2025-02-24 01:11:53 -- STEP: 28824/36938 -- GLOBAL_STEP: 102700 | > loss_text_ce: 0.051592811942100525 (0.061583531643189714) | > loss_mel_ce: 3.7935545444488525 (nan) | > loss: 0.015258521772921085 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13639234650343426) | > loader_time: 0.0046 (0.0046455761210441905)  --> TIME: 2025-02-24 01:12:02 -- STEP: 28874/36938 -- GLOBAL_STEP: 102750 | > loss_text_ce: 0.04313265532255173 (0.06157291825275448) | > loss_mel_ce: 3.8119261264801025 (nan) | > loss: 0.015297852456569672 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13639839620993768) | > loader_time: 0.0048 (0.004645776449746411)  --> TIME: 2025-02-24 01:12:12 -- STEP: 28924/36938 -- GLOBAL_STEP: 102800 | > loss_text_ce: 0.058571528643369675 (0.06156407993706919) | > loss_mel_ce: 3.732164144515991 (nan) | > loss: 0.015042603015899658 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.13640629549505187) | > loader_time: 0.0046 (0.004645887342433806)  --> TIME: 2025-02-24 01:12:21 -- STEP: 28974/36938 -- GLOBAL_STEP: 102850 | > loss_text_ce: 0.04672100394964218 (0.06155630724980637) | > loss_mel_ce: 1.8195559978485107 (nan) | > loss: 0.007405861280858517 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.13640190892283025) | > loader_time: 0.0045 (0.004645876125115353)  --> TIME: 2025-02-24 01:12:30 -- STEP: 29024/36938 -- GLOBAL_STEP: 102900 | > loss_text_ce: 0.06141490489244461 (0.061543921175755696) | > loss_mel_ce: 3.758754253387451 (nan) | > loss: 0.015159402042627335 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13639508961579713) | > loader_time: 0.0046 (0.004645900515369233)  --> TIME: 2025-02-24 01:12:39 -- STEP: 29074/36938 -- GLOBAL_STEP: 102950 | > loss_text_ce: 0.05499143898487091 (0.061534377878771196) | > loss_mel_ce: 3.3983144760131836 (nan) | > loss: 0.013703595846891403 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13638941212681815) | > loader_time: 0.0046 (0.00464586166221245)  --> TIME: 2025-02-24 01:12:48 -- STEP: 29124/36938 -- GLOBAL_STEP: 103000 | > loss_text_ce: 0.059846971184015274 (0.06152874949004283) | > loss_mel_ce: 3.6081855297088623 (nan) | > loss: 0.014555685222148895 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.13638067167713244) | > loader_time: 0.0045 (0.0046457358890634095)  --> TIME: 2025-02-24 01:12:57 -- STEP: 29174/36938 -- GLOBAL_STEP: 103050 | > loss_text_ce: 0.05799010768532753 (0.06152032081077539) | > loss_mel_ce: 3.65840220451355 (nan) | > loss: 0.014747588895261288 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13638121742442783) | > loader_time: 0.0045 (0.004645763712209852)  --> TIME: 2025-02-24 01:13:07 -- STEP: 29224/36938 -- GLOBAL_STEP: 103100 | > loss_text_ce: 0.05908519774675369 (0.06151069987552726) | > loss_mel_ce: 3.5532402992248535 (nan) | > loss: 0.01433462556451559 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.13638513719842968) | > loader_time: 0.0047 (0.004645725178320499)  --> TIME: 2025-02-24 01:13:16 -- STEP: 29274/36938 -- GLOBAL_STEP: 103150 | > loss_text_ce: 0.04889049008488655 (0.0615031377789623) | > loss_mel_ce: 4.661799430847168 (nan) | > loss: 0.018693216145038605 (nan) | > current_lr: 5e-06 | > step_time: 0.1727 (0.13639023090277017) | > loader_time: 0.0046 (0.004645657146808619)  --> TIME: 2025-02-24 01:13:26 -- STEP: 29324/36938 -- GLOBAL_STEP: 103200 | > loss_text_ce: 0.05240069329738617 (0.06149118933721182) | > loss_mel_ce: 0.01226806640625 (nan) | > loss: 0.00025662206462584436 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.1363986284618011) | > loader_time: 0.0048 (0.004645756713496234)  --> TIME: 2025-02-24 01:13:35 -- STEP: 29374/36938 -- GLOBAL_STEP: 103250 | > loss_text_ce: 0.04540542885661125 (0.061481658976973164) | > loss_mel_ce: 1.2708196640014648 (nan) | > loss: 0.005223115440458059 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13639973725634236) | > loader_time: 0.0045 (0.004645658950042106)  --> TIME: 2025-02-24 01:13:44 -- STEP: 29424/36938 -- GLOBAL_STEP: 103300 | > loss_text_ce: 0.04077998921275139 (0.06147231773227391) | > loss_mel_ce: 2.9219489097595215 (nan) | > loss: 0.011756861582398415 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13640053661022528) | > loader_time: 0.0051 (0.00464575259240837)  --> TIME: 2025-02-24 01:13:54 -- STEP: 29474/36938 -- GLOBAL_STEP: 103350 | > loss_text_ce: 0.05897308140993118 (0.061465272152046405) | > loss_mel_ce: 0.1378987580537796 (nan) | > loss: 0.0007812375552020967 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13639877180138388) | > loader_time: 0.0045 (0.004645796241806904)  --> TIME: 2025-02-24 01:14:03 -- STEP: 29524/36938 -- GLOBAL_STEP: 103400 | > loss_text_ce: 0.062311649322509766 (0.061456682353485934) | > loss_mel_ce: 3.6742136478424072 (nan) | > loss: 0.014827482402324677 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13639569190471942) | > loader_time: 0.0045 (0.004645654792111243)  --> TIME: 2025-02-24 01:14:12 -- STEP: 29574/36938 -- GLOBAL_STEP: 103450 | > loss_text_ce: 0.056617483496665955 (0.06144761381628071) | > loss_mel_ce: 2.962505578994751 (nan) | > loss: 0.011980648152530193 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13640630653133895) | > loader_time: 0.0044 (0.004645557394535336)  --> TIME: 2025-02-24 01:14:22 -- STEP: 29624/36938 -- GLOBAL_STEP: 103500 | > loss_text_ce: 0.05371018871665001 (0.06143871941686657) | > loss_mel_ce: 0.566790759563446 (nan) | > loss: 0.002462305361405015 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.1364103640184955) | > loader_time: 0.0065 (0.0046455157211978005)  --> TIME: 2025-02-24 01:14:31 -- STEP: 29674/36938 -- GLOBAL_STEP: 103550 | > loss_text_ce: 0.05491309612989426 (0.06143233699915719) | > loss_mel_ce: 4.524614334106445 (nan) | > loss: 0.018172727897763252 (nan) | > current_lr: 5e-06 | > step_time: 0.1987 (0.13641292277376738) | > loader_time: 0.0048 (0.004645469030087332)  --> TIME: 2025-02-24 01:14:41 -- STEP: 29724/36938 -- GLOBAL_STEP: 103600 | > loss_text_ce: 0.05807914957404137 (0.06142287576784609) | > loss_mel_ce: 4.014838695526123 (nan) | > loss: 0.01616237312555313 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13641380156021707) | > loader_time: 0.0044 (0.004645543646449942)  --> TIME: 2025-02-24 01:14:50 -- STEP: 29774/36938 -- GLOBAL_STEP: 103650 | > loss_text_ce: 0.04931749403476715 (0.06141524626717914) | > loss_mel_ce: 3.4383575916290283 (nan) | > loss: 0.013839981518685818 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.1364080618601254) | > loader_time: 0.0046 (0.004645456394611496)  --> TIME: 2025-02-24 01:14:59 -- STEP: 29824/36938 -- GLOBAL_STEP: 103700 | > loss_text_ce: 0.05839847773313522 (0.06140584147767786) | > loss_mel_ce: 4.378024578094482 (nan) | > loss: 0.017604853957891464 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.1364086249067797) | > loader_time: 0.0046 (0.004645356700591831)  --> TIME: 2025-02-24 01:15:08 -- STEP: 29874/36938 -- GLOBAL_STEP: 103750 | > loss_text_ce: 0.05493604391813278 (0.06139914547101571) | > loss_mel_ce: 3.6989963054656982 (nan) | > loss: 0.014896557666361332 (nan) | > current_lr: 5e-06 | > step_time: 0.1689 (0.1364033412141647) | > loader_time: 0.0048 (0.004645320707881667)  --> TIME: 2025-02-24 01:15:17 -- STEP: 29924/36938 -- GLOBAL_STEP: 103800 | > loss_text_ce: 0.05648226663470268 (0.061389143369125765) | > loss_mel_ce: 3.856785535812378 (nan) | > loss: 0.015528841875493526 (nan) | > current_lr: 5e-06 | > step_time: 0.168 (0.13640349226956489) | > loader_time: 0.0048 (0.0046453716649424584)  --> TIME: 2025-02-24 01:15:27 -- STEP: 29974/36938 -- GLOBAL_STEP: 103850 | > loss_text_ce: 0.06375623494386673 (0.06138342495447349) | > loss_mel_ce: 3.9760706424713135 (nan) | > loss: 0.016031060367822647 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13641002154806764) | > loader_time: 0.0046 (0.004645319874899021)  --> TIME: 2025-02-24 01:15:36 -- STEP: 30024/36938 -- GLOBAL_STEP: 103900 | > loss_text_ce: 0.05741290748119354 (0.06137324267505196) | > loss_mel_ce: 2.83858323097229 (nan) | > loss: 0.011492048390209675 (nan) | > current_lr: 5e-06 | > step_time: 0.1777 (0.13641096864768082) | > loader_time: 0.0048 (0.004645203165521431)  --> TIME: 2025-02-24 01:15:45 -- STEP: 30074/36938 -- GLOBAL_STEP: 103950 | > loss_text_ce: 0.05045656859874725 (0.06136350294987154) | > loss_mel_ce: 3.760632276535034 (nan) | > loss: 0.015123369172215462 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13640812014601064) | > loader_time: 0.0045 (0.004645297484027877)  --> TIME: 2025-02-24 01:15:55 -- STEP: 30124/36938 -- GLOBAL_STEP: 104000 | > loss_text_ce: 0.05011671036481857 (0.061358041823952957) | > loss_mel_ce: 3.874342203140259 (nan) | > loss: 0.015573251061141491 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1364057947243523) | > loader_time: 0.0045 (0.004645335074361356)  --> TIME: 2025-02-24 01:16:04 -- STEP: 30174/36938 -- GLOBAL_STEP: 104050 | > loss_text_ce: 0.04548842832446098 (0.0613463529319384) | > loss_mel_ce: 2.913198709487915 (nan) | > loss: 0.011740822345018387 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1364012844023485) | > loader_time: 0.0051 (0.004645484053386209)  --> TIME: 2025-02-24 01:16:13 -- STEP: 30224/36938 -- GLOBAL_STEP: 104100 | > loss_text_ce: 0.054354481399059296 (0.061337086255045276) | > loss_mel_ce: 4.067978382110596 (nan) | > loss: 0.01635846495628357 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.1363951447087023) | > loader_time: 0.0047 (0.004645589658228343)  --> TIME: 2025-02-24 01:16:22 -- STEP: 30274/36938 -- GLOBAL_STEP: 104150 | > loss_text_ce: 0.051927875727415085 (0.061326922510241244) | > loss_mel_ce: 4.875 (nan) | > loss: 0.01955130323767662 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13639025333150182) | > loader_time: 0.0044 (0.0046455112451479685)  --> TIME: 2025-02-24 01:16:31 -- STEP: 30324/36938 -- GLOBAL_STEP: 104200 | > loss_text_ce: 0.05573166161775589 (0.061320443876672334) | > loss_mel_ce: 3.9324872493743896 (nan) | > loss: 0.015826266258955002 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.13639155873489903) | > loader_time: 0.0046 (0.00464551008686423)  --> TIME: 2025-02-24 01:16:40 -- STEP: 30374/36938 -- GLOBAL_STEP: 104250 | > loss_text_ce: 0.05896575003862381 (0.061313797249546144) | > loss_mel_ce: 3.5942342281341553 (nan) | > loss: 0.014496825635433197 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13638502557588278) | > loader_time: 0.0045 (0.004645491977625645)  --> TIME: 2025-02-24 01:16:49 -- STEP: 30424/36938 -- GLOBAL_STEP: 104300 | > loss_text_ce: 0.059864286333322525 (0.06130428995297639) | > loss_mel_ce: 4.2014546394348145 (nan) | > loss: 0.016909997910261154 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.1363834137042627) | > loader_time: 0.0045 (0.004645457165572939)  --> TIME: 2025-02-24 01:16:59 -- STEP: 30474/36938 -- GLOBAL_STEP: 104350 | > loss_text_ce: 0.060602109879255295 (0.061297055100547844) | > loss_mel_ce: 3.4799604415893555 (nan) | > loss: 0.014049852266907692 (nan) | > current_lr: 5e-06 | > step_time: 0.1439 (0.13638010278980725) | > loader_time: 0.0046 (0.0046453210103776016)  --> TIME: 2025-02-24 01:17:08 -- STEP: 30524/36938 -- GLOBAL_STEP: 104400 | > loss_text_ce: 0.06760358065366745 (0.061288382701148995) | > loss_mel_ce: 3.078369617462158 (nan) | > loss: 0.012484021484851837 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13637556114760327) | > loader_time: 0.0046 (0.0046453633340904354)  --> TIME: 2025-02-24 01:17:17 -- STEP: 30574/36938 -- GLOBAL_STEP: 104450 | > loss_text_ce: 0.06977448612451553 (0.06128163260356574) | > loss_mel_ce: 2.9389572143554688 (nan) | > loss: 0.011939411982893944 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.136369024833079) | > loader_time: 0.0045 (0.004645320793204812)  --> TIME: 2025-02-24 01:17:26 -- STEP: 30624/36938 -- GLOBAL_STEP: 104500 | > loss_text_ce: 0.062113240361213684 (0.06127307561216156) | > loss_mel_ce: 3.5108935832977295 (nan) | > loss: 0.01417859923094511 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.13637047986594283) | > loader_time: 0.0052 (0.004645461417041781)  --> TIME: 2025-02-24 01:17:35 -- STEP: 30674/36938 -- GLOBAL_STEP: 104550 | > loss_text_ce: 0.06320882588624954 (0.061265135644034865) | > loss_mel_ce: 3.9970104694366455 (nan) | > loss: 0.016111982986330986 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.1363733805103447) | > loader_time: 0.0047 (0.0046455650198387885)  --> TIME: 2025-02-24 01:17:45 -- STEP: 30724/36938 -- GLOBAL_STEP: 104600 | > loss_text_ce: 0.04903363436460495 (0.06125564906034838) | > loss_mel_ce: 0.8108203411102295 (nan) | > loss: 0.00341211911290884 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13636747962126908) | > loader_time: 0.0046 (0.004645428128609441)  --> TIME: 2025-02-24 01:17:54 -- STEP: 30774/36938 -- GLOBAL_STEP: 104650 | > loss_text_ce: 0.053761061280965805 (0.061249082110353875) | > loss_mel_ce: 3.5674796104431152 (nan) | > loss: 0.014370003715157509 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13636155127549235) | > loader_time: 0.0045 (0.0046452496990309275)  --> TIME: 2025-02-24 01:18:03 -- STEP: 30824/36938 -- GLOBAL_STEP: 104700 | > loss_text_ce: 0.057788971811532974 (0.06123983206440314) | > loss_mel_ce: 3.728175640106201 (nan) | > loss: 0.0150236701592803 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13636072210110828) | > loader_time: 0.0045 (0.004645117553462982)  --> TIME: 2025-02-24 01:18:12 -- STEP: 30874/36938 -- GLOBAL_STEP: 104750 | > loss_text_ce: 0.055724672973155975 (0.06123070539369018) | > loss_mel_ce: 3.7319130897521973 (nan) | > loss: 0.015030309557914734 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.13636126829921838) | > loader_time: 0.0047 (0.0046451392550307434)  --> TIME: 2025-02-24 01:18:22 -- STEP: 30924/36938 -- GLOBAL_STEP: 104800 | > loss_text_ce: 0.05560198053717613 (0.06122246445690843) | > loss_mel_ce: 3.739655017852783 (nan) | > loss: 0.015060544945299625 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13636999727143473) | > loader_time: 0.0045 (0.004645081953249339)  --> TIME: 2025-02-24 01:18:31 -- STEP: 30974/36938 -- GLOBAL_STEP: 104850 | > loss_text_ce: 0.051662568002939224 (0.06121672801827303) | > loss_mel_ce: 4.124551773071289 (nan) | > loss: 0.01657227985560894 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.13636930509049794) | > loader_time: 0.0046 (0.004644921560749988)  --> TIME: 2025-02-24 01:18:40 -- STEP: 31024/36938 -- GLOBAL_STEP: 104900 | > loss_text_ce: 0.0799117311835289 (0.06120898490344118) | > loss_mel_ce: 3.739428758621216 (nan) | > loss: 0.015156113542616367 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13637099161843944) | > loader_time: 0.0045 (0.0046449059705995464)  --> TIME: 2025-02-24 01:18:49 -- STEP: 31074/36938 -- GLOBAL_STEP: 104950 | > loss_text_ce: 0.05632037669420242 (0.0612021602978896) | > loss_mel_ce: 3.0496022701263428 (nan) | > loss: 0.012325090356171131 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13636594336718874) | > loader_time: 0.0046 (0.004644946993076388)  --> TIME: 2025-02-24 01:18:59 -- STEP: 31124/36938 -- GLOBAL_STEP: 105000 | > loss_text_ce: 0.05974220484495163 (0.0611947775422102) | > loss_mel_ce: 3.801990509033203 (nan) | > loss: 0.015324337407946587 (nan) | > current_lr: 5e-06 | > step_time: 0.1693 (0.136365598699853) | > loader_time: 0.0049 (0.004644880111259531)  --> TIME: 2025-02-24 01:19:08 -- STEP: 31174/36938 -- GLOBAL_STEP: 105050 | > loss_text_ce: 0.04804331064224243 (0.06118732363488888) | > loss_mel_ce: 1.3903096914291382 (nan) | > loss: 0.00570775056257844 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13636829514710508) | > loader_time: 0.0045 (0.004644792511425438)  --> TIME: 2025-02-24 01:19:18 -- STEP: 31224/36938 -- GLOBAL_STEP: 105100 | > loss_text_ce: 0.057307466864585876 (0.06117719928145757) | > loss_mel_ce: 4.223089694976807 (nan) | > loss: 0.016985703259706497 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13638351550323285) | > loader_time: 0.0045 (0.0046447574588356876)  --> TIME: 2025-02-24 01:19:27 -- STEP: 31274/36938 -- GLOBAL_STEP: 105150 | > loss_text_ce: 0.040311090648174286 (0.061169684702043814) | > loss_mel_ce: 3.5695605278015137 (nan) | > loss: 0.014324888586997986 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13638117022623128) | > loader_time: 0.0045 (0.004644745419441263)  --> TIME: 2025-02-24 01:19:37 -- STEP: 31324/36938 -- GLOBAL_STEP: 105200 | > loss_text_ce: 0.06968986988067627 (0.06116549507075515) | > loss_mel_ce: 4.082564830780029 (nan) | > loss: 0.016477201133966446 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13638008569425783) | > loader_time: 0.0044 (0.004644892633136388)  --> TIME: 2025-02-24 01:19:46 -- STEP: 31374/36938 -- GLOBAL_STEP: 105250 | > loss_text_ce: 0.05190422758460045 (0.06115627149847188) | > loss_mel_ce: 3.8481760025024414 (nan) | > loss: 0.015476509928703308 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.13638505413145074) | > loader_time: 0.0053 (0.004644992003942764)  --> TIME: 2025-02-24 01:19:55 -- STEP: 31424/36938 -- GLOBAL_STEP: 105300 | > loss_text_ce: 0.06443311274051666 (0.061148599194183896) | > loss_mel_ce: 3.431173324584961 (nan) | > loss: 0.013871454633772373 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13637642408370165) | > loader_time: 0.0046 (0.004645115314639566)  --> TIME: 2025-02-24 01:20:04 -- STEP: 31474/36938 -- GLOBAL_STEP: 105350 | > loss_text_ce: 0.0594022274017334 (0.061141264945118456) | > loss_mel_ce: 3.93768048286438 (nan) | > loss: 0.01586144044995308 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.13638309554631853) | > loader_time: 0.0047 (0.004645142499484161)  --> TIME: 2025-02-24 01:20:14 -- STEP: 31524/36938 -- GLOBAL_STEP: 105400 | > loss_text_ce: 0.032577913254499435 (0.061131722816827945) | > loss_mel_ce: 2.0297985076904297 (nan) | > loss: 0.008184034377336502 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13638683198717716) | > loader_time: 0.0045 (0.004645138090296114)  --> TIME: 2025-02-24 01:20:23 -- STEP: 31574/36938 -- GLOBAL_STEP: 105450 | > loss_text_ce: 0.0484396368265152 (0.06112307661891403) | > loss_mel_ce: 2.539332628250122 (nan) | > loss: 0.010268938727676868 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13638798144844186) | > loader_time: 0.0047 (0.004645100787358498)  --> TIME: 2025-02-24 01:20:33 -- STEP: 31624/36938 -- GLOBAL_STEP: 105500 | > loss_text_ce: 0.05506785586476326 (0.06111452931254507) | > loss_mel_ce: 4.242193698883057 (nan) | > loss: 0.017052626237273216 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.1363957950111698) | > loader_time: 0.0045 (0.004645244188016853)  --> TIME: 2025-02-24 01:20:42 -- STEP: 31674/36938 -- GLOBAL_STEP: 105550 | > loss_text_ce: 0.05655791237950325 (0.06110307375175951) | > loss_mel_ce: 3.888936758041382 (nan) | > loss: 0.01565672643482685 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.13639301861402037) | > loader_time: 0.0045 (0.004645112902631775)  --> TIME: 2025-02-24 01:20:51 -- STEP: 31724/36938 -- GLOBAL_STEP: 105600 | > loss_text_ce: 0.04873347282409668 (0.061092727038106455) | > loss_mel_ce: 0.013671875 (nan) | > loss: 0.00024764027330093086 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13639025326079293) | > loader_time: 0.0046 (0.004645022576670879)  --> TIME: 2025-02-24 01:21:01 -- STEP: 31774/36938 -- GLOBAL_STEP: 105650 | > loss_text_ce: 0.08987266570329666 (0.06108694345620905) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13639133709030857) | > loader_time: 0.0051 (0.004645271163804097)  --> TIME: 2025-02-24 01:21:10 -- STEP: 31824/36938 -- GLOBAL_STEP: 105700 | > loss_text_ce: 0.06161988899111748 (0.06107822694542189) | > loss_mel_ce: 3.875236749649048 (nan) | > loss: 0.015622448176145554 (nan) | > current_lr: 5e-06 | > step_time: 0.161 (0.13639555358299285) | > loader_time: 0.0046 (0.0046453452353024905)  --> TIME: 2025-02-24 01:21:19 -- STEP: 31874/36938 -- GLOBAL_STEP: 105750 | > loss_text_ce: 0.06294996291399002 (0.061068995173954754) | > loss_mel_ce: 4.087719440460205 (nan) | > loss: 0.016470912843942642 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13639728492794903) | > loader_time: 0.0046 (0.00464545807530567)  --> TIME: 2025-02-24 01:21:29 -- STEP: 31924/36938 -- GLOBAL_STEP: 105800 | > loss_text_ce: 0.058870043605566025 (0.0610628559427626) | > loss_mel_ce: 2.566462993621826 (nan) | > loss: 0.010417988523840904 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13640394857272947) | > loader_time: 0.0046 (0.004645702773475293)  --> TIME: 2025-02-24 01:21:38 -- STEP: 31974/36938 -- GLOBAL_STEP: 105850 | > loss_text_ce: 0.044967517256736755 (0.061050354667897974) | > loss_mel_ce: 3.9452526569366455 (nan) | > loss: 0.015834206715226173 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13639729611457618) | > loader_time: 0.0044 (0.004645576730278298)  --> TIME: 2025-02-24 01:21:47 -- STEP: 32024/36938 -- GLOBAL_STEP: 105900 | > loss_text_ce: 0.05370045453310013 (0.06104157028724148) | > loss_mel_ce: 3.526123046875 (nan) | > loss: 0.014205649495124817 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13639463183881562) | > loader_time: 0.0047 (0.0046455166859833376)  --> TIME: 2025-02-24 01:21:57 -- STEP: 32074/36938 -- GLOBAL_STEP: 105950 | > loss_text_ce: 0.05872143432497978 (0.0610340792063079) | > loss_mel_ce: 4.190480709075928 (nan) | > loss: 0.016861915588378906 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.1363992099639453) | > loader_time: 0.0046 (0.004645471494974092)  --> TIME: 2025-02-24 01:22:06 -- STEP: 32124/36938 -- GLOBAL_STEP: 106000 | > loss_text_ce: 0.056907303631305695 (0.061028517317838066) | > loss_mel_ce: 3.571117401123047 (nan) | > loss: 0.014396924525499344 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.13639670632693243) | > loader_time: 0.0053 (0.004645447455816995)  --> TIME: 2025-02-24 01:22:15 -- STEP: 32174/36938 -- GLOBAL_STEP: 106050 | > loss_text_ce: 0.06170780211687088 (0.061021408490449805) | > loss_mel_ce: 4.327117443084717 (nan) | > loss: 0.01741597428917885 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.13639505200591384) | > loader_time: 0.0054 (0.004645456570897878)  --> TIME: 2025-02-24 01:22:24 -- STEP: 32224/36938 -- GLOBAL_STEP: 106100 | > loss_text_ce: 0.06230638921260834 (0.061013427911621225) | > loss_mel_ce: 3.622535228729248 (nan) | > loss: 0.014622388407588005 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13638985333104042) | > loader_time: 0.0046 (0.00464537862573627)  --> TIME: 2025-02-24 01:22:34 -- STEP: 32274/36938 -- GLOBAL_STEP: 106150 | > loss_text_ce: 0.05727459862828255 (0.06100590179720501) | > loss_mel_ce: 3.3910348415374756 (nan) | > loss: 0.01368376798927784 (nan) | > current_lr: 5e-06 | > step_time: 0.1072 (0.13639654048410882) | > loader_time: 0.0044 (0.004645395376443428)  --> TIME: 2025-02-24 01:22:43 -- STEP: 32324/36938 -- GLOBAL_STEP: 106200 | > loss_text_ce: 0.07581725716590881 (0.060997570905019775) | > loss_mel_ce: 3.0849215984344482 (nan) | > loss: 0.01254261564463377 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.1363945162513139) | > loader_time: 0.0044 (0.004645441254388186)  --> TIME: 2025-02-24 01:22:52 -- STEP: 32374/36938 -- GLOBAL_STEP: 106250 | > loss_text_ce: 0.05536030977964401 (0.060989971419821924) | > loss_mel_ce: 4.1317033767700195 (nan) | > loss: 0.016615333035588264 (nan) | > current_lr: 5e-06 | > step_time: 0.1856 (0.1363878403966633) | > loader_time: 0.0048 (0.004645611089943433)  --> TIME: 2025-02-24 01:23:01 -- STEP: 32424/36938 -- GLOBAL_STEP: 106300 | > loss_text_ce: 0.05045695602893829 (0.060978788093925394) | > loss_mel_ce: 4.3154497146606445 (nan) | > loss: 0.01732502691447735 (nan) | > current_lr: 5e-06 | > step_time: 0.1996 (0.13639190208461807) | > loader_time: 0.0054 (0.004645739907898426)  --> TIME: 2025-02-24 01:23:11 -- STEP: 32474/36938 -- GLOBAL_STEP: 106350 | > loss_text_ce: 0.060161471366882324 (0.0609704048443981) | > loss_mel_ce: 4.308135509490967 (nan) | > loss: 0.01733451336622238 (nan) | > current_lr: 5e-06 | > step_time: 0.1605 (0.13639360386184143) | > loader_time: 0.0049 (0.00464580892642868)  --> TIME: 2025-02-24 01:23:20 -- STEP: 32524/36938 -- GLOBAL_STEP: 106400 | > loss_text_ce: 0.06361352652311325 (0.060962586939989946) | > loss_mel_ce: 3.3207473754882812 (nan) | > loss: 0.013430004008114338 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13640189993324248) | > loader_time: 0.0046 (0.004645790447975954)  --> TIME: 2025-02-24 01:23:30 -- STEP: 32574/36938 -- GLOBAL_STEP: 106450 | > loss_text_ce: 0.05613569915294647 (0.06095375404622512) | > loss_mel_ce: 3.8147385120391846 (nan) | > loss: 0.01536061242222786 (nan) | > current_lr: 5e-06 | > step_time: 0.1777 (0.13640563559881952) | > loader_time: 0.0048 (0.0046457279568008275)  --> TIME: 2025-02-24 01:23:39 -- STEP: 32624/36938 -- GLOBAL_STEP: 106500 | > loss_text_ce: 0.05194583162665367 (0.06094630242592741) | > loss_mel_ce: 3.060903787612915 (nan) | > loss: 0.012352579273283482 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.1364008298014932) | > loader_time: 0.0045 (0.004645666863007478)  --> TIME: 2025-02-24 01:23:48 -- STEP: 32674/36938 -- GLOBAL_STEP: 106550 | > loss_text_ce: 0.05379834398627281 (0.06093901493125645) | > loss_mel_ce: 1.0060138702392578 (nan) | > loss: 0.004205604083836079 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13640365537394797) | > loader_time: 0.0051 (0.004645748369593256)  --> TIME: 2025-02-24 01:23:58 -- STEP: 32724/36938 -- GLOBAL_STEP: 106600 | > loss_text_ce: 0.07426705211400986 (0.060930685891989335) | > loss_mel_ce: 3.65533447265625 (nan) | > loss: 0.014800007455050945 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.1364016949205166) | > loader_time: 0.0044 (0.0046460123971977605)  --> TIME: 2025-02-24 01:24:07 -- STEP: 32774/36938 -- GLOBAL_STEP: 106650 | > loss_text_ce: 0.054246921092271805 (0.06092193394070916) | > loss_mel_ce: 3.7077839374542236 (nan) | > loss: 0.014928694814443588 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13639746469132244) | > loader_time: 0.0046 (0.0046461649139492335)  --> TIME: 2025-02-24 01:24:16 -- STEP: 32824/36938 -- GLOBAL_STEP: 106700 | > loss_text_ce: 0.06122715398669243 (0.06091390679563793) | > loss_mel_ce: 3.6561172008514404 (nan) | > loss: 0.014751367270946503 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13640039980542198) | > loader_time: 0.0045 (0.004646093691961425)  --> TIME: 2025-02-24 01:24:25 -- STEP: 32874/36938 -- GLOBAL_STEP: 106750 | > loss_text_ce: 0.06054893881082535 (0.0609067028766072) | > loss_mel_ce: 3.6245758533477783 (nan) | > loss: 0.014623512513935566 (nan) | > current_lr: 5e-06 | > step_time: 0.1621 (0.13639224077464743) | > loader_time: 0.0046 (0.004645899764057137)  --> TIME: 2025-02-24 01:24:35 -- STEP: 32924/36938 -- GLOBAL_STEP: 106800 | > loss_text_ce: 0.05051306262612343 (0.06089949486820128) | > loss_mel_ce: 3.6708364486694336 (nan) | > loss: 0.014767260290682316 (nan) | > current_lr: 5e-06 | > step_time: 0.1621 (0.13640082143170112) | > loader_time: 0.0048 (0.00464595562631665)  --> TIME: 2025-02-24 01:24:44 -- STEP: 32974/36938 -- GLOBAL_STEP: 106850 | > loss_text_ce: 0.0569571852684021 (0.06089160881900117) | > loss_mel_ce: 3.9471213817596436 (nan) | > loss: 0.015889201313257217 (nan) | > current_lr: 5e-06 | > step_time: 0.1796 (0.13639771889156485) | > loader_time: 0.0048 (0.0046460668277284)  --> TIME: 2025-02-24 01:24:53 -- STEP: 33024/36938 -- GLOBAL_STEP: 106900 | > loss_text_ce: 0.06375729292631149 (0.06087977595654553) | > loss_mel_ce: 4.129223346710205 (nan) | > loss: 0.01663881354033947 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13639754594787304) | > loader_time: 0.0045 (0.004646128440607936)  --> TIME: 2025-02-24 01:25:03 -- STEP: 33074/36938 -- GLOBAL_STEP: 106950 | > loss_text_ce: 0.05496272072196007 (0.060867809320415876) | > loss_mel_ce: 3.757129669189453 (nan) | > loss: 0.015127351507544518 (nan) | > current_lr: 5e-06 | > step_time: 0.1887 (0.13640240111416968) | > loader_time: 0.0048 (0.004646182305335526)  --> TIME: 2025-02-24 01:25:12 -- STEP: 33124/36938 -- GLOBAL_STEP: 107000 | > loss_text_ce: 0.06468667089939117 (0.06086017945707068) | > loss_mel_ce: 4.336688995361328 (nan) | > loss: 0.017465777695178986 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.1364100156969349) | > loader_time: 0.005 (0.004646316003344587)  --> TIME: 2025-02-24 01:25:21 -- STEP: 33174/36938 -- GLOBAL_STEP: 107050 | > loss_text_ce: 0.06256210803985596 (0.0608535000434305) | > loss_mel_ce: 3.630831003189087 (nan) | > loss: 0.01465632300823927 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.1364073524238209) | > loader_time: 0.0045 (0.004646380289615617)  --> TIME: 2025-02-24 01:25:31 -- STEP: 33224/36938 -- GLOBAL_STEP: 107100 | > loss_text_ce: 0.05063392594456673 (0.06084734740978253) | > loss_mel_ce: 4.198042392730713 (nan) | > loss: 0.01685982756316662 (nan) | > current_lr: 5e-06 | > step_time: 0.1875 (0.13640907960801404) | > loader_time: 0.0049 (0.004646548062613224)  --> TIME: 2025-02-24 01:25:40 -- STEP: 33274/36938 -- GLOBAL_STEP: 107150 | > loss_text_ce: 0.05670047178864479 (0.0608417383978599) | > loss_mel_ce: 3.9302353858947754 (nan) | > loss: 0.015821175649762154 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13640032844517014) | > loader_time: 0.0052 (0.004646588104121069)  --> TIME: 2025-02-24 01:25:49 -- STEP: 33324/36938 -- GLOBAL_STEP: 107200 | > loss_text_ce: 0.04532264173030853 (0.06083468650632528) | > loss_mel_ce: 2.711257219314575 (nan) | > loss: 0.010938809253275394 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1364035048400085) | > loader_time: 0.0045 (0.004646618559987007)  --> TIME: 2025-02-24 01:25:59 -- STEP: 33374/36938 -- GLOBAL_STEP: 107250 | > loss_text_ce: 0.047524817287921906 (0.06082664617516551) | > loss_mel_ce: 3.528001546859741 (nan) | > loss: 0.014188597910106182 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13640877242827487) | > loader_time: 0.0047 (0.004646634229713918)  --> TIME: 2025-02-24 01:26:08 -- STEP: 33424/36938 -- GLOBAL_STEP: 107300 | > loss_text_ce: 0.05667512118816376 (0.06081754933823514) | > loss_mel_ce: 3.965165138244629 (nan) | > loss: 0.015959683805704117 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.1364143464398639) | > loader_time: 0.0047 (0.004646691517318637)  --> TIME: 2025-02-24 01:26:18 -- STEP: 33474/36938 -- GLOBAL_STEP: 107350 | > loss_text_ce: 0.06403329968452454 (0.060811245002723564) | > loss_mel_ce: 3.2421913146972656 (nan) | > loss: 0.013119939714670181 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13641536442361765) | > loader_time: 0.0045 (0.004646732316134292)  --> TIME: 2025-02-24 01:26:27 -- STEP: 33524/36938 -- GLOBAL_STEP: 107400 | > loss_text_ce: 0.05316505953669548 (0.06080287239965283) | > loss_mel_ce: 2.742685079574585 (nan) | > loss: 0.011094643734395504 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13641717437912024) | > loader_time: 0.0051 (0.004646872637764355)  --> TIME: 2025-02-24 01:26:36 -- STEP: 33574/36938 -- GLOBAL_STEP: 107450 | > loss_text_ce: 0.05614393949508667 (0.060796760677955204) | > loss_mel_ce: 3.888073205947876 (nan) | > loss: 0.015651656314730644 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13641958524438372) | > loader_time: 0.0046 (0.004647135762963296)  --> TIME: 2025-02-24 01:26:46 -- STEP: 33624/36938 -- GLOBAL_STEP: 107500 | > loss_text_ce: 0.05146483704447746 (0.060789709447082785) | > loss_mel_ce: 3.1949639320373535 (nan) | > loss: 0.012882654555141926 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13641644508362644) | > loader_time: 0.0045 (0.004647122723075501)  --> TIME: 2025-02-24 01:26:55 -- STEP: 33674/36938 -- GLOBAL_STEP: 107550 | > loss_text_ce: 0.04992857202887535 (0.060784139329862465) | > loss_mel_ce: 3.9827826023101807 (nan) | > loss: 0.016002822667360306 (nan) | > current_lr: 5e-06 | > step_time: 0.1879 (0.13642035044237108) | > loader_time: 0.0048 (0.0046472020335299)  --> TIME: 2025-02-24 01:27:04 -- STEP: 33724/36938 -- GLOBAL_STEP: 107600 | > loss_text_ce: 0.05043919011950493 (0.0607760866152612) | > loss_mel_ce: 2.622751235961914 (nan) | > loss: 0.01060789916664362 (nan) | > current_lr: 5e-06 | > step_time: 0.1642 (0.13642168503867422) | > loader_time: 0.0046 (0.004647216300864858)  --> TIME: 2025-02-24 01:27:14 -- STEP: 33774/36938 -- GLOBAL_STEP: 107650 | > loss_text_ce: 0.05673445016145706 (0.06077101902400077) | > loss_mel_ce: 4.2321696281433105 (nan) | > loss: 0.01701946184039116 (nan) | > current_lr: 5e-06 | > step_time: 0.193 (0.13643092126344178) | > loader_time: 0.0048 (0.004647268631702999)  --> TIME: 2025-02-24 01:27:23 -- STEP: 33824/36938 -- GLOBAL_STEP: 107700 | > loss_text_ce: 0.053287070244550705 (0.06076636724779722) | > loss_mel_ce: 3.572505235671997 (nan) | > loss: 0.014388065785169601 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.136430940211044) | > loader_time: 0.0048 (0.004647266856318096)  --> TIME: 2025-02-24 01:27:33 -- STEP: 33874/36938 -- GLOBAL_STEP: 107750 | > loss_text_ce: 0.05634590610861778 (0.06075916844398359) | > loss_mel_ce: 3.3316705226898193 (nan) | > loss: 0.013444510288536549 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.13643028630960805) | > loader_time: 0.0047 (0.004647238340274809)  --> TIME: 2025-02-24 01:27:42 -- STEP: 33924/36938 -- GLOBAL_STEP: 107800 | > loss_text_ce: 0.05691282078623772 (0.06075278701650026) | > loss_mel_ce: 3.7728755474090576 (nan) | > loss: 0.0151975741609931 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.13643174463251098) | > loader_time: 0.0045 (0.00464723967898643)  --> TIME: 2025-02-24 01:27:51 -- STEP: 33974/36938 -- GLOBAL_STEP: 107850 | > loss_text_ce: 0.041797101497650146 (0.06074461099342558) | > loss_mel_ce: 3.623809814453125 (nan) | > loss: 0.014546060003340244 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13643254899442406) | > loader_time: 0.0045 (0.004647124695751272)  --> TIME: 2025-02-24 01:28:01 -- STEP: 34024/36938 -- GLOBAL_STEP: 107900 | > loss_text_ce: 0.07347626239061356 (0.060736765330860455) | > loss_mel_ce: 3.2702393531799316 (nan) | > loss: 0.013268713839352131 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13643184200528286) | > loader_time: 0.0044 (0.00464701143091441)  --> TIME: 2025-02-24 01:28:10 -- STEP: 34074/36938 -- GLOBAL_STEP: 107950 | > loss_text_ce: 0.05668334662914276 (0.060727877998716914) | > loss_mel_ce: 1.9613037109375 (nan) | > loss: 0.00800788588821888 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13643805228635889) | > loader_time: 0.0045 (0.004646893810440865)  --> TIME: 2025-02-24 01:28:19 -- STEP: 34124/36938 -- GLOBAL_STEP: 108000 | > loss_text_ce: 0.04942455142736435 (0.06072046501281815) | > loss_mel_ce: 4.176041603088379 (nan) | > loss: 0.016767723485827446 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13643241903312564) | > loader_time: 0.0044 (0.004646762183704449)  --> TIME: 2025-02-24 01:28:29 -- STEP: 34174/36938 -- GLOBAL_STEP: 108050 | > loss_text_ce: 0.048350218683481216 (0.060714533300060894) | > loss_mel_ce: 2.7969560623168945 (nan) | > loss: 0.011290899477899075 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13643619946796556) | > loader_time: 0.0046 (0.004646653323090683)  --> TIME: 2025-02-24 01:28:37 -- STEP: 34224/36938 -- GLOBAL_STEP: 108100 | > loss_text_ce: 0.0573677234351635 (0.060705390435962084) | > loss_mel_ce: 4.269893169403076 (nan) | > loss: 0.017171671614050865 (nan) | > current_lr: 5e-06 | > step_time: 0.1842 (0.13642554133503412) | > loader_time: 0.0048 (0.0046465225994949625)  --> TIME: 2025-02-24 01:28:47 -- STEP: 34274/36938 -- GLOBAL_STEP: 108150 | > loss_text_ce: 0.05183202773332596 (0.06069945394531921) | > loss_mel_ce: 2.981987237930298 (nan) | > loss: 0.012038965709507465 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13642465872797058) | > loader_time: 0.0046 (0.004646483516375953)  --> TIME: 2025-02-24 01:28:56 -- STEP: 34324/36938 -- GLOBAL_STEP: 108200 | > loss_text_ce: 0.044982828199863434 (0.06068892873245757) | > loss_mel_ce: 3.94252610206604 (nan) | > loss: 0.015823449939489365 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.1364215257603746) | > loader_time: 0.0048 (0.004646374544137715)  --> TIME: 2025-02-24 01:29:05 -- STEP: 34374/36938 -- GLOBAL_STEP: 108250 | > loss_text_ce: 0.05469852313399315 (0.06067784624769704) | > loss_mel_ce: 3.74003005027771 (nan) | > loss: 0.015058447606861591 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.13642951804333178) | > loader_time: 0.0045 (0.004646287473797484)  --> TIME: 2025-02-24 01:29:15 -- STEP: 34424/36938 -- GLOBAL_STEP: 108300 | > loss_text_ce: 0.04155898466706276 (0.060669629389978864) | > loss_mel_ce: 3.7037200927734375 (nan) | > loss: 0.01486221980303526 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13643141401498265) | > loader_time: 0.0045 (0.004646220326063536)  --> TIME: 2025-02-24 01:29:24 -- STEP: 34474/36938 -- GLOBAL_STEP: 108350 | > loss_text_ce: 0.05492309480905533 (0.06066139738359634) | > loss_mel_ce: 4.112071990966797 (nan) | > loss: 0.016535695642232895 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13643101692310286) | > loader_time: 0.0046 (0.0046461114143699096)  --> TIME: 2025-02-24 01:29:33 -- STEP: 34524/36938 -- GLOBAL_STEP: 108400 | > loss_text_ce: 0.06168602034449577 (0.0606518718388106) | > loss_mel_ce: 0.00750732421875 (nan) | > loss: 0.0002745767997112125 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13643070217383751) | > loader_time: 0.0046 (0.004646031933330285)  --> TIME: 2025-02-24 01:29:43 -- STEP: 34574/36938 -- GLOBAL_STEP: 108450 | > loss_text_ce: 0.045208610594272614 (0.060642104780741266) | > loss_mel_ce: 2.3476977348327637 (nan) | > loss: 0.009495660662651062 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13642718331040152) | > loader_time: 0.0045 (0.0046460403496557355)  --> TIME: 2025-02-24 01:29:52 -- STEP: 34624/36938 -- GLOBAL_STEP: 108500 | > loss_text_ce: 0.06477785855531693 (0.06063427993580738) | > loss_mel_ce: 4.142153739929199 (nan) | > loss: 0.016694173216819763 (nan) | > current_lr: 5e-06 | > step_time: 0.1271 (0.13642693371572173) | > loader_time: 0.0045 (0.004645998839309649)  --> TIME: 2025-02-24 01:30:01 -- STEP: 34674/36938 -- GLOBAL_STEP: 108550 | > loss_text_ce: 0.06773392856121063 (0.060627801802717665) | > loss_mel_ce: 3.2376041412353516 (nan) | > loss: 0.013116422109305859 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13642476634623824) | > loader_time: 0.0045 (0.004645891005848396)  --> TIME: 2025-02-24 01:30:10 -- STEP: 34724/36938 -- GLOBAL_STEP: 108600 | > loss_text_ce: 0.0554484985768795 (0.0606207216898541) | > loss_mel_ce: 3.07080078125 (nan) | > loss: 0.012405752204358578 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.1364185986410643) | > loader_time: 0.0047 (0.004645771487848742)  --> TIME: 2025-02-24 01:30:19 -- STEP: 34774/36938 -- GLOBAL_STEP: 108650 | > loss_text_ce: 0.04888768121600151 (0.060614509578932316) | > loss_mel_ce: 3.701082468032837 (nan) | > loss: 0.014880835078656673 (nan) | > current_lr: 5e-06 | > step_time: 0.1435 (0.13641813879571912) | > loader_time: 0.0046 (0.004645749911997631)  --> TIME: 2025-02-24 01:30:28 -- STEP: 34824/36938 -- GLOBAL_STEP: 108700 | > loss_text_ce: 0.05513063445687294 (0.060605462767662904) | > loss_mel_ce: 3.6725053787231445 (nan) | > loss: 0.01479220762848854 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13641399258091658) | > loader_time: 0.0045 (0.004645755756265607)  --> TIME: 2025-02-24 01:30:38 -- STEP: 34874/36938 -- GLOBAL_STEP: 108750 | > loss_text_ce: 0.0680580586194992 (0.06059954231926946) | > loss_mel_ce: 3.811807632446289 (nan) | > loss: 0.015396293252706528 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.1364107301163134) | > loader_time: 0.0055 (0.004645808742442509)  --> TIME: 2025-02-24 01:30:47 -- STEP: 34924/36938 -- GLOBAL_STEP: 108800 | > loss_text_ce: 0.05447882413864136 (0.060592424201440065) | > loss_mel_ce: 2.420654296875 (nan) | > loss: 0.009821957908570766 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13641026821923935) | > loader_time: 0.0046 (0.004645778426680597)  --> TIME: 2025-02-24 01:30:56 -- STEP: 34974/36938 -- GLOBAL_STEP: 108850 | > loss_text_ce: 0.07352020591497421 (0.06058401914317961) | > loss_mel_ce: 4.2320237159729 (nan) | > loss: 0.017085492610931396 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13641318669141556) | > loader_time: 0.0046 (0.004645742246338044)  --> TIME: 2025-02-24 01:31:06 -- STEP: 35024/36938 -- GLOBAL_STEP: 108900 | > loss_text_ce: 0.07255774736404419 (0.06057647796069648) | > loss_mel_ce: 2.3898987770080566 (nan) | > loss: 0.009771653451025486 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13641547859940278) | > loader_time: 0.0045 (0.004645801900238071)  --> TIME: 2025-02-24 01:31:15 -- STEP: 35074/36938 -- GLOBAL_STEP: 108950 | > loss_text_ce: 0.06598507612943649 (0.06056800095713092) | > loss_mel_ce: 3.713609218597412 (nan) | > loss: 0.014998390339314938 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13642215676418246) | > loader_time: 0.0046 (0.004645895596315172)  --> TIME: 2025-02-24 01:31:25 -- STEP: 35124/36938 -- GLOBAL_STEP: 109000 | > loss_text_ce: 0.053439877927303314 (0.06056055776302221) | > loss_mel_ce: 4.492680549621582 (nan) | > loss: 0.018040161579847336 (nan) | > current_lr: 5e-06 | > step_time: 0.1659 (0.13642767747460363) | > loader_time: 0.0047 (0.004645876943651075)  --> TIME: 2025-02-24 01:31:34 -- STEP: 35174/36938 -- GLOBAL_STEP: 109050 | > loss_text_ce: 0.06509732455015182 (0.060553209864095264) | > loss_mel_ce: 3.1115849018096924 (nan) | > loss: 0.012605882249772549 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13643043357909534) | > loader_time: 0.0046 (0.0046458218905249295)  --> TIME: 2025-02-24 01:31:43 -- STEP: 35224/36938 -- GLOBAL_STEP: 109100 | > loss_text_ce: 0.07182766497135162 (0.06054904887229725) | > loss_mel_ce: 3.159393787384033 (nan) | > loss: 0.012822307646274567 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13642817615866967) | > loader_time: 0.0045 (0.0046457641982447844)  --> TIME: 2025-02-24 01:31:53 -- STEP: 35274/36938 -- GLOBAL_STEP: 109150 | > loss_text_ce: 0.06264922767877579 (0.06054169138728785) | > loss_mel_ce: 4.050380706787109 (nan) | > loss: 0.01632154919207096 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.1364345013598506) | > loader_time: 0.0046 (0.004645771988937265)  --> TIME: 2025-02-24 01:32:02 -- STEP: 35324/36938 -- GLOBAL_STEP: 109200 | > loss_text_ce: 0.058146703988313675 (0.060534134591758125) | > loss_mel_ce: 4.353200912475586 (nan) | > loss: 0.017505347728729248 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.13643174953172252) | > loader_time: 0.0046 (0.004645694990874109)  --> TIME: 2025-02-24 01:32:11 -- STEP: 35374/36938 -- GLOBAL_STEP: 109250 | > loss_text_ce: 0.06343187391757965 (0.06052590609788874) | > loss_mel_ce: 2.5703506469726562 (nan) | > loss: 0.010451518930494785 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13643400561393587) | > loader_time: 0.0045 (0.004645675055119498)  --> TIME: 2025-02-24 01:32:21 -- STEP: 35424/36938 -- GLOBAL_STEP: 109300 | > loss_text_ce: 0.05558691546320915 (0.06051781830691629) | > loss_mel_ce: 3.5564918518066406 (nan) | > loss: 0.01433364674448967 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.13643667974796092) | > loader_time: 0.0048 (0.004645667041381891)  --> TIME: 2025-02-24 01:32:30 -- STEP: 35474/36938 -- GLOBAL_STEP: 109350 | > loss_text_ce: 0.07105311751365662 (0.0605108720007996) | > loss_mel_ce: 1.9432373046875 (nan) | > loss: 0.007993215695023537 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13643624635776863) | > loader_time: 0.0045 (0.004645636037740663)  --> TIME: 2025-02-24 01:32:39 -- STEP: 35524/36938 -- GLOBAL_STEP: 109400 | > loss_text_ce: 0.05901622027158737 (0.06050192176047296) | > loss_mel_ce: 4.064244747161865 (nan) | > loss: 0.0163621474057436 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13644014846740246) | > loader_time: 0.0046 (0.004645645289576294)  --> TIME: 2025-02-24 01:32:49 -- STEP: 35574/36938 -- GLOBAL_STEP: 109450 | > loss_text_ce: 0.041062209755182266 (0.060492281088177835) | > loss_mel_ce: 3.037855625152588 (nan) | > loss: 0.012217928655445576 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13643643439449327) | > loader_time: 0.0044 (0.004645511922677108)  --> TIME: 2025-02-24 01:32:58 -- STEP: 35624/36938 -- GLOBAL_STEP: 109500 | > loss_text_ce: 0.0668613612651825 (0.06048531651366327) | > loss_mel_ce: 3.2197062969207764 (nan) | > loss: 0.013041935861110687 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13643473732773886) | > loader_time: 0.0047 (0.0046454763950564215)  --> TIME: 2025-02-24 01:33:07 -- STEP: 35674/36938 -- GLOBAL_STEP: 109550 | > loss_text_ce: 0.06173156574368477 (0.06047546323861811) | > loss_mel_ce: 3.585348129272461 (nan) | > loss: 0.01447253953665495 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.13643484872947398) | > loader_time: 0.0045 (0.0046454015558474926)  --> TIME: 2025-02-24 01:33:16 -- STEP: 35724/36938 -- GLOBAL_STEP: 109600 | > loss_text_ce: 0.05575811117887497 (0.060467593241935014) | > loss_mel_ce: 3.892024278640747 (nan) | > loss: 0.015665803104639053 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.13643413236861632) | > loader_time: 0.0048 (0.004645352587295757)  --> TIME: 2025-02-24 01:33:26 -- STEP: 35774/36938 -- GLOBAL_STEP: 109650 | > loss_text_ce: 0.046049293130636215 (0.06046175201389934) | > loss_mel_ce: 2.907437801361084 (nan) | > loss: 0.011720187962055206 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13644079579697505) | > loader_time: 0.0044 (0.004645309193921967)  --> TIME: 2025-02-24 01:33:35 -- STEP: 35824/36938 -- GLOBAL_STEP: 109700 | > loss_text_ce: 0.05166691169142723 (0.060453709910297505) | > loss_mel_ce: 3.8185980319976807 (nan) | > loss: 0.015358195640146732 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13643697686842707) | > loader_time: 0.0047 (0.004645239779756493)  --> TIME: 2025-02-24 01:33:45 -- STEP: 35874/36938 -- GLOBAL_STEP: 109750 | > loss_text_ce: 0.04871547967195511 (0.06044506560870834) | > loss_mel_ce: 2.4864418506622314 (nan) | > loss: 0.010060149244964123 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13644699466303495) | > loader_time: 0.0046 (0.004645293018268585)  --> TIME: 2025-02-24 01:33:54 -- STEP: 35924/36938 -- GLOBAL_STEP: 109800 | > loss_text_ce: 0.04718654602766037 (0.06043755967892108) | > loss_mel_ce: 4.082510948181152 (nan) | > loss: 0.016387687996029854 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13644806355944678) | > loader_time: 0.0046 (0.004645212145799987)  --> TIME: 2025-02-24 01:34:03 -- STEP: 35974/36938 -- GLOBAL_STEP: 109850 | > loss_text_ce: 0.05004587024450302 (0.06042635384212803) | > loss_mel_ce: 2.418595314025879 (nan) | > loss: 0.009796195663511753 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13644101439849332) | > loader_time: 0.0045 (0.0046450925813135546)  --> TIME: 2025-02-24 01:34:12 -- STEP: 36024/36938 -- GLOBAL_STEP: 109900 | > loss_text_ce: 0.053685449063777924 (0.060421541586866044) | > loss_mel_ce: 2.4740753173828125 (nan) | > loss: 0.010030796751379967 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13643813048525072) | > loader_time: 0.0044 (0.004644988802520848)  --> TIME: 2025-02-24 01:34:22 -- STEP: 36074/36938 -- GLOBAL_STEP: 109950 | > loss_text_ce: 0.05930107831954956 (0.060413817469863046) | > loss_mel_ce: 4.123804569244385 (nan) | > loss: 0.0165996253490448 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13643573408031026) | > loader_time: 0.0048 (0.004644866918138895)  --> TIME: 2025-02-24 01:34:31 -- STEP: 36124/36938 -- GLOBAL_STEP: 110000 | > loss_text_ce: 0.06285696476697922 (0.06040561355046265) | > loss_mel_ce: 3.689094066619873 (nan) | > loss: 0.014888695441186428 (nan) | > current_lr: 5e-06 | > step_time: 0.152 (0.13642934295059447) | > loader_time: 0.0045 (0.004644729742350755) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_110000.pth  --> TIME: 2025-02-24 01:34:46 -- STEP: 36174/36938 -- GLOBAL_STEP: 110050 | > loss_text_ce: 0.07772991061210632 (0.06039649406649285) | > loss_mel_ce: 2.1056668758392334 (nan) | > loss: 0.008664273656904697 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13643112081512954) | > loader_time: 0.0047 (0.004644788437935801)  --> TIME: 2025-02-24 01:34:55 -- STEP: 36224/36938 -- GLOBAL_STEP: 110100 | > loss_text_ce: 0.05816415324807167 (0.06038887405827837) | > loss_mel_ce: 3.682111978530884 (nan) | > loss: 0.014842365868389606 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13643026272947206) | > loader_time: 0.0046 (0.004644899132131738)  --> TIME: 2025-02-24 01:35:05 -- STEP: 36274/36938 -- GLOBAL_STEP: 110150 | > loss_text_ce: 0.060190796852111816 (0.060382974302942934) | > loss_mel_ce: 3.9961016178131104 (nan) | > loss: 0.01609640009701252 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13643343579304362) | > loader_time: 0.0047 (0.00464492098015414)  --> TIME: 2025-02-24 01:35:14 -- STEP: 36324/36938 -- GLOBAL_STEP: 110200 | > loss_text_ce: 0.0463423877954483 (0.06037385993983838) | > loss_mel_ce: 2.1346435546875 (nan) | > loss: 0.008654706180095673 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13643159084337841) | > loader_time: 0.0045 (0.00464488503402876)  --> TIME: 2025-02-24 01:35:23 -- STEP: 36374/36938 -- GLOBAL_STEP: 110250 | > loss_text_ce: 0.037013184279203415 (0.06036571859654826) | > loss_mel_ce: 3.852395534515381 (nan) | > loss: 0.015434162691235542 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13642962454648258) | > loader_time: 0.0046 (0.004644864976861689)  --> TIME: 2025-02-24 01:35:32 -- STEP: 36424/36938 -- GLOBAL_STEP: 110300 | > loss_text_ce: 0.05224697291851044 (0.06035845805260787) | > loss_mel_ce: 3.432648181915283 (nan) | > loss: 0.013828950002789497 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13642664486409317) | > loader_time: 0.0046 (0.004644923031578886)  --> TIME: 2025-02-24 01:35:41 -- STEP: 36474/36938 -- GLOBAL_STEP: 110350 | > loss_text_ce: 0.04835030063986778 (0.06035027598790854) | > loss_mel_ce: 4.188023567199707 (nan) | > loss: 0.016811007633805275 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13642704907808542) | > loader_time: 0.0046 (0.004644905709642537)  --> TIME: 2025-02-24 01:35:51 -- STEP: 36524/36938 -- GLOBAL_STEP: 110400 | > loss_text_ce: 0.05335312336683273 (0.060342893638473054) | > loss_mel_ce: 3.9156768321990967 (nan) | > loss: 0.01575011946260929 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13643502327427326) | > loader_time: 0.0049 (0.004645050081153454)  --> TIME: 2025-02-24 01:36:01 -- STEP: 36574/36938 -- GLOBAL_STEP: 110450 | > loss_text_ce: 0.058599743992090225 (0.06033752296621416) | > loss_mel_ce: 4.269131660461426 (nan) | > loss: 0.017173539847135544 (nan) | > current_lr: 5e-06 | > step_time: 0.1435 (0.13643919651730035) | > loader_time: 0.0046 (0.004644998096289421)  --> TIME: 2025-02-24 01:36:10 -- STEP: 36624/36938 -- GLOBAL_STEP: 110500 | > loss_text_ce: 0.058849871158599854 (0.06032856708926871) | > loss_mel_ce: 3.023284912109375 (nan) | > loss: 0.01223069429397583 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13644551874092367) | > loader_time: 0.0045 (0.004644996913409577)  --> TIME: 2025-02-24 01:36:19 -- STEP: 36674/36938 -- GLOBAL_STEP: 110550 | > loss_text_ce: 0.05023643746972084 (0.06032219448981025) | > loss_mel_ce: 3.5266342163085938 (nan) | > loss: 0.01419393252581358 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13644747800510235) | > loader_time: 0.0045 (0.004644946182945014)  --> TIME: 2025-02-24 01:36:28 -- STEP: 36724/36938 -- GLOBAL_STEP: 110600 | > loss_text_ce: 0.055317338556051254 (0.060317859928685214) | > loss_mel_ce: 3.507509231567383 (nan) | > loss: 0.01413820218294859 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13643741052585115) | > loader_time: 0.0046 (0.004644874744251198)  --> TIME: 2025-02-24 01:36:38 -- STEP: 36774/36938 -- GLOBAL_STEP: 110650 | > loss_text_ce: 0.058552760630846024 (0.06030786110651114) | > loss_mel_ce: 3.4912307262420654 (nan) | > loss: 0.01408644299954176 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13643876100147875) | > loader_time: 0.0049 (0.004645000146200018)  --> TIME: 2025-02-24 01:36:47 -- STEP: 36824/36938 -- GLOBAL_STEP: 110700 | > loss_text_ce: 0.052938733249902725 (0.060300305332055645) | > loss_mel_ce: 3.7543911933898926 (nan) | > loss: 0.01510845310986042 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.1364394777266061) | > loader_time: 0.0048 (0.004644998384252401)  --> TIME: 2025-02-24 01:36:56 -- STEP: 36874/36938 -- GLOBAL_STEP: 110750 | > loss_text_ce: 0.052295949310064316 (0.060293783067509946) | > loss_mel_ce: 3.5346314907073975 (nan) | > loss: 0.014233839698135853 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13643428592275253) | > loader_time: 0.0045 (0.004644929758152528)  --> TIME: 2025-02-24 01:37:05 -- STEP: 36924/36938 -- GLOBAL_STEP: 110800 | > loss_text_ce: 0.0602317713201046 (0.06028791096572125) | > loss_mel_ce: 4.362471580505371 (nan) | > loss: 0.017550410702824593 (nan) | > current_lr: 5e-06 | > step_time: 0.1771 (0.13643481377958105) | > loader_time: 0.0046 (0.004644862351031711)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.0213831627723014 (-0.00012163124462164637) | > avg_loss_text_ce: 0.05231616657116626 (-0.012137102073814611) | > avg_loss_mel_ce: 3.267754073780362 (-0.18713962205565826) | > avg_loss: 3.320070239576963 (-0.19927671935298585) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_110814.pth  > EPOCH: 4/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-24 01:37:33)   --> TIME: 2025-02-24 01:37:40 -- STEP: 36/36938 -- GLOBAL_STEP: 110850 | > loss_text_ce: 0.052744898945093155 (0.054422185031904116) | > loss_mel_ce: 4.0009565353393555 (3.2812475520703526) | > loss: 0.016086118295788765 (0.013236785627668723) | > current_lr: 5e-06 | > step_time: 0.1222 (0.127527740266588) | > loader_time: 0.005 (0.004830784267849392)  --> TIME: 2025-02-24 01:37:49 -- STEP: 86/36938 -- GLOBAL_STEP: 110900 | > loss_text_ce: 0.055052440613508224 (0.05499641462987245) | > loss_mel_ce: 3.5825159549713135 (3.2469175545888587) | > loss: 0.014434796757996082 (0.013102834079777392) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13041695883107743) | > loader_time: 0.0047 (0.004752342091050259)  --> TIME: 2025-02-24 01:37:58 -- STEP: 136/36938 -- GLOBAL_STEP: 110950 | > loss_text_ce: 0.05276013910770416 (0.05566502937718349) | > loss_mel_ce: 4.105266571044922 (3.271166114547454) | > loss: 0.01650010608136654 (0.01320171175629024) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13139229662278124) | > loader_time: 0.0046 (0.004733886788873112)  --> TIME: 2025-02-24 01:38:07 -- STEP: 186/36938 -- GLOBAL_STEP: 111000 | > loss_text_ce: 0.05104334279894829 (0.05560842192461414) | > loss_mel_ce: 3.3441576957702637 (nan) | > loss: 0.013473020866513252 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13270140719670132) | > loader_time: 0.0046 (0.0047272533498784585)  --> TIME: 2025-02-24 01:38:16 -- STEP: 236/36938 -- GLOBAL_STEP: 111050 | > loss_text_ce: 0.04497415944933891 (0.05506425232516002) | > loss_mel_ce: 3.6729047298431396 (nan) | > loss: 0.014753487892448902 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.1327687451394939) | > loader_time: 0.0045 (0.004713362556392865)  --> TIME: 2025-02-24 01:38:26 -- STEP: 286/36938 -- GLOBAL_STEP: 111100 | > loss_text_ce: 0.05053699016571045 (0.054928848845558556) | > loss_mel_ce: 3.4069011211395264 (nan) | > loss: 0.013719992712140083 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13411637369569374) | > loader_time: 0.0048 (0.004711471237502733)  --> TIME: 2025-02-24 01:38:35 -- STEP: 336/36938 -- GLOBAL_STEP: 111150 | > loss_text_ce: 0.06561767309904099 (0.05518575809851641) | > loss_mel_ce: 4.004680156707764 (nan) | > loss: 0.016151975840330124 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13419801280612031) | > loader_time: 0.0046 (0.004708320611999151)  --> TIME: 2025-02-24 01:38:44 -- STEP: 386/36938 -- GLOBAL_STEP: 111200 | > loss_text_ce: 0.058331456035375595 (0.05535659389908139) | > loss_mel_ce: 3.9516289234161377 (nan) | > loss: 0.01591254211962223 (nan) | > current_lr: 5e-06 | > step_time: 0.1876 (0.13381300503725832) | > loader_time: 0.005 (0.004702634761988198)  --> TIME: 2025-02-24 01:38:53 -- STEP: 436/36938 -- GLOBAL_STEP: 111250 | > loss_text_ce: 0.06347232311964035 (0.0553627265959021) | > loss_mel_ce: 3.7217442989349365 (nan) | > loss: 0.015020701102912426 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13398412225443285) | > loader_time: 0.0046 (0.004693006703613009)  --> TIME: 2025-02-24 01:39:02 -- STEP: 486/36938 -- GLOBAL_STEP: 111300 | > loss_text_ce: 0.05630763992667198 (0.055324035907478475) | > loss_mel_ce: 3.5878164768218994 (nan) | > loss: 0.014460810460150242 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13437201064309948) | > loader_time: 0.0048 (0.004688175617422102)  --> TIME: 2025-02-24 01:39:12 -- STEP: 536/36938 -- GLOBAL_STEP: 111350 | > loss_text_ce: 0.03777197375893593 (0.05511910105541125) | > loss_mel_ce: 3.448418617248535 (nan) | > loss: 0.0138340899720788 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13468812428303611) | > loader_time: 0.0046 (0.0046899781298281595)  --> TIME: 2025-02-24 01:39:21 -- STEP: 586/36938 -- GLOBAL_STEP: 111400 | > loss_text_ce: 0.056534793227910995 (0.055134544390612876) | > loss_mel_ce: 2.261737108230591 (nan) | > loss: 0.009199491702020168 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13465090660511836) | > loader_time: 0.0046 (0.004682088467855099)  --> TIME: 2025-02-24 01:39:31 -- STEP: 636/36938 -- GLOBAL_STEP: 111450 | > loss_text_ce: 0.06560340523719788 (0.05516155212004696) | > loss_mel_ce: 4.52348518371582 (nan) | > loss: 0.018210669979453087 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13572017951581455) | > loader_time: 0.0045 (0.0046845412104384735)  --> TIME: 2025-02-24 01:39:40 -- STEP: 686/36938 -- GLOBAL_STEP: 111500 | > loss_text_ce: 0.05321408063173294 (0.055100245287223716) | > loss_mel_ce: 4.361752033233643 (nan) | > loss: 0.01751970686018467 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13587537486073573) | > loader_time: 0.0045 (0.004682308730856673)  --> TIME: 2025-02-24 01:39:50 -- STEP: 736/36938 -- GLOBAL_STEP: 111550 | > loss_text_ce: 0.057265181094408035 (0.055051523002633884) | > loss_mel_ce: 3.2437498569488525 (nan) | > loss: 0.013099267147481441 (nan) | > current_lr: 5e-06 | > step_time: 0.1519 (0.1359936857353086) | > loader_time: 0.0046 (0.004680220199667893)  --> TIME: 2025-02-24 01:39:59 -- STEP: 786/36938 -- GLOBAL_STEP: 111600 | > loss_text_ce: 0.04523676633834839 (0.055115523847630445) | > loss_mel_ce: 4.438686847686768 (nan) | > loss: 0.01779334805905819 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.1364256025270653) | > loader_time: 0.0049 (0.004682569710049624)  --> TIME: 2025-02-24 01:40:09 -- STEP: 836/36938 -- GLOBAL_STEP: 111650 | > loss_text_ce: 0.05435806140303612 (0.05506657193856018) | > loss_mel_ce: 3.444688320159912 (nan) | > loss: 0.013885105028748512 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.13633649257951966) | > loader_time: 0.0048 (0.004681773162914812)  --> TIME: 2025-02-24 01:40:18 -- STEP: 886/36938 -- GLOBAL_STEP: 111700 | > loss_text_ce: 0.056686632335186005 (0.05499516096403379) | > loss_mel_ce: 3.8048486709594727 (nan) | > loss: 0.015323554165661335 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13654773956496916) | > loader_time: 0.0046 (0.004684944454218954)  --> TIME: 2025-02-24 01:40:28 -- STEP: 936/36938 -- GLOBAL_STEP: 111750 | > loss_text_ce: 0.05993657559156418 (0.05509020466333588) | > loss_mel_ce: 3.7196357250213623 (nan) | > loss: 0.014998303726315498 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13656750448748614) | > loader_time: 0.0054 (0.004687641930376368)  --> TIME: 2025-02-24 01:40:37 -- STEP: 986/36938 -- GLOBAL_STEP: 111800 | > loss_text_ce: 0.05776115134358406 (0.05500853586465905) | > loss_mel_ce: 3.1001665592193604 (nan) | > loss: 0.012531460262835026 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.1369046093726015) | > loader_time: 0.0047 (0.004686120073887209)  --> TIME: 2025-02-24 01:40:47 -- STEP: 1036/36938 -- GLOBAL_STEP: 111850 | > loss_text_ce: 0.05855877324938774 (0.055001602979182725) | > loss_mel_ce: 3.584392547607422 (nan) | > loss: 0.014456156641244888 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13691004201712312) | > loader_time: 0.0045 (0.0046876298414694345)  --> TIME: 2025-02-24 01:40:56 -- STEP: 1086/36938 -- GLOBAL_STEP: 111900 | > loss_text_ce: 0.0363161563873291 (0.05492695710839595) | > loss_mel_ce: 0.1419556885957718 (nan) | > loss: 0.0007074279710650444 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13686441079907233) | > loader_time: 0.0045 (0.004688188515974017)  --> TIME: 2025-02-24 01:41:05 -- STEP: 1136/36938 -- GLOBAL_STEP: 111950 | > loss_text_ce: 0.06507255136966705 (0.0549705461402174) | > loss_mel_ce: 0.5857747197151184 (nan) | > loss: 0.0025827274657785892 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13697490243005098) | > loader_time: 0.0044 (0.004687367610528439)  --> TIME: 2025-02-24 01:41:15 -- STEP: 1186/36938 -- GLOBAL_STEP: 112000 | > loss_text_ce: 0.0445081889629364 (0.054944687166438085) | > loss_mel_ce: 3.8886005878448486 (nan) | > loss: 0.015607575885951519 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13704295604667227) | > loader_time: 0.0046 (0.004684739128936202)  --> TIME: 2025-02-24 01:41:24 -- STEP: 1236/36938 -- GLOBAL_STEP: 112050 | > loss_text_ce: 0.05289793387055397 (0.05489304811847437) | > loss_mel_ce: 3.402897834777832 (nan) | > loss: 0.013713476248085499 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13706303645877788) | > loader_time: 0.0045 (0.004685047182064614)  --> TIME: 2025-02-24 01:41:33 -- STEP: 1286/36938 -- GLOBAL_STEP: 112100 | > loss_text_ce: 0.042176954448223114 (0.05482672604487431) | > loss_mel_ce: 3.499722957611084 (nan) | > loss: 0.014055158942937851 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13697125733732998) | > loader_time: 0.0047 (0.004687579942528234)  --> TIME: 2025-02-24 01:41:42 -- STEP: 1336/36938 -- GLOBAL_STEP: 112150 | > loss_text_ce: 0.051138803362846375 (0.054769498404829225) | > loss_mel_ce: 3.341686725616455 (nan) | > loss: 0.013463594019412994 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13677503266734278) | > loader_time: 0.0044 (0.004685618741783553)  --> TIME: 2025-02-24 01:41:52 -- STEP: 1386/36938 -- GLOBAL_STEP: 112200 | > loss_text_ce: 0.05038168653845787 (0.054751942442222035) | > loss_mel_ce: 0.6150919795036316 (nan) | > loss: 0.002640768652781844 (nan) | > current_lr: 5e-06 | > step_time: 0.1773 (0.1368186786921337) | > loader_time: 0.0057 (0.004685840049347322)  --> TIME: 2025-02-24 01:42:01 -- STEP: 1436/36938 -- GLOBAL_STEP: 112250 | > loss_text_ce: 0.06980493664741516 (0.05475211078638316) | > loss_mel_ce: 0.050311747938394547 (nan) | > loss: 0.000476653513032943 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.1368936240507034) | > loader_time: 0.0045 (0.004687157181976233)  --> TIME: 2025-02-24 01:42:10 -- STEP: 1486/36938 -- GLOBAL_STEP: 112300 | > loss_text_ce: 0.054450392723083496 (0.05475452234956005) | > loss_mel_ce: 2.2304415702819824 (nan) | > loss: 0.009067032486200333 (nan) | > current_lr: 5e-06 | > step_time: 0.1521 (0.13676938763537594) | > loader_time: 0.0045 (0.004683005537030516)  --> TIME: 2025-02-24 01:42:19 -- STEP: 1536/36938 -- GLOBAL_STEP: 112350 | > loss_text_ce: 0.053377583622932434 (0.0547059579621418) | > loss_mel_ce: 3.5686283111572266 (nan) | > loss: 0.014373039826750755 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13674698630347848) | > loader_time: 0.0045 (0.00467996687317888)  --> TIME: 2025-02-24 01:42:29 -- STEP: 1586/36938 -- GLOBAL_STEP: 112400 | > loss_text_ce: 0.05849279835820198 (0.05471593168567661) | > loss_mel_ce: 2.202629566192627 (nan) | > loss: 0.008972709067165852 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13669137350397476) | > loader_time: 0.0045 (0.004676961958934502)  --> TIME: 2025-02-24 01:42:38 -- STEP: 1636/36938 -- GLOBAL_STEP: 112450 | > loss_text_ce: 0.0529920719563961 (0.05474000849276049) | > loss_mel_ce: 3.1118507385253906 (nan) | > loss: 0.012558900751173496 (nan) | > current_lr: 5e-06 | > step_time: 0.1441 (0.13666636174348845) | > loader_time: 0.0046 (0.004674229616057901)  --> TIME: 2025-02-24 01:42:47 -- STEP: 1686/36938 -- GLOBAL_STEP: 112500 | > loss_text_ce: 0.06086812540888786 (0.054728795578736396) | > loss_mel_ce: 4.293689250946045 (nan) | > loss: 0.017279991880059242 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13665184162972502) | > loader_time: 0.0046 (0.004673089138270837)  --> TIME: 2025-02-24 01:42:57 -- STEP: 1736/36938 -- GLOBAL_STEP: 112550 | > loss_text_ce: 0.05390797555446625 (0.05469845492212519) | > loss_mel_ce: 4.112401962280273 (nan) | > loss: 0.016532976180315018 (nan) | > current_lr: 5e-06 | > step_time: 0.1895 (0.1367103563201041) | > loader_time: 0.0048 (0.004674384396197058)  --> TIME: 2025-02-24 01:43:06 -- STEP: 1786/36938 -- GLOBAL_STEP: 112600 | > loss_text_ce: 0.06244424358010292 (0.05469371673357978) | > loss_mel_ce: 3.501856565475464 (nan) | > loss: 0.01414405182003975 (nan) | > current_lr: 5e-06 | > step_time: 0.1264 (0.13690970926509027) | > loader_time: 0.0045 (0.004673516630191951)  --> TIME: 2025-02-24 01:43:16 -- STEP: 1836/36938 -- GLOBAL_STEP: 112650 | > loss_text_ce: 0.048523690551519394 (0.05471199964556621) | > loss_mel_ce: 3.4624507427215576 (nan) | > loss: 0.01393243856728077 (nan) | > current_lr: 5e-06 | > step_time: 0.1853 (0.13696975671647635) | > loader_time: 0.005 (0.004672988437619342)  --> TIME: 2025-02-24 01:43:25 -- STEP: 1886/36938 -- GLOBAL_STEP: 112700 | > loss_text_ce: 0.07199080288410187 (0.05475185288082754) | > loss_mel_ce: 0.045204855501651764 (nan) | > loss: 0.00046506215585395694 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13695861817923188) | > loader_time: 0.0064 (0.004671289331713257)  --> TIME: 2025-02-24 01:43:34 -- STEP: 1936/36938 -- GLOBAL_STEP: 112750 | > loss_text_ce: 0.04439796507358551 (0.05473233423626802) | > loss_mel_ce: 3.403843402862549 (nan) | > loss: 0.013683498837053776 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.1368695664996939) | > loader_time: 0.0045 (0.004668478384490839)  --> TIME: 2025-02-24 01:43:44 -- STEP: 1986/36938 -- GLOBAL_STEP: 112800 | > loss_text_ce: 0.05272888392210007 (0.054681608462771834) | > loss_mel_ce: 3.2818315029144287 (nan) | > loss: 0.013232382945716381 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13690839325193152) | > loader_time: 0.0046 (0.004668108886342273)  --> TIME: 2025-02-24 01:43:53 -- STEP: 2036/36938 -- GLOBAL_STEP: 112850 | > loss_text_ce: 0.05294833704829216 (0.05471994992060134) | > loss_mel_ce: 3.4732024669647217 (nan) | > loss: 0.01399266254156828 (nan) | > current_lr: 5e-06 | > step_time: 0.1634 (0.1368533644554419) | > loader_time: 0.0048 (0.004665167369168023)  --> TIME: 2025-02-24 01:44:02 -- STEP: 2086/36938 -- GLOBAL_STEP: 112900 | > loss_text_ce: 0.05292888358235359 (0.05469795613088004) | > loss_mel_ce: 3.44429349899292 (nan) | > loss: 0.013877867721021175 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13686070163320793) | > loader_time: 0.0045 (0.004664630012109888)  --> TIME: 2025-02-24 01:44:11 -- STEP: 2136/36938 -- GLOBAL_STEP: 112950 | > loss_text_ce: 0.06434442847967148 (0.054681613409982385) | > loss_mel_ce: 1.2157796621322632 (nan) | > loss: 0.005079857539385557 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13674587167604102) | > loader_time: 0.0045 (0.00466232331058059)  --> TIME: 2025-02-24 01:44:21 -- STEP: 2186/36938 -- GLOBAL_STEP: 113000 | > loss_text_ce: 0.05089382454752922 (0.0546958511379996) | > loss_mel_ce: 3.0853145122528076 (nan) | > loss: 0.012445271946489811 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13680347268557366) | > loader_time: 0.0046 (0.004660305252668511)  --> TIME: 2025-02-24 01:44:30 -- STEP: 2236/36938 -- GLOBAL_STEP: 113050 | > loss_text_ce: 0.05526132881641388 (0.05466189779164616) | > loss_mel_ce: 4.098320960998535 (nan) | > loss: 0.01648246869444847 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.1366568345289961) | > loader_time: 0.0048 (0.004658315497348895)  --> TIME: 2025-02-24 01:44:39 -- STEP: 2286/36938 -- GLOBAL_STEP: 113100 | > loss_text_ce: 0.05435265973210335 (0.05467098558440552) | > loss_mel_ce: 3.085249662399292 (nan) | > loss: 0.0124587407335639 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13665930993287864) | > loader_time: 0.0048 (0.004659597642152635)  --> TIME: 2025-02-24 01:44:48 -- STEP: 2336/36938 -- GLOBAL_STEP: 113150 | > loss_text_ce: 0.06646008044481277 (0.054682226806930044) | > loss_mel_ce: 3.8486216068267822 (nan) | > loss: 0.015536039136350155 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13672089617546254) | > loader_time: 0.0045 (0.004661360525921593)  --> TIME: 2025-02-24 01:44:58 -- STEP: 2386/36938 -- GLOBAL_STEP: 113200 | > loss_text_ce: 0.04630005732178688 (0.05467115404444985) | > loss_mel_ce: 4.405205726623535 (nan) | > loss: 0.017664706334471703 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13669270951510035) | > loader_time: 0.0047 (0.00466317303122738)  --> TIME: 2025-02-24 01:45:07 -- STEP: 2436/36938 -- GLOBAL_STEP: 113250 | > loss_text_ce: 0.0463796891272068 (0.0546690052610704) | > loss_mel_ce: 3.850275993347168 (nan) | > loss: 0.015462921001017094 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13674108862681103) | > loader_time: 0.0052 (0.004664162599002976)  --> TIME: 2025-02-24 01:45:16 -- STEP: 2486/36938 -- GLOBAL_STEP: 113300 | > loss_text_ce: 0.046785227954387665 (0.05468044360187007) | > loss_mel_ce: 4.089282989501953 (nan) | > loss: 0.016412969678640366 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1366694734063915) | > loader_time: 0.0045 (0.0046638307893650185)  --> TIME: 2025-02-24 01:45:25 -- STEP: 2536/36938 -- GLOBAL_STEP: 113350 | > loss_text_ce: 0.04825134575366974 (0.054722305183072646) | > loss_mel_ce: 3.4196410179138184 (nan) | > loss: 0.01376147847622633 (nan) | > current_lr: 5e-06 | > step_time: 0.1782 (0.13663353265647957) | > loader_time: 0.0054 (0.004662569671023148)  --> TIME: 2025-02-24 01:45:35 -- STEP: 2586/36938 -- GLOBAL_STEP: 113400 | > loss_text_ce: 0.06643016636371613 (0.054722907466775) | > loss_mel_ce: 4.169790744781494 (nan) | > loss: 0.016810400411486626 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.1366422656701681) | > loader_time: 0.0046 (0.004663026803054271)  --> TIME: 2025-02-24 01:45:44 -- STEP: 2636/36938 -- GLOBAL_STEP: 113450 | > loss_text_ce: 0.057820968329906464 (0.05468508853309974) | > loss_mel_ce: 3.5932087898254395 (nan) | > loss: 0.014488214626908302 (nan) | > current_lr: 5e-06 | > step_time: 0.1599 (0.1366848077325425) | > loader_time: 0.0046 (0.004661014933868308)  --> TIME: 2025-02-24 01:45:53 -- STEP: 2686/36938 -- GLOBAL_STEP: 113500 | > loss_text_ce: 0.05296389386057854 (0.05467950082741182) | > loss_mel_ce: 3.7105050086975098 (nan) | > loss: 0.014934401959180832 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.1366361840305622) | > loader_time: 0.0045 (0.00466046738074239)  --> TIME: 2025-02-24 01:46:03 -- STEP: 2736/36938 -- GLOBAL_STEP: 113550 | > loss_text_ce: 0.04719604551792145 (0.05465708170916709) | > loss_mel_ce: 3.637641191482544 (nan) | > loss: 0.014622371643781662 (nan) | > current_lr: 5e-06 | > step_time: 0.1452 (0.1366544547136759) | > loader_time: 0.0048 (0.004659948467511176)  --> TIME: 2025-02-24 01:46:12 -- STEP: 2786/36938 -- GLOBAL_STEP: 113600 | > loss_text_ce: 0.052976515144109726 (0.05464244993672318) | > loss_mel_ce: 1.9751170873641968 (nan) | > loss: 0.008047990500926971 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13664808062795036) | > loader_time: 0.0046 (0.004660253377587173)  --> TIME: 2025-02-24 01:46:21 -- STEP: 2836/36938 -- GLOBAL_STEP: 113650 | > loss_text_ce: 0.042585019022226334 (0.054614845495828644) | > loss_mel_ce: 3.0642757415771484 (nan) | > loss: 0.012328813783824444 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.136719617588059) | > loader_time: 0.0045 (0.004659067340226697)  --> TIME: 2025-02-24 01:46:31 -- STEP: 2886/36938 -- GLOBAL_STEP: 113700 | > loss_text_ce: 0.04904340207576752 (0.05460869938287461) | > loss_mel_ce: 0.02215576171875 (nan) | > loss: 0.00028253637719899416 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13664407591314143) | > loader_time: 0.0044 (0.004657564358172602)  --> TIME: 2025-02-24 01:46:40 -- STEP: 2936/36938 -- GLOBAL_STEP: 113750 | > loss_text_ce: 0.06014832854270935 (0.05461061106073574) | > loss_mel_ce: 2.8990962505340576 (nan) | > loss: 0.011743034236133099 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.1367087546908561) | > loader_time: 0.0049 (0.004658283670851568)  --> TIME: 2025-02-24 01:46:49 -- STEP: 2986/36938 -- GLOBAL_STEP: 113800 | > loss_text_ce: 0.05689680948853493 (0.054605229631720036) | > loss_mel_ce: 3.91229510307312 (nan) | > loss: 0.015750762075185776 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.13670862152204352) | > loader_time: 0.0046 (0.004660332594154178)  --> TIME: 2025-02-24 01:46:58 -- STEP: 3036/36938 -- GLOBAL_STEP: 113850 | > loss_text_ce: 0.06410592049360275 (0.05459741331521562) | > loss_mel_ce: 4.12701416015625 (nan) | > loss: 0.01663143001496792 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13664045344856565) | > loader_time: 0.0045 (0.004661200388932246)  --> TIME: 2025-02-24 01:47:08 -- STEP: 3086/36938 -- GLOBAL_STEP: 113900 | > loss_text_ce: 0.0532875657081604 (0.054590824188100795) | > loss_mel_ce: 3.5412561893463135 (nan) | > loss: 0.014264062978327274 (nan) | > current_lr: 5e-06 | > step_time: 0.1867 (0.13663012720282308) | > loader_time: 0.0048 (0.004660766385857795)  --> TIME: 2025-02-24 01:47:17 -- STEP: 3136/36938 -- GLOBAL_STEP: 113950 | > loss_text_ce: 0.07083892077207565 (0.05458409266786325) | > loss_mel_ce: 3.4866626262664795 (nan) | > loss: 0.014117070473730564 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13658854144872448) | > loader_time: 0.0045 (0.004659444701914872)  --> TIME: 2025-02-24 01:47:26 -- STEP: 3186/36938 -- GLOBAL_STEP: 114000 | > loss_text_ce: 0.0468805730342865 (0.054596766005740795) | > loss_mel_ce: 3.450639009475708 (nan) | > loss: 0.013879046775400639 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13656296059536024) | > loader_time: 0.0047 (0.00465976166710191)  --> TIME: 2025-02-24 01:47:36 -- STEP: 3236/36938 -- GLOBAL_STEP: 114050 | > loss_text_ce: 0.05954502150416374 (0.05455603328279053) | > loss_mel_ce: 3.6846489906311035 (nan) | > loss: 0.014857913367450237 (nan) | > current_lr: 5e-06 | > step_time: 0.1886 (0.13661823107668736) | > loader_time: 0.0048 (0.004660694428221097)  --> TIME: 2025-02-24 01:47:45 -- STEP: 3286/36938 -- GLOBAL_STEP: 114100 | > loss_text_ce: 0.05647886544466019 (0.0545495720857973) | > loss_mel_ce: 3.557690382003784 (nan) | > loss: 0.014341942965984344 (nan) | > current_lr: 5e-06 | > step_time: 0.184 (0.13662614672922618) | > loader_time: 0.0047 (0.004662692800790007)  --> TIME: 2025-02-24 01:47:54 -- STEP: 3336/36938 -- GLOBAL_STEP: 114150 | > loss_text_ce: 0.05831456556916237 (0.0545416776095327) | > loss_mel_ce: 3.4100372791290283 (nan) | > loss: 0.013763302005827427 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13657968400193646) | > loader_time: 0.0045 (0.0046634566154983015)  --> TIME: 2025-02-24 01:48:03 -- STEP: 3386/36938 -- GLOBAL_STEP: 114200 | > loss_text_ce: 0.05028010159730911 (0.05453479603566646) | > loss_mel_ce: 4.106840133666992 (nan) | > loss: 0.016496509313583374 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13660593738375662) | > loader_time: 0.0046 (0.004664726181278118)  --> TIME: 2025-02-24 01:48:13 -- STEP: 3436/36938 -- GLOBAL_STEP: 114250 | > loss_text_ce: 0.06034328415989876 (0.05453072291960433) | > loss_mel_ce: 3.304251194000244 (nan) | > loss: 0.013351566158235073 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.13666623650385412) | > loader_time: 0.0046 (0.004663832182101509)  --> TIME: 2025-02-24 01:48:23 -- STEP: 3486/36938 -- GLOBAL_STEP: 114300 | > loss_text_ce: 0.04204162582755089 (0.054512734710315265) | > loss_mel_ce: 0.4021106958389282 (nan) | > loss: 0.0017625093460083008 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13669853961406192) | > loader_time: 0.0046 (0.004664415678482793)  --> TIME: 2025-02-24 01:48:32 -- STEP: 3536/36938 -- GLOBAL_STEP: 114350 | > loss_text_ce: 0.05213989317417145 (0.05448534121163761) | > loss_mel_ce: 2.652212619781494 (nan) | > loss: 0.010731558315455914 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13662537433443034) | > loader_time: 0.0045 (0.00466342728871565)  --> TIME: 2025-02-24 01:48:41 -- STEP: 3586/36938 -- GLOBAL_STEP: 114400 | > loss_text_ce: 0.05057539790868759 (0.054481624297868306) | > loss_mel_ce: 2.7602381706237793 (nan) | > loss: 0.011154022999107838 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13665685365137561) | > loader_time: 0.0045 (0.004662506286484925)  --> TIME: 2025-02-24 01:48:50 -- STEP: 3636/36938 -- GLOBAL_STEP: 114450 | > loss_text_ce: 0.06225410848855972 (0.05449056552212819) | > loss_mel_ce: 3.1359362602233887 (nan) | > loss: 0.01269123237580061 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13666680766673256) | > loader_time: 0.0048 (0.004661153907691941)  --> TIME: 2025-02-24 01:49:00 -- STEP: 3686/36938 -- GLOBAL_STEP: 114500 | > loss_text_ce: 0.05149082466959953 (0.05448501061383336) | > loss_mel_ce: 3.576629400253296 (nan) | > loss: 0.014397303573787212 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13666074979712778) | > loader_time: 0.0045 (0.004660106455574713)  --> TIME: 2025-02-24 01:49:09 -- STEP: 3736/36938 -- GLOBAL_STEP: 114550 | > loss_text_ce: 0.043047819286584854 (0.05445195969126367) | > loss_mel_ce: 3.3084769248962402 (nan) | > loss: 0.013299702666699886 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.13661186949088702) | > loader_time: 0.0048 (0.004658986592956472)  --> TIME: 2025-02-24 01:49:18 -- STEP: 3786/36938 -- GLOBAL_STEP: 114600 | > loss_text_ce: 0.05458640307188034 (0.05442960045167095) | > loss_mel_ce: 3.059849500656128 (nan) | > loss: 0.012358873151242733 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.1365457927494282) | > loader_time: 0.0045 (0.0046579203653511845)  --> TIME: 2025-02-24 01:49:27 -- STEP: 3836/36938 -- GLOBAL_STEP: 114650 | > loss_text_ce: 0.05081620067358017 (0.05443927831996359) | > loss_mel_ce: 0.03289794921875 (nan) | > loss: 0.0003321990370750427 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13656896309260916) | > loader_time: 0.0045 (0.0046576290110726245)  --> TIME: 2025-02-24 01:49:36 -- STEP: 3886/36938 -- GLOBAL_STEP: 114700 | > loss_text_ce: 0.06675688922405243 (0.05442428254444252) | > loss_mel_ce: 3.3674020767211914 (nan) | > loss: 0.013627615757286549 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13650227280173557) | > loader_time: 0.0046 (0.0046565071921709274)  --> TIME: 2025-02-24 01:49:45 -- STEP: 3936/36938 -- GLOBAL_STEP: 114750 | > loss_text_ce: 0.04901150241494179 (0.054397432387208296) | > loss_mel_ce: 4.110671520233154 (nan) | > loss: 0.016506681218743324 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13649213368572852) | > loader_time: 0.0046 (0.004655727647184351)  --> TIME: 2025-02-24 01:49:54 -- STEP: 3986/36938 -- GLOBAL_STEP: 114800 | > loss_text_ce: 0.05458839237689972 (0.05439626660461527) | > loss_mel_ce: 3.579801321029663 (nan) | > loss: 0.014422181993722916 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13645864328546148) | > loader_time: 0.0045 (0.0046548679493446125)  --> TIME: 2025-02-24 01:50:04 -- STEP: 4036/36938 -- GLOBAL_STEP: 114850 | > loss_text_ce: 0.0680183470249176 (0.05438725073610471) | > loss_mel_ce: 2.201399326324463 (nan) | > loss: 0.009005626663565636 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1364871831786173) | > loader_time: 0.0045 (0.004654194306807423)  --> TIME: 2025-02-24 01:50:13 -- STEP: 4086/36938 -- GLOBAL_STEP: 114900 | > loss_text_ce: 0.053176149725914 (0.05439802634015102) | > loss_mel_ce: 3.7906322479248047 (nan) | > loss: 0.015253208577632904 (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.13649804231528856) | > loader_time: 0.0047 (0.004653472732342282)  --> TIME: 2025-02-24 01:50:22 -- STEP: 4136/36938 -- GLOBAL_STEP: 114950 | > loss_text_ce: 0.05717165023088455 (0.05438369866712756) | > loss_mel_ce: 4.017068862915039 (nan) | > loss: 0.016167622059583664 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13645738184567358) | > loader_time: 0.0046 (0.004652281506149169)  --> TIME: 2025-02-24 01:50:31 -- STEP: 4186/36938 -- GLOBAL_STEP: 115000 | > loss_text_ce: 0.0526297502219677 (0.0543769607511765) | > loss_mel_ce: 3.7086570262908936 (nan) | > loss: 0.014925741590559483 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.13636598208847092) | > loader_time: 0.0045 (0.004650959089163435)  --> TIME: 2025-02-24 01:50:40 -- STEP: 4236/36938 -- GLOBAL_STEP: 115050 | > loss_text_ce: 0.05869656056165695 (0.05439441055815747) | > loss_mel_ce: 4.064598560333252 (nan) | > loss: 0.016362283378839493 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13640275560987805) | > loader_time: 0.0045 (0.004650322810110893)  --> TIME: 2025-02-24 01:50:50 -- STEP: 4286/36938 -- GLOBAL_STEP: 115100 | > loss_text_ce: 0.05737824738025665 (0.05439386712200331) | > loss_mel_ce: 3.68363094329834 (nan) | > loss: 0.014845275320112705 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.1364416362968941) | > loader_time: 0.0046 (0.004649938460103937)  --> TIME: 2025-02-24 01:50:59 -- STEP: 4336/36938 -- GLOBAL_STEP: 115150 | > loss_text_ce: 0.06575828045606613 (0.05439299614790731) | > loss_mel_ce: 0.142333984375 (nan) | > loss: 0.0008257630397565663 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13641517441651044) | > loader_time: 0.0046 (0.004649445524514813)  --> TIME: 2025-02-24 01:51:09 -- STEP: 4386/36938 -- GLOBAL_STEP: 115200 | > loss_text_ce: 0.04428710788488388 (0.054400494932020795) | > loss_mel_ce: 3.467719793319702 (nan) | > loss: 0.013936536386609077 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13649011968316088) | > loader_time: 0.0045 (0.004649335643335169)  --> TIME: 2025-02-24 01:51:19 -- STEP: 4436/36938 -- GLOBAL_STEP: 115250 | > loss_text_ce: 0.049644701182842255 (0.054395162911936155) | > loss_mel_ce: 4.094467639923096 (nan) | > loss: 0.016444889828562737 (nan) | > current_lr: 5e-06 | > step_time: 0.1844 (0.13656895546573464) | > loader_time: 0.0048 (0.004649137945407347)  --> TIME: 2025-02-24 01:51:28 -- STEP: 4486/36938 -- GLOBAL_STEP: 115300 | > loss_text_ce: 0.054493170231580734 (0.054383462408704515) | > loss_mel_ce: 3.220323085784912 (nan) | > loss: 0.012995303608477116 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1366069407745816) | > loader_time: 0.0044 (0.004648296683163067)  --> TIME: 2025-02-24 01:51:37 -- STEP: 4536/36938 -- GLOBAL_STEP: 115350 | > loss_text_ce: 0.05411989241838455 (0.05439603088072105) | > loss_mel_ce: 3.5074994564056396 (nan) | > loss: 0.014133410528302193 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.1365644532326457) | > loader_time: 0.0047 (0.004647037129343282)  --> TIME: 2025-02-24 01:51:46 -- STEP: 4586/36938 -- GLOBAL_STEP: 115400 | > loss_text_ce: 0.04867974668741226 (0.05438336542764061) | > loss_mel_ce: 3.9977755546569824 (nan) | > loss: 0.016057362779974937 (nan) | > current_lr: 5e-06 | > step_time: 0.1869 (0.13655809180163625) | > loader_time: 0.0052 (0.004645852246140987)  --> TIME: 2025-02-24 01:51:56 -- STEP: 4636/36938 -- GLOBAL_STEP: 115450 | > loss_text_ce: 0.04920476675033569 (0.05438004109391345) | > loss_mel_ce: 3.4048314094543457 (nan) | > loss: 0.013706494122743607 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13654576976099742) | > loader_time: 0.0045 (0.004644750005937621)  --> TIME: 2025-02-24 01:52:05 -- STEP: 4686/36938 -- GLOBAL_STEP: 115500 | > loss_text_ce: 0.052538078278303146 (0.05436360486475031) | > loss_mel_ce: 3.87803316116333 (nan) | > loss: 0.015597506426274776 (nan) | > current_lr: 5e-06 | > step_time: 0.1905 (0.13655660735507624) | > loader_time: 0.0049 (0.004643860709906327)  --> TIME: 2025-02-24 01:52:14 -- STEP: 4736/36938 -- GLOBAL_STEP: 115550 | > loss_text_ce: 0.06211211532354355 (0.054362942785578405) | > loss_mel_ce: 3.0228312015533447 (nan) | > loss: 0.012241839431226254 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13655771737968594) | > loader_time: 0.0045 (0.00464300111540265)  --> TIME: 2025-02-24 01:52:23 -- STEP: 4786/36938 -- GLOBAL_STEP: 115600 | > loss_text_ce: 0.05812051519751549 (0.05434973743879242) | > loss_mel_ce: 2.569476366043091 (nan) | > loss: 0.010426972061395645 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13656516718754835) | > loader_time: 0.0045 (0.004642587100617605)  --> TIME: 2025-02-24 01:52:33 -- STEP: 4836/36938 -- GLOBAL_STEP: 115650 | > loss_text_ce: 0.057250652462244034 (0.05434767486615379) | > loss_mel_ce: 3.429133892059326 (nan) | > loss: 0.013834860175848007 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.136584682243732) | > loader_time: 0.0046 (0.004642208713061351)  --> TIME: 2025-02-24 01:52:42 -- STEP: 4886/36938 -- GLOBAL_STEP: 115700 | > loss_text_ce: 0.04288898780941963 (0.05434093992307788) | > loss_mel_ce: 3.6427764892578125 (nan) | > loss: 0.014625657349824905 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1365666062530317) | > loader_time: 0.0046 (0.004641809377619529)  --> TIME: 2025-02-24 01:52:51 -- STEP: 4936/36938 -- GLOBAL_STEP: 115750 | > loss_text_ce: 0.058171432465314865 (0.054350047174545314) | > loss_mel_ce: 3.5478599071502686 (nan) | > loss: 0.014309649355709553 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.1365001489026426) | > loader_time: 0.0046 (0.004641297329186995)  --> TIME: 2025-02-24 01:53:00 -- STEP: 4986/36938 -- GLOBAL_STEP: 115800 | > loss_text_ce: 0.052736274898052216 (0.05435017513233019) | > loss_mel_ce: 3.6260321140289307 (nan) | > loss: 0.01459828857332468 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.1364923879605815) | > loader_time: 0.0048 (0.004640961525385691)  --> TIME: 2025-02-24 01:53:09 -- STEP: 5036/36938 -- GLOBAL_STEP: 115850 | > loss_text_ce: 0.05745060369372368 (0.054368318074824465) | > loss_mel_ce: 2.4677608013153076 (nan) | > loss: 0.010020680725574493 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13648768546942966) | > loader_time: 0.0045 (0.004640682999154123)  --> TIME: 2025-02-24 01:53:19 -- STEP: 5086/36938 -- GLOBAL_STEP: 115900 | > loss_text_ce: 0.05135592818260193 (0.05436573709488337) | > loss_mel_ce: 3.6487698554992676 (nan) | > loss: 0.014683038927614689 (nan) | > current_lr: 5e-06 | > step_time: 0.1954 (0.1364720229600013) | > loader_time: 0.0048 (0.004640057852921676)  --> TIME: 2025-02-24 01:53:28 -- STEP: 5136/36938 -- GLOBAL_STEP: 115950 | > loss_text_ce: 0.04404306411743164 (0.05436585482276651) | > loss_mel_ce: 3.6084346771240234 (nan) | > loss: 0.014493959955871105 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13653148418275) | > loader_time: 0.0045 (0.0046396899353306435)  --> TIME: 2025-02-24 01:53:37 -- STEP: 5186/36938 -- GLOBAL_STEP: 116000 | > loss_text_ce: 0.05318775400519371 (0.05436499113259378) | > loss_mel_ce: 4.038891315460205 (nan) | > loss: 0.016238410025835037 (nan) | > current_lr: 5e-06 | > step_time: 0.1809 (0.1364958075267027) | > loader_time: 0.0048 (0.004639986533250656)  --> TIME: 2025-02-24 01:53:47 -- STEP: 5236/36938 -- GLOBAL_STEP: 116050 | > loss_text_ce: 0.04707614332437515 (0.054338018579594936) | > loss_mel_ce: 2.818324089050293 (nan) | > loss: 0.011370636522769928 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.1364927015475774) | > loader_time: 0.0046 (0.004639407074843018)  --> TIME: 2025-02-24 01:53:56 -- STEP: 5286/36938 -- GLOBAL_STEP: 116100 | > loss_text_ce: 0.06473111361265182 (0.05433457780691414) | > loss_mel_ce: 3.6433703899383545 (nan) | > loss: 0.014714688993990421 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13645125621654206) | > loader_time: 0.0045 (0.004639345680363805)  --> TIME: 2025-02-24 01:54:05 -- STEP: 5336/36938 -- GLOBAL_STEP: 116150 | > loss_text_ce: 0.0562891848385334 (0.05431702698097806) | > loss_mel_ce: 3.2781660556793213 (nan) | > loss: 0.013231965713202953 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.136406933796638) | > loader_time: 0.0046 (0.004638656906817067)  --> TIME: 2025-02-24 01:54:14 -- STEP: 5386/36938 -- GLOBAL_STEP: 116200 | > loss_text_ce: 0.056956224143505096 (0.054305806171911226) | > loss_mel_ce: 3.2397544384002686 (nan) | > loss: 0.013082185760140419 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13636853404528487) | > loader_time: 0.0045 (0.0046384728977417015)  --> TIME: 2025-02-24 01:54:23 -- STEP: 5436/36938 -- GLOBAL_STEP: 116250 | > loss_text_ce: 0.05464506521821022 (0.054278200882883786) | > loss_mel_ce: 2.299443006515503 (nan) | > loss: 0.009341619908809662 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13636957506996644) | > loader_time: 0.0047 (0.004637659297785898)  --> TIME: 2025-02-24 01:54:32 -- STEP: 5486/36938 -- GLOBAL_STEP: 116300 | > loss_text_ce: 0.048109062016010284 (0.054269619847638) | > loss_mel_ce: 3.894862651824951 (nan) | > loss: 0.015646714717149734 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13637338308709154) | > loader_time: 0.0049 (0.004636797294797464)  --> TIME: 2025-02-24 01:54:42 -- STEP: 5536/36938 -- GLOBAL_STEP: 116350 | > loss_text_ce: 0.055884603410959244 (0.054268322773331594) | > loss_mel_ce: 2.8214964866638184 (nan) | > loss: 0.011418179608881474 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.1363934920185564) | > loader_time: 0.005 (0.004635952671483737)  --> TIME: 2025-02-24 01:54:51 -- STEP: 5586/36938 -- GLOBAL_STEP: 116400 | > loss_text_ce: 0.04852347820997238 (0.05425445479032411) | > loss_mel_ce: 3.378058433532715 (nan) | > loss: 0.013597548007965088 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.13638297255816886) | > loader_time: 0.0046 (0.004635114973691703)  --> TIME: 2025-02-24 01:55:00 -- STEP: 5636/36938 -- GLOBAL_STEP: 116450 | > loss_text_ce: 0.03104359842836857 (0.054232661514176154) | > loss_mel_ce: 3.614957571029663 (nan) | > loss: 0.014468259178102016 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13636802027968345) | > loader_time: 0.0044 (0.004634733442045233)  --> TIME: 2025-02-24 01:55:09 -- STEP: 5686/36938 -- GLOBAL_STEP: 116500 | > loss_text_ce: 0.05748943239450455 (0.054216089266648164) | > loss_mel_ce: 3.9120683670043945 (nan) | > loss: 0.015752214938402176 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.1363413322990813) | > loader_time: 0.0046 (0.004634495608369667)  --> TIME: 2025-02-24 01:55:18 -- STEP: 5736/36938 -- GLOBAL_STEP: 116550 | > loss_text_ce: 0.041133854538202286 (0.05422329648483591) | > loss_mel_ce: 4.021319389343262 (nan) | > loss: 0.016120847314596176 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1363288106363025) | > loader_time: 0.0045 (0.004634346797376471)  --> TIME: 2025-02-24 01:55:27 -- STEP: 5786/36938 -- GLOBAL_STEP: 116600 | > loss_text_ce: 0.04766470938920975 (0.054227463020906086) | > loss_mel_ce: 3.7496607303619385 (nan) | > loss: 0.015068752691149712 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.13627215862439018) | > loader_time: 0.0046 (0.0046339737598433415)  --> TIME: 2025-02-24 01:55:36 -- STEP: 5836/36938 -- GLOBAL_STEP: 116650 | > loss_text_ce: 0.04603859782218933 (0.05423415304851642) | > loss_mel_ce: 3.2104175090789795 (nan) | > loss: 0.012922445312142372 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13626496754254774) | > loader_time: 0.0045 (0.004633718602374584)  --> TIME: 2025-02-24 01:55:46 -- STEP: 5886/36938 -- GLOBAL_STEP: 116700 | > loss_text_ce: 0.05612972751259804 (0.05422214049253533) | > loss_mel_ce: 4.0428290367126465 (nan) | > loss: 0.016265710815787315 (nan) | > current_lr: 5e-06 | > step_time: 0.1435 (0.13632473968301081) | > loader_time: 0.0048 (0.004634026722805915)  --> TIME: 2025-02-24 01:55:56 -- STEP: 5936/36938 -- GLOBAL_STEP: 116750 | > loss_text_ce: 0.05002007260918617 (0.0542130380560772) | > loss_mel_ce: 3.0786244869232178 (nan) | > loss: 0.012415256351232529 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13634076927389438) | > loader_time: 0.0045 (0.004634010864075271)  --> TIME: 2025-02-24 01:56:05 -- STEP: 5986/36938 -- GLOBAL_STEP: 116800 | > loss_text_ce: 0.04712480306625366 (0.05419487253328081) | > loss_mel_ce: 0.2582198679447174 (nan) | > loss: 0.0012116852449253201 (nan) | > current_lr: 5e-06 | > step_time: 0.1487 (0.13639699823275675) | > loader_time: 0.0046 (0.004634256949203262)  --> TIME: 2025-02-24 01:56:15 -- STEP: 6036/36938 -- GLOBAL_STEP: 116850 | > loss_text_ce: 0.054897431284189224 (0.05418928658387286) | > loss_mel_ce: 4.615530490875244 (nan) | > loss: 0.01853344403207302 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13638258742685774) | > loader_time: 0.0046 (0.004634622551100834)  --> TIME: 2025-02-24 01:56:24 -- STEP: 6086/36938 -- GLOBAL_STEP: 116900 | > loss_text_ce: 0.05089737847447395 (0.05418985888681451) | > loss_mel_ce: 3.6069135665893555 (nan) | > loss: 0.014515123330056667 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13634206137686974) | > loader_time: 0.0046 (0.004634219174944459)  --> TIME: 2025-02-24 01:56:33 -- STEP: 6136/36938 -- GLOBAL_STEP: 116950 | > loss_text_ce: 0.05161135643720627 (0.0541910408796303) | > loss_mel_ce: 2.105421781539917 (nan) | > loss: 0.008559656329452991 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13637790993108614) | > loader_time: 0.0046 (0.00463442525776608)  --> TIME: 2025-02-24 01:56:42 -- STEP: 6186/36938 -- GLOBAL_STEP: 117000 | > loss_text_ce: 0.0597260519862175 (0.05419815125951355) | > loss_mel_ce: 3.069899082183838 (nan) | > loss: 0.012419147416949272 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1363790806851418) | > loader_time: 0.0045 (0.004634064144969464)  --> TIME: 2025-02-24 01:56:52 -- STEP: 6236/36938 -- GLOBAL_STEP: 117050 | > loss_text_ce: 0.045899633318185806 (0.05421129686906073) | > loss_mel_ce: 2.666903257369995 (nan) | > loss: 0.010765091516077518 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.136382292982399) | > loader_time: 0.0045 (0.004633785097013282)  --> TIME: 2025-02-24 01:57:01 -- STEP: 6286/36938 -- GLOBAL_STEP: 117100 | > loss_text_ce: 0.039497170597314835 (0.0542063398133801) | > loss_mel_ce: 2.6712238788604736 (nan) | > loss: 0.010756829753518105 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13634977756394673) | > loader_time: 0.0045 (0.00463323087931967)  --> TIME: 2025-02-24 01:57:10 -- STEP: 6336/36938 -- GLOBAL_STEP: 117150 | > loss_text_ce: 0.046592626720666885 (0.054216216291557304) | > loss_mel_ce: 1.0831151008605957 (nan) | > loss: 0.004482967313379049 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13631545639399342) | > loader_time: 0.0047 (0.004632712313623121)  --> TIME: 2025-02-24 01:57:19 -- STEP: 6386/36938 -- GLOBAL_STEP: 117200 | > loss_text_ce: 0.05012365058064461 (0.054205972568598375) | > loss_mel_ce: 1.0349090099334717 (nan) | > loss: 0.00430568540468812 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13627987982447998) | > loader_time: 0.0044 (0.004632088632520877)  --> TIME: 2025-02-24 01:57:28 -- STEP: 6436/36938 -- GLOBAL_STEP: 117250 | > loss_text_ce: 0.05967915058135986 (0.05420521368875155) | > loss_mel_ce: 3.912114143371582 (nan) | > loss: 0.015761084854602814 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13630092218101528) | > loader_time: 0.0046 (0.004631642527725458)  --> TIME: 2025-02-24 01:57:37 -- STEP: 6486/36938 -- GLOBAL_STEP: 117300 | > loss_text_ce: 0.036840055137872696 (0.05419938189399406) | > loss_mel_ce: 1.3161282539367676 (nan) | > loss: 0.005368922371417284 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1362850761619427) | > loader_time: 0.0045 (0.004631017521138605)  --> TIME: 2025-02-24 01:57:47 -- STEP: 6536/36938 -- GLOBAL_STEP: 117350 | > loss_text_ce: 0.04758559912443161 (0.05418515496207869) | > loss_mel_ce: 3.8853747844696045 (nan) | > loss: 0.015606986358761787 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13630504550519476) | > loader_time: 0.0045 (0.0046308685910307614)  --> TIME: 2025-02-24 01:57:56 -- STEP: 6586/36938 -- GLOBAL_STEP: 117400 | > loss_text_ce: 0.05264173448085785 (0.054172862527878986) | > loss_mel_ce: 2.64276123046875 (nan) | > loss: 0.010696043260395527 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.136304026239374) | > loader_time: 0.0045 (0.004630345759549604)  --> TIME: 2025-02-24 01:58:05 -- STEP: 6636/36938 -- GLOBAL_STEP: 117450 | > loss_text_ce: 0.06000711768865585 (0.05417655008767) | > loss_mel_ce: 4.00656270980835 (nan) | > loss: 0.01613718271255493 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13628486587456598) | > loader_time: 0.0046 (0.004630030710485899)  --> TIME: 2025-02-24 01:58:14 -- STEP: 6686/36938 -- GLOBAL_STEP: 117500 | > loss_text_ce: 0.047201190143823624 (0.05416885404110695) | > loss_mel_ce: 1.7295430898666382 (nan) | > loss: 0.007050572894513607 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1362806065083265) | > loader_time: 0.0045 (0.004629742304008579)  --> TIME: 2025-02-24 01:58:23 -- STEP: 6736/36938 -- GLOBAL_STEP: 117550 | > loss_text_ce: 0.061079807579517365 (0.05416559594469875) | > loss_mel_ce: 3.4155333042144775 (nan) | > loss: 0.013796084560453892 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13621991384340879) | > loader_time: 0.0045 (0.0046293104063020365)  --> TIME: 2025-02-24 01:58:32 -- STEP: 6786/36938 -- GLOBAL_STEP: 117600 | > loss_text_ce: 0.058756954967975616 (0.05417672402727465) | > loss_mel_ce: 3.137723445892334 (nan) | > loss: 0.012684446759521961 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13620563330567417) | > loader_time: 0.0047 (0.004628972848396954)  --> TIME: 2025-02-24 01:58:42 -- STEP: 6836/36938 -- GLOBAL_STEP: 117650 | > loss_text_ce: 0.0560922846198082 (0.05415358001937367) | > loss_mel_ce: 3.419132947921753 (nan) | > loss: 0.013790576718747616 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13622013928947424) | > loader_time: 0.0045 (0.004628439755939184)  --> TIME: 2025-02-24 01:58:51 -- STEP: 6886/36938 -- GLOBAL_STEP: 117700 | > loss_text_ce: 0.038512252271175385 (0.05414072050535286) | > loss_mel_ce: 3.122652292251587 (nan) | > loss: 0.012544304132461548 (nan) | > current_lr: 5e-06 | > step_time: 0.1837 (0.13619521967449957) | > loader_time: 0.0047 (0.004627941100014999)  --> TIME: 2025-02-24 01:59:00 -- STEP: 6936/36938 -- GLOBAL_STEP: 117750 | > loss_text_ce: 0.05161944404244423 (0.054147418392091695) | > loss_mel_ce: 1.6598423719406128 (nan) | > loss: 0.006791515275835991 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13616973385128855) | > loader_time: 0.0044 (0.004627562277319783)  --> TIME: 2025-02-24 01:59:09 -- STEP: 6986/36938 -- GLOBAL_STEP: 117800 | > loss_text_ce: 0.04720680043101311 (0.05415346099726585) | > loss_mel_ce: 3.908984422683716 (nan) | > loss: 0.015699172392487526 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.1361626773995857) | > loader_time: 0.0045 (0.004627560429336879)  --> TIME: 2025-02-24 01:59:18 -- STEP: 7036/36938 -- GLOBAL_STEP: 117850 | > loss_text_ce: 0.04238179698586464 (0.05414925478509987) | > loss_mel_ce: 1.5336871147155762 (nan) | > loss: 0.006254241801798344 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13612506323879983) | > loader_time: 0.0045 (0.004627622142984746)  --> TIME: 2025-02-24 01:59:28 -- STEP: 7086/36938 -- GLOBAL_STEP: 117900 | > loss_text_ce: 0.04909711703658104 (0.05413429328455213) | > loss_mel_ce: 3.7026591300964355 (nan) | > loss: 0.014887922443449497 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13614950128238196) | > loader_time: 0.0045 (0.004627317215405094)  --> TIME: 2025-02-24 01:59:37 -- STEP: 7136/36938 -- GLOBAL_STEP: 117950 | > loss_text_ce: 0.0498419888317585 (0.05412588439878669) | > loss_mel_ce: 1.3026597499847412 (nan) | > loss: 0.0053670709021389484 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13614570803840034) | > loader_time: 0.0045 (0.004626956154412739)  --> TIME: 2025-02-24 01:59:46 -- STEP: 7186/36938 -- GLOBAL_STEP: 118000 | > loss_text_ce: 0.05594974756240845 (0.05411291389961719) | > loss_mel_ce: 3.7965407371520996 (nan) | > loss: 0.015287660993635654 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13616525227732199) | > loader_time: 0.0045 (0.00462683541775412)  --> TIME: 2025-02-24 01:59:56 -- STEP: 7236/36938 -- GLOBAL_STEP: 118050 | > loss_text_ce: 0.053377553820610046 (0.05409045943592903) | > loss_mel_ce: 4.195940971374512 (nan) | > loss: 0.016862375661730766 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13620199654364568) | > loader_time: 0.0048 (0.004627010386052073)  --> TIME: 2025-02-24 02:00:05 -- STEP: 7286/36938 -- GLOBAL_STEP: 118100 | > loss_text_ce: 0.0292312353849411 (0.05407816583263598) | > loss_mel_ce: 3.2033417224884033 (nan) | > loss: 0.012827671132981777 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13622561801504074) | > loader_time: 0.0045 (0.004627228404936985)  --> TIME: 2025-02-24 02:00:15 -- STEP: 7336/36938 -- GLOBAL_STEP: 118150 | > loss_text_ce: 0.04998583719134331 (0.05407686987886715) | > loss_mel_ce: 3.398399591445923 (nan) | > loss: 0.013684069737792015 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13623194437120534) | > loader_time: 0.0045 (0.004627399219681405)  --> TIME: 2025-02-24 02:00:24 -- STEP: 7386/36938 -- GLOBAL_STEP: 118200 | > loss_text_ce: 0.06621401757001877 (0.05406882787123827) | > loss_mel_ce: 3.405569553375244 (nan) | > loss: 0.013776919804513454 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13625531515518588) | > loader_time: 0.0045 (0.004627574242262074)  --> TIME: 2025-02-24 02:00:33 -- STEP: 7436/36938 -- GLOBAL_STEP: 118250 | > loss_text_ce: 0.041349858045578 (0.05405348922076398) | > loss_mel_ce: 3.369673728942871 (nan) | > loss: 0.013535808771848679 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.136241273222077) | > loader_time: 0.0045 (0.004627360458845974)  --> TIME: 2025-02-24 02:00:43 -- STEP: 7486/36938 -- GLOBAL_STEP: 118300 | > loss_text_ce: 0.02953101135790348 (0.05404654277263161) | > loss_mel_ce: 2.651350975036621 (nan) | > loss: 0.010638421401381493 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.136250334087935) | > loader_time: 0.0045 (0.0046272312228513275)  --> TIME: 2025-02-24 02:00:52 -- STEP: 7536/36938 -- GLOBAL_STEP: 118350 | > loss_text_ce: 0.04661348834633827 (0.05403531771528904) | > loss_mel_ce: 3.086233377456665 (nan) | > loss: 0.012431932613253593 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13627656003472122) | > loader_time: 0.0044 (0.004627412196936868)  --> TIME: 2025-02-24 02:01:02 -- STEP: 7586/36938 -- GLOBAL_STEP: 118400 | > loss_text_ce: 0.05788067355751991 (0.05403117099843159) | > loss_mel_ce: 4.278379917144775 (nan) | > loss: 0.01720738597214222 (nan) | > current_lr: 5e-06 | > step_time: 0.1811 (0.1362888955391082) | > loader_time: 0.0047 (0.004627061085056058)  --> TIME: 2025-02-24 02:01:11 -- STEP: 7636/36938 -- GLOBAL_STEP: 118450 | > loss_text_ce: 0.051838431507349014 (0.05402709384809354) | > loss_mel_ce: 3.935605049133301 (nan) | > loss: 0.015823189169168472 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.1363066755207872) | > loader_time: 0.0046 (0.004626882207399878)  --> TIME: 2025-02-24 02:01:20 -- STEP: 7686/36938 -- GLOBAL_STEP: 118500 | > loss_text_ce: 0.0609574019908905 (0.05402171991197448) | > loss_mel_ce: 1.0819294452667236 (nan) | > loss: 0.0045352657325565815 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1362656671306658) | > loader_time: 0.0046 (0.004626344027632125)  --> TIME: 2025-02-24 02:01:29 -- STEP: 7736/36938 -- GLOBAL_STEP: 118550 | > loss_text_ce: 0.07033208757638931 (0.05403658476221879) | > loss_mel_ce: 4.3009867668151855 (nan) | > loss: 0.01734650507569313 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13628018491024038) | > loader_time: 0.0045 (0.004626371590526533)  --> TIME: 2025-02-24 02:01:39 -- STEP: 7786/36938 -- GLOBAL_STEP: 118600 | > loss_text_ce: 0.06650948524475098 (0.05402905188812837) | > loss_mel_ce: 3.2388393878936768 (nan) | > loss: 0.013116464950144291 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13627778663140383) | > loader_time: 0.0045 (0.004626379905982878)  --> TIME: 2025-02-24 02:01:49 -- STEP: 7836/36938 -- GLOBAL_STEP: 118650 | > loss_text_ce: 0.046354446560144424 (0.054030793672403765) | > loss_mel_ce: 3.1108081340789795 (nan) | > loss: 0.01252842415124178 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.1363193226323069) | > loader_time: 0.0045 (0.004626539089413173)  --> TIME: 2025-02-24 02:01:58 -- STEP: 7886/36938 -- GLOBAL_STEP: 118700 | > loss_text_ce: 0.052015628665685654 (0.05401956998312531) | > loss_mel_ce: 2.940810441970825 (nan) | > loss: 0.011876294389367104 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13632981326112614) | > loader_time: 0.0046 (0.004626467691712218)  --> TIME: 2025-02-24 02:02:07 -- STEP: 7936/36938 -- GLOBAL_STEP: 118750 | > loss_text_ce: 0.048982083797454834 (0.05400601029137878) | > loss_mel_ce: 4.0185065269470215 (nan) | > loss: 0.016140829771757126 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13633485396783168) | > loader_time: 0.0046 (0.004626360090990208)  --> TIME: 2025-02-24 02:02:16 -- STEP: 7986/36938 -- GLOBAL_STEP: 118800 | > loss_text_ce: 0.04695688560605049 (0.05402062225232268) | > loss_mel_ce: 4.391270160675049 (nan) | > loss: 0.017612013965845108 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.1363222298095494) | > loader_time: 0.0045 (0.004626052737265865)  --> TIME: 2025-02-24 02:02:26 -- STEP: 8036/36938 -- GLOBAL_STEP: 118850 | > loss_text_ce: 0.04657721891999245 (0.05402918323302422) | > loss_mel_ce: 3.1520562171936035 (nan) | > loss: 0.012692990712821484 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13633911935413193) | > loader_time: 0.0046 (0.004627035529296619)  --> TIME: 2025-02-24 02:02:35 -- STEP: 8086/36938 -- GLOBAL_STEP: 118900 | > loss_text_ce: 0.05707911401987076 (0.05403466254492082) | > loss_mel_ce: 4.375473499298096 (nan) | > loss: 0.017589496448636055 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.1363290813987629) | > loader_time: 0.0047 (0.004626952625871901)  --> TIME: 2025-02-24 02:02:44 -- STEP: 8136/36938 -- GLOBAL_STEP: 118950 | > loss_text_ce: 0.055988941341638565 (0.054029002836736006) | > loss_mel_ce: 3.019115686416626 (nan) | > loss: 0.012202797457575798 (nan) | > current_lr: 5e-06 | > step_time: 0.1084 (0.13631677261261552) | > loader_time: 0.0045 (0.004626727502723464)  --> TIME: 2025-02-24 02:02:54 -- STEP: 8186/36938 -- GLOBAL_STEP: 119000 | > loss_text_ce: 0.0658249780535698 (0.054022429340068255) | > loss_mel_ce: 3.1818501949310303 (nan) | > loss: 0.012887600809335709 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13633413195988645) | > loader_time: 0.0045 (0.004626789915512151)  --> TIME: 2025-02-24 02:03:03 -- STEP: 8236/36938 -- GLOBAL_STEP: 119050 | > loss_text_ce: 0.056906942278146744 (0.05402846447060508) | > loss_mel_ce: 3.000458240509033 (nan) | > loss: 0.012132402509450912 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13634626129519975) | > loader_time: 0.0045 (0.0046269388208115885)  --> TIME: 2025-02-24 02:03:13 -- STEP: 8286/36938 -- GLOBAL_STEP: 119100 | > loss_text_ce: 0.06145860627293587 (0.054014365953077896) | > loss_mel_ce: 3.3921027183532715 (nan) | > loss: 0.013704609125852585 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13637107022281061) | > loader_time: 0.0045 (0.004627016555736399)  --> TIME: 2025-02-24 02:03:22 -- STEP: 8336/36938 -- GLOBAL_STEP: 119150 | > loss_text_ce: 0.05939002335071564 (0.05400936333029111) | > loss_mel_ce: 2.285843849182129 (nan) | > loss: 0.009306483902037144 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13635580862323496) | > loader_time: 0.0046 (0.004626712105782142)  --> TIME: 2025-02-24 02:03:31 -- STEP: 8386/36938 -- GLOBAL_STEP: 119200 | > loss_text_ce: 0.048825301229953766 (0.05399963675160561) | > loss_mel_ce: 3.153548240661621 (nan) | > loss: 0.012707832269370556 (nan) | > current_lr: 5e-06 | > step_time: 0.144 (0.13635155468319105) | > loader_time: 0.0046 (0.004626557874190547)  --> TIME: 2025-02-24 02:03:40 -- STEP: 8436/36938 -- GLOBAL_STEP: 119250 | > loss_text_ce: 0.0502522848546505 (0.05399917138364182) | > loss_mel_ce: 2.7614834308624268 (nan) | > loss: 0.011157682165503502 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.136334690537821) | > loader_time: 0.0047 (0.004626696795974074)  --> TIME: 2025-02-24 02:03:49 -- STEP: 8486/36938 -- GLOBAL_STEP: 119300 | > loss_text_ce: 0.0537070706486702 (0.0540029749732123) | > loss_mel_ce: 1.751765251159668 (nan) | > loss: 0.007164573296904564 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13633485386008035) | > loader_time: 0.0046 (0.004626968602035046)  --> TIME: 2025-02-24 02:03:59 -- STEP: 8536/36938 -- GLOBAL_STEP: 119350 | > loss_text_ce: 0.051617007702589035 (0.05400623535347234) | > loss_mel_ce: 4.0675740242004395 (nan) | > loss: 0.016345998272299767 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.136376732347608) | > loader_time: 0.0046 (0.004627552592206904)  --> TIME: 2025-02-24 02:04:08 -- STEP: 8586/36938 -- GLOBAL_STEP: 119400 | > loss_text_ce: 0.04846249893307686 (0.05399642792488487) | > loss_mel_ce: 3.781062602996826 (nan) | > loss: 0.015196528285741806 (nan) | > current_lr: 5e-06 | > step_time: 0.1809 (0.13635028252755999) | > loader_time: 0.005 (0.004627595574267658)  --> TIME: 2025-02-24 02:04:17 -- STEP: 8636/36938 -- GLOBAL_STEP: 119450 | > loss_text_ce: 0.07356266677379608 (0.05400020834627926) | > loss_mel_ce: 2.8810815811157227 (nan) | > loss: 0.01172477938234806 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13635805761333727) | > loader_time: 0.0045 (0.0046276218253961995)  --> TIME: 2025-02-24 02:04:27 -- STEP: 8686/36938 -- GLOBAL_STEP: 119500 | > loss_text_ce: 0.05931689590215683 (0.05399528910576843) | > loss_mel_ce: 3.291917085647583 (nan) | > loss: 0.013298547826707363 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13635171461621368) | > loader_time: 0.0046 (0.004627439137458127)  --> TIME: 2025-02-24 02:04:36 -- STEP: 8736/36938 -- GLOBAL_STEP: 119550 | > loss_text_ce: 0.06724131107330322 (0.053983121278402366) | > loss_mel_ce: 3.8649790287017822 (nan) | > loss: 0.015604049898684025 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13635701697060465) | > loader_time: 0.0044 (0.004628071964005396)  --> TIME: 2025-02-24 02:04:45 -- STEP: 8786/36938 -- GLOBAL_STEP: 119600 | > loss_text_ce: 0.054726164788007736 (0.053986203051372514) | > loss_mel_ce: 3.3298518657684326 (nan) | > loss: 0.013430865481495857 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13633909044736814) | > loader_time: 0.0045 (0.004628183519870075)  --> TIME: 2025-02-24 02:04:54 -- STEP: 8836/36938 -- GLOBAL_STEP: 119650 | > loss_text_ce: 0.058756835758686066 (0.05397845381506599) | > loss_mel_ce: 2.0427043437957764 (nan) | > loss: 0.00833913218230009 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13631301044392424) | > loader_time: 0.0046 (0.004628096920918303)  --> TIME: 2025-02-24 02:05:04 -- STEP: 8886/36938 -- GLOBAL_STEP: 119700 | > loss_text_ce: 0.05169754475355148 (0.05397099352330839) | > loss_mel_ce: 0.0982818603515625 (nan) | > loss: 0.0005951564526185393 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13633559360821804) | > loader_time: 0.0046 (0.004628240351147824)  --> TIME: 2025-02-24 02:05:13 -- STEP: 8936/36938 -- GLOBAL_STEP: 119750 | > loss_text_ce: 0.04052931070327759 (0.053972885980426696) | > loss_mel_ce: 2.894894599914551 (nan) | > loss: 0.011648507788777351 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13635285384119228) | > loader_time: 0.0045 (0.004628189408362915)  --> TIME: 2025-02-24 02:05:22 -- STEP: 8986/36938 -- GLOBAL_STEP: 119800 | > loss_text_ce: 0.05568782985210419 (0.053969518543939665) | > loss_mel_ce: 3.5698137283325195 (nan) | > loss: 0.014386911876499653 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.1363288819750085) | > loader_time: 0.0046 (0.004628018337396628)  --> TIME: 2025-02-24 02:05:32 -- STEP: 9036/36938 -- GLOBAL_STEP: 119850 | > loss_text_ce: 0.052470069378614426 (0.05395460882644242) | > loss_mel_ce: 3.6623992919921875 (nan) | > loss: 0.014741545543074608 (nan) | > current_lr: 5e-06 | > step_time: 0.2021 (0.1363429326354636) | > loader_time: 0.0049 (0.004627880188890018)  --> TIME: 2025-02-24 02:05:41 -- STEP: 9086/36938 -- GLOBAL_STEP: 119900 | > loss_text_ce: 0.053865671157836914 (0.05394351150473151) | > loss_mel_ce: 4.443817138671875 (nan) | > loss: 0.017847947776317596 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.1363404771973583) | > loader_time: 0.0046 (0.004627698558872017)  --> TIME: 2025-02-24 02:05:50 -- STEP: 9136/36938 -- GLOBAL_STEP: 119950 | > loss_text_ce: 0.059440262615680695 (0.05394910972287315) | > loss_mel_ce: 3.910090208053589 (nan) | > loss: 0.01575210690498352 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.1363493643583639) | > loader_time: 0.0047 (0.004627652975180923)  --> TIME: 2025-02-24 02:06:00 -- STEP: 9186/36938 -- GLOBAL_STEP: 120000 | > loss_text_ce: 0.058512430638074875 (0.0539415575531058) | > loss_mel_ce: 2.2142333984375 (nan) | > loss: 0.009018832817673683 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13639575148262045) | > loader_time: 0.0046 (0.004627862865307622) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_120000.pth  --> TIME: 2025-02-24 02:06:16 -- STEP: 9236/36938 -- GLOBAL_STEP: 120050 | > loss_text_ce: 0.058663852512836456 (0.053935049632309974) | > loss_mel_ce: 3.6223974227905273 (nan) | > loss: 0.014607386663556099 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.13641500981559232) | > loader_time: 0.0046 (0.004628036640690937)  --> TIME: 2025-02-24 02:06:25 -- STEP: 9286/36938 -- GLOBAL_STEP: 120100 | > loss_text_ce: 0.060789745301008224 (0.05393535768801357) | > loss_mel_ce: 2.871577739715576 (nan) | > loss: 0.011636380106210709 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13639832682941244) | > loader_time: 0.0045 (0.004628134061364332)  --> TIME: 2025-02-24 02:06:34 -- STEP: 9336/36938 -- GLOBAL_STEP: 120150 | > loss_text_ce: 0.05986623466014862 (0.05392913676414469) | > loss_mel_ce: 3.51836895942688 (nan) | > loss: 0.01419934630393982 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13639428156915614) | > loader_time: 0.0046 (0.004628254724754241)  --> TIME: 2025-02-24 02:06:43 -- STEP: 9386/36938 -- GLOBAL_STEP: 120200 | > loss_text_ce: 0.07653927803039551 (0.05393256946775835) | > loss_mel_ce: 3.5019898414611816 (nan) | > loss: 0.014200513251125813 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1363603432448257) | > loader_time: 0.0045 (0.004628135455383696)  --> TIME: 2025-02-24 02:06:52 -- STEP: 9436/36938 -- GLOBAL_STEP: 120250 | > loss_text_ce: 0.06185431405901909 (0.053924039242320775) | > loss_mel_ce: 2.5489535331726074 (nan) | > loss: 0.010360348969697952 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13637862444530302) | > loader_time: 0.0046 (0.004628211702012855)  --> TIME: 2025-02-24 02:07:02 -- STEP: 9486/36938 -- GLOBAL_STEP: 120300 | > loss_text_ce: 0.04933379963040352 (0.05391518362243466) | > loss_mel_ce: 3.248934507369995 (nan) | > loss: 0.013088366948068142 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1363894640833869) | > loader_time: 0.0049 (0.004628200911027867)  --> TIME: 2025-02-24 02:07:11 -- STEP: 9536/36938 -- GLOBAL_STEP: 120350 | > loss_text_ce: 0.05232813581824303 (0.05390406050133322) | > loss_mel_ce: 3.6438632011413574 (nan) | > loss: 0.014667426235973835 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13639613208454682) | > loader_time: 0.0046 (0.004628261063722941)  --> TIME: 2025-02-24 02:07:20 -- STEP: 9586/36938 -- GLOBAL_STEP: 120400 | > loss_text_ce: 0.04688870906829834 (0.0538879997679298) | > loss_mel_ce: 0.17863769829273224 (nan) | > loss: 0.0008949461043812335 (nan) | > current_lr: 5e-06 | > step_time: 0.1523 (0.13638869290458258) | > loader_time: 0.0047 (0.004628193246629311)  --> TIME: 2025-02-24 02:07:30 -- STEP: 9636/36938 -- GLOBAL_STEP: 120450 | > loss_text_ce: 0.04521309211850166 (0.053877313265668816) | > loss_mel_ce: 2.168872833251953 (nan) | > loss: 0.008786055259406567 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13639444088035166) | > loader_time: 0.0045 (0.004628326869099982)  --> TIME: 2025-02-24 02:07:39 -- STEP: 9686/36938 -- GLOBAL_STEP: 120500 | > loss_text_ce: 0.06507015228271484 (0.05387397999478127) | > loss_mel_ce: 3.2678894996643066 (nan) | > loss: 0.013226031325757504 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.13639068130912152) | > loader_time: 0.0046 (0.004628325330800006)  --> TIME: 2025-02-24 02:07:48 -- STEP: 9736/36938 -- GLOBAL_STEP: 120550 | > loss_text_ce: 0.047918517142534256 (0.05387290905481503) | > loss_mel_ce: 3.9433772563934326 (nan) | > loss: 0.01583847589790821 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13638341395014467) | > loader_time: 0.0046 (0.004628336566730686)  --> TIME: 2025-02-24 02:07:58 -- STEP: 9786/36938 -- GLOBAL_STEP: 120600 | > loss_text_ce: 0.045761026442050934 (0.053869755181485485) | > loss_mel_ce: 1.9485212564468384 (nan) | > loss: 0.007913818582892418 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1364074397764391) | > loader_time: 0.0045 (0.004628459685618153)  --> TIME: 2025-02-24 02:08:08 -- STEP: 9836/36938 -- GLOBAL_STEP: 120650 | > loss_text_ce: 0.057298194617033005 (0.05387066657972689) | > loss_mel_ce: 3.666450262069702 (nan) | > loss: 0.014776780270040035 (nan) | > current_lr: 5e-06 | > step_time: 0.1799 (0.1364508225404908) | > loader_time: 0.0048 (0.004628630128513731)  --> TIME: 2025-02-24 02:08:16 -- STEP: 9886/36938 -- GLOBAL_STEP: 120700 | > loss_text_ce: 0.07105927914381027 (0.05386541969150941) | > loss_mel_ce: 3.952387571334839 (nan) | > loss: 0.015966061502695084 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13640147750251735) | > loader_time: 0.0046 (0.004628345162285329)  --> TIME: 2025-02-24 02:08:26 -- STEP: 9936/36938 -- GLOBAL_STEP: 120750 | > loss_text_ce: 0.05957874655723572 (0.05385377314095423) | > loss_mel_ce: 3.8964407444000244 (nan) | > loss: 0.015698490664362907 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13642211155231032) | > loader_time: 0.0047 (0.0046284973573377285)  --> TIME: 2025-02-24 02:08:35 -- STEP: 9986/36938 -- GLOBAL_STEP: 120800 | > loss_text_ce: 0.05511230230331421 (0.05384693515031691) | > loss_mel_ce: 1.369964599609375 (nan) | > loss: 0.005655067507177591 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13638342465424164) | > loader_time: 0.0045 (0.0046280633226177025)  --> TIME: 2025-02-24 02:08:44 -- STEP: 10036/36938 -- GLOBAL_STEP: 120850 | > loss_text_ce: 0.0817018672823906 (0.05383755904839509) | > loss_mel_ce: 4.57717227935791 (nan) | > loss: 0.01848759688436985 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13638626906728282) | > loader_time: 0.0045 (0.004627929901687087)  --> TIME: 2025-02-24 02:08:53 -- STEP: 10086/36938 -- GLOBAL_STEP: 120900 | > loss_text_ce: 0.056924156844615936 (0.05383134415366174) | > loss_mel_ce: 4.268847942352295 (nan) | > loss: 0.017165763303637505 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.13637239429985204) | > loader_time: 0.0048 (0.004627693652066563)  --> TIME: 2025-02-24 02:09:02 -- STEP: 10136/36938 -- GLOBAL_STEP: 120950 | > loss_text_ce: 0.0593886598944664 (0.053832850993618855) | > loss_mel_ce: 3.728132486343384 (nan) | > loss: 0.01502984669059515 (nan) | > current_lr: 5e-06 | > step_time: 0.1441 (0.13635896594390762) | > loader_time: 0.0046 (0.004627576425683309)  --> TIME: 2025-02-24 02:09:11 -- STEP: 10186/36938 -- GLOBAL_STEP: 121000 | > loss_text_ce: 0.04951385036110878 (0.053825347290984774) | > loss_mel_ce: 4.168883323669434 (nan) | > loss: 0.016739672049880028 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.13634345127935601) | > loader_time: 0.0046 (0.004627496021659462)  --> TIME: 2025-02-24 02:09:21 -- STEP: 10236/36938 -- GLOBAL_STEP: 121050 | > loss_text_ce: 0.061403822153806686 (0.053819239674184664) | > loss_mel_ce: 3.131373405456543 (nan) | > loss: 0.012669751420617104 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.13638593799763235) | > loader_time: 0.0046 (0.0046276792784881015)  --> TIME: 2025-02-24 02:09:30 -- STEP: 10286/36938 -- GLOBAL_STEP: 121100 | > loss_text_ce: 0.05133748799562454 (0.05381010555023512) | > loss_mel_ce: 3.4670469760894775 (nan) | > loss: 0.013961844146251678 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13635769053560587) | > loader_time: 0.0045 (0.0046275994343305885)  --> TIME: 2025-02-24 02:09:40 -- STEP: 10336/36938 -- GLOBAL_STEP: 121150 | > loss_text_ce: 0.05142251402139664 (0.05380461248605218) | > loss_mel_ce: 4.237753868103027 (nan) | > loss: 0.017020542174577713 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.13638083083157188) | > loader_time: 0.0048 (0.004627862189760868)  --> TIME: 2025-02-24 02:09:49 -- STEP: 10386/36938 -- GLOBAL_STEP: 121200 | > loss_text_ce: 0.04950875788927078 (0.05380163650874349) | > loss_mel_ce: 4.082091808319092 (nan) | > loss: 0.016395241022109985 (nan) | > current_lr: 5e-06 | > step_time: 0.1634 (0.13638077164669601) | > loader_time: 0.0049 (0.0046279723304970205)  --> TIME: 2025-02-24 02:09:58 -- STEP: 10436/36938 -- GLOBAL_STEP: 121250 | > loss_text_ce: 0.0502789244055748 (0.053800104581456294) | > loss_mel_ce: 3.598381757736206 (nan) | > loss: 0.014478812925517559 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13638131181688393) | > loader_time: 0.0046 (0.004628069124810293)  --> TIME: 2025-02-24 02:10:08 -- STEP: 10486/36938 -- GLOBAL_STEP: 121300 | > loss_text_ce: 0.06050094589591026 (0.053804060667340975) | > loss_mel_ce: 4.149152755737305 (nan) | > loss: 0.016704976558685303 (nan) | > current_lr: 5e-06 | > step_time: 0.187 (0.13638588721893566) | > loader_time: 0.0048 (0.0046282575122731946)  --> TIME: 2025-02-24 02:10:17 -- STEP: 10536/36938 -- GLOBAL_STEP: 121350 | > loss_text_ce: 0.05315845459699631 (0.05379812881646094) | > loss_mel_ce: 4.1228132247924805 (nan) | > loss: 0.016571316868066788 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13636353016804784) | > loader_time: 0.0046 (0.004628087524763075)  --> TIME: 2025-02-24 02:10:26 -- STEP: 10586/36938 -- GLOBAL_STEP: 121400 | > loss_text_ce: 0.04768004268407822 (0.053799264275146925) | > loss_mel_ce: 3.7684590816497803 (nan) | > loss: 0.015143410302698612 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13634373392609203) | > loader_time: 0.0046 (0.004628022181476499)  --> TIME: 2025-02-24 02:10:35 -- STEP: 10636/36938 -- GLOBAL_STEP: 121450 | > loss_text_ce: 0.06831682473421097 (0.05379104881045465) | > loss_mel_ce: 3.616462230682373 (nan) | > loss: 0.014622140675783157 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.1363460669036911) | > loader_time: 0.0045 (0.004628147384913257)  --> TIME: 2025-02-24 02:10:45 -- STEP: 10686/36938 -- GLOBAL_STEP: 121500 | > loss_text_ce: 0.06536941975355148 (0.05379113109425692) | > loss_mel_ce: 3.1719024181365967 (nan) | > loss: 0.012846317142248154 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13635622225694805) | > loader_time: 0.0045 (0.004628379470329699)  --> TIME: 2025-02-24 02:10:54 -- STEP: 10736/36938 -- GLOBAL_STEP: 121550 | > loss_text_ce: 0.058740463107824326 (0.0537859685090959) | > loss_mel_ce: 4.232461929321289 (nan) | > loss: 0.017028583213686943 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13633561574044753) | > loader_time: 0.0048 (0.004628302065874882)  --> TIME: 2025-02-24 02:11:03 -- STEP: 10786/36938 -- GLOBAL_STEP: 121600 | > loss_text_ce: 0.03345959633588791 (0.053782189848588985) | > loss_mel_ce: 1.5051642656326294 (nan) | > loss: 0.006105650216341019 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13633809632545874) | > loader_time: 0.0045 (0.0046287696568881416)  --> TIME: 2025-02-24 02:11:12 -- STEP: 10836/36938 -- GLOBAL_STEP: 121650 | > loss_text_ce: 0.05874757468700409 (0.05376846949033901) | > loss_mel_ce: 4.256729602813721 (nan) | > loss: 0.01712491177022457 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.1363482567030989) | > loader_time: 0.0047 (0.004629383429516363)  --> TIME: 2025-02-24 02:11:21 -- STEP: 10886/36938 -- GLOBAL_STEP: 121700 | > loss_text_ce: 0.05076904222369194 (0.05376562164508131) | > loss_mel_ce: 0.8510159254074097 (nan) | > loss: 0.0035785119980573654 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13633579551220917) | > loader_time: 0.0045 (0.004629368162672016)  --> TIME: 2025-02-24 02:11:31 -- STEP: 10936/36938 -- GLOBAL_STEP: 121750 | > loss_text_ce: 0.04574500769376755 (0.053752825644831824) | > loss_mel_ce: 3.861708879470825 (nan) | > loss: 0.015505770221352577 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.13632685628439623) | > loader_time: 0.0048 (0.0046293309507530345)  --> TIME: 2025-02-24 02:11:40 -- STEP: 10986/36938 -- GLOBAL_STEP: 121800 | > loss_text_ce: 0.05519545450806618 (0.05374965353770721) | > loss_mel_ce: 3.2281768321990967 (nan) | > loss: 0.013029255904257298 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13634907051272543) | > loader_time: 0.0045 (0.004629378238055037)  --> TIME: 2025-02-24 02:11:49 -- STEP: 11036/36938 -- GLOBAL_STEP: 121850 | > loss_text_ce: 0.0544123500585556 (0.05374254998941692) | > loss_mel_ce: 3.755009889602661 (nan) | > loss: 0.015116755850613117 (nan) | > current_lr: 5e-06 | > step_time: 0.1751 (0.13633619164072464) | > loader_time: 0.0048 (0.004629706809295177)  --> TIME: 2025-02-24 02:11:58 -- STEP: 11086/36938 -- GLOBAL_STEP: 121900 | > loss_text_ce: 0.0561564676463604 (0.05373963530976987) | > loss_mel_ce: 4.034550666809082 (nan) | > loss: 0.016232965514063835 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.1363277405839137) | > loader_time: 0.0046 (0.004630389958539419)  --> TIME: 2025-02-24 02:12:08 -- STEP: 11136/36938 -- GLOBAL_STEP: 121950 | > loss_text_ce: 0.060861051082611084 (0.05373115395496843) | > loss_mel_ce: 4.379240989685059 (nan) | > loss: 0.017619453370571136 (nan) | > current_lr: 5e-06 | > step_time: 0.1455 (0.1363318632517397) | > loader_time: 0.0046 (0.004631070762701411)  --> TIME: 2025-02-24 02:12:17 -- STEP: 11186/36938 -- GLOBAL_STEP: 122000 | > loss_text_ce: 0.04738703742623329 (0.053732350398698815) | > loss_mel_ce: 3.3973817825317383 (nan) | > loss: 0.013669718988239765 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13633365207380904) | > loader_time: 0.0045 (0.004631594790556374)  --> TIME: 2025-02-24 02:12:26 -- STEP: 11236/36938 -- GLOBAL_STEP: 122050 | > loss_text_ce: 0.038143936544656754 (0.053723256113640795) | > loss_mel_ce: 3.186908483505249 (nan) | > loss: 0.012797827832400799 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13630854151944097) | > loader_time: 0.0046 (0.004632115151883013)  --> TIME: 2025-02-24 02:12:35 -- STEP: 11286/36938 -- GLOBAL_STEP: 122100 | > loss_text_ce: 0.06329643726348877 (0.05372570117533817) | > loss_mel_ce: 2.6295461654663086 (nan) | > loss: 0.010685883462429047 (nan) | > current_lr: 5e-06 | > step_time: 0.1667 (0.1363146972639353) | > loader_time: 0.0048 (0.004632224369910972)  --> TIME: 2025-02-24 02:12:45 -- STEP: 11336/36938 -- GLOBAL_STEP: 122150 | > loss_text_ce: 0.05562730133533478 (0.053720208117245295) | > loss_mel_ce: 3.383791208267212 (nan) | > loss: 0.013648486696183681 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.13632414521006717) | > loader_time: 0.0048 (0.004632144198925786)  --> TIME: 2025-02-24 02:12:54 -- STEP: 11386/36938 -- GLOBAL_STEP: 122200 | > loss_text_ce: 0.04364448040723801 (0.05372079042169772) | > loss_mel_ce: 2.5557215213775635 (nan) | > loss: 0.010314945131540298 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13631328762928246) | > loader_time: 0.0045 (0.004632012257357374)  --> TIME: 2025-02-24 02:13:03 -- STEP: 11436/36938 -- GLOBAL_STEP: 122250 | > loss_text_ce: 0.04878021031618118 (0.053717954355953494) | > loss_mel_ce: 3.3878376483917236 (nan) | > loss: 0.013637373223900795 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.13631006288294975) | > loader_time: 0.0048 (0.004632463777094458)  --> TIME: 2025-02-24 02:13:12 -- STEP: 11486/36938 -- GLOBAL_STEP: 122300 | > loss_text_ce: 0.0561898835003376 (0.0537061102515766) | > loss_mel_ce: 3.2187814712524414 (nan) | > loss: 0.012995918281376362 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13630898553445447) | > loader_time: 0.0046 (0.0046328334428075435)  --> TIME: 2025-02-24 02:13:22 -- STEP: 11536/36938 -- GLOBAL_STEP: 122350 | > loss_text_ce: 0.04023038595914841 (0.053699092535190826) | > loss_mel_ce: 3.5722572803497314 (nan) | > loss: 0.01433526910841465 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13630975049180274) | > loader_time: 0.0052 (0.004633111819810901)  --> TIME: 2025-02-24 02:13:32 -- STEP: 11586/36938 -- GLOBAL_STEP: 122400 | > loss_text_ce: 0.04364216700196266 (0.05369510395645609) | > loss_mel_ce: 4.169862270355225 (nan) | > loss: 0.016720255836844444 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.13635299338715) | > loader_time: 0.0046 (0.0046333753031686615)  --> TIME: 2025-02-24 02:13:41 -- STEP: 11636/36938 -- GLOBAL_STEP: 122450 | > loss_text_ce: 0.06530552357435226 (0.053694835651426326) | > loss_mel_ce: 3.8242828845977783 (nan) | > loss: 0.015434875153005123 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13634871330077689) | > loader_time: 0.0047 (0.004633767390095425)  --> TIME: 2025-02-24 02:13:50 -- STEP: 11686/36938 -- GLOBAL_STEP: 122500 | > loss_text_ce: 0.053815849125385284 (0.05368413559227899) | > loss_mel_ce: 3.678053379058838 (nan) | > loss: 0.01480900589376688 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.1363474804918882) | > loader_time: 0.0046 (0.004634206372133236)  --> TIME: 2025-02-24 02:14:00 -- STEP: 11736/36938 -- GLOBAL_STEP: 122550 | > loss_text_ce: 0.06065510958433151 (0.05367785621111935) | > loss_mel_ce: 4.297666072845459 (nan) | > loss: 0.017294926568865776 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1363675270303226) | > loader_time: 0.0054 (0.004635317295525609)  --> TIME: 2025-02-24 02:14:09 -- STEP: 11786/36938 -- GLOBAL_STEP: 122600 | > loss_text_ce: 0.05453215911984444 (0.053677394342934656) | > loss_mel_ce: 3.687676429748535 (nan) | > loss: 0.01485003437846899 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13636238514699414) | > loader_time: 0.0047 (0.0046362858483731785)  --> TIME: 2025-02-24 02:14:19 -- STEP: 11836/36938 -- GLOBAL_STEP: 122650 | > loss_text_ce: 0.05820374935865402 (0.05366759044698913) | > loss_mel_ce: 3.651308059692383 (nan) | > loss: 0.014720285311341286 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13638299884889646) | > loader_time: 0.0047 (0.004636220712039705)  --> TIME: 2025-02-24 02:14:28 -- STEP: 11886/36938 -- GLOBAL_STEP: 122700 | > loss_text_ce: 0.053435396403074265 (0.05366398775099131) | > loss_mel_ce: 2.9611663818359375 (nan) | > loss: 0.011962706223130226 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.1363667294009457) | > loader_time: 0.0045 (0.004636243078497223)  --> TIME: 2025-02-24 02:14:37 -- STEP: 11936/36938 -- GLOBAL_STEP: 122750 | > loss_text_ce: 0.049120306968688965 (0.05366247472597958) | > loss_mel_ce: 0.21004445850849152 (nan) | > loss: 0.0010284315794706345 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13637382840742707) | > loader_time: 0.0046 (0.0046364652247595)  --> TIME: 2025-02-24 02:14:46 -- STEP: 11986/36938 -- GLOBAL_STEP: 122800 | > loss_text_ce: 0.0640982836484909 (0.05365760150713153) | > loss_mel_ce: 4.211565971374512 (nan) | > loss: 0.0169669222086668 (nan) | > current_lr: 5e-06 | > step_time: 0.1695 (0.13637875213699427) | > loader_time: 0.0055 (0.004636811191642219)  --> TIME: 2025-02-24 02:14:56 -- STEP: 12036/36938 -- GLOBAL_STEP: 122850 | > loss_text_ce: 0.04309981316328049 (0.053649667714994395) | > loss_mel_ce: 3.0448718070983887 (nan) | > loss: 0.012253856286406517 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13640543195487415) | > loader_time: 0.0045 (0.004637569159358473)  --> TIME: 2025-02-24 02:15:05 -- STEP: 12086/36938 -- GLOBAL_STEP: 122900 | > loss_text_ce: 0.054298851639032364 (0.0536483596187702) | > loss_mel_ce: 3.7820332050323486 (nan) | > loss: 0.015223541297018528 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.13640600459654173) | > loader_time: 0.0046 (0.004637829617871642)  --> TIME: 2025-02-24 02:15:15 -- STEP: 12136/36938 -- GLOBAL_STEP: 122950 | > loss_text_ce: 0.04320365935564041 (0.053643120618214024) | > loss_mel_ce: 3.621523857116699 (nan) | > loss: 0.01454257033765316 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13640926411926713) | > loader_time: 0.0045 (0.0046380357908904)  --> TIME: 2025-02-24 02:15:24 -- STEP: 12186/36938 -- GLOBAL_STEP: 123000 | > loss_text_ce: 0.05065375566482544 (0.05363774744877217) | > loss_mel_ce: 3.2940773963928223 (nan) | > loss: 0.013272742740809917 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13639546747613027) | > loader_time: 0.0051 (0.004638439717207487)  --> TIME: 2025-02-24 02:15:33 -- STEP: 12236/36938 -- GLOBAL_STEP: 123050 | > loss_text_ce: 0.04278765246272087 (0.053632630701729286) | > loss_mel_ce: 2.1975555419921875 (nan) | > loss: 0.008890250697731972 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.1363925264138107) | > loader_time: 0.0046 (0.0046389221989200525)  --> TIME: 2025-02-24 02:15:42 -- STEP: 12286/36938 -- GLOBAL_STEP: 123100 | > loss_text_ce: 0.04000219702720642 (0.053622143605371216) | > loss_mel_ce: 2.42598819732666 (nan) | > loss: 0.009785676375031471 (nan) | > current_lr: 5e-06 | > step_time: 0.1071 (0.1363767264585202) | > loader_time: 0.0045 (0.004639038448901943)  --> TIME: 2025-02-24 02:15:51 -- STEP: 12336/36938 -- GLOBAL_STEP: 123150 | > loss_text_ce: 0.06184244528412819 (0.053614528879013405) | > loss_mel_ce: 3.1258273124694824 (nan) | > loss: 0.012649483978748322 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13635271953856135) | > loader_time: 0.0045 (0.004638846243711146)  --> TIME: 2025-02-24 02:16:00 -- STEP: 12386/36938 -- GLOBAL_STEP: 123200 | > loss_text_ce: 0.04529613256454468 (0.05360772892268507) | > loss_mel_ce: 0.05316162109375 (nan) | > loss: 0.0003907053906004876 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13634023829228717) | > loader_time: 0.0045 (0.004638642250731211)  --> TIME: 2025-02-24 02:16:09 -- STEP: 12436/36938 -- GLOBAL_STEP: 123250 | > loss_text_ce: 0.04330788180232048 (0.053605825169657076) | > loss_mel_ce: 2.0210089683532715 (nan) | > loss: 0.008191733621060848 (nan) | > current_lr: 5e-06 | > step_time: 0.1482 (0.13633450441093628) | > loader_time: 0.0053 (0.004638977662740166)  --> TIME: 2025-02-24 02:16:18 -- STEP: 12486/36938 -- GLOBAL_STEP: 123300 | > loss_text_ce: 0.06689386814832687 (0.05360002296611395) | > loss_mel_ce: 1.6586207151412964 (nan) | > loss: 0.006847280077636242 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.1363109746614026) | > loader_time: 0.0045 (0.004639654955420193)  --> TIME: 2025-02-24 02:16:28 -- STEP: 12536/36938 -- GLOBAL_STEP: 123350 | > loss_text_ce: 0.04358512908220291 (0.05359273418057853) | > loss_mel_ce: 2.5089666843414307 (nan) | > loss: 0.010129174217581749 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.136311585022982) | > loader_time: 0.0045 (0.004639543008104437)  --> TIME: 2025-02-24 02:16:37 -- STEP: 12586/36938 -- GLOBAL_STEP: 123400 | > loss_text_ce: 0.044121358543634415 (0.05358795811869541) | > loss_mel_ce: 3.3773398399353027 (nan) | > loss: 0.013577227480709553 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.13631983523563593) | > loader_time: 0.0048 (0.004639438675068605)  --> TIME: 2025-02-24 02:16:46 -- STEP: 12636/36938 -- GLOBAL_STEP: 123450 | > loss_text_ce: 0.054780084639787674 (0.053592750748389956) | > loss_mel_ce: 3.919825315475464 (nan) | > loss: 0.015772243961691856 (nan) | > current_lr: 5e-06 | > step_time: 0.1824 (0.13630330417306447) | > loader_time: 0.005 (0.004639675512310824)  --> TIME: 2025-02-24 02:16:55 -- STEP: 12686/36938 -- GLOBAL_STEP: 123500 | > loss_text_ce: 0.05871625617146492 (0.0535957337608617) | > loss_mel_ce: 3.9985878467559814 (nan) | > loss: 0.016100414097309113 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.13628756614695917) | > loader_time: 0.0046 (0.00463963372463423)  --> TIME: 2025-02-24 02:17:05 -- STEP: 12736/36938 -- GLOBAL_STEP: 123550 | > loss_text_ce: 0.055303070694208145 (0.05359480775285491) | > loss_mel_ce: 3.5103557109832764 (nan) | > loss: 0.014149440452456474 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13629039747631164) | > loader_time: 0.0047 (0.004639795939227445)  --> TIME: 2025-02-24 02:17:14 -- STEP: 12786/36938 -- GLOBAL_STEP: 123600 | > loss_text_ce: 0.0479271300137043 (0.053590864933130866) | > loss_mel_ce: 3.6711573600769043 (nan) | > loss: 0.014758273027837276 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13629162941817394) | > loader_time: 0.0045 (0.004639829974694374)  --> TIME: 2025-02-24 02:17:23 -- STEP: 12836/36938 -- GLOBAL_STEP: 123650 | > loss_text_ce: 0.04623769596219063 (0.05358574352055619) | > loss_mel_ce: 2.425173759460449 (nan) | > loss: 0.009807188995182514 (nan) | > current_lr: 5e-06 | > step_time: 0.1521 (0.13627501034075426) | > loader_time: 0.0046 (0.004640258521296664)  --> TIME: 2025-02-24 02:17:32 -- STEP: 12886/36938 -- GLOBAL_STEP: 123700 | > loss_text_ce: 0.05407560616731644 (0.05358555795280527) | > loss_mel_ce: 4.013527870178223 (nan) | > loss: 0.016141284257173538 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.13627208465192545) | > loader_time: 0.0046 (0.004640251365707795)  --> TIME: 2025-02-24 02:17:41 -- STEP: 12936/36938 -- GLOBAL_STEP: 123750 | > loss_text_ce: 0.050181109458208084 (0.05358253491869048) | > loss_mel_ce: 3.6962695121765137 (nan) | > loss: 0.01486686896532774 (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.13625445526989463) | > loader_time: 0.0048 (0.00464046158094647)  --> TIME: 2025-02-24 02:17:50 -- STEP: 12986/36938 -- GLOBAL_STEP: 123800 | > loss_text_ce: 0.049464549869298935 (0.05358211513770477) | > loss_mel_ce: 4.247025966644287 (nan) | > loss: 0.01704956777393818 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.13626328126759632) | > loader_time: 0.0047 (0.0046406587760510435)  --> TIME: 2025-02-24 02:18:00 -- STEP: 13036/36938 -- GLOBAL_STEP: 123850 | > loss_text_ce: 0.07069684565067291 (0.05358084992969402) | > loss_mel_ce: 0.306640625 (nan) | > loss: 0.0014973708894103765 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13626624828981274) | > loader_time: 0.0046 (0.004640597732347292)  --> TIME: 2025-02-24 02:18:09 -- STEP: 13086/36938 -- GLOBAL_STEP: 123900 | > loss_text_ce: 0.045380573719739914 (0.05357315183046243) | > loss_mel_ce: 3.5647315979003906 (nan) | > loss: 0.014325843192636967 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.1362423653925451) | > loader_time: 0.0047 (0.004640373836758115)  --> TIME: 2025-02-24 02:18:18 -- STEP: 13136/36938 -- GLOBAL_STEP: 123950 | > loss_text_ce: 0.051231928169727325 (0.05357676243790086) | > loss_mel_ce: 3.9966847896575928 (nan) | > loss: 0.016063163056969643 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13624312878381067) | > loader_time: 0.0046 (0.004640395345844669)  --> TIME: 2025-02-24 02:18:27 -- STEP: 13186/36938 -- GLOBAL_STEP: 124000 | > loss_text_ce: 0.056089773774147034 (0.05357016243583551) | > loss_mel_ce: 3.639127492904663 (nan) | > loss: 0.014663562178611755 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13622767426583263) | > loader_time: 0.0052 (0.004640640808167944)  --> TIME: 2025-02-24 02:18:36 -- STEP: 13236/36938 -- GLOBAL_STEP: 124050 | > loss_text_ce: 0.054355066269636154 (0.053567361280949036) | > loss_mel_ce: 3.273076295852661 (nan) | > loss: 0.013204094022512436 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13623894093151254) | > loader_time: 0.0045 (0.0046407026299251425)  --> TIME: 2025-02-24 02:18:46 -- STEP: 13286/36938 -- GLOBAL_STEP: 124100 | > loss_text_ce: 0.05087563768029213 (0.053564136811262375) | > loss_mel_ce: 2.6855878829956055 (nan) | > loss: 0.010858982801437378 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13625484707663132) | > loader_time: 0.0044 (0.004640563126879417)  --> TIME: 2025-02-24 02:18:55 -- STEP: 13336/36938 -- GLOBAL_STEP: 124150 | > loss_text_ce: 0.05026528239250183 (0.05356339634011752) | > loss_mel_ce: 3.6031041145324707 (nan) | > loss: 0.014497498981654644 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.1362681308226027) | > loader_time: 0.0046 (0.004640543021051614)  --> TIME: 2025-02-24 02:19:05 -- STEP: 13386/36938 -- GLOBAL_STEP: 124200 | > loss_text_ce: 0.06523727625608444 (0.053570607395047466) | > loss_mel_ce: 4.2821807861328125 (nan) | > loss: 0.01725165918469429 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13625469240536028) | > loader_time: 0.0045 (0.004640258660538023)  --> TIME: 2025-02-24 02:19:14 -- STEP: 13436/36938 -- GLOBAL_STEP: 124250 | > loss_text_ce: 0.0649360865354538 (0.05356704637291241) | > loss_mel_ce: 3.7997970581054688 (nan) | > loss: 0.015336243435740471 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.1362441113996667) | > loader_time: 0.0045 (0.004639974180592336)  --> TIME: 2025-02-24 02:19:23 -- STEP: 13486/36938 -- GLOBAL_STEP: 124300 | > loss_text_ce: 0.048277854919433594 (0.053556326362530104) | > loss_mel_ce: 4.3387932777404785 (nan) | > loss: 0.017409013584256172 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.1362333548063703) | > loader_time: 0.0045 (0.004639846907690182)  --> TIME: 2025-02-24 02:19:32 -- STEP: 13536/36938 -- GLOBAL_STEP: 124350 | > loss_text_ce: 0.04066020995378494 (0.05355421104674805) | > loss_mel_ce: 2.9301629066467285 (nan) | > loss: 0.011788981035351753 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13624121807075343) | > loader_time: 0.0047 (0.0046399810459314935)  --> TIME: 2025-02-24 02:19:42 -- STEP: 13586/36938 -- GLOBAL_STEP: 124400 | > loss_text_ce: 0.05457013100385666 (0.05355217815584514) | > loss_mel_ce: 2.4171385765075684 (nan) | > loss: 0.009808368980884552 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13624914710658304) | > loader_time: 0.0052 (0.004640619147250306)  --> TIME: 2025-02-24 02:19:51 -- STEP: 13636/36938 -- GLOBAL_STEP: 124450 | > loss_text_ce: 0.058536283671855927 (0.0535496830557499) | > loss_mel_ce: 3.989393711090088 (nan) | > loss: 0.016063213348388672 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.13626376942267898) | > loader_time: 0.0048 (0.004641150949147471)  --> TIME: 2025-02-24 02:20:01 -- STEP: 13686/36938 -- GLOBAL_STEP: 124500 | > loss_text_ce: 0.05514548718929291 (0.05354481694865191) | > loss_mel_ce: 2.7419705390930176 (nan) | > loss: 0.01109966728836298 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.1362660276807931) | > loader_time: 0.0045 (0.004641118426603744)  --> TIME: 2025-02-24 02:20:10 -- STEP: 13736/36938 -- GLOBAL_STEP: 124550 | > loss_text_ce: 0.04666315019130707 (0.053544586498195816) | > loss_mel_ce: 1.4527184963226318 (nan) | > loss: 0.005949927493929863 (nan) | > current_lr: 5e-06 | > step_time: 0.1118 (0.13626478037031636) | > loader_time: 0.0046 (0.0046413557503066395)  --> TIME: 2025-02-24 02:20:19 -- STEP: 13786/36938 -- GLOBAL_STEP: 124600 | > loss_text_ce: 0.05700451135635376 (0.05354239999500508) | > loss_mel_ce: 3.660128593444824 (nan) | > loss: 0.014750529080629349 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.1362824734350644) | > loader_time: 0.0046 (0.004641212435431382)  --> TIME: 2025-02-24 02:20:29 -- STEP: 13836/36938 -- GLOBAL_STEP: 124650 | > loss_text_ce: 0.05780743062496185 (0.053537728605389565) | > loss_mel_ce: 3.8680827617645264 (nan) | > loss: 0.015578930266201496 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13628537482903863) | > loader_time: 0.0045 (0.004641314416512057)  --> TIME: 2025-02-24 02:20:38 -- STEP: 13886/36938 -- GLOBAL_STEP: 124700 | > loss_text_ce: 0.047448933124542236 (0.05353156743169005) | > loss_mel_ce: 3.06962513923645 (nan) | > loss: 0.012369342148303986 (nan) | > current_lr: 5e-06 | > step_time: 0.1613 (0.13630427181695057) | > loader_time: 0.0046 (0.004641674084466273)  --> TIME: 2025-02-24 02:20:48 -- STEP: 13936/36938 -- GLOBAL_STEP: 124750 | > loss_text_ce: 0.04274877533316612 (0.053522488747345845) | > loss_mel_ce: 1.7454578876495361 (nan) | > loss: 0.007096058689057827 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13632003607145082) | > loader_time: 0.0044 (0.004642033737781528)  --> TIME: 2025-02-24 02:20:57 -- STEP: 13986/36938 -- GLOBAL_STEP: 124800 | > loss_text_ce: 0.06306520104408264 (0.0535245823768327) | > loss_mel_ce: 3.4004743099212646 (nan) | > loss: 0.013744205236434937 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.1363236018044071) | > loader_time: 0.0046 (0.004641847448186702)  --> TIME: 2025-02-24 02:21:06 -- STEP: 14036/36938 -- GLOBAL_STEP: 124850 | > loss_text_ce: 0.05265264958143234 (0.05352366675194645) | > loss_mel_ce: 4.128125190734863 (nan) | > loss: 0.016590390354394913 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13631503184183555) | > loader_time: 0.0046 (0.0046418504043670315)  --> TIME: 2025-02-24 02:21:16 -- STEP: 14086/36938 -- GLOBAL_STEP: 124900 | > loss_text_ce: 0.05280913785099983 (0.05352147887208972) | > loss_mel_ce: 3.2918457984924316 (nan) | > loss: 0.013272440992295742 (nan) | > current_lr: 5e-06 | > step_time: 0.1705 (0.13631590528446275) | > loader_time: 0.0047 (0.004641626007462459)  --> TIME: 2025-02-24 02:21:25 -- STEP: 14136/36938 -- GLOBAL_STEP: 124950 | > loss_text_ce: 0.031828008592128754 (0.053518321091835096) | > loss_mel_ce: 3.4332387447357178 (nan) | > loss: 0.01375026535242796 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13630897154157776) | > loader_time: 0.0044 (0.004641402236607082)  --> TIME: 2025-02-24 02:21:34 -- STEP: 14186/36938 -- GLOBAL_STEP: 125000 | > loss_text_ce: 0.06541140377521515 (0.05351331742919197) | > loss_mel_ce: 1.5231705904006958 (nan) | > loss: 0.006303897127509117 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.136325642199741) | > loader_time: 0.0052 (0.004641380058631688)  --> TIME: 2025-02-24 02:21:44 -- STEP: 14236/36938 -- GLOBAL_STEP: 125050 | > loss_text_ce: 0.04418618604540825 (0.05350781857274435) | > loss_mel_ce: 3.1559128761291504 (nan) | > loss: 0.012698805890977383 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13631764942757715) | > loader_time: 0.0046 (0.0046411537661851104)  --> TIME: 2025-02-24 02:21:53 -- STEP: 14286/36938 -- GLOBAL_STEP: 125100 | > loss_text_ce: 0.050117310136556625 (0.05350449907782604) | > loss_mel_ce: 2.9565589427948 (nan) | > loss: 0.011931255459785461 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13631633099731344) | > loader_time: 0.0046 (0.004640890005571766)  --> TIME: 2025-02-24 02:22:02 -- STEP: 14336/36938 -- GLOBAL_STEP: 125150 | > loss_text_ce: 0.06338679790496826 (0.05349689015773547) | > loss_mel_ce: 4.0058112144470215 (nan) | > loss: 0.016147611662745476 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13632428449844736) | > loader_time: 0.0053 (0.004640969098545601)  --> TIME: 2025-02-24 02:22:12 -- STEP: 14386/36938 -- GLOBAL_STEP: 125200 | > loss_text_ce: 0.05625919625163078 (0.05349880790714168) | > loss_mel_ce: 3.323526620864868 (nan) | > loss: 0.013411848805844784 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.1363372319466915) | > loader_time: 0.0045 (0.004641062938568729)  --> TIME: 2025-02-24 02:22:21 -- STEP: 14436/36938 -- GLOBAL_STEP: 125250 | > loss_text_ce: 0.03646702691912651 (0.05349184360729825) | > loss_mel_ce: 2.5445351600646973 (nan) | > loss: 0.010242072865366936 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1363395689497637) | > loader_time: 0.0047 (0.00464095386252504)  --> TIME: 2025-02-24 02:22:30 -- STEP: 14486/36938 -- GLOBAL_STEP: 125300 | > loss_text_ce: 0.05438721179962158 (0.0534851688429523) | > loss_mel_ce: 4.379179000854492 (nan) | > loss: 0.017593517899513245 (nan) | > current_lr: 5e-06 | > step_time: 0.1615 (0.13633477786587073) | > loader_time: 0.0048 (0.004641080501740196)  --> TIME: 2025-02-24 02:22:39 -- STEP: 14536/36938 -- GLOBAL_STEP: 125350 | > loss_text_ce: 0.07243864238262177 (0.05348219486539297) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13633128951457998) | > loader_time: 0.0066 (0.004641172760583955)  --> TIME: 2025-02-24 02:22:49 -- STEP: 14586/36938 -- GLOBAL_STEP: 125400 | > loss_text_ce: 0.03563893586397171 (0.0534833579818515) | > loss_mel_ce: 0.06303711235523224 (nan) | > loss: 0.0003915716370102018 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.1363238692855736) | > loader_time: 0.0046 (0.004641031117231997)  --> TIME: 2025-02-24 02:22:58 -- STEP: 14636/36938 -- GLOBAL_STEP: 125450 | > loss_text_ce: 0.04997400566935539 (0.05348832746478967) | > loss_mel_ce: 3.8557586669921875 (nan) | > loss: 0.015498939901590347 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.1363196746838426) | > loader_time: 0.0053 (0.0046411991999037805)  --> TIME: 2025-02-24 02:23:07 -- STEP: 14686/36938 -- GLOBAL_STEP: 125500 | > loss_text_ce: 0.05735553428530693 (0.05348815693911021) | > loss_mel_ce: 3.8112447261810303 (nan) | > loss: 0.015351589769124985 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.1363098446199104) | > loader_time: 0.0048 (0.004641237772567928)  --> TIME: 2025-02-24 02:23:16 -- STEP: 14736/36938 -- GLOBAL_STEP: 125550 | > loss_text_ce: 0.051062945276498795 (0.053482590428397696) | > loss_mel_ce: 3.8529722690582275 (nan) | > loss: 0.015492204576730728 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13630234154829166) | > loader_time: 0.0046 (0.004641342046594765)  --> TIME: 2025-02-24 02:23:26 -- STEP: 14786/36938 -- GLOBAL_STEP: 125600 | > loss_text_ce: 0.046194009482860565 (0.05347969469255418) | > loss_mel_ce: 1.580103874206543 (nan) | > loss: 0.006453562993556261 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13630197216850795) | > loader_time: 0.0045 (0.004641125574012602)  --> TIME: 2025-02-24 02:23:35 -- STEP: 14836/36938 -- GLOBAL_STEP: 125650 | > loss_text_ce: 0.046174757182598114 (0.05346641325748127) | > loss_mel_ce: 3.578705310821533 (nan) | > loss: 0.014384445734322071 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13632061903159848) | > loader_time: 0.0046 (0.004641085389040322)  --> TIME: 2025-02-24 02:23:44 -- STEP: 14886/36938 -- GLOBAL_STEP: 125700 | > loss_text_ce: 0.05590830743312836 (0.05346091833512659) | > loss_mel_ce: 3.6517131328582764 (nan) | > loss: 0.014712783508002758 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13631266551714183) | > loader_time: 0.0047 (0.004641110724408531)  --> TIME: 2025-02-24 02:23:54 -- STEP: 14936/36938 -- GLOBAL_STEP: 125750 | > loss_text_ce: 0.0470886155962944 (0.05346108665320679) | > loss_mel_ce: 3.4610488414764404 (nan) | > loss: 0.013921181671321392 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13633302309070008) | > loader_time: 0.0045 (0.004641094004079874)  --> TIME: 2025-02-24 02:24:04 -- STEP: 14986/36938 -- GLOBAL_STEP: 125800 | > loss_text_ce: 0.05332968011498451 (0.053462335564187065) | > loss_mel_ce: 1.5636065006256104 (nan) | > loss: 0.0064164139330387115 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13634733687283387) | > loader_time: 0.0046 (0.0046409366446789856)  --> TIME: 2025-02-24 02:24:13 -- STEP: 15036/36938 -- GLOBAL_STEP: 125850 | > loss_text_ce: 0.041638415306806564 (0.05346119097411924) | > loss_mel_ce: 3.5264995098114014 (nan) | > loss: 0.014159278012812138 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.1363382273831567) | > loader_time: 0.0044 (0.004640883082054679)  --> TIME: 2025-02-24 02:24:22 -- STEP: 15086/36938 -- GLOBAL_STEP: 125900 | > loss_text_ce: 0.061063606292009354 (0.05345821141728453) | > loss_mel_ce: 3.939507007598877 (nan) | > loss: 0.015875281766057014 (nan) | > current_lr: 5e-06 | > step_time: 0.1877 (0.1363365463848289) | > loader_time: 0.0048 (0.0046409133984210245)  --> TIME: 2025-02-24 02:24:31 -- STEP: 15136/36938 -- GLOBAL_STEP: 125950 | > loss_text_ce: 0.04165755212306976 (0.053454913060627704) | > loss_mel_ce: 3.6363353729248047 (nan) | > loss: 0.014595210552215576 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13631905089221147) | > loader_time: 0.0046 (0.004641024478520188)  --> TIME: 2025-02-24 02:24:40 -- STEP: 15186/36938 -- GLOBAL_STEP: 126000 | > loss_text_ce: 0.04703524336218834 (0.05345236686232872) | > loss_mel_ce: 2.318258047103882 (nan) | > loss: 0.009386084973812103 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13632276674260888) | > loader_time: 0.0051 (0.004641257035124674)  --> TIME: 2025-02-24 02:24:50 -- STEP: 15236/36938 -- GLOBAL_STEP: 126050 | > loss_text_ce: 0.04896823316812515 (0.053450822637162776) | > loss_mel_ce: 3.6833748817443848 (nan) | > loss: 0.014810886234045029 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.1363272142397765) | > loader_time: 0.0045 (0.0046411990086592195)  --> TIME: 2025-02-24 02:24:59 -- STEP: 15286/36938 -- GLOBAL_STEP: 126100 | > loss_text_ce: 0.055785614997148514 (0.05344673231242615) | > loss_mel_ce: 3.6192891597747803 (nan) | > loss: 0.014583631418645382 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13633024506928945) | > loader_time: 0.0046 (0.004641361796823163)  --> TIME: 2025-02-24 02:25:08 -- STEP: 15336/36938 -- GLOBAL_STEP: 126150 | > loss_text_ce: 0.05308406054973602 (0.05344169017961788) | > loss_mel_ce: 4.4005513191223145 (nan) | > loss: 0.017673157155513763 (nan) | > current_lr: 5e-06 | > step_time: 0.1767 (0.13633067053186493) | > loader_time: 0.0051 (0.004641699570196637)  --> TIME: 2025-02-24 02:25:18 -- STEP: 15386/36938 -- GLOBAL_STEP: 126200 | > loss_text_ce: 0.04975399374961853 (0.05343929061117231) | > loss_mel_ce: 0.646728515625 (nan) | > loss: 0.0027638196479529142 (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.1363320063819479) | > loader_time: 0.0045 (0.0046418450456066525)  --> TIME: 2025-02-24 02:25:26 -- STEP: 15436/36938 -- GLOBAL_STEP: 126250 | > loss_text_ce: 0.053805865347385406 (0.05343094193703604) | > loss_mel_ce: 2.9682722091674805 (nan) | > loss: 0.011992373503744602 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13631283396666324) | > loader_time: 0.0045 (0.004641632970116481)  --> TIME: 2025-02-24 02:25:36 -- STEP: 15486/36938 -- GLOBAL_STEP: 126300 | > loss_text_ce: 0.05127612501382828 (0.05342094160856466) | > loss_mel_ce: 2.883056640625 (nan) | > loss: 0.011644179001450539 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13630343855281854) | > loader_time: 0.0045 (0.004641939361135756)  --> TIME: 2025-02-24 02:25:45 -- STEP: 15536/36938 -- GLOBAL_STEP: 126350 | > loss_text_ce: 0.05094350874423981 (0.05341751159523172) | > loss_mel_ce: 3.4303066730499268 (nan) | > loss: 0.013814485631883144 (nan) | > current_lr: 5e-06 | > step_time: 0.1823 (0.13630673605430804) | > loader_time: 0.0048 (0.004641960044108499)  --> TIME: 2025-02-24 02:25:54 -- STEP: 15586/36938 -- GLOBAL_STEP: 126400 | > loss_text_ce: 0.04473597928881645 (0.05341043007599144) | > loss_mel_ce: 2.4375 (nan) | > loss: 0.00985014345496893 (nan) | > current_lr: 5e-06 | > step_time: 0.1074 (0.1363123039463195) | > loader_time: 0.0045 (0.004641886976920752)  --> TIME: 2025-02-24 02:26:04 -- STEP: 15636/36938 -- GLOBAL_STEP: 126450 | > loss_text_ce: 0.058331750333309174 (0.05341016188376815) | > loss_mel_ce: 3.910216808319092 (nan) | > loss: 0.01574821025133133 (nan) | > current_lr: 5e-06 | > step_time: 0.1526 (0.1363133907196136) | > loader_time: 0.0046 (0.004641919283648521)  --> TIME: 2025-02-24 02:26:13 -- STEP: 15686/36938 -- GLOBAL_STEP: 126500 | > loss_text_ce: 0.0476599745452404 (0.05341206966605668) | > loss_mel_ce: 0.0785624161362648 (nan) | > loss: 0.0005008825100958347 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13631273972719152) | > loader_time: 0.0045 (0.004642052855948221)  --> TIME: 2025-02-24 02:26:22 -- STEP: 15736/36938 -- GLOBAL_STEP: 126550 | > loss_text_ce: 0.0432603545486927 (0.05340930665230031) | > loss_mel_ce: 3.1806490421295166 (nan) | > loss: 0.012793292291462421 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.13631705502658967) | > loader_time: 0.0047 (0.004642080642514009)  --> TIME: 2025-02-24 02:26:32 -- STEP: 15786/36938 -- GLOBAL_STEP: 126600 | > loss_text_ce: 0.05778774246573448 (0.053405760223064494) | > loss_mel_ce: 3.890608549118042 (nan) | > loss: 0.015668239444494247 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13632075145914324) | > loader_time: 0.0047 (0.0046421564472623215)  --> TIME: 2025-02-24 02:26:41 -- STEP: 15836/36938 -- GLOBAL_STEP: 126650 | > loss_text_ce: 0.05730242654681206 (0.05340109098924728) | > loss_mel_ce: 3.4471147060394287 (nan) | > loss: 0.013906418345868587 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.1363213065297478) | > loader_time: 0.0048 (0.004641879535317563)  --> TIME: 2025-02-24 02:26:50 -- STEP: 15886/36938 -- GLOBAL_STEP: 126700 | > loss_text_ce: 0.04475386440753937 (0.05339467336216035) | > loss_mel_ce: 4.275055408477783 (nan) | > loss: 0.017142102122306824 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13630720908104876) | > loader_time: 0.0044 (0.004641706841757078)  --> TIME: 2025-02-24 02:26:59 -- STEP: 15936/36938 -- GLOBAL_STEP: 126750 | > loss_text_ce: 0.040180034935474396 (0.053385021786362466) | > loss_mel_ce: 3.8771920204162598 (nan) | > loss: 0.015545127913355827 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13630323225714647) | > loader_time: 0.0045 (0.004641746974972827)  --> TIME: 2025-02-24 02:27:09 -- STEP: 15986/36938 -- GLOBAL_STEP: 126800 | > loss_text_ce: 0.05777312442660332 (0.05338355675936258) | > loss_mel_ce: 3.9453892707824707 (nan) | > loss: 0.01588556542992592 (nan) | > current_lr: 5e-06 | > step_time: 0.179 (0.1363040536780036) | > loader_time: 0.0048 (0.004642267959758893)  --> TIME: 2025-02-24 02:27:18 -- STEP: 16036/36938 -- GLOBAL_STEP: 126850 | > loss_text_ce: 0.04169551283121109 (0.05338292325370378) | > loss_mel_ce: 3.176694869995117 (nan) | > loss: 0.012771391309797764 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.13631374237096755) | > loader_time: 0.0046 (0.004642298495748677)  --> TIME: 2025-02-24 02:27:28 -- STEP: 16086/36938 -- GLOBAL_STEP: 126900 | > loss_text_ce: 0.04276005178689957 (0.05337973033686729) | > loss_mel_ce: 3.811282157897949 (nan) | > loss: 0.015293819829821587 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.1363282972613516) | > loader_time: 0.0045 (0.004642518645982213)  --> TIME: 2025-02-24 02:27:37 -- STEP: 16136/36938 -- GLOBAL_STEP: 126950 | > loss_text_ce: 0.059011638164520264 (0.05337466335376133) | > loss_mel_ce: 2.397387742996216 (nan) | > loss: 0.009747616946697235 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13633031842433221) | > loader_time: 0.0046 (0.004642794583774042)  --> TIME: 2025-02-24 02:27:47 -- STEP: 16186/36938 -- GLOBAL_STEP: 127000 | > loss_text_ce: 0.0444754920899868 (0.053373517619208076) | > loss_mel_ce: 3.648336172103882 (nan) | > loss: 0.014654015190899372 (nan) | > current_lr: 5e-06 | > step_time: 0.1695 (0.13633896325584094) | > loader_time: 0.0048 (0.00464281764207804)  --> TIME: 2025-02-24 02:27:56 -- STEP: 16236/36938 -- GLOBAL_STEP: 127050 | > loss_text_ce: 0.04793131351470947 (0.053368007478330044) | > loss_mel_ce: 3.11928129196167 (nan) | > loss: 0.012568304315209389 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13633583899817808) | > loader_time: 0.0047 (0.004642788105100259)  --> TIME: 2025-02-24 02:28:05 -- STEP: 16286/36938 -- GLOBAL_STEP: 127100 | > loss_text_ce: 0.04534929245710373 (0.053367129466072576) | > loss_mel_ce: 3.8466012477874756 (nan) | > loss: 0.015444248914718628 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1363377742241489) | > loader_time: 0.0045 (0.004642678583691596)  --> TIME: 2025-02-24 02:28:14 -- STEP: 16336/36938 -- GLOBAL_STEP: 127150 | > loss_text_ce: 0.05225260183215141 (0.05336601295792119) | > loss_mel_ce: 3.9183340072631836 (nan) | > loss: 0.01575629599392414 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.1363231634737584) | > loader_time: 0.0045 (0.004642418021086256)  --> TIME: 2025-02-24 02:28:23 -- STEP: 16386/36938 -- GLOBAL_STEP: 127200 | > loss_text_ce: 0.04884512722492218 (0.0533662389613297) | > loss_mel_ce: 0.10337135940790176 (nan) | > loss: 0.0006040337611921132 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13631530759914584) | > loader_time: 0.0044 (0.004642357585282598)  --> TIME: 2025-02-24 02:28:33 -- STEP: 16436/36938 -- GLOBAL_STEP: 127250 | > loss_text_ce: 0.05150643363595009 (0.053357631136951245) | > loss_mel_ce: 1.0762939453125 (nan) | > loss: 0.004475398454815149 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1363298280391712) | > loader_time: 0.0046 (0.0046422394936785)  --> TIME: 2025-02-24 02:28:42 -- STEP: 16486/36938 -- GLOBAL_STEP: 127300 | > loss_text_ce: 0.04760932922363281 (0.05334822337585896) | > loss_mel_ce: 3.6557915210723877 (nan) | > loss: 0.014696035534143448 (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.13632736627763203) | > loader_time: 0.0047 (0.004642394580237128)  --> TIME: 2025-02-24 02:28:51 -- STEP: 16536/36938 -- GLOBAL_STEP: 127350 | > loss_text_ce: 0.04785573482513428 (0.053343770863334125) | > loss_mel_ce: 4.019403457641602 (nan) | > loss: 0.016139918938279152 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.13632100559572857) | > loader_time: 0.0048 (0.004642387433045134)  --> TIME: 2025-02-24 02:29:01 -- STEP: 16586/36938 -- GLOBAL_STEP: 127400 | > loss_text_ce: 0.05273277312517166 (0.053341708548863755) | > loss_mel_ce: 3.6694114208221436 (nan) | > loss: 0.014770413748919964 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13632960187559445) | > loader_time: 0.0046 (0.004642457477896623)  --> TIME: 2025-02-24 02:29:10 -- STEP: 16636/36938 -- GLOBAL_STEP: 127450 | > loss_text_ce: 0.05486242473125458 (0.053337722066382216) | > loss_mel_ce: 4.1201982498168945 (nan) | > loss: 0.01656770147383213 (nan) | > current_lr: 5e-06 | > step_time: 0.1872 (0.1363468674204336) | > loader_time: 0.0055 (0.0046426744150352885)  --> TIME: 2025-02-24 02:29:19 -- STEP: 16686/36938 -- GLOBAL_STEP: 127500 | > loss_text_ce: 0.04510340467095375 (0.05333314633842388) | > loss_mel_ce: 3.6624622344970703 (nan) | > loss: 0.014712562784552574 (nan) | > current_lr: 5e-06 | > step_time: 0.1517 (0.13633561781609282) | > loader_time: 0.0048 (0.004643373361926096)  --> TIME: 2025-02-24 02:29:29 -- STEP: 16736/36938 -- GLOBAL_STEP: 127550 | > loss_text_ce: 0.05090648680925369 (0.05333336580537579) | > loss_mel_ce: 3.6846203804016113 (nan) | > loss: 0.014823519624769688 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13633321721398842) | > loader_time: 0.0045 (0.004643638235433143)  --> TIME: 2025-02-24 02:29:38 -- STEP: 16786/36938 -- GLOBAL_STEP: 127600 | > loss_text_ce: 0.050671037286520004 (0.05333237521726321) | > loss_mel_ce: 2.459301233291626 (nan) | > loss: 0.009960208088159561 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13634590845234196) | > loader_time: 0.0046 (0.004643721971383432)  --> TIME: 2025-02-24 02:29:47 -- STEP: 16836/36938 -- GLOBAL_STEP: 127650 | > loss_text_ce: 0.04823492839932442 (0.0533247452903468) | > loss_mel_ce: 3.574963331222534 (nan) | > loss: 0.014377771876752377 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.136339502610456) | > loader_time: 0.0046 (0.004643691877594853)  --> TIME: 2025-02-24 02:29:56 -- STEP: 16886/36938 -- GLOBAL_STEP: 127700 | > loss_text_ce: 0.04094108194112778 (0.05332214929029752) | > loss_mel_ce: 5.9375 (nan) | > loss: 0.02372397482395172 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13633733106057694) | > loader_time: 0.0044 (0.004643903049151357)  --> TIME: 2025-02-24 02:30:06 -- STEP: 16936/36938 -- GLOBAL_STEP: 127750 | > loss_text_ce: 0.04561666399240494 (0.05332048667208085) | > loss_mel_ce: 3.310385227203369 (nan) | > loss: 0.013317468576133251 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13633451908897265) | > loader_time: 0.0045 (0.004644091097205992)  --> TIME: 2025-02-24 02:30:14 -- STEP: 16986/36938 -- GLOBAL_STEP: 127800 | > loss_text_ce: 0.04921668767929077 (0.05332108035492378) | > loss_mel_ce: 4.108968257904053 (nan) | > loss: 0.01650073565542698 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1363052874224555) | > loader_time: 0.0045 (0.00464410987069324)  --> TIME: 2025-02-24 02:30:24 -- STEP: 17036/36938 -- GLOBAL_STEP: 127850 | > loss_text_ce: 0.06134263053536415 (0.05331539885217698) | > loss_mel_ce: 3.9609320163726807 (nan) | > loss: 0.015961406752467155 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.1363018513089705) | > loader_time: 0.0046 (0.00464400747735389)  --> TIME: 2025-02-24 02:30:33 -- STEP: 17086/36938 -- GLOBAL_STEP: 127900 | > loss_text_ce: 0.049370117485523224 (0.05331386487741751) | > loss_mel_ce: 4.3816962242126465 (nan) | > loss: 0.01758359745144844 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13630669193442027) | > loader_time: 0.0046 (0.004644433578255434)  --> TIME: 2025-02-24 02:30:42 -- STEP: 17136/36938 -- GLOBAL_STEP: 127950 | > loss_text_ce: 0.05276009812951088 (0.0533130548632008) | > loss_mel_ce: 3.836759090423584 (nan) | > loss: 0.015434601344168186 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13631062185953594) | > loader_time: 0.0047 (0.0046445023195416215)  --> TIME: 2025-02-24 02:30:52 -- STEP: 17186/36938 -- GLOBAL_STEP: 128000 | > loss_text_ce: 0.05340336263179779 (0.053311658543035385) | > loss_mel_ce: 3.4220550060272217 (nan) | > loss: 0.013791502453386784 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13632198388898692) | > loader_time: 0.0047 (0.0046449628736952445)  --> TIME: 2025-02-24 02:31:01 -- STEP: 17236/36938 -- GLOBAL_STEP: 128050 | > loss_text_ce: 0.05822015553712845 (0.05331018059169723) | > loss_mel_ce: 2.6147286891937256 (nan) | > loss: 0.01060694083571434 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13631699048717608) | > loader_time: 0.0047 (0.004644852402205245)  --> TIME: 2025-02-24 02:31:10 -- STEP: 17286/36938 -- GLOBAL_STEP: 128100 | > loss_text_ce: 0.05621631443500519 (0.053305279596283124) | > loss_mel_ce: 2.4874794483184814 (nan) | > loss: 0.010094030760228634 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13631071385638882) | > loader_time: 0.0046 (0.004645103604432449)  --> TIME: 2025-02-24 02:31:19 -- STEP: 17336/36938 -- GLOBAL_STEP: 128150 | > loss_text_ce: 0.058001648634672165 (0.053298585137707286) | > loss_mel_ce: 3.642444133758545 (nan) | > loss: 0.014684310182929039 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13630066320375803) | > loader_time: 0.0045 (0.004644903132153032)  --> TIME: 2025-02-24 02:31:29 -- STEP: 17386/36938 -- GLOBAL_STEP: 128200 | > loss_text_ce: 0.050278596580028534 (0.053294174973961626) | > loss_mel_ce: 3.2636289596557617 (nan) | > loss: 0.013150427490472794 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.13629512484899514) | > loader_time: 0.0047 (0.004644751370494821)  --> TIME: 2025-02-24 02:31:38 -- STEP: 17436/36938 -- GLOBAL_STEP: 128250 | > loss_text_ce: 0.04969451203942299 (0.05329437705086309) | > loss_mel_ce: 3.761970281600952 (nan) | > loss: 0.015125654637813568 (nan) | > current_lr: 5e-06 | > step_time: 0.1914 (0.1362998484197799) | > loader_time: 0.0049 (0.0046448258528826395)  --> TIME: 2025-02-24 02:31:47 -- STEP: 17486/36938 -- GLOBAL_STEP: 128300 | > loss_text_ce: 0.05546860024333 (0.05328450542990207) | > loss_mel_ce: 4.063068866729736 (nan) | > loss: 0.016343403607606888 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13630335464311796) | > loader_time: 0.0045 (0.004644826826645114)  --> TIME: 2025-02-24 02:31:56 -- STEP: 17536/36938 -- GLOBAL_STEP: 128350 | > loss_text_ce: 0.04318169876933098 (0.05327079844292263) | > loss_mel_ce: 4.057711124420166 (nan) | > loss: 0.016273386776447296 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13629372881548735) | > loader_time: 0.0045 (0.004644806598768614)  --> TIME: 2025-02-24 02:32:05 -- STEP: 17586/36938 -- GLOBAL_STEP: 128400 | > loss_text_ce: 0.059098850935697556 (0.053272348629279126) | > loss_mel_ce: 1.7999894618988037 (nan) | > loss: 0.007377334870398045 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13628170504635675) | > loader_time: 0.0047 (0.004644677607283496)  --> TIME: 2025-02-24 02:32:15 -- STEP: 17636/36938 -- GLOBAL_STEP: 128450 | > loss_text_ce: 0.05154944211244583 (0.05326936479409054) | > loss_mel_ce: 3.1549479961395264 (nan) | > loss: 0.012724196538329124 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1362874355816906) | > loader_time: 0.0046 (0.004644968959040908)  --> TIME: 2025-02-24 02:32:24 -- STEP: 17686/36938 -- GLOBAL_STEP: 128500 | > loss_text_ce: 0.024365181103348732 (0.053261094481337604) | > loss_mel_ce: 3.2080912590026855 (nan) | > loss: 0.012827209196984768 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.1362970308603432) | > loader_time: 0.0045 (0.004645271631814794)  --> TIME: 2025-02-24 02:32:34 -- STEP: 17736/36938 -- GLOBAL_STEP: 128550 | > loss_text_ce: 0.04602444916963577 (0.05325902935813134) | > loss_mel_ce: 4.212308883666992 (nan) | > loss: 0.016898147761821747 (nan) | > current_lr: 5e-06 | > step_time: 0.1889 (0.1363140243531139) | > loader_time: 0.0049 (0.004645738399507333)  --> TIME: 2025-02-24 02:32:43 -- STEP: 17786/36938 -- GLOBAL_STEP: 128600 | > loss_text_ce: 0.061106547713279724 (0.05325894658371547) | > loss_mel_ce: 3.8753159046173096 (nan) | > loss: 0.01562072429805994 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13630376816010428) | > loader_time: 0.0046 (0.0046457003437197905)  --> TIME: 2025-02-24 02:32:52 -- STEP: 17836/36938 -- GLOBAL_STEP: 128650 | > loss_text_ce: 0.05510504171252251 (0.053254274481602054) | > loss_mel_ce: 4.149708271026611 (nan) | > loss: 0.016685768961906433 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.13631173685597467) | > loader_time: 0.0046 (0.004645654133387968)  --> TIME: 2025-02-24 02:33:02 -- STEP: 17886/36938 -- GLOBAL_STEP: 128700 | > loss_text_ce: 0.053164880722761154 (0.05324832875484431) | > loss_mel_ce: 3.720961093902588 (nan) | > loss: 0.014976691454648972 (nan) | > current_lr: 5e-06 | > step_time: 0.1765 (0.13632316565628294) | > loader_time: 0.0048 (0.0046458991997782835)  --> TIME: 2025-02-24 02:33:11 -- STEP: 17936/36938 -- GLOBAL_STEP: 128750 | > loss_text_ce: 0.04967939853668213 (0.05324161673931734) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1116 (0.13631563528003107) | > loader_time: 0.0052 (0.004646165337975161)  --> TIME: 2025-02-24 02:33:20 -- STEP: 17986/36938 -- GLOBAL_STEP: 128800 | > loss_text_ce: 0.05677376687526703 (0.05324089377464006) | > loss_mel_ce: 2.9840803146362305 (nan) | > loss: 0.01206688117235899 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.1363108005537471) | > loader_time: 0.0047 (0.004646105905747254)  --> TIME: 2025-02-24 02:33:30 -- STEP: 18036/36938 -- GLOBAL_STEP: 128850 | > loss_text_ce: 0.05256812274456024 (0.0532415431031259) | > loss_mel_ce: 3.0632030963897705 (nan) | > loss: 0.012364172376692295 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.136313126710599) | > loader_time: 0.0047 (0.004646022942674787)  --> TIME: 2025-02-24 02:33:39 -- STEP: 18086/36938 -- GLOBAL_STEP: 128900 | > loss_text_ce: 0.05201994627714157 (0.053240624511121895) | > loss_mel_ce: 3.8014862537384033 (nan) | > loss: 0.01529169175773859 (nan) | > current_lr: 5e-06 | > step_time: 0.1889 (0.13631360500269926) | > loader_time: 0.0049 (0.004645977652496057)  --> TIME: 2025-02-24 02:33:48 -- STEP: 18136/36938 -- GLOBAL_STEP: 128950 | > loss_text_ce: 0.04976418986916542 (0.053237184863266375) | > loss_mel_ce: 2.754803419113159 (nan) | > loss: 0.011129236780107021 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.1363077303571331) | > loader_time: 0.0044 (0.004645838879992053)  --> TIME: 2025-02-24 02:33:57 -- STEP: 18186/36938 -- GLOBAL_STEP: 129000 | > loss_text_ce: 0.06032322719693184 (0.053237432543502536) | > loss_mel_ce: 4.225885391235352 (nan) | > loss: 0.017008764669299126 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13630088179546504) | > loader_time: 0.0044 (0.004645828745565195)  --> TIME: 2025-02-24 02:34:07 -- STEP: 18236/36938 -- GLOBAL_STEP: 129050 | > loss_text_ce: 0.06116754561662674 (0.053233625544192184) | > loss_mel_ce: 4.1747145652771 (nan) | > loss: 0.01680905744433403 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.13630974939921192) | > loader_time: 0.0046 (0.004645935771077781)  --> TIME: 2025-02-24 02:34:16 -- STEP: 18286/36938 -- GLOBAL_STEP: 129100 | > loss_text_ce: 0.05316705256700516 (0.05323379270587346) | > loss_mel_ce: 4.027425765991211 (nan) | > loss: 0.016192829236388206 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.13630861646952044) | > loader_time: 0.0046 (0.0046459040834230065)  --> TIME: 2025-02-24 02:34:26 -- STEP: 18336/36938 -- GLOBAL_STEP: 129150 | > loss_text_ce: 0.06405576318502426 (0.05323158780120578) | > loss_mel_ce: 4.260252952575684 (nan) | > loss: 0.01715995743870735 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.1363373742833815) | > loader_time: 0.0045 (0.004645853974640257)  --> TIME: 2025-02-24 02:34:35 -- STEP: 18386/36938 -- GLOBAL_STEP: 129200 | > loss_text_ce: 0.05351591110229492 (0.05322921907635199) | > loss_mel_ce: 3.6109702587127686 (nan) | > loss: 0.014541612938046455 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.1363352967543094) | > loader_time: 0.0045 (0.004645821968567425)  --> TIME: 2025-02-24 02:34:44 -- STEP: 18436/36938 -- GLOBAL_STEP: 129250 | > loss_text_ce: 0.0541665144264698 (0.05322699225684481) | > loss_mel_ce: 4.081431865692139 (nan) | > loss: 0.016411105170845985 (nan) | > current_lr: 5e-06 | > step_time: 0.1448 (0.13632731114118768) | > loader_time: 0.0046 (0.004645738006284725)  --> TIME: 2025-02-24 02:34:54 -- STEP: 18486/36938 -- GLOBAL_STEP: 129300 | > loss_text_ce: 0.06013920158147812 (0.05322312808231953) | > loss_mel_ce: 3.4996774196624756 (nan) | > loss: 0.014126257039606571 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13633036105312712) | > loader_time: 0.0048 (0.004645730527489641)  --> TIME: 2025-02-24 02:35:03 -- STEP: 18536/36938 -- GLOBAL_STEP: 129350 | > loss_text_ce: 0.0357762910425663 (0.05321676924816271) | > loss_mel_ce: 3.0025863647460938 (nan) | > loss: 0.012056996114552021 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13633134930258364) | > loader_time: 0.0045 (0.004645618002734714)  --> TIME: 2025-02-24 02:35:12 -- STEP: 18586/36938 -- GLOBAL_STEP: 129400 | > loss_text_ce: 0.06749243289232254 (0.0532151974193863) | > loss_mel_ce: 3.950258731842041 (nan) | > loss: 0.015943458303809166 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13633465116534357) | > loader_time: 0.0046 (0.004645509354511209)  --> TIME: 2025-02-24 02:35:22 -- STEP: 18636/36938 -- GLOBAL_STEP: 129450 | > loss_text_ce: 0.056799061596393585 (0.053211210703348916) | > loss_mel_ce: 3.0252647399902344 (nan) | > loss: 0.012230413034558296 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13633843268107743) | > loader_time: 0.0048 (0.004645457887475536)  --> TIME: 2025-02-24 02:35:31 -- STEP: 18686/36938 -- GLOBAL_STEP: 129500 | > loss_text_ce: 0.04486628249287605 (0.053202084950659354) | > loss_mel_ce: 3.6953437328338623 (nan) | > loss: 0.01484210416674614 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13633811995698136) | > loader_time: 0.0046 (0.004645349215634377)  --> TIME: 2025-02-24 02:35:41 -- STEP: 18736/36938 -- GLOBAL_STEP: 129550 | > loss_text_ce: 0.056963324546813965 (0.0531982947790114) | > loss_mel_ce: 3.5024197101593018 (nan) | > loss: 0.014124535955488682 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.1363421380519872) | > loader_time: 0.0046 (0.004645251113059262)  --> TIME: 2025-02-24 02:35:50 -- STEP: 18786/36938 -- GLOBAL_STEP: 129600 | > loss_text_ce: 0.04472611844539642 (0.05319569346331472) | > loss_mel_ce: 4.132617473602295 (nan) | > loss: 0.01657675951719284 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.1363413418289908) | > loader_time: 0.0046 (0.004645112501753782)  --> TIME: 2025-02-24 02:35:59 -- STEP: 18836/36938 -- GLOBAL_STEP: 129650 | > loss_text_ce: 0.05039399862289429 (0.05319680041473546) | > loss_mel_ce: 2.2521708011627197 (nan) | > loss: 0.009137162938714027 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13635058253908647) | > loader_time: 0.0045 (0.004645016485021014)  --> TIME: 2025-02-24 02:36:09 -- STEP: 18886/36938 -- GLOBAL_STEP: 129700 | > loss_text_ce: 0.04896530508995056 (0.05319476920348379) | > loss_mel_ce: 3.588109254837036 (nan) | > loss: 0.014432836323976517 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13635488466907003) | > loader_time: 0.0045 (0.004644878774354297)  --> TIME: 2025-02-24 02:36:18 -- STEP: 18936/36938 -- GLOBAL_STEP: 129750 | > loss_text_ce: 0.05699364095926285 (0.053192252671837116) | > loss_mel_ce: 3.2382588386535645 (nan) | > loss: 0.013076399452984333 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13634639313616884) | > loader_time: 0.0047 (0.004644712920323869)  --> TIME: 2025-02-24 02:36:27 -- STEP: 18986/36938 -- GLOBAL_STEP: 129800 | > loss_text_ce: 0.05990982428193092 (0.05319358281903837) | > loss_mel_ce: 3.6304051876068115 (nan) | > loss: 0.014644107781350613 (nan) | > current_lr: 5e-06 | > step_time: 0.1927 (0.13635395206868173) | > loader_time: 0.0049 (0.004644613101235499)  --> TIME: 2025-02-24 02:36:36 -- STEP: 19036/36938 -- GLOBAL_STEP: 129850 | > loss_text_ce: 0.04567273333668709 (0.0531915982947009) | > loss_mel_ce: 3.544323444366455 (nan) | > loss: 0.01424601674079895 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13635088740389137) | > loader_time: 0.0045 (0.004644566823214275)  --> TIME: 2025-02-24 02:36:46 -- STEP: 19086/36938 -- GLOBAL_STEP: 129900 | > loss_text_ce: 0.04622436314821243 (0.05318608761097633) | > loss_mel_ce: 3.5602164268493652 (nan) | > loss: 0.014311273582279682 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13635166207537122) | > loader_time: 0.0045 (0.004644680970072124)  --> TIME: 2025-02-24 02:36:55 -- STEP: 19136/36938 -- GLOBAL_STEP: 129950 | > loss_text_ce: 0.05742621421813965 (0.05318483450534886) | > loss_mel_ce: 2.993922472000122 (nan) | > loss: 0.01210852712392807 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13634290865133472) | > loader_time: 0.0045 (0.0046446814908829735)  --> TIME: 2025-02-24 02:37:04 -- STEP: 19186/36938 -- GLOBAL_STEP: 130000 | > loss_text_ce: 0.05926119536161423 (0.0531822049897679) | > loss_mel_ce: 4.008507251739502 (nan) | > loss: 0.016141939908266068 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.1363448039664975) | > loader_time: 0.0047 (0.004644742005064968) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_130000.pth  --> TIME: 2025-02-24 02:37:19 -- STEP: 19236/36938 -- GLOBAL_STEP: 130050 | > loss_text_ce: 0.05916593596339226 (0.05317674011393322) | > loss_mel_ce: 4.727898120880127 (nan) | > loss: 0.018996287137269974 (nan) | > current_lr: 5e-06 | > step_time: 0.1618 (0.136335176077026) | > loader_time: 0.0049 (0.004644803568042215)  --> TIME: 2025-02-24 02:37:29 -- STEP: 19286/36938 -- GLOBAL_STEP: 130100 | > loss_text_ce: 0.049040112644433975 (0.053174714896340035) | > loss_mel_ce: 3.6219000816345215 (nan) | > loss: 0.014567223377525806 (nan) | > current_lr: 5e-06 | > step_time: 0.1446 (0.13636071212457634) | > loader_time: 0.0046 (0.004645028797210171)  --> TIME: 2025-02-24 02:37:38 -- STEP: 19336/36938 -- GLOBAL_STEP: 130150 | > loss_text_ce: 0.05612843856215477 (0.05316791879178871) | > loss_mel_ce: 2.863377809524536 (nan) | > loss: 0.01158534362912178 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13633857585449688) | > loader_time: 0.0046 (0.004645318877958384)  --> TIME: 2025-02-24 02:37:47 -- STEP: 19386/36938 -- GLOBAL_STEP: 130200 | > loss_text_ce: 0.055107664316892624 (0.05316073416681234) | > loss_mel_ce: 3.191300392150879 (nan) | > loss: 0.012882571667432785 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.1363512559361618) | > loader_time: 0.0048 (0.004645345996410782)  --> TIME: 2025-02-24 02:37:56 -- STEP: 19436/36938 -- GLOBAL_STEP: 130250 | > loss_text_ce: 0.07004957646131516 (0.05315884466504021) | > loss_mel_ce: 3.2543680667877197 (nan) | > loss: 0.013192133978009224 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13634563770576824) | > loader_time: 0.0046 (0.004645313346157781)  --> TIME: 2025-02-24 02:38:06 -- STEP: 19486/36938 -- GLOBAL_STEP: 130300 | > loss_text_ce: 0.05259173735976219 (0.05315297184794773) | > loss_mel_ce: 3.3753726482391357 (nan) | > loss: 0.013603034429252148 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1363471354790248) | > loader_time: 0.0046 (0.004645484655921134)  --> TIME: 2025-02-24 02:38:15 -- STEP: 19536/36938 -- GLOBAL_STEP: 130350 | > loss_text_ce: 0.0852440744638443 (0.05315252760273424) | > loss_mel_ce: 3.6451003551483154 (nan) | > loss: 0.014802955090999603 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13633298151420828) | > loader_time: 0.0045 (0.004645439357566192)  --> TIME: 2025-02-24 02:38:24 -- STEP: 19586/36938 -- GLOBAL_STEP: 130400 | > loss_text_ce: 0.05225173756480217 (0.0531475784807308) | > loss_mel_ce: 0.21771240234375 (nan) | > loss: 0.0010712862713262439 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13633550284625343) | > loader_time: 0.0045 (0.004645507948933937)  --> TIME: 2025-02-24 02:38:33 -- STEP: 19636/36938 -- GLOBAL_STEP: 130450 | > loss_text_ce: 0.039564959704875946 (0.053146406281933005) | > loss_mel_ce: 0.7990560531616211 (nan) | > loss: 0.0033278614282608032 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13634484085975485) | > loader_time: 0.0045 (0.004645545253396184)  --> TIME: 2025-02-24 02:38:43 -- STEP: 19686/36938 -- GLOBAL_STEP: 130500 | > loss_text_ce: 0.045201607048511505 (0.05314313586096201) | > loss_mel_ce: 1.6953667402267456 (nan) | > loss: 0.0069070179015398026 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.1363511085631364) | > loader_time: 0.0045 (0.004645571819616604)  --> TIME: 2025-02-24 02:38:52 -- STEP: 19736/36938 -- GLOBAL_STEP: 130550 | > loss_text_ce: 0.03633315861225128 (0.05313735160739573) | > loss_mel_ce: 3.5406904220581055 (nan) | > loss: 0.014194538816809654 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13634642244689993) | > loader_time: 0.0044 (0.00464540035027841)  --> TIME: 2025-02-24 02:39:01 -- STEP: 19786/36938 -- GLOBAL_STEP: 130600 | > loss_text_ce: 0.05185099318623543 (0.053131780081701524) | > loss_mel_ce: 0.1160956472158432 (nan) | > loss: 0.0006664549582637846 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.1363442847139009) | > loader_time: 0.0046 (0.004645332219590295)  --> TIME: 2025-02-24 02:39:11 -- STEP: 19836/36938 -- GLOBAL_STEP: 130650 | > loss_text_ce: 0.04828726500272751 (0.05312724086679678) | > loss_mel_ce: 3.532116413116455 (nan) | > loss: 0.014207951724529266 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13636477235392913) | > loader_time: 0.0047 (0.0046455483326966526)  --> TIME: 2025-02-24 02:39:20 -- STEP: 19886/36938 -- GLOBAL_STEP: 130700 | > loss_text_ce: 0.047254785895347595 (0.05311858108604201) | > loss_mel_ce: 3.7632689476013184 (nan) | > loss: 0.015121126547455788 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1363648480480854) | > loader_time: 0.0045 (0.004645486970452781)  --> TIME: 2025-02-24 02:39:30 -- STEP: 19936/36938 -- GLOBAL_STEP: 130750 | > loss_text_ce: 0.045929111540317535 (0.05311507945913931) | > loss_mel_ce: 4.055198669433594 (nan) | > loss: 0.016274316236376762 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13637302921155103) | > loader_time: 0.0046 (0.004645449810483459)  --> TIME: 2025-02-24 02:39:39 -- STEP: 19986/36938 -- GLOBAL_STEP: 130800 | > loss_text_ce: 0.06442373245954514 (0.05310992553021568) | > loss_mel_ce: 3.4364986419677734 (nan) | > loss: 0.013892550021409988 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13637551214677568) | > loader_time: 0.0047 (0.00464545073576495)  --> TIME: 2025-02-24 02:39:49 -- STEP: 20036/36938 -- GLOBAL_STEP: 130850 | > loss_text_ce: 0.05741959810256958 (0.05310964972042197) | > loss_mel_ce: 3.184481143951416 (nan) | > loss: 0.012864685617387295 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13638257425829528) | > loader_time: 0.0046 (0.0046453816635114815)  --> TIME: 2025-02-24 02:39:58 -- STEP: 20086/36938 -- GLOBAL_STEP: 130900 | > loss_text_ce: 0.05106062442064285 (0.053107972029642216) | > loss_mel_ce: 3.7706596851348877 (nan) | > loss: 0.015165558084845543 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13638294920626484) | > loader_time: 0.0044 (0.0046453328765531585)  --> TIME: 2025-02-24 02:40:08 -- STEP: 20136/36938 -- GLOBAL_STEP: 130950 | > loss_text_ce: 0.0384228490293026 (0.0531046533642428) | > loss_mel_ce: 2.408724784851074 (nan) | > loss: 0.009710904210805893 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.1363835940770228) | > loader_time: 0.0044 (0.0046452899205575455)  --> TIME: 2025-02-24 02:40:17 -- STEP: 20186/36938 -- GLOBAL_STEP: 131000 | > loss_text_ce: 0.07079530507326126 (0.053103101726561984) | > loss_mel_ce: 3.600215435028076 (nan) | > loss: 0.014567503705620766 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.136382261624625) | > loader_time: 0.0045 (0.004645216078773802)  --> TIME: 2025-02-24 02:40:26 -- STEP: 20236/36938 -- GLOBAL_STEP: 131050 | > loss_text_ce: 0.05903215333819389 (0.05310026243986996) | > loss_mel_ce: 3.025542974472046 (nan) | > loss: 0.012240378186106682 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.1363909778257538) | > loader_time: 0.0046 (0.004645202041591264)  --> TIME: 2025-02-24 02:40:36 -- STEP: 20286/36938 -- GLOBAL_STEP: 131100 | > loss_text_ce: 0.043571289628744125 (0.05309785575926656) | > loss_mel_ce: 3.7135493755340576 (nan) | > loss: 0.014909209683537483 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13639589689027798) | > loader_time: 0.0049 (0.004645175885886183)  --> TIME: 2025-02-24 02:40:45 -- STEP: 20336/36938 -- GLOBAL_STEP: 131150 | > loss_text_ce: 0.0531839020550251 (0.05309724961369015) | > loss_mel_ce: 3.5626943111419678 (nan) | > loss: 0.01434872392565012 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.1363907418060641) | > loader_time: 0.0045 (0.004645133914204652)  --> TIME: 2025-02-24 02:40:54 -- STEP: 20386/36938 -- GLOBAL_STEP: 131200 | > loss_text_ce: 0.05315869674086571 (0.05309079588674772) | > loss_mel_ce: 3.301624298095703 (nan) | > loss: 0.013312632218003273 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13639069506385954) | > loader_time: 0.0046 (0.004645086558096915)  --> TIME: 2025-02-24 02:41:04 -- STEP: 20436/36938 -- GLOBAL_STEP: 131250 | > loss_text_ce: 0.03125247731804848 (0.0530861409500068) | > loss_mel_ce: 6.333333492279053 (nan) | > loss: 0.025256294757127762 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13640188449841711) | > loader_time: 0.0044 (0.00464510326167096)  --> TIME: 2025-02-24 02:41:13 -- STEP: 20486/36938 -- GLOBAL_STEP: 131300 | > loss_text_ce: 0.0492030568420887 (0.05308115881290622) | > loss_mel_ce: 0.1638009250164032 (nan) | > loss: 0.0008452539332211018 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13641195308646586) | > loader_time: 0.0045 (0.004645072376892122)  --> TIME: 2025-02-24 02:41:23 -- STEP: 20536/36938 -- GLOBAL_STEP: 131350 | > loss_text_ce: 0.04751504212617874 (0.05308108145262603) | > loss_mel_ce: 3.956223964691162 (nan) | > loss: 0.015887852758169174 (nan) | > current_lr: 5e-06 | > step_time: 0.1789 (0.13642563957345138) | > loader_time: 0.0048 (0.004645097009579277)  --> TIME: 2025-02-24 02:41:32 -- STEP: 20586/36938 -- GLOBAL_STEP: 131400 | > loss_text_ce: 0.03645145148038864 (0.053073059092585716) | > loss_mel_ce: 2.8416264057159424 (nan) | > loss: 0.011420943774282932 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13642006981052038) | > loader_time: 0.0044 (0.004645068409444487)  --> TIME: 2025-02-24 02:41:42 -- STEP: 20636/36938 -- GLOBAL_STEP: 131450 | > loss_text_ce: 0.03380883112549782 (0.05306479973226779) | > loss_mel_ce: 2.3154025077819824 (nan) | > loss: 0.009322267957031727 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1364280043079958) | > loader_time: 0.0044 (0.004645044211154496)  --> TIME: 2025-02-24 02:41:51 -- STEP: 20686/36938 -- GLOBAL_STEP: 131500 | > loss_text_ce: 0.04846729710698128 (0.053062257891981726) | > loss_mel_ce: 3.863788366317749 (nan) | > loss: 0.01552482508122921 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.13642997430738532) | > loader_time: 0.0049 (0.004645036357889564)  --> TIME: 2025-02-24 02:42:00 -- STEP: 20736/36938 -- GLOBAL_STEP: 131550 | > loss_text_ce: 0.05216987431049347 (0.053057505067741165) | > loss_mel_ce: 3.3919804096221924 (nan) | > loss: 0.013667263090610504 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.1364260418340568) | > loader_time: 0.0048 (0.004644966891242361)  --> TIME: 2025-02-24 02:42:09 -- STEP: 20786/36938 -- GLOBAL_STEP: 131600 | > loss_text_ce: 0.04859438166022301 (0.053052241573006115) | > loss_mel_ce: 2.3661158084869385 (nan) | > loss: 0.009582184255123138 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13643088364158434) | > loader_time: 0.0046 (0.004644978279261283)  --> TIME: 2025-02-24 02:42:19 -- STEP: 20836/36938 -- GLOBAL_STEP: 131650 | > loss_text_ce: 0.06273332238197327 (0.05304850348378527) | > loss_mel_ce: 4.163181781768799 (nan) | > loss: 0.01676950417459011 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13643679347187526) | > loader_time: 0.0047 (0.004644952778807949)  --> TIME: 2025-02-24 02:42:28 -- STEP: 20886/36938 -- GLOBAL_STEP: 131700 | > loss_text_ce: 0.042028430849313736 (0.05304443786493465) | > loss_mel_ce: 4.0853095054626465 (nan) | > loss: 0.01637832634150982 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13642485950181074) | > loader_time: 0.0044 (0.004644935025824503)  --> TIME: 2025-02-24 02:42:37 -- STEP: 20936/36938 -- GLOBAL_STEP: 131750 | > loss_text_ce: 0.05084022879600525 (0.053040954643891795) | > loss_mel_ce: 3.7472987174987793 (nan) | > loss: 0.015071980655193329 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13641311616035795) | > loader_time: 0.0045 (0.004644835899474706)  --> TIME: 2025-02-24 02:42:46 -- STEP: 20986/36938 -- GLOBAL_STEP: 131800 | > loss_text_ce: 0.07013533264398575 (0.053039079996135316) | > loss_mel_ce: 4.28576135635376 (nan) | > loss: 0.01728530414402485 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1364082973814328) | > loader_time: 0.0045 (0.004644742585064638)  --> TIME: 2025-02-24 02:42:56 -- STEP: 21036/36938 -- GLOBAL_STEP: 131850 | > loss_text_ce: 0.06046003848314285 (0.053038885329842425) | > loss_mel_ce: 3.9201409816741943 (nan) | > loss: 0.015796037390828133 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13642224264271877) | > loader_time: 0.0047 (0.0046447498146901045)  --> TIME: 2025-02-24 02:43:05 -- STEP: 21086/36938 -- GLOBAL_STEP: 131900 | > loss_text_ce: 0.05281905084848404 (0.053039088886505756) | > loss_mel_ce: 3.8648815155029297 (nan) | > loss: 0.015546431764960289 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13642105110499214) | > loader_time: 0.0045 (0.00464469020850376)  --> TIME: 2025-02-24 02:43:14 -- STEP: 21136/36938 -- GLOBAL_STEP: 131950 | > loss_text_ce: 0.05515864118933678 (0.05303580882998903) | > loss_mel_ce: 4.1550726890563965 (nan) | > loss: 0.016707267612218857 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13642717610246635) | > loader_time: 0.0046 (0.0046447092028479514)  --> TIME: 2025-02-24 02:43:24 -- STEP: 21186/36938 -- GLOBAL_STEP: 132000 | > loss_text_ce: 0.03767067939043045 (0.05302715757415988) | > loss_mel_ce: 3.8870224952697754 (nan) | > loss: 0.015574179589748383 (nan) | > current_lr: 5e-06 | > step_time: 0.1789 (0.13643780052183682) | > loader_time: 0.005 (0.004644765942109947)  --> TIME: 2025-02-24 02:43:33 -- STEP: 21236/36938 -- GLOBAL_STEP: 132050 | > loss_text_ce: 0.05151023715734482 (0.05302516992247223) | > loss_mel_ce: 4.181497097015381 (nan) | > loss: 0.01679764874279499 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13642833571488874) | > loader_time: 0.0047 (0.004644717294901687)  --> TIME: 2025-02-24 02:43:42 -- STEP: 21286/36938 -- GLOBAL_STEP: 132100 | > loss_text_ce: 0.049547065049409866 (0.05302413969325104) | > loss_mel_ce: 3.6375789642333984 (nan) | > loss: 0.014631452970206738 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.136423053311796) | > loader_time: 0.0062 (0.004644782709177041)  --> TIME: 2025-02-24 02:43:52 -- STEP: 21336/36938 -- GLOBAL_STEP: 132150 | > loss_text_ce: 0.045081157237291336 (0.053016866050345654) | > loss_mel_ce: 2.797388792037964 (nan) | > loss: 0.011279643513262272 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13642913082989652) | > loader_time: 0.0046 (0.004644966299810037)  --> TIME: 2025-02-24 02:44:01 -- STEP: 21386/36938 -- GLOBAL_STEP: 132200 | > loss_text_ce: 0.04982732608914375 (0.05300864610050381) | > loss_mel_ce: 3.5646939277648926 (nan) | > loss: 0.014343339018523693 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13643200488016113) | > loader_time: 0.0046 (0.00464490099237749)  --> TIME: 2025-02-24 02:44:10 -- STEP: 21436/36938 -- GLOBAL_STEP: 132250 | > loss_text_ce: 0.058839842677116394 (0.05300260337370631) | > loss_mel_ce: 3.432962417602539 (nan) | > loss: 0.013856358826160431 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.13642279664445617) | > loader_time: 0.0045 (0.004645111712530276)  --> TIME: 2025-02-24 02:44:20 -- STEP: 21486/36938 -- GLOBAL_STEP: 132300 | > loss_text_ce: 0.0618864968419075 (0.05299419266346265) | > loss_mel_ce: 3.859070301055908 (nan) | > loss: 0.015559353865683079 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13642996696634538) | > loader_time: 0.0046 (0.0046453209858993115)  --> TIME: 2025-02-24 02:44:29 -- STEP: 21536/36938 -- GLOBAL_STEP: 132350 | > loss_text_ce: 0.04486103728413582 (0.05298598845216709) | > loss_mel_ce: 2.1158463954925537 (nan) | > loss: 0.008574236184358597 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.136427425866585) | > loader_time: 0.0046 (0.0046454886912767794)  --> TIME: 2025-02-24 02:44:38 -- STEP: 21586/36938 -- GLOBAL_STEP: 132400 | > loss_text_ce: 0.05201127380132675 (0.05298236635141094) | > loss_mel_ce: 3.6177468299865723 (nan) | > loss: 0.014562533237040043 (nan) | > current_lr: 5e-06 | > step_time: 0.1504 (0.13642980651905037) | > loader_time: 0.0047 (0.0046455705728056)  --> TIME: 2025-02-24 02:44:48 -- STEP: 21636/36938 -- GLOBAL_STEP: 132450 | > loss_text_ce: 0.05318562686443329 (0.052978433889692804) | > loss_mel_ce: 3.984729766845703 (nan) | > loss: 0.01602347381412983 (nan) | > current_lr: 5e-06 | > step_time: 0.1262 (0.1364333720078515) | > loader_time: 0.0047 (0.004645847716642637)  --> TIME: 2025-02-24 02:44:57 -- STEP: 21686/36938 -- GLOBAL_STEP: 132500 | > loss_text_ce: 0.04697881639003754 (0.0529766735306176) | > loss_mel_ce: 3.6307730674743652 (nan) | > loss: 0.014594254083931446 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13643299182116891) | > loader_time: 0.0045 (0.004646100593780762)  --> TIME: 2025-02-24 02:45:07 -- STEP: 21736/36938 -- GLOBAL_STEP: 132550 | > loss_text_ce: 0.060409121215343475 (0.05297603749354394) | > loss_mel_ce: 3.7384867668151855 (nan) | > loss: 0.015074984170496464 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13644144563145) | > loader_time: 0.0045 (0.0046463912578465875)  --> TIME: 2025-02-24 02:45:16 -- STEP: 21786/36938 -- GLOBAL_STEP: 132600 | > loss_text_ce: 0.051889147609472275 (0.05297406321005188) | > loss_mel_ce: 1.6469168663024902 (nan) | > loss: 0.0067412941716611385 (nan) | > current_lr: 5e-06 | > step_time: 0.1118 (0.1364411926243275) | > loader_time: 0.0046 (0.00464653695219529)  --> TIME: 2025-02-24 02:45:25 -- STEP: 21836/36938 -- GLOBAL_STEP: 132650 | > loss_text_ce: 0.04658257216215134 (0.05297081017319779) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1364419950651025) | > loader_time: 0.0045 (0.00464638496061057)  --> TIME: 2025-02-24 02:45:35 -- STEP: 21886/36938 -- GLOBAL_STEP: 132700 | > loss_text_ce: 0.05934889242053032 (0.05296745572780782) | > loss_mel_ce: 3.0876262187957764 (nan) | > loss: 0.01248799730092287 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.1364611798924117) | > loader_time: 0.0044 (0.004646263828031789)  --> TIME: 2025-02-24 02:45:44 -- STEP: 21936/36938 -- GLOBAL_STEP: 132750 | > loss_text_ce: 0.04618241637945175 (0.05296823864308189) | > loss_mel_ce: 2.160283327102661 (nan) | > loss: 0.008755817078053951 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.1364566466718767) | > loader_time: 0.0045 (0.004646085566799594)  --> TIME: 2025-02-24 02:45:54 -- STEP: 21986/36938 -- GLOBAL_STEP: 132800 | > loss_text_ce: 0.050248607993125916 (0.05296447717502578) | > loss_mel_ce: 3.9513473510742188 (nan) | > loss: 0.015879349783062935 (nan) | > current_lr: 5e-06 | > step_time: 0.1791 (0.13646518428278231) | > loader_time: 0.0048 (0.004646099959492901)  --> TIME: 2025-02-24 02:46:03 -- STEP: 22036/36938 -- GLOBAL_STEP: 132850 | > loss_text_ce: 0.05137583985924721 (0.05296318805128213) | > loss_mel_ce: 3.748931646347046 (nan) | > loss: 0.015080586075782776 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13646273664693379) | > loader_time: 0.0046 (0.004646025718416104)  --> TIME: 2025-02-24 02:46:12 -- STEP: 22086/36938 -- GLOBAL_STEP: 132900 | > loss_text_ce: 0.056366726756095886 (0.05295784545185764) | > loss_mel_ce: 3.9805963039398193 (nan) | > loss: 0.01601969450712204 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13645039676249052) | > loader_time: 0.0046 (0.004645876852938985)  --> TIME: 2025-02-24 02:46:21 -- STEP: 22136/36938 -- GLOBAL_STEP: 132950 | > loss_text_ce: 0.04798877239227295 (0.05295264098803704) | > loss_mel_ce: 3.9453697204589844 (nan) | > loss: 0.01584666222333908 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.1364578230849759) | > loader_time: 0.0046 (0.004646080385531384)  --> TIME: 2025-02-24 02:46:31 -- STEP: 22186/36938 -- GLOBAL_STEP: 133000 | > loss_text_ce: 0.05329287424683571 (0.05294805684495566) | > loss_mel_ce: 7.651041507720947 (nan) | > loss: 0.030572757124900818 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13645999120825691) | > loader_time: 0.0047 (0.004646240971746697)  --> TIME: 2025-02-24 02:46:40 -- STEP: 22236/36938 -- GLOBAL_STEP: 133050 | > loss_text_ce: 0.060671236366033554 (0.05294640508595468) | > loss_mel_ce: 3.849289655685425 (nan) | > loss: 0.015515719540417194 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.13646652657129305) | > loader_time: 0.0053 (0.00464645516509993)  --> TIME: 2025-02-24 02:46:50 -- STEP: 22286/36938 -- GLOBAL_STEP: 133100 | > loss_text_ce: 0.05500119924545288 (0.05294455690691364) | > loss_mel_ce: 3.1106152534484863 (nan) | > loss: 0.012561971321702003 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.136468743631788) | > loader_time: 0.0045 (0.004646533568778574)  --> TIME: 2025-02-24 02:46:59 -- STEP: 22336/36938 -- GLOBAL_STEP: 133150 | > loss_text_ce: 0.057035211473703384 (0.05293894255898073) | > loss_mel_ce: 3.996929883956909 (nan) | > loss: 0.016087163239717484 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.13646309432122739) | > loader_time: 0.0047 (0.004646374889379896)  --> TIME: 2025-02-24 02:47:08 -- STEP: 22386/36938 -- GLOBAL_STEP: 133200 | > loss_text_ce: 0.06161193549633026 (0.0529362442877011) | > loss_mel_ce: 2.986886739730835 (nan) | > loss: 0.01209721714258194 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.1364692498774298) | > loader_time: 0.0045 (0.004646318831946877)  --> TIME: 2025-02-24 02:47:18 -- STEP: 22436/36938 -- GLOBAL_STEP: 133250 | > loss_text_ce: 0.049312371760606766 (0.05293379342181812) | > loss_mel_ce: 3.921614646911621 (nan) | > loss: 0.015757648274302483 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.13647010079210709) | > loader_time: 0.0046 (0.0046463340101107415)  --> TIME: 2025-02-24 02:47:27 -- STEP: 22486/36938 -- GLOBAL_STEP: 133300 | > loss_text_ce: 0.047904517501592636 (0.05292622587019564) | > loss_mel_ce: 3.8412516117095947 (nan) | > loss: 0.015433159656822681 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.1364689111539958) | > loader_time: 0.0045 (0.004646441430306681)  --> TIME: 2025-02-24 02:47:36 -- STEP: 22536/36938 -- GLOBAL_STEP: 133350 | > loss_text_ce: 0.04172854498028755 (0.05291920748017241) | > loss_mel_ce: 4.231586933135986 (nan) | > loss: 0.01695760153234005 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.1364686029956924) | > loader_time: 0.0048 (0.004646581096093925)  --> TIME: 2025-02-24 02:47:46 -- STEP: 22586/36938 -- GLOBAL_STEP: 133400 | > loss_text_ce: 0.062213897705078125 (0.05291425211363834) | > loss_mel_ce: 1.5880126953125 (nan) | > loss: 0.006548518780618906 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.1364687251250872) | > loader_time: 0.0046 (0.004646720270180301)  --> TIME: 2025-02-24 02:47:55 -- STEP: 22636/36938 -- GLOBAL_STEP: 133450 | > loss_text_ce: 0.0512223094701767 (0.052911786234572766) | > loss_mel_ce: 3.808109760284424 (nan) | > loss: 0.015314810909330845 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13646141354493016) | > loader_time: 0.0046 (0.004646838585549693)  --> TIME: 2025-02-24 02:48:04 -- STEP: 22686/36938 -- GLOBAL_STEP: 133500 | > loss_text_ce: 0.0548313669860363 (0.052911407261365014) | > loss_mel_ce: 3.7392807006835938 (nan) | > loss: 0.015056001022458076 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13645226519479733) | > loader_time: 0.0046 (0.004647054170307323)  --> TIME: 2025-02-24 02:48:13 -- STEP: 22736/36938 -- GLOBAL_STEP: 133550 | > loss_text_ce: 0.058051954954862595 (0.05290883075028731) | > loss_mel_ce: 4.012012004852295 (nan) | > loss: 0.016151048243045807 (nan) | > current_lr: 5e-06 | > step_time: 0.1797 (0.13646675649451087) | > loader_time: 0.0049 (0.004647134025276383)  --> TIME: 2025-02-24 02:48:23 -- STEP: 22786/36938 -- GLOBAL_STEP: 133600 | > loss_text_ce: 0.058113183826208115 (0.05290960021525294) | > loss_mel_ce: 4.150508880615234 (nan) | > loss: 0.016700882464647293 (nan) | > current_lr: 5e-06 | > step_time: 0.1264 (0.13646970552357315) | > loader_time: 0.0052 (0.004647227529791739)  --> TIME: 2025-02-24 02:48:32 -- STEP: 22836/36938 -- GLOBAL_STEP: 133650 | > loss_text_ce: 0.05517682433128357 (0.05291205966508451) | > loss_mel_ce: 3.8580539226531982 (nan) | > loss: 0.015528693795204163 (nan) | > current_lr: 5e-06 | > step_time: 0.1764 (0.13647444295390473) | > loader_time: 0.0055 (0.004647281285154111)  --> TIME: 2025-02-24 02:48:42 -- STEP: 22886/36938 -- GLOBAL_STEP: 133700 | > loss_text_ce: 0.04725249111652374 (0.052906140325916375) | > loss_mel_ce: 3.0462985038757324 (nan) | > loss: 0.01227599661797285 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13648119230511324) | > loader_time: 0.0045 (0.00464769785068544)  --> TIME: 2025-02-24 02:48:51 -- STEP: 22936/36938 -- GLOBAL_STEP: 133750 | > loss_text_ce: 0.049506861716508865 (0.05290487491334704) | > loss_mel_ce: 4.153928279876709 (nan) | > loss: 0.016680298373103142 (nan) | > current_lr: 5e-06 | > step_time: 0.178 (0.13648062760421686) | > loader_time: 0.0048 (0.004648022590134214)  --> TIME: 2025-02-24 02:49:00 -- STEP: 22986/36938 -- GLOBAL_STEP: 133800 | > loss_text_ce: 0.05713414028286934 (0.05290402363506706) | > loss_mel_ce: 4.082693576812744 (nan) | > loss: 0.016427889466285706 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.1364776817965865) | > loader_time: 0.0048 (0.0046483038466063175)  --> TIME: 2025-02-24 02:49:10 -- STEP: 23036/36938 -- GLOBAL_STEP: 133850 | > loss_text_ce: 0.04587102681398392 (0.05290606306389721) | > loss_mel_ce: 3.6826932430267334 (nan) | > loss: 0.014795891009271145 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.1364750804162567) | > loader_time: 0.0046 (0.004648331491291779)  --> TIME: 2025-02-24 02:49:19 -- STEP: 23086/36938 -- GLOBAL_STEP: 133900 | > loss_text_ce: 0.04814471676945686 (0.05290293243517539) | > loss_mel_ce: 4.154252052307129 (nan) | > loss: 0.016676178202033043 (nan) | > current_lr: 5e-06 | > step_time: 0.1691 (0.13647585929106285) | > loader_time: 0.0046 (0.00464850348636425)  --> TIME: 2025-02-24 02:49:28 -- STEP: 23136/36938 -- GLOBAL_STEP: 133950 | > loss_text_ce: 0.049374714493751526 (0.0528948616411474) | > loss_mel_ce: 3.3583984375 (nan) | > loss: 0.013522910885512829 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13646151035776835) | > loader_time: 0.0045 (0.004648582311669122)  --> TIME: 2025-02-24 02:49:37 -- STEP: 23186/36938 -- GLOBAL_STEP: 134000 | > loss_text_ce: 0.05673709884285927 (0.052888294011511926) | > loss_mel_ce: 3.4222114086151123 (nan) | > loss: 0.01380535215139389 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13645469461638704) | > loader_time: 0.0046 (0.004648566163769733)  --> TIME: 2025-02-24 02:49:46 -- STEP: 23236/36938 -- GLOBAL_STEP: 134050 | > loss_text_ce: 0.048497289419174194 (0.05288705474485909) | > loss_mel_ce: 3.559136390686035 (nan) | > loss: 0.014316007494926453 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13644809219303297) | > loader_time: 0.0045 (0.004648647377712184)  --> TIME: 2025-02-24 02:49:55 -- STEP: 23286/36938 -- GLOBAL_STEP: 134100 | > loss_text_ce: 0.059883616864681244 (0.052884822953957966) | > loss_mel_ce: 3.2402894496917725 (nan) | > loss: 0.013095925562083721 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.1364397942625685) | > loader_time: 0.0045 (0.00464849839410776)  --> TIME: 2025-02-24 02:50:04 -- STEP: 23336/36938 -- GLOBAL_STEP: 134150 | > loss_text_ce: 0.04655971750617027 (0.05288251567471367) | > loss_mel_ce: 3.7984061241149902 (nan) | > loss: 0.015257801860570908 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.13644605703373577) | > loader_time: 0.0046 (0.004648505313207615)  --> TIME: 2025-02-24 02:50:14 -- STEP: 23386/36938 -- GLOBAL_STEP: 134200 | > loss_text_ce: 0.05329684168100357 (0.05287944534679602) | > loss_mel_ce: 4.1547017097473145 (nan) | > loss: 0.016698408871889114 (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.1364390862708857) | > loader_time: 0.0048 (0.004648533347001005)  --> TIME: 2025-02-24 02:50:23 -- STEP: 23436/36938 -- GLOBAL_STEP: 134250 | > loss_text_ce: 0.05587254837155342 (0.05287179048105575) | > loss_mel_ce: 3.639106512069702 (nan) | > loss: 0.014662615954875946 (nan) | > current_lr: 5e-06 | > step_time: 0.1718 (0.13643531924243113) | > loader_time: 0.005 (0.00464873251643961)  --> TIME: 2025-02-24 02:50:32 -- STEP: 23486/36938 -- GLOBAL_STEP: 134300 | > loss_text_ce: 0.038841407746076584 (0.05286888777257983) | > loss_mel_ce: 3.5611674785614014 (nan) | > loss: 0.014285750687122345 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13643923065056024) | > loader_time: 0.0045 (0.004648705788815115)  --> TIME: 2025-02-24 02:50:42 -- STEP: 23536/36938 -- GLOBAL_STEP: 134350 | > loss_text_ce: 0.049834903329610825 (0.0528723790733249) | > loss_mel_ce: 3.3940372467041016 (nan) | > loss: 0.01366616040468216 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.1364432227684486) | > loader_time: 0.0045 (0.004648608771576313)  --> TIME: 2025-02-24 02:50:51 -- STEP: 23586/36938 -- GLOBAL_STEP: 134400 | > loss_text_ce: 0.04864900931715965 (0.05287136723492971) | > loss_mel_ce: 3.7965734004974365 (nan) | > loss: 0.015258819796144962 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.136452169664174) | > loader_time: 0.0046 (0.004648727263990214)  --> TIME: 2025-02-24 02:51:01 -- STEP: 23636/36938 -- GLOBAL_STEP: 134450 | > loss_text_ce: 0.054815974086523056 (0.05286923355999299) | > loss_mel_ce: 4.068929195404053 (nan) | > loss: 0.016364067792892456 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.13645478979668574) | > loader_time: 0.0047 (0.004648611062996073)  --> TIME: 2025-02-24 02:51:10 -- STEP: 23686/36938 -- GLOBAL_STEP: 134500 | > loss_text_ce: 0.04687918722629547 (0.052860225388833885) | > loss_mel_ce: 0.120452880859375 (nan) | > loss: 0.0006640161736868322 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.1364556939016279) | > loader_time: 0.0045 (0.004648656777853171)  --> TIME: 2025-02-24 02:51:19 -- STEP: 23736/36938 -- GLOBAL_STEP: 134550 | > loss_text_ce: 0.042821697890758514 (0.05285307591015749) | > loss_mel_ce: 3.1597402095794678 (nan) | > loss: 0.012708579190075397 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.1364502323007287) | > loader_time: 0.0052 (0.004648703384929725)  --> TIME: 2025-02-24 02:51:28 -- STEP: 23786/36938 -- GLOBAL_STEP: 134600 | > loss_text_ce: 0.058050643652677536 (0.05284621116886838) | > loss_mel_ce: 4.0166239738464355 (nan) | > loss: 0.016169345006346703 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13644718140536685) | > loader_time: 0.0046 (0.0046489513683166325)  --> TIME: 2025-02-24 02:51:37 -- STEP: 23836/36938 -- GLOBAL_STEP: 134650 | > loss_text_ce: 0.051365580409765244 (0.05284177152389004) | > loss_mel_ce: 3.1435680389404297 (nan) | > loss: 0.012678309343755245 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13644956914265188) | > loader_time: 0.005 (0.004648943418703461)  --> TIME: 2025-02-24 02:51:46 -- STEP: 23886/36938 -- GLOBAL_STEP: 134700 | > loss_text_ce: 0.04473854973912239 (0.052833338326040834) | > loss_mel_ce: 3.9749245643615723 (nan) | > loss: 0.015951046720147133 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.1364406889371124) | > loader_time: 0.0047 (0.0046487638401962825)  --> TIME: 2025-02-24 02:51:56 -- STEP: 23936/36938 -- GLOBAL_STEP: 134750 | > loss_text_ce: 0.04795369878411293 (0.052829226024634406) | > loss_mel_ce: 0.8367933630943298 (nan) | > loss: 0.003510901238769293 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1364428543430927) | > loader_time: 0.0045 (0.004648615730637495)  --> TIME: 2025-02-24 02:52:05 -- STEP: 23986/36938 -- GLOBAL_STEP: 134800 | > loss_text_ce: 0.05843145772814751 (0.0528225844716182) | > loss_mel_ce: 4.040775299072266 (nan) | > loss: 0.01626669429242611 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13644339616251258) | > loader_time: 0.0046 (0.004648534259416003)  --> TIME: 2025-02-24 02:52:14 -- STEP: 24036/36938 -- GLOBAL_STEP: 134850 | > loss_text_ce: 0.04363173991441727 (0.05281801229556084) | > loss_mel_ce: 3.7475063800811768 (nan) | > loss: 0.015044200234115124 (nan) | > current_lr: 5e-06 | > step_time: 0.1605 (0.13644218370430086) | > loader_time: 0.0046 (0.004648465754327534)  --> TIME: 2025-02-24 02:52:24 -- STEP: 24086/36938 -- GLOBAL_STEP: 134900 | > loss_text_ce: 0.06821110099554062 (0.052813635426642684) | > loss_mel_ce: 1.0441616773605347 (nan) | > loss: 0.004414177965372801 (nan) | > current_lr: 5e-06 | > step_time: 0.17 (0.13644191555175944) | > loader_time: 0.0047 (0.004648632764875639)  --> TIME: 2025-02-24 02:52:33 -- STEP: 24136/36938 -- GLOBAL_STEP: 134950 | > loss_text_ce: 0.05628059431910515 (0.05281118139143656) | > loss_mel_ce: 3.741982936859131 (nan) | > loss: 0.01507247518748045 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13644318707857855) | > loader_time: 0.0047 (0.004648672910095371)  --> TIME: 2025-02-24 02:52:42 -- STEP: 24186/36938 -- GLOBAL_STEP: 135000 | > loss_text_ce: 0.047299813479185104 (0.05280563953617323) | > loss_mel_ce: 3.7950048446655273 (nan) | > loss: 0.015247241593897343 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.1364506127904715) | > loader_time: 0.0047 (0.004648967168964568)  --> TIME: 2025-02-24 02:52:52 -- STEP: 24236/36938 -- GLOBAL_STEP: 135050 | > loss_text_ce: 0.0453396700322628 (0.052795968869942984) | > loss_mel_ce: 4.003824710845947 (nan) | > loss: 0.0160681139677763 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1364419945377519) | > loader_time: 0.0045 (0.00464912946516576)  --> TIME: 2025-02-24 02:53:01 -- STEP: 24286/36938 -- GLOBAL_STEP: 135100 | > loss_text_ce: 0.05012938752770424 (0.052794349252751845) | > loss_mel_ce: 2.280123710632324 (nan) | > loss: 0.009247036650776863 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13643416237872458) | > loader_time: 0.0045 (0.004649056797707685)  --> TIME: 2025-02-24 02:53:10 -- STEP: 24336/36938 -- GLOBAL_STEP: 135150 | > loss_text_ce: 0.04533849656581879 (0.05279212874234249) | > loss_mel_ce: 4.115146160125732 (nan) | > loss: 0.01650986075401306 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.13643348987050882) | > loader_time: 0.0055 (0.004649211933626899)  --> TIME: 2025-02-24 02:53:19 -- STEP: 24386/36938 -- GLOBAL_STEP: 135200 | > loss_text_ce: 0.057722460478544235 (0.052791303852285094) | > loss_mel_ce: 3.8346476554870605 (nan) | > loss: 0.01544591411948204 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.13643961924383524) | > loader_time: 0.0047 (0.004649521357547933)  --> TIME: 2025-02-24 02:53:29 -- STEP: 24436/36938 -- GLOBAL_STEP: 135250 | > loss_text_ce: 0.0453731007874012 (0.052787490949043246) | > loss_mel_ce: 4.404306411743164 (nan) | > loss: 0.017657458782196045 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1364387098360158) | > loader_time: 0.0046 (0.00464973055139695)  --> TIME: 2025-02-24 02:53:38 -- STEP: 24486/36938 -- GLOBAL_STEP: 135300 | > loss_text_ce: 0.03838357329368591 (0.05278227603476657) | > loss_mel_ce: 3.0556640625 (nan) | > loss: 0.012277967296540737 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.1364312761507432) | > loader_time: 0.0045 (0.00464987285501529)  --> TIME: 2025-02-24 02:53:47 -- STEP: 24536/36938 -- GLOBAL_STEP: 135350 | > loss_text_ce: 0.041408054530620575 (0.05277904189857685) | > loss_mel_ce: 2.7161526679992676 (nan) | > loss: 0.010942702181637287 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.1364328936446878) | > loader_time: 0.0045 (0.004649895243042994)  --> TIME: 2025-02-24 02:53:57 -- STEP: 24586/36938 -- GLOBAL_STEP: 135400 | > loss_text_ce: 0.0555262565612793 (0.05277128270845797) | > loss_mel_ce: 1.425721526145935 (nan) | > loss: 0.005877967923879623 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13644066388735324) | > loader_time: 0.0045 (0.004650267400317178)  --> TIME: 2025-02-24 02:54:06 -- STEP: 24636/36938 -- GLOBAL_STEP: 135450 | > loss_text_ce: 0.04472691938281059 (0.05276787496590905) | > loss_mel_ce: 3.9761664867401123 (nan) | > loss: 0.015955928713083267 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.13643665467788985) | > loader_time: 0.0046 (0.004650279644886961)  --> TIME: 2025-02-24 02:54:15 -- STEP: 24686/36938 -- GLOBAL_STEP: 135500 | > loss_text_ce: 0.05573858693242073 (0.05276395854601515) | > loss_mel_ce: 3.201054811477661 (nan) | > loss: 0.01292378455400467 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13644271609482525) | > loader_time: 0.0045 (0.00465035189359949)  --> TIME: 2025-02-24 02:54:25 -- STEP: 24736/36938 -- GLOBAL_STEP: 135550 | > loss_text_ce: 0.05688217654824257 (0.05276070385303876) | > loss_mel_ce: 4.196464538574219 (nan) | > loss: 0.016878360882401466 (nan) | > current_lr: 5e-06 | > step_time: 0.1844 (0.13645245698684344) | > loader_time: 0.0049 (0.004650396390072237)  --> TIME: 2025-02-24 02:54:34 -- STEP: 24786/36938 -- GLOBAL_STEP: 135600 | > loss_text_ce: 0.04805862903594971 (0.052756900124400354) | > loss_mel_ce: 3.0929291248321533 (nan) | > loss: 0.012464238330721855 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13645130122226493) | > loader_time: 0.0048 (0.004650347401922034)  --> TIME: 2025-02-24 02:54:43 -- STEP: 24836/36938 -- GLOBAL_STEP: 135650 | > loss_text_ce: 0.06304895132780075 (0.05275027171652697) | > loss_mel_ce: 3.7119548320770264 (nan) | > loss: 0.014980174601078033 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.13643847853497415) | > loader_time: 0.0048 (0.004650186495919907)  --> TIME: 2025-02-24 02:54:52 -- STEP: 24886/36938 -- GLOBAL_STEP: 135700 | > loss_text_ce: 0.03239517658948898 (0.052744194238457075) | > loss_mel_ce: 3.2688100337982178 (nan) | > loss: 0.01310002151876688 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13643291876418281) | > loader_time: 0.0045 (0.004650091057679025)  --> TIME: 2025-02-24 02:55:02 -- STEP: 24936/36938 -- GLOBAL_STEP: 135750 | > loss_text_ce: 0.05693089962005615 (0.052741539800963314) | > loss_mel_ce: 3.193387031555176 (nan) | > loss: 0.012898088432848454 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13644286129656005) | > loader_time: 0.0052 (0.0046503892838285175)  --> TIME: 2025-02-24 02:55:11 -- STEP: 24986/36938 -- GLOBAL_STEP: 135800 | > loss_text_ce: 0.0517507866024971 (0.05273931953117622) | > loss_mel_ce: 4.0663018226623535 (nan) | > loss: 0.016341479495167732 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.13644461650668296) | > loader_time: 0.0048 (0.004650307972666062)  --> TIME: 2025-02-24 02:55:21 -- STEP: 25036/36938 -- GLOBAL_STEP: 135850 | > loss_text_ce: 0.050185564905405045 (0.05273439127247753) | > loss_mel_ce: 3.2141659259796143 (nan) | > loss: 0.012953776866197586 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.13644402083125606) | > loader_time: 0.0046 (0.004650326216253188)  --> TIME: 2025-02-24 02:55:30 -- STEP: 25086/36938 -- GLOBAL_STEP: 135900 | > loss_text_ce: 0.04228145629167557 (0.052732353140344995) | > loss_mel_ce: 3.125511646270752 (nan) | > loss: 0.012570608407258987 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.136438871232135) | > loader_time: 0.0044 (0.004650418756301397)  --> TIME: 2025-02-24 02:55:39 -- STEP: 25136/36938 -- GLOBAL_STEP: 135950 | > loss_text_ce: 0.06173840910196304 (0.05272904851587691) | > loss_mel_ce: 3.9425113201141357 (nan) | > loss: 0.015889879316091537 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13643290836632033) | > loader_time: 0.0045 (0.004650464726055864)  --> TIME: 2025-02-24 02:55:48 -- STEP: 25186/36938 -- GLOBAL_STEP: 136000 | > loss_text_ce: 0.035367608070373535 (0.052730755983273314) | > loss_mel_ce: 2.9859089851379395 (nan) | > loss: 0.01198919303715229 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13643164152111958) | > loader_time: 0.0048 (0.004650542197041517)  --> TIME: 2025-02-24 02:55:57 -- STEP: 25236/36938 -- GLOBAL_STEP: 136050 | > loss_text_ce: 0.044286489486694336 (0.0527282548313413) | > loss_mel_ce: 3.711894989013672 (nan) | > loss: 0.014905482530593872 (nan) | > current_lr: 5e-06 | > step_time: 0.1566 (0.1364234367893023) | > loader_time: 0.0048 (0.004650650698591777)  --> TIME: 2025-02-24 02:56:06 -- STEP: 25286/36938 -- GLOBAL_STEP: 136100 | > loss_text_ce: 0.064713254570961 (0.05272202506981999) | > loss_mel_ce: 3.8064064979553223 (nan) | > loss: 0.015361587516963482 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.1364213274332922) | > loader_time: 0.0058 (0.004650736170026987)  --> TIME: 2025-02-24 02:56:16 -- STEP: 25336/36938 -- GLOBAL_STEP: 136150 | > loss_text_ce: 0.0473308265209198 (0.052717347125330785) | > loss_mel_ce: 2.93462872505188 (nan) | > loss: 0.011833173222839832 (nan) | > current_lr: 5e-06 | > step_time: 0.1577 (0.13642469285638917) | > loader_time: 0.0049 (0.0046507471511895845)  --> TIME: 2025-02-24 02:56:25 -- STEP: 25386/36938 -- GLOBAL_STEP: 136200 | > loss_text_ce: 0.06264042854309082 (0.052716447389693805) | > loss_mel_ce: 3.5678396224975586 (nan) | > loss: 0.014406668022274971 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.13642622455678735) | > loader_time: 0.0047 (0.004650655258991321)  --> TIME: 2025-02-24 02:56:34 -- STEP: 25436/36938 -- GLOBAL_STEP: 136250 | > loss_text_ce: 0.04678962007164955 (0.052715388237466354) | > loss_mel_ce: 3.7032382488250732 (nan) | > loss: 0.014881064184010029 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13642731818374548) | > loader_time: 0.0046 (0.0046507851341494365)  --> TIME: 2025-02-24 02:56:43 -- STEP: 25486/36938 -- GLOBAL_STEP: 136300 | > loss_text_ce: 0.043831709772348404 (0.05271168527347798) | > loss_mel_ce: 0.2444196492433548 (nan) | > loss: 0.0011438546935096383 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.1364174143332367) | > loader_time: 0.0045 (0.004650618389563372)  --> TIME: 2025-02-24 02:56:53 -- STEP: 25536/36938 -- GLOBAL_STEP: 136350 | > loss_text_ce: 0.05643551051616669 (0.05270773139678674) | > loss_mel_ce: 3.301630973815918 (nan) | > loss: 0.013325661420822144 (nan) | > current_lr: 5e-06 | > step_time: 0.1268 (0.13641700139245963) | > loader_time: 0.0046 (0.004650601935117734)  --> TIME: 2025-02-24 02:57:02 -- STEP: 25586/36938 -- GLOBAL_STEP: 136400 | > loss_text_ce: 0.05861907824873924 (0.0527050521360581) | > loss_mel_ce: 3.6409683227539062 (nan) | > loss: 0.014680903404951096 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13640859724721907) | > loader_time: 0.0045 (0.00465043663820162)  --> TIME: 2025-02-24 02:57:11 -- STEP: 25636/36938 -- GLOBAL_STEP: 136450 | > loss_text_ce: 0.055814310908317566 (0.0527016655496909) | > loss_mel_ce: 3.634951591491699 (nan) | > loss: 0.014645896852016449 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.13640109401613085) | > loader_time: 0.0046 (0.004650321639556902)  --> TIME: 2025-02-24 02:57:20 -- STEP: 25686/36938 -- GLOBAL_STEP: 136500 | > loss_text_ce: 0.03989021107554436 (0.05269662113807315) | > loss_mel_ce: 2.708808422088623 (nan) | > loss: 0.010907535441219807 (nan) | > current_lr: 5e-06 | > step_time: 0.161 (0.13639024167587036) | > loader_time: 0.0048 (0.004650265890369595)  --> TIME: 2025-02-24 02:57:29 -- STEP: 25736/36938 -- GLOBAL_STEP: 136550 | > loss_text_ce: 0.04623040184378624 (0.05268919099924059) | > loss_mel_ce: 3.6052653789520264 (nan) | > loss: 0.014490063302218914 (nan) | > current_lr: 5e-06 | > step_time: 0.1757 (0.13638899428784498) | > loader_time: 0.0048 (0.004650205855492566)  --> TIME: 2025-02-24 02:57:39 -- STEP: 25786/36938 -- GLOBAL_STEP: 136600 | > loss_text_ce: 0.04048129916191101 (0.05268356602700072) | > loss_mel_ce: 3.9044740200042725 (nan) | > loss: 0.015654586255550385 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.13639915669056002) | > loader_time: 0.0047 (0.004650153367058712)  --> TIME: 2025-02-24 02:57:48 -- STEP: 25836/36938 -- GLOBAL_STEP: 136650 | > loss_text_ce: 0.05095376446843147 (0.05267884338380291) | > loss_mel_ce: 3.7016992568969727 (nan) | > loss: 0.014891481958329678 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13640004797863356) | > loader_time: 0.0046 (0.004650121180703686)  --> TIME: 2025-02-24 02:57:57 -- STEP: 25886/36938 -- GLOBAL_STEP: 136700 | > loss_text_ce: 0.05041690543293953 (0.05267112193847029) | > loss_mel_ce: 3.057046413421631 (nan) | > loss: 0.012331204488873482 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.136396268894562) | > loader_time: 0.0046 (0.004650053944440677)  --> TIME: 2025-02-24 02:58:06 -- STEP: 25936/36938 -- GLOBAL_STEP: 136750 | > loss_text_ce: 0.036911915987730026 (0.05266838008117761) | > loss_mel_ce: 3.899765968322754 (nan) | > loss: 0.015621738508343697 (nan) | > current_lr: 5e-06 | > step_time: 0.1457 (0.13639465237311038) | > loader_time: 0.0046 (0.004650090586102173)  --> TIME: 2025-02-24 02:58:16 -- STEP: 25986/36938 -- GLOBAL_STEP: 136800 | > loss_text_ce: 0.03932752460241318 (0.05266371753829012) | > loss_mel_ce: 3.15815806388855 (nan) | > loss: 0.01268843561410904 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13639674346900105) | > loader_time: 0.0045 (0.004650106204718482)  --> TIME: 2025-02-24 02:58:25 -- STEP: 26036/36938 -- GLOBAL_STEP: 136850 | > loss_text_ce: 0.04616628214716911 (0.05266099252669667) | > loss_mel_ce: 4.266221523284912 (nan) | > loss: 0.017112651839852333 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.13639931199663868) | > loader_time: 0.0046 (0.004649989349279043)  --> TIME: 2025-02-24 02:58:34 -- STEP: 26086/36938 -- GLOBAL_STEP: 136900 | > loss_text_ce: 0.05818898230791092 (0.05265892008453738) | > loss_mel_ce: 4.431692600250244 (nan) | > loss: 0.017816990613937378 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.1363974266016424) | > loader_time: 0.0045 (0.004649888022329965)  --> TIME: 2025-02-24 02:58:43 -- STEP: 26136/36938 -- GLOBAL_STEP: 136950 | > loss_text_ce: 0.04008275270462036 (0.05265609797270315) | > loss_mel_ce: 3.5683743953704834 (nan) | > loss: 0.014319274574518204 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13639107125776848) | > loader_time: 0.0047 (0.00464973946503623)  --> TIME: 2025-02-24 02:58:53 -- STEP: 26186/36938 -- GLOBAL_STEP: 137000 | > loss_text_ce: 0.05470709502696991 (0.05265284765129493) | > loss_mel_ce: 0.099212646484375 (nan) | > loss: 0.0006107926601544023 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13639477113620224) | > loader_time: 0.0044 (0.004649643427109398)  --> TIME: 2025-02-24 02:59:02 -- STEP: 26236/36938 -- GLOBAL_STEP: 137050 | > loss_text_ce: 0.049407437443733215 (0.052647255946688835) | > loss_mel_ce: 3.0208089351654053 (nan) | > loss: 0.012183398939669132 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13640062899683364) | > loader_time: 0.0044 (0.004649557069882224)  --> TIME: 2025-02-24 02:59:11 -- STEP: 26286/36938 -- GLOBAL_STEP: 137100 | > loss_text_ce: 0.062445420771837234 (0.05264195873072771) | > loss_mel_ce: 3.803056001663208 (nan) | > loss: 0.015339292585849762 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13640092475532412) | > loader_time: 0.0046 (0.004649472655675281)  --> TIME: 2025-02-24 02:59:21 -- STEP: 26336/36938 -- GLOBAL_STEP: 137150 | > loss_text_ce: 0.052614592015743256 (0.05263864083481102) | > loss_mel_ce: 3.7606027126312256 (nan) | > loss: 0.015131816267967224 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13640619019546638) | > loader_time: 0.0045 (0.0046494132312940055)  --> TIME: 2025-02-24 02:59:30 -- STEP: 26386/36938 -- GLOBAL_STEP: 137200 | > loss_text_ce: 0.047499869018793106 (0.052632448881253804) | > loss_mel_ce: 3.366318941116333 (nan) | > loss: 0.013546900823712349 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13641779708110569) | > loader_time: 0.0046 (0.004649623606136829)  --> TIME: 2025-02-24 02:59:40 -- STEP: 26436/36938 -- GLOBAL_STEP: 137250 | > loss_text_ce: 0.03670959919691086 (0.05262977499724082) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1118 (0.13642704871945824) | > loader_time: 0.0044 (0.004649723193020995)  --> TIME: 2025-02-24 02:59:49 -- STEP: 26486/36938 -- GLOBAL_STEP: 137300 | > loss_text_ce: 0.045527081936597824 (0.0526262182633845) | > loss_mel_ce: 3.701537609100342 (nan) | > loss: 0.01486930437386036 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.1364307866701054) | > loader_time: 0.0046 (0.004649731873979824)  --> TIME: 2025-02-24 02:59:58 -- STEP: 26536/36938 -- GLOBAL_STEP: 137350 | > loss_text_ce: 0.0584569051861763 (0.052626134618732986) | > loss_mel_ce: 3.274462938308716 (nan) | > loss: 0.013225873000919819 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1364230108271895) | > loader_time: 0.0045 (0.004649707826821935)  --> TIME: 2025-02-24 03:00:08 -- STEP: 26586/36938 -- GLOBAL_STEP: 137400 | > loss_text_ce: 0.05852177366614342 (0.052623345687536204) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.1364269332369917) | > loader_time: 0.0055 (0.004649940968671797)  --> TIME: 2025-02-24 03:00:17 -- STEP: 26636/36938 -- GLOBAL_STEP: 137450 | > loss_text_ce: 0.05887600779533386 (0.052621118612074184) | > loss_mel_ce: 4.134895324707031 (nan) | > loss: 0.016641950234770775 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.13642710307675987) | > loader_time: 0.0047 (0.004650203534335497)  --> TIME: 2025-02-24 03:00:27 -- STEP: 26686/36938 -- GLOBAL_STEP: 137500 | > loss_text_ce: 0.05046863853931427 (0.05261689334233787) | > loss_mel_ce: 4.386224746704102 (nan) | > loss: 0.01760592684149742 (nan) | > current_lr: 5e-06 | > step_time: 0.1499 (0.13642922896348272) | > loader_time: 0.0046 (0.004650616426031168)  --> TIME: 2025-02-24 03:00:36 -- STEP: 26736/36938 -- GLOBAL_STEP: 137550 | > loss_text_ce: 0.04824935272336006 (0.05261738337617495) | > loss_mel_ce: 1.358123779296875 (nan) | > loss: 0.005580846220254898 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13642552039317957) | > loader_time: 0.0052 (0.004650551159177811)  --> TIME: 2025-02-24 03:00:45 -- STEP: 26786/36938 -- GLOBAL_STEP: 137600 | > loss_text_ce: 0.055455125868320465 (0.05261392234533583) | > loss_mel_ce: 3.7592368125915527 (nan) | > loss: 0.015137667767703533 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13641789428934836) | > loader_time: 0.0046 (0.004650846282798319)  --> TIME: 2025-02-24 03:00:54 -- STEP: 26836/36938 -- GLOBAL_STEP: 137650 | > loss_text_ce: 0.05352704972028732 (0.05261275471866389) | > loss_mel_ce: 3.935153007507324 (nan) | > loss: 0.015828097239136696 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13641598927741969) | > loader_time: 0.0045 (0.004651062702489385)  --> TIME: 2025-02-24 03:01:03 -- STEP: 26886/36938 -- GLOBAL_STEP: 137700 | > loss_text_ce: 0.05985286086797714 (0.05261251436127168) | > loss_mel_ce: 4.078609943389893 (nan) | > loss: 0.016422472894191742 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13641097396113389) | > loader_time: 0.0046 (0.0046513755078703214)  --> TIME: 2025-02-24 03:01:13 -- STEP: 26936/36938 -- GLOBAL_STEP: 137750 | > loss_text_ce: 0.05041540041565895 (0.05260685111203362) | > loss_mel_ce: 3.119758367538452 (nan) | > loss: 0.012580055743455887 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.136414126982348) | > loader_time: 0.0046 (0.004651495423981139)  --> TIME: 2025-02-24 03:01:22 -- STEP: 26986/36938 -- GLOBAL_STEP: 137800 | > loss_text_ce: 0.07921149581670761 (0.05260311769026698) | > loss_mel_ce: 1.4134047031402588 (nan) | > loss: 0.005923080258071423 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13641571977392478) | > loader_time: 0.0045 (0.004651646047580979)  --> TIME: 2025-02-24 03:01:31 -- STEP: 27036/36938 -- GLOBAL_STEP: 137850 | > loss_text_ce: 0.04458017274737358 (0.052596600682946346) | > loss_mel_ce: 2.7870686054229736 (nan) | > loss: 0.011236702091991901 (nan) | > current_lr: 5e-06 | > step_time: 0.1818 (0.13641237349248495) | > loader_time: 0.0051 (0.004651692522438074)  --> TIME: 2025-02-24 03:01:40 -- STEP: 27086/36938 -- GLOBAL_STEP: 137900 | > loss_text_ce: 0.05726366862654686 (0.05259578259257991) | > loss_mel_ce: 4.1725945472717285 (nan) | > loss: 0.01678515411913395 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13640114214134647) | > loader_time: 0.0048 (0.004651550597738901)  --> TIME: 2025-02-24 03:01:50 -- STEP: 27136/36938 -- GLOBAL_STEP: 137950 | > loss_text_ce: 0.05001034960150719 (0.05258997880346731) | > loss_mel_ce: 2.957385301589966 (nan) | > loss: 0.011934110894799232 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13640978861213635) | > loader_time: 0.0045 (0.00465160002053344)  --> TIME: 2025-02-24 03:01:59 -- STEP: 27186/36938 -- GLOBAL_STEP: 138000 | > loss_text_ce: 0.05161659047007561 (0.052584704345997005) | > loss_mel_ce: 3.667964220046997 (nan) | > loss: 0.014760242775082588 (nan) | > current_lr: 5e-06 | > step_time: 0.1438 (0.13640969710608625) | > loader_time: 0.0047 (0.004651773890622475)  --> TIME: 2025-02-24 03:02:08 -- STEP: 27236/36938 -- GLOBAL_STEP: 138050 | > loss_text_ce: 0.05178648978471756 (0.052580654039098965) | > loss_mel_ce: 4.152270317077637 (nan) | > loss: 0.016682766377925873 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13640477625279257) | > loader_time: 0.0047 (0.004651717151168972)  --> TIME: 2025-02-24 03:02:17 -- STEP: 27286/36938 -- GLOBAL_STEP: 138100 | > loss_text_ce: 0.07299478352069855 (0.05257938800259218) | > loss_mel_ce: 2.3633573055267334 (nan) | > loss: 0.009668064303696156 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13640920372116988) | > loader_time: 0.0045 (0.004651701905585283)  --> TIME: 2025-02-24 03:02:27 -- STEP: 27336/36938 -- GLOBAL_STEP: 138150 | > loss_text_ce: 0.050126370042562485 (0.05257558129424079) | > loss_mel_ce: 2.825103759765625 (nan) | > loss: 0.011409644037485123 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1364180212101818) | > loader_time: 0.0045 (0.004651763519767685)  --> TIME: 2025-02-24 03:02:36 -- STEP: 27386/36938 -- GLOBAL_STEP: 138200 | > loss_text_ce: 0.05618984252214432 (0.05257154960412595) | > loss_mel_ce: 4.028183460235596 (nan) | > loss: 0.016207832843065262 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13642300907727897) | > loader_time: 0.0047 (0.0046518247783780085)  --> TIME: 2025-02-24 03:02:45 -- STEP: 27436/36938 -- GLOBAL_STEP: 138250 | > loss_text_ce: 0.04992280900478363 (0.052564259432874044) | > loss_mel_ce: 3.5091099739074707 (nan) | > loss: 0.014123146422207355 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13641436829270165) | > loader_time: 0.0046 (0.0046520361157856484)  --> TIME: 2025-02-24 03:02:55 -- STEP: 27486/36938 -- GLOBAL_STEP: 138300 | > loss_text_ce: 0.05296853557229042 (0.05256291946109172) | > loss_mel_ce: 3.74165940284729 (nan) | > loss: 0.015058048069477081 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13640781551575717) | > loader_time: 0.0048 (0.004652339567447972)  --> TIME: 2025-02-24 03:03:04 -- STEP: 27536/36938 -- GLOBAL_STEP: 138350 | > loss_text_ce: 0.04859921336174011 (0.052558867414331786) | > loss_mel_ce: 3.833423137664795 (nan) | > loss: 0.01540485117584467 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13640966596712908) | > loader_time: 0.0053 (0.004652431768293201)  --> TIME: 2025-02-24 03:03:13 -- STEP: 27586/36938 -- GLOBAL_STEP: 138400 | > loss_text_ce: 0.04146445542573929 (0.05255559583483615) | > loss_mel_ce: 0.0107421875 (nan) | > loss: 0.00020716922881547362 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13641239536653033) | > loader_time: 0.0045 (0.0046525787755782935)  --> TIME: 2025-02-24 03:03:22 -- STEP: 27636/36938 -- GLOBAL_STEP: 138450 | > loss_text_ce: 0.04402941092848778 (0.052554413873977565) | > loss_mel_ce: 3.9660189151763916 (nan) | > loss: 0.015912890434265137 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13640291859227524) | > loader_time: 0.0046 (0.004652612184128902)  --> TIME: 2025-02-24 03:03:32 -- STEP: 27686/36938 -- GLOBAL_STEP: 138500 | > loss_text_ce: 0.051168859004974365 (0.0525507319008889) | > loss_mel_ce: 3.057987689971924 (nan) | > loss: 0.012337923981249332 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13640825563984907) | > loader_time: 0.0045 (0.004652691913944483)  --> TIME: 2025-02-24 03:03:42 -- STEP: 27736/36938 -- GLOBAL_STEP: 138550 | > loss_text_ce: 0.029003936797380447 (0.052546664560945706) | > loss_mel_ce: 2.8119168281555176 (nan) | > loss: 0.011273495852947235 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13642316402500584) | > loader_time: 0.0053 (0.004652868001097217)  --> TIME: 2025-02-24 03:03:51 -- STEP: 27786/36938 -- GLOBAL_STEP: 138600 | > loss_text_ce: 0.050163548439741135 (0.052541951040779515) | > loss_mel_ce: 3.779341220855713 (nan) | > loss: 0.015196448192000389 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13643175611089006) | > loader_time: 0.0047 (0.00465313268344497)  --> TIME: 2025-02-24 03:04:00 -- STEP: 27836/36938 -- GLOBAL_STEP: 138650 | > loss_text_ce: 0.054874539375305176 (0.05253732703954175) | > loss_mel_ce: 2.928114652633667 (nan) | > loss: 0.011837259866297245 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13643115281819) | > loader_time: 0.0046 (0.00465326137730469)  --> TIME: 2025-02-24 03:04:09 -- STEP: 27886/36938 -- GLOBAL_STEP: 138700 | > loss_text_ce: 0.037557002156972885 (0.05253674734690679) | > loss_mel_ce: 3.362413167953491 (nan) | > loss: 0.013491945341229439 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13642866654400615) | > loader_time: 0.0045 (0.004653405520765993)  --> TIME: 2025-02-24 03:04:19 -- STEP: 27936/36938 -- GLOBAL_STEP: 138750 | > loss_text_ce: 0.04435073211789131 (0.05253032844422044) | > loss_mel_ce: 3.0433156490325928 (nan) | > loss: 0.012252644635736942 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13642889008625914) | > loader_time: 0.0054 (0.004653508609581074)  --> TIME: 2025-02-24 03:04:28 -- STEP: 27986/36938 -- GLOBAL_STEP: 138800 | > loss_text_ce: 0.04954219609498978 (0.05252485049176074) | > loss_mel_ce: 4.317896842956543 (nan) | > loss: 0.017331108450889587 (nan) | > current_lr: 5e-06 | > step_time: 0.1872 (0.13642699444327286) | > loader_time: 0.0048 (0.004653546319954839)  --> TIME: 2025-02-24 03:04:37 -- STEP: 28036/36938 -- GLOBAL_STEP: 138850 | > loss_text_ce: 0.055547673255205154 (0.05251868337882188) | > loss_mel_ce: 3.043487548828125 (nan) | > loss: 0.012297759763896465 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13641896995706984) | > loader_time: 0.0046 (0.004653697560493069)  --> TIME: 2025-02-24 03:04:46 -- STEP: 28086/36938 -- GLOBAL_STEP: 138900 | > loss_text_ce: 0.05716433748602867 (0.05251374106630971) | > loss_mel_ce: 3.937746524810791 (nan) | > loss: 0.015852821990847588 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.1364186666956291) | > loader_time: 0.0046 (0.004653846692098344)  --> TIME: 2025-02-24 03:04:55 -- STEP: 28136/36938 -- GLOBAL_STEP: 138950 | > loss_text_ce: 0.05419779568910599 (0.052510212684472056) | > loss_mel_ce: 2.2930352687835693 (nan) | > loss: 0.009314417839050293 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1364137720902408) | > loader_time: 0.0052 (0.004653977303807259)  --> TIME: 2025-02-24 03:05:05 -- STEP: 28186/36938 -- GLOBAL_STEP: 139000 | > loss_text_ce: 0.061775654554367065 (0.05250947882259344) | > loss_mel_ce: 1.526629090309143 (nan) | > loss: 0.0063031939789652824 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13641740578278172) | > loader_time: 0.0046 (0.004654065944993982)  --> TIME: 2025-02-24 03:05:14 -- STEP: 28236/36938 -- GLOBAL_STEP: 139050 | > loss_text_ce: 0.04515782743692398 (0.05250627315348038) | > loss_mel_ce: 0.054931640625 (nan) | > loss: 0.00039718043990433216 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.13642047879074176) | > loader_time: 0.0044 (0.0046539855148589665)  --> TIME: 2025-02-24 03:05:24 -- STEP: 28286/36938 -- GLOBAL_STEP: 139100 | > loss_text_ce: 0.055107660591602325 (0.05250346378691879) | > loss_mel_ce: 3.6679790019989014 (nan) | > loss: 0.01477415394037962 (nan) | > current_lr: 5e-06 | > step_time: 0.1886 (0.13642022249461525) | > loader_time: 0.0048 (0.004654154332146153)  --> TIME: 2025-02-24 03:05:33 -- STEP: 28336/36938 -- GLOBAL_STEP: 139150 | > loss_text_ce: 0.05319899320602417 (0.05249874759567345) | > loss_mel_ce: 3.890463352203369 (nan) | > loss: 0.015649454668164253 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.13641432653807636) | > loader_time: 0.0046 (0.004654258986033642)  --> TIME: 2025-02-24 03:05:42 -- STEP: 28386/36938 -- GLOBAL_STEP: 139200 | > loss_text_ce: 0.05115163326263428 (0.05249711856379727) | > loss_mel_ce: 3.319929599761963 (nan) | > loss: 0.013377307914197445 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13641569184877383) | > loader_time: 0.0047 (0.004654223685586078)  --> TIME: 2025-02-24 03:05:51 -- STEP: 28436/36938 -- GLOBAL_STEP: 139250 | > loss_text_ce: 0.04981895908713341 (0.052494903923254566) | > loss_mel_ce: 3.9418013095855713 (nan) | > loss: 0.01583976484835148 (nan) | > current_lr: 5e-06 | > step_time: 0.1623 (0.13641345086757128) | > loader_time: 0.0047 (0.0046542467472673365)  --> TIME: 2025-02-24 03:06:00 -- STEP: 28486/36938 -- GLOBAL_STEP: 139300 | > loss_text_ce: 0.04743044450879097 (0.05249159493373949) | > loss_mel_ce: 4.169311046600342 (nan) | > loss: 0.01673310250043869 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.1364046242902822) | > loader_time: 0.0046 (0.004654149321837484)  --> TIME: 2025-02-24 03:06:09 -- STEP: 28536/36938 -- GLOBAL_STEP: 139350 | > loss_text_ce: 0.06052625551819801 (0.052490060012491256) | > loss_mel_ce: 3.577260732650757 (nan) | > loss: 0.01443566381931305 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.1363991093478491) | > loader_time: 0.0046 (0.004654181063726603)  --> TIME: 2025-02-24 03:06:19 -- STEP: 28586/36938 -- GLOBAL_STEP: 139400 | > loss_text_ce: 0.04535109922289848 (0.05248980906380245) | > loss_mel_ce: 3.3157694339752197 (nan) | > loss: 0.013337780721485615 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13640335483080018) | > loader_time: 0.0046 (0.004654189166316745)  --> TIME: 2025-02-24 03:06:28 -- STEP: 28636/36938 -- GLOBAL_STEP: 139450 | > loss_text_ce: 0.048203807324171066 (0.052485608271137) | > loss_mel_ce: 3.872436046600342 (nan) | > loss: 0.015558094717562199 (nan) | > current_lr: 5e-06 | > step_time: 0.1888 (0.13640430948057963) | > loader_time: 0.0049 (0.0046543097725825445)  --> TIME: 2025-02-24 03:06:37 -- STEP: 28686/36938 -- GLOBAL_STEP: 139500 | > loss_text_ce: 0.037818703800439835 (0.052482600150714175) | > loss_mel_ce: 0.05724334716796875 (nan) | > loss: 0.0003772303753066808 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13639415996822615) | > loader_time: 0.0045 (0.004654224660450903)  --> TIME: 2025-02-24 03:06:46 -- STEP: 28736/36938 -- GLOBAL_STEP: 139550 | > loss_text_ce: 0.044239893555641174 (0.0524795766630304) | > loss_mel_ce: 3.1778066158294678 (nan) | > loss: 0.012785900384187698 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13639427152335865) | > loader_time: 0.0045 (0.004654363312142435)  --> TIME: 2025-02-24 03:06:55 -- STEP: 28786/36938 -- GLOBAL_STEP: 139600 | > loss_text_ce: 0.06600473821163177 (0.05247448360655355) | > loss_mel_ce: 3.4532361030578613 (nan) | > loss: 0.01396524254232645 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13638933239309242) | > loader_time: 0.0045 (0.0046544956844559865)  --> TIME: 2025-02-24 03:07:04 -- STEP: 28836/36938 -- GLOBAL_STEP: 139650 | > loss_text_ce: 0.0418081097304821 (0.05247311386839668) | > loss_mel_ce: 3.705897808074951 (nan) | > loss: 0.014871849678456783 (nan) | > current_lr: 5e-06 | > step_time: 0.1518 (0.13638382264588594) | > loader_time: 0.0048 (0.004654474042791094)  --> TIME: 2025-02-24 03:07:13 -- STEP: 28886/36938 -- GLOBAL_STEP: 139700 | > loss_text_ce: 0.06806104630231857 (0.0524697109785341) | > loss_mel_ce: 3.620203971862793 (nan) | > loss: 0.014635973609983921 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.1363778159384474) | > loader_time: 0.0046 (0.004654376929233657)  --> TIME: 2025-02-24 03:07:22 -- STEP: 28936/36938 -- GLOBAL_STEP: 139750 | > loss_text_ce: 0.044291432946920395 (0.05246401781114945) | > loss_mel_ce: 2.99344539642334 (nan) | > loss: 0.012054512277245522 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.1363717706651039) | > loader_time: 0.0046 (0.004654258085872547)  --> TIME: 2025-02-24 03:07:32 -- STEP: 28986/36938 -- GLOBAL_STEP: 139800 | > loss_text_ce: 0.058268092572689056 (0.05245996429686132) | > loss_mel_ce: 1.2713557481765747 (nan) | > loss: 0.005276285111904144 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13636925130340288) | > loader_time: 0.0045 (0.004654476346991291)  --> TIME: 2025-02-24 03:07:41 -- STEP: 29036/36938 -- GLOBAL_STEP: 139850 | > loss_text_ce: 0.04109329730272293 (0.05245543160000572) | > loss_mel_ce: 3.2460973262786865 (nan) | > loss: 0.013044408522546291 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.13636925263515945) | > loader_time: 0.0046 (0.0046544720736034426)  --> TIME: 2025-02-24 03:07:50 -- STEP: 29086/36938 -- GLOBAL_STEP: 139900 | > loss_text_ce: 0.05149422958493233 (0.05245514126404361) | > loss_mel_ce: 3.4595675468444824 (nan) | > loss: 0.013932785019278526 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.136374253809621) | > loader_time: 0.0046 (0.00465458527823173)  --> TIME: 2025-02-24 03:08:00 -- STEP: 29136/36938 -- GLOBAL_STEP: 139950 | > loss_text_ce: 0.04908960312604904 (0.052452500635661654) | > loss_mel_ce: 3.8785438537597656 (nan) | > loss: 0.01558584813028574 (nan) | > current_lr: 5e-06 | > step_time: 0.1931 (0.1363844330791333) | > loader_time: 0.0048 (0.0046546340136001405)  --> TIME: 2025-02-24 03:08:09 -- STEP: 29186/36938 -- GLOBAL_STEP: 140000 | > loss_text_ce: 0.04876802861690521 (0.05244601850528653) | > loss_mel_ce: 4.368833541870117 (nan) | > loss: 0.0175301656126976 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13638419490023013) | > loader_time: 0.0046 (0.004654547255385387) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_140000.pth  --> TIME: 2025-02-24 03:08:25 -- STEP: 29236/36938 -- GLOBAL_STEP: 140050 | > loss_text_ce: 0.05148107931017876 (0.05244044585139828) | > loss_mel_ce: 1.6896693706512451 (nan) | > loss: 0.006909327581524849 (nan) | > current_lr: 5e-06 | > step_time: 0.1533 (0.136389496943289) | > loader_time: 0.0046 (0.004654827220998661)  --> TIME: 2025-02-24 03:08:34 -- STEP: 29286/36938 -- GLOBAL_STEP: 140100 | > loss_text_ce: 0.053732071071863174 (0.052435545233441926) | > loss_mel_ce: 3.2300233840942383 (nan) | > loss: 0.013030776754021645 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13639641482998002) | > loader_time: 0.0047 (0.004654787199456327)  --> TIME: 2025-02-24 03:08:44 -- STEP: 29336/36938 -- GLOBAL_STEP: 140150 | > loss_text_ce: 0.05904039368033409 (0.05243386542837904) | > loss_mel_ce: 3.6924970149993896 (nan) | > loss: 0.014887053519487381 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13639185410414623) | > loader_time: 0.0047 (0.004654805415457496)  --> TIME: 2025-02-24 03:08:52 -- STEP: 29386/36938 -- GLOBAL_STEP: 140200 | > loss_text_ce: 0.04301275312900543 (0.05243355221269905) | > loss_mel_ce: 4.312798023223877 (nan) | > loss: 0.01728496327996254 (nan) | > current_lr: 5e-06 | > step_time: 0.1583 (0.13638243883420303) | > loader_time: 0.0048 (0.004654714566760839)  --> TIME: 2025-02-24 03:09:02 -- STEP: 29436/36938 -- GLOBAL_STEP: 140250 | > loss_text_ce: 0.0524737685918808 (0.0524306571949071) | > loss_mel_ce: 3.6099436283111572 (nan) | > loss: 0.01453340332955122 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13638653232280631) | > loader_time: 0.0046 (0.004654722922294393)  --> TIME: 2025-02-24 03:09:11 -- STEP: 29486/36938 -- GLOBAL_STEP: 140300 | > loss_text_ce: 0.056365158408880234 (0.05243031797597818) | > loss_mel_ce: 3.946448564529419 (nan) | > loss: 0.01588418334722519 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.1363929381228566) | > loader_time: 0.0046 (0.004654686373172606)  --> TIME: 2025-02-24 03:09:20 -- STEP: 29536/36938 -- GLOBAL_STEP: 140350 | > loss_text_ce: 0.06188386678695679 (0.05242695686336631) | > loss_mel_ce: 3.9173970222473145 (nan) | > loss: 0.015790797770023346 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.13638528932858748) | > loader_time: 0.0048 (0.004654589535936863)  --> TIME: 2025-02-24 03:09:30 -- STEP: 29586/36938 -- GLOBAL_STEP: 140400 | > loss_text_ce: 0.0356270931661129 (0.0524256621273511) | > loss_mel_ce: 3.890625 (nan) | > loss: 0.015580366365611553 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13638264338497882) | > loader_time: 0.005 (0.004654667186163268)  --> TIME: 2025-02-24 03:09:39 -- STEP: 29636/36938 -- GLOBAL_STEP: 140450 | > loss_text_ce: 0.03579041734337807 (0.05242239889825303) | > loss_mel_ce: 3.5448944568634033 (nan) | > loss: 0.014209067448973656 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13637848709868283) | > loader_time: 0.0045 (0.004654791516352772)  --> TIME: 2025-02-24 03:09:48 -- STEP: 29686/36938 -- GLOBAL_STEP: 140500 | > loss_text_ce: 0.06033388897776604 (0.052421053774477436) | > loss_mel_ce: 4.009430408477783 (nan) | > loss: 0.01614985801279545 (nan) | > current_lr: 5e-06 | > step_time: 0.1642 (0.1363739233441674) | > loader_time: 0.0047 (0.00465475874416814)  --> TIME: 2025-02-24 03:09:57 -- STEP: 29736/36938 -- GLOBAL_STEP: 140550 | > loss_text_ce: 0.0492282509803772 (0.05241958814714344) | > loss_mel_ce: 4.163272380828857 (nan) | > loss: 0.01671627350151539 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.1363769009577073) | > loader_time: 0.0048 (0.004654871173080692)  --> TIME: 2025-02-24 03:10:07 -- STEP: 29786/36938 -- GLOBAL_STEP: 140600 | > loss_text_ce: 0.06691300868988037 (0.05241401801014619) | > loss_mel_ce: 2.4929616451263428 (nan) | > loss: 0.010158232413232327 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13637932505287856) | > loader_time: 0.0046 (0.004655022726172274)  --> TIME: 2025-02-24 03:10:16 -- STEP: 29836/36938 -- GLOBAL_STEP: 140650 | > loss_text_ce: 0.056052468717098236 (0.052413430440898) | > loss_mel_ce: 3.7251968383789062 (nan) | > loss: 0.015004958026111126 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13637539899801607) | > loader_time: 0.0045 (0.004655087756509647)  --> TIME: 2025-02-24 03:10:25 -- STEP: 29886/36938 -- GLOBAL_STEP: 140700 | > loss_text_ce: 0.059451874345541 (0.05240902929117875) | > loss_mel_ce: 4.18326473236084 (nan) | > loss: 0.016836179420351982 (nan) | > current_lr: 5e-06 | > step_time: 0.1825 (0.13637564298435478) | > loader_time: 0.0048 (0.0046550633078764215)  --> TIME: 2025-02-24 03:10:34 -- STEP: 29936/36938 -- GLOBAL_STEP: 140750 | > loss_text_ce: 0.044316500425338745 (0.05240536247654971) | > loss_mel_ce: 1.7303577661514282 (nan) | > loss: 0.0070423586294054985 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13637318576420895) | > loader_time: 0.0045 (0.004655001938884591)  --> TIME: 2025-02-24 03:10:44 -- STEP: 29986/36938 -- GLOBAL_STEP: 140800 | > loss_text_ce: 0.047834161669015884 (0.05240345073826347) | > loss_mel_ce: 3.5947132110595703 (nan) | > loss: 0.014454553835093975 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.136370928659773) | > loader_time: 0.0045 (0.0046551762194580095)  --> TIME: 2025-02-24 03:10:53 -- STEP: 30036/36938 -- GLOBAL_STEP: 140850 | > loss_text_ce: 0.03755265101790428 (0.052401883636439675) | > loss_mel_ce: 1.2922779321670532 (nan) | > loss: 0.0052771056070923805 (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.13637164233130727) | > loader_time: 0.0046 (0.004655112810943877)  --> TIME: 2025-02-24 03:11:02 -- STEP: 30086/36938 -- GLOBAL_STEP: 140900 | > loss_text_ce: 0.039784371852874756 (0.052398712590307345) | > loss_mel_ce: 3.509127140045166 (nan) | > loss: 0.014082983136177063 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.1363721070736252) | > loader_time: 0.0044 (0.004655013088849352)  --> TIME: 2025-02-24 03:11:12 -- STEP: 30136/36938 -- GLOBAL_STEP: 140950 | > loss_text_ce: 0.04270809143781662 (0.05239589866424435) | > loss_mel_ce: 3.629666328430176 (nan) | > loss: 0.014572915621101856 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.13637957828159564) | > loader_time: 0.0048 (0.004654913484053335)  --> TIME: 2025-02-24 03:11:21 -- STEP: 30186/36938 -- GLOBAL_STEP: 141000 | > loss_text_ce: 0.04717927798628807 (0.05239421828872512) | > loss_mel_ce: 4.023443222045898 (nan) | > loss: 0.01615326479077339 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13636867845065664) | > loader_time: 0.0047 (0.004655006391062337)  --> TIME: 2025-02-24 03:11:30 -- STEP: 30236/36938 -- GLOBAL_STEP: 141050 | > loss_text_ce: 0.040133461356163025 (0.05239138818609136) | > loss_mel_ce: 3.9460859298706055 (nan) | > loss: 0.015818331390619278 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.1363775682900389) | > loader_time: 0.0046 (0.004654986807271439)  --> TIME: 2025-02-24 03:11:40 -- STEP: 30286/36938 -- GLOBAL_STEP: 141100 | > loss_text_ce: 0.03769524767994881 (0.052386499553303094) | > loss_mel_ce: 3.46250581741333 (nan) | > loss: 0.01388968713581562 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13637854954860035) | > loader_time: 0.0045 (0.004654906050009374)  --> TIME: 2025-02-24 03:11:49 -- STEP: 30336/36938 -- GLOBAL_STEP: 141150 | > loss_text_ce: 0.05961240455508232 (0.052381891893589626) | > loss_mel_ce: 3.549708366394043 (nan) | > loss: 0.014322702772915363 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1363742747717403) | > loader_time: 0.0045 (0.004654810484891675)  --> TIME: 2025-02-24 03:11:58 -- STEP: 30386/36938 -- GLOBAL_STEP: 141200 | > loss_text_ce: 0.055632248520851135 (0.052376494739098736) | > loss_mel_ce: 2.9362947940826416 (nan) | > loss: 0.011872727423906326 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13637657251209379) | > loader_time: 0.0065 (0.004654866299666533)  --> TIME: 2025-02-24 03:12:07 -- STEP: 30436/36938 -- GLOBAL_STEP: 141250 | > loss_text_ce: 0.05382464826107025 (0.052373915890923044) | > loss_mel_ce: 2.735405445098877 (nan) | > loss: 0.011068373918533325 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13637653201963498) | > loader_time: 0.0046 (0.004654863955768062)  --> TIME: 2025-02-24 03:12:17 -- STEP: 30486/36938 -- GLOBAL_STEP: 141300 | > loss_text_ce: 0.053360018879175186 (0.05236976861395664) | > loss_mel_ce: 3.4320452213287354 (nan) | > loss: 0.013830973766744137 (nan) | > current_lr: 5e-06 | > step_time: 0.1766 (0.1363729503080783) | > loader_time: 0.0048 (0.00465523988596996)  --> TIME: 2025-02-24 03:12:26 -- STEP: 30536/36938 -- GLOBAL_STEP: 141350 | > loss_text_ce: 0.038422632962465286 (0.052367539058587306) | > loss_mel_ce: 0.154541015625 (nan) | > loss: 0.0007657288224436343 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13637565110315153) | > loader_time: 0.0046 (0.004655346348549864)  --> TIME: 2025-02-24 03:12:35 -- STEP: 30586/36938 -- GLOBAL_STEP: 141400 | > loss_text_ce: 0.06500482559204102 (0.0523658182112019) | > loss_mel_ce: 4.112112045288086 (nan) | > loss: 0.01657586172223091 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13637192880850987) | > loader_time: 0.0046 (0.004655323969891426)  --> TIME: 2025-02-24 03:12:45 -- STEP: 30636/36938 -- GLOBAL_STEP: 141450 | > loss_text_ce: 0.050766780972480774 (0.05236136250749095) | > loss_mel_ce: 3.792531967163086 (nan) | > loss: 0.01525118574500084 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.13637602274778154) | > loader_time: 0.005 (0.004655343696488621)  --> TIME: 2025-02-24 03:12:54 -- STEP: 30686/36938 -- GLOBAL_STEP: 141500 | > loss_text_ce: 0.055870164185762405 (0.05235806928050664) | > loss_mel_ce: 2.929816484451294 (nan) | > loss: 0.0118479635566473 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13637324451102795) | > loader_time: 0.0063 (0.004655415034547391)  --> TIME: 2025-02-24 03:13:03 -- STEP: 30736/36938 -- GLOBAL_STEP: 141550 | > loss_text_ce: 0.03402578458189964 (0.052354051503514035) | > loss_mel_ce: 2.8772406578063965 (nan) | > loss: 0.011552644893527031 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13637268607626465) | > loader_time: 0.0046 (0.004655354388178392)  --> TIME: 2025-02-24 03:13:12 -- STEP: 30786/36938 -- GLOBAL_STEP: 141600 | > loss_text_ce: 0.041928499937057495 (0.05234809462112351) | > loss_mel_ce: 3.169081449508667 (nan) | > loss: 0.012742104008793831 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13637594756021998) | > loader_time: 0.0047 (0.004655514653721993)  --> TIME: 2025-02-24 03:13:22 -- STEP: 30836/36938 -- GLOBAL_STEP: 141650 | > loss_text_ce: 0.04758405685424805 (0.0523435732634925) | > loss_mel_ce: 3.643242835998535 (nan) | > loss: 0.014646138995885849 (nan) | > current_lr: 5e-06 | > step_time: 0.1511 (0.13638592735155303) | > loader_time: 0.0047 (0.004655664324961437)  --> TIME: 2025-02-24 03:13:32 -- STEP: 30886/36938 -- GLOBAL_STEP: 141700 | > loss_text_ce: 0.04288026690483093 (0.05233611979462169) | > loss_mel_ce: 2.7051899433135986 (nan) | > loss: 0.010905041359364986 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1363900856967738) | > loader_time: 0.0046 (0.004655756072229795)  --> TIME: 2025-02-24 03:13:41 -- STEP: 30936/36938 -- GLOBAL_STEP: 141750 | > loss_text_ce: 0.0637359544634819 (0.052333123058066) | > loss_mel_ce: 3.06854248046875 (nan) | > loss: 0.012429676949977875 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13639841319086252) | > loader_time: 0.0045 (0.004656046120301036)  --> TIME: 2025-02-24 03:13:50 -- STEP: 30986/36938 -- GLOBAL_STEP: 141800 | > loss_text_ce: 0.05126224830746651 (0.05232855679609428) | > loss_mel_ce: 3.457601308822632 (nan) | > loss: 0.013924062252044678 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.13639345330957603) | > loader_time: 0.0048 (0.004656131207685174)  --> TIME: 2025-02-24 03:14:00 -- STEP: 31036/36938 -- GLOBAL_STEP: 141850 | > loss_text_ce: 0.03776323050260544 (0.05232382499104417) | > loss_mel_ce: 3.1023852825164795 (nan) | > loss: 0.012460906989872456 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13639294425918005) | > loader_time: 0.0045 (0.004656123775445769)  --> TIME: 2025-02-24 03:14:09 -- STEP: 31086/36938 -- GLOBAL_STEP: 141900 | > loss_text_ce: 0.043470341712236404 (0.05232090212570466) | > loss_mel_ce: 2.596890926361084 (nan) | > loss: 0.010477624833583832 (nan) | > current_lr: 5e-06 | > step_time: 0.156 (0.1363897691367879) | > loader_time: 0.0048 (0.00465607743599786)  --> TIME: 2025-02-24 03:14:18 -- STEP: 31136/36938 -- GLOBAL_STEP: 141950 | > loss_text_ce: 0.03854598477482796 (0.05231831156234603) | > loss_mel_ce: 2.8839571475982666 (nan) | > loss: 0.011597235687077045 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13639137930670464) | > loader_time: 0.0047 (0.004656190954278917)  --> TIME: 2025-02-24 03:14:27 -- STEP: 31186/36938 -- GLOBAL_STEP: 142000 | > loss_text_ce: 0.06811396777629852 (0.05231637717781613) | > loss_mel_ce: 3.9583582878112793 (nan) | > loss: 0.015978064388036728 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.136387657264486) | > loader_time: 0.0045 (0.004656245012307152)  --> TIME: 2025-02-24 03:14:36 -- STEP: 31236/36938 -- GLOBAL_STEP: 142050 | > loss_text_ce: 0.0530310720205307 (0.052311518118242244) | > loss_mel_ce: 3.6945672035217285 (nan) | > loss: 0.014871422201395035 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13638347315565502) | > loader_time: 0.0046 (0.0046563869265163925)  --> TIME: 2025-02-24 03:14:46 -- STEP: 31286/36938 -- GLOBAL_STEP: 142100 | > loss_text_ce: 0.0477541945874691 (0.05230862905428468) | > loss_mel_ce: 2.8399498462677 (nan) | > loss: 0.011459143832325935 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13638926198596477) | > loader_time: 0.0045 (0.0046564148704330045)  --> TIME: 2025-02-24 03:14:55 -- STEP: 31336/36938 -- GLOBAL_STEP: 142150 | > loss_text_ce: 0.06674488633871078 (0.052303458096449776) | > loss_mel_ce: 3.3836801052093506 (nan) | > loss: 0.013692162930965424 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13638015049953245) | > loader_time: 0.0045 (0.004656404226386828)  --> TIME: 2025-02-24 03:15:04 -- STEP: 31386/36938 -- GLOBAL_STEP: 142200 | > loss_text_ce: 0.04754730686545372 (0.05230030555467142) | > loss_mel_ce: 3.410104751586914 (nan) | > loss: 0.013720842078328133 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13638017629259994) | > loader_time: 0.0046 (0.004656403552260555)  --> TIME: 2025-02-24 03:15:13 -- STEP: 31436/36938 -- GLOBAL_STEP: 142250 | > loss_text_ce: 0.05070345103740692 (0.05229562019439515) | > loss_mel_ce: 3.6659939289093018 (nan) | > loss: 0.014748800545930862 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13637715202658807) | > loader_time: 0.0046 (0.004656328691115318)  --> TIME: 2025-02-24 03:15:23 -- STEP: 31486/36938 -- GLOBAL_STEP: 142300 | > loss_text_ce: 0.0794629156589508 (0.05229514007744287) | > loss_mel_ce: 1.4789352416992188 (nan) | > loss: 0.0061841197311878204 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13638313579292893) | > loader_time: 0.0045 (0.0046563586853725905)  --> TIME: 2025-02-24 03:15:32 -- STEP: 31536/36938 -- GLOBAL_STEP: 142350 | > loss_text_ce: 0.05086245760321617 (0.05228889181265299) | > loss_mel_ce: 3.534766912460327 (nan) | > loss: 0.01422868948429823 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13637890271702727) | > loader_time: 0.0048 (0.004656312649834257)  --> TIME: 2025-02-24 03:15:41 -- STEP: 31586/36938 -- GLOBAL_STEP: 142400 | > loss_text_ce: 0.04867248982191086 (0.052286796018551795) | > loss_mel_ce: 3.8734052181243896 (nan) | > loss: 0.015563800930976868 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13637769470596592) | > loader_time: 0.0046 (0.004656229849169032)  --> TIME: 2025-02-24 03:15:51 -- STEP: 31636/36938 -- GLOBAL_STEP: 142450 | > loss_text_ce: 0.046079277992248535 (0.052285970992088766) | > loss_mel_ce: 3.4962143898010254 (nan) | > loss: 0.014056721702218056 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13638114986475255) | > loader_time: 0.0046 (0.004656307607463348)  --> TIME: 2025-02-24 03:16:00 -- STEP: 31686/36938 -- GLOBAL_STEP: 142500 | > loss_text_ce: 0.045788057148456573 (0.052281381617815356) | > loss_mel_ce: 3.582711935043335 (nan) | > loss: 0.01439881045371294 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.13637819662920708) | > loader_time: 0.0048 (0.004656334751928277)  --> TIME: 2025-02-24 03:16:09 -- STEP: 31736/36938 -- GLOBAL_STEP: 142550 | > loss_text_ce: 0.05329738184809685 (0.05227727302914604) | > loss_mel_ce: 4.135270595550537 (nan) | > loss: 0.01662130281329155 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13637628888404835) | > loader_time: 0.0047 (0.004656338762329456)  --> TIME: 2025-02-24 03:16:18 -- STEP: 31786/36938 -- GLOBAL_STEP: 142600 | > loss_text_ce: 0.05492164567112923 (0.05227784825966395) | > loss_mel_ce: 2.6030962467193604 (nan) | > loss: 0.010547691024839878 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13637181945679197) | > loader_time: 0.0044 (0.0046563553163555425)  --> TIME: 2025-02-24 03:16:28 -- STEP: 31836/36938 -- GLOBAL_STEP: 142650 | > loss_text_ce: 0.0452306792140007 (0.05227273187541804) | > loss_mel_ce: 4.1704535484313965 (nan) | > loss: 0.016728907823562622 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.13637459054186848) | > loader_time: 0.0046 (0.0046564357291814604)  --> TIME: 2025-02-24 03:16:37 -- STEP: 31886/36938 -- GLOBAL_STEP: 142700 | > loss_text_ce: 0.05128747224807739 (0.052267481836527206) | > loss_mel_ce: 2.817901611328125 (nan) | > loss: 0.01138567179441452 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.13637076875971127) | > loader_time: 0.0045 (0.004656437214529115)  --> TIME: 2025-02-24 03:16:46 -- STEP: 31936/36938 -- GLOBAL_STEP: 142750 | > loss_text_ce: 0.047696102410554886 (0.052264742295411504) | > loss_mel_ce: 4.034658432006836 (nan) | > loss: 0.01619981974363327 (nan) | > current_lr: 5e-06 | > step_time: 0.1495 (0.13637436759137547) | > loader_time: 0.0046 (0.004656452752784137)  --> TIME: 2025-02-24 03:16:55 -- STEP: 31986/36938 -- GLOBAL_STEP: 142800 | > loss_text_ce: 0.05224204063415527 (0.05226030304284559) | > loss_mel_ce: 4.020432472229004 (nan) | > loss: 0.01616140827536583 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13637179095652915) | > loader_time: 0.0046 (0.0046563629271321495)  --> TIME: 2025-02-24 03:17:05 -- STEP: 32036/36938 -- GLOBAL_STEP: 142850 | > loss_text_ce: 0.04628686234354973 (0.05225680519853224) | > loss_mel_ce: 3.4073047637939453 (nan) | > loss: 0.01370472926646471 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13637243947489924) | > loader_time: 0.0052 (0.004656255877796404)  --> TIME: 2025-02-24 03:17:14 -- STEP: 32086/36938 -- GLOBAL_STEP: 142900 | > loss_text_ce: 0.04177059233188629 (0.05225402246976651) | > loss_mel_ce: 3.996894121170044 (nan) | > loss: 0.01602644845843315 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.13636772656228327) | > loader_time: 0.0048 (0.004656243768630287)  --> TIME: 2025-02-24 03:17:23 -- STEP: 32136/36938 -- GLOBAL_STEP: 142950 | > loss_text_ce: 0.05704715475440025 (0.05225111430519342) | > loss_mel_ce: 3.043381929397583 (nan) | > loss: 0.012303290888667107 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.1363637397946318) | > loader_time: 0.0044 (0.004656313803767491)  --> TIME: 2025-02-24 03:17:32 -- STEP: 32186/36938 -- GLOBAL_STEP: 143000 | > loss_text_ce: 0.051110848784446716 (0.05224723862516507) | > loss_mel_ce: 2.4139606952667236 (nan) | > loss: 0.009782030247151852 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.13636202859010485) | > loader_time: 0.0049 (0.004656457052098659)  --> TIME: 2025-02-24 03:17:41 -- STEP: 32236/36938 -- GLOBAL_STEP: 143050 | > loss_text_ce: 0.0570208914577961 (0.05224221109545278) | > loss_mel_ce: 3.279086112976074 (nan) | > loss: 0.013238520361483097 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13636128370512585) | > loader_time: 0.0047 (0.0046566266149101)  --> TIME: 2025-02-24 03:17:50 -- STEP: 32286/36938 -- GLOBAL_STEP: 143100 | > loss_text_ce: 0.048711422830820084 (0.05224001266788373) | > loss_mel_ce: 3.7643821239471436 (nan) | > loss: 0.015131324529647827 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1363536803273262) | > loader_time: 0.0045 (0.004656691692401658)  --> TIME: 2025-02-24 03:18:00 -- STEP: 32336/36938 -- GLOBAL_STEP: 143150 | > loss_text_ce: 0.03861406818032265 (0.052235516838786765) | > loss_mel_ce: 3.497136354446411 (nan) | > loss: 0.014030756428837776 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.13636332173739243) | > loader_time: 0.0046 (0.004656859645449459)  --> TIME: 2025-02-24 03:18:09 -- STEP: 32386/36938 -- GLOBAL_STEP: 143200 | > loss_text_ce: 0.052771762013435364 (0.052233254440276745) | > loss_mel_ce: 3.0396556854248047 (nan) | > loss: 0.012271538376808167 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13635672469773202) | > loader_time: 0.0045 (0.004656870384430658)  --> TIME: 2025-02-24 03:18:18 -- STEP: 32436/36938 -- GLOBAL_STEP: 143250 | > loss_text_ce: 0.06075231730937958 (0.052228342422127656) | > loss_mel_ce: 3.575181484222412 (nan) | > loss: 0.014428310096263885 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13634989825846386) | > loader_time: 0.0048 (0.004656769349051961)  --> TIME: 2025-02-24 03:18:27 -- STEP: 32486/36938 -- GLOBAL_STEP: 143300 | > loss_text_ce: 0.05844949930906296 (0.052229181716212526) | > loss_mel_ce: 3.8542673587799072 (nan) | > loss: 0.01552665513008833 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13634659403218816) | > loader_time: 0.0045 (0.0046568836021223465)  --> TIME: 2025-02-24 03:18:37 -- STEP: 32536/36938 -- GLOBAL_STEP: 143350 | > loss_text_ce: 0.05797208100557327 (0.05222713250055098) | > loss_mel_ce: 4.256096839904785 (nan) | > loss: 0.01711932197213173 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13634804533100775) | > loader_time: 0.0046 (0.004656974318750289)  --> TIME: 2025-02-24 03:18:46 -- STEP: 32586/36938 -- GLOBAL_STEP: 143400 | > loss_text_ce: 0.06332208961248398 (0.052225954351095144) | > loss_mel_ce: 3.2087578773498535 (nan) | > loss: 0.012984445318579674 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13635130518452898) | > loader_time: 0.0045 (0.004657111480765311)  --> TIME: 2025-02-24 03:18:55 -- STEP: 32636/36938 -- GLOBAL_STEP: 143450 | > loss_text_ce: 0.05595940351486206 (0.052223956912034804) | > loss_mel_ce: 3.9958438873291016 (nan) | > loss: 0.016078583896160126 (nan) | > current_lr: 5e-06 | > step_time: 0.1864 (0.13634912898865253) | > loader_time: 0.0048 (0.004657154808530655)  --> TIME: 2025-02-24 03:19:04 -- STEP: 32686/36938 -- GLOBAL_STEP: 143500 | > loss_text_ce: 0.04747314751148224 (0.052223205385219205) | > loss_mel_ce: 3.747152090072632 (nan) | > loss: 0.01505803782492876 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13634448714350633) | > loader_time: 0.0045 (0.0046572066838494654)  --> TIME: 2025-02-24 03:19:14 -- STEP: 32736/36938 -- GLOBAL_STEP: 143550 | > loss_text_ce: 0.05391787737607956 (0.05222048308448945) | > loss_mel_ce: 3.576097249984741 (nan) | > loss: 0.014404823072254658 (nan) | > current_lr: 5e-06 | > step_time: 0.1511 (0.13634519428306183) | > loader_time: 0.0046 (0.004657285377193847)  --> TIME: 2025-02-24 03:19:23 -- STEP: 32786/36938 -- GLOBAL_STEP: 143600 | > loss_text_ce: 0.04346858710050583 (0.052219402566746474) | > loss_mel_ce: 4.046897888183594 (nan) | > loss: 0.016231613233685493 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.1363476233900364) | > loader_time: 0.0047 (0.004657294834132051)  --> TIME: 2025-02-24 03:19:33 -- STEP: 32836/36938 -- GLOBAL_STEP: 143650 | > loss_text_ce: 0.049601197242736816 (0.052216783181559645) | > loss_mel_ce: 3.8730523586273193 (nan) | > loss: 0.015566087327897549 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.1363518510060571) | > loader_time: 0.0046 (0.0046573507029212145)  --> TIME: 2025-02-24 03:19:42 -- STEP: 32886/36938 -- GLOBAL_STEP: 143700 | > loss_text_ce: 0.050660938024520874 (0.052215871658925654) | > loss_mel_ce: 3.5556094646453857 (nan) | > loss: 0.014310597442090511 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.1363491954532623) | > loader_time: 0.0048 (0.004657470120758995)  --> TIME: 2025-02-24 03:19:51 -- STEP: 32936/36938 -- GLOBAL_STEP: 143750 | > loss_text_ce: 0.041744690388441086 (0.05221346751021602) | > loss_mel_ce: 3.2511913776397705 (nan) | > loss: 0.013067207299172878 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13634737710771025) | > loader_time: 0.0045 (0.004657506949987843)  --> TIME: 2025-02-24 03:20:00 -- STEP: 32986/36938 -- GLOBAL_STEP: 143800 | > loss_text_ce: 0.07905831187963486 (0.05221035294012837) | > loss_mel_ce: 1.8215745687484741 (nan) | > loss: 0.0075421943329274654 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13634660885389485) | > loader_time: 0.0045 (0.004657794228969321)  --> TIME: 2025-02-24 03:20:09 -- STEP: 33036/36938 -- GLOBAL_STEP: 143850 | > loss_text_ce: 0.061475906521081924 (0.05220640444657745) | > loss_mel_ce: 3.4646968841552734 (nan) | > loss: 0.013992750085890293 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13634684603588784) | > loader_time: 0.0046 (0.004657833559459084)  --> TIME: 2025-02-24 03:20:19 -- STEP: 33086/36938 -- GLOBAL_STEP: 143900 | > loss_text_ce: 0.056875936686992645 (0.052203773632045515) | > loss_mel_ce: 3.9250190258026123 (nan) | > loss: 0.015801170840859413 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13634774552747592) | > loader_time: 0.0045 (0.004657878708840833)  --> TIME: 2025-02-24 03:20:28 -- STEP: 33136/36938 -- GLOBAL_STEP: 143950 | > loss_text_ce: 0.049488749355077744 (0.05220042742252644) | > loss_mel_ce: 2.789271116256714 (nan) | > loss: 0.01126492116600275 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.1363356980449746) | > loader_time: 0.0046 (0.004657766824468535)  --> TIME: 2025-02-24 03:20:37 -- STEP: 33186/36938 -- GLOBAL_STEP: 144000 | > loss_text_ce: 0.040387678891420364 (0.052198063610105365) | > loss_mel_ce: 1.7350952625274658 (nan) | > loss: 0.007045567501336336 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13633854384305452) | > loader_time: 0.0045 (0.004657776404740392)  --> TIME: 2025-02-24 03:20:46 -- STEP: 33236/36938 -- GLOBAL_STEP: 144050 | > loss_text_ce: 0.049214452505111694 (0.05219350848215252) | > loss_mel_ce: 3.8631179332733154 (nan) | > loss: 0.015525128692388535 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.1363314169955933) | > loader_time: 0.0046 (0.0046578188036621015)  --> TIME: 2025-02-24 03:20:55 -- STEP: 33286/36938 -- GLOBAL_STEP: 144100 | > loss_text_ce: 0.05979951098561287 (0.05218946631996821) | > loss_mel_ce: 2.934981346130371 (nan) | > loss: 0.011884051375091076 (nan) | > current_lr: 5e-06 | > step_time: 0.1508 (0.13633075063707037) | > loader_time: 0.0046 (0.004657782886862436)  --> TIME: 2025-02-24 03:21:04 -- STEP: 33336/36938 -- GLOBAL_STEP: 144150 | > loss_text_ce: 0.07629874348640442 (0.05218514474290413) | > loss_mel_ce: 3.3023719787597656 (nan) | > loss: 0.013407424092292786 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13633025717414987) | > loader_time: 0.0047 (0.004657930626353089)  --> TIME: 2025-02-24 03:21:14 -- STEP: 33386/36938 -- GLOBAL_STEP: 144200 | > loss_text_ce: 0.058503832668066025 (0.052183355006228074) | > loss_mel_ce: 1.39404296875 (nan) | > loss: 0.005764075089246035 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1363327989168063) | > loader_time: 0.0052 (0.004658077337739969)  --> TIME: 2025-02-24 03:21:23 -- STEP: 33436/36938 -- GLOBAL_STEP: 144250 | > loss_text_ce: 0.048072654753923416 (0.052180124442878055) | > loss_mel_ce: 4.0121965408325195 (nan) | > loss: 0.016112180426716805 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.13633962057754712) | > loader_time: 0.0047 (0.004658073910629769)  --> TIME: 2025-02-24 03:21:33 -- STEP: 33486/36938 -- GLOBAL_STEP: 144300 | > loss_text_ce: 0.05376512557268143 (0.05217888468338263) | > loss_mel_ce: 4.184876918792725 (nan) | > loss: 0.016820009797811508 (nan) | > current_lr: 5e-06 | > step_time: 0.1493 (0.13633658763777617) | > loader_time: 0.0046 (0.004658272757237128)  --> TIME: 2025-02-24 03:21:42 -- STEP: 33536/36938 -- GLOBAL_STEP: 144350 | > loss_text_ce: 0.05472347512841225 (0.052176516734510014) | > loss_mel_ce: 3.663254737854004 (nan) | > loss: 0.014753882773220539 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13633470990844465) | > loader_time: 0.0047 (0.004658319532359347)  --> TIME: 2025-02-24 03:21:51 -- STEP: 33586/36938 -- GLOBAL_STEP: 144400 | > loss_text_ce: 0.04495488479733467 (0.05217064313031175) | > loss_mel_ce: 3.957629680633545 (nan) | > loss: 0.015883272513747215 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.13634086566691866) | > loader_time: 0.0047 (0.0046583234337437465)  --> TIME: 2025-02-24 03:22:00 -- STEP: 33636/36938 -- GLOBAL_STEP: 144450 | > loss_text_ce: 0.05397067964076996 (0.05216451453858277) | > loss_mel_ce: 3.8015177249908447 (nan) | > loss: 0.015299557708203793 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13633578753468653) | > loader_time: 0.0047 (0.004658301713875182)  --> TIME: 2025-02-24 03:22:09 -- STEP: 33686/36938 -- GLOBAL_STEP: 144500 | > loss_text_ce: 0.05065478757023811 (0.05216168451647635) | > loss_mel_ce: 4.000033855438232 (nan) | > loss: 0.01607416197657585 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.1363319413562739) | > loader_time: 0.0046 (0.004658334535352878)  --> TIME: 2025-02-24 03:22:19 -- STEP: 33736/36938 -- GLOBAL_STEP: 144550 | > loss_text_ce: 0.05079566314816475 (0.0521594503174093) | > loss_mel_ce: 3.356764078140259 (nan) | > loss: 0.013522062450647354 (nan) | > current_lr: 5e-06 | > step_time: 0.1828 (0.1363406039002339) | > loader_time: 0.0048 (0.004658447408495201)  --> TIME: 2025-02-24 03:22:28 -- STEP: 33786/36938 -- GLOBAL_STEP: 144600 | > loss_text_ce: 0.054493680596351624 (0.052154791394943) | > loss_mel_ce: 3.0645618438720703 (nan) | > loss: 0.012377205304801464 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13634280612821242) | > loader_time: 0.0048 (0.004658425114671765)  --> TIME: 2025-02-24 03:22:38 -- STEP: 33836/36938 -- GLOBAL_STEP: 144650 | > loss_text_ce: 0.050295084714889526 (0.05215179040606464) | > loss_mel_ce: 3.1665425300598145 (nan) | > loss: 0.012765229679644108 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13634176949178595) | > loader_time: 0.0045 (0.004658504367537021)  --> TIME: 2025-02-24 03:22:48 -- STEP: 33886/36938 -- GLOBAL_STEP: 144700 | > loss_text_ce: 0.043142180889844894 (0.05215044388963545) | > loss_mel_ce: 3.6960790157318115 (nan) | > loss: 0.01483817957341671 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13635568553199914) | > loader_time: 0.0044 (0.004658527444063701)  --> TIME: 2025-02-24 03:22:57 -- STEP: 33936/36938 -- GLOBAL_STEP: 144750 | > loss_text_ce: 0.04462583735585213 (0.05214632985248308) | > loss_mel_ce: 3.436244249343872 (nan) | > loss: 0.013812976889312267 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13636296837187215) | > loader_time: 0.0046 (0.004658468401358946)  --> TIME: 2025-02-24 03:23:06 -- STEP: 33986/36938 -- GLOBAL_STEP: 144800 | > loss_text_ce: 0.06416093558073044 (0.052144290317796625) | > loss_mel_ce: 3.531851053237915 (nan) | > loss: 0.014269889332354069 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.13636381462113883) | > loader_time: 0.0044 (0.004658355318916981)  --> TIME: 2025-02-24 03:23:16 -- STEP: 34036/36938 -- GLOBAL_STEP: 144850 | > loss_text_ce: 0.05095922201871872 (0.052141851867338065) | > loss_mel_ce: 3.9225006103515625 (nan) | > loss: 0.015767699107527733 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.13636794126976745) | > loader_time: 0.0048 (0.004658315272515581)  --> TIME: 2025-02-24 03:23:25 -- STEP: 34086/36938 -- GLOBAL_STEP: 144900 | > loss_text_ce: 0.048350173979997635 (0.05213878383469393) | > loss_mel_ce: 4.185494899749756 (nan) | > loss: 0.016800973564386368 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.1363653581354932) | > loader_time: 0.0047 (0.004658432575638319)  --> TIME: 2025-02-24 03:23:35 -- STEP: 34136/36938 -- GLOBAL_STEP: 144950 | > loss_text_ce: 0.03629478067159653 (0.052135672871928614) | > loss_mel_ce: 2.3053691387176514 (nan) | > loss: 0.009292317554354668 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13636892881023635) | > loader_time: 0.0045 (0.00465852639589649)  --> TIME: 2025-02-24 03:23:44 -- STEP: 34186/36938 -- GLOBAL_STEP: 145000 | > loss_text_ce: 0.0573844239115715 (0.05213223806294878) | > loss_mel_ce: 3.2732701301574707 (nan) | > loss: 0.01321688387542963 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13636641371824212) | > loader_time: 0.0052 (0.004658719414135351)  --> TIME: 2025-02-24 03:23:53 -- STEP: 34236/36938 -- GLOBAL_STEP: 145050 | > loss_text_ce: 0.05184219405055046 (0.052131329855776926) | > loss_mel_ce: 3.4633615016937256 (nan) | > loss: 0.013949221931397915 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.1363665335213761) | > loader_time: 0.0047 (0.0046588965548102655)  --> TIME: 2025-02-24 03:24:02 -- STEP: 34286/36938 -- GLOBAL_STEP: 145100 | > loss_text_ce: 0.06346926838159561 (0.05212742071110379) | > loss_mel_ce: 3.558241605758667 (nan) | > loss: 0.014371869154274464 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13636651868296798) | > loader_time: 0.0046 (0.004658850503692394)  --> TIME: 2025-02-24 03:24:12 -- STEP: 34336/36938 -- GLOBAL_STEP: 145150 | > loss_text_ce: 0.04794086515903473 (0.05212550662020825) | > loss_mel_ce: 3.9015233516693115 (nan) | > loss: 0.015672478824853897 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13637358674437647) | > loader_time: 0.0046 (0.004658828507709563)  --> TIME: 2025-02-24 03:24:21 -- STEP: 34386/36938 -- GLOBAL_STEP: 145200 | > loss_text_ce: 0.0500573106110096 (0.052120946016823995) | > loss_mel_ce: 4.033565044403076 (nan) | > loss: 0.016204852610826492 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1363725691768567) | > loader_time: 0.0045 (0.004658802568076977)  --> TIME: 2025-02-24 03:24:31 -- STEP: 34436/36938 -- GLOBAL_STEP: 145250 | > loss_text_ce: 0.04545409977436066 (0.05212124850760963) | > loss_mel_ce: 3.4143452644348145 (nan) | > loss: 0.01372936274856329 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.1363801297887811) | > loader_time: 0.0047 (0.0046588091404959195)  --> TIME: 2025-02-24 03:24:40 -- STEP: 34486/36938 -- GLOBAL_STEP: 145300 | > loss_text_ce: 0.04814460128545761 (0.052117127873063385) | > loss_mel_ce: 3.3925998210906982 (nan) | > loss: 0.013653748668730259 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13638014543266064) | > loader_time: 0.0045 (0.004658748702148941)  --> TIME: 2025-02-24 03:24:49 -- STEP: 34536/36938 -- GLOBAL_STEP: 145350 | > loss_text_ce: 0.04855778068304062 (0.05211337380805779) | > loss_mel_ce: 3.5766561031341553 (nan) | > loss: 0.014385770075023174 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13637685965602622) | > loader_time: 0.0046 (0.004658688445706452)  --> TIME: 2025-02-24 03:24:58 -- STEP: 34586/36938 -- GLOBAL_STEP: 145400 | > loss_text_ce: 0.0510726124048233 (0.05211105203621804) | > loss_mel_ce: 3.810710906982422 (nan) | > loss: 0.015324538573622704 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13637325122331279) | > loader_time: 0.0047 (0.004658651815171853)  --> TIME: 2025-02-24 03:25:08 -- STEP: 34636/36938 -- GLOBAL_STEP: 145450 | > loss_text_ce: 0.04626636952161789 (0.05210850281248424) | > loss_mel_ce: 3.822734832763672 (nan) | > loss: 0.015353180468082428 (nan) | > current_lr: 5e-06 | > step_time: 0.1578 (0.1363763547491605) | > loader_time: 0.0047 (0.0046586409797980885)  --> TIME: 2025-02-24 03:25:17 -- STEP: 34686/36938 -- GLOBAL_STEP: 145500 | > loss_text_ce: 0.05404061824083328 (0.052105619215936504) | > loss_mel_ce: 3.756517171859741 (nan) | > loss: 0.015121262520551682 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13636873063005794) | > loader_time: 0.0047 (0.004658618332405804)  --> TIME: 2025-02-24 03:25:26 -- STEP: 34736/36938 -- GLOBAL_STEP: 145550 | > loss_text_ce: 0.05248939245939255 (0.05209998992626818) | > loss_mel_ce: 3.7526636123657227 (nan) | > loss: 0.015099814161658287 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13637476416158084) | > loader_time: 0.0045 (0.00465861890844239)  --> TIME: 2025-02-24 03:25:36 -- STEP: 34786/36938 -- GLOBAL_STEP: 145600 | > loss_text_ce: 0.04694170877337456 (0.05209636436478686) | > loss_mel_ce: 3.6962521076202393 (nan) | > loss: 0.0148539450019598 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13637179029698726) | > loader_time: 0.0045 (0.004658583404072947)  --> TIME: 2025-02-24 03:25:45 -- STEP: 34836/36938 -- GLOBAL_STEP: 145650 | > loss_text_ce: 0.05517246574163437 (0.052090459019579864) | > loss_mel_ce: 3.6573827266693115 (nan) | > loss: 0.0147323627024889 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13636949455468508) | > loader_time: 0.0047 (0.004658547454099835)  --> TIME: 2025-02-24 03:25:54 -- STEP: 34886/36938 -- GLOBAL_STEP: 145700 | > loss_text_ce: 0.048668306320905685 (0.052087344263525945) | > loss_mel_ce: 4.181796073913574 (nan) | > loss: 0.016787558794021606 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13637271777879564) | > loader_time: 0.0046 (0.0046585262734114815)  --> TIME: 2025-02-24 03:26:04 -- STEP: 34936/36938 -- GLOBAL_STEP: 145750 | > loss_text_ce: 0.06071234494447708 (0.05208473261105074) | > loss_mel_ce: 3.0119471549987793 (nan) | > loss: 0.01219309400767088 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13637950630845416) | > loader_time: 0.0046 (0.004658676337562462)  --> TIME: 2025-02-24 03:26:13 -- STEP: 34986/36938 -- GLOBAL_STEP: 145800 | > loss_text_ce: 0.04658181592822075 (0.052081067206560944) | > loss_mel_ce: 3.544682264328003 (nan) | > loss: 0.014251048676669598 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13638461222201945) | > loader_time: 0.0046 (0.004658773758668561)  --> TIME: 2025-02-24 03:26:22 -- STEP: 35036/36938 -- GLOBAL_STEP: 145850 | > loss_text_ce: 0.060134079307317734 (0.05207806482500372) | > loss_mel_ce: 3.6765246391296387 (nan) | > loss: 0.014828012324869633 (nan) | > current_lr: 5e-06 | > step_time: 0.1903 (0.13637767403293796) | > loader_time: 0.0048 (0.004658709215864682)  --> TIME: 2025-02-24 03:26:31 -- STEP: 35086/36938 -- GLOBAL_STEP: 145900 | > loss_text_ce: 0.058633316308259964 (0.05207288868406034) | > loss_mel_ce: 3.3431296348571777 (nan) | > loss: 0.013499059714376926 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.1363737838186609) | > loader_time: 0.0045 (0.004658728479516789)  --> TIME: 2025-02-24 03:26:41 -- STEP: 35136/36938 -- GLOBAL_STEP: 145950 | > loss_text_ce: 0.0484425351023674 (0.052069876413703996) | > loss_mel_ce: 3.3942902088165283 (nan) | > loss: 0.013661638833582401 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13637744783269556) | > loader_time: 0.0048 (0.004658786929424527)  --> TIME: 2025-02-24 03:26:50 -- STEP: 35186/36938 -- GLOBAL_STEP: 146000 | > loss_text_ce: 0.05127745121717453 (0.05206386993719587) | > loss_mel_ce: 3.427401542663574 (nan) | > loss: 0.013804282061755657 (nan) | > current_lr: 5e-06 | > step_time: 0.1837 (0.136375979370834) | > loader_time: 0.0048 (0.0046587099855798075)  --> TIME: 2025-02-24 03:26:59 -- STEP: 35236/36938 -- GLOBAL_STEP: 146050 | > loss_text_ce: 0.03511509671807289 (0.052059832192414675) | > loss_mel_ce: 0.555206298828125 (nan) | > loss: 0.0023425454273819923 (nan) | > current_lr: 5e-06 | > step_time: 0.1071 (0.13637081135387125) | > loader_time: 0.0045 (0.0046586842850342555)  --> TIME: 2025-02-24 03:27:09 -- STEP: 35286/36938 -- GLOBAL_STEP: 146100 | > loss_text_ce: 0.0425708144903183 (0.05205660101097776) | > loss_mel_ce: 3.8737387657165527 (nan) | > loss: 0.015540911816060543 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.1363733614922823) | > loader_time: 0.0047 (0.004658730853166714)  --> TIME: 2025-02-24 03:27:18 -- STEP: 35336/36938 -- GLOBAL_STEP: 146150 | > loss_text_ce: 0.060067318379879 (0.05205280522844668) | > loss_mel_ce: 3.5010757446289062 (nan) | > loss: 0.014131520874798298 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13638270294636703) | > loader_time: 0.0047 (0.004658769671939058)  --> TIME: 2025-02-24 03:27:27 -- STEP: 35386/36938 -- GLOBAL_STEP: 146200 | > loss_text_ce: 0.05522812902927399 (0.05205010707210485) | > loss_mel_ce: 3.1366567611694336 (nan) | > loss: 0.01266621146351099 (nan) | > current_lr: 5e-06 | > step_time: 0.1452 (0.13637835288987118) | > loader_time: 0.0046 (0.004658741381791834)  --> TIME: 2025-02-24 03:27:37 -- STEP: 35436/36938 -- GLOBAL_STEP: 146250 | > loss_text_ce: 0.04489781707525253 (0.05204662615644118) | > loss_mel_ce: 3.884237289428711 (nan) | > loss: 0.01559180673211813 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.1363772276643043) | > loader_time: 0.0048 (0.004658682121085633)  --> TIME: 2025-02-24 03:27:46 -- STEP: 35486/36938 -- GLOBAL_STEP: 146300 | > loss_text_ce: 0.060925811529159546 (0.05204420058941271) | > loss_mel_ce: 3.2229466438293457 (nan) | > loss: 0.013031240552663803 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1363768261395275) | > loader_time: 0.0045 (0.0046587309089956555)  --> TIME: 2025-02-24 03:27:56 -- STEP: 35536/36938 -- GLOBAL_STEP: 146350 | > loss_text_ce: 0.051565323024988174 (0.05203958163274297) | > loss_mel_ce: 3.429964542388916 (nan) | > loss: 0.013815595768392086 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1363820729492924) | > loader_time: 0.0045 (0.004658764879856992)  --> TIME: 2025-02-24 03:28:04 -- STEP: 35586/36938 -- GLOBAL_STEP: 146400 | > loss_text_ce: 0.05682450532913208 (0.052036997622773806) | > loss_mel_ce: 4.024117469787598 (nan) | > loss: 0.016194215044379234 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13637162886280751) | > loader_time: 0.0048 (0.004658855080356623)  --> TIME: 2025-02-24 03:28:14 -- STEP: 35636/36938 -- GLOBAL_STEP: 146450 | > loss_text_ce: 0.06078391522169113 (0.05203284059465337) | > loss_mel_ce: 0.0913035050034523 (nan) | > loss: 0.0006035215337760746 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13637184903388666) | > loader_time: 0.0045 (0.004658763517580327)  --> TIME: 2025-02-24 03:28:23 -- STEP: 35686/36938 -- GLOBAL_STEP: 146500 | > loss_text_ce: 0.056848958134651184 (0.05202914374963856) | > loss_mel_ce: 3.4063591957092285 (nan) | > loss: 0.01374289020895958 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.1363742901312433) | > loader_time: 0.0046 (0.004658751394732215)  --> TIME: 2025-02-24 03:28:32 -- STEP: 35736/36938 -- GLOBAL_STEP: 146550 | > loss_text_ce: 0.05417848005890846 (0.05202632009687953) | > loss_mel_ce: 2.868162155151367 (nan) | > loss: 0.011596590280532837 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.1363717617472474) | > loader_time: 0.0048 (0.004658788902956461)  --> TIME: 2025-02-24 03:28:41 -- STEP: 35786/36938 -- GLOBAL_STEP: 146600 | > loss_text_ce: 0.040837034583091736 (0.05202468300319665) | > loss_mel_ce: 3.7341532707214355 (nan) | > loss: 0.01498012151569128 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13636546162797406) | > loader_time: 0.0045 (0.004658767444574527)  --> TIME: 2025-02-24 03:28:51 -- STEP: 35836/36938 -- GLOBAL_STEP: 146650 | > loss_text_ce: 0.0590408556163311 (0.052022900434817235) | > loss_mel_ce: 2.6910324096679688 (nan) | > loss: 0.010912989266216755 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.1363664776476019) | > loader_time: 0.0046 (0.004658679036586317)  --> TIME: 2025-02-24 03:29:00 -- STEP: 35886/36938 -- GLOBAL_STEP: 146700 | > loss_text_ce: 0.056796845048666 (0.05202312936836324) | > loss_mel_ce: 3.6685149669647217 (nan) | > loss: 0.014782983809709549 (nan) | > current_lr: 5e-06 | > step_time: 0.19 (0.13637504042547002) | > loader_time: 0.0048 (0.004658634205669159)  --> TIME: 2025-02-24 03:29:09 -- STEP: 35936/36938 -- GLOBAL_STEP: 146750 | > loss_text_ce: 0.050020188093185425 (0.052020590041094696) | > loss_mel_ce: 3.7617199420928955 (nan) | > loss: 0.01512595359236002 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13636697042577595) | > loader_time: 0.0045 (0.004658533596928765)  --> TIME: 2025-02-24 03:29:18 -- STEP: 35986/36938 -- GLOBAL_STEP: 146800 | > loss_text_ce: 0.05453458055853844 (0.052017469495044925) | > loss_mel_ce: 2.957170248031616 (nan) | > loss: 0.011951210908591747 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13636824115137788) | > loader_time: 0.0045 (0.004658560559939448)  --> TIME: 2025-02-24 03:29:28 -- STEP: 36036/36938 -- GLOBAL_STEP: 146850 | > loss_text_ce: 0.06031224504113197 (0.052014918793947096) | > loss_mel_ce: 3.3388671875 (nan) | > loss: 0.01348880771547556 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13637381978643207) | > loader_time: 0.0048 (0.004658596856253451)  --> TIME: 2025-02-24 03:29:37 -- STEP: 36086/36938 -- GLOBAL_STEP: 146900 | > loss_text_ce: 0.040642183274030685 (0.05200738031315453) | > loss_mel_ce: 4.795572757720947 (nan) | > loss: 0.019191330298781395 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13637060458516312) | > loader_time: 0.0046 (0.004658565931928523)  --> TIME: 2025-02-24 03:29:46 -- STEP: 36136/36938 -- GLOBAL_STEP: 146950 | > loss_text_ce: 0.03555203974246979 (0.05200432441729807) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13636860867002756) | > loader_time: 0.0045 (0.004658573446269545)  --> TIME: 2025-02-24 03:29:55 -- STEP: 36186/36938 -- GLOBAL_STEP: 147000 | > loss_text_ce: 0.04228159412741661 (0.05200179049720295) | > loss_mel_ce: 3.233914613723755 (nan) | > loss: 0.013000779785215855 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13636389938560778) | > loader_time: 0.0046 (0.004658558854533319)  --> TIME: 2025-02-24 03:30:05 -- STEP: 36236/36938 -- GLOBAL_STEP: 147050 | > loss_text_ce: 0.0526430681347847 (0.05199882632616669) | > loss_mel_ce: 2.144160270690918 (nan) | > loss: 0.008717473596334457 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13636040979037634) | > loader_time: 0.0045 (0.004658433798594026)  --> TIME: 2025-02-24 03:30:14 -- STEP: 36286/36938 -- GLOBAL_STEP: 147100 | > loss_text_ce: 0.07231427729129791 (0.05199668212245391) | > loss_mel_ce: 0.03108724020421505 (nan) | > loss: 0.0004103235260117799 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13636349488440155) | > loader_time: 0.0045 (0.004658340422194986)  --> TIME: 2025-02-24 03:30:23 -- STEP: 36336/36938 -- GLOBAL_STEP: 147150 | > loss_text_ce: 0.053339842706918716 (0.05199214295241255) | > loss_mel_ce: 3.9355945587158203 (nan) | > loss: 0.01582910679280758 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13635392553320308) | > loader_time: 0.0046 (0.0046582427031664885)  --> TIME: 2025-02-24 03:30:32 -- STEP: 36386/36938 -- GLOBAL_STEP: 147200 | > loss_text_ce: 0.06538469344377518 (0.051992376672034143) | > loss_mel_ce: 4.0311079025268555 (nan) | > loss: 0.016255924478173256 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.13636218536314976) | > loader_time: 0.0047 (0.004658232295859738)  --> TIME: 2025-02-24 03:30:41 -- STEP: 36436/36938 -- GLOBAL_STEP: 147250 | > loss_text_ce: 0.04796044901013374 (0.05199151177298015) | > loss_mel_ce: 3.9931795597076416 (nan) | > loss: 0.016036270186305046 (nan) | > current_lr: 5e-06 | > step_time: 0.1558 (0.13635607080619866) | > loader_time: 0.0046 (0.004658168509155912)  --> TIME: 2025-02-24 03:30:51 -- STEP: 36486/36938 -- GLOBAL_STEP: 147300 | > loss_text_ce: 0.04586286097764969 (0.05198749904502187) | > loss_mel_ce: 3.407073497772217 (nan) | > loss: 0.013702129013836384 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13636073711729715) | > loader_time: 0.0047 (0.0046581509264520225)  --> TIME: 2025-02-24 03:31:00 -- STEP: 36536/36938 -- GLOBAL_STEP: 147350 | > loss_text_ce: 0.05193392187356949 (0.05198443257203508) | > loss_mel_ce: 2.375110149383545 (nan) | > loss: 0.009631128050386906 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.1363611724566191) | > loader_time: 0.0046 (0.004658121091154763)  --> TIME: 2025-02-24 03:31:09 -- STEP: 36586/36938 -- GLOBAL_STEP: 147400 | > loss_text_ce: 0.038412533700466156 (0.0519793518489621) | > loss_mel_ce: 2.958587646484375 (nan) | > loss: 0.011892858892679214 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1363611976724891) | > loader_time: 0.0045 (0.004658112979238881)  --> TIME: 2025-02-24 03:31:19 -- STEP: 36636/36938 -- GLOBAL_STEP: 147450 | > loss_text_ce: 0.05644857883453369 (0.05197824481149202) | > loss_mel_ce: 3.7448630332946777 (nan) | > loss: 0.015084570273756981 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.1363629830145196) | > loader_time: 0.0048 (0.004658129514858067)  --> TIME: 2025-02-24 03:31:28 -- STEP: 36686/36938 -- GLOBAL_STEP: 147500 | > loss_text_ce: 0.03989894315600395 (0.051975625592803894) | > loss_mel_ce: 3.180692672729492 (nan) | > loss: 0.012780126184225082 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13636155025276586) | > loader_time: 0.0044 (0.0046582359306581705)  --> TIME: 2025-02-24 03:31:37 -- STEP: 36736/36938 -- GLOBAL_STEP: 147550 | > loss_text_ce: 0.0607181191444397 (0.051973289033266944) | > loss_mel_ce: 2.5256993770599365 (nan) | > loss: 0.010263562202453613 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13635566844264269) | > loader_time: 0.0052 (0.004658365265209877)  --> TIME: 2025-02-24 03:31:46 -- STEP: 36786/36938 -- GLOBAL_STEP: 147600 | > loss_text_ce: 0.0380135141313076 (0.05196761005139692) | > loss_mel_ce: 3.6701512336730957 (nan) | > loss: 0.014714940451085567 (nan) | > current_lr: 5e-06 | > step_time: 0.188 (0.13635739033551406) | > loader_time: 0.0049 (0.0046584991544674275)  --> TIME: 2025-02-24 03:31:56 -- STEP: 36836/36938 -- GLOBAL_STEP: 147650 | > loss_text_ce: 0.05467161908745766 (0.051965930244208626) | > loss_mel_ce: 3.949254035949707 (nan) | > loss: 0.015888595953583717 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.13636520898799176) | > loader_time: 0.0044 (0.004658547244122147)  --> TIME: 2025-02-24 03:32:05 -- STEP: 36886/36938 -- GLOBAL_STEP: 147700 | > loss_text_ce: 0.04204922914505005 (0.051961598736278834) | > loss_mel_ce: 3.658188581466675 (nan) | > loss: 0.014683484099805355 (nan) | > current_lr: 5e-06 | > step_time: 0.1357 (0.1363683794969756) | > loader_time: 0.0046 (0.004658661733860077)  --> TIME: 2025-02-24 03:32:15 -- STEP: 36936/36938 -- GLOBAL_STEP: 147750 | > loss_text_ce: 0.05922519043087959 (0.0519567012295638) | > loss_mel_ce: 2.7919921875 (nan) | > loss: 0.011314354836940765 (nan) | > current_lr: 5e-06 | > step_time: 0.1082 (0.13636493316222498) | > loader_time: 0.0045 (0.004658560332524834)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.020762822415569048 (-0.0006203403567323525) | > avg_loss_text_ce: 0.047880328518550605 (-0.004435838052615652) | > avg_loss_mel_ce: 3.1384985671185026 (-0.12925550666185925) | > avg_loss: 3.1863788980068546 (-0.13369134157010842) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_147752.pth  > EPOCH: 5/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-24 03:32:39)   --> TIME: 2025-02-24 03:32:48 -- STEP: 48/36938 -- GLOBAL_STEP: 147800 | > loss_text_ce: 0.031120311468839645 (0.05075840139761567) | > loss_mel_ce: 3.340609073638916 (nan) | > loss: 0.013379879295825958 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13113042215506232) | > loader_time: 0.0066 (0.004916826883951823)  --> TIME: 2025-02-24 03:32:58 -- STEP: 98/36938 -- GLOBAL_STEP: 147850 | > loss_text_ce: 0.047175660729408264 (0.05066579268599043) | > loss_mel_ce: 0.2068583220243454 (nan) | > loss: 0.0010080714710056782 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13578042935351936) | > loader_time: 0.0046 (0.004847906073745416)  --> TIME: 2025-02-24 03:33:07 -- STEP: 148/36938 -- GLOBAL_STEP: 147900 | > loss_text_ce: 0.05340677499771118 (0.05064649951669413) | > loss_mel_ce: 4.233011722564697 (nan) | > loss: 0.017009597271680832 (nan) | > current_lr: 5e-06 | > step_time: 0.1773 (0.13577959505287382) | > loader_time: 0.0049 (0.004816164841523042)  --> TIME: 2025-02-24 03:33:17 -- STEP: 198/36938 -- GLOBAL_STEP: 147950 | > loss_text_ce: 0.04121340811252594 (0.0504341506577953) | > loss_mel_ce: 2.894226312637329 (nan) | > loss: 0.011648571118712425 (nan) | > current_lr: 5e-06 | > step_time: 0.1083 (0.13632421902935926) | > loader_time: 0.0045 (0.004801387738699867)  --> TIME: 2025-02-24 03:33:26 -- STEP: 248/36938 -- GLOBAL_STEP: 148000 | > loss_text_ce: 0.049170441925525665 (0.05021455791598607) | > loss_mel_ce: 1.4073575735092163 (nan) | > loss: 0.005779873579740524 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13657539506112382) | > loader_time: 0.0046 (0.004788192049149544)  --> TIME: 2025-02-24 03:33:35 -- STEP: 298/36938 -- GLOBAL_STEP: 148050 | > loss_text_ce: 0.0505484864115715 (0.05032089430738814) | > loss_mel_ce: 4.008098125457764 (nan) | > loss: 0.016105741262435913 (nan) | > current_lr: 5e-06 | > step_time: 0.1832 (0.13672674182277403) | > loader_time: 0.0049 (0.004776164189281081)  --> TIME: 2025-02-24 03:33:45 -- STEP: 348/36938 -- GLOBAL_STEP: 148100 | > loss_text_ce: 0.04409318417310715 (0.05020676423185345) | > loss_mel_ce: 3.370713710784912 (nan) | > loss: 0.01355082169175148 (nan) | > current_lr: 5e-06 | > step_time: 0.1357 (0.13753626606930258) | > loader_time: 0.0046 (0.004776222267370115)  --> TIME: 2025-02-24 03:33:55 -- STEP: 398/36938 -- GLOBAL_STEP: 148150 | > loss_text_ce: 0.04723824933171272 (0.05006838082125289) | > loss_mel_ce: 3.54379940032959 (nan) | > loss: 0.014250150881707668 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13803268736930355) | > loader_time: 0.0048 (0.004778449858852373)  --> TIME: 2025-02-24 03:34:04 -- STEP: 448/36938 -- GLOBAL_STEP: 148200 | > loss_text_ce: 0.048431411385536194 (0.05007283981623394) | > loss_mel_ce: 3.5347511768341064 (nan) | > loss: 0.014218979515135288 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.1377097118113723) | > loader_time: 0.0051 (0.004781096109322142)  --> TIME: 2025-02-24 03:34:13 -- STEP: 498/36938 -- GLOBAL_STEP: 148250 | > loss_text_ce: 0.05027807503938675 (0.05001362784948933) | > loss_mel_ce: 3.344407081604004 (nan) | > loss: 0.013470973819494247 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13755389916370192) | > loader_time: 0.0047 (0.0047689628409573375)  --> TIME: 2025-02-24 03:34:23 -- STEP: 548/36938 -- GLOBAL_STEP: 148300 | > loss_text_ce: 0.05470757186412811 (0.050115666175465076) | > loss_mel_ce: 4.427523136138916 (nan) | > loss: 0.017786631360650063 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.1377560013402117) | > loader_time: 0.0046 (0.0047696650463299175)  --> TIME: 2025-02-24 03:34:32 -- STEP: 598/36938 -- GLOBAL_STEP: 148350 | > loss_text_ce: 0.045220959931612015 (0.04986712569526606) | > loss_mel_ce: 4.261001110076904 (nan) | > loss: 0.017088184133172035 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13788245713033) | > loader_time: 0.0046 (0.00476430092368238)  --> TIME: 2025-02-24 03:34:42 -- STEP: 648/36938 -- GLOBAL_STEP: 148400 | > loss_text_ce: 0.048167433589696884 (0.049760330037248376) | > loss_mel_ce: 3.2988789081573486 (nan) | > loss: 0.013281931169331074 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13786480235464774) | > loader_time: 0.0046 (0.004759409545380397)  --> TIME: 2025-02-24 03:34:51 -- STEP: 698/36938 -- GLOBAL_STEP: 148450 | > loss_text_ce: 0.04000741243362427 (0.04969994650144975) | > loss_mel_ce: 3.807241439819336 (nan) | > loss: 0.015266860835254192 (nan) | > current_lr: 5e-06 | > step_time: 0.1466 (0.13765099397019873) | > loader_time: 0.0046 (0.004755626093692293)  --> TIME: 2025-02-24 03:35:00 -- STEP: 748/36938 -- GLOBAL_STEP: 148500 | > loss_text_ce: 0.051066506654024124 (0.04971693179017323) | > loss_mel_ce: 3.59700083732605 (nan) | > loss: 0.014476458542048931 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.1370706108802143) | > loader_time: 0.0067 (0.004763840672804077)  --> TIME: 2025-02-24 03:35:09 -- STEP: 798/36938 -- GLOBAL_STEP: 148550 | > loss_text_ce: 0.05394855886697769 (0.04968789877150592) | > loss_mel_ce: 3.4580225944519043 (nan) | > loss: 0.013936394825577736 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13678861859448274) | > loader_time: 0.0045 (0.004756263921732892)  --> TIME: 2025-02-24 03:35:18 -- STEP: 848/36938 -- GLOBAL_STEP: 148600 | > loss_text_ce: 0.057402342557907104 (0.04974330203906406) | > loss_mel_ce: 2.9579994678497314 (nan) | > loss: 0.01196588110178709 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13679192819685312) | > loader_time: 0.0052 (0.004758057166945261)  --> TIME: 2025-02-24 03:35:27 -- STEP: 898/36938 -- GLOBAL_STEP: 148650 | > loss_text_ce: 0.06615813821554184 (0.04974465630118859) | > loss_mel_ce: 3.6983511447906494 (nan) | > loss: 0.014938529580831528 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1368057974728285) | > loader_time: 0.0047 (0.004754263202437846)  --> TIME: 2025-02-24 03:35:37 -- STEP: 948/36938 -- GLOBAL_STEP: 148700 | > loss_text_ce: 0.044145368039608 (0.049715929540888056) | > loss_mel_ce: 2.9564788341522217 (nan) | > loss: 0.011907239444553852 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13685242083505247) | > loader_time: 0.0045 (0.0047604512061750846)  --> TIME: 2025-02-24 03:35:46 -- STEP: 998/36938 -- GLOBAL_STEP: 148750 | > loss_text_ce: 0.06183424964547157 (0.04975871109629503) | > loss_mel_ce: 3.126399040222168 (nan) | > loss: 0.012651720084249973 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13684642959931104) | > loader_time: 0.0046 (0.004755125017108804)  --> TIME: 2025-02-24 03:35:56 -- STEP: 1048/36938 -- GLOBAL_STEP: 148800 | > loss_text_ce: 0.05729827657341957 (0.04971254251378354) | > loss_mel_ce: 3.972627878189087 (nan) | > loss: 0.01599177159368992 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13702750023994748) | > loader_time: 0.0046 (0.004752173906064216)  --> TIME: 2025-02-24 03:36:05 -- STEP: 1098/36938 -- GLOBAL_STEP: 148850 | > loss_text_ce: 0.06406117975711823 (0.049798268300628744) | > loss_mel_ce: 3.477365493774414 (nan) | > loss: 0.014053281396627426 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13685497277855665) | > loader_time: 0.0046 (0.004750900364095832)  --> TIME: 2025-02-24 03:36:14 -- STEP: 1148/36938 -- GLOBAL_STEP: 148900 | > loss_text_ce: 0.06382457911968231 (0.04973292749745038) | > loss_mel_ce: 0.032867431640625 (nan) | > loss: 0.0003836984687950462 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13664359996543116) | > loader_time: 0.0046 (0.004750841170653233)  --> TIME: 2025-02-24 03:36:23 -- STEP: 1198/36938 -- GLOBAL_STEP: 148950 | > loss_text_ce: 0.056278329342603683 (0.04974199362403911) | > loss_mel_ce: 4.276085376739502 (nan) | > loss: 0.017191920429468155 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13671323910778488) | > loader_time: 0.0049 (0.0047504911836678265)  --> TIME: 2025-02-24 03:36:32 -- STEP: 1248/36938 -- GLOBAL_STEP: 149000 | > loss_text_ce: 0.05557064339518547 (0.0497210866201502) | > loss_mel_ce: 2.0286896228790283 (nan) | > loss: 0.008270874619483948 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13671613713869699) | > loader_time: 0.0046 (0.004750730708623546)  --> TIME: 2025-02-24 03:36:42 -- STEP: 1298/36938 -- GLOBAL_STEP: 149050 | > loss_text_ce: 0.04108746349811554 (0.049767988136982445) | > loss_mel_ce: 3.211040496826172 (nan) | > loss: 0.012905269861221313 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.13658907674310397) | > loader_time: 0.0046 (0.004747424728146686)  --> TIME: 2025-02-24 03:36:51 -- STEP: 1348/36938 -- GLOBAL_STEP: 149100 | > loss_text_ce: 0.0432780459523201 (0.04975630637004045) | > loss_mel_ce: 3.7550899982452393 (nan) | > loss: 0.015072889626026154 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.1365891530534989) | > loader_time: 0.0048 (0.004742076205924283)  --> TIME: 2025-02-24 03:37:00 -- STEP: 1398/36938 -- GLOBAL_STEP: 149150 | > loss_text_ce: 0.05455564334988594 (0.049770873546259274) | > loss_mel_ce: 3.973726749420166 (nan) | > loss: 0.015985246747732162 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.13663521805546325) | > loader_time: 0.0047 (0.0047388489495360595)  --> TIME: 2025-02-24 03:37:09 -- STEP: 1448/36938 -- GLOBAL_STEP: 149200 | > loss_text_ce: 0.05587116628885269 (0.04978800476888886) | > loss_mel_ce: 3.9734458923339844 (nan) | > loss: 0.015989353880286217 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13662173734844066) | > loader_time: 0.0047 (0.00473766761590104)  --> TIME: 2025-02-24 03:37:19 -- STEP: 1498/36938 -- GLOBAL_STEP: 149250 | > loss_text_ce: 0.05058814585208893 (0.04984630450720743) | > loss_mel_ce: 3.3067188262939453 (nan) | > loss: 0.013322647660970688 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1365781420859221) | > loader_time: 0.0046 (0.004734253533214372)  --> TIME: 2025-02-24 03:37:28 -- STEP: 1548/36938 -- GLOBAL_STEP: 149300 | > loss_text_ce: 0.02839127741754055 (0.04982309320397696) | > loss_mel_ce: 1.3404507637023926 (nan) | > loss: 0.005431912839412689 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1366743850461579) | > loader_time: 0.0046 (0.00473161122595617)  --> TIME: 2025-02-24 03:37:37 -- STEP: 1598/36938 -- GLOBAL_STEP: 149350 | > loss_text_ce: 0.045161809772253036 (0.04978361434521195) | > loss_mel_ce: 3.631509780883789 (nan) | > loss: 0.014589967206120491 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13657076143949876) | > loader_time: 0.0045 (0.004729552024297034)  --> TIME: 2025-02-24 03:37:47 -- STEP: 1648/36938 -- GLOBAL_STEP: 149400 | > loss_text_ce: 0.06226596236228943 (0.04977416113421359) | > loss_mel_ce: 3.3015010356903076 (nan) | > loss: 0.013348282314836979 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13666715972053176) | > loader_time: 0.0046 (0.004728293823964386)  --> TIME: 2025-02-24 03:37:56 -- STEP: 1698/36938 -- GLOBAL_STEP: 149450 | > loss_text_ce: 0.04601903632283211 (0.04975749214714037) | > loss_mel_ce: 3.107456922531128 (nan) | > loss: 0.012513794004917145 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.1366642116798247) | > loader_time: 0.0047 (0.004730190629813081)  --> TIME: 2025-02-24 03:38:05 -- STEP: 1748/36938 -- GLOBAL_STEP: 149500 | > loss_text_ce: 0.04665398597717285 (0.04975066395813187) | > loss_mel_ce: 3.6765427589416504 (nan) | > loss: 0.014774590730667114 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13665501108431433) | > loader_time: 0.0047 (0.004728039696778533)  --> TIME: 2025-02-24 03:38:15 -- STEP: 1798/36938 -- GLOBAL_STEP: 149550 | > loss_text_ce: 0.05052092298865318 (0.0497837547871706) | > loss_mel_ce: 0.0323079414665699 (nan) | > loss: 0.00032868600101210177 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1365533309995929) | > loader_time: 0.0047 (0.004726565719579035)  --> TIME: 2025-02-24 03:38:24 -- STEP: 1848/36938 -- GLOBAL_STEP: 149600 | > loss_text_ce: 0.0789356455206871 (0.04976771116857438) | > loss_mel_ce: 0.01495361328125 (nan) | > loss: 0.0003725764399860054 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13655127681695017) | > loader_time: 0.0045 (0.004724491855282806)  --> TIME: 2025-02-24 03:38:33 -- STEP: 1898/36938 -- GLOBAL_STEP: 149650 | > loss_text_ce: 0.05328184366226196 (0.04973831962601718) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13645300649114347) | > loader_time: 0.0045 (0.004723230327771011)  --> TIME: 2025-02-24 03:38:42 -- STEP: 1948/36938 -- GLOBAL_STEP: 149700 | > loss_text_ce: 0.03455963730812073 (0.04974676153069895) | > loss_mel_ce: 2.569566488265991 (nan) | > loss: 0.010333835147321224 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13637986724136786) | > loader_time: 0.0046 (0.0047219196139419856)  --> TIME: 2025-02-24 03:38:51 -- STEP: 1998/36938 -- GLOBAL_STEP: 149750 | > loss_text_ce: 0.038636308163404465 (0.04976128121918149) | > loss_mel_ce: 1.6201035976409912 (nan) | > loss: 0.006582301575690508 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.1363661891824609) | > loader_time: 0.0045 (0.004718766675458419)  --> TIME: 2025-02-24 03:39:00 -- STEP: 2048/36938 -- GLOBAL_STEP: 149800 | > loss_text_ce: 0.04879150912165642 (0.049746109891202664) | > loss_mel_ce: 3.8185782432556152 (nan) | > loss: 0.015346705913543701 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.1363478180719539) | > loader_time: 0.0046 (0.004717675619758667)  --> TIME: 2025-02-24 03:39:10 -- STEP: 2098/36938 -- GLOBAL_STEP: 149850 | > loss_text_ce: 0.0416712760925293 (0.04972258821889773) | > loss_mel_ce: 3.4076194763183594 (nan) | > loss: 0.013687662780284882 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.13633957127370186) | > loader_time: 0.0048 (0.004719814081437483)  --> TIME: 2025-02-24 03:39:19 -- STEP: 2148/36938 -- GLOBAL_STEP: 149900 | > loss_text_ce: 0.052911680191755295 (0.04969547360483384) | > loss_mel_ce: 2.5291173458099365 (nan) | > loss: 0.010246147401630878 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13639910660642485) | > loader_time: 0.0046 (0.004720071729557032)  --> TIME: 2025-02-24 03:39:29 -- STEP: 2198/36938 -- GLOBAL_STEP: 149950 | > loss_text_ce: 0.049667567014694214 (0.049699190010853087) | > loss_mel_ce: 3.7474048137664795 (nan) | > loss: 0.015067748725414276 (nan) | > current_lr: 5e-06 | > step_time: 0.1844 (0.13640885867239455) | > loader_time: 0.005 (0.004718943549026893)  --> TIME: 2025-02-24 03:39:38 -- STEP: 2248/36938 -- GLOBAL_STEP: 150000 | > loss_text_ce: 0.03773786127567291 (0.04968514077007452) | > loss_mel_ce: 3.2866625785827637 (nan) | > loss: 0.013192065991461277 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13640856827705355) | > loader_time: 0.0049 (0.004718316089216077) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_150000.pth  --> TIME: 2025-02-24 03:39:53 -- STEP: 2298/36938 -- GLOBAL_STEP: 150050 | > loss_text_ce: 0.04724150151014328 (0.049665687578179914) | > loss_mel_ce: 3.555957317352295 (nan) | > loss: 0.014298408292233944 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13625942529855964) | > loader_time: 0.0047 (0.004720118379883399)  --> TIME: 2025-02-24 03:40:02 -- STEP: 2348/36938 -- GLOBAL_STEP: 150100 | > loss_text_ce: 0.049995847046375275 (0.04963364397703939) | > loss_mel_ce: 3.824002742767334 (nan) | > loss: 0.015373011119663715 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1362677203329498) | > loader_time: 0.0047 (0.004720446807562395)  --> TIME: 2025-02-24 03:40:11 -- STEP: 2398/36938 -- GLOBAL_STEP: 150150 | > loss_text_ce: 0.06044125556945801 (0.049609534072692016) | > loss_mel_ce: 2.3580260276794434 (nan) | > loss: 0.009597092866897583 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13616804623226) | > loader_time: 0.0045 (0.004718352000448881)  --> TIME: 2025-02-24 03:40:20 -- STEP: 2448/36938 -- GLOBAL_STEP: 150200 | > loss_text_ce: 0.03601740673184395 (0.04960189590358409) | > loss_mel_ce: 3.545138359069824 (nan) | > loss: 0.014210936613380909 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13614336123653464) | > loader_time: 0.0046 (0.004716998903579962)  --> TIME: 2025-02-24 03:40:29 -- STEP: 2498/36938 -- GLOBAL_STEP: 150250 | > loss_text_ce: 0.03756294026970863 (0.049590253151857736) | > loss_mel_ce: 2.1641652584075928 (nan) | > loss: 0.008737016469240189 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.1360235993054696) | > loader_time: 0.0046 (0.004715674585872313)  --> TIME: 2025-02-24 03:40:38 -- STEP: 2548/36938 -- GLOBAL_STEP: 150300 | > loss_text_ce: 0.04898839071393013 (0.04953430532996819) | > loss_mel_ce: 3.363257646560669 (nan) | > loss: 0.013540659099817276 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13594874210971472) | > loader_time: 0.0051 (0.004715751254202994)  --> TIME: 2025-02-24 03:40:48 -- STEP: 2598/36938 -- GLOBAL_STEP: 150350 | > loss_text_ce: 0.04358363896608353 (0.04951374127376846) | > loss_mel_ce: 3.998914957046509 (nan) | > loss: 0.01604166254401207 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13592941647221982) | > loader_time: 0.0049 (0.004717593930885369)  --> TIME: 2025-02-24 03:40:57 -- STEP: 2648/36938 -- GLOBAL_STEP: 150400 | > loss_text_ce: 0.05094415694475174 (0.0495079497643535) | > loss_mel_ce: 3.8490850925445557 (nan) | > loss: 0.015476306900382042 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13592851936276948) | > loader_time: 0.0045 (0.0047158486353306595)  --> TIME: 2025-02-24 03:41:06 -- STEP: 2698/36938 -- GLOBAL_STEP: 150450 | > loss_text_ce: 0.06263667345046997 (0.04950303371976353) | > loss_mel_ce: 3.487199306488037 (nan) | > loss: 0.014086651615798473 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.1358729530211464) | > loader_time: 0.0047 (0.004713923158779949)  --> TIME: 2025-02-24 03:41:15 -- STEP: 2748/36938 -- GLOBAL_STEP: 150500 | > loss_text_ce: 0.04579286649823189 (0.049487040057545145) | > loss_mel_ce: 3.204543113708496 (nan) | > loss: 0.012898159213364124 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13596928865802008) | > loader_time: 0.0046 (0.004712690258997841)  --> TIME: 2025-02-24 03:41:25 -- STEP: 2798/36938 -- GLOBAL_STEP: 150550 | > loss_text_ce: 0.043886683881282806 (0.04947940345914808) | > loss_mel_ce: 4.1976637840271 (nan) | > loss: 0.016831550747156143 (nan) | > current_lr: 5e-06 | > step_time: 0.1623 (0.13606557818802037) | > loader_time: 0.0048 (0.004711579049460113)  --> TIME: 2025-02-24 03:41:34 -- STEP: 2848/36938 -- GLOBAL_STEP: 150600 | > loss_text_ce: 0.020956063643097878 (0.04944676008883839) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13610261628467057) | > loader_time: 0.0045 (0.004710430425874298)  --> TIME: 2025-02-24 03:41:44 -- STEP: 2898/36938 -- GLOBAL_STEP: 150650 | > loss_text_ce: 0.0435195155441761 (0.049431019309674154) | > loss_mel_ce: 2.026730537414551 (nan) | > loss: 0.008215278387069702 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13611022565511288) | > loader_time: 0.0046 (0.004708932629118628)  --> TIME: 2025-02-24 03:41:53 -- STEP: 2948/36938 -- GLOBAL_STEP: 150700 | > loss_text_ce: 0.056378964334726334 (0.0494373523316812) | > loss_mel_ce: 3.5733823776245117 (nan) | > loss: 0.014403816312551498 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.1360595388845964) | > loader_time: 0.0047 (0.004707757540279683)  --> TIME: 2025-02-24 03:42:02 -- STEP: 2998/36938 -- GLOBAL_STEP: 150750 | > loss_text_ce: 0.06255564093589783 (0.049439034580046136) | > loss_mel_ce: 3.8840651512145996 (nan) | > loss: 0.015661194920539856 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13607128697128754) | > loader_time: 0.0045 (0.0047071543910489075)  --> TIME: 2025-02-24 03:42:11 -- STEP: 3048/36938 -- GLOBAL_STEP: 150800 | > loss_text_ce: 0.07334961742162704 (0.04943676314101578) | > loss_mel_ce: 1.5887960195541382 (nan) | > loss: 0.00659581646323204 (nan) | > current_lr: 5e-06 | > step_time: 0.1086 (0.13594535564187318) | > loader_time: 0.0047 (0.00470618297421713)  --> TIME: 2025-02-24 03:42:21 -- STEP: 3098/36938 -- GLOBAL_STEP: 150850 | > loss_text_ce: 0.05033579841256142 (0.049414130768890764) | > loss_mel_ce: 4.003546237945557 (nan) | > loss: 0.01608683541417122 (nan) | > current_lr: 5e-06 | > step_time: 0.1452 (0.13601797969207388) | > loader_time: 0.0047 (0.004705594846554303)  --> TIME: 2025-02-24 03:42:30 -- STEP: 3148/36938 -- GLOBAL_STEP: 150900 | > loss_text_ce: 0.05316748842597008 (0.04940373518680523) | > loss_mel_ce: 3.857032060623169 (nan) | > loss: 0.015516665764153004 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13601714221161856) | > loader_time: 0.005 (0.004704347938261306)  --> TIME: 2025-02-24 03:42:39 -- STEP: 3198/36938 -- GLOBAL_STEP: 150950 | > loss_text_ce: 0.04225385934114456 (0.049405396179184796) | > loss_mel_ce: 3.609311819076538 (nan) | > loss: 0.01449034083634615 (nan) | > current_lr: 5e-06 | > step_time: 0.1873 (0.13601931771164494) | > loader_time: 0.0048 (0.0047039628550736385)  --> TIME: 2025-02-24 03:42:48 -- STEP: 3248/36938 -- GLOBAL_STEP: 151000 | > loss_text_ce: 0.05250788480043411 (0.04941376280200118) | > loss_mel_ce: 3.299468755722046 (nan) | > loss: 0.013301495462656021 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.13596617052414173) | > loader_time: 0.0047 (0.004702761695890001)  --> TIME: 2025-02-24 03:42:58 -- STEP: 3298/36938 -- GLOBAL_STEP: 151050 | > loss_text_ce: 0.04678474739193916 (0.04940347242400074) | > loss_mel_ce: 3.5385398864746094 (nan) | > loss: 0.014227478764951229 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13599245131847168) | > loader_time: 0.0046 (0.004702306068471304)  --> TIME: 2025-02-24 03:43:07 -- STEP: 3348/36938 -- GLOBAL_STEP: 151100 | > loss_text_ce: 0.054730091243982315 (0.04938262465310902) | > loss_mel_ce: 3.555554151535034 (nan) | > loss: 0.01432652585208416 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13592992696283573) | > loader_time: 0.0047 (0.004702132947413586)  --> TIME: 2025-02-24 03:43:16 -- STEP: 3398/36938 -- GLOBAL_STEP: 151150 | > loss_text_ce: 0.05034324526786804 (0.049364788964250395) | > loss_mel_ce: 3.7462682723999023 (nan) | > loss: 0.015065919607877731 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.1360511094980765) | > loader_time: 0.0046 (0.00470106008124394)  --> TIME: 2025-02-24 03:43:26 -- STEP: 3448/36938 -- GLOBAL_STEP: 151200 | > loss_text_ce: 0.04518077149987221 (0.049392012785255804) | > loss_mel_ce: 3.5120365619659424 (nan) | > loss: 0.014115942642092705 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.13608381124494284) | > loader_time: 0.0047 (0.0047004768439069455)  --> TIME: 2025-02-24 03:43:35 -- STEP: 3498/36938 -- GLOBAL_STEP: 151250 | > loss_text_ce: 0.044280651956796646 (0.049379362527168125) | > loss_mel_ce: 2.879124164581299 (nan) | > loss: 0.011600813828408718 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.13610008670780754) | > loader_time: 0.0046 (0.004699782687231364)  --> TIME: 2025-02-24 03:43:45 -- STEP: 3548/36938 -- GLOBAL_STEP: 151300 | > loss_text_ce: 0.05019516870379448 (0.04937473626118549) | > loss_mel_ce: 3.2237985134124756 (nan) | > loss: 0.012992039322853088 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13613306723425908) | > loader_time: 0.0047 (0.004700914848482464)  --> TIME: 2025-02-24 03:43:54 -- STEP: 3598/36938 -- GLOBAL_STEP: 151350 | > loss_text_ce: 0.051757022738456726 (0.04936330652474244) | > loss_mel_ce: 2.9960904121398926 (nan) | > loss: 0.012094633653759956 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.136179692020279) | > loader_time: 0.0081 (0.004703467988252245)  --> TIME: 2025-02-24 03:44:03 -- STEP: 3648/36938 -- GLOBAL_STEP: 151400 | > loss_text_ce: 0.05203019082546234 (0.04933293234799953) | > loss_mel_ce: 3.4775848388671875 (nan) | > loss: 0.014006408862769604 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.1361385593027404) | > loader_time: 0.0046 (0.004702801048232801)  --> TIME: 2025-02-24 03:44:13 -- STEP: 3698/36938 -- GLOBAL_STEP: 151450 | > loss_text_ce: 0.03749259561300278 (0.04935919771926941) | > loss_mel_ce: 2.652770757675171 (nan) | > loss: 0.010675648227334023 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13617586728493733) | > loader_time: 0.0046 (0.004702518410525369)  --> TIME: 2025-02-24 03:44:22 -- STEP: 3748/36938 -- GLOBAL_STEP: 151500 | > loss_text_ce: 0.044485386461019516 (0.04935032056680338) | > loss_mel_ce: 3.458568811416626 (nan) | > loss: 0.01390100922435522 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.13617330089831733) | > loader_time: 0.0047 (0.004703257610601959)  --> TIME: 2025-02-24 03:44:31 -- STEP: 3798/36938 -- GLOBAL_STEP: 151550 | > loss_text_ce: 0.05198240280151367 (0.049348473427263014) | > loss_mel_ce: 3.2094204425811768 (nan) | > loss: 0.012942075729370117 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13615616476491899) | > loader_time: 0.0045 (0.004703620787103785)  --> TIME: 2025-02-24 03:44:40 -- STEP: 3848/36938 -- GLOBAL_STEP: 151600 | > loss_text_ce: 0.059795159846544266 (0.04935049265913523) | > loss_mel_ce: 2.881072998046875 (nan) | > loss: 0.011670112609863281 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13613589792638256) | > loader_time: 0.0046 (0.004702361172797033)  --> TIME: 2025-02-24 03:44:50 -- STEP: 3898/36938 -- GLOBAL_STEP: 151650 | > loss_text_ce: 0.04893181100487709 (0.04937122819066932) | > loss_mel_ce: 3.6710877418518066 (nan) | > loss: 0.014761983416974545 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.13617817104138047) | > loader_time: 0.0046 (0.004702090177247317)  --> TIME: 2025-02-24 03:44:59 -- STEP: 3948/36938 -- GLOBAL_STEP: 151700 | > loss_text_ce: 0.045083895325660706 (0.04935607913194604) | > loss_mel_ce: 3.2631494998931885 (nan) | > loss: 0.01312791183590889 (nan) | > current_lr: 5e-06 | > step_time: 0.1648 (0.13620962971008096) | > loader_time: 0.0048 (0.004701880034221586)  --> TIME: 2025-02-24 03:45:08 -- STEP: 3998/36938 -- GLOBAL_STEP: 151750 | > loss_text_ce: 0.052877411246299744 (0.04933866533288182) | > loss_mel_ce: 4.464117050170898 (nan) | > loss: 0.017924582585692406 (nan) | > current_lr: 5e-06 | > step_time: 0.186 (0.1361741502622539) | > loader_time: 0.0048 (0.004701008374479906)  --> TIME: 2025-02-24 03:45:17 -- STEP: 4048/36938 -- GLOBAL_STEP: 151800 | > loss_text_ce: 0.05903692916035652 (0.049322801917743485) | > loss_mel_ce: 4.349740505218506 (nan) | > loss: 0.01749514788389206 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13611058348959185) | > loader_time: 0.0048 (0.0047006649462130725)  --> TIME: 2025-02-24 03:45:27 -- STEP: 4098/36938 -- GLOBAL_STEP: 151850 | > loss_text_ce: 0.039924900978803635 (0.04932850534155446) | > loss_mel_ce: 2.906411647796631 (nan) | > loss: 0.011691812425851822 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13618475882351994) | > loader_time: 0.0045 (0.004700337519815577)  --> TIME: 2025-02-24 03:45:36 -- STEP: 4148/36938 -- GLOBAL_STEP: 151900 | > loss_text_ce: 0.056670818477869034 (0.04933086793560787) | > loss_mel_ce: 2.5330944061279297 (nan) | > loss: 0.010276847518980503 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13614204810463515) | > loader_time: 0.0046 (0.004699753358485962)  --> TIME: 2025-02-24 03:45:45 -- STEP: 4198/36938 -- GLOBAL_STEP: 151950 | > loss_text_ce: 0.06883038580417633 (0.0493326325084004) | > loss_mel_ce: 2.6275107860565186 (nan) | > loss: 0.010699767619371414 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.1361080283718831) | > loader_time: 0.0046 (0.004699312317308901)  --> TIME: 2025-02-24 03:45:55 -- STEP: 4248/36938 -- GLOBAL_STEP: 152000 | > loss_text_ce: 0.04947818070650101 (0.049329840296198864) | > loss_mel_ce: 1.000695824623108 (nan) | > loss: 0.004167357459664345 (nan) | > current_lr: 5e-06 | > step_time: 0.1514 (0.1362320466454649) | > loader_time: 0.0046 (0.004699536187051611)  --> TIME: 2025-02-24 03:46:05 -- STEP: 4298/36938 -- GLOBAL_STEP: 152050 | > loss_text_ce: 0.04853643849492073 (0.04933226107250895) | > loss_mel_ce: 3.3542048931121826 (nan) | > loss: 0.013502943329513073 (nan) | > current_lr: 5e-06 | > step_time: 0.1625 (0.13627868499684903) | > loader_time: 0.0047 (0.004699223981229806)  --> TIME: 2025-02-24 03:46:14 -- STEP: 4348/36938 -- GLOBAL_STEP: 152100 | > loss_text_ce: 0.04579103738069534 (0.04932978783803536) | > loss_mel_ce: 3.556972026824951 (nan) | > loss: 0.014296679757535458 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13628561988048019) | > loader_time: 0.0046 (0.004698696768163216)  --> TIME: 2025-02-24 03:46:23 -- STEP: 4398/36938 -- GLOBAL_STEP: 152150 | > loss_text_ce: 0.048546914011240005 (0.04932238036074607) | > loss_mel_ce: 3.4472286701202393 (nan) | > loss: 0.013872127048671246 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.13627670444429588) | > loader_time: 0.0047 (0.004698020157460576)  --> TIME: 2025-02-24 03:46:32 -- STEP: 4448/36938 -- GLOBAL_STEP: 152200 | > loss_text_ce: 0.035313546657562256 (0.04932449993228518) | > loss_mel_ce: 1.3214843273162842 (nan) | > loss: 0.005384119227528572 (nan) | > current_lr: 5e-06 | > step_time: 0.1072 (0.1362560428947) | > loader_time: 0.0104 (0.00469967546008474)  --> TIME: 2025-02-24 03:46:42 -- STEP: 4498/36938 -- GLOBAL_STEP: 152250 | > loss_text_ce: 0.05612017586827278 (0.04930934994061179) | > loss_mel_ce: 3.4692819118499756 (nan) | > loss: 0.013989691622555256 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13624003650136118) | > loader_time: 0.0046 (0.004698608386988322)  --> TIME: 2025-02-24 03:46:51 -- STEP: 4548/36938 -- GLOBAL_STEP: 152300 | > loss_text_ce: 0.03657492995262146 (0.04930932517183926) | > loss_mel_ce: 3.347944974899292 (nan) | > loss: 0.013430634513497353 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13619479670688212) | > loader_time: 0.0045 (0.004698090261813312)  --> TIME: 2025-02-24 03:47:00 -- STEP: 4598/36938 -- GLOBAL_STEP: 152350 | > loss_text_ce: 0.04214192181825638 (0.049324270309868135) | > loss_mel_ce: 3.5786349773406982 (nan) | > loss: 0.014368163421750069 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13621465518921608) | > loader_time: 0.0046 (0.004697169205996203)  --> TIME: 2025-02-24 03:47:09 -- STEP: 4648/36938 -- GLOBAL_STEP: 152400 | > loss_text_ce: 0.04773937165737152 (0.04930923807534112) | > loss_mel_ce: 1.146643042564392 (nan) | > loss: 0.004739613272249699 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13621388911790638) | > loader_time: 0.0045 (0.004697020872529733)  --> TIME: 2025-02-24 03:47:19 -- STEP: 4698/36938 -- GLOBAL_STEP: 152450 | > loss_text_ce: 0.05491093546152115 (0.0493059606493952) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13623750285931852) | > loader_time: 0.0046 (0.004696609365935022)  --> TIME: 2025-02-24 03:47:28 -- STEP: 4748/36938 -- GLOBAL_STEP: 152500 | > loss_text_ce: 0.04377617686986923 (0.049311253660387404) | > loss_mel_ce: 3.3712382316589355 (nan) | > loss: 0.013551645912230015 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13621699599127127) | > loader_time: 0.0046 (0.004697150910915005)  --> TIME: 2025-02-24 03:47:37 -- STEP: 4798/36938 -- GLOBAL_STEP: 152550 | > loss_text_ce: 0.03702930361032486 (0.04930778193043353) | > loss_mel_ce: 2.9783661365509033 (nan) | > loss: 0.011965855024755001 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13624306493324317) | > loader_time: 0.0045 (0.004697379791622321)  --> TIME: 2025-02-24 03:47:47 -- STEP: 4848/36938 -- GLOBAL_STEP: 152600 | > loss_text_ce: 0.04296872019767761 (0.04931553073992739) | > loss_mel_ce: 3.2512547969818115 (nan) | > loss: 0.013072316534817219 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.1362187714859997) | > loader_time: 0.0046 (0.004696652932529015)  --> TIME: 2025-02-24 03:47:56 -- STEP: 4898/36938 -- GLOBAL_STEP: 152650 | > loss_text_ce: 0.03951219096779823 (0.04930838558209835) | > loss_mel_ce: 3.4753665924072266 (nan) | > loss: 0.013947932049632072 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.1361849211147248) | > loader_time: 0.0045 (0.004695767672902089)  --> TIME: 2025-02-24 03:48:05 -- STEP: 4948/36938 -- GLOBAL_STEP: 152700 | > loss_text_ce: 0.0517437681555748 (0.049312994448752555) | > loss_mel_ce: 3.882173538208008 (nan) | > loss: 0.015610783360898495 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.13625297453871899) | > loader_time: 0.0049 (0.004695428699471831)  --> TIME: 2025-02-24 03:48:14 -- STEP: 4998/36938 -- GLOBAL_STEP: 152750 | > loss_text_ce: 0.05179966241121292 (0.04930337823378808) | > loss_mel_ce: 1.4123260974884033 (nan) | > loss: 0.005810023285448551 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13622145349381284) | > loader_time: 0.0045 (0.004694629354732623)  --> TIME: 2025-02-24 03:48:24 -- STEP: 5048/36938 -- GLOBAL_STEP: 152800 | > loss_text_ce: 0.016583118587732315 (0.04930582719747257) | > loss_mel_ce: 2.087895631790161 (nan) | > loss: 0.008351107127964497 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13620683621680202) | > loader_time: 0.0045 (0.004694662267923741)  --> TIME: 2025-02-24 03:48:33 -- STEP: 5098/36938 -- GLOBAL_STEP: 152850 | > loss_text_ce: 0.044274043291807175 (0.04928498438359606) | > loss_mel_ce: 3.678119421005249 (nan) | > loss: 0.01477140374481678 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13616644223094687) | > loader_time: 0.0045 (0.00469391287051262)  --> TIME: 2025-02-24 03:48:42 -- STEP: 5148/36938 -- GLOBAL_STEP: 152900 | > loss_text_ce: 0.044843919575214386 (0.049293557004662855) | > loss_mel_ce: 3.2037932872772217 (nan) | > loss: 0.012891418300569057 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13613522062027567) | > loader_time: 0.0045 (0.0046936507454748465)  --> TIME: 2025-02-24 03:48:51 -- STEP: 5198/36938 -- GLOBAL_STEP: 152950 | > loss_text_ce: 0.033490922302007675 (0.04927734766913714) | > loss_mel_ce: 2.87565279006958 (nan) | > loss: 0.011544222012162209 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13613459907068032) | > loader_time: 0.0048 (0.004693423110093013)  --> TIME: 2025-02-24 03:49:00 -- STEP: 5248/36938 -- GLOBAL_STEP: 153000 | > loss_text_ce: 0.040862083435058594 (0.04927688613519765) | > loss_mel_ce: 4.625 (nan) | > loss: 0.01851532608270645 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.13611840547585008) | > loader_time: 0.0048 (0.004693271773980896)  --> TIME: 2025-02-24 03:49:10 -- STEP: 5298/36938 -- GLOBAL_STEP: 153050 | > loss_text_ce: 0.06448089331388474 (0.049296775941760435) | > loss_mel_ce: 3.57930064201355 (nan) | > loss: 0.014459450729191303 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.13611837966165902) | > loader_time: 0.0048 (0.004692809093039016)  --> TIME: 2025-02-24 03:49:19 -- STEP: 5348/36938 -- GLOBAL_STEP: 153100 | > loss_text_ce: 0.05051952227950096 (0.04927676418627599) | > loss_mel_ce: 1.1808627843856812 (nan) | > loss: 0.004886437673121691 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13609292784464835) | > loader_time: 0.0046 (0.004692423004843723)  --> TIME: 2025-02-24 03:49:28 -- STEP: 5398/36938 -- GLOBAL_STEP: 153150 | > loss_text_ce: 0.052802566438913345 (0.049276954001667934) | > loss_mel_ce: 3.6435484886169434 (nan) | > loss: 0.014668060466647148 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.1361004062916191) | > loader_time: 0.0047 (0.0046917872766337266)  --> TIME: 2025-02-24 03:49:37 -- STEP: 5448/36938 -- GLOBAL_STEP: 153200 | > loss_text_ce: 0.04952643811702728 (0.04927694934308134) | > loss_mel_ce: 3.634922742843628 (nan) | > loss: 0.014620831236243248 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13607343217349885) | > loader_time: 0.0045 (0.004690875084564734)  --> TIME: 2025-02-24 03:49:47 -- STEP: 5498/36938 -- GLOBAL_STEP: 153250 | > loss_text_ce: 0.037733081728219986 (0.049270334317141784) | > loss_mel_ce: 4.244258403778076 (nan) | > loss: 0.016992030665278435 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.1360969159246672) | > loader_time: 0.0046 (0.004690465037282407)  --> TIME: 2025-02-24 03:49:56 -- STEP: 5548/36938 -- GLOBAL_STEP: 153300 | > loss_text_ce: 0.04548191279172897 (0.049268459809894236) | > loss_mel_ce: 3.8112473487854004 (nan) | > loss: 0.01530448254197836 (nan) | > current_lr: 5e-06 | > step_time: 0.1507 (0.13612550346646893) | > loader_time: 0.0049 (0.00469020294816527)  --> TIME: 2025-02-24 03:50:05 -- STEP: 5598/36938 -- GLOBAL_STEP: 153350 | > loss_text_ce: 0.049543220549821854 (0.04926377972211943) | > loss_mel_ce: 4.010482311248779 (nan) | > loss: 0.016111213713884354 (nan) | > current_lr: 5e-06 | > step_time: 0.1529 (0.13613189148366367) | > loader_time: 0.0047 (0.004690206830269021)  --> TIME: 2025-02-24 03:50:15 -- STEP: 5648/36938 -- GLOBAL_STEP: 153400 | > loss_text_ce: 0.04794943332672119 (0.04926017087471341) | > loss_mel_ce: 3.6688079833984375 (nan) | > loss: 0.01474903803318739 (nan) | > current_lr: 5e-06 | > step_time: 0.1965 (0.13615444213911848) | > loader_time: 0.0049 (0.004689990081125216)  --> TIME: 2025-02-24 03:50:24 -- STEP: 5698/36938 -- GLOBAL_STEP: 153450 | > loss_text_ce: 0.04136599972844124 (0.0492624484238607) | > loss_mel_ce: 3.284109115600586 (nan) | > loss: 0.013196331448853016 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.13620524203078638) | > loader_time: 0.0048 (0.0046897716964073655)  --> TIME: 2025-02-24 03:50:34 -- STEP: 5748/36938 -- GLOBAL_STEP: 153500 | > loss_text_ce: 0.04957260191440582 (0.04926574394844394) | > loss_mel_ce: 3.455669403076172 (nan) | > loss: 0.013909691944718361 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13620571349342508) | > loader_time: 0.0044 (0.004689068079160668)  --> TIME: 2025-02-24 03:50:43 -- STEP: 5798/36938 -- GLOBAL_STEP: 153550 | > loss_text_ce: 0.045010823756456375 (0.04926435986489417) | > loss_mel_ce: 3.4841134548187256 (nan) | > loss: 0.014004462398588657 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13618113255245942) | > loader_time: 0.0045 (0.004688198050781549)  --> TIME: 2025-02-24 03:50:52 -- STEP: 5848/36938 -- GLOBAL_STEP: 153600 | > loss_text_ce: 0.04626811295747757 (0.049269621903853825) | > loss_mel_ce: 3.671255350112915 (nan) | > loss: 0.014752078801393509 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.1361929042884012) | > loader_time: 0.0047 (0.004687664161465568)  --> TIME: 2025-02-24 03:51:01 -- STEP: 5898/36938 -- GLOBAL_STEP: 153650 | > loss_text_ce: 0.037152376025915146 (0.04927736114257056) | > loss_mel_ce: 3.3990211486816406 (nan) | > loss: 0.013635609298944473 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13618982436496177) | > loader_time: 0.0045 (0.004687147772728531)  --> TIME: 2025-02-24 03:51:11 -- STEP: 5948/36938 -- GLOBAL_STEP: 153700 | > loss_text_ce: 0.04466145858168602 (0.04927721818897765) | > loss_mel_ce: 3.8226919174194336 (nan) | > loss: 0.015346641652286053 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13622603070920433) | > loader_time: 0.0047 (0.004687063471134194)  --> TIME: 2025-02-24 03:51:20 -- STEP: 5998/36938 -- GLOBAL_STEP: 153750 | > loss_text_ce: 0.05509171262383461 (0.04929009427432309) | > loss_mel_ce: 3.5915122032165527 (nan) | > loss: 0.014470650814473629 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13623207241108082) | > loader_time: 0.0046 (0.004686416328330968)  --> TIME: 2025-02-24 03:51:30 -- STEP: 6048/36938 -- GLOBAL_STEP: 153800 | > loss_text_ce: 0.06398694962263107 (0.04927869233864512) | > loss_mel_ce: 3.6603848934173584 (nan) | > loss: 0.014779254794120789 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13625577155244523) | > loader_time: 0.0046 (0.004686378928088644)  --> TIME: 2025-02-24 03:51:39 -- STEP: 6098/36938 -- GLOBAL_STEP: 153850 | > loss_text_ce: 0.04611733928322792 (0.04926733885610404) | > loss_mel_ce: 4.101203918457031 (nan) | > loss: 0.01645762473344803 (nan) | > current_lr: 5e-06 | > step_time: 0.1799 (0.13628201943688845) | > loader_time: 0.0047 (0.0046868007978324916)  --> TIME: 2025-02-24 03:51:49 -- STEP: 6148/36938 -- GLOBAL_STEP: 153900 | > loss_text_ce: 0.04769332706928253 (0.04927408155544651) | > loss_mel_ce: 3.7474191188812256 (nan) | > loss: 0.015059970319271088 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.13631686581833263) | > loader_time: 0.0047 (0.004686507103733225)  --> TIME: 2025-02-24 03:51:58 -- STEP: 6198/36938 -- GLOBAL_STEP: 153950 | > loss_text_ce: 0.04921805113554001 (0.04927138028388082) | > loss_mel_ce: 0.1059722900390625 (nan) | > loss: 0.0006158347241580486 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13633518651363538) | > loader_time: 0.0045 (0.004685988423131142)  --> TIME: 2025-02-24 03:52:08 -- STEP: 6248/36938 -- GLOBAL_STEP: 154000 | > loss_text_ce: 0.050785839557647705 (0.04926983865005373) | > loss_mel_ce: 3.3182551860809326 (nan) | > loss: 0.013369210995733738 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13635043782705641) | > loader_time: 0.0046 (0.0046858353285111854)  --> TIME: 2025-02-24 03:52:17 -- STEP: 6298/36938 -- GLOBAL_STEP: 154050 | > loss_text_ce: 0.04022862762212753 (0.049258390461943684) | > loss_mel_ce: 3.1874799728393555 (nan) | > loss: 0.012808368541300297 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13634898360474473) | > loader_time: 0.0048 (0.004685315831421528)  --> TIME: 2025-02-24 03:52:26 -- STEP: 6348/36938 -- GLOBAL_STEP: 154100 | > loss_text_ce: 0.04874037951231003 (0.04925412863872058) | > loss_mel_ce: 2.820108413696289 (nan) | > loss: 0.011384321376681328 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.13631847904550357) | > loader_time: 0.0045 (0.0046847001065528915)  --> TIME: 2025-02-24 03:52:36 -- STEP: 6398/36938 -- GLOBAL_STEP: 154150 | > loss_text_ce: 0.05418429896235466 (0.04924421253718633) | > loss_mel_ce: 3.9142098426818848 (nan) | > loss: 0.015747595578432083 (nan) | > current_lr: 5e-06 | > step_time: 0.1791 (0.136374214881285) | > loader_time: 0.0049 (0.004684614367543187)  --> TIME: 2025-02-24 03:52:45 -- STEP: 6448/36938 -- GLOBAL_STEP: 154200 | > loss_text_ce: 0.04712270572781563 (0.04925143736889507) | > loss_mel_ce: 1.4783257246017456 (nan) | > loss: 0.006053367163985968 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13638359555979312) | > loader_time: 0.0045 (0.0046842581137238395)  --> TIME: 2025-02-24 03:52:54 -- STEP: 6498/36938 -- GLOBAL_STEP: 154250 | > loss_text_ce: 0.04668205976486206 (0.04924404874206653) | > loss_mel_ce: 2.1445212364196777 (nan) | > loss: 0.008695252239704132 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1363767622507115) | > loader_time: 0.0045 (0.004683915744674659)  --> TIME: 2025-02-24 03:53:04 -- STEP: 6548/36938 -- GLOBAL_STEP: 154300 | > loss_text_ce: 0.0569712370634079 (0.049235681131329555) | > loss_mel_ce: 0.0846354141831398 (nan) | > loss: 0.0005619312287308276 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13639498922106924) | > loader_time: 0.0046 (0.004683458885188222)  --> TIME: 2025-02-24 03:53:13 -- STEP: 6598/36938 -- GLOBAL_STEP: 154350 | > loss_text_ce: 0.0461096428334713 (0.04922868497980222) | > loss_mel_ce: 3.195096731185913 (nan) | > loss: 0.012861930765211582 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1364185667717311) | > loader_time: 0.0046 (0.004683083785451092)  --> TIME: 2025-02-24 03:53:23 -- STEP: 6648/36938 -- GLOBAL_STEP: 154400 | > loss_text_ce: 0.057126447558403015 (0.0492255356766296) | > loss_mel_ce: 3.3227245807647705 (nan) | > loss: 0.013412108644843102 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.1364249452572015) | > loader_time: 0.0045 (0.004682864200337277)  --> TIME: 2025-02-24 03:53:32 -- STEP: 6698/36938 -- GLOBAL_STEP: 154450 | > loss_text_ce: 0.047219499945640564 (0.0492386079420301) | > loss_mel_ce: 3.8001041412353516 (nan) | > loss: 0.015267157927155495 (nan) | > current_lr: 5e-06 | > step_time: 0.1532 (0.136478748809904) | > loader_time: 0.0047 (0.004682650136107586)  --> TIME: 2025-02-24 03:53:42 -- STEP: 6748/36938 -- GLOBAL_STEP: 154500 | > loss_text_ce: 0.05536944419145584 (0.04925011306842112) | > loss_mel_ce: 3.9639925956726074 (nan) | > loss: 0.015949850901961327 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.13650013231708302) | > loader_time: 0.0049 (0.004682526760858836)  --> TIME: 2025-02-24 03:53:51 -- STEP: 6798/36938 -- GLOBAL_STEP: 154550 | > loss_text_ce: 0.059912871569395065 (0.04926256821103107) | > loss_mel_ce: 2.6960537433624268 (nan) | > loss: 0.010936376638710499 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.1365051940806869) | > loader_time: 0.0046 (0.004682571791873054)  --> TIME: 2025-02-24 03:54:00 -- STEP: 6848/36938 -- GLOBAL_STEP: 154600 | > loss_text_ce: 0.04831812158226967 (0.04924146074009615) | > loss_mel_ce: 4.063084125518799 (nan) | > loss: 0.01631508767604828 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.1364988063540422) | > loader_time: 0.0046 (0.004682129440463607)  --> TIME: 2025-02-24 03:54:09 -- STEP: 6898/36938 -- GLOBAL_STEP: 154650 | > loss_text_ce: 0.058083221316337585 (0.04925231277298905) | > loss_mel_ce: 3.3769233226776123 (nan) | > loss: 0.01363097969442606 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.13648979316140794) | > loader_time: 0.0048 (0.004681982694068983)  --> TIME: 2025-02-24 03:54:19 -- STEP: 6948/36938 -- GLOBAL_STEP: 154700 | > loss_text_ce: 0.05681673809885979 (0.04925006898827777) | > loss_mel_ce: 4.15525484085083 (nan) | > loss: 0.016714569181203842 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13647311754314612) | > loader_time: 0.0046 (0.004681419145807029)  --> TIME: 2025-02-24 03:54:28 -- STEP: 6998/36938 -- GLOBAL_STEP: 154750 | > loss_text_ce: 0.040369004011154175 (0.04924807124447775) | > loss_mel_ce: 2.0288729667663574 (nan) | > loss: 0.008211278356611729 (nan) | > current_lr: 5e-06 | > step_time: 0.1506 (0.13645008800029165) | > loader_time: 0.0046 (0.004681034679581841)  --> TIME: 2025-02-24 03:54:37 -- STEP: 7048/36938 -- GLOBAL_STEP: 154800 | > loss_text_ce: 0.03786954656243324 (0.04923673859579643) | > loss_mel_ce: 3.5746195316314697 (nan) | > loss: 0.014335274696350098 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13642453629773074) | > loader_time: 0.0046 (0.0046808816720895555)  --> TIME: 2025-02-24 03:54:46 -- STEP: 7098/36938 -- GLOBAL_STEP: 154850 | > loss_text_ce: 0.04264341667294502 (0.04921614297419761) | > loss_mel_ce: 3.4874985218048096 (nan) | > loss: 0.014008499681949615 (nan) | > current_lr: 5e-06 | > step_time: 0.1606 (0.13642085082230768) | > loader_time: 0.0045 (0.004680496365226616)  --> TIME: 2025-02-24 03:54:56 -- STEP: 7148/36938 -- GLOBAL_STEP: 154900 | > loss_text_ce: 0.05228479579091072 (0.049221124991234544) | > loss_mel_ce: 0.03515625 (nan) | > loss: 0.00034698829404078424 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1364498495321811) | > loader_time: 0.0046 (0.004680194431800194)  --> TIME: 2025-02-24 03:55:05 -- STEP: 7198/36938 -- GLOBAL_STEP: 154950 | > loss_text_ce: 0.04676791653037071 (0.049225234847839715) | > loss_mel_ce: 1.9138474464416504 (nan) | > loss: 0.0077802203595638275 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13645336534553726) | > loader_time: 0.0048 (0.004680044189298389)  --> TIME: 2025-02-24 03:55:14 -- STEP: 7248/36938 -- GLOBAL_STEP: 155000 | > loss_text_ce: 0.04793810099363327 (0.04921825220587369) | > loss_mel_ce: 3.7428743839263916 (nan) | > loss: 0.015042907558381557 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13646487105498062) | > loader_time: 0.0045 (0.004679556615305276)  --> TIME: 2025-02-24 03:55:24 -- STEP: 7298/36938 -- GLOBAL_STEP: 155050 | > loss_text_ce: 0.048391036689281464 (0.049212143558393144) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.1364849625825563) | > loader_time: 0.0048 (0.004679384183217232)  --> TIME: 2025-02-24 03:55:33 -- STEP: 7348/36938 -- GLOBAL_STEP: 155100 | > loss_text_ce: 0.056607864797115326 (0.04920772716877533) | > loss_mel_ce: 4.308212757110596 (nan) | > loss: 0.017320716753602028 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13650617696178632) | > loader_time: 0.0045 (0.004679133305422444)  --> TIME: 2025-02-24 03:55:43 -- STEP: 7398/36938 -- GLOBAL_STEP: 155150 | > loss_text_ce: 0.05023546516895294 (0.04919745184875922) | > loss_mel_ce: 2.967059850692749 (nan) | > loss: 0.011973395012319088 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13653188829455568) | > loader_time: 0.0046 (0.004678920431052018)  --> TIME: 2025-02-24 03:55:52 -- STEP: 7448/36938 -- GLOBAL_STEP: 155200 | > loss_text_ce: 0.0531572625041008 (0.049174044015902287) | > loss_mel_ce: 4.243844985961914 (nan) | > loss: 0.01705159805715084 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.13654169667548252) | > loader_time: 0.0047 (0.0046783895036715956)  --> TIME: 2025-02-24 03:56:01 -- STEP: 7498/36938 -- GLOBAL_STEP: 155250 | > loss_text_ce: 0.06119035184383392 (0.049184119577523736) | > loss_mel_ce: 0.0322062186896801 (nan) | > loss: 0.0003706213610712439 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13651704358940164) | > loader_time: 0.0046 (0.004677862668234573)  --> TIME: 2025-02-24 03:56:11 -- STEP: 7548/36938 -- GLOBAL_STEP: 155300 | > loss_text_ce: 0.042303770780563354 (0.04918928230260296) | > loss_mel_ce: 2.674332618713379 (nan) | > loss: 0.010780303739011288 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13652892354688329) | > loader_time: 0.0045 (0.004677561899942271)  --> TIME: 2025-02-24 03:56:20 -- STEP: 7598/36938 -- GLOBAL_STEP: 155350 | > loss_text_ce: 0.058171581476926804 (0.04919383051297976) | > loss_mel_ce: 3.4263839721679688 (nan) | > loss: 0.013827602379024029 (nan) | > current_lr: 5e-06 | > step_time: 0.1897 (0.13654854404954614) | > loader_time: 0.0049 (0.004677246388211706)  --> TIME: 2025-02-24 03:56:30 -- STEP: 7648/36938 -- GLOBAL_STEP: 155400 | > loss_text_ce: 0.045848578214645386 (0.04917207050841551) | > loss_mel_ce: 3.8927910327911377 (nan) | > loss: 0.015629522502422333 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.13657068723041216) | > loader_time: 0.0047 (0.004676767753507316)  --> TIME: 2025-02-24 03:56:39 -- STEP: 7698/36938 -- GLOBAL_STEP: 155450 | > loss_text_ce: 0.058260053396224976 (0.04917318269196153) | > loss_mel_ce: 3.228214740753174 (nan) | > loss: 0.01304156705737114 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13655849192538372) | > loader_time: 0.0046 (0.0046762385037571715)  --> TIME: 2025-02-24 03:56:48 -- STEP: 7748/36938 -- GLOBAL_STEP: 155500 | > loss_text_ce: 0.046431463211774826 (0.049153754070533816) | > loss_mel_ce: 3.576627254486084 (nan) | > loss: 0.014377218671143055 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13658130695109716) | > loader_time: 0.0045 (0.004675907945903738)  --> TIME: 2025-02-24 03:56:58 -- STEP: 7798/36938 -- GLOBAL_STEP: 155550 | > loss_text_ce: 0.04882824793457985 (0.049152912256967315) | > loss_mel_ce: 1.3908262252807617 (nan) | > loss: 0.0057129147462546825 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13656458624756446) | > loader_time: 0.0047 (0.004675393564391069)  --> TIME: 2025-02-24 03:57:07 -- STEP: 7848/36938 -- GLOBAL_STEP: 155600 | > loss_text_ce: 0.05280270799994469 (0.0491555763015031) | > loss_mel_ce: 3.3070127964019775 (nan) | > loss: 0.013332602567970753 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.13654395910578107) | > loader_time: 0.0048 (0.004675156084647857)  --> TIME: 2025-02-24 03:57:16 -- STEP: 7898/36938 -- GLOBAL_STEP: 155650 | > loss_text_ce: 0.05608813092112541 (0.04914701029433792) | > loss_mel_ce: 3.87994647026062 (nan) | > loss: 0.015619185753166676 (nan) | > current_lr: 5e-06 | > step_time: 0.1822 (0.1365132833257698) | > loader_time: 0.0048 (0.0046745482261466575)  --> TIME: 2025-02-24 03:57:25 -- STEP: 7948/36938 -- GLOBAL_STEP: 155700 | > loss_text_ce: 0.04535768926143646 (0.04913367432971943) | > loss_mel_ce: 3.41986083984375 (nan) | > loss: 0.013750867918133736 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13648016976542318) | > loader_time: 0.0046 (0.0046740159894811855)  --> TIME: 2025-02-24 03:57:34 -- STEP: 7998/36938 -- GLOBAL_STEP: 155750 | > loss_text_ce: 0.04939436912536621 (0.049120936559983105) | > loss_mel_ce: 4.102673053741455 (nan) | > loss: 0.016476457938551903 (nan) | > current_lr: 5e-06 | > step_time: 0.1948 (0.1364939700785093) | > loader_time: 0.0048 (0.004673627413162813)  --> TIME: 2025-02-24 03:57:44 -- STEP: 8048/36938 -- GLOBAL_STEP: 155800 | > loss_text_ce: 0.05424495413899422 (0.049128355445417965) | > loss_mel_ce: 3.8476526737213135 (nan) | > loss: 0.01548372209072113 (nan) | > current_lr: 5e-06 | > step_time: 0.1783 (0.13649578706526208) | > loader_time: 0.0047 (0.00467322049866141)  --> TIME: 2025-02-24 03:57:53 -- STEP: 8098/36938 -- GLOBAL_STEP: 155850 | > loss_text_ce: 0.04345560818910599 (0.04912362293501037) | > loss_mel_ce: 3.0628490447998047 (nan) | > loss: 0.012326606549322605 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13648262837104555) | > loader_time: 0.0048 (0.004672601565104069)  --> TIME: 2025-02-24 03:58:02 -- STEP: 8148/36938 -- GLOBAL_STEP: 155900 | > loss_text_ce: 0.05417004972696304 (0.04912859909181559) | > loss_mel_ce: 4.1225905418396 (nan) | > loss: 0.016574447974562645 (nan) | > current_lr: 5e-06 | > step_time: 0.1605 (0.13650287540742045) | > loader_time: 0.0046 (0.004672112772730454)  --> TIME: 2025-02-24 03:58:12 -- STEP: 8198/36938 -- GLOBAL_STEP: 155950 | > loss_text_ce: 0.05025097355246544 (0.0491231767745621) | > loss_mel_ce: 0.7566325068473816 (nan) | > loss: 0.003201918676495552 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13650372965390006) | > loader_time: 0.0046 (0.004671613394734231)  --> TIME: 2025-02-24 03:58:21 -- STEP: 8248/36938 -- GLOBAL_STEP: 156000 | > loss_text_ce: 0.037321850657463074 (0.049119227036931244) | > loss_mel_ce: 3.174133777618408 (nan) | > loss: 0.012743871659040451 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13650894115898535) | > loader_time: 0.0046 (0.004671140883757477)  --> TIME: 2025-02-24 03:58:30 -- STEP: 8298/36938 -- GLOBAL_STEP: 156050 | > loss_text_ce: 0.04738602414727211 (0.0491134464540077) | > loss_mel_ce: 3.6078734397888184 (nan) | > loss: 0.014504997991025448 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13650491057203143) | > loader_time: 0.0045 (0.004670620453044529)  --> TIME: 2025-02-24 03:58:39 -- STEP: 8348/36938 -- GLOBAL_STEP: 156100 | > loss_text_ce: 0.04172063618898392 (0.04910513608273295) | > loss_mel_ce: 3.207545757293701 (nan) | > loss: 0.012893915176391602 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13648463232299102) | > loader_time: 0.0045 (0.00466989174263868)  --> TIME: 2025-02-24 03:58:49 -- STEP: 8398/36938 -- GLOBAL_STEP: 156150 | > loss_text_ce: 0.05454251170158386 (0.04909845538637864) | > loss_mel_ce: 4.056417942047119 (nan) | > loss: 0.016313336789608 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.1365165507177371) | > loader_time: 0.0045 (0.004669447318801838)  --> TIME: 2025-02-24 03:58:58 -- STEP: 8448/36938 -- GLOBAL_STEP: 156200 | > loss_text_ce: 0.053529832512140274 (0.049098754928060956) | > loss_mel_ce: 2.9045376777648926 (nan) | > loss: 0.011738363653421402 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.1365381285096666) | > loader_time: 0.0045 (0.004668989529212328)  --> TIME: 2025-02-24 03:59:08 -- STEP: 8498/36938 -- GLOBAL_STEP: 156250 | > loss_text_ce: 0.055009663105010986 (0.04909521204661189) | > loss_mel_ce: 3.668937921524048 (nan) | > loss: 0.01477757003158331 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13654002051377084) | > loader_time: 0.0045 (0.004668465135349457)  --> TIME: 2025-02-24 03:59:17 -- STEP: 8548/36938 -- GLOBAL_STEP: 156300 | > loss_text_ce: 0.07828494161367416 (0.049096114482211066) | > loss_mel_ce: 3.5368008613586426 (nan) | > loss: 0.014345579780638218 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.1365128328825868) | > loader_time: 0.0052 (0.004668332814166197)  --> TIME: 2025-02-24 03:59:26 -- STEP: 8598/36938 -- GLOBAL_STEP: 156350 | > loss_text_ce: 0.04020870104432106 (0.049085169320674256) | > loss_mel_ce: 2.475177049636841 (nan) | > loss: 0.009981690905988216 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13653403979508047) | > loader_time: 0.0045 (0.004668103786201429)  --> TIME: 2025-02-24 03:59:36 -- STEP: 8648/36938 -- GLOBAL_STEP: 156400 | > loss_text_ce: 0.050961125642061234 (0.04908326157919233) | > loss_mel_ce: 3.8118038177490234 (nan) | > loss: 0.015328432433307171 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13653784008846642) | > loader_time: 0.0045 (0.004667961272125895)  --> TIME: 2025-02-24 03:59:45 -- STEP: 8698/36938 -- GLOBAL_STEP: 156450 | > loss_text_ce: 0.04978364706039429 (0.04908956278215722) | > loss_mel_ce: 3.407756805419922 (nan) | > loss: 0.01372039970010519 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.13655548727246533) | > loader_time: 0.0046 (0.004667508544579806)  --> TIME: 2025-02-24 03:59:54 -- STEP: 8748/36938 -- GLOBAL_STEP: 156500 | > loss_text_ce: 0.04298640042543411 (0.04908787143115622) | > loss_mel_ce: 4.752297878265381 (nan) | > loss: 0.019028907641768456 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.1365713313610308) | > loader_time: 0.0045 (0.00466717867530751)  --> TIME: 2025-02-24 04:00:04 -- STEP: 8798/36938 -- GLOBAL_STEP: 156550 | > loss_text_ce: 0.03946586698293686 (0.049090078911420275) | > loss_mel_ce: 3.1721599102020264 (nan) | > loss: 0.012744547799229622 (nan) | > current_lr: 5e-06 | > step_time: 0.1789 (0.1365554850869468) | > loader_time: 0.0048 (0.004666939245676244)  --> TIME: 2025-02-24 04:00:13 -- STEP: 8848/36938 -- GLOBAL_STEP: 156600 | > loss_text_ce: 0.05911355838179588 (0.04908917190348738) | > loss_mel_ce: 3.7291204929351807 (nan) | > loss: 0.015032675117254257 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.13655097003871294) | > loader_time: 0.0046 (0.004666817312214746)  --> TIME: 2025-02-24 04:00:22 -- STEP: 8898/36938 -- GLOBAL_STEP: 156650 | > loss_text_ce: 0.03856487572193146 (0.049075365822639085) | > loss_mel_ce: 3.2298176288604736 (nan) | > loss: 0.012969772331416607 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13655692324206495) | > loader_time: 0.0045 (0.004666296115857252)  --> TIME: 2025-02-24 04:00:31 -- STEP: 8948/36938 -- GLOBAL_STEP: 156700 | > loss_text_ce: 0.049977414309978485 (0.04907339427388688) | > loss_mel_ce: 3.474546194076538 (nan) | > loss: 0.013986204750835896 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13652992141955608) | > loader_time: 0.0045 (0.0046656365420171516)  --> TIME: 2025-02-24 04:00:41 -- STEP: 8998/36938 -- GLOBAL_STEP: 156750 | > loss_text_ce: 0.045700762420892715 (0.049068136697551765) | > loss_mel_ce: 4.352192401885986 (nan) | > loss: 0.017451956868171692 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.1365321387606591) | > loader_time: 0.0047 (0.004665430823175833)  --> TIME: 2025-02-24 04:00:50 -- STEP: 9048/36938 -- GLOBAL_STEP: 156800 | > loss_text_ce: 0.049488525837659836 (0.04906583721148358) | > loss_mel_ce: 2.1208314895629883 (nan) | > loss: 0.008612381294369698 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13653371441585693) | > loader_time: 0.0045 (0.00466516980231073)  --> TIME: 2025-02-24 04:00:59 -- STEP: 9098/36938 -- GLOBAL_STEP: 156850 | > loss_text_ce: 0.026089763268828392 (0.049048634736118334) | > loss_mel_ce: 2.9340384006500244 (nan) | > loss: 0.011746540665626526 (nan) | > current_lr: 5e-06 | > step_time: 0.108 (0.13650047281176844) | > loader_time: 0.0045 (0.004664850512764761)  --> TIME: 2025-02-24 04:01:08 -- STEP: 9148/36938 -- GLOBAL_STEP: 156900 | > loss_text_ce: 0.04670768603682518 (0.04904690012865165) | > loss_mel_ce: 0.1958867907524109 (nan) | > loss: 0.0009626765386201441 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13648157139037176) | > loader_time: 0.0046 (0.004664562522035853)  --> TIME: 2025-02-24 04:01:17 -- STEP: 9198/36938 -- GLOBAL_STEP: 156950 | > loss_text_ce: 0.05345231294631958 (0.04905071987394439) | > loss_mel_ce: 3.9112143516540527 (nan) | > loss: 0.01573280431330204 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.13645110490089388) | > loader_time: 0.0047 (0.004664112003141459)  --> TIME: 2025-02-24 04:01:26 -- STEP: 9248/36938 -- GLOBAL_STEP: 157000 | > loss_text_ce: 0.05567308887839317 (0.04904956358064371) | > loss_mel_ce: 3.6346681118011475 (nan) | > loss: 0.01464421208947897 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.1364510266374555) | > loader_time: 0.0046 (0.00466390397515678)  --> TIME: 2025-02-24 04:01:36 -- STEP: 9298/36938 -- GLOBAL_STEP: 157050 | > loss_text_ce: 0.05153003707528114 (0.04905465213635201) | > loss_mel_ce: 3.3543245792388916 (nan) | > loss: 0.013515297323465347 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13650709940659492) | > loader_time: 0.0046 (0.004664230818491046)  --> TIME: 2025-02-24 04:01:46 -- STEP: 9348/36938 -- GLOBAL_STEP: 157100 | > loss_text_ce: 0.05187656730413437 (0.049050512452237974) | > loss_mel_ce: 4.404422283172607 (nan) | > loss: 0.0176837258040905 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1365035188264405) | > loader_time: 0.0048 (0.004663683483628176)  --> TIME: 2025-02-24 04:01:55 -- STEP: 9398/36938 -- GLOBAL_STEP: 157150 | > loss_text_ce: 0.04805827885866165 (0.04906082971512656) | > loss_mel_ce: 4.0959272384643555 (nan) | > loss: 0.016444388777017593 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.13652483530058793) | > loader_time: 0.0046 (0.004663263515960324)  --> TIME: 2025-02-24 04:02:04 -- STEP: 9448/36938 -- GLOBAL_STEP: 157200 | > loss_text_ce: 0.04881126433610916 (0.049055332930089804) | > loss_mel_ce: 0.215545654296875 (nan) | > loss: 0.0010490354616194963 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13651133701520723) | > loader_time: 0.0048 (0.004663064028629313)  --> TIME: 2025-02-24 04:02:13 -- STEP: 9498/36938 -- GLOBAL_STEP: 157250 | > loss_text_ce: 0.05083678290247917 (0.049059040352402425) | > loss_mel_ce: 3.334003448486328 (nan) | > loss: 0.013431906700134277 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13649572706594085) | > loader_time: 0.0046 (0.004662772735110544)  --> TIME: 2025-02-24 04:02:23 -- STEP: 9548/36938 -- GLOBAL_STEP: 157300 | > loss_text_ce: 0.041675813496112823 (0.049054205678848085) | > loss_mel_ce: 3.6311094760894775 (nan) | > loss: 0.014574545435607433 (nan) | > current_lr: 5e-06 | > step_time: 0.1527 (0.1364847195413159) | > loader_time: 0.0046 (0.0046625860974935625)  --> TIME: 2025-02-24 04:02:32 -- STEP: 9598/36938 -- GLOBAL_STEP: 157350 | > loss_text_ce: 0.04327784478664398 (0.04906475223708757) | > loss_mel_ce: 3.3654890060424805 (nan) | > loss: 0.013526853173971176 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13648474298434685) | > loader_time: 0.0045 (0.004662260633628811)  --> TIME: 2025-02-24 04:02:41 -- STEP: 9648/36938 -- GLOBAL_STEP: 157400 | > loss_text_ce: 0.04271573945879936 (0.049067690464819264) | > loss_mel_ce: 3.7148783206939697 (nan) | > loss: 0.014911088161170483 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.1364719315399581) | > loader_time: 0.0046 (0.004661916747417431)  --> TIME: 2025-02-24 04:02:50 -- STEP: 9698/36938 -- GLOBAL_STEP: 157450 | > loss_text_ce: 0.045494358986616135 (0.0490756352916267) | > loss_mel_ce: 3.843574047088623 (nan) | > loss: 0.01543281227350235 (nan) | > current_lr: 5e-06 | > step_time: 0.1506 (0.13647545352624302) | > loader_time: 0.0047 (0.004661788864710045)  --> TIME: 2025-02-24 04:03:00 -- STEP: 9748/36938 -- GLOBAL_STEP: 157500 | > loss_text_ce: 0.0566006638109684 (0.04907575817885703) | > loss_mel_ce: 3.2477331161499023 (nan) | > loss: 0.013112436048686504 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13647183023812048) | > loader_time: 0.0045 (0.004661737869741697)  --> TIME: 2025-02-24 04:03:09 -- STEP: 9798/36938 -- GLOBAL_STEP: 157550 | > loss_text_ce: 0.04527397081255913 (0.04907737727060061) | > loss_mel_ce: 3.7345221042633057 (nan) | > loss: 0.014999192208051682 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1364743569890542) | > loader_time: 0.0045 (0.0046615267218461254)  --> TIME: 2025-02-24 04:03:18 -- STEP: 9848/36938 -- GLOBAL_STEP: 157600 | > loss_text_ce: 0.04175035282969475 (0.04906881874984014) | > loss_mel_ce: 2.750117540359497 (nan) | > loss: 0.011078841984272003 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.1364700300096986) | > loader_time: 0.0045 (0.004661388338144787)  --> TIME: 2025-02-24 04:03:27 -- STEP: 9898/36938 -- GLOBAL_STEP: 157650 | > loss_text_ce: 0.050954073667526245 (0.049075286420298056) | > loss_mel_ce: 3.3815360069274902 (nan) | > loss: 0.013620993122458458 (nan) | > current_lr: 5e-06 | > step_time: 0.1069 (0.13646067118591776) | > loader_time: 0.0046 (0.0046610674922648076)  --> TIME: 2025-02-24 04:03:37 -- STEP: 9948/36938 -- GLOBAL_STEP: 157700 | > loss_text_ce: 0.03828071430325508 (0.04907506598840454) | > loss_mel_ce: 2.6137826442718506 (nan) | > loss: 0.010524061508476734 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13644589678465693) | > loader_time: 0.0046 (0.004661092448531762)  --> TIME: 2025-02-24 04:03:46 -- STEP: 9998/36938 -- GLOBAL_STEP: 157750 | > loss_text_ce: 0.053730230778455734 (0.04907297206568237) | > loss_mel_ce: 3.71990966796875 (nan) | > loss: 0.014974762685596943 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13644478854763029) | > loader_time: 0.0045 (0.004660794295699978)  --> TIME: 2025-02-24 04:03:55 -- STEP: 10048/36938 -- GLOBAL_STEP: 157800 | > loss_text_ce: 0.04907476156949997 (0.04907377209561821) | > loss_mel_ce: 3.397850513458252 (nan) | > loss: 0.013678275048732758 (nan) | > current_lr: 5e-06 | > step_time: 0.1828 (0.1364426795796611) | > loader_time: 0.0048 (0.004660455118508874)  --> TIME: 2025-02-24 04:04:05 -- STEP: 10098/36938 -- GLOBAL_STEP: 157850 | > loss_text_ce: 0.041953958570957184 (0.04908042883725113) | > loss_mel_ce: 3.8521313667297363 (nan) | > loss: 0.015452721156179905 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13646327110535383) | > loader_time: 0.0048 (0.004660166615565304)  --> TIME: 2025-02-24 04:04:14 -- STEP: 10148/36938 -- GLOBAL_STEP: 157900 | > loss_text_ce: 0.04659991338849068 (0.04906925064161743) | > loss_mel_ce: 3.5319454669952393 (nan) | > loss: 0.014200577512383461 (nan) | > current_lr: 5e-06 | > step_time: 0.1519 (0.13644938629633122) | > loader_time: 0.0046 (0.004659685037903885)  --> TIME: 2025-02-24 04:04:23 -- STEP: 10198/36938 -- GLOBAL_STEP: 157950 | > loss_text_ce: 0.037098757922649384 (0.04906051231237313) | > loss_mel_ce: 0.20220114290714264 (nan) | > loss: 0.0009496028069406748 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13642614176283224) | > loader_time: 0.0047 (0.004659238808948565)  --> TIME: 2025-02-24 04:04:32 -- STEP: 10248/36938 -- GLOBAL_STEP: 158000 | > loss_text_ce: 0.05970855429768562 (0.04906376312337052) | > loss_mel_ce: 0.9644213914871216 (nan) | > loss: 0.004064008127897978 (nan) | > current_lr: 5e-06 | > step_time: 0.1072 (0.1364252575433977) | > loader_time: 0.0045 (0.004658827899862901)  --> TIME: 2025-02-24 04:04:42 -- STEP: 10298/36938 -- GLOBAL_STEP: 158050 | > loss_text_ce: 0.04524529352784157 (0.04905987280851614) | > loss_mel_ce: 3.0003662109375 (nan) | > loss: 0.012085760943591595 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13645310836246785) | > loader_time: 0.0045 (0.004658534795850401)  --> TIME: 2025-02-24 04:04:51 -- STEP: 10348/36938 -- GLOBAL_STEP: 158100 | > loss_text_ce: 0.035526297986507416 (0.04905760234490371) | > loss_mel_ce: 3.081019878387451 (nan) | > loss: 0.012367247603833675 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.1364368265505512) | > loader_time: 0.0047 (0.004658155820062844)  --> TIME: 2025-02-24 04:05:00 -- STEP: 10398/36938 -- GLOBAL_STEP: 158150 | > loss_text_ce: 0.057343415915966034 (0.04905044976181191) | > loss_mel_ce: 3.4012248516082764 (nan) | > loss: 0.013724477961659431 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.1364538228656272) | > loader_time: 0.0045 (0.004657806972284289)  --> TIME: 2025-02-24 04:05:10 -- STEP: 10448/36938 -- GLOBAL_STEP: 158200 | > loss_text_ce: 0.052487824112176895 (0.049049732694616224) | > loss_mel_ce: 3.1124281883239746 (nan) | > loss: 0.012559191323816776 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13646134147691574) | > loader_time: 0.0045 (0.004657482137906437)  --> TIME: 2025-02-24 04:05:19 -- STEP: 10498/36938 -- GLOBAL_STEP: 158250 | > loss_text_ce: 0.04200298339128494 (0.04904983103222776) | > loss_mel_ce: 3.33870005607605 (nan) | > loss: 0.013415488414466381 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1364478173540267) | > loader_time: 0.0045 (0.00465703226312227)  --> TIME: 2025-02-24 04:05:29 -- STEP: 10548/36938 -- GLOBAL_STEP: 158300 | > loss_text_ce: 0.04750736430287361 (0.049054971076745835) | > loss_mel_ce: 3.41288423538208 (nan) | > loss: 0.013731713406741619 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13646815430724799) | > loader_time: 0.0046 (0.004656755770523093)  --> TIME: 2025-02-24 04:05:38 -- STEP: 10598/36938 -- GLOBAL_STEP: 158350 | > loss_text_ce: 0.05551061034202576 (0.04905310563094928) | > loss_mel_ce: 3.68112850189209 (nan) | > loss: 0.01482793316245079 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13646473804044904) | > loader_time: 0.0045 (0.004656373318421853)  --> TIME: 2025-02-24 04:05:47 -- STEP: 10648/36938 -- GLOBAL_STEP: 158400 | > loss_text_ce: 0.058097634464502335 (0.04904457377962579) | > loss_mel_ce: 4.098283290863037 (nan) | > loss: 0.016493577510118484 (nan) | > current_lr: 5e-06 | > step_time: 0.1473 (0.13646250161615994) | > loader_time: 0.0047 (0.00465632046997593)  --> TIME: 2025-02-24 04:05:57 -- STEP: 10698/36938 -- GLOBAL_STEP: 158450 | > loss_text_ce: 0.05211861804127693 (0.04904952431353567) | > loss_mel_ce: 3.082249641418457 (nan) | > loss: 0.012437969446182251 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13646387227073284) | > loader_time: 0.0045 (0.004656281219863262)  --> TIME: 2025-02-24 04:06:06 -- STEP: 10748/36938 -- GLOBAL_STEP: 158500 | > loss_text_ce: 0.05420922115445137 (0.04904206847244468) | > loss_mel_ce: 0.02099609375 (nan) | > loss: 0.00029843379161320627 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13647359558583982) | > loader_time: 0.0046 (0.004656053583196971)  --> TIME: 2025-02-24 04:06:15 -- STEP: 10798/36938 -- GLOBAL_STEP: 158550 | > loss_text_ce: 0.04879980534315109 (0.049038868617238086) | > loss_mel_ce: 2.032428503036499 (nan) | > loss: 0.00825884286314249 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13649070945177394) | > loader_time: 0.0048 (0.004655880869748668)  --> TIME: 2025-02-24 04:06:25 -- STEP: 10848/36938 -- GLOBAL_STEP: 158600 | > loss_text_ce: 0.050028905272483826 (0.049038084965971174) | > loss_mel_ce: 3.372159004211426 (nan) | > loss: 0.013580110855400562 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1364980616049083) | > loader_time: 0.0045 (0.004655837353351907)  --> TIME: 2025-02-24 04:06:34 -- STEP: 10898/36938 -- GLOBAL_STEP: 158650 | > loss_text_ce: 0.03985898196697235 (0.04903323522323332) | > loss_mel_ce: 0.02738443948328495 (nan) | > loss: 0.0002668389934115112 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13649161817131011) | > loader_time: 0.0046 (0.004655555642664082)  --> TIME: 2025-02-24 04:06:43 -- STEP: 10948/36938 -- GLOBAL_STEP: 158700 | > loss_text_ce: 0.052046336233615875 (0.049034137090284896) | > loss_mel_ce: 3.6707823276519775 (nan) | > loss: 0.014773130416870117 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13650118705779535) | > loader_time: 0.0046 (0.004655507867859807)  --> TIME: 2025-02-24 04:06:53 -- STEP: 10998/36938 -- GLOBAL_STEP: 158750 | > loss_text_ce: 0.027984675019979477 (0.04903644264512577) | > loss_mel_ce: 3.2170839309692383 (nan) | > loss: 0.012877256609499454 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.1365083932096171) | > loader_time: 0.0045 (0.004655602087127366)  --> TIME: 2025-02-24 04:07:02 -- STEP: 11048/36938 -- GLOBAL_STEP: 158800 | > loss_text_ce: 0.059640053659677505 (0.04903435869787119) | > loss_mel_ce: 3.9359898567199707 (nan) | > loss: 0.01585567556321621 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13649863829049957) | > loader_time: 0.0045 (0.00465552911302649)  --> TIME: 2025-02-24 04:07:11 -- STEP: 11098/36938 -- GLOBAL_STEP: 158850 | > loss_text_ce: 0.03779905289411545 (0.049036582086317816) | > loss_mel_ce: 3.8522257804870605 (nan) | > loss: 0.015436607412993908 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13649230270605764) | > loader_time: 0.0046 (0.004655252793690701)  --> TIME: 2025-02-24 04:07:21 -- STEP: 11148/36938 -- GLOBAL_STEP: 158900 | > loss_text_ce: 0.05844326689839363 (0.049038062809462206) | > loss_mel_ce: 3.4289443492889404 (nan) | > loss: 0.013838840648531914 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13648114550443075) | > loader_time: 0.0065 (0.004655087597275203)  --> TIME: 2025-02-24 04:07:30 -- STEP: 11198/36938 -- GLOBAL_STEP: 158950 | > loss_text_ce: 0.049335114657878876 (0.04903640163036741) | > loss_mel_ce: 3.587275743484497 (nan) | > loss: 0.014430996030569077 (nan) | > current_lr: 5e-06 | > step_time: 0.1765 (0.1364837585336124) | > loader_time: 0.0047 (0.004654780437444286)  --> TIME: 2025-02-24 04:07:40 -- STEP: 11248/36938 -- GLOBAL_STEP: 159000 | > loss_text_ce: 0.0637260228395462 (0.049037717270020986) | > loss_mel_ce: 3.6744298934936523 (nan) | > loss: 0.01483395230025053 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.13651597773282706) | > loader_time: 0.0048 (0.004654956237011598)  --> TIME: 2025-02-24 04:07:49 -- STEP: 11298/36938 -- GLOBAL_STEP: 159050 | > loss_text_ce: 0.047050684690475464 (0.049037101191538895) | > loss_mel_ce: 2.9894845485687256 (nan) | > loss: 0.01204974390566349 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13649842979254267) | > loader_time: 0.0045 (0.004654576660658679)  --> TIME: 2025-02-24 04:07:58 -- STEP: 11348/36938 -- GLOBAL_STEP: 159100 | > loss_text_ce: 0.04434549808502197 (0.049039844044208074) | > loss_mel_ce: 4.09009313583374 (nan) | > loss: 0.016406502574682236 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13649806116599644) | > loader_time: 0.0046 (0.004654288691116314)  --> TIME: 2025-02-24 04:08:07 -- STEP: 11398/36938 -- GLOBAL_STEP: 159150 | > loss_text_ce: 0.03670882061123848 (0.049034096183430784) | > loss_mel_ce: 0.01904296875 (nan) | > loss: 0.00022123727831058204 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1365086205446173) | > loader_time: 0.0045 (0.004654201714066728)  --> TIME: 2025-02-24 04:08:17 -- STEP: 11448/36938 -- GLOBAL_STEP: 159200 | > loss_text_ce: 0.04602200537919998 (0.049036160353369385) | > loss_mel_ce: 4.124030113220215 (nan) | > loss: 0.016547827050089836 (nan) | > current_lr: 5e-06 | > step_time: 0.1667 (0.13650269332588547) | > loader_time: 0.0046 (0.004654039460241518)  --> TIME: 2025-02-24 04:08:26 -- STEP: 11498/36938 -- GLOBAL_STEP: 159250 | > loss_text_ce: 0.06554563343524933 (0.0490367503569131) | > loss_mel_ce: 3.515277862548828 (nan) | > loss: 0.014209617860615253 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13651804244711155) | > loader_time: 0.0045 (0.004653789703069238)  --> TIME: 2025-02-24 04:08:36 -- STEP: 11548/36938 -- GLOBAL_STEP: 159300 | > loss_text_ce: 0.05295475199818611 (0.04903328849230465) | > loss_mel_ce: 3.7173476219177246 (nan) | > loss: 0.014961518347263336 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.1365383273389956) | > loader_time: 0.0046 (0.004653874734383712)  --> TIME: 2025-02-24 04:08:45 -- STEP: 11598/36938 -- GLOBAL_STEP: 159350 | > loss_text_ce: 0.047328054904937744 (0.0490338974821028) | > loss_mel_ce: 4.231867790222168 (nan) | > loss: 0.016980936750769615 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.13653585419076827) | > loader_time: 0.0048 (0.004653814661644691)  --> TIME: 2025-02-24 04:08:55 -- STEP: 11648/36938 -- GLOBAL_STEP: 159400 | > loss_text_ce: 0.045860160142183304 (0.049031519108441626) | > loss_mel_ce: 3.672868013381958 (nan) | > loss: 0.014756858348846436 (nan) | > current_lr: 5e-06 | > step_time: 0.1886 (0.13655232709760884) | > loader_time: 0.005 (0.004653735393351262)  --> TIME: 2025-02-24 04:09:04 -- STEP: 11698/36938 -- GLOBAL_STEP: 159450 | > loss_text_ce: 0.05630270391702652 (0.049025136691461246) | > loss_mel_ce: 3.2146105766296387 (nan) | > loss: 0.01297981571406126 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.13655976155493557) | > loader_time: 0.005 (0.00465398228949086)  --> TIME: 2025-02-24 04:09:13 -- STEP: 11748/36938 -- GLOBAL_STEP: 159500 | > loss_text_ce: 0.039338648319244385 (0.04902831280231726) | > loss_mel_ce: 2.7184460163116455 (nan) | > loss: 0.01094359066337347 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13656908442140447) | > loader_time: 0.0049 (0.004653774011610003)  --> TIME: 2025-02-24 04:09:22 -- STEP: 11798/36938 -- GLOBAL_STEP: 159550 | > loss_text_ce: 0.041145551949739456 (0.04902645438871005) | > loss_mel_ce: 2.8748972415924072 (nan) | > loss: 0.011571599170565605 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13655923398637607) | > loader_time: 0.0045 (0.004653509824351564)  --> TIME: 2025-02-24 04:09:32 -- STEP: 11848/36938 -- GLOBAL_STEP: 159600 | > loss_text_ce: 0.055171217769384384 (0.04902719126326946) | > loss_mel_ce: 3.8403842449188232 (nan) | > loss: 0.015458554029464722 (nan) | > current_lr: 5e-06 | > step_time: 0.1082 (0.1365744871798603) | > loader_time: 0.0045 (0.004653373897920838)  --> TIME: 2025-02-24 04:09:41 -- STEP: 11898/36938 -- GLOBAL_STEP: 159650 | > loss_text_ce: 0.04529031738638878 (0.049022444960805066) | > loss_mel_ce: 3.6487550735473633 (nan) | > loss: 0.014658911153674126 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13655371033939012) | > loader_time: 0.0046 (0.004653581592651524)  --> TIME: 2025-02-24 04:09:50 -- STEP: 11948/36938 -- GLOBAL_STEP: 159700 | > loss_text_ce: 0.05106665566563606 (0.04901776129612462) | > loss_mel_ce: 2.007488489151001 (nan) | > loss: 0.008168869651854038 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.1365628213757935) | > loader_time: 0.0048 (0.004653558908276718)  --> TIME: 2025-02-24 04:09:59 -- STEP: 11998/36938 -- GLOBAL_STEP: 159750 | > loss_text_ce: 0.046417031437158585 (0.04901733445439867) | > loss_mel_ce: 3.5538556575775146 (nan) | > loss: 0.014286797493696213 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.1365445493201179) | > loader_time: 0.0045 (0.004653311367928346)  --> TIME: 2025-02-24 04:10:09 -- STEP: 12048/36938 -- GLOBAL_STEP: 159800 | > loss_text_ce: 0.050799883902072906 (0.04902081230654072) | > loss_mel_ce: 3.7216358184814453 (nan) | > loss: 0.014969984069466591 (nan) | > current_lr: 5e-06 | > step_time: 0.1488 (0.13654452018133095) | > loader_time: 0.0046 (0.0046531002953708045)  --> TIME: 2025-02-24 04:10:18 -- STEP: 12098/36938 -- GLOBAL_STEP: 159850 | > loss_text_ce: 0.0638398751616478 (0.049026352494270196) | > loss_mel_ce: 3.7756617069244385 (nan) | > loss: 0.01523611880838871 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13651025985012152) | > loader_time: 0.0045 (0.004652673221852377)  --> TIME: 2025-02-24 04:10:27 -- STEP: 12148/36938 -- GLOBAL_STEP: 159900 | > loss_text_ce: 0.043371181935071945 (0.04902446864492964) | > loss_mel_ce: 3.5844531059265137 (nan) | > loss: 0.014396129176020622 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13649733625119387) | > loader_time: 0.0048 (0.004652315509684355)  --> TIME: 2025-02-24 04:10:36 -- STEP: 12198/36938 -- GLOBAL_STEP: 159950 | > loss_text_ce: 0.04446995630860329 (0.04901467620911707) | > loss_mel_ce: 4.456960201263428 (nan) | > loss: 0.017862819135189056 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13647550910315556) | > loader_time: 0.0046 (0.00465223724323329)  --> TIME: 2025-02-24 04:10:45 -- STEP: 12248/36938 -- GLOBAL_STEP: 160000 | > loss_text_ce: 0.05166350305080414 (0.04900432126806433) | > loss_mel_ce: 2.202042818069458 (nan) | > loss: 0.008943279273808002 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13648125147757292) | > loader_time: 0.0046 (0.004652238005053221) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_160000.pth  --> TIME: 2025-02-24 04:11:00 -- STEP: 12298/36938 -- GLOBAL_STEP: 160050 | > loss_text_ce: 0.05312233790755272 (0.04899881411238394) | > loss_mel_ce: 4.0286784172058105 (nan) | > loss: 0.016197623685002327 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13645252789456958) | > loader_time: 0.0047 (0.004652215767961392)  --> TIME: 2025-02-24 04:11:09 -- STEP: 12348/36938 -- GLOBAL_STEP: 160100 | > loss_text_ce: 0.05842765048146248 (0.04899652643538923) | > loss_mel_ce: 2.3593993186950684 (nan) | > loss: 0.009594551287591457 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13645184349002032) | > loader_time: 0.0046 (0.004652206029635567)  --> TIME: 2025-02-24 04:11:19 -- STEP: 12398/36938 -- GLOBAL_STEP: 160150 | > loss_text_ce: 0.06584601104259491 (0.04898367281765785) | > loss_mel_ce: 3.8449721336364746 (nan) | > loss: 0.01551912073045969 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13645887992174052) | > loader_time: 0.0048 (0.004652124755823701)  --> TIME: 2025-02-24 04:11:28 -- STEP: 12448/36938 -- GLOBAL_STEP: 160200 | > loss_text_ce: 0.02906510978937149 (0.0489863160935381) | > loss_mel_ce: 2.4989869594573975 (nan) | > loss: 0.01003195345401764 (nan) | > current_lr: 5e-06 | > step_time: 0.106 (0.13643910189980482) | > loader_time: 0.0046 (0.004652349781101363)  --> TIME: 2025-02-24 04:11:37 -- STEP: 12498/36938 -- GLOBAL_STEP: 160250 | > loss_text_ce: 0.050136107951402664 (0.04898366314318915) | > loss_mel_ce: 3.8404273986816406 (nan) | > loss: 0.01543874479830265 (nan) | > current_lr: 5e-06 | > step_time: 0.1434 (0.13642089978125216) | > loader_time: 0.0047 (0.0046526198197143825)  --> TIME: 2025-02-24 04:11:46 -- STEP: 12548/36938 -- GLOBAL_STEP: 160300 | > loss_text_ce: 0.056045226752758026 (0.048979251473164696) | > loss_mel_ce: 2.7989249229431152 (nan) | > loss: 0.011329247616231441 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13640653832459704) | > loader_time: 0.0047 (0.004652615846780458)  --> TIME: 2025-02-24 04:11:55 -- STEP: 12598/36938 -- GLOBAL_STEP: 160350 | > loss_text_ce: 0.050470203161239624 (0.04897956061069759) | > loss_mel_ce: 3.859952211380005 (nan) | > loss: 0.015517549589276314 (nan) | > current_lr: 5e-06 | > step_time: 0.1532 (0.13639857324650873) | > loader_time: 0.0048 (0.004652720062406065)  --> TIME: 2025-02-24 04:12:04 -- STEP: 12648/36938 -- GLOBAL_STEP: 160400 | > loss_text_ce: 0.05128498747944832 (0.04897684313944178) | > loss_mel_ce: 2.0579845905303955 (nan) | > loss: 0.008370118215680122 (nan) | > current_lr: 5e-06 | > step_time: 0.1571 (0.13640589713296367) | > loader_time: 0.0046 (0.004652511161162397)  --> TIME: 2025-02-24 04:12:14 -- STEP: 12698/36938 -- GLOBAL_STEP: 160450 | > loss_text_ce: 0.048476219177246094 (0.04896858953227602) | > loss_mel_ce: 4.134194374084473 (nan) | > loss: 0.016597900539636612 (nan) | > current_lr: 5e-06 | > step_time: 0.2017 (0.13641819170580777) | > loader_time: 0.0048 (0.004652381337595896)  --> TIME: 2025-02-24 04:12:24 -- STEP: 12748/36938 -- GLOBAL_STEP: 160500 | > loss_text_ce: 0.049260709434747696 (0.04896818856209873) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.1364553893364305) | > loader_time: 0.0047 (0.004652365850437254)  --> TIME: 2025-02-24 04:12:33 -- STEP: 12798/36938 -- GLOBAL_STEP: 160550 | > loss_text_ce: 0.042489826679229736 (0.04896881046164547) | > loss_mel_ce: 3.878525972366333 (nan) | > loss: 0.015559587627649307 (nan) | > current_lr: 5e-06 | > step_time: 0.1773 (0.13648247601446156) | > loader_time: 0.0047 (0.0046522937765120145)  --> TIME: 2025-02-24 04:12:43 -- STEP: 12848/36938 -- GLOBAL_STEP: 160600 | > loss_text_ce: 0.046623408794403076 (0.04895873808004492) | > loss_mel_ce: 3.665076732635498 (nan) | > loss: 0.014728969894349575 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13648493684466514) | > loader_time: 0.0046 (0.004652183609614502)  --> TIME: 2025-02-24 04:12:52 -- STEP: 12898/36938 -- GLOBAL_STEP: 160650 | > loss_text_ce: 0.059216197580099106 (0.0489573666033349) | > loss_mel_ce: 4.345417022705078 (nan) | > loss: 0.01747870445251465 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.1364899994440098) | > loader_time: 0.0046 (0.00465206668106675)  --> TIME: 2025-02-24 04:13:02 -- STEP: 12948/36938 -- GLOBAL_STEP: 160700 | > loss_text_ce: 0.048209838569164276 (0.04895553160945032) | > loss_mel_ce: 3.3988451957702637 (nan) | > loss: 0.013678791001439095 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13649607384171805) | > loader_time: 0.0045 (0.0046520280108836515)  --> TIME: 2025-02-24 04:13:11 -- STEP: 12998/36938 -- GLOBAL_STEP: 160750 | > loss_text_ce: 0.04520261287689209 (0.04895308634283737) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13650159930097056) | > loader_time: 0.0045 (0.004651861422647806)  --> TIME: 2025-02-24 04:13:20 -- STEP: 13048/36938 -- GLOBAL_STEP: 160800 | > loss_text_ce: 0.04696742445230484 (0.048958425452204066) | > loss_mel_ce: 0.0814615860581398 (nan) | > loss: 0.0005096389795653522 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13648997034345436) | > loader_time: 0.0047 (0.004651638973278202)  --> TIME: 2025-02-24 04:13:29 -- STEP: 13098/36938 -- GLOBAL_STEP: 160850 | > loss_text_ce: 0.049158815294504166 (0.04895324008031632) | > loss_mel_ce: 3.795306921005249 (nan) | > loss: 0.015255817212164402 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13649175399713945) | > loader_time: 0.0046 (0.0046515465146768475)  --> TIME: 2025-02-24 04:13:39 -- STEP: 13148/36938 -- GLOBAL_STEP: 160900 | > loss_text_ce: 0.04850609600543976 (0.04894996531449928) | > loss_mel_ce: 2.984722137451172 (nan) | > loss: 0.012036620639264584 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13650278153664747) | > loader_time: 0.0046 (0.004651583125970646)  --> TIME: 2025-02-24 04:13:48 -- STEP: 13198/36938 -- GLOBAL_STEP: 160950 | > loss_text_ce: 0.05152055248618126 (0.04894740089767354) | > loss_mel_ce: 3.6426408290863037 (nan) | > loss: 0.014659371227025986 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13649585615922094) | > loader_time: 0.0047 (0.004651444918243336)  --> TIME: 2025-02-24 04:13:57 -- STEP: 13248/36938 -- GLOBAL_STEP: 161000 | > loss_text_ce: 0.057545460760593414 (0.04894917410464306) | > loss_mel_ce: 3.909489393234253 (nan) | > loss: 0.01574220322072506 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.1364962545533984) | > loader_time: 0.0049 (0.004651600072061401)  --> TIME: 2025-02-24 04:14:07 -- STEP: 13298/36938 -- GLOBAL_STEP: 161050 | > loss_text_ce: 0.04017670452594757 (0.04894568863625827) | > loss_mel_ce: 3.714250326156616 (nan) | > loss: 0.014898520894348621 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1365148874669537) | > loader_time: 0.0046 (0.004651730464694927)  --> TIME: 2025-02-24 04:14:16 -- STEP: 13348/36938 -- GLOBAL_STEP: 161100 | > loss_text_ce: 0.046787843108177185 (0.048938117758363923) | > loss_mel_ce: 4.183676719665527 (nan) | > loss: 0.016787558794021606 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.136513679531974) | > loader_time: 0.0047 (0.004651659632429326)  --> TIME: 2025-02-24 04:14:26 -- STEP: 13398/36938 -- GLOBAL_STEP: 161150 | > loss_text_ce: 0.05935855209827423 (0.0489339669521301) | > loss_mel_ce: 3.5522775650024414 (nan) | > loss: 0.014331890270113945 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13652153978918724) | > loader_time: 0.0045 (0.004651684692714244)  --> TIME: 2025-02-24 04:14:35 -- STEP: 13448/36938 -- GLOBAL_STEP: 161200 | > loss_text_ce: 0.045973729342222214 (0.04893425934596769) | > loss_mel_ce: 3.7579941749572754 (nan) | > loss: 0.015095111913979053 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13651393159807415) | > loader_time: 0.0046 (0.004651650688890773)  --> TIME: 2025-02-24 04:14:44 -- STEP: 13498/36938 -- GLOBAL_STEP: 161250 | > loss_text_ce: 0.0451539121568203 (0.048936104836903624) | > loss_mel_ce: 2.3603971004486084 (nan) | > loss: 0.00954583752900362 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13652128780448164) | > loader_time: 0.0046 (0.004651506665300894)  --> TIME: 2025-02-24 04:14:53 -- STEP: 13548/36938 -- GLOBAL_STEP: 161300 | > loss_text_ce: 0.03728405758738518 (0.04892796316660623) | > loss_mel_ce: 1.370554804801941 (nan) | > loss: 0.005586662329733372 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13649488880390126) | > loader_time: 0.0045 (0.00465146978590601)  --> TIME: 2025-02-24 04:15:03 -- STEP: 13598/36938 -- GLOBAL_STEP: 161350 | > loss_text_ce: 0.05661071836948395 (0.048922256138112685) | > loss_mel_ce: 4.298346996307373 (nan) | > loss: 0.017281578853726387 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13650037907803728) | > loader_time: 0.0045 (0.004651430126919035)  --> TIME: 2025-02-24 04:15:12 -- STEP: 13648/36938 -- GLOBAL_STEP: 161400 | > loss_text_ce: 0.04211816564202309 (0.048918324489373534) | > loss_mel_ce: 4.302589416503906 (nan) | > loss: 0.01724090427160263 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13650002822297466) | > loader_time: 0.0045 (0.004651273697929138)  --> TIME: 2025-02-24 04:15:21 -- STEP: 13698/36938 -- GLOBAL_STEP: 161450 | > loss_text_ce: 0.06165862828493118 (0.0489138843828086) | > loss_mel_ce: 3.2180638313293457 (nan) | > loss: 0.013014771975576878 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.1365148872855048) | > loader_time: 0.0046 (0.004651170418129879)  --> TIME: 2025-02-24 04:15:31 -- STEP: 13748/36938 -- GLOBAL_STEP: 161500 | > loss_text_ce: 0.036281805485486984 (0.0489148967522734) | > loss_mel_ce: 0.7119395136833191 (nan) | > loss: 0.0029691325034946203 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.1365104389461218) | > loader_time: 0.0067 (0.004651335130641051)  --> TIME: 2025-02-24 04:15:40 -- STEP: 13798/36938 -- GLOBAL_STEP: 161550 | > loss_text_ce: 0.04250313341617584 (0.048916618565427075) | > loss_mel_ce: 0.6242268681526184 (nan) | > loss: 0.002645754022523761 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13650048276724666) | > loader_time: 0.0045 (0.0046513580830136484)  --> TIME: 2025-02-24 04:15:49 -- STEP: 13848/36938 -- GLOBAL_STEP: 161600 | > loss_text_ce: 0.04853295162320137 (0.048915023274546436) | > loss_mel_ce: 3.131523609161377 (nan) | > loss: 0.01261927280575037 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13648105055042206) | > loader_time: 0.0045 (0.0046510822093948434)  --> TIME: 2025-02-24 04:15:58 -- STEP: 13898/36938 -- GLOBAL_STEP: 161650 | > loss_text_ce: 0.049346037209033966 (0.04891632522761514) | > loss_mel_ce: 3.4683926105499268 (nan) | > loss: 0.013959281146526337 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.1365035054391718) | > loader_time: 0.0047 (0.00465112240953677)  --> TIME: 2025-02-24 04:16:08 -- STEP: 13948/36938 -- GLOBAL_STEP: 161700 | > loss_text_ce: 0.03597961366176605 (0.048917018538802946) | > loss_mel_ce: 3.0231637954711914 (nan) | > loss: 0.0121394582092762 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.1364871843653633) | > loader_time: 0.0047 (0.0046509407569660126)  --> TIME: 2025-02-24 04:16:17 -- STEP: 13998/36938 -- GLOBAL_STEP: 161750 | > loss_text_ce: 0.04582642763853073 (0.04891345518263133) | > loss_mel_ce: 3.579627513885498 (nan) | > loss: 0.014386722818017006 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13647297568484137) | > loader_time: 0.0046 (0.004650805231196033)  --> TIME: 2025-02-24 04:16:26 -- STEP: 14048/36938 -- GLOBAL_STEP: 161800 | > loss_text_ce: 0.039540186524391174 (0.048912564371972354) | > loss_mel_ce: 3.516493320465088 (nan) | > loss: 0.014111245051026344 (nan) | > current_lr: 5e-06 | > step_time: 0.1452 (0.1364584432953048) | > loader_time: 0.0047 (0.004650813588927733)  --> TIME: 2025-02-24 04:16:35 -- STEP: 14098/36938 -- GLOBAL_STEP: 161850 | > loss_text_ce: 0.04692262038588524 (0.048900375776214804) | > loss_mel_ce: 3.1195967197418213 (nan) | > loss: 0.01256555411964655 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13645485073543742) | > loader_time: 0.0046 (0.004650780352688413)  --> TIME: 2025-02-24 04:16:44 -- STEP: 14148/36938 -- GLOBAL_STEP: 161900 | > loss_text_ce: 0.055485017597675323 (0.0489000055812834) | > loss_mel_ce: 3.9798848628997803 (nan) | > loss: 0.01601337268948555 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13643459340829822) | > loader_time: 0.0046 (0.00465073226904986)  --> TIME: 2025-02-24 04:16:54 -- STEP: 14198/36938 -- GLOBAL_STEP: 161950 | > loss_text_ce: 0.03961345925927162 (0.04890088943714974) | > loss_mel_ce: 3.461557626724243 (nan) | > loss: 0.01389353722333908 (nan) | > current_lr: 5e-06 | > step_time: 0.1869 (0.13646034849145225) | > loader_time: 0.0048 (0.0046507675625234)  --> TIME: 2025-02-24 04:17:03 -- STEP: 14248/36938 -- GLOBAL_STEP: 162000 | > loss_text_ce: 0.03554810583591461 (0.048890670552036074) | > loss_mel_ce: 3.6178090572357178 (nan) | > loss: 0.014497450552880764 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13645994817186638) | > loader_time: 0.0045 (0.004650481526333608)  --> TIME: 2025-02-24 04:17:12 -- STEP: 14298/36938 -- GLOBAL_STEP: 162050 | > loss_text_ce: 0.061636194586753845 (0.04888782055678794) | > loss_mel_ce: 4.095163345336914 (nan) | > loss: 0.01649523712694645 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.136450154912408) | > loader_time: 0.0045 (0.00465017477938107)  --> TIME: 2025-02-24 04:17:22 -- STEP: 14348/36938 -- GLOBAL_STEP: 162100 | > loss_text_ce: 0.044521354138851166 (0.04889106674089709) | > loss_mel_ce: 3.4137229919433594 (nan) | > loss: 0.013723192736506462 (nan) | > current_lr: 5e-06 | > step_time: 0.1599 (0.13647490738160842) | > loader_time: 0.0046 (0.004650129758783283)  --> TIME: 2025-02-24 04:17:31 -- STEP: 14398/36938 -- GLOBAL_STEP: 162150 | > loss_text_ce: 0.048491332679986954 (0.048887558343647795) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13647632327969478) | > loader_time: 0.0047 (0.00465023063888054)  --> TIME: 2025-02-24 04:17:41 -- STEP: 14448/36938 -- GLOBAL_STEP: 162200 | > loss_text_ce: 0.051567886024713516 (0.048883880660683506) | > loss_mel_ce: 3.718580722808838 (nan) | > loss: 0.014960907399654388 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.1364905347063708) | > loader_time: 0.0046 (0.004650053523835704)  --> TIME: 2025-02-24 04:17:50 -- STEP: 14498/36938 -- GLOBAL_STEP: 162250 | > loss_text_ce: 0.051978230476379395 (0.04888718564639908) | > loss_mel_ce: 3.9412269592285156 (nan) | > loss: 0.015846053138375282 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.1364948895079872) | > loader_time: 0.0046 (0.0046499224099838185)  --> TIME: 2025-02-24 04:18:00 -- STEP: 14548/36938 -- GLOBAL_STEP: 162300 | > loss_text_ce: 0.051931124180555344 (0.04888044081982719) | > loss_mel_ce: 4.039793491363525 (nan) | > loss: 0.01623700186610222 (nan) | > current_lr: 5e-06 | > step_time: 0.1583 (0.13650702835276665) | > loader_time: 0.0046 (0.004649684509144161)  --> TIME: 2025-02-24 04:18:09 -- STEP: 14598/36938 -- GLOBAL_STEP: 162350 | > loss_text_ce: 0.037556976079940796 (0.048874331124515794) | > loss_mel_ce: 0.8939366936683655 (nan) | > loss: 0.0036964034661650658 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13650513683415397) | > loader_time: 0.0045 (0.004649806388487069)  --> TIME: 2025-02-24 04:18:19 -- STEP: 14648/36938 -- GLOBAL_STEP: 162400 | > loss_text_ce: 0.0475204735994339 (0.04887479867512517) | > loss_mel_ce: 3.1629135608673096 (nan) | > loss: 0.01273981761187315 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13652763638191848) | > loader_time: 0.0047 (0.004650048663354476)  --> TIME: 2025-02-24 04:18:28 -- STEP: 14698/36938 -- GLOBAL_STEP: 162450 | > loss_text_ce: 0.05356239527463913 (0.04888006567738056) | > loss_mel_ce: 3.9097630977630615 (nan) | > loss: 0.015727482736110687 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1365239190996916) | > loader_time: 0.0046 (0.004649824294156345)  --> TIME: 2025-02-24 04:18:37 -- STEP: 14748/36938 -- GLOBAL_STEP: 162500 | > loss_text_ce: 0.04946010187268257 (0.048874444862994845) | > loss_mel_ce: 3.507458209991455 (nan) | > loss: 0.014114756137132645 (nan) | > current_lr: 5e-06 | > step_time: 0.1896 (0.13652494797902043) | > loader_time: 0.0051 (0.004649692882132287)  --> TIME: 2025-02-24 04:18:47 -- STEP: 14798/36938 -- GLOBAL_STEP: 162550 | > loss_text_ce: 0.05322899669408798 (0.04887135579561778) | > loss_mel_ce: 4.045472145080566 (nan) | > loss: 0.016264688223600388 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.13652480252773028) | > loader_time: 0.0045 (0.0046495510639572935)  --> TIME: 2025-02-24 04:18:56 -- STEP: 14848/36938 -- GLOBAL_STEP: 162600 | > loss_text_ce: 0.03669581562280655 (0.04887386926766964) | > loss_mel_ce: 2.926508665084839 (nan) | > loss: 0.011758748441934586 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13651957184654534) | > loader_time: 0.0047 (0.00464942341606167)  --> TIME: 2025-02-24 04:19:05 -- STEP: 14898/36938 -- GLOBAL_STEP: 162650 | > loss_text_ce: 0.05129349231719971 (0.04887310244995187) | > loss_mel_ce: 2.7375922203063965 (nan) | > loss: 0.01106700673699379 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1364962966158225) | > loader_time: 0.0045 (0.004649299889670926)  --> TIME: 2025-02-24 04:19:14 -- STEP: 14948/36938 -- GLOBAL_STEP: 162700 | > loss_text_ce: 0.05248105898499489 (0.04886719959062552) | > loss_mel_ce: 3.9005126953125 (nan) | > loss: 0.015686484053730965 (nan) | > current_lr: 5e-06 | > step_time: 0.1505 (0.13649741693202888) | > loader_time: 0.0046 (0.004649116037872557)  --> TIME: 2025-02-24 04:19:23 -- STEP: 14998/36938 -- GLOBAL_STEP: 162750 | > loss_text_ce: 0.04510074853897095 (0.048869752614859596) | > loss_mel_ce: 4.068022727966309 (nan) | > loss: 0.016321919858455658 (nan) | > current_lr: 5e-06 | > step_time: 0.1624 (0.13649899054406764) | > loader_time: 0.0046 (0.00464884858717045)  --> TIME: 2025-02-24 04:19:33 -- STEP: 15048/36938 -- GLOBAL_STEP: 162800 | > loss_text_ce: 0.05200923979282379 (0.0488712953598917) | > loss_mel_ce: 3.9149718284606934 (nan) | > loss: 0.015741989016532898 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13650331802662397) | > loader_time: 0.0046 (0.0046486520247533145)  --> TIME: 2025-02-24 04:19:42 -- STEP: 15098/36938 -- GLOBAL_STEP: 162850 | > loss_text_ce: 0.05589686706662178 (0.04886498645036583) | > loss_mel_ce: 3.6075897216796875 (nan) | > loss: 0.014537645503878593 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13649275505332897) | > loader_time: 0.0046 (0.004648403168229956)  --> TIME: 2025-02-24 04:19:51 -- STEP: 15148/36938 -- GLOBAL_STEP: 162900 | > loss_text_ce: 0.0488346628844738 (0.048860411846012745) | > loss_mel_ce: 4.335094928741455 (nan) | > loss: 0.01739654690027237 (nan) | > current_lr: 5e-06 | > step_time: 0.1883 (0.13649984850303187) | > loader_time: 0.0048 (0.004648223775088541)  --> TIME: 2025-02-24 04:20:00 -- STEP: 15198/36938 -- GLOBAL_STEP: 162950 | > loss_text_ce: 0.049094609916210175 (0.048854745645713314) | > loss_mel_ce: 3.5261707305908203 (nan) | > loss: 0.01418756227940321 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.1364937079142625) | > loader_time: 0.0044 (0.004647978623770861)  --> TIME: 2025-02-24 04:20:10 -- STEP: 15248/36938 -- GLOBAL_STEP: 163000 | > loss_text_ce: 0.02903309091925621 (0.04885203704358381) | > loss_mel_ce: 3.327805519104004 (nan) | > loss: 0.01332078967243433 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13650121923696498) | > loader_time: 0.0045 (0.0046477679784750975)  --> TIME: 2025-02-24 04:20:19 -- STEP: 15298/36938 -- GLOBAL_STEP: 163050 | > loss_text_ce: 0.044478729367256165 (0.04885083048638883) | > loss_mel_ce: 0.1436360627412796 (nan) | > loss: 0.0007464872905984521 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.1365135055785882) | > loader_time: 0.0047 (0.004647543140760002)  --> TIME: 2025-02-24 04:20:29 -- STEP: 15348/36938 -- GLOBAL_STEP: 163100 | > loss_text_ce: 0.042386893182992935 (0.048846705402178406) | > loss_mel_ce: 4.074999809265137 (nan) | > loss: 0.016338838264346123 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1365124493301444) | > loader_time: 0.0046 (0.00464753263283126)  --> TIME: 2025-02-24 04:20:38 -- STEP: 15398/36938 -- GLOBAL_STEP: 163150 | > loss_text_ce: 0.05155419185757637 (0.048845008754920545) | > loss_mel_ce: 3.472321033477783 (nan) | > loss: 0.013983632437884808 (nan) | > current_lr: 5e-06 | > step_time: 0.187 (0.1365163602183634) | > loader_time: 0.0048 (0.0046476065485427185)  --> TIME: 2025-02-24 04:20:47 -- STEP: 15448/36938 -- GLOBAL_STEP: 163200 | > loss_text_ce: 0.048071540892124176 (0.048846344442486804) | > loss_mel_ce: 1.28955078125 (nan) | > loss: 0.005308025050908327 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13651674688016896) | > loader_time: 0.0045 (0.004647671096007626)  --> TIME: 2025-02-24 04:20:57 -- STEP: 15498/36938 -- GLOBAL_STEP: 163250 | > loss_text_ce: 0.04281262680888176 (0.048842476931042295) | > loss_mel_ce: 2.008975028991699 (nan) | > loss: 0.008142014965415001 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13651602268834512) | > loader_time: 0.0045 (0.004647376922226417)  --> TIME: 2025-02-24 04:21:06 -- STEP: 15548/36938 -- GLOBAL_STEP: 163300 | > loss_text_ce: 0.054276932030916214 (0.0488436765786584) | > loss_mel_ce: 3.885756015777588 (nan) | > loss: 0.0156350526958704 (nan) | > current_lr: 5e-06 | > step_time: 0.1846 (0.13651038176117422) | > loader_time: 0.0047 (0.004647331707637681)  --> TIME: 2025-02-24 04:21:15 -- STEP: 15598/36938 -- GLOBAL_STEP: 163350 | > loss_text_ce: 0.06484174728393555 (0.04883989180677168) | > loss_mel_ce: 0.02381388284265995 (nan) | > loss: 0.0003518080629874021 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13649549129942112) | > loader_time: 0.0045 (0.004647107655642252)  --> TIME: 2025-02-24 04:21:24 -- STEP: 15648/36938 -- GLOBAL_STEP: 163400 | > loss_text_ce: 0.04290638491511345 (0.04883652412793729) | > loss_mel_ce: 1.7519935369491577 (nan) | > loss: 0.007122619077563286 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1364971908354093) | > loader_time: 0.0045 (0.004646894893397604)  --> TIME: 2025-02-24 04:21:34 -- STEP: 15698/36938 -- GLOBAL_STEP: 163450 | > loss_text_ce: 0.0462544821202755 (0.048836676641011975) | > loss_mel_ce: 3.0852978229522705 (nan) | > loss: 0.012426795437932014 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13650269521423514) | > loader_time: 0.0046 (0.004646778182628632)  --> TIME: 2025-02-24 04:21:43 -- STEP: 15748/36938 -- GLOBAL_STEP: 163500 | > loss_text_ce: 0.05222275108098984 (0.0488285907547862) | > loss_mel_ce: 3.6723058223724365 (nan) | > loss: 0.014779875986278057 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13651124706479215) | > loader_time: 0.0047 (0.004646667254464672)  --> TIME: 2025-02-24 04:21:53 -- STEP: 15798/36938 -- GLOBAL_STEP: 163550 | > loss_text_ce: 0.03370281681418419 (0.048819838165362) | > loss_mel_ce: 2.957291603088379 (nan) | > loss: 0.011869026347994804 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13651523198911347) | > loader_time: 0.0046 (0.004646511300634281)  --> TIME: 2025-02-24 04:22:02 -- STEP: 15848/36938 -- GLOBAL_STEP: 163600 | > loss_text_ce: 0.06669189035892487 (0.04881695336242641) | > loss_mel_ce: 2.120891809463501 (nan) | > loss: 0.008680888451635838 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13649684123051922) | > loader_time: 0.0045 (0.0046462779662514225)  --> TIME: 2025-02-24 04:22:11 -- STEP: 15898/36938 -- GLOBAL_STEP: 163650 | > loss_text_ce: 0.03920980170369148 (0.04881273800270986) | > loss_mel_ce: 3.3056063652038574 (nan) | > loss: 0.013273080810904503 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13649794548198954) | > loader_time: 0.0045 (0.004646071864008279)  --> TIME: 2025-02-24 04:22:20 -- STEP: 15948/36938 -- GLOBAL_STEP: 163700 | > loss_text_ce: 0.046576131135225296 (0.048812294444573794) | > loss_mel_ce: 3.731344699859619 (nan) | > loss: 0.01499175000935793 (nan) | > current_lr: 5e-06 | > step_time: 0.1583 (0.13649237608891826) | > loader_time: 0.0046 (0.004645970625957532)  --> TIME: 2025-02-24 04:22:30 -- STEP: 15998/36938 -- GLOBAL_STEP: 163750 | > loss_text_ce: 0.04389306902885437 (0.04880770415286325) | > loss_mel_ce: 3.3455376625061035 (nan) | > loss: 0.013450122438371181 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13650691087491404) | > loader_time: 0.0045 (0.004645790751538665)  --> TIME: 2025-02-24 04:22:39 -- STEP: 16048/36938 -- GLOBAL_STEP: 163800 | > loss_text_ce: 0.04835699871182442 (0.04880662850754942) | > loss_mel_ce: 3.420673131942749 (nan) | > loss: 0.013765993528068066 (nan) | > current_lr: 5e-06 | > step_time: 0.184 (0.1364965669417088) | > loader_time: 0.0048 (0.004645567933322208)  --> TIME: 2025-02-24 04:22:48 -- STEP: 16098/36938 -- GLOBAL_STEP: 163850 | > loss_text_ce: 0.043433282524347305 (0.048801600926074676) | > loss_mel_ce: 3.7151317596435547 (nan) | > loss: 0.014914941042661667 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.13649024094302023) | > loader_time: 0.0047 (0.004645366419293261)  --> TIME: 2025-02-24 04:22:58 -- STEP: 16148/36938 -- GLOBAL_STEP: 163900 | > loss_text_ce: 0.05058445408940315 (0.0488059569364411) | > loss_mel_ce: 4.0106940269470215 (nan) | > loss: 0.01611618511378765 (nan) | > current_lr: 5e-06 | > step_time: 0.1355 (0.13649590242195617) | > loader_time: 0.0046 (0.004645226407470421)  --> TIME: 2025-02-24 04:23:07 -- STEP: 16198/36938 -- GLOBAL_STEP: 163950 | > loss_text_ce: 0.043555717915296555 (0.04880513070238141) | > loss_mel_ce: 3.443547010421753 (nan) | > loss: 0.01383771002292633 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.1364962594540639) | > loader_time: 0.0047 (0.004645123748459031)  --> TIME: 2025-02-24 04:23:16 -- STEP: 16248/36938 -- GLOBAL_STEP: 164000 | > loss_text_ce: 0.03873727470636368 (0.04880466829561903) | > loss_mel_ce: 3.273970127105713 (nan) | > loss: 0.01314566470682621 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.1365050922497822) | > loader_time: 0.0046 (0.004645077173255591)  --> TIME: 2025-02-24 04:23:25 -- STEP: 16298/36938 -- GLOBAL_STEP: 164050 | > loss_text_ce: 0.04529193416237831 (0.04879811373340979) | > loss_mel_ce: 3.423238515853882 (nan) | > loss: 0.01376401074230671 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1364916028501316) | > loader_time: 0.0045 (0.0046453705769051185)  --> TIME: 2025-02-24 04:23:34 -- STEP: 16348/36938 -- GLOBAL_STEP: 164100 | > loss_text_ce: 0.04873710125684738 (0.04879452438405451) | > loss_mel_ce: 2.968623161315918 (nan) | > loss: 0.011973652057349682 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13646662465686735) | > loader_time: 0.0045 (0.004645164843611321)  --> TIME: 2025-02-24 04:23:44 -- STEP: 16398/36938 -- GLOBAL_STEP: 164150 | > loss_text_ce: 0.04367670789361 (0.048791590822068395) | > loss_mel_ce: 3.83664608001709 (nan) | > loss: 0.015398106537759304 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.1364806205823039) | > loader_time: 0.0045 (0.004645120444741556)  --> TIME: 2025-02-24 04:23:54 -- STEP: 16448/36938 -- GLOBAL_STEP: 164200 | > loss_text_ce: 0.05816511809825897 (0.04879319877459996) | > loss_mel_ce: 3.6828298568725586 (nan) | > loss: 0.014845218509435654 (nan) | > current_lr: 5e-06 | > step_time: 0.1842 (0.13649930371558774) | > loader_time: 0.0048 (0.00464525610092551)  --> TIME: 2025-02-24 04:24:03 -- STEP: 16498/36938 -- GLOBAL_STEP: 164250 | > loss_text_ce: 0.050295472145080566 (0.048787979727988234) | > loss_mel_ce: 3.1334543228149414 (nan) | > loss: 0.01263392809778452 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13649519515306488) | > loader_time: 0.0044 (0.004645100968261842)  --> TIME: 2025-02-24 04:24:12 -- STEP: 16548/36938 -- GLOBAL_STEP: 164300 | > loss_text_ce: 0.05035513639450073 (0.04878641066932947) | > loss_mel_ce: 2.903886079788208 (nan) | > loss: 0.011723180301487446 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13650966118521568) | > loader_time: 0.0045 (0.004644860456552909)  --> TIME: 2025-02-24 04:24:22 -- STEP: 16598/36938 -- GLOBAL_STEP: 164350 | > loss_text_ce: 0.04100016877055168 (0.04878060000386714) | > loss_mel_ce: 2.440563917160034 (nan) | > loss: 0.009847477078437805 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.13651079565003094) | > loader_time: 0.0045 (0.004644644218750055)  --> TIME: 2025-02-24 04:24:31 -- STEP: 16648/36938 -- GLOBAL_STEP: 164400 | > loss_text_ce: 0.05408197641372681 (0.04878109389566311) | > loss_mel_ce: 3.7613229751586914 (nan) | > loss: 0.01514049619436264 (nan) | > current_lr: 5e-06 | > step_time: 0.1664 (0.13651448451000622) | > loader_time: 0.0049 (0.0046445217658441324)  --> TIME: 2025-02-24 04:24:40 -- STEP: 16698/36938 -- GLOBAL_STEP: 164450 | > loss_text_ce: 0.061680715531110764 (0.048785046486853036) | > loss_mel_ce: 2.1520423889160156 (nan) | > loss: 0.008784616366028786 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.136515060689354) | > loader_time: 0.0045 (0.0046442450699170695)  --> TIME: 2025-02-24 04:24:50 -- STEP: 16748/36938 -- GLOBAL_STEP: 164500 | > loss_text_ce: 0.039745818823575974 (0.0487812064603751) | > loss_mel_ce: 3.192502021789551 (nan) | > loss: 0.012826381251215935 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.1365403708934457) | > loader_time: 0.0046 (0.004644125906413813)  --> TIME: 2025-02-24 04:24:59 -- STEP: 16798/36938 -- GLOBAL_STEP: 164550 | > loss_text_ce: 0.0330517403781414 (0.04878048447946178) | > loss_mel_ce: 3.8376877307891846 (nan) | > loss: 0.01536007784307003 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13654166188236222) | > loader_time: 0.0046 (0.004644294795088449)  --> TIME: 2025-02-24 04:25:09 -- STEP: 16848/36938 -- GLOBAL_STEP: 164600 | > loss_text_ce: 0.04635467007756233 (0.04878152110986453) | > loss_mel_ce: 4.136905193328857 (nan) | > loss: 0.016600238159298897 (nan) | > current_lr: 5e-06 | > step_time: 0.1725 (0.1365432595340629) | > loader_time: 0.0048 (0.00464438503397611)  --> TIME: 2025-02-24 04:25:18 -- STEP: 16898/36938 -- GLOBAL_STEP: 164650 | > loss_text_ce: 0.0487070307135582 (0.04877825476204277) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.1365378484145793) | > loader_time: 0.0045 (0.0046445642622113615)  --> TIME: 2025-02-24 04:25:27 -- STEP: 16948/36938 -- GLOBAL_STEP: 164700 | > loss_text_ce: 0.05565278232097626 (0.04877533790963602) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13653428741072132) | > loader_time: 0.0047 (0.0046446800006851325)  --> TIME: 2025-02-24 04:25:36 -- STEP: 16998/36938 -- GLOBAL_STEP: 164750 | > loss_text_ce: 0.04826013371348381 (0.04877036899519558) | > loss_mel_ce: 3.9901041984558105 (nan) | > loss: 0.016025256365537643 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13652449983024004) | > loader_time: 0.0045 (0.0046448229985989655)  --> TIME: 2025-02-24 04:25:45 -- STEP: 17048/36938 -- GLOBAL_STEP: 164800 | > loss_text_ce: 0.04758621007204056 (0.04876794444458497) | > loss_mel_ce: 3.455775499343872 (nan) | > loss: 0.013902230188250542 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1365164275294695) | > loader_time: 0.0046 (0.004644986513017898)  --> TIME: 2025-02-24 04:25:54 -- STEP: 17098/36938 -- GLOBAL_STEP: 164850 | > loss_text_ce: 0.05799774080514908 (0.04876535917857897) | > loss_mel_ce: 2.4746294021606445 (nan) | > loss: 0.010050107724964619 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13649949017624266) | > loader_time: 0.0046 (0.004644890809870715)  --> TIME: 2025-02-24 04:26:04 -- STEP: 17148/36938 -- GLOBAL_STEP: 164900 | > loss_text_ce: 0.042685579508543015 (0.04875791129203836) | > loss_mel_ce: 0.013580322265625 (nan) | > loss: 0.00022327739861793816 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13650042736255164) | > loader_time: 0.0046 (0.0046449346728110385)  --> TIME: 2025-02-24 04:26:13 -- STEP: 17198/36938 -- GLOBAL_STEP: 164950 | > loss_text_ce: 0.04167606681585312 (0.048751050020055905) | > loss_mel_ce: 3.4752774238586426 (nan) | > loss: 0.013956164941191673 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13650132019945002) | > loader_time: 0.0047 (0.004644877869856349)  --> TIME: 2025-02-24 04:26:22 -- STEP: 17248/36938 -- GLOBAL_STEP: 165000 | > loss_text_ce: 0.056145112961530685 (0.048747298995463595) | > loss_mel_ce: 0.7079569697380066 (nan) | > loss: 0.0030321513768285513 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13649378906320475) | > loader_time: 0.0046 (0.004645055612628687)  --> TIME: 2025-02-24 04:26:32 -- STEP: 17298/36938 -- GLOBAL_STEP: 165050 | > loss_text_ce: 0.04919786751270294 (0.048744034306852056) | > loss_mel_ce: 3.6892287731170654 (nan) | > loss: 0.014835027046501637 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13649961154217646) | > loader_time: 0.0046 (0.004645137679909063)  --> TIME: 2025-02-24 04:26:41 -- STEP: 17348/36938 -- GLOBAL_STEP: 165100 | > loss_text_ce: 0.04380233958363533 (0.048746693694099945) | > loss_mel_ce: 4.9921875 (nan) | > loss: 0.019984086975455284 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13650950270642584) | > loader_time: 0.0046 (0.004645279002459023)  --> TIME: 2025-02-24 04:26:51 -- STEP: 17398/36938 -- GLOBAL_STEP: 165150 | > loss_text_ce: 0.043576937168836594 (0.0487437754273025) | > loss_mel_ce: 3.8287763595581055 (nan) | > loss: 0.01536648254841566 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13652208503984645) | > loader_time: 0.0045 (0.004645335960695165)  --> TIME: 2025-02-24 04:27:00 -- STEP: 17448/36938 -- GLOBAL_STEP: 165200 | > loss_text_ce: 0.06582050025463104 (0.048741393866338176) | > loss_mel_ce: 2.901411533355713 (nan) | > loss: 0.011774730868637562 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13652299122491296) | > loader_time: 0.0046 (0.004645362571530926)  --> TIME: 2025-02-24 04:27:09 -- STEP: 17498/36938 -- GLOBAL_STEP: 165250 | > loss_text_ce: 0.06037210673093796 (0.04874275426506691) | > loss_mel_ce: 3.2762858867645264 (nan) | > loss: 0.013240707106888294 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13650906624174458) | > loader_time: 0.0045 (0.004645306214644241)  --> TIME: 2025-02-24 04:27:18 -- STEP: 17548/36938 -- GLOBAL_STEP: 165300 | > loss_text_ce: 0.05387166887521744 (0.04874104804455757) | > loss_mel_ce: 3.9550936222076416 (nan) | > loss: 0.01590859331190586 (nan) | > current_lr: 5e-06 | > step_time: 0.1809 (0.13651236156029428) | > loader_time: 0.005 (0.004645257855374714)  --> TIME: 2025-02-24 04:27:28 -- STEP: 17598/36938 -- GLOBAL_STEP: 165350 | > loss_text_ce: 0.05100966989994049 (0.048740065567957916) | > loss_mel_ce: 2.2783591747283936 (nan) | > loss: 0.009243527427315712 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13650406780724295) | > loader_time: 0.0045 (0.0046452099741257375)  --> TIME: 2025-02-24 04:27:37 -- STEP: 17648/36938 -- GLOBAL_STEP: 165400 | > loss_text_ce: 0.048373494297266006 (0.048744752896679046) | > loss_mel_ce: 3.484476089477539 (nan) | > loss: 0.014019245281815529 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.1364943260970394) | > loader_time: 0.0046 (0.004645151313282877)  --> TIME: 2025-02-24 04:27:46 -- STEP: 17698/36938 -- GLOBAL_STEP: 165450 | > loss_text_ce: 0.055042725056409836 (0.04874511209596192) | > loss_mel_ce: 2.8205742835998535 (nan) | > loss: 0.011411179788410664 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13648434400100584) | > loader_time: 0.0045 (0.004644939449227127)  --> TIME: 2025-02-24 04:27:55 -- STEP: 17748/36938 -- GLOBAL_STEP: 165500 | > loss_text_ce: 0.04703633114695549 (0.04874205026948348) | > loss_mel_ce: 0.563385009765625 (nan) | > loss: 0.002422307152301073 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.1364729609445394) | > loader_time: 0.0045 (0.004644776723232815)  --> TIME: 2025-02-24 04:28:04 -- STEP: 17798/36938 -- GLOBAL_STEP: 165550 | > loss_text_ce: 0.057178743183612823 (0.04874240322985033) | > loss_mel_ce: 3.840672492980957 (nan) | > loss: 0.01546766422688961 (nan) | > current_lr: 5e-06 | > step_time: 0.1701 (0.13647979083309456) | > loader_time: 0.0048 (0.00464472988774664)  --> TIME: 2025-02-24 04:28:14 -- STEP: 17848/36938 -- GLOBAL_STEP: 165600 | > loss_text_ce: 0.05104854330420494 (0.048740848880363494) | > loss_mel_ce: 3.7733211517333984 (nan) | > loss: 0.015176070854067802 (nan) | > current_lr: 5e-06 | > step_time: 0.1723 (0.13647378209672914) | > loader_time: 0.0047 (0.00464475431393105)  --> TIME: 2025-02-24 04:28:23 -- STEP: 17898/36938 -- GLOBAL_STEP: 165650 | > loss_text_ce: 0.0535365529358387 (0.048735073630073955) | > loss_mel_ce: 4.106443881988525 (nan) | > loss: 0.016507858410477638 (nan) | > current_lr: 5e-06 | > step_time: 0.1693 (0.1364807312096135) | > loader_time: 0.0048 (0.0046446697580600155)  --> TIME: 2025-02-24 04:28:32 -- STEP: 17948/36938 -- GLOBAL_STEP: 165700 | > loss_text_ce: 0.04049188643693924 (0.0487394544298769) | > loss_mel_ce: 3.4268417358398438 (nan) | > loss: 0.01375926099717617 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13646902690253793) | > loader_time: 0.0046 (0.004644602317971616)  --> TIME: 2025-02-24 04:28:41 -- STEP: 17998/36938 -- GLOBAL_STEP: 165750 | > loss_text_ce: 0.048143867403268814 (0.048737773343004644) | > loss_mel_ce: 3.2346649169921875 (nan) | > loss: 0.013027019798755646 (nan) | > current_lr: 5e-06 | > step_time: 0.1436 (0.1364694520465175) | > loader_time: 0.0049 (0.004644663522794333)  --> TIME: 2025-02-24 04:28:50 -- STEP: 18048/36938 -- GLOBAL_STEP: 165800 | > loss_text_ce: 0.05025996267795563 (0.04873741919818589) | > loss_mel_ce: 4.370069980621338 (nan) | > loss: 0.01754099316895008 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13645388314117496) | > loader_time: 0.0047 (0.004644489364949549)  --> TIME: 2025-02-24 04:29:00 -- STEP: 18098/36938 -- GLOBAL_STEP: 165850 | > loss_text_ce: 0.05764129385352135 (0.04873809278012769) | > loss_mel_ce: 4.284993648529053 (nan) | > loss: 0.017232680693268776 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.1364670574026156) | > loader_time: 0.0046 (0.00464470638914234)  --> TIME: 2025-02-24 04:29:09 -- STEP: 18148/36938 -- GLOBAL_STEP: 165900 | > loss_text_ce: 0.054047465324401855 (0.04873543201195663) | > loss_mel_ce: 3.2978076934814453 (nan) | > loss: 0.013301013968884945 (nan) | > current_lr: 5e-06 | > step_time: 0.1571 (0.13646969226112515) | > loader_time: 0.0046 (0.004644616955514805)  --> TIME: 2025-02-24 04:29:18 -- STEP: 18198/36938 -- GLOBAL_STEP: 165950 | > loss_text_ce: 0.05580156669020653 (0.04873546492272565) | > loss_mel_ce: 3.7358627319335938 (nan) | > loss: 0.015046288259327412 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.13645788284039445) | > loader_time: 0.0046 (0.0046446518081009565)  --> TIME: 2025-02-24 04:29:27 -- STEP: 18248/36938 -- GLOBAL_STEP: 166000 | > loss_text_ce: 0.053854722529649734 (0.048736302131398776) | > loss_mel_ce: 3.198361396789551 (nan) | > loss: 0.012905620038509369 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13645111781523483) | > loader_time: 0.0046 (0.004644720113261776)  --> TIME: 2025-02-24 04:29:37 -- STEP: 18298/36938 -- GLOBAL_STEP: 166050 | > loss_text_ce: 0.03967728838324547 (0.0487315542563811) | > loss_mel_ce: 1.2932801246643066 (nan) | > loss: 0.0052895136177539825 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13644664530442038) | > loader_time: 0.0046 (0.004644765607879953)  --> TIME: 2025-02-24 04:29:46 -- STEP: 18348/36938 -- GLOBAL_STEP: 166100 | > loss_text_ce: 0.05079283565282822 (0.048727571867974585) | > loss_mel_ce: 3.639090061187744 (nan) | > loss: 0.014642393216490746 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13644238414394463) | > loader_time: 0.0045 (0.004644728406000084)  --> TIME: 2025-02-24 04:29:55 -- STEP: 18398/36938 -- GLOBAL_STEP: 166150 | > loss_text_ce: 0.04987754300236702 (0.048724368936511456) | > loss_mel_ce: 1.6702393293380737 (nan) | > loss: 0.006825861055403948 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1364356865894809) | > loader_time: 0.0047 (0.004644633026301384)  --> TIME: 2025-02-24 04:30:04 -- STEP: 18448/36938 -- GLOBAL_STEP: 166200 | > loss_text_ce: 0.033433910459280014 (0.04871947600584812) | > loss_mel_ce: 0.1606852263212204 (nan) | > loss: 0.0007703140727244318 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.1364323484411692) | > loader_time: 0.0049 (0.0046446782190285064)  --> TIME: 2025-02-24 04:30:13 -- STEP: 18498/36938 -- GLOBAL_STEP: 166250 | > loss_text_ce: 0.04530223086476326 (0.048718693963259056) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.1364206030943921) | > loader_time: 0.0045 (0.004644619167037953)  --> TIME: 2025-02-24 04:30:22 -- STEP: 18548/36938 -- GLOBAL_STEP: 166300 | > loss_text_ce: 0.043678782880306244 (0.04871852063965153) | > loss_mel_ce: 3.6597940921783447 (nan) | > loss: 0.014696321450173855 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13640446542845308) | > loader_time: 0.0045 (0.004644491702337494)  --> TIME: 2025-02-24 04:30:32 -- STEP: 18598/36938 -- GLOBAL_STEP: 166350 | > loss_text_ce: 0.04996242746710777 (0.04871241876363548) | > loss_mel_ce: 4.128756046295166 (nan) | > loss: 0.016582217067480087 (nan) | > current_lr: 5e-06 | > step_time: 0.1576 (0.13641672696410018) | > loader_time: 0.0046 (0.004644400343508851)  --> TIME: 2025-02-24 04:30:41 -- STEP: 18648/36938 -- GLOBAL_STEP: 166400 | > loss_text_ce: 0.040434710681438446 (0.048706069427697486) | > loss_mel_ce: 3.281773090362549 (nan) | > loss: 0.013183364644646645 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13642134474920592) | > loader_time: 0.0045 (0.004644264918138743)  --> TIME: 2025-02-24 04:30:50 -- STEP: 18698/36938 -- GLOBAL_STEP: 166450 | > loss_text_ce: 0.049463901668787 (0.04870628775926752) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1364025165083743) | > loader_time: 0.0045 (0.004643975011625848)  --> TIME: 2025-02-24 04:30:59 -- STEP: 18748/36938 -- GLOBAL_STEP: 166500 | > loss_text_ce: 0.046720605343580246 (0.048705115330206696) | > loss_mel_ce: 3.625089645385742 (nan) | > loss: 0.014570675790309906 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.13640200117643236) | > loader_time: 0.0047 (0.004643949092464573)  --> TIME: 2025-02-24 04:31:09 -- STEP: 18798/36938 -- GLOBAL_STEP: 166550 | > loss_text_ce: 0.04372325912117958 (0.048704764353449725) | > loss_mel_ce: 3.779926061630249 (nan) | > loss: 0.015173212625086308 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.136397542554226) | > loader_time: 0.0046 (0.004643776313030095)  --> TIME: 2025-02-24 04:31:18 -- STEP: 18848/36938 -- GLOBAL_STEP: 166600 | > loss_text_ce: 0.047418445348739624 (0.04870195344612459) | > loss_mel_ce: 3.937845468521118 (nan) | > loss: 0.015814539045095444 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13640345948003707) | > loader_time: 0.0046 (0.004643743519366104)  --> TIME: 2025-02-24 04:31:28 -- STEP: 18898/36938 -- GLOBAL_STEP: 166650 | > loss_text_ce: 0.05634298548102379 (0.0486989484926737) | > loss_mel_ce: 2.77642822265625 (nan) | > loss: 0.011241156607866287 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13642487832258401) | > loader_time: 0.0048 (0.004643818312491661)  --> TIME: 2025-02-24 04:31:37 -- STEP: 18948/36938 -- GLOBAL_STEP: 166700 | > loss_text_ce: 0.040481746196746826 (0.04869355024528677) | > loss_mel_ce: 2.9120140075683594 (nan) | > loss: 0.01171625405550003 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13641950351567478) | > loader_time: 0.0045 (0.004643658281052102)  --> TIME: 2025-02-24 04:31:47 -- STEP: 18998/36938 -- GLOBAL_STEP: 166750 | > loss_text_ce: 0.05003533139824867 (0.04869464171784862) | > loss_mel_ce: 4.028285026550293 (nan) | > loss: 0.01618381217122078 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13643046185423052) | > loader_time: 0.0047 (0.004643682417511662)  --> TIME: 2025-02-24 04:31:56 -- STEP: 19048/36938 -- GLOBAL_STEP: 166800 | > loss_text_ce: 0.05402694642543793 (0.04869349192665036) | > loss_mel_ce: 3.020963430404663 (nan) | > loss: 0.012202342972159386 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13642552090712684) | > loader_time: 0.0046 (0.00464372304946824)  --> TIME: 2025-02-24 04:32:05 -- STEP: 19098/36938 -- GLOBAL_STEP: 166850 | > loss_text_ce: 0.04839705675840378 (0.048693494764023626) | > loss_mel_ce: 0.053955078125 (nan) | > loss: 0.00040615929174236953 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.13642291354064584) | > loader_time: 0.0046 (0.0046436859682750225)  --> TIME: 2025-02-24 04:32:14 -- STEP: 19148/36938 -- GLOBAL_STEP: 166900 | > loss_text_ce: 0.04353206232190132 (0.04869382284237899) | > loss_mel_ce: 3.309457778930664 (nan) | > loss: 0.013305515982210636 (nan) | > current_lr: 5e-06 | > step_time: 0.161 (0.1364307003377861) | > loader_time: 0.0046 (0.004643643851167669)  --> TIME: 2025-02-24 04:32:24 -- STEP: 19198/36938 -- GLOBAL_STEP: 166950 | > loss_text_ce: 0.034631796181201935 (0.04869374514627581) | > loss_mel_ce: 0.027405261993408203 (nan) | > loss: 0.000246178824454546 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1364183951348176) | > loader_time: 0.0045 (0.004643583610686491)  --> TIME: 2025-02-24 04:32:33 -- STEP: 19248/36938 -- GLOBAL_STEP: 167000 | > loss_text_ce: 0.03988324850797653 (0.04869346726161858) | > loss_mel_ce: 2.9397101402282715 (nan) | > loss: 0.011823783628642559 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.136423863253789) | > loader_time: 0.0046 (0.0046435924415675735)  --> TIME: 2025-02-24 04:32:42 -- STEP: 19298/36938 -- GLOBAL_STEP: 167050 | > loss_text_ce: 0.05857672169804573 (0.048691374962605796) | > loss_mel_ce: 3.839381694793701 (nan) | > loss: 0.01546808984130621 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13640945158371573) | > loader_time: 0.0045 (0.0046434510444801815)  --> TIME: 2025-02-24 04:32:51 -- STEP: 19348/36938 -- GLOBAL_STEP: 167100 | > loss_text_ce: 0.050944674760103226 (0.04869014454443513) | > loss_mel_ce: 4.049781322479248 (nan) | > loss: 0.01627272367477417 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13639516882938948) | > loader_time: 0.0048 (0.004643332250901498)  --> TIME: 2025-02-24 04:33:00 -- STEP: 19398/36938 -- GLOBAL_STEP: 167150 | > loss_text_ce: 0.05585747957229614 (0.048690081875293605) | > loss_mel_ce: 4.403249740600586 (nan) | > loss: 0.017694871872663498 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13640176893472933) | > loader_time: 0.0046 (0.004643276089527583)  --> TIME: 2025-02-24 04:33:10 -- STEP: 19448/36938 -- GLOBAL_STEP: 167200 | > loss_text_ce: 0.042272113263607025 (0.048687129599867554) | > loss_mel_ce: 3.842717409133911 (nan) | > loss: 0.015416625887155533 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.13640489115091411) | > loader_time: 0.0047 (0.004643183831372229)  --> TIME: 2025-02-24 04:33:19 -- STEP: 19498/36938 -- GLOBAL_STEP: 167250 | > loss_text_ce: 0.04839594289660454 (0.048683808947442425) | > loss_mel_ce: 0.1632603257894516 (nan) | > loss: 0.0008399058715440333 (nan) | > current_lr: 5e-06 | > step_time: 0.1733 (0.13638905905885013) | > loader_time: 0.0048 (0.004643283241014516)  --> TIME: 2025-02-24 04:33:28 -- STEP: 19548/36938 -- GLOBAL_STEP: 167300 | > loss_text_ce: 0.052944693714380264 (0.048685398162852514) | > loss_mel_ce: 3.9341201782226562 (nan) | > loss: 0.015821686014533043 (nan) | > current_lr: 5e-06 | > step_time: 0.1831 (0.13639479690176814) | > loader_time: 0.0049 (0.004643429940479152)  --> TIME: 2025-02-24 04:33:37 -- STEP: 19598/36938 -- GLOBAL_STEP: 167350 | > loss_text_ce: 0.048418596386909485 (0.0486829884566333) | > loss_mel_ce: 3.6826171875 (nan) | > loss: 0.014805697835981846 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13638935845559425) | > loader_time: 0.0046 (0.004643577351255301)  --> TIME: 2025-02-24 04:33:47 -- STEP: 19648/36938 -- GLOBAL_STEP: 167400 | > loss_text_ce: 0.0538063608109951 (0.048679949900832145) | > loss_mel_ce: 2.9743144512176514 (nan) | > loss: 0.012016353197395802 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13639812715084765) | > loader_time: 0.0045 (0.004643912084327867)  --> TIME: 2025-02-24 04:33:56 -- STEP: 19698/36938 -- GLOBAL_STEP: 167450 | > loss_text_ce: 0.04778675362467766 (0.04867751538986711) | > loss_mel_ce: 3.927536725997925 (nan) | > loss: 0.01577509380877018 (nan) | > current_lr: 5e-06 | > step_time: 0.1783 (0.13639825635155475) | > loader_time: 0.0051 (0.0046438682211245655)  --> TIME: 2025-02-24 04:34:05 -- STEP: 19748/36938 -- GLOBAL_STEP: 167500 | > loss_text_ce: 0.048632245510816574 (0.04867631487450464) | > loss_mel_ce: 3.8053646087646484 (nan) | > loss: 0.015293639153242111 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13639761847669432) | > loader_time: 0.0045 (0.004643764540761373)  --> TIME: 2025-02-24 04:34:14 -- STEP: 19798/36938 -- GLOBAL_STEP: 167550 | > loss_text_ce: 0.04131675139069557 (0.04867084692829283) | > loss_mel_ce: 2.688359022140503 (nan) | > loss: 0.010832047089934349 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13638450312053385) | > loader_time: 0.0046 (0.00464361492389743)  --> TIME: 2025-02-24 04:34:24 -- STEP: 19848/36938 -- GLOBAL_STEP: 167600 | > loss_text_ce: 0.04725424200296402 (0.04866512265376027) | > loss_mel_ce: 9.15625 (nan) | > loss: 0.03652184456586838 (nan) | > current_lr: 5e-06 | > step_time: 0.1071 (0.13638446082131067) | > loader_time: 0.0045 (0.004643713332433385)  --> TIME: 2025-02-24 04:34:33 -- STEP: 19898/36938 -- GLOBAL_STEP: 167650 | > loss_text_ce: 0.048931583762168884 (0.048660590849992566) | > loss_mel_ce: 1.306054711341858 (nan) | > loss: 0.005376929882913828 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.1363929177495235) | > loader_time: 0.0044 (0.004643625968160906)  --> TIME: 2025-02-24 04:34:42 -- STEP: 19948/36938 -- GLOBAL_STEP: 167700 | > loss_text_ce: 0.04431251809000969 (0.04865623747567465) | > loss_mel_ce: 0.1768798828125 (nan) | > loss: 0.000877747661434114 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13638523562245788) | > loader_time: 0.0046 (0.004643585606856715)  --> TIME: 2025-02-24 04:34:52 -- STEP: 19998/36938 -- GLOBAL_STEP: 167750 | > loss_text_ce: 0.03762417659163475 (0.0486535268987732) | > loss_mel_ce: 3.557786464691162 (nan) | > loss: 0.014267503283917904 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.1363833943347084) | > loader_time: 0.0045 (0.0046434841080658235)  --> TIME: 2025-02-24 04:35:01 -- STEP: 20048/36938 -- GLOBAL_STEP: 167800 | > loss_text_ce: 0.04587884247303009 (0.04865253681982402) | > loss_mel_ce: 2.837552547454834 (nan) | > loss: 0.01144218910485506 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13637718018397227) | > loader_time: 0.0047 (0.004643477945449552)  --> TIME: 2025-02-24 04:35:10 -- STEP: 20098/36938 -- GLOBAL_STEP: 167850 | > loss_text_ce: 0.04981669783592224 (0.04864871560657386) | > loss_mel_ce: 4.142569541931152 (nan) | > loss: 0.016636453568935394 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1363758528834032) | > loader_time: 0.0046 (0.0046435447459980735)  --> TIME: 2025-02-24 04:35:19 -- STEP: 20148/36938 -- GLOBAL_STEP: 167900 | > loss_text_ce: 0.04777142032980919 (0.048646566748657284) | > loss_mel_ce: 3.767807722091675 (nan) | > loss: 0.01514118816703558 (nan) | > current_lr: 5e-06 | > step_time: 0.1605 (0.1363754623556804) | > loader_time: 0.0047 (0.004643632266548448)  --> TIME: 2025-02-24 04:35:29 -- STEP: 20198/36938 -- GLOBAL_STEP: 167950 | > loss_text_ce: 0.0565764494240284 (0.04864551014214262) | > loss_mel_ce: 3.4538021087646484 (nan) | > loss: 0.013930074870586395 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13638235054484635) | > loader_time: 0.0046 (0.00464373995188571)  --> TIME: 2025-02-24 04:35:38 -- STEP: 20248/36938 -- GLOBAL_STEP: 168000 | > loss_text_ce: 0.05946018174290657 (0.0486450891058024) | > loss_mel_ce: 3.819134473800659 (nan) | > loss: 0.01539124920964241 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13639099085251022) | > loader_time: 0.0047 (0.0046436934544606)  --> TIME: 2025-02-24 04:35:48 -- STEP: 20298/36938 -- GLOBAL_STEP: 168050 | > loss_text_ce: 0.05580582097172737 (0.04864465183442538) | > loss_mel_ce: 3.6871142387390137 (nan) | > loss: 0.014852859079837799 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.13639677206346457) | > loader_time: 0.0047 (0.004643664757997992)  --> TIME: 2025-02-24 04:35:57 -- STEP: 20348/36938 -- GLOBAL_STEP: 168100 | > loss_text_ce: 0.04651729017496109 (0.04864658996277374) | > loss_mel_ce: 3.643315076828003 (nan) | > loss: 0.014642192982137203 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13640199806405848) | > loader_time: 0.0048 (0.004643762442084703)  --> TIME: 2025-02-24 04:36:06 -- STEP: 20398/36938 -- GLOBAL_STEP: 168150 | > loss_text_ce: 0.028250016272068024 (0.04864621003887004) | > loss_mel_ce: 3.9934821128845215 (nan) | > loss: 0.015959255397319794 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13639814037355283) | > loader_time: 0.0047 (0.004643759735426103)  --> TIME: 2025-02-24 04:36:16 -- STEP: 20448/36938 -- GLOBAL_STEP: 168200 | > loss_text_ce: 0.028076592832803726 (0.04864022749470134) | > loss_mel_ce: 3.147733688354492 (nan) | > loss: 0.012602422386407852 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.1363926424946594) | > loader_time: 0.0046 (0.004643732952959672)  --> TIME: 2025-02-24 04:36:25 -- STEP: 20498/36938 -- GLOBAL_STEP: 168250 | > loss_text_ce: 0.05216674506664276 (0.04863750654236309) | > loss_mel_ce: 2.986781597137451 (nan) | > loss: 0.012059319764375687 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.13640198468906206) | > loader_time: 0.0046 (0.004643789395226178)  --> TIME: 2025-02-24 04:36:34 -- STEP: 20548/36938 -- GLOBAL_STEP: 168300 | > loss_text_ce: 0.06061091646552086 (0.04863948521515409) | > loss_mel_ce: 1.4750398397445679 (nan) | > loss: 0.006093852687627077 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.1363911753571506) | > loader_time: 0.0046 (0.004643807887845367)  --> TIME: 2025-02-24 04:36:44 -- STEP: 20598/36938 -- GLOBAL_STEP: 168350 | > loss_text_ce: 0.05392630025744438 (0.04863666824240318) | > loss_mel_ce: 2.802382469177246 (nan) | > loss: 0.011334558948874474 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.13640753387629437) | > loader_time: 0.0046 (0.004643802284465198)  --> TIME: 2025-02-24 04:36:53 -- STEP: 20648/36938 -- GLOBAL_STEP: 168400 | > loss_text_ce: 0.044147372245788574 (0.04863475925305882) | > loss_mel_ce: 2.9706521034240723 (nan) | > loss: 0.011963491328060627 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.1364100251674477) | > loader_time: 0.0044 (0.004643723894024691)  --> TIME: 2025-02-24 04:37:02 -- STEP: 20698/36938 -- GLOBAL_STEP: 168450 | > loss_text_ce: 0.04878536984324455 (0.04863087033798229) | > loss_mel_ce: 3.768528699874878 (nan) | > loss: 0.015148072503507137 (nan) | > current_lr: 5e-06 | > step_time: 0.177 (0.13639760394063752) | > loader_time: 0.0049 (0.004643565411152566)  --> TIME: 2025-02-24 04:37:12 -- STEP: 20748/36938 -- GLOBAL_STEP: 168500 | > loss_text_ce: 0.05216395854949951 (0.04862552690931989) | > loss_mel_ce: 3.317653179168701 (nan) | > loss: 0.013372291810810566 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13640900480510598) | > loader_time: 0.0047 (0.004643637630230267)  --> TIME: 2025-02-24 04:37:21 -- STEP: 20798/36938 -- GLOBAL_STEP: 168550 | > loss_text_ce: 0.04509945213794708 (0.048624449780094336) | > loss_mel_ce: 3.787736654281616 (nan) | > loss: 0.015209668315947056 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13638990016405078) | > loader_time: 0.0046 (0.004643592023771551)  --> TIME: 2025-02-24 04:37:30 -- STEP: 20848/36938 -- GLOBAL_STEP: 168600 | > loss_text_ce: 0.04104207083582878 (0.048621410149600315) | > loss_mel_ce: 3.4001612663269043 (nan) | > loss: 0.013655569404363632 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1364024092464025) | > loader_time: 0.0046 (0.004643632943869935)  --> TIME: 2025-02-24 04:37:40 -- STEP: 20898/36938 -- GLOBAL_STEP: 168650 | > loss_text_ce: 0.040807295590639114 (0.048621829229028544) | > loss_mel_ce: 3.6908013820648193 (nan) | > loss: 0.014807971194386482 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13640931076108662) | > loader_time: 0.0047 (0.00464355718554443)  --> TIME: 2025-02-24 04:37:50 -- STEP: 20948/36938 -- GLOBAL_STEP: 168700 | > loss_text_ce: 0.04857945814728737 (0.04862064393031842) | > loss_mel_ce: 4.10038948059082 (nan) | > loss: 0.01646416448056698 (nan) | > current_lr: 5e-06 | > step_time: 0.1693 (0.13642886887075684) | > loader_time: 0.0048 (0.00464360569869607)  --> TIME: 2025-02-24 04:37:59 -- STEP: 20998/36938 -- GLOBAL_STEP: 168750 | > loss_text_ce: 0.03947385400533676 (0.0486140661870795) | > loss_mel_ce: 3.6436150074005127 (nan) | > loss: 0.014615432359278202 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13642906404606664) | > loader_time: 0.0045 (0.004643564122508241)  --> TIME: 2025-02-24 04:38:08 -- STEP: 21048/36938 -- GLOBAL_STEP: 168800 | > loss_text_ce: 0.05512426793575287 (0.0486156493429352) | > loss_mel_ce: 0.040771484375 (nan) | > loss: 0.00038053872413001955 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13642797861793426) | > loader_time: 0.0046 (0.004643539961460043)  --> TIME: 2025-02-24 04:38:17 -- STEP: 21098/36938 -- GLOBAL_STEP: 168850 | > loss_text_ce: 0.03952272608876228 (0.048609754245312276) | > loss_mel_ce: 2.154123306274414 (nan) | > loss: 0.008704944513738155 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13642287221561994) | > loader_time: 0.0045 (0.0046434747584002695)  --> TIME: 2025-02-24 04:38:27 -- STEP: 21148/36938 -- GLOBAL_STEP: 168900 | > loss_text_ce: 0.04437292367219925 (0.04860826411662533) | > loss_mel_ce: 3.8501229286193848 (nan) | > loss: 0.015454350039362907 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.13643780705246536) | > loader_time: 0.0049 (0.0046435216435056995)  --> TIME: 2025-02-24 04:38:36 -- STEP: 21198/36938 -- GLOBAL_STEP: 168950 | > loss_text_ce: 0.041407112032175064 (0.04860610875919399) | > loss_mel_ce: 3.0858240127563477 (nan) | > loss: 0.012409647926688194 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13644030437411922) | > loader_time: 0.0046 (0.004643519415766812)  --> TIME: 2025-02-24 04:38:46 -- STEP: 21248/36938 -- GLOBAL_STEP: 169000 | > loss_text_ce: 0.04945240914821625 (0.04860407383305111) | > loss_mel_ce: 3.356834650039673 (nan) | > loss: 0.013517012819647789 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1364441416955679) | > loader_time: 0.0046 (0.0046435563477227795)  --> TIME: 2025-02-24 04:38:55 -- STEP: 21298/36938 -- GLOBAL_STEP: 169050 | > loss_text_ce: 0.03856439143419266 (0.04860074299566309) | > loss_mel_ce: 3.546523332595825 (nan) | > loss: 0.014226539991796017 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13644967663287025) | > loader_time: 0.0045 (0.004643511398262443)  --> TIME: 2025-02-24 04:39:04 -- STEP: 21348/36938 -- GLOBAL_STEP: 169100 | > loss_text_ce: 0.03699591010808945 (0.048594552676097635) | > loss_mel_ce: 2.486409902572632 (nan) | > loss: 0.010013516061007977 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13644282589339385) | > loader_time: 0.0045 (0.0046434096010611195)  --> TIME: 2025-02-24 04:39:13 -- STEP: 21398/36938 -- GLOBAL_STEP: 169150 | > loss_text_ce: 0.05927915498614311 (0.04859353130669725) | > loss_mel_ce: 3.5697622299194336 (nan) | > loss: 0.014400959014892578 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13642856521866759) | > loader_time: 0.0046 (0.0046433238005324185)  --> TIME: 2025-02-24 04:39:22 -- STEP: 21448/36938 -- GLOBAL_STEP: 169200 | > loss_text_ce: 0.031983889639377594 (0.04858889203753633) | > loss_mel_ce: 3.2707746028900146 (nan) | > loss: 0.013106185011565685 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.1364201809745459) | > loader_time: 0.0045 (0.004643294102929529)  --> TIME: 2025-02-24 04:39:31 -- STEP: 21498/36938 -- GLOBAL_STEP: 169250 | > loss_text_ce: 0.043231476098299026 (0.04858314542287677) | > loss_mel_ce: 3.5997653007507324 (nan) | > loss: 0.014456337317824364 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.1364107013292144) | > loader_time: 0.0045 (0.004643346733443208)  --> TIME: 2025-02-24 04:39:40 -- STEP: 21548/36938 -- GLOBAL_STEP: 169300 | > loss_text_ce: 0.06420741230249405 (0.04858250622426654) | > loss_mel_ce: 3.434748649597168 (nan) | > loss: 0.013884746469557285 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.136401495321443) | > loader_time: 0.0045 (0.0046432808619694755)  --> TIME: 2025-02-24 04:39:49 -- STEP: 21598/36938 -- GLOBAL_STEP: 169350 | > loss_text_ce: 0.0412096343934536 (0.0485768839556071) | > loss_mel_ce: 3.9431369304656982 (nan) | > loss: 0.015810899436473846 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.13639804306643183) | > loader_time: 0.0049 (0.004643268249180404)  --> TIME: 2025-02-24 04:39:59 -- STEP: 21648/36938 -- GLOBAL_STEP: 169400 | > loss_text_ce: 0.043144602328538895 (0.048576461386267) | > loss_mel_ce: 3.765272617340088 (nan) | > loss: 0.015112767927348614 (nan) | > current_lr: 5e-06 | > step_time: 0.1582 (0.136402576611118) | > loader_time: 0.0046 (0.00464321194933861)  --> TIME: 2025-02-24 04:40:08 -- STEP: 21698/36938 -- GLOBAL_STEP: 169450 | > loss_text_ce: 0.052539799362421036 (0.04857663402918628) | > loss_mel_ce: 3.8714048862457275 (nan) | > loss: 0.015571210533380508 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.136406988556544) | > loader_time: 0.0046 (0.004643134811922492)  --> TIME: 2025-02-24 04:40:17 -- STEP: 21748/36938 -- GLOBAL_STEP: 169500 | > loss_text_ce: 0.05617714673280716 (0.04857450810922094) | > loss_mel_ce: 3.5885307788848877 (nan) | > loss: 0.014463127590715885 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1364011303212509) | > loader_time: 0.0045 (0.004643001844936926)  --> TIME: 2025-02-24 04:40:27 -- STEP: 21798/36938 -- GLOBAL_STEP: 169550 | > loss_text_ce: 0.0539558008313179 (0.048572579804640095) | > loss_mel_ce: 2.0840256214141846 (nan) | > loss: 0.008484053425490856 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13640331191659608) | > loader_time: 0.0046 (0.004642926166774903)  --> TIME: 2025-02-24 04:40:36 -- STEP: 21848/36938 -- GLOBAL_STEP: 169600 | > loss_text_ce: 0.0444086529314518 (0.04857280760330325) | > loss_mel_ce: 0.939038097858429 (nan) | > loss: 0.0039025668520480394 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13639778009005774) | > loader_time: 0.0046 (0.004642802885014361)  --> TIME: 2025-02-24 04:40:45 -- STEP: 21898/36938 -- GLOBAL_STEP: 169650 | > loss_text_ce: 0.04043840616941452 (0.04856981526890493) | > loss_mel_ce: 2.7902746200561523 (nan) | > loss: 0.01123298890888691 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13639178299143928) | > loader_time: 0.0046 (0.004642842479879688)  --> TIME: 2025-02-24 04:40:54 -- STEP: 21948/36938 -- GLOBAL_STEP: 169700 | > loss_text_ce: 0.046032655984163284 (0.04857223049326932) | > loss_mel_ce: 3.591744899749756 (nan) | > loss: 0.01443562563508749 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.136397744954469) | > loader_time: 0.0047 (0.00464296667915195)  --> TIME: 2025-02-24 04:41:04 -- STEP: 21998/36938 -- GLOBAL_STEP: 169750 | > loss_text_ce: 0.03863648697733879 (0.04856877731832824) | > loss_mel_ce: 3.3141355514526367 (nan) | > loss: 0.013304651714861393 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13640328770020269) | > loader_time: 0.0045 (0.004642935045871642)  --> TIME: 2025-02-24 04:41:13 -- STEP: 22048/36938 -- GLOBAL_STEP: 169800 | > loss_text_ce: 0.04379279166460037 (0.04857003046551124) | > loss_mel_ce: 4.023151397705078 (nan) | > loss: 0.01613866724073887 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13640140821002927) | > loader_time: 0.0047 (0.004642913764118305)  --> TIME: 2025-02-24 04:41:22 -- STEP: 22098/36938 -- GLOBAL_STEP: 169850 | > loss_text_ce: 0.0537477470934391 (0.04856581978111482) | > loss_mel_ce: 3.1246321201324463 (nan) | > loss: 0.012612618505954742 (nan) | > current_lr: 5e-06 | > step_time: 0.1872 (0.1363980735477922) | > loader_time: 0.0049 (0.004642947948575634)  --> TIME: 2025-02-24 04:41:32 -- STEP: 22148/36938 -- GLOBAL_STEP: 169900 | > loss_text_ce: 0.05083222687244415 (0.04856181062077771) | > loss_mel_ce: 3.9555892944335938 (nan) | > loss: 0.01589849963784218 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13639608206937573) | > loader_time: 0.0045 (0.004642874309253218)  --> TIME: 2025-02-24 04:41:41 -- STEP: 22198/36938 -- GLOBAL_STEP: 169950 | > loss_text_ce: 0.043844401836395264 (0.04856268897906373) | > loss_mel_ce: 3.2460882663726807 (nan) | > loss: 0.013055289164185524 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13639934928215935) | > loader_time: 0.0045 (0.0046428170158193316)  --> TIME: 2025-02-24 04:41:50 -- STEP: 22248/36938 -- GLOBAL_STEP: 170000 | > loss_text_ce: 0.05793709680438042 (0.04856189046581189) | > loss_mel_ce: 3.5002543926239014 (nan) | > loss: 0.014119808562099934 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13639813679793225) | > loader_time: 0.0045 (0.004642788828469516) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_170000.pth  --> TIME: 2025-02-24 04:42:05 -- STEP: 22298/36938 -- GLOBAL_STEP: 170050 | > loss_text_ce: 0.04056683927774429 (0.048555549224712495) | > loss_mel_ce: 4.040170669555664 (nan) | > loss: 0.01619340479373932 (nan) | > current_lr: 5e-06 | > step_time: 0.1763 (0.13639306000972928) | > loader_time: 0.0048 (0.004642816250265894)  --> TIME: 2025-02-24 04:42:14 -- STEP: 22348/36938 -- GLOBAL_STEP: 170100 | > loss_text_ce: 0.06097191199660301 (0.04855532605804109) | > loss_mel_ce: 2.181493043899536 (nan) | > loss: 0.00889867078512907 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13638096815233536) | > loader_time: 0.0045 (0.004642957179047521)  --> TIME: 2025-02-24 04:42:24 -- STEP: 22398/36938 -- GLOBAL_STEP: 170150 | > loss_text_ce: 0.05009574815630913 (0.04855356756699216) | > loss_mel_ce: 3.79854154586792 (nan) | > loss: 0.015272371470928192 (nan) | > current_lr: 5e-06 | > step_time: 0.1494 (0.1363746378504601) | > loader_time: 0.0047 (0.004643169734422453)  --> TIME: 2025-02-24 04:42:33 -- STEP: 22448/36938 -- GLOBAL_STEP: 170200 | > loss_text_ce: 0.04316170513629913 (0.04854665227375093) | > loss_mel_ce: 1.7630615234375 (nan) | > loss: 0.007167553063482046 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13637871883082064) | > loader_time: 0.0045 (0.00464330629544522)  --> TIME: 2025-02-24 04:42:42 -- STEP: 22498/36938 -- GLOBAL_STEP: 170250 | > loss_text_ce: 0.04925762116909027 (0.048545622459071104) | > loss_mel_ce: 2.5637333393096924 (nan) | > loss: 0.010369012132287025 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13637692219989156) | > loader_time: 0.0049 (0.004643483017378803)  --> TIME: 2025-02-24 04:42:52 -- STEP: 22548/36938 -- GLOBAL_STEP: 170300 | > loss_text_ce: 0.04624903202056885 (0.04854581996156488) | > loss_mel_ce: 3.3828301429748535 (nan) | > loss: 0.013607457280158997 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13638139079812825) | > loader_time: 0.0046 (0.0046435676505850515)  --> TIME: 2025-02-24 04:43:01 -- STEP: 22598/36938 -- GLOBAL_STEP: 170350 | > loss_text_ce: 0.05417494475841522 (0.048543783609562234) | > loss_mel_ce: 1.533233642578125 (nan) | > loss: 0.00629924051463604 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13638685243152351) | > loader_time: 0.0046 (0.004643825073454905)  --> TIME: 2025-02-24 04:43:10 -- STEP: 22648/36938 -- GLOBAL_STEP: 170400 | > loss_text_ce: 0.046772778034210205 (0.048546794590503074) | > loss_mel_ce: 3.156055450439453 (nan) | > loss: 0.012709636241197586 (nan) | > current_lr: 5e-06 | > step_time: 0.1507 (0.13637867742568488) | > loader_time: 0.0047 (0.004643924284301367)  --> TIME: 2025-02-24 04:43:20 -- STEP: 22698/36938 -- GLOBAL_STEP: 170450 | > loss_text_ce: 0.0473027229309082 (0.048542654271113574) | > loss_mel_ce: 3.486269950866699 (nan) | > loss: 0.01402211468666792 (nan) | > current_lr: 5e-06 | > step_time: 0.1527 (0.13638084242969795) | > loader_time: 0.0047 (0.0046438953930905994)  --> TIME: 2025-02-24 04:43:29 -- STEP: 22748/36938 -- GLOBAL_STEP: 170500 | > loss_text_ce: 0.05250159278512001 (0.04854405034891575) | > loss_mel_ce: 3.1088271141052246 (nan) | > loss: 0.012544956058263779 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13638935816273384) | > loader_time: 0.0046 (0.004644079746670956)  --> TIME: 2025-02-24 04:43:39 -- STEP: 22798/36938 -- GLOBAL_STEP: 170550 | > loss_text_ce: 0.04557735472917557 (0.04854476322212502) | > loss_mel_ce: 4.0676703453063965 (nan) | > loss: 0.016322413459420204 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.1364036721820973) | > loader_time: 0.0048 (0.0046441883295813835)  --> TIME: 2025-02-24 04:43:48 -- STEP: 22848/36938 -- GLOBAL_STEP: 170600 | > loss_text_ce: 0.052261754870414734 (0.04854162569736271) | > loss_mel_ce: 0.0555013008415699 (nan) | > loss: 0.0004276311956346035 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.13640450826045245) | > loader_time: 0.0046 (0.004644392971314654)  --> TIME: 2025-02-24 04:43:57 -- STEP: 22898/36938 -- GLOBAL_STEP: 170650 | > loss_text_ce: 0.04399164393544197 (0.0485387983573221) | > loss_mel_ce: 3.615760326385498 (nan) | > loss: 0.014522825367748737 (nan) | > current_lr: 5e-06 | > step_time: 0.1493 (0.1364058271975844) | > loader_time: 0.0046 (0.00464442694840011)  --> TIME: 2025-02-24 04:44:06 -- STEP: 22948/36938 -- GLOBAL_STEP: 170700 | > loss_text_ce: 0.04736584424972534 (0.04853512623446531) | > loss_mel_ce: 2.8392269611358643 (nan) | > loss: 0.011454734019935131 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.1363942433425899) | > loader_time: 0.0049 (0.0046446640171073144)  --> TIME: 2025-02-24 04:44:16 -- STEP: 22998/36938 -- GLOBAL_STEP: 170750 | > loss_text_ce: 0.05631348490715027 (0.04853122942220914) | > loss_mel_ce: 2.4509329795837402 (nan) | > loss: 0.009949391707777977 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13639907156221726) | > loader_time: 0.0046 (0.004644852584834237)  --> TIME: 2025-02-24 04:44:25 -- STEP: 23048/36938 -- GLOBAL_STEP: 170800 | > loss_text_ce: 0.04695114865899086 (0.0485311082888528) | > loss_mel_ce: 3.078648090362549 (nan) | > loss: 0.012403171509504318 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13640355586964303) | > loader_time: 0.0048 (0.0046449325143442825)  --> TIME: 2025-02-24 04:44:34 -- STEP: 23098/36938 -- GLOBAL_STEP: 170850 | > loss_text_ce: 0.041038788855075836 (0.04852220298817061) | > loss_mel_ce: 3.3038809299468994 (nan) | > loss: 0.013273491524159908 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.1363918420215416) | > loader_time: 0.0045 (0.004644930914930797)  --> TIME: 2025-02-24 04:44:44 -- STEP: 23148/36938 -- GLOBAL_STEP: 170900 | > loss_text_ce: 0.03592758998274803 (0.0485210170436999) | > loss_mel_ce: 6.385416507720947 (nan) | > loss: 0.02548152580857277 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13639183010032496) | > loader_time: 0.0045 (0.004644915963652975)  --> TIME: 2025-02-24 04:44:53 -- STEP: 23198/36938 -- GLOBAL_STEP: 170950 | > loss_text_ce: 0.05052837356925011 (0.048515712899755) | > loss_mel_ce: 3.7948827743530273 (nan) | > loss: 0.015259568579494953 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13638897196610858) | > loader_time: 0.0047 (0.004644875958496389)  --> TIME: 2025-02-24 04:45:02 -- STEP: 23248/36938 -- GLOBAL_STEP: 171000 | > loss_text_ce: 0.04746232554316521 (0.048515031768886566) | > loss_mel_ce: 3.1568078994750977 (nan) | > loss: 0.012715358287096024 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13638257656274697) | > loader_time: 0.0046 (0.004644873301406945)  --> TIME: 2025-02-24 04:45:11 -- STEP: 23298/36938 -- GLOBAL_STEP: 171050 | > loss_text_ce: 0.04907101392745972 (0.04851105733636213) | > loss_mel_ce: 4.2349677085876465 (nan) | > loss: 0.01700015366077423 (nan) | > current_lr: 5e-06 | > step_time: 0.1865 (0.13638521567036999) | > loader_time: 0.0048 (0.0046448941721569615)  --> TIME: 2025-02-24 04:45:21 -- STEP: 23348/36938 -- GLOBAL_STEP: 171100 | > loss_text_ce: 0.046961501240730286 (0.048510654135777126) | > loss_mel_ce: 1.874321699142456 (nan) | > loss: 0.007624140474945307 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.1363889966310047) | > loader_time: 0.0047 (0.0046449691664790784)  --> TIME: 2025-02-24 04:45:30 -- STEP: 23398/36938 -- GLOBAL_STEP: 171150 | > loss_text_ce: 0.03889664262533188 (0.04851325178665848) | > loss_mel_ce: 3.3059120178222656 (nan) | > loss: 0.013273051008582115 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.1363894323964789) | > loader_time: 0.0049 (0.0046450058428973715)  --> TIME: 2025-02-24 04:45:40 -- STEP: 23448/36938 -- GLOBAL_STEP: 171200 | > loss_text_ce: 0.05358850955963135 (0.048513302595409605) | > loss_mel_ce: 4.067242622375488 (nan) | > loss: 0.01635250449180603 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13639165260246014) | > loader_time: 0.0046 (0.004644999484566865)  --> TIME: 2025-02-24 04:45:49 -- STEP: 23498/36938 -- GLOBAL_STEP: 171250 | > loss_text_ce: 0.05393599718809128 (0.048510118842559985) | > loss_mel_ce: 4.184712886810303 (nan) | > loss: 0.016820035874843597 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.13639924558479982) | > loader_time: 0.0047 (0.0046450050042146284)  --> TIME: 2025-02-24 04:45:58 -- STEP: 23548/36938 -- GLOBAL_STEP: 171300 | > loss_text_ce: 0.05004342272877693 (0.04850747716884112) | > loss_mel_ce: 3.5311038494110107 (nan) | > loss: 0.014210902154445648 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13640089147014395) | > loader_time: 0.0046 (0.004644919468426287)  --> TIME: 2025-02-24 04:46:08 -- STEP: 23598/36938 -- GLOBAL_STEP: 171350 | > loss_text_ce: 0.041221484541893005 (0.04850655868971528) | > loss_mel_ce: 3.3778421878814697 (nan) | > loss: 0.013567713089287281 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13641281894368965) | > loader_time: 0.0046 (0.004644836063192659)  --> TIME: 2025-02-24 04:46:18 -- STEP: 23648/36938 -- GLOBAL_STEP: 171400 | > loss_text_ce: 0.03400854393839836 (0.04850448209816637) | > loss_mel_ce: 1.034562349319458 (nan) | > loss: 0.004240360576659441 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13642052205150954) | > loader_time: 0.0045 (0.0046447540793431935)  --> TIME: 2025-02-24 04:46:26 -- STEP: 23698/36938 -- GLOBAL_STEP: 171450 | > loss_text_ce: 0.05502473562955856 (0.04850293876021142) | > loss_mel_ce: 1.4449485540390015 (nan) | > loss: 0.005952275358140469 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13640861794338724) | > loader_time: 0.0046 (0.004644659644230008)  --> TIME: 2025-02-24 04:46:36 -- STEP: 23748/36938 -- GLOBAL_STEP: 171500 | > loss_text_ce: 0.05451243370771408 (0.04850096341480731) | > loss_mel_ce: 2.011536121368408 (nan) | > loss: 0.008198605850338936 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.1364097249654483) | > loader_time: 0.0046 (0.004644625422246587)  --> TIME: 2025-02-24 04:46:45 -- STEP: 23798/36938 -- GLOBAL_STEP: 171550 | > loss_text_ce: 0.04977063462138176 (0.04850025147461677) | > loss_mel_ce: 3.1086623668670654 (nan) | > loss: 0.012533464469015598 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.1364052029495151) | > loader_time: 0.0045 (0.0046445827482528065)  --> TIME: 2025-02-24 04:46:54 -- STEP: 23848/36938 -- GLOBAL_STEP: 171600 | > loss_text_ce: 0.04386718571186066 (0.04849888290889708) | > loss_mel_ce: 3.071117401123047 (nan) | > loss: 0.012361050583422184 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1364043214824208) | > loader_time: 0.0048 (0.004644726435236799)  --> TIME: 2025-02-24 04:47:04 -- STEP: 23898/36938 -- GLOBAL_STEP: 171650 | > loss_text_ce: 0.04386042058467865 (0.04849585910076062) | > loss_mel_ce: 3.449824571609497 (nan) | > loss: 0.013863829895853996 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.1364078996111455) | > loader_time: 0.0047 (0.004644736783535827)  --> TIME: 2025-02-24 04:47:13 -- STEP: 23948/36938 -- GLOBAL_STEP: 171700 | > loss_text_ce: 0.0542098805308342 (0.048492467954038665) | > loss_mel_ce: 2.1293106079101562 (nan) | > loss: 0.00866476446390152 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13640010922385618) | > loader_time: 0.0044 (0.004644735121898244)  --> TIME: 2025-02-24 04:47:22 -- STEP: 23998/36938 -- GLOBAL_STEP: 171750 | > loss_text_ce: 0.04953246936202049 (0.048494990087278145) | > loss_mel_ce: 3.2423458099365234 (nan) | > loss: 0.013063009828329086 (nan) | > current_lr: 5e-06 | > step_time: 0.1829 (0.13640153718020556) | > loader_time: 0.0048 (0.004644722995762843)  --> TIME: 2025-02-24 04:47:32 -- STEP: 24048/36938 -- GLOBAL_STEP: 171800 | > loss_text_ce: 0.046695832163095474 (0.048494056951259586) | > loss_mel_ce: 2.725921630859375 (nan) | > loss: 0.011002451181411743 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13640846112769087) | > loader_time: 0.0047 (0.004644786655228708)  --> TIME: 2025-02-24 04:47:41 -- STEP: 24098/36938 -- GLOBAL_STEP: 171850 | > loss_text_ce: 0.05107800289988518 (0.04849084127434351) | > loss_mel_ce: 3.382744789123535 (nan) | > loss: 0.0136262821033597 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13640546756163163) | > loader_time: 0.0046 (0.0046447130721360965)  --> TIME: 2025-02-24 04:47:50 -- STEP: 24148/36938 -- GLOBAL_STEP: 171900 | > loss_text_ce: 0.05160285532474518 (0.04848784013093322) | > loss_mel_ce: 3.816554307937622 (nan) | > loss: 0.015349830500781536 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.13639512715877622) | > loader_time: 0.0046 (0.004644634264755966)  --> TIME: 2025-02-24 04:47:59 -- STEP: 24198/36938 -- GLOBAL_STEP: 171950 | > loss_text_ce: 0.05099540576338768 (0.04848448738408973) | > loss_mel_ce: 3.3589882850646973 (nan) | > loss: 0.013531682081520557 (nan) | > current_lr: 5e-06 | > step_time: 0.1623 (0.1363987783743678) | > loader_time: 0.0047 (0.004644745508987415)  --> TIME: 2025-02-24 04:48:08 -- STEP: 24248/36938 -- GLOBAL_STEP: 172000 | > loss_text_ce: 0.04381654039025307 (0.0484834644524519) | > loss_mel_ce: 3.011829376220703 (nan) | > loss: 0.01212557964026928 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13639466801282868) | > loader_time: 0.0046 (0.004644829894166244)  --> TIME: 2025-02-24 04:48:18 -- STEP: 24298/36938 -- GLOBAL_STEP: 172050 | > loss_text_ce: 0.05008067563176155 (0.04848257080517746) | > loss_mel_ce: 0.025146484375 (nan) | > loss: 0.0002985204919241369 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13638795236568288) | > loader_time: 0.0048 (0.004644847385225335)  --> TIME: 2025-02-24 04:48:27 -- STEP: 24348/36938 -- GLOBAL_STEP: 172100 | > loss_text_ce: 0.06507539004087448 (0.04848284043857613) | > loss_mel_ce: 2.7397494316101074 (nan) | > loss: 0.0111302575096488 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.136378316362641) | > loader_time: 0.0047 (0.004644807403029267)  --> TIME: 2025-02-24 04:48:36 -- STEP: 24398/36938 -- GLOBAL_STEP: 172150 | > loss_text_ce: 0.04041450470685959 (0.048482881115123624) | > loss_mel_ce: 3.68277907371521 (nan) | > loss: 0.014774578623473644 (nan) | > current_lr: 5e-06 | > step_time: 0.1664 (0.13636758944296284) | > loader_time: 0.0047 (0.004644737848347071)  --> TIME: 2025-02-24 04:48:45 -- STEP: 24448/36938 -- GLOBAL_STEP: 172200 | > loss_text_ce: 0.042300209403038025 (0.04847946726125533) | > loss_mel_ce: 3.6325559616088867 (nan) | > loss: 0.01458276342600584 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13637532337680952) | > loader_time: 0.0046 (0.004644908810629283)  --> TIME: 2025-02-24 04:48:54 -- STEP: 24498/36938 -- GLOBAL_STEP: 172250 | > loss_text_ce: 0.03197190538048744 (0.04847761101150018) | > loss_mel_ce: 3.1673226356506348 (nan) | > loss: 0.012695614248514175 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13637122100436536) | > loader_time: 0.0046 (0.0046449718947157315)  --> TIME: 2025-02-24 04:49:04 -- STEP: 24548/36938 -- GLOBAL_STEP: 172300 | > loss_text_ce: 0.04463403671979904 (0.048474633188547056) | > loss_mel_ce: 3.611917495727539 (nan) | > loss: 0.01451012585312128 (nan) | > current_lr: 5e-06 | > step_time: 0.151 (0.13636874023042597) | > loader_time: 0.0048 (0.004644934053391515)  --> TIME: 2025-02-24 04:49:12 -- STEP: 24598/36938 -- GLOBAL_STEP: 172350 | > loss_text_ce: 0.029001127928495407 (0.04847267757209508) | > loss_mel_ce: 2.2185704708099365 (nan) | > loss: 0.008918935433030128 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1363586451970667) | > loader_time: 0.0046 (0.004645068991115437)  --> TIME: 2025-02-24 04:49:22 -- STEP: 24648/36938 -- GLOBAL_STEP: 172400 | > loss_text_ce: 0.04771820083260536 (0.0484695635827126) | > loss_mel_ce: 2.973884344100952 (nan) | > loss: 0.011990487575531006 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13636564110556684) | > loader_time: 0.0046 (0.004645154610426171)  --> TIME: 2025-02-24 04:49:31 -- STEP: 24698/36938 -- GLOBAL_STEP: 172450 | > loss_text_ce: 0.05328471586108208 (0.048470412103308574) | > loss_mel_ce: 3.836517572402954 (nan) | > loss: 0.015435723587870598 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13636822697218454) | > loader_time: 0.0046 (0.0046451519506548675)  --> TIME: 2025-02-24 04:49:41 -- STEP: 24748/36938 -- GLOBAL_STEP: 172500 | > loss_text_ce: 0.04597744718194008 (0.048467347474191645) | > loss_mel_ce: 3.152634620666504 (nan) | > loss: 0.012692905031144619 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13636785926737255) | > loader_time: 0.0047 (0.004645244724938974)  --> TIME: 2025-02-24 04:49:50 -- STEP: 24798/36938 -- GLOBAL_STEP: 172550 | > loss_text_ce: 0.05075312405824661 (0.04846179247236269) | > loss_mel_ce: 2.820207118988037 (nan) | > loss: 0.011392699554562569 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.13636251008936615) | > loader_time: 0.0048 (0.004645226636099459)  --> TIME: 2025-02-24 04:49:59 -- STEP: 24848/36938 -- GLOBAL_STEP: 172600 | > loss_text_ce: 0.049907881766557693 (0.0484602020284178) | > loss_mel_ce: 2.795673370361328 (nan) | > loss: 0.01129199005663395 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13636387152512267) | > loader_time: 0.0046 (0.004645358706318956)  --> TIME: 2025-02-24 04:50:08 -- STEP: 24898/36938 -- GLOBAL_STEP: 172650 | > loss_text_ce: 0.05264635756611824 (0.048462521220256745) | > loss_mel_ce: 3.576955795288086 (nan) | > loss: 0.014403183944523335 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13636361036791814) | > loader_time: 0.0045 (0.0046454878234128)  --> TIME: 2025-02-24 04:50:18 -- STEP: 24948/36938 -- GLOBAL_STEP: 172700 | > loss_text_ce: 0.026990696787834167 (0.0484602633059289) | > loss_mel_ce: 3.2442891597747803 (nan) | > loss: 0.012981269508600235 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.1363599085742826) | > loader_time: 0.0045 (0.004645602030690648)  --> TIME: 2025-02-24 04:50:27 -- STEP: 24998/36938 -- GLOBAL_STEP: 172750 | > loss_text_ce: 0.042983803898096085 (0.04845381282183225) | > loss_mel_ce: 3.5191731452941895 (nan) | > loss: 0.014135544188320637 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13635437260037375) | > loader_time: 0.0045 (0.00464554511696752)  --> TIME: 2025-02-24 04:50:36 -- STEP: 25048/36938 -- GLOBAL_STEP: 172800 | > loss_text_ce: 0.0512424111366272 (0.048452445642674064) | > loss_mel_ce: 1.873725175857544 (nan) | > loss: 0.0076387603767216206 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13635400690540608) | > loader_time: 0.0047 (0.004645548644289742)  --> TIME: 2025-02-24 04:50:45 -- STEP: 25098/36938 -- GLOBAL_STEP: 172850 | > loss_text_ce: 0.058780066668987274 (0.048452081356094606) | > loss_mel_ce: 3.0538523197174072 (nan) | > loss: 0.012351715937256813 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13635344886810305) | > loader_time: 0.0046 (0.004645575573837847)  --> TIME: 2025-02-24 04:50:55 -- STEP: 25148/36938 -- GLOBAL_STEP: 172900 | > loss_text_ce: 0.043997496366500854 (0.04845404558048763) | > loss_mel_ce: 3.148815870285034 (nan) | > loss: 0.012669894844293594 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13634970810259142) | > loader_time: 0.0046 (0.004645634270139094)  --> TIME: 2025-02-24 04:51:04 -- STEP: 25198/36938 -- GLOBAL_STEP: 172950 | > loss_text_ce: 0.060007594525814056 (0.04845219928175017) | > loss_mel_ce: 0.046445343643426895 (nan) | > loss: 0.00042243232019245625 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13635021553634624) | > loader_time: 0.0049 (0.004645756515532003)  --> TIME: 2025-02-24 04:51:13 -- STEP: 25248/36938 -- GLOBAL_STEP: 173000 | > loss_text_ce: 0.042998962104320526 (0.04845044081025117) | > loss_mel_ce: 3.651812791824341 (nan) | > loss: 0.01466195285320282 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.1363546099650705) | > loader_time: 0.0047 (0.004645833006678731)  --> TIME: 2025-02-24 04:51:23 -- STEP: 25298/36938 -- GLOBAL_STEP: 173050 | > loss_text_ce: 0.043275922536849976 (0.04844454027870942) | > loss_mel_ce: 3.007960796356201 (nan) | > loss: 0.012108082883059978 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13635630582724664) | > loader_time: 0.0045 (0.004646032994439443)  --> TIME: 2025-02-24 04:51:32 -- STEP: 25348/36938 -- GLOBAL_STEP: 173100 | > loss_text_ce: 0.03786135092377663 (0.048440336075650965) | > loss_mel_ce: 0.08661498874425888 (nan) | > loss: 0.0004939537611790001 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13636286690048252) | > loader_time: 0.0047 (0.004646122427379487)  --> TIME: 2025-02-24 04:51:41 -- STEP: 25398/36938 -- GLOBAL_STEP: 173150 | > loss_text_ce: 0.051021598279476166 (0.048438337524413094) | > loss_mel_ce: 3.4907987117767334 (nan) | > loss: 0.014054843224585056 (nan) | > current_lr: 5e-06 | > step_time: 0.1812 (0.1363611365971741) | > loader_time: 0.0049 (0.004646255281161341)  --> TIME: 2025-02-24 04:51:50 -- STEP: 25448/36938 -- GLOBAL_STEP: 173200 | > loss_text_ce: 0.044901490211486816 (0.04843515564106537) | > loss_mel_ce: 3.6373283863067627 (nan) | > loss: 0.014612024649977684 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13635255869961757) | > loader_time: 0.0046 (0.004646179586864428)  --> TIME: 2025-02-24 04:52:00 -- STEP: 25498/36938 -- GLOBAL_STEP: 173250 | > loss_text_ce: 0.043110813945531845 (0.048433637349746615) | > loss_mel_ce: 3.0732390880584717 (nan) | > loss: 0.012366469018161297 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13636039539322012) | > loader_time: 0.0045 (0.004646130417531984)  --> TIME: 2025-02-24 04:52:09 -- STEP: 25548/36938 -- GLOBAL_STEP: 173300 | > loss_text_ce: 0.047322433441877365 (0.048427626491358504) | > loss_mel_ce: 3.7718448638916016 (nan) | > loss: 0.015155427157878876 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13635260407981126) | > loader_time: 0.0046 (0.004646206473229137)  --> TIME: 2025-02-24 04:52:18 -- STEP: 25598/36938 -- GLOBAL_STEP: 173350 | > loss_text_ce: 0.05238904431462288 (0.04843009481867538) | > loss_mel_ce: 4.151779651641846 (nan) | > loss: 0.01668320968747139 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13636189313109714) | > loader_time: 0.0046 (0.004646381434728159)  --> TIME: 2025-02-24 04:52:28 -- STEP: 25648/36938 -- GLOBAL_STEP: 173400 | > loss_text_ce: 0.06301973015069962 (0.048430588572753494) | > loss_mel_ce: 2.860053539276123 (nan) | > loss: 0.011599497869610786 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13636639562719663) | > loader_time: 0.0048 (0.004646394088055599)  --> TIME: 2025-02-24 04:52:37 -- STEP: 25698/36938 -- GLOBAL_STEP: 173450 | > loss_text_ce: 0.0412517674267292 (0.04842846755464441) | > loss_mel_ce: 3.8089053630828857 (nan) | > loss: 0.015278401784598827 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13637237312617753) | > loader_time: 0.0046 (0.004646413334984483)  --> TIME: 2025-02-24 04:52:47 -- STEP: 25748/36938 -- GLOBAL_STEP: 173500 | > loss_text_ce: 0.050083842128515244 (0.0484248629588656) | > loss_mel_ce: 3.9723362922668457 (nan) | > loss: 0.015961984172463417 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.13637211672986135) | > loader_time: 0.0049 (0.004646465980952926)  --> TIME: 2025-02-24 04:52:56 -- STEP: 25798/36938 -- GLOBAL_STEP: 173550 | > loss_text_ce: 0.04746909439563751 (0.048418341675095025) | > loss_mel_ce: 2.508859157562256 (nan) | > loss: 0.010144160129129887 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1363717632355078) | > loader_time: 0.0046 (0.004646496963515772)  --> TIME: 2025-02-24 04:53:05 -- STEP: 25848/36938 -- GLOBAL_STEP: 173600 | > loss_text_ce: 0.054280251264572144 (0.04841343698735892) | > loss_mel_ce: 1.738339900970459 (nan) | > loss: 0.007113572675734758 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13636955321849856) | > loader_time: 0.0046 (0.004646653446093032)  --> TIME: 2025-02-24 04:53:14 -- STEP: 25898/36938 -- GLOBAL_STEP: 173650 | > loss_text_ce: 0.05066504329442978 (0.0484141042475522) | > loss_mel_ce: 2.648350954055786 (nan) | > loss: 0.010710381902754307 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13636233455122795) | > loader_time: 0.0044 (0.0046465618839244835)  --> TIME: 2025-02-24 04:53:23 -- STEP: 25948/36938 -- GLOBAL_STEP: 173700 | > loss_text_ce: 0.041907742619514465 (0.04841242110150958) | > loss_mel_ce: 3.521524429321289 (nan) | > loss: 0.014140604995191097 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13635856492026635) | > loader_time: 0.0047 (0.004646637874519186)  --> TIME: 2025-02-24 04:53:33 -- STEP: 25998/36938 -- GLOBAL_STEP: 173750 | > loss_text_ce: 0.0424463264644146 (0.04841024191942346) | > loss_mel_ce: 3.7496180534362793 (nan) | > loss: 0.015047875232994556 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13636330072949054) | > loader_time: 0.0046 (0.004646678018206794)  --> TIME: 2025-02-24 04:53:42 -- STEP: 26048/36938 -- GLOBAL_STEP: 173800 | > loss_text_ce: 0.04725966230034828 (0.0484089925385716) | > loss_mel_ce: 2.535806179046631 (nan) | > loss: 0.010250261053442955 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13636931393487858) | > loader_time: 0.0062 (0.004646804266098975)  --> TIME: 2025-02-24 04:53:51 -- STEP: 26098/36938 -- GLOBAL_STEP: 173850 | > loss_text_ce: 0.06320062279701233 (0.04840770059177626) | > loss_mel_ce: 3.367887020111084 (nan) | > loss: 0.01361542847007513 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13635673721372651) | > loader_time: 0.0047 (0.004646853739589161)  --> TIME: 2025-02-24 04:54:00 -- STEP: 26148/36938 -- GLOBAL_STEP: 173900 | > loss_text_ce: 0.04693406820297241 (0.048410326975843854) | > loss_mel_ce: 3.1310269832611084 (nan) | > loss: 0.012610957957804203 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13635467941359583) | > loader_time: 0.0045 (0.004646847494993998)  --> TIME: 2025-02-24 04:54:10 -- STEP: 26198/36938 -- GLOBAL_STEP: 173950 | > loss_text_ce: 0.03319571912288666 (0.04840737076334589) | > loss_mel_ce: 2.27553391456604 (nan) | > loss: 0.009161625988781452 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13635247956251387) | > loader_time: 0.0045 (0.004646809804220233)  --> TIME: 2025-02-24 04:54:19 -- STEP: 26248/36938 -- GLOBAL_STEP: 174000 | > loss_text_ce: 0.03920697420835495 (0.048407219743865075) | > loss_mel_ce: 1.6701217889785767 (nan) | > loss: 0.006783050950616598 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.1363452520631368) | > loader_time: 0.0045 (0.004646716485551071)  --> TIME: 2025-02-24 04:54:28 -- STEP: 26298/36938 -- GLOBAL_STEP: 174050 | > loss_text_ce: 0.04314062371850014 (0.04841024155559028) | > loss_mel_ce: 0.2680460512638092 (nan) | > loss: 0.0012348678428679705 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13633967949071518) | > loader_time: 0.0047 (0.004646629505315594)  --> TIME: 2025-02-24 04:54:37 -- STEP: 26348/36938 -- GLOBAL_STEP: 174100 | > loss_text_ce: 0.05224420502781868 (0.04841137678035542) | > loss_mel_ce: 2.508563756942749 (nan) | > loss: 0.010161937214434147 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13633809297602878) | > loader_time: 0.0045 (0.00464651710223009)  --> TIME: 2025-02-24 04:54:47 -- STEP: 26398/36938 -- GLOBAL_STEP: 174150 | > loss_text_ce: 0.0427822545170784 (0.04840973166714167) | > loss_mel_ce: 2.8697869777679443 (nan) | > loss: 0.011557815596461296 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.1363483854239356) | > loader_time: 0.0044 (0.004646451231728676)  --> TIME: 2025-02-24 04:54:56 -- STEP: 26448/36938 -- GLOBAL_STEP: 174200 | > loss_text_ce: 0.04513131454586983 (0.04840878238185258) | > loss_mel_ce: 3.5322582721710205 (nan) | > loss: 0.01419599074870348 (nan) | > current_lr: 5e-06 | > step_time: 0.1495 (0.1363554731128592) | > loader_time: 0.0046 (0.0046463270603492686)  --> TIME: 2025-02-24 04:55:06 -- STEP: 26498/36938 -- GLOBAL_STEP: 174250 | > loss_text_ce: 0.04571601748466492 (0.048405354760475604) | > loss_mel_ce: 4.400582790374756 (nan) | > loss: 0.017644042149186134 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13636074894661837) | > loader_time: 0.0046 (0.004646180656615077)  --> TIME: 2025-02-24 04:55:15 -- STEP: 26548/36938 -- GLOBAL_STEP: 174300 | > loss_text_ce: 0.06821727007627487 (0.04840244241769563) | > loss_mel_ce: 1.5994973182678223 (nan) | > loss: 0.006617915350943804 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13635242353528246) | > loader_time: 0.0047 (0.004645973340707388)  --> TIME: 2025-02-24 04:55:24 -- STEP: 26598/36938 -- GLOBAL_STEP: 174350 | > loss_text_ce: 0.04823720455169678 (0.04839957409037409) | > loss_mel_ce: 3.603849172592163 (nan) | > loss: 0.014492406509816647 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.1363554614694053) | > loader_time: 0.0045 (0.004645830841474635)  --> TIME: 2025-02-24 04:55:34 -- STEP: 26648/36938 -- GLOBAL_STEP: 174400 | > loss_text_ce: 0.04391423240303993 (0.04839697129966699) | > loss_mel_ce: 3.750580310821533 (nan) | > loss: 0.0150575190782547 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.13636007862061128) | > loader_time: 0.0045 (0.004645707477717311)  --> TIME: 2025-02-24 04:55:43 -- STEP: 26698/36938 -- GLOBAL_STEP: 174450 | > loss_text_ce: 0.04385463520884514 (0.048395709554811386) | > loss_mel_ce: 3.565598964691162 (nan) | > loss: 0.014323229901492596 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.13636128154441562) | > loader_time: 0.0046 (0.004645563081030775)  --> TIME: 2025-02-24 04:55:53 -- STEP: 26748/36938 -- GLOBAL_STEP: 174500 | > loss_text_ce: 0.05077134445309639 (0.04839240760575601) | > loss_mel_ce: 3.827934741973877 (nan) | > loss: 0.015391691587865353 (nan) | > current_lr: 5e-06 | > step_time: 0.1902 (0.13637304295334202) | > loader_time: 0.0048 (0.0046454900152857635)  --> TIME: 2025-02-24 04:56:02 -- STEP: 26798/36938 -- GLOBAL_STEP: 174550 | > loss_text_ce: 0.055539652705192566 (0.04839198957542471) | > loss_mel_ce: 2.3712642192840576 (nan) | > loss: 0.0096301743760705 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13637396395851614) | > loader_time: 0.0047 (0.004645365664777285)  --> TIME: 2025-02-24 04:56:12 -- STEP: 26848/36938 -- GLOBAL_STEP: 174600 | > loss_text_ce: 0.0478961355984211 (0.048386167500261625) | > loss_mel_ce: 3.296133041381836 (nan) | > loss: 0.013269958086311817 (nan) | > current_lr: 5e-06 | > step_time: 0.1618 (0.13638586747206005) | > loader_time: 0.0047 (0.004645277209872998)  --> TIME: 2025-02-24 04:56:21 -- STEP: 26898/36938 -- GLOBAL_STEP: 174650 | > loss_text_ce: 0.03748323768377304 (0.04838457470683578) | > loss_mel_ce: 3.4357008934020996 (nan) | > loss: 0.013782477006316185 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13639357738117763) | > loader_time: 0.0045 (0.004645175211969758)  --> TIME: 2025-02-24 04:56:30 -- STEP: 26948/36938 -- GLOBAL_STEP: 174700 | > loss_text_ce: 0.040442898869514465 (0.048381241789672146) | > loss_mel_ce: 2.9511892795562744 (nan) | > loss: 0.011871556751430035 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.1363936751236008) | > loader_time: 0.0046 (0.0046450569506867895)  --> TIME: 2025-02-24 04:56:40 -- STEP: 26998/36938 -- GLOBAL_STEP: 174750 | > loss_text_ce: 0.04586496204137802 (0.048379829735469296) | > loss_mel_ce: 3.8524787425994873 (nan) | > loss: 0.015469619072973728 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.13639894852206375) | > loader_time: 0.0048 (0.004644896367875127)  --> TIME: 2025-02-24 04:56:49 -- STEP: 27048/36938 -- GLOBAL_STEP: 174800 | > loss_text_ce: 0.03070989064872265 (0.048375422882553316) | > loss_mel_ce: 0.192047119140625 (nan) | > loss: 0.0008839564397931099 (nan) | > current_lr: 5e-06 | > step_time: 0.1271 (0.13638907353647575) | > loader_time: 0.0046 (0.004644711433307409)  --> TIME: 2025-02-24 04:56:58 -- STEP: 27098/36938 -- GLOBAL_STEP: 174850 | > loss_text_ce: 0.047031838446855545 (0.04837390596882342) | > loss_mel_ce: 4.044521331787109 (nan) | > loss: 0.0162363238632679 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13639097626741362) | > loader_time: 0.0046 (0.004644593529810691)  --> TIME: 2025-02-24 04:57:08 -- STEP: 27148/36938 -- GLOBAL_STEP: 174900 | > loss_text_ce: 0.04807785898447037 (0.04836853442530978) | > loss_mel_ce: 3.5647525787353516 (nan) | > loss: 0.014336629770696163 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13639662608107056) | > loader_time: 0.0046 (0.004644470352197027)  --> TIME: 2025-02-24 04:57:17 -- STEP: 27198/36938 -- GLOBAL_STEP: 174950 | > loss_text_ce: 0.03797757625579834 (0.0483667403248274) | > loss_mel_ce: 2.1355998516082764 (nan) | > loss: 0.008625307120382786 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.1363914488378606) | > loader_time: 0.0044 (0.0046442990285858205)  --> TIME: 2025-02-24 04:57:26 -- STEP: 27248/36938 -- GLOBAL_STEP: 175000 | > loss_text_ce: 0.043793752789497375 (0.048363276927753) | > loss_mel_ce: 3.025106906890869 (nan) | > loss: 0.012178177945315838 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13638837737891643) | > loader_time: 0.0046 (0.004644167034750613)  --> TIME: 2025-02-24 04:57:35 -- STEP: 27298/36938 -- GLOBAL_STEP: 175050 | > loss_text_ce: 0.05129818245768547 (0.04835978248225184) | > loss_mel_ce: 3.3832616806030273 (nan) | > loss: 0.013629206456243992 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13638491281721532) | > loader_time: 0.0045 (0.004644179887427146)  --> TIME: 2025-02-24 04:57:45 -- STEP: 27348/36938 -- GLOBAL_STEP: 175100 | > loss_text_ce: 0.048400189727544785 (0.04835707856102046) | > loss_mel_ce: 3.129021167755127 (nan) | > loss: 0.012608815915882587 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.1363898277614012) | > loader_time: 0.0045 (0.004644123149995695)  --> TIME: 2025-02-24 04:57:54 -- STEP: 27398/36938 -- GLOBAL_STEP: 175150 | > loss_text_ce: 0.049099937081336975 (0.04835612161504613) | > loss_mel_ce: 2.229020118713379 (nan) | > loss: 0.009040159173309803 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13639440478369538) | > loader_time: 0.0044 (0.004644035414123425)  --> TIME: 2025-02-24 04:58:03 -- STEP: 27448/36938 -- GLOBAL_STEP: 175200 | > loss_text_ce: 0.02701738104224205 (0.04835475417525441) | > loss_mel_ce: 3.9908640384674072 (nan) | > loss: 0.015943974256515503 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13639186830397101) | > loader_time: 0.0048 (0.004644006968447925)  --> TIME: 2025-02-24 04:58:13 -- STEP: 27498/36938 -- GLOBAL_STEP: 175250 | > loss_text_ce: 0.043004799634218216 (0.04835596794518036) | > loss_mel_ce: 0.7974938154220581 (nan) | > loss: 0.0033353122416883707 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13639278395044813) | > loader_time: 0.0045 (0.004643900818075307)  --> TIME: 2025-02-24 04:58:22 -- STEP: 27548/36938 -- GLOBAL_STEP: 175300 | > loss_text_ce: 0.05762159824371338 (0.04835442255420959) | > loss_mel_ce: 1.573797345161438 (nan) | > loss: 0.006473884917795658 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13638551887414305) | > loader_time: 0.0045 (0.004643781023827288)  --> TIME: 2025-02-24 04:58:31 -- STEP: 27598/36938 -- GLOBAL_STEP: 175350 | > loss_text_ce: 0.04231645166873932 (0.048356143347788066) | > loss_mel_ce: 3.8301119804382324 (nan) | > loss: 0.015366780571639538 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1363862923966252) | > loader_time: 0.0045 (0.0046436497418550025)  --> TIME: 2025-02-24 04:58:41 -- STEP: 27648/36938 -- GLOBAL_STEP: 175400 | > loss_text_ce: 0.04862402379512787 (0.048356643710677535) | > loss_mel_ce: 0.0135498046875 (nan) | > loss: 0.00024672155268490314 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13639575118820005) | > loader_time: 0.0045 (0.004643744901167583)  --> TIME: 2025-02-24 04:58:50 -- STEP: 27698/36938 -- GLOBAL_STEP: 175450 | > loss_text_ce: 0.03238803893327713 (0.048353665153695644) | > loss_mel_ce: 2.841869831085205 (nan) | > loss: 0.01140578556805849 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13638877436541394) | > loader_time: 0.0045 (0.004643696698482075)  --> TIME: 2025-02-24 04:58:59 -- STEP: 27748/36938 -- GLOBAL_STEP: 175500 | > loss_text_ce: 0.04921463876962662 (0.04835399039434312) | > loss_mel_ce: 3.3252270221710205 (nan) | > loss: 0.01339064259082079 (nan) | > current_lr: 5e-06 | > step_time: 0.1651 (0.13639148677548632) | > loader_time: 0.0048 (0.004643683347953617)  --> TIME: 2025-02-24 04:59:09 -- STEP: 27798/36938 -- GLOBAL_STEP: 175550 | > loss_text_ce: 0.04486146196722984 (0.0483524474227969) | > loss_mel_ce: 2.29791259765625 (nan) | > loss: 0.009296723641455173 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.1363937766339796) | > loader_time: 0.0045 (0.004643693322954928)  --> TIME: 2025-02-24 04:59:18 -- STEP: 27848/36938 -- GLOBAL_STEP: 175600 | > loss_text_ce: 0.04116920009255409 (0.04835102247857217) | > loss_mel_ce: 3.448272466659546 (nan) | > loss: 0.013846991583704948 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.1363947316255094) | > loader_time: 0.0046 (0.0046436353015406615)  --> TIME: 2025-02-24 04:59:27 -- STEP: 27898/36938 -- GLOBAL_STEP: 175650 | > loss_text_ce: 0.04168781265616417 (0.048350578197663675) | > loss_mel_ce: 3.038410186767578 (nan) | > loss: 0.012222611345350742 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13639362175085842) | > loader_time: 0.0048 (0.0046435380821219915)  --> TIME: 2025-02-24 04:59:37 -- STEP: 27948/36938 -- GLOBAL_STEP: 175700 | > loss_text_ce: 0.0476791150867939 (0.04835031969815732) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13640061126206568) | > loader_time: 0.0046 (0.004643528122272012)  --> TIME: 2025-02-24 04:59:46 -- STEP: 27998/36938 -- GLOBAL_STEP: 175750 | > loss_text_ce: 0.0328451506793499 (0.048349834737982725) | > loss_mel_ce: 3.082294225692749 (nan) | > loss: 0.01236166525632143 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13639873030765073) | > loader_time: 0.0045 (0.004643442937498213)  --> TIME: 2025-02-24 04:59:55 -- STEP: 28048/36938 -- GLOBAL_STEP: 175800 | > loss_text_ce: 0.047011326998472214 (0.04835006627859177) | > loss_mel_ce: 3.9334421157836914 (nan) | > loss: 0.015795450657606125 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13639581899641723) | > loader_time: 0.0045 (0.004643381908493465)  --> TIME: 2025-02-24 05:00:05 -- STEP: 28098/36938 -- GLOBAL_STEP: 175850 | > loss_text_ce: 0.035962846130132675 (0.048349600522712984) | > loss_mel_ce: 2.791533946990967 (nan) | > loss: 0.011220226064324379 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13639456061283878) | > loader_time: 0.0047 (0.0046432905328155916)  --> TIME: 2025-02-24 05:00:13 -- STEP: 28148/36938 -- GLOBAL_STEP: 175900 | > loss_text_ce: 0.04902608320116997 (0.04834677357235479) | > loss_mel_ce: 3.078082323074341 (nan) | > loss: 0.01240916084498167 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.1363806411998776) | > loader_time: 0.0046 (0.0046431394112589645)  --> TIME: 2025-02-24 05:00:23 -- STEP: 28198/36938 -- GLOBAL_STEP: 175950 | > loss_text_ce: 0.05058453604578972 (0.04834418279452486) | > loss_mel_ce: 3.8412492275238037 (nan) | > loss: 0.015443786047399044 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.1363832397370992) | > loader_time: 0.0046 (0.00464309674728441)  --> TIME: 2025-02-24 05:00:32 -- STEP: 28248/36938 -- GLOBAL_STEP: 176000 | > loss_text_ce: 0.027143046259880066 (0.04834153900623636) | > loss_mel_ce: 3.5856332778930664 (nan) | > loss: 0.014336414635181427 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13637953315534906) | > loader_time: 0.0046 (0.004643119316673666)  --> TIME: 2025-02-24 05:00:42 -- STEP: 28298/36938 -- GLOBAL_STEP: 176050 | > loss_text_ce: 0.03151726350188255 (0.048341781468707205) | > loss_mel_ce: 3.789306640625 (nan) | > loss: 0.015162000432610512 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13638785047340565) | > loader_time: 0.0047 (0.004643161176025973)  --> TIME: 2025-02-24 05:00:51 -- STEP: 28348/36938 -- GLOBAL_STEP: 176100 | > loss_text_ce: 0.054448921233415604 (0.04833815222413808) | > loss_mel_ce: 2.8779234886169434 (nan) | > loss: 0.011636398732662201 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1363868526667606) | > loader_time: 0.0047 (0.004643162997094948)  --> TIME: 2025-02-24 05:01:00 -- STEP: 28398/36938 -- GLOBAL_STEP: 176150 | > loss_text_ce: 0.04835205152630806 (0.04833735051902076) | > loss_mel_ce: 1.3847572803497314 (nan) | > loss: 0.005686942022293806 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.1363873957026081) | > loader_time: 0.0045 (0.004643210366339968)  --> TIME: 2025-02-24 05:01:10 -- STEP: 28448/36938 -- GLOBAL_STEP: 176200 | > loss_text_ce: 0.05833149328827858 (0.04833570090763527) | > loss_mel_ce: 3.6958725452423096 (nan) | > loss: 0.014897636137902737 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13639009189116452) | > loader_time: 0.0047 (0.0046433291918649535)  --> TIME: 2025-02-24 05:01:19 -- STEP: 28498/36938 -- GLOBAL_STEP: 176250 | > loss_text_ce: 0.05095149949193001 (0.04833344447758727) | > loss_mel_ce: 3.546757698059082 (nan) | > loss: 0.014276624657213688 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.1363982135885297) | > loader_time: 0.0045 (0.00464327840338558)  --> TIME: 2025-02-24 05:01:28 -- STEP: 28548/36938 -- GLOBAL_STEP: 176300 | > loss_text_ce: 0.05250726267695427 (0.04833134538808538) | > loss_mel_ce: 3.538769483566284 (nan) | > loss: 0.014251098968088627 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13639471201698408) | > loader_time: 0.0045 (0.004643154583575939)  --> TIME: 2025-02-24 05:01:37 -- STEP: 28598/36938 -- GLOBAL_STEP: 176350 | > loss_text_ce: 0.05914271995425224 (0.048330690462769596) | > loss_mel_ce: 2.631190538406372 (nan) | > loss: 0.010675926692783833 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13638492283833617) | > loader_time: 0.0045 (0.004643001550727404)  --> TIME: 2025-02-24 05:01:47 -- STEP: 28648/36938 -- GLOBAL_STEP: 176400 | > loss_text_ce: 0.05447517707943916 (0.048329242979734476) | > loss_mel_ce: 2.7398757934570312 (nan) | > loss: 0.01108869444578886 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.1363887711131899) | > loader_time: 0.0047 (0.004643439081453547)  --> TIME: 2025-02-24 05:01:56 -- STEP: 28698/36938 -- GLOBAL_STEP: 176450 | > loss_text_ce: 0.04574311152100563 (0.04832625630263575) | > loss_mel_ce: 2.8535878658294678 (nan) | > loss: 0.01150528248399496 (nan) | > current_lr: 5e-06 | > step_time: 0.1752 (0.13637944958967677) | > loader_time: 0.0048 (0.004643379964847774)  --> TIME: 2025-02-24 05:02:05 -- STEP: 28748/36938 -- GLOBAL_STEP: 176500 | > loss_text_ce: 0.051888611167669296 (0.04832492775097932) | > loss_mel_ce: 2.9965920448303223 (nan) | > loss: 0.012097146362066269 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13637328809701452) | > loader_time: 0.0049 (0.004643436970690196)  --> TIME: 2025-02-24 05:02:14 -- STEP: 28798/36938 -- GLOBAL_STEP: 176550 | > loss_text_ce: 0.04349040240049362 (0.04831999670629411) | > loss_mel_ce: 0.9074918031692505 (nan) | > loss: 0.003773739095777273 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13638357305967355) | > loader_time: 0.0047 (0.004643531837134877)  --> TIME: 2025-02-24 05:02:24 -- STEP: 28848/36938 -- GLOBAL_STEP: 176600 | > loss_text_ce: 0.039406776428222656 (0.04832013466820165) | > loss_mel_ce: 2.5146987438201904 (nan) | > loss: 0.010135339573025703 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13638846872088434) | > loader_time: 0.0045 (0.004643543893538519)  --> TIME: 2025-02-24 05:02:33 -- STEP: 28898/36938 -- GLOBAL_STEP: 176650 | > loss_text_ce: 0.04780260846018791 (0.048319374202978875) | > loss_mel_ce: 3.094003200531006 (nan) | > loss: 0.01246748398989439 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.1363826659123589) | > loader_time: 0.0046 (0.004643464997628386)  --> TIME: 2025-02-24 05:02:43 -- STEP: 28948/36938 -- GLOBAL_STEP: 176700 | > loss_text_ce: 0.046246130019426346 (0.04831872014809493) | > loss_mel_ce: 4.033062934875488 (nan) | > loss: 0.01618773490190506 (nan) | > current_lr: 5e-06 | > step_time: 0.1891 (0.1363925193659737) | > loader_time: 0.005 (0.004643708273935215)  --> TIME: 2025-02-24 05:02:52 -- STEP: 28998/36938 -- GLOBAL_STEP: 176750 | > loss_text_ce: 0.046954311430454254 (0.048317584316960184) | > loss_mel_ce: 3.659627914428711 (nan) | > loss: 0.014708660542964935 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13638391373231515) | > loader_time: 0.0048 (0.0046436631339970644)  --> TIME: 2025-02-24 05:03:01 -- STEP: 29048/36938 -- GLOBAL_STEP: 176800 | > loss_text_ce: 0.04957177862524986 (0.04831557915714901) | > loss_mel_ce: 4.455388069152832 (nan) | > loss: 0.017876826226711273 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.1363808456464648) | > loader_time: 0.0047 (0.0046436990614076665)  --> TIME: 2025-02-24 05:03:10 -- STEP: 29098/36938 -- GLOBAL_STEP: 176850 | > loss_text_ce: 0.037761230021715164 (0.04831454176377002) | > loss_mel_ce: 2.352158546447754 (nan) | > loss: 0.009483808651566505 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13638920336296198) | > loader_time: 0.0044 (0.004643761609393606)  --> TIME: 2025-02-24 05:03:19 -- STEP: 29148/36938 -- GLOBAL_STEP: 176900 | > loss_text_ce: 0.04403981193900108 (0.04831236174121921) | > loss_mel_ce: 3.7448108196258545 (nan) | > loss: 0.015035121701657772 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13638037416168564) | > loader_time: 0.0047 (0.004643789261346404)  --> TIME: 2025-02-24 05:03:28 -- STEP: 29198/36938 -- GLOBAL_STEP: 176950 | > loss_text_ce: 0.06088714301586151 (0.04831293669511845) | > loss_mel_ce: 3.4732112884521484 (nan) | > loss: 0.014024200849235058 (nan) | > current_lr: 5e-06 | > step_time: 0.1508 (0.13637249538674753) | > loader_time: 0.0047 (0.004643739016956406)  --> TIME: 2025-02-24 05:03:38 -- STEP: 29248/36938 -- GLOBAL_STEP: 177000 | > loss_text_ce: 0.05178813263773918 (0.04830989996863544) | > loss_mel_ce: 3.925281047821045 (nan) | > loss: 0.01578202098608017 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13639070942215564) | > loader_time: 0.0049 (0.004643901122868461)  --> TIME: 2025-02-24 05:03:47 -- STEP: 29298/36938 -- GLOBAL_STEP: 177050 | > loss_text_ce: 0.04988941550254822 (0.0483083691402771) | > loss_mel_ce: 4.0252251625061035 (nan) | > loss: 0.016171090304851532 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13638574923742658) | > loader_time: 0.0049 (0.0046439682292371935)  --> TIME: 2025-02-24 05:03:57 -- STEP: 29348/36938 -- GLOBAL_STEP: 177100 | > loss_text_ce: 0.03630676493048668 (0.04830398876166042) | > loss_mel_ce: 2.877467155456543 (nan) | > loss: 0.011562596075236797 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13638516788962488) | > loader_time: 0.0046 (0.004644005024353531)  --> TIME: 2025-02-24 05:04:06 -- STEP: 29398/36938 -- GLOBAL_STEP: 177150 | > loss_text_ce: 0.04855237901210785 (0.048305286653219495) | > loss_mel_ce: 3.108842372894287 (nan) | > loss: 0.0125293442979455 (nan) | > current_lr: 5e-06 | > step_time: 0.1835 (0.13637871598376872) | > loader_time: 0.0048 (0.00464424326903351)  --> TIME: 2025-02-24 05:04:15 -- STEP: 29448/36938 -- GLOBAL_STEP: 177200 | > loss_text_ce: 0.03884681686758995 (0.048304383155456675) | > loss_mel_ce: 2.078125 (nan) | > loss: 0.008400682359933853 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13638077341516217) | > loader_time: 0.0045 (0.004644318018488127)  --> TIME: 2025-02-24 05:04:24 -- STEP: 29498/36938 -- GLOBAL_STEP: 177250 | > loss_text_ce: 0.045037854462862015 (0.048304237251836996) | > loss_mel_ce: 2.977590560913086 (nan) | > loss: 0.011994557455182076 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.13636985825913814) | > loader_time: 0.0046 (0.004644253608113403)  --> TIME: 2025-02-24 05:04:33 -- STEP: 29548/36938 -- GLOBAL_STEP: 177300 | > loss_text_ce: 0.04297329857945442 (0.04830508681777067) | > loss_mel_ce: 3.977294445037842 (nan) | > loss: 0.015953445807099342 (nan) | > current_lr: 5e-06 | > step_time: 0.1878 (0.13637442887838203) | > loader_time: 0.005 (0.004644270798213374)  --> TIME: 2025-02-24 05:04:43 -- STEP: 29598/36938 -- GLOBAL_STEP: 177350 | > loss_text_ce: 0.03612169250845909 (0.04830010085561063) | > loss_mel_ce: 3.836332321166992 (nan) | > loss: 0.015366881154477596 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13637914641643636) | > loader_time: 0.0047 (0.004644257030387829)  --> TIME: 2025-02-24 05:04:52 -- STEP: 29648/36938 -- GLOBAL_STEP: 177400 | > loss_text_ce: 0.04257553815841675 (0.048298748750653014) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13637525989150023) | > loader_time: 0.0046 (0.0046442979278271184)  --> TIME: 2025-02-24 05:05:01 -- STEP: 29698/36938 -- GLOBAL_STEP: 177450 | > loss_text_ce: 0.050677698105573654 (0.048301082260647554) | > loss_mel_ce: 0.208984375 (nan) | > loss: 0.0010304050520062447 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.1363735868759196) | > loader_time: 0.0048 (0.00464448634041096)  --> TIME: 2025-02-24 05:05:11 -- STEP: 29748/36938 -- GLOBAL_STEP: 177500 | > loss_text_ce: 0.04455020651221275 (0.0483017963590758) | > loss_mel_ce: 3.192187547683716 (nan) | > loss: 0.012844198383390903 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13637576546258226) | > loader_time: 0.0049 (0.0046444720392892795)  --> TIME: 2025-02-24 05:05:20 -- STEP: 29798/36938 -- GLOBAL_STEP: 177550 | > loss_text_ce: 0.0421617366373539 (0.04829884896577941) | > loss_mel_ce: 3.5042264461517334 (nan) | > loss: 0.014072969555854797 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13637918609526148) | > loader_time: 0.0047 (0.004644458146213381)  --> TIME: 2025-02-24 05:05:30 -- STEP: 29848/36938 -- GLOBAL_STEP: 177600 | > loss_text_ce: 0.07012857496738434 (0.04829781650767382) | > loss_mel_ce: 0.8803939819335938 (nan) | > loss: 0.0037719151005148888 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13638167250607114) | > loader_time: 0.0046 (0.004644476769916888)  --> TIME: 2025-02-24 05:05:39 -- STEP: 29898/36938 -- GLOBAL_STEP: 177650 | > loss_text_ce: 0.05004293844103813 (0.048297192290054254) | > loss_mel_ce: 2.3404972553253174 (nan) | > loss: 0.009486271068453789 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13638072873016915) | > loader_time: 0.0046 (0.004644489079400835)  --> TIME: 2025-02-24 05:05:48 -- STEP: 29948/36938 -- GLOBAL_STEP: 177700 | > loss_text_ce: 0.04501322656869888 (0.048292522214016496) | > loss_mel_ce: 3.246366500854492 (nan) | > loss: 0.013061031699180603 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13637995354496046) | > loader_time: 0.0046 (0.004644432683422209)  --> TIME: 2025-02-24 05:05:57 -- STEP: 29998/36938 -- GLOBAL_STEP: 177750 | > loss_text_ce: 0.0692937821149826 (0.0482905937914898) | > loss_mel_ce: 2.639335870742798 (nan) | > loss: 0.010748530738055706 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13637370685393446) | > loader_time: 0.0045 (0.004644399643325839)  --> TIME: 2025-02-24 05:06:07 -- STEP: 30048/36938 -- GLOBAL_STEP: 177800 | > loss_text_ce: 0.05017640069127083 (0.048292398406236066) | > loss_mel_ce: 3.7530345916748047 (nan) | > loss: 0.015092107467353344 (nan) | > current_lr: 5e-06 | > step_time: 0.1767 (0.13637501321915674) | > loader_time: 0.0049 (0.004644360849524721)  --> TIME: 2025-02-24 05:06:16 -- STEP: 30098/36938 -- GLOBAL_STEP: 177850 | > loss_text_ce: 0.039007123559713364 (0.04828868777755345) | > loss_mel_ce: 0.0590413399040699 (nan) | > loss: 0.0003890812222380191 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1363811305974679) | > loader_time: 0.0046 (0.004644453291861645)  --> TIME: 2025-02-24 05:06:25 -- STEP: 30148/36938 -- GLOBAL_STEP: 177900 | > loss_text_ce: 0.06286683678627014 (0.0482849636160174) | > loss_mel_ce: 3.6163220405578613 (nan) | > loss: 0.014599957503378391 (nan) | > current_lr: 5e-06 | > step_time: 0.1718 (0.1363782951966909) | > loader_time: 0.0049 (0.004644417020960459)  --> TIME: 2025-02-24 05:06:34 -- STEP: 30198/36938 -- GLOBAL_STEP: 177950 | > loss_text_ce: 0.03823801130056381 (0.04827875469037966) | > loss_mel_ce: 3.0195178985595703 (nan) | > loss: 0.012133953161537647 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13637039371496723) | > loader_time: 0.0045 (0.0046442614950812635)  --> TIME: 2025-02-24 05:06:44 -- STEP: 30248/36938 -- GLOBAL_STEP: 178000 | > loss_text_ce: 0.0366724357008934 (0.04827568637358267) | > loss_mel_ce: 3.351179361343384 (nan) | > loss: 0.013443856500089169 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.1363751574330044) | > loader_time: 0.0045 (0.004644233677255457)  --> TIME: 2025-02-24 05:06:53 -- STEP: 30298/36938 -- GLOBAL_STEP: 178050 | > loss_text_ce: 0.05359991267323494 (0.04827194080963296) | > loss_mel_ce: 3.598459243774414 (nan) | > loss: 0.014492298476397991 (nan) | > current_lr: 5e-06 | > step_time: 0.1702 (0.13637561703351686) | > loader_time: 0.0069 (0.004644330574487403)  --> TIME: 2025-02-24 05:07:02 -- STEP: 30348/36938 -- GLOBAL_STEP: 178100 | > loss_text_ce: 0.03212421387434006 (0.04826756003056216) | > loss_mel_ce: 2.404264211654663 (nan) | > loss: 0.009668208658695221 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.1363674393359502) | > loader_time: 0.0045 (0.004644260515537574)  --> TIME: 2025-02-24 05:07:11 -- STEP: 30398/36938 -- GLOBAL_STEP: 178150 | > loss_text_ce: 0.0476699136197567 (0.048263964796456844) | > loss_mel_ce: 3.9265449047088623 (nan) | > loss: 0.015770694240927696 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13636914789147428) | > loader_time: 0.0046 (0.004644263668337647)  --> TIME: 2025-02-24 05:07:21 -- STEP: 30448/36938 -- GLOBAL_STEP: 178200 | > loss_text_ce: 0.049879543483257294 (0.04826228934703191) | > loss_mel_ce: 3.9067890644073486 (nan) | > loss: 0.015701066702604294 (nan) | > current_lr: 5e-06 | > step_time: 0.1889 (0.13636561204775902) | > loader_time: 0.0048 (0.0046441892276484365)  --> TIME: 2025-02-24 05:07:30 -- STEP: 30498/36938 -- GLOBAL_STEP: 178250 | > loss_text_ce: 0.05183000862598419 (0.04826205595310895) | > loss_mel_ce: 3.6312334537506104 (nan) | > loss: 0.014615331776440144 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.13636125687404174) | > loader_time: 0.0047 (0.004644137608027012)  --> TIME: 2025-02-24 05:07:39 -- STEP: 30548/36938 -- GLOBAL_STEP: 178300 | > loss_text_ce: 0.0475994236767292 (0.0482638443269763) | > loss_mel_ce: 3.9372670650482178 (nan) | > loss: 0.015812963247299194 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13635888246555414) | > loader_time: 0.0046 (0.004644080108726503)  --> TIME: 2025-02-24 05:07:48 -- STEP: 30598/36938 -- GLOBAL_STEP: 178350 | > loss_text_ce: 0.05101815611124039 (0.04826260638651607) | > loss_mel_ce: 3.873570203781128 (nan) | > loss: 0.015573764219880104 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.13636540069994166) | > loader_time: 0.005 (0.004644125596753405)  --> TIME: 2025-02-24 05:07:58 -- STEP: 30648/36938 -- GLOBAL_STEP: 178400 | > loss_text_ce: 0.03751214221119881 (0.048261448931155866) | > loss_mel_ce: 3.8222668170928955 (nan) | > loss: 0.015316584147512913 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13636748184270497) | > loader_time: 0.0046 (0.004644179205706532)  --> TIME: 2025-02-24 05:08:07 -- STEP: 30698/36938 -- GLOBAL_STEP: 178450 | > loss_text_ce: 0.040335558354854584 (0.04825666795858915) | > loss_mel_ce: 2.9632480144500732 (nan) | > loss: 0.011918983422219753 (nan) | > current_lr: 5e-06 | > step_time: 0.1531 (0.13636686249861826) | > loader_time: 0.0046 (0.004644121787580537)  --> TIME: 2025-02-24 05:08:16 -- STEP: 30748/36938 -- GLOBAL_STEP: 178500 | > loss_text_ce: 0.06799663603305817 (0.04825423778210666) | > loss_mel_ce: 3.0130717754364014 (nan) | > loss: 0.012226463295519352 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13635781803195793) | > loader_time: 0.0045 (0.00464413486904762)  --> TIME: 2025-02-24 05:08:25 -- STEP: 30798/36938 -- GLOBAL_STEP: 178550 | > loss_text_ce: 0.049173761159181595 (0.04825087519896241) | > loss_mel_ce: 2.571192502975464 (nan) | > loss: 0.010398279875516891 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.13634994551921828) | > loader_time: 0.0045 (0.004644057512113023)  --> TIME: 2025-02-24 05:08:34 -- STEP: 30848/36938 -- GLOBAL_STEP: 178600 | > loss_text_ce: 0.04465996474027634 (0.048249131072844204) | > loss_mel_ce: 3.8403332233428955 (nan) | > loss: 0.015416639856994152 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.13634686914748106) | > loader_time: 0.0049 (0.004644080455496111)  --> TIME: 2025-02-24 05:08:44 -- STEP: 30898/36938 -- GLOBAL_STEP: 178650 | > loss_text_ce: 0.04743983969092369 (0.048247252996995046) | > loss_mel_ce: 3.509453535079956 (nan) | > loss: 0.014114657416939735 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13634694612591997) | > loader_time: 0.0045 (0.004644112098069637)  --> TIME: 2025-02-24 05:08:53 -- STEP: 30948/36938 -- GLOBAL_STEP: 178700 | > loss_text_ce: 0.03794223070144653 (0.048247490467646204) | > loss_mel_ce: 3.543808937072754 (nan) | > loss: 0.014213298447430134 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13634491908206847) | > loader_time: 0.0046 (0.004644174076288396)  --> TIME: 2025-02-24 05:09:02 -- STEP: 30998/36938 -- GLOBAL_STEP: 178750 | > loss_text_ce: 0.0450437106192112 (0.04824668443427972) | > loss_mel_ce: 4.515625 (nan) | > loss: 0.01809789426624775 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13634216786200226) | > loader_time: 0.0046 (0.004644164755095351)  --> TIME: 2025-02-24 05:09:11 -- STEP: 31048/36938 -- GLOBAL_STEP: 178800 | > loss_text_ce: 0.04103296250104904 (0.048242649861037434) | > loss_mel_ce: 3.312448263168335 (nan) | > loss: 0.013307466171681881 (nan) | > current_lr: 5e-06 | > step_time: 0.1709 (0.13634082550580023) | > loader_time: 0.0049 (0.004644182954858004)  --> TIME: 2025-02-24 05:09:21 -- STEP: 31098/36938 -- GLOBAL_STEP: 178850 | > loss_text_ce: 0.05329687520861626 (0.04823708155767018) | > loss_mel_ce: 3.9833168983459473 (nan) | > loss: 0.01601831056177616 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.1363383973094858) | > loader_time: 0.0047 (0.004644178832042677)  --> TIME: 2025-02-24 05:09:30 -- STEP: 31148/36938 -- GLOBAL_STEP: 178900 | > loss_text_ce: 0.040763989090919495 (0.04823642224094341) | > loss_mel_ce: 3.349838972091675 (nan) | > loss: 0.013454774394631386 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13634023947910048) | > loader_time: 0.0045 (0.004644178151625096)  --> TIME: 2025-02-24 05:09:39 -- STEP: 31198/36938 -- GLOBAL_STEP: 178950 | > loss_text_ce: 0.04035045951604843 (0.048233938212404695) | > loss_mel_ce: 3.419023275375366 (nan) | > loss: 0.013727674260735512 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13633698026708757) | > loader_time: 0.0045 (0.004644063338460637)  --> TIME: 2025-02-24 05:09:48 -- STEP: 31248/36938 -- GLOBAL_STEP: 179000 | > loss_text_ce: 0.03945940360426903 (0.048231517526096286) | > loss_mel_ce: 3.562594175338745 (nan) | > loss: 0.014293864369392395 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13632965119173318) | > loader_time: 0.0046 (0.004643956461565599)  --> TIME: 2025-02-24 05:09:58 -- STEP: 31298/36938 -- GLOBAL_STEP: 179050 | > loss_text_ce: 0.04822349548339844 (0.0482299279599096) | > loss_mel_ce: 3.337576389312744 (nan) | > loss: 0.01343571487814188 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1363352533346322) | > loader_time: 0.0044 (0.00464389631790707)  --> TIME: 2025-02-24 05:10:07 -- STEP: 31348/36938 -- GLOBAL_STEP: 179100 | > loss_text_ce: 0.04424281045794487 (0.048228408237995284) | > loss_mel_ce: 2.76039719581604 (nan) | > loss: 0.011129524558782578 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.1363410110207152) | > loader_time: 0.0045 (0.00464389121728664)  --> TIME: 2025-02-24 05:10:17 -- STEP: 31398/36938 -- GLOBAL_STEP: 179150 | > loss_text_ce: 0.052040599286556244 (0.04822730145518777) | > loss_mel_ce: 3.8455753326416016 (nan) | > loss: 0.015466731041669846 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.13634622447072772) | > loader_time: 0.0047 (0.004643926074363774)  --> TIME: 2025-02-24 05:10:26 -- STEP: 31448/36938 -- GLOBAL_STEP: 179200 | > loss_text_ce: 0.057637427002191544 (0.04822653299786888) | > loss_mel_ce: 3.03173828125 (nan) | > loss: 0.012259428389370441 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13633939046133317) | > loader_time: 0.0045 (0.00464384070912927)  --> TIME: 2025-02-24 05:10:35 -- STEP: 31498/36938 -- GLOBAL_STEP: 179250 | > loss_text_ce: 0.04866533726453781 (0.04822788735131615) | > loss_mel_ce: 2.1960580348968506 (nan) | > loss: 0.008907632902264595 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13634408436182613) | > loader_time: 0.0047 (0.004644025090429721)  --> TIME: 2025-02-24 05:10:44 -- STEP: 31548/36938 -- GLOBAL_STEP: 179300 | > loss_text_ce: 0.047601375728845596 (0.04822707665577949) | > loss_mel_ce: 3.700546979904175 (nan) | > loss: 0.014873606152832508 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1363377981846743) | > loader_time: 0.0045 (0.004644024798334129)  --> TIME: 2025-02-24 05:10:53 -- STEP: 31598/36938 -- GLOBAL_STEP: 179350 | > loss_text_ce: 0.03773237019777298 (0.04822368446168208) | > loss_mel_ce: 2.52508282661438 (nan) | > loss: 0.010169901885092258 (nan) | > current_lr: 5e-06 | > step_time: 0.1071 (0.1363328266041199) | > loader_time: 0.0045 (0.004644014720818968)  --> TIME: 2025-02-24 05:11:02 -- STEP: 31648/36938 -- GLOBAL_STEP: 179400 | > loss_text_ce: 0.033347781747579575 (0.04821939992448726) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1082 (0.1363269008537894) | > loader_time: 0.0045 (0.004644016329391187)  --> TIME: 2025-02-24 05:11:12 -- STEP: 31698/36938 -- GLOBAL_STEP: 179450 | > loss_text_ce: 0.056470438838005066 (0.04821666390950334) | > loss_mel_ce: 3.56955885887146 (nan) | > loss: 0.014389005489647388 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13633043629580704) | > loader_time: 0.0047 (0.004644076292714431)  --> TIME: 2025-02-24 05:11:21 -- STEP: 31748/36938 -- GLOBAL_STEP: 179500 | > loss_text_ce: 0.052783869206905365 (0.04821067308039352) | > loss_mel_ce: 4.282515525817871 (nan) | > loss: 0.017203571274876595 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13633374076984978) | > loader_time: 0.0046 (0.0046441141538030115)  --> TIME: 2025-02-24 05:11:30 -- STEP: 31798/36938 -- GLOBAL_STEP: 179550 | > loss_text_ce: 0.05081323906779289 (0.04821212848749652) | > loss_mel_ce: 2.7328124046325684 (nan) | > loss: 0.011046133935451508 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13633195999561243) | > loader_time: 0.0045 (0.004644106346013772)  --> TIME: 2025-02-24 05:11:39 -- STEP: 31848/36938 -- GLOBAL_STEP: 179600 | > loss_text_ce: 0.05185170844197273 (0.04821412055726854) | > loss_mel_ce: 1.3982056379318237 (nan) | > loss: 0.005754196085035801 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13632300332903705) | > loader_time: 0.0047 (0.004644176201495783)  --> TIME: 2025-02-24 05:11:48 -- STEP: 31898/36938 -- GLOBAL_STEP: 179650 | > loss_text_ce: 0.04688067361712456 (0.04821426978477734) | > loss_mel_ce: 3.1857476234436035 (nan) | > loss: 0.012827890925109386 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13631577552751573) | > loader_time: 0.0045 (0.004644147272999149)  --> TIME: 2025-02-24 05:11:57 -- STEP: 31948/36938 -- GLOBAL_STEP: 179700 | > loss_text_ce: 0.05218225717544556 (0.04821245188803145) | > loss_mel_ce: 3.4417624473571777 (nan) | > loss: 0.013864860869944096 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.1363050458994546) | > loader_time: 0.0049 (0.004644141644071523)  --> TIME: 2025-02-24 05:12:06 -- STEP: 31998/36938 -- GLOBAL_STEP: 179750 | > loss_text_ce: 0.04686666280031204 (0.048211121314079335) | > loss_mel_ce: 3.2737019062042236 (nan) | > loss: 0.01317686028778553 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13629853555549665) | > loader_time: 0.0047 (0.004644076782015318)  --> TIME: 2025-02-24 05:12:16 -- STEP: 32048/36938 -- GLOBAL_STEP: 179800 | > loss_text_ce: 0.04119983687996864 (0.04821035418387098) | > loss_mel_ce: 3.046131134033203 (nan) | > loss: 0.012251314707100391 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13629397072169694) | > loader_time: 0.0046 (0.004644100815132157)  --> TIME: 2025-02-24 05:12:25 -- STEP: 32098/36938 -- GLOBAL_STEP: 179850 | > loss_text_ce: 0.04570280760526657 (0.04820808577658542) | > loss_mel_ce: 3.2694036960601807 (nan) | > loss: 0.01315518468618393 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13629320139632276) | > loader_time: 0.0045 (0.004644059319311655)  --> TIME: 2025-02-24 05:12:34 -- STEP: 32148/36938 -- GLOBAL_STEP: 179900 | > loss_text_ce: 0.04688522219657898 (0.048206846043426975) | > loss_mel_ce: 3.396498918533325 (nan) | > loss: 0.01366422325372696 (nan) | > current_lr: 5e-06 | > step_time: 0.1599 (0.1362891193422304) | > loader_time: 0.0047 (0.004643998670239829)  --> TIME: 2025-02-24 05:12:44 -- STEP: 32198/36938 -- GLOBAL_STEP: 179950 | > loss_text_ce: 0.05508612096309662 (0.04820389283446816) | > loss_mel_ce: 4.359567165374756 (nan) | > loss: 0.01751846633851528 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13629497027603177) | > loader_time: 0.0046 (0.0046439460215564774)  --> TIME: 2025-02-24 05:12:53 -- STEP: 32248/36938 -- GLOBAL_STEP: 180000 | > loss_text_ce: 0.03469211980700493 (0.04820294656467454) | > loss_mel_ce: 2.7529163360595703 (nan) | > loss: 0.011061938479542732 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13629093284170604) | > loader_time: 0.0045 (0.00464388339994446) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_180000.pth  --> TIME: 2025-02-24 05:13:09 -- STEP: 32298/36938 -- GLOBAL_STEP: 180050 | > loss_text_ce: 0.039287157356739044 (0.04820141771033636) | > loss_mel_ce: 3.8855271339416504 (nan) | > loss: 0.015574660152196884 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13630121955562305) | > loader_time: 0.0046 (0.004644066912890655)  --> TIME: 2025-02-24 05:13:18 -- STEP: 32348/36938 -- GLOBAL_STEP: 180100 | > loss_text_ce: 0.05098183453083038 (0.04820008163871983) | > loss_mel_ce: 3.0262610912323 (nan) | > loss: 0.012211281806230545 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.1362958766185607) | > loader_time: 0.0045 (0.004644076587155301)  --> TIME: 2025-02-24 05:13:27 -- STEP: 32398/36938 -- GLOBAL_STEP: 180150 | > loss_text_ce: 0.04174108803272247 (0.048199763745629495) | > loss_mel_ce: 2.9525086879730225 (nan) | > loss: 0.011881944723427296 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13628984357157115) | > loader_time: 0.0047 (0.004644032716530335)  --> TIME: 2025-02-24 05:13:36 -- STEP: 32448/36938 -- GLOBAL_STEP: 180200 | > loss_text_ce: 0.05451451614499092 (0.04820111934958405) | > loss_mel_ce: 3.499539375305176 (nan) | > loss: 0.014103388413786888 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13628819749435428) | > loader_time: 0.0047 (0.004644012537874892)  --> TIME: 2025-02-24 05:13:45 -- STEP: 32498/36938 -- GLOBAL_STEP: 180250 | > loss_text_ce: 0.044769953936338425 (0.04819889272580298) | > loss_mel_ce: 3.6164050102233887 (nan) | > loss: 0.014528472907841206 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.136284495582655) | > loader_time: 0.0045 (0.004644021091992932)  --> TIME: 2025-02-24 05:13:54 -- STEP: 32548/36938 -- GLOBAL_STEP: 180300 | > loss_text_ce: 0.04846515506505966 (0.04819609052612642) | > loss_mel_ce: 2.801339626312256 (nan) | > loss: 0.01130875013768673 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13627578550142952) | > loader_time: 0.0046 (0.004643940897762629)  --> TIME: 2025-02-24 05:14:03 -- STEP: 32598/36938 -- GLOBAL_STEP: 180350 | > loss_text_ce: 0.04225240275263786 (0.04819360571511215) | > loss_mel_ce: 3.1915481090545654 (nan) | > loss: 0.012832541950047016 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.1362690945565114) | > loader_time: 0.0047 (0.004643842540450525)  --> TIME: 2025-02-24 05:14:12 -- STEP: 32648/36938 -- GLOBAL_STEP: 180400 | > loss_text_ce: 0.05934198200702667 (0.04819320961190022) | > loss_mel_ce: 4.310763835906982 (nan) | > loss: 0.01734169013798237 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13627455208091976) | > loader_time: 0.0046 (0.004643855375917607)  --> TIME: 2025-02-24 05:14:22 -- STEP: 32698/36938 -- GLOBAL_STEP: 180450 | > loss_text_ce: 0.04789546877145767 (0.04819015255230924) | > loss_mel_ce: 0.017333984375 (nan) | > loss: 0.00025884705246426165 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13627935319189377) | > loader_time: 0.0046 (0.004643863505548327)  --> TIME: 2025-02-24 05:14:31 -- STEP: 32748/36938 -- GLOBAL_STEP: 180500 | > loss_text_ce: 0.050362419337034225 (0.04818809394756748) | > loss_mel_ce: 3.6784708499908447 (nan) | > loss: 0.014796958304941654 (nan) | > current_lr: 5e-06 | > step_time: 0.1512 (0.13627702125164146) | > loader_time: 0.0046 (0.0046438101346463435)  --> TIME: 2025-02-24 05:14:40 -- STEP: 32798/36938 -- GLOBAL_STEP: 180550 | > loss_text_ce: 0.04142200946807861 (0.0481865659104929) | > loss_mel_ce: 3.5739223957061768 (nan) | > loss: 0.014346606098115444 (nan) | > current_lr: 5e-06 | > step_time: 0.1499 (0.13627972252609416) | > loader_time: 0.0046 (0.004643823949438289)  --> TIME: 2025-02-24 05:14:50 -- STEP: 32848/36938 -- GLOBAL_STEP: 180600 | > loss_text_ce: 0.042843759059906006 (0.04818761300822837) | > loss_mel_ce: 3.6407134532928467 (nan) | > loss: 0.014617291279137135 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.1362744387306373) | > loader_time: 0.005 (0.0046437384802947975)  --> TIME: 2025-02-24 05:14:59 -- STEP: 32898/36938 -- GLOBAL_STEP: 180650 | > loss_text_ce: 0.04082576557993889 (0.04818564534522615) | > loss_mel_ce: 4.106815338134766 (nan) | > loss: 0.016458895057439804 (nan) | > current_lr: 5e-06 | > step_time: 0.1876 (0.1362717100911421) | > loader_time: 0.0048 (0.004643644146719525)  --> TIME: 2025-02-24 05:15:08 -- STEP: 32948/36938 -- GLOBAL_STEP: 180700 | > loss_text_ce: 0.04983888566493988 (0.04818673864838823) | > loss_mel_ce: 4.054898738861084 (nan) | > loss: 0.016288643702864647 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13626675849923747) | > loader_time: 0.0045 (0.004643572951399392)  --> TIME: 2025-02-24 05:15:17 -- STEP: 32998/36938 -- GLOBAL_STEP: 180750 | > loss_text_ce: 0.04306158423423767 (0.04818539259135901) | > loss_mel_ce: 1.7752280235290527 (nan) | > loss: 0.007215435616672039 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13627838827608824) | > loader_time: 0.0045 (0.004643540496833116)  --> TIME: 2025-02-24 05:15:27 -- STEP: 33048/36938 -- GLOBAL_STEP: 180800 | > loss_text_ce: 0.04874696955084801 (0.048184953581753044) | > loss_mel_ce: 3.9926793575286865 (nan) | > loss: 0.016037406399846077 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13627893657495188) | > loader_time: 0.0046 (0.0046434807260860135)  --> TIME: 2025-02-24 05:15:36 -- STEP: 33098/36938 -- GLOBAL_STEP: 180850 | > loss_text_ce: 0.04341466352343559 (0.048182848233754835) | > loss_mel_ce: 3.4553322792053223 (nan) | > loss: 0.013883916661143303 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13626828090799972) | > loader_time: 0.0045 (0.004643392545578026)  --> TIME: 2025-02-24 05:15:45 -- STEP: 33148/36938 -- GLOBAL_STEP: 180900 | > loss_text_ce: 0.04096757620573044 (0.04817845243724054) | > loss_mel_ce: 3.6625163555145264 (nan) | > loss: 0.014696365222334862 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13626946537115345) | > loader_time: 0.0046 (0.004643334149307384)  --> TIME: 2025-02-24 05:15:54 -- STEP: 33198/36938 -- GLOBAL_STEP: 180950 | > loss_text_ce: 0.04337332770228386 (0.04817817583225125) | > loss_mel_ce: 3.8823912143707275 (nan) | > loss: 0.015578432008624077 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.1362660298886693) | > loader_time: 0.0045 (0.004643313647307613)  --> TIME: 2025-02-24 05:16:04 -- STEP: 33248/36938 -- GLOBAL_STEP: 181000 | > loss_text_ce: 0.04996555671095848 (0.048174583963554375) | > loss_mel_ce: 0.5862019658088684 (nan) | > loss: 0.0025244744028896093 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13626955303294644) | > loader_time: 0.0046 (0.004643268962108375)  --> TIME: 2025-02-24 05:16:13 -- STEP: 33298/36938 -- GLOBAL_STEP: 181050 | > loss_text_ce: 0.03865998983383179 (0.04817547257560598) | > loss_mel_ce: 3.4733638763427734 (nan) | > loss: 0.01393660344183445 (nan) | > current_lr: 5e-06 | > step_time: 0.1556 (0.13627422617050317) | > loader_time: 0.0048 (0.0046432231509210865)  --> TIME: 2025-02-24 05:16:23 -- STEP: 33348/36938 -- GLOBAL_STEP: 181100 | > loss_text_ce: 0.06861498951911926 (0.04817329224780954) | > loss_mel_ce: 1.5920614004135132 (nan) | > loss: 0.00658998591825366 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13627829273012879) | > loader_time: 0.0045 (0.004643165566187139)  --> TIME: 2025-02-24 05:16:32 -- STEP: 33398/36938 -- GLOBAL_STEP: 181150 | > loss_text_ce: 0.04854285717010498 (0.04816997087900187) | > loss_mel_ce: 3.706974744796753 (nan) | > loss: 0.014902847819030285 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13627833530241848) | > loader_time: 0.0046 (0.0046431055910766275)  --> TIME: 2025-02-24 05:16:41 -- STEP: 33448/36938 -- GLOBAL_STEP: 181200 | > loss_text_ce: 0.04785547032952309 (0.048167661700104385) | > loss_mel_ce: 4.085267066955566 (nan) | > loss: 0.016401279717683792 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.13627599574994537) | > loader_time: 0.0047 (0.004643019236209038)  --> TIME: 2025-02-24 05:16:50 -- STEP: 33498/36938 -- GLOBAL_STEP: 181250 | > loss_text_ce: 0.047647394239902496 (0.04816700585610303) | > loss_mel_ce: 4.118401050567627 (nan) | > loss: 0.016531938686966896 (nan) | > current_lr: 5e-06 | > step_time: 0.187 (0.13627186909925784) | > loader_time: 0.0047 (0.004642925993265546)  --> TIME: 2025-02-24 05:16:59 -- STEP: 33548/36938 -- GLOBAL_STEP: 181300 | > loss_text_ce: 0.04526042938232422 (0.048160499424219586) | > loss_mel_ce: 3.21087646484375 (nan) | > loss: 0.012921178713440895 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13626175941181132) | > loader_time: 0.0047 (0.004642789961115692)  --> TIME: 2025-02-24 05:17:09 -- STEP: 33598/36938 -- GLOBAL_STEP: 181350 | > loss_text_ce: 0.0603339709341526 (0.048161744995154576) | > loss_mel_ce: 3.5644373893737793 (nan) | > loss: 0.014384014531970024 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.1362594989405013) | > loader_time: 0.0046 (0.0046427040499188756)  --> TIME: 2025-02-24 05:17:18 -- STEP: 33648/36938 -- GLOBAL_STEP: 181400 | > loss_text_ce: 0.03932631388306618 (0.04816146673258999) | > loss_mel_ce: 2.868980646133423 (nan) | > loss: 0.011540901847183704 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1362585350693961) | > loader_time: 0.0045 (0.004642646134438204)  --> TIME: 2025-02-24 05:17:27 -- STEP: 33698/36938 -- GLOBAL_STEP: 181450 | > loss_text_ce: 0.047319915145635605 (0.048157708881631504) | > loss_mel_ce: 4.043426513671875 (nan) | > loss: 0.01623312197625637 (nan) | > current_lr: 5e-06 | > step_time: 0.1884 (0.1362543571412074) | > loader_time: 0.0048 (0.004642576582389772)  --> TIME: 2025-02-24 05:17:36 -- STEP: 33748/36938 -- GLOBAL_STEP: 181500 | > loss_text_ce: 0.03291948139667511 (0.04815848163418259) | > loss_mel_ce: 0.04805755615234375 (nan) | > loss: 0.00032133745844475925 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.1362519485698701) | > loader_time: 0.0049 (0.004642599946135412)  --> TIME: 2025-02-24 05:17:45 -- STEP: 33798/36938 -- GLOBAL_STEP: 181550 | > loss_text_ce: 0.05093240737915039 (0.0481537412927827) | > loss_mel_ce: 2.513920307159424 (nan) | > loss: 0.01017798762768507 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.13624132315825854) | > loader_time: 0.0045 (0.004642512940945691)  --> TIME: 2025-02-24 05:17:54 -- STEP: 33848/36938 -- GLOBAL_STEP: 181600 | > loss_text_ce: 0.045915812253952026 (0.04815619653873911) | > loss_mel_ce: 3.689680337905884 (nan) | > loss: 0.014823795296251774 (nan) | > current_lr: 5e-06 | > step_time: 0.1508 (0.13624257812881088) | > loader_time: 0.0047 (0.0046424902632173975)  --> TIME: 2025-02-24 05:18:04 -- STEP: 33898/36938 -- GLOBAL_STEP: 181650 | > loss_text_ce: 0.05754517391324043 (0.04815339478279188) | > loss_mel_ce: 0.8554088473320007 (nan) | > loss: 0.003622833639383316 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13624163239155293) | > loader_time: 0.0045 (0.004642464585821645)  --> TIME: 2025-02-24 05:18:13 -- STEP: 33948/36938 -- GLOBAL_STEP: 181700 | > loss_text_ce: 0.036355894058942795 (0.048150510235105935) | > loss_mel_ce: 2.3407673835754395 (nan) | > loss: 0.009433030150830746 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13624279200855313) | > loader_time: 0.0045 (0.004642500807992622)  --> TIME: 2025-02-24 05:18:23 -- STEP: 33998/36938 -- GLOBAL_STEP: 181750 | > loss_text_ce: 0.05038030073046684 (0.04815049194454423) | > loss_mel_ce: 3.181889533996582 (nan) | > loss: 0.012826467864215374 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.1362524659951048) | > loader_time: 0.0046 (0.004642508746497857)  --> TIME: 2025-02-24 05:18:32 -- STEP: 34048/36938 -- GLOBAL_STEP: 181800 | > loss_text_ce: 0.049881529062986374 (0.048151532226489234) | > loss_mel_ce: 3.901155710220337 (nan) | > loss: 0.01567872054874897 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.1362565920401128) | > loader_time: 0.0047 (0.004642545567699401)  --> TIME: 2025-02-24 05:18:42 -- STEP: 34098/36938 -- GLOBAL_STEP: 181850 | > loss_text_ce: 0.047809310257434845 (0.0481504844447262) | > loss_mel_ce: 3.8684539794921875 (nan) | > loss: 0.015540728345513344 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13626016911275182) | > loader_time: 0.0046 (0.004642567072972455)  --> TIME: 2025-02-24 05:18:51 -- STEP: 34148/36938 -- GLOBAL_STEP: 181900 | > loss_text_ce: 0.048842981457710266 (0.04815109307570958) | > loss_mel_ce: 3.5188236236572266 (nan) | > loss: 0.01415740791708231 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.1362593588776553) | > loader_time: 0.0045 (0.004642578391485902)  --> TIME: 2025-02-24 05:19:00 -- STEP: 34198/36938 -- GLOBAL_STEP: 181950 | > loss_text_ce: 0.04913762956857681 (0.04814768943529211) | > loss_mel_ce: 3.845072031021118 (nan) | > loss: 0.015453213825821877 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13625906708575455) | > loader_time: 0.0048 (0.004642597736200476)  --> TIME: 2025-02-24 05:19:10 -- STEP: 34248/36938 -- GLOBAL_STEP: 182000 | > loss_text_ce: 0.047954414039850235 (0.04814794912533043) | > loss_mel_ce: 3.779788017272949 (nan) | > loss: 0.015189454890787601 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1362701401243268) | > loader_time: 0.0046 (0.0046426076263614054)  --> TIME: 2025-02-24 05:19:19 -- STEP: 34298/36938 -- GLOBAL_STEP: 182050 | > loss_text_ce: 0.0356544628739357 (0.048145757175190686) | > loss_mel_ce: 2.519014596939087 (nan) | > loss: 0.01013757660984993 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13627244003368805) | > loader_time: 0.0047 (0.004642584350437353)  --> TIME: 2025-02-24 05:19:29 -- STEP: 34348/36938 -- GLOBAL_STEP: 182100 | > loss_text_ce: 0.05728267878293991 (0.048142628035991564) | > loss_mel_ce: 3.517534017562866 (nan) | > loss: 0.014185781590640545 (nan) | > current_lr: 5e-06 | > step_time: 0.1913 (0.13627506969721276) | > loader_time: 0.0048 (0.004642583666684071)  --> TIME: 2025-02-24 05:19:38 -- STEP: 34398/36938 -- GLOBAL_STEP: 182150 | > loss_text_ce: 0.04414840415120125 (0.048140461483538365) | > loss_mel_ce: 3.277191400527954 (nan) | > loss: 0.013179920613765717 (nan) | > current_lr: 5e-06 | > step_time: 0.1862 (0.13628732773431484) | > loader_time: 0.0048 (0.00464257599829353)  --> TIME: 2025-02-24 05:19:48 -- STEP: 34448/36938 -- GLOBAL_STEP: 182200 | > loss_text_ce: 0.053138237446546555 (0.048137065447778106) | > loss_mel_ce: 3.3940043449401855 (nan) | > loss: 0.013679138384759426 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13629226363015454) | > loader_time: 0.0045 (0.004642538750553715)  --> TIME: 2025-02-24 05:19:57 -- STEP: 34498/36938 -- GLOBAL_STEP: 182250 | > loss_text_ce: 0.047431424260139465 (0.048134558883134115) | > loss_mel_ce: 3.94735050201416 (nan) | > loss: 0.01585230976343155 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13628786601525036) | > loader_time: 0.0047 (0.004642477691522568)  --> TIME: 2025-02-24 05:20:06 -- STEP: 34548/36938 -- GLOBAL_STEP: 182300 | > loss_text_ce: 0.04797695577144623 (0.04813368421955749) | > loss_mel_ce: 3.1145496368408203 (nan) | > loss: 0.012549709528684616 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.136286505756245) | > loader_time: 0.0047 (0.004642483114823575)  --> TIME: 2025-02-24 05:20:16 -- STEP: 34598/36938 -- GLOBAL_STEP: 182350 | > loss_text_ce: 0.051739566028118134 (0.04813082369097836) | > loss_mel_ce: 3.0923945903778076 (nan) | > loss: 0.012476722709834576 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13628984920633727) | > loader_time: 0.0047 (0.0046424891357580685)  --> TIME: 2025-02-24 05:20:25 -- STEP: 34648/36938 -- GLOBAL_STEP: 182400 | > loss_text_ce: 0.0361672081053257 (0.04812880159105513) | > loss_mel_ce: 0.02131144143640995 (nan) | > loss: 0.00022808990615885705 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.13629373455262447) | > loader_time: 0.0048 (0.004642528389100806)  --> TIME: 2025-02-24 05:20:35 -- STEP: 34698/36938 -- GLOBAL_STEP: 182450 | > loss_text_ce: 0.04306403174996376 (0.04813050232295472) | > loss_mel_ce: 3.148442268371582 (nan) | > loss: 0.01266470830887556 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.1362957173214516) | > loader_time: 0.0047 (0.004642527092017805)  --> TIME: 2025-02-24 05:20:44 -- STEP: 34748/36938 -- GLOBAL_STEP: 182500 | > loss_text_ce: 0.04567300155758858 (0.04812890113475258) | > loss_mel_ce: 3.5182089805603027 (nan) | > loss: 0.01414238940924406 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.13630019355262268) | > loader_time: 0.0047 (0.004642588298806835)  --> TIME: 2025-02-24 05:20:54 -- STEP: 34798/36938 -- GLOBAL_STEP: 182550 | > loss_text_ce: 0.028417138382792473 (0.04812657149413109) | > loss_mel_ce: 2.9092540740966797 (nan) | > loss: 0.01165742613375187 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13630655128535718) | > loader_time: 0.0046 (0.004642774006492611)  --> TIME: 2025-02-24 05:21:03 -- STEP: 34848/36938 -- GLOBAL_STEP: 182600 | > loss_text_ce: 0.04851682484149933 (0.04812640292065748) | > loss_mel_ce: 3.534325122833252 (nan) | > loss: 0.014217627234756947 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1363124536800224) | > loader_time: 0.0046 (0.0046428707937125925)  --> TIME: 2025-02-24 05:21:12 -- STEP: 34898/36938 -- GLOBAL_STEP: 182650 | > loss_text_ce: 0.048913102596998215 (0.048124499602754806) | > loss_mel_ce: 3.296236038208008 (nan) | > loss: 0.013274401426315308 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13630766099049685) | > loader_time: 0.0045 (0.0046428799239794935)  --> TIME: 2025-02-24 05:21:21 -- STEP: 34948/36938 -- GLOBAL_STEP: 182700 | > loss_text_ce: 0.04930510371923447 (0.04812324605943467) | > loss_mel_ce: 3.6007044315338135 (nan) | > loss: 0.014484166167676449 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13630179975758003) | > loader_time: 0.0054 (0.0046428910611054355)  --> TIME: 2025-02-24 05:21:30 -- STEP: 34998/36938 -- GLOBAL_STEP: 182750 | > loss_text_ce: 0.04434765875339508 (0.04812080543205664) | > loss_mel_ce: 0.0478515625 (nan) | > loss: 0.0003658699570223689 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13629652068331574) | > loader_time: 0.0046 (0.004643029925196346)  --> TIME: 2025-02-24 05:21:39 -- STEP: 35048/36938 -- GLOBAL_STEP: 182800 | > loss_text_ce: 0.04392196238040924 (0.04811942769145747) | > loss_mel_ce: 3.2060835361480713 (nan) | > loss: 0.012896847911179066 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13629258908901262) | > loader_time: 0.0046 (0.004643041578446101)  --> TIME: 2025-02-24 05:21:49 -- STEP: 35098/36938 -- GLOBAL_STEP: 182850 | > loss_text_ce: 0.05460439622402191 (0.0481159013144726) | > loss_mel_ce: 3.819263458251953 (nan) | > loss: 0.015372492372989655 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13629703020681472) | > loader_time: 0.0049 (0.004643103242076715)  --> TIME: 2025-02-24 05:21:58 -- STEP: 35148/36938 -- GLOBAL_STEP: 182900 | > loss_text_ce: 0.042603474110364914 (0.048113271832654646) | > loss_mel_ce: 3.607415199279785 (nan) | > loss: 0.014484201557934284 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13629452792453578) | > loader_time: 0.0046 (0.0046431184479763556)  --> TIME: 2025-02-24 05:22:07 -- STEP: 35198/36938 -- GLOBAL_STEP: 182950 | > loss_text_ce: 0.051559243351221085 (0.04811217376458097) | > loss_mel_ce: 0.8434637784957886 (nan) | > loss: 0.003551678964868188 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13628879342745032) | > loader_time: 0.0045 (0.004643148682023861)  --> TIME: 2025-02-24 05:22:16 -- STEP: 35248/36938 -- GLOBAL_STEP: 183000 | > loss_text_ce: 0.041897471994161606 (0.048110393642920794) | > loss_mel_ce: 3.4273688793182373 (nan) | > loss: 0.013766931369900703 (nan) | > current_lr: 5e-06 | > step_time: 0.1612 (0.13627896208656134) | > loader_time: 0.0046 (0.004643105961385781)  --> TIME: 2025-02-24 05:22:25 -- STEP: 35298/36938 -- GLOBAL_STEP: 183050 | > loss_text_ce: 0.041109006851911545 (0.04810749339035831) | > loss_mel_ce: 3.4056148529052734 (nan) | > loss: 0.013677476905286312 (nan) | > current_lr: 5e-06 | > step_time: 0.187 (0.13628208887152363) | > loader_time: 0.0048 (0.004643142449632241)  --> TIME: 2025-02-24 05:22:34 -- STEP: 35348/36938 -- GLOBAL_STEP: 183100 | > loss_text_ce: 0.038169097155332565 (0.04810416754077491) | > loss_mel_ce: 3.2015631198883057 (nan) | > loss: 0.012856081128120422 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13627956605435285) | > loader_time: 0.0044 (0.0046431797249791825)  --> TIME: 2025-02-24 05:22:44 -- STEP: 35398/36938 -- GLOBAL_STEP: 183150 | > loss_text_ce: 0.05298352241516113 (0.04810187092867155) | > loss_mel_ce: 3.800607204437256 (nan) | > loss: 0.015292027965188026 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13627712331496464) | > loader_time: 0.0046 (0.00464318205296061)  --> TIME: 2025-02-24 05:22:53 -- STEP: 35448/36938 -- GLOBAL_STEP: 183200 | > loss_text_ce: 0.0554104708135128 (0.0480994934800084) | > loss_mel_ce: 3.050835609436035 (nan) | > loss: 0.012326373718678951 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13627079150069413) | > loader_time: 0.0046 (0.004643126679876546)  --> TIME: 2025-02-24 05:23:02 -- STEP: 35498/36938 -- GLOBAL_STEP: 183250 | > loss_text_ce: 0.04046979919075966 (0.048099131552725376) | > loss_mel_ce: 3.9796667098999023 (nan) | > loss: 0.01595292240381241 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13626763682827947) | > loader_time: 0.0046 (0.0046430632083327)  --> TIME: 2025-02-24 05:23:11 -- STEP: 35548/36938 -- GLOBAL_STEP: 183300 | > loss_text_ce: 0.04934130981564522 (0.048096346927509616) | > loss_mel_ce: 3.5140485763549805 (nan) | > loss: 0.014140436425805092 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.1362661370560939) | > loader_time: 0.0048 (0.004643068306086859)  --> TIME: 2025-02-24 05:23:21 -- STEP: 35598/36938 -- GLOBAL_STEP: 183350 | > loss_text_ce: 0.03951161727309227 (0.048093998572655824) | > loss_mel_ce: 3.8248353004455566 (nan) | > loss: 0.015334711410105228 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13627218754079135) | > loader_time: 0.0046 (0.004643099322346631)  --> TIME: 2025-02-24 05:23:29 -- STEP: 35648/36938 -- GLOBAL_STEP: 183400 | > loss_text_ce: 0.03745776414871216 (0.04809146869613205) | > loss_mel_ce: 2.904863119125366 (nan) | > loss: 0.011675876565277576 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13626281771962873) | > loader_time: 0.0046 (0.004643026906562244)  --> TIME: 2025-02-24 05:23:39 -- STEP: 35698/36938 -- GLOBAL_STEP: 183450 | > loss_text_ce: 0.05048755556344986 (0.048089124613829116) | > loss_mel_ce: 4.033655643463135 (nan) | > loss: 0.01620691828429699 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13627044158367022) | > loader_time: 0.0046 (0.004643060411817325)  --> TIME: 2025-02-24 05:23:48 -- STEP: 35748/36938 -- GLOBAL_STEP: 183500 | > loss_text_ce: 0.051892299205064774 (0.04808829866344598) | > loss_mel_ce: 3.154625177383423 (nan) | > loss: 0.012724276632070541 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.13626862809444015) | > loader_time: 0.0046 (0.004643065104805229)  --> TIME: 2025-02-24 05:23:58 -- STEP: 35798/36938 -- GLOBAL_STEP: 183550 | > loss_text_ce: 0.041472289711236954 (0.0480849814505088) | > loss_mel_ce: 3.393195629119873 (nan) | > loss: 0.013629634864628315 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13627371943072422) | > loader_time: 0.0045 (0.004643121433828166)  --> TIME: 2025-02-24 05:24:07 -- STEP: 35848/36938 -- GLOBAL_STEP: 183600 | > loss_text_ce: 0.04971727356314659 (0.04808221853976149) | > loss_mel_ce: 3.5350143909454346 (nan) | > loss: 0.01422512624412775 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13626712024597815) | > loader_time: 0.0046 (0.00464303989384862)  --> TIME: 2025-02-24 05:24:16 -- STEP: 35898/36938 -- GLOBAL_STEP: 183650 | > loss_text_ce: 0.04812496528029442 (0.04807816132369393) | > loss_mel_ce: 4.285416603088379 (nan) | > loss: 0.017196593806147575 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13626918028148863) | > loader_time: 0.0046 (0.004643014144270507)  --> TIME: 2025-02-24 05:24:26 -- STEP: 35948/36938 -- GLOBAL_STEP: 183700 | > loss_text_ce: 0.05115777999162674 (0.04807658414171955) | > loss_mel_ce: 3.9558427333831787 (nan) | > loss: 0.01590079627931118 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13627860033964623) | > loader_time: 0.0045 (0.004643041140194589)  --> TIME: 2025-02-24 05:24:35 -- STEP: 35998/36938 -- GLOBAL_STEP: 183750 | > loss_text_ce: 0.057685572654008865 (0.0480738572405683) | > loss_mel_ce: 3.190004348754883 (nan) | > loss: 0.01288765948265791 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.1362856551073458) | > loader_time: 0.0048 (0.004643001598254521)  --> TIME: 2025-02-24 05:24:45 -- STEP: 36048/36938 -- GLOBAL_STEP: 183800 | > loss_text_ce: 0.03647099435329437 (0.04807117068062736) | > loss_mel_ce: 2.4272851943969727 (nan) | > loss: 0.009776810184121132 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.1362848290260048) | > loader_time: 0.0045 (0.004642992947189429)  --> TIME: 2025-02-24 05:24:54 -- STEP: 36098/36938 -- GLOBAL_STEP: 183850 | > loss_text_ce: 0.04482521861791611 (0.0480697890398154) | > loss_mel_ce: 3.8052711486816406 (nan) | > loss: 0.015278161503374577 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.1362834856103196) | > loader_time: 0.0065 (0.0046430868061303875)  --> TIME: 2025-02-24 05:25:03 -- STEP: 36148/36938 -- GLOBAL_STEP: 183900 | > loss_text_ce: 0.045636095106601715 (0.04806750004060858) | > loss_mel_ce: 3.924877882003784 (nan) | > loss: 0.01575600914657116 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.13628558717075456) | > loader_time: 0.0048 (0.00464313913001098)  --> TIME: 2025-02-24 05:25:13 -- STEP: 36198/36938 -- GLOBAL_STEP: 183950 | > loss_text_ce: 0.05445847287774086 (0.04806504276443162) | > loss_mel_ce: 4.328125 (nan) | > loss: 0.017391206696629524 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13629301029435023) | > loader_time: 0.0046 (0.004643160668886381)  --> TIME: 2025-02-24 05:25:23 -- STEP: 36248/36938 -- GLOBAL_STEP: 184000 | > loss_text_ce: 0.04395530745387077 (0.04806382396403642) | > loss_mel_ce: 3.834829807281494 (nan) | > loss: 0.015392005443572998 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13629748766139882) | > loader_time: 0.0047 (0.004643268273171936)  --> TIME: 2025-02-24 05:25:32 -- STEP: 36298/36938 -- GLOBAL_STEP: 184050 | > loss_text_ce: 0.05563425272703171 (0.04806304973132907) | > loss_mel_ce: 3.3137497901916504 (nan) | > loss: 0.013370572589337826 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.1362933030344727) | > loader_time: 0.0045 (0.004643219753076005)  --> TIME: 2025-02-24 05:25:41 -- STEP: 36348/36938 -- GLOBAL_STEP: 184100 | > loss_text_ce: 0.05747814103960991 (0.048060470460457576) | > loss_mel_ce: 3.6680102348327637 (nan) | > loss: 0.014783685095608234 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13629503579138302) | > loader_time: 0.0045 (0.004643265532229856)  --> TIME: 2025-02-24 05:25:51 -- STEP: 36398/36938 -- GLOBAL_STEP: 184150 | > loss_text_ce: 0.04608539491891861 (0.0480597464559294) | > loss_mel_ce: 3.779895782470703 (nan) | > loss: 0.015182465314865112 (nan) | > current_lr: 5e-06 | > step_time: 0.1519 (0.13630048762547547) | > loader_time: 0.0046 (0.004643181908477161)  --> TIME: 2025-02-24 05:26:00 -- STEP: 36448/36938 -- GLOBAL_STEP: 184200 | > loss_text_ce: 0.047444503754377365 (0.04805655035587784) | > loss_mel_ce: 3.391042947769165 (nan) | > loss: 0.013644792139530182 (nan) | > current_lr: 5e-06 | > step_time: 0.1455 (0.136302412210379) | > loader_time: 0.0048 (0.00464315406317456)  --> TIME: 2025-02-24 05:26:10 -- STEP: 36498/36938 -- GLOBAL_STEP: 184250 | > loss_text_ce: 0.04203660786151886 (0.04805560593350246) | > loss_mel_ce: 3.5985071659088135 (nan) | > loss: 0.014446603134274483 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13631184191643603) | > loader_time: 0.0045 (0.004643121427546153)  --> TIME: 2025-02-24 05:26:19 -- STEP: 36548/36938 -- GLOBAL_STEP: 184300 | > loss_text_ce: 0.048043135553598404 (0.04805534066561765) | > loss_mel_ce: 2.9917373657226562 (nan) | > loss: 0.012062622234225273 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13631229819700527) | > loader_time: 0.0045 (0.00464308856156458)  --> TIME: 2025-02-24 05:26:28 -- STEP: 36598/36938 -- GLOBAL_STEP: 184350 | > loss_text_ce: 0.0459190309047699 (0.04805370989070414) | > loss_mel_ce: 3.1063055992126465 (nan) | > loss: 0.012508828192949295 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13630814171598465) | > loader_time: 0.0046 (0.004643005877611086)  --> TIME: 2025-02-24 05:26:37 -- STEP: 36648/36938 -- GLOBAL_STEP: 184400 | > loss_text_ce: 0.04494330286979675 (0.04805345421568443) | > loss_mel_ce: 3.491488218307495 (nan) | > loss: 0.01403345912694931 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.13631075057913833) | > loader_time: 0.0046 (0.0046429531240119795)  --> TIME: 2025-02-24 05:26:46 -- STEP: 36698/36938 -- GLOBAL_STEP: 184450 | > loss_text_ce: 0.04570797458291054 (0.048051356595424664) | > loss_mel_ce: 2.984334707260132 (nan) | > loss: 0.012023979797959328 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1363068049113459) | > loader_time: 0.0046 (0.00464286237160863)  --> TIME: 2025-02-24 05:26:56 -- STEP: 36748/36938 -- GLOBAL_STEP: 184500 | > loss_text_ce: 0.05062633752822876 (0.04805203983240382) | > loss_mel_ce: 3.565469741821289 (nan) | > loss: 0.014349588192999363 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13630654520680852) | > loader_time: 0.0046 (0.0046427876772579015)  --> TIME: 2025-02-24 05:27:05 -- STEP: 36798/36938 -- GLOBAL_STEP: 184550 | > loss_text_ce: 0.04989631101489067 (0.048050166304340336) | > loss_mel_ce: 3.4582929611206055 (nan) | > loss: 0.013921386562287807 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.1363123847165911) | > loader_time: 0.0045 (0.0046427515163532575)  --> TIME: 2025-02-24 05:27:15 -- STEP: 36848/36938 -- GLOBAL_STEP: 184600 | > loss_text_ce: 0.04643874242901802 (0.048047755507903474) | > loss_mel_ce: 2.6683573722839355 (nan) | > loss: 0.010773001238703728 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13631386951395155) | > loader_time: 0.0046 (0.004642764246313671)  --> TIME: 2025-02-24 05:27:24 -- STEP: 36898/36938 -- GLOBAL_STEP: 184650 | > loss_text_ce: 0.03543824702501297 (0.048047690301998576) | > loss_mel_ce: 2.8750152587890625 (nan) | > loss: 0.011549419723451138 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.1363169042080298) | > loader_time: 0.0048 (0.0046427964750644915)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.020912884485603543 (+0.00015006207003449498) | > avg_loss_text_ce: 0.0452646742768512 (-0.0026156542416994055) | > avg_loss_mel_ce: 3.0340980368085426 (-0.10440053030996) | > avg_loss: 3.079362717595431 (-0.1070161804114238) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_184690.pth  > EPOCH: 6/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-24 05:27:56)   --> TIME: 2025-02-24 05:27:59 -- STEP: 10/36938 -- GLOBAL_STEP: 184700 | > loss_text_ce: 0.04844748228788376 (0.04653867930173874) | > loss_mel_ce: 3.504575729370117 (2.7448235154151917) | > loss: 0.014099298976361752 (0.01107683481532149) | > current_lr: 5e-06 | > step_time: 0.1615 (0.15061943531036376) | > loader_time: 0.0047 (0.005012297630310058)  --> TIME: 2025-02-24 05:28:08 -- STEP: 60/36938 -- GLOBAL_STEP: 184750 | > loss_text_ce: 0.04117259010672569 (0.045334544871002434) | > loss_mel_ce: 2.6605942249298096 (3.167501280705134) | > loss: 0.0107212970033288 (0.01274934919953618) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13578875462214152) | > loader_time: 0.0047 (0.004716726144154868)  --> TIME: 2025-02-24 05:28:16 -- STEP: 110/36938 -- GLOBAL_STEP: 184800 | > loss_text_ce: 0.04066731408238411 (0.04552612261507998) | > loss_mel_ce: 3.531916379928589 (3.0900796961377965) | > loss: 0.014176920056343079 (0.01244288094233277) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13281968940388078) | > loader_time: 0.0046 (0.004692235859957606)  --> TIME: 2025-02-24 05:28:26 -- STEP: 160/36938 -- GLOBAL_STEP: 184850 | > loss_text_ce: 0.0582268051803112 (0.0461462196020875) | > loss_mel_ce: 3.6449949741363525 (3.1307168215978884) | > loss: 0.014695325866341591 (0.012606600133494794) | > current_lr: 5e-06 | > step_time: 0.115 (0.13327196091413496) | > loader_time: 0.0044 (0.004673793911933898)  --> TIME: 2025-02-24 05:28:35 -- STEP: 210/36938 -- GLOBAL_STEP: 184900 | > loss_text_ce: 0.04617580771446228 (0.046077338175936816) | > loss_mel_ce: 3.310039520263672 (3.198559716805105) | > loss: 0.013318315148353577 (0.012875544641186327) | > current_lr: 5e-06 | > step_time: 0.1376 (0.1342078163510277) | > loader_time: 0.0064 (0.004676832471575055)  --> TIME: 2025-02-24 05:28:44 -- STEP: 260/36938 -- GLOBAL_STEP: 184950 | > loss_text_ce: 0.04841660335659981 (0.046438538709368836) | > loss_mel_ce: 3.6509013175964355 (nan) | > loss: 0.014679834246635437 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13394880294799796) | > loader_time: 0.0048 (0.00466390481361976)  --> TIME: 2025-02-24 05:28:53 -- STEP: 310/36938 -- GLOBAL_STEP: 185000 | > loss_text_ce: 0.05846844241023064 (0.04632871135047848) | > loss_mel_ce: 0.079665407538414 (nan) | > loss: 0.0005481502157635987 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13454167458318897) | > loader_time: 0.0045 (0.00465112501575101)  --> TIME: 2025-02-24 05:29:03 -- STEP: 360/36938 -- GLOBAL_STEP: 185050 | > loss_text_ce: 0.040989067405462265 (0.04646460301397992) | > loss_mel_ce: 3.6228580474853516 (nan) | > loss: 0.014539076946675777 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13521022399266558) | > loader_time: 0.0046 (0.004644296566645304)  --> TIME: 2025-02-24 05:29:12 -- STEP: 410/36938 -- GLOBAL_STEP: 185100 | > loss_text_ce: 0.05944555252790451 (0.0465809353536404) | > loss_mel_ce: 0.15777587890625 (nan) | > loss: 0.0008619898580946028 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.13523389479009112) | > loader_time: 0.0046 (0.004661914778918752)  --> TIME: 2025-02-24 05:29:22 -- STEP: 460/36938 -- GLOBAL_STEP: 185150 | > loss_text_ce: 0.0510401576757431 (0.04645425669808429) | > loss_mel_ce: 3.3465871810913086 (nan) | > loss: 0.013482648879289627 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13538509970125936) | > loader_time: 0.0049 (0.004655156964841095)  --> TIME: 2025-02-24 05:29:31 -- STEP: 510/36938 -- GLOBAL_STEP: 185200 | > loss_text_ce: 0.042122047394514084 (0.04652876546241199) | > loss_mel_ce: 3.8206117153167725 (nan) | > loss: 0.015328309498727322 (nan) | > current_lr: 5e-06 | > step_time: 0.1789 (0.13566384455736943) | > loader_time: 0.0048 (0.004656760365355247)  --> TIME: 2025-02-24 05:29:40 -- STEP: 560/36938 -- GLOBAL_STEP: 185250 | > loss_text_ce: 0.03275241330265999 (0.04667798007805169) | > loss_mel_ce: 3.3613831996917725 (nan) | > loss: 0.013468793593347073 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.13555254127298078) | > loader_time: 0.0046 (0.004652423092297144)  --> TIME: 2025-02-24 05:29:49 -- STEP: 610/36938 -- GLOBAL_STEP: 185300 | > loss_text_ce: 0.053694795817136765 (0.046717213947692404) | > loss_mel_ce: 3.5991575717926025 (nan) | > loss: 0.014495446346700191 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13547262207406463) | > loader_time: 0.0047 (0.004647416364951209)  --> TIME: 2025-02-24 05:29:59 -- STEP: 660/36938 -- GLOBAL_STEP: 185350 | > loss_text_ce: 0.04786481335759163 (0.04664022249000323) | > loss_mel_ce: 3.487074136734009 (nan) | > loss: 0.01402753684669733 (nan) | > current_lr: 5e-06 | > step_time: 0.1441 (0.1359788529800645) | > loader_time: 0.0045 (0.0046475775314099846)  --> TIME: 2025-02-24 05:30:08 -- STEP: 710/36938 -- GLOBAL_STEP: 185400 | > loss_text_ce: 0.0538492277264595 (0.046629624256313326) | > loss_mel_ce: 3.197641611099243 (nan) | > loss: 0.012902742251753807 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1360413222245766) | > loader_time: 0.0065 (0.0046498634445835164)  --> TIME: 2025-02-24 05:30:18 -- STEP: 760/36938 -- GLOBAL_STEP: 185450 | > loss_text_ce: 0.04383327439427376 (0.046570121503042995) | > loss_mel_ce: 1.554840087890625 (nan) | > loss: 0.0063439421355724335 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13610290351666895) | > loader_time: 0.0045 (0.00464764864821183)  --> TIME: 2025-02-24 05:30:27 -- STEP: 810/36938 -- GLOBAL_STEP: 185500 | > loss_text_ce: 0.042597997933626175 (0.046638888153994945) | > loss_mel_ce: 4.3556413650512695 (nan) | > loss: 0.017453331500291824 (nan) | > current_lr: 5e-06 | > step_time: 0.1449 (0.13608599327228682) | > loader_time: 0.0045 (0.004651132042025338)  --> TIME: 2025-02-24 05:30:36 -- STEP: 860/36938 -- GLOBAL_STEP: 185550 | > loss_text_ce: 0.05093085765838623 (0.04667507069900117) | > loss_mel_ce: 3.1208248138427734 (nan) | > loss: 0.012586332857608795 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13575167240098449) | > loader_time: 0.0045 (0.004645291871802743)  --> TIME: 2025-02-24 05:30:45 -- STEP: 910/36938 -- GLOBAL_STEP: 185600 | > loss_text_ce: 0.046243876218795776 (0.046612750668242456) | > loss_mel_ce: 4.1639885902404785 (nan) | > loss: 0.016707271337509155 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.1359373881266665) | > loader_time: 0.0045 (0.004645269257681703)  --> TIME: 2025-02-24 05:30:54 -- STEP: 960/36938 -- GLOBAL_STEP: 185650 | > loss_text_ce: 0.03862005099654198 (0.04659890205269528) | > loss_mel_ce: 2.0961825847625732 (nan) | > loss: 0.008471439592540264 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.13579599807659765) | > loader_time: 0.0044 (0.004640079289674751)  --> TIME: 2025-02-24 05:31:04 -- STEP: 1010/36938 -- GLOBAL_STEP: 185700 | > loss_text_ce: 0.031845930963754654 (0.04655164173057325) | > loss_mel_ce: 3.4603562355041504 (nan) | > loss: 0.01385794673115015 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1356894849550604) | > loader_time: 0.0045 (0.0046380312135904074)  --> TIME: 2025-02-24 05:31:13 -- STEP: 1060/36938 -- GLOBAL_STEP: 185750 | > loss_text_ce: 0.03887965530157089 (0.046570281883842296) | > loss_mel_ce: 3.359142541885376 (nan) | > loss: 0.01348421536386013 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.1356239399820002) | > loader_time: 0.0044 (0.0046382485695604945)  --> TIME: 2025-02-24 05:31:22 -- STEP: 1110/36938 -- GLOBAL_STEP: 185800 | > loss_text_ce: 0.05088960379362106 (0.04662414620638837) | > loss_mel_ce: 3.227999687194824 (nan) | > loss: 0.013011465780436993 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.13569865699286912) | > loader_time: 0.0047 (0.004637534553940225)  --> TIME: 2025-02-24 05:31:31 -- STEP: 1160/36938 -- GLOBAL_STEP: 185850 | > loss_text_ce: 0.037113647907972336 (0.046652317646858794) | > loss_mel_ce: 2.295558452606201 (nan) | > loss: 0.009256635792553425 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13557842949341067) | > loader_time: 0.0046 (0.0046373342645579275)  --> TIME: 2025-02-24 05:31:40 -- STEP: 1210/36938 -- GLOBAL_STEP: 185900 | > loss_text_ce: 0.05854546278715134 (0.04659478053342834) | > loss_mel_ce: 3.2535181045532227 (nan) | > loss: 0.01314311008900404 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1355868761204489) | > loader_time: 0.0044 (0.004635130472419666)  --> TIME: 2025-02-24 05:31:50 -- STEP: 1260/36938 -- GLOBAL_STEP: 185950 | > loss_text_ce: 0.03882225602865219 (0.046589023517149136) | > loss_mel_ce: 3.694819211959839 (nan) | > loss: 0.014816038310527802 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.1356405702848281) | > loader_time: 0.0046 (0.004635214427160828)  --> TIME: 2025-02-24 05:31:59 -- STEP: 1310/36938 -- GLOBAL_STEP: 186000 | > loss_text_ce: 0.04624224081635475 (0.04664046820110953) | > loss_mel_ce: 3.437364339828491 (nan) | > loss: 0.013823836110532284 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13566308221744192) | > loader_time: 0.0045 (0.004634766542274523)  --> TIME: 2025-02-24 05:32:08 -- STEP: 1360/36938 -- GLOBAL_STEP: 186050 | > loss_text_ce: 0.05276624858379364 (0.046705610746079554) | > loss_mel_ce: 3.7322604656219482 (nan) | > loss: 0.015019948594272137 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13569962399847343) | > loader_time: 0.0045 (0.0046364090021918675)  --> TIME: 2025-02-24 05:32:18 -- STEP: 1410/36938 -- GLOBAL_STEP: 186100 | > loss_text_ce: 0.05513044819235802 (0.04667330719266383) | > loss_mel_ce: 3.4478232860565186 (nan) | > loss: 0.013900610618293285 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.13565184545855122) | > loader_time: 0.0045 (0.0046362298600217035)  --> TIME: 2025-02-24 05:32:27 -- STEP: 1460/36938 -- GLOBAL_STEP: 186150 | > loss_text_ce: 0.05736511945724487 (0.04668384104814028) | > loss_mel_ce: 2.83564829826355 (nan) | > loss: 0.011480213142931461 (nan) | > current_lr: 5e-06 | > step_time: 0.1505 (0.13554853837783987) | > loader_time: 0.0047 (0.004637014702574839)  --> TIME: 2025-02-24 05:32:36 -- STEP: 1510/36938 -- GLOBAL_STEP: 186200 | > loss_text_ce: 0.03484770655632019 (0.046646205258142524) | > loss_mel_ce: 3.685112953186035 (nan) | > loss: 0.014761749655008316 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13552792088085427) | > loader_time: 0.0045 (0.004637895514633475)  --> TIME: 2025-02-24 05:32:45 -- STEP: 1560/36938 -- GLOBAL_STEP: 186250 | > loss_text_ce: 0.044507257640361786 (0.04664971760999512) | > loss_mel_ce: 0.01220703125 (nan) | > loss: 0.0002250567195005715 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.1356538462333188) | > loader_time: 0.0046 (0.004639087579189191)  --> TIME: 2025-02-24 05:32:55 -- STEP: 1610/36938 -- GLOBAL_STEP: 186300 | > loss_text_ce: 0.051573023200035095 (0.046651560061189) | > loss_mel_ce: 3.0700881481170654 (nan) | > loss: 0.012387544848024845 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13573047552049514) | > loader_time: 0.0046 (0.004640305116309871)  --> TIME: 2025-02-24 05:33:04 -- STEP: 1660/36938 -- GLOBAL_STEP: 186350 | > loss_text_ce: 0.05758129805326462 (0.04662500824218239) | > loss_mel_ce: 2.9804000854492188 (nan) | > loss: 0.012055481784045696 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13556744018232958) | > loader_time: 0.0046 (0.00463838663445897)  --> TIME: 2025-02-24 05:33:13 -- STEP: 1710/36938 -- GLOBAL_STEP: 186400 | > loss_text_ce: 0.045968350023031235 (0.04662136226478554) | > loss_mel_ce: 3.8588461875915527 (nan) | > loss: 0.015495296567678452 (nan) | > current_lr: 5e-06 | > step_time: 0.1582 (0.13575050147653311) | > loader_time: 0.0047 (0.004638479188171741)  --> TIME: 2025-02-24 05:33:23 -- STEP: 1760/36938 -- GLOBAL_STEP: 186450 | > loss_text_ce: 0.03924431651830673 (0.04660493495281451) | > loss_mel_ce: 3.4549570083618164 (nan) | > loss: 0.013865879736840725 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.13584500998258578) | > loader_time: 0.0052 (0.004640109972520304)  --> TIME: 2025-02-24 05:33:32 -- STEP: 1810/36938 -- GLOBAL_STEP: 186500 | > loss_text_ce: 0.0376402884721756 (0.046590553483416404) | > loss_mel_ce: 0.14172668755054474 (nan) | > loss: 0.0007117737550288439 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13595233806589022) | > loader_time: 0.0045 (0.004641650790009047)  --> TIME: 2025-02-24 05:33:41 -- STEP: 1860/36938 -- GLOBAL_STEP: 186550 | > loss_text_ce: 0.0578717365860939 (0.04660019630826613) | > loss_mel_ce: 2.2607364654541016 (nan) | > loss: 0.009200827218592167 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13584539672379842) | > loader_time: 0.0045 (0.004640231593962635)  --> TIME: 2025-02-24 05:33:51 -- STEP: 1910/36938 -- GLOBAL_STEP: 186600 | > loss_text_ce: 0.048140645027160645 (0.04661282238730901) | > loss_mel_ce: 3.4429540634155273 (nan) | > loss: 0.013853550888597965 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13592728207872795) | > loader_time: 0.0048 (0.00464132206602246)  --> TIME: 2025-02-24 05:34:00 -- STEP: 1960/36938 -- GLOBAL_STEP: 186650 | > loss_text_ce: 0.04688012972474098 (0.04661190758402254) | > loss_mel_ce: 3.617621421813965 (nan) | > loss: 0.01454167440533638 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.1359550126961299) | > loader_time: 0.0061 (0.004642127363049251)  --> TIME: 2025-02-24 05:34:09 -- STEP: 2010/36938 -- GLOBAL_STEP: 186700 | > loss_text_ce: 0.05174628645181656 (0.04663236023607987) | > loss_mel_ce: 3.8387625217437744 (nan) | > loss: 0.015438527800142765 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13607901471171208) | > loader_time: 0.0047 (0.004643068028919729)  --> TIME: 2025-02-24 05:34:19 -- STEP: 2060/36938 -- GLOBAL_STEP: 186750 | > loss_text_ce: 0.05119648203253746 (0.046641555984585935) | > loss_mel_ce: 3.70530104637146 (nan) | > loss: 0.014906737022101879 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13611557819310896) | > loader_time: 0.0047 (0.004642632864053964)  --> TIME: 2025-02-24 05:34:28 -- STEP: 2110/36938 -- GLOBAL_STEP: 186800 | > loss_text_ce: 0.03808221220970154 (0.04665294466054698) | > loss_mel_ce: 1.2782371044158936 (nan) | > loss: 0.005223489832133055 (nan) | > current_lr: 5e-06 | > step_time: 0.1116 (0.13614330291748045) | > loader_time: 0.0045 (0.00464454714155875)  --> TIME: 2025-02-24 05:34:37 -- STEP: 2160/36938 -- GLOBAL_STEP: 186850 | > loss_text_ce: 0.039600200951099396 (0.04664318140216729) | > loss_mel_ce: 2.711752414703369 (nan) | > loss: 0.010918065905570984 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.1361460743127046) | > loader_time: 0.0046 (0.004645341082855505)  --> TIME: 2025-02-24 05:34:47 -- STEP: 2210/36938 -- GLOBAL_STEP: 186900 | > loss_text_ce: 0.04771808162331581 (0.04664328219242636) | > loss_mel_ce: 4.032164096832275 (nan) | > loss: 0.01619000919163227 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.1360995232249817) | > loader_time: 0.0045 (0.004644733748284939)  --> TIME: 2025-02-24 05:34:56 -- STEP: 2260/36938 -- GLOBAL_STEP: 186950 | > loss_text_ce: 0.039419692009687424 (0.04662474270300138) | > loss_mel_ce: 0.1806773543357849 (nan) | > loss: 0.0008734010625630617 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13614040317788592) | > loader_time: 0.0046 (0.004644680234183254)  --> TIME: 2025-02-24 05:35:05 -- STEP: 2310/36938 -- GLOBAL_STEP: 187000 | > loss_text_ce: 0.042054761201143265 (0.04662348763267334) | > loss_mel_ce: 3.288606882095337 (nan) | > loss: 0.013216911815106869 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13612337969082264) | > loader_time: 0.0046 (0.004643386679810363)  --> TIME: 2025-02-24 05:35:15 -- STEP: 2360/36938 -- GLOBAL_STEP: 187050 | > loss_text_ce: 0.04058080539107323 (0.04662231767423836) | > loss_mel_ce: 2.9998726844787598 (nan) | > loss: 0.012065292336046696 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.13621369236606662) | > loader_time: 0.0048 (0.00464286581944611)  --> TIME: 2025-02-24 05:35:24 -- STEP: 2410/36938 -- GLOBAL_STEP: 187100 | > loss_text_ce: 0.04389386624097824 (0.04660152771753647) | > loss_mel_ce: 4.85546875 (nan) | > loss: 0.019441915675997734 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1361712704060978) | > loader_time: 0.0046 (0.004642305928147182)  --> TIME: 2025-02-24 05:35:33 -- STEP: 2460/36938 -- GLOBAL_STEP: 187150 | > loss_text_ce: 0.04657474532723427 (0.046600220096486886) | > loss_mel_ce: 3.0564866065979004 (nan) | > loss: 0.012313736602663994 (nan) | > current_lr: 5e-06 | > step_time: 0.178 (0.13622103454620862) | > loader_time: 0.0048 (0.004642303300097703)  --> TIME: 2025-02-24 05:35:43 -- STEP: 2510/36938 -- GLOBAL_STEP: 187200 | > loss_text_ce: 0.041458144783973694 (0.04657776954609803) | > loss_mel_ce: 2.924330472946167 (nan) | > loss: 0.011769003234803677 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13626890258485105) | > loader_time: 0.0047 (0.004642408492555657)  --> TIME: 2025-02-24 05:35:52 -- STEP: 2560/36938 -- GLOBAL_STEP: 187250 | > loss_text_ce: 0.03717166930437088 (0.046553842193679966) | > loss_mel_ce: 2.8033013343811035 (nan) | > loss: 0.011271718889474869 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.1362930947914721) | > loader_time: 0.0047 (0.004644400905817751)  --> TIME: 2025-02-24 05:36:01 -- STEP: 2610/36938 -- GLOBAL_STEP: 187300 | > loss_text_ce: 0.036836303770542145 (0.046563507623656596) | > loss_mel_ce: 2.5246315002441406 (nan) | > loss: 0.010164556093513966 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13629436355897764) | > loader_time: 0.0046 (0.004644189301121738)  --> TIME: 2025-02-24 05:36:11 -- STEP: 2660/36938 -- GLOBAL_STEP: 187350 | > loss_text_ce: 0.04912816360592842 (0.04657129458802993) | > loss_mel_ce: 3.8051562309265137 (nan) | > loss: 0.015294780023396015 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13632106189440982) | > loader_time: 0.0046 (0.00464484951549903)  --> TIME: 2025-02-24 05:36:20 -- STEP: 2710/36938 -- GLOBAL_STEP: 187400 | > loss_text_ce: 0.0357915461063385 (0.04656665232857336) | > loss_mel_ce: 2.89688777923584 (nan) | > loss: 0.011637617833912373 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13636934458109737) | > loader_time: 0.0046 (0.004645042665770135)  --> TIME: 2025-02-24 05:36:30 -- STEP: 2760/36938 -- GLOBAL_STEP: 187450 | > loss_text_ce: 0.05390485003590584 (0.046571207327930235) | > loss_mel_ce: 3.0821146965026855 (nan) | > loss: 0.012444522231817245 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.136383278646331) | > loader_time: 0.0045 (0.0046459908934607065)  --> TIME: 2025-02-24 05:36:39 -- STEP: 2810/36938 -- GLOBAL_STEP: 187500 | > loss_text_ce: 0.04441409930586815 (0.04659055805702127) | > loss_mel_ce: 2.988999605178833 (nan) | > loss: 0.01203735638409853 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13634137224896536) | > loader_time: 0.0046 (0.0046456285218751375)  --> TIME: 2025-02-24 05:36:48 -- STEP: 2860/36938 -- GLOBAL_STEP: 187550 | > loss_text_ce: 0.036526355892419815 (0.046589537779637834) | > loss_mel_ce: 3.460110902786255 (nan) | > loss: 0.013875545933842659 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13640179542394806) | > loader_time: 0.0046 (0.004645456884290797)  --> TIME: 2025-02-24 05:36:58 -- STEP: 2910/36938 -- GLOBAL_STEP: 187600 | > loss_text_ce: 0.04455798864364624 (0.04658349979720068) | > loss_mel_ce: 3.3744165897369385 (nan) | > loss: 0.013567360118031502 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.1363857746943578) | > loader_time: 0.0045 (0.00464491057641728)  --> TIME: 2025-02-24 05:37:06 -- STEP: 2960/36938 -- GLOBAL_STEP: 187650 | > loss_text_ce: 0.03552884981036186 (0.04657164705552267) | > loss_mel_ce: 3.1979167461395264 (nan) | > loss: 0.012831133790314198 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13627534766454977) | > loader_time: 0.0046 (0.0046431473783544635)  --> TIME: 2025-02-24 05:37:16 -- STEP: 3010/36938 -- GLOBAL_STEP: 187700 | > loss_text_ce: 0.050941165536642075 (0.046582465468475845) | > loss_mel_ce: 3.5458498001098633 (nan) | > loss: 0.014272981323301792 (nan) | > current_lr: 5e-06 | > step_time: 0.1463 (0.1362562740363949) | > loader_time: 0.0046 (0.004641572343946694)  --> TIME: 2025-02-24 05:37:25 -- STEP: 3060/36938 -- GLOBAL_STEP: 187750 | > loss_text_ce: 0.0465632900595665 (0.04657074405075196) | > loss_mel_ce: 3.503293514251709 (nan) | > loss: 0.014086734503507614 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13625865917579813) | > loader_time: 0.0046 (0.0046415720889770975)  --> TIME: 2025-02-24 05:37:34 -- STEP: 3110/36938 -- GLOBAL_STEP: 187800 | > loss_text_ce: 0.04601694643497467 (0.04655821287938642) | > loss_mel_ce: 3.2227301597595215 (nan) | > loss: 0.01297121960669756 (nan) | > current_lr: 5e-06 | > step_time: 0.1579 (0.1362480862730953) | > loader_time: 0.0048 (0.004641466355400459)  --> TIME: 2025-02-24 05:37:44 -- STEP: 3160/36938 -- GLOBAL_STEP: 187850 | > loss_text_ce: 0.04715574532747269 (0.04654404942677185) | > loss_mel_ce: 3.9107038974761963 (nan) | > loss: 0.015705794095993042 (nan) | > current_lr: 5e-06 | > step_time: 0.187 (0.13625708478915552) | > loader_time: 0.0049 (0.004641820433773577)  --> TIME: 2025-02-24 05:37:53 -- STEP: 3210/36938 -- GLOBAL_STEP: 187900 | > loss_text_ce: 0.040109410881996155 (0.04655022251411024) | > loss_mel_ce: 2.867489814758301 (nan) | > loss: 0.011538092978298664 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.13635905331168988) | > loader_time: 0.0048 (0.00464218925464191)  --> TIME: 2025-02-24 05:38:02 -- STEP: 3260/36938 -- GLOBAL_STEP: 187950 | > loss_text_ce: 0.059031885117292404 (0.04653307695878801) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13633068902360893) | > loader_time: 0.0045 (0.004641271222588482)  --> TIME: 2025-02-24 05:38:12 -- STEP: 3310/36938 -- GLOBAL_STEP: 188000 | > loss_text_ce: 0.043377846479415894 (0.046514419222031166) | > loss_mel_ce: 2.3899307250976562 (nan) | > loss: 0.009655986912548542 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13636975014677952) | > loader_time: 0.0045 (0.004640781843410164)  --> TIME: 2025-02-24 05:38:21 -- STEP: 3360/36938 -- GLOBAL_STEP: 188050 | > loss_text_ce: 0.0496697761118412 (0.04652078885673772) | > loss_mel_ce: 3.6694698333740234 (nan) | > loss: 0.014758490957319736 (nan) | > current_lr: 5e-06 | > step_time: 0.1769 (0.13632469979070483) | > loader_time: 0.0049 (0.004639897389071334)  --> TIME: 2025-02-24 05:38:30 -- STEP: 3410/36938 -- GLOBAL_STEP: 188100 | > loss_text_ce: 0.0501994825899601 (0.04654685237180021) | > loss_mel_ce: 3.391242504119873 (nan) | > loss: 0.013656516559422016 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13629235102633921) | > loader_time: 0.0048 (0.004640741012662738)  --> TIME: 2025-02-24 05:38:39 -- STEP: 3460/36938 -- GLOBAL_STEP: 188150 | > loss_text_ce: 0.04730859771370888 (0.046540764864060453) | > loss_mel_ce: 3.3480851650238037 (nan) | > loss: 0.013473785482347012 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.13622713337054837) | > loader_time: 0.0047 (0.004640447886692995)  --> TIME: 2025-02-24 05:38:48 -- STEP: 3510/36938 -- GLOBAL_STEP: 188200 | > loss_text_ce: 0.05199151858687401 (0.04654142610155624) | > loss_mel_ce: 2.131899833679199 (nan) | > loss: 0.008666235953569412 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13625659168276005) | > loader_time: 0.0045 (0.004639988882928832)  --> TIME: 2025-02-24 05:38:58 -- STEP: 3560/36938 -- GLOBAL_STEP: 188250 | > loss_text_ce: 0.06631407886743546 (0.04655214009618161) | > loss_mel_ce: 3.730797290802002 (nan) | > loss: 0.015067902393639088 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.1362767433182575) | > loader_time: 0.0045 (0.004639277766259869)  --> TIME: 2025-02-24 05:39:07 -- STEP: 3610/36938 -- GLOBAL_STEP: 188300 | > loss_text_ce: 0.048485755920410156 (0.046567818109162835) | > loss_mel_ce: 3.1792819499969482 (nan) | > loss: 0.012808602303266525 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.1363218014920516) | > loader_time: 0.0046 (0.004639393603042222)  --> TIME: 2025-02-24 05:39:17 -- STEP: 3660/36938 -- GLOBAL_STEP: 188350 | > loss_text_ce: 0.04613460972905159 (0.04655740794781702) | > loss_mel_ce: 3.538802146911621 (nan) | > loss: 0.014225941151380539 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13633389121196288) | > loader_time: 0.0046 (0.004639049957358772)  --> TIME: 2025-02-24 05:39:26 -- STEP: 3710/36938 -- GLOBAL_STEP: 188400 | > loss_text_ce: 0.042926352471113205 (0.046559287176901804) | > loss_mel_ce: 2.7964038848876953 (nan) | > loss: 0.011267184279859066 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.13635777880881983) | > loader_time: 0.0047 (0.004638946988190602)  --> TIME: 2025-02-24 05:39:35 -- STEP: 3760/36938 -- GLOBAL_STEP: 188450 | > loss_text_ce: 0.04880993440747261 (0.04655176321282353) | > loss_mel_ce: 2.387911558151245 (nan) | > loss: 0.0096695302054286 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13634447277860426) | > loader_time: 0.0046 (0.004638889685590217)  --> TIME: 2025-02-24 05:39:45 -- STEP: 3810/36938 -- GLOBAL_STEP: 188500 | > loss_text_ce: 0.04837719351053238 (0.04655212286194874) | > loss_mel_ce: 3.5800206661224365 (nan) | > loss: 0.014398405328392982 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13633770060351502) | > loader_time: 0.0046 (0.004638652976729426)  --> TIME: 2025-02-24 05:39:54 -- STEP: 3860/36938 -- GLOBAL_STEP: 188550 | > loss_text_ce: 0.0690559670329094 (0.04656908103202141) | > loss_mel_ce: 2.797558069229126 (nan) | > loss: 0.01137545332312584 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13633481684126372) | > loader_time: 0.0047 (0.004637903438330932)  --> TIME: 2025-02-24 05:40:03 -- STEP: 3910/36938 -- GLOBAL_STEP: 188600 | > loss_text_ce: 0.04250268265604973 (0.04659603007823767) | > loss_mel_ce: 2.7977828979492188 (nan) | > loss: 0.011270974762737751 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1363462288361375) | > loader_time: 0.0046 (0.004638025461865203)  --> TIME: 2025-02-24 05:40:13 -- STEP: 3960/36938 -- GLOBAL_STEP: 188650 | > loss_text_ce: 0.04170116409659386 (0.04659058284484833) | > loss_mel_ce: 3.3642804622650146 (nan) | > loss: 0.01351580023765564 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13632688407946103) | > loader_time: 0.0045 (0.004638049939666131)  --> TIME: 2025-02-24 05:40:22 -- STEP: 4010/36938 -- GLOBAL_STEP: 188700 | > loss_text_ce: 0.041705917567014694 (0.04659295350852011) | > loss_mel_ce: 3.4968414306640625 (nan) | > loss: 0.014041854999959469 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1363595915554169) | > loader_time: 0.0045 (0.0046385284076605054)  --> TIME: 2025-02-24 05:40:31 -- STEP: 4060/36938 -- GLOBAL_STEP: 188750 | > loss_text_ce: 0.04546903073787689 (0.0465843174046851) | > loss_mel_ce: 3.749131441116333 (nan) | > loss: 0.015057939104735851 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.13634506434642613) | > loader_time: 0.0046 (0.004639124929024081)  --> TIME: 2025-02-24 05:40:41 -- STEP: 4110/36938 -- GLOBAL_STEP: 188800 | > loss_text_ce: 0.05222136527299881 (0.0465679508382387) | > loss_mel_ce: 3.6418018341064453 (nan) | > loss: 0.014658822678029537 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13633897327738656) | > loader_time: 0.0045 (0.004639125624422319)  --> TIME: 2025-02-24 05:40:50 -- STEP: 4160/36938 -- GLOBAL_STEP: 188850 | > loss_text_ce: 0.04184432327747345 (0.046566875201041066) | > loss_mel_ce: 3.9068331718444824 (nan) | > loss: 0.015669357031583786 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.13638251458223039) | > loader_time: 0.0047 (0.004638855732404269)  --> TIME: 2025-02-24 05:40:59 -- STEP: 4210/36938 -- GLOBAL_STEP: 188900 | > loss_text_ce: 0.04740607738494873 (0.046563623019996316) | > loss_mel_ce: 3.2448878288269043 (nan) | > loss: 0.01306465920060873 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13638905442525526) | > loader_time: 0.0046 (0.004638132573306703)  --> TIME: 2025-02-24 05:41:09 -- STEP: 4260/36938 -- GLOBAL_STEP: 188950 | > loss_text_ce: 0.044623810797929764 (0.046567013077088125) | > loss_mel_ce: 2.363616943359375 (nan) | > loss: 0.009556511417031288 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13643529672577895) | > loader_time: 0.0045 (0.004637694134958474)  --> TIME: 2025-02-24 05:41:18 -- STEP: 4310/36938 -- GLOBAL_STEP: 189000 | > loss_text_ce: 0.04440917447209358 (0.04654096121837122) | > loss_mel_ce: 3.2049612998962402 (nan) | > loss: 0.01289432868361473 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.1364568305955961) | > loader_time: 0.0048 (0.004637336896909635)  --> TIME: 2025-02-24 05:41:27 -- STEP: 4360/36938 -- GLOBAL_STEP: 189050 | > loss_text_ce: 0.032016631215810776 (0.04653561292045299) | > loss_mel_ce: 2.3597004413604736 (nan) | > loss: 0.00949094071984291 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13639776520772803) | > loader_time: 0.0046 (0.004636849613364677)  --> TIME: 2025-02-24 05:41:37 -- STEP: 4410/36938 -- GLOBAL_STEP: 189100 | > loss_text_ce: 0.04440590739250183 (0.04653491528742777) | > loss_mel_ce: 2.775226354598999 (nan) | > loss: 0.011189017444849014 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13644070500959882) | > loader_time: 0.0047 (0.004636848432406808)  --> TIME: 2025-02-24 05:41:46 -- STEP: 4460/36938 -- GLOBAL_STEP: 189150 | > loss_text_ce: 0.043591029942035675 (0.04654122147199621) | > loss_mel_ce: 3.7713115215301514 (nan) | > loss: 0.015138503164052963 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13646319104951585) | > loader_time: 0.0045 (0.0046371201228667695)  --> TIME: 2025-02-24 05:41:55 -- STEP: 4510/36938 -- GLOBAL_STEP: 189200 | > loss_text_ce: 0.05124196782708168 (0.04653789112654137) | > loss_mel_ce: 3.259190082550049 (nan) | > loss: 0.013136635534465313 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.1364234471268239) | > loader_time: 0.0045 (0.00463684583186045)  --> TIME: 2025-02-24 05:42:05 -- STEP: 4560/36938 -- GLOBAL_STEP: 189250 | > loss_text_ce: 0.0494520328938961 (0.04654929187075282) | > loss_mel_ce: 3.413985013961792 (nan) | > loss: 0.013743799179792404 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13644487272229083) | > loader_time: 0.0045 (0.004636602757269864)  --> TIME: 2025-02-24 05:42:14 -- STEP: 4610/36938 -- GLOBAL_STEP: 189300 | > loss_text_ce: 0.032049499452114105 (0.046531817523071645) | > loss_mel_ce: 3.4959359169006348 (nan) | > loss: 0.013999942690134048 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.13643403854907957) | > loader_time: 0.0048 (0.004636070268014496)  --> TIME: 2025-02-24 05:42:23 -- STEP: 4660/36938 -- GLOBAL_STEP: 189350 | > loss_text_ce: 0.05618201941251755 (0.04654864743911523) | > loss_mel_ce: 3.2388112545013428 (nan) | > loss: 0.013075370341539383 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13642982873793819) | > loader_time: 0.0046 (0.0046356634520665655)  --> TIME: 2025-02-24 05:42:33 -- STEP: 4710/36938 -- GLOBAL_STEP: 189400 | > loss_text_ce: 0.04346952214837074 (0.04655017243528183) | > loss_mel_ce: 1.065232753753662 (nan) | > loss: 0.004399612545967102 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13642609240902445) | > loader_time: 0.0047 (0.004635353554079731)  --> TIME: 2025-02-24 05:42:42 -- STEP: 4760/36938 -- GLOBAL_STEP: 189450 | > loss_text_ce: 0.04648253321647644 (0.04654489314934907) | > loss_mel_ce: 3.012368679046631 (nan) | > loss: 0.012138299643993378 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.1364697678750302) | > loader_time: 0.0048 (0.0046352811220313215)  --> TIME: 2025-02-24 05:42:52 -- STEP: 4810/36938 -- GLOBAL_STEP: 189500 | > loss_text_ce: 0.04583913832902908 (0.04654803660440712) | > loss_mel_ce: 2.9307000637054443 (nan) | > loss: 0.011811664327979088 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.13651343061125948) | > loader_time: 0.0045 (0.004635141743443848)  --> TIME: 2025-02-24 05:43:01 -- STEP: 4860/36938 -- GLOBAL_STEP: 189550 | > loss_text_ce: 0.042572300881147385 (0.04653356072412406) | > loss_mel_ce: 0.0146484375 (nan) | > loss: 0.00022706644085701555 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1365354161203644) | > loader_time: 0.0046 (0.004635019949924787)  --> TIME: 2025-02-24 05:43:11 -- STEP: 4910/36938 -- GLOBAL_STEP: 189600 | > loss_text_ce: 0.054519519209861755 (0.046519558030406284) | > loss_mel_ce: 2.8359375 (nan) | > loss: 0.011470068246126175 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13653428336025014) | > loader_time: 0.0045 (0.004634708833791562)  --> TIME: 2025-02-24 05:43:20 -- STEP: 4960/36938 -- GLOBAL_STEP: 189650 | > loss_text_ce: 0.04704009369015694 (0.04652640190568839) | > loss_mel_ce: 4.2958598136901855 (nan) | > loss: 0.017233731225132942 (nan) | > current_lr: 5e-06 | > step_time: 0.1353 (0.1365384329711238) | > loader_time: 0.0046 (0.004634331407085531)  --> TIME: 2025-02-24 05:43:29 -- STEP: 5010/36938 -- GLOBAL_STEP: 189700 | > loss_text_ce: 0.05014025419950485 (0.04652291484504281) | > loss_mel_ce: 3.7295947074890137 (nan) | > loss: 0.014998949132859707 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13652781707322062) | > loader_time: 0.0045 (0.004634234005819526)  --> TIME: 2025-02-24 05:43:38 -- STEP: 5060/36938 -- GLOBAL_STEP: 189750 | > loss_text_ce: 0.056155748665332794 (0.04650694774684637) | > loss_mel_ce: 3.8804688453674316 (nan) | > loss: 0.015621527098119259 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13649870769779684) | > loader_time: 0.0045 (0.004633892996037891)  --> TIME: 2025-02-24 05:43:47 -- STEP: 5110/36938 -- GLOBAL_STEP: 189800 | > loss_text_ce: 0.04076169431209564 (0.046499221977903515) | > loss_mel_ce: 4.192708492279053 (nan) | > loss: 0.016799485310912132 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.1364490389590628) | > loader_time: 0.0045 (0.004633519700828585)  --> TIME: 2025-02-24 05:43:57 -- STEP: 5160/36938 -- GLOBAL_STEP: 189850 | > loss_text_ce: 0.04461875930428505 (0.04649139352236663) | > loss_mel_ce: 3.343168020248413 (nan) | > loss: 0.013443599455058575 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.1364950154178827) | > loader_time: 0.0045 (0.004633649559908125)  --> TIME: 2025-02-24 05:44:06 -- STEP: 5210/36938 -- GLOBAL_STEP: 189900 | > loss_text_ce: 0.0631929486989975 (0.046487391313570096) | > loss_mel_ce: 3.364443778991699 (nan) | > loss: 0.01360173337161541 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13646226071121598) | > loader_time: 0.0046 (0.004633725261505177)  --> TIME: 2025-02-24 05:44:16 -- STEP: 5260/36938 -- GLOBAL_STEP: 189950 | > loss_text_ce: 0.048028867691755295 (0.046487200075668395) | > loss_mel_ce: 2.345691680908203 (nan) | > loss: 0.009498892351984978 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13649106157143306) | > loader_time: 0.0045 (0.00463328007962766)  --> TIME: 2025-02-24 05:44:25 -- STEP: 5310/36938 -- GLOBAL_STEP: 190000 | > loss_text_ce: 0.05385281518101692 (0.0464941731648846) | > loss_mel_ce: 3.283621311187744 (nan) | > loss: 0.013243945315480232 (nan) | > current_lr: 5e-06 | > step_time: 0.1508 (0.1364525987603569) | > loader_time: 0.0047 (0.004633386840030058) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_190000.pth  --> TIME: 2025-02-24 05:44:40 -- STEP: 5360/36938 -- GLOBAL_STEP: 190050 | > loss_text_ce: 0.050359781831502914 (0.04649610724717158) | > loss_mel_ce: 3.890550136566162 (nan) | > loss: 0.015638532117009163 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13641998127325242) | > loader_time: 0.0046 (0.00463387636996026)  --> TIME: 2025-02-24 05:44:49 -- STEP: 5410/36938 -- GLOBAL_STEP: 190100 | > loss_text_ce: 0.032963935285806656 (0.04651257444978753) | > loss_mel_ce: 3.612062931060791 (nan) | > loss: 0.014464393258094788 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.1364488137627704) | > loader_time: 0.0047 (0.004634370028201387)  --> TIME: 2025-02-24 05:44:58 -- STEP: 5460/36938 -- GLOBAL_STEP: 190150 | > loss_text_ce: 0.036089275032281876 (0.04650053366704849) | > loss_mel_ce: 0.02923583984375 (nan) | > loss: 0.00025922665372490883 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13635400459443245) | > loader_time: 0.0046 (0.004634354097065896)  --> TIME: 2025-02-24 05:45:07 -- STEP: 5510/36938 -- GLOBAL_STEP: 190200 | > loss_text_ce: 0.046186741441488266 (0.046484105432314814) | > loss_mel_ce: 3.2057008743286133 (nan) | > loss: 0.012904316186904907 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13636397480315599) | > loader_time: 0.0046 (0.004634735155884443)  --> TIME: 2025-02-24 05:45:16 -- STEP: 5560/36938 -- GLOBAL_STEP: 190250 | > loss_text_ce: 0.04915099963545799 (0.046499495415298436) | > loss_mel_ce: 3.381510019302368 (nan) | > loss: 0.013613734394311905 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13632809005195273) | > loader_time: 0.0045 (0.004635080545068635)  --> TIME: 2025-02-24 05:45:25 -- STEP: 5610/36938 -- GLOBAL_STEP: 190300 | > loss_text_ce: 0.05092719942331314 (0.04650298016155788) | > loss_mel_ce: 2.934368133544922 (nan) | > loss: 0.011846411041915417 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13629479914000522) | > loader_time: 0.0046 (0.004635278723881621)  --> TIME: 2025-02-24 05:45:35 -- STEP: 5660/36938 -- GLOBAL_STEP: 190350 | > loss_text_ce: 0.04645664989948273 (0.04652509697360834) | > loss_mel_ce: 3.8445165157318115 (nan) | > loss: 0.015440369956195354 (nan) | > current_lr: 5e-06 | > step_time: 0.1713 (0.1363139482353263) | > loader_time: 0.005 (0.004635038603320972)  --> TIME: 2025-02-24 05:45:44 -- STEP: 5710/36938 -- GLOBAL_STEP: 190400 | > loss_text_ce: 0.05340312793850899 (0.0465311276946615) | > loss_mel_ce: 3.2273905277252197 (nan) | > loss: 0.013019023463129997 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13636595188214348) | > loader_time: 0.0045 (0.004635218480004941)  --> TIME: 2025-02-24 05:45:53 -- STEP: 5760/36938 -- GLOBAL_STEP: 190450 | > loss_text_ce: 0.05886901542544365 (0.046545026226926405) | > loss_mel_ce: 2.7184696197509766 (nan) | > loss: 0.011021185666322708 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13633597050276064) | > loader_time: 0.0044 (0.004635132269726853)  --> TIME: 2025-02-24 05:46:03 -- STEP: 5810/36938 -- GLOBAL_STEP: 190500 | > loss_text_ce: 0.04429693892598152 (0.04654270257807231) | > loss_mel_ce: 3.194145441055298 (nan) | > loss: 0.012850962579250336 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.1363730037479105) | > loader_time: 0.0045 (0.004635766410171055)  --> TIME: 2025-02-24 05:46:12 -- STEP: 5860/36938 -- GLOBAL_STEP: 190550 | > loss_text_ce: 0.04817129671573639 (0.046525936981555056) | > loss_mel_ce: 1.5606155395507812 (nan) | > loss: 0.006384075153619051 (nan) | > current_lr: 5e-06 | > step_time: 0.1648 (0.1363349783949477) | > loader_time: 0.0047 (0.004635696199566829)  --> TIME: 2025-02-24 05:46:21 -- STEP: 5910/36938 -- GLOBAL_STEP: 190600 | > loss_text_ce: 0.055728986859321594 (0.046513260502916626) | > loss_mel_ce: 2.682685613632202 (nan) | > loss: 0.010866724886000156 (nan) | > current_lr: 5e-06 | > step_time: 0.1118 (0.13634901599432203) | > loader_time: 0.0045 (0.004635659974642001)  --> TIME: 2025-02-24 05:46:31 -- STEP: 5960/36938 -- GLOBAL_STEP: 190650 | > loss_text_ce: 0.048341549932956696 (0.046507993551446206) | > loss_mel_ce: 2.4909238815307617 (nan) | > loss: 0.010076450183987617 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.1363515824279529) | > loader_time: 0.0046 (0.004635524069702858)  --> TIME: 2025-02-24 05:46:40 -- STEP: 6010/36938 -- GLOBAL_STEP: 190700 | > loss_text_ce: 0.05304834619164467 (0.04651920727784197) | > loss_mel_ce: 3.6894712448120117 (nan) | > loss: 0.01485126931220293 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13637811328329383) | > loader_time: 0.0046 (0.004635919628048094)  --> TIME: 2025-02-24 05:46:49 -- STEP: 6060/36938 -- GLOBAL_STEP: 190750 | > loss_text_ce: 0.047076791524887085 (0.04652557162684774) | > loss_mel_ce: 3.8581957817077637 (nan) | > loss: 0.015497113578021526 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13634721331863914) | > loader_time: 0.0046 (0.004635651630930376)  --> TIME: 2025-02-24 05:46:58 -- STEP: 6110/36938 -- GLOBAL_STEP: 190800 | > loss_text_ce: 0.040993280708789825 (0.04652921131243489) | > loss_mel_ce: 3.0943503379821777 (nan) | > loss: 0.012441840022802353 (nan) | > current_lr: 5e-06 | > step_time: 0.1261 (0.13630585198316567) | > loader_time: 0.0047 (0.0046356084108742845)  --> TIME: 2025-02-24 05:47:08 -- STEP: 6160/36938 -- GLOBAL_STEP: 190850 | > loss_text_ce: 0.037603724747896194 (0.04651499036064543) | > loss_mel_ce: 3.5749707221984863 (nan) | > loss: 0.014335613697767258 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1362985876086472) | > loader_time: 0.0045 (0.004635132055778008)  --> TIME: 2025-02-24 05:47:17 -- STEP: 6210/36938 -- GLOBAL_STEP: 190900 | > loss_text_ce: 0.03454884886741638 (0.04652757163574972) | > loss_mel_ce: 3.167543888092041 (nan) | > loss: 0.012706717476248741 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13633018423009827) | > loader_time: 0.0045 (0.0046350218055712855)  --> TIME: 2025-02-24 05:47:26 -- STEP: 6260/36938 -- GLOBAL_STEP: 190950 | > loss_text_ce: 0.049126897007226944 (0.04651950312266574) | > loss_mel_ce: 3.3139424324035645 (nan) | > loss: 0.013345513492822647 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13630057767557283) | > loader_time: 0.0047 (0.004634448057546405)  --> TIME: 2025-02-24 05:47:35 -- STEP: 6310/36938 -- GLOBAL_STEP: 191000 | > loss_text_ce: 0.0400124117732048 (0.04652777458858426) | > loss_mel_ce: 2.991180419921875 (nan) | > loss: 0.012028543278574944 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13626116614334197) | > loader_time: 0.0045 (0.004634114566204481)  --> TIME: 2025-02-24 05:47:44 -- STEP: 6360/36938 -- GLOBAL_STEP: 191050 | > loss_text_ce: 0.05442320555448532 (0.04651804025308377) | > loss_mel_ce: 3.4231958389282227 (nan) | > loss: 0.013800076209008694 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13623483147261298) | > loader_time: 0.0047 (0.0046336298093855805)  --> TIME: 2025-02-24 05:47:54 -- STEP: 6410/36938 -- GLOBAL_STEP: 191100 | > loss_text_ce: 0.04563665762543678 (0.046513971276058254) | > loss_mel_ce: 3.845299005508423 (nan) | > loss: 0.01544022187590599 (nan) | > current_lr: 5e-06 | > step_time: 0.1615 (0.13624248043423348) | > loader_time: 0.0046 (0.00463328938030416)  --> TIME: 2025-02-24 05:48:03 -- STEP: 6460/36938 -- GLOBAL_STEP: 191150 | > loss_text_ce: 0.05448091775178909 (0.04652505605553651) | > loss_mel_ce: 3.3617498874664307 (nan) | > loss: 0.013556472957134247 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.13623382480520968) | > loader_time: 0.0047 (0.004632930932768364)  --> TIME: 2025-02-24 05:48:12 -- STEP: 6510/36938 -- GLOBAL_STEP: 191200 | > loss_text_ce: 0.03956993669271469 (0.04653012747468626) | > loss_mel_ce: 4.046553134918213 (nan) | > loss: 0.016214774921536446 (nan) | > current_lr: 5e-06 | > step_time: 0.199 (0.13624732355185207) | > loader_time: 0.0048 (0.004632665448108216)  --> TIME: 2025-02-24 05:48:21 -- STEP: 6560/36938 -- GLOBAL_STEP: 191250 | > loss_text_ce: 0.0382179357111454 (0.04653276636566196) | > loss_mel_ce: 2.9994053840637207 (nan) | > loss: 0.012054061517119408 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13621461758526382) | > loader_time: 0.0047 (0.004632514133685982)  --> TIME: 2025-02-24 05:48:30 -- STEP: 6610/36938 -- GLOBAL_STEP: 191300 | > loss_text_ce: 0.05026756227016449 (0.04653467659312957) | > loss_mel_ce: 0.7814127802848816 (nan) | > loss: 0.0033003189601004124 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.1361623450595205) | > loader_time: 0.0045 (0.00463251504883644)  --> TIME: 2025-02-24 05:48:40 -- STEP: 6660/36938 -- GLOBAL_STEP: 191350 | > loss_text_ce: 0.04842115938663483 (0.04653350933267549) | > loss_mel_ce: 3.181006908416748 (nan) | > loss: 0.01281519141048193 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13615017731984475) | > loader_time: 0.0052 (0.004634110455040459)  --> TIME: 2025-02-24 05:48:49 -- STEP: 6710/36938 -- GLOBAL_STEP: 191400 | > loss_text_ce: 0.04822741448879242 (0.046517438037895605) | > loss_mel_ce: 3.158087968826294 (nan) | > loss: 0.012723473832011223 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13612818856530698) | > loader_time: 0.0045 (0.0046343252662397185)  --> TIME: 2025-02-24 05:48:58 -- STEP: 6760/36938 -- GLOBAL_STEP: 191450 | > loss_text_ce: 0.04575665295124054 (0.046519024850326725) | > loss_mel_ce: 3.720452070236206 (nan) | > loss: 0.014945273287594318 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13616265746263378) | > loader_time: 0.0045 (0.00463436901922057)  --> TIME: 2025-02-24 05:49:08 -- STEP: 6810/36938 -- GLOBAL_STEP: 191500 | > loss_text_ce: 0.040932849049568176 (0.046523814766275304) | > loss_mel_ce: 3.472071647644043 (nan) | > loss: 0.013940495438873768 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13617591413282287) | > loader_time: 0.0045 (0.004634400331326344)  --> TIME: 2025-02-24 05:49:17 -- STEP: 6860/36938 -- GLOBAL_STEP: 191550 | > loss_text_ce: 0.03686867281794548 (0.04650322969204615) | > loss_mel_ce: 2.8806629180908203 (nan) | > loss: 0.011577506549656391 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.1361714193146712) | > loader_time: 0.0046 (0.004634604648667938)  --> TIME: 2025-02-24 05:49:26 -- STEP: 6910/36938 -- GLOBAL_STEP: 191600 | > loss_text_ce: 0.05207739397883415 (0.04650257228484161) | > loss_mel_ce: 3.3692312240600586 (nan) | > loss: 0.013576622121036053 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13616835819139525) | > loader_time: 0.0045 (0.00463494799071559)  --> TIME: 2025-02-24 05:49:35 -- STEP: 6960/36938 -- GLOBAL_STEP: 191650 | > loss_text_ce: 0.04140765964984894 (0.04649591303494728) | > loss_mel_ce: 3.8526618480682373 (nan) | > loss: 0.015452657826244831 (nan) | > current_lr: 5e-06 | > step_time: 0.1651 (0.1361544464853991) | > loader_time: 0.0047 (0.00463551231499376)  --> TIME: 2025-02-24 05:49:44 -- STEP: 7010/36938 -- GLOBAL_STEP: 191700 | > loss_text_ce: 0.051494501531124115 (0.04649191377750904) | > loss_mel_ce: 3.864760398864746 (nan) | > loss: 0.015540695749223232 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.13614289209608035) | > loader_time: 0.0048 (0.004635821871682683)  --> TIME: 2025-02-24 05:49:53 -- STEP: 7060/36938 -- GLOBAL_STEP: 191750 | > loss_text_ce: 0.06533125042915344 (0.04648603388311465) | > loss_mel_ce: 1.141137719154358 (nan) | > loss: 0.004787575453519821 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13612204975852243) | > loader_time: 0.0046 (0.004636078954756091)  --> TIME: 2025-02-24 05:50:03 -- STEP: 7110/36938 -- GLOBAL_STEP: 191800 | > loss_text_ce: 0.044606178998947144 (0.046483388940807654) | > loss_mel_ce: 2.907893657684326 (nan) | > loss: 0.011716269887983799 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.1361217970921858) | > loader_time: 0.0046 (0.00463604846584143)  --> TIME: 2025-02-24 05:50:12 -- STEP: 7160/36938 -- GLOBAL_STEP: 191850 | > loss_text_ce: 0.038018848747015 (0.04647223582496483) | > loss_mel_ce: 2.257843017578125 (nan) | > loss: 0.009110563434660435 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13613677377807376) | > loader_time: 0.0046 (0.00463582760128895)  --> TIME: 2025-02-24 05:50:21 -- STEP: 7210/36938 -- GLOBAL_STEP: 191900 | > loss_text_ce: 0.06908797472715378 (0.0464697747825853) | > loss_mel_ce: 3.0135245323181152 (nan) | > loss: 0.012232590466737747 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13612368748356676) | > loader_time: 0.0045 (0.00463558552830626)  --> TIME: 2025-02-24 05:50:31 -- STEP: 7260/36938 -- GLOBAL_STEP: 191950 | > loss_text_ce: 0.039895977824926376 (0.04648088322151916) | > loss_mel_ce: 0.1498565673828125 (nan) | > loss: 0.0007529863505624235 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13613251381997576) | > loader_time: 0.0048 (0.004635863067690006)  --> TIME: 2025-02-24 05:50:40 -- STEP: 7310/36938 -- GLOBAL_STEP: 192000 | > loss_text_ce: 0.04677155986428261 (0.046491907711017244) | > loss_mel_ce: 1.3391927480697632 (nan) | > loss: 0.005499858409166336 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13614475381358068) | > loader_time: 0.0046 (0.004636379338484945)  --> TIME: 2025-02-24 05:50:50 -- STEP: 7360/36938 -- GLOBAL_STEP: 192050 | > loss_text_ce: 0.04016607627272606 (0.04647815596924732) | > loss_mel_ce: 4.0865373611450195 (nan) | > loss: 0.01637580804526806 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13616689314012956) | > loader_time: 0.0046 (0.004636687752993211)  --> TIME: 2025-02-24 05:50:59 -- STEP: 7410/36938 -- GLOBAL_STEP: 192100 | > loss_text_ce: 0.030631965026259422 (0.04647257782936138) | > loss_mel_ce: 3.969928026199341 (nan) | > loss: 0.015875238925218582 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13615257814023826) | > loader_time: 0.0045 (0.004636561951013069)  --> TIME: 2025-02-24 05:51:08 -- STEP: 7460/36938 -- GLOBAL_STEP: 192150 | > loss_text_ce: 0.04056882858276367 (0.04646108252503199) | > loss_mel_ce: 3.2957959175109863 (nan) | > loss: 0.013239542953670025 (nan) | > current_lr: 5e-06 | > step_time: 0.151 (0.13617034444220905) | > loader_time: 0.0048 (0.004636481907668127)  --> TIME: 2025-02-24 05:51:18 -- STEP: 7510/36938 -- GLOBAL_STEP: 192200 | > loss_text_ce: 0.046890877187252045 (0.04647028640324323) | > loss_mel_ce: 3.4650280475616455 (nan) | > loss: 0.013936187140643597 (nan) | > current_lr: 5e-06 | > step_time: 0.1664 (0.13620112729612277) | > loader_time: 0.0047 (0.0046367512244517885)  --> TIME: 2025-02-24 05:51:27 -- STEP: 7560/36938 -- GLOBAL_STEP: 192250 | > loss_text_ce: 0.051094841212034225 (0.046474166317946396) | > loss_mel_ce: 2.2833235263824463 (nan) | > loss: 0.009263564832508564 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13619444550029827) | > loader_time: 0.0045 (0.004637215755603932)  --> TIME: 2025-02-24 05:51:36 -- STEP: 7610/36938 -- GLOBAL_STEP: 192300 | > loss_text_ce: 0.0608377680182457 (0.0464770634933993) | > loss_mel_ce: 1.9271069765090942 (nan) | > loss: 0.007888670079410076 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13617245439787587) | > loader_time: 0.0046 (0.004637889479838909)  --> TIME: 2025-02-24 05:51:46 -- STEP: 7660/36938 -- GLOBAL_STEP: 192350 | > loss_text_ce: 0.052572742104530334 (0.04647861109978421) | > loss_mel_ce: 1.604257583618164 (nan) | > loss: 0.006574723869562149 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13619712858847488) | > loader_time: 0.0046 (0.0046383981293860056)  --> TIME: 2025-02-24 05:51:55 -- STEP: 7710/36938 -- GLOBAL_STEP: 192400 | > loss_text_ce: 0.04690990224480629 (0.04647601207913941) | > loss_mel_ce: 3.7431464195251465 (nan) | > loss: 0.015039906837046146 (nan) | > current_lr: 5e-06 | > step_time: 0.1925 (0.13619547623140815) | > loader_time: 0.0048 (0.00463829677547153)  --> TIME: 2025-02-24 05:52:04 -- STEP: 7760/36938 -- GLOBAL_STEP: 192450 | > loss_text_ce: 0.04281720519065857 (0.046479500583181) | > loss_mel_ce: 0.380859375 (nan) | > loss: 0.001681256340816617 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13618246178651605) | > loader_time: 0.0046 (0.004638891889876926)  --> TIME: 2025-02-24 05:52:13 -- STEP: 7810/36938 -- GLOBAL_STEP: 192500 | > loss_text_ce: 0.033259209245443344 (0.04647239641323401) | > loss_mel_ce: 3.3988969326019287 (nan) | > loss: 0.01361966785043478 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1362039321821265) | > loader_time: 0.0046 (0.004639813475663775)  --> TIME: 2025-02-24 05:52:23 -- STEP: 7860/36938 -- GLOBAL_STEP: 192550 | > loss_text_ce: 0.051518358290195465 (0.0464657929323083) | > loss_mel_ce: 2.543919086456299 (nan) | > loss: 0.01029935572296381 (nan) | > current_lr: 5e-06 | > step_time: 0.1734 (0.1362475538981781) | > loader_time: 0.0049 (0.0046402362466768455)  --> TIME: 2025-02-24 05:52:32 -- STEP: 7910/36938 -- GLOBAL_STEP: 192600 | > loss_text_ce: 0.03619394823908806 (0.046472237492980996) | > loss_mel_ce: 3.47841215133667 (nan) | > loss: 0.01394684985280037 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13622326793622438) | > loader_time: 0.0048 (0.004640249687561344)  --> TIME: 2025-02-24 05:52:42 -- STEP: 7960/36938 -- GLOBAL_STEP: 192650 | > loss_text_ce: 0.046934980899095535 (0.046471861085793646) | > loss_mel_ce: 3.2202391624450684 (nan) | > loss: 0.012964977882802486 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.1362748013968445) | > loader_time: 0.0046 (0.00464057823521408)  --> TIME: 2025-02-24 05:52:52 -- STEP: 8010/36938 -- GLOBAL_STEP: 192700 | > loss_text_ce: 0.04540335386991501 (0.04647312759283406) | > loss_mel_ce: 4.463517189025879 (nan) | > loss: 0.017892543226480484 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.1363036357508171) | > loader_time: 0.0047 (0.004640448852424765)  --> TIME: 2025-02-24 05:53:01 -- STEP: 8060/36938 -- GLOBAL_STEP: 192750 | > loss_text_ce: 0.04680359736084938 (0.046472939595385995) | > loss_mel_ce: 3.503491163253784 (nan) | > loss: 0.014088472351431847 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.13632165589936032) | > loader_time: 0.0047 (0.0046403623396350496)  --> TIME: 2025-02-24 05:53:10 -- STEP: 8110/36938 -- GLOBAL_STEP: 192800 | > loss_text_ce: 0.03622755408287048 (0.04646639019117661) | > loss_mel_ce: 2.3928756713867188 (nan) | > loss: 0.009639298543334007 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13632893609353863) | > loader_time: 0.0045 (0.004640182937264589)  --> TIME: 2025-02-24 05:53:20 -- STEP: 8160/36938 -- GLOBAL_STEP: 192850 | > loss_text_ce: 0.042247261852025986 (0.04646503417182021) | > loss_mel_ce: 3.7485873699188232 (nan) | > loss: 0.015042996034026146 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13634692673005322) | > loader_time: 0.0046 (0.004639992439279369)  --> TIME: 2025-02-24 05:53:29 -- STEP: 8210/36938 -- GLOBAL_STEP: 192900 | > loss_text_ce: 0.05081607401371002 (0.04646130933127981) | > loss_mel_ce: 2.6471378803253174 (nan) | > loss: 0.010706166736781597 (nan) | > current_lr: 5e-06 | > step_time: 0.178 (0.13636504474133446) | > loader_time: 0.0048 (0.004639810330975215)  --> TIME: 2025-02-24 05:53:39 -- STEP: 8260/36938 -- GLOBAL_STEP: 192950 | > loss_text_ce: 0.04511559382081032 (0.04645813340569237) | > loss_mel_ce: 3.4815666675567627 (nan) | > loss: 0.013994771987199783 (nan) | > current_lr: 5e-06 | > step_time: 0.156 (0.13636211436828177) | > loader_time: 0.0046 (0.004639774026940002)  --> TIME: 2025-02-24 05:53:48 -- STEP: 8310/36938 -- GLOBAL_STEP: 193000 | > loss_text_ce: 0.042703501880168915 (0.04644246290535382) | > loss_mel_ce: 3.536440134048462 (nan) | > loss: 0.014202951453626156 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.1363586323642846) | > loader_time: 0.0046 (0.004639770207135399)  --> TIME: 2025-02-24 05:53:57 -- STEP: 8360/36938 -- GLOBAL_STEP: 193050 | > loss_text_ce: 0.03962776064872742 (0.04644454959000163) | > loss_mel_ce: 3.385011911392212 (nan) | > loss: 0.013589841313660145 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1363221481085965) | > loader_time: 0.0048 (0.004639716667421697)  --> TIME: 2025-02-24 05:54:07 -- STEP: 8410/36938 -- GLOBAL_STEP: 193100 | > loss_text_ce: 0.0535195954144001 (0.046446521853082394) | > loss_mel_ce: 3.169297456741333 (nan) | > loss: 0.012788956984877586 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13638735994005605) | > loader_time: 0.0045 (0.004639874485528427)  --> TIME: 2025-02-24 05:54:17 -- STEP: 8460/36938 -- GLOBAL_STEP: 193150 | > loss_text_ce: 0.032328106462955475 (0.046435800242077045) | > loss_mel_ce: 3.965860366821289 (nan) | > loss: 0.015865828841924667 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.1364206365941547) | > loader_time: 0.0046 (0.0046399148079802255)  --> TIME: 2025-02-24 05:54:26 -- STEP: 8510/36938 -- GLOBAL_STEP: 193200 | > loss_text_ce: 0.045970551669597626 (0.046425791015601216) | > loss_mel_ce: 2.250720500946045 (nan) | > loss: 0.009113853797316551 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13640526049845939) | > loader_time: 0.0046 (0.004639694721522262)  --> TIME: 2025-02-24 05:54:35 -- STEP: 8560/36938 -- GLOBAL_STEP: 193250 | > loss_text_ce: 0.04463666304945946 (0.046424978470786665) | > loss_mel_ce: 3.6545844078063965 (nan) | > loss: 0.01467944961041212 (nan) | > current_lr: 5e-06 | > step_time: 0.1925 (0.13641791909097525) | > loader_time: 0.0048 (0.004639614463966588)  --> TIME: 2025-02-24 05:54:44 -- STEP: 8610/36938 -- GLOBAL_STEP: 193300 | > loss_text_ce: 0.04532596468925476 (0.04643557674287195) | > loss_mel_ce: 3.431380271911621 (nan) | > loss: 0.013796454295516014 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1364089852010748) | > loader_time: 0.0046 (0.00463957346274878)  --> TIME: 2025-02-24 05:54:54 -- STEP: 8660/36938 -- GLOBAL_STEP: 193350 | > loss_text_ce: 0.04710482060909271 (0.046430778111746306) | > loss_mel_ce: 1.3270670175552368 (nan) | > loss: 0.005453063175082207 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13640944431890803) | > loader_time: 0.0046 (0.004639474541705973)  --> TIME: 2025-02-24 05:55:03 -- STEP: 8710/36938 -- GLOBAL_STEP: 193400 | > loss_text_ce: 0.05519753694534302 (0.04642920633189887) | > loss_mel_ce: 3.439648389816284 (nan) | > loss: 0.013868437148630619 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13641977225324467) | > loader_time: 0.0045 (0.00463944195879861)  --> TIME: 2025-02-24 05:55:13 -- STEP: 8760/36938 -- GLOBAL_STEP: 193450 | > loss_text_ce: 0.050100624561309814 (0.04642901692058911) | > loss_mel_ce: 3.2780940532684326 (nan) | > loss: 0.013207122683525085 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.13643664698622557) | > loader_time: 0.0047 (0.004639246692396193)  --> TIME: 2025-02-24 05:55:22 -- STEP: 8810/36938 -- GLOBAL_STEP: 193500 | > loss_text_ce: 0.05294837802648544 (0.046424479766513115) | > loss_mel_ce: 4.046669006347656 (nan) | > loss: 0.016268324106931686 (nan) | > current_lr: 5e-06 | > step_time: 0.1493 (0.1364439368112676) | > loader_time: 0.0048 (0.00463905900074053)  --> TIME: 2025-02-24 05:55:31 -- STEP: 8860/36938 -- GLOBAL_STEP: 193550 | > loss_text_ce: 0.05234254151582718 (0.04642578117007024) | > loss_mel_ce: 3.4340476989746094 (nan) | > loss: 0.013834883458912373 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.13646532808415793) | > loader_time: 0.0046 (0.004638810593710528)  --> TIME: 2025-02-24 05:55:41 -- STEP: 8910/36938 -- GLOBAL_STEP: 193600 | > loss_text_ce: 0.041150569915771484 (0.0464261832361671) | > loss_mel_ce: 3.246242046356201 (nan) | > loss: 0.01304520945996046 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13647726414313363) | > loader_time: 0.0045 (0.0046385495617199705)  --> TIME: 2025-02-24 05:55:50 -- STEP: 8960/36938 -- GLOBAL_STEP: 193650 | > loss_text_ce: 0.03530345484614372 (0.04642117637995397) | > loss_mel_ce: 1.7951011657714844 (nan) | > loss: 0.007263510953634977 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1364971911268576) | > loader_time: 0.0045 (0.004638328616108216)  --> TIME: 2025-02-24 05:55:59 -- STEP: 9010/36938 -- GLOBAL_STEP: 193700 | > loss_text_ce: 0.050223931670188904 (0.04642545781475589) | > loss_mel_ce: 1.967974305152893 (nan) | > loss: 0.008008724078536034 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13646601984953388) | > loader_time: 0.0046 (0.004638209988618402)  --> TIME: 2025-02-24 05:56:09 -- STEP: 9060/36938 -- GLOBAL_STEP: 193750 | > loss_text_ce: 0.0492703802883625 (0.0464236549218956) | > loss_mel_ce: 3.9612293243408203 (nan) | > loss: 0.015914682298898697 (nan) | > current_lr: 5e-06 | > step_time: 0.1567 (0.13649474785027918) | > loader_time: 0.0047 (0.004637989513658269)  --> TIME: 2025-02-24 05:56:18 -- STEP: 9110/36938 -- GLOBAL_STEP: 193800 | > loss_text_ce: 0.041500743478536606 (0.04642269556031334) | > loss_mel_ce: 2.7612078189849854 (nan) | > loss: 0.011121860705316067 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13647911823410883) | > loader_time: 0.0044 (0.004637785957621169)  --> TIME: 2025-02-24 05:56:28 -- STEP: 9160/36938 -- GLOBAL_STEP: 193850 | > loss_text_ce: 0.04624427855014801 (0.04641764267372157) | > loss_mel_ce: 3.7269413471221924 (nan) | > loss: 0.014972959645092487 (nan) | > current_lr: 5e-06 | > step_time: 0.1688 (0.1365102061017632) | > loader_time: 0.0047 (0.004637783453454098)  --> TIME: 2025-02-24 05:56:37 -- STEP: 9210/36938 -- GLOBAL_STEP: 193900 | > loss_text_ce: 0.05378776416182518 (0.046414806566177906) | > loss_mel_ce: 3.680586576461792 (nan) | > loss: 0.014818946830928326 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13652214566478263) | > loader_time: 0.0046 (0.004637723015651642)  --> TIME: 2025-02-24 05:56:47 -- STEP: 9260/36938 -- GLOBAL_STEP: 193950 | > loss_text_ce: 0.04316609725356102 (0.04641290664789519) | > loss_mel_ce: 3.338414192199707 (nan) | > loss: 0.013418970629572868 (nan) | > current_lr: 5e-06 | > step_time: 0.165 (0.13655714759558904) | > loader_time: 0.0049 (0.00463776539520363)  --> TIME: 2025-02-24 05:56:56 -- STEP: 9310/36938 -- GLOBAL_STEP: 194000 | > loss_text_ce: 0.0481623150408268 (0.04640997680970474) | > loss_mel_ce: 3.3275721073150635 (nan) | > loss: 0.013395772315561771 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13653538426830494) | > loader_time: 0.0046 (0.004637680325164964)  --> TIME: 2025-02-24 05:57:05 -- STEP: 9360/36938 -- GLOBAL_STEP: 194050 | > loss_text_ce: 0.03507765755057335 (0.04640508139661204) | > loss_mel_ce: 3.6068880558013916 (nan) | > loss: 0.014452245086431503 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.13652129361772142) | > loader_time: 0.0046 (0.004637447381630928)  --> TIME: 2025-02-24 05:57:15 -- STEP: 9410/36938 -- GLOBAL_STEP: 194100 | > loss_text_ce: 0.03840077668428421 (0.046407937637939045) | > loss_mel_ce: 3.3125691413879395 (nan) | > loss: 0.013297501020133495 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13652897258620722) | > loader_time: 0.0046 (0.00463715357177443)  --> TIME: 2025-02-24 05:57:24 -- STEP: 9460/36938 -- GLOBAL_STEP: 194150 | > loss_text_ce: 0.03391903266310692 (0.046399860882670896) | > loss_mel_ce: 2.1991143226623535 (nan) | > loss: 0.008861244656145573 (nan) | > current_lr: 5e-06 | > step_time: 0.1444 (0.13651253670517038) | > loader_time: 0.0046 (0.004637037245970198)  --> TIME: 2025-02-24 05:57:33 -- STEP: 9510/36938 -- GLOBAL_STEP: 194200 | > loss_text_ce: 0.05823702737689018 (0.046404027998580635) | > loss_mel_ce: 3.50896954536438 (nan) | > loss: 0.014155582524836063 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13649287148605002) | > loader_time: 0.0045 (0.004637035385918042)  --> TIME: 2025-02-24 05:57:42 -- STEP: 9560/36938 -- GLOBAL_STEP: 194250 | > loss_text_ce: 0.037659142166376114 (0.04640063015124512) | > loss_mel_ce: 3.4935457706451416 (nan) | > loss: 0.014012718573212624 (nan) | > current_lr: 5e-06 | > step_time: 0.1579 (0.1365006588742323) | > loader_time: 0.0046 (0.00463762654919006)  --> TIME: 2025-02-24 05:57:52 -- STEP: 9610/36938 -- GLOBAL_STEP: 194300 | > loss_text_ce: 0.03620941564440727 (0.046400404446656614) | > loss_mel_ce: 3.465325355529785 (nan) | > loss: 0.01389497984200716 (nan) | > current_lr: 5e-06 | > step_time: 0.1442 (0.1365268839509628) | > loader_time: 0.0046 (0.004637894025082149)  --> TIME: 2025-02-24 05:58:01 -- STEP: 9660/36938 -- GLOBAL_STEP: 194350 | > loss_text_ce: 0.059879131615161896 (0.046402193993468885) | > loss_mel_ce: 3.4128787517547607 (nan) | > loss: 0.013780785724520683 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13652518871654876) | > loader_time: 0.0046 (0.004637707489124246)  --> TIME: 2025-02-24 05:58:10 -- STEP: 9710/36938 -- GLOBAL_STEP: 194400 | > loss_text_ce: 0.049620721489191055 (0.04639336711541974) | > loss_mel_ce: 3.428391933441162 (nan) | > loss: 0.013801638036966324 (nan) | > current_lr: 5e-06 | > step_time: 0.1824 (0.1365214497372738) | > loader_time: 0.005 (0.004637701233923745)  --> TIME: 2025-02-24 05:58:20 -- STEP: 9760/36938 -- GLOBAL_STEP: 194450 | > loss_text_ce: 0.04741011559963226 (0.04638961010391172) | > loss_mel_ce: 3.426313638687134 (nan) | > loss: 0.013784618116915226 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13651534582747776) | > loader_time: 0.0046 (0.004637885533395365)  --> TIME: 2025-02-24 05:58:29 -- STEP: 9810/36938 -- GLOBAL_STEP: 194500 | > loss_text_ce: 0.035392966121435165 (0.04639550087884241) | > loss_mel_ce: 2.0101318359375 (nan) | > loss: 0.008117162622511387 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13652371482868544) | > loader_time: 0.0046 (0.004638229233044738)  --> TIME: 2025-02-24 05:58:39 -- STEP: 9860/36938 -- GLOBAL_STEP: 194550 | > loss_text_ce: 0.051690299063920975 (0.04639261019761864) | > loss_mel_ce: 3.8137428760528564 (nan) | > loss: 0.015339021570980549 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.13654063543489925) | > loader_time: 0.0046 (0.004638553342761184)  --> TIME: 2025-02-24 05:58:48 -- STEP: 9910/36938 -- GLOBAL_STEP: 194600 | > loss_text_ce: 0.052312616258859634 (0.04639072848519638) | > loss_mel_ce: 4.229977607727051 (nan) | > loss: 0.016993217170238495 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13654948755662671) | > loader_time: 0.0045 (0.004638785382452698)  --> TIME: 2025-02-24 05:58:57 -- STEP: 9960/36938 -- GLOBAL_STEP: 194650 | > loss_text_ce: 0.028192095458507538 (0.046393822710661954) | > loss_mel_ce: 0.013239297084510326 (nan) | > loss: 0.00016441030311398208 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.13655264341687562) | > loader_time: 0.0047 (0.0046388930106258785)  --> TIME: 2025-02-24 05:59:07 -- STEP: 10010/36938 -- GLOBAL_STEP: 194700 | > loss_text_ce: 0.05055129900574684 (0.046395244265903815) | > loss_mel_ce: 3.6327619552612305 (nan) | > loss: 0.014616323634982109 (nan) | > current_lr: 5e-06 | > step_time: 0.1688 (0.13656408041269014) | > loader_time: 0.0048 (0.00463886592057082)  --> TIME: 2025-02-24 05:59:16 -- STEP: 10060/36938 -- GLOBAL_STEP: 194750 | > loss_text_ce: 0.04909152537584305 (0.046401019532892704) | > loss_mel_ce: 0.07302093505859375 (nan) | > loss: 0.00048457327648065984 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.13657290053888985) | > loader_time: 0.0046 (0.004639299987087664)  --> TIME: 2025-02-24 05:59:25 -- STEP: 10110/36938 -- GLOBAL_STEP: 194800 | > loss_text_ce: 0.05065925419330597 (0.04639228627244137) | > loss_mel_ce: 2.95196533203125 (nan) | > loss: 0.0119151771068573 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13654682346196825) | > loader_time: 0.0046 (0.004639613215694554)  --> TIME: 2025-02-24 05:59:35 -- STEP: 10160/36938 -- GLOBAL_STEP: 194850 | > loss_text_ce: 0.059206120669841766 (0.04639292108756678) | > loss_mel_ce: 3.9038052558898926 (nan) | > loss: 0.015726236626505852 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.13655262251538547) | > loader_time: 0.0048 (0.0046396686570850795)  --> TIME: 2025-02-24 05:59:44 -- STEP: 10210/36938 -- GLOBAL_STEP: 194900 | > loss_text_ce: 0.04040335863828659 (0.04639389305562517) | > loss_mel_ce: 0.07916259765625 (nan) | > loss: 0.0004744681064039469 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13654421452327054) | > loader_time: 0.0045 (0.0046396915405901035)  --> TIME: 2025-02-24 05:59:53 -- STEP: 10260/36938 -- GLOBAL_STEP: 194950 | > loss_text_ce: 0.04139988496899605 (0.04639336217256528) | > loss_mel_ce: 3.6056582927703857 (nan) | > loss: 0.014472453854978085 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13655066759730403) | > loader_time: 0.0046 (0.004639514798541739)  --> TIME: 2025-02-24 06:00:02 -- STEP: 10310/36938 -- GLOBAL_STEP: 195000 | > loss_text_ce: 0.034531913697719574 (0.04639061318238921) | > loss_mel_ce: 3.5343453884124756 (nan) | > loss: 0.014162211678922176 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.13653504966417637) | > loader_time: 0.0045 (0.004639423899460935)  --> TIME: 2025-02-24 06:00:11 -- STEP: 10360/36938 -- GLOBAL_STEP: 195050 | > loss_text_ce: 0.04039386659860611 (0.04639192476048785) | > loss_mel_ce: 3.125746488571167 (nan) | > loss: 0.012564050033688545 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13651085360169876) | > loader_time: 0.0046 (0.004639290244422809)  --> TIME: 2025-02-24 06:00:21 -- STEP: 10410/36938 -- GLOBAL_STEP: 195100 | > loss_text_ce: 0.04605758562684059 (0.046391852444447615) | > loss_mel_ce: 3.281916379928589 (nan) | > loss: 0.013206247240304947 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.13650554731187634) | > loader_time: 0.0047 (0.004639199900008745)  --> TIME: 2025-02-24 06:00:30 -- STEP: 10460/36938 -- GLOBAL_STEP: 195150 | > loss_text_ce: 0.05578560382127762 (0.046401923007899935) | > loss_mel_ce: 3.016845464706421 (nan) | > loss: 0.012192981317639351 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.1364914693522408) | > loader_time: 0.0048 (0.0046396704066551725)  --> TIME: 2025-02-24 06:00:39 -- STEP: 10510/36938 -- GLOBAL_STEP: 195200 | > loss_text_ce: 0.044679369777441025 (0.046404635608302644) | > loss_mel_ce: 2.6791436672210693 (nan) | > loss: 0.010808822698891163 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13648905189234692) | > loader_time: 0.0048 (0.004639748728468106)  --> TIME: 2025-02-24 06:00:49 -- STEP: 10560/36938 -- GLOBAL_STEP: 195250 | > loss_text_ce: 0.04347068816423416 (0.04640907937198684) | > loss_mel_ce: 3.7763941287994385 (nan) | > loss: 0.015158194117248058 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.1364954501616233) | > loader_time: 0.0046 (0.00463973444068069)  --> TIME: 2025-02-24 06:00:58 -- STEP: 10610/36938 -- GLOBAL_STEP: 195300 | > loss_text_ce: 0.033648788928985596 (0.046413177126820235) | > loss_mel_ce: 2.275546073913574 (nan) | > loss: 0.00916347187012434 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13647478952596598) | > loader_time: 0.0048 (0.004639722916677675)  --> TIME: 2025-02-24 06:01:07 -- STEP: 10660/36938 -- GLOBAL_STEP: 195350 | > loss_text_ce: 0.04935530200600624 (0.04641208112515919) | > loss_mel_ce: 2.9868273735046387 (nan) | > loss: 0.012048344127833843 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.1364708323192419) | > loader_time: 0.0045 (0.004639692691208929)  --> TIME: 2025-02-24 06:01:16 -- STEP: 10710/36938 -- GLOBAL_STEP: 195400 | > loss_text_ce: 0.04399966076016426 (0.046400270086689684) | > loss_mel_ce: 3.4738476276397705 (nan) | > loss: 0.013959712348878384 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13645458504154145) | > loader_time: 0.0046 (0.004639585679318708)  --> TIME: 2025-02-24 06:01:25 -- STEP: 10760/36938 -- GLOBAL_STEP: 195450 | > loss_text_ce: 0.04705361649394035 (0.046406548062242516) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1364447638891003) | > loader_time: 0.0045 (0.004639463996355402)  --> TIME: 2025-02-24 06:01:34 -- STEP: 10810/36938 -- GLOBAL_STEP: 195500 | > loss_text_ce: 0.056848425418138504 (0.046411558801857954) | > loss_mel_ce: 3.895958423614502 (nan) | > loss: 0.01568574272096157 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13644349998505906) | > loader_time: 0.0046 (0.004639284374755784)  --> TIME: 2025-02-24 06:01:44 -- STEP: 10860/36938 -- GLOBAL_STEP: 195550 | > loss_text_ce: 0.049595292657613754 (0.04640574328958108) | > loss_mel_ce: 2.902470588684082 (nan) | > loss: 0.011714547872543335 (nan) | > current_lr: 5e-06 | > step_time: 0.1075 (0.13646965202486128) | > loader_time: 0.0044 (0.0046393706434119724)  --> TIME: 2025-02-24 06:01:53 -- STEP: 10910/36938 -- GLOBAL_STEP: 195600 | > loss_text_ce: 0.04524601995944977 (0.04640406315803087) | > loss_mel_ce: 2.5562143325805664 (nan) | > loss: 0.010323256254196167 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13644934471980052) | > loader_time: 0.0046 (0.004639136736596192)  --> TIME: 2025-02-24 06:02:02 -- STEP: 10960/36938 -- GLOBAL_STEP: 195650 | > loss_text_ce: 0.04582596570253372 (0.046403211910943744) | > loss_mel_ce: 3.741076946258545 (nan) | > loss: 0.015027393586933613 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13644141435188128) | > loader_time: 0.0047 (0.004639111274350281)  --> TIME: 2025-02-24 06:02:11 -- STEP: 11010/36938 -- GLOBAL_STEP: 195700 | > loss_text_ce: 0.04677581787109375 (0.046405721155860516) | > loss_mel_ce: 2.891099691390991 (nan) | > loss: 0.011658236384391785 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13643633959836934) | > loader_time: 0.0045 (0.004639207807916805)  --> TIME: 2025-02-24 06:02:21 -- STEP: 11060/36938 -- GLOBAL_STEP: 195750 | > loss_text_ce: 0.049037277698516846 (0.04640487504088955) | > loss_mel_ce: 0.015869140625 (nan) | > loss: 0.0002575651742517948 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.13645661976626322) | > loader_time: 0.0048 (0.004639240070018787)  --> TIME: 2025-02-24 06:02:31 -- STEP: 11110/36938 -- GLOBAL_STEP: 195800 | > loss_text_ce: 0.04205741360783577 (0.0464045763706484) | > loss_mel_ce: 3.490269660949707 (nan) | > loss: 0.014017172157764435 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.136488782383106) | > loader_time: 0.0046 (0.004639619240129868)  --> TIME: 2025-02-24 06:02:40 -- STEP: 11160/36938 -- GLOBAL_STEP: 195850 | > loss_text_ce: 0.05228816717863083 (0.046401123636265224) | > loss_mel_ce: 6.828125 (nan) | > loss: 0.02730322815477848 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.13648782019974104) | > loader_time: 0.0046 (0.00463974638224501)  --> TIME: 2025-02-24 06:02:50 -- STEP: 11210/36938 -- GLOBAL_STEP: 195900 | > loss_text_ce: 0.039872124791145325 (0.04640050293274552) | > loss_mel_ce: 3.6845548152923584 (nan) | > loss: 0.014779472723603249 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.13649599547900867) | > loader_time: 0.0047 (0.004639861607104923)  --> TIME: 2025-02-24 06:02:59 -- STEP: 11260/36938 -- GLOBAL_STEP: 195950 | > loss_text_ce: 0.048336293548345566 (0.046397513232059796) | > loss_mel_ce: 3.293067693710327 (nan) | > loss: 0.013259540311992168 (nan) | > current_lr: 5e-06 | > step_time: 0.1818 (0.13651963296622618) | > loader_time: 0.0049 (0.004640019829277441)  --> TIME: 2025-02-24 06:03:08 -- STEP: 11310/36938 -- GLOBAL_STEP: 196000 | > loss_text_ce: 0.04808877781033516 (0.04639697343598511) | > loss_mel_ce: 3.4548122882843018 (nan) | > loss: 0.013900402002036572 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13650825417853374) | > loader_time: 0.0046 (0.004639894133853645)  --> TIME: 2025-02-24 06:03:17 -- STEP: 11360/36938 -- GLOBAL_STEP: 196050 | > loss_text_ce: 0.056353434920310974 (0.04639820535372093) | > loss_mel_ce: 0.09690163284540176 (nan) | > loss: 0.0006081550964154303 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13649098614991562) | > loader_time: 0.0045 (0.004639757015335713)  --> TIME: 2025-02-24 06:03:27 -- STEP: 11410/36938 -- GLOBAL_STEP: 196100 | > loss_text_ce: 0.04377981647849083 (0.04639460216623237) | > loss_mel_ce: 2.850412368774414 (nan) | > loss: 0.01148489024490118 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13648255554653901) | > loader_time: 0.0045 (0.004639668155406033)  --> TIME: 2025-02-24 06:03:36 -- STEP: 11460/36938 -- GLOBAL_STEP: 196150 | > loss_text_ce: 0.04125290736556053 (0.046391334501704524) | > loss_mel_ce: 3.0386412143707275 (nan) | > loss: 0.012221802957355976 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13647824373960904) | > loader_time: 0.0046 (0.004639471014132665)  --> TIME: 2025-02-24 06:03:45 -- STEP: 11510/36938 -- GLOBAL_STEP: 196200 | > loss_text_ce: 0.04922829195857048 (0.04639224474400039) | > loss_mel_ce: 3.489903211593628 (nan) | > loss: 0.014044173061847687 (nan) | > current_lr: 5e-06 | > step_time: 0.1832 (0.13650232024238368) | > loader_time: 0.0072 (0.0046398802698435215)  --> TIME: 2025-02-24 06:03:54 -- STEP: 11560/36938 -- GLOBAL_STEP: 196250 | > loss_text_ce: 0.04223083332180977 (0.04639554371876845) | > loss_mel_ce: 3.27490234375 (nan) | > loss: 0.013163227587938309 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13647656240677722) | > loader_time: 0.0045 (0.004639908413573522)  --> TIME: 2025-02-24 06:04:04 -- STEP: 11610/36938 -- GLOBAL_STEP: 196300 | > loss_text_ce: 0.03931732103228569 (0.04638832120046742) | > loss_mel_ce: 4.654186248779297 (nan) | > loss: 0.018625015392899513 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13649239663313797) | > loader_time: 0.0046 (0.004640109532871283)  --> TIME: 2025-02-24 06:04:14 -- STEP: 11660/36938 -- GLOBAL_STEP: 196350 | > loss_text_ce: 0.040446165949106216 (0.04638518384924567) | > loss_mel_ce: 3.641659736633301 (nan) | > loss: 0.014611532911658287 (nan) | > current_lr: 5e-06 | > step_time: 0.1784 (0.136506481011163) | > loader_time: 0.0048 (0.004639948109615305)  --> TIME: 2025-02-24 06:04:23 -- STEP: 11710/36938 -- GLOBAL_STEP: 196400 | > loss_text_ce: 0.05837402120232582 (0.04638622260114039) | > loss_mel_ce: 3.764134168624878 (nan) | > loss: 0.015168683603405952 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13649749081934928) | > loader_time: 0.0045 (0.0046397591940263196)  --> TIME: 2025-02-24 06:04:32 -- STEP: 11760/36938 -- GLOBAL_STEP: 196450 | > loss_text_ce: 0.04414163902401924 (0.04637747747319912) | > loss_mel_ce: 2.632891893386841 (nan) | > loss: 0.010623149573802948 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.13649653015898972) | > loader_time: 0.0046 (0.004639485478401176)  --> TIME: 2025-02-24 06:04:41 -- STEP: 11810/36938 -- GLOBAL_STEP: 196500 | > loss_text_ce: 0.05153842270374298 (0.04636881980496141) | > loss_mel_ce: 3.858105421066284 (nan) | > loss: 0.015514460392296314 (nan) | > current_lr: 5e-06 | > step_time: 0.1728 (0.13650449557792893) | > loader_time: 0.0048 (0.004639637884096404)  --> TIME: 2025-02-24 06:04:51 -- STEP: 11860/36938 -- GLOBAL_STEP: 196550 | > loss_text_ce: 0.03947075828909874 (0.046365416389033985) | > loss_mel_ce: 3.653724431991577 (nan) | > loss: 0.014655536971986294 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13650282050465612) | > loader_time: 0.0046 (0.004639621187020309)  --> TIME: 2025-02-24 06:05:00 -- STEP: 11910/36938 -- GLOBAL_STEP: 196600 | > loss_text_ce: 0.04092073440551758 (0.046360329993532) | > loss_mel_ce: 3.705773115158081 (nan) | > loss: 0.014867833815515041 (nan) | > current_lr: 5e-06 | > step_time: 0.202 (0.1365094519181973) | > loader_time: 0.0048 (0.004639628171720587)  --> TIME: 2025-02-24 06:05:10 -- STEP: 11960/36938 -- GLOBAL_STEP: 196650 | > loss_text_ce: 0.04247654229402542 (0.046344453104176654) | > loss_mel_ce: 3.1985199451446533 (nan) | > loss: 0.012861098162829876 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13651674860297192) | > loader_time: 0.0047 (0.004639666395442537)  --> TIME: 2025-02-24 06:05:19 -- STEP: 12010/36938 -- GLOBAL_STEP: 196700 | > loss_text_ce: 0.04906228929758072 (0.046339571012259925) | > loss_mel_ce: 3.464874267578125 (nan) | > loss: 0.013944193720817566 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13651681748357614) | > loader_time: 0.0065 (0.004639962392484608)  --> TIME: 2025-02-24 06:05:28 -- STEP: 12060/36938 -- GLOBAL_STEP: 196750 | > loss_text_ce: 0.03839266672730446 (0.046338272023216956) | > loss_mel_ce: 1.406788945198059 (nan) | > loss: 0.005734847858548164 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13651933434788444) | > loader_time: 0.0047 (0.00463973325283372)  --> TIME: 2025-02-24 06:05:37 -- STEP: 12110/36938 -- GLOBAL_STEP: 196800 | > loss_text_ce: 0.04232708737254143 (0.046337660006920287) | > loss_mel_ce: 2.927624225616455 (nan) | > loss: 0.011785522103309631 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13651479828170643) | > loader_time: 0.0045 (0.004639553374047114)  --> TIME: 2025-02-24 06:05:47 -- STEP: 12160/36938 -- GLOBAL_STEP: 196850 | > loss_text_ce: 0.04125319793820381 (0.04633470566692067) | > loss_mel_ce: 3.4196937084198 (nan) | > loss: 0.01373391691595316 (nan) | > current_lr: 5e-06 | > step_time: 0.1491 (0.13649990268443737) | > loader_time: 0.0046 (0.004639276254333937)  --> TIME: 2025-02-24 06:05:56 -- STEP: 12210/36938 -- GLOBAL_STEP: 196900 | > loss_text_ce: 0.05094654858112335 (0.046329688421167484) | > loss_mel_ce: 2.568014144897461 (nan) | > loss: 0.010392701253294945 (nan) | > current_lr: 5e-06 | > step_time: 0.1081 (0.13650040577709588) | > loader_time: 0.0064 (0.004639818607833315)  --> TIME: 2025-02-24 06:06:05 -- STEP: 12260/36938 -- GLOBAL_STEP: 196950 | > loss_text_ce: 0.05530480667948723 (0.04633608901140926) | > loss_mel_ce: 3.333808660507202 (nan) | > loss: 0.013448863290250301 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13650793436106387) | > loader_time: 0.0045 (0.004639642098599494)  --> TIME: 2025-02-24 06:06:14 -- STEP: 12310/36938 -- GLOBAL_STEP: 197000 | > loss_text_ce: 0.04005851596593857 (0.04633505649344152) | > loss_mel_ce: 2.721644401550293 (nan) | > loss: 0.01095914002507925 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13649099702083564) | > loader_time: 0.0045 (0.004639489451237161)  --> TIME: 2025-02-24 06:06:24 -- STEP: 12360/36938 -- GLOBAL_STEP: 197050 | > loss_text_ce: 0.04887273162603378 (0.04633565105115051) | > loss_mel_ce: 3.689607858657837 (nan) | > loss: 0.014835241250693798 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13650288514334752) | > loader_time: 0.0045 (0.00464048784913367)  --> TIME: 2025-02-24 06:06:33 -- STEP: 12410/36938 -- GLOBAL_STEP: 197100 | > loss_text_ce: 0.04025580734014511 (0.0463363509207939) | > loss_mel_ce: 3.011561393737793 (nan) | > loss: 0.012110386043787003 (nan) | > current_lr: 5e-06 | > step_time: 0.1506 (0.1365000529216235) | > loader_time: 0.0049 (0.004640380535079612)  --> TIME: 2025-02-24 06:06:42 -- STEP: 12460/36938 -- GLOBAL_STEP: 197150 | > loss_text_ce: 0.0413629524409771 (0.04634022416627656) | > loss_mel_ce: 2.1180338859558105 (nan) | > loss: 0.008569035679101944 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13649894858248499) | > loader_time: 0.0046 (0.004640299014830845)  --> TIME: 2025-02-24 06:06:52 -- STEP: 12510/36938 -- GLOBAL_STEP: 197200 | > loss_text_ce: 0.03860507905483246 (0.046342868840755086) | > loss_mel_ce: 3.3175647258758545 (nan) | > loss: 0.013318134471774101 (nan) | > current_lr: 5e-06 | > step_time: 0.1946 (0.13648341407211714) | > loader_time: 0.0049 (0.0046402230060738885)  --> TIME: 2025-02-24 06:07:01 -- STEP: 12560/36938 -- GLOBAL_STEP: 197250 | > loss_text_ce: 0.049279678612947464 (0.04633806688002165) | > loss_mel_ce: 3.434455156326294 (nan) | > loss: 0.013824345543980598 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.13648310506799385) | > loader_time: 0.0047 (0.00464015333515822)  --> TIME: 2025-02-24 06:07:10 -- STEP: 12610/36938 -- GLOBAL_STEP: 197300 | > loss_text_ce: 0.04323995113372803 (0.04633727461250282) | > loss_mel_ce: 3.0227415561676025 (nan) | > loss: 0.012166593223810196 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13647130395948237) | > loader_time: 0.0045 (0.0046402041256002205)  --> TIME: 2025-02-24 06:07:19 -- STEP: 12660/36938 -- GLOBAL_STEP: 197350 | > loss_text_ce: 0.05066314712166786 (0.04633012188357214) | > loss_mel_ce: 3.5211570262908936 (nan) | > loss: 0.014173890464007854 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13645270782247682) | > loader_time: 0.0046 (0.004639932762766698)  --> TIME: 2025-02-24 06:07:29 -- STEP: 12710/36938 -- GLOBAL_STEP: 197400 | > loss_text_ce: 0.04592983052134514 (0.04632556354136063) | > loss_mel_ce: 3.524470806121826 (nan) | > loss: 0.01416825782507658 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.13646428335566438) | > loader_time: 0.0049 (0.0046399342554183775)  --> TIME: 2025-02-24 06:07:38 -- STEP: 12760/36938 -- GLOBAL_STEP: 197450 | > loss_text_ce: 0.04497508332133293 (0.04632585316686225) | > loss_mel_ce: 3.6047933101654053 (nan) | > loss: 0.014483208768069744 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.1364517732473746) | > loader_time: 0.0048 (0.004639930635410401)  --> TIME: 2025-02-24 06:07:47 -- STEP: 12810/36938 -- GLOBAL_STEP: 197500 | > loss_text_ce: 0.04210112988948822 (0.04632519088659331) | > loss_mel_ce: 0.035502783954143524 (nan) | > loss: 0.00030795205384492874 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13643380769242675) | > loader_time: 0.0044 (0.004639873757760646)  --> TIME: 2025-02-24 06:07:56 -- STEP: 12860/36938 -- GLOBAL_STEP: 197550 | > loss_text_ce: 0.055739276111125946 (0.04631789205494454) | > loss_mel_ce: 3.419191598892212 (nan) | > loss: 0.013789408840239048 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1364324515010478) | > loader_time: 0.0048 (0.004639860519539685)  --> TIME: 2025-02-24 06:08:05 -- STEP: 12910/36938 -- GLOBAL_STEP: 197600 | > loss_text_ce: 0.04993591085076332 (0.04631878174119999) | > loss_mel_ce: 3.514849901199341 (nan) | > loss: 0.014145975932478905 (nan) | > current_lr: 5e-06 | > step_time: 0.1699 (0.13642594299050403) | > loader_time: 0.0049 (0.004639785424578924)  --> TIME: 2025-02-24 06:08:15 -- STEP: 12960/36938 -- GLOBAL_STEP: 197650 | > loss_text_ce: 0.050718754529953 (0.04631640890075667) | > loss_mel_ce: 3.732118606567383 (nan) | > loss: 0.015011260285973549 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13643364810649197) | > loader_time: 0.0047 (0.004639813708670329)  --> TIME: 2025-02-24 06:08:24 -- STEP: 13010/36938 -- GLOBAL_STEP: 197700 | > loss_text_ce: 0.055045291781425476 (0.046312496007250376) | > loss_mel_ce: 0.2079634964466095 (nan) | > loss: 0.0010436857119202614 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13643697578113442) | > loader_time: 0.0046 (0.004639778478066783)  --> TIME: 2025-02-24 06:08:33 -- STEP: 13060/36938 -- GLOBAL_STEP: 197750 | > loss_text_ce: 0.043325167149305344 (0.04630920169732395) | > loss_mel_ce: 3.4916539192199707 (nan) | > loss: 0.01402769610285759 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.1364374163322384) | > loader_time: 0.0048 (0.00463963476841153)  --> TIME: 2025-02-24 06:08:43 -- STEP: 13110/36938 -- GLOBAL_STEP: 197800 | > loss_text_ce: 0.03992026671767235 (0.04630962390179825) | > loss_mel_ce: 1.718890905380249 (nan) | > loss: 0.006979409605264664 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1364409254308326) | > loader_time: 0.0046 (0.004639527781331555)  --> TIME: 2025-02-24 06:08:52 -- STEP: 13160/36938 -- GLOBAL_STEP: 197850 | > loss_text_ce: 0.022894028574228287 (0.046302819621146185) | > loss_mel_ce: 1.5038543939590454 (nan) | > loss: 0.006058525759726763 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13643638837663535) | > loader_time: 0.0045 (0.004639565926554702)  --> TIME: 2025-02-24 06:09:01 -- STEP: 13210/36938 -- GLOBAL_STEP: 197900 | > loss_text_ce: 0.03596651181578636 (0.046299769344297484) | > loss_mel_ce: 0.02227783203125 (nan) | > loss: 0.00023112836061045527 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13645508574500884) | > loader_time: 0.0046 (0.0046398243878845915)  --> TIME: 2025-02-24 06:09:11 -- STEP: 13260/36938 -- GLOBAL_STEP: 197950 | > loss_text_ce: 0.06102510169148445 (0.0463031573461292) | > loss_mel_ce: 4.0372090339660645 (nan) | > loss: 0.016262834891676903 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13645522269008695) | > loader_time: 0.0046 (0.0046399215587483505)  --> TIME: 2025-02-24 06:09:20 -- STEP: 13310/36938 -- GLOBAL_STEP: 198000 | > loss_text_ce: 0.053414344787597656 (0.04630548528288528) | > loss_mel_ce: 3.470651388168335 (nan) | > loss: 0.013984388671815395 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.13646777177734662) | > loader_time: 0.0049 (0.004640079798508535)  --> TIME: 2025-02-24 06:09:30 -- STEP: 13360/36938 -- GLOBAL_STEP: 198050 | > loss_text_ce: 0.048209089785814285 (0.04630798820732432) | > loss_mel_ce: 3.9692304134368896 (nan) | > loss: 0.015942221507430077 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13649019109988572) | > loader_time: 0.0046 (0.0046403423933211755)  --> TIME: 2025-02-24 06:09:39 -- STEP: 13410/36938 -- GLOBAL_STEP: 198100 | > loss_text_ce: 0.04652230069041252 (0.04630570839229327) | > loss_mel_ce: 3.4456822872161865 (nan) | > loss: 0.013857956044375896 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13648324988832344) | > loader_time: 0.0047 (0.004640467552708459)  --> TIME: 2025-02-24 06:09:48 -- STEP: 13460/36938 -- GLOBAL_STEP: 198150 | > loss_text_ce: 0.04908272251486778 (0.04629778880999591) | > loss_mel_ce: 3.842857599258423 (nan) | > loss: 0.01544420886784792 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13646640775816518) | > loader_time: 0.0045 (0.004640414084226186)  --> TIME: 2025-02-24 06:09:58 -- STEP: 13510/36938 -- GLOBAL_STEP: 198200 | > loss_text_ce: 0.05431503802537918 (0.046298411034246055) | > loss_mel_ce: 3.217216730117798 (nan) | > loss: 0.012982269749045372 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.13646235668950324) | > loader_time: 0.0046 (0.004640428831099036)  --> TIME: 2025-02-24 06:10:07 -- STEP: 13560/36938 -- GLOBAL_STEP: 198250 | > loss_text_ce: 0.048728447407484055 (0.046299457864473134) | > loss_mel_ce: 0.04931640625 (nan) | > loss: 0.0003890669031534344 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13645293914814544) | > loader_time: 0.0045 (0.0046404106427082835)  --> TIME: 2025-02-24 06:10:16 -- STEP: 13610/36938 -- GLOBAL_STEP: 198300 | > loss_text_ce: 0.043393824249506 (0.046292884534390386) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13645373105477207) | > loader_time: 0.0045 (0.004640357604716987)  --> TIME: 2025-02-24 06:10:25 -- STEP: 13660/36938 -- GLOBAL_STEP: 198350 | > loss_text_ce: 0.04395781084895134 (0.046292470345521704) | > loss_mel_ce: 2.713113784790039 (nan) | > loss: 0.010940760374069214 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13642393313948198) | > loader_time: 0.0045 (0.004640244809286127)  --> TIME: 2025-02-24 06:10:34 -- STEP: 13710/36938 -- GLOBAL_STEP: 198400 | > loss_text_ce: 0.055507466197013855 (0.04629286767793967) | > loss_mel_ce: 3.546935558319092 (nan) | > loss: 0.014295409433543682 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.1364426011851365) | > loader_time: 0.0045 (0.0046401409403761225)  --> TIME: 2025-02-24 06:10:44 -- STEP: 13760/36938 -- GLOBAL_STEP: 198450 | > loss_text_ce: 0.04049498587846756 (0.046286548446690536) | > loss_mel_ce: 3.717327356338501 (nan) | > loss: 0.014911994338035583 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.1364400486093615) | > loader_time: 0.0047 (0.004639863517395279)  --> TIME: 2025-02-24 06:10:53 -- STEP: 13810/36938 -- GLOBAL_STEP: 198500 | > loss_text_ce: 0.04026009514927864 (0.04628545062928024) | > loss_mel_ce: 4.020132541656494 (nan) | > loss: 0.016112670302391052 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13644149063463876) | > loader_time: 0.0046 (0.004639628708664698)  --> TIME: 2025-02-24 06:11:03 -- STEP: 13860/36938 -- GLOBAL_STEP: 198550 | > loss_text_ce: 0.06413769721984863 (0.04628669578974459) | > loss_mel_ce: 2.1665074825286865 (nan) | > loss: 0.008851767517626286 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13646809069109136) | > loader_time: 0.0045 (0.004639508507468478)  --> TIME: 2025-02-24 06:11:12 -- STEP: 13910/36938 -- GLOBAL_STEP: 198600 | > loss_text_ce: 0.04274826869368553 (0.046287438269680396) | > loss_mel_ce: 2.8849287033081055 (nan) | > loss: 0.011617765761911869 (nan) | > current_lr: 5e-06 | > step_time: 0.1714 (0.13646394114627985) | > loader_time: 0.0048 (0.004639623183984703)  --> TIME: 2025-02-24 06:11:21 -- STEP: 13960/36938 -- GLOBAL_STEP: 198650 | > loss_text_ce: 0.049346666783094406 (0.04628623929541859) | > loss_mel_ce: 0.2009938508272171 (nan) | > loss: 0.000993414781987667 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13645464383428596) | > loader_time: 0.0046 (0.004639517731516267)  --> TIME: 2025-02-24 06:11:30 -- STEP: 14010/36938 -- GLOBAL_STEP: 198700 | > loss_text_ce: 0.051471732556819916 (0.04627834911591802) | > loss_mel_ce: 3.1573355197906494 (nan) | > loss: 0.01273336261510849 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.1364506721326403) | > loader_time: 0.0048 (0.0046394507941137746)  --> TIME: 2025-02-24 06:11:40 -- STEP: 14060/36938 -- GLOBAL_STEP: 198750 | > loss_text_ce: 0.03842701017856598 (0.0462802027611811) | > loss_mel_ce: 0.9919179081916809 (nan) | > loss: 0.004088670946657658 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13644256301828314) | > loader_time: 0.0045 (0.004639338124357953)  --> TIME: 2025-02-24 06:11:49 -- STEP: 14110/36938 -- GLOBAL_STEP: 198800 | > loss_text_ce: 0.03691427782177925 (0.04628184688687422) | > loss_mel_ce: 2.28841233253479 (nan) | > loss: 0.00922748725861311 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13645356708515777) | > loader_time: 0.0045 (0.004639471836076565)  --> TIME: 2025-02-24 06:11:58 -- STEP: 14160/36938 -- GLOBAL_STEP: 198850 | > loss_text_ce: 0.04765281453728676 (0.04627686165244242) | > loss_mel_ce: 3.8171093463897705 (nan) | > loss: 0.015336357988417149 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13646028770902047) | > loader_time: 0.0046 (0.004639353270584575)  --> TIME: 2025-02-24 06:12:08 -- STEP: 14210/36938 -- GLOBAL_STEP: 198900 | > loss_text_ce: 0.04711068049073219 (0.046274353351042824) | > loss_mel_ce: 3.1423845291137695 (nan) | > loss: 0.01265672780573368 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.1364468972167859) | > loader_time: 0.0048 (0.00463913351108623)  --> TIME: 2025-02-24 06:12:17 -- STEP: 14260/36938 -- GLOBAL_STEP: 198950 | > loss_text_ce: 0.04157756268978119 (0.0462737374335558) | > loss_mel_ce: 0.016845703125 (nan) | > loss: 0.00023183836310636252 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13645002922202545) | > loader_time: 0.0045 (0.004639004791602174)  --> TIME: 2025-02-24 06:12:26 -- STEP: 14310/36938 -- GLOBAL_STEP: 199000 | > loss_text_ce: 0.05496537685394287 (0.04627384214065206) | > loss_mel_ce: 3.3538925647735596 (nan) | > loss: 0.013527214527130127 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13645756277981358) | > loader_time: 0.0045 (0.00463890787773745)  --> TIME: 2025-02-24 06:12:36 -- STEP: 14360/36938 -- GLOBAL_STEP: 199050 | > loss_text_ce: 0.04351390153169632 (0.046268657314354956) | > loss_mel_ce: 3.509892702102661 (nan) | > loss: 0.014100820757448673 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13645770338916505) | > loader_time: 0.0046 (0.004638761331776054)  --> TIME: 2025-02-24 06:12:45 -- STEP: 14410/36938 -- GLOBAL_STEP: 199100 | > loss_text_ce: 0.033421073108911514 (0.04626483542199635) | > loss_mel_ce: 2.859776020050049 (nan) | > loss: 0.01148094143718481 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.1364631770304058) | > loader_time: 0.0046 (0.004638858473524291)  --> TIME: 2025-02-24 06:12:54 -- STEP: 14460/36938 -- GLOBAL_STEP: 199150 | > loss_text_ce: 0.04040282964706421 (0.046267528766779) | > loss_mel_ce: 1.1596828699111938 (nan) | > loss: 0.004762244876474142 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1364738538222359) | > loader_time: 0.0045 (0.0046387556018011485)  --> TIME: 2025-02-24 06:13:04 -- STEP: 14510/36938 -- GLOBAL_STEP: 199200 | > loss_text_ce: 0.038777392357587814 (0.0462643118165715) | > loss_mel_ce: 3.4204704761505127 (nan) | > loss: 0.013727175071835518 (nan) | > current_lr: 5e-06 | > step_time: 0.1499 (0.13647178050816103) | > loader_time: 0.0046 (0.004638598443392138)  --> TIME: 2025-02-24 06:13:13 -- STEP: 14560/36938 -- GLOBAL_STEP: 199250 | > loss_text_ce: 0.05417737737298012 (0.04626305251251796) | > loss_mel_ce: 3.1849710941314697 (nan) | > loss: 0.012853763997554779 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13646717430143565) | > loader_time: 0.0044 (0.0046384871497258985)  --> TIME: 2025-02-24 06:13:22 -- STEP: 14610/36938 -- GLOBAL_STEP: 199300 | > loss_text_ce: 0.042639873921871185 (0.0462628066003026) | > loss_mel_ce: 3.7259092330932617 (nan) | > loss: 0.014954561367630959 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13646326197246267) | > loader_time: 0.0046 (0.00463837082462715)  --> TIME: 2025-02-24 06:13:32 -- STEP: 14660/36938 -- GLOBAL_STEP: 199350 | > loss_text_ce: 0.03905543312430382 (0.046263437362649436) | > loss_mel_ce: 3.5320074558258057 (nan) | > loss: 0.01417088508605957 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13648029642248222) | > loader_time: 0.0048 (0.00463837954389772)  --> TIME: 2025-02-24 06:13:41 -- STEP: 14710/36938 -- GLOBAL_STEP: 199400 | > loss_text_ce: 0.04179532080888748 (0.04626392305358214) | > loss_mel_ce: 3.7017695903778076 (nan) | > loss: 0.014855416491627693 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.13645641688665205) | > loader_time: 0.0048 (0.0046381850051361185)  --> TIME: 2025-02-24 06:13:50 -- STEP: 14760/36938 -- GLOBAL_STEP: 199450 | > loss_text_ce: 0.04255576431751251 (0.04626515224616371) | > loss_mel_ce: 4.06997013092041 (nan) | > loss: 0.016319548711180687 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13643375672299057) | > loader_time: 0.0046 (0.004637983336358208)  --> TIME: 2025-02-24 06:13:59 -- STEP: 14810/36938 -- GLOBAL_STEP: 199500 | > loss_text_ce: 0.045670513063669205 (0.04626404294664892) | > loss_mel_ce: 3.60477876663208 (nan) | > loss: 0.014485910534858704 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13642264326869608) | > loader_time: 0.0045 (0.004637875724370051)  --> TIME: 2025-02-24 06:14:08 -- STEP: 14860/36938 -- GLOBAL_STEP: 199550 | > loss_text_ce: 0.045699648559093475 (0.04626390162321404) | > loss_mel_ce: 2.144136428833008 (nan) | > loss: 0.008689826354384422 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13643316157252436) | > loader_time: 0.0046 (0.004637834521995734)  --> TIME: 2025-02-24 06:14:18 -- STEP: 14910/36938 -- GLOBAL_STEP: 199600 | > loss_text_ce: 0.03968137502670288 (0.04626384395931101) | > loss_mel_ce: 3.5228679180145264 (nan) | > loss: 0.014137101359665394 (nan) | > current_lr: 5e-06 | > step_time: 0.1583 (0.13644151641249572) | > loader_time: 0.0046 (0.004637750821334251)  --> TIME: 2025-02-24 06:14:27 -- STEP: 14960/36938 -- GLOBAL_STEP: 199650 | > loss_text_ce: 0.04577481746673584 (0.046261781736844726) | > loss_mel_ce: 3.9287467002868652 (nan) | > loss: 0.015771912410855293 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13645537079655548) | > loader_time: 0.0047 (0.004638037754890111)  --> TIME: 2025-02-24 06:14:37 -- STEP: 15010/36938 -- GLOBAL_STEP: 199700 | > loss_text_ce: 0.04673457145690918 (0.04625769730428999) | > loss_mel_ce: 3.304018974304199 (nan) | > loss: 0.013296641409397125 (nan) | > current_lr: 5e-06 | > step_time: 0.1828 (0.13646614715149216) | > loader_time: 0.0048 (0.004638137887272332)  --> TIME: 2025-02-24 06:14:46 -- STEP: 15060/36938 -- GLOBAL_STEP: 199750 | > loss_text_ce: 0.038087718188762665 (0.0462602466477172) | > loss_mel_ce: 2.891324043273926 (nan) | > loss: 0.011624650098383427 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13647018131190253) | > loader_time: 0.0046 (0.004638140942470956)  --> TIME: 2025-02-24 06:14:56 -- STEP: 15110/36938 -- GLOBAL_STEP: 199800 | > loss_text_ce: 0.05243804678320885 (0.04627017775213944) | > loss_mel_ce: 3.625328540802002 (nan) | > loss: 0.014594312757253647 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1364775142316232) | > loader_time: 0.0047 (0.004638066124553158)  --> TIME: 2025-02-24 06:15:05 -- STEP: 15160/36938 -- GLOBAL_STEP: 199850 | > loss_text_ce: 0.04558621346950531 (0.046263892753762276) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.1364848906729656) | > loader_time: 0.0045 (0.004638016868707053)  --> TIME: 2025-02-24 06:15:14 -- STEP: 15210/36938 -- GLOBAL_STEP: 199900 | > loss_text_ce: 0.04860946163535118 (0.04626188839698187) | > loss_mel_ce: 3.090456962585449 (nan) | > loss: 0.012456613592803478 (nan) | > current_lr: 5e-06 | > step_time: 0.1758 (0.1364925586573785) | > loader_time: 0.0048 (0.004637955177151627)  --> TIME: 2025-02-24 06:15:24 -- STEP: 15260/36938 -- GLOBAL_STEP: 199950 | > loss_text_ce: 0.052420876920223236 (0.04625887184954323) | > loss_mel_ce: 3.800438404083252 (nan) | > loss: 0.01528912503272295 (nan) | > current_lr: 5e-06 | > step_time: 0.1882 (0.13648938729597426) | > loader_time: 0.0048 (0.004638009068225341)  --> TIME: 2025-02-24 06:15:33 -- STEP: 15310/36938 -- GLOBAL_STEP: 200000 | > loss_text_ce: 0.03709941357374191 (0.046256931088889064) | > loss_mel_ce: 3.5644307136535645 (nan) | > loss: 0.014291786588728428 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13649576496719776) | > loader_time: 0.0046 (0.004637904948926606) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_200000.pth  --> TIME: 2025-02-24 06:15:48 -- STEP: 15360/36938 -- GLOBAL_STEP: 200050 | > loss_text_ce: 0.048532817512750626 (0.04625387181295074) | > loss_mel_ce: 3.6658456325531006 (nan) | > loss: 0.014739597216248512 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.13649771935306496) | > loader_time: 0.0048 (0.004638006134579574)  --> TIME: 2025-02-24 06:15:57 -- STEP: 15410/36938 -- GLOBAL_STEP: 200100 | > loss_text_ce: 0.051742393523454666 (0.04625498711250565) | > loss_mel_ce: 3.343709707260132 (nan) | > loss: 0.013474016450345516 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13648948725131646) | > loader_time: 0.0046 (0.004638146951862309)  --> TIME: 2025-02-24 06:16:07 -- STEP: 15460/36938 -- GLOBAL_STEP: 200150 | > loss_text_ce: 0.044678740203380585 (0.046256293666297146) | > loss_mel_ce: 3.535778045654297 (nan) | > loss: 0.01420816220343113 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.136475583035764) | > loader_time: 0.0045 (0.004638121121913614)  --> TIME: 2025-02-24 06:16:16 -- STEP: 15510/36938 -- GLOBAL_STEP: 200200 | > loss_text_ce: 0.036777954548597336 (0.04625332224093308) | > loss_mel_ce: 0.006378173828125 (nan) | > loss: 0.00017125449085142463 (nan) | > current_lr: 5e-06 | > step_time: 0.1265 (0.13646415043922494) | > loader_time: 0.0046 (0.004637976940949446)  --> TIME: 2025-02-24 06:16:25 -- STEP: 15560/36938 -- GLOBAL_STEP: 200250 | > loss_text_ce: 0.05006532371044159 (0.046253670837758824) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13645692194948472) | > loader_time: 0.0047 (0.004638151261371324)  --> TIME: 2025-02-24 06:16:34 -- STEP: 15610/36938 -- GLOBAL_STEP: 200300 | > loss_text_ce: 0.053968485444784164 (0.046255786130341946) | > loss_mel_ce: 2.878051996231079 (nan) | > loss: 0.011635002680122852 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13644398271388392) | > loader_time: 0.0045 (0.0046381790312645405)  --> TIME: 2025-02-24 06:16:44 -- STEP: 15660/36938 -- GLOBAL_STEP: 200350 | > loss_text_ce: 0.05206207558512688 (0.04625239763337526) | > loss_mel_ce: 3.232915163040161 (nan) | > loss: 0.013035624288022518 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13647162844242575) | > loader_time: 0.0046 (0.004638304609905253)  --> TIME: 2025-02-24 06:16:53 -- STEP: 15710/36938 -- GLOBAL_STEP: 200400 | > loss_text_ce: 0.05730476230382919 (0.046252854779969654) | > loss_mel_ce: 2.863588809967041 (nan) | > loss: 0.011590848676860332 (nan) | > current_lr: 5e-06 | > step_time: 0.1492 (0.1364706056122565) | > loader_time: 0.0047 (0.004638665819684078)  --> TIME: 2025-02-24 06:17:02 -- STEP: 15760/36938 -- GLOBAL_STEP: 200450 | > loss_text_ce: 0.04377990588545799 (0.046253636268530025) | > loss_mel_ce: 3.509948968887329 (nan) | > loss: 0.01410209946334362 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.13646768331527714) | > loader_time: 0.0048 (0.00463867744213433)  --> TIME: 2025-02-24 06:17:12 -- STEP: 15810/36938 -- GLOBAL_STEP: 200500 | > loss_text_ce: 0.03926531597971916 (0.04626002864915679) | > loss_mel_ce: 3.7789535522460938 (nan) | > loss: 0.015151663683354855 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13648017995497766) | > loader_time: 0.0045 (0.004638775038915216)  --> TIME: 2025-02-24 06:17:21 -- STEP: 15860/36938 -- GLOBAL_STEP: 200550 | > loss_text_ce: 0.03153730183839798 (0.04625516506099848) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.1364806965801485) | > loader_time: 0.0046 (0.00463877983983197)  --> TIME: 2025-02-24 06:17:30 -- STEP: 15910/36938 -- GLOBAL_STEP: 200600 | > loss_text_ce: 0.04357718676328659 (0.04625378338846012) | > loss_mel_ce: 3.461496114730835 (nan) | > loss: 0.013909022323787212 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13648205972332245) | > loader_time: 0.0046 (0.004638831215336362)  --> TIME: 2025-02-24 06:17:40 -- STEP: 15960/36938 -- GLOBAL_STEP: 200650 | > loss_text_ce: 0.05315662920475006 (0.04625406871587124) | > loss_mel_ce: 3.6606099605560303 (nan) | > loss: 0.014737170189619064 (nan) | > current_lr: 5e-06 | > step_time: 0.1857 (0.13648965750123032) | > loader_time: 0.005 (0.004638822664294322)  --> TIME: 2025-02-24 06:17:49 -- STEP: 16010/36938 -- GLOBAL_STEP: 200700 | > loss_text_ce: 0.041809115558862686 (0.04625349093780035) | > loss_mel_ce: 4.222705364227295 (nan) | > loss: 0.01692267693579197 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.1365078117458766) | > loader_time: 0.0048 (0.004638910055309436)  --> TIME: 2025-02-24 06:17:59 -- STEP: 16060/36938 -- GLOBAL_STEP: 200750 | > loss_text_ce: 0.044294312596321106 (0.046257523322944104) | > loss_mel_ce: 3.599702835083008 (nan) | > loss: 0.01446030754595995 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13651502470892962) | > loader_time: 0.0046 (0.00463896737449047)  --> TIME: 2025-02-24 06:18:08 -- STEP: 16110/36938 -- GLOBAL_STEP: 200800 | > loss_text_ce: 0.035316918045282364 (0.0462584372394799) | > loss_mel_ce: 3.5119576454162598 (nan) | > loss: 0.01407648716121912 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.1365099438815884) | > loader_time: 0.0045 (0.004638920653169747)  --> TIME: 2025-02-24 06:18:17 -- STEP: 16160/36938 -- GLOBAL_STEP: 200850 | > loss_text_ce: 0.05512731894850731 (0.0462604102484955) | > loss_mel_ce: 3.9431240558624268 (nan) | > loss: 0.015866078436374664 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.1365051493933886) | > loader_time: 0.0047 (0.0046388888713156775)  --> TIME: 2025-02-24 06:18:27 -- STEP: 16210/36938 -- GLOBAL_STEP: 200900 | > loss_text_ce: 0.04356580227613449 (0.04625743274082279) | > loss_mel_ce: 3.2194135189056396 (nan) | > loss: 0.01294833142310381 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.1364984385545132) | > loader_time: 0.0047 (0.004639232460177286)  --> TIME: 2025-02-24 06:18:36 -- STEP: 16260/36938 -- GLOBAL_STEP: 200950 | > loss_text_ce: 0.05418722331523895 (0.04625660648012726) | > loss_mel_ce: 3.4379825592041016 (nan) | > loss: 0.013857817277312279 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.1364855177874642) | > loader_time: 0.0046 (0.004639190618695657)  --> TIME: 2025-02-24 06:18:45 -- STEP: 16310/36938 -- GLOBAL_STEP: 201000 | > loss_text_ce: 0.04745886102318764 (0.0462558524503819) | > loss_mel_ce: 3.63004994392395 (nan) | > loss: 0.014593290165066719 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13649332530654923) | > loader_time: 0.0046 (0.004639202973946404)  --> TIME: 2025-02-24 06:18:55 -- STEP: 16360/36938 -- GLOBAL_STEP: 201050 | > loss_text_ce: 0.034873031079769135 (0.04625782394692896) | > loss_mel_ce: 3.4289658069610596 (nan) | > loss: 0.013745392672717571 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13649646397907758) | > loader_time: 0.0044 (0.0046395738288067916)  --> TIME: 2025-02-24 06:19:04 -- STEP: 16410/36938 -- GLOBAL_STEP: 201100 | > loss_text_ce: 0.06685424596071243 (0.04626145028803742) | > loss_mel_ce: 2.2212626934051514 (nan) | > loss: 0.009079829789698124 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13650020344238473) | > loader_time: 0.0046 (0.004639664602889871)  --> TIME: 2025-02-24 06:19:13 -- STEP: 16460/36938 -- GLOBAL_STEP: 201150 | > loss_text_ce: 0.05124276876449585 (0.04625972637053423) | > loss_mel_ce: 3.7062623500823975 (nan) | > loss: 0.014910736121237278 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.1365071543366768) | > loader_time: 0.0049 (0.004639880842600525)  --> TIME: 2025-02-24 06:19:23 -- STEP: 16510/36938 -- GLOBAL_STEP: 201200 | > loss_text_ce: 0.061182651668787 (0.0462593620765992) | > loss_mel_ce: 2.8724234104156494 (nan) | > loss: 0.011641294695436954 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.1365135730793086) | > loader_time: 0.0046 (0.004640146488857438)  --> TIME: 2025-02-24 06:19:32 -- STEP: 16560/36938 -- GLOBAL_STEP: 201250 | > loss_text_ce: 0.03210426867008209 (0.04625488672020798) | > loss_mel_ce: 2.7371256351470947 (nan) | > loss: 0.010989008471369743 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13649530079629696) | > loader_time: 0.0047 (0.004640158305421538)  --> TIME: 2025-02-24 06:19:41 -- STEP: 16610/36938 -- GLOBAL_STEP: 201300 | > loss_text_ce: 0.04378654435276985 (0.046251131599003015) | > loss_mel_ce: 2.2940151691436768 (nan) | > loss: 0.009276991710066795 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13650007137399645) | > loader_time: 0.0044 (0.004640328833886038)  --> TIME: 2025-02-24 06:19:50 -- STEP: 16660/36938 -- GLOBAL_STEP: 201350 | > loss_text_ce: 0.044361770153045654 (0.04625348536239535) | > loss_mel_ce: 2.539705753326416 (nan) | > loss: 0.010254236869513988 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13648351166142428) | > loader_time: 0.0046 (0.004640200115194695)  --> TIME: 2025-02-24 06:20:00 -- STEP: 16710/36938 -- GLOBAL_STEP: 201400 | > loss_text_ce: 0.03217437490820885 (0.046254448969762) | > loss_mel_ce: 2.9555087089538574 (nan) | > loss: 0.01185588538646698 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.13648951540158805) | > loader_time: 0.0046 (0.00464020424585439)  --> TIME: 2025-02-24 06:20:09 -- STEP: 16760/36938 -- GLOBAL_STEP: 201450 | > loss_text_ce: 0.058394916355609894 (0.046253119719376935) | > loss_mel_ce: 4.051640033721924 (nan) | > loss: 0.016309663653373718 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13650023792125743) | > loader_time: 0.0044 (0.0046402644855025816)  --> TIME: 2025-02-24 06:20:18 -- STEP: 16810/36938 -- GLOBAL_STEP: 201500 | > loss_text_ce: 0.051357533782720566 (0.046247929043886075) | > loss_mel_ce: 3.0704920291900635 (nan) | > loss: 0.012388292700052261 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13649361930667908) | > loader_time: 0.0046 (0.004640175273629752)  --> TIME: 2025-02-24 06:20:27 -- STEP: 16860/36938 -- GLOBAL_STEP: 201550 | > loss_text_ce: 0.055753860622644424 (0.04624314768105068) | > loss_mel_ce: 3.391141891479492 (nan) | > loss: 0.01367815863341093 (nan) | > current_lr: 5e-06 | > step_time: 0.1556 (0.13647801782737015) | > loader_time: 0.0046 (0.004640083677828099)  --> TIME: 2025-02-24 06:20:37 -- STEP: 16910/36938 -- GLOBAL_STEP: 201600 | > loss_text_ce: 0.049154896289110184 (0.04623975912973692) | > loss_mel_ce: 2.4437644481658936 (nan) | > loss: 0.009892538189888 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13648924601987458) | > loader_time: 0.0048 (0.004640255363504959)  --> TIME: 2025-02-24 06:20:46 -- STEP: 16960/36938 -- GLOBAL_STEP: 201650 | > loss_text_ce: 0.04110315069556236 (0.04623678035974795) | > loss_mel_ce: 3.718315839767456 (nan) | > loss: 0.014918330125510693 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.1364858403942497) | > loader_time: 0.0048 (0.004640235695636495)  --> TIME: 2025-02-24 06:20:55 -- STEP: 17010/36938 -- GLOBAL_STEP: 201700 | > loss_text_ce: 0.03237001225352287 (0.046233442841072475) | > loss_mel_ce: 2.978637456893921 (nan) | > loss: 0.01194844301789999 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.1364739378503603) | > loader_time: 0.0046 (0.0046402089950409)  --> TIME: 2025-02-24 06:21:05 -- STEP: 17060/36938 -- GLOBAL_STEP: 201750 | > loss_text_ce: 0.03414458781480789 (0.046233741563590836) | > loss_mel_ce: 3.2463533878326416 (nan) | > loss: 0.013017849996685982 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13647155985323528) | > loader_time: 0.0045 (0.00464027826559641)  --> TIME: 2025-02-24 06:21:14 -- STEP: 17110/36938 -- GLOBAL_STEP: 201800 | > loss_text_ce: 0.04180293157696724 (0.04623064014969212) | > loss_mel_ce: 2.362548828125 (nan) | > loss: 0.009541078470647335 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13647296503926512) | > loader_time: 0.0046 (0.00464042994378255)  --> TIME: 2025-02-24 06:21:23 -- STEP: 17160/36938 -- GLOBAL_STEP: 201850 | > loss_text_ce: 0.03281165286898613 (0.0462257214872891) | > loss_mel_ce: 2.181993246078491 (nan) | > loss: 0.008788908831775188 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.1364727245760966) | > loader_time: 0.0045 (0.004640491497822291)  --> TIME: 2025-02-24 06:21:33 -- STEP: 17210/36938 -- GLOBAL_STEP: 201900 | > loss_text_ce: 0.04770180210471153 (0.04622561731091643) | > loss_mel_ce: 1.9866807460784912 (nan) | > loss: 0.008072947151958942 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13648639091012946) | > loader_time: 0.0049 (0.00464075876487541)  --> TIME: 2025-02-24 06:21:42 -- STEP: 17260/36938 -- GLOBAL_STEP: 201950 | > loss_text_ce: 0.04099090024828911 (0.04622321965923984) | > loss_mel_ce: 3.7137391567230225 (nan) | > loss: 0.01489972323179245 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.1364845070336124) | > loader_time: 0.0046 (0.004640823568142991)  --> TIME: 2025-02-24 06:21:51 -- STEP: 17310/36938 -- GLOBAL_STEP: 202000 | > loss_text_ce: 0.032588597387075424 (0.046217846157107764) | > loss_mel_ce: 3.0672221183776855 (nan) | > loss: 0.012300836853682995 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13648075870115062) | > loader_time: 0.0045 (0.004640805273645532)  --> TIME: 2025-02-24 06:22:00 -- STEP: 17360/36938 -- GLOBAL_STEP: 202050 | > loss_text_ce: 0.041635435074567795 (0.04621311427868709) | > loss_mel_ce: 3.9061367511749268 (nan) | > loss: 0.01566576398909092 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13646639205069067) | > loader_time: 0.0047 (0.004640889016713961)  --> TIME: 2025-02-24 06:22:09 -- STEP: 17410/36938 -- GLOBAL_STEP: 202100 | > loss_text_ce: 0.04193549603223801 (0.04620817581620671) | > loss_mel_ce: 2.275341749191284 (nan) | > loss: 0.009195544756948948 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.1364555755748894) | > loader_time: 0.0046 (0.004640765932358109)  --> TIME: 2025-02-24 06:22:19 -- STEP: 17460/36938 -- GLOBAL_STEP: 202150 | > loss_text_ce: 0.05206878483295441 (0.046206980127892386) | > loss_mel_ce: 3.6614389419555664 (nan) | > loss: 0.014736142009496689 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13647422543369514) | > loader_time: 0.0047 (0.004640936127779542)  --> TIME: 2025-02-24 06:22:28 -- STEP: 17510/36938 -- GLOBAL_STEP: 202200 | > loss_text_ce: 0.05356338247656822 (0.046207148199427575) | > loss_mel_ce: 2.759453296661377 (nan) | > loss: 0.011162765324115753 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13647824116668203) | > loader_time: 0.0045 (0.004640940364465103)  --> TIME: 2025-02-24 06:22:38 -- STEP: 17560/36938 -- GLOBAL_STEP: 202250 | > loss_text_ce: 0.05101826414465904 (0.04620860883210842) | > loss_mel_ce: 3.7218515872955322 (nan) | > loss: 0.014971706084907055 (nan) | > current_lr: 5e-06 | > step_time: 0.1662 (0.13647352766610515) | > loader_time: 0.0048 (0.004640929736960718)  --> TIME: 2025-02-24 06:22:47 -- STEP: 17610/36938 -- GLOBAL_STEP: 202300 | > loss_text_ce: 0.061702948063611984 (0.04621119579022097) | > loss_mel_ce: 3.2687995433807373 (nan) | > loss: 0.01321628037840128 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13647044189676266) | > loader_time: 0.0049 (0.00464089253895124)  --> TIME: 2025-02-24 06:22:56 -- STEP: 17660/36938 -- GLOBAL_STEP: 202350 | > loss_text_ce: 0.037371329963207245 (0.046213262355416986) | > loss_mel_ce: 0.8340725302696228 (nan) | > loss: 0.00345811084844172 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1364709665794202) | > loader_time: 0.0046 (0.004640946787859007)  --> TIME: 2025-02-24 06:23:05 -- STEP: 17710/36938 -- GLOBAL_STEP: 202400 | > loss_text_ce: 0.04753384366631508 (0.04620865440361072) | > loss_mel_ce: 3.902606725692749 (nan) | > loss: 0.01567516103386879 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13646957390999034) | > loader_time: 0.0047 (0.004641071798301965)  --> TIME: 2025-02-24 06:23:14 -- STEP: 17760/36938 -- GLOBAL_STEP: 202450 | > loss_text_ce: 0.036627039313316345 (0.046210375210052064) | > loss_mel_ce: 2.7680859565734863 (nan) | > loss: 0.011129814200103283 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13645485271741717) | > loader_time: 0.0046 (0.004641018915283783)  --> TIME: 2025-02-24 06:23:24 -- STEP: 17810/36938 -- GLOBAL_STEP: 202500 | > loss_text_ce: 0.0427379235625267 (0.046208578316679856) | > loss_mel_ce: 3.2740697860717773 (nan) | > loss: 0.013161935843527317 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.1364582702592814) | > loader_time: 0.0048 (0.004641006837595314)  --> TIME: 2025-02-24 06:23:33 -- STEP: 17860/36938 -- GLOBAL_STEP: 202550 | > loss_text_ce: 0.057844508439302444 (0.04620952193989241) | > loss_mel_ce: 4.131009578704834 (nan) | > loss: 0.01662243902683258 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.1364584945350856) | > loader_time: 0.0046 (0.004641110165522315)  --> TIME: 2025-02-24 06:23:43 -- STEP: 17910/36938 -- GLOBAL_STEP: 202600 | > loss_text_ce: 0.04920392856001854 (0.04620226889657341) | > loss_mel_ce: 3.592315435409546 (nan) | > loss: 0.01445047464221716 (nan) | > current_lr: 5e-06 | > step_time: 0.1446 (0.13646522086136603) | > loader_time: 0.0047 (0.004641315219523446)  --> TIME: 2025-02-24 06:23:52 -- STEP: 17960/36938 -- GLOBAL_STEP: 202650 | > loss_text_ce: 0.04016612470149994 (0.04620494881960119) | > loss_mel_ce: 3.3287341594696045 (nan) | > loss: 0.013368653133511543 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13645223103015655) | > loader_time: 0.0046 (0.00464126534079655)  --> TIME: 2025-02-24 06:24:01 -- STEP: 18010/36938 -- GLOBAL_STEP: 202700 | > loss_text_ce: 0.04354618862271309 (0.04620073831938555) | > loss_mel_ce: 3.5191376209259033 (nan) | > loss: 0.014137635007500648 (nan) | > current_lr: 5e-06 | > step_time: 0.1798 (0.13645246596815613) | > loader_time: 0.0049 (0.004641277693431281)  --> TIME: 2025-02-24 06:24:10 -- STEP: 18060/36938 -- GLOBAL_STEP: 202750 | > loss_text_ce: 0.04440689459443092 (0.04620170477253782) | > loss_mel_ce: 3.204658031463623 (nan) | > loss: 0.012893115170300007 (nan) | > current_lr: 5e-06 | > step_time: 0.1929 (0.13645776082510983) | > loader_time: 0.0049 (0.004641323086959309)  --> TIME: 2025-02-24 06:24:20 -- STEP: 18110/36938 -- GLOBAL_STEP: 202800 | > loss_text_ce: 0.04750020056962967 (0.04620667811213667) | > loss_mel_ce: 0.1597900390625 (nan) | > loss: 0.0008225803612731397 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1364546038313245) | > loader_time: 0.0046 (0.0046413661234290535)  --> TIME: 2025-02-24 06:24:29 -- STEP: 18160/36938 -- GLOBAL_STEP: 202850 | > loss_text_ce: 0.05415454879403114 (0.04621019893313552) | > loss_mel_ce: 3.656907081604004 (nan) | > loss: 0.014726436696946621 (nan) | > current_lr: 5e-06 | > step_time: 0.1786 (0.1364507581300149) | > loader_time: 0.0048 (0.004641297144511717)  --> TIME: 2025-02-24 06:24:38 -- STEP: 18210/36938 -- GLOBAL_STEP: 202900 | > loss_text_ce: 0.04181846231222153 (0.04621006872950916) | > loss_mel_ce: 2.86006498336792 (nan) | > loss: 0.011515410616993904 (nan) | > current_lr: 5e-06 | > step_time: 0.1695 (0.13645452033812247) | > loader_time: 0.0048 (0.004641231372420564)  --> TIME: 2025-02-24 06:24:48 -- STEP: 18260/36938 -- GLOBAL_STEP: 202950 | > loss_text_ce: 0.027598831802606583 (0.04621143598335934) | > loss_mel_ce: 3.2951643466949463 (nan) | > loss: 0.01318556908518076 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13647093864286333) | > loader_time: 0.0048 (0.004641290418681276)  --> TIME: 2025-02-24 06:24:57 -- STEP: 18310/36938 -- GLOBAL_STEP: 203000 | > loss_text_ce: 0.04861795902252197 (0.046210603818708326) | > loss_mel_ce: 3.9431815147399902 (nan) | > loss: 0.015840474516153336 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13647423047045426) | > loader_time: 0.0045 (0.004641317592039579)  --> TIME: 2025-02-24 06:25:07 -- STEP: 18360/36938 -- GLOBAL_STEP: 203050 | > loss_text_ce: 0.04165961220860481 (0.04620527053822015) | > loss_mel_ce: 1.9891412258148193 (nan) | > loss: 0.008058734238147736 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.1364758046623928) | > loader_time: 0.0046 (0.004641313036023139)  --> TIME: 2025-02-24 06:25:16 -- STEP: 18410/36938 -- GLOBAL_STEP: 203100 | > loss_text_ce: 0.04986361041665077 (0.046207338745165985) | > loss_mel_ce: 2.1075363159179688 (nan) | > loss: 0.00856111105531454 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13646203844784272) | > loader_time: 0.0045 (0.004641173482912506)  --> TIME: 2025-02-24 06:25:25 -- STEP: 18460/36938 -- GLOBAL_STEP: 203150 | > loss_text_ce: 0.046518564224243164 (0.046200555680119854) | > loss_mel_ce: 3.254999876022339 (nan) | > loss: 0.013101264834403992 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1364592520542289) | > loader_time: 0.0045 (0.004641113366491743)  --> TIME: 2025-02-24 06:25:34 -- STEP: 18510/36938 -- GLOBAL_STEP: 203200 | > loss_text_ce: 0.042639195919036865 (0.04619659488434035) | > loss_mel_ce: 3.6735315322875977 (nan) | > loss: 0.014746710658073425 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13647057624973513) | > loader_time: 0.0046 (0.004641142720985509)  --> TIME: 2025-02-24 06:25:44 -- STEP: 18560/36938 -- GLOBAL_STEP: 203250 | > loss_text_ce: 0.03470027446746826 (0.04618936808035175) | > loss_mel_ce: 3.4430902004241943 (nan) | > loss: 0.013800756074488163 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13647354977398077) | > loader_time: 0.0045 (0.004641108805763304)  --> TIME: 2025-02-24 06:25:53 -- STEP: 18610/36938 -- GLOBAL_STEP: 203300 | > loss_text_ce: 0.04426370561122894 (0.04619049887267456) | > loss_mel_ce: 3.1914758682250977 (nan) | > loss: 0.01284023653715849 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.13647344231541472) | > loader_time: 0.0046 (0.004641129687420462)  --> TIME: 2025-02-24 06:26:03 -- STEP: 18660/36938 -- GLOBAL_STEP: 203350 | > loss_text_ce: 0.038897257298231125 (0.046186443429144734) | > loss_mel_ce: 3.4698150157928467 (nan) | > loss: 0.013923462480306625 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.136487528033323) | > loader_time: 0.0046 (0.0046411726507982086)  --> TIME: 2025-02-24 06:26:12 -- STEP: 18710/36938 -- GLOBAL_STEP: 203400 | > loss_text_ce: 0.053734634071588516 (0.04619329979255957) | > loss_mel_ce: 2.7879931926727295 (nan) | > loss: 0.011276697739958763 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.1364981272971165) | > loader_time: 0.0048 (0.004641167917460567)  --> TIME: 2025-02-24 06:26:22 -- STEP: 18760/36938 -- GLOBAL_STEP: 203450 | > loss_text_ce: 0.05309345945715904 (0.04619423832668895) | > loss_mel_ce: 3.74483323097229 (nan) | > loss: 0.015071138739585876 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13651138896118606) | > loader_time: 0.0045 (0.004641187381642709)  --> TIME: 2025-02-24 06:26:31 -- STEP: 18810/36938 -- GLOBAL_STEP: 203500 | > loss_text_ce: 0.02682776190340519 (0.04619080666373421) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.1365207875681203) | > loader_time: 0.0046 (0.004641329424613206)  --> TIME: 2025-02-24 06:26:41 -- STEP: 18860/36938 -- GLOBAL_STEP: 203550 | > loss_text_ce: 0.04641282930970192 (0.046189808340354) | > loss_mel_ce: 3.1499717235565186 (nan) | > loss: 0.012684066779911518 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13651726368502626) | > loader_time: 0.0045 (0.004641320925375811)  --> TIME: 2025-02-24 06:26:50 -- STEP: 18910/36938 -- GLOBAL_STEP: 203600 | > loss_text_ce: 0.042877040803432465 (0.04619004354915133) | > loss_mel_ce: 3.767683744430542 (nan) | > loss: 0.015121273696422577 (nan) | > current_lr: 5e-06 | > step_time: 0.1885 (0.13651868638886405) | > loader_time: 0.0049 (0.004641300153001027)  --> TIME: 2025-02-24 06:27:00 -- STEP: 18960/36938 -- GLOBAL_STEP: 203650 | > loss_text_ce: 0.04368186742067337 (0.046190193972554236) | > loss_mel_ce: 3.9562864303588867 (nan) | > loss: 0.015872890129685402 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.1365276625890772) | > loader_time: 0.0063 (0.004641402635393259)  --> TIME: 2025-02-24 06:27:09 -- STEP: 19010/36938 -- GLOBAL_STEP: 203700 | > loss_text_ce: 0.043492391705513 (0.04619146669288406) | > loss_mel_ce: 3.7450294494628906 (nan) | > loss: 0.01503381785005331 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.1365306263908093) | > loader_time: 0.0048 (0.004641363346093828)  --> TIME: 2025-02-24 06:27:18 -- STEP: 19060/36938 -- GLOBAL_STEP: 203750 | > loss_text_ce: 0.04334787651896477 (0.046195066090602824) | > loss_mel_ce: 3.3873472213745117 (nan) | > loss: 0.013613870367407799 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13652807406588602) | > loader_time: 0.0047 (0.004641268536026301)  --> TIME: 2025-02-24 06:27:27 -- STEP: 19110/36938 -- GLOBAL_STEP: 203800 | > loss_text_ce: 0.043764784932136536 (0.04619300007081306) | > loss_mel_ce: 3.086824655532837 (nan) | > loss: 0.012422975152730942 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13652196596705574) | > loader_time: 0.0046 (0.00464120468876238)  --> TIME: 2025-02-24 06:27:37 -- STEP: 19160/36938 -- GLOBAL_STEP: 203850 | > loss_text_ce: 0.03834822401404381 (0.04619044740480176) | > loss_mel_ce: 3.168874502182007 (nan) | > loss: 0.012727075256407261 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13653123834187916) | > loader_time: 0.0047 (0.004641225367846712)  --> TIME: 2025-02-24 06:27:46 -- STEP: 19210/36938 -- GLOBAL_STEP: 203900 | > loss_text_ce: 0.04746280238032341 (0.046186471002671876) | > loss_mel_ce: 3.8250999450683594 (nan) | > loss: 0.015367313288152218 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13653455654324997) | > loader_time: 0.0046 (0.004641121244256779)  --> TIME: 2025-02-24 06:27:55 -- STEP: 19260/36938 -- GLOBAL_STEP: 203950 | > loss_text_ce: 0.038781750947237015 (0.04618548081437299) | > loss_mel_ce: 2.3186252117156982 (nan) | > loss: 0.0093547897413373 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1365229718657916) | > loader_time: 0.0044 (0.004640941295544548)  --> TIME: 2025-02-24 06:28:05 -- STEP: 19310/36938 -- GLOBAL_STEP: 204000 | > loss_text_ce: 0.03634767606854439 (0.046188608967616646) | > loss_mel_ce: 3.1112797260284424 (nan) | > loss: 0.012490585446357727 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13651812926035115) | > loader_time: 0.0046 (0.00464093581436212)  --> TIME: 2025-02-24 06:28:14 -- STEP: 19360/36938 -- GLOBAL_STEP: 204050 | > loss_text_ce: 0.04254714772105217 (0.04619306805889964) | > loss_mel_ce: 3.866938352584839 (nan) | > loss: 0.015513832680881023 (nan) | > current_lr: 5e-06 | > step_time: 0.1691 (0.13651102734498743) | > loader_time: 0.0047 (0.00464083809744227)  --> TIME: 2025-02-24 06:28:23 -- STEP: 19410/36938 -- GLOBAL_STEP: 204100 | > loss_text_ce: 0.04519842192530632 (0.04618919243328833) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.1364976738728794) | > loader_time: 0.0048 (0.0046409461622321565)  --> TIME: 2025-02-24 06:28:32 -- STEP: 19460/36938 -- GLOBAL_STEP: 204150 | > loss_text_ce: 0.03611834719777107 (0.04618119856798289) | > loss_mel_ce: 2.047764539718628 (nan) | > loss: 0.008269377052783966 (nan) | > current_lr: 5e-06 | > step_time: 0.1075 (0.1364865760028912) | > loader_time: 0.0044 (0.004641363297312489)  --> TIME: 2025-02-24 06:28:41 -- STEP: 19510/36938 -- GLOBAL_STEP: 204200 | > loss_text_ce: 0.04615896940231323 (0.046181028304303706) | > loss_mel_ce: 2.0997557640075684 (nan) | > loss: 0.008515534922480583 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13648745407513124) | > loader_time: 0.0045 (0.00464141125681459)  --> TIME: 2025-02-24 06:28:50 -- STEP: 19560/36938 -- GLOBAL_STEP: 204250 | > loss_text_ce: 0.04692560434341431 (0.046180681879432244) | > loss_mel_ce: 2.382391929626465 (nan) | > loss: 0.009640148840844631 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13648103640123363) | > loader_time: 0.0045 (0.0046413469168306)  --> TIME: 2025-02-24 06:29:00 -- STEP: 19610/36938 -- GLOBAL_STEP: 204300 | > loss_text_ce: 0.03545811027288437 (0.04617960170084262) | > loss_mel_ce: 3.9830729961395264 (nan) | > loss: 0.015946554020047188 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13649833281389118) | > loader_time: 0.0049 (0.00464138084987423)  --> TIME: 2025-02-24 06:29:09 -- STEP: 19660/36938 -- GLOBAL_STEP: 204350 | > loss_text_ce: 0.039449844509363174 (0.046179713155991) | > loss_mel_ce: 3.227621555328369 (nan) | > loss: 0.012964569963514805 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.13649421604094397) | > loader_time: 0.0049 (0.00464136771486345)  --> TIME: 2025-02-24 06:29:19 -- STEP: 19710/36938 -- GLOBAL_STEP: 204400 | > loss_text_ce: 0.03798873722553253 (0.046177300008959886) | > loss_mel_ce: 3.9029948711395264 (nan) | > loss: 0.015638824552297592 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13649069378667514) | > loader_time: 0.0045 (0.004641418067529548)  --> TIME: 2025-02-24 06:29:28 -- STEP: 19760/36938 -- GLOBAL_STEP: 204450 | > loss_text_ce: 0.0437452495098114 (0.046172948390988316) | > loss_mel_ce: 4.216938018798828 (nan) | > loss: 0.016907474026083946 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.1364885912491725) | > loader_time: 0.0049 (0.0046413135794010065)  --> TIME: 2025-02-24 06:29:37 -- STEP: 19810/36938 -- GLOBAL_STEP: 204500 | > loss_text_ce: 0.04385538399219513 (0.04617078277136217) | > loss_mel_ce: 2.881324052810669 (nan) | > loss: 0.011607855558395386 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.1364871294314305) | > loader_time: 0.0045 (0.004641350034148554)  --> TIME: 2025-02-24 06:29:46 -- STEP: 19860/36938 -- GLOBAL_STEP: 204550 | > loss_text_ce: 0.05136622488498688 (0.04617130250931696) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1364907531699866) | > loader_time: 0.0046 (0.004641502729475421)  --> TIME: 2025-02-24 06:29:56 -- STEP: 19910/36938 -- GLOBAL_STEP: 204600 | > loss_text_ce: 0.06036671996116638 (0.04617079652187256) | > loss_mel_ce: 3.39021372795105 (nan) | > loss: 0.013692780397832394 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13649622216888085) | > loader_time: 0.0047 (0.004641561541708176)  --> TIME: 2025-02-24 06:30:05 -- STEP: 19960/36938 -- GLOBAL_STEP: 204650 | > loss_text_ce: 0.04285523667931557 (0.04616678173136807) | > loss_mel_ce: 2.850491523742676 (nan) | > loss: 0.011481535620987415 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.1364953941596534) | > loader_time: 0.0046 (0.004641638287083638)  --> TIME: 2025-02-24 06:30:15 -- STEP: 20010/36938 -- GLOBAL_STEP: 204700 | > loss_text_ce: 0.03067045845091343 (0.046168517952655634) | > loss_mel_ce: 3.4509100914001465 (nan) | > loss: 0.013815796002745628 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13651106717406594) | > loader_time: 0.0045 (0.004641770387160528)  --> TIME: 2025-02-24 06:30:25 -- STEP: 20060/36938 -- GLOBAL_STEP: 204750 | > loss_text_ce: 0.048395827412605286 (0.046169673791201765) | > loss_mel_ce: 3.0782976150512695 (nan) | > loss: 0.01240751426666975 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13653665633643736) | > loader_time: 0.0047 (0.004641922307038218)  --> TIME: 2025-02-24 06:30:34 -- STEP: 20110/36938 -- GLOBAL_STEP: 204800 | > loss_text_ce: 0.03633405640721321 (0.04616943171107989) | > loss_mel_ce: 3.539039134979248 (nan) | > loss: 0.014187989756464958 (nan) | > current_lr: 5e-06 | > step_time: 0.1606 (0.13654040204299034) | > loader_time: 0.0047 (0.004641985027781753)  --> TIME: 2025-02-24 06:30:44 -- STEP: 20160/36938 -- GLOBAL_STEP: 204850 | > loss_text_ce: 0.03508530557155609 (0.04616928858385017) | > loss_mel_ce: 3.9662387371063232 (nan) | > loss: 0.01587827131152153 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.1365445928559417) | > loader_time: 0.0046 (0.004642055171822727)  --> TIME: 2025-02-24 06:30:53 -- STEP: 20210/36938 -- GLOBAL_STEP: 204900 | > loss_text_ce: 0.04431362450122833 (0.046168108268713356) | > loss_mel_ce: 3.169708251953125 (nan) | > loss: 0.012754055671393871 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.136539301950114) | > loader_time: 0.0047 (0.004642091878034974)  --> TIME: 2025-02-24 06:31:02 -- STEP: 20260/36938 -- GLOBAL_STEP: 204950 | > loss_text_ce: 0.03482014685869217 (0.0461664062706793) | > loss_mel_ce: 2.53472900390625 (nan) | > loss: 0.0101966243237257 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13653099675757502) | > loader_time: 0.0047 (0.004642074564720049)  --> TIME: 2025-02-24 06:31:11 -- STEP: 20310/36938 -- GLOBAL_STEP: 205000 | > loss_text_ce: 0.04516430199146271 (0.04616859021781414) | > loss_mel_ce: 3.7962684631347656 (nan) | > loss: 0.015243781730532646 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.1365418638636007) | > loader_time: 0.0055 (0.004642240429441052)  --> TIME: 2025-02-24 06:31:21 -- STEP: 20360/36938 -- GLOBAL_STEP: 205050 | > loss_text_ce: 0.04624949023127556 (0.046168494613544823) | > loss_mel_ce: 3.104909658432007 (nan) | > loss: 0.01250459998846054 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13655228437983916) | > loader_time: 0.0048 (0.004642327091782848)  --> TIME: 2025-02-24 06:31:30 -- STEP: 20410/36938 -- GLOBAL_STEP: 205100 | > loss_text_ce: 0.05615353584289551 (0.04616680691837322) | > loss_mel_ce: 2.8462917804718018 (nan) | > loss: 0.011517641134560108 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13655063292958) | > loader_time: 0.0046 (0.004642356475856222)  --> TIME: 2025-02-24 06:31:40 -- STEP: 20460/36938 -- GLOBAL_STEP: 205150 | > loss_text_ce: 0.04398610442876816 (0.04616851706452901) | > loss_mel_ce: 3.7059292793273926 (nan) | > loss: 0.014880617149174213 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.1365530456499153) | > loader_time: 0.0046 (0.004642405060146303)  --> TIME: 2025-02-24 06:31:49 -- STEP: 20510/36938 -- GLOBAL_STEP: 205200 | > loss_text_ce: 0.05573839694261551 (0.046164644249909785) | > loss_mel_ce: 3.8679351806640625 (nan) | > loss: 0.015570133924484253 (nan) | > current_lr: 5e-06 | > step_time: 0.1902 (0.1365610675077098) | > loader_time: 0.0051 (0.004642452372974536)  --> TIME: 2025-02-24 06:31:59 -- STEP: 20560/36938 -- GLOBAL_STEP: 205250 | > loss_text_ce: 0.04135873541235924 (0.04616488693300135) | > loss_mel_ce: 3.1617932319641113 (nan) | > loss: 0.012710921466350555 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.136570366730022) | > loader_time: 0.0045 (0.004642451238539403)  --> TIME: 2025-02-24 06:32:08 -- STEP: 20610/36938 -- GLOBAL_STEP: 205300 | > loss_text_ce: 0.06720507144927979 (0.04616548433006979) | > loss_mel_ce: 1.8607492446899414 (nan) | > loss: 0.0076506128534674644 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13656785680159148) | > loader_time: 0.0045 (0.004642339992847072)  --> TIME: 2025-02-24 06:32:17 -- STEP: 20660/36938 -- GLOBAL_STEP: 205350 | > loss_text_ce: 0.03182696923613548 (0.046164542960592775) | > loss_mel_ce: 3.5270559787750244 (nan) | > loss: 0.01412255223840475 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.1365674281397261) | > loader_time: 0.0046 (0.004642326014778132)  --> TIME: 2025-02-24 06:32:26 -- STEP: 20710/36938 -- GLOBAL_STEP: 205400 | > loss_text_ce: 0.055005062371492386 (0.04616533695593785) | > loss_mel_ce: 3.683330774307251 (nan) | > loss: 0.01483466662466526 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.1365575017012701) | > loader_time: 0.0046 (0.00464225641947789)  --> TIME: 2025-02-24 06:32:35 -- STEP: 20760/36938 -- GLOBAL_STEP: 205450 | > loss_text_ce: 0.04833356291055679 (0.04616242385288148) | > loss_mel_ce: 3.3743295669555664 (nan) | > loss: 0.013581997714936733 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13655455584470916) | > loader_time: 0.0046 (0.00464225991383221)  --> TIME: 2025-02-24 06:32:44 -- STEP: 20810/36938 -- GLOBAL_STEP: 205500 | > loss_text_ce: 0.05484055355191231 (0.04616336127817429) | > loss_mel_ce: 3.386732816696167 (nan) | > loss: 0.0136570381000638 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13654190257785964) | > loader_time: 0.0047 (0.004642221894417714)  --> TIME: 2025-02-24 06:32:53 -- STEP: 20860/36938 -- GLOBAL_STEP: 205550 | > loss_text_ce: 0.049360886216163635 (0.046165421435989985) | > loss_mel_ce: 3.5890116691589355 (nan) | > loss: 0.014437987469136715 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.1365371538145772) | > loader_time: 0.0046 (0.00464215044321513)  --> TIME: 2025-02-24 06:33:03 -- STEP: 20910/36938 -- GLOBAL_STEP: 205600 | > loss_text_ce: 0.034388650208711624 (0.04616257219630146) | > loss_mel_ce: 2.9087226390838623 (nan) | > loss: 0.011679013259708881 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.1365400801316108) | > loader_time: 0.0047 (0.004642224574077623)  --> TIME: 2025-02-24 06:33:13 -- STEP: 20960/36938 -- GLOBAL_STEP: 205650 | > loss_text_ce: 0.049993712455034256 (0.046160098863676904) | > loss_mel_ce: 3.825997829437256 (nan) | > loss: 0.01538091991096735 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.1365513840818221) | > loader_time: 0.0048 (0.00464232579897376)  --> TIME: 2025-02-24 06:33:22 -- STEP: 21010/36938 -- GLOBAL_STEP: 205700 | > loss_text_ce: 0.04605371877551079 (0.04615644374432354) | > loss_mel_ce: 2.6505682468414307 (nan) | > loss: 0.010700881481170654 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.13656561499490763) | > loader_time: 0.0046 (0.004642321948832616)  --> TIME: 2025-02-24 06:33:31 -- STEP: 21060/36938 -- GLOBAL_STEP: 205750 | > loss_text_ce: 0.03760691359639168 (0.046153206819649156) | > loss_mel_ce: 0.5574618577957153 (nan) | > loss: 0.0023613839875906706 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13656038281358293) | > loader_time: 0.0045 (0.004642199960529274)  --> TIME: 2025-02-24 06:33:41 -- STEP: 21110/36938 -- GLOBAL_STEP: 205800 | > loss_text_ce: 0.03981015458703041 (0.046148804295058306) | > loss_mel_ce: 3.0577774047851562 (nan) | > loss: 0.012292015366256237 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1365580413873472) | > loader_time: 0.0046 (0.004642162386245014)  --> TIME: 2025-02-24 06:33:50 -- STEP: 21160/36938 -- GLOBAL_STEP: 205850 | > loss_text_ce: 0.04920308664441109 (0.04614913941427933) | > loss_mel_ce: 3.9051008224487305 (nan) | > loss: 0.015691682696342468 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.1365611391053982) | > loader_time: 0.0045 (0.0046420885845032634)  --> TIME: 2025-02-24 06:34:00 -- STEP: 21210/36938 -- GLOBAL_STEP: 205900 | > loss_text_ce: 0.04526418074965477 (0.04614835326331095) | > loss_mel_ce: 3.0049495697021484 (nan) | > loss: 0.012104024179279804 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1365737779699141) | > loader_time: 0.0045 (0.004641966086859276)  --> TIME: 2025-02-24 06:34:08 -- STEP: 21260/36938 -- GLOBAL_STEP: 205950 | > loss_text_ce: 0.049379996955394745 (0.0461494379732953) | > loss_mel_ce: 3.434093952178955 (nan) | > loss: 0.0138233108446002 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13656402956396463) | > loader_time: 0.0044 (0.00464182999037617)  --> TIME: 2025-02-24 06:34:18 -- STEP: 21310/36938 -- GLOBAL_STEP: 206000 | > loss_text_ce: 0.050360649824142456 (0.04615167182940594) | > loss_mel_ce: 2.681452751159668 (nan) | > loss: 0.010840530507266521 (nan) | > current_lr: 5e-06 | > step_time: 0.1456 (0.13656711446349443) | > loader_time: 0.0045 (0.0046417465348693185)  --> TIME: 2025-02-24 06:34:27 -- STEP: 21360/36938 -- GLOBAL_STEP: 206050 | > loss_text_ce: 0.05533544719219208 (0.04614610809646221) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13655281732144842) | > loader_time: 0.0044 (0.0046416040886653655)  --> TIME: 2025-02-24 06:34:36 -- STEP: 21410/36938 -- GLOBAL_STEP: 206100 | > loss_text_ce: 0.05237586796283722 (0.04614265086881868) | > loss_mel_ce: 3.7167510986328125 (nan) | > loss: 0.014956853352487087 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13656224188878102) | > loader_time: 0.0049 (0.004641551138473834)  --> TIME: 2025-02-24 06:34:46 -- STEP: 21460/36938 -- GLOBAL_STEP: 206150 | > loss_text_ce: 0.056579478085041046 (0.04614362994882492) | > loss_mel_ce: 2.517306327819824 (nan) | > loss: 0.010213833302259445 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.1365620845184075) | > loader_time: 0.0046 (0.004641506611912003)  --> TIME: 2025-02-24 06:34:55 -- STEP: 21510/36938 -- GLOBAL_STEP: 206200 | > loss_text_ce: 0.04288702458143234 (0.04613923731864886) | > loss_mel_ce: 3.32069993019104 (nan) | > loss: 0.013347567990422249 (nan) | > current_lr: 5e-06 | > step_time: 0.1446 (0.13655795001362908) | > loader_time: 0.0047 (0.00464150637374372)  --> TIME: 2025-02-24 06:35:04 -- STEP: 21560/36938 -- GLOBAL_STEP: 206250 | > loss_text_ce: 0.04631766676902771 (0.046139605147444654) | > loss_mel_ce: 3.378300666809082 (nan) | > loss: 0.01358975563198328 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13655247330002057) | > loader_time: 0.0046 (0.004641638505436692)  --> TIME: 2025-02-24 06:35:14 -- STEP: 21610/36938 -- GLOBAL_STEP: 206300 | > loss_text_ce: 0.045406874269247055 (0.046138115298289345) | > loss_mel_ce: 3.8503246307373047 (nan) | > loss: 0.015459252521395683 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.13655892940981942) | > loader_time: 0.0046 (0.004641614044115292)  --> TIME: 2025-02-24 06:35:23 -- STEP: 21660/36938 -- GLOBAL_STEP: 206350 | > loss_text_ce: 0.037174876779317856 (0.04613790040494363) | > loss_mel_ce: 2.11212158203125 (nan) | > loss: 0.008528955280780792 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13655715198948323) | > loader_time: 0.0046 (0.004641731657149697)  --> TIME: 2025-02-24 06:35:32 -- STEP: 21710/36938 -- GLOBAL_STEP: 206400 | > loss_text_ce: 0.0431261770427227 (0.04613499900854666) | > loss_mel_ce: 3.0489516258239746 (nan) | > loss: 0.01227015070617199 (nan) | > current_lr: 5e-06 | > step_time: 0.1084 (0.13654980842883763) | > loader_time: 0.0044 (0.004641712167635302)  --> TIME: 2025-02-24 06:35:42 -- STEP: 21760/36938 -- GLOBAL_STEP: 206450 | > loss_text_ce: 0.04171181097626686 (0.04613366713991366) | > loss_mel_ce: 3.129185914993286 (nan) | > loss: 0.012582927942276001 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.136561604786445) | > loader_time: 0.0045 (0.004641810004763729)  --> TIME: 2025-02-24 06:35:51 -- STEP: 21810/36938 -- GLOBAL_STEP: 206500 | > loss_text_ce: 0.039257533848285675 (0.046133158378113735) | > loss_mel_ce: 3.569570302963257 (nan) | > loss: 0.014320746064186096 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13655616351630032) | > loader_time: 0.0046 (0.004641729284458771)  --> TIME: 2025-02-24 06:36:00 -- STEP: 21860/36938 -- GLOBAL_STEP: 206550 | > loss_text_ce: 0.041867997497320175 (0.04612888564238112) | > loss_mel_ce: 3.5876898765563965 (nan) | > loss: 0.014403007924556732 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13656024872874953) | > loader_time: 0.0048 (0.004641638779356611)  --> TIME: 2025-02-24 06:36:09 -- STEP: 21910/36938 -- GLOBAL_STEP: 206600 | > loss_text_ce: 0.054795343428850174 (0.046129507550588256) | > loss_mel_ce: 3.4762160778045654 (nan) | > loss: 0.014011950232088566 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13654916746765441) | > loader_time: 0.0046 (0.004641603922963617)  --> TIME: 2025-02-24 06:36:19 -- STEP: 21960/36938 -- GLOBAL_STEP: 206650 | > loss_text_ce: 0.045597631484270096 (0.046125825414330836) | > loss_mel_ce: 2.862717628479004 (nan) | > loss: 0.011540933512151241 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13656127401822676) | > loader_time: 0.0045 (0.0046415987236252095)  --> TIME: 2025-02-24 06:36:28 -- STEP: 22010/36938 -- GLOBAL_STEP: 206700 | > loss_text_ce: 0.04824813827872276 (0.04612322003643699) | > loss_mel_ce: 0.9329426884651184 (nan) | > loss: 0.003893614513799548 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13656566772391596) | > loader_time: 0.0046 (0.004641663318653083)  --> TIME: 2025-02-24 06:36:38 -- STEP: 22060/36938 -- GLOBAL_STEP: 206750 | > loss_text_ce: 0.046247247606515884 (0.04612055314220985) | > loss_mel_ce: 0.2476719468832016 (nan) | > loss: 0.0011663461336866021 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1365632311279732) | > loader_time: 0.0045 (0.004641740741453042)  --> TIME: 2025-02-24 06:36:47 -- STEP: 22110/36938 -- GLOBAL_STEP: 206800 | > loss_text_ce: 0.048407573252916336 (0.04612063011087163) | > loss_mel_ce: 0.8903121948242188 (nan) | > loss: 0.0037250786554068327 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.1365554556378624) | > loader_time: 0.0045 (0.0046416974186304066)  --> TIME: 2025-02-24 06:36:56 -- STEP: 22160/36938 -- GLOBAL_STEP: 206850 | > loss_text_ce: 0.03508613258600235 (0.04611539122138352) | > loss_mel_ce: 3.5817904472351074 (nan) | > loss: 0.014352685771882534 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13654844769501948) | > loader_time: 0.0045 (0.004641626673054594)  --> TIME: 2025-02-24 06:37:05 -- STEP: 22210/36938 -- GLOBAL_STEP: 206900 | > loss_text_ce: 0.0389152355492115 (0.04611632887761306) | > loss_mel_ce: 2.9152798652648926 (nan) | > loss: 0.011722996830940247 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13654169217255238) | > loader_time: 0.0046 (0.004641619505832864)  --> TIME: 2025-02-24 06:37:14 -- STEP: 22260/36938 -- GLOBAL_STEP: 206950 | > loss_text_ce: 0.053893882781267166 (0.04611424934340752) | > loss_mel_ce: 3.1685428619384766 (nan) | > loss: 0.01278744824230671 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13654671150183517) | > loader_time: 0.0046 (0.004641585272920074)  --> TIME: 2025-02-24 06:37:24 -- STEP: 22310/36938 -- GLOBAL_STEP: 207000 | > loss_text_ce: 0.039279159158468246 (0.04611406004004594) | > loss_mel_ce: 3.265699625015259 (nan) | > loss: 0.013114996254444122 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13654497234023022) | > loader_time: 0.0047 (0.004641595136846758)  --> TIME: 2025-02-24 06:37:33 -- STEP: 22360/36938 -- GLOBAL_STEP: 207050 | > loss_text_ce: 0.04804918169975281 (0.046113410797045326) | > loss_mel_ce: 3.8305675983428955 (nan) | > loss: 0.015391337685286999 (nan) | > current_lr: 5e-06 | > step_time: 0.1857 (0.13654880284836549) | > loader_time: 0.0048 (0.004641526489650171)  --> TIME: 2025-02-24 06:37:42 -- STEP: 22410/36938 -- GLOBAL_STEP: 207100 | > loss_text_ce: 0.05589943379163742 (0.0461125146694457) | > loss_mel_ce: 3.961216449737549 (nan) | > loss: 0.015940938144922256 (nan) | > current_lr: 5e-06 | > step_time: 0.169 (0.1365467029934957) | > loader_time: 0.0049 (0.0046417498471533385)  --> TIME: 2025-02-24 06:37:52 -- STEP: 22460/36938 -- GLOBAL_STEP: 207150 | > loss_text_ce: 0.03577214106917381 (0.04611449262325509) | > loss_mel_ce: 3.0035409927368164 (nan) | > loss: 0.012060767039656639 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13655010624220604) | > loader_time: 0.0046 (0.004641972815257966)  --> TIME: 2025-02-24 06:38:01 -- STEP: 22510/36938 -- GLOBAL_STEP: 207200 | > loss_text_ce: 0.05269157886505127 (0.04611352500172709) | > loss_mel_ce: 2.4820244312286377 (nan) | > loss: 0.010058398358523846 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13654393564060693) | > loader_time: 0.0046 (0.004642005403112773)  --> TIME: 2025-02-24 06:38:10 -- STEP: 22560/36938 -- GLOBAL_STEP: 207250 | > loss_text_ce: 0.04729882627725601 (0.046114420643322565) | > loss_mel_ce: 2.2594943046569824 (nan) | > loss: 0.0091539416462183 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13653948368332902) | > loader_time: 0.0048 (0.004642224037055394)  --> TIME: 2025-02-24 06:38:19 -- STEP: 22610/36938 -- GLOBAL_STEP: 207300 | > loss_text_ce: 0.06572238355875015 (0.04611380356732631) | > loss_mel_ce: 2.950629711151123 (nan) | > loss: 0.011969652026891708 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13653622169190927) | > loader_time: 0.0045 (0.0046421890921004)  --> TIME: 2025-02-24 06:38:29 -- STEP: 22660/36938 -- GLOBAL_STEP: 207350 | > loss_text_ce: 0.04320111125707626 (0.046110800476268234) | > loss_mel_ce: 3.305814743041992 (nan) | > loss: 0.013289746828377247 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13655218658253654) | > loader_time: 0.0045 (0.004642355547262703)  --> TIME: 2025-02-24 06:38:39 -- STEP: 22710/36938 -- GLOBAL_STEP: 207400 | > loss_text_ce: 0.04361100494861603 (0.046110588075074156) | > loss_mel_ce: 2.9473068714141846 (nan) | > loss: 0.01186872273683548 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13656308080907523) | > loader_time: 0.0046 (0.00464243625345231)  --> TIME: 2025-02-24 06:38:48 -- STEP: 22760/36938 -- GLOBAL_STEP: 207450 | > loss_text_ce: 0.05616512522101402 (0.04610798044006652) | > loss_mel_ce: 4.383852481842041 (nan) | > loss: 0.017619118094444275 (nan) | > current_lr: 5e-06 | > step_time: 0.1895 (0.1365673549355436) | > loader_time: 0.005 (0.004642535869182696)  --> TIME: 2025-02-24 06:38:58 -- STEP: 22810/36938 -- GLOBAL_STEP: 207500 | > loss_text_ce: 0.0469992570579052 (0.0461047188498095) | > loss_mel_ce: 2.0090575218200684 (nan) | > loss: 0.008158955723047256 (nan) | > current_lr: 5e-06 | > step_time: 0.1519 (0.13656427152634482) | > loader_time: 0.0046 (0.004642607464386853)  --> TIME: 2025-02-24 06:39:07 -- STEP: 22860/36938 -- GLOBAL_STEP: 207550 | > loss_text_ce: 0.047839369624853134 (0.04609996586038296) | > loss_mel_ce: 2.5006766319274902 (nan) | > loss: 0.010113159194588661 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13657466652929395) | > loader_time: 0.0047 (0.004642763089841606)  --> TIME: 2025-02-24 06:39:17 -- STEP: 22910/36938 -- GLOBAL_STEP: 207600 | > loss_text_ce: 0.05354682728648186 (0.046102639272628754) | > loss_mel_ce: 2.5146162509918213 (nan) | > loss: 0.010191123932600021 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.13657852043391597) | > loader_time: 0.0067 (0.004642920523218447)  --> TIME: 2025-02-24 06:39:26 -- STEP: 22960/36938 -- GLOBAL_STEP: 207650 | > loss_text_ce: 0.04644329473376274 (0.04610233000939643) | > loss_mel_ce: 3.087604284286499 (nan) | > loss: 0.012436697259545326 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.1365780642639051) | > loader_time: 0.0047 (0.004642959318094523)  --> TIME: 2025-02-24 06:39:35 -- STEP: 23010/36938 -- GLOBAL_STEP: 207700 | > loss_text_ce: 0.04596191272139549 (0.04610040906281667) | > loss_mel_ce: 3.4365808963775635 (nan) | > loss: 0.013819615356624126 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1365818089062831) | > loader_time: 0.0047 (0.0046430139218346265)  --> TIME: 2025-02-24 06:39:45 -- STEP: 23060/36938 -- GLOBAL_STEP: 207750 | > loss_text_ce: 0.04303833469748497 (0.046098736664668225) | > loss_mel_ce: 2.689864158630371 (nan) | > loss: 0.010844851844012737 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13658482831969831) | > loader_time: 0.0045 (0.0046430230037295215)  --> TIME: 2025-02-24 06:39:54 -- STEP: 23110/36938 -- GLOBAL_STEP: 207800 | > loss_text_ce: 0.04558953270316124 (0.04609842877790728) | > loss_mel_ce: 3.3853070735931396 (nan) | > loss: 0.013614669442176819 (nan) | > current_lr: 5e-06 | > step_time: 0.1815 (0.13659115501009284) | > loader_time: 0.0049 (0.004643164749838759)  --> TIME: 2025-02-24 06:40:04 -- STEP: 23160/36938 -- GLOBAL_STEP: 207850 | > loss_text_ce: 0.03699386492371559 (0.04609424620026355) | > loss_mel_ce: 3.1485254764556885 (nan) | > loss: 0.012640951201319695 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13659894397427583) | > loader_time: 0.0045 (0.0046432608561606074)  --> TIME: 2025-02-24 06:40:13 -- STEP: 23210/36938 -- GLOBAL_STEP: 207900 | > loss_text_ce: 0.0487247072160244 (0.04608936027356095) | > loss_mel_ce: 2.2620768547058105 (nan) | > loss: 0.009169847704470158 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13659753474835842) | > loader_time: 0.0048 (0.004643318232972233)  --> TIME: 2025-02-24 06:40:22 -- STEP: 23260/36938 -- GLOBAL_STEP: 207950 | > loss_text_ce: 0.03911843150854111 (0.046084914867016075) | > loss_mel_ce: 3.2858123779296875 (nan) | > loss: 0.013194170780479908 (nan) | > current_lr: 5e-06 | > step_time: 0.1619 (0.13659779390997642) | > loader_time: 0.0048 (0.004643386935531699)  --> TIME: 2025-02-24 06:40:32 -- STEP: 23310/36938 -- GLOBAL_STEP: 208000 | > loss_text_ce: 0.04810230806469917 (0.04608542743086651) | > loss_mel_ce: 1.6759788990020752 (nan) | > loss: 0.006841592490673065 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13659654600535373) | > loader_time: 0.0046 (0.004643421007399361)  --> TIME: 2025-02-24 06:40:41 -- STEP: 23360/36938 -- GLOBAL_STEP: 208050 | > loss_text_ce: 0.03350236266851425 (0.046081188926433606) | > loss_mel_ce: 3.3404793739318848 (nan) | > loss: 0.013388817198574543 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13658817947319093) | > loader_time: 0.0045 (0.004643386541164051)  --> TIME: 2025-02-24 06:40:50 -- STEP: 23410/36938 -- GLOBAL_STEP: 208100 | > loss_text_ce: 0.049292657524347305 (0.04608022297176784) | > loss_mel_ce: 3.70978045463562 (nan) | > loss: 0.01491695735603571 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13657784388646863) | > loader_time: 0.0046 (0.004643341650671521)  --> TIME: 2025-02-24 06:40:59 -- STEP: 23460/36938 -- GLOBAL_STEP: 208150 | > loss_text_ce: 0.03886988013982773 (0.04607607554402208) | > loss_mel_ce: 3.6613314151763916 (nan) | > loss: 0.014683338813483715 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13657737273186157) | > loader_time: 0.0046 (0.0046432577537250126)  --> TIME: 2025-02-24 06:41:08 -- STEP: 23510/36938 -- GLOBAL_STEP: 208200 | > loss_text_ce: 0.038416434079408646 (0.04607746217622487) | > loss_mel_ce: 2.5380382537841797 (nan) | > loss: 0.010224026627838612 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1365578468177533) | > loader_time: 0.0046 (0.004643174903233545)  --> TIME: 2025-02-24 06:41:17 -- STEP: 23560/36938 -- GLOBAL_STEP: 208250 | > loss_text_ce: 0.034421391785144806 (0.04606951196683538) | > loss_mel_ce: 3.251422166824341 (nan) | > loss: 0.013039062730967999 (nan) | > current_lr: 5e-06 | > step_time: 0.146 (0.13654330141070753) | > loader_time: 0.0047 (0.004643101471591653)  --> TIME: 2025-02-24 06:41:26 -- STEP: 23610/36938 -- GLOBAL_STEP: 208300 | > loss_text_ce: 0.030831603333353996 (0.04606466196056939) | > loss_mel_ce: 2.716437816619873 (nan) | > loss: 0.010901863686740398 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13654789538870218) | > loader_time: 0.0046 (0.004643057272223149)  --> TIME: 2025-02-24 06:41:36 -- STEP: 23660/36938 -- GLOBAL_STEP: 208350 | > loss_text_ce: 0.03846375271677971 (0.04606216715681069) | > loss_mel_ce: 2.0097291469573975 (nan) | > loss: 0.00812775082886219 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13655119751432854) | > loader_time: 0.0047 (0.004643300268877757)  --> TIME: 2025-02-24 06:41:45 -- STEP: 23710/36938 -- GLOBAL_STEP: 208400 | > loss_text_ce: 0.040012504905462265 (0.04606166972760297) | > loss_mel_ce: 3.405587911605835 (nan) | > loss: 0.013673018664121628 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13656192586071042) | > loader_time: 0.0047 (0.004643412543368777)  --> TIME: 2025-02-24 06:41:54 -- STEP: 23760/36938 -- GLOBAL_STEP: 208450 | > loss_text_ce: 0.048124998807907104 (0.04605945666945444) | > loss_mel_ce: 3.760315179824829 (nan) | > loss: 0.015112859196960926 (nan) | > current_lr: 5e-06 | > step_time: 0.1886 (0.1365466086852432) | > loader_time: 0.0048 (0.0046433635733343975)  --> TIME: 2025-02-24 06:42:04 -- STEP: 23810/36938 -- GLOBAL_STEP: 208500 | > loss_text_ce: 0.05496684089303017 (0.04605875468696733) | > loss_mel_ce: 3.553220272064209 (nan) | > loss: 0.014318203553557396 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13654765618142858) | > loader_time: 0.0045 (0.004643544205493722)  --> TIME: 2025-02-24 06:42:13 -- STEP: 23860/36938 -- GLOBAL_STEP: 208550 | > loss_text_ce: 0.043080829083919525 (0.046052030547836825) | > loss_mel_ce: 2.9182069301605225 (nan) | > loss: 0.011751142330467701 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1365443702115799) | > loader_time: 0.0045 (0.004643421091158016)  --> TIME: 2025-02-24 06:42:22 -- STEP: 23910/36938 -- GLOBAL_STEP: 208600 | > loss_text_ce: 0.05082652345299721 (0.04604809369299066) | > loss_mel_ce: 3.468879222869873 (nan) | > loss: 0.013967087492346764 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1365402529101053) | > loader_time: 0.0046 (0.004643351510198247)  --> TIME: 2025-02-24 06:42:31 -- STEP: 23960/36938 -- GLOBAL_STEP: 208650 | > loss_text_ce: 0.0422353558242321 (0.04604906315012749) | > loss_mel_ce: 3.3757410049438477 (nan) | > loss: 0.013563399203121662 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13653643371067797) | > loader_time: 0.0045 (0.004643407478953407)  --> TIME: 2025-02-24 06:42:41 -- STEP: 24010/36938 -- GLOBAL_STEP: 208700 | > loss_text_ce: 0.047320060431957245 (0.04604800119697417) | > loss_mel_ce: 3.6060869693756104 (nan) | > loss: 0.014497647993266582 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.13655243383651489) | > loader_time: 0.0047 (0.00464357103421654)  --> TIME: 2025-02-24 06:42:50 -- STEP: 24060/36938 -- GLOBAL_STEP: 208750 | > loss_text_ce: 0.05301133170723915 (0.046045097007578195) | > loss_mel_ce: 3.055980682373047 (nan) | > loss: 0.012337271124124527 (nan) | > current_lr: 5e-06 | > step_time: 0.1613 (0.13653718248963478) | > loader_time: 0.0047 (0.004643495084837059)  --> TIME: 2025-02-24 06:43:00 -- STEP: 24110/36938 -- GLOBAL_STEP: 208800 | > loss_text_ce: 0.05341155827045441 (0.04604552987169561) | > loss_mel_ce: 3.9693715572357178 (nan) | > loss: 0.015963425859808922 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.13654891284656268) | > loader_time: 0.0048 (0.0046435327067111815)  --> TIME: 2025-02-24 06:43:08 -- STEP: 24160/36938 -- GLOBAL_STEP: 208850 | > loss_text_ce: 0.053532809019088745 (0.046042210883530534) | > loss_mel_ce: 3.0026535987854004 (nan) | > loss: 0.012127724476158619 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13653403365651517) | > loader_time: 0.0045 (0.004643366421690019)  --> TIME: 2025-02-24 06:43:18 -- STEP: 24210/36938 -- GLOBAL_STEP: 208900 | > loss_text_ce: 0.05135330185294151 (0.046042639149783196) | > loss_mel_ce: 0.05615234375 (nan) | > loss: 0.0004266097384970635 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13654077437139672) | > loader_time: 0.0045 (0.0046434041046495254)  --> TIME: 2025-02-24 06:43:27 -- STEP: 24260/36938 -- GLOBAL_STEP: 208950 | > loss_text_ce: 0.04800058901309967 (0.04604191745497823) | > loss_mel_ce: 1.0375279188156128 (nan) | > loss: 0.004307652823626995 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13653555342901494) | > loader_time: 0.0047 (0.004643382027634656)  --> TIME: 2025-02-24 06:43:37 -- STEP: 24310/36938 -- GLOBAL_STEP: 209000 | > loss_text_ce: 0.05177227780222893 (0.04604440416281827) | > loss_mel_ce: 4.0528998374938965 (nan) | > loss: 0.016288381069898605 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13654307864027168) | > loader_time: 0.0046 (0.004643366406455216)  --> TIME: 2025-02-24 06:43:46 -- STEP: 24360/36938 -- GLOBAL_STEP: 209050 | > loss_text_ce: 0.04488225653767586 (0.046045902508467164) | > loss_mel_ce: 5.474999904632568 (nan) | > loss: 0.021904295310378075 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13654933885987808) | > loader_time: 0.0047 (0.004643348011085714)  --> TIME: 2025-02-24 06:43:55 -- STEP: 24410/36938 -- GLOBAL_STEP: 209100 | > loss_text_ce: 0.042706429958343506 (0.04604146662212102) | > loss_mel_ce: 3.252689838409424 (nan) | > loss: 0.013076970353722572 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13653979898231672) | > loader_time: 0.0046 (0.0046432794288062944)  --> TIME: 2025-02-24 06:44:05 -- STEP: 24460/36938 -- GLOBAL_STEP: 209150 | > loss_text_ce: 0.05143082141876221 (0.04603972937100765) | > loss_mel_ce: 3.559288501739502 (nan) | > loss: 0.014328251592814922 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13655033663268615) | > loader_time: 0.0046 (0.004643272690761278)  --> TIME: 2025-02-24 06:44:14 -- STEP: 24510/36938 -- GLOBAL_STEP: 209200 | > loss_text_ce: 0.03578382730484009 (0.04604107465542591) | > loss_mel_ce: 3.224515438079834 (nan) | > loss: 0.0129376957193017 (nan) | > current_lr: 5e-06 | > step_time: 0.1871 (0.13655967070393968) | > loader_time: 0.0048 (0.004643275094781261)  --> TIME: 2025-02-24 06:44:24 -- STEP: 24560/36938 -- GLOBAL_STEP: 209250 | > loss_text_ce: 0.04757712408900261 (0.04603801184983835) | > loss_mel_ce: 3.720391273498535 (nan) | > loss: 0.014952256344258785 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13656070624578315) | > loader_time: 0.0048 (0.004643275052406077)  --> TIME: 2025-02-24 06:44:33 -- STEP: 24610/36938 -- GLOBAL_STEP: 209300 | > loss_text_ce: 0.040784962475299835 (0.04603941829031984) | > loss_mel_ce: 3.6091084480285645 (nan) | > loss: 0.014483705163002014 (nan) | > current_lr: 5e-06 | > step_time: 0.1818 (0.13655672195431506) | > loader_time: 0.0047 (0.004643314362347264)  --> TIME: 2025-02-24 06:44:42 -- STEP: 24660/36938 -- GLOBAL_STEP: 209350 | > loss_text_ce: 0.03616384044289589 (0.046037305362291676) | > loss_mel_ce: 4.232610702514648 (nan) | > loss: 0.01693958230316639 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13654361330002993) | > loader_time: 0.0046 (0.004643272415758215)  --> TIME: 2025-02-24 06:44:51 -- STEP: 24710/36938 -- GLOBAL_STEP: 209400 | > loss_text_ce: 0.03313293308019638 (0.04603666664108365) | > loss_mel_ce: 0.01171875 (nan) | > loss: 0.00017798287444747984 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.1365425886388062) | > loader_time: 0.0045 (0.004643258292006337)  --> TIME: 2025-02-24 06:45:00 -- STEP: 24760/36938 -- GLOBAL_STEP: 209450 | > loss_text_ce: 0.06230510398745537 (0.046037007621053345) | > loss_mel_ce: 3.277914524078369 (nan) | > loss: 0.013254840858280659 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13653821927857931) | > loader_time: 0.0044 (0.004643176849901359)  --> TIME: 2025-02-24 06:45:10 -- STEP: 24810/36938 -- GLOBAL_STEP: 209500 | > loss_text_ce: 0.05197632685303688 (0.04603695774875851) | > loss_mel_ce: 3.71364164352417 (nan) | > loss: 0.014942930079996586 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.136536964318869) | > loader_time: 0.0046 (0.004643151165256067)  --> TIME: 2025-02-24 06:45:19 -- STEP: 24860/36938 -- GLOBAL_STEP: 209550 | > loss_text_ce: 0.04848184809088707 (0.0460358418498418) | > loss_mel_ce: 3.672755241394043 (nan) | > loss: 0.01476681511849165 (nan) | > current_lr: 5e-06 | > step_time: 0.176 (0.13654315950212662) | > loader_time: 0.0048 (0.004643098980020514)  --> TIME: 2025-02-24 06:45:28 -- STEP: 24910/36938 -- GLOBAL_STEP: 209600 | > loss_text_ce: 0.05379946902394295 (0.046036887305087135) | > loss_mel_ce: 2.18424391746521 (nan) | > loss: 0.008881124667823315 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13653694560182328) | > loader_time: 0.0046 (0.004643027900165914)  --> TIME: 2025-02-24 06:45:37 -- STEP: 24960/36938 -- GLOBAL_STEP: 209650 | > loss_text_ce: 0.04673364385962486 (0.04603368379000314) | > loss_mel_ce: 3.4898898601531982 (nan) | > loss: 0.014034220948815346 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.1365311141675104) | > loader_time: 0.0047 (0.004643087614423167)  --> TIME: 2025-02-24 06:45:47 -- STEP: 25010/36938 -- GLOBAL_STEP: 209700 | > loss_text_ce: 0.03706399351358414 (0.04602912885260925) | > loss_mel_ce: 1.7568882703781128 (nan) | > loss: 0.0071188583970069885 (nan) | > current_lr: 5e-06 | > step_time: 0.1784 (0.1365303171677195) | > loader_time: 0.0051 (0.004643067337426576)  --> TIME: 2025-02-24 06:45:56 -- STEP: 25060/36938 -- GLOBAL_STEP: 209750 | > loss_text_ce: 0.046835556626319885 (0.0460333929792218) | > loss_mel_ce: 2.0711669921875 (nan) | > loss: 0.008404772728681564 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13654510050132035) | > loader_time: 0.0047 (0.004643117696498714)  --> TIME: 2025-02-24 06:46:05 -- STEP: 25110/36938 -- GLOBAL_STEP: 209800 | > loss_text_ce: 0.05093865096569061 (0.046034303006476304) | > loss_mel_ce: 1.9813964366912842 (nan) | > loss: 0.008064822293817997 (nan) | > current_lr: 5e-06 | > step_time: 0.1079 (0.13653933222385992) | > loader_time: 0.0044 (0.004643025601446627)  --> TIME: 2025-02-24 06:46:15 -- STEP: 25160/36938 -- GLOBAL_STEP: 209850 | > loss_text_ce: 0.06261550635099411 (0.04603434289196327) | > loss_mel_ce: 3.2890241146087646 (nan) | > loss: 0.013300157152116299 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13654116293584265) | > loader_time: 0.0045 (0.004642975074498158)  --> TIME: 2025-02-24 06:46:24 -- STEP: 25210/36938 -- GLOBAL_STEP: 209900 | > loss_text_ce: 0.03606424853205681 (0.046035259542081854) | > loss_mel_ce: 1.4616349935531616 (nan) | > loss: 0.005943251308053732 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.13654920192334216) | > loader_time: 0.0048 (0.004642925372155842)  --> TIME: 2025-02-24 06:46:34 -- STEP: 25260/36938 -- GLOBAL_STEP: 209950 | > loss_text_ce: 0.05177144333720207 (0.0460324040858515) | > loss_mel_ce: 2.974929094314575 (nan) | > loss: 0.012010716833174229 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.1365473858062376) | > loader_time: 0.0046 (0.004642831731767544)  --> TIME: 2025-02-24 06:46:43 -- STEP: 25310/36938 -- GLOBAL_STEP: 210000 | > loss_text_ce: 0.051166024059057236 (0.04603071199996466) | > loss_mel_ce: 2.156480550765991 (nan) | > loss: 0.008760502561926842 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13655430612145905) | > loader_time: 0.0045 (0.004642803204572959) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_210000.pth  --> TIME: 2025-02-24 06:46:58 -- STEP: 25360/36938 -- GLOBAL_STEP: 210050 | > loss_text_ce: 0.04513092711567879 (0.046030531595321715) | > loss_mel_ce: 3.8712687492370605 (nan) | > loss: 0.015541269443929195 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.13654217467135135) | > loader_time: 0.0046 (0.004642764250938796)  --> TIME: 2025-02-24 06:47:07 -- STEP: 25410/36938 -- GLOBAL_STEP: 210100 | > loss_text_ce: 0.038415487855672836 (0.04602995401802531) | > loss_mel_ce: 3.15873646736145 (nan) | > loss: 0.012687111273407936 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13654237591061594) | > loader_time: 0.0046 (0.004642721903882381)  --> TIME: 2025-02-24 06:47:17 -- STEP: 25460/36938 -- GLOBAL_STEP: 210150 | > loss_text_ce: 0.05040398985147476 (0.046029055061752505) | > loss_mel_ce: 4.161871433258057 (nan) | > loss: 0.016715379431843758 (nan) | > current_lr: 5e-06 | > step_time: 0.1759 (0.13654403118102168) | > loader_time: 0.005 (0.00464271070071305)  --> TIME: 2025-02-24 06:47:26 -- STEP: 25510/36938 -- GLOBAL_STEP: 210200 | > loss_text_ce: 0.051035113632678986 (0.04602730059017849) | > loss_mel_ce: 3.035046100616455 (nan) | > loss: 0.01224635448306799 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.1365444147834502) | > loader_time: 0.0049 (0.0046426397919514355)  --> TIME: 2025-02-24 06:47:36 -- STEP: 25560/36938 -- GLOBAL_STEP: 210250 | > loss_text_ce: 0.04254075139760971 (0.0460264239408856) | > loss_mel_ce: 3.135981321334839 (nan) | > loss: 0.012613183818757534 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13654652092378536) | > loader_time: 0.0045 (0.004642577425041826)  --> TIME: 2025-02-24 06:47:45 -- STEP: 25610/36938 -- GLOBAL_STEP: 210300 | > loss_text_ce: 0.045799821615219116 (0.04602228519392221) | > loss_mel_ce: 3.7854700088500977 (nan) | > loss: 0.015203452669084072 (nan) | > current_lr: 5e-06 | > step_time: 0.177 (0.13654537929868643) | > loader_time: 0.0048 (0.004642607410733249)  --> TIME: 2025-02-24 06:47:54 -- STEP: 25660/36938 -- GLOBAL_STEP: 210350 | > loss_text_ce: 0.07575348019599915 (0.04602126359866279) | > loss_mel_ce: 3.9150588512420654 (nan) | > loss: 0.015836557373404503 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13654156822946492) | > loader_time: 0.0045 (0.004642646868342232)  --> TIME: 2025-02-24 06:48:03 -- STEP: 25710/36938 -- GLOBAL_STEP: 210400 | > loss_text_ce: 0.050167832523584366 (0.04601944646994066) | > loss_mel_ce: 2.925510883331299 (nan) | > loss: 0.011808249168097973 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.136530730929277) | > loader_time: 0.0046 (0.0046426678390087146)  --> TIME: 2025-02-24 06:48:13 -- STEP: 25760/36938 -- GLOBAL_STEP: 210450 | > loss_text_ce: 0.05305393040180206 (0.04601987020046794) | > loss_mel_ce: 3.5739684104919434 (nan) | > loss: 0.014392945915460587 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.1365325320488923) | > loader_time: 0.0047 (0.004642747027903575)  --> TIME: 2025-02-24 06:48:22 -- STEP: 25810/36938 -- GLOBAL_STEP: 210500 | > loss_text_ce: 0.04676756262779236 (0.04601724691951002) | > loss_mel_ce: 0.0461832694709301 (nan) | > loss: 0.00036885254667140543 (nan) | > current_lr: 5e-06 | > step_time: 0.178 (0.1365337180426217) | > loader_time: 0.0048 (0.004642778789752273)  --> TIME: 2025-02-24 06:48:31 -- STEP: 25860/36938 -- GLOBAL_STEP: 210550 | > loss_text_ce: 0.034966301172971725 (0.046015433586049335) | > loss_mel_ce: 4.4375 (nan) | > loss: 0.01774788461625576 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.1365428933226966) | > loader_time: 0.0045 (0.004642840263369613)  --> TIME: 2025-02-24 06:48:41 -- STEP: 25910/36938 -- GLOBAL_STEP: 210600 | > loss_text_ce: 0.05407826229929924 (0.04601463322987456) | > loss_mel_ce: 3.726301431655884 (nan) | > loss: 0.015001507475972176 (nan) | > current_lr: 5e-06 | > step_time: 0.1899 (0.13654668028407146) | > loader_time: 0.005 (0.004642872247213714)  --> TIME: 2025-02-24 06:48:50 -- STEP: 25960/36938 -- GLOBAL_STEP: 210650 | > loss_text_ce: 0.035428132861852646 (0.046014207765318144) | > loss_mel_ce: 3.293466806411743 (nan) | > loss: 0.013209900818765163 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13654792696375373) | > loader_time: 0.0045 (0.004642776035931849)  --> TIME: 2025-02-24 06:48:59 -- STEP: 26010/36938 -- GLOBAL_STEP: 210700 | > loss_text_ce: 0.040217626839876175 (0.046010838248092205) | > loss_mel_ce: 2.3973515033721924 (nan) | > loss: 0.009672894142568111 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13653972423887906) | > loader_time: 0.0045 (0.004642752820095217)  --> TIME: 2025-02-24 06:49:08 -- STEP: 26060/36938 -- GLOBAL_STEP: 210750 | > loss_text_ce: 0.03401953727006912 (0.04600829077860987) | > loss_mel_ce: 2.5999948978424072 (nan) | > loss: 0.01045243814587593 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13652360780552045) | > loader_time: 0.0044 (0.0046425826716038465)  --> TIME: 2025-02-24 06:49:17 -- STEP: 26110/36938 -- GLOBAL_STEP: 210800 | > loss_text_ce: 0.045668769627809525 (0.046007549929693504) | > loss_mel_ce: 3.2232348918914795 (nan) | > loss: 0.012971840798854828 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.1365231408966957) | > loader_time: 0.0046 (0.004642632664140241)  --> TIME: 2025-02-24 06:49:27 -- STEP: 26160/36938 -- GLOBAL_STEP: 210850 | > loss_text_ce: 0.0398360937833786 (0.046006553512193536) | > loss_mel_ce: 3.1555004119873047 (nan) | > loss: 0.012679907493293285 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13652112190512025) | > loader_time: 0.0046 (0.004642619342978909)  --> TIME: 2025-02-24 06:49:36 -- STEP: 26210/36938 -- GLOBAL_STEP: 210900 | > loss_text_ce: 0.04314819350838661 (0.04600482735571814) | > loss_mel_ce: 3.586170196533203 (nan) | > loss: 0.014402057975530624 (nan) | > current_lr: 5e-06 | > step_time: 0.1695 (0.13651407532726578) | > loader_time: 0.0049 (0.004642570469043734)  --> TIME: 2025-02-24 06:49:45 -- STEP: 26260/36938 -- GLOBAL_STEP: 210950 | > loss_text_ce: 0.06086111441254616 (0.04600017193193451) | > loss_mel_ce: 3.6468489170074463 (nan) | > loss: 0.014713135547935963 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13651523970504503) | > loader_time: 0.0045 (0.004642591037009351)  --> TIME: 2025-02-24 06:49:54 -- STEP: 26310/36938 -- GLOBAL_STEP: 211000 | > loss_text_ce: 0.047985561192035675 (0.04599599052841484) | > loss_mel_ce: 3.08439302444458 (nan) | > loss: 0.012430074624717236 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13651246245212825) | > loader_time: 0.0046 (0.004642592804914883)  --> TIME: 2025-02-24 06:50:04 -- STEP: 26360/36938 -- GLOBAL_STEP: 211050 | > loss_text_ce: 0.050459444522857666 (0.04599140641998168) | > loss_mel_ce: 3.0761218070983887 (nan) | > loss: 0.012407069094479084 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13650472207568728) | > loader_time: 0.0046 (0.0046425193955214215)  --> TIME: 2025-02-24 06:50:12 -- STEP: 26410/36938 -- GLOBAL_STEP: 211100 | > loss_text_ce: 0.047479838132858276 (0.04598841496317644) | > loss_mel_ce: 3.5772199630737305 (nan) | > loss: 0.014383730478584766 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.13649410497325298) | > loader_time: 0.0048 (0.004642498028273809)  --> TIME: 2025-02-24 06:50:22 -- STEP: 26460/36938 -- GLOBAL_STEP: 211150 | > loss_text_ce: 0.042362041771411896 (0.04598830851961371) | > loss_mel_ce: 2.9748497009277344 (nan) | > loss: 0.01197306253015995 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13649255697779528) | > loader_time: 0.0046 (0.004642493699593489)  --> TIME: 2025-02-24 06:50:31 -- STEP: 26510/36938 -- GLOBAL_STEP: 211200 | > loss_text_ce: 0.05015433579683304 (0.04598855666478293) | > loss_mel_ce: 3.8685805797576904 (nan) | > loss: 0.015550537034869194 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13648367860730096) | > loader_time: 0.0047 (0.004642475267393701)  --> TIME: 2025-02-24 06:50:40 -- STEP: 26560/36938 -- GLOBAL_STEP: 211250 | > loss_text_ce: 0.05233434960246086 (0.04598889714419013) | > loss_mel_ce: 3.4191741943359375 (nan) | > loss: 0.013775828294456005 (nan) | > current_lr: 5e-06 | > step_time: 0.1504 (0.13648867922195534) | > loader_time: 0.0047 (0.004642452667635577)  --> TIME: 2025-02-24 06:50:50 -- STEP: 26610/36938 -- GLOBAL_STEP: 211300 | > loss_text_ce: 0.047605808824300766 (0.045989988968357466) | > loss_mel_ce: 3.3439652919769287 (nan) | > loss: 0.013458616100251675 (nan) | > current_lr: 5e-06 | > step_time: 0.1794 (0.1364942201408671) | > loader_time: 0.0047 (0.004642424266260363)  --> TIME: 2025-02-24 06:50:59 -- STEP: 26660/36938 -- GLOBAL_STEP: 211350 | > loss_text_ce: 0.04811793193221092 (0.04599075860774271) | > loss_mel_ce: 3.666621208190918 (nan) | > loss: 0.014741028659045696 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13649274468511433) | > loader_time: 0.0045 (0.00464240461028969)  --> TIME: 2025-02-24 06:51:08 -- STEP: 26710/36938 -- GLOBAL_STEP: 211400 | > loss_text_ce: 0.05126908794045448 (0.045987522832384206) | > loss_mel_ce: 3.5527257919311523 (nan) | > loss: 0.01430156733840704 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.13649397707753186) | > loader_time: 0.0047 (0.004642395659003582)  --> TIME: 2025-02-24 06:51:18 -- STEP: 26760/36938 -- GLOBAL_STEP: 211450 | > loss_text_ce: 0.03255778178572655 (0.04598778309645108) | > loss_mel_ce: 3.310096263885498 (nan) | > loss: 0.01326450053602457 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1364939765188911) | > loader_time: 0.0045 (0.004642340082045949)  --> TIME: 2025-02-24 06:51:28 -- STEP: 26810/36938 -- GLOBAL_STEP: 211500 | > loss_text_ce: 0.049182992428541183 (0.04598554960076522) | > loss_mel_ce: 2.8197708129882812 (nan) | > loss: 0.011384737677872181 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13651235428198008) | > loader_time: 0.0047 (0.0046427461381546905)  --> TIME: 2025-02-24 06:51:37 -- STEP: 26860/36938 -- GLOBAL_STEP: 211550 | > loss_text_ce: 0.04750221222639084 (0.04598481808248191) | > loss_mel_ce: 2.9148240089416504 (nan) | > loss: 0.011755263432860374 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13651607655271786) | > loader_time: 0.0045 (0.00464276015891656)  --> TIME: 2025-02-24 06:51:46 -- STEP: 26910/36938 -- GLOBAL_STEP: 211600 | > loss_text_ce: 0.043437495827674866 (0.04598201413848984) | > loss_mel_ce: 2.4368324279785156 (nan) | > loss: 0.009842341765761375 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13651843148064677) | > loader_time: 0.0047 (0.004642798846559126)  --> TIME: 2025-02-24 06:51:56 -- STEP: 26960/36938 -- GLOBAL_STEP: 211650 | > loss_text_ce: 0.04685983434319496 (0.04597828618830464) | > loss_mel_ce: 3.392620086669922 (nan) | > loss: 0.01364873070269823 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13651957555234692) | > loader_time: 0.0046 (0.004642792032804936)  --> TIME: 2025-02-24 06:52:05 -- STEP: 27010/36938 -- GLOBAL_STEP: 211700 | > loss_text_ce: 0.04657838121056557 (0.0459759817321458) | > loss_mel_ce: 3.868441581726074 (nan) | > loss: 0.015535794198513031 (nan) | > current_lr: 5e-06 | > step_time: 0.1923 (0.136522877697414) | > loader_time: 0.0049 (0.0046429106236739615)  --> TIME: 2025-02-24 06:52:14 -- STEP: 27060/36938 -- GLOBAL_STEP: 211750 | > loss_text_ce: 0.05328119173645973 (0.04597891943480395) | > loss_mel_ce: 3.4234485626220703 (nan) | > loss: 0.01379654835909605 (nan) | > current_lr: 5e-06 | > step_time: 0.1353 (0.13651860211394312) | > loader_time: 0.0047 (0.0046429560612680325)  --> TIME: 2025-02-24 06:52:24 -- STEP: 27110/36938 -- GLOBAL_STEP: 211800 | > loss_text_ce: 0.04201577603816986 (0.0459788644007175) | > loss_mel_ce: 3.6003665924072266 (nan) | > loss: 0.01445389911532402 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.13652326275351745) | > loader_time: 0.0047 (0.0046431818821033215)  --> TIME: 2025-02-24 06:52:33 -- STEP: 27160/36938 -- GLOBAL_STEP: 211850 | > loss_text_ce: 0.04492107778787613 (0.04597754992578274) | > loss_mel_ce: 3.6974334716796875 (nan) | > loss: 0.014850614592432976 (nan) | > current_lr: 5e-06 | > step_time: 0.1785 (0.13652030884429622) | > loader_time: 0.0049 (0.004643542785883266)  --> TIME: 2025-02-24 06:52:42 -- STEP: 27210/36938 -- GLOBAL_STEP: 211900 | > loss_text_ce: 0.05449879541993141 (0.0459786537855671) | > loss_mel_ce: 3.7800045013427734 (nan) | > loss: 0.015216284431517124 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.13652499658752223) | > loader_time: 0.0048 (0.004643701718423929)  --> TIME: 2025-02-24 06:52:52 -- STEP: 27260/36938 -- GLOBAL_STEP: 211950 | > loss_text_ce: 0.028557948768138885 (0.04597760373210622) | > loss_mel_ce: 1.8238097429275513 (nan) | > loss: 0.007350665517151356 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13653084468246918) | > loader_time: 0.0047 (0.004643666062036652)  --> TIME: 2025-02-24 06:53:01 -- STEP: 27310/36938 -- GLOBAL_STEP: 212000 | > loss_text_ce: 0.040812186896800995 (0.045977762434563564) | > loss_mel_ce: 3.7004988193511963 (nan) | > loss: 0.01484647300094366 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13653797201515) | > loader_time: 0.0047 (0.004643762369201246)  --> TIME: 2025-02-24 06:53:11 -- STEP: 27360/36938 -- GLOBAL_STEP: 212050 | > loss_text_ce: 0.030716856941580772 (0.04597624285056067) | > loss_mel_ce: 2.832367420196533 (nan) | > loss: 0.011361446231603622 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13654176600669665) | > loader_time: 0.0045 (0.0046438084272613)  --> TIME: 2025-02-24 06:53:20 -- STEP: 27410/36938 -- GLOBAL_STEP: 212100 | > loss_text_ce: 0.04269610345363617 (0.0459696736326392) | > loss_mel_ce: 3.5566582679748535 (nan) | > loss: 0.014283152297139168 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13654347426349533) | > loader_time: 0.0047 (0.004643762089918206)  --> TIME: 2025-02-24 06:53:29 -- STEP: 27460/36938 -- GLOBAL_STEP: 212150 | > loss_text_ce: 0.06231316551566124 (0.04596914134757545) | > loss_mel_ce: 3.8500866889953613 (nan) | > loss: 0.015525396913290024 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.1365443768157469) | > loader_time: 0.0047 (0.004643963100436096)  --> TIME: 2025-02-24 06:53:39 -- STEP: 27510/36938 -- GLOBAL_STEP: 212200 | > loss_text_ce: 0.05444899573922157 (0.04596701198511768) | > loss_mel_ce: 3.3728973865509033 (nan) | > loss: 0.013600582256913185 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.1365464840399844) | > loader_time: 0.0045 (0.0046439483702204075)  --> TIME: 2025-02-24 06:53:48 -- STEP: 27560/36938 -- GLOBAL_STEP: 212250 | > loss_text_ce: 0.03810251131653786 (0.04596228442558326) | > loss_mel_ce: 3.879610776901245 (nan) | > loss: 0.015546482987701893 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13655369158061165) | > loader_time: 0.0046 (0.004643980927321973)  --> TIME: 2025-02-24 06:53:58 -- STEP: 27610/36938 -- GLOBAL_STEP: 212300 | > loss_text_ce: 0.04436279088258743 (0.04596074984534751) | > loss_mel_ce: 2.530466079711914 (nan) | > loss: 0.01021757535636425 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13656019565205202) | > loader_time: 0.0046 (0.004644010845019874)  --> TIME: 2025-02-24 06:54:07 -- STEP: 27660/36938 -- GLOBAL_STEP: 212350 | > loss_text_ce: 0.035513609647750854 (0.04595859637541212) | > loss_mel_ce: 3.0159876346588135 (nan) | > loss: 0.01210913248360157 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13656677413659193) | > loader_time: 0.0045 (0.004644012545987614)  --> TIME: 2025-02-24 06:54:16 -- STEP: 27710/36938 -- GLOBAL_STEP: 212400 | > loss_text_ce: 0.05104973539710045 (0.0459612614141785) | > loss_mel_ce: 3.0475666522979736 (nan) | > loss: 0.012296097353100777 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13655643781485194) | > loader_time: 0.005 (0.004644067990396959)  --> TIME: 2025-02-24 06:54:26 -- STEP: 27760/36938 -- GLOBAL_STEP: 212450 | > loss_text_ce: 0.0519968718290329 (0.04595787115848) | > loss_mel_ce: 3.4686193466186523 (nan) | > loss: 0.013970700092613697 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.13655651974403316) | > loader_time: 0.0047 (0.0046441209333430425)  --> TIME: 2025-02-24 06:54:35 -- STEP: 27810/36938 -- GLOBAL_STEP: 212500 | > loss_text_ce: 0.043859824538230896 (0.04595984905406968) | > loss_mel_ce: 3.79913330078125 (nan) | > loss: 0.01524997316300869 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13655616935317214) | > loader_time: 0.0046 (0.004644074400566034)  --> TIME: 2025-02-24 06:54:45 -- STEP: 27860/36938 -- GLOBAL_STEP: 212550 | > loss_text_ce: 0.04006699100136757 (0.04595780661045502) | > loss_mel_ce: 2.506887674331665 (nan) | > loss: 0.010106963105499744 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.1365670688958463) | > loader_time: 0.0047 (0.004644092911023982)  --> TIME: 2025-02-24 06:54:54 -- STEP: 27910/36938 -- GLOBAL_STEP: 212600 | > loss_text_ce: 0.049330711364746094 (0.04595461334397187) | > loss_mel_ce: 3.286708116531372 (nan) | > loss: 0.013238250277936459 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13656160044695828) | > loader_time: 0.0068 (0.004644135521629349)  --> TIME: 2025-02-24 06:55:03 -- STEP: 27960/36938 -- GLOBAL_STEP: 212650 | > loss_text_ce: 0.04174259677529335 (0.045955207210883024) | > loss_mel_ce: 3.3789236545562744 (nan) | > loss: 0.01357407309114933 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13655349037507503) | > loader_time: 0.0046 (0.004644175020921859)  --> TIME: 2025-02-24 06:55:12 -- STEP: 28010/36938 -- GLOBAL_STEP: 212700 | > loss_text_ce: 0.046752095222473145 (0.045955410247362244) | > loss_mel_ce: 4.093556880950928 (nan) | > loss: 0.016429798677563667 (nan) | > current_lr: 5e-06 | > step_time: 0.1842 (0.1365527130203231) | > loader_time: 0.0048 (0.00464418542168044)  --> TIME: 2025-02-24 06:55:21 -- STEP: 28060/36938 -- GLOBAL_STEP: 212750 | > loss_text_ce: 0.05683010071516037 (0.04595034047712325) | > loss_mel_ce: 3.783546209335327 (nan) | > loss: 0.015239589847624302 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13654689520321436) | > loader_time: 0.0045 (0.0046441526814009725)  --> TIME: 2025-02-24 06:55:31 -- STEP: 28110/36938 -- GLOBAL_STEP: 212800 | > loss_text_ce: 0.05119921267032623 (0.045948969063454276) | > loss_mel_ce: 3.459317684173584 (nan) | > loss: 0.013930623419582844 (nan) | > current_lr: 5e-06 | > step_time: 0.1823 (0.13654750176317582) | > loader_time: 0.0049 (0.004644212134153261)  --> TIME: 2025-02-24 06:55:40 -- STEP: 28160/36938 -- GLOBAL_STEP: 212850 | > loss_text_ce: 0.04245790094137192 (0.04595099087557852) | > loss_mel_ce: 4.185914993286133 (nan) | > loss: 0.016779258847236633 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.13654929535802243) | > loader_time: 0.0048 (0.004644205700606072)  --> TIME: 2025-02-24 06:55:49 -- STEP: 28210/36938 -- GLOBAL_STEP: 212900 | > loss_text_ce: 0.04527173563838005 (0.04595170223581015) | > loss_mel_ce: 3.323251962661743 (nan) | > loss: 0.013367158360779285 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.13655150843528802) | > loader_time: 0.0047 (0.004644179775206899)  --> TIME: 2025-02-24 06:55:59 -- STEP: 28260/36938 -- GLOBAL_STEP: 212950 | > loss_text_ce: 0.03823848068714142 (0.04595121167562603) | > loss_mel_ce: 0.01263427734375 (nan) | > loss: 0.00020187604241073132 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13654877272212526) | > loader_time: 0.0047 (0.004644197246662166)  --> TIME: 2025-02-24 06:56:08 -- STEP: 28310/36938 -- GLOBAL_STEP: 213000 | > loss_text_ce: 0.048589419573545456 (0.04595020682850679) | > loss_mel_ce: 2.6120574474334717 (nan) | > loss: 0.01055812370032072 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1365470859196794) | > loader_time: 0.0046 (0.004644205822030008)  --> TIME: 2025-02-24 06:56:18 -- STEP: 28360/36938 -- GLOBAL_STEP: 213050 | > loss_text_ce: 0.05542239919304848 (0.04595117963307354) | > loss_mel_ce: 3.0470733642578125 (nan) | > loss: 0.012311491183936596 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.1365547802956058) | > loader_time: 0.0047 (0.004644267069772165)  --> TIME: 2025-02-24 06:56:27 -- STEP: 28410/36938 -- GLOBAL_STEP: 213100 | > loss_text_ce: 0.04072221368551254 (0.04594852689628083) | > loss_mel_ce: 3.0262680053710938 (nan) | > loss: 0.012170596048235893 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13655362700711196) | > loader_time: 0.0046 (0.004644324845807779)  --> TIME: 2025-02-24 06:56:36 -- STEP: 28460/36938 -- GLOBAL_STEP: 213150 | > loss_text_ce: 0.039977267384529114 (0.04594517449451758) | > loss_mel_ce: 3.484051465988159 (nan) | > loss: 0.01398424245417118 (nan) | > current_lr: 5e-06 | > step_time: 0.157 (0.13655280837356032) | > loader_time: 0.0047 (0.004644315006526823)  --> TIME: 2025-02-24 06:56:45 -- STEP: 28510/36938 -- GLOBAL_STEP: 213200 | > loss_text_ce: 0.046212244778871536 (0.04594464346329411) | > loss_mel_ce: 3.3740341663360596 (nan) | > loss: 0.013572406955063343 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.1365478812080243) | > loader_time: 0.0046 (0.004644323449016041)  --> TIME: 2025-02-24 06:56:55 -- STEP: 28560/36938 -- GLOBAL_STEP: 213250 | > loss_text_ce: 0.048529114574193954 (0.04594467189394513) | > loss_mel_ce: 3.8577373027801514 (nan) | > loss: 0.015501058660447598 (nan) | > current_lr: 5e-06 | > step_time: 0.1974 (0.1365527205357043) | > loader_time: 0.0049 (0.004644350102301704)  --> TIME: 2025-02-24 06:57:04 -- STEP: 28610/36938 -- GLOBAL_STEP: 213300 | > loss_text_ce: 0.04583471640944481 (0.04594496062659565) | > loss_mel_ce: 1.7143208980560303 (nan) | > loss: 0.0069847446866333485 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13655440992344142) | > loader_time: 0.0047 (0.004644350545550474)  --> TIME: 2025-02-24 06:57:13 -- STEP: 28660/36938 -- GLOBAL_STEP: 213350 | > loss_text_ce: 0.04551083594560623 (0.04593988319632107) | > loss_mel_ce: 3.7843270301818848 (nan) | > loss: 0.015197769738733768 (nan) | > current_lr: 5e-06 | > step_time: 0.1902 (0.13655100820452326) | > loader_time: 0.005 (0.0046443462455014435)  --> TIME: 2025-02-24 06:57:23 -- STEP: 28710/36938 -- GLOBAL_STEP: 213400 | > loss_text_ce: 0.04205567389726639 (0.04593827692176794) | > loss_mel_ce: 1.27813720703125 (nan) | > loss: 0.005238861311227083 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13655634569150288) | > loader_time: 0.0046 (0.004644427404134589)  --> TIME: 2025-02-24 06:57:32 -- STEP: 28760/36938 -- GLOBAL_STEP: 213450 | > loss_text_ce: 0.04005889967083931 (0.04593567299894275) | > loss_mel_ce: 2.1566967964172363 (nan) | > loss: 0.00871728453785181 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13655825066632923) | > loader_time: 0.0045 (0.0046445265101457646)  --> TIME: 2025-02-24 06:57:42 -- STEP: 28810/36938 -- GLOBAL_STEP: 213500 | > loss_text_ce: 0.047166526317596436 (0.045936433413266246) | > loss_mel_ce: 3.688821315765381 (nan) | > loss: 0.014825349673628807 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13656591202227264) | > loader_time: 0.0046 (0.004644577265696041)  --> TIME: 2025-02-24 06:57:51 -- STEP: 28860/36938 -- GLOBAL_STEP: 213550 | > loss_text_ce: 0.04669106751680374 (0.04593236835603119) | > loss_mel_ce: 2.6137120723724365 (nan) | > loss: 0.010557156056165695 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13656493960695937) | > loader_time: 0.0045 (0.004644525604618113)  --> TIME: 2025-02-24 06:58:00 -- STEP: 28910/36938 -- GLOBAL_STEP: 213600 | > loss_text_ce: 0.04767480120062828 (0.04593118091147064) | > loss_mel_ce: 2.8449175357818604 (nan) | > loss: 0.011478542350232601 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13656222466489065) | > loader_time: 0.0046 (0.004644444878643137)  --> TIME: 2025-02-24 06:58:10 -- STEP: 28960/36938 -- GLOBAL_STEP: 213650 | > loss_text_ce: 0.04853373020887375 (0.045931398526486436) | > loss_mel_ce: 3.0908355712890625 (nan) | > loss: 0.012457814998924732 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.1365626549770174) | > loader_time: 0.0045 (0.0046444508416876255)  --> TIME: 2025-02-24 06:58:19 -- STEP: 29010/36938 -- GLOBAL_STEP: 213700 | > loss_text_ce: 0.038342442363500595 (0.04592725661711997) | > loss_mel_ce: 2.851787567138672 (nan) | > loss: 0.011468770913779736 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13655692004039885) | > loader_time: 0.0045 (0.0046444794590084085)  --> TIME: 2025-02-24 06:58:28 -- STEP: 29060/36938 -- GLOBAL_STEP: 213750 | > loss_text_ce: 0.04094310849905014 (0.0459210006157047) | > loss_mel_ce: 3.616370439529419 (nan) | > loss: 0.014513149857521057 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.1365581991913736) | > loader_time: 0.0048 (0.004644399869056346)  --> TIME: 2025-02-24 06:58:38 -- STEP: 29110/36938 -- GLOBAL_STEP: 213800 | > loss_text_ce: 0.03515361621975899 (0.04592037597224543) | > loss_mel_ce: 3.3340096473693848 (nan) | > loss: 0.013369696214795113 (nan) | > current_lr: 5e-06 | > step_time: 0.1493 (0.13656460817881644) | > loader_time: 0.0047 (0.004644356778052088)  --> TIME: 2025-02-24 06:58:47 -- STEP: 29160/36938 -- GLOBAL_STEP: 213850 | > loss_text_ce: 0.039356593042612076 (0.045915422716627866) | > loss_mel_ce: 2.8749325275421143 (nan) | > loss: 0.011564640328288078 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1365655296133391) | > loader_time: 0.0045 (0.004644394632229554)  --> TIME: 2025-02-24 06:58:56 -- STEP: 29210/36938 -- GLOBAL_STEP: 213900 | > loss_text_ce: 0.03995155543088913 (0.04591344874361799) | > loss_mel_ce: 3.3862392902374268 (nan) | > loss: 0.013595996424555779 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13656363063624322) | > loader_time: 0.0046 (0.004644611591102698)  --> TIME: 2025-02-24 06:59:05 -- STEP: 29260/36938 -- GLOBAL_STEP: 213950 | > loss_text_ce: 0.04156056419014931 (0.04591487433251471) | > loss_mel_ce: 3.3880417346954346 (nan) | > loss: 0.013609534129500389 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13655774202236115) | > loader_time: 0.0045 (0.004644561645028007)  --> TIME: 2025-02-24 06:59:14 -- STEP: 29310/36938 -- GLOBAL_STEP: 214000 | > loss_text_ce: 0.04246553033590317 (0.0459105744036771) | > loss_mel_ce: 0.01403045654296875 (nan) | > loss: 0.0002241904439870268 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13654744581166064) | > loader_time: 0.0046 (0.0046445854691993475)  --> TIME: 2025-02-24 06:59:24 -- STEP: 29360/36938 -- GLOBAL_STEP: 214050 | > loss_text_ce: 0.048934124410152435 (0.045907201843332586) | > loss_mel_ce: 0.119049072265625 (nan) | > loss: 0.0006666000699624419 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13655138016559262) | > loader_time: 0.0046 (0.004644659892414796)  --> TIME: 2025-02-24 06:59:33 -- STEP: 29410/36938 -- GLOBAL_STEP: 214100 | > loss_text_ce: 0.04727435111999512 (0.045904186177639215) | > loss_mel_ce: 3.5220484733581543 (nan) | > loss: 0.014163980260491371 (nan) | > current_lr: 5e-06 | > step_time: 0.17 (0.13654410960681718) | > loader_time: 0.0048 (0.004644672751467196)  --> TIME: 2025-02-24 06:59:42 -- STEP: 29460/36938 -- GLOBAL_STEP: 214150 | > loss_text_ce: 0.042576227337121964 (0.045903803663611714) | > loss_mel_ce: 3.521075487136841 (nan) | > loss: 0.014141475781798363 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.1365504417302909) | > loader_time: 0.0045 (0.004644793923501617)  --> TIME: 2025-02-24 06:59:51 -- STEP: 29510/36938 -- GLOBAL_STEP: 214200 | > loss_text_ce: 0.048014797270298004 (0.045903975594920245) | > loss_mel_ce: 2.674957275390625 (nan) | > loss: 0.010805445723235607 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.1365348466894513) | > loader_time: 0.0049 (0.0046447323040006904)  --> TIME: 2025-02-24 07:00:00 -- STEP: 29560/36938 -- GLOBAL_STEP: 214250 | > loss_text_ce: 0.05681811645627022 (0.0459015299459916) | > loss_mel_ce: 3.077322006225586 (nan) | > loss: 0.012437064200639725 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13653130306281477) | > loader_time: 0.0046 (0.004644736869409343)  --> TIME: 2025-02-24 07:00:10 -- STEP: 29610/36938 -- GLOBAL_STEP: 214300 | > loss_text_ce: 0.040570881217718124 (0.045901259600679784) | > loss_mel_ce: 2.4534664154052734 (nan) | > loss: 0.009896974079310894 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13653419909144032) | > loader_time: 0.0044 (0.004644703599336507)  --> TIME: 2025-02-24 07:00:19 -- STEP: 29660/36938 -- GLOBAL_STEP: 214350 | > loss_text_ce: 0.06474094092845917 (0.045900741194243776) | > loss_mel_ce: 3.638138771057129 (nan) | > loss: 0.014693967998027802 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1365366400212607) | > loader_time: 0.0047 (0.004644652298794157)  --> TIME: 2025-02-24 07:00:28 -- STEP: 29710/36938 -- GLOBAL_STEP: 214400 | > loss_text_ce: 0.049878936260938644 (0.045897017035430365) | > loss_mel_ce: 2.209416627883911 (nan) | > loss: 0.00896545872092247 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13653353040606114) | > loader_time: 0.0046 (0.004644686996796329)  --> TIME: 2025-02-24 07:00:38 -- STEP: 29760/36938 -- GLOBAL_STEP: 214450 | > loss_text_ce: 0.05139319226145744 (0.045896763439066325) | > loss_mel_ce: 2.7706480026245117 (nan) | > loss: 0.011198577471077442 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13654740791647835) | > loader_time: 0.0047 (0.004644784651776762)  --> TIME: 2025-02-24 07:00:48 -- STEP: 29810/36938 -- GLOBAL_STEP: 214500 | > loss_text_ce: 0.051756665110588074 (0.0458918633366537) | > loss_mel_ce: 3.056206703186035 (nan) | > loss: 0.012333188205957413 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13655545569154193) | > loader_time: 0.0047 (0.004644819227331844)  --> TIME: 2025-02-24 07:00:57 -- STEP: 29860/36938 -- GLOBAL_STEP: 214550 | > loss_text_ce: 0.04551570490002632 (0.04589094056377337) | > loss_mel_ce: 3.3164114952087402 (nan) | > loss: 0.013340981677174568 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13654937415177454) | > loader_time: 0.0046 (0.00464476904290995)  --> TIME: 2025-02-24 07:01:06 -- STEP: 29910/36938 -- GLOBAL_STEP: 214600 | > loss_text_ce: 0.025389377027750015 (0.04588777965541282) | > loss_mel_ce: 2.764014959335327 (nan) | > loss: 0.011069065891206264 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13655134159280907) | > loader_time: 0.0045 (0.004644862260757866)  --> TIME: 2025-02-24 07:01:15 -- STEP: 29960/36938 -- GLOBAL_STEP: 214650 | > loss_text_ce: 0.03829013928771019 (0.04588287354565711) | > loss_mel_ce: 3.4008686542510986 (nan) | > loss: 0.013647455722093582 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13654804863185077) | > loader_time: 0.0048 (0.00464483897421799)  --> TIME: 2025-02-24 07:01:25 -- STEP: 30010/36938 -- GLOBAL_STEP: 214700 | > loss_text_ce: 0.05453796312212944 (0.04588456736711236) | > loss_mel_ce: 3.86279296875 (nan) | > loss: 0.01554496493190527 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13655466139614672) | > loader_time: 0.0046 (0.004644900129382396)  --> TIME: 2025-02-24 07:01:34 -- STEP: 30060/36938 -- GLOBAL_STEP: 214750 | > loss_text_ce: 0.037592969834804535 (0.04588799474619008) | > loss_mel_ce: 3.386481523513794 (nan) | > loss: 0.013587597757577896 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.13655247812023846) | > loader_time: 0.0046 (0.004644936105051724)  --> TIME: 2025-02-24 07:01:44 -- STEP: 30110/36938 -- GLOBAL_STEP: 214800 | > loss_text_ce: 0.04071579873561859 (0.04588587886428991) | > loss_mel_ce: 2.3722453117370605 (nan) | > loss: 0.009575242176651955 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13656208425691171) | > loader_time: 0.0046 (0.004645008535647984)  --> TIME: 2025-02-24 07:01:53 -- STEP: 30160/36938 -- GLOBAL_STEP: 214850 | > loss_text_ce: 0.04176577180624008 (0.04588295040532978) | > loss_mel_ce: 3.828317165374756 (nan) | > loss: 0.01535747293382883 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13656314677047662) | > loader_time: 0.0046 (0.0046450140068322)  --> TIME: 2025-02-24 07:02:02 -- STEP: 30210/36938 -- GLOBAL_STEP: 214900 | > loss_text_ce: 0.037851881235837936 (0.04588321827741117) | > loss_mel_ce: 2.277714729309082 (nan) | > loss: 0.009188756346702576 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13655969526782913) | > loader_time: 0.0048 (0.004645089186094137)  --> TIME: 2025-02-24 07:02:11 -- STEP: 30260/36938 -- GLOBAL_STEP: 214950 | > loss_text_ce: 0.04266693815588951 (0.04587927819543608) | > loss_mel_ce: 2.0898540019989014 (nan) | > loss: 0.008462385274469852 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13655048058037902) | > loader_time: 0.0045 (0.004645214605551999)  --> TIME: 2025-02-24 07:02:21 -- STEP: 30310/36938 -- GLOBAL_STEP: 215000 | > loss_text_ce: 0.03955377638339996 (0.045878358717445056) | > loss_mel_ce: 3.8203125 (nan) | > loss: 0.01531693059951067 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13655763472397103) | > loader_time: 0.0045 (0.004645198274862217)  --> TIME: 2025-02-24 07:02:30 -- STEP: 30360/36938 -- GLOBAL_STEP: 215050 | > loss_text_ce: 0.04461861401796341 (0.04587871864857107) | > loss_mel_ce: 3.715449810028076 (nan) | > loss: 0.014920907095074654 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13656079751858) | > loader_time: 0.0046 (0.004645136308921308)  --> TIME: 2025-02-24 07:02:40 -- STEP: 30410/36938 -- GLOBAL_STEP: 215100 | > loss_text_ce: 0.04088879004120827 (0.045876922718696866) | > loss_mel_ce: 3.617124319076538 (nan) | > loss: 0.014515926130115986 (nan) | > current_lr: 5e-06 | > step_time: 0.1503 (0.13655960536324743) | > loader_time: 0.0048 (0.004645132265840426)  --> TIME: 2025-02-24 07:02:49 -- STEP: 30460/36938 -- GLOBAL_STEP: 215150 | > loss_text_ce: 0.048105139285326004 (0.0458751185070994) | > loss_mel_ce: 3.2679920196533203 (nan) | > loss: 0.013159116730093956 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.13655326423563802) | > loader_time: 0.0047 (0.004645063324500568)  --> TIME: 2025-02-24 07:02:58 -- STEP: 30510/36938 -- GLOBAL_STEP: 215200 | > loss_text_ce: 0.04470229521393776 (0.04587473990957344) | > loss_mel_ce: 3.7350454330444336 (nan) | > loss: 0.014998999424278736 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13655121341294155) | > loader_time: 0.0046 (0.004645118678995335)  --> TIME: 2025-02-24 07:03:07 -- STEP: 30560/36938 -- GLOBAL_STEP: 215250 | > loss_text_ce: 0.04474173113703728 (0.04587077490692238) | > loss_mel_ce: 3.7760350704193115 (nan) | > loss: 0.015161813236773014 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13655176080022366) | > loader_time: 0.0046 (0.004645197584991022)  --> TIME: 2025-02-24 07:03:16 -- STEP: 30610/36938 -- GLOBAL_STEP: 215300 | > loss_text_ce: 0.045512180775403976 (0.04587238487528641) | > loss_mel_ce: 0.5251855850219727 (nan) | > loss: 0.0022646738216280937 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13654377940276866) | > loader_time: 0.0045 (0.004645165809810771)  --> TIME: 2025-02-24 07:03:25 -- STEP: 30660/36938 -- GLOBAL_STEP: 215350 | > loss_text_ce: 0.04098404198884964 (0.045871950763345305) | > loss_mel_ce: 3.7815322875976562 (nan) | > loss: 0.015168717131018639 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13654088796701633) | > loader_time: 0.0046 (0.004645205974889872)  --> TIME: 2025-02-24 07:03:35 -- STEP: 30710/36938 -- GLOBAL_STEP: 215400 | > loss_text_ce: 0.04101863130927086 (0.04587237467226331) | > loss_mel_ce: 3.8632757663726807 (nan) | > loss: 0.015493232756853104 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13653964270446275) | > loader_time: 0.0045 (0.004645183295234689)  --> TIME: 2025-02-24 07:03:44 -- STEP: 30760/36938 -- GLOBAL_STEP: 215450 | > loss_text_ce: 0.03790565952658653 (0.04587005968099677) | > loss_mel_ce: 3.6432297229766846 (nan) | > loss: 0.014607680961489677 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.1365399007291899) | > loader_time: 0.0049 (0.004645225045274818)  --> TIME: 2025-02-24 07:03:53 -- STEP: 30810/36938 -- GLOBAL_STEP: 215500 | > loss_text_ce: 0.03819020837545395 (0.04586968901934447) | > loss_mel_ce: 3.479773759841919 (nan) | > loss: 0.013960175216197968 (nan) | > current_lr: 5e-06 | > step_time: 0.1452 (0.13653098837349048) | > loader_time: 0.0045 (0.0046451444944674205)  --> TIME: 2025-02-24 07:04:02 -- STEP: 30860/36938 -- GLOBAL_STEP: 215550 | > loss_text_ce: 0.04892304912209511 (0.045869250656561444) | > loss_mel_ce: 0.1942342072725296 (nan) | > loss: 0.0009649097919464111 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13652505121367237) | > loader_time: 0.0047 (0.0046450202834350885)  --> TIME: 2025-02-24 07:04:11 -- STEP: 30910/36938 -- GLOBAL_STEP: 215600 | > loss_text_ce: 0.03830927237868309 (0.045868443470781055) | > loss_mel_ce: 3.2798709869384766 (nan) | > loss: 0.013167383149266243 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1365289399246708) | > loader_time: 0.0044 (0.004644914685387826)  --> TIME: 2025-02-24 07:04:21 -- STEP: 30960/36938 -- GLOBAL_STEP: 215650 | > loss_text_ce: 0.039251573383808136 (0.04586647400197539) | > loss_mel_ce: 2.897855758666992 (nan) | > loss: 0.011655188165605068 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13653178268933047) | > loader_time: 0.0045 (0.004644843396906364)  --> TIME: 2025-02-24 07:04:30 -- STEP: 31010/36938 -- GLOBAL_STEP: 215700 | > loss_text_ce: 0.03817692771553993 (0.045866101057266136) | > loss_mel_ce: 1.875854253768921 (nan) | > loss: 0.007595362141728401 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13653047782919045) | > loader_time: 0.0045 (0.004644896491271192)  --> TIME: 2025-02-24 07:04:39 -- STEP: 31060/36938 -- GLOBAL_STEP: 215750 | > loss_text_ce: 0.0468391589820385 (0.04586626810371935) | > loss_mel_ce: 2.51265549659729 (nan) | > loss: 0.010156725533306599 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13653185875739912) | > loader_time: 0.0044 (0.00464497416232831)  --> TIME: 2025-02-24 07:04:49 -- STEP: 31110/36938 -- GLOBAL_STEP: 215800 | > loss_text_ce: 0.04096390679478645 (0.04586472756599671) | > loss_mel_ce: 3.087296962738037 (nan) | > loss: 0.012413734570145607 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.13652808473869857) | > loader_time: 0.0047 (0.004644920587616277)  --> TIME: 2025-02-24 07:04:58 -- STEP: 31160/36938 -- GLOBAL_STEP: 215850 | > loss_text_ce: 0.049770288169384 (0.04586149132771694) | > loss_mel_ce: 3.0914225578308105 (nan) | > loss: 0.012465052306652069 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.1365328461658682) | > loader_time: 0.0046 (0.004644880475373535)  --> TIME: 2025-02-24 07:05:07 -- STEP: 31210/36938 -- GLOBAL_STEP: 215900 | > loss_text_ce: 0.049105532467365265 (0.04586126229148977) | > loss_mel_ce: 3.4266841411590576 (nan) | > loss: 0.01379281748086214 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.1365310790408176) | > loader_time: 0.0049 (0.004644890505317966)  --> TIME: 2025-02-24 07:05:17 -- STEP: 31260/36938 -- GLOBAL_STEP: 215950 | > loss_text_ce: 0.03332749754190445 (0.04585977691769144) | > loss_mel_ce: 3.127732992172241 (nan) | > loss: 0.012543891556560993 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13653929003217197) | > loader_time: 0.0046 (0.004644854749080359)  --> TIME: 2025-02-24 07:05:26 -- STEP: 31310/36938 -- GLOBAL_STEP: 216000 | > loss_text_ce: 0.04668883606791496 (0.045858547919323214) | > loss_mel_ce: 3.5062294006347656 (nan) | > loss: 0.014098882675170898 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13653368444269312) | > loader_time: 0.0048 (0.004644832539505075)  --> TIME: 2025-02-24 07:05:35 -- STEP: 31360/36938 -- GLOBAL_STEP: 216050 | > loss_text_ce: 0.04024185985326767 (0.04585735381118214) | > loss_mel_ce: 3.182072162628174 (nan) | > loss: 0.012786961160600185 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13653281236789844) | > loader_time: 0.0047 (0.0046447505528221)  --> TIME: 2025-02-24 07:05:44 -- STEP: 31410/36938 -- GLOBAL_STEP: 216100 | > loss_text_ce: 0.039139118045568466 (0.04585537680417946) | > loss_mel_ce: 3.5446882247924805 (nan) | > loss: 0.014221537858247757 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13653314998673038) | > loader_time: 0.0045 (0.004644668090878575)  --> TIME: 2025-02-24 07:05:54 -- STEP: 31460/36938 -- GLOBAL_STEP: 216150 | > loss_text_ce: 0.0616234727203846 (0.04585675458271056) | > loss_mel_ce: 2.9739415645599365 (nan) | > loss: 0.01204589381814003 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13653500405225066) | > loader_time: 0.0044 (0.004644617940389767)  --> TIME: 2025-02-24 07:06:03 -- STEP: 31510/36938 -- GLOBAL_STEP: 216200 | > loss_text_ce: 0.047522276639938354 (0.04585394871612404) | > loss_mel_ce: 2.968316078186035 (nan) | > loss: 0.011967613361775875 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13654019410403814) | > loader_time: 0.0045 (0.0046446061141904495)  --> TIME: 2025-02-24 07:06:13 -- STEP: 31560/36938 -- GLOBAL_STEP: 216250 | > loss_text_ce: 0.029951125383377075 (0.04585180963193058) | > loss_mel_ce: 3.3046715259552 (nan) | > loss: 0.013232630677521229 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13654403027775197) | > loader_time: 0.0047 (0.004644620191914921)  --> TIME: 2025-02-24 07:06:22 -- STEP: 31610/36938 -- GLOBAL_STEP: 216300 | > loss_text_ce: 0.04812314733862877 (0.045850314013984206) | > loss_mel_ce: 4.272345066070557 (nan) | > loss: 0.017144717276096344 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13654533155882548) | > loader_time: 0.0046 (0.004644633749925919)  --> TIME: 2025-02-24 07:06:32 -- STEP: 31660/36938 -- GLOBAL_STEP: 216350 | > loss_text_ce: 0.04447990283370018 (0.04584992086248334) | > loss_mel_ce: 3.4897420406341553 (nan) | > loss: 0.014024690724909306 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13655233091744934) | > loader_time: 0.0048 (0.0046446723269271285)  --> TIME: 2025-02-24 07:06:41 -- STEP: 31710/36938 -- GLOBAL_STEP: 216400 | > loss_text_ce: 0.05422373488545418 (0.04584934539680019) | > loss_mel_ce: 2.2358038425445557 (nan) | > loss: 0.009087411686778069 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13655255869411764) | > loader_time: 0.0047 (0.00464470481240983)  --> TIME: 2025-02-24 07:06:51 -- STEP: 31760/36938 -- GLOBAL_STEP: 216450 | > loss_text_ce: 0.03774752840399742 (0.04584733192361461) | > loss_mel_ce: 3.5080413818359375 (nan) | > loss: 0.014070591889321804 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13655462197452886) | > loader_time: 0.0049 (0.004644724508977324)  --> TIME: 2025-02-24 07:06:59 -- STEP: 31810/36938 -- GLOBAL_STEP: 216500 | > loss_text_ce: 0.05724388360977173 (0.045847556625365084) | > loss_mel_ce: 4.075333118438721 (nan) | > loss: 0.016399115324020386 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.13654670637383348) | > loader_time: 0.0049 (0.00464479394844961)  --> TIME: 2025-02-24 07:07:09 -- STEP: 31860/36938 -- GLOBAL_STEP: 216550 | > loss_text_ce: 0.028490843251347542 (0.045844460873352026) | > loss_mel_ce: 1.9591796398162842 (nan) | > loss: 0.007887582294642925 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1365465408140824) | > loader_time: 0.0046 (0.004644853890653052)  --> TIME: 2025-02-24 07:07:18 -- STEP: 31910/36938 -- GLOBAL_STEP: 216600 | > loss_text_ce: 0.049035873264074326 (0.045840214807026665) | > loss_mel_ce: 2.2693052291870117 (nan) | > loss: 0.00919976644217968 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.13653927801993704) | > loader_time: 0.0047 (0.004644821027123907)  --> TIME: 2025-02-24 07:07:27 -- STEP: 31960/36938 -- GLOBAL_STEP: 216650 | > loss_text_ce: 0.04246899485588074 (0.045838607802902165) | > loss_mel_ce: 3.253160238265991 (nan) | > loss: 0.013077894225716591 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13653806600165036) | > loader_time: 0.0047 (0.004644797225768304)  --> TIME: 2025-02-24 07:07:36 -- STEP: 32010/36938 -- GLOBAL_STEP: 216700 | > loss_text_ce: 0.04902414605021477 (0.0458394592807399) | > loss_mel_ce: 3.744908332824707 (nan) | > loss: 0.015055288560688496 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13653827103850646) | > loader_time: 0.0045 (0.004644780567160815)  --> TIME: 2025-02-24 07:07:46 -- STEP: 32060/36938 -- GLOBAL_STEP: 216750 | > loss_text_ce: 0.03797360882163048 (0.045838395565026786) | > loss_mel_ce: 4.114057540893555 (nan) | > loss: 0.016476314514875412 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.1365451481590717) | > loader_time: 0.0046 (0.0046448217059995986)  --> TIME: 2025-02-24 07:07:55 -- STEP: 32110/36938 -- GLOBAL_STEP: 216800 | > loss_text_ce: 0.04083242639899254 (0.04583887555600226) | > loss_mel_ce: 3.4933066368103027 (nan) | > loss: 0.014024362899363041 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13654969884242374) | > loader_time: 0.0048 (0.004644853509649282)  --> TIME: 2025-02-24 07:08:05 -- STEP: 32160/36938 -- GLOBAL_STEP: 216850 | > loss_text_ce: 0.048257239162921906 (0.04583791769981349) | > loss_mel_ce: 0.023088904097676277 (nan) | > loss: 0.0002831196179613471 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13655684829054957) | > loader_time: 0.0046 (0.004644880002707359)  --> TIME: 2025-02-24 07:08:14 -- STEP: 32210/36938 -- GLOBAL_STEP: 216900 | > loss_text_ce: 0.03908944129943848 (0.045838624295957704) | > loss_mel_ce: 3.470400333404541 (nan) | > loss: 0.013926547951996326 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13654829950223266) | > loader_time: 0.0048 (0.004644809358425635)  --> TIME: 2025-02-24 07:08:23 -- STEP: 32260/36938 -- GLOBAL_STEP: 216950 | > loss_text_ce: 0.04333745688199997 (0.04584072244795885) | > loss_mel_ce: 4.215610504150391 (nan) | > loss: 0.01690058782696724 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13654685157910856) | > loader_time: 0.0046 (0.004644835549559682)  --> TIME: 2025-02-24 07:08:32 -- STEP: 32310/36938 -- GLOBAL_STEP: 217000 | > loss_text_ce: 0.052024539560079575 (0.04583886213423668) | > loss_mel_ce: 3.2985382080078125 (nan) | > loss: 0.013295885175466537 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13654293606651147) | > loader_time: 0.0046 (0.004644795720033369)  --> TIME: 2025-02-24 07:08:42 -- STEP: 32360/36938 -- GLOBAL_STEP: 217050 | > loss_text_ce: 0.03300882875919342 (0.04583677691853608) | > loss_mel_ce: 2.1546294689178467 (nan) | > loss: 0.008681104518473148 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13654222142121875) | > loader_time: 0.0047 (0.00464473685021745)  --> TIME: 2025-02-24 07:08:51 -- STEP: 32410/36938 -- GLOBAL_STEP: 217100 | > loss_text_ce: 0.04743392765522003 (0.045833118916372956) | > loss_mel_ce: 3.8242738246917725 (nan) | > loss: 0.015363920480012894 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.1365514322917665) | > loader_time: 0.0046 (0.004644786675231582)  --> TIME: 2025-02-24 07:09:01 -- STEP: 32460/36938 -- GLOBAL_STEP: 217150 | > loss_text_ce: 0.03969044238328934 (0.0458341796729527) | > loss_mel_ce: 3.530806541442871 (nan) | > loss: 0.014168639667332172 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.1365583528505767) | > loader_time: 0.0046 (0.004644783558260827)  --> TIME: 2025-02-24 07:09:10 -- STEP: 32510/36938 -- GLOBAL_STEP: 217200 | > loss_text_ce: 0.04217955470085144 (0.04583540833270137) | > loss_mel_ce: 3.212918996810913 (nan) | > loss: 0.012917058542370796 (nan) | > current_lr: 5e-06 | > step_time: 0.1628 (0.13655527896494057) | > loader_time: 0.0048 (0.004644792485479102)  --> TIME: 2025-02-24 07:09:20 -- STEP: 32560/36938 -- GLOBAL_STEP: 217250 | > loss_text_ce: 0.048983532935380936 (0.045833326825164) | > loss_mel_ce: 2.157597541809082 (nan) | > loss: 0.008756274357438087 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.1365553422407688) | > loader_time: 0.0046 (0.004644814492443531)  --> TIME: 2025-02-24 07:09:29 -- STEP: 32610/36938 -- GLOBAL_STEP: 217300 | > loss_text_ce: 0.03868561238050461 (0.045830888301505536) | > loss_mel_ce: 2.300870180130005 (nan) | > loss: 0.009283951483666897 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1365475037097217) | > loader_time: 0.0046 (0.004644770645649261)  --> TIME: 2025-02-24 07:09:38 -- STEP: 32660/36938 -- GLOBAL_STEP: 217350 | > loss_text_ce: 0.05585451051592827 (0.045828972114318654) | > loss_mel_ce: 3.006010055541992 (nan) | > loss: 0.012150256894528866 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13653985140736993) | > loader_time: 0.0048 (0.004644735233227429)  --> TIME: 2025-02-24 07:09:47 -- STEP: 32710/36938 -- GLOBAL_STEP: 217400 | > loss_text_ce: 0.045550644397735596 (0.045828926036842334) | > loss_mel_ce: 3.7056119441986084 (nan) | > loss: 0.01488556619733572 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13654271016898076) | > loader_time: 0.0047 (0.004644746359099425)  --> TIME: 2025-02-24 07:09:56 -- STEP: 32760/36938 -- GLOBAL_STEP: 217450 | > loss_text_ce: 0.03362874314188957 (0.04582835909622249) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1076 (0.13654339191969087) | > loader_time: 0.0045 (0.004644733223431828)  --> TIME: 2025-02-24 07:10:06 -- STEP: 32810/36938 -- GLOBAL_STEP: 217500 | > loss_text_ce: 0.04135314002633095 (0.04582482204349322) | > loss_mel_ce: 3.192389726638794 (nan) | > loss: 0.012832313776016235 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.13655402252711335) | > loader_time: 0.0047 (0.004644812610373892)  --> TIME: 2025-02-24 07:10:15 -- STEP: 32860/36938 -- GLOBAL_STEP: 217550 | > loss_text_ce: 0.06513626128435135 (0.04582403798369968) | > loss_mel_ce: 2.2477972507476807 (nan) | > loss: 0.009178307838737965 (nan) | > current_lr: 5e-06 | > step_time: 0.1086 (0.1365509101550769) | > loader_time: 0.0045 (0.004644814084668863)  --> TIME: 2025-02-24 07:10:25 -- STEP: 32910/36938 -- GLOBAL_STEP: 217600 | > loss_text_ce: 0.04361772537231445 (0.045821116937777906) | > loss_mel_ce: 3.2669191360473633 (nan) | > loss: 0.013137051835656166 (nan) | > current_lr: 5e-06 | > step_time: 0.1789 (0.1365498534816642) | > loader_time: 0.0049 (0.00464481029493108)  --> TIME: 2025-02-24 07:10:34 -- STEP: 32960/36938 -- GLOBAL_STEP: 217650 | > loss_text_ce: 0.051097385585308075 (0.045819461581187736) | > loss_mel_ce: 3.457015037536621 (nan) | > loss: 0.013921082019805908 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.13655676288367646) | > loader_time: 0.0048 (0.004644840355347629)  --> TIME: 2025-02-24 07:10:43 -- STEP: 33010/36938 -- GLOBAL_STEP: 217700 | > loss_text_ce: 0.03910539671778679 (0.045815955523753754) | > loss_mel_ce: 3.1521999835968018 (nan) | > loss: 0.01266391109675169 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13654264219238582) | > loader_time: 0.0046 (0.004644831322569123)  --> TIME: 2025-02-24 07:10:52 -- STEP: 33060/36938 -- GLOBAL_STEP: 217750 | > loss_text_ce: 0.049194879829883575 (0.04581509711752175) | > loss_mel_ce: 3.2134854793548584 (nan) | > loss: 0.01294714491814375 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.136540294097399) | > loader_time: 0.0045 (0.0046448094802989965)  --> TIME: 2025-02-24 07:11:01 -- STEP: 33110/36938 -- GLOBAL_STEP: 217800 | > loss_text_ce: 0.036304403096437454 (0.04581670754157951) | > loss_mel_ce: 3.930288553237915 (nan) | > loss: 0.015740448608994484 (nan) | > current_lr: 5e-06 | > step_time: 0.1072 (0.13653570825835604) | > loader_time: 0.0046 (0.004644836820673241)  --> TIME: 2025-02-24 07:11:11 -- STEP: 33160/36938 -- GLOBAL_STEP: 217850 | > loss_text_ce: 0.034522589296102524 (0.04581355468487433) | > loss_mel_ce: 0.1540934294462204 (nan) | > loss: 0.0007484763045795262 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13653675903744972) | > loader_time: 0.0045 (0.00464490093425614)  --> TIME: 2025-02-24 07:11:20 -- STEP: 33210/36938 -- GLOBAL_STEP: 217900 | > loss_text_ce: 0.04567090794444084 (0.04581218492264967) | > loss_mel_ce: 3.6442933082580566 (nan) | > loss: 0.014642716385424137 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.1365350755429076) | > loader_time: 0.0045 (0.00464480791921822)  --> TIME: 2025-02-24 07:11:29 -- STEP: 33260/36938 -- GLOBAL_STEP: 217950 | > loss_text_ce: 0.0390041284263134 (0.04581062683073446) | > loss_mel_ce: 3.0125744342803955 (nan) | > loss: 0.012109438888728619 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13653538480840263) | > loader_time: 0.0046 (0.004644724309121821)  --> TIME: 2025-02-24 07:11:39 -- STEP: 33310/36938 -- GLOBAL_STEP: 218000 | > loss_text_ce: 0.043738119304180145 (0.045806697467317005) | > loss_mel_ce: 3.4402413368225098 (nan) | > loss: 0.013825315982103348 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.1365320502512824) | > loader_time: 0.0049 (0.004644605505756311)  --> TIME: 2025-02-24 07:11:48 -- STEP: 33360/36938 -- GLOBAL_STEP: 218050 | > loss_text_ce: 0.06632432341575623 (0.0458057346804308) | > loss_mel_ce: 1.45533287525177 (nan) | > loss: 0.006038322579115629 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13653000915508062) | > loader_time: 0.0078 (0.004644864718976893)  --> TIME: 2025-02-24 07:11:57 -- STEP: 33410/36938 -- GLOBAL_STEP: 218100 | > loss_text_ce: 0.052243299782276154 (0.04580226546968523) | > loss_mel_ce: 3.4832141399383545 (nan) | > loss: 0.014029593206942081 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13653417114582242) | > loader_time: 0.0046 (0.004644806311537967)  --> TIME: 2025-02-24 07:12:06 -- STEP: 33460/36938 -- GLOBAL_STEP: 218150 | > loss_text_ce: 0.04812351241707802 (0.04580140975198195) | > loss_mel_ce: 2.8864848613739014 (nan) | > loss: 0.011645272374153137 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13653147683055422) | > loader_time: 0.0045 (0.00464470504788101)  --> TIME: 2025-02-24 07:12:16 -- STEP: 33510/36938 -- GLOBAL_STEP: 218200 | > loss_text_ce: 0.030487770214676857 (0.045800666151786906) | > loss_mel_ce: 4.703125 (nan) | > loss: 0.0187841784209013 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13653139332379008) | > loader_time: 0.0046 (0.004644707223259813)  --> TIME: 2025-02-24 07:12:25 -- STEP: 33560/36938 -- GLOBAL_STEP: 218250 | > loss_text_ce: 0.05734318494796753 (0.045799722171974824) | > loss_mel_ce: 2.4082932472229004 (nan) | > loss: 0.009784271940588951 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13653181582440801) | > loader_time: 0.0069 (0.004644785904912723)  --> TIME: 2025-02-24 07:12:34 -- STEP: 33610/36938 -- GLOBAL_STEP: 218300 | > loss_text_ce: 0.0436716265976429 (0.04579891171325783) | > loss_mel_ce: 3.635061740875244 (nan) | > loss: 0.014598148874938488 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13652464472371628) | > loader_time: 0.0048 (0.004644835275470099)  --> TIME: 2025-02-24 07:12:44 -- STEP: 33660/36938 -- GLOBAL_STEP: 218350 | > loss_text_ce: 0.041557855904102325 (0.04579736039280991) | > loss_mel_ce: 2.831939697265625 (nan) | > loss: 0.011402768082916737 (nan) | > current_lr: 5e-06 | > step_time: 0.1788 (0.13653019610715098) | > loader_time: 0.0048 (0.004644818859870958)  --> TIME: 2025-02-24 07:12:53 -- STEP: 33710/36938 -- GLOBAL_STEP: 218400 | > loss_text_ce: 0.04957490786910057 (0.0457971340792658) | > loss_mel_ce: 3.0286970138549805 (nan) | > loss: 0.012215365655720234 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.1365335813512171) | > loader_time: 0.0049 (0.004644858232417228)  --> TIME: 2025-02-24 07:13:02 -- STEP: 33760/36938 -- GLOBAL_STEP: 218450 | > loss_text_ce: 0.056315548717975616 (0.045796208471414156) | > loss_mel_ce: 3.538540840148926 (nan) | > loss: 0.014265304431319237 (nan) | > current_lr: 5e-06 | > step_time: 0.1773 (0.136532122313413) | > loader_time: 0.0048 (0.004644872332918671)  --> TIME: 2025-02-24 07:13:12 -- STEP: 33810/36938 -- GLOBAL_STEP: 218500 | > loss_text_ce: 0.05574949458241463 (0.04579330426058229) | > loss_mel_ce: 3.778576612472534 (nan) | > loss: 0.015215580351650715 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.1365296114693393) | > loader_time: 0.0046 (0.004644844412838878)  --> TIME: 2025-02-24 07:13:21 -- STEP: 33860/36938 -- GLOBAL_STEP: 218550 | > loss_text_ce: 0.03703887388110161 (0.045790329599165645) | > loss_mel_ce: 1.2400975227355957 (nan) | > loss: 0.0050680022686719894 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13652857824394454) | > loader_time: 0.0046 (0.004644904676021628)  --> TIME: 2025-02-24 07:13:30 -- STEP: 33910/36938 -- GLOBAL_STEP: 218600 | > loss_text_ce: 0.04294777661561966 (0.04578865396036867) | > loss_mel_ce: 0.036865234375 (nan) | > loss: 0.0003167183022014797 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13652408896417986) | > loader_time: 0.0046 (0.0046449205229254645)  --> TIME: 2025-02-24 07:13:39 -- STEP: 33960/36938 -- GLOBAL_STEP: 218650 | > loss_text_ce: 0.03172704204916954 (0.04578722682323548) | > loss_mel_ce: 2.7711963653564453 (nan) | > loss: 0.01112271286547184 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13651814126575398) | > loader_time: 0.0045 (0.00464486304244947)  --> TIME: 2025-02-24 07:13:48 -- STEP: 34010/36938 -- GLOBAL_STEP: 218700 | > loss_text_ce: 0.04232284426689148 (0.04578780740523059) | > loss_mel_ce: 3.5693814754486084 (nan) | > loss: 0.014332161284983158 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.1365214306619106) | > loader_time: 0.0048 (0.004645188960283131)  --> TIME: 2025-02-24 07:13:58 -- STEP: 34060/36938 -- GLOBAL_STEP: 218750 | > loss_text_ce: 0.049550846219062805 (0.045786144611405594) | > loss_mel_ce: 2.907698154449463 (nan) | > loss: 0.011735115200281143 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13651993483287642) | > loader_time: 0.0046 (0.004645339496263079)  --> TIME: 2025-02-24 07:14:07 -- STEP: 34110/36938 -- GLOBAL_STEP: 218800 | > loss_text_ce: 0.04240238666534424 (0.045786414382014494) | > loss_mel_ce: 1.2615245580673218 (nan) | > loss: 0.005174313671886921 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13652452786484084) | > loader_time: 0.0045 (0.00464539910928482)  --> TIME: 2025-02-24 07:14:16 -- STEP: 34160/36938 -- GLOBAL_STEP: 218850 | > loss_text_ce: 0.04105108976364136 (0.045785421348777916) | > loss_mel_ce: 1.0483770370483398 (nan) | > loss: 0.004323128145188093 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.1365226419943574) | > loader_time: 0.0045 (0.004645383762810739)  --> TIME: 2025-02-24 07:14:26 -- STEP: 34210/36938 -- GLOBAL_STEP: 218900 | > loss_text_ce: 0.0522758811712265 (0.045787288397752174) | > loss_mel_ce: 4.302250862121582 (nan) | > loss: 0.017279868945479393 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13651969263477828) | > loader_time: 0.0045 (0.004645489217662809)  --> TIME: 2025-02-24 07:14:35 -- STEP: 34260/36938 -- GLOBAL_STEP: 218950 | > loss_text_ce: 0.04842504858970642 (0.04578383669512165) | > loss_mel_ce: 3.007087230682373 (nan) | > loss: 0.012125048786401749 (nan) | > current_lr: 5e-06 | > step_time: 0.1588 (0.13651515296935096) | > loader_time: 0.0048 (0.004645441083690927)  --> TIME: 2025-02-24 07:14:44 -- STEP: 34310/36938 -- GLOBAL_STEP: 219000 | > loss_text_ce: 0.04244629666209221 (0.04578356810820205) | > loss_mel_ce: 1.4267222881317139 (nan) | > loss: 0.005830034147948027 (nan) | > current_lr: 5e-06 | > step_time: 0.1068 (0.1365116788916409) | > loader_time: 0.0044 (0.004645391707168767)  --> TIME: 2025-02-24 07:14:53 -- STEP: 34360/36938 -- GLOBAL_STEP: 219050 | > loss_text_ce: 0.057282570749521255 (0.04578328998588643) | > loss_mel_ce: 3.2098958492279053 (nan) | > loss: 0.01296499464660883 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13650965850900193) | > loader_time: 0.0045 (0.004645431992338725)  --> TIME: 2025-02-24 07:15:02 -- STEP: 34410/36938 -- GLOBAL_STEP: 219100 | > loss_text_ce: 0.045468199998140335 (0.045781086461222996) | > loss_mel_ce: 3.862569570541382 (nan) | > loss: 0.015508086420595646 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.136505873506066) | > loader_time: 0.0046 (0.004645377437377563)  --> TIME: 2025-02-24 07:15:12 -- STEP: 34460/36938 -- GLOBAL_STEP: 219150 | > loss_text_ce: 0.03513070568442345 (0.045780067742806144) | > loss_mel_ce: 3.030695676803589 (nan) | > loss: 0.01216597855091095 (nan) | > current_lr: 5e-06 | > step_time: 0.1902 (0.13650918767954703) | > loader_time: 0.005 (0.004645368641915848)  --> TIME: 2025-02-24 07:15:21 -- STEP: 34510/36938 -- GLOBAL_STEP: 219200 | > loss_text_ce: 0.041085027158260345 (0.04577982321230998) | > loss_mel_ce: 2.7047119140625 (nan) | > loss: 0.010896020568907261 (nan) | > current_lr: 5e-06 | > step_time: 0.1441 (0.13650801966619536) | > loader_time: 0.0046 (0.004645305369356933)  --> TIME: 2025-02-24 07:15:31 -- STEP: 34560/36938 -- GLOBAL_STEP: 219250 | > loss_text_ce: 0.05132238194346428 (0.04577806084823738) | > loss_mel_ce: 3.6910557746887207 (nan) | > loss: 0.014850707724690437 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.13651079482364606) | > loader_time: 0.0047 (0.004645325057208506)  --> TIME: 2025-02-24 07:15:40 -- STEP: 34610/36938 -- GLOBAL_STEP: 219300 | > loss_text_ce: 0.03159128874540329 (0.04577709140338943) | > loss_mel_ce: 2.111279249191284 (nan) | > loss: 0.008503454737365246 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.1365106198113301) | > loader_time: 0.0045 (0.0046452555619233725)  --> TIME: 2025-02-24 07:15:49 -- STEP: 34660/36938 -- GLOBAL_STEP: 219350 | > loss_text_ce: 0.049611929804086685 (0.04577681080059879) | > loss_mel_ce: 3.722252130508423 (nan) | > loss: 0.014967716298997402 (nan) | > current_lr: 5e-06 | > step_time: 0.1526 (0.1365085067462883) | > loader_time: 0.0047 (0.004645288417054514)  --> TIME: 2025-02-24 07:15:58 -- STEP: 34710/36938 -- GLOBAL_STEP: 219400 | > loss_text_ce: 0.03569727763533592 (0.04577660729279003) | > loss_mel_ce: 2.4345686435699463 (nan) | > loss: 0.009802643209695816 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1365067243850822) | > loader_time: 0.0046 (0.004645328575305695)  --> TIME: 2025-02-24 07:16:08 -- STEP: 34760/36938 -- GLOBAL_STEP: 219450 | > loss_text_ce: 0.04173245653510094 (0.045774398041354435) | > loss_mel_ce: 3.6543514728546143 (nan) | > loss: 0.014667000621557236 (nan) | > current_lr: 5e-06 | > step_time: 0.1457 (0.13651117528815712) | > loader_time: 0.0046 (0.004645369557708781)  --> TIME: 2025-02-24 07:16:17 -- STEP: 34810/36938 -- GLOBAL_STEP: 219500 | > loss_text_ce: 0.033190153539180756 (0.0457725834483226) | > loss_mel_ce: 4.403702259063721 (nan) | > loss: 0.017606716603040695 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1365134188787121) | > loader_time: 0.0045 (0.004645483064363797)  --> TIME: 2025-02-24 07:16:27 -- STEP: 34860/36938 -- GLOBAL_STEP: 219550 | > loss_text_ce: 0.04994624853134155 (0.045770738538481945) | > loss_mel_ce: 3.9889349937438965 (nan) | > loss: 0.016027307137846947 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.1365133091354278) | > loader_time: 0.0046 (0.004645514912257819)  --> TIME: 2025-02-24 07:16:36 -- STEP: 34910/36938 -- GLOBAL_STEP: 219600 | > loss_text_ce: 0.05056378245353699 (0.04577174863247981) | > loss_mel_ce: 3.6873931884765625 (nan) | > loss: 0.014833163470029831 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.13651247607777406) | > loader_time: 0.005 (0.004645508956308153)  --> TIME: 2025-02-24 07:16:45 -- STEP: 34960/36938 -- GLOBAL_STEP: 219650 | > loss_text_ce: 0.06727050989866257 (0.04577074863793966) | > loss_mel_ce: 1.871399998664856 (nan) | > loss: 0.0076931375078856945 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13651156133590805) | > loader_time: 0.0045 (0.004645549146206996)  --> TIME: 2025-02-24 07:16:54 -- STEP: 35010/36938 -- GLOBAL_STEP: 219700 | > loss_text_ce: 0.04210525378584862 (0.04577042896843773) | > loss_mel_ce: 2.371811628341675 (nan) | > loss: 0.009579035453498363 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.13651293726929153) | > loader_time: 0.0046 (0.004645548252268031)  --> TIME: 2025-02-24 07:17:04 -- STEP: 35060/36938 -- GLOBAL_STEP: 219750 | > loss_text_ce: 0.04335303232073784 (0.04576915850200378) | > loss_mel_ce: 2.2106516361236572 (nan) | > loss: 0.008944463916122913 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.1365209828902859) | > loader_time: 0.0047 (0.004645618492035037)  --> TIME: 2025-02-24 07:17:13 -- STEP: 35110/36938 -- GLOBAL_STEP: 219800 | > loss_text_ce: 0.0481349378824234 (0.04577049230795972) | > loss_mel_ce: 3.049875020980835 (nan) | > loss: 0.012293691746890545 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13651544709587438) | > loader_time: 0.0045 (0.004645618934707372)  --> TIME: 2025-02-24 07:17:22 -- STEP: 35160/36938 -- GLOBAL_STEP: 219850 | > loss_text_ce: 0.038504429161548615 (0.045766455414535626) | > loss_mel_ce: 3.7508559226989746 (nan) | > loss: 0.015037144534289837 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13650851496521973) | > loader_time: 0.0048 (0.004645597019293172)  --> TIME: 2025-02-24 07:17:32 -- STEP: 35210/36938 -- GLOBAL_STEP: 219900 | > loss_text_ce: 0.05719386413693428 (0.04576329073578546) | > loss_mel_ce: 3.3060359954833984 (nan) | > loss: 0.013346150517463684 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.1365089751346262) | > loader_time: 0.0045 (0.004645630406642668)  --> TIME: 2025-02-24 07:17:41 -- STEP: 35260/36938 -- GLOBAL_STEP: 219950 | > loss_text_ce: 0.043182387948036194 (0.04576030608926407) | > loss_mel_ce: 3.4334733486175537 (nan) | > loss: 0.013796254061162472 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.13651416858893975) | > loader_time: 0.0048 (0.004645629113057193)  --> TIME: 2025-02-24 07:17:51 -- STEP: 35310/36938 -- GLOBAL_STEP: 220000 | > loss_text_ce: 0.03917964920401573 (0.045756867678329254) | > loss_mel_ce: 3.1400303840637207 (nan) | > loss: 0.012615913525223732 (nan) | > current_lr: 5e-06 | > step_time: 0.1446 (0.13652119226004805) | > loader_time: 0.0047 (0.004645685412261854) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_220000.pth  --> TIME: 2025-02-24 07:18:06 -- STEP: 35360/36938 -- GLOBAL_STEP: 220050 | > loss_text_ce: 0.054223764687776566 (0.04575615747232847) | > loss_mel_ce: 3.9712531566619873 (nan) | > loss: 0.01597411558032036 (nan) | > current_lr: 5e-06 | > step_time: 0.1479 (0.13651578421641428) | > loader_time: 0.0049 (0.004645716833852501)  --> TIME: 2025-02-24 07:18:15 -- STEP: 35410/36938 -- GLOBAL_STEP: 220100 | > loss_text_ce: 0.04407111927866936 (0.045753479778605846) | > loss_mel_ce: 0.15159280598163605 (nan) | > loss: 0.0007764442125335336 (nan) | > current_lr: 5e-06 | > step_time: 0.1264 (0.1365127561254661) | > loader_time: 0.0046 (0.004645757236173655)  --> TIME: 2025-02-24 07:18:24 -- STEP: 35460/36938 -- GLOBAL_STEP: 220150 | > loss_text_ce: 0.05752179026603699 (0.045753327823752195) | > loss_mel_ce: 2.7739875316619873 (nan) | > loss: 0.01123614888638258 (nan) | > current_lr: 5e-06 | > step_time: 0.107 (0.13650264364141607) | > loader_time: 0.0044 (0.004645696065550902)  --> TIME: 2025-02-24 07:18:33 -- STEP: 35510/36938 -- GLOBAL_STEP: 220200 | > loss_text_ce: 0.04853564128279686 (0.04575311804528104) | > loss_mel_ce: 3.8479909896850586 (nan) | > loss: 0.01546240784227848 (nan) | > current_lr: 5e-06 | > step_time: 0.1355 (0.13650598650883877) | > loader_time: 0.0046 (0.00464571765092886)  --> TIME: 2025-02-24 07:18:43 -- STEP: 35560/36938 -- GLOBAL_STEP: 220250 | > loss_text_ce: 0.04704634100198746 (0.045752558337092075) | > loss_mel_ce: 2.745415210723877 (nan) | > loss: 0.011081197299063206 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13650502142139345) | > loader_time: 0.0047 (0.004645813463509257)  --> TIME: 2025-02-24 07:18:52 -- STEP: 35610/36938 -- GLOBAL_STEP: 220300 | > loss_text_ce: 0.034947119653224945 (0.04575152108226095) | > loss_mel_ce: 3.407489061355591 (nan) | > loss: 0.013660461641848087 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13650882652521773) | > loader_time: 0.0045 (0.0046458294270447405)  --> TIME: 2025-02-24 07:19:02 -- STEP: 35660/36938 -- GLOBAL_STEP: 220350 | > loss_text_ce: 0.04045550152659416 (0.04574932687873049) | > loss_mel_ce: 4.281005859375 (nan) | > loss: 0.017148656770586967 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13651517823514192) | > loader_time: 0.0046 (0.0046458501596498645)  --> TIME: 2025-02-24 07:19:11 -- STEP: 35710/36938 -- GLOBAL_STEP: 220400 | > loss_text_ce: 0.05447943136096001 (0.045752523668518445) | > loss_mel_ce: 3.6460812091827393 (nan) | > loss: 0.014684764668345451 (nan) | > current_lr: 5e-06 | > step_time: 0.156 (0.13651348425540336) | > loader_time: 0.0047 (0.004645809577112907)  --> TIME: 2025-02-24 07:19:21 -- STEP: 35760/36938 -- GLOBAL_STEP: 220450 | > loss_text_ce: 0.0400344617664814 (0.0457481793913694) | > loss_mel_ce: 3.789858102798462 (nan) | > loss: 0.015197987668216228 (nan) | > current_lr: 5e-06 | > step_time: 0.2003 (0.1365237528579099) | > loader_time: 0.0049 (0.004645828839383103)  --> TIME: 2025-02-24 07:19:30 -- STEP: 35810/36938 -- GLOBAL_STEP: 220500 | > loss_text_ce: 0.04706390202045441 (0.04574752234937305) | > loss_mel_ce: 3.135608196258545 (nan) | > loss: 0.012629651464521885 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13652333521503585) | > loader_time: 0.0048 (0.0046458540199788645)  --> TIME: 2025-02-24 07:19:39 -- STEP: 35860/36938 -- GLOBAL_STEP: 220550 | > loss_text_ce: 0.06182677671313286 (0.045747421482812876) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13652928404747056) | > loader_time: 0.0047 (0.004645862934380589)  --> TIME: 2025-02-24 07:19:49 -- STEP: 35910/36938 -- GLOBAL_STEP: 220600 | > loss_text_ce: 0.04213036596775055 (0.04574735416498845) | > loss_mel_ce: 3.4971773624420166 (nan) | > loss: 0.014044873416423798 (nan) | > current_lr: 5e-06 | > step_time: 0.1909 (0.13653499787978002) | > loader_time: 0.0048 (0.004645862767902835)  --> TIME: 2025-02-24 07:19:58 -- STEP: 35960/36938 -- GLOBAL_STEP: 220650 | > loss_text_ce: 0.04858330264687538 (0.04574164661505817) | > loss_mel_ce: 3.4754889011383057 (nan) | > loss: 0.01398441381752491 (nan) | > current_lr: 5e-06 | > step_time: 0.1723 (0.13653953656869322) | > loader_time: 0.005 (0.004645883765183485)  --> TIME: 2025-02-24 07:20:08 -- STEP: 36010/36938 -- GLOBAL_STEP: 220700 | > loss_text_ce: 0.04186522960662842 (0.045739329003818836) | > loss_mel_ce: 2.89316987991333 (nan) | > loss: 0.011646965518593788 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.136542743426237) | > loader_time: 0.0046 (0.004645906121026976)  --> TIME: 2025-02-24 07:20:17 -- STEP: 36060/36938 -- GLOBAL_STEP: 220750 | > loss_text_ce: 0.0505714975297451 (0.04573885991136833) | > loss_mel_ce: 3.2174808979034424 (nan) | > loss: 0.012968461960554123 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13653965549077762) | > loader_time: 0.0046 (0.004645880638594346)  --> TIME: 2025-02-24 07:20:27 -- STEP: 36110/36938 -- GLOBAL_STEP: 220800 | > loss_text_ce: 0.03254522755742073 (0.045736417350519615) | > loss_mel_ce: 2.999894380569458 (nan) | > loss: 0.012033491395413876 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13654383721030922) | > loader_time: 0.0047 (0.004645885598524138)  --> TIME: 2025-02-24 07:20:36 -- STEP: 36160/36938 -- GLOBAL_STEP: 220850 | > loss_text_ce: 0.04229990765452385 (0.04573278495651973) | > loss_mel_ce: 3.151432991027832 (nan) | > loss: 0.012673544697463512 (nan) | > current_lr: 5e-06 | > step_time: 0.1513 (0.13654487968414405) | > loader_time: 0.0047 (0.004645909922840299)  --> TIME: 2025-02-24 07:20:46 -- STEP: 36210/36938 -- GLOBAL_STEP: 220900 | > loss_text_ce: 0.03546043485403061 (0.04572997701317564) | > loss_mel_ce: 3.2028167247772217 (nan) | > loss: 0.012850306928157806 (nan) | > current_lr: 5e-06 | > step_time: 0.1444 (0.13655413170277145) | > loader_time: 0.0047 (0.004645955664648643)  --> TIME: 2025-02-24 07:20:55 -- STEP: 36260/36938 -- GLOBAL_STEP: 220950 | > loss_text_ce: 0.03765507787466049 (0.04572913179030766) | > loss_mel_ce: 3.5415456295013428 (nan) | > loss: 0.014203178696334362 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13655453537684978) | > loader_time: 0.0049 (0.004645963636999008)  --> TIME: 2025-02-24 07:21:05 -- STEP: 36310/36938 -- GLOBAL_STEP: 221000 | > loss_text_ce: 0.051536764949560165 (0.04572717044207551) | > loss_mel_ce: 3.3813068866729736 (nan) | > loss: 0.013622396625578403 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.13655993190556484) | > loader_time: 0.0049 (0.004645966807202292)  --> TIME: 2025-02-24 07:21:14 -- STEP: 36360/36938 -- GLOBAL_STEP: 221050 | > loss_text_ce: 0.041309770196676254 (0.04572682233465339) | > loss_mel_ce: 2.788111686706543 (nan) | > loss: 0.0112278638407588 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13655139262920335) | > loader_time: 0.0046 (0.004645947005489062)  --> TIME: 2025-02-24 07:21:23 -- STEP: 36410/36938 -- GLOBAL_STEP: 221100 | > loss_text_ce: 0.05009860545396805 (0.04572278276979595) | > loss_mel_ce: 3.0973448753356934 (nan) | > loss: 0.012489856220781803 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13655537043715432) | > loader_time: 0.0046 (0.0046459558081476055)  --> TIME: 2025-02-24 07:21:32 -- STEP: 36460/36938 -- GLOBAL_STEP: 221150 | > loss_text_ce: 0.020019210875034332 (0.045722564493775694) | > loss_mel_ce: 2.860102891921997 (nan) | > loss: 0.011429056525230408 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13655278877576535) | > loader_time: 0.0046 (0.00464593484646266)  --> TIME: 2025-02-24 07:21:41 -- STEP: 36510/36938 -- GLOBAL_STEP: 221200 | > loss_text_ce: 0.050342220813035965 (0.045721939400831454) | > loss_mel_ce: 1.32342529296875 (nan) | > loss: 0.005451458506286144 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13654403644077143) | > loader_time: 0.0048 (0.004645913458954479)  --> TIME: 2025-02-24 07:21:51 -- STEP: 36560/36938 -- GLOBAL_STEP: 221250 | > loss_text_ce: 0.031600404530763626 (0.045720310943253344) | > loss_mel_ce: 3.059185266494751 (nan) | > loss: 0.012265023775398731 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13654853688549082) | > loader_time: 0.0045 (0.004645966192303892)  --> TIME: 2025-02-24 07:22:00 -- STEP: 36610/36938 -- GLOBAL_STEP: 221300 | > loss_text_ce: 0.0577634759247303 (0.04572169772343974) | > loss_mel_ce: 3.8538239002227783 (nan) | > loss: 0.01552217360585928 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.1365506409752162) | > loader_time: 0.0045 (0.00464598452644377)  --> TIME: 2025-02-24 07:22:10 -- STEP: 36660/36938 -- GLOBAL_STEP: 221350 | > loss_text_ce: 0.03932701423764229 (0.04572085145877697) | > loss_mel_ce: 2.86212420463562 (nan) | > loss: 0.011513695120811462 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13655499570433638) | > loader_time: 0.0045 (0.004645926374850584)  --> TIME: 2025-02-24 07:22:19 -- STEP: 36710/36938 -- GLOBAL_STEP: 221400 | > loss_text_ce: 0.03755688667297363 (0.04572270896772384) | > loss_mel_ce: 4.1083502769470215 (nan) | > loss: 0.016452014446258545 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.1365529783932677) | > loader_time: 0.0047 (0.004645843507158281)  --> TIME: 2025-02-24 07:22:28 -- STEP: 36760/36938 -- GLOBAL_STEP: 221450 | > loss_text_ce: 0.04618026316165924 (0.045718990796817405) | > loss_mel_ce: 3.3316476345062256 (nan) | > loss: 0.013404079712927341 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.1365553535172436) | > loader_time: 0.0046 (0.004645825230128333)  --> TIME: 2025-02-24 07:22:37 -- STEP: 36810/36938 -- GLOBAL_STEP: 221500 | > loss_text_ce: 0.04738302901387215 (0.04571708606002497) | > loss_mel_ce: 3.586287260055542 (nan) | > loss: 0.01441932749003172 (nan) | > current_lr: 5e-06 | > step_time: 0.1662 (0.13655186933980354) | > loader_time: 0.0046 (0.0046457770271684466)  --> TIME: 2025-02-24 07:22:46 -- STEP: 36860/36938 -- GLOBAL_STEP: 221550 | > loss_text_ce: 0.03641660511493683 (0.045716621365061834) | > loss_mel_ce: 3.162489175796509 (nan) | > loss: 0.012694071047008038 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1365428105877994) | > loader_time: 0.0045 (0.004645776619292812)  --> TIME: 2025-02-24 07:22:55 -- STEP: 36910/36938 -- GLOBAL_STEP: 221600 | > loss_text_ce: 0.048573654145002365 (0.045717373537694714) | > loss_mel_ce: 3.3696188926696777 (nan) | > loss: 0.013564256951212883 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1365431176262881) | > loader_time: 0.0045 (0.004645796133489391)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.02072063176938803 (-0.00019225271621551437) | > avg_loss_text_ce: 0.04341344736368939 (-0.0018512269131618075) | > avg_loss_mel_ce: 2.9513396715173648 (-0.08275836529117786) | > avg_loss: 2.9947531158381158 (-0.08460960175731502) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_221628.pth  > EPOCH: 7/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-24 07:23:25)   --> TIME: 2025-02-24 07:23:29 -- STEP: 22/36938 -- GLOBAL_STEP: 221650 | > loss_text_ce: 0.04635746031999588 (0.04495734721422195) | > loss_mel_ce: 2.904710292816162 (3.076383314349435) | > loss: 0.011710586957633495 (0.01238627310059118) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13109085776589133) | > loader_time: 0.0048 (0.004880580035122958)  --> TIME: 2025-02-24 07:23:39 -- STEP: 72/36938 -- GLOBAL_STEP: 221700 | > loss_text_ce: 0.06153290718793869 (0.044548732694238424) | > loss_mel_ce: 2.9518182277679443 (3.056031490624365) | > loss: 0.0119577432051301 (0.012303890446572848) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13386779692437914) | > loader_time: 0.0046 (0.004926535818311904)  --> TIME: 2025-02-24 07:23:48 -- STEP: 122/36938 -- GLOBAL_STEP: 221750 | > loss_text_ce: 0.048672426491975784 (0.04540531092979869) | > loss_mel_ce: 2.8070130348205566 (3.062309199211294) | > loss: 0.01133208628743887 (0.01233220114740047) | > current_lr: 5e-06 | > step_time: 0.1348 (0.13372362441703922) | > loader_time: 0.0046 (0.0048972075102759185)  --> TIME: 2025-02-24 07:23:57 -- STEP: 172/36938 -- GLOBAL_STEP: 221800 | > loss_text_ce: 0.04866533353924751 (0.04530076892656644) | > loss_mel_ce: 3.687727451324463 (3.0587363619043297) | > loss: 0.014826956205070019 (0.012317608372794962) | > current_lr: 5e-06 | > step_time: 0.118 (0.13460596079050108) | > loader_time: 0.0046 (0.0048692392748455665)  --> TIME: 2025-02-24 07:24:06 -- STEP: 222/36938 -- GLOBAL_STEP: 221850 | > loss_text_ce: 0.033321913331747055 (0.04542931849724271) | > loss_mel_ce: 3.1789984703063965 (nan) | > loss: 0.012747303582727909 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13465703070700707) | > loader_time: 0.0046 (0.004841037698694176)  --> TIME: 2025-02-24 07:24:16 -- STEP: 272/36938 -- GLOBAL_STEP: 221900 | > loss_text_ce: 0.047306206077337265 (0.045605093192802204) | > loss_mel_ce: 1.5552291870117188 (nan) | > loss: 0.0063592675141990185 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.1349008425193675) | > loader_time: 0.0048 (0.004833458101048188)  --> TIME: 2025-02-24 07:24:25 -- STEP: 322/36938 -- GLOBAL_STEP: 221950 | > loss_text_ce: 0.04667072743177414 (0.045431117214817816) | > loss_mel_ce: 3.1578691005706787 (nan) | > loss: 0.012716428376734257 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13490750848876765) | > loader_time: 0.0046 (0.004812809991540373)  --> TIME: 2025-02-24 07:24:34 -- STEP: 372/36938 -- GLOBAL_STEP: 222000 | > loss_text_ce: 0.034759145230054855 (0.04542189437936072) | > loss_mel_ce: 2.93285870552063 (nan) | > loss: 0.01177626196295023 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13508281271944766) | > loader_time: 0.0046 (0.004794969353624566)  --> TIME: 2025-02-24 07:24:43 -- STEP: 422/36938 -- GLOBAL_STEP: 222050 | > loss_text_ce: 0.045470986515283585 (0.045225036043620805) | > loss_mel_ce: 3.7350966930389404 (nan) | > loss: 0.015002253465354443 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.1350520758832234) | > loader_time: 0.0048 (0.004777360866420075)  --> TIME: 2025-02-24 07:24:53 -- STEP: 472/36938 -- GLOBAL_STEP: 222100 | > loss_text_ce: 0.04666897654533386 (0.04518298859613302) | > loss_mel_ce: 3.2638020515441895 (nan) | > loss: 0.013136791065335274 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13512221534373378) | > loader_time: 0.0046 (0.004761016975014895)  --> TIME: 2025-02-24 07:25:02 -- STEP: 522/36938 -- GLOBAL_STEP: 222150 | > loss_text_ce: 0.037002287805080414 (0.045137263110114484) | > loss_mel_ce: 3.342240571975708 (nan) | > loss: 0.013409694656729698 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13568580561670748) | > loader_time: 0.0046 (0.004750016091883866)  --> TIME: 2025-02-24 07:25:12 -- STEP: 572/36938 -- GLOBAL_STEP: 222200 | > loss_text_ce: 0.04084155336022377 (0.04497808505214389) | > loss_mel_ce: 3.3653533458709717 (nan) | > loss: 0.01351664774119854 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1362217677223097) | > loader_time: 0.0046 (0.004743077121414503)  --> TIME: 2025-02-24 07:25:21 -- STEP: 622/36938 -- GLOBAL_STEP: 222250 | > loss_text_ce: 0.05147518590092659 (0.04499308697555897) | > loss_mel_ce: 0.74444580078125 (nan) | > loss: 0.0031584168318659067 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13619158812274482) | > loader_time: 0.0046 (0.004734920918749841)  --> TIME: 2025-02-24 07:25:30 -- STEP: 672/36938 -- GLOBAL_STEP: 222300 | > loss_text_ce: 0.039439741522073746 (0.04509913548827172) | > loss_mel_ce: 3.2979540824890137 (nan) | > loss: 0.013243626803159714 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.1361683177806081) | > loader_time: 0.0047 (0.004745607219991228)  --> TIME: 2025-02-24 07:25:40 -- STEP: 722/36938 -- GLOBAL_STEP: 222350 | > loss_text_ce: 0.03271975368261337 (0.045066407257355647) | > loss_mel_ce: 2.2527661323547363 (nan) | > loss: 0.009069389663636684 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13633491391950683) | > loader_time: 0.0049 (0.004742978682478378)  --> TIME: 2025-02-24 07:25:49 -- STEP: 772/36938 -- GLOBAL_STEP: 222400 | > loss_text_ce: 0.029689664021134377 (0.04501015518265993) | > loss_mel_ce: 0.4516037702560425 (nan) | > loss: 0.0019098947523161769 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13597891954560357) | > loader_time: 0.0047 (0.004735209163606474)  --> TIME: 2025-02-24 07:25:58 -- STEP: 822/36938 -- GLOBAL_STEP: 222450 | > loss_text_ce: 0.03339080512523651 (0.04504134529100282) | > loss_mel_ce: 0.5518689751625061 (nan) | > loss: 0.0023224595934152603 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13596373810965362) | > loader_time: 0.0046 (0.004729117790277859)  --> TIME: 2025-02-24 07:26:07 -- STEP: 872/36938 -- GLOBAL_STEP: 222500 | > loss_text_ce: 0.038490429520606995 (0.04504903242068895) | > loss_mel_ce: 2.249931812286377 (nan) | > loss: 0.009081041440367699 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13596995571337703) | > loader_time: 0.0054 (0.004726305467273121)  --> TIME: 2025-02-24 07:26:17 -- STEP: 922/36938 -- GLOBAL_STEP: 222550 | > loss_text_ce: 0.043195392936468124 (0.04506010346219687) | > loss_mel_ce: 3.6525275707244873 (nan) | > loss: 0.014665568247437477 (nan) | > current_lr: 5e-06 | > step_time: 0.1718 (0.13624512244201786) | > loader_time: 0.0048 (0.004726264843971769)  --> TIME: 2025-02-24 07:26:26 -- STEP: 972/36938 -- GLOBAL_STEP: 222600 | > loss_text_ce: 0.03321482986211777 (0.045182792740057647) | > loss_mel_ce: 3.1545329093933105 (nan) | > loss: 0.012649793177843094 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.1363776203536201) | > loader_time: 0.0047 (0.004730634728576915)  --> TIME: 2025-02-24 07:26:36 -- STEP: 1022/36938 -- GLOBAL_STEP: 222650 | > loss_text_ce: 0.037800729274749756 (0.045197431617753615) | > loss_mel_ce: 2.159392833709717 (nan) | > loss: 0.008719022385776043 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13625179788137823) | > loader_time: 0.0046 (0.004732909501182127)  --> TIME: 2025-02-24 07:26:45 -- STEP: 1072/36938 -- GLOBAL_STEP: 222700 | > loss_text_ce: 0.03585098311305046 (0.045237289289874374) | > loss_mel_ce: 3.0292434692382812 (nan) | > loss: 0.012163073755800724 (nan) | > current_lr: 5e-06 | > step_time: 0.1769 (0.1362394528157675) | > loader_time: 0.0049 (0.004736351877895758)  --> TIME: 2025-02-24 07:26:54 -- STEP: 1122/36938 -- GLOBAL_STEP: 222750 | > loss_text_ce: 0.05394411459565163 (0.04529421069325611) | > loss_mel_ce: 3.15875244140625 (nan) | > loss: 0.012748796492815018 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.1362855313295986) | > loader_time: 0.0045 (0.004736363356551177)  --> TIME: 2025-02-24 07:27:04 -- STEP: 1172/36938 -- GLOBAL_STEP: 222800 | > loss_text_ce: 0.05273601785302162 (0.04527168345253015) | > loss_mel_ce: 3.5597662925720215 (nan) | > loss: 0.01433532778173685 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13648316620152956) | > loader_time: 0.0048 (0.004738702505521814)  --> TIME: 2025-02-24 07:27:13 -- STEP: 1222/36938 -- GLOBAL_STEP: 222850 | > loss_text_ce: 0.052133508026599884 (0.04522301897760207) | > loss_mel_ce: 3.5744123458862305 (nan) | > loss: 0.014391056261956692 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.1364949260717913) | > loader_time: 0.0045 (0.004741353216030797)  --> TIME: 2025-02-24 07:27:22 -- STEP: 1272/36938 -- GLOBAL_STEP: 222900 | > loss_text_ce: 0.049267545342445374 (0.045301044595562515) | > loss_mel_ce: 2.5987131595611572 (nan) | > loss: 0.010507860220968723 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13636997210904486) | > loader_time: 0.0046 (0.004742702785527939)  --> TIME: 2025-02-24 07:27:31 -- STEP: 1322/36938 -- GLOBAL_STEP: 222950 | > loss_text_ce: 0.05155494436621666 (0.04530271408873723) | > loss_mel_ce: 3.62890625 (nan) | > loss: 0.014605005271732807 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.13626085575698174) | > loader_time: 0.0049 (0.004742889649568644)  --> TIME: 2025-02-24 07:27:41 -- STEP: 1372/36938 -- GLOBAL_STEP: 223000 | > loss_text_ce: 0.03457392007112503 (0.04528467537585438) | > loss_mel_ce: 3.095486640930176 (nan) | > loss: 0.012420875951647758 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13619865724713728) | > loader_time: 0.0047 (0.004744337008228807)  --> TIME: 2025-02-24 07:27:50 -- STEP: 1422/36938 -- GLOBAL_STEP: 223050 | > loss_text_ce: 0.04207232967019081 (0.04525571371233495) | > loss_mel_ce: 3.0387685298919678 (nan) | > loss: 0.012225559912621975 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13627441573243607) | > loader_time: 0.0046 (0.0047434343735041715)  --> TIME: 2025-02-24 07:27:59 -- STEP: 1472/36938 -- GLOBAL_STEP: 223100 | > loss_text_ce: 0.03328540921211243 (0.045232038324921495) | > loss_mel_ce: 2.573204278945923 (nan) | > loss: 0.010343213565647602 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13636616747016492) | > loader_time: 0.0047 (0.004742691860250806)  --> TIME: 2025-02-24 07:28:09 -- STEP: 1522/36938 -- GLOBAL_STEP: 223150 | > loss_text_ce: 0.05106515437364578 (0.0452509754252927) | > loss_mel_ce: 3.3060357570648193 (nan) | > loss: 0.013321829959750175 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13646645003956653) | > loader_time: 0.0046 (0.004740289257014471)  --> TIME: 2025-02-24 07:28:18 -- STEP: 1572/36938 -- GLOBAL_STEP: 223200 | > loss_text_ce: 0.04851128160953522 (0.04526403088018334) | > loss_mel_ce: 3.258305788040161 (nan) | > loss: 0.01312229037284851 (nan) | > current_lr: 5e-06 | > step_time: 0.1769 (0.1364754703815353) | > loader_time: 0.0049 (0.004740162811813162)  --> TIME: 2025-02-24 07:28:28 -- STEP: 1622/36938 -- GLOBAL_STEP: 223250 | > loss_text_ce: 0.040180109441280365 (0.04523591617404788) | > loss_mel_ce: 3.6178948879241943 (nan) | > loss: 0.014516171999275684 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13641150400465135) | > loader_time: 0.0047 (0.004739663309880456)  --> TIME: 2025-02-24 07:28:37 -- STEP: 1672/36938 -- GLOBAL_STEP: 223300 | > loss_text_ce: 0.04435357823967934 (0.045273917309414026) | > loss_mel_ce: 3.2120347023010254 (nan) | > loss: 0.012922176159918308 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13629692678816588) | > loader_time: 0.0045 (0.004738594784120623)  --> TIME: 2025-02-24 07:28:46 -- STEP: 1722/36938 -- GLOBAL_STEP: 223350 | > loss_text_ce: 0.03855571523308754 (0.04529173710382391) | > loss_mel_ce: 3.464385509490967 (nan) | > loss: 0.01390056125819683 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.1362152980179737) | > loader_time: 0.0047 (0.0047366880789035705)  --> TIME: 2025-02-24 07:28:55 -- STEP: 1772/36938 -- GLOBAL_STEP: 223400 | > loss_text_ce: 0.03918038681149483 (0.04526706844564046) | > loss_mel_ce: 3.6178112030029297 (nan) | > loss: 0.014511872082948685 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.13621710076019952) | > loader_time: 0.0047 (0.004735356932569035)  --> TIME: 2025-02-24 07:29:05 -- STEP: 1822/36938 -- GLOBAL_STEP: 223450 | > loss_text_ce: 0.043188709765672684 (0.0452421049390123) | > loss_mel_ce: 3.68738055229187 (nan) | > loss: 0.014803847298026085 (nan) | > current_lr: 5e-06 | > step_time: 0.1899 (0.13624834595605917) | > loader_time: 0.0048 (0.004733485919060482)  --> TIME: 2025-02-24 07:29:14 -- STEP: 1872/36938 -- GLOBAL_STEP: 223500 | > loss_text_ce: 0.0529504232108593 (0.045224790831502416) | > loss_mel_ce: 3.173341751098633 (nan) | > loss: 0.012802747078239918 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13617202996188768) | > loader_time: 0.0045 (0.004731379513047699)  --> TIME: 2025-02-24 07:29:23 -- STEP: 1922/36938 -- GLOBAL_STEP: 223550 | > loss_text_ce: 0.04246438294649124 (0.0451466742123823) | > loss_mel_ce: 3.7065649032592773 (nan) | > loss: 0.01487710140645504 (nan) | > current_lr: 5e-06 | > step_time: 0.1825 (0.13620051806724795) | > loader_time: 0.0049 (0.004731516064019655)  --> TIME: 2025-02-24 07:29:32 -- STEP: 1972/36938 -- GLOBAL_STEP: 223600 | > loss_text_ce: 0.038743022829294205 (0.045103313123644836) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13624419281013614) | > loader_time: 0.0046 (0.00473048346037797)  --> TIME: 2025-02-24 07:29:42 -- STEP: 2022/36938 -- GLOBAL_STEP: 223650 | > loss_text_ce: 0.049404535442590714 (0.0451422290596466) | > loss_mel_ce: 3.1847128868103027 (nan) | > loss: 0.012833800166845322 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.1361679790281756) | > loader_time: 0.0046 (0.004728574545282049)  --> TIME: 2025-02-24 07:29:51 -- STEP: 2072/36938 -- GLOBAL_STEP: 223700 | > loss_text_ce: 0.06066140905022621 (0.04512870148575816) | > loss_mel_ce: 0.2734375 (nan) | > loss: 0.0013257893733680248 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1360642104535491) | > loader_time: 0.0048 (0.0047268589030821815)  --> TIME: 2025-02-24 07:30:00 -- STEP: 2122/36938 -- GLOBAL_STEP: 223750 | > loss_text_ce: 0.05231882631778717 (0.045118681421111924) | > loss_mel_ce: 3.5672354698181152 (nan) | > loss: 0.014363311231136322 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13602137498199451) | > loader_time: 0.0046 (0.004725823191176261)  --> TIME: 2025-02-24 07:30:09 -- STEP: 2172/36938 -- GLOBAL_STEP: 223800 | > loss_text_ce: 0.04942180961370468 (0.04512222107108783) | > loss_mel_ce: 2.014082908630371 (nan) | > loss: 0.008188511244952679 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13600429074759873) | > loader_time: 0.0045 (0.0047246165038472366)  --> TIME: 2025-02-24 07:30:18 -- STEP: 2222/36938 -- GLOBAL_STEP: 223850 | > loss_text_ce: 0.04835642874240875 (0.045067782400630245) | > loss_mel_ce: 3.2473654747009277 (nan) | > loss: 0.013078263029456139 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13601418436378326) | > loader_time: 0.0045 (0.004722338018923807)  --> TIME: 2025-02-24 07:30:28 -- STEP: 2272/36938 -- GLOBAL_STEP: 223900 | > loss_text_ce: 0.046699341386556625 (0.045032526702541355) | > loss_mel_ce: 3.4108948707580566 (nan) | > loss: 0.013720612972974777 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.1359891568271208) | > loader_time: 0.0046 (0.00472022414627209)  --> TIME: 2025-02-24 07:30:37 -- STEP: 2322/36938 -- GLOBAL_STEP: 223950 | > loss_text_ce: 0.03849257901310921 (0.04501820144281237) | > loss_mel_ce: 2.6052770614624023 (nan) | > loss: 0.010491150431334972 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13603101610418647) | > loader_time: 0.0046 (0.004719102002750891)  --> TIME: 2025-02-24 07:30:46 -- STEP: 2372/36938 -- GLOBAL_STEP: 224000 | > loss_text_ce: 0.04301297664642334 (0.04496517663815293) | > loss_mel_ce: 2.976978302001953 (nan) | > loss: 0.011984093114733696 (nan) | > current_lr: 5e-06 | > step_time: 0.1704 (0.1359850232556941) | > loader_time: 0.0048 (0.004717483890398358)  --> TIME: 2025-02-24 07:30:55 -- STEP: 2422/36938 -- GLOBAL_STEP: 224050 | > loss_text_ce: 0.04812246561050415 (0.044991865205023976) | > loss_mel_ce: 3.693532705307007 (nan) | > loss: 0.014847838319838047 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13589864314636071) | > loader_time: 0.0045 (0.004716442302669207)  --> TIME: 2025-02-24 07:31:05 -- STEP: 2472/36938 -- GLOBAL_STEP: 224100 | > loss_text_ce: 0.036702368408441544 (0.04495943343857583) | > loss_mel_ce: 3.5578742027282715 (nan) | > loss: 0.01426419336348772 (nan) | > current_lr: 5e-06 | > step_time: 0.1874 (0.1359822085567279) | > loader_time: 0.0049 (0.004715354022084699)  --> TIME: 2025-02-24 07:31:14 -- STEP: 2522/36938 -- GLOBAL_STEP: 224150 | > loss_text_ce: 0.0419364869594574 (0.04493881638097579) | > loss_mel_ce: 3.7952489852905273 (nan) | > loss: 0.015226926654577255 (nan) | > current_lr: 5e-06 | > step_time: 0.1945 (0.13597761790587948) | > loader_time: 0.0051 (0.004713894354739328)  --> TIME: 2025-02-24 07:31:24 -- STEP: 2572/36938 -- GLOBAL_STEP: 224200 | > loss_text_ce: 0.040708694607019424 (0.04492037522955613) | > loss_mel_ce: 3.3173913955688477 (nan) | > loss: 0.013325795531272888 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13603359377402757) | > loader_time: 0.0046 (0.004716041484850559)  --> TIME: 2025-02-24 07:31:33 -- STEP: 2622/36938 -- GLOBAL_STEP: 224250 | > loss_text_ce: 0.04237411916255951 (0.04490983353687615) | > loss_mel_ce: 3.358419179916382 (nan) | > loss: 0.01349521242082119 (nan) | > current_lr: 5e-06 | > step_time: 0.1621 (0.13604374526567292) | > loader_time: 0.0046 (0.004716052161565177)  --> TIME: 2025-02-24 07:31:43 -- STEP: 2672/36938 -- GLOBAL_STEP: 224300 | > loss_text_ce: 0.04336212947964668 (0.04492564835030762) | > loss_mel_ce: 3.7931671142578125 (nan) | > loss: 0.015224323607981205 (nan) | > current_lr: 5e-06 | > step_time: 0.1457 (0.13616579021522413) | > loader_time: 0.0046 (0.004715512880308183)  --> TIME: 2025-02-24 07:31:52 -- STEP: 2722/36938 -- GLOBAL_STEP: 224350 | > loss_text_ce: 0.044757626950740814 (0.0449051824586519) | > loss_mel_ce: 3.066868305206299 (nan) | > loss: 0.012347722426056862 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13610376924561055) | > loader_time: 0.0046 (0.004714031328093383)  --> TIME: 2025-02-24 07:32:01 -- STEP: 2772/36938 -- GLOBAL_STEP: 224400 | > loss_text_ce: 0.04852670431137085 (0.04489258014762295) | > loss_mel_ce: 3.7021329402923584 (nan) | > loss: 0.014883571304380894 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13613426960563976) | > loader_time: 0.0046 (0.004713248122822158)  --> TIME: 2025-02-24 07:32:11 -- STEP: 2822/36938 -- GLOBAL_STEP: 224450 | > loss_text_ce: 0.05278840288519859 (0.044875923140221256) | > loss_mel_ce: 2.065373659133911 (nan) | > loss: 0.008405406028032303 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13622854931991596) | > loader_time: 0.0046 (0.004712714530490635)  --> TIME: 2025-02-24 07:32:20 -- STEP: 2872/36938 -- GLOBAL_STEP: 224500 | > loss_text_ce: 0.047890834510326385 (0.04485779546115612) | > loss_mel_ce: 3.7837774753570557 (nan) | > loss: 0.01520503405481577 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13624155396870613) | > loader_time: 0.0046 (0.004712093507347002)  --> TIME: 2025-02-24 07:32:29 -- STEP: 2922/36938 -- GLOBAL_STEP: 224550 | > loss_text_ce: 0.04145931079983711 (0.04484762576798162) | > loss_mel_ce: 3.5324604511260986 (nan) | > loss: 0.014182222075760365 (nan) | > current_lr: 5e-06 | > step_time: 0.1855 (0.13623200411016564) | > loader_time: 0.0048 (0.004710773749354769)  --> TIME: 2025-02-24 07:32:38 -- STEP: 2972/36938 -- GLOBAL_STEP: 224600 | > loss_text_ce: 0.05164455249905586 (0.0448764913394535) | > loss_mel_ce: 3.4605140686035156 (nan) | > loss: 0.013937138020992279 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.13625285635084386) | > loader_time: 0.0049 (0.004710300774144295)  --> TIME: 2025-02-24 07:32:48 -- STEP: 3022/36938 -- GLOBAL_STEP: 224650 | > loss_text_ce: 0.03791138157248497 (0.044908062509285036) | > loss_mel_ce: 3.357928514480591 (nan) | > loss: 0.013475555926561356 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13627663722186628) | > loader_time: 0.0046 (0.0047092870873855835)  --> TIME: 2025-02-24 07:32:57 -- STEP: 3072/36938 -- GLOBAL_STEP: 224700 | > loss_text_ce: 0.05131705105304718 (0.04488666402812661) | > loss_mel_ce: 3.3817291259765625 (nan) | > loss: 0.013623199425637722 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13628171229114186) | > loader_time: 0.0046 (0.004708464800690616)  --> TIME: 2025-02-24 07:33:07 -- STEP: 3122/36938 -- GLOBAL_STEP: 224750 | > loss_text_ce: 0.04232149198651314 (0.04487496816620586) | > loss_mel_ce: 2.843900680541992 (nan) | > loss: 0.011453263461589813 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.13627381106357755) | > loader_time: 0.0046 (0.004707600809228037)  --> TIME: 2025-02-24 07:33:16 -- STEP: 3172/36938 -- GLOBAL_STEP: 224800 | > loss_text_ce: 0.04207630082964897 (0.04489290032243351) | > loss_mel_ce: 3.599820852279663 (nan) | > loss: 0.01445197407156229 (nan) | > current_lr: 5e-06 | > step_time: 0.1713 (0.13635748497023756) | > loader_time: 0.0047 (0.004706633045874816)  --> TIME: 2025-02-24 07:33:25 -- STEP: 3222/36938 -- GLOBAL_STEP: 224850 | > loss_text_ce: 0.04931734502315521 (0.044891490447516974) | > loss_mel_ce: 2.7313520908355713 (nan) | > loss: 0.011034402996301651 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13633811185257882) | > loader_time: 0.0048 (0.004705630127204133)  --> TIME: 2025-02-24 07:33:35 -- STEP: 3272/36938 -- GLOBAL_STEP: 224900 | > loss_text_ce: 0.04505071043968201 (0.044824434823156116) | > loss_mel_ce: 2.989732503890991 (nan) | > loss: 0.012042790651321411 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13628101188571107) | > loader_time: 0.0046 (0.004704513584780531)  --> TIME: 2025-02-24 07:33:44 -- STEP: 3322/36938 -- GLOBAL_STEP: 224950 | > loss_text_ce: 0.048740461468696594 (0.04482570867721951) | > loss_mel_ce: 4.087745666503906 (nan) | > loss: 0.01641462743282318 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.136362855151646) | > loader_time: 0.0048 (0.004703999211600154)  --> TIME: 2025-02-24 07:33:53 -- STEP: 3372/36938 -- GLOBAL_STEP: 225000 | > loss_text_ce: 0.043619345873594284 (0.044852115881429254) | > loss_mel_ce: 3.2405831813812256 (nan) | > loss: 0.013032550923526287 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13635488612796115) | > loader_time: 0.0046 (0.004702472531215731)  --> TIME: 2025-02-24 07:34:03 -- STEP: 3422/36938 -- GLOBAL_STEP: 225050 | > loss_text_ce: 0.039786696434020996 (0.04485647316476675) | > loss_mel_ce: 0.16287842392921448 (nan) | > loss: 0.0008042267290875316 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13633928613869004) | > loader_time: 0.0048 (0.004700824296105203)  --> TIME: 2025-02-24 07:34:12 -- STEP: 3472/36938 -- GLOBAL_STEP: 225100 | > loss_text_ce: 0.04055667668581009 (0.04489712608567205) | > loss_mel_ce: 3.817767858505249 (nan) | > loss: 0.01531081274151802 (nan) | > current_lr: 5e-06 | > step_time: 0.1829 (0.13633812661819258) | > loader_time: 0.0048 (0.004699798017602921)  --> TIME: 2025-02-24 07:34:21 -- STEP: 3522/36938 -- GLOBAL_STEP: 225150 | > loss_text_ce: 0.041198890656232834 (0.0448871146464981) | > loss_mel_ce: 3.540501356124878 (nan) | > loss: 0.014213097281754017 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13632577065376375) | > loader_time: 0.0047 (0.004699298226650579)  --> TIME: 2025-02-24 07:34:31 -- STEP: 3572/36938 -- GLOBAL_STEP: 225200 | > loss_text_ce: 0.043024711310863495 (0.044868851515171365) | > loss_mel_ce: 2.4901282787323 (nan) | > loss: 0.010052194818854332 (nan) | > current_lr: 5e-06 | > step_time: 0.1486 (0.1363363667805283) | > loader_time: 0.0047 (0.004699423425275036)  --> TIME: 2025-02-24 07:34:40 -- STEP: 3622/36938 -- GLOBAL_STEP: 225250 | > loss_text_ce: 0.055109862238168716 (0.04487586925927315) | > loss_mel_ce: 2.567115068435669 (nan) | > loss: 0.010405655950307846 (nan) | > current_lr: 5e-06 | > step_time: 0.1605 (0.13629135922264013) | > loader_time: 0.0048 (0.0046989046230558485)  --> TIME: 2025-02-24 07:34:49 -- STEP: 3672/36938 -- GLOBAL_STEP: 225300 | > loss_text_ce: 0.03301448002457619 (0.04486504661336267) | > loss_mel_ce: 3.262172222137451 (nan) | > loss: 0.01307613868266344 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.1362755451846492) | > loader_time: 0.0046 (0.004698050762313652)  --> TIME: 2025-02-24 07:34:59 -- STEP: 3722/36938 -- GLOBAL_STEP: 225350 | > loss_text_ce: 0.053336579352617264 (0.044851932488732114) | > loss_mel_ce: 3.607553720474243 (nan) | > loss: 0.014527343213558197 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13634120771027905) | > loader_time: 0.0045 (0.004698090627589329)  --> TIME: 2025-02-24 07:35:08 -- STEP: 3772/36938 -- GLOBAL_STEP: 225400 | > loss_text_ce: 0.03900047764182091 (0.044864659886244204) | > loss_mel_ce: 2.7660627365112305 (nan) | > loss: 0.01113120373338461 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13627946281989392) | > loader_time: 0.0045 (0.004698862200197974)  --> TIME: 2025-02-24 07:35:17 -- STEP: 3822/36938 -- GLOBAL_STEP: 225450 | > loss_text_ce: 0.04758712276816368 (0.04485293411469926) | > loss_mel_ce: 3.1338610649108887 (nan) | > loss: 0.01262479554861784 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13632084498662525) | > loader_time: 0.0046 (0.004699098009641487)  --> TIME: 2025-02-24 07:35:26 -- STEP: 3872/36938 -- GLOBAL_STEP: 225500 | > loss_text_ce: 0.04880385845899582 (0.04484855021857112) | > loss_mel_ce: 3.131394147872925 (nan) | > loss: 0.012619833461940289 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13628592360610864) | > loader_time: 0.0047 (0.004697940258447797)  --> TIME: 2025-02-24 07:35:36 -- STEP: 3922/36938 -- GLOBAL_STEP: 225550 | > loss_text_ce: 0.05261225253343582 (0.04485775275186291) | > loss_mel_ce: 3.548841953277588 (nan) | > loss: 0.014291485771536827 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13625682196648795) | > loader_time: 0.0046 (0.004697302237631783)  --> TIME: 2025-02-24 07:35:44 -- STEP: 3972/36938 -- GLOBAL_STEP: 225600 | > loss_text_ce: 0.05467190966010094 (0.04484823393853036) | > loss_mel_ce: 1.5522867441177368 (nan) | > loss: 0.006376820150762796 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1361694879162113) | > loader_time: 0.0046 (0.0046967279394827905)  --> TIME: 2025-02-24 07:35:53 -- STEP: 4022/36938 -- GLOBAL_STEP: 225650 | > loss_text_ce: 0.05625821650028229 (0.04483163101998745) | > loss_mel_ce: 3.6623551845550537 (nan) | > loss: 0.014756402932107449 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13611109164269805) | > loader_time: 0.0045 (0.00469631410131166)  --> TIME: 2025-02-24 07:36:02 -- STEP: 4072/36938 -- GLOBAL_STEP: 225700 | > loss_text_ce: 0.041512273252010345 (0.044842046496827895) | > loss_mel_ce: 3.5123507976531982 (nan) | > loss: 0.0141026321798563 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.1360872441752726) | > loader_time: 0.0047 (0.004695751987880722)  --> TIME: 2025-02-24 07:36:12 -- STEP: 4122/36938 -- GLOBAL_STEP: 225750 | > loss_text_ce: 0.046834126114845276 (0.04483461597652387) | > loss_mel_ce: 2.09619140625 (nan) | > loss: 0.008504070341587067 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13605996030791817) | > loader_time: 0.0045 (0.004694837091733745)  --> TIME: 2025-02-24 07:36:21 -- STEP: 4172/36938 -- GLOBAL_STEP: 225800 | > loss_text_ce: 0.04827191308140755 (0.04483062464476862) | > loss_mel_ce: 3.046082019805908 (nan) | > loss: 0.01227918267250061 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13607929216935385) | > loader_time: 0.0045 (0.0046945116549644665)  --> TIME: 2025-02-24 07:36:30 -- STEP: 4222/36938 -- GLOBAL_STEP: 225850 | > loss_text_ce: 0.05830048769712448 (0.04481269447053529) | > loss_mel_ce: 2.519195556640625 (nan) | > loss: 0.010228159837424755 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13608725666152813) | > loader_time: 0.0045 (0.004694214425116348)  --> TIME: 2025-02-24 07:36:39 -- STEP: 4272/36938 -- GLOBAL_STEP: 225900 | > loss_text_ce: 0.036532215774059296 (0.04480158345322975) | > loss_mel_ce: 0.1070963516831398 (nan) | > loss: 0.0005699546891264617 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.1360695171222264) | > loader_time: 0.0045 (0.004694605978687169)  --> TIME: 2025-02-24 07:36:49 -- STEP: 4322/36938 -- GLOBAL_STEP: 225950 | > loss_text_ce: 0.04201516881585121 (0.044788525380971625) | > loss_mel_ce: 2.966188669204712 (nan) | > loss: 0.011937317438423634 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.13607118897612416) | > loader_time: 0.0049 (0.004694414105695588)  --> TIME: 2025-02-24 07:36:58 -- STEP: 4372/36938 -- GLOBAL_STEP: 226000 | > loss_text_ce: 0.047970421612262726 (0.044767364044354184) | > loss_mel_ce: 3.5269978046417236 (nan) | > loss: 0.014186383225023746 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1360391141729194) | > loader_time: 0.0046 (0.004693760036440595)  --> TIME: 2025-02-24 07:37:07 -- STEP: 4422/36938 -- GLOBAL_STEP: 226050 | > loss_text_ce: 0.052438702434301376 (0.04475757810600987) | > loss_mel_ce: 3.025197982788086 (nan) | > loss: 0.01221284456551075 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13608030213968988) | > loader_time: 0.0067 (0.004694321262302692)  --> TIME: 2025-02-24 07:37:17 -- STEP: 4472/36938 -- GLOBAL_STEP: 226100 | > loss_text_ce: 0.05498946085572243 (0.044753909059167096) | > loss_mel_ce: 3.8600242137908936 (nan) | > loss: 0.015535769984126091 (nan) | > current_lr: 5e-06 | > step_time: 0.1712 (0.1360775721734245) | > loader_time: 0.0048 (0.004693876385475537)  --> TIME: 2025-02-24 07:37:26 -- STEP: 4522/36938 -- GLOBAL_STEP: 226150 | > loss_text_ce: 0.045423056930303574 (0.04474533563941117) | > loss_mel_ce: 3.5465075969696045 (nan) | > loss: 0.014253693632781506 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.136051192214035) | > loader_time: 0.0045 (0.004693343543415849)  --> TIME: 2025-02-24 07:37:35 -- STEP: 4572/36938 -- GLOBAL_STEP: 226200 | > loss_text_ce: 0.042219918221235275 (0.04474367928684691) | > loss_mel_ce: 3.8373165130615234 (nan) | > loss: 0.015394986607134342 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13605047951428856) | > loader_time: 0.0048 (0.0046928690278728365)  --> TIME: 2025-02-24 07:37:44 -- STEP: 4622/36938 -- GLOBAL_STEP: 226250 | > loss_text_ce: 0.03601352125406265 (0.044729731254213534) | > loss_mel_ce: 3.373821973800659 (nan) | > loss: 0.013531094416975975 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13606027781009106) | > loader_time: 0.0045 (0.004692130045476084)  --> TIME: 2025-02-24 07:37:53 -- STEP: 4672/36938 -- GLOBAL_STEP: 226300 | > loss_text_ce: 0.06263526529073715 (0.044733473125483196) | > loss_mel_ce: 0.096771240234375 (nan) | > loss: 0.0006325655849650502 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13602718476155085) | > loader_time: 0.0045 (0.004690947853130842)  --> TIME: 2025-02-24 07:38:03 -- STEP: 4722/36938 -- GLOBAL_STEP: 226350 | > loss_text_ce: 0.04673211649060249 (0.044726335238633765) | > loss_mel_ce: 2.8478732109069824 (nan) | > loss: 0.011486530303955078 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.136016271923216) | > loader_time: 0.0047 (0.004690032296138319)  --> TIME: 2025-02-24 07:38:12 -- STEP: 4772/36938 -- GLOBAL_STEP: 226400 | > loss_text_ce: 0.04733443260192871 (0.04473423060831478) | > loss_mel_ce: 3.4456095695495605 (nan) | > loss: 0.013860889710485935 (nan) | > current_lr: 5e-06 | > step_time: 0.1754 (0.13603134836754932) | > loader_time: 0.005 (0.004689291606704833)  --> TIME: 2025-02-24 07:38:21 -- STEP: 4822/36938 -- GLOBAL_STEP: 226450 | > loss_text_ce: 0.04757832735776901 (0.04473228122879086) | > loss_mel_ce: 3.8956336975097656 (nan) | > loss: 0.01564766839146614 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.13603715232080238) | > loader_time: 0.0047 (0.004688992929478405)  --> TIME: 2025-02-24 07:38:31 -- STEP: 4872/36938 -- GLOBAL_STEP: 226500 | > loss_text_ce: 0.03749550133943558 (0.04469668016289563) | > loss_mel_ce: 0.0416259765625 (nan) | > loss: 0.0003139741311315447 (nan) | > current_lr: 5e-06 | > step_time: 0.1068 (0.13604248646640688) | > loader_time: 0.0044 (0.004688823360136194)  --> TIME: 2025-02-24 07:38:40 -- STEP: 4922/36938 -- GLOBAL_STEP: 226550 | > loss_text_ce: 0.051215704530477524 (0.04469026067895506) | > loss_mel_ce: 3.4407410621643066 (nan) | > loss: 0.013856971636414528 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13605411074017248) | > loader_time: 0.0046 (0.0046890236797588565)  --> TIME: 2025-02-24 07:38:49 -- STEP: 4972/36938 -- GLOBAL_STEP: 226600 | > loss_text_ce: 0.04230385646224022 (0.044688604765909516) | > loss_mel_ce: 3.414799690246582 (nan) | > loss: 0.013718665577471256 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13605395307103643) | > loader_time: 0.0045 (0.00468894189533839)  --> TIME: 2025-02-24 07:38:59 -- STEP: 5022/36938 -- GLOBAL_STEP: 226650 | > loss_text_ce: 0.03498286381363869 (0.04468030910230558) | > loss_mel_ce: 3.053633451461792 (nan) | > loss: 0.012256414629518986 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13608649459696137) | > loader_time: 0.0045 (0.004687963278347736)  --> TIME: 2025-02-24 07:39:08 -- STEP: 5072/36938 -- GLOBAL_STEP: 226700 | > loss_text_ce: 0.05260998755693436 (0.044675391203208174) | > loss_mel_ce: 2.7209079265594482 (nan) | > loss: 0.011006023734807968 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.13605571587567156) | > loader_time: 0.0046 (0.004687692652741444)  --> TIME: 2025-02-24 07:39:17 -- STEP: 5122/36938 -- GLOBAL_STEP: 226750 | > loss_text_ce: 0.0479147844016552 (0.04465836477736844) | > loss_mel_ce: 3.7528443336486816 (nan) | > loss: 0.015082377940416336 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13603109421556958) | > loader_time: 0.0046 (0.0046874239127275685)  --> TIME: 2025-02-24 07:39:26 -- STEP: 5172/36938 -- GLOBAL_STEP: 226800 | > loss_text_ce: 0.04623664915561676 (0.044647848289582306) | > loss_mel_ce: 3.5446877479553223 (nan) | > loss: 0.014249701052904129 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13599404109510896) | > loader_time: 0.0046 (0.0046869696615651334)  --> TIME: 2025-02-24 07:39:35 -- STEP: 5222/36938 -- GLOBAL_STEP: 226850 | > loss_text_ce: 0.04486076906323433 (0.04465212791852075) | > loss_mel_ce: 3.850668430328369 (nan) | > loss: 0.01545845065265894 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13599571917806788) | > loader_time: 0.0047 (0.004687254751198873)  --> TIME: 2025-02-24 07:39:45 -- STEP: 5272/36938 -- GLOBAL_STEP: 226900 | > loss_text_ce: 0.05026642233133316 (0.044651755288826846) | > loss_mel_ce: 3.581441879272461 (nan) | > loss: 0.014411541633307934 (nan) | > current_lr: 5e-06 | > step_time: 0.1664 (0.13601716199666558) | > loader_time: 0.0048 (0.004686736463595961)  --> TIME: 2025-02-24 07:39:54 -- STEP: 5322/36938 -- GLOBAL_STEP: 226950 | > loss_text_ce: 0.04283432289958 (0.044657708180501916) | > loss_mel_ce: 3.7808780670166016 (nan) | > loss: 0.015173462219536304 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13596542559813374) | > loader_time: 0.0046 (0.004685987569657724)  --> TIME: 2025-02-24 07:40:03 -- STEP: 5372/36938 -- GLOBAL_STEP: 227000 | > loss_text_ce: 0.050119977444410324 (0.04465822152441814) | > loss_mel_ce: 3.406660318374634 (nan) | > loss: 0.013717382214963436 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.13600147489461167) | > loader_time: 0.0049 (0.004685645794034094)  --> TIME: 2025-02-24 07:40:13 -- STEP: 5422/36938 -- GLOBAL_STEP: 227050 | > loss_text_ce: 0.03912512585520744 (0.044656763749031485) | > loss_mel_ce: 2.91367506980896 (nan) | > loss: 0.011717461980879307 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13607259537187877) | > loader_time: 0.0046 (0.004685462378611296)  --> TIME: 2025-02-24 07:40:23 -- STEP: 5472/36938 -- GLOBAL_STEP: 227100 | > loss_text_ce: 0.029592910781502724 (0.04467405145800648) | > loss_mel_ce: 3.2771739959716797 (nan) | > loss: 0.013122092001140118 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13610967836882004) | > loader_time: 0.0046 (0.00468499257027754)  --> TIME: 2025-02-24 07:40:32 -- STEP: 5522/36938 -- GLOBAL_STEP: 227150 | > loss_text_ce: 0.05570147931575775 (0.04468031895014011) | > loss_mel_ce: 2.3030080795288086 (nan) | > loss: 0.00935995951294899 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13607292579421595) | > loader_time: 0.0046 (0.00468467746425132)  --> TIME: 2025-02-24 07:40:41 -- STEP: 5572/36938 -- GLOBAL_STEP: 227200 | > loss_text_ce: 0.04066484048962593 (0.044683867143012417) | > loss_mel_ce: 3.3391940593719482 (nan) | > loss: 0.013412139378488064 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13607484846772414) | > loader_time: 0.0045 (0.004684373190825731)  --> TIME: 2025-02-24 07:40:50 -- STEP: 5622/36938 -- GLOBAL_STEP: 227250 | > loss_text_ce: 0.036403194069862366 (0.04465902242405331) | > loss_mel_ce: 2.8837335109710693 (nan) | > loss: 0.011587844230234623 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13609472066641173) | > loader_time: 0.0048 (0.004684249814487101)  --> TIME: 2025-02-24 07:41:00 -- STEP: 5672/36938 -- GLOBAL_STEP: 227300 | > loss_text_ce: 0.050680775195360184 (0.04466642590426174) | > loss_mel_ce: 3.7811858654022217 (nan) | > loss: 0.01520582102239132 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1360689576909646) | > loader_time: 0.0047 (0.004683510877181507)  --> TIME: 2025-02-24 07:41:09 -- STEP: 5722/36938 -- GLOBAL_STEP: 227350 | > loss_text_ce: 0.034057676792144775 (0.04466984860471437) | > loss_mel_ce: 1.763462781906128 (nan) | > loss: 0.007133017759770155 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13605602997576902) | > loader_time: 0.0046 (0.004682995438867408)  --> TIME: 2025-02-24 07:41:18 -- STEP: 5772/36938 -- GLOBAL_STEP: 227400 | > loss_text_ce: 0.0449175164103508 (0.04467227748335081) | > loss_mel_ce: 2.696803569793701 (nan) | > loss: 0.010879846289753914 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.13604506645777667) | > loader_time: 0.0047 (0.004682440230686133)  --> TIME: 2025-02-24 07:41:28 -- STEP: 5822/36938 -- GLOBAL_STEP: 227450 | > loss_text_ce: 0.045668162405490875 (0.04467334050291013) | > loss_mel_ce: 4.3569135665893555 (nan) | > loss: 0.01747056283056736 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13609868486654117) | > loader_time: 0.0047 (0.00468210631815767)  --> TIME: 2025-02-24 07:41:37 -- STEP: 5872/36938 -- GLOBAL_STEP: 227500 | > loss_text_ce: 0.04407468065619469 (0.04467278740518508) | > loss_mel_ce: 3.5214829444885254 (nan) | > loss: 0.014149039052426815 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.1361273385442251) | > loader_time: 0.0045 (0.004682109571932444)  --> TIME: 2025-02-24 07:41:46 -- STEP: 5922/36938 -- GLOBAL_STEP: 227550 | > loss_text_ce: 0.040030088275671005 (0.04466868933587161) | > loss_mel_ce: 3.130430221557617 (nan) | > loss: 0.012581191956996918 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13608645297755573) | > loader_time: 0.0045 (0.004681389382544983)  --> TIME: 2025-02-24 07:41:56 -- STEP: 5972/36938 -- GLOBAL_STEP: 227600 | > loss_text_ce: 0.04691173881292343 (0.044656835270300774) | > loss_mel_ce: 3.1999104022979736 (nan) | > loss: 0.012884215451776981 (nan) | > current_lr: 5e-06 | > step_time: 0.1776 (0.1361058552394523) | > loader_time: 0.0047 (0.004680885018235632)  --> TIME: 2025-02-24 07:42:05 -- STEP: 6022/36938 -- GLOBAL_STEP: 227650 | > loss_text_ce: 0.01833890564739704 (0.044661763497810474) | > loss_mel_ce: 2.3364908695220947 (nan) | > loss: 0.009344562888145447 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13611321154720873) | > loader_time: 0.0044 (0.0046808309469536565)  --> TIME: 2025-02-24 07:42:14 -- STEP: 6072/36938 -- GLOBAL_STEP: 227700 | > loss_text_ce: 0.041391633450984955 (0.044659441618901626) | > loss_mel_ce: 3.4384689331054688 (nan) | > loss: 0.013808971270918846 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.1361269265688148) | > loader_time: 0.0048 (0.00468044267655675)  --> TIME: 2025-02-24 07:42:24 -- STEP: 6122/36938 -- GLOBAL_STEP: 227750 | > loss_text_ce: 0.03791411221027374 (0.04466294981594017) | > loss_mel_ce: 3.7006936073303223 (nan) | > loss: 0.014835745096206665 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.13617229749859722) | > loader_time: 0.0049 (0.004679953728731604)  --> TIME: 2025-02-24 07:42:33 -- STEP: 6172/36938 -- GLOBAL_STEP: 227800 | > loss_text_ce: 0.04603027552366257 (0.04466440177468883) | > loss_mel_ce: 3.400134325027466 (nan) | > loss: 0.013675257563591003 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.1361572521890214) | > loader_time: 0.0046 (0.004679167726483338)  --> TIME: 2025-02-24 07:42:42 -- STEP: 6222/36938 -- GLOBAL_STEP: 227850 | > loss_text_ce: 0.04656403511762619 (0.044653100704601247) | > loss_mel_ce: 2.995898723602295 (nan) | > loss: 0.012073266319930553 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1361020251279941) | > loader_time: 0.0045 (0.004678851536561028)  --> TIME: 2025-02-24 07:42:51 -- STEP: 6272/36938 -- GLOBAL_STEP: 227900 | > loss_text_ce: 0.039522863924503326 (0.044641407415430004) | > loss_mel_ce: 3.2543466091156006 (nan) | > loss: 0.013070911169052124 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.1361026128716014) | > loader_time: 0.0048 (0.004678619265252219)  --> TIME: 2025-02-24 07:43:01 -- STEP: 6322/36938 -- GLOBAL_STEP: 227950 | > loss_text_ce: 0.04147212579846382 (0.044624170737163696) | > loss_mel_ce: 3.8721463680267334 (nan) | > loss: 0.015530233271420002 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.13614566218283605) | > loader_time: 0.0047 (0.004678429662709887)  --> TIME: 2025-02-24 07:43:11 -- STEP: 6372/36938 -- GLOBAL_STEP: 228000 | > loss_text_ce: 0.04951978474855423 (0.04461920993608393) | > loss_mel_ce: 2.498778820037842 (nan) | > loss: 0.010112296789884567 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13615682640692375) | > loader_time: 0.0046 (0.004678400522197322)  --> TIME: 2025-02-24 07:43:20 -- STEP: 6422/36938 -- GLOBAL_STEP: 228050 | > loss_text_ce: 0.049731940031051636 (0.04461726786122679) | > loss_mel_ce: 3.128910779953003 (nan) | > loss: 0.012613662518560886 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13616584591760186) | > loader_time: 0.0045 (0.004677677072865378)  --> TIME: 2025-02-24 07:43:29 -- STEP: 6472/36938 -- GLOBAL_STEP: 228100 | > loss_text_ce: 0.049763891845941544 (0.044634170479996536) | > loss_mel_ce: 2.907043218612671 (nan) | > loss: 0.011733362451195717 (nan) | > current_lr: 5e-06 | > step_time: 0.1521 (0.1361770811864863) | > loader_time: 0.0046 (0.004677075280247266)  --> TIME: 2025-02-24 07:43:39 -- STEP: 6522/36938 -- GLOBAL_STEP: 228150 | > loss_text_ce: 0.046370163559913635 (0.044629947037905314) | > loss_mel_ce: 3.1639795303344727 (nan) | > loss: 0.012739484198391438 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13622948217523648) | > loader_time: 0.0046 (0.004676586936054617)  --> TIME: 2025-02-24 07:43:48 -- STEP: 6572/36938 -- GLOBAL_STEP: 228200 | > loss_text_ce: 0.03836046904325485 (0.04464120034732683) | > loss_mel_ce: 3.1612868309020996 (nan) | > loss: 0.012697013095021248 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1362351215763645) | > loader_time: 0.0048 (0.00467599617295779)  --> TIME: 2025-02-24 07:43:58 -- STEP: 6622/36938 -- GLOBAL_STEP: 228250 | > loss_text_ce: 0.04617546126246452 (0.04463913214931053) | > loss_mel_ce: 1.15237295627594 (nan) | > loss: 0.004756144713610411 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.136263454871321) | > loader_time: 0.0046 (0.004675762921940774)  --> TIME: 2025-02-24 07:44:07 -- STEP: 6672/36938 -- GLOBAL_STEP: 228300 | > loss_text_ce: 0.041846368461847305 (0.044632948619845104) | > loss_mel_ce: 3.606684684753418 (nan) | > loss: 0.014478297904133797 (nan) | > current_lr: 5e-06 | > step_time: 0.1779 (0.13626409695445849) | > loader_time: 0.0049 (0.004675539634782349)  --> TIME: 2025-02-24 07:44:17 -- STEP: 6722/36938 -- GLOBAL_STEP: 228350 | > loss_text_ce: 0.046539634466171265 (0.044630223105427996) | > loss_mel_ce: 1.2917886972427368 (nan) | > loss: 0.005310826934874058 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.1362544367335382) | > loader_time: 0.0046 (0.004675963101589904)  --> TIME: 2025-02-24 07:44:26 -- STEP: 6772/36938 -- GLOBAL_STEP: 228400 | > loss_text_ce: 0.04474867880344391 (0.04463124388033428) | > loss_mel_ce: 1.9795485734939575 (nan) | > loss: 0.008032926358282566 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13631466824981694) | > loader_time: 0.0045 (0.004675879115123264)  --> TIME: 2025-02-24 07:44:36 -- STEP: 6822/36938 -- GLOBAL_STEP: 228450 | > loss_text_ce: 0.04385192692279816 (0.04462482536889285) | > loss_mel_ce: 2.7777183055877686 (nan) | > loss: 0.011196707375347614 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1363278816117467) | > loader_time: 0.0046 (0.00467557401749368)  --> TIME: 2025-02-24 07:44:45 -- STEP: 6872/36938 -- GLOBAL_STEP: 228500 | > loss_text_ce: 0.050253160297870636 (0.04462748261198958) | > loss_mel_ce: 3.5454912185668945 (nan) | > loss: 0.014268827624619007 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13632999976383592) | > loader_time: 0.0046 (0.00467540766085401)  --> TIME: 2025-02-24 07:44:55 -- STEP: 6922/36938 -- GLOBAL_STEP: 228550 | > loss_text_ce: 0.05433013662695885 (0.04463215233562734) | > loss_mel_ce: 1.4359979629516602 (nan) | > loss: 0.005914000794291496 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13636070526049276) | > loader_time: 0.0045 (0.004675306601525726)  --> TIME: 2025-02-24 07:45:04 -- STEP: 6972/36938 -- GLOBAL_STEP: 228600 | > loss_text_ce: 0.03584115207195282 (0.04462405108503157) | > loss_mel_ce: 0.1689961701631546 (nan) | > loss: 0.0008128465851768851 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13637261389318256) | > loader_time: 0.0046 (0.004675213557444145)  --> TIME: 2025-02-24 07:45:14 -- STEP: 7022/36938 -- GLOBAL_STEP: 228650 | > loss_text_ce: 0.049420859664678574 (0.04462027685794736) | > loss_mel_ce: 3.2714974880218506 (nan) | > loss: 0.013178247958421707 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13640293753137026) | > loader_time: 0.0047 (0.004675059500531023)  --> TIME: 2025-02-24 07:45:23 -- STEP: 7072/36938 -- GLOBAL_STEP: 228700 | > loss_text_ce: 0.051867783069610596 (0.04461847409859883) | > loss_mel_ce: 1.9178358316421509 (nan) | > loss: 0.007816284894943237 (nan) | > current_lr: 5e-06 | > step_time: 0.1891 (0.1363798678258434) | > loader_time: 0.0048 (0.004674539441976068)  --> TIME: 2025-02-24 07:45:32 -- STEP: 7122/36938 -- GLOBAL_STEP: 228750 | > loss_text_ce: 0.037339020520448685 (0.044618453673298655) | > loss_mel_ce: 3.0929572582244873 (nan) | > loss: 0.012421810999512672 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13638903062954966) | > loader_time: 0.0045 (0.004674469343365924)  --> TIME: 2025-02-24 07:45:41 -- STEP: 7172/36938 -- GLOBAL_STEP: 228800 | > loss_text_ce: 0.0430067777633667 (0.044618510060963935) | > loss_mel_ce: 0.1160888671875 (nan) | > loss: 0.0006313319317996502 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1363459780637747) | > loader_time: 0.0046 (0.004674167521360257)  --> TIME: 2025-02-24 07:45:50 -- STEP: 7222/36938 -- GLOBAL_STEP: 228850 | > loss_text_ce: 0.03078390844166279 (0.044623895962749696) | > loss_mel_ce: 3.04710054397583 (nan) | > loss: 0.012213828042149544 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13634122467807008) | > loader_time: 0.0046 (0.004673845944263302)  --> TIME: 2025-02-24 07:45:59 -- STEP: 7272/36938 -- GLOBAL_STEP: 228900 | > loss_text_ce: 0.043075185269117355 (0.044622501651157116) | > loss_mel_ce: 3.463674545288086 (nan) | > loss: 0.013915673829615116 (nan) | > current_lr: 5e-06 | > step_time: 0.1605 (0.13630470679407022) | > loader_time: 0.0047 (0.004673789928455159)  --> TIME: 2025-02-24 07:46:09 -- STEP: 7322/36938 -- GLOBAL_STEP: 228950 | > loss_text_ce: 0.04443509504199028 (0.04462871051550409) | > loss_mel_ce: 2.347625732421875 (nan) | > loss: 0.009492305107414722 (nan) | > current_lr: 5e-06 | > step_time: 0.1066 (0.13630035486224001) | > loader_time: 0.0044 (0.004673428855870984)  --> TIME: 2025-02-24 07:46:18 -- STEP: 7372/36938 -- GLOBAL_STEP: 229000 | > loss_text_ce: 0.048593901097774506 (0.04463042571106791) | > loss_mel_ce: 3.137500762939453 (nan) | > loss: 0.012643233872950077 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.1362889847933962) | > loader_time: 0.0046 (0.004673016213620409)  --> TIME: 2025-02-24 07:46:27 -- STEP: 7422/36938 -- GLOBAL_STEP: 229050 | > loss_text_ce: 0.033246107399463654 (0.04462320597782608) | > loss_mel_ce: 3.3211095333099365 (nan) | > loss: 0.013310935348272324 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.1362359643592831) | > loader_time: 0.0045 (0.004672581119635004)  --> TIME: 2025-02-24 07:46:36 -- STEP: 7472/36938 -- GLOBAL_STEP: 229100 | > loss_text_ce: 0.04221082106232643 (0.04461615763722304) | > loss_mel_ce: 3.458709716796875 (nan) | > loss: 0.013892542570829391 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13624979255669184) | > loader_time: 0.0048 (0.004672615571501929)  --> TIME: 2025-02-24 07:46:45 -- STEP: 7522/36938 -- GLOBAL_STEP: 229150 | > loss_text_ce: 0.04671434313058853 (0.04460493523287778) | > loss_mel_ce: 3.0609374046325684 (nan) | > loss: 0.012331952340900898 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.13622194793242778) | > loader_time: 0.0046 (0.004672206421223002)  --> TIME: 2025-02-24 07:46:54 -- STEP: 7572/36938 -- GLOBAL_STEP: 229200 | > loss_text_ce: 0.04265265539288521 (0.0446000871956014) | > loss_mel_ce: 3.523857593536377 (nan) | > loss: 0.01415281929075718 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13621124951095157) | > loader_time: 0.0046 (0.004671772037683433)  --> TIME: 2025-02-24 07:47:03 -- STEP: 7622/36938 -- GLOBAL_STEP: 229250 | > loss_text_ce: 0.052389297634363174 (0.04459553048170067) | > loss_mel_ce: 2.6042120456695557 (nan) | > loss: 0.010542069561779499 (nan) | > current_lr: 5e-06 | > step_time: 0.1072 (0.13617962300073688) | > loader_time: 0.0045 (0.004671300374063343)  --> TIME: 2025-02-24 07:47:13 -- STEP: 7672/36938 -- GLOBAL_STEP: 229300 | > loss_text_ce: 0.04776749014854431 (0.044599477293715675) | > loss_mel_ce: 3.1913957595825195 (nan) | > loss: 0.01285382267087698 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.1361789681304862) | > loader_time: 0.0047 (0.00467121653934713)  --> TIME: 2025-02-24 07:47:22 -- STEP: 7722/36938 -- GLOBAL_STEP: 229350 | > loss_text_ce: 0.04120488464832306 (0.04459562729436272) | > loss_mel_ce: 4.029084205627441 (nan) | > loss: 0.016151942312717438 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13615168416108034) | > loader_time: 0.0046 (0.004670716881968137)  --> TIME: 2025-02-24 07:47:31 -- STEP: 7772/36938 -- GLOBAL_STEP: 229400 | > loss_text_ce: 0.04336483031511307 (0.04459579398891997) | > loss_mel_ce: 3.6633591651916504 (nan) | > loss: 0.014709223061800003 (nan) | > current_lr: 5e-06 | > step_time: 0.1626 (0.13617162473166225) | > loader_time: 0.0046 (0.004670361176754352)  --> TIME: 2025-02-24 07:47:41 -- STEP: 7822/36938 -- GLOBAL_STEP: 229450 | > loss_text_ce: 0.03991532698273659 (0.044588424061784385) | > loss_mel_ce: 3.6308951377868652 (nan) | > loss: 0.014566709287464619 (nan) | > current_lr: 5e-06 | > step_time: 0.1622 (0.13617360918029983) | > loader_time: 0.0046 (0.0046703665496077035)  --> TIME: 2025-02-24 07:47:50 -- STEP: 7872/36938 -- GLOBAL_STEP: 229500 | > loss_text_ce: 0.03998221829533577 (0.044587962292475504) | > loss_mel_ce: 2.892387866973877 (nan) | > loss: 0.01163639035075903 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.1361788169034134) | > loader_time: 0.0045 (0.0046702563096352664)  --> TIME: 2025-02-24 07:47:59 -- STEP: 7922/36938 -- GLOBAL_STEP: 229550 | > loss_text_ce: 0.05568147078156471 (0.04458620553101233) | > loss_mel_ce: 3.662900447845459 (nan) | > loss: 0.01475627813488245 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.13619807434033934) | > loader_time: 0.0048 (0.00466997272527088)  --> TIME: 2025-02-24 07:48:09 -- STEP: 7972/36938 -- GLOBAL_STEP: 229600 | > loss_text_ce: 0.053179070353507996 (0.044589144048293644) | > loss_mel_ce: 3.7333905696868896 (nan) | > loss: 0.015026071108877659 (nan) | > current_lr: 5e-06 | > step_time: 0.1749 (0.13620081389901448) | > loader_time: 0.0048 (0.00466995555075699)  --> TIME: 2025-02-24 07:48:18 -- STEP: 8022/36938 -- GLOBAL_STEP: 229650 | > loss_text_ce: 0.04734291881322861 (0.04458161362343985) | > loss_mel_ce: 1.8225504159927368 (nan) | > loss: 0.007420212030410767 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13623189970728403) | > loader_time: 0.0044 (0.004669655293664616)  --> TIME: 2025-02-24 07:48:28 -- STEP: 8072/36938 -- GLOBAL_STEP: 229700 | > loss_text_ce: 0.043059203773736954 (0.044582839868778174) | > loss_mel_ce: 3.94132137298584 (nan) | > loss: 0.01581103540956974 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.13623650958327813) | > loader_time: 0.0048 (0.0046692370364167365)  --> TIME: 2025-02-24 07:48:37 -- STEP: 8122/36938 -- GLOBAL_STEP: 229750 | > loss_text_ce: 0.04071477800607681 (0.044589622429191594) | > loss_mel_ce: 3.647794008255005 (nan) | > loss: 0.014636940322816372 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.13627031465317688) | > loader_time: 0.0048 (0.0046689051532064286)  --> TIME: 2025-02-24 07:48:46 -- STEP: 8172/36938 -- GLOBAL_STEP: 229800 | > loss_text_ce: 0.03830258175730705 (0.04458265090329329) | > loss_mel_ce: 3.0394372940063477 (nan) | > loss: 0.01221325434744358 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13625614557322308) | > loader_time: 0.0046 (0.004668590868447141)  --> TIME: 2025-02-24 07:48:55 -- STEP: 8222/36938 -- GLOBAL_STEP: 229850 | > loss_text_ce: 0.04597628861665726 (0.04458362324608705) | > loss_mel_ce: 2.9479057788848877 (nan) | > loss: 0.011880485340952873 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13622467969056085) | > loader_time: 0.0047 (0.004668276955454457)  --> TIME: 2025-02-24 07:49:05 -- STEP: 8272/36938 -- GLOBAL_STEP: 229900 | > loss_text_ce: 0.03913628309965134 (0.044581681617843705) | > loss_mel_ce: 2.6738665103912354 (nan) | > loss: 0.010765884071588516 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13621095586676255) | > loader_time: 0.0045 (0.004667907405637445)  --> TIME: 2025-02-24 07:49:14 -- STEP: 8322/36938 -- GLOBAL_STEP: 229950 | > loss_text_ce: 0.04685053601861 (0.04457231637245331) | > loss_mel_ce: 3.660215139389038 (nan) | > loss: 0.014710578136146069 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.13616833890810914) | > loader_time: 0.0047 (0.004667458583517328)  --> TIME: 2025-02-24 07:49:23 -- STEP: 8372/36938 -- GLOBAL_STEP: 230000 | > loss_text_ce: 0.04216957837343216 (0.04456323472897773) | > loss_mel_ce: 2.944411039352417 (nan) | > loss: 0.011851510964334011 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.1361580552863442) | > loader_time: 0.0049 (0.004666908500414396) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_230000.pth  --> TIME: 2025-02-24 07:49:38 -- STEP: 8422/36938 -- GLOBAL_STEP: 230050 | > loss_text_ce: 0.03573671355843544 (0.044572342765068364) | > loss_mel_ce: 2.819284439086914 (nan) | > loss: 0.011329449713230133 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13612002965192949) | > loader_time: 0.0046 (0.004667033183266102)  --> TIME: 2025-02-24 07:49:47 -- STEP: 8472/36938 -- GLOBAL_STEP: 230100 | > loss_text_ce: 0.05121457949280739 (0.04457236728010958) | > loss_mel_ce: 4.350559711456299 (nan) | > loss: 0.017467359080910683 (nan) | > current_lr: 5e-06 | > step_time: 0.1868 (0.13612332104285835) | > loader_time: 0.0049 (0.004667070842892405)  --> TIME: 2025-02-24 07:49:56 -- STEP: 8522/36938 -- GLOBAL_STEP: 230150 | > loss_text_ce: 0.04007534310221672 (0.044565864202842496) | > loss_mel_ce: 3.9954192638397217 (nan) | > loss: 0.016013870015740395 (nan) | > current_lr: 5e-06 | > step_time: 0.1785 (0.13613877333377222) | > loader_time: 0.0048 (0.00466734549999796)  --> TIME: 2025-02-24 07:50:06 -- STEP: 8572/36938 -- GLOBAL_STEP: 230200 | > loss_text_ce: 0.03580208867788315 (0.04456672424926823) | > loss_mel_ce: 3.758225440979004 (nan) | > loss: 0.015055665746331215 (nan) | > current_lr: 5e-06 | > step_time: 0.184 (0.13616187212014433) | > loader_time: 0.0048 (0.004667593033248108)  --> TIME: 2025-02-24 07:50:15 -- STEP: 8622/36938 -- GLOBAL_STEP: 230250 | > loss_text_ce: 0.046023815870285034 (0.044571430246081886) | > loss_mel_ce: 3.3999416828155518 (nan) | > loss: 0.013674466870725155 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.1361492656549818) | > loader_time: 0.0048 (0.004667780233961441)  --> TIME: 2025-02-24 07:50:24 -- STEP: 8672/36938 -- GLOBAL_STEP: 230300 | > loss_text_ce: 0.046188242733478546 (0.04457458024230231) | > loss_mel_ce: 3.3081891536712646 (nan) | > loss: 0.013311022892594337 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.1361453416052784) | > loader_time: 0.0046 (0.004667842025246561)  --> TIME: 2025-02-24 07:50:34 -- STEP: 8722/36938 -- GLOBAL_STEP: 230350 | > loss_text_ce: 0.05649303272366524 (0.044577008447909) | > loss_mel_ce: 2.7904739379882812 (nan) | > loss: 0.011297488585114479 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13614623613845248) | > loader_time: 0.0046 (0.004667853767842875)  --> TIME: 2025-02-24 07:50:43 -- STEP: 8772/36938 -- GLOBAL_STEP: 230400 | > loss_text_ce: 0.04735312983393669 (0.04458334952286229) | > loss_mel_ce: 3.5436813831329346 (nan) | > loss: 0.014250137843191624 (nan) | > current_lr: 5e-06 | > step_time: 0.1903 (0.13618461600841625) | > loader_time: 0.0049 (0.004667945610460378)  --> TIME: 2025-02-24 07:50:52 -- STEP: 8822/36938 -- GLOBAL_STEP: 230450 | > loss_text_ce: 0.05713982135057449 (0.04458780423321067) | > loss_mel_ce: 2.475844383239746 (nan) | > loss: 0.010051525197923183 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.136154135401572) | > loader_time: 0.0047 (0.004667555521037724)  --> TIME: 2025-02-24 07:51:02 -- STEP: 8872/36938 -- GLOBAL_STEP: 230500 | > loss_text_ce: 0.03930578753352165 (0.044592235064993956) | > loss_mel_ce: 1.867450475692749 (nan) | > loss: 0.007566493470221758 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13619980414887642) | > loader_time: 0.0045 (0.004667959173484568)  --> TIME: 2025-02-24 07:51:12 -- STEP: 8922/36938 -- GLOBAL_STEP: 230550 | > loss_text_ce: 0.04031633585691452 (0.04459127980582726) | > loss_mel_ce: 3.3081154823303223 (nan) | > loss: 0.013287428766489029 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13622104996016604) | > loader_time: 0.0046 (0.004668023281101683)  --> TIME: 2025-02-24 07:51:21 -- STEP: 8972/36938 -- GLOBAL_STEP: 230600 | > loss_text_ce: 0.04340430721640587 (0.044592414004998124) | > loss_mel_ce: 3.4789180755615234 (nan) | > loss: 0.013977470807731152 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.13623968209212692) | > loader_time: 0.0044 (0.004667758516472356)  --> TIME: 2025-02-24 07:51:30 -- STEP: 9022/36938 -- GLOBAL_STEP: 230650 | > loss_text_ce: 0.057094696909189224 (0.04458676037285501) | > loss_mel_ce: 3.4809296131134033 (nan) | > loss: 0.014039780013263226 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13623496175895813) | > loader_time: 0.0045 (0.0046676010177500605)  --> TIME: 2025-02-24 07:51:40 -- STEP: 9072/36938 -- GLOBAL_STEP: 230700 | > loss_text_ce: 0.05554325506091118 (0.044576837045774594) | > loss_mel_ce: 3.101747751235962 (nan) | > loss: 0.01252893265336752 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13624000596621696) | > loader_time: 0.0046 (0.004667950843377081)  --> TIME: 2025-02-24 07:51:49 -- STEP: 9122/36938 -- GLOBAL_STEP: 230750 | > loss_text_ce: 0.04430066794157028 (0.04457184258273421) | > loss_mel_ce: 2.913625955581665 (nan) | > loss: 0.011737803928554058 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13621926171767623) | > loader_time: 0.0046 (0.00466794053913861)  --> TIME: 2025-02-24 07:51:58 -- STEP: 9172/36938 -- GLOBAL_STEP: 230800 | > loss_text_ce: 0.05013379082083702 (0.04457703806139846) | > loss_mel_ce: 2.693131446838379 (nan) | > loss: 0.010885973460972309 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13623211691070758) | > loader_time: 0.0045 (0.004668056834903382)  --> TIME: 2025-02-24 07:52:07 -- STEP: 9222/36938 -- GLOBAL_STEP: 230850 | > loss_text_ce: 0.05736800655722618 (0.04457698575081835) | > loss_mel_ce: 0.7328414916992188 (nan) | > loss: 0.0031357521656900644 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.136195611653838) | > loader_time: 0.0046 (0.004667797747363592)  --> TIME: 2025-02-24 07:52:16 -- STEP: 9272/36938 -- GLOBAL_STEP: 230900 | > loss_text_ce: 0.04506448656320572 (0.044583205736143376) | > loss_mel_ce: 2.5587825775146484 (nan) | > loss: 0.010332726873457432 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13618921760447195) | > loader_time: 0.0048 (0.00466823457025884)  --> TIME: 2025-02-24 07:52:26 -- STEP: 9322/36938 -- GLOBAL_STEP: 230950 | > loss_text_ce: 0.052106037735939026 (0.044578495866210814) | > loss_mel_ce: 2.9692094326019287 (nan) | > loss: 0.011989348568022251 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.1361696520886999) | > loader_time: 0.0046 (0.004668186570768872)  --> TIME: 2025-02-24 07:52:35 -- STEP: 9372/36938 -- GLOBAL_STEP: 231000 | > loss_text_ce: 0.04293680936098099 (0.04457245339674063) | > loss_mel_ce: 2.9237215518951416 (nan) | > loss: 0.011772454716265202 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13616865435778172) | > loader_time: 0.0046 (0.0046681320112686615)  --> TIME: 2025-02-24 07:52:44 -- STEP: 9422/36938 -- GLOBAL_STEP: 231050 | > loss_text_ce: 0.04793654754757881 (0.044574449382678065) | > loss_mel_ce: 0.268525630235672 (nan) | > loss: 0.0012558024609461427 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13618510125220365) | > loader_time: 0.0046 (0.004668165811989309)  --> TIME: 2025-02-24 07:52:53 -- STEP: 9472/36938 -- GLOBAL_STEP: 231100 | > loss_text_ce: 0.040162257850170135 (0.0445786275740829) | > loss_mel_ce: 2.9459574222564697 (nan) | > loss: 0.011849681846797466 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.1361497321946397) | > loader_time: 0.0045 (0.00466804118273225)  --> TIME: 2025-02-24 07:53:03 -- STEP: 9522/36938 -- GLOBAL_STEP: 231150 | > loss_text_ce: 0.057081904262304306 (0.044577394672774065) | > loss_mel_ce: 2.8268115520477295 (nan) | > loss: 0.011444022879004478 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13617467910295486) | > loader_time: 0.0048 (0.004668027131253486)  --> TIME: 2025-02-24 07:53:12 -- STEP: 9572/36938 -- GLOBAL_STEP: 231200 | > loss_text_ce: 0.02876141294836998 (0.044575530740074705) | > loss_mel_ce: 3.0705459117889404 (nan) | > loss: 0.012298839166760445 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13615885119133306) | > loader_time: 0.0046 (0.004668241881248189)  --> TIME: 2025-02-24 07:53:21 -- STEP: 9622/36938 -- GLOBAL_STEP: 231250 | > loss_text_ce: 0.035323988646268845 (0.04457482521355801) | > loss_mel_ce: 3.6435530185699463 (nan) | > loss: 0.014598719775676727 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13615527224377177) | > loader_time: 0.0046 (0.004668180423673923)  --> TIME: 2025-02-24 07:53:31 -- STEP: 9672/36938 -- GLOBAL_STEP: 231300 | > loss_text_ce: 0.04285259544849396 (0.04457379271340995) | > loss_mel_ce: 3.3531229496002197 (nan) | > loss: 0.013476094231009483 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.1361477435423975) | > loader_time: 0.0046 (0.004668008970563514)  --> TIME: 2025-02-24 07:53:40 -- STEP: 9722/36938 -- GLOBAL_STEP: 231350 | > loss_text_ce: 0.05018435791134834 (0.04457083810408199) | > loss_mel_ce: 2.542332887649536 (nan) | > loss: 0.01028776727616787 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13612220236313877) | > loader_time: 0.0048 (0.004667950765362142)  --> TIME: 2025-02-24 07:53:49 -- STEP: 9772/36938 -- GLOBAL_STEP: 231400 | > loss_text_ce: 0.04493672400712967 (0.044573818294371764) | > loss_mel_ce: 3.8843510150909424 (nan) | > loss: 0.01559241209179163 (nan) | > current_lr: 5e-06 | > step_time: 0.1688 (0.1361327714184196) | > loader_time: 0.0048 (0.004668091561430252)  --> TIME: 2025-02-24 07:53:58 -- STEP: 9822/36938 -- GLOBAL_STEP: 231450 | > loss_text_ce: 0.04297835752367973 (0.044568770694427044) | > loss_mel_ce: 3.126453399658203 (nan) | > loss: 0.012577110901474953 (nan) | > current_lr: 5e-06 | > step_time: 0.19 (0.13614027511318583) | > loader_time: 0.0048 (0.0046678207673601126)  --> TIME: 2025-02-24 07:54:08 -- STEP: 9872/36938 -- GLOBAL_STEP: 231500 | > loss_text_ce: 0.03836453706026077 (0.044555890379357255) | > loss_mel_ce: 0.4929111897945404 (nan) | > loss: 0.0021082372404634953 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13612564354893436) | > loader_time: 0.0046 (0.004667701027574948)  --> TIME: 2025-02-24 07:54:17 -- STEP: 9922/36938 -- GLOBAL_STEP: 231550 | > loss_text_ce: 0.04789932072162628 (0.04456172020709016) | > loss_mel_ce: 2.239124298095703 (nan) | > loss: 0.00907549075782299 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13613453434734787) | > loader_time: 0.0046 (0.004667625146584018)  --> TIME: 2025-02-24 07:54:26 -- STEP: 9972/36938 -- GLOBAL_STEP: 231600 | > loss_text_ce: 0.04879944398999214 (0.04457107593701016) | > loss_mel_ce: 2.953125 (nan) | > loss: 0.011912399902939796 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13611001766115302) | > loader_time: 0.0045 (0.0046673916307166914)  --> TIME: 2025-02-24 07:54:35 -- STEP: 10022/36938 -- GLOBAL_STEP: 231650 | > loss_text_ce: 0.0314258374273777 (0.044576157056534095) | > loss_mel_ce: 3.0153682231903076 (nan) | > loss: 0.012090453878045082 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13610464423698607) | > loader_time: 0.0046 (0.004667293475883195)  --> TIME: 2025-02-24 07:54:44 -- STEP: 10072/36938 -- GLOBAL_STEP: 231700 | > loss_text_ce: 0.03776365891098976 (0.044570709145394835) | > loss_mel_ce: 2.7931458950042725 (nan) | > loss: 0.011233768425881863 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.1360878371311804) | > loader_time: 0.0047 (0.004667090058232024)  --> TIME: 2025-02-24 07:54:54 -- STEP: 10122/36938 -- GLOBAL_STEP: 231750 | > loss_text_ce: 0.041788287460803986 (0.04456568000324004) | > loss_mel_ce: 2.7002460956573486 (nan) | > loss: 0.010881089605391026 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13609628447947372) | > loader_time: 0.0046 (0.004667121038868394)  --> TIME: 2025-02-24 07:55:03 -- STEP: 10172/36938 -- GLOBAL_STEP: 231800 | > loss_text_ce: 0.05237044394016266 (0.044553876706522354) | > loss_mel_ce: 3.6929850578308105 (nan) | > loss: 0.014862523414194584 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13607568931635816) | > loader_time: 0.0049 (0.004666927781481924)  --> TIME: 2025-02-24 07:55:12 -- STEP: 10222/36938 -- GLOBAL_STEP: 231850 | > loss_text_ce: 0.043740611523389816 (0.04455586893779548) | > loss_mel_ce: 3.91245698928833 (nan) | > loss: 0.015699196606874466 (nan) | > current_lr: 5e-06 | > step_time: 0.161 (0.13607883353075648) | > loader_time: 0.0047 (0.004666961771801375)  --> TIME: 2025-02-24 07:55:21 -- STEP: 10272/36938 -- GLOBAL_STEP: 231900 | > loss_text_ce: 0.05257917195558548 (0.04455565783845083) | > loss_mel_ce: 3.1867120265960693 (nan) | > loss: 0.01285433117300272 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13609332709668934) | > loader_time: 0.0045 (0.0046668143426517724)  --> TIME: 2025-02-24 07:55:31 -- STEP: 10322/36938 -- GLOBAL_STEP: 231950 | > loss_text_ce: 0.04832198843359947 (0.04456049216865353) | > loss_mel_ce: 4.118162631988525 (nan) | > loss: 0.01653367094695568 (nan) | > current_lr: 5e-06 | > step_time: 0.1821 (0.13608307820885354) | > loader_time: 0.0048 (0.0046667284892525085)  --> TIME: 2025-02-24 07:55:40 -- STEP: 10372/36938 -- GLOBAL_STEP: 232000 | > loss_text_ce: 0.04784376919269562 (0.044554027421468086) | > loss_mel_ce: 3.5136024951934814 (nan) | > loss: 0.014132724143564701 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13607402618814946) | > loader_time: 0.0045 (0.004666434720911389)  --> TIME: 2025-02-24 07:55:49 -- STEP: 10422/36938 -- GLOBAL_STEP: 232050 | > loss_text_ce: 0.041357602924108505 (0.04455257236429951) | > loss_mel_ce: 1.5885416269302368 (nan) | > loss: 0.006467854604125023 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.1360675293478252) | > loader_time: 0.0045 (0.0046664179229297074)  --> TIME: 2025-02-24 07:55:58 -- STEP: 10472/36938 -- GLOBAL_STEP: 232100 | > loss_text_ce: 0.03412903472781181 (0.04454364701077432) | > loss_mel_ce: 2.3435139656066895 (nan) | > loss: 0.009435092099010944 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13605744650657148) | > loader_time: 0.0046 (0.004666476713235912)  --> TIME: 2025-02-24 07:56:07 -- STEP: 10522/36938 -- GLOBAL_STEP: 232150 | > loss_text_ce: 0.032351620495319366 (0.04453737470109808) | > loss_mel_ce: 3.086780071258545 (nan) | > loss: 0.012377507984638214 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1360343506616729) | > loader_time: 0.0046 (0.004666482013252111)  --> TIME: 2025-02-24 07:56:16 -- STEP: 10572/36938 -- GLOBAL_STEP: 232200 | > loss_text_ce: 0.043278563767671585 (0.04453824296987612) | > loss_mel_ce: 3.8189358711242676 (nan) | > loss: 0.015326248481869698 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1360244350003959) | > loader_time: 0.0047 (0.004666662130489338)  --> TIME: 2025-02-24 07:56:25 -- STEP: 10622/36938 -- GLOBAL_STEP: 232250 | > loss_text_ce: 0.039852168411016464 (0.04453545221082149) | > loss_mel_ce: 3.106130599975586 (nan) | > loss: 0.012484059669077396 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13601051643280337) | > loader_time: 0.0047 (0.0046665274564929434)  --> TIME: 2025-02-24 07:56:35 -- STEP: 10672/36938 -- GLOBAL_STEP: 232300 | > loss_text_ce: 0.06251978129148483 (0.04453548726069317) | > loss_mel_ce: 3.0543746948242188 (nan) | > loss: 0.012368629686534405 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.1359959492261618) | > loader_time: 0.0045 (0.004666349497334711)  --> TIME: 2025-02-24 07:56:44 -- STEP: 10722/36938 -- GLOBAL_STEP: 232350 | > loss_text_ce: 0.04489422217011452 (0.044523445282543435) | > loss_mel_ce: 3.4543614387512207 (nan) | > loss: 0.01388593576848507 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.1359748868572244) | > loader_time: 0.005 (0.004666561823330628)  --> TIME: 2025-02-24 07:56:53 -- STEP: 10772/36938 -- GLOBAL_STEP: 232400 | > loss_text_ce: 0.056394319981336594 (0.04451284918956925) | > loss_mel_ce: 3.6036789417266846 (nan) | > loss: 0.01452410127967596 (nan) | > current_lr: 5e-06 | > step_time: 0.1488 (0.1360142151086778) | > loader_time: 0.0047 (0.004666525703179454)  --> TIME: 2025-02-24 07:57:02 -- STEP: 10822/36938 -- GLOBAL_STEP: 232450 | > loss_text_ce: 0.03094925545156002 (0.044508194615931855) | > loss_mel_ce: 3.071268081665039 (nan) | > loss: 0.012310387566685677 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13599027243014736) | > loader_time: 0.0046 (0.004666452354081482)  --> TIME: 2025-02-24 07:57:12 -- STEP: 10872/36938 -- GLOBAL_STEP: 232500 | > loss_text_ce: 0.033305201679468155 (0.0445062089396189) | > loss_mel_ce: 3.3214375972747803 (nan) | > loss: 0.01331247203052044 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13598578602536052) | > loader_time: 0.0047 (0.004666452683272656)  --> TIME: 2025-02-24 07:57:21 -- STEP: 10922/36938 -- GLOBAL_STEP: 232550 | > loss_text_ce: 0.03493743762373924 (0.04449848266261472) | > loss_mel_ce: 3.0500526428222656 (nan) | > loss: 0.012242024764418602 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13597816059690207) | > loader_time: 0.0046 (0.004666201558835212)  --> TIME: 2025-02-24 07:57:30 -- STEP: 10972/36938 -- GLOBAL_STEP: 232600 | > loss_text_ce: 0.03734589368104935 (0.04450265784546549) | > loss_mel_ce: 4.196948528289795 (nan) | > loss: 0.016802756115794182 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13597399543942262) | > loader_time: 0.0045 (0.004665983514198198)  --> TIME: 2025-02-24 07:57:39 -- STEP: 11022/36938 -- GLOBAL_STEP: 232650 | > loss_text_ce: 0.040377020835876465 (0.04450330125160185) | > loss_mel_ce: 3.4576733112335205 (nan) | > loss: 0.013881152495741844 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13598816811421893) | > loader_time: 0.0045 (0.0046658461419667825)  --> TIME: 2025-02-24 07:57:48 -- STEP: 11072/36938 -- GLOBAL_STEP: 232700 | > loss_text_ce: 0.04370884224772453 (0.04450115453851578) | > loss_mel_ce: 3.101160764694214 (nan) | > loss: 0.01247964147478342 (nan) | > current_lr: 5e-06 | > step_time: 0.1606 (0.13596573592610403) | > loader_time: 0.0048 (0.004665628355534791)  --> TIME: 2025-02-24 07:57:58 -- STEP: 11122/36938 -- GLOBAL_STEP: 232750 | > loss_text_ce: 0.023375196382403374 (0.044499752977238224) | > loss_mel_ce: 3.5277693271636963 (nan) | > loss: 0.01409184467047453 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1359709842752536) | > loader_time: 0.0045 (0.0046656028419663495)  --> TIME: 2025-02-24 07:58:07 -- STEP: 11172/36938 -- GLOBAL_STEP: 232800 | > loss_text_ce: 0.04328839108347893 (0.04450291780937645) | > loss_mel_ce: 3.564500570297241 (nan) | > loss: 0.014316624030470848 (nan) | > current_lr: 5e-06 | > step_time: 0.1885 (0.13597590563504655) | > loader_time: 0.0048 (0.004665700906158745)  --> TIME: 2025-02-24 07:58:17 -- STEP: 11222/36938 -- GLOBAL_STEP: 232850 | > loss_text_ce: 0.0335020087659359 (0.04449974404785039) | > loss_mel_ce: 3.465198516845703 (nan) | > loss: 0.013883733190596104 (nan) | > current_lr: 5e-06 | > step_time: 0.1775 (0.13600993156433053) | > loader_time: 0.0048 (0.0046656468912701626)  --> TIME: 2025-02-24 07:58:27 -- STEP: 11272/36938 -- GLOBAL_STEP: 232900 | > loss_text_ce: 0.04323473945260048 (0.044498040452959686) | > loss_mel_ce: 3.3498003482818604 (nan) | > loss: 0.013464425690472126 (nan) | > current_lr: 5e-06 | > step_time: 0.17 (0.13602667069756572) | > loader_time: 0.0048 (0.004665639698293899)  --> TIME: 2025-02-24 07:58:36 -- STEP: 11322/36938 -- GLOBAL_STEP: 232950 | > loss_text_ce: 0.03930151090025902 (0.04449323668223437) | > loss_mel_ce: 3.5456886291503906 (nan) | > loss: 0.014226151630282402 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.1360319796604138) | > loader_time: 0.0045 (0.0046654578296356245)  --> TIME: 2025-02-24 07:58:45 -- STEP: 11372/36938 -- GLOBAL_STEP: 233000 | > loss_text_ce: 0.0446433387696743 (0.0445022669276356) | > loss_mel_ce: 0.91766357421875 (nan) | > loss: 0.0038186784368008375 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.1360422944544567) | > loader_time: 0.0046 (0.004665135855075983)  --> TIME: 2025-02-24 07:58:55 -- STEP: 11422/36938 -- GLOBAL_STEP: 233050 | > loss_text_ce: 0.03473168611526489 (0.04450159996408441) | > loss_mel_ce: 1.633982539176941 (nan) | > loss: 0.006621882319450378 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13605513359364352) | > loader_time: 0.0044 (0.004664866483017404)  --> TIME: 2025-02-24 07:59:04 -- STEP: 11472/36938 -- GLOBAL_STEP: 233100 | > loss_text_ce: 0.036854956299066544 (0.04450237589056637) | > loss_mel_ce: 3.50058913230896 (nan) | > loss: 0.014037477783858776 (nan) | > current_lr: 5e-06 | > step_time: 0.1065 (0.13607346554347782) | > loader_time: 0.0045 (0.004665085461349178)  --> TIME: 2025-02-24 07:59:14 -- STEP: 11522/36938 -- GLOBAL_STEP: 233150 | > loss_text_ce: 0.03905162587761879 (0.04449737139880026) | > loss_mel_ce: 3.5398335456848145 (nan) | > loss: 0.014201926067471504 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13608303731813345) | > loader_time: 0.0047 (0.004665040448398531)  --> TIME: 2025-02-24 07:59:23 -- STEP: 11572/36938 -- GLOBAL_STEP: 233200 | > loss_text_ce: 0.03519197925925255 (0.044496587778702607) | > loss_mel_ce: 3.0138230323791504 (nan) | > loss: 0.012099266983568668 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.1361138668597627) | > loader_time: 0.0046 (0.004664919799148033)  --> TIME: 2025-02-24 07:59:32 -- STEP: 11622/36938 -- GLOBAL_STEP: 233250 | > loss_text_ce: 0.05256522074341774 (0.04449207289561088) | > loss_mel_ce: 3.156663179397583 (nan) | > loss: 0.012735034339129925 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13608932246892677) | > loader_time: 0.0044 (0.004664799141773079)  --> TIME: 2025-02-24 07:59:42 -- STEP: 11672/36938 -- GLOBAL_STEP: 233300 | > loss_text_ce: 0.048453815281391144 (0.04448137367937344) | > loss_mel_ce: 3.2523679733276367 (nan) | > loss: 0.013098499737679958 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13608827486358463) | > loader_time: 0.0046 (0.004664688076852041)  --> TIME: 2025-02-24 07:59:51 -- STEP: 11722/36938 -- GLOBAL_STEP: 233350 | > loss_text_ce: 0.04447249323129654 (0.0444787616219871) | > loss_mel_ce: 3.982921838760376 (nan) | > loss: 0.015981724485754967 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13610310032595074) | > loader_time: 0.0046 (0.004664730667722167)  --> TIME: 2025-02-24 08:00:00 -- STEP: 11772/36938 -- GLOBAL_STEP: 233400 | > loss_text_ce: 0.046355098485946655 (0.04448244551197003) | > loss_mel_ce: 3.7761294841766357 (nan) | > loss: 0.015168590471148491 (nan) | > current_lr: 5e-06 | > step_time: 0.1578 (0.13609183004750608) | > loader_time: 0.0046 (0.004664377456532838)  --> TIME: 2025-02-24 08:00:10 -- STEP: 11822/36938 -- GLOBAL_STEP: 233450 | > loss_text_ce: 0.03322659060359001 (0.04448217635557068) | > loss_mel_ce: 2.896925926208496 (nan) | > loss: 0.011627589352428913 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.1360978536689729) | > loader_time: 0.0045 (0.0046641488422941805)  --> TIME: 2025-02-24 08:00:19 -- STEP: 11872/36938 -- GLOBAL_STEP: 233500 | > loss_text_ce: 0.03292672708630562 (0.04447096984369379) | > loss_mel_ce: 3.243156909942627 (nan) | > loss: 0.01300033275038004 (nan) | > current_lr: 5e-06 | > step_time: 0.1822 (0.1361199609152067) | > loader_time: 0.0048 (0.004663988224900968)  --> TIME: 2025-02-24 08:00:29 -- STEP: 11922/36938 -- GLOBAL_STEP: 233550 | > loss_text_ce: 0.041400711983442307 (0.044466421656879807) | > loss_mel_ce: 0.03466796875 (nan) | > loss: 0.0003018598654307425 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13611892333188275) | > loader_time: 0.0045 (0.004663797717549592)  --> TIME: 2025-02-24 08:00:38 -- STEP: 11972/36938 -- GLOBAL_STEP: 233600 | > loss_text_ce: 0.04653755947947502 (0.044466010198470615) | > loss_mel_ce: 3.8233065605163574 (nan) | > loss: 0.015356525778770447 (nan) | > current_lr: 5e-06 | > step_time: 0.1591 (0.13613564080710855) | > loader_time: 0.0049 (0.004663658468690644)  --> TIME: 2025-02-24 08:00:47 -- STEP: 12022/36938 -- GLOBAL_STEP: 233650 | > loss_text_ce: 0.03469393029808998 (0.044470376368783024) | > loss_mel_ce: 0.36529314517974854 (nan) | > loss: 0.001587250386364758 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.1361283456559307) | > loader_time: 0.0045 (0.004663316308311734)  --> TIME: 2025-02-24 08:00:57 -- STEP: 12072/36938 -- GLOBAL_STEP: 233700 | > loss_text_ce: 0.039995864033699036 (0.04446854340376953) | > loss_mel_ce: 2.375446081161499 (nan) | > loss: 0.00958508811891079 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.1361348868364366) | > loader_time: 0.0045 (0.004663071958650582)  --> TIME: 2025-02-24 08:01:06 -- STEP: 12122/36938 -- GLOBAL_STEP: 233750 | > loss_text_ce: 0.04364345967769623 (0.04446756721227909) | > loss_mel_ce: 3.4088621139526367 (nan) | > loss: 0.013700420036911964 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13613043927336013) | > loader_time: 0.0046 (0.004662735590819108)  --> TIME: 2025-02-24 08:01:15 -- STEP: 12172/36938 -- GLOBAL_STEP: 233800 | > loss_text_ce: 0.08420948684215546 (0.04446703253775727) | > loss_mel_ce: 0.7716869115829468 (nan) | > loss: 0.003396414453163743 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13612963392904548) | > loader_time: 0.0045 (0.004662403240040894)  --> TIME: 2025-02-24 08:01:24 -- STEP: 12222/36938 -- GLOBAL_STEP: 233850 | > loss_text_ce: 0.04047754034399986 (0.04446304770265175) | > loss_mel_ce: 3.31803035736084 (nan) | > loss: 0.013327413238584995 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13612317693660703) | > loader_time: 0.0045 (0.004662051623785606)  --> TIME: 2025-02-24 08:01:33 -- STEP: 12272/36938 -- GLOBAL_STEP: 233900 | > loss_text_ce: 0.05124653875827789 (0.04446774445945569) | > loss_mel_ce: 3.4830729961395264 (nan) | > loss: 0.01402507908642292 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13611626648374456) | > loader_time: 0.0046 (0.004661782662902698)  --> TIME: 2025-02-24 08:01:43 -- STEP: 12322/36938 -- GLOBAL_STEP: 233950 | > loss_text_ce: 0.04526190087199211 (0.04446617479709246) | > loss_mel_ce: 2.984572172164917 (nan) | > loss: 0.012023151852190495 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.13614009309368397) | > loader_time: 0.0046 (0.004661693702249483)  --> TIME: 2025-02-24 08:01:52 -- STEP: 12372/36938 -- GLOBAL_STEP: 234000 | > loss_text_ce: 0.04143014922738075 (0.04446594422643065) | > loss_mel_ce: 3.123185873031616 (nan) | > loss: 0.012558001093566418 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13614065528417008) | > loader_time: 0.0045 (0.004661497023743107)  --> TIME: 2025-02-24 08:02:01 -- STEP: 12422/36938 -- GLOBAL_STEP: 234050 | > loss_text_ce: 0.042812395840883255 (0.04446070286526434) | > loss_mel_ce: 3.7099080085754395 (nan) | > loss: 0.014891748316586018 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13613319690629425) | > loader_time: 0.0048 (0.004661318357968527)  --> TIME: 2025-02-24 08:02:11 -- STEP: 12472/36938 -- GLOBAL_STEP: 234100 | > loss_text_ce: 0.05504225939512253 (0.044459419563574486) | > loss_mel_ce: 3.676051139831543 (nan) | > loss: 0.014805926941335201 (nan) | > current_lr: 5e-06 | > step_time: 0.1701 (0.1361382388013385) | > loader_time: 0.0049 (0.004661083144970327)  --> TIME: 2025-02-24 08:02:20 -- STEP: 12522/36938 -- GLOBAL_STEP: 234150 | > loss_text_ce: 0.036256227642297745 (0.04445439722498158) | > loss_mel_ce: 2.6023662090301514 (nan) | > loss: 0.010470724664628506 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.13613365509890463) | > loader_time: 0.0049 (0.004660877456415244)  --> TIME: 2025-02-24 08:02:29 -- STEP: 12572/36938 -- GLOBAL_STEP: 234200 | > loss_text_ce: 0.052263591438531876 (0.044461651746763746) | > loss_mel_ce: 3.121969699859619 (nan) | > loss: 0.012596163898706436 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1361239131301822) | > loader_time: 0.0045 (0.00466069568694113)  --> TIME: 2025-02-24 08:02:38 -- STEP: 12622/36938 -- GLOBAL_STEP: 234250 | > loss_text_ce: 0.04046522080898285 (0.04446484097331989) | > loss_mel_ce: 2.4304370880126953 (nan) | > loss: 0.009805168025195599 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13609379238115968) | > loader_time: 0.0046 (0.004660428165350574)  --> TIME: 2025-02-24 08:02:47 -- STEP: 12672/36938 -- GLOBAL_STEP: 234300 | > loss_text_ce: 0.04314512014389038 (0.044460680805022565) | > loss_mel_ce: 0.7731765508651733 (nan) | > loss: 0.0032393718138337135 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.1360896491831301) | > loader_time: 0.0046 (0.004660372086095096)  --> TIME: 2025-02-24 08:02:56 -- STEP: 12722/36938 -- GLOBAL_STEP: 234350 | > loss_text_ce: 0.0578087642788887 (0.044462302166381695) | > loss_mel_ce: 2.7718026638031006 (nan) | > loss: 0.011228618212044239 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13606574068352834) | > loader_time: 0.0045 (0.004660208239118334)  --> TIME: 2025-02-24 08:03:05 -- STEP: 12772/36938 -- GLOBAL_STEP: 234400 | > loss_text_ce: 0.047445494681596756 (0.04446234407973539) | > loss_mel_ce: 3.530944347381592 (nan) | > loss: 0.014199960976839066 (nan) | > current_lr: 5e-06 | > step_time: 0.1478 (0.13606821646930822) | > loader_time: 0.0046 (0.0046601572680391255)  --> TIME: 2025-02-24 08:03:14 -- STEP: 12822/36938 -- GLOBAL_STEP: 234450 | > loss_text_ce: 0.05667184665799141 (0.04445813666827792) | > loss_mel_ce: 2.547550678253174 (nan) | > loss: 0.010334216989576817 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1360475809111869) | > loader_time: 0.0046 (0.0046601787109565454)  --> TIME: 2025-02-24 08:03:24 -- STEP: 12872/36938 -- GLOBAL_STEP: 234500 | > loss_text_ce: 0.05632399022579193 (0.04444946693376595) | > loss_mel_ce: 3.276040554046631 (nan) | > loss: 0.013223669491708279 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13604491932411392) | > loader_time: 0.0048 (0.004660131325256465)  --> TIME: 2025-02-24 08:03:33 -- STEP: 12922/36938 -- GLOBAL_STEP: 234550 | > loss_text_ce: 0.04644610360264778 (0.04445059536987235) | > loss_mel_ce: 3.2461416721343994 (nan) | > loss: 0.013065825216472149 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.13604843950441353) | > loader_time: 0.005 (0.004660153920354169)  --> TIME: 2025-02-24 08:03:43 -- STEP: 12972/36938 -- GLOBAL_STEP: 234600 | > loss_text_ce: 0.051888272166252136 (0.04444873836467273) | > loss_mel_ce: 3.633011817932129 (nan) | > loss: 0.014622620306909084 (nan) | > current_lr: 5e-06 | > step_time: 0.173 (0.13605732448006205) | > loader_time: 0.0048 (0.004660290018335157)  --> TIME: 2025-02-24 08:03:52 -- STEP: 13022/36938 -- GLOBAL_STEP: 234650 | > loss_text_ce: 0.04934549331665039 (0.04444543847564164) | > loss_mel_ce: 2.948373317718506 (nan) | > loss: 0.011895710602402687 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13605291908227837) | > loader_time: 0.0047 (0.004660282206780683)  --> TIME: 2025-02-24 08:04:01 -- STEP: 13072/36938 -- GLOBAL_STEP: 234700 | > loss_text_ce: 0.04316512495279312 (0.04444036711994244) | > loss_mel_ce: 3.951780319213867 (nan) | > loss: 0.01585295982658863 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13606066106290834) | > loader_time: 0.0046 (0.0046601718430711765)  --> TIME: 2025-02-24 08:04:10 -- STEP: 13122/36938 -- GLOBAL_STEP: 234750 | > loss_text_ce: 0.05257118493318558 (0.04444214924333863) | > loss_mel_ce: 2.50205135345459 (nan) | > loss: 0.010137392207980156 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13604158930726912) | > loader_time: 0.0045 (0.004660091645796482)  --> TIME: 2025-02-24 08:04:20 -- STEP: 13172/36938 -- GLOBAL_STEP: 234800 | > loss_text_ce: 0.04679809510707855 (0.044444475860246906) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13605548656352412) | > loader_time: 0.0046 (0.004660064747657171)  --> TIME: 2025-02-24 08:04:29 -- STEP: 13222/36938 -- GLOBAL_STEP: 234850 | > loss_text_ce: 0.04981119558215141 (0.04444557425465004) | > loss_mel_ce: 3.168701171875 (nan) | > loss: 0.01277187466621399 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13605788309082573) | > loader_time: 0.0045 (0.0046601586867669985)  --> TIME: 2025-02-24 08:04:38 -- STEP: 13272/36938 -- GLOBAL_STEP: 234900 | > loss_text_ce: 0.04887084290385246 (0.044452859529809347) | > loss_mel_ce: 0.0660807266831398 (nan) | > loss: 0.00045615702401846647 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13605506377740384) | > loader_time: 0.0045 (0.0046600450622956485)  --> TIME: 2025-02-24 08:04:47 -- STEP: 13322/36938 -- GLOBAL_STEP: 234950 | > loss_text_ce: 0.044427260756492615 (0.044451289405259595) | > loss_mel_ce: 2.644831657409668 (nan) | > loss: 0.010671662166714668 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13603257975444444) | > loader_time: 0.0047 (0.004659861957073009)  --> TIME: 2025-02-24 08:04:57 -- STEP: 13372/36938 -- GLOBAL_STEP: 235000 | > loss_text_ce: 0.048430416733026505 (0.04445478611100863) | > loss_mel_ce: 0.009978711605072021 (nan) | > loss: 0.00023178226547315717 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13605959546305704) | > loader_time: 0.0045 (0.0046598960317040675)  --> TIME: 2025-02-24 08:05:06 -- STEP: 13422/36938 -- GLOBAL_STEP: 235050 | > loss_text_ce: 0.05101795494556427 (0.04445326671337627) | > loss_mel_ce: 3.6373908519744873 (nan) | > loss: 0.014636543579399586 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13605779372993881) | > loader_time: 0.0046 (0.0046597577619261425)  --> TIME: 2025-02-24 08:05:15 -- STEP: 13472/36938 -- GLOBAL_STEP: 235100 | > loss_text_ce: 0.039024196565151215 (0.04445583138700884) | > loss_mel_ce: 0.0986497700214386 (nan) | > loss: 0.0005463253473863006 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13605527236456003) | > loader_time: 0.0045 (0.004659745249357485)  --> TIME: 2025-02-24 08:05:25 -- STEP: 13522/36938 -- GLOBAL_STEP: 235150 | > loss_text_ce: 0.05985351279377937 (0.0444618985182917) | > loss_mel_ce: 1.884385347366333 (nan) | > loss: 0.007715234067291021 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13606541704415695) | > loader_time: 0.0045 (0.004659727275274584)  --> TIME: 2025-02-24 08:05:34 -- STEP: 13572/36938 -- GLOBAL_STEP: 235200 | > loss_text_ce: 0.04344726353883743 (0.04445876038524585) | > loss_mel_ce: 3.2192609310150146 (nan) | > loss: 0.012947255745530128 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13605293258256104) | > loader_time: 0.0045 (0.004659701528501422)  --> TIME: 2025-02-24 08:05:43 -- STEP: 13622/36938 -- GLOBAL_STEP: 235250 | > loss_text_ce: 0.05268734693527222 (0.04445683890094938) | > loss_mel_ce: 3.4251585006713867 (nan) | > loss: 0.013800976797938347 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13604303035349666) | > loader_time: 0.0046 (0.0046596388480086145)  --> TIME: 2025-02-24 08:05:52 -- STEP: 13672/36938 -- GLOBAL_STEP: 235300 | > loss_text_ce: 0.04746603965759277 (0.044457217392532306) | > loss_mel_ce: 3.430426597595215 (nan) | > loss: 0.013801162131130695 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13603846591909877) | > loader_time: 0.0047 (0.004659668753341048)  --> TIME: 2025-02-24 08:06:01 -- STEP: 13722/36938 -- GLOBAL_STEP: 235350 | > loss_text_ce: 0.051341429352760315 (0.04445570010417405) | > loss_mel_ce: 3.302868127822876 (nan) | > loss: 0.013310356996953487 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.1360283170387884) | > loader_time: 0.0046 (0.004659594121748674)  --> TIME: 2025-02-24 08:06:11 -- STEP: 13772/36938 -- GLOBAL_STEP: 235400 | > loss_text_ce: 0.042235683649778366 (0.04445257209285475) | > loss_mel_ce: 3.182642698287964 (nan) | > loss: 0.012797136791050434 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13603158063608825) | > loader_time: 0.0047 (0.004659573975739494)  --> TIME: 2025-02-24 08:06:20 -- STEP: 13822/36938 -- GLOBAL_STEP: 235450 | > loss_text_ce: 0.03814280778169632 (0.04445127352934436) | > loss_mel_ce: 3.504687547683716 (nan) | > loss: 0.014058851636946201 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13602884556446485) | > loader_time: 0.0045 (0.004659540365856925)  --> TIME: 2025-02-24 08:06:29 -- STEP: 13872/36938 -- GLOBAL_STEP: 235500 | > loss_text_ce: 0.03287581726908684 (0.04445002167444023) | > loss_mel_ce: 1.846402883529663 (nan) | > loss: 0.00745745562016964 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13603285061584386) | > loader_time: 0.0046 (0.004659463411933963)  --> TIME: 2025-02-24 08:06:38 -- STEP: 13922/36938 -- GLOBAL_STEP: 235550 | > loss_text_ce: 0.04487016424536705 (0.04445348649775806) | > loss_mel_ce: 3.7134153842926025 (nan) | > loss: 0.0149138318374753 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.1360306797176924) | > loader_time: 0.0047 (0.004659424977575873)  --> TIME: 2025-02-24 08:06:48 -- STEP: 13972/36938 -- GLOBAL_STEP: 235600 | > loss_text_ce: 0.030927544459700584 (0.0444447514772358) | > loss_mel_ce: 2.070026159286499 (nan) | > loss: 0.008337118662893772 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13602299452032818) | > loader_time: 0.0045 (0.004659349021480245)  --> TIME: 2025-02-24 08:06:57 -- STEP: 14022/36938 -- GLOBAL_STEP: 235650 | > loss_text_ce: 0.04662678763270378 (0.044444695686621995) | > loss_mel_ce: 3.5857017040252686 (nan) | > loss: 0.014414003118872643 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.13602243464813696) | > loader_time: 0.0048 (0.004659206274487192)  --> TIME: 2025-02-24 08:07:06 -- STEP: 14072/36938 -- GLOBAL_STEP: 235700 | > loss_text_ce: 0.03550438955426216 (0.04444631155255451) | > loss_mel_ce: 3.067091941833496 (nan) | > loss: 0.012311890721321106 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13603266485717425) | > loader_time: 0.0045 (0.004659233579857911)  --> TIME: 2025-02-24 08:07:16 -- STEP: 14122/36938 -- GLOBAL_STEP: 235750 | > loss_text_ce: 0.035364579409360886 (0.04445591404953074) | > loss_mel_ce: 1.5429606437683105 (nan) | > loss: 0.006263196002691984 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13603526395198912) | > loader_time: 0.0045 (0.004659144031885006)  --> TIME: 2025-02-24 08:07:25 -- STEP: 14172/36938 -- GLOBAL_STEP: 235800 | > loss_text_ce: 0.0378105528652668 (0.0444529035394538) | > loss_mel_ce: 3.647421360015869 (nan) | > loss: 0.01462393719702959 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.1360569616983145) | > loader_time: 0.0046 (0.004659169311749543)  --> TIME: 2025-02-24 08:07:35 -- STEP: 14222/36938 -- GLOBAL_STEP: 235850 | > loss_text_ce: 0.04867622256278992 (0.0444527230412265) | > loss_mel_ce: 3.68758487701416 (nan) | > loss: 0.01482643373310566 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.13606339809579773) | > loader_time: 0.0049 (0.004659105681648841)  --> TIME: 2025-02-24 08:07:44 -- STEP: 14272/36938 -- GLOBAL_STEP: 235900 | > loss_text_ce: 0.04244389757514 (0.044450122178070116) | > loss_mel_ce: 0.4674173891544342 (nan) | > loss: 0.0020232591778039932 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.13605635768562735) | > loader_time: 0.0046 (0.0046590321901935116)  --> TIME: 2025-02-24 08:07:53 -- STEP: 14322/36938 -- GLOBAL_STEP: 235950 | > loss_text_ce: 0.043419964611530304 (0.04445022892334615) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13604826323564265) | > loader_time: 0.0045 (0.004658911801170263)  --> TIME: 2025-02-24 08:08:03 -- STEP: 14372/36938 -- GLOBAL_STEP: 236000 | > loss_text_ce: 0.03505992516875267 (0.04444554243284727) | > loss_mel_ce: 0.0125732421875 (nan) | > loss: 0.00018902051670011133 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1360551488455912) | > loader_time: 0.0046 (0.004658970367002715)  --> TIME: 2025-02-24 08:08:12 -- STEP: 14422/36938 -- GLOBAL_STEP: 236050 | > loss_text_ce: 0.05008311942219734 (0.04444625765242006) | > loss_mel_ce: 3.2012386322021484 (nan) | > loss: 0.012902071699500084 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13606530405615136) | > loader_time: 0.0046 (0.004659023567271654)  --> TIME: 2025-02-24 08:08:21 -- STEP: 14472/36938 -- GLOBAL_STEP: 236100 | > loss_text_ce: 0.049224987626075745 (0.04445012846739978) | > loss_mel_ce: 3.5656025409698486 (nan) | > loss: 0.014344555325806141 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13606793347348717) | > loader_time: 0.0046 (0.004659053961699547)  --> TIME: 2025-02-24 08:08:30 -- STEP: 14522/36938 -- GLOBAL_STEP: 236150 | > loss_text_ce: 0.03202212229371071 (0.04444798346150457) | > loss_mel_ce: 3.7760417461395264 (nan) | > loss: 0.015111366286873817 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1360583176643723) | > loader_time: 0.0046 (0.004658891320343503)  --> TIME: 2025-02-24 08:08:40 -- STEP: 14572/36938 -- GLOBAL_STEP: 236200 | > loss_text_ce: 0.05555140972137451 (0.0444494124359164) | > loss_mel_ce: 2.614435911178589 (nan) | > loss: 0.010595187544822693 (nan) | > current_lr: 5e-06 | > step_time: 0.1083 (0.13604419330565024) | > loader_time: 0.0047 (0.0046589181477507185)  --> TIME: 2025-02-24 08:08:49 -- STEP: 14622/36938 -- GLOBAL_STEP: 236250 | > loss_text_ce: 0.0531349778175354 (0.04444968454432001) | > loss_mel_ce: 2.2139272689819336 (nan) | > loss: 0.00899627897888422 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13603734655397237) | > loader_time: 0.0045 (0.0046589613743478)  --> TIME: 2025-02-24 08:08:58 -- STEP: 14672/36938 -- GLOBAL_STEP: 236300 | > loss_text_ce: 0.04119911044836044 (0.04444636872911417) | > loss_mel_ce: 3.4159433841705322 (nan) | > loss: 0.013718820177018642 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.1360412427389767) | > loader_time: 0.0048 (0.004658934529241067)  --> TIME: 2025-02-24 08:09:07 -- STEP: 14722/36938 -- GLOBAL_STEP: 236350 | > loss_text_ce: 0.05478765815496445 (0.044445483303910814) | > loss_mel_ce: 3.531498908996582 (nan) | > loss: 0.014231296256184578 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13604829915757605) | > loader_time: 0.0045 (0.004659042574112029)  --> TIME: 2025-02-24 08:09:16 -- STEP: 14772/36938 -- GLOBAL_STEP: 236400 | > loss_text_ce: 0.03140964359045029 (0.04444274550545657) | > loss_mel_ce: 2.562852382659912 (nan) | > loss: 0.010294691659510136 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13603401926517394) | > loader_time: 0.0045 (0.004658934904119688)  --> TIME: 2025-02-24 08:09:26 -- STEP: 14822/36938 -- GLOBAL_STEP: 236450 | > loss_text_ce: 0.046698544174432755 (0.04443977392749733) | > loss_mel_ce: 0.00799560546875 (nan) | > loss: 0.00021704028767999262 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.1360391973081897) | > loader_time: 0.0045 (0.004658772707017646)  --> TIME: 2025-02-24 08:09:35 -- STEP: 14872/36938 -- GLOBAL_STEP: 236500 | > loss_text_ce: 0.04261099547147751 (0.044433498075955764) | > loss_mel_ce: 3.423396110534668 (nan) | > loss: 0.013753997161984444 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.1360434703669159) | > loader_time: 0.0048 (0.004658748957340023)  --> TIME: 2025-02-24 08:09:44 -- STEP: 14922/36938 -- GLOBAL_STEP: 236550 | > loss_text_ce: 0.03544212132692337 (0.04442597162998301) | > loss_mel_ce: 2.99609375 (nan) | > loss: 0.012029904872179031 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13603614915585266) | > loader_time: 0.0045 (0.0046586598903879456)  --> TIME: 2025-02-24 08:09:54 -- STEP: 14972/36938 -- GLOBAL_STEP: 236600 | > loss_text_ce: 0.05174864083528519 (0.044424550251680885) | > loss_mel_ce: 2.130218505859375 (nan) | > loss: 0.008658600971102715 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.1360594503165504) | > loader_time: 0.0046 (0.00465860038462279)  --> TIME: 2025-02-24 08:10:03 -- STEP: 15022/36938 -- GLOBAL_STEP: 236650 | > loss_text_ce: 0.04551605135202408 (0.04442114797539208) | > loss_mel_ce: 3.8983638286590576 (nan) | > loss: 0.015650317072868347 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13605711885781663) | > loader_time: 0.0046 (0.004658472171367494)  --> TIME: 2025-02-24 08:10:13 -- STEP: 15072/36938 -- GLOBAL_STEP: 236700 | > loss_text_ce: 0.045656468719244 (0.04442190247702291) | > loss_mel_ce: 2.915701389312744 (nan) | > loss: 0.011751420795917511 (nan) | > current_lr: 5e-06 | > step_time: 0.168 (0.136062919682252) | > loader_time: 0.0049 (0.004658668584757805)  --> TIME: 2025-02-24 08:10:22 -- STEP: 15122/36938 -- GLOBAL_STEP: 236750 | > loss_text_ce: 0.04275472089648247 (0.044420644497485756) | > loss_mel_ce: 2.1578621864318848 (nan) | > loss: 0.008732606656849384 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.136062168315075) | > loader_time: 0.0047 (0.004658640179900863)  --> TIME: 2025-02-24 08:10:31 -- STEP: 15172/36938 -- GLOBAL_STEP: 236800 | > loss_text_ce: 0.03123231790959835 (0.0444190836990472) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13606054637352302) | > loader_time: 0.0046 (0.0046588433407995485)  --> TIME: 2025-02-24 08:10:40 -- STEP: 15222/36938 -- GLOBAL_STEP: 236850 | > loss_text_ce: 0.0433688722550869 (0.044421254105402375) | > loss_mel_ce: 3.502976179122925 (nan) | > loss: 0.014072798192501068 (nan) | > current_lr: 5e-06 | > step_time: 0.1623 (0.13605190083069332) | > loader_time: 0.0047 (0.004658690452450384)  --> TIME: 2025-02-24 08:10:50 -- STEP: 15272/36938 -- GLOBAL_STEP: 236900 | > loss_text_ce: 0.0454397052526474 (0.04442369568008229) | > loss_mel_ce: 0.582080066204071 (nan) | > loss: 0.002490157959982753 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13604910467981138) | > loader_time: 0.0046 (0.004658710275912799)  --> TIME: 2025-02-24 08:10:59 -- STEP: 15322/36938 -- GLOBAL_STEP: 236950 | > loss_text_ce: 0.04270799458026886 (0.04442823572084788) | > loss_mel_ce: 0.7866760492324829 (nan) | > loss: 0.003291206667199731 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13603901729277543) | > loader_time: 0.0046 (0.00465881298547189)  --> TIME: 2025-02-24 08:11:08 -- STEP: 15372/36938 -- GLOBAL_STEP: 237000 | > loss_text_ce: 0.04882888123393059 (0.044428014079464534) | > loss_mel_ce: 3.7698261737823486 (nan) | > loss: 0.01515339408069849 (nan) | > current_lr: 5e-06 | > step_time: 0.1735 (0.13604101530127716) | > loader_time: 0.0049 (0.0046587538942520195)  --> TIME: 2025-02-24 08:11:17 -- STEP: 15422/36938 -- GLOBAL_STEP: 237050 | > loss_text_ce: 0.046137843281030655 (0.04443138577723987) | > loss_mel_ce: 2.5539631843566895 (nan) | > loss: 0.010317862033843994 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13603592766133657) | > loader_time: 0.0047 (0.0046587097800948295)  --> TIME: 2025-02-24 08:11:27 -- STEP: 15472/36938 -- GLOBAL_STEP: 237100 | > loss_text_ce: 0.0476159006357193 (0.04443218923640223) | > loss_mel_ce: 3.2936220169067383 (nan) | > loss: 0.013258881866931915 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.1360402780950011) | > loader_time: 0.0046 (0.00465889715790872)  --> TIME: 2025-02-24 08:11:36 -- STEP: 15522/36938 -- GLOBAL_STEP: 237150 | > loss_text_ce: 0.040120162069797516 (0.04442704157282663) | > loss_mel_ce: 3.227177381515503 (nan) | > loss: 0.01296546682715416 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.1360373678411382) | > loader_time: 0.0045 (0.004658825924955315)  --> TIME: 2025-02-24 08:11:45 -- STEP: 15572/36938 -- GLOBAL_STEP: 237200 | > loss_text_ce: 0.03745439276099205 (0.04443165537113671) | > loss_mel_ce: 3.0873963832855225 (nan) | > loss: 0.012400202453136444 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13602443123243507) | > loader_time: 0.0048 (0.004659264384092602)  --> TIME: 2025-02-24 08:11:54 -- STEP: 15622/36938 -- GLOBAL_STEP: 237250 | > loss_text_ce: 0.03891858085989952 (0.044429116510796667) | > loss_mel_ce: 3.189913034439087 (nan) | > loss: 0.012812823988497257 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13603276572143877) | > loader_time: 0.0046 (0.00465929302176018)  --> TIME: 2025-02-24 08:12:04 -- STEP: 15672/36938 -- GLOBAL_STEP: 237300 | > loss_text_ce: 0.03649051859974861 (0.04442632122856909) | > loss_mel_ce: 2.748325824737549 (nan) | > loss: 0.011050858534872532 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13602777446880246) | > loader_time: 0.0047 (0.004659334438196431)  --> TIME: 2025-02-24 08:12:13 -- STEP: 15722/36938 -- GLOBAL_STEP: 237350 | > loss_text_ce: 0.04871130362153053 (0.04442571689716062) | > loss_mel_ce: 3.2140090465545654 (nan) | > loss: 0.012947304174304008 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.1360141140497791) | > loader_time: 0.0046 (0.004659309640153344)  --> TIME: 2025-02-24 08:12:22 -- STEP: 15772/36938 -- GLOBAL_STEP: 237400 | > loss_text_ce: 0.05653712525963783 (0.04442542901251061) | > loss_mel_ce: 2.78682279586792 (nan) | > loss: 0.011283175088465214 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13600956716423832) | > loader_time: 0.0048 (0.0046593151417837955)  --> TIME: 2025-02-24 08:12:31 -- STEP: 15822/36938 -- GLOBAL_STEP: 237450 | > loss_text_ce: 0.04220752790570259 (0.0444251385394042) | > loss_mel_ce: 3.713641881942749 (nan) | > loss: 0.014904164709150791 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13601309311945994) | > loader_time: 0.0046 (0.004659289416222881)  --> TIME: 2025-02-24 08:12:40 -- STEP: 15872/36938 -- GLOBAL_STEP: 237500 | > loss_text_ce: 0.05290202051401138 (0.044427007545832946) | > loss_mel_ce: 3.5377533435821533 (nan) | > loss: 0.014248632825911045 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.13601166314836022) | > loader_time: 0.0046 (0.0046592848075013025)  --> TIME: 2025-02-24 08:12:50 -- STEP: 15922/36938 -- GLOBAL_STEP: 237550 | > loss_text_ce: 0.050364211201667786 (0.04442470645260132) | > loss_mel_ce: 3.4705634117126465 (nan) | > loss: 0.013971935957670212 (nan) | > current_lr: 5e-06 | > step_time: 0.1513 (0.13602994375081567) | > loader_time: 0.0046 (0.004659482813139027)  --> TIME: 2025-02-24 08:12:59 -- STEP: 15972/36938 -- GLOBAL_STEP: 237600 | > loss_text_ce: 0.04438618943095207 (0.04442193771741865) | > loss_mel_ce: 0.16688843071460724 (nan) | > loss: 0.0008383914246223867 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.1360275878095403) | > loader_time: 0.0047 (0.004659647380920572)  --> TIME: 2025-02-24 08:13:08 -- STEP: 16022/36938 -- GLOBAL_STEP: 237650 | > loss_text_ce: 0.032226186245679855 (0.04442045642035115) | > loss_mel_ce: 2.6202008724212646 (nan) | > loss: 0.010525504127144814 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13601426357425983) | > loader_time: 0.0047 (0.00465985179884933)  --> TIME: 2025-02-24 08:13:18 -- STEP: 16072/36938 -- GLOBAL_STEP: 237700 | > loss_text_ce: 0.04375918582081795 (0.044417188419926076) | > loss_mel_ce: 2.3171169757843018 (nan) | > loss: 0.009368556551635265 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.13602804986323364) | > loader_time: 0.0049 (0.004659797137741311)  --> TIME: 2025-02-24 08:13:27 -- STEP: 16122/36938 -- GLOBAL_STEP: 237750 | > loss_text_ce: 0.03603594750165939 (0.04441277664775182) | > loss_mel_ce: 2.3910889625549316 (nan) | > loss: 0.009631448425352573 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13603040521633802) | > loader_time: 0.0045 (0.004659778086013143)  --> TIME: 2025-02-24 08:13:37 -- STEP: 16172/36938 -- GLOBAL_STEP: 237800 | > loss_text_ce: 0.040836151689291 (0.04441191955131383) | > loss_mel_ce: 1.337255835533142 (nan) | > loss: 0.005468619056046009 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13604176022350126) | > loader_time: 0.0045 (0.004659750851964468)  --> TIME: 2025-02-24 08:13:46 -- STEP: 16222/36938 -- GLOBAL_STEP: 237850 | > loss_text_ce: 0.05370163172483444 (0.04440871938942963) | > loss_mel_ce: 2.7366714477539062 (nan) | > loss: 0.011072909459471703 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13602618191718732) | > loader_time: 0.0045 (0.0046596958022599114)  --> TIME: 2025-02-24 08:13:55 -- STEP: 16272/36938 -- GLOBAL_STEP: 237900 | > loss_text_ce: 0.03930725157260895 (0.04440108021922017) | > loss_mel_ce: 2.9452483654022217 (nan) | > loss: 0.011843475513160229 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.1360128004261412) | > loader_time: 0.0046 (0.004659546086804589)  --> TIME: 2025-02-24 08:14:04 -- STEP: 16322/36938 -- GLOBAL_STEP: 237950 | > loss_text_ce: 0.04070599377155304 (0.0444010276858764) | > loss_mel_ce: 3.661763906478882 (nan) | > loss: 0.014692340977489948 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.13602946156921436) | > loader_time: 0.005 (0.004659544616744094)  --> TIME: 2025-02-24 08:14:14 -- STEP: 16372/36938 -- GLOBAL_STEP: 238000 | > loss_text_ce: 0.05427693575620651 (0.04439386258188711) | > loss_mel_ce: 0.03409423679113388 (nan) | > loss: 0.00035067927092313766 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1360354097175508) | > loader_time: 0.0045 (0.004659466556485698)  --> TIME: 2025-02-24 08:14:23 -- STEP: 16422/36938 -- GLOBAL_STEP: 238050 | > loss_text_ce: 0.04058295115828514 (0.04439140899724334) | > loss_mel_ce: 3.6159279346466064 (nan) | > loss: 0.014509964734315872 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13604234992884354) | > loader_time: 0.0048 (0.004659377923184258)  --> TIME: 2025-02-24 08:14:32 -- STEP: 16472/36938 -- GLOBAL_STEP: 238100 | > loss_text_ce: 0.05672363191843033 (0.04438965386425417) | > loss_mel_ce: 3.069091796875 (nan) | > loss: 0.012404030188918114 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13602795570205897) | > loader_time: 0.0048 (0.004659264773173841)  --> TIME: 2025-02-24 08:14:41 -- STEP: 16522/36938 -- GLOBAL_STEP: 238150 | > loss_text_ce: 0.05310656502842903 (0.04439313154397717) | > loss_mel_ce: 0.2240753173828125 (nan) | > loss: 0.0010999281657859683 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.1360302575635157) | > loader_time: 0.0046 (0.00465939593595074)  --> TIME: 2025-02-24 08:14:51 -- STEP: 16572/36938 -- GLOBAL_STEP: 238200 | > loss_text_ce: 0.04469987377524376 (0.04439040304016835) | > loss_mel_ce: 3.9619016647338867 (nan) | > loss: 0.015899213030934334 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.13602895360334846) | > loader_time: 0.0047 (0.00465929988729416)  --> TIME: 2025-02-24 08:15:00 -- STEP: 16622/36938 -- GLOBAL_STEP: 238250 | > loss_text_ce: 0.050369586795568466 (0.044392263564688894) | > loss_mel_ce: 3.3811354637145996 (nan) | > loss: 0.013617084361612797 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1360319539565731) | > loader_time: 0.0046 (0.004659350649026198)  --> TIME: 2025-02-24 08:15:10 -- STEP: 16672/36938 -- GLOBAL_STEP: 238300 | > loss_text_ce: 0.04579014331102371 (0.04439055333520645) | > loss_mel_ce: 2.509730815887451 (nan) | > loss: 0.010140957310795784 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.1360563952869054) | > loader_time: 0.0046 (0.004659388493210253)  --> TIME: 2025-02-24 08:15:19 -- STEP: 16722/36938 -- GLOBAL_STEP: 238350 | > loss_text_ce: 0.048866529017686844 (0.04439153965887164) | > loss_mel_ce: 3.4336888790130615 (nan) | > loss: 0.013819664716720581 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.1360641676990944) | > loader_time: 0.0048 (0.004659376023503218)  --> TIME: 2025-02-24 08:15:29 -- STEP: 16772/36938 -- GLOBAL_STEP: 238400 | > loss_text_ce: 0.038641203194856644 (0.0443916309140369) | > loss_mel_ce: 3.2261905670166016 (nan) | > loss: 0.01295568235218525 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.13608395953807323) | > loader_time: 0.0046 (0.004659376237092771)  --> TIME: 2025-02-24 08:15:38 -- STEP: 16822/36938 -- GLOBAL_STEP: 238450 | > loss_text_ce: 0.044665634632110596 (0.0443899371403695) | > loss_mel_ce: 3.682793617248535 (nan) | > loss: 0.01479150541126728 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.1360921449999143) | > loader_time: 0.0046 (0.00465922537916921)  --> TIME: 2025-02-24 08:15:48 -- STEP: 16872/36938 -- GLOBAL_STEP: 238500 | > loss_text_ce: 0.047697652131319046 (0.044385881644020915) | > loss_mel_ce: 3.0174217224121094 (nan) | > loss: 0.012163172475993633 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.13609880806711316) | > loader_time: 0.0048 (0.004659225175933514)  --> TIME: 2025-02-24 08:15:57 -- STEP: 16922/36938 -- GLOBAL_STEP: 238550 | > loss_text_ce: 0.04368576034903526 (0.04438673134087664) | > loss_mel_ce: 2.7449164390563965 (nan) | > loss: 0.011065881699323654 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.13611561758075) | > loader_time: 0.0048 (0.00465925639296463)  --> TIME: 2025-02-24 08:16:07 -- STEP: 16972/36938 -- GLOBAL_STEP: 238600 | > loss_text_ce: 0.04886182025074959 (0.04438564096421179) | > loss_mel_ce: 3.4064109325408936 (nan) | > loss: 0.013711400330066681 (nan) | > current_lr: 5e-06 | > step_time: 0.1625 (0.13612770146422332) | > loader_time: 0.0048 (0.004659406340349798)  --> TIME: 2025-02-24 08:16:16 -- STEP: 17022/36938 -- GLOBAL_STEP: 238650 | > loss_text_ce: 0.05043541267514229 (0.044390849734263306) | > loss_mel_ce: 2.6003010272979736 (nan) | > loss: 0.010518795810639858 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13612704870353784) | > loader_time: 0.0045 (0.0046594511844929515)  --> TIME: 2025-02-24 08:16:25 -- STEP: 17072/36938 -- GLOBAL_STEP: 238700 | > loss_text_ce: 0.05456490442156792 (0.04439104807814282) | > loss_mel_ce: 3.5006370544433594 (nan) | > loss: 0.014107945375144482 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1361281124503122) | > loader_time: 0.0045 (0.0046593960943686836)  --> TIME: 2025-02-24 08:16:35 -- STEP: 17122/36938 -- GLOBAL_STEP: 238750 | > loss_text_ce: 0.044438473880290985 (0.04439037702695896) | > loss_mel_ce: 2.533032178878784 (nan) | > loss: 0.010228058323264122 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13612505353899354) | > loader_time: 0.0045 (0.004659406368232002)  --> TIME: 2025-02-24 08:16:44 -- STEP: 17172/36938 -- GLOBAL_STEP: 238800 | > loss_text_ce: 0.03978738188743591 (0.044386064449707864) | > loss_mel_ce: 3.191986560821533 (nan) | > loss: 0.012824499979615211 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13611963151802514) | > loader_time: 0.0046 (0.004659224897727197)  --> TIME: 2025-02-24 08:16:53 -- STEP: 17222/36938 -- GLOBAL_STEP: 238850 | > loss_text_ce: 0.039127979427576065 (0.044384395307988316) | > loss_mel_ce: 3.794896364212036 (nan) | > loss: 0.015214383602142334 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13612846221509664) | > loader_time: 0.0046 (0.004659333166490797)  --> TIME: 2025-02-24 08:17:03 -- STEP: 17272/36938 -- GLOBAL_STEP: 238900 | > loss_text_ce: 0.049461543560028076 (0.04438487398208448) | > loss_mel_ce: 3.7077243328094482 (nan) | > loss: 0.014909468591213226 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.136136712714333) | > loader_time: 0.0048 (0.0046591904219679976)  --> TIME: 2025-02-24 08:17:12 -- STEP: 17322/36938 -- GLOBAL_STEP: 238950 | > loss_text_ce: 0.043969977647066116 (0.0443796700145909) | > loss_mel_ce: 0.12003707885742188 (nan) | > loss: 0.0006508216611109674 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13613399991591674) | > loader_time: 0.0045 (0.0046590951336626255)  --> TIME: 2025-02-24 08:17:21 -- STEP: 17372/36938 -- GLOBAL_STEP: 239000 | > loss_text_ce: 0.03809804469347 (0.04437238773696237) | > loss_mel_ce: 3.5696322917938232 (nan) | > loss: 0.014316391199827194 (nan) | > current_lr: 5e-06 | > step_time: 0.1503 (0.13614056730995072) | > loader_time: 0.0048 (0.004659067958613957)  --> TIME: 2025-02-24 08:17:31 -- STEP: 17422/36938 -- GLOBAL_STEP: 239050 | > loss_text_ce: 0.048812150955200195 (0.04436946745752135) | > loss_mel_ce: 2.5088701248168945 (nan) | > loss: 0.010149533860385418 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13614353029507112) | > loader_time: 0.0045 (0.004659078354102546)  --> TIME: 2025-02-24 08:17:40 -- STEP: 17472/36938 -- GLOBAL_STEP: 239100 | > loss_text_ce: 0.04970739781856537 (0.044369380106591236) | > loss_mel_ce: 3.191866159439087 (nan) | > loss: 0.012863388285040855 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13614806734816973) | > loader_time: 0.0046 (0.004658955002660715)  --> TIME: 2025-02-24 08:17:49 -- STEP: 17522/36938 -- GLOBAL_STEP: 239150 | > loss_text_ce: 0.041502151638269424 (0.0443724923801804) | > loss_mel_ce: 1.933856725692749 (nan) | > loss: 0.007838726043701172 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13614605262581353) | > loader_time: 0.0045 (0.004658864113501034)  --> TIME: 2025-02-24 08:17:59 -- STEP: 17572/36938 -- GLOBAL_STEP: 239200 | > loss_text_ce: 0.04597635939717293 (0.04437116318225014) | > loss_mel_ce: 3.4447391033172607 (nan) | > loss: 0.013852046802639961 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1361522879031755) | > loader_time: 0.0046 (0.004658966883423384)  --> TIME: 2025-02-24 08:18:08 -- STEP: 17622/36938 -- GLOBAL_STEP: 239250 | > loss_text_ce: 0.04100152105093002 (0.04436601372636514) | > loss_mel_ce: 3.6545183658599854 (nan) | > loss: 0.014664762653410435 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13614439176871876) | > loader_time: 0.0045 (0.004658896743662711)  --> TIME: 2025-02-24 08:18:17 -- STEP: 17672/36938 -- GLOBAL_STEP: 239300 | > loss_text_ce: 0.04002423584461212 (0.04436432844751556) | > loss_mel_ce: 0.002899169921875 (nan) | > loss: 0.0001703309826552868 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13615743939344524) | > loader_time: 0.0045 (0.004658898100047799)  --> TIME: 2025-02-24 08:18:27 -- STEP: 17722/36938 -- GLOBAL_STEP: 239350 | > loss_text_ce: 0.05183896794915199 (0.044366481571272536) | > loss_mel_ce: 2.7312073707580566 (nan) | > loss: 0.011043834500014782 (nan) | > current_lr: 5e-06 | > step_time: 0.1754 (0.13616242670821904) | > loader_time: 0.0049 (0.0046586945960704895)  --> TIME: 2025-02-24 08:18:36 -- STEP: 17772/36938 -- GLOBAL_STEP: 239400 | > loss_text_ce: 0.044455911964178085 (0.04436604255589646) | > loss_mel_ce: 3.322798252105713 (nan) | > loss: 0.01336212083697319 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.1361664388728743) | > loader_time: 0.0045 (0.004658491593235756)  --> TIME: 2025-02-24 08:18:45 -- STEP: 17822/36938 -- GLOBAL_STEP: 239450 | > loss_text_ce: 0.04324393346905708 (0.04436553222818189) | > loss_mel_ce: 3.303895950317383 (nan) | > loss: 0.01328230183571577 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13614781973619136) | > loader_time: 0.0048 (0.004658520709998727)  --> TIME: 2025-02-24 08:18:54 -- STEP: 17872/36938 -- GLOBAL_STEP: 239500 | > loss_text_ce: 0.049925535917282104 (0.04436342048598608) | > loss_mel_ce: 3.728515625 (nan) | > loss: 0.014993814751505852 (nan) | > current_lr: 5e-06 | > step_time: 0.1069 (0.1361444946128343) | > loader_time: 0.0044 (0.00465848685751669)  --> TIME: 2025-02-24 08:19:04 -- STEP: 17922/36938 -- GLOBAL_STEP: 239550 | > loss_text_ce: 0.048224132508039474 (0.04436183465900791) | > loss_mel_ce: 2.5807347297668457 (nan) | > loss: 0.010432377457618713 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13615176809670898) | > loader_time: 0.0045 (0.004658418313132739)  --> TIME: 2025-02-24 08:19:13 -- STEP: 17972/36938 -- GLOBAL_STEP: 239600 | > loss_text_ce: 0.06067086011171341 (0.04435605386044258) | > loss_mel_ce: 2.6322498321533203 (nan) | > loss: 0.010686193592846394 (nan) | > current_lr: 5e-06 | > step_time: 0.1086 (0.13615205740042524) | > loader_time: 0.0045 (0.004658437653424178)  --> TIME: 2025-02-24 08:19:22 -- STEP: 18022/36938 -- GLOBAL_STEP: 239650 | > loss_text_ce: 0.04985819384455681 (0.044353779846831795) | > loss_mel_ce: 3.256075620651245 (nan) | > loss: 0.013118784874677658 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1361505345622999) | > loader_time: 0.0046 (0.0046583669403392714)  --> TIME: 2025-02-24 08:19:32 -- STEP: 18072/36938 -- GLOBAL_STEP: 239700 | > loss_text_ce: 0.044404055923223495 (0.04435406816395002) | > loss_mel_ce: 3.067960262298584 (nan) | > loss: 0.012350652366876602 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13615233636155755) | > loader_time: 0.0046 (0.004658302700377073)  --> TIME: 2025-02-24 08:19:41 -- STEP: 18122/36938 -- GLOBAL_STEP: 239750 | > loss_text_ce: 0.046540964394807816 (0.04435191351373336) | > loss_mel_ce: 3.4625627994537354 (nan) | > loss: 0.013925015926361084 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.136153323617996) | > loader_time: 0.0047 (0.004658256141756052)  --> TIME: 2025-02-24 08:19:51 -- STEP: 18172/36938 -- GLOBAL_STEP: 239800 | > loss_text_ce: 0.04411384463310242 (0.04434947456268449) | > loss_mel_ce: 3.289980173110962 (nan) | > loss: 0.01323053240776062 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.13616746810992478) | > loader_time: 0.0046 (0.004658254867552023)  --> TIME: 2025-02-24 08:20:00 -- STEP: 18222/36938 -- GLOBAL_STEP: 239850 | > loss_text_ce: 0.03327258676290512 (0.044350233733510346) | > loss_mel_ce: 0.019775390625 (nan) | > loss: 0.0002105078601744026 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1361620313785512) | > loader_time: 0.0047 (0.004658132781706076)  --> TIME: 2025-02-24 08:20:09 -- STEP: 18272/36938 -- GLOBAL_STEP: 239900 | > loss_text_ce: 0.04647576063871384 (0.044349136517410306) | > loss_mel_ce: 3.339437961578369 (nan) | > loss: 0.013436165638267994 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.1361690977826432) | > loader_time: 0.0047 (0.004657990277961746)  --> TIME: 2025-02-24 08:20:19 -- STEP: 18322/36938 -- GLOBAL_STEP: 239950 | > loss_text_ce: 0.0500650554895401 (0.0443491775034899) | > loss_mel_ce: 3.416869640350342 (nan) | > loss: 0.013757677748799324 (nan) | > current_lr: 5e-06 | > step_time: 0.1827 (0.13618701794641755) | > loader_time: 0.0048 (0.004657959068801348)  --> TIME: 2025-02-24 08:20:28 -- STEP: 18372/36938 -- GLOBAL_STEP: 240000 | > loss_text_ce: 0.046201080083847046 (0.044348296773383124) | > loss_mel_ce: 2.9382808208465576 (nan) | > loss: 0.011843182146549225 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.136179387387746) | > loader_time: 0.0045 (0.004657726816795699) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_240000.pth  --> TIME: 2025-02-24 08:20:44 -- STEP: 18422/36938 -- GLOBAL_STEP: 240050 | > loss_text_ce: 0.05347607657313347 (0.044348356861664634) | > loss_mel_ce: 1.7453835010528564 (nan) | > loss: 0.007138332352042198 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1361884746861092) | > loader_time: 0.0045 (0.004657729028451834)  --> TIME: 2025-02-24 08:20:52 -- STEP: 18472/36938 -- GLOBAL_STEP: 240100 | > loss_text_ce: 0.04616241529583931 (0.044350187948056924) | > loss_mel_ce: 1.3152819871902466 (nan) | > loss: 0.005402557086199522 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13616726570389642) | > loader_time: 0.0045 (0.004657686195522212)  --> TIME: 2025-02-24 08:21:02 -- STEP: 18522/36938 -- GLOBAL_STEP: 240150 | > loss_text_ce: 0.042929936200380325 (0.04434829745368244) | > loss_mel_ce: 3.3076584339141846 (nan) | > loss: 0.013295985758304596 (nan) | > current_lr: 5e-06 | > step_time: 0.1069 (0.13616459979286066) | > loader_time: 0.0045 (0.004657618081179631)  --> TIME: 2025-02-24 08:21:11 -- STEP: 18572/36938 -- GLOBAL_STEP: 240200 | > loss_text_ce: 0.034833911806344986 (0.04434914600117913) | > loss_mel_ce: 3.3068408966064453 (nan) | > loss: 0.013260615058243275 (nan) | > current_lr: 5e-06 | > step_time: 0.1268 (0.13617025432262178) | > loader_time: 0.0046 (0.0046575843530458)  --> TIME: 2025-02-24 08:21:20 -- STEP: 18622/36938 -- GLOBAL_STEP: 240250 | > loss_text_ce: 0.044044096022844315 (0.04434979857773653) | > loss_mel_ce: 3.3398070335388184 (nan) | > loss: 0.013427981175482273 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.1361709831224687) | > loader_time: 0.0047 (0.00465756922963481)  --> TIME: 2025-02-24 08:21:30 -- STEP: 18672/36938 -- GLOBAL_STEP: 240300 | > loss_text_ce: 0.045758314430713654 (0.04435215706432428) | > loss_mel_ce: 3.305676221847534 (nan) | > loss: 0.01329934410750866 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.1361703820839439) | > loader_time: 0.0047 (0.004657502843972026)  --> TIME: 2025-02-24 08:21:39 -- STEP: 18722/36938 -- GLOBAL_STEP: 240350 | > loss_text_ce: 0.04960176348686218 (0.04435177497527323) | > loss_mel_ce: 3.674023389816284 (nan) | > loss: 0.014776291325688362 (nan) | > current_lr: 5e-06 | > step_time: 0.1716 (0.1361615842077142) | > loader_time: 0.005 (0.004657418003781143)  --> TIME: 2025-02-24 08:21:48 -- STEP: 18772/36938 -- GLOBAL_STEP: 240400 | > loss_text_ce: 0.04616712033748627 (0.04434939107356341) | > loss_mel_ce: 3.5892176628112793 (nan) | > loss: 0.014426130801439285 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.1361703441490023) | > loader_time: 0.0047 (0.004657436072534254)  --> TIME: 2025-02-24 08:21:58 -- STEP: 18822/36938 -- GLOBAL_STEP: 240450 | > loss_text_ce: 0.040368061512708664 (0.04434662972156409) | > loss_mel_ce: 3.258467674255371 (nan) | > loss: 0.013090618886053562 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1361642965866625) | > loader_time: 0.0045 (0.004657367529568599)  --> TIME: 2025-02-24 08:22:07 -- STEP: 18872/36938 -- GLOBAL_STEP: 240500 | > loss_text_ce: 0.0408642552793026 (0.04434716705608485) | > loss_mel_ce: 3.158055305480957 (nan) | > loss: 0.01269412599503994 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13615903404871793) | > loader_time: 0.0045 (0.004657316253366163)  --> TIME: 2025-02-24 08:22:16 -- STEP: 18922/36938 -- GLOBAL_STEP: 240550 | > loss_text_ce: 0.043651215732097626 (0.04434583802450449) | > loss_mel_ce: 3.213082790374756 (nan) | > loss: 0.012923547998070717 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.1361617781800686) | > loader_time: 0.0046 (0.0046571777910450664)  --> TIME: 2025-02-24 08:22:25 -- STEP: 18972/36938 -- GLOBAL_STEP: 240600 | > loss_text_ce: 0.05935779958963394 (0.04434084921476161) | > loss_mel_ce: 3.5054168701171875 (nan) | > loss: 0.014145932160317898 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13616180903746253) | > loader_time: 0.0046 (0.004657051519530224)  --> TIME: 2025-02-24 08:22:35 -- STEP: 19022/36938 -- GLOBAL_STEP: 240650 | > loss_text_ce: 0.050934355705976486 (0.04434613066480312) | > loss_mel_ce: 3.0197832584381104 (nan) | > loss: 0.012185388244688511 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13616626460198056) | > loader_time: 0.0045 (0.004656906434256558)  --> TIME: 2025-02-24 08:22:44 -- STEP: 19072/36938 -- GLOBAL_STEP: 240700 | > loss_text_ce: 0.03654002398252487 (0.04434422256165019) | > loss_mel_ce: 3.124494791030884 (nan) | > loss: 0.01254379004240036 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.1361776981002737) | > loader_time: 0.0048 (0.004656807900775184)  --> TIME: 2025-02-24 08:22:53 -- STEP: 19122/36938 -- GLOBAL_STEP: 240750 | > loss_text_ce: 0.03778655827045441 (0.04434061580682486) | > loss_mel_ce: 2.689088821411133 (nan) | > loss: 0.010820934548974037 (nan) | > current_lr: 5e-06 | > step_time: 0.1069 (0.13616836955015973) | > loader_time: 0.0044 (0.004656667029080846)  --> TIME: 2025-02-24 08:23:03 -- STEP: 19172/36938 -- GLOBAL_STEP: 240800 | > loss_text_ce: 0.05114509165287018 (0.04434458582079875) | > loss_mel_ce: 3.071044921875 (nan) | > loss: 0.012389643117785454 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1361808680229541) | > loader_time: 0.0047 (0.004656630880072088)  --> TIME: 2025-02-24 08:23:13 -- STEP: 19222/36938 -- GLOBAL_STEP: 240850 | > loss_text_ce: 0.059026047587394714 (0.044345458175606166) | > loss_mel_ce: 2.5532805919647217 (nan) | > loss: 0.010366296395659447 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13620588042934303) | > loader_time: 0.0046 (0.004656769410465662)  --> TIME: 2025-02-24 08:23:22 -- STEP: 19272/36938 -- GLOBAL_STEP: 240900 | > loss_text_ce: 0.05223912000656128 (0.04434003595442708) | > loss_mel_ce: 3.0920848846435547 (nan) | > loss: 0.012477477081120014 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13621247600301695) | > loader_time: 0.0046 (0.004656748214466717)  --> TIME: 2025-02-24 08:23:31 -- STEP: 19322/36938 -- GLOBAL_STEP: 240950 | > loss_text_ce: 0.040551524609327316 (0.044340199825839886) | > loss_mel_ce: 1.7674403190612793 (nan) | > loss: 0.007174571510404348 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.1362075326408419) | > loader_time: 0.0045 (0.004656609720404413)  --> TIME: 2025-02-24 08:23:40 -- STEP: 19372/36938 -- GLOBAL_STEP: 241000 | > loss_text_ce: 0.04142541438341141 (0.04433839615237661) | > loss_mel_ce: 3.3911778926849365 (nan) | > loss: 0.013621442951261997 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13619093346787792) | > loader_time: 0.0044 (0.004656361014838539)  --> TIME: 2025-02-24 08:23:50 -- STEP: 19422/36938 -- GLOBAL_STEP: 241050 | > loss_text_ce: 0.052793171256780624 (0.04433887734635722) | > loss_mel_ce: 1.3717825412750244 (nan) | > loss: 0.0056530786678195 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13619768851548056) | > loader_time: 0.0045 (0.0046562905808013475)  --> TIME: 2025-02-24 08:23:59 -- STEP: 19472/36938 -- GLOBAL_STEP: 241100 | > loss_text_ce: 0.03921591117978096 (0.044340394245280176) | > loss_mel_ce: 3.337589740753174 (nan) | > loss: 0.013400022871792316 (nan) | > current_lr: 5e-06 | > step_time: 0.1642 (0.13620277006261775) | > loader_time: 0.0046 (0.00465632035973233)  --> TIME: 2025-02-24 08:24:08 -- STEP: 19522/36938 -- GLOBAL_STEP: 241150 | > loss_text_ce: 0.05051209032535553 (0.04434479525224595) | > loss_mel_ce: 0.11421342194080353 (nan) | > loss: 0.0006536727305501699 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13618901731928623) | > loader_time: 0.0046 (0.004656406091796154)  --> TIME: 2025-02-24 08:24:17 -- STEP: 19572/36938 -- GLOBAL_STEP: 241200 | > loss_text_ce: 0.04226081818342209 (0.04434206063778057) | > loss_mel_ce: 3.8801329135894775 (nan) | > loss: 0.015565055422484875 (nan) | > current_lr: 5e-06 | > step_time: 0.1823 (0.13618897827714865) | > loader_time: 0.0047 (0.004656244915198757)  --> TIME: 2025-02-24 08:24:27 -- STEP: 19622/36938 -- GLOBAL_STEP: 241250 | > loss_text_ce: 0.044345445930957794 (0.0443385504962174) | > loss_mel_ce: 2.691188335418701 (nan) | > loss: 0.010855292901396751 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.13619661679898282) | > loader_time: 0.0048 (0.004656098508868843)  --> TIME: 2025-02-24 08:24:36 -- STEP: 19672/36938 -- GLOBAL_STEP: 241300 | > loss_text_ce: 0.05127789080142975 (0.04433593194521357) | > loss_mel_ce: 3.1099956035614014 (nan) | > loss: 0.012544737197458744 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.13620401598322263) | > loader_time: 0.005 (0.004656069559409088)  --> TIME: 2025-02-24 08:24:46 -- STEP: 19722/36938 -- GLOBAL_STEP: 241350 | > loss_text_ce: 0.045367978513240814 (0.04433286855875215) | > loss_mel_ce: 3.9226388931274414 (nan) | > loss: 0.015746060758829117 (nan) | > current_lr: 5e-06 | > step_time: 0.1799 (0.1362185402170858) | > loader_time: 0.0048 (0.0046560964626903584)  --> TIME: 2025-02-24 08:24:55 -- STEP: 19772/36938 -- GLOBAL_STEP: 241400 | > loss_text_ce: 0.04307706281542778 (0.044327922837790426) | > loss_mel_ce: 3.4767372608184814 (nan) | > loss: 0.013967517763376236 (nan) | > current_lr: 5e-06 | > step_time: 0.1828 (0.13621146950720803) | > loader_time: 0.0047 (0.004655984896000581)  --> TIME: 2025-02-24 08:25:04 -- STEP: 19822/36938 -- GLOBAL_STEP: 241450 | > loss_text_ce: 0.041126277297735214 (0.044327789799046634) | > loss_mel_ce: 2.9604828357696533 (nan) | > loss: 0.011911148205399513 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13620353116720696) | > loader_time: 0.0046 (0.004655960289119227)  --> TIME: 2025-02-24 08:25:13 -- STEP: 19872/36938 -- GLOBAL_STEP: 241500 | > loss_text_ce: 0.03992195799946785 (0.0443235650745682) | > loss_mel_ce: 0.1285400390625 (nan) | > loss: 0.0006685000262223184 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13618714159621548) | > loader_time: 0.0046 (0.004655755240464154)  --> TIME: 2025-02-24 08:25:22 -- STEP: 19922/36938 -- GLOBAL_STEP: 241550 | > loss_text_ce: 0.03961166366934776 (0.04432148447233305) | > loss_mel_ce: 2.654603958129883 (nan) | > loss: 0.010691331699490547 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1361889762220682) | > loader_time: 0.0045 (0.004655599318861336)  --> TIME: 2025-02-24 08:25:32 -- STEP: 19972/36938 -- GLOBAL_STEP: 241600 | > loss_text_ce: 0.0460556335747242 (0.0443198688389537) | > loss_mel_ce: 3.232128858566284 (nan) | > loss: 0.013008669018745422 (nan) | > current_lr: 5e-06 | > step_time: 0.1713 (0.13620912515398498) | > loader_time: 0.0047 (0.004655570907963303)  --> TIME: 2025-02-24 08:25:41 -- STEP: 20022/36938 -- GLOBAL_STEP: 241650 | > loss_text_ce: 0.04546774923801422 (0.04431713098193447) | > loss_mel_ce: 3.3561763763427734 (nan) | > loss: 0.013498589396476746 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1362087352329914) | > loader_time: 0.0046 (0.004655416082542801)  --> TIME: 2025-02-24 08:25:50 -- STEP: 20072/36938 -- GLOBAL_STEP: 241700 | > loss_text_ce: 0.04709748551249504 (0.04431904444249929) | > loss_mel_ce: 1.5438019037246704 (nan) | > loss: 0.00631309300661087 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13620046259825752) | > loader_time: 0.0047 (0.004655216784532334)  --> TIME: 2025-02-24 08:26:00 -- STEP: 20122/36938 -- GLOBAL_STEP: 241750 | > loss_text_ce: 0.03658827766776085 (0.04431730784179008) | > loss_mel_ce: 3.8542544841766357 (nan) | > loss: 0.015439853072166443 (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.13620302622887745) | > loader_time: 0.0048 (0.004655080824577936)  --> TIME: 2025-02-24 08:26:09 -- STEP: 20172/36938 -- GLOBAL_STEP: 241800 | > loss_text_ce: 0.03724268451333046 (0.04432025974644637) | > loss_mel_ce: 2.906453847885132 (nan) | > loss: 0.011681335978209972 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1361992509266276) | > loader_time: 0.0045 (0.004654955159523345)  --> TIME: 2025-02-24 08:26:18 -- STEP: 20222/36938 -- GLOBAL_STEP: 241850 | > loss_text_ce: 0.053804200142621994 (0.04431630247793406) | > loss_mel_ce: 3.7887699604034424 (nan) | > loss: 0.015248310752213001 (nan) | > current_lr: 5e-06 | > step_time: 0.1853 (0.13620357452826867) | > loader_time: 0.0048 (0.004654870319998382)  --> TIME: 2025-02-24 08:26:28 -- STEP: 20272/36938 -- GLOBAL_STEP: 241900 | > loss_text_ce: 0.05065228417515755 (0.044313276222557235) | > loss_mel_ce: 3.385517120361328 (nan) | > loss: 0.013635593466460705 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.1362096309285517) | > loader_time: 0.0049 (0.004654789450795213)  --> TIME: 2025-02-24 08:26:37 -- STEP: 20322/36938 -- GLOBAL_STEP: 241950 | > loss_text_ce: 0.04334931820631027 (0.044315025442775574) | > loss_mel_ce: 3.752595901489258 (nan) | > loss: 0.015063275583088398 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.13619330914608813) | > loader_time: 0.0047 (0.004654565320663949)  --> TIME: 2025-02-24 08:26:46 -- STEP: 20372/36938 -- GLOBAL_STEP: 242000 | > loss_text_ce: 0.04291590303182602 (0.044313431053890094) | > loss_mel_ce: 0.041748046875 (nan) | > loss: 0.0003359680704306811 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13620120325562285) | > loader_time: 0.0045 (0.004654514316770042)  --> TIME: 2025-02-24 08:26:55 -- STEP: 20422/36938 -- GLOBAL_STEP: 242050 | > loss_text_ce: 0.04266244173049927 (0.04431198925461555) | > loss_mel_ce: 3.607680320739746 (nan) | > loss: 0.014485487714409828 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.1361859144560683) | > loader_time: 0.0047 (0.004654545950242801)  --> TIME: 2025-02-24 08:27:04 -- STEP: 20472/36938 -- GLOBAL_STEP: 242100 | > loss_text_ce: 0.03971320390701294 (0.04431104410548708) | > loss_mel_ce: 3.3820152282714844 (nan) | > loss: 0.01357828825712204 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13618559128560834) | > loader_time: 0.0046 (0.0046544659645494845)  --> TIME: 2025-02-24 08:27:14 -- STEP: 20522/36938 -- GLOBAL_STEP: 242150 | > loss_text_ce: 0.03307448700070381 (0.04430667921985499) | > loss_mel_ce: 3.1140520572662354 (nan) | > loss: 0.012488598003983498 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13618869994562915) | > loader_time: 0.0047 (0.004654479077912334)  --> TIME: 2025-02-24 08:27:23 -- STEP: 20572/36938 -- GLOBAL_STEP: 242200 | > loss_text_ce: 0.04388504847884178 (0.04430599820520507) | > loss_mel_ce: 3.3585896492004395 (nan) | > loss: 0.013501884415745735 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13618276762679588) | > loader_time: 0.0045 (0.004654222289898868)  --> TIME: 2025-02-24 08:27:33 -- STEP: 20622/36938 -- GLOBAL_STEP: 242250 | > loss_text_ce: 0.0448528416454792 (0.04430501366734312) | > loss_mel_ce: 2.7476487159729004 (nan) | > loss: 0.01108135562390089 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13619786659899638) | > loader_time: 0.0045 (0.00465411213133131)  --> TIME: 2025-02-24 08:27:42 -- STEP: 20672/36938 -- GLOBAL_STEP: 242300 | > loss_text_ce: 0.04457242414355278 (0.044304877414950636) | > loss_mel_ce: 3.862217426300049 (nan) | > loss: 0.015503134578466415 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.13620269045971872) | > loader_time: 0.0047 (0.0046539903955747355)  --> TIME: 2025-02-24 08:27:51 -- STEP: 20722/36938 -- GLOBAL_STEP: 242350 | > loss_text_ce: 0.042649801820516586 (0.04430174914878722) | > loss_mel_ce: 3.1952199935913086 (nan) | > loss: 0.012848690152168274 (nan) | > current_lr: 5e-06 | > step_time: 0.1862 (0.13620894311443404) | > loader_time: 0.0048 (0.004653972130164173)  --> TIME: 2025-02-24 08:28:01 -- STEP: 20772/36938 -- GLOBAL_STEP: 242400 | > loss_text_ce: 0.03920654207468033 (0.04429868642873276) | > loss_mel_ce: 3.3083088397979736 (nan) | > loss: 0.013283791951835155 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13620677899699124) | > loader_time: 0.0046 (0.004653851627362334)  --> TIME: 2025-02-24 08:28:10 -- STEP: 20822/36938 -- GLOBAL_STEP: 242450 | > loss_text_ce: 0.042533908039331436 (0.04429320528343418) | > loss_mel_ce: 3.1683216094970703 (nan) | > loss: 0.012741491198539734 (nan) | > current_lr: 5e-06 | > step_time: 0.1809 (0.13621151612180774) | > loader_time: 0.0047 (0.004653711825531909)  --> TIME: 2025-02-24 08:28:19 -- STEP: 20872/36938 -- GLOBAL_STEP: 242500 | > loss_text_ce: 0.049362316727638245 (0.04428959861202277) | > loss_mel_ce: 3.6410858631134033 (nan) | > loss: 0.014644636772572994 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13620246781750928) | > loader_time: 0.0045 (0.004653664864813875)  --> TIME: 2025-02-24 08:28:28 -- STEP: 20922/36938 -- GLOBAL_STEP: 242550 | > loss_text_ce: 0.05451424792408943 (0.044285935528842094) | > loss_mel_ce: 4.132580280303955 (nan) | > loss: 0.016615455970168114 (nan) | > current_lr: 5e-06 | > step_time: 0.1866 (0.13619965640209294) | > loader_time: 0.0048 (0.004653484868497321)  --> TIME: 2025-02-24 08:28:38 -- STEP: 20972/36938 -- GLOBAL_STEP: 242600 | > loss_text_ce: 0.04879133775830269 (0.044283923184130776) | > loss_mel_ce: 2.8612983226776123 (nan) | > loss: 0.011547976173460484 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.13621290017011067) | > loader_time: 0.005 (0.004653346622670022)  --> TIME: 2025-02-24 08:28:47 -- STEP: 21022/36938 -- GLOBAL_STEP: 242650 | > loss_text_ce: 0.03111361525952816 (0.044281348724526784) | > loss_mel_ce: 2.5996134281158447 (nan) | > loss: 0.010439394041895866 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13621883533647658) | > loader_time: 0.0046 (0.0046532952856942495)  --> TIME: 2025-02-24 08:28:57 -- STEP: 21072/36938 -- GLOBAL_STEP: 242700 | > loss_text_ce: 0.04634467884898186 (0.04427680271834453) | > loss_mel_ce: 3.3177647590637207 (nan) | > loss: 0.013349642045795918 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13622054075235326) | > loader_time: 0.0045 (0.004653105499532592)  --> TIME: 2025-02-24 08:29:06 -- STEP: 21122/36938 -- GLOBAL_STEP: 242750 | > loss_text_ce: 0.032796066254377365 (0.044271859003193755) | > loss_mel_ce: 2.862309217453003 (nan) | > loss: 0.011488514021039009 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13622272490958445) | > loader_time: 0.0045 (0.004652929671751401)  --> TIME: 2025-02-24 08:29:15 -- STEP: 21172/36938 -- GLOBAL_STEP: 242800 | > loss_text_ce: 0.035819195210933685 (0.044269710425654904) | > loss_mel_ce: 3.3357770442962646 (nan) | > loss: 0.013379351235926151 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13622485938199247) | > loader_time: 0.0046 (0.004652895437349052)  --> TIME: 2025-02-24 08:29:24 -- STEP: 21222/36938 -- GLOBAL_STEP: 242850 | > loss_text_ce: 0.049833670258522034 (0.044264592191130575) | > loss_mel_ce: 3.2043368816375732 (nan) | > loss: 0.012913376092910767 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.1362127035999488) | > loader_time: 0.0045 (0.004652868531873937)  --> TIME: 2025-02-24 08:29:34 -- STEP: 21272/36938 -- GLOBAL_STEP: 242900 | > loss_text_ce: 0.04909735172986984 (0.04426133020492168) | > loss_mel_ce: 2.0061886310577393 (nan) | > loss: 0.00815589725971222 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13621671965759763) | > loader_time: 0.0046 (0.004652895495693401)  --> TIME: 2025-02-24 08:29:43 -- STEP: 21322/36938 -- GLOBAL_STEP: 242950 | > loss_text_ce: 0.04050357639789581 (0.04425605016303353) | > loss_mel_ce: 3.2097554206848145 (nan) | > loss: 0.01289785373955965 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13620996332092566) | > loader_time: 0.0044 (0.0046528610679090355)  --> TIME: 2025-02-24 08:29:52 -- STEP: 21372/36938 -- GLOBAL_STEP: 243000 | > loss_text_ce: 0.04226243868470192 (0.04425416161754487) | > loss_mel_ce: 2.7313520908355713 (nan) | > loss: 0.011006408371031284 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13621136365971193) | > loader_time: 0.0046 (0.004652676478811973)  --> TIME: 2025-02-24 08:30:02 -- STEP: 21422/36938 -- GLOBAL_STEP: 243050 | > loss_text_ce: 0.04075368866324425 (0.044253856713579176) | > loss_mel_ce: 3.0584006309509277 (nan) | > loss: 0.012298231944441795 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13622251057555826) | > loader_time: 0.0047 (0.004652638360127574)  --> TIME: 2025-02-24 08:30:12 -- STEP: 21472/36938 -- GLOBAL_STEP: 243100 | > loss_text_ce: 0.03526703268289566 (0.04425398492305116) | > loss_mel_ce: 2.80143404006958 (nan) | > loss: 0.011256751604378223 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13623791703213164) | > loader_time: 0.0046 (0.004652552917355388)  --> TIME: 2025-02-24 08:30:20 -- STEP: 21522/36938 -- GLOBAL_STEP: 243150 | > loss_text_ce: 0.04881656914949417 (0.044249133734878064) | > loss_mel_ce: 1.3201497793197632 (nan) | > loss: 0.005432406440377235 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.1362254256907919) | > loader_time: 0.0045 (0.004652367483648963)  --> TIME: 2025-02-24 08:30:29 -- STEP: 21572/36938 -- GLOBAL_STEP: 243200 | > loss_text_ce: 0.045169804245233536 (0.04424677573757625) | > loss_mel_ce: 2.4247848987579346 (nan) | > loss: 0.009801408275961876 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.1362152114087789) | > loader_time: 0.0047 (0.004652159854606673)  --> TIME: 2025-02-24 08:30:39 -- STEP: 21622/36938 -- GLOBAL_STEP: 243250 | > loss_text_ce: 0.050676267594099045 (0.04424334015147076) | > loss_mel_ce: 1.533642053604126 (nan) | > loss: 0.006286977790296078 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1362217955491819) | > loader_time: 0.0045 (0.0046520363158692605)  --> TIME: 2025-02-24 08:30:48 -- STEP: 21672/36938 -- GLOBAL_STEP: 243300 | > loss_text_ce: 0.04113636538386345 (0.044240564585803284) | > loss_mel_ce: 2.264782190322876 (nan) | > loss: 0.009150470606982708 (nan) | > current_lr: 5e-06 | > step_time: 0.1054 (0.13622192701276004) | > loader_time: 0.0044 (0.0046518347984136395)  --> TIME: 2025-02-24 08:30:57 -- STEP: 21722/36938 -- GLOBAL_STEP: 243350 | > loss_text_ce: 0.050434716045856476 (0.04423767692820931) | > loss_mel_ce: 2.3196356296539307 (nan) | > loss: 0.009405042044818401 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13622029571174946) | > loader_time: 0.0046 (0.004651782438248154)  --> TIME: 2025-02-24 08:31:06 -- STEP: 21772/36938 -- GLOBAL_STEP: 243400 | > loss_text_ce: 0.04778311029076576 (0.044234998222573964) | > loss_mel_ce: 2.85343861579895 (nan) | > loss: 0.011512785218656063 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13620734867444254) | > loader_time: 0.0045 (0.004651651374999802)  --> TIME: 2025-02-24 08:31:16 -- STEP: 21822/36938 -- GLOBAL_STEP: 243450 | > loss_text_ce: 0.04601796343922615 (0.04423571894334186) | > loss_mel_ce: 0.14915160834789276 (nan) | > loss: 0.000774482439737767 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.1362054765022332) | > loader_time: 0.0045 (0.0046514783680422465)  --> TIME: 2025-02-24 08:31:25 -- STEP: 21872/36938 -- GLOBAL_STEP: 243500 | > loss_text_ce: 0.041835203766822815 (0.044233420805795086) | > loss_mel_ce: 3.7932913303375244 (nan) | > loss: 0.015218757092952728 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13619744271890208) | > loader_time: 0.0046 (0.004651252848001476)  --> TIME: 2025-02-24 08:31:34 -- STEP: 21922/36938 -- GLOBAL_STEP: 243550 | > loss_text_ce: 0.0484369695186615 (0.04423225280005326) | > loss_mel_ce: 4.299201965332031 (nan) | > loss: 0.017252536490559578 (nan) | > current_lr: 5e-06 | > step_time: 0.1353 (0.13619791399920328) | > loader_time: 0.0046 (0.0046510777326844175)  --> TIME: 2025-02-24 08:31:44 -- STEP: 21972/36938 -- GLOBAL_STEP: 243600 | > loss_text_ce: 0.041679542511701584 (0.044232878160694676) | > loss_mel_ce: 3.6349897384643555 (nan) | > loss: 0.01458995882421732 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.1362026233436977) | > loader_time: 0.0046 (0.004650910597734699)  --> TIME: 2025-02-24 08:31:53 -- STEP: 22022/36938 -- GLOBAL_STEP: 243650 | > loss_text_ce: 0.043480515480041504 (0.044231442253289996) | > loss_mel_ce: 3.5891201496124268 (nan) | > loss: 0.014415083453059196 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13619377713536449) | > loader_time: 0.0046 (0.00465070732542131)  --> TIME: 2025-02-24 08:32:02 -- STEP: 22072/36938 -- GLOBAL_STEP: 243700 | > loss_text_ce: 0.051322776824235916 (0.04423300615376275) | > loss_mel_ce: 3.361790418624878 (nan) | > loss: 0.01354410033673048 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13620067259547236) | > loader_time: 0.0046 (0.004650578955974885)  --> TIME: 2025-02-24 08:32:12 -- STEP: 22122/36938 -- GLOBAL_STEP: 243750 | > loss_text_ce: 0.04463014006614685 (0.044234114775223175) | > loss_mel_ce: 0.0140533447265625 (nan) | > loss: 0.00023287098156288266 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13621326451404592) | > loader_time: 0.0045 (0.004650386146498487)  --> TIME: 2025-02-24 08:32:21 -- STEP: 22172/36938 -- GLOBAL_STEP: 243800 | > loss_text_ce: 0.03577476739883423 (0.04423307677849005) | > loss_mel_ce: 3.2587361335754395 (nan) | > loss: 0.013073456473648548 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.13622070913729442) | > loader_time: 0.0046 (0.0046503726549576765)  --> TIME: 2025-02-24 08:32:30 -- STEP: 22222/36938 -- GLOBAL_STEP: 243850 | > loss_text_ce: 0.04721282422542572 (0.04423287382271223) | > loss_mel_ce: 4.440680980682373 (nan) | > loss: 0.017809102311730385 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.1362205146938551) | > loader_time: 0.0045 (0.004650213407060358)  --> TIME: 2025-02-24 08:32:39 -- STEP: 22272/36938 -- GLOBAL_STEP: 243900 | > loss_text_ce: 0.04382823035120964 (0.04423065528141583) | > loss_mel_ce: 2.3011138439178467 (nan) | > loss: 0.009305326268076897 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13621190252403378) | > loader_time: 0.0046 (0.004650112691110566)  --> TIME: 2025-02-24 08:32:49 -- STEP: 22322/36938 -- GLOBAL_STEP: 243950 | > loss_text_ce: 0.03878689929842949 (0.04422916898087648) | > loss_mel_ce: 2.975755453109741 (nan) | > loss: 0.011962470598518848 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13622032770406842) | > loader_time: 0.0048 (0.004650061408867372)  --> TIME: 2025-02-24 08:32:58 -- STEP: 22372/36938 -- GLOBAL_STEP: 244000 | > loss_text_ce: 0.04472637549042702 (0.04422461131853853) | > loss_mel_ce: 3.7990410327911377 (nan) | > loss: 0.015253046527504921 (nan) | > current_lr: 5e-06 | > step_time: 0.156 (0.13621343928842952) | > loader_time: 0.0046 (0.0046498823225572675)  --> TIME: 2025-02-24 08:33:07 -- STEP: 22422/36938 -- GLOBAL_STEP: 244050 | > loss_text_ce: 0.03297048062086105 (0.04422027902742753) | > loss_mel_ce: 2.4758822917938232 (nan) | > loss: 0.009955765679478645 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13621821729731254) | > loader_time: 0.0045 (0.004649856186032167)  --> TIME: 2025-02-24 08:33:17 -- STEP: 22472/36938 -- GLOBAL_STEP: 244100 | > loss_text_ce: 0.05083828791975975 (0.044217531647257315) | > loss_mel_ce: 2.999974250793457 (nan) | > loss: 0.012106399051845074 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.13622228667039155) | > loader_time: 0.0046 (0.004649812872191675)  --> TIME: 2025-02-24 08:33:26 -- STEP: 22522/36938 -- GLOBAL_STEP: 244150 | > loss_text_ce: 0.034494638442993164 (0.044217465476454054) | > loss_mel_ce: 2.7069461345672607 (nan) | > loss: 0.010878734290599823 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.1362296396520901) | > loader_time: 0.0046 (0.004649748398645565)  --> TIME: 2025-02-24 08:33:36 -- STEP: 22572/36938 -- GLOBAL_STEP: 244200 | > loss_text_ce: 0.04850584268569946 (0.044213941121284094) | > loss_mel_ce: 3.3741321563720703 (nan) | > loss: 0.013581897132098675 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13623726846435297) | > loader_time: 0.0045 (0.004649664162955712)  --> TIME: 2025-02-24 08:33:45 -- STEP: 22622/36938 -- GLOBAL_STEP: 244250 | > loss_text_ce: 0.03885497525334358 (0.04421238793898677) | > loss_mel_ce: 1.9681047201156616 (nan) | > loss: 0.007964125834405422 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13623151640267614) | > loader_time: 0.0046 (0.00464957036112886)  --> TIME: 2025-02-24 08:33:54 -- STEP: 22672/36938 -- GLOBAL_STEP: 244300 | > loss_text_ce: 0.037943724542856216 (0.04421340209408414) | > loss_mel_ce: 2.817949056625366 (nan) | > loss: 0.011332908645272255 (nan) | > current_lr: 5e-06 | > step_time: 0.1063 (0.13623011609619493) | > loader_time: 0.0044 (0.004649483471919535)  --> TIME: 2025-02-24 08:34:03 -- STEP: 22722/36938 -- GLOBAL_STEP: 244350 | > loss_text_ce: 0.04400892183184624 (0.04420661314015691) | > loss_mel_ce: 3.615793228149414 (nan) | > loss: 0.01452302560210228 (nan) | > current_lr: 5e-06 | > step_time: 0.1574 (0.1362185100572704) | > loader_time: 0.0046 (0.004649444382308833)  --> TIME: 2025-02-24 08:34:13 -- STEP: 22772/36938 -- GLOBAL_STEP: 244400 | > loss_text_ce: 0.03966738283634186 (0.04420191711296571) | > loss_mel_ce: 3.402858257293701 (nan) | > loss: 0.013660816475749016 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.13621832255254998) | > loader_time: 0.0048 (0.00464934414266051)  --> TIME: 2025-02-24 08:34:22 -- STEP: 22822/36938 -- GLOBAL_STEP: 244450 | > loss_text_ce: 0.04745589569211006 (0.044202057546009636) | > loss_mel_ce: 3.313572645187378 (nan) | > loss: 0.013337415643036366 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.13622367897080417) | > loader_time: 0.0046 (0.004649190875130187)  --> TIME: 2025-02-24 08:34:31 -- STEP: 22872/36938 -- GLOBAL_STEP: 244500 | > loss_text_ce: 0.043109942227602005 (0.044197670159114744) | > loss_mel_ce: 3.5601806640625 (nan) | > loss: 0.014298772439360619 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.13621936102103518) | > loader_time: 0.0046 (0.004649001543402605)  --> TIME: 2025-02-24 08:34:41 -- STEP: 22922/36938 -- GLOBAL_STEP: 244550 | > loss_text_ce: 0.04077346622943878 (0.04419506959823983) | > loss_mel_ce: 3.9057958126068115 (nan) | > loss: 0.01566099002957344 (nan) | > current_lr: 5e-06 | > step_time: 0.1794 (0.13622878317786705) | > loader_time: 0.0048 (0.004648974715451581)  --> TIME: 2025-02-24 08:34:50 -- STEP: 22972/36938 -- GLOBAL_STEP: 244600 | > loss_text_ce: 0.04654200002551079 (0.044197764051069025) | > loss_mel_ce: 3.605743885040283 (nan) | > loss: 0.01449319813400507 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13623131630251656) | > loader_time: 0.0065 (0.004649059087084714)  --> TIME: 2025-02-24 08:34:59 -- STEP: 23022/36938 -- GLOBAL_STEP: 244650 | > loss_text_ce: 0.03218753635883331 (0.04419725613513824) | > loss_mel_ce: 2.9449965953826904 (nan) | > loss: 0.011814222671091557 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.1362187934785508) | > loader_time: 0.0045 (0.004648976068742736)  --> TIME: 2025-02-24 08:35:09 -- STEP: 23072/36938 -- GLOBAL_STEP: 244700 | > loss_text_ce: 0.03799888864159584 (0.0441921159665657) | > loss_mel_ce: 3.544844627380371 (nan) | > loss: 0.014217633754014969 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13622419013229653) | > loader_time: 0.0046 (0.004648939426084137)  --> TIME: 2025-02-24 08:35:18 -- STEP: 23122/36938 -- GLOBAL_STEP: 244750 | > loss_text_ce: 0.09588170051574707 (0.044191713140502506) | > loss_mel_ce: 3.1601669788360596 (nan) | > loss: 0.01292082853615284 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.136228829145369) | > loader_time: 0.0044 (0.004648808469492664)  --> TIME: 2025-02-24 08:35:27 -- STEP: 23172/36938 -- GLOBAL_STEP: 244800 | > loss_text_ce: 0.051876768469810486 (0.04418982047006093) | > loss_mel_ce: 3.5872232913970947 (nan) | > loss: 0.014440874569118023 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.1362350619808489) | > loader_time: 0.0046 (0.004648681596916953)  --> TIME: 2025-02-24 08:35:37 -- STEP: 23222/36938 -- GLOBAL_STEP: 244850 | > loss_text_ce: 0.057177603244781494 (0.04419147760463118) | > loss_mel_ce: 3.6275458335876465 (nan) | > loss: 0.014621919021010399 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13624963588096872) | > loader_time: 0.0047 (0.004648570825082128)  --> TIME: 2025-02-24 08:35:46 -- STEP: 23272/36938 -- GLOBAL_STEP: 244900 | > loss_text_ce: 0.03422718495130539 (0.044189571418854066) | > loss_mel_ce: 3.433098793029785 (nan) | > loss: 0.013759230263531208 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13624495488132649) | > loader_time: 0.0045 (0.0046484664405240135)  --> TIME: 2025-02-24 08:35:56 -- STEP: 23322/36938 -- GLOBAL_STEP: 244950 | > loss_text_ce: 0.044028714299201965 (0.044188768471886304) | > loss_mel_ce: 3.659111976623535 (nan) | > loss: 0.014695003628730774 (nan) | > current_lr: 5e-06 | > step_time: 0.1642 (0.13624887425130072) | > loader_time: 0.0047 (0.00464838123190676)  --> TIME: 2025-02-24 08:36:05 -- STEP: 23372/36938 -- GLOBAL_STEP: 245000 | > loss_text_ce: 0.043357040733098984 (0.04418539687516589) | > loss_mel_ce: 3.3709051609039307 (nan) | > loss: 0.013548661023378372 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13625258801021317) | > loader_time: 0.0045 (0.004648217482863031)  --> TIME: 2025-02-24 08:36:14 -- STEP: 23422/36938 -- GLOBAL_STEP: 245050 | > loss_text_ce: 0.0377902127802372 (0.044181753838212164) | > loss_mel_ce: 3.6806819438934326 (nan) | > loss: 0.014755843207240105 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.13625398666320535) | > loader_time: 0.0047 (0.0046480600111775865)  --> TIME: 2025-02-24 08:36:24 -- STEP: 23472/36938 -- GLOBAL_STEP: 245100 | > loss_text_ce: 0.04315563663840294 (0.04418042632296411) | > loss_mel_ce: 2.2870609760284424 (nan) | > loss: 0.009246892295777798 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13625173905550692) | > loader_time: 0.0045 (0.004647891600276593)  --> TIME: 2025-02-24 08:36:33 -- STEP: 23522/36938 -- GLOBAL_STEP: 245150 | > loss_text_ce: 0.037901803851127625 (0.044177683904896764) | > loss_mel_ce: 0.056182861328125 (nan) | > loss: 0.0003733518533408642 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13625615326870258) | > loader_time: 0.0045 (0.004647788512099569)  --> TIME: 2025-02-24 08:36:42 -- STEP: 23572/36938 -- GLOBAL_STEP: 245200 | > loss_text_ce: 0.058662910014390945 (0.0441781881788209) | > loss_mel_ce: 3.696899175643921 (nan) | > loss: 0.014903024770319462 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.13625911638284763) | > loader_time: 0.0046 (0.004647674856698882)  --> TIME: 2025-02-24 08:36:52 -- STEP: 23622/36938 -- GLOBAL_STEP: 245250 | > loss_text_ce: 0.04057300090789795 (0.044181311792400174) | > loss_mel_ce: 1.4091821908950806 (nan) | > loss: 0.0057529970072209835 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13626901653878543) | > loader_time: 0.0044 (0.004647574197851327)  --> TIME: 2025-02-24 08:37:01 -- STEP: 23672/36938 -- GLOBAL_STEP: 245300 | > loss_text_ce: 0.033215153962373734 (0.04418182876036558) | > loss_mel_ce: 3.591545820236206 (nan) | > loss: 0.014383972622454166 (nan) | > current_lr: 5e-06 | > step_time: 0.162 (0.136264842359872) | > loader_time: 0.0048 (0.004647425398225482)  --> TIME: 2025-02-24 08:37:11 -- STEP: 23722/36938 -- GLOBAL_STEP: 245350 | > loss_text_ce: 0.042428597807884216 (0.04418382110339043) | > loss_mel_ce: 3.9293107986450195 (nan) | > loss: 0.015760870650410652 (nan) | > current_lr: 5e-06 | > step_time: 0.1573 (0.13626550565772166) | > loader_time: 0.0046 (0.00464726748690414)  --> TIME: 2025-02-24 08:37:20 -- STEP: 23772/36938 -- GLOBAL_STEP: 245400 | > loss_text_ce: 0.03833017125725746 (0.04418328214400995) | > loss_mel_ce: 3.5137939453125 (nan) | > loss: 0.0140957310795784 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13627325752337882) | > loader_time: 0.0045 (0.004647136887937535)  --> TIME: 2025-02-24 08:37:29 -- STEP: 23822/36938 -- GLOBAL_STEP: 245450 | > loss_text_ce: 0.04936366155743599 (0.04418477450496856) | > loss_mel_ce: 2.988539457321167 (nan) | > loss: 0.01205517165362835 (nan) | > current_lr: 5e-06 | > step_time: 0.1556 (0.13626893969249104) | > loader_time: 0.0046 (0.004646998490247126)  --> TIME: 2025-02-24 08:37:39 -- STEP: 23872/36938 -- GLOBAL_STEP: 245500 | > loss_text_ce: 0.04864726588129997 (0.04418270194502852) | > loss_mel_ce: 1.4035016298294067 (nan) | > loss: 0.005762496031820774 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13627256609839022) | > loader_time: 0.0044 (0.004646918379349581)  --> TIME: 2025-02-24 08:37:48 -- STEP: 23922/36938 -- GLOBAL_STEP: 245550 | > loss_text_ce: 0.04151131212711334 (0.04417793134605921) | > loss_mel_ce: 3.756701707839966 (nan) | > loss: 0.015072274953126907 (nan) | > current_lr: 5e-06 | > step_time: 0.1718 (0.13626826282887433) | > loader_time: 0.0048 (0.004646883970837364)  --> TIME: 2025-02-24 08:37:58 -- STEP: 23972/36938 -- GLOBAL_STEP: 245600 | > loss_text_ce: 0.037876926362514496 (0.04417265068741784) | > loss_mel_ce: 2.2590153217315674 (nan) | > loss: 0.009114651940762997 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13628234707929252) | > loader_time: 0.0046 (0.004646911249919784)  --> TIME: 2025-02-24 08:38:07 -- STEP: 24022/36938 -- GLOBAL_STEP: 245650 | > loss_text_ce: 0.04443950206041336 (0.044170678791407916) | > loss_mel_ce: 3.3912768363952637 (nan) | > loss: 0.013633796013891697 (nan) | > current_lr: 5e-06 | > step_time: 0.1956 (0.13628871836299836) | > loader_time: 0.0048 (0.0046468700321277475)  --> TIME: 2025-02-24 08:38:17 -- STEP: 24072/36938 -- GLOBAL_STEP: 245700 | > loss_text_ce: 0.024732284247875214 (0.04416662921326425) | > loss_mel_ce: 2.236678123474121 (nan) | > loss: 0.008973851799964905 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.13629740499388382) | > loader_time: 0.0045 (0.004646776838926946)  --> TIME: 2025-02-24 08:38:26 -- STEP: 24122/36938 -- GLOBAL_STEP: 245750 | > loss_text_ce: 0.040296245366334915 (0.04416401216399009) | > loss_mel_ce: 3.42799711227417 (nan) | > loss: 0.013763070106506348 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13629414937199874) | > loader_time: 0.0047 (0.004646670540592225)  --> TIME: 2025-02-24 08:38:35 -- STEP: 24172/36938 -- GLOBAL_STEP: 245800 | > loss_text_ce: 0.047120869159698486 (0.0441620103223792) | > loss_mel_ce: 2.6044323444366455 (nan) | > loss: 0.01052203681319952 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13628662132613237) | > loader_time: 0.0045 (0.004646548584914806)  --> TIME: 2025-02-24 08:38:44 -- STEP: 24222/36938 -- GLOBAL_STEP: 245850 | > loss_text_ce: 0.05237526819109917 (0.04416437789863257) | > loss_mel_ce: 0.979076623916626 (nan) | > loss: 0.0040930635295808315 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13627340425282364) | > loader_time: 0.0045 (0.004646365013182364)  --> TIME: 2025-02-24 08:38:53 -- STEP: 24272/36938 -- GLOBAL_STEP: 245900 | > loss_text_ce: 0.031475361436605453 (0.04416098169848554) | > loss_mel_ce: 3.769986867904663 (nan) | > loss: 0.015085168182849884 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13627498682856026) | > loader_time: 0.0045 (0.004646295415152782)  --> TIME: 2025-02-24 08:39:03 -- STEP: 24322/36938 -- GLOBAL_STEP: 245950 | > loss_text_ce: 0.04255004972219467 (0.04416091338363018) | > loss_mel_ce: 2.9630632400512695 (nan) | > loss: 0.011927037499845028 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.1362778568099219) | > loader_time: 0.0045 (0.004646301004861295)  --> TIME: 2025-02-24 08:39:12 -- STEP: 24372/36938 -- GLOBAL_STEP: 246000 | > loss_text_ce: 0.04903567209839821 (0.04416085232558789) | > loss_mel_ce: 3.2505176067352295 (nan) | > loss: 0.013093466870486736 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13627916554913194) | > loader_time: 0.0046 (0.0046462479354554935)  --> TIME: 2025-02-24 08:39:21 -- STEP: 24422/36938 -- GLOBAL_STEP: 246050 | > loss_text_ce: 0.03696680814027786 (0.04415802600676881) | > loss_mel_ce: 1.2134323120117188 (nan) | > loss: 0.004961901344358921 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13627327064479489) | > loader_time: 0.0044 (0.00464609092776127)  --> TIME: 2025-02-24 08:39:30 -- STEP: 24472/36938 -- GLOBAL_STEP: 246100 | > loss_text_ce: 0.040878456085920334 (0.044156363262998766) | > loss_mel_ce: 3.473855972290039 (nan) | > loss: 0.013947360217571259 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13626455892414482) | > loader_time: 0.0047 (0.004645900618761059)  --> TIME: 2025-02-24 08:39:39 -- STEP: 24522/36938 -- GLOBAL_STEP: 246150 | > loss_text_ce: 0.05295172706246376 (0.0441547178296963) | > loss_mel_ce: 1.8853813409805298 (nan) | > loss: 0.0076917982660233974 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13626077264394224) | > loader_time: 0.0045 (0.004645742295509931)  --> TIME: 2025-02-24 08:39:48 -- STEP: 24572/36938 -- GLOBAL_STEP: 246200 | > loss_text_ce: 0.05329431965947151 (0.044154976357253815) | > loss_mel_ce: 3.6153972148895264 (nan) | > loss: 0.01455830130726099 (nan) | > current_lr: 5e-06 | > step_time: 0.1558 (0.1362583043713844) | > loader_time: 0.0048 (0.0046457311879262685)  --> TIME: 2025-02-24 08:39:58 -- STEP: 24622/36938 -- GLOBAL_STEP: 246250 | > loss_text_ce: 0.032502904534339905 (0.04415555397928657) | > loss_mel_ce: 2.9221444129943848 (nan) | > loss: 0.01172479148954153 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.1362530226905382) | > loader_time: 0.0046 (0.004645563684444405)  --> TIME: 2025-02-24 08:40:07 -- STEP: 24672/36938 -- GLOBAL_STEP: 246300 | > loss_text_ce: 0.039725564420223236 (0.04415310899172558) | > loss_mel_ce: 1.8264837265014648 (nan) | > loss: 0.00740559259429574 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13624768491578318) | > loader_time: 0.0045 (0.004645427705867399)  --> TIME: 2025-02-24 08:40:16 -- STEP: 24722/36938 -- GLOBAL_STEP: 246350 | > loss_text_ce: 0.041718896478414536 (0.044151112735909936) | > loss_mel_ce: 3.35471510887146 (nan) | > loss: 0.013477914035320282 (nan) | > current_lr: 5e-06 | > step_time: 0.1865 (0.13625511024850978) | > loader_time: 0.0048 (0.00464532615663778)  --> TIME: 2025-02-24 08:40:26 -- STEP: 24772/36938 -- GLOBAL_STEP: 246400 | > loss_text_ce: 0.0444931797683239 (0.044151013534489796) | > loss_mel_ce: 3.0677926540374756 (nan) | > loss: 0.01235034130513668 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13625792765605813) | > loader_time: 0.0046 (0.004645206519018603)  --> TIME: 2025-02-24 08:40:35 -- STEP: 24822/36938 -- GLOBAL_STEP: 246450 | > loss_text_ce: 0.0473753958940506 (0.04414807098002356) | > loss_mel_ce: 3.4896082878112793 (nan) | > loss: 0.01403565052896738 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.13626563140823642) | > loader_time: 0.0046 (0.004645185239673656)  --> TIME: 2025-02-24 08:40:44 -- STEP: 24872/36938 -- GLOBAL_STEP: 246500 | > loss_text_ce: 0.039586812257766724 (0.04414781436257276) | > loss_mel_ce: 3.450474977493286 (nan) | > loss: 0.013849452137947083 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.13626298015046315) | > loader_time: 0.0045 (0.004645119519738264)  --> TIME: 2025-02-24 08:40:54 -- STEP: 24922/36938 -- GLOBAL_STEP: 246550 | > loss_text_ce: 0.043712206184864044 (0.04414511359661497) | > loss_mel_ce: 3.2894694805145264 (nan) | > loss: 0.01322691235691309 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13626076639219947) | > loader_time: 0.0046 (0.004645075636167914)  --> TIME: 2025-02-24 08:41:03 -- STEP: 24972/36938 -- GLOBAL_STEP: 246600 | > loss_text_ce: 0.04409611597657204 (0.04414566310536388) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13626146866460648) | > loader_time: 0.0045 (0.00464491818017267)  --> TIME: 2025-02-24 08:41:12 -- STEP: 25022/36938 -- GLOBAL_STEP: 246650 | > loss_text_ce: 0.04161052778363228 (0.044143272351138754) | > loss_mel_ce: 0.05095471814274788 (nan) | > loss: 0.00036732241278514266 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.1362621069106922) | > loader_time: 0.0046 (0.0046447871276566076)  --> TIME: 2025-02-24 08:41:22 -- STEP: 25072/36938 -- GLOBAL_STEP: 246700 | > loss_text_ce: 0.06911590695381165 (0.0441461870307574) | > loss_mel_ce: 2.0021162033081055 (nan) | > loss: 0.008219175040721893 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13625853884805908) | > loader_time: 0.0045 (0.004644615669576057)  --> TIME: 2025-02-24 08:41:31 -- STEP: 25122/36938 -- GLOBAL_STEP: 246750 | > loss_text_ce: 0.043614357709884644 (0.04414118771878613) | > loss_mel_ce: 3.2206733226776123 (nan) | > loss: 0.01295352354645729 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13625820939987496) | > loader_time: 0.0045 (0.004644436466495493)  --> TIME: 2025-02-24 08:41:40 -- STEP: 25172/36938 -- GLOBAL_STEP: 246800 | > loss_text_ce: 0.039640162140131 (0.04414049126662126) | > loss_mel_ce: 2.643275499343872 (nan) | > loss: 0.010646491311490536 (nan) | > current_lr: 5e-06 | > step_time: 0.1442 (0.13626005455209866) | > loader_time: 0.0045 (0.0046443427364870945)  --> TIME: 2025-02-24 08:41:49 -- STEP: 25222/36938 -- GLOBAL_STEP: 246850 | > loss_text_ce: 0.03647163137793541 (0.04413729450176411) | > loss_mel_ce: 2.2667980194091797 (nan) | > loss: 0.009139959700405598 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.13625413623206667) | > loader_time: 0.0046 (0.004644170551180531)  --> TIME: 2025-02-24 08:41:58 -- STEP: 25272/36938 -- GLOBAL_STEP: 246900 | > loss_text_ce: 0.04129882901906967 (0.04413690138350211) | > loss_mel_ce: 1.9615217447280884 (nan) | > loss: 0.007947701029479504 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13624800359119768) | > loader_time: 0.0046 (0.004644119869723017)  --> TIME: 2025-02-24 08:42:08 -- STEP: 25322/36938 -- GLOBAL_STEP: 246950 | > loss_text_ce: 0.04476042464375496 (0.04413244683357529) | > loss_mel_ce: 2.830148935317993 (nan) | > loss: 0.011408370919525623 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13624474364108136) | > loader_time: 0.0046 (0.004644010042694132)  --> TIME: 2025-02-24 08:42:17 -- STEP: 25372/36938 -- GLOBAL_STEP: 247000 | > loss_text_ce: 0.04130282253026962 (0.04413284897272999) | > loss_mel_ce: 3.3282206058502197 (nan) | > loss: 0.013371125794947147 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.136246231195022) | > loader_time: 0.0045 (0.004644014633135974)  --> TIME: 2025-02-24 08:42:26 -- STEP: 25422/36938 -- GLOBAL_STEP: 247050 | > loss_text_ce: 0.04399603232741356 (0.04413105059514999) | > loss_mel_ce: 2.8286094665527344 (nan) | > loss: 0.011399229057133198 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.1362467520862136) | > loader_time: 0.0045 (0.004644065431827851)  --> TIME: 2025-02-24 08:42:35 -- STEP: 25472/36938 -- GLOBAL_STEP: 247100 | > loss_text_ce: 0.05431031435728073 (0.04412531208766971) | > loss_mel_ce: 2.801152229309082 (nan) | > loss: 0.011331201530992985 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.13624039972809396) | > loader_time: 0.0047 (0.004643913349089875)  --> TIME: 2025-02-24 08:42:45 -- STEP: 25522/36938 -- GLOBAL_STEP: 247150 | > loss_text_ce: 0.04388350993394852 (0.044122615285029344) | > loss_mel_ce: 3.612715244293213 (nan) | > loss: 0.014510313980281353 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13623916466644728) | > loader_time: 0.0046 (0.0046438634344219696)  --> TIME: 2025-02-24 08:42:54 -- STEP: 25572/36938 -- GLOBAL_STEP: 247200 | > loss_text_ce: 0.05384290590882301 (0.04412032915918756) | > loss_mel_ce: 3.627424478530884 (nan) | > loss: 0.014608205296099186 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13623857776939302) | > loader_time: 0.0046 (0.00464382191023576)  --> TIME: 2025-02-24 08:43:03 -- STEP: 25622/36938 -- GLOBAL_STEP: 247250 | > loss_text_ce: 0.04526254162192345 (0.04411665781110777) | > loss_mel_ce: 2.3569233417510986 (nan) | > loss: 0.009532484225928783 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13624698734610993) | > loader_time: 0.0045 (0.0046438694037464695)  --> TIME: 2025-02-24 08:43:13 -- STEP: 25672/36938 -- GLOBAL_STEP: 247300 | > loss_text_ce: 0.03927840292453766 (0.04411395525058528) | > loss_mel_ce: 3.803901195526123 (nan) | > loss: 0.01525071356445551 (nan) | > current_lr: 5e-06 | > step_time: 0.1435 (0.13623979216834478) | > loader_time: 0.0046 (0.004643786572042938)  --> TIME: 2025-02-24 08:43:22 -- STEP: 25722/36938 -- GLOBAL_STEP: 247350 | > loss_text_ce: 0.045815661549568176 (0.04411349070112337) | > loss_mel_ce: 3.458338737487793 (nan) | > loss: 0.013905375264585018 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.1362434085610589) | > loader_time: 0.0047 (0.004643771856219118)  --> TIME: 2025-02-24 08:43:31 -- STEP: 25772/36938 -- GLOBAL_STEP: 247400 | > loss_text_ce: 0.040196314454078674 (0.044108662273024374) | > loss_mel_ce: 3.0031683444976807 (nan) | > loss: 0.0120768453925848 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13623817713016806) | > loader_time: 0.0047 (0.0046437557543284)  --> TIME: 2025-02-24 08:43:40 -- STEP: 25822/36938 -- GLOBAL_STEP: 247450 | > loss_text_ce: 0.03669329732656479 (0.04410850375994708) | > loss_mel_ce: 3.3206560611724854 (nan) | > loss: 0.013322816230356693 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13623447957798582) | > loader_time: 0.0045 (0.004643707657273967)  --> TIME: 2025-02-24 08:43:50 -- STEP: 25872/36938 -- GLOBAL_STEP: 247500 | > loss_text_ce: 0.03800245746970177 (0.04410849454384008) | > loss_mel_ce: 2.7545697689056396 (nan) | > loss: 0.011081636883318424 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13624936953214642) | > loader_time: 0.0047 (0.004643724225668798)  --> TIME: 2025-02-24 08:43:59 -- STEP: 25922/36938 -- GLOBAL_STEP: 247550 | > loss_text_ce: 0.05884235352277756 (0.04410868396787924) | > loss_mel_ce: 2.602682113647461 (nan) | > loss: 0.010561605915427208 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13625003057040896) | > loader_time: 0.0047 (0.004643770879676731)  --> TIME: 2025-02-24 08:44:09 -- STEP: 25972/36938 -- GLOBAL_STEP: 247600 | > loss_text_ce: 0.017828073352575302 (0.04410519128565313) | > loss_mel_ce: 1.4968371391296387 (nan) | > loss: 0.006010576616972685 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.136248375378789) | > loader_time: 0.0045 (0.004643779248572904)  --> TIME: 2025-02-24 08:44:18 -- STEP: 26022/36938 -- GLOBAL_STEP: 247650 | > loss_text_ce: 0.05601939931511879 (0.04410522781921854) | > loss_mel_ce: 3.866642475128174 (nan) | > loss: 0.015566118992865086 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1362423423928785) | > loader_time: 0.0047 (0.004643820569202037)  --> TIME: 2025-02-24 08:44:27 -- STEP: 26072/36938 -- GLOBAL_STEP: 247700 | > loss_text_ce: 0.037773653864860535 (0.04410318304684107) | > loss_mel_ce: 3.247551441192627 (nan) | > loss: 0.013037004508078098 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13623952551940058) | > loader_time: 0.0047 (0.004643811198164598)  --> TIME: 2025-02-24 08:44:36 -- STEP: 26122/36938 -- GLOBAL_STEP: 247750 | > loss_text_ce: 0.03710407018661499 (0.04410156319442639) | > loss_mel_ce: 3.2063870429992676 (nan) | > loss: 0.012870997190475464 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13623081360903036) | > loader_time: 0.0046 (0.004643740702182861)  --> TIME: 2025-02-24 08:44:45 -- STEP: 26172/36938 -- GLOBAL_STEP: 247800 | > loss_text_ce: 0.052562277764081955 (0.044100241639605256) | > loss_mel_ce: 3.9694833755493164 (nan) | > loss: 0.01596049964427948 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13622824730866545) | > loader_time: 0.0047 (0.004643722765128303)  --> TIME: 2025-02-24 08:44:54 -- STEP: 26222/36938 -- GLOBAL_STEP: 247850 | > loss_text_ce: 0.05187175050377846 (0.04409761275092643) | > loss_mel_ce: 0.018689313903450966 (nan) | > loss: 0.00028000425663776696 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13623097385515306) | > loader_time: 0.0045 (0.0046437812264292135)  --> TIME: 2025-02-24 08:45:03 -- STEP: 26272/36938 -- GLOBAL_STEP: 247900 | > loss_text_ce: 0.05072521045804024 (0.0440997394197347) | > loss_mel_ce: 2.7369000911712646 (nan) | > loss: 0.011062005534768105 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13622473721164446) | > loader_time: 0.0046 (0.004643808210883424)  --> TIME: 2025-02-24 08:45:13 -- STEP: 26322/36938 -- GLOBAL_STEP: 247950 | > loss_text_ce: 0.046765778213739395 (0.04409974556238332) | > loss_mel_ce: 1.9716578722000122 (nan) | > loss: 0.008009618148207664 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13623132254430467) | > loader_time: 0.0044 (0.004643854331520928)  --> TIME: 2025-02-24 08:45:23 -- STEP: 26372/36938 -- GLOBAL_STEP: 248000 | > loss_text_ce: 0.045970119535923004 (0.044098928309735094) | > loss_mel_ce: 3.2562739849090576 (nan) | > loss: 0.013104144483804703 (nan) | > current_lr: 5e-06 | > step_time: 0.1533 (0.13624827305427684) | > loader_time: 0.0051 (0.0046439508684448265)  --> TIME: 2025-02-24 08:45:32 -- STEP: 26422/36938 -- GLOBAL_STEP: 248050 | > loss_text_ce: 0.03992447629570961 (0.044098091349714486) | > loss_mel_ce: 1.3379796743392944 (nan) | > loss: 0.005467873997986317 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13624969222473288) | > loader_time: 0.0046 (0.004643953962732446)  --> TIME: 2025-02-24 08:45:41 -- STEP: 26472/36938 -- GLOBAL_STEP: 248100 | > loss_text_ce: 0.0450972244143486 (0.04409544655853174) | > loss_mel_ce: 3.3945364952087402 (nan) | > loss: 0.013649340718984604 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13625329846376116) | > loader_time: 0.0045 (0.004643919155222272)  --> TIME: 2025-02-24 08:45:51 -- STEP: 26522/36938 -- GLOBAL_STEP: 248150 | > loss_text_ce: 0.03871710225939751 (0.044092149298888286) | > loss_mel_ce: 2.1984481811523438 (nan) | > loss: 0.008877640590071678 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.136250112078487) | > loader_time: 0.0045 (0.004643839891208017)  --> TIME: 2025-02-24 08:46:00 -- STEP: 26572/36938 -- GLOBAL_STEP: 248200 | > loss_text_ce: 0.052534740418195724 (0.04409146674253381) | > loss_mel_ce: 0.0867513045668602 (nan) | > loss: 0.0005527224275283515 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13624439613836722) | > loader_time: 0.0045 (0.0046438251510004264)  --> TIME: 2025-02-24 08:46:09 -- STEP: 26622/36938 -- GLOBAL_STEP: 248250 | > loss_text_ce: 0.040624115616083145 (0.04409116550364961) | > loss_mel_ce: 3.542935371398926 (nan) | > loss: 0.01422047521919012 (nan) | > current_lr: 5e-06 | > step_time: 0.1436 (0.13624539385872914) | > loader_time: 0.0046 (0.004643800113375035)  --> TIME: 2025-02-24 08:46:18 -- STEP: 26672/36938 -- GLOBAL_STEP: 248300 | > loss_text_ce: 0.02978951297700405 (0.04408963830497251) | > loss_mel_ce: 3.3443119525909424 (nan) | > loss: 0.013389292173087597 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13624247260521358) | > loader_time: 0.0045 (0.004643770217466415)  --> TIME: 2025-02-24 08:46:28 -- STEP: 26722/36938 -- GLOBAL_STEP: 248350 | > loss_text_ce: 0.04733841493725777 (0.044090237508525934) | > loss_mel_ce: 2.4885406494140625 (nan) | > loss: 0.010063013061881065 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13624418385861664) | > loader_time: 0.0046 (0.004643819296612488)  --> TIME: 2025-02-24 08:46:37 -- STEP: 26772/36938 -- GLOBAL_STEP: 248400 | > loss_text_ce: 0.03735533729195595 (0.044085871063715934) | > loss_mel_ce: 3.2515714168548584 (nan) | > loss: 0.013051297515630722 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1362389973348771) | > loader_time: 0.0045 (0.00464391415382416)  --> TIME: 2025-02-24 08:46:46 -- STEP: 26822/36938 -- GLOBAL_STEP: 248450 | > loss_text_ce: 0.03907657414674759 (0.04408294839686407) | > loss_mel_ce: 3.9503347873687744 (nan) | > loss: 0.015830999240279198 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.1362406258685828) | > loader_time: 0.0047 (0.004643894550331146)  --> TIME: 2025-02-24 08:46:56 -- STEP: 26872/36938 -- GLOBAL_STEP: 248500 | > loss_text_ce: 0.05794405937194824 (0.04408072257289237) | > loss_mel_ce: 3.4034407138824463 (nan) | > loss: 0.013735654763877392 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13624360999488982) | > loader_time: 0.0045 (0.00464383926410054)  --> TIME: 2025-02-24 08:47:05 -- STEP: 26922/36938 -- GLOBAL_STEP: 248550 | > loss_text_ce: 0.04694535583257675 (0.04407862817250575) | > loss_mel_ce: 3.4306204319000244 (nan) | > loss: 0.013799864798784256 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13625206846192303) | > loader_time: 0.0045 (0.004643909945883637)  --> TIME: 2025-02-24 08:47:14 -- STEP: 26972/36938 -- GLOBAL_STEP: 248600 | > loss_text_ce: 0.03527770936489105 (0.0440775677657402) | > loss_mel_ce: 2.7256381511688232 (nan) | > loss: 0.010956015437841415 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1362541355772757) | > loader_time: 0.0045 (0.004643898803686026)  --> TIME: 2025-02-24 08:47:23 -- STEP: 27022/36938 -- GLOBAL_STEP: 248650 | > loss_text_ce: 0.03417767956852913 (0.04407628289866447) | > loss_mel_ce: 3.150373935699463 (nan) | > loss: 0.01263711042702198 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.1362472680220115) | > loader_time: 0.0047 (0.0046439472323703945)  --> TIME: 2025-02-24 08:47:33 -- STEP: 27072/36938 -- GLOBAL_STEP: 248700 | > loss_text_ce: 0.03981119394302368 (0.044076592672138905) | > loss_mel_ce: 2.190523624420166 (nan) | > loss: 0.00885053537786007 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.13624980014442875) | > loader_time: 0.0045 (0.004644019736183994)  --> TIME: 2025-02-24 08:47:42 -- STEP: 27122/36938 -- GLOBAL_STEP: 248750 | > loss_text_ce: 0.038643695414066315 (0.044070238805269904) | > loss_mel_ce: 0.0174560546875 (nan) | > loss: 0.0002226180658908561 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1362447241890815) | > loader_time: 0.0044 (0.004644009622357651)  --> TIME: 2025-02-24 08:47:51 -- STEP: 27172/36938 -- GLOBAL_STEP: 248800 | > loss_text_ce: 0.042991459369659424 (0.044070207224278235) | > loss_mel_ce: 3.0826539993286133 (nan) | > loss: 0.012403355911374092 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13624508665532187) | > loader_time: 0.0046 (0.0046439832692010555)  --> TIME: 2025-02-24 08:48:01 -- STEP: 27222/36938 -- GLOBAL_STEP: 248850 | > loss_text_ce: 0.04663852974772453 (0.044068816074321476) | > loss_mel_ce: 3.365015745162964 (nan) | > loss: 0.01353831123560667 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1362508543968277) | > loader_time: 0.0045 (0.004643977875135314)  --> TIME: 2025-02-24 08:48:10 -- STEP: 27272/36938 -- GLOBAL_STEP: 248900 | > loss_text_ce: 0.03967411071062088 (0.044064029258452556) | > loss_mel_ce: 3.091815710067749 (nan) | > loss: 0.012426547706127167 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13625553141034763) | > loader_time: 0.0047 (0.004644002067130043)  --> TIME: 2025-02-24 08:48:19 -- STEP: 27322/36938 -- GLOBAL_STEP: 248950 | > loss_text_ce: 0.044470496475696564 (0.04406308540452611) | > loss_mel_ce: 3.5746257305145264 (nan) | > loss: 0.014361494220793247 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13625763602438698) | > loader_time: 0.0045 (0.0046439332796586255)  --> TIME: 2025-02-24 08:48:29 -- STEP: 27372/36938 -- GLOBAL_STEP: 249000 | > loss_text_ce: 0.04393262416124344 (0.04406308701053472) | > loss_mel_ce: 0.4919951558113098 (nan) | > loss: 0.0021266976837068796 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1362629772173325) | > loader_time: 0.0046 (0.0046440344203133215)  --> TIME: 2025-02-24 08:48:38 -- STEP: 27422/36938 -- GLOBAL_STEP: 249050 | > loss_text_ce: 0.05047185346484184 (0.0440624531350801) | > loss_mel_ce: 4.069377899169922 (nan) | > loss: 0.016348611563444138 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.1362537736487893) | > loader_time: 0.0046 (0.004644011148746679)  --> TIME: 2025-02-24 08:48:47 -- STEP: 27472/36938 -- GLOBAL_STEP: 249100 | > loss_text_ce: 0.048448801040649414 (0.044063494591093266) | > loss_mel_ce: 3.5114941596984863 (nan) | > loss: 0.014126759022474289 (nan) | > current_lr: 5e-06 | > step_time: 0.152 (0.13624889077700122) | > loader_time: 0.0046 (0.004643943440657248)  --> TIME: 2025-02-24 08:48:56 -- STEP: 27522/36938 -- GLOBAL_STEP: 249150 | > loss_text_ce: 0.038778435438871384 (0.04406246868399413) | > loss_mel_ce: 1.5132886171340942 (nan) | > loss: 0.006158996373414993 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.13624150305499164) | > loader_time: 0.0045 (0.004643863010316383)  --> TIME: 2025-02-24 08:49:06 -- STEP: 27572/36938 -- GLOBAL_STEP: 249200 | > loss_text_ce: 0.03013782948255539 (0.04406030545787069) | > loss_mel_ce: 1.9027913808822632 (nan) | > loss: 0.007670354098081589 (nan) | > current_lr: 5e-06 | > step_time: 0.1084 (0.13624211860017857) | > loader_time: 0.0044 (0.004643754379403049)  --> TIME: 2025-02-24 08:49:15 -- STEP: 27622/36938 -- GLOBAL_STEP: 249250 | > loss_text_ce: 0.04921204596757889 (0.04405819512746532) | > loss_mel_ce: 3.499040126800537 (nan) | > loss: 0.014080366119742393 (nan) | > current_lr: 5e-06 | > step_time: 0.1697 (0.13624443634821276) | > loader_time: 0.0048 (0.004643753370569923)  --> TIME: 2025-02-24 08:49:24 -- STEP: 27672/36938 -- GLOBAL_STEP: 249300 | > loss_text_ce: 0.044811490923166275 (0.04405888618860812) | > loss_mel_ce: 2.7898330688476562 (nan) | > loss: 0.011248590424656868 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13624433009532858) | > loader_time: 0.0045 (0.004643799847486752)  --> TIME: 2025-02-24 08:49:33 -- STEP: 27722/36938 -- GLOBAL_STEP: 249350 | > loss_text_ce: 0.04135037213563919 (0.04406076838081532) | > loss_mel_ce: 2.797473192214966 (nan) | > loss: 0.011265173554420471 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.1362330055057934) | > loader_time: 0.0047 (0.004643673909174019)  --> TIME: 2025-02-24 08:49:42 -- STEP: 27772/36938 -- GLOBAL_STEP: 249400 | > loss_text_ce: 0.051593974232673645 (0.04406004135355226) | > loss_mel_ce: 3.9226768016815186 (nan) | > loss: 0.015770917758345604 (nan) | > current_lr: 5e-06 | > step_time: 0.1817 (0.1362267608912492) | > loader_time: 0.0049 (0.004643566401018852)  --> TIME: 2025-02-24 08:49:52 -- STEP: 27822/36938 -- GLOBAL_STEP: 249450 | > loss_text_ce: 0.04294325411319733 (0.044058907285768) | > loss_mel_ce: 3.6555428504943848 (nan) | > loss: 0.014676532708108425 (nan) | > current_lr: 5e-06 | > step_time: 0.186 (0.13622684788458891) | > loader_time: 0.0049 (0.004643567099799842)  --> TIME: 2025-02-24 08:50:01 -- STEP: 27872/36938 -- GLOBAL_STEP: 249500 | > loss_text_ce: 0.047540392726659775 (0.04405799320543785) | > loss_mel_ce: 3.3724124431610107 (nan) | > loss: 0.013571241870522499 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13622927220044329) | > loader_time: 0.0045 (0.004643514735279875)  --> TIME: 2025-02-24 08:50:10 -- STEP: 27922/36938 -- GLOBAL_STEP: 249550 | > loss_text_ce: 0.044490594416856766 (0.044054446982256926) | > loss_mel_ce: 2.2334330081939697 (nan) | > loss: 0.00903937965631485 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13623198269346204) | > loader_time: 0.0045 (0.004643466332419673)  --> TIME: 2025-02-24 08:50:20 -- STEP: 27972/36938 -- GLOBAL_STEP: 249600 | > loss_text_ce: 0.04882190749049187 (0.044055475032632664) | > loss_mel_ce: 3.551072359085083 (nan) | > loss: 0.014285295270383358 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.1362259149909383) | > loader_time: 0.0046 (0.004643377829191238)  --> TIME: 2025-02-24 08:50:29 -- STEP: 28022/36938 -- GLOBAL_STEP: 249650 | > loss_text_ce: 0.04262283816933632 (0.04405296945126268) | > loss_mel_ce: 2.7977046966552734 (nan) | > loss: 0.011271141469478607 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.1362280885886186) | > loader_time: 0.0045 (0.004643339261994355)  --> TIME: 2025-02-24 08:50:39 -- STEP: 28072/36938 -- GLOBAL_STEP: 249700 | > loss_text_ce: 0.05869530141353607 (0.04405434812674801) | > loss_mel_ce: 4.236080646514893 (nan) | > loss: 0.01704276353120804 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13623719142861976) | > loader_time: 0.0046 (0.0046434618785327455)  --> TIME: 2025-02-24 08:50:48 -- STEP: 28122/36938 -- GLOBAL_STEP: 249750 | > loss_text_ce: 0.04598534479737282 (0.044053074621726404) | > loss_mel_ce: 2.5131282806396484 (nan) | > loss: 0.010155213996767998 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.136234010446887) | > loader_time: 0.0047 (0.00464338870028513)  --> TIME: 2025-02-24 08:50:57 -- STEP: 28172/36938 -- GLOBAL_STEP: 249800 | > loss_text_ce: 0.04261002317070961 (0.04405007301697483) | > loss_mel_ce: 2.0390660762786865 (nan) | > loss: 0.008260619826614857 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13623042611359568) | > loader_time: 0.0047 (0.004643337455439682)  --> TIME: 2025-02-24 08:51:06 -- STEP: 28222/36938 -- GLOBAL_STEP: 249850 | > loss_text_ce: 0.04708695784211159 (0.04404742394480691) | > loss_mel_ce: 3.1379692554473877 (nan) | > loss: 0.012639112770557404 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13622859198301546) | > loader_time: 0.0046 (0.004643301032502039)  --> TIME: 2025-02-24 08:51:15 -- STEP: 28272/36938 -- GLOBAL_STEP: 249900 | > loss_text_ce: 0.04326662793755531 (0.04404314324219795) | > loss_mel_ce: 3.6281261444091797 (nan) | > loss: 0.014569018967449665 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13622110720300185) | > loader_time: 0.0048 (0.004643185459494775)  --> TIME: 2025-02-24 08:51:25 -- STEP: 28322/36938 -- GLOBAL_STEP: 249950 | > loss_text_ce: 0.03716671094298363 (0.04404031280827338) | > loss_mel_ce: 3.937089204788208 (nan) | > loss: 0.01577085815370083 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13622185716049268) | > loader_time: 0.0047 (0.004643175294023626)  --> TIME: 2025-02-24 08:51:34 -- STEP: 28372/36938 -- GLOBAL_STEP: 250000 | > loss_text_ce: 0.043271731585264206 (0.04403674385056569) | > loss_mel_ce: 3.776606321334839 (nan) | > loss: 0.015158247202634811 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1362315955071972) | > loader_time: 0.0046 (0.004643126315901128) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_250000.pth  --> TIME: 2025-02-24 08:51:49 -- STEP: 28422/36938 -- GLOBAL_STEP: 250050 | > loss_text_ce: 0.045897092670202255 (0.04403639477331252) | > loss_mel_ce: 3.962071418762207 (nan) | > loss: 0.015904637053608894 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.13622414237818325) | > loader_time: 0.0045 (0.00464315519620778)  --> TIME: 2025-02-24 08:51:59 -- STEP: 28472/36938 -- GLOBAL_STEP: 250100 | > loss_text_ce: 0.03025057725608349 (0.044032241606616875) | > loss_mel_ce: 1.7117726802825928 (nan) | > loss: 0.006912791170179844 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13622131714260918) | > loader_time: 0.0045 (0.004643131730997937)  --> TIME: 2025-02-24 08:52:08 -- STEP: 28522/36938 -- GLOBAL_STEP: 250150 | > loss_text_ce: 0.044895242899656296 (0.04403087908107389) | > loss_mel_ce: 3.86354398727417 (nan) | > loss: 0.015509679913520813 (nan) | > current_lr: 5e-06 | > step_time: 0.187 (0.13621460167675137) | > loader_time: 0.0048 (0.004643083680320526)  --> TIME: 2025-02-24 08:52:17 -- STEP: 28572/36938 -- GLOBAL_STEP: 250200 | > loss_text_ce: 0.035822544246912 (0.044027269878483655) | > loss_mel_ce: 3.2657532691955566 (nan) | > loss: 0.013101492077112198 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13620888383285473) | > loader_time: 0.0046 (0.004643104840072599)  --> TIME: 2025-02-24 08:52:26 -- STEP: 28622/36938 -- GLOBAL_STEP: 250250 | > loss_text_ce: 0.055678028613328934 (0.04402712143970723) | > loss_mel_ce: 3.6255199909210205 (nan) | > loss: 0.0146079296246171 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13620597660962286) | > loader_time: 0.0048 (0.0046430700822043276)  --> TIME: 2025-02-24 08:52:35 -- STEP: 28672/36938 -- GLOBAL_STEP: 250300 | > loss_text_ce: 0.03228513523936272 (0.044027965860842434) | > loss_mel_ce: 3.049830436706543 (nan) | > loss: 0.012230617925524712 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13620161162024189) | > loader_time: 0.0044 (0.004643100280580758)  --> TIME: 2025-02-24 08:52:44 -- STEP: 28722/36938 -- GLOBAL_STEP: 250350 | > loss_text_ce: 0.042987048625946045 (0.044028038539135485) | > loss_mel_ce: 3.5493052005767822 (nan) | > loss: 0.014255128800868988 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.1362047681369529) | > loader_time: 0.0046 (0.004643105645422449)  --> TIME: 2025-02-24 08:52:54 -- STEP: 28772/36938 -- GLOBAL_STEP: 250400 | > loss_text_ce: 0.049226049333810806 (0.04403083928932935) | > loss_mel_ce: 3.5218346118927 (nan) | > loss: 0.014170876704156399 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13620656335136833) | > loader_time: 0.0045 (0.004643139082782526)  --> TIME: 2025-02-24 08:53:03 -- STEP: 28822/36938 -- GLOBAL_STEP: 250450 | > loss_text_ce: 0.04744332656264305 (0.044030387490842905) | > loss_mel_ce: 3.6298115253448486 (nan) | > loss: 0.014592282474040985 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13621002646209338) | > loader_time: 0.0046 (0.004643183248857514)  --> TIME: 2025-02-24 08:53:12 -- STEP: 28872/36938 -- GLOBAL_STEP: 250500 | > loss_text_ce: 0.045473404228687286 (0.044028856582433724) | > loss_mel_ce: 1.8088170289993286 (nan) | > loss: 0.00735829584300518 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.1362043184714169) | > loader_time: 0.0044 (0.004643100174750323)  --> TIME: 2025-02-24 08:53:22 -- STEP: 28922/36938 -- GLOBAL_STEP: 250550 | > loss_text_ce: 0.06219116225838661 (0.04402842547202337) | > loss_mel_ce: 2.2302653789520264 (nan) | > loss: 0.009097050875425339 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13621190153013488) | > loader_time: 0.0047 (0.00464318343674885)  --> TIME: 2025-02-24 08:53:31 -- STEP: 28972/36938 -- GLOBAL_STEP: 250600 | > loss_text_ce: 0.05267960578203201 (0.044031814107800625) | > loss_mel_ce: 2.838792562484741 (nan) | > loss: 0.011474096216261387 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13620881493309242) | > loader_time: 0.0049 (0.004643306545537741)  --> TIME: 2025-02-24 08:53:40 -- STEP: 29022/36938 -- GLOBAL_STEP: 250650 | > loss_text_ce: 0.04578714445233345 (0.04402936615097167) | > loss_mel_ce: 2.926194906234741 (nan) | > loss: 0.011793579906225204 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13620552142740533) | > loader_time: 0.0046 (0.0046432705555142055)  --> TIME: 2025-02-24 08:53:50 -- STEP: 29072/36938 -- GLOBAL_STEP: 250700 | > loss_text_ce: 0.038296520709991455 (0.04402553924970189) | > loss_mel_ce: 2.991400957107544 (nan) | > loss: 0.012022609822452068 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13621057256703656) | > loader_time: 0.0048 (0.004643243275701687)  --> TIME: 2025-02-24 08:53:59 -- STEP: 29122/36938 -- GLOBAL_STEP: 250750 | > loss_text_ce: 0.039074018597602844 (0.044022441121311276) | > loss_mel_ce: 3.4750726222991943 (nan) | > loss: 0.013945027254521847 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13620987614549898) | > loader_time: 0.0046 (0.004643174582036589)  --> TIME: 2025-02-24 08:54:08 -- STEP: 29172/36938 -- GLOBAL_STEP: 250800 | > loss_text_ce: 0.0379122756421566 (0.04402298831082558) | > loss_mel_ce: 3.778813123703003 (nan) | > loss: 0.015145736746490002 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13620935112593494) | > loader_time: 0.0045 (0.004643118718221927)  --> TIME: 2025-02-24 08:54:17 -- STEP: 29222/36938 -- GLOBAL_STEP: 250850 | > loss_text_ce: 0.03776898980140686 (0.04402095361530921) | > loss_mel_ce: 3.202000856399536 (nan) | > loss: 0.012856231071054935 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13620573052569604) | > loader_time: 0.0045 (0.004643093518967337)  --> TIME: 2025-02-24 08:54:27 -- STEP: 29272/36938 -- GLOBAL_STEP: 250900 | > loss_text_ce: 0.0335313156247139 (0.04402010594226712) | > loss_mel_ce: 1.7207050323486328 (nan) | > loss: 0.0069612557999789715 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13620421696830018) | > loader_time: 0.0046 (0.004643120362348766)  --> TIME: 2025-02-24 08:54:36 -- STEP: 29322/36938 -- GLOBAL_STEP: 250950 | > loss_text_ce: 0.03524719178676605 (0.04402183085927333) | > loss_mel_ce: 2.0311646461486816 (nan) | > loss: 0.008200046606361866 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13619837911512347) | > loader_time: 0.0046 (0.004643254500929569)  --> TIME: 2025-02-24 08:54:45 -- STEP: 29372/36938 -- GLOBAL_STEP: 251000 | > loss_text_ce: 0.04123447462916374 (0.04401962515866558) | > loss_mel_ce: 3.306727409362793 (nan) | > loss: 0.013285564258694649 (nan) | > current_lr: 5e-06 | > step_time: 0.1438 (0.13619179971111325) | > loader_time: 0.0046 (0.00464321238037091)  --> TIME: 2025-02-24 08:54:54 -- STEP: 29422/36938 -- GLOBAL_STEP: 251050 | > loss_text_ce: 0.03915882110595703 (0.044016259336328016) | > loss_mel_ce: 0.8650789856910706 (nan) | > loss: 0.0035882454831153154 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.1361816663744013) | > loader_time: 0.0046 (0.004643159090609595)  --> TIME: 2025-02-24 08:55:03 -- STEP: 29472/36938 -- GLOBAL_STEP: 251100 | > loss_text_ce: 0.019163070246577263 (0.044013711663568635) | > loss_mel_ce: 2.9781527519226074 (nan) | > loss: 0.011894111521542072 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13618265431329837) | > loader_time: 0.0045 (0.00464322523140103)  --> TIME: 2025-02-24 08:55:12 -- STEP: 29522/36938 -- GLOBAL_STEP: 251150 | > loss_text_ce: 0.044598810374736786 (0.044010782593055915) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13617785147685096) | > loader_time: 0.0048 (0.004643265951148975)  --> TIME: 2025-02-24 08:55:21 -- STEP: 29572/36938 -- GLOBAL_STEP: 251200 | > loss_text_ce: 0.04408637061715126 (0.044008391488580705) | > loss_mel_ce: 3.266637086868286 (nan) | > loss: 0.013137792237102985 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.13617660315453134) | > loader_time: 0.0048 (0.004643365766978547)  --> TIME: 2025-02-24 08:55:31 -- STEP: 29622/36938 -- GLOBAL_STEP: 251250 | > loss_text_ce: 0.03078039363026619 (0.04400732472829294) | > loss_mel_ce: 2.928997278213501 (nan) | > loss: 0.011745150201022625 (nan) | > current_lr: 5e-06 | > step_time: 0.1822 (0.13617965412223623) | > loader_time: 0.0049 (0.004643374818703089)  --> TIME: 2025-02-24 08:55:40 -- STEP: 29672/36938 -- GLOBAL_STEP: 251300 | > loss_text_ce: 0.04530150443315506 (0.0440083133808732) | > loss_mel_ce: 3.4622061252593994 (nan) | > loss: 0.013918682001531124 (nan) | > current_lr: 5e-06 | > step_time: 0.1871 (0.13618902725876442) | > loader_time: 0.0049 (0.004643380304461493)  --> TIME: 2025-02-24 08:55:50 -- STEP: 29722/36938 -- GLOBAL_STEP: 251350 | > loss_text_ce: 0.043429601937532425 (0.04400706594099439) | > loss_mel_ce: 3.872814416885376 (nan) | > loss: 0.015540651977062225 (nan) | > current_lr: 5e-06 | > step_time: 0.1881 (0.1361961107171889) | > loader_time: 0.0049 (0.00464338354977436)  --> TIME: 2025-02-24 08:55:59 -- STEP: 29772/36938 -- GLOBAL_STEP: 251400 | > loss_text_ce: 0.06087502837181091 (0.044007944833016684) | > loss_mel_ce: 3.4952690601348877 (nan) | > loss: 0.014111683703958988 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13619373867496723) | > loader_time: 0.0046 (0.0046433953128641806)  --> TIME: 2025-02-24 08:56:08 -- STEP: 29822/36938 -- GLOBAL_STEP: 251450 | > loss_text_ce: 0.038454558700323105 (0.04400706693736036) | > loss_mel_ce: 3.477849245071411 (nan) | > loss: 0.013953587040305138 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13619054881198175) | > loader_time: 0.0046 (0.004643341735625952)  --> TIME: 2025-02-24 08:56:18 -- STEP: 29872/36938 -- GLOBAL_STEP: 251500 | > loss_text_ce: 0.032970260828733444 (0.04400609131298457) | > loss_mel_ce: 3.7057316303253174 (nan) | > loss: 0.01483611948788166 (nan) | > current_lr: 5e-06 | > step_time: 0.1271 (0.13618632653816942) | > loader_time: 0.0057 (0.0046432952016959)  --> TIME: 2025-02-24 08:56:27 -- STEP: 29922/36938 -- GLOBAL_STEP: 251550 | > loss_text_ce: 0.04133134335279465 (0.0440086311851562) | > loss_mel_ce: 1.5305079221725464 (nan) | > loss: 0.0062374575063586235 (nan) | > current_lr: 5e-06 | > step_time: 0.1075 (0.13618403954840932) | > loader_time: 0.0044 (0.004643370749661791)  --> TIME: 2025-02-24 08:56:36 -- STEP: 29972/36938 -- GLOBAL_STEP: 251600 | > loss_text_ce: 0.035448770970106125 (0.044005446279524005) | > loss_mel_ce: 3.2010862827301025 (nan) | > loss: 0.012843393720686436 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13618657615368407) | > loader_time: 0.0047 (0.004643382901074983)  --> TIME: 2025-02-24 08:56:46 -- STEP: 30022/36938 -- GLOBAL_STEP: 251650 | > loss_text_ce: 0.045024141669273376 (0.0440055315437869) | > loss_mel_ce: 1.9096598625183105 (nan) | > loss: 0.0077566830441355705 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13619036874942042) | > loader_time: 0.0045 (0.004643320608453503)  --> TIME: 2025-02-24 08:56:55 -- STEP: 30072/36938 -- GLOBAL_STEP: 251700 | > loss_text_ce: 0.030123654752969742 (0.04400384059165532) | > loss_mel_ce: 3.1078178882598877 (nan) | > loss: 0.012452149763703346 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.1361965045281379) | > loader_time: 0.0046 (0.004643353733429076)  --> TIME: 2025-02-24 08:57:04 -- STEP: 30122/36938 -- GLOBAL_STEP: 251750 | > loss_text_ce: 0.058601900935173035 (0.04400429166736521) | > loss_mel_ce: 3.6119353771209717 (nan) | > loss: 0.014565625227987766 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1361956197057826) | > loader_time: 0.0045 (0.004643266589333973)  --> TIME: 2025-02-24 08:57:14 -- STEP: 30172/36938 -- GLOBAL_STEP: 251800 | > loss_text_ce: 0.044979438185691833 (0.04400653078611651) | > loss_mel_ce: 3.328575611114502 (nan) | > loss: 0.013387123122811317 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.1361973038580098) | > loader_time: 0.0046 (0.004643189714265614)  --> TIME: 2025-02-24 08:57:23 -- STEP: 30222/36938 -- GLOBAL_STEP: 251850 | > loss_text_ce: 0.03237432241439819 (0.04400249546784218) | > loss_mel_ce: 3.4531009197235107 (nan) | > loss: 0.013831252232193947 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13620134561526237) | > loader_time: 0.0045 (0.004643161570905131)  --> TIME: 2025-02-24 08:57:33 -- STEP: 30272/36938 -- GLOBAL_STEP: 251900 | > loss_text_ce: 0.06160687655210495 (0.044004915354581814) | > loss_mel_ce: 3.463427782058716 (nan) | > loss: 0.013988233171403408 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13620325396069802) | > loader_time: 0.0046 (0.004643149162006663)  --> TIME: 2025-02-24 08:57:42 -- STEP: 30322/36938 -- GLOBAL_STEP: 251950 | > loss_text_ce: 0.04673343524336815 (0.044004800343035125) | > loss_mel_ce: 3.1409499645233154 (nan) | > loss: 0.01264953799545765 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13620477219337512) | > loader_time: 0.0045 (0.004643113677132577)  --> TIME: 2025-02-24 08:57:51 -- STEP: 30372/36938 -- GLOBAL_STEP: 252000 | > loss_text_ce: 0.04335477203130722 (0.04400016691132262) | > loss_mel_ce: 3.1149702072143555 (nan) | > loss: 0.012533036060631275 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13620757376050288) | > loader_time: 0.0046 (0.004643146203281079)  --> TIME: 2025-02-24 08:58:01 -- STEP: 30422/36938 -- GLOBAL_STEP: 252050 | > loss_text_ce: 0.037247613072395325 (0.043997005723377516) | > loss_mel_ce: 3.457791805267334 (nan) | > loss: 0.013869204558432102 (nan) | > current_lr: 5e-06 | > step_time: 0.1764 (0.13621085471917627) | > loader_time: 0.005 (0.004643160370035539)  --> TIME: 2025-02-24 08:58:10 -- STEP: 30472/36938 -- GLOBAL_STEP: 252100 | > loss_text_ce: 0.04549838975071907 (0.043997241596688025) | > loss_mel_ce: 3.542463779449463 (nan) | > loss: 0.014237945899367332 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13620315113071418) | > loader_time: 0.0046 (0.0046431232731809725)  --> TIME: 2025-02-24 08:58:19 -- STEP: 30522/36938 -- GLOBAL_STEP: 252150 | > loss_text_ce: 0.05121924728155136 (0.04399274812466167) | > loss_mel_ce: 3.3264169692993164 (nan) | > loss: 0.01340331882238388 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13620385217694783) | > loader_time: 0.0046 (0.004643110286579399)  --> TIME: 2025-02-24 08:58:28 -- STEP: 30572/36938 -- GLOBAL_STEP: 252200 | > loss_text_ce: 0.045052528381347656 (0.043990774180167794) | > loss_mel_ce: 3.362574577331543 (nan) | > loss: 0.013522330671548843 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13620122517339106) | > loader_time: 0.0046 (0.004643049154951655)  --> TIME: 2025-02-24 08:58:37 -- STEP: 30622/36938 -- GLOBAL_STEP: 252250 | > loss_text_ce: 0.04275274649262428 (0.043989440941830354) | > loss_mel_ce: 3.6319518089294434 (nan) | > loss: 0.014582161791622639 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13619817650206845) | > loader_time: 0.0046 (0.004642957663459754)  --> TIME: 2025-02-24 08:58:47 -- STEP: 30672/36938 -- GLOBAL_STEP: 252300 | > loss_text_ce: 0.03656306117773056 (0.043987936181073974) | > loss_mel_ce: 2.886496067047119 (nan) | > loss: 0.01159944199025631 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13620134480744558) | > loader_time: 0.0045 (0.004642976965827109)  --> TIME: 2025-02-24 08:58:56 -- STEP: 30722/36938 -- GLOBAL_STEP: 252350 | > loss_text_ce: 0.04006271809339523 (0.043989268529553066) | > loss_mel_ce: 3.2074620723724365 (nan) | > loss: 0.01288700383156538 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13619640663370014) | > loader_time: 0.0046 (0.004642938467124702)  --> TIME: 2025-02-24 08:59:05 -- STEP: 30772/36938 -- GLOBAL_STEP: 252400 | > loss_text_ce: 0.04455423355102539 (0.04398631154444268) | > loss_mel_ce: 3.522603750228882 (nan) | > loss: 0.014155389741063118 (nan) | > current_lr: 5e-06 | > step_time: 0.1944 (0.13619651218609216) | > loader_time: 0.0049 (0.0046429483165019315)  --> TIME: 2025-02-24 08:59:14 -- STEP: 30822/36938 -- GLOBAL_STEP: 252450 | > loss_text_ce: 0.03650146722793579 (0.04398482009949386) | > loss_mel_ce: 2.319991111755371 (nan) | > loss: 0.009351161308586597 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.1361867958686688) | > loader_time: 0.0046 (0.004642955596732718)  --> TIME: 2025-02-24 08:59:23 -- STEP: 30872/36938 -- GLOBAL_STEP: 252500 | > loss_text_ce: 0.06040360778570175 (0.043983868732967245) | > loss_mel_ce: 3.0599234104156494 (nan) | > loss: 0.012382250279188156 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13618934268808117) | > loader_time: 0.0049 (0.004642982044563464)  --> TIME: 2025-02-24 08:59:33 -- STEP: 30922/36938 -- GLOBAL_STEP: 252550 | > loss_text_ce: 0.040407247841358185 (0.04398391135153168) | > loss_mel_ce: 3.3072750568389893 (nan) | > loss: 0.013284454122185707 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.13618600027703556) | > loader_time: 0.0047 (0.00464297806674607)  --> TIME: 2025-02-24 08:59:42 -- STEP: 30972/36938 -- GLOBAL_STEP: 252600 | > loss_text_ce: 0.04492739215493202 (0.04398375566471774) | > loss_mel_ce: 2.2107014656066895 (nan) | > loss: 0.008950908668339252 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.1361835371781335) | > loader_time: 0.0046 (0.004642879571869085)  --> TIME: 2025-02-24 08:59:51 -- STEP: 31022/36938 -- GLOBAL_STEP: 252650 | > loss_text_ce: 0.04508465155959129 (0.0439827212673615) | > loss_mel_ce: 3.5170352458953857 (nan) | > loss: 0.014135397039353848 (nan) | > current_lr: 5e-06 | > step_time: 0.1831 (0.13619567407844535) | > loader_time: 0.0048 (0.004642858579644334)  --> TIME: 2025-02-24 09:00:01 -- STEP: 31072/36938 -- GLOBAL_STEP: 252700 | > loss_text_ce: 0.028593262657523155 (0.043979418896725554) | > loss_mel_ce: 2.3832104206085205 (nan) | > loss: 0.009570649825036526 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13620520638633865) | > loader_time: 0.0046 (0.004642837018112182)  --> TIME: 2025-02-24 09:00:11 -- STEP: 31122/36938 -- GLOBAL_STEP: 252750 | > loss_text_ce: 0.03731124475598335 (0.043979394457266556) | > loss_mel_ce: 3.2005984783172607 (nan) | > loss: 0.012848849408328533 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.136208946137508) | > loader_time: 0.0048 (0.004642842338566408)  --> TIME: 2025-02-24 09:00:20 -- STEP: 31172/36938 -- GLOBAL_STEP: 252800 | > loss_text_ce: 0.04474808648228645 (0.043977013845972725) | > loss_mel_ce: 2.5814208984375 (nan) | > loss: 0.010421305894851685 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13621558610578904) | > loader_time: 0.0045 (0.004642804588629668)  --> TIME: 2025-02-24 09:00:29 -- STEP: 31222/36938 -- GLOBAL_STEP: 252850 | > loss_text_ce: 0.03799879550933838 (0.04397373565508841) | > loss_mel_ce: 2.516080379486084 (nan) | > loss: 0.010135235264897346 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.1362166700475568) | > loader_time: 0.0047 (0.00464283564754927)  --> TIME: 2025-02-24 09:00:39 -- STEP: 31272/36938 -- GLOBAL_STEP: 252900 | > loss_text_ce: 0.044871073216199875 (0.04397331043567625) | > loss_mel_ce: 3.148501396179199 (nan) | > loss: 0.012672114185988903 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13621812416910092) | > loader_time: 0.0046 (0.004642794072155315)  --> TIME: 2025-02-24 09:00:48 -- STEP: 31322/36938 -- GLOBAL_STEP: 252950 | > loss_text_ce: 0.0502038337290287 (0.04397224648274891) | > loss_mel_ce: 3.1485705375671387 (nan) | > loss: 0.012693549506366253 (nan) | > current_lr: 5e-06 | > step_time: 0.1355 (0.13621917235867306) | > loader_time: 0.0047 (0.004642726726348064)  --> TIME: 2025-02-24 09:00:58 -- STEP: 31372/36938 -- GLOBAL_STEP: 253000 | > loss_text_ce: 0.04457946494221687 (0.04397133811386217) | > loss_mel_ce: 3.6769118309020996 (nan) | > loss: 0.014767823740839958 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.136228638576577) | > loader_time: 0.0047 (0.004642690123305474)  --> TIME: 2025-02-24 09:01:07 -- STEP: 31422/36938 -- GLOBAL_STEP: 253050 | > loss_text_ce: 0.04479680582880974 (0.043971559406221605) | > loss_mel_ce: 4.153308868408203 (nan) | > loss: 0.016659151762723923 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13623010392146873) | > loader_time: 0.0045 (0.004642698919740459)  --> TIME: 2025-02-24 09:01:17 -- STEP: 31472/36938 -- GLOBAL_STEP: 253100 | > loss_text_ce: 0.040819596499204636 (0.04397167990093138) | > loss_mel_ce: 3.625648260116577 (nan) | > loss: 0.014549477025866508 (nan) | > current_lr: 5e-06 | > step_time: 0.1904 (0.13623266114489255) | > loader_time: 0.0048 (0.004642623675069516)  --> TIME: 2025-02-24 09:01:26 -- STEP: 31522/36938 -- GLOBAL_STEP: 253150 | > loss_text_ce: 0.05574687942862511 (0.04397146776770557) | > loss_mel_ce: 3.7363553047180176 (nan) | > loss: 0.01504802517592907 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13622998307860254) | > loader_time: 0.0047 (0.004642567706587607)  --> TIME: 2025-02-24 09:01:35 -- STEP: 31572/36938 -- GLOBAL_STEP: 253200 | > loss_text_ce: 0.04388180002570152 (0.043970491010198606) | > loss_mel_ce: 2.9854838848114014 (nan) | > loss: 0.012021293863654137 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.13623213728737174) | > loader_time: 0.0046 (0.004642490672775189)  --> TIME: 2025-02-24 09:01:44 -- STEP: 31622/36938 -- GLOBAL_STEP: 253250 | > loss_text_ce: 0.04441557824611664 (0.04396848610892856) | > loss_mel_ce: 3.1798510551452637 (nan) | > loss: 0.012794708833098412 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.13623083041911743) | > loader_time: 0.0046 (0.004642433757069017)  --> TIME: 2025-02-24 09:01:54 -- STEP: 31672/36938 -- GLOBAL_STEP: 253300 | > loss_text_ce: 0.04055298864841461 (0.04396663173424678) | > loss_mel_ce: 3.467665672302246 (nan) | > loss: 0.013921503908932209 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.1362296659240644) | > loader_time: 0.0047 (0.004642370893486086)  --> TIME: 2025-02-24 09:02:03 -- STEP: 31722/36938 -- GLOBAL_STEP: 253350 | > loss_text_ce: 0.04734443128108978 (0.04396638868713149) | > loss_mel_ce: 3.167886972427368 (nan) | > loss: 0.01275885570794344 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13623310916699363) | > loader_time: 0.0048 (0.00464232695011748)  --> TIME: 2025-02-24 09:02:13 -- STEP: 31772/36938 -- GLOBAL_STEP: 253400 | > loss_text_ce: 0.04742338880896568 (0.043965090062236985) | > loss_mel_ce: 3.537731885910034 (nan) | > loss: 0.014226807281374931 (nan) | > current_lr: 5e-06 | > step_time: 0.1507 (0.13623627478869607) | > loader_time: 0.0048 (0.0046423141592833155)  --> TIME: 2025-02-24 09:02:22 -- STEP: 31822/36938 -- GLOBAL_STEP: 253450 | > loss_text_ce: 0.045757077634334564 (0.04396600291400624) | > loss_mel_ce: 3.6222851276397705 (nan) | > loss: 0.014555723406374454 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13623899847398335) | > loader_time: 0.0045 (0.004642283127538787)  --> TIME: 2025-02-24 09:02:31 -- STEP: 31872/36938 -- GLOBAL_STEP: 253500 | > loss_text_ce: 0.053328316658735275 (0.04396441906695123) | > loss_mel_ce: 3.09755802154541 (nan) | > loss: 0.012503517791628838 (nan) | > current_lr: 5e-06 | > step_time: 0.1263 (0.13624472190517534) | > loader_time: 0.0046 (0.004642297457679192)  --> TIME: 2025-02-24 09:02:41 -- STEP: 31922/36938 -- GLOBAL_STEP: 253550 | > loss_text_ce: 0.04722413420677185 (0.04396393690215636) | > loss_mel_ce: 4.167834281921387 (nan) | > loss: 0.016726423054933548 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.1362442507360541) | > loader_time: 0.0045 (0.004642290061043419)  --> TIME: 2025-02-24 09:02:50 -- STEP: 31972/36938 -- GLOBAL_STEP: 253600 | > loss_text_ce: 0.05017827823758125 (0.04396225284000488) | > loss_mel_ce: 3.3009626865386963 (nan) | > loss: 0.013298179022967815 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13624636669503787) | > loader_time: 0.0047 (0.004642299622629112)  --> TIME: 2025-02-24 09:03:00 -- STEP: 32022/36938 -- GLOBAL_STEP: 253650 | > loss_text_ce: 0.04564405977725983 (0.043960298017141915) | > loss_mel_ce: 3.014638662338257 (nan) | > loss: 0.012143979780375957 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13625389064663532) | > loader_time: 0.0046 (0.004642299847528075)  --> TIME: 2025-02-24 09:03:09 -- STEP: 32072/36938 -- GLOBAL_STEP: 253700 | > loss_text_ce: 0.04207964986562729 (0.04395744665179101) | > loss_mel_ce: 3.7456424236297607 (nan) | > loss: 0.01503064390271902 (nan) | > current_lr: 5e-06 | > step_time: 0.1825 (0.13625323685454357) | > loader_time: 0.0048 (0.0046422484956438114)  --> TIME: 2025-02-24 09:03:18 -- STEP: 32122/36938 -- GLOBAL_STEP: 253750 | > loss_text_ce: 0.03952663019299507 (0.043954617279680495) | > loss_mel_ce: 3.797670602798462 (nan) | > loss: 0.01522697415202856 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13625241206210548) | > loader_time: 0.0046 (0.004642208622606491)  --> TIME: 2025-02-24 09:03:27 -- STEP: 32172/36938 -- GLOBAL_STEP: 253800 | > loss_text_ce: 0.039731208235025406 (0.043952870121814536) | > loss_mel_ce: 2.633528232574463 (nan) | > loss: 0.010608172975480556 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.13624727215382174) | > loader_time: 0.0049 (0.004642164597504754)  --> TIME: 2025-02-24 09:03:37 -- STEP: 32222/36938 -- GLOBAL_STEP: 253850 | > loss_text_ce: 0.041141100227832794 (0.04394965408233211) | > loss_mel_ce: 3.565828800201416 (nan) | > loss: 0.014313372783362865 (nan) | > current_lr: 5e-06 | > step_time: 0.1651 (0.136251540669708) | > loader_time: 0.0048 (0.004642134508631687)  --> TIME: 2025-02-24 09:03:46 -- STEP: 32272/36938 -- GLOBAL_STEP: 253900 | > loss_text_ce: 0.03644176945090294 (0.04394765704521926) | > loss_mel_ce: 3.586258888244629 (nan) | > loss: 0.014375797472894192 (nan) | > current_lr: 5e-06 | > step_time: 0.1822 (0.13625611915970212) | > loader_time: 0.0048 (0.004642142161143301)  --> TIME: 2025-02-24 09:03:56 -- STEP: 32322/36938 -- GLOBAL_STEP: 253950 | > loss_text_ce: 0.04522886127233505 (0.04394780768925875) | > loss_mel_ce: 2.7224860191345215 (nan) | > loss: 0.0109829967841506 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.1362527240643771) | > loader_time: 0.0046 (0.004642093146941219)  --> TIME: 2025-02-24 09:04:05 -- STEP: 32372/36938 -- GLOBAL_STEP: 254000 | > loss_text_ce: 0.0467846654355526 (0.043948533570466776) | > loss_mel_ce: 3.1860570907592773 (nan) | > loss: 0.012828737497329712 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.13626118437934276) | > loader_time: 0.0046 (0.00464209310648747)  --> TIME: 2025-02-24 09:04:14 -- STEP: 32422/36938 -- GLOBAL_STEP: 254050 | > loss_text_ce: 0.04456733912229538 (0.04394629931709968) | > loss_mel_ce: 3.5745599269866943 (nan) | > loss: 0.014361617155373096 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13625934365779482) | > loader_time: 0.0046 (0.0046419925497326806)  --> TIME: 2025-02-24 09:04:24 -- STEP: 32472/36938 -- GLOBAL_STEP: 254100 | > loss_text_ce: 0.03973126783967018 (0.043945716210066106) | > loss_mel_ce: 2.8990116119384766 (nan) | > loss: 0.011661678552627563 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1362602533260926) | > loader_time: 0.0046 (0.004641924718824842)  --> TIME: 2025-02-24 09:04:33 -- STEP: 32522/36938 -- GLOBAL_STEP: 254150 | > loss_text_ce: 0.046056218445301056 (0.043944552977116094) | > loss_mel_ce: 3.1344799995422363 (nan) | > loss: 0.012621176429092884 (nan) | > current_lr: 5e-06 | > step_time: 0.1619 (0.13626549123112258) | > loader_time: 0.0048 (0.004641900393335458)  --> TIME: 2025-02-24 09:04:43 -- STEP: 32572/36938 -- GLOBAL_STEP: 254200 | > loss_text_ce: 0.04880700632929802 (0.04394697091481764) | > loss_mel_ce: 1.3483871221542358 (nan) | > loss: 0.005544421728700399 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13627078922436428) | > loader_time: 0.0045 (0.004641893160924342)  --> TIME: 2025-02-24 09:04:52 -- STEP: 32622/36938 -- GLOBAL_STEP: 254250 | > loss_text_ce: 0.04730391502380371 (0.04394493681622951) | > loss_mel_ce: 2.3264617919921875 (nan) | > loss: 0.009419705718755722 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13627315460667244) | > loader_time: 0.0046 (0.0046418403674367515)  --> TIME: 2025-02-24 09:05:02 -- STEP: 32672/36938 -- GLOBAL_STEP: 254300 | > loss_text_ce: 0.040097326040267944 (0.04394340507887808) | > loss_mel_ce: 2.2649285793304443 (nan) | > loss: 0.009146928787231445 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13628261912902384) | > loader_time: 0.0048 (0.004641838867972561)  --> TIME: 2025-02-24 09:05:11 -- STEP: 32722/36938 -- GLOBAL_STEP: 254350 | > loss_text_ce: 0.04856876656413078 (0.04394391849143474) | > loss_mel_ce: 3.142674207687378 (nan) | > loss: 0.012663663364946842 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13627648948858093) | > loader_time: 0.0045 (0.0046417882641969985)  --> TIME: 2025-02-24 09:05:20 -- STEP: 32772/36938 -- GLOBAL_STEP: 254400 | > loss_text_ce: 0.04719999060034752 (0.04394412538987318) | > loss_mel_ce: 3.048527240753174 (nan) | > loss: 0.01228463277220726 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13627696426508734) | > loader_time: 0.0046 (0.004641735515911078)  --> TIME: 2025-02-24 09:05:29 -- STEP: 32822/36938 -- GLOBAL_STEP: 254450 | > loss_text_ce: 0.03711841255426407 (0.04394304281339856) | > loss_mel_ce: 3.1795551776885986 (nan) | > loss: 0.012764578685164452 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1362763576944573) | > loader_time: 0.0045 (0.0046416894441315875)  --> TIME: 2025-02-24 09:05:39 -- STEP: 32872/36938 -- GLOBAL_STEP: 254500 | > loss_text_ce: 0.06191863864660263 (0.043939074425267385) | > loss_mel_ce: 3.528188705444336 (nan) | > loss: 0.014246459119021893 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.1362738948896527) | > loader_time: 0.0046 (0.00464168357773689)  --> TIME: 2025-02-24 09:05:48 -- STEP: 32922/36938 -- GLOBAL_STEP: 254550 | > loss_text_ce: 0.040468111634254456 (0.04394033468728896) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13627294275698545) | > loader_time: 0.0046 (0.0046417207461820955)  --> TIME: 2025-02-24 09:05:57 -- STEP: 32972/36938 -- GLOBAL_STEP: 254600 | > loss_text_ce: 0.04966186359524727 (0.0439376629832008) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13627223023855786) | > loader_time: 0.0046 (0.004641780998757996)  --> TIME: 2025-02-24 09:06:07 -- STEP: 33022/36938 -- GLOBAL_STEP: 254650 | > loss_text_ce: 0.06143493577837944 (0.04393788586447835) | > loss_mel_ce: 3.3280282020568848 (nan) | > loss: 0.013450251892209053 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13627636549611608) | > loader_time: 0.0045 (0.004641829596304528)  --> TIME: 2025-02-24 09:06:16 -- STEP: 33072/36938 -- GLOBAL_STEP: 254700 | > loss_text_ce: 0.04082903638482094 (0.04393871529820222) | > loss_mel_ce: 3.5333175659179688 (nan) | > loss: 0.014183121733367443 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.1362754059662612) | > loader_time: 0.0046 (0.004641867089110046)  --> TIME: 2025-02-24 09:06:25 -- STEP: 33122/36938 -- GLOBAL_STEP: 254750 | > loss_text_ce: 0.038972437381744385 (0.04393643974481449) | > loss_mel_ce: 3.505124092102051 (nan) | > loss: 0.014063876122236252 (nan) | > current_lr: 5e-06 | > step_time: 0.1705 (0.13627958143688176) | > loader_time: 0.0049 (0.004641940423698126)  --> TIME: 2025-02-24 09:06:35 -- STEP: 33172/36938 -- GLOBAL_STEP: 254800 | > loss_text_ce: 0.04572560265660286 (0.043937048705639845) | > loss_mel_ce: 3.404299020767212 (nan) | > loss: 0.013690575025975704 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.1362843801557525) | > loader_time: 0.0045 (0.0046419733096474125)  --> TIME: 2025-02-24 09:06:44 -- STEP: 33222/36938 -- GLOBAL_STEP: 254850 | > loss_text_ce: 0.05256350338459015 (0.04393634445367925) | > loss_mel_ce: 3.385211706161499 (nan) | > loss: 0.013641965575516224 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13628078009375963) | > loader_time: 0.0046 (0.004641993760158364)  --> TIME: 2025-02-24 09:06:53 -- STEP: 33272/36938 -- GLOBAL_STEP: 254900 | > loss_text_ce: 0.04837071895599365 (0.043934597414747735) | > loss_mel_ce: 0.0269622802734375 (nan) | > loss: 0.000298940489301458 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13627706778023704) | > loader_time: 0.0046 (0.004641997639334338)  --> TIME: 2025-02-24 09:07:02 -- STEP: 33322/36938 -- GLOBAL_STEP: 254950 | > loss_text_ce: 0.03963296860456467 (0.04393302298120014) | > loss_mel_ce: 0.7338206171989441 (nan) | > loss: 0.0030692603904753923 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.1362775604476323) | > loader_time: 0.0047 (0.004641979884488835)  --> TIME: 2025-02-24 09:07:11 -- STEP: 33372/36938 -- GLOBAL_STEP: 255000 | > loss_text_ce: 0.057910941541194916 (0.04393140566940476) | > loss_mel_ce: 2.852987289428711 (nan) | > loss: 0.01155118364840746 (nan) | > current_lr: 5e-06 | > step_time: 0.1493 (0.13627085948782036) | > loader_time: 0.0046 (0.0046418832672395155)  --> TIME: 2025-02-24 09:07:21 -- STEP: 33422/36938 -- GLOBAL_STEP: 255050 | > loss_text_ce: 0.03660847246646881 (0.04393060409563279) | > loss_mel_ce: 2.138115167617798 (nan) | > loss: 0.008629855699837208 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13626715208282697) | > loader_time: 0.0045 (0.0046418503066822875)  --> TIME: 2025-02-24 09:07:30 -- STEP: 33472/36938 -- GLOBAL_STEP: 255100 | > loss_text_ce: 0.04043113440275192 (0.043927504239289934) | > loss_mel_ce: 3.614037275314331 (nan) | > loss: 0.014501859433948994 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.1362598348359081) | > loader_time: 0.0046 (0.004641802351116892)  --> TIME: 2025-02-24 09:07:39 -- STEP: 33522/36938 -- GLOBAL_STEP: 255150 | > loss_text_ce: 0.03422790765762329 (0.04392539150604465) | > loss_mel_ce: 1.9839887619018555 (nan) | > loss: 0.008008796721696854 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1362623079004742) | > loader_time: 0.0045 (0.004641783279419841)  --> TIME: 2025-02-24 09:07:48 -- STEP: 33572/36938 -- GLOBAL_STEP: 255200 | > loss_text_ce: 0.0499204657971859 (0.043923724962615145) | > loss_mel_ce: 3.439495086669922 (nan) | > loss: 0.013846888206899166 (nan) | > current_lr: 5e-06 | > step_time: 0.1799 (0.1362634098431816) | > loader_time: 0.0051 (0.004641790228379734)  --> TIME: 2025-02-24 09:07:57 -- STEP: 33622/36938 -- GLOBAL_STEP: 255250 | > loss_text_ce: 0.037831537425518036 (0.04392016455984812) | > loss_mel_ce: 2.6160786151885986 (nan) | > loss: 0.010531390085816383 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1362607831667343) | > loader_time: 0.0046 (0.0046418160900369735)  --> TIME: 2025-02-24 09:08:07 -- STEP: 33672/36938 -- GLOBAL_STEP: 255300 | > loss_text_ce: 0.043388914316892624 (0.04391939727461452) | > loss_mel_ce: 0.1495884507894516 (nan) | > loss: 0.0007657832466065884 (nan) | > current_lr: 5e-06 | > step_time: 0.1116 (0.1362640896144294) | > loader_time: 0.0045 (0.004641811350344125)  --> TIME: 2025-02-24 09:08:16 -- STEP: 33722/36938 -- GLOBAL_STEP: 255350 | > loss_text_ce: 0.03935224562883377 (0.04391684897121211) | > loss_mel_ce: 3.514751672744751 (nan) | > loss: 0.014103587716817856 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13626931486559096) | > loader_time: 0.0046 (0.004641796033667045)  --> TIME: 2025-02-24 09:08:25 -- STEP: 33772/36938 -- GLOBAL_STEP: 255400 | > loss_text_ce: 0.04270414635539055 (0.04391546457391242) | > loss_mel_ce: 2.450395107269287 (nan) | > loss: 0.009893251582980156 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13626337025976887) | > loader_time: 0.0046 (0.004641713321484267)  --> TIME: 2025-02-24 09:08:35 -- STEP: 33822/36938 -- GLOBAL_STEP: 255450 | > loss_text_ce: 0.05196436122059822 (0.043913717166104045) | > loss_mel_ce: 3.486525774002075 (nan) | > loss: 0.014041627757251263 (nan) | > current_lr: 5e-06 | > step_time: 0.1628 (0.1362643069669984) | > loader_time: 0.0047 (0.0046416748480091466)  --> TIME: 2025-02-24 09:08:44 -- STEP: 33872/36938 -- GLOBAL_STEP: 255500 | > loss_text_ce: 0.04442916437983513 (0.04391238086694756) | > loss_mel_ce: 3.0282907485961914 (nan) | > loss: 0.012193333357572556 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13625998517703414) | > loader_time: 0.0047 (0.004641628147128833)  --> TIME: 2025-02-24 09:08:53 -- STEP: 33922/36938 -- GLOBAL_STEP: 255550 | > loss_text_ce: 0.039223991334438324 (0.04391211784460585) | > loss_mel_ce: 3.2282283306121826 (nan) | > loss: 0.012966081500053406 (nan) | > current_lr: 5e-06 | > step_time: 0.1512 (0.13625395202417462) | > loader_time: 0.0046 (0.004641623065744478)  --> TIME: 2025-02-24 09:09:02 -- STEP: 33972/36938 -- GLOBAL_STEP: 255600 | > loss_text_ce: 0.05066678300499916 (0.04391089901510086) | > loss_mel_ce: 3.0536203384399414 (nan) | > loss: 0.01231860090047121 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.1362581809375815) | > loader_time: 0.0045 (0.004641612595387809)  --> TIME: 2025-02-24 09:09:12 -- STEP: 34022/36938 -- GLOBAL_STEP: 255650 | > loss_text_ce: 0.0510869026184082 (0.04391247985258084) | > loss_mel_ce: 0.1496785432100296 (nan) | > loss: 0.0007966883131302893 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13625844837769777) | > loader_time: 0.0048 (0.004641600508978868)  --> TIME: 2025-02-24 09:09:21 -- STEP: 34072/36938 -- GLOBAL_STEP: 255700 | > loss_text_ce: 0.047347910702228546 (0.04391109649500623) | > loss_mel_ce: 3.2784459590911865 (nan) | > loss: 0.013197596184909344 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.13625935368007241) | > loader_time: 0.0046 (0.004641503613441291)  --> TIME: 2025-02-24 09:09:30 -- STEP: 34122/36938 -- GLOBAL_STEP: 255750 | > loss_text_ce: 0.025435740128159523 (0.043906954679591735) | > loss_mel_ce: 0.090972900390625 (nan) | > loss: 0.00046193908201530576 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13626445101361312) | > loader_time: 0.0045 (0.004641444264818791)  --> TIME: 2025-02-24 09:09:39 -- STEP: 34172/36938 -- GLOBAL_STEP: 255800 | > loss_text_ce: 0.041979994624853134 (0.04390515474636124) | > loss_mel_ce: 3.127488613128662 (nan) | > loss: 0.012577257119119167 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13625450292934707) | > loader_time: 0.0046 (0.004641320964121541)  --> TIME: 2025-02-24 09:09:49 -- STEP: 34222/36938 -- GLOBAL_STEP: 255850 | > loss_text_ce: 0.04601692408323288 (0.0439037750285967) | > loss_mel_ce: 3.308751344680786 (nan) | > loss: 0.013312573544681072 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13625903956818095) | > loader_time: 0.0044 (0.004641319239455137)  --> TIME: 2025-02-24 09:09:58 -- STEP: 34272/36938 -- GLOBAL_STEP: 255900 | > loss_text_ce: 0.03517227992415428 (0.04390228504242593) | > loss_mel_ce: 2.292403221130371 (nan) | > loss: 0.009236411191523075 (nan) | > current_lr: 5e-06 | > step_time: 0.1506 (0.13625454888039965) | > loader_time: 0.0046 (0.004641254040317803)  --> TIME: 2025-02-24 09:10:07 -- STEP: 34322/36938 -- GLOBAL_STEP: 255950 | > loss_text_ce: 0.04751449450850487 (0.043899964748181176) | > loss_mel_ce: 2.9032037258148193 (nan) | > loss: 0.011709199286997318 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13624234622150433) | > loader_time: 0.0046 (0.0046411797714277685)  --> TIME: 2025-02-24 09:10:16 -- STEP: 34372/36938 -- GLOBAL_STEP: 256000 | > loss_text_ce: 0.03660212457180023 (0.04389892948621626) | > loss_mel_ce: 3.7836825847625732 (nan) | > loss: 0.015159860253334045 (nan) | > current_lr: 5e-06 | > step_time: 0.1702 (0.13624001357409188) | > loader_time: 0.0049 (0.004641184370661715)  --> TIME: 2025-02-24 09:10:25 -- STEP: 34422/36938 -- GLOBAL_STEP: 256050 | > loss_text_ce: 0.045457709580659866 (0.04389781173059719) | > loss_mel_ce: 3.1759164333343506 (nan) | > loss: 0.01278323121368885 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13623602407189603) | > loader_time: 0.0045 (0.0046411922534736125)  --> TIME: 2025-02-24 09:10:34 -- STEP: 34472/36938 -- GLOBAL_STEP: 256100 | > loss_text_ce: 0.04868914932012558 (0.04389494636286657) | > loss_mel_ce: 3.1756210327148438 (nan) | > loss: 0.01279488205909729 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1362321642386331) | > loader_time: 0.0047 (0.004641222100325362)  --> TIME: 2025-02-24 09:10:43 -- STEP: 34522/36938 -- GLOBAL_STEP: 256150 | > loss_text_ce: 0.05657770484685898 (0.0438945600327001) | > loss_mel_ce: 2.204401969909668 (nan) | > loss: 0.00897214189171791 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13622923509511495) | > loader_time: 0.0045 (0.004641150821851308)  --> TIME: 2025-02-24 09:10:52 -- STEP: 34572/36938 -- GLOBAL_STEP: 256200 | > loss_text_ce: 0.05156758800148964 (0.04389397247177068) | > loss_mel_ce: 3.4414448738098145 (nan) | > loss: 0.013861161656677723 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.13622714620655554) | > loader_time: 0.0048 (0.004641069480973664)  --> TIME: 2025-02-24 09:11:02 -- STEP: 34622/36938 -- GLOBAL_STEP: 256250 | > loss_text_ce: 0.04591335356235504 (0.043889726554258705) | > loss_mel_ce: 2.505204916000366 (nan) | > loss: 0.010123486630618572 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1362222581628389) | > loader_time: 0.0045 (0.004641097791961626)  --> TIME: 2025-02-24 09:11:11 -- STEP: 34672/36938 -- GLOBAL_STEP: 256300 | > loss_text_ce: 0.04124898463487625 (0.043889189158294324) | > loss_mel_ce: 1.8485435247421265 (nan) | > loss: 0.007499177474528551 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13622493858319815) | > loader_time: 0.0045 (0.004641095682614167)  --> TIME: 2025-02-24 09:11:20 -- STEP: 34722/36938 -- GLOBAL_STEP: 256350 | > loss_text_ce: 0.04634466394782066 (0.0438879996629998) | > loss_mel_ce: 3.1015625 (nan) | > loss: 0.012491696514189243 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13621704835618734) | > loader_time: 0.0045 (0.004641149568774705)  --> TIME: 2025-02-24 09:11:29 -- STEP: 34772/36938 -- GLOBAL_STEP: 256400 | > loss_text_ce: 0.031585514545440674 (0.04388502469261859) | > loss_mel_ce: 2.542210817337036 (nan) | > loss: 0.010213477537035942 (nan) | > current_lr: 5e-06 | > step_time: 0.1069 (0.136212501463621) | > loader_time: 0.0045 (0.0046411482755462745)  --> TIME: 2025-02-24 09:11:38 -- STEP: 34822/36938 -- GLOBAL_STEP: 256450 | > loss_text_ce: 0.04603934288024902 (0.04388436803363222) | > loss_mel_ce: 3.4093549251556396 (nan) | > loss: 0.013711882755160332 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13621001821870385) | > loader_time: 0.0046 (0.0046410955461722424)  --> TIME: 2025-02-24 09:11:47 -- STEP: 34872/36938 -- GLOBAL_STEP: 256500 | > loss_text_ce: 0.04049932211637497 (0.04388134433356074) | > loss_mel_ce: 3.919830322265625 (nan) | > loss: 0.015715593472123146 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1362057514941648) | > loader_time: 0.0045 (0.004641072223371898)  --> TIME: 2025-02-24 09:11:56 -- STEP: 34922/36938 -- GLOBAL_STEP: 256550 | > loss_text_ce: 0.03211493045091629 (0.04387988408682966) | > loss_mel_ce: 2.819974660873413 (nan) | > loss: 0.011317816562950611 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13620262866672628) | > loader_time: 0.0047 (0.004641023160638298)  --> TIME: 2025-02-24 09:12:06 -- STEP: 34972/36938 -- GLOBAL_STEP: 256600 | > loss_text_ce: 0.04039284214377403 (0.043877118770701704) | > loss_mel_ce: 2.891848564147949 (nan) | > loss: 0.011635879054665565 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13620150775095718) | > loader_time: 0.0046 (0.004640966220918091)  --> TIME: 2025-02-24 09:12:15 -- STEP: 35022/36938 -- GLOBAL_STEP: 256650 | > loss_text_ce: 0.044293198734521866 (0.043876443536119185) | > loss_mel_ce: 4.021289825439453 (nan) | > loss: 0.016133267432451248 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13620115449063558) | > loader_time: 0.0047 (0.004640986329721796)  --> TIME: 2025-02-24 09:12:24 -- STEP: 35072/36938 -- GLOBAL_STEP: 256700 | > loss_text_ce: 0.0374879315495491 (0.04387375383160249) | > loss_mel_ce: 2.997124195098877 (nan) | > loss: 0.012042112648487091 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13620384345442954) | > loader_time: 0.0045 (0.004641035048250286)  --> TIME: 2025-02-24 09:12:34 -- STEP: 35122/36938 -- GLOBAL_STEP: 256750 | > loss_text_ce: 0.04938265681266785 (0.04387247451621275) | > loss_mel_ce: 3.760141134262085 (nan) | > loss: 0.015117159113287926 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.1362030976834574) | > loader_time: 0.0047 (0.004641020442594868)  --> TIME: 2025-02-24 09:12:43 -- STEP: 35172/36938 -- GLOBAL_STEP: 256800 | > loss_text_ce: 0.038055114448070526 (0.043870762236325696) | > loss_mel_ce: 3.490884780883789 (nan) | > loss: 0.01400373037904501 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13620741866519703) | > loader_time: 0.0046 (0.00464105233931369)  --> TIME: 2025-02-24 09:12:52 -- STEP: 35222/36938 -- GLOBAL_STEP: 256850 | > loss_text_ce: 0.04425975680351257 (0.0438661846146047) | > loss_mel_ce: 3.9780242443084717 (nan) | > loss: 0.01596144586801529 (nan) | > current_lr: 5e-06 | > step_time: 0.1794 (0.13619916161255663) | > loader_time: 0.0048 (0.004640980708010852)  --> TIME: 2025-02-24 09:13:01 -- STEP: 35272/36938 -- GLOBAL_STEP: 256900 | > loss_text_ce: 0.052333295345306396 (0.04386370091359571) | > loss_mel_ce: 3.3461976051330566 (nan) | > loss: 0.013486234471201897 (nan) | > current_lr: 5e-06 | > step_time: 0.1901 (0.13619838691833885) | > loader_time: 0.0049 (0.004640909617762082)  --> TIME: 2025-02-24 09:13:10 -- STEP: 35322/36938 -- GLOBAL_STEP: 256950 | > loss_text_ce: 0.029884815216064453 (0.043863797277683365) | > loss_mel_ce: 1.316210389137268 (nan) | > loss: 0.005341648124158382 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13619239884725337) | > loader_time: 0.0045 (0.004640831067684811)  --> TIME: 2025-02-24 09:13:20 -- STEP: 35372/36938 -- GLOBAL_STEP: 257000 | > loss_text_ce: 0.03811068460345268 (0.043861716688390805) | > loss_mel_ce: 3.5232155323028564 (nan) | > loss: 0.014132248237729073 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.13619524525609322) | > loader_time: 0.0048 (0.0046407708980966186)  --> TIME: 2025-02-24 09:13:29 -- STEP: 35422/36938 -- GLOBAL_STEP: 257050 | > loss_text_ce: 0.03824537992477417 (0.04385838349556989) | > loss_mel_ce: 3.214772939682007 (nan) | > loss: 0.01290880423039198 (nan) | > current_lr: 5e-06 | > step_time: 0.1486 (0.13619286134560615) | > loader_time: 0.0045 (0.004640917466782702)  --> TIME: 2025-02-24 09:13:38 -- STEP: 35472/36938 -- GLOBAL_STEP: 257100 | > loss_text_ce: 0.03738318011164665 (0.04385739216127464) | > loss_mel_ce: 2.653628349304199 (nan) | > loss: 0.010678617283701897 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13619213353340123) | > loader_time: 0.0044 (0.00464088702637384)  --> TIME: 2025-02-24 09:13:48 -- STEP: 35522/36938 -- GLOBAL_STEP: 257150 | > loss_text_ce: 0.04383881390094757 (0.043858034354287784) | > loss_mel_ce: 2.8534297943115234 (nan) | > loss: 0.01149709802120924 (nan) | > current_lr: 5e-06 | > step_time: 0.171 (0.1362013398416345) | > loader_time: 0.0047 (0.00464104407341467)  --> TIME: 2025-02-24 09:13:57 -- STEP: 35572/36938 -- GLOBAL_STEP: 257200 | > loss_text_ce: 0.04541900008916855 (0.04386019672360401) | > loss_mel_ce: 3.267085075378418 (nan) | > loss: 0.013144858181476593 (nan) | > current_lr: 5e-06 | > step_time: 0.1606 (0.13619831909844393) | > loader_time: 0.0047 (0.0046409779440548275)  --> TIME: 2025-02-24 09:14:07 -- STEP: 35622/36938 -- GLOBAL_STEP: 257250 | > loss_text_ce: 0.03847352787852287 (0.043857662598864235) | > loss_mel_ce: 3.128040313720703 (nan) | > loss: 0.012565531767904758 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13620013288723173) | > loader_time: 0.0045 (0.004641018847627801)  --> TIME: 2025-02-24 09:14:16 -- STEP: 35672/36938 -- GLOBAL_STEP: 257300 | > loss_text_ce: 0.05289756506681442 (0.043856137258539714) | > loss_mel_ce: 3.147444009780884 (nan) | > loss: 0.012699768878519535 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13620107197713455) | > loader_time: 0.0045 (0.004641050807454971)  --> TIME: 2025-02-24 09:14:25 -- STEP: 35722/36938 -- GLOBAL_STEP: 257350 | > loss_text_ce: 0.0379423126578331 (0.043859986452839284) | > loss_mel_ce: 3.667283296585083 (nan) | > loss: 0.014703277498483658 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.1362055187282477) | > loader_time: 0.0047 (0.004641201660178029)  --> TIME: 2025-02-24 09:14:34 -- STEP: 35772/36938 -- GLOBAL_STEP: 257400 | > loss_text_ce: 0.03807850182056427 (0.043859402734550884) | > loss_mel_ce: 3.5917539596557617 (nan) | > loss: 0.014404098503291607 (nan) | > current_lr: 5e-06 | > step_time: 0.1858 (0.1361958059330964) | > loader_time: 0.0049 (0.0046412384138136056)  --> TIME: 2025-02-24 09:14:44 -- STEP: 35822/36938 -- GLOBAL_STEP: 257450 | > loss_text_ce: 0.039390187710523605 (0.04385646144661806) | > loss_mel_ce: 0.047119140625 (nan) | > loss: 0.00034329103073105216 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1361960576185065) | > loader_time: 0.0047 (0.004641192361781504)  --> TIME: 2025-02-24 09:14:53 -- STEP: 35872/36938 -- GLOBAL_STEP: 257500 | > loss_text_ce: 0.026665737852454185 (0.043854449972024694) | > loss_mel_ce: 1.7367017269134521 (nan) | > loss: 0.006997490301728249 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13619348821525196) | > loader_time: 0.0046 (0.004641239234743956)  --> TIME: 2025-02-24 09:15:02 -- STEP: 35922/36938 -- GLOBAL_STEP: 257550 | > loss_text_ce: 0.03420080989599228 (0.04385258474816887) | > loss_mel_ce: 3.044642448425293 (nan) | > loss: 0.012217633426189423 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13619152636512374) | > loader_time: 0.0046 (0.004641296842184381)  --> TIME: 2025-02-24 09:15:11 -- STEP: 35972/36938 -- GLOBAL_STEP: 257600 | > loss_text_ce: 0.026830382645130157 (0.04385162774963436) | > loss_mel_ce: 0.0252979826182127 (nan) | > loss: 0.00020685861818492413 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.136186083206462) | > loader_time: 0.0046 (0.00464128942891539)  --> TIME: 2025-02-24 09:15:20 -- STEP: 36022/36938 -- GLOBAL_STEP: 257650 | > loss_text_ce: 0.046056926250457764 (0.043850043247215115) | > loss_mel_ce: 0.561065673828125 (nan) | > loss: 0.0024092167150229216 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.1361856387236166) | > loader_time: 0.0045 (0.004641386677771279)  --> TIME: 2025-02-24 09:15:30 -- STEP: 36072/36938 -- GLOBAL_STEP: 257700 | > loss_text_ce: 0.041134536266326904 (0.04384971946052566) | > loss_mel_ce: 3.2056617736816406 (nan) | > loss: 0.012884113006293774 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13618754417226347) | > loader_time: 0.0045 (0.004641473498199533)  --> TIME: 2025-02-24 09:15:39 -- STEP: 36122/36938 -- GLOBAL_STEP: 257750 | > loss_text_ce: 0.03805599734187126 (0.0438467273967679) | > loss_mel_ce: 2.9771552085876465 (nan) | > loss: 0.011965123936533928 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.13619012355038812) | > loader_time: 0.0048 (0.00464152473989141)  --> TIME: 2025-02-24 09:15:49 -- STEP: 36172/36938 -- GLOBAL_STEP: 257800 | > loss_text_ce: 0.041136790066957474 (0.04384542056822238) | > loss_mel_ce: 1.7807337045669556 (nan) | > loss: 0.007229645270854235 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13619489032030774) | > loader_time: 0.0045 (0.004641590465898723)  --> TIME: 2025-02-24 09:15:58 -- STEP: 36222/36938 -- GLOBAL_STEP: 257850 | > loss_text_ce: 0.045132048428058624 (0.04384334504634737) | > loss_mel_ce: 2.8860061168670654 (nan) | > loss: 0.011631501838564873 (nan) | > current_lr: 5e-06 | > step_time: 0.1068 (0.13620097346508855) | > loader_time: 0.0045 (0.004641739162733409)  --> TIME: 2025-02-24 09:16:08 -- STEP: 36272/36938 -- GLOBAL_STEP: 257900 | > loss_text_ce: 0.04618389159440994 (0.043840486155128144) | > loss_mel_ce: 2.2587006092071533 (nan) | > loss: 0.009146367199718952 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.1362025506324479) | > loader_time: 0.0046 (0.004641828318235355)  --> TIME: 2025-02-24 09:16:17 -- STEP: 36322/36938 -- GLOBAL_STEP: 257950 | > loss_text_ce: 0.04297750070691109 (0.043840276224002966) | > loss_mel_ce: 3.6927649974823 (nan) | > loss: 0.014824376441538334 (nan) | > current_lr: 5e-06 | > step_time: 0.1075 (0.13619731847203195) | > loader_time: 0.0045 (0.004641798727903956)  --> TIME: 2025-02-24 09:16:26 -- STEP: 36372/36938 -- GLOBAL_STEP: 258000 | > loss_text_ce: 0.05064476281404495 (0.04383960778018473) | > loss_mel_ce: 2.189037561416626 (nan) | > loss: 0.00888762902468443 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13619397800380134) | > loader_time: 0.0045 (0.0046417924236388484)  --> TIME: 2025-02-24 09:16:35 -- STEP: 36422/36938 -- GLOBAL_STEP: 258050 | > loss_text_ce: 0.03742295131087303 (0.0438410237702414) | > loss_mel_ce: 2.7779836654663086 (nan) | > loss: 0.011172248981893063 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13619487542610173) | > loader_time: 0.0046 (0.004641841424237438)  --> TIME: 2025-02-24 09:16:45 -- STEP: 36472/36938 -- GLOBAL_STEP: 258100 | > loss_text_ce: 0.05452374741435051 (0.04384246608827108) | > loss_mel_ce: 3.0389950275421143 (nan) | > loss: 0.012275869026780128 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13619921518721395) | > loader_time: 0.0045 (0.004642000870893947)  --> TIME: 2025-02-24 09:16:54 -- STEP: 36522/36938 -- GLOBAL_STEP: 258150 | > loss_text_ce: 0.04128740727901459 (0.043843153026790954) | > loss_mel_ce: 1.5385690927505493 (nan) | > loss: 0.006269272416830063 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13620001623725983) | > loader_time: 0.0046 (0.004642037603485759)  --> TIME: 2025-02-24 09:17:03 -- STEP: 36572/36938 -- GLOBAL_STEP: 258200 | > loss_text_ce: 0.03806522488594055 (0.04384081037079925) | > loss_mel_ce: 3.2695746421813965 (nan) | > loss: 0.013125555589795113 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13620222591407247) | > loader_time: 0.0045 (0.00464200877679145)  --> TIME: 2025-02-24 09:17:13 -- STEP: 36622/36938 -- GLOBAL_STEP: 258250 | > loss_text_ce: 0.042350102216005325 (0.04383833067669421) | > loss_mel_ce: 3.822497844696045 (nan) | > loss: 0.015336698852479458 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.13620757834640917) | > loader_time: 0.0047 (0.004641984058659478)  --> TIME: 2025-02-24 09:17:22 -- STEP: 36672/36938 -- GLOBAL_STEP: 258300 | > loss_text_ce: 0.04336415231227875 (0.043836718016761) | > loss_mel_ce: 2.326575756072998 (nan) | > loss: 0.00940452329814434 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.136212939874352) | > loader_time: 0.0045 (0.004642050030655869)  --> TIME: 2025-02-24 09:17:32 -- STEP: 36722/36938 -- GLOBAL_STEP: 258350 | > loss_text_ce: 0.037082917988300323 (0.04383573652952191) | > loss_mel_ce: 0.196075439453125 (nan) | > loss: 0.0009252316085621715 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13622188354511097) | > loader_time: 0.0048 (0.004642134067000816)  --> TIME: 2025-02-24 09:17:42 -- STEP: 36772/36938 -- GLOBAL_STEP: 258400 | > loss_text_ce: 0.03807312995195389 (0.04383500379512819) | > loss_mel_ce: 3.026055335998535 (nan) | > loss: 0.012159240432083607 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13622380954376656) | > loader_time: 0.0046 (0.004642198650644752)  --> TIME: 2025-02-24 09:17:51 -- STEP: 36822/36938 -- GLOBAL_STEP: 258450 | > loss_text_ce: 0.03890328109264374 (0.043834699059954914) | > loss_mel_ce: 2.405778169631958 (nan) | > loss: 0.009701116941869259 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.1362216494640376) | > loader_time: 0.0047 (0.004642174268661084)  --> TIME: 2025-02-24 09:18:00 -- STEP: 36872/36938 -- GLOBAL_STEP: 258500 | > loss_text_ce: 0.04243231564760208 (0.043835269702095135) | > loss_mel_ce: 3.271122694015503 (nan) | > loss: 0.013149028643965721 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.13622640451890428) | > loader_time: 0.0047 (0.004642298453592311)  --> TIME: 2025-02-24 09:18:10 -- STEP: 36922/36938 -- GLOBAL_STEP: 258550 | > loss_text_ce: 0.04392256960272789 (0.04383536342014919) | > loss_mel_ce: 3.043851375579834 (nan) | > loss: 0.01225307211279869 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13622672592313115) | > loader_time: 0.0045 (0.004642257374794001)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.020214955405433565 (-0.0005056763639544638) | > avg_loss_text_ce: 0.04186495054181256 (-0.0015484968218768339) | > avg_loss_mel_ce: 2.8923728294891897 (-0.05896684202817504) | > avg_loss: 2.9342377864488283 (-0.0605153293892875) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_258566.pth  > EPOCH: 8/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-24 09:18:37)   --> TIME: 2025-02-24 09:18:43 -- STEP: 34/36938 -- GLOBAL_STEP: 258600 | > loss_text_ce: 0.04777878150343895 (0.04276994166567045) | > loss_mel_ce: 2.6117186546325684 (2.7766395339961436) | > loss: 0.010553562082350254 (0.011188133577781948) | > current_lr: 5e-06 | > step_time: 0.1485 (0.13018155799192543) | > loader_time: 0.0047 (0.004949177012724035)  --> TIME: 2025-02-24 09:18:52 -- STEP: 84/36938 -- GLOBAL_STEP: 258650 | > loss_text_ce: 0.04695006087422371 (0.04361914572793813) | > loss_mel_ce: 3.710418224334717 (nan) | > loss: 0.014910192228853703 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13118689968472436) | > loader_time: 0.0047 (0.004894807225181943)  --> TIME: 2025-02-24 09:19:02 -- STEP: 134/36938 -- GLOBAL_STEP: 258700 | > loss_text_ce: 0.04390903189778328 (0.043926448950460606) | > loss_mel_ce: 3.4084525108337402 (nan) | > loss: 0.01369984820485115 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.13388999896263) | > loader_time: 0.0046 (0.004813379316187618)  --> TIME: 2025-02-24 09:19:11 -- STEP: 184/36938 -- GLOBAL_STEP: 258750 | > loss_text_ce: 0.0484854131937027 (0.0436153026783596) | > loss_mel_ce: 2.840554714202881 (nan) | > loss: 0.011464445851743221 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13402496213498322) | > loader_time: 0.0046 (0.004796920911125516)  --> TIME: 2025-02-24 09:19:20 -- STEP: 234/36938 -- GLOBAL_STEP: 258800 | > loss_text_ce: 0.03743570297956467 (0.04366351674414344) | > loss_mel_ce: 3.6167638301849365 (nan) | > loss: 0.014500793069601059 (nan) | > current_lr: 5e-06 | > step_time: 0.163 (0.13423548702500818) | > loader_time: 0.0049 (0.004785547908554731)  --> TIME: 2025-02-24 09:19:30 -- STEP: 284/36938 -- GLOBAL_STEP: 258850 | > loss_text_ce: 0.051027946174144745 (0.04400241834728021) | > loss_mel_ce: 2.958660364151001 (nan) | > loss: 0.011943208985030651 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.1355371021888625) | > loader_time: 0.0045 (0.004773495902477857)  --> TIME: 2025-02-24 09:19:39 -- STEP: 334/36938 -- GLOBAL_STEP: 258900 | > loss_text_ce: 0.026853332296013832 (0.04367105631592745) | > loss_mel_ce: 0.034912109375 (nan) | > loss: 0.00024510096409358084 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1353838943435761) | > loader_time: 0.0044 (0.004768745627945772)  --> TIME: 2025-02-24 09:19:48 -- STEP: 384/36938 -- GLOBAL_STEP: 258950 | > loss_text_ce: 0.03956788033246994 (0.04352097301549898) | > loss_mel_ce: 3.332153558731079 (nan) | > loss: 0.013379847630858421 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.13623339186112096) | > loader_time: 0.0048 (0.0047531283150116605)  --> TIME: 2025-02-24 09:19:58 -- STEP: 434/36938 -- GLOBAL_STEP: 259000 | > loss_text_ce: 0.052666034549474716 (0.043284753110036586) | > loss_mel_ce: 3.1452181339263916 (nan) | > loss: 0.012690016999840736 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.13660320378668303) | > loader_time: 0.0053 (0.004749661217087419)  --> TIME: 2025-02-24 09:20:07 -- STEP: 484/36938 -- GLOBAL_STEP: 259050 | > loss_text_ce: 0.04685104638338089 (0.043389237809095) | > loss_mel_ce: 2.877946138381958 (nan) | > loss: 0.011606339365243912 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13676331013687404) | > loader_time: 0.0047 (0.004744079979983238)  --> TIME: 2025-02-24 09:20:17 -- STEP: 534/36938 -- GLOBAL_STEP: 259100 | > loss_text_ce: 0.0449993833899498 (0.04344894107752078) | > loss_mel_ce: 0.0631866455078125 (nan) | > loss: 0.00042930967174470425 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.1368488573402949) | > loader_time: 0.0045 (0.004749284701400924)  --> TIME: 2025-02-24 09:20:26 -- STEP: 584/36938 -- GLOBAL_STEP: 259150 | > loss_text_ce: 0.04216673597693443 (0.043317029606678506) | > loss_mel_ce: 3.172699213027954 (nan) | > loss: 0.012757405638694763 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13652887654631127) | > loader_time: 0.0046 (0.004755265092196529)  --> TIME: 2025-02-24 09:20:35 -- STEP: 634/36938 -- GLOBAL_STEP: 259200 | > loss_text_ce: 0.05383231118321419 (0.04339036367516318) | > loss_mel_ce: 2.7905545234680176 (nan) | > loss: 0.011287249624729156 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1361681997588008) | > loader_time: 0.0045 (0.004744321389905289)  --> TIME: 2025-02-24 09:20:44 -- STEP: 684/36938 -- GLOBAL_STEP: 259250 | > loss_text_ce: 0.06314321607351303 (0.0434171612842869) | > loss_mel_ce: 3.4005398750305176 (nan) | > loss: 0.013744775205850601 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.1360851320606926) | > loader_time: 0.0047 (0.004735383373952054)  --> TIME: 2025-02-24 09:20:54 -- STEP: 734/36938 -- GLOBAL_STEP: 259300 | > loss_text_ce: 0.04985211789608002 (0.0433984776471592) | > loss_mel_ce: 3.360466480255127 (nan) | > loss: 0.013533011078834534 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13615270628916168) | > loader_time: 0.0045 (0.004726685360277065)  --> TIME: 2025-02-24 09:21:03 -- STEP: 784/36938 -- GLOBAL_STEP: 259350 | > loss_text_ce: 0.04345018416643143 (0.043451928136851274) | > loss_mel_ce: 3.445134401321411 (nan) | > loss: 0.013843590393662453 (nan) | > current_lr: 5e-06 | > step_time: 0.1688 (0.13598638985838224) | > loader_time: 0.0048 (0.004722528919881714)  --> TIME: 2025-02-24 09:21:12 -- STEP: 834/36938 -- GLOBAL_STEP: 259400 | > loss_text_ce: 0.054411664605140686 (0.04337614957618415) | > loss_mel_ce: 3.425243377685547 (nan) | > loss: 0.013808155432343483 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.1358924986933063) | > loader_time: 0.0048 (0.004716548702413809)  --> TIME: 2025-02-24 09:21:21 -- STEP: 884/36938 -- GLOBAL_STEP: 259450 | > loss_text_ce: 0.04850541800260544 (0.04337363386299142) | > loss_mel_ce: 4.061217308044434 (nan) | > loss: 0.01630842313170433 (nan) | > current_lr: 5e-06 | > step_time: 0.1788 (0.13586815153311838) | > loader_time: 0.0048 (0.004711365537945506)  --> TIME: 2025-02-24 09:21:31 -- STEP: 934/36938 -- GLOBAL_STEP: 259500 | > loss_text_ce: 0.042303938418626785 (0.043274852650626965) | > loss_mel_ce: 3.933229684829712 (nan) | > loss: 0.015775928273797035 (nan) | > current_lr: 5e-06 | > step_time: 0.1442 (0.1363505699394604) | > loader_time: 0.0048 (0.004710298789389899)  --> TIME: 2025-02-24 09:21:41 -- STEP: 984/36938 -- GLOBAL_STEP: 259550 | > loss_text_ce: 0.029275519773364067 (0.043368234984364706) | > loss_mel_ce: 3.0135796070098877 (nan) | > loss: 0.012074822559952736 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13679522154777052) | > loader_time: 0.0045 (0.004710064186313293)  --> TIME: 2025-02-24 09:21:50 -- STEP: 1034/36938 -- GLOBAL_STEP: 259600 | > loss_text_ce: 0.04218962416052818 (0.04338049248934544) | > loss_mel_ce: 1.6908055543899536 (nan) | > loss: 0.006876965519040823 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13667500595520748) | > loader_time: 0.0045 (0.004706245557021361)  --> TIME: 2025-02-24 09:21:59 -- STEP: 1084/36938 -- GLOBAL_STEP: 259650 | > loss_text_ce: 0.040790658444166183 (0.04339993147289459) | > loss_mel_ce: 3.3414981365203857 (nan) | > loss: 0.013421781361103058 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.13682052108194576) | > loader_time: 0.0047 (0.004703741671854284)  --> TIME: 2025-02-24 09:22:09 -- STEP: 1134/36938 -- GLOBAL_STEP: 259700 | > loss_text_ce: 0.03557959944009781 (0.043347738734142605) | > loss_mel_ce: 2.380842685699463 (nan) | > loss: 0.009588978253304958 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13687440411333918) | > loader_time: 0.0046 (0.004700450670151488)  --> TIME: 2025-02-24 09:22:18 -- STEP: 1184/36938 -- GLOBAL_STEP: 259750 | > loss_text_ce: 0.04462463781237602 (0.04325979354325682) | > loss_mel_ce: 3.883110284805298 (nan) | > loss: 0.015586250461637974 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13683745647604417) | > loader_time: 0.0048 (0.004697198400626316)  --> TIME: 2025-02-24 09:22:27 -- STEP: 1234/36938 -- GLOBAL_STEP: 259800 | > loss_text_ce: 0.03921804577112198 (0.0432350110893577) | > loss_mel_ce: 1.685317039489746 (nan) | > loss: 0.0068433936685323715 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.13669808853773893) | > loader_time: 0.0047 (0.004693364981121147)  --> TIME: 2025-02-24 09:22:37 -- STEP: 1284/36938 -- GLOBAL_STEP: 259850 | > loss_text_ce: 0.04482441768050194 (0.043281894011823384) | > loss_mel_ce: 2.4164912700653076 (nan) | > loss: 0.00976712629199028 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13694517541897283) | > loader_time: 0.0044 (0.0046905667239631865)  --> TIME: 2025-02-24 09:22:46 -- STEP: 1334/36938 -- GLOBAL_STEP: 259900 | > loss_text_ce: 0.03607336804270744 (0.04321468783182276) | > loss_mel_ce: 2.9853413105010986 (nan) | > loss: 0.011989741586148739 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.13705345376141972) | > loader_time: 0.0048 (0.004687268873383443)  --> TIME: 2025-02-24 09:22:55 -- STEP: 1384/36938 -- GLOBAL_STEP: 259950 | > loss_text_ce: 0.04739592596888542 (0.04318146364941034) | > loss_mel_ce: 2.9490668773651123 (nan) | > loss: 0.011890726163983345 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.1367388472736227) | > loader_time: 0.0046 (0.004683388106395753)  --> TIME: 2025-02-24 09:23:05 -- STEP: 1434/36938 -- GLOBAL_STEP: 260000 | > loss_text_ce: 0.027388233691453934 (0.043075651882509926) | > loss_mel_ce: 2.915886878967285 (nan) | > loss: 0.011679664254188538 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13675930476753953) | > loader_time: 0.0046 (0.004682306465244699) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_260000.pth  --> TIME: 2025-02-24 09:23:20 -- STEP: 1484/36938 -- GLOBAL_STEP: 260050 | > loss_text_ce: 0.04396313056349754 (0.04307848864016591) | > loss_mel_ce: 3.1224632263183594 (nan) | > loss: 0.012565185315907001 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.1367601818151552) | > loader_time: 0.0048 (0.00468170626786842)  --> TIME: 2025-02-24 09:23:29 -- STEP: 1534/36938 -- GLOBAL_STEP: 260100 | > loss_text_ce: 0.05693575739860535 (0.04312696886968935) | > loss_mel_ce: 3.188115119934082 (nan) | > loss: 0.012877186760306358 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13661494248849018) | > loader_time: 0.0045 (0.004680002528343557)  --> TIME: 2025-02-24 09:23:39 -- STEP: 1584/36938 -- GLOBAL_STEP: 260150 | > loss_text_ce: 0.03768865019083023 (0.04310173424304175) | > loss_mel_ce: 0.025146484375 (nan) | > loss: 0.0002493457868695259 (nan) | > current_lr: 5e-06 | > step_time: 0.1786 (0.13675335666748017) | > loader_time: 0.005 (0.004679648262081733)  --> TIME: 2025-02-24 09:23:48 -- STEP: 1634/36938 -- GLOBAL_STEP: 260200 | > loss_text_ce: 0.052348144352436066 (0.04316531179411252) | > loss_mel_ce: 2.6543312072753906 (nan) | > loss: 0.010740791447460651 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13688224896952755) | > loader_time: 0.0047 (0.004678933244002497)  --> TIME: 2025-02-24 09:23:58 -- STEP: 1684/36938 -- GLOBAL_STEP: 260250 | > loss_text_ce: 0.04096372425556183 (0.04311634189252371) | > loss_mel_ce: 3.477811336517334 (nan) | > loss: 0.013963393867015839 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13710468398703557) | > loader_time: 0.0046 (0.004678361489483982)  --> TIME: 2025-02-24 09:24:07 -- STEP: 1734/36938 -- GLOBAL_STEP: 260300 | > loss_text_ce: 0.06253140419721603 (0.04313970679436591) | > loss_mel_ce: 3.512436628341675 (nan) | > loss: 0.014186382293701172 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.136997259061917) | > loader_time: 0.0046 (0.004677364432962989)  --> TIME: 2025-02-24 09:24:16 -- STEP: 1784/36938 -- GLOBAL_STEP: 260350 | > loss_text_ce: 0.045001447200775146 (0.04313794889324452) | > loss_mel_ce: 3.52474045753479 (nan) | > loss: 0.014165643602609634 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.13693180078882833) | > loader_time: 0.0047 (0.004676557709817935)  --> TIME: 2025-02-24 09:24:26 -- STEP: 1834/36938 -- GLOBAL_STEP: 260400 | > loss_text_ce: 0.04613789543509483 (0.04315387500062052) | > loss_mel_ce: 3.7697019577026367 (nan) | > loss: 0.015142221935093403 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13708606528091877) | > loader_time: 0.0047 (0.004675589704981553)  --> TIME: 2025-02-24 09:24:35 -- STEP: 1884/36938 -- GLOBAL_STEP: 260450 | > loss_text_ce: 0.0484626479446888 (0.04313985797459947) | > loss_mel_ce: 3.7314422130584717 (nan) | > loss: 0.014999623410403728 (nan) | > current_lr: 5e-06 | > step_time: 0.1502 (0.13704797113017694) | > loader_time: 0.0047 (0.004674848224453618)  --> TIME: 2025-02-24 09:24:44 -- STEP: 1934/36938 -- GLOBAL_STEP: 260500 | > loss_text_ce: 0.029872795566916466 (0.04316585762821133) | > loss_mel_ce: 3.995854616165161 (nan) | > loss: 0.0159751083701849 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1369948464810789) | > loader_time: 0.0046 (0.004677179434183594)  --> TIME: 2025-02-24 09:24:53 -- STEP: 1984/36938 -- GLOBAL_STEP: 260550 | > loss_text_ce: 0.04403354600071907 (0.04316469888116476) | > loss_mel_ce: 3.6277637481689453 (nan) | > loss: 0.014570624567568302 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13686139461013602) | > loader_time: 0.0048 (0.004679458756600662)  --> TIME: 2025-02-24 09:25:03 -- STEP: 2034/36938 -- GLOBAL_STEP: 260600 | > loss_text_ce: 0.046860892325639725 (0.043110559885861745) | > loss_mel_ce: 3.241284132003784 (nan) | > loss: 0.013048195280134678 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13686505941749227) | > loader_time: 0.0048 (0.004680884622894562)  --> TIME: 2025-02-24 09:25:12 -- STEP: 2084/36938 -- GLOBAL_STEP: 260650 | > loss_text_ce: 0.04238962009549141 (0.043112754874510606) | > loss_mel_ce: 3.582832098007202 (nan) | > loss: 0.014385801739990711 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.1368583542600476) | > loader_time: 0.0046 (0.004681502102432696)  --> TIME: 2025-02-24 09:25:21 -- STEP: 2134/36938 -- GLOBAL_STEP: 260700 | > loss_text_ce: 0.04066644236445427 (0.04308542531182696) | > loss_mel_ce: 2.838228940963745 (nan) | > loss: 0.011424188502132893 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.13673137367162752) | > loader_time: 0.0048 (0.004679787684067906)  --> TIME: 2025-02-24 09:25:31 -- STEP: 2184/36938 -- GLOBAL_STEP: 260750 | > loss_text_ce: 0.042980194091796875 (0.043071332294664505) | > loss_mel_ce: 2.3442609310150146 (nan) | > loss: 0.009473179467022419 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13680733691205058) | > loader_time: 0.0047 (0.004680798385606147)  --> TIME: 2025-02-24 09:25:40 -- STEP: 2234/36938 -- GLOBAL_STEP: 260800 | > loss_text_ce: 0.04537858068943024 (0.04310693576074836) | > loss_mel_ce: 3.7191288471221924 (nan) | > loss: 0.014938523061573505 (nan) | > current_lr: 5e-06 | > step_time: 0.144 (0.13679690433594427) | > loader_time: 0.0053 (0.0046807984315399235)  --> TIME: 2025-02-24 09:25:49 -- STEP: 2284/36938 -- GLOBAL_STEP: 260850 | > loss_text_ce: 0.033307116478681564 (0.043128117521810484) | > loss_mel_ce: 3.3511483669281006 (nan) | > loss: 0.013430379331111908 (nan) | > current_lr: 5e-06 | > step_time: 0.1578 (0.13679067088509633) | > loader_time: 0.0047 (0.004682066352897688)  --> TIME: 2025-02-24 09:25:59 -- STEP: 2334/36938 -- GLOBAL_STEP: 260900 | > loss_text_ce: 0.043235134333372116 (0.043113432910002816) | > loss_mel_ce: 3.1640005111694336 (nan) | > loss: 0.012727126479148865 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13677149137678396) | > loader_time: 0.0047 (0.004682352222125559)  --> TIME: 2025-02-24 09:26:08 -- STEP: 2384/36938 -- GLOBAL_STEP: 260950 | > loss_text_ce: 0.04493068903684616 (0.043122840152222085) | > loss_mel_ce: 2.4390485286712646 (nan) | > loss: 0.009857061319053173 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13673381967432569) | > loader_time: 0.0046 (0.0046817487318243835)  --> TIME: 2025-02-24 09:26:17 -- STEP: 2434/36938 -- GLOBAL_STEP: 261000 | > loss_text_ce: 0.04589874669909477 (0.04314826962353638) | > loss_mel_ce: 3.016439199447632 (nan) | > loss: 0.012152135372161865 (nan) | > current_lr: 5e-06 | > step_time: 0.1827 (0.13672799611542227) | > loader_time: 0.0049 (0.004683252720922951)  --> TIME: 2025-02-24 09:26:26 -- STEP: 2484/36938 -- GLOBAL_STEP: 261050 | > loss_text_ce: 0.03199088200926781 (0.043116247350419044) | > loss_mel_ce: 3.48337984085083 (nan) | > loss: 0.013949884101748466 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13666176959128415) | > loader_time: 0.0048 (0.00468232752620311)  --> TIME: 2025-02-24 09:26:36 -- STEP: 2534/36938 -- GLOBAL_STEP: 261100 | > loss_text_ce: 0.03393617644906044 (0.043101420994309976) | > loss_mel_ce: 1.9067314863204956 (nan) | > loss: 0.007701062597334385 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1366483812362264) | > loader_time: 0.0045 (0.004681571977008687)  --> TIME: 2025-02-24 09:26:45 -- STEP: 2584/36938 -- GLOBAL_STEP: 261150 | > loss_text_ce: 0.0390312559902668 (0.04308630308372783) | > loss_mel_ce: 3.0929462909698486 (nan) | > loss: 0.012428482994437218 (nan) | > current_lr: 5e-06 | > step_time: 0.1859 (0.13664157065813778) | > loader_time: 0.0049 (0.004681774429492545)  --> TIME: 2025-02-24 09:26:54 -- STEP: 2634/36938 -- GLOBAL_STEP: 261200 | > loss_text_ce: 0.049080729484558105 (0.04307134547140019) | > loss_mel_ce: 3.211733341217041 (nan) | > loss: 0.012939739972352982 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.13660255054135007) | > loader_time: 0.0047 (0.004681670584276768)  --> TIME: 2025-02-24 09:27:03 -- STEP: 2684/36938 -- GLOBAL_STEP: 261250 | > loss_text_ce: 0.054494213312864304 (0.043081660698766254) | > loss_mel_ce: 3.405473470687866 (nan) | > loss: 0.013730031438171864 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1365208563612755) | > loader_time: 0.0046 (0.004681336364518698)  --> TIME: 2025-02-24 09:27:12 -- STEP: 2734/36938 -- GLOBAL_STEP: 261300 | > loss_text_ce: 0.03930576518177986 (0.04309444859332323) | > loss_mel_ce: 3.370692491531372 (nan) | > loss: 0.013531739823520184 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13647584158221396) | > loader_time: 0.0046 (0.00468163300188424)  --> TIME: 2025-02-24 09:27:21 -- STEP: 2784/36938 -- GLOBAL_STEP: 261350 | > loss_text_ce: 0.04349398985505104 (0.043086411887057315) | > loss_mel_ce: 3.213271141052246 (nan) | > loss: 0.01292367186397314 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13643532092201313) | > loader_time: 0.0046 (0.004680866653891824)  --> TIME: 2025-02-24 09:27:30 -- STEP: 2834/36938 -- GLOBAL_STEP: 261400 | > loss_text_ce: 0.04167300462722778 (0.043063470962324026) | > loss_mel_ce: 0.02459716796875 (nan) | > loss: 0.00026297688600607216 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13638320563120512) | > loader_time: 0.0046 (0.004680452585725117)  --> TIME: 2025-02-24 09:27:40 -- STEP: 2884/36938 -- GLOBAL_STEP: 261450 | > loss_text_ce: 0.04678487032651901 (0.04305030458223153) | > loss_mel_ce: 2.141169786453247 (nan) | > loss: 0.00868235994130373 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1364383554491687) | > loader_time: 0.0045 (0.0046800153430721905)  --> TIME: 2025-02-24 09:27:49 -- STEP: 2934/36938 -- GLOBAL_STEP: 261500 | > loss_text_ce: 0.032497622072696686 (0.043042274135474186) | > loss_mel_ce: 3.55708646774292 (nan) | > loss: 0.0142443822696805 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.13642576127614642) | > loader_time: 0.005 (0.004679102514094059)  --> TIME: 2025-02-24 09:27:58 -- STEP: 2984/36938 -- GLOBAL_STEP: 261550 | > loss_text_ce: 0.027242396026849747 (0.04306972016587902) | > loss_mel_ce: 3.57900071144104 (nan) | > loss: 0.014310489408671856 (nan) | > current_lr: 5e-06 | > step_time: 0.1776 (0.1363659769536344) | > loader_time: 0.0048 (0.004678393736601532)  --> TIME: 2025-02-24 09:28:08 -- STEP: 3034/36938 -- GLOBAL_STEP: 261600 | > loss_text_ce: 0.04375864565372467 (0.043054672689272336) | > loss_mel_ce: 2.4637227058410645 (nan) | > loss: 0.009950323030352592 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13637159581080652) | > loader_time: 0.0047 (0.004677646711773219)  --> TIME: 2025-02-24 09:28:17 -- STEP: 3084/36938 -- GLOBAL_STEP: 261650 | > loss_text_ce: 0.047093700617551804 (0.043032829624983705) | > loss_mel_ce: 3.6686172485351562 (nan) | > loss: 0.014744885265827179 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13643534893191744) | > loader_time: 0.0046 (0.004678056088557677)  --> TIME: 2025-02-24 09:28:26 -- STEP: 3134/36938 -- GLOBAL_STEP: 261700 | > loss_text_ce: 0.039407361298799515 (0.04303893405913212) | > loss_mel_ce: 3.829179286956787 (nan) | > loss: 0.015351534821093082 (nan) | > current_lr: 5e-06 | > step_time: 0.1735 (0.13646095873218633) | > loader_time: 0.0049 (0.004678549930862222)  --> TIME: 2025-02-24 09:28:35 -- STEP: 3184/36938 -- GLOBAL_STEP: 261750 | > loss_text_ce: 0.05005206912755966 (0.043031341640217935) | > loss_mel_ce: 3.223862409591675 (nan) | > loss: 0.012991725467145443 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13638190880193188) | > loader_time: 0.0045 (0.004678198889871359)  --> TIME: 2025-02-24 09:28:45 -- STEP: 3234/36938 -- GLOBAL_STEP: 261800 | > loss_text_ce: 0.047446101903915405 (0.04301095803440473) | > loss_mel_ce: 3.7224321365356445 (nan) | > loss: 0.014959834516048431 (nan) | > current_lr: 5e-06 | > step_time: 0.1438 (0.13636973299357946) | > loader_time: 0.0047 (0.00467778616063598)  --> TIME: 2025-02-24 09:28:54 -- STEP: 3284/36938 -- GLOBAL_STEP: 261850 | > loss_text_ce: 0.03146115317940712 (0.04302507395607662) | > loss_mel_ce: 3.2618563175201416 (nan) | > loss: 0.013068721629679203 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.13640066229905065) | > loader_time: 0.0047 (0.004678348385605264)  --> TIME: 2025-02-24 09:29:04 -- STEP: 3334/36938 -- GLOBAL_STEP: 261900 | > loss_text_ce: 0.03461563587188721 (0.04300371549093917) | > loss_mel_ce: 1.6814749240875244 (nan) | > loss: 0.006809883750975132 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.1364259977289213) | > loader_time: 0.0045 (0.0046782695968207045)  --> TIME: 2025-02-24 09:29:13 -- STEP: 3384/36938 -- GLOBAL_STEP: 261950 | > loss_text_ce: 0.054605793207883835 (0.04300433642838607) | > loss_mel_ce: 0.01470947265625 (nan) | > loss: 0.00027506062178872526 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.1364952243388971) | > loader_time: 0.0048 (0.004678893948841339)  --> TIME: 2025-02-24 09:29:23 -- STEP: 3434/36938 -- GLOBAL_STEP: 262000 | > loss_text_ce: 0.043614380061626434 (0.04298909681311658) | > loss_mel_ce: 3.331503391265869 (nan) | > loss: 0.013393325731158257 (nan) | > current_lr: 5e-06 | > step_time: 0.1435 (0.13653656939260517) | > loader_time: 0.0047 (0.004678745497501949)  --> TIME: 2025-02-24 09:29:32 -- STEP: 3484/36938 -- GLOBAL_STEP: 262050 | > loss_text_ce: 0.03924109414219856 (0.04298035257702314) | > loss_mel_ce: 3.832195520401001 (nan) | > loss: 0.015362844802439213 (nan) | > current_lr: 5e-06 | > step_time: 0.1975 (0.13652211961461563) | > loader_time: 0.0049 (0.004678500027114589)  --> TIME: 2025-02-24 09:29:41 -- STEP: 3534/36938 -- GLOBAL_STEP: 262100 | > loss_text_ce: 0.02736060321331024 (0.04297514487190752) | > loss_mel_ce: 3.32544207572937 (nan) | > loss: 0.013304773718118668 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1364869716549301) | > loader_time: 0.0045 (0.004679536711515262)  --> TIME: 2025-02-24 09:29:51 -- STEP: 3584/36938 -- GLOBAL_STEP: 262150 | > loss_text_ce: 0.04703928902745247 (0.04296783854130946) | > loss_mel_ce: 3.7374675273895264 (nan) | > loss: 0.015017884783446789 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13653236953541675) | > loader_time: 0.0047 (0.004679465732936366)  --> TIME: 2025-02-24 09:30:00 -- STEP: 3634/36938 -- GLOBAL_STEP: 262200 | > loss_text_ce: 0.04590674862265587 (0.04298372481871213) | > loss_mel_ce: 2.2325241565704346 (nan) | > loss: 0.009041393175721169 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13658154299446867) | > loader_time: 0.0045 (0.004679232753759978)  --> TIME: 2025-02-24 09:30:10 -- STEP: 3684/36938 -- GLOBAL_STEP: 262250 | > loss_text_ce: 0.05302022024989128 (0.04297041312167499) | > loss_mel_ce: 3.742967367172241 (nan) | > loss: 0.015063444152474403 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13659125831823551) | > loader_time: 0.0046 (0.00467947711126555)  --> TIME: 2025-02-24 09:30:19 -- STEP: 3734/36938 -- GLOBAL_STEP: 262300 | > loss_text_ce: 0.04279392957687378 (0.04296658564524569) | > loss_mel_ce: 2.4684793949127197 (nan) | > loss: 0.00996537134051323 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1366295984995579) | > loader_time: 0.0046 (0.004679667036611271)  --> TIME: 2025-02-24 09:30:28 -- STEP: 3784/36938 -- GLOBAL_STEP: 262350 | > loss_text_ce: 0.04753008112311363 (0.04295627757991586) | > loss_mel_ce: 3.87190842628479 (nan) | > loss: 0.015553329139947891 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13661544828283892) | > loader_time: 0.0046 (0.004679581516640904)  --> TIME: 2025-02-24 09:30:38 -- STEP: 3834/36938 -- GLOBAL_STEP: 262400 | > loss_text_ce: 0.047655537724494934 (0.04294447121528617) | > loss_mel_ce: 2.843862771987915 (nan) | > loss: 0.011474279686808586 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1366034186502021) | > loader_time: 0.0054 (0.004679580436252855)  --> TIME: 2025-02-24 09:30:47 -- STEP: 3884/36938 -- GLOBAL_STEP: 262450 | > loss_text_ce: 0.03486407548189163 (0.04294221661745143) | > loss_mel_ce: 2.4416561126708984 (nan) | > loss: 0.009827462024986744 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13656657265094507) | > loader_time: 0.0044 (0.004680255414528671)  --> TIME: 2025-02-24 09:30:56 -- STEP: 3934/36938 -- GLOBAL_STEP: 262500 | > loss_text_ce: 0.04415231943130493 (0.0429510112310205) | > loss_mel_ce: 3.7675657272338867 (nan) | > loss: 0.015125866048038006 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13658558189050793) | > loader_time: 0.0045 (0.004680443125202639)  --> TIME: 2025-02-24 09:31:05 -- STEP: 3984/36938 -- GLOBAL_STEP: 262550 | > loss_text_ce: 0.04561300575733185 (0.04295253589652096) | > loss_mel_ce: 3.917224407196045 (nan) | > loss: 0.015725545585155487 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13659140952380336) | > loader_time: 0.0047 (0.0046803802850256164)  --> TIME: 2025-02-24 09:31:15 -- STEP: 4034/36938 -- GLOBAL_STEP: 262600 | > loss_text_ce: 0.043402593582868576 (0.042960722934423344) | > loss_mel_ce: 3.130455732345581 (nan) | > loss: 0.012594676576554775 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.136600929141104) | > loader_time: 0.0047 (0.004681641356922595)  --> TIME: 2025-02-24 09:31:24 -- STEP: 4084/36938 -- GLOBAL_STEP: 262650 | > loss_text_ce: 0.035025257617235184 (0.04296253497979585) | > loss_mel_ce: 3.1851868629455566 (nan) | > loss: 0.01277862023562193 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13656999694728972) | > loader_time: 0.0046 (0.004680971557549951)  --> TIME: 2025-02-24 09:31:34 -- STEP: 4134/36938 -- GLOBAL_STEP: 262700 | > loss_text_ce: 0.043630197644233704 (0.042957637537438455) | > loss_mel_ce: 3.7292656898498535 (nan) | > loss: 0.014971809461712837 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.13666089093218028) | > loader_time: 0.0047 (0.004680633371904054)  --> TIME: 2025-02-24 09:31:43 -- STEP: 4184/36938 -- GLOBAL_STEP: 262750 | > loss_text_ce: 0.05094826966524124 (0.042950561953473534) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13667584371612387) | > loader_time: 0.0052 (0.004680733778736575)  --> TIME: 2025-02-24 09:31:52 -- STEP: 4234/36938 -- GLOBAL_STEP: 262800 | > loss_text_ce: 0.025217439979314804 (0.042943474118688936) | > loss_mel_ce: 2.6297607421875 (nan) | > loss: 0.010535628534853458 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13660409149833064) | > loader_time: 0.0053 (0.004681792302012289)  --> TIME: 2025-02-24 09:32:01 -- STEP: 4284/36938 -- GLOBAL_STEP: 262850 | > loss_text_ce: 0.03667442128062248 (0.042958059925133865) | > loss_mel_ce: 2.5153589248657227 (nan) | > loss: 0.010127116926014423 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.1365814475006975) | > loader_time: 0.0047 (0.0046820569327788225)  --> TIME: 2025-02-24 09:32:11 -- STEP: 4334/36938 -- GLOBAL_STEP: 262900 | > loss_text_ce: 0.042495694011449814 (0.04295043357792796) | > loss_mel_ce: 4.241020679473877 (nan) | > loss: 0.016998082399368286 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1365825437322214) | > loader_time: 0.0046 (0.004681789385430771)  --> TIME: 2025-02-24 09:32:20 -- STEP: 4384/36938 -- GLOBAL_STEP: 262950 | > loss_text_ce: 0.04532098025083542 (0.042938574050284686) | > loss_mel_ce: 3.823676109313965 (nan) | > loss: 0.015353163704276085 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.1365489043893609) | > loader_time: 0.0047 (0.00468237159678537)  --> TIME: 2025-02-24 09:32:29 -- STEP: 4434/36938 -- GLOBAL_STEP: 263000 | > loss_text_ce: 0.05441143363714218 (0.04293846476186253) | > loss_mel_ce: 2.9357569217681885 (nan) | > loss: 0.011865748092532158 (nan) | > current_lr: 5e-06 | > step_time: 0.1267 (0.13652498645722286) | > loader_time: 0.0046 (0.004682768665566669)  --> TIME: 2025-02-24 09:32:38 -- STEP: 4484/36938 -- GLOBAL_STEP: 263050 | > loss_text_ce: 0.0417022705078125 (0.04292919810813168) | > loss_mel_ce: 3.176988363265991 (nan) | > loss: 0.012772582471370697 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.1364689919695488) | > loader_time: 0.0046 (0.004682833971879412)  --> TIME: 2025-02-24 09:32:47 -- STEP: 4534/36938 -- GLOBAL_STEP: 263100 | > loss_text_ce: 0.044065263122320175 (0.04293073626562962) | > loss_mel_ce: 0.0096435546875 (nan) | > loss: 0.00021313024626579136 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13640921648031426) | > loader_time: 0.0046 (0.004682995014165451)  --> TIME: 2025-02-24 09:32:56 -- STEP: 4584/36938 -- GLOBAL_STEP: 263150 | > loss_text_ce: 0.03239510953426361 (0.042933803406441665) | > loss_mel_ce: 0.6000925898551941 (nan) | > loss: 0.0025098719634115696 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.1363674522382429) | > loader_time: 0.0047 (0.004683111090518642)  --> TIME: 2025-02-24 09:33:05 -- STEP: 4634/36938 -- GLOBAL_STEP: 263200 | > loss_text_ce: 0.03389777988195419 (0.042944322546793735) | > loss_mel_ce: 2.948052167892456 (nan) | > loss: 0.011833135969936848 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.13636702296545566) | > loader_time: 0.0046 (0.0046838749365710015)  --> TIME: 2025-02-24 09:33:15 -- STEP: 4684/36938 -- GLOBAL_STEP: 263250 | > loss_text_ce: 0.04813162237405777 (0.04294840763796228) | > loss_mel_ce: 3.551588535308838 (nan) | > loss: 0.014284605160355568 (nan) | > current_lr: 5e-06 | > step_time: 0.1784 (0.13640176674731277) | > loader_time: 0.0048 (0.004683215937789555)  --> TIME: 2025-02-24 09:33:24 -- STEP: 4734/36938 -- GLOBAL_STEP: 263300 | > loss_text_ce: 0.030521942302584648 (0.042943115841107875) | > loss_mel_ce: 1.9959712028503418 (nan) | > loss: 0.008041639812290668 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13648920362389555) | > loader_time: 0.0047 (0.004683086224432295)  --> TIME: 2025-02-24 09:33:34 -- STEP: 4784/36938 -- GLOBAL_STEP: 263350 | > loss_text_ce: 0.03990969434380531 (0.042929498839696725) | > loss_mel_ce: 3.578436851501465 (nan) | > loss: 0.01435851864516735 (nan) | > current_lr: 5e-06 | > step_time: 0.1688 (0.13649165072170003) | > loader_time: 0.0048 (0.004682277108115912)  --> TIME: 2025-02-24 09:33:43 -- STEP: 4834/36938 -- GLOBAL_STEP: 263400 | > loss_text_ce: 0.041345637291669846 (0.04292809075021249) | > loss_mel_ce: 3.552722215652466 (nan) | > loss: 0.014262174256145954 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.1365154818360058) | > loader_time: 0.0047 (0.00468279287545802)  --> TIME: 2025-02-24 09:33:52 -- STEP: 4884/36938 -- GLOBAL_STEP: 263450 | > loss_text_ce: 0.044668395072221756 (0.042938882940014904) | > loss_mel_ce: 3.0724713802337646 (nan) | > loss: 0.012369602918624878 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.1364822468730293) | > loader_time: 0.0045 (0.004682764618441687)  --> TIME: 2025-02-24 09:34:02 -- STEP: 4934/36938 -- GLOBAL_STEP: 263500 | > loss_text_ce: 0.05579402297735214 (0.04293592249575211) | > loss_mel_ce: 3.5925960540771484 (nan) | > loss: 0.014477739110589027 (nan) | > current_lr: 5e-06 | > step_time: 0.1588 (0.13648290821642944) | > loader_time: 0.0046 (0.004683753813042752)  --> TIME: 2025-02-24 09:34:11 -- STEP: 4984/36938 -- GLOBAL_STEP: 263550 | > loss_text_ce: 0.046494584530591965 (0.04292125983894735) | > loss_mel_ce: 3.5778493881225586 (nan) | > loss: 0.014382317662239075 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.13648528061556034) | > loader_time: 0.0047 (0.00468443135579939)  --> TIME: 2025-02-24 09:34:20 -- STEP: 5034/36938 -- GLOBAL_STEP: 263600 | > loss_text_ce: 0.06271057575941086 (0.042922128172481255) | > loss_mel_ce: 3.315168857574463 (nan) | > loss: 0.013404283672571182 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13649032136775438) | > loader_time: 0.0045 (0.0046837417483566735)  --> TIME: 2025-02-24 09:34:29 -- STEP: 5084/36938 -- GLOBAL_STEP: 263650 | > loss_text_ce: 0.029834356158971786 (0.04293237393245672) | > loss_mel_ce: 1.8705533742904663 (nan) | > loss: 0.00754122156649828 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13641584618273564) | > loader_time: 0.0046 (0.004682766415770853)  --> TIME: 2025-02-24 09:34:39 -- STEP: 5134/36938 -- GLOBAL_STEP: 263700 | > loss_text_ce: 0.04617904871702194 (0.04293725684025414) | > loss_mel_ce: 3.995361328125 (nan) | > loss: 0.01603785902261734 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13642318939763293) | > loader_time: 0.0045 (0.004682321561464893)  --> TIME: 2025-02-24 09:34:48 -- STEP: 5184/36938 -- GLOBAL_STEP: 263750 | > loss_text_ce: 0.03808065131306648 (0.04294111787587318) | > loss_mel_ce: 1.941671371459961 (nan) | > loss: 0.007856159470975399 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13640505337604777) | > loader_time: 0.0047 (0.004681720363873041)  --> TIME: 2025-02-24 09:34:57 -- STEP: 5234/36938 -- GLOBAL_STEP: 263800 | > loss_text_ce: 0.035914111882448196 (0.04294394642679784) | > loss_mel_ce: 2.9005050659179688 (nan) | > loss: 0.01165245845913887 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13642293340065675) | > loader_time: 0.0045 (0.004681130242703045)  --> TIME: 2025-02-24 09:35:06 -- STEP: 5284/36938 -- GLOBAL_STEP: 263850 | > loss_text_ce: 0.03884779289364815 (0.04295427258423265) | > loss_mel_ce: 3.2411136627197266 (nan) | > loss: 0.013015720061957836 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13640259550341516) | > loader_time: 0.0046 (0.004681172874459369)  --> TIME: 2025-02-24 09:35:16 -- STEP: 5334/36938 -- GLOBAL_STEP: 263900 | > loss_text_ce: 0.040537480264902115 (0.04295542498615408) | > loss_mel_ce: 3.143686294555664 (nan) | > loss: 0.012635809369385242 (nan) | > current_lr: 5e-06 | > step_time: 0.1704 (0.13645210447586759) | > loader_time: 0.0047 (0.004681814105760148)  --> TIME: 2025-02-24 09:35:26 -- STEP: 5384/36938 -- GLOBAL_STEP: 263950 | > loss_text_ce: 0.04009177163243294 (0.042946366937125446) | > loss_mel_ce: 2.613621711730957 (nan) | > loss: 0.010530609637498856 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1364974143069355) | > loader_time: 0.0051 (0.004682335819921758)  --> TIME: 2025-02-24 09:35:35 -- STEP: 5434/36938 -- GLOBAL_STEP: 264000 | > loss_text_ce: 0.04160766676068306 (0.04293878842539413) | > loss_mel_ce: 2.410604953765869 (nan) | > loss: 0.00973100308328867 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13652358513173263) | > loader_time: 0.0044 (0.004682744694493236)  --> TIME: 2025-02-24 09:35:45 -- STEP: 5484/36938 -- GLOBAL_STEP: 264050 | > loss_text_ce: 0.04526239261031151 (0.04293595964648118) | > loss_mel_ce: 3.773933172225952 (nan) | > loss: 0.01515553891658783 (nan) | > current_lr: 5e-06 | > step_time: 0.1775 (0.1365841004119833) | > loader_time: 0.0048 (0.0046833053198633125)  --> TIME: 2025-02-24 09:35:54 -- STEP: 5534/36938 -- GLOBAL_STEP: 264100 | > loss_text_ce: 0.039789024740457535 (0.042952467169596636) | > loss_mel_ce: 3.3234620094299316 (nan) | > loss: 0.0133462343364954 (nan) | > current_lr: 5e-06 | > step_time: 0.1357 (0.13654622486191692) | > loader_time: 0.0046 (0.004683533385207541)  --> TIME: 2025-02-24 09:36:03 -- STEP: 5584/36938 -- GLOBAL_STEP: 264150 | > loss_text_ce: 0.021877670660614967 (0.042964828051700224) | > loss_mel_ce: 2.2304606437683105 (nan) | > loss: 0.008937851525843143 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.13655913866693442) | > loader_time: 0.0044 (0.004683603615337276)  --> TIME: 2025-02-24 09:36:12 -- STEP: 5634/36938 -- GLOBAL_STEP: 264200 | > loss_text_ce: 0.03707504644989967 (0.042953342202466493) | > loss_mel_ce: 0.6603197455406189 (nan) | > loss: 0.0027674396988004446 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13652913122681534) | > loader_time: 0.0047 (0.0046839344615753715)  --> TIME: 2025-02-24 09:36:22 -- STEP: 5684/36938 -- GLOBAL_STEP: 264250 | > loss_text_ce: 0.037461332976818085 (0.042962690243985246) | > loss_mel_ce: 3.6235227584838867 (nan) | > loss: 0.014527715742588043 (nan) | > current_lr: 5e-06 | > step_time: 0.1876 (0.1365746298279246) | > loader_time: 0.0048 (0.004683831768250676)  --> TIME: 2025-02-24 09:36:32 -- STEP: 5734/36938 -- GLOBAL_STEP: 264300 | > loss_text_ce: 0.05743451789021492 (0.04296082759212378) | > loss_mel_ce: 3.5300633907318115 (nan) | > loss: 0.014236103743314743 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1366188317007163) | > loader_time: 0.0075 (0.004685100047148457)  --> TIME: 2025-02-24 09:36:41 -- STEP: 5784/36938 -- GLOBAL_STEP: 264350 | > loss_text_ce: 0.04224977269768715 (0.04297155330622943) | > loss_mel_ce: 1.389315128326416 (nan) | > loss: 0.005680813454091549 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13661734345542914) | > loader_time: 0.0045 (0.004684667823060775)  --> TIME: 2025-02-24 09:36:51 -- STEP: 5834/36938 -- GLOBAL_STEP: 264400 | > loss_text_ce: 0.04254644736647606 (0.04297604509043458) | > loss_mel_ce: 3.3987207412719727 (nan) | > loss: 0.013655823655426502 (nan) | > current_lr: 5e-06 | > step_time: 0.1815 (0.13666666898573734) | > loader_time: 0.0049 (0.004684891037063177)  --> TIME: 2025-02-24 09:36:59 -- STEP: 5884/36938 -- GLOBAL_STEP: 264450 | > loss_text_ce: 0.04327879473567009 (0.042980465836497735) | > loss_mel_ce: 2.7148873805999756 (nan) | > loss: 0.01094510406255722 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.1366119157044275) | > loader_time: 0.0046 (0.00468450229561793)  --> TIME: 2025-02-24 09:37:09 -- STEP: 5934/36938 -- GLOBAL_STEP: 264500 | > loss_text_ce: 0.06061769649386406 (0.04298975932473702) | > loss_mel_ce: 3.716967821121216 (nan) | > loss: 0.014990419149398804 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13658500420834255) | > loader_time: 0.0046 (0.004684534424838713)  --> TIME: 2025-02-24 09:37:18 -- STEP: 5984/36938 -- GLOBAL_STEP: 264550 | > loss_text_ce: 0.04596438631415367 (0.04298637292822863) | > loss_mel_ce: 3.3667385578155518 (nan) | > loss: 0.013542473316192627 (nan) | > current_lr: 5e-06 | > step_time: 0.1817 (0.13657999759848738) | > loader_time: 0.0049 (0.004685494192143832)  --> TIME: 2025-02-24 09:37:27 -- STEP: 6034/36938 -- GLOBAL_STEP: 264600 | > loss_text_ce: 0.04304900020360947 (0.042983871859794506) | > loss_mel_ce: 3.2138874530792236 (nan) | > loss: 0.01292435172945261 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13661250635057962) | > loader_time: 0.0045 (0.004685389944208689)  --> TIME: 2025-02-24 09:37:37 -- STEP: 6084/36938 -- GLOBAL_STEP: 264650 | > loss_text_ce: 0.0313054658472538 (0.042979651291008) | > loss_mel_ce: 3.7684812545776367 (nan) | > loss: 0.015078520402312279 (nan) | > current_lr: 5e-06 | > step_time: 0.151 (0.13660838147827403) | > loader_time: 0.0048 (0.004684957095778844)  --> TIME: 2025-02-24 09:37:46 -- STEP: 6134/36938 -- GLOBAL_STEP: 264700 | > loss_text_ce: 0.04961720481514931 (0.042976428734130864) | > loss_mel_ce: 2.1476364135742188 (nan) | > loss: 0.008719260804355145 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.13659368110939102) | > loader_time: 0.0046 (0.004684757984224821)  --> TIME: 2025-02-24 09:37:56 -- STEP: 6184/36938 -- GLOBAL_STEP: 264750 | > loss_text_ce: 0.034220337867736816 (0.0429915451017881) | > loss_mel_ce: 3.372992992401123 (nan) | > loss: 0.013520687818527222 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13667130867383173) | > loader_time: 0.0047 (0.004686678481441276)  --> TIME: 2025-02-24 09:38:05 -- STEP: 6234/36938 -- GLOBAL_STEP: 264800 | > loss_text_ce: 0.040953148156404495 (0.042987822569198135) | > loss_mel_ce: 2.8562848567962646 (nan) | > loss: 0.01149697694927454 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13664064022605166) | > loader_time: 0.0047 (0.004687371374821111)  --> TIME: 2025-02-24 09:38:14 -- STEP: 6284/36938 -- GLOBAL_STEP: 264850 | > loss_text_ce: 0.05110332742333412 (0.0429924971474138) | > loss_mel_ce: 3.2453691959381104 (nan) | > loss: 0.013081241399049759 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13664414073008038) | > loader_time: 0.0045 (0.0046879777780844956)  --> TIME: 2025-02-24 09:38:24 -- STEP: 6334/36938 -- GLOBAL_STEP: 264900 | > loss_text_ce: 0.040680039674043655 (0.04298247620278121) | > loss_mel_ce: 2.839191198348999 (nan) | > loss: 0.011428060941398144 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13665712101812605) | > loader_time: 0.0046 (0.00468786616486483)  --> TIME: 2025-02-24 09:38:33 -- STEP: 6384/36938 -- GLOBAL_STEP: 264950 | > loss_text_ce: 0.051882002502679825 (0.04297262686844532) | > loss_mel_ce: 3.13682222366333 (nan) | > loss: 0.012653589248657227 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.1366754331086814) | > loader_time: 0.0046 (0.004688159004787467)  --> TIME: 2025-02-24 09:38:43 -- STEP: 6434/36938 -- GLOBAL_STEP: 265000 | > loss_text_ce: 0.04422467574477196 (0.04297351943100865) | > loss_mel_ce: 3.246262788772583 (nan) | > loss: 0.013057490810751915 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13668291576908487) | > loader_time: 0.0056 (0.004688580028011013)  --> TIME: 2025-02-24 09:38:52 -- STEP: 6484/36938 -- GLOBAL_STEP: 265050 | > loss_text_ce: 0.03198361024260521 (0.04296985361164934) | > loss_mel_ce: 2.8819048404693604 (nan) | > loss: 0.011563049629330635 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13665279567719726) | > loader_time: 0.0046 (0.004688954037108017)  --> TIME: 2025-02-24 09:39:01 -- STEP: 6534/36938 -- GLOBAL_STEP: 265100 | > loss_text_ce: 0.04578177258372307 (0.042968245309398266) | > loss_mel_ce: 3.9396963119506836 (nan) | > loss: 0.015815390273928642 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13666813228339827) | > loader_time: 0.0048 (0.0046899536898788)  --> TIME: 2025-02-24 09:39:10 -- STEP: 6584/36938 -- GLOBAL_STEP: 265150 | > loss_text_ce: 0.03520979359745979 (0.042967177033656496) | > loss_mel_ce: 3.345672845840454 (nan) | > loss: 0.013416201807558537 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13668345239008947) | > loader_time: 0.0046 (0.004690740044757217)  --> TIME: 2025-02-24 09:39:20 -- STEP: 6634/36938 -- GLOBAL_STEP: 265200 | > loss_text_ce: 0.030504222959280014 (0.04296993224718474) | > loss_mel_ce: 0.1158447265625 (nan) | > loss: 0.0005807498237118125 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1366640668983451) | > loader_time: 0.0046 (0.0046909772958028265)  --> TIME: 2025-02-24 09:39:29 -- STEP: 6684/36938 -- GLOBAL_STEP: 265250 | > loss_text_ce: 0.045055173337459564 (0.04296737560934949) | > loss_mel_ce: 3.098504066467285 (nan) | > loss: 0.012474441900849342 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.1366624003181483) | > loader_time: 0.0045 (0.004690893070202701)  --> TIME: 2025-02-24 09:39:38 -- STEP: 6734/36938 -- GLOBAL_STEP: 265300 | > loss_text_ce: 0.04772607237100601 (0.04297464217988991) | > loss_mel_ce: 0.11611060798168182 (nan) | > loss: 0.0006501456373371184 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13663482311963712) | > loader_time: 0.0046 (0.004690677219659403)  --> TIME: 2025-02-24 09:39:47 -- STEP: 6784/36938 -- GLOBAL_STEP: 265350 | > loss_text_ce: 0.03563099727034569 (0.0429814311197712) | > loss_mel_ce: 3.370307445526123 (nan) | > loss: 0.01351562887430191 (nan) | > current_lr: 5e-06 | > step_time: 0.1773 (0.1366297966045032) | > loader_time: 0.0049 (0.004690920020330645)  --> TIME: 2025-02-24 09:39:57 -- STEP: 6834/36938 -- GLOBAL_STEP: 265400 | > loss_text_ce: 0.030466847121715546 (0.04297815426355004) | > loss_mel_ce: 2.8139703273773193 (nan) | > loss: 0.011287449859082699 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.1366163298579244) | > loader_time: 0.0046 (0.004690620890129807)  --> TIME: 2025-02-24 09:40:06 -- STEP: 6884/36938 -- GLOBAL_STEP: 265450 | > loss_text_ce: 0.037226662039756775 (0.0429702599532633) | > loss_mel_ce: 3.3119561672210693 (nan) | > loss: 0.013290408998727798 (nan) | > current_lr: 5e-06 | > step_time: 0.1267 (0.13660223556077472) | > loader_time: 0.0045 (0.004690908471350769)  --> TIME: 2025-02-24 09:40:15 -- STEP: 6934/36938 -- GLOBAL_STEP: 265500 | > loss_text_ce: 0.04020165652036667 (0.04296763540445194) | > loss_mel_ce: 1.044023036956787 (nan) | > loss: 0.004302479326725006 (nan) | > current_lr: 5e-06 | > step_time: 0.1626 (0.13663330300245594) | > loader_time: 0.0047 (0.004691285395230295)  --> TIME: 2025-02-24 09:40:24 -- STEP: 6984/36938 -- GLOBAL_STEP: 265550 | > loss_text_ce: 0.03389272466301918 (0.0429528569705297) | > loss_mel_ce: 3.2890625 (nan) | > loss: 0.013186330907046795 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13660783135467677) | > loader_time: 0.0048 (0.00469176008665823)  --> TIME: 2025-02-24 09:40:34 -- STEP: 7034/36938 -- GLOBAL_STEP: 265600 | > loss_text_ce: 0.04358339682221413 (0.04295208424931939) | > loss_mel_ce: 3.157681941986084 (nan) | > loss: 0.012703434564173222 (nan) | > current_lr: 5e-06 | > step_time: 0.1615 (0.13663701228808053) | > loader_time: 0.0049 (0.004692419164384822)  --> TIME: 2025-02-24 09:40:44 -- STEP: 7084/36938 -- GLOBAL_STEP: 265650 | > loss_text_ce: 0.04559624195098877 (0.042952658126461075) | > loss_mel_ce: 3.67756724357605 (nan) | > loss: 0.014774459414184093 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13666229450655581) | > loader_time: 0.0047 (0.004692407565760402)  --> TIME: 2025-02-24 09:40:53 -- STEP: 7134/36938 -- GLOBAL_STEP: 265700 | > loss_text_ce: 0.05212509632110596 (0.04295203229125246) | > loss_mel_ce: 2.5079991817474365 (nan) | > loss: 0.010159224271774292 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.1366801539956608) | > loader_time: 0.0046 (0.004693016539376643)  --> TIME: 2025-02-24 09:41:02 -- STEP: 7184/36938 -- GLOBAL_STEP: 265750 | > loss_text_ce: 0.04199856519699097 (0.042932486595748894) | > loss_mel_ce: 2.654696226119995 (nan) | > loss: 0.01070117112249136 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13667022309085616) | > loader_time: 0.0046 (0.004693211651591805)  --> TIME: 2025-02-24 09:41:11 -- STEP: 7234/36938 -- GLOBAL_STEP: 265800 | > loss_text_ce: 0.03448253124952316 (0.042937360831271525) | > loss_mel_ce: 3.375279188156128 (nan) | > loss: 0.013530801050364971 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.136666008013535) | > loader_time: 0.0047 (0.004693233245045978)  --> TIME: 2025-02-24 09:41:21 -- STEP: 7284/36938 -- GLOBAL_STEP: 265850 | > loss_text_ce: 0.04038284346461296 (0.04293749503418509) | > loss_mel_ce: 3.4078805446624756 (nan) | > loss: 0.013683585450053215 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13664943238394045) | > loader_time: 0.0044 (0.004693824730359367)  --> TIME: 2025-02-24 09:41:30 -- STEP: 7334/36938 -- GLOBAL_STEP: 265900 | > loss_text_ce: 0.04776691645383835 (0.04293431253862009) | > loss_mel_ce: 3.836899995803833 (nan) | > loss: 0.015415346249938011 (nan) | > current_lr: 5e-06 | > step_time: 0.1842 (0.13662795094660957) | > loader_time: 0.0049 (0.004694052018227242)  --> TIME: 2025-02-24 09:41:39 -- STEP: 7384/36938 -- GLOBAL_STEP: 265950 | > loss_text_ce: 0.05198607221245766 (0.04293745255336392) | > loss_mel_ce: 3.2051825523376465 (nan) | > loss: 0.012925272807478905 (nan) | > current_lr: 5e-06 | > step_time: 0.1441 (0.13661746650062478) | > loader_time: 0.0046 (0.00469448335896594)  --> TIME: 2025-02-24 09:41:48 -- STEP: 7434/36938 -- GLOBAL_STEP: 266000 | > loss_text_ce: 0.04459675773978233 (0.042928697637194486) | > loss_mel_ce: 0.3866373598575592 (nan) | > loss: 0.001711246557533741 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1365959384539519) | > loader_time: 0.0048 (0.004694533085098263)  --> TIME: 2025-02-24 09:41:57 -- STEP: 7484/36938 -- GLOBAL_STEP: 266050 | > loss_text_ce: 0.0351954847574234 (0.04292979503171416) | > loss_mel_ce: 3.7616145610809326 (nan) | > loss: 0.015066707506775856 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.13657042439897626) | > loader_time: 0.0047 (0.004694681636539152)  --> TIME: 2025-02-24 09:42:07 -- STEP: 7534/36938 -- GLOBAL_STEP: 266100 | > loss_text_ce: 0.04181310907006264 (0.042931738056767706) | > loss_mel_ce: 3.5031096935272217 (nan) | > loss: 0.014067155309021473 (nan) | > current_lr: 5e-06 | > step_time: 0.1796 (0.13664001284969726) | > loader_time: 0.0048 (0.004695293755881963)  --> TIME: 2025-02-24 09:42:16 -- STEP: 7584/36938 -- GLOBAL_STEP: 266150 | > loss_text_ce: 0.040005724877119064 (0.042931602753130134) | > loss_mel_ce: 3.1437580585479736 (nan) | > loss: 0.012633983977138996 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13661466774684006) | > loader_time: 0.0046 (0.004695017503786708)  --> TIME: 2025-02-24 09:42:25 -- STEP: 7634/36938 -- GLOBAL_STEP: 266200 | > loss_text_ce: 0.039473991841077805 (0.042939535214670484) | > loss_mel_ce: 3.4898202419281006 (nan) | > loss: 0.014005136676132679 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13658753142857863) | > loader_time: 0.0054 (0.004696228037587583)  --> TIME: 2025-02-24 09:42:34 -- STEP: 7684/36938 -- GLOBAL_STEP: 266250 | > loss_text_ce: 0.044092703610658646 (0.04293620443763781) | > loss_mel_ce: 3.285332202911377 (nan) | > loss: 0.013212003745138645 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.1365878724108635) | > loader_time: 0.0046 (0.004696161718433085)  --> TIME: 2025-02-24 09:42:44 -- STEP: 7734/36938 -- GLOBAL_STEP: 266300 | > loss_text_ce: 0.04090297222137451 (0.04293090288166305) | > loss_mel_ce: 1.8043866157531738 (nan) | > loss: 0.007322578225284815 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13661529800828412) | > loader_time: 0.0045 (0.004696715670344702)  --> TIME: 2025-02-24 09:42:54 -- STEP: 7784/36938 -- GLOBAL_STEP: 266350 | > loss_text_ce: 0.031280767172575 (0.04293028973875711) | > loss_mel_ce: 3.203679323196411 (nan) | > loss: 0.012837143614888191 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13664333150173671) | > loader_time: 0.0052 (0.004697281894066367)  --> TIME: 2025-02-24 09:43:03 -- STEP: 7834/36938 -- GLOBAL_STEP: 266400 | > loss_text_ce: 0.05024530366063118 (0.0429276364418221) | > loss_mel_ce: 3.706854820251465 (nan) | > loss: 0.014909128658473492 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13663865334834904) | > loader_time: 0.0046 (0.004697390956357651)  --> TIME: 2025-02-24 09:43:12 -- STEP: 7884/36938 -- GLOBAL_STEP: 266450 | > loss_text_ce: 0.037758298218250275 (0.042931982336631175) | > loss_mel_ce: 3.135183811187744 (nan) | > loss: 0.012591040693223476 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.1366634515262029) | > loader_time: 0.0047 (0.004697818613366889)  --> TIME: 2025-02-24 09:43:22 -- STEP: 7934/36938 -- GLOBAL_STEP: 266500 | > loss_text_ce: 0.04715913534164429 (0.04294037794140734) | > loss_mel_ce: 3.393773317337036 (nan) | > loss: 0.013654494658112526 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13664047986096248) | > loader_time: 0.0046 (0.004697600149044575)  --> TIME: 2025-02-24 09:43:32 -- STEP: 7984/36938 -- GLOBAL_STEP: 266550 | > loss_text_ce: 0.03564154729247093 (0.04294242397863392) | > loss_mel_ce: 2.6508750915527344 (nan) | > loss: 0.010660780593752861 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13670592604275975) | > loader_time: 0.0047 (0.004697996085296907)  --> TIME: 2025-02-24 09:43:40 -- STEP: 8034/36938 -- GLOBAL_STEP: 266600 | > loss_text_ce: 0.04031306132674217 (0.04294579554296334) | > loss_mel_ce: 3.087592363357544 (nan) | > loss: 0.012412323616445065 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13665628563802854) | > loader_time: 0.0046 (0.004697378518007469)  --> TIME: 2025-02-24 09:43:50 -- STEP: 8084/36938 -- GLOBAL_STEP: 266650 | > loss_text_ce: 0.03884897008538246 (0.04293698140819564) | > loss_mel_ce: 1.2835965156555176 (nan) | > loss: 0.005247800145298243 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13664855103514012) | > loader_time: 0.0045 (0.004696937848411549)  --> TIME: 2025-02-24 09:43:59 -- STEP: 8134/36938 -- GLOBAL_STEP: 266700 | > loss_text_ce: 0.04127219691872597 (0.042942848644385016) | > loss_mel_ce: 4.206848621368408 (nan) | > loss: 0.016857624053955078 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.1366390416919383) | > loader_time: 0.0045 (0.004696541111563667)  --> TIME: 2025-02-24 09:44:08 -- STEP: 8184/36938 -- GLOBAL_STEP: 266750 | > loss_text_ce: 0.04178216680884361 (0.04293712604274749) | > loss_mel_ce: 2.7549281120300293 (nan) | > loss: 0.01109805703163147 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.1366711275784971) | > loader_time: 0.0046 (0.004696592207877883)  --> TIME: 2025-02-24 09:44:18 -- STEP: 8234/36938 -- GLOBAL_STEP: 266800 | > loss_text_ce: 0.0463658943772316 (0.042935811125017216) | > loss_mel_ce: 2.9718477725982666 (nan) | > loss: 0.011977039277553558 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.13668132890970502) | > loader_time: 0.0046 (0.004696998110899293)  --> TIME: 2025-02-24 09:44:27 -- STEP: 8284/36938 -- GLOBAL_STEP: 266850 | > loss_text_ce: 0.04875806346535683 (0.04293283180615645) | > loss_mel_ce: 3.598447561264038 (nan) | > loss: 0.014473038725554943 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.1366625939343418) | > loader_time: 0.0046 (0.004696739836627896)  --> TIME: 2025-02-24 09:44:36 -- STEP: 8334/36938 -- GLOBAL_STEP: 266900 | > loss_text_ce: 0.04975135624408722 (0.042938022625764335) | > loss_mel_ce: 3.5654854774475098 (nan) | > loss: 0.014346178621053696 (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.1366673520293948) | > loader_time: 0.0048 (0.004696915783068545)  --> TIME: 2025-02-24 09:44:45 -- STEP: 8384/36938 -- GLOBAL_STEP: 266950 | > loss_text_ce: 0.045756325125694275 (0.042934389906457406) | > loss_mel_ce: 3.077108383178711 (nan) | > loss: 0.012392320670187473 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.1366426040141186) | > loader_time: 0.0046 (0.004696145881223329)  --> TIME: 2025-02-24 09:44:55 -- STEP: 8434/36938 -- GLOBAL_STEP: 267000 | > loss_text_ce: 0.04061920568346977 (0.04293199587491447) | > loss_mel_ce: 2.9652657508850098 (nan) | > loss: 0.011928115040063858 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13666993036127706) | > loader_time: 0.0047 (0.0046972220108524474)  --> TIME: 2025-02-24 09:45:05 -- STEP: 8484/36938 -- GLOBAL_STEP: 267050 | > loss_text_ce: 0.040190670639276505 (0.04292801224272262) | > loss_mel_ce: 0.2808837890625 (nan) | > loss: 0.0012741050450131297 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13670132861054246) | > loader_time: 0.0045 (0.004697537208603905)  --> TIME: 2025-02-24 09:45:14 -- STEP: 8534/36938 -- GLOBAL_STEP: 267100 | > loss_text_ce: 0.04161650687456131 (0.04293041284525012) | > loss_mel_ce: 0.104522705078125 (nan) | > loss: 0.0005799175123684108 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13670074696671616) | > loader_time: 0.0045 (0.004697519273983683)  --> TIME: 2025-02-24 09:45:23 -- STEP: 8584/36938 -- GLOBAL_STEP: 267150 | > loss_text_ce: 0.05359658598899841 (0.042938028659475505) | > loss_mel_ce: 2.2774136066436768 (nan) | > loss: 0.009250040166079998 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.13669501951889615) | > loader_time: 0.0054 (0.004697069706254672)  --> TIME: 2025-02-24 09:45:33 -- STEP: 8634/36938 -- GLOBAL_STEP: 267200 | > loss_text_ce: 0.04246386140584946 (0.04294546414986489) | > loss_mel_ce: 3.4215102195739746 (nan) | > loss: 0.01374592911452055 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.136717612251298) | > loader_time: 0.0045 (0.004697604037999072)  --> TIME: 2025-02-24 09:45:42 -- STEP: 8684/36938 -- GLOBAL_STEP: 267250 | > loss_text_ce: 0.03895382210612297 (0.04294884082089516) | > loss_mel_ce: 2.3485586643218994 (nan) | > loss: 0.009474256075918674 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1367228848491604) | > loader_time: 0.0045 (0.004697056367945323)  --> TIME: 2025-02-24 09:45:51 -- STEP: 8734/36938 -- GLOBAL_STEP: 267300 | > loss_text_ce: 0.03792627900838852 (0.04296194643672811) | > loss_mel_ce: 3.168346881866455 (nan) | > loss: 0.012723306193947792 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13671051309396498) | > loader_time: 0.0046 (0.004696548135214563)  --> TIME: 2025-02-24 09:46:01 -- STEP: 8784/36938 -- GLOBAL_STEP: 267350 | > loss_text_ce: 0.04870301112532616 (0.04296246876784073) | > loss_mel_ce: 4.330626964569092 (nan) | > loss: 0.017378294840455055 (nan) | > current_lr: 5e-06 | > step_time: 0.1903 (0.1367222046265842) | > loader_time: 0.0051 (0.004696195704255177)  --> TIME: 2025-02-24 09:46:10 -- STEP: 8834/36938 -- GLOBAL_STEP: 267400 | > loss_text_ce: 0.03344423696398735 (0.042961122448083304) | > loss_mel_ce: 2.5977540016174316 (nan) | > loss: 0.01044126320630312 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13671135726977987) | > loader_time: 0.0046 (0.004695699499396694)  --> TIME: 2025-02-24 09:46:19 -- STEP: 8884/36938 -- GLOBAL_STEP: 267450 | > loss_text_ce: 0.03978845849633217 (0.042949290229917635) | > loss_mel_ce: 3.0602619647979736 (nan) | > loss: 0.012301788665354252 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1366804121368272) | > loader_time: 0.0045 (0.004695154884171789)  --> TIME: 2025-02-24 09:46:28 -- STEP: 8934/36938 -- GLOBAL_STEP: 267500 | > loss_text_ce: 0.03474085405468941 (0.042947101601065366) | > loss_mel_ce: 3.8439927101135254 (nan) | > loss: 0.015391801483929157 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13666400478167523) | > loader_time: 0.0046 (0.004694930306744853)  --> TIME: 2025-02-24 09:46:37 -- STEP: 8984/36938 -- GLOBAL_STEP: 267550 | > loss_text_ce: 0.046490974724292755 (0.042939680461578644) | > loss_mel_ce: 3.23681902885437 (nan) | > loss: 0.01302900817245245 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.1366466382254692) | > loader_time: 0.0047 (0.004694454657214107)  --> TIME: 2025-02-24 09:46:46 -- STEP: 9034/36938 -- GLOBAL_STEP: 267600 | > loss_text_ce: 0.050699084997177124 (0.042942312629371884) | > loss_mel_ce: 3.844700574874878 (nan) | > loss: 0.015457935631275177 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13660768027429163) | > loader_time: 0.0046 (0.004694187353479161)  --> TIME: 2025-02-24 09:46:55 -- STEP: 9084/36938 -- GLOBAL_STEP: 267650 | > loss_text_ce: 0.04531827196478844 (0.042943426462006086) | > loss_mel_ce: 2.205019235610962 (nan) | > loss: 0.008929912000894547 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13659996442572273) | > loader_time: 0.0046 (0.004694251828338561)  --> TIME: 2025-02-24 09:47:04 -- STEP: 9134/36938 -- GLOBAL_STEP: 267700 | > loss_text_ce: 0.0488157756626606 (0.04294317285096938) | > loss_mel_ce: 3.0102908611297607 (nan) | > loss: 0.01213931292295456 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13657818684725198) | > loader_time: 0.0045 (0.004693734037659501)  --> TIME: 2025-02-24 09:47:14 -- STEP: 9184/36938 -- GLOBAL_STEP: 267750 | > loss_text_ce: 0.0421573668718338 (0.04293928233738704) | > loss_mel_ce: 3.353296995162964 (nan) | > loss: 0.013474025763571262 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13659847667927566) | > loader_time: 0.0052 (0.004693956766602074)  --> TIME: 2025-02-24 09:47:23 -- STEP: 9234/36938 -- GLOBAL_STEP: 267800 | > loss_text_ce: 0.04911885783076286 (0.04293864009285041) | > loss_mel_ce: 3.5027997493743896 (nan) | > loss: 0.014094915241003036 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.1365975643975517) | > loader_time: 0.0056 (0.0046942584928248805)  --> TIME: 2025-02-24 09:47:33 -- STEP: 9284/36938 -- GLOBAL_STEP: 267850 | > loss_text_ce: 0.039739981293678284 (0.04292773183407923) | > loss_mel_ce: 3.292375326156616 (nan) | > loss: 0.013222681358456612 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13661641075096526) | > loader_time: 0.0045 (0.004694068755429678)  --> TIME: 2025-02-24 09:47:42 -- STEP: 9334/36938 -- GLOBAL_STEP: 267900 | > loss_text_ce: 0.038731422275304794 (0.04292484249828661) | > loss_mel_ce: 3.3461267948150635 (nan) | > loss: 0.013431977480649948 (nan) | > current_lr: 5e-06 | > step_time: 0.1567 (0.13661474892432612) | > loader_time: 0.0046 (0.004693951779421346)  --> TIME: 2025-02-24 09:47:51 -- STEP: 9384/36938 -- GLOBAL_STEP: 267950 | > loss_text_ce: 0.04667503759264946 (0.04292304648776391) | > loss_mel_ce: 3.6858484745025635 (nan) | > loss: 0.014811601489782333 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.13661716135261662) | > loader_time: 0.0048 (0.004693707694178063)  --> TIME: 2025-02-24 09:48:01 -- STEP: 9434/36938 -- GLOBAL_STEP: 268000 | > loss_text_ce: 0.040087055414915085 (0.0429154372504955) | > loss_mel_ce: 3.322687864303589 (nan) | > loss: 0.01334434561431408 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13661690826812334) | > loader_time: 0.0046 (0.004693317544745895)  --> TIME: 2025-02-24 09:48:10 -- STEP: 9484/36938 -- GLOBAL_STEP: 268050 | > loss_text_ce: 0.04860047250986099 (0.04291468131484) | > loss_mel_ce: 2.3331594467163086 (nan) | > loss: 0.009451428428292274 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13661908718766416) | > loader_time: 0.0067 (0.004694137579275578)  --> TIME: 2025-02-24 09:48:19 -- STEP: 9534/36938 -- GLOBAL_STEP: 268100 | > loss_text_ce: 0.04698876291513443 (0.042921350875122) | > loss_mel_ce: 3.044849395751953 (nan) | > loss: 0.012269199825823307 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.1366009232331207) | > loader_time: 0.0046 (0.00469447647672495)  --> TIME: 2025-02-24 09:48:28 -- STEP: 9584/36938 -- GLOBAL_STEP: 268150 | > loss_text_ce: 0.030035104602575302 (0.042917051240323835) | > loss_mel_ce: 2.3228862285614014 (nan) | > loss: 0.009336989372968674 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13660507839490307) | > loader_time: 0.0046 (0.0046946610602393414)  --> TIME: 2025-02-24 09:48:38 -- STEP: 9634/36938 -- GLOBAL_STEP: 268200 | > loss_text_ce: 0.040858715772628784 (0.042913966222987494) | > loss_mel_ce: 2.3733084201812744 (nan) | > loss: 0.009580029174685478 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13659778209501802) | > loader_time: 0.0045 (0.0046944480133373654)  --> TIME: 2025-02-24 09:48:47 -- STEP: 9684/36938 -- GLOBAL_STEP: 268250 | > loss_text_ce: 0.04475700855255127 (0.04290786647150913) | > loss_mel_ce: 3.5425634384155273 (nan) | > loss: 0.014235398732125759 (nan) | > current_lr: 5e-06 | > step_time: 0.1854 (0.1366160306887215) | > loader_time: 0.005 (0.004694875632864761)  --> TIME: 2025-02-24 09:48:56 -- STEP: 9734/36938 -- GLOBAL_STEP: 268300 | > loss_text_ce: 0.03254367783665657 (0.042912214653149844) | > loss_mel_ce: 3.374023675918579 (nan) | > loss: 0.01351812481880188 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13660626130253992) | > loader_time: 0.0046 (0.004695042487113731)  --> TIME: 2025-02-24 09:49:05 -- STEP: 9784/36938 -- GLOBAL_STEP: 268350 | > loss_text_ce: 0.023644205182790756 (0.042917896139904696) | > loss_mel_ce: 2.913217782974243 (nan) | > loss: 0.011654214933514595 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1365694883945824) | > loader_time: 0.0045 (0.004695418031120167)  --> TIME: 2025-02-24 09:49:15 -- STEP: 9834/36938 -- GLOBAL_STEP: 268400 | > loss_text_ce: 0.034537509083747864 (0.04291560065324395) | > loss_mel_ce: 1.88751220703125 (nan) | > loss: 0.007627182174474001 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13656919211518984) | > loader_time: 0.0046 (0.0046954867993522385)  --> TIME: 2025-02-24 09:49:24 -- STEP: 9884/36938 -- GLOBAL_STEP: 268450 | > loss_text_ce: 0.025759590789675713 (0.042921153131705955) | > loss_mel_ce: 1.6604115962982178 (nan) | > loss: 0.006691155955195427 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13657356497731066) | > loader_time: 0.0045 (0.004695944846858401)  --> TIME: 2025-02-24 09:49:33 -- STEP: 9934/36938 -- GLOBAL_STEP: 268500 | > loss_text_ce: 0.06749116629362106 (0.042923065375398826) | > loss_mel_ce: 3.253424882888794 (nan) | > loss: 0.013178238645195961 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.1365423748280558) | > loader_time: 0.0046 (0.004695591730732605)  --> TIME: 2025-02-24 09:49:42 -- STEP: 9984/36938 -- GLOBAL_STEP: 268550 | > loss_text_ce: 0.03925619274377823 (0.04291809947468366) | > loss_mel_ce: 2.32537841796875 (nan) | > loss: 0.009383470751345158 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13652856206187142) | > loader_time: 0.0046 (0.004695122846617163)  --> TIME: 2025-02-24 09:49:51 -- STEP: 10034/36938 -- GLOBAL_STEP: 268600 | > loss_text_ce: 0.03728353977203369 (0.04291522159449624) | > loss_mel_ce: 2.447244167327881 (nan) | > loss: 0.009859236888587475 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13652433537857186) | > loader_time: 0.0046 (0.004694948900689276)  --> TIME: 2025-02-24 09:50:00 -- STEP: 10084/36938 -- GLOBAL_STEP: 268650 | > loss_text_ce: 0.04817856848239899 (0.042913300490142955) | > loss_mel_ce: 3.7709174156188965 (nan) | > loss: 0.015155144035816193 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1365059161886064) | > loader_time: 0.0044 (0.004694304547580373)  --> TIME: 2025-02-24 09:50:10 -- STEP: 10134/36938 -- GLOBAL_STEP: 268700 | > loss_text_ce: 0.0556037463247776 (0.04290648636191103) | > loss_mel_ce: 4.552734375 (nan) | > loss: 0.018287058919668198 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13652654147745746) | > loader_time: 0.0055 (0.00469407010459563)  --> TIME: 2025-02-24 09:50:19 -- STEP: 10184/36938 -- GLOBAL_STEP: 268750 | > loss_text_ce: 0.043674033135175705 (0.04291009929209299) | > loss_mel_ce: 3.5851926803588867 (nan) | > loss: 0.01440026517957449 (nan) | > current_lr: 5e-06 | > step_time: 0.1723 (0.1365326132394161) | > loader_time: 0.0064 (0.004694264724511212)  --> TIME: 2025-02-24 09:50:28 -- STEP: 10234/36938 -- GLOBAL_STEP: 268800 | > loss_text_ce: 0.03363923355937004 (0.042917244651864486) | > loss_mel_ce: 3.483980655670166 (nan) | > loss: 0.013958809897303581 (nan) | > current_lr: 5e-06 | > step_time: 0.1901 (0.136524959469718) | > loader_time: 0.0049 (0.00469413231377588)  --> TIME: 2025-02-24 09:50:38 -- STEP: 10284/36938 -- GLOBAL_STEP: 268850 | > loss_text_ce: 0.04594234749674797 (0.04291692899856326) | > loss_mel_ce: 0.06829833984375 (nan) | > loss: 0.0004533361061476171 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.1365185720778685) | > loader_time: 0.0046 (0.004694160692703884)  --> TIME: 2025-02-24 09:50:47 -- STEP: 10334/36938 -- GLOBAL_STEP: 268900 | > loss_text_ce: 0.047842442989349365 (0.042917777834547234) | > loss_mel_ce: 2.847102642059326 (nan) | > loss: 0.011487877927720547 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13650160537643735) | > loader_time: 0.0045 (0.0046941208521033145)  --> TIME: 2025-02-24 09:50:56 -- STEP: 10384/36938 -- GLOBAL_STEP: 268950 | > loss_text_ce: 0.048028357326984406 (0.042909559277973554) | > loss_mel_ce: 1.7423118352890015 (nan) | > loss: 0.007104524876922369 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13649803875315578) | > loader_time: 0.0044 (0.004694081211494214)  --> TIME: 2025-02-24 09:51:05 -- STEP: 10434/36938 -- GLOBAL_STEP: 269000 | > loss_text_ce: 0.03861596807837486 (0.04289746178222715) | > loss_mel_ce: 3.349559783935547 (nan) | > loss: 0.013445142656564713 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13650087954236217) | > loader_time: 0.0047 (0.004693577658382924)  --> TIME: 2025-02-24 09:51:14 -- STEP: 10484/36938 -- GLOBAL_STEP: 269050 | > loss_text_ce: 0.058685313910245895 (0.0428952648867072) | > loss_mel_ce: 6.21875 (nan) | > loss: 0.0249104592949152 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1364848905281003) | > loader_time: 0.0045 (0.004693057054056121)  --> TIME: 2025-02-24 09:51:24 -- STEP: 10534/36938 -- GLOBAL_STEP: 269100 | > loss_text_ce: 0.047438401728868484 (0.04288894974074763) | > loss_mel_ce: 1.3800092935562134 (nan) | > loss: 0.00566447526216507 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13650695182622166) | > loader_time: 0.0045 (0.00469279379778579)  --> TIME: 2025-02-24 09:51:33 -- STEP: 10584/36938 -- GLOBAL_STEP: 269150 | > loss_text_ce: 0.04392460361123085 (0.04288637061761425) | > loss_mel_ce: 3.3179919719696045 (nan) | > loss: 0.01334093976765871 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13650472278587816) | > loader_time: 0.0052 (0.004692511989235992)  --> TIME: 2025-02-24 09:51:43 -- STEP: 10634/36938 -- GLOBAL_STEP: 269200 | > loss_text_ce: 0.03910155966877937 (0.04287872202556062) | > loss_mel_ce: 3.542792320251465 (nan) | > loss: 0.014213865622878075 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13652232055581595) | > loader_time: 0.0046 (0.004692878650502448)  --> TIME: 2025-02-24 09:51:52 -- STEP: 10684/36938 -- GLOBAL_STEP: 269250 | > loss_text_ce: 0.04928271844983101 (0.042879927929477785) | > loss_mel_ce: 3.1704137325286865 (nan) | > loss: 0.012776574119925499 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13651385083264628) | > loader_time: 0.0045 (0.004692635188965018)  --> TIME: 2025-02-24 09:52:01 -- STEP: 10734/36938 -- GLOBAL_STEP: 269300 | > loss_text_ce: 0.04951552674174309 (0.042874773026282696) | > loss_mel_ce: 2.7254655361175537 (nan) | > loss: 0.011011830531060696 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13651291125087717) | > loader_time: 0.0047 (0.004692624018182095)  --> TIME: 2025-02-24 09:52:10 -- STEP: 10784/36938 -- GLOBAL_STEP: 269350 | > loss_text_ce: 0.03328021615743637 (0.042871350062460636) | > loss_mel_ce: 3.741414785385132 (nan) | > loss: 0.01497894898056984 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.1365052489038747) | > loader_time: 0.0046 (0.004692306725490714)  --> TIME: 2025-02-24 09:52:19 -- STEP: 10834/36938 -- GLOBAL_STEP: 269400 | > loss_text_ce: 0.060974158346652985 (0.04287341775944127) | > loss_mel_ce: 1.63934326171875 (nan) | > loss: 0.006747291423380375 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13649415553925523) | > loader_time: 0.0046 (0.004692087297578675)  --> TIME: 2025-02-24 09:52:28 -- STEP: 10884/36938 -- GLOBAL_STEP: 269450 | > loss_text_ce: 0.04428238794207573 (0.04287166854215056) | > loss_mel_ce: 3.4927713871002197 (nan) | > loss: 0.01403592899441719 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.1364713756532541) | > loader_time: 0.0045 (0.004692168719450478)  --> TIME: 2025-02-24 09:52:38 -- STEP: 10934/36938 -- GLOBAL_STEP: 269500 | > loss_text_ce: 0.042116954922676086 (0.042867931398577426) | > loss_mel_ce: 2.708543539047241 (nan) | > loss: 0.010915319435298443 (nan) | > current_lr: 5e-06 | > step_time: 0.1343 (0.13647695365718765) | > loader_time: 0.0047 (0.004691978553672913)  --> TIME: 2025-02-24 09:52:47 -- STEP: 10984/36938 -- GLOBAL_STEP: 269550 | > loss_text_ce: 0.046498145908117294 (0.04286694524442432) | > loss_mel_ce: 0.053070068359375 (nan) | > loss: 0.00039511200156994164 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13647968223819068) | > loader_time: 0.0045 (0.004692113212834138)  --> TIME: 2025-02-24 09:52:56 -- STEP: 11034/36938 -- GLOBAL_STEP: 269600 | > loss_text_ce: 0.04663052782416344 (0.0428644255643825) | > loss_mel_ce: 3.540037155151367 (nan) | > loss: 0.014232809655368328 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.1364790932389304) | > loader_time: 0.0048 (0.004691771305361523)  --> TIME: 2025-02-24 09:53:06 -- STEP: 11084/36938 -- GLOBAL_STEP: 269650 | > loss_text_ce: 0.042625393718481064 (0.04286217171833604) | > loss_mel_ce: 3.3409411907196045 (nan) | > loss: 0.01342685241252184 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.13647690822999878) | > loader_time: 0.0046 (0.004691315402798797)  --> TIME: 2025-02-24 09:53:15 -- STEP: 11134/36938 -- GLOBAL_STEP: 269700 | > loss_text_ce: 0.04947895556688309 (0.04286231535363267) | > loss_mel_ce: 3.1401045322418213 (nan) | > loss: 0.012657077983021736 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1364774355795005) | > loader_time: 0.0046 (0.004691500733993825)  --> TIME: 2025-02-24 09:53:24 -- STEP: 11184/36938 -- GLOBAL_STEP: 269750 | > loss_text_ce: 0.03738611564040184 (0.042858334176508334) | > loss_mel_ce: 2.6847338676452637 (nan) | > loss: 0.010802064090967178 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13645925301424272) | > loader_time: 0.0045 (0.004691249375179628)  --> TIME: 2025-02-24 09:53:33 -- STEP: 11234/36938 -- GLOBAL_STEP: 269800 | > loss_text_ce: 0.04156064987182617 (0.04286315259912516) | > loss_mel_ce: 3.33656644821167 (nan) | > loss: 0.013405267149209976 (nan) | > current_lr: 5e-06 | > step_time: 0.1893 (0.13643769725947616) | > loader_time: 0.0048 (0.004690890679776281)  --> TIME: 2025-02-24 09:53:42 -- STEP: 11284/36938 -- GLOBAL_STEP: 269850 | > loss_text_ce: 0.04182860627770424 (0.04285249565856777) | > loss_mel_ce: 3.726087808609009 (nan) | > loss: 0.014952050521969795 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13644418508099648) | > loader_time: 0.0047 (0.00469077402545563)  --> TIME: 2025-02-24 09:53:52 -- STEP: 11334/36938 -- GLOBAL_STEP: 269900 | > loss_text_ce: 0.04519132152199745 (0.04284846854690513) | > loss_mel_ce: 3.2716593742370605 (nan) | > loss: 0.013162106275558472 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.13647536070695612) | > loader_time: 0.0046 (0.00469100153465133)  --> TIME: 2025-02-24 09:54:02 -- STEP: 11384/36938 -- GLOBAL_STEP: 269950 | > loss_text_ce: 0.047743089497089386 (0.0428470875037893) | > loss_mel_ce: 3.3872451782226562 (nan) | > loss: 0.013630906119942665 (nan) | > current_lr: 5e-06 | > step_time: 0.1588 (0.13648723877406194) | > loader_time: 0.0053 (0.004691439661775301)  --> TIME: 2025-02-24 09:54:11 -- STEP: 11434/36938 -- GLOBAL_STEP: 270000 | > loss_text_ce: 0.0433897003531456 (0.04284416628970233) | > loss_mel_ce: 2.555629253387451 (nan) | > loss: 0.010313568636775017 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13648853190236743) | > loader_time: 0.0067 (0.00469136924119772) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_270000.pth  --> TIME: 2025-02-24 09:54:26 -- STEP: 11484/36938 -- GLOBAL_STEP: 270050 | > loss_text_ce: 0.054290998727083206 (0.04284427134879397) | > loss_mel_ce: 3.3077759742736816 (nan) | > loss: 0.013341536745429039 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.1364751907019448) | > loader_time: 0.0046 (0.004691657954415007)  --> TIME: 2025-02-24 09:54:36 -- STEP: 11534/36938 -- GLOBAL_STEP: 270100 | > loss_text_ce: 0.040120333433151245 (0.042841885838343106) | > loss_mel_ce: 3.503657579421997 (nan) | > loss: 0.014062611386179924 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13650327779317806) | > loader_time: 0.0046 (0.004691415339920236)  --> TIME: 2025-02-24 09:54:45 -- STEP: 11584/36938 -- GLOBAL_STEP: 270150 | > loss_text_ce: 0.04977639392018318 (0.04284063354681271) | > loss_mel_ce: 3.4813575744628906 (nan) | > loss: 0.01401243731379509 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13651355030786946) | > loader_time: 0.0046 (0.004691603495765158)  --> TIME: 2025-02-24 09:54:55 -- STEP: 11634/36938 -- GLOBAL_STEP: 270200 | > loss_text_ce: 0.04464711993932724 (0.042842678889985844) | > loss_mel_ce: 3.3347771167755127 (nan) | > loss: 0.013410414569079876 (nan) | > current_lr: 5e-06 | > step_time: 0.1878 (0.13652810315526792) | > loader_time: 0.0048 (0.004691618239324275)  --> TIME: 2025-02-24 09:55:04 -- STEP: 11684/36938 -- GLOBAL_STEP: 270250 | > loss_text_ce: 0.052250832319259644 (0.042843228442934395) | > loss_mel_ce: 3.7185754776000977 (nan) | > loss: 0.014963597059249878 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13652568594391593) | > loader_time: 0.0046 (0.00469128771703241)  --> TIME: 2025-02-24 09:55:13 -- STEP: 11734/36938 -- GLOBAL_STEP: 270300 | > loss_text_ce: 0.04919886589050293 (0.04284380821268926) | > loss_mel_ce: 0.01641845703125 (nan) | > loss: 0.000260386208537966 (nan) | > current_lr: 5e-06 | > step_time: 0.1077 (0.13652152622581182) | > loader_time: 0.0044 (0.004690873941896508)  --> TIME: 2025-02-24 09:55:23 -- STEP: 11784/36938 -- GLOBAL_STEP: 270350 | > loss_text_ce: 0.055745791643857956 (0.04283970658628199) | > loss_mel_ce: 2.530942678451538 (nan) | > loss: 0.010264637880027294 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13651680238216185) | > loader_time: 0.0046 (0.004690653518681965)  --> TIME: 2025-02-24 09:55:32 -- STEP: 11834/36938 -- GLOBAL_STEP: 270400 | > loss_text_ce: 0.04309249669313431 (0.042840602773073264) | > loss_mel_ce: 1.4147491455078125 (nan) | > loss: 0.005785086192190647 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1365184676955437) | > loader_time: 0.0046 (0.004691024416500786)  --> TIME: 2025-02-24 09:55:41 -- STEP: 11884/36938 -- GLOBAL_STEP: 270450 | > loss_text_ce: 0.047031451016664505 (0.04283436487461142) | > loss_mel_ce: 3.6556997299194336 (nan) | > loss: 0.01469337847083807 (nan) | > current_lr: 5e-06 | > step_time: 0.1873 (0.13652280722351384) | > loader_time: 0.0048 (0.004690812637973687)  --> TIME: 2025-02-24 09:55:51 -- STEP: 11934/36938 -- GLOBAL_STEP: 270500 | > loss_text_ce: 0.04313621297478676 (0.04283614939077762) | > loss_mel_ce: 0.8752380609512329 (nan) | > loss: 0.003644342767074704 (nan) | > current_lr: 5e-06 | > step_time: 0.1352 (0.13653877228173847) | > loader_time: 0.0049 (0.004690705381367569)  --> TIME: 2025-02-24 09:56:00 -- STEP: 11984/36938 -- GLOBAL_STEP: 270550 | > loss_text_ce: 0.040280718356370926 (0.0428254342262133) | > loss_mel_ce: 3.294412612915039 (nan) | > loss: 0.013232911005616188 (nan) | > current_lr: 5e-06 | > step_time: 0.1692 (0.13653221109680597) | > loader_time: 0.0055 (0.004690572818385608)  --> TIME: 2025-02-24 09:56:10 -- STEP: 12034/36938 -- GLOBAL_STEP: 270600 | > loss_text_ce: 0.037559568881988525 (0.04282708404403396) | > loss_mel_ce: 3.4483561515808105 (nan) | > loss: 0.013832999393343925 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13654775048522705) | > loader_time: 0.0046 (0.00469042544787325)  --> TIME: 2025-02-24 09:56:19 -- STEP: 12084/36938 -- GLOBAL_STEP: 270650 | > loss_text_ce: 0.041749510914087296 (0.04282659071105185) | > loss_mel_ce: 3.094500780105591 (nan) | > loss: 0.012445438653230667 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1365431699284024) | > loader_time: 0.0046 (0.004690448462825014)  --> TIME: 2025-02-24 09:56:28 -- STEP: 12134/36938 -- GLOBAL_STEP: 270700 | > loss_text_ce: 0.04035824537277222 (0.042819923586014934) | > loss_mel_ce: 3.543344497680664 (nan) | > loss: 0.014221043325960636 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13655473711621527) | > loader_time: 0.0046 (0.004690883323508491)  --> TIME: 2025-02-24 09:56:37 -- STEP: 12184/36938 -- GLOBAL_STEP: 270750 | > loss_text_ce: 0.04477974772453308 (0.04282326395705949) | > loss_mel_ce: 2.6232106685638428 (nan) | > loss: 0.010587263852357864 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.136540164543623) | > loader_time: 0.0046 (0.004691071813230227)  --> TIME: 2025-02-24 09:56:47 -- STEP: 12234/36938 -- GLOBAL_STEP: 270800 | > loss_text_ce: 0.045917414128780365 (0.04282671873327869) | > loss_mel_ce: 3.05047345161438 (nan) | > loss: 0.012287266552448273 (nan) | > current_lr: 5e-06 | > step_time: 0.1518 (0.1365358355155626) | > loader_time: 0.0053 (0.0046913821809877075)  --> TIME: 2025-02-24 09:56:56 -- STEP: 12284/36938 -- GLOBAL_STEP: 270850 | > loss_text_ce: 0.04071369394659996 (0.04282910953063887) | > loss_mel_ce: 3.482424259185791 (nan) | > loss: 0.01398070715367794 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13654621339939) | > loader_time: 0.0046 (0.0046912124668569985)  --> TIME: 2025-02-24 09:57:06 -- STEP: 12334/36938 -- GLOBAL_STEP: 270900 | > loss_text_ce: 0.03974997252225876 (0.042818268260417125) | > loss_mel_ce: 3.1288020610809326 (nan) | > loss: 0.01257361937314272 (nan) | > current_lr: 5e-06 | > step_time: 0.1572 (0.1365493245076108) | > loader_time: 0.0047 (0.004691046332354233)  --> TIME: 2025-02-24 09:57:15 -- STEP: 12384/36938 -- GLOBAL_STEP: 270950 | > loss_text_ce: 0.04175316542387009 (0.042818850263843806) | > loss_mel_ce: 3.4872629642486572 (nan) | > loss: 0.01400403305888176 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13656609509265416) | > loader_time: 0.0048 (0.004690916077727209)  --> TIME: 2025-02-24 09:57:24 -- STEP: 12434/36938 -- GLOBAL_STEP: 271000 | > loss_text_ce: 0.04447625204920769 (0.042824350749405465) | > loss_mel_ce: 3.345203399658203 (nan) | > loss: 0.013451110571622849 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13656505892969767) | > loader_time: 0.0048 (0.004690933135500387)  --> TIME: 2025-02-24 09:57:34 -- STEP: 12484/36938 -- GLOBAL_STEP: 271050 | > loss_text_ce: 0.03681458532810211 (0.04283270290335103) | > loss_mel_ce: 0.46484375 (nan) | > loss: 0.0019907078240066767 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13655991888092123) | > loader_time: 0.0045 (0.004691205453276848)  --> TIME: 2025-02-24 09:57:43 -- STEP: 12534/36938 -- GLOBAL_STEP: 271100 | > loss_text_ce: 0.048213452100753784 (0.042835373645513215) | > loss_mel_ce: 3.1649560928344727 (nan) | > loss: 0.012750674039125443 (nan) | > current_lr: 5e-06 | > step_time: 0.1507 (0.13657166757366024) | > loader_time: 0.0046 (0.004691048369979089)  --> TIME: 2025-02-24 09:57:52 -- STEP: 12584/36938 -- GLOBAL_STEP: 271150 | > loss_text_ce: 0.04178377985954285 (0.04282851981981065) | > loss_mel_ce: 1.455010175704956 (nan) | > loss: 0.00593965919688344 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.1365738181801109) | > loader_time: 0.0045 (0.004691204957270903)  --> TIME: 2025-02-24 09:58:02 -- STEP: 12634/36938 -- GLOBAL_STEP: 271200 | > loss_text_ce: 0.026195639744400978 (0.04283261228447445) | > loss_mel_ce: 1.6992151737213135 (nan) | > loss: 0.006846868898719549 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.1365592849203654) | > loader_time: 0.0046 (0.004691061159091357)  --> TIME: 2025-02-24 09:58:11 -- STEP: 12684/36938 -- GLOBAL_STEP: 271250 | > loss_text_ce: 0.028769752010703087 (0.042825773790598036) | > loss_mel_ce: 3.5220556259155273 (nan) | > loss: 0.014090578071773052 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13654597596541737) | > loader_time: 0.0047 (0.00469127320897427)  --> TIME: 2025-02-24 09:58:20 -- STEP: 12734/36938 -- GLOBAL_STEP: 271300 | > loss_text_ce: 0.04000755399465561 (0.042827739161255755) | > loss_mel_ce: 3.8561949729919434 (nan) | > loss: 0.015461122617125511 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13655532061707304) | > loader_time: 0.0046 (0.004691066482817171)  --> TIME: 2025-02-24 09:58:30 -- STEP: 12784/36938 -- GLOBAL_STEP: 271350 | > loss_text_ce: 0.047019679099321365 (0.04282589159034334) | > loss_mel_ce: 2.860492706298828 (nan) | > loss: 0.011537748388946056 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13658440734104751) | > loader_time: 0.0045 (0.00469113153420642)  --> TIME: 2025-02-24 09:58:39 -- STEP: 12834/36938 -- GLOBAL_STEP: 271400 | > loss_text_ce: 0.03211117908358574 (0.04282542226359475) | > loss_mel_ce: 2.419074773788452 (nan) | > loss: 0.009726928547024727 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13657343404323155) | > loader_time: 0.0048 (0.004691053592324626)  --> TIME: 2025-02-24 09:58:48 -- STEP: 12884/36938 -- GLOBAL_STEP: 271450 | > loss_text_ce: 0.03419853746891022 (0.04283099172925149) | > loss_mel_ce: 3.4471726417541504 (nan) | > loss: 0.013814966194331646 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13657835439583116) | > loader_time: 0.0046 (0.004691080105523094)  --> TIME: 2025-02-24 09:58:58 -- STEP: 12934/36938 -- GLOBAL_STEP: 271500 | > loss_text_ce: 0.04766133800148964 (0.04283296100952826) | > loss_mel_ce: 3.3368802070617676 (nan) | > loss: 0.013430721126496792 (nan) | > current_lr: 5e-06 | > step_time: 0.1438 (0.13657857823523048) | > loader_time: 0.0048 (0.004691159797079782)  --> TIME: 2025-02-24 09:59:07 -- STEP: 12984/36938 -- GLOBAL_STEP: 271550 | > loss_text_ce: 0.06371233612298965 (0.042832614541313804) | > loss_mel_ce: 4.721354007720947 (nan) | > loss: 0.018988357856869698 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1366003857339082) | > loader_time: 0.0045 (0.004691233843546641)  --> TIME: 2025-02-24 09:59:17 -- STEP: 13034/36938 -- GLOBAL_STEP: 271600 | > loss_text_ce: 0.05128033086657524 (0.0428272898819682) | > loss_mel_ce: 2.817201852798462 (nan) | > loss: 0.011382865719497204 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13660816788142924) | > loader_time: 0.0046 (0.004691418757079422)  --> TIME: 2025-02-24 09:59:26 -- STEP: 13084/36938 -- GLOBAL_STEP: 271650 | > loss_text_ce: 0.035365279763936996 (0.04282553779103385) | > loss_mel_ce: 3.186574935913086 (nan) | > loss: 0.012785478495061398 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13661245571222183) | > loader_time: 0.0046 (0.004691375792482089)  --> TIME: 2025-02-24 09:59:36 -- STEP: 13134/36938 -- GLOBAL_STEP: 271700 | > loss_text_ce: 0.04450869560241699 (0.042827101662379) | > loss_mel_ce: 2.73492431640625 (nan) | > loss: 0.011029496788978577 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13661919092866998) | > loader_time: 0.0047 (0.004691482225618219)  --> TIME: 2025-02-24 09:59:45 -- STEP: 13184/36938 -- GLOBAL_STEP: 271750 | > loss_text_ce: 0.04276541247963905 (0.04282810875488447) | > loss_mel_ce: 3.0907864570617676 (nan) | > loss: 0.012434730306267738 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.13663930400699678) | > loader_time: 0.0057 (0.004691643640398993)  --> TIME: 2025-02-24 09:59:54 -- STEP: 13234/36938 -- GLOBAL_STEP: 271800 | > loss_text_ce: 0.05128927901387215 (0.04282343996482116) | > loss_mel_ce: 2.8714101314544678 (nan) | > loss: 0.011598014272749424 (nan) | > current_lr: 5e-06 | > step_time: 0.1352 (0.136634861160049) | > loader_time: 0.0046 (0.004691673816826772)  --> TIME: 2025-02-24 10:00:04 -- STEP: 13284/36938 -- GLOBAL_STEP: 271850 | > loss_text_ce: 0.04622387886047363 (0.04282168552184974) | > loss_mel_ce: 2.8971457481384277 (nan) | > loss: 0.011680038645863533 (nan) | > current_lr: 5e-06 | > step_time: 0.1748 (0.13665728316325754) | > loader_time: 0.0051 (0.004692018911516197)  --> TIME: 2025-02-24 10:00:13 -- STEP: 13334/36938 -- GLOBAL_STEP: 271900 | > loss_text_ce: 0.03344395011663437 (0.04281977791646768) | > loss_mel_ce: 2.0747780799865723 (nan) | > loss: 0.008365960791707039 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13664952550349957) | > loader_time: 0.0046 (0.0046921274517901105)  --> TIME: 2025-02-24 10:00:23 -- STEP: 13384/36938 -- GLOBAL_STEP: 271950 | > loss_text_ce: 0.05009709298610687 (0.04282490441761307) | > loss_mel_ce: 2.625244140625 (nan) | > loss: 0.010616433806717396 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13666730062803423) | > loader_time: 0.0047 (0.004692212700202528)  --> TIME: 2025-02-24 10:00:32 -- STEP: 13434/36938 -- GLOBAL_STEP: 272000 | > loss_text_ce: 0.03457847237586975 (0.04282518443283386) | > loss_mel_ce: 3.6013665199279785 (nan) | > loss: 0.014428353868424892 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13666458211165944) | > loader_time: 0.0053 (0.0046921231233063845)  --> TIME: 2025-02-24 10:00:41 -- STEP: 13484/36938 -- GLOBAL_STEP: 272050 | > loss_text_ce: 0.03853023052215576 (0.04282214821554459) | > loss_mel_ce: 2.170999050140381 (nan) | > loss: 0.008767974562942982 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13665010306008427) | > loader_time: 0.0045 (0.004692462317422324)  --> TIME: 2025-02-24 10:00:51 -- STEP: 13534/36938 -- GLOBAL_STEP: 272100 | > loss_text_ce: 0.04593292623758316 (0.04281655028736296) | > loss_mel_ce: 2.71944260597229 (nan) | > loss: 0.010973713360726833 (nan) | > current_lr: 5e-06 | > step_time: 0.1848 (0.13664917736580579) | > loader_time: 0.0048 (0.004692604891637664)  --> TIME: 2025-02-24 10:01:00 -- STEP: 13584/36938 -- GLOBAL_STEP: 272150 | > loss_text_ce: 0.045549243688583374 (0.0428142579633977) | > loss_mel_ce: 3.1349425315856934 (nan) | > loss: 0.012620999477803707 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.1366487409572862) | > loader_time: 0.0047 (0.004692861378122697)  --> TIME: 2025-02-24 10:01:09 -- STEP: 13634/36938 -- GLOBAL_STEP: 272200 | > loss_text_ce: 0.04882822558283806 (0.04281877843158712) | > loss_mel_ce: 2.1655662059783936 (nan) | > loss: 0.008787279948592186 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.1366464783758193) | > loader_time: 0.0055 (0.004692950503329858)  --> TIME: 2025-02-24 10:01:18 -- STEP: 13684/36938 -- GLOBAL_STEP: 272250 | > loss_text_ce: 0.051126010715961456 (0.04282216254293163) | > loss_mel_ce: 3.9008290767669678 (nan) | > loss: 0.015682362020015717 (nan) | > current_lr: 5e-06 | > step_time: 0.1267 (0.13662889231300207) | > loader_time: 0.0046 (0.004693154928385253)  --> TIME: 2025-02-24 10:01:27 -- STEP: 13734/36938 -- GLOBAL_STEP: 272300 | > loss_text_ce: 0.044063400477170944 (0.042815174011046314) | > loss_mel_ce: 1.6499459743499756 (nan) | > loss: 0.006722259800881147 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.1366150144477314) | > loader_time: 0.0048 (0.004693282072384735)  --> TIME: 2025-02-24 10:01:37 -- STEP: 13784/36938 -- GLOBAL_STEP: 272350 | > loss_text_ce: 0.048824191093444824 (0.04281241175586456) | > loss_mel_ce: 3.703075408935547 (nan) | > loss: 0.014888491481542587 (nan) | > current_lr: 5e-06 | > step_time: 0.1571 (0.1366164031688794) | > loader_time: 0.0047 (0.004693910349819872)  --> TIME: 2025-02-24 10:01:46 -- STEP: 13834/36938 -- GLOBAL_STEP: 272400 | > loss_text_ce: 0.0289629977196455 (0.0428113386273149) | > loss_mel_ce: 2.4161343574523926 (nan) | > loss: 0.009702768176794052 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13660367136317328) | > loader_time: 0.0045 (0.004693663273654578)  --> TIME: 2025-02-24 10:01:55 -- STEP: 13884/36938 -- GLOBAL_STEP: 272450 | > loss_text_ce: 0.04004254564642906 (0.04281420760344775) | > loss_mel_ce: 3.889835834503174 (nan) | > loss: 0.015594757162034512 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13660187003111776) | > loader_time: 0.0046 (0.00469393479106269)  --> TIME: 2025-02-24 10:02:04 -- STEP: 13934/36938 -- GLOBAL_STEP: 272500 | > loss_text_ce: 0.03804333508014679 (0.04281471405115221) | > loss_mel_ce: 0.081939697265625 (nan) | > loss: 0.0004761231830343604 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13659320030592922) | > loader_time: 0.0045 (0.004693816326536132)  --> TIME: 2025-02-24 10:02:13 -- STEP: 13984/36938 -- GLOBAL_STEP: 272550 | > loss_text_ce: 0.044258277863264084 (0.04281232324793828) | > loss_mel_ce: 4.361019611358643 (nan) | > loss: 0.017481261864304543 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13658175452077398) | > loader_time: 0.0046 (0.004693544872565455)  --> TIME: 2025-02-24 10:02:22 -- STEP: 14034/36938 -- GLOBAL_STEP: 272600 | > loss_text_ce: 0.04916847497224808 (0.042812016333105836) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.1365740517868653) | > loader_time: 0.0046 (0.004693351139178962)  --> TIME: 2025-02-24 10:02:32 -- STEP: 14084/36938 -- GLOBAL_STEP: 272650 | > loss_text_ce: 0.03891882672905922 (0.042816354131505495) | > loss_mel_ce: 2.2807505130767822 (nan) | > loss: 0.009205036796629429 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13656968667201763) | > loader_time: 0.0044 (0.004693383538767006)  --> TIME: 2025-02-24 10:02:41 -- STEP: 14134/36938 -- GLOBAL_STEP: 272700 | > loss_text_ce: 0.03341643139719963 (0.04281309052858495) | > loss_mel_ce: 2.055047035217285 (nan) | > loss: 0.008287554606795311 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13655396140041753) | > loader_time: 0.0065 (0.004693613272332666)  --> TIME: 2025-02-24 10:02:50 -- STEP: 14184/36938 -- GLOBAL_STEP: 272750 | > loss_text_ce: 0.04418788477778435 (0.04281096053097008) | > loss_mel_ce: 3.21270489692688 (nan) | > loss: 0.012924178503453732 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13654455113935363) | > loader_time: 0.0049 (0.004693530117530844)  --> TIME: 2025-02-24 10:02:59 -- STEP: 14234/36938 -- GLOBAL_STEP: 272800 | > loss_text_ce: 0.04164920374751091 (0.04280746742314605) | > loss_mel_ce: 4.071075439453125 (nan) | > loss: 0.01632033661007881 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.13656209934556726) | > loader_time: 0.0048 (0.004693732798325642)  --> TIME: 2025-02-24 10:03:09 -- STEP: 14284/36938 -- GLOBAL_STEP: 272850 | > loss_text_ce: 0.04055365175008774 (0.04280085394430723) | > loss_mel_ce: 1.8254038095474243 (nan) | > loss: 0.007404593750834465 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.1365529134539919) | > loader_time: 0.0046 (0.004693403393572634)  --> TIME: 2025-02-24 10:03:18 -- STEP: 14334/36938 -- GLOBAL_STEP: 272900 | > loss_text_ce: 0.03537089750170708 (0.042801664678661436) | > loss_mel_ce: 2.7480337619781494 (nan) | > loss: 0.01104525662958622 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1365519556620634) | > loader_time: 0.0045 (0.004693099772795417)  --> TIME: 2025-02-24 10:03:27 -- STEP: 14384/36938 -- GLOBAL_STEP: 272950 | > loss_text_ce: 0.04594706371426582 (0.04280065765950911) | > loss_mel_ce: 3.3272979259490967 (nan) | > loss: 0.013385893777012825 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13655949013516913) | > loader_time: 0.0047 (0.004693058129810253)  --> TIME: 2025-02-24 10:03:37 -- STEP: 14434/36938 -- GLOBAL_STEP: 273000 | > loss_text_ce: 0.047725871205329895 (0.042796819281641946) | > loss_mel_ce: 3.3061914443969727 (nan) | > loss: 0.013309196569025517 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.13656315217804982) | > loader_time: 0.0049 (0.004693106830623881)  --> TIME: 2025-02-24 10:03:46 -- STEP: 14484/36938 -- GLOBAL_STEP: 273050 | > loss_text_ce: 0.03287893161177635 (0.042795307068933613) | > loss_mel_ce: 1.8922348022460938 (nan) | > loss: 0.007639340590685606 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1365601887725323) | > loader_time: 0.0046 (0.004693150602628829)  --> TIME: 2025-02-24 10:03:55 -- STEP: 14534/36938 -- GLOBAL_STEP: 273100 | > loss_text_ce: 0.04413313418626785 (0.042801347085459) | > loss_mel_ce: 3.7657251358032227 (nan) | > loss: 0.015118486247956753 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13657036030392516) | > loader_time: 0.0045 (0.004693404194805732)  --> TIME: 2025-02-24 10:04:05 -- STEP: 14584/36938 -- GLOBAL_STEP: 273150 | > loss_text_ce: 0.045406512916088104 (0.04279809612862557) | > loss_mel_ce: 3.011091709136963 (nan) | > loss: 0.012128962203860283 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1365762747419575) | > loader_time: 0.0047 (0.004693536822506836)  --> TIME: 2025-02-24 10:04:14 -- STEP: 14634/36938 -- GLOBAL_STEP: 273200 | > loss_text_ce: 0.04665695130825043 (0.04279706148410093) | > loss_mel_ce: 3.295283794403076 (nan) | > loss: 0.013261670246720314 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13656765138645074) | > loader_time: 0.0047 (0.004693402493930309)  --> TIME: 2025-02-24 10:04:23 -- STEP: 14684/36938 -- GLOBAL_STEP: 273250 | > loss_text_ce: 0.04228537902235985 (0.04279375590817329) | > loss_mel_ce: 3.0205674171447754 (nan) | > loss: 0.012154178693890572 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13655873583500294) | > loader_time: 0.0046 (0.004693291437979843)  --> TIME: 2025-02-24 10:04:32 -- STEP: 14734/36938 -- GLOBAL_STEP: 273300 | > loss_text_ce: 0.045627813786268234 (0.042793005074811735) | > loss_mel_ce: 4.204339981079102 (nan) | > loss: 0.016864951699972153 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13655782103522285) | > loader_time: 0.0046 (0.004693304989155717)  --> TIME: 2025-02-24 10:04:42 -- STEP: 14784/36938 -- GLOBAL_STEP: 273350 | > loss_text_ce: 0.03758632019162178 (0.04279821664392271) | > loss_mel_ce: 3.215970039367676 (nan) | > loss: 0.012910937890410423 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.1365481603132693) | > loader_time: 0.0047 (0.004693010362324777)  --> TIME: 2025-02-24 10:04:51 -- STEP: 14834/36938 -- GLOBAL_STEP: 273400 | > loss_text_ce: 0.04162484034895897 (0.0427925064888576) | > loss_mel_ce: 3.4612579345703125 (nan) | > loss: 0.013900329358875751 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.1365364998261591) | > loader_time: 0.0047 (0.00469295152744161)  --> TIME: 2025-02-24 10:05:00 -- STEP: 14884/36938 -- GLOBAL_STEP: 273450 | > loss_text_ce: 0.03508986532688141 (0.04279136874489504) | > loss_mel_ce: 2.6103055477142334 (nan) | > loss: 0.010497601702809334 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13654036660951704) | > loader_time: 0.0045 (0.004692981157274664)  --> TIME: 2025-02-24 10:05:09 -- STEP: 14934/36938 -- GLOBAL_STEP: 273500 | > loss_text_ce: 0.03414304926991463 (0.04279269765250795) | > loss_mel_ce: 3.145090103149414 (nan) | > loss: 0.012616004794836044 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.1365372116536644) | > loader_time: 0.0046 (0.004692969223861317)  --> TIME: 2025-02-24 10:05:19 -- STEP: 14984/36938 -- GLOBAL_STEP: 273550 | > loss_text_ce: 0.03716278076171875 (0.04278414843972428) | > loss_mel_ce: 1.064658522605896 (nan) | > loss: 0.004372307099401951 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1365351031409564) | > loader_time: 0.0047 (0.004692873102550639)  --> TIME: 2025-02-24 10:05:28 -- STEP: 15034/36938 -- GLOBAL_STEP: 273600 | > loss_text_ce: 0.039291862398386 (0.04278237807175537) | > loss_mel_ce: 3.802063226699829 (nan) | > loss: 0.015243473462760448 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.13654004828130914) | > loader_time: 0.0046 (0.00469298525758606)  --> TIME: 2025-02-24 10:05:37 -- STEP: 15084/36938 -- GLOBAL_STEP: 273650 | > loss_text_ce: 0.03772905096411705 (0.04277788617348618) | > loss_mel_ce: 2.67646861076355 (nan) | > loss: 0.010770626366138458 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.13654835622148123) | > loader_time: 0.0049 (0.004692954888744653)  --> TIME: 2025-02-24 10:05:47 -- STEP: 15134/36938 -- GLOBAL_STEP: 273700 | > loss_text_ce: 0.04449565336108208 (0.042776216522550446) | > loss_mel_ce: 3.384115695953369 (nan) | > loss: 0.013605601154267788 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13655148854387347) | > loader_time: 0.0046 (0.0046928597990045845)  --> TIME: 2025-02-24 10:05:56 -- STEP: 15184/36938 -- GLOBAL_STEP: 273750 | > loss_text_ce: 0.04234761744737625 (0.04277309493575884) | > loss_mel_ce: 1.7700306177139282 (nan) | > loss: 0.007191977929323912 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13655807883395255) | > loader_time: 0.0046 (0.004692735658807423)  --> TIME: 2025-02-24 10:06:06 -- STEP: 15234/36938 -- GLOBAL_STEP: 273800 | > loss_text_ce: 0.03521609306335449 (0.04276788217189561) | > loss_mel_ce: 3.2910706996917725 (nan) | > loss: 0.013199551962316036 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13656653538766866) | > loader_time: 0.0045 (0.004692557557010738)  --> TIME: 2025-02-24 10:06:15 -- STEP: 15284/36938 -- GLOBAL_STEP: 273850 | > loss_text_ce: 0.04400127753615379 (0.04276822093173233) | > loss_mel_ce: 3.904149055480957 (nan) | > loss: 0.0156672652810812 (nan) | > current_lr: 5e-06 | > step_time: 0.1756 (0.13656857693204671) | > loader_time: 0.0049 (0.004692507083426592)  --> TIME: 2025-02-24 10:06:24 -- STEP: 15334/36938 -- GLOBAL_STEP: 273900 | > loss_text_ce: 0.047508977353572845 (0.04276591595745753) | > loss_mel_ce: 3.5223984718322754 (nan) | > loss: 0.014166300185024738 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.13656526133058497) | > loader_time: 0.0046 (0.004692412408495627)  --> TIME: 2025-02-24 10:06:34 -- STEP: 15384/36938 -- GLOBAL_STEP: 273950 | > loss_text_ce: 0.04983789473772049 (0.04276787307419983) | > loss_mel_ce: 3.76458740234375 (nan) | > loss: 0.015136608853936195 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13656136266464844) | > loader_time: 0.0045 (0.004692183533374431)  --> TIME: 2025-02-24 10:06:43 -- STEP: 15434/36938 -- GLOBAL_STEP: 274000 | > loss_text_ce: 0.04152698442339897 (0.042758730314025196) | > loss_mel_ce: 1.7824294567108154 (nan) | > loss: 0.007237922865897417 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1365691195254776) | > loader_time: 0.0046 (0.004692032931304216)  --> TIME: 2025-02-24 10:06:52 -- STEP: 15484/36938 -- GLOBAL_STEP: 274050 | > loss_text_ce: 0.05088787525892258 (0.042763416327798905) | > loss_mel_ce: 2.501816987991333 (nan) | > loss: 0.010129781439900398 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13656365106161822) | > loader_time: 0.0045 (0.004691766125076164)  --> TIME: 2025-02-24 10:07:02 -- STEP: 15534/36938 -- GLOBAL_STEP: 274100 | > loss_text_ce: 0.037615563720464706 (0.042766477106765295) | > loss_mel_ce: 0.1148274764418602 (nan) | > loss: 0.0006049327203072608 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13656474671585428) | > loader_time: 0.0048 (0.004691661317421845)  --> TIME: 2025-02-24 10:07:11 -- STEP: 15584/36938 -- GLOBAL_STEP: 274150 | > loss_text_ce: 0.04136171564459801 (0.04276134297409479) | > loss_mel_ce: 2.5483522415161133 (nan) | > loss: 0.010276643559336662 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.1365619538011489) | > loader_time: 0.0046 (0.0046915191950494504)  --> TIME: 2025-02-24 10:07:20 -- STEP: 15634/36938 -- GLOBAL_STEP: 274200 | > loss_text_ce: 0.027868727222085 (0.04275886680265102) | > loss_mel_ce: 2.196946859359741 (nan) | > loss: 0.008828633464872837 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13656434276852109) | > loader_time: 0.0046 (0.0046914269647528725)  --> TIME: 2025-02-24 10:07:30 -- STEP: 15684/36938 -- GLOBAL_STEP: 274250 | > loss_text_ce: 0.046510618180036545 (0.04276155632236746) | > loss_mel_ce: 3.3800222873687744 (nan) | > loss: 0.013597354292869568 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1365603547988875) | > loader_time: 0.0045 (0.004691454668611751)  --> TIME: 2025-02-24 10:07:39 -- STEP: 15734/36938 -- GLOBAL_STEP: 274300 | > loss_text_ce: 0.04374213144183159 (0.04275563990279852) | > loss_mel_ce: 1.8672070503234863 (nan) | > loss: 0.007583132479339838 (nan) | > current_lr: 5e-06 | > step_time: 0.1515 (0.13656819627414382) | > loader_time: 0.0047 (0.004691218350944941)  --> TIME: 2025-02-24 10:07:48 -- STEP: 15784/36938 -- GLOBAL_STEP: 274350 | > loss_text_ce: 0.03282713145017624 (0.04275379442842094) | > loss_mel_ce: 3.2228047847747803 (nan) | > loss: 0.012919175438582897 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13655928074860388) | > loader_time: 0.0047 (0.00469113805544177)  --> TIME: 2025-02-24 10:07:57 -- STEP: 15834/36938 -- GLOBAL_STEP: 274400 | > loss_text_ce: 0.04197291284799576 (0.042752337279107676) | > loss_mel_ce: 3.7810871601104736 (nan) | > loss: 0.015170874074101448 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13655655044068016) | > loader_time: 0.0045 (0.004691007132179075)  --> TIME: 2025-02-24 10:08:07 -- STEP: 15884/36938 -- GLOBAL_STEP: 274450 | > loss_text_ce: 0.045041754841804504 (0.04274310500791866) | > loss_mel_ce: 3.182741165161133 (nan) | > loss: 0.012808662839233875 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.13655572377653724) | > loader_time: 0.0049 (0.00469087961487973)  --> TIME: 2025-02-24 10:08:16 -- STEP: 15934/36938 -- GLOBAL_STEP: 274500 | > loss_text_ce: 0.036968059837818146 (0.042739688361408984) | > loss_mel_ce: 3.30788516998291 (nan) | > loss: 0.013273227959871292 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13655343002756462) | > loader_time: 0.0047 (0.004690717645312128)  --> TIME: 2025-02-24 10:08:25 -- STEP: 15984/36938 -- GLOBAL_STEP: 274550 | > loss_text_ce: 0.04639244079589844 (0.042741623047778496) | > loss_mel_ce: 3.431370735168457 (nan) | > loss: 0.013800648041069508 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13653909728990035) | > loader_time: 0.0048 (0.004690476649397009)  --> TIME: 2025-02-24 10:08:34 -- STEP: 16034/36938 -- GLOBAL_STEP: 274600 | > loss_text_ce: 0.02146018296480179 (0.04274120152826527) | > loss_mel_ce: 1.6909399032592773 (nan) | > loss: 0.006795238703489304 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13652718752656032) | > loader_time: 0.0045 (0.004690224785035695)  --> TIME: 2025-02-24 10:08:43 -- STEP: 16084/36938 -- GLOBAL_STEP: 274650 | > loss_text_ce: 0.04625566676259041 (0.04273910003556078) | > loss_mel_ce: 3.82582426071167 (nan) | > loss: 0.015365397557616234 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13652005113496352) | > loader_time: 0.0046 (0.004690031452695993)  --> TIME: 2025-02-24 10:08:52 -- STEP: 16134/36938 -- GLOBAL_STEP: 274700 | > loss_text_ce: 0.03247370570898056 (0.042741852520615416) | > loss_mel_ce: 1.6945455074310303 (nan) | > loss: 0.006853251252323389 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13650335116768825) | > loader_time: 0.0045 (0.00468977243613322)  --> TIME: 2025-02-24 10:09:02 -- STEP: 16184/36938 -- GLOBAL_STEP: 274750 | > loss_text_ce: 0.0529404953122139 (0.04273968467786289) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13650408799673006) | > loader_time: 0.0045 (0.004689589091238266)  --> TIME: 2025-02-24 10:09:11 -- STEP: 16234/36938 -- GLOBAL_STEP: 274800 | > loss_text_ce: 0.03839440643787384 (0.042742244646392) | > loss_mel_ce: 0.7976440191268921 (nan) | > loss: 0.003317612921819091 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13649690889939134) | > loader_time: 0.0046 (0.004689342857831744)  --> TIME: 2025-02-24 10:09:20 -- STEP: 16284/36938 -- GLOBAL_STEP: 274850 | > loss_text_ce: 0.040370874106884 (0.04273943188301704) | > loss_mel_ce: 3.1284122467041016 (nan) | > loss: 0.012574536725878716 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.13649966240339126) | > loader_time: 0.0047 (0.004689211342909271)  --> TIME: 2025-02-24 10:09:29 -- STEP: 16334/36938 -- GLOBAL_STEP: 274900 | > loss_text_ce: 0.01688106171786785 (0.04273243258144242) | > loss_mel_ce: 1.415330410003662 (nan) | > loss: 0.0056833792477846146 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13648831910013876) | > loader_time: 0.0045 (0.004689097024972018)  --> TIME: 2025-02-24 10:09:39 -- STEP: 16384/36938 -- GLOBAL_STEP: 274950 | > loss_text_ce: 0.044127847999334335 (0.0427266675000056) | > loss_mel_ce: 2.7150912284851074 (nan) | > loss: 0.01094928290694952 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13649478854495073) | > loader_time: 0.0046 (0.00468894986261148)  --> TIME: 2025-02-24 10:09:48 -- STEP: 16434/36938 -- GLOBAL_STEP: 275000 | > loss_text_ce: 0.031122110784053802 (0.042728076114533435) | > loss_mel_ce: 3.567887783050537 (nan) | > loss: 0.014281786046922207 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1364952375146901) | > loader_time: 0.0047 (0.004688818262951412)  --> TIME: 2025-02-24 10:09:57 -- STEP: 16484/36938 -- GLOBAL_STEP: 275050 | > loss_text_ce: 0.04646114632487297 (0.042727000466951374) | > loss_mel_ce: 3.4875476360321045 (nan) | > loss: 0.014023845084011555 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.13648512595719808) | > loader_time: 0.0048 (0.004688676628375684)  --> TIME: 2025-02-24 10:10:06 -- STEP: 16534/36938 -- GLOBAL_STEP: 275100 | > loss_text_ce: 0.047333501279354095 (0.04272558366120756) | > loss_mel_ce: 2.979426383972168 (nan) | > loss: 0.012010952457785606 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.1364717503996759) | > loader_time: 0.0046 (0.004688546506730132)  --> TIME: 2025-02-24 10:10:16 -- STEP: 16584/36938 -- GLOBAL_STEP: 275150 | > loss_text_ce: 0.03982318937778473 (0.04272595798377507) | > loss_mel_ce: 3.4772427082061768 (nan) | > loss: 0.013956611976027489 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.1364784737571791) | > loader_time: 0.0046 (0.004688511364021798)  --> TIME: 2025-02-24 10:10:25 -- STEP: 16634/36938 -- GLOBAL_STEP: 275200 | > loss_text_ce: 0.05089893564581871 (0.04272328724441243) | > loss_mel_ce: 1.3069525957107544 (nan) | > loss: 0.005388299934566021 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13646630801101886) | > loader_time: 0.0048 (0.004688376702123699)  --> TIME: 2025-02-24 10:10:34 -- STEP: 16684/36938 -- GLOBAL_STEP: 275250 | > loss_text_ce: 0.05537589266896248 (0.04272350903992997) | > loss_mel_ce: 3.372567892074585 (nan) | > loss: 0.013602951541543007 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13647081631317698) | > loader_time: 0.0046 (0.004688217039160864)  --> TIME: 2025-02-24 10:10:44 -- STEP: 16734/36938 -- GLOBAL_STEP: 275300 | > loss_text_ce: 0.04205634072422981 (0.04272620735668411) | > loss_mel_ce: 3.3415915966033936 (nan) | > loss: 0.013427174650132656 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13647810979970543) | > loader_time: 0.0046 (0.004688146109505197)  --> TIME: 2025-02-24 10:10:53 -- STEP: 16784/36938 -- GLOBAL_STEP: 275350 | > loss_text_ce: 0.06771599501371384 (0.042730865100362336) | > loss_mel_ce: 2.4753382205963135 (nan) | > loss: 0.01009148545563221 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13648459546843267) | > loader_time: 0.0045 (0.004688069522664934)  --> TIME: 2025-02-24 10:11:02 -- STEP: 16834/36938 -- GLOBAL_STEP: 275400 | > loss_text_ce: 0.031192168593406677 (0.042726398698274846) | > loss_mel_ce: 2.7181918621063232 (nan) | > loss: 0.01091025397181511 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13648330666778938) | > loader_time: 0.0045 (0.004687950321341182)  --> TIME: 2025-02-24 10:11:12 -- STEP: 16884/36938 -- GLOBAL_STEP: 275450 | > loss_text_ce: 0.04223522171378136 (0.042725708690758625) | > loss_mel_ce: 2.6099295616149902 (nan) | > loss: 0.010524463839828968 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13649624670831184) | > loader_time: 0.0047 (0.004687917625076596)  --> TIME: 2025-02-24 10:11:21 -- STEP: 16934/36938 -- GLOBAL_STEP: 275500 | > loss_text_ce: 0.042019233107566833 (0.042727205406923326) | > loss_mel_ce: 3.385575771331787 (nan) | > loss: 0.013601567596197128 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13650002393106397) | > loader_time: 0.0045 (0.0046878029270381654)  --> TIME: 2025-02-24 10:11:30 -- STEP: 16984/36938 -- GLOBAL_STEP: 275550 | > loss_text_ce: 0.05003601312637329 (0.042726133657124576) | > loss_mel_ce: 3.694484233856201 (nan) | > loss: 0.014859207905828953 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13649378130866038) | > loader_time: 0.0049 (0.004687618771309122)  --> TIME: 2025-02-24 10:11:40 -- STEP: 17034/36938 -- GLOBAL_STEP: 275600 | > loss_text_ce: 0.030122743919491768 (0.04272246903924567) | > loss_mel_ce: 3.1313695907592773 (nan) | > loss: 0.012545605190098286 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13650460674041315) | > loader_time: 0.0046 (0.004687530929750862)  --> TIME: 2025-02-24 10:11:49 -- STEP: 17084/36938 -- GLOBAL_STEP: 275650 | > loss_text_ce: 0.04158296808600426 (0.04272482654117307) | > loss_mel_ce: 3.4808990955352783 (nan) | > loss: 0.013978105038404465 (nan) | > current_lr: 5e-06 | > step_time: 0.161 (0.13649712915695159) | > loader_time: 0.0046 (0.004687294486087598)  --> TIME: 2025-02-24 10:11:58 -- STEP: 17134/36938 -- GLOBAL_STEP: 275700 | > loss_text_ce: 0.03662267327308655 (0.04272344758155993) | > loss_mel_ce: 0.10112762451171875 (nan) | > loss: 0.0005466282018460333 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13650049556021046) | > loader_time: 0.0046 (0.004687162323372086)  --> TIME: 2025-02-24 10:12:07 -- STEP: 17184/36938 -- GLOBAL_STEP: 275750 | > loss_text_ce: 0.039849307388067245 (0.04272183085105856) | > loss_mel_ce: 3.0239686965942383 (nan) | > loss: 0.01215800829231739 (nan) | > current_lr: 5e-06 | > step_time: 0.1818 (0.13648539576228527) | > loader_time: 0.0048 (0.00468696680204385)  --> TIME: 2025-02-24 10:12:17 -- STEP: 17234/36938 -- GLOBAL_STEP: 275800 | > loss_text_ce: 0.04306446388363838 (0.04271898988731195) | > loss_mel_ce: 3.4663944244384766 (nan) | > loss: 0.013926424086093903 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.1364841575313777) | > loader_time: 0.0047 (0.0046868072912396195)  --> TIME: 2025-02-24 10:12:26 -- STEP: 17284/36938 -- GLOBAL_STEP: 275850 | > loss_text_ce: 0.05587757006287575 (0.04271785761394584) | > loss_mel_ce: 2.9262540340423584 (nan) | > loss: 0.0118338568136096 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13648974720291202) | > loader_time: 0.0048 (0.004686693120571072)  --> TIME: 2025-02-24 10:12:35 -- STEP: 17334/36938 -- GLOBAL_STEP: 275900 | > loss_text_ce: 0.03907300531864166 (0.042711445748603166) | > loss_mel_ce: 3.902406930923462 (nan) | > loss: 0.01564079336822033 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.1364855426287888) | > loader_time: 0.005 (0.004686544768686766)  --> TIME: 2025-02-24 10:12:45 -- STEP: 17384/36938 -- GLOBAL_STEP: 275950 | > loss_text_ce: 0.05470150336623192 (0.042707587597554) | > loss_mel_ce: 3.266885757446289 (nan) | > loss: 0.013180903159081936 (nan) | > current_lr: 5e-06 | > step_time: 0.157 (0.13649259774409475) | > loader_time: 0.0046 (0.004686374133268489)  --> TIME: 2025-02-24 10:12:54 -- STEP: 17434/36938 -- GLOBAL_STEP: 276000 | > loss_text_ce: 0.04672211408615112 (0.042711744601738426) | > loss_mel_ce: 1.9666526317596436 (nan) | > loss: 0.007989582605659962 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13649404252684516) | > loader_time: 0.0046 (0.004686204476601035)  --> TIME: 2025-02-24 10:13:04 -- STEP: 17484/36938 -- GLOBAL_STEP: 276050 | > loss_text_ce: 0.05286422744393349 (0.042707940104716464) | > loss_mel_ce: 2.341805934906006 (nan) | > loss: 0.009502660483121872 (nan) | > current_lr: 5e-06 | > step_time: 0.1086 (0.13649710849885413) | > loader_time: 0.0045 (0.004686058140324674)  --> TIME: 2025-02-24 10:13:13 -- STEP: 17534/36938 -- GLOBAL_STEP: 276100 | > loss_text_ce: 0.016343729570508003 (0.042708466554514636) | > loss_mel_ce: 2.107177734375 (nan) | > loss: 0.008426673710346222 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13649838775977602) | > loader_time: 0.0044 (0.004685888788617712)  --> TIME: 2025-02-24 10:13:23 -- STEP: 17584/36938 -- GLOBAL_STEP: 276150 | > loss_text_ce: 0.03684461861848831 (0.042705938805596394) | > loss_mel_ce: 3.424255847930908 (nan) | > loss: 0.013734526932239532 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13651185633921392) | > loader_time: 0.0047 (0.004685860761086229)  --> TIME: 2025-02-24 10:13:32 -- STEP: 17634/36938 -- GLOBAL_STEP: 276200 | > loss_text_ce: 0.04281430318951607 (0.04270299673521975) | > loss_mel_ce: 3.6510419845581055 (nan) | > loss: 0.014658160507678986 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13651611926104834) | > loader_time: 0.0045 (0.0046857285961970216)  --> TIME: 2025-02-24 10:13:41 -- STEP: 17684/36938 -- GLOBAL_STEP: 276250 | > loss_text_ce: 0.04678621515631676 (0.04270301625822262) | > loss_mel_ce: 3.265699625015259 (nan) | > loss: 0.013144786469638348 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13651996284716508) | > loader_time: 0.0046 (0.004685583507763916)  --> TIME: 2025-02-24 10:13:51 -- STEP: 17734/36938 -- GLOBAL_STEP: 276300 | > loss_text_ce: 0.04195968061685562 (0.04269901154771009) | > loss_mel_ce: 2.8567981719970703 (nan) | > loss: 0.01150300819426775 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.13652031864242803) | > loader_time: 0.0047 (0.004685421491188201)  --> TIME: 2025-02-24 10:14:00 -- STEP: 17784/36938 -- GLOBAL_STEP: 276350 | > loss_text_ce: 0.04221583902835846 (0.04269873345218597) | > loss_mel_ce: 2.45585560798645 (nan) | > loss: 0.00991298258304596 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1365083868150712) | > loader_time: 0.0048 (0.004685261444738741)  --> TIME: 2025-02-24 10:14:09 -- STEP: 17834/36938 -- GLOBAL_STEP: 276400 | > loss_text_ce: 0.04294433444738388 (0.04270109272397679) | > loss_mel_ce: 3.104543924331665 (nan) | > loss: 0.012490034103393555 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13650498636717773) | > loader_time: 0.0046 (0.004685117843587723)  --> TIME: 2025-02-24 10:14:18 -- STEP: 17884/36938 -- GLOBAL_STEP: 276450 | > loss_text_ce: 0.04647049680352211 (0.04270500369338532) | > loss_mel_ce: 2.80244779586792 (nan) | > loss: 0.011305231600999832 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.13650920111923417) | > loader_time: 0.0048 (0.004685046594965735)  --> TIME: 2025-02-24 10:14:28 -- STEP: 17934/36938 -- GLOBAL_STEP: 276500 | > loss_text_ce: 0.03515171632170677 (0.04270331281831078) | > loss_mel_ce: 6.1785712242126465 (nan) | > loss: 0.024657631292939186 (nan) | > current_lr: 5e-06 | > step_time: 0.1065 (0.13649784892488345) | > loader_time: 0.0044 (0.004684876435787055)  --> TIME: 2025-02-24 10:14:37 -- STEP: 17984/36938 -- GLOBAL_STEP: 276550 | > loss_text_ce: 0.0436687134206295 (0.04270406884924892) | > loss_mel_ce: 0.08690643310546875 (nan) | > loss: 0.0005181554006412625 (nan) | > current_lr: 5e-06 | > step_time: 0.1118 (0.13650743416725916) | > loader_time: 0.0045 (0.004684790504890837)  --> TIME: 2025-02-24 10:14:46 -- STEP: 18034/36938 -- GLOBAL_STEP: 276600 | > loss_text_ce: 0.03594944253563881 (0.042706004029475116) | > loss_mel_ce: 2.9588253498077393 (nan) | > loss: 0.01188402809202671 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13648793523375305) | > loader_time: 0.0046 (0.0046846088448768124)  --> TIME: 2025-02-24 10:14:55 -- STEP: 18084/36938 -- GLOBAL_STEP: 276650 | > loss_text_ce: 0.022801654413342476 (0.042706745432947675) | > loss_mel_ce: 0.724578857421875 (nan) | > loss: 0.0029657958075404167 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1364822100285167) | > loader_time: 0.0045 (0.004684481057042817)  --> TIME: 2025-02-24 10:15:05 -- STEP: 18134/36938 -- GLOBAL_STEP: 276700 | > loss_text_ce: 0.03618865832686424 (0.04270919391261295) | > loss_mel_ce: 2.9448776245117188 (nan) | > loss: 0.011829628609120846 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.1364869035511723) | > loader_time: 0.0045 (0.004684331281137564)  --> TIME: 2025-02-24 10:15:14 -- STEP: 18184/36938 -- GLOBAL_STEP: 276750 | > loss_text_ce: 0.022840524092316628 (0.04270563283753878) | > loss_mel_ce: 3.0891926288604736 (nan) | > loss: 0.01234933827072382 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13649936322593165) | > loader_time: 0.0045 (0.004684247309239601)  --> TIME: 2025-02-24 10:15:23 -- STEP: 18234/36938 -- GLOBAL_STEP: 276800 | > loss_text_ce: 0.04288895055651665 (0.04270353540364437) | > loss_mel_ce: 3.1403894424438477 (nan) | > loss: 0.012632058002054691 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.1364965905507194) | > loader_time: 0.0049 (0.00468407925171354)  --> TIME: 2025-02-24 10:15:33 -- STEP: 18284/36938 -- GLOBAL_STEP: 276850 | > loss_text_ce: 0.03688134625554085 (0.04270330057283837) | > loss_mel_ce: 3.0326991081237793 (nan) | > loss: 0.012180875986814499 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13649095249655857) | > loader_time: 0.0045 (0.004683888094139896)  --> TIME: 2025-02-24 10:15:42 -- STEP: 18334/36938 -- GLOBAL_STEP: 276900 | > loss_text_ce: 0.0339936688542366 (0.04270234239631364) | > loss_mel_ce: 3.006526231765747 (nan) | > loss: 0.012065555900335312 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13650116696019077) | > loader_time: 0.0046 (0.004683777590776933)  --> TIME: 2025-02-24 10:15:51 -- STEP: 18384/36938 -- GLOBAL_STEP: 276950 | > loss_text_ce: 0.04336574301123619 (0.04270187361629713) | > loss_mel_ce: 2.864107370376587 (nan) | > loss: 0.011537591926753521 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.13647912214433117) | > loader_time: 0.0047 (0.004683631531459755)  --> TIME: 2025-02-24 10:16:00 -- STEP: 18434/36938 -- GLOBAL_STEP: 277000 | > loss_text_ce: 0.017518596723675728 (0.04269705802173726) | > loss_mel_ce: 2.4471657276153564 (nan) | > loss: 0.009780493564903736 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13648624119847041) | > loader_time: 0.0045 (0.004683722781814256)  --> TIME: 2025-02-24 10:16:10 -- STEP: 18484/36938 -- GLOBAL_STEP: 277050 | > loss_text_ce: 0.04111069068312645 (0.04269553559743188) | > loss_mel_ce: 4.014265060424805 (nan) | > loss: 0.016092760488390923 (nan) | > current_lr: 5e-06 | > step_time: 0.1878 (0.13648663562140223) | > loader_time: 0.0048 (0.004683582239970323)  --> TIME: 2025-02-24 10:16:19 -- STEP: 18534/36938 -- GLOBAL_STEP: 277100 | > loss_text_ce: 0.035564761608839035 (0.04269585101602422) | > loss_mel_ce: 3.257174253463745 (nan) | > loss: 0.013066424988210201 (nan) | > current_lr: 5e-06 | > step_time: 0.1617 (0.13648736292945118) | > loader_time: 0.0047 (0.004683541160732512)  --> TIME: 2025-02-24 10:16:28 -- STEP: 18584/36938 -- GLOBAL_STEP: 277150 | > loss_text_ce: 0.04305579885840416 (0.042694757932111846) | > loss_mel_ce: 4.084653854370117 (nan) | > loss: 0.01637980155646801 (nan) | > current_lr: 5e-06 | > step_time: 0.1502 (0.13648977625539335) | > loader_time: 0.0047 (0.004683510065591905)  --> TIME: 2025-02-24 10:16:38 -- STEP: 18634/36938 -- GLOBAL_STEP: 277200 | > loss_text_ce: 0.04720936343073845 (0.04269201833271167) | > loss_mel_ce: 3.6839005947113037 (nan) | > loss: 0.014805992133915424 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.1364944697552332) | > loader_time: 0.0047 (0.004683516677311081)  --> TIME: 2025-02-24 10:16:47 -- STEP: 18684/36938 -- GLOBAL_STEP: 277250 | > loss_text_ce: 0.047820452600717545 (0.04269043528084066) | > loss_mel_ce: 3.4467499256134033 (nan) | > loss: 0.01386734377592802 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13650300116897288) | > loader_time: 0.0048 (0.004683675244849341)  --> TIME: 2025-02-24 10:16:57 -- STEP: 18734/36938 -- GLOBAL_STEP: 277300 | > loss_text_ce: 0.048455242067575455 (0.042690133061828624) | > loss_mel_ce: 3.7498221397399902 (nan) | > loss: 0.015072530135512352 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13651605208299036) | > loader_time: 0.0046 (0.004683805667591721)  --> TIME: 2025-02-24 10:17:06 -- STEP: 18784/36938 -- GLOBAL_STEP: 277350 | > loss_text_ce: 0.04660160094499588 (0.04269224167296701) | > loss_mel_ce: 2.959948778152466 (nan) | > loss: 0.011930756270885468 (nan) | > current_lr: 5e-06 | > step_time: 0.1494 (0.1365072880022769) | > loader_time: 0.0048 (0.004683789468705151)  --> TIME: 2025-02-24 10:17:15 -- STEP: 18834/36938 -- GLOBAL_STEP: 277400 | > loss_text_ce: 0.04217870533466339 (0.04268798443656005) | > loss_mel_ce: 3.4391279220581055 (nan) | > loss: 0.013814709149301052 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.1365036779469306) | > loader_time: 0.0045 (0.004683752101457871)  --> TIME: 2025-02-24 10:17:25 -- STEP: 18884/36938 -- GLOBAL_STEP: 277450 | > loss_text_ce: 0.028711190447211266 (0.042689403157187576) | > loss_mel_ce: 3.8272805213928223 (nan) | > loss: 0.015301554463803768 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13651356874343482) | > loader_time: 0.0044 (0.004683742695404201)  --> TIME: 2025-02-24 10:17:34 -- STEP: 18934/36938 -- GLOBAL_STEP: 277500 | > loss_text_ce: 0.04164840281009674 (0.042685105627819915) | > loss_mel_ce: 3.5906362533569336 (nan) | > loss: 0.01441382896155119 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13652666131105476) | > loader_time: 0.0046 (0.004683633559329647)  --> TIME: 2025-02-24 10:17:44 -- STEP: 18984/36938 -- GLOBAL_STEP: 277550 | > loss_text_ce: 0.03840342536568642 (0.04267993668113698) | > loss_mel_ce: 3.305072546005249 (nan) | > loss: 0.013267762959003448 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13652134283253023) | > loader_time: 0.0052 (0.004683776804529767)  --> TIME: 2025-02-24 10:17:53 -- STEP: 19034/36938 -- GLOBAL_STEP: 277600 | > loss_text_ce: 0.032118719071149826 (0.04267605346364971) | > loss_mel_ce: 3.333444118499756 (nan) | > loss: 0.013355408795177937 (nan) | > current_lr: 5e-06 | > step_time: 0.1634 (0.13651315131532607) | > loader_time: 0.0047 (0.004683768096636036)  --> TIME: 2025-02-24 10:18:02 -- STEP: 19084/36938 -- GLOBAL_STEP: 277650 | > loss_text_ce: 0.0479666069149971 (0.042675070785674236) | > loss_mel_ce: 0.8095963001251221 (nan) | > loss: 0.003403027541935444 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13651585994939094) | > loader_time: 0.0047 (0.004683775513008744)  --> TIME: 2025-02-24 10:18:11 -- STEP: 19134/36938 -- GLOBAL_STEP: 277700 | > loss_text_ce: 0.04283391684293747 (0.042673104507148604) | > loss_mel_ce: 3.364858388900757 (nan) | > loss: 0.013522588647902012 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.13651869768531624) | > loader_time: 0.0049 (0.0046837968338840534)  --> TIME: 2025-02-24 10:18:21 -- STEP: 19184/36938 -- GLOBAL_STEP: 277750 | > loss_text_ce: 0.0402505062520504 (0.04267281743602082) | > loss_mel_ce: 2.618278741836548 (nan) | > loss: 0.010549720376729965 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13651698313224672) | > loader_time: 0.0045 (0.004683909078654503)  --> TIME: 2025-02-24 10:18:30 -- STEP: 19234/36938 -- GLOBAL_STEP: 277800 | > loss_text_ce: 0.04606412351131439 (0.04266935094350993) | > loss_mel_ce: 3.2745375633239746 (nan) | > loss: 0.013176991604268551 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13652059830337038) | > loader_time: 0.0046 (0.004684079371433795)  --> TIME: 2025-02-24 10:18:40 -- STEP: 19284/36938 -- GLOBAL_STEP: 277850 | > loss_text_ce: 0.027744116261601448 (0.04266727518730144) | > loss_mel_ce: 2.5674023628234863 (nan) | > loss: 0.010298200882971287 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13653320080263176) | > loader_time: 0.0045 (0.004684280369989468)  --> TIME: 2025-02-24 10:18:49 -- STEP: 19334/36938 -- GLOBAL_STEP: 277900 | > loss_text_ce: 0.04615344479680061 (0.042663910695326254) | > loss_mel_ce: 3.7762460708618164 (nan) | > loss: 0.01516825333237648 (nan) | > current_lr: 5e-06 | > step_time: 0.1566 (0.13653041916400604) | > loader_time: 0.0047 (0.0046841362411287595)  --> TIME: 2025-02-24 10:18:58 -- STEP: 19384/36938 -- GLOBAL_STEP: 277950 | > loss_text_ce: 0.040199488401412964 (0.04266291861575139) | > loss_mel_ce: 2.233039379119873 (nan) | > loss: 0.00902078952640295 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13653532634871668) | > loader_time: 0.0054 (0.004684231102195998)  --> TIME: 2025-02-24 10:19:07 -- STEP: 19434/36938 -- GLOBAL_STEP: 278000 | > loss_text_ce: 0.04153545945882797 (0.04266243495423373) | > loss_mel_ce: 3.2678773403167725 (nan) | > loss: 0.013132590800523758 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13652399651881797) | > loader_time: 0.0046 (0.00468436013257368)  --> TIME: 2025-02-24 10:19:17 -- STEP: 19484/36938 -- GLOBAL_STEP: 278050 | > loss_text_ce: 0.05382648855447769 (0.04266418141081159) | > loss_mel_ce: 1.7482025623321533 (nan) | > loss: 0.007150909397751093 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13651833078303735) | > loader_time: 0.0046 (0.004684321568551144)  --> TIME: 2025-02-24 10:19:26 -- STEP: 19534/36938 -- GLOBAL_STEP: 278100 | > loss_text_ce: 0.044646572321653366 (0.042665432347027556) | > loss_mel_ce: 3.6843068599700928 (nan) | > loss: 0.014797435142099857 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13653642491453608) | > loader_time: 0.0045 (0.004684545176781301)  --> TIME: 2025-02-24 10:19:36 -- STEP: 19584/36938 -- GLOBAL_STEP: 278150 | > loss_text_ce: 0.0407453253865242 (0.042662418869582996) | > loss_mel_ce: 3.131075859069824 (nan) | > loss: 0.012586592696607113 (nan) | > current_lr: 5e-06 | > step_time: 0.1876 (0.13653300205866492) | > loader_time: 0.0051 (0.004684529723782173)  --> TIME: 2025-02-24 10:19:45 -- STEP: 19634/36938 -- GLOBAL_STEP: 278200 | > loss_text_ce: 0.04148035869002342 (0.042661698620039205) | > loss_mel_ce: 4.119900226593018 (nan) | > loss: 0.016513416543602943 (nan) | > current_lr: 5e-06 | > step_time: 0.189 (0.13654384915887224) | > loader_time: 0.0055 (0.004684749295126955)  --> TIME: 2025-02-24 10:19:54 -- STEP: 19684/36938 -- GLOBAL_STEP: 278250 | > loss_text_ce: 0.03771085664629936 (0.04265781851318626) | > loss_mel_ce: 2.968118190765381 (nan) | > loss: 0.011927894316613674 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13653221781528518) | > loader_time: 0.0046 (0.004684647429016066)  --> TIME: 2025-02-24 10:20:03 -- STEP: 19734/36938 -- GLOBAL_STEP: 278300 | > loss_text_ce: 0.03612889349460602 (0.04266262181927976) | > loss_mel_ce: 2.8817107677459717 (nan) | > loss: 0.011578730307519436 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.13652917620402835) | > loader_time: 0.0048 (0.004684526084029259)  --> TIME: 2025-02-24 10:20:13 -- STEP: 19784/36938 -- GLOBAL_STEP: 278350 | > loss_text_ce: 0.04191247373819351 (0.042662856900912614) | > loss_mel_ce: 3.3336551189422607 (nan) | > loss: 0.01339511014521122 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.13653962917285464) | > loader_time: 0.0049 (0.004684500182345004)  --> TIME: 2025-02-24 10:20:22 -- STEP: 19834/36938 -- GLOBAL_STEP: 278400 | > loss_text_ce: 0.03144229203462601 (0.04266239432531608) | > loss_mel_ce: 3.007946252822876 (nan) | > loss: 0.012061066925525665 (nan) | > current_lr: 5e-06 | > step_time: 0.1271 (0.13654415914220683) | > loader_time: 0.0046 (0.004684434298174609)  --> TIME: 2025-02-24 10:20:31 -- STEP: 19884/36938 -- GLOBAL_STEP: 278450 | > loss_text_ce: 0.048663996160030365 (0.04266364981479901) | > loss_mel_ce: 2.954108238220215 (nan) | > loss: 0.011915763840079308 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13653828175200763) | > loader_time: 0.0046 (0.004684287102211416)  --> TIME: 2025-02-24 10:20:41 -- STEP: 19934/36938 -- GLOBAL_STEP: 278500 | > loss_text_ce: 0.0441746711730957 (0.04265817490480978) | > loss_mel_ce: 3.395716667175293 (nan) | > loss: 0.013650363311171532 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13655776611311676) | > loader_time: 0.0046 (0.004684286166830558)  --> TIME: 2025-02-24 10:20:51 -- STEP: 19984/36938 -- GLOBAL_STEP: 278550 | > loss_text_ce: 0.04100221395492554 (0.042656656627248955) | > loss_mel_ce: 3.8308091163635254 (nan) | > loss: 0.015364332124590874 (nan) | > current_lr: 5e-06 | > step_time: 0.1629 (0.1365647163268946) | > loader_time: 0.0046 (0.004684184292395472)  --> TIME: 2025-02-24 10:21:00 -- STEP: 20034/36938 -- GLOBAL_STEP: 278600 | > loss_text_ce: 0.04099613428115845 (0.04265837103755622) | > loss_mel_ce: 2.9215033054351807 (nan) | > loss: 0.01175595074892044 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13654463444925224) | > loader_time: 0.0051 (0.004684221105469889)  --> TIME: 2025-02-24 10:21:09 -- STEP: 20084/36938 -- GLOBAL_STEP: 278650 | > loss_text_ce: 0.04195099696516991 (0.04265751381363136) | > loss_mel_ce: 3.274218797683716 (nan) | > loss: 0.013159404508769512 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13654646641275991) | > loader_time: 0.0047 (0.004684614140397166)  --> TIME: 2025-02-24 10:21:18 -- STEP: 20134/36938 -- GLOBAL_STEP: 278700 | > loss_text_ce: 0.058077335357666016 (0.042652785527568134) | > loss_mel_ce: 3.1315979957580566 (nan) | > loss: 0.01265744213014841 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.1365528411373572) | > loader_time: 0.0045 (0.004684856244181289)  --> TIME: 2025-02-24 10:21:28 -- STEP: 20184/36938 -- GLOBAL_STEP: 278750 | > loss_text_ce: 0.04420333728194237 (0.04265487439908462) | > loss_mel_ce: 2.6457114219665527 (nan) | > loss: 0.010674265213310719 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.1365406485573433) | > loader_time: 0.0047 (0.00468492541008888)  --> TIME: 2025-02-24 10:21:37 -- STEP: 20234/36938 -- GLOBAL_STEP: 278800 | > loss_text_ce: 0.04155384376645088 (0.04265350251683831) | > loss_mel_ce: 2.8204426765441895 (nan) | > loss: 0.01135712955147028 (nan) | > current_lr: 5e-06 | > step_time: 0.1508 (0.1365378234795788) | > loader_time: 0.0047 (0.004684957789139897)  --> TIME: 2025-02-24 10:21:46 -- STEP: 20284/36938 -- GLOBAL_STEP: 278850 | > loss_text_ce: 0.031882405281066895 (0.04265290111397166) | > loss_mel_ce: 3.6928088665008545 (nan) | > loss: 0.01478052232414484 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13653059664902864) | > loader_time: 0.0046 (0.004685059686403879)  --> TIME: 2025-02-24 10:21:55 -- STEP: 20334/36938 -- GLOBAL_STEP: 278900 | > loss_text_ce: 0.039740994572639465 (0.04265347520174162) | > loss_mel_ce: 3.3185133934020996 (nan) | > loss: 0.01332640741020441 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.1365312028801218) | > loader_time: 0.0046 (0.00468508610041126)  --> TIME: 2025-02-24 10:22:05 -- STEP: 20384/36938 -- GLOBAL_STEP: 278950 | > loss_text_ce: 0.04217620939016342 (0.04265437613876806) | > loss_mel_ce: 2.240550994873047 (nan) | > loss: 0.00905844196677208 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13653613019915928) | > loader_time: 0.0052 (0.004685250413567539)  --> TIME: 2025-02-24 10:22:14 -- STEP: 20434/36938 -- GLOBAL_STEP: 279000 | > loss_text_ce: 0.04740317165851593 (0.04265298587334336) | > loss_mel_ce: 2.9928038120269775 (nan) | > loss: 0.012064313516020775 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13653585990604467) | > loader_time: 0.0047 (0.00468533565541403)  --> TIME: 2025-02-24 10:22:23 -- STEP: 20484/36938 -- GLOBAL_STEP: 279050 | > loss_text_ce: 0.04537240043282509 (0.04265148507745219) | > loss_mel_ce: 3.6254167556762695 (nan) | > loss: 0.014566624537110329 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.13652860774958536) | > loader_time: 0.005 (0.004685403080429809)  --> TIME: 2025-02-24 10:22:32 -- STEP: 20534/36938 -- GLOBAL_STEP: 279100 | > loss_text_ce: 0.043621305376291275 (0.04265378348236697) | > loss_mel_ce: 3.5674853324890137 (nan) | > loss: 0.014329789206385612 (nan) | > current_lr: 5e-06 | > step_time: 0.168 (0.13652224663553975) | > loader_time: 0.0049 (0.004685861302453182)  --> TIME: 2025-02-24 10:22:42 -- STEP: 20584/36938 -- GLOBAL_STEP: 279150 | > loss_text_ce: 0.039899636059999466 (0.042646800807034285) | > loss_mel_ce: 3.5160350799560547 (nan) | > loss: 0.014110852964222431 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.1365360092709051) | > loader_time: 0.0046 (0.004686091701853769)  --> TIME: 2025-02-24 10:22:51 -- STEP: 20634/36938 -- GLOBAL_STEP: 279200 | > loss_text_ce: 0.07075898349285126 (0.04264707936608057) | > loss_mel_ce: 0.9906294941902161 (nan) | > loss: 0.0042118593119084835 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13654304192003125) | > loader_time: 0.0046 (0.0046861863036921705)  --> TIME: 2025-02-24 10:23:01 -- STEP: 20684/36938 -- GLOBAL_STEP: 279250 | > loss_text_ce: 0.03642420843243599 (0.042646562748096334) | > loss_mel_ce: 3.5075294971466064 (nan) | > loss: 0.014063308946788311 (nan) | > current_lr: 5e-06 | > step_time: 0.1588 (0.13654780076657122) | > loader_time: 0.0046 (0.004686134335520297)  --> TIME: 2025-02-24 10:23:10 -- STEP: 20734/36938 -- GLOBAL_STEP: 279300 | > loss_text_ce: 0.03889690712094307 (0.04264496093940369) | > loss_mel_ce: 3.0253005027770996 (nan) | > loss: 0.012159514240920544 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13655023721095355) | > loader_time: 0.0046 (0.004686102269642669)  --> TIME: 2025-02-24 10:23:19 -- STEP: 20784/36938 -- GLOBAL_STEP: 279350 | > loss_text_ce: 0.042061712592840195 (0.042644434506129134) | > loss_mel_ce: 3.149724006652832 (nan) | > loss: 0.012665817514061928 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.13654066991631686) | > loader_time: 0.0048 (0.004686174195853463)  --> TIME: 2025-02-24 10:23:29 -- STEP: 20834/36938 -- GLOBAL_STEP: 279400 | > loss_text_ce: 0.028652962297201157 (0.042646873527280436) | > loss_mel_ce: 3.2271299362182617 (nan) | > loss: 0.012919774278998375 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.13655735885180334) | > loader_time: 0.0047 (0.004686297902999591)  --> TIME: 2025-02-24 10:23:38 -- STEP: 20884/36938 -- GLOBAL_STEP: 279450 | > loss_text_ce: 0.03649662807583809 (0.042648932502200664) | > loss_mel_ce: 0.1113433837890625 (nan) | > loss: 0.0005866667488589883 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13654091735608612) | > loader_time: 0.0046 (0.004686322529136503)  --> TIME: 2025-02-24 10:23:47 -- STEP: 20934/36938 -- GLOBAL_STEP: 279500 | > loss_text_ce: 0.04191478714346886 (0.042647185021497294) | > loss_mel_ce: 3.7228660583496094 (nan) | > loss: 0.014939607121050358 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13654741756187397) | > loader_time: 0.0047 (0.004686312813511496)  --> TIME: 2025-02-24 10:23:57 -- STEP: 20984/36938 -- GLOBAL_STEP: 279550 | > loss_text_ce: 0.03841172531247139 (0.042649232858184415) | > loss_mel_ce: 3.847147226333618 (nan) | > loss: 0.01541888527572155 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13655120083225017) | > loader_time: 0.0046 (0.004686445804483977)  --> TIME: 2025-02-24 10:24:06 -- STEP: 21034/36938 -- GLOBAL_STEP: 279600 | > loss_text_ce: 0.024160869419574738 (0.04264504786769773) | > loss_mel_ce: 2.349327564239502 (nan) | > loss: 0.009418605826795101 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13654546224426034) | > loader_time: 0.0045 (0.004686503613461026)  --> TIME: 2025-02-24 10:24:15 -- STEP: 21084/36938 -- GLOBAL_STEP: 279650 | > loss_text_ce: 0.04340910166501999 (0.04264314619295463) | > loss_mel_ce: 3.3355681896209717 (nan) | > loss: 0.013408640399575233 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13653625946211415) | > loader_time: 0.0045 (0.004686624258389143)  --> TIME: 2025-02-24 10:24:24 -- STEP: 21134/36938 -- GLOBAL_STEP: 279700 | > loss_text_ce: 0.042939286679029465 (0.04264297663104876) | > loss_mel_ce: 2.7978873252868652 (nan) | > loss: 0.01127312146127224 (nan) | > current_lr: 5e-06 | > step_time: 0.177 (0.13652477255830556) | > loader_time: 0.0049 (0.004686617891869953)  --> TIME: 2025-02-24 10:24:33 -- STEP: 21184/36938 -- GLOBAL_STEP: 279750 | > loss_text_ce: 0.035724937915802 (0.04263649179154033) | > loss_mel_ce: 2.7422919273376465 (nan) | > loss: 0.011023877188563347 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13650809015707327) | > loader_time: 0.0046 (0.004686435678934422)  --> TIME: 2025-02-24 10:24:42 -- STEP: 21234/36938 -- GLOBAL_STEP: 279800 | > loss_text_ce: 0.03156151995062828 (0.04264073843189759) | > loss_mel_ce: 3.02907657623291 (nan) | > loss: 0.0121453907340765 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13650254862135824) | > loader_time: 0.0053 (0.0046866937490734565)  --> TIME: 2025-02-24 10:24:52 -- STEP: 21284/36938 -- GLOBAL_STEP: 279850 | > loss_text_ce: 0.0407429113984108 (0.042640703791413555) | > loss_mel_ce: 3.0397377014160156 (nan) | > loss: 0.01222413033246994 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13651166624692215) | > loader_time: 0.0046 (0.004686763615474709)  --> TIME: 2025-02-24 10:25:01 -- STEP: 21334/36938 -- GLOBAL_STEP: 279900 | > loss_text_ce: 0.04446718469262123 (0.04263764067551323) | > loss_mel_ce: 3.6741464138031006 (nan) | > loss: 0.014756403863430023 (nan) | > current_lr: 5e-06 | > step_time: 0.1701 (0.13652050430329682) | > loader_time: 0.0048 (0.004686693113925516)  --> TIME: 2025-02-24 10:25:10 -- STEP: 21384/36938 -- GLOBAL_STEP: 279950 | > loss_text_ce: 0.032215118408203125 (0.042632691741741904) | > loss_mel_ce: 3.1473515033721924 (nan) | > loss: 0.012617328204214573 (nan) | > current_lr: 5e-06 | > step_time: 0.1086 (0.13651565996127935) | > loader_time: 0.0048 (0.0046865727698094045)  --> TIME: 2025-02-24 10:25:19 -- STEP: 21434/36938 -- GLOBAL_STEP: 280000 | > loss_text_ce: 0.041154585778713226 (0.042635036237060615) | > loss_mel_ce: 3.266632318496704 (nan) | > loss: 0.013126139529049397 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13650597786963434) | > loader_time: 0.0046 (0.004686481885704492) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_280000.pth  --> TIME: 2025-02-24 10:25:35 -- STEP: 21484/36938 -- GLOBAL_STEP: 280050 | > loss_text_ce: 0.03604206070303917 (0.042635870318323325) | > loss_mel_ce: 2.163647174835205 (nan) | > loss: 0.008728926070034504 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13650406381311944) | > loader_time: 0.0046 (0.004686378551537152)  --> TIME: 2025-02-24 10:25:44 -- STEP: 21534/36938 -- GLOBAL_STEP: 280100 | > loss_text_ce: 0.04242180660367012 (0.04263496262272123) | > loss_mel_ce: 3.8282392024993896 (nan) | > loss: 0.015359766781330109 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13649749056126204) | > loader_time: 0.0046 (0.004686271135683276)  --> TIME: 2025-02-24 10:25:53 -- STEP: 21584/36938 -- GLOBAL_STEP: 280150 | > loss_text_ce: 0.0479731485247612 (0.04263337951817494) | > loss_mel_ce: 2.7760417461395264 (nan) | > loss: 0.011206408962607384 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.1364810209061323) | > loader_time: 0.0046 (0.004686113317159502)  --> TIME: 2025-02-24 10:26:02 -- STEP: 21634/36938 -- GLOBAL_STEP: 280200 | > loss_text_ce: 0.0342361181974411 (0.04263017963189446) | > loss_mel_ce: 3.4465019702911377 (nan) | > loss: 0.013812453486025333 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.1364812784828848) | > loader_time: 0.0046 (0.004686006184281996)  --> TIME: 2025-02-24 10:26:11 -- STEP: 21684/36938 -- GLOBAL_STEP: 280250 | > loss_text_ce: 0.05428663641214371 (0.04263019799284044) | > loss_mel_ce: 1.9997482299804688 (nan) | > loss: 0.008150932379066944 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13647126298266526) | > loader_time: 0.0046 (0.0046858667909516465)  --> TIME: 2025-02-24 10:26:20 -- STEP: 21734/36938 -- GLOBAL_STEP: 280300 | > loss_text_ce: 0.041990600526332855 (0.042630886113233644) | > loss_mel_ce: 3.3789567947387695 (nan) | > loss: 0.01357518881559372 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13647337239350488) | > loader_time: 0.0047 (0.00468579838858849)  --> TIME: 2025-02-24 10:26:29 -- STEP: 21784/36938 -- GLOBAL_STEP: 280350 | > loss_text_ce: 0.03898845985531807 (0.04262829803292387) | > loss_mel_ce: 2.4925663471221924 (nan) | > loss: 0.0100458525121212 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13646843748148688) | > loader_time: 0.0046 (0.0046856620383534265)  --> TIME: 2025-02-24 10:26:39 -- STEP: 21834/36938 -- GLOBAL_STEP: 280400 | > loss_text_ce: 0.05508754029870033 (0.04263073728199149) | > loss_mel_ce: 3.0104126930236816 (nan) | > loss: 0.012164684012532234 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13647355175699688) | > loader_time: 0.0047 (0.0046855661036320434)  --> TIME: 2025-02-24 10:26:48 -- STEP: 21884/36938 -- GLOBAL_STEP: 280450 | > loss_text_ce: 0.03669523447751999 (0.042633069799961065) | > loss_mel_ce: 3.141263246536255 (nan) | > loss: 0.012610946781933308 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13647329238580627) | > loader_time: 0.0046 (0.004685420971251445)  --> TIME: 2025-02-24 10:26:57 -- STEP: 21934/36938 -- GLOBAL_STEP: 280500 | > loss_text_ce: 0.046166468411684036 (0.04263140400294176) | > loss_mel_ce: 0.04218083992600441 (nan) | > loss: 0.00035058456705883145 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13646510324124225) | > loader_time: 0.0046 (0.004685225021092398)  --> TIME: 2025-02-24 10:27:07 -- STEP: 21984/36938 -- GLOBAL_STEP: 280550 | > loss_text_ce: 0.034650593996047974 (0.042631649891474804) | > loss_mel_ce: 1.4837239980697632 (nan) | > loss: 0.0060252961702644825 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13647424406880368) | > loader_time: 0.0045 (0.004685209448594381)  --> TIME: 2025-02-24 10:27:16 -- STEP: 22034/36938 -- GLOBAL_STEP: 280600 | > loss_text_ce: 0.04484277218580246 (0.042629585456369565) | > loss_mel_ce: 2.33366322517395 (nan) | > loss: 0.009438516572117805 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.1364658790826257) | > loader_time: 0.0048 (0.00468507874106133)  --> TIME: 2025-02-24 10:27:25 -- STEP: 22084/36938 -- GLOBAL_STEP: 280650 | > loss_text_ce: 0.042877864092588425 (0.04262817084518196) | > loss_mel_ce: 3.3327245712280273 (nan) | > loss: 0.013395248912274837 (nan) | > current_lr: 5e-06 | > step_time: 0.1574 (0.13645722715562295) | > loader_time: 0.0046 (0.004684889279852495)  --> TIME: 2025-02-24 10:27:34 -- STEP: 22134/36938 -- GLOBAL_STEP: 280700 | > loss_text_ce: 0.03297962620854378 (0.04262781182946617) | > loss_mel_ce: 2.661829710006714 (nan) | > loss: 0.010693688876926899 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.1364576340131001) | > loader_time: 0.0045 (0.004684800053392535)  --> TIME: 2025-02-24 10:27:44 -- STEP: 22184/36938 -- GLOBAL_STEP: 280750 | > loss_text_ce: 0.037375662475824356 (0.04262589008801438) | > loss_mel_ce: 3.4626352787017822 (nan) | > loss: 0.013888932764530182 (nan) | > current_lr: 5e-06 | > step_time: 0.1824 (0.13646482255241307) | > loader_time: 0.0047 (0.004684718548069362)  --> TIME: 2025-02-24 10:27:53 -- STEP: 22234/36938 -- GLOBAL_STEP: 280800 | > loss_text_ce: 0.04884892702102661 (0.04262309833937007) | > loss_mel_ce: 3.897153615951538 (nan) | > loss: 0.01565874181687832 (nan) | > current_lr: 5e-06 | > step_time: 0.1873 (0.13647411117732144) | > loader_time: 0.0048 (0.004684620284450848)  --> TIME: 2025-02-24 10:28:03 -- STEP: 22284/36938 -- GLOBAL_STEP: 280850 | > loss_text_ce: 0.046500176191329956 (0.042623995189728654) | > loss_mel_ce: 0.1447906494140625 (nan) | > loss: 0.0007590906461700797 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13647181526417576) | > loader_time: 0.0048 (0.004684592155681826)  --> TIME: 2025-02-24 10:28:12 -- STEP: 22334/36938 -- GLOBAL_STEP: 280900 | > loss_text_ce: 0.05797457695007324 (0.042621550492195355) | > loss_mel_ce: 2.0438475608825684 (nan) | > loss: 0.00834056455641985 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13645968078937123) | > loader_time: 0.0046 (0.0046843930090126025)  --> TIME: 2025-02-24 10:28:21 -- STEP: 22384/36938 -- GLOBAL_STEP: 280950 | > loss_text_ce: 0.039298806339502335 (0.042620546375246496) | > loss_mel_ce: 2.916182041168213 (nan) | > loss: 0.011728099547326565 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.1364530097211234) | > loader_time: 0.0046 (0.004684220293831028)  --> TIME: 2025-02-24 10:28:30 -- STEP: 22434/36938 -- GLOBAL_STEP: 281000 | > loss_text_ce: 0.046409882605075836 (0.04262199694472749) | > loss_mel_ce: 2.5908212661743164 (nan) | > loss: 0.010465203784406185 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13645545223918285) | > loader_time: 0.0047 (0.004684226434477575)  --> TIME: 2025-02-24 10:28:39 -- STEP: 22484/36938 -- GLOBAL_STEP: 281050 | > loss_text_ce: 0.041185762733221054 (0.04262059487391996) | > loss_mel_ce: 3.5717074871063232 (nan) | > loss: 0.01433687936514616 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.136440510142) | > loader_time: 0.0046 (0.004684021540371853)  --> TIME: 2025-02-24 10:28:48 -- STEP: 22534/36938 -- GLOBAL_STEP: 281100 | > loss_text_ce: 0.03333301842212677 (0.04261905236402463) | > loss_mel_ce: 2.626718044281006 (nan) | > loss: 0.010555759072303772 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1364301667827525) | > loader_time: 0.0045 (0.0046838195658072896)  --> TIME: 2025-02-24 10:28:57 -- STEP: 22584/36938 -- GLOBAL_STEP: 281150 | > loss_text_ce: 0.04134272783994675 (0.04261857061486049) | > loss_mel_ce: 3.4795379638671875 (nan) | > loss: 0.013971749693155289 (nan) | > current_lr: 5e-06 | > step_time: 0.1854 (0.13642626224871526) | > loader_time: 0.0049 (0.004683668916207697)  --> TIME: 2025-02-24 10:29:07 -- STEP: 22634/36938 -- GLOBAL_STEP: 281200 | > loss_text_ce: 0.04041604325175285 (0.04261543299579656) | > loss_mel_ce: 3.6897261142730713 (nan) | > loss: 0.014802152290940285 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.13642908496212883) | > loader_time: 0.0047 (0.004683544507894222)  --> TIME: 2025-02-24 10:29:16 -- STEP: 22684/36938 -- GLOBAL_STEP: 281250 | > loss_text_ce: 0.04682609438896179 (0.042615209052633496) | > loss_mel_ce: 3.714078903198242 (nan) | > loss: 0.014924227260053158 (nan) | > current_lr: 5e-06 | > step_time: 0.1271 (0.13643715644803472) | > loader_time: 0.0046 (0.004683510974651293)  --> TIME: 2025-02-24 10:29:26 -- STEP: 22734/36938 -- GLOBAL_STEP: 281300 | > loss_text_ce: 0.04423128440976143 (0.04261285854287866) | > loss_mel_ce: 3.5556793212890625 (nan) | > loss: 0.014285359531641006 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13644279246071456) | > loader_time: 0.0048 (0.004683528074836888)  --> TIME: 2025-02-24 10:29:35 -- STEP: 22784/36938 -- GLOBAL_STEP: 281350 | > loss_text_ce: 0.038931071758270264 (0.0426124981492107) | > loss_mel_ce: 2.72218656539917 (nan) | > loss: 0.01095681730657816 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13644109643326058) | > loader_time: 0.0067 (0.004683549494974403)  --> TIME: 2025-02-24 10:29:45 -- STEP: 22834/36938 -- GLOBAL_STEP: 281400 | > loss_text_ce: 0.04718102887272835 (0.04261133069861958) | > loss_mel_ce: 4.026322364807129 (nan) | > loss: 0.01616469770669937 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.13644163628598513) | > loader_time: 0.0048 (0.004683376308747556)  --> TIME: 2025-02-24 10:29:54 -- STEP: 22884/36938 -- GLOBAL_STEP: 281450 | > loss_text_ce: 0.040441833436489105 (0.04261169054607644) | > loss_mel_ce: 5.790625095367432 (nan) | > loss: 0.02313915640115738 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13644693595215812) | > loader_time: 0.0046 (0.004683360543373563)  --> TIME: 2025-02-24 10:30:04 -- STEP: 22934/36938 -- GLOBAL_STEP: 281500 | > loss_text_ce: 0.03885867819190025 (0.04261156120697741) | > loss_mel_ce: 2.8579914569854736 (nan) | > loss: 0.0114954374730587 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13645683865585884) | > loader_time: 0.0045 (0.004683335750367262)  --> TIME: 2025-02-24 10:30:13 -- STEP: 22984/36938 -- GLOBAL_STEP: 281550 | > loss_text_ce: 0.04403200000524521 (0.04260937222586748) | > loss_mel_ce: 3.0959036350250244 (nan) | > loss: 0.012460063211619854 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.1364660322189002) | > loader_time: 0.0047 (0.004683221990208246)  --> TIME: 2025-02-24 10:30:22 -- STEP: 23034/36938 -- GLOBAL_STEP: 281600 | > loss_text_ce: 0.03274610638618469 (0.04260933820155487) | > loss_mel_ce: 2.0168352127075195 (nan) | > loss: 0.00813325960189104 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13646353584947724) | > loader_time: 0.0045 (0.004683085041471776)  --> TIME: 2025-02-24 10:30:32 -- STEP: 23084/36938 -- GLOBAL_STEP: 281650 | > loss_text_ce: 0.04006245359778404 (0.04260520092963687) | > loss_mel_ce: 3.4527716636657715 (nan) | > loss: 0.013860453851521015 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.1364694486621186) | > loader_time: 0.0046 (0.004682998137919427)  --> TIME: 2025-02-24 10:30:41 -- STEP: 23134/36938 -- GLOBAL_STEP: 281700 | > loss_text_ce: 0.04483979940414429 (0.042602579510458524) | > loss_mel_ce: 2.808441638946533 (nan) | > loss: 0.011322545818984509 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13646230052889669) | > loader_time: 0.0047 (0.004682947330552364)  --> TIME: 2025-02-24 10:30:51 -- STEP: 23184/36938 -- GLOBAL_STEP: 281750 | > loss_text_ce: 0.04467971995472908 (0.04260119086192022) | > loss_mel_ce: 2.84912109375 (nan) | > loss: 0.011483336798846722 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.13646947235136409) | > loader_time: 0.005 (0.00468285762292917)  --> TIME: 2025-02-24 10:31:00 -- STEP: 23234/36938 -- GLOBAL_STEP: 281800 | > loss_text_ce: 0.042352430522441864 (0.04260073995384104) | > loss_mel_ce: 3.7735517024993896 (nan) | > loss: 0.015142478048801422 (nan) | > current_lr: 5e-06 | > step_time: 0.1764 (0.13646383456686378) | > loader_time: 0.0048 (0.004682742575518787)  --> TIME: 2025-02-24 10:31:09 -- STEP: 23284/36938 -- GLOBAL_STEP: 281850 | > loss_text_ce: 0.0375015065073967 (0.042599027530931526) | > loss_mel_ce: 3.2051515579223633 (nan) | > loss: 0.012867671437561512 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.1364787622141975) | > loader_time: 0.0048 (0.004683011146657484)  --> TIME: 2025-02-24 10:31:19 -- STEP: 23334/36938 -- GLOBAL_STEP: 281900 | > loss_text_ce: 0.037469469010829926 (0.042598281615144916) | > loss_mel_ce: 1.7591829299926758 (nan) | > loss: 0.007129573728889227 (nan) | > current_lr: 5e-06 | > step_time: 0.1123 (0.13648205066264701) | > loader_time: 0.0048 (0.004683115799690451)  --> TIME: 2025-02-24 10:31:28 -- STEP: 23384/36938 -- GLOBAL_STEP: 281950 | > loss_text_ce: 0.04620249196887016 (0.04259434159421504) | > loss_mel_ce: 3.612250328063965 (nan) | > loss: 0.01451767049729824 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13649151788522995) | > loader_time: 0.0047 (0.004683204752265941)  --> TIME: 2025-02-24 10:31:38 -- STEP: 23434/36938 -- GLOBAL_STEP: 282000 | > loss_text_ce: 0.0402601882815361 (0.042593057828490224) | > loss_mel_ce: 3.193971872329712 (nan) | > loss: 0.012834254652261734 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13649013735912915) | > loader_time: 0.0047 (0.004683100130306311)  --> TIME: 2025-02-24 10:31:47 -- STEP: 23484/36938 -- GLOBAL_STEP: 282050 | > loss_text_ce: 0.05151582136750221 (0.04259270977637665) | > loss_mel_ce: 3.241438388824463 (nan) | > loss: 0.013067279011011124 (nan) | > current_lr: 5e-06 | > step_time: 0.1711 (0.13649600268100676) | > loader_time: 0.0049 (0.004683197722323731)  --> TIME: 2025-02-24 10:31:57 -- STEP: 23534/36938 -- GLOBAL_STEP: 282100 | > loss_text_ce: 0.042420294135808945 (0.04259271707000012) | > loss_mel_ce: 3.106804609298706 (nan) | > loss: 0.012496924959123135 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.13649263542632747) | > loader_time: 0.0047 (0.00468315205517709)  --> TIME: 2025-02-24 10:32:06 -- STEP: 23584/36938 -- GLOBAL_STEP: 282150 | > loss_text_ce: 0.05165468528866768 (0.04259143255803737) | > loss_mel_ce: 0.1044921875 (nan) | > loss: 0.0006196304457262158 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13649513810023867) | > loader_time: 0.0046 (0.004683021198215131)  --> TIME: 2025-02-24 10:32:15 -- STEP: 23634/36938 -- GLOBAL_STEP: 282200 | > loss_text_ce: 0.03290246054530144 (0.04259129482419715) | > loss_mel_ce: 1.7660281658172607 (nan) | > loss: 0.007138614077121019 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13649623757380383) | > loader_time: 0.0047 (0.0046828639096714265)  --> TIME: 2025-02-24 10:32:25 -- STEP: 23684/36938 -- GLOBAL_STEP: 282250 | > loss_text_ce: 0.04573025926947594 (0.04259389308356741) | > loss_mel_ce: 3.8036000728607178 (nan) | > loss: 0.015275121666491032 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.13649568977921314) | > loader_time: 0.0046 (0.0046828013480343916)  --> TIME: 2025-02-24 10:32:34 -- STEP: 23734/36938 -- GLOBAL_STEP: 282300 | > loss_text_ce: 0.03270171955227852 (0.04259418674775347) | > loss_mel_ce: 3.949232578277588 (nan) | > loss: 0.01580132730305195 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.1364971136016152) | > loader_time: 0.0046 (0.004682899716831244)  --> TIME: 2025-02-24 10:32:43 -- STEP: 23784/36938 -- GLOBAL_STEP: 282350 | > loss_text_ce: 0.03319434076547623 (0.042589992881339626) | > loss_mel_ce: 2.8252410888671875 (nan) | > loss: 0.011342998594045639 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.1365025029453494) | > loader_time: 0.0046 (0.004682869641338989)  --> TIME: 2025-02-24 10:32:53 -- STEP: 23834/36938 -- GLOBAL_STEP: 282400 | > loss_text_ce: 0.054578643292188644 (0.042589813727370265) | > loss_mel_ce: 3.5877108573913574 (nan) | > loss: 0.014453530311584473 (nan) | > current_lr: 5e-06 | > step_time: 0.1572 (0.13650300229286993) | > loader_time: 0.0047 (0.004682764537264225)  --> TIME: 2025-02-24 10:33:02 -- STEP: 23884/36938 -- GLOBAL_STEP: 282450 | > loss_text_ce: 0.05052783340215683 (0.042590468006287706) | > loss_mel_ce: 3.5125045776367188 (nan) | > loss: 0.014139018021523952 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.1365078083336047) | > loader_time: 0.0046 (0.004682678440431322)  --> TIME: 2025-02-24 10:33:11 -- STEP: 23934/36938 -- GLOBAL_STEP: 282500 | > loss_text_ce: 0.03912464529275894 (0.042585695096303784) | > loss_mel_ce: 3.145568609237671 (nan) | > loss: 0.012637672945857048 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13650260721122073) | > loader_time: 0.0045 (0.004682552787587591)  --> TIME: 2025-02-24 10:33:21 -- STEP: 23984/36938 -- GLOBAL_STEP: 282550 | > loss_text_ce: 0.04525558277964592 (0.042584579865849834) | > loss_mel_ce: 2.8873674869537354 (nan) | > loss: 0.011637394316494465 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13650729462971642) | > loader_time: 0.0046 (0.004682441844074945)  --> TIME: 2025-02-24 10:33:30 -- STEP: 24034/36938 -- GLOBAL_STEP: 282600 | > loss_text_ce: 0.03768104687333107 (0.04258547637829714) | > loss_mel_ce: 2.339254379272461 (nan) | > loss: 0.009432284161448479 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.13649837047375055) | > loader_time: 0.0049 (0.004682328594477902)  --> TIME: 2025-02-24 10:33:39 -- STEP: 24084/36938 -- GLOBAL_STEP: 282650 | > loss_text_ce: 0.04469139873981476 (0.04258424192151153) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13649290526869565) | > loader_time: 0.0048 (0.004682270806605105)  --> TIME: 2025-02-24 10:33:48 -- STEP: 24134/36938 -- GLOBAL_STEP: 282700 | > loss_text_ce: 0.03390033170580864 (0.042586534508068145) | > loss_mel_ce: 0.6195204257965088 (nan) | > loss: 0.0025929396506398916 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13649083691365702) | > loader_time: 0.0045 (0.004682272008293809)  --> TIME: 2025-02-24 10:33:57 -- STEP: 24184/36938 -- GLOBAL_STEP: 282750 | > loss_text_ce: 0.03932049125432968 (0.042583027387090557) | > loss_mel_ce: 3.0316178798675537 (nan) | > loss: 0.01218626368790865 (nan) | > current_lr: 5e-06 | > step_time: 0.1512 (0.13648395024898108) | > loader_time: 0.0047 (0.004682219101425987)  --> TIME: 2025-02-24 10:34:06 -- STEP: 24234/36938 -- GLOBAL_STEP: 282800 | > loss_text_ce: 0.04052147641777992 (0.042578967573636) | > loss_mel_ce: 3.6373050212860107 (nan) | > loss: 0.014594550244510174 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13647611698833192) | > loader_time: 0.0047 (0.0046820642236456295)  --> TIME: 2025-02-24 10:34:16 -- STEP: 24284/36938 -- GLOBAL_STEP: 282850 | > loss_text_ce: 0.04171985015273094 (0.04257939417301606) | > loss_mel_ce: 3.5579373836517334 (nan) | > loss: 0.014284355565905571 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.13648507279317687) | > loader_time: 0.0046 (0.004681999474061888)  --> TIME: 2025-02-24 10:34:25 -- STEP: 24334/36938 -- GLOBAL_STEP: 282900 | > loss_text_ce: 0.03100111149251461 (0.04257469718970877) | > loss_mel_ce: 3.322140693664551 (nan) | > loss: 0.013306118547916412 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13648580342808014) | > loader_time: 0.0046 (0.004681937263644278)  --> TIME: 2025-02-24 10:34:34 -- STEP: 24384/36938 -- GLOBAL_STEP: 282950 | > loss_text_ce: 0.04358886927366257 (0.04257418890036808) | > loss_mel_ce: 3.668456792831421 (nan) | > loss: 0.014730340801179409 (nan) | > current_lr: 5e-06 | > step_time: 0.1949 (0.13647454082379215) | > loader_time: 0.0049 (0.004681834701712677)  --> TIME: 2025-02-24 10:34:43 -- STEP: 24434/36938 -- GLOBAL_STEP: 283000 | > loss_text_ce: 0.06574327498674393 (0.04257565871994216) | > loss_mel_ce: 0.7498576045036316 (nan) | > loss: 0.0032365114893764257 (nan) | > current_lr: 5e-06 | > step_time: 0.1071 (0.1364788007275647) | > loader_time: 0.0045 (0.004681741185300314)  --> TIME: 2025-02-24 10:34:53 -- STEP: 24484/36938 -- GLOBAL_STEP: 283050 | > loss_text_ce: 0.042746901512145996 (0.04257570856020769) | > loss_mel_ce: 2.4918079376220703 (nan) | > loss: 0.010057758539915085 (nan) | > current_lr: 5e-06 | > step_time: 0.1084 (0.13647607862316447) | > loader_time: 0.0045 (0.0046815899944913095)  --> TIME: 2025-02-24 10:35:02 -- STEP: 24534/36938 -- GLOBAL_STEP: 283100 | > loss_text_ce: 0.04054129123687744 (0.042578618539643176) | > loss_mel_ce: 3.4887924194335938 (nan) | > loss: 0.014005293138325214 (nan) | > current_lr: 5e-06 | > step_time: 0.1064 (0.13647672676598935) | > loader_time: 0.0046 (0.004681444871665667)  --> TIME: 2025-02-24 10:35:11 -- STEP: 24584/36938 -- GLOBAL_STEP: 283150 | > loss_text_ce: 0.04783247038722038 (0.042578410579119236) | > loss_mel_ce: 3.372701406478882 (nan) | > loss: 0.013573547825217247 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.1364681567891275) | > loader_time: 0.0045 (0.004681322179320119)  --> TIME: 2025-02-24 10:35:20 -- STEP: 24634/36938 -- GLOBAL_STEP: 283200 | > loss_text_ce: 0.0490935854613781 (0.04257904332710492) | > loss_mel_ce: 3.0151541233062744 (nan) | > loss: 0.012159713543951511 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13647442017126543) | > loader_time: 0.0046 (0.00468123266911211)  --> TIME: 2025-02-24 10:35:30 -- STEP: 24684/36938 -- GLOBAL_STEP: 283250 | > loss_text_ce: 0.04562518373131752 (0.04257790013829168) | > loss_mel_ce: 3.4322805404663086 (nan) | > loss: 0.013801214285194874 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1364718954657028) | > loader_time: 0.0045 (0.004681149481027104)  --> TIME: 2025-02-24 10:35:39 -- STEP: 24734/36938 -- GLOBAL_STEP: 283300 | > loss_text_ce: 0.04336738586425781 (0.04257986969200425) | > loss_mel_ce: 1.3650766611099243 (nan) | > loss: 0.005589064210653305 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13647621309515331) | > loader_time: 0.0046 (0.004681081290656471)  --> TIME: 2025-02-24 10:35:48 -- STEP: 24784/36938 -- GLOBAL_STEP: 283350 | > loss_text_ce: 0.030669018626213074 (0.04257598102578696) | > loss_mel_ce: 3.191924571990967 (nan) | > loss: 0.012788070365786552 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.136477394170189) | > loader_time: 0.0047 (0.004681013991095473)  --> TIME: 2025-02-24 10:35:58 -- STEP: 24834/36938 -- GLOBAL_STEP: 283400 | > loss_text_ce: 0.04092361778020859 (0.04257676853728051) | > loss_mel_ce: 3.2558155059814453 (nan) | > loss: 0.013082298450171947 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.1364806860822161) | > loader_time: 0.0047 (0.00468106775590262)  --> TIME: 2025-02-24 10:36:07 -- STEP: 24884/36938 -- GLOBAL_STEP: 283450 | > loss_text_ce: 0.05779970437288284 (0.042574514844009446) | > loss_mel_ce: 3.285229206085205 (nan) | > loss: 0.013265988789498806 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.13647042077140775) | > loader_time: 0.0048 (0.004681017281560349)  --> TIME: 2025-02-24 10:36:16 -- STEP: 24934/36938 -- GLOBAL_STEP: 283500 | > loss_text_ce: 0.037165749818086624 (0.04257280770615293) | > loss_mel_ce: 2.6693522930145264 (nan) | > loss: 0.010740150697529316 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13647777587474252) | > loader_time: 0.0046 (0.004680964227111708)  --> TIME: 2025-02-24 10:36:26 -- STEP: 24984/36938 -- GLOBAL_STEP: 283550 | > loss_text_ce: 0.04034354165196419 (0.04257269201743119) | > loss_mel_ce: 3.2338132858276367 (nan) | > loss: 0.012992686592042446 (nan) | > current_lr: 5e-06 | > step_time: 0.1833 (0.13647878202459188) | > loader_time: 0.005 (0.004680950462989926)  --> TIME: 2025-02-24 10:36:35 -- STEP: 25034/36938 -- GLOBAL_STEP: 283600 | > loss_text_ce: 0.043805044144392014 (0.042574396038376906) | > loss_mel_ce: 3.205629587173462 (nan) | > loss: 0.0128945829346776 (nan) | > current_lr: 5e-06 | > step_time: 0.1515 (0.13647822356789838) | > loader_time: 0.0046 (0.004681048286964013)  --> TIME: 2025-02-24 10:36:44 -- STEP: 25084/36938 -- GLOBAL_STEP: 283650 | > loss_text_ce: 0.042286887764930725 (0.04257319997511198) | > loss_mel_ce: 1.8668607473373413 (nan) | > loss: 0.007575983181595802 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.13647919925581173) | > loader_time: 0.0048 (0.00468095554877385)  --> TIME: 2025-02-24 10:36:53 -- STEP: 25134/36938 -- GLOBAL_STEP: 283700 | > loss_text_ce: 0.04817899689078331 (0.042574576217547185) | > loss_mel_ce: 3.6822493076324463 (nan) | > loss: 0.014803287573158741 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13646835803492371) | > loader_time: 0.0048 (0.00468085683349248)  --> TIME: 2025-02-24 10:37:02 -- STEP: 25184/36938 -- GLOBAL_STEP: 283750 | > loss_text_ce: 0.04566894471645355 (0.04257280977064622) | > loss_mel_ce: 3.186662197113037 (nan) | > loss: 0.012826710939407349 (nan) | > current_lr: 5e-06 | > step_time: 0.1642 (0.1364504464888971) | > loader_time: 0.0048 (0.00468082518673004)  --> TIME: 2025-02-24 10:37:11 -- STEP: 25234/36938 -- GLOBAL_STEP: 283800 | > loss_text_ce: 0.03861771151423454 (0.04257130511790801) | > loss_mel_ce: 0.00799560546875 (nan) | > loss: 0.00018497348355595022 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.1364456636928161) | > loader_time: 0.0046 (0.004680721848799193)  --> TIME: 2025-02-24 10:37:20 -- STEP: 25284/36938 -- GLOBAL_STEP: 283850 | > loss_text_ce: 0.058877132833004 (0.04257075016272178) | > loss_mel_ce: 2.1563944816589355 (nan) | > loss: 0.008790761232376099 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13643312932305535) | > loader_time: 0.0045 (0.004680559899568969)  --> TIME: 2025-02-24 10:37:30 -- STEP: 25334/36938 -- GLOBAL_STEP: 283900 | > loss_text_ce: 0.034062180668115616 (0.0425731744938154) | > loss_mel_ce: 3.4751124382019043 (nan) | > loss: 0.013925297185778618 (nan) | > current_lr: 5e-06 | > step_time: 0.1853 (0.13644164869814882) | > loader_time: 0.0049 (0.004680484201571759)  --> TIME: 2025-02-24 10:37:39 -- STEP: 25384/36938 -- GLOBAL_STEP: 283950 | > loss_text_ce: 0.03632880002260208 (0.04257100714923737) | > loss_mel_ce: 3.289984941482544 (nan) | > loss: 0.013199658133089542 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13644813784957888) | > loader_time: 0.0046 (0.004680584600735938)  --> TIME: 2025-02-24 10:37:49 -- STEP: 25434/36938 -- GLOBAL_STEP: 284000 | > loss_text_ce: 0.042317647486925125 (0.04257170307168996) | > loss_mel_ce: 1.270745873451233 (nan) | > loss: 0.005210569594055414 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13645561766753825) | > loader_time: 0.005 (0.004680690904491165)  --> TIME: 2025-02-24 10:37:58 -- STEP: 25484/36938 -- GLOBAL_STEP: 284050 | > loss_text_ce: 0.03801083564758301 (0.04257158708791982) | > loss_mel_ce: 3.0601894855499268 (nan) | > loss: 0.012294446118175983 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13645993069976725) | > loader_time: 0.0045 (0.004680773579817956)  --> TIME: 2025-02-24 10:38:08 -- STEP: 25534/36938 -- GLOBAL_STEP: 284100 | > loss_text_ce: 0.030639873817563057 (0.04256732864712425) | > loss_mel_ce: 2.617971897125244 (nan) | > loss: 0.010510364547371864 (nan) | > current_lr: 5e-06 | > step_time: 0.1503 (0.13645507784749566) | > loader_time: 0.0046 (0.0046806842464525485)  --> TIME: 2025-02-24 10:38:17 -- STEP: 25584/36938 -- GLOBAL_STEP: 284150 | > loss_text_ce: 0.03569207713007927 (0.042566522104912304) | > loss_mel_ce: 1.7053303718566895 (nan) | > loss: 0.006908819545060396 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13644813656285107) | > loader_time: 0.0045 (0.004680601906746599)  --> TIME: 2025-02-24 10:38:26 -- STEP: 25634/36938 -- GLOBAL_STEP: 284200 | > loss_text_ce: 0.05327758565545082 (0.042567071858287765) | > loss_mel_ce: 3.0610413551330566 (nan) | > loss: 0.012358408421278 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13643339239807872) | > loader_time: 0.0045 (0.004680482991690222)  --> TIME: 2025-02-24 10:38:35 -- STEP: 25684/36938 -- GLOBAL_STEP: 284250 | > loss_text_ce: 0.04522814229130745 (0.04256538012704054) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.1364304848018791) | > loader_time: 0.0045 (0.004680343217111555)  --> TIME: 2025-02-24 10:38:44 -- STEP: 25734/36938 -- GLOBAL_STEP: 284300 | > loss_text_ce: 0.042217567563056946 (0.04256722721450504) | > loss_mel_ce: 4.512422561645508 (nan) | > loss: 0.018073970451951027 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13643120127679642) | > loader_time: 0.0047 (0.004680214019387597)  --> TIME: 2025-02-24 10:38:53 -- STEP: 25784/36938 -- GLOBAL_STEP: 284350 | > loss_text_ce: 0.04789038002490997 (0.04256538756130615) | > loss_mel_ce: 2.653197765350342 (nan) | > loss: 0.010718604549765587 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13642856506431406) | > loader_time: 0.0047 (0.004680155339242498)  --> TIME: 2025-02-24 10:39:03 -- STEP: 25834/36938 -- GLOBAL_STEP: 284400 | > loss_text_ce: 0.04209388047456741 (0.04256406563056652) | > loss_mel_ce: 2.8635122776031494 (nan) | > loss: 0.01153018418699503 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13642418913366144) | > loader_time: 0.0047 (0.004680084021198322)  --> TIME: 2025-02-24 10:39:12 -- STEP: 25884/36938 -- GLOBAL_STEP: 284450 | > loss_text_ce: 0.036129843443632126 (0.04256200471922745) | > loss_mel_ce: 3.4370980262756348 (nan) | > loss: 0.013782651163637638 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13641840297791527) | > loader_time: 0.0047 (0.004679972090873657)  --> TIME: 2025-02-24 10:39:21 -- STEP: 25934/36938 -- GLOBAL_STEP: 284500 | > loss_text_ce: 0.03959428891539574 (0.0425617971027755) | > loss_mel_ce: 2.5026135444641113 (nan) | > loss: 0.010088126175105572 (nan) | > current_lr: 5e-06 | > step_time: 0.1492 (0.13642548285628683) | > loader_time: 0.0046 (0.004679940987398177)  --> TIME: 2025-02-24 10:39:30 -- STEP: 25984/36938 -- GLOBAL_STEP: 284550 | > loss_text_ce: 0.04545789957046509 (0.04256485616400456) | > loss_mel_ce: 3.0544135570526123 (nan) | > loss: 0.01230107806622982 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.1364195007794156) | > loader_time: 0.0047 (0.004679817853155977)  --> TIME: 2025-02-24 10:39:40 -- STEP: 26034/36938 -- GLOBAL_STEP: 284600 | > loss_text_ce: 0.03756926208734512 (0.04256103968548553) | > loss_mel_ce: 3.0938172340393066 (nan) | > loss: 0.012426137924194336 (nan) | > current_lr: 5e-06 | > step_time: 0.1752 (0.13641806049739622) | > loader_time: 0.0049 (0.00467976664236689)  --> TIME: 2025-02-24 10:39:49 -- STEP: 26084/36938 -- GLOBAL_STEP: 284650 | > loss_text_ce: 0.047769103199243546 (0.042563459481291324) | > loss_mel_ce: 3.167851448059082 (nan) | > loss: 0.012760399840772152 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.1364126422437783) | > loader_time: 0.005 (0.004679811556561186)  --> TIME: 2025-02-24 10:39:58 -- STEP: 26134/36938 -- GLOBAL_STEP: 284700 | > loss_text_ce: 0.046022433787584305 (0.04256670840893215) | > loss_mel_ce: 3.371356725692749 (nan) | > loss: 0.013561028987169266 (nan) | > current_lr: 5e-06 | > step_time: 0.1786 (0.13641065533819663) | > loader_time: 0.0048 (0.004679759860431036)  --> TIME: 2025-02-24 10:40:07 -- STEP: 26184/36938 -- GLOBAL_STEP: 284750 | > loss_text_ce: 0.04032168164849281 (0.042566576542759874) | > loss_mel_ce: 3.736589193344116 (nan) | > loss: 0.014987742528319359 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13640243528889426) | > loader_time: 0.0046 (0.004679830147882787)  --> TIME: 2025-02-24 10:40:17 -- STEP: 26234/36938 -- GLOBAL_STEP: 284800 | > loss_text_ce: 0.043827347457408905 (0.0425662811786397) | > loss_mel_ce: 3.330751895904541 (nan) | > loss: 0.013391188345849514 (nan) | > current_lr: 5e-06 | > step_time: 0.1869 (0.13641345401885202) | > loader_time: 0.0049 (0.004679904229813446)  --> TIME: 2025-02-24 10:40:26 -- STEP: 26284/36938 -- GLOBAL_STEP: 284850 | > loss_text_ce: 0.0442931242287159 (0.0425663617540694) | > loss_mel_ce: 3.1936209201812744 (nan) | > loss: 0.012848866172134876 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.1364117206603385) | > loader_time: 0.0046 (0.004679985313795767)  --> TIME: 2025-02-24 10:40:35 -- STEP: 26334/36938 -- GLOBAL_STEP: 284900 | > loss_text_ce: 0.045541953295469284 (0.042566301325205814) | > loss_mel_ce: 2.5652925968170166 (nan) | > loss: 0.010360455140471458 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13642024632656233) | > loader_time: 0.0046 (0.004680033877017756)  --> TIME: 2025-02-24 10:40:45 -- STEP: 26384/36938 -- GLOBAL_STEP: 284950 | > loss_text_ce: 0.04113558679819107 (0.04256671383095738) | > loss_mel_ce: 3.626962184906006 (nan) | > loss: 0.014555944129824638 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13643115085677884) | > loader_time: 0.0046 (0.004680008879931051)  --> TIME: 2025-02-24 10:40:54 -- STEP: 26434/36938 -- GLOBAL_STEP: 285000 | > loss_text_ce: 0.049016740173101425 (0.04256621381440401) | > loss_mel_ce: 3.8245298862457275 (nan) | > loss: 0.015371217392385006 (nan) | > current_lr: 5e-06 | > step_time: 0.1791 (0.1364336244960937) | > loader_time: 0.0049 (0.004680004947492341)  --> TIME: 2025-02-24 10:41:04 -- STEP: 26484/36938 -- GLOBAL_STEP: 285050 | > loss_text_ce: 0.05084316059947014 (0.042565161523172454) | > loss_mel_ce: 3.757866144180298 (nan) | > loss: 0.01511392742395401 (nan) | > current_lr: 5e-06 | > step_time: 0.1798 (0.13643763171025905) | > loader_time: 0.0048 (0.0046799328189982775)  --> TIME: 2025-02-24 10:41:13 -- STEP: 26534/36938 -- GLOBAL_STEP: 285100 | > loss_text_ce: 0.02364508807659149 (0.04256134439912473) | > loss_mel_ce: 2.68963885307312 (nan) | > loss: 0.01076700072735548 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13643187946244273) | > loader_time: 0.0045 (0.0046798166373648)  --> TIME: 2025-02-24 10:41:22 -- STEP: 26584/36938 -- GLOBAL_STEP: 285150 | > loss_text_ce: 0.037583161145448685 (0.04256111656952809) | > loss_mel_ce: 2.042294502258301 (nan) | > loss: 0.008253483101725578 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13643577841079446) | > loader_time: 0.0044 (0.004679792846788156)  --> TIME: 2025-02-24 10:41:32 -- STEP: 26634/36938 -- GLOBAL_STEP: 285200 | > loss_text_ce: 0.019572466611862183 (0.04255628000485101) | > loss_mel_ce: 2.3227951526641846 (nan) | > loss: 0.00929511059075594 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13644342095928735) | > loader_time: 0.0045 (0.004679764615994366)  --> TIME: 2025-02-24 10:41:41 -- STEP: 26684/36938 -- GLOBAL_STEP: 285250 | > loss_text_ce: 0.04354016110301018 (0.04255303680962235) | > loss_mel_ce: 2.475135326385498 (nan) | > loss: 0.009994745254516602 (nan) | > current_lr: 5e-06 | > step_time: 0.1856 (0.13644990376349714) | > loader_time: 0.0048 (0.0046797273774104725)  --> TIME: 2025-02-24 10:41:51 -- STEP: 26734/36938 -- GLOBAL_STEP: 285300 | > loss_text_ce: 0.04720134660601616 (0.0425540023255365) | > loss_mel_ce: 3.3981075286865234 (nan) | > loss: 0.013671861961483955 (nan) | > current_lr: 5e-06 | > step_time: 0.1445 (0.1364497593527043) | > loader_time: 0.0046 (0.004679621001710388)  --> TIME: 2025-02-24 10:42:00 -- STEP: 26784/36938 -- GLOBAL_STEP: 285350 | > loss_text_ce: 0.03531336784362793 (0.04255234994343482) | > loss_mel_ce: 2.7811968326568604 (nan) | > loss: 0.011176628060638905 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1364402145663616) | > loader_time: 0.0048 (0.0046794995078049085)  --> TIME: 2025-02-24 10:42:09 -- STEP: 26834/36938 -- GLOBAL_STEP: 285400 | > loss_text_ce: 0.042397331446409225 (0.04255461230900578) | > loss_mel_ce: 3.604961633682251 (nan) | > loss: 0.014473646879196167 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13644203280002348) | > loader_time: 0.0047 (0.004679548195739898)  --> TIME: 2025-02-24 10:42:19 -- STEP: 26884/36938 -- GLOBAL_STEP: 285450 | > loss_text_ce: 0.044737137854099274 (0.04255482587564125) | > loss_mel_ce: 2.777587890625 (nan) | > loss: 0.011199703440070152 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13644225834877896) | > loader_time: 0.0045 (0.004679620665039538)  --> TIME: 2025-02-24 10:42:28 -- STEP: 26934/36938 -- GLOBAL_STEP: 285500 | > loss_text_ce: 0.02573815919458866 (0.04255435197727065) | > loss_mel_ce: 2.643397569656372 (nan) | > loss: 0.010591809637844563 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13644360974591516) | > loader_time: 0.0046 (0.004679623581018154)  --> TIME: 2025-02-24 10:42:37 -- STEP: 26984/36938 -- GLOBAL_STEP: 285550 | > loss_text_ce: 0.04121709242463112 (0.04255133559727854) | > loss_mel_ce: 2.7443594932556152 (nan) | > loss: 0.011053876020014286 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13643340986877023) | > loader_time: 0.0047 (0.004679644837635325)  --> TIME: 2025-02-24 10:42:46 -- STEP: 27034/36938 -- GLOBAL_STEP: 285600 | > loss_text_ce: 0.04264814406633377 (0.042549215813570525) | > loss_mel_ce: 1.2127091884613037 (nan) | > loss: 0.004981577396392822 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13642120871078645) | > loader_time: 0.0045 (0.004679643606005427)  --> TIME: 2025-02-24 10:42:55 -- STEP: 27084/36938 -- GLOBAL_STEP: 285650 | > loss_text_ce: 0.05115027353167534 (0.04254649469423024) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13641731760347336) | > loader_time: 0.0045 (0.004679602255164171)  --> TIME: 2025-02-24 10:43:04 -- STEP: 27134/36938 -- GLOBAL_STEP: 285700 | > loss_text_ce: 0.040815141052007675 (0.04254505808275788) | > loss_mel_ce: 3.2734375 (nan) | > loss: 0.013151796534657478 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.13641999516831957) | > loader_time: 0.0046 (0.0046796291888643185)  --> TIME: 2025-02-24 10:43:14 -- STEP: 27184/36938 -- GLOBAL_STEP: 285750 | > loss_text_ce: 0.03403447940945625 (0.04254521081831033) | > loss_mel_ce: 3.017381191253662 (nan) | > loss: 0.01210879348218441 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13641663048287048) | > loader_time: 0.0045 (0.004679543751699208)  --> TIME: 2025-02-24 10:43:23 -- STEP: 27234/36938 -- GLOBAL_STEP: 285800 | > loss_text_ce: 0.020888676866889 (0.04254688831837407) | > loss_mel_ce: 2.515495777130127 (nan) | > loss: 0.010065018199384212 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13642761726040467) | > loader_time: 0.0047 (0.004679449068394278)  --> TIME: 2025-02-24 10:43:33 -- STEP: 27284/36938 -- GLOBAL_STEP: 285850 | > loss_text_ce: 0.048529136925935745 (0.04254329224251453) | > loss_mel_ce: 2.965531587600708 (nan) | > loss: 0.011960559524595737 (nan) | > current_lr: 5e-06 | > step_time: 0.1493 (0.1364294165670833) | > loader_time: 0.0047 (0.004679510555314865)  --> TIME: 2025-02-24 10:43:42 -- STEP: 27334/36938 -- GLOBAL_STEP: 285900 | > loss_text_ce: 0.053720034658908844 (0.042542653435618645) | > loss_mel_ce: 3.6489994525909424 (nan) | > loss: 0.01469333190470934 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13642975922395684) | > loader_time: 0.0046 (0.004679451997429966)  --> TIME: 2025-02-24 10:43:52 -- STEP: 27384/36938 -- GLOBAL_STEP: 285950 | > loss_text_ce: 0.04369870945811272 (0.04254091014773913) | > loss_mel_ce: 3.128714084625244 (nan) | > loss: 0.012588940560817719 (nan) | > current_lr: 5e-06 | > step_time: 0.1591 (0.13644152838794496) | > loader_time: 0.0048 (0.004679465760552245)  --> TIME: 2025-02-24 10:44:01 -- STEP: 27434/36938 -- GLOBAL_STEP: 286000 | > loss_text_ce: 0.04088938608765602 (0.04253956259061877) | > loss_mel_ce: 3.440135955810547 (nan) | > loss: 0.013813593424856663 (nan) | > current_lr: 5e-06 | > step_time: 0.1565 (0.13644297382631213) | > loader_time: 0.0047 (0.004679499818257214)  --> TIME: 2025-02-24 10:44:11 -- STEP: 27484/36938 -- GLOBAL_STEP: 286050 | > loss_text_ce: 0.04230707138776779 (0.04253514812722625) | > loss_mel_ce: 3.0245423316955566 (nan) | > loss: 0.012170038186013699 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.1364513849122544) | > loader_time: 0.0046 (0.004679499937620171)  --> TIME: 2025-02-24 10:44:20 -- STEP: 27534/36938 -- GLOBAL_STEP: 286100 | > loss_text_ce: 0.04301416128873825 (0.04253751094513718) | > loss_mel_ce: 3.4019978046417236 (nan) | > loss: 0.013670682907104492 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.1364496850911994) | > loader_time: 0.0045 (0.004679388129530169)  --> TIME: 2025-02-24 10:44:29 -- STEP: 27584/36938 -- GLOBAL_STEP: 286150 | > loss_text_ce: 0.03360159695148468 (0.0425360864148869) | > loss_mel_ce: 3.225339412689209 (nan) | > loss: 0.0129323061555624 (nan) | > current_lr: 5e-06 | > step_time: 0.171 (0.13645873388184787) | > loader_time: 0.005 (0.004679376704579433)  --> TIME: 2025-02-24 10:44:39 -- STEP: 27634/36938 -- GLOBAL_STEP: 286200 | > loss_text_ce: 0.03556762635707855 (0.042536710760321395) | > loss_mel_ce: 2.9241483211517334 (nan) | > loss: 0.011744905263185501 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.1364551606688702) | > loader_time: 0.0045 (0.004679460260468947)  --> TIME: 2025-02-24 10:44:48 -- STEP: 27684/36938 -- GLOBAL_STEP: 286250 | > loss_text_ce: 0.03945927694439888 (0.04253451566450328) | > loss_mel_ce: 3.3558247089385986 (nan) | > loss: 0.013473349623382092 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13645134329055203) | > loader_time: 0.0047 (0.004679474324577886)  --> TIME: 2025-02-24 10:44:57 -- STEP: 27734/36938 -- GLOBAL_STEP: 286300 | > loss_text_ce: 0.04057693853974342 (0.04253648004371098) | > loss_mel_ce: 2.8187026977539062 (nan) | > loss: 0.011346348561346531 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13644336405025317) | > loader_time: 0.0047 (0.004679333684834191)  --> TIME: 2025-02-24 10:45:06 -- STEP: 27784/36938 -- GLOBAL_STEP: 286350 | > loss_text_ce: 0.0388636477291584 (0.042536671976963786) | > loss_mel_ce: 2.864201307296753 (nan) | > loss: 0.011520099826157093 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13643636094744682) | > loader_time: 0.0046 (0.0046792899690441805)  --> TIME: 2025-02-24 10:45:15 -- STEP: 27834/36938 -- GLOBAL_STEP: 286400 | > loss_text_ce: 0.04042334482073784 (0.04253923068907956) | > loss_mel_ce: 2.7292208671569824 (nan) | > loss: 0.01099065225571394 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13643344607411131) | > loader_time: 0.0045 (0.004679349010648704)  --> TIME: 2025-02-24 10:45:24 -- STEP: 27884/36938 -- GLOBAL_STEP: 286450 | > loss_text_ce: 0.03774341195821762 (0.04253694889053629) | > loss_mel_ce: 3.6864802837371826 (nan) | > loss: 0.014778666198253632 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.13643601439704092) | > loader_time: 0.0046 (0.0046793114008648285)  --> TIME: 2025-02-24 10:45:34 -- STEP: 27934/36938 -- GLOBAL_STEP: 286500 | > loss_text_ce: 0.03926993906497955 (0.042535634786366676) | > loss_mel_ce: 3.4591753482818604 (nan) | > loss: 0.013882719911634922 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.136447775378432) | > loader_time: 0.0047 (0.004679328089258134)  --> TIME: 2025-02-24 10:45:44 -- STEP: 27984/36938 -- GLOBAL_STEP: 286550 | > loss_text_ce: 0.03892974928021431 (0.042535525178045555) | > loss_mel_ce: 2.4956119060516357 (nan) | > loss: 0.010057705454528332 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1364537916008308) | > loader_time: 0.0045 (0.004679337501730348)  --> TIME: 2025-02-24 10:45:53 -- STEP: 28034/36938 -- GLOBAL_STEP: 286600 | > loss_text_ce: 0.04713495448231697 (0.04253324819456319) | > loss_mel_ce: 2.921051502227783 (nan) | > loss: 0.011778517626225948 (nan) | > current_lr: 5e-06 | > step_time: 0.1443 (0.13645532848645034) | > loader_time: 0.0047 (0.004679317922388726)  --> TIME: 2025-02-24 10:46:03 -- STEP: 28084/36938 -- GLOBAL_STEP: 286650 | > loss_text_ce: 0.04189250245690346 (0.04253067355612928) | > loss_mel_ce: 3.4600918292999268 (nan) | > loss: 0.013896764256060123 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13646328020530527) | > loader_time: 0.0045 (0.004679310068821326)  --> TIME: 2025-02-24 10:46:11 -- STEP: 28134/36938 -- GLOBAL_STEP: 286700 | > loss_text_ce: 0.03462783992290497 (0.04252738127978286) | > loss_mel_ce: 3.5521180629730225 (nan) | > loss: 0.014233119785785675 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13645256414777548) | > loader_time: 0.0048 (0.0046793247681059915)  --> TIME: 2025-02-24 10:46:21 -- STEP: 28184/36938 -- GLOBAL_STEP: 286750 | > loss_text_ce: 0.04537393897771835 (0.04252493934871208) | > loss_mel_ce: 2.6652069091796875 (nan) | > loss: 0.010756273753941059 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13645046391914867) | > loader_time: 0.0045 (0.00467921610315719)  --> TIME: 2025-02-24 10:46:30 -- STEP: 28234/36938 -- GLOBAL_STEP: 286800 | > loss_text_ce: 0.05346114933490753 (0.0425257897284535) | > loss_mel_ce: 4.174328327178955 (nan) | > loss: 0.01677694357931614 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13644346788397915) | > loader_time: 0.0046 (0.004679425762371335)  --> TIME: 2025-02-24 10:46:39 -- STEP: 28284/36938 -- GLOBAL_STEP: 286850 | > loss_text_ce: 0.04627634584903717 (0.04252590535932594) | > loss_mel_ce: 1.1144671440124512 (nan) | > loss: 0.004606124944984913 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13644001609010328) | > loader_time: 0.0045 (0.004679348669064255)  --> TIME: 2025-02-24 10:46:48 -- STEP: 28334/36938 -- GLOBAL_STEP: 286900 | > loss_text_ce: 0.044012680649757385 (0.04252525246271216) | > loss_mel_ce: 2.9021434783935547 (nan) | > loss: 0.011691097170114517 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13644459948186713) | > loader_time: 0.0046 (0.00467929529926641)  --> TIME: 2025-02-24 10:46:57 -- STEP: 28384/36938 -- GLOBAL_STEP: 286950 | > loss_text_ce: 0.037759311497211456 (0.042524696540659955) | > loss_mel_ce: 3.839062452316284 (nan) | > loss: 0.015384213998913765 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.1364406432532275) | > loader_time: 0.0047 (0.004679248618904868)  --> TIME: 2025-02-24 10:47:07 -- STEP: 28434/36938 -- GLOBAL_STEP: 287000 | > loss_text_ce: 0.04392440989613533 (0.04252096230707622) | > loss_mel_ce: 2.1423542499542236 (nan) | > loss: 0.008675709366798401 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1364427213889521) | > loader_time: 0.0047 (0.004679246249644652)  --> TIME: 2025-02-24 10:47:16 -- STEP: 28484/36938 -- GLOBAL_STEP: 287050 | > loss_text_ce: 0.03456138074398041 (0.042521610211447854) | > loss_mel_ce: 3.9338290691375732 (nan) | > loss: 0.01574758253991604 (nan) | > current_lr: 5e-06 | > step_time: 0.1566 (0.136446970983296) | > loader_time: 0.0047 (0.004679261934989919)  --> TIME: 2025-02-24 10:47:26 -- STEP: 28534/36938 -- GLOBAL_STEP: 287100 | > loss_text_ce: 0.03721627593040466 (0.042521598633614266) | > loss_mel_ce: 3.0456252098083496 (nan) | > loss: 0.012233498506247997 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13644786377197826) | > loader_time: 0.0045 (0.004679232478567913)  --> TIME: 2025-02-24 10:47:35 -- STEP: 28584/36938 -- GLOBAL_STEP: 287150 | > loss_text_ce: 0.047625258564949036 (0.042520066122709504) | > loss_mel_ce: 2.4115421772003174 (nan) | > loss: 0.009758601896464825 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1364500596264385) | > loader_time: 0.0047 (0.004679196202184445)  --> TIME: 2025-02-24 10:47:44 -- STEP: 28634/36938 -- GLOBAL_STEP: 287200 | > loss_text_ce: 0.04332665726542473 (0.04251986736922976) | > loss_mel_ce: 2.8557541370391846 (nan) | > loss: 0.01150428969413042 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13644263422455064) | > loader_time: 0.0047 (0.004679219894438482)  --> TIME: 2025-02-24 10:47:53 -- STEP: 28684/36938 -- GLOBAL_STEP: 287250 | > loss_text_ce: 0.05197261646389961 (0.042521347066274806) | > loss_mel_ce: 3.3931379318237305 (nan) | > loss: 0.01367107406258583 (nan) | > current_lr: 5e-06 | > step_time: 0.1513 (0.1364394608459588) | > loader_time: 0.0047 (0.004679211420155748)  --> TIME: 2025-02-24 10:48:02 -- STEP: 28734/36938 -- GLOBAL_STEP: 287300 | > loss_text_ce: 0.04309815913438797 (0.0425196732920766) | > loss_mel_ce: 3.7691099643707275 (nan) | > loss: 0.015127810649573803 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.13643148442088718) | > loader_time: 0.0047 (0.004679298528659442)  --> TIME: 2025-02-24 10:48:12 -- STEP: 28784/36938 -- GLOBAL_STEP: 287350 | > loss_text_ce: 0.0407639741897583 (0.04251874098635142) | > loss_mel_ce: 2.6733429431915283 (nan) | > loss: 0.010770266875624657 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13643554499939228) | > loader_time: 0.0047 (0.004679284215371028)  --> TIME: 2025-02-24 10:48:21 -- STEP: 28834/36938 -- GLOBAL_STEP: 287400 | > loss_text_ce: 0.035048957914114 (0.04251776609797564) | > loss_mel_ce: 2.5621330738067627 (nan) | > loss: 0.010306278243660927 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13643393984877542) | > loader_time: 0.0046 (0.004679309004610114)  --> TIME: 2025-02-24 10:48:30 -- STEP: 28884/36938 -- GLOBAL_STEP: 287450 | > loss_text_ce: 0.046412814408540726 (0.042522115663619756) | > loss_mel_ce: 2.653569221496582 (nan) | > loss: 0.01071421429514885 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13643013575849358) | > loader_time: 0.0046 (0.004679426990409181)  --> TIME: 2025-02-24 10:48:40 -- STEP: 28934/36938 -- GLOBAL_STEP: 287500 | > loss_text_ce: 0.029980570077896118 (0.042519695153877705) | > loss_mel_ce: 3.713402271270752 (nan) | > loss: 0.014854694716632366 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13643920361447767) | > loader_time: 0.0046 (0.004679572271593714)  --> TIME: 2025-02-24 10:48:49 -- STEP: 28984/36938 -- GLOBAL_STEP: 287550 | > loss_text_ce: 0.035630207508802414 (0.04252038619434371) | > loss_mel_ce: 3.621469259262085 (nan) | > loss: 0.014512300491333008 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.136438472508529) | > loader_time: 0.0047 (0.004679558629526336)  --> TIME: 2025-02-24 10:48:58 -- STEP: 29034/36938 -- GLOBAL_STEP: 287600 | > loss_text_ce: 0.040062323212623596 (0.04251772962324411) | > loss_mel_ce: 3.9320576190948486 (nan) | > loss: 0.01576238125562668 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.1364333534188171) | > loader_time: 0.0048 (0.004679562574642663)  --> TIME: 2025-02-24 10:49:08 -- STEP: 29084/36938 -- GLOBAL_STEP: 287650 | > loss_text_ce: 0.041118863970041275 (0.04251885681803524) | > loss_mel_ce: 3.6792335510253906 (nan) | > loss: 0.01476330403238535 (nan) | > current_lr: 5e-06 | > step_time: 0.1879 (0.13644375840106965) | > loader_time: 0.0048 (0.004679548225578581)  --> TIME: 2025-02-24 10:49:17 -- STEP: 29134/36938 -- GLOBAL_STEP: 287700 | > loss_text_ce: 0.042863063514232635 (0.04251818855532143) | > loss_mel_ce: 3.662003993988037 (nan) | > loss: 0.014701854437589645 (nan) | > current_lr: 5e-06 | > step_time: 0.1449 (0.13644339643526746) | > loader_time: 0.0046 (0.004679600032215343)  --> TIME: 2025-02-24 10:49:27 -- STEP: 29184/36938 -- GLOBAL_STEP: 287750 | > loss_text_ce: 0.04162823408842087 (0.04251710590027582) | > loss_mel_ce: 0.024658203125 (nan) | > loss: 0.0002630414383020252 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1364445054511495) | > loader_time: 0.0046 (0.004679698970887738)  --> TIME: 2025-02-24 10:49:36 -- STEP: 29234/36938 -- GLOBAL_STEP: 287800 | > loss_text_ce: 0.027301035821437836 (0.04251465193805337) | > loss_mel_ce: 2.389942169189453 (nan) | > loss: 0.009592236019670963 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.136448993144271) | > loader_time: 0.0045 (0.004679793428117107)  --> TIME: 2025-02-24 10:49:45 -- STEP: 29284/36938 -- GLOBAL_STEP: 287850 | > loss_text_ce: 0.042130470275878906 (0.04251479023818566) | > loss_mel_ce: 4.640625 (nan) | > loss: 0.018582364544272423 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.136441728421011) | > loader_time: 0.0046 (0.004679721726564865)  --> TIME: 2025-02-24 10:49:54 -- STEP: 29334/36938 -- GLOBAL_STEP: 287900 | > loss_text_ce: 0.0492505244910717 (0.042513476627446235) | > loss_mel_ce: 3.34765625 (nan) | > loss: 0.013479789718985558 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13643479948680437) | > loader_time: 0.0048 (0.0046796842026809605)  --> TIME: 2025-02-24 10:50:03 -- STEP: 29384/36938 -- GLOBAL_STEP: 287950 | > loss_text_ce: 0.04995567351579666 (0.04251014514730012) | > loss_mel_ce: 2.6373116970062256 (nan) | > loss: 0.010663759894669056 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13643242579826806) | > loader_time: 0.0047 (0.004679649792410773)  --> TIME: 2025-02-24 10:50:13 -- STEP: 29434/36938 -- GLOBAL_STEP: 288000 | > loss_text_ce: 0.03847741708159447 (0.04250962532124084) | > loss_mel_ce: 2.848184108734131 (nan) | > loss: 0.011455006897449493 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.136432929119031) | > loader_time: 0.0047 (0.004679676719663447)  --> TIME: 2025-02-24 10:50:22 -- STEP: 29484/36938 -- GLOBAL_STEP: 288050 | > loss_text_ce: 0.03559550270438194 (0.04250862743944185) | > loss_mel_ce: 2.874180316925049 (nan) | > loss: 0.011546730063855648 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13643258128362262) | > loader_time: 0.0047 (0.004679679377336797)  --> TIME: 2025-02-24 10:50:31 -- STEP: 29534/36938 -- GLOBAL_STEP: 288100 | > loss_text_ce: 0.049250368028879166 (0.04250805854078769) | > loss_mel_ce: 3.405818223953247 (nan) | > loss: 0.013710590079426765 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.136432662784848) | > loader_time: 0.0045 (0.004679653811989178)  --> TIME: 2025-02-24 10:50:41 -- STEP: 29584/36938 -- GLOBAL_STEP: 288150 | > loss_text_ce: 0.03561093658208847 (0.04250612768130891) | > loss_mel_ce: 3.838787794113159 (nan) | > loss: 0.015374599024653435 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.1364340636780488) | > loader_time: 0.0047 (0.004679626495597032)  --> TIME: 2025-02-24 10:50:50 -- STEP: 29634/36938 -- GLOBAL_STEP: 288200 | > loss_text_ce: 0.039658766239881516 (0.042506947141985975) | > loss_mel_ce: 2.5590808391571045 (nan) | > loss: 0.01031245943158865 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.13643751291962597) | > loader_time: 0.0047 (0.004679581297870389)  --> TIME: 2025-02-24 10:50:59 -- STEP: 29684/36938 -- GLOBAL_STEP: 288250 | > loss_text_ce: 0.039368048310279846 (0.042506687461873305) | > loss_mel_ce: 3.36177134513855 (nan) | > loss: 0.013496586121618748 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.1364345757197606) | > loader_time: 0.0046 (0.0046795912869068736)  --> TIME: 2025-02-24 10:51:09 -- STEP: 29734/36938 -- GLOBAL_STEP: 288300 | > loss_text_ce: 0.04256033897399902 (0.04250874060723614) | > loss_mel_ce: 2.8933966159820557 (nan) | > loss: 0.011650623753666878 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13643476338732022) | > loader_time: 0.0046 (0.004679641526701337)  --> TIME: 2025-02-24 10:51:18 -- STEP: 29784/36938 -- GLOBAL_STEP: 288350 | > loss_text_ce: 0.03651193156838417 (0.04250673802184021) | > loss_mel_ce: 2.2408478260040283 (nan) | > loss: 0.009037142619490623 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.1364374556770069) | > loader_time: 0.0045 (0.004679657424805844)  --> TIME: 2025-02-24 10:51:27 -- STEP: 29834/36938 -- GLOBAL_STEP: 288400 | > loss_text_ce: 0.04540073126554489 (0.04250499041532634) | > loss_mel_ce: 2.9789235591888428 (nan) | > loss: 0.012001287192106247 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.136430095881474) | > loader_time: 0.0047 (0.004679639825170727)  --> TIME: 2025-02-24 10:51:36 -- STEP: 29884/36938 -- GLOBAL_STEP: 288450 | > loss_text_ce: 0.040002577006816864 (0.042503464078759295) | > loss_mel_ce: 3.279146909713745 (nan) | > loss: 0.01317122858017683 (nan) | > current_lr: 5e-06 | > step_time: 0.1447 (0.13643392925679942) | > loader_time: 0.0048 (0.004679748961252465)  --> TIME: 2025-02-24 10:51:46 -- STEP: 29934/36938 -- GLOBAL_STEP: 288500 | > loss_text_ce: 0.03882373124361038 (0.04250345326204177) | > loss_mel_ce: 3.510239601135254 (nan) | > loss: 0.01408358570188284 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.1364331124277631) | > loader_time: 0.005 (0.00467986155585327)  --> TIME: 2025-02-24 10:51:55 -- STEP: 29984/36938 -- GLOBAL_STEP: 288550 | > loss_text_ce: 0.04841489717364311 (0.04250423074097362) | > loss_mel_ce: 2.6651291847229004 (nan) | > loss: 0.010768032632768154 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.1364221926480378) | > loader_time: 0.0046 (0.004679810227935565)  --> TIME: 2025-02-24 10:52:04 -- STEP: 30034/36938 -- GLOBAL_STEP: 288600 | > loss_text_ce: 0.046815235167741776 (0.042503157227384514) | > loss_mel_ce: 3.055475950241089 (nan) | > loss: 0.012310680001974106 (nan) | > current_lr: 5e-06 | > step_time: 0.1537 (0.13642569312863934) | > loader_time: 0.0048 (0.0046799731189483594)  --> TIME: 2025-02-24 10:52:14 -- STEP: 30084/36938 -- GLOBAL_STEP: 288650 | > loss_text_ce: 0.0385480597615242 (0.04250134291075811) | > loss_mel_ce: 3.516845703125 (nan) | > loss: 0.014108706265687943 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.1364353687470625) | > loader_time: 0.0045 (0.0046801057810986835)  --> TIME: 2025-02-24 10:52:23 -- STEP: 30134/36938 -- GLOBAL_STEP: 288700 | > loss_text_ce: 0.03627913072705269 (0.04249854893880099) | > loss_mel_ce: 2.989561080932617 (nan) | > loss: 0.012007303535938263 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13643121080075174) | > loader_time: 0.0046 (0.004680089432487296)  --> TIME: 2025-02-24 10:52:33 -- STEP: 30184/36938 -- GLOBAL_STEP: 288750 | > loss_text_ce: 0.04548738896846771 (0.04249784293387795) | > loss_mel_ce: 3.5361833572387695 (nan) | > loss: 0.014212979935109615 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.1364421387218594) | > loader_time: 0.0047 (0.004680234377054956)  --> TIME: 2025-02-24 10:52:42 -- STEP: 30234/36938 -- GLOBAL_STEP: 288800 | > loss_text_ce: 0.03530823811888695 (0.042497992179494334) | > loss_mel_ce: 2.430367946624756 (nan) | > loss: 0.009784429334104061 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1364368758851708) | > loader_time: 0.0045 (0.004680332600017646)  --> TIME: 2025-02-24 10:52:51 -- STEP: 30284/36938 -- GLOBAL_STEP: 288850 | > loss_text_ce: 0.044878605753183365 (0.042495884321393326) | > loss_mel_ce: 3.6309592723846436 (nan) | > loss: 0.014586659148335457 (nan) | > current_lr: 5e-06 | > step_time: 0.1694 (0.13643753845079865) | > loader_time: 0.0048 (0.004680365375193301)  --> TIME: 2025-02-24 10:53:00 -- STEP: 30334/36938 -- GLOBAL_STEP: 288900 | > loss_text_ce: 0.04225103184580803 (0.04249391620691954) | > loss_mel_ce: 3.3353066444396973 (nan) | > loss: 0.013403007760643959 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.13643431419074067) | > loader_time: 0.0046 (0.004680290496945886)  --> TIME: 2025-02-24 10:53:09 -- STEP: 30384/36938 -- GLOBAL_STEP: 288950 | > loss_text_ce: 0.041770994663238525 (0.04249252773397869) | > loss_mel_ce: 3.594316005706787 (nan) | > loss: 0.014428917318582535 (nan) | > current_lr: 5e-06 | > step_time: 0.1599 (0.13643148204820846) | > loader_time: 0.0048 (0.004680207955517092)  --> TIME: 2025-02-24 10:53:19 -- STEP: 30434/36938 -- GLOBAL_STEP: 289000 | > loss_text_ce: 0.043828681111335754 (0.04249159187298072) | > loss_mel_ce: 4.113593578338623 (nan) | > loss: 0.01649770699441433 (nan) | > current_lr: 5e-06 | > step_time: 0.1705 (0.13644068470027595) | > loader_time: 0.0048 (0.004680196394578469)  --> TIME: 2025-02-24 10:53:28 -- STEP: 30484/36938 -- GLOBAL_STEP: 289050 | > loss_text_ce: 0.04167759418487549 (0.042492084180831856) | > loss_mel_ce: 3.267678737640381 (nan) | > loss: 0.013132366351783276 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13643946900735068) | > loader_time: 0.0045 (0.004680180992296145)  --> TIME: 2025-02-24 10:53:38 -- STEP: 30534/36938 -- GLOBAL_STEP: 289100 | > loss_text_ce: 0.0425097830593586 (0.04248906406915259) | > loss_mel_ce: 2.6424121856689453 (nan) | > loss: 0.010654453188180923 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.13644415584834618) | > loader_time: 0.0062 (0.004680161166301973)  --> TIME: 2025-02-24 10:53:47 -- STEP: 30584/36938 -- GLOBAL_STEP: 289150 | > loss_text_ce: 0.043263353407382965 (0.04248704619424551) | > loss_mel_ce: 0.03190422058105469 (nan) | > loss: 0.00029828405240550637 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13644588356893714) | > loader_time: 0.0046 (0.004680336706618997)  --> TIME: 2025-02-24 10:53:57 -- STEP: 30634/36938 -- GLOBAL_STEP: 289200 | > loss_text_ce: 0.052251871675252914 (0.042485406626972456) | > loss_mel_ce: 3.5631368160247803 (nan) | > loss: 0.014346781186759472 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13645214372012193) | > loader_time: 0.0046 (0.004680443955688111)  --> TIME: 2025-02-24 10:54:06 -- STEP: 30684/36938 -- GLOBAL_STEP: 289250 | > loss_text_ce: 0.0376877523958683 (0.0424833323318706) | > loss_mel_ce: 3.009284496307373 (nan) | > loss: 0.012091160751879215 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13645237005917837) | > loader_time: 0.0046 (0.0046804101615386525)  --> TIME: 2025-02-24 10:54:16 -- STEP: 30734/36938 -- GLOBAL_STEP: 289300 | > loss_text_ce: 0.04289911687374115 (0.04248170381458091) | > loss_mel_ce: 2.8042097091674805 (nan) | > loss: 0.011298052035272121 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.1364598660631347) | > loader_time: 0.0046 (0.004680456782842627)  --> TIME: 2025-02-24 10:54:25 -- STEP: 30784/36938 -- GLOBAL_STEP: 289350 | > loss_text_ce: 0.04962296783924103 (0.04248283925463093) | > loss_mel_ce: 3.667926549911499 (nan) | > loss: 0.014752181246876717 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1364587618506629) | > loader_time: 0.0046 (0.004680469043537379)  --> TIME: 2025-02-24 10:54:34 -- STEP: 30834/36938 -- GLOBAL_STEP: 289400 | > loss_text_ce: 0.03743433207273483 (0.0424803732791364) | > loss_mel_ce: 3.3489701747894287 (nan) | > loss: 0.01343811396509409 (nan) | > current_lr: 5e-06 | > step_time: 0.179 (0.13646127943044065) | > loader_time: 0.0052 (0.004680578266514202)  --> TIME: 2025-02-24 10:54:44 -- STEP: 30884/36938 -- GLOBAL_STEP: 289450 | > loss_text_ce: 0.05804428085684776 (0.04248088144097396) | > loss_mel_ce: 4.214410781860352 (nan) | > loss: 0.01695418916642666 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1364646401527004) | > loader_time: 0.0045 (0.004680547515371114)  --> TIME: 2025-02-24 10:54:53 -- STEP: 30934/36938 -- GLOBAL_STEP: 289500 | > loss_text_ce: 0.05103989318013191 (0.04247983298803656) | > loss_mel_ce: 3.5695858001708984 (nan) | > loss: 0.014367563650012016 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13646678579426313) | > loader_time: 0.0049 (0.0046805589610791724)  --> TIME: 2025-02-24 10:55:03 -- STEP: 30984/36938 -- GLOBAL_STEP: 289550 | > loss_text_ce: 0.051176298409700394 (0.04247782187755904) | > loss_mel_ce: 3.5213093757629395 (nan) | > loss: 0.014176531694829464 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13647085615223878) | > loader_time: 0.0048 (0.0046806066666589975)  --> TIME: 2025-02-24 10:55:12 -- STEP: 31034/36938 -- GLOBAL_STEP: 289600 | > loss_text_ce: 0.04757700860500336 (0.042477471408782747) | > loss_mel_ce: 3.58620548248291 (nan) | > loss: 0.014419772662222385 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.1364763092424654) | > loader_time: 0.0046 (0.004680609975024314)  --> TIME: 2025-02-24 10:55:21 -- STEP: 31084/36938 -- GLOBAL_STEP: 289650 | > loss_text_ce: 0.045363783836364746 (0.04247413605886385) | > loss_mel_ce: 3.8457703590393066 (nan) | > loss: 0.015441009774804115 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13647046189344422) | > loader_time: 0.0046 (0.004680530312530409)  --> TIME: 2025-02-24 10:55:31 -- STEP: 31134/36938 -- GLOBAL_STEP: 289700 | > loss_text_ce: 0.040553998202085495 (0.04247368556651516) | > loss_mel_ce: 3.158296823501587 (nan) | > loss: 0.012693853117525578 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13647328949605667) | > loader_time: 0.0045 (0.004680491967139045)  --> TIME: 2025-02-24 10:55:40 -- STEP: 31184/36938 -- GLOBAL_STEP: 289750 | > loss_text_ce: 0.04141276329755783 (0.042471194684600075) | > loss_mel_ce: 2.0651800632476807 (nan) | > loss: 0.008359496481716633 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1364707525813811) | > loader_time: 0.0045 (0.004680441450681235)  --> TIME: 2025-02-24 10:55:49 -- STEP: 31234/36938 -- GLOBAL_STEP: 289800 | > loss_text_ce: 0.04499030485749245 (0.04246793136290699) | > loss_mel_ce: 2.615450382232666 (nan) | > loss: 0.010557305067777634 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.1364730367161615) | > loader_time: 0.0046 (0.004680424255026184)  --> TIME: 2025-02-24 10:55:59 -- STEP: 31284/36938 -- GLOBAL_STEP: 289850 | > loss_text_ce: 0.03619881346821785 (0.04246742529381367) | > loss_mel_ce: 2.7216427326202393 (nan) | > loss: 0.010943816043436527 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.13648041366021896) | > loader_time: 0.0046 (0.004680429764254863)  --> TIME: 2025-02-24 10:56:08 -- STEP: 31334/36938 -- GLOBAL_STEP: 289900 | > loss_text_ce: 0.03672328218817711 (0.042466180669474905) | > loss_mel_ce: 3.3190720081329346 (nan) | > loss: 0.01331664901226759 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13647293777420827) | > loader_time: 0.0046 (0.00468042098152665)  --> TIME: 2025-02-24 10:56:17 -- STEP: 31384/36938 -- GLOBAL_STEP: 289950 | > loss_text_ce: 0.04323222488164902 (0.04246897350970856) | > loss_mel_ce: 2.804007053375244 (nan) | > loss: 0.011298568919301033 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13646999205857485) | > loader_time: 0.0046 (0.004680301206864898)  --> TIME: 2025-02-24 10:56:26 -- STEP: 31434/36938 -- GLOBAL_STEP: 290000 | > loss_text_ce: 0.0463130958378315 (0.042465361541997135) | > loss_mel_ce: 2.479649782180786 (nan) | > loss: 0.010023662820458412 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13646835512337394) | > loader_time: 0.0046 (0.004680370695909374) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_290000.pth  --> TIME: 2025-02-24 10:56:41 -- STEP: 31484/36938 -- GLOBAL_STEP: 290050 | > loss_text_ce: 0.03599747270345688 (0.042463083959038694) | > loss_mel_ce: 3.45564603805542 (nan) | > loss: 0.013855728320777416 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13646403611986166) | > loader_time: 0.0046 (0.004680400525670899)  --> TIME: 2025-02-24 10:56:51 -- STEP: 31534/36938 -- GLOBAL_STEP: 290100 | > loss_text_ce: 0.03973971679806709 (0.04246411336862435) | > loss_mel_ce: 3.479698896408081 (nan) | > loss: 0.013966026715934277 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13646175393049986) | > loader_time: 0.0046 (0.004680460647222024)  --> TIME: 2025-02-24 10:57:00 -- STEP: 31584/36938 -- GLOBAL_STEP: 290150 | > loss_text_ce: 0.0385456457734108 (0.042462892333041354) | > loss_mel_ce: 2.770500898361206 (nan) | > loss: 0.011147010140120983 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13646043698929053) | > loader_time: 0.0046 (0.004680615911126252)  --> TIME: 2025-02-24 10:57:09 -- STEP: 31634/36938 -- GLOBAL_STEP: 290200 | > loss_text_ce: 0.04499535635113716 (0.04246118952358092) | > loss_mel_ce: 2.9771580696105957 (nan) | > loss: 0.011992672458291054 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.1364644826298883) | > loader_time: 0.0053 (0.004680766704795915)  --> TIME: 2025-02-24 10:57:19 -- STEP: 31684/36938 -- GLOBAL_STEP: 290250 | > loss_text_ce: 0.04719557613134384 (0.042459822464367476) | > loss_mel_ce: 3.672130823135376 (nan) | > loss: 0.014759233221411705 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.1364696398330393) | > loader_time: 0.0048 (0.00468091555518222)  --> TIME: 2025-02-24 10:57:28 -- STEP: 31734/36938 -- GLOBAL_STEP: 290300 | > loss_text_ce: 0.042080797255039215 (0.04245797279824691) | > loss_mel_ce: 2.567831516265869 (nan) | > loss: 0.010356795974075794 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13647089113004846) | > loader_time: 0.0049 (0.004681019271532748)  --> TIME: 2025-02-24 10:57:38 -- STEP: 31784/36938 -- GLOBAL_STEP: 290350 | > loss_text_ce: 0.04457961022853851 (0.042457349757398316) | > loss_mel_ce: 3.0997862815856934 (nan) | > loss: 0.012477642856538296 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13646921391694808) | > loader_time: 0.0045 (0.004681047604421863)  --> TIME: 2025-02-24 10:57:47 -- STEP: 31834/36938 -- GLOBAL_STEP: 290400 | > loss_text_ce: 0.055497992783784866 (0.042459413358503895) | > loss_mel_ce: 2.561479091644287 (nan) | > loss: 0.010384829714894295 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13647532288570016) | > loader_time: 0.0047 (0.004681110015097903)  --> TIME: 2025-02-24 10:57:57 -- STEP: 31884/36938 -- GLOBAL_STEP: 290450 | > loss_text_ce: 0.036415502429008484 (0.042456900741645126) | > loss_mel_ce: 2.9553494453430176 (nan) | > loss: 0.011872083880007267 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13647557875246008) | > loader_time: 0.0048 (0.00468106592722959)  --> TIME: 2025-02-24 10:58:06 -- STEP: 31934/36938 -- GLOBAL_STEP: 290500 | > loss_text_ce: 0.04710039123892784 (0.04245569308048609) | > loss_mel_ce: 2.266183614730835 (nan) | > loss: 0.009179698303341866 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13647548290203043) | > loader_time: 0.0046 (0.004681056798747504)  --> TIME: 2025-02-24 10:58:15 -- STEP: 31984/36938 -- GLOBAL_STEP: 290550 | > loss_text_ce: 0.030462922528386116 (0.04245667436985712) | > loss_mel_ce: 3.436826467514038 (nan) | > loss: 0.013759085908532143 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.13648349131686313) | > loader_time: 0.0047 (0.004681149532104379)  --> TIME: 2025-02-24 10:58:25 -- STEP: 32034/36938 -- GLOBAL_STEP: 290600 | > loss_text_ce: 0.038046497851610184 (0.042456406845864735) | > loss_mel_ce: 3.437380790710449 (nan) | > loss: 0.013791379518806934 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.1364801453615227) | > loader_time: 0.0047 (0.004681215435406038)  --> TIME: 2025-02-24 10:58:34 -- STEP: 32084/36938 -- GLOBAL_STEP: 290650 | > loss_text_ce: 0.03229403495788574 (0.042455693538683975) | > loss_mel_ce: 3.141145944595337 (nan) | > loss: 0.01259301695972681 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13648400215596743) | > loader_time: 0.0045 (0.0046813586691254825)  --> TIME: 2025-02-24 10:58:43 -- STEP: 32134/36938 -- GLOBAL_STEP: 290700 | > loss_text_ce: 0.04891496151685715 (0.04245616718014934) | > loss_mel_ce: 2.770427942276001 (nan) | > loss: 0.011187869124114513 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13648290588747913) | > loader_time: 0.0046 (0.004681471170660578)  --> TIME: 2025-02-24 10:58:52 -- STEP: 32184/36938 -- GLOBAL_STEP: 290750 | > loss_text_ce: 0.03944521024823189 (0.04245386018470789) | > loss_mel_ce: 3.4356799125671387 (nan) | > loss: 0.01379017997533083 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13647417328175807) | > loader_time: 0.0048 (0.004681442422769622)  --> TIME: 2025-02-24 10:59:01 -- STEP: 32234/36938 -- GLOBAL_STEP: 290800 | > loss_text_ce: 0.05101100727915764 (0.04245253116672047) | > loss_mel_ce: 3.9001591205596924 (nan) | > loss: 0.015679247677326202 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13646785821054955) | > loader_time: 0.0047 (0.004681456012835884)  --> TIME: 2025-02-24 10:59:11 -- STEP: 32284/36938 -- GLOBAL_STEP: 290850 | > loss_text_ce: 0.027323922142386436 (0.04245357009304244) | > loss_mel_ce: 3.653805732727051 (nan) | > loss: 0.014607658609747887 (nan) | > current_lr: 5e-06 | > step_time: 0.1481 (0.13646848604088652) | > loader_time: 0.0046 (0.00468140543651852)  --> TIME: 2025-02-24 10:59:19 -- STEP: 32334/36938 -- GLOBAL_STEP: 290900 | > loss_text_ce: 0.04451905936002731 (0.04245026921936952) | > loss_mel_ce: 3.371659517288208 (nan) | > loss: 0.013556264340877533 (nan) | > current_lr: 5e-06 | > step_time: 0.1511 (0.13645795412583342) | > loader_time: 0.0047 (0.004681338661933799)  --> TIME: 2025-02-24 10:59:28 -- STEP: 32384/36938 -- GLOBAL_STEP: 290950 | > loss_text_ce: 0.03216104581952095 (0.04244881508562009) | > loss_mel_ce: 3.1056482791900635 (nan) | > loss: 0.012451624497771263 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.1364457780649898) | > loader_time: 0.0047 (0.004681274685051595)  --> TIME: 2025-02-24 10:59:38 -- STEP: 32434/36938 -- GLOBAL_STEP: 291000 | > loss_text_ce: 0.04106675833463669 (0.0424472579455788) | > loss_mel_ce: 3.245220184326172 (nan) | > loss: 0.013040821067988873 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13644373320604103) | > loader_time: 0.0046 (0.004681359481493959)  --> TIME: 2025-02-24 10:59:47 -- STEP: 32484/36938 -- GLOBAL_STEP: 291050 | > loss_text_ce: 0.0382133387029171 (0.04244174459909456) | > loss_mel_ce: 2.403162956237793 (nan) | > loss: 0.009688001126050949 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.1364500900160991) | > loader_time: 0.0046 (0.004681392551833598)  --> TIME: 2025-02-24 10:59:57 -- STEP: 32534/36938 -- GLOBAL_STEP: 291100 | > loss_text_ce: 0.04544205218553543 (0.04244060751179872) | > loss_mel_ce: 3.794477939605713 (nan) | > loss: 0.015237778425216675 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.1364586954514899) | > loader_time: 0.0047 (0.004681511972359003)  --> TIME: 2025-02-24 11:00:06 -- STEP: 32584/36938 -- GLOBAL_STEP: 291150 | > loss_text_ce: 0.0468161478638649 (0.04243865438350315) | > loss_mel_ce: 3.661515474319458 (nan) | > loss: 0.014715602621436119 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13645830477539683) | > loader_time: 0.0045 (0.004681472378224311)  --> TIME: 2025-02-24 11:00:15 -- STEP: 32634/36938 -- GLOBAL_STEP: 291200 | > loss_text_ce: 0.04990415647625923 (0.04243848883216014) | > loss_mel_ce: 3.4105517864227295 (nan) | > loss: 0.013731968589127064 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13645722520930811) | > loader_time: 0.005 (0.004681619160343498)  --> TIME: 2025-02-24 11:00:25 -- STEP: 32684/36938 -- GLOBAL_STEP: 291250 | > loss_text_ce: 0.05715307220816612 (0.042440445138762245) | > loss_mel_ce: 3.1078174114227295 (nan) | > loss: 0.012559407390654087 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13646626708820092) | > loader_time: 0.0046 (0.004681726736853741)  --> TIME: 2025-02-24 11:00:34 -- STEP: 32734/36938 -- GLOBAL_STEP: 291300 | > loss_text_ce: 0.04460251331329346 (0.042435800100284034) | > loss_mel_ce: 3.7351834774017334 (nan) | > loss: 0.014999152161180973 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13646026239461034) | > loader_time: 0.0047 (0.004681715074060098)  --> TIME: 2025-02-24 11:00:43 -- STEP: 32784/36938 -- GLOBAL_STEP: 291350 | > loss_text_ce: 0.035659607499837875 (0.04243334382886077) | > loss_mel_ce: 2.7119131088256836 (nan) | > loss: 0.010903066955506802 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13645304444215592) | > loader_time: 0.0046 (0.004681654321734291)  --> TIME: 2025-02-24 11:00:53 -- STEP: 32834/36938 -- GLOBAL_STEP: 291400 | > loss_text_ce: 0.05321992188692093 (0.04243492509415474) | > loss_mel_ce: 3.41845703125 (nan) | > loss: 0.013776496984064579 (nan) | > current_lr: 5e-06 | > step_time: 0.1853 (0.13645675977167995) | > loader_time: 0.0049 (0.004681645629393915)  --> TIME: 2025-02-24 11:01:02 -- STEP: 32884/36938 -- GLOBAL_STEP: 291450 | > loss_text_ce: 0.04146720468997955 (0.04243341592626378) | > loss_mel_ce: 2.8629565238952637 (nan) | > loss: 0.011525491252541542 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13645689431775085) | > loader_time: 0.0046 (0.004681512533987596)  --> TIME: 2025-02-24 11:01:11 -- STEP: 32934/36938 -- GLOBAL_STEP: 291500 | > loss_text_ce: 0.04020824283361435 (0.04243401136215822) | > loss_mel_ce: 2.6265952587127686 (nan) | > loss: 0.010582554154098034 (nan) | > current_lr: 5e-06 | > step_time: 0.1073 (0.13646055554304112) | > loader_time: 0.0044 (0.004681510439392221)  --> TIME: 2025-02-24 11:01:20 -- STEP: 32984/36938 -- GLOBAL_STEP: 291550 | > loss_text_ce: 0.03806831315159798 (0.042434360067299795) | > loss_mel_ce: 3.3373465538024902 (nan) | > loss: 0.013394503854215145 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.13645167607952363) | > loader_time: 0.0046 (0.004681538644994656)  --> TIME: 2025-02-24 11:01:30 -- STEP: 33034/36938 -- GLOBAL_STEP: 291600 | > loss_text_ce: 0.037540219724178314 (0.04243359064292339) | > loss_mel_ce: 3.246089220046997 (nan) | > loss: 0.013030276633799076 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.1364514469602409) | > loader_time: 0.0046 (0.004681644142630337)  --> TIME: 2025-02-24 11:01:39 -- STEP: 33084/36938 -- GLOBAL_STEP: 291650 | > loss_text_ce: 0.041421178728342056 (0.0424338173651114) | > loss_mel_ce: 3.311690330505371 (nan) | > loss: 0.013305998407304287 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.13644872311596878) | > loader_time: 0.005 (0.004681642348670332)  --> TIME: 2025-02-24 11:01:48 -- STEP: 33134/36938 -- GLOBAL_STEP: 291700 | > loss_text_ce: 0.04242704436182976 (0.04243275291740798) | > loss_mel_ce: 3.512122631072998 (nan) | > loss: 0.01410535629838705 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13644494613611358) | > loader_time: 0.0046 (0.0046816216357308165)  --> TIME: 2025-02-24 11:01:57 -- STEP: 33184/36938 -- GLOBAL_STEP: 291750 | > loss_text_ce: 0.05192871019244194 (0.042429879028383785) | > loss_mel_ce: 3.1854071617126465 (nan) | > loss: 0.012846571393311024 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13643885870880007) | > loader_time: 0.0045 (0.004681621454548347)  --> TIME: 2025-02-24 11:02:06 -- STEP: 33234/36938 -- GLOBAL_STEP: 291800 | > loss_text_ce: 0.04314116761088371 (0.04243079334483551) | > loss_mel_ce: 3.3840694427490234 (nan) | > loss: 0.013600043021142483 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13643175830922605) | > loader_time: 0.0047 (0.004681614401280649)  --> TIME: 2025-02-24 11:02:15 -- STEP: 33284/36938 -- GLOBAL_STEP: 291850 | > loss_text_ce: 0.03770015016198158 (0.04242964128219016) | > loss_mel_ce: 3.353828191757202 (nan) | > loss: 0.013458446599543095 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13642870401364837) | > loader_time: 0.0046 (0.004681632633665373)  --> TIME: 2025-02-24 11:02:24 -- STEP: 33334/36938 -- GLOBAL_STEP: 291900 | > loss_text_ce: 0.04120871424674988 (0.042430405134975854) | > loss_mel_ce: 3.5412471294403076 (nan) | > loss: 0.014216095209121704 (nan) | > current_lr: 5e-06 | > step_time: 0.1688 (0.1364280981115459) | > loader_time: 0.0047 (0.0046816971804561035)  --> TIME: 2025-02-24 11:02:34 -- STEP: 33384/36938 -- GLOBAL_STEP: 291950 | > loss_text_ce: 0.035341691225767136 (0.04242974537985591) | > loss_mel_ce: 3.1653807163238525 (nan) | > loss: 0.012701280415058136 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13643109456343916) | > loader_time: 0.0045 (0.004681937726830046)  --> TIME: 2025-02-24 11:02:44 -- STEP: 33434/36938 -- GLOBAL_STEP: 292000 | > loss_text_ce: 0.03899779170751572 (0.04242810017014858) | > loss_mel_ce: 3.542186975479126 (nan) | > loss: 0.014211052097380161 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13643805105135573) | > loader_time: 0.0053 (0.004682035239937838)  --> TIME: 2025-02-24 11:02:53 -- STEP: 33484/36938 -- GLOBAL_STEP: 292050 | > loss_text_ce: 0.04688392952084541 (0.042428161882332685) | > loss_mel_ce: 3.4670398235321045 (nan) | > loss: 0.013944142498075962 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13643477409775479) | > loader_time: 0.0046 (0.004682019098359552)  --> TIME: 2025-02-24 11:03:02 -- STEP: 33534/36938 -- GLOBAL_STEP: 292100 | > loss_text_ce: 0.03508604317903519 (0.04242542530832966) | > loss_mel_ce: 2.824324607849121 (nan) | > loss: 0.011346869170665741 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13643237746322878) | > loader_time: 0.0047 (0.004682056690688602)  --> TIME: 2025-02-24 11:03:11 -- STEP: 33584/36938 -- GLOBAL_STEP: 292150 | > loss_text_ce: 0.038803599774837494 (0.04242413118276424) | > loss_mel_ce: 1.64501953125 (nan) | > loss: 0.006681838072836399 (nan) | > current_lr: 5e-06 | > step_time: 0.1778 (0.13643135509501234) | > loader_time: 0.0048 (0.004682040231576581)  --> TIME: 2025-02-24 11:03:20 -- STEP: 33634/36938 -- GLOBAL_STEP: 292200 | > loss_text_ce: 0.03700214624404907 (0.04242364403333124) | > loss_mel_ce: 2.7686262130737305 (nan) | > loss: 0.011133446358144283 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13642899041766515) | > loader_time: 0.0045 (0.004682184165940519)  --> TIME: 2025-02-24 11:03:30 -- STEP: 33684/36938 -- GLOBAL_STEP: 292250 | > loss_text_ce: 0.05162947624921799 (0.04242129456664378) | > loss_mel_ce: 0.02588907815515995 (nan) | > loss: 0.0003076133143622428 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13642832620199713) | > loader_time: 0.0046 (0.004682116936850239)  --> TIME: 2025-02-24 11:03:39 -- STEP: 33734/36938 -- GLOBAL_STEP: 292300 | > loss_text_ce: 0.035460881888866425 (0.04242038176334094) | > loss_mel_ce: 1.9117151498794556 (nan) | > loss: 0.007726889103651047 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13643962489397465) | > loader_time: 0.0045 (0.0046821503165126194)  --> TIME: 2025-02-24 11:03:48 -- STEP: 33784/36938 -- GLOBAL_STEP: 292350 | > loss_text_ce: 0.04540156200528145 (0.04241844119649425) | > loss_mel_ce: 3.5183348655700684 (nan) | > loss: 0.014141811989247799 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13642907057953474) | > loader_time: 0.0046 (0.0046820898361463435)  --> TIME: 2025-02-24 11:03:58 -- STEP: 33834/36938 -- GLOBAL_STEP: 292400 | > loss_text_ce: 0.04116518050432205 (0.042417972562399126) | > loss_mel_ce: 3.9126837253570557 (nan) | > loss: 0.015689877793192863 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13642741159364838) | > loader_time: 0.0046 (0.004682044494713686)  --> TIME: 2025-02-24 11:04:07 -- STEP: 33884/36938 -- GLOBAL_STEP: 292450 | > loss_text_ce: 0.03328575938940048 (0.04241730905092988) | > loss_mel_ce: 3.182558059692383 (nan) | > loss: 0.012761286459863186 (nan) | > current_lr: 5e-06 | > step_time: 0.1579 (0.1364237975235495) | > loader_time: 0.0047 (0.004681974265950015)  --> TIME: 2025-02-24 11:04:16 -- STEP: 33934/36938 -- GLOBAL_STEP: 292500 | > loss_text_ce: 0.03462277352809906 (0.04241646050053486) | > loss_mel_ce: 0.0414021797478199 (nan) | > loss: 0.000301686319289729 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.1364169186129912) | > loader_time: 0.0046 (0.0046819553579277645)  --> TIME: 2025-02-24 11:04:25 -- STEP: 33984/36938 -- GLOBAL_STEP: 292550 | > loss_text_ce: 0.05290863290429115 (0.04241458742516273) | > loss_mel_ce: 3.2126429080963135 (nan) | > loss: 0.012958538718521595 (nan) | > current_lr: 5e-06 | > step_time: 0.1786 (0.13641813264812502) | > loader_time: 0.0051 (0.0046819262346866584)  --> TIME: 2025-02-24 11:04:35 -- STEP: 34034/36938 -- GLOBAL_STEP: 292600 | > loss_text_ce: 0.054908279329538345 (0.04241464547754924) | > loss_mel_ce: 0.8342593908309937 (nan) | > loss: 0.0035284433979541063 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13642311316158057) | > loader_time: 0.0045 (0.004681907543853305)  --> TIME: 2025-02-24 11:04:44 -- STEP: 34084/36938 -- GLOBAL_STEP: 292650 | > loss_text_ce: 0.04291039705276489 (0.04241476854498085) | > loss_mel_ce: 2.7900798320770264 (nan) | > loss: 0.011242025531828403 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13642082131394467) | > loader_time: 0.0045 (0.004681954360346792)  --> TIME: 2025-02-24 11:04:53 -- STEP: 34134/36938 -- GLOBAL_STEP: 292700 | > loss_text_ce: 0.04188695177435875 (0.0424136271166691) | > loss_mel_ce: 3.452249526977539 (nan) | > loss: 0.013865621760487556 (nan) | > current_lr: 5e-06 | > step_time: 0.1724 (0.13641972605929437) | > loader_time: 0.0048 (0.004682034566643122)  --> TIME: 2025-02-24 11:05:02 -- STEP: 34184/36938 -- GLOBAL_STEP: 292750 | > loss_text_ce: 0.0502595417201519 (0.042411037400890576) | > loss_mel_ce: 3.0523681640625 (nan) | > loss: 0.012312015518546104 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.1364184048592616) | > loader_time: 0.0045 (0.004682128564161454)  --> TIME: 2025-02-24 11:05:11 -- STEP: 34234/36938 -- GLOBAL_STEP: 292800 | > loss_text_ce: 0.03998282551765442 (0.04240942595953904) | > loss_mel_ce: 2.9594550132751465 (nan) | > loss: 0.01190253160893917 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1364138639578445) | > loader_time: 0.0046 (0.004682232322777547)  --> TIME: 2025-02-24 11:05:21 -- STEP: 34284/36938 -- GLOBAL_STEP: 292850 | > loss_text_ce: 0.05985577777028084 (0.04240791853693321) | > loss_mel_ce: 3.633821487426758 (nan) | > loss: 0.014657449908554554 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13641487539558692) | > loader_time: 0.0046 (0.004682364325834357)  --> TIME: 2025-02-24 11:05:31 -- STEP: 34334/36938 -- GLOBAL_STEP: 292900 | > loss_text_ce: 0.038563087582588196 (0.042406684713926494) | > loss_mel_ce: 3.608870267868042 (nan) | > loss: 0.01447394210845232 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.1364281344387374) | > loader_time: 0.0047 (0.004682405726733382)  --> TIME: 2025-02-24 11:05:40 -- STEP: 34384/36938 -- GLOBAL_STEP: 292950 | > loss_text_ce: 0.0392955020070076 (0.04240858431442011) | > loss_mel_ce: 3.0502004623413086 (nan) | > loss: 0.012259905226528645 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.1364268311626586) | > loader_time: 0.0045 (0.004682344439530036)  --> TIME: 2025-02-24 11:05:49 -- STEP: 34434/36938 -- GLOBAL_STEP: 293000 | > loss_text_ce: 0.03110005147755146 (0.04240689806346095) | > loss_mel_ce: 3.5704474449157715 (nan) | > loss: 0.01429185550659895 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.13642564322285602) | > loader_time: 0.0048 (0.004682312853945485)  --> TIME: 2025-02-24 11:05:58 -- STEP: 34484/36938 -- GLOBAL_STEP: 293050 | > loss_text_ce: 0.051487307995557785 (0.04240705798492532) | > loss_mel_ce: 3.6201467514038086 (nan) | > loss: 0.014569977298378944 (nan) | > current_lr: 5e-06 | > step_time: 0.1574 (0.13642191449252672) | > loader_time: 0.0048 (0.004682298914322271)  --> TIME: 2025-02-24 11:06:07 -- STEP: 34534/36938 -- GLOBAL_STEP: 293100 | > loss_text_ce: 0.04355570673942566 (0.04240579979057648) | > loss_mel_ce: 3.591007947921753 (nan) | > loss: 0.01442287303507328 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.13641649460816066) | > loader_time: 0.0048 (0.004682329966224246)  --> TIME: 2025-02-24 11:06:17 -- STEP: 34584/36938 -- GLOBAL_STEP: 293150 | > loss_text_ce: 0.03912746161222458 (0.04240484927859988) | > loss_mel_ce: 3.8165886402130127 (nan) | > loss: 0.01530046109110117 (nan) | > current_lr: 5e-06 | > step_time: 0.1759 (0.1364192116420472) | > loader_time: 0.0049 (0.004682405752466123)  --> TIME: 2025-02-24 11:06:26 -- STEP: 34634/36938 -- GLOBAL_STEP: 293200 | > loss_text_ce: 0.042915087193250656 (0.04240579563015813) | > loss_mel_ce: 3.247321367263794 (nan) | > loss: 0.013056494295597076 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.1364120269968587) | > loader_time: 0.0045 (0.004682314218987107)  --> TIME: 2025-02-24 11:06:35 -- STEP: 34684/36938 -- GLOBAL_STEP: 293250 | > loss_text_ce: 0.039068326354026794 (0.04240353086886175) | > loss_mel_ce: 3.6918389797210693 (nan) | > loss: 0.014805188402533531 (nan) | > current_lr: 5e-06 | > step_time: 0.1912 (0.13641210257312825) | > loader_time: 0.0048 (0.004682241901093681)  --> TIME: 2025-02-24 11:06:44 -- STEP: 34734/36938 -- GLOBAL_STEP: 293300 | > loss_text_ce: 0.04116420820355415 (0.04240360867553621) | > loss_mel_ce: 3.7576096057891846 (nan) | > loss: 0.015074499882757664 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13641215037234306) | > loader_time: 0.0047 (0.004682175673961804)  --> TIME: 2025-02-24 11:06:53 -- STEP: 34784/36938 -- GLOBAL_STEP: 293350 | > loss_text_ce: 0.03874655067920685 (0.04240099373063634) | > loss_mel_ce: 2.940950632095337 (nan) | > loss: 0.011824196204543114 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13640784988352933) | > loader_time: 0.0046 (0.004682174930905873)  --> TIME: 2025-02-24 11:07:03 -- STEP: 34834/36938 -- GLOBAL_STEP: 293400 | > loss_text_ce: 0.044731784611940384 (0.04240086256897126) | > loss_mel_ce: 3.5960333347320557 (nan) | > loss: 0.014447481371462345 (nan) | > current_lr: 5e-06 | > step_time: 0.1504 (0.1364141963903553) | > loader_time: 0.0046 (0.004682162752954724)  --> TIME: 2025-02-24 11:07:12 -- STEP: 34884/36938 -- GLOBAL_STEP: 293450 | > loss_text_ce: 0.04411625862121582 (0.04239787856750582) | > loss_mel_ce: 0.0621337890625 (nan) | > loss: 0.00042162719182670116 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.13641541354249762) | > loader_time: 0.0049 (0.0046820997399018285)  --> TIME: 2025-02-24 11:07:22 -- STEP: 34934/36938 -- GLOBAL_STEP: 293500 | > loss_text_ce: 0.0401531383395195 (0.042397027555111974) | > loss_mel_ce: 3.7534286975860596 (nan) | > loss: 0.015053896233439445 (nan) | > current_lr: 5e-06 | > step_time: 0.186 (0.13642229785860288) | > loader_time: 0.0049 (0.0046821688106980135)  --> TIME: 2025-02-24 11:07:31 -- STEP: 34984/36938 -- GLOBAL_STEP: 293550 | > loss_text_ce: 0.04310033842921257 (0.042394946848306164) | > loss_mel_ce: 3.2583818435668945 (nan) | > loss: 0.013101120479404926 (nan) | > current_lr: 5e-06 | > step_time: 0.1702 (0.13641528012058549) | > loader_time: 0.0048 (0.0046822086927303325)  --> TIME: 2025-02-24 11:07:40 -- STEP: 35034/36938 -- GLOBAL_STEP: 293600 | > loss_text_ce: 0.0487838014960289 (0.042393272965185205) | > loss_mel_ce: 3.521716356277466 (nan) | > loss: 0.014168651774525642 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13642017705368314) | > loader_time: 0.0046 (0.004682211712042711)  --> TIME: 2025-02-24 11:07:50 -- STEP: 35084/36938 -- GLOBAL_STEP: 293650 | > loss_text_ce: 0.04969539865851402 (0.0423921902009712) | > loss_mel_ce: 2.368983745574951 (nan) | > loss: 0.00959793385118246 (nan) | > current_lr: 5e-06 | > step_time: 0.1842 (0.1364155557486972) | > loader_time: 0.0049 (0.004682259716746748)  --> TIME: 2025-02-24 11:07:59 -- STEP: 35134/36938 -- GLOBAL_STEP: 293700 | > loss_text_ce: 0.028603311628103256 (0.04239018795001435) | > loss_mel_ce: 2.280893087387085 (nan) | > loss: 0.009164668619632721 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13642015526777532) | > loader_time: 0.0046 (0.004682536821967384)  --> TIME: 2025-02-24 11:08:09 -- STEP: 35184/36938 -- GLOBAL_STEP: 293750 | > loss_text_ce: 0.0503358393907547 (0.042388892682267504) | > loss_mel_ce: 3.6957061290740967 (nan) | > loss: 0.01486524660140276 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13642455119065416) | > loader_time: 0.0054 (0.00468260137912302)  --> TIME: 2025-02-24 11:08:18 -- STEP: 35234/36938 -- GLOBAL_STEP: 293800 | > loss_text_ce: 0.037091970443725586 (0.04238889196358643) | > loss_mel_ce: 0.0611640065908432 (nan) | > loss: 0.00038990468601696193 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13642417422324632) | > loader_time: 0.0048 (0.004682799409289354)  --> TIME: 2025-02-24 11:08:27 -- STEP: 35284/36938 -- GLOBAL_STEP: 293850 | > loss_text_ce: 0.04435151815414429 (0.04238881822686023) | > loss_mel_ce: 3.053520441055298 (nan) | > loss: 0.012293143197894096 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13642445940804412) | > loader_time: 0.0047 (0.004682897257839987)  --> TIME: 2025-02-24 11:08:37 -- STEP: 35334/36938 -- GLOBAL_STEP: 293900 | > loss_text_ce: 0.02949257753789425 (0.042388079897549315) | > loss_mel_ce: 3.122387409210205 (nan) | > loss: 0.01250746101140976 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13642514395026978) | > loader_time: 0.0048 (0.004683095773088091)  --> TIME: 2025-02-24 11:08:46 -- STEP: 35384/36938 -- GLOBAL_STEP: 293950 | > loss_text_ce: 0.041610583662986755 (0.04238796637928574) | > loss_mel_ce: 3.8946025371551514 (nan) | > loss: 0.015619893558323383 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13642909523860297) | > loader_time: 0.0046 (0.004683154182630276)  --> TIME: 2025-02-24 11:08:56 -- STEP: 35434/36938 -- GLOBAL_STEP: 294000 | > loss_text_ce: 0.03620576485991478 (0.04238713531345782) | > loss_mel_ce: 3.6251068115234375 (nan) | > loss: 0.01452901866286993 (nan) | > current_lr: 5e-06 | > step_time: 0.1522 (0.13644284811637383) | > loader_time: 0.0047 (0.004683249440951867)  --> TIME: 2025-02-24 11:09:06 -- STEP: 35484/36938 -- GLOBAL_STEP: 294050 | > loss_text_ce: 0.04470953345298767 (0.04238602349033009) | > loss_mel_ce: 0.03564453125 (nan) | > loss: 0.00031886534998193383 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13644997361493602) | > loader_time: 0.0047 (0.004683278006344105)  --> TIME: 2025-02-24 11:09:15 -- STEP: 35534/36938 -- GLOBAL_STEP: 294100 | > loss_text_ce: 0.04275159910321236 (0.042386916355034505) | > loss_mel_ce: 2.7862977981567383 (nan) | > loss: 0.01122638676315546 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13644820676843272) | > loader_time: 0.0045 (0.004683348271965778)  --> TIME: 2025-02-24 11:09:24 -- STEP: 35584/36938 -- GLOBAL_STEP: 294150 | > loss_text_ce: 0.04796958714723587 (0.0423859420458919) | > loss_mel_ce: 3.138561248779297 (nan) | > loss: 0.012644964270293713 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.13645161392850702) | > loader_time: 0.0057 (0.004683546541030868)  --> TIME: 2025-02-24 11:09:34 -- STEP: 35634/36938 -- GLOBAL_STEP: 294200 | > loss_text_ce: 0.038620878010988235 (0.04238304683346173) | > loss_mel_ce: 3.5408260822296143 (nan) | > loss: 0.014204155653715134 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13645603090462846) | > loader_time: 0.0059 (0.0046837664737092105)  --> TIME: 2025-02-24 11:09:43 -- STEP: 35684/36938 -- GLOBAL_STEP: 294250 | > loss_text_ce: 0.05490216612815857 (0.042382642012335825) | > loss_mel_ce: 1.7609387636184692 (nan) | > loss: 0.0072057186625897884 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.1364565851268533) | > loader_time: 0.0045 (0.004683757660808787)  --> TIME: 2025-02-24 11:09:53 -- STEP: 35734/36938 -- GLOBAL_STEP: 294300 | > loss_text_ce: 0.044638365507125854 (0.04238058382762735) | > loss_mel_ce: 2.0890166759490967 (nan) | > loss: 0.008466885425150394 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13646127376731534) | > loader_time: 0.0045 (0.004683705691145787)  --> TIME: 2025-02-24 11:10:02 -- STEP: 35784/36938 -- GLOBAL_STEP: 294350 | > loss_text_ce: 0.03745301067829132 (0.04238168049412975) | > loss_mel_ce: 1.325225830078125 (nan) | > loss: 0.005407456308603287 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13645678932518138) | > loader_time: 0.0048 (0.004683680317691298)  --> TIME: 2025-02-24 11:10:11 -- STEP: 35834/36938 -- GLOBAL_STEP: 294400 | > loss_text_ce: 0.0462941974401474 (0.042380311941391004) | > loss_mel_ce: 3.549692392349243 (nan) | > loss: 0.01426978874951601 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13645812705564284) | > loader_time: 0.0046 (0.0046836245756579735)  --> TIME: 2025-02-24 11:10:21 -- STEP: 35884/36938 -- GLOBAL_STEP: 294450 | > loss_text_ce: 0.04502840340137482 (0.04237955887889145) | > loss_mel_ce: 2.227189779281616 (nan) | > loss: 0.009016739204525948 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.13646702145323641) | > loader_time: 0.0048 (0.00468359606386412)  --> TIME: 2025-02-24 11:10:30 -- STEP: 35934/36938 -- GLOBAL_STEP: 294500 | > loss_text_ce: 0.04189363867044449 (0.04237774428838156) | > loss_mel_ce: 2.6536002159118652 (nan) | > loss: 0.010696405544877052 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13646913045846198) | > loader_time: 0.0047 (0.004683807297355893)  --> TIME: 2025-02-24 11:10:39 -- STEP: 35984/36938 -- GLOBAL_STEP: 294550 | > loss_text_ce: 0.03803297504782677 (0.042376771121345434) | > loss_mel_ce: 2.7973263263702393 (nan) | > loss: 0.011251426301896572 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13646819662152068) | > loader_time: 0.0045 (0.004683949752297051)  --> TIME: 2025-02-24 11:10:48 -- STEP: 36034/36938 -- GLOBAL_STEP: 294600 | > loss_text_ce: 0.04050081595778465 (0.04237506526094381) | > loss_mel_ce: 3.0556249618530273 (nan) | > loss: 0.012286214157938957 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13646127783697123) | > loader_time: 0.0045 (0.0046838936017304066)  --> TIME: 2025-02-24 11:10:58 -- STEP: 36084/36938 -- GLOBAL_STEP: 294650 | > loss_text_ce: 0.05740804225206375 (0.04237816694428281) | > loss_mel_ce: 3.4251277446746826 (nan) | > loss: 0.013819587416946888 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13646146978511156) | > loader_time: 0.0045 (0.004683852942367029)  --> TIME: 2025-02-24 11:11:07 -- STEP: 36134/36938 -- GLOBAL_STEP: 294700 | > loss_text_ce: 0.0719732940196991 (0.04237791803234155) | > loss_mel_ce: 2.7978997230529785 (nan) | > loss: 0.01138838566839695 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13645842288205218) | > loader_time: 0.0046 (0.004684043483530366)  --> TIME: 2025-02-24 11:11:16 -- STEP: 36184/36938 -- GLOBAL_STEP: 294750 | > loss_text_ce: 0.04142800346016884 (0.04237926977596189) | > loss_mel_ce: 2.9845352172851562 (nan) | > loss: 0.012007791548967361 (nan) | > current_lr: 5e-06 | > step_time: 0.1844 (0.13645256802410122) | > loader_time: 0.005 (0.004684128139020582)  --> TIME: 2025-02-24 11:11:26 -- STEP: 36234/36938 -- GLOBAL_STEP: 294800 | > loss_text_ce: 0.051765479147434235 (0.042379234962699144) | > loss_mel_ce: 3.8132271766662598 (nan) | > loss: 0.015337273478507996 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.13645714890215402) | > loader_time: 0.0056 (0.004684200519532909)  --> TIME: 2025-02-24 11:11:35 -- STEP: 36284/36938 -- GLOBAL_STEP: 294850 | > loss_text_ce: 0.04787937551736832 (0.04237999123609546) | > loss_mel_ce: 3.2238194942474365 (nan) | > loss: 0.012982932850718498 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.1364629067970926) | > loader_time: 0.0045 (0.004684227032798629)  --> TIME: 2025-02-24 11:11:45 -- STEP: 36334/36938 -- GLOBAL_STEP: 294900 | > loss_text_ce: 0.03517954424023628 (0.042378013499875515) | > loss_mel_ce: 2.100149393081665 (nan) | > loss: 0.008473528549075127 (nan) | > current_lr: 5e-06 | > step_time: 0.1077 (0.13646609528441334) | > loader_time: 0.0044 (0.004684171830445701)  --> TIME: 2025-02-24 11:11:54 -- STEP: 36384/36938 -- GLOBAL_STEP: 294950 | > loss_text_ce: 0.04337313026189804 (0.042377648777746135) | > loss_mel_ce: 3.605289936065674 (nan) | > loss: 0.014478822238743305 (nan) | > current_lr: 5e-06 | > step_time: 0.1911 (0.13646436342437204) | > loader_time: 0.0049 (0.004684195977454645)  --> TIME: 2025-02-24 11:12:04 -- STEP: 36434/36938 -- GLOBAL_STEP: 295000 | > loss_text_ce: 0.04148682951927185 (0.04237667714708949) | > loss_mel_ce: 3.785527229309082 (nan) | > loss: 0.015186564065515995 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13647708211869475) | > loader_time: 0.0046 (0.004684219639381251)  --> TIME: 2025-02-24 11:12:13 -- STEP: 36484/36938 -- GLOBAL_STEP: 295050 | > loss_text_ce: 0.04517687112092972 (0.04237529180974393) | > loss_mel_ce: 3.466763973236084 (nan) | > loss: 0.01393627468496561 (nan) | > current_lr: 5e-06 | > step_time: 0.1848 (0.13648248574226984) | > loader_time: 0.0049 (0.004684220383975224)  --> TIME: 2025-02-24 11:12:23 -- STEP: 36534/36938 -- GLOBAL_STEP: 295100 | > loss_text_ce: 0.03975076228380203 (0.04237495713961106) | > loss_mel_ce: 2.9153852462768555 (nan) | > loss: 0.011726731434464455 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13648448258894622) | > loader_time: 0.0048 (0.00468414656117597)  --> TIME: 2025-02-24 11:12:32 -- STEP: 36584/36938 -- GLOBAL_STEP: 295150 | > loss_text_ce: 0.04782623052597046 (0.042373259495512024) | > loss_mel_ce: 3.342031240463257 (nan) | > loss: 0.013451816514134407 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13648705073344636) | > loader_time: 0.0045 (0.004684068599832714)  --> TIME: 2025-02-24 11:12:41 -- STEP: 36634/36938 -- GLOBAL_STEP: 295200 | > loss_text_ce: 0.026951661333441734 (0.04237329846816169) | > loss_mel_ce: 2.762573719024658 (nan) | > loss: 0.011069545522332191 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13648472648654458) | > loader_time: 0.0046 (0.004684007557652719)  --> TIME: 2025-02-24 11:12:51 -- STEP: 36684/36938 -- GLOBAL_STEP: 295250 | > loss_text_ce: 0.029045412316918373 (0.04236961979524198) | > loss_mel_ce: 3.2957940101623535 (nan) | > loss: 0.013193807564675808 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13648669901155294) | > loader_time: 0.0047 (0.004683915914411346)  --> TIME: 2025-02-24 11:13:00 -- STEP: 36734/36938 -- GLOBAL_STEP: 295300 | > loss_text_ce: 0.0347839891910553 (0.04236865305162936) | > loss_mel_ce: 2.7459142208099365 (nan) | > loss: 0.011034517548978329 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.136491547666989) | > loader_time: 0.0048 (0.004683854220746448)  --> TIME: 2025-02-24 11:13:10 -- STEP: 36784/36938 -- GLOBAL_STEP: 295350 | > loss_text_ce: 0.04072495549917221 (0.04236641289522888) | > loss_mel_ce: 1.8863056898117065 (nan) | > loss: 0.007646947633475065 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13649378385088676) | > loader_time: 0.0045 (0.004683893347315379)  --> TIME: 2025-02-24 11:13:19 -- STEP: 36834/36938 -- GLOBAL_STEP: 295400 | > loss_text_ce: 0.026700295507907867 (0.04236341961221314) | > loss_mel_ce: 2.8919708728790283 (nan) | > loss: 0.011582029052078724 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13649044925837697) | > loader_time: 0.0055 (0.004683880086972215)  --> TIME: 2025-02-24 11:13:28 -- STEP: 36884/36938 -- GLOBAL_STEP: 295450 | > loss_text_ce: 0.03005756065249443 (0.04236185780182284) | > loss_mel_ce: 3.161315441131592 (nan) | > loss: 0.01266417931765318 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13649557543969704) | > loader_time: 0.0045 (0.004683839093189576)  --> TIME: 2025-02-24 11:13:37 -- STEP: 36934/36938 -- GLOBAL_STEP: 295500 | > loss_text_ce: 0.04437622055411339 (0.04236040173570788) | > loss_mel_ce: 3.2433106899261475 (nan) | > loss: 0.013046377338469028 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13649236559448377) | > loader_time: 0.0044 (0.004683695829972642)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.021157283594112588 (+0.000942328188679023) | > avg_loss_text_ce: 0.04068429041469451 (-0.0011806601271180453) | > avg_loss_mel_ce: 2.8393145463254186 (-0.05305828316377115) | > avg_loss: 2.8799988355967066 (-0.054238950852121715) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_295504.pth  > EPOCH: 9/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-24 11:14:03)   --> TIME: 2025-02-24 11:14:12 -- STEP: 46/36938 -- GLOBAL_STEP: 295550 | > loss_text_ce: 0.037034399807453156 (0.039811675436794744) | > loss_mel_ce: 2.669921875 (nan) | > loss: 0.010741891339421272 (nan) | > current_lr: 5e-06 | > step_time: 0.1077 (0.1298752349355947) | > loader_time: 0.0045 (0.00466274696847667)  --> TIME: 2025-02-24 11:14:21 -- STEP: 96/36938 -- GLOBAL_STEP: 295600 | > loss_text_ce: 0.05197882652282715 (0.04038740847803029) | > loss_mel_ce: 3.8484039306640625 (nan) | > loss: 0.015477710403501987 (nan) | > current_lr: 5e-06 | > step_time: 0.161 (0.13400372614463174) | > loader_time: 0.0048 (0.004661706586678824)  --> TIME: 2025-02-24 11:14:30 -- STEP: 146/36938 -- GLOBAL_STEP: 295650 | > loss_text_ce: 0.049751222133636475 (0.041055206209421144) | > loss_mel_ce: 3.5151147842407227 (nan) | > loss: 0.014146294444799423 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.1335981022821714) | > loader_time: 0.0046 (0.004654647552803773)  --> TIME: 2025-02-24 11:14:40 -- STEP: 196/36938 -- GLOBAL_STEP: 295700 | > loss_text_ce: 0.04562119022011757 (0.04126516850285078) | > loss_mel_ce: 3.8173935413360596 (nan) | > loss: 0.015329424291849136 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13426847238929912) | > loader_time: 0.0048 (0.004656871970818968)  --> TIME: 2025-02-24 11:14:49 -- STEP: 246/36938 -- GLOBAL_STEP: 295750 | > loss_text_ce: 0.03137831762433052 (0.04134512399664011) | > loss_mel_ce: 3.383413553237915 (nan) | > loss: 0.013550762087106705 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13482013756666733) | > loader_time: 0.0048 (0.004654385210052737)  --> TIME: 2025-02-24 11:14:59 -- STEP: 296/36938 -- GLOBAL_STEP: 295800 | > loss_text_ce: 0.04384899139404297 (0.0416177395743796) | > loss_mel_ce: 3.6786296367645264 (nan) | > loss: 0.014771741814911366 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13569857784219694) | > loader_time: 0.0045 (0.0046539451624896064)  --> TIME: 2025-02-24 11:15:08 -- STEP: 346/36938 -- GLOBAL_STEP: 295850 | > loss_text_ce: 0.04518824443221092 (0.04137439783016551) | > loss_mel_ce: 3.939284324645996 (nan) | > loss: 0.01581140048801899 (nan) | > current_lr: 5e-06 | > step_time: 0.1931 (0.13619685104127563) | > loader_time: 0.0048 (0.004648414650404384)  --> TIME: 2025-02-24 11:15:17 -- STEP: 396/36938 -- GLOBAL_STEP: 295900 | > loss_text_ce: 0.03780654817819595 (0.04146776680430077) | > loss_mel_ce: 3.3009419441223145 (nan) | > loss: 0.013249002397060394 (nan) | > current_lr: 5e-06 | > step_time: 0.1355 (0.13623873633567718) | > loader_time: 0.0046 (0.004646760646743008)  --> TIME: 2025-02-24 11:15:27 -- STEP: 446/36938 -- GLOBAL_STEP: 295950 | > loss_text_ce: 0.038487065583467484 (0.04138259565449348) | > loss_mel_ce: 2.5574238300323486 (nan) | > loss: 0.01030123420059681 (nan) | > current_lr: 5e-06 | > step_time: 0.169 (0.1369483952030471) | > loader_time: 0.0048 (0.004646439188798986)  --> TIME: 2025-02-24 11:15:36 -- STEP: 496/36938 -- GLOBAL_STEP: 296000 | > loss_text_ce: 0.033257897943258286 (0.04134858283589803) | > loss_mel_ce: 3.718838930130005 (nan) | > loss: 0.014889273792505264 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13702049418803172) | > loader_time: 0.0045 (0.004641917924727167)  --> TIME: 2025-02-24 11:15:46 -- STEP: 546/36938 -- GLOBAL_STEP: 296050 | > loss_text_ce: 0.028551246970891953 (0.04126312701720676) | > loss_mel_ce: 3.4007222652435303 (nan) | > loss: 0.013608229346573353 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.13727983581277467) | > loader_time: 0.0046 (0.004637004254938486)  --> TIME: 2025-02-24 11:15:55 -- STEP: 596/36938 -- GLOBAL_STEP: 296100 | > loss_text_ce: 0.04473426192998886 (0.0413618854984591) | > loss_mel_ce: 3.7971200942993164 (nan) | > loss: 0.015245454385876656 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13694414636432717) | > loader_time: 0.0047 (0.004632595401482297)  --> TIME: 2025-02-24 11:16:04 -- STEP: 646/36938 -- GLOBAL_STEP: 296150 | > loss_text_ce: 0.04536474496126175 (0.041465093514337074) | > loss_mel_ce: 2.8696835041046143 (nan) | > loss: 0.01156765315681696 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.1368677900302521) | > loader_time: 0.0047 (0.004628709226200829)  --> TIME: 2025-02-24 11:16:14 -- STEP: 696/36938 -- GLOBAL_STEP: 296200 | > loss_text_ce: 0.04781198501586914 (0.04143478172221062) | > loss_mel_ce: 2.5092074871063232 (nan) | > loss: 0.01014690287411213 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13676913305260668) | > loader_time: 0.0045 (0.004628450020976454)  --> TIME: 2025-02-24 11:16:23 -- STEP: 746/36938 -- GLOBAL_STEP: 296250 | > loss_text_ce: 0.051650527864694595 (0.04141902046193226) | > loss_mel_ce: 3.4130165576934814 (nan) | > loss: 0.01374867931008339 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.13700826334250193) | > loader_time: 0.0046 (0.004627653485009245)  --> TIME: 2025-02-24 11:16:33 -- STEP: 796/36938 -- GLOBAL_STEP: 296300 | > loss_text_ce: 0.04694833606481552 (0.041408504139463495) | > loss_mel_ce: 3.7805819511413574 (nan) | > loss: 0.015188613906502724 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13691621240059942) | > loader_time: 0.0045 (0.004627327224118028)  --> TIME: 2025-02-24 11:16:42 -- STEP: 846/36938 -- GLOBAL_STEP: 296350 | > loss_text_ce: 0.038866445422172546 (0.04141559912856165) | > loss_mel_ce: 3.778961181640625 (nan) | > loss: 0.01515011116862297 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13703802826838582) | > loader_time: 0.0049 (0.0046305335159842875)  --> TIME: 2025-02-24 11:16:51 -- STEP: 896/36938 -- GLOBAL_STEP: 296400 | > loss_text_ce: 0.04271755740046501 (0.04143571949264566) | > loss_mel_ce: 3.330901622772217 (nan) | > loss: 0.013387378305196762 (nan) | > current_lr: 5e-06 | > step_time: 0.1778 (0.13694453292659353) | > loader_time: 0.0049 (0.0046294911631516045)  --> TIME: 2025-02-24 11:17:00 -- STEP: 946/36938 -- GLOBAL_STEP: 296450 | > loss_text_ce: 0.04775845259428024 (0.04142223746490213) | > loss_mel_ce: 3.149049758911133 (nan) | > loss: 0.012685746885836124 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13676757817540572) | > loader_time: 0.0046 (0.004630839597851227)  --> TIME: 2025-02-24 11:17:10 -- STEP: 996/36938 -- GLOBAL_STEP: 296500 | > loss_text_ce: 0.0372808538377285 (0.04145641780521974) | > loss_mel_ce: 3.363313674926758 (nan) | > loss: 0.01349442359060049 (nan) | > current_lr: 5e-06 | > step_time: 0.1527 (0.13666443484854038) | > loader_time: 0.0046 (0.004630938112496372)  --> TIME: 2025-02-24 11:17:19 -- STEP: 1046/36938 -- GLOBAL_STEP: 296550 | > loss_text_ce: 0.0384533628821373 (0.0414871306053539) | > loss_mel_ce: 2.224236488342285 (nan) | > loss: 0.008978928439319134 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13648035307459352) | > loader_time: 0.005 (0.004630036837284242)  --> TIME: 2025-02-24 11:17:28 -- STEP: 1096/36938 -- GLOBAL_STEP: 296600 | > loss_text_ce: 0.04097846522927284 (0.041493121908474574) | > loss_mel_ce: 3.214830160140991 (nan) | > loss: 0.012919875793159008 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13657150825444808) | > loader_time: 0.0048 (0.004631216943699079)  --> TIME: 2025-02-24 11:17:37 -- STEP: 1146/36938 -- GLOBAL_STEP: 296650 | > loss_text_ce: 0.04143500328063965 (0.041428040734201296) | > loss_mel_ce: 3.319812536239624 (nan) | > loss: 0.013338284566998482 (nan) | > current_lr: 5e-06 | > step_time: 0.1701 (0.13659342338069372) | > loader_time: 0.0047 (0.004629870567854368)  --> TIME: 2025-02-24 11:17:47 -- STEP: 1196/36938 -- GLOBAL_STEP: 296700 | > loss_text_ce: 0.04153827950358391 (0.04146919562247733) | > loss_mel_ce: 3.3453569412231445 (nan) | > loss: 0.01344006136059761 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.1367229368774389) | > loader_time: 0.0046 (0.004629353615751231)  --> TIME: 2025-02-24 11:17:56 -- STEP: 1246/36938 -- GLOBAL_STEP: 296750 | > loss_text_ce: 0.03152844309806824 (0.0414816683866861) | > loss_mel_ce: 2.8741700649261475 (nan) | > loss: 0.011530550196766853 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.13681156218147575) | > loader_time: 0.0045 (0.004628139552488373)  --> TIME: 2025-02-24 11:18:06 -- STEP: 1296/36938 -- GLOBAL_STEP: 296800 | > loss_text_ce: 0.045721277594566345 (0.04149014827576868) | > loss_mel_ce: 3.353605031967163 (nan) | > loss: 0.013489390723407269 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.1369669998501554) | > loader_time: 0.0049 (0.004629177075845223)  --> TIME: 2025-02-24 11:18:15 -- STEP: 1346/36938 -- GLOBAL_STEP: 296850 | > loss_text_ce: 0.0344163253903389 (0.041528450982454154) | > loss_mel_ce: 0.14266204833984375 (nan) | > loss: 0.000702691962942481 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13697491305012563) | > loader_time: 0.0044 (0.004630375935808159)  --> TIME: 2025-02-24 11:18:25 -- STEP: 1396/36938 -- GLOBAL_STEP: 296900 | > loss_text_ce: 0.03287486732006073 (0.0414756713755279) | > loss_mel_ce: 1.9311124086380005 (nan) | > loss: 0.007793600670993328 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13694431993544603) | > loader_time: 0.0045 (0.004632141494477719)  --> TIME: 2025-02-24 11:18:34 -- STEP: 1446/36938 -- GLOBAL_STEP: 296950 | > loss_text_ce: 0.03414805606007576 (0.041448060521034086) | > loss_mel_ce: 0.86749267578125 (nan) | > loss: 0.0035779394675046206 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13674692428293397) | > loader_time: 0.0047 (0.0046336634360243534)  --> TIME: 2025-02-24 11:18:43 -- STEP: 1496/36938 -- GLOBAL_STEP: 297000 | > loss_text_ce: 0.04382285475730896 (0.041455714010887504) | > loss_mel_ce: 2.4386136531829834 (nan) | > loss: 0.009850938804447651 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13681643978159697) | > loader_time: 0.0045 (0.004633992113531596)  --> TIME: 2025-02-24 11:18:52 -- STEP: 1546/36938 -- GLOBAL_STEP: 297050 | > loss_text_ce: 0.04227619990706444 (0.04144661154063055) | > loss_mel_ce: 3.4137346744537354 (nan) | > loss: 0.013714329339563847 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.13672980793389122) | > loader_time: 0.0049 (0.004632390177727981)  --> TIME: 2025-02-24 11:19:02 -- STEP: 1596/36938 -- GLOBAL_STEP: 297100 | > loss_text_ce: 0.03781198710203171 (0.041445273227924274) | > loss_mel_ce: 2.8342416286468506 (nan) | > loss: 0.011397038586437702 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.13682034365216597) | > loader_time: 0.0047 (0.00463403600797916)  --> TIME: 2025-02-24 11:19:11 -- STEP: 1646/36938 -- GLOBAL_STEP: 297150 | > loss_text_ce: 0.04055579751729965 (0.0414784894294285) | > loss_mel_ce: 3.236366033554077 (nan) | > loss: 0.013003658503293991 (nan) | > current_lr: 5e-06 | > step_time: 0.1533 (0.13697713780084486) | > loader_time: 0.0046 (0.004636302898170589)  --> TIME: 2025-02-24 11:19:21 -- STEP: 1696/36938 -- GLOBAL_STEP: 297200 | > loss_text_ce: 0.045441437512636185 (0.04145977306421315) | > loss_mel_ce: 3.3462305068969727 (nan) | > loss: 0.01345901656895876 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13702375793232127) | > loader_time: 0.0045 (0.004635133130370441)  --> TIME: 2025-02-24 11:19:30 -- STEP: 1746/36938 -- GLOBAL_STEP: 297250 | > loss_text_ce: 0.04299358278512955 (0.04142666324004442) | > loss_mel_ce: 3.730811595916748 (nan) | > loss: 0.014975418336689472 (nan) | > current_lr: 5e-06 | > step_time: 0.186 (0.13697742173215663) | > loader_time: 0.0048 (0.004633328212768618)  --> TIME: 2025-02-24 11:19:39 -- STEP: 1796/36938 -- GLOBAL_STEP: 297300 | > loss_text_ce: 0.0370325893163681 (0.04147331627658284) | > loss_mel_ce: 3.404557466506958 (nan) | > loss: 0.013657104223966599 (nan) | > current_lr: 5e-06 | > step_time: 0.195 (0.13703424667727987) | > loader_time: 0.0048 (0.004634167409952068)  --> TIME: 2025-02-24 11:19:49 -- STEP: 1846/36938 -- GLOBAL_STEP: 297350 | > loss_text_ce: 0.04657970368862152 (0.041459298769328884) | > loss_mel_ce: 1.3958390951156616 (nan) | > loss: 0.005723884329199791 (nan) | > current_lr: 5e-06 | > step_time: 0.1512 (0.1370443621022732) | > loader_time: 0.0046 (0.004633348011376404)  --> TIME: 2025-02-24 11:19:58 -- STEP: 1896/36938 -- GLOBAL_STEP: 297400 | > loss_text_ce: 0.03013656847178936 (0.041435056284486516) | > loss_mel_ce: 3.054737091064453 (nan) | > loss: 0.012241562828421593 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13685820399457416) | > loader_time: 0.0045 (0.004632563404896082)  --> TIME: 2025-02-24 11:20:07 -- STEP: 1946/36938 -- GLOBAL_STEP: 297450 | > loss_text_ce: 0.03449234366416931 (0.041457752361204456) | > loss_mel_ce: 2.9999420642852783 (nan) | > loss: 0.012041406705975533 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13682454095469984) | > loader_time: 0.0045 (0.004631982187818029)  --> TIME: 2025-02-24 11:20:16 -- STEP: 1996/36938 -- GLOBAL_STEP: 297500 | > loss_text_ce: 0.05112269148230553 (0.04144635808491751) | > loss_mel_ce: 3.8687596321105957 (nan) | > loss: 0.015555089339613914 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13671534309884134) | > loader_time: 0.0047 (0.004631032805165684)  --> TIME: 2025-02-24 11:20:25 -- STEP: 2046/36938 -- GLOBAL_STEP: 297550 | > loss_text_ce: 0.054674241691827774 (0.041458706503491795) | > loss_mel_ce: 3.9281563758850098 (nan) | > loss: 0.01580488309264183 (nan) | > current_lr: 5e-06 | > step_time: 0.1632 (0.1366548968200462) | > loader_time: 0.0049 (0.004632000000246103)  --> TIME: 2025-02-24 11:20:35 -- STEP: 2096/36938 -- GLOBAL_STEP: 297600 | > loss_text_ce: 0.036117859184741974 (0.0414811561669699) | > loss_mel_ce: 3.342005491256714 (nan) | > loss: 0.013405252248048782 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13659870044420716) | > loader_time: 0.0046 (0.004633269464696631)  --> TIME: 2025-02-24 11:20:44 -- STEP: 2146/36938 -- GLOBAL_STEP: 297650 | > loss_text_ce: 0.04289159178733826 (0.041501950475197155) | > loss_mel_ce: 3.4282329082489014 (nan) | > loss: 0.01377430371940136 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13663511562880704) | > loader_time: 0.0046 (0.0046320361504461356)  --> TIME: 2025-02-24 11:20:53 -- STEP: 2196/36938 -- GLOBAL_STEP: 297700 | > loss_text_ce: 0.0430554524064064 (0.04149285956491152) | > loss_mel_ce: 0.6912597417831421 (nan) | > loss: 0.0029139495454728603 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.1366227808762988) | > loader_time: 0.0045 (0.004632279195420722)  --> TIME: 2025-02-24 11:21:03 -- STEP: 2246/36938 -- GLOBAL_STEP: 297750 | > loss_text_ce: 0.042971257120370865 (0.04148649007748751) | > loss_mel_ce: 3.2649033069610596 (nan) | > loss: 0.013126487843692303 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13678340633745628) | > loader_time: 0.0047 (0.004632590078182334)  --> TIME: 2025-02-24 11:21:12 -- STEP: 2296/36938 -- GLOBAL_STEP: 297800 | > loss_text_ce: 0.037245601415634155 (0.04148937457886873) | > loss_mel_ce: 3.445888042449951 (nan) | > loss: 0.013821959495544434 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13677878724573397) | > loader_time: 0.0046 (0.00463268389269865)  --> TIME: 2025-02-24 11:21:22 -- STEP: 2346/36938 -- GLOBAL_STEP: 297850 | > loss_text_ce: 0.03468222916126251 (0.04148878867421151) | > loss_mel_ce: 3.50496506690979 (nan) | > loss: 0.014046220108866692 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.1368102364731162) | > loader_time: 0.0047 (0.004633064668400102)  --> TIME: 2025-02-24 11:21:31 -- STEP: 2396/36938 -- GLOBAL_STEP: 297900 | > loss_text_ce: 0.05482469126582146 (0.04151037826711223) | > loss_mel_ce: 3.486328125 (nan) | > loss: 0.01405219454318285 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13679685238406744) | > loader_time: 0.0044 (0.004632750019207219)  --> TIME: 2025-02-24 11:21:40 -- STEP: 2446/36938 -- GLOBAL_STEP: 297950 | > loss_text_ce: 0.0408807173371315 (0.04153861083434501) | > loss_mel_ce: 3.440347194671631 (nan) | > loss: 0.013814397156238556 (nan) | > current_lr: 5e-06 | > step_time: 0.1567 (0.1367351879180697) | > loader_time: 0.0048 (0.004633249460627377)  --> TIME: 2025-02-24 11:21:49 -- STEP: 2496/36938 -- GLOBAL_STEP: 298000 | > loss_text_ce: 0.04170352220535278 (0.041554125948982175) | > loss_mel_ce: 3.215376615524292 (nan) | > loss: 0.012924921698868275 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.1366554172948386) | > loader_time: 0.0047 (0.004632945626209938)  --> TIME: 2025-02-24 11:21:59 -- STEP: 2546/36938 -- GLOBAL_STEP: 298050 | > loss_text_ce: 0.03620180860161781 (0.0415364533168236) | > loss_mel_ce: 0.109893798828125 (nan) | > loss: 0.000579744519200176 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13659431376236228) | > loader_time: 0.0045 (0.004632997119510998)  --> TIME: 2025-02-24 11:22:08 -- STEP: 2596/36938 -- GLOBAL_STEP: 298100 | > loss_text_ce: 0.03697530925273895 (0.04150633046831155) | > loss_mel_ce: 2.6330461502075195 (nan) | > loss: 0.010595324449241161 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.1365642442908972) | > loader_time: 0.0047 (0.004632976371077428)  --> TIME: 2025-02-24 11:22:17 -- STEP: 2646/36938 -- GLOBAL_STEP: 298150 | > loss_text_ce: 0.03839031979441643 (0.041492741579613614) | > loss_mel_ce: 2.6115732192993164 (nan) | > loss: 0.01051572896540165 (nan) | > current_lr: 5e-06 | > step_time: 0.1521 (0.13662669718130405) | > loader_time: 0.0046 (0.004632527325429067)  --> TIME: 2025-02-24 11:22:27 -- STEP: 2696/36938 -- GLOBAL_STEP: 298200 | > loss_text_ce: 0.02962435409426689 (0.041462304075428114) | > loss_mel_ce: 3.1342058181762695 (nan) | > loss: 0.012554883025586605 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.1366259249629184) | > loader_time: 0.0047 (0.004635877442996646)  --> TIME: 2025-02-24 11:22:36 -- STEP: 2746/36938 -- GLOBAL_STEP: 298250 | > loss_text_ce: 0.042455147951841354 (0.04145257716058311) | > loss_mel_ce: 3.4675378799438477 (nan) | > loss: 0.013928544707596302 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13674051303196716) | > loader_time: 0.0045 (0.004636372342564635)  --> TIME: 2025-02-24 11:22:46 -- STEP: 2796/36938 -- GLOBAL_STEP: 298300 | > loss_text_ce: 0.048085711896419525 (0.04144989151989926) | > loss_mel_ce: 3.3704545497894287 (nan) | > loss: 0.013565636239945889 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13678677501596603) | > loader_time: 0.0046 (0.004638678867247308)  --> TIME: 2025-02-24 11:22:55 -- STEP: 2846/36938 -- GLOBAL_STEP: 298350 | > loss_text_ce: 0.03824457898736 (0.04147307979412899) | > loss_mel_ce: 3.925100564956665 (nan) | > loss: 0.015727560967206955 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.13682575839050043) | > loader_time: 0.0049 (0.004639591537389802)  --> TIME: 2025-02-24 11:23:05 -- STEP: 2896/36938 -- GLOBAL_STEP: 298400 | > loss_text_ce: 0.04968588799238205 (0.04149045860677829) | > loss_mel_ce: 3.2624692916870117 (nan) | > loss: 0.013143474236130714 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.13689186782138793) | > loader_time: 0.005 (0.0046404344930174556)  --> TIME: 2025-02-24 11:23:14 -- STEP: 2946/36938 -- GLOBAL_STEP: 298450 | > loss_text_ce: 0.03834149241447449 (0.04152096118031915) | > loss_mel_ce: 3.0824856758117676 (nan) | > loss: 0.012384235858917236 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.13688777768522906) | > loader_time: 0.0046 (0.004640395675317331)  --> TIME: 2025-02-24 11:23:24 -- STEP: 2996/36938 -- GLOBAL_STEP: 298500 | > loss_text_ce: 0.03557825833559036 (0.041515481757753554) | > loss_mel_ce: 1.387728214263916 (nan) | > loss: 0.0056480420753359795 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13697970987798705) | > loader_time: 0.0048 (0.0046407301053504255)  --> TIME: 2025-02-24 11:23:33 -- STEP: 3046/36938 -- GLOBAL_STEP: 298550 | > loss_text_ce: 0.03881191089749336 (0.041529753551777404) | > loss_mel_ce: 3.159374713897705 (nan) | > loss: 0.012691217474639416 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.1369785253577381) | > loader_time: 0.0047 (0.00464008266175959)  --> TIME: 2025-02-24 11:23:43 -- STEP: 3096/36938 -- GLOBAL_STEP: 298600 | > loss_text_ce: 0.03957260027527809 (0.04150515847396608) | > loss_mel_ce: 3.545572519302368 (nan) | > loss: 0.014226767234504223 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13701270655452133) | > loader_time: 0.0045 (0.004640186094806477)  --> TIME: 2025-02-24 11:23:52 -- STEP: 3146/36938 -- GLOBAL_STEP: 298650 | > loss_text_ce: 0.03867727890610695 (0.04151145812746862) | > loss_mel_ce: 3.1732914447784424 (nan) | > loss: 0.012745908461511135 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13697880601246334) | > loader_time: 0.0046 (0.004639972567330927)  --> TIME: 2025-02-24 11:24:01 -- STEP: 3196/36938 -- GLOBAL_STEP: 298700 | > loss_text_ce: 0.03813532367348671 (0.041482631396269065) | > loss_mel_ce: 2.045532703399658 (nan) | > loss: 0.008268523961305618 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13690178862799685) | > loader_time: 0.0047 (0.004639372733119726)  --> TIME: 2025-02-24 11:24:11 -- STEP: 3246/36938 -- GLOBAL_STEP: 298750 | > loss_text_ce: 0.0458461232483387 (0.04148627139222804) | > loss_mel_ce: 0.7848946452140808 (nan) | > loss: 0.0032965904101729393 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.1369457653989339) | > loader_time: 0.0046 (0.004638982788692338)  --> TIME: 2025-02-24 11:24:20 -- STEP: 3296/36938 -- GLOBAL_STEP: 298800 | > loss_text_ce: 0.033544935286045074 (0.04147223233232874) | > loss_mel_ce: 3.523900270462036 (nan) | > loss: 0.014116846956312656 (nan) | > current_lr: 5e-06 | > step_time: 0.1648 (0.1369559948594824) | > loader_time: 0.0047 (0.004638909208543093)  --> TIME: 2025-02-24 11:24:29 -- STEP: 3346/36938 -- GLOBAL_STEP: 298850 | > loss_text_ce: 0.04483889043331146 (0.04148147098894864) | > loss_mel_ce: 3.3800222873687744 (nan) | > loss: 0.013590719550848007 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.1369078224174149) | > loader_time: 0.0048 (0.004637868619577943)  --> TIME: 2025-02-24 11:24:38 -- STEP: 3396/36938 -- GLOBAL_STEP: 298900 | > loss_text_ce: 0.05603301525115967 (0.04148411099662947) | > loss_mel_ce: 3.311974048614502 (nan) | > loss: 0.013365108519792557 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13687445769180823) | > loader_time: 0.0046 (0.004637166945476559)  --> TIME: 2025-02-24 11:24:48 -- STEP: 3446/36938 -- GLOBAL_STEP: 298950 | > loss_text_ce: 0.042186588048934937 (0.041470536921341915) | > loss_mel_ce: 3.5551223754882812 (nan) | > loss: 0.014275035820901394 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13696152258943278) | > loader_time: 0.0048 (0.004637393644225513)  --> TIME: 2025-02-24 11:24:58 -- STEP: 3496/36938 -- GLOBAL_STEP: 299000 | > loss_text_ce: 0.040895652025938034 (0.04145544170198457) | > loss_mel_ce: 3.1216344833374023 (nan) | > loss: 0.012549723498523235 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1370246777272606) | > loader_time: 0.0045 (0.004637911131507478)  --> TIME: 2025-02-24 11:25:07 -- STEP: 3546/36938 -- GLOBAL_STEP: 299050 | > loss_text_ce: 0.05346377566456795 (0.04145425406257835) | > loss_mel_ce: 3.7843589782714844 (nan) | > loss: 0.015229456126689911 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.1369455007261565) | > loader_time: 0.0046 (0.004637096066698144)  --> TIME: 2025-02-24 11:25:16 -- STEP: 3596/36938 -- GLOBAL_STEP: 299100 | > loss_text_ce: 0.032096266746520996 (0.04143740634384193) | > loss_mel_ce: 2.5498735904693604 (nan) | > loss: 0.010245912708342075 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.1370047750541975) | > loader_time: 0.0045 (0.004636860463987872)  --> TIME: 2025-02-24 11:25:25 -- STEP: 3646/36938 -- GLOBAL_STEP: 299150 | > loss_text_ce: 0.037995267659425735 (0.041428986230524815) | > loss_mel_ce: 2.5683822631835938 (nan) | > loss: 0.010342768393456936 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13691620243426142) | > loader_time: 0.0045 (0.004636063656830493)  --> TIME: 2025-02-24 11:25:34 -- STEP: 3696/36938 -- GLOBAL_STEP: 299200 | > loss_text_ce: 0.04411382973194122 (0.041416428702756365) | > loss_mel_ce: 3.1468005180358887 (nan) | > loss: 0.01266235951334238 (nan) | > current_lr: 5e-06 | > step_time: 0.185 (0.13688637425889166) | > loader_time: 0.0048 (0.004636197121112383)  --> TIME: 2025-02-24 11:25:44 -- STEP: 3746/36938 -- GLOBAL_STEP: 299250 | > loss_text_ce: 0.03687561675906181 (0.04142337054778948) | > loss_mel_ce: 0.03668212890625 (nan) | > loss: 0.0002918958489317447 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13688905590432107) | > loader_time: 0.0046 (0.00463717580796307)  --> TIME: 2025-02-24 11:25:53 -- STEP: 3796/36938 -- GLOBAL_STEP: 299300 | > loss_text_ce: 0.04150491952896118 (0.0414540909009409) | > loss_mel_ce: 3.5653984546661377 (nan) | > loss: 0.01431310921907425 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13685057983508964) | > loader_time: 0.0048 (0.0046374208683460155)  --> TIME: 2025-02-24 11:26:02 -- STEP: 3846/36938 -- GLOBAL_STEP: 299350 | > loss_text_ce: 0.046996645629405975 (0.04146185710428551) | > loss_mel_ce: 3.176910877227783 (nan) | > loss: 0.01279328390955925 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13688961336281652) | > loader_time: 0.0045 (0.004637775418662931)  --> TIME: 2025-02-24 11:26:11 -- STEP: 3896/36938 -- GLOBAL_STEP: 299400 | > loss_text_ce: 0.04136090725660324 (0.04145661645566033) | > loss_mel_ce: 3.421834707260132 (nan) | > loss: 0.01374283991754055 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13682190777095177) | > loader_time: 0.0047 (0.004637707369038707)  --> TIME: 2025-02-24 11:26:20 -- STEP: 3946/36938 -- GLOBAL_STEP: 299450 | > loss_text_ce: 0.05057172477245331 (0.04145531438680967) | > loss_mel_ce: 3.349027156829834 (nan) | > loss: 0.013490471988916397 (nan) | > current_lr: 5e-06 | > step_time: 0.1769 (0.1367653117709151) | > loader_time: 0.0048 (0.004637908041326158)  --> TIME: 2025-02-24 11:26:30 -- STEP: 3996/36938 -- GLOBAL_STEP: 299500 | > loss_text_ce: 0.04769647866487503 (0.04144979409152847) | > loss_mel_ce: 2.762840986251831 (nan) | > loss: 0.01115292776376009 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13680457496070275) | > loader_time: 0.0046 (0.004638460723010156)  --> TIME: 2025-02-24 11:26:40 -- STEP: 4046/36938 -- GLOBAL_STEP: 299550 | > loss_text_ce: 0.04347153380513191 (0.04145743301206411) | > loss_mel_ce: 3.541308641433716 (nan) | > loss: 0.014225319027900696 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.13683195734047604) | > loader_time: 0.0047 (0.004638212421302216)  --> TIME: 2025-02-24 11:26:49 -- STEP: 4096/36938 -- GLOBAL_STEP: 299600 | > loss_text_ce: 0.03416765481233597 (0.04146564232701161) | > loss_mel_ce: 3.3153233528137207 (nan) | > loss: 0.013291631825268269 (nan) | > current_lr: 5e-06 | > step_time: 0.1567 (0.13683266611769782) | > loader_time: 0.0046 (0.0046383414301089994)  --> TIME: 2025-02-24 11:26:58 -- STEP: 4146/36938 -- GLOBAL_STEP: 299650 | > loss_text_ce: 0.0342465378344059 (0.041469723807249725) | > loss_mel_ce: 3.5057036876678467 (nan) | > loss: 0.014047421514987946 (nan) | > current_lr: 5e-06 | > step_time: 0.1894 (0.13687484803777109) | > loader_time: 0.0048 (0.004639343943966228)  --> TIME: 2025-02-24 11:27:08 -- STEP: 4196/36938 -- GLOBAL_STEP: 299700 | > loss_text_ce: 0.04813478887081146 (0.04149827184142261) | > loss_mel_ce: 3.028862714767456 (nan) | > loss: 0.012210308574140072 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13686077582256573) | > loader_time: 0.0046 (0.0046391532009732095)  --> TIME: 2025-02-24 11:27:17 -- STEP: 4246/36938 -- GLOBAL_STEP: 299750 | > loss_text_ce: 0.04550381377339363 (0.041493387720659414) | > loss_mel_ce: 3.3138575553894043 (nan) | > loss: 0.013330800458788872 (nan) | > current_lr: 5e-06 | > step_time: 0.1623 (0.1369110100459108) | > loader_time: 0.005 (0.0046395579431060385)  --> TIME: 2025-02-24 11:27:26 -- STEP: 4296/36938 -- GLOBAL_STEP: 299800 | > loss_text_ce: 0.04445572569966316 (0.041488660214376846) | > loss_mel_ce: 2.9623632431030273 (nan) | > loss: 0.011931821703910828 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13685156002612964) | > loader_time: 0.0044 (0.004639105035384056)  --> TIME: 2025-02-24 11:27:36 -- STEP: 4346/36938 -- GLOBAL_STEP: 299850 | > loss_text_ce: 0.03471748158335686 (0.04148511096403971) | > loss_mel_ce: 0.01123046875 (nan) | > loss: 0.00018233314040116966 (nan) | > current_lr: 5e-06 | > step_time: 0.1754 (0.13689526099517918) | > loader_time: 0.0048 (0.004639051885468812)  --> TIME: 2025-02-24 11:27:45 -- STEP: 4396/36938 -- GLOBAL_STEP: 299900 | > loss_text_ce: 0.046094708144664764 (0.041485944210515165) | > loss_mel_ce: 3.2231979370117188 (nan) | > loss: 0.012973384000360966 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13687115999869137) | > loader_time: 0.0046 (0.00463906996243645)  --> TIME: 2025-02-24 11:27:55 -- STEP: 4446/36938 -- GLOBAL_STEP: 299950 | > loss_text_ce: 0.050081245601177216 (0.04148396469541204) | > loss_mel_ce: 2.797748565673828 (nan) | > loss: 0.01130091305822134 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.136885110665912) | > loader_time: 0.0045 (0.0046389319582453145)  --> TIME: 2025-02-24 11:28:04 -- STEP: 4496/36938 -- GLOBAL_STEP: 300000 | > loss_text_ce: 0.03586005046963692 (0.04147522873721602) | > loss_mel_ce: 2.0868234634399414 (nan) | > loss: 0.00842334795743227 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13687445918011903) | > loader_time: 0.0046 (0.004639709759437321) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_300000.pth  --> TIME: 2025-02-24 11:28:19 -- STEP: 4546/36938 -- GLOBAL_STEP: 300050 | > loss_text_ce: 0.04082787409424782 (0.041471104425436625) | > loss_mel_ce: 3.3978874683380127 (nan) | > loss: 0.013645697385072708 (nan) | > current_lr: 5e-06 | > step_time: 0.1485 (0.1368276837424878) | > loader_time: 0.0047 (0.004640567265991804)  --> TIME: 2025-02-24 11:28:29 -- STEP: 4596/36938 -- GLOBAL_STEP: 300100 | > loss_text_ce: 0.029728775843977928 (0.04147774034408312) | > loss_mel_ce: 3.6047468185424805 (nan) | > loss: 0.0144225237891078 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13683410558210846) | > loader_time: 0.0046 (0.004641358337784143)  --> TIME: 2025-02-24 11:28:38 -- STEP: 4646/36938 -- GLOBAL_STEP: 300150 | > loss_text_ce: 0.050020381808280945 (0.041469759731767884) | > loss_mel_ce: 1.9600684642791748 (nan) | > loss: 0.007976544089615345 (nan) | > current_lr: 5e-06 | > step_time: 0.1614 (0.13680764189887226) | > loader_time: 0.0049 (0.004641653030943979)  --> TIME: 2025-02-24 11:28:47 -- STEP: 4696/36938 -- GLOBAL_STEP: 300200 | > loss_text_ce: 0.04761137068271637 (0.04146534971217735) | > loss_mel_ce: 2.8829476833343506 (nan) | > loss: 0.011629203334450722 (nan) | > current_lr: 5e-06 | > step_time: 0.1788 (0.13678993363396663) | > loader_time: 0.005 (0.004642351421018268)  --> TIME: 2025-02-24 11:28:56 -- STEP: 4746/36938 -- GLOBAL_STEP: 300250 | > loss_text_ce: 0.04414768144488335 (0.041449889950210844) | > loss_mel_ce: 2.7264246940612793 (nan) | > loss: 0.010994335636496544 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13671796647396137) | > loader_time: 0.0046 (0.004642203393687789)  --> TIME: 2025-02-24 11:29:06 -- STEP: 4796/36938 -- GLOBAL_STEP: 300300 | > loss_text_ce: 0.04665292799472809 (0.041477712668017044) | > loss_mel_ce: 3.655871868133545 (nan) | > loss: 0.014692559838294983 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13673736752819446) | > loader_time: 0.0046 (0.004642149674285936)  --> TIME: 2025-02-24 11:29:15 -- STEP: 4846/36938 -- GLOBAL_STEP: 300350 | > loss_text_ce: 0.04493880644440651 (0.04148977011303038) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13667985165950602) | > loader_time: 0.0046 (0.004641625834947964)  --> TIME: 2025-02-24 11:29:24 -- STEP: 4896/36938 -- GLOBAL_STEP: 300400 | > loss_text_ce: 0.04088974371552467 (0.041484583127563396) | > loss_mel_ce: 3.26139760017395 (nan) | > loss: 0.013104315847158432 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.1366738158405993) | > loader_time: 0.0048 (0.004641849036310231)  --> TIME: 2025-02-24 11:29:33 -- STEP: 4946/36938 -- GLOBAL_STEP: 300450 | > loss_text_ce: 0.040640998631715775 (0.041471736200019704) | > loss_mel_ce: 3.2940824031829834 (nan) | > loss: 0.013233030214905739 (nan) | > current_lr: 5e-06 | > step_time: 0.1578 (0.13665822614699843) | > loader_time: 0.0046 (0.004642530341790454)  --> TIME: 2025-02-24 11:29:42 -- STEP: 4996/36938 -- GLOBAL_STEP: 300500 | > loss_text_ce: 0.041625939309597015 (0.041471391919071135) | > loss_mel_ce: 2.703653335571289 (nan) | > loss: 0.01089396607130766 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13662167784688514) | > loader_time: 0.0046 (0.004643105477500281)  --> TIME: 2025-02-24 11:29:51 -- STEP: 5046/36938 -- GLOBAL_STEP: 300550 | > loss_text_ce: 0.04395459592342377 (0.04147526765540467) | > loss_mel_ce: 1.6982008218765259 (nan) | > loss: 0.006913315504789352 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.1366153018038936) | > loader_time: 0.0046 (0.004643264054968579)  --> TIME: 2025-02-24 11:30:01 -- STEP: 5096/36938 -- GLOBAL_STEP: 300600 | > loss_text_ce: 0.03822793811559677 (0.04146331741534498) | > loss_mel_ce: 3.245269775390625 (nan) | > loss: 0.013029754161834717 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.1365998547810768) | > loader_time: 0.0046 (0.004643515851845734)  --> TIME: 2025-02-24 11:30:09 -- STEP: 5146/36938 -- GLOBAL_STEP: 300650 | > loss_text_ce: 0.051976706832647324 (0.04146748901157686) | > loss_mel_ce: 3.471426010131836 (nan) | > loss: 0.013981757685542107 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.1365201057648308) | > loader_time: 0.0048 (0.004643482361329095)  --> TIME: 2025-02-24 11:30:19 -- STEP: 5196/36938 -- GLOBAL_STEP: 300700 | > loss_text_ce: 0.04159194976091385 (0.041469264623161826) | > loss_mel_ce: 3.3198494911193848 (nan) | > loss: 0.013339053839445114 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.1365367298038113) | > loader_time: 0.0046 (0.004643615068519362)  --> TIME: 2025-02-24 11:30:28 -- STEP: 5246/36938 -- GLOBAL_STEP: 300750 | > loss_text_ce: 0.03465209901332855 (0.041449073751095064) | > loss_mel_ce: 3.011122465133667 (nan) | > loss: 0.012086407281458378 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13651250029537274) | > loader_time: 0.0045 (0.004643626673004348)  --> TIME: 2025-02-24 11:30:38 -- STEP: 5296/36938 -- GLOBAL_STEP: 300800 | > loss_text_ce: 0.04258428514003754 (0.04145250552871285) | > loss_mel_ce: 3.4739952087402344 (nan) | > loss: 0.01395468134433031 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.1365459269627335) | > loader_time: 0.0047 (0.0046443938758798245)  --> TIME: 2025-02-24 11:30:47 -- STEP: 5346/36938 -- GLOBAL_STEP: 300850 | > loss_text_ce: 0.04077526181936264 (0.04145206852017805) | > loss_mel_ce: 3.3373451232910156 (nan) | > loss: 0.013405240140855312 (nan) | > current_lr: 5e-06 | > step_time: 0.17 (0.13654876018435594) | > loader_time: 0.0048 (0.004644914062291642)  --> TIME: 2025-02-24 11:30:56 -- STEP: 5396/36938 -- GLOBAL_STEP: 300900 | > loss_text_ce: 0.053345125168561935 (0.04144315886655721) | > loss_mel_ce: 1.937546730041504 (nan) | > loss: 0.007900364696979523 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13656424760288177) | > loader_time: 0.0047 (0.004645246015468117)  --> TIME: 2025-02-24 11:31:06 -- STEP: 5446/36938 -- GLOBAL_STEP: 300950 | > loss_text_ce: 0.03823418915271759 (0.041443594029221716) | > loss_mel_ce: 2.8285086154937744 (nan) | > loss: 0.011375964619219303 (nan) | > current_lr: 5e-06 | > step_time: 0.1588 (0.13654676347894246) | > loader_time: 0.0047 (0.004645246028725034)  --> TIME: 2025-02-24 11:31:15 -- STEP: 5496/36938 -- GLOBAL_STEP: 301000 | > loss_text_ce: 0.04050125926733017 (0.04145551087935289) | > loss_mel_ce: 3.4818339347839355 (nan) | > loss: 0.013977522030472755 (nan) | > current_lr: 5e-06 | > step_time: 0.1446 (0.13657289504831258) | > loader_time: 0.0047 (0.004645354102272135)  --> TIME: 2025-02-24 11:31:25 -- STEP: 5546/36938 -- GLOBAL_STEP: 301050 | > loss_text_ce: 0.042097773402929306 (0.04145573309041335) | > loss_mel_ce: 3.6391525268554688 (nan) | > loss: 0.014608137309551239 (nan) | > current_lr: 5e-06 | > step_time: 0.1697 (0.1366124175061732) | > loader_time: 0.0048 (0.004645694260815616)  --> TIME: 2025-02-24 11:31:34 -- STEP: 5596/36938 -- GLOBAL_STEP: 301100 | > loss_text_ce: 0.043375447392463684 (0.041449996275964344) | > loss_mel_ce: 3.8673908710479736 (nan) | > loss: 0.0155189149081707 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.1365816259145555) | > loader_time: 0.0049 (0.004646524052009827)  --> TIME: 2025-02-24 11:31:43 -- STEP: 5646/36938 -- GLOBAL_STEP: 301150 | > loss_text_ce: 0.03066125139594078 (0.04143861352241375) | > loss_mel_ce: 2.3900651931762695 (nan) | > loss: 0.009606058709323406 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13660564931030902) | > loader_time: 0.0045 (0.004647371197185993)  --> TIME: 2025-02-24 11:31:53 -- STEP: 5696/36938 -- GLOBAL_STEP: 301200 | > loss_text_ce: 0.0468197762966156 (0.04144080755107221) | > loss_mel_ce: 3.7115535736083984 (nan) | > loss: 0.014914180152118206 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.13665164567613747) | > loader_time: 0.0048 (0.004648083214009744)  --> TIME: 2025-02-24 11:32:02 -- STEP: 5746/36938 -- GLOBAL_STEP: 301250 | > loss_text_ce: 0.04142720252275467 (0.0414390465166202) | > loss_mel_ce: 3.1222853660583496 (nan) | > loss: 0.012554415501654148 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.13668120314010088) | > loader_time: 0.0047 (0.004648498571006171)  --> TIME: 2025-02-24 11:32:12 -- STEP: 5796/36938 -- GLOBAL_STEP: 301300 | > loss_text_ce: 0.03661283478140831 (0.04143655941814555) | > loss_mel_ce: 0.12081298977136612 (nan) | > loss: 0.0006247056880965829 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13672266749368073) | > loader_time: 0.0046 (0.004649458423493397)  --> TIME: 2025-02-24 11:32:21 -- STEP: 5846/36938 -- GLOBAL_STEP: 301350 | > loss_text_ce: 0.04397899657487869 (0.04143615240046985) | > loss_mel_ce: 3.120415687561035 (nan) | > loss: 0.01255712192505598 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13672306490319536) | > loader_time: 0.0046 (0.004649375221560432)  --> TIME: 2025-02-24 11:32:31 -- STEP: 5896/36938 -- GLOBAL_STEP: 301400 | > loss_text_ce: 0.03478776291012764 (0.04142189568276361) | > loss_mel_ce: 3.615152359008789 (nan) | > loss: 0.014483889564871788 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.1367283388668989) | > loader_time: 0.0046 (0.0046494460817593205)  --> TIME: 2025-02-24 11:32:40 -- STEP: 5946/36938 -- GLOBAL_STEP: 301450 | > loss_text_ce: 0.04546520486474037 (0.041419202895077493) | > loss_mel_ce: 2.9294140338897705 (nan) | > loss: 0.011805077083408833 (nan) | > current_lr: 5e-06 | > step_time: 0.1482 (0.13673407298484755) | > loader_time: 0.0046 (0.00464973307762569)  --> TIME: 2025-02-24 11:32:49 -- STEP: 5996/36938 -- GLOBAL_STEP: 301500 | > loss_text_ce: 0.046475719660520554 (0.04141871216927482) | > loss_mel_ce: 2.9068500995635986 (nan) | > loss: 0.01171954721212387 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.1367032484979599) | > loader_time: 0.0047 (0.00464982987245454)  --> TIME: 2025-02-24 11:32:59 -- STEP: 6046/36938 -- GLOBAL_STEP: 301550 | > loss_text_ce: 0.040265388786792755 (0.0414253142435907) | > loss_mel_ce: 3.1831467151641846 (nan) | > loss: 0.012791317887604237 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13670597457349315) | > loader_time: 0.0048 (0.004649941470896688)  --> TIME: 2025-02-24 11:33:08 -- STEP: 6096/36938 -- GLOBAL_STEP: 301600 | > loss_text_ce: 0.0356028787791729 (0.04142857515335614) | > loss_mel_ce: 3.002570867538452 (nan) | > loss: 0.012056245468556881 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13669262959418851) | > loader_time: 0.0046 (0.004650016547500929)  --> TIME: 2025-02-24 11:33:17 -- STEP: 6146/36938 -- GLOBAL_STEP: 301650 | > loss_text_ce: 0.036787617951631546 (0.04142236108902263) | > loss_mel_ce: 2.851374626159668 (nan) | > loss: 0.011460961773991585 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.13665280988384657) | > loader_time: 0.0046 (0.004649945085924611)  --> TIME: 2025-02-24 11:33:26 -- STEP: 6196/36938 -- GLOBAL_STEP: 301700 | > loss_text_ce: 0.04279144108295441 (0.04142659181669425) | > loss_mel_ce: 2.7477569580078125 (nan) | > loss: 0.01107360515743494 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.136640756802838) | > loader_time: 0.0045 (0.004650321485459072)  --> TIME: 2025-02-24 11:33:36 -- STEP: 6246/36938 -- GLOBAL_STEP: 301750 | > loss_text_ce: 0.028361495584249496 (0.041431334591820954) | > loss_mel_ce: 3.8054141998291016 (nan) | > loss: 0.015213396400213242 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.1366960987081282) | > loader_time: 0.0045 (0.004650686934633283)  --> TIME: 2025-02-24 11:33:45 -- STEP: 6296/36938 -- GLOBAL_STEP: 301800 | > loss_text_ce: 0.04794362932443619 (0.04141814853203753) | > loss_mel_ce: 3.521092176437378 (nan) | > loss: 0.014162841252982616 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13666343340576725) | > loader_time: 0.0047 (0.004650944372901772)  --> TIME: 2025-02-24 11:33:55 -- STEP: 6346/36938 -- GLOBAL_STEP: 301850 | > loss_text_ce: 0.0450810082256794 (0.04141670960496796) | > loss_mel_ce: 3.7582714557647705 (nan) | > loss: 0.015092669054865837 (nan) | > current_lr: 5e-06 | > step_time: 0.1857 (0.13667257242385988) | > loader_time: 0.0048 (0.0046508684543040985)  --> TIME: 2025-02-24 11:34:04 -- STEP: 6396/36938 -- GLOBAL_STEP: 301900 | > loss_text_ce: 0.03460146486759186 (0.041407049994238825) | > loss_mel_ce: 3.4162850379943848 (nan) | > loss: 0.013693994842469692 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.1366421755512767) | > loader_time: 0.0047 (0.004651094839824898)  --> TIME: 2025-02-24 11:34:13 -- STEP: 6446/36938 -- GLOBAL_STEP: 301950 | > loss_text_ce: 0.0321628600358963 (0.04141936834489502) | > loss_mel_ce: 3.086858034133911 (nan) | > loss: 0.012377067469060421 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13660521485810567) | > loader_time: 0.0045 (0.004650970293874153)  --> TIME: 2025-02-24 11:34:22 -- STEP: 6496/36938 -- GLOBAL_STEP: 302000 | > loss_text_ce: 0.04107310250401497 (0.041433223105853394) | > loss_mel_ce: 3.3239386081695557 (nan) | > loss: 0.01335322204977274 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.1365662563243511) | > loader_time: 0.0048 (0.004650882715955751)  --> TIME: 2025-02-24 11:34:31 -- STEP: 6546/36938 -- GLOBAL_STEP: 302050 | > loss_text_ce: 0.036985330283641815 (0.04143463332377298) | > loss_mel_ce: 3.1921792030334473 (nan) | > loss: 0.012814145535230637 (nan) | > current_lr: 5e-06 | > step_time: 0.1709 (0.13656817599243126) | > loader_time: 0.0048 (0.004651026881655315)  --> TIME: 2025-02-24 11:34:41 -- STEP: 6596/36938 -- GLOBAL_STEP: 302100 | > loss_text_ce: 0.04345831274986267 (0.041445445085651365) | > loss_mel_ce: 3.0685362815856934 (nan) | > loss: 0.012349185533821583 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.1365967225565762) | > loader_time: 0.0047 (0.004651861164770825)  --> TIME: 2025-02-24 11:34:50 -- STEP: 6646/36938 -- GLOBAL_STEP: 302150 | > loss_text_ce: 0.03865464776754379 (0.041447639950051275) | > loss_mel_ce: 3.185137987136841 (nan) | > loss: 0.012792828492820263 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.1366124490352759) | > loader_time: 0.0045 (0.004652067189327103)  --> TIME: 2025-02-24 11:35:00 -- STEP: 6696/36938 -- GLOBAL_STEP: 302200 | > loss_text_ce: 0.04634150490164757 (0.04143933304783009) | > loss_mel_ce: 3.4325056076049805 (nan) | > loss: 0.013804948888719082 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1366420177789559) | > loader_time: 0.0046 (0.004652331557609989)  --> TIME: 2025-02-24 11:35:09 -- STEP: 6746/36938 -- GLOBAL_STEP: 302250 | > loss_text_ce: 0.04080270230770111 (0.041434902977087816) | > loss_mel_ce: 3.538628101348877 (nan) | > loss: 0.014204091392457485 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.13663378579659546) | > loader_time: 0.0048 (0.004652370729398353)  --> TIME: 2025-02-24 11:35:18 -- STEP: 6796/36938 -- GLOBAL_STEP: 302300 | > loss_text_ce: 0.035385631024837494 (0.04144584975510129) | > loss_mel_ce: 2.672342300415039 (nan) | > loss: 0.010744952596724033 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13658547868021267) | > loader_time: 0.0046 (0.004652527867519841)  --> TIME: 2025-02-24 11:35:27 -- STEP: 6846/36938 -- GLOBAL_STEP: 302350 | > loss_text_ce: 0.0438021644949913 (0.04144808326148873) | > loss_mel_ce: 3.017876148223877 (nan) | > loss: 0.012149518355727196 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.136576691462287) | > loader_time: 0.0048 (0.004653011746062481)  --> TIME: 2025-02-24 11:35:37 -- STEP: 6896/36938 -- GLOBAL_STEP: 302400 | > loss_text_ce: 0.038070231676101685 (0.04144903305221637) | > loss_mel_ce: 1.1529676914215088 (nan) | > loss: 0.0047263409942388535 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13660723575997957) | > loader_time: 0.0046 (0.0046529621923052855)  --> TIME: 2025-02-24 11:35:46 -- STEP: 6946/36938 -- GLOBAL_STEP: 302450 | > loss_text_ce: 0.03740020841360092 (0.041455402094634726) | > loss_mel_ce: 3.306483507156372 (nan) | > loss: 0.013269380666315556 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13656965110063643) | > loader_time: 0.0047 (0.0046525257244049755)  --> TIME: 2025-02-24 11:35:55 -- STEP: 6996/36938 -- GLOBAL_STEP: 302500 | > loss_text_ce: 0.051836322993040085 (0.041456233273010464) | > loss_mel_ce: 3.074331760406494 (nan) | > loss: 0.012405429035425186 (nan) | > current_lr: 5e-06 | > step_time: 0.163 (0.1365645680514783) | > loader_time: 0.0048 (0.004652292712066028)  --> TIME: 2025-02-24 11:36:04 -- STEP: 7046/36938 -- GLOBAL_STEP: 302550 | > loss_text_ce: 0.028166117146611214 (0.04145655922368032) | > loss_mel_ce: 1.7073423862457275 (nan) | > loss: 0.00688693905249238 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1365538068092436) | > loader_time: 0.0045 (0.004652429567694907)  --> TIME: 2025-02-24 11:36:14 -- STEP: 7096/36938 -- GLOBAL_STEP: 302600 | > loss_text_ce: 0.046612370759248734 (0.0414660537683403) | > loss_mel_ce: 3.9321341514587402 (nan) | > loss: 0.01578867807984352 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.1365776249158195) | > loader_time: 0.0046 (0.004652310452574141)  --> TIME: 2025-02-24 11:36:23 -- STEP: 7146/36938 -- GLOBAL_STEP: 302650 | > loss_text_ce: 0.03273528814315796 (0.041459439328217065) | > loss_mel_ce: 3.329136848449707 (nan) | > loss: 0.013340763747692108 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13656775513650735) | > loader_time: 0.0046 (0.004652114665845252)  --> TIME: 2025-02-24 11:36:33 -- STEP: 7196/36938 -- GLOBAL_STEP: 302700 | > loss_text_ce: 0.05275670439004898 (0.041468212646709086) | > loss_mel_ce: 3.031848907470703 (nan) | > loss: 0.012240499258041382 (nan) | > current_lr: 5e-06 | > step_time: 0.1606 (0.13656692215970795) | > loader_time: 0.0046 (0.004652027920791352)  --> TIME: 2025-02-24 11:36:42 -- STEP: 7246/36938 -- GLOBAL_STEP: 302750 | > loss_text_ce: 0.03985725715756416 (0.041458793813884454) | > loss_mel_ce: 2.914409637451172 (nan) | > loss: 0.011723281815648079 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13652862457487955) | > loader_time: 0.0046 (0.004652119722611141)  --> TIME: 2025-02-24 11:36:51 -- STEP: 7296/36938 -- GLOBAL_STEP: 302800 | > loss_text_ce: 0.03924598917365074 (0.04146253922839309) | > loss_mel_ce: 2.7907204627990723 (nan) | > loss: 0.011230026371777058 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13654012961737827) | > loader_time: 0.0045 (0.00465212125135096)  --> TIME: 2025-02-24 11:37:00 -- STEP: 7346/36938 -- GLOBAL_STEP: 302850 | > loss_text_ce: 0.039333898574113846 (0.04146442320205418) | > loss_mel_ce: 3.2597382068634033 (nan) | > loss: 0.0130915567278862 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13655348262895797) | > loader_time: 0.0046 (0.004652021140906851)  --> TIME: 2025-02-24 11:37:09 -- STEP: 7396/36938 -- GLOBAL_STEP: 302900 | > loss_text_ce: 0.038940805941820145 (0.04146294496117857) | > loss_mel_ce: 2.968529462814331 (nan) | > loss: 0.01193440705537796 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13652131214471822) | > loader_time: 0.0049 (0.004651916452586302)  --> TIME: 2025-02-24 11:37:19 -- STEP: 7446/36938 -- GLOBAL_STEP: 302950 | > loss_text_ce: 0.03305816650390625 (0.041457402855212794) | > loss_mel_ce: 1.6155142784118652 (nan) | > loss: 0.006541954353451729 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13650577189363144) | > loader_time: 0.0045 (0.0046518594386787075)  --> TIME: 2025-02-24 11:37:28 -- STEP: 7496/36938 -- GLOBAL_STEP: 303000 | > loss_text_ce: 0.048628516495227814 (0.04145893041507144) | > loss_mel_ce: 3.455789089202881 (nan) | > loss: 0.013906420208513737 (nan) | > current_lr: 5e-06 | > step_time: 0.1458 (0.13652107253145773) | > loader_time: 0.0047 (0.004651950479444349)  --> TIME: 2025-02-24 11:37:37 -- STEP: 7546/36938 -- GLOBAL_STEP: 303050 | > loss_text_ce: 0.03499330207705498 (0.04145327338770126) | > loss_mel_ce: 3.4720802307128906 (nan) | > loss: 0.01391695998609066 (nan) | > current_lr: 5e-06 | > step_time: 0.1985 (0.13647595704344206) | > loader_time: 0.0049 (0.004651779430853482)  --> TIME: 2025-02-24 11:37:46 -- STEP: 7596/36938 -- GLOBAL_STEP: 303100 | > loss_text_ce: 0.049736134707927704 (0.041470426536302896) | > loss_mel_ce: 2.9220314025878906 (nan) | > loss: 0.011792728677392006 (nan) | > current_lr: 5e-06 | > step_time: 0.1796 (0.1364987504174661) | > loader_time: 0.0048 (0.00465188361017751)  --> TIME: 2025-02-24 11:37:56 -- STEP: 7646/36938 -- GLOBAL_STEP: 303150 | > loss_text_ce: 0.04591919481754303 (0.041466546417638986) | > loss_mel_ce: 2.820960760116577 (nan) | > loss: 0.011376508511602879 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13650968618614842) | > loader_time: 0.0046 (0.004652165911318031)  --> TIME: 2025-02-24 11:38:05 -- STEP: 7696/36938 -- GLOBAL_STEP: 303200 | > loss_text_ce: 0.0392637737095356 (0.041462026447882466) | > loss_mel_ce: 3.5288031101226807 (nan) | > loss: 0.014158995822072029 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.13649065445589553) | > loader_time: 0.0047 (0.004652074090904114)  --> TIME: 2025-02-24 11:38:15 -- STEP: 7746/36938 -- GLOBAL_STEP: 303250 | > loss_text_ce: 0.04552985727787018 (0.04146555794478454) | > loss_mel_ce: 3.8987183570861816 (nan) | > loss: 0.015651779249310493 (nan) | > current_lr: 5e-06 | > step_time: 0.1996 (0.1365065586650018) | > loader_time: 0.0049 (0.00465392958677742)  --> TIME: 2025-02-24 11:38:24 -- STEP: 7796/36938 -- GLOBAL_STEP: 303300 | > loss_text_ce: 0.03207360580563545 (0.04145818675943226) | > loss_mel_ce: 3.492255926132202 (nan) | > loss: 0.013985435478389263 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.1364756269354635) | > loader_time: 0.0046 (0.0046543244094711675)  --> TIME: 2025-02-24 11:38:33 -- STEP: 7846/36938 -- GLOBAL_STEP: 303350 | > loss_text_ce: 0.03122291900217533 (0.041454945104787706) | > loss_mel_ce: 3.8546009063720703 (nan) | > loss: 0.01541993673890829 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1364616320323422) | > loader_time: 0.0047 (0.00465494584222853)  --> TIME: 2025-02-24 11:38:42 -- STEP: 7896/36938 -- GLOBAL_STEP: 303400 | > loss_text_ce: 0.0437033548951149 (0.041452092161966284) | > loss_mel_ce: 2.8656771183013916 (nan) | > loss: 0.011545160785317421 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.1364537726298884) | > loader_time: 0.0046 (0.004655360511250294)  --> TIME: 2025-02-24 11:38:51 -- STEP: 7946/36938 -- GLOBAL_STEP: 303450 | > loss_text_ce: 0.033991649746894836 (0.04144805938421756) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13644257358120762) | > loader_time: 0.0045 (0.0046558811296721824)  --> TIME: 2025-02-24 11:39:00 -- STEP: 7996/36938 -- GLOBAL_STEP: 303500 | > loss_text_ce: 0.04629060998558998 (0.04145184188956324) | > loss_mel_ce: 3.1678755283355713 (nan) | > loss: 0.01275462843477726 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13644023288065465) | > loader_time: 0.0046 (0.004655765407737825)  --> TIME: 2025-02-24 11:39:10 -- STEP: 8046/36938 -- GLOBAL_STEP: 303550 | > loss_text_ce: 0.031441859900951385 (0.04146473055065092) | > loss_mel_ce: 2.3349180221557617 (nan) | > loss: 0.00939031783491373 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.13642969443790248) | > loader_time: 0.0047 (0.0046559778458142315)  --> TIME: 2025-02-24 11:39:19 -- STEP: 8096/36938 -- GLOBAL_STEP: 303600 | > loss_text_ce: 0.04378104582428932 (0.04146991008266857) | > loss_mel_ce: 3.2572927474975586 (nan) | > loss: 0.013099499978125095 (nan) | > current_lr: 5e-06 | > step_time: 0.1522 (0.1364423428246144) | > loader_time: 0.0047 (0.004656054432919851)  --> TIME: 2025-02-24 11:39:28 -- STEP: 8146/36938 -- GLOBAL_STEP: 303650 | > loss_text_ce: 0.047417476773262024 (0.0414675038427721) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13642086234632306) | > loader_time: 0.0046 (0.004656210216116682)  --> TIME: 2025-02-24 11:39:37 -- STEP: 8196/36938 -- GLOBAL_STEP: 303700 | > loss_text_ce: 0.035381197929382324 (0.04146423754943622) | > loss_mel_ce: 3.3203125 (nan) | > loss: 0.013316245749592781 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13641358855993477) | > loader_time: 0.0045 (0.004656658340163676)  --> TIME: 2025-02-24 11:39:47 -- STEP: 8246/36938 -- GLOBAL_STEP: 303750 | > loss_text_ce: 0.042912185192108154 (0.04146530280301256) | > loss_mel_ce: 2.8544886112213135 (nan) | > loss: 0.011497623287141323 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1364379170852861) | > loader_time: 0.0046 (0.004657066218227898)  --> TIME: 2025-02-24 11:39:56 -- STEP: 8296/36938 -- GLOBAL_STEP: 303800 | > loss_text_ce: 0.04170411825180054 (0.04146346844204674) | > loss_mel_ce: 3.573845386505127 (nan) | > loss: 0.014347420074045658 (nan) | > current_lr: 5e-06 | > step_time: 0.1664 (0.13643448493059834) | > loader_time: 0.005 (0.00465788327969374)  --> TIME: 2025-02-24 11:40:06 -- STEP: 8346/36938 -- GLOBAL_STEP: 303850 | > loss_text_ce: 0.041451167315244675 (0.04146260368217488) | > loss_mel_ce: 0.2472686767578125 (nan) | > loss: 0.0011457137297838926 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13645855320062625) | > loader_time: 0.0048 (0.004658717718336931)  --> TIME: 2025-02-24 11:40:15 -- STEP: 8396/36938 -- GLOBAL_STEP: 303900 | > loss_text_ce: 0.042111124843358994 (0.04145776612791498) | > loss_mel_ce: 2.9692678451538086 (nan) | > loss: 0.011949917301535606 (nan) | > current_lr: 5e-06 | > step_time: 0.1689 (0.13642262120426246) | > loader_time: 0.0049 (0.004658840536332917)  --> TIME: 2025-02-24 11:40:24 -- STEP: 8446/36938 -- GLOBAL_STEP: 303950 | > loss_text_ce: 0.04089992865920067 (0.04146124092893316) | > loss_mel_ce: 2.746431827545166 (nan) | > loss: 0.011060841381549835 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.1364214528998623) | > loader_time: 0.0047 (0.004659013671387213)  --> TIME: 2025-02-24 11:40:33 -- STEP: 8496/36938 -- GLOBAL_STEP: 304000 | > loss_text_ce: 0.039545729756355286 (0.04146655170788907) | > loss_mel_ce: 3.4234931468963623 (nan) | > loss: 0.013742218725383282 (nan) | > current_lr: 5e-06 | > step_time: 0.1765 (0.13643141792431063) | > loader_time: 0.0049 (0.004659180559231965)  --> TIME: 2025-02-24 11:40:42 -- STEP: 8546/36938 -- GLOBAL_STEP: 304050 | > loss_text_ce: 0.026030007749795914 (0.041462947533302597) | > loss_mel_ce: 0.8617570400238037 (nan) | > loss: 0.003522964660078287 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13639159010635515) | > loader_time: 0.0045 (0.00465895921487092)  --> TIME: 2025-02-24 11:40:52 -- STEP: 8596/36938 -- GLOBAL_STEP: 304100 | > loss_text_ce: 0.034099481999874115 (0.04146093528740717) | > loss_mel_ce: 3.1456401348114014 (nan) | > loss: 0.012618015520274639 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1363948238289807) | > loader_time: 0.0046 (0.004660038184764611)  --> TIME: 2025-02-24 11:41:01 -- STEP: 8646/36938 -- GLOBAL_STEP: 304150 | > loss_text_ce: 0.04384159296751022 (0.04146341022986004) | > loss_mel_ce: 0.04559021070599556 (nan) | > loss: 0.00035488815046846867 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13638182671401458) | > loader_time: 0.0045 (0.004660207834228336)  --> TIME: 2025-02-24 11:41:10 -- STEP: 8696/36938 -- GLOBAL_STEP: 304200 | > loss_text_ce: 0.036878593266010284 (0.04146247209495707) | > loss_mel_ce: 1.549353003501892 (nan) | > loss: 0.006294569931924343 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13638198277000466) | > loader_time: 0.0045 (0.004660432642495403)  --> TIME: 2025-02-24 11:41:19 -- STEP: 8746/36938 -- GLOBAL_STEP: 304250 | > loss_text_ce: 0.05119723826646805 (0.04145931076629406) | > loss_mel_ce: 3.5949747562408447 (nan) | > loss: 0.014468937180936337 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.13635631704014134) | > loader_time: 0.0048 (0.004660348229241432)  --> TIME: 2025-02-24 11:41:28 -- STEP: 8796/36938 -- GLOBAL_STEP: 304300 | > loss_text_ce: 0.04510453715920448 (0.04144815928467869) | > loss_mel_ce: 3.430755853652954 (nan) | > loss: 0.0137930978089571 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.1363422034436208) | > loader_time: 0.0068 (0.004660673361574862)  --> TIME: 2025-02-24 11:41:38 -- STEP: 8846/36938 -- GLOBAL_STEP: 304350 | > loss_text_ce: 0.04731201007962227 (0.04144406427367972) | > loss_mel_ce: 2.8737361431121826 (nan) | > loss: 0.01159146148711443 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13634492532715775) | > loader_time: 0.0049 (0.004660626140191712)  --> TIME: 2025-02-24 11:41:47 -- STEP: 8896/36938 -- GLOBAL_STEP: 304400 | > loss_text_ce: 0.04291904345154762 (0.04144168779869109) | > loss_mel_ce: 2.318732261657715 (nan) | > loss: 0.00937163271009922 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13636539136763007) | > loader_time: 0.0046 (0.004660689975503539)  --> TIME: 2025-02-24 11:41:57 -- STEP: 8946/36938 -- GLOBAL_STEP: 304450 | > loss_text_ce: 0.040698979049921036 (0.0414360930014458) | > loss_mel_ce: 2.658707857131958 (nan) | > loss: 0.010711932554841042 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13638659041119144) | > loader_time: 0.0046 (0.0046605723777820495)  --> TIME: 2025-02-24 11:42:06 -- STEP: 8996/36938 -- GLOBAL_STEP: 304500 | > loss_text_ce: 0.05222605913877487 (0.041431482228160066) | > loss_mel_ce: 2.73193359375 (nan) | > loss: 0.011048253625631332 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13638590062231645) | > loader_time: 0.0047 (0.004660610598211772)  --> TIME: 2025-02-24 11:42:15 -- STEP: 9046/36938 -- GLOBAL_STEP: 304550 | > loss_text_ce: 0.03780805692076683 (0.04143183802870355) | > loss_mel_ce: 3.21709942817688 (nan) | > loss: 0.012916299514472485 (nan) | > current_lr: 5e-06 | > step_time: 0.1778 (0.13638580006201453) | > loader_time: 0.0048 (0.004660610680345047)  --> TIME: 2025-02-24 11:42:25 -- STEP: 9096/36938 -- GLOBAL_STEP: 304600 | > loss_text_ce: 0.036412496119737625 (0.04142338955431276) | > loss_mel_ce: 3.5748953819274902 (nan) | > loss: 0.014330587349832058 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.13636210296063747) | > loader_time: 0.0047 (0.004660545600121133)  --> TIME: 2025-02-24 11:42:34 -- STEP: 9146/36938 -- GLOBAL_STEP: 304650 | > loss_text_ce: 0.051717597991228104 (0.04142835061875376) | > loss_mel_ce: 3.3745832443237305 (nan) | > loss: 0.013596432283520699 (nan) | > current_lr: 5e-06 | > step_time: 0.1892 (0.1364145454890134) | > loader_time: 0.0048 (0.004660572130831806)  --> TIME: 2025-02-24 11:42:44 -- STEP: 9196/36938 -- GLOBAL_STEP: 304700 | > loss_text_ce: 0.035386040806770325 (0.041427640144608986) | > loss_mel_ce: 3.5442447662353516 (nan) | > loss: 0.014204884879291058 (nan) | > current_lr: 5e-06 | > step_time: 0.1445 (0.13642421482748968) | > loader_time: 0.0049 (0.004660525260774303)  --> TIME: 2025-02-24 11:42:53 -- STEP: 9246/36938 -- GLOBAL_STEP: 304750 | > loss_text_ce: 0.04393433779478073 (0.041429560601194626) | > loss_mel_ce: 0.0107421875 (nan) | > loss: 0.00021697035117540509 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13645211832362855) | > loader_time: 0.0046 (0.004660689982918215)  --> TIME: 2025-02-24 11:43:02 -- STEP: 9296/36938 -- GLOBAL_STEP: 304800 | > loss_text_ce: 0.041873663663864136 (0.04143002229109926) | > loss_mel_ce: 3.251448392868042 (nan) | > loss: 0.01306873932480812 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13641843525452277) | > loader_time: 0.0045 (0.004660422928156832)  --> TIME: 2025-02-24 11:43:11 -- STEP: 9346/36938 -- GLOBAL_STEP: 304850 | > loss_text_ce: 0.03886470943689346 (0.041434451187415615) | > loss_mel_ce: 2.7577672004699707 (nan) | > loss: 0.011097745969891548 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13639379869098106) | > loader_time: 0.0045 (0.004660453067814644)  --> TIME: 2025-02-24 11:43:21 -- STEP: 9396/36938 -- GLOBAL_STEP: 304900 | > loss_text_ce: 0.04082223400473595 (0.041438103721783456) | > loss_mel_ce: 3.380011796951294 (nan) | > loss: 0.01357473898679018 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.1364207110185723) | > loader_time: 0.0046 (0.004660426758359779)  --> TIME: 2025-02-24 11:43:30 -- STEP: 9446/36938 -- GLOBAL_STEP: 304950 | > loss_text_ce: 0.030843591317534447 (0.04144166740960537) | > loss_mel_ce: 3.2509560585021973 (nan) | > loss: 0.0130230151116848 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.13643186876840874) | > loader_time: 0.0047 (0.004660398531535564)  --> TIME: 2025-02-24 11:43:40 -- STEP: 9496/36938 -- GLOBAL_STEP: 305000 | > loss_text_ce: 0.05319318175315857 (0.0414410709085102) | > loss_mel_ce: 3.2428245544433594 (nan) | > loss: 0.01307943556457758 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1364421001380487) | > loader_time: 0.0046 (0.004660326990643376)  --> TIME: 2025-02-24 11:43:49 -- STEP: 9546/36938 -- GLOBAL_STEP: 305050 | > loss_text_ce: 0.03639267012476921 (0.04143944693547417) | > loss_mel_ce: 2.9450533390045166 (nan) | > loss: 0.011831135489046574 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.13643387643740865) | > loader_time: 0.0047 (0.004660370163564891)  --> TIME: 2025-02-24 11:43:59 -- STEP: 9596/36938 -- GLOBAL_STEP: 305100 | > loss_text_ce: 0.05365791171789169 (0.04144005405948739) | > loss_mel_ce: 3.226733922958374 (nan) | > loss: 0.013017429038882256 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13644617217041144) | > loader_time: 0.0045 (0.004660501734521708)  --> TIME: 2025-02-24 11:44:08 -- STEP: 9646/36938 -- GLOBAL_STEP: 305150 | > loss_text_ce: 0.037334419786930084 (0.041434368274185875) | > loss_mel_ce: 3.586966037750244 (nan) | > loss: 0.01438214536756277 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13645721764676136) | > loader_time: 0.0051 (0.004660504254319978)  --> TIME: 2025-02-24 11:44:17 -- STEP: 9696/36938 -- GLOBAL_STEP: 305200 | > loss_text_ce: 0.03359292075037956 (0.0414258539583494) | > loss_mel_ce: 3.4864540100097656 (nan) | > loss: 0.01396844070404768 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.13643084350887205) | > loader_time: 0.0046 (0.004660483511170938)  --> TIME: 2025-02-24 11:44:27 -- STEP: 9746/36938 -- GLOBAL_STEP: 305250 | > loss_text_ce: 0.05511830374598503 (0.041422001367024536) | > loss_mel_ce: 2.4185383319854736 (nan) | > loss: 0.009816098026931286 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1364448287514908) | > loader_time: 0.0046 (0.004660570814749956)  --> TIME: 2025-02-24 11:44:36 -- STEP: 9796/36938 -- GLOBAL_STEP: 305300 | > loss_text_ce: 0.03961827605962753 (0.04142094746951896) | > loss_mel_ce: 2.8365416526794434 (nan) | > loss: 0.01141333393752575 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13643041526896357) | > loader_time: 0.0045 (0.004660455997548047)  --> TIME: 2025-02-24 11:44:45 -- STEP: 9846/36938 -- GLOBAL_STEP: 305350 | > loss_text_ce: 0.05065292865037918 (0.04142273080597834) | > loss_mel_ce: 2.87890625 (nan) | > loss: 0.01162523590028286 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13645604803095684) | > loader_time: 0.0045 (0.004660455768437213)  --> TIME: 2025-02-24 11:44:55 -- STEP: 9896/36938 -- GLOBAL_STEP: 305400 | > loss_text_ce: 0.050932202488183975 (0.04141687604392768) | > loss_mel_ce: 3.3925654888153076 (nan) | > loss: 0.013664674013853073 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13646867695409715) | > loader_time: 0.0047 (0.00466063488560911)  --> TIME: 2025-02-24 11:45:04 -- STEP: 9946/36938 -- GLOBAL_STEP: 305450 | > loss_text_ce: 0.04293717443943024 (0.041406481064688826) | > loss_mel_ce: 3.607447862625122 (nan) | > loss: 0.014485656283795834 (nan) | > current_lr: 5e-06 | > step_time: 0.1807 (0.1364709118494622) | > loader_time: 0.0049 (0.004660770563731229)  --> TIME: 2025-02-24 11:45:13 -- STEP: 9996/36938 -- GLOBAL_STEP: 305500 | > loss_text_ce: 0.03694940358400345 (0.041401121420554556) | > loss_mel_ce: 2.801117181777954 (nan) | > loss: 0.011262170039117336 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13645249543165103) | > loader_time: 0.0046 (0.004660718915175331)  --> TIME: 2025-02-24 11:45:23 -- STEP: 10046/36938 -- GLOBAL_STEP: 305550 | > loss_text_ce: 0.04515387862920761 (0.04139837388958963) | > loss_mel_ce: 3.883843183517456 (nan) | > loss: 0.015591259114444256 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13647059000703624) | > loader_time: 0.0045 (0.0046607933741225375)  --> TIME: 2025-02-24 11:45:32 -- STEP: 10096/36938 -- GLOBAL_STEP: 305600 | > loss_text_ce: 0.035286057740449905 (0.04140294099489667) | > loss_mel_ce: 3.5868008136749268 (nan) | > loss: 0.014373361133038998 (nan) | > current_lr: 5e-06 | > step_time: 0.1816 (0.13648919314854432) | > loader_time: 0.0049 (0.004660876092933444)  --> TIME: 2025-02-24 11:45:42 -- STEP: 10146/36938 -- GLOBAL_STEP: 305650 | > loss_text_ce: 0.04227902367711067 (0.04140033023912875) | > loss_mel_ce: 2.6408348083496094 (nan) | > loss: 0.010647277347743511 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13651080747668595) | > loader_time: 0.0046 (0.004660914265237667)  --> TIME: 2025-02-24 11:45:51 -- STEP: 10196/36938 -- GLOBAL_STEP: 305700 | > loss_text_ce: 0.035028714686632156 (0.041398179939048756) | > loss_mel_ce: 3.255666494369507 (nan) | > loss: 0.01305831503123045 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13652610498206044) | > loader_time: 0.0046 (0.004661155172589722)  --> TIME: 2025-02-24 11:46:01 -- STEP: 10246/36938 -- GLOBAL_STEP: 305750 | > loss_text_ce: 0.046324580907821655 (0.04138871709269496) | > loss_mel_ce: 3.5855886936187744 (nan) | > loss: 0.014412354677915573 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13652239968807825) | > loader_time: 0.0046 (0.004661138998259103)  --> TIME: 2025-02-24 11:46:10 -- STEP: 10296/36938 -- GLOBAL_STEP: 305800 | > loss_text_ce: 0.042333316057920456 (0.04138969460535906) | > loss_mel_ce: 2.8742880821228027 (nan) | > loss: 0.011573894880712032 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.1365297779197731) | > loader_time: 0.0045 (0.004660931361583014)  --> TIME: 2025-02-24 11:46:20 -- STEP: 10346/36938 -- GLOBAL_STEP: 305850 | > loss_text_ce: 0.04282200336456299 (0.041392216475209756) | > loss_mel_ce: 3.6733384132385254 (nan) | > loss: 0.014746668748557568 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13653331371656108) | > loader_time: 0.0046 (0.004661002312134255)  --> TIME: 2025-02-24 11:46:28 -- STEP: 10396/36938 -- GLOBAL_STEP: 305900 | > loss_text_ce: 0.014350582845509052 (0.041379462882966615) | > loss_mel_ce: 0.044921875 (nan) | > loss: 0.0002352081792196259 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1365070082261583) | > loader_time: 0.0045 (0.004660867576004688)  --> TIME: 2025-02-24 11:46:38 -- STEP: 10446/36938 -- GLOBAL_STEP: 305950 | > loss_text_ce: 0.04086148738861084 (0.041375479795018655) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.13651016690642875) | > loader_time: 0.0047 (0.00466057287877533)  --> TIME: 2025-02-24 11:46:47 -- STEP: 10496/36938 -- GLOBAL_STEP: 306000 | > loss_text_ce: 0.03177293762564659 (0.041375936362849725) | > loss_mel_ce: 2.716007947921753 (nan) | > loss: 0.010903893038630486 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.1364962321410811) | > loader_time: 0.0046 (0.004660485993798196)  --> TIME: 2025-02-24 11:46:56 -- STEP: 10546/36938 -- GLOBAL_STEP: 306050 | > loss_text_ce: 0.05327412113547325 (0.04137383886942693) | > loss_mel_ce: 2.4442853927612305 (nan) | > loss: 0.009910951368510723 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13651339618480943) | > loader_time: 0.0047 (0.004660582126441649)  --> TIME: 2025-02-24 11:47:06 -- STEP: 10596/36938 -- GLOBAL_STEP: 306100 | > loss_text_ce: 0.03516661003232002 (0.04136749121623097) | > loss_mel_ce: 1.6523486375808716 (nan) | > loss: 0.00669648963958025 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13650676843219015) | > loader_time: 0.0046 (0.004660494420248518)  --> TIME: 2025-02-24 11:47:15 -- STEP: 10646/36938 -- GLOBAL_STEP: 306150 | > loss_text_ce: 0.03796304389834404 (0.041368157018648544) | > loss_mel_ce: 2.5459604263305664 (nan) | > loss: 0.01025366596877575 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13648307881094518) | > loader_time: 0.0047 (0.004660451387566298)  --> TIME: 2025-02-24 11:47:24 -- STEP: 10696/36938 -- GLOBAL_STEP: 306200 | > loss_text_ce: 0.04653893783688545 (0.041377730787924616) | > loss_mel_ce: 3.317927122116089 (nan) | > loss: 0.013351055793464184 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13647827497149115) | > loader_time: 0.0048 (0.004660709566738338)  --> TIME: 2025-02-24 11:47:33 -- STEP: 10746/36938 -- GLOBAL_STEP: 306250 | > loss_text_ce: 0.03852018713951111 (0.04138458120760916) | > loss_mel_ce: 3.1225616931915283 (nan) | > loss: 0.012543976306915283 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13645632316252745) | > loader_time: 0.0046 (0.00466050922859544)  --> TIME: 2025-02-24 11:47:43 -- STEP: 10796/36938 -- GLOBAL_STEP: 306300 | > loss_text_ce: 0.0437987856566906 (0.04137848226267946) | > loss_mel_ce: 3.1456804275512695 (nan) | > loss: 0.012656663544476032 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13646762719548267) | > loader_time: 0.0045 (0.004660333603035315)  --> TIME: 2025-02-24 11:47:52 -- STEP: 10846/36938 -- GLOBAL_STEP: 306350 | > loss_text_ce: 0.04745291545987129 (0.041374018667057956) | > loss_mel_ce: 1.7801164388656616 (nan) | > loss: 0.007252259645611048 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13646479449148102) | > loader_time: 0.0046 (0.004660413292925812)  --> TIME: 2025-02-24 11:48:01 -- STEP: 10896/36938 -- GLOBAL_STEP: 306400 | > loss_text_ce: 0.039047304540872574 (0.04137178884170231) | > loss_mel_ce: 2.985124111175537 (nan) | > loss: 0.012000680901110172 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13643973842591375) | > loader_time: 0.0045 (0.004660366565295438)  --> TIME: 2025-02-24 11:48:10 -- STEP: 10946/36938 -- GLOBAL_STEP: 306450 | > loss_text_ce: 0.03522409498691559 (0.04136792032710735) | > loss_mel_ce: 3.343501329421997 (nan) | > loss: 0.013407642021775246 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13642248021778644) | > loader_time: 0.0048 (0.004660148061260596)  --> TIME: 2025-02-24 11:48:19 -- STEP: 10996/36938 -- GLOBAL_STEP: 306500 | > loss_text_ce: 0.047851160168647766 (0.04135795802531154) | > loss_mel_ce: 3.6905031204223633 (nan) | > loss: 0.014834740199148655 (nan) | > current_lr: 5e-06 | > step_time: 0.163 (0.13639208946543663) | > loader_time: 0.0049 (0.004659964653221913)  --> TIME: 2025-02-24 11:48:28 -- STEP: 11046/36938 -- GLOBAL_STEP: 306550 | > loss_text_ce: 0.052753746509552 (0.04135606424987894) | > loss_mel_ce: 2.206392765045166 (nan) | > loss: 0.00896486733108759 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13639370196758735) | > loader_time: 0.0045 (0.004659801079444273)  --> TIME: 2025-02-24 11:48:38 -- STEP: 11096/36938 -- GLOBAL_STEP: 306600 | > loss_text_ce: 0.04385817050933838 (0.04134925348209465) | > loss_mel_ce: 2.753690242767334 (nan) | > loss: 0.011101382784545422 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13640241018713276) | > loader_time: 0.0045 (0.004659779590071767)  --> TIME: 2025-02-24 11:48:47 -- STEP: 11146/36938 -- GLOBAL_STEP: 306650 | > loss_text_ce: 0.05262044817209244 (0.0413443238856951) | > loss_mel_ce: 3.294316530227661 (nan) | > loss: 0.013281496241688728 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13640858094738792) | > loader_time: 0.0045 (0.0046596135867256055)  --> TIME: 2025-02-24 11:48:56 -- STEP: 11196/36938 -- GLOBAL_STEP: 306700 | > loss_text_ce: 0.03977231681346893 (0.04135096099170438) | > loss_mel_ce: 3.4290921688079834 (nan) | > loss: 0.013765336014330387 (nan) | > current_lr: 5e-06 | > step_time: 0.1567 (0.13638972825090842) | > loader_time: 0.0046 (0.00465926790969975)  --> TIME: 2025-02-24 11:49:05 -- STEP: 11246/36938 -- GLOBAL_STEP: 306750 | > loss_text_ce: 0.03692184388637543 (0.04134966269797655) | > loss_mel_ce: 2.5711822509765625 (nan) | > loss: 0.010349619202315807 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13637802625452877) | > loader_time: 0.0046 (0.0046589436659116796)  --> TIME: 2025-02-24 11:49:14 -- STEP: 11296/36938 -- GLOBAL_STEP: 306800 | > loss_text_ce: 0.042350586503744125 (0.041343521643459676) | > loss_mel_ce: 3.4846231937408447 (nan) | > loss: 0.013995928689837456 (nan) | > current_lr: 5e-06 | > step_time: 0.1852 (0.13638067061857112) | > loader_time: 0.005 (0.0046589588561747036)  --> TIME: 2025-02-24 11:49:24 -- STEP: 11346/36938 -- GLOBAL_STEP: 306850 | > loss_text_ce: 0.04755721986293793 (0.04133550589296769) | > loss_mel_ce: 0.194488525390625 (nan) | > loss: 0.0009604990482330322 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13636973560665355) | > loader_time: 0.0046 (0.004658835244813384)  --> TIME: 2025-02-24 11:49:33 -- STEP: 11396/36938 -- GLOBAL_STEP: 306900 | > loss_text_ce: 0.02618064545094967 (0.04134432040506345) | > loss_mel_ce: 2.4985363483428955 (nan) | > loss: 0.010018719360232353 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13636559733343528) | > loader_time: 0.0045 (0.00465854955330444)  --> TIME: 2025-02-24 11:49:42 -- STEP: 11446/36938 -- GLOBAL_STEP: 306950 | > loss_text_ce: 0.04170594736933708 (0.041337619587130066) | > loss_mel_ce: 2.9223620891571045 (nan) | > loss: 0.011762174777686596 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13634265627286873) | > loader_time: 0.0048 (0.004658415332949492)  --> TIME: 2025-02-24 11:49:51 -- STEP: 11496/36938 -- GLOBAL_STEP: 307000 | > loss_text_ce: 0.03898964449763298 (0.04133790639002602) | > loss_mel_ce: 0.2262369841337204 (nan) | > loss: 0.0010524867102503777 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13633933980678892) | > loader_time: 0.0046 (0.004658353083981509)  --> TIME: 2025-02-24 11:50:01 -- STEP: 11546/36938 -- GLOBAL_STEP: 307050 | > loss_text_ce: 0.036409489810466766 (0.041340166441763845) | > loss_mel_ce: 2.6792991161346436 (nan) | > loss: 0.010776621289551258 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.1363456156934) | > loader_time: 0.0046 (0.004658511972253901)  --> TIME: 2025-02-24 11:50:10 -- STEP: 11596/36938 -- GLOBAL_STEP: 307100 | > loss_text_ce: 0.03063374198973179 (0.04134604948519482) | > loss_mel_ce: 1.8385553359985352 (nan) | > loss: 0.0074174171313643456 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13633097959576257) | > loader_time: 0.0046 (0.004658781585713086)  --> TIME: 2025-02-24 11:50:19 -- STEP: 11646/36938 -- GLOBAL_STEP: 307150 | > loss_text_ce: 0.03877231851220131 (0.041342745814888227) | > loss_mel_ce: 2.5016677379608154 (nan) | > loss: 0.010081112384796143 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13634128601674225) | > loader_time: 0.0045 (0.004659601877629491)  --> TIME: 2025-02-24 11:50:28 -- STEP: 11696/36938 -- GLOBAL_STEP: 307200 | > loss_text_ce: 0.05050278082489967 (0.04134325576010036) | > loss_mel_ce: 0.07708740234375 (nan) | > loss: 0.0005063102580606937 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13632654548506654) | > loader_time: 0.0046 (0.004659534499518023)  --> TIME: 2025-02-24 11:50:38 -- STEP: 11746/36938 -- GLOBAL_STEP: 307250 | > loss_text_ce: 0.03861842676997185 (0.041336609467451046) | > loss_mel_ce: 2.6042399406433105 (nan) | > loss: 0.010487533174455166 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1363348698222242) | > loader_time: 0.0045 (0.004659518784728085)  --> TIME: 2025-02-24 11:50:47 -- STEP: 11796/36938 -- GLOBAL_STEP: 307300 | > loss_text_ce: 0.049222178757190704 (0.04133605478761257) | > loss_mel_ce: 3.5008554458618164 (nan) | > loss: 0.014087610878050327 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13632471109091363) | > loader_time: 0.0045 (0.004659528831741921)  --> TIME: 2025-02-24 11:50:56 -- STEP: 11846/36938 -- GLOBAL_STEP: 307350 | > loss_text_ce: 0.04034892097115517 (0.04133420723410059) | > loss_mel_ce: 3.5939137935638428 (nan) | > loss: 0.014421678148210049 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.1363003684241437) | > loader_time: 0.0046 (0.004659486948065133)  --> TIME: 2025-02-24 11:51:05 -- STEP: 11896/36938 -- GLOBAL_STEP: 307400 | > loss_text_ce: 0.041874997317790985 (0.04133076660756872) | > loss_mel_ce: 3.6960721015930176 (nan) | > loss: 0.014833123423159122 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13628107287295244) | > loader_time: 0.0048 (0.004659316065973601)  --> TIME: 2025-02-24 11:51:14 -- STEP: 11946/36938 -- GLOBAL_STEP: 307450 | > loss_text_ce: 0.03968590497970581 (0.041335505582364054) | > loss_mel_ce: 2.748852491378784 (nan) | > loss: 0.011065629310905933 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13626710390218272) | > loader_time: 0.0045 (0.004658984096130476)  --> TIME: 2025-02-24 11:51:23 -- STEP: 11996/36938 -- GLOBAL_STEP: 307500 | > loss_text_ce: 0.04448484256863594 (0.041335158710820996) | > loss_mel_ce: 3.2352664470672607 (nan) | > loss: 0.013014886528253555 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13626948117255766) | > loader_time: 0.0048 (0.004658859028105487)  --> TIME: 2025-02-24 11:51:32 -- STEP: 12046/36938 -- GLOBAL_STEP: 307550 | > loss_text_ce: 0.04669465869665146 (0.04133266570877197) | > loss_mel_ce: 2.7220523357391357 (nan) | > loss: 0.010987091809511185 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13626325219528126) | > loader_time: 0.0046 (0.00465916385884975)  --> TIME: 2025-02-24 11:51:42 -- STEP: 12096/36938 -- GLOBAL_STEP: 307600 | > loss_text_ce: 0.041473813354969025 (0.04133436108485694) | > loss_mel_ce: 3.077134847640991 (nan) | > loss: 0.012375432066619396 (nan) | > current_lr: 5e-06 | > step_time: 0.1444 (0.13626647632194502) | > loader_time: 0.0048 (0.004659299300145839)  --> TIME: 2025-02-24 11:51:51 -- STEP: 12146/36938 -- GLOBAL_STEP: 307650 | > loss_text_ce: 0.03501462563872337 (0.04133730145476759) | > loss_mel_ce: 3.2729084491729736 (nan) | > loss: 0.013126679696142673 (nan) | > current_lr: 5e-06 | > step_time: 0.1818 (0.13624622628261993) | > loader_time: 0.0048 (0.004659252270391221)  --> TIME: 2025-02-24 11:52:00 -- STEP: 12196/36938 -- GLOBAL_STEP: 307700 | > loss_text_ce: 0.042418573051691055 (0.04134038731169588) | > loss_mel_ce: 3.322380542755127 (nan) | > loss: 0.013352378271520138 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13624719005211186) | > loader_time: 0.0046 (0.004659232036838287)  --> TIME: 2025-02-24 11:52:09 -- STEP: 12246/36938 -- GLOBAL_STEP: 307750 | > loss_text_ce: 0.04115895554423332 (0.04133755180818327) | > loss_mel_ce: 3.8299560546875 (nan) | > loss: 0.015361567959189415 (nan) | > current_lr: 5e-06 | > step_time: 0.1523 (0.13623574273406694) | > loader_time: 0.0046 (0.0046591950888028475)  --> TIME: 2025-02-24 11:52:19 -- STEP: 12296/36938 -- GLOBAL_STEP: 307800 | > loss_text_ce: 0.02915610931813717 (0.04133562338588446) | > loss_mel_ce: 2.918825149536133 (nan) | > loss: 0.011698339134454727 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13623862818765656) | > loader_time: 0.0046 (0.004659117819352344)  --> TIME: 2025-02-24 11:52:28 -- STEP: 12346/36938 -- GLOBAL_STEP: 307850 | > loss_text_ce: 0.03391272574663162 (0.04133067046710477) | > loss_mel_ce: 0.058099109679460526 (nan) | > loss: 0.00036512635415419936 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13623299313657455) | > loader_time: 0.0048 (0.004659136187869943)  --> TIME: 2025-02-24 11:52:37 -- STEP: 12396/36938 -- GLOBAL_STEP: 307900 | > loss_text_ce: 0.0530911423265934 (0.04132644511768587) | > loss_mel_ce: 2.126574754714966 (nan) | > loss: 0.008649468421936035 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.1362166904603014) | > loader_time: 0.0045 (0.00465900275198709)  --> TIME: 2025-02-24 11:52:46 -- STEP: 12446/36938 -- GLOBAL_STEP: 307950 | > loss_text_ce: 0.0347784049808979 (0.041327887674095794) | > loss_mel_ce: 2.9743361473083496 (nan) | > loss: 0.011940930970013142 (nan) | > current_lr: 5e-06 | > step_time: 0.1964 (0.13619641150076464) | > loader_time: 0.0049 (0.004658781292544613)  --> TIME: 2025-02-24 11:52:55 -- STEP: 12496/36938 -- GLOBAL_STEP: 308000 | > loss_text_ce: 0.038650281727313995 (0.041327634063044225) | > loss_mel_ce: 3.607783317565918 (nan) | > loss: 0.014469975605607033 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.1362109245319821) | > loader_time: 0.0048 (0.004658782222664751)  --> TIME: 2025-02-24 11:53:05 -- STEP: 12546/36938 -- GLOBAL_STEP: 308050 | > loss_text_ce: 0.03097463957965374 (0.04133185836905626) | > loss_mel_ce: 2.387251138687134 (nan) | > loss: 0.009596134535968304 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1362211938757989) | > loader_time: 0.0045 (0.004658946100840345)  --> TIME: 2025-02-24 11:53:14 -- STEP: 12596/36938 -- GLOBAL_STEP: 308100 | > loss_text_ce: 0.048135556280612946 (0.041330039631986344) | > loss_mel_ce: 3.4476137161254883 (nan) | > loss: 0.013872021809220314 (nan) | > current_lr: 5e-06 | > step_time: 0.1716 (0.1362138387512568) | > loader_time: 0.0048 (0.004658842170075698)  --> TIME: 2025-02-24 11:53:23 -- STEP: 12646/36938 -- GLOBAL_STEP: 308150 | > loss_text_ce: 0.04849196970462799 (0.04133246973353995) | > loss_mel_ce: 3.539782762527466 (nan) | > loss: 0.01423918642103672 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13621599178760582) | > loader_time: 0.0046 (0.004658860966466591)  --> TIME: 2025-02-24 11:53:32 -- STEP: 12696/36938 -- GLOBAL_STEP: 308200 | > loss_text_ce: 0.04643087834119797 (0.041338122493608465) | > loss_mel_ce: 3.5781147480010986 (nan) | > loss: 0.014383118599653244 (nan) | > current_lr: 5e-06 | > step_time: 0.1357 (0.136197362257086) | > loader_time: 0.0046 (0.004658810357740999)  --> TIME: 2025-02-24 11:53:41 -- STEP: 12746/36938 -- GLOBAL_STEP: 308250 | > loss_text_ce: 0.057609666138887405 (0.04134107811306882) | > loss_mel_ce: 0.08061523735523224 (nan) | > loss: 0.0005485115107148886 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13617840092690509) | > loader_time: 0.0046 (0.004658880477682591)  --> TIME: 2025-02-24 11:53:51 -- STEP: 12796/36938 -- GLOBAL_STEP: 308300 | > loss_text_ce: 0.03763587027788162 (0.041342561937086526) | > loss_mel_ce: 3.279991626739502 (nan) | > loss: 0.013165188953280449 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13621753316329457) | > loader_time: 0.0048 (0.0046590493223971794)  --> TIME: 2025-02-24 11:54:01 -- STEP: 12846/36938 -- GLOBAL_STEP: 308350 | > loss_text_ce: 0.04149327799677849 (0.04134205774320435) | > loss_mel_ce: 2.1130847930908203 (nan) | > loss: 0.008549912832677364 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13622140503924407) | > loader_time: 0.0049 (0.004659047532114992)  --> TIME: 2025-02-24 11:54:10 -- STEP: 12896/36938 -- GLOBAL_STEP: 308400 | > loss_text_ce: 0.0392305850982666 (0.04134563062732588) | > loss_mel_ce: 3.230832576751709 (nan) | > loss: 0.012976441532373428 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13622296340855913) | > loader_time: 0.0047 (0.004659337844357581)  --> TIME: 2025-02-24 11:54:19 -- STEP: 12946/36938 -- GLOBAL_STEP: 308450 | > loss_text_ce: 0.04635179415345192 (0.04134774016588744) | > loss_mel_ce: 2.9691851139068604 (nan) | > loss: 0.011966417543590069 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13621249151767528) | > loader_time: 0.0045 (0.004659653262452608)  --> TIME: 2025-02-24 11:54:28 -- STEP: 12996/36938 -- GLOBAL_STEP: 308500 | > loss_text_ce: 0.04029424116015434 (0.04134592955160066) | > loss_mel_ce: 3.694735288619995 (nan) | > loss: 0.014821546152234077 (nan) | > current_lr: 5e-06 | > step_time: 0.2009 (0.13620123534101217) | > loader_time: 0.0048 (0.0046596092860784364)  --> TIME: 2025-02-24 11:54:38 -- STEP: 13046/36938 -- GLOBAL_STEP: 308550 | > loss_text_ce: 0.04637209326028824 (0.04134842624534303) | > loss_mel_ce: 2.9801385402679443 (nan) | > loss: 0.012009963393211365 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.1362092212977875) | > loader_time: 0.0044 (0.0046597602231067715)  --> TIME: 2025-02-24 11:54:47 -- STEP: 13096/36938 -- GLOBAL_STEP: 308600 | > loss_text_ce: 0.037036288529634476 (0.04133875399724689) | > loss_mel_ce: 3.0721328258514404 (nan) | > loss: 0.012337973341345787 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13621193643889087) | > loader_time: 0.0047 (0.004660034405435063)  --> TIME: 2025-02-24 11:54:56 -- STEP: 13146/36938 -- GLOBAL_STEP: 308650 | > loss_text_ce: 0.031599145382642746 (0.04133888726706002) | > loss_mel_ce: 0.006008148193359375 (nan) | > loss: 0.00014923530397936702 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13621123149372033) | > loader_time: 0.0046 (0.004660296200726556)  --> TIME: 2025-02-24 11:55:06 -- STEP: 13196/36938 -- GLOBAL_STEP: 308700 | > loss_text_ce: 0.03830176219344139 (0.04133646542193129) | > loss_mel_ce: 3.762991428375244 (nan) | > loss: 0.015084497630596161 (nan) | > current_lr: 5e-06 | > step_time: 0.1563 (0.1362170092298526) | > loader_time: 0.0046 (0.004660471131057928)  --> TIME: 2025-02-24 11:55:15 -- STEP: 13246/36938 -- GLOBAL_STEP: 308750 | > loss_text_ce: 0.03947041556239128 (0.04133281710299031) | > loss_mel_ce: 0.030029296875 (nan) | > loss: 0.00027579255402088165 (nan) | > current_lr: 5e-06 | > step_time: 0.1075 (0.13620226874229716) | > loader_time: 0.0047 (0.004660490774845528)  --> TIME: 2025-02-24 11:55:24 -- STEP: 13296/36938 -- GLOBAL_STEP: 308800 | > loss_text_ce: 0.04623815789818764 (0.04133155926181976) | > loss_mel_ce: 2.5097603797912598 (nan) | > loss: 0.010142851620912552 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13620724476129595) | > loader_time: 0.0046 (0.004660499099503048)  --> TIME: 2025-02-24 11:55:34 -- STEP: 13346/36938 -- GLOBAL_STEP: 308850 | > loss_text_ce: 0.03705428168177605 (0.04132917306212724) | > loss_mel_ce: 2.8048930168151855 (nan) | > loss: 0.011277569457888603 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13621636178742413) | > loader_time: 0.0047 (0.004660856432595299)  --> TIME: 2025-02-24 11:55:43 -- STEP: 13396/36938 -- GLOBAL_STEP: 308900 | > loss_text_ce: 0.04547226428985596 (0.04132692308944331) | > loss_mel_ce: 1.809814453125 (nan) | > loss: 0.007362249307334423 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13620750040536606) | > loader_time: 0.0045 (0.004660909995992828)  --> TIME: 2025-02-24 11:55:53 -- STEP: 13446/36938 -- GLOBAL_STEP: 308950 | > loss_text_ce: 0.036233510822057724 (0.041324768841217226) | > loss_mel_ce: 2.999826192855835 (nan) | > loss: 0.012047856114804745 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.13622480165421041) | > loader_time: 0.0046 (0.004660900391302465)  --> TIME: 2025-02-24 11:56:02 -- STEP: 13496/36938 -- GLOBAL_STEP: 309000 | > loss_text_ce: 0.02807454764842987 (0.041317384911076616) | > loss_mel_ce: 3.2631430625915527 (nan) | > loss: 0.013060388155281544 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1362454215308645) | > loader_time: 0.0046 (0.004661028298246249)  --> TIME: 2025-02-24 11:56:11 -- STEP: 13546/36938 -- GLOBAL_STEP: 309050 | > loss_text_ce: 0.03322243317961693 (0.041317488617904005) | > loss_mel_ce: 2.2469654083251953 (nan) | > loss: 0.00904836505651474 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.1362415578364261) | > loader_time: 0.0046 (0.004660813156864598)  --> TIME: 2025-02-24 11:56:21 -- STEP: 13596/36938 -- GLOBAL_STEP: 309100 | > loss_text_ce: 0.040317825973033905 (0.04131602821960679) | > loss_mel_ce: 2.0561492443084717 (nan) | > loss: 0.008319313637912273 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.1362448781509537) | > loader_time: 0.0046 (0.004660665550523705)  --> TIME: 2025-02-24 11:56:30 -- STEP: 13646/36938 -- GLOBAL_STEP: 309150 | > loss_text_ce: 0.03801094368100166 (0.041317369348066284) | > loss_mel_ce: 3.2681190967559814 (nan) | > loss: 0.013119564391672611 (nan) | > current_lr: 5e-06 | > step_time: 0.1705 (0.13625182727032026) | > loader_time: 0.0047 (0.004660551173760683)  --> TIME: 2025-02-24 11:56:40 -- STEP: 13696/36938 -- GLOBAL_STEP: 309200 | > loss_text_ce: 0.039369385689496994 (0.04131471587144532) | > loss_mel_ce: 3.009594678878784 (nan) | > loss: 0.012099063955247402 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13626805155007002) | > loader_time: 0.0048 (0.004660710849076758)  --> TIME: 2025-02-24 11:56:49 -- STEP: 13746/36938 -- GLOBAL_STEP: 309250 | > loss_text_ce: 0.040681205689907074 (0.04131536938787467) | > loss_mel_ce: 2.437849998474121 (nan) | > loss: 0.00983544159680605 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.1362456789742414) | > loader_time: 0.0046 (0.004660805500041032)  --> TIME: 2025-02-24 11:56:58 -- STEP: 13796/36938 -- GLOBAL_STEP: 309300 | > loss_text_ce: 0.04137792810797691 (0.04131745069641984) | > loss_mel_ce: 3.3879778385162354 (nan) | > loss: 0.013608556240797043 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.1362553217785011) | > loader_time: 0.0047 (0.0046607146888029164)  --> TIME: 2025-02-24 11:57:07 -- STEP: 13846/36938 -- GLOBAL_STEP: 309350 | > loss_text_ce: 0.045899443328380585 (0.041312698164472285) | > loss_mel_ce: 3.367173671722412 (nan) | > loss: 0.013543941080570221 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13625927800028723) | > loader_time: 0.0045 (0.004660569862118836)  --> TIME: 2025-02-24 11:57:17 -- STEP: 13896/36938 -- GLOBAL_STEP: 309400 | > loss_text_ce: 0.05073060467839241 (0.041314207685821176) | > loss_mel_ce: 3.027195930480957 (nan) | > loss: 0.012213995680212975 (nan) | > current_lr: 5e-06 | > step_time: 0.1871 (0.13625656733172412) | > loader_time: 0.0049 (0.0046605982002198695)  --> TIME: 2025-02-24 11:57:26 -- STEP: 13946/36938 -- GLOBAL_STEP: 309450 | > loss_text_ce: 0.04686622694134712 (0.04131363048901347) | > loss_mel_ce: 3.855287551879883 (nan) | > loss: 0.015484738163650036 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.1362654301009998) | > loader_time: 0.0047 (0.0046606338230998755)  --> TIME: 2025-02-24 11:57:35 -- STEP: 13996/36938 -- GLOBAL_STEP: 309500 | > loss_text_ce: 0.040263861417770386 (0.041305678500747206) | > loss_mel_ce: 1.258172631263733 (nan) | > loss: 0.005152526311576366 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13625336149891015) | > loader_time: 0.0046 (0.004660637183133488)  --> TIME: 2025-02-24 11:57:44 -- STEP: 14046/36938 -- GLOBAL_STEP: 309550 | > loss_text_ce: 0.03592206537723541 (0.04131187663291441) | > loss_mel_ce: 3.4494965076446533 (nan) | > loss: 0.01383102685213089 (nan) | > current_lr: 5e-06 | > step_time: 0.1887 (0.13624650802636665) | > loader_time: 0.0048 (0.004660600375436576)  --> TIME: 2025-02-24 11:57:54 -- STEP: 14096/36938 -- GLOBAL_STEP: 309600 | > loss_text_ce: 0.04511067643761635 (0.041310758513932753) | > loss_mel_ce: 2.426561117172241 (nan) | > loss: 0.009808221831917763 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13623743432401572) | > loader_time: 0.0046 (0.004660520732199804)  --> TIME: 2025-02-24 11:58:03 -- STEP: 14146/36938 -- GLOBAL_STEP: 309650 | > loss_text_ce: 0.035137150436639786 (0.04131285913221958) | > loss_mel_ce: 2.114980936050415 (nan) | > loss: 0.008532214909791946 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13624270517148787) | > loader_time: 0.0048 (0.0046604959391374256)  --> TIME: 2025-02-24 11:58:12 -- STEP: 14196/36938 -- GLOBAL_STEP: 309700 | > loss_text_ce: 0.036159247159957886 (0.04131097906197222) | > loss_mel_ce: 2.952794313430786 (nan) | > loss: 0.011860927566885948 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13623761465664258) | > loader_time: 0.0046 (0.004660380141907697)  --> TIME: 2025-02-24 11:58:21 -- STEP: 14246/36938 -- GLOBAL_STEP: 309750 | > loss_text_ce: 0.0424891896545887 (0.04131552575953128) | > loss_mel_ce: 3.3064777851104736 (nan) | > loss: 0.013289553113281727 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13622810372672897) | > loader_time: 0.0049 (0.004660320352272862)  --> TIME: 2025-02-24 11:58:30 -- STEP: 14296/36938 -- GLOBAL_STEP: 309800 | > loss_text_ce: 0.04266483336687088 (0.04131602633828638) | > loss_mel_ce: 3.2166054248809814 (nan) | > loss: 0.012933612801134586 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.13621669672492015) | > loader_time: 0.0047 (0.004660398935431102)  --> TIME: 2025-02-24 11:58:40 -- STEP: 14346/36938 -- GLOBAL_STEP: 309850 | > loss_text_ce: 0.042158909142017365 (0.04131848256349208) | > loss_mel_ce: 3.4718081951141357 (nan) | > loss: 0.013944314792752266 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.1362145881049215) | > loader_time: 0.0046 (0.004660365190297215)  --> TIME: 2025-02-24 11:58:49 -- STEP: 14396/36938 -- GLOBAL_STEP: 309900 | > loss_text_ce: 0.05944151431322098 (0.041321306494518685) | > loss_mel_ce: 2.844449281692505 (nan) | > loss: 0.011523377150297165 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13620823303770016) | > loader_time: 0.0046 (0.004660500109477224)  --> TIME: 2025-02-24 11:58:58 -- STEP: 14446/36938 -- GLOBAL_STEP: 309950 | > loss_text_ce: 0.04220488294959068 (0.04132179157727441) | > loss_mel_ce: 3.0676350593566895 (nan) | > loss: 0.012340635061264038 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.1362095087762511) | > loader_time: 0.0046 (0.004660356313786483)  --> TIME: 2025-02-24 11:59:08 -- STEP: 14496/36938 -- GLOBAL_STEP: 310000 | > loss_text_ce: 0.04339054599404335 (0.04132614313684155) | > loss_mel_ce: 3.7813079357147217 (nan) | > loss: 0.015177375636994839 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13622433795834113) | > loader_time: 0.0045 (0.004660361534843878) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_310000.pth  --> TIME: 2025-02-24 11:59:23 -- STEP: 14546/36938 -- GLOBAL_STEP: 310050 | > loss_text_ce: 0.038995161652565 (0.04131879356642235) | > loss_mel_ce: 3.1211562156677246 (nan) | > loss: 0.012540284544229507 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13622780387410993) | > loader_time: 0.0047 (0.004660679847227372)  --> TIME: 2025-02-24 11:59:33 -- STEP: 14596/36938 -- GLOBAL_STEP: 310100 | > loss_text_ce: 0.052221521735191345 (0.041316427308496814) | > loss_mel_ce: 4.21484375 (nan) | > loss: 0.01693279854953289 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13623002892685146) | > loader_time: 0.0048 (0.004660726001929737)  --> TIME: 2025-02-24 11:59:42 -- STEP: 14646/36938 -- GLOBAL_STEP: 310150 | > loss_text_ce: 0.03583122417330742 (0.04131995120629961) | > loss_mel_ce: 1.3356527090072632 (nan) | > loss: 0.005442396737635136 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.1362483877961814) | > loader_time: 0.0048 (0.0046607033567937495)  --> TIME: 2025-02-24 11:59:51 -- STEP: 14696/36938 -- GLOBAL_STEP: 310200 | > loss_text_ce: 0.039590731263160706 (0.04131320807985217) | > loss_mel_ce: 3.2664144039154053 (nan) | > loss: 0.013119068928062916 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13624214021226308) | > loader_time: 0.0046 (0.004660620563498745)  --> TIME: 2025-02-24 12:00:01 -- STEP: 14746/36938 -- GLOBAL_STEP: 310250 | > loss_text_ce: 0.023530272766947746 (0.04131592473706883) | > loss_mel_ce: 2.0687150955200195 (nan) | > loss: 0.00830256100744009 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13624155333839785) | > loader_time: 0.0045 (0.004660636037042408)  --> TIME: 2025-02-24 12:00:10 -- STEP: 14796/36938 -- GLOBAL_STEP: 310300 | > loss_text_ce: 0.040481649339199066 (0.04132131093131809) | > loss_mel_ce: 2.5449378490448 (nan) | > loss: 0.010259601287543774 (nan) | > current_lr: 5e-06 | > step_time: 0.19 (0.1362407682682562) | > loader_time: 0.0049 (0.004660597280026356)  --> TIME: 2025-02-24 12:00:20 -- STEP: 14846/36938 -- GLOBAL_STEP: 310350 | > loss_text_ce: 0.043974101543426514 (0.04132200647932163) | > loss_mel_ce: 3.5084574222564697 (nan) | > loss: 0.014096951112151146 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13625551885661016) | > loader_time: 0.0046 (0.004660590758433487)  --> TIME: 2025-02-24 12:00:29 -- STEP: 14896/36938 -- GLOBAL_STEP: 310400 | > loss_text_ce: 0.056517429649829865 (0.041324252918250995) | > loss_mel_ce: 3.2009525299072266 (nan) | > loss: 0.012926468625664711 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13624564421420088) | > loader_time: 0.0046 (0.004660778988060682)  --> TIME: 2025-02-24 12:00:38 -- STEP: 14946/36938 -- GLOBAL_STEP: 310450 | > loss_text_ce: 0.044489357620477676 (0.0413284537199972) | > loss_mel_ce: 3.2109930515289307 (nan) | > loss: 0.012918582186102867 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13625941708211273) | > loader_time: 0.0046 (0.004660672951969714)  --> TIME: 2025-02-24 12:00:48 -- STEP: 14996/36938 -- GLOBAL_STEP: 310500 | > loss_text_ce: 0.03626236692070961 (0.041322399426942664) | > loss_mel_ce: 2.354649543762207 (nan) | > loss: 0.00948774628341198 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13626372925470034) | > loader_time: 0.0046 (0.004660490640928699)  --> TIME: 2025-02-24 12:00:57 -- STEP: 15046/36938 -- GLOBAL_STEP: 310550 | > loss_text_ce: 0.0423615463078022 (0.04132222246112984) | > loss_mel_ce: 4.3203125 (nan) | > loss: 0.017312200739979744 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1362904777365849) | > loader_time: 0.0046 (0.004660466892136426)  --> TIME: 2025-02-24 12:01:07 -- STEP: 15096/36938 -- GLOBAL_STEP: 310600 | > loss_text_ce: 0.04104530066251755 (0.04132344754738256) | > loss_mel_ce: 3.503476142883301 (nan) | > loss: 0.014065561816096306 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13631906034646882) | > loader_time: 0.0045 (0.00466040936044489)  --> TIME: 2025-02-24 12:01:17 -- STEP: 15146/36938 -- GLOBAL_STEP: 310650 | > loss_text_ce: 0.03102768398821354 (0.04131932567728058) | > loss_mel_ce: 3.182687520980835 (nan) | > loss: 0.012752839364111423 (nan) | > current_lr: 5e-06 | > step_time: 0.1523 (0.13632858372515277) | > loader_time: 0.0046 (0.004660335428315209)  --> TIME: 2025-02-24 12:01:26 -- STEP: 15196/36938 -- GLOBAL_STEP: 310700 | > loss_text_ce: 0.0395578071475029 (0.041318147929795115) | > loss_mel_ce: 4.541275978088379 (nan) | > loss: 0.01817791350185871 (nan) | > current_lr: 5e-06 | > step_time: 0.1267 (0.13631871495632109) | > loader_time: 0.0046 (0.0046601543366265475)  --> TIME: 2025-02-24 12:01:35 -- STEP: 15246/36938 -- GLOBAL_STEP: 310750 | > loss_text_ce: 0.034327149391174316 (0.041314330367694135) | > loss_mel_ce: 0.21901854872703552 (nan) | > loss: 0.0010053401347249746 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13631630534967987) | > loader_time: 0.0045 (0.00465996511242416)  --> TIME: 2025-02-24 12:01:44 -- STEP: 15296/36938 -- GLOBAL_STEP: 310800 | > loss_text_ce: 0.05239903926849365 (0.04131421734532382) | > loss_mel_ce: 3.552011489868164 (nan) | > loss: 0.014303217642009258 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.13632277004950602) | > loader_time: 0.0049 (0.004659828297388576)  --> TIME: 2025-02-24 12:01:54 -- STEP: 15346/36938 -- GLOBAL_STEP: 310850 | > loss_text_ce: 0.039164282381534576 (0.04131814273206319) | > loss_mel_ce: 2.999375820159912 (nan) | > loss: 0.012057699263095856 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.1363240316205291) | > loader_time: 0.0047 (0.004659688986996111)  --> TIME: 2025-02-24 12:02:03 -- STEP: 15396/36938 -- GLOBAL_STEP: 310900 | > loss_text_ce: 0.046211760491132736 (0.04131419851810504) | > loss_mel_ce: 3.0078375339508057 (nan) | > loss: 0.01211924385279417 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13633232882933927) | > loader_time: 0.0046 (0.004659566361429014)  --> TIME: 2025-02-24 12:02:13 -- STEP: 15446/36938 -- GLOBAL_STEP: 310950 | > loss_text_ce: 0.04918020963668823 (0.041309830892245415) | > loss_mel_ce: 3.233994960784912 (nan) | > loss: 0.01302847359329462 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.1363293684070679) | > loader_time: 0.0045 (0.004659342315087988)  --> TIME: 2025-02-24 12:02:22 -- STEP: 15496/36938 -- GLOBAL_STEP: 311000 | > loss_text_ce: 0.05254679545760155 (0.041299454942905676) | > loss_mel_ce: 2.4312400817871094 (nan) | > loss: 0.009856297634541988 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.1363450239537966) | > loader_time: 0.0048 (0.004659408183213513)  --> TIME: 2025-02-24 12:02:31 -- STEP: 15546/36938 -- GLOBAL_STEP: 311050 | > loss_text_ce: 0.04857701063156128 (0.04129750049999748) | > loss_mel_ce: 3.8064255714416504 (nan) | > loss: 0.015297630801796913 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.136327155228671) | > loader_time: 0.0046 (0.0046591926354109)  --> TIME: 2025-02-24 12:02:40 -- STEP: 15596/36938 -- GLOBAL_STEP: 311100 | > loss_text_ce: 0.03582345321774483 (0.04130002707485402) | > loss_mel_ce: 2.0511348247528076 (nan) | > loss: 0.008281580172479153 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.1363177139442914) | > loader_time: 0.0045 (0.0046589857922299605)  --> TIME: 2025-02-24 12:02:50 -- STEP: 15646/36938 -- GLOBAL_STEP: 311150 | > loss_text_ce: 0.04014815762639046 (0.041303922551577214) | > loss_mel_ce: 3.8535447120666504 (nan) | > loss: 0.015451163053512573 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13632778371369259) | > loader_time: 0.0046 (0.004658838816650902)  --> TIME: 2025-02-24 12:02:59 -- STEP: 15696/36938 -- GLOBAL_STEP: 311200 | > loss_text_ce: 0.041974496096372604 (0.04130337834751005) | > loss_mel_ce: 3.4205994606018066 (nan) | > loss: 0.013740373775362968 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.1363248570188945) | > loader_time: 0.0046 (0.004658687354713864)  --> TIME: 2025-02-24 12:03:08 -- STEP: 15746/36938 -- GLOBAL_STEP: 311250 | > loss_text_ce: 0.03730271756649017 (0.041310308395902474) | > loss_mel_ce: 3.431985378265381 (nan) | > loss: 0.013767017051577568 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13633925447193754) | > loader_time: 0.0045 (0.004658657139022609)  --> TIME: 2025-02-24 12:03:18 -- STEP: 15796/36938 -- GLOBAL_STEP: 311300 | > loss_text_ce: 0.027849284932017326 (0.04131262162545814) | > loss_mel_ce: 1.5038602352142334 (nan) | > loss: 0.006078212987631559 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13633156393292054) | > loader_time: 0.0046 (0.004658593259663095)  --> TIME: 2025-02-24 12:03:27 -- STEP: 15846/36938 -- GLOBAL_STEP: 311350 | > loss_text_ce: 0.0590948760509491 (0.041310559495144186) | > loss_mel_ce: 3.3890562057495117 (nan) | > loss: 0.013683140277862549 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13633354407104187) | > loader_time: 0.0046 (0.004658518468853655)  --> TIME: 2025-02-24 12:03:36 -- STEP: 15896/36938 -- GLOBAL_STEP: 311400 | > loss_text_ce: 0.03532944992184639 (0.04130897051575068) | > loss_mel_ce: 2.997096061706543 (nan) | > loss: 0.012033434584736824 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.13632752496990908) | > loader_time: 0.0046 (0.0046583584012490605)  --> TIME: 2025-02-24 12:03:45 -- STEP: 15946/36938 -- GLOBAL_STEP: 311450 | > loss_text_ce: 0.039400506764650345 (0.04130851680247712) | > loss_mel_ce: 0.01654052734375 (nan) | > loss: 0.00022198824444785714 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1363094378695877) | > loader_time: 0.0048 (0.004658204884506959)  --> TIME: 2025-02-24 12:03:55 -- STEP: 15996/36938 -- GLOBAL_STEP: 311500 | > loss_text_ce: 0.03540877625346184 (0.041309808131827094) | > loss_mel_ce: 3.3294553756713867 (nan) | > loss: 0.013352636247873306 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13631638845046312) | > loader_time: 0.0049 (0.004658128416934682)  --> TIME: 2025-02-24 12:04:03 -- STEP: 16046/36938 -- GLOBAL_STEP: 311550 | > loss_text_ce: 0.04154296964406967 (0.041308730852291066) | > loss_mel_ce: 3.185394763946533 (nan) | > loss: 0.012805309146642685 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.13629771696426035) | > loader_time: 0.0046 (0.004657941240184405)  --> TIME: 2025-02-24 12:04:13 -- STEP: 16096/36938 -- GLOBAL_STEP: 311600 | > loss_text_ce: 0.04324612021446228 (0.04130418834560701) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1362938279837774) | > loader_time: 0.0045 (0.004657751212181649)  --> TIME: 2025-02-24 12:04:22 -- STEP: 16146/36938 -- GLOBAL_STEP: 311650 | > loss_text_ce: 0.04599977284669876 (0.04130227203037553) | > loss_mel_ce: 4.3203125 (nan) | > loss: 0.017326638102531433 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13630639752180337) | > loader_time: 0.0045 (0.004657725589094888)  --> TIME: 2025-02-24 12:04:31 -- STEP: 16196/36938 -- GLOBAL_STEP: 311700 | > loss_text_ce: 0.030696410685777664 (0.04130220721643912) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13628379306135968) | > loader_time: 0.0045 (0.004657487452368735)  --> TIME: 2025-02-24 12:04:40 -- STEP: 16246/36938 -- GLOBAL_STEP: 311750 | > loss_text_ce: 0.04665793851017952 (0.04130512609789843) | > loss_mel_ce: 2.7251601219177246 (nan) | > loss: 0.010999279096722603 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1362824516315161) | > loader_time: 0.0046 (0.004657330322805658)  --> TIME: 2025-02-24 12:04:49 -- STEP: 16296/36938 -- GLOBAL_STEP: 311800 | > loss_text_ce: 0.043533384799957275 (0.04130952454925995) | > loss_mel_ce: 3.860323190689087 (nan) | > loss: 0.015491494908928871 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.1362705128153096) | > loader_time: 0.0046 (0.00465714031712469)  --> TIME: 2025-02-24 12:04:58 -- STEP: 16346/36938 -- GLOBAL_STEP: 311850 | > loss_text_ce: 0.03491063416004181 (0.041311106780628395) | > loss_mel_ce: 3.175048828125 (nan) | > loss: 0.012737935408949852 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13625633649488722) | > loader_time: 0.0045 (0.004656919137246869)  --> TIME: 2025-02-24 12:05:08 -- STEP: 16396/36938 -- GLOBAL_STEP: 311900 | > loss_text_ce: 0.03919300064444542 (0.04131282452381658) | > loss_mel_ce: 3.716417074203491 (nan) | > loss: 0.014903214760124683 (nan) | > current_lr: 5e-06 | > step_time: 0.1852 (0.13625881288888494) | > loader_time: 0.005 (0.0046568308611793004)  --> TIME: 2025-02-24 12:05:17 -- STEP: 16446/36938 -- GLOBAL_STEP: 311950 | > loss_text_ce: 0.03738434240221977 (0.041314376125124105) | > loss_mel_ce: 2.8322997093200684 (nan) | > loss: 0.011387635953724384 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13625315017658338) | > loader_time: 0.0045 (0.0046567820174754975)  --> TIME: 2025-02-24 12:05:26 -- STEP: 16496/36938 -- GLOBAL_STEP: 312000 | > loss_text_ce: 0.03244617208838463 (0.04131487119935218) | > loss_mel_ce: 1.3166279792785645 (nan) | > loss: 0.005353468935936689 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13624146641399082) | > loader_time: 0.0045 (0.004656615315653741)  --> TIME: 2025-02-24 12:05:35 -- STEP: 16546/36938 -- GLOBAL_STEP: 312050 | > loss_text_ce: 0.03830510750412941 (0.041313107932727335) | > loss_mel_ce: 3.845226526260376 (nan) | > loss: 0.015410840511322021 (nan) | > current_lr: 5e-06 | > step_time: 0.1879 (0.13624123685178002) | > loader_time: 0.0049 (0.004656489766034544)  --> TIME: 2025-02-24 12:05:44 -- STEP: 16596/36938 -- GLOBAL_STEP: 312100 | > loss_text_ce: 0.039872173219919205 (0.04131237510928441) | > loss_mel_ce: 3.2757487297058105 (nan) | > loss: 0.013157226145267487 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13622963437116828) | > loader_time: 0.0047 (0.004656312292700737)  --> TIME: 2025-02-24 12:05:54 -- STEP: 16646/36938 -- GLOBAL_STEP: 312150 | > loss_text_ce: 0.03578132390975952 (0.04131486907240592) | > loss_mel_ce: 2.8320271968841553 (nan) | > loss: 0.011380193755030632 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13622559503367052) | > loader_time: 0.0045 (0.004656210865780183)  --> TIME: 2025-02-24 12:06:03 -- STEP: 16696/36938 -- GLOBAL_STEP: 312200 | > loss_text_ce: 0.039524488151073456 (0.041312654706960446) | > loss_mel_ce: 2.5565528869628906 (nan) | > loss: 0.010301895439624786 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13621772215248595) | > loader_time: 0.0046 (0.004656108489833929)  --> TIME: 2025-02-24 12:06:12 -- STEP: 16746/36938 -- GLOBAL_STEP: 312250 | > loss_text_ce: 0.03892575576901436 (0.04131164936359689) | > loss_mel_ce: 3.1909961700439453 (nan) | > loss: 0.01281715091317892 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.1362228357249448) | > loader_time: 0.0046 (0.0046560433294473005)  --> TIME: 2025-02-24 12:06:21 -- STEP: 16796/36938 -- GLOBAL_STEP: 312300 | > loss_text_ce: 0.043876465409994125 (0.04131581066052081) | > loss_mel_ce: 1.4367324113845825 (nan) | > loss: 0.005875432398170233 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13620700134098732) | > loader_time: 0.0045 (0.004656128214495188)  --> TIME: 2025-02-24 12:06:31 -- STEP: 16846/36938 -- GLOBAL_STEP: 312350 | > loss_text_ce: 0.048201195895671844 (0.04131731742208612) | > loss_mel_ce: 3.546978235244751 (nan) | > loss: 0.014266584999859333 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.13620589248757214) | > loader_time: 0.0049 (0.004656101524242498)  --> TIME: 2025-02-24 12:06:40 -- STEP: 16896/36938 -- GLOBAL_STEP: 312400 | > loss_text_ce: 0.0456191711127758 (0.04131878115136211) | > loss_mel_ce: 2.0564005374908447 (nan) | > loss: 0.008341348730027676 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.1362026174214066) | > loader_time: 0.0047 (0.004655948543751757)  --> TIME: 2025-02-24 12:06:49 -- STEP: 16946/36938 -- GLOBAL_STEP: 312450 | > loss_text_ce: 0.04209054261445999 (0.04132021517676054) | > loss_mel_ce: 3.1023361682891846 (nan) | > loss: 0.012477885000407696 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13619318415588672) | > loader_time: 0.0047 (0.004655862957587342)  --> TIME: 2025-02-24 12:06:58 -- STEP: 16996/36938 -- GLOBAL_STEP: 312500 | > loss_text_ce: 0.0433785654604435 (0.041320025980267315) | > loss_mel_ce: 2.401747941970825 (nan) | > loss: 0.009702883660793304 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1362003096835018) | > loader_time: 0.0047 (0.004656111711500704)  --> TIME: 2025-02-24 12:07:08 -- STEP: 17046/36938 -- GLOBAL_STEP: 312550 | > loss_text_ce: 0.04002613574266434 (0.04131805045379615) | > loss_mel_ce: 3.7468643188476562 (nan) | > loss: 0.015027344226837158 (nan) | > current_lr: 5e-06 | > step_time: 0.176 (0.13620127194310622) | > loader_time: 0.0049 (0.004656089522896476)  --> TIME: 2025-02-24 12:07:17 -- STEP: 17096/36938 -- GLOBAL_STEP: 312600 | > loss_text_ce: 0.0392417311668396 (0.04131840039330066) | > loss_mel_ce: 2.270674228668213 (nan) | > loss: 0.009166333824396133 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.1361919008352281) | > loader_time: 0.0046 (0.004656183423982561)  --> TIME: 2025-02-24 12:07:26 -- STEP: 17146/36938 -- GLOBAL_STEP: 312650 | > loss_text_ce: 0.04889620840549469 (0.041320655440393816) | > loss_mel_ce: 3.493220806121826 (nan) | > loss: 0.014056021347641945 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.1362037798994876) | > loader_time: 0.0046 (0.004656364547042442)  --> TIME: 2025-02-24 12:07:36 -- STEP: 17196/36938 -- GLOBAL_STEP: 312700 | > loss_text_ce: 0.03972858190536499 (0.04132129783974141) | > loss_mel_ce: 3.161407470703125 (nan) | > loss: 0.012702921405434608 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.13620142752027714) | > loader_time: 0.0048 (0.004656309803077337)  --> TIME: 2025-02-24 12:07:45 -- STEP: 17246/36938 -- GLOBAL_STEP: 312750 | > loss_text_ce: 0.04432804882526398 (0.04132216774418842) | > loss_mel_ce: 1.5779151916503906 (nan) | > loss: 0.006437473930418491 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13620654979244745) | > loader_time: 0.0045 (0.0046563616875054704)  --> TIME: 2025-02-24 12:07:54 -- STEP: 17296/36938 -- GLOBAL_STEP: 312800 | > loss_text_ce: 0.0468827486038208 (0.04132446825577958) | > loss_mel_ce: 2.752963066101074 (nan) | > loss: 0.011110499501228333 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.136215384515427) | > loader_time: 0.0047 (0.004656381098679301)  --> TIME: 2025-02-24 12:08:04 -- STEP: 17346/36938 -- GLOBAL_STEP: 312850 | > loss_text_ce: 0.047601278871297836 (0.041322701509363666) | > loss_mel_ce: 3.061159610748291 (nan) | > loss: 0.012336352840065956 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1362258700176482) | > loader_time: 0.0046 (0.0046564930658858136)  --> TIME: 2025-02-24 12:08:13 -- STEP: 17396/36938 -- GLOBAL_STEP: 312900 | > loss_text_ce: 0.03703832998871803 (0.04132326212044561) | > loss_mel_ce: 3.1710076332092285 (nan) | > loss: 0.012730341404676437 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13622461082908377) | > loader_time: 0.0047 (0.004656579267515405)  --> TIME: 2025-02-24 12:08:23 -- STEP: 17446/36938 -- GLOBAL_STEP: 312950 | > loss_text_ce: 0.04530225321650505 (0.041320290700263926) | > loss_mel_ce: 1.903478980064392 (nan) | > loss: 0.007733259350061417 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13622722532256284) | > loader_time: 0.0045 (0.004656650270328415)  --> TIME: 2025-02-24 12:08:32 -- STEP: 17496/36938 -- GLOBAL_STEP: 313000 | > loss_text_ce: 0.034949179738759995 (0.04132226946992369) | > loss_mel_ce: 3.2300970554351807 (nan) | > loss: 0.012956532649695873 (nan) | > current_lr: 5e-06 | > step_time: 0.1511 (0.13621608602929688) | > loader_time: 0.0049 (0.004656608294394098)  --> TIME: 2025-02-24 12:08:41 -- STEP: 17546/36938 -- GLOBAL_STEP: 313050 | > loss_text_ce: 0.05391706898808479 (0.041322041057376295) | > loss_mel_ce: 3.0843544006347656 (nan) | > loss: 0.012453459203243256 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.1362210851186867) | > loader_time: 0.0045 (0.004656785952790775)  --> TIME: 2025-02-24 12:08:51 -- STEP: 17596/36938 -- GLOBAL_STEP: 313100 | > loss_text_ce: 0.053244367241859436 (0.04132140385570048) | > loss_mel_ce: 3.178478717803955 (nan) | > loss: 0.012824298813939095 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13625327258739747) | > loader_time: 0.0045 (0.0046574268240255345)  --> TIME: 2025-02-24 12:09:01 -- STEP: 17646/36938 -- GLOBAL_STEP: 313150 | > loss_text_ce: 0.04541686549782753 (0.04132188543089167) | > loss_mel_ce: 3.2248547077178955 (nan) | > loss: 0.012977268546819687 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13625822834156195) | > loader_time: 0.0045 (0.0046574250513850765)  --> TIME: 2025-02-24 12:09:10 -- STEP: 17696/36938 -- GLOBAL_STEP: 313200 | > loss_text_ce: 0.0399838425219059 (0.0413206582805807) | > loss_mel_ce: 3.0877087116241455 (nan) | > loss: 0.012411478906869888 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.13624387864986692) | > loader_time: 0.0048 (0.004657561454591857)  --> TIME: 2025-02-24 12:09:19 -- STEP: 17746/36938 -- GLOBAL_STEP: 313250 | > loss_text_ce: 0.03025013580918312 (0.041320620210471615) | > loss_mel_ce: 3.077568292617798 (nan) | > loss: 0.012332613579928875 (nan) | > current_lr: 5e-06 | > step_time: 0.1856 (0.13623775235294985) | > loader_time: 0.0055 (0.004657641011215191)  --> TIME: 2025-02-24 12:09:28 -- STEP: 17796/36938 -- GLOBAL_STEP: 313300 | > loss_text_ce: 0.04528714716434479 (0.0413224617048393) | > loss_mel_ce: 2.742100954055786 (nan) | > loss: 0.011061064898967743 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.1362319622020495) | > loader_time: 0.0047 (0.004657773160752234)  --> TIME: 2025-02-24 12:09:37 -- STEP: 17846/36938 -- GLOBAL_STEP: 313350 | > loss_text_ce: 0.04830173775553703 (0.041319823740683756) | > loss_mel_ce: 3.2702205181121826 (nan) | > loss: 0.013168740086257458 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.1362283930548173) | > loader_time: 0.0046 (0.004657782033915788)  --> TIME: 2025-02-24 12:09:46 -- STEP: 17896/36938 -- GLOBAL_STEP: 313400 | > loss_text_ce: 0.03536504507064819 (0.041321324097836926) | > loss_mel_ce: 3.080946207046509 (nan) | > loss: 0.012366315349936485 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.1362145370572704) | > loader_time: 0.0046 (0.004658116458194503)  --> TIME: 2025-02-24 12:09:56 -- STEP: 17946/36938 -- GLOBAL_STEP: 313450 | > loss_text_ce: 0.0351497083902359 (0.041316011802664004) | > loss_mel_ce: 3.34731125831604 (nan) | > loss: 0.013422464951872826 (nan) | > current_lr: 5e-06 | > step_time: 0.1884 (0.13622007260523897) | > loader_time: 0.0049 (0.004658086409101665)  --> TIME: 2025-02-24 12:10:05 -- STEP: 17996/36938 -- GLOBAL_STEP: 313500 | > loss_text_ce: 0.03621945157647133 (0.04131248077666659) | > loss_mel_ce: 2.987109422683716 (nan) | > loss: 0.011997337453067303 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13622242539531446) | > loader_time: 0.0046 (0.004658300403913772)  --> TIME: 2025-02-24 12:10:14 -- STEP: 18046/36938 -- GLOBAL_STEP: 313550 | > loss_text_ce: 0.038766562938690186 (0.04130892481753557) | > loss_mel_ce: 2.4715027809143066 (nan) | > loss: 0.009961387142539024 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.13620768065834582) | > loader_time: 0.0048 (0.004658197492582355)  --> TIME: 2025-02-24 12:10:23 -- STEP: 18096/36938 -- GLOBAL_STEP: 313600 | > loss_text_ce: 0.04364420473575592 (0.04131177016445388) | > loss_mel_ce: 0.13348388671875 (nan) | > loss: 0.0007028892869129777 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13620829130304624) | > loader_time: 0.0045 (0.0046582922510717)  --> TIME: 2025-02-24 12:10:33 -- STEP: 18146/36938 -- GLOBAL_STEP: 313650 | > loss_text_ce: 0.03714340180158615 (0.041306622460534556) | > loss_mel_ce: 2.589221239089966 (nan) | > loss: 0.010422082617878914 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13623170329469855) | > loader_time: 0.0045 (0.00465847781589271)  --> TIME: 2025-02-24 12:10:42 -- STEP: 18196/36938 -- GLOBAL_STEP: 313700 | > loss_text_ce: 0.03780073672533035 (0.04130943133232663) | > loss_mel_ce: 3.0731139183044434 (nan) | > loss: 0.012344900518655777 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13623237134231528) | > loader_time: 0.0048 (0.004658475383922895)  --> TIME: 2025-02-24 12:10:52 -- STEP: 18246/36938 -- GLOBAL_STEP: 313750 | > loss_text_ce: 0.04203866049647331 (0.04130764898930727) | > loss_mel_ce: 1.4930213689804077 (nan) | > loss: 0.00609150854870677 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13624221571937192) | > loader_time: 0.0045 (0.004658389023543832)  --> TIME: 2025-02-24 12:11:01 -- STEP: 18296/36938 -- GLOBAL_STEP: 313800 | > loss_text_ce: 0.033694762736558914 (0.04130039232219821) | > loss_mel_ce: 1.726435661315918 (nan) | > loss: 0.006984645035117865 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13624824843502545) | > loader_time: 0.0046 (0.0046582763170017845)  --> TIME: 2025-02-24 12:11:11 -- STEP: 18346/36938 -- GLOBAL_STEP: 313850 | > loss_text_ce: 0.03561285510659218 (0.041300589509006135) | > loss_mel_ce: 3.1904990673065186 (nan) | > loss: 0.012802031822502613 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.1362498847372353) | > loader_time: 0.0047 (0.004658351310477125)  --> TIME: 2025-02-24 12:11:20 -- STEP: 18396/36938 -- GLOBAL_STEP: 313900 | > loss_text_ce: 0.04673541337251663 (0.041300835888399404) | > loss_mel_ce: 3.406611919403076 (nan) | > loss: 0.01370376069098711 (nan) | > current_lr: 5e-06 | > step_time: 0.1753 (0.13627232601135042) | > loader_time: 0.0048 (0.004658316459104159)  --> TIME: 2025-02-24 12:11:30 -- STEP: 18446/36938 -- GLOBAL_STEP: 313950 | > loss_text_ce: 0.05558813363313675 (0.04130411056655658) | > loss_mel_ce: 3.4986722469329834 (nan) | > loss: 0.014104208908975124 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13627407940636319) | > loader_time: 0.0047 (0.004658203069163043)  --> TIME: 2025-02-24 12:11:39 -- STEP: 18496/36938 -- GLOBAL_STEP: 314000 | > loss_text_ce: 0.05451356619596481 (0.041303625366884936) | > loss_mel_ce: 3.089186191558838 (nan) | > loss: 0.012474999763071537 (nan) | > current_lr: 5e-06 | > step_time: 0.1827 (0.13626794357200772) | > loader_time: 0.0048 (0.004658034077771383)  --> TIME: 2025-02-24 12:11:48 -- STEP: 18546/36938 -- GLOBAL_STEP: 314050 | > loss_text_ce: 0.04093556851148605 (0.041304303833323494) | > loss_mel_ce: 3.41656494140625 (nan) | > loss: 0.013720240443944931 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13626478030228287) | > loader_time: 0.0045 (0.004658019505413057)  --> TIME: 2025-02-24 12:11:57 -- STEP: 18596/36938 -- GLOBAL_STEP: 314100 | > loss_text_ce: 0.03886348381638527 (0.04130405863361563) | > loss_mel_ce: 3.2932968139648438 (nan) | > loss: 0.013222859241068363 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.1362543873798205) | > loader_time: 0.0049 (0.0046578305694307525)  --> TIME: 2025-02-24 12:12:07 -- STEP: 18646/36938 -- GLOBAL_STEP: 314150 | > loss_text_ce: 0.044590823352336884 (0.041302737278441554) | > loss_mel_ce: 3.3320674896240234 (nan) | > loss: 0.013399438001215458 (nan) | > current_lr: 5e-06 | > step_time: 0.1878 (0.1362666424719874) | > loader_time: 0.005 (0.004657914208122977)  --> TIME: 2025-02-24 12:12:16 -- STEP: 18696/36938 -- GLOBAL_STEP: 314200 | > loss_text_ce: 0.048053570091724396 (0.041301281452334276) | > loss_mel_ce: 3.7091634273529053 (nan) | > loss: 0.014909591525793076 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.1362583145760648) | > loader_time: 0.0047 (0.0046579617947964756)  --> TIME: 2025-02-24 12:12:25 -- STEP: 18746/36938 -- GLOBAL_STEP: 314250 | > loss_text_ce: 0.05274081975221634 (0.0413038086624644) | > loss_mel_ce: 3.2019360065460205 (nan) | > loss: 0.012915384955704212 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.1362629853380557) | > loader_time: 0.0046 (0.004658038392591451)  --> TIME: 2025-02-24 12:12:34 -- STEP: 18796/36938 -- GLOBAL_STEP: 314300 | > loss_text_ce: 0.034550271928310394 (0.04130374527978101) | > loss_mel_ce: 3.2194530963897705 (nan) | > loss: 0.012912712059915066 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.1362588576343008) | > loader_time: 0.0046 (0.0046581191366138345)  --> TIME: 2025-02-24 12:12:43 -- STEP: 18846/36938 -- GLOBAL_STEP: 314350 | > loss_text_ce: 0.042232945561409 (0.04130576505733616) | > loss_mel_ce: 3.5052778720855713 (nan) | > loss: 0.014077425003051758 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13624679312310717) | > loader_time: 0.0046 (0.0046582189598597555)  --> TIME: 2025-02-24 12:12:52 -- STEP: 18896/36938 -- GLOBAL_STEP: 314400 | > loss_text_ce: 0.0423954613506794 (0.04130344870206505) | > loss_mel_ce: 3.932969570159912 (nan) | > loss: 0.015775257721543312 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13623755047323277) | > loader_time: 0.0046 (0.004658109436693273)  --> TIME: 2025-02-24 12:13:02 -- STEP: 18946/36938 -- GLOBAL_STEP: 314450 | > loss_text_ce: 0.048966217786073685 (0.04130324534560608) | > loss_mel_ce: 3.548323392868042 (nan) | > loss: 0.014274959452450275 (nan) | > current_lr: 5e-06 | > step_time: 0.1929 (0.13624337894113853) | > loader_time: 0.0049 (0.004658117460925155)  --> TIME: 2025-02-24 12:13:11 -- STEP: 18996/36938 -- GLOBAL_STEP: 314500 | > loss_text_ce: 0.048902325332164764 (0.04130407626832599) | > loss_mel_ce: 3.7257080078125 (nan) | > loss: 0.014978612773120403 (nan) | > current_lr: 5e-06 | > step_time: 0.1436 (0.13624148174044037) | > loader_time: 0.0046 (0.004658014793400003)  --> TIME: 2025-02-24 12:13:21 -- STEP: 19046/36938 -- GLOBAL_STEP: 314550 | > loss_text_ce: 0.044024743139743805 (0.041306639914806154) | > loss_mel_ce: 3.51271653175354 (nan) | > loss: 0.01411405298858881 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13624105734646313) | > loader_time: 0.0047 (0.004658119099913521)  --> TIME: 2025-02-24 12:13:30 -- STEP: 19096/36938 -- GLOBAL_STEP: 314600 | > loss_text_ce: 0.07185561954975128 (0.04131280983174712) | > loss_mel_ce: 1.0015575885772705 (nan) | > loss: 0.004259576555341482 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13623934693300477) | > loader_time: 0.0053 (0.004658299682027472)  --> TIME: 2025-02-24 12:13:39 -- STEP: 19146/36938 -- GLOBAL_STEP: 314650 | > loss_text_ce: 0.0337405800819397 (0.0413117678151091) | > loss_mel_ce: 2.9551711082458496 (nan) | > loss: 0.011860760860145092 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13623846399115327) | > loader_time: 0.0046 (0.0046588673582149635)  --> TIME: 2025-02-24 12:13:48 -- STEP: 19196/36938 -- GLOBAL_STEP: 314700 | > loss_text_ce: 0.05900166556239128 (0.0413118668855754) | > loss_mel_ce: 3.9213168621063232 (nan) | > loss: 0.015794916078448296 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13623883080099888) | > loader_time: 0.0045 (0.004658855393817699)  --> TIME: 2025-02-24 12:13:58 -- STEP: 19246/36938 -- GLOBAL_STEP: 314750 | > loss_text_ce: 0.04579009860754013 (0.04130982753258317) | > loss_mel_ce: 3.347273349761963 (nan) | > loss: 0.013464538380503654 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1362460273771212) | > loader_time: 0.0045 (0.004658867710037802)  --> TIME: 2025-02-24 12:14:07 -- STEP: 19296/36938 -- GLOBAL_STEP: 314800 | > loss_text_ce: 0.04024985060095787 (0.04131018100511028) | > loss_mel_ce: 3.618558883666992 (nan) | > loss: 0.014519083313643932 (nan) | > current_lr: 5e-06 | > step_time: 0.1629 (0.1362465493022697) | > loader_time: 0.0046 (0.00465882927871264)  --> TIME: 2025-02-24 12:14:17 -- STEP: 19346/36938 -- GLOBAL_STEP: 314850 | > loss_text_ce: 0.04517906904220581 (0.04131451097957204) | > loss_mel_ce: 3.6801469326019287 (nan) | > loss: 0.0147830406203866 (nan) | > current_lr: 5e-06 | > step_time: 0.1634 (0.1362586393605752) | > loader_time: 0.0047 (0.004658918327497961)  --> TIME: 2025-02-24 12:14:26 -- STEP: 19396/36938 -- GLOBAL_STEP: 314900 | > loss_text_ce: 0.04200141876935959 (0.04131482587114106) | > loss_mel_ce: 3.274780035018921 (nan) | > loss: 0.013161832466721535 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13625363594085782) | > loader_time: 0.0045 (0.004658912783874678)  --> TIME: 2025-02-24 12:14:35 -- STEP: 19446/36938 -- GLOBAL_STEP: 314950 | > loss_text_ce: 0.04054202884435654 (0.04131468152847648) | > loss_mel_ce: 3.4106876850128174 (nan) | > loss: 0.013695357367396355 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.13626037322725698) | > loader_time: 0.0047 (0.004658993264229394)  --> TIME: 2025-02-24 12:14:44 -- STEP: 19496/36938 -- GLOBAL_STEP: 315000 | > loss_text_ce: 0.042576864361763 (0.041312928272020946) | > loss_mel_ce: 0.1362202912569046 (nan) | > loss: 0.0007095125620253384 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1362536941266318) | > loader_time: 0.0048 (0.004659378435653788)  --> TIME: 2025-02-24 12:14:54 -- STEP: 19546/36938 -- GLOBAL_STEP: 315050 | > loss_text_ce: 0.03342096135020256 (0.04131390038544373) | > loss_mel_ce: 3.748779296875 (nan) | > loss: 0.015008732676506042 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13627279572021844) | > loader_time: 0.0046 (0.004659507324150824)  --> TIME: 2025-02-24 12:15:03 -- STEP: 19596/36938 -- GLOBAL_STEP: 315100 | > loss_text_ce: 0.035666197538375854 (0.041314431644052525) | > loss_mel_ce: 2.623074531555176 (nan) | > loss: 0.010550559498369694 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13627285985563117) | > loader_time: 0.0046 (0.0046595549262234155)  --> TIME: 2025-02-24 12:15:13 -- STEP: 19646/36938 -- GLOBAL_STEP: 315150 | > loss_text_ce: 0.0446937158703804 (0.04131482240239442) | > loss_mel_ce: 3.4155867099761963 (nan) | > loss: 0.013731271959841251 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.13628294665010263) | > loader_time: 0.0046 (0.004659600538451635)  --> TIME: 2025-02-24 12:15:22 -- STEP: 19696/36938 -- GLOBAL_STEP: 315200 | > loss_text_ce: 0.04121747240424156 (0.041314693533837696) | > loss_mel_ce: 3.6883883476257324 (nan) | > loss: 0.014800024218857288 (nan) | > current_lr: 5e-06 | > step_time: 0.15 (0.1362800649469403) | > loader_time: 0.0046 (0.004659532544382395)  --> TIME: 2025-02-24 12:15:32 -- STEP: 19746/36938 -- GLOBAL_STEP: 315250 | > loss_text_ce: 0.046070534735918045 (0.04131507025897658) | > loss_mel_ce: 0.01495361328125 (nan) | > loss: 0.00024215933808591217 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1362847890412441) | > loader_time: 0.0046 (0.004659566596251509)  --> TIME: 2025-02-24 12:15:41 -- STEP: 19796/36938 -- GLOBAL_STEP: 315300 | > loss_text_ce: 0.04112159088253975 (0.04131451214934287) | > loss_mel_ce: 3.313908100128174 (nan) | > loss: 0.01331361010670662 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.1362845561773656) | > loader_time: 0.0047 (0.004659583229420267)  --> TIME: 2025-02-24 12:15:51 -- STEP: 19846/36938 -- GLOBAL_STEP: 315350 | > loss_text_ce: 0.05178692191839218 (0.0413171448398508) | > loss_mel_ce: 3.7740464210510254 (nan) | > loss: 0.01518187951296568 (nan) | > current_lr: 5e-06 | > step_time: 0.1599 (0.1363032987302047) | > loader_time: 0.0048 (0.004659602013276265)  --> TIME: 2025-02-24 12:16:00 -- STEP: 19896/36938 -- GLOBAL_STEP: 315400 | > loss_text_ce: 0.020655976608395576 (0.04131762312823853) | > loss_mel_ce: 2.0548324584960938 (nan) | > loss: 0.00823606550693512 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13629253851538098) | > loader_time: 0.0044 (0.0046594726617526965)  --> TIME: 2025-02-24 12:16:09 -- STEP: 19946/36938 -- GLOBAL_STEP: 315450 | > loss_text_ce: 0.042038023471832275 (0.041317803657128095) | > loss_mel_ce: 0.19776611030101776 (nan) | > loss: 0.0009516037534922361 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13629557681659754) | > loader_time: 0.0046 (0.004659300114390574)  --> TIME: 2025-02-24 12:16:18 -- STEP: 19996/36938 -- GLOBAL_STEP: 315500 | > loss_text_ce: 0.03728686273097992 (0.041322736269508345) | > loss_mel_ce: 3.540059804916382 (nan) | > loss: 0.014195820316672325 (nan) | > current_lr: 5e-06 | > step_time: 0.1767 (0.13628577456328286) | > loader_time: 0.005 (0.004659370544744163)  --> TIME: 2025-02-24 12:16:28 -- STEP: 20046/36938 -- GLOBAL_STEP: 315550 | > loss_text_ce: 0.04311112314462662 (0.04132421674882549) | > loss_mel_ce: 3.4905176162719727 (nan) | > loss: 0.014022337272763252 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.13630127070681006) | > loader_time: 0.0046 (0.004659265038925022)  --> TIME: 2025-02-24 12:16:37 -- STEP: 20096/36938 -- GLOBAL_STEP: 315600 | > loss_text_ce: 0.04109852761030197 (0.04132407730679568) | > loss_mel_ce: 2.0049214363098145 (nan) | > loss: 0.008119127713143826 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13630438151110816) | > loader_time: 0.0044 (0.004659075764523944)  --> TIME: 2025-02-24 12:16:46 -- STEP: 20146/36938 -- GLOBAL_STEP: 315650 | > loss_text_ce: 0.04889928549528122 (0.04131808682128216) | > loss_mel_ce: 3.375659942626953 (nan) | > loss: 0.013589520938694477 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13629185394817991) | > loader_time: 0.0046 (0.004659046864516678)  --> TIME: 2025-02-24 12:16:56 -- STEP: 20196/36938 -- GLOBAL_STEP: 315700 | > loss_text_ce: 0.04498005285859108 (0.04132016454719089) | > loss_mel_ce: 3.007878541946411 (nan) | > loss: 0.012114519253373146 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.1362980887497449) | > loader_time: 0.0045 (0.00465894997958075)  --> TIME: 2025-02-24 12:17:05 -- STEP: 20246/36938 -- GLOBAL_STEP: 315750 | > loss_text_ce: 0.052751004695892334 (0.04132115766188926) | > loss_mel_ce: 1.265625 (nan) | > loss: 0.0052316514775156975 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13631387318851604) | > loader_time: 0.0045 (0.004658936382600062)  --> TIME: 2025-02-24 12:17:15 -- STEP: 20296/36938 -- GLOBAL_STEP: 315800 | > loss_text_ce: 0.04286009073257446 (0.04132138557400237) | > loss_mel_ce: 3.59244704246521 (nan) | > loss: 0.014425822533667088 (nan) | > current_lr: 5e-06 | > step_time: 0.1619 (0.136320015566058) | > loader_time: 0.0046 (0.004658870073016998)  --> TIME: 2025-02-24 12:17:24 -- STEP: 20346/36938 -- GLOBAL_STEP: 315850 | > loss_text_ce: 0.049673572182655334 (0.04132017467705949) | > loss_mel_ce: 2.8083620071411133 (nan) | > loss: 0.011341411620378494 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13631421726595738) | > loader_time: 0.0049 (0.0046587874612121)  --> TIME: 2025-02-24 12:17:33 -- STEP: 20396/36938 -- GLOBAL_STEP: 315900 | > loss_text_ce: 0.04281807690858841 (0.041321509827134985) | > loss_mel_ce: 2.6269288063049316 (nan) | > loss: 0.010594233870506287 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13631349022329026) | > loader_time: 0.0044 (0.004658878445742194)  --> TIME: 2025-02-24 12:17:43 -- STEP: 20446/36938 -- GLOBAL_STEP: 315950 | > loss_text_ce: 0.03915262967348099 (0.04132253066952273) | > loss_mel_ce: 3.651984930038452 (nan) | > loss: 0.01464737206697464 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1363283961903469) | > loader_time: 0.0047 (0.004658867733755837)  --> TIME: 2025-02-24 12:17:52 -- STEP: 20496/36938 -- GLOBAL_STEP: 316000 | > loss_text_ce: 0.03587697073817253 (0.04132089171461182) | > loss_mel_ce: 3.159868001937866 (nan) | > loss: 0.012681528925895691 (nan) | > current_lr: 5e-06 | > step_time: 0.1859 (0.13633500309450136) | > loader_time: 0.0049 (0.004659179920893362)  --> TIME: 2025-02-24 12:18:02 -- STEP: 20546/36938 -- GLOBAL_STEP: 316050 | > loss_text_ce: 0.04550290107727051 (0.041321698309316646) | > loss_mel_ce: 3.3142483234405518 (nan) | > loss: 0.013332346454262733 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13633918918862808) | > loader_time: 0.0045 (0.004659208921382178)  --> TIME: 2025-02-24 12:18:11 -- STEP: 20596/36938 -- GLOBAL_STEP: 316100 | > loss_text_ce: 0.04308055713772774 (0.04132144912121599) | > loss_mel_ce: 2.640235662460327 (nan) | > loss: 0.010648081079125404 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13635144004036412) | > loader_time: 0.0045 (0.004659107644063885)  --> TIME: 2025-02-24 12:18:21 -- STEP: 20646/36938 -- GLOBAL_STEP: 316150 | > loss_text_ce: 0.043671634048223495 (0.041320463529163784) | > loss_mel_ce: 3.241922378540039 (nan) | > loss: 0.013038071803748608 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13634506540336305) | > loader_time: 0.0045 (0.004658936645868467)  --> TIME: 2025-02-24 12:18:30 -- STEP: 20696/36938 -- GLOBAL_STEP: 316200 | > loss_text_ce: 0.050378307700157166 (0.04132162187628981) | > loss_mel_ce: 3.4108145236968994 (nan) | > loss: 0.013734892942011356 (nan) | > current_lr: 5e-06 | > step_time: 0.1558 (0.13634257520263496) | > loader_time: 0.0046 (0.004658792082941749)  --> TIME: 2025-02-24 12:18:39 -- STEP: 20746/36938 -- GLOBAL_STEP: 316250 | > loss_text_ce: 0.03023991733789444 (0.04132031640309929) | > loss_mel_ce: 1.4265908002853394 (nan) | > loss: 0.005781074520200491 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13634821638403138) | > loader_time: 0.0046 (0.004658826013706821)  --> TIME: 2025-02-24 12:18:48 -- STEP: 20796/36938 -- GLOBAL_STEP: 316300 | > loss_text_ce: 0.05082850903272629 (0.04131660205852155) | > loss_mel_ce: 2.462757110595703 (nan) | > loss: 0.009974546730518341 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13634284308700048) | > loader_time: 0.0046 (0.004658785341427531)  --> TIME: 2025-02-24 12:18:58 -- STEP: 20846/36938 -- GLOBAL_STEP: 316350 | > loss_text_ce: 0.044258151203393936 (0.04132009938017565) | > loss_mel_ce: 3.6676816940307617 (nan) | > loss: 0.014729920774698257 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.13634403767769201) | > loader_time: 0.0046 (0.00465870727038389)  --> TIME: 2025-02-24 12:19:07 -- STEP: 20896/36938 -- GLOBAL_STEP: 316400 | > loss_text_ce: 0.04361347109079361 (0.04132088024308605) | > loss_mel_ce: 3.32981014251709 (nan) | > loss: 0.013386602513492107 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.136342242462803) | > loader_time: 0.0048 (0.004658609742775066)  --> TIME: 2025-02-24 12:19:16 -- STEP: 20946/36938 -- GLOBAL_STEP: 316450 | > loss_text_ce: 0.04849190264940262 (0.04131986448246211) | > loss_mel_ce: 2.589472532272339 (nan) | > loss: 0.010468113236129284 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13633360690172688) | > loader_time: 0.0044 (0.004658454049342694)  --> TIME: 2025-02-24 12:19:25 -- STEP: 20996/36938 -- GLOBAL_STEP: 316500 | > loss_text_ce: 0.03506734222173691 (0.04131925022559555) | > loss_mel_ce: 3.3803317546844482 (nan) | > loss: 0.013553171418607235 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13632640186367226) | > loader_time: 0.0046 (0.004658315369778063)  --> TIME: 2025-02-24 12:19:35 -- STEP: 21046/36938 -- GLOBAL_STEP: 316550 | > loss_text_ce: 0.04032375290989876 (0.041316819212538325) | > loss_mel_ce: 3.3858883380889893 (nan) | > loss: 0.013596080243587494 (nan) | > current_lr: 5e-06 | > step_time: 0.1904 (0.13633418511406356) | > loader_time: 0.0049 (0.00465827959974738)  --> TIME: 2025-02-24 12:19:44 -- STEP: 21096/36938 -- GLOBAL_STEP: 316600 | > loss_text_ce: 0.03898508846759796 (0.041318283914042624) | > loss_mel_ce: 1.0186035633087158 (nan) | > loss: 0.004196780733764172 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13633667754774342) | > loader_time: 0.0045 (0.004658213643174208)  --> TIME: 2025-02-24 12:19:54 -- STEP: 21146/36938 -- GLOBAL_STEP: 316650 | > loss_text_ce: 0.04876060038805008 (0.04131821433559103) | > loss_mel_ce: 3.442131996154785 (nan) | > loss: 0.013852749019861221 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13633925664687602) | > loader_time: 0.0047 (0.004658247420389351)  --> TIME: 2025-02-24 12:20:03 -- STEP: 21196/36938 -- GLOBAL_STEP: 316700 | > loss_text_ce: 0.04837053269147873 (0.04131938251642988) | > loss_mel_ce: 3.240846872329712 (nan) | > loss: 0.01305245142430067 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.1363333042083225) | > loader_time: 0.0047 (0.004658297708201434)  --> TIME: 2025-02-24 12:20:12 -- STEP: 21246/36938 -- GLOBAL_STEP: 316750 | > loss_text_ce: 0.049548737704753876 (0.04131902496450628) | > loss_mel_ce: 0.7979927062988281 (nan) | > loss: 0.0033632598351687193 (nan) | > current_lr: 5e-06 | > step_time: 0.1517 (0.1363402447165893) | > loader_time: 0.0047 (0.00465831858261396)  --> TIME: 2025-02-24 12:20:22 -- STEP: 21296/36938 -- GLOBAL_STEP: 316800 | > loss_text_ce: 0.041849106550216675 (0.0413180496259385) | > loss_mel_ce: 3.878979206085205 (nan) | > loss: 0.01555884350091219 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13634314024743943) | > loader_time: 0.0046 (0.004658280302939057)  --> TIME: 2025-02-24 12:20:31 -- STEP: 21346/36938 -- GLOBAL_STEP: 316850 | > loss_text_ce: 0.039004649966955185 (0.041315904435754176) | > loss_mel_ce: 2.5017905235290527 (nan) | > loss: 0.010082520544528961 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.1363490382655316) | > loader_time: 0.0048 (0.004658246558596927)  --> TIME: 2025-02-24 12:20:41 -- STEP: 21396/36938 -- GLOBAL_STEP: 316900 | > loss_text_ce: 0.049364712089300156 (0.04131402856772858) | > loss_mel_ce: 0.015625 (nan) | > loss: 0.00025789570645429194 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13635780347086682) | > loader_time: 0.0047 (0.004658398170920477)  --> TIME: 2025-02-24 12:20:50 -- STEP: 21446/36938 -- GLOBAL_STEP: 316950 | > loss_text_ce: 0.04282178357243538 (0.04131086167685623) | > loss_mel_ce: 3.1353917121887207 (nan) | > loss: 0.012611959129571915 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13635018859505038) | > loader_time: 0.0047 (0.004658315793842911)  --> TIME: 2025-02-24 12:20:59 -- STEP: 21496/36938 -- GLOBAL_STEP: 317000 | > loss_text_ce: 0.04159444198012352 (0.04131231357736532) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13635320505068377) | > loader_time: 0.0046 (0.004658301257270832)  --> TIME: 2025-02-24 12:21:08 -- STEP: 21546/36938 -- GLOBAL_STEP: 317050 | > loss_text_ce: 0.04568206146359444 (0.041314405861664044) | > loss_mel_ce: 3.4706246852874756 (nan) | > loss: 0.013953598216176033 (nan) | > current_lr: 5e-06 | > step_time: 0.1871 (0.13634916833767613) | > loader_time: 0.0049 (0.004658201273510319)  --> TIME: 2025-02-24 12:21:18 -- STEP: 21596/36938 -- GLOBAL_STEP: 317100 | > loss_text_ce: 0.046291474252939224 (0.041312669017107964) | > loss_mel_ce: 2.6580874919891357 (nan) | > loss: 0.010731663554906845 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.1363503666007094) | > loader_time: 0.0046 (0.004658216833074704)  --> TIME: 2025-02-24 12:21:27 -- STEP: 21646/36938 -- GLOBAL_STEP: 317150 | > loss_text_ce: 0.039115309715270996 (0.04131093892381293) | > loss_mel_ce: 3.777113914489746 (nan) | > loss: 0.015143767930567265 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1363552140405348) | > loader_time: 0.0046 (0.004658233168869067)  --> TIME: 2025-02-24 12:21:37 -- STEP: 21696/36938 -- GLOBAL_STEP: 317200 | > loss_text_ce: 0.049990952014923096 (0.04130944423583548) | > loss_mel_ce: 3.040478467941284 (nan) | > loss: 0.01226376835256815 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13636617703160017) | > loader_time: 0.0046 (0.004658321209889246)  --> TIME: 2025-02-24 12:21:46 -- STEP: 21746/36938 -- GLOBAL_STEP: 317250 | > loss_text_ce: 0.03661318123340607 (0.041306699423488843) | > loss_mel_ce: 3.071556568145752 (nan) | > loss: 0.012334007769823074 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13637987844619254) | > loader_time: 0.0046 (0.004658288924112761)  --> TIME: 2025-02-24 12:21:56 -- STEP: 21796/36938 -- GLOBAL_STEP: 317300 | > loss_text_ce: 0.02855035290122032 (0.04130615775288304) | > loss_mel_ce: 3.590366840362549 (nan) | > loss: 0.014360783621668816 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1363729864090905) | > loader_time: 0.0067 (0.004658297587587407)  --> TIME: 2025-02-24 12:22:05 -- STEP: 21846/36938 -- GLOBAL_STEP: 317350 | > loss_text_ce: 0.030701512470841408 (0.04130436532856919) | > loss_mel_ce: 2.887195348739624 (nan) | > loss: 0.011578955687582493 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13638000406210948) | > loader_time: 0.0045 (0.004658336627617606)  --> TIME: 2025-02-24 12:22:15 -- STEP: 21896/36938 -- GLOBAL_STEP: 317400 | > loss_text_ce: 0.04005340859293938 (0.04130147403387831) | > loss_mel_ce: 3.16141676902771 (nan) | > loss: 0.012704247608780861 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13638648765340602) | > loader_time: 0.0047 (0.004658319946183837)  --> TIME: 2025-02-24 12:22:24 -- STEP: 21946/36938 -- GLOBAL_STEP: 317450 | > loss_text_ce: 0.0472695492208004 (0.041300103843506054) | > loss_mel_ce: 3.106928586959839 (nan) | > loss: 0.012516659684479237 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1363900271303333) | > loader_time: 0.0045 (0.004658334009477679)  --> TIME: 2025-02-24 12:22:34 -- STEP: 21996/36938 -- GLOBAL_STEP: 317500 | > loss_text_ce: 0.05437943711876869 (0.04129775601801096) | > loss_mel_ce: 3.250122547149658 (nan) | > loss: 0.013113103806972504 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13640238623507198) | > loader_time: 0.0046 (0.004658410258240261)  --> TIME: 2025-02-24 12:22:43 -- STEP: 22046/36938 -- GLOBAL_STEP: 317550 | > loss_text_ce: 0.03921349346637726 (0.0412988558222849) | > loss_mel_ce: 3.0825588703155518 (nan) | > loss: 0.012387986294925213 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13640876996433887) | > loader_time: 0.0045 (0.004658433796867837)  --> TIME: 2025-02-24 12:22:52 -- STEP: 22096/36938 -- GLOBAL_STEP: 317600 | > loss_text_ce: 0.03552883863449097 (0.04129975317868787) | > loss_mel_ce: 2.4840078353881836 (nan) | > loss: 0.009998162277042866 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13640380913498248) | > loader_time: 0.0046 (0.004658335678220058)  --> TIME: 2025-02-24 12:23:02 -- STEP: 22146/36938 -- GLOBAL_STEP: 317650 | > loss_text_ce: 0.03216182440519333 (0.04129766141096673) | > loss_mel_ce: 4.791666507720947 (nan) | > loss: 0.01914217695593834 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13641053883350915) | > loader_time: 0.0045 (0.004658313729000173)  --> TIME: 2025-02-24 12:23:11 -- STEP: 22196/36938 -- GLOBAL_STEP: 317700 | > loss_text_ce: 0.031211283057928085 (0.04129688359681526) | > loss_mel_ce: 1.6948096752166748 (nan) | > loss: 0.006849289871752262 (nan) | > current_lr: 5e-06 | > step_time: 0.1072 (0.13640286087194914) | > loader_time: 0.0044 (0.004658118306591528)  --> TIME: 2025-02-24 12:23:20 -- STEP: 22246/36938 -- GLOBAL_STEP: 317750 | > loss_text_ce: 0.04303096979856491 (0.04129182812334534) | > loss_mel_ce: 3.411595344543457 (nan) | > loss: 0.01370883546769619 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.1363951738903355) | > loader_time: 0.0048 (0.004657990767631477)  --> TIME: 2025-02-24 12:23:30 -- STEP: 22296/36938 -- GLOBAL_STEP: 317800 | > loss_text_ce: 0.047207534313201904 (0.04129381906551952) | > loss_mel_ce: 2.9328081607818604 (nan) | > loss: 0.011825460009276867 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13640232042552808) | > loader_time: 0.0046 (0.004658013870949646)  --> TIME: 2025-02-24 12:23:39 -- STEP: 22346/36938 -- GLOBAL_STEP: 317850 | > loss_text_ce: 0.04065671190619469 (0.04129216665645281) | > loss_mel_ce: 3.8369140625 (nan) | > loss: 0.015387186780571938 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13641372564964988) | > loader_time: 0.0046 (0.0046579834918332305)  --> TIME: 2025-02-24 12:23:48 -- STEP: 22396/36938 -- GLOBAL_STEP: 317900 | > loss_text_ce: 0.04062177985906601 (0.04129450701872074) | > loss_mel_ce: 0.013331095688045025 (nan) | > loss: 0.00021409873443190008 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13640376186132364) | > loader_time: 0.0045 (0.004657882817154766)  --> TIME: 2025-02-24 12:23:58 -- STEP: 22446/36938 -- GLOBAL_STEP: 317950 | > loss_text_ce: 0.04816789925098419 (0.041290314907011745) | > loss_mel_ce: 0.5943756103515625 (nan) | > loss: 0.0025497758761048317 (nan) | > current_lr: 5e-06 | > step_time: 0.1118 (0.1364098561699137) | > loader_time: 0.0045 (0.004657840214661078)  --> TIME: 2025-02-24 12:24:07 -- STEP: 22496/36938 -- GLOBAL_STEP: 318000 | > loss_text_ce: 0.04179446026682854 (0.04128593115301957) | > loss_mel_ce: 2.283914804458618 (nan) | > loss: 0.009229006245732307 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.1364095273452832) | > loader_time: 0.0045 (0.004657854449274197)  --> TIME: 2025-02-24 12:24:16 -- STEP: 22546/36938 -- GLOBAL_STEP: 318050 | > loss_text_ce: 0.04804474860429764 (0.04128547419389004) | > loss_mel_ce: 2.6206817626953125 (nan) | > loss: 0.010590184479951859 (nan) | > current_lr: 5e-06 | > step_time: 0.1758 (0.13639742725544513) | > loader_time: 0.0048 (0.004657693058556362)  --> TIME: 2025-02-24 12:24:26 -- STEP: 22596/36938 -- GLOBAL_STEP: 318100 | > loss_text_ce: 0.04010846093297005 (0.0412828426194491) | > loss_mel_ce: 3.8233652114868164 (nan) | > loss: 0.015331245958805084 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.1363968943933356) | > loader_time: 0.0047 (0.004657668579065774)  --> TIME: 2025-02-24 12:24:35 -- STEP: 22646/36938 -- GLOBAL_STEP: 318150 | > loss_text_ce: 0.04258660599589348 (0.04128484365583035) | > loss_mel_ce: 2.4940261840820312 (nan) | > loss: 0.010065924376249313 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13638950516517848) | > loader_time: 0.0046 (0.004657589703874621)  --> TIME: 2025-02-24 12:24:44 -- STEP: 22696/36938 -- GLOBAL_STEP: 318200 | > loss_text_ce: 0.02775319665670395 (0.04128214930340774) | > loss_mel_ce: 0.215087890625 (nan) | > loss: 0.0009636551840230823 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13638275709419423) | > loader_time: 0.0047 (0.004657474251586785)  --> TIME: 2025-02-24 12:24:53 -- STEP: 22746/36938 -- GLOBAL_STEP: 318250 | > loss_text_ce: 0.053708624094724655 (0.041278993501649354) | > loss_mel_ce: 2.477959394454956 (nan) | > loss: 0.01004630234092474 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.13636823687045632) | > loader_time: 0.0046 (0.004657430121779789)  --> TIME: 2025-02-24 12:25:02 -- STEP: 22796/36938 -- GLOBAL_STEP: 318300 | > loss_text_ce: 0.04596421495079994 (0.04127857327936138) | > loss_mel_ce: 3.504732847213745 (nan) | > loss: 0.014090068638324738 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1363527958362554) | > loader_time: 0.0045 (0.00465732308767787)  --> TIME: 2025-02-24 12:25:11 -- STEP: 22846/36938 -- GLOBAL_STEP: 318350 | > loss_text_ce: 0.03504597768187523 (0.041280509842063835) | > loss_mel_ce: 3.1368913650512695 (nan) | > loss: 0.012587052769958973 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13636639360724867) | > loader_time: 0.0046 (0.004657242528342185)  --> TIME: 2025-02-24 12:25:21 -- STEP: 22896/36938 -- GLOBAL_STEP: 318400 | > loss_text_ce: 0.030003804713487625 (0.04127692712991866) | > loss_mel_ce: 2.803151845932007 (nan) | > loss: 0.011242682114243507 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1363753340695802) | > loader_time: 0.0045 (0.004657226580160645)  --> TIME: 2025-02-24 12:25:30 -- STEP: 22946/36938 -- GLOBAL_STEP: 318450 | > loss_text_ce: 0.01567203551530838 (0.041275816130628216) | > loss_mel_ce: 2.2659912109375 (nan) | > loss: 0.009054219350218773 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13637384764248114) | > loader_time: 0.0045 (0.004657266996778183)  --> TIME: 2025-02-24 12:25:39 -- STEP: 22996/36938 -- GLOBAL_STEP: 318500 | > loss_text_ce: 0.032006751745939255 (0.041275166864297094) | > loss_mel_ce: 2.2988226413726807 (nan) | > loss: 0.009249323047697544 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13637186553420558) | > loader_time: 0.0046 (0.0046572242431753555)  --> TIME: 2025-02-24 12:25:49 -- STEP: 23046/36938 -- GLOBAL_STEP: 318550 | > loss_text_ce: 0.03889589384198189 (0.04127571434789628) | > loss_mel_ce: 3.7311816215515137 (nan) | > loss: 0.014960626140236855 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13637449086917636) | > loader_time: 0.0046 (0.004657284093898366)  --> TIME: 2025-02-24 12:25:58 -- STEP: 23096/36938 -- GLOBAL_STEP: 318600 | > loss_text_ce: 0.041728854179382324 (0.04127166509603456) | > loss_mel_ce: 1.2841806411743164 (nan) | > loss: 0.00526154600083828 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.1363771683231618) | > loader_time: 0.0045 (0.004657448607825643)  --> TIME: 2025-02-24 12:26:08 -- STEP: 23146/36938 -- GLOBAL_STEP: 318650 | > loss_text_ce: 0.039129626005887985 (0.041273031165829914) | > loss_mel_ce: 3.669523239135742 (nan) | > loss: 0.014716877602040768 (nan) | > current_lr: 5e-06 | > step_time: 0.1901 (0.13638665720821688) | > loader_time: 0.005 (0.004657382202016593)  --> TIME: 2025-02-24 12:26:18 -- STEP: 23196/36938 -- GLOBAL_STEP: 318700 | > loss_text_ce: 0.03859328106045723 (0.04126972999586911) | > loss_mel_ce: 2.335421085357666 (nan) | > loss: 0.009420692920684814 (nan) | > current_lr: 5e-06 | > step_time: 0.1822 (0.1364061405975706) | > loader_time: 0.0048 (0.004657507795859137)  --> TIME: 2025-02-24 12:26:27 -- STEP: 23246/36938 -- GLOBAL_STEP: 318750 | > loss_text_ce: 0.031834106892347336 (0.041262803718468175) | > loss_mel_ce: 1.0341033935546875 (nan) | > loss: 0.004229911137372255 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.136405685698115) | > loader_time: 0.0045 (0.004657457066230482)  --> TIME: 2025-02-24 12:26:36 -- STEP: 23296/36938 -- GLOBAL_STEP: 318800 | > loss_text_ce: 0.054404523223638535 (0.041262627314139044) | > loss_mel_ce: 3.670983076095581 (nan) | > loss: 0.01478328462690115 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.13640391299562435) | > loader_time: 0.0047 (0.0046573690046648455)  --> TIME: 2025-02-24 12:26:45 -- STEP: 23346/36938 -- GLOBAL_STEP: 318850 | > loss_text_ce: 0.03813236579298973 (0.04126391655883631) | > loss_mel_ce: 3.4986326694488525 (nan) | > loss: 0.014034782536327839 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13639435600235641) | > loader_time: 0.0045 (0.004657173548894259)  --> TIME: 2025-02-24 12:26:55 -- STEP: 23396/36938 -- GLOBAL_STEP: 318900 | > loss_text_ce: 0.044272832572460175 (0.041265067171749936) | > loss_mel_ce: 3.6163601875305176 (nan) | > loss: 0.014526322484016418 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.13640169981993797) | > loader_time: 0.0047 (0.004657305536320685)  --> TIME: 2025-02-24 12:27:04 -- STEP: 23446/36938 -- GLOBAL_STEP: 318950 | > loss_text_ce: 0.04040953516960144 (0.04126390569322909) | > loss_mel_ce: 3.729011058807373 (nan) | > loss: 0.014958019368350506 (nan) | > current_lr: 5e-06 | > step_time: 0.1812 (0.13640875924338242) | > loader_time: 0.0048 (0.004657236807563754)  --> TIME: 2025-02-24 12:27:13 -- STEP: 23496/36938 -- GLOBAL_STEP: 319000 | > loss_text_ce: 0.037788938730955124 (0.04126625800396509) | > loss_mel_ce: 2.718345880508423 (nan) | > loss: 0.010937043465673923 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13640672574247806) | > loader_time: 0.0046 (0.00465713998960483)  --> TIME: 2025-02-24 12:27:23 -- STEP: 23546/36938 -- GLOBAL_STEP: 319050 | > loss_text_ce: 0.039865024387836456 (0.041263708931244446) | > loss_mel_ce: 2.663612127304077 (nan) | > loss: 0.010728084482252598 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13640934853143447) | > loader_time: 0.0045 (0.004657065444112917)  --> TIME: 2025-02-24 12:27:32 -- STEP: 23596/36938 -- GLOBAL_STEP: 319100 | > loss_text_ce: 0.045326583087444305 (0.04126372346365612) | > loss_mel_ce: 3.0394742488861084 (nan) | > loss: 0.012241273187100887 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1364150003854206) | > loader_time: 0.0047 (0.00465699372038557)  --> TIME: 2025-02-24 12:27:42 -- STEP: 23646/36938 -- GLOBAL_STEP: 319150 | > loss_text_ce: 0.03740508481860161 (0.04125871048307725) | > loss_mel_ce: 1.7244689464569092 (nan) | > loss: 0.00699156429618597 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13642362456106727) | > loader_time: 0.0045 (0.004656922461306364)  --> TIME: 2025-02-24 12:27:51 -- STEP: 23696/36938 -- GLOBAL_STEP: 319200 | > loss_text_ce: 0.04738527536392212 (0.04126002726257966) | > loss_mel_ce: 3.1605300903320312 (nan) | > loss: 0.012729823589324951 (nan) | > current_lr: 5e-06 | > step_time: 0.1529 (0.13642774044380707) | > loader_time: 0.0047 (0.004656829508392501)  --> TIME: 2025-02-24 12:28:01 -- STEP: 23746/36938 -- GLOBAL_STEP: 319250 | > loss_text_ce: 0.03975536674261093 (0.04126025596158989) | > loss_mel_ce: 3.673753499984741 (nan) | > loss: 0.014736146666109562 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13643347380647827) | > loader_time: 0.0046 (0.004656796084692922)  --> TIME: 2025-02-24 12:28:10 -- STEP: 23796/36938 -- GLOBAL_STEP: 319300 | > loss_text_ce: 0.03567368537187576 (0.04125682715394492) | > loss_mel_ce: 3.1091296672821045 (nan) | > loss: 0.012479378841817379 (nan) | > current_lr: 5e-06 | > step_time: 0.1455 (0.13644186906162095) | > loader_time: 0.0047 (0.004656742813006478)  --> TIME: 2025-02-24 12:28:19 -- STEP: 23846/36938 -- GLOBAL_STEP: 319350 | > loss_text_ce: 0.05620839446783066 (0.04125785585741246) | > loss_mel_ce: 3.3540375232696533 (nan) | > loss: 0.013532722368836403 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.1364374638583586) | > loader_time: 0.0046 (0.004656642692896745)  --> TIME: 2025-02-24 12:28:29 -- STEP: 23896/36938 -- GLOBAL_STEP: 319400 | > loss_text_ce: 0.035296883434057236 (0.04126035293823304) | > loss_mel_ce: 3.6251800060272217 (nan) | > loss: 0.0145257031545043 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13644124924198678) | > loader_time: 0.0045 (0.004656557718327433)  --> TIME: 2025-02-24 12:28:38 -- STEP: 23946/36938 -- GLOBAL_STEP: 319450 | > loss_text_ce: 0.050302982330322266 (0.041259749238841015) | > loss_mel_ce: 3.6134848594665527 (nan) | > loss: 0.0145388413220644 (nan) | > current_lr: 5e-06 | > step_time: 0.1758 (0.1364423712423187) | > loader_time: 0.0048 (0.004656484897081135)  --> TIME: 2025-02-24 12:28:47 -- STEP: 23996/36938 -- GLOBAL_STEP: 319500 | > loss_text_ce: 0.031816449016332626 (0.04125810262178447) | > loss_mel_ce: 3.469391345977783 (nan) | > loss: 0.01389368250966072 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.1364493464843331) | > loader_time: 0.0045 (0.004656422821794639)  --> TIME: 2025-02-24 12:28:57 -- STEP: 24046/36938 -- GLOBAL_STEP: 319550 | > loss_text_ce: 0.04121588543057442 (0.0412572823472772) | > loss_mel_ce: 2.7658543586730957 (nan) | > loss: 0.01113916840404272 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.1364447830481462) | > loader_time: 0.0048 (0.004656300453123218)  --> TIME: 2025-02-24 12:29:06 -- STEP: 24096/36938 -- GLOBAL_STEP: 319600 | > loss_text_ce: 0.043853964656591415 (0.04125372210313763) | > loss_mel_ce: 3.9263203144073486 (nan) | > loss: 0.015754660591483116 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.1364520110120647) | > loader_time: 0.0047 (0.004656265802671558)  --> TIME: 2025-02-24 12:29:16 -- STEP: 24146/36938 -- GLOBAL_STEP: 319650 | > loss_text_ce: 0.03291130065917969 (0.04125423472848941) | > loss_mel_ce: 3.51092529296875 (nan) | > loss: 0.014062844216823578 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.13646152051492572) | > loader_time: 0.0047 (0.004656203273197192)  --> TIME: 2025-02-24 12:29:25 -- STEP: 24196/36938 -- GLOBAL_STEP: 319700 | > loss_text_ce: 0.04335075244307518 (0.04125515099685374) | > loss_mel_ce: 3.4842472076416016 (nan) | > loss: 0.013998405076563358 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13646932688010024) | > loader_time: 0.0046 (0.004656150836080626)  --> TIME: 2025-02-24 12:29:34 -- STEP: 24246/36938 -- GLOBAL_STEP: 319750 | > loss_text_ce: 0.052701037377119064 (0.0412519624332157) | > loss_mel_ce: 3.3083014488220215 (nan) | > loss: 0.013337312266230583 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13646552760953234) | > loader_time: 0.0045 (0.004656013694718514)  --> TIME: 2025-02-24 12:29:44 -- STEP: 24296/36938 -- GLOBAL_STEP: 319800 | > loss_text_ce: 0.036785900592803955 (0.04125265150722262) | > loss_mel_ce: 2.7501113414764404 (nan) | > loss: 0.011059116572141647 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.13646329084155473) | > loader_time: 0.0048 (0.004655907263597239)  --> TIME: 2025-02-24 12:29:53 -- STEP: 24346/36938 -- GLOBAL_STEP: 319850 | > loss_text_ce: 0.032988741993904114 (0.041250933060779246) | > loss_mel_ce: 3.3054094314575195 (nan) | > loss: 0.013247612863779068 (nan) | > current_lr: 5e-06 | > step_time: 0.1511 (0.1364687003625558) | > loader_time: 0.0046 (0.004655801367565824)  --> TIME: 2025-02-24 12:30:02 -- STEP: 24396/36938 -- GLOBAL_STEP: 319900 | > loss_text_ce: 0.03977557644248009 (0.041252917713258314) | > loss_mel_ce: 0.1304713636636734 (nan) | > loss: 0.0006755831418558955 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13646834454471488) | > loader_time: 0.0046 (0.0046557253609917195)  --> TIME: 2025-02-24 12:30:12 -- STEP: 24446/36938 -- GLOBAL_STEP: 319950 | > loss_text_ce: 0.033612728118896484 (0.04125187330453575) | > loss_mel_ce: 2.2566168308258057 (nan) | > loss: 0.009088212624192238 (nan) | > current_lr: 5e-06 | > step_time: 0.1632 (0.13647443024409697) | > loader_time: 0.0046 (0.004655643813613803)  --> TIME: 2025-02-24 12:30:21 -- STEP: 24496/36938 -- GLOBAL_STEP: 320000 | > loss_text_ce: 0.03722316026687622 (0.04125469942515742) | > loss_mel_ce: 1.3197970390319824 (nan) | > loss: 0.005385000724345446 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13648085311726713) | > loader_time: 0.0047 (0.004655606154131026) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_320000.pth  --> TIME: 2025-02-24 12:30:36 -- STEP: 24546/36938 -- GLOBAL_STEP: 320050 | > loss_text_ce: 0.038601137697696686 (0.04125662380598007) | > loss_mel_ce: 3.4702765941619873 (nan) | > loss: 0.01392411906272173 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13647161992499693) | > loader_time: 0.0047 (0.0046556505977811465)  --> TIME: 2025-02-24 12:30:45 -- STEP: 24596/36938 -- GLOBAL_STEP: 320100 | > loss_text_ce: 0.0412166602909565 (0.04125947316350912) | > loss_mel_ce: 1.6170076131820679 (nan) | > loss: 0.006580255459994078 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13645459672535734) | > loader_time: 0.0046 (0.0046556198339110485)  --> TIME: 2025-02-24 12:30:54 -- STEP: 24646/36938 -- GLOBAL_STEP: 320150 | > loss_text_ce: 0.04192323237657547 (0.04125776289920136) | > loss_mel_ce: 2.576122522354126 (nan) | > loss: 0.010389070957899094 (nan) | > current_lr: 5e-06 | > step_time: 0.1777 (0.13644998470974384) | > loader_time: 0.0048 (0.004655616668237481)  --> TIME: 2025-02-24 12:31:04 -- STEP: 24696/36938 -- GLOBAL_STEP: 320200 | > loss_text_ce: 0.04216707497835159 (0.04125908954545086) | > loss_mel_ce: 0.0344151072204113 (nan) | > loss: 0.00030389754101634026 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.13645390302647534) | > loader_time: 0.0045 (0.004655594689814402)  --> TIME: 2025-02-24 12:31:13 -- STEP: 24746/36938 -- GLOBAL_STEP: 320250 | > loss_text_ce: 0.03947683423757553 (0.04125844142889357) | > loss_mel_ce: 3.276644229888916 (nan) | > loss: 0.013159211724996567 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.13644849906248613) | > loader_time: 0.0048 (0.004655530022445678)  --> TIME: 2025-02-24 12:31:23 -- STEP: 24796/36938 -- GLOBAL_STEP: 320300 | > loss_text_ce: 0.04552247002720833 (0.04126157657648798) | > loss_mel_ce: 3.6291756629943848 (nan) | > loss: 0.014582136645913124 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.13646449390236876) | > loader_time: 0.0048 (0.004655646554768777)  --> TIME: 2025-02-24 12:31:32 -- STEP: 24846/36938 -- GLOBAL_STEP: 320350 | > loss_text_ce: 0.04021161422133446 (0.04126276405045242) | > loss_mel_ce: 3.3403964042663574 (nan) | > loss: 0.013415112160146236 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13646958146436475) | > loader_time: 0.0046 (0.004655726614430634)  --> TIME: 2025-02-24 12:31:41 -- STEP: 24896/36938 -- GLOBAL_STEP: 320400 | > loss_text_ce: 0.042526595294475555 (0.041259595509594536) | > loss_mel_ce: 2.7976529598236084 (nan) | > loss: 0.011270553804934025 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13646064820318698) | > loader_time: 0.0047 (0.00465580904353556)  --> TIME: 2025-02-24 12:31:50 -- STEP: 24946/36938 -- GLOBAL_STEP: 320450 | > loss_text_ce: 0.04924168065190315 (0.0412606480940943) | > loss_mel_ce: 2.4397575855255127 (nan) | > loss: 0.009876982308924198 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13645758151358117) | > loader_time: 0.0047 (0.004655797737866012)  --> TIME: 2025-02-24 12:32:00 -- STEP: 24996/36938 -- GLOBAL_STEP: 320500 | > loss_text_ce: 0.043198585510253906 (0.0412598491654844) | > loss_mel_ce: 3.359307289123535 (nan) | > loss: 0.013502008281648159 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13645752548580034) | > loader_time: 0.0045 (0.004655648897430178)  --> TIME: 2025-02-24 12:32:09 -- STEP: 25046/36938 -- GLOBAL_STEP: 320550 | > loss_text_ce: 0.04481733590364456 (0.04126016740364202) | > loss_mel_ce: 3.003500461578369 (nan) | > loss: 0.012096500024199486 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13646049947105765) | > loader_time: 0.0046 (0.004655526257985379)  --> TIME: 2025-02-24 12:32:19 -- STEP: 25096/36938 -- GLOBAL_STEP: 320600 | > loss_text_ce: 0.04427266865968704 (0.04125588592181941) | > loss_mel_ce: 3.5270397663116455 (nan) | > loss: 0.014171875081956387 (nan) | > current_lr: 5e-06 | > step_time: 0.1689 (0.1364606384129744) | > loader_time: 0.0047 (0.00465537576793981)  --> TIME: 2025-02-24 12:32:28 -- STEP: 25146/36938 -- GLOBAL_STEP: 320650 | > loss_text_ce: 0.03894105181097984 (0.04125617153740805) | > loss_mel_ce: 3.2801947593688965 (nan) | > loss: 0.013171174563467503 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13646318753598274) | > loader_time: 0.0046 (0.004655432778858119)  --> TIME: 2025-02-24 12:32:38 -- STEP: 25196/36938 -- GLOBAL_STEP: 320700 | > loss_text_ce: 0.04204403609037399 (0.04125578822202859) | > loss_mel_ce: 2.892937183380127 (nan) | > loss: 0.011646751314401627 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13647024992780618) | > loader_time: 0.0045 (0.004655439762070693)  --> TIME: 2025-02-24 12:32:47 -- STEP: 25246/36938 -- GLOBAL_STEP: 320750 | > loss_text_ce: 0.0487704873085022 (0.04125504693165293) | > loss_mel_ce: 3.769205331802368 (nan) | > loss: 0.015150697901844978 (nan) | > current_lr: 5e-06 | > step_time: 0.1707 (0.13646946787824668) | > loader_time: 0.0048 (0.0046553601272735605)  --> TIME: 2025-02-24 12:32:57 -- STEP: 25296/36938 -- GLOBAL_STEP: 320800 | > loss_text_ce: 0.04699521139264107 (0.04125294699360112) | > loss_mel_ce: 3.4584927558898926 (nan) | > loss: 0.013910667039453983 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.1364809237255612) | > loader_time: 0.0045 (0.004655418329643071)  --> TIME: 2025-02-24 12:33:06 -- STEP: 25346/36938 -- GLOBAL_STEP: 320850 | > loss_text_ce: 0.038734320551157 (0.04125066149811095) | > loss_mel_ce: 3.4459683895111084 (nan) | > loss: 0.013828186318278313 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1364869524796131) | > loader_time: 0.0045 (0.004655381710049807)  --> TIME: 2025-02-24 12:33:15 -- STEP: 25396/36938 -- GLOBAL_STEP: 320900 | > loss_text_ce: 0.0475006029009819 (0.04125091253751513) | > loss_mel_ce: 3.110699415206909 (nan) | > loss: 0.01253254059702158 (nan) | > current_lr: 5e-06 | > step_time: 0.1501 (0.13648868690496058) | > loader_time: 0.0048 (0.00465532674021865)  --> TIME: 2025-02-24 12:33:24 -- STEP: 25446/36938 -- GLOBAL_STEP: 320950 | > loss_text_ce: 0.04093754664063454 (0.041248374070834835) | > loss_mel_ce: 0.7750105857849121 (nan) | > loss: 0.003237889613956213 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1364805312648762) | > loader_time: 0.0046 (0.004655212573844739)  --> TIME: 2025-02-24 12:33:34 -- STEP: 25496/36938 -- GLOBAL_STEP: 321000 | > loss_text_ce: 0.04448997229337692 (0.04125021930736058) | > loss_mel_ce: 2.72998046875 (nan) | > loss: 0.011009803973138332 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.1364933332006941) | > loader_time: 0.0045 (0.004655225246275057)  --> TIME: 2025-02-24 12:33:43 -- STEP: 25546/36938 -- GLOBAL_STEP: 321050 | > loss_text_ce: 0.058533381670713425 (0.04124827347045436) | > loss_mel_ce: 3.7313616275787354 (nan) | > loss: 0.015039267018437386 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.1364954567119819) | > loader_time: 0.0047 (0.0046552406129752874)  --> TIME: 2025-02-24 12:33:53 -- STEP: 25596/36938 -- GLOBAL_STEP: 321100 | > loss_text_ce: 0.04272616282105446 (0.04124789500471062) | > loss_mel_ce: 2.33784818649292 (nan) | > loss: 0.009446724317967892 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1364920981732502) | > loader_time: 0.0045 (0.00465524852378312)  --> TIME: 2025-02-24 12:34:02 -- STEP: 25646/36938 -- GLOBAL_STEP: 321150 | > loss_text_ce: 0.043172918260097504 (0.041247884230081716) | > loss_mel_ce: 3.0915067195892334 (nan) | > loss: 0.012439205311238766 (nan) | > current_lr: 5e-06 | > step_time: 0.173 (0.13650150272298225) | > loader_time: 0.0048 (0.00465526030828363)  --> TIME: 2025-02-24 12:34:11 -- STEP: 25696/36938 -- GLOBAL_STEP: 321200 | > loss_text_ce: 0.038484640419483185 (0.04124534775766541) | > loss_mel_ce: 3.8828799724578857 (nan) | > loss: 0.015560971572995186 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.13649737181728772) | > loader_time: 0.0049 (0.0046551863977205635)  --> TIME: 2025-02-24 12:34:21 -- STEP: 25746/36938 -- GLOBAL_STEP: 321250 | > loss_text_ce: 0.03532703220844269 (0.04124462632101367) | > loss_mel_ce: 3.0146799087524414 (nan) | > loss: 0.012103202752768993 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13650526035862628) | > loader_time: 0.0047 (0.004655201359338955)  --> TIME: 2025-02-24 12:34:30 -- STEP: 25796/36938 -- GLOBAL_STEP: 321300 | > loss_text_ce: 0.032555822283029556 (0.04124314240966283) | > loss_mel_ce: 3.3006591796875 (nan) | > loss: 0.013227044604718685 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13649875365554595) | > loader_time: 0.0045 (0.0046551517598073505)  --> TIME: 2025-02-24 12:34:39 -- STEP: 25846/36938 -- GLOBAL_STEP: 321350 | > loss_text_ce: 0.051185786724090576 (0.041245871153243886) | > loss_mel_ce: 3.1487739086151123 (nan) | > loss: 0.012698253616690636 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1364980633299586) | > loader_time: 0.0048 (0.004655102905654838)  --> TIME: 2025-02-24 12:34:49 -- STEP: 25896/36938 -- GLOBAL_STEP: 321400 | > loss_text_ce: 0.042765144258737564 (0.04124283177618728) | > loss_mel_ce: 3.080946207046509 (nan) | > loss: 0.012395680882036686 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13649381853232626) | > loader_time: 0.0046 (0.0046550256439223305)  --> TIME: 2025-02-24 12:34:58 -- STEP: 25946/36938 -- GLOBAL_STEP: 321450 | > loss_text_ce: 0.04292178899049759 (0.04123958221185943) | > loss_mel_ce: 3.874940872192383 (nan) | > loss: 0.015547075308859348 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.13649621922664867) | > loader_time: 0.0046 (0.00465504812365426)  --> TIME: 2025-02-24 12:35:07 -- STEP: 25996/36938 -- GLOBAL_STEP: 321500 | > loss_text_ce: 0.0420871376991272 (0.041236424306472015) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13649769441258544) | > loader_time: 0.0046 (0.004655047863662251)  --> TIME: 2025-02-24 12:35:17 -- STEP: 26046/36938 -- GLOBAL_STEP: 321550 | > loss_text_ce: 0.04265390336513519 (0.041236526372199614) | > loss_mel_ce: 3.3639423847198486 (nan) | > loss: 0.013518239371478558 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13649035644340804) | > loader_time: 0.0045 (0.004654979335999055)  --> TIME: 2025-02-24 12:35:26 -- STEP: 26096/36938 -- GLOBAL_STEP: 321600 | > loss_text_ce: 0.041657138615846634 (0.04123712786496741) | > loss_mel_ce: 3.3076066970825195 (nan) | > loss: 0.01329073030501604 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13649197410652741) | > loader_time: 0.0047 (0.004654958396510527)  --> TIME: 2025-02-24 12:35:35 -- STEP: 26146/36938 -- GLOBAL_STEP: 321650 | > loss_text_ce: 0.04424482583999634 (0.04123849992558457) | > loss_mel_ce: 2.9231507778167725 (nan) | > loss: 0.011775380000472069 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13648766348067018) | > loader_time: 0.0046 (0.004654861377391313)  --> TIME: 2025-02-24 12:35:44 -- STEP: 26196/36938 -- GLOBAL_STEP: 321700 | > loss_text_ce: 0.05332868918776512 (0.0412396364777075) | > loss_mel_ce: 1.9217219352722168 (nan) | > loss: 0.007837502285838127 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.1364881001568724) | > loader_time: 0.0045 (0.0046548081237972605)  --> TIME: 2025-02-24 12:35:54 -- STEP: 26246/36938 -- GLOBAL_STEP: 321750 | > loss_text_ce: 0.039672888815402985 (0.04123772716300563) | > loss_mel_ce: 2.7479054927825928 (nan) | > loss: 0.011061819270253181 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13648453504040106) | > loader_time: 0.0045 (0.004654713304887831)  --> TIME: 2025-02-24 12:36:03 -- STEP: 26296/36938 -- GLOBAL_STEP: 321800 | > loss_text_ce: 0.036029402166604996 (0.041238105316624536) | > loss_mel_ce: 3.2180891036987305 (nan) | > loss: 0.012913169339299202 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13647885387125838) | > loader_time: 0.0048 (0.004654708661531171)  --> TIME: 2025-02-24 12:36:12 -- STEP: 26346/36938 -- GLOBAL_STEP: 321850 | > loss_text_ce: 0.04002772271633148 (0.041236248212358606) | > loss_mel_ce: 3.5449936389923096 (nan) | > loss: 0.014226275496184826 (nan) | > current_lr: 5e-06 | > step_time: 0.1752 (0.13648312901578136) | > loader_time: 0.0048 (0.004654756233414009)  --> TIME: 2025-02-24 12:36:22 -- STEP: 26396/36938 -- GLOBAL_STEP: 321900 | > loss_text_ce: 0.041148360818624496 (0.041239522542600646) | > loss_mel_ce: 3.6197917461395264 (nan) | > loss: 0.01452754158526659 (nan) | > current_lr: 5e-06 | > step_time: 0.1448 (0.1364832253939523) | > loader_time: 0.0046 (0.004654716589104377)  --> TIME: 2025-02-24 12:36:31 -- STEP: 26446/36938 -- GLOBAL_STEP: 321950 | > loss_text_ce: 0.04139433428645134 (0.04123971040881752) | > loss_mel_ce: 2.809461832046509 (nan) | > loss: 0.011312921531498432 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13647870435788959) | > loader_time: 0.0046 (0.004654659731236841)  --> TIME: 2025-02-24 12:36:40 -- STEP: 26496/36938 -- GLOBAL_STEP: 322000 | > loss_text_ce: 0.03752859681844711 (0.04123899884688425) | > loss_mel_ce: 3.318145275115967 (nan) | > loss: 0.013316166587173939 (nan) | > current_lr: 5e-06 | > step_time: 0.1789 (0.13648713084954503) | > loader_time: 0.0048 (0.004654652272589568)  --> TIME: 2025-02-24 12:36:49 -- STEP: 26546/36938 -- GLOBAL_STEP: 322050 | > loss_text_ce: 0.04658246412873268 (0.04123766015638873) | > loss_mel_ce: 2.495534658432007 (nan) | > loss: 0.01008776668459177 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13646812877433673) | > loader_time: 0.0045 (0.004654497467276139)  --> TIME: 2025-02-24 12:36:58 -- STEP: 26596/36938 -- GLOBAL_STEP: 322100 | > loss_text_ce: 0.03651648387312889 (0.04123934288652757) | > loss_mel_ce: 1.8930109739303589 (nan) | > loss: 0.007656855043023825 (nan) | > current_lr: 5e-06 | > step_time: 0.1445 (0.13646494232455367) | > loader_time: 0.0047 (0.004654434851765359)  --> TIME: 2025-02-24 12:37:07 -- STEP: 26646/36938 -- GLOBAL_STEP: 322150 | > loss_text_ce: 0.052079133689403534 (0.04123901767122281) | > loss_mel_ce: 3.3739991188049316 (nan) | > loss: 0.013595549389719963 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13646204232865283) | > loader_time: 0.0066 (0.00465446938303763)  --> TIME: 2025-02-24 12:37:17 -- STEP: 26696/36938 -- GLOBAL_STEP: 322200 | > loss_text_ce: 0.0334603376686573 (0.04123709154903331) | > loss_mel_ce: 2.983466863632202 (nan) | > loss: 0.011971933767199516 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1364656965068827) | > loader_time: 0.0046 (0.004654460854254577)  --> TIME: 2025-02-24 12:37:26 -- STEP: 26746/36938 -- GLOBAL_STEP: 322250 | > loss_text_ce: 0.03864024579524994 (0.04123780432862061) | > loss_mel_ce: 3.0903406143188477 (nan) | > loss: 0.012416591867804527 (nan) | > current_lr: 5e-06 | > step_time: 0.1844 (0.13646569331706856) | > loader_time: 0.0051 (0.004654422575094102)  --> TIME: 2025-02-24 12:37:36 -- STEP: 26796/36938 -- GLOBAL_STEP: 322300 | > loss_text_ce: 0.037039536982774734 (0.041237777697720655) | > loss_mel_ce: 3.66048526763916 (nan) | > loss: 0.0146727180108428 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.13647195847644072) | > loader_time: 0.0047 (0.004654516700773695)  --> TIME: 2025-02-24 12:37:45 -- STEP: 26846/36938 -- GLOBAL_STEP: 322350 | > loss_text_ce: 0.03322717174887657 (0.041236876419886256) | > loss_mel_ce: 3.3658878803253174 (nan) | > loss: 0.013488552533090115 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.13646086866379734) | > loader_time: 0.0048 (0.004654473016152486)  --> TIME: 2025-02-24 12:37:54 -- STEP: 26896/36938 -- GLOBAL_STEP: 322400 | > loss_text_ce: 0.0466233491897583 (0.04123948586963119) | > loss_mel_ce: 2.0659120082855225 (nan) | > loss: 0.008383077569305897 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13646577236589919) | > loader_time: 0.0045 (0.004654466893118407)  --> TIME: 2025-02-24 12:38:03 -- STEP: 26946/36938 -- GLOBAL_STEP: 322450 | > loss_text_ce: 0.032408297061920166 (0.041238140910330186) | > loss_mel_ce: 1.0260035991668701 (nan) | > loss: 0.004200047347694635 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13646511115608723) | > loader_time: 0.0048 (0.004654381974700996)  --> TIME: 2025-02-24 12:38:13 -- STEP: 26996/36938 -- GLOBAL_STEP: 322500 | > loss_text_ce: 0.034994132816791534 (0.04123840569165927) | > loss_mel_ce: 3.435468912124634 (nan) | > loss: 0.013771679252386093 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13646653772795794) | > loader_time: 0.0045 (0.004654300603218006)  --> TIME: 2025-02-24 12:38:22 -- STEP: 27046/36938 -- GLOBAL_STEP: 322550 | > loss_text_ce: 0.0405103825032711 (0.041239082521324986) | > loss_mel_ce: 1.9774107933044434 (nan) | > loss: 0.008007624186575413 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13646312944901176) | > loader_time: 0.0044 (0.0046544073249147454)  --> TIME: 2025-02-24 12:38:31 -- STEP: 27096/36938 -- GLOBAL_STEP: 322600 | > loss_text_ce: 0.03812597692012787 (0.04123971727271105) | > loss_mel_ce: 3.483748197555542 (nan) | > loss: 0.013975691981613636 (nan) | > current_lr: 5e-06 | > step_time: 0.1651 (0.13645945593675074) | > loader_time: 0.0047 (0.00465431804145937)  --> TIME: 2025-02-24 12:38:40 -- STEP: 27146/36938 -- GLOBAL_STEP: 322650 | > loss_text_ce: 0.037677422165870667 (0.04123938757991269) | > loss_mel_ce: 3.1081361770629883 (nan) | > loss: 0.012483388185501099 (nan) | > current_lr: 5e-06 | > step_time: 0.1968 (0.13645919349632124) | > loader_time: 0.0048 (0.004654296091080157)  --> TIME: 2025-02-24 12:38:50 -- STEP: 27196/36938 -- GLOBAL_STEP: 322700 | > loss_text_ce: 0.04819454997777939 (0.04124080831708788) | > loss_mel_ce: 3.5821051597595215 (nan) | > loss: 0.014405952766537666 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13645624944367304) | > loader_time: 0.005 (0.0046542372347976)  --> TIME: 2025-02-24 12:38:59 -- STEP: 27246/36938 -- GLOBAL_STEP: 322750 | > loss_text_ce: 0.04242414981126785 (0.04123934062496141) | > loss_mel_ce: 3.3190507888793945 (nan) | > loss: 0.013339187018573284 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13645296994437234) | > loader_time: 0.0046 (0.004654227055301333)  --> TIME: 2025-02-24 12:39:08 -- STEP: 27296/36938 -- GLOBAL_STEP: 322800 | > loss_text_ce: 0.03451438248157501 (0.041238570059487195) | > loss_mel_ce: 2.0758819580078125 (nan) | > loss: 0.008374589495360851 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13645649198246115) | > loader_time: 0.0046 (0.004654285846254071)  --> TIME: 2025-02-24 12:39:18 -- STEP: 27346/36938 -- GLOBAL_STEP: 322850 | > loss_text_ce: 0.038471244275569916 (0.041232551328609236) | > loss_mel_ce: 3.6430411338806152 (nan) | > loss: 0.01460917666554451 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.1364548530835282) | > loader_time: 0.0046 (0.004654265196384618)  --> TIME: 2025-02-24 12:39:27 -- STEP: 27396/36938 -- GLOBAL_STEP: 322900 | > loss_text_ce: 0.03911585360765457 (0.041233637326417764) | > loss_mel_ce: 2.8145103454589844 (nan) | > loss: 0.011323913931846619 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13645477245490598) | > loader_time: 0.0046 (0.004654280076602706)  --> TIME: 2025-02-24 12:39:37 -- STEP: 27446/36938 -- GLOBAL_STEP: 322950 | > loss_text_ce: 0.05193344131112099 (0.041233221173389824) | > loss_mel_ce: 2.531583070755005 (nan) | > loss: 0.010252050124108791 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13647142525430808) | > loader_time: 0.0046 (0.0046542623443870616)  --> TIME: 2025-02-24 12:39:46 -- STEP: 27496/36938 -- GLOBAL_STEP: 323000 | > loss_text_ce: 0.04008420556783676 (0.04123457167642407) | > loss_mel_ce: 3.4108645915985107 (nan) | > loss: 0.013694241642951965 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13646662691356026) | > loader_time: 0.0046 (0.004654289202389088)  --> TIME: 2025-02-24 12:39:56 -- STEP: 27546/36938 -- GLOBAL_STEP: 323050 | > loss_text_ce: 0.05295608937740326 (0.04123431627257869) | > loss_mel_ce: 3.0344271659851074 (nan) | > loss: 0.012251521460711956 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.1364754942138059) | > loader_time: 0.0048 (0.004654343175116031)  --> TIME: 2025-02-24 12:40:05 -- STEP: 27596/36938 -- GLOBAL_STEP: 323100 | > loss_text_ce: 0.03277882933616638 (0.04123558503825898) | > loss_mel_ce: 3.703695774078369 (nan) | > loss: 0.014827280305325985 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13647961505065148) | > loader_time: 0.0045 (0.004654397142332613)  --> TIME: 2025-02-24 12:40:15 -- STEP: 27646/36938 -- GLOBAL_STEP: 323150 | > loss_text_ce: 0.048121120780706406 (0.041235587130833126) | > loss_mel_ce: 3.2538561820983887 (nan) | > loss: 0.013103085570037365 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13648698352002883) | > loader_time: 0.0048 (0.004654400360564446)  --> TIME: 2025-02-24 12:40:24 -- STEP: 27696/36938 -- GLOBAL_STEP: 323200 | > loss_text_ce: 0.03712194040417671 (0.04123146099874107) | > loss_mel_ce: 2.9623966217041016 (nan) | > loss: 0.011902851983904839 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.136490810630226) | > loader_time: 0.0045 (0.004654478348451589)  --> TIME: 2025-02-24 12:40:33 -- STEP: 27746/36938 -- GLOBAL_STEP: 323250 | > loss_text_ce: 0.03128879889845848 (0.04123021870851378) | > loss_mel_ce: 3.4470486640930176 (nan) | > loss: 0.013802927918732166 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13648914167740553) | > loader_time: 0.0046 (0.004654511741680001)  --> TIME: 2025-02-24 12:40:42 -- STEP: 27796/36938 -- GLOBAL_STEP: 323300 | > loss_text_ce: 0.05592450872063637 (0.04122720663218637) | > loss_mel_ce: 3.2084248065948486 (nan) | > loss: 0.01295376755297184 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13648766057064796) | > loader_time: 0.0046 (0.004654469353155316)  --> TIME: 2025-02-24 12:40:52 -- STEP: 27846/36938 -- GLOBAL_STEP: 323350 | > loss_text_ce: 0.049078360199928284 (0.04122647056646375) | > loss_mel_ce: 3.3175649642944336 (nan) | > loss: 0.013359696604311466 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13649632637420073) | > loader_time: 0.0046 (0.004654485895263325)  --> TIME: 2025-02-24 12:41:02 -- STEP: 27896/36938 -- GLOBAL_STEP: 323400 | > loss_text_ce: 0.03283287584781647 (0.041226391911733364) | > loss_mel_ce: 3.6104726791381836 (nan) | > loss: 0.014457562938332558 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13650989078641207) | > loader_time: 0.0046 (0.004654515924582299)  --> TIME: 2025-02-24 12:41:11 -- STEP: 27946/36938 -- GLOBAL_STEP: 323450 | > loss_text_ce: 0.04024222120642662 (0.041224248295420964) | > loss_mel_ce: 2.8664281368255615 (nan) | > loss: 0.011534406803548336 (nan) | > current_lr: 5e-06 | > step_time: 0.1791 (0.13651502457462178) | > loader_time: 0.0048 (0.004654714341649264)  --> TIME: 2025-02-24 12:41:21 -- STEP: 27996/36938 -- GLOBAL_STEP: 323500 | > loss_text_ce: 0.04800490289926529 (0.041223753790948184) | > loss_mel_ce: 2.967496633529663 (nan) | > loss: 0.011966276913881302 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.1365132031653295) | > loader_time: 0.0045 (0.0046547179239820775)  --> TIME: 2025-02-24 12:41:30 -- STEP: 28046/36938 -- GLOBAL_STEP: 323550 | > loss_text_ce: 0.04623890668153763 (0.041224102082499596) | > loss_mel_ce: 3.411837339401245 (nan) | > loss: 0.013722525909543037 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13650168505798296) | > loader_time: 0.0045 (0.004654550725788246)  --> TIME: 2025-02-24 12:41:39 -- STEP: 28096/36938 -- GLOBAL_STEP: 323600 | > loss_text_ce: 0.042494501918554306 (0.041221985967445536) | > loss_mel_ce: 2.8903656005859375 (nan) | > loss: 0.011638334952294827 (nan) | > current_lr: 5e-06 | > step_time: 0.1812 (0.1364955500039802) | > loader_time: 0.005 (0.004654451271260267)  --> TIME: 2025-02-24 12:41:48 -- STEP: 28146/36938 -- GLOBAL_STEP: 323650 | > loss_text_ce: 0.044920824468135834 (0.04122080330887883) | > loss_mel_ce: 3.186439037322998 (nan) | > loss: 0.012822858057916164 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13649816878321538) | > loader_time: 0.0045 (0.0046543610389914065)  --> TIME: 2025-02-24 12:41:57 -- STEP: 28196/36938 -- GLOBAL_STEP: 323700 | > loss_text_ce: 0.043780744075775146 (0.04121887061461788) | > loss_mel_ce: 3.2310075759887695 (nan) | > loss: 0.012995192781090736 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1365043442373426) | > loader_time: 0.0045 (0.004654373847300139)  --> TIME: 2025-02-24 12:42:07 -- STEP: 28246/36938 -- GLOBAL_STEP: 323750 | > loss_text_ce: 0.046983517706394196 (0.041216055037419305) | > loss_mel_ce: 3.135974407196045 (nan) | > loss: 0.012630785815417767 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13650470330138983) | > loader_time: 0.0046 (0.004654367154240334)  --> TIME: 2025-02-24 12:42:16 -- STEP: 28296/36938 -- GLOBAL_STEP: 323800 | > loss_text_ce: 0.0426388718187809 (0.04121813198616964) | > loss_mel_ce: 3.022524118423462 (nan) | > loss: 0.012163345701992512 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.13650107486523697) | > loader_time: 0.0049 (0.004654405790767955)  --> TIME: 2025-02-24 12:42:25 -- STEP: 28346/36938 -- GLOBAL_STEP: 323850 | > loss_text_ce: 0.03633620962500572 (0.04122040880869489) | > loss_mel_ce: 3.547142744064331 (nan) | > loss: 0.014220154844224453 (nan) | > current_lr: 5e-06 | > step_time: 0.1791 (0.13650397949747622) | > loader_time: 0.0049 (0.004654343880317164)  --> TIME: 2025-02-24 12:42:35 -- STEP: 28396/36938 -- GLOBAL_STEP: 323900 | > loss_text_ce: 0.035783689469099045 (0.04121886143266835) | > loss_mel_ce: 1.8187519311904907 (nan) | > loss: 0.007359268609434366 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1365036370720992) | > loader_time: 0.0045 (0.00465429146569589)  --> TIME: 2025-02-24 12:42:44 -- STEP: 28446/36938 -- GLOBAL_STEP: 323950 | > loss_text_ce: 0.04355541244149208 (0.04121822883392468) | > loss_mel_ce: 1.293626308441162 (nan) | > loss: 0.005306276958435774 (nan) | > current_lr: 5e-06 | > step_time: 0.1078 (0.136508987915966) | > loader_time: 0.0044 (0.004654371217946136)  --> TIME: 2025-02-24 12:42:53 -- STEP: 28496/36938 -- GLOBAL_STEP: 324000 | > loss_text_ce: 0.03069121763110161 (0.04121912365907) | > loss_mel_ce: 0.20085261762142181 (nan) | > loss: 0.0009188248077407479 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13650179662308226) | > loader_time: 0.0046 (0.004654274277644256)  --> TIME: 2025-02-24 12:43:03 -- STEP: 28546/36938 -- GLOBAL_STEP: 324050 | > loss_text_ce: 0.04114016145467758 (0.04121816200280207) | > loss_mel_ce: 1.9004415273666382 (nan) | > loss: 0.007704689633101225 (nan) | > current_lr: 5e-06 | > step_time: 0.1505 (0.1365054826850708) | > loader_time: 0.0048 (0.004654206466567919)  --> TIME: 2025-02-24 12:43:12 -- STEP: 28596/36938 -- GLOBAL_STEP: 324100 | > loss_text_ce: 0.04314242675900459 (0.0412190588017432) | > loss_mel_ce: 3.0926477909088135 (nan) | > loss: 0.012443612329661846 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13650769695633355) | > loader_time: 0.0046 (0.004654105834518071)  --> TIME: 2025-02-24 12:43:21 -- STEP: 28646/36938 -- GLOBAL_STEP: 324150 | > loss_text_ce: 0.04819425940513611 (0.04121535827686581) | > loss_mel_ce: 3.505878210067749 (nan) | > loss: 0.014103462919592857 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.136510165869786) | > loader_time: 0.0046 (0.004654090639050614)  --> TIME: 2025-02-24 12:43:31 -- STEP: 28696/36938 -- GLOBAL_STEP: 324200 | > loss_text_ce: 0.04885271564126015 (0.04121452591294075) | > loss_mel_ce: 3.106402635574341 (nan) | > loss: 0.01252085529267788 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.1365168235462824) | > loader_time: 0.0047 (0.004654154202251492)  --> TIME: 2025-02-24 12:43:40 -- STEP: 28746/36938 -- GLOBAL_STEP: 324250 | > loss_text_ce: 0.03529229387640953 (0.04121907200124003) | > loss_mel_ce: 3.950491189956665 (nan) | > loss: 0.01581660285592079 (nan) | > current_lr: 5e-06 | > step_time: 0.1349 (0.13651419395877296) | > loader_time: 0.0047 (0.004654219352418563)  --> TIME: 2025-02-24 12:43:49 -- STEP: 28796/36938 -- GLOBAL_STEP: 324300 | > loss_text_ce: 0.05009106546640396 (0.04121878127828014) | > loss_mel_ce: 3.3087573051452637 (nan) | > loss: 0.013328763656318188 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13651478901590375) | > loader_time: 0.0046 (0.00465425716901692)  --> TIME: 2025-02-24 12:43:59 -- STEP: 28846/36938 -- GLOBAL_STEP: 324350 | > loss_text_ce: 0.03619369491934776 (0.041216403911760485) | > loss_mel_ce: 3.23372220993042 (nan) | > loss: 0.012975857593119144 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13650822460936599) | > loader_time: 0.0045 (0.0046541659666531204)  --> TIME: 2025-02-24 12:44:08 -- STEP: 28896/36938 -- GLOBAL_STEP: 324400 | > loss_text_ce: 0.032368991523981094 (0.0412131772550446) | > loss_mel_ce: 3.289642333984375 (nan) | > loss: 0.013182585127651691 (nan) | > current_lr: 5e-06 | > step_time: 0.2083 (0.13650776518473773) | > loader_time: 0.0048 (0.004654161937345059)  --> TIME: 2025-02-24 12:44:17 -- STEP: 28946/36938 -- GLOBAL_STEP: 324450 | > loss_text_ce: 0.04262346774339676 (0.041212463607840526) | > loss_mel_ce: 2.6992132663726807 (nan) | > loss: 0.0108803054317832 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13650722034336835) | > loader_time: 0.0045 (0.004654030887833353)  --> TIME: 2025-02-24 12:44:26 -- STEP: 28996/36938 -- GLOBAL_STEP: 324500 | > loss_text_ce: 0.03736585006117821 (0.04120838522191217) | > loss_mel_ce: 2.415154457092285 (nan) | > loss: 0.00973222404718399 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13650324876135156) | > loader_time: 0.0047 (0.00465399391948151)  --> TIME: 2025-02-24 12:44:36 -- STEP: 29046/36938 -- GLOBAL_STEP: 324550 | > loss_text_ce: 0.048093147575855255 (0.041208448578664895) | > loss_mel_ce: 3.4636106491088867 (nan) | > loss: 0.013935333117842674 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.136508300671653) | > loader_time: 0.0046 (0.004653970170659685)  --> TIME: 2025-02-24 12:44:45 -- STEP: 29096/36938 -- GLOBAL_STEP: 324600 | > loss_text_ce: 0.04840916022658348 (0.04121071491748319) | > loss_mel_ce: 3.011077404022217 (nan) | > loss: 0.012140820734202862 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13650989978235475) | > loader_time: 0.0045 (0.004653957983540893)  --> TIME: 2025-02-24 12:44:54 -- STEP: 29146/36938 -- GLOBAL_STEP: 324650 | > loss_text_ce: 0.040330495685338974 (0.041211775460755336) | > loss_mel_ce: 3.1619760990142822 (nan) | > loss: 0.012707565911114216 (nan) | > current_lr: 5e-06 | > step_time: 0.1619 (0.13650856089724533) | > loader_time: 0.0049 (0.004653981217374085)  --> TIME: 2025-02-24 12:45:04 -- STEP: 29196/36938 -- GLOBAL_STEP: 324700 | > loss_text_ce: 0.03818192332983017 (0.041210337039875575) | > loss_mel_ce: 2.4021737575531006 (nan) | > loss: 0.009683951735496521 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.1365190912050851) | > loader_time: 0.0045 (0.004654064686727276)  --> TIME: 2025-02-24 12:45:14 -- STEP: 29246/36938 -- GLOBAL_STEP: 324750 | > loss_text_ce: 0.04169602319598198 (0.0412108651420316) | > loss_mel_ce: 3.8174619674682617 (nan) | > loss: 0.01531411986798048 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13651984153543592) | > loader_time: 0.0046 (0.0046541168516412695)  --> TIME: 2025-02-24 12:45:23 -- STEP: 29296/36938 -- GLOBAL_STEP: 324800 | > loss_text_ce: 0.035344939678907394 (0.04121071649602764) | > loss_mel_ce: 2.7524681091308594 (nan) | > loss: 0.011062750592827797 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13652162653445435) | > loader_time: 0.0044 (0.0046541416322775)  --> TIME: 2025-02-24 12:45:33 -- STEP: 29346/36938 -- GLOBAL_STEP: 324850 | > loss_text_ce: 0.042638663202524185 (0.041209941684996844) | > loss_mel_ce: 3.55349063873291 (nan) | > loss: 0.014270355924963951 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.13653545579336393) | > loader_time: 0.0047 (0.004654155750504762)  --> TIME: 2025-02-24 12:45:42 -- STEP: 29396/36938 -- GLOBAL_STEP: 324900 | > loss_text_ce: 0.026194419711828232 (0.04120990900742387) | > loss_mel_ce: 1.4222066402435303 (nan) | > loss: 0.0057476237416267395 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1365367812937761) | > loader_time: 0.0045 (0.004654103597827101)  --> TIME: 2025-02-24 12:45:52 -- STEP: 29446/36938 -- GLOBAL_STEP: 324950 | > loss_text_ce: 0.03506854176521301 (0.04120990860994575) | > loss_mel_ce: 3.698718786239624 (nan) | > loss: 0.014816616661846638 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.13654269336852565) | > loader_time: 0.0048 (0.004654050650645543)  --> TIME: 2025-02-24 12:46:01 -- STEP: 29496/36938 -- GLOBAL_STEP: 325000 | > loss_text_ce: 0.04878786951303482 (0.041208262991377094) | > loss_mel_ce: 2.801901340484619 (nan) | > loss: 0.01131225936114788 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13655078025539327) | > loader_time: 0.0046 (0.0046541225402569435)  --> TIME: 2025-02-24 12:46:10 -- STEP: 29546/36938 -- GLOBAL_STEP: 325050 | > loss_text_ce: 0.03287382423877716 (0.04120717707141708) | > loss_mel_ce: 2.0754916667938232 (nan) | > loss: 0.008366530761122704 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13654633557829599) | > loader_time: 0.0048 (0.004654059976244545)  --> TIME: 2025-02-24 12:46:20 -- STEP: 29596/36938 -- GLOBAL_STEP: 325100 | > loss_text_ce: 0.03257948160171509 (0.04120517015058263) | > loss_mel_ce: 2.848721504211426 (nan) | > loss: 0.011433734558522701 (nan) | > current_lr: 5e-06 | > step_time: 0.1084 (0.13654188541709356) | > loader_time: 0.0045 (0.00465402150898469)  --> TIME: 2025-02-24 12:46:29 -- STEP: 29646/36938 -- GLOBAL_STEP: 325150 | > loss_text_ce: 0.04303211718797684 (0.041204488992234604) | > loss_mel_ce: 3.6616315841674805 (nan) | > loss: 0.014701047912240028 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13655112000891145) | > loader_time: 0.0046 (0.004654019103957088)  --> TIME: 2025-02-24 12:46:38 -- STEP: 29696/36938 -- GLOBAL_STEP: 325200 | > loss_text_ce: 0.0385727696120739 (0.04120476528234949) | > loss_mel_ce: 3.424577474594116 (nan) | > loss: 0.01374266017228365 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.136542324683276) | > loader_time: 0.0047 (0.004653940033488766)  --> TIME: 2025-02-24 12:46:48 -- STEP: 29746/36938 -- GLOBAL_STEP: 325250 | > loss_text_ce: 0.0539265051484108 (0.04120554904531913) | > loss_mel_ce: 3.430649995803833 (nan) | > loss: 0.01382768526673317 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13654968056947792) | > loader_time: 0.0045 (0.0046539921963502815)  --> TIME: 2025-02-24 12:46:57 -- STEP: 29796/36938 -- GLOBAL_STEP: 325300 | > loss_text_ce: 0.03728193789720535 (0.04120438010878792) | > loss_mel_ce: 2.5226528644561768 (nan) | > loss: 0.010158472694456577 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13654946567932574) | > loader_time: 0.0045 (0.004653992181113001)  --> TIME: 2025-02-24 12:47:06 -- STEP: 29846/36938 -- GLOBAL_STEP: 325350 | > loss_text_ce: 0.05399477481842041 (0.04120703942558002) | > loss_mel_ce: 3.198199987411499 (nan) | > loss: 0.012905536219477654 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.136549000421159) | > loader_time: 0.0047 (0.004653996823101323)  --> TIME: 2025-02-24 12:47:16 -- STEP: 29896/36938 -- GLOBAL_STEP: 325400 | > loss_text_ce: 0.04937805235385895 (0.041206203694790486) | > loss_mel_ce: 3.0870683193206787 (nan) | > loss: 0.012446217238903046 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13655099322172584) | > loader_time: 0.0048 (0.004654011665450931)  --> TIME: 2025-02-24 12:47:25 -- STEP: 29946/36938 -- GLOBAL_STEP: 325450 | > loss_text_ce: 0.03725401684641838 (0.041206222907858636) | > loss_mel_ce: 3.435615301132202 (nan) | > loss: 0.013781228102743626 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13655821323108205) | > loader_time: 0.0045 (0.004654045765157761)  --> TIME: 2025-02-24 12:47:35 -- STEP: 29996/36938 -- GLOBAL_STEP: 325500 | > loss_text_ce: 0.039192866533994675 (0.0412016664952703) | > loss_mel_ce: 3.052480936050415 (nan) | > loss: 0.012268547900021076 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1365588022273518) | > loader_time: 0.0047 (0.004654043935937796)  --> TIME: 2025-02-24 12:47:44 -- STEP: 30046/36938 -- GLOBAL_STEP: 325550 | > loss_text_ce: 0.04424890875816345 (0.04120063193491803) | > loss_mel_ce: 3.023263931274414 (nan) | > loss: 0.012172670103609562 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13656172733812233) | > loader_time: 0.0047 (0.004654051857131768)  --> TIME: 2025-02-24 12:47:53 -- STEP: 30096/36938 -- GLOBAL_STEP: 325600 | > loss_text_ce: 0.03695295378565788 (0.04120168401532917) | > loss_mel_ce: 3.656907558441162 (nan) | > loss: 0.014658177271485329 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.13656751676039303) | > loader_time: 0.0046 (0.0046540537471786644)  --> TIME: 2025-02-24 12:48:03 -- STEP: 30146/36938 -- GLOBAL_STEP: 325650 | > loss_text_ce: 0.03693124279379845 (0.04120297433741341) | > loss_mel_ce: 3.326256275177002 (nan) | > loss: 0.013345982879400253 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.1365647407958311) | > loader_time: 0.0048 (0.004654107354484988)  --> TIME: 2025-02-24 12:48:12 -- STEP: 30196/36938 -- GLOBAL_STEP: 325700 | > loss_text_ce: 0.039525747299194336 (0.04120185702673619) | > loss_mel_ce: 2.6195027828216553 (nan) | > loss: 0.010551701299846172 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.136564705763168) | > loader_time: 0.0047 (0.00465416210284218)  --> TIME: 2025-02-24 12:48:21 -- STEP: 30246/36938 -- GLOBAL_STEP: 325750 | > loss_text_ce: 0.04654846712946892 (0.04120213920861437) | > loss_mel_ce: 2.930633068084717 (nan) | > loss: 0.011814212426543236 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13656521871403232) | > loader_time: 0.0047 (0.004654212468737613)  --> TIME: 2025-02-24 12:48:31 -- STEP: 30296/36938 -- GLOBAL_STEP: 325800 | > loss_text_ce: 0.04660002514719963 (0.04119923481586008) | > loss_mel_ce: 2.536973714828491 (nan) | > loss: 0.010252277366816998 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.1365625294289871) | > loader_time: 0.0047 (0.004654238469247508)  --> TIME: 2025-02-24 12:48:40 -- STEP: 30346/36938 -- GLOBAL_STEP: 325850 | > loss_text_ce: 0.033791687339544296 (0.04119939545329303) | > loss_mel_ce: 1.54520583152771 (nan) | > loss: 0.006265863310545683 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13655837574638727) | > loader_time: 0.0048 (0.004654326993899163)  --> TIME: 2025-02-24 12:48:49 -- STEP: 30396/36938 -- GLOBAL_STEP: 325900 | > loss_text_ce: 0.0362890250980854 (0.04119836097491374) | > loss_mel_ce: 3.2405412197113037 (nan) | > loss: 0.013003295287489891 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13655977479594877) | > loader_time: 0.0047 (0.004654454453899184)  --> TIME: 2025-02-24 12:48:59 -- STEP: 30446/36938 -- GLOBAL_STEP: 325950 | > loss_text_ce: 0.0416378527879715 (0.04119729557734292) | > loss_mel_ce: 3.1058290004730225 (nan) | > loss: 0.01248994842171669 (nan) | > current_lr: 5e-06 | > step_time: 0.1469 (0.13655895460280804) | > loader_time: 0.0047 (0.004654541150629168)  --> TIME: 2025-02-24 12:49:08 -- STEP: 30496/36938 -- GLOBAL_STEP: 326000 | > loss_text_ce: 0.04059123247861862 (0.04119822232109239) | > loss_mel_ce: 2.0503392219543457 (nan) | > loss: 0.008297343738377094 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.136552497564746) | > loader_time: 0.0046 (0.00465453057261606)  --> TIME: 2025-02-24 12:49:17 -- STEP: 30546/36938 -- GLOBAL_STEP: 326050 | > loss_text_ce: 0.04471433907747269 (0.041194304346445276) | > loss_mel_ce: 3.828706741333008 (nan) | > loss: 0.015370720066130161 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.1365465150790798) | > loader_time: 0.0045 (0.004654477904402268)  --> TIME: 2025-02-24 12:49:26 -- STEP: 30596/36938 -- GLOBAL_STEP: 326100 | > loss_text_ce: 0.039520181715488434 (0.041191659076857665) | > loss_mel_ce: 3.5544538497924805 (nan) | > loss: 0.014261802658438683 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.13654846951983063) | > loader_time: 0.0048 (0.00465452290778225)  --> TIME: 2025-02-24 12:49:35 -- STEP: 30646/36938 -- GLOBAL_STEP: 326150 | > loss_text_ce: 0.03213627636432648 (0.041188515721201704) | > loss_mel_ce: 3.0321149826049805 (nan) | > loss: 0.01215972751379013 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13655248202068962) | > loader_time: 0.0046 (0.004654596160443564)  --> TIME: 2025-02-24 12:49:45 -- STEP: 30696/36938 -- GLOBAL_STEP: 326200 | > loss_text_ce: 0.04351692274212837 (0.04118805285151811) | > loss_mel_ce: 3.7306900024414062 (nan) | > loss: 0.01497701182961464 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13655286336864791) | > loader_time: 0.0047 (0.004654639302239576)  --> TIME: 2025-02-24 12:49:54 -- STEP: 30746/36938 -- GLOBAL_STEP: 326250 | > loss_text_ce: 0.03661540150642395 (0.041188974371191414) | > loss_mel_ce: 4.197012901306152 (nan) | > loss: 0.016800113022327423 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13655595577647975) | > loader_time: 0.0045 (0.004654637048699372)  --> TIME: 2025-02-24 12:50:04 -- STEP: 30796/36938 -- GLOBAL_STEP: 326300 | > loss_text_ce: 0.04590831696987152 (0.04118906473532264) | > loss_mel_ce: 3.1787898540496826 (nan) | > loss: 0.01279642153531313 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13655654553980628) | > loader_time: 0.0047 (0.004654700964484114)  --> TIME: 2025-02-24 12:50:13 -- STEP: 30846/36938 -- GLOBAL_STEP: 326350 | > loss_text_ce: 0.04138677567243576 (0.04118909577569815) | > loss_mel_ce: 3.126091480255127 (nan) | > loss: 0.012569359503686428 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.1365572315457965) | > loader_time: 0.0047 (0.00465477820709741)  --> TIME: 2025-02-24 12:50:22 -- STEP: 30896/36938 -- GLOBAL_STEP: 326400 | > loss_text_ce: 0.04377563297748566 (0.04118691563119979) | > loss_mel_ce: 3.723707914352417 (nan) | > loss: 0.01495033223181963 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13655266225924498) | > loader_time: 0.0046 (0.00465479531725229)  --> TIME: 2025-02-24 12:50:31 -- STEP: 30946/36938 -- GLOBAL_STEP: 326450 | > loss_text_ce: 0.042703721672296524 (0.041186461607028546) | > loss_mel_ce: 2.779017925262451 (nan) | > loss: 0.011197308078408241 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.1365530751436107) | > loader_time: 0.0046 (0.004654762910238771)  --> TIME: 2025-02-24 12:50:41 -- STEP: 30996/36938 -- GLOBAL_STEP: 326500 | > loss_text_ce: 0.04431377723813057 (0.04118871054527245) | > loss_mel_ce: 2.701500415802002 (nan) | > loss: 0.010896088555455208 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1365599819484384) | > loader_time: 0.0046 (0.004654831279553314)  --> TIME: 2025-02-24 12:50:50 -- STEP: 31046/36938 -- GLOBAL_STEP: 326550 | > loss_text_ce: 0.03681178018450737 (0.041186760623671055) | > loss_mel_ce: 3.003023624420166 (nan) | > loss: 0.012062840163707733 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13656328890669717) | > loader_time: 0.0046 (0.004654813402592829)  --> TIME: 2025-02-24 12:51:00 -- STEP: 31096/36938 -- GLOBAL_STEP: 326600 | > loss_text_ce: 0.04827801138162613 (0.041185743726585296) | > loss_mel_ce: 3.6902804374694824 (nan) | > loss: 0.01483555044978857 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.13656276623939628) | > loader_time: 0.0047 (0.004654791343172277)  --> TIME: 2025-02-24 12:51:09 -- STEP: 31146/36938 -- GLOBAL_STEP: 326650 | > loss_text_ce: 0.03902773931622505 (0.041184803429286) | > loss_mel_ce: 3.891796827316284 (nan) | > loss: 0.015598511323332787 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.1365686433665721) | > loader_time: 0.0045 (0.0046547836844941055)  --> TIME: 2025-02-24 12:51:18 -- STEP: 31196/36938 -- GLOBAL_STEP: 326700 | > loss_text_ce: 0.043623462319374084 (0.04118537693460586) | > loss_mel_ce: 3.05157470703125 (nan) | > loss: 0.012282532639801502 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13656495609930355) | > loader_time: 0.0047 (0.004654713296969761)  --> TIME: 2025-02-24 12:51:28 -- STEP: 31246/36938 -- GLOBAL_STEP: 326750 | > loss_text_ce: 0.04037041217088699 (0.04118372984834983) | > loss_mel_ce: 4.084348678588867 (nan) | > loss: 0.01636793464422226 (nan) | > current_lr: 5e-06 | > step_time: 0.162 (0.13656730630643074) | > loader_time: 0.0049 (0.004654692060654584)  --> TIME: 2025-02-24 12:51:37 -- STEP: 31296/36938 -- GLOBAL_STEP: 326800 | > loss_text_ce: 0.04302509129047394 (0.041185317109831705) | > loss_mel_ce: 3.119288682937622 (nan) | > loss: 0.01254886481910944 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13656654478582175) | > loader_time: 0.0047 (0.004654651595344348)  --> TIME: 2025-02-24 12:51:46 -- STEP: 31346/36938 -- GLOBAL_STEP: 326850 | > loss_text_ce: 0.04284409061074257 (0.041186645918784265) | > loss_mel_ce: 2.9742743968963623 (nan) | > loss: 0.011972692795097828 (nan) | > current_lr: 5e-06 | > step_time: 0.1688 (0.13656266719998508) | > loader_time: 0.0069 (0.004654732620922237)  --> TIME: 2025-02-24 12:51:56 -- STEP: 31396/36938 -- GLOBAL_STEP: 326900 | > loss_text_ce: 0.03462403640151024 (0.04118910839744141) | > loss_mel_ce: 2.781996726989746 (nan) | > loss: 0.011177067644894123 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.1365594099336167) | > loader_time: 0.0046 (0.004654753335738204)  --> TIME: 2025-02-24 12:52:05 -- STEP: 31446/36938 -- GLOBAL_STEP: 326950 | > loss_text_ce: 0.040980156511068344 (0.041189022166703594) | > loss_mel_ce: 2.8322854042053223 (nan) | > loss: 0.011401847936213017 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13655773479429198) | > loader_time: 0.0045 (0.004654776554924097)  --> TIME: 2025-02-24 12:52:14 -- STEP: 31496/36938 -- GLOBAL_STEP: 327000 | > loss_text_ce: 0.04986172169446945 (0.041188004667590816) | > loss_mel_ce: 3.215383768081665 (nan) | > loss: 0.012957324273884296 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13656105909185276) | > loader_time: 0.0046 (0.004654793856499449)  --> TIME: 2025-02-24 12:52:24 -- STEP: 31546/36938 -- GLOBAL_STEP: 327050 | > loss_text_ce: 0.0351276658475399 (0.041187603148173345) | > loss_mel_ce: 0.003814697265625 (nan) | > loss: 0.00015453319065272808 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13655848005960286) | > loader_time: 0.0046 (0.00465485456061941)  --> TIME: 2025-02-24 12:52:33 -- STEP: 31596/36938 -- GLOBAL_STEP: 327100 | > loss_text_ce: 0.03434083238244057 (0.041185064542681796) | > loss_mel_ce: 2.96822190284729 (nan) | > loss: 0.011914932169020176 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13656307700495618) | > loader_time: 0.0046 (0.004654875502192188)  --> TIME: 2025-02-24 12:52:42 -- STEP: 31646/36938 -- GLOBAL_STEP: 327150 | > loss_text_ce: 0.042621415108442307 (0.04118298768833208) | > loss_mel_ce: 3.1146514415740967 (nan) | > loss: 0.012528860941529274 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13656657704027483) | > loader_time: 0.0045 (0.004654872615592188)  --> TIME: 2025-02-24 12:52:51 -- STEP: 31696/36938 -- GLOBAL_STEP: 327200 | > loss_text_ce: 0.03888815641403198 (0.0411813120458593) | > loss_mel_ce: 2.602999687194824 (nan) | > loss: 0.010483683086931705 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13655790837048198) | > loader_time: 0.0047 (0.004654812901565266)  --> TIME: 2025-02-24 12:53:01 -- STEP: 31746/36938 -- GLOBAL_STEP: 327250 | > loss_text_ce: 0.030013279989361763 (0.04118039854991409) | > loss_mel_ce: 3.2654337882995605 (nan) | > loss: 0.01307717151939869 (nan) | > current_lr: 5e-06 | > step_time: 0.1501 (0.1365726864766757) | > loader_time: 0.0047 (0.004654893395671384)  --> TIME: 2025-02-24 12:53:11 -- STEP: 31796/36938 -- GLOBAL_STEP: 327300 | > loss_text_ce: 0.048211317509412766 (0.04118090136654921) | > loss_mel_ce: 2.56793212890625 (nan) | > loss: 0.010381522588431835 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.1365690118472585) | > loader_time: 0.0045 (0.004654890149625585)  --> TIME: 2025-02-24 12:53:20 -- STEP: 31846/36938 -- GLOBAL_STEP: 327350 | > loss_text_ce: 0.039777617901563644 (0.04117913819454218) | > loss_mel_ce: 0.5878829956054688 (nan) | > loss: 0.0024907169863581657 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13657316855806947) | > loader_time: 0.0046 (0.004654887452808646)  --> TIME: 2025-02-24 12:53:29 -- STEP: 31896/36938 -- GLOBAL_STEP: 327400 | > loss_text_ce: 0.03287373483181 (0.04117788877164727) | > loss_mel_ce: 3.0071072578430176 (nan) | > loss: 0.01206341665238142 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13657272832268702) | > loader_time: 0.0045 (0.004654870943404829)  --> TIME: 2025-02-24 12:53:38 -- STEP: 31946/36938 -- GLOBAL_STEP: 327450 | > loss_text_ce: 0.03765325993299484 (0.04117918490988847) | > loss_mel_ce: 3.0299837589263916 (nan) | > loss: 0.012173163704574108 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13656675055831127) | > loader_time: 0.0046 (0.0046548416490196165)  --> TIME: 2025-02-24 12:53:47 -- STEP: 31996/36938 -- GLOBAL_STEP: 327500 | > loss_text_ce: 0.0422951839864254 (0.04118003313367244) | > loss_mel_ce: 3.436469316482544 (nan) | > loss: 0.013804621994495392 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13655776330151734) | > loader_time: 0.0046 (0.004654794353919579)  --> TIME: 2025-02-24 12:53:57 -- STEP: 32046/36938 -- GLOBAL_STEP: 327550 | > loss_text_ce: 0.033280372619628906 (0.04117967362437099) | > loss_mel_ce: 3.63584041595459 (nan) | > loss: 0.014560003764927387 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.13656714601938752) | > loader_time: 0.0046 (0.004654832988287402)  --> TIME: 2025-02-24 12:54:06 -- STEP: 32096/36938 -- GLOBAL_STEP: 327600 | > loss_text_ce: 0.04132208600640297 (0.04117966593099371) | > loss_mel_ce: 3.092501163482666 (nan) | > loss: 0.012435806915163994 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13656299520406542) | > loader_time: 0.0046 (0.004654809320020552)  --> TIME: 2025-02-24 12:54:16 -- STEP: 32146/36938 -- GLOBAL_STEP: 327650 | > loss_text_ce: 0.04159404709935188 (0.041177866505037665) | > loss_mel_ce: 2.80899715423584 (nan) | > loss: 0.011311870068311691 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13656095169257051) | > loader_time: 0.0047 (0.004654836923149154)  --> TIME: 2025-02-24 12:54:25 -- STEP: 32196/36938 -- GLOBAL_STEP: 327700 | > loss_text_ce: 0.04070206731557846 (0.04117416956336768) | > loss_mel_ce: 3.1551625728607178 (nan) | > loss: 0.01268200296908617 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13655858159672513) | > loader_time: 0.0046 (0.004654772534312976)  --> TIME: 2025-02-24 12:54:34 -- STEP: 32246/36938 -- GLOBAL_STEP: 327750 | > loss_text_ce: 0.035981446504592896 (0.0411726686301924) | > loss_mel_ce: 2.270092725753784 (nan) | > loss: 0.009151088073849678 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13656621782860914) | > loader_time: 0.0045 (0.004654877750555506)  --> TIME: 2025-02-24 12:54:43 -- STEP: 32296/36938 -- GLOBAL_STEP: 327800 | > loss_text_ce: 0.03872225433588028 (0.04117006644600704) | > loss_mel_ce: 3.3543574810028076 (nan) | > loss: 0.013464602641761303 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13655917427313669) | > loader_time: 0.0046 (0.004654850084533029)  --> TIME: 2025-02-24 12:54:53 -- STEP: 32346/36938 -- GLOBAL_STEP: 327850 | > loss_text_ce: 0.03430468961596489 (0.04116929143337025) | > loss_mel_ce: 3.1273109912872314 (nan) | > loss: 0.012546094134449959 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.1365592693092875) | > loader_time: 0.0045 (0.004654826491689727)  --> TIME: 2025-02-24 12:55:02 -- STEP: 32396/36938 -- GLOBAL_STEP: 327900 | > loss_text_ce: 0.0435306541621685 (0.04116909915700589) | > loss_mel_ce: 3.853015899658203 (nan) | > loss: 0.015462487004697323 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13655920627897594) | > loader_time: 0.0045 (0.004654890844182849)  --> TIME: 2025-02-24 12:55:12 -- STEP: 32446/36938 -- GLOBAL_STEP: 327950 | > loss_text_ce: 0.04198195040225983 (0.041164950711714256) | > loss_mel_ce: 0.7263819575309753 (nan) | > loss: 0.0030490632634609938 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13656262173382566) | > loader_time: 0.0046 (0.004654913326891979)  --> TIME: 2025-02-24 12:55:21 -- STEP: 32496/36938 -- GLOBAL_STEP: 328000 | > loss_text_ce: 0.035166509449481964 (0.041163376882916) | > loss_mel_ce: 3.5645906925201416 (nan) | > loss: 0.014284751377999783 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13655453519038213) | > loader_time: 0.0047 (0.00465490602613495)  --> TIME: 2025-02-24 12:55:30 -- STEP: 32546/36938 -- GLOBAL_STEP: 328050 | > loss_text_ce: 0.034838367253541946 (0.04116483041645764) | > loss_mel_ce: 3.2200868129730225 (nan) | > loss: 0.01291637122631073 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13655228783442194) | > loader_time: 0.0046 (0.00465482951368016)  --> TIME: 2025-02-24 12:55:39 -- STEP: 32596/36938 -- GLOBAL_STEP: 328100 | > loss_text_ce: 0.016404129564762115 (0.04116108631311172) | > loss_mel_ce: 1.875516653060913 (nan) | > loss: 0.007507622707635164 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.1365428511416759) | > loader_time: 0.0047 (0.004654744839080218)  --> TIME: 2025-02-24 12:55:48 -- STEP: 32646/36938 -- GLOBAL_STEP: 328150 | > loss_text_ce: 0.030492540448904037 (0.041159262665097396) | > loss_mel_ce: 2.217996120452881 (nan) | > loss: 0.008922575041651726 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.1365439965430924) | > loader_time: 0.0046 (0.0046547098150442195)  --> TIME: 2025-02-24 12:55:57 -- STEP: 32696/36938 -- GLOBAL_STEP: 328200 | > loss_text_ce: 0.04691644385457039 (0.04115942673218632) | > loss_mel_ce: 3.4635391235351562 (nan) | > loss: 0.01393038034439087 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.13654101172752453) | > loader_time: 0.0046 (0.004654725526345824)  --> TIME: 2025-02-24 12:56:07 -- STEP: 32746/36938 -- GLOBAL_STEP: 328250 | > loss_text_ce: 0.04275675117969513 (0.04115797236039257) | > loss_mel_ce: 1.5187572240829468 (nan) | > loss: 0.006196484435349703 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13654019383674162) | > loader_time: 0.0046 (0.004654658930671406)  --> TIME: 2025-02-24 12:56:16 -- STEP: 32796/36938 -- GLOBAL_STEP: 328300 | > loss_text_ce: 0.02465124987065792 (0.04115607172764965) | > loss_mel_ce: 3.7359914779663086 (nan) | > loss: 0.014923186041414738 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13653513169431158) | > loader_time: 0.0045 (0.004654597321549997)  --> TIME: 2025-02-24 12:56:25 -- STEP: 32846/36938 -- GLOBAL_STEP: 328350 | > loss_text_ce: 0.03949384018778801 (0.04115486875716177) | > loss_mel_ce: 3.342684507369995 (nan) | > loss: 0.013421343639492989 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13653543558290906) | > loader_time: 0.0045 (0.00465460720200463)  --> TIME: 2025-02-24 12:56:35 -- STEP: 32896/36938 -- GLOBAL_STEP: 328400 | > loss_text_ce: 0.03712289035320282 (0.04115443836217802) | > loss_mel_ce: 3.6397206783294678 (nan) | > loss: 0.014590649865567684 (nan) | > current_lr: 5e-06 | > step_time: 0.1861 (0.136541940677723) | > loader_time: 0.0047 (0.004654625481436707)  --> TIME: 2025-02-24 12:56:44 -- STEP: 32946/36938 -- GLOBAL_STEP: 328450 | > loss_text_ce: 0.041978973895311356 (0.04115367579087702) | > loss_mel_ce: 4.0302734375 (nan) | > loss: 0.01615973189473152 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13654819674912516) | > loader_time: 0.0045 (0.004654623623689758)  --> TIME: 2025-02-24 12:56:54 -- STEP: 32996/36938 -- GLOBAL_STEP: 328500 | > loss_text_ce: 0.04452302306890488 (0.04115127053024628) | > loss_mel_ce: 2.0330865383148193 (nan) | > loss: 0.008244482800364494 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.1365523735224547) | > loader_time: 0.0046 (0.004654633441048274)  --> TIME: 2025-02-24 12:57:03 -- STEP: 33046/36938 -- GLOBAL_STEP: 328550 | > loss_text_ce: 0.03747837617993355 (0.041151305944178825) | > loss_mel_ce: 3.9093589782714844 (nan) | > loss: 0.01566205359995365 (nan) | > current_lr: 5e-06 | > step_time: 0.1707 (0.1365552525582953) | > loader_time: 0.0049 (0.00465462976597792)  --> TIME: 2025-02-24 12:57:12 -- STEP: 33096/36938 -- GLOBAL_STEP: 328600 | > loss_text_ce: 0.03655841574072838 (0.04115170430364334) | > loss_mel_ce: 0.0113525390625 (nan) | > loss: 0.00019012285338249058 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13654930754534356) | > loader_time: 0.0045 (0.004654552204931092)  --> TIME: 2025-02-24 12:57:22 -- STEP: 33146/36938 -- GLOBAL_STEP: 328650 | > loss_text_ce: 0.042000751942396164 (0.041152167984894204) | > loss_mel_ce: 3.338090181350708 (nan) | > loss: 0.013413060456514359 (nan) | > current_lr: 5e-06 | > step_time: 0.1567 (0.13654946750761066) | > loader_time: 0.0046 (0.004654489407705976)  --> TIME: 2025-02-24 12:57:31 -- STEP: 33196/36938 -- GLOBAL_STEP: 328700 | > loss_text_ce: 0.03194805234670639 (0.04114967039855293) | > loss_mel_ce: 1.8450915813446045 (nan) | > loss: 0.007448570337146521 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13655256096721238) | > loader_time: 0.0045 (0.004654435102214802)  --> TIME: 2025-02-24 12:57:40 -- STEP: 33246/36938 -- GLOBAL_STEP: 328750 | > loss_text_ce: 0.043835319578647614 (0.04114913063956705) | > loss_mel_ce: 2.956120014190674 (nan) | > loss: 0.011904586106538773 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.136554721588909) | > loader_time: 0.0047 (0.004654445538041499)  --> TIME: 2025-02-24 12:57:50 -- STEP: 33296/36938 -- GLOBAL_STEP: 328800 | > loss_text_ce: 0.04731622710824013 (0.04114613867848608) | > loss_mel_ce: 0.01280523557215929 (nan) | > loss: 0.00023857723863329738 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13655414527164908) | > loader_time: 0.0045 (0.00465439612306352)  --> TIME: 2025-02-24 12:57:59 -- STEP: 33346/36938 -- GLOBAL_STEP: 328850 | > loss_text_ce: 0.04176286235451698 (0.04114436918068272) | > loss_mel_ce: 3.378028392791748 (nan) | > loss: 0.013570601120591164 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13655017423678423) | > loader_time: 0.0046 (0.004654343267054266)  --> TIME: 2025-02-24 12:58:08 -- STEP: 33396/36938 -- GLOBAL_STEP: 328900 | > loss_text_ce: 0.031139103695750237 (0.041145623745127875) | > loss_mel_ce: 3.06787109375 (nan) | > loss: 0.012297660112380981 (nan) | > current_lr: 5e-06 | > step_time: 0.1074 (0.1365449302052868) | > loader_time: 0.0045 (0.004654292475464747)  --> TIME: 2025-02-24 12:58:17 -- STEP: 33446/36938 -- GLOBAL_STEP: 328950 | > loss_text_ce: 0.04158888757228851 (0.041144971865090224) | > loss_mel_ce: 3.374302864074707 (nan) | > loss: 0.013555126264691353 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13654154567978602) | > loader_time: 0.0046 (0.004654246119943072)  --> TIME: 2025-02-24 12:58:26 -- STEP: 33496/36938 -- GLOBAL_STEP: 329000 | > loss_text_ce: 0.0449286587536335 (0.041143971388118095) | > loss_mel_ce: 3.116490125656128 (nan) | > loss: 0.012545312754809856 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.1365369090266355) | > loader_time: 0.0048 (0.004654184784557834)  --> TIME: 2025-02-24 12:58:36 -- STEP: 33546/36938 -- GLOBAL_STEP: 329050 | > loss_text_ce: 0.03869447484612465 (0.04114399737430433) | > loss_mel_ce: 3.1330652236938477 (nan) | > loss: 0.012586348690092564 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.13653819368967787) | > loader_time: 0.0046 (0.004654143454030273)  --> TIME: 2025-02-24 12:58:45 -- STEP: 33596/36938 -- GLOBAL_STEP: 329100 | > loss_text_ce: 0.03993057832121849 (0.041143317743863046) | > loss_mel_ce: 3.743520975112915 (nan) | > loss: 0.015013697557151318 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.13653774323357332) | > loader_time: 0.0046 (0.00465414407409677)  --> TIME: 2025-02-24 12:58:54 -- STEP: 33646/36938 -- GLOBAL_STEP: 329150 | > loss_text_ce: 0.04662972688674927 (0.04114223693593304) | > loss_mel_ce: 2.944725275039673 (nan) | > loss: 0.011870456859469414 (nan) | > current_lr: 5e-06 | > step_time: 0.1757 (0.13652994873222898) | > loader_time: 0.0049 (0.00465404220620285)  --> TIME: 2025-02-24 12:59:03 -- STEP: 33696/36938 -- GLOBAL_STEP: 329200 | > loss_text_ce: 0.03593428060412407 (0.04114178942071676) | > loss_mel_ce: 3.5404207706451416 (nan) | > loss: 0.014191885478794575 (nan) | > current_lr: 5e-06 | > step_time: 0.1858 (0.136524366481975) | > loader_time: 0.0048 (0.004654037024107658)  --> TIME: 2025-02-24 12:59:13 -- STEP: 33746/36938 -- GLOBAL_STEP: 329250 | > loss_text_ce: 0.04249916970729828 (0.041141675025493346) | > loss_mel_ce: 3.657594680786133 (nan) | > loss: 0.014682912267744541 (nan) | > current_lr: 5e-06 | > step_time: 0.1628 (0.13653282036688394) | > loader_time: 0.0046 (0.00465402684821874)  --> TIME: 2025-02-24 12:59:22 -- STEP: 33796/36938 -- GLOBAL_STEP: 329300 | > loss_text_ce: 0.026302793994545937 (0.04114018988771104) | > loss_mel_ce: 2.3593051433563232 (nan) | > loss: 0.00946669839322567 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.1365311340208327) | > loader_time: 0.0045 (0.004654033076256648)  --> TIME: 2025-02-24 12:59:32 -- STEP: 33846/36938 -- GLOBAL_STEP: 329350 | > loss_text_ce: 0.0480664037168026 (0.04113927677266372) | > loss_mel_ce: 3.4175384044647217 (nan) | > loss: 0.013752400875091553 (nan) | > current_lr: 5e-06 | > step_time: 0.1807 (0.1365345915546028) | > loader_time: 0.0049 (0.004654024845247963)  --> TIME: 2025-02-24 12:59:41 -- STEP: 33896/36938 -- GLOBAL_STEP: 329400 | > loss_text_ce: 0.04625748470425606 (0.0411378482602796) | > loss_mel_ce: 1.7843749523162842 (nan) | > loss: 0.007264414802193642 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13653244290932645) | > loader_time: 0.0046 (0.004654020886953541)  --> TIME: 2025-02-24 12:59:50 -- STEP: 33946/36938 -- GLOBAL_STEP: 329450 | > loss_text_ce: 0.04145912081003189 (0.041137145845980945) | > loss_mel_ce: 3.3011813163757324 (nan) | > loss: 0.013264446519315243 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13652564258881164) | > loader_time: 0.0048 (0.0046539687733893964)  --> TIME: 2025-02-24 12:59:59 -- STEP: 33996/36938 -- GLOBAL_STEP: 329500 | > loss_text_ce: 0.027722351253032684 (0.04113365514605513) | > loss_mel_ce: 3.1182234287261963 (nan) | > loss: 0.012483912520110607 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.1365258372423305) | > loader_time: 0.0046 (0.00465393227708385)  --> TIME: 2025-02-24 13:00:08 -- STEP: 34046/36938 -- GLOBAL_STEP: 329550 | > loss_text_ce: 0.044732529670000076 (0.041134643522348566) | > loss_mel_ce: 1.006082534790039 (nan) | > loss: 0.004169901367276907 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1365224813620864) | > loader_time: 0.0044 (0.0046538522883139075)  --> TIME: 2025-02-24 13:00:17 -- STEP: 34096/36938 -- GLOBAL_STEP: 329600 | > loss_text_ce: 0.03500061482191086 (0.041133216784667984) | > loss_mel_ce: 3.7023773193359375 (nan) | > loss: 0.014830865897238255 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13651623466047375) | > loader_time: 0.0044 (0.004653741962637848)  --> TIME: 2025-02-24 13:00:27 -- STEP: 34146/36938 -- GLOBAL_STEP: 329650 | > loss_text_ce: 0.044604234397411346 (0.041130175090406916) | > loss_mel_ce: 3.2257680892944336 (nan) | > loss: 0.012977669015526772 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.1365237844951799) | > loader_time: 0.0046 (0.004653799424221463)  --> TIME: 2025-02-24 13:00:36 -- STEP: 34196/36938 -- GLOBAL_STEP: 329700 | > loss_text_ce: 0.05080275237560272 (0.04112864994927202) | > loss_mel_ce: 2.9099302291870117 (nan) | > loss: 0.01174894068390131 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13652402755182053) | > loader_time: 0.0046 (0.0046539305525086175)  --> TIME: 2025-02-24 13:00:46 -- STEP: 34246/36938 -- GLOBAL_STEP: 329750 | > loss_text_ce: 0.03533686324954033 (0.04113008398847126) | > loss_mel_ce: 3.0145187377929688 (nan) | > loss: 0.012102602981030941 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13652565431519717) | > loader_time: 0.0044 (0.0046538634674227536)  --> TIME: 2025-02-24 13:00:55 -- STEP: 34296/36938 -- GLOBAL_STEP: 329800 | > loss_text_ce: 0.05404256656765938 (0.04113035023060684) | > loss_mel_ce: 3.461527109146118 (nan) | > loss: 0.01395067386329174 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13653466759624586) | > loader_time: 0.0046 (0.004653953952557972)  --> TIME: 2025-02-24 13:01:05 -- STEP: 34346/36938 -- GLOBAL_STEP: 329850 | > loss_text_ce: 0.0388738252222538 (0.041130540895437405) | > loss_mel_ce: 2.9819185733795166 (nan) | > loss: 0.011987272650003433 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13653177443252099) | > loader_time: 0.0046 (0.004653961547549927)  --> TIME: 2025-02-24 13:01:14 -- STEP: 34396/36938 -- GLOBAL_STEP: 329900 | > loss_text_ce: 0.05101117119193077 (0.04113261090173594) | > loss_mel_ce: 3.6235718727111816 (nan) | > loss: 0.014581679366528988 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13653025841044794) | > loader_time: 0.0049 (0.004653971650487054)  --> TIME: 2025-02-24 13:01:23 -- STEP: 34446/36938 -- GLOBAL_STEP: 329950 | > loss_text_ce: 0.04473508149385452 (0.041130743692474096) | > loss_mel_ce: 3.70232892036438 (nan) | > loss: 0.01486930251121521 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13653685081325612) | > loader_time: 0.0047 (0.0046539495736473225)  --> TIME: 2025-02-24 13:01:33 -- STEP: 34496/36938 -- GLOBAL_STEP: 330000 | > loss_text_ce: 0.04656370356678963 (0.04113120227975571) | > loss_mel_ce: 3.128239870071411 (nan) | > loss: 0.012598427012562752 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13654149026508058) | > loader_time: 0.0045 (0.004654042602470075) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_330000.pth  --> TIME: 2025-02-24 13:01:48 -- STEP: 34546/36938 -- GLOBAL_STEP: 330050 | > loss_text_ce: 0.03498310595750809 (0.04112973997887236) | > loss_mel_ce: 2.960813283920288 (nan) | > loss: 0.011888082139194012 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.13654157458089908) | > loader_time: 0.0046 (0.004654084360055007)  --> TIME: 2025-02-24 13:01:58 -- STEP: 34596/36938 -- GLOBAL_STEP: 330100 | > loss_text_ce: 0.03738060221076012 (0.041127415704956004) | > loss_mel_ce: 3.270602226257324 (nan) | > loss: 0.0131269171833992 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13654007792045605) | > loader_time: 0.0046 (0.004654204629035159)  --> TIME: 2025-02-24 13:02:07 -- STEP: 34646/36938 -- GLOBAL_STEP: 330150 | > loss_text_ce: 0.04616871103644371 (0.041127589833897124) | > loss_mel_ce: 1.5231837034225464 (nan) | > loss: 0.006227589212357998 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13653836016745013) | > loader_time: 0.0046 (0.004654151644764819)  --> TIME: 2025-02-24 13:02:16 -- STEP: 34696/36938 -- GLOBAL_STEP: 330200 | > loss_text_ce: 0.04142948240041733 (0.041127792967004285) | > loss_mel_ce: 3.2433035373687744 (nan) | > loss: 0.013034655712544918 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13653089255541215) | > loader_time: 0.0046 (0.0046540683580758874)  --> TIME: 2025-02-24 13:02:25 -- STEP: 34746/36938 -- GLOBAL_STEP: 330250 | > loss_text_ce: 0.0466155931353569 (0.041126647360472966) | > loss_mel_ce: 3.3613908290863037 (nan) | > loss: 0.013523835688829422 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.1365304966084887) | > loader_time: 0.0045 (0.0046540730729495985)  --> TIME: 2025-02-24 13:02:34 -- STEP: 34796/36938 -- GLOBAL_STEP: 330300 | > loss_text_ce: 0.03917517513036728 (0.04112335699392005) | > loss_mel_ce: 1.7732124328613281 (nan) | > loss: 0.007192014716565609 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13653176583727444) | > loader_time: 0.0046 (0.004654112602461492)  --> TIME: 2025-02-24 13:02:44 -- STEP: 34846/36938 -- GLOBAL_STEP: 330350 | > loss_text_ce: 0.04632502421736717 (0.041122551916325016) | > loss_mel_ce: 3.169447422027588 (nan) | > loss: 0.012761002406477928 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13653450335699854) | > loader_time: 0.0048 (0.004654113771392306)  --> TIME: 2025-02-24 13:02:53 -- STEP: 34896/36938 -- GLOBAL_STEP: 330400 | > loss_text_ce: 0.05155574530363083 (0.041122149214506555) | > loss_mel_ce: 0.0372314453125 (nan) | > loss: 0.00035233012749813497 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.1365300382648462) | > loader_time: 0.0046 (0.004654099332089287)  --> TIME: 2025-02-24 13:03:02 -- STEP: 34946/36938 -- GLOBAL_STEP: 330450 | > loss_text_ce: 0.04082898050546646 (0.041119481277459256) | > loss_mel_ce: 3.483873128890991 (nan) | > loss: 0.013986913487315178 (nan) | > current_lr: 5e-06 | > step_time: 0.1612 (0.13652993840903282) | > loader_time: 0.0047 (0.004654156160854032)  --> TIME: 2025-02-24 13:03:12 -- STEP: 34996/36938 -- GLOBAL_STEP: 330500 | > loss_text_ce: 0.028745731338858604 (0.04111960963552571) | > loss_mel_ce: 1.66015625 (nan) | > loss: 0.006701992359012365 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.1365313597823132) | > loader_time: 0.0045 (0.004654160410026139)  --> TIME: 2025-02-24 13:03:21 -- STEP: 35046/36938 -- GLOBAL_STEP: 330550 | > loss_text_ce: 0.04127349331974983 (0.04111860077622479) | > loss_mel_ce: 3.2541871070861816 (nan) | > loss: 0.013077225536108017 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.13653714462399563) | > loader_time: 0.0048 (0.004654151619293656)  --> TIME: 2025-02-24 13:03:31 -- STEP: 35096/36938 -- GLOBAL_STEP: 330600 | > loss_text_ce: 0.04238149896264076 (0.04111828110973168) | > loss_mel_ce: 2.986715078353882 (nan) | > loss: 0.012020225636661053 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13653727120681422) | > loader_time: 0.0046 (0.00465415992523779)  --> TIME: 2025-02-24 13:03:40 -- STEP: 35146/36938 -- GLOBAL_STEP: 330650 | > loss_text_ce: 0.04132023826241493 (0.04111650306776506) | > loss_mel_ce: 2.635159492492676 (nan) | > loss: 0.01062095258384943 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.1365330386621736) | > loader_time: 0.0046 (0.004654139960381329)  --> TIME: 2025-02-24 13:03:49 -- STEP: 35196/36938 -- GLOBAL_STEP: 330700 | > loss_text_ce: 0.04789368063211441 (0.04111355679355352) | > loss_mel_ce: 2.9333455562591553 (nan) | > loss: 0.011830314993858337 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13653071897519492) | > loader_time: 0.0046 (0.004654115425590673)  --> TIME: 2025-02-24 13:03:59 -- STEP: 35246/36938 -- GLOBAL_STEP: 330750 | > loss_text_ce: 0.03969384729862213 (0.04111337302365103) | > loss_mel_ce: 0.0713297501206398 (nan) | > loss: 0.00044056985643692315 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.1365386827412915) | > loader_time: 0.0047 (0.004654202390622199)  --> TIME: 2025-02-24 13:04:08 -- STEP: 35296/36938 -- GLOBAL_STEP: 330800 | > loss_text_ce: 0.05230105295777321 (0.041113710253110364) | > loss_mel_ce: 3.110578775405884 (nan) | > loss: 0.012551111169159412 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13653980340075708) | > loader_time: 0.0048 (0.004654293837648914)  --> TIME: 2025-02-24 13:04:17 -- STEP: 35346/36938 -- GLOBAL_STEP: 330850 | > loss_text_ce: 0.02751237526535988 (0.041113626762702) | > loss_mel_ce: 1.7493082284927368 (nan) | > loss: 0.007050876040011644 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13654188363639605) | > loader_time: 0.0045 (0.004654331691046713)  --> TIME: 2025-02-24 13:04:27 -- STEP: 35396/36938 -- GLOBAL_STEP: 330900 | > loss_text_ce: 0.039275530725717545 (0.04111370382321) | > loss_mel_ce: 3.355522871017456 (nan) | > loss: 0.013471423648297787 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.1365493977321768) | > loader_time: 0.0046 (0.00465442555302861)  --> TIME: 2025-02-24 13:04:36 -- STEP: 35446/36938 -- GLOBAL_STEP: 330950 | > loss_text_ce: 0.04186251014471054 (0.04111175154369581) | > loss_mel_ce: 2.620971202850342 (nan) | > loss: 0.010566800832748413 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13654661224999942) | > loader_time: 0.0045 (0.004654442753485378)  --> TIME: 2025-02-24 13:04:45 -- STEP: 35496/36938 -- GLOBAL_STEP: 331000 | > loss_text_ce: 0.03339499607682228 (0.04111118489602573) | > loss_mel_ce: 2.9353394508361816 (nan) | > loss: 0.01178069319576025 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1365421979255922) | > loader_time: 0.0045 (0.004654395189375566)  --> TIME: 2025-02-24 13:04:55 -- STEP: 35546/36938 -- GLOBAL_STEP: 331050 | > loss_text_ce: 0.046273455023765564 (0.04111001070003298) | > loss_mel_ce: 3.485774517059326 (nan) | > loss: 0.014016063883900642 (nan) | > current_lr: 5e-06 | > step_time: 0.1116 (0.13654189189485233) | > loader_time: 0.0045 (0.004654342406630323)  --> TIME: 2025-02-24 13:05:04 -- STEP: 35596/36938 -- GLOBAL_STEP: 331100 | > loss_text_ce: 0.04483422636985779 (0.04111098854233017) | > loss_mel_ce: 3.511319398880005 (nan) | > loss: 0.014111720956861973 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13654357513720686) | > loader_time: 0.0049 (0.004654425819987261)  --> TIME: 2025-02-24 13:05:14 -- STEP: 35646/36938 -- GLOBAL_STEP: 331150 | > loss_text_ce: 0.03173918277025223 (0.04110877648214265) | > loss_mel_ce: 2.9002418518066406 (nan) | > loss: 0.011634846217930317 (nan) | > current_lr: 5e-06 | > step_time: 0.1565 (0.13655056855376604) | > loader_time: 0.0048 (0.004654489555844754)  --> TIME: 2025-02-24 13:05:23 -- STEP: 35696/36938 -- GLOBAL_STEP: 331200 | > loss_text_ce: 0.042426567524671555 (0.04110886101416949) | > loss_mel_ce: 3.51025390625 (nan) | > loss: 0.014097939245402813 (nan) | > current_lr: 5e-06 | > step_time: 0.1574 (0.1365581036068192) | > loader_time: 0.0046 (0.004654528046338256)  --> TIME: 2025-02-24 13:05:33 -- STEP: 35746/36938 -- GLOBAL_STEP: 331250 | > loss_text_ce: 0.0391380749642849 (0.041107485896498625) | > loss_mel_ce: 3.2288146018981934 (nan) | > loss: 0.012968067079782486 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13656409438546252) | > loader_time: 0.0047 (0.004654500451517768)  --> TIME: 2025-02-24 13:05:42 -- STEP: 35796/36938 -- GLOBAL_STEP: 331300 | > loss_text_ce: 0.05028394237160683 (0.04110702358173027) | > loss_mel_ce: 2.1623971462249756 (nan) | > loss: 0.008780481293797493 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13656613248355498) | > loader_time: 0.0046 (0.004654468984121144)  --> TIME: 2025-02-24 13:05:51 -- STEP: 35846/36938 -- GLOBAL_STEP: 331350 | > loss_text_ce: 0.037415921688079834 (0.041107396176752596) | > loss_mel_ce: 1.7562505006790161 (nan) | > loss: 0.007117724046111107 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13656332053450676) | > loader_time: 0.0046 (0.004654437950371393)  --> TIME: 2025-02-24 13:06:01 -- STEP: 35896/36938 -- GLOBAL_STEP: 331400 | > loss_text_ce: 0.0359501913189888 (0.041107086685580746) | > loss_mel_ce: 3.3190486431121826 (nan) | > loss: 0.013313488103449345 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.13657096691192122) | > loader_time: 0.0048 (0.0046545049582023094)  --> TIME: 2025-02-24 13:06:10 -- STEP: 35946/36938 -- GLOBAL_STEP: 331450 | > loss_text_ce: 0.04177362844347954 (0.04110638790611619) | > loss_mel_ce: 1.6443129777908325 (nan) | > loss: 0.006690820213407278 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13656609889986426) | > loader_time: 0.0045 (0.004654448935618528)  --> TIME: 2025-02-24 13:06:20 -- STEP: 35996/36938 -- GLOBAL_STEP: 331500 | > loss_text_ce: 0.043366458266973495 (0.041103498234146225) | > loss_mel_ce: 2.323836088180542 (nan) | > loss: 0.0093936612829566 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13657037780501824) | > loader_time: 0.0045 (0.0046545092973964645)  --> TIME: 2025-02-24 13:06:29 -- STEP: 36046/36938 -- GLOBAL_STEP: 331550 | > loss_text_ce: 0.042311035096645355 (0.041100498490683104) | > loss_mel_ce: 1.6015625 (nan) | > loss: 0.006523308344185352 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13657188870590228) | > loader_time: 0.0045 (0.004654530500496556)  --> TIME: 2025-02-24 13:06:38 -- STEP: 36096/36938 -- GLOBAL_STEP: 331600 | > loss_text_ce: 0.047226693481206894 (0.04109911286008005) | > loss_mel_ce: 2.8516147136688232 (nan) | > loss: 0.011503339745104313 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13656346410750658) | > loader_time: 0.0047 (0.004654439285080497)  --> TIME: 2025-02-24 13:06:48 -- STEP: 36146/36938 -- GLOBAL_STEP: 331650 | > loss_text_ce: 0.03570065647363663 (0.04109781576755039) | > loss_mel_ce: 3.1547746658325195 (nan) | > loss: 0.012660617008805275 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1365736991666709) | > loader_time: 0.0045 (0.004654498539045892)  --> TIME: 2025-02-24 13:06:57 -- STEP: 36196/36938 -- GLOBAL_STEP: 331700 | > loss_text_ce: 0.04342879727482796 (0.04109813161601371) | > loss_mel_ce: 3.3946895599365234 (nan) | > loss: 0.013643328100442886 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13657608145125572) | > loader_time: 0.0048 (0.004654614157881695)  --> TIME: 2025-02-24 13:07:06 -- STEP: 36246/36938 -- GLOBAL_STEP: 331750 | > loss_text_ce: 0.03349301591515541 (0.041095135023520704) | > loss_mel_ce: 3.47387433052063 (nan) | > loss: 0.013918125070631504 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13657123961735673) | > loader_time: 0.0046 (0.00465451988934196)  --> TIME: 2025-02-24 13:07:16 -- STEP: 36296/36938 -- GLOBAL_STEP: 331800 | > loss_text_ce: 0.04558565467596054 (0.041092762657800826) | > loss_mel_ce: 3.4244792461395264 (nan) | > loss: 0.013770099729299545 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.13657043653225992) | > loader_time: 0.0048 (0.004654574333793783)  --> TIME: 2025-02-24 13:07:25 -- STEP: 36346/36938 -- GLOBAL_STEP: 331850 | > loss_text_ce: 0.03568799048662186 (0.0410925176394417) | > loss_mel_ce: 0.021516814827919006 (nan) | > loss: 0.0002270032127853483 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13657103613136193) | > loader_time: 0.0045 (0.004654559102271219)  --> TIME: 2025-02-24 13:07:34 -- STEP: 36396/36938 -- GLOBAL_STEP: 331900 | > loss_text_ce: 0.05069578438997269 (0.041091988345195424) | > loss_mel_ce: 2.751861572265625 (nan) | > loss: 0.01112126000225544 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.13657434069988436) | > loader_time: 0.0047 (0.004654603576199004)  --> TIME: 2025-02-24 13:07:44 -- STEP: 36446/36938 -- GLOBAL_STEP: 331950 | > loss_text_ce: 0.04223569855093956 (0.04109205438436596) | > loss_mel_ce: 3.3407156467437744 (nan) | > loss: 0.013424410484731197 (nan) | > current_lr: 5e-06 | > step_time: 0.1505 (0.13657726432570189) | > loader_time: 0.0046 (0.00465462495367059)  --> TIME: 2025-02-24 13:07:53 -- STEP: 36496/36938 -- GLOBAL_STEP: 332000 | > loss_text_ce: 0.04361717030405998 (0.04109037745173295) | > loss_mel_ce: 3.41591477394104 (nan) | > loss: 0.013728302903473377 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.1365808552291921) | > loader_time: 0.0047 (0.00465463612723488)  --> TIME: 2025-02-24 13:08:03 -- STEP: 36546/36938 -- GLOBAL_STEP: 332050 | > loss_text_ce: 0.04557766392827034 (0.0410902187887361) | > loss_mel_ce: 1.3575458526611328 (nan) | > loss: 0.005567950662225485 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13658187528721977) | > loader_time: 0.0046 (0.004654625428562557)  --> TIME: 2025-02-24 13:08:12 -- STEP: 36596/36938 -- GLOBAL_STEP: 332100 | > loss_text_ce: 0.04271356388926506 (0.041089490361092156) | > loss_mel_ce: 2.614626884460449 (nan) | > loss: 0.010545002296566963 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.13657929491040333) | > loader_time: 0.0046 (0.004654658115656746)  --> TIME: 2025-02-24 13:08:21 -- STEP: 36646/36938 -- GLOBAL_STEP: 332150 | > loss_text_ce: 0.036021918058395386 (0.04109016711451231) | > loss_mel_ce: 3.101886510848999 (nan) | > loss: 0.012452018447220325 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.1365754943952371) | > loader_time: 0.0047 (0.004654573677265784)  --> TIME: 2025-02-24 13:08:30 -- STEP: 36696/36938 -- GLOBAL_STEP: 332200 | > loss_text_ce: 0.04487666115164757 (0.04108934618900867) | > loss_mel_ce: 3.1805615425109863 (nan) | > loss: 0.012799357995390892 (nan) | > current_lr: 5e-06 | > step_time: 0.1883 (0.13657323153600515) | > loader_time: 0.0048 (0.004654525112216984)  --> TIME: 2025-02-24 13:08:39 -- STEP: 36746/36938 -- GLOBAL_STEP: 332250 | > loss_text_ce: 0.05016870051622391 (0.041090940001274175) | > loss_mel_ce: 3.479645252227783 (nan) | > loss: 0.014007199555635452 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13657142373996803) | > loader_time: 0.0047 (0.004654472883684708)  --> TIME: 2025-02-24 13:08:49 -- STEP: 36796/36938 -- GLOBAL_STEP: 332300 | > loss_text_ce: 0.047117751091718674 (0.04108857041597306) | > loss_mel_ce: 3.2101218700408936 (nan) | > loss: 0.012925554998219013 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.13657778414037716) | > loader_time: 0.0049 (0.0046544762937539945)  --> TIME: 2025-02-24 13:08:59 -- STEP: 36846/36938 -- GLOBAL_STEP: 332350 | > loss_text_ce: 0.046033117920160294 (0.041088000262466604) | > loss_mel_ce: 2.485325336456299 (nan) | > loss: 0.010045073926448822 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13658636181063155) | > loader_time: 0.0045 (0.004654502503706158)  --> TIME: 2025-02-24 13:09:08 -- STEP: 36896/36938 -- GLOBAL_STEP: 332400 | > loss_text_ce: 0.031825046986341476 (0.041087145508477606) | > loss_mel_ce: 2.856858968734741 (nan) | > loss: 0.011463032104074955 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.13658351606890456) | > loader_time: 0.0046 (0.004654463700439673)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.020726638265175405 (-0.0004306453289371834) | > avg_loss_text_ce: 0.03971957007773442 (-0.000964720336960094) | > avg_loss_mel_ce: 2.7995588885675566 (-0.03975565775786194) | > avg_loss: 2.839278453057354 (-0.040720382539352507) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_332442.pth  > EPOCH: 10/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-24 13:09:41)   --> TIME: 2025-02-24 13:09:43 -- STEP: 8/36938 -- GLOBAL_STEP: 332450 | > loss_text_ce: 0.029057621955871582 (0.03515444602817297) | > loss_mel_ce: 3.4347972869873047 (2.8695815801620483) | > loss: 0.013745456002652645 (0.011526730901096016) | > current_lr: 5e-06 | > step_time: 0.1633 (0.1420532763004303) | > loader_time: 0.0046 (0.0050397515296936035)  --> TIME: 2025-02-24 13:09:52 -- STEP: 58/36938 -- GLOBAL_STEP: 332500 | > loss_text_ce: 0.030859047546982765 (0.04041565980376868) | > loss_mel_ce: 2.4365015029907227 (nan) | > loss: 0.009791113436222076 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13627811958049899) | > loader_time: 0.0048 (0.004825111093192267)  --> TIME: 2025-02-24 13:10:01 -- STEP: 108/36938 -- GLOBAL_STEP: 332550 | > loss_text_ce: 0.05073358863592148 (0.041112431980393535) | > loss_mel_ce: 3.433067560195923 (nan) | > loss: 0.01382460817694664 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13652549408100262) | > loader_time: 0.0047 (0.004839031784622761)  --> TIME: 2025-02-24 13:10:10 -- STEP: 158/36938 -- GLOBAL_STEP: 332600 | > loss_text_ce: 0.051006708294153214 (0.040686276705959167) | > loss_mel_ce: 1.8093483448028564 (nan) | > loss: 0.007382361683994532 (nan) | > current_lr: 5e-06 | > step_time: 0.1738 (0.13482778434511977) | > loader_time: 0.0048 (0.004791449896896944)  --> TIME: 2025-02-24 13:10:19 -- STEP: 208/36938 -- GLOBAL_STEP: 332650 | > loss_text_ce: 0.04948268458247185 (0.04080018350102295) | > loss_mel_ce: 3.450038194656372 (nan) | > loss: 0.013886988162994385 (nan) | > current_lr: 5e-06 | > step_time: 0.1626 (0.13493214318385494) | > loader_time: 0.0051 (0.004791372097455542)  --> TIME: 2025-02-24 13:10:29 -- STEP: 258/36938 -- GLOBAL_STEP: 332700 | > loss_text_ce: 0.0402977354824543 (0.04070898863060993) | > loss_mel_ce: 3.15978741645813 (nan) | > loss: 0.01269875094294548 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13519289049991343) | > loader_time: 0.0046 (0.004793080248573957)  --> TIME: 2025-02-24 13:10:38 -- STEP: 308/36938 -- GLOBAL_STEP: 332750 | > loss_text_ce: 0.03975571319460869 (0.040464516844019895) | > loss_mel_ce: 2.742900848388672 (nan) | > loss: 0.011042289435863495 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13522947995693652) | > loader_time: 0.0046 (0.004767991505660021)  --> TIME: 2025-02-24 13:10:47 -- STEP: 358/36938 -- GLOBAL_STEP: 332800 | > loss_text_ce: 0.045695800334215164 (0.040584162038922636) | > loss_mel_ce: 2.9373281002044678 (nan) | > loss: 0.011837396770715714 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.135756416693746) | > loader_time: 0.0045 (0.004758769573446089)  --> TIME: 2025-02-24 13:10:56 -- STEP: 408/36938 -- GLOBAL_STEP: 332850 | > loss_text_ce: 0.04051032289862633 (0.04051991616485312) | > loss_mel_ce: 3.2523930072784424 (nan) | > loss: 0.013067077845335007 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.13530301229626518) | > loader_time: 0.0046 (0.004744160409067186)  --> TIME: 2025-02-24 13:11:06 -- STEP: 458/36938 -- GLOBAL_STEP: 332900 | > loss_text_ce: 0.04816621541976929 (0.04062847924056792) | > loss_mel_ce: 2.0814425945281982 (nan) | > loss: 0.008450829423964024 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13577134432230453) | > loader_time: 0.0047 (0.004752709355416772)  --> TIME: 2025-02-24 13:11:15 -- STEP: 508/36938 -- GLOBAL_STEP: 332950 | > loss_text_ce: 0.034542016685009 (0.04065572639824602) | > loss_mel_ce: 3.5031118392944336 (nan) | > loss: 0.014038310386240482 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13600918439429577) | > loader_time: 0.0052 (0.004759430885314937)  --> TIME: 2025-02-24 13:11:25 -- STEP: 558/36938 -- GLOBAL_STEP: 333000 | > loss_text_ce: 0.03129081055521965 (0.040600867451278756) | > loss_mel_ce: 3.4355719089508057 (nan) | > loss: 0.013757392764091492 (nan) | > current_lr: 5e-06 | > step_time: 0.2004 (0.13620414656977498) | > loader_time: 0.0049 (0.004751506672110604)  --> TIME: 2025-02-24 13:11:34 -- STEP: 608/36938 -- GLOBAL_STEP: 333050 | > loss_text_ce: 0.03844607621431351 (0.04075392877290907) | > loss_mel_ce: 3.2740392684936523 (nan) | > loss: 0.013144784606993198 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.136482944221873) | > loader_time: 0.0048 (0.0047511728970628)  --> TIME: 2025-02-24 13:11:43 -- STEP: 658/36938 -- GLOBAL_STEP: 333100 | > loss_text_ce: 0.04318918287754059 (0.040737474691066165) | > loss_mel_ce: 3.3978397846221924 (nan) | > loss: 0.013654878363013268 (nan) | > current_lr: 5e-06 | > step_time: 0.1556 (0.13638176200600013) | > loader_time: 0.0047 (0.004758867086972872)  --> TIME: 2025-02-24 13:11:53 -- STEP: 708/36938 -- GLOBAL_STEP: 333150 | > loss_text_ce: 0.031977809965610504 (0.04062450186765128) | > loss_mel_ce: 2.8395450115203857 (nan) | > loss: 0.011394932866096497 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13635403726060522) | > loader_time: 0.0045 (0.004750361887075134)  --> TIME: 2025-02-24 13:12:02 -- STEP: 758/36938 -- GLOBAL_STEP: 333200 | > loss_text_ce: 0.04672432318329811 (0.040693407536712814) | > loss_mel_ce: 2.803272008895874 (nan) | > loss: 0.011309510096907616 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.13613866470105435) | > loader_time: 0.0048 (0.004748622786086595)  --> TIME: 2025-02-24 13:12:11 -- STEP: 808/36938 -- GLOBAL_STEP: 333250 | > loss_text_ce: 0.052999161183834076 (0.040743876658933274) | > loss_mel_ce: 3.1077864170074463 (nan) | > loss: 0.012542800977826118 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13582743512521897) | > loader_time: 0.0045 (0.004749222262070908)  --> TIME: 2025-02-24 13:12:20 -- STEP: 858/36938 -- GLOBAL_STEP: 333300 | > loss_text_ce: 0.04034535214304924 (0.04068509674259831) | > loss_mel_ce: 2.270129919052124 (nan) | > loss: 0.009168553166091442 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13592167389698537) | > loader_time: 0.0053 (0.004750144787323778)  --> TIME: 2025-02-24 13:12:29 -- STEP: 908/36938 -- GLOBAL_STEP: 333350 | > loss_text_ce: 0.04517999663949013 (0.04065128080423772) | > loss_mel_ce: 0.0894063338637352 (nan) | > loss: 0.0005340728093869984 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13594267783186) | > loader_time: 0.0045 (0.004744788098440293)  --> TIME: 2025-02-24 13:12:39 -- STEP: 958/36938 -- GLOBAL_STEP: 333400 | > loss_text_ce: 0.03771824389696121 (0.04060120610734116) | > loss_mel_ce: 3.507317066192627 (nan) | > loss: 0.014067601412534714 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13604722466000957) | > loader_time: 0.0046 (0.004739177998520882)  --> TIME: 2025-02-24 13:12:48 -- STEP: 1008/36938 -- GLOBAL_STEP: 333450 | > loss_text_ce: 0.0401015542447567 (0.04058031230566223) | > loss_mel_ce: 0.19681549072265625 (nan) | > loss: 0.0009401470888406038 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13611898228289612) | > loader_time: 0.0046 (0.004736098741728159)  --> TIME: 2025-02-24 13:12:57 -- STEP: 1058/36938 -- GLOBAL_STEP: 333500 | > loss_text_ce: 0.05547801032662392 (0.04063115344296554) | > loss_mel_ce: 3.0662107467651367 (nan) | > loss: 0.01238765474408865 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13588571052695514) | > loader_time: 0.0045 (0.00473545922674168)  --> TIME: 2025-02-24 13:13:06 -- STEP: 1108/36938 -- GLOBAL_STEP: 333550 | > loss_text_ce: 0.03276791051030159 (0.040613410701901234) | > loss_mel_ce: 1.2408398389816284 (nan) | > loss: 0.005053999368101358 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13578568705582897) | > loader_time: 0.0045 (0.0047303298320150525)  --> TIME: 2025-02-24 13:13:16 -- STEP: 1158/36938 -- GLOBAL_STEP: 333600 | > loss_text_ce: 0.02839007042348385 (0.040575699869459664) | > loss_mel_ce: 0.0281982421875 (nan) | > loss: 0.0002245568175567314 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.1359990639595995) | > loader_time: 0.0045 (0.004726592740864333)  --> TIME: 2025-02-24 13:13:25 -- STEP: 1208/36938 -- GLOBAL_STEP: 333650 | > loss_text_ce: 0.03898952528834343 (0.040550528945314926) | > loss_mel_ce: 2.823826789855957 (nan) | > loss: 0.011360382661223412 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13603250297489555) | > loader_time: 0.0045 (0.004724583088956922)  --> TIME: 2025-02-24 13:13:34 -- STEP: 1258/36938 -- GLOBAL_STEP: 333700 | > loss_text_ce: 0.0413525328040123 (0.04047619853544803) | > loss_mel_ce: 2.4428372383117676 (nan) | > loss: 0.009857896715402603 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.1357639023533687) | > loader_time: 0.0046 (0.004727507812609166)  --> TIME: 2025-02-24 13:13:44 -- STEP: 1308/36938 -- GLOBAL_STEP: 333750 | > loss_text_ce: 0.03882533311843872 (0.040455765812159324) | > loss_mel_ce: 1.6879781484603882 (nan) | > loss: 0.006852395366877317 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.1359069793960736) | > loader_time: 0.0052 (0.004733369073372011)  --> TIME: 2025-02-24 13:13:53 -- STEP: 1358/36938 -- GLOBAL_STEP: 333800 | > loss_text_ce: 0.043370794504880905 (0.04043471746884599) | > loss_mel_ce: 2.297375440597534 (nan) | > loss: 0.009288676083087921 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13598441024240948) | > loader_time: 0.0045 (0.004738226203749921)  --> TIME: 2025-02-24 13:14:02 -- STEP: 1408/36938 -- GLOBAL_STEP: 333850 | > loss_text_ce: 0.04003477096557617 (0.040457540434462376) | > loss_mel_ce: 2.9724578857421875 (nan) | > loss: 0.011954336427152157 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.1359653030945497) | > loader_time: 0.0053 (0.0047361451116475295)  --> TIME: 2025-02-24 13:14:11 -- STEP: 1458/36938 -- GLOBAL_STEP: 333900 | > loss_text_ce: 0.03945381939411163 (0.040415179586761045) | > loss_mel_ce: 3.15975022315979 (nan) | > loss: 0.012695254758000374 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13594845047703508) | > loader_time: 0.0046 (0.0047304890283341346)  --> TIME: 2025-02-24 13:14:21 -- STEP: 1508/36938 -- GLOBAL_STEP: 333950 | > loss_text_ce: 0.0402158759534359 (0.04035588960405386) | > loss_mel_ce: 2.081310987472534 (nan) | > loss: 0.008418758399784565 (nan) | > current_lr: 5e-06 | > step_time: 0.1082 (0.13604805782555918) | > loader_time: 0.0044 (0.004733448951883097)  --> TIME: 2025-02-24 13:14:30 -- STEP: 1558/36938 -- GLOBAL_STEP: 334000 | > loss_text_ce: 0.03701420873403549 (0.04037677226898966) | > loss_mel_ce: 2.880262613296509 (nan) | > loss: 0.011576496064662933 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13612738301420407) | > loader_time: 0.0046 (0.004733660254766157)  --> TIME: 2025-02-24 13:14:40 -- STEP: 1608/36938 -- GLOBAL_STEP: 334050 | > loss_text_ce: 0.03156605735421181 (0.040405066079802925) | > loss_mel_ce: 1.6068551540374756 (nan) | > loss: 0.0065016718581318855 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13617044583481935) | > loader_time: 0.0045 (0.004736011745917868)  --> TIME: 2025-02-24 13:14:49 -- STEP: 1658/36938 -- GLOBAL_STEP: 334100 | > loss_text_ce: 0.04221925511956215 (0.04037629050540816) | > loss_mel_ce: 3.4834909439086914 (nan) | > loss: 0.013990913517773151 (nan) | > current_lr: 5e-06 | > step_time: 0.1753 (0.13623356574694698) | > loader_time: 0.0047 (0.004736273510177129)  --> TIME: 2025-02-24 13:14:59 -- STEP: 1708/36938 -- GLOBAL_STEP: 334150 | > loss_text_ce: 0.040939636528491974 (0.040370910166189114) | > loss_mel_ce: 1.7394810914993286 (nan) | > loss: 0.0070651620626449585 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.1363002870066105) | > loader_time: 0.0051 (0.004734468823015286)  --> TIME: 2025-02-24 13:15:08 -- STEP: 1758/36938 -- GLOBAL_STEP: 334200 | > loss_text_ce: 0.0413530208170414 (0.04037671341307551) | > loss_mel_ce: 3.277386426925659 (nan) | > loss: 0.013169601559638977 (nan) | > current_lr: 5e-06 | > step_time: 0.1859 (0.13635270248365355) | > loader_time: 0.0048 (0.004736352563582468)  --> TIME: 2025-02-24 13:15:17 -- STEP: 1808/36938 -- GLOBAL_STEP: 334250 | > loss_text_ce: 0.03887363150715828 (0.04037285793780766) | > loss_mel_ce: 2.2309579849243164 (nan) | > loss: 0.009007268585264683 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13639476512913165) | > loader_time: 0.0052 (0.004736861164590968)  --> TIME: 2025-02-24 13:15:27 -- STEP: 1858/36938 -- GLOBAL_STEP: 334300 | > loss_text_ce: 0.03743840754032135 (0.04034893302872485) | > loss_mel_ce: 3.119016647338867 (nan) | > loss: 0.012525616213679314 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13652698113407616) | > loader_time: 0.0046 (0.00473798950983712)  --> TIME: 2025-02-24 13:15:36 -- STEP: 1908/36938 -- GLOBAL_STEP: 334350 | > loss_text_ce: 0.04132743179798126 (0.04036167614304495) | > loss_mel_ce: 3.4916818141937256 (nan) | > loss: 0.014019878581166267 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.1365419227622093) | > loader_time: 0.0046 (0.004739594159636121)  --> TIME: 2025-02-24 13:15:46 -- STEP: 1958/36938 -- GLOBAL_STEP: 334400 | > loss_text_ce: 0.03769371286034584 (0.040371407692409086) | > loss_mel_ce: 3.0184123516082764 (nan) | > loss: 0.0121274059638381 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13649110616288493) | > loader_time: 0.0047 (0.004743223779656926)  --> TIME: 2025-02-24 13:15:55 -- STEP: 2008/36938 -- GLOBAL_STEP: 334450 | > loss_text_ce: 0.03375055268406868 (0.04036830980247218) | > loss_mel_ce: 2.7713675498962402 (nan) | > loss: 0.01113142166286707 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13651217965490817) | > loader_time: 0.0045 (0.004743445679486048)  --> TIME: 2025-02-24 13:16:05 -- STEP: 2058/36938 -- GLOBAL_STEP: 334500 | > loss_text_ce: 0.02993365004658699 (0.04035010087947323) | > loss_mel_ce: 3.2627220153808594 (nan) | > loss: 0.013066094368696213 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13664086937788536) | > loader_time: 0.0045 (0.004743741251405185)  --> TIME: 2025-02-24 13:16:14 -- STEP: 2108/36938 -- GLOBAL_STEP: 334550 | > loss_text_ce: 0.05443190038204193 (0.04033951517026621) | > loss_mel_ce: 3.1996541023254395 (nan) | > loss: 0.012913040816783905 (nan) | > current_lr: 5e-06 | > step_time: 0.1736 (0.13661806237313082) | > loader_time: 0.0047 (0.0047442863290631234)  --> TIME: 2025-02-24 13:16:23 -- STEP: 2158/36938 -- GLOBAL_STEP: 334600 | > loss_text_ce: 0.042287372052669525 (0.04037435502426712) | > loss_mel_ce: 2.4683632850646973 (nan) | > loss: 0.0099628996104002 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13662831113778218) | > loader_time: 0.0044 (0.004745109640303544)  --> TIME: 2025-02-24 13:16:32 -- STEP: 2208/36938 -- GLOBAL_STEP: 334650 | > loss_text_ce: 0.035099759697914124 (0.0403984455932505) | > loss_mel_ce: 1.8339205980300903 (nan) | > loss: 0.007416747976094484 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13662952735372244) | > loader_time: 0.0045 (0.004743324997632409)  --> TIME: 2025-02-24 13:16:42 -- STEP: 2258/36938 -- GLOBAL_STEP: 334700 | > loss_text_ce: 0.039025723934173584 (0.040368113550873046) | > loss_mel_ce: 3.2590909004211426 (nan) | > loss: 0.01308776531368494 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13672054183493706) | > loader_time: 0.0046 (0.004743595689244782)  --> TIME: 2025-02-24 13:16:51 -- STEP: 2308/36938 -- GLOBAL_STEP: 334750 | > loss_text_ce: 0.039880018681287766 (0.040364754796625275) | > loss_mel_ce: 2.780268669128418 (nan) | > loss: 0.011191067285835743 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13671931526276668) | > loader_time: 0.0046 (0.004742803462043385)  --> TIME: 2025-02-24 13:17:01 -- STEP: 2358/36938 -- GLOBAL_STEP: 334800 | > loss_text_ce: 0.03443540632724762 (0.04035880338152557) | > loss_mel_ce: 3.6286635398864746 (nan) | > loss: 0.014536107890307903 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13678613651395377) | > loader_time: 0.0046 (0.004741376188470717)  --> TIME: 2025-02-24 13:17:10 -- STEP: 2408/36938 -- GLOBAL_STEP: 334850 | > loss_text_ce: 0.033163972198963165 (0.04035575464530297) | > loss_mel_ce: 1.1684489250183105 (nan) | > loss: 0.004768305458128452 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13684045862517913) | > loader_time: 0.0046 (0.004739398081041254)  --> TIME: 2025-02-24 13:17:19 -- STEP: 2458/36938 -- GLOBAL_STEP: 334900 | > loss_text_ce: 0.03872590512037277 (0.040324298841113305) | > loss_mel_ce: 3.447628974914551 (nan) | > loss: 0.013834741897881031 (nan) | > current_lr: 5e-06 | > step_time: 0.1833 (0.13678012658562316) | > loader_time: 0.0048 (0.004737159505522862)  --> TIME: 2025-02-24 13:17:29 -- STEP: 2508/36938 -- GLOBAL_STEP: 334950 | > loss_text_ce: 0.04849785193800926 (0.04031653006923517) | > loss_mel_ce: 3.0672924518585205 (nan) | > loss: 0.012364247813820839 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13676748197805028) | > loader_time: 0.0045 (0.004735690649996915)  --> TIME: 2025-02-24 13:17:38 -- STEP: 2558/36938 -- GLOBAL_STEP: 335000 | > loss_text_ce: 0.03704153746366501 (0.04031856613527854) | > loss_mel_ce: 2.2380869388580322 (nan) | > loss: 0.00902828760445118 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13681030804706454) | > loader_time: 0.0045 (0.0047334757887636724)  --> TIME: 2025-02-24 13:17:48 -- STEP: 2608/36938 -- GLOBAL_STEP: 335050 | > loss_text_ce: 0.05114244297146797 (0.04032319531157502) | > loss_mel_ce: 2.293937921524048 (nan) | > loss: 0.009305874817073345 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1368922131376035) | > loader_time: 0.0044 (0.004732952436055145)  --> TIME: 2025-02-24 13:17:57 -- STEP: 2658/36938 -- GLOBAL_STEP: 335100 | > loss_text_ce: 0.03541716933250427 (0.040343476282439644) | > loss_mel_ce: 3.736567258834839 (nan) | > loss: 0.014968193136155605 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13682789180043803) | > loader_time: 0.0052 (0.004734370114661589)  --> TIME: 2025-02-24 13:18:06 -- STEP: 2708/36938 -- GLOBAL_STEP: 335150 | > loss_text_ce: 0.0362420417368412 (0.04037287671076328) | > loss_mel_ce: 3.3555245399475098 (nan) | > loss: 0.01345939189195633 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13675704189030116) | > loader_time: 0.0046 (0.004735280703230476)  --> TIME: 2025-02-24 13:18:15 -- STEP: 2758/36938 -- GLOBAL_STEP: 335200 | > loss_text_ce: 0.04277688264846802 (0.040392243977030944) | > loss_mel_ce: 3.690504789352417 (nan) | > loss: 0.014814610593020916 (nan) | > current_lr: 5e-06 | > step_time: 0.1504 (0.13669925119844056) | > loader_time: 0.0045 (0.004735118500818103)  --> TIME: 2025-02-24 13:18:24 -- STEP: 2808/36938 -- GLOBAL_STEP: 335250 | > loss_text_ce: 0.03921843320131302 (0.04041351735203171) | > loss_mel_ce: 2.9391417503356934 (nan) | > loss: 0.01181889045983553 (nan) | > current_lr: 5e-06 | > step_time: 0.1527 (0.13667045385070042) | > loader_time: 0.0053 (0.004735524922694226)  --> TIME: 2025-02-24 13:18:33 -- STEP: 2858/36938 -- GLOBAL_STEP: 335300 | > loss_text_ce: 0.04672258347272873 (0.040471888369860404) | > loss_mel_ce: 3.008594036102295 (nan) | > loss: 0.012124272994697094 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13662443918477796) | > loader_time: 0.0046 (0.004734280561549908)  --> TIME: 2025-02-24 13:18:42 -- STEP: 2908/36938 -- GLOBAL_STEP: 335350 | > loss_text_ce: 0.0433616116642952 (0.04046688765444968) | > loss_mel_ce: 2.383422613143921 (nan) | > loss: 0.009630097076296806 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13652174394101235) | > loader_time: 0.0045 (0.00473201889775313)  --> TIME: 2025-02-24 13:18:51 -- STEP: 2958/36938 -- GLOBAL_STEP: 335400 | > loss_text_ce: 0.037657756358385086 (0.04046597063857523) | > loss_mel_ce: 3.5610499382019043 (nan) | > loss: 0.014280586503446102 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13647845663518213) | > loader_time: 0.0046 (0.004730952600114153)  --> TIME: 2025-02-24 13:19:01 -- STEP: 3008/36938 -- GLOBAL_STEP: 335450 | > loss_text_ce: 0.026257406920194626 (0.04046553656851446) | > loss_mel_ce: 3.010406494140625 (nan) | > loss: 0.01205025427043438 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1364833796595008) | > loader_time: 0.0045 (0.0047290897432793865)  --> TIME: 2025-02-24 13:19:10 -- STEP: 3058/36938 -- GLOBAL_STEP: 335500 | > loss_text_ce: 0.0437958799302578 (0.040464358896740694) | > loss_mel_ce: 2.359564781188965 (nan) | > loss: 0.009537145495414734 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13647538380657182) | > loader_time: 0.0045 (0.004728624055087752)  --> TIME: 2025-02-24 13:19:19 -- STEP: 3108/36938 -- GLOBAL_STEP: 335550 | > loss_text_ce: 0.04137473180890083 (0.04046203776926732) | > loss_mel_ce: 2.7939035892486572 (nan) | > loss: 0.01125110499560833 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13652949121467864) | > loader_time: 0.0045 (0.004728772081770332)  --> TIME: 2025-02-24 13:19:29 -- STEP: 3158/36938 -- GLOBAL_STEP: 335600 | > loss_text_ce: 0.05002812296152115 (0.04043792544624064) | > loss_mel_ce: 3.3465731143951416 (nan) | > loss: 0.013478577136993408 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13651100829706095) | > loader_time: 0.0047 (0.00472673862474186)  --> TIME: 2025-02-24 13:19:38 -- STEP: 3208/36938 -- GLOBAL_STEP: 335650 | > loss_text_ce: 0.04204479977488518 (0.04045807149178789) | > loss_mel_ce: 2.220777988433838 (nan) | > loss: 0.00897945649921894 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.1365190183283982) | > loader_time: 0.0046 (0.0047257359039753685)  --> TIME: 2025-02-24 13:19:47 -- STEP: 3258/36938 -- GLOBAL_STEP: 335700 | > loss_text_ce: 0.041875455528497696 (0.040440903446294654) | > loss_mel_ce: 3.580652952194214 (nan) | > loss: 0.01437511295080185 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.13645718008407887) | > loader_time: 0.0049 (0.004723974939794754)  --> TIME: 2025-02-24 13:19:56 -- STEP: 3308/36938 -- GLOBAL_STEP: 335750 | > loss_text_ce: 0.040748052299022675 (0.04045687696644239) | > loss_mel_ce: 2.899050712585449 (nan) | > loss: 0.011665869504213333 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13640492340097135) | > loader_time: 0.0045 (0.004724673737637794)  --> TIME: 2025-02-24 13:20:05 -- STEP: 3358/36938 -- GLOBAL_STEP: 335800 | > loss_text_ce: 0.04833884909749031 (0.040460062843091255) | > loss_mel_ce: 3.177908420562744 (nan) | > loss: 0.012802569195628166 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13639141442592556) | > loader_time: 0.0045 (0.004725232374249214)  --> TIME: 2025-02-24 13:20:15 -- STEP: 3408/36938 -- GLOBAL_STEP: 335850 | > loss_text_ce: 0.039393823593854904 (0.04046497577249332) | > loss_mel_ce: 2.888202428817749 (nan) | > loss: 0.011617446318268776 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13644546927980422) | > loader_time: 0.0066 (0.004727370022608073)  --> TIME: 2025-02-24 13:20:24 -- STEP: 3458/36938 -- GLOBAL_STEP: 335900 | > loss_text_ce: 0.05637931078672409 (0.04047196160785386) | > loss_mel_ce: 2.3201730251312256 (nan) | > loss: 0.009430764243006706 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13637235131547806) | > loader_time: 0.0047 (0.004727293536316935)  --> TIME: 2025-02-24 13:20:33 -- STEP: 3508/36938 -- GLOBAL_STEP: 335950 | > loss_text_ce: 0.04832055792212486 (0.04047147171415747) | > loss_mel_ce: 4.127683639526367 (nan) | > loss: 0.016571447253227234 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13636808819482657) | > loader_time: 0.0046 (0.004725555533558089)  --> TIME: 2025-02-24 13:20:43 -- STEP: 3558/36938 -- GLOBAL_STEP: 336000 | > loss_text_ce: 0.041424572467803955 (0.04046942818802042) | > loss_mel_ce: 3.221611976623535 (nan) | > loss: 0.012948558665812016 (nan) | > current_lr: 5e-06 | > step_time: 0.1446 (0.13646777738943214) | > loader_time: 0.0046 (0.0047243954158351885)  --> TIME: 2025-02-24 13:20:52 -- STEP: 3608/36938 -- GLOBAL_STEP: 336050 | > loss_text_ce: 0.04220994934439659 (0.040470869017882646) | > loss_mel_ce: 3.6323189735412598 (nan) | > loss: 0.014581464231014252 (nan) | > current_lr: 5e-06 | > step_time: 0.1521 (0.1364411374021264) | > loader_time: 0.0047 (0.004723028637088845)  --> TIME: 2025-02-24 13:21:02 -- STEP: 3658/36938 -- GLOBAL_STEP: 336100 | > loss_text_ce: 0.047398582100868225 (0.04046593854527671) | > loss_mel_ce: 3.6238534450531006 (nan) | > loss: 0.01456846110522747 (nan) | > current_lr: 5e-06 | > step_time: 0.1916 (0.1365830987577819) | > loader_time: 0.0048 (0.004722344347399522)  --> TIME: 2025-02-24 13:21:12 -- STEP: 3708/36938 -- GLOBAL_STEP: 336150 | > loss_text_ce: 0.04671446233987808 (0.04048408822512141) | > loss_mel_ce: 2.538994550704956 (nan) | > loss: 0.01026075053960085 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13663483206122437) | > loader_time: 0.0046 (0.004721002349832934)  --> TIME: 2025-02-24 13:21:21 -- STEP: 3758/36938 -- GLOBAL_STEP: 336200 | > loss_text_ce: 0.04111143574118614 (0.04048464920777212) | > loss_mel_ce: 3.7391133308410645 (nan) | > loss: 0.01500089280307293 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.1366064179254504) | > loader_time: 0.0049 (0.00472042039076402)  --> TIME: 2025-02-24 13:21:30 -- STEP: 3808/36938 -- GLOBAL_STEP: 336250 | > loss_text_ce: 0.031382106244564056 (0.040474140457270935) | > loss_mel_ce: 2.6642770767211914 (nan) | > loss: 0.010697061195969582 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13664244422141236) | > loader_time: 0.0048 (0.004719245834510861)  --> TIME: 2025-02-24 13:21:39 -- STEP: 3858/36938 -- GLOBAL_STEP: 336300 | > loss_text_ce: 0.041959457099437714 (0.040470420887202284) | > loss_mel_ce: 3.025970935821533 (nan) | > loss: 0.012174327857792377 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.1365807565028584) | > loader_time: 0.0046 (0.004717616357600639)  --> TIME: 2025-02-24 13:21:49 -- STEP: 3908/36938 -- GLOBAL_STEP: 336350 | > loss_text_ce: 0.03924292325973511 (0.04044968386928046) | > loss_mel_ce: 2.894557476043701 (nan) | > loss: 0.011642065830528736 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.1366192766807462) | > loader_time: 0.0045 (0.004716763781915539)  --> TIME: 2025-02-24 13:21:58 -- STEP: 3958/36938 -- GLOBAL_STEP: 336400 | > loss_text_ce: 0.036827873438596725 (0.04044116450193189) | > loss_mel_ce: 2.7657785415649414 (nan) | > loss: 0.011121454648673534 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13665424837735995) | > loader_time: 0.0045 (0.004719236633161989)  --> TIME: 2025-02-24 13:22:08 -- STEP: 4008/36938 -- GLOBAL_STEP: 336450 | > loss_text_ce: 0.041119419038295746 (0.04044334799727006) | > loss_mel_ce: 2.442626953125 (nan) | > loss: 0.00985613651573658 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13665848726521942) | > loader_time: 0.0045 (0.004720795356822817)  --> TIME: 2025-02-24 13:22:17 -- STEP: 4058/36938 -- GLOBAL_STEP: 336500 | > loss_text_ce: 0.04101575165987015 (0.04045490265335979) | > loss_mel_ce: 2.4445037841796875 (nan) | > loss: 0.009863173589110374 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13671814907707558) | > loader_time: 0.0045 (0.004722372365738142)  --> TIME: 2025-02-24 13:22:26 -- STEP: 4108/36938 -- GLOBAL_STEP: 336550 | > loss_text_ce: 0.04302079603075981 (0.04045223396705844) | > loss_mel_ce: 2.7503061294555664 (nan) | > loss: 0.011084631085395813 (nan) | > current_lr: 5e-06 | > step_time: 0.1854 (0.1366960228708261) | > loader_time: 0.0048 (0.004721380893536406)  --> TIME: 2025-02-24 13:22:36 -- STEP: 4158/36938 -- GLOBAL_STEP: 336600 | > loss_text_ce: 0.03307301551103592 (0.04043030898463468) | > loss_mel_ce: 3.4143524169921875 (nan) | > loss: 0.013680260628461838 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13664881623439223) | > loader_time: 0.0052 (0.0047216246253082815)  --> TIME: 2025-02-24 13:22:45 -- STEP: 4208/36938 -- GLOBAL_STEP: 336650 | > loss_text_ce: 0.04385417327284813 (0.040431457821115124) | > loss_mel_ce: 3.18925142288208 (nan) | > loss: 0.012829785235226154 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.13667450357979236) | > loader_time: 0.0048 (0.004721487769609163)  --> TIME: 2025-02-24 13:22:54 -- STEP: 4258/36938 -- GLOBAL_STEP: 336700 | > loss_text_ce: 0.041029542684555054 (0.04043605270961318) | > loss_mel_ce: 3.0933032035827637 (nan) | > loss: 0.012437828816473484 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13667689264520558) | > loader_time: 0.0045 (0.004720390598893775)  --> TIME: 2025-02-24 13:23:03 -- STEP: 4308/36938 -- GLOBAL_STEP: 336750 | > loss_text_ce: 0.03183886036276817 (0.040429293429444436) | > loss_mel_ce: 3.3636913299560547 (nan) | > loss: 0.013474327512085438 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13663823607904324) | > loader_time: 0.0046 (0.004720730314458417)  --> TIME: 2025-02-24 13:23:13 -- STEP: 4358/36938 -- GLOBAL_STEP: 336800 | > loss_text_ce: 0.04845047742128372 (0.04042224428104004) | > loss_mel_ce: 2.6144750118255615 (nan) | > loss: 0.010567164979875088 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.1366535559768466) | > loader_time: 0.0053 (0.004721139865820966)  --> TIME: 2025-02-24 13:23:22 -- STEP: 4408/36938 -- GLOBAL_STEP: 336850 | > loss_text_ce: 0.0481637641787529 (0.04042611712570424) | > loss_mel_ce: 3.8455402851104736 (nan) | > loss: 0.01545120682567358 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13669296477104917) | > loader_time: 0.0047 (0.004720375888360601)  --> TIME: 2025-02-24 13:23:32 -- STEP: 4458/36938 -- GLOBAL_STEP: 336900 | > loss_text_ce: 0.03320673853158951 (0.04041076926623142) | > loss_mel_ce: 1.2067315578460693 (nan) | > loss: 0.004920389968901873 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1366805986953776) | > loader_time: 0.0045 (0.004719025460846914)  --> TIME: 2025-02-24 13:23:41 -- STEP: 4508/36938 -- GLOBAL_STEP: 336950 | > loss_text_ce: 0.04849421977996826 (0.04042100772768187) | > loss_mel_ce: 3.09014892578125 (nan) | > loss: 0.012454934418201447 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.13667625818193319) | > loader_time: 0.0046 (0.004718645177484081)  --> TIME: 2025-02-24 13:23:50 -- STEP: 4558/36938 -- GLOBAL_STEP: 337000 | > loss_text_ce: 0.03651077672839165 (0.04040813639685643) | > loss_mel_ce: 2.909193992614746 (nan) | > loss: 0.011689305305480957 (nan) | > current_lr: 5e-06 | > step_time: 0.1503 (0.13666574910001916) | > loader_time: 0.0053 (0.004718286523488376)  --> TIME: 2025-02-24 13:24:00 -- STEP: 4608/36938 -- GLOBAL_STEP: 337050 | > loss_text_ce: 0.04059534892439842 (0.04040686596413759) | > loss_mel_ce: 2.9882383346557617 (nan) | > loss: 0.012019181624054909 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13668739878469027) | > loader_time: 0.0045 (0.004719275101605378)  --> TIME: 2025-02-24 13:24:09 -- STEP: 4658/36938 -- GLOBAL_STEP: 337100 | > loss_text_ce: 0.04651468247175217 (0.040414138448961676) | > loss_mel_ce: 1.64263916015625 (nan) | > loss: 0.006702991668134928 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.1366818684168053) | > loader_time: 0.0045 (0.004718548548277883)  --> TIME: 2025-02-24 13:24:18 -- STEP: 4708/36938 -- GLOBAL_STEP: 337150 | > loss_text_ce: 0.02563495747745037 (0.0403999103573068) | > loss_mel_ce: 0.9566445350646973 (nan) | > loss: 0.0038979346863925457 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.1366633198579286) | > loader_time: 0.0046 (0.00471911393897251)  --> TIME: 2025-02-24 13:24:27 -- STEP: 4758/36938 -- GLOBAL_STEP: 337200 | > loss_text_ce: 0.03862946853041649 (0.04040628862001564) | > loss_mel_ce: 3.6729583740234375 (nan) | > loss: 0.014728524722158909 (nan) | > current_lr: 5e-06 | > step_time: 0.1987 (0.1366636602254213) | > loader_time: 0.0048 (0.004720556781090513)  --> TIME: 2025-02-24 13:24:37 -- STEP: 4808/36938 -- GLOBAL_STEP: 337250 | > loss_text_ce: 0.05109446868300438 (0.04040807824773242) | > loss_mel_ce: 0.11376953125 (nan) | > loss: 0.000654222269076854 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13665606862495044) | > loader_time: 0.0051 (0.00472109949529269)  --> TIME: 2025-02-24 13:24:46 -- STEP: 4858/36938 -- GLOBAL_STEP: 337300 | > loss_text_ce: 0.035802122205495834 (0.04040623434749118) | > loss_mel_ce: 3.629676580429077 (nan) | > loss: 0.014545551501214504 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13669967950740058) | > loader_time: 0.0047 (0.004721602229435967)  --> TIME: 2025-02-24 13:24:55 -- STEP: 4908/36938 -- GLOBAL_STEP: 337350 | > loss_text_ce: 0.044228147715330124 (0.04040768608878225) | > loss_mel_ce: 0.11350097507238388 (nan) | > loss: 0.0006259092479012907 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13666555438473066) | > loader_time: 0.0046 (0.00472059071112572)  --> TIME: 2025-02-24 13:25:05 -- STEP: 4958/36938 -- GLOBAL_STEP: 337400 | > loss_text_ce: 0.03711098060011864 (0.040421278263844646) | > loss_mel_ce: 3.0531837940216064 (nan) | > loss: 0.012263075448572636 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.13664914398916597) | > loader_time: 0.0047 (0.004720288690610861)  --> TIME: 2025-02-24 13:25:14 -- STEP: 5008/36938 -- GLOBAL_STEP: 337450 | > loss_text_ce: 0.04180683195590973 (0.04041861033326725) | > loss_mel_ce: 3.4496383666992188 (nan) | > loss: 0.01385494228452444 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.1365992523515567) | > loader_time: 0.0055 (0.004721341946254513)  --> TIME: 2025-02-24 13:25:23 -- STEP: 5058/36938 -- GLOBAL_STEP: 337500 | > loss_text_ce: 0.039315368980169296 (0.040417015645874035) | > loss_mel_ce: 1.70928955078125 (nan) | > loss: 0.006938908714801073 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.13659133313436564) | > loader_time: 0.0045 (0.004721112381379647)  --> TIME: 2025-02-24 13:25:32 -- STEP: 5108/36938 -- GLOBAL_STEP: 337550 | > loss_text_ce: 0.0447128489613533 (0.04041738618955098) | > loss_mel_ce: 2.6534879207611084 (nan) | > loss: 0.010707146488130093 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.1365692519357452) | > loader_time: 0.0046 (0.004723630291274856)  --> TIME: 2025-02-24 13:25:41 -- STEP: 5158/36938 -- GLOBAL_STEP: 337600 | > loss_text_ce: 0.04342595487833023 (0.04042627261055999) | > loss_mel_ce: 2.935987710952759 (nan) | > loss: 0.011823071166872978 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13653315957917303) | > loader_time: 0.0051 (0.004723158304244782)  --> TIME: 2025-02-24 13:25:50 -- STEP: 5208/36938 -- GLOBAL_STEP: 337650 | > loss_text_ce: 0.037177570164203644 (0.04042521754108781) | > loss_mel_ce: 3.5725479125976562 (nan) | > loss: 0.014324308373034 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13654291876999455) | > loader_time: 0.0046 (0.004722614396369202)  --> TIME: 2025-02-24 13:25:59 -- STEP: 5258/36938 -- GLOBAL_STEP: 337700 | > loss_text_ce: 0.042447347193956375 (0.04043295502661956) | > loss_mel_ce: 3.624824047088623 (nan) | > loss: 0.014552664943039417 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.13650402545747575) | > loader_time: 0.0054 (0.00472224987519017)  --> TIME: 2025-02-24 13:26:09 -- STEP: 5308/36938 -- GLOBAL_STEP: 337750 | > loss_text_ce: 0.03407452628016472 (0.04044297487880639) | > loss_mel_ce: 2.8222920894622803 (nan) | > loss: 0.011334788985550404 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13649895551425664) | > loader_time: 0.0045 (0.004720941870933002)  --> TIME: 2025-02-24 13:26:18 -- STEP: 5358/36938 -- GLOBAL_STEP: 337800 | > loss_text_ce: 0.05124912038445473 (0.04044339981346566) | > loss_mel_ce: 3.1799776554107666 (nan) | > loss: 0.012822329066693783 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.1364731581276056) | > loader_time: 0.0045 (0.004720328903411811)  --> TIME: 2025-02-24 13:26:27 -- STEP: 5408/36938 -- GLOBAL_STEP: 337850 | > loss_text_ce: 0.046413931995630264 (0.04044212988311203) | > loss_mel_ce: 1.240486979484558 (nan) | > loss: 0.005106749944388866 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.1364752071848988) | > loader_time: 0.0046 (0.004719331640110899)  --> TIME: 2025-02-24 13:26:36 -- STEP: 5458/36938 -- GLOBAL_STEP: 337900 | > loss_text_ce: 0.04867013916373253 (0.04041971307648532) | > loss_mel_ce: 3.1647136211395264 (nan) | > loss: 0.012751523405313492 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13644751916223508) | > loader_time: 0.0044 (0.004717834837142237)  --> TIME: 2025-02-24 13:26:45 -- STEP: 5508/36938 -- GLOBAL_STEP: 337950 | > loss_text_ce: 0.033431075513362885 (0.04040991008155235) | > loss_mel_ce: 2.887967824935913 (nan) | > loss: 0.01159285381436348 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13642793564841474) | > loader_time: 0.0044 (0.004717238977843617)  --> TIME: 2025-02-24 13:26:55 -- STEP: 5558/36938 -- GLOBAL_STEP: 338000 | > loss_text_ce: 0.03824838250875473 (0.04041523903560738) | > loss_mel_ce: 3.286302328109741 (nan) | > loss: 0.013192662037909031 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.1364021629427525) | > loader_time: 0.0064 (0.004718467502038033)  --> TIME: 2025-02-24 13:27:04 -- STEP: 5608/36938 -- GLOBAL_STEP: 338050 | > loss_text_ce: 0.040894102305173874 (0.04041307867314009) | > loss_mel_ce: 1.1418113708496094 (nan) | > loss: 0.00469327624887228 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1364040305458019) | > loader_time: 0.0047 (0.004718161938704008)  --> TIME: 2025-02-24 13:27:13 -- STEP: 5658/36938 -- GLOBAL_STEP: 338100 | > loss_text_ce: 0.038351185619831085 (0.04042527312215337) | > loss_mel_ce: 0.02272796630859375 (nan) | > loss: 0.00024237760226242244 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13643858284442892) | > loader_time: 0.0052 (0.00471848508586052)  --> TIME: 2025-02-24 13:27:23 -- STEP: 5708/36938 -- GLOBAL_STEP: 338150 | > loss_text_ce: 0.04030010104179382 (0.04042522239233818) | > loss_mel_ce: 3.2415380477905273 (nan) | > loss: 0.013023167848587036 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13642819485801363) | > loader_time: 0.0045 (0.004718552165375589)  --> TIME: 2025-02-24 13:27:32 -- STEP: 5758/36938 -- GLOBAL_STEP: 338200 | > loss_text_ce: 0.03758002817630768 (0.04042155625875718) | > loss_mel_ce: 2.3134031295776367 (nan) | > loss: 0.009329298511147499 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13643680313636367) | > loader_time: 0.0046 (0.004718764907330739)  --> TIME: 2025-02-24 13:27:42 -- STEP: 5808/36938 -- GLOBAL_STEP: 338250 | > loss_text_ce: 0.04094711318612099 (0.04042002408442624) | > loss_mel_ce: 2.233787775039673 (nan) | > loss: 0.009026726707816124 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.1364612194706913) | > loader_time: 0.0046 (0.004719608004099716)  --> TIME: 2025-02-24 13:27:51 -- STEP: 5858/36938 -- GLOBAL_STEP: 338300 | > loss_text_ce: 0.03933519124984741 (0.040417730108573705) | > loss_mel_ce: 3.267056941986084 (nan) | > loss: 0.013120604678988457 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13646155658863854) | > loader_time: 0.0045 (0.004719586818443685)  --> TIME: 2025-02-24 13:28:00 -- STEP: 5908/36938 -- GLOBAL_STEP: 338350 | > loss_text_ce: 0.022605080157518387 (0.04040658443307997) | > loss_mel_ce: 2.5728304386138916 (nan) | > loss: 0.010299348272383213 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13645394109113812) | > loader_time: 0.0046 (0.004720870271758907)  --> TIME: 2025-02-24 13:28:09 -- STEP: 5958/36938 -- GLOBAL_STEP: 338400 | > loss_text_ce: 0.045129019767045975 (0.04041040274066036) | > loss_mel_ce: 3.6932809352874756 (nan) | > loss: 0.014834960922598839 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13646005071542014) | > loader_time: 0.0044 (0.004719982414527302)  --> TIME: 2025-02-24 13:28:19 -- STEP: 6008/36938 -- GLOBAL_STEP: 338450 | > loss_text_ce: 0.036996882408857346 (0.04040598807426735) | > loss_mel_ce: 2.9939820766448975 (nan) | > loss: 0.012027694843709469 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.136447804229713) | > loader_time: 0.0046 (0.004719290411107208)  --> TIME: 2025-02-24 13:28:28 -- STEP: 6058/36938 -- GLOBAL_STEP: 338500 | > loss_text_ce: 0.0357227586209774 (0.04041010552227668) | > loss_mel_ce: 3.480225086212158 (nan) | > loss: 0.013952174223959446 (nan) | > current_lr: 5e-06 | > step_time: 0.1757 (0.13645368064736754) | > loader_time: 0.008 (0.004719586606937157)  --> TIME: 2025-02-24 13:28:38 -- STEP: 6108/36938 -- GLOBAL_STEP: 338550 | > loss_text_ce: 0.028684191405773163 (0.04040117273073614) | > loss_mel_ce: 2.551165819168091 (nan) | > loss: 0.010237500071525574 (nan) | > current_lr: 5e-06 | > step_time: 0.108 (0.1364568784179134) | > loader_time: 0.0044 (0.004719215393378652)  --> TIME: 2025-02-24 13:28:47 -- STEP: 6158/36938 -- GLOBAL_STEP: 338600 | > loss_text_ce: 0.037117186933755875 (0.04039800450669791) | > loss_mel_ce: 3.686328887939453 (nan) | > loss: 0.01477558072656393 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13643698515772754) | > loader_time: 0.0052 (0.0047191245945369335)  --> TIME: 2025-02-24 13:28:56 -- STEP: 6208/36938 -- GLOBAL_STEP: 338650 | > loss_text_ce: 0.03786266967654228 (0.04040371407571685) | > loss_mel_ce: 2.6731512546539307 (nan) | > loss: 0.0107579929754138 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13645246686394658) | > loader_time: 0.0046 (0.00471928212599654)  --> TIME: 2025-02-24 13:29:05 -- STEP: 6258/36938 -- GLOBAL_STEP: 338700 | > loss_text_ce: 0.047533366829156876 (0.04042227906253822) | > loss_mel_ce: 3.414966583251953 (nan) | > loss: 0.01374007947742939 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.1364340047632094) | > loader_time: 0.0054 (0.0047192934718365016)  --> TIME: 2025-02-24 13:29:15 -- STEP: 6308/36938 -- GLOBAL_STEP: 338750 | > loss_text_ce: 0.030360737815499306 (0.04041162230350191) | > loss_mel_ce: 3.4650840759277344 (nan) | > loss: 0.013870812952518463 (nan) | > current_lr: 5e-06 | > step_time: 0.1861 (0.1364840457313936) | > loader_time: 0.0055 (0.00471986961576403)  --> TIME: 2025-02-24 13:29:24 -- STEP: 6358/36938 -- GLOBAL_STEP: 338800 | > loss_text_ce: 0.03791886195540428 (0.04042445574780159) | > loss_mel_ce: 3.2470011711120605 (nan) | > loss: 0.013035397045314312 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13648668382482032) | > loader_time: 0.0045 (0.004719606282839254)  --> TIME: 2025-02-24 13:29:33 -- STEP: 6408/36938 -- GLOBAL_STEP: 338850 | > loss_text_ce: 0.04004570096731186 (0.04043099948059128) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1364498328924473) | > loader_time: 0.0052 (0.004721421873971296)  --> TIME: 2025-02-24 13:29:43 -- STEP: 6458/36938 -- GLOBAL_STEP: 338900 | > loss_text_ce: 0.033976439386606216 (0.040426092874604795) | > loss_mel_ce: 3.4267578125 (nan) | > loss: 0.013733074069023132 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13645061659495386) | > loader_time: 0.0051 (0.004721992707688119)  --> TIME: 2025-02-24 13:29:52 -- STEP: 6508/36938 -- GLOBAL_STEP: 338950 | > loss_text_ce: 0.03869631513953209 (0.04042096704959556) | > loss_mel_ce: 1.58837890625 (nan) | > loss: 0.006456647999584675 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13646123818847056) | > loader_time: 0.0045 (0.0047214512599342276)  --> TIME: 2025-02-24 13:30:02 -- STEP: 6558/36938 -- GLOBAL_STEP: 339000 | > loss_text_ce: 0.0411747582256794 (0.04041905632866275) | > loss_mel_ce: 3.6829323768615723 (nan) | > loss: 0.014778203330934048 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.1364951042476021) | > loader_time: 0.0053 (0.004721320886079673)  --> TIME: 2025-02-24 13:30:11 -- STEP: 6608/36938 -- GLOBAL_STEP: 339050 | > loss_text_ce: 0.04923645779490471 (0.04042531574524359) | > loss_mel_ce: 3.7979230880737305 (nan) | > loss: 0.015266506932675838 (nan) | > current_lr: 5e-06 | > step_time: 0.1816 (0.1365079685287958) | > loader_time: 0.0048 (0.0047203974198486605)  --> TIME: 2025-02-24 13:30:20 -- STEP: 6658/36938 -- GLOBAL_STEP: 339100 | > loss_text_ce: 0.033081624656915665 (0.040415803936961044) | > loss_mel_ce: 2.3103280067443848 (nan) | > loss: 0.009299244731664658 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.1364822239802739) | > loader_time: 0.0044 (0.004719518476436337)  --> TIME: 2025-02-24 13:30:29 -- STEP: 6708/36938 -- GLOBAL_STEP: 339150 | > loss_text_ce: 0.033511094748973846 (0.0404232597395335) | > loss_mel_ce: 2.8081860542297363 (nan) | > loss: 0.011276576668024063 (nan) | > current_lr: 5e-06 | > step_time: 0.1799 (0.1365054067430852) | > loader_time: 0.0048 (0.00471897351293954)  --> TIME: 2025-02-24 13:30:39 -- STEP: 6758/36938 -- GLOBAL_STEP: 339200 | > loss_text_ce: 0.04323273152112961 (0.04042844132917973) | > loss_mel_ce: 3.079267740249634 (nan) | > loss: 0.01239087525755167 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.13650243237865972) | > loader_time: 0.0046 (0.004718480254108625)  --> TIME: 2025-02-24 13:30:48 -- STEP: 6808/36938 -- GLOBAL_STEP: 339250 | > loss_text_ce: 0.03152359649538994 (0.04042708194572756) | > loss_mel_ce: 3.0330753326416016 (nan) | > loss: 0.01216110773384571 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13650549844905407) | > loader_time: 0.0051 (0.00471851520336611)  --> TIME: 2025-02-24 13:30:57 -- STEP: 6858/36938 -- GLOBAL_STEP: 339300 | > loss_text_ce: 0.044257406145334244 (0.04042997863306637) | > loss_mel_ce: 3.2373504638671875 (nan) | > loss: 0.013022254221141338 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13649447755515984) | > loader_time: 0.0046 (0.004718788165884814)  --> TIME: 2025-02-24 13:31:07 -- STEP: 6908/36938 -- GLOBAL_STEP: 339350 | > loss_text_ce: 0.04565753787755966 (0.0404175357193673) | > loss_mel_ce: 1.8054938316345215 (nan) | > loss: 0.007345838937908411 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13651134361058329) | > loader_time: 0.0052 (0.004719704303288621)  --> TIME: 2025-02-24 13:31:16 -- STEP: 6958/36938 -- GLOBAL_STEP: 339400 | > loss_text_ce: 0.0342937670648098 (0.04040863707454959) | > loss_mel_ce: 3.132638454437256 (nan) | > loss: 0.012567191384732723 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.13647367345015507) | > loader_time: 0.0055 (0.004720876942629355)  --> TIME: 2025-02-24 13:31:25 -- STEP: 7008/36938 -- GLOBAL_STEP: 339450 | > loss_text_ce: 0.048986561596393585 (0.04041405513560495) | > loss_mel_ce: 3.8103206157684326 (nan) | > loss: 0.015314712189137936 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1364547061525521) | > loader_time: 0.0045 (0.004720515924501618)  --> TIME: 2025-02-24 13:31:34 -- STEP: 7058/36938 -- GLOBAL_STEP: 339500 | > loss_text_ce: 0.034498896449804306 (0.040418720703078996) | > loss_mel_ce: 3.419844627380371 (nan) | > loss: 0.013707713223993778 (nan) | > current_lr: 5e-06 | > step_time: 0.1563 (0.13647920192216909) | > loader_time: 0.0053 (0.0047213111392632005)  --> TIME: 2025-02-24 13:31:44 -- STEP: 7108/36938 -- GLOBAL_STEP: 339550 | > loss_text_ce: 0.05027829110622406 (0.0404229378820379) | > loss_mel_ce: 2.31866717338562 (nan) | > loss: 0.009400577284395695 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13646729262812957) | > loader_time: 0.0045 (0.0047222119606674775)  --> TIME: 2025-02-24 13:31:53 -- STEP: 7158/36938 -- GLOBAL_STEP: 339600 | > loss_text_ce: 0.036925058811903 (0.04043126831361489) | > loss_mel_ce: 3.457996368408203 (nan) | > loss: 0.013868737034499645 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.1364740695191279) | > loader_time: 0.0053 (0.004723801896638045)  --> TIME: 2025-02-24 13:32:02 -- STEP: 7208/36938 -- GLOBAL_STEP: 339650 | > loss_text_ce: 0.05550558492541313 (0.04042827224640064) | > loss_mel_ce: 3.126012086868286 (nan) | > loss: 0.012625070288777351 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13645054034466983) | > loader_time: 0.0045 (0.004725036524509069)  --> TIME: 2025-02-24 13:32:11 -- STEP: 7258/36938 -- GLOBAL_STEP: 339700 | > loss_text_ce: 0.03602535277605057 (0.04042236662792386) | > loss_mel_ce: 2.5620815753936768 (nan) | > loss: 0.010309948585927486 (nan) | > current_lr: 5e-06 | > step_time: 0.1451 (0.13644797334200587) | > loader_time: 0.0047 (0.004724574962835644)  --> TIME: 2025-02-24 13:32:21 -- STEP: 7308/36938 -- GLOBAL_STEP: 339750 | > loss_text_ce: 0.02911190129816532 (0.04042450938226989) | > loss_mel_ce: 3.1410069465637207 (nan) | > loss: 0.012579836882650852 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13648253229059773) | > loader_time: 0.0044 (0.004724189728808372)  --> TIME: 2025-02-24 13:32:31 -- STEP: 7358/36938 -- GLOBAL_STEP: 339800 | > loss_text_ce: 0.0366632379591465 (0.040418381937411096) | > loss_mel_ce: 3.1338086128234863 (nan) | > loss: 0.012581238523125648 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13652778582327943) | > loader_time: 0.0047 (0.004724152615032091)  --> TIME: 2025-02-24 13:32:40 -- STEP: 7408/36938 -- GLOBAL_STEP: 339850 | > loss_text_ce: 0.030158525332808495 (0.04042278928757633) | > loss_mel_ce: 2.994159460067749 (nan) | > loss: 0.012001262977719307 (nan) | > current_lr: 5e-06 | > step_time: 0.1785 (0.13654388773775994) | > loader_time: 0.0047 (0.004723961583738948)  --> TIME: 2025-02-24 13:32:49 -- STEP: 7458/36938 -- GLOBAL_STEP: 339900 | > loss_text_ce: 0.05483762547373772 (0.040431887897160505) | > loss_mel_ce: 3.292917013168335 (nan) | > loss: 0.013284741900861263 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13652813255579052) | > loader_time: 0.0046 (0.004723102645536423)  --> TIME: 2025-02-24 13:32:59 -- STEP: 7508/36938 -- GLOBAL_STEP: 339950 | > loss_text_ce: 0.04119843244552612 (0.04043966110921759) | > loss_mel_ce: 3.4574716091156006 (nan) | > loss: 0.013883612118661404 (nan) | > current_lr: 5e-06 | > step_time: 0.1872 (0.1365551200075608) | > loader_time: 0.0055 (0.004722957637758395)  --> TIME: 2025-02-24 13:33:08 -- STEP: 7558/36938 -- GLOBAL_STEP: 340000 | > loss_text_ce: 0.03913533687591553 (0.04044511954177615) | > loss_mel_ce: 3.253263235092163 (nan) | > loss: 0.013065073639154434 (nan) | > current_lr: 5e-06 | > step_time: 0.1994 (0.1365653773911193) | > loader_time: 0.0048 (0.004722634046978012) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_340000.pth  --> TIME: 2025-02-24 13:33:23 -- STEP: 7608/36938 -- GLOBAL_STEP: 340050 | > loss_text_ce: 0.04411112517118454 (0.04044134819147599) | > loss_mel_ce: 2.6506423950195312 (nan) | > loss: 0.010693466290831566 (nan) | > current_lr: 5e-06 | > step_time: 0.1073 (0.13653264145369748) | > loader_time: 0.0045 (0.004723016176564694)  --> TIME: 2025-02-24 13:33:33 -- STEP: 7658/36938 -- GLOBAL_STEP: 340100 | > loss_text_ce: 0.030576441437005997 (0.0404396789967157) | > loss_mel_ce: 3.2432749271392822 (nan) | > loss: 0.012991474941372871 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13655416549544097) | > loader_time: 0.0053 (0.00472326741326909)  --> TIME: 2025-02-24 13:33:42 -- STEP: 7708/36938 -- GLOBAL_STEP: 340150 | > loss_text_ce: 0.04099316895008087 (0.04044720542230012) | > loss_mel_ce: 3.1324305534362793 (nan) | > loss: 0.012592951767146587 (nan) | > current_lr: 5e-06 | > step_time: 0.1576 (0.13656265323002434) | > loader_time: 0.0049 (0.0047239171264699745)  --> TIME: 2025-02-24 13:33:52 -- STEP: 7758/36938 -- GLOBAL_STEP: 340200 | > loss_text_ce: 0.035476379096508026 (0.04044820083226963) | > loss_mel_ce: 2.9683356285095215 (nan) | > loss: 0.011919889599084854 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13657044233443558) | > loader_time: 0.0046 (0.004723429464746839)  --> TIME: 2025-02-24 13:34:01 -- STEP: 7808/36938 -- GLOBAL_STEP: 340250 | > loss_text_ce: 0.05345646291971207 (0.04045323487161271) | > loss_mel_ce: 1.9671059846878052 (nan) | > loss: 0.008018105290830135 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1365388504489031) | > loader_time: 0.0046 (0.004722856412656956)  --> TIME: 2025-02-24 13:34:10 -- STEP: 7858/36938 -- GLOBAL_STEP: 340300 | > loss_text_ce: 0.039706215262413025 (0.04045130480621284) | > loss_mel_ce: 4.026072025299072 (nan) | > loss: 0.016134042292833328 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1365255863498442) | > loader_time: 0.0045 (0.0047224620184117844)  --> TIME: 2025-02-24 13:34:19 -- STEP: 7908/36938 -- GLOBAL_STEP: 340350 | > loss_text_ce: 0.03327954187989235 (0.04044860920722303) | > loss_mel_ce: 3.1271824836730957 (nan) | > loss: 0.012541517615318298 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.13652276667188223) | > loader_time: 0.0046 (0.004722158023668272)  --> TIME: 2025-02-24 13:34:28 -- STEP: 7958/36938 -- GLOBAL_STEP: 340400 | > loss_text_ce: 0.033671148121356964 (0.04044193365676901) | > loss_mel_ce: 2.608444929122925 (nan) | > loss: 0.010484588332474232 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13652817293398606) | > loader_time: 0.0046 (0.004721744481748463)  --> TIME: 2025-02-24 13:34:38 -- STEP: 8008/36938 -- GLOBAL_STEP: 340450 | > loss_text_ce: 0.04268547520041466 (0.04043691243569768) | > loss_mel_ce: 3.9467179775238037 (nan) | > loss: 0.01583096757531166 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.13651023517836305) | > loader_time: 0.0049 (0.004720971419975577)  --> TIME: 2025-02-24 13:34:47 -- STEP: 8058/36938 -- GLOBAL_STEP: 340500 | > loss_text_ce: 0.034717101603746414 (0.040438342122899595) | > loss_mel_ce: 3.5475547313690186 (nan) | > loss: 0.014215365052223206 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.1364929572498567) | > loader_time: 0.0046 (0.004720556525918959)  --> TIME: 2025-02-24 13:34:56 -- STEP: 8108/36938 -- GLOBAL_STEP: 340550 | > loss_text_ce: 0.03479628264904022 (0.040441383952968646) | > loss_mel_ce: 3.108049154281616 (nan) | > loss: 0.012471609748899937 (nan) | > current_lr: 5e-06 | > step_time: 0.1713 (0.13652501849629423) | > loader_time: 0.0048 (0.004720448330228514)  --> TIME: 2025-02-24 13:35:06 -- STEP: 8158/36938 -- GLOBAL_STEP: 340600 | > loss_text_ce: 0.03816229850053787 (0.04043926940360983) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13650683633075295) | > loader_time: 0.0046 (0.004719983745246679)  --> TIME: 2025-02-24 13:35:15 -- STEP: 8208/36938 -- GLOBAL_STEP: 340650 | > loss_text_ce: 0.036507584154605865 (0.04043003553311293) | > loss_mel_ce: 3.356236457824707 (nan) | > loss: 0.013463270850479603 (nan) | > current_lr: 5e-06 | > step_time: 0.1572 (0.13652494988362424) | > loader_time: 0.0046 (0.004719491096732661)  --> TIME: 2025-02-24 13:35:24 -- STEP: 8258/36938 -- GLOBAL_STEP: 340700 | > loss_text_ce: 0.04330180957913399 (0.040437036749678294) | > loss_mel_ce: 3.0741569995880127 (nan) | > loss: 0.01237086858600378 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13651971148358985) | > loader_time: 0.0046 (0.004718943640001447)  --> TIME: 2025-02-24 13:35:33 -- STEP: 8308/36938 -- GLOBAL_STEP: 340750 | > loss_text_ce: 0.04038511961698532 (0.040430515470520345) | > loss_mel_ce: 3.4215452671051025 (nan) | > loss: 0.013737819157540798 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13649961764156987) | > loader_time: 0.0046 (0.0047185603218923315)  --> TIME: 2025-02-24 13:35:43 -- STEP: 8358/36938 -- GLOBAL_STEP: 340800 | > loss_text_ce: 0.031196190044283867 (0.04042493153455278) | > loss_mel_ce: 3.3465731143951416 (nan) | > loss: 0.013403846882283688 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13649953769479334) | > loader_time: 0.0047 (0.004718042070370979)  --> TIME: 2025-02-24 13:35:53 -- STEP: 8408/36938 -- GLOBAL_STEP: 340850 | > loss_text_ce: 0.04097336158156395 (0.04042252731579506) | > loss_mel_ce: 3.2731776237487793 (nan) | > loss: 0.013151394203305244 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13654463122165472) | > loader_time: 0.0046 (0.0047182310602984666)  --> TIME: 2025-02-24 13:36:02 -- STEP: 8458/36938 -- GLOBAL_STEP: 340900 | > loss_text_ce: 0.052403438836336136 (0.04042528468373781) | > loss_mel_ce: 3.5869927406311035 (nan) | > loss: 0.014442048966884613 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13652759247791313) | > loader_time: 0.0044 (0.004717629974574493)  --> TIME: 2025-02-24 13:36:11 -- STEP: 8508/36938 -- GLOBAL_STEP: 340950 | > loss_text_ce: 0.037526533007621765 (0.04042915143199579) | > loss_mel_ce: 3.0429792404174805 (nan) | > loss: 0.012224229983985424 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13653177844004014) | > loader_time: 0.0045 (0.004717160487432786)  --> TIME: 2025-02-24 13:36:20 -- STEP: 8558/36938 -- GLOBAL_STEP: 341000 | > loss_text_ce: 0.02912708930671215 (0.04042548574916413) | > loss_mel_ce: 3.190479278564453 (nan) | > loss: 0.012776216492056847 (nan) | > current_lr: 5e-06 | > step_time: 0.1487 (0.13650676228504718) | > loader_time: 0.0046 (0.004716466592555559)  --> TIME: 2025-02-24 13:36:29 -- STEP: 8608/36938 -- GLOBAL_STEP: 341050 | > loss_text_ce: 0.03266036882996559 (0.04042759475356214) | > loss_mel_ce: 2.642937183380127 (nan) | > loss: 0.01061745174229145 (nan) | > current_lr: 5e-06 | > step_time: 0.108 (0.13646197410431896) | > loader_time: 0.0045 (0.004715522508860511)  --> TIME: 2025-02-24 13:36:38 -- STEP: 8658/36938 -- GLOBAL_STEP: 341100 | > loss_text_ce: 0.041391126811504364 (0.04042192131515823) | > loss_mel_ce: 2.5651254653930664 (nan) | > loss: 0.010343320667743683 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13644738427539063) | > loader_time: 0.0046 (0.0047149625709024265)  --> TIME: 2025-02-24 13:36:47 -- STEP: 8708/36938 -- GLOBAL_STEP: 341150 | > loss_text_ce: 0.048771362751722336 (0.04042711765329971) | > loss_mel_ce: 3.5101819038391113 (nan) | > loss: 0.014122831635177135 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.13641842675417085) | > loader_time: 0.0045 (0.0047146588440712305)  --> TIME: 2025-02-24 13:36:56 -- STEP: 8758/36938 -- GLOBAL_STEP: 341200 | > loss_text_ce: 0.03894427791237831 (0.040436213010346535) | > loss_mel_ce: 3.335085153579712 (nan) | > loss: 0.01338900625705719 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.1363795688365192) | > loader_time: 0.0046 (0.004714505044693642)  --> TIME: 2025-02-24 13:37:05 -- STEP: 8808/36938 -- GLOBAL_STEP: 341250 | > loss_text_ce: 0.04329138249158859 (0.040431579060536973) | > loss_mel_ce: 3.5852444171905518 (nan) | > loss: 0.014398952014744282 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13639491609721435) | > loader_time: 0.0047 (0.004714195886381522)  --> TIME: 2025-02-24 13:37:15 -- STEP: 8858/36938 -- GLOBAL_STEP: 341300 | > loss_text_ce: 0.03168687969446182 (0.04043362744530621) | > loss_mel_ce: 3.5590081214904785 (nan) | > loss: 0.014248790219426155 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.13639025460465184) | > loader_time: 0.0047 (0.0047141420407875445)  --> TIME: 2025-02-24 13:37:24 -- STEP: 8908/36938 -- GLOBAL_STEP: 341350 | > loss_text_ce: 0.03472689539194107 (0.04043361243706558) | > loss_mel_ce: 4.495233058929443 (nan) | > loss: 0.017976032570004463 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13638919260023272) | > loader_time: 0.0046 (0.004713568951872673)  --> TIME: 2025-02-24 13:37:34 -- STEP: 8958/36938 -- GLOBAL_STEP: 341400 | > loss_text_ce: 0.03778992220759392 (0.04043021390662023) | > loss_mel_ce: 3.493926525115967 (nan) | > loss: 0.01401474792510271 (nan) | > current_lr: 5e-06 | > step_time: 0.1859 (0.13639710105990485) | > loader_time: 0.0049 (0.004713110584157673)  --> TIME: 2025-02-24 13:37:43 -- STEP: 9008/36938 -- GLOBAL_STEP: 341450 | > loss_text_ce: 0.04145798459649086 (0.04043240304430443) | > loss_mel_ce: 3.252368450164795 (nan) | > loss: 0.013070739805698395 (nan) | > current_lr: 5e-06 | > step_time: 0.1908 (0.1364396755479364) | > loader_time: 0.0054 (0.004713569266546253)  --> TIME: 2025-02-24 13:37:52 -- STEP: 9058/36938 -- GLOBAL_STEP: 341500 | > loss_text_ce: 0.020962359383702278 (0.04042932765412764) | > loss_mel_ce: 2.4812324047088623 (nan) | > loss: 0.009929344989359379 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1364334996737869) | > loader_time: 0.0044 (0.004713625011919975)  --> TIME: 2025-02-24 13:38:02 -- STEP: 9108/36938 -- GLOBAL_STEP: 341550 | > loss_text_ce: 0.051132410764694214 (0.04042050766073105) | > loss_mel_ce: 3.5901758670806885 (nan) | > loss: 0.01444963738322258 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13643777438512608) | > loader_time: 0.0047 (0.004713563972468114)  --> TIME: 2025-02-24 13:38:11 -- STEP: 9158/36938 -- GLOBAL_STEP: 341600 | > loss_text_ce: 0.028727775439620018 (0.040424270417376854) | > loss_mel_ce: 3.0583839416503906 (nan) | > loss: 0.012250443920493126 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13642251379600773) | > loader_time: 0.0045 (0.004714131277183899)  --> TIME: 2025-02-24 13:38:20 -- STEP: 9208/36938 -- GLOBAL_STEP: 341650 | > loss_text_ce: 0.03814838454127312 (0.040418849464527666) | > loss_mel_ce: 3.665275812149048 (nan) | > loss: 0.01469612866640091 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.1364092107289358) | > loader_time: 0.0047 (0.004714306725511124)  --> TIME: 2025-02-24 13:38:29 -- STEP: 9258/36938 -- GLOBAL_STEP: 341700 | > loss_text_ce: 0.038932330906391144 (0.04041151717604144) | > loss_mel_ce: 3.056990146636963 (nan) | > loss: 0.01228540763258934 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13639958073753403) | > loader_time: 0.0044 (0.0047142737162605)  --> TIME: 2025-02-24 13:38:38 -- STEP: 9308/36938 -- GLOBAL_STEP: 341750 | > loss_text_ce: 0.039578262716531754 (0.04040930959012638) | > loss_mel_ce: 3.5631470680236816 (nan) | > loss: 0.014296529814600945 (nan) | > current_lr: 5e-06 | > step_time: 0.1651 (0.13638632869740905) | > loader_time: 0.0049 (0.0047144734864589385)  --> TIME: 2025-02-24 13:38:48 -- STEP: 9358/36938 -- GLOBAL_STEP: 341800 | > loss_text_ce: 0.03474077209830284 (0.04040601085160863) | > loss_mel_ce: 3.4411985874176025 (nan) | > loss: 0.01379341073334217 (nan) | > current_lr: 5e-06 | > step_time: 0.1899 (0.13639011378470653) | > loader_time: 0.0049 (0.004715740999163505)  --> TIME: 2025-02-24 13:38:57 -- STEP: 9408/36938 -- GLOBAL_STEP: 341850 | > loss_text_ce: 0.04132455214858055 (0.04040840211521178) | > loss_mel_ce: 3.1955204010009766 (nan) | > loss: 0.012844623997807503 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.136407088691077) | > loader_time: 0.0048 (0.004715688095814512)  --> TIME: 2025-02-24 13:39:06 -- STEP: 9458/36938 -- GLOBAL_STEP: 341900 | > loss_text_ce: 0.038975995033979416 (0.04040836829380555) | > loss_mel_ce: 2.747020721435547 (nan) | > loss: 0.011055543087422848 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.1363815671673128) | > loader_time: 0.0046 (0.004715422566583879)  --> TIME: 2025-02-24 13:39:15 -- STEP: 9508/36938 -- GLOBAL_STEP: 341950 | > loss_text_ce: 0.04041349142789841 (0.04041530655316531) | > loss_mel_ce: 3.328718662261963 (nan) | > loss: 0.013369572348892689 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13637042053115891) | > loader_time: 0.0047 (0.004715403389118272)  --> TIME: 2025-02-24 13:39:25 -- STEP: 9558/36938 -- GLOBAL_STEP: 342000 | > loss_text_ce: 0.03544212132692337 (0.04040918298068378) | > loss_mel_ce: 0.015297376550734043 (nan) | > loss: 0.00020134722581133246 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13636857529368976) | > loader_time: 0.0045 (0.004715156395531614)  --> TIME: 2025-02-24 13:39:34 -- STEP: 9608/36938 -- GLOBAL_STEP: 342050 | > loss_text_ce: 0.03811100497841835 (0.0404083318774743) | > loss_mel_ce: 2.860827684402466 (nan) | > loss: 0.011503725312650204 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.1363786701903948) | > loader_time: 0.0046 (0.00471556037788486)  --> TIME: 2025-02-24 13:39:43 -- STEP: 9658/36938 -- GLOBAL_STEP: 342100 | > loss_text_ce: 0.059508346021175385 (0.04040552588117207) | > loss_mel_ce: 3.668473243713379 (nan) | > loss: 0.014793578535318375 (nan) | > current_lr: 5e-06 | > step_time: 0.1822 (0.13636277086815224) | > loader_time: 0.0049 (0.004715471169943144)  --> TIME: 2025-02-24 13:39:53 -- STEP: 9708/36938 -- GLOBAL_STEP: 342150 | > loss_text_ce: 0.04629424586892128 (0.04040119210981451) | > loss_mel_ce: 3.004154920578003 (nan) | > loss: 0.012104957364499569 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.13636725635060937) | > loader_time: 0.0048 (0.004715743234828508)  --> TIME: 2025-02-24 13:40:02 -- STEP: 9758/36938 -- GLOBAL_STEP: 342200 | > loss_text_ce: 0.05859597399830818 (0.040397510588967574) | > loss_mel_ce: 0.08123779296875 (nan) | > loss: 0.0005548959015868604 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1363814971710378) | > loader_time: 0.0045 (0.004715956095472467)  --> TIME: 2025-02-24 13:40:11 -- STEP: 9808/36938 -- GLOBAL_STEP: 342250 | > loss_text_ce: 0.04307955130934715 (0.040402746972244906) | > loss_mel_ce: 0.1265428364276886 (nan) | > loss: 0.0006731047760695219 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13635962054157702) | > loader_time: 0.0047 (0.0047157473655548175)  --> TIME: 2025-02-24 13:40:21 -- STEP: 9858/36938 -- GLOBAL_STEP: 342300 | > loss_text_ce: 0.04237731918692589 (0.04040121428478023) | > loss_mel_ce: 4.901785850524902 (nan) | > loss: 0.019619697704911232 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.1363788736739633) | > loader_time: 0.0047 (0.00471601224521227)  --> TIME: 2025-02-24 13:40:30 -- STEP: 9908/36938 -- GLOBAL_STEP: 342350 | > loss_text_ce: 0.03851708024740219 (0.04040472960358242) | > loss_mel_ce: 2.8015310764312744 (nan) | > loss: 0.01127003226429224 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13636050052973908) | > loader_time: 0.0046 (0.004715562733036881)  --> TIME: 2025-02-24 13:40:39 -- STEP: 9958/36938 -- GLOBAL_STEP: 342400 | > loss_text_ce: 0.04262324422597885 (0.04040356767952104) | > loss_mel_ce: 3.066257953643799 (nan) | > loss: 0.012336830608546734 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13633979499328058) | > loader_time: 0.0047 (0.004715087830864724)  --> TIME: 2025-02-24 13:40:48 -- STEP: 10008/36938 -- GLOBAL_STEP: 342450 | > loss_text_ce: 0.052722904831171036 (0.04039633252153932) | > loss_mel_ce: 3.461113452911377 (nan) | > loss: 0.013943796046078205 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13635226138394677) | > loader_time: 0.0046 (0.004715124170461906)  --> TIME: 2025-02-24 13:40:57 -- STEP: 10058/36938 -- GLOBAL_STEP: 342500 | > loss_text_ce: 0.03912602737545967 (0.0403938267302588) | > loss_mel_ce: 2.07861328125 (nan) | > loss: 0.008403727784752846 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13636084765841766) | > loader_time: 0.0047 (0.0047148953472129045)  --> TIME: 2025-02-24 13:41:07 -- STEP: 10108/36938 -- GLOBAL_STEP: 342550 | > loss_text_ce: 0.033612947911024094 (0.040388869585673406) | > loss_mel_ce: 3.564131736755371 (nan) | > loss: 0.014276765286922455 (nan) | > current_lr: 5e-06 | > step_time: 0.1822 (0.1363774843257453) | > loader_time: 0.0049 (0.004715123004860487)  --> TIME: 2025-02-24 13:41:16 -- STEP: 10158/36938 -- GLOBAL_STEP: 342600 | > loss_text_ce: 0.042304445058107376 (0.04039257626593992) | > loss_mel_ce: 2.745422840118408 (nan) | > loss: 0.011062410660088062 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13639367748654035) | > loader_time: 0.0047 (0.004715484623647051)  --> TIME: 2025-02-24 13:41:26 -- STEP: 10208/36938 -- GLOBAL_STEP: 342650 | > loss_text_ce: 0.0411541722714901 (0.040383641310452086) | > loss_mel_ce: 3.23514461517334 (nan) | > loss: 0.013001186773180962 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13640958243589768) | > loader_time: 0.0049 (0.004715624951456775)  --> TIME: 2025-02-24 13:41:35 -- STEP: 10258/36938 -- GLOBAL_STEP: 342700 | > loss_text_ce: 0.04331415519118309 (0.040385490870054094) | > loss_mel_ce: 2.661001205444336 (nan) | > loss: 0.010731411166489124 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13641342965221032) | > loader_time: 0.0053 (0.004715609908406083)  --> TIME: 2025-02-24 13:41:45 -- STEP: 10308/36938 -- GLOBAL_STEP: 342750 | > loss_text_ce: 0.03919444605708122 (0.04039545671282152) | > loss_mel_ce: 2.144259452819824 (nan) | > loss: 0.008664499968290329 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.1364162419024575) | > loader_time: 0.0046 (0.004716057300012511)  --> TIME: 2025-02-24 13:41:54 -- STEP: 10358/36938 -- GLOBAL_STEP: 342800 | > loss_text_ce: 0.04604443535208702 (0.04039427161225356) | > loss_mel_ce: 3.4624693393707275 (nan) | > loss: 0.0139226745814085 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13643545981515978) | > loader_time: 0.0046 (0.004716267869333927)  --> TIME: 2025-02-24 13:42:03 -- STEP: 10408/36938 -- GLOBAL_STEP: 342850 | > loss_text_ce: 0.03982783108949661 (0.04039309765419826) | > loss_mel_ce: 3.372335910797119 (nan) | > loss: 0.01354033313691616 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.1364401793085914) | > loader_time: 0.0047 (0.004715968424498705)  --> TIME: 2025-02-24 13:42:13 -- STEP: 10458/36938 -- GLOBAL_STEP: 342900 | > loss_text_ce: 0.05083303898572922 (0.040390813780735065) | > loss_mel_ce: 2.986259698867798 (nan) | > loss: 0.012051955796778202 (nan) | > current_lr: 5e-06 | > step_time: 0.1074 (0.13643732370028952) | > loader_time: 0.0045 (0.0047161772000810075)  --> TIME: 2025-02-24 13:42:22 -- STEP: 10508/36938 -- GLOBAL_STEP: 342950 | > loss_text_ce: 0.014493699185550213 (0.040388007890337294) | > loss_mel_ce: 2.700882911682129 (nan) | > loss: 0.010775304399430752 (nan) | > current_lr: 5e-06 | > step_time: 0.1068 (0.13644607320320115) | > loader_time: 0.0054 (0.0047164512397583)  --> TIME: 2025-02-24 13:42:31 -- STEP: 10558/36938 -- GLOBAL_STEP: 343000 | > loss_text_ce: 0.021267611533403397 (0.04038321127642953) | > loss_mel_ce: 2.2758829593658447 (nan) | > loss: 0.009115677326917648 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.1364490775182224) | > loader_time: 0.0045 (0.004716369165889135)  --> TIME: 2025-02-24 13:42:41 -- STEP: 10608/36938 -- GLOBAL_STEP: 343050 | > loss_text_ce: 0.045605216175317764 (0.04038881839084086) | > loss_mel_ce: 3.0436532497406006 (nan) | > loss: 0.012258962728083134 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13644115130106588) | > loader_time: 0.0046 (0.004716314813669977)  --> TIME: 2025-02-24 13:42:49 -- STEP: 10658/36938 -- GLOBAL_STEP: 343100 | > loss_text_ce: 0.04101632907986641 (0.04038994845296839) | > loss_mel_ce: 3.7825100421905518 (nan) | > loss: 0.015172724612057209 (nan) | > current_lr: 5e-06 | > step_time: 0.1352 (0.13639509127586955) | > loader_time: 0.0046 (0.004716126170295276)  --> TIME: 2025-02-24 13:42:59 -- STEP: 10708/36938 -- GLOBAL_STEP: 343150 | > loss_text_ce: 0.04259197786450386 (0.04038985730475872) | > loss_mel_ce: 3.408453941345215 (nan) | > loss: 0.01369462814182043 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13639621639768237) | > loader_time: 0.0049 (0.004715963023609291)  --> TIME: 2025-02-24 13:43:08 -- STEP: 10758/36938 -- GLOBAL_STEP: 343200 | > loss_text_ce: 0.03339871019124985 (0.04038532186141308) | > loss_mel_ce: 2.793111801147461 (nan) | > loss: 0.01121631171554327 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13640763613787174) | > loader_time: 0.0046 (0.004716294275869935)  --> TIME: 2025-02-24 13:43:17 -- STEP: 10808/36938 -- GLOBAL_STEP: 343250 | > loss_text_ce: 0.03703717514872551 (0.04037175296268442) | > loss_mel_ce: 2.9798731803894043 (nan) | > loss: 0.011971866711974144 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13640968661498842) | > loader_time: 0.0045 (0.004716991738510679)  --> TIME: 2025-02-24 13:43:27 -- STEP: 10858/36938 -- GLOBAL_STEP: 343300 | > loss_text_ce: 0.04187418520450592 (0.0403713205427674) | > loss_mel_ce: 3.0766804218292236 (nan) | > loss: 0.01237521693110466 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13642317147781013) | > loader_time: 0.0046 (0.004717003071718724)  --> TIME: 2025-02-24 13:43:36 -- STEP: 10908/36938 -- GLOBAL_STEP: 343350 | > loss_text_ce: 0.04642971232533455 (0.04037901782703926) | > loss_mel_ce: 3.3709096908569336 (nan) | > loss: 0.013560870662331581 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.1364360955288698) | > loader_time: 0.0048 (0.0047173912288244425)  --> TIME: 2025-02-24 13:43:46 -- STEP: 10958/36938 -- GLOBAL_STEP: 343400 | > loss_text_ce: 0.0321427658200264 (0.040379160641791034) | > loss_mel_ce: 3.135847806930542 (nan) | > loss: 0.012571392580866814 (nan) | > current_lr: 5e-06 | > step_time: 0.1451 (0.136444163204958) | > loader_time: 0.0046 (0.004717117592591721)  --> TIME: 2025-02-24 13:43:55 -- STEP: 11008/36938 -- GLOBAL_STEP: 343450 | > loss_text_ce: 0.03887369483709335 (0.04038303319358359) | > loss_mel_ce: 2.658395528793335 (nan) | > loss: 0.010703450068831444 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13641785483720678) | > loader_time: 0.0047 (0.00471731697664009)  --> TIME: 2025-02-24 13:44:04 -- STEP: 11058/36938 -- GLOBAL_STEP: 343500 | > loss_text_ce: 0.03675886616110802 (0.04038178583413033) | > loss_mel_ce: 6.3828125 (nan) | > loss: 0.025474490597844124 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13642946208530218) | > loader_time: 0.0044 (0.004717721174112102)  --> TIME: 2025-02-24 13:44:13 -- STEP: 11108/36938 -- GLOBAL_STEP: 343550 | > loss_text_ce: 0.04378574341535568 (0.040389710076694604) | > loss_mel_ce: 2.903437852859497 (nan) | > loss: 0.011695332825183868 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13641693682648268) | > loader_time: 0.0046 (0.0047180743195145754)  --> TIME: 2025-02-24 13:44:23 -- STEP: 11158/36938 -- GLOBAL_STEP: 343600 | > loss_text_ce: 0.04007657989859581 (0.04038428370940177) | > loss_mel_ce: 3.50628662109375 (nan) | > loss: 0.014072869904339314 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13641568690906997) | > loader_time: 0.0047 (0.0047183247274086345)  --> TIME: 2025-02-24 13:44:32 -- STEP: 11208/36938 -- GLOBAL_STEP: 343650 | > loss_text_ce: 0.04150014370679855 (0.04038111789098518) | > loss_mel_ce: 3.46728253364563 (nan) | > loss: 0.013923740945756435 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13643864018418475) | > loader_time: 0.0046 (0.004718580941997336)  --> TIME: 2025-02-24 13:44:41 -- STEP: 11258/36938 -- GLOBAL_STEP: 343700 | > loss_text_ce: 0.05141199380159378 (0.04038255424097287) | > loss_mel_ce: 3.399505853652954 (nan) | > loss: 0.01369411963969469 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13642364474591545) | > loader_time: 0.0045 (0.0047185912736250986)  --> TIME: 2025-02-24 13:44:51 -- STEP: 11308/36938 -- GLOBAL_STEP: 343750 | > loss_text_ce: 0.028816835954785347 (0.04038155749352903) | > loss_mel_ce: 2.5767054557800293 (nan) | > loss: 0.010339375585317612 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13644224457998863) | > loader_time: 0.0048 (0.004719100932076383)  --> TIME: 2025-02-24 13:45:00 -- STEP: 11358/36938 -- GLOBAL_STEP: 343800 | > loss_text_ce: 0.04441523924469948 (0.040379070902929165) | > loss_mel_ce: 2.4027373790740967 (nan) | > loss: 0.00971092376857996 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.13645362142400658) | > loader_time: 0.0048 (0.004719287917991378)  --> TIME: 2025-02-24 13:45:10 -- STEP: 11408/36938 -- GLOBAL_STEP: 343850 | > loss_text_ce: 0.04387127235531807 (0.0403794431653192) | > loss_mel_ce: 3.485767364501953 (nan) | > loss: 0.01400650292634964 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.13645991461474302) | > loader_time: 0.0048 (0.004718722208686469)  --> TIME: 2025-02-24 13:45:19 -- STEP: 11458/36938 -- GLOBAL_STEP: 343900 | > loss_text_ce: 0.04140964522957802 (0.040380556943250476) | > loss_mel_ce: 3.083768844604492 (nan) | > loss: 0.012401502579450607 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13644042883187332) | > loader_time: 0.0046 (0.004718933685597016)  --> TIME: 2025-02-24 13:45:29 -- STEP: 11508/36938 -- GLOBAL_STEP: 343950 | > loss_text_ce: 0.035201482474803925 (0.04038105885657499) | > loss_mel_ce: 3.294043779373169 (nan) | > loss: 0.013211292214691639 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.13647269224764208) | > loader_time: 0.0048 (0.0047197841628375675)  --> TIME: 2025-02-24 13:45:38 -- STEP: 11558/36938 -- GLOBAL_STEP: 344000 | > loss_text_ce: 0.040140409022569656 (0.04038425136045227) | > loss_mel_ce: 2.967756986618042 (nan) | > loss: 0.011936101131141186 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1364656544243265) | > loader_time: 0.0046 (0.0047202364015092105)  --> TIME: 2025-02-24 13:45:47 -- STEP: 11608/36938 -- GLOBAL_STEP: 344050 | > loss_text_ce: 0.027913613244891167 (0.040382578270108736) | > loss_mel_ce: 3.0062479972839355 (nan) | > loss: 0.01204032450914383 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13646585094279723) | > loader_time: 0.0045 (0.004720023280583406)  --> TIME: 2025-02-24 13:45:57 -- STEP: 11658/36938 -- GLOBAL_STEP: 344100 | > loss_text_ce: 0.03359474241733551 (0.040387602389318336) | > loss_mel_ce: 3.413832664489746 (nan) | > loss: 0.01368026901036501 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13647471394132327) | > loader_time: 0.0054 (0.0047203647188470005)  --> TIME: 2025-02-24 13:46:06 -- STEP: 11708/36938 -- GLOBAL_STEP: 344150 | > loss_text_ce: 0.04720776900649071 (0.04037903970609211) | > loss_mel_ce: 2.9312245845794678 (nan) | > loss: 0.011819177307188511 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13647775994986064) | > loader_time: 0.0046 (0.004719879446397805)  --> TIME: 2025-02-24 13:46:15 -- STEP: 11758/36938 -- GLOBAL_STEP: 344200 | > loss_text_ce: 0.03380388766527176 (0.04037767325411216) | > loss_mel_ce: 3.481201171875 (nan) | > loss: 0.01394843403249979 (nan) | > current_lr: 5e-06 | > step_time: 0.1357 (0.13644875113829058) | > loader_time: 0.0046 (0.004719357990168179)  --> TIME: 2025-02-24 13:46:24 -- STEP: 11808/36938 -- GLOBAL_STEP: 344250 | > loss_text_ce: 0.04435222968459129 (0.0403724162375427) | > loss_mel_ce: 3.2685546875 (nan) | > loss: 0.013146457262337208 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13645349366232012) | > loader_time: 0.0046 (0.0047200294691049334)  --> TIME: 2025-02-24 13:46:34 -- STEP: 11858/36938 -- GLOBAL_STEP: 344300 | > loss_text_ce: 0.039215393364429474 (0.0403710078759231) | > loss_mel_ce: 3.11004638671875 (nan) | > loss: 0.01249707117676735 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13646613103902316) | > loader_time: 0.0045 (0.004719907868710683)  --> TIME: 2025-02-24 13:46:43 -- STEP: 11908/36938 -- GLOBAL_STEP: 344350 | > loss_text_ce: 0.04455410689115524 (0.04037007678252904) | > loss_mel_ce: 3.690882444381714 (nan) | > loss: 0.01482316106557846 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.13646168461733357) | > loader_time: 0.0048 (0.004719966403742036)  --> TIME: 2025-02-24 13:46:52 -- STEP: 11958/36938 -- GLOBAL_STEP: 344400 | > loss_text_ce: 0.03948317840695381 (0.04036910246958299) | > loss_mel_ce: 3.4371261596679688 (nan) | > loss: 0.013796068727970123 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.13646030764971415) | > loader_time: 0.0048 (0.004719794544214203)  --> TIME: 2025-02-24 13:47:01 -- STEP: 12008/36938 -- GLOBAL_STEP: 344450 | > loss_text_ce: 0.0313127301633358 (0.04037527777361614) | > loss_mel_ce: 2.927236557006836 (nan) | > loss: 0.011740275658667088 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13646177682457317) | > loader_time: 0.0045 (0.0047198348963443035)  --> TIME: 2025-02-24 13:47:10 -- STEP: 12058/36938 -- GLOBAL_STEP: 344500 | > loss_text_ce: 0.040079303085803986 (0.040379562746760654) | > loss_mel_ce: 3.3511595726013184 (nan) | > loss: 0.013457298278808594 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13644770722540336) | > loader_time: 0.0047 (0.004719699134991141)  --> TIME: 2025-02-24 13:47:19 -- STEP: 12108/36938 -- GLOBAL_STEP: 344550 | > loss_text_ce: 0.02338443696498871 (0.04037764417935198) | > loss_mel_ce: 1.689149022102356 (nan) | > loss: 0.0067957681603729725 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13643115815999823) | > loader_time: 0.0045 (0.0047197318880479785)  --> TIME: 2025-02-24 13:47:29 -- STEP: 12158/36938 -- GLOBAL_STEP: 344600 | > loss_text_ce: 0.043702974915504456 (0.04038067128155997) | > loss_mel_ce: 3.48730206489563 (nan) | > loss: 0.014011926017701626 (nan) | > current_lr: 5e-06 | > step_time: 0.1818 (0.13644028678695808) | > loader_time: 0.0055 (0.004720327277229784)  --> TIME: 2025-02-24 13:47:38 -- STEP: 12208/36938 -- GLOBAL_STEP: 344650 | > loss_text_ce: 0.04304708540439606 (0.04038624225242607) | > loss_mel_ce: 3.0672237873077393 (nan) | > loss: 0.012342345900833607 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.13642734177850085) | > loader_time: 0.0048 (0.0047202189776406524)  --> TIME: 2025-02-24 13:47:47 -- STEP: 12258/36938 -- GLOBAL_STEP: 344700 | > loss_text_ce: 0.03477853536605835 (0.04038301534878711) | > loss_mel_ce: 2.899081230163574 (nan) | > loss: 0.011642301455140114 (nan) | > current_lr: 5e-06 | > step_time: 0.1881 (0.13643716505445638) | > loader_time: 0.0048 (0.004719979437437319)  --> TIME: 2025-02-24 13:47:57 -- STEP: 12308/36938 -- GLOBAL_STEP: 344750 | > loss_text_ce: 0.0357808880507946 (0.04037579393287022) | > loss_mel_ce: 3.131223678588867 (nan) | > loss: 0.01256747916340828 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13642636391881843) | > loader_time: 0.0046 (0.004719495657089245)  --> TIME: 2025-02-24 13:48:06 -- STEP: 12358/36938 -- GLOBAL_STEP: 344800 | > loss_text_ce: 0.043856311589479446 (0.04038057488664084) | > loss_mel_ce: 2.56632661819458 (nan) | > loss: 0.010357869789004326 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13644363079898383) | > loader_time: 0.0046 (0.004719722269036275)  --> TIME: 2025-02-24 13:48:15 -- STEP: 12408/36938 -- GLOBAL_STEP: 344850 | > loss_text_ce: 0.04653111472725868 (0.040380491758846886) | > loss_mel_ce: 0.9910496473312378 (nan) | > loss: 0.00411738408729434 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13643480851756373) | > loader_time: 0.0046 (0.004719726198492741)  --> TIME: 2025-02-24 13:48:25 -- STEP: 12458/36938 -- GLOBAL_STEP: 344900 | > loss_text_ce: 0.032634783536195755 (0.04038243039105517) | > loss_mel_ce: 3.6320853233337402 (nan) | > loss: 0.014542540535330772 (nan) | > current_lr: 5e-06 | > step_time: 0.1868 (0.136454914484654) | > loader_time: 0.0049 (0.00471993245939164)  --> TIME: 2025-02-24 13:48:34 -- STEP: 12508/36938 -- GLOBAL_STEP: 344950 | > loss_text_ce: 0.024530701339244843 (0.040383468671628335) | > loss_mel_ce: 2.72271466255188 (nan) | > loss: 0.010901767760515213 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1364506469544496) | > loader_time: 0.0045 (0.0047195587708692656)  --> TIME: 2025-02-24 13:48:43 -- STEP: 12558/36938 -- GLOBAL_STEP: 345000 | > loss_text_ce: 0.04498957842588425 (0.04038240130583452) | > loss_mel_ce: 2.5081987380981445 (nan) | > loss: 0.010131699964404106 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13644569615760407) | > loader_time: 0.0046 (0.004719470674624386)  --> TIME: 2025-02-24 13:48:53 -- STEP: 12608/36938 -- GLOBAL_STEP: 345050 | > loss_text_ce: 0.03588739037513733 (0.04038239402280179) | > loss_mel_ce: 2.9947187900543213 (nan) | > loss: 0.01202621590346098 (nan) | > current_lr: 5e-06 | > step_time: 0.1867 (0.13643751144031052) | > loader_time: 0.0048 (0.004719296820121326)  --> TIME: 2025-02-24 13:49:02 -- STEP: 12658/36938 -- GLOBAL_STEP: 345100 | > loss_text_ce: 0.029803141951560974 (0.04037509946185535) | > loss_mel_ce: 2.9420604705810547 (nan) | > loss: 0.011793109588325024 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13643426860484534) | > loader_time: 0.0046 (0.004719307513424558)  --> TIME: 2025-02-24 13:49:11 -- STEP: 12708/36938 -- GLOBAL_STEP: 345150 | > loss_text_ce: 0.04606416076421738 (0.04037733782582028) | > loss_mel_ce: 3.2148571014404297 (nan) | > loss: 0.012940164655447006 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13642708449233193) | > loader_time: 0.0047 (0.0047192659098883255)  --> TIME: 2025-02-24 13:49:21 -- STEP: 12758/36938 -- GLOBAL_STEP: 345200 | > loss_text_ce: 0.038249671459198 (0.040375931252630434) | > loss_mel_ce: 3.265434503555298 (nan) | > loss: 0.013109859079122543 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13644720168486776) | > loader_time: 0.0054 (0.004720373706096989)  --> TIME: 2025-02-24 13:49:30 -- STEP: 12808/36938 -- GLOBAL_STEP: 345250 | > loss_text_ce: 0.03982068598270416 (0.040375235704934946) | > loss_mel_ce: 2.602321147918701 (nan) | > loss: 0.01048469077795744 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13643983233578186) | > loader_time: 0.0046 (0.004720506837113325)  --> TIME: 2025-02-24 13:49:39 -- STEP: 12858/36938 -- GLOBAL_STEP: 345300 | > loss_text_ce: 0.026287488639354706 (0.04038080786718306) | > loss_mel_ce: 3.473750114440918 (nan) | > loss: 0.013889038935303688 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13643793850965696) | > loader_time: 0.0046 (0.004720543717351519)  --> TIME: 2025-02-24 13:49:48 -- STEP: 12908/36938 -- GLOBAL_STEP: 345350 | > loss_text_ce: 0.02929459512233734 (0.04037283838214026) | > loss_mel_ce: 1.4714730978012085 (nan) | > loss: 0.005955427885055542 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13644157813213817) | > loader_time: 0.0045 (0.004721393424880183)  --> TIME: 2025-02-24 13:49:58 -- STEP: 12958/36938 -- GLOBAL_STEP: 345400 | > loss_text_ce: 0.04896744713187218 (0.04038059918297887) | > loss_mel_ce: 3.2819082736968994 (nan) | > loss: 0.013217761181294918 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13643330294218628) | > loader_time: 0.0046 (0.004721311990043414)  --> TIME: 2025-02-24 13:50:07 -- STEP: 13008/36938 -- GLOBAL_STEP: 345450 | > loss_text_ce: 0.051755599677562714 (0.040380142693511555) | > loss_mel_ce: 1.745649814605713 (nan) | > loss: 0.007132561411708593 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13645036727519994) | > loader_time: 0.0046 (0.004722043212049525)  --> TIME: 2025-02-24 13:50:17 -- STEP: 13058/36938 -- GLOBAL_STEP: 345500 | > loss_text_ce: 0.05437209829688072 (0.040386538891608786) | > loss_mel_ce: 3.818397283554077 (nan) | > loss: 0.01536813285201788 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13645387643474394) | > loader_time: 0.0053 (0.004722564869333034)  --> TIME: 2025-02-24 13:50:26 -- STEP: 13108/36938 -- GLOBAL_STEP: 345550 | > loss_text_ce: 0.04497896879911423 (0.04038863109336898) | > loss_mel_ce: 2.7237932682037354 (nan) | > loss: 0.010987191461026669 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13646878687244884) | > loader_time: 0.0053 (0.004723247009914062)  --> TIME: 2025-02-24 13:50:35 -- STEP: 13158/36938 -- GLOBAL_STEP: 345600 | > loss_text_ce: 0.03272589296102524 (0.040385295609701694) | > loss_mel_ce: 0.3346630930900574 (nan) | > loss: 0.0014578929403796792 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13645614385786115) | > loader_time: 0.0052 (0.004723939331745813)  --> TIME: 2025-02-24 13:50:45 -- STEP: 13208/36938 -- GLOBAL_STEP: 345650 | > loss_text_ce: 0.037344761192798615 (0.04038407742774919) | > loss_mel_ce: 2.7944416999816895 (nan) | > loss: 0.011237247847020626 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1364738391717953) | > loader_time: 0.0045 (0.004724617115589112)  --> TIME: 2025-02-24 13:50:54 -- STEP: 13258/36938 -- GLOBAL_STEP: 345700 | > loss_text_ce: 0.044384367763996124 (0.04037942043822424) | > loss_mel_ce: 3.130005359649658 (nan) | > loss: 0.01259678602218628 (nan) | > current_lr: 5e-06 | > step_time: 0.1837 (0.13646505808359147) | > loader_time: 0.005 (0.004724907576543244)  --> TIME: 2025-02-24 13:51:03 -- STEP: 13308/36938 -- GLOBAL_STEP: 345750 | > loss_text_ce: 0.03838687390089035 (0.04038397327107742) | > loss_mel_ce: 3.5943219661712646 (nan) | > loss: 0.01441551186144352 (nan) | > current_lr: 5e-06 | > step_time: 0.1906 (0.1364635173752034) | > loader_time: 0.0048 (0.0047249535844961325)  --> TIME: 2025-02-24 13:51:13 -- STEP: 13358/36938 -- GLOBAL_STEP: 345800 | > loss_text_ce: 0.04827180877327919 (0.04037819131711027) | > loss_mel_ce: 3.14395809173584 (nan) | > loss: 0.0126675795763731 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.1364622531242048) | > loader_time: 0.0048 (0.004725158009312761)  --> TIME: 2025-02-24 13:51:22 -- STEP: 13408/36938 -- GLOBAL_STEP: 345850 | > loss_text_ce: 0.04313015192747116 (0.04037779402219793) | > loss_mel_ce: 3.4305367469787598 (nan) | > loss: 0.013784393668174744 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1364623007135633) | > loader_time: 0.0045 (0.004724948833694289)  --> TIME: 2025-02-24 13:51:31 -- STEP: 13458/36938 -- GLOBAL_STEP: 345900 | > loss_text_ce: 0.03644006699323654 (0.04037411877902628) | > loss_mel_ce: 3.1974432468414307 (nan) | > loss: 0.01283287163823843 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.1364470186515558) | > loader_time: 0.0053 (0.004724966379557632)  --> TIME: 2025-02-24 13:51:40 -- STEP: 13508/36938 -- GLOBAL_STEP: 345950 | > loss_text_ce: 0.033545609563589096 (0.04037047626288226) | > loss_mel_ce: 2.745521306991577 (nan) | > loss: 0.011028043925762177 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.13644701104775556) | > loader_time: 0.0046 (0.004725476694347066)  --> TIME: 2025-02-24 13:51:49 -- STEP: 13558/36938 -- GLOBAL_STEP: 346000 | > loss_text_ce: 0.04289349913597107 (0.040375712778208284) | > loss_mel_ce: 2.5498483180999756 (nan) | > loss: 0.010288658551871777 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1364364384690435) | > loader_time: 0.0045 (0.004725657622183546)  --> TIME: 2025-02-24 13:51:59 -- STEP: 13608/36938 -- GLOBAL_STEP: 346050 | > loss_text_ce: 0.04782513529062271 (0.04036897973592471) | > loss_mel_ce: 1.4300462007522583 (nan) | > loss: 0.0058645689859986305 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13644907495191988) | > loader_time: 0.0045 (0.004725610155136956)  --> TIME: 2025-02-24 13:52:08 -- STEP: 13658/36938 -- GLOBAL_STEP: 346100 | > loss_text_ce: 0.04614018648862839 (0.04037253877288779) | > loss_mel_ce: 1.9107208251953125 (nan) | > loss: 0.007765321992337704 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13644101213337054) | > loader_time: 0.0052 (0.004725709965574988)  --> TIME: 2025-02-24 13:52:17 -- STEP: 13708/36938 -- GLOBAL_STEP: 346150 | > loss_text_ce: 0.03559369966387749 (0.04037126754562756) | > loss_mel_ce: 3.047394275665283 (nan) | > loss: 0.012234080582857132 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13644564276991067) | > loader_time: 0.0046 (0.004725806925990372)  --> TIME: 2025-02-24 13:52:27 -- STEP: 13758/36938 -- GLOBAL_STEP: 346200 | > loss_text_ce: 0.0334865041077137 (0.04036965866424641) | > loss_mel_ce: 1.4414775371551514 (nan) | > loss: 0.005853032227605581 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13644713138248157) | > loader_time: 0.0046 (0.004725404093476575)  --> TIME: 2025-02-24 13:52:36 -- STEP: 13808/36938 -- GLOBAL_STEP: 346250 | > loss_text_ce: 0.05009107291698456 (0.040373805287870955) | > loss_mel_ce: 3.851454973220825 (nan) | > loss: 0.015482326038181782 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13643858900351996) | > loader_time: 0.0046 (0.004724925580334108)  --> TIME: 2025-02-24 13:52:45 -- STEP: 13858/36938 -- GLOBAL_STEP: 346300 | > loss_text_ce: 0.03864071145653725 (0.040372050997339745) | > loss_mel_ce: 0.02771504782140255 (nan) | > loss: 0.0002633164986036718 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13642590309844468) | > loader_time: 0.0045 (0.00472464178148361)  --> TIME: 2025-02-24 13:52:54 -- STEP: 13908/36938 -- GLOBAL_STEP: 346350 | > loss_text_ce: 0.040501970797777176 (0.04037171651286217) | > loss_mel_ce: 3.3919215202331543 (nan) | > loss: 0.013620729558169842 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.1364273678169605) | > loader_time: 0.0046 (0.004725098575639187)  --> TIME: 2025-02-24 13:53:04 -- STEP: 13958/36938 -- GLOBAL_STEP: 346400 | > loss_text_ce: 0.03049415536224842 (0.04036756160032894) | > loss_mel_ce: 3.131603479385376 (nan) | > loss: 0.012548007071018219 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13645039562646807) | > loader_time: 0.0046 (0.004724819486983958)  --> TIME: 2025-02-24 13:53:13 -- STEP: 14008/36938 -- GLOBAL_STEP: 346450 | > loss_text_ce: 0.037547558546066284 (0.04036498791411862) | > loss_mel_ce: 3.384172201156616 (nan) | > loss: 0.013578253798186779 (nan) | > current_lr: 5e-06 | > step_time: 0.1783 (0.1364569812587576) | > loader_time: 0.0048 (0.004725112634546586)  --> TIME: 2025-02-24 13:53:23 -- STEP: 14058/36938 -- GLOBAL_STEP: 346500 | > loss_text_ce: 0.030896300449967384 (0.04036134133525627) | > loss_mel_ce: 1.54498291015625 (nan) | > loss: 0.006253489293158054 (nan) | > current_lr: 5e-06 | > step_time: 0.1628 (0.13648477260716085) | > loader_time: 0.0067 (0.004725529062474875)  --> TIME: 2025-02-24 13:53:32 -- STEP: 14108/36938 -- GLOBAL_STEP: 346550 | > loss_text_ce: 0.035648833960294724 (0.04036234786924373) | > loss_mel_ce: 3.461700677871704 (nan) | > loss: 0.013878371566534042 (nan) | > current_lr: 5e-06 | > step_time: 0.1563 (0.13648260658781722) | > loader_time: 0.0046 (0.004726010255107844)  --> TIME: 2025-02-24 13:53:41 -- STEP: 14158/36938 -- GLOBAL_STEP: 346600 | > loss_text_ce: 0.04653003439307213 (0.04036098266078835) | > loss_mel_ce: 0.1572265625 (nan) | > loss: 0.0008085579611361027 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13646662442390167) | > loader_time: 0.0046 (0.004726054776538313)  --> TIME: 2025-02-24 13:53:51 -- STEP: 14208/36938 -- GLOBAL_STEP: 346650 | > loss_text_ce: 0.040373969823122025 (0.04035295895589227) | > loss_mel_ce: 0.0250244140625 (nan) | > loss: 0.0002595174009911716 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13647105318267613) | > loader_time: 0.0051 (0.004726144711713503)  --> TIME: 2025-02-24 13:54:00 -- STEP: 14258/36938 -- GLOBAL_STEP: 346700 | > loss_text_ce: 0.054388225078582764 (0.0403482714240206) | > loss_mel_ce: 0.1854248046875 (nan) | > loss: 0.0009516390855424106 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.1364603381552225) | > loader_time: 0.0046 (0.004725719320939594)  --> TIME: 2025-02-24 13:54:09 -- STEP: 14308/36938 -- GLOBAL_STEP: 346750 | > loss_text_ce: 0.03709631785750389 (0.040353044865953425) | > loss_mel_ce: 1.7439810037612915 (nan) | > loss: 0.007067767437547445 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1364712168925764) | > loader_time: 0.0045 (0.004725530089564156)  --> TIME: 2025-02-24 13:54:19 -- STEP: 14358/36938 -- GLOBAL_STEP: 346800 | > loss_text_ce: 0.043414901942014694 (0.040356180733628955) | > loss_mel_ce: 3.1301968097686768 (nan) | > loss: 0.012593697756528854 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13646591501758054) | > loader_time: 0.0046 (0.004725812454212976)  --> TIME: 2025-02-24 13:54:28 -- STEP: 14408/36938 -- GLOBAL_STEP: 346850 | > loss_text_ce: 0.045766547322273254 (0.040355934609303805) | > loss_mel_ce: 2.2034287452697754 (nan) | > loss: 0.008925379253923893 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.1364860808697099) | > loader_time: 0.0047 (0.00472561727954834)  --> TIME: 2025-02-24 13:54:38 -- STEP: 14458/36938 -- GLOBAL_STEP: 346900 | > loss_text_ce: 0.04036372900009155 (0.04035639197012031) | > loss_mel_ce: 3.205458164215088 (nan) | > loss: 0.01288024615496397 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.13648158487466938) | > loader_time: 0.0046 (0.004725507374603964)  --> TIME: 2025-02-24 13:54:47 -- STEP: 14508/36938 -- GLOBAL_STEP: 346950 | > loss_text_ce: 0.03662516176700592 (0.040356313845957774) | > loss_mel_ce: 2.887956142425537 (nan) | > loss: 0.011605481617152691 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13648270796506853) | > loader_time: 0.0045 (0.004725513032293172)  --> TIME: 2025-02-24 13:54:56 -- STEP: 14558/36938 -- GLOBAL_STEP: 347000 | > loss_text_ce: 0.04536987096071243 (0.04035396946099994) | > loss_mel_ce: 1.807824730873108 (nan) | > loss: 0.007353947032243013 (nan) | > current_lr: 5e-06 | > step_time: 0.1516 (0.13648234811977542) | > loader_time: 0.0046 (0.004725420044286976)  --> TIME: 2025-02-24 13:55:05 -- STEP: 14608/36938 -- GLOBAL_STEP: 347050 | > loss_text_ce: 0.04271727427840233 (0.040349265941885946) | > loss_mel_ce: 3.613675594329834 (nan) | > loss: 0.01450949627906084 (nan) | > current_lr: 5e-06 | > step_time: 0.1452 (0.136473231499062) | > loader_time: 0.0053 (0.004725360824662229)  --> TIME: 2025-02-24 13:55:15 -- STEP: 14658/36938 -- GLOBAL_STEP: 347100 | > loss_text_ce: 0.0399140790104866 (0.04035246080868505) | > loss_mel_ce: 2.5956532955169678 (nan) | > loss: 0.010458601638674736 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1364742618804902) | > loader_time: 0.0045 (0.00472569872272794)  --> TIME: 2025-02-24 13:55:24 -- STEP: 14708/36938 -- GLOBAL_STEP: 347150 | > loss_text_ce: 0.046081721782684326 (0.04035174173561555) | > loss_mel_ce: 3.0374345779418945 (nan) | > loss: 0.012236176989972591 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.13648725977597564) | > loader_time: 0.0055 (0.004725833285396278)  --> TIME: 2025-02-24 13:55:34 -- STEP: 14758/36938 -- GLOBAL_STEP: 347200 | > loss_text_ce: 0.03853197395801544 (0.04035076102349664) | > loss_mel_ce: 3.673011541366577 (nan) | > loss: 0.014728348702192307 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.1364904929344725) | > loader_time: 0.0052 (0.00472596816406578)  --> TIME: 2025-02-24 13:55:43 -- STEP: 14808/36938 -- GLOBAL_STEP: 347250 | > loss_text_ce: 0.045494962483644485 (0.040348741532314006) | > loss_mel_ce: 2.721695899963379 (nan) | > loss: 0.010980917140841484 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13650217074307167) | > loader_time: 0.0052 (0.00472608448556148)  --> TIME: 2025-02-24 13:55:52 -- STEP: 14858/36938 -- GLOBAL_STEP: 347300 | > loss_text_ce: 0.045038554817438126 (0.04034839692964466) | > loss_mel_ce: 2.7125473022460938 (nan) | > loss: 0.010942800901830196 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13650858744042207) | > loader_time: 0.0046 (0.00472624960778809)  --> TIME: 2025-02-24 13:56:02 -- STEP: 14908/36938 -- GLOBAL_STEP: 347350 | > loss_text_ce: 0.033223338425159454 (0.04035122653665358) | > loss_mel_ce: 3.3163976669311523 (nan) | > loss: 0.013292147777974606 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13650683223449556) | > loader_time: 0.0045 (0.004726437179595084)  --> TIME: 2025-02-24 13:56:11 -- STEP: 14958/36938 -- GLOBAL_STEP: 347400 | > loss_text_ce: 0.05004413425922394 (0.04035186933669593) | > loss_mel_ce: 3.6040523052215576 (nan) | > loss: 0.014500383287668228 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13648955366767362) | > loader_time: 0.0046 (0.004726413386855318)  --> TIME: 2025-02-24 13:56:20 -- STEP: 15008/36938 -- GLOBAL_STEP: 347450 | > loss_text_ce: 0.0487874336540699 (0.04034855719797242) | > loss_mel_ce: 1.1922880411148071 (nan) | > loss: 0.004924903158098459 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.1364907738147005) | > loader_time: 0.0045 (0.004726964400521191)  --> TIME: 2025-02-24 13:56:29 -- STEP: 15058/36938 -- GLOBAL_STEP: 347500 | > loss_text_ce: 0.04842492938041687 (0.040348464768191455) | > loss_mel_ce: 3.3739302158355713 (nan) | > loss: 0.013580774888396263 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13648419425154695) | > loader_time: 0.0045 (0.00472733298056087)  --> TIME: 2025-02-24 13:56:39 -- STEP: 15108/36938 -- GLOBAL_STEP: 347550 | > loss_text_ce: 0.04285064712166786 (0.04034609871836093) | > loss_mel_ce: 3.4021193981170654 (nan) | > loss: 0.01367051713168621 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.13648111604968796) | > loader_time: 0.0046 (0.004727018662221132)  --> TIME: 2025-02-24 13:56:48 -- STEP: 15158/36938 -- GLOBAL_STEP: 347600 | > loss_text_ce: 0.048247359693050385 (0.04034696467795138) | > loss_mel_ce: 3.424943208694458 (nan) | > loss: 0.013782503083348274 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1364707843281219) | > loader_time: 0.0045 (0.004726602166383813)  --> TIME: 2025-02-24 13:56:57 -- STEP: 15208/36938 -- GLOBAL_STEP: 347650 | > loss_text_ce: 0.06688845157623291 (0.04035159083775767) | > loss_mel_ce: 3.541135787963867 (nan) | > loss: 0.014317557215690613 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13646592921736608) | > loader_time: 0.0046 (0.004726973663562825)  --> TIME: 2025-02-24 13:57:07 -- STEP: 15258/36938 -- GLOBAL_STEP: 347700 | > loss_text_ce: 0.039257243275642395 (0.04034614884617466) | > loss_mel_ce: 3.756260395050049 (nan) | > loss: 0.015061578713357449 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13648420940268657) | > loader_time: 0.0046 (0.004727025428313921)  --> TIME: 2025-02-24 13:57:16 -- STEP: 15308/36938 -- GLOBAL_STEP: 347750 | > loss_text_ce: 0.04312072694301605 (0.040346965318529195) | > loss_mel_ce: 2.5245397090911865 (nan) | > loss: 0.010189129039645195 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13648832384774348) | > loader_time: 0.0045 (0.004727063585206087)  --> TIME: 2025-02-24 13:57:26 -- STEP: 15358/36938 -- GLOBAL_STEP: 347800 | > loss_text_ce: 0.03430571034550667 (0.04034077763482972) | > loss_mel_ce: 3.067328453063965 (nan) | > loss: 0.012308073230087757 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.1365062079092443) | > loader_time: 0.0046 (0.004726948861048706)  --> TIME: 2025-02-24 13:57:35 -- STEP: 15408/36938 -- GLOBAL_STEP: 347850 | > loss_text_ce: 0.03438033536076546 (0.04033520129026447) | > loss_mel_ce: 2.4443767070770264 (nan) | > loss: 0.009836338460445404 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13649649913437265) | > loader_time: 0.0045 (0.0047271692214353525)  --> TIME: 2025-02-24 13:57:44 -- STEP: 15458/36938 -- GLOBAL_STEP: 347900 | > loss_text_ce: 0.03464900329709053 (0.04033205363804763) | > loss_mel_ce: 3.1668593883514404 (nan) | > loss: 0.012704398483037949 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13648992569422805) | > loader_time: 0.0046 (0.004727021197110352)  --> TIME: 2025-02-24 13:57:53 -- STEP: 15508/36938 -- GLOBAL_STEP: 347950 | > loss_text_ce: 0.04957495257258415 (0.04033474107311362) | > loss_mel_ce: 3.2581841945648193 (nan) | > loss: 0.013126028701663017 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13648095649328612) | > loader_time: 0.0052 (0.004727438795588055)  --> TIME: 2025-02-24 13:58:02 -- STEP: 15558/36938 -- GLOBAL_STEP: 348000 | > loss_text_ce: 0.0372459813952446 (0.04033464765516673) | > loss_mel_ce: 2.9286437034606934 (nan) | > loss: 0.011769404634833336 (nan) | > current_lr: 5e-06 | > step_time: 0.1688 (0.1364826711656379) | > loader_time: 0.0046 (0.004727183477571134)  --> TIME: 2025-02-24 13:58:12 -- STEP: 15608/36938 -- GLOBAL_STEP: 348050 | > loss_text_ce: 0.027370980009436607 (0.0403268912071143) | > loss_mel_ce: 3.1473262310028076 (nan) | > loss: 0.012598005123436451 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.136484265419106) | > loader_time: 0.0045 (0.004727244209106707)  --> TIME: 2025-02-24 13:58:21 -- STEP: 15658/36938 -- GLOBAL_STEP: 348100 | > loss_text_ce: 0.039635494351387024 (0.04032661973163661) | > loss_mel_ce: 2.671246290206909 (nan) | > loss: 0.010757467709481716 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13648403621234564) | > loader_time: 0.0045 (0.004727520953651135)  --> TIME: 2025-02-24 13:58:30 -- STEP: 15708/36938 -- GLOBAL_STEP: 348150 | > loss_text_ce: 0.031711358577013016 (0.04032518439686681) | > loss_mel_ce: 1.6301474571228027 (nan) | > loss: 0.006594677921384573 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13648970124980764) | > loader_time: 0.0045 (0.004727252801016481)  --> TIME: 2025-02-24 13:58:40 -- STEP: 15758/36938 -- GLOBAL_STEP: 348200 | > loss_text_ce: 0.041966043412685394 (0.040322913906994134) | > loss_mel_ce: 3.945716142654419 (nan) | > loss: 0.01582413539290428 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13650313124527763) | > loader_time: 0.0045 (0.004727151630220992)  --> TIME: 2025-02-24 13:58:49 -- STEP: 15808/36938 -- GLOBAL_STEP: 348250 | > loss_text_ce: 0.04114215821027756 (0.040328665093345865) | > loss_mel_ce: 2.9676318168640137 (nan) | > loss: 0.01193958055227995 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13650693074773673) | > loader_time: 0.0045 (0.0047272569858110595)  --> TIME: 2025-02-24 13:58:59 -- STEP: 15858/36938 -- GLOBAL_STEP: 348300 | > loss_text_ce: 0.037590544670820236 (0.040324164122878245) | > loss_mel_ce: 1.1362121105194092 (nan) | > loss: 0.004657946992665529 (nan) | > current_lr: 5e-06 | > step_time: 0.1444 (0.13651296080308922) | > loader_time: 0.0048 (0.004726990818721741)  --> TIME: 2025-02-24 13:59:08 -- STEP: 15908/36938 -- GLOBAL_STEP: 348350 | > loss_text_ce: 0.03630170598626137 (0.040325762868582825) | > loss_mel_ce: 2.2513620853424072 (nan) | > loss: 0.009078031405806541 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13650833265488244) | > loader_time: 0.0045 (0.004727054787305575)  --> TIME: 2025-02-24 13:59:17 -- STEP: 15958/36938 -- GLOBAL_STEP: 348400 | > loss_text_ce: 0.03073124587535858 (0.04032426652332686) | > loss_mel_ce: 2.590602159500122 (nan) | > loss: 0.010402116924524307 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13651748573978761) | > loader_time: 0.0044 (0.004727082229195423)  --> TIME: 2025-02-24 13:59:27 -- STEP: 16008/36938 -- GLOBAL_STEP: 348450 | > loss_text_ce: 0.03900782763957977 (0.04032026187567717) | > loss_mel_ce: 3.4201254844665527 (nan) | > loss: 0.013726720586419106 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13651683055300043) | > loader_time: 0.0045 (0.004726714652398391)  --> TIME: 2025-02-24 13:59:36 -- STEP: 16058/36938 -- GLOBAL_STEP: 348500 | > loss_text_ce: 0.012698063626885414 (0.04031605708355645) | > loss_mel_ce: 1.8169409036636353 (nan) | > loss: 0.007260472513735294 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13652940281361858) | > loader_time: 0.0045 (0.00472637669861069)  --> TIME: 2025-02-24 13:59:45 -- STEP: 16108/36938 -- GLOBAL_STEP: 348550 | > loss_text_ce: 0.015102271921932697 (0.04031482086197075) | > loss_mel_ce: 2.921278715133667 (nan) | > loss: 0.011652306653559208 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1365253061818401) | > loader_time: 0.0053 (0.00472624433049497)  --> TIME: 2025-02-24 13:59:55 -- STEP: 16158/36938 -- GLOBAL_STEP: 348600 | > loss_text_ce: 0.03797810524702072 (0.04031456934041686) | > loss_mel_ce: 2.4979488849639893 (nan) | > loss: 0.010063203051686287 (nan) | > current_lr: 5e-06 | > step_time: 0.1532 (0.13655454951032633) | > loader_time: 0.0046 (0.004726722963760185)  --> TIME: 2025-02-24 14:00:05 -- STEP: 16208/36938 -- GLOBAL_STEP: 348650 | > loss_text_ce: 0.03458436578512192 (0.0403116138583934) | > loss_mel_ce: 3.5542917251586914 (nan) | > loss: 0.014241572469472885 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.13656826226108748) | > loader_time: 0.0047 (0.0047269091230742405)  --> TIME: 2025-02-24 14:00:14 -- STEP: 16258/36938 -- GLOBAL_STEP: 348700 | > loss_text_ce: 0.03390217944979668 (0.04030626640136226) | > loss_mel_ce: 2.826915740966797 (nan) | > loss: 0.01135245244950056 (nan) | > current_lr: 5e-06 | > step_time: 0.1462 (0.13657911436219597) | > loader_time: 0.0051 (0.004727276624796153)  --> TIME: 2025-02-24 14:00:23 -- STEP: 16308/36938 -- GLOBAL_STEP: 348750 | > loss_text_ce: 0.04931829497218132 (0.040307944856729394) | > loss_mel_ce: 3.162870407104492 (nan) | > loss: 0.01274678111076355 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13657121696921282) | > loader_time: 0.0046 (0.004727175664398789)  --> TIME: 2025-02-24 14:00:33 -- STEP: 16358/36938 -- GLOBAL_STEP: 348800 | > loss_text_ce: 0.042718078941106796 (0.04031111468230896) | > loss_mel_ce: 0.0211181640625 (nan) | > loss: 0.00025331845972687006 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13656726708722905) | > loader_time: 0.0047 (0.004726995726875813)  --> TIME: 2025-02-24 14:00:42 -- STEP: 16408/36938 -- GLOBAL_STEP: 348850 | > loss_text_ce: 0.04363296926021576 (0.040310489423940866) | > loss_mel_ce: 2.6835107803344727 (nan) | > loss: 0.010821999981999397 (nan) | > current_lr: 5e-06 | > step_time: 0.1896 (0.13655843419717145) | > loader_time: 0.0049 (0.00472730174406394)  --> TIME: 2025-02-24 14:00:51 -- STEP: 16458/36938 -- GLOBAL_STEP: 348900 | > loss_text_ce: 0.0424814410507679 (0.040306120487091895) | > loss_mel_ce: 1.2101783752441406 (nan) | > loss: 0.004970872309058905 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1365569691015399) | > loader_time: 0.0045 (0.004727400237234184)  --> TIME: 2025-02-24 14:01:00 -- STEP: 16508/36938 -- GLOBAL_STEP: 348950 | > loss_text_ce: 0.029890259727835655 (0.04030338105790074) | > loss_mel_ce: 2.5641815662384033 (nan) | > loss: 0.010293936356902122 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.1365437867683621) | > loader_time: 0.0045 (0.004727336896485473)  --> TIME: 2025-02-24 14:01:10 -- STEP: 16558/36938 -- GLOBAL_STEP: 349000 | > loss_text_ce: 0.04793693125247955 (0.0403038090342857) | > loss_mel_ce: 2.4389712810516357 (nan) | > loss: 0.009868684224784374 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13655247857049685) | > loader_time: 0.0045 (0.004727615237682366)  --> TIME: 2025-02-24 14:01:19 -- STEP: 16608/36938 -- GLOBAL_STEP: 349050 | > loss_text_ce: 0.037176959216594696 (0.04029645128316932) | > loss_mel_ce: 2.747880458831787 (nan) | > loss: 0.011051815003156662 (nan) | > current_lr: 5e-06 | > step_time: 0.1265 (0.13654119860172725) | > loader_time: 0.0046 (0.004727653125462479)  --> TIME: 2025-02-24 14:01:28 -- STEP: 16658/36938 -- GLOBAL_STEP: 349100 | > loss_text_ce: 0.03718097135424614 (0.04029272248847051) | > loss_mel_ce: 3.0408897399902344 (nan) | > loss: 0.012214566580951214 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13654713858818424) | > loader_time: 0.0046 (0.004727839393079835)  --> TIME: 2025-02-24 14:01:38 -- STEP: 16708/36938 -- GLOBAL_STEP: 349150 | > loss_text_ce: 0.03506636992096901 (0.040292180344968107) | > loss_mel_ce: 2.8737072944641113 (nan) | > loss: 0.01154275331646204 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13656435382117874) | > loader_time: 0.0045 (0.0047278050489336365)  --> TIME: 2025-02-24 14:01:47 -- STEP: 16758/36938 -- GLOBAL_STEP: 349200 | > loss_text_ce: 0.04734383895993233 (0.04029087220661068) | > loss_mel_ce: 3.5084362030029297 (nan) | > loss: 0.014110238291323185 (nan) | > current_lr: 5e-06 | > step_time: 0.156 (0.1365661158853646) | > loader_time: 0.0053 (0.004728343026110749)  --> TIME: 2025-02-24 14:01:57 -- STEP: 16808/36938 -- GLOBAL_STEP: 349250 | > loss_text_ce: 0.04093417525291443 (0.04029036020567072) | > loss_mel_ce: 3.532944917678833 (nan) | > loss: 0.014182060025632381 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13657072812020904) | > loader_time: 0.0045 (0.00472816197943651)  --> TIME: 2025-02-24 14:02:06 -- STEP: 16858/36938 -- GLOBAL_STEP: 349300 | > loss_text_ce: 0.04095296189188957 (0.04028766782664189) | > loss_mel_ce: 2.9502813816070557 (nan) | > loss: 0.011869978159666061 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13657540053938855) | > loader_time: 0.0048 (0.004727830764081677)  --> TIME: 2025-02-24 14:02:15 -- STEP: 16908/36938 -- GLOBAL_STEP: 349350 | > loss_text_ce: 0.03650076687335968 (0.04028889773951581) | > loss_mel_ce: 3.518486261367798 (nan) | > loss: 0.014107092283666134 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13657767585978195) | > loader_time: 0.0045 (0.004727803112514631)  --> TIME: 2025-02-24 14:02:25 -- STEP: 16958/36938 -- GLOBAL_STEP: 349400 | > loss_text_ce: 0.03714102879166603 (0.04028974054265924) | > loss_mel_ce: 3.0671322345733643 (nan) | > loss: 0.012318545952439308 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13657919687684342) | > loader_time: 0.0046 (0.004727920913403837)  --> TIME: 2025-02-24 14:02:34 -- STEP: 17008/36938 -- GLOBAL_STEP: 349450 | > loss_text_ce: 0.04712231457233429 (0.04029032781541435) | > loss_mel_ce: 0.9559659361839294 (nan) | > loss: 0.003980509005486965 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.13658888769788993) | > loader_time: 0.0045 (0.004727929381974311)  --> TIME: 2025-02-24 14:02:43 -- STEP: 17058/36938 -- GLOBAL_STEP: 349500 | > loss_text_ce: 0.04336998239159584 (0.04029168159384161) | > loss_mel_ce: 3.5424039363861084 (nan) | > loss: 0.014229262247681618 (nan) | > current_lr: 5e-06 | > step_time: 0.1614 (0.13658545676674272) | > loader_time: 0.0046 (0.004727678934062162)  --> TIME: 2025-02-24 14:02:52 -- STEP: 17108/36938 -- GLOBAL_STEP: 349550 | > loss_text_ce: 0.04070819169282913 (0.04028889763917843) | > loss_mel_ce: 4.132813930511475 (nan) | > loss: 0.01656159572303295 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13657857797846532) | > loader_time: 0.0045 (0.004727298700003955)  --> TIME: 2025-02-24 14:03:02 -- STEP: 17158/36938 -- GLOBAL_STEP: 349600 | > loss_text_ce: 0.0375857874751091 (0.04028344824065584) | > loss_mel_ce: 3.866162061691284 (nan) | > loss: 0.015491063706576824 (nan) | > current_lr: 5e-06 | > step_time: 0.2065 (0.13659429166655743) | > loader_time: 0.0049 (0.004727574728185261)  --> TIME: 2025-02-24 14:03:11 -- STEP: 17208/36938 -- GLOBAL_STEP: 349650 | > loss_text_ce: 0.03874969854950905 (0.04028477693448105) | > loss_mel_ce: 3.249713182449341 (nan) | > loss: 0.013049456290900707 (nan) | > current_lr: 5e-06 | > step_time: 0.1632 (0.13658846451315068) | > loader_time: 0.0047 (0.004728051062685436)  --> TIME: 2025-02-24 14:03:21 -- STEP: 17258/36938 -- GLOBAL_STEP: 349700 | > loss_text_ce: 0.039035070687532425 (0.040281847898365634) | > loss_mel_ce: 3.2141640186309814 (nan) | > loss: 0.012909521348774433 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13658522896452208) | > loader_time: 0.0048 (0.004728087283103609)  --> TIME: 2025-02-24 14:03:30 -- STEP: 17308/36938 -- GLOBAL_STEP: 349750 | > loss_text_ce: 0.039978377521038055 (0.040281896977482044) | > loss_mel_ce: 3.0618128776550293 (nan) | > loss: 0.0123086953535676 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.13658109294202794) | > loader_time: 0.0047 (0.004728400833975322)  --> TIME: 2025-02-24 14:03:39 -- STEP: 17358/36938 -- GLOBAL_STEP: 349800 | > loss_text_ce: 0.04356355965137482 (0.04028360131481585) | > loss_mel_ce: 2.948615789413452 (nan) | > loss: 0.011873728595674038 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.1365805002302186) | > loader_time: 0.0047 (0.004728628971263778)  --> TIME: 2025-02-24 14:03:48 -- STEP: 17408/36938 -- GLOBAL_STEP: 349850 | > loss_text_ce: 0.03813200443983078 (0.04028425886677086) | > loss_mel_ce: 3.67822265625 (nan) | > loss: 0.01474743988364935 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13658302844337653) | > loader_time: 0.0045 (0.004728709279066483)  --> TIME: 2025-02-24 14:03:58 -- STEP: 17458/36938 -- GLOBAL_STEP: 349900 | > loss_text_ce: 0.031578704714775085 (0.04028838450917679) | > loss_mel_ce: 3.0706405639648438 (nan) | > loss: 0.012310395017266273 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13657942842762527) | > loader_time: 0.0052 (0.004728507553128537)  --> TIME: 2025-02-24 14:04:07 -- STEP: 17508/36938 -- GLOBAL_STEP: 349950 | > loss_text_ce: 0.039894044399261475 (0.04028699295912331) | > loss_mel_ce: 3.0361428260803223 (nan) | > loss: 0.01220649667084217 (nan) | > current_lr: 5e-06 | > step_time: 0.1867 (0.13658356161348484) | > loader_time: 0.0048 (0.004728237243174726)  --> TIME: 2025-02-24 14:04:16 -- STEP: 17558/36938 -- GLOBAL_STEP: 350000 | > loss_text_ce: 0.026884347200393677 (0.04028560290946293) | > loss_mel_ce: 2.5961050987243652 (nan) | > loss: 0.010408688336610794 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.1365756916013986) | > loader_time: 0.0045 (0.004727948810478737) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_350000.pth  --> TIME: 2025-02-24 14:04:32 -- STEP: 17608/36938 -- GLOBAL_STEP: 350050 | > loss_text_ce: 0.03491609916090965 (0.040290431246654074) | > loss_mel_ce: 2.6785848140716553 (nan) | > loss: 0.010767861269414425 (nan) | > current_lr: 5e-06 | > step_time: 0.1757 (0.13657244706847121) | > loader_time: 0.0048 (0.0047279282463295385)  --> TIME: 2025-02-24 14:04:41 -- STEP: 17658/36938 -- GLOBAL_STEP: 350100 | > loss_text_ce: 0.04455047473311424 (0.04029227690727981) | > loss_mel_ce: 3.240734338760376 (nan) | > loss: 0.013036845251917839 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13656620690906032) | > loader_time: 0.0046 (0.0047277765185474475)  --> TIME: 2025-02-24 14:04:50 -- STEP: 17708/36938 -- GLOBAL_STEP: 350150 | > loss_text_ce: 0.038020722568035126 (0.040289987351237164) | > loss_mel_ce: 2.9482319355010986 (nan) | > loss: 0.011850208975374699 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13655249449639648) | > loader_time: 0.0045 (0.004727703024577884)  --> TIME: 2025-02-24 14:04:59 -- STEP: 17758/36938 -- GLOBAL_STEP: 350200 | > loss_text_ce: 0.04734120890498161 (0.04029219425452776) | > loss_mel_ce: 3.444465398788452 (nan) | > loss: 0.013856375589966774 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13654766795414758) | > loader_time: 0.0045 (0.004727762042704095)  --> TIME: 2025-02-24 14:05:08 -- STEP: 17808/36938 -- GLOBAL_STEP: 350250 | > loss_text_ce: 0.046172432601451874 (0.04029376818781752) | > loss_mel_ce: 3.304572105407715 (nan) | > loss: 0.01329660601913929 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1365492588640437) | > loader_time: 0.0046 (0.004728069711352709)  --> TIME: 2025-02-24 14:05:18 -- STEP: 17858/36938 -- GLOBAL_STEP: 350300 | > loss_text_ce: 0.04697515442967415 (0.04029274213143754) | > loss_mel_ce: 3.3558924198150635 (nan) | > loss: 0.013503443449735641 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13655270296804503) | > loader_time: 0.0045 (0.004727966201466921)  --> TIME: 2025-02-24 14:05:27 -- STEP: 17908/36938 -- GLOBAL_STEP: 350350 | > loss_text_ce: 0.04243449494242668 (0.04029258993677493) | > loss_mel_ce: 2.9310688972473145 (nan) | > loss: 0.011799616739153862 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13654479217337584) | > loader_time: 0.0053 (0.004728179771984838)  --> TIME: 2025-02-24 14:05:36 -- STEP: 17958/36938 -- GLOBAL_STEP: 350400 | > loss_text_ce: 0.030213825404644012 (0.040294650781110206) | > loss_mel_ce: 1.8366191387176514 (nan) | > loss: 0.007408067584037781 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13655954926645109) | > loader_time: 0.0046 (0.004728520204633486)  --> TIME: 2025-02-24 14:05:46 -- STEP: 18008/36938 -- GLOBAL_STEP: 350450 | > loss_text_ce: 0.048121966421604156 (0.040291978151594215) | > loss_mel_ce: 3.1522634029388428 (nan) | > loss: 0.012699942104518414 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13655985431743153) | > loader_time: 0.0049 (0.00472838107557732)  --> TIME: 2025-02-24 14:05:55 -- STEP: 18058/36938 -- GLOBAL_STEP: 350500 | > loss_text_ce: 0.03210505470633507 (0.040288574181011096) | > loss_mel_ce: 2.9083962440490723 (nan) | > loss: 0.011668656021356583 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13656125406395575) | > loader_time: 0.0046 (0.004728246294972837)  --> TIME: 2025-02-24 14:06:04 -- STEP: 18108/36938 -- GLOBAL_STEP: 350550 | > loss_text_ce: 0.05107253044843674 (0.04028898304188842) | > loss_mel_ce: 2.842877149581909 (nan) | > loss: 0.011483928188681602 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1365644050560855) | > loader_time: 0.0045 (0.0047286775482393295)  --> TIME: 2025-02-24 14:06:14 -- STEP: 18158/36938 -- GLOBAL_STEP: 350600 | > loss_text_ce: 0.04946710541844368 (0.04028923997885498) | > loss_mel_ce: 3.6837270259857178 (nan) | > loss: 0.014814263209700584 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1365672181600357) | > loader_time: 0.0045 (0.004728530692507964)  --> TIME: 2025-02-24 14:06:23 -- STEP: 18208/36938 -- GLOBAL_STEP: 350650 | > loss_text_ce: 0.039534859359264374 (0.040288085656024576) | > loss_mel_ce: 3.587883710861206 (nan) | > loss: 0.014394518919289112 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.1365698569557368) | > loader_time: 0.0047 (0.00472852535323223)  --> TIME: 2025-02-24 14:06:33 -- STEP: 18258/36938 -- GLOBAL_STEP: 350700 | > loss_text_ce: 0.03616180643439293 (0.04028907696056332) | > loss_mel_ce: 3.5326483249664307 (nan) | > loss: 0.014161946251988411 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13657579626751318) | > loader_time: 0.0053 (0.004728937621758558)  --> TIME: 2025-02-24 14:06:42 -- STEP: 18308/36938 -- GLOBAL_STEP: 350750 | > loss_text_ce: 0.03458015248179436 (0.040286151941736306) | > loss_mel_ce: 3.1751708984375 (nan) | > loss: 0.012737108394503593 (nan) | > current_lr: 5e-06 | > step_time: 0.1591 (0.1365796326939809) | > loader_time: 0.0053 (0.0047291969794593346)  --> TIME: 2025-02-24 14:06:51 -- STEP: 18358/36938 -- GLOBAL_STEP: 350800 | > loss_text_ce: 0.047597892582416534 (0.040280745332460594) | > loss_mel_ce: 2.5554726123809814 (nan) | > loss: 0.01032964512705803 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.1365762781585167) | > loader_time: 0.0045 (0.00472891247601064)  --> TIME: 2025-02-24 14:07:01 -- STEP: 18408/36938 -- GLOBAL_STEP: 350850 | > loss_text_ce: 0.0371779128909111 (0.040283317702568634) | > loss_mel_ce: 3.783461332321167 (nan) | > loss: 0.01516126748174429 (nan) | > current_lr: 5e-06 | > step_time: 0.1773 (0.13658456453184922) | > loader_time: 0.0048 (0.004728984167035973)  --> TIME: 2025-02-24 14:07:10 -- STEP: 18458/36938 -- GLOBAL_STEP: 350900 | > loss_text_ce: 0.037668392062187195 (0.040281556112965675) | > loss_mel_ce: 3.1004366874694824 (nan) | > loss: 0.012452798895537853 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13659174862862758) | > loader_time: 0.0048 (0.004728826299521213)  --> TIME: 2025-02-24 14:07:19 -- STEP: 18508/36938 -- GLOBAL_STEP: 350950 | > loss_text_ce: 0.04241613671183586 (0.04027970558775525) | > loss_mel_ce: 3.074037551879883 (nan) | > loss: 0.012366880662739277 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13658963044312686) | > loader_time: 0.0052 (0.004728685722305701)  --> TIME: 2025-02-24 14:07:29 -- STEP: 18558/36938 -- GLOBAL_STEP: 351000 | > loss_text_ce: 0.043116629123687744 (0.040278360431989665) | > loss_mel_ce: 1.847808837890625 (nan) | > loss: 0.007503672502934933 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13658740301046873) | > loader_time: 0.0046 (0.00472874883716892)  --> TIME: 2025-02-24 14:07:38 -- STEP: 18608/36938 -- GLOBAL_STEP: 351050 | > loss_text_ce: 0.034515514969825745 (0.04027615382681632) | > loss_mel_ce: 3.1937246322631836 (nan) | > loss: 0.0128104779869318 (nan) | > current_lr: 5e-06 | > step_time: 0.17 (0.13658353445454063) | > loader_time: 0.0049 (0.004728422747638218)  --> TIME: 2025-02-24 14:07:47 -- STEP: 18658/36938 -- GLOBAL_STEP: 351100 | > loss_text_ce: 0.045440033078193665 (0.0402739147635401) | > loss_mel_ce: 3.1378798484802246 (nan) | > loss: 0.012632221914827824 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13657593366762127) | > loader_time: 0.0053 (0.004728311242136168)  --> TIME: 2025-02-24 14:07:56 -- STEP: 18708/36938 -- GLOBAL_STEP: 351150 | > loss_text_ce: 0.03641488403081894 (0.04027950743451063) | > loss_mel_ce: 3.272965431213379 (nan) | > loss: 0.013132462278008461 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13657484648190443) | > loader_time: 0.0047 (0.004728348560508595)  --> TIME: 2025-02-24 14:08:06 -- STEP: 18758/36938 -- GLOBAL_STEP: 351200 | > loss_text_ce: 0.03939204290509224 (0.040278300987434505) | > loss_mel_ce: 3.3674607276916504 (nan) | > loss: 0.013519257307052612 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.13658671415186394) | > loader_time: 0.0049 (0.004728513366950558)  --> TIME: 2025-02-24 14:08:15 -- STEP: 18808/36938 -- GLOBAL_STEP: 351250 | > loss_text_ce: 0.03607086092233658 (0.04027470680859273) | > loss_mel_ce: 3.483996868133545 (nan) | > loss: 0.01396852359175682 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13658566962450605) | > loader_time: 0.0045 (0.004728462368009023)  --> TIME: 2025-02-24 14:08:25 -- STEP: 18858/36938 -- GLOBAL_STEP: 351300 | > loss_text_ce: 0.043870892375707626 (0.04027309182366394) | > loss_mel_ce: 3.50565505027771 (nan) | > loss: 0.014085421338677406 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13659371004995657) | > loader_time: 0.0047 (0.004728913686474973)  --> TIME: 2025-02-24 14:08:34 -- STEP: 18908/36938 -- GLOBAL_STEP: 351350 | > loss_text_ce: 0.023595599457621574 (0.040268837860154054) | > loss_mel_ce: 0.24846163392066956 (nan) | > loss: 0.001079592271707952 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13659532742997602) | > loader_time: 0.0046 (0.004728863878332821)  --> TIME: 2025-02-24 14:08:43 -- STEP: 18958/36938 -- GLOBAL_STEP: 351400 | > loss_text_ce: 0.04628967121243477 (0.04026887524792067) | > loss_mel_ce: 2.81083345413208 (nan) | > loss: 0.01133779063820839 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.1365866142934554) | > loader_time: 0.0053 (0.004729255768226295)  --> TIME: 2025-02-24 14:08:52 -- STEP: 19008/36938 -- GLOBAL_STEP: 351450 | > loss_text_ce: 0.02273290976881981 (0.0402700722601496) | > loss_mel_ce: 0.8753593564033508 (nan) | > loss: 0.003563858335837722 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13657266578891045) | > loader_time: 0.0046 (0.0047294291783663385)  --> TIME: 2025-02-24 14:09:02 -- STEP: 19058/36938 -- GLOBAL_STEP: 351500 | > loss_text_ce: 0.03519608825445175 (0.04027020956127388) | > loss_mel_ce: 2.6172449588775635 (nan) | > loss: 0.01052556000649929 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13658145939219157) | > loader_time: 0.0046 (0.004729530433252844)  --> TIME: 2025-02-24 14:09:11 -- STEP: 19108/36938 -- GLOBAL_STEP: 351550 | > loss_text_ce: 0.04271223023533821 (0.04027155103799065) | > loss_mel_ce: 2.7015647888183594 (nan) | > loss: 0.010889988392591476 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.1365930884858167) | > loader_time: 0.0047 (0.004729406227745061)  --> TIME: 2025-02-24 14:09:21 -- STEP: 19158/36938 -- GLOBAL_STEP: 351600 | > loss_text_ce: 0.038405805826187134 (0.04027069386706168) | > loss_mel_ce: 2.7634170055389404 (nan) | > loss: 0.011118345893919468 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.136600357628227) | > loader_time: 0.0047 (0.0047292317337576046)  --> TIME: 2025-02-24 14:09:30 -- STEP: 19208/36938 -- GLOBAL_STEP: 351650 | > loss_text_ce: 0.043967366218566895 (0.04026672202296271) | > loss_mel_ce: 3.5751469135284424 (nan) | > loss: 0.014361565932631493 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.13659602213730063) | > loader_time: 0.0047 (0.00472905452377544)  --> TIME: 2025-02-24 14:09:39 -- STEP: 19258/36938 -- GLOBAL_STEP: 351700 | > loss_text_ce: 0.03882266581058502 (0.04026321469069999) | > loss_mel_ce: 2.783195734024048 (nan) | > loss: 0.01119848620146513 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13659028269591159) | > loader_time: 0.0045 (0.004728804905846145)  --> TIME: 2025-02-24 14:09:48 -- STEP: 19308/36938 -- GLOBAL_STEP: 351750 | > loss_text_ce: 0.040021661669015884 (0.040265264816509695) | > loss_mel_ce: 2.703514337539673 (nan) | > loss: 0.010887048207223415 (nan) | > current_lr: 5e-06 | > step_time: 0.1754 (0.13658994314017078) | > loader_time: 0.0049 (0.0047288049025542385)  --> TIME: 2025-02-24 14:09:58 -- STEP: 19358/36938 -- GLOBAL_STEP: 351800 | > loss_text_ce: 0.04085686430335045 (0.04026713726867153) | > loss_mel_ce: 3.3834733963012695 (nan) | > loss: 0.013588612899184227 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13659737074174386) | > loader_time: 0.0046 (0.004729097571078864)  --> TIME: 2025-02-24 14:10:07 -- STEP: 19408/36938 -- GLOBAL_STEP: 351850 | > loss_text_ce: 0.04535813629627228 (0.04027032191302365) | > loss_mel_ce: 2.5627245903015137 (nan) | > loss: 0.010349535383284092 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13660023777276542) | > loader_time: 0.0046 (0.004729268207888109)  --> TIME: 2025-02-24 14:10:16 -- STEP: 19458/36938 -- GLOBAL_STEP: 351900 | > loss_text_ce: 0.049520161002874374 (0.040274629089378296) | > loss_mel_ce: 3.468005418777466 (nan) | > loss: 0.01395843643695116 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1365911696427455) | > loader_time: 0.0046 (0.004729025164688572)  --> TIME: 2025-02-24 14:10:26 -- STEP: 19508/36938 -- GLOBAL_STEP: 351950 | > loss_text_ce: 0.039398156106472015 (0.040270779682636565) | > loss_mel_ce: 3.4459166526794434 (nan) | > loss: 0.01383061520755291 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1365898796860401) | > loader_time: 0.0049 (0.0047294490890275945)  --> TIME: 2025-02-24 14:10:35 -- STEP: 19558/36938 -- GLOBAL_STEP: 352000 | > loss_text_ce: 0.04358326643705368 (0.040270183306882285) | > loss_mel_ce: 2.9499104022979736 (nan) | > loss: 0.01187894307076931 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.136597246440572) | > loader_time: 0.0045 (0.004729469320576334)  --> TIME: 2025-02-24 14:10:44 -- STEP: 19608/36938 -- GLOBAL_STEP: 352050 | > loss_text_ce: 0.021030353382229805 (0.04026770293525482) | > loss_mel_ce: 2.5256268978118896 (nan) | > loss: 0.010105784051120281 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13660027171241565) | > loader_time: 0.0044 (0.004729685358785196)  --> TIME: 2025-02-24 14:10:54 -- STEP: 19658/36938 -- GLOBAL_STEP: 352100 | > loss_text_ce: 0.04207684472203255 (0.04027003054204348) | > loss_mel_ce: 3.611253023147583 (nan) | > loss: 0.014497341588139534 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13660080512453637) | > loader_time: 0.0054 (0.0047301486739429075)  --> TIME: 2025-02-24 14:11:03 -- STEP: 19708/36938 -- GLOBAL_STEP: 352150 | > loss_text_ce: 0.029683109372854233 (0.0402687649545114) | > loss_mel_ce: 1.9107812643051147 (nan) | > loss: 0.007700256071984768 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.1365843470788082) | > loader_time: 0.0045 (0.004730393708974065)  --> TIME: 2025-02-24 14:11:12 -- STEP: 19758/36938 -- GLOBAL_STEP: 352200 | > loss_text_ce: 0.04065466672182083 (0.04026497588098403) | > loss_mel_ce: 3.171489715576172 (nan) | > loss: 0.012746605090796947 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.1365909573157939) | > loader_time: 0.0046 (0.004730559877205828)  --> TIME: 2025-02-24 14:11:21 -- STEP: 19808/36938 -- GLOBAL_STEP: 352250 | > loss_text_ce: 0.03114231675863266 (0.040266605889470265) | > loss_mel_ce: 3.2084646224975586 (nan) | > loss: 0.012855583801865578 (nan) | > current_lr: 5e-06 | > step_time: 0.1794 (0.13658198558985676) | > loader_time: 0.0048 (0.00473081452925485)  --> TIME: 2025-02-24 14:11:31 -- STEP: 19858/36938 -- GLOBAL_STEP: 352300 | > loss_text_ce: 0.03504357114434242 (0.040262409283319374) | > loss_mel_ce: 1.581884741783142 (nan) | > loss: 0.006416382733732462 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1365818878545152) | > loader_time: 0.0045 (0.004730938988661677)  --> TIME: 2025-02-24 14:11:40 -- STEP: 19908/36938 -- GLOBAL_STEP: 352350 | > loss_text_ce: 0.03644862398505211 (0.04026506017330873) | > loss_mel_ce: 1.4313310384750366 (nan) | > loss: 0.005824522580951452 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13658173916113267) | > loader_time: 0.0045 (0.004730885841292205)  --> TIME: 2025-02-24 14:11:49 -- STEP: 19958/36938 -- GLOBAL_STEP: 352400 | > loss_text_ce: 0.03470664471387863 (0.04026252071574107) | > loss_mel_ce: 3.335602045059204 (nan) | > loss: 0.013374241068959236 (nan) | > current_lr: 5e-06 | > step_time: 0.1766 (0.13658254069404657) | > loader_time: 0.0048 (0.004730968690847907)  --> TIME: 2025-02-24 14:11:58 -- STEP: 20008/36938 -- GLOBAL_STEP: 352450 | > loss_text_ce: 0.046093858778476715 (0.04026386758387253) | > loss_mel_ce: 2.6569228172302246 (nan) | > loss: 0.010726257227361202 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13657659400658137) | > loader_time: 0.0052 (0.004731056452846099)  --> TIME: 2025-02-24 14:12:07 -- STEP: 20058/36938 -- GLOBAL_STEP: 352500 | > loss_text_ce: 0.038779713213443756 (0.04025902641372166) | > loss_mel_ce: 3.2556560039520264 (nan) | > loss: 0.013073158450424671 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13657063082334506) | > loader_time: 0.0047 (0.0047310472711488295)  --> TIME: 2025-02-24 14:12:17 -- STEP: 20108/36938 -- GLOBAL_STEP: 352550 | > loss_text_ce: 0.0478324294090271 (0.040257126212637766) | > loss_mel_ce: 2.8897454738616943 (nan) | > loss: 0.011657056398689747 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.136569425591138) | > loader_time: 0.0047 (0.0047313930951792425)  --> TIME: 2025-02-24 14:12:26 -- STEP: 20158/36938 -- GLOBAL_STEP: 352600 | > loss_text_ce: 0.03039293736219406 (0.04025791522052829) | > loss_mel_ce: 2.240844964981079 (nan) | > loss: 0.009012849070131779 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13657973162509773) | > loader_time: 0.0045 (0.004731759640394378)  --> TIME: 2025-02-24 14:12:36 -- STEP: 20208/36938 -- GLOBAL_STEP: 352650 | > loss_text_ce: 0.04774116724729538 (0.04025973057202369) | > loss_mel_ce: 2.434126615524292 (nan) | > loss: 0.009848682209849358 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13658264733597267) | > loader_time: 0.0044 (0.004731650354365743)  --> TIME: 2025-02-24 14:12:45 -- STEP: 20258/36938 -- GLOBAL_STEP: 352700 | > loss_text_ce: 0.04495716467499733 (0.04025903572222202) | > loss_mel_ce: 3.4271488189697266 (nan) | > loss: 0.013778199441730976 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.136581642087043) | > loader_time: 0.0053 (0.004731670514842179)  --> TIME: 2025-02-24 14:12:54 -- STEP: 20308/36938 -- GLOBAL_STEP: 352750 | > loss_text_ce: 0.03441031277179718 (0.040258207115958795) | > loss_mel_ce: 0.758126974105835 (nan) | > loss: 0.0031449892558157444 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13657846671754909) | > loader_time: 0.0045 (0.00473153157945174)  --> TIME: 2025-02-24 14:13:04 -- STEP: 20358/36938 -- GLOBAL_STEP: 352800 | > loss_text_ce: 0.03900032490491867 (0.04025150112016699) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13658656632692934) | > loader_time: 0.0045 (0.004731457516139657)  --> TIME: 2025-02-24 14:13:13 -- STEP: 20408/36938 -- GLOBAL_STEP: 352850 | > loss_text_ce: 0.03557823970913887 (0.04024982827226193) | > loss_mel_ce: 3.151231050491333 (nan) | > loss: 0.012646069750189781 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.1365757190868087) | > loader_time: 0.0046 (0.00473125341451668)  --> TIME: 2025-02-24 14:13:22 -- STEP: 20458/36938 -- GLOBAL_STEP: 352900 | > loss_text_ce: 0.03076663240790367 (0.0402497341173962) | > loss_mel_ce: 3.3884336948394775 (nan) | > loss: 0.013568256050348282 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.136564588334649) | > loader_time: 0.0052 (0.004731298879813046)  --> TIME: 2025-02-24 14:13:31 -- STEP: 20508/36938 -- GLOBAL_STEP: 352950 | > loss_text_ce: 0.03605615720152855 (0.040245987079328874) | > loss_mel_ce: 3.2427561283111572 (nan) | > loss: 0.013011160306632519 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13657426155169186) | > loader_time: 0.0046 (0.004731546444387864)  --> TIME: 2025-02-24 14:13:41 -- STEP: 20558/36938 -- GLOBAL_STEP: 353000 | > loss_text_ce: 0.03547770529985428 (0.04024358831335353) | > loss_mel_ce: 3.4332423210144043 (nan) | > loss: 0.013764762319624424 (nan) | > current_lr: 5e-06 | > step_time: 0.1919 (0.13658529384925505) | > loader_time: 0.0048 (0.004731358715457809)  --> TIME: 2025-02-24 14:13:50 -- STEP: 20608/36938 -- GLOBAL_STEP: 353050 | > loss_text_ce: 0.03462045267224312 (0.04024511588700679) | > loss_mel_ce: 3.1104378700256348 (nan) | > loss: 0.012480391189455986 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13658704487201953) | > loader_time: 0.0053 (0.00473156296040701)  --> TIME: 2025-02-24 14:13:59 -- STEP: 20658/36938 -- GLOBAL_STEP: 353100 | > loss_text_ce: 0.039140358567237854 (0.04024134749858267) | > loss_mel_ce: 4.055941581726074 (nan) | > loss: 0.016250325366854668 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13657753157562566) | > loader_time: 0.0052 (0.00473156663429496)  --> TIME: 2025-02-24 14:14:08 -- STEP: 20708/36938 -- GLOBAL_STEP: 353150 | > loss_text_ce: 0.04511939361691475 (0.040242480334969986) | > loss_mel_ce: 0.0901285782456398 (nan) | > loss: 0.0005366983241401613 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13656864116522058) | > loader_time: 0.0045 (0.004731691917535856)  --> TIME: 2025-02-24 14:14:17 -- STEP: 20758/36938 -- GLOBAL_STEP: 353200 | > loss_text_ce: 0.05995635315775871 (0.04024626860444969) | > loss_mel_ce: 3.733844518661499 (nan) | > loss: 0.015054766088724136 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13656140724488658) | > loader_time: 0.0048 (0.004731577558432001)  --> TIME: 2025-02-24 14:14:27 -- STEP: 20808/36938 -- GLOBAL_STEP: 353250 | > loss_text_ce: 0.029265237972140312 (0.040243325957977305) | > loss_mel_ce: 1.3053693771362305 (nan) | > loss: 0.005296169780194759 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13655976174841117) | > loader_time: 0.0046 (0.004731776438415716)  --> TIME: 2025-02-24 14:14:36 -- STEP: 20858/36938 -- GLOBAL_STEP: 353300 | > loss_text_ce: 0.05107060819864273 (0.040238645939210615) | > loss_mel_ce: 3.2056784629821777 (nan) | > loss: 0.012923608534038067 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13655484898891201) | > loader_time: 0.0046 (0.004731667877352356)  --> TIME: 2025-02-24 14:14:45 -- STEP: 20908/36938 -- GLOBAL_STEP: 353350 | > loss_text_ce: 0.040819354355335236 (0.04023890693312809) | > loss_mel_ce: 3.090975761413574 (nan) | > loss: 0.012427759356796741 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13656012193637124) | > loader_time: 0.0051 (0.004732123052090928)  --> TIME: 2025-02-24 14:14:55 -- STEP: 20958/36938 -- GLOBAL_STEP: 353400 | > loss_text_ce: 0.036137618124485016 (0.04023834099871892) | > loss_mel_ce: 3.2854816913604736 (nan) | > loss: 0.013181029818952084 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13655341546355076) | > loader_time: 0.0051 (0.004732186733533076)  --> TIME: 2025-02-24 14:15:04 -- STEP: 21008/36938 -- GLOBAL_STEP: 353450 | > loss_text_ce: 0.052582912147045135 (0.04023554178942692) | > loss_mel_ce: 2.132473945617676 (nan) | > loss: 0.008670860901474953 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13654512001282365) | > loader_time: 0.0046 (0.0047319147846486345)  --> TIME: 2025-02-24 14:15:13 -- STEP: 21058/36938 -- GLOBAL_STEP: 353500 | > loss_text_ce: 0.034963615238666534 (0.040233947093383585) | > loss_mel_ce: 3.414165496826172 (nan) | > loss: 0.013687021099030972 (nan) | > current_lr: 5e-06 | > step_time: 0.191 (0.13656453226284931) | > loader_time: 0.0048 (0.004731795638134503)  --> TIME: 2025-02-24 14:15:23 -- STEP: 21108/36938 -- GLOBAL_STEP: 353550 | > loss_text_ce: 0.03536882624030113 (0.04023071363807808) | > loss_mel_ce: 2.713076591491699 (nan) | > loss: 0.010906530544161797 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13656290849998676) | > loader_time: 0.0045 (0.0047315873723828)  --> TIME: 2025-02-24 14:15:32 -- STEP: 21158/36938 -- GLOBAL_STEP: 353600 | > loss_text_ce: 0.0328458771109581 (0.04022793773258356) | > loss_mel_ce: 3.117034435272217 (nan) | > loss: 0.012499526143074036 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1365603130113222) | > loader_time: 0.0052 (0.004731780110871322)  --> TIME: 2025-02-24 14:15:41 -- STEP: 21208/36938 -- GLOBAL_STEP: 353650 | > loss_text_ce: 0.04166484624147415 (0.04023232377649467) | > loss_mel_ce: 2.8682632446289062 (nan) | > loss: 0.011547334492206573 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13655617890471278) | > loader_time: 0.0045 (0.004731883905105706)  --> TIME: 2025-02-24 14:15:50 -- STEP: 21258/36938 -- GLOBAL_STEP: 353700 | > loss_text_ce: 0.03668154031038284 (0.04022841877875751) | > loss_mel_ce: 3.591749906539917 (nan) | > loss: 0.014398538507521152 (nan) | > current_lr: 5e-06 | > step_time: 0.186 (0.1365529362037628) | > loader_time: 0.0048 (0.004732077260133586)  --> TIME: 2025-02-24 14:15:59 -- STEP: 21308/36938 -- GLOBAL_STEP: 353750 | > loss_text_ce: 0.03612428531050682 (0.04022605501589821) | > loss_mel_ce: 3.2719321250915527 (nan) | > loss: 0.013127208687365055 (nan) | > current_lr: 5e-06 | > step_time: 0.1819 (0.13654424875879512) | > loader_time: 0.0047 (0.0047319737656917145)  --> TIME: 2025-02-24 14:16:09 -- STEP: 21358/36938 -- GLOBAL_STEP: 353800 | > loss_text_ce: 0.04000292718410492 (0.040221951291176575) | > loss_mel_ce: 3.5455124378204346 (nan) | > loss: 0.014228235930204391 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13654346096123793) | > loader_time: 0.0047 (0.00473195283709198)  --> TIME: 2025-02-24 14:16:18 -- STEP: 21408/36938 -- GLOBAL_STEP: 353850 | > loss_text_ce: 0.044629644602537155 (0.040220660495610905) | > loss_mel_ce: 3.0885348320007324 (nan) | > loss: 0.012433192692697048 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.1365478417776264) | > loader_time: 0.0045 (0.004732200973326137)  --> TIME: 2025-02-24 14:16:28 -- STEP: 21458/36938 -- GLOBAL_STEP: 353900 | > loss_text_ce: 0.04195338115096092 (0.040222313534841805) | > loss_mel_ce: 3.524007558822632 (nan) | > loss: 0.014150639064610004 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13655620179490505) | > loader_time: 0.0047 (0.004732152635469632)  --> TIME: 2025-02-24 14:16:37 -- STEP: 21508/36938 -- GLOBAL_STEP: 353950 | > loss_text_ce: 0.02962522767484188 (0.04022108621177216) | > loss_mel_ce: 1.5606166124343872 (nan) | > loss: 0.006310483440756798 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13655511809033838) | > loader_time: 0.0046 (0.004732059649825209)  --> TIME: 2025-02-24 14:16:46 -- STEP: 21558/36938 -- GLOBAL_STEP: 354000 | > loss_text_ce: 0.040146250277757645 (0.040224251540998054) | > loss_mel_ce: 3.513301134109497 (nan) | > loss: 0.014100982807576656 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13654655592485082) | > loader_time: 0.0045 (0.004732076395574318)  --> TIME: 2025-02-24 14:16:55 -- STEP: 21608/36938 -- GLOBAL_STEP: 354050 | > loss_text_ce: 0.04503409191966057 (0.040225309324018216) | > loss_mel_ce: 2.687864303588867 (nan) | > loss: 0.010844836011528969 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13654330503009285) | > loader_time: 0.0051 (0.004732004947814184)  --> TIME: 2025-02-24 14:17:04 -- STEP: 21658/36938 -- GLOBAL_STEP: 354100 | > loss_text_ce: 0.050139665603637695 (0.04022686415049126) | > loss_mel_ce: 3.6407971382141113 (nan) | > loss: 0.014646575786173344 (nan) | > current_lr: 5e-06 | > step_time: 0.1872 (0.13654594378894488) | > loader_time: 0.0049 (0.00473192755519238)  --> TIME: 2025-02-24 14:17:14 -- STEP: 21708/36938 -- GLOBAL_STEP: 354150 | > loss_text_ce: 0.04942822828888893 (0.040225319533710446) | > loss_mel_ce: 2.2251760959625244 (nan) | > loss: 0.009026207961142063 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.136539778328082) | > loader_time: 0.0044 (0.0047316880588134415)  --> TIME: 2025-02-24 14:17:23 -- STEP: 21758/36938 -- GLOBAL_STEP: 354200 | > loss_text_ce: 0.046131063252687454 (0.040224070696554017) | > loss_mel_ce: 2.1287949085235596 (nan) | > loss: 0.008630659431219101 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1365508785558122) | > loader_time: 0.0051 (0.004732056974170301)  --> TIME: 2025-02-24 14:17:32 -- STEP: 21808/36938 -- GLOBAL_STEP: 354250 | > loss_text_ce: 0.027379734441637993 (0.040222354556631514) | > loss_mel_ce: 2.0783824920654297 (nan) | > loss: 0.008356199599802494 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13654204962538655) | > loader_time: 0.0045 (0.004732278783095739)  --> TIME: 2025-02-24 14:17:42 -- STEP: 21858/36938 -- GLOBAL_STEP: 354300 | > loss_text_ce: 0.018458545207977295 (0.040223984731797245) | > loss_mel_ce: 2.493879795074463 (nan) | > loss: 0.009969597682356834 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.1365451405562988) | > loader_time: 0.0045 (0.004732063719988292)  --> TIME: 2025-02-24 14:17:51 -- STEP: 21908/36938 -- GLOBAL_STEP: 354350 | > loss_text_ce: 0.04366990923881531 (0.04022429750566399) | > loss_mel_ce: 3.2052743434906006 (nan) | > loss: 0.012892636470496655 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13653122230626913) | > loader_time: 0.0047 (0.004731929670051443)  --> TIME: 2025-02-24 14:18:00 -- STEP: 21958/36938 -- GLOBAL_STEP: 354400 | > loss_text_ce: 0.03976518660783768 (0.04022575904867095) | > loss_mel_ce: 2.8018879890441895 (nan) | > loss: 0.011276401579380035 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1365234943803881) | > loader_time: 0.0046 (0.004731859803818676)  --> TIME: 2025-02-24 14:18:09 -- STEP: 22008/36938 -- GLOBAL_STEP: 354450 | > loss_text_ce: 0.044028133153915405 (0.04022387617947473) | > loss_mel_ce: 3.5942223072052 (nan) | > loss: 0.01443750225007534 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.13652315336069312) | > loader_time: 0.0045 (0.004732211192535611)  --> TIME: 2025-02-24 14:18:18 -- STEP: 22058/36938 -- GLOBAL_STEP: 354500 | > loss_text_ce: 0.03889593109488487 (0.040223183951747205) | > loss_mel_ce: 3.015105962753296 (nan) | > loss: 0.012119055725634098 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13652055078041758) | > loader_time: 0.0045 (0.004732274503307539)  --> TIME: 2025-02-24 14:18:27 -- STEP: 22108/36938 -- GLOBAL_STEP: 354550 | > loss_text_ce: 0.039030034095048904 (0.040225809530840664) | > loss_mel_ce: 3.5439178943634033 (nan) | > loss: 0.014218048192560673 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13651437542204245) | > loader_time: 0.0046 (0.004732052208351926)  --> TIME: 2025-02-24 14:18:37 -- STEP: 22158/36938 -- GLOBAL_STEP: 354600 | > loss_text_ce: 0.031016403809189796 (0.040227523400807465) | > loss_mel_ce: 3.451777696609497 (nan) | > loss: 0.013820611871778965 (nan) | > current_lr: 5e-06 | > step_time: 0.1667 (0.13651783269001916) | > loader_time: 0.0046 (0.004732245486053133)  --> TIME: 2025-02-24 14:18:46 -- STEP: 22208/36938 -- GLOBAL_STEP: 354650 | > loss_text_ce: 0.03307038173079491 (0.04022291057302985) | > loss_mel_ce: 0.006378173828125 (nan) | > loss: 0.00015654189337510616 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.1365153320369869) | > loader_time: 0.0046 (0.00473218230847324)  --> TIME: 2025-02-24 14:18:55 -- STEP: 22258/36938 -- GLOBAL_STEP: 354700 | > loss_text_ce: 0.03460056334733963 (0.04022024811831057) | > loss_mel_ce: 2.9353623390197754 (nan) | > loss: 0.011785567738115788 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13650503581891363) | > loader_time: 0.0046 (0.004731925609930375)  --> TIME: 2025-02-24 14:19:04 -- STEP: 22308/36938 -- GLOBAL_STEP: 354750 | > loss_text_ce: 0.05429147928953171 (0.04021864910588329) | > loss_mel_ce: 0.1337890625 (nan) | > loss: 0.0007463514339178801 (nan) | > current_lr: 5e-06 | > step_time: 0.1116 (0.13650084868199014) | > loader_time: 0.0044 (0.00473171653169575)  --> TIME: 2025-02-24 14:19:13 -- STEP: 22358/36938 -- GLOBAL_STEP: 354800 | > loss_text_ce: 0.04380267485976219 (0.04021338389305693) | > loss_mel_ce: 3.257565975189209 (nan) | > loss: 0.013100670650601387 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.136490184342203) | > loader_time: 0.0048 (0.004731425435825231)  --> TIME: 2025-02-24 14:19:22 -- STEP: 22408/36938 -- GLOBAL_STEP: 354850 | > loss_text_ce: 0.039804957807064056 (0.04021316956390733) | > loss_mel_ce: 3.758058786392212 (nan) | > loss: 0.015070888213813305 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.1364819041876567) | > loader_time: 0.0046 (0.00473163153222101)  --> TIME: 2025-02-24 14:19:31 -- STEP: 22458/36938 -- GLOBAL_STEP: 354900 | > loss_text_ce: 0.04053827375173569 (0.04021556954009808) | > loss_mel_ce: 3.565790891647339 (nan) | > loss: 0.01431083120405674 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.13647854297368506) | > loader_time: 0.0047 (0.004732173541656159)  --> TIME: 2025-02-24 14:19:41 -- STEP: 22508/36938 -- GLOBAL_STEP: 354950 | > loss_text_ce: 0.04345962405204773 (0.04021532929431112) | > loss_mel_ce: 3.5705482959747314 (nan) | > loss: 0.014341302216053009 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.13648695097906793) | > loader_time: 0.0047 (0.004732090180734146)  --> TIME: 2025-02-24 14:19:50 -- STEP: 22558/36938 -- GLOBAL_STEP: 355000 | > loss_text_ce: 0.047851841896772385 (0.04021783882174244) | > loss_mel_ce: 3.077481746673584 (nan) | > loss: 0.012402118183672428 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13649007896548765) | > loader_time: 0.0047 (0.004732429236107588)  --> TIME: 2025-02-24 14:20:00 -- STEP: 22608/36938 -- GLOBAL_STEP: 355050 | > loss_text_ce: 0.027897825464606285 (0.040217022409218525) | > loss_mel_ce: 2.7645230293273926 (nan) | > loss: 0.011081036180257797 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13649929126609858) | > loader_time: 0.0045 (0.0047324658473670315)  --> TIME: 2025-02-24 14:20:09 -- STEP: 22658/36938 -- GLOBAL_STEP: 355100 | > loss_text_ce: 0.03675157576799393 (0.040216698718522674) | > loss_mel_ce: 3.373957633972168 (nan) | > loss: 0.013534560799598694 (nan) | > current_lr: 5e-06 | > step_time: 0.1722 (0.13650192304431505) | > loader_time: 0.0048 (0.004732396009372643)  --> TIME: 2025-02-24 14:20:18 -- STEP: 22708/36938 -- GLOBAL_STEP: 355150 | > loss_text_ce: 0.029180234298110008 (0.04021397820643221) | > loss_mel_ce: 2.37454891204834 (nan) | > loss: 0.009538608603179455 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13649598743076802) | > loader_time: 0.0045 (0.004732140199974739)  --> TIME: 2025-02-24 14:20:28 -- STEP: 22758/36938 -- GLOBAL_STEP: 355200 | > loss_text_ce: 0.044384606182575226 (0.040213426105925106) | > loss_mel_ce: 2.546530246734619 (nan) | > loss: 0.01028140913695097 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13650011585722302) | > loader_time: 0.0048 (0.004732070441446771)  --> TIME: 2025-02-24 14:20:37 -- STEP: 22808/36938 -- GLOBAL_STEP: 355250 | > loss_text_ce: 0.04008636996150017 (0.040214392928973865) | > loss_mel_ce: 3.897865056991577 (nan) | > loss: 0.01562679186463356 (nan) | > current_lr: 5e-06 | > step_time: 0.1945 (0.13649479555105193) | > loader_time: 0.0048 (0.004731828331821903)  --> TIME: 2025-02-24 14:20:46 -- STEP: 22858/36938 -- GLOBAL_STEP: 355300 | > loss_text_ce: 0.0432494692504406 (0.04021182073089063) | > loss_mel_ce: 3.6113317012786865 (nan) | > loss: 0.014502306468784809 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13649413417524398) | > loader_time: 0.0045 (0.004731639746408048)  --> TIME: 2025-02-24 14:20:56 -- STEP: 22908/36938 -- GLOBAL_STEP: 355350 | > loss_text_ce: 0.041851840913295746 (0.040209969144733626) | > loss_mel_ce: 3.228614568710327 (nan) | > loss: 0.012978041544556618 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.13650376697433347) | > loader_time: 0.0053 (0.004731521975708959)  --> TIME: 2025-02-24 14:21:05 -- STEP: 22958/36938 -- GLOBAL_STEP: 355400 | > loss_text_ce: 0.04280048608779907 (0.04021222763667664) | > loss_mel_ce: 3.3536016941070557 (nan) | > loss: 0.013477787375450134 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.13650019971197996) | > loader_time: 0.0046 (0.004731249805111159)  --> TIME: 2025-02-24 14:21:14 -- STEP: 23008/36938 -- GLOBAL_STEP: 355450 | > loss_text_ce: 0.04203815013170242 (0.0402114212241606) | > loss_mel_ce: 2.742886781692505 (nan) | > loss: 0.011051290668547153 (nan) | > current_lr: 5e-06 | > step_time: 0.1504 (0.13649556926750828) | > loader_time: 0.0047 (0.004731034681263176)  --> TIME: 2025-02-24 14:21:24 -- STEP: 23058/36938 -- GLOBAL_STEP: 355500 | > loss_text_ce: 0.043363068252801895 (0.040212014115366665) | > loss_mel_ce: 3.3389458656311035 (nan) | > loss: 0.013421861454844475 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13649434972609698) | > loader_time: 0.0047 (0.0047308700911262115)  --> TIME: 2025-02-24 14:21:33 -- STEP: 23108/36938 -- GLOBAL_STEP: 355550 | > loss_text_ce: 0.04712195694446564 (0.04021067530196) | > loss_mel_ce: 2.9433016777038574 (nan) | > loss: 0.01186676137149334 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13649532422281482) | > loader_time: 0.0051 (0.0047312189246289)  --> TIME: 2025-02-24 14:21:42 -- STEP: 23158/36938 -- GLOBAL_STEP: 355600 | > loss_text_ce: 0.05348111689090729 (0.04021172402988843) | > loss_mel_ce: 3.2757811546325684 (nan) | > loss: 0.013211359269917011 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13649491358219257) | > loader_time: 0.0044 (0.00473141294448569)  --> TIME: 2025-02-24 14:21:52 -- STEP: 23208/36938 -- GLOBAL_STEP: 355650 | > loss_text_ce: 0.02511265128850937 (0.04021069569136329) | > loss_mel_ce: 0.0016326904296875 (nan) | > loss: 0.00010613231279421598 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13650215399425525) | > loader_time: 0.0045 (0.004731249414777338)  --> TIME: 2025-02-24 14:22:01 -- STEP: 23258/36938 -- GLOBAL_STEP: 355700 | > loss_text_ce: 0.047775786370038986 (0.040208917997873135) | > loss_mel_ce: 3.7314951419830322 (nan) | > loss: 0.014997107908129692 (nan) | > current_lr: 5e-06 | > step_time: 0.1964 (0.13649116178321144) | > loader_time: 0.0049 (0.004731167612363682)  --> TIME: 2025-02-24 14:22:10 -- STEP: 23308/36938 -- GLOBAL_STEP: 355750 | > loss_text_ce: 0.03426972031593323 (0.04021182137787468) | > loss_mel_ce: 3.5180835723876953 (nan) | > loss: 0.014096640981733799 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.1364889764777581) | > loader_time: 0.0052 (0.004731107723739545)  --> TIME: 2025-02-24 14:22:19 -- STEP: 23358/36938 -- GLOBAL_STEP: 355800 | > loss_text_ce: 0.035177964717149734 (0.040209676703241055) | > loss_mel_ce: 2.5095722675323486 (nan) | > loss: 0.010098216123878956 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13649325254835332) | > loader_time: 0.0045 (0.004731134688959379)  --> TIME: 2025-02-24 14:22:28 -- STEP: 23408/36938 -- GLOBAL_STEP: 355850 | > loss_text_ce: 0.030492274090647697 (0.04021265286753664) | > loss_mel_ce: 3.591520071029663 (nan) | > loss: 0.014373065903782845 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13647780440600016) | > loader_time: 0.0045 (0.00473078442654707)  --> TIME: 2025-02-24 14:22:37 -- STEP: 23458/36938 -- GLOBAL_STEP: 355900 | > loss_text_ce: 0.04050590842962265 (0.0402112567117771) | > loss_mel_ce: 3.562870740890503 (nan) | > loss: 0.014299114234745502 (nan) | > current_lr: 5e-06 | > step_time: 0.1532 (0.13648027101840782) | > loader_time: 0.0046 (0.004730708215514968)  --> TIME: 2025-02-24 14:22:46 -- STEP: 23508/36938 -- GLOBAL_STEP: 355950 | > loss_text_ce: 0.03560599684715271 (0.040211667617875375) | > loss_mel_ce: 3.2781171798706055 (nan) | > loss: 0.013149695470929146 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.13647030013261094) | > loader_time: 0.0049 (0.004731094642461986)  --> TIME: 2025-02-24 14:22:56 -- STEP: 23558/36938 -- GLOBAL_STEP: 356000 | > loss_text_ce: 0.041471004486083984 (0.040210259094567595) | > loss_mel_ce: 3.5256481170654297 (nan) | > loss: 0.014155235141515732 (nan) | > current_lr: 5e-06 | > step_time: 0.1502 (0.13647626250696338) | > loader_time: 0.0047 (0.004731027164486787)  --> TIME: 2025-02-24 14:23:05 -- STEP: 23608/36938 -- GLOBAL_STEP: 356050 | > loss_text_ce: 0.05046621710062027 (0.04021151936079683) | > loss_mel_ce: 3.8276140689849854 (nan) | > loss: 0.015389208681881428 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13647373165570406) | > loader_time: 0.0046 (0.004730999792926987)  --> TIME: 2025-02-24 14:23:14 -- STEP: 23658/36938 -- GLOBAL_STEP: 356100 | > loss_text_ce: 0.03758396953344345 (0.04021352046574909) | > loss_mel_ce: 1.974626064300537 (nan) | > loss: 0.00798496138304472 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13647478392102166) | > loader_time: 0.0046 (0.00473106886994077)  --> TIME: 2025-02-24 14:23:24 -- STEP: 23708/36938 -- GLOBAL_STEP: 356150 | > loss_text_ce: 0.03295288607478142 (0.040209161959587465) | > loss_mel_ce: 3.2281157970428467 (nan) | > loss: 0.012940749526023865 (nan) | > current_lr: 5e-06 | > step_time: 0.1573 (0.13647475681277524) | > loader_time: 0.0047 (0.004730977788027038)  --> TIME: 2025-02-24 14:23:33 -- STEP: 23758/36938 -- GLOBAL_STEP: 356200 | > loss_text_ce: 0.04412730038166046 (0.04020504562861771) | > loss_mel_ce: 2.947256565093994 (nan) | > loss: 0.011870571412146091 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13648216692684173) | > loader_time: 0.0046 (0.004730810670702566)  --> TIME: 2025-02-24 14:23:43 -- STEP: 23808/36938 -- GLOBAL_STEP: 356250 | > loss_text_ce: 0.04096909612417221 (0.04020397499886233) | > loss_mel_ce: 1.9306741952896118 (nan) | > loss: 0.007823982276022434 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13648585382328202) | > loader_time: 0.0051 (0.004730682409498669)  --> TIME: 2025-02-24 14:23:52 -- STEP: 23858/36938 -- GLOBAL_STEP: 356300 | > loss_text_ce: 0.03923778608441353 (0.04020674716422373) | > loss_mel_ce: 2.4808290004730225 (nan) | > loss: 0.010000265203416348 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13648771633095694) | > loader_time: 0.0046 (0.004730849646194898)  --> TIME: 2025-02-24 14:24:01 -- STEP: 23908/36938 -- GLOBAL_STEP: 356350 | > loss_text_ce: 0.046923067420721054 (0.040208740360095176) | > loss_mel_ce: 2.7916359901428223 (nan) | > loss: 0.01126412395387888 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13648268891074883) | > loader_time: 0.0045 (0.0047308817064560926)  --> TIME: 2025-02-24 14:24:10 -- STEP: 23958/36938 -- GLOBAL_STEP: 356400 | > loss_text_ce: 0.0505487322807312 (0.04020716031250313) | > loss_mel_ce: 2.976534128189087 (nan) | > loss: 0.012012234888970852 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.1364729699014528) | > loader_time: 0.0046 (0.00473068196145192)  --> TIME: 2025-02-24 14:24:19 -- STEP: 24008/36938 -- GLOBAL_STEP: 356450 | > loss_text_ce: 0.03896712139248848 (0.040208314384697984) | > loss_mel_ce: 3.7325093746185303 (nan) | > loss: 0.014966177754104137 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13646755044518244) | > loader_time: 0.0054 (0.004730503317198349)  --> TIME: 2025-02-24 14:24:28 -- STEP: 24058/36938 -- GLOBAL_STEP: 356500 | > loss_text_ce: 0.03727218136191368 (0.04020433117660264) | > loss_mel_ce: 3.262258529663086 (nan) | > loss: 0.013093376532196999 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13646350705481855) | > loader_time: 0.0047 (0.0047305314774148365)  --> TIME: 2025-02-24 14:24:38 -- STEP: 24108/36938 -- GLOBAL_STEP: 356550 | > loss_text_ce: 0.04159557819366455 (0.04020705241592525) | > loss_mel_ce: 2.340355634689331 (nan) | > loss: 0.00945218838751316 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.136473560493463) | > loader_time: 0.0046 (0.004730826817046225)  --> TIME: 2025-02-24 14:24:47 -- STEP: 24158/36938 -- GLOBAL_STEP: 356600 | > loss_text_ce: 0.04139852896332741 (0.04020394244117883) | > loss_mel_ce: 3.6686673164367676 (nan) | > loss: 0.014722484163939953 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.136476809827092) | > loader_time: 0.0047 (0.004730871106844242)  --> TIME: 2025-02-24 14:24:57 -- STEP: 24208/36938 -- GLOBAL_STEP: 356650 | > loss_text_ce: 0.040393877774477005 (0.04020451746744989) | > loss_mel_ce: 2.679821729660034 (nan) | > loss: 0.010794506408274174 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.1364752116360095) | > loader_time: 0.0046 (0.004730834247100014)  --> TIME: 2025-02-24 14:25:06 -- STEP: 24258/36938 -- GLOBAL_STEP: 356700 | > loss_text_ce: 0.06665323674678802 (0.040204880501737573) | > loss_mel_ce: 1.8634048700332642 (nan) | > loss: 0.007658961229026318 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13647117561868974) | > loader_time: 0.0045 (0.004730742785000695)  --> TIME: 2025-02-24 14:25:16 -- STEP: 24308/36938 -- GLOBAL_STEP: 356750 | > loss_text_ce: 0.056126222014427185 (0.040207087335550694) | > loss_mel_ce: 3.7274861335754395 (nan) | > loss: 0.01501433551311493 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.1364839810946079) | > loader_time: 0.0049 (0.004730553351991277)  --> TIME: 2025-02-24 14:25:25 -- STEP: 24358/36938 -- GLOBAL_STEP: 356800 | > loss_text_ce: 0.034907303750514984 (0.040203058366468196) | > loss_mel_ce: 2.701894998550415 (nan) | > loss: 0.010860327631235123 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13648630256419436) | > loader_time: 0.0046 (0.004730527188960206)  --> TIME: 2025-02-24 14:25:34 -- STEP: 24408/36938 -- GLOBAL_STEP: 356850 | > loss_text_ce: 0.039533831179142 (0.040203552430612355) | > loss_mel_ce: 3.333784580230713 (nan) | > loss: 0.01338618528097868 (nan) | > current_lr: 5e-06 | > step_time: 0.1859 (0.13649177584480854) | > loader_time: 0.0049 (0.004730488415117648)  --> TIME: 2025-02-24 14:25:44 -- STEP: 24458/36938 -- GLOBAL_STEP: 356900 | > loss_text_ce: 0.055147260427474976 (0.040207725515948235) | > loss_mel_ce: 3.3419535160064697 (nan) | > loss: 0.01348055899143219 (nan) | > current_lr: 5e-06 | > step_time: 0.1076 (0.1365047386967835) | > loader_time: 0.0045 (0.004730347883614601)  --> TIME: 2025-02-24 14:25:53 -- STEP: 24508/36938 -- GLOBAL_STEP: 356950 | > loss_text_ce: 0.03225333243608475 (0.04020899849776167) | > loss_mel_ce: 0.205902099609375 (nan) | > loss: 0.0009450612706132233 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13650446345137932) | > loader_time: 0.0045 (0.004730312464695249)  --> TIME: 2025-02-24 14:26:03 -- STEP: 24558/36938 -- GLOBAL_STEP: 357000 | > loss_text_ce: 0.05136733874678612 (0.04020836738881237) | > loss_mel_ce: 2.303264856338501 (nan) | > loss: 0.00934377871453762 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.1365122287453395) | > loader_time: 0.0045 (0.004730517249298067)  --> TIME: 2025-02-24 14:26:12 -- STEP: 24608/36938 -- GLOBAL_STEP: 357050 | > loss_text_ce: 0.043266382068395615 (0.0402042949586588) | > loss_mel_ce: 0.19869790971279144 (nan) | > loss: 0.0009601758210919797 (nan) | > current_lr: 5e-06 | > step_time: 0.1617 (0.13650956039737147) | > loader_time: 0.0047 (0.004730555128369157)  --> TIME: 2025-02-24 14:26:22 -- STEP: 24658/36938 -- GLOBAL_STEP: 357100 | > loss_text_ce: 0.03949422016739845 (0.04020554963243995) | > loss_mel_ce: 2.1947851181030273 (nan) | > loss: 0.008866189047694206 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13651418762298245) | > loader_time: 0.0045 (0.004730669654810951)  --> TIME: 2025-02-24 14:26:31 -- STEP: 24708/36938 -- GLOBAL_STEP: 357150 | > loss_text_ce: 0.03160691261291504 (0.040201635856750816) | > loss_mel_ce: 3.0804755687713623 (nan) | > loss: 0.012349534779787064 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.1365062527157095) | > loader_time: 0.0067 (0.004730639448825171)  --> TIME: 2025-02-24 14:26:40 -- STEP: 24758/36938 -- GLOBAL_STEP: 357200 | > loss_text_ce: 0.037825945764780045 (0.0401975240870458) | > loss_mel_ce: 3.1736092567443848 (nan) | > loss: 0.012743791565299034 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13650478624509754) | > loader_time: 0.0051 (0.004730720610155993)  --> TIME: 2025-02-24 14:26:49 -- STEP: 24808/36938 -- GLOBAL_STEP: 357250 | > loss_text_ce: 0.04450121149420738 (0.040194549003447315) | > loss_mel_ce: 2.6030635833740234 (nan) | > loss: 0.01050621084868908 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.1365052337840228) | > loader_time: 0.0045 (0.004730654796912824)  --> TIME: 2025-02-24 14:26:59 -- STEP: 24858/36938 -- GLOBAL_STEP: 357300 | > loss_text_ce: 0.0344977043569088 (0.0401910512528818) | > loss_mel_ce: 3.558302164077759 (nan) | > loss: 0.014257143251597881 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13651170687622102) | > loader_time: 0.0046 (0.004730466941172416)  --> TIME: 2025-02-24 14:27:08 -- STEP: 24908/36938 -- GLOBAL_STEP: 357350 | > loss_text_ce: 0.032847631722688675 (0.04019167003402349) | > loss_mel_ce: 3.1683509349823 (nan) | > loss: 0.01270317006856203 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13652568897619724) | > loader_time: 0.0047 (0.004730310345492456)  --> TIME: 2025-02-24 14:27:17 -- STEP: 24958/36938 -- GLOBAL_STEP: 357400 | > loss_text_ce: 0.033257532864809036 (0.0401909292078283) | > loss_mel_ce: 2.4505953788757324 (nan) | > loss: 0.009856559336185455 (nan) | > current_lr: 5e-06 | > step_time: 0.1451 (0.13651759913548622) | > loader_time: 0.0047 (0.00473044775303995)  --> TIME: 2025-02-24 14:27:27 -- STEP: 25008/36938 -- GLOBAL_STEP: 357450 | > loss_text_ce: 0.029420720413327217 (0.040190368264667035) | > loss_mel_ce: 3.484375 (nan) | > loss: 0.013943633995950222 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13651527333778163) | > loader_time: 0.0046 (0.004730554618136844)  --> TIME: 2025-02-24 14:27:36 -- STEP: 25058/36938 -- GLOBAL_STEP: 357500 | > loss_text_ce: 0.03448772430419922 (0.040189502779648764) | > loss_mel_ce: 3.1608822345733643 (nan) | > loss: 0.012680040672421455 (nan) | > current_lr: 5e-06 | > step_time: 0.1862 (0.13652440654621523) | > loader_time: 0.007 (0.00473074983814043)  --> TIME: 2025-02-24 14:27:45 -- STEP: 25108/36938 -- GLOBAL_STEP: 357550 | > loss_text_ce: 0.06037683039903641 (0.04018495278655534) | > loss_mel_ce: 2.5717945098876953 (nan) | > loss: 0.01044512540102005 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13651636117997717) | > loader_time: 0.0044 (0.0047306099837423265)  --> TIME: 2025-02-24 14:27:54 -- STEP: 25158/36938 -- GLOBAL_STEP: 357600 | > loss_text_ce: 0.04674624279141426 (0.040183434461293424) | > loss_mel_ce: 3.705488920211792 (nan) | > loss: 0.014889823272824287 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.13650569425628772) | > loader_time: 0.0048 (0.004730477679163413)  --> TIME: 2025-02-24 14:28:03 -- STEP: 25208/36938 -- GLOBAL_STEP: 357650 | > loss_text_ce: 0.03891725465655327 (0.04018075867844666) | > loss_mel_ce: 0.06591796875 (nan) | > loss: 0.0004160128301009536 (nan) | > current_lr: 5e-06 | > step_time: 0.1533 (0.13649503400082832) | > loader_time: 0.0047 (0.004730205674354576)  --> TIME: 2025-02-24 14:28:13 -- STEP: 25258/36938 -- GLOBAL_STEP: 357700 | > loss_text_ce: 0.034868162125349045 (0.04017756208012707) | > loss_mel_ce: 3.345688819885254 (nan) | > loss: 0.013414910063147545 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1364931089123497) | > loader_time: 0.0045 (0.004730359223649966)  --> TIME: 2025-02-24 14:28:21 -- STEP: 25308/36938 -- GLOBAL_STEP: 357750 | > loss_text_ce: 0.03405117243528366 (0.0401742196092741) | > loss_mel_ce: 3.528942823410034 (nan) | > loss: 0.01413886621594429 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13647953624952894) | > loader_time: 0.0047 (0.0047301478685126295)  --> TIME: 2025-02-24 14:28:31 -- STEP: 25358/36938 -- GLOBAL_STEP: 357800 | > loss_text_ce: 0.035378798842430115 (0.040174403422114806) | > loss_mel_ce: 2.6917693614959717 (nan) | > loss: 0.01082201674580574 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13648201031718285) | > loader_time: 0.0052 (0.0047303177090189325)  --> TIME: 2025-02-24 14:28:40 -- STEP: 25408/36938 -- GLOBAL_STEP: 357850 | > loss_text_ce: 0.03676985576748848 (0.04017512361163211) | > loss_mel_ce: 2.892514944076538 (nan) | > loss: 0.011624147184193134 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.1364832808092343) | > loader_time: 0.0045 (0.004730292743714714)  --> TIME: 2025-02-24 14:28:49 -- STEP: 25458/36938 -- GLOBAL_STEP: 357900 | > loss_text_ce: 0.037648607045412064 (0.04017068969316446) | > loss_mel_ce: 2.4695091247558594 (nan) | > loss: 0.009949039667844772 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.1364748499850291) | > loader_time: 0.0047 (0.00473019678544547)  --> TIME: 2025-02-24 14:28:58 -- STEP: 25508/36938 -- GLOBAL_STEP: 357950 | > loss_text_ce: 0.049339406192302704 (0.04017413635995248) | > loss_mel_ce: 3.303882122039795 (nan) | > loss: 0.013306435197591782 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1364689404470556) | > loader_time: 0.0045 (0.004729979891042033)  --> TIME: 2025-02-24 14:29:08 -- STEP: 25558/36938 -- GLOBAL_STEP: 358000 | > loss_text_ce: 0.030538609251379967 (0.04017349457977832) | > loss_mel_ce: 2.5517263412475586 (nan) | > loss: 0.010247083380818367 (nan) | > current_lr: 5e-06 | > step_time: 0.1121 (0.13646666218542428) | > loader_time: 0.0045 (0.004729815711903481)  --> TIME: 2025-02-24 14:29:17 -- STEP: 25608/36938 -- GLOBAL_STEP: 358050 | > loss_text_ce: 0.03015858307480812 (0.04017476868196455) | > loss_mel_ce: 2.568777322769165 (nan) | > loss: 0.010313238017261028 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1364627939971) | > loader_time: 0.0046 (0.004729543662153186)  --> TIME: 2025-02-24 14:29:26 -- STEP: 25658/36938 -- GLOBAL_STEP: 358100 | > loss_text_ce: 0.04233279079198837 (0.040174890390116645) | > loss_mel_ce: 3.074431896209717 (nan) | > loss: 0.012368114665150642 (nan) | > current_lr: 5e-06 | > step_time: 0.1821 (0.13646027644877906) | > loader_time: 0.0049 (0.004729417742102474)  --> TIME: 2025-02-24 14:29:35 -- STEP: 25708/36938 -- GLOBAL_STEP: 358150 | > loss_text_ce: 0.05098990350961685 (0.04017296887195049) | > loss_mel_ce: 1.5096696615219116 (nan) | > loss: 0.006193093489855528 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.13646314996238837) | > loader_time: 0.0046 (0.004729433686250649)  --> TIME: 2025-02-24 14:29:45 -- STEP: 25758/36938 -- GLOBAL_STEP: 358200 | > loss_text_ce: 0.04296881705522537 (0.040174116883415496) | > loss_mel_ce: 3.2873458862304688 (nan) | > loss: 0.013215535320341587 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13646988342411576) | > loader_time: 0.0045 (0.004729287697852576)  --> TIME: 2025-02-24 14:29:54 -- STEP: 25808/36938 -- GLOBAL_STEP: 358250 | > loss_text_ce: 0.03249093517661095 (0.04017431900968043) | > loss_mel_ce: 3.3153159618377686 (nan) | > loss: 0.013284948654472828 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.13645690164984284) | > loader_time: 0.0053 (0.004729254657410741)  --> TIME: 2025-02-24 14:30:03 -- STEP: 25858/36938 -- GLOBAL_STEP: 358300 | > loss_text_ce: 0.03804744407534599 (0.04017463704072383) | > loss_mel_ce: 4.000091075897217 (nan) | > loss: 0.016024360433220863 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13646781770026184) | > loader_time: 0.0047 (0.004729343139246609)  --> TIME: 2025-02-24 14:30:12 -- STEP: 25908/36938 -- GLOBAL_STEP: 358350 | > loss_text_ce: 0.028149470686912537 (0.04017584722335136) | > loss_mel_ce: 3.3550362586975098 (nan) | > loss: 0.013425340875983238 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13645521777637226) | > loader_time: 0.0045 (0.004729378383542856)  --> TIME: 2025-02-24 14:30:21 -- STEP: 25958/36938 -- GLOBAL_STEP: 358400 | > loss_text_ce: 0.03564099967479706 (0.04017640686771132) | > loss_mel_ce: 3.0959670543670654 (nan) | > loss: 0.012427017092704773 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13644776014381868) | > loader_time: 0.0045 (0.0047292031513652)  --> TIME: 2025-02-24 14:30:31 -- STEP: 26008/36938 -- GLOBAL_STEP: 358450 | > loss_text_ce: 0.04519856721162796 (0.04017422467945586) | > loss_mel_ce: 3.8683671951293945 (nan) | > loss: 0.015530024655163288 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13645418096013967) | > loader_time: 0.0046 (0.004729081129741784)  --> TIME: 2025-02-24 14:30:40 -- STEP: 26058/36938 -- GLOBAL_STEP: 358500 | > loss_text_ce: 0.030600961297750473 (0.0401710472976058) | > loss_mel_ce: 2.9469316005706787 (nan) | > loss: 0.011815606616437435 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1364467867191678) | > loader_time: 0.0045 (0.00472891298695625)  --> TIME: 2025-02-24 14:30:49 -- STEP: 26108/36938 -- GLOBAL_STEP: 358550 | > loss_text_ce: 0.04650216922163963 (0.0401712051883923) | > loss_mel_ce: 3.4531376361846924 (nan) | > loss: 0.013887460343539715 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.13644735123125934) | > loader_time: 0.0047 (0.004728846433463343)  --> TIME: 2025-02-24 14:30:58 -- STEP: 26158/36938 -- GLOBAL_STEP: 358600 | > loss_text_ce: 0.0457410030066967 (0.04017271859004826) | > loss_mel_ce: 3.270505428314209 (nan) | > loss: 0.013159709051251411 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13644743840695622) | > loader_time: 0.0047 (0.0047290215152093525)  --> TIME: 2025-02-24 14:31:08 -- STEP: 26208/36938 -- GLOBAL_STEP: 358650 | > loss_text_ce: 0.035554807633161545 (0.04017509286352608) | > loss_mel_ce: 2.8043055534362793 (nan) | > loss: 0.011269288137555122 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13645116762873225) | > loader_time: 0.0048 (0.004728945156375157)  --> TIME: 2025-02-24 14:31:17 -- STEP: 26258/36938 -- GLOBAL_STEP: 358700 | > loss_text_ce: 0.04772300645709038 (0.040173699235499356) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.1364529012353461) | > loader_time: 0.0046 (0.004729107933674965)  --> TIME: 2025-02-24 14:31:26 -- STEP: 26308/36938 -- GLOBAL_STEP: 358750 | > loss_text_ce: 0.04373518377542496 (0.040172766771084695) | > loss_mel_ce: 3.4023218154907227 (nan) | > loss: 0.013674830086529255 (nan) | > current_lr: 5e-06 | > step_time: 0.1877 (0.13645266191528282) | > loader_time: 0.0049 (0.004729259398383414)  --> TIME: 2025-02-24 14:31:36 -- STEP: 26358/36938 -- GLOBAL_STEP: 358800 | > loss_text_ce: 0.04654484987258911 (0.040172291796804996) | > loss_mel_ce: 3.2186546325683594 (nan) | > loss: 0.012957141734659672 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13646175616409204) | > loader_time: 0.0046 (0.004729412983976072)  --> TIME: 2025-02-24 14:31:45 -- STEP: 26408/36938 -- GLOBAL_STEP: 358850 | > loss_text_ce: 0.047733332961797714 (0.040174392768014426) | > loss_mel_ce: 3.3997559547424316 (nan) | > loss: 0.013680513948202133 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13646240196997872) | > loader_time: 0.0046 (0.004729404679864392)  --> TIME: 2025-02-24 14:31:55 -- STEP: 26458/36938 -- GLOBAL_STEP: 358900 | > loss_text_ce: 0.05216919258236885 (0.04017506555207576) | > loss_mel_ce: 2.9133758544921875 (nan) | > loss: 0.011768036521971226 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13647655244406615) | > loader_time: 0.0044 (0.004729358560059119)  --> TIME: 2025-02-24 14:32:04 -- STEP: 26508/36938 -- GLOBAL_STEP: 358950 | > loss_text_ce: 0.03866977244615555 (0.040173158048072845) | > loss_mel_ce: 0.041015625 (nan) | > loss: 0.00031621192465536296 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.136475293978461) | > loader_time: 0.0046 (0.004729417001066038)  --> TIME: 2025-02-24 14:32:14 -- STEP: 26558/36938 -- GLOBAL_STEP: 359000 | > loss_text_ce: 0.02657121792435646 (0.04017151433940715) | > loss_mel_ce: 1.5416988134384155 (nan) | > loss: 0.006223294418305159 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13647499450027367) | > loader_time: 0.0046 (0.004729431816877887)  --> TIME: 2025-02-24 14:32:24 -- STEP: 26608/36938 -- GLOBAL_STEP: 359050 | > loss_text_ce: 0.04782748222351074 (0.040171037205506015) | > loss_mel_ce: 3.1300456523895264 (nan) | > loss: 0.012610608711838722 (nan) | > current_lr: 5e-06 | > step_time: 0.1832 (0.1364900193888947) | > loader_time: 0.005 (0.004729506441504596)  --> TIME: 2025-02-24 14:32:33 -- STEP: 26658/36938 -- GLOBAL_STEP: 359100 | > loss_text_ce: 0.046474166214466095 (0.040172933033308586) | > loss_mel_ce: 3.0114362239837646 (nan) | > loss: 0.012134565971791744 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13648988156157943) | > loader_time: 0.0048 (0.004729503701280384)  --> TIME: 2025-02-24 14:32:42 -- STEP: 26708/36938 -- GLOBAL_STEP: 359150 | > loss_text_ce: 0.04563000425696373 (0.04017709477845624) | > loss_mel_ce: 3.382115364074707 (nan) | > loss: 0.013602164573967457 (nan) | > current_lr: 5e-06 | > step_time: 0.1357 (0.13649446595227427) | > loader_time: 0.0047 (0.004729612226773324)  --> TIME: 2025-02-24 14:32:52 -- STEP: 26758/36938 -- GLOBAL_STEP: 359200 | > loss_text_ce: 0.031677015125751495 (0.040176511950763466) | > loss_mel_ce: 3.321809768676758 (nan) | > loss: 0.013307487592101097 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.136500943160589) | > loader_time: 0.0046 (0.004729623225729844)  --> TIME: 2025-02-24 14:33:01 -- STEP: 26808/36938 -- GLOBAL_STEP: 359250 | > loss_text_ce: 0.052231431007385254 (0.04017420551243971) | > loss_mel_ce: 2.18812894821167 (nan) | > loss: 0.008890318684279919 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13650640348148618) | > loader_time: 0.0046 (0.0047296145466753856)  --> TIME: 2025-02-24 14:33:11 -- STEP: 26858/36938 -- GLOBAL_STEP: 359300 | > loss_text_ce: 0.047266390174627304 (0.04017342968220214) | > loss_mel_ce: 3.543046712875366 (nan) | > loss: 0.014247274957597256 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13650493907133) | > loader_time: 0.0046 (0.004729643183353211)  --> TIME: 2025-02-24 14:33:20 -- STEP: 26908/36938 -- GLOBAL_STEP: 359350 | > loss_text_ce: 0.044928405433893204 (0.040174712354311366) | > loss_mel_ce: 3.714510917663574 (nan) | > loss: 0.01491841021925211 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.13649494066375836) | > loader_time: 0.0049 (0.004729856526105632)  --> TIME: 2025-02-24 14:33:29 -- STEP: 26958/36938 -- GLOBAL_STEP: 359400 | > loss_text_ce: 0.039234887808561325 (0.04017101637320556) | > loss_mel_ce: 3.3731236457824707 (nan) | > loss: 0.013541106134653091 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.13650749664347592) | > loader_time: 0.0048 (0.004730074551951206)  --> TIME: 2025-02-24 14:33:39 -- STEP: 27008/36938 -- GLOBAL_STEP: 359450 | > loss_text_ce: 0.03464248403906822 (0.04017103027578037) | > loss_mel_ce: 3.227804183959961 (nan) | > loss: 0.012946217320859432 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.1365077985247579) | > loader_time: 0.0046 (0.004730227778520102)  --> TIME: 2025-02-24 14:33:48 -- STEP: 27058/36938 -- GLOBAL_STEP: 359500 | > loss_text_ce: 0.039243075996637344 (0.04017151497386786) | > loss_mel_ce: 3.2878198623657227 (nan) | > loss: 0.013202630914747715 (nan) | > current_lr: 5e-06 | > step_time: 0.1879 (0.1365033098683661) | > loader_time: 0.005 (0.004730214352917572)  --> TIME: 2025-02-24 14:33:57 -- STEP: 27108/36938 -- GLOBAL_STEP: 359550 | > loss_text_ce: 0.053985241800546646 (0.040172659297142276) | > loss_mel_ce: 2.961024045944214 (nan) | > loss: 0.011964323930442333 (nan) | > current_lr: 5e-06 | > step_time: 0.1456 (0.13650733411039376) | > loader_time: 0.0048 (0.004730361074723764)  --> TIME: 2025-02-24 14:34:07 -- STEP: 27158/36938 -- GLOBAL_STEP: 359600 | > loss_text_ce: 0.04040086269378662 (0.040172543224973205) | > loss_mel_ce: 3.534963846206665 (nan) | > loss: 0.014187955297529697 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13650949533842963) | > loader_time: 0.0045 (0.004730197868709598)  --> TIME: 2025-02-24 14:34:16 -- STEP: 27208/36938 -- GLOBAL_STEP: 359650 | > loss_text_ce: 0.03719654306769371 (0.040171934898956006) | > loss_mel_ce: 4.035196304321289 (nan) | > loss: 0.01616029068827629 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.13651286286839384) | > loader_time: 0.0046 (0.00472996976928692)  --> TIME: 2025-02-24 14:34:26 -- STEP: 27258/36938 -- GLOBAL_STEP: 359700 | > loss_text_ce: 0.033673688769340515 (0.04017045232824413) | > loss_mel_ce: 2.728830099105835 (nan) | > loss: 0.010962317697703838 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13652457469486967) | > loader_time: 0.0046 (0.00473010526013768)  --> TIME: 2025-02-24 14:34:35 -- STEP: 27308/36938 -- GLOBAL_STEP: 359750 | > loss_text_ce: 0.04059514403343201 (0.04016668406329214) | > loss_mel_ce: 3.2018883228302 (nan) | > loss: 0.012866998091340065 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.13651892695190093) | > loader_time: 0.0049 (0.004730313540517804)  --> TIME: 2025-02-24 14:34:44 -- STEP: 27358/36938 -- GLOBAL_STEP: 359800 | > loss_text_ce: 0.040427159518003464 (0.04016611556216117) | > loss_mel_ce: 3.2064993381500244 (nan) | > loss: 0.012884629890322685 (nan) | > current_lr: 5e-06 | > step_time: 0.152 (0.1365144247461085) | > loader_time: 0.0067 (0.004730295957167532)  --> TIME: 2025-02-24 14:34:53 -- STEP: 27408/36938 -- GLOBAL_STEP: 359850 | > loss_text_ce: 0.03782631456851959 (0.04016737581764336) | > loss_mel_ce: 0.07861328125 (nan) | > loss: 0.00046206192928366363 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.13651767733218104) | > loader_time: 0.0048 (0.004730246112977178)  --> TIME: 2025-02-24 14:35:03 -- STEP: 27458/36938 -- GLOBAL_STEP: 359900 | > loss_text_ce: 0.04984842240810394 (0.04016912766800873) | > loss_mel_ce: 3.583866596221924 (nan) | > loss: 0.014419504441320896 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13652483496039675) | > loader_time: 0.0047 (0.004730454457805635)  --> TIME: 2025-02-24 14:35:12 -- STEP: 27508/36938 -- GLOBAL_STEP: 359950 | > loss_text_ce: 0.03804992511868477 (0.040168038533096834) | > loss_mel_ce: 2.953158140182495 (nan) | > loss: 0.011869873851537704 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.1365203025973043) | > loader_time: 0.0045 (0.004730261557841034)  --> TIME: 2025-02-24 14:35:21 -- STEP: 27558/36938 -- GLOBAL_STEP: 360000 | > loss_text_ce: 0.04051674157381058 (0.040170711149737554) | > loss_mel_ce: 2.8214480876922607 (nan) | > loss: 0.011357004754245281 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13652431136058607) | > loader_time: 0.0046 (0.0047301138180141245) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_360000.pth  --> TIME: 2025-02-24 14:35:37 -- STEP: 27608/36938 -- GLOBAL_STEP: 360050 | > loss_text_ce: 0.02719239704310894 (0.04016671178022361) | > loss_mel_ce: 1.3216356039047241 (nan) | > loss: 0.005352491978555918 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13651638255883347) | > loader_time: 0.0045 (0.004730140703445067)  --> TIME: 2025-02-24 14:35:45 -- STEP: 27658/36938 -- GLOBAL_STEP: 360100 | > loss_text_ce: 0.04408121481537819 (0.04016364087045673) | > loss_mel_ce: 1.8059407472610474 (nan) | > loss: 0.007341357413679361 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.1365029551441694) | > loader_time: 0.0046 (0.004730156569826086)  --> TIME: 2025-02-24 14:35:55 -- STEP: 27708/36938 -- GLOBAL_STEP: 360150 | > loss_text_ce: 0.03467992693185806 (0.04016230260037461) | > loss_mel_ce: 3.564680337905884 (nan) | > loss: 0.014283176511526108 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.13650077974639616) | > loader_time: 0.0047 (0.004730320078341945)  --> TIME: 2025-02-24 14:36:03 -- STEP: 27758/36938 -- GLOBAL_STEP: 360200 | > loss_text_ce: 0.039938438683748245 (0.040161732482510044) | > loss_mel_ce: 3.3405094146728516 (nan) | > loss: 0.013414476066827774 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13648868906660216) | > loader_time: 0.0046 (0.004730161453102532)  --> TIME: 2025-02-24 14:36:13 -- STEP: 27808/36938 -- GLOBAL_STEP: 360250 | > loss_text_ce: 0.04815831407904625 (0.040162305134952635) | > loss_mel_ce: 2.892526149749756 (nan) | > loss: 0.011669383384287357 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.1364840252541418) | > loader_time: 0.0046 (0.004730013643913391)  --> TIME: 2025-02-24 14:36:22 -- STEP: 27858/36938 -- GLOBAL_STEP: 360300 | > loss_text_ce: 0.04386332258582115 (0.04016311830744805) | > loss_mel_ce: 1.185272216796875 (nan) | > loss: 0.004877522122114897 (nan) | > current_lr: 5e-06 | > step_time: 0.151 (0.13648413883500557) | > loader_time: 0.0047 (0.004730152967407713)  --> TIME: 2025-02-24 14:36:31 -- STEP: 27908/36938 -- GLOBAL_STEP: 360350 | > loss_text_ce: 0.04266577959060669 (0.040164081810059225) | > loss_mel_ce: 2.266165018081665 (nan) | > loss: 0.009162027388811111 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13647834873992185) | > loader_time: 0.0046 (0.004729997459038223)  --> TIME: 2025-02-24 14:36:40 -- STEP: 27958/36938 -- GLOBAL_STEP: 360400 | > loss_text_ce: 0.04447830468416214 (0.040162815533842905) | > loss_mel_ce: 3.4286892414093018 (nan) | > loss: 0.013782411813735962 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13647430280271527) | > loader_time: 0.0053 (0.004729895208328645)  --> TIME: 2025-02-24 14:36:50 -- STEP: 28008/36938 -- GLOBAL_STEP: 360450 | > loss_text_ce: 0.03960029408335686 (0.04016155445915272) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13648117582003214) | > loader_time: 0.0045 (0.004729781762697635)  --> TIME: 2025-02-24 14:36:59 -- STEP: 28058/36938 -- GLOBAL_STEP: 360500 | > loss_text_ce: 0.03689512982964516 (0.040159140374415765) | > loss_mel_ce: 1.7854660749435425 (nan) | > loss: 0.007231592666357756 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.1364832806123258) | > loader_time: 0.0046 (0.004729589602588511)  --> TIME: 2025-02-24 14:37:08 -- STEP: 28108/36938 -- GLOBAL_STEP: 360550 | > loss_text_ce: 0.04210028052330017 (0.04015927922478435) | > loss_mel_ce: 3.341665029525757 (nan) | > loss: 0.013427640311419964 (nan) | > current_lr: 5e-06 | > step_time: 0.1823 (0.13648555499146406) | > loader_time: 0.0048 (0.004729773040786176)  --> TIME: 2025-02-24 14:37:18 -- STEP: 28158/36938 -- GLOBAL_STEP: 360600 | > loss_text_ce: 0.03640947863459587 (0.04015746891454392) | > loss_mel_ce: 3.229743242263794 (nan) | > loss: 0.012960923835635185 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13649195776006662) | > loader_time: 0.0045 (0.00472961593837648)  --> TIME: 2025-02-24 14:37:27 -- STEP: 28208/36938 -- GLOBAL_STEP: 360650 | > loss_text_ce: 0.043864794075489044 (0.04015593451477646) | > loss_mel_ce: 3.4796714782714844 (nan) | > loss: 0.013982287608087063 (nan) | > current_lr: 5e-06 | > step_time: 0.1343 (0.136490047747143) | > loader_time: 0.0046 (0.004729479982375592)  --> TIME: 2025-02-24 14:37:36 -- STEP: 28258/36938 -- GLOBAL_STEP: 360700 | > loss_text_ce: 0.03450998663902283 (0.04015752253562893) | > loss_mel_ce: 3.305366039276123 (nan) | > loss: 0.013253476470708847 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13648748048526732) | > loader_time: 0.0045 (0.004729344642493939)  --> TIME: 2025-02-24 14:37:46 -- STEP: 28308/36938 -- GLOBAL_STEP: 360750 | > loss_text_ce: 0.04333712160587311 (0.04015738049154107) | > loss_mel_ce: 3.3586831092834473 (nan) | > loss: 0.01350008137524128 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13649306756149276) | > loader_time: 0.0045 (0.0047291755861560555)  --> TIME: 2025-02-24 14:37:55 -- STEP: 28358/36938 -- GLOBAL_STEP: 360800 | > loss_text_ce: 0.03599001467227936 (0.040156354357884866) | > loss_mel_ce: 2.694455623626709 (nan) | > loss: 0.010835102759301662 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13649154240762623) | > loader_time: 0.0045 (0.0047290677689339365)  --> TIME: 2025-02-24 14:38:04 -- STEP: 28408/36938 -- GLOBAL_STEP: 360850 | > loss_text_ce: 0.03762126713991165 (0.040155466909275774) | > loss_mel_ce: 3.201690673828125 (nan) | > loss: 0.012854413129389286 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.13648565360708098) | > loader_time: 0.0046 (0.004728888221540407)  --> TIME: 2025-02-24 14:38:14 -- STEP: 28458/36938 -- GLOBAL_STEP: 360900 | > loss_text_ce: 0.049869611859321594 (0.040155111161065425) | > loss_mel_ce: 3.4252052307128906 (nan) | > loss: 0.013789979740977287 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.1364942947420259) | > loader_time: 0.0048 (0.004728782117488305)  --> TIME: 2025-02-24 14:38:23 -- STEP: 28508/36938 -- GLOBAL_STEP: 360950 | > loss_text_ce: 0.04304828122258186 (0.04015369966437487) | > loss_mel_ce: 3.4648430347442627 (nan) | > loss: 0.01392020471394062 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13649959277016036) | > loader_time: 0.0045 (0.004728693463314964)  --> TIME: 2025-02-24 14:38:33 -- STEP: 28558/36938 -- GLOBAL_STEP: 361000 | > loss_text_ce: 0.04893193021416664 (0.040154432412139236) | > loss_mel_ce: 3.1519615650177 (nan) | > loss: 0.012701958417892456 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13649833813477588) | > loader_time: 0.0052 (0.004728820087011593)  --> TIME: 2025-02-24 14:38:42 -- STEP: 28608/36938 -- GLOBAL_STEP: 361050 | > loss_text_ce: 0.03763251751661301 (0.04015512607553655) | > loss_mel_ce: 2.2334957122802734 (nan) | > loss: 0.009012414142489433 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13649042279061502) | > loader_time: 0.0045 (0.004728836651216451)  --> TIME: 2025-02-24 14:38:51 -- STEP: 28658/36938 -- GLOBAL_STEP: 361100 | > loss_text_ce: 0.04181583225727081 (0.040156172443267475) | > loss_mel_ce: 2.3063902854919434 (nan) | > loss: 0.009318278171122074 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13649657632178222) | > loader_time: 0.0046 (0.004728962408538773)  --> TIME: 2025-02-24 14:39:00 -- STEP: 28708/36938 -- GLOBAL_STEP: 361150 | > loss_text_ce: 0.037704844027757645 (0.04015828425800983) | > loss_mel_ce: 2.644000291824341 (nan) | > loss: 0.010641688480973244 (nan) | > current_lr: 5e-06 | > step_time: 0.1779 (0.13649455671674435) | > loader_time: 0.0049 (0.004729111720812426)  --> TIME: 2025-02-24 14:39:10 -- STEP: 28758/36938 -- GLOBAL_STEP: 361200 | > loss_text_ce: 0.04979056119918823 (0.040157197874488536) | > loss_mel_ce: 0.0137481689453125 (nan) | > loss: 0.0002521378337405622 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.1364947651795373) | > loader_time: 0.0053 (0.0047292411721161776)  --> TIME: 2025-02-24 14:39:19 -- STEP: 28808/36938 -- GLOBAL_STEP: 361250 | > loss_text_ce: 0.03146477788686752 (0.04015777747425343) | > loss_mel_ce: 3.3910698890686035 (nan) | > loss: 0.013581487350165844 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13649740894613357) | > loader_time: 0.0053 (0.0047295906168989136)  --> TIME: 2025-02-24 14:39:28 -- STEP: 28858/36938 -- GLOBAL_STEP: 361300 | > loss_text_ce: 0.041199855506420135 (0.040156916928655806) | > loss_mel_ce: 2.9876866340637207 (nan) | > loss: 0.012019392102956772 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.1364933047690182) | > loader_time: 0.0046 (0.0047295794466180145)  --> TIME: 2025-02-24 14:39:37 -- STEP: 28908/36938 -- GLOBAL_STEP: 361350 | > loss_text_ce: 0.04178667068481445 (0.04015673159689841) | > loss_mel_ce: 3.7439608573913574 (nan) | > loss: 0.01502280868589878 (nan) | > current_lr: 5e-06 | > step_time: 0.1964 (0.13649520415297692) | > loader_time: 0.0048 (0.004729534277566954)  --> TIME: 2025-02-24 14:39:47 -- STEP: 28958/36938 -- GLOBAL_STEP: 361400 | > loss_text_ce: 0.03850933536887169 (0.0401588389581807) | > loss_mel_ce: 2.9413375854492188 (nan) | > loss: 0.011824790388345718 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13649606714591722) | > loader_time: 0.0054 (0.004729523243137796)  --> TIME: 2025-02-24 14:39:56 -- STEP: 29008/36938 -- GLOBAL_STEP: 361450 | > loss_text_ce: 0.04569435492157936 (0.04015793304979534) | > loss_mel_ce: 1.7746080160140991 (nan) | > loss: 0.007223422639071941 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.1364959963385193) | > loader_time: 0.0053 (0.004729711493272809)  --> TIME: 2025-02-24 14:40:05 -- STEP: 29058/36938 -- GLOBAL_STEP: 361500 | > loss_text_ce: 0.03681885823607445 (0.04015651309260326) | > loss_mel_ce: 3.7101104259490967 (nan) | > loss: 0.014868767000734806 (nan) | > current_lr: 5e-06 | > step_time: 0.1448 (0.13649898808085578) | > loader_time: 0.0047 (0.004729956612057627)  --> TIME: 2025-02-24 14:40:15 -- STEP: 29108/36938 -- GLOBAL_STEP: 361550 | > loss_text_ce: 0.0352240726351738 (0.04015542318651837) | > loss_mel_ce: 3.471726894378662 (nan) | > loss: 0.013916472904384136 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.136500660757515) | > loader_time: 0.0047 (0.004729800562093919)  --> TIME: 2025-02-24 14:40:24 -- STEP: 29158/36938 -- GLOBAL_STEP: 361600 | > loss_text_ce: 0.05049628019332886 (0.04015492852248321) | > loss_mel_ce: 3.765221357345581 (nan) | > loss: 0.015141737647354603 (nan) | > current_lr: 5e-06 | > step_time: 0.168 (0.13649933727707025) | > loader_time: 0.0047 (0.004729690215855684)  --> TIME: 2025-02-24 14:40:33 -- STEP: 29208/36938 -- GLOBAL_STEP: 361650 | > loss_text_ce: 0.036465924233198166 (0.04015150655456278) | > loss_mel_ce: 2.3339455127716064 (nan) | > loss: 0.009406395256519318 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13649167205757684) | > loader_time: 0.0045 (0.0047296682503804284)  --> TIME: 2025-02-24 14:40:42 -- STEP: 29258/36938 -- GLOBAL_STEP: 361700 | > loss_text_ce: 0.04716407507658005 (0.04015341190952572) | > loss_mel_ce: 3.4035255908966064 (nan) | > loss: 0.01369321346282959 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13648659043419872) | > loader_time: 0.0046 (0.004729470972630566)  --> TIME: 2025-02-24 14:40:52 -- STEP: 29308/36938 -- GLOBAL_STEP: 361750 | > loss_text_ce: 0.03605266660451889 (0.04015389945698321) | > loss_mel_ce: 2.203403949737549 (nan) | > loss: 0.008886733092367649 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13648288038786563) | > loader_time: 0.0045 (0.004729308965864458)  --> TIME: 2025-02-24 14:41:01 -- STEP: 29358/36938 -- GLOBAL_STEP: 361800 | > loss_text_ce: 0.03873114660382271 (0.040153026024485186) | > loss_mel_ce: 3.347609519958496 (nan) | > loss: 0.013437860645353794 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13648338369521051) | > loader_time: 0.0051 (0.004729461130682394)  --> TIME: 2025-02-24 14:41:10 -- STEP: 29408/36938 -- GLOBAL_STEP: 361850 | > loss_text_ce: 0.03223740682005882 (0.04015501663483408) | > loss_mel_ce: 2.6885180473327637 (nan) | > loss: 0.01079664845019579 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1364796408599028) | > loader_time: 0.0046 (0.0047293261050270295)  --> TIME: 2025-02-24 14:41:19 -- STEP: 29458/36938 -- GLOBAL_STEP: 361900 | > loss_text_ce: 0.04632890969514847 (0.04015645708532036) | > loss_mel_ce: 3.0673985481262207 (nan) | > loss: 0.012356062419712543 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.136479966589806) | > loader_time: 0.0045 (0.00472931949611079)  --> TIME: 2025-02-24 14:41:29 -- STEP: 29508/36938 -- GLOBAL_STEP: 361950 | > loss_text_ce: 0.03044023923575878 (0.04015598253348761) | > loss_mel_ce: 2.6681151390075684 (nan) | > loss: 0.010708553716540337 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.13648451980866327) | > loader_time: 0.0046 (0.004729261384741407)  --> TIME: 2025-02-24 14:41:38 -- STEP: 29558/36938 -- GLOBAL_STEP: 362000 | > loss_text_ce: 0.03629682958126068 (0.04015298619752577) | > loss_mel_ce: 3.866084575653076 (nan) | > loss: 0.015485641546547413 (nan) | > current_lr: 5e-06 | > step_time: 0.1691 (0.1364928875089496) | > loader_time: 0.0048 (0.004729116242876814)  --> TIME: 2025-02-24 14:41:48 -- STEP: 29608/36938 -- GLOBAL_STEP: 362050 | > loss_text_ce: 0.03555986285209656 (0.04015264897030394) | > loss_mel_ce: 2.2432968616485596 (nan) | > loss: 0.009043082594871521 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13649576934599442) | > loader_time: 0.0044 (0.004728978806268683)  --> TIME: 2025-02-24 14:41:57 -- STEP: 29658/36938 -- GLOBAL_STEP: 362100 | > loss_text_ce: 0.03707495331764221 (0.04014923280263024) | > loss_mel_ce: 3.2948968410491943 (nan) | > loss: 0.013222111389040947 (nan) | > current_lr: 5e-06 | > step_time: 0.1435 (0.1364904917358048) | > loader_time: 0.0046 (0.004728730855639927)  --> TIME: 2025-02-24 14:42:07 -- STEP: 29708/36938 -- GLOBAL_STEP: 362150 | > loss_text_ce: 0.035327862948179245 (0.040148196212105594) | > loss_mel_ce: 3.2265725135803223 (nan) | > loss: 0.012944050133228302 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1364993831102395) | > loader_time: 0.0045 (0.00472854414682172)  --> TIME: 2025-02-24 14:42:16 -- STEP: 29758/36938 -- GLOBAL_STEP: 362200 | > loss_text_ce: 0.03152715042233467 (0.04014971695259634) | > loss_mel_ce: 3.742466449737549 (nan) | > loss: 0.014976165257394314 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13649192382126066) | > loader_time: 0.0045 (0.004728430453083289)  --> TIME: 2025-02-24 14:42:25 -- STEP: 29808/36938 -- GLOBAL_STEP: 362250 | > loss_text_ce: 0.040743473917245865 (0.040149511780193284) | > loss_mel_ce: 3.2682926654815674 (nan) | > loss: 0.013131096959114075 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.13650016189901176) | > loader_time: 0.0055 (0.0047284655844852036)  --> TIME: 2025-02-24 14:42:34 -- STEP: 29858/36938 -- GLOBAL_STEP: 362300 | > loss_text_ce: 0.0408821664750576 (0.04014749561751961) | > loss_mel_ce: 3.617058753967285 (nan) | > loss: 0.014515639282763004 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.13649111356536456) | > loader_time: 0.0049 (0.0047286026475737925)  --> TIME: 2025-02-24 14:42:44 -- STEP: 29908/36938 -- GLOBAL_STEP: 362350 | > loss_text_ce: 0.04633011668920517 (0.040147167446614555) | > loss_mel_ce: 1.2916967868804932 (nan) | > loss: 0.005309630651026964 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1364933185212595) | > loader_time: 0.0045 (0.004728707620544997)  --> TIME: 2025-02-24 14:42:53 -- STEP: 29958/36938 -- GLOBAL_STEP: 362400 | > loss_text_ce: 0.04645158722996712 (0.04014511480972869) | > loss_mel_ce: 3.034315347671509 (nan) | > loss: 0.012225266546010971 (nan) | > current_lr: 5e-06 | > step_time: 0.1458 (0.13649424997888981) | > loader_time: 0.0046 (0.004728729634635494)  --> TIME: 2025-02-24 14:43:02 -- STEP: 30008/36938 -- GLOBAL_STEP: 362450 | > loss_text_ce: 0.03475479781627655 (0.04014184006503607) | > loss_mel_ce: 2.7764978408813477 (nan) | > loss: 0.011155764572322369 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13648955833431892) | > loader_time: 0.0049 (0.004728714956089358)  --> TIME: 2025-02-24 14:43:11 -- STEP: 30058/36938 -- GLOBAL_STEP: 362500 | > loss_text_ce: 0.045416440814733505 (0.04014226089894931) | > loss_mel_ce: 2.6711819171905518 (nan) | > loss: 0.0107801528647542 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.136483343785469) | > loader_time: 0.0046 (0.004728508642852691)  --> TIME: 2025-02-24 14:43:20 -- STEP: 30108/36938 -- GLOBAL_STEP: 362550 | > loss_text_ce: 0.04754439368844032 (0.04014411267669326) | > loss_mel_ce: 1.916783094406128 (nan) | > loss: 0.007794950623065233 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13648383999402877) | > loader_time: 0.0047 (0.004728350567124035)  --> TIME: 2025-02-24 14:43:30 -- STEP: 30158/36938 -- GLOBAL_STEP: 362600 | > loss_text_ce: 0.04167814180254936 (0.040141720081192905) | > loss_mel_ce: 3.346360445022583 (nan) | > loss: 0.013444598764181137 (nan) | > current_lr: 5e-06 | > step_time: 0.1928 (0.1364913890687329) | > loader_time: 0.0057 (0.004728357634894467)  --> TIME: 2025-02-24 14:43:39 -- STEP: 30208/36938 -- GLOBAL_STEP: 362650 | > loss_text_ce: 0.038109246641397476 (0.040140142387919554) | > loss_mel_ce: 2.989147424697876 (nan) | > loss: 0.012012924067676067 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13648497880736282) | > loader_time: 0.0048 (0.004728222076418068)  --> TIME: 2025-02-24 14:43:48 -- STEP: 30258/36938 -- GLOBAL_STEP: 362700 | > loss_text_ce: 0.037177834659814835 (0.04014040233079333) | > loss_mel_ce: 3.2206296920776367 (nan) | > loss: 0.012927807867527008 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.136485908672698) | > loader_time: 0.0046 (0.00472822748934716)  --> TIME: 2025-02-24 14:43:57 -- STEP: 30308/36938 -- GLOBAL_STEP: 362750 | > loss_text_ce: 0.04075009748339653 (0.040141406585109146) | > loss_mel_ce: 2.8168225288391113 (nan) | > loss: 0.011339574120938778 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13648120325977492) | > loader_time: 0.0045 (0.004728023037048952)  --> TIME: 2025-02-24 14:44:07 -- STEP: 30358/36938 -- GLOBAL_STEP: 362800 | > loss_text_ce: 0.04326257482171059 (0.040143223573234264) | > loss_mel_ce: 3.014353036880493 (nan) | > loss: 0.012133395299315453 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13648871433374302) | > loader_time: 0.0046 (0.004728046893853618)  --> TIME: 2025-02-24 14:44:17 -- STEP: 30408/36938 -- GLOBAL_STEP: 362850 | > loss_text_ce: 0.03678365796804428 (0.04014577384307622) | > loss_mel_ce: 1.3881767988204956 (nan) | > loss: 0.005654605571180582 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13649318359990095) | > loader_time: 0.0046 (0.004728104661433737)  --> TIME: 2025-02-24 14:44:26 -- STEP: 30458/36938 -- GLOBAL_STEP: 362900 | > loss_text_ce: 0.034779325127601624 (0.040143433854054215) | > loss_mel_ce: 3.39635968208313 (nan) | > loss: 0.013615631498396397 (nan) | > current_lr: 5e-06 | > step_time: 0.1505 (0.13648863167981506) | > loader_time: 0.0047 (0.004728241315602865)  --> TIME: 2025-02-24 14:44:35 -- STEP: 30508/36938 -- GLOBAL_STEP: 362950 | > loss_text_ce: 0.033031366765499115 (0.04014287490560232) | > loss_mel_ce: 2.1031601428985596 (nan) | > loss: 0.008476951159536839 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13648753500991193) | > loader_time: 0.0045 (0.004728182444991987)  --> TIME: 2025-02-24 14:44:44 -- STEP: 30558/36938 -- GLOBAL_STEP: 363000 | > loss_text_ce: 0.05431309714913368 (0.04014505283602053) | > loss_mel_ce: 4.116643905639648 (nan) | > loss: 0.016551418229937553 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13649223740870509) | > loader_time: 0.0045 (0.004728108482590567)  --> TIME: 2025-02-24 14:44:54 -- STEP: 30608/36938 -- GLOBAL_STEP: 363050 | > loss_text_ce: 0.03472592681646347 (0.04014271382908572) | > loss_mel_ce: 3.207512140274048 (nan) | > loss: 0.012866024859249592 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13649335331431883) | > loader_time: 0.0045 (0.004727915054020712)  --> TIME: 2025-02-24 14:45:03 -- STEP: 30658/36938 -- GLOBAL_STEP: 363100 | > loss_text_ce: 0.03837674856185913 (0.04014287313955062) | > loss_mel_ce: 3.629774808883667 (nan) | > loss: 0.0145561583340168 (nan) | > current_lr: 5e-06 | > step_time: 0.1578 (0.1364916065946318) | > loader_time: 0.0047 (0.004727731339579546)  --> TIME: 2025-02-24 14:45:12 -- STEP: 30708/36938 -- GLOBAL_STEP: 363150 | > loss_text_ce: 0.03505950793623924 (0.04014248902166063) | > loss_mel_ce: 3.0640037059783936 (nan) | > loss: 0.012297870591282845 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.1364913444834563) | > loader_time: 0.0046 (0.004727765220954174)  --> TIME: 2025-02-24 14:45:22 -- STEP: 30758/36938 -- GLOBAL_STEP: 363200 | > loss_text_ce: 0.04026694595813751 (0.040141049954629526) | > loss_mel_ce: 2.75520658493042 (nan) | > loss: 0.01109314989298582 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13649423629814703) | > loader_time: 0.0047 (0.0047277322678380565)  --> TIME: 2025-02-24 14:45:31 -- STEP: 30808/36938 -- GLOBAL_STEP: 363250 | > loss_text_ce: 0.040310971438884735 (0.04014353041091047) | > loss_mel_ce: 0.011199951171875 (nan) | > loss: 0.00020440843945834786 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13649637586604163) | > loader_time: 0.0045 (0.004727804376007365)  --> TIME: 2025-02-24 14:45:40 -- STEP: 30858/36938 -- GLOBAL_STEP: 363300 | > loss_text_ce: 0.041707638651132584 (0.04014418785721613) | > loss_mel_ce: 2.9259109497070312 (nan) | > loss: 0.011776265688240528 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13648538565834908) | > loader_time: 0.0046 (0.004727683540790294)  --> TIME: 2025-02-24 14:45:49 -- STEP: 30908/36938 -- GLOBAL_STEP: 363350 | > loss_text_ce: 0.04266076162457466 (0.04014507806435664) | > loss_mel_ce: 3.4554412364959717 (nan) | > loss: 0.01388135738670826 (nan) | > current_lr: 5e-06 | > step_time: 0.1815 (0.13648766010008342) | > loader_time: 0.0048 (0.004727761457264802)  --> TIME: 2025-02-24 14:45:59 -- STEP: 30958/36938 -- GLOBAL_STEP: 363400 | > loss_text_ce: 0.026041259989142418 (0.040145932210280816) | > loss_mel_ce: 2.0065314769744873 (nan) | > loss: 0.008065765723586082 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.1364889957544535) | > loader_time: 0.0053 (0.004727756709844277)  --> TIME: 2025-02-24 14:46:08 -- STEP: 31008/36938 -- GLOBAL_STEP: 363450 | > loss_text_ce: 0.0469249002635479 (0.040144667882162206) | > loss_mel_ce: 3.4358088970184326 (nan) | > loss: 0.013820372521877289 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.13648925675511617) | > loader_time: 0.0048 (0.004727871756004999)  --> TIME: 2025-02-24 14:46:18 -- STEP: 31058/36938 -- GLOBAL_STEP: 363500 | > loss_text_ce: 0.0351414829492569 (0.040146049143237696) | > loss_mel_ce: 3.1998274326324463 (nan) | > loss: 0.012837179005146027 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.13649722315998003) | > loader_time: 0.0048 (0.004727774336084902)  --> TIME: 2025-02-24 14:46:27 -- STEP: 31108/36938 -- GLOBAL_STEP: 363550 | > loss_text_ce: 0.02791740372776985 (0.04014365389900384) | > loss_mel_ce: 1.9182623624801636 (nan) | > loss: 0.007722936104983091 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.13650237864020504) | > loader_time: 0.0052 (0.004727707004829194)  --> TIME: 2025-02-24 14:46:36 -- STEP: 31158/36938 -- GLOBAL_STEP: 363600 | > loss_text_ce: 0.04047010838985443 (0.040141391719997845) | > loss_mel_ce: 3.346409797668457 (nan) | > loss: 0.01344000082463026 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.1365020879923854) | > loader_time: 0.0046 (0.004727681952283765)  --> TIME: 2025-02-24 14:46:46 -- STEP: 31208/36938 -- GLOBAL_STEP: 363650 | > loss_text_ce: 0.031181268393993378 (0.040143197569804064) | > loss_mel_ce: 2.5782554149627686 (nan) | > loss: 0.010354908183217049 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13650321809183796) | > loader_time: 0.0046 (0.004727635268094295)  --> TIME: 2025-02-24 14:46:55 -- STEP: 31258/36938 -- GLOBAL_STEP: 363700 | > loss_text_ce: 0.03831033036112785 (0.04014669883519189) | > loss_mel_ce: 2.887579917907715 (nan) | > loss: 0.011610676534473896 (nan) | > current_lr: 5e-06 | > step_time: 0.1503 (0.1365042323228353) | > loader_time: 0.0048 (0.00472757125878659)  --> TIME: 2025-02-24 14:47:04 -- STEP: 31308/36938 -- GLOBAL_STEP: 363750 | > loss_text_ce: 0.03711285442113876 (0.04014527449458474) | > loss_mel_ce: 3.3696444034576416 (nan) | > loss: 0.013518879190087318 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13650621644561323) | > loader_time: 0.0046 (0.004727429176665425)  --> TIME: 2025-02-24 14:47:14 -- STEP: 31358/36938 -- GLOBAL_STEP: 363800 | > loss_text_ce: 0.052438147366046906 (0.04014295674682886) | > loss_mel_ce: 0.9729708433151245 (nan) | > loss: 0.004069083370268345 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13650934049476168) | > loader_time: 0.0046 (0.004727402157044212)  --> TIME: 2025-02-24 14:47:23 -- STEP: 31408/36938 -- GLOBAL_STEP: 363850 | > loss_text_ce: 0.03403114154934883 (0.04014299443206667) | > loss_mel_ce: 3.29064679145813 (nan) | > loss: 0.013193166814744473 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13650837322764303) | > loader_time: 0.0047 (0.004727394208577751)  --> TIME: 2025-02-24 14:47:33 -- STEP: 31458/36938 -- GLOBAL_STEP: 363900 | > loss_text_ce: 0.03973400965332985 (0.04014322979854628) | > loss_mel_ce: 0.02044677734375 (nan) | > loss: 0.0002388126595178619 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13651055259390407) | > loader_time: 0.0046 (0.00472727578430111)  --> TIME: 2025-02-24 14:47:42 -- STEP: 31508/36938 -- GLOBAL_STEP: 363950 | > loss_text_ce: 0.04114029183983803 (0.04014362811229869) | > loss_mel_ce: 2.9953696727752686 (nan) | > loss: 0.012049643322825432 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.13651954044132722) | > loader_time: 0.0056 (0.004727239140826222)  --> TIME: 2025-02-24 14:47:51 -- STEP: 31558/36938 -- GLOBAL_STEP: 364000 | > loss_text_ce: 0.02820521779358387 (0.040142671648690245) | > loss_mel_ce: 2.2617223262786865 (nan) | > loss: 0.009087014012038708 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13651469931024265) | > loader_time: 0.0045 (0.004727187276951812)  --> TIME: 2025-02-24 14:48:00 -- STEP: 31608/36938 -- GLOBAL_STEP: 364050 | > loss_text_ce: 0.03834902122616768 (0.04014248814432294) | > loss_mel_ce: 2.4834041595458984 (nan) | > loss: 0.01000695675611496 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13650466051019508) | > loader_time: 0.0045 (0.004726921047207345)  --> TIME: 2025-02-24 14:48:10 -- STEP: 31658/36938 -- GLOBAL_STEP: 364100 | > loss_text_ce: 0.044146452099084854 (0.04014364465438885) | > loss_mel_ce: 3.723560094833374 (nan) | > loss: 0.014951217919588089 (nan) | > current_lr: 5e-06 | > step_time: 0.1874 (0.13650367039921305) | > loader_time: 0.0049 (0.004726808885546296)  --> TIME: 2025-02-24 14:48:19 -- STEP: 31708/36938 -- GLOBAL_STEP: 364150 | > loss_text_ce: 0.039383415132761 (0.04014284710032789) | > loss_mel_ce: 3.272876501083374 (nan) | > loss: 0.013143889605998993 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.136502235823325) | > loader_time: 0.0045 (0.00472657940224412)  --> TIME: 2025-02-24 14:48:28 -- STEP: 31758/36938 -- GLOBAL_STEP: 364200 | > loss_text_ce: 0.0450332909822464 (0.040143362174166274) | > loss_mel_ce: 1.0274434089660645 (nan) | > loss: 0.004255860112607479 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13650252952091502) | > loader_time: 0.0046 (0.004726406256027536)  --> TIME: 2025-02-24 14:48:38 -- STEP: 31808/36938 -- GLOBAL_STEP: 364250 | > loss_text_ce: 0.04482020437717438 (0.040142183267644835) | > loss_mel_ce: 3.8447248935699463 (nan) | > loss: 0.015434703789651394 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13650944804251686) | > loader_time: 0.0047 (0.004726306413500872)  --> TIME: 2025-02-24 14:48:47 -- STEP: 31858/36938 -- GLOBAL_STEP: 364300 | > loss_text_ce: 0.03366278484463692 (0.04014103728483294) | > loss_mel_ce: 3.597628116607666 (nan) | > loss: 0.014409885741770267 (nan) | > current_lr: 5e-06 | > step_time: 0.1959 (0.13651468590817645) | > loader_time: 0.0048 (0.00472625338864334)  --> TIME: 2025-02-24 14:48:57 -- STEP: 31908/36938 -- GLOBAL_STEP: 364350 | > loss_text_ce: 0.03460024669766426 (0.040141557309825944) | > loss_mel_ce: 1.9830118417739868 (nan) | > loss: 0.008006397634744644 (nan) | > current_lr: 5e-06 | > step_time: 0.1577 (0.13651935230359985) | > loader_time: 0.0046 (0.004726228460536918)  --> TIME: 2025-02-24 14:49:06 -- STEP: 31958/36938 -- GLOBAL_STEP: 364400 | > loss_text_ce: 0.033244770020246506 (0.04014195235675981) | > loss_mel_ce: 2.0509705543518066 (nan) | > loss: 0.008270696736872196 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.1365210053783315) | > loader_time: 0.0046 (0.0047262312884145735)  --> TIME: 2025-02-24 14:49:15 -- STEP: 32008/36938 -- GLOBAL_STEP: 364450 | > loss_text_ce: 0.04655905067920685 (0.04014157716408213) | > loss_mel_ce: 3.561924695968628 (nan) | > loss: 0.01431938074529171 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.1365208268627398) | > loader_time: 0.0046 (0.004726353003900697)  --> TIME: 2025-02-24 14:49:25 -- STEP: 32058/36938 -- GLOBAL_STEP: 364500 | > loss_text_ce: 0.04421944543719292 (0.04014229838890281) | > loss_mel_ce: 3.122468948364258 (nan) | > loss: 0.012566224671900272 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.1365177255655828) | > loader_time: 0.0046 (0.00472616998379883)  --> TIME: 2025-02-24 14:49:34 -- STEP: 32108/36938 -- GLOBAL_STEP: 364550 | > loss_text_ce: 0.04006975144147873 (0.040140803134597114) | > loss_mel_ce: 3.439899206161499 (nan) | > loss: 0.013809401541948318 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13652192305490754) | > loader_time: 0.0046 (0.004726026413730924)  --> TIME: 2025-02-24 14:49:43 -- STEP: 32158/36938 -- GLOBAL_STEP: 364600 | > loss_text_ce: 0.035598184913396835 (0.04014008913608654) | > loss_mel_ce: 3.6380796432495117 (nan) | > loss: 0.01457808818668127 (nan) | > current_lr: 5e-06 | > step_time: 0.1824 (0.13651377288505798) | > loader_time: 0.0048 (0.004725925623907232)  --> TIME: 2025-02-24 14:49:52 -- STEP: 32208/36938 -- GLOBAL_STEP: 364650 | > loss_text_ce: 0.035723090171813965 (0.04014002045653417) | > loss_mel_ce: 2.864346504211426 (nan) | > loss: 0.011508213356137276 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13651328134524793) | > loader_time: 0.0045 (0.004725731550255553)  --> TIME: 2025-02-24 14:50:02 -- STEP: 32258/36938 -- GLOBAL_STEP: 364700 | > loss_text_ce: 0.04422562196850777 (0.04013804663208866) | > loss_mel_ce: 0.2603929936885834 (nan) | > loss: 0.0012088041985407472 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13651410145623186) | > loader_time: 0.0046 (0.004725537878676648)  --> TIME: 2025-02-24 14:50:11 -- STEP: 32308/36938 -- GLOBAL_STEP: 364750 | > loss_text_ce: 0.04191116243600845 (0.04013926997911454) | > loss_mel_ce: 2.475107192993164 (nan) | > loss: 0.009988168254494667 (nan) | > current_lr: 5e-06 | > step_time: 0.107 (0.13650545715857584) | > loader_time: 0.0044 (0.0047254688568276115)  --> TIME: 2025-02-24 14:50:20 -- STEP: 32358/36938 -- GLOBAL_STEP: 364800 | > loss_text_ce: 0.04684441536664963 (0.0401390942300981) | > loss_mel_ce: 3.056887626647949 (nan) | > loss: 0.012316398322582245 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.13650165062338437) | > loader_time: 0.0046 (0.004725385230936988)  --> TIME: 2025-02-24 14:50:29 -- STEP: 32408/36938 -- GLOBAL_STEP: 364850 | > loss_text_ce: 0.027194706723093987 (0.04014028657862405) | > loss_mel_ce: 3.2947754859924316 (nan) | > loss: 0.013182422146201134 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13650308663066313) | > loader_time: 0.0045 (0.0047253543831513155)  --> TIME: 2025-02-24 14:50:39 -- STEP: 32458/36938 -- GLOBAL_STEP: 364900 | > loss_text_ce: 0.049261827021837234 (0.04013974105927116) | > loss_mel_ce: 3.2188477516174316 (nan) | > loss: 0.012968689203262329 (nan) | > current_lr: 5e-06 | > step_time: 0.1854 (0.1365073909967534) | > loader_time: 0.0049 (0.004725315161102191)  --> TIME: 2025-02-24 14:50:48 -- STEP: 32508/36938 -- GLOBAL_STEP: 364950 | > loss_text_ce: 0.03683524951338768 (0.04013678287105517) | > loss_mel_ce: 2.484886884689331 (nan) | > loss: 0.01000683382153511 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13651950312112443) | > loader_time: 0.0045 (0.004725429211446745)  --> TIME: 2025-02-24 14:50:58 -- STEP: 32558/36938 -- GLOBAL_STEP: 365000 | > loss_text_ce: 0.03817279636859894 (0.040135770623692266) | > loss_mel_ce: 1.3668086528778076 (nan) | > loss: 0.005575323943048716 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13652659438665243) | > loader_time: 0.0054 (0.004725532805907228)  --> TIME: 2025-02-24 14:51:07 -- STEP: 32608/36938 -- GLOBAL_STEP: 365050 | > loss_text_ce: 0.02925863489508629 (0.04013626583311167) | > loss_mel_ce: 2.2239582538604736 (nan) | > loss: 0.00894133746623993 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13652962144972308) | > loader_time: 0.0047 (0.004725676859793913)  --> TIME: 2025-02-24 14:51:16 -- STEP: 32658/36938 -- GLOBAL_STEP: 365100 | > loss_text_ce: 0.030717773362994194 (0.0401361115076109) | > loss_mel_ce: 3.271484375 (nan) | > loss: 0.013103977777063847 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13652334003587382) | > loader_time: 0.0045 (0.004725672813381449)  --> TIME: 2025-02-24 14:51:26 -- STEP: 32708/36938 -- GLOBAL_STEP: 365150 | > loss_text_ce: 0.03614261746406555 (0.04013729558606043) | > loss_mel_ce: 2.9558205604553223 (nan) | > loss: 0.011872869916260242 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13652638959989505) | > loader_time: 0.0046 (0.004725605041659752)  --> TIME: 2025-02-24 14:51:35 -- STEP: 32758/36938 -- GLOBAL_STEP: 365200 | > loss_text_ce: 0.038283053785562515 (0.04013664993061815) | > loss_mel_ce: 3.456102132797241 (nan) | > loss: 0.013866608962416649 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.1365209031551911) | > loader_time: 0.0047 (0.004725672123130727)  --> TIME: 2025-02-24 14:51:44 -- STEP: 32808/36938 -- GLOBAL_STEP: 365250 | > loss_text_ce: 0.03417886048555374 (0.04013661464512878) | > loss_mel_ce: 2.8070452213287354 (nan) | > loss: 0.011274700053036213 (nan) | > current_lr: 5e-06 | > step_time: 0.15 (0.13652586125943708) | > loader_time: 0.0047 (0.004725948924467204)  --> TIME: 2025-02-24 14:51:54 -- STEP: 32858/36938 -- GLOBAL_STEP: 365300 | > loss_text_ce: 0.04460916295647621 (0.04013674435108777) | > loss_mel_ce: 3.335703134536743 (nan) | > loss: 0.013413937762379646 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.1365301940059309) | > loader_time: 0.0046 (0.00472593766076111)  --> TIME: 2025-02-24 14:52:03 -- STEP: 32908/36938 -- GLOBAL_STEP: 365350 | > loss_text_ce: 0.034861668944358826 (0.04013629377773423) | > loss_mel_ce: 2.806288719177246 (nan) | > loss: 0.011274406686425209 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13652513177017733) | > loader_time: 0.0045 (0.004726074055476628)  --> TIME: 2025-02-24 14:52:12 -- STEP: 32958/36938 -- GLOBAL_STEP: 365400 | > loss_text_ce: 0.03027508035302162 (0.04013626921457534) | > loss_mel_ce: 2.0552875995635986 (nan) | > loss: 0.008276042528450489 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.1365203457742682) | > loader_time: 0.0046 (0.0047260643288713185)  --> TIME: 2025-02-24 14:52:21 -- STEP: 33008/36938 -- GLOBAL_STEP: 365450 | > loss_text_ce: 0.03551182523369789 (0.040135491969576616) | > loss_mel_ce: 3.2981953620910645 (nan) | > loss: 0.013228997588157654 (nan) | > current_lr: 5e-06 | > step_time: 0.1941 (0.13651672450593427) | > loader_time: 0.0057 (0.0047260637761202765)  --> TIME: 2025-02-24 14:52:30 -- STEP: 33058/36938 -- GLOBAL_STEP: 365500 | > loss_text_ce: 0.03260858356952667 (0.04013388315054573) | > loss_mel_ce: 3.547886371612549 (nan) | > loss: 0.014208314009010792 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13651283396459357) | > loader_time: 0.0046 (0.0047261230352317685)  --> TIME: 2025-02-24 14:52:40 -- STEP: 33108/36938 -- GLOBAL_STEP: 365550 | > loss_text_ce: 0.045852746814489365 (0.04013115243784965) | > loss_mel_ce: 2.811983585357666 (nan) | > loss: 0.011340620927512646 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13651047063879032) | > loader_time: 0.0048 (0.004726190540804127)  --> TIME: 2025-02-24 14:52:49 -- STEP: 33158/36938 -- GLOBAL_STEP: 365600 | > loss_text_ce: 0.037728238850831985 (0.04013048621603266) | > loss_mel_ce: 3.316230535507202 (nan) | > loss: 0.013309361413121223 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13651798631279252) | > loader_time: 0.0046 (0.004726224299671957)  --> TIME: 2025-02-24 14:52:59 -- STEP: 33208/36938 -- GLOBAL_STEP: 365650 | > loss_text_ce: 0.03595176711678505 (0.040129709451095334) | > loss_mel_ce: 3.606123924255371 (nan) | > loss: 0.014452682808041573 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13652265826439763) | > loader_time: 0.0049 (0.004726137649119865)  --> TIME: 2025-02-24 14:53:08 -- STEP: 33258/36938 -- GLOBAL_STEP: 365700 | > loss_text_ce: 0.029566247016191483 (0.04012870635459625) | > loss_mel_ce: 3.1705965995788574 (nan) | > loss: 0.012699060142040253 (nan) | > current_lr: 5e-06 | > step_time: 0.1756 (0.13652990282144528) | > loader_time: 0.0048 (0.0047261355565426)  --> TIME: 2025-02-24 14:53:18 -- STEP: 33308/36938 -- GLOBAL_STEP: 365750 | > loss_text_ce: 0.0438111387193203 (0.04012857772387782) | > loss_mel_ce: 3.049295425415039 (nan) | > loss: 0.012274232693016529 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13653249448267868) | > loader_time: 0.0047 (0.0047260345753550575)  --> TIME: 2025-02-24 14:53:28 -- STEP: 33358/36938 -- GLOBAL_STEP: 365800 | > loss_text_ce: 0.04751548171043396 (0.04012820268526369) | > loss_mel_ce: 3.5535976886749268 (nan) | > loss: 0.01429013255983591 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.13654226740456132) | > loader_time: 0.0047 (0.004726191584501758)  --> TIME: 2025-02-24 14:53:37 -- STEP: 33408/36938 -- GLOBAL_STEP: 365850 | > loss_text_ce: 0.03746452182531357 (0.040127378197184455) | > loss_mel_ce: 3.5829741954803467 (nan) | > loss: 0.014366821385920048 (nan) | > current_lr: 5e-06 | > step_time: 0.165 (0.13653809296045621) | > loader_time: 0.0046 (0.004726072623469392)  --> TIME: 2025-02-24 14:53:46 -- STEP: 33458/36938 -- GLOBAL_STEP: 365900 | > loss_text_ce: 0.04017471522092819 (0.04012629567601575) | > loss_mel_ce: 3.4117136001586914 (nan) | > loss: 0.013697970658540726 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13654069901510224) | > loader_time: 0.0046 (0.004726169868823621)  --> TIME: 2025-02-24 14:53:55 -- STEP: 33508/36938 -- GLOBAL_STEP: 365950 | > loss_text_ce: 0.05039786547422409 (0.04012548695511857) | > loss_mel_ce: 0.3984375 (nan) | > loss: 0.0017810928402468562 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.13653871041003302) | > loader_time: 0.0045 (0.00472622630248084)  --> TIME: 2025-02-24 14:54:05 -- STEP: 33558/36938 -- GLOBAL_STEP: 366000 | > loss_text_ce: 0.03710215538740158 (0.040123456878784956) | > loss_mel_ce: 3.4797840118408203 (nan) | > loss: 0.013955898582935333 (nan) | > current_lr: 5e-06 | > step_time: 0.1499 (0.1365382739774182) | > loader_time: 0.0047 (0.004726417244130782)  --> TIME: 2025-02-24 14:54:14 -- STEP: 33608/36938 -- GLOBAL_STEP: 366050 | > loss_text_ce: 0.024952441453933716 (0.040121451652918395) | > loss_mel_ce: 2.316823720932007 (nan) | > loss: 0.009292762726545334 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13653671344215784) | > loader_time: 0.0049 (0.0047265825399868035)  --> TIME: 2025-02-24 14:54:23 -- STEP: 33658/36938 -- GLOBAL_STEP: 366100 | > loss_text_ce: 0.03398881480097771 (0.040119490841866015) | > loss_mel_ce: 3.1379473209381104 (nan) | > loss: 0.0125870481133461 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.13653384507945485) | > loader_time: 0.0047 (0.00472658021514222)  --> TIME: 2025-02-24 14:54:33 -- STEP: 33708/36938 -- GLOBAL_STEP: 366150 | > loss_text_ce: 0.047488730400800705 (0.04011951058183415) | > loss_mel_ce: 2.754671573638916 (nan) | > loss: 0.01111968420445919 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13653698125399807) | > loader_time: 0.0047 (0.004726509437083485)  --> TIME: 2025-02-24 14:54:42 -- STEP: 33758/36938 -- GLOBAL_STEP: 366200 | > loss_text_ce: 0.04214637726545334 (0.04011890501680769) | > loss_mel_ce: 2.9607081413269043 (nan) | > loss: 0.011916090734302998 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13654655432473592) | > loader_time: 0.0046 (0.004726546460033098)  --> TIME: 2025-02-24 14:54:51 -- STEP: 33808/36938 -- GLOBAL_STEP: 366250 | > loss_text_ce: 0.046397414058446884 (0.0401181391195657) | > loss_mel_ce: 2.8653972148895264 (nan) | > loss: 0.01155474130064249 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13654024192381856) | > loader_time: 0.0046 (0.004726773379145805)  --> TIME: 2025-02-24 14:55:01 -- STEP: 33858/36938 -- GLOBAL_STEP: 366300 | > loss_text_ce: 0.04567393660545349 (0.04011721817523069) | > loss_mel_ce: 2.929380178451538 (nan) | > loss: 0.011805770918726921 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13654306930041862) | > loader_time: 0.0045 (0.004726833894112106)  --> TIME: 2025-02-24 14:55:10 -- STEP: 33908/36938 -- GLOBAL_STEP: 366350 | > loss_text_ce: 0.03841875493526459 (0.04011581424542768) | > loss_mel_ce: 3.4396002292633057 (nan) | > loss: 0.013801663182675838 (nan) | > current_lr: 5e-06 | > step_time: 0.1868 (0.13654473059382885) | > loader_time: 0.0051 (0.004727069465594028)  --> TIME: 2025-02-24 14:55:20 -- STEP: 33958/36938 -- GLOBAL_STEP: 366400 | > loss_text_ce: 0.0376262441277504 (0.04011734699175717) | > loss_mel_ce: 1.7170708179473877 (nan) | > loss: 0.006963083986192942 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13654798968912812) | > loader_time: 0.0053 (0.004727102588361739)  --> TIME: 2025-02-24 14:55:29 -- STEP: 34008/36938 -- GLOBAL_STEP: 366450 | > loss_text_ce: 0.0412549190223217 (0.04011749520112354) | > loss_mel_ce: 2.7266616821289062 (nan) | > loss: 0.010983796790242195 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.1365461390920304) | > loader_time: 0.0046 (0.004727101317575999)  --> TIME: 2025-02-24 14:55:38 -- STEP: 34058/36938 -- GLOBAL_STEP: 366500 | > loss_text_ce: 0.046513061970472336 (0.04011862641717806) | > loss_mel_ce: 2.974301815032959 (nan) | > loss: 0.011987361125648022 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13655085819593063) | > loader_time: 0.0049 (0.004727026518641914)  --> TIME: 2025-02-24 14:55:47 -- STEP: 34108/36938 -- GLOBAL_STEP: 366550 | > loss_text_ce: 0.03166720271110535 (0.040119272805906665) | > loss_mel_ce: 1.1379190683364868 (nan) | > loss: 0.004641215782612562 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13654408621119665) | > loader_time: 0.0046 (0.004726934037343349)  --> TIME: 2025-02-24 14:55:57 -- STEP: 34158/36938 -- GLOBAL_STEP: 366600 | > loss_text_ce: 0.037372175604104996 (0.04011540149247326) | > loss_mel_ce: 2.767011880874634 (nan) | > loss: 0.011128508485853672 (nan) | > current_lr: 5e-06 | > step_time: 0.1837 (0.136544559897998) | > loader_time: 0.0049 (0.0047269051063395335)  --> TIME: 2025-02-24 14:56:06 -- STEP: 34208/36938 -- GLOBAL_STEP: 366650 | > loss_text_ce: 0.03623725846409798 (0.040114423236706324) | > loss_mel_ce: 3.312530994415283 (nan) | > loss: 0.013288763351738453 (nan) | > current_lr: 5e-06 | > step_time: 0.1451 (0.1365423815079211) | > loader_time: 0.0046 (0.0047268082498389185)  --> TIME: 2025-02-24 14:56:15 -- STEP: 34258/36938 -- GLOBAL_STEP: 366700 | > loss_text_ce: 0.031986113637685776 (0.040113501575638484) | > loss_mel_ce: 3.15293025970459 (nan) | > loss: 0.012638557702302933 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13654007256932188) | > loader_time: 0.0051 (0.0047267794894409595)  --> TIME: 2025-02-24 14:56:24 -- STEP: 34308/36938 -- GLOBAL_STEP: 366750 | > loss_text_ce: 0.03160669282078743 (0.040111958966525973) | > loss_mel_ce: 2.537753105163574 (nan) | > loss: 0.010195872746407986 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13653636019699839) | > loader_time: 0.0047 (0.004726838138507406)  --> TIME: 2025-02-24 14:56:33 -- STEP: 34358/36938 -- GLOBAL_STEP: 366800 | > loss_text_ce: 0.015071652829647064 (0.040112339187078745) | > loss_mel_ce: 1.9359984397888184 (nan) | > loss: 0.007742342073470354 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.1365344328349863) | > loader_time: 0.0045 (0.004726839104411555)  --> TIME: 2025-02-24 14:56:43 -- STEP: 34408/36938 -- GLOBAL_STEP: 366850 | > loss_text_ce: 0.030315769836306572 (0.04011321980859688) | > loss_mel_ce: 0.1769932359457016 (nan) | > loss: 0.000822654867079109 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13654213136252893) | > loader_time: 0.0046 (0.00472691540134923)  --> TIME: 2025-02-24 14:56:53 -- STEP: 34458/36938 -- GLOBAL_STEP: 366900 | > loss_text_ce: 0.04552885517477989 (0.04011179296683836) | > loss_mel_ce: 2.957509756088257 (nan) | > loss: 0.011916820891201496 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13654841201982929) | > loader_time: 0.0045 (0.004726930311963467)  --> TIME: 2025-02-24 14:57:02 -- STEP: 34508/36938 -- GLOBAL_STEP: 366950 | > loss_text_ce: 0.04181181266903877 (0.04011160037265476) | > loss_mel_ce: 3.3338403701782227 (nan) | > loss: 0.013395445421338081 (nan) | > current_lr: 5e-06 | > step_time: 0.1599 (0.13654380573849853) | > loader_time: 0.0054 (0.004726836458064485)  --> TIME: 2025-02-24 14:57:11 -- STEP: 34558/36938 -- GLOBAL_STEP: 367000 | > loss_text_ce: 0.04067168012261391 (0.04011478963656719) | > loss_mel_ce: 3.150390625 (nan) | > loss: 0.012662946246564388 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13653992438552168) | > loader_time: 0.0045 (0.004726753583129119)  --> TIME: 2025-02-24 14:57:20 -- STEP: 34608/36938 -- GLOBAL_STEP: 367050 | > loss_text_ce: 0.037650346755981445 (0.04011301608876001) | > loss_mel_ce: 3.257288932800293 (nan) | > loss: 0.013075157068669796 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.13654487570760523) | > loader_time: 0.0047 (0.004726875912685749)  --> TIME: 2025-02-24 14:57:30 -- STEP: 34658/36938 -- GLOBAL_STEP: 367100 | > loss_text_ce: 0.04423599690198898 (0.040111656140594636) | > loss_mel_ce: 2.8784983158111572 (nan) | > loss: 0.011598152108490467 (nan) | > current_lr: 5e-06 | > step_time: 0.1444 (0.13654674586796148) | > loader_time: 0.0045 (0.004726901745877406)  --> TIME: 2025-02-24 14:57:39 -- STEP: 34708/36938 -- GLOBAL_STEP: 367150 | > loss_text_ce: 0.041060615330934525 (0.04011276049807895) | > loss_mel_ce: 1.5792397260665894 (nan) | > loss: 0.00642976351082325 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13654838771167083) | > loader_time: 0.0045 (0.004726807484775725)  --> TIME: 2025-02-24 14:57:48 -- STEP: 34758/36938 -- GLOBAL_STEP: 367200 | > loss_text_ce: 0.03237959370017052 (0.040112086543270586) | > loss_mel_ce: 3.5110929012298584 (nan) | > loss: 0.01406139973551035 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.1365428691396245) | > loader_time: 0.0045 (0.0047267701808421155)  --> TIME: 2025-02-24 14:57:58 -- STEP: 34808/36938 -- GLOBAL_STEP: 367250 | > loss_text_ce: 0.022267550230026245 (0.040110096446475285) | > loss_mel_ce: 3.203139066696167 (nan) | > loss: 0.012799233198165894 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1365466877677679) | > loader_time: 0.0046 (0.004726875673571146)  --> TIME: 2025-02-24 14:58:08 -- STEP: 34858/36938 -- GLOBAL_STEP: 367300 | > loss_text_ce: 0.0333239808678627 (0.04010834547352438) | > loss_mel_ce: 2.374169111251831 (nan) | > loss: 0.009553544223308563 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.1365554116803526) | > loader_time: 0.0048 (0.004726939216675956)  --> TIME: 2025-02-24 14:58:17 -- STEP: 34908/36938 -- GLOBAL_STEP: 367350 | > loss_text_ce: 0.04267212003469467 (0.04010683168564074) | > loss_mel_ce: 0.05021476745605469 (nan) | > loss: 0.0003685987903736532 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.1365631847047186) | > loader_time: 0.0045 (0.004726936238812241)  --> TIME: 2025-02-24 14:58:26 -- STEP: 34958/36938 -- GLOBAL_STEP: 367400 | > loss_text_ce: 0.043042223900556564 (0.04010932892792745) | > loss_mel_ce: 3.0049691200256348 (nan) | > loss: 0.012095283716917038 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13655712966126582) | > loader_time: 0.0049 (0.004726902278734526)  --> TIME: 2025-02-24 14:58:35 -- STEP: 35008/36938 -- GLOBAL_STEP: 367450 | > loss_text_ce: 0.031229909509420395 (0.040108073135309966) | > loss_mel_ce: 2.674240827560425 (nan) | > loss: 0.010735996067523956 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13655604209607736) | > loader_time: 0.0048 (0.0047269575910769005)  --> TIME: 2025-02-24 14:58:44 -- STEP: 35058/36938 -- GLOBAL_STEP: 367500 | > loss_text_ce: 0.044889725744724274 (0.04010654639332522) | > loss_mel_ce: 3.153895139694214 (nan) | > loss: 0.01269359141588211 (nan) | > current_lr: 5e-06 | > step_time: 0.1833 (0.136544415686771) | > loader_time: 0.0055 (0.004727090293909983)  --> TIME: 2025-02-24 14:58:54 -- STEP: 35108/36938 -- GLOBAL_STEP: 367550 | > loss_text_ce: 0.027039488777518272 (0.04010653208908439) | > loss_mel_ce: 0.0665690079331398 (nan) | > loss: 0.0003714623162522912 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13654695124781946) | > loader_time: 0.0046 (0.004727111871060278)  --> TIME: 2025-02-24 14:59:03 -- STEP: 35158/36938 -- GLOBAL_STEP: 367600 | > loss_text_ce: 0.026713082566857338 (0.04010453221496615) | > loss_mel_ce: 1.7701572179794312 (nan) | > loss: 0.007130438461899757 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.1365497654171553) | > loader_time: 0.0045 (0.00472705125117021)  --> TIME: 2025-02-24 14:59:12 -- STEP: 35208/36938 -- GLOBAL_STEP: 367650 | > loss_text_ce: 0.046080321073532104 (0.040104211798614744) | > loss_mel_ce: 0.036865234375 (nan) | > loss: 0.0003291490429546684 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.1365496772537162) | > loader_time: 0.0046 (0.004727232641752323)  --> TIME: 2025-02-24 14:59:22 -- STEP: 35258/36938 -- GLOBAL_STEP: 367700 | > loss_text_ce: 0.041064586490392685 (0.040103345862282605) | > loss_mel_ce: 3.9027318954467773 (nan) | > loss: 0.015649987384676933 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13654713060224286) | > loader_time: 0.0046 (0.004727181530058311)  --> TIME: 2025-02-24 14:59:31 -- STEP: 35308/36938 -- GLOBAL_STEP: 367750 | > loss_text_ce: 0.03345707431435585 (0.04010257876049353) | > loss_mel_ce: 0.040771484375 (nan) | > loss: 0.00029455777257680893 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13654151777995727) | > loader_time: 0.0044 (0.0047271381327178945)  --> TIME: 2025-02-24 14:59:40 -- STEP: 35358/36938 -- GLOBAL_STEP: 367800 | > loss_text_ce: 0.04696110635995865 (0.0401052340045283) | > loss_mel_ce: 2.2880873680114746 (nan) | > loss: 0.009266065433621407 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13653992025336487) | > loader_time: 0.0045 (0.0047271392000052875)  --> TIME: 2025-02-24 14:59:49 -- STEP: 35408/36938 -- GLOBAL_STEP: 367850 | > loss_text_ce: 0.03821685537695885 (0.04010356798894513) | > loss_mel_ce: 3.056472063064575 (nan) | > loss: 0.012280512601137161 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.13653722465361362) | > loader_time: 0.0047 (0.004727124898508048)  --> TIME: 2025-02-24 14:59:58 -- STEP: 35458/36938 -- GLOBAL_STEP: 367900 | > loss_text_ce: 0.035616692155599594 (0.04010189342041933) | > loss_mel_ce: 2.810223340988159 (nan) | > loss: 0.011293016374111176 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13653730000663547) | > loader_time: 0.0045 (0.0047270792162259255)  --> TIME: 2025-02-24 15:00:08 -- STEP: 35508/36938 -- GLOBAL_STEP: 367950 | > loss_text_ce: 0.03992002084851265 (0.040102279145353324) | > loss_mel_ce: 3.0849711894989014 (nan) | > loss: 0.012400362640619278 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.1365350512444616) | > loader_time: 0.0048 (0.004727039927228769)  --> TIME: 2025-02-24 15:00:17 -- STEP: 35558/36938 -- GLOBAL_STEP: 368000 | > loss_text_ce: 0.04558632895350456 (0.04010376438307413) | > loss_mel_ce: 3.5181760787963867 (nan) | > loss: 0.014141915366053581 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13653689926953422) | > loader_time: 0.0046 (0.004726950869771343)  --> TIME: 2025-02-24 15:00:26 -- STEP: 35608/36938 -- GLOBAL_STEP: 368050 | > loss_text_ce: 0.04100190848112106 (0.04010302218761521) | > loss_mel_ce: 3.5600154399871826 (nan) | > loss: 0.014289751648902893 (nan) | > current_lr: 5e-06 | > step_time: 0.1827 (0.13653936524172486) | > loader_time: 0.0049 (0.00472692602592973)  --> TIME: 2025-02-24 15:00:36 -- STEP: 35658/36938 -- GLOBAL_STEP: 368100 | > loss_text_ce: 0.03233930841088295 (0.04010393332804509) | > loss_mel_ce: 0.347004771232605 (nan) | > loss: 0.0015053336974233389 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1365358157395647) | > loader_time: 0.0045 (0.004726902682619838)  --> TIME: 2025-02-24 15:00:45 -- STEP: 35708/36938 -- GLOBAL_STEP: 368150 | > loss_text_ce: 0.03663456067442894 (0.04010166891365316) | > loss_mel_ce: 2.6910877227783203 (nan) | > loss: 0.01082429476082325 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13652932541066048) | > loader_time: 0.0046 (0.00472679114280311)  --> TIME: 2025-02-24 15:00:54 -- STEP: 35758/36938 -- GLOBAL_STEP: 368200 | > loss_text_ce: 0.038695916533470154 (0.04009978121139381) | > loss_mel_ce: 3.5023884773254395 (nan) | > loss: 0.014051922596991062 (nan) | > current_lr: 5e-06 | > step_time: 0.1578 (0.13652925444259892) | > loader_time: 0.0047 (0.004726718886799135)  --> TIME: 2025-02-24 15:01:03 -- STEP: 35808/36938 -- GLOBAL_STEP: 368250 | > loss_text_ce: 0.04102255776524544 (0.04009969079239892) | > loss_mel_ce: 3.5009610652923584 (nan) | > loss: 0.014055491425096989 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13653182775660916) | > loader_time: 0.0066 (0.004726662239770596)  --> TIME: 2025-02-24 15:01:12 -- STEP: 35858/36938 -- GLOBAL_STEP: 368300 | > loss_text_ce: 0.03357720747590065 (0.04009597579317429) | > loss_mel_ce: 2.284693479537964 (nan) | > loss: 0.009199487045407295 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13652638878663387) | > loader_time: 0.0047 (0.004726589148252987)  --> TIME: 2025-02-24 15:01:22 -- STEP: 35908/36938 -- GLOBAL_STEP: 368350 | > loss_text_ce: 0.03640113025903702 (0.040093677451443446) | > loss_mel_ce: 3.3819680213928223 (nan) | > loss: 0.013564957305788994 (nan) | > current_lr: 5e-06 | > step_time: 0.1825 (0.13652642814064103) | > loader_time: 0.0048 (0.0047265689596376285)  --> TIME: 2025-02-24 15:01:31 -- STEP: 35958/36938 -- GLOBAL_STEP: 368400 | > loss_text_ce: 0.0411313995718956 (0.040096081032202475) | > loss_mel_ce: 2.731943130493164 (nan) | > loss: 0.011004265397787094 (nan) | > current_lr: 5e-06 | > step_time: 0.17 (0.1365265285248033) | > loader_time: 0.0048 (0.004726514368594605)  --> TIME: 2025-02-24 15:01:40 -- STEP: 36008/36938 -- GLOBAL_STEP: 368450 | > loss_text_ce: 0.03782881796360016 (0.040094724622089815) | > loss_mel_ce: 3.346729278564453 (nan) | > loss: 0.01343078725039959 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13652186431081578) | > loader_time: 0.0046 (0.004726498358992854)  --> TIME: 2025-02-24 15:01:50 -- STEP: 36058/36938 -- GLOBAL_STEP: 368500 | > loss_text_ce: 0.040518589317798615 (0.04009409065767111) | > loss_mel_ce: 3.837024688720703 (nan) | > loss: 0.015387076884508133 (nan) | > current_lr: 5e-06 | > step_time: 0.1789 (0.13652251232112572) | > loader_time: 0.0049 (0.004726511976261233)  --> TIME: 2025-02-24 15:01:59 -- STEP: 36108/36938 -- GLOBAL_STEP: 368550 | > loss_text_ce: 0.04589573293924332 (0.040093046123389174) | > loss_mel_ce: 3.5040948390960693 (nan) | > loss: 0.014087265357375145 (nan) | > current_lr: 5e-06 | > step_time: 0.1578 (0.13652028870553376) | > loader_time: 0.0046 (0.004726452580239952)  --> TIME: 2025-02-24 15:02:08 -- STEP: 36158/36938 -- GLOBAL_STEP: 368600 | > loss_text_ce: 0.031098177656531334 (0.04009205779063097) | > loss_mel_ce: 3.9961609840393066 (nan) | > loss: 0.015981188043951988 (nan) | > current_lr: 5e-06 | > step_time: 0.1913 (0.13651886742109204) | > loader_time: 0.0049 (0.004726396262945762)  --> TIME: 2025-02-24 15:02:17 -- STEP: 36208/36938 -- GLOBAL_STEP: 368650 | > loss_text_ce: 0.05705111846327782 (0.04009017193564636) | > loss_mel_ce: 2.5541069507598877 (nan) | > loss: 0.010361739434301853 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13651176269766477) | > loader_time: 0.0046 (0.004726253815367349)  --> TIME: 2025-02-24 15:02:26 -- STEP: 36258/36938 -- GLOBAL_STEP: 368700 | > loss_text_ce: 0.043155644088983536 (0.040088911644475055) | > loss_mel_ce: 3.752983808517456 (nan) | > loss: 0.01506404671818018 (nan) | > current_lr: 5e-06 | > step_time: 0.169 (0.13651097006263455) | > loader_time: 0.0048 (0.004726129337274836)  --> TIME: 2025-02-24 15:02:36 -- STEP: 36308/36938 -- GLOBAL_STEP: 368750 | > loss_text_ce: 0.04178391769528389 (0.04008825855830141) | > loss_mel_ce: 2.399827003479004 (nan) | > loss: 0.009688932448625565 (nan) | > current_lr: 5e-06 | > step_time: 0.1893 (0.13651447850622586) | > loader_time: 0.0049 (0.004726080553271899)  --> TIME: 2025-02-24 15:02:45 -- STEP: 36358/36938 -- GLOBAL_STEP: 368800 | > loss_text_ce: 0.044308293610811234 (0.04008940690533598) | > loss_mel_ce: 2.82331919670105 (nan) | > loss: 0.011379474774003029 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.1365115972325643) | > loader_time: 0.0046 (0.00472606905839569)  --> TIME: 2025-02-24 15:02:54 -- STEP: 36408/36938 -- GLOBAL_STEP: 368850 | > loss_text_ce: 0.03728223592042923 (0.04008956119268564) | > loss_mel_ce: 3.373417377471924 (nan) | > loss: 0.013534522615373135 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13650204437403873) | > loader_time: 0.0046 (0.004725947540635697)  --> TIME: 2025-02-24 15:03:03 -- STEP: 36458/36938 -- GLOBAL_STEP: 368900 | > loss_text_ce: 0.0502765066921711 (0.040088521708815744) | > loss_mel_ce: 3.542555093765259 (nan) | > loss: 0.014257268980145454 (nan) | > current_lr: 5e-06 | > step_time: 0.1832 (0.13650589721947837) | > loader_time: 0.005 (0.004725901698234358)  --> TIME: 2025-02-24 15:03:13 -- STEP: 36508/36938 -- GLOBAL_STEP: 368950 | > loss_text_ce: 0.04127594456076622 (0.040087167652805775) | > loss_mel_ce: 3.5875957012176514 (nan) | > loss: 0.014400285668671131 (nan) | > current_lr: 5e-06 | > step_time: 0.1708 (0.13650164634096956) | > loader_time: 0.0049 (0.004725769307483569)  --> TIME: 2025-02-24 15:03:22 -- STEP: 36558/36938 -- GLOBAL_STEP: 369000 | > loss_text_ce: 0.049397606402635574 (0.040085527343184726) | > loss_mel_ce: 2.680201292037964 (nan) | > loss: 0.01083174254745245 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.1365025371070151) | > loader_time: 0.0046 (0.004725726638556893)  --> TIME: 2025-02-24 15:03:31 -- STEP: 36608/36938 -- GLOBAL_STEP: 369050 | > loss_text_ce: 0.04410380870103836 (0.040083274613283225) | > loss_mel_ce: 2.778186082839966 (nan) | > loss: 0.01119956374168396 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1364988953370094) | > loader_time: 0.0047 (0.004725625087227045)  --> TIME: 2025-02-24 15:03:40 -- STEP: 36658/36938 -- GLOBAL_STEP: 369100 | > loss_text_ce: 0.04092780873179436 (0.040083952316946045) | > loss_mel_ce: 3.378384590148926 (nan) | > loss: 0.013568701222538948 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13649568486716956) | > loader_time: 0.0045 (0.004725524938089001)  --> TIME: 2025-02-24 15:03:50 -- STEP: 36708/36938 -- GLOBAL_STEP: 369150 | > loss_text_ce: 0.042636431753635406 (0.0400861082865472) | > loss_mel_ce: 2.339878559112549 (nan) | > loss: 0.009454425424337387 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.1364961395703241) | > loader_time: 0.0046 (0.004725623055456325)  --> TIME: 2025-02-24 15:03:59 -- STEP: 36758/36938 -- GLOBAL_STEP: 369200 | > loss_text_ce: 0.03780733048915863 (0.04008736876819813) | > loss_mel_ce: 3.563807487487793 (nan) | > loss: 0.014292122796177864 (nan) | > current_lr: 5e-06 | > step_time: 0.1508 (0.13649628635744468) | > loader_time: 0.0048 (0.004725629904952669)  --> TIME: 2025-02-24 15:04:08 -- STEP: 36808/36938 -- GLOBAL_STEP: 369250 | > loss_text_ce: 0.037078674882650375 (0.040086844608248906) | > loss_mel_ce: 3.3454184532165527 (nan) | > loss: 0.013422608375549316 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.1364919692051187) | > loader_time: 0.0048 (0.004725536291455756)  --> TIME: 2025-02-24 15:04:17 -- STEP: 36858/36938 -- GLOBAL_STEP: 369300 | > loss_text_ce: 0.04373075067996979 (0.04008612585563252) | > loss_mel_ce: 3.0881869792938232 (nan) | > loss: 0.01242824550718069 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13649430691021705) | > loader_time: 0.0047 (0.004725776160362989)  --> TIME: 2025-02-24 15:04:27 -- STEP: 36908/36938 -- GLOBAL_STEP: 369350 | > loss_text_ce: 0.028044339269399643 (0.040086038108213484) | > loss_mel_ce: 1.74346923828125 (nan) | > loss: 0.0070298160426318645 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13649944187670404) | > loader_time: 0.0048 (0.004726008667619787)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.020537844978936824 (-0.00018879328623858105) | > avg_loss_text_ce: 0.038847126767481914 (-0.0008724433102525048) | > avg_loss_mel_ce: 2.764363928596572 (-0.03519495997098465) | > avg_loss: 2.8032110545894886 (-0.03606739846786544) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_369380.pth  > EPOCH: 11/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-24 15:04:58)   --> TIME: 2025-02-24 15:05:02 -- STEP: 20/36938 -- GLOBAL_STEP: 369400 | > loss_text_ce: 0.03800134360790253 (0.040333830192685126) | > loss_mel_ce: 3.2970290184020996 (3.2057291865348816) | > loss: 0.013234248384833336 (0.012881203135475517) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13384971618652344) | > loader_time: 0.0045 (0.004644989967346191)  --> TIME: 2025-02-24 15:05:11 -- STEP: 70/36938 -- GLOBAL_STEP: 369450 | > loss_text_ce: 0.0462854839861393 (0.03905375653079578) | > loss_mel_ce: 3.8126232624053955 (2.7493929783148423) | > loss: 0.015313130803406239 (0.011065265404925282) | > current_lr: 5e-06 | > step_time: 0.1491 (0.1330579280853271) | > loader_time: 0.0047 (0.004783347674778528)  --> TIME: 2025-02-24 15:05:20 -- STEP: 120/36938 -- GLOBAL_STEP: 369500 | > loss_text_ce: 0.04091250151395798 (0.038741920577983056) | > loss_mel_ce: 2.9028968811035156 (nan) | > loss: 0.011681783944368362 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13286142547925317) | > loader_time: 0.0046 (0.004784806569417318)  --> TIME: 2025-02-24 15:05:29 -- STEP: 170/36938 -- GLOBAL_STEP: 369550 | > loss_text_ce: 0.041251424700021744 (0.03910771755172926) | > loss_mel_ce: 1.6131068468093872 (nan) | > loss: 0.006564914248883724 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.133393973462722) | > loader_time: 0.0044 (0.004753878537346335)  --> TIME: 2025-02-24 15:05:39 -- STEP: 220/36938 -- GLOBAL_STEP: 369600 | > loss_text_ce: 0.05328581854701042 (0.03953992705792188) | > loss_mel_ce: 0.0272216796875 (nan) | > loss: 0.00031947423121891916 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13411198095841845) | > loader_time: 0.0046 (0.004738553003831345)  --> TIME: 2025-02-24 15:05:48 -- STEP: 270/36938 -- GLOBAL_STEP: 369650 | > loss_text_ce: 0.033152226358652115 (0.03945789704168284) | > loss_mel_ce: 2.2180187702178955 (nan) | > loss: 0.008933219127357006 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13451952404446083) | > loader_time: 0.0048 (0.004719161987304689)  --> TIME: 2025-02-24 15:05:57 -- STEP: 320/36938 -- GLOBAL_STEP: 369700 | > loss_text_ce: 0.03663712367415428 (0.0393755108525511) | > loss_mel_ce: 1.1532025337219238 (nan) | > loss: 0.004721586126834154 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.1351095080375672) | > loader_time: 0.0046 (0.004715611785650254)  --> TIME: 2025-02-24 15:06:07 -- STEP: 370/36938 -- GLOBAL_STEP: 369750 | > loss_text_ce: 0.0498117059469223 (0.03950999336367522) | > loss_mel_ce: 3.687166690826416 (nan) | > loss: 0.014829279854893684 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.1354221814387554) | > loader_time: 0.0047 (0.004704954817488383)  --> TIME: 2025-02-24 15:06:16 -- STEP: 420/36938 -- GLOBAL_STEP: 369800 | > loss_text_ce: 0.034534554928541183 (0.039576167227434254) | > loss_mel_ce: 3.6705217361450195 (nan) | > loss: 0.014702605083584785 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.1362463962464106) | > loader_time: 0.0047 (0.004697162764413018)  --> TIME: 2025-02-24 15:06:26 -- STEP: 470/36938 -- GLOBAL_STEP: 369850 | > loss_text_ce: 0.039043083786964417 (0.03981975741842959) | > loss_mel_ce: 0.04575347900390625 (nan) | > loss: 0.00033649432589299977 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13663635152451542) | > loader_time: 0.0048 (0.004698228328785998)  --> TIME: 2025-02-24 15:06:35 -- STEP: 520/36938 -- GLOBAL_STEP: 369900 | > loss_text_ce: 0.04640626907348633 (0.03982478689688902) | > loss_mel_ce: 3.3555898666381836 (nan) | > loss: 0.013499985449016094 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13674970636001005) | > loader_time: 0.0049 (0.004703319989717924)  --> TIME: 2025-02-24 15:06:45 -- STEP: 570/36938 -- GLOBAL_STEP: 369950 | > loss_text_ce: 0.04263954237103462 (0.03965519082716159) | > loss_mel_ce: 3.1991307735443115 (nan) | > loss: 0.012864168733358383 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.1366369552779616) | > loader_time: 0.0046 (0.004709526112205107)  --> TIME: 2025-02-24 15:06:54 -- STEP: 620/36938 -- GLOBAL_STEP: 370000 | > loss_text_ce: 0.028751352801918983 (0.03965842179773796) | > loss_mel_ce: 3.030505657196045 (nan) | > loss: 0.012139909900724888 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.1367118566266952) | > loader_time: 0.0048 (0.004712633932790449) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_370000.pth  --> TIME: 2025-02-24 15:07:09 -- STEP: 670/36938 -- GLOBAL_STEP: 370050 | > loss_text_ce: 0.04426950961351395 (0.03965969767643891) | > loss_mel_ce: 2.3252482414245605 (nan) | > loss: 0.009402848780155182 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13636911804996313) | > loader_time: 0.0052 (0.0047303833178619845)  --> TIME: 2025-02-24 15:07:18 -- STEP: 720/36938 -- GLOBAL_STEP: 370100 | > loss_text_ce: 0.04471622034907341 (0.03972492146616184) | > loss_mel_ce: 2.186739921569824 (nan) | > loss: 0.008854985237121582 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13607915143171945) | > loader_time: 0.0048 (0.00473173062006632)  --> TIME: 2025-02-24 15:07:27 -- STEP: 770/36938 -- GLOBAL_STEP: 370150 | > loss_text_ce: 0.034318845719099045 (0.03972461378419557) | > loss_mel_ce: 2.8257229328155518 (nan) | > loss: 0.011349373497068882 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13560467292736095) | > loader_time: 0.0046 (0.004733891920609904)  --> TIME: 2025-02-24 15:07:36 -- STEP: 820/36938 -- GLOBAL_STEP: 370200 | > loss_text_ce: 0.04169321060180664 (0.03973468623873666) | > loss_mel_ce: 2.5090606212615967 (nan) | > loss: 0.01012203935533762 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.1357190100158132) | > loader_time: 0.005 (0.004742429314590082)  --> TIME: 2025-02-24 15:07:46 -- STEP: 870/36938 -- GLOBAL_STEP: 370250 | > loss_text_ce: 0.04291572421789169 (0.03976291585499529) | > loss_mel_ce: 2.923031806945801 (nan) | > loss: 0.011769633740186691 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1358110693679457) | > loader_time: 0.0052 (0.0047498401554151495)  --> TIME: 2025-02-24 15:07:55 -- STEP: 920/36938 -- GLOBAL_STEP: 370300 | > loss_text_ce: 0.030918346717953682 (0.039725510814510624) | > loss_mel_ce: 3.0011987686157227 (nan) | > loss: 0.012032211758196354 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13580373473789378) | > loader_time: 0.0046 (0.0047658775163733425)  --> TIME: 2025-02-24 15:08:04 -- STEP: 970/36938 -- GLOBAL_STEP: 370350 | > loss_text_ce: 0.03325393423438072 (0.03968707384138379) | > loss_mel_ce: 2.4561612606048584 (nan) | > loss: 0.009878632612526417 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13576810654905652) | > loader_time: 0.0046 (0.004767411517113753)  --> TIME: 2025-02-24 15:08:14 -- STEP: 1020/36938 -- GLOBAL_STEP: 370400 | > loss_text_ce: 0.042602527886629105 (0.039643000261675505) | > loss_mel_ce: 2.806328058242798 (nan) | > loss: 0.011305280961096287 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.1358155418844784) | > loader_time: 0.0047 (0.004766574092939786)  --> TIME: 2025-02-24 15:08:23 -- STEP: 1070/36938 -- GLOBAL_STEP: 370450 | > loss_text_ce: 0.045364007353782654 (0.039711432874411645) | > loss_mel_ce: 3.074190616607666 (nan) | > loss: 0.012379185296595097 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.1358788474697933) | > loader_time: 0.0049 (0.004769510420683388)  --> TIME: 2025-02-24 15:08:32 -- STEP: 1120/36938 -- GLOBAL_STEP: 370500 | > loss_text_ce: 0.03157345950603485 (0.0396558826373491) | > loss_mel_ce: 4.077104568481445 (nan) | > loss: 0.01630427874624729 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.13590594381093984) | > loader_time: 0.0069 (0.004775599070957729)  --> TIME: 2025-02-24 15:08:41 -- STEP: 1170/36938 -- GLOBAL_STEP: 370550 | > loss_text_ce: 0.04129258915781975 (0.039613430773536865) | > loss_mel_ce: 1.8960331678390503 (nan) | > loss: 0.007687801029533148 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13575812323480588) | > loader_time: 0.0045 (0.004774896915142352)  --> TIME: 2025-02-24 15:08:51 -- STEP: 1220/36938 -- GLOBAL_STEP: 370600 | > loss_text_ce: 0.039926327764987946 (0.03963503670100063) | > loss_mel_ce: 2.427938461303711 (nan) | > loss: 0.009793114848434925 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13598050680316864) | > loader_time: 0.0046 (0.004772167323065585)  --> TIME: 2025-02-24 15:09:00 -- STEP: 1270/36938 -- GLOBAL_STEP: 370650 | > loss_text_ce: 0.03494016453623772 (0.039635079180983114) | > loss_mel_ce: 2.1749165058135986 (nan) | > loss: 0.00876927375793457 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13599910736083992) | > loader_time: 0.0052 (0.004768321833272617)  --> TIME: 2025-02-24 15:09:09 -- STEP: 1320/36938 -- GLOBAL_STEP: 370700 | > loss_text_ce: 0.03588550165295601 (0.039618549655389156) | > loss_mel_ce: 2.9603517055511475 (nan) | > loss: 0.01188983116298914 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.1358936725240765) | > loader_time: 0.0045 (0.0047642431475899425)  --> TIME: 2025-02-24 15:09:19 -- STEP: 1370/36938 -- GLOBAL_STEP: 370750 | > loss_text_ce: 0.04309406131505966 (0.03961019941000608) | > loss_mel_ce: 3.4806158542633057 (nan) | > loss: 0.013982976786792278 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13584088670076253) | > loader_time: 0.0047 (0.004758410732241445)  --> TIME: 2025-02-24 15:09:28 -- STEP: 1420/36938 -- GLOBAL_STEP: 370800 | > loss_text_ce: 0.047908321022987366 (0.03970010681733696) | > loss_mel_ce: 3.0778520107269287 (nan) | > loss: 0.012403811328113079 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.13589917689981595) | > loader_time: 0.0047 (0.004755103420203842)  --> TIME: 2025-02-24 15:09:37 -- STEP: 1470/36938 -- GLOBAL_STEP: 370850 | > loss_text_ce: 0.04096631333231926 (0.039726300891108665) | > loss_mel_ce: 1.8266688585281372 (nan) | > loss: 0.0074112508445978165 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.1360089832422685) | > loader_time: 0.0047 (0.00474994669155199)  --> TIME: 2025-02-24 15:09:47 -- STEP: 1520/36938 -- GLOBAL_STEP: 370900 | > loss_text_ce: 0.03867993503808975 (0.039701287999847235) | > loss_mel_ce: 2.1308114528656006 (nan) | > loss: 0.00860909279435873 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.1361339005984758) | > loader_time: 0.0046 (0.0047487158524362715)  --> TIME: 2025-02-24 15:09:56 -- STEP: 1570/36938 -- GLOBAL_STEP: 370950 | > loss_text_ce: 0.045087628066539764 (0.03967223167182154) | > loss_mel_ce: 3.919440746307373 (nan) | > loss: 0.015732256695628166 (nan) | > current_lr: 5e-06 | > step_time: 0.1895 (0.13616178931703998) | > loader_time: 0.005 (0.004749975690416475)  --> TIME: 2025-02-24 15:10:05 -- STEP: 1620/36938 -- GLOBAL_STEP: 371000 | > loss_text_ce: 0.04408491402864456 (0.03972534138571331) | > loss_mel_ce: 2.941761016845703 (nan) | > loss: 0.011848595924675465 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13609566997598724) | > loader_time: 0.005 (0.004749703995975452)  --> TIME: 2025-02-24 15:10:15 -- STEP: 1670/36938 -- GLOBAL_STEP: 371050 | > loss_text_ce: 0.03674275428056717 (0.039729197096904936) | > loss_mel_ce: 1.7370719909667969 (nan) | > loss: 0.007038948126137257 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13620021900017104) | > loader_time: 0.0046 (0.0047498106242653895)  --> TIME: 2025-02-24 15:10:24 -- STEP: 1720/36938 -- GLOBAL_STEP: 371100 | > loss_text_ce: 0.03538769111037254 (0.03969256152983667) | > loss_mel_ce: 3.5496747493743896 (nan) | > loss: 0.014226439408957958 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13617938507434926) | > loader_time: 0.0046 (0.004749848399051403)  --> TIME: 2025-02-24 15:10:34 -- STEP: 1770/36938 -- GLOBAL_STEP: 371150 | > loss_text_ce: 0.0354125052690506 (0.03971243144176099) | > loss_mel_ce: 2.5029704570770264 (nan) | > loss: 0.010072949342429638 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13636171507970088) | > loader_time: 0.0044 (0.004752182960510257)  --> TIME: 2025-02-24 15:10:43 -- STEP: 1820/36938 -- GLOBAL_STEP: 371200 | > loss_text_ce: 0.04357706010341644 (0.03973333253474024) | > loss_mel_ce: 1.6696265935897827 (nan) | > loss: 0.006798427551984787 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13641834586531246) | > loader_time: 0.0046 (0.004753074148199064)  --> TIME: 2025-02-24 15:10:53 -- STEP: 1870/36938 -- GLOBAL_STEP: 371250 | > loss_text_ce: 0.045025620609521866 (0.039709953257545096) | > loss_mel_ce: 2.791663646697998 (nan) | > loss: 0.011256704106926918 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.13647918905166373) | > loader_time: 0.005 (0.004749563416057733)  --> TIME: 2025-02-24 15:11:02 -- STEP: 1920/36938 -- GLOBAL_STEP: 371300 | > loss_text_ce: 0.05328662693500519 (0.03969765090150757) | > loss_mel_ce: 3.616508722305298 (nan) | > loss: 0.014562680386006832 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.13640905047456417) | > loader_time: 0.0053 (0.004747866590817772)  --> TIME: 2025-02-24 15:11:11 -- STEP: 1970/36938 -- GLOBAL_STEP: 371350 | > loss_text_ce: 0.03199175000190735 (0.039694366487792586) | > loss_mel_ce: 3.060807228088379 (nan) | > loss: 0.012273012660443783 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.13642720038515657) | > loader_time: 0.0045 (0.004745817184448245)  --> TIME: 2025-02-24 15:11:21 -- STEP: 2020/36938 -- GLOBAL_STEP: 371400 | > loss_text_ce: 0.03392958641052246 (0.0396774882625899) | > loss_mel_ce: 2.9441187381744385 (nan) | > loss: 0.011817652732133865 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13657103257604158) | > loader_time: 0.0046 (0.00474374317886806)  --> TIME: 2025-02-24 15:11:30 -- STEP: 2070/36938 -- GLOBAL_STEP: 371450 | > loss_text_ce: 0.05536504089832306 (0.03968756893730681) | > loss_mel_ce: 2.3526809215545654 (nan) | > loss: 0.009555738419294357 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13653846293831787) | > loader_time: 0.0048 (0.004743357557029542)  --> TIME: 2025-02-24 15:11:40 -- STEP: 2120/36938 -- GLOBAL_STEP: 371500 | > loss_text_ce: 0.03986508399248123 (0.03970604541904801) | > loss_mel_ce: 2.8856568336486816 (nan) | > loss: 0.01160921435803175 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13655706520350472) | > loader_time: 0.0045 (0.004743758237586834)  --> TIME: 2025-02-24 15:11:49 -- STEP: 2170/36938 -- GLOBAL_STEP: 371550 | > loss_text_ce: 0.03665287420153618 (0.039674638533612805) | > loss_mel_ce: 3.200627565383911 (nan) | > loss: 0.012846351601183414 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.1366463664489957) | > loader_time: 0.0048 (0.0047412340542138465)  --> TIME: 2025-02-24 15:11:58 -- STEP: 2220/36938 -- GLOBAL_STEP: 371600 | > loss_text_ce: 0.04904087260365486 (0.03971934230024882) | > loss_mel_ce: 2.980593681335449 (nan) | > loss: 0.012022359296679497 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.1366176824311952) | > loader_time: 0.0052 (0.004739395562592928)  --> TIME: 2025-02-24 15:12:07 -- STEP: 2270/36938 -- GLOBAL_STEP: 371650 | > loss_text_ce: 0.020989853888750076 (0.03968758556072798) | > loss_mel_ce: 0.614147961139679 (nan) | > loss: 0.002520388225093484 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13655417354096389) | > loader_time: 0.0045 (0.004739555388295175)  --> TIME: 2025-02-24 15:12:17 -- STEP: 2320/36938 -- GLOBAL_STEP: 371700 | > loss_text_ce: 0.02770368941128254 (0.039677293547268545) | > loss_mel_ce: 2.8638916015625 (nan) | > loss: 0.01147458516061306 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.1365367066243597) | > loader_time: 0.0045 (0.004738925962612542)  --> TIME: 2025-02-24 15:12:26 -- STEP: 2370/36938 -- GLOBAL_STEP: 371750 | > loss_text_ce: 0.04277623072266579 (0.039673902497544294) | > loss_mel_ce: 3.1790707111358643 (nan) | > loss: 0.0127851078286767 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13656703922818964) | > loader_time: 0.0045 (0.004740367559441028)  --> TIME: 2025-02-24 15:12:35 -- STEP: 2420/36938 -- GLOBAL_STEP: 371800 | > loss_text_ce: 0.04308345168828964 (0.039639769043496316) | > loss_mel_ce: 3.062835931777954 (nan) | > loss: 0.012325077317655087 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13655193167284493) | > loader_time: 0.0046 (0.0047387492558187685)  --> TIME: 2025-02-24 15:12:44 -- STEP: 2470/36938 -- GLOBAL_STEP: 371850 | > loss_text_ce: 0.05178894102573395 (0.03965765507778654) | > loss_mel_ce: 3.519381046295166 (nan) | > loss: 0.01417131070047617 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.13649171740419946) | > loader_time: 0.0046 (0.004738723604302655)  --> TIME: 2025-02-24 15:12:54 -- STEP: 2520/36938 -- GLOBAL_STEP: 371900 | > loss_text_ce: 0.0413636788725853 (0.039642087568248004) | > loss_mel_ce: 2.6434695720672607 (nan) | > loss: 0.010654101148247719 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13656521486857576) | > loader_time: 0.0046 (0.004736444779804771)  --> TIME: 2025-02-24 15:13:03 -- STEP: 2570/36938 -- GLOBAL_STEP: 371950 | > loss_text_ce: 0.03456532210111618 (0.039659153525119555) | > loss_mel_ce: 2.962151527404785 (nan) | > loss: 0.011891733855009079 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.1365257789188785) | > loader_time: 0.0048 (0.004733497530569818)  --> TIME: 2025-02-24 15:13:12 -- STEP: 2620/36938 -- GLOBAL_STEP: 372000 | > loss_text_ce: 0.047283634543418884 (0.03964224825079994) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13651425601871864) | > loader_time: 0.0045 (0.004730773609103131)  --> TIME: 2025-02-24 15:13:22 -- STEP: 2670/36938 -- GLOBAL_STEP: 372050 | > loss_text_ce: 0.03988470882177353 (0.03961170794937599) | > loss_mel_ce: 2.8900959491729736 (nan) | > loss: 0.01162690855562687 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.1364364977633014) | > loader_time: 0.0045 (0.004728187425306222)  --> TIME: 2025-02-24 15:13:31 -- STEP: 2720/36938 -- GLOBAL_STEP: 372100 | > loss_text_ce: 0.03004520758986473 (0.03961054446203079) | > loss_mel_ce: 2.8039705753326416 (nan) | > loss: 0.011246095411479473 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.1364061814897198) | > loader_time: 0.0046 (0.004725957705694082)  --> TIME: 2025-02-24 15:13:40 -- STEP: 2770/36938 -- GLOBAL_STEP: 372150 | > loss_text_ce: 0.03656519949436188 (0.03962176736672857) | > loss_mel_ce: 2.633911371231079 (nan) | > loss: 0.01059713028371334 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.1363865822230864) | > loader_time: 0.0047 (0.004725620376504281)  --> TIME: 2025-02-24 15:13:49 -- STEP: 2820/36938 -- GLOBAL_STEP: 372200 | > loss_text_ce: 0.03619910031557083 (0.03959165953218615) | > loss_mel_ce: 3.281665086746216 (nan) | > loss: 0.013166128657758236 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13643673337097703) | > loader_time: 0.0046 (0.004727542315814503)  --> TIME: 2025-02-24 15:13:59 -- STEP: 2870/36938 -- GLOBAL_STEP: 372250 | > loss_text_ce: 0.03369196876883507 (0.039600929958609504) | > loss_mel_ce: 1.5877562761306763 (nan) | > loss: 0.006434319075196981 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13646609957625214) | > loader_time: 0.0044 (0.0047280151254208555)  --> TIME: 2025-02-24 15:14:08 -- STEP: 2920/36938 -- GLOBAL_STEP: 372300 | > loss_text_ce: 0.04830445721745491 (0.03961942834167241) | > loss_mel_ce: 3.433962345123291 (nan) | > loss: 0.013818520121276379 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13649872435282337) | > loader_time: 0.0046 (0.0047263985627317975)  --> TIME: 2025-02-24 15:14:18 -- STEP: 2970/36938 -- GLOBAL_STEP: 372350 | > loss_text_ce: 0.03991067782044411 (0.03964125108701308) | > loss_mel_ce: 3.590285301208496 (nan) | > loss: 0.014405541121959686 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13653592726196892) | > loader_time: 0.0047 (0.0047275219702158635)  --> TIME: 2025-02-24 15:14:27 -- STEP: 3020/36938 -- GLOBAL_STEP: 372400 | > loss_text_ce: 0.036186300218105316 (0.039629561312508936) | > loss_mel_ce: 3.532599925994873 (nan) | > loss: 0.014161850325763226 (nan) | > current_lr: 5e-06 | > step_time: 0.1687 (0.13654873955328725) | > loader_time: 0.0046 (0.004725116136058269)  --> TIME: 2025-02-24 15:14:36 -- STEP: 3070/36938 -- GLOBAL_STEP: 372450 | > loss_text_ce: 0.044444143772125244 (0.03964122584429398) | > loss_mel_ce: 0.09578529000282288 (nan) | > loss: 0.0005564660532400012 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13648300931974028) | > loader_time: 0.0044 (0.004724590863777678)  --> TIME: 2025-02-24 15:14:45 -- STEP: 3120/36938 -- GLOBAL_STEP: 372500 | > loss_text_ce: 0.04713790863752365 (0.03964887584756626) | > loss_mel_ce: 3.616609811782837 (nan) | > loss: 0.014538682997226715 (nan) | > current_lr: 5e-06 | > step_time: 0.1609 (0.13643875565284344) | > loader_time: 0.0047 (0.004722414719752767)  --> TIME: 2025-02-24 15:14:55 -- STEP: 3170/36938 -- GLOBAL_STEP: 372550 | > loss_text_ce: 0.0394183024764061 (0.03963903910230708) | > loss_mel_ce: 3.343080997467041 (nan) | > loss: 0.013422616757452488 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13648137865758475) | > loader_time: 0.0045 (0.004721161243667352)  --> TIME: 2025-02-24 15:15:04 -- STEP: 3220/36938 -- GLOBAL_STEP: 372600 | > loss_text_ce: 0.036417245864868164 (0.039638149020730505) | > loss_mel_ce: 1.6635239124298096 (nan) | > loss: 0.006745798513293266 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1364567304249876) | > loader_time: 0.0045 (0.00471986991278132)  --> TIME: 2025-02-24 15:15:13 -- STEP: 3270/36938 -- GLOBAL_STEP: 372650 | > loss_text_ce: 0.04746101796627045 (0.039615513474056646) | > loss_mel_ce: 3.639092445373535 (nan) | > loss: 0.01462918147444725 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.136486269283003) | > loader_time: 0.0046 (0.004719172486471461)  --> TIME: 2025-02-24 15:15:23 -- STEP: 3320/36938 -- GLOBAL_STEP: 372700 | > loss_text_ce: 0.053386230021715164 (0.03961989710483331) | > loss_mel_ce: 3.426072120666504 (nan) | > loss: 0.013807374984025955 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13648194074630732) | > loader_time: 0.0045 (0.004717377869479612)  --> TIME: 2025-02-24 15:15:32 -- STEP: 3370/36938 -- GLOBAL_STEP: 372750 | > loss_text_ce: 0.030401146039366722 (0.039631559221399985) | > loss_mel_ce: 1.8331869840621948 (nan) | > loss: 0.007395191118121147 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13646084158639882) | > loader_time: 0.0045 (0.004716685829954948)  --> TIME: 2025-02-24 15:15:41 -- STEP: 3420/36938 -- GLOBAL_STEP: 372800 | > loss_text_ce: 0.039503782987594604 (0.03962568010377813) | > loss_mel_ce: 3.0027518272399902 (nan) | > loss: 0.012072443962097168 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13649034611662927) | > loader_time: 0.0047 (0.004716979551036435)  --> TIME: 2025-02-24 15:15:51 -- STEP: 3470/36938 -- GLOBAL_STEP: 372850 | > loss_text_ce: 0.043527908623218536 (0.039640876916787283) | > loss_mel_ce: 2.153003215789795 (nan) | > loss: 0.008716393262147903 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13647568136539512) | > loader_time: 0.0045 (0.0047153096377677605)  --> TIME: 2025-02-24 15:16:00 -- STEP: 3520/36938 -- GLOBAL_STEP: 372900 | > loss_text_ce: 0.03348476067185402 (0.03964049005748652) | > loss_mel_ce: 2.965047597885132 (nan) | > loss: 0.011898938566446304 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13654293905604958) | > loader_time: 0.0046 (0.004713569649241167)  --> TIME: 2025-02-24 15:16:10 -- STEP: 3570/36938 -- GLOBAL_STEP: 372950 | > loss_text_ce: 0.039644937962293625 (0.039616435249949025) | > loss_mel_ce: 3.6004135608673096 (nan) | > loss: 0.014444677159190178 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.13669161195514568) | > loader_time: 0.0046 (0.004712452340860641)  --> TIME: 2025-02-24 15:16:19 -- STEP: 3620/36938 -- GLOBAL_STEP: 373000 | > loss_text_ce: 0.03954833745956421 (0.03963800622742017) | > loss_mel_ce: 1.8285260200500488 (nan) | > loss: 0.007412994280457497 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13665140755268734) | > loader_time: 0.0044 (0.004710640564807851)  --> TIME: 2025-02-24 15:16:29 -- STEP: 3670/36938 -- GLOBAL_STEP: 373050 | > loss_text_ce: 0.045969899743795395 (0.039639466817488124) | > loss_mel_ce: 1.8121734857559204 (nan) | > loss: 0.007373584900051355 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13662078562484434) | > loader_time: 0.0045 (0.004708939417181594)  --> TIME: 2025-02-24 15:16:38 -- STEP: 3720/36938 -- GLOBAL_STEP: 373100 | > loss_text_ce: 0.04671840742230415 (0.039637460535822684) | > loss_mel_ce: 3.611391067504883 (nan) | > loss: 0.014516307972371578 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.1366071021685035) | > loader_time: 0.0045 (0.004707357511725458)  --> TIME: 2025-02-24 15:16:47 -- STEP: 3770/36938 -- GLOBAL_STEP: 373150 | > loss_text_ce: 0.04483703896403313 (0.039634041030661406) | > loss_mel_ce: 2.5156664848327637 (nan) | > loss: 0.010160728357732296 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.13661970997362605) | > loader_time: 0.0046 (0.0047068178495614345)  --> TIME: 2025-02-24 15:16:57 -- STEP: 3820/36938 -- GLOBAL_STEP: 373200 | > loss_text_ce: 0.033330079168081284 (0.03962416423738228) | > loss_mel_ce: 2.170135021209717 (nan) | > loss: 0.008743909187614918 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13663518116736276) | > loader_time: 0.0044 (0.004705751691189093)  --> TIME: 2025-02-24 15:17:06 -- STEP: 3870/36938 -- GLOBAL_STEP: 373250 | > loss_text_ce: 0.041292931884527206 (0.03962840992495293) | > loss_mel_ce: 2.3984527587890625 (nan) | > loss: 0.009681531228125095 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13663643225834957) | > loader_time: 0.0046 (0.004704141801641875)  --> TIME: 2025-02-24 15:17:15 -- STEP: 3920/36938 -- GLOBAL_STEP: 373300 | > loss_text_ce: 0.044659972190856934 (0.03964330497036242) | > loss_mel_ce: 3.681032180786133 (nan) | > loss: 0.014784494414925575 (nan) | > current_lr: 5e-06 | > step_time: 0.1821 (0.1366198325643733) | > loader_time: 0.0048 (0.004705096446737933)  --> TIME: 2025-02-24 15:17:25 -- STEP: 3970/36938 -- GLOBAL_STEP: 373350 | > loss_text_ce: 0.0397200845181942 (0.03965782705337852) | > loss_mel_ce: 3.026160478591919 (nan) | > loss: 0.012166193686425686 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1366722789159048) | > loader_time: 0.0046 (0.004705883633880213)  --> TIME: 2025-02-24 15:17:34 -- STEP: 4020/36938 -- GLOBAL_STEP: 373400 | > loss_text_ce: 0.040360841900110245 (0.039666192622771816) | > loss_mel_ce: 2.3770065307617188 (nan) | > loss: 0.009592728689312935 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13674388293603154) | > loader_time: 0.0048 (0.004705769742899255)  --> TIME: 2025-02-24 15:17:44 -- STEP: 4070/36938 -- GLOBAL_STEP: 373450 | > loss_text_ce: 0.03511188551783562 (0.03965580258152327) | > loss_mel_ce: 0.0692138671875 (nan) | > loss: 0.00041399113251827657 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13674484271675116) | > loader_time: 0.0044 (0.004704586352411576)  --> TIME: 2025-02-24 15:17:53 -- STEP: 4120/36938 -- GLOBAL_STEP: 373500 | > loss_text_ce: 0.038331981748342514 (0.039674250347973714) | > loss_mel_ce: 3.439152240753174 (nan) | > loss: 0.013799541629850864 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13678654947327168) | > loader_time: 0.0046 (0.004706394556656607)  --> TIME: 2025-02-24 15:18:03 -- STEP: 4170/36938 -- GLOBAL_STEP: 373550 | > loss_text_ce: 0.04135271534323692 (0.03968086200509418) | > loss_mel_ce: 3.14959716796875 (nan) | > loss: 0.012662500143051147 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.1367662839752306) | > loader_time: 0.0046 (0.004707055240512164)  --> TIME: 2025-02-24 15:18:12 -- STEP: 4220/36938 -- GLOBAL_STEP: 373600 | > loss_text_ce: 0.042837582528591156 (0.039677156227247885) | > loss_mel_ce: 3.1203272342681885 (nan) | > loss: 0.012552241794764996 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13673053013769923) | > loader_time: 0.0053 (0.004707168346332697)  --> TIME: 2025-02-24 15:18:21 -- STEP: 4270/36938 -- GLOBAL_STEP: 373650 | > loss_text_ce: 0.03292069211602211 (0.03967975732942367) | > loss_mel_ce: 2.709092140197754 (nan) | > loss: 0.010881003923714161 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.1367911647577753) | > loader_time: 0.0046 (0.0047074861213808996)  --> TIME: 2025-02-24 15:18:31 -- STEP: 4320/36938 -- GLOBAL_STEP: 373700 | > loss_text_ce: 0.030325034633278847 (0.03966614264451579) | > loss_mel_ce: 2.7507474422454834 (nan) | > loss: 0.011036002077162266 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.1368055791214659) | > loader_time: 0.0049 (0.004706545781206186)  --> TIME: 2025-02-24 15:18:40 -- STEP: 4370/36938 -- GLOBAL_STEP: 373750 | > loss_text_ce: 0.03948581591248512 (0.03966083306887388) | > loss_mel_ce: 0.04974365234375 (nan) | > loss: 0.00035408520488999784 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1367982359718129) | > loader_time: 0.0046 (0.004705926700915009)  --> TIME: 2025-02-24 15:18:49 -- STEP: 4420/36938 -- GLOBAL_STEP: 373800 | > loss_text_ce: 0.05008413642644882 (0.039686214121909744) | > loss_mel_ce: 2.8342628479003906 (nan) | > loss: 0.011445822194218636 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13676837121739088) | > loader_time: 0.0045 (0.004705223886136)  --> TIME: 2025-02-24 15:18:58 -- STEP: 4470/36938 -- GLOBAL_STEP: 373850 | > loss_text_ce: 0.03626718372106552 (0.03969357012535623) | > loss_mel_ce: 2.6735217571258545 (nan) | > loss: 0.010753131471574306 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13676573947505383) | > loader_time: 0.0044 (0.0047057822246679365)  --> TIME: 2025-02-24 15:19:08 -- STEP: 4520/36938 -- GLOBAL_STEP: 373900 | > loss_text_ce: 0.0336478091776371 (0.039694116187726856) | > loss_mel_ce: 3.1074578762054443 (nan) | > loss: 0.012464705854654312 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13672211449758143) | > loader_time: 0.0049 (0.004705876692206439)  --> TIME: 2025-02-24 15:19:17 -- STEP: 4570/36938 -- GLOBAL_STEP: 373950 | > loss_text_ce: 0.03466479852795601 (0.039698995584397774) | > loss_mel_ce: 1.6022355556488037 (nan) | > loss: 0.00649563642218709 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13672722726734254) | > loader_time: 0.0044 (0.004704526686303185)  --> TIME: 2025-02-24 15:19:26 -- STEP: 4620/36938 -- GLOBAL_STEP: 374000 | > loss_text_ce: 0.0405895859003067 (0.03969054954767509) | > loss_mel_ce: 2.7520804405212402 (nan) | > loss: 0.011082024313509464 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13674910790992503) | > loader_time: 0.0045 (0.004703840342434954)  --> TIME: 2025-02-24 15:19:35 -- STEP: 4670/36938 -- GLOBAL_STEP: 374050 | > loss_text_ce: 0.02725076675415039 (0.0396834119885154) | > loss_mel_ce: 2.8074183464050293 (nan) | > loss: 0.011248687282204628 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13670973440819678) | > loader_time: 0.0046 (0.004704551370036661)  --> TIME: 2025-02-24 15:19:45 -- STEP: 4720/36938 -- GLOBAL_STEP: 374100 | > loss_text_ce: 0.04168666899204254 (0.03967563430798406) | > loss_mel_ce: 1.5367867946624756 (nan) | > loss: 0.0062637836672365665 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.1366758725400698) | > loader_time: 0.0045 (0.0047052476870811395)  --> TIME: 2025-02-24 15:19:54 -- STEP: 4770/36938 -- GLOBAL_STEP: 374150 | > loss_text_ce: 0.03421693667769432 (0.03966473175595417) | > loss_mel_ce: 3.3201112747192383 (nan) | > loss: 0.013310826383531094 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.136707413321521) | > loader_time: 0.0047 (0.004705301770624107)  --> TIME: 2025-02-24 15:20:04 -- STEP: 4820/36938 -- GLOBAL_STEP: 374200 | > loss_text_ce: 0.03903761878609657 (0.03965878720526652) | > loss_mel_ce: 3.0076165199279785 (nan) | > loss: 0.012089898809790611 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13672528895105052) | > loader_time: 0.0044 (0.004705736597543913)  --> TIME: 2025-02-24 15:20:13 -- STEP: 4870/36938 -- GLOBAL_STEP: 374250 | > loss_text_ce: 0.049462493509054184 (0.03966990472687673) | > loss_mel_ce: 3.122086763381958 (nan) | > loss: 0.012585514225065708 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.13674540735123322) | > loader_time: 0.0048 (0.004705916342059675)  --> TIME: 2025-02-24 15:20:22 -- STEP: 4920/36938 -- GLOBAL_STEP: 374300 | > loss_text_ce: 0.047436077147722244 (0.03966059494040507) | > loss_mel_ce: 0.0384521484375 (nan) | > loss: 0.0003408262855373323 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13673916546309847) | > loader_time: 0.0046 (0.004705165169103342)  --> TIME: 2025-02-24 15:20:32 -- STEP: 4970/36938 -- GLOBAL_STEP: 374350 | > loss_text_ce: 0.04786297678947449 (0.039663803060042455) | > loss_mel_ce: 2.854170083999634 (nan) | > loss: 0.011516004800796509 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1367106306241071) | > loader_time: 0.0044 (0.0047049789601408334)  --> TIME: 2025-02-24 15:20:40 -- STEP: 5020/36938 -- GLOBAL_STEP: 374400 | > loss_text_ce: 0.032803259789943695 (0.03966078829016547) | > loss_mel_ce: 3.4732768535614014 (nan) | > loss: 0.013913017697632313 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.13665621309166398) | > loader_time: 0.0045 (0.004704981306159625)  --> TIME: 2025-02-24 15:20:49 -- STEP: 5070/36938 -- GLOBAL_STEP: 374450 | > loss_text_ce: 0.035498738288879395 (0.03967059293497324) | > loss_mel_ce: 2.2745046615600586 (nan) | > loss: 0.00916668027639389 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13660078255617625) | > loader_time: 0.0045 (0.00470532848990174)  --> TIME: 2025-02-24 15:20:59 -- STEP: 5120/36938 -- GLOBAL_STEP: 374500 | > loss_text_ce: 0.03654852136969566 (0.03966128966949322) | > loss_mel_ce: 3.2749462127685547 (nan) | > loss: 0.013140853494405746 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13661446925252654) | > loader_time: 0.0046 (0.0047044030856341025)  --> TIME: 2025-02-24 15:21:09 -- STEP: 5170/36938 -- GLOBAL_STEP: 374550 | > loss_text_ce: 0.04803409427404404 (0.03966859345643177) | > loss_mel_ce: 2.908271551132202 (nan) | > loss: 0.011731372214853764 (nan) | > current_lr: 5e-06 | > step_time: 0.1505 (0.1366765765664192) | > loader_time: 0.0047 (0.00470483593599477)  --> TIME: 2025-02-24 15:21:18 -- STEP: 5220/36938 -- GLOBAL_STEP: 374600 | > loss_text_ce: 0.044120900332927704 (0.0396497561309086) | > loss_mel_ce: 3.3437273502349854 (nan) | > loss: 0.01344384253025055 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.1366454305319947) | > loader_time: 0.0048 (0.004704742733089395)  --> TIME: 2025-02-24 15:21:27 -- STEP: 5270/36938 -- GLOBAL_STEP: 374650 | > loss_text_ce: 0.03426302969455719 (0.03965503846646842) | > loss_mel_ce: 3.1328752040863037 (nan) | > loss: 0.01256801001727581 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.1366507760487649) | > loader_time: 0.0046 (0.004706517387838915)  --> TIME: 2025-02-24 15:21:36 -- STEP: 5320/36938 -- GLOBAL_STEP: 374700 | > loss_text_ce: 0.03726211190223694 (0.03966097128875119) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.1366357974540018) | > loader_time: 0.0046 (0.004707825632023626)  --> TIME: 2025-02-24 15:21:45 -- STEP: 5370/36938 -- GLOBAL_STEP: 374750 | > loss_text_ce: 0.043294861912727356 (0.03965464163609738) | > loss_mel_ce: 3.2510106563568115 (nan) | > loss: 0.01307264156639576 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13659715474872794) | > loader_time: 0.0045 (0.004707319376854918)  --> TIME: 2025-02-24 15:21:54 -- STEP: 5420/36938 -- GLOBAL_STEP: 374800 | > loss_text_ce: 0.02967795543372631 (0.03964913837470762) | > loss_mel_ce: 0.929810643196106 (nan) | > loss: 0.0038074946496635675 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13654435512324506) | > loader_time: 0.0047 (0.004707792456299612)  --> TIME: 2025-02-24 15:22:04 -- STEP: 5470/36938 -- GLOBAL_STEP: 374850 | > loss_text_ce: 0.050702355802059174 (0.0396577099030002) | > loss_mel_ce: 3.204540491104126 (nan) | > loss: 0.012917631305754185 (nan) | > current_lr: 5e-06 | > step_time: 0.1266 (0.13655992944672038) | > loader_time: 0.0048 (0.004707875373812429)  --> TIME: 2025-02-24 15:22:13 -- STEP: 5520/36938 -- GLOBAL_STEP: 374900 | > loss_text_ce: 0.048673342913389206 (0.03966411275576548) | > loss_mel_ce: 3.1628010272979736 (nan) | > loss: 0.012743947096168995 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.1366180257952728) | > loader_time: 0.0046 (0.004708475090455309)  --> TIME: 2025-02-24 15:22:22 -- STEP: 5570/36938 -- GLOBAL_STEP: 374950 | > loss_text_ce: 0.03713636472821236 (0.03966304508006092) | > loss_mel_ce: 3.2709429264068604 (nan) | > loss: 0.013127299025654793 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.1365802758471966) | > loader_time: 0.0046 (0.004708670927763405)  --> TIME: 2025-02-24 15:22:32 -- STEP: 5620/36938 -- GLOBAL_STEP: 375000 | > loss_text_ce: 0.044736169278621674 (0.03966057292298406) | > loss_mel_ce: 3.640922784805298 (nan) | > loss: 0.014625631272792816 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.13658898712477197) | > loader_time: 0.0049 (0.004708318396395202)  --> TIME: 2025-02-24 15:22:41 -- STEP: 5670/36938 -- GLOBAL_STEP: 375050 | > loss_text_ce: 0.04051412642002106 (0.03964943662269371) | > loss_mel_ce: 2.07951283454895 (nan) | > loss: 0.008412806317210197 (nan) | > current_lr: 5e-06 | > step_time: 0.1086 (0.13659677724148622) | > loader_time: 0.0047 (0.004707495749942827)  --> TIME: 2025-02-24 15:22:50 -- STEP: 5720/36938 -- GLOBAL_STEP: 375100 | > loss_text_ce: 0.041950859129428864 (0.03964829277361154) | > loss_mel_ce: 2.685537815093994 (nan) | > loss: 0.010823369026184082 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13655800448430974) | > loader_time: 0.0046 (0.004706499043044496)  --> TIME: 2025-02-24 15:22:59 -- STEP: 5770/36938 -- GLOBAL_STEP: 375150 | > loss_text_ce: 0.03647841513156891 (0.03966331942897906) | > loss_mel_ce: 2.008549213409424 (nan) | > loss: 0.008115190081298351 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.1365507974988251) | > loader_time: 0.0047 (0.004706347422541927)  --> TIME: 2025-02-24 15:23:09 -- STEP: 5820/36938 -- GLOBAL_STEP: 375200 | > loss_text_ce: 0.047710247337818146 (0.039653462414946494) | > loss_mel_ce: 1.469415307044983 (nan) | > loss: 0.006020340137183666 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13656016215425978) | > loader_time: 0.0045 (0.004705680031137355)  --> TIME: 2025-02-24 15:23:18 -- STEP: 5870/36938 -- GLOBAL_STEP: 375250 | > loss_text_ce: 0.04023299738764763 (0.03965270010360984) | > loss_mel_ce: 3.3325424194335938 (nan) | > loss: 0.01338403020054102 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.1365573337577675) | > loader_time: 0.0046 (0.004704798750414336)  --> TIME: 2025-02-24 15:23:28 -- STEP: 5920/36938 -- GLOBAL_STEP: 375300 | > loss_text_ce: 0.04977056011557579 (0.03965856351439731) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13658761506950517) | > loader_time: 0.0045 (0.004703992846849782)  --> TIME: 2025-02-24 15:23:37 -- STEP: 5970/36938 -- GLOBAL_STEP: 375350 | > loss_text_ce: 0.04446988180279732 (0.03965654480575232) | > loss_mel_ce: 3.1803321838378906 (nan) | > loss: 0.012796834111213684 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.13659315380818474) | > loader_time: 0.0047 (0.004703287103866991)  --> TIME: 2025-02-24 15:23:46 -- STEP: 6020/36938 -- GLOBAL_STEP: 375400 | > loss_text_ce: 0.032368388026952744 (0.039665791979430785) | > loss_mel_ce: 3.479417085647583 (nan) | > loss: 0.01393565721809864 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13657782659974205) | > loader_time: 0.0044 (0.004703137011226997)  --> TIME: 2025-02-24 15:23:55 -- STEP: 6070/36938 -- GLOBAL_STEP: 375450 | > loss_text_ce: 0.04096763953566551 (0.03965410984533982) | > loss_mel_ce: 2.962672233581543 (nan) | > loss: 0.011919206939637661 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13657050670862564) | > loader_time: 0.0045 (0.00470299736475432)  --> TIME: 2025-02-24 15:24:05 -- STEP: 6120/36938 -- GLOBAL_STEP: 375500 | > loss_text_ce: 0.037552688270807266 (0.039661555309183326) | > loss_mel_ce: 2.9288527965545654 (nan) | > loss: 0.011771450750529766 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13655535200842048) | > loader_time: 0.0047 (0.004702314286450138)  --> TIME: 2025-02-24 15:24:14 -- STEP: 6170/36938 -- GLOBAL_STEP: 375550 | > loss_text_ce: 0.033704545348882675 (0.03966262001088378) | > loss_mel_ce: 3.5315957069396973 (nan) | > loss: 0.014148017391562462 (nan) | > current_lr: 5e-06 | > step_time: 0.188 (0.13654468600521974) | > loader_time: 0.0048 (0.004701861067761075)  --> TIME: 2025-02-24 15:24:23 -- STEP: 6220/36938 -- GLOBAL_STEP: 375600 | > loss_text_ce: 0.04279472678899765 (0.03965957134354168) | > loss_mel_ce: 2.3662109375 (nan) | > loss: 0.00955954659730196 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13653012667438225) | > loader_time: 0.0047 (0.004701178365198345)  --> TIME: 2025-02-24 15:24:33 -- STEP: 6270/36938 -- GLOBAL_STEP: 375650 | > loss_text_ce: 0.03192804381251335 (0.03964987598312044) | > loss_mel_ce: 3.2730557918548584 (nan) | > loss: 0.01311501581221819 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1365691240894737) | > loader_time: 0.0046 (0.0047007344366070385)  --> TIME: 2025-02-24 15:24:42 -- STEP: 6320/36938 -- GLOBAL_STEP: 375700 | > loss_text_ce: 0.03314374014735222 (0.039648917313853727) | > loss_mel_ce: 2.9747977256774902 (nan) | > loss: 0.011936276219785213 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.1365777776588367) | > loader_time: 0.0046 (0.004700392899633944)  --> TIME: 2025-02-24 15:24:51 -- STEP: 6370/36938 -- GLOBAL_STEP: 375750 | > loss_text_ce: 0.03652608394622803 (0.039653700275209175) | > loss_mel_ce: 2.919731378555298 (nan) | > loss: 0.011731180362403393 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13655961203612554) | > loader_time: 0.0047 (0.004699907010915308)  --> TIME: 2025-02-24 15:25:01 -- STEP: 6420/36938 -- GLOBAL_STEP: 375800 | > loss_text_ce: 0.03799464926123619 (0.03966216853358477) | > loss_mel_ce: 2.9813976287841797 (nan) | > loss: 0.011981716379523277 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13657968891001196) | > loader_time: 0.0045 (0.0047015351669810425)  --> TIME: 2025-02-24 15:25:10 -- STEP: 6470/36938 -- GLOBAL_STEP: 375850 | > loss_text_ce: 0.04048988223075867 (0.03966391475891356) | > loss_mel_ce: 0.5356200933456421 (nan) | > loss: 0.0022861510515213013 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13661281266573597) | > loader_time: 0.0054 (0.0047017352474159405)  --> TIME: 2025-02-24 15:25:20 -- STEP: 6520/36938 -- GLOBAL_STEP: 375900 | > loss_text_ce: 0.03443161025643349 (0.03965900024467723) | > loss_mel_ce: 3.263148546218872 (nan) | > loss: 0.01308563631027937 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13663430159077322) | > loader_time: 0.0046 (0.00470274010318918)  --> TIME: 2025-02-24 15:25:29 -- STEP: 6570/36938 -- GLOBAL_STEP: 375950 | > loss_text_ce: 0.03295310586690903 (0.039672025216281215) | > loss_mel_ce: 3.116766929626465 (nan) | > loss: 0.012498889118432999 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13666376359930857) | > loader_time: 0.0047 (0.004703550941142117)  --> TIME: 2025-02-24 15:25:39 -- STEP: 6620/36938 -- GLOBAL_STEP: 376000 | > loss_text_ce: 0.031246647238731384 (0.03966600046580335) | > loss_mel_ce: 2.5191073417663574 (nan) | > loss: 0.01012045331299305 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13669740447825168) | > loader_time: 0.0046 (0.004703240380186311)  --> TIME: 2025-02-24 15:25:48 -- STEP: 6670/36938 -- GLOBAL_STEP: 376050 | > loss_text_ce: 0.045641038566827774 (0.039663571908209884) | > loss_mel_ce: 3.167306661605835 (nan) | > loss: 0.012749793007969856 (nan) | > current_lr: 5e-06 | > step_time: 0.1889 (0.1366910311771834) | > loader_time: 0.0062 (0.004702807688105395)  --> TIME: 2025-02-24 15:25:57 -- STEP: 6720/36938 -- GLOBAL_STEP: 376100 | > loss_text_ce: 0.04256473481655121 (0.039666324399322966) | > loss_mel_ce: 3.6397879123687744 (nan) | > loss: 0.014612510800361633 (nan) | > current_lr: 5e-06 | > step_time: 0.1271 (0.13668955685127349) | > loader_time: 0.0046 (0.0047023308064256236)  --> TIME: 2025-02-24 15:26:07 -- STEP: 6770/36938 -- GLOBAL_STEP: 376150 | > loss_text_ce: 0.04636649042367935 (0.03966399215992698) | > loss_mel_ce: 3.383160352706909 (nan) | > loss: 0.013609234243631363 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.136691738935834) | > loader_time: 0.0045 (0.0047024305321088965)  --> TIME: 2025-02-24 15:26:16 -- STEP: 6820/36938 -- GLOBAL_STEP: 376200 | > loss_text_ce: 0.04326585680246353 (0.039668794946411824) | > loss_mel_ce: 3.019371509552002 (nan) | > loss: 0.012153323739767075 (nan) | > current_lr: 5e-06 | > step_time: 0.1904 (0.13670290736508853) | > loader_time: 0.0048 (0.0047026011950808775)  --> TIME: 2025-02-24 15:26:26 -- STEP: 6870/36938 -- GLOBAL_STEP: 376250 | > loss_text_ce: 0.026914052665233612 (0.039658596965564286) | > loss_mel_ce: 3.0743284225463867 (nan) | > loss: 0.012306518852710724 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13671997599900185) | > loader_time: 0.0045 (0.004702166421271123)  --> TIME: 2025-02-24 15:26:34 -- STEP: 6920/36938 -- GLOBAL_STEP: 376300 | > loss_text_ce: 0.03736511245369911 (0.03966620907085176) | > loss_mel_ce: 0.9114466905593872 (nan) | > loss: 0.003765126457437873 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.1366766523419086) | > loader_time: 0.0046 (0.0047014890723145655)  --> TIME: 2025-02-24 15:26:43 -- STEP: 6970/36938 -- GLOBAL_STEP: 376350 | > loss_text_ce: 0.036312613636255264 (0.03966829730905965) | > loss_mel_ce: 2.559738874435425 (nan) | > loss: 0.010301792062819004 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13665104408346232) | > loader_time: 0.0046 (0.004700932167523891)  --> TIME: 2025-02-24 15:26:53 -- STEP: 7020/36938 -- GLOBAL_STEP: 376400 | > loss_text_ce: 0.027904875576496124 (0.03966834981901904) | > loss_mel_ce: 2.827127695083618 (nan) | > loss: 0.01132949534803629 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13664523699344713) | > loader_time: 0.0049 (0.004700535009389578)  --> TIME: 2025-02-24 15:27:02 -- STEP: 7070/36938 -- GLOBAL_STEP: 376450 | > loss_text_ce: 0.04412570968270302 (0.03967426517554142) | > loss_mel_ce: 3.2219789028167725 (nan) | > loss: 0.012960733845829964 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.136660372047478) | > loader_time: 0.0046 (0.00470029366910035)  --> TIME: 2025-02-24 15:27:11 -- STEP: 7120/36938 -- GLOBAL_STEP: 376500 | > loss_text_ce: 0.03325227275490761 (0.039674301744501476) | > loss_mel_ce: 2.817268133163452 (nan) | > loss: 0.011311589740216732 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13665278661117078) | > loader_time: 0.0045 (0.004699939891193679)  --> TIME: 2025-02-24 15:27:21 -- STEP: 7170/36938 -- GLOBAL_STEP: 376550 | > loss_text_ce: 0.04283373802900314 (0.03967150775754675) | > loss_mel_ce: 2.7010085582733154 (nan) | > loss: 0.010888263583183289 (nan) | > current_lr: 5e-06 | > step_time: 0.1741 (0.13665077400340478) | > loader_time: 0.0046 (0.0046995233791143716)  --> TIME: 2025-02-24 15:27:30 -- STEP: 7220/36938 -- GLOBAL_STEP: 376600 | > loss_text_ce: 0.048786815255880356 (0.039680236166245046) | > loss_mel_ce: 2.4163877964019775 (nan) | > loss: 0.009782440029084682 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13660322820710966) | > loader_time: 0.0046 (0.004698880095230895)  --> TIME: 2025-02-24 15:27:39 -- STEP: 7270/36938 -- GLOBAL_STEP: 376650 | > loss_text_ce: 0.036277856677770615 (0.039679392269124585) | > loss_mel_ce: 0.0084228515625 (nan) | > loss: 0.00017738377209752798 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13656811986369596) | > loader_time: 0.0045 (0.0046978999692767216)  --> TIME: 2025-02-24 15:27:48 -- STEP: 7320/36938 -- GLOBAL_STEP: 376700 | > loss_text_ce: 0.04587515816092491 (0.03968192298433493) | > loss_mel_ce: 0.1106131449341774 (nan) | > loss: 0.0006209853454492986 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13657797138547612) | > loader_time: 0.0045 (0.004697124717014073)  --> TIME: 2025-02-24 15:27:57 -- STEP: 7370/36938 -- GLOBAL_STEP: 376750 | > loss_text_ce: 0.03711460903286934 (0.03967770771698436) | > loss_mel_ce: 1.5188626050949097 (nan) | > loss: 0.006174513138830662 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13656840143242532) | > loader_time: 0.0044 (0.004696171836904776)  --> TIME: 2025-02-24 15:28:06 -- STEP: 7420/36938 -- GLOBAL_STEP: 376800 | > loss_text_ce: 0.042181823402643204 (0.039672219903883574) | > loss_mel_ce: 3.089376449584961 (nan) | > loss: 0.01242681872099638 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13656116637258162) | > loader_time: 0.0045 (0.004695374335882788)  --> TIME: 2025-02-24 15:28:16 -- STEP: 7470/36938 -- GLOBAL_STEP: 376850 | > loss_text_ce: 0.03749309852719307 (0.03968314828583034) | > loss_mel_ce: 2.7696192264556885 (nan) | > loss: 0.011139335110783577 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13656296873667123) | > loader_time: 0.0048 (0.004694621757650301)  --> TIME: 2025-02-24 15:28:25 -- STEP: 7520/36938 -- GLOBAL_STEP: 376900 | > loss_text_ce: 0.03753633052110672 (0.03968120948733522) | > loss_mel_ce: 3.6417183876037598 (nan) | > loss: 0.014600218273699284 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13658130669847413) | > loader_time: 0.0046 (0.004694137357650909)  --> TIME: 2025-02-24 15:28:35 -- STEP: 7570/36938 -- GLOBAL_STEP: 376950 | > loss_text_ce: 0.03630216419696808 (0.039672365707875544) | > loss_mel_ce: 2.219099760055542 (nan) | > loss: 0.0089500080794096 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.1365832963424437) | > loader_time: 0.0047 (0.004693517596749985)  --> TIME: 2025-02-24 15:28:44 -- STEP: 7620/36938 -- GLOBAL_STEP: 377000 | > loss_text_ce: 0.03554588556289673 (0.039673987220547435) | > loss_mel_ce: 3.3482182025909424 (nan) | > loss: 0.01342763565480709 (nan) | > current_lr: 5e-06 | > step_time: 0.1725 (0.13660096136603733) | > loader_time: 0.0046 (0.004692943859600948)  --> TIME: 2025-02-24 15:28:53 -- STEP: 7670/36938 -- GLOBAL_STEP: 377050 | > loss_text_ce: 0.044061243534088135 (0.039675310325130046) | > loss_mel_ce: 0.015449181199073792 (nan) | > loss: 0.0002361524966545403 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1365923422722645) | > loader_time: 0.0052 (0.004692923105680015)  --> TIME: 2025-02-24 15:29:02 -- STEP: 7720/36938 -- GLOBAL_STEP: 377100 | > loss_text_ce: 0.03768924996256828 (0.03967207807919656) | > loss_mel_ce: 3.5537259578704834 (nan) | > loss: 0.01425164844840765 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.13656458694082432) | > loader_time: 0.0046 (0.004692642676397916)  --> TIME: 2025-02-24 15:29:12 -- STEP: 7770/36938 -- GLOBAL_STEP: 377150 | > loss_text_ce: 0.038329921662807465 (0.039673314372166264) | > loss_mel_ce: 0.2472941130399704 (nan) | > loss: 0.0011334287701174617 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1365881387791575) | > loader_time: 0.0045 (0.004692466532094742)  --> TIME: 2025-02-24 15:29:21 -- STEP: 7820/36938 -- GLOBAL_STEP: 377200 | > loss_text_ce: 0.04393695667386055 (0.03966769739601979) | > loss_mel_ce: 3.0088746547698975 (nan) | > loss: 0.012114332988858223 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13658902642062218) | > loader_time: 0.0045 (0.004692356513284346)  --> TIME: 2025-02-24 15:29:30 -- STEP: 7870/36938 -- GLOBAL_STEP: 377250 | > loss_text_ce: 0.03672610595822334 (0.039674404414851094) | > loss_mel_ce: 3.2401530742645264 (nan) | > loss: 0.013003489002585411 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13656345989076998) | > loader_time: 0.0053 (0.004692591736246993)  --> TIME: 2025-02-24 15:29:40 -- STEP: 7920/36938 -- GLOBAL_STEP: 377300 | > loss_text_ce: 0.03870774060487747 (0.03967233142883732) | > loss_mel_ce: 2.739675998687744 (nan) | > loss: 0.01102533284574747 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.13656924743242904) | > loader_time: 0.006 (0.004692599207463881)  --> TIME: 2025-02-24 15:29:49 -- STEP: 7970/36938 -- GLOBAL_STEP: 377350 | > loss_text_ce: 0.031535059213638306 (0.03966655194691592) | > loss_mel_ce: 2.6251304149627686 (nan) | > loss: 0.010542323812842369 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13655006092794233) | > loader_time: 0.0046 (0.004693180225424954)  --> TIME: 2025-02-24 15:29:58 -- STEP: 8020/36938 -- GLOBAL_STEP: 377400 | > loss_text_ce: 0.04613835737109184 (0.03967843955450056) | > loss_mel_ce: 3.0893030166625977 (nan) | > loss: 0.012442228384315968 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13654185524605125) | > loader_time: 0.0045 (0.004692659175901332)  --> TIME: 2025-02-24 15:30:08 -- STEP: 8070/36938 -- GLOBAL_STEP: 377450 | > loss_text_ce: 0.0351310670375824 (0.039679408589172564) | > loss_mel_ce: 3.3693575859069824 (nan) | > loss: 0.01350987609475851 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13657843112354648) | > loader_time: 0.0045 (0.004692451188703147)  --> TIME: 2025-02-24 15:30:17 -- STEP: 8120/36938 -- GLOBAL_STEP: 377500 | > loss_text_ce: 0.04262484982609749 (0.03967729703043792) | > loss_mel_ce: 2.0743062496185303 (nan) | > loss: 0.008400521241128445 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.1365659006417088) | > loader_time: 0.0046 (0.004691819751203931)  --> TIME: 2025-02-24 15:30:26 -- STEP: 8170/36938 -- GLOBAL_STEP: 377550 | > loss_text_ce: 0.041690852493047714 (0.039676263186205145) | > loss_mel_ce: 3.152843475341797 (nan) | > loss: 0.01267672423273325 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.1365647719277013) | > loader_time: 0.0045 (0.004691153027874156)  --> TIME: 2025-02-24 15:30:36 -- STEP: 8220/36938 -- GLOBAL_STEP: 377600 | > loss_text_ce: 0.046946171671152115 (0.039675836402292906) | > loss_mel_ce: 3.3851206302642822 (nan) | > loss: 0.013619313947856426 (nan) | > current_lr: 5e-06 | > step_time: 0.1667 (0.13658234875857653) | > loader_time: 0.0046 (0.004691023490144669)  --> TIME: 2025-02-24 15:30:45 -- STEP: 8270/36938 -- GLOBAL_STEP: 377650 | > loss_text_ce: 0.05653199180960655 (0.03967024623309379) | > loss_mel_ce: 3.6950225830078125 (nan) | > loss: 0.014887121506035328 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13656897593870673) | > loader_time: 0.0046 (0.004690986878926684)  --> TIME: 2025-02-24 15:30:54 -- STEP: 8320/36938 -- GLOBAL_STEP: 377700 | > loss_text_ce: 0.020311227068305016 (0.03966504823194833) | > loss_mel_ce: 3.034670352935791 (nan) | > loss: 0.012122943066060543 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.13654762165477613) | > loader_time: 0.0045 (0.004690304398536673)  --> TIME: 2025-02-24 15:31:03 -- STEP: 8370/36938 -- GLOBAL_STEP: 377750 | > loss_text_ce: 0.04960118979215622 (0.03966503583947407) | > loss_mel_ce: 2.893984794616699 (nan) | > loss: 0.011680897325277328 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13652683947960792) | > loader_time: 0.0047 (0.004689781583195794)  --> TIME: 2025-02-24 15:31:13 -- STEP: 8420/36938 -- GLOBAL_STEP: 377800 | > loss_text_ce: 0.04395216330885887 (0.039671867268133425) | > loss_mel_ce: 3.474210500717163 (nan) | > loss: 0.013960964046418667 (nan) | > current_lr: 5e-06 | > step_time: 0.1624 (0.1365399426066108) | > loader_time: 0.0047 (0.004690055665947186)  --> TIME: 2025-02-24 15:31:22 -- STEP: 8470/36938 -- GLOBAL_STEP: 377850 | > loss_text_ce: 0.03240801393985748 (0.039670203793837885) | > loss_mel_ce: 2.83671498298645 (nan) | > loss: 0.011385409161448479 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1365518987389914) | > loader_time: 0.0045 (0.004689680192940622)  --> TIME: 2025-02-24 15:31:31 -- STEP: 8520/36938 -- GLOBAL_STEP: 377900 | > loss_text_ce: 0.04417526721954346 (0.039682607736958815) | > loss_mel_ce: 2.2541911602020264 (nan) | > loss: 0.009120502509176731 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13655530734800914) | > loader_time: 0.0046 (0.004689286964040397)  --> TIME: 2025-02-24 15:31:40 -- STEP: 8570/36938 -- GLOBAL_STEP: 377950 | > loss_text_ce: 0.03300890699028969 (0.03968412568770963) | > loss_mel_ce: 3.428438186645508 (nan) | > loss: 0.013735901564359665 (nan) | > current_lr: 5e-06 | > step_time: 0.169 (0.13653623334446152) | > loader_time: 0.0047 (0.0046892163355681474)  --> TIME: 2025-02-24 15:31:50 -- STEP: 8620/36938 -- GLOBAL_STEP: 378000 | > loss_text_ce: 0.05379651486873627 (0.0396822916416196) | > loss_mel_ce: 1.7344344854354858 (nan) | > loss: 0.0070961550809443 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13655033097742844) | > loader_time: 0.0044 (0.0046891089105274095)  --> TIME: 2025-02-24 15:31:59 -- STEP: 8670/36938 -- GLOBAL_STEP: 378050 | > loss_text_ce: 0.04607020691037178 (0.03968063631119694) | > loss_mel_ce: 2.877004623413086 (nan) | > loss: 0.011599503457546234 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13655407219747046) | > loader_time: 0.0045 (0.004690015879744332)  --> TIME: 2025-02-24 15:32:09 -- STEP: 8720/36938 -- GLOBAL_STEP: 378100 | > loss_text_ce: 0.046425413340330124 (0.03968556296163654) | > loss_mel_ce: 0.025503264740109444 (nan) | > loss: 0.00028543127700686455 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13656745099146395) | > loader_time: 0.0045 (0.004690255703182388)  --> TIME: 2025-02-24 15:32:18 -- STEP: 8770/36938 -- GLOBAL_STEP: 378150 | > loss_text_ce: 0.027562668547034264 (0.039675337440539166) | > loss_mel_ce: 3.2690327167510986 (nan) | > loss: 0.013081728480756283 (nan) | > current_lr: 5e-06 | > step_time: 0.1263 (0.13659908945503332) | > loader_time: 0.0045 (0.004689967292472459)  --> TIME: 2025-02-24 15:32:28 -- STEP: 8820/36938 -- GLOBAL_STEP: 378200 | > loss_text_ce: 0.035538218915462494 (0.03966122165949516) | > loss_mel_ce: 3.0395946502685547 (nan) | > loss: 0.012202909216284752 (nan) | > current_lr: 5e-06 | > step_time: 0.1708 (0.13659870673199054) | > loader_time: 0.005 (0.004689885834717692)  --> TIME: 2025-02-24 15:32:37 -- STEP: 8870/36938 -- GLOBAL_STEP: 378250 | > loss_text_ce: 0.037682559341192245 (0.039652130883523) | > loss_mel_ce: 2.701007843017578 (nan) | > loss: 0.01086781919002533 (nan) | > current_lr: 5e-06 | > step_time: 0.1073 (0.1365818639484303) | > loader_time: 0.0045 (0.0046894403294215955)  --> TIME: 2025-02-24 15:32:46 -- STEP: 8920/36938 -- GLOBAL_STEP: 378300 | > loss_text_ce: 0.03575372323393822 (0.0396559676072965) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13657848998035516) | > loader_time: 0.0046 (0.004689461126455805)  --> TIME: 2025-02-24 15:32:56 -- STEP: 8970/36938 -- GLOBAL_STEP: 378350 | > loss_text_ce: 0.03023618459701538 (0.03965621952563439) | > loss_mel_ce: 1.926688313484192 (nan) | > loss: 0.00776557344943285 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13658187713644276) | > loader_time: 0.0046 (0.004689553483008268)  --> TIME: 2025-02-24 15:33:05 -- STEP: 9020/36938 -- GLOBAL_STEP: 378400 | > loss_text_ce: 0.03298989310860634 (0.039650132780803746) | > loss_mel_ce: 3.0658230781555176 (nan) | > loss: 0.012296877801418304 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.1365774714233074) | > loader_time: 0.0046 (0.004689529121847211)  --> TIME: 2025-02-24 15:33:14 -- STEP: 9070/36938 -- GLOBAL_STEP: 378450 | > loss_text_ce: 0.0388837531208992 (0.03965550243963363) | > loss_mel_ce: 3.086575746536255 (nan) | > loss: 0.012402617372572422 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13653346677336392) | > loader_time: 0.0047 (0.004689192272468066)  --> TIME: 2025-02-24 15:33:23 -- STEP: 9120/36938 -- GLOBAL_STEP: 378500 | > loss_text_ce: 0.03102392517030239 (0.039655366855399804) | > loss_mel_ce: 2.9610025882720947 (nan) | > loss: 0.011873122304677963 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1365325668140457) | > loader_time: 0.0045 (0.004689075471016396)  --> TIME: 2025-02-24 15:33:32 -- STEP: 9170/36938 -- GLOBAL_STEP: 378550 | > loss_text_ce: 0.06850702315568924 (0.03964297321554976) | > loss_mel_ce: 3.0135087966918945 (nan) | > loss: 0.0122302221134305 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.1365266897946169) | > loader_time: 0.0044 (0.004688764138466136)  --> TIME: 2025-02-24 15:33:41 -- STEP: 9220/36938 -- GLOBAL_STEP: 378600 | > loss_text_ce: 0.02890862338244915 (0.03963692944028176) | > loss_mel_ce: 3.471975088119507 (nan) | > loss: 0.013892397284507751 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13650476102456552) | > loader_time: 0.0045 (0.004688352798950124)  --> TIME: 2025-02-24 15:33:50 -- STEP: 9270/36938 -- GLOBAL_STEP: 378650 | > loss_text_ce: 0.055125392973423004 (0.03963530930331798) | > loss_mel_ce: 0.0362803153693676 (nan) | > loss: 0.0003627210680861026 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13647331221846892) | > loader_time: 0.0048 (0.004687857602149274)  --> TIME: 2025-02-24 15:33:59 -- STEP: 9320/36938 -- GLOBAL_STEP: 378700 | > loss_text_ce: 0.04592819884419441 (0.03963102078856698) | > loss_mel_ce: 2.148712158203125 (nan) | > loss: 0.008708890527486801 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1364685516756485) | > loader_time: 0.0045 (0.004687491634884615)  --> TIME: 2025-02-24 15:34:09 -- STEP: 9370/36938 -- GLOBAL_STEP: 378750 | > loss_text_ce: 0.04410446435213089 (0.039629757657583475) | > loss_mel_ce: 3.343876838684082 (nan) | > loss: 0.013444371521472931 (nan) | > current_lr: 5e-06 | > step_time: 0.1619 (0.13647877187840776) | > loader_time: 0.0046 (0.004687341783700655)  --> TIME: 2025-02-24 15:34:18 -- STEP: 9420/36938 -- GLOBAL_STEP: 378800 | > loss_text_ce: 0.040490832179784775 (0.039627755970759376) | > loss_mel_ce: 3.3185644149780273 (nan) | > loss: 0.013329585082828999 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.13648929846514496) | > loader_time: 0.0046 (0.00468694105775999)  --> TIME: 2025-02-24 15:34:27 -- STEP: 9470/36938 -- GLOBAL_STEP: 378850 | > loss_text_ce: 0.03964446112513542 (0.03963107589825029) | > loss_mel_ce: 3.6812500953674316 (nan) | > loss: 0.014765455387532711 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13647875405667012) | > loader_time: 0.0045 (0.004686421879745206)  --> TIME: 2025-02-24 15:34:37 -- STEP: 9520/36938 -- GLOBAL_STEP: 378900 | > loss_text_ce: 0.04813000559806824 (0.03962433887704299) | > loss_mel_ce: 3.5020346641540527 (nan) | > loss: 0.01408795639872551 (nan) | > current_lr: 5e-06 | > step_time: 0.1528 (0.13648726379170087) | > loader_time: 0.0053 (0.004686485218400713)  --> TIME: 2025-02-24 15:34:46 -- STEP: 9570/36938 -- GLOBAL_STEP: 378950 | > loss_text_ce: 0.03730805963277817 (0.03961920498334099) | > loss_mel_ce: 3.26240873336792 (nan) | > loss: 0.013094115070998669 (nan) | > current_lr: 5e-06 | > step_time: 0.1693 (0.1365027720153389) | > loader_time: 0.0048 (0.004686495079391304)  --> TIME: 2025-02-24 15:34:56 -- STEP: 9620/36938 -- GLOBAL_STEP: 379000 | > loss_text_ce: 0.04578213021159172 (0.03962413275869962) | > loss_mel_ce: 2.5291531085968018 (nan) | > loss: 0.010217997245490551 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13650138336011078) | > loader_time: 0.0046 (0.004686517155343927)  --> TIME: 2025-02-24 15:35:05 -- STEP: 9670/36938 -- GLOBAL_STEP: 379050 | > loss_text_ce: 0.024513673037290573 (0.03961570927586249) | > loss_mel_ce: 2.5480709075927734 (nan) | > loss: 0.010208670049905777 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13653209966365606) | > loader_time: 0.0047 (0.004686786741357306)  --> TIME: 2025-02-24 15:35:15 -- STEP: 9720/36938 -- GLOBAL_STEP: 379100 | > loss_text_ce: 0.031976643949747086 (0.039614905613578014) | > loss_mel_ce: 3.159572124481201 (nan) | > loss: 0.012664876878261566 (nan) | > current_lr: 5e-06 | > step_time: 0.1617 (0.13653794742906006) | > loader_time: 0.0049 (0.004687418908248712)  --> TIME: 2025-02-24 15:35:24 -- STEP: 9770/36938 -- GLOBAL_STEP: 379150 | > loss_text_ce: 0.033094730228185654 (0.039609422444908296) | > loss_mel_ce: 3.0836939811706543 (nan) | > loss: 0.012368209660053253 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13652355502547311) | > loader_time: 0.0045 (0.004688214499240279)  --> TIME: 2025-02-24 15:35:33 -- STEP: 9820/36938 -- GLOBAL_STEP: 379200 | > loss_text_ce: 0.04142298176884651 (0.039602714434812336) | > loss_mel_ce: 2.258401870727539 (nan) | > loss: 0.009126289747655392 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13655077066052418) | > loader_time: 0.0045 (0.004689224892866827)  --> TIME: 2025-02-24 15:35:43 -- STEP: 9870/36938 -- GLOBAL_STEP: 379250 | > loss_text_ce: 0.035148780792951584 (0.03960050977955226) | > loss_mel_ce: 4.005669116973877 (nan) | > loss: 0.016034992411732674 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.13656963456606983) | > loader_time: 0.0053 (0.004689657676062925)  --> TIME: 2025-02-24 15:35:52 -- STEP: 9920/36938 -- GLOBAL_STEP: 379300 | > loss_text_ce: 0.04151323810219765 (0.03960222636195527) | > loss_mel_ce: 3.753800392150879 (nan) | > loss: 0.015060769394040108 (nan) | > current_lr: 5e-06 | > step_time: 0.1857 (0.13655424461730012) | > loader_time: 0.0056 (0.004689990104206141)  --> TIME: 2025-02-24 15:36:02 -- STEP: 9970/36938 -- GLOBAL_STEP: 379350 | > loss_text_ce: 0.04055255651473999 (0.039601376808716414) | > loss_mel_ce: 2.699450731277466 (nan) | > loss: 0.010873029939830303 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13656974843177241) | > loader_time: 0.0045 (0.00469001389315039)  --> TIME: 2025-02-24 15:36:11 -- STEP: 10020/36938 -- GLOBAL_STEP: 379400 | > loss_text_ce: 0.024569010362029076 (0.03960804238974925) | > loss_mel_ce: 1.8123548030853271 (nan) | > loss: 0.007289380766451359 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13657315491202301) | > loader_time: 0.0044 (0.0046902181859501775)  --> TIME: 2025-02-24 15:36:20 -- STEP: 10070/36938 -- GLOBAL_STEP: 379450 | > loss_text_ce: 0.028905265033245087 (0.03960511707446455) | > loss_mel_ce: 3.595675230026245 (nan) | > loss: 0.014383256435394287 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13658319703398494) | > loader_time: 0.0053 (0.004690968623345992)  --> TIME: 2025-02-24 15:36:30 -- STEP: 10120/36938 -- GLOBAL_STEP: 379500 | > loss_text_ce: 0.03831060603260994 (0.0396032575562066) | > loss_mel_ce: 2.98130464553833 (nan) | > loss: 0.011982600204646587 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.1365752641862558) | > loader_time: 0.0046 (0.004691261265117666)  --> TIME: 2025-02-24 15:36:39 -- STEP: 10170/36938 -- GLOBAL_STEP: 379550 | > loss_text_ce: 0.04826199263334274 (0.039606676889892965) | > loss_mel_ce: 3.51611328125 (nan) | > loss: 0.014144347049295902 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.1365732896996105) | > loader_time: 0.0048 (0.00469189784988659)  --> TIME: 2025-02-24 15:36:48 -- STEP: 10220/36938 -- GLOBAL_STEP: 379600 | > loss_text_ce: 0.04362734779715538 (0.03960725849680234) | > loss_mel_ce: 3.0814857482910156 (nan) | > loss: 0.01240124274045229 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13657381739868327) | > loader_time: 0.0047 (0.0046923233338298205)  --> TIME: 2025-02-24 15:36:57 -- STEP: 10270/36938 -- GLOBAL_STEP: 379650 | > loss_text_ce: 0.03904604911804199 (0.03960737244451836) | > loss_mel_ce: 2.43544340133667 (nan) | > loss: 0.009819403290748596 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13653764234893837) | > loader_time: 0.0045 (0.004691697536680231)  --> TIME: 2025-02-24 15:37:07 -- STEP: 10320/36938 -- GLOBAL_STEP: 379700 | > loss_text_ce: 0.03393401950597763 (0.03960792765151272) | > loss_mel_ce: 0.12158203125 (nan) | > loss: 0.0006171272252686322 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13653988498587916) | > loader_time: 0.0045 (0.004691318601600888)  --> TIME: 2025-02-24 15:37:16 -- STEP: 10370/36938 -- GLOBAL_STEP: 379750 | > loss_text_ce: 0.04390175640583038 (0.03960569455713144) | > loss_mel_ce: 3.2758233547210693 (nan) | > loss: 0.013173513114452362 (nan) | > current_lr: 5e-06 | > step_time: 0.1499 (0.1365322087516893) | > loader_time: 0.0046 (0.004692069405874843)  --> TIME: 2025-02-24 15:37:25 -- STEP: 10420/36938 -- GLOBAL_STEP: 379800 | > loss_text_ce: 0.04060658812522888 (0.03960899073029941) | > loss_mel_ce: 3.1910462379455566 (nan) | > loss: 0.01282401941716671 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13652502426487914) | > loader_time: 0.0045 (0.0046928837981196625)  --> TIME: 2025-02-24 15:37:34 -- STEP: 10470/36938 -- GLOBAL_STEP: 379850 | > loss_text_ce: 0.03938344120979309 (0.03960773146648125) | > loss_mel_ce: 2.8581976890563965 (nan) | > loss: 0.011498338542878628 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13650869571718582) | > loader_time: 0.0046 (0.00469252735290965)  --> TIME: 2025-02-24 15:37:43 -- STEP: 10520/36938 -- GLOBAL_STEP: 379900 | > loss_text_ce: 0.04523635655641556 (0.03960275578873838) | > loss_mel_ce: 2.961458444595337 (nan) | > loss: 0.011931329034268856 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.13650580050827443) | > loader_time: 0.0046 (0.004693108275816019)  --> TIME: 2025-02-24 15:37:53 -- STEP: 10570/36938 -- GLOBAL_STEP: 379950 | > loss_text_ce: 0.04309946298599243 (0.039605371071185946) | > loss_mel_ce: 2.7242541313171387 (nan) | > loss: 0.010981562547385693 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.13654468945996778) | > loader_time: 0.0046 (0.0046927321261692)  --> TIME: 2025-02-24 15:38:02 -- STEP: 10620/36938 -- GLOBAL_STEP: 380000 | > loss_text_ce: 0.051794249564409256 (0.03960057696750243) | > loss_mel_ce: 3.6818485260009766 (nan) | > loss: 0.01481604389846325 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13651560464147805) | > loader_time: 0.0047 (0.004692167975359273) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_380000.pth  --> TIME: 2025-02-24 15:38:18 -- STEP: 10670/36938 -- GLOBAL_STEP: 380050 | > loss_text_ce: 0.03861298784613609 (0.03960751325002011) | > loss_mel_ce: 2.643217086791992 (nan) | > loss: 0.01064218394458294 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13650346946358793) | > loader_time: 0.0044 (0.0046921287838722295)  --> TIME: 2025-02-24 15:38:27 -- STEP: 10720/36938 -- GLOBAL_STEP: 380100 | > loss_text_ce: 0.04015854373574257 (0.03960930397534217) | > loss_mel_ce: 2.5679683685302734 (nan) | > loss: 0.01034971047192812 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.1364926176062271) | > loader_time: 0.0046 (0.004691834036094044)  --> TIME: 2025-02-24 15:38:36 -- STEP: 10770/36938 -- GLOBAL_STEP: 380150 | > loss_text_ce: 0.028966907411813736 (0.03960019773021053) | > loss_mel_ce: 2.4792280197143555 (nan) | > loss: 0.009953155182301998 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.13650219329773775) | > loader_time: 0.0046 (0.004692116509794419)  --> TIME: 2025-02-24 15:38:45 -- STEP: 10820/36938 -- GLOBAL_STEP: 380200 | > loss_text_ce: 0.03873582184314728 (0.03959508707562655) | > loss_mel_ce: 3.093380928039551 (nan) | > loss: 0.012429035268723965 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.136490932233673) | > loader_time: 0.0046 (0.004692154979529536)  --> TIME: 2025-02-24 15:38:55 -- STEP: 10870/36938 -- GLOBAL_STEP: 380250 | > loss_text_ce: 0.04926060512661934 (0.03959745704132126) | > loss_mel_ce: 3.5658278465270996 (nan) | > loss: 0.014345590025186539 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.1365154241791756) | > loader_time: 0.0046 (0.004692094240478129)  --> TIME: 2025-02-24 15:39:04 -- STEP: 10920/36938 -- GLOBAL_STEP: 380300 | > loss_text_ce: 0.043253324925899506 (0.0395933377011525) | > loss_mel_ce: 2.939770460128784 (nan) | > loss: 0.011837396770715714 (nan) | > current_lr: 5e-06 | > step_time: 0.1827 (0.13651191985214162) | > loader_time: 0.0049 (0.004691695206331252)  --> TIME: 2025-02-24 15:39:13 -- STEP: 10970/36938 -- GLOBAL_STEP: 380350 | > loss_text_ce: 0.03425400331616402 (0.03959697952592504) | > loss_mel_ce: 3.1896634101867676 (nan) | > loss: 0.012793323956429958 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13650937347707703) | > loader_time: 0.0046 (0.004691211614374047)  --> TIME: 2025-02-24 15:39:22 -- STEP: 11020/36938 -- GLOBAL_STEP: 380400 | > loss_text_ce: 0.04699605703353882 (0.03959732907824214) | > loss_mel_ce: 0.18203124403953552 (nan) | > loss: 0.0009088385268114507 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13649550072727115) | > loader_time: 0.0045 (0.004692175686900289)  --> TIME: 2025-02-24 15:39:32 -- STEP: 11070/36938 -- GLOBAL_STEP: 380450 | > loss_text_ce: 0.04224837198853493 (0.039597389794963495) | > loss_mel_ce: 2.252246618270874 (nan) | > loss: 0.009105139411985874 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.136490501534756) | > loader_time: 0.0047 (0.0046923736677368)  --> TIME: 2025-02-24 15:39:41 -- STEP: 11120/36938 -- GLOBAL_STEP: 380500 | > loss_text_ce: 0.025586280971765518 (0.03960448577790687) | > loss_mel_ce: 2.405066967010498 (nan) | > loss: 0.009645449928939342 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13648093627939997) | > loader_time: 0.0064 (0.0046925334407271315)  --> TIME: 2025-02-24 15:39:50 -- STEP: 11170/36938 -- GLOBAL_STEP: 380550 | > loss_text_ce: 0.051097720861434937 (0.039603857769261466) | > loss_mel_ce: 3.6470775604248047 (nan) | > loss: 0.014675299637019634 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13648303965958733) | > loader_time: 0.0046 (0.004692241349386768)  --> TIME: 2025-02-24 15:39:59 -- STEP: 11220/36938 -- GLOBAL_STEP: 380600 | > loss_text_ce: 0.03978392109274864 (0.03960918463740007) | > loss_mel_ce: 3.4246506690979004 (nan) | > loss: 0.01374775730073452 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13648040148026183) | > loader_time: 0.0046 (0.004692574894577008)  --> TIME: 2025-02-24 15:40:09 -- STEP: 11270/36938 -- GLOBAL_STEP: 380650 | > loss_text_ce: 0.036670099943876266 (0.039604514271364584) | > loss_mel_ce: 0.021361803635954857 (nan) | > loss: 0.00023028533905744553 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13648006882197905) | > loader_time: 0.0045 (0.004693072055860661)  --> TIME: 2025-02-24 15:40:18 -- STEP: 11320/36938 -- GLOBAL_STEP: 380700 | > loss_text_ce: 0.04383477568626404 (0.03960856875575834) | > loss_mel_ce: 2.98447322845459 (nan) | > loss: 0.01201709546148777 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1364685315125401) | > loader_time: 0.0045 (0.004693500650223918)  --> TIME: 2025-02-24 15:40:27 -- STEP: 11370/36938 -- GLOBAL_STEP: 380750 | > loss_text_ce: 0.038227200508117676 (0.03960402558443828) | > loss_mel_ce: 2.560835123062134 (nan) | > loss: 0.010313740000128746 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.1364835295639441) | > loader_time: 0.0046 (0.004693951791278506)  --> TIME: 2025-02-24 15:40:37 -- STEP: 11420/36938 -- GLOBAL_STEP: 380800 | > loss_text_ce: 0.04206136614084244 (0.03959729552265714) | > loss_mel_ce: 2.6273224353790283 (nan) | > loss: 0.010592793114483356 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1364894760677986) | > loader_time: 0.0045 (0.0046940770750496805)  --> TIME: 2025-02-24 15:40:46 -- STEP: 11470/36938 -- GLOBAL_STEP: 380850 | > loss_text_ce: 0.03510304167866707 (0.03960090355887524) | > loss_mel_ce: 3.216373920440674 (nan) | > loss: 0.012902687303721905 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13646153796727287) | > loader_time: 0.0045 (0.004693834097778273)  --> TIME: 2025-02-24 15:40:55 -- STEP: 11520/36938 -- GLOBAL_STEP: 380900 | > loss_text_ce: 0.03266868367791176 (0.039604480844961154) | > loss_mel_ce: 3.0896034240722656 (nan) | > loss: 0.01238996908068657 (nan) | > current_lr: 5e-06 | > step_time: 0.1591 (0.13647259825633642) | > loader_time: 0.0048 (0.004693425446748738)  --> TIME: 2025-02-24 15:41:04 -- STEP: 11570/36938 -- GLOBAL_STEP: 380950 | > loss_text_ce: 0.03937740996479988 (0.03960677217305762) | > loss_mel_ce: 3.017545461654663 (nan) | > loss: 0.012130646966397762 (nan) | > current_lr: 5e-06 | > step_time: 0.1613 (0.13645503040831378) | > loader_time: 0.0048 (0.004693048867874349)  --> TIME: 2025-02-24 15:41:13 -- STEP: 11620/36938 -- GLOBAL_STEP: 381000 | > loss_text_ce: 0.0385495163500309 (0.039615133876323996) | > loss_mel_ce: 3.0562121868133545 (nan) | > loss: 0.012280800379812717 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13643225712538193) | > loader_time: 0.0046 (0.004693554416170633)  --> TIME: 2025-02-24 15:41:23 -- STEP: 11670/36938 -- GLOBAL_STEP: 381050 | > loss_text_ce: 0.04741039499640465 (0.03961323869335625) | > loss_mel_ce: 3.40046763420105 (nan) | > loss: 0.013682057149708271 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.1364255359260672) | > loader_time: 0.0047 (0.004693584107086002)  --> TIME: 2025-02-24 15:41:32 -- STEP: 11720/36938 -- GLOBAL_STEP: 381100 | > loss_text_ce: 0.03896617889404297 (0.03961493984300251) | > loss_mel_ce: 3.3324763774871826 (nan) | > loss: 0.013378741219639778 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13645332154560427) | > loader_time: 0.0046 (0.004694014889388361)  --> TIME: 2025-02-24 15:41:41 -- STEP: 11770/36938 -- GLOBAL_STEP: 381150 | > loss_text_ce: 0.036000095307826996 (0.03960983135556277) | > loss_mel_ce: 2.835078239440918 (nan) | > loss: 0.0113931680098176 (nan) | > current_lr: 5e-06 | > step_time: 0.1717 (0.13644981688124316) | > loader_time: 0.0048 (0.004694768485115464)  --> TIME: 2025-02-24 15:41:51 -- STEP: 11820/36938 -- GLOBAL_STEP: 381200 | > loss_text_ce: 0.04569769650697708 (0.03961026973757648) | > loss_mel_ce: 3.4162752628326416 (nan) | > loss: 0.013737988658249378 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13645223250042174) | > loader_time: 0.0048 (0.004695593282050895)  --> TIME: 2025-02-24 15:42:01 -- STEP: 11870/36938 -- GLOBAL_STEP: 381250 | > loss_text_ce: 0.04514898732304573 (0.039605242412666046) | > loss_mel_ce: 3.054140329360962 (nan) | > loss: 0.012298768386244774 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13648347002975864) | > loader_time: 0.0046 (0.004695922884503235)  --> TIME: 2025-02-24 15:42:10 -- STEP: 11920/36938 -- GLOBAL_STEP: 381300 | > loss_text_ce: 0.03134789690375328 (0.03960684867272621) | > loss_mel_ce: 2.6190412044525146 (nan) | > loss: 0.010517418384552002 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13648052751617956) | > loader_time: 0.0045 (0.00469591393566772)  --> TIME: 2025-02-24 15:42:19 -- STEP: 11970/36938 -- GLOBAL_STEP: 381350 | > loss_text_ce: 0.04639548808336258 (0.039602711759725545) | > loss_mel_ce: 3.1056292057037354 (nan) | > loss: 0.012508035637438297 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13649557196506398) | > loader_time: 0.0047 (0.004696302983595517)  --> TIME: 2025-02-24 15:42:29 -- STEP: 12020/36938 -- GLOBAL_STEP: 381400 | > loss_text_ce: 0.039190903306007385 (0.03959934398510221) | > loss_mel_ce: 2.945746898651123 (nan) | > loss: 0.011844992637634277 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13649235100992121) | > loader_time: 0.0046 (0.004696640952454629)  --> TIME: 2025-02-24 15:42:38 -- STEP: 12070/36938 -- GLOBAL_STEP: 381450 | > loss_text_ce: 0.03545583784580231 (0.03960171189072742) | > loss_mel_ce: 3.3379738330841064 (nan) | > loss: 0.013386626727879047 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13649390883552426) | > loader_time: 0.0065 (0.00469710344900844)  --> TIME: 2025-02-24 15:42:47 -- STEP: 12120/36938 -- GLOBAL_STEP: 381500 | > loss_text_ce: 0.032572004944086075 (0.03959376119621253) | > loss_mel_ce: 3.3929905891418457 (nan) | > loss: 0.013593503274023533 (nan) | > current_lr: 5e-06 | > step_time: 0.1619 (0.13647990980164085) | > loader_time: 0.0047 (0.0046976890894446085)  --> TIME: 2025-02-24 15:42:56 -- STEP: 12170/36938 -- GLOBAL_STEP: 381550 | > loss_text_ce: 0.039151404052972794 (0.03959455412729125) | > loss_mel_ce: 3.0734102725982666 (nan) | > loss: 0.012351435609161854 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13647016650257124) | > loader_time: 0.0046 (0.004697923338991168)  --> TIME: 2025-02-24 15:43:06 -- STEP: 12220/36938 -- GLOBAL_STEP: 381600 | > loss_text_ce: 0.04287481680512428 (0.03959719133174663) | > loss_mel_ce: 2.4759879112243652 (nan) | > loss: 0.00999548751860857 (nan) | > current_lr: 5e-06 | > step_time: 0.1264 (0.13648529983386296) | > loader_time: 0.0046 (0.004697851912284646)  --> TIME: 2025-02-24 15:43:15 -- STEP: 12270/36938 -- GLOBAL_STEP: 381650 | > loss_text_ce: 0.04155987128615379 (0.03959250829403449) | > loss_mel_ce: 0.19021987915039062 (nan) | > loss: 0.0009197609615512192 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13648269265865848) | > loader_time: 0.0046 (0.0046975746131467985)  --> TIME: 2025-02-24 15:43:24 -- STEP: 12320/36938 -- GLOBAL_STEP: 381700 | > loss_text_ce: 0.047358326613903046 (0.03958386525257755) | > loss_mel_ce: 3.2402095794677734 (nan) | > loss: 0.013045905157923698 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13645191049420988) | > loader_time: 0.0046 (0.0046974438932034885)  --> TIME: 2025-02-24 15:43:33 -- STEP: 12370/36938 -- GLOBAL_STEP: 381750 | > loss_text_ce: 0.033657584339380264 (0.039580816101876234) | > loss_mel_ce: 1.5438565015792847 (nan) | > loss: 0.006259976886212826 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13643245774117232) | > loader_time: 0.0045 (0.004697270054150323)  --> TIME: 2025-02-24 15:43:42 -- STEP: 12420/36938 -- GLOBAL_STEP: 381800 | > loss_text_ce: 0.03689970076084137 (0.03958153498438521) | > loss_mel_ce: 3.185129404067993 (nan) | > loss: 0.012785830534994602 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.13642764511891614) | > loader_time: 0.0048 (0.0046975048651825765)  --> TIME: 2025-02-24 15:43:51 -- STEP: 12470/36938 -- GLOBAL_STEP: 381850 | > loss_text_ce: 0.028400534763932228 (0.0395807375299425) | > loss_mel_ce: 2.371044158935547 (nan) | > loss: 0.009521606378257275 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13643741915487934) | > loader_time: 0.0045 (0.004697427351950832)  --> TIME: 2025-02-24 15:44:01 -- STEP: 12520/36938 -- GLOBAL_STEP: 381900 | > loss_text_ce: 0.043342042714357376 (0.03958091357612795) | > loss_mel_ce: 1.9296125173568726 (nan) | > loss: 0.007829184643924236 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13644986251672636) | > loader_time: 0.0045 (0.004697623725135483)  --> TIME: 2025-02-24 15:44:10 -- STEP: 12570/36938 -- GLOBAL_STEP: 381950 | > loss_text_ce: 0.029293479397892952 (0.03958366811171765) | > loss_mel_ce: 2.9928476810455322 (nan) | > loss: 0.011992624960839748 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1364436592293241) | > loader_time: 0.0045 (0.00469782762140533)  --> TIME: 2025-02-24 15:44:20 -- STEP: 12620/36938 -- GLOBAL_STEP: 382000 | > loss_text_ce: 0.04879871383309364 (0.03958451600255315) | > loss_mel_ce: 3.0025250911712646 (nan) | > loss: 0.01210842840373516 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.1364508015418014) | > loader_time: 0.0047 (0.004698738469941484)  --> TIME: 2025-02-24 15:44:29 -- STEP: 12670/36938 -- GLOBAL_STEP: 382050 | > loss_text_ce: 0.044300198554992676 (0.039587552379596304) | > loss_mel_ce: 3.63032865524292 (nan) | > loss: 0.014581860974431038 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.136448700520215) | > loader_time: 0.0046 (0.004698727802677044)  --> TIME: 2025-02-24 15:44:38 -- STEP: 12720/36938 -- GLOBAL_STEP: 382100 | > loss_text_ce: 0.042330194264650345 (0.039580706780067705) | > loss_mel_ce: 2.8692827224731445 (nan) | > loss: 0.011554020456969738 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13644543065215056) | > loader_time: 0.0047 (0.004699058746391869)  --> TIME: 2025-02-24 15:44:47 -- STEP: 12770/36938 -- GLOBAL_STEP: 382150 | > loss_text_ce: 0.041220102459192276 (0.039581382129933645) | > loss_mel_ce: 3.2981815338134766 (nan) | > loss: 0.013251595199108124 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.13645146804203157) | > loader_time: 0.0048 (0.004699569954569671)  --> TIME: 2025-02-24 15:44:57 -- STEP: 12820/36938 -- GLOBAL_STEP: 382200 | > loss_text_ce: 0.03880573436617851 (0.03958311318441675) | > loss_mel_ce: 3.2261595726013184 (nan) | > loss: 0.012956212274730206 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13643711130258246) | > loader_time: 0.0046 (0.0047001160995078705)  --> TIME: 2025-02-24 15:45:06 -- STEP: 12870/36938 -- GLOBAL_STEP: 382250 | > loss_text_ce: 0.03940357640385628 (0.03958239937570206) | > loss_mel_ce: 0.0803680419921875 (nan) | > loss: 0.00047528420691378415 (nan) | > current_lr: 5e-06 | > step_time: 0.1688 (0.13642845618789642) | > loader_time: 0.005 (0.004699759327726209)  --> TIME: 2025-02-24 15:45:15 -- STEP: 12920/36938 -- GLOBAL_STEP: 382300 | > loss_text_ce: 0.051814958453178406 (0.03957823979201223) | > loss_mel_ce: 3.234602689743042 (nan) | > loss: 0.013041340745985508 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.1364353721355873) | > loader_time: 0.0045 (0.004699485479124564)  --> TIME: 2025-02-24 15:45:24 -- STEP: 12970/36938 -- GLOBAL_STEP: 382350 | > loss_text_ce: 0.03722689673304558 (0.039576056850937735) | > loss_mel_ce: 3.1187267303466797 (nan) | > loss: 0.01252362597733736 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.13643009590937136) | > loader_time: 0.0047 (0.004699143613405018)  --> TIME: 2025-02-24 15:45:34 -- STEP: 13020/36938 -- GLOBAL_STEP: 382400 | > loss_text_ce: 0.05818890035152435 (0.03957505759750192) | > loss_mel_ce: 3.3444480895996094 (nan) | > loss: 0.013502528890967369 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13643635392372125) | > loader_time: 0.0046 (0.0046990535409402925)  --> TIME: 2025-02-24 15:45:43 -- STEP: 13070/36938 -- GLOBAL_STEP: 382450 | > loss_text_ce: 0.04834839329123497 (0.039578060023971226) | > loss_mel_ce: 3.9630825519561768 (nan) | > loss: 0.01591837778687477 (nan) | > current_lr: 5e-06 | > step_time: 0.1882 (0.13643210258936272) | > loader_time: 0.0049 (0.004698996719004274)  --> TIME: 2025-02-24 15:45:52 -- STEP: 13120/36938 -- GLOBAL_STEP: 382500 | > loss_text_ce: 0.0419284962117672 (0.03957546331022487) | > loss_mel_ce: 2.25738525390625 (nan) | > loss: 0.00912426132708788 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13643075825601078) | > loader_time: 0.0045 (0.004698765568616915)  --> TIME: 2025-02-24 15:46:02 -- STEP: 13170/36938 -- GLOBAL_STEP: 382550 | > loss_text_ce: 0.033054739236831665 (0.039577650648507164) | > loss_mel_ce: 3.3488950729370117 (nan) | > loss: 0.013420436531305313 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.1364384041107687) | > loader_time: 0.0046 (0.004698691426033131)  --> TIME: 2025-02-24 15:46:11 -- STEP: 13220/36938 -- GLOBAL_STEP: 382600 | > loss_text_ce: 0.03798394650220871 (0.03957700879945682) | > loss_mel_ce: 3.078423261642456 (nan) | > loss: 0.012366695329546928 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.1364377441637094) | > loader_time: 0.0046 (0.0046984083713814285)  --> TIME: 2025-02-24 15:46:20 -- STEP: 13270/36938 -- GLOBAL_STEP: 382650 | > loss_text_ce: 0.034195370972156525 (0.039573422402491655) | > loss_mel_ce: 4.189732074737549 (nan) | > loss: 0.016761617735028267 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13643520866557893) | > loader_time: 0.0045 (0.004698511255173733)  --> TIME: 2025-02-24 15:46:30 -- STEP: 13320/36938 -- GLOBAL_STEP: 382700 | > loss_text_ce: 0.04333009198307991 (0.03957001427803811) | > loss_mel_ce: 3.6056461334228516 (nan) | > loss: 0.014480065554380417 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13644956359276186) | > loader_time: 0.0048 (0.00469818926072336)  --> TIME: 2025-02-24 15:46:39 -- STEP: 13370/36938 -- GLOBAL_STEP: 382750 | > loss_text_ce: 0.04312869533896446 (0.039570868648477274) | > loss_mel_ce: 2.6197268962860107 (nan) | > loss: 0.010566888377070427 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.1364380570249764) | > loader_time: 0.0047 (0.004698500280159133)  --> TIME: 2025-02-24 15:46:48 -- STEP: 13420/36938 -- GLOBAL_STEP: 382800 | > loss_text_ce: 0.04179945960640907 (0.0395757308660073) | > loss_mel_ce: 3.1537904739379883 (nan) | > loss: 0.01268091332167387 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.13644553351864197) | > loader_time: 0.0048 (0.004698233597324791)  --> TIME: 2025-02-24 15:46:57 -- STEP: 13470/36938 -- GLOBAL_STEP: 382850 | > loss_text_ce: 0.03515582159161568 (0.039574012589248954) | > loss_mel_ce: 2.4774680137634277 (nan) | > loss: 0.009970730170607567 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.13641197686736997) | > loader_time: 0.0045 (0.004697825772371133)  --> TIME: 2025-02-24 15:47:07 -- STEP: 13520/36938 -- GLOBAL_STEP: 382900 | > loss_text_ce: 0.04725427180528641 (0.03957556716948975) | > loss_mel_ce: 3.288621664047241 (nan) | > loss: 0.013237603940069675 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13642671399215275) | > loader_time: 0.0045 (0.004697950598756242)  --> TIME: 2025-02-24 15:47:16 -- STEP: 13570/36938 -- GLOBAL_STEP: 382950 | > loss_text_ce: 0.03160213679075241 (0.039576850296873266) | > loss_mel_ce: 3.189286947250366 (nan) | > loss: 0.012781307101249695 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1364201224017162) | > loader_time: 0.0045 (0.0046978770939848785)  --> TIME: 2025-02-24 15:47:25 -- STEP: 13620/36938 -- GLOBAL_STEP: 383000 | > loss_text_ce: 0.037489186972379684 (0.03958101638383547) | > loss_mel_ce: 2.77044415473938 (nan) | > loss: 0.011142592877149582 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13642191001147094) | > loader_time: 0.0047 (0.00469794637481197)  --> TIME: 2025-02-24 15:47:34 -- STEP: 13670/36938 -- GLOBAL_STEP: 383050 | > loss_text_ce: 0.029544273391366005 (0.03957837940355185) | > loss_mel_ce: 1.9686156511306763 (nan) | > loss: 0.007929205894470215 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13642675308040603) | > loader_time: 0.0045 (0.004697988097826414)  --> TIME: 2025-02-24 15:47:44 -- STEP: 13720/36938 -- GLOBAL_STEP: 383100 | > loss_text_ce: 0.03490234911441803 (0.03957881626276626) | > loss_mel_ce: 1.0736192464828491 (nan) | > loss: 0.004398895427584648 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13643059892140086) | > loader_time: 0.0045 (0.004698433246974006)  --> TIME: 2025-02-24 15:47:53 -- STEP: 13770/36938 -- GLOBAL_STEP: 383150 | > loss_text_ce: 0.033095039427280426 (0.039579835659895396) | > loss_mel_ce: 3.399744749069214 (nan) | > loss: 0.013622380793094635 (nan) | > current_lr: 5e-06 | > step_time: 0.1714 (0.13643868419686037) | > loader_time: 0.0053 (0.004698488049171249)  --> TIME: 2025-02-24 15:48:02 -- STEP: 13820/36938 -- GLOBAL_STEP: 383200 | > loss_text_ce: 0.04544292017817497 (0.039576787844788235) | > loss_mel_ce: 3.440548896789551 (nan) | > loss: 0.013833301141858101 (nan) | > current_lr: 5e-06 | > step_time: 0.1679 (0.1364275220506618) | > loader_time: 0.0047 (0.00469804032666639)  --> TIME: 2025-02-24 15:48:12 -- STEP: 13870/36938 -- GLOBAL_STEP: 383250 | > loss_text_ce: 0.04079412668943405 (0.03957313365889176) | > loss_mel_ce: 2.9180448055267334 (nan) | > loss: 0.011741424910724163 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.13642788473100403) | > loader_time: 0.0047 (0.004698222785350143)  --> TIME: 2025-02-24 15:48:21 -- STEP: 13920/36938 -- GLOBAL_STEP: 383300 | > loss_text_ce: 0.02890537492930889 (0.039566251357693566) | > loss_mel_ce: 3.2682292461395264 (nan) | > loss: 0.01308386866003275 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13641898743380074) | > loader_time: 0.0046 (0.0046982833023729055)  --> TIME: 2025-02-24 15:48:30 -- STEP: 13970/36938 -- GLOBAL_STEP: 383350 | > loss_text_ce: 0.032981302589178085 (0.039565849885592565) | > loss_mel_ce: 3.11362624168396 (nan) | > loss: 0.012486538849771023 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.1364133702744393) | > loader_time: 0.0046 (0.004698351287978335)  --> TIME: 2025-02-24 15:48:39 -- STEP: 14020/36938 -- GLOBAL_STEP: 383400 | > loss_text_ce: 0.03899253159761429 (0.03956290492675036) | > loss_mel_ce: 3.5826871395111084 (nan) | > loss: 0.014371746219694614 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13642016159484796) | > loader_time: 0.0054 (0.004698545684487952)  --> TIME: 2025-02-24 15:48:49 -- STEP: 14070/36938 -- GLOBAL_STEP: 383450 | > loss_text_ce: 0.046018581837415695 (0.03955778570497559) | > loss_mel_ce: 3.1853809356689453 (nan) | > loss: 0.0128230145201087 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13643391352641973) | > loader_time: 0.0046 (0.004698936618975745)  --> TIME: 2025-02-24 15:48:58 -- STEP: 14120/36938 -- GLOBAL_STEP: 383500 | > loss_text_ce: 0.03886299207806587 (0.03956608256238068) | > loss_mel_ce: 3.4433250427246094 (nan) | > loss: 0.013818207196891308 (nan) | > current_lr: 5e-06 | > step_time: 0.1619 (0.13642648732020546) | > loader_time: 0.0047 (0.004699096666179371)  --> TIME: 2025-02-24 15:49:07 -- STEP: 14170/36938 -- GLOBAL_STEP: 383550 | > loss_text_ce: 0.04340313374996185 (0.039567760044633984) | > loss_mel_ce: 3.06134295463562 (nan) | > loss: 0.012320421636104584 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.13642920142088266) | > loader_time: 0.0047 (0.004699278803214952)  --> TIME: 2025-02-24 15:49:16 -- STEP: 14220/36938 -- GLOBAL_STEP: 383600 | > loss_text_ce: 0.03914925828576088 (0.03956211811342871) | > loss_mel_ce: 1.4900144338607788 (nan) | > loss: 0.006068110466003418 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13641143416888904) | > loader_time: 0.0046 (0.004699562990883406)  --> TIME: 2025-02-24 15:49:26 -- STEP: 14270/36938 -- GLOBAL_STEP: 383650 | > loss_text_ce: 0.03256816044449806 (0.039564090997852665) | > loss_mel_ce: 3.0230047702789307 (nan) | > loss: 0.012125289998948574 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13643456557954842) | > loader_time: 0.0046 (0.0046997760172565305)  --> TIME: 2025-02-24 15:49:36 -- STEP: 14320/36938 -- GLOBAL_STEP: 383700 | > loss_text_ce: 0.03476742282509804 (0.03956350922914925) | > loss_mel_ce: 1.3642760515213013 (nan) | > loss: 0.005551760084927082 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13643850945560596) | > loader_time: 0.0045 (0.004699911468521849)  --> TIME: 2025-02-24 15:49:45 -- STEP: 14370/36938 -- GLOBAL_STEP: 383750 | > loss_text_ce: 0.036538053303956985 (0.03956213338377068) | > loss_mel_ce: 3.485283851623535 (nan) | > loss: 0.013975485228002071 (nan) | > current_lr: 5e-06 | > step_time: 0.1606 (0.13644691530664316) | > loader_time: 0.0046 (0.004699660658255671)  --> TIME: 2025-02-24 15:49:54 -- STEP: 14420/36938 -- GLOBAL_STEP: 383800 | > loss_text_ce: 0.05821359530091286 (0.03956259909071727) | > loss_mel_ce: 2.263214111328125 (nan) | > loss: 0.009212016128003597 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1364426588714533) | > loader_time: 0.0045 (0.0046993672930417275)  --> TIME: 2025-02-24 15:50:03 -- STEP: 14470/36938 -- GLOBAL_STEP: 383850 | > loss_text_ce: 0.052083566784858704 (0.039565214385750364) | > loss_mel_ce: 0.2265625 (nan) | > loss: 0.0011057383380830288 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13642733072361266) | > loader_time: 0.0044 (0.00469953781502282)  --> TIME: 2025-02-24 15:50:13 -- STEP: 14520/36938 -- GLOBAL_STEP: 383900 | > loss_text_ce: 0.04273874685168266 (0.039561940922040156) | > loss_mel_ce: 2.7389140129089355 (nan) | > loss: 0.01103830523788929 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13642630985945686) | > loader_time: 0.0047 (0.004699772103758889)  --> TIME: 2025-02-24 15:50:22 -- STEP: 14570/36938 -- GLOBAL_STEP: 383950 | > loss_text_ce: 0.036865830421447754 (0.03955888607876104) | > loss_mel_ce: 2.7143802642822266 (nan) | > loss: 0.010917643085122108 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1364277223895979) | > loader_time: 0.0045 (0.004699683893029632)  --> TIME: 2025-02-24 15:50:31 -- STEP: 14620/36938 -- GLOBAL_STEP: 384000 | > loss_text_ce: 0.03876696154475212 (0.039552192945446166) | > loss_mel_ce: 3.8121678829193115 (nan) | > loss: 0.015281488187611103 (nan) | > current_lr: 5e-06 | > step_time: 0.151 (0.13641687036375968) | > loader_time: 0.0053 (0.004700018622480573)  --> TIME: 2025-02-24 15:50:40 -- STEP: 14670/36938 -- GLOBAL_STEP: 384050 | > loss_text_ce: 0.046056732535362244 (0.03955460294062848) | > loss_mel_ce: 3.6241538524627686 (nan) | > loss: 0.014564328826963902 (nan) | > current_lr: 5e-06 | > step_time: 0.1504 (0.13641663179482053) | > loader_time: 0.0055 (0.004700595973378235)  --> TIME: 2025-02-24 15:50:50 -- STEP: 14720/36938 -- GLOBAL_STEP: 384100 | > loss_text_ce: 0.03568755462765694 (0.0395549032141192) | > loss_mel_ce: 0.5546875 (nan) | > loss: 0.002342758234590292 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13642248099912732) | > loader_time: 0.0045 (0.0047005004209020925)  --> TIME: 2025-02-24 15:50:59 -- STEP: 14770/36938 -- GLOBAL_STEP: 384150 | > loss_text_ce: 0.035560835152864456 (0.039557107439997415) | > loss_mel_ce: 2.436971664428711 (nan) | > loss: 0.009811637923121452 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13642267812807626) | > loader_time: 0.0046 (0.004700637202692264)  --> TIME: 2025-02-24 15:51:08 -- STEP: 14820/36938 -- GLOBAL_STEP: 384200 | > loss_text_ce: 0.0363803431391716 (0.039555142155638615) | > loss_mel_ce: 3.373032808303833 (nan) | > loss: 0.013529418036341667 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.13643003443152926) | > loader_time: 0.0048 (0.004700880382064228)  --> TIME: 2025-02-24 15:51:18 -- STEP: 14870/36938 -- GLOBAL_STEP: 384250 | > loss_text_ce: 0.03968290239572525 (0.03955999459262139) | > loss_mel_ce: 2.801288604736328 (nan) | > loss: 0.011273697018623352 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13642398100534198) | > loader_time: 0.0045 (0.00470086699295301)  --> TIME: 2025-02-24 15:51:27 -- STEP: 14920/36938 -- GLOBAL_STEP: 384300 | > loss_text_ce: 0.034025777131319046 (0.03956059056082022) | > loss_mel_ce: 2.9203481674194336 (nan) | > loss: 0.011723706498742104 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13644115752253372) | > loader_time: 0.0048 (0.004700543174155601)  --> TIME: 2025-02-24 15:51:36 -- STEP: 14970/36938 -- GLOBAL_STEP: 384350 | > loss_text_ce: 0.04652564600110054 (0.03955695395710658) | > loss_mel_ce: 3.2905731201171875 (nan) | > loss: 0.013242456130683422 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.136429884852611) | > loader_time: 0.0045 (0.004700283838576614)  --> TIME: 2025-02-24 15:51:46 -- STEP: 15020/36938 -- GLOBAL_STEP: 384400 | > loss_text_ce: 0.04253869876265526 (0.039556388119074484) | > loss_mel_ce: 3.0809450149536133 (nan) | > loss: 0.012394777499139309 (nan) | > current_lr: 5e-06 | > step_time: 0.1923 (0.13644393920580936) | > loader_time: 0.0047 (0.004700257187676975)  --> TIME: 2025-02-24 15:51:55 -- STEP: 15070/36938 -- GLOBAL_STEP: 384450 | > loss_text_ce: 0.03617769479751587 (0.03954929177511512) | > loss_mel_ce: 3.3653175830841064 (nan) | > loss: 0.013497998006641865 (nan) | > current_lr: 5e-06 | > step_time: 0.1777 (0.13643563561037633) | > loader_time: 0.0048 (0.004699897623726611)  --> TIME: 2025-02-24 15:52:04 -- STEP: 15120/36938 -- GLOBAL_STEP: 384500 | > loss_text_ce: 0.04601828381419182 (0.03954624382054646) | > loss_mel_ce: 2.384986639022827 (nan) | > loss: 0.009646845981478691 (nan) | > current_lr: 5e-06 | > step_time: 0.1619 (0.13643803869290294) | > loader_time: 0.005 (0.0046995817038117315)  --> TIME: 2025-02-24 15:52:14 -- STEP: 15170/36938 -- GLOBAL_STEP: 384550 | > loss_text_ce: 0.03539174422621727 (0.03954793188215074) | > loss_mel_ce: 2.890700101852417 (nan) | > loss: 0.01161147654056549 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.13645529072773446) | > loader_time: 0.0047 (0.004699465280783049)  --> TIME: 2025-02-24 15:52:23 -- STEP: 15220/36938 -- GLOBAL_STEP: 384600 | > loss_text_ce: 0.03986339271068573 (0.03954645335703663) | > loss_mel_ce: 2.607571601867676 (nan) | > loss: 0.010505694895982742 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.136464652610984) | > loader_time: 0.0045 (0.004699516233727439)  --> TIME: 2025-02-24 15:52:33 -- STEP: 15270/36938 -- GLOBAL_STEP: 384650 | > loss_text_ce: 0.040577683597803116 (0.039541585508858466) | > loss_mel_ce: 3.1366946697235107 (nan) | > loss: 0.012608224526047707 (nan) | > current_lr: 5e-06 | > step_time: 0.1349 (0.13645694269881) | > loader_time: 0.0046 (0.0046992457451661175)  --> TIME: 2025-02-24 15:52:42 -- STEP: 15320/36938 -- GLOBAL_STEP: 384700 | > loss_text_ce: 0.037886764854192734 (0.039538809324690144) | > loss_mel_ce: 3.897061586380005 (nan) | > loss: 0.015614875592291355 (nan) | > current_lr: 5e-06 | > step_time: 0.1892 (0.13647191642781117) | > loader_time: 0.0048 (0.00469895154626812)  --> TIME: 2025-02-24 15:52:52 -- STEP: 15370/36938 -- GLOBAL_STEP: 384750 | > loss_text_ce: 0.03581951931118965 (0.0395406386881059) | > loss_mel_ce: 0.9259982705116272 (nan) | > loss: 0.003816737560555339 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13647732337574378) | > loader_time: 0.0045 (0.004699157753313377)  --> TIME: 2025-02-24 15:53:01 -- STEP: 15420/36938 -- GLOBAL_STEP: 384800 | > loss_text_ce: 0.041517529636621475 (0.03953971077691668) | > loss_mel_ce: 3.4758644104003906 (nan) | > loss: 0.013957865536212921 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13650166655947385) | > loader_time: 0.0046 (0.004699286613637532)  --> TIME: 2025-02-24 15:53:11 -- STEP: 15470/36938 -- GLOBAL_STEP: 384850 | > loss_text_ce: 0.03884678706526756 (0.0395401748584773) | > loss_mel_ce: 3.1174275875091553 (nan) | > loss: 0.01252489909529686 (nan) | > current_lr: 5e-06 | > step_time: 0.1762 (0.1364928766458973) | > loader_time: 0.0048 (0.004698994672907198)  --> TIME: 2025-02-24 15:53:20 -- STEP: 15520/36938 -- GLOBAL_STEP: 384900 | > loss_text_ce: 0.036345165222883224 (0.03953908271877552) | > loss_mel_ce: 2.035446882247925 (nan) | > loss: 0.0082213981077075 (nan) | > current_lr: 5e-06 | > step_time: 0.1082 (0.1364849256793245) | > loader_time: 0.0045 (0.004699180282882814)  --> TIME: 2025-02-24 15:53:29 -- STEP: 15570/36938 -- GLOBAL_STEP: 384950 | > loss_text_ce: 0.04067236930131912 (0.03953690045784181) | > loss_mel_ce: 3.4872238636016846 (nan) | > loss: 0.013999588787555695 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13648814378675894) | > loader_time: 0.0049 (0.004699734548153451)  --> TIME: 2025-02-24 15:53:38 -- STEP: 15620/36938 -- GLOBAL_STEP: 385000 | > loss_text_ce: 0.04453824460506439 (0.039534099139905236) | > loss_mel_ce: 3.7257239818573 (nan) | > loss: 0.014961359091103077 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13648960338511906) | > loader_time: 0.0046 (0.004699921302697728)  --> TIME: 2025-02-24 15:53:48 -- STEP: 15670/36938 -- GLOBAL_STEP: 385050 | > loss_text_ce: 0.03797633945941925 (0.0395342548984034) | > loss_mel_ce: 2.4404079914093018 (nan) | > loss: 0.00983485858887434 (nan) | > current_lr: 5e-06 | > step_time: 0.1599 (0.1364892020526778) | > loader_time: 0.0046 (0.004699589982486188)  --> TIME: 2025-02-24 15:53:57 -- STEP: 15720/36938 -- GLOBAL_STEP: 385100 | > loss_text_ce: 0.03378797695040703 (0.039530478567939606) | > loss_mel_ce: 2.511918067932129 (nan) | > loss: 0.01010200846940279 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13648278503927533) | > loader_time: 0.0045 (0.004699211023539386)  --> TIME: 2025-02-24 15:54:06 -- STEP: 15770/36938 -- GLOBAL_STEP: 385150 | > loss_text_ce: 0.03791925683617592 (0.039529844409902565) | > loss_mel_ce: 2.537320137023926 (nan) | > loss: 0.010219204239547253 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1364786388001714) | > loader_time: 0.0045 (0.004699294432553604)  --> TIME: 2025-02-24 15:54:15 -- STEP: 15820/36938 -- GLOBAL_STEP: 385200 | > loss_text_ce: 0.03576739877462387 (0.039534282623984535) | > loss_mel_ce: 2.7702229022979736 (nan) | > loss: 0.01113488245755434 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.1364721524429077) | > loader_time: 0.0046 (0.004699035480561967)  --> TIME: 2025-02-24 15:54:25 -- STEP: 15870/36938 -- GLOBAL_STEP: 385250 | > loss_text_ce: 0.034569140523672104 (0.039533071586640434) | > loss_mel_ce: 2.572800874710083 (nan) | > loss: 0.010346706956624985 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.1364745280542833) | > loader_time: 0.0046 (0.004698751343821463)  --> TIME: 2025-02-24 15:54:34 -- STEP: 15920/36938 -- GLOBAL_STEP: 385300 | > loss_text_ce: 0.04473131522536278 (0.03953810616825765) | > loss_mel_ce: 2.3362252712249756 (nan) | > loss: 0.00944824144244194 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13646889778237814) | > loader_time: 0.0045 (0.004698611593725699)  --> TIME: 2025-02-24 15:54:44 -- STEP: 15970/36938 -- GLOBAL_STEP: 385350 | > loss_text_ce: 0.038862548768520355 (0.03953439422627087) | > loss_mel_ce: 3.508776903152466 (nan) | > loss: 0.014077934436500072 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.1364872542484593) | > loader_time: 0.0047 (0.004698441576494904)  --> TIME: 2025-02-24 15:54:53 -- STEP: 16020/36938 -- GLOBAL_STEP: 385400 | > loss_text_ce: 0.035461440682411194 (0.03953751193076964) | > loss_mel_ce: 0.016326904296875 (nan) | > loss: 0.00020550932094920427 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1364743953638157) | > loader_time: 0.0048 (0.0046983755185511875)  --> TIME: 2025-02-24 15:55:02 -- STEP: 16070/36938 -- GLOBAL_STEP: 385450 | > loss_text_ce: 0.04270736128091812 (0.03953205525762666) | > loss_mel_ce: 3.272160291671753 (nan) | > loss: 0.013154237531125546 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13648427158835189) | > loader_time: 0.0046 (0.004698295435997446)  --> TIME: 2025-02-24 15:55:12 -- STEP: 16120/36938 -- GLOBAL_STEP: 385500 | > loss_text_ce: 0.036753833293914795 (0.03953438976337034) | > loss_mel_ce: 3.825859308242798 (nan) | > loss: 0.01532783079892397 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13648775941680708) | > loader_time: 0.0047 (0.004698043573583036)  --> TIME: 2025-02-24 15:55:21 -- STEP: 16170/36938 -- GLOBAL_STEP: 385550 | > loss_text_ce: 0.03500425070524216 (0.039531806697487314) | > loss_mel_ce: 3.1745200157165527 (nan) | > loss: 0.01273620780557394 (nan) | > current_lr: 5e-06 | > step_time: 0.1695 (0.13649616383002255) | > loader_time: 0.0047 (0.004697806347131587)  --> TIME: 2025-02-24 15:55:30 -- STEP: 16220/36938 -- GLOBAL_STEP: 385600 | > loss_text_ce: 0.0345647931098938 (0.039530599217014455) | > loss_mel_ce: 2.87341046333313 (nan) | > loss: 0.011539584957063198 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13649526524337946) | > loader_time: 0.0046 (0.004697727260401457)  --> TIME: 2025-02-24 15:55:39 -- STEP: 16270/36938 -- GLOBAL_STEP: 385650 | > loss_text_ce: 0.031890809535980225 (0.03953128460406594) | > loss_mel_ce: 1.7314168214797974 (nan) | > loss: 0.006997252814471722 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.13648512466010945) | > loader_time: 0.0044 (0.004697936945483231)  --> TIME: 2025-02-24 15:55:49 -- STEP: 16320/36938 -- GLOBAL_STEP: 385700 | > loss_text_ce: 0.026278266683220863 (0.03952773172213181) | > loss_mel_ce: 2.830291748046875 (nan) | > loss: 0.01133559551090002 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13649757408043886) | > loader_time: 0.0045 (0.004697751677503778)  --> TIME: 2025-02-24 15:55:58 -- STEP: 16370/36938 -- GLOBAL_STEP: 385750 | > loss_text_ce: 0.04147087037563324 (0.03952851149719498) | > loss_mel_ce: 3.271425485610962 (nan) | > loss: 0.013146414421498775 (nan) | > current_lr: 5e-06 | > step_time: 0.1767 (0.13651188009747922) | > loader_time: 0.0048 (0.00469775877132894)  --> TIME: 2025-02-24 15:56:08 -- STEP: 16420/36938 -- GLOBAL_STEP: 385800 | > loss_text_ce: 0.036978404968976974 (0.03952709632896201) | > loss_mel_ce: 2.3832411766052246 (nan) | > loss: 0.009604047052562237 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13651498126344733) | > loader_time: 0.0046 (0.004697782156964027)  --> TIME: 2025-02-24 15:56:17 -- STEP: 16470/36938 -- GLOBAL_STEP: 385850 | > loss_text_ce: 0.03463095426559448 (0.03952243817827665) | > loss_mel_ce: 2.5897879600524902 (nan) | > loss: 0.010414361953735352 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.1365186503531355) | > loader_time: 0.0047 (0.0046975579924777465)  --> TIME: 2025-02-24 15:56:26 -- STEP: 16520/36938 -- GLOBAL_STEP: 385900 | > loss_text_ce: 0.031135953962802887 (0.03952114413132712) | > loss_mel_ce: 3.309208631515503 (nan) | > loss: 0.013255336321890354 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1365140665674322) | > loader_time: 0.0045 (0.00469739761825913)  --> TIME: 2025-02-24 15:56:36 -- STEP: 16570/36938 -- GLOBAL_STEP: 385950 | > loss_text_ce: 0.040921811014413834 (0.039515878322242605) | > loss_mel_ce: 3.0129499435424805 (nan) | > loss: 0.012118539772927761 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13651551562903771) | > loader_time: 0.0045 (0.004697333823936712)  --> TIME: 2025-02-24 15:56:45 -- STEP: 16620/36938 -- GLOBAL_STEP: 386000 | > loss_text_ce: 0.04160074517130852 (0.039518553141121286) | > loss_mel_ce: 2.7732529640197754 (nan) | > loss: 0.011170054785907269 (nan) | > current_lr: 5e-06 | > step_time: 0.179 (0.1365187982933326) | > loader_time: 0.0048 (0.00469722776493298)  --> TIME: 2025-02-24 15:56:54 -- STEP: 16670/36938 -- GLOBAL_STEP: 386050 | > loss_text_ce: 0.04790617898106575 (0.03951804651524014) | > loss_mel_ce: 0.1257731169462204 (nan) | > loss: 0.0006892035598866642 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13649936085628303) | > loader_time: 0.0046 (0.004696989974792522)  --> TIME: 2025-02-24 15:57:04 -- STEP: 16720/36938 -- GLOBAL_STEP: 386100 | > loss_text_ce: 0.04829683527350426 (0.03951392189558358) | > loss_mel_ce: 3.672849416732788 (nan) | > loss: 0.014766454696655273 (nan) | > current_lr: 5e-06 | > step_time: 0.1886 (0.13652426188357109) | > loader_time: 0.0049 (0.004697126763289061)  --> TIME: 2025-02-24 15:57:13 -- STEP: 16770/36938 -- GLOBAL_STEP: 386150 | > loss_text_ce: 0.028352219611406326 (0.039519320543495585) | > loss_mel_ce: 3.800680160522461 (nan) | > loss: 0.015194574370980263 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13652906616884222) | > loader_time: 0.0045 (0.004697266816382053)  --> TIME: 2025-02-24 15:57:23 -- STEP: 16820/36938 -- GLOBAL_STEP: 386200 | > loss_text_ce: 0.03516087308526039 (0.03952008367584762) | > loss_mel_ce: 3.183149576187134 (nan) | > loss: 0.01277107372879982 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.1365352089424902) | > loader_time: 0.0045 (0.004697406745552315)  --> TIME: 2025-02-24 15:57:32 -- STEP: 16870/36938 -- GLOBAL_STEP: 386250 | > loss_text_ce: 0.031362731009721756 (0.03951585200396063) | > loss_mel_ce: 2.9322197437286377 (nan) | > loss: 0.011760248802602291 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13653133787484137) | > loader_time: 0.0045 (0.0046973464894591605)  --> TIME: 2025-02-24 15:57:41 -- STEP: 16920/36938 -- GLOBAL_STEP: 386300 | > loss_text_ce: 0.0366046242415905 (0.03951216309885089) | > loss_mel_ce: 3.225520610809326 (nan) | > loss: 0.012944942340254784 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.13653541407404757) | > loader_time: 0.0049 (0.004697133477416171)  --> TIME: 2025-02-24 15:57:50 -- STEP: 16970/36938 -- GLOBAL_STEP: 386350 | > loss_text_ce: 0.04226880520582199 (0.039506842806533604) | > loss_mel_ce: 3.56766939163208 (nan) | > loss: 0.014325152151286602 (nan) | > current_lr: 5e-06 | > step_time: 0.1529 (0.13652197221622206) | > loader_time: 0.0047 (0.004696870313789408)  --> TIME: 2025-02-24 15:58:00 -- STEP: 17020/36938 -- GLOBAL_STEP: 386400 | > loss_text_ce: 0.04618718475103378 (0.03950558920094328) | > loss_mel_ce: 2.974534273147583 (nan) | > loss: 0.011986990459263325 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13652200251992783) | > loader_time: 0.0046 (0.004697203748234579)  --> TIME: 2025-02-24 15:58:09 -- STEP: 17070/36938 -- GLOBAL_STEP: 386450 | > loss_text_ce: 0.038482196629047394 (0.03950436059167304) | > loss_mel_ce: 2.0243802070617676 (nan) | > loss: 0.008185962215065956 (nan) | > current_lr: 5e-06 | > step_time: 0.1814 (0.13652431724639935) | > loader_time: 0.0049 (0.004697110447607067)  --> TIME: 2025-02-24 15:58:18 -- STEP: 17120/36938 -- GLOBAL_STEP: 386500 | > loss_text_ce: 0.04182739183306694 (0.03950285869857263) | > loss_mel_ce: 3.3892388343811035 (nan) | > loss: 0.013615342788398266 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.13651411740857836) | > loader_time: 0.0046 (0.004697291692283674)  --> TIME: 2025-02-24 15:58:27 -- STEP: 17170/36938 -- GLOBAL_STEP: 386550 | > loss_text_ce: 0.01967698149383068 (0.03949911849183804) | > loss_mel_ce: 1.2771683931350708 (nan) | > loss: 0.0051462119445204735 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13650660688403574) | > loader_time: 0.0045 (0.00469748694742567)  --> TIME: 2025-02-24 15:58:36 -- STEP: 17220/36938 -- GLOBAL_STEP: 386600 | > loss_text_ce: 0.033302199095487595 (0.03949399848342049) | > loss_mel_ce: 3.0187036991119385 (nan) | > loss: 0.012111135758459568 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13649839389891824) | > loader_time: 0.0046 (0.004697419514362554)  --> TIME: 2025-02-24 15:58:46 -- STEP: 17270/36938 -- GLOBAL_STEP: 386650 | > loss_text_ce: 0.04132819548249245 (0.0394970907152916) | > loss_mel_ce: 3.8933053016662598 (nan) | > loss: 0.015613625757396221 (nan) | > current_lr: 5e-06 | > step_time: 0.1705 (0.13651079154111206) | > loader_time: 0.0047 (0.004697325786000821)  --> TIME: 2025-02-24 15:58:55 -- STEP: 17320/36938 -- GLOBAL_STEP: 386700 | > loss_text_ce: 0.05017025023698807 (0.03950039883075375) | > loss_mel_ce: 3.5685012340545654 (nan) | > loss: 0.014359807595610619 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13651718632178256) | > loader_time: 0.0048 (0.004697114793588064)  --> TIME: 2025-02-24 15:59:05 -- STEP: 17370/36938 -- GLOBAL_STEP: 386750 | > loss_text_ce: 0.03873880207538605 (0.03949694224152338) | > loss_mel_ce: 3.676502227783203 (nan) | > loss: 0.014743020758032799 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13652584706494758) | > loader_time: 0.0047 (0.004697270549846653)  --> TIME: 2025-02-24 15:59:14 -- STEP: 17420/36938 -- GLOBAL_STEP: 386800 | > loss_text_ce: 0.043147776275873184 (0.03949819160176253) | > loss_mel_ce: 2.7608089447021484 (nan) | > loss: 0.011126813478767872 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13650215835439874) | > loader_time: 0.0045 (0.004697005839079594)  --> TIME: 2025-02-24 15:59:23 -- STEP: 17470/36938 -- GLOBAL_STEP: 386850 | > loss_text_ce: 0.04795075207948685 (0.03949791946382818) | > loss_mel_ce: 3.7840044498443604 (nan) | > loss: 0.01520617213100195 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1364844255059801) | > loader_time: 0.0046 (0.004697015057581808)  --> TIME: 2025-02-24 15:59:32 -- STEP: 17520/36938 -- GLOBAL_STEP: 386900 | > loss_text_ce: 0.034876421093940735 (0.039500932342938465) | > loss_mel_ce: 3.299363851547241 (nan) | > loss: 0.013231112621724606 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.13648007931230244) | > loader_time: 0.0046 (0.004696886343498763)  --> TIME: 2025-02-24 15:59:41 -- STEP: 17570/36938 -- GLOBAL_STEP: 386950 | > loss_text_ce: 0.02748923748731613 (0.039499921521079714) | > loss_mel_ce: 2.654001474380493 (nan) | > loss: 0.01064083632081747 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13648605605859782) | > loader_time: 0.0045 (0.004696927683965425)  --> TIME: 2025-02-24 15:59:50 -- STEP: 17620/36938 -- GLOBAL_STEP: 387000 | > loss_text_ce: 0.04110011085867882 (0.03949253287083903) | > loss_mel_ce: 3.0678184032440186 (nan) | > loss: 0.012336978688836098 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13646922108804968) | > loader_time: 0.0046 (0.0046969624843012986)  --> TIME: 2025-02-24 15:59:59 -- STEP: 17670/36938 -- GLOBAL_STEP: 387050 | > loss_text_ce: 0.030027205124497414 (0.03949335073176235) | > loss_mel_ce: 3.641737699508667 (nan) | > loss: 0.014570496045053005 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13645183710844108) | > loader_time: 0.0052 (0.0046968750449042015)  --> TIME: 2025-02-24 16:00:08 -- STEP: 17720/36938 -- GLOBAL_STEP: 387100 | > loss_text_ce: 0.0358295775949955 (0.03949546936970903) | > loss_mel_ce: 3.6637094020843506 (nan) | > loss: 0.01468071062117815 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.13643135310026597) | > loader_time: 0.0047 (0.004697060692660171)  --> TIME: 2025-02-24 16:00:18 -- STEP: 17770/36938 -- GLOBAL_STEP: 387150 | > loss_text_ce: 0.0365789532661438 (0.039492243728362) | > loss_mel_ce: 3.237009048461914 (nan) | > loss: 0.012990429066121578 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1364611818472451) | > loader_time: 0.0045 (0.004697567529737393)  --> TIME: 2025-02-24 16:00:27 -- STEP: 17820/36938 -- GLOBAL_STEP: 387200 | > loss_text_ce: 0.035887476056814194 (0.03949381511827764) | > loss_mel_ce: 2.9152801036834717 (nan) | > loss: 0.011710982769727707 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13645942399798788) | > loader_time: 0.0047 (0.004697830147213421)  --> TIME: 2025-02-24 16:00:37 -- STEP: 17870/36938 -- GLOBAL_STEP: 387250 | > loss_text_ce: 0.04288520663976669 (0.039492734993177916) | > loss_mel_ce: 1.2868303060531616 (nan) | > loss: 0.005276649259030819 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13645766685978925) | > loader_time: 0.0045 (0.004697696923809353)  --> TIME: 2025-02-24 16:00:46 -- STEP: 17920/36938 -- GLOBAL_STEP: 387300 | > loss_text_ce: 0.03373490273952484 (0.039492596481432024) | > loss_mel_ce: 2.652311325073242 (nan) | > loss: 0.010658914223313332 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1364640012117366) | > loader_time: 0.0046 (0.004697622132620654)  --> TIME: 2025-02-24 16:00:56 -- STEP: 17970/36938 -- GLOBAL_STEP: 387350 | > loss_text_ce: 0.05148572474718094 (0.0394954483686601) | > loss_mel_ce: 3.4727091789245605 (nan) | > loss: 0.013984901830554008 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13646775632018676) | > loader_time: 0.0045 (0.004697734804105696)  --> TIME: 2025-02-24 16:01:05 -- STEP: 18020/36938 -- GLOBAL_STEP: 387400 | > loss_text_ce: 0.043737273663282394 (0.03949414788250226) | > loss_mel_ce: 3.3137905597686768 (nan) | > loss: 0.013323523104190826 (nan) | > current_lr: 5e-06 | > step_time: 0.19 (0.1364800001090426) | > loader_time: 0.0049 (0.004697789891843146)  --> TIME: 2025-02-24 16:01:14 -- STEP: 18070/36938 -- GLOBAL_STEP: 387450 | > loss_text_ce: 0.036588165909051895 (0.039491323787898394) | > loss_mel_ce: 2.5258076190948486 (nan) | > loss: 0.01016823761165142 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1364734696497491) | > loader_time: 0.0046 (0.004697627155702965)  --> TIME: 2025-02-24 16:01:24 -- STEP: 18120/36938 -- GLOBAL_STEP: 387500 | > loss_text_ce: 0.04086754843592644 (0.03949112775211307) | > loss_mel_ce: 3.370671033859253 (nan) | > loss: 0.013537852093577385 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.1364881476029653) | > loader_time: 0.0048 (0.004697562790338059)  --> TIME: 2025-02-24 16:01:33 -- STEP: 18170/36938 -- GLOBAL_STEP: 387550 | > loss_text_ce: 0.04048002511262894 (0.03948685149880042) | > loss_mel_ce: 0.0625 (nan) | > loss: 0.0004086509288754314 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13649594753603367) | > loader_time: 0.0046 (0.004697594474683324)  --> TIME: 2025-02-24 16:01:43 -- STEP: 18220/36938 -- GLOBAL_STEP: 387600 | > loss_text_ce: 0.042134642601013184 (0.039492932541150806) | > loss_mel_ce: 3.4525725841522217 (nan) | > loss: 0.013867885805666447 (nan) | > current_lr: 5e-06 | > step_time: 0.1514 (0.1364996168799246) | > loader_time: 0.0047 (0.00469763918844197)  --> TIME: 2025-02-24 16:01:52 -- STEP: 18270/36938 -- GLOBAL_STEP: 387650 | > loss_text_ce: 0.03601793572306633 (0.03949771779719352) | > loss_mel_ce: 3.1114296913146973 (nan) | > loss: 0.012489872053265572 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13648532080271866) | > loader_time: 0.0047 (0.004697454634688898)  --> TIME: 2025-02-24 16:02:01 -- STEP: 18320/36938 -- GLOBAL_STEP: 387700 | > loss_text_ce: 0.041168391704559326 (0.0394991436690565) | > loss_mel_ce: 1.635824203491211 (nan) | > loss: 0.0066547333262860775 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13648589516310686) | > loader_time: 0.0045 (0.0046974123565390715)  --> TIME: 2025-02-24 16:02:11 -- STEP: 18370/36938 -- GLOBAL_STEP: 387750 | > loss_text_ce: 0.04809831827878952 (0.039502876399264523) | > loss_mel_ce: 3.3357160091400146 (nan) | > loss: 0.013427835889160633 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13649670530027422) | > loader_time: 0.0046 (0.004697385740202289)  --> TIME: 2025-02-24 16:02:20 -- STEP: 18420/36938 -- GLOBAL_STEP: 387800 | > loss_text_ce: 0.0412101224064827 (0.03950145882741437) | > loss_mel_ce: 2.6514384746551514 (nan) | > loss: 0.010685114189982414 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13648863150682525) | > loader_time: 0.0045 (0.004697222184151194)  --> TIME: 2025-02-24 16:02:29 -- STEP: 18470/36938 -- GLOBAL_STEP: 387850 | > loss_text_ce: 0.04532017931342125 (0.039506034129796) | > loss_mel_ce: 0.8744655251502991 (nan) | > loss: 0.0036499432753771544 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13648769372724195) | > loader_time: 0.0046 (0.004697205598249791)  --> TIME: 2025-02-24 16:02:38 -- STEP: 18520/36938 -- GLOBAL_STEP: 387900 | > loss_text_ce: 0.03642471134662628 (0.03949860706764484) | > loss_mel_ce: 3.302008867263794 (nan) | > loss: 0.01324775256216526 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.1364872233141571) | > loader_time: 0.0046 (0.004697145795204221)  --> TIME: 2025-02-24 16:02:47 -- STEP: 18570/36938 -- GLOBAL_STEP: 387950 | > loss_text_ce: 0.03991558775305748 (0.03950238963935135) | > loss_mel_ce: 3.14077091217041 (nan) | > loss: 0.012621772475540638 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13647531116271916) | > loader_time: 0.0046 (0.004696867860127479)  --> TIME: 2025-02-24 16:02:57 -- STEP: 18620/36938 -- GLOBAL_STEP: 388000 | > loss_text_ce: 0.039869606494903564 (0.03950066917468871) | > loss_mel_ce: 2.8946902751922607 (nan) | > loss: 0.011645079590380192 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13647390886997143) | > loader_time: 0.0046 (0.004696674748983096)  --> TIME: 2025-02-24 16:03:06 -- STEP: 18670/36938 -- GLOBAL_STEP: 388050 | > loss_text_ce: 0.03880230337381363 (0.039498440147316294) | > loss_mel_ce: 3.0176682472229004 (nan) | > loss: 0.012128852307796478 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.13648332547043218) | > loader_time: 0.0046 (0.004696608330390011)  --> TIME: 2025-02-24 16:03:16 -- STEP: 18720/36938 -- GLOBAL_STEP: 388100 | > loss_text_ce: 0.05166734755039215 (0.039497998369455764) | > loss_mel_ce: 3.4284780025482178 (nan) | > loss: 0.01381010189652443 (nan) | > current_lr: 5e-06 | > step_time: 0.1651 (0.13649478296184098) | > loader_time: 0.0049 (0.004696587326689674)  --> TIME: 2025-02-24 16:03:25 -- STEP: 18770/36938 -- GLOBAL_STEP: 388150 | > loss_text_ce: 0.039400614798069 (0.039501242597424234) | > loss_mel_ce: 3.048666477203369 (nan) | > loss: 0.012254235334694386 (nan) | > current_lr: 5e-06 | > step_time: 0.1867 (0.13649475131189792) | > loader_time: 0.0048 (0.004696772653623423)  --> TIME: 2025-02-24 16:03:34 -- STEP: 18820/36938 -- GLOBAL_STEP: 388200 | > loss_text_ce: 0.038983654230833054 (0.03949807187399807) | > loss_mel_ce: 1.8214569091796875 (nan) | > loss: 0.007382701151072979 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13648723894680692) | > loader_time: 0.0044 (0.004696563503820776)  --> TIME: 2025-02-24 16:03:44 -- STEP: 18870/36938 -- GLOBAL_STEP: 388250 | > loss_text_ce: 0.03909849375486374 (0.03949680163800946) | > loss_mel_ce: 2.3676235675811768 (nan) | > loss: 0.009550484828650951 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.1364925542582765) | > loader_time: 0.0046 (0.004696363750815593)  --> TIME: 2025-02-24 16:03:53 -- STEP: 18920/36938 -- GLOBAL_STEP: 388300 | > loss_text_ce: 0.034727104008197784 (0.039494635234242376) | > loss_mel_ce: 2.9759681224823 (nan) | > loss: 0.011947204358875751 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.13650211757383657) | > loader_time: 0.0049 (0.0046963705068915)  --> TIME: 2025-02-24 16:04:02 -- STEP: 18970/36938 -- GLOBAL_STEP: 388350 | > loss_text_ce: 0.036911025643348694 (0.03949440447412313) | > loss_mel_ce: 2.660629987716675 (nan) | > loss: 0.010704528540372849 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13649871412175643) | > loader_time: 0.0046 (0.004696284360991203)  --> TIME: 2025-02-24 16:04:12 -- STEP: 19020/36938 -- GLOBAL_STEP: 388400 | > loss_text_ce: 0.04095759615302086 (0.039495781387988864) | > loss_mel_ce: 3.259352684020996 (nan) | > loss: 0.013096470385789871 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13649225618560953) | > loader_time: 0.0047 (0.004696193628882771)  --> TIME: 2025-02-24 16:04:21 -- STEP: 19070/36938 -- GLOBAL_STEP: 388450 | > loss_text_ce: 0.033828649669885635 (0.03949602144439702) | > loss_mel_ce: 3.5974490642547607 (nan) | > loss: 0.014409833587706089 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13650355146515072) | > loader_time: 0.0046 (0.004696220881537621)  --> TIME: 2025-02-24 16:04:31 -- STEP: 19120/36938 -- GLOBAL_STEP: 388500 | > loss_text_ce: 0.03554805740714073 (0.03949565304108123) | > loss_mel_ce: 2.8904013633728027 (nan) | > loss: 0.011610910296440125 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1365038158379836) | > loader_time: 0.0053 (0.004696265623659277)  --> TIME: 2025-02-24 16:04:40 -- STEP: 19170/36938 -- GLOBAL_STEP: 388550 | > loss_text_ce: 0.03278406336903572 (0.039492189829093485) | > loss_mel_ce: 3.3391401767730713 (nan) | > loss: 0.013380652293562889 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13649756201991303) | > loader_time: 0.0049 (0.004696129384486084)  --> TIME: 2025-02-24 16:04:49 -- STEP: 19220/36938 -- GLOBAL_STEP: 388600 | > loss_text_ce: 0.04270769655704498 (0.03949200782806662) | > loss_mel_ce: 1.9206503629684448 (nan) | > loss: 0.007791103795170784 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13649748409948018) | > loader_time: 0.0046 (0.0046959426350945905)  --> TIME: 2025-02-24 16:04:58 -- STEP: 19270/36938 -- GLOBAL_STEP: 388650 | > loss_text_ce: 0.03942050039768219 (0.03948863330478413) | > loss_mel_ce: 1.6472103595733643 (nan) | > loss: 0.006692979950457811 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13650262809876132) | > loader_time: 0.0045 (0.004695786412787918)  --> TIME: 2025-02-24 16:05:08 -- STEP: 19320/36938 -- GLOBAL_STEP: 388700 | > loss_text_ce: 0.04177633300423622 (0.03948813311429794) | > loss_mel_ce: 2.004747152328491 (nan) | > loss: 0.008121125400066376 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1365098220714621) | > loader_time: 0.0045 (0.004695662084820352)  --> TIME: 2025-02-24 16:05:17 -- STEP: 19370/36938 -- GLOBAL_STEP: 388750 | > loss_text_ce: 0.0434391088783741 (0.039484241803233955) | > loss_mel_ce: 3.33137583732605 (nan) | > loss: 0.013392123393714428 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.1365081695286831) | > loader_time: 0.0045 (0.004695508340984256)  --> TIME: 2025-02-24 16:05:26 -- STEP: 19420/36938 -- GLOBAL_STEP: 388800 | > loss_text_ce: 0.0442071370780468 (0.03948441226256921) | > loss_mel_ce: 2.975924253463745 (nan) | > loss: 0.011984649114310741 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.136509036980991) | > loader_time: 0.0045 (0.004695308368066065)  --> TIME: 2025-02-24 16:05:36 -- STEP: 19470/36938 -- GLOBAL_STEP: 388850 | > loss_text_ce: 0.03523045778274536 (0.039486826173712485) | > loss_mel_ce: 2.6831302642822266 (nan) | > loss: 0.010787146165966988 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13650922018500758) | > loader_time: 0.0046 (0.00469505783222122)  --> TIME: 2025-02-24 16:05:45 -- STEP: 19520/36938 -- GLOBAL_STEP: 388900 | > loss_text_ce: 0.03914802521467209 (0.03948937114552391) | > loss_mel_ce: 2.3268795013427734 (nan) | > loss: 0.009388999082148075 (nan) | > current_lr: 5e-06 | > step_time: 0.1869 (0.1365027938465599) | > loader_time: 0.005 (0.004694858877385243)  --> TIME: 2025-02-24 16:05:54 -- STEP: 19570/36938 -- GLOBAL_STEP: 388950 | > loss_text_ce: 0.05133097618818283 (0.03949131630718003) | > loss_mel_ce: 2.7555770874023438 (nan) | > loss: 0.011138524860143661 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13650826096961088) | > loader_time: 0.0046 (0.004694656967568523)  --> TIME: 2025-02-24 16:06:04 -- STEP: 19620/36938 -- GLOBAL_STEP: 389000 | > loss_text_ce: 0.03909904882311821 (0.039491643600126) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13652020601686674) | > loader_time: 0.0047 (0.00469447263762369)  --> TIME: 2025-02-24 16:06:13 -- STEP: 19670/36938 -- GLOBAL_STEP: 389050 | > loss_text_ce: 0.03881432116031647 (0.039491964800280485) | > loss_mel_ce: 2.9421803951263428 (nan) | > loss: 0.011829344555735588 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13652113321534076) | > loader_time: 0.0046 (0.004694259815186965)  --> TIME: 2025-02-24 16:06:22 -- STEP: 19720/36938 -- GLOBAL_STEP: 389100 | > loss_text_ce: 0.03183792904019356 (0.03949234412532511) | > loss_mel_ce: 3.005927801132202 (nan) | > loss: 0.0120546268299222 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.1365082755533719) | > loader_time: 0.0046 (0.004694036767642339)  --> TIME: 2025-02-24 16:06:32 -- STEP: 19770/36938 -- GLOBAL_STEP: 389150 | > loss_text_ce: 0.03874894604086876 (0.0394882079834268) | > loss_mel_ce: 3.579073905944824 (nan) | > loss: 0.014356440864503384 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13652468814232346) | > loader_time: 0.0045 (0.004693935589170472)  --> TIME: 2025-02-24 16:06:41 -- STEP: 19820/36938 -- GLOBAL_STEP: 389200 | > loss_text_ce: 0.04221070185303688 (0.03949344100907074) | > loss_mel_ce: 3.1372482776641846 (nan) | > loss: 0.012616901658475399 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1365248274490397) | > loader_time: 0.0046 (0.004693762276896789)  --> TIME: 2025-02-24 16:06:50 -- STEP: 19870/36938 -- GLOBAL_STEP: 389250 | > loss_text_ce: 0.028971970081329346 (0.03949141925497039) | > loss_mel_ce: 0.035614013671875 (nan) | > loss: 0.00025629359879530966 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13653195721453923) | > loader_time: 0.0046 (0.004693610966955523)  --> TIME: 2025-02-24 16:07:00 -- STEP: 19920/36938 -- GLOBAL_STEP: 389300 | > loss_text_ce: 0.050667211413383484 (0.03949301922548909) | > loss_mel_ce: 2.1909220218658447 (nan) | > loss: 0.00889519602060318 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13653489888671838) | > loader_time: 0.0046 (0.0046934491778951965)  --> TIME: 2025-02-24 16:07:09 -- STEP: 19970/36938 -- GLOBAL_STEP: 389350 | > loss_text_ce: 0.046301182359457016 (0.03949035693628777) | > loss_mel_ce: 2.8335044384002686 (nan) | > loss: 0.011427800171077251 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13653663060757001) | > loader_time: 0.0045 (0.0046932677238895985)  --> TIME: 2025-02-24 16:07:18 -- STEP: 20020/36938 -- GLOBAL_STEP: 389400 | > loss_text_ce: 0.03890246897935867 (0.03948958143875639) | > loss_mel_ce: 3.728541851043701 (nan) | > loss: 0.014950176700949669 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13652042186224436) | > loader_time: 0.0045 (0.004693446388015971)  --> TIME: 2025-02-24 16:07:27 -- STEP: 20070/36938 -- GLOBAL_STEP: 389450 | > loss_text_ce: 0.0394798144698143 (0.03949263768166656) | > loss_mel_ce: 2.534066915512085 (nan) | > loss: 0.010212487541139126 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13651653680863104) | > loader_time: 0.0047 (0.004693388701316779)  --> TIME: 2025-02-24 16:07:37 -- STEP: 20120/36938 -- GLOBAL_STEP: 389500 | > loss_text_ce: 0.04256249964237213 (0.039491839945699464) | > loss_mel_ce: 3.364325523376465 (nan) | > loss: 0.01351939793676138 (nan) | > current_lr: 5e-06 | > step_time: 0.1946 (0.13652602898671604) | > loader_time: 0.0049 (0.004693322935350843)  --> TIME: 2025-02-24 16:07:46 -- STEP: 20170/36938 -- GLOBAL_STEP: 389550 | > loss_text_ce: 0.026330526918172836 (0.03948857047104246) | > loss_mel_ce: 0.47353601455688477 (nan) | > loss: 0.001983597408980131 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13651851881951052) | > loader_time: 0.0045 (0.004693083273755714)  --> TIME: 2025-02-24 16:07:55 -- STEP: 20220/36938 -- GLOBAL_STEP: 389600 | > loss_text_ce: 0.03550642728805542 (0.03948808483006541) | > loss_mel_ce: 2.956709623336792 (nan) | > loss: 0.011873873881995678 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.136515413795097) | > loader_time: 0.0046 (0.004692867743155365)  --> TIME: 2025-02-24 16:08:04 -- STEP: 20270/36938 -- GLOBAL_STEP: 389650 | > loss_text_ce: 0.030393900349736214 (0.0394840523819739) | > loss_mel_ce: 3.370800733566284 (nan) | > loss: 0.013496804982423782 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13650755590562053) | > loader_time: 0.0046 (0.004692674083079264)  --> TIME: 2025-02-24 16:08:14 -- STEP: 20320/36938 -- GLOBAL_STEP: 389700 | > loss_text_ce: 0.04049941152334213 (0.03948551218843948) | > loss_mel_ce: 3.069232940673828 (nan) | > loss: 0.012340208515524864 (nan) | > current_lr: 5e-06 | > step_time: 0.1822 (0.13651593553972946) | > loader_time: 0.0056 (0.0046927011505825305)  --> TIME: 2025-02-24 16:08:23 -- STEP: 20370/36938 -- GLOBAL_STEP: 389750 | > loss_text_ce: 0.03950217738747597 (0.03948426943809095) | > loss_mel_ce: 3.4297549724578857 (nan) | > loss: 0.013766894116997719 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13650562218959586) | > loader_time: 0.0046 (0.004692558909843644)  --> TIME: 2025-02-24 16:08:32 -- STEP: 20420/36938 -- GLOBAL_STEP: 389800 | > loss_text_ce: 0.038729969412088394 (0.03948693693465885) | > loss_mel_ce: 3.3610141277313232 (nan) | > loss: 0.013491048477590084 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.1365091205110275) | > loader_time: 0.0047 (0.004692360037804586)  --> TIME: 2025-02-24 16:08:41 -- STEP: 20470/36938 -- GLOBAL_STEP: 389850 | > loss_text_ce: 0.045587532222270966 (0.03948627753023078) | > loss_mel_ce: 2.6086854934692383 (nan) | > loss: 0.010532829910516739 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1365017249517805) | > loader_time: 0.0045 (0.004692149733006227)  --> TIME: 2025-02-24 16:08:51 -- STEP: 20520/36938 -- GLOBAL_STEP: 389900 | > loss_text_ce: 0.04410707205533981 (0.03948596662494062) | > loss_mel_ce: 3.904599189758301 (nan) | > loss: 0.01566947065293789 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13650558495614457) | > loader_time: 0.0047 (0.004692173329477892)  --> TIME: 2025-02-24 16:09:00 -- STEP: 20570/36938 -- GLOBAL_STEP: 389950 | > loss_text_ce: 0.03349792957305908 (0.039484532019390374) | > loss_mel_ce: 1.6502007246017456 (nan) | > loss: 0.006681344471871853 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13650425668474375) | > loader_time: 0.0046 (0.0046920023905634095)  --> TIME: 2025-02-24 16:09:09 -- STEP: 20620/36938 -- GLOBAL_STEP: 390000 | > loss_text_ce: 0.05133143812417984 (0.03948419006432052) | > loss_mel_ce: 2.938387632369995 (nan) | > loss: 0.011863965541124344 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1365031168717528) | > loader_time: 0.0046 (0.004691818821904275) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_390000.pth  --> TIME: 2025-02-24 16:09:25 -- STEP: 20670/36938 -- GLOBAL_STEP: 390050 | > loss_text_ce: 0.04399116337299347 (0.03948667004373742) | > loss_mel_ce: 3.178133726119995 (nan) | > loss: 0.012786210514605045 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13650023160951916) | > loader_time: 0.0046 (0.004692021117558737)  --> TIME: 2025-02-24 16:09:34 -- STEP: 20720/36938 -- GLOBAL_STEP: 390100 | > loss_text_ce: 0.041960231959819794 (0.03948410687807205) | > loss_mel_ce: 2.6928088665008545 (nan) | > loss: 0.010852258652448654 (nan) | > current_lr: 5e-06 | > step_time: 0.1758 (0.1365043678338917) | > loader_time: 0.0049 (0.004692352140271991)  --> TIME: 2025-02-24 16:09:43 -- STEP: 20770/36938 -- GLOBAL_STEP: 390150 | > loss_text_ce: 0.05921948328614235 (0.03948530862098888) | > loss_mel_ce: 2.0981976985931396 (nan) | > loss: 0.008561179973185062 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13650046680544037) | > loader_time: 0.0046 (0.0046924961376603475)  --> TIME: 2025-02-24 16:09:52 -- STEP: 20820/36938 -- GLOBAL_STEP: 390200 | > loss_text_ce: 0.03970477730035782 (0.03948158431484183) | > loss_mel_ce: 3.1785025596618652 (nan) | > loss: 0.012770664878189564 (nan) | > current_lr: 5e-06 | > step_time: 0.1865 (0.13649892260774) | > loader_time: 0.0049 (0.004692481917675366)  --> TIME: 2025-02-24 16:10:02 -- STEP: 20870/36938 -- GLOBAL_STEP: 390250 | > loss_text_ce: 0.03906237334012985 (0.039481149426119966) | > loss_mel_ce: 3.2628209590911865 (nan) | > loss: 0.013102712109684944 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13650382347636944) | > loader_time: 0.0046 (0.004692464715622241)  --> TIME: 2025-02-24 16:10:11 -- STEP: 20920/36938 -- GLOBAL_STEP: 390300 | > loss_text_ce: 0.046479176729917526 (0.039479489428931265) | > loss_mel_ce: 3.693735361099243 (nan) | > loss: 0.014842121861875057 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.13650545154201296) | > loader_time: 0.005 (0.004692826740600419)  --> TIME: 2025-02-24 16:10:21 -- STEP: 20970/36938 -- GLOBAL_STEP: 390350 | > loss_text_ce: 0.04423494264483452 (0.03948159174785767) | > loss_mel_ce: 2.377244234085083 (nan) | > loss: 0.009609045460820198 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13651243437456337) | > loader_time: 0.0047 (0.004692876310534977)  --> TIME: 2025-02-24 16:10:30 -- STEP: 21020/36938 -- GLOBAL_STEP: 390400 | > loss_text_ce: 0.030914507806301117 (0.03948221243461566) | > loss_mel_ce: 3.2784290313720703 (nan) | > loss: 0.013132316991686821 (nan) | > current_lr: 5e-06 | > step_time: 0.1506 (0.13652620684181746) | > loader_time: 0.0046 (0.004693177277195701)  --> TIME: 2025-02-24 16:10:40 -- STEP: 21070/36938 -- GLOBAL_STEP: 390450 | > loss_text_ce: 0.023301051929593086 (0.039476901669175266) | > loss_mel_ce: 3.5827507972717285 (nan) | > loss: 0.014309730380773544 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.13652597027024022) | > loader_time: 0.0045 (0.00469316585289115)  --> TIME: 2025-02-24 16:10:49 -- STEP: 21120/36938 -- GLOBAL_STEP: 390500 | > loss_text_ce: 0.03348260000348091 (0.039475669439178074) | > loss_mel_ce: 2.1913249492645264 (nan) | > loss: 0.0088286017999053 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13652721535075701) | > loader_time: 0.0047 (0.00469316329919932)  --> TIME: 2025-02-24 16:10:58 -- STEP: 21170/36938 -- GLOBAL_STEP: 390550 | > loss_text_ce: 0.049710050225257874 (0.03947732451344993) | > loss_mel_ce: 0.02325439453125 (nan) | > loss: 0.00028954146546311677 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13652687911779984) | > loader_time: 0.0047 (0.004693046650010361)  --> TIME: 2025-02-24 16:11:08 -- STEP: 21220/36938 -- GLOBAL_STEP: 390600 | > loss_text_ce: 0.03632139787077904 (0.039476107915557716) | > loss_mel_ce: 3.2289915084838867 (nan) | > loss: 0.012957591563463211 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.1365334496534053) | > loader_time: 0.0046 (0.004692832801164502)  --> TIME: 2025-02-24 16:11:17 -- STEP: 21270/36938 -- GLOBAL_STEP: 390650 | > loss_text_ce: 0.04640928655862808 (0.0394769358312946) | > loss_mel_ce: 3.6287970542907715 (nan) | > loss: 0.014584152959287167 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13653386635683842) | > loader_time: 0.0048 (0.004692584391093121)  --> TIME: 2025-02-24 16:11:26 -- STEP: 21320/36938 -- GLOBAL_STEP: 390700 | > loss_text_ce: 0.036626022309064865 (0.03947814508943183) | > loss_mel_ce: 3.244774103164673 (nan) | > loss: 0.013021430000662804 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13652666364333327) | > loader_time: 0.0045 (0.00469232596331199)  --> TIME: 2025-02-24 16:11:36 -- STEP: 21370/36938 -- GLOBAL_STEP: 390750 | > loss_text_ce: 0.05091148614883423 (0.03947882651755365) | > loss_mel_ce: 2.361240863800049 (nan) | > loss: 0.009572033770382404 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.1365321806674247) | > loader_time: 0.0045 (0.004692281012117849)  --> TIME: 2025-02-24 16:11:45 -- STEP: 21420/36938 -- GLOBAL_STEP: 390800 | > loss_text_ce: 0.04196751490235329 (0.0394797778765327) | > loss_mel_ce: 3.6117098331451416 (nan) | > loss: 0.01449872087687254 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13654137419051401) | > loader_time: 0.0048 (0.004692378765394724)  --> TIME: 2025-02-24 16:11:54 -- STEP: 21470/36938 -- GLOBAL_STEP: 390850 | > loss_text_ce: 0.03944443538784981 (0.039478389864625006) | > loss_mel_ce: 2.79229998588562 (nan) | > loss: 0.011237082071602345 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13652433672781852) | > loader_time: 0.0046 (0.0046925824133806915)  --> TIME: 2025-02-24 16:12:03 -- STEP: 21520/36938 -- GLOBAL_STEP: 390900 | > loss_text_ce: 0.04025885835289955 (0.039478651495073736) | > loss_mel_ce: 3.195399761199951 (nan) | > loss: 0.012839916162192822 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13652454992430726) | > loader_time: 0.0046 (0.00469264449018529)  --> TIME: 2025-02-24 16:12:13 -- STEP: 21570/36938 -- GLOBAL_STEP: 390950 | > loss_text_ce: 0.04238956794142723 (0.039478247480636844) | > loss_mel_ce: 3.363713502883911 (nan) | > loss: 0.01351628266274929 (nan) | > current_lr: 5e-06 | > step_time: 0.1662 (0.13652176181641043) | > loader_time: 0.0047 (0.004692583997104821)  --> TIME: 2025-02-24 16:12:23 -- STEP: 21620/36938 -- GLOBAL_STEP: 391000 | > loss_text_ce: 0.04538711532950401 (0.039477415117637395) | > loss_mel_ce: 0.18333740532398224 (nan) | > loss: 0.0009076370624825358 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13654444202243038) | > loader_time: 0.0046 (0.004693131949700896)  --> TIME: 2025-02-24 16:12:32 -- STEP: 21670/36938 -- GLOBAL_STEP: 391050 | > loss_text_ce: 0.04275498166680336 (0.03947961870189794) | > loss_mel_ce: 3.147181987762451 (nan) | > loss: 0.012658480554819107 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13654692826537232) | > loader_time: 0.0046 (0.004693076486239876)  --> TIME: 2025-02-24 16:12:41 -- STEP: 21720/36938 -- GLOBAL_STEP: 391100 | > loss_text_ce: 0.03776140883564949 (0.039480012892676594) | > loss_mel_ce: 1.4878692626953125 (nan) | > loss: 0.0060540903359651566 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13654467942086737) | > loader_time: 0.0046 (0.00469290101506216)  --> TIME: 2025-02-24 16:12:51 -- STEP: 21770/36938 -- GLOBAL_STEP: 391150 | > loss_text_ce: 0.03551868721842766 (0.03947809379052415) | > loss_mel_ce: 3.5403401851654053 (nan) | > loss: 0.014189916662871838 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13654269407690175) | > loader_time: 0.0046 (0.004692875621622201)  --> TIME: 2025-02-24 16:13:00 -- STEP: 21820/36938 -- GLOBAL_STEP: 391200 | > loss_text_ce: 0.0329006090760231 (0.0394804036666873) | > loss_mel_ce: 3.679054021835327 (nan) | > loss: 0.014729979448020458 (nan) | > current_lr: 5e-06 | > step_time: 0.1591 (0.13654228924175138) | > loader_time: 0.0054 (0.004692935550245661)  --> TIME: 2025-02-24 16:13:09 -- STEP: 21870/36938 -- GLOBAL_STEP: 391250 | > loss_text_ce: 0.0464954599738121 (0.03947851144788299) | > loss_mel_ce: 3.0420520305633545 (nan) | > loss: 0.012256141752004623 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.1365439649908077) | > loader_time: 0.0048 (0.0046932128231221455)  --> TIME: 2025-02-24 16:13:19 -- STEP: 21920/36938 -- GLOBAL_STEP: 391300 | > loss_text_ce: 0.04497085511684418 (0.03948224541381905) | > loss_mel_ce: 3.0012431144714355 (nan) | > loss: 0.012088150717318058 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13654301426488907) | > loader_time: 0.0045 (0.0046935327296709645)  --> TIME: 2025-02-24 16:13:28 -- STEP: 21970/36938 -- GLOBAL_STEP: 391350 | > loss_text_ce: 0.0437638945877552 (0.03948252885040711) | > loss_mel_ce: 3.3325483798980713 (nan) | > loss: 0.013398065231740475 (nan) | > current_lr: 5e-06 | > step_time: 0.1848 (0.1365522715955299) | > loader_time: 0.0048 (0.004693778927455357)  --> TIME: 2025-02-24 16:13:37 -- STEP: 22020/36938 -- GLOBAL_STEP: 391400 | > loss_text_ce: 0.03631838783621788 (0.03948201724204576) | > loss_mel_ce: 3.630931854248047 (nan) | > loss: 0.014552581124007702 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1365513572684201) | > loader_time: 0.0046 (0.004693913308194735)  --> TIME: 2025-02-24 16:13:46 -- STEP: 22070/36938 -- GLOBAL_STEP: 391450 | > loss_text_ce: 0.046262145042419434 (0.03948457335241325) | > loss_mel_ce: 3.4366819858551025 (nan) | > loss: 0.013821206986904144 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.13653494637640043) | > loader_time: 0.0048 (0.004694061026510535)  --> TIME: 2025-02-24 16:13:55 -- STEP: 22120/36938 -- GLOBAL_STEP: 391500 | > loss_text_ce: 0.0359722301363945 (0.0394906798688773) | > loss_mel_ce: 3.1105644702911377 (nan) | > loss: 0.012486256659030914 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13652953704916665) | > loader_time: 0.0047 (0.004693899286159781)  --> TIME: 2025-02-24 16:14:05 -- STEP: 22170/36938 -- GLOBAL_STEP: 391550 | > loss_text_ce: 0.03846224024891853 (0.03949256919918367) | > loss_mel_ce: 3.2062478065490723 (nan) | > loss: 0.012875833548605442 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.13653875684544484) | > loader_time: 0.0046 (0.004693790948977068)  --> TIME: 2025-02-24 16:14:14 -- STEP: 22220/36938 -- GLOBAL_STEP: 391600 | > loss_text_ce: 0.046421222388744354 (0.03949488771976216) | > loss_mel_ce: 3.1366679668426514 (nan) | > loss: 0.012631307356059551 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.13652645477188372) | > loader_time: 0.0046 (0.0046941134867423615)  --> TIME: 2025-02-24 16:14:23 -- STEP: 22270/36938 -- GLOBAL_STEP: 391650 | > loss_text_ce: 0.04637253284454346 (0.039493405937163) | > loss_mel_ce: 2.9839847087860107 (nan) | > loss: 0.012025228701531887 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13652297823108886) | > loader_time: 0.0045 (0.004693928394394718)  --> TIME: 2025-02-24 16:14:33 -- STEP: 22320/36938 -- GLOBAL_STEP: 391700 | > loss_text_ce: 0.031376998871564865 (0.039493003372715274) | > loss_mel_ce: 1.4837342500686646 (nan) | > loss: 0.006012346595525742 (nan) | > current_lr: 5e-06 | > step_time: 0.1078 (0.1365317769802588) | > loader_time: 0.0046 (0.004694055282514165)  --> TIME: 2025-02-24 16:14:42 -- STEP: 22370/36938 -- GLOBAL_STEP: 391750 | > loss_text_ce: 0.037057679146528244 (0.03949417957759602) | > loss_mel_ce: 3.4779131412506104 (nan) | > loss: 0.013948297128081322 (nan) | > current_lr: 5e-06 | > step_time: 0.178 (0.1365252383649471) | > loader_time: 0.0047 (0.0046940315606394875)  --> TIME: 2025-02-24 16:14:51 -- STEP: 22420/36938 -- GLOBAL_STEP: 391800 | > loss_text_ce: 0.03187820687890053 (0.03949248873982572) | > loss_mel_ce: 1.30078125 (nan) | > loss: 0.0052883317694067955 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13651411535482433) | > loader_time: 0.0045 (0.004694013761474453)  --> TIME: 2025-02-24 16:15:01 -- STEP: 22470/36938 -- GLOBAL_STEP: 391850 | > loss_text_ce: 0.04301003739237785 (0.039492218910861764) | > loss_mel_ce: 3.72173810005188 (nan) | > loss: 0.014939477667212486 (nan) | > current_lr: 5e-06 | > step_time: 0.1814 (0.13652093881599311) | > loader_time: 0.0048 (0.004694008137525117)  --> TIME: 2025-02-24 16:15:10 -- STEP: 22520/36938 -- GLOBAL_STEP: 391900 | > loss_text_ce: 0.03281747177243233 (0.03949355949940502) | > loss_mel_ce: 0.01263427734375 (nan) | > loss: 0.00018036409164778888 (nan) | > current_lr: 5e-06 | > step_time: 0.1442 (0.1365182270059355) | > loader_time: 0.0047 (0.004693962456278024)  --> TIME: 2025-02-24 16:15:19 -- STEP: 22570/36938 -- GLOBAL_STEP: 391950 | > loss_text_ce: 0.03726796433329582 (0.03949200711520451) | > loss_mel_ce: 3.5926272869110107 (nan) | > loss: 0.01440434716641903 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13651502284530306) | > loader_time: 0.0045 (0.004693904068811214)  --> TIME: 2025-02-24 16:15:28 -- STEP: 22620/36938 -- GLOBAL_STEP: 392000 | > loss_text_ce: 0.032911915332078934 (0.039487510448968755) | > loss_mel_ce: 3.6245789527893066 (nan) | > loss: 0.014513853937387466 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.13651273993661495) | > loader_time: 0.0046 (0.004694013064356648)  --> TIME: 2025-02-24 16:15:38 -- STEP: 22670/36938 -- GLOBAL_STEP: 392050 | > loss_text_ce: 0.0497361496090889 (0.039484814487736004) | > loss_mel_ce: 2.673953056335449 (nan) | > loss: 0.010808291845023632 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.1365152427999773) | > loader_time: 0.0054 (0.004694121905134608)  --> TIME: 2025-02-24 16:15:47 -- STEP: 22720/36938 -- GLOBAL_STEP: 392100 | > loss_text_ce: 0.040724318474531174 (0.0394860836320627) | > loss_mel_ce: 3.4177792072296143 (nan) | > loss: 0.013724220916628838 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.13650715339561456) | > loader_time: 0.0047 (0.004694162634476821)  --> TIME: 2025-02-24 16:15:56 -- STEP: 22770/36938 -- GLOBAL_STEP: 392150 | > loss_text_ce: 0.036094289273023605 (0.039485291527157795) | > loss_mel_ce: 2.6513094902038574 (nan) | > loss: 0.010664300993084908 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13649529780428382) | > loader_time: 0.0047 (0.004694077284858589)  --> TIME: 2025-02-24 16:16:05 -- STEP: 22820/36938 -- GLOBAL_STEP: 392200 | > loss_text_ce: 0.0264898668974638 (0.03948208779426341) | > loss_mel_ce: 2.146484375 (nan) | > loss: 0.008622914552688599 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13649602784700784) | > loader_time: 0.0044 (0.004694121579764503)  --> TIME: 2025-02-24 16:16:14 -- STEP: 22870/36938 -- GLOBAL_STEP: 392250 | > loss_text_ce: 0.04078944772481918 (0.0394807140972051) | > loss_mel_ce: 6.809374809265137 (nan) | > loss: 0.02718319371342659 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13647763046261177) | > loader_time: 0.0046 (0.00469393746843493)  --> TIME: 2025-02-24 16:16:23 -- STEP: 22920/36938 -- GLOBAL_STEP: 392300 | > loss_text_ce: 0.030645376071333885 (0.03948081471416003) | > loss_mel_ce: 2.334298610687256 (nan) | > loss: 0.009384699165821075 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13647542487680256) | > loader_time: 0.0053 (0.004694198636276358)  --> TIME: 2025-02-24 16:16:32 -- STEP: 22970/36938 -- GLOBAL_STEP: 392350 | > loss_text_ce: 0.037993140518665314 (0.039479088638923104) | > loss_mel_ce: 3.010552167892456 (nan) | > loss: 0.012097402475774288 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.1364714733039899) | > loader_time: 0.0046 (0.004694643828159946)  --> TIME: 2025-02-24 16:16:42 -- STEP: 23020/36938 -- GLOBAL_STEP: 392400 | > loss_text_ce: 0.03920746222138405 (0.039479550479754856) | > loss_mel_ce: 1.7680704593658447 (nan) | > loss: 0.007171737961471081 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.1364710307452689) | > loader_time: 0.0046 (0.004694717019459771)  --> TIME: 2025-02-24 16:16:51 -- STEP: 23070/36938 -- GLOBAL_STEP: 392450 | > loss_text_ce: 0.038856420665979385 (0.03947950050325195) | > loss_mel_ce: 2.6421446800231934 (nan) | > loss: 0.010638894513249397 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13646113997576287) | > loader_time: 0.0045 (0.004694532748678314)  --> TIME: 2025-02-24 16:17:00 -- STEP: 23120/36938 -- GLOBAL_STEP: 392500 | > loss_text_ce: 0.04583185911178589 (0.03947965102229936) | > loss_mel_ce: 3.3524587154388428 (nan) | > loss: 0.01348528079688549 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13646155420471529) | > loader_time: 0.0045 (0.0046946583837786365)  --> TIME: 2025-02-24 16:17:09 -- STEP: 23170/36938 -- GLOBAL_STEP: 392550 | > loss_text_ce: 0.038591209799051285 (0.03948086407734956) | > loss_mel_ce: 2.8347108364105225 (nan) | > loss: 0.011401992291212082 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13645949015907427) | > loader_time: 0.0045 (0.004694454794461019)  --> TIME: 2025-02-24 16:17:19 -- STEP: 23220/36938 -- GLOBAL_STEP: 392600 | > loss_text_ce: 0.03737829625606537 (0.03947694413532359) | > loss_mel_ce: 3.3525733947753906 (nan) | > loss: 0.013452189974486828 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1364675826804571) | > loader_time: 0.0047 (0.004694375316199311)  --> TIME: 2025-02-24 16:17:28 -- STEP: 23270/36938 -- GLOBAL_STEP: 392650 | > loss_text_ce: 0.038097597658634186 (0.03947680384364086) | > loss_mel_ce: 3.5832486152648926 (nan) | > loss: 0.014370422810316086 (nan) | > current_lr: 5e-06 | > step_time: 0.1769 (0.13647101427487135) | > loader_time: 0.0049 (0.004694421136640514)  --> TIME: 2025-02-24 16:17:37 -- STEP: 23320/36938 -- GLOBAL_STEP: 392700 | > loss_text_ce: 0.035046424716711044 (0.039474447647039015) | > loss_mel_ce: 2.9399755001068115 (nan) | > loss: 0.011805643327534199 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.1364746479493255) | > loader_time: 0.0048 (0.0046943119009813)  --> TIME: 2025-02-24 16:17:47 -- STEP: 23370/36938 -- GLOBAL_STEP: 392750 | > loss_text_ce: 0.03782593086361885 (0.03947343672540003) | > loss_mel_ce: 3.0065340995788574 (nan) | > loss: 0.01208079420030117 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.1364726616268121) | > loader_time: 0.0046 (0.004694451620602234)  --> TIME: 2025-02-24 16:17:56 -- STEP: 23420/36938 -- GLOBAL_STEP: 392800 | > loss_text_ce: 0.04275640472769737 (0.03947721376130677) | > loss_mel_ce: 2.9705729484558105 (nan) | > loss: 0.011957656592130661 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.136477068964992) | > loader_time: 0.0047 (0.004694361792579867)  --> TIME: 2025-02-24 16:18:05 -- STEP: 23470/36938 -- GLOBAL_STEP: 392850 | > loss_text_ce: 0.04029062017798424 (0.03947533517075932) | > loss_mel_ce: 3.5639021396636963 (nan) | > loss: 0.014302353374660015 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.13647699924951606) | > loader_time: 0.0047 (0.004694412442334519)  --> TIME: 2025-02-24 16:18:14 -- STEP: 23520/36938 -- GLOBAL_STEP: 392900 | > loss_text_ce: 0.03488648682832718 (0.03947382918638169) | > loss_mel_ce: 3.169511079788208 (nan) | > loss: 0.012715864926576614 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.13647289825539904) | > loader_time: 0.0049 (0.004694448827075308)  --> TIME: 2025-02-24 16:18:24 -- STEP: 23570/36938 -- GLOBAL_STEP: 392950 | > loss_text_ce: 0.029632439836859703 (0.03947415261572413) | > loss_mel_ce: 2.4537150859832764 (nan) | > loss: 0.009854554198682308 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.1364706713515) | > loader_time: 0.0046 (0.004694345789437177)  --> TIME: 2025-02-24 16:18:33 -- STEP: 23620/36938 -- GLOBAL_STEP: 393000 | > loss_text_ce: 0.04354414716362953 (0.03947522730742375) | > loss_mel_ce: 3.1225647926330566 (nan) | > loss: 0.012563924305140972 (nan) | > current_lr: 5e-06 | > step_time: 0.1874 (0.13648181081726513) | > loader_time: 0.0049 (0.004694321436159295)  --> TIME: 2025-02-24 16:18:43 -- STEP: 23670/36938 -- GLOBAL_STEP: 393050 | > loss_text_ce: 0.03502679616212845 (0.039474104294717455) | > loss_mel_ce: 3.061490774154663 (nan) | > loss: 0.01228776853531599 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13648394515960477) | > loader_time: 0.0048 (0.0046945904698309435)  --> TIME: 2025-02-24 16:18:52 -- STEP: 23720/36938 -- GLOBAL_STEP: 393100 | > loss_text_ce: 0.03617532551288605 (0.039472204159542765) | > loss_mel_ce: 3.639624834060669 (nan) | > loss: 0.014586509205400944 (nan) | > current_lr: 5e-06 | > step_time: 0.1579 (0.136478042924182) | > loader_time: 0.0053 (0.004694617437874088)  --> TIME: 2025-02-24 16:19:01 -- STEP: 23770/36938 -- GLOBAL_STEP: 393150 | > loss_text_ce: 0.03057655692100525 (0.03946707461575427) | > loss_mel_ce: 3.3526580333709717 (nan) | > loss: 0.013425534591078758 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13648757809192325) | > loader_time: 0.0046 (0.004694826832628071)  --> TIME: 2025-02-24 16:19:11 -- STEP: 23820/36938 -- GLOBAL_STEP: 393200 | > loss_text_ce: 0.04660793021321297 (0.03946418562754604) | > loss_mel_ce: 3.123302459716797 (nan) | > loss: 0.012579010799527168 (nan) | > current_lr: 5e-06 | > step_time: 0.1488 (0.13648160918233282) | > loader_time: 0.0046 (0.0046948261465173564)  --> TIME: 2025-02-24 16:19:20 -- STEP: 23870/36938 -- GLOBAL_STEP: 393250 | > loss_text_ce: 0.037632543593645096 (0.03946362300926119) | > loss_mel_ce: 3.0181832313537598 (nan) | > loss: 0.012126253917813301 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13647749351066688) | > loader_time: 0.0046 (0.004694954451028406)  --> TIME: 2025-02-24 16:19:29 -- STEP: 23920/36938 -- GLOBAL_STEP: 393300 | > loss_text_ce: 0.04458975791931152 (0.03946314262219177) | > loss_mel_ce: 3.566760778427124 (nan) | > loss: 0.014330756850540638 (nan) | > current_lr: 5e-06 | > step_time: 0.1687 (0.13647903967461766) | > loader_time: 0.0049 (0.004694995383753827)  --> TIME: 2025-02-24 16:19:38 -- STEP: 23970/36938 -- GLOBAL_STEP: 393350 | > loss_text_ce: 0.038008563220500946 (0.03946131634870549) | > loss_mel_ce: 2.9940428733825684 (nan) | > loss: 0.012031950056552887 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.1364789303909694) | > loader_time: 0.0048 (0.0046952341614437945)  --> TIME: 2025-02-24 16:19:48 -- STEP: 24020/36938 -- GLOBAL_STEP: 393400 | > loss_text_ce: 0.037164587527513504 (0.03946013084657723) | > loss_mel_ce: 2.4782609939575195 (nan) | > loss: 0.009981848299503326 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13648379437631528) | > loader_time: 0.0046 (0.004695279830500645)  --> TIME: 2025-02-24 16:19:57 -- STEP: 24070/36938 -- GLOBAL_STEP: 393450 | > loss_text_ce: 0.034324951469898224 (0.039461204476420424) | > loss_mel_ce: 2.7667899131774902 (nan) | > loss: 0.011115535162389278 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13648026826523238) | > loader_time: 0.0047 (0.004695616651977991)  --> TIME: 2025-02-24 16:20:07 -- STEP: 24120/36938 -- GLOBAL_STEP: 393500 | > loss_text_ce: 0.04332742467522621 (0.03946091591169672) | > loss_mel_ce: 2.5345966815948486 (nan) | > loss: 0.010229857638478279 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13649430347121472) | > loader_time: 0.0047 (0.004696147447794817)  --> TIME: 2025-02-24 16:20:16 -- STEP: 24170/36938 -- GLOBAL_STEP: 393550 | > loss_text_ce: 0.04067811742424965 (0.03946302100683853) | > loss_mel_ce: 3.5169906616210938 (nan) | > loss: 0.01411773357540369 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.13649672679680058) | > loader_time: 0.0048 (0.004696115315034457)  --> TIME: 2025-02-24 16:20:26 -- STEP: 24220/36938 -- GLOBAL_STEP: 393600 | > loss_text_ce: 0.044759564101696014 (0.039463035673368646) | > loss_mel_ce: 3.5558416843414307 (nan) | > loss: 0.0142881004139781 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13649768682671906) | > loader_time: 0.0046 (0.004696195269300361)  --> TIME: 2025-02-24 16:20:35 -- STEP: 24270/36938 -- GLOBAL_STEP: 393650 | > loss_text_ce: 0.03979435935616493 (0.039459389702849686) | > loss_mel_ce: 2.7557666301727295 (nan) | > loss: 0.011093497276306152 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13649788616024705) | > loader_time: 0.0046 (0.004696377383665535)  --> TIME: 2025-02-24 16:20:44 -- STEP: 24320/36938 -- GLOBAL_STEP: 393700 | > loss_text_ce: 0.024635126814246178 (0.03945675616648064) | > loss_mel_ce: 2.9754247665405273 (nan) | > loss: 0.011904999613761902 (nan) | > current_lr: 5e-06 | > step_time: 0.189 (0.13649808361538038) | > loader_time: 0.0048 (0.004696309527284216)  --> TIME: 2025-02-24 16:20:53 -- STEP: 24370/36938 -- GLOBAL_STEP: 393750 | > loss_text_ce: 0.03553106263279915 (0.039454161973801805) | > loss_mel_ce: 3.02535080909729 (nan) | > loss: 0.01214635744690895 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.1364879816733857) | > loader_time: 0.0046 (0.004696150984390378)  --> TIME: 2025-02-24 16:21:03 -- STEP: 24420/36938 -- GLOBAL_STEP: 393800 | > loss_text_ce: 0.036370061337947845 (0.03945491962645068) | > loss_mel_ce: 3.457890272140503 (nan) | > loss: 0.013866113498806953 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13649071328079485) | > loader_time: 0.0047 (0.004696086261430593)  --> TIME: 2025-02-24 16:21:12 -- STEP: 24470/36938 -- GLOBAL_STEP: 393850 | > loss_text_ce: 0.04287802428007126 (0.03945595487806793) | > loss_mel_ce: 3.005239486694336 (nan) | > loss: 0.012095704674720764 (nan) | > current_lr: 5e-06 | > step_time: 0.1776 (0.13650034953975862) | > loader_time: 0.0049 (0.0046960224265413094)  --> TIME: 2025-02-24 16:21:22 -- STEP: 24520/36938 -- GLOBAL_STEP: 393900 | > loss_text_ce: 0.043332070112228394 (0.039456539274554406) | > loss_mel_ce: 3.6537041664123535 (nan) | > loss: 0.014670779928565025 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13651115264239078) | > loader_time: 0.0048 (0.004696105432743742)  --> TIME: 2025-02-24 16:21:32 -- STEP: 24570/36938 -- GLOBAL_STEP: 393950 | > loss_text_ce: 0.030584944412112236 (0.03945574725939743) | > loss_mel_ce: 3.5029516220092773 (nan) | > loss: 0.014021972194314003 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13652446502876445) | > loader_time: 0.0047 (0.004696132586552551)  --> TIME: 2025-02-24 16:21:41 -- STEP: 24620/36938 -- GLOBAL_STEP: 394000 | > loss_text_ce: 0.041258327662944794 (0.03945653347629826) | > loss_mel_ce: 3.20444393157959 (nan) | > loss: 0.012879771180450916 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13652052378673782) | > loader_time: 0.0048 (0.004696008560620708)  --> TIME: 2025-02-24 16:21:50 -- STEP: 24670/36938 -- GLOBAL_STEP: 394050 | > loss_text_ce: 0.026006774976849556 (0.039452371213706976) | > loss_mel_ce: 2.559609889984131 (nan) | > loss: 0.01026038359850645 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13653334831969127) | > loader_time: 0.0044 (0.004695963463321313)  --> TIME: 2025-02-24 16:22:00 -- STEP: 24720/36938 -- GLOBAL_STEP: 394100 | > loss_text_ce: 0.04018602892756462 (0.03945323380733523) | > loss_mel_ce: 3.1839470863342285 (nan) | > loss: 0.012794178910553455 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13655101412901147) | > loader_time: 0.0046 (0.004695922396715409)  --> TIME: 2025-02-24 16:22:09 -- STEP: 24770/36938 -- GLOBAL_STEP: 394150 | > loss_text_ce: 0.034040313214063644 (0.03945273970414271) | > loss_mel_ce: 3.1039834022521973 (nan) | > loss: 0.012452475726604462 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13654585282168225) | > loader_time: 0.0046 (0.004695754422744916)  --> TIME: 2025-02-24 16:22:19 -- STEP: 24820/36938 -- GLOBAL_STEP: 394200 | > loss_text_ce: 0.040360450744628906 (0.03945356412422202) | > loss_mel_ce: 3.5789706707000732 (nan) | > loss: 0.014362425543367863 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13655671309694703) | > loader_time: 0.0046 (0.00469577424858579)  --> TIME: 2025-02-24 16:22:28 -- STEP: 24870/36938 -- GLOBAL_STEP: 394250 | > loss_text_ce: 0.04598718881607056 (0.0394500063850597) | > loss_mel_ce: 2.754579544067383 (nan) | > loss: 0.011113360524177551 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13655084557067304) | > loader_time: 0.0046 (0.0046957440293837055)  --> TIME: 2025-02-24 16:22:37 -- STEP: 24920/36938 -- GLOBAL_STEP: 394300 | > loss_text_ce: 0.03743356466293335 (0.03944834041279099) | > loss_mel_ce: 3.086968421936035 (nan) | > loss: 0.01239842176437378 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1365473971894996) | > loader_time: 0.0046 (0.004695722714281774)  --> TIME: 2025-02-24 16:22:47 -- STEP: 24970/36938 -- GLOBAL_STEP: 394350 | > loss_text_ce: 0.038910988718271255 (0.03944619890596284) | > loss_mel_ce: 3.482314348220825 (nan) | > loss: 0.013973117806017399 (nan) | > current_lr: 5e-06 | > step_time: 0.1972 (0.13655283529375523) | > loader_time: 0.0049 (0.004695727923511266)  --> TIME: 2025-02-24 16:22:56 -- STEP: 25020/36938 -- GLOBAL_STEP: 394400 | > loss_text_ce: 0.04127666726708412 (0.03944684066768727) | > loss_mel_ce: 2.998927116394043 (nan) | > loss: 0.012064301408827305 (nan) | > current_lr: 5e-06 | > step_time: 0.1762 (0.1365592009634324) | > loader_time: 0.005 (0.004695663282530104)  --> TIME: 2025-02-24 16:23:06 -- STEP: 25070/36938 -- GLOBAL_STEP: 394450 | > loss_text_ce: 0.046571459621191025 (0.03944804884283792) | > loss_mel_ce: 3.5477664470672607 (nan) | > loss: 0.014263246208429337 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13656916935984967) | > loader_time: 0.0046 (0.004695543217668512)  --> TIME: 2025-02-24 16:23:15 -- STEP: 25120/36938 -- GLOBAL_STEP: 394500 | > loss_text_ce: 0.033559344708919525 (0.03944994735985266) | > loss_mel_ce: 2.746173858642578 (nan) | > loss: 0.01103068795055151 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.1365671253223325) | > loader_time: 0.0046 (0.004695403518950114)  --> TIME: 2025-02-24 16:23:24 -- STEP: 25170/36938 -- GLOBAL_STEP: 394550 | > loss_text_ce: 0.035306479781866074 (0.039448444762836224) | > loss_mel_ce: 2.620378255844116 (nan) | > loss: 0.010538431815803051 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.1365610211436408) | > loader_time: 0.0046 (0.0046952720383684745)  --> TIME: 2025-02-24 16:23:34 -- STEP: 25220/36938 -- GLOBAL_STEP: 394600 | > loss_text_ce: 0.030574528500437737 (0.03944995556592398) | > loss_mel_ce: 3.117380380630493 (nan) | > loss: 0.01249188557267189 (nan) | > current_lr: 5e-06 | > step_time: 0.1971 (0.13656392403768214) | > loader_time: 0.007 (0.004695569892235932)  --> TIME: 2025-02-24 16:23:43 -- STEP: 25270/36938 -- GLOBAL_STEP: 394650 | > loss_text_ce: 0.02459753304719925 (0.03944784792609624) | > loss_mel_ce: 3.4377033710479736 (nan) | > loss: 0.01373929064720869 (nan) | > current_lr: 5e-06 | > step_time: 0.1614 (0.13655980872126683) | > loader_time: 0.0047 (0.004695542527603745)  --> TIME: 2025-02-24 16:23:52 -- STEP: 25320/36938 -- GLOBAL_STEP: 394700 | > loss_text_ce: 0.03733748197555542 (0.03944429665359551) | > loss_mel_ce: 2.33589506149292 (nan) | > loss: 0.00941759068518877 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13655418480553708) | > loader_time: 0.0048 (0.004695475873600634)  --> TIME: 2025-02-24 16:24:02 -- STEP: 25370/36938 -- GLOBAL_STEP: 394750 | > loss_text_ce: 0.038246918469667435 (0.03944754906983962) | > loss_mel_ce: 3.8078296184539795 (nan) | > loss: 0.015262208878993988 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.1365580779903743) | > loader_time: 0.0046 (0.00469562485784728)  --> TIME: 2025-02-24 16:24:11 -- STEP: 25420/36938 -- GLOBAL_STEP: 394800 | > loss_text_ce: 0.03664511442184448 (0.039447919326525596) | > loss_mel_ce: 3.434551239013672 (nan) | > loss: 0.013774589635431767 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13656818579727034) | > loader_time: 0.0045 (0.00469591788285179)  --> TIME: 2025-02-24 16:24:21 -- STEP: 25470/36938 -- GLOBAL_STEP: 394850 | > loss_text_ce: 0.03481846675276756 (0.039446889223771216) | > loss_mel_ce: 3.348609209060669 (nan) | > loss: 0.013426301069557667 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.136575692393614) | > loader_time: 0.005 (0.004695846484977692)  --> TIME: 2025-02-24 16:24:30 -- STEP: 25520/36938 -- GLOBAL_STEP: 394900 | > loss_text_ce: 0.04534522071480751 (0.03944749387096421) | > loss_mel_ce: 3.3634033203125 (nan) | > loss: 0.01352678146213293 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.13658304738587335) | > loader_time: 0.0046 (0.004696098437129901)  --> TIME: 2025-02-24 16:24:39 -- STEP: 25570/36938 -- GLOBAL_STEP: 394950 | > loss_text_ce: 0.04568543657660484 (0.03944802605727269) | > loss_mel_ce: 2.86161208152771 (nan) | > loss: 0.011536896228790283 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13657380300058616) | > loader_time: 0.0046 (0.004695991244593974)  --> TIME: 2025-02-24 16:24:49 -- STEP: 25620/36938 -- GLOBAL_STEP: 395000 | > loss_text_ce: 0.04160809516906738 (0.039450147086979484) | > loss_mel_ce: 3.338500738143921 (nan) | > loss: 0.01341313123703003 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13657571624536982) | > loader_time: 0.0046 (0.004696297580427172)  --> TIME: 2025-02-24 16:24:58 -- STEP: 25670/36938 -- GLOBAL_STEP: 395050 | > loss_text_ce: 0.0406438373029232 (0.03944959930050978) | > loss_mel_ce: 3.725633382797241 (nan) | > loss: 0.01494554616510868 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13657271696706672) | > loader_time: 0.0066 (0.004696352592367224)  --> TIME: 2025-02-24 16:25:08 -- STEP: 25720/36938 -- GLOBAL_STEP: 395100 | > loss_text_ce: 0.026540953665971756 (0.039449397793972156) | > loss_mel_ce: 1.3191615343093872 (nan) | > loss: 0.005340089555829763 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13658443033046347) | > loader_time: 0.0046 (0.004696673201179813)  --> TIME: 2025-02-24 16:25:17 -- STEP: 25770/36938 -- GLOBAL_STEP: 395150 | > loss_text_ce: 0.03810490667819977 (0.039449124673701354) | > loss_mel_ce: 2.2558295726776123 (nan) | > loss: 0.009102915413677692 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.13658985091495496) | > loader_time: 0.0049 (0.004696751704824781)  --> TIME: 2025-02-24 16:25:27 -- STEP: 25820/36938 -- GLOBAL_STEP: 395200 | > loss_text_ce: 0.040447819977998734 (0.03944947072459856) | > loss_mel_ce: 0.686657726764679 (nan) | > loss: 0.00288533978164196 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13659777706856904) | > loader_time: 0.0045 (0.004696789653979014)  --> TIME: 2025-02-24 16:25:36 -- STEP: 25870/36938 -- GLOBAL_STEP: 395250 | > loss_text_ce: 0.03796711936593056 (0.03944985118903737) | > loss_mel_ce: 3.518167734146118 (nan) | > loss: 0.014111647382378578 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13660258104960643) | > loader_time: 0.0046 (0.00469661902680931)  --> TIME: 2025-02-24 16:25:46 -- STEP: 25920/36938 -- GLOBAL_STEP: 395300 | > loss_text_ce: 0.04023503512144089 (0.039447790273338466) | > loss_mel_ce: 3.576472759246826 (nan) | > loss: 0.014352016150951385 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13660715610524785) | > loader_time: 0.0047 (0.004696526875098555)  --> TIME: 2025-02-24 16:25:55 -- STEP: 25970/36938 -- GLOBAL_STEP: 395350 | > loss_text_ce: 0.03419980779290199 (0.03944817615517281) | > loss_mel_ce: 3.1744308471679688 (nan) | > loss: 0.012732661329209805 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13660233200721061) | > loader_time: 0.0045 (0.004696545033533854)  --> TIME: 2025-02-24 16:26:04 -- STEP: 26020/36938 -- GLOBAL_STEP: 395400 | > loss_text_ce: 0.02958749234676361 (0.03944379738671572) | > loss_mel_ce: 2.9229023456573486 (nan) | > loss: 0.01171622984111309 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13659533977875093) | > loader_time: 0.0045 (0.004696599040738814)  --> TIME: 2025-02-24 16:26:13 -- STEP: 26070/36938 -- GLOBAL_STEP: 395450 | > loss_text_ce: 0.03937065973877907 (0.03944067778941011) | > loss_mel_ce: 3.714723587036133 (nan) | > loss: 0.014897200278937817 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13659078322199547) | > loader_time: 0.0046 (0.0046965481451200705)  --> TIME: 2025-02-24 16:26:22 -- STEP: 26120/36938 -- GLOBAL_STEP: 395500 | > loss_text_ce: 0.03801107779145241 (0.03944139578101551) | > loss_mel_ce: 3.010256052017212 (nan) | > loss: 0.012096298858523369 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13659205182040474) | > loader_time: 0.0045 (0.004696597412205773)  --> TIME: 2025-02-24 16:26:32 -- STEP: 26170/36938 -- GLOBAL_STEP: 395550 | > loss_text_ce: 0.02514190413057804 (0.03944098522653866) | > loss_mel_ce: 3.962357997894287 (nan) | > loss: 0.01582341268658638 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13659200610939615) | > loader_time: 0.0045 (0.004696809047499286)  --> TIME: 2025-02-24 16:26:41 -- STEP: 26220/36938 -- GLOBAL_STEP: 395600 | > loss_text_ce: 0.03825543448328972 (0.03944212536467462) | > loss_mel_ce: 3.2880265712738037 (nan) | > loss: 0.01319953240454197 (nan) | > current_lr: 5e-06 | > step_time: 0.1777 (0.1365960079236038) | > loader_time: 0.0048 (0.004696701847806777)  --> TIME: 2025-02-24 16:26:50 -- STEP: 26270/36938 -- GLOBAL_STEP: 395650 | > loss_text_ce: 0.04293641820549965 (0.03944116064234209) | > loss_mel_ce: 3.7795894145965576 (nan) | > loss: 0.015168753452599049 (nan) | > current_lr: 5e-06 | > step_time: 0.1884 (0.13659617963470727) | > loader_time: 0.0048 (0.00469656595949625)  --> TIME: 2025-02-24 16:27:00 -- STEP: 26320/36938 -- GLOBAL_STEP: 395700 | > loss_text_ce: 0.035503827035427094 (0.03944093600764001) | > loss_mel_ce: 2.0874946117401123 (nan) | > loss: 0.008424597792327404 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1365937215216618) | > loader_time: 0.0045 (0.004696399000640353)  --> TIME: 2025-02-24 16:27:09 -- STEP: 26370/36938 -- GLOBAL_STEP: 395750 | > loss_text_ce: 0.03508662432432175 (0.039437093999897706) | > loss_mel_ce: 3.607611894607544 (nan) | > loss: 0.014455153606832027 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13660168543791415) | > loader_time: 0.0047 (0.004696276317187062)  --> TIME: 2025-02-24 16:27:18 -- STEP: 26420/36938 -- GLOBAL_STEP: 395800 | > loss_text_ce: 0.026926502585411072 (0.03943639964187445) | > loss_mel_ce: 2.7018210887908936 (nan) | > loss: 0.010828363709151745 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13659610531710958) | > loader_time: 0.0054 (0.004696609764546735)  --> TIME: 2025-02-24 16:27:28 -- STEP: 26470/36938 -- GLOBAL_STEP: 395850 | > loss_text_ce: 0.04248002544045448 (0.039436225522404986) | > loss_mel_ce: 3.1572279930114746 (nan) | > loss: 0.012697254307568073 (nan) | > current_lr: 5e-06 | > step_time: 0.1852 (0.13660109547520316) | > loader_time: 0.0048 (0.00469656546340154)  --> TIME: 2025-02-24 16:27:37 -- STEP: 26520/36938 -- GLOBAL_STEP: 395900 | > loss_text_ce: 0.03861331194639206 (0.039437351257365846) | > loss_mel_ce: 3.5178029537200928 (nan) | > loss: 0.014112764038145542 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.13659351575967976) | > loader_time: 0.0048 (0.0046965742992240165)  --> TIME: 2025-02-24 16:27:46 -- STEP: 26570/36938 -- GLOBAL_STEP: 395950 | > loss_text_ce: 0.043463997542858124 (0.039437206528518384) | > loss_mel_ce: 3.421207904815674 (nan) | > loss: 0.013748698867857456 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13659545208755475) | > loader_time: 0.0046 (0.004696695437591072)  --> TIME: 2025-02-24 16:27:56 -- STEP: 26620/36938 -- GLOBAL_STEP: 396000 | > loss_text_ce: 0.03524826094508171 (0.03943631203574328) | > loss_mel_ce: 2.53424072265625 (nan) | > loss: 0.010196385905146599 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.1366012752100905) | > loader_time: 0.0045 (0.004697021750200084)  --> TIME: 2025-02-24 16:28:06 -- STEP: 26670/36938 -- GLOBAL_STEP: 396050 | > loss_text_ce: 0.049417801201343536 (0.03943748761909172) | > loss_mel_ce: 3.204019784927368 (nan) | > loss: 0.012910466641187668 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13661218296511593) | > loader_time: 0.0046 (0.004697094108444242)  --> TIME: 2025-02-24 16:28:15 -- STEP: 26720/36938 -- GLOBAL_STEP: 396100 | > loss_text_ce: 0.04163191840052605 (0.0394342507721929) | > loss_mel_ce: 2.8994956016540527 (nan) | > loss: 0.011671141721308231 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13661656150382429) | > loader_time: 0.0045 (0.004696990005270457)  --> TIME: 2025-02-24 16:28:24 -- STEP: 26770/36938 -- GLOBAL_STEP: 396150 | > loss_text_ce: 0.041673626750707626 (0.0394320415458502) | > loss_mel_ce: 3.0825035572052 (nan) | > loss: 0.012397529557347298 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13661217317235602) | > loader_time: 0.0045 (0.004697240463245444)  --> TIME: 2025-02-24 16:28:34 -- STEP: 26820/36938 -- GLOBAL_STEP: 396200 | > loss_text_ce: 0.04137855023145676 (0.03943149966054406) | > loss_mel_ce: 3.4065799713134766 (nan) | > loss: 0.01368237566202879 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.13661471886211665) | > loader_time: 0.0055 (0.004697536062074194)  --> TIME: 2025-02-24 16:28:43 -- STEP: 26870/36938 -- GLOBAL_STEP: 396250 | > loss_text_ce: 0.041306715458631516 (0.039429659362754575) | > loss_mel_ce: 1.625280737876892 (nan) | > loss: 0.006613442674279213 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.1366087524252925) | > loader_time: 0.0046 (0.004697795370319854)  --> TIME: 2025-02-24 16:28:52 -- STEP: 26920/36938 -- GLOBAL_STEP: 396300 | > loss_text_ce: 0.028569472953677177 (0.03943372654626854) | > loss_mel_ce: 2.639474868774414 (nan) | > loss: 0.010587478056550026 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.1366126663086137) | > loader_time: 0.0046 (0.004697996351375272)  --> TIME: 2025-02-24 16:29:02 -- STEP: 26970/36938 -- GLOBAL_STEP: 396350 | > loss_text_ce: 0.042766984552145004 (0.03943404072984022) | > loss_mel_ce: 3.2400925159454346 (nan) | > loss: 0.013027220964431763 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13661995395537305) | > loader_time: 0.0046 (0.004697941151379571)  --> TIME: 2025-02-24 16:29:12 -- STEP: 27020/36938 -- GLOBAL_STEP: 396400 | > loss_text_ce: 0.041908569633960724 (0.039434471152561275) | > loss_mel_ce: 2.638937473297119 (nan) | > loss: 0.010638277977705002 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13663205282499274) | > loader_time: 0.0049 (0.0046980098033286705)  --> TIME: 2025-02-24 16:29:20 -- STEP: 27070/36938 -- GLOBAL_STEP: 396450 | > loss_text_ce: 0.03654501214623451 (0.039433516498852995) | > loss_mel_ce: 3.021627187728882 (nan) | > loss: 0.01213560439646244 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13661963878014333) | > loader_time: 0.0046 (0.00469779306819706)  --> TIME: 2025-02-24 16:29:30 -- STEP: 27120/36938 -- GLOBAL_STEP: 396500 | > loss_text_ce: 0.037136469036340714 (0.039432832841406765) | > loss_mel_ce: 2.5848631858825684 (nan) | > loss: 0.010404761880636215 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13661784796939816) | > loader_time: 0.0046 (0.004697632455544491)  --> TIME: 2025-02-24 16:29:38 -- STEP: 27170/36938 -- GLOBAL_STEP: 396550 | > loss_text_ce: 0.0354171060025692 (0.039430884742491415) | > loss_mel_ce: 0.0987548828125 (nan) | > loss: 0.0005324285593815148 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13660410434216025) | > loader_time: 0.0046 (0.00469778358914373)  --> TIME: 2025-02-24 16:29:48 -- STEP: 27220/36938 -- GLOBAL_STEP: 396600 | > loss_text_ce: 0.054750628769397736 (0.039428433093798426) | > loss_mel_ce: 1.1088343858718872 (nan) | > loss: 0.004617401398718357 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.1366085925144274) | > loader_time: 0.0045 (0.004697725450528364)  --> TIME: 2025-02-24 16:29:57 -- STEP: 27270/36938 -- GLOBAL_STEP: 396650 | > loss_text_ce: 0.03836129978299141 (0.03943183922564342) | > loss_mel_ce: 2.9648425579071045 (nan) | > loss: 0.011917476542294025 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13661041698658613) | > loader_time: 0.0047 (0.004697556350534551)  --> TIME: 2025-02-24 16:30:06 -- STEP: 27320/36938 -- GLOBAL_STEP: 396700 | > loss_text_ce: 0.03271898254752159 (0.0394284692822679) | > loss_mel_ce: 3.4225645065307617 (nan) | > loss: 0.013711443170905113 (nan) | > current_lr: 5e-06 | > step_time: 0.1771 (0.13660388041729424) | > loader_time: 0.0048 (0.004697308114472108)  --> TIME: 2025-02-24 16:30:16 -- STEP: 27370/36938 -- GLOBAL_STEP: 396750 | > loss_text_ce: 0.03361029550433159 (0.03942650096000283) | > loss_mel_ce: 2.5463619232177734 (nan) | > loss: 0.010237986221909523 (nan) | > current_lr: 5e-06 | > step_time: 0.1526 (0.13661092823336565) | > loader_time: 0.0046 (0.004697222234038138)  --> TIME: 2025-02-24 16:30:25 -- STEP: 27420/36938 -- GLOBAL_STEP: 396800 | > loss_text_ce: 0.03242199122905731 (0.039427657303604556) | > loss_mel_ce: 3.633244752883911 (nan) | > loss: 0.014546297490596771 (nan) | > current_lr: 5e-06 | > step_time: 0.1577 (0.13660947977409643) | > loader_time: 0.0046 (0.0046971164594854795)  --> TIME: 2025-02-24 16:30:34 -- STEP: 27470/36938 -- GLOBAL_STEP: 396850 | > loss_text_ce: 0.04655423387885094 (0.039427302630152136) | > loss_mel_ce: 0.02197265625 (nan) | > loss: 0.0002719321346376091 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13660347597096487) | > loader_time: 0.0045 (0.004696998346229447)  --> TIME: 2025-02-24 16:30:44 -- STEP: 27520/36938 -- GLOBAL_STEP: 396900 | > loss_text_ce: 0.031158404424786568 (0.03942615684483251) | > loss_mel_ce: 2.6907806396484375 (nan) | > loss: 0.01080134604126215 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13660172482216068) | > loader_time: 0.0046 (0.004696998364010525)  --> TIME: 2025-02-24 16:30:53 -- STEP: 27570/36938 -- GLOBAL_STEP: 396950 | > loss_text_ce: 0.04520198702812195 (0.03942611696591085) | > loss_mel_ce: 2.9485294818878174 (nan) | > loss: 0.01187988743185997 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13660399176999188) | > loader_time: 0.0046 (0.004697144407701961)  --> TIME: 2025-02-24 16:31:02 -- STEP: 27620/36938 -- GLOBAL_STEP: 397000 | > loss_text_ce: 0.044611476361751556 (0.039425781912154595) | > loss_mel_ce: 3.3149592876434326 (nan) | > loss: 0.013331630267202854 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13659796565447385) | > loader_time: 0.0045 (0.004697016354145823)  --> TIME: 2025-02-24 16:31:12 -- STEP: 27670/36938 -- GLOBAL_STEP: 397050 | > loss_text_ce: 0.038275208324193954 (0.03942609065897902) | > loss_mel_ce: 2.3651225566864014 (nan) | > loss: 0.009537293575704098 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13660508039672312) | > loader_time: 0.0045 (0.004696902765192021)  --> TIME: 2025-02-24 16:31:21 -- STEP: 27720/36938 -- GLOBAL_STEP: 397100 | > loss_text_ce: 0.03762306645512581 (0.03942758168932429) | > loss_mel_ce: 1.0604643821716309 (nan) | > loss: 0.004357490222901106 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13659936316945673) | > loader_time: 0.0047 (0.004697333579455621)  --> TIME: 2025-02-24 16:31:30 -- STEP: 27770/36938 -- GLOBAL_STEP: 397150 | > loss_text_ce: 0.04185579717159271 (0.039429085302143924) | > loss_mel_ce: 3.351206064224243 (nan) | > loss: 0.013464531861245632 (nan) | > current_lr: 5e-06 | > step_time: 0.1728 (0.1365957551007653) | > loader_time: 0.0048 (0.0046973201066931695)  --> TIME: 2025-02-24 16:31:40 -- STEP: 27820/36938 -- GLOBAL_STEP: 397200 | > loss_text_ce: 0.0362158939242363 (0.03943105969178961) | > loss_mel_ce: 3.487136125564575 (nan) | > loss: 0.013981555588543415 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.13660527285185753) | > loader_time: 0.0048 (0.004697258621561367)  --> TIME: 2025-02-24 16:31:49 -- STEP: 27870/36938 -- GLOBAL_STEP: 397250 | > loss_text_ce: 0.03940282762050629 (0.03943170730457632) | > loss_mel_ce: 3.1112852096557617 (nan) | > loss: 0.012502730824053288 (nan) | > current_lr: 5e-06 | > step_time: 0.1809 (0.13660574285283403) | > loader_time: 0.0048 (0.004697396176067528)  --> TIME: 2025-02-24 16:31:58 -- STEP: 27920/36938 -- GLOBAL_STEP: 397300 | > loss_text_ce: 0.046126607805490494 (0.0394297137118628) | > loss_mel_ce: 3.8698418140411377 (nan) | > loss: 0.015539557673037052 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13659669689428477) | > loader_time: 0.0046 (0.004697244932453409)  --> TIME: 2025-02-24 16:32:07 -- STEP: 27970/36938 -- GLOBAL_STEP: 397350 | > loss_text_ce: 0.046927206218242645 (0.03942904730777248) | > loss_mel_ce: 3.0276336669921875 (nan) | > loss: 0.012200639583170414 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1366023035274472) | > loader_time: 0.0046 (0.00469730669232322)  --> TIME: 2025-02-24 16:32:17 -- STEP: 28020/36938 -- GLOBAL_STEP: 397400 | > loss_text_ce: 0.03720966726541519 (0.0394271840968916) | > loss_mel_ce: 3.3030498027801514 (nan) | > loss: 0.013254999183118343 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.13661395213503547) | > loader_time: 0.0048 (0.004697474218282763)  --> TIME: 2025-02-24 16:32:26 -- STEP: 28070/36938 -- GLOBAL_STEP: 397450 | > loss_text_ce: 0.04891335964202881 (0.03942687072033668) | > loss_mel_ce: 1.142954707145691 (nan) | > loss: 0.004729635547846556 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13661553948045316) | > loader_time: 0.0045 (0.004697352530992793)  --> TIME: 2025-02-24 16:32:36 -- STEP: 28120/36938 -- GLOBAL_STEP: 397500 | > loss_text_ce: 0.03637364134192467 (0.039424820479659776) | > loss_mel_ce: 2.9339420795440674 (nan) | > loss: 0.011786967515945435 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13661345310435002) | > loader_time: 0.0045 (0.004697189307314575)  --> TIME: 2025-02-24 16:32:45 -- STEP: 28170/36938 -- GLOBAL_STEP: 397550 | > loss_text_ce: 0.04263269901275635 (0.03942349466138466) | > loss_mel_ce: 2.831328868865967 (nan) | > loss: 0.011404609307646751 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.13661127320663005) | > loader_time: 0.005 (0.004697271615510129)  --> TIME: 2025-02-24 16:32:54 -- STEP: 28220/36938 -- GLOBAL_STEP: 397600 | > loss_text_ce: 0.039136167615652084 (0.03942104034036015) | > loss_mel_ce: 3.443740129470825 (nan) | > loss: 0.013820938766002655 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.1366077661091851) | > loader_time: 0.0054 (0.004697364437757531)  --> TIME: 2025-02-24 16:33:03 -- STEP: 28270/36938 -- GLOBAL_STEP: 397650 | > loss_text_ce: 0.035005029290914536 (0.039420623150872976) | > loss_mel_ce: 2.3402099609375 (nan) | > loss: 0.009425457566976547 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13660488707208754) | > loader_time: 0.0046 (0.004697355854667285)  --> TIME: 2025-02-24 16:33:12 -- STEP: 28320/36938 -- GLOBAL_STEP: 397700 | > loss_text_ce: 0.031925901770591736 (0.03941712043896599) | > loss_mel_ce: 2.584404706954956 (nan) | > loss: 0.010382264852523804 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13659716336067065) | > loader_time: 0.0045 (0.004697361462197063)  --> TIME: 2025-02-24 16:33:22 -- STEP: 28370/36938 -- GLOBAL_STEP: 397750 | > loss_text_ce: 0.03620247542858124 (0.03941514776710359) | > loss_mel_ce: 0.1413530558347702 (nan) | > loss: 0.0007045854581519961 (nan) | > current_lr: 5e-06 | > step_time: 0.1777 (0.1365904911160254) | > loader_time: 0.0048 (0.004697496150638339)  --> TIME: 2025-02-24 16:33:31 -- STEP: 28420/36938 -- GLOBAL_STEP: 397800 | > loss_text_ce: 0.03588501736521721 (0.03941322231663919) | > loss_mel_ce: 3.2364649772644043 (nan) | > loss: 0.012985517270863056 (nan) | > current_lr: 5e-06 | > step_time: 0.1505 (0.13658835636235508) | > loader_time: 0.0047 (0.004697794791764564)  --> TIME: 2025-02-24 16:33:40 -- STEP: 28470/36938 -- GLOBAL_STEP: 397850 | > loss_text_ce: 0.035996004939079285 (0.039412521011669185) | > loss_mel_ce: 3.8132596015930176 (nan) | > loss: 0.01527482457458973 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.1365893346115203) | > loader_time: 0.0053 (0.00469812212134396)  --> TIME: 2025-02-24 16:33:49 -- STEP: 28520/36938 -- GLOBAL_STEP: 397900 | > loss_text_ce: 0.05172552540898323 (0.039414326272033395) | > loss_mel_ce: 1.4161537885665894 (nan) | > loss: 0.005824918393045664 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13658916098372573) | > loader_time: 0.0045 (0.0046982452545059035)  --> TIME: 2025-02-24 16:33:59 -- STEP: 28570/36938 -- GLOBAL_STEP: 397950 | > loss_text_ce: 0.043418120592832565 (0.03941505280276688) | > loss_mel_ce: 3.2292728424072266 (nan) | > loss: 0.012986869551241398 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13659809319172644) | > loader_time: 0.0047 (0.004698327241095791)  --> TIME: 2025-02-24 16:34:09 -- STEP: 28620/36938 -- GLOBAL_STEP: 398000 | > loss_text_ce: 0.04440280422568321 (0.03941211560474207) | > loss_mel_ce: 2.937952995300293 (nan) | > loss: 0.011834746226668358 (nan) | > current_lr: 5e-06 | > step_time: 0.1811 (0.13660495903506906) | > loader_time: 0.0051 (0.004698349786421207)  --> TIME: 2025-02-24 16:34:18 -- STEP: 28670/36938 -- GLOBAL_STEP: 398050 | > loss_text_ce: 0.03968295827507973 (0.03941166048052813) | > loss_mel_ce: 2.3992722034454346 (nan) | > loss: 0.009678393602371216 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.1366043417803861) | > loader_time: 0.0047 (0.004698332918617509)  --> TIME: 2025-02-24 16:34:27 -- STEP: 28720/36938 -- GLOBAL_STEP: 398100 | > loss_text_ce: 0.036904558539390564 (0.039411676480515016) | > loss_mel_ce: 2.8713550567626953 (nan) | > loss: 0.011540713720023632 (nan) | > current_lr: 5e-06 | > step_time: 0.1648 (0.13659472348464172) | > loader_time: 0.0048 (0.004698248344543587)  --> TIME: 2025-02-24 16:34:36 -- STEP: 28770/36938 -- GLOBAL_STEP: 398150 | > loss_text_ce: 0.04185152053833008 (0.03941193238498932) | > loss_mel_ce: 2.8507564067840576 (nan) | > loss: 0.011478603817522526 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13657992187787676) | > loader_time: 0.0045 (0.004698160070075235)  --> TIME: 2025-02-24 16:34:45 -- STEP: 28820/36938 -- GLOBAL_STEP: 398200 | > loss_text_ce: 0.04019467160105705 (0.03941064313335849) | > loss_mel_ce: 2.7866528034210205 (nan) | > loss: 0.011217650026082993 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1365796647495393) | > loader_time: 0.0046 (0.004698260247417825)  --> TIME: 2025-02-24 16:34:54 -- STEP: 28870/36938 -- GLOBAL_STEP: 398250 | > loss_text_ce: 0.023543450981378555 (0.039408891728185694) | > loss_mel_ce: 2.6518797874450684 (nan) | > loss: 0.010616758838295937 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.1365832885161455) | > loader_time: 0.0046 (0.004698241033643269)  --> TIME: 2025-02-24 16:35:04 -- STEP: 28920/36938 -- GLOBAL_STEP: 398300 | > loss_text_ce: 0.048870425671339035 (0.0394099629430657) | > loss_mel_ce: 3.0386343002319336 (nan) | > loss: 0.012252002954483032 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.1365807147853762) | > loader_time: 0.0046 (0.0046980938970795335)  --> TIME: 2025-02-24 16:35:13 -- STEP: 28970/36938 -- GLOBAL_STEP: 398350 | > loss_text_ce: 0.04421767592430115 (0.039410596950031504) | > loss_mel_ce: 2.7169029712677 (nan) | > loss: 0.010956828482449055 (nan) | > current_lr: 5e-06 | > step_time: 0.1659 (0.13658161869450636) | > loader_time: 0.0048 (0.004698071399144895)  --> TIME: 2025-02-24 16:35:22 -- STEP: 29020/36938 -- GLOBAL_STEP: 398400 | > loss_text_ce: 0.044461287558078766 (0.039410876417994456) | > loss_mel_ce: 3.8047122955322266 (nan) | > loss: 0.015274499543011189 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13658098426873985) | > loader_time: 0.0047 (0.004698081964628194)  --> TIME: 2025-02-24 16:35:31 -- STEP: 29070/36938 -- GLOBAL_STEP: 398450 | > loss_text_ce: 0.039825864136219025 (0.03941154775379819) | > loss_mel_ce: 3.181913137435913 (nan) | > loss: 0.012784679420292377 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.13657113423985462) | > loader_time: 0.0049 (0.004698194192782031)  --> TIME: 2025-02-24 16:35:40 -- STEP: 29120/36938 -- GLOBAL_STEP: 398500 | > loss_text_ce: 0.039388444274663925 (0.03941221610597531) | > loss_mel_ce: 3.319160223007202 (nan) | > loss: 0.013327574357390404 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.13656687555568464) | > loader_time: 0.0053 (0.004698463472036217)  --> TIME: 2025-02-24 16:35:50 -- STEP: 29170/36938 -- GLOBAL_STEP: 398550 | > loss_text_ce: 0.03464384377002716 (0.039410723214454775) | > loss_mel_ce: 2.103278398513794 (nan) | > loss: 0.008483818732202053 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13656646249880225) | > loader_time: 0.0046 (0.00469856147779191)  --> TIME: 2025-02-24 16:35:59 -- STEP: 29220/36938 -- GLOBAL_STEP: 398600 | > loss_text_ce: 0.05402262881398201 (0.03941224935307724) | > loss_mel_ce: 3.5558862686157227 (nan) | > loss: 0.014325035735964775 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1365634187699008) | > loader_time: 0.0045 (0.00469855561311893)  --> TIME: 2025-02-24 16:36:08 -- STEP: 29270/36938 -- GLOBAL_STEP: 398650 | > loss_text_ce: 0.03696565330028534 (0.03940943399008924) | > loss_mel_ce: 1.8598557710647583 (nan) | > loss: 0.007527069188654423 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.1365640209525112) | > loader_time: 0.0053 (0.00469879135565828)  --> TIME: 2025-02-24 16:36:17 -- STEP: 29320/36938 -- GLOBAL_STEP: 398700 | > loss_text_ce: 0.031317051500082016 (0.03940902799961869) | > loss_mel_ce: 2.926374912261963 (nan) | > loss: 0.011736873537302017 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.13655755915062726) | > loader_time: 0.0053 (0.004699337165319905)  --> TIME: 2025-02-24 16:36:27 -- STEP: 29370/36938 -- GLOBAL_STEP: 398750 | > loss_text_ce: 0.042185328900814056 (0.039408917552298216) | > loss_mel_ce: 3.0914721488952637 (nan) | > loss: 0.012435149401426315 (nan) | > current_lr: 5e-06 | > step_time: 0.1499 (0.13656403107589457) | > loader_time: 0.0047 (0.004699484937484063)  --> TIME: 2025-02-24 16:36:36 -- STEP: 29420/36938 -- GLOBAL_STEP: 398800 | > loss_text_ce: 0.04019326716661453 (0.03940814585731368) | > loss_mel_ce: 2.698138475418091 (nan) | > loss: 0.010866397060453892 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.1365597537713006) | > loader_time: 0.0047 (0.004699464682249049)  --> TIME: 2025-02-24 16:36:46 -- STEP: 29470/36938 -- GLOBAL_STEP: 398850 | > loss_text_ce: 0.04022582992911339 (0.039405859483757764) | > loss_mel_ce: 3.190204620361328 (nan) | > loss: 0.012819169089198112 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13656857477271522) | > loader_time: 0.0047 (0.004699805497233247)  --> TIME: 2025-02-24 16:36:55 -- STEP: 29520/36938 -- GLOBAL_STEP: 398900 | > loss_text_ce: 0.04707472398877144 (0.039405345702499456) | > loss_mel_ce: 3.68386173248291 (nan) | > loss: 0.014805304817855358 (nan) | > current_lr: 5e-06 | > step_time: 0.1821 (0.1365731779395072) | > loader_time: 0.005 (0.004700055548815225)  --> TIME: 2025-02-24 16:37:05 -- STEP: 29570/36938 -- GLOBAL_STEP: 398950 | > loss_text_ce: 0.036998286843299866 (0.039405150236048285) | > loss_mel_ce: 3.325176239013672 (nan) | > loss: 0.013341963291168213 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.13657396938658017) | > loader_time: 0.0047 (0.004699951295397918)  --> TIME: 2025-02-24 16:37:14 -- STEP: 29620/36938 -- GLOBAL_STEP: 399000 | > loss_text_ce: 0.031567711383104324 (0.039404630511942304) | > loss_mel_ce: 3.466416835784912 (nan) | > loss: 0.013880892656743526 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13657463182073204) | > loader_time: 0.0046 (0.0047001542310953)  --> TIME: 2025-02-24 16:37:23 -- STEP: 29670/36938 -- GLOBAL_STEP: 399050 | > loss_text_ce: 0.03570534288883209 (0.03940564963485072) | > loss_mel_ce: 2.2450222969055176 (nan) | > loss: 0.009050507098436356 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13656850614891566) | > loader_time: 0.0046 (0.00469999043355697)  --> TIME: 2025-02-24 16:37:33 -- STEP: 29720/36938 -- GLOBAL_STEP: 399100 | > loss_text_ce: 0.036081310361623764 (0.03940635579917055) | > loss_mel_ce: 3.1684792041778564 (nan) | > loss: 0.012716510333120823 (nan) | > current_lr: 5e-06 | > step_time: 0.1563 (0.13657763536974285) | > loader_time: 0.0051 (0.004700083567348498)  --> TIME: 2025-02-24 16:37:42 -- STEP: 29770/36938 -- GLOBAL_STEP: 399150 | > loss_text_ce: 0.0453585721552372 (0.03940641533243636) | > loss_mel_ce: 3.1974644660949707 (nan) | > loss: 0.012868346646428108 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13658026246604799) | > loader_time: 0.0046 (0.004700247585513188)  --> TIME: 2025-02-24 16:37:51 -- STEP: 29820/36938 -- GLOBAL_STEP: 399200 | > loss_text_ce: 0.047976408153772354 (0.03940520204218253) | > loss_mel_ce: 2.9061028957366943 (nan) | > loss: 0.011722537688910961 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1365803243408732) | > loader_time: 0.0045 (0.004700554128783731)  --> TIME: 2025-02-24 16:38:01 -- STEP: 29870/36938 -- GLOBAL_STEP: 399250 | > loss_text_ce: 0.0312768891453743 (0.03940662808088325) | > loss_mel_ce: 5.40625 (nan) | > loss: 0.021577488631010056 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13657765359753954) | > loader_time: 0.0045 (0.004700550068809631)  --> TIME: 2025-02-24 16:38:10 -- STEP: 29920/36938 -- GLOBAL_STEP: 399300 | > loss_text_ce: 0.03998424857854843 (0.03940642883189941) | > loss_mel_ce: 3.168738842010498 (nan) | > loss: 0.012733029201626778 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.136579928455505) | > loader_time: 0.0045 (0.004700520778084823)  --> TIME: 2025-02-24 16:38:19 -- STEP: 29970/36938 -- GLOBAL_STEP: 399350 | > loss_text_ce: 0.05831930413842201 (0.03940571058497387) | > loss_mel_ce: 2.4923338890075684 (nan) | > loss: 0.010121640749275684 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1365821969879037) | > loader_time: 0.0047 (0.0047004718640505355)  --> TIME: 2025-02-24 16:38:29 -- STEP: 30020/36938 -- GLOBAL_STEP: 399400 | > loss_text_ce: 0.038979507982730865 (0.03940320068807117) | > loss_mel_ce: 2.824620008468628 (nan) | > loss: 0.011363490484654903 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13658333239278914) | > loader_time: 0.0045 (0.004700619486631513)  --> TIME: 2025-02-24 16:38:38 -- STEP: 30070/36938 -- GLOBAL_STEP: 399450 | > loss_text_ce: 0.034165386110544205 (0.039399514256536206) | > loss_mel_ce: 1.7158126831054688 (nan) | > loss: 0.006944357883185148 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13657933083094287) | > loader_time: 0.0045 (0.004700605021706048)  --> TIME: 2025-02-24 16:38:47 -- STEP: 30120/36938 -- GLOBAL_STEP: 399500 | > loss_text_ce: 0.06310277432203293 (0.0394013444388748) | > loss_mel_ce: 3.3726775646209717 (nan) | > loss: 0.013634049333631992 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13658245097435107) | > loader_time: 0.0045 (0.004700500200469182)  --> TIME: 2025-02-24 16:38:57 -- STEP: 30170/36938 -- GLOBAL_STEP: 399550 | > loss_text_ce: 0.03949372470378876 (0.03940084430567816) | > loss_mel_ce: 3.3019654750823975 (nan) | > loss: 0.013259760104119778 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.13658586534633452) | > loader_time: 0.0048 (0.004700587433539049)  --> TIME: 2025-02-24 16:39:06 -- STEP: 30220/36938 -- GLOBAL_STEP: 399600 | > loss_text_ce: 0.03989699110388756 (0.039400741573722475) | > loss_mel_ce: 2.322352409362793 (nan) | > loss: 0.00937400572001934 (nan) | > current_lr: 5e-06 | > step_time: 0.1077 (0.13658645390517438) | > loader_time: 0.0045 (0.004700658512304827)  --> TIME: 2025-02-24 16:39:15 -- STEP: 30270/36938 -- GLOBAL_STEP: 399650 | > loss_text_ce: 0.03583475202322006 (0.03939740169503518) | > loss_mel_ce: 3.081066131591797 (nan) | > loss: 0.01236865483224392 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13658064053639052) | > loader_time: 0.0045 (0.004700675654970542)  --> TIME: 2025-02-24 16:39:25 -- STEP: 30320/36938 -- GLOBAL_STEP: 399700 | > loss_text_ce: 0.04167168587446213 (0.03939560339948887) | > loss_mel_ce: 2.154571056365967 (nan) | > loss: 0.008715249598026276 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13657813952906403) | > loader_time: 0.0047 (0.004700926488810921)  --> TIME: 2025-02-24 16:39:34 -- STEP: 30370/36938 -- GLOBAL_STEP: 399750 | > loss_text_ce: 0.03331673890352249 (0.0393967370264068) | > loss_mel_ce: 1.742003083229065 (nan) | > loss: 0.007044920232146978 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13658332075772742) | > loader_time: 0.0046 (0.0047010165906321445)  --> TIME: 2025-02-24 16:39:43 -- STEP: 30420/36938 -- GLOBAL_STEP: 399800 | > loss_text_ce: 0.038574256002902985 (0.039392749158626124) | > loss_mel_ce: 1.5384042263031006 (nan) | > loss: 0.006257851142436266 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13657966728354598) | > loader_time: 0.0047 (0.004701290398659793)  --> TIME: 2025-02-24 16:39:52 -- STEP: 30470/36938 -- GLOBAL_STEP: 399850 | > loss_text_ce: 0.041437968611717224 (0.039392679354159195) | > loss_mel_ce: 3.489086389541626 (nan) | > loss: 0.014010017737746239 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.1365752335333372) | > loader_time: 0.0048 (0.004701611868154429)  --> TIME: 2025-02-24 16:40:02 -- STEP: 30520/36938 -- GLOBAL_STEP: 399900 | > loss_text_ce: 0.034423936158418655 (0.03939119207524449) | > loss_mel_ce: 3.094356060028076 (nan) | > loss: 0.012415793724358082 (nan) | > current_lr: 5e-06 | > step_time: 0.194 (0.13658749428364053) | > loader_time: 0.0056 (0.004701817832486816)  --> TIME: 2025-02-24 16:40:11 -- STEP: 30570/36938 -- GLOBAL_STEP: 399950 | > loss_text_ce: 0.04360288381576538 (0.03939202920094081) | > loss_mel_ce: 2.9986283779144287 (nan) | > loss: 0.012072347104549408 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13658449462632255) | > loader_time: 0.0046 (0.004701865924352286)  --> TIME: 2025-02-24 16:40:21 -- STEP: 30620/36938 -- GLOBAL_STEP: 400000 | > loss_text_ce: 0.04126032814383507 (0.039388529722232474) | > loss_mel_ce: 3.2812306880950928 (nan) | > loss: 0.013184488750994205 (nan) | > current_lr: 5e-06 | > step_time: 0.1899 (0.13658603868166752) | > loader_time: 0.0048 (0.004701861098729889) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_400000.pth  --> TIME: 2025-02-24 16:40:36 -- STEP: 30670/36938 -- GLOBAL_STEP: 400050 | > loss_text_ce: 0.037661585956811905 (0.03939038627715058) | > loss_mel_ce: 3.7842154502868652 (nan) | > loss: 0.015166179277002811 (nan) | > current_lr: 5e-06 | > step_time: 0.1846 (0.13658806397699494) | > loader_time: 0.0048 (0.004701794760212435)  --> TIME: 2025-02-24 16:40:45 -- STEP: 30720/36938 -- GLOBAL_STEP: 400100 | > loss_text_ce: 0.042734745889902115 (0.03938990105322448) | > loss_mel_ce: 2.720106363296509 (nan) | > loss: 0.01096365600824356 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13658026518920996) | > loader_time: 0.0045 (0.004701628830904764)  --> TIME: 2025-02-24 16:40:55 -- STEP: 30770/36938 -- GLOBAL_STEP: 400150 | > loss_text_ce: 0.03259100764989853 (0.039388535516036695) | > loss_mel_ce: 2.940577745437622 (nan) | > loss: 0.011798289604485035 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13657778616574448) | > loader_time: 0.0047 (0.004701488003313911)  --> TIME: 2025-02-24 16:41:03 -- STEP: 30820/36938 -- GLOBAL_STEP: 400200 | > loss_text_ce: 0.032016001641750336 (0.039389902844965616) | > loss_mel_ce: 2.0543971061706543 (nan) | > loss: 0.008279417641460896 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1365666956115583) | > loader_time: 0.0046 (0.004701328416833007)  --> TIME: 2025-02-24 16:41:13 -- STEP: 30870/36938 -- GLOBAL_STEP: 400250 | > loss_text_ce: 0.057659633457660675 (0.039390939026907865) | > loss_mel_ce: 3.005791425704956 (nan) | > loss: 0.012156552635133266 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13656448116957365) | > loader_time: 0.0047 (0.004701219301684066)  --> TIME: 2025-02-24 16:41:22 -- STEP: 30920/36938 -- GLOBAL_STEP: 400300 | > loss_text_ce: 0.034864190965890884 (0.03938760025879132) | > loss_mel_ce: 3.1778321266174316 (nan) | > loss: 0.012748795561492443 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.13657012406789562) | > loader_time: 0.0047 (0.004701130271420598)  --> TIME: 2025-02-24 16:41:32 -- STEP: 30970/36938 -- GLOBAL_STEP: 400350 | > loss_text_ce: 0.05077442526817322 (0.039384315485296285) | > loss_mel_ce: 2.1970062255859375 (nan) | > loss: 0.008919764310121536 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.1365781221306622) | > loader_time: 0.0047 (0.00470112708063714)  --> TIME: 2025-02-24 16:41:41 -- STEP: 31020/36938 -- GLOBAL_STEP: 400400 | > loss_text_ce: 0.03740419074892998 (0.03938407584316862) | > loss_mel_ce: 0.009240374900400639 (nan) | > loss: 0.00018509749497752637 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13657708898964827) | > loader_time: 0.0045 (0.004701039385288322)  --> TIME: 2025-02-24 16:41:50 -- STEP: 31070/36938 -- GLOBAL_STEP: 400450 | > loss_text_ce: 0.03960482403635979 (0.0393818262456409) | > loss_mel_ce: 2.7745680809020996 (nan) | > loss: 0.011167353950440884 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13657482638787194) | > loader_time: 0.0045 (0.00470086950407099)  --> TIME: 2025-02-24 16:41:59 -- STEP: 31120/36938 -- GLOBAL_STEP: 400500 | > loss_text_ce: 0.03855494037270546 (0.03938174548150068) | > loss_mel_ce: 2.7003543376922607 (nan) | > loss: 0.010868688113987446 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13656559974951057) | > loader_time: 0.0046 (0.004700690921597127)  --> TIME: 2025-02-24 16:42:09 -- STEP: 31170/36938 -- GLOBAL_STEP: 400550 | > loss_text_ce: 0.03976805508136749 (0.039379575816399705) | > loss_mel_ce: 3.170332670211792 (nan) | > loss: 0.012738495133817196 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13656390234182553) | > loader_time: 0.0046 (0.004700667712028341)  --> TIME: 2025-02-24 16:42:18 -- STEP: 31220/36938 -- GLOBAL_STEP: 400600 | > loss_text_ce: 0.03861576318740845 (0.03937829069892838) | > loss_mel_ce: 2.792015790939331 (nan) | > loss: 0.011232665739953518 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.13655772063305957) | > loader_time: 0.0048 (0.00470073627096808)  --> TIME: 2025-02-24 16:42:27 -- STEP: 31270/36938 -- GLOBAL_STEP: 400650 | > loss_text_ce: 0.04043584689497948 (0.03937703731803659) | > loss_mel_ce: 3.5135254859924316 (nan) | > loss: 0.01410302147269249 (nan) | > current_lr: 5e-06 | > step_time: 0.1465 (0.13655987231160277) | > loader_time: 0.0047 (0.004700760975561092)  --> TIME: 2025-02-24 16:42:36 -- STEP: 31320/36938 -- GLOBAL_STEP: 400700 | > loss_text_ce: 0.032488878816366196 (0.03937971292139101) | > loss_mel_ce: 2.634089946746826 (nan) | > loss: 0.010581661947071552 (nan) | > current_lr: 5e-06 | > step_time: 0.1264 (0.13655794736373714) | > loader_time: 0.0047 (0.004700819757858609)  --> TIME: 2025-02-24 16:42:45 -- STEP: 31370/36938 -- GLOBAL_STEP: 400750 | > loss_text_ce: 0.031260937452316284 (0.039378847088842685) | > loss_mel_ce: 2.959763288497925 (nan) | > loss: 0.01186914462596178 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.1365551961847724) | > loader_time: 0.0045 (0.004700865478017053)  --> TIME: 2025-02-24 16:42:55 -- STEP: 31420/36938 -- GLOBAL_STEP: 400800 | > loss_text_ce: 0.038558535277843475 (0.03937898857254018) | > loss_mel_ce: 0.8787606954574585 (nan) | > loss: 0.0036401560064405203 (nan) | > current_lr: 5e-06 | > step_time: 0.1268 (0.13655964621004157) | > loader_time: 0.0045 (0.00470080585103366)  --> TIME: 2025-02-24 16:43:05 -- STEP: 31470/36938 -- GLOBAL_STEP: 400850 | > loss_text_ce: 0.033231209963560104 (0.03937820397210258) | > loss_mel_ce: 3.483271837234497 (nan) | > loss: 0.013954377733170986 (nan) | > current_lr: 5e-06 | > step_time: 0.1873 (0.1365742344381995) | > loader_time: 0.0048 (0.004700742989144025)  --> TIME: 2025-02-24 16:43:14 -- STEP: 31520/36938 -- GLOBAL_STEP: 400900 | > loss_text_ce: 0.04405521601438522 (0.03937760155738385) | > loss_mel_ce: 3.4312589168548584 (nan) | > loss: 0.013790929690003395 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.1365675383866735) | > loader_time: 0.0047 (0.004700647793748054)  --> TIME: 2025-02-24 16:43:23 -- STEP: 31570/36938 -- GLOBAL_STEP: 400950 | > loss_text_ce: 0.034626830369234085 (0.03937399685837125) | > loss_mel_ce: 3.142746925354004 (nan) | > loss: 0.012608625926077366 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.1365678906705051) | > loader_time: 0.0046 (0.004700629183262032)  --> TIME: 2025-02-24 16:43:32 -- STEP: 31620/36938 -- GLOBAL_STEP: 401000 | > loss_text_ce: 0.04373537376523018 (0.039372936296903895) | > loss_mel_ce: 3.7109375 (nan) | > loss: 0.014899495989084244 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.1365688329177286) | > loader_time: 0.0046 (0.004701043826579145)  --> TIME: 2025-02-24 16:43:42 -- STEP: 31670/36938 -- GLOBAL_STEP: 401050 | > loss_text_ce: 0.051650308072566986 (0.03937096027936735) | > loss_mel_ce: 2.5291218757629395 (nan) | > loss: 0.010241160169243813 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1365652460384935) | > loader_time: 0.0046 (0.004701352970134133)  --> TIME: 2025-02-24 16:43:51 -- STEP: 31720/36938 -- GLOBAL_STEP: 401100 | > loss_text_ce: 0.03214230015873909 (0.03937070732994963) | > loss_mel_ce: 2.5402791500091553 (nan) | > loss: 0.01020802278071642 (nan) | > current_lr: 5e-06 | > step_time: 0.1626 (0.13656961236144316) | > loader_time: 0.0054 (0.004701619528581638)  --> TIME: 2025-02-24 16:44:01 -- STEP: 31770/36938 -- GLOBAL_STEP: 401150 | > loss_text_ce: 0.04451204836368561 (0.0393713872332146) | > loss_mel_ce: 2.381326198577881 (nan) | > loss: 0.009626342914998531 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13657273600827097) | > loader_time: 0.0047 (0.004701611415597679)  --> TIME: 2025-02-24 16:44:10 -- STEP: 31820/36938 -- GLOBAL_STEP: 401200 | > loss_text_ce: 0.0399845652282238 (0.03936990251686194) | > loss_mel_ce: 3.2708675861358643 (nan) | > loss: 0.013138302601873875 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.13657716126804542) | > loader_time: 0.0049 (0.00470149459365327)  --> TIME: 2025-02-24 16:44:19 -- STEP: 31870/36938 -- GLOBAL_STEP: 401250 | > loss_text_ce: 0.03151907026767731 (0.039369750686006004) | > loss_mel_ce: 3.4030921459198 (nan) | > loss: 0.013629410415887833 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13656860178901606) | > loader_time: 0.0045 (0.00470131568711194)  --> TIME: 2025-02-24 16:44:29 -- STEP: 31920/36938 -- GLOBAL_STEP: 401300 | > loss_text_ce: 0.04132568836212158 (0.039368394980344584) | > loss_mel_ce: 1.4939671754837036 (nan) | > loss: 0.0060924324207007885 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.1365759471977558) | > loader_time: 0.0052 (0.004701380286001623)  --> TIME: 2025-02-24 16:44:38 -- STEP: 31970/36938 -- GLOBAL_STEP: 401350 | > loss_text_ce: 0.048245448619127274 (0.03936938265717806) | > loss_mel_ce: 2.1714260578155518 (nan) | > loss: 0.008808220736682415 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.1365662173625259) | > loader_time: 0.0046 (0.00470125617032354)  --> TIME: 2025-02-24 16:44:47 -- STEP: 32020/36938 -- GLOBAL_STEP: 401400 | > loss_text_ce: 0.02772011235356331 (0.03936872226159898) | > loss_mel_ce: 3.1049818992614746 (nan) | > loss: 0.012431357987225056 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.1365586009791009) | > loader_time: 0.0052 (0.004701419989069429)  --> TIME: 2025-02-24 16:44:56 -- STEP: 32070/36938 -- GLOBAL_STEP: 401450 | > loss_text_ce: 0.04254843294620514 (0.03936658186287986) | > loss_mel_ce: 3.1874313354492188 (nan) | > loss: 0.01281738094985485 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.13656295290014042) | > loader_time: 0.0047 (0.004701446416335647)  --> TIME: 2025-02-24 16:45:05 -- STEP: 32120/36938 -- GLOBAL_STEP: 401500 | > loss_text_ce: 0.04709184169769287 (0.03936551724129091) | > loss_mel_ce: 3.3201053142547607 (nan) | > loss: 0.013361893594264984 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1365626955017497) | > loader_time: 0.0045 (0.00470133840963523)  --> TIME: 2025-02-24 16:45:15 -- STEP: 32170/36938 -- GLOBAL_STEP: 401550 | > loss_text_ce: 0.035995300859212875 (0.03936442006356185) | > loss_mel_ce: 3.1292786598205566 (nan) | > loss: 0.012560611590743065 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.13656402699206918) | > loader_time: 0.0048 (0.004701219962774912)  --> TIME: 2025-02-24 16:45:24 -- STEP: 32220/36938 -- GLOBAL_STEP: 401600 | > loss_text_ce: 0.032826486974954605 (0.03936287129822748) | > loss_mel_ce: 0.11456604301929474 (nan) | > loss: 0.0005848910077475011 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13656852024819466) | > loader_time: 0.0047 (0.00470121776295478)  --> TIME: 2025-02-24 16:45:33 -- STEP: 32270/36938 -- GLOBAL_STEP: 401650 | > loss_text_ce: 0.04654248058795929 (0.03936420767399198) | > loss_mel_ce: 1.9816601276397705 (nan) | > loss: 0.008048423565924168 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13656502387676528) | > loader_time: 0.0044 (0.004701011669240504)  --> TIME: 2025-02-24 16:45:43 -- STEP: 32320/36938 -- GLOBAL_STEP: 401700 | > loss_text_ce: 0.037067681550979614 (0.03936477722438714) | > loss_mel_ce: 2.8418376445770264 (nan) | > loss: 0.011424227617681026 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1365656189546721) | > loader_time: 0.0045 (0.004700856382894045)  --> TIME: 2025-02-24 16:45:52 -- STEP: 32370/36938 -- GLOBAL_STEP: 401750 | > loss_text_ce: 0.04411115497350693 (0.039364582098269915) | > loss_mel_ce: 3.365548849105835 (nan) | > loss: 0.013530397787690163 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.1365696536489274) | > loader_time: 0.0045 (0.004700771061623581)  --> TIME: 2025-02-24 16:46:01 -- STEP: 32420/36938 -- GLOBAL_STEP: 401800 | > loss_text_ce: 0.03969549387693405 (0.03936300788259711) | > loss_mel_ce: 3.153240442276001 (nan) | > loss: 0.012670380994677544 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.13656580886452468) | > loader_time: 0.0047 (0.004700607462182303)  --> TIME: 2025-02-24 16:46:11 -- STEP: 32470/36938 -- GLOBAL_STEP: 401850 | > loss_text_ce: 0.03790368139743805 (0.03936123269285113) | > loss_mel_ce: 3.32621169090271 (nan) | > loss: 0.013349665328860283 (nan) | > current_lr: 5e-06 | > step_time: 0.152 (0.13656776555692463) | > loader_time: 0.0046 (0.004700462444400584)  --> TIME: 2025-02-24 16:46:20 -- STEP: 32520/36938 -- GLOBAL_STEP: 401900 | > loss_text_ce: 0.04352377727627754 (0.03936057155121321) | > loss_mel_ce: 3.172116756439209 (nan) | > loss: 0.012760479003190994 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13656979179030357) | > loader_time: 0.0044 (0.00470040603317458)  --> TIME: 2025-02-24 16:46:29 -- STEP: 32570/36938 -- GLOBAL_STEP: 401950 | > loss_text_ce: 0.04383517801761627 (0.03936115417973737) | > loss_mel_ce: 3.70654296875 (nan) | > loss: 0.014882453717291355 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13656693502624612) | > loader_time: 0.0046 (0.004700253124723123)  --> TIME: 2025-02-24 16:46:39 -- STEP: 32620/36938 -- GLOBAL_STEP: 402000 | > loss_text_ce: 0.04234135150909424 (0.03936214092422249) | > loss_mel_ce: 0.033935546875 (nan) | > loss: 0.00030268612317740917 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13656513939453385) | > loader_time: 0.0044 (0.004700101006623065)  --> TIME: 2025-02-24 16:46:48 -- STEP: 32670/36938 -- GLOBAL_STEP: 402050 | > loss_text_ce: 0.03554078936576843 (0.03936088828003355) | > loss_mel_ce: 3.3170905113220215 (nan) | > loss: 0.013304092921316624 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.1365640024325876) | > loader_time: 0.0048 (0.004699949244676398)  --> TIME: 2025-02-24 16:46:57 -- STEP: 32720/36938 -- GLOBAL_STEP: 402100 | > loss_text_ce: 0.036063745617866516 (0.03936107346012685) | > loss_mel_ce: 2.9231116771698 (nan) | > loss: 0.011742760427296162 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13656036673081767) | > loader_time: 0.0046 (0.004699784400702107)  --> TIME: 2025-02-24 16:47:06 -- STEP: 32770/36938 -- GLOBAL_STEP: 402150 | > loss_text_ce: 0.03074009157717228 (0.03936011045756236) | > loss_mel_ce: 3.099335193634033 (nan) | > loss: 0.012420933693647385 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13656182678363452) | > loader_time: 0.0045 (0.00469965987864667)  --> TIME: 2025-02-24 16:47:16 -- STEP: 32820/36938 -- GLOBAL_STEP: 402200 | > loss_text_ce: 0.04186425730586052 (0.03935870561183459) | > loss_mel_ce: 3.4652838706970215 (nan) | > loss: 0.013917255215346813 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13656186599109552) | > loader_time: 0.0045 (0.004699562519952193)  --> TIME: 2025-02-24 16:47:25 -- STEP: 32870/36938 -- GLOBAL_STEP: 402250 | > loss_text_ce: 0.030880382284522057 (0.03935848782878832) | > loss_mel_ce: 3.0757853984832764 (nan) | > loss: 0.012328039854764938 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13655615119829576) | > loader_time: 0.0045 (0.004699387599676985)  --> TIME: 2025-02-24 16:47:34 -- STEP: 32920/36938 -- GLOBAL_STEP: 402300 | > loss_text_ce: 0.03443586453795433 (0.03935839911710047) | > loss_mel_ce: 3.0127241611480713 (nan) | > loss: 0.012091904878616333 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.13655755718307622) | > loader_time: 0.0046 (0.0046992845842354514)  --> TIME: 2025-02-24 16:47:43 -- STEP: 32970/36938 -- GLOBAL_STEP: 402350 | > loss_text_ce: 0.02912663109600544 (0.039355789904598544) | > loss_mel_ce: 2.736506223678589 (nan) | > loss: 0.010974734090268612 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13655647872106733) | > loader_time: 0.0047 (0.004699145304928921)  --> TIME: 2025-02-24 16:47:52 -- STEP: 33020/36938 -- GLOBAL_STEP: 402400 | > loss_text_ce: 0.03021363727748394 (0.039357055362983916) | > loss_mel_ce: 1.826545238494873 (nan) | > loss: 0.007368091028183699 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13654982881210934) | > loader_time: 0.0047 (0.004699028310885509)  --> TIME: 2025-02-24 16:48:02 -- STEP: 33070/36938 -- GLOBAL_STEP: 402450 | > loss_text_ce: 0.04460745304822922 (0.03935827474538717) | > loss_mel_ce: 1.9050025939941406 (nan) | > loss: 0.007736547850072384 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.136549615708585) | > loader_time: 0.0049 (0.004698934041736421)  --> TIME: 2025-02-24 16:48:11 -- STEP: 33120/36938 -- GLOBAL_STEP: 402500 | > loss_text_ce: 0.040316276252269745 (0.039357961670907035) | > loss_mel_ce: 1.3799612522125244 (nan) | > loss: 0.005636021960526705 (nan) | > current_lr: 5e-06 | > step_time: 0.1642 (0.1365454696226805) | > loader_time: 0.0056 (0.004698961303717849)  --> TIME: 2025-02-24 16:48:20 -- STEP: 33170/36938 -- GLOBAL_STEP: 402550 | > loss_text_ce: 0.04395253583788872 (0.03935592910794272) | > loss_mel_ce: 3.0162353515625 (nan) | > loss: 0.012143603526055813 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13654114369135587) | > loader_time: 0.0046 (0.004699164706251153)  --> TIME: 2025-02-24 16:48:29 -- STEP: 33220/36938 -- GLOBAL_STEP: 402600 | > loss_text_ce: 0.02583984285593033 (0.03935445845732641) | > loss_mel_ce: 0.02421264722943306 (nan) | > loss: 0.00019862099725287408 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13653673279507855) | > loader_time: 0.0046 (0.004699368587392844)  --> TIME: 2025-02-24 16:48:39 -- STEP: 33270/36938 -- GLOBAL_STEP: 402650 | > loss_text_ce: 0.03868081048130989 (0.03935478888211293) | > loss_mel_ce: 3.322295665740967 (nan) | > loss: 0.013337208889424801 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13654343425457147) | > loader_time: 0.0045 (0.004699455018440523)  --> TIME: 2025-02-24 16:48:48 -- STEP: 33320/36938 -- GLOBAL_STEP: 402700 | > loss_text_ce: 0.05451303720474243 (0.03935592848084184) | > loss_mel_ce: 2.9354684352874756 (nan) | > loss: 0.01186500582844019 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.136543730742075) | > loader_time: 0.0045 (0.004699722909125962)  --> TIME: 2025-02-24 16:48:58 -- STEP: 33370/36938 -- GLOBAL_STEP: 402750 | > loss_text_ce: 0.03380843997001648 (0.03935358344722638) | > loss_mel_ce: 2.806499719619751 (nan) | > loss: 0.011271065101027489 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13654918241257874) | > loader_time: 0.0049 (0.00470006984616955)  --> TIME: 2025-02-24 16:49:07 -- STEP: 33420/36938 -- GLOBAL_STEP: 402800 | > loss_text_ce: 0.03961467742919922 (0.03935047314250914) | > loss_mel_ce: 3.2131564617156982 (nan) | > loss: 0.012907822616398335 (nan) | > current_lr: 5e-06 | > step_time: 0.1837 (0.13655902929180408) | > loader_time: 0.0057 (0.004700364030670222)  --> TIME: 2025-02-24 16:49:16 -- STEP: 33470/36938 -- GLOBAL_STEP: 402850 | > loss_text_ce: 0.02977403998374939 (0.03935209802525223) | > loss_mel_ce: 3.5895743370056152 (nan) | > loss: 0.014362494461238384 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1365582872156525) | > loader_time: 0.0046 (0.004700320772103207)  --> TIME: 2025-02-24 16:49:26 -- STEP: 33520/36938 -- GLOBAL_STEP: 402900 | > loss_text_ce: 0.043283868581056595 (0.03934900561163651) | > loss_mel_ce: 2.8974859714508057 (nan) | > loss: 0.011669722385704517 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13656440067120493) | > loader_time: 0.0047 (0.004700288261889273)  --> TIME: 2025-02-24 16:49:35 -- STEP: 33570/36938 -- GLOBAL_STEP: 402950 | > loss_text_ce: 0.05072823539376259 (0.0393492904660951) | > loss_mel_ce: 0.8619317412376404 (nan) | > loss: 0.0036216669250279665 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.1365589769390101) | > loader_time: 0.0046 (0.004700317757805082)  --> TIME: 2025-02-24 16:49:44 -- STEP: 33620/36938 -- GLOBAL_STEP: 403000 | > loss_text_ce: 0.03888176381587982 (0.0393481790297815) | > loss_mel_ce: 3.901226043701172 (nan) | > loss: 0.015635348856449127 (nan) | > current_lr: 5e-06 | > step_time: 0.1844 (0.13655493767895427) | > loader_time: 0.0048 (0.004700315026992988)  --> TIME: 2025-02-24 16:49:54 -- STEP: 33670/36938 -- GLOBAL_STEP: 403050 | > loss_text_ce: 0.02500390075147152 (0.03934589394085798) | > loss_mel_ce: 1.727836012840271 (nan) | > loss: 0.006955714430660009 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13655456142040265) | > loader_time: 0.0044 (0.004700248567860452)  --> TIME: 2025-02-24 16:50:03 -- STEP: 33720/36938 -- GLOBAL_STEP: 403100 | > loss_text_ce: 0.04464371129870415 (0.039345576990619836) | > loss_mel_ce: 1.6223396062850952 (nan) | > loss: 0.006615013815462589 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1365556329170574) | > loader_time: 0.0046 (0.0047002766126698296)  --> TIME: 2025-02-24 16:50:12 -- STEP: 33770/36938 -- GLOBAL_STEP: 403150 | > loss_text_ce: 0.04498791694641113 (0.03934352875336639) | > loss_mel_ce: 2.8244004249572754 (nan) | > loss: 0.011386462487280369 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13654950549696349) | > loader_time: 0.0045 (0.004700300825535281)  --> TIME: 2025-02-24 16:50:21 -- STEP: 33820/36938 -- GLOBAL_STEP: 403200 | > loss_text_ce: 0.03636208549141884 (0.03934303737936122) | > loss_mel_ce: 0.014434814453125 (nan) | > loss: 0.0002015750069404021 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1365388900552673) | > loader_time: 0.0045 (0.00470042874445089)  --> TIME: 2025-02-24 16:50:30 -- STEP: 33870/36938 -- GLOBAL_STEP: 403250 | > loss_text_ce: 0.03304971754550934 (0.039342415051307436) | > loss_mel_ce: 3.2590506076812744 (nan) | > loss: 0.013063891790807247 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1365364324388366) | > loader_time: 0.0052 (0.00470045469558306)  --> TIME: 2025-02-24 16:50:39 -- STEP: 33920/36938 -- GLOBAL_STEP: 403300 | > loss_text_ce: 0.03973010927438736 (0.039345347352258725) | > loss_mel_ce: 0.2080485075712204 (nan) | > loss: 0.0009832485811784863 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13652386163484315) | > loader_time: 0.0045 (0.004700295106982285)  --> TIME: 2025-02-24 16:50:48 -- STEP: 33970/36938 -- GLOBAL_STEP: 403350 | > loss_text_ce: 0.05299985408782959 (0.03934503813897353) | > loss_mel_ce: 2.386302947998047 (nan) | > loss: 0.009679773822426796 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13652941012052575) | > loader_time: 0.0045 (0.0047002257127287395)  --> TIME: 2025-02-24 16:50:58 -- STEP: 34020/36938 -- GLOBAL_STEP: 403400 | > loss_text_ce: 0.04811614006757736 (0.03934380737963339) | > loss_mel_ce: 3.5361087322235107 (nan) | > loss: 0.014223115518689156 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.1365319176606043) | > loader_time: 0.0047 (0.004700274000722896)  --> TIME: 2025-02-24 16:51:07 -- STEP: 34070/36938 -- GLOBAL_STEP: 403450 | > loss_text_ce: 0.038843654096126556 (0.03934138176157959) | > loss_mel_ce: 2.50093150138855 (nan) | > loss: 0.010078473016619682 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13653385119246358) | > loader_time: 0.0053 (0.004700273658510595)  --> TIME: 2025-02-24 16:51:17 -- STEP: 34120/36938 -- GLOBAL_STEP: 403500 | > loss_text_ce: 0.04225960001349449 (0.03934139032372676) | > loss_mel_ce: 2.9825048446655273 (nan) | > loss: 0.012003034353256226 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.136537264268255) | > loader_time: 0.0045 (0.004700207640389625)  --> TIME: 2025-02-24 16:51:26 -- STEP: 34170/36938 -- GLOBAL_STEP: 403550 | > loss_text_ce: 0.03459227830171585 (0.03933980340690804) | > loss_mel_ce: 3.2266721725463867 (nan) | > loss: 0.012941527180373669 (nan) | > current_lr: 5e-06 | > step_time: 0.1778 (0.1365317762613219) | > loader_time: 0.0048 (0.004700012286573883)  --> TIME: 2025-02-24 16:51:35 -- STEP: 34220/36938 -- GLOBAL_STEP: 403600 | > loss_text_ce: 0.03874167799949646 (0.039336437374883404) | > loss_mel_ce: 3.408081531524658 (nan) | > loss: 0.0136778699234128 (nan) | > current_lr: 5e-06 | > step_time: 0.194 (0.13652931979913366) | > loader_time: 0.0048 (0.004699821203232924)  --> TIME: 2025-02-24 16:51:44 -- STEP: 34270/36938 -- GLOBAL_STEP: 403650 | > loss_text_ce: 0.033380839973688126 (0.03933639793890736) | > loss_mel_ce: 3.5787501335144043 (nan) | > loss: 0.014333853498101234 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13653059613784604) | > loader_time: 0.0046 (0.004699787816675753)  --> TIME: 2025-02-24 16:51:54 -- STEP: 34320/36938 -- GLOBAL_STEP: 403700 | > loss_text_ce: 0.031233269721269608 (0.039335357639064636) | > loss_mel_ce: 2.880159854888916 (nan) | > loss: 0.011553147807717323 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13653118266509046) | > loader_time: 0.0045 (0.004699833875213748)  --> TIME: 2025-02-24 16:52:03 -- STEP: 34370/36938 -- GLOBAL_STEP: 403750 | > loss_text_ce: 0.03541235253214836 (0.03933382963827788) | > loss_mel_ce: 3.2973427772521973 (nan) | > loss: 0.013225219212472439 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.13653234701569966) | > loader_time: 0.0047 (0.004699829154004879)  --> TIME: 2025-02-24 16:52:12 -- STEP: 34420/36938 -- GLOBAL_STEP: 403800 | > loss_text_ce: 0.03424476087093353 (0.039333206625170265) | > loss_mel_ce: 3.578017473220825 (nan) | > loss: 0.014334375038743019 (nan) | > current_lr: 5e-06 | > step_time: 0.1687 (0.136531028970599) | > loader_time: 0.0055 (0.0046998355223784845)  --> TIME: 2025-02-24 16:52:21 -- STEP: 34470/36938 -- GLOBAL_STEP: 403850 | > loss_text_ce: 0.02394998073577881 (0.039333888132088234) | > loss_mel_ce: 3.0325520038604736 (nan) | > loss: 0.012128976173698902 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13652564149889146) | > loader_time: 0.0047 (0.004700089303313532)  --> TIME: 2025-02-24 16:52:30 -- STEP: 34520/36938 -- GLOBAL_STEP: 403900 | > loss_text_ce: 0.03460131958127022 (0.039331219294043526) | > loss_mel_ce: 3.3381595611572266 (nan) | > loss: 0.013383972458541393 (nan) | > current_lr: 5e-06 | > step_time: 0.1619 (0.13652427936208) | > loader_time: 0.0053 (0.004700334765543672)  --> TIME: 2025-02-24 16:52:40 -- STEP: 34570/36938 -- GLOBAL_STEP: 403950 | > loss_text_ce: 0.04249715059995651 (0.03932922096984401) | > loss_mel_ce: 2.8130605220794678 (nan) | > loss: 0.011331578716635704 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.136528452069627) | > loader_time: 0.0046 (0.004700473350081311)  --> TIME: 2025-02-24 16:52:49 -- STEP: 34620/36938 -- GLOBAL_STEP: 404000 | > loss_text_ce: 0.031667862087488174 (0.03932796053581379) | > loss_mel_ce: 2.7902956008911133 (nan) | > loss: 0.01119826827198267 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.1365299790460612) | > loader_time: 0.0045 (0.004700672867530902)  --> TIME: 2025-02-24 16:52:59 -- STEP: 34670/36938 -- GLOBAL_STEP: 404050 | > loss_text_ce: 0.033925868570804596 (0.03932717729094983) | > loss_mel_ce: 2.9159440994262695 (nan) | > loss: 0.011705833487212658 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13653362398364827) | > loader_time: 0.0046 (0.004700640329614582)  --> TIME: 2025-02-24 16:53:08 -- STEP: 34720/36938 -- GLOBAL_STEP: 404100 | > loss_text_ce: 0.04066665098071098 (0.03932833237064371) | > loss_mel_ce: 3.6142168045043945 (nan) | > loss: 0.014503506012260914 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13653555334575648) | > loader_time: 0.0054 (0.004701067603403528)  --> TIME: 2025-02-24 16:53:17 -- STEP: 34770/36938 -- GLOBAL_STEP: 404150 | > loss_text_ce: 0.03653569519519806 (0.03932678115524405) | > loss_mel_ce: 2.9411983489990234 (nan) | > loss: 0.011816405691206455 (nan) | > current_lr: 5e-06 | > step_time: 0.15 (0.1365314157042717) | > loader_time: 0.0046 (0.004701189848071794)  --> TIME: 2025-02-24 16:53:27 -- STEP: 34820/36938 -- GLOBAL_STEP: 404200 | > loss_text_ce: 0.035144396126270294 (0.03932759316409679) | > loss_mel_ce: 3.4647653102874756 (nan) | > loss: 0.013888531364500523 (nan) | > current_lr: 5e-06 | > step_time: 0.1815 (0.13653626572736505) | > loader_time: 0.0048 (0.004701390312979242)  --> TIME: 2025-02-24 16:53:36 -- STEP: 34870/36938 -- GLOBAL_STEP: 404250 | > loss_text_ce: 0.04131709039211273 (0.039325932473139626) | > loss_mel_ce: 2.4643282890319824 (nan) | > loss: 0.009943037293851376 (nan) | > current_lr: 5e-06 | > step_time: 0.1578 (0.13653052823400574) | > loader_time: 0.0048 (0.004701416465796476)  --> TIME: 2025-02-24 16:53:45 -- STEP: 34920/36938 -- GLOBAL_STEP: 404300 | > loss_text_ce: 0.03956060856580734 (0.039324677452196324) | > loss_mel_ce: 3.46875 (nan) | > loss: 0.013921868056058884 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13652718414692194) | > loader_time: 0.0054 (0.00470146197497094)  --> TIME: 2025-02-24 16:53:54 -- STEP: 34970/36938 -- GLOBAL_STEP: 404350 | > loss_text_ce: 0.03343435376882553 (0.03932633684667905) | > loss_mel_ce: 2.3098177909851074 (nan) | > loss: 0.009298620745539665 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13652644766239436) | > loader_time: 0.0045 (0.004701446361939911)  --> TIME: 2025-02-24 16:54:04 -- STEP: 35020/36938 -- GLOBAL_STEP: 404400 | > loss_text_ce: 0.04394596815109253 (0.03932632472824031) | > loss_mel_ce: 2.597923755645752 (nan) | > loss: 0.010483610443770885 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1365267334360174) | > loader_time: 0.0046 (0.004701664398766329)  --> TIME: 2025-02-24 16:54:13 -- STEP: 35070/36938 -- GLOBAL_STEP: 404450 | > loss_text_ce: 0.03541039302945137 (0.03932624822004442) | > loss_mel_ce: 2.837721109390259 (nan) | > loss: 0.011401316151022911 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13653058099515505) | > loader_time: 0.0046 (0.004701560924357081)  --> TIME: 2025-02-24 16:54:22 -- STEP: 35120/36938 -- GLOBAL_STEP: 404500 | > loss_text_ce: 0.03402134031057358 (0.03932879213235752) | > loss_mel_ce: 2.8873536586761475 (nan) | > loss: 0.011592758819460869 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13652209300115006) | > loader_time: 0.0046 (0.004701413468782345)  --> TIME: 2025-02-24 16:54:31 -- STEP: 35170/36938 -- GLOBAL_STEP: 404550 | > loss_text_ce: 0.045967746526002884 (0.03932986185783471) | > loss_mel_ce: 3.315420389175415 (nan) | > loss: 0.013338842429220676 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13652392321091145) | > loader_time: 0.0045 (0.004701395349337158)  --> TIME: 2025-02-24 16:54:41 -- STEP: 35220/36938 -- GLOBAL_STEP: 404600 | > loss_text_ce: 0.029980184510350227 (0.03932736945874796) | > loss_mel_ce: 2.29630970954895 (nan) | > loss: 0.009231309406459332 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13652360616659007) | > loader_time: 0.0051 (0.004701390373375121)  --> TIME: 2025-02-24 16:54:50 -- STEP: 35270/36938 -- GLOBAL_STEP: 404650 | > loss_text_ce: 0.038355689495801926 (0.03932814383736698) | > loss_mel_ce: 3.3470723628997803 (nan) | > loss: 0.013434238731861115 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13652740391449203) | > loader_time: 0.0046 (0.00470137430514971)  --> TIME: 2025-02-24 16:54:59 -- STEP: 35320/36938 -- GLOBAL_STEP: 404700 | > loss_text_ce: 0.04783353582024574 (0.03932739841782853) | > loss_mel_ce: 2.397653579711914 (nan) | > loss: 0.009704314172267914 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.13652092980501432) | > loader_time: 0.0046 (0.004701254085668224)  --> TIME: 2025-02-24 16:55:08 -- STEP: 35370/36938 -- GLOBAL_STEP: 404750 | > loss_text_ce: 0.03646043688058853 (0.03932678285174651) | > loss_mel_ce: 2.9757792949676514 (nan) | > loss: 0.011953332461416721 (nan) | > current_lr: 5e-06 | > step_time: 0.1846 (0.13651876730672133) | > loader_time: 0.005 (0.004701122349185535)  --> TIME: 2025-02-24 16:55:18 -- STEP: 35420/36938 -- GLOBAL_STEP: 404800 | > loss_text_ce: 0.057290200144052505 (0.039326323178280595) | > loss_mel_ce: 1.8568488359451294 (nan) | > loss: 0.007595790084451437 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13651784147535007) | > loader_time: 0.0053 (0.004701197457003911)  --> TIME: 2025-02-24 16:55:27 -- STEP: 35470/36938 -- GLOBAL_STEP: 404850 | > loss_text_ce: 0.041197966784238815 (0.03932474511508792) | > loss_mel_ce: 2.0221900939941406 (nan) | > loss: 0.008188048377633095 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13651593793503855) | > loader_time: 0.0045 (0.004701155462767807)  --> TIME: 2025-02-24 16:55:36 -- STEP: 35520/36938 -- GLOBAL_STEP: 404900 | > loss_text_ce: 0.031998161226511 (0.03932397976160551) | > loss_mel_ce: 3.5881950855255127 (nan) | > loss: 0.014365847222507 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13651642968525718) | > loader_time: 0.0047 (0.0047014273703098386)  --> TIME: 2025-02-24 16:55:46 -- STEP: 35570/36938 -- GLOBAL_STEP: 404950 | > loss_text_ce: 0.03420830890536308 (0.039323128268435696) | > loss_mel_ce: 3.4707419872283936 (nan) | > loss: 0.01390853337943554 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.13651733791221504) | > loader_time: 0.0046 (0.0047014644047434495)  --> TIME: 2025-02-24 16:55:55 -- STEP: 35620/36938 -- GLOBAL_STEP: 405000 | > loss_text_ce: 0.03835861384868622 (0.039322662372732846) | > loss_mel_ce: 3.0027594566345215 (nan) | > loss: 0.012067929841578007 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.13652123174661954) | > loader_time: 0.0056 (0.004701576568918383)  --> TIME: 2025-02-24 16:56:04 -- STEP: 35670/36938 -- GLOBAL_STEP: 405050 | > loss_text_ce: 0.03257715702056885 (0.03932320485293603) | > loss_mel_ce: 3.449284076690674 (nan) | > loss: 0.013816909864544868 (nan) | > current_lr: 5e-06 | > step_time: 0.1434 (0.1365214223249434) | > loader_time: 0.0046 (0.004701745827591725)  --> TIME: 2025-02-24 16:56:13 -- STEP: 35720/36938 -- GLOBAL_STEP: 405100 | > loss_text_ce: 0.04684920608997345 (0.03932245960101899) | > loss_mel_ce: 3.0536012649536133 (nan) | > loss: 0.012303375639021397 (nan) | > current_lr: 5e-06 | > step_time: 0.1755 (0.1365134783247679) | > loader_time: 0.0049 (0.00470171562100846)  --> TIME: 2025-02-24 16:56:23 -- STEP: 35770/36938 -- GLOBAL_STEP: 405150 | > loss_text_ce: 0.042718786746263504 (0.03932359516539908) | > loss_mel_ce: 3.4320573806762695 (nan) | > loss: 0.013788795098662376 (nan) | > current_lr: 5e-06 | > step_time: 0.172 (0.13651418114694913) | > loader_time: 0.0048 (0.0047017738277231646)  --> TIME: 2025-02-24 16:56:32 -- STEP: 35820/36938 -- GLOBAL_STEP: 405200 | > loss_text_ce: 0.039706550538539886 (0.03932250998610026) | > loss_mel_ce: 2.6721365451812744 (nan) | > loss: 0.010761282406747341 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13651889937340594) | > loader_time: 0.0047 (0.004701818014109757)  --> TIME: 2025-02-24 16:56:41 -- STEP: 35870/36938 -- GLOBAL_STEP: 405250 | > loss_text_ce: 0.03683873638510704 (0.039322538888416705) | > loss_mel_ce: 3.423015832901001 (nan) | > loss: 0.013729582540690899 (nan) | > current_lr: 5e-06 | > step_time: 0.176 (0.13652024370134402) | > loader_time: 0.0049 (0.004701863825404285)  --> TIME: 2025-02-24 16:56:51 -- STEP: 35920/36938 -- GLOBAL_STEP: 405300 | > loss_text_ce: 0.032553091645240784 (0.03932314459081492) | > loss_mel_ce: 2.9154207706451416 (nan) | > loss: 0.011698310263454914 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.1365183830991349) | > loader_time: 0.0047 (0.004701804464272242)  --> TIME: 2025-02-24 16:57:00 -- STEP: 35970/36938 -- GLOBAL_STEP: 405350 | > loss_text_ce: 0.04366298392415047 (0.03932296532507346) | > loss_mel_ce: 3.423356294631958 (nan) | > loss: 0.01375801395624876 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13652333248181858) | > loader_time: 0.0047 (0.00470196503481999)  --> TIME: 2025-02-24 16:57:09 -- STEP: 36020/36938 -- GLOBAL_STEP: 405400 | > loss_text_ce: 0.04016768932342529 (0.039320114123113106) | > loss_mel_ce: 3.66430926322937 (nan) | > loss: 0.01470030564814806 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13651891429843888) | > loader_time: 0.0046 (0.0047020835389301975)  --> TIME: 2025-02-24 16:57:19 -- STEP: 36070/36938 -- GLOBAL_STEP: 405450 | > loss_text_ce: 0.03920358046889305 (0.03931661919944467) | > loss_mel_ce: 3.065446376800537 (nan) | > loss: 0.012320040725171566 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13652747650373823) | > loader_time: 0.0053 (0.004702344309566491)  --> TIME: 2025-02-24 16:57:28 -- STEP: 36120/36938 -- GLOBAL_STEP: 405500 | > loss_text_ce: 0.030628660693764687 (0.03931516185596242) | > loss_mel_ce: 2.5586373805999756 (nan) | > loss: 0.010274865664541721 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13652932653791433) | > loader_time: 0.0052 (0.004702444739251969)  --> TIME: 2025-02-24 16:57:37 -- STEP: 36170/36938 -- GLOBAL_STEP: 405550 | > loss_text_ce: 0.04302263259887695 (0.039315077373132175) | > loss_mel_ce: 3.36161208152771 (nan) | > loss: 0.013510456308722496 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13652385880283288) | > loader_time: 0.0045 (0.004702365678550462)  --> TIME: 2025-02-24 16:57:47 -- STEP: 36220/36938 -- GLOBAL_STEP: 405600 | > loss_text_ce: 0.04275146871805191 (0.039315506942135044) | > loss_mel_ce: 3.5791146755218506 (nan) | > loss: 0.014372485689818859 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13652700256208591) | > loader_time: 0.0045 (0.004702386469027404)  --> TIME: 2025-02-24 16:57:56 -- STEP: 36270/36938 -- GLOBAL_STEP: 405650 | > loss_text_ce: 0.028510885313153267 (0.0393132295603045) | > loss_mel_ce: 1.2071901559829712 (nan) | > loss: 0.004903575871139765 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13652757831864773) | > loader_time: 0.0047 (0.004702447162109659)  --> TIME: 2025-02-24 16:58:05 -- STEP: 36320/36938 -- GLOBAL_STEP: 405700 | > loss_text_ce: 0.04208697751164436 (0.03931187151296315) | > loss_mel_ce: 3.3257241249084473 (nan) | > loss: 0.013364330865442753 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13652520226355652) | > loader_time: 0.0053 (0.004702498031870385)  --> TIME: 2025-02-24 16:58:15 -- STEP: 36370/36938 -- GLOBAL_STEP: 405750 | > loss_text_ce: 0.029989033937454224 (0.03930968005068801) | > loss_mel_ce: 1.91845703125 (nan) | > loss: 0.007731928955763578 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13652425628459638) | > loader_time: 0.0045 (0.004702546237948705)  --> TIME: 2025-02-24 16:58:24 -- STEP: 36420/36938 -- GLOBAL_STEP: 405800 | > loss_text_ce: 0.044803958386182785 (0.03930935694655536) | > loss_mel_ce: 3.0990958213806152 (nan) | > loss: 0.01247579324990511 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13652321856994637) | > loader_time: 0.0047 (0.004702639691061139)  --> TIME: 2025-02-24 16:58:34 -- STEP: 36470/36938 -- GLOBAL_STEP: 405850 | > loss_text_ce: 0.039962414652109146 (0.039308941282724115) | > loss_mel_ce: 2.8803634643554688 (nan) | > loss: 0.011588595807552338 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13652900587803085) | > loader_time: 0.0047 (0.004702691545478607)  --> TIME: 2025-02-24 16:58:43 -- STEP: 36520/36938 -- GLOBAL_STEP: 405900 | > loss_text_ce: 0.03690357506275177 (0.03930924906036304) | > loss_mel_ce: 3.4450128078460693 (nan) | > loss: 0.013817129656672478 (nan) | > current_lr: 5e-06 | > step_time: 0.1445 (0.1365360088899385) | > loader_time: 0.0053 (0.004702843443326335)  --> TIME: 2025-02-24 16:58:53 -- STEP: 36570/36938 -- GLOBAL_STEP: 405950 | > loss_text_ce: 0.03721190616488457 (0.03930867219602009) | > loss_mel_ce: 3.2128286361694336 (nan) | > loss: 0.012896986678242683 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.1365407947115729) | > loader_time: 0.0046 (0.004702849319015725)  --> TIME: 2025-02-24 16:59:02 -- STEP: 36620/36938 -- GLOBAL_STEP: 406000 | > loss_text_ce: 0.04608350619673729 (0.03930851677702231) | > loss_mel_ce: 3.2177329063415527 (nan) | > loss: 0.012951652519404888 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13654279693236807) | > loader_time: 0.0046 (0.004702803048196871)  --> TIME: 2025-02-24 16:59:12 -- STEP: 36670/36938 -- GLOBAL_STEP: 406050 | > loss_text_ce: 0.03321167081594467 (0.039308582300552355) | > loss_mel_ce: 0.0648193359375 (nan) | > loss: 0.0003890119551215321 (nan) | > current_lr: 5e-06 | > step_time: 0.1076 (0.13654528967132) | > loader_time: 0.0045 (0.004702683479479688)  --> TIME: 2025-02-24 16:59:21 -- STEP: 36720/36938 -- GLOBAL_STEP: 406100 | > loss_text_ce: 0.033569153398275375 (0.039309889729964555) | > loss_mel_ce: 3.3495290279388428 (nan) | > loss: 0.013424993492662907 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13654639918949593) | > loader_time: 0.0048 (0.004702595499605912)  --> TIME: 2025-02-24 16:59:30 -- STEP: 36770/36938 -- GLOBAL_STEP: 406150 | > loss_text_ce: 0.03162549436092377 (0.03930975697526328) | > loss_mel_ce: 2.9737560749053955 (nan) | > loss: 0.011926118284463882 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.136541359139941) | > loader_time: 0.0045 (0.004702438457458927)  --> TIME: 2025-02-24 16:59:39 -- STEP: 36820/36938 -- GLOBAL_STEP: 406200 | > loss_text_ce: 0.04388796538114548 (0.03930921887210981) | > loss_mel_ce: 2.718083143234253 (nan) | > loss: 0.010960202664136887 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13653659150368594) | > loader_time: 0.0046 (0.004702285798200765)  --> TIME: 2025-02-24 16:59:48 -- STEP: 36870/36938 -- GLOBAL_STEP: 406250 | > loss_text_ce: 0.043020062148571014 (0.03930893748349968) | > loss_mel_ce: 3.080859899520874 (nan) | > loss: 0.01239634957164526 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13652863604672566) | > loader_time: 0.0046 (0.0047022073353601085)  --> TIME: 2025-02-24 16:59:57 -- STEP: 36920/36938 -- GLOBAL_STEP: 406300 | > loss_text_ce: 0.03794913738965988 (0.03930917668352067) | > loss_mel_ce: 3.4121506214141846 (nan) | > loss: 0.013690873049199581 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.136522761087376) | > loader_time: 0.0046 (0.0047022031097474085)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.02056877093740028 (+3.092595846345553e-05) | > avg_loss_text_ce: 0.03815364756501547 (-0.0006934792024664432) | > avg_loss_mel_ce: 2.7313802519647203 (-0.032983676631851644) | > avg_loss: 2.769533896800314 (-0.03367715778917457) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_406318.pth  > EPOCH: 12/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-24 17:00:25)   --> TIME: 2025-02-24 17:00:31 -- STEP: 32/36938 -- GLOBAL_STEP: 406350 | > loss_text_ce: 0.04817384108901024 (0.03943884471664205) | > loss_mel_ce: 1.3655569553375244 (2.465608275961131) | > loss: 0.005610042717307806 (0.009940663877387124) | > current_lr: 5e-06 | > step_time: 0.1089 (0.12921485304832456) | > loader_time: 0.0045 (0.004672609269618988)  --> TIME: 2025-02-24 17:00:40 -- STEP: 82/36938 -- GLOBAL_STEP: 406400 | > loss_text_ce: 0.034181199967861176 (0.03995395722094831) | > loss_mel_ce: 3.980665922164917 (2.7650038343740677) | > loss: 0.015931934118270874 (0.01113078558977932) | > current_lr: 5e-06 | > step_time: 0.1296 (0.1329034043521416) | > loader_time: 0.0047 (0.004693316250312621)  --> TIME: 2025-02-24 17:00:50 -- STEP: 132/36938 -- GLOBAL_STEP: 406450 | > loss_text_ce: 0.055934276431798935 (0.04019689063231149) | > loss_mel_ce: 1.4114990234375 (2.7199749343774537) | > loss: 0.005823148414492607 (0.010953063471375075) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13352429144310227) | > loader_time: 0.0046 (0.004673919894478541)  --> TIME: 2025-02-24 17:00:59 -- STEP: 182/36938 -- GLOBAL_STEP: 406500 | > loss_text_ce: 0.04167841002345085 (0.039699726881316066) | > loss_mel_ce: 2.64919376373291 (2.708622723613139) | > loss: 0.01067806500941515 (0.010906042106248761) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13401401828933548) | > loader_time: 0.0045 (0.004658847064762326)  --> TIME: 2025-02-24 17:01:08 -- STEP: 232/36938 -- GLOBAL_STEP: 406550 | > loss_text_ce: 0.03434492275118828 (0.03941648306549878) | > loss_mel_ce: 3.2094566822052 (nan) | > loss: 0.01287222933024168 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.13367460822236954) | > loader_time: 0.0048 (0.004646513996453123)  --> TIME: 2025-02-24 17:01:17 -- STEP: 282/36938 -- GLOBAL_STEP: 406600 | > loss_text_ce: 0.0379447266459465 (0.03944069109460774) | > loss_mel_ce: 2.6471798419952393 (nan) | > loss: 0.010655256919562817 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13410368540608297) | > loader_time: 0.0046 (0.0046571315603053335)  --> TIME: 2025-02-24 17:01:27 -- STEP: 332/36938 -- GLOBAL_STEP: 406650 | > loss_text_ce: 0.03898731246590614 (0.03930939046149873) | > loss_mel_ce: 2.734692335128784 (nan) | > loss: 0.011006666347384453 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13456061231084623) | > loader_time: 0.0045 (0.004652069275637711)  --> TIME: 2025-02-24 17:01:36 -- STEP: 382/36938 -- GLOBAL_STEP: 406700 | > loss_text_ce: 0.04412281885743141 (0.039291242239948065) | > loss_mel_ce: 3.6821248531341553 (nan) | > loss: 0.01478669885545969 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.13507009551163124) | > loader_time: 0.0048 (0.0046508362155934305)  --> TIME: 2025-02-24 17:01:45 -- STEP: 432/36938 -- GLOBAL_STEP: 406750 | > loss_text_ce: 0.03843213990330696 (0.03907816606806595) | > loss_mel_ce: 2.6403985023498535 (nan) | > loss: 0.010630280710756779 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.13483747674359206) | > loader_time: 0.0049 (0.0046531878135822454)  --> TIME: 2025-02-24 17:01:55 -- STEP: 482/36938 -- GLOBAL_STEP: 406800 | > loss_text_ce: 0.04377951845526695 (0.03894136742692583) | > loss_mel_ce: 0.0561116524040699 (nan) | > loss: 0.00039639355964027345 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13512663227888552) | > loader_time: 0.0046 (0.0046750415904888)  --> TIME: 2025-02-24 17:02:04 -- STEP: 532/36938 -- GLOBAL_STEP: 406850 | > loss_text_ce: 0.03754011541604996 (0.03872499018522417) | > loss_mel_ce: 2.469240665435791 (nan) | > loss: 0.00994754396378994 (nan) | > current_lr: 5e-06 | > step_time: 0.151 (0.13503200084643252) | > loader_time: 0.0046 (0.004673299932838384)  --> TIME: 2025-02-24 17:02:13 -- STEP: 582/36938 -- GLOBAL_STEP: 406900 | > loss_text_ce: 0.0445663146674633 (0.03869962394781594) | > loss_mel_ce: 2.280426263809204 (nan) | > loss: 0.009226161986589432 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1351316401229282) | > loader_time: 0.0045 (0.004671752657677303)  --> TIME: 2025-02-24 17:02:22 -- STEP: 632/36938 -- GLOBAL_STEP: 406950 | > loss_text_ce: 0.030273351818323135 (0.038722447253751936) | > loss_mel_ce: 3.2168495655059814 (nan) | > loss: 0.012885409407317638 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1353234056430527) | > loader_time: 0.0045 (0.004667840049236636)  --> TIME: 2025-02-24 17:02:32 -- STEP: 682/36938 -- GLOBAL_STEP: 407000 | > loss_text_ce: 0.037238288670778275 (0.03876265195332189) | > loss_mel_ce: 0.02178955078125 (nan) | > loss: 0.00023423746461048722 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1352855037034781) | > loader_time: 0.0046 (0.004667104514113621)  --> TIME: 2025-02-24 17:02:41 -- STEP: 732/36938 -- GLOBAL_STEP: 407050 | > loss_text_ce: 0.030282652005553246 (0.038722421094785416) | > loss_mel_ce: 3.3743667602539062 (nan) | > loss: 0.013510514982044697 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.1350863432623649) | > loader_time: 0.0046 (0.004666336898595259)  --> TIME: 2025-02-24 17:02:50 -- STEP: 782/36938 -- GLOBAL_STEP: 407100 | > loss_text_ce: 0.030901417136192322 (0.03868674136379073) | > loss_mel_ce: 2.5186591148376465 (nan) | > loss: 0.010117304511368275 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13525789747457673) | > loader_time: 0.0045 (0.004667205578835726)  --> TIME: 2025-02-24 17:03:00 -- STEP: 832/36938 -- GLOBAL_STEP: 407150 | > loss_text_ce: 0.03698631748557091 (0.03863234332507444) | > loss_mel_ce: 3.448073625564575 (nan) | > loss: 0.013829603791236877 (nan) | > current_lr: 5e-06 | > step_time: 0.171 (0.1355036138915099) | > loader_time: 0.0048 (0.004668915214446876)  --> TIME: 2025-02-24 17:03:09 -- STEP: 882/36938 -- GLOBAL_STEP: 407200 | > loss_text_ce: 0.026373278349637985 (0.03872376362858618) | > loss_mel_ce: 3.262152671813965 (nan) | > loss: 0.013049707747995853 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13548948570173616) | > loader_time: 0.0046 (0.0046695292401476)  --> TIME: 2025-02-24 17:03:19 -- STEP: 932/36938 -- GLOBAL_STEP: 407250 | > loss_text_ce: 0.038132745772600174 (0.0387466759111157) | > loss_mel_ce: 0.874965488910675 (nan) | > loss: 0.0036234059371054173 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.13575072554559667) | > loader_time: 0.0046 (0.004672771871345752)  --> TIME: 2025-02-24 17:03:28 -- STEP: 982/36938 -- GLOBAL_STEP: 407300 | > loss_text_ce: 0.033958591520786285 (0.03880545812091829) | > loss_mel_ce: 3.046459913253784 (nan) | > loss: 0.012223883531987667 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13604417157998638) | > loader_time: 0.0047 (0.004675670456740621)  --> TIME: 2025-02-24 17:03:38 -- STEP: 1032/36938 -- GLOBAL_STEP: 407350 | > loss_text_ce: 0.04010296240448952 (0.038805034294067516) | > loss_mel_ce: 2.7112650871276855 (nan) | > loss: 0.010918127372860909 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1362556630326796) | > loader_time: 0.0046 (0.004675779000733253)  --> TIME: 2025-02-24 17:03:47 -- STEP: 1082/36938 -- GLOBAL_STEP: 407400 | > loss_text_ce: 0.028073757886886597 (0.03880052955635239) | > loss_mel_ce: 2.067284345626831 (nan) | > loss: 0.00831491407006979 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13624869521135766) | > loader_time: 0.0046 (0.004675172393291145)  --> TIME: 2025-02-24 17:03:56 -- STEP: 1132/36938 -- GLOBAL_STEP: 407450 | > loss_text_ce: 0.04915401712059975 (0.03879424774206438) | > loss_mel_ce: 3.6046316623687744 (nan) | > loss: 0.014499150216579437 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13615098648273488) | > loader_time: 0.0048 (0.00467283586731227)  --> TIME: 2025-02-24 17:04:05 -- STEP: 1182/36938 -- GLOBAL_STEP: 407500 | > loss_text_ce: 0.03838934749364853 (0.03872620772062951) | > loss_mel_ce: 3.2062289714813232 (nan) | > loss: 0.012875470332801342 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13606544174316987) | > loader_time: 0.0046 (0.004670858786594247)  --> TIME: 2025-02-24 17:04:15 -- STEP: 1232/36938 -- GLOBAL_STEP: 407550 | > loss_text_ce: 0.04072483628988266 (0.03870177145687525) | > loss_mel_ce: 3.463853597640991 (nan) | > loss: 0.013907058164477348 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13630297992910664) | > loader_time: 0.0047 (0.0046719643589738175)  --> TIME: 2025-02-24 17:04:25 -- STEP: 1282/36938 -- GLOBAL_STEP: 407600 | > loss_text_ce: 0.030535323545336723 (0.03875506850825437) | > loss_mel_ce: 2.7835021018981934 (nan) | > loss: 0.011166815645992756 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1364410433122037) | > loader_time: 0.0045 (0.004669968869868384)  --> TIME: 2025-02-24 17:04:34 -- STEP: 1332/36938 -- GLOBAL_STEP: 407650 | > loss_text_ce: 0.03992893546819687 (0.03882284789603469) | > loss_mel_ce: 3.3622100353240967 (nan) | > loss: 0.013500552624464035 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13648605543571907) | > loader_time: 0.0046 (0.004669006164367495)  --> TIME: 2025-02-24 17:04:43 -- STEP: 1382/36938 -- GLOBAL_STEP: 407700 | > loss_text_ce: 0.043345704674720764 (0.038850067344436315) | > loss_mel_ce: 3.269111156463623 (nan) | > loss: 0.013144670985639095 (nan) | > current_lr: 5e-06 | > step_time: 0.1435 (0.1363942243600892) | > loader_time: 0.0047 (0.004666756099971434)  --> TIME: 2025-02-24 17:04:52 -- STEP: 1432/36938 -- GLOBAL_STEP: 407750 | > loss_text_ce: 0.03695748373866081 (0.038870670062011246) | > loss_mel_ce: 2.777425765991211 (nan) | > loss: 0.011168188415467739 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.13635730793356227) | > loader_time: 0.0067 (0.004667030699426234)  --> TIME: 2025-02-24 17:05:02 -- STEP: 1482/36938 -- GLOBAL_STEP: 407800 | > loss_text_ce: 0.04661129042506218 (0.038912089985849455) | > loss_mel_ce: 1.7795820236206055 (nan) | > loss: 0.007246799301356077 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13638508110715628) | > loader_time: 0.0045 (0.0046699518455989245)  --> TIME: 2025-02-24 17:05:11 -- STEP: 1532/36938 -- GLOBAL_STEP: 407850 | > loss_text_ce: 0.03956596180796623 (0.03890177115917156) | > loss_mel_ce: 3.6949024200439453 (nan) | > loss: 0.014819320291280746 (nan) | > current_lr: 5e-06 | > step_time: 0.1438 (0.13634882847883067) | > loader_time: 0.0046 (0.004668907770600075)  --> TIME: 2025-02-24 17:05:20 -- STEP: 1582/36938 -- GLOBAL_STEP: 407900 | > loss_text_ce: 0.036669839173555374 (0.03891198882037213) | > loss_mel_ce: 3.4545300006866455 (nan) | > loss: 0.013853968121111393 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.13626690760876528) | > loader_time: 0.0048 (0.004666919352583269)  --> TIME: 2025-02-24 17:05:29 -- STEP: 1632/36938 -- GLOBAL_STEP: 407950 | > loss_text_ce: 0.032912421971559525 (0.03885659292698665) | > loss_mel_ce: 3.6587154865264893 (nan) | > loss: 0.014649318531155586 (nan) | > current_lr: 5e-06 | > step_time: 0.1626 (0.13625345393723134) | > loader_time: 0.0047 (0.004664511365049025)  --> TIME: 2025-02-24 17:05:39 -- STEP: 1682/36938 -- GLOBAL_STEP: 408000 | > loss_text_ce: 0.04343423619866371 (0.038843539021311345) | > loss_mel_ce: 2.691359758377075 (nan) | > loss: 0.010852357372641563 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13627968220024686) | > loader_time: 0.0046 (0.0046621690040252696)  --> TIME: 2025-02-24 17:05:48 -- STEP: 1732/36938 -- GLOBAL_STEP: 408050 | > loss_text_ce: 0.03691362217068672 (0.03888292865038324) | > loss_mel_ce: 1.371263027191162 (nan) | > loss: 0.005588002968579531 (nan) | > current_lr: 5e-06 | > step_time: 0.1075 (0.1364429619240705) | > loader_time: 0.0045 (0.004661609201453297)  --> TIME: 2025-02-24 17:05:57 -- STEP: 1782/36938 -- GLOBAL_STEP: 408100 | > loss_text_ce: 0.044221460819244385 (0.038927720439976846) | > loss_mel_ce: 3.142793655395508 (nan) | > loss: 0.012646885588765144 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.13635339225850385) | > loader_time: 0.0048 (0.004660831557379826)  --> TIME: 2025-02-24 17:06:07 -- STEP: 1832/36938 -- GLOBAL_STEP: 408150 | > loss_text_ce: 0.03968137130141258 (0.03891861634318804) | > loss_mel_ce: 3.5168404579162598 (nan) | > loss: 0.014113183133304119 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.13640739511714753) | > loader_time: 0.0048 (0.00466044696137374)  --> TIME: 2025-02-24 17:06:16 -- STEP: 1882/36938 -- GLOBAL_STEP: 408200 | > loss_text_ce: 0.028686676174402237 (0.03891477377008454) | > loss_mel_ce: 3.1068129539489746 (nan) | > loss: 0.012442460283637047 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13648236646662362) | > loader_time: 0.0047 (0.00465862099853256)  --> TIME: 2025-02-24 17:06:26 -- STEP: 1932/36938 -- GLOBAL_STEP: 408250 | > loss_text_ce: 0.039732445031404495 (0.038917452760723434) | > loss_mel_ce: 3.46948504447937 (nan) | > loss: 0.013925466686487198 (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.13650379553591496) | > loader_time: 0.0046 (0.004656863632162657)  --> TIME: 2025-02-24 17:06:35 -- STEP: 1982/36938 -- GLOBAL_STEP: 408300 | > loss_text_ce: 0.051303938031196594 (0.03895852937833927) | > loss_mel_ce: 2.4001376628875732 (nan) | > loss: 0.009727942757308483 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13648734528169146) | > loader_time: 0.0045 (0.004655426373756015)  --> TIME: 2025-02-24 17:06:44 -- STEP: 2032/36938 -- GLOBAL_STEP: 408350 | > loss_text_ce: 0.035780902951955795 (0.03894736606391214) | > loss_mel_ce: 0.691113293170929 (nan) | > loss: 0.0028845008928328753 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13647984191188667) | > loader_time: 0.0047 (0.0046547284041802695)  --> TIME: 2025-02-24 17:06:53 -- STEP: 2082/36938 -- GLOBAL_STEP: 408400 | > loss_text_ce: 0.037752665579319 (0.038948772361503924) | > loss_mel_ce: 2.6119368076324463 (nan) | > loss: 0.010514641180634499 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13645104601564148) | > loader_time: 0.0045 (0.0046529883266525915)  --> TIME: 2025-02-24 17:07:03 -- STEP: 2132/36938 -- GLOBAL_STEP: 408450 | > loss_text_ce: 0.04208466038107872 (0.03893138427640308) | > loss_mel_ce: 2.8477132320404053 (nan) | > loss: 0.011467453092336655 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.1364691722683791) | > loader_time: 0.0046 (0.00465126344008025)  --> TIME: 2025-02-24 17:07:12 -- STEP: 2182/36938 -- GLOBAL_STEP: 408500 | > loss_text_ce: 0.038151223212480545 (0.03895299405791728) | > loss_mel_ce: 3.582505226135254 (nan) | > loss: 0.014367685653269291 (nan) | > current_lr: 5e-06 | > step_time: 0.1444 (0.13648596212035688) | > loader_time: 0.0045 (0.004650839854554226)  --> TIME: 2025-02-24 17:07:21 -- STEP: 2232/36938 -- GLOBAL_STEP: 408550 | > loss_text_ce: 0.04294959828257561 (0.03895077048917727) | > loss_mel_ce: 3.1682536602020264 (nan) | > loss: 0.012742871418595314 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13656523236237128) | > loader_time: 0.0046 (0.0046500148098101305)  --> TIME: 2025-02-24 17:07:30 -- STEP: 2282/36938 -- GLOBAL_STEP: 408600 | > loss_text_ce: 0.05210411548614502 (0.038965588503278116) | > loss_mel_ce: 3.0669643878936768 (nan) | > loss: 0.012377257458865643 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13643706670254613) | > loader_time: 0.0045 (0.004649209725449313)  --> TIME: 2025-02-24 17:07:40 -- STEP: 2332/36938 -- GLOBAL_STEP: 408650 | > loss_text_ce: 0.03530732914805412 (0.03898207058069361) | > loss_mel_ce: 2.9300246238708496 (nan) | > loss: 0.011767190881073475 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.1363903281005743) | > loader_time: 0.0046 (0.0046507430935396705)  --> TIME: 2025-02-24 17:07:49 -- STEP: 2382/36938 -- GLOBAL_STEP: 408700 | > loss_text_ce: 0.03849503770470619 (0.03899363684370231) | > loss_mel_ce: 3.3198859691619873 (nan) | > loss: 0.013326909393072128 (nan) | > current_lr: 5e-06 | > step_time: 0.169 (0.1364919428661231) | > loader_time: 0.0046 (0.004651402846591791)  --> TIME: 2025-02-24 17:07:59 -- STEP: 2432/36938 -- GLOBAL_STEP: 408750 | > loss_text_ce: 0.0403028279542923 (0.03896726390858509) | > loss_mel_ce: 1.13006591796875 (nan) | > loss: 0.00464432081207633 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13651949775062136) | > loader_time: 0.0046 (0.0046509984684617815)  --> TIME: 2025-02-24 17:08:08 -- STEP: 2482/36938 -- GLOBAL_STEP: 408800 | > loss_text_ce: 0.04001889005303383 (0.03893766478439513) | > loss_mel_ce: 2.2061517238616943 (nan) | > loss: 0.008913375437259674 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13660943210365137) | > loader_time: 0.0047 (0.004651090485928608)  --> TIME: 2025-02-24 17:08:18 -- STEP: 2532/36938 -- GLOBAL_STEP: 408850 | > loss_text_ce: 0.042903099209070206 (0.03893827035578827) | > loss_mel_ce: 2.5625338554382324 (nan) | > loss: 0.010339036583900452 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1366391207369584) | > loader_time: 0.0047 (0.004650749301458412)  --> TIME: 2025-02-24 17:08:27 -- STEP: 2582/36938 -- GLOBAL_STEP: 408900 | > loss_text_ce: 0.038079358637332916 (0.03892092847017878) | > loss_mel_ce: 1.8155266046524048 (nan) | > loss: 0.007355579640716314 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13666706127680894) | > loader_time: 0.0045 (0.004650635187428269)  --> TIME: 2025-02-24 17:08:37 -- STEP: 2632/36938 -- GLOBAL_STEP: 408950 | > loss_text_ce: 0.044875603169202805 (0.03891949836762016) | > loss_mel_ce: 2.8276960849761963 (nan) | > loss: 0.011399094946682453 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13671316718258042) | > loader_time: 0.0045 (0.004650724332745665)  --> TIME: 2025-02-24 17:08:46 -- STEP: 2682/36938 -- GLOBAL_STEP: 409000 | > loss_text_ce: 0.0338241383433342 (0.03891245875227027) | > loss_mel_ce: 2.746377944946289 (nan) | > loss: 0.011032548733055592 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.136672479105387) | > loader_time: 0.0046 (0.004650767928713021)  --> TIME: 2025-02-24 17:08:55 -- STEP: 2732/36938 -- GLOBAL_STEP: 409050 | > loss_text_ce: 0.0389457531273365 (0.03890664769625668) | > loss_mel_ce: 3.1493875980377197 (nan) | > loss: 0.01265211682766676 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13672104045901445) | > loader_time: 0.0045 (0.004651398819327171)  --> TIME: 2025-02-24 17:09:05 -- STEP: 2782/36938 -- GLOBAL_STEP: 409100 | > loss_text_ce: 0.031875431537628174 (0.038899379047229794) | > loss_mel_ce: 3.30720853805542 (nan) | > loss: 0.013250334188342094 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13673102024388276) | > loader_time: 0.0047 (0.004651192192719355)  --> TIME: 2025-02-24 17:09:14 -- STEP: 2832/36938 -- GLOBAL_STEP: 409150 | > loss_text_ce: 0.034019846469163895 (0.03890276105290466) | > loss_mel_ce: 3.2146358489990234 (nan) | > loss: 0.012891491875052452 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13682895612581988) | > loader_time: 0.0047 (0.004651714263662773)  --> TIME: 2025-02-24 17:09:24 -- STEP: 2882/36938 -- GLOBAL_STEP: 409200 | > loss_text_ce: 0.03815824165940285 (0.038899271613236475) | > loss_mel_ce: 3.2524969577789307 (nan) | > loss: 0.01305815577507019 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.13682587298976312) | > loader_time: 0.0046 (0.004651494939487727)  --> TIME: 2025-02-24 17:09:33 -- STEP: 2932/36938 -- GLOBAL_STEP: 409250 | > loss_text_ce: 0.02442658692598343 (0.03890950039936311) | > loss_mel_ce: 2.7661244869232178 (nan) | > loss: 0.011073616333305836 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13679159988647932) | > loader_time: 0.0047 (0.004651046384730734)  --> TIME: 2025-02-24 17:09:42 -- STEP: 2982/36938 -- GLOBAL_STEP: 409300 | > loss_text_ce: 0.036815863102674484 (0.0388927591875906) | > loss_mel_ce: 2.9312632083892822 (nan) | > loss: 0.011778092011809349 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13684252937874658) | > loader_time: 0.0047 (0.0046505515484263)  --> TIME: 2025-02-24 17:09:51 -- STEP: 3032/36938 -- GLOBAL_STEP: 409350 | > loss_text_ce: 0.04562807083129883 (0.0389039139251046) | > loss_mel_ce: 1.2660945653915405 (nan) | > loss: 0.005205248948186636 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13679062014833937) | > loader_time: 0.0045 (0.004649638028761332)  --> TIME: 2025-02-24 17:10:01 -- STEP: 3082/36938 -- GLOBAL_STEP: 409400 | > loss_text_ce: 0.03597268834710121 (0.03893046649710793) | > loss_mel_ce: 2.93300461769104 (nan) | > loss: 0.011781656183302402 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1368050946576655) | > loader_time: 0.0045 (0.004649541116549866)  --> TIME: 2025-02-24 17:10:10 -- STEP: 3132/36938 -- GLOBAL_STEP: 409450 | > loss_text_ce: 0.04015478864312172 (0.038930768003190154) | > loss_mel_ce: 2.80841064453125 (nan) | > loss: 0.011303831823170185 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.1368469104913006) | > loader_time: 0.0047 (0.004649195482295444)  --> TIME: 2025-02-24 17:10:20 -- STEP: 3182/36938 -- GLOBAL_STEP: 409500 | > loss_text_ce: 0.034872326999902725 (0.03894793921771348) | > loss_mel_ce: 2.2870590686798096 (nan) | > loss: 0.009214013814926147 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.1368923537915495) | > loader_time: 0.0047 (0.00465001001963445)  --> TIME: 2025-02-24 17:10:29 -- STEP: 3232/36938 -- GLOBAL_STEP: 409550 | > loss_text_ce: 0.03661623224616051 (0.03896838783961955) | > loss_mel_ce: 2.2659189701080322 (nan) | > loss: 0.009137045592069626 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.1368928002101364) | > loader_time: 0.0045 (0.004649645990074277)  --> TIME: 2025-02-24 17:10:39 -- STEP: 3282/36938 -- GLOBAL_STEP: 409600 | > loss_text_ce: 0.034950658679008484 (0.03900937960493918) | > loss_mel_ce: 3.1549525260925293 (nan) | > loss: 0.012658347375690937 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.1369295568454567) | > loader_time: 0.0047 (0.004649412842656987)  --> TIME: 2025-02-24 17:10:48 -- STEP: 3332/36938 -- GLOBAL_STEP: 409650 | > loss_text_ce: 0.042721860110759735 (0.03902295225110948) | > loss_mel_ce: 3.0195815563201904 (nan) | > loss: 0.012151997536420822 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.13698315534557368) | > loader_time: 0.0049 (0.004650317821182125)  --> TIME: 2025-02-24 17:10:57 -- STEP: 3382/36938 -- GLOBAL_STEP: 409700 | > loss_text_ce: 0.05136379599571228 (0.03903162107367283) | > loss_mel_ce: 3.2547457218170166 (nan) | > loss: 0.013119482435286045 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13697014230020976) | > loader_time: 0.0048 (0.004650281421677171)  --> TIME: 2025-02-24 17:11:07 -- STEP: 3432/36938 -- GLOBAL_STEP: 409750 | > loss_text_ce: 0.03729911893606186 (0.039005105114500706) | > loss_mel_ce: 2.460158109664917 (nan) | > loss: 0.00991054531186819 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13692828144504798) | > loader_time: 0.0046 (0.004650089012715091)  --> TIME: 2025-02-24 17:11:16 -- STEP: 3482/36938 -- GLOBAL_STEP: 409800 | > loss_text_ce: 0.043974608182907104 (0.038984006335493454) | > loss_mel_ce: 1.6371160745620728 (nan) | > loss: 0.006670995149761438 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13696185540084863) | > loader_time: 0.0045 (0.004650373009687729)  --> TIME: 2025-02-24 17:11:26 -- STEP: 3532/36938 -- GLOBAL_STEP: 409850 | > loss_text_ce: 0.03500936180353165 (0.03896743508682314) | > loss_mel_ce: 3.167570114135742 (nan) | > loss: 0.012708649970591068 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.1370384843001151) | > loader_time: 0.0046 (0.004650720788560396)  --> TIME: 2025-02-24 17:11:35 -- STEP: 3582/36938 -- GLOBAL_STEP: 409900 | > loss_text_ce: 0.040807731449604034 (0.038964554278511775) | > loss_mel_ce: 2.7562146186828613 (nan) | > loss: 0.011099295690655708 (nan) | > current_lr: 5e-06 | > step_time: 0.182 (0.13708036881728597) | > loader_time: 0.0049 (0.004650494494563307)  --> TIME: 2025-02-24 17:11:45 -- STEP: 3632/36938 -- GLOBAL_STEP: 409950 | > loss_text_ce: 0.024264739826321602 (0.03894678440605525) | > loss_mel_ce: 2.9236767292022705 (nan) | > loss: 0.011698181740939617 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13706819693422603) | > loader_time: 0.0045 (0.004649652455346702)  --> TIME: 2025-02-24 17:11:54 -- STEP: 3682/36938 -- GLOBAL_STEP: 410000 | > loss_text_ce: 0.033743616193532944 (0.038951515715360846) | > loss_mel_ce: 2.5382449626922607 (nan) | > loss: 0.010206304490566254 (nan) | > current_lr: 5e-06 | > step_time: 0.1511 (0.1370698027737702) | > loader_time: 0.0046 (0.004649513445102543) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_410000.pth  --> TIME: 2025-02-24 17:12:09 -- STEP: 3732/36938 -- GLOBAL_STEP: 410050 | > loss_text_ce: 0.032677892595529556 (0.03895491466678815) | > loss_mel_ce: 2.9503908157348633 (nan) | > loss: 0.011837574653327465 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.1370746854289432) | > loader_time: 0.0047 (0.004650314137866644)  --> TIME: 2025-02-24 17:12:19 -- STEP: 3782/36938 -- GLOBAL_STEP: 410100 | > loss_text_ce: 0.050705280154943466 (0.03894573472891376) | > loss_mel_ce: 3.444580078125 (nan) | > loss: 0.0138701805844903 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13706863613622977) | > loader_time: 0.0046 (0.004650603899055535)  --> TIME: 2025-02-24 17:12:28 -- STEP: 3832/36938 -- GLOBAL_STEP: 410150 | > loss_text_ce: 0.03423751890659332 (0.0389358000798221) | > loss_mel_ce: 2.9049184322357178 (nan) | > loss: 0.011663318611681461 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13699321165712072) | > loader_time: 0.0048 (0.004649770844704425)  --> TIME: 2025-02-24 17:12:37 -- STEP: 3882/36938 -- GLOBAL_STEP: 410200 | > loss_text_ce: 0.043741725385189056 (0.03893897403007598) | > loss_mel_ce: 3.0296952724456787 (nan) | > loss: 0.01219617947936058 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13690993258905265) | > loader_time: 0.0046 (0.004649881048954254)  --> TIME: 2025-02-24 17:12:46 -- STEP: 3932/36938 -- GLOBAL_STEP: 410250 | > loss_text_ce: 0.033439941704273224 (0.038955956010341214) | > loss_mel_ce: 3.5170629024505615 (nan) | > loss: 0.014089297503232956 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.1368676187548252) | > loader_time: 0.0047 (0.004649993180015931)  --> TIME: 2025-02-24 17:12:55 -- STEP: 3982/36938 -- GLOBAL_STEP: 410300 | > loss_text_ce: 0.031531307846307755 (0.03894667630493155) | > loss_mel_ce: 0.02520751953125 (nan) | > loss: 0.0002251540863653645 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13689054878796317) | > loader_time: 0.0046 (0.0046520294225856585)  --> TIME: 2025-02-24 17:13:04 -- STEP: 4032/36938 -- GLOBAL_STEP: 410350 | > loss_text_ce: 0.0378757044672966 (0.038949061737401376) | > loss_mel_ce: 3.15811824798584 (nan) | > loss: 0.012682516127824783 (nan) | > current_lr: 5e-06 | > step_time: 0.1071 (0.13685638209184084) | > loader_time: 0.0046 (0.004652162629460534)  --> TIME: 2025-02-24 17:13:14 -- STEP: 4082/36938 -- GLOBAL_STEP: 410400 | > loss_text_ce: 0.03780987113714218 (0.03896983121378035) | > loss_mel_ce: 1.7123326063156128 (nan) | > loss: 0.006945010274648666 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13680689899083928) | > loader_time: 0.0046 (0.004652128331746501)  --> TIME: 2025-02-24 17:13:23 -- STEP: 4132/36938 -- GLOBAL_STEP: 410450 | > loss_text_ce: 0.038550522178411484 (0.03894623468849999) | > loss_mel_ce: 3.081573963165283 (nan) | > loss: 0.012381447479128838 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.13673824793140815) | > loader_time: 0.0048 (0.004652647577435328)  --> TIME: 2025-02-24 17:13:32 -- STEP: 4182/36938 -- GLOBAL_STEP: 410500 | > loss_text_ce: 0.03092372976243496 (0.038936143058887984) | > loss_mel_ce: 3.5952706336975098 (nan) | > loss: 0.014389661140739918 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13679574460632157) | > loader_time: 0.0047 (0.004653441398580472)  --> TIME: 2025-02-24 17:13:42 -- STEP: 4232/36938 -- GLOBAL_STEP: 410550 | > loss_text_ce: 0.0365077406167984 (0.038953464999152365) | > loss_mel_ce: 3.613830089569092 (nan) | > loss: 0.014485469087958336 (nan) | > current_lr: 5e-06 | > step_time: 0.1529 (0.13679495070742365) | > loader_time: 0.0047 (0.0046536073454837065)  --> TIME: 2025-02-24 17:13:51 -- STEP: 4282/36938 -- GLOBAL_STEP: 410600 | > loss_text_ce: 0.03462911769747734 (0.03894571046506319) | > loss_mel_ce: 2.116469144821167 (nan) | > loss: 0.008536105044186115 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13675132570618675) | > loader_time: 0.0048 (0.0046538111206974086)  --> TIME: 2025-02-24 17:14:00 -- STEP: 4332/36938 -- GLOBAL_STEP: 410650 | > loss_text_ce: 0.04438963904976845 (0.03893464772802083) | > loss_mel_ce: 1.8995205163955688 (nan) | > loss: 0.007713929284363985 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13673164287542547) | > loader_time: 0.0045 (0.004654016411095337)  --> TIME: 2025-02-24 17:14:10 -- STEP: 4382/36938 -- GLOBAL_STEP: 410700 | > loss_text_ce: 0.03866839408874512 (0.03891197165953771) | > loss_mel_ce: 3.006072998046875 (nan) | > loss: 0.01208230759948492 (nan) | > current_lr: 5e-06 | > step_time: 0.1352 (0.13677912327537092) | > loader_time: 0.0047 (0.004654482857161383)  --> TIME: 2025-02-24 17:14:19 -- STEP: 4432/36938 -- GLOBAL_STEP: 410750 | > loss_text_ce: 0.03891323134303093 (0.038905329318315356) | > loss_mel_ce: 1.9299756288528442 (nan) | > loss: 0.007813051342964172 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13679639037550628) | > loader_time: 0.0045 (0.004654287323624648)  --> TIME: 2025-02-24 17:14:28 -- STEP: 4482/36938 -- GLOBAL_STEP: 410800 | > loss_text_ce: 0.03897634893655777 (0.03889692796099998) | > loss_mel_ce: 3.3820297718048096 (nan) | > loss: 0.013575422577559948 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.1368051059327988) | > loader_time: 0.0046 (0.00465400912622318)  --> TIME: 2025-02-24 17:14:38 -- STEP: 4532/36938 -- GLOBAL_STEP: 410850 | > loss_text_ce: 0.02844269759953022 (0.03886957604923871) | > loss_mel_ce: 2.5607218742370605 (nan) | > loss: 0.010274463333189487 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.1368030243917371) | > loader_time: 0.0045 (0.004654128679846205)  --> TIME: 2025-02-24 17:14:47 -- STEP: 4582/36938 -- GLOBAL_STEP: 410900 | > loss_text_ce: 0.04859491065144539 (0.038884176068790126) | > loss_mel_ce: 2.7664499282836914 (nan) | > loss: 0.01117081381380558 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13675438910267498) | > loader_time: 0.0046 (0.004653785021634982)  --> TIME: 2025-02-24 17:14:56 -- STEP: 4632/36938 -- GLOBAL_STEP: 410950 | > loss_text_ce: 0.041536830365657806 (0.038863150252709776) | > loss_mel_ce: 3.335334539413452 (nan) | > loss: 0.013400283642113209 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13675175753899868) | > loader_time: 0.0046 (0.004654268423716229)  --> TIME: 2025-02-24 17:15:06 -- STEP: 4682/36938 -- GLOBAL_STEP: 411000 | > loss_text_ce: 0.05103312432765961 (0.03884971687937818) | > loss_mel_ce: 2.99992299079895 (nan) | > loss: 0.012106969021260738 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.1367864715192427) | > loader_time: 0.0047 (0.004654368902057325)  --> TIME: 2025-02-24 17:15:15 -- STEP: 4732/36938 -- GLOBAL_STEP: 411050 | > loss_text_ce: 0.07011858373880386 (0.03884541539814637) | > loss_mel_ce: 3.226898670196533 (nan) | > loss: 0.013083402998745441 (nan) | > current_lr: 5e-06 | > step_time: 0.1071 (0.13679823009824774) | > loader_time: 0.0046 (0.004654343311603256)  --> TIME: 2025-02-24 17:15:24 -- STEP: 4782/36938 -- GLOBAL_STEP: 411100 | > loss_text_ce: 0.03779757767915726 (0.03884148911098962) | > loss_mel_ce: 3.0737111568450928 (nan) | > loss: 0.012347257696092129 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.1367965724766385) | > loader_time: 0.0046 (0.004654671596413829)  --> TIME: 2025-02-24 17:15:34 -- STEP: 4832/36938 -- GLOBAL_STEP: 411150 | > loss_text_ce: 0.041507892310619354 (0.038846779845590255) | > loss_mel_ce: 2.909456968307495 (nan) | > loss: 0.011710179038345814 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13679797365176025) | > loader_time: 0.0047 (0.004654679768132852)  --> TIME: 2025-02-24 17:15:43 -- STEP: 4882/36938 -- GLOBAL_STEP: 411200 | > loss_text_ce: 0.04528094083070755 (0.03884862849678861) | > loss_mel_ce: 3.0611190795898438 (nan) | > loss: 0.012326984666287899 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13679246993301314) | > loader_time: 0.0045 (0.004654500143589518)  --> TIME: 2025-02-24 17:15:52 -- STEP: 4932/36938 -- GLOBAL_STEP: 411250 | > loss_text_ce: 0.03280648961663246 (0.0388669337807654) | > loss_mel_ce: 2.018786668777466 (nan) | > loss: 0.008141242899000645 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1367651311633851) | > loader_time: 0.0045 (0.004654087579453368)  --> TIME: 2025-02-24 17:16:02 -- STEP: 4982/36938 -- GLOBAL_STEP: 411300 | > loss_text_ce: 0.0327904149889946 (0.03887181806844292) | > loss_mel_ce: 2.42038631439209 (nan) | > loss: 0.00973482895642519 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.1368038260951101) | > loader_time: 0.0046 (0.004653910660255585)  --> TIME: 2025-02-24 17:16:11 -- STEP: 5032/36938 -- GLOBAL_STEP: 411350 | > loss_text_ce: 0.04118119552731514 (0.03887148173592563) | > loss_mel_ce: 2.588761568069458 (nan) | > loss: 0.010436280630528927 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.13676774378611362) | > loader_time: 0.0048 (0.004653633304165354)  --> TIME: 2025-02-24 17:16:20 -- STEP: 5082/36938 -- GLOBAL_STEP: 411400 | > loss_text_ce: 0.03196629136800766 (0.038864577159033444) | > loss_mel_ce: 3.496299982070923 (nan) | > loss: 0.014001057483255863 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13676897173367047) | > loader_time: 0.0045 (0.0046534848841847725)  --> TIME: 2025-02-24 17:16:30 -- STEP: 5132/36938 -- GLOBAL_STEP: 411450 | > loss_text_ce: 0.04419941455125809 (0.03885944508922031) | > loss_mel_ce: 3.7945985794067383 (nan) | > loss: 0.015233326703310013 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.13676294974767952) | > loader_time: 0.0047 (0.004653549668199833)  --> TIME: 2025-02-24 17:16:39 -- STEP: 5182/36938 -- GLOBAL_STEP: 411500 | > loss_text_ce: 0.05269791558384895 (0.03885736805683389) | > loss_mel_ce: 3.2427995204925537 (nan) | > loss: 0.013077371753752232 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13679168892385293) | > loader_time: 0.0048 (0.004653545890890553)  --> TIME: 2025-02-24 17:16:48 -- STEP: 5232/36938 -- GLOBAL_STEP: 411550 | > loss_text_ce: 0.03626403585076332 (0.038854144407872035) | > loss_mel_ce: 3.1343612670898438 (nan) | > loss: 0.012581846676766872 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13673295732302823) | > loader_time: 0.0046 (0.0046535466515689825)  --> TIME: 2025-02-24 17:16:57 -- STEP: 5282/36938 -- GLOBAL_STEP: 411600 | > loss_text_ce: 0.026679303497076035 (0.03885668575332074) | > loss_mel_ce: 2.740551710128784 (nan) | > loss: 0.010981076397001743 (nan) | > current_lr: 5e-06 | > step_time: 0.1265 (0.13672610919884554) | > loader_time: 0.0046 (0.004653360707341881)  --> TIME: 2025-02-24 17:17:06 -- STEP: 5332/36938 -- GLOBAL_STEP: 411650 | > loss_text_ce: 0.031960248947143555 (0.03885028202894349) | > loss_mel_ce: 6.434895992279053 (nan) | > loss: 0.025662127882242203 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13669321645227359) | > loader_time: 0.0044 (0.004653184063346961)  --> TIME: 2025-02-24 17:17:16 -- STEP: 5382/36938 -- GLOBAL_STEP: 411700 | > loss_text_ce: 0.03526749461889267 (0.03884723327260724) | > loss_mel_ce: 2.905367851257324 (nan) | > loss: 0.011669188737869263 (nan) | > current_lr: 5e-06 | > step_time: 0.1828 (0.13673024508023734) | > loader_time: 0.0049 (0.004653267415648097)  --> TIME: 2025-02-24 17:17:26 -- STEP: 5432/36938 -- GLOBAL_STEP: 411750 | > loss_text_ce: 0.042609717696905136 (0.03883972705104595) | > loss_mel_ce: 2.3129916191101074 (nan) | > loss: 0.009347625076770782 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13679629625733405) | > loader_time: 0.0046 (0.0046535758452492665)  --> TIME: 2025-02-24 17:17:35 -- STEP: 5482/36938 -- GLOBAL_STEP: 411800 | > loss_text_ce: 0.047762852162122726 (0.03883348098170837) | > loss_mel_ce: 3.1905267238616943 (nan) | > loss: 0.012850356288254261 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13682009668430267) | > loader_time: 0.0045 (0.004653454740218805)  --> TIME: 2025-02-24 17:17:44 -- STEP: 5532/36938 -- GLOBAL_STEP: 411850 | > loss_text_ce: 0.0363827608525753 (0.03884057224231875) | > loss_mel_ce: 2.6854491233825684 (nan) | > loss: 0.01080092042684555 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.13683383226049894) | > loader_time: 0.0046 (0.004653411375295394)  --> TIME: 2025-02-24 17:17:54 -- STEP: 5582/36938 -- GLOBAL_STEP: 411900 | > loss_text_ce: 0.04207582771778107 (0.03882335447441008) | > loss_mel_ce: 2.5499074459075928 (nan) | > loss: 0.01028564851731062 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13686170094502717) | > loader_time: 0.0046 (0.004653721417803261)  --> TIME: 2025-02-24 17:18:03 -- STEP: 5632/36938 -- GLOBAL_STEP: 411950 | > loss_text_ce: 0.041149724274873734 (0.03881594229857335) | > loss_mel_ce: 2.340216636657715 (nan) | > loss: 0.009449866600334644 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13682380983267345) | > loader_time: 0.0047 (0.004653619221326968)  --> TIME: 2025-02-24 17:18:12 -- STEP: 5682/36938 -- GLOBAL_STEP: 412000 | > loss_text_ce: 0.04066042974591255 (0.0388076672247234) | > loss_mel_ce: 3.124445676803589 (nan) | > loss: 0.012559945695102215 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.1368439322582061) | > loader_time: 0.0047 (0.004653596785739029)  --> TIME: 2025-02-24 17:18:22 -- STEP: 5732/36938 -- GLOBAL_STEP: 412050 | > loss_text_ce: 0.04223546385765076 (0.038818190737048446) | > loss_mel_ce: 3.4807941913604736 (nan) | > loss: 0.013980276882648468 (nan) | > current_lr: 5e-06 | > step_time: 0.1875 (0.1368655255434821) | > loader_time: 0.0049 (0.0046535093553067815)  --> TIME: 2025-02-24 17:18:31 -- STEP: 5782/36938 -- GLOBAL_STEP: 412100 | > loss_text_ce: 0.03675953671336174 (0.038813132273090675) | > loss_mel_ce: 3.40661883354187 (nan) | > loss: 0.013664200901985168 (nan) | > current_lr: 5e-06 | > step_time: 0.1825 (0.13686716049363276) | > loader_time: 0.0049 (0.004653343730465998)  --> TIME: 2025-02-24 17:18:40 -- STEP: 5832/36938 -- GLOBAL_STEP: 412150 | > loss_text_ce: 0.03741256520152092 (0.03879880741595036) | > loss_mel_ce: 0.36402344703674316 (nan) | > loss: 0.0015930001391097903 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.1368396013198407) | > loader_time: 0.0045 (0.0046531385109748295)  --> TIME: 2025-02-24 17:18:50 -- STEP: 5882/36938 -- GLOBAL_STEP: 412200 | > loss_text_ce: 0.031903475522994995 (0.03879107694834767) | > loss_mel_ce: 3.592782497406006 (nan) | > loss: 0.014383675530552864 (nan) | > current_lr: 5e-06 | > step_time: 0.1606 (0.13684527515676007) | > loader_time: 0.0046 (0.004653100333461866)  --> TIME: 2025-02-24 17:18:59 -- STEP: 5932/36938 -- GLOBAL_STEP: 412250 | > loss_text_ce: 0.040226805955171585 (0.03879400795647183) | > loss_mel_ce: 2.9420242309570312 (nan) | > loss: 0.011834329925477505 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.1368219833155383) | > loader_time: 0.0047 (0.004652819236387529)  --> TIME: 2025-02-24 17:19:08 -- STEP: 5982/36938 -- GLOBAL_STEP: 412300 | > loss_text_ce: 0.03356204554438591 (0.038797395433688896) | > loss_mel_ce: 1.83934485912323 (nan) | > loss: 0.007432170677930117 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13681856994444005) | > loader_time: 0.0045 (0.004652710831392491)  --> TIME: 2025-02-24 17:19:17 -- STEP: 6032/36938 -- GLOBAL_STEP: 412350 | > loss_text_ce: 0.028751373291015625 (0.03879777043971396) | > loss_mel_ce: 3.0262298583984375 (nan) | > loss: 0.012122942134737968 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13674148778858514) | > loader_time: 0.0046 (0.004651772446278234)  --> TIME: 2025-02-24 17:19:26 -- STEP: 6082/36938 -- GLOBAL_STEP: 412400 | > loss_text_ce: 0.045367948710918427 (0.03879184869388472) | > loss_mel_ce: 3.5596940517425537 (nan) | > loss: 0.014305802062153816 (nan) | > current_lr: 5e-06 | > step_time: 0.1757 (0.13675004414373726) | > loader_time: 0.0049 (0.0046517184516859066)  --> TIME: 2025-02-24 17:19:36 -- STEP: 6132/36938 -- GLOBAL_STEP: 412450 | > loss_text_ce: 0.04216337949037552 (0.03880038798105685) | > loss_mel_ce: 2.8250732421875 (nan) | > loss: 0.011377924121916294 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13678537831147863) | > loader_time: 0.0045 (0.0046516926710075315)  --> TIME: 2025-02-24 17:19:45 -- STEP: 6182/36938 -- GLOBAL_STEP: 412500 | > loss_text_ce: 0.03334987163543701 (0.03878949357866191) | > loss_mel_ce: 2.613125801086426 (nan) | > loss: 0.010501888580620289 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13675501635066178) | > loader_time: 0.0046 (0.00465157860423014)  --> TIME: 2025-02-24 17:19:54 -- STEP: 6232/36938 -- GLOBAL_STEP: 412550 | > loss_text_ce: 0.034892719238996506 (0.03878897322886977) | > loss_mel_ce: 3.070878267288208 (nan) | > loss: 0.01232448872178793 (nan) | > current_lr: 5e-06 | > step_time: 0.1844 (0.13677457885503497) | > loader_time: 0.0049 (0.004652460479919964)  --> TIME: 2025-02-24 17:20:03 -- STEP: 6282/36938 -- GLOBAL_STEP: 412600 | > loss_text_ce: 0.02961178869009018 (0.03878987612352809) | > loss_mel_ce: 1.709008812904358 (nan) | > loss: 0.006899288855493069 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13673673948259446) | > loader_time: 0.0045 (0.004652262986446413)  --> TIME: 2025-02-24 17:20:13 -- STEP: 6332/36938 -- GLOBAL_STEP: 412650 | > loss_text_ce: 0.02998611144721508 (0.038778836500906994) | > loss_mel_ce: 3.176722526550293 (nan) | > loss: 0.012725034728646278 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.1367116684561205) | > loader_time: 0.0046 (0.004652280259267942)  --> TIME: 2025-02-24 17:20:22 -- STEP: 6382/36938 -- GLOBAL_STEP: 412700 | > loss_text_ce: 0.03622608259320259 (0.038776966582356936) | > loss_mel_ce: 3.0835509300231934 (nan) | > loss: 0.012380068190395832 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13671449186360354) | > loader_time: 0.0046 (0.004653240438166632)  --> TIME: 2025-02-24 17:20:31 -- STEP: 6432/36938 -- GLOBAL_STEP: 412750 | > loss_text_ce: 0.0501980185508728 (0.03878875247136801) | > loss_mel_ce: 2.8778889179229736 (nan) | > loss: 0.011619393713772297 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1367097618568004) | > loader_time: 0.0045 (0.00465375970845199)  --> TIME: 2025-02-24 17:20:41 -- STEP: 6482/36938 -- GLOBAL_STEP: 412800 | > loss_text_ce: 0.029268668964505196 (0.03879037177042495) | > loss_mel_ce: 2.776515483856201 (nan) | > loss: 0.011134064756333828 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13672297124471322) | > loader_time: 0.0045 (0.0046540048158452045)  --> TIME: 2025-02-24 17:20:50 -- STEP: 6532/36938 -- GLOBAL_STEP: 412850 | > loss_text_ce: 0.044745635241270065 (0.03878981045396544) | > loss_mel_ce: 3.3584463596343994 (nan) | > loss: 0.013504731468856335 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.13670182100338069) | > loader_time: 0.0049 (0.0046533883822453435)  --> TIME: 2025-02-24 17:20:59 -- STEP: 6582/36938 -- GLOBAL_STEP: 412900 | > loss_text_ce: 0.04558582603931427 (0.038797127448002136) | > loss_mel_ce: 3.3646538257598877 (nan) | > loss: 0.013532698154449463 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.136701967824161) | > loader_time: 0.0049 (0.004653632622727219)  --> TIME: 2025-02-24 17:21:09 -- STEP: 6632/36938 -- GLOBAL_STEP: 412950 | > loss_text_ce: 0.04255225881934166 (0.03880256797558901) | > loss_mel_ce: 2.8885390758514404 (nan) | > loss: 0.01163131557404995 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1367137754926922) | > loader_time: 0.0045 (0.00465404573913363)  --> TIME: 2025-02-24 17:21:18 -- STEP: 6682/36938 -- GLOBAL_STEP: 413000 | > loss_text_ce: 0.04185623675584793 (0.03879891635375645) | > loss_mel_ce: 3.443323850631714 (nan) | > loss: 0.01383008062839508 (nan) | > current_lr: 5e-06 | > step_time: 0.1662 (0.1367190766570026) | > loader_time: 0.0049 (0.004654122376577551)  --> TIME: 2025-02-24 17:21:27 -- STEP: 6732/36938 -- GLOBAL_STEP: 413050 | > loss_text_ce: 0.021221699193120003 (0.03879057727444244) | > loss_mel_ce: 3.4293673038482666 (nan) | > loss: 0.01369281392544508 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13672573440356825) | > loader_time: 0.0046 (0.004654315314120089)  --> TIME: 2025-02-24 17:21:37 -- STEP: 6782/36938 -- GLOBAL_STEP: 413100 | > loss_text_ce: 0.047164954245090485 (0.03880202370129741) | > loss_mel_ce: 3.806373119354248 (nan) | > loss: 0.015291818417608738 (nan) | > current_lr: 5e-06 | > step_time: 0.165 (0.1367224756857879) | > loader_time: 0.0047 (0.004654276233899223)  --> TIME: 2025-02-24 17:21:46 -- STEP: 6832/36938 -- GLOBAL_STEP: 413150 | > loss_text_ce: 0.0417572446167469 (0.03879358947540995) | > loss_mel_ce: 3.1470067501068115 (nan) | > loss: 0.012653826735913754 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13673927552945722) | > loader_time: 0.0045 (0.0046548431944233)  --> TIME: 2025-02-24 17:21:56 -- STEP: 6882/36938 -- GLOBAL_STEP: 413200 | > loss_text_ce: 0.03689834475517273 (0.03879106290686097) | > loss_mel_ce: 1.712677001953125 (nan) | > loss: 0.006942759733647108 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13675291771737677) | > loader_time: 0.0045 (0.004655572606601964)  --> TIME: 2025-02-24 17:22:05 -- STEP: 6932/36938 -- GLOBAL_STEP: 413250 | > loss_text_ce: 0.038471248000860214 (0.038796217421008625) | > loss_mel_ce: 3.3993217945098877 (nan) | > loss: 0.013642036356031895 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13672735164155866) | > loader_time: 0.0046 (0.004655276188187299)  --> TIME: 2025-02-24 17:22:14 -- STEP: 6982/36938 -- GLOBAL_STEP: 413300 | > loss_text_ce: 0.03706818073987961 (0.03879502846743895) | > loss_mel_ce: 2.9728329181671143 (nan) | > loss: 0.011944052763283253 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13672640862488125) | > loader_time: 0.0048 (0.004655869530934869)  --> TIME: 2025-02-24 17:22:23 -- STEP: 7032/36938 -- GLOBAL_STEP: 413350 | > loss_text_ce: 0.03730964660644531 (0.03880212339519647) | > loss_mel_ce: 2.627406597137451 (nan) | > loss: 0.010574270971119404 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13670744698609102) | > loader_time: 0.0047 (0.004656156582230197)  --> TIME: 2025-02-24 17:22:33 -- STEP: 7082/36938 -- GLOBAL_STEP: 413400 | > loss_text_ce: 0.03930011764168739 (0.03880244132238661) | > loss_mel_ce: 2.6311557292938232 (nan) | > loss: 0.010597048327326775 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13671187551908326) | > loader_time: 0.0046 (0.004656448232289335)  --> TIME: 2025-02-24 17:22:42 -- STEP: 7132/36938 -- GLOBAL_STEP: 413450 | > loss_text_ce: 0.03853372856974602 (0.03880042225577942) | > loss_mel_ce: 2.610513210296631 (nan) | > loss: 0.010512091219425201 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13667096793484437) | > loader_time: 0.0047 (0.004656371378992233)  --> TIME: 2025-02-24 17:22:51 -- STEP: 7182/36938 -- GLOBAL_STEP: 413500 | > loss_text_ce: 0.03624918684363365 (0.038799739847739455) | > loss_mel_ce: 1.9980863332748413 (nan) | > loss: 0.00807276088744402 (nan) | > current_lr: 5e-06 | > step_time: 0.1774 (0.13669806049913125) | > loader_time: 0.0048 (0.004656381342810076)  --> TIME: 2025-02-24 17:23:01 -- STEP: 7232/36938 -- GLOBAL_STEP: 413550 | > loss_text_ce: 0.042547523975372314 (0.03880660753711079) | > loss_mel_ce: 2.1001272201538086 (nan) | > loss: 0.008502678014338017 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13673497776014626) | > loader_time: 0.0046 (0.0046564091029947935)  --> TIME: 2025-02-24 17:23:10 -- STEP: 7282/36938 -- GLOBAL_STEP: 413600 | > loss_text_ce: 0.03303222730755806 (0.03880981633115335) | > loss_mel_ce: 2.64724063873291 (nan) | > loss: 0.01063600368797779 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13670977116024005) | > loader_time: 0.0046 (0.004656170135472912)  --> TIME: 2025-02-24 17:23:19 -- STEP: 7332/36938 -- GLOBAL_STEP: 413650 | > loss_text_ce: 0.0334121510386467 (0.038815222425382385) | > loss_mel_ce: 3.9460179805755615 (nan) | > loss: 0.0157913900911808 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13667694321110566) | > loader_time: 0.0046 (0.004655944865320926)  --> TIME: 2025-02-24 17:23:28 -- STEP: 7382/36938 -- GLOBAL_STEP: 413700 | > loss_text_ce: 0.037707410752773285 (0.038811426822841104) | > loss_mel_ce: 2.4256591796875 (nan) | > loss: 0.00977526418864727 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.13667059254756128) | > loader_time: 0.0048 (0.004656112959507012)  --> TIME: 2025-02-24 17:23:38 -- STEP: 7432/36938 -- GLOBAL_STEP: 413750 | > loss_text_ce: 0.045601677149534225 (0.03881255407034222) | > loss_mel_ce: 3.1967580318450928 (nan) | > loss: 0.012866507284343243 (nan) | > current_lr: 5e-06 | > step_time: 0.1618 (0.13672825100604133) | > loader_time: 0.0048 (0.004656166832721667)  --> TIME: 2025-02-24 17:23:47 -- STEP: 7482/36938 -- GLOBAL_STEP: 413800 | > loss_text_ce: 0.04874309524893761 (0.03881504495417331) | > loss_mel_ce: 1.1181776523590088 (nan) | > loss: 0.004630638286471367 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13671690316418225) | > loader_time: 0.0046 (0.004655804362599153)  --> TIME: 2025-02-24 17:23:56 -- STEP: 7532/36938 -- GLOBAL_STEP: 413850 | > loss_text_ce: 0.04388616606593132 (0.038822094949909125) | > loss_mel_ce: 3.7686941623687744 (nan) | > loss: 0.015129287727177143 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13670294153772639) | > loader_time: 0.0045 (0.004655318125979554)  --> TIME: 2025-02-24 17:24:06 -- STEP: 7582/36938 -- GLOBAL_STEP: 413900 | > loss_text_ce: 0.03614645078778267 (0.03883063165037124) | > loss_mel_ce: 0.101806640625 (nan) | > loss: 0.000547432922758162 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.1367156546778114) | > loader_time: 0.0044 (0.004654998453523147)  --> TIME: 2025-02-24 17:24:15 -- STEP: 7632/36938 -- GLOBAL_STEP: 413950 | > loss_text_ce: 0.03439446911215782 (0.038830442051208114) | > loss_mel_ce: 3.2222464084625244 (nan) | > loss: 0.012923179194331169 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.1366834649874729) | > loader_time: 0.0045 (0.004654331246012171)  --> TIME: 2025-02-24 17:24:24 -- STEP: 7682/36938 -- GLOBAL_STEP: 414000 | > loss_text_ce: 0.04053207486867905 (0.03882819920753071) | > loss_mel_ce: 3.490015745162964 (nan) | > loss: 0.014010111801326275 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13668338115312756) | > loader_time: 0.0046 (0.004653928522578266)  --> TIME: 2025-02-24 17:24:34 -- STEP: 7732/36938 -- GLOBAL_STEP: 414050 | > loss_text_ce: 0.037899408489465714 (0.0388293158023514) | > loss_mel_ce: 3.0475337505340576 (nan) | > loss: 0.012243783101439476 (nan) | > current_lr: 5e-06 | > step_time: 0.1664 (0.1366909967144721) | > loader_time: 0.0046 (0.004653577476477793)  --> TIME: 2025-02-24 17:24:43 -- STEP: 7782/36938 -- GLOBAL_STEP: 414100 | > loss_text_ce: 0.03071117028594017 (0.038833152175819105) | > loss_mel_ce: 3.5222251415252686 (nan) | > loss: 0.014098954387009144 (nan) | > current_lr: 5e-06 | > step_time: 0.1458 (0.1366786597719293) | > loader_time: 0.0047 (0.004653472147941353)  --> TIME: 2025-02-24 17:24:52 -- STEP: 7832/36938 -- GLOBAL_STEP: 414150 | > loss_text_ce: 0.04106825217604637 (0.03882892991999316) | > loss_mel_ce: 3.275914430618286 (nan) | > loss: 0.01316263061016798 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.1366871992341101) | > loader_time: 0.0046 (0.004653156990902669)  --> TIME: 2025-02-24 17:25:02 -- STEP: 7882/36938 -- GLOBAL_STEP: 414200 | > loss_text_ce: 0.04111069440841675 (0.0388196046345752) | > loss_mel_ce: 3.0184154510498047 (nan) | > loss: 0.012140978127717972 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13670473941477745) | > loader_time: 0.0049 (0.004653001309530116)  --> TIME: 2025-02-24 17:25:11 -- STEP: 7932/36938 -- GLOBAL_STEP: 414250 | > loss_text_ce: 0.04305381327867508 (0.0388201268777756) | > loss_mel_ce: 2.5092415809631348 (nan) | > loss: 0.010128157213330269 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13672278616323774) | > loader_time: 0.0045 (0.004653062233706054)  --> TIME: 2025-02-24 17:25:21 -- STEP: 7982/36938 -- GLOBAL_STEP: 414300 | > loss_text_ce: 0.03071015328168869 (0.03882251609867277) | > loss_mel_ce: 2.9760546684265137 (nan) | > loss: 0.011931607499718666 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13674327010530124) | > loader_time: 0.0046 (0.004653026961467831)  --> TIME: 2025-02-24 17:25:30 -- STEP: 8032/36938 -- GLOBAL_STEP: 414350 | > loss_text_ce: 0.03542295843362808 (0.038820496549740915) | > loss_mel_ce: 0.00946807861328125 (nan) | > loss: 0.00017813904560171068 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.1367563862665241) | > loader_time: 0.0045 (0.004653460683813142)  --> TIME: 2025-02-24 17:25:40 -- STEP: 8082/36938 -- GLOBAL_STEP: 414400 | > loss_text_ce: 0.03391677886247635 (0.03881514356862255) | > loss_mel_ce: 2.388336181640625 (nan) | > loss: 0.00961211510002613 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13677008610437247) | > loader_time: 0.0066 (0.0046540472773566426)  --> TIME: 2025-02-24 17:25:49 -- STEP: 8132/36938 -- GLOBAL_STEP: 414450 | > loss_text_ce: 0.0455322265625 (0.03882004698441618) | > loss_mel_ce: 2.4113361835479736 (nan) | > loss: 0.009749478660523891 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13675486587035615) | > loader_time: 0.0045 (0.004653863202042033)  --> TIME: 2025-02-24 17:25:58 -- STEP: 8182/36938 -- GLOBAL_STEP: 414500 | > loss_text_ce: 0.04743097722530365 (0.03881271194144859) | > loss_mel_ce: 3.4321796894073486 (nan) | > loss: 0.01380797941237688 (nan) | > current_lr: 5e-06 | > step_time: 0.1693 (0.1367562980705472) | > loader_time: 0.0048 (0.004653990079301178)  --> TIME: 2025-02-24 17:26:07 -- STEP: 8232/36938 -- GLOBAL_STEP: 414550 | > loss_text_ce: 0.037531424313783646 (0.038806968191647764) | > loss_mel_ce: 3.2260825634002686 (nan) | > loss: 0.01295084971934557 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.1367464308256774) | > loader_time: 0.0045 (0.004653709072645027)  --> TIME: 2025-02-24 17:26:17 -- STEP: 8282/36938 -- GLOBAL_STEP: 414600 | > loss_text_ce: 0.04512101039290428 (0.0388095355847934) | > loss_mel_ce: 3.1953585147857666 (nan) | > loss: 0.012859046459197998 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.1367622702284672) | > loader_time: 0.0046 (0.004653495626442685)  --> TIME: 2025-02-24 17:26:26 -- STEP: 8332/36938 -- GLOBAL_STEP: 414650 | > loss_text_ce: 0.04229619726538658 (0.03880254167033281) | > loss_mel_ce: 3.05078125 (nan) | > loss: 0.012274117209017277 (nan) | > current_lr: 5e-06 | > step_time: 0.1076 (0.13675040507396707) | > loader_time: 0.0046 (0.004653149851800393)  --> TIME: 2025-02-24 17:26:35 -- STEP: 8382/36938 -- GLOBAL_STEP: 414700 | > loss_text_ce: 0.03197651728987694 (0.038804576880216685) | > loss_mel_ce: 3.1050350666046143 (nan) | > loss: 0.012448458932340145 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13675382734056132) | > loader_time: 0.0046 (0.00465296327968226)  --> TIME: 2025-02-24 17:26:44 -- STEP: 8432/36938 -- GLOBAL_STEP: 414750 | > loss_text_ce: 0.03613775596022606 (0.038809983947568925) | > loss_mel_ce: 3.008657217025757 (nan) | > loss: 0.012082520872354507 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1367297892439297) | > loader_time: 0.0045 (0.004652814999703448)  --> TIME: 2025-02-24 17:26:54 -- STEP: 8482/36938 -- GLOBAL_STEP: 414800 | > loss_text_ce: 0.028252413496375084 (0.03880412096719418) | > loss_mel_ce: 3.026549816131592 (nan) | > loss: 0.012122231535613537 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.1367186111145721) | > loader_time: 0.0046 (0.004652452564441894)  --> TIME: 2025-02-24 17:27:03 -- STEP: 8532/36938 -- GLOBAL_STEP: 414850 | > loss_text_ce: 0.045031484216451645 (0.03880112177892778) | > loss_mel_ce: 2.487180709838867 (nan) | > loss: 0.010048462077975273 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13671345055187126) | > loader_time: 0.0045 (0.00465243227650625)  --> TIME: 2025-02-24 17:27:12 -- STEP: 8582/36938 -- GLOBAL_STEP: 414900 | > loss_text_ce: 0.039924029260873795 (0.03879775438979506) | > loss_mel_ce: 2.0642051696777344 (nan) | > loss: 0.008349720388650894 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1367086905528548) | > loader_time: 0.0045 (0.0046521286063193265)  --> TIME: 2025-02-24 17:27:21 -- STEP: 8632/36938 -- GLOBAL_STEP: 414950 | > loss_text_ce: 0.029896726831793785 (0.03879520546669105) | > loss_mel_ce: 2.718106269836426 (nan) | > loss: 0.010904774069786072 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13669504002910496) | > loader_time: 0.0046 (0.004651785255914723)  --> TIME: 2025-02-24 17:27:30 -- STEP: 8682/36938 -- GLOBAL_STEP: 415000 | > loss_text_ce: 0.03689145669341087 (0.038794765954949316) | > loss_mel_ce: 3.0671942234039307 (nan) | > loss: 0.012317800894379616 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.1366695988373228) | > loader_time: 0.0047 (0.004651427186598192)  --> TIME: 2025-02-24 17:27:40 -- STEP: 8732/36938 -- GLOBAL_STEP: 415050 | > loss_text_ce: 0.031943466514348984 (0.03879485078487921) | > loss_mel_ce: 1.704208493232727 (nan) | > loss: 0.0068894922733306885 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13667458317665349) | > loader_time: 0.0045 (0.004651162720469242)  --> TIME: 2025-02-24 17:27:49 -- STEP: 8782/36938 -- GLOBAL_STEP: 415100 | > loss_text_ce: 0.04188000410795212 (0.03878685371643588) | > loss_mel_ce: 3.4271528720855713 (nan) | > loss: 0.013766003772616386 (nan) | > current_lr: 5e-06 | > step_time: 0.1533 (0.13665592768175458) | > loader_time: 0.0068 (0.004651296630152222)  --> TIME: 2025-02-24 17:27:58 -- STEP: 8832/36938 -- GLOBAL_STEP: 415150 | > loss_text_ce: 0.04145572707056999 (0.038784418924478946) | > loss_mel_ce: 2.896487236022949 (nan) | > loss: 0.011658504605293274 (nan) | > current_lr: 5e-06 | > step_time: 0.1591 (0.13666796149766977) | > loader_time: 0.0047 (0.004651470271789521)  --> TIME: 2025-02-24 17:28:07 -- STEP: 8882/36938 -- GLOBAL_STEP: 415200 | > loss_text_ce: 0.03500929847359657 (0.03878068157417834) | > loss_mel_ce: 2.6621322631835938 (nan) | > loss: 0.010702943429350853 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13665301500207508) | > loader_time: 0.0046 (0.004651653474044555)  --> TIME: 2025-02-24 17:28:17 -- STEP: 8932/36938 -- GLOBAL_STEP: 415250 | > loss_text_ce: 0.04347538203001022 (0.03877934232997302) | > loss_mel_ce: 2.8192532062530518 (nan) | > loss: 0.011360034346580505 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13663020677968019) | > loader_time: 0.0046 (0.0046516058159899645)  --> TIME: 2025-02-24 17:28:26 -- STEP: 8982/36938 -- GLOBAL_STEP: 415300 | > loss_text_ce: 0.04237913340330124 (0.03878192776891676) | > loss_mel_ce: 3.115156412124634 (nan) | > loss: 0.012529904022812843 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1366171857291942) | > loader_time: 0.0046 (0.004651592213654687)  --> TIME: 2025-02-24 17:28:35 -- STEP: 9032/36938 -- GLOBAL_STEP: 415350 | > loss_text_ce: 0.026044292375445366 (0.03877843691215947) | > loss_mel_ce: 2.335374116897583 (nan) | > loss: 0.009370708838105202 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13659718165469475) | > loader_time: 0.0046 (0.004651447647540748)  --> TIME: 2025-02-24 17:28:44 -- STEP: 9082/36938 -- GLOBAL_STEP: 415400 | > loss_text_ce: 0.037532269954681396 (0.03877459283084662) | > loss_mel_ce: 3.164588451385498 (nan) | > loss: 0.012706829234957695 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13658183486260575) | > loader_time: 0.0049 (0.004651105684794107)  --> TIME: 2025-02-24 17:28:53 -- STEP: 9132/36938 -- GLOBAL_STEP: 415450 | > loss_text_ce: 0.036652930080890656 (0.038768129117919194) | > loss_mel_ce: 1.3780618906021118 (nan) | > loss: 0.005613948218524456 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.1365570866168723) | > loader_time: 0.0045 (0.004650746632499562)  --> TIME: 2025-02-24 17:29:02 -- STEP: 9182/36938 -- GLOBAL_STEP: 415500 | > loss_text_ce: 0.027193866670131683 (0.038765543079207904) | > loss_mel_ce: 2.456148624420166 (nan) | > loss: 0.009854533709585667 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.136571497549499) | > loader_time: 0.0046 (0.004650985745401134)  --> TIME: 2025-02-24 17:29:12 -- STEP: 9232/36938 -- GLOBAL_STEP: 415550 | > loss_text_ce: 0.0362086296081543 (0.03876491923323416) | > loss_mel_ce: 3.0270707607269287 (nan) | > loss: 0.012155870907008648 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13655754342752724) | > loader_time: 0.0048 (0.00465145508276116)  --> TIME: 2025-02-24 17:29:21 -- STEP: 9282/36938 -- GLOBAL_STEP: 415600 | > loss_text_ce: 0.03441442549228668 (0.03876190104310519) | > loss_mel_ce: 2.492288112640381 (nan) | > loss: 0.0100265983492136 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13654426754201687) | > loader_time: 0.0046 (0.004651368781683451)  --> TIME: 2025-02-24 17:29:30 -- STEP: 9332/36938 -- GLOBAL_STEP: 415650 | > loss_text_ce: 0.04114919528365135 (0.03876379155284436) | > loss_mel_ce: 0.025390625 (nan) | > loss: 0.0002640469465404749 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13655933299598125) | > loader_time: 0.0045 (0.004651379876518996)  --> TIME: 2025-02-24 17:29:39 -- STEP: 9382/36938 -- GLOBAL_STEP: 415700 | > loss_text_ce: 0.04140226170420647 (0.0387738302351634) | > loss_mel_ce: 3.242640256881714 (nan) | > loss: 0.0130319157615304 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.13655743075952048) | > loader_time: 0.0048 (0.004651321324934165)  --> TIME: 2025-02-24 17:29:49 -- STEP: 9432/36938 -- GLOBAL_STEP: 415750 | > loss_text_ce: 0.04643375426530838 (0.03877320729746666) | > loss_mel_ce: 2.189730167388916 (nan) | > loss: 0.008873666636645794 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1365713768332972) | > loader_time: 0.0046 (0.004651576685035914)  --> TIME: 2025-02-24 17:29:58 -- STEP: 9482/36938 -- GLOBAL_STEP: 415800 | > loss_text_ce: 0.02798883616924286 (0.0387643676797671) | > loss_mel_ce: 3.671961784362793 (nan) | > loss: 0.0146823450922966 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.13656274627668682) | > loader_time: 0.0048 (0.004651801894420673)  --> TIME: 2025-02-24 17:30:08 -- STEP: 9532/36938 -- GLOBAL_STEP: 415850 | > loss_text_ce: 0.029782697558403015 (0.038751794815148045) | > loss_mel_ce: 2.7581775188446045 (nan) | > loss: 0.011063335463404655 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13658483034414892) | > loader_time: 0.0046 (0.0046516775684701)  --> TIME: 2025-02-24 17:30:17 -- STEP: 9582/36938 -- GLOBAL_STEP: 415900 | > loss_text_ce: 0.040696363896131516 (0.03875410616043308) | > loss_mel_ce: 2.6642935276031494 (nan) | > loss: 0.01073408778756857 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13661337909009788) | > loader_time: 0.0046 (0.004651831202107926)  --> TIME: 2025-02-24 17:30:27 -- STEP: 9632/36938 -- GLOBAL_STEP: 415950 | > loss_text_ce: 0.03179578855633736 (0.03875561643169864) | > loss_mel_ce: 2.351590156555176 (nan) | > loss: 0.00945788063108921 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13660641591315287) | > loader_time: 0.0046 (0.004651740638916687)  --> TIME: 2025-02-24 17:30:36 -- STEP: 9682/36938 -- GLOBAL_STEP: 416000 | > loss_text_ce: 0.04894851893186569 (0.038752669140702876) | > loss_mel_ce: 2.770221710205078 (nan) | > loss: 0.011187183670699596 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.1366093243677939) | > loader_time: 0.0047 (0.004651792481329251)  --> TIME: 2025-02-24 17:30:45 -- STEP: 9732/36938 -- GLOBAL_STEP: 416050 | > loss_text_ce: 0.03840010240674019 (0.038760455758944545) | > loss_mel_ce: 3.1856095790863037 (nan) | > loss: 0.012793689966201782 (nan) | > current_lr: 5e-06 | > step_time: 0.1924 (0.13658844660301583) | > loader_time: 0.0049 (0.004651823040883442)  --> TIME: 2025-02-24 17:30:55 -- STEP: 9782/36938 -- GLOBAL_STEP: 416100 | > loss_text_ce: 0.04156494140625 (0.03875510976768847) | > loss_mel_ce: 3.008556842803955 (nan) | > loss: 0.01210365816950798 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.1365959624575345) | > loader_time: 0.0048 (0.004651815314596121)  --> TIME: 2025-02-24 17:31:04 -- STEP: 9832/36938 -- GLOBAL_STEP: 416150 | > loss_text_ce: 0.04545825347304344 (0.038761510299079316) | > loss_mel_ce: 2.8135287761688232 (nan) | > loss: 0.011345187202095985 (nan) | > current_lr: 5e-06 | > step_time: 0.1755 (0.1365987360962437) | > loader_time: 0.005 (0.004651989875720529)  --> TIME: 2025-02-24 17:31:13 -- STEP: 9882/36938 -- GLOBAL_STEP: 416200 | > loss_text_ce: 0.03070932626724243 (0.03875750096873339) | > loss_mel_ce: 2.52252197265625 (nan) | > loss: 0.01013187039643526 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13660037425866758) | > loader_time: 0.0045 (0.0046521251777289305)  --> TIME: 2025-02-24 17:31:23 -- STEP: 9932/36938 -- GLOBAL_STEP: 416250 | > loss_text_ce: 0.032313015311956406 (0.038754745394042604) | > loss_mel_ce: 2.373133420944214 (nan) | > loss: 0.009545423090457916 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13659460454327083) | > loader_time: 0.0046 (0.004651833747213922)  --> TIME: 2025-02-24 17:31:32 -- STEP: 9982/36938 -- GLOBAL_STEP: 416300 | > loss_text_ce: 0.04738590121269226 (0.0387572820043292) | > loss_mel_ce: 3.213775396347046 (nan) | > loss: 0.012941117398440838 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13659414980412063) | > loader_time: 0.0046 (0.004651710113383243)  --> TIME: 2025-02-24 17:31:41 -- STEP: 10032/36938 -- GLOBAL_STEP: 416350 | > loss_text_ce: 0.039914436638355255 (0.03875662505097898) | > loss_mel_ce: 2.9619784355163574 (nan) | > loss: 0.011912274174392223 (nan) | > current_lr: 5e-06 | > step_time: 0.1265 (0.13659998049671435) | > loader_time: 0.0045 (0.004651509760099179)  --> TIME: 2025-02-24 17:31:51 -- STEP: 10082/36938 -- GLOBAL_STEP: 416400 | > loss_text_ce: 0.03571387007832527 (0.03876832615753375) | > loss_mel_ce: 2.5348265171051025 (nan) | > loss: 0.010200558230280876 (nan) | > current_lr: 5e-06 | > step_time: 0.1844 (0.13659492065877726) | > loader_time: 0.0048 (0.004651468936660619)  --> TIME: 2025-02-24 17:32:00 -- STEP: 10132/36938 -- GLOBAL_STEP: 416450 | > loss_text_ce: 0.03954680636525154 (0.03877159456056585) | > loss_mel_ce: 3.7239582538604736 (nan) | > loss: 0.014934544451534748 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.1366189337746226) | > loader_time: 0.0048 (0.004651622366538137)  --> TIME: 2025-02-24 17:32:10 -- STEP: 10182/36938 -- GLOBAL_STEP: 416500 | > loss_text_ce: 0.039366260170936584 (0.0387686028768514) | > loss_mel_ce: 3.4752187728881836 (nan) | > loss: 0.013946766965091228 (nan) | > current_lr: 5e-06 | > step_time: 0.163 (0.1366219499728886) | > loader_time: 0.0047 (0.004651710224207819)  --> TIME: 2025-02-24 17:32:19 -- STEP: 10232/36938 -- GLOBAL_STEP: 416550 | > loss_text_ce: 0.03093121387064457 (0.03877144970456354) | > loss_mel_ce: 2.931933879852295 (nan) | > loss: 0.011757401749491692 (nan) | > current_lr: 5e-06 | > step_time: 0.1707 (0.13660266545715563) | > loader_time: 0.0048 (0.00465159058757273)  --> TIME: 2025-02-24 17:32:27 -- STEP: 10282/36938 -- GLOBAL_STEP: 416600 | > loss_text_ce: 0.0391450859606266 (0.038771580599474526) | > loss_mel_ce: 2.7880327701568604 (nan) | > loss: 0.011218960396945477 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13655878282015405) | > loader_time: 0.0045 (0.004651483244711369)  --> TIME: 2025-02-24 17:32:37 -- STEP: 10332/36938 -- GLOBAL_STEP: 416650 | > loss_text_ce: 0.037413742393255234 (0.03876903298517538) | > loss_mel_ce: 2.9214694499969482 (nan) | > loss: 0.011741600930690765 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13657441494126196) | > loader_time: 0.0046 (0.0046517745589176275)  --> TIME: 2025-02-24 17:32:47 -- STEP: 10382/36938 -- GLOBAL_STEP: 416700 | > loss_text_ce: 0.04769250378012657 (0.0387637491776256) | > loss_mel_ce: 3.28570818901062 (nan) | > loss: 0.013227781280875206 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.1365848735356656) | > loader_time: 0.0046 (0.004651739817321528)  --> TIME: 2025-02-24 17:32:56 -- STEP: 10432/36938 -- GLOBAL_STEP: 416750 | > loss_text_ce: 0.04547097906470299 (0.03876173663727274) | > loss_mel_ce: 0.141357421875 (nan) | > loss: 0.0007413825951516628 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13657212428794385) | > loader_time: 0.0045 (0.004651752946201287)  --> TIME: 2025-02-24 17:33:05 -- STEP: 10482/36938 -- GLOBAL_STEP: 416800 | > loss_text_ce: 0.0357191376388073 (0.03875794122395391) | > loss_mel_ce: 3.5504603385925293 (nan) | > loss: 0.014230871573090553 (nan) | > current_lr: 5e-06 | > step_time: 0.2024 (0.13657242331516722) | > loader_time: 0.005 (0.0046520316175458305)  --> TIME: 2025-02-24 17:33:14 -- STEP: 10532/36938 -- GLOBAL_STEP: 416850 | > loss_text_ce: 0.031106458976864815 (0.03875815374790865) | > loss_mel_ce: 3.425748586654663 (nan) | > loss: 0.013717679306864738 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13657229246859648) | > loader_time: 0.0045 (0.004652185173846003)  --> TIME: 2025-02-24 17:33:23 -- STEP: 10582/36938 -- GLOBAL_STEP: 416900 | > loss_text_ce: 0.04696531593799591 (0.03876648615623966) | > loss_mel_ce: 3.0493690967559814 (nan) | > loss: 0.01228704210370779 (nan) | > current_lr: 5e-06 | > step_time: 0.1743 (0.13655859039312476) | > loader_time: 0.0051 (0.004652113572926187)  --> TIME: 2025-02-24 17:33:33 -- STEP: 10632/36938 -- GLOBAL_STEP: 416950 | > loss_text_ce: 0.044602155685424805 (0.038767723170753075) | > loss_mel_ce: 2.7756550312042236 (nan) | > loss: 0.011191497556865215 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13656003267537287) | > loader_time: 0.0046 (0.004651974026110951)  --> TIME: 2025-02-24 17:33:42 -- STEP: 10682/36938 -- GLOBAL_STEP: 417000 | > loss_text_ce: 0.04085010290145874 (0.03876510577510352) | > loss_mel_ce: 2.808133602142334 (nan) | > loss: 0.01130549143999815 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13655017128019364) | > loader_time: 0.0045 (0.00465181835401686)  --> TIME: 2025-02-24 17:33:51 -- STEP: 10732/36938 -- GLOBAL_STEP: 417050 | > loss_text_ce: 0.03032671846449375 (0.03877010500107819) | > loss_mel_ce: 0.13936766982078552 (nan) | > loss: 0.0006733905174769461 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.1365347689894484) | > loader_time: 0.0045 (0.004651400121432214)  --> TIME: 2025-02-24 17:34:00 -- STEP: 10782/36938 -- GLOBAL_STEP: 417100 | > loss_text_ce: 0.037056077271699905 (0.038777259580287916) | > loss_mel_ce: 3.421935796737671 (nan) | > loss: 0.013726158998906612 (nan) | > current_lr: 5e-06 | > step_time: 0.1892 (0.13654251457742286) | > loader_time: 0.0049 (0.004651281436063268)  --> TIME: 2025-02-24 17:34:10 -- STEP: 10832/36938 -- GLOBAL_STEP: 417150 | > loss_text_ce: 0.03337051719427109 (0.03877593118176717) | > loss_mel_ce: 2.993802309036255 (nan) | > loss: 0.01201259158551693 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13656443333256058) | > loader_time: 0.0047 (0.004651539743034679)  --> TIME: 2025-02-24 17:34:20 -- STEP: 10882/36938 -- GLOBAL_STEP: 417200 | > loss_text_ce: 0.034878991544246674 (0.03877196711620039) | > loss_mel_ce: 2.8372483253479004 (nan) | > loss: 0.011397331021726131 (nan) | > current_lr: 5e-06 | > step_time: 0.1618 (0.13659193210798173) | > loader_time: 0.0047 (0.004651681287085794)  --> TIME: 2025-02-24 17:34:29 -- STEP: 10932/36938 -- GLOBAL_STEP: 417250 | > loss_text_ce: 0.04404054582118988 (0.03876963698378796) | > loss_mel_ce: 3.2238478660583496 (nan) | > loss: 0.012967811897397041 (nan) | > current_lr: 5e-06 | > step_time: 0.1609 (0.13656930606355677) | > loader_time: 0.0048 (0.004651748889800279)  --> TIME: 2025-02-24 17:34:38 -- STEP: 10982/36938 -- GLOBAL_STEP: 417300 | > loss_text_ce: 0.035273872315883636 (0.03877303414659054) | > loss_mel_ce: 2.80755352973938 (nan) | > loss: 0.011281060986220837 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.1365779125067391) | > loader_time: 0.0046 (0.004651579824741862)  --> TIME: 2025-02-24 17:34:48 -- STEP: 11032/36938 -- GLOBAL_STEP: 417350 | > loss_text_ce: 0.04919729754328728 (0.038777371081442566) | > loss_mel_ce: 3.1659371852874756 (nan) | > loss: 0.012758470140397549 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.13659196657664252) | > loader_time: 0.0048 (0.0046514198778331035)  --> TIME: 2025-02-24 17:34:57 -- STEP: 11082/36938 -- GLOBAL_STEP: 417400 | > loss_text_ce: 0.03946223855018616 (0.03877720132578036) | > loss_mel_ce: 1.94073486328125 (nan) | > loss: 0.00785792525857687 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.1365844252264004) | > loader_time: 0.0045 (0.004651153524350971)  --> TIME: 2025-02-24 17:35:06 -- STEP: 11132/36938 -- GLOBAL_STEP: 417450 | > loss_text_ce: 0.04280605539679527 (0.03877483807098249) | > loss_mel_ce: 3.2733030319213867 (nan) | > loss: 0.013159164227545261 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13658281924096685) | > loader_time: 0.0045 (0.004651119329498449)  --> TIME: 2025-02-24 17:35:16 -- STEP: 11182/36938 -- GLOBAL_STEP: 417500 | > loss_text_ce: 0.04241562262177467 (0.038774510798817566) | > loss_mel_ce: 3.309889078140259 (nan) | > loss: 0.013302797451615334 (nan) | > current_lr: 5e-06 | > step_time: 0.1659 (0.1366019714627028) | > loader_time: 0.0048 (0.004651231578969515)  --> TIME: 2025-02-24 17:35:25 -- STEP: 11232/36938 -- GLOBAL_STEP: 417550 | > loss_text_ce: 0.043689027428627014 (0.03877356189475236) | > loss_mel_ce: 3.2500147819519043 (nan) | > loss: 0.013070253655314445 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.13659249938111767) | > loader_time: 0.0046 (0.004651377471084282)  --> TIME: 2025-02-24 17:35:34 -- STEP: 11282/36938 -- GLOBAL_STEP: 417600 | > loss_text_ce: 0.014864634722471237 (0.03876696249481343) | > loss_mel_ce: 2.9523019790649414 (nan) | > loss: 0.011774471960961819 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.1365871483754282) | > loader_time: 0.0044 (0.004651084349439042)  --> TIME: 2025-02-24 17:35:44 -- STEP: 11332/36938 -- GLOBAL_STEP: 417650 | > loss_text_ce: 0.03134993091225624 (0.0387662404805765) | > loss_mel_ce: 3.3385989665985107 (nan) | > loss: 0.01337281335145235 (nan) | > current_lr: 5e-06 | > step_time: 0.1687 (0.1366171979921013) | > loader_time: 0.0048 (0.004650888765316921)  --> TIME: 2025-02-24 17:35:53 -- STEP: 11382/36938 -- GLOBAL_STEP: 417700 | > loss_text_ce: 0.037602748721838 (0.03877234948338179) | > loss_mel_ce: 3.2238540649414062 (nan) | > loss: 0.012942289002239704 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.1366083740538699) | > loader_time: 0.0046 (0.0046506005543394055)  --> TIME: 2025-02-24 17:36:03 -- STEP: 11432/36938 -- GLOBAL_STEP: 417750 | > loss_text_ce: 0.03681488335132599 (0.03877189717517201) | > loss_mel_ce: 3.2074496746063232 (nan) | > loss: 0.012874066829681396 (nan) | > current_lr: 5e-06 | > step_time: 0.1754 (0.1366536463366435) | > loader_time: 0.0048 (0.0046506269441942425)  --> TIME: 2025-02-24 17:36:12 -- STEP: 11482/36938 -- GLOBAL_STEP: 417800 | > loss_text_ce: 0.04207080602645874 (0.03876714212699888) | > loss_mel_ce: 3.2847378253936768 (nan) | > loss: 0.013201622292399406 (nan) | > current_lr: 5e-06 | > step_time: 0.1862 (0.13663727129155698) | > loader_time: 0.005 (0.004650358745931782)  --> TIME: 2025-02-24 17:36:22 -- STEP: 11532/36938 -- GLOBAL_STEP: 417850 | > loss_text_ce: 0.0393308661878109 (0.03876521857721883) | > loss_mel_ce: 2.3995540142059326 (nan) | > loss: 0.009678115136921406 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13664151106367067) | > loader_time: 0.0046 (0.004650123409624537)  --> TIME: 2025-02-24 17:36:30 -- STEP: 11582/36938 -- GLOBAL_STEP: 417900 | > loss_text_ce: 0.03860211372375488 (0.03876454687901352) | > loss_mel_ce: 3.4608054161071777 (nan) | > loss: 0.013886538334190845 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.13661449172916368) | > loader_time: 0.0047 (0.004649806899576489)  --> TIME: 2025-02-24 17:36:40 -- STEP: 11632/36938 -- GLOBAL_STEP: 417950 | > loss_text_ce: 0.03853415697813034 (0.03876790981952735) | > loss_mel_ce: 2.7843101024627686 (nan) | > loss: 0.011201763525605202 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13664568940169913) | > loader_time: 0.0046 (0.0046499389565794556)  --> TIME: 2025-02-24 17:36:50 -- STEP: 11682/36938 -- GLOBAL_STEP: 418000 | > loss_text_ce: 0.03568928688764572 (0.03876819676891649) | > loss_mel_ce: 2.3695077896118164 (nan) | > loss: 0.009544434025883675 (nan) | > current_lr: 5e-06 | > step_time: 0.1831 (0.13665094616029044) | > loader_time: 0.0047 (0.004649726459984509)  --> TIME: 2025-02-24 17:36:59 -- STEP: 11732/36938 -- GLOBAL_STEP: 418050 | > loss_text_ce: 0.03793945163488388 (0.038770249947360017) | > loss_mel_ce: 2.3723883628845215 (nan) | > loss: 0.009564793668687344 (nan) | > current_lr: 5e-06 | > step_time: 0.1058 (0.13664493859931978) | > loader_time: 0.0044 (0.004649498358627985)  --> TIME: 2025-02-24 17:37:08 -- STEP: 11782/36938 -- GLOBAL_STEP: 418100 | > loss_text_ce: 0.034879542887210846 (0.03877338181564246) | > loss_mel_ce: 3.007502555847168 (nan) | > loss: 0.012072945013642311 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.1366498577556311) | > loader_time: 0.0046 (0.004649561848994172)  --> TIME: 2025-02-24 17:37:17 -- STEP: 11832/36938 -- GLOBAL_STEP: 418150 | > loss_text_ce: 0.034452732652425766 (0.03877087312199847) | > loss_mel_ce: 2.6627509593963623 (nan) | > loss: 0.010703189298510551 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13663531004616844) | > loader_time: 0.0046 (0.004649587343317826)  --> TIME: 2025-02-24 17:37:27 -- STEP: 11882/36938 -- GLOBAL_STEP: 418200 | > loss_text_ce: 0.04251245781779289 (0.03877060197300343) | > loss_mel_ce: 3.168065309524536 (nan) | > loss: 0.01274038851261139 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13663242196739803) | > loader_time: 0.0045 (0.00464941200795067)  --> TIME: 2025-02-24 17:37:36 -- STEP: 11932/36938 -- GLOBAL_STEP: 418250 | > loss_text_ce: 0.04504460096359253 (0.03876533620679229) | > loss_mel_ce: 3.1110470294952393 (nan) | > loss: 0.012524174526333809 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13662528222413686) | > loader_time: 0.0048 (0.0046492150235088224)  --> TIME: 2025-02-24 17:37:45 -- STEP: 11982/36938 -- GLOBAL_STEP: 418300 | > loss_text_ce: 0.03656012937426567 (0.03877270813887414) | > loss_mel_ce: 3.12060546875 (nan) | > loss: 0.012528435327112675 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13663127555251847) | > loader_time: 0.0046 (0.00464948428292005)  --> TIME: 2025-02-24 17:37:54 -- STEP: 12032/36938 -- GLOBAL_STEP: 418350 | > loss_text_ce: 0.04710818827152252 (0.03877181961224973) | > loss_mel_ce: 3.428529977798462 (nan) | > loss: 0.013792215846478939 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13662279048498666) | > loader_time: 0.0046 (0.004649573560566349)  --> TIME: 2025-02-24 17:38:03 -- STEP: 12082/36938 -- GLOBAL_STEP: 418400 | > loss_text_ce: 0.03248405456542969 (0.03876684850252619) | > loss_mel_ce: 3.1756045818328857 (nan) | > loss: 0.012730510905385017 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.1366032819798445) | > loader_time: 0.0047 (0.004649875890614679)  --> TIME: 2025-02-24 17:38:13 -- STEP: 12132/36938 -- GLOBAL_STEP: 418450 | > loss_text_ce: 0.04019075259566307 (0.0387647951706081) | > loss_mel_ce: 3.7420644760131836 (nan) | > loss: 0.015008949674665928 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.1366005608676255) | > loader_time: 0.0046 (0.004649710270032371)  --> TIME: 2025-02-24 17:38:22 -- STEP: 12182/36938 -- GLOBAL_STEP: 418500 | > loss_text_ce: 0.0367569737136364 (0.038765344727249906) | > loss_mel_ce: 1.5905557870864868 (nan) | > loss: 0.006457590963691473 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1365858627606489) | > loader_time: 0.0046 (0.004649834393319817)  --> TIME: 2025-02-24 17:38:31 -- STEP: 12232/36938 -- GLOBAL_STEP: 418550 | > loss_text_ce: 0.036964550614356995 (0.038766138495659956) | > loss_mel_ce: 3.0722084045410156 (nan) | > loss: 0.012337989173829556 (nan) | > current_lr: 5e-06 | > step_time: 0.1786 (0.13657461614527705) | > loader_time: 0.0048 (0.004650252386976337)  --> TIME: 2025-02-24 17:38:40 -- STEP: 12282/36938 -- GLOBAL_STEP: 418600 | > loss_text_ce: 0.03481519967317581 (0.038760929558234423) | > loss_mel_ce: 3.515432357788086 (nan) | > loss: 0.014088285155594349 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1365694066752561) | > loader_time: 0.0046 (0.004650435081193708)  --> TIME: 2025-02-24 17:38:50 -- STEP: 12332/36938 -- GLOBAL_STEP: 418650 | > loss_text_ce: 0.033060189336538315 (0.03875850042878932) | > loss_mel_ce: 1.2804981470108032 (nan) | > loss: 0.005212533287703991 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13658326796575937) | > loader_time: 0.0045 (0.004650463096669391)  --> TIME: 2025-02-24 17:38:59 -- STEP: 12382/36938 -- GLOBAL_STEP: 418700 | > loss_text_ce: 0.02987666055560112 (0.03875605384850043) | > loss_mel_ce: 3.1114563941955566 (nan) | > loss: 0.012465608306229115 (nan) | > current_lr: 5e-06 | > step_time: 0.2018 (0.13658268966576237) | > loader_time: 0.005 (0.004650619164400472)  --> TIME: 2025-02-24 17:39:08 -- STEP: 12432/36938 -- GLOBAL_STEP: 418750 | > loss_text_ce: 0.036543652415275574 (0.03875898625984833) | > loss_mel_ce: 2.2054269313812256 (nan) | > loss: 0.00889670941978693 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1365770959562625) | > loader_time: 0.0045 (0.004650637852639311)  --> TIME: 2025-02-24 17:39:18 -- STEP: 12482/36938 -- GLOBAL_STEP: 418800 | > loss_text_ce: 0.043070238083601 (0.03876193988648341) | > loss_mel_ce: 0.0245361328125 (nan) | > loss: 0.000268279283773154 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13657359799175914) | > loader_time: 0.0045 (0.004650654920380279)  --> TIME: 2025-02-24 17:39:27 -- STEP: 12532/36938 -- GLOBAL_STEP: 418850 | > loss_text_ce: 0.03049306385219097 (0.03876216290615251) | > loss_mel_ce: 2.715181350708008 (nan) | > loss: 0.010895533487200737 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.1365723951799669) | > loader_time: 0.0046 (0.004650576518740936)  --> TIME: 2025-02-24 17:39:36 -- STEP: 12582/36938 -- GLOBAL_STEP: 418900 | > loss_text_ce: 0.029959945008158684 (0.038767358498312365) | > loss_mel_ce: 2.4367456436157227 (nan) | > loss: 0.0097885150462389 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13656581063998996) | > loader_time: 0.0045 (0.004650613534357689)  --> TIME: 2025-02-24 17:39:45 -- STEP: 12632/36938 -- GLOBAL_STEP: 418950 | > loss_text_ce: 0.034554317593574524 (0.038768010960975596) | > loss_mel_ce: 3.3255057334899902 (nan) | > loss: 0.013333572074770927 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.1365643767067872) | > loader_time: 0.0045 (0.0046505598873938184)  --> TIME: 2025-02-24 17:39:55 -- STEP: 12682/36938 -- GLOBAL_STEP: 419000 | > loss_text_ce: 0.05005275085568428 (0.038769372807046926) | > loss_mel_ce: 3.017594337463379 (nan) | > loss: 0.012173202820122242 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13655918892052954) | > loader_time: 0.0045 (0.004650607467769289)  --> TIME: 2025-02-24 17:40:04 -- STEP: 12732/36938 -- GLOBAL_STEP: 419050 | > loss_text_ce: 0.049458470195531845 (0.0387667971496399) | > loss_mel_ce: 3.0858254432678223 (nan) | > loss: 0.0124416034668684 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13656724392525127) | > loader_time: 0.0048 (0.004650462190572333)  --> TIME: 2025-02-24 17:40:13 -- STEP: 12782/36938 -- GLOBAL_STEP: 419100 | > loss_text_ce: 0.03907332196831703 (0.038770461990669226) | > loss_mel_ce: 2.591620445251465 (nan) | > loss: 0.010439260862767696 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13657355733813883) | > loader_time: 0.0048 (0.004650664803306936)  --> TIME: 2025-02-24 17:40:23 -- STEP: 12832/36938 -- GLOBAL_STEP: 419150 | > loss_text_ce: 0.016539329662919044 (0.03876804779405633) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13657265344165534) | > loader_time: 0.0047 (0.0046506995646436455)  --> TIME: 2025-02-24 17:40:32 -- STEP: 12882/36938 -- GLOBAL_STEP: 419200 | > loss_text_ce: 0.04604858160018921 (0.038757545145208565) | > loss_mel_ce: 1.7882808446884155 (nan) | > loss: 0.007279084995388985 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1365443353715142) | > loader_time: 0.0045 (0.0046506929316148474)  --> TIME: 2025-02-24 17:40:41 -- STEP: 12932/36938 -- GLOBAL_STEP: 419250 | > loss_text_ce: 0.029602913185954094 (0.038757660317742006) | > loss_mel_ce: 2.85591983795166 (nan) | > loss: 0.011450488120317459 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13653349065205436) | > loader_time: 0.0045 (0.004650631169951312)  --> TIME: 2025-02-24 17:40:50 -- STEP: 12982/36938 -- GLOBAL_STEP: 419300 | > loss_text_ce: 0.034866638481616974 (0.03875974542299926) | > loss_mel_ce: 3.1523146629333496 (nan) | > loss: 0.012647544965147972 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13652886925263769) | > loader_time: 0.0046 (0.004650535485792611)  --> TIME: 2025-02-24 17:40:59 -- STEP: 13032/36938 -- GLOBAL_STEP: 419350 | > loss_text_ce: 0.03339041769504547 (0.03876260602570889) | > loss_mel_ce: 3.2694616317749023 (nan) | > loss: 0.013106556609272957 (nan) | > current_lr: 5e-06 | > step_time: 0.1831 (0.1365302571165847) | > loader_time: 0.0049 (0.0046510520048709545)  --> TIME: 2025-02-24 17:41:09 -- STEP: 13082/36938 -- GLOBAL_STEP: 419400 | > loss_text_ce: 0.041909921914339066 (0.0387624815224427) | > loss_mel_ce: 3.6468098163604736 (nan) | > loss: 0.014637777581810951 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13652850347380985) | > loader_time: 0.0045 (0.004651113216929124)  --> TIME: 2025-02-24 17:41:18 -- STEP: 13132/36938 -- GLOBAL_STEP: 419450 | > loss_text_ce: 0.034904539585113525 (0.038759846694383406) | > loss_mel_ce: 2.7510712146759033 (nan) | > loss: 0.011055460199713707 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13650223219717836) | > loader_time: 0.0046 (0.004651045457931196)  --> TIME: 2025-02-24 17:41:27 -- STEP: 13182/36938 -- GLOBAL_STEP: 419500 | > loss_text_ce: 0.04119410738348961 (0.038753178086868914) | > loss_mel_ce: 3.172621488571167 (nan) | > loss: 0.012753237970173359 (nan) | > current_lr: 5e-06 | > step_time: 0.176 (0.1364950789357691) | > loader_time: 0.0049 (0.004651112560798336)  --> TIME: 2025-02-24 17:41:36 -- STEP: 13232/36938 -- GLOBAL_STEP: 419550 | > loss_text_ce: 0.025778992101550102 (0.038750387233202864) | > loss_mel_ce: 2.449110269546509 (nan) | > loss: 0.009820989333093166 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.1365033013536678) | > loader_time: 0.0046 (0.00465134935771109)  --> TIME: 2025-02-24 17:41:45 -- STEP: 13282/36938 -- GLOBAL_STEP: 419600 | > loss_text_ce: 0.029427720233798027 (0.038747540979070245) | > loss_mel_ce: 2.5122430324554443 (nan) | > loss: 0.010085996240377426 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13650635706518544) | > loader_time: 0.0048 (0.004651638097638143)  --> TIME: 2025-02-24 17:41:55 -- STEP: 13332/36938 -- GLOBAL_STEP: 419650 | > loss_text_ce: 0.03826915845274925 (0.038749729386890495) | > loss_mel_ce: 3.302960157394409 (nan) | > loss: 0.01325884647667408 (nan) | > current_lr: 5e-06 | > step_time: 0.1436 (0.13652946313794495) | > loader_time: 0.0046 (0.004651772020959739)  --> TIME: 2025-02-24 17:42:04 -- STEP: 13382/36938 -- GLOBAL_STEP: 419700 | > loss_text_ce: 0.030105922371149063 (0.03875539734943741) | > loss_mel_ce: 2.6973116397857666 (nan) | > loss: 0.010823085904121399 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13652130444298197) | > loader_time: 0.0047 (0.004652257956939697)  --> TIME: 2025-02-24 17:42:13 -- STEP: 13432/36938 -- GLOBAL_STEP: 419750 | > loss_text_ce: 0.03864559903740883 (0.038750779773763254) | > loss_mel_ce: 3.2689552307128906 (nan) | > loss: 0.013125400058925152 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.136510470479354) | > loader_time: 0.0047 (0.0046522602785052675)  --> TIME: 2025-02-24 17:42:23 -- STEP: 13482/36938 -- GLOBAL_STEP: 419800 | > loss_text_ce: 0.039327286183834076 (0.03874694699329794) | > loss_mel_ce: 3.3432700634002686 (nan) | > loss: 0.013423006050288677 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13652142958831331) | > loader_time: 0.0049 (0.004652669337467466)  --> TIME: 2025-02-24 17:42:32 -- STEP: 13532/36938 -- GLOBAL_STEP: 419850 | > loss_text_ce: 0.0456341877579689 (0.03874607956893924) | > loss_mel_ce: 3.219151258468628 (nan) | > loss: 0.01295549888163805 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1365159362898117) | > loader_time: 0.0045 (0.004652588775515252)  --> TIME: 2025-02-24 17:42:41 -- STEP: 13582/36938 -- GLOBAL_STEP: 419900 | > loss_text_ce: 0.0400177426636219 (0.03874847636773569) | > loss_mel_ce: 3.576977014541626 (nan) | > loss: 0.01435315515846014 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.13652166337577906) | > loader_time: 0.0049 (0.004652559590469573)  --> TIME: 2025-02-24 17:42:51 -- STEP: 13632/36938 -- GLOBAL_STEP: 419950 | > loss_text_ce: 0.03526625782251358 (0.03874704728886533) | > loss_mel_ce: 3.408177137374878 (nan) | > loss: 0.013664457947015762 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.1365205265686543) | > loader_time: 0.0046 (0.004652385613028444)  --> TIME: 2025-02-24 17:43:00 -- STEP: 13682/36938 -- GLOBAL_STEP: 420000 | > loss_text_ce: 0.026139693334698677 (0.03874160268536899) | > loss_mel_ce: 2.198773145675659 (nan) | > loss: 0.008829019963741302 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13650541386745793) | > loader_time: 0.0046 (0.004652232284557062) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_420000.pth  --> TIME: 2025-02-24 17:43:15 -- STEP: 13732/36938 -- GLOBAL_STEP: 420050 | > loss_text_ce: 0.04131854325532913 (0.03874050191128315) | > loss_mel_ce: 0.8069773316383362 (nan) | > loss: 0.0033662535715848207 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13650174787500793) | > loader_time: 0.0048 (0.004652071999211799)  --> TIME: 2025-02-24 17:43:24 -- STEP: 13782/36938 -- GLOBAL_STEP: 420100 | > loss_text_ce: 0.04148601368069649 (0.03873628112309778) | > loss_mel_ce: 1.4958621263504028 (nan) | > loss: 0.0061005884781479836 (nan) | > current_lr: 5e-06 | > step_time: 0.1069 (0.13648327876721927) | > loader_time: 0.0044 (0.004652179233776343)  --> TIME: 2025-02-24 17:43:33 -- STEP: 13832/36938 -- GLOBAL_STEP: 420150 | > loss_text_ce: 0.03829707205295563 (0.03873612577783218) | > loss_mel_ce: 2.7205116748809814 (nan) | > loss: 0.010947654955089092 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.136464182147406) | > loader_time: 0.0045 (0.004651911793725912)  --> TIME: 2025-02-24 17:43:43 -- STEP: 13882/36938 -- GLOBAL_STEP: 420200 | > loss_text_ce: 0.04088601469993591 (0.03873649181528679) | > loss_mel_ce: 3.1228139400482178 (nan) | > loss: 0.012554365210235119 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13646313151782874) | > loader_time: 0.0046 (0.0046517232570090486)  --> TIME: 2025-02-24 17:43:52 -- STEP: 13932/36938 -- GLOBAL_STEP: 420250 | > loss_text_ce: 0.0397321917116642 (0.03873794402761651) | > loss_mel_ce: 3.9085819721221924 (nan) | > loss: 0.015667913481593132 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.1364687741607076) | > loader_time: 0.0047 (0.004651980310413243)  --> TIME: 2025-02-24 17:44:02 -- STEP: 13982/36938 -- GLOBAL_STEP: 420300 | > loss_text_ce: 0.033625345677137375 (0.03874197902990251) | > loss_mel_ce: 2.7429616451263428 (nan) | > loss: 0.011018202640116215 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13649251186153663) | > loader_time: 0.0047 (0.004652489392888045)  --> TIME: 2025-02-24 17:44:11 -- STEP: 14032/36938 -- GLOBAL_STEP: 420350 | > loss_text_ce: 0.03536757081747055 (0.038738100368095815) | > loss_mel_ce: 2.9411447048187256 (nan) | > loss: 0.011811557225883007 (nan) | > current_lr: 5e-06 | > step_time: 0.1774 (0.13648704027368366) | > loader_time: 0.0048 (0.004652442535897357)  --> TIME: 2025-02-24 17:44:20 -- STEP: 14082/36938 -- GLOBAL_STEP: 420400 | > loss_text_ce: 0.0440993569791317 (0.03873786073583602) | > loss_mel_ce: 3.099522829055786 (nan) | > loss: 0.01247469149529934 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.1364695389252834) | > loader_time: 0.0046 (0.004652395063529496)  --> TIME: 2025-02-24 17:44:29 -- STEP: 14132/36938 -- GLOBAL_STEP: 420450 | > loss_text_ce: 0.04379330575466156 (0.03873635066994502) | > loss_mel_ce: 2.6962194442749023 (nan) | > loss: 0.010873066261410713 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.1364553361549073) | > loader_time: 0.0047 (0.0046522557786055725)  --> TIME: 2025-02-24 17:44:39 -- STEP: 14182/36938 -- GLOBAL_STEP: 420500 | > loss_text_ce: 0.04178837314248085 (0.038737168171191064) | > loss_mel_ce: 3.5661253929138184 (nan) | > loss: 0.014317118562757969 (nan) | > current_lr: 5e-06 | > step_time: 0.1567 (0.13647579170050045) | > loader_time: 0.0051 (0.004652458779853227)  --> TIME: 2025-02-24 17:44:49 -- STEP: 14232/36938 -- GLOBAL_STEP: 420550 | > loss_text_ce: 0.04232858121395111 (0.03873869975731191) | > loss_mel_ce: 1.4273681640625 (nan) | > loss: 0.005832130555063486 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13650762430062904) | > loader_time: 0.0047 (0.004652617418611913)  --> TIME: 2025-02-24 17:44:58 -- STEP: 14282/36938 -- GLOBAL_STEP: 420600 | > loss_text_ce: 0.04178408160805702 (0.03874286454873385) | > loss_mel_ce: 3.468752145767212 (nan) | > loss: 0.013930699788033962 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1365227270754022) | > loader_time: 0.0046 (0.00465271167771154)  --> TIME: 2025-02-24 17:45:07 -- STEP: 14332/36938 -- GLOBAL_STEP: 420650 | > loss_text_ce: 0.047347161918878555 (0.03873688513131055) | > loss_mel_ce: 1.377907156944275 (nan) | > loss: 0.005655771587044001 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13651618854712735) | > loader_time: 0.0045 (0.004652559807138883)  --> TIME: 2025-02-24 17:45:17 -- STEP: 14382/36938 -- GLOBAL_STEP: 420700 | > loss_text_ce: 0.04863293841481209 (0.03873580101336872) | > loss_mel_ce: 3.1581497192382812 (nan) | > loss: 0.012725328095257282 (nan) | > current_lr: 5e-06 | > step_time: 0.1695 (0.1365140907921257) | > loader_time: 0.0048 (0.004652503716764965)  --> TIME: 2025-02-24 17:45:26 -- STEP: 14432/36938 -- GLOBAL_STEP: 420750 | > loss_text_ce: 0.03722972795367241 (0.038729290224515274) | > loss_mel_ce: 1.6359723806381226 (nan) | > loss: 0.006639691535383463 (nan) | > current_lr: 5e-06 | > step_time: 0.1438 (0.13650204644367056) | > loader_time: 0.0046 (0.004652282747619708)  --> TIME: 2025-02-24 17:45:35 -- STEP: 14482/36938 -- GLOBAL_STEP: 420800 | > loss_text_ce: 0.034288037568330765 (0.03873114992158709) | > loss_mel_ce: 2.894296884536743 (nan) | > loss: 0.011621369048953056 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.13650982868208786) | > loader_time: 0.0046 (0.004652120727578059)  --> TIME: 2025-02-24 17:45:44 -- STEP: 14532/36938 -- GLOBAL_STEP: 420850 | > loss_text_ce: 0.03919155150651932 (0.038729037390124676) | > loss_mel_ce: 2.6025826930999756 (nan) | > loss: 0.010483231395483017 (nan) | > current_lr: 5e-06 | > step_time: 0.1578 (0.13649730924532266) | > loader_time: 0.0047 (0.004652045595393257)  --> TIME: 2025-02-24 17:45:54 -- STEP: 14582/36938 -- GLOBAL_STEP: 420900 | > loss_text_ce: 0.02745361253619194 (0.038724603281311616) | > loss_mel_ce: 3.4795660972595215 (nan) | > loss: 0.013916745781898499 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13650542151828185) | > loader_time: 0.0045 (0.004652184413925097)  --> TIME: 2025-02-24 17:46:03 -- STEP: 14632/36938 -- GLOBAL_STEP: 420950 | > loss_text_ce: 0.05292801558971405 (0.038725170774241285) | > loss_mel_ce: 1.8427810668945312 (nan) | > loss: 0.007522655185312033 (nan) | > current_lr: 5e-06 | > step_time: 0.1121 (0.13649486426514545) | > loader_time: 0.0047 (0.004652166721793469)  --> TIME: 2025-02-24 17:46:12 -- STEP: 14682/36938 -- GLOBAL_STEP: 421000 | > loss_text_ce: 0.03561518341302872 (0.038726249271340316) | > loss_mel_ce: 0.9489352107048035 (nan) | > loss: 0.003906946163624525 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13648412741102556) | > loader_time: 0.0046 (0.00465214249224158)  --> TIME: 2025-02-24 17:46:21 -- STEP: 14732/36938 -- GLOBAL_STEP: 421050 | > loss_text_ce: 0.03948401287198067 (0.038730070381565715) | > loss_mel_ce: 3.2416832447052 (nan) | > loss: 0.013020506128668785 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13648336145044665) | > loader_time: 0.0045 (0.004652240355313241)  --> TIME: 2025-02-24 17:46:31 -- STEP: 14782/36938 -- GLOBAL_STEP: 421100 | > loss_text_ce: 0.03633233532309532 (0.0387305100759039) | > loss_mel_ce: 2.725348472595215 (nan) | > loss: 0.010959051549434662 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13648222223732664) | > loader_time: 0.0046 (0.004652365169156127)  --> TIME: 2025-02-24 17:46:40 -- STEP: 14832/36938 -- GLOBAL_STEP: 421150 | > loss_text_ce: 0.036447130143642426 (0.038727451105172145) | > loss_mel_ce: 3.5113365650177 (nan) | > loss: 0.014078507199883461 (nan) | > current_lr: 5e-06 | > step_time: 0.1582 (0.13648384037912833) | > loader_time: 0.0048 (0.004652490331002693)  --> TIME: 2025-02-24 17:46:49 -- STEP: 14882/36938 -- GLOBAL_STEP: 421200 | > loss_text_ce: 0.038318827748298645 (0.03872305461557375) | > loss_mel_ce: 2.8022608757019043 (nan) | > loss: 0.011272142641246319 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1364785670401058) | > loader_time: 0.0045 (0.00465249119707213)  --> TIME: 2025-02-24 17:46:58 -- STEP: 14932/36938 -- GLOBAL_STEP: 421250 | > loss_text_ce: 0.023391298949718475 (0.038717825774541546) | > loss_mel_ce: 2.596660852432251 (nan) | > loss: 0.010397032834589481 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13648070057690892) | > loader_time: 0.0047 (0.004652511856366262)  --> TIME: 2025-02-24 17:47:08 -- STEP: 14982/36938 -- GLOBAL_STEP: 421300 | > loss_text_ce: 0.049977224320173264 (0.038721969731857285) | > loss_mel_ce: 3.1501152515411377 (nan) | > loss: 0.012698780745267868 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13648399770729303) | > loader_time: 0.0047 (0.004652556455146496)  --> TIME: 2025-02-24 17:47:17 -- STEP: 15032/36938 -- GLOBAL_STEP: 421350 | > loss_text_ce: 0.034082137048244476 (0.038719344433737125) | > loss_mel_ce: 3.54142689704895 (nan) | > loss: 0.01418852899223566 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13647154655591037) | > loader_time: 0.0046 (0.00465248805084656)  --> TIME: 2025-02-24 17:47:27 -- STEP: 15082/36938 -- GLOBAL_STEP: 421400 | > loss_text_ce: 0.03654824197292328 (0.03871724832365328) | > loss_mel_ce: 3.2175629138946533 (nan) | > loss: 0.012913139536976814 (nan) | > current_lr: 5e-06 | > step_time: 0.1664 (0.13649462153370628) | > loader_time: 0.0067 (0.004652925581629195)  --> TIME: 2025-02-24 17:47:36 -- STEP: 15132/36938 -- GLOBAL_STEP: 421450 | > loss_text_ce: 0.030570102855563164 (0.0387155813329462) | > loss_mel_ce: 3.158936023712158 (nan) | > loss: 0.012656770646572113 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13649742101633508) | > loader_time: 0.0047 (0.004653072927542044)  --> TIME: 2025-02-24 17:47:46 -- STEP: 15182/36938 -- GLOBAL_STEP: 421500 | > loss_text_ce: 0.041921794414520264 (0.038715894286906194) | > loss_mel_ce: 2.4552483558654785 (nan) | > loss: 0.009909406304359436 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13650818971003986) | > loader_time: 0.0047 (0.004653476016698292)  --> TIME: 2025-02-24 17:47:55 -- STEP: 15232/36938 -- GLOBAL_STEP: 421550 | > loss_text_ce: 0.0279095396399498 (0.03871721462305042) | > loss_mel_ce: 0.025390625 (nan) | > loss: 0.00021150859538465738 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13651805087736169) | > loader_time: 0.0046 (0.0046537090718996735)  --> TIME: 2025-02-24 17:48:04 -- STEP: 15282/36938 -- GLOBAL_STEP: 421600 | > loss_text_ce: 0.04217269644141197 (0.03871841361435878) | > loss_mel_ce: 2.4229960441589355 (nan) | > loss: 0.009782415814697742 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.13651918910932292) | > loader_time: 0.0047 (0.004654005487744318)  --> TIME: 2025-02-24 17:48:14 -- STEP: 15332/36938 -- GLOBAL_STEP: 421650 | > loss_text_ce: 0.031389158219099045 (0.038712874775025496) | > loss_mel_ce: 0.69140625 (nan) | > loss: 0.002868236042559147 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13652892965515315) | > loader_time: 0.0046 (0.004654276240378933)  --> TIME: 2025-02-24 17:48:23 -- STEP: 15382/36938 -- GLOBAL_STEP: 421700 | > loss_text_ce: 0.035203538835048676 (0.03871151205292019) | > loss_mel_ce: 3.0062646865844727 (nan) | > loss: 0.012069318443536758 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.1365202583179813) | > loader_time: 0.0048 (0.004654490905873825)  --> TIME: 2025-02-24 17:48:32 -- STEP: 15432/36938 -- GLOBAL_STEP: 421750 | > loss_text_ce: 0.04407636076211929 (0.03871004079402651) | > loss_mel_ce: 3.2162253856658936 (nan) | > loss: 0.012937705963850021 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.136505394741437) | > loader_time: 0.0047 (0.004654709124206704)  --> TIME: 2025-02-24 17:48:41 -- STEP: 15482/36938 -- GLOBAL_STEP: 421800 | > loss_text_ce: 0.04098818823695183 (0.038716325581546876) | > loss_mel_ce: 3.236562728881836 (nan) | > loss: 0.013006155379116535 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13649435756958322) | > loader_time: 0.0048 (0.004655028787821623)  --> TIME: 2025-02-24 17:48:50 -- STEP: 15532/36938 -- GLOBAL_STEP: 421850 | > loss_text_ce: 0.05018554627895355 (0.03871603203592518) | > loss_mel_ce: 2.8264036178588867 (nan) | > loss: 0.011415036395192146 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13648372417315355) | > loader_time: 0.0049 (0.004655099025621211)  --> TIME: 2025-02-24 17:49:00 -- STEP: 15582/36938 -- GLOBAL_STEP: 421900 | > loss_text_ce: 0.040695976465940475 (0.03871297172794656) | > loss_mel_ce: 2.907626152038574 (nan) | > loss: 0.011699691414833069 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13649043792356894) | > loader_time: 0.0047 (0.004655095398953948)  --> TIME: 2025-02-24 17:49:09 -- STEP: 15632/36938 -- GLOBAL_STEP: 421950 | > loss_text_ce: 0.05225103348493576 (0.038715536789439606) | > loss_mel_ce: 3.5153729915618896 (nan) | > loss: 0.014157239347696304 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1364744063714631) | > loader_time: 0.0045 (0.004655225707662275)  --> TIME: 2025-02-24 17:49:18 -- STEP: 15682/36938 -- GLOBAL_STEP: 422000 | > loss_text_ce: 0.043552983552217484 (0.038717106264286545) | > loss_mel_ce: 1.8101863861083984 (nan) | > loss: 0.007356109097599983 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13648986489534706) | > loader_time: 0.0045 (0.004655297975305433)  --> TIME: 2025-02-24 17:49:28 -- STEP: 15732/36938 -- GLOBAL_STEP: 422050 | > loss_text_ce: 0.038338325917720795 (0.03871875322366339) | > loss_mel_ce: 2.6294913291931152 (nan) | > loss: 0.010586626827716827 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13649042674733147) | > loader_time: 0.0046 (0.004655487022719987)  --> TIME: 2025-02-24 17:49:37 -- STEP: 15782/36938 -- GLOBAL_STEP: 422100 | > loss_text_ce: 0.04274166002869606 (0.03872046292433857) | > loss_mel_ce: 2.72806453704834 (nan) | > loss: 0.010995264165103436 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13648538276158478) | > loader_time: 0.0047 (0.004655467377704848)  --> TIME: 2025-02-24 17:49:46 -- STEP: 15832/36938 -- GLOBAL_STEP: 422150 | > loss_text_ce: 0.04096117988228798 (0.03872563462046052) | > loss_mel_ce: 3.5935871601104736 (nan) | > loss: 0.01442281249910593 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13648013634293768) | > loader_time: 0.0046 (0.004655464437045262)  --> TIME: 2025-02-24 17:49:55 -- STEP: 15882/36938 -- GLOBAL_STEP: 422200 | > loss_text_ce: 0.031132521107792854 (0.03872746359786788) | > loss_mel_ce: 2.3284099102020264 (nan) | > loss: 0.009363263845443726 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13646680788434787) | > loader_time: 0.0046 (0.004655412230919788)  --> TIME: 2025-02-24 17:50:04 -- STEP: 15932/36938 -- GLOBAL_STEP: 422250 | > loss_text_ce: 0.04438182711601257 (0.0387311543749766) | > loss_mel_ce: 2.7705776691436768 (nan) | > loss: 0.01117047481238842 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13645791438785349) | > loader_time: 0.0046 (0.0046554042291557535)  --> TIME: 2025-02-24 17:50:14 -- STEP: 15982/36938 -- GLOBAL_STEP: 422300 | > loss_text_ce: 0.04029955714941025 (0.03872955084127228) | > loss_mel_ce: 0.02136739157140255 (nan) | > loss: 0.00024471012875437737 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13646653609704854) | > loader_time: 0.0049 (0.004655537073013418)  --> TIME: 2025-02-24 17:50:23 -- STEP: 16032/36938 -- GLOBAL_STEP: 422350 | > loss_text_ce: 0.04288895055651665 (0.03872771734723124) | > loss_mel_ce: 0.03167724609375 (nan) | > loss: 0.0002958976256195456 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13647405792793585) | > loader_time: 0.0045 (0.004655492891689509)  --> TIME: 2025-02-24 17:50:32 -- STEP: 16082/36938 -- GLOBAL_STEP: 422400 | > loss_text_ce: 0.0407368429005146 (0.03872680831471241) | > loss_mel_ce: 1.822265625 (nan) | > loss: 0.007392867002636194 (nan) | > current_lr: 5e-06 | > step_time: 0.1067 (0.13645190062058557) | > loader_time: 0.0044 (0.004655548862342954)  --> TIME: 2025-02-24 17:50:42 -- STEP: 16132/36938 -- GLOBAL_STEP: 422450 | > loss_text_ce: 0.043252166360616684 (0.03872978892833177) | > loss_mel_ce: 3.1449246406555176 (nan) | > loss: 0.012651496566832066 (nan) | > current_lr: 5e-06 | > step_time: 0.1809 (0.13645641942075065) | > loader_time: 0.0048 (0.004655777683868154)  --> TIME: 2025-02-24 17:50:51 -- STEP: 16182/36938 -- GLOBAL_STEP: 422500 | > loss_text_ce: 0.03523245453834534 (0.03873014628982489) | > loss_mel_ce: 3.8966097831726074 (nan) | > loss: 0.01560254953801632 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.13646270450563275) | > loader_time: 0.0047 (0.0046559561317009255)  --> TIME: 2025-02-24 17:51:00 -- STEP: 16232/36938 -- GLOBAL_STEP: 422550 | > loss_text_ce: 0.039486587047576904 (0.03873066313081908) | > loss_mel_ce: 2.98529314994812 (nan) | > loss: 0.012003094889223576 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13646688970689877) | > loader_time: 0.0046 (0.004655830374840399)  --> TIME: 2025-02-24 17:51:10 -- STEP: 16282/36938 -- GLOBAL_STEP: 422600 | > loss_text_ce: 0.029230903834104538 (0.03872853757061061) | > loss_mel_ce: 2.948723793029785 (nan) | > loss: 0.011817281134426594 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.1364683152826569) | > loader_time: 0.0047 (0.004655975965123385)  --> TIME: 2025-02-24 17:51:19 -- STEP: 16332/36938 -- GLOBAL_STEP: 422650 | > loss_text_ce: 0.046181000769138336 (0.03872574963649097) | > loss_mel_ce: 3.20173978805542 (nan) | > loss: 0.012888574972748756 (nan) | > current_lr: 5e-06 | > step_time: 0.1713 (0.13646487828536616) | > loader_time: 0.0047 (0.0046558722893805865)  --> TIME: 2025-02-24 17:51:28 -- STEP: 16382/36938 -- GLOBAL_STEP: 422700 | > loss_text_ce: 0.04089207947254181 (0.03871993476711651) | > loss_mel_ce: 2.731111764907837 (nan) | > loss: 0.011000015772879124 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13646215474761278) | > loader_time: 0.0045 (0.0046558009589959945)  --> TIME: 2025-02-24 17:51:37 -- STEP: 16432/36938 -- GLOBAL_STEP: 422750 | > loss_text_ce: 0.045185547322034836 (0.03872524560898449) | > loss_mel_ce: 1.7694838047027588 (nan) | > loss: 0.007201069500297308 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1364428341359683) | > loader_time: 0.0048 (0.004655842699233848)  --> TIME: 2025-02-24 17:51:46 -- STEP: 16482/36938 -- GLOBAL_STEP: 422800 | > loss_text_ce: 0.038729432970285416 (0.03872498986783082) | > loss_mel_ce: 3.3901774883270264 (nan) | > loss: 0.013606774620711803 (nan) | > current_lr: 5e-06 | > step_time: 0.1904 (0.1364361778536318) | > loader_time: 0.0049 (0.004655821493218479)  --> TIME: 2025-02-24 17:51:56 -- STEP: 16532/36938 -- GLOBAL_STEP: 422850 | > loss_text_ce: 0.03339696675539017 (0.03872413464269575) | > loss_mel_ce: 2.9320080280303955 (nan) | > loss: 0.01176748052239418 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.13645628344888633) | > loader_time: 0.0046 (0.004655940867843287)  --> TIME: 2025-02-24 17:52:06 -- STEP: 16582/36938 -- GLOBAL_STEP: 422900 | > loss_text_ce: 0.038079168647527695 (0.03872221186887394) | > loss_mel_ce: 2.9316325187683105 (nan) | > loss: 0.011784571222960949 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13647199966783558) | > loader_time: 0.0046 (0.004656003160192582)  --> TIME: 2025-02-24 17:52:15 -- STEP: 16632/36938 -- GLOBAL_STEP: 422950 | > loss_text_ce: 0.04000288248062134 (0.038721217859416394) | > loss_mel_ce: 3.4173007011413574 (nan) | > loss: 0.013719459064304829 (nan) | > current_lr: 5e-06 | > step_time: 0.1786 (0.1364718434929905) | > loader_time: 0.0048 (0.00465607435270937)  --> TIME: 2025-02-24 17:52:24 -- STEP: 16682/36938 -- GLOBAL_STEP: 423000 | > loss_text_ce: 0.0375140979886055 (0.03871919350220038) | > loss_mel_ce: 3.4540796279907227 (nan) | > loss: 0.013855531811714172 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13647206281818172) | > loader_time: 0.0045 (0.004656018820358924)  --> TIME: 2025-02-24 17:52:34 -- STEP: 16732/36938 -- GLOBAL_STEP: 423050 | > loss_text_ce: 0.04112417995929718 (0.038714513469072474) | > loss_mel_ce: 3.603579521179199 (nan) | > loss: 0.014463110826909542 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.13647352836931723) | > loader_time: 0.0049 (0.004656029308979355)  --> TIME: 2025-02-24 17:52:43 -- STEP: 16782/36938 -- GLOBAL_STEP: 423100 | > loss_text_ce: 0.031998928636312485 (0.03871390850200745) | > loss_mel_ce: 3.2471892833709717 (nan) | > loss: 0.013012652285397053 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13646773335306037) | > loader_time: 0.0046 (0.004656004303332218)  --> TIME: 2025-02-24 17:52:52 -- STEP: 16832/36938 -- GLOBAL_STEP: 423150 | > loss_text_ce: 0.03770839795470238 (0.038708901509034895) | > loss_mel_ce: 2.6277971267700195 (nan) | > loss: 0.010577403008937836 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.1364678020939629) | > loader_time: 0.0045 (0.00465595981410702)  --> TIME: 2025-02-24 17:53:02 -- STEP: 16882/36938 -- GLOBAL_STEP: 423200 | > loss_text_ce: 0.04529048874974251 (0.03870918622521556) | > loss_mel_ce: 3.3597500324249268 (nan) | > loss: 0.013512066565454006 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.1364794274159426) | > loader_time: 0.0051 (0.004656028668471765)  --> TIME: 2025-02-24 17:53:11 -- STEP: 16932/36938 -- GLOBAL_STEP: 423250 | > loss_text_ce: 0.040649838745594025 (0.0387078432452388) | > loss_mel_ce: 3.2784578800201416 (nan) | > loss: 0.013171063736081123 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13646650960646495) | > loader_time: 0.0047 (0.0046560533244419024)  --> TIME: 2025-02-24 17:53:20 -- STEP: 16982/36938 -- GLOBAL_STEP: 423300 | > loss_text_ce: 0.03160254657268524 (0.03870628834047817) | > loss_mel_ce: 3.03271484375 (nan) | > loss: 0.012159991078078747 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13645670061860615) | > loader_time: 0.0045 (0.004655987252448939)  --> TIME: 2025-02-24 17:53:30 -- STEP: 17032/36938 -- GLOBAL_STEP: 423350 | > loss_text_ce: 0.05119403824210167 (0.038706912607763454) | > loss_mel_ce: 3.4454376697540283 (nan) | > loss: 0.013875522650778294 (nan) | > current_lr: 5e-06 | > step_time: 0.1833 (0.13648333809587432) | > loader_time: 0.0049 (0.004656065582613153)  --> TIME: 2025-02-24 17:53:38 -- STEP: 17082/36938 -- GLOBAL_STEP: 423400 | > loss_text_ce: 0.04385855421423912 (0.03870920721892288) | > loss_mel_ce: 3.591667413711548 (nan) | > loss: 0.014426691457629204 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.1364677686236836) | > loader_time: 0.0048 (0.004655994194869552)  --> TIME: 2025-02-24 17:53:48 -- STEP: 17132/36938 -- GLOBAL_STEP: 423450 | > loss_text_ce: 0.04586031287908554 (0.03870811320126136) | > loss_mel_ce: 2.9929592609405518 (nan) | > loss: 0.012058808468282223 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.1364638115545315) | > loader_time: 0.0047 (0.0046559536175947306)  --> TIME: 2025-02-24 17:53:57 -- STEP: 17182/36938 -- GLOBAL_STEP: 423500 | > loss_text_ce: 0.05348360538482666 (0.03871086080744086) | > loss_mel_ce: 2.5750620365142822 (nan) | > loss: 0.010430737398564816 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.136469384758343) | > loader_time: 0.0045 (0.004656002097184784)  --> TIME: 2025-02-24 17:54:07 -- STEP: 17232/36938 -- GLOBAL_STEP: 423550 | > loss_text_ce: 0.03467415273189545 (0.03870805861548799) | > loss_mel_ce: 2.4697492122650146 (nan) | > loss: 0.009938188828527927 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.13646904753387928) | > loader_time: 0.0046 (0.004656121715861787)  --> TIME: 2025-02-24 17:54:16 -- STEP: 17282/36938 -- GLOBAL_STEP: 423600 | > loss_text_ce: 0.04275071620941162 (0.03870937691412129) | > loss_mel_ce: 3.1269166469573975 (nan) | > loss: 0.012578045018017292 (nan) | > current_lr: 5e-06 | > step_time: 0.1484 (0.13646979270441076) | > loader_time: 0.0047 (0.004656110092964798)  --> TIME: 2025-02-24 17:54:25 -- STEP: 17332/36938 -- GLOBAL_STEP: 423650 | > loss_text_ce: 0.023886946961283684 (0.0387091943376758) | > loss_mel_ce: 1.1808208227157593 (nan) | > loss: 0.004780586343258619 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.1364693156901998) | > loader_time: 0.0047 (0.0046560508451587365)  --> TIME: 2025-02-24 17:54:34 -- STEP: 17382/36938 -- GLOBAL_STEP: 423700 | > loss_text_ce: 0.0500120148062706 (0.038708167504082056) | > loss_mel_ce: 3.2647953033447266 (nan) | > loss: 0.01315399818122387 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.1364646201275831) | > loader_time: 0.0046 (0.004656063976777255)  --> TIME: 2025-02-24 17:54:44 -- STEP: 17432/36938 -- GLOBAL_STEP: 423750 | > loss_text_ce: 0.03757144883275032 (0.03870664118856883) | > loss_mel_ce: 3.2062087059020996 (nan) | > loss: 0.01287214457988739 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1364699864726704) | > loader_time: 0.0047 (0.004656188419074199)  --> TIME: 2025-02-24 17:54:53 -- STEP: 17482/36938 -- GLOBAL_STEP: 423800 | > loss_text_ce: 0.038972679525613785 (0.03870081479983254) | > loss_mel_ce: 3.3048534393310547 (nan) | > loss: 0.0132691515609622 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.1364591954133834) | > loader_time: 0.0046 (0.0046561515627211855)  --> TIME: 2025-02-24 17:55:02 -- STEP: 17532/36938 -- GLOBAL_STEP: 423850 | > loss_text_ce: 0.03761260583996773 (0.038701139405820616) | > loss_mel_ce: 3.8294270038604736 (nan) | > loss: 0.015345396474003792 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.1364567086981118) | > loader_time: 0.0046 (0.004656029854521435)  --> TIME: 2025-02-24 17:55:12 -- STEP: 17582/36938 -- GLOBAL_STEP: 423900 | > loss_text_ce: 0.02957157976925373 (0.038701368565795584) | > loss_mel_ce: 1.4612040519714355 (nan) | > loss: 0.005915776360780001 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13646404221933636) | > loader_time: 0.0044 (0.004655841240065465)  --> TIME: 2025-02-24 17:55:21 -- STEP: 17632/36938 -- GLOBAL_STEP: 423950 | > loss_text_ce: 0.03634519502520561 (0.03870146237025346) | > loss_mel_ce: 3.486159563064575 (nan) | > loss: 0.013978194445371628 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13647106157348735) | > loader_time: 0.0047 (0.004655953462889765)  --> TIME: 2025-02-24 17:55:30 -- STEP: 17682/36938 -- GLOBAL_STEP: 424000 | > loss_text_ce: 0.040289342403411865 (0.03869955529688567) | > loss_mel_ce: 2.0235226154327393 (nan) | > loss: 0.00818973034620285 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13645633133190455) | > loader_time: 0.0046 (0.004655849217640424)  --> TIME: 2025-02-24 17:55:39 -- STEP: 17732/36938 -- GLOBAL_STEP: 424050 | > loss_text_ce: 0.03243682160973549 (0.03869632275697548) | > loss_mel_ce: 2.947439193725586 (nan) | > loss: 0.01182490587234497 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13642960975058827) | > loader_time: 0.0046 (0.004655854900453917)  --> TIME: 2025-02-24 17:55:48 -- STEP: 17782/36938 -- GLOBAL_STEP: 424100 | > loss_text_ce: 0.04160217195749283 (0.03869582621283058) | > loss_mel_ce: 2.5207877159118652 (nan) | > loss: 0.010168214328587055 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13641860015533644) | > loader_time: 0.0048 (0.0046558085422174905)  --> TIME: 2025-02-24 17:55:58 -- STEP: 17832/36938 -- GLOBAL_STEP: 424150 | > loss_text_ce: 0.03083869442343712 (0.038699441556459456) | > loss_mel_ce: 2.972306966781616 (nan) | > loss: 0.01191724557429552 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13643192704775364) | > loader_time: 0.0046 (0.004655843828760933)  --> TIME: 2025-02-24 17:56:07 -- STEP: 17882/36938 -- GLOBAL_STEP: 424200 | > loss_text_ce: 0.03630804643034935 (0.03870032061454497) | > loss_mel_ce: 3.0310325622558594 (nan) | > loss: 0.012171987444162369 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13641623012865714) | > loader_time: 0.0046 (0.004655845599104648)  --> TIME: 2025-02-24 17:56:16 -- STEP: 17932/36938 -- GLOBAL_STEP: 424250 | > loss_text_ce: 0.03220337629318237 (0.038701738948176974) | > loss_mel_ce: 3.4951436519622803 (nan) | > loss: 0.013997409492731094 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.13641968186668021) | > loader_time: 0.0049 (0.004655993545850489)  --> TIME: 2025-02-24 17:56:26 -- STEP: 17982/36938 -- GLOBAL_STEP: 424300 | > loss_text_ce: 0.03976427763700485 (0.03870271336006135) | > loss_mel_ce: 3.5112247467041016 (nan) | > loss: 0.014091226272284985 (nan) | > current_lr: 5e-06 | > step_time: 0.1888 (0.1364349074974141) | > loader_time: 0.0048 (0.004655897411829707)  --> TIME: 2025-02-24 17:56:35 -- STEP: 18032/36938 -- GLOBAL_STEP: 424350 | > loss_text_ce: 0.028597211465239525 (0.03870468647147966) | > loss_mel_ce: 2.1049561500549316 (nan) | > loss: 0.008466482162475586 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.1364301673426709) | > loader_time: 0.0047 (0.004656061622667033)  --> TIME: 2025-02-24 17:56:44 -- STEP: 18082/36938 -- GLOBAL_STEP: 424400 | > loss_text_ce: 0.0350990928709507 (0.038703455936601246) | > loss_mel_ce: 3.2321150302886963 (nan) | > loss: 0.012965136207640171 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.1364412318683148) | > loader_time: 0.0047 (0.004656088456368122)  --> TIME: 2025-02-24 17:56:54 -- STEP: 18132/36938 -- GLOBAL_STEP: 424450 | > loss_text_ce: 0.024874288588762283 (0.038707724461798744) | > loss_mel_ce: 3.0764122009277344 (nan) | > loss: 0.012306693010032177 (nan) | > current_lr: 5e-06 | > step_time: 0.1435 (0.1364401783765401) | > loader_time: 0.0046 (0.004656060823351783)  --> TIME: 2025-02-24 17:57:03 -- STEP: 18182/36938 -- GLOBAL_STEP: 424500 | > loss_text_ce: 0.03749420493841171 (0.038708009184129664) | > loss_mel_ce: 3.1022582054138184 (nan) | > loss: 0.01245933584868908 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.1364414054376302) | > loader_time: 0.0046 (0.004655927783545207)  --> TIME: 2025-02-24 17:57:13 -- STEP: 18232/36938 -- GLOBAL_STEP: 424550 | > loss_text_ce: 0.038612574338912964 (0.038709081793771964) | > loss_mel_ce: 2.9051897525787354 (nan) | > loss: 0.011681756004691124 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.1364504732469447) | > loader_time: 0.0048 (0.0046560144204746195)  --> TIME: 2025-02-24 17:57:22 -- STEP: 18282/36938 -- GLOBAL_STEP: 424600 | > loss_text_ce: 0.038315970450639725 (0.038705535416921594) | > loss_mel_ce: 2.7699790000915527 (nan) | > loss: 0.011144028045237064 (nan) | > current_lr: 5e-06 | > step_time: 0.152 (0.1364367146967225) | > loader_time: 0.0047 (0.004655955526633426)  --> TIME: 2025-02-24 17:57:31 -- STEP: 18332/36938 -- GLOBAL_STEP: 424650 | > loss_text_ce: 0.036564458161592484 (0.038708552006004565) | > loss_mel_ce: 3.287907600402832 (nan) | > loss: 0.013192350044846535 (nan) | > current_lr: 5e-06 | > step_time: 0.1832 (0.13643627041374107) | > loader_time: 0.0048 (0.004655832069138126)  --> TIME: 2025-02-24 17:57:40 -- STEP: 18382/36938 -- GLOBAL_STEP: 424700 | > loss_text_ce: 0.027038738131523132 (0.03870599932977534) | > loss_mel_ce: 2.4878830909729004 (nan) | > loss: 0.009979849681258202 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13641950114223927) | > loader_time: 0.0047 (0.004655906378723531)  --> TIME: 2025-02-24 17:57:49 -- STEP: 18432/36938 -- GLOBAL_STEP: 424750 | > loss_text_ce: 0.044517502188682556 (0.03870263600629186) | > loss_mel_ce: 3.5868020057678223 (nan) | > loss: 0.01440999936312437 (nan) | > current_lr: 5e-06 | > step_time: 0.1746 (0.13641043632136027) | > loader_time: 0.0048 (0.004655815958459355)  --> TIME: 2025-02-24 17:57:58 -- STEP: 18482/36938 -- GLOBAL_STEP: 424800 | > loss_text_ce: 0.037920013070106506 (0.03870113658062776) | > loss_mel_ce: 2.8181281089782715 (nan) | > loss: 0.01133352518081665 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.13640427406235903) | > loader_time: 0.0048 (0.004655744796990713)  --> TIME: 2025-02-24 17:58:07 -- STEP: 18532/36938 -- GLOBAL_STEP: 424850 | > loss_text_ce: 0.03390951082110405 (0.038699503599909466) | > loss_mel_ce: 3.264559507369995 (nan) | > loss: 0.013089164160192013 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13639854057134163) | > loader_time: 0.0046 (0.004655668577519046)  --> TIME: 2025-02-24 17:58:17 -- STEP: 18582/36938 -- GLOBAL_STEP: 424900 | > loss_text_ce: 0.0426018089056015 (0.03870301656714688) | > loss_mel_ce: 2.9137508869171143 (nan) | > loss: 0.011731559410691261 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13640223329796783) | > loader_time: 0.0046 (0.004655745683554477)  --> TIME: 2025-02-24 17:58:26 -- STEP: 18632/36938 -- GLOBAL_STEP: 424950 | > loss_text_ce: 0.048464085906744 (0.038699722008132986) | > loss_mel_ce: 1.3013153076171875 (nan) | > loss: 0.005356267560273409 (nan) | > current_lr: 5e-06 | > step_time: 0.1917 (0.13639511332434107) | > loader_time: 0.0048 (0.004655786047374923)  --> TIME: 2025-02-24 17:58:35 -- STEP: 18682/36938 -- GLOBAL_STEP: 425000 | > loss_text_ce: 0.05498147010803223 (0.03869728248472865) | > loss_mel_ce: 2.5073978900909424 (nan) | > loss: 0.010168172419071198 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13639979873731153) | > loader_time: 0.0047 (0.004655846231385234)  --> TIME: 2025-02-24 17:58:44 -- STEP: 18732/36938 -- GLOBAL_STEP: 425050 | > loss_text_ce: 0.04604824259877205 (0.038695325878918704) | > loss_mel_ce: 2.0653929710388184 (nan) | > loss: 0.008378734812140465 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1363941214926194) | > loader_time: 0.0046 (0.004655822611964981)  --> TIME: 2025-02-24 17:58:54 -- STEP: 18782/36938 -- GLOBAL_STEP: 425100 | > loss_text_ce: 0.0482427217066288 (0.03869681137811996) | > loss_mel_ce: 3.0996251106262207 (nan) | > loss: 0.012491540051996708 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.1364036380463981) | > loader_time: 0.0047 (0.004655815861677296)  --> TIME: 2025-02-24 17:59:04 -- STEP: 18832/36938 -- GLOBAL_STEP: 425150 | > loss_text_ce: 0.034063372761011124 (0.03869408375893963) | > loss_mel_ce: 3.302489757537842 (nan) | > loss: 0.013240290805697441 (nan) | > current_lr: 5e-06 | > step_time: 0.1116 (0.1364153195030121) | > loader_time: 0.0045 (0.004655918532138984)  --> TIME: 2025-02-24 17:59:13 -- STEP: 18882/36938 -- GLOBAL_STEP: 425200 | > loss_text_ce: 0.035887449979782104 (0.03869327258064349) | > loss_mel_ce: 2.3518080711364746 (nan) | > loss: 0.009474982507526875 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.136402779774019) | > loader_time: 0.0046 (0.00465591785173121)  --> TIME: 2025-02-24 17:59:22 -- STEP: 18932/36938 -- GLOBAL_STEP: 425250 | > loss_text_ce: 0.04082568734884262 (0.038689187392814774) | > loss_mel_ce: 3.712404251098633 (nan) | > loss: 0.014893770217895508 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13639598014413667) | > loader_time: 0.0046 (0.004655894607513326)  --> TIME: 2025-02-24 17:59:31 -- STEP: 18982/36938 -- GLOBAL_STEP: 425300 | > loss_text_ce: 0.03660878911614418 (0.03868770759748775) | > loss_mel_ce: 3.5521206855773926 (nan) | > loss: 0.01424099039286375 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13638985491115488) | > loader_time: 0.0048 (0.004655880579367145)  --> TIME: 2025-02-24 17:59:40 -- STEP: 19032/36938 -- GLOBAL_STEP: 425350 | > loss_text_ce: 0.041936591267585754 (0.03868858570930012) | > loss_mel_ce: 0.04074910655617714 (nan) | > loss: 0.00032811786513775587 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13638647007761823) | > loader_time: 0.0045 (0.004655905910378103)  --> TIME: 2025-02-24 17:59:50 -- STEP: 19082/36938 -- GLOBAL_STEP: 425400 | > loss_text_ce: 0.036968205124139786 (0.03868888130834363) | > loss_mel_ce: 3.4359350204467773 (nan) | > loss: 0.013781363144516945 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.13639454667441536) | > loader_time: 0.0054 (0.004656021805657202)  --> TIME: 2025-02-24 17:59:59 -- STEP: 19132/36938 -- GLOBAL_STEP: 425450 | > loss_text_ce: 0.03779630735516548 (0.03869046786292407) | > loss_mel_ce: 3.582206964492798 (nan) | > loss: 0.014365092851221561 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.13640980619081236) | > loader_time: 0.005 (0.004656150815578137)  --> TIME: 2025-02-24 18:00:09 -- STEP: 19182/36938 -- GLOBAL_STEP: 425500 | > loss_text_ce: 0.03491717204451561 (0.03869900516729667) | > loss_mel_ce: 2.390049695968628 (nan) | > loss: 0.009622884914278984 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13642343192338688) | > loader_time: 0.0044 (0.004656396398404118)  --> TIME: 2025-02-24 18:00:18 -- STEP: 19232/36938 -- GLOBAL_STEP: 425550 | > loss_text_ce: 0.031809158623218536 (0.03869532882369599) | > loss_mel_ce: 2.2529296875 (nan) | > loss: 0.009066424332559109 (nan) | > current_lr: 5e-06 | > step_time: 0.108 (0.13641596446666082) | > loader_time: 0.0045 (0.004656319089618192)  --> TIME: 2025-02-24 18:00:28 -- STEP: 19282/36938 -- GLOBAL_STEP: 425600 | > loss_text_ce: 0.03926459699869156 (0.03869475085155364) | > loss_mel_ce: 3.5062482357025146 (nan) | > loss: 0.014069496653974056 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.13641659864287772) | > loader_time: 0.0048 (0.004656273984102881)  --> TIME: 2025-02-24 18:00:37 -- STEP: 19332/36938 -- GLOBAL_STEP: 425650 | > loss_text_ce: 0.029897311702370644 (0.03869469535240111) | > loss_mel_ce: 1.3957791328430176 (nan) | > loss: 0.0056574465706944466 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13642423323856837) | > loader_time: 0.0046 (0.004656283536757501)  --> TIME: 2025-02-24 18:00:46 -- STEP: 19382/36938 -- GLOBAL_STEP: 425700 | > loss_text_ce: 0.03974588215351105 (0.03869276792318403) | > loss_mel_ce: 2.8173415660858154 (nan) | > loss: 0.011337649077177048 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13642851696972894) | > loader_time: 0.0046 (0.004656374067015832)  --> TIME: 2025-02-24 18:00:56 -- STEP: 19432/36938 -- GLOBAL_STEP: 425750 | > loss_text_ce: 0.0338621586561203 (0.03869557741894465) | > loss_mel_ce: 3.424862861633301 (nan) | > loss: 0.013725100085139275 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.13643782792291598) | > loader_time: 0.0046 (0.004656322518215608)  --> TIME: 2025-02-24 18:01:05 -- STEP: 19482/36938 -- GLOBAL_STEP: 425800 | > loss_text_ce: 0.036066871136426926 (0.03869791172907375) | > loss_mel_ce: 2.586561918258667 (nan) | > loss: 0.01040725689381361 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.136442693458814) | > loader_time: 0.0045 (0.004656342617626648)  --> TIME: 2025-02-24 18:01:15 -- STEP: 19532/36938 -- GLOBAL_STEP: 425850 | > loss_text_ce: 0.025626687332987785 (0.03870080119997727) | > loss_mel_ce: 3.0715649127960205 (nan) | > loss: 0.012290443293750286 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13644653795538395) | > loader_time: 0.0045 (0.004656291059263177)  --> TIME: 2025-02-24 18:01:24 -- STEP: 19582/36938 -- GLOBAL_STEP: 425900 | > loss_text_ce: 0.04212222993373871 (0.03869374083385887) | > loss_mel_ce: 2.8385798931121826 (nan) | > loss: 0.011431357823312283 (nan) | > current_lr: 5e-06 | > step_time: 0.1537 (0.1364495655499493) | > loader_time: 0.0048 (0.004656194995268056)  --> TIME: 2025-02-24 18:01:34 -- STEP: 19632/36938 -- GLOBAL_STEP: 425950 | > loss_text_ce: 0.04352915287017822 (0.038695527084891684) | > loss_mel_ce: 3.356017589569092 (nan) | > loss: 0.013490265235304832 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13645148104832436) | > loader_time: 0.0047 (0.004656322488672101)  --> TIME: 2025-02-24 18:01:43 -- STEP: 19682/36938 -- GLOBAL_STEP: 426000 | > loss_text_ce: 0.03948504850268364 (0.03869061000370143) | > loss_mel_ce: 3.7983715534210205 (nan) | > loss: 0.015229590237140656 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13644590418751365) | > loader_time: 0.0046 (0.0046565011317816546)  --> TIME: 2025-02-24 18:01:52 -- STEP: 19732/36938 -- GLOBAL_STEP: 426050 | > loss_text_ce: 0.03976379334926605 (0.038687962541823356) | > loss_mel_ce: 3.258030652999878 (nan) | > loss: 0.013086486607789993 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13644959944487825) | > loader_time: 0.0046 (0.004656492008438912)  --> TIME: 2025-02-24 18:02:01 -- STEP: 19782/36938 -- GLOBAL_STEP: 426100 | > loss_text_ce: 0.04878900945186615 (0.03868652600441933) | > loss_mel_ce: 3.8158700466156006 (nan) | > loss: 0.015335949137806892 (nan) | > current_lr: 5e-06 | > step_time: 0.1689 (0.13644406584827515) | > loader_time: 0.0048 (0.00465647965299025)  --> TIME: 2025-02-24 18:02:11 -- STEP: 19832/36938 -- GLOBAL_STEP: 426150 | > loss_text_ce: 0.03573812544345856 (0.03868556737548699) | > loss_mel_ce: 2.984947919845581 (nan) | > loss: 0.011986850760877132 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13645011917532035) | > loader_time: 0.0046 (0.004656418370546583)  --> TIME: 2025-02-24 18:02:20 -- STEP: 19882/36938 -- GLOBAL_STEP: 426200 | > loss_text_ce: 0.033654600381851196 (0.03868879539565678) | > loss_mel_ce: 3.1855626106262207 (nan) | > loss: 0.012774672359228134 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13645334825850536) | > loader_time: 0.0046 (0.004656478260475066)  --> TIME: 2025-02-24 18:02:30 -- STEP: 19932/36938 -- GLOBAL_STEP: 426250 | > loss_text_ce: 0.0325201041996479 (0.03868590075123259) | > loss_mel_ce: 2.926727294921875 (nan) | > loss: 0.011743045412003994 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.1364600446935687) | > loader_time: 0.0046 (0.004656630803513394)  --> TIME: 2025-02-24 18:02:39 -- STEP: 19982/36938 -- GLOBAL_STEP: 426300 | > loss_text_ce: 0.03569002449512482 (0.03868425895380067) | > loss_mel_ce: 2.732861042022705 (nan) | > loss: 0.01098631415516138 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13646626261282613) | > loader_time: 0.0045 (0.004656503238950985)  --> TIME: 2025-02-24 18:02:48 -- STEP: 20032/36938 -- GLOBAL_STEP: 426350 | > loss_text_ce: 0.03709675371646881 (0.03868287302267826) | > loss_mel_ce: 2.2474417686462402 (nan) | > loss: 0.009065629914402962 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13646882573684163) | > loader_time: 0.0048 (0.004656381571826092)  --> TIME: 2025-02-24 18:02:58 -- STEP: 20082/36938 -- GLOBAL_STEP: 426400 | > loss_text_ce: 0.03404147922992706 (0.03868365822797342) | > loss_mel_ce: 3.044761896133423 (nan) | > loss: 0.012217474170029163 (nan) | > current_lr: 5e-06 | > step_time: 0.17 (0.13647465124843514) | > loader_time: 0.0048 (0.0046565262946744535)  --> TIME: 2025-02-24 18:03:07 -- STEP: 20132/36938 -- GLOBAL_STEP: 426450 | > loss_text_ce: 0.03581893816590309 (0.03868228065555178) | > loss_mel_ce: 2.371638298034668 (nan) | > loss: 0.009553402662277222 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13648258519414247) | > loader_time: 0.0045 (0.004656709569267469)  --> TIME: 2025-02-24 18:03:16 -- STEP: 20182/36938 -- GLOBAL_STEP: 426500 | > loss_text_ce: 0.039962247014045715 (0.03868586602179166) | > loss_mel_ce: 3.4523022174835205 (nan) | > loss: 0.013858193531632423 (nan) | > current_lr: 5e-06 | > step_time: 0.178 (0.13646895940281817) | > loader_time: 0.0047 (0.004656605649166934)  --> TIME: 2025-02-24 18:03:26 -- STEP: 20232/36938 -- GLOBAL_STEP: 426550 | > loss_text_ce: 0.02640937827527523 (0.03868753835531278) | > loss_mel_ce: 3.0533854961395264 (nan) | > loss: 0.012221409007906914 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1364913471794354) | > loader_time: 0.0046 (0.004656607251998807)  --> TIME: 2025-02-24 18:03:36 -- STEP: 20282/36938 -- GLOBAL_STEP: 426600 | > loss_text_ce: 0.03347579017281532 (0.038685798899225354) | > loss_mel_ce: 2.661972999572754 (nan) | > loss: 0.010696226730942726 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13649297442359573) | > loader_time: 0.0046 (0.004656506894243966)  --> TIME: 2025-02-24 18:03:45 -- STEP: 20332/36938 -- GLOBAL_STEP: 426650 | > loss_text_ce: 0.03663603216409683 (0.038686312768060886) | > loss_mel_ce: 3.257075309753418 (nan) | > loss: 0.013070284388959408 (nan) | > current_lr: 5e-06 | > step_time: 0.1623 (0.1364965699166028) | > loader_time: 0.0047 (0.004656494097659953)  --> TIME: 2025-02-24 18:03:54 -- STEP: 20382/36938 -- GLOBAL_STEP: 426700 | > loss_text_ce: 0.039918046444654465 (0.038688493356384274) | > loss_mel_ce: 3.0364561080932617 (nan) | > loss: 0.012207834050059319 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.1364937747335446) | > loader_time: 0.0046 (0.004656466379353764)  --> TIME: 2025-02-24 18:04:04 -- STEP: 20432/36938 -- GLOBAL_STEP: 426750 | > loss_text_ce: 0.039373189210891724 (0.038689861266626505) | > loss_mel_ce: 2.202670097351074 (nan) | > loss: 0.008896997198462486 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13650105570383872) | > loader_time: 0.0045 (0.004656410277963359)  --> TIME: 2025-02-24 18:04:13 -- STEP: 20482/36938 -- GLOBAL_STEP: 426800 | > loss_text_ce: 0.05098078027367592 (0.03869074271909452) | > loss_mel_ce: 2.727377414703369 (nan) | > loss: 0.011025231331586838 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.1365056169904914) | > loader_time: 0.0048 (0.004656320157873851)  --> TIME: 2025-02-24 18:04:22 -- STEP: 20532/36938 -- GLOBAL_STEP: 426850 | > loss_text_ce: 0.02592930756509304 (0.038686157584233126) | > loss_mel_ce: 2.355656147003174 (nan) | > loss: 0.009450736455619335 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13649362161938425) | > loader_time: 0.0046 (0.004656109270147493)  --> TIME: 2025-02-24 18:04:31 -- STEP: 20582/36938 -- GLOBAL_STEP: 426900 | > loss_text_ce: 0.04911951348185539 (0.03868820521775631) | > loss_mel_ce: 2.7060585021972656 (nan) | > loss: 0.010933246463537216 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13649241520704897) | > loader_time: 0.0045 (0.0046560166007447905)  --> TIME: 2025-02-24 18:04:40 -- STEP: 20632/36938 -- GLOBAL_STEP: 426950 | > loss_text_ce: 0.0386507585644722 (0.03868501318682573) | > loss_mel_ce: 2.2593817710876465 (nan) | > loss: 0.009119177237153053 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.1364799122007928) | > loader_time: 0.0045 (0.0046558446572612005)  --> TIME: 2025-02-24 18:04:50 -- STEP: 20682/36938 -- GLOBAL_STEP: 427000 | > loss_text_ce: 0.03495946153998375 (0.03868359591663834) | > loss_mel_ce: 3.174429178237915 (nan) | > loss: 0.012735670432448387 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13648605822779045) | > loader_time: 0.0045 (0.004655783612865781)  --> TIME: 2025-02-24 18:04:59 -- STEP: 20732/36938 -- GLOBAL_STEP: 427050 | > loss_text_ce: 0.04652886465191841 (0.03868418655645936) | > loss_mel_ce: 2.063462257385254 (nan) | > loss: 0.00837298110127449 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13649134934465995) | > loader_time: 0.0045 (0.004655710132384721)  --> TIME: 2025-02-24 18:05:09 -- STEP: 20782/36938 -- GLOBAL_STEP: 427100 | > loss_text_ce: 0.038421060889959335 (0.038682833279704135) | > loss_mel_ce: 3.4135355949401855 (nan) | > loss: 0.013698241673409939 (nan) | > current_lr: 5e-06 | > step_time: 0.1987 (0.1365002530587783) | > loader_time: 0.0048 (0.004655936434069125)  --> TIME: 2025-02-24 18:05:18 -- STEP: 20832/36938 -- GLOBAL_STEP: 427150 | > loss_text_ce: 0.02789030410349369 (0.038682871803601276) | > loss_mel_ce: 2.3621928691864014 (nan) | > loss: 0.009484456852078438 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13649722039928422) | > loader_time: 0.0045 (0.004655947665556622)  --> TIME: 2025-02-24 18:05:27 -- STEP: 20882/36938 -- GLOBAL_STEP: 427200 | > loss_text_ce: 0.03461506590247154 (0.03868340380891579) | > loss_mel_ce: 3.607020378112793 (nan) | > loss: 0.01445093471556902 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.13649145228551487) | > loader_time: 0.0047 (0.00465604533022044)  --> TIME: 2025-02-24 18:05:37 -- STEP: 20932/36938 -- GLOBAL_STEP: 427250 | > loss_text_ce: 0.043206002563238144 (0.03869075402512455) | > loss_mel_ce: 3.153442144393921 (nan) | > loss: 0.012685112655162811 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13648877500599263) | > loader_time: 0.0048 (0.004656004605000794)  --> TIME: 2025-02-24 18:05:46 -- STEP: 20982/36938 -- GLOBAL_STEP: 427300 | > loss_text_ce: 0.04004669561982155 (0.03869055638384684) | > loss_mel_ce: 3.4613444805145264 (nan) | > loss: 0.013894409872591496 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.13648790577303116) | > loader_time: 0.0047 (0.004656115781362097)  --> TIME: 2025-02-24 18:05:55 -- STEP: 21032/36938 -- GLOBAL_STEP: 427350 | > loss_text_ce: 0.04038979858160019 (0.0386882428403628) | > loss_mel_ce: 3.5308470726013184 (nan) | > loss: 0.01417157519608736 (nan) | > current_lr: 5e-06 | > step_time: 0.1537 (0.1364837165520913) | > loader_time: 0.0067 (0.0046561344035363885)  --> TIME: 2025-02-24 18:06:05 -- STEP: 21082/36938 -- GLOBAL_STEP: 427400 | > loss_text_ce: 0.03697511553764343 (0.03868874225466132) | > loss_mel_ce: 2.9999873638153076 (nan) | > loss: 0.01205143891274929 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13650114672447014) | > loader_time: 0.0047 (0.0046561226968283185)  --> TIME: 2025-02-24 18:06:14 -- STEP: 21132/36938 -- GLOBAL_STEP: 427450 | > loss_text_ce: 0.03487354516983032 (0.03868523251239553) | > loss_mel_ce: 2.8781421184539795 (nan) | > loss: 0.01155958604067564 (nan) | > current_lr: 5e-06 | > step_time: 0.1527 (0.1364990616643514) | > loader_time: 0.0046 (0.0046560568676828964)  --> TIME: 2025-02-24 18:06:24 -- STEP: 21182/36938 -- GLOBAL_STEP: 427500 | > loss_text_ce: 0.0393393412232399 (0.03868444399566472) | > loss_mel_ce: 2.924743175506592 (nan) | > loss: 0.011762232519686222 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13650533194222686) | > loader_time: 0.0047 (0.0046561580014492275)  --> TIME: 2025-02-24 18:06:33 -- STEP: 21232/36938 -- GLOBAL_STEP: 427550 | > loss_text_ce: 0.03951183333992958 (0.03868558209629124) | > loss_mel_ce: 3.5928657054901123 (nan) | > loss: 0.014414197765290737 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13650241376272038) | > loader_time: 0.0046 (0.00465630679851021)  --> TIME: 2025-02-24 18:06:42 -- STEP: 21282/36938 -- GLOBAL_STEP: 427600 | > loss_text_ce: 0.04831254482269287 (0.038684216839269335) | > loss_mel_ce: 2.9158267974853516 (nan) | > loss: 0.011762458831071854 (nan) | > current_lr: 5e-06 | > step_time: 0.1075 (0.13651807464364413) | > loader_time: 0.0046 (0.004656357543829884)  --> TIME: 2025-02-24 18:06:52 -- STEP: 21332/36938 -- GLOBAL_STEP: 427650 | > loss_text_ce: 0.03578963503241539 (0.03868541380447458) | > loss_mel_ce: 0.0020599365234375 (nan) | > loss: 0.00015019671991467476 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1365116661180144) | > loader_time: 0.0045 (0.004656270110046091)  --> TIME: 2025-02-24 18:07:01 -- STEP: 21382/36938 -- GLOBAL_STEP: 427700 | > loss_text_ce: 0.03498141095042229 (0.03868965392289579) | > loss_mel_ce: 3.121767282485962 (nan) | > loss: 0.012526781298220158 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13650406012420457) | > loader_time: 0.0048 (0.0046563480335450125)  --> TIME: 2025-02-24 18:07:10 -- STEP: 21432/36938 -- GLOBAL_STEP: 427750 | > loss_text_ce: 0.03184324875473976 (0.03868459945509143) | > loss_mel_ce: 3.0725388526916504 (nan) | > loss: 0.012318977154791355 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.1364879038583969) | > loader_time: 0.0047 (0.0046562569583695)  --> TIME: 2025-02-24 18:07:19 -- STEP: 21482/36938 -- GLOBAL_STEP: 427800 | > loss_text_ce: 0.018270201981067657 (0.03868592257838022) | > loss_mel_ce: 1.8254730701446533 (nan) | > loss: 0.007316442206501961 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.136480850598941) | > loader_time: 0.0046 (0.004656144098999814)  --> TIME: 2025-02-24 18:07:29 -- STEP: 21532/36938 -- GLOBAL_STEP: 427850 | > loss_text_ce: 0.03955644741654396 (0.03867961831298125) | > loss_mel_ce: 2.051978349685669 (nan) | > loss: 0.008299741894006729 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13649253286705018) | > loader_time: 0.0045 (0.004656022706263011)  --> TIME: 2025-02-24 18:07:38 -- STEP: 21582/36938 -- GLOBAL_STEP: 427900 | > loss_text_ce: 0.05021265894174576 (0.038683472646737105) | > loss_mel_ce: 3.0760059356689453 (nan) | > loss: 0.012405630201101303 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13649516802282136) | > loader_time: 0.0046 (0.00465613274940603)  --> TIME: 2025-02-24 18:07:47 -- STEP: 21632/36938 -- GLOBAL_STEP: 427950 | > loss_text_ce: 0.05024411901831627 (0.038686487943842655) | > loss_mel_ce: 3.406001091003418 (nan) | > loss: 0.013715259730815887 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13648536864161095) | > loader_time: 0.0045 (0.00465614819270973)  --> TIME: 2025-02-24 18:07:56 -- STEP: 21682/36938 -- GLOBAL_STEP: 428000 | > loss_text_ce: 0.033665042370557785 (0.038681695410172044) | > loss_mel_ce: 3.0740444660186768 (nan) | > loss: 0.012332180514931679 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.1364764507907018) | > loader_time: 0.0045 (0.004656004351494609)  --> TIME: 2025-02-24 18:08:05 -- STEP: 21732/36938 -- GLOBAL_STEP: 428050 | > loss_text_ce: 0.04479055479168892 (0.03867537303871075) | > loss_mel_ce: 0.009124755859375 (nan) | > loss: 0.0002139496646123007 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13647077162797144) | > loader_time: 0.0047 (0.004655913689639862)  --> TIME: 2025-02-24 18:08:14 -- STEP: 21782/36938 -- GLOBAL_STEP: 428100 | > loss_text_ce: 0.05672149360179901 (0.038672097961360806) | > loss_mel_ce: 3.8745718002319336 (nan) | > loss: 0.015600371174514294 (nan) | > current_lr: 5e-06 | > step_time: 0.1613 (0.13646453412677365) | > loader_time: 0.0049 (0.0046558465393720825)  --> TIME: 2025-02-24 18:08:23 -- STEP: 21832/36938 -- GLOBAL_STEP: 428150 | > loss_text_ce: 0.03177756816148758 (0.03867382966729342) | > loss_mel_ce: 1.2546764612197876 (nan) | > loss: 0.0051049767062067986 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13645347284501816) | > loader_time: 0.0047 (0.004655801090140548)  --> TIME: 2025-02-24 18:08:33 -- STEP: 21882/36938 -- GLOBAL_STEP: 428200 | > loss_text_ce: 0.04960256814956665 (0.038674154969969246) | > loss_mel_ce: 3.3759191036224365 (nan) | > loss: 0.013593340292572975 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13646928728656357) | > loader_time: 0.0045 (0.004655767049337057)  --> TIME: 2025-02-24 18:08:42 -- STEP: 21932/36938 -- GLOBAL_STEP: 428250 | > loss_text_ce: 0.03038746304810047 (0.038672940970612364) | > loss_mel_ce: 2.8784027099609375 (nan) | > loss: 0.011542818509042263 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13646838690228735) | > loader_time: 0.0045 (0.004655788333099009)  --> TIME: 2025-02-24 18:08:52 -- STEP: 21982/36938 -- GLOBAL_STEP: 428300 | > loss_text_ce: 0.048794325441122055 (0.038670585018746824) | > loss_mel_ce: 2.9672765731811523 (nan) | > loss: 0.011968535371124744 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.13646235600277823) | > loader_time: 0.0046 (0.004655664225920107)  --> TIME: 2025-02-24 18:09:01 -- STEP: 22032/36938 -- GLOBAL_STEP: 428350 | > loss_text_ce: 0.03001738153398037 (0.03866546164347426) | > loss_mel_ce: 3.0788214206695557 (nan) | > loss: 0.012336662970483303 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13647086020172108) | > loader_time: 0.0045 (0.00465551875775158)  --> TIME: 2025-02-24 18:09:10 -- STEP: 22082/36938 -- GLOBAL_STEP: 428400 | > loss_text_ce: 0.04356598109006882 (0.03866541638143568) | > loss_mel_ce: 1.3734955787658691 (nan) | > loss: 0.005623260512948036 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13647751413645315) | > loader_time: 0.0047 (0.004655550608286439)  --> TIME: 2025-02-24 18:09:20 -- STEP: 22132/36938 -- GLOBAL_STEP: 428450 | > loss_text_ce: 0.03238464891910553 (0.03866523782151709) | > loss_mel_ce: 2.9031155109405518 (nan) | > loss: 0.011648810468614101 (nan) | > current_lr: 5e-06 | > step_time: 0.1917 (0.1364812640994698) | > loader_time: 0.0048 (0.004655438220977109)  --> TIME: 2025-02-24 18:09:29 -- STEP: 22182/36938 -- GLOBAL_STEP: 428500 | > loss_text_ce: 0.04853271320462227 (0.03866594269586717) | > loss_mel_ce: 2.7043402194976807 (nan) | > loss: 0.010924099944531918 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.1364914832021869) | > loader_time: 0.0045 (0.0046553588646546734)  --> TIME: 2025-02-24 18:09:39 -- STEP: 22232/36938 -- GLOBAL_STEP: 428550 | > loss_text_ce: 0.03908586502075195 (0.03866642163018322) | > loss_mel_ce: 3.3699400424957275 (nan) | > loss: 0.013527881354093552 (nan) | > current_lr: 5e-06 | > step_time: 0.1909 (0.13649494187271566) | > loader_time: 0.0051 (0.004655406731588044)  --> TIME: 2025-02-24 18:09:48 -- STEP: 22282/36938 -- GLOBAL_STEP: 428600 | > loss_text_ce: 0.029134588316082954 (0.03865998552927572) | > loss_mel_ce: 2.5699234008789062 (nan) | > loss: 0.010313722304999828 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13647745077405132) | > loader_time: 0.0048 (0.004655415831407422)  --> TIME: 2025-02-24 18:09:57 -- STEP: 22332/36938 -- GLOBAL_STEP: 428650 | > loss_text_ce: 0.03792367875576019 (0.03865714416306711) | > loss_mel_ce: 3.1088907718658447 (nan) | > loss: 0.012487359344959259 (nan) | > current_lr: 5e-06 | > step_time: 0.1837 (0.13647027926783986) | > loader_time: 0.0048 (0.00465533886249618)  --> TIME: 2025-02-24 18:10:06 -- STEP: 22382/36938 -- GLOBAL_STEP: 428700 | > loss_text_ce: 0.039595652371644974 (0.03865562064174375) | > loss_mel_ce: 2.5149385929107666 (nan) | > loss: 0.010137041099369526 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13646014272958837) | > loader_time: 0.0046 (0.00465525104196151)  --> TIME: 2025-02-24 18:10:15 -- STEP: 22432/36938 -- GLOBAL_STEP: 428750 | > loss_text_ce: 0.03554035350680351 (0.0386533702447756) | > loss_mel_ce: 3.096068859100342 (nan) | > loss: 0.012427021749317646 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13646280212851197) | > loader_time: 0.0046 (0.004655221676435358)  --> TIME: 2025-02-24 18:10:24 -- STEP: 22482/36938 -- GLOBAL_STEP: 428800 | > loss_text_ce: 0.046682894229888916 (0.03865518828372769) | > loss_mel_ce: 3.238417148590088 (nan) | > loss: 0.013036111369729042 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13645040100234326) | > loader_time: 0.0045 (0.004655147773834989)  --> TIME: 2025-02-24 18:10:34 -- STEP: 22532/36938 -- GLOBAL_STEP: 428850 | > loss_text_ce: 0.03657623380422592 (0.038653507624029824) | > loss_mel_ce: 0.2974446713924408 (nan) | > loss: 0.0013254799414426088 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.1364549690043752) | > loader_time: 0.0045 (0.004655028043452711)  --> TIME: 2025-02-24 18:10:43 -- STEP: 22582/36938 -- GLOBAL_STEP: 428900 | > loss_text_ce: 0.03841541334986687 (0.03865219621234731) | > loss_mel_ce: 3.384385824203491 (nan) | > loss: 0.013582545332610607 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.1364520464487418) | > loader_time: 0.0045 (0.004654839857931053)  --> TIME: 2025-02-24 18:10:52 -- STEP: 22632/36938 -- GLOBAL_STEP: 428950 | > loss_text_ce: 0.03865586966276169 (0.038651006304424876) | > loss_mel_ce: 2.5269815921783447 (nan) | > loss: 0.01018110103905201 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13644393724407422) | > loader_time: 0.0046 (0.004654748811027374)  --> TIME: 2025-02-24 18:11:01 -- STEP: 22682/36938 -- GLOBAL_STEP: 429000 | > loss_text_ce: 0.037337541580200195 (0.0386516690851552) | > loss_mel_ce: 3.6331820487976074 (nan) | > loss: 0.014565554447472095 (nan) | > current_lr: 5e-06 | > step_time: 0.1518 (0.13644385802431488) | > loader_time: 0.0046 (0.004654645972356596)  --> TIME: 2025-02-24 18:11:10 -- STEP: 22732/36938 -- GLOBAL_STEP: 429050 | > loss_text_ce: 0.0381157249212265 (0.03865213523974985) | > loss_mel_ce: 3.2509191036224365 (nan) | > loss: 0.013051725924015045 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13643964807814746) | > loader_time: 0.0046 (0.004654572680448337)  --> TIME: 2025-02-24 18:11:20 -- STEP: 22782/36938 -- GLOBAL_STEP: 429100 | > loss_text_ce: 0.04654419794678688 (0.03865219063289312) | > loss_mel_ce: 2.7450900077819824 (nan) | > loss: 0.011077914386987686 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13643956912766286) | > loader_time: 0.0046 (0.00465450005560196)  --> TIME: 2025-02-24 18:11:29 -- STEP: 22832/36938 -- GLOBAL_STEP: 429150 | > loss_text_ce: 0.04043400287628174 (0.03865395324792873) | > loss_mel_ce: 3.555600643157959 (nan) | > loss: 0.014269978739321232 (nan) | > current_lr: 5e-06 | > step_time: 0.1997 (0.1364377507523545) | > loader_time: 0.0048 (0.004654490308654676)  --> TIME: 2025-02-24 18:11:38 -- STEP: 22882/36938 -- GLOBAL_STEP: 429200 | > loss_text_ce: 0.036144793033599854 (0.038654907906834755) | > loss_mel_ce: 3.3542068004608154 (nan) | > loss: 0.013453776948153973 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13644475878957307) | > loader_time: 0.0046 (0.004654360353066252)  --> TIME: 2025-02-24 18:11:48 -- STEP: 22932/36938 -- GLOBAL_STEP: 429250 | > loss_text_ce: 0.03689500316977501 (0.0386556245935029) | > loss_mel_ce: 2.583970069885254 (nan) | > loss: 0.01040025893598795 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.1364502095955734) | > loader_time: 0.0046 (0.00465428343664264)  --> TIME: 2025-02-24 18:11:57 -- STEP: 22982/36938 -- GLOBAL_STEP: 429300 | > loss_text_ce: 0.02981499582529068 (0.03865489701445989) | > loss_mel_ce: 3.3593552112579346 (nan) | > loss: 0.013449088670313358 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13644152313805377) | > loader_time: 0.0046 (0.004654174072604915)  --> TIME: 2025-02-24 18:12:06 -- STEP: 23032/36938 -- GLOBAL_STEP: 429350 | > loss_text_ce: 0.04514961689710617 (0.03865359567646638) | > loss_mel_ce: 2.523186683654785 (nan) | > loss: 0.010191811248660088 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13644040711599292) | > loader_time: 0.0047 (0.004654197270110824)  --> TIME: 2025-02-24 18:12:16 -- STEP: 23082/36938 -- GLOBAL_STEP: 429400 | > loss_text_ce: 0.03518887609243393 (0.03865593293725798) | > loss_mel_ce: 1.6477868556976318 (nan) | > loss: 0.00667847553268075 (nan) | > current_lr: 5e-06 | > step_time: 0.1817 (0.13644252834860462) | > loader_time: 0.0049 (0.004654181580972809)  --> TIME: 2025-02-24 18:12:25 -- STEP: 23132/36938 -- GLOBAL_STEP: 429450 | > loss_text_ce: 0.04120314121246338 (0.03865996663362495) | > loss_mel_ce: 2.8839893341064453 (nan) | > loss: 0.01160790678113699 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.13644308322815443) | > loader_time: 0.0047 (0.004654181100455171)  --> TIME: 2025-02-24 18:12:34 -- STEP: 23182/36938 -- GLOBAL_STEP: 429500 | > loss_text_ce: 0.057604674249887466 (0.038658790590618955) | > loss_mel_ce: 3.802891969680786 (nan) | > loss: 0.015319432131946087 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.13644873531289858) | > loader_time: 0.005 (0.004654177804018327)  --> TIME: 2025-02-24 18:12:44 -- STEP: 23232/36938 -- GLOBAL_STEP: 429550 | > loss_text_ce: 0.04249969497323036 (0.03866064314827954) | > loss_mel_ce: 2.6373019218444824 (nan) | > loss: 0.010634134523570538 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.1364422806387267) | > loader_time: 0.0045 (0.004654157270495235)  --> TIME: 2025-02-24 18:12:53 -- STEP: 23282/36938 -- GLOBAL_STEP: 429600 | > loss_text_ce: 0.04055003076791763 (0.03865922498798745) | > loss_mel_ce: 2.5537502765655518 (nan) | > loss: 0.010294842533767223 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13644421669986465) | > loader_time: 0.0045 (0.0046539872631298825)  --> TIME: 2025-02-24 18:13:02 -- STEP: 23332/36938 -- GLOBAL_STEP: 429650 | > loss_text_ce: 0.026399999856948853 (0.038659627810960336) | > loss_mel_ce: 3.0065386295318604 (nan) | > loss: 0.012035471387207508 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13644164382706303) | > loader_time: 0.0045 (0.004653811464991474)  --> TIME: 2025-02-24 18:13:12 -- STEP: 23382/36938 -- GLOBAL_STEP: 429700 | > loss_text_ce: 0.03762108087539673 (0.03865685779067598) | > loss_mel_ce: 2.912691354751587 (nan) | > loss: 0.011707589030265808 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13644933601653775) | > loader_time: 0.0046 (0.004653692072139984)  --> TIME: 2025-02-24 18:13:21 -- STEP: 23432/36938 -- GLOBAL_STEP: 429750 | > loss_text_ce: 0.03981192409992218 (0.03865754094918231) | > loss_mel_ce: 3.0487241744995117 (nan) | > loss: 0.012256096117198467 (nan) | > current_lr: 5e-06 | > step_time: 0.1865 (0.1364547316736633) | > loader_time: 0.0048 (0.004653676911888375)  --> TIME: 2025-02-24 18:13:30 -- STEP: 23482/36938 -- GLOBAL_STEP: 429800 | > loss_text_ce: 0.03587598726153374 (0.03865557788164384) | > loss_mel_ce: 2.6470932960510254 (nan) | > loss: 0.010646704584360123 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13645193590215005) | > loader_time: 0.0045 (0.0046535803262262075)  --> TIME: 2025-02-24 18:13:40 -- STEP: 23532/36938 -- GLOBAL_STEP: 429850 | > loss_text_ce: 0.03332527354359627 (0.038654001448436406) | > loss_mel_ce: 3.1923537254333496 (nan) | > loss: 0.012800313532352448 (nan) | > current_lr: 5e-06 | > step_time: 0.1881 (0.1364518065710322) | > loader_time: 0.0048 (0.004653700867538938)  --> TIME: 2025-02-24 18:13:49 -- STEP: 23582/36938 -- GLOBAL_STEP: 429900 | > loss_text_ce: 0.03899335116147995 (0.03865503500453535) | > loss_mel_ce: 2.688227891921997 (nan) | > loss: 0.01082230731844902 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13645099256999305) | > loader_time: 0.0045 (0.004653640794709552)  --> TIME: 2025-02-24 18:13:58 -- STEP: 23632/36938 -- GLOBAL_STEP: 429950 | > loss_text_ce: 0.040153492242097855 (0.0386543007724059) | > loss_mel_ce: 2.5496673583984375 (nan) | > loss: 0.010277067311108112 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13644193032139734) | > loader_time: 0.0048 (0.004653529835941833)  --> TIME: 2025-02-24 18:14:07 -- STEP: 23682/36938 -- GLOBAL_STEP: 430000 | > loss_text_ce: 0.039932798594236374 (0.038652030818042284) | > loss_mel_ce: 3.065873622894287 (nan) | > loss: 0.012324629351496696 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13642815447832166) | > loader_time: 0.0045 (0.004653420241718091) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_430000.pth  --> TIME: 2025-02-24 18:14:22 -- STEP: 23732/36938 -- GLOBAL_STEP: 430050 | > loss_text_ce: 0.0522330142557621 (0.038654148014874226) | > loss_mel_ce: 3.157149076461792 (nan) | > loss: 0.012735643424093723 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.13642012353653052) | > loader_time: 0.0048 (0.0046534226432168955)  --> TIME: 2025-02-24 18:14:31 -- STEP: 23782/36938 -- GLOBAL_STEP: 430100 | > loss_text_ce: 0.032856520265340805 (0.03865335386936614) | > loss_mel_ce: 2.8722617626190186 (nan) | > loss: 0.011528247967362404 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13640365296021242) | > loader_time: 0.0047 (0.004653663974505647)  --> TIME: 2025-02-24 18:14:40 -- STEP: 23832/36938 -- GLOBAL_STEP: 430150 | > loss_text_ce: 0.037577174603939056 (0.038652523603092126) | > loss_mel_ce: 2.62146258354187 (nan) | > loss: 0.010551746003329754 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13641242210238314) | > loader_time: 0.0046 (0.0046538911677319285)  --> TIME: 2025-02-24 18:14:50 -- STEP: 23882/36938 -- GLOBAL_STEP: 430200 | > loss_text_ce: 0.037867799401283264 (0.03865174982441264) | > loss_mel_ce: 2.6009981632232666 (nan) | > loss: 0.010471690446138382 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13641547326136702) | > loader_time: 0.0046 (0.004654040608950357)  --> TIME: 2025-02-24 18:14:59 -- STEP: 23932/36938 -- GLOBAL_STEP: 430250 | > loss_text_ce: 0.04473618417978287 (0.0386511409967626) | > loss_mel_ce: 3.280188798904419 (nan) | > loss: 0.013194147497415543 (nan) | > current_lr: 5e-06 | > step_time: 0.1572 (0.1364188422711217) | > loader_time: 0.0047 (0.004654044274520312)  --> TIME: 2025-02-24 18:15:08 -- STEP: 23982/36938 -- GLOBAL_STEP: 430300 | > loss_text_ce: 0.03178027272224426 (0.03864957620304713) | > loss_mel_ce: 2.1703884601593018 (nan) | > loss: 0.008738765493035316 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13640391716278227) | > loader_time: 0.0046 (0.004654204196244377)  --> TIME: 2025-02-24 18:15:17 -- STEP: 24032/36938 -- GLOBAL_STEP: 430350 | > loss_text_ce: 0.0322737991809845 (0.03865012517056308) | > loss_mel_ce: 2.7413241863250732 (nan) | > loss: 0.011006341315805912 (nan) | > current_lr: 5e-06 | > step_time: 0.1628 (0.13639749238359256) | > loader_time: 0.0047 (0.004654140143435757)  --> TIME: 2025-02-24 18:15:27 -- STEP: 24082/36938 -- GLOBAL_STEP: 430400 | > loss_text_ce: 0.031775444746017456 (0.038651002840692075) | > loss_mel_ce: 2.068060874938965 (nan) | > loss: 0.008332684636116028 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13639420369427543) | > loader_time: 0.0048 (0.004654190685055854)  --> TIME: 2025-02-24 18:15:36 -- STEP: 24132/36938 -- GLOBAL_STEP: 430450 | > loss_text_ce: 0.044260844588279724 (0.03865065579280225) | > loss_mel_ce: 2.5406429767608643 (nan) | > loss: 0.010257555171847343 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13639933945783664) | > loader_time: 0.0046 (0.004654204738726981)  --> TIME: 2025-02-24 18:15:45 -- STEP: 24182/36938 -- GLOBAL_STEP: 430500 | > loss_text_ce: 0.028446139767766 (0.03864826597515212) | > loss_mel_ce: 0.08422103524208069 (nan) | > loss: 0.0004470919957384467 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13640501850957257) | > loader_time: 0.0046 (0.00465432962228778)  --> TIME: 2025-02-24 18:15:55 -- STEP: 24232/36938 -- GLOBAL_STEP: 430550 | > loss_text_ce: 0.03788641467690468 (0.038646986645244526) | > loss_mel_ce: 3.612628698348999 (nan) | > loss: 0.01448617223650217 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13641333831895372) | > loader_time: 0.0047 (0.0046543359618723915)  --> TIME: 2025-02-24 18:16:04 -- STEP: 24282/36938 -- GLOBAL_STEP: 430600 | > loss_text_ce: 0.02769654057919979 (0.0386465751479235) | > loss_mel_ce: 1.0001397132873535 (nan) | > loss: 0.004078715108335018 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13641049339354708) | > loader_time: 0.0046 (0.0046542735049127934)  --> TIME: 2025-02-24 18:16:14 -- STEP: 24332/36938 -- GLOBAL_STEP: 430650 | > loss_text_ce: 0.03272835165262222 (0.038647012046051296) | > loss_mel_ce: 2.934004306793213 (nan) | > loss: 0.011772749945521355 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13641308930757726) | > loader_time: 0.0045 (0.00465425198826375)  --> TIME: 2025-02-24 18:16:23 -- STEP: 24382/36938 -- GLOBAL_STEP: 430700 | > loss_text_ce: 0.028164679184556007 (0.03864562770588743) | > loss_mel_ce: 3.249295234680176 (nan) | > loss: 0.013005794025957584 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.1364177698068564) | > loader_time: 0.0048 (0.004654516922264936)  --> TIME: 2025-02-24 18:16:32 -- STEP: 24432/36938 -- GLOBAL_STEP: 430750 | > loss_text_ce: 0.0399978905916214 (0.038646347898064044) | > loss_mel_ce: 3.4520444869995117 (nan) | > loss: 0.013857311569154263 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13641401345657958) | > loader_time: 0.0046 (0.004654396552233905)  --> TIME: 2025-02-24 18:16:42 -- STEP: 24482/36938 -- GLOBAL_STEP: 430800 | > loss_text_ce: 0.04095149412751198 (0.03864688365352349) | > loss_mel_ce: 3.5701076984405518 (nan) | > loss: 0.01432960107922554 (nan) | > current_lr: 5e-06 | > step_time: 0.1563 (0.13642139178106885) | > loader_time: 0.0048 (0.004654484007076341)  --> TIME: 2025-02-24 18:16:51 -- STEP: 24532/36938 -- GLOBAL_STEP: 430850 | > loss_text_ce: 0.04077167436480522 (0.03864761492627405) | > loss_mel_ce: 3.423609733581543 (nan) | > loss: 0.013747545890510082 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13642563769153723) | > loader_time: 0.005 (0.004654404449556203)  --> TIME: 2025-02-24 18:17:00 -- STEP: 24582/36938 -- GLOBAL_STEP: 430900 | > loss_text_ce: 0.0414271280169487 (0.03864623424017037) | > loss_mel_ce: 2.9959075450897217 (nan) | > loss: 0.01205291599035263 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13642306100283386) | > loader_time: 0.0046 (0.004654307156309014)  --> TIME: 2025-02-24 18:17:09 -- STEP: 24632/36938 -- GLOBAL_STEP: 430950 | > loss_text_ce: 0.033247631043195724 (0.03864320205062072) | > loss_mel_ce: 2.946636438369751 (nan) | > loss: 0.011824937537312508 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13642038961980701) | > loader_time: 0.0047 (0.004654210470991898)  --> TIME: 2025-02-24 18:17:19 -- STEP: 24682/36938 -- GLOBAL_STEP: 431000 | > loss_text_ce: 0.03540367633104324 (0.03864413946710018) | > loss_mel_ce: 3.0118093490600586 (nan) | > loss: 0.01209211628884077 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.1364187963410137) | > loader_time: 0.0047 (0.0046541695559626705)  --> TIME: 2025-02-24 18:17:28 -- STEP: 24732/36938 -- GLOBAL_STEP: 431050 | > loss_text_ce: 0.02644813060760498 (0.03864391075932818) | > loss_mel_ce: 3.9176137447357178 (nan) | > loss: 0.01565103977918625 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1364229905476135) | > loader_time: 0.0045 (0.004654130107778468)  --> TIME: 2025-02-24 18:17:37 -- STEP: 24782/36938 -- GLOBAL_STEP: 431100 | > loss_text_ce: 0.031175795942544937 (0.03864130481672328) | > loss_mel_ce: 2.8271584510803223 (nan) | > loss: 0.01134259719401598 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13642734931346673) | > loader_time: 0.0046 (0.00465416687951618)  --> TIME: 2025-02-24 18:17:47 -- STEP: 24832/36938 -- GLOBAL_STEP: 431150 | > loss_text_ce: 0.03851151466369629 (0.038638795239637244) | > loss_mel_ce: 1.7542258501052856 (nan) | > loss: 0.0071140374056994915 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13642472192913882) | > loader_time: 0.0046 (0.004654209916815942)  --> TIME: 2025-02-24 18:17:56 -- STEP: 24882/36938 -- GLOBAL_STEP: 431200 | > loss_text_ce: 0.03914707154035568 (0.03863757871683947) | > loss_mel_ce: 3.191110610961914 (nan) | > loss: 0.012818483635783195 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.1364305593626126) | > loader_time: 0.0047 (0.004654274484312305)  --> TIME: 2025-02-24 18:18:05 -- STEP: 24932/36938 -- GLOBAL_STEP: 431250 | > loss_text_ce: 0.04184156656265259 (0.03863631752855292) | > loss_mel_ce: 3.3483664989471436 (nan) | > loss: 0.013453206978738308 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.1364307647414675) | > loader_time: 0.0047 (0.00465422256712392)  --> TIME: 2025-02-24 18:18:15 -- STEP: 24982/36938 -- GLOBAL_STEP: 431300 | > loss_text_ce: 0.01987103745341301 (0.03863539214148069) | > loss_mel_ce: 3.061255693435669 (nan) | > loss: 0.012226694263517857 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.1364425556853051) | > loader_time: 0.0049 (0.004654309870043695)  --> TIME: 2025-02-24 18:18:24 -- STEP: 25032/36938 -- GLOBAL_STEP: 431350 | > loss_text_ce: 0.042803965508937836 (0.03863236945774012) | > loss_mel_ce: 3.507450819015503 (nan) | > loss: 0.014088314026594162 (nan) | > current_lr: 5e-06 | > step_time: 0.1823 (0.136444224169095) | > loader_time: 0.0049 (0.004654393528703311)  --> TIME: 2025-02-24 18:18:34 -- STEP: 25082/36938 -- GLOBAL_STEP: 431400 | > loss_text_ce: 0.03913445770740509 (0.03862957603805854) | > loss_mel_ce: 3.7383224964141846 (nan) | > loss: 0.014989909715950489 (nan) | > current_lr: 5e-06 | > step_time: 0.1761 (0.13644323682834458) | > loader_time: 0.0048 (0.004654642003507068)  --> TIME: 2025-02-24 18:18:43 -- STEP: 25132/36938 -- GLOBAL_STEP: 431450 | > loss_text_ce: 0.03526449203491211 (0.03863349652189659) | > loss_mel_ce: 3.6677379608154297 (nan) | > loss: 0.014694455079734325 (nan) | > current_lr: 5e-06 | > step_time: 0.1571 (0.13643305824001406) | > loader_time: 0.0047 (0.004654770745188314)  --> TIME: 2025-02-24 18:18:52 -- STEP: 25182/36938 -- GLOBAL_STEP: 431500 | > loss_text_ce: 0.03815072029829025 (0.03863350401348669) | > loss_mel_ce: 3.0431604385375977 (nan) | > loss: 0.012227426283061504 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.1364300107190932) | > loader_time: 0.0046 (0.004654841515440574)  --> TIME: 2025-02-24 18:19:01 -- STEP: 25232/36938 -- GLOBAL_STEP: 431550 | > loss_text_ce: 0.02984705939888954 (0.038632389328683986) | > loss_mel_ce: 2.880836009979248 (nan) | > loss: 0.011550330556929111 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1364297474213015) | > loader_time: 0.0045 (0.00465481930997604)  --> TIME: 2025-02-24 18:19:11 -- STEP: 25282/36938 -- GLOBAL_STEP: 431600 | > loss_text_ce: 0.03460361808538437 (0.03863176388979736) | > loss_mel_ce: 3.8766887187957764 (nan) | > loss: 0.015521002002060413 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13643760980472244) | > loader_time: 0.0047 (0.004654902812471019)  --> TIME: 2025-02-24 18:19:20 -- STEP: 25332/36938 -- GLOBAL_STEP: 431650 | > loss_text_ce: 0.03969793766736984 (0.03863037126875569) | > loss_mel_ce: 3.0584845542907715 (nan) | > loss: 0.012294375337660313 (nan) | > current_lr: 5e-06 | > step_time: 0.1865 (0.13643919018307665) | > loader_time: 0.0048 (0.004654928921862624)  --> TIME: 2025-02-24 18:19:29 -- STEP: 25382/36938 -- GLOBAL_STEP: 431700 | > loss_text_ce: 0.040623344480991364 (0.03862870031881604) | > loss_mel_ce: 3.7786355018615723 (nan) | > loss: 0.015155790373682976 (nan) | > current_lr: 5e-06 | > step_time: 0.1565 (0.13643274637140682) | > loader_time: 0.0047 (0.004654987738887893)  --> TIME: 2025-02-24 18:19:38 -- STEP: 25432/36938 -- GLOBAL_STEP: 431750 | > loss_text_ce: 0.05465368181467056 (0.038629269735526134) | > loss_mel_ce: 1.7346351146697998 (nan) | > loss: 0.007100352551788092 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13643157274027842) | > loader_time: 0.0047 (0.0046549976228280315)  --> TIME: 2025-02-24 18:19:48 -- STEP: 25482/36938 -- GLOBAL_STEP: 431800 | > loss_text_ce: 0.032136138528585434 (0.03862772489902365) | > loss_mel_ce: 3.2348790168762207 (nan) | > loss: 0.012964346446096897 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.1364249200314186) | > loader_time: 0.0047 (0.004654964363263426)  --> TIME: 2025-02-24 18:19:57 -- STEP: 25532/36938 -- GLOBAL_STEP: 431850 | > loss_text_ce: 0.03113423101603985 (0.038627231819839694) | > loss_mel_ce: 3.0960168838500977 (nan) | > loss: 0.01240933034569025 (nan) | > current_lr: 5e-06 | > step_time: 0.172 (0.13642478683012427) | > loader_time: 0.0048 (0.004654920775371543)  --> TIME: 2025-02-24 18:20:06 -- STEP: 25582/36938 -- GLOBAL_STEP: 431900 | > loss_text_ce: 0.02769308350980282 (0.038627268786614) | > loss_mel_ce: 3.334859609603882 (nan) | > loss: 0.013343463651835918 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.13642826056836455) | > loader_time: 0.0046 (0.004654873210563136)  --> TIME: 2025-02-24 18:20:16 -- STEP: 25632/36938 -- GLOBAL_STEP: 431950 | > loss_text_ce: 0.034844182431697845 (0.038626349632269) | > loss_mel_ce: 3.096003770828247 (nan) | > loss: 0.012424000538885593 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.1364354476630318) | > loader_time: 0.0047 (0.004654930809166266)  --> TIME: 2025-02-24 18:20:25 -- STEP: 25682/36938 -- GLOBAL_STEP: 432000 | > loss_text_ce: 0.036315977573394775 (0.03862598309306882) | > loss_mel_ce: 3.18070912361145 (nan) | > loss: 0.012765972875058651 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.1364323005861199) | > loader_time: 0.0048 (0.004654894039625526)  --> TIME: 2025-02-24 18:20:34 -- STEP: 25732/36938 -- GLOBAL_STEP: 432050 | > loss_text_ce: 0.04916271194815636 (0.03862383914119734) | > loss_mel_ce: 3.504108428955078 (nan) | > loss: 0.014100282453000546 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.1364328559553498) | > loader_time: 0.0049 (0.004654940579665074)  --> TIME: 2025-02-24 18:20:44 -- STEP: 25782/36938 -- GLOBAL_STEP: 432100 | > loss_text_ce: 0.03955918923020363 (0.03862739557303683) | > loss_mel_ce: 2.199866533279419 (nan) | > loss: 0.0088866101577878 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13643443405965766) | > loader_time: 0.0045 (0.004654924453955542)  --> TIME: 2025-02-24 18:20:53 -- STEP: 25832/36938 -- GLOBAL_STEP: 432150 | > loss_text_ce: 0.049198806285858154 (0.038627955779466726) | > loss_mel_ce: 3.3700616359710693 (nan) | > loss: 0.013568494468927383 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13643230358008918) | > loader_time: 0.0046 (0.004654847992282439)  --> TIME: 2025-02-24 18:21:02 -- STEP: 25882/36938 -- GLOBAL_STEP: 432200 | > loss_text_ce: 0.026902595534920692 (0.0386292967916119) | > loss_mel_ce: 3.1149795055389404 (nan) | > loss: 0.012467787601053715 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.1364307588628563) | > loader_time: 0.0047 (0.004654887369045511)  --> TIME: 2025-02-24 18:21:11 -- STEP: 25932/36938 -- GLOBAL_STEP: 432250 | > loss_text_ce: 0.03065473958849907 (0.0386287489154345) | > loss_mel_ce: 2.4251301288604736 (nan) | > loss: 0.009745178744196892 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.1364348150965574) | > loader_time: 0.0044 (0.004654858117116101)  --> TIME: 2025-02-24 18:21:21 -- STEP: 25982/36938 -- GLOBAL_STEP: 432300 | > loss_text_ce: 0.04694313555955887 (0.038630159420142904) | > loss_mel_ce: 2.7908787727355957 (nan) | > loss: 0.011261198669672012 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1364249656294192) | > loader_time: 0.0045 (0.004654806578428216)  --> TIME: 2025-02-24 18:21:30 -- STEP: 26032/36938 -- GLOBAL_STEP: 432350 | > loss_text_ce: 0.03816422075033188 (0.03863060095462811) | > loss_mel_ce: 3.5452213287353516 (nan) | > loss: 0.014219784177839756 (nan) | > current_lr: 5e-06 | > step_time: 0.1688 (0.13642165193949005) | > loader_time: 0.0048 (0.0046547659716873035)  --> TIME: 2025-02-24 18:21:39 -- STEP: 26082/36938 -- GLOBAL_STEP: 432400 | > loss_text_ce: 0.038832228630781174 (0.03863175657523756) | > loss_mel_ce: 2.314206123352051 (nan) | > loss: 0.009337454102933407 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13641863703480925) | > loader_time: 0.0044 (0.004654702704410415)  --> TIME: 2025-02-24 18:21:48 -- STEP: 26132/36938 -- GLOBAL_STEP: 432450 | > loss_text_ce: 0.04373398795723915 (0.038633494601269525) | > loss_mel_ce: 3.275564193725586 (nan) | > loss: 0.013171819038689137 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13642168628043616) | > loader_time: 0.0045 (0.004654678171815633)  --> TIME: 2025-02-24 18:21:58 -- STEP: 26182/36938 -- GLOBAL_STEP: 432500 | > loss_text_ce: 0.04338871315121651 (0.038636248684610255) | > loss_mel_ce: 2.3250365257263184 (nan) | > loss: 0.009398512542247772 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1364263204063619) | > loader_time: 0.0045 (0.004654687972240365)  --> TIME: 2025-02-24 18:22:07 -- STEP: 26232/36938 -- GLOBAL_STEP: 432550 | > loss_text_ce: 0.031088871881365776 (0.03863359657263926) | > loss_mel_ce: 3.170720338821411 (nan) | > loss: 0.012705592438578606 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13641731696871956) | > loader_time: 0.0045 (0.004654577998861972)  --> TIME: 2025-02-24 18:22:16 -- STEP: 26282/36938 -- GLOBAL_STEP: 432600 | > loss_text_ce: 0.04311758279800415 (0.038633065747525036) | > loss_mel_ce: 4.110190391540527 (nan) | > loss: 0.01648138090968132 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1364230849446975) | > loader_time: 0.0046 (0.004654665523427763)  --> TIME: 2025-02-24 18:22:26 -- STEP: 26332/36938 -- GLOBAL_STEP: 432650 | > loss_text_ce: 0.04583938792347908 (0.03863358960267522) | > loss_mel_ce: 3.1179733276367188 (nan) | > loss: 0.012554813176393509 (nan) | > current_lr: 5e-06 | > step_time: 0.1609 (0.13642271069721235) | > loader_time: 0.0048 (0.004654639436900585)  --> TIME: 2025-02-24 18:22:35 -- STEP: 26382/36938 -- GLOBAL_STEP: 432700 | > loss_text_ce: 0.035607900470495224 (0.03863426452523364) | > loss_mel_ce: 3.2119972705841064 (nan) | > loss: 0.012887322343885899 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13641095768735714) | > loader_time: 0.0045 (0.004654623733551177)  --> TIME: 2025-02-24 18:22:44 -- STEP: 26432/36938 -- GLOBAL_STEP: 432750 | > loss_text_ce: 0.02028479613363743 (0.03863328157665633) | > loss_mel_ce: 1.4485247135162354 (nan) | > loss: 0.005828609224408865 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.1364007924500736) | > loader_time: 0.0047 (0.004654573046627129)  --> TIME: 2025-02-24 18:22:54 -- STEP: 26482/36938 -- GLOBAL_STEP: 432800 | > loss_text_ce: 0.03067639470100403 (0.03862942379338251) | > loss_mel_ce: 2.8190014362335205 (nan) | > loss: 0.011308246292173862 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.13641452834321868) | > loader_time: 0.0048 (0.004654665402384417)  --> TIME: 2025-02-24 18:23:03 -- STEP: 26532/36938 -- GLOBAL_STEP: 432850 | > loss_text_ce: 0.03366956487298012 (0.03862928988581356) | > loss_mel_ce: 2.8549439907073975 (nan) | > loss: 0.01146275270730257 (nan) | > current_lr: 5e-06 | > step_time: 0.1615 (0.13642484121295326) | > loader_time: 0.0048 (0.004654605410567402)  --> TIME: 2025-02-24 18:23:12 -- STEP: 26582/36938 -- GLOBAL_STEP: 432900 | > loss_text_ce: 0.03927599638700485 (0.03862704524841714) | > loss_mel_ce: 3.5097272396087646 (nan) | > loss: 0.014083346351981163 (nan) | > current_lr: 5e-06 | > step_time: 0.181 (0.13642749300427123) | > loader_time: 0.0048 (0.004654538603634789)  --> TIME: 2025-02-24 18:23:22 -- STEP: 26632/36938 -- GLOBAL_STEP: 432950 | > loss_text_ce: 0.039409466087818146 (0.03862660580617398) | > loss_mel_ce: 2.5787911415100098 (nan) | > loss: 0.01038968563079834 (nan) | > current_lr: 5e-06 | > step_time: 0.1271 (0.1364358144144498) | > loader_time: 0.0046 (0.004654509817454597)  --> TIME: 2025-02-24 18:23:31 -- STEP: 26682/36938 -- GLOBAL_STEP: 433000 | > loss_text_ce: 0.032689597457647324 (0.038625440147919216) | > loss_mel_ce: 2.259756326675415 (nan) | > loss: 0.009097008034586906 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.13642851699284128) | > loader_time: 0.0046 (0.004654577732550852)  --> TIME: 2025-02-24 18:23:40 -- STEP: 26732/36938 -- GLOBAL_STEP: 433050 | > loss_text_ce: 0.035302259027957916 (0.03862420937956969) | > loss_mel_ce: 0.1561715304851532 (nan) | > loss: 0.0007598166121169925 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13642110988513365) | > loader_time: 0.0045 (0.0046544944064517094)  --> TIME: 2025-02-24 18:23:49 -- STEP: 26782/36938 -- GLOBAL_STEP: 433100 | > loss_text_ce: 0.04890480265021324 (0.038623677046634185) | > loss_mel_ce: 2.7193737030029297 (nan) | > loss: 0.010985232889652252 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.13641903567960473) | > loader_time: 0.0047 (0.004654532933144384)  --> TIME: 2025-02-24 18:23:59 -- STEP: 26832/36938 -- GLOBAL_STEP: 433150 | > loss_text_ce: 0.03877425193786621 (0.03862045159061943) | > loss_mel_ce: 2.941460371017456 (nan) | > loss: 0.011826328933238983 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.1364156422191388) | > loader_time: 0.0046 (0.004654633942002991)  --> TIME: 2025-02-24 18:24:08 -- STEP: 26882/36938 -- GLOBAL_STEP: 433200 | > loss_text_ce: 0.03910832479596138 (0.03861906603502538) | > loss_mel_ce: 3.0269312858581543 (nan) | > loss: 0.0121668241918087 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13642095604181295) | > loader_time: 0.0046 (0.004654628350154517)  --> TIME: 2025-02-24 18:24:18 -- STEP: 26932/36938 -- GLOBAL_STEP: 433250 | > loss_text_ce: 0.03987106308341026 (0.038620359486390296) | > loss_mel_ce: 2.7052161693573 (nan) | > loss: 0.010893203318119049 (nan) | > current_lr: 5e-06 | > step_time: 0.1953 (0.13642792851861532) | > loader_time: 0.0048 (0.004654621716755296)  --> TIME: 2025-02-24 18:24:27 -- STEP: 26982/36938 -- GLOBAL_STEP: 433300 | > loss_text_ce: 0.03968021646142006 (0.03861982719746994) | > loss_mel_ce: 3.39035701751709 (nan) | > loss: 0.013611259870231152 (nan) | > current_lr: 5e-06 | > step_time: 0.165 (0.13643650504588647) | > loader_time: 0.0046 (0.004654646441136128)  --> TIME: 2025-02-24 18:24:37 -- STEP: 27032/36938 -- GLOBAL_STEP: 433350 | > loss_text_ce: 0.04839107394218445 (0.03862185443290678) | > loss_mel_ce: 2.917588472366333 (nan) | > loss: 0.011769761331379414 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13643557250446844) | > loader_time: 0.0045 (0.004654624064175416)  --> TIME: 2025-02-24 18:24:46 -- STEP: 27082/36938 -- GLOBAL_STEP: 433400 | > loss_text_ce: 0.0409713014960289 (0.03862097163404535) | > loss_mel_ce: 0.0559895820915699 (nan) | > loss: 0.0003847654443234205 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13643699201697018) | > loader_time: 0.0045 (0.0046545476630329135)  --> TIME: 2025-02-24 18:24:55 -- STEP: 27132/36938 -- GLOBAL_STEP: 433450 | > loss_text_ce: 0.03768594563007355 (0.03862130124204013) | > loss_mel_ce: 0.13066336512565613 (nan) | > loss: 0.0006680528749711812 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1364429483859396) | > loader_time: 0.0045 (0.004654472967032834)  --> TIME: 2025-02-24 18:25:04 -- STEP: 27182/36938 -- GLOBAL_STEP: 433500 | > loss_text_ce: 0.024084867909550667 (0.03862043689060926) | > loss_mel_ce: 1.212351679801941 (nan) | > loss: 0.00490649463608861 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13643120407189852) | > loader_time: 0.0044 (0.0046543469273590725)  --> TIME: 2025-02-24 18:25:13 -- STEP: 27232/36938 -- GLOBAL_STEP: 433550 | > loss_text_ce: 0.04044260084629059 (0.03861942000189999) | > loss_mel_ce: 2.691399335861206 (nan) | > loss: 0.01084064319729805 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13642326572561986) | > loader_time: 0.0048 (0.004654354970687723)  --> TIME: 2025-02-24 18:25:23 -- STEP: 27282/36938 -- GLOBAL_STEP: 433600 | > loss_text_ce: 0.027390114963054657 (0.03861747286048644) | > loss_mel_ce: 2.6025004386901855 (nan) | > loss: 0.010436073876917362 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1364229168328069) | > loader_time: 0.0044 (0.0046542594181321366)  --> TIME: 2025-02-24 18:25:32 -- STEP: 27332/36938 -- GLOBAL_STEP: 433650 | > loss_text_ce: 0.03973737731575966 (0.038618086886060804) | > loss_mel_ce: 3.0409955978393555 (nan) | > loss: 0.012225131504237652 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13641539336855954) | > loader_time: 0.0046 (0.00465454831752112)  --> TIME: 2025-02-24 18:25:41 -- STEP: 27382/36938 -- GLOBAL_STEP: 433700 | > loss_text_ce: 0.031318388879299164 (0.03861517384398675) | > loss_mel_ce: 2.9574193954467773 (nan) | > loss: 0.011860071681439877 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13641634988572396) | > loader_time: 0.0047 (0.004654556654323495)  --> TIME: 2025-02-24 18:25:50 -- STEP: 27432/36938 -- GLOBAL_STEP: 433750 | > loss_text_ce: 0.02597236819565296 (0.03861512619417139) | > loss_mel_ce: 2.816642999649048 (nan) | > loss: 0.011280220001935959 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13641522514051854) | > loader_time: 0.0046 (0.004654625938598556)  --> TIME: 2025-02-24 18:26:00 -- STEP: 27482/36938 -- GLOBAL_STEP: 433800 | > loss_text_ce: 0.03666462004184723 (0.03861485021092614) | > loss_mel_ce: 3.281827926635742 (nan) | > loss: 0.013168622739613056 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13641494894711417) | > loader_time: 0.0046 (0.004654592930181857)  --> TIME: 2025-02-24 18:26:09 -- STEP: 27532/36938 -- GLOBAL_STEP: 433850 | > loss_text_ce: 0.03501589968800545 (0.03861359018679143) | > loss_mel_ce: 3.056009531021118 (nan) | > loss: 0.012265974655747414 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13640756774534143) | > loader_time: 0.0045 (0.004654470344589483)  --> TIME: 2025-02-24 18:26:18 -- STEP: 27582/36938 -- GLOBAL_STEP: 433900 | > loss_text_ce: 0.035566266626119614 (0.038616059648197074) | > loss_mel_ce: 2.636216402053833 (nan) | > loss: 0.010602313093841076 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.13641426401793258) | > loader_time: 0.0046 (0.004654485988665375)  --> TIME: 2025-02-24 18:26:28 -- STEP: 27632/36938 -- GLOBAL_STEP: 433950 | > loss_text_ce: 0.05040973052382469 (0.038613803953302585) | > loss_mel_ce: 1.9213652610778809 (nan) | > loss: 0.007824503816664219 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13642122982621077) | > loader_time: 0.0046 (0.004654525235062356)  --> TIME: 2025-02-24 18:26:37 -- STEP: 27682/36938 -- GLOBAL_STEP: 434000 | > loss_text_ce: 0.046603765338659286 (0.03861355794013408) | > loss_mel_ce: 3.593353748321533 (nan) | > loss: 0.014444276690483093 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.1364264921183953) | > loader_time: 0.0048 (0.004654522886442325)  --> TIME: 2025-02-24 18:26:46 -- STEP: 27732/36938 -- GLOBAL_STEP: 434050 | > loss_text_ce: 0.036722954362630844 (0.03860883277710075) | > loss_mel_ce: 3.0354440212249756 (nan) | > loss: 0.012191139161586761 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13642792672146772) | > loader_time: 0.0049 (0.004654469745217292)  --> TIME: 2025-02-24 18:26:55 -- STEP: 27782/36938 -- GLOBAL_STEP: 434100 | > loss_text_ce: 0.03826846182346344 (0.03860866013753038) | > loss_mel_ce: 2.6243736743927 (nan) | > loss: 0.010566040873527527 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13642238488457262) | > loader_time: 0.0046 (0.004654442051404282)  --> TIME: 2025-02-24 18:27:05 -- STEP: 27832/36938 -- GLOBAL_STEP: 434150 | > loss_text_ce: 0.03501052036881447 (0.03860843957885143) | > loss_mel_ce: 3.149007558822632 (nan) | > loss: 0.01263499353080988 (nan) | > current_lr: 5e-06 | > step_time: 0.1349 (0.13642276775120893) | > loader_time: 0.0046 (0.004654424000005701)  --> TIME: 2025-02-24 18:27:14 -- STEP: 27882/36938 -- GLOBAL_STEP: 434200 | > loss_text_ce: 0.04438094422221184 (0.038607695450352024) | > loss_mel_ce: 3.007603168487549 (nan) | > loss: 0.012111048214137554 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13642334184707589) | > loader_time: 0.0046 (0.004654448537400382)  --> TIME: 2025-02-24 18:27:24 -- STEP: 27932/36938 -- GLOBAL_STEP: 434250 | > loss_text_ce: 0.03146002069115639 (0.03860596758268133) | > loss_mel_ce: 3.133650541305542 (nan) | > loss: 0.012559963390231133 (nan) | > current_lr: 5e-06 | > step_time: 0.1679 (0.13642466588090757) | > loader_time: 0.0048 (0.0046544042918055155)  --> TIME: 2025-02-24 18:27:33 -- STEP: 27982/36938 -- GLOBAL_STEP: 434300 | > loss_text_ce: 0.03707108646631241 (0.03861070862398049) | > loss_mel_ce: 2.7866709232330322 (nan) | > loss: 0.011205325834453106 (nan) | > current_lr: 5e-06 | > step_time: 0.1827 (0.13642322794737308) | > loader_time: 0.0049 (0.00465441015959385)  --> TIME: 2025-02-24 18:27:42 -- STEP: 28032/36938 -- GLOBAL_STEP: 434350 | > loss_text_ce: 0.03372395783662796 (0.038607212455674) | > loss_mel_ce: 2.8522257804870605 (nan) | > loss: 0.011452182196080685 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13641681756024843) | > loader_time: 0.0046 (0.0046544537101298415)  --> TIME: 2025-02-24 18:27:51 -- STEP: 28082/36938 -- GLOBAL_STEP: 434400 | > loss_text_ce: 0.03645789623260498 (0.038609993824373544) | > loss_mel_ce: 3.328125 (nan) | > loss: 0.013351520523428917 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.1364158512426834) | > loader_time: 0.0046 (0.004654484625156793)  --> TIME: 2025-02-24 18:28:01 -- STEP: 28132/36938 -- GLOBAL_STEP: 434450 | > loss_text_ce: 0.04272139444947243 (0.03860863801781166) | > loss_mel_ce: 3.090578556060791 (nan) | > loss: 0.012433730997145176 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13642299519347337) | > loader_time: 0.0047 (0.004654432493983055)  --> TIME: 2025-02-24 18:28:10 -- STEP: 28182/36938 -- GLOBAL_STEP: 434500 | > loss_text_ce: 0.03834879770874977 (0.038607120733114904) | > loss_mel_ce: 2.9726383686065674 (nan) | > loss: 0.011948362924158573 (nan) | > current_lr: 5e-06 | > step_time: 0.1799 (0.13642395869215795) | > loader_time: 0.0049 (0.004654334314110949)  --> TIME: 2025-02-24 18:28:20 -- STEP: 28232/36938 -- GLOBAL_STEP: 434550 | > loss_text_ce: 0.03658071532845497 (0.038608517588528464) | > loss_mel_ce: 3.4389262199401855 (nan) | > loss: 0.013791695237159729 (nan) | > current_lr: 5e-06 | > step_time: 0.2025 (0.13642974403499772) | > loader_time: 0.0048 (0.004654353335140877)  --> TIME: 2025-02-24 18:28:29 -- STEP: 28282/36938 -- GLOBAL_STEP: 434600 | > loss_text_ce: 0.03814138099551201 (0.03860698130951081) | > loss_mel_ce: 3.1973516941070557 (nan) | > loss: 0.012839259579777718 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.13643187556338662) | > loader_time: 0.0046 (0.00465434771532995)  --> TIME: 2025-02-24 18:28:39 -- STEP: 28332/36938 -- GLOBAL_STEP: 434650 | > loss_text_ce: 0.03915790468454361 (0.038606768513330315) | > loss_mel_ce: 3.3006534576416016 (nan) | > loss: 0.013253220357000828 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.1364402685161372) | > loader_time: 0.0049 (0.004654423498457735)  --> TIME: 2025-02-24 18:28:48 -- STEP: 28382/36938 -- GLOBAL_STEP: 434700 | > loss_text_ce: 0.03607022762298584 (0.03860354617449499) | > loss_mel_ce: 3.0536811351776123 (nan) | > loss: 0.012260918505489826 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.1364446834499432) | > loader_time: 0.0047 (0.004654424822731196)  --> TIME: 2025-02-24 18:28:57 -- STEP: 28432/36938 -- GLOBAL_STEP: 434750 | > loss_text_ce: 0.04061606153845787 (0.03860884386682245) | > loss_mel_ce: 2.887115478515625 (nan) | > loss: 0.011617982760071754 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.1364348795585714) | > loader_time: 0.0046 (0.004654431140147799)  --> TIME: 2025-02-24 18:29:07 -- STEP: 28482/36938 -- GLOBAL_STEP: 434800 | > loss_text_ce: 0.04862760752439499 (0.038609227771845236) | > loss_mel_ce: 3.6838901042938232 (nan) | > loss: 0.014811579138040543 (nan) | > current_lr: 5e-06 | > step_time: 0.1447 (0.13644001602113723) | > loader_time: 0.0067 (0.00465457181265564)  --> TIME: 2025-02-24 18:29:16 -- STEP: 28532/36938 -- GLOBAL_STEP: 434850 | > loss_text_ce: 0.039638452231884 (0.03861085406250493) | > loss_mel_ce: 2.711188554763794 (nan) | > loss: 0.01091598067432642 (nan) | > current_lr: 5e-06 | > step_time: 0.1757 (0.13645152782936013) | > loader_time: 0.0048 (0.004654587167485597)  --> TIME: 2025-02-24 18:29:26 -- STEP: 28582/36938 -- GLOBAL_STEP: 434900 | > loss_text_ce: 0.042851127684116364 (0.038610017537588706) | > loss_mel_ce: 2.435688018798828 (nan) | > loss: 0.009835473261773586 (nan) | > current_lr: 5e-06 | > step_time: 0.1512 (0.13645120922792317) | > loader_time: 0.0049 (0.004654576050860688)  --> TIME: 2025-02-24 18:29:35 -- STEP: 28632/36938 -- GLOBAL_STEP: 434950 | > loss_text_ce: 0.03457066789269447 (0.038610671353138705) | > loss_mel_ce: 3.363123893737793 (nan) | > loss: 0.013482916168868542 (nan) | > current_lr: 5e-06 | > step_time: 0.1753 (0.13645491879157623) | > loader_time: 0.0049 (0.004654612378655773)  --> TIME: 2025-02-24 18:29:45 -- STEP: 28682/36938 -- GLOBAL_STEP: 435000 | > loss_text_ce: 0.03871648758649826 (0.038611398342909574) | > loss_mel_ce: 2.5702760219573975 (nan) | > loss: 0.010353146120905876 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13645862212166845) | > loader_time: 0.0047 (0.004654602495399257)  --> TIME: 2025-02-24 18:29:54 -- STEP: 28732/36938 -- GLOBAL_STEP: 435050 | > loss_text_ce: 0.035309478640556335 (0.03861097174056901) | > loss_mel_ce: 3.3915560245513916 (nan) | > loss: 0.013598673976957798 (nan) | > current_lr: 5e-06 | > step_time: 0.1439 (0.13646965541000555) | > loader_time: 0.0047 (0.004654645886607768)  --> TIME: 2025-02-24 18:30:04 -- STEP: 28782/36938 -- GLOBAL_STEP: 435100 | > loss_text_ce: 0.03356349468231201 (0.03861097937801091) | > loss_mel_ce: 3.5347342491149902 (nan) | > loss: 0.014159913174808025 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.1364717884281088) | > loader_time: 0.0049 (0.00465476878215704)  --> TIME: 2025-02-24 18:30:13 -- STEP: 28832/36938 -- GLOBAL_STEP: 435150 | > loss_text_ce: 0.03779347613453865 (0.038610091810568216) | > loss_mel_ce: 3.3035502433776855 (nan) | > loss: 0.013259300962090492 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13647643661128617) | > loader_time: 0.0047 (0.0046547964364125095)  --> TIME: 2025-02-24 18:30:22 -- STEP: 28882/36938 -- GLOBAL_STEP: 435200 | > loss_text_ce: 0.038196954876184464 (0.03861002301193373) | > loss_mel_ce: 2.864597797393799 (nan) | > loss: 0.01151902787387371 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13647794645089062) | > loader_time: 0.0045 (0.004654808128963992)  --> TIME: 2025-02-24 18:30:32 -- STEP: 28932/36938 -- GLOBAL_STEP: 435250 | > loss_text_ce: 0.03882573917508125 (0.03860834303449006) | > loss_mel_ce: 3.2527222633361816 (nan) | > loss: 0.013061699457466602 (nan) | > current_lr: 5e-06 | > step_time: 0.1629 (0.1364835327251418) | > loader_time: 0.0047 (0.0046548353888987756)  --> TIME: 2025-02-24 18:30:41 -- STEP: 28982/36938 -- GLOBAL_STEP: 435300 | > loss_text_ce: 0.04212747886776924 (0.038609585686691636) | > loss_mel_ce: 2.803839683532715 (nan) | > loss: 0.011293521150946617 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13648540662210495) | > loader_time: 0.0046 (0.00465483376224808)  --> TIME: 2025-02-24 18:30:50 -- STEP: 29032/36938 -- GLOBAL_STEP: 435350 | > loss_text_ce: 0.04351029917597771 (0.03860757733960441) | > loss_mel_ce: 3.319784641265869 (nan) | > loss: 0.013346408493816853 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13647425919565942) | > loader_time: 0.0048 (0.004654727952149879)  --> TIME: 2025-02-24 18:31:00 -- STEP: 29082/36938 -- GLOBAL_STEP: 435400 | > loss_text_ce: 0.03419184684753418 (0.038609042100652684) | > loss_mel_ce: 3.1147913932800293 (nan) | > loss: 0.01249596569687128 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13647520051075684) | > loader_time: 0.0045 (0.004654666292200939)  --> TIME: 2025-02-24 18:31:09 -- STEP: 29132/36938 -- GLOBAL_STEP: 435450 | > loss_text_ce: 0.036451131105422974 (0.03860965677775898) | > loss_mel_ce: 3.2429614067077637 (nan) | > loss: 0.013013542629778385 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.1364719578071761) | > loader_time: 0.0047 (0.0046545867570953)  --> TIME: 2025-02-24 18:31:19 -- STEP: 29182/36938 -- GLOBAL_STEP: 435500 | > loss_text_ce: 0.04252992570400238 (0.03861206763289111) | > loss_mel_ce: 3.2712371349334717 (nan) | > loss: 0.013149869628250599 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.13648086763802536) | > loader_time: 0.0049 (0.004654589211439965)  --> TIME: 2025-02-24 18:31:28 -- STEP: 29232/36938 -- GLOBAL_STEP: 435550 | > loss_text_ce: 0.057649463415145874 (0.03861403390193629) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13648747341614623) | > loader_time: 0.0045 (0.004654565150123144)  --> TIME: 2025-02-24 18:31:37 -- STEP: 29282/36938 -- GLOBAL_STEP: 435600 | > loss_text_ce: 0.03373713418841362 (0.038612461215407116) | > loss_mel_ce: 3.905956745147705 (nan) | > loss: 0.015633707866072655 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13648455710157406) | > loader_time: 0.0046 (0.004654483491952193)  --> TIME: 2025-02-24 18:31:46 -- STEP: 29332/36938 -- GLOBAL_STEP: 435650 | > loss_text_ce: 0.03692222759127617 (0.03861561490870662) | > loss_mel_ce: 3.366394281387329 (nan) | > loss: 0.013505225069820881 (nan) | > current_lr: 5e-06 | > step_time: 0.1624 (0.13648254379120087) | > loader_time: 0.0047 (0.00465441788915691)  --> TIME: 2025-02-24 18:31:56 -- STEP: 29382/36938 -- GLOBAL_STEP: 435700 | > loss_text_ce: 0.03908693790435791 (0.03861479961800365) | > loss_mel_ce: 3.414623975753784 (nan) | > loss: 0.01370520330965519 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.1364865326571442) | > loader_time: 0.0046 (0.004654385308217737)  --> TIME: 2025-02-24 18:32:06 -- STEP: 29432/36938 -- GLOBAL_STEP: 435750 | > loss_text_ce: 0.03532509505748749 (0.03861450999323989) | > loss_mel_ce: 2.184626817703247 (nan) | > loss: 0.00880933366715908 (nan) | > current_lr: 5e-06 | > step_time: 0.1599 (0.13650068785710762) | > loader_time: 0.0047 (0.004654380404518257)  --> TIME: 2025-02-24 18:32:15 -- STEP: 29482/36938 -- GLOBAL_STEP: 435800 | > loss_text_ce: 0.03742517530918121 (0.038615867541878665) | > loss_mel_ce: 2.425335168838501 (nan) | > loss: 0.009772859513759613 (nan) | > current_lr: 5e-06 | > step_time: 0.1694 (0.1364973523255958) | > loader_time: 0.005 (0.004654279000059954)  --> TIME: 2025-02-24 18:32:24 -- STEP: 29532/36938 -- GLOBAL_STEP: 435850 | > loss_text_ce: 0.04422711580991745 (0.03861756005116791) | > loss_mel_ce: 2.4447598457336426 (nan) | > loss: 0.009876932948827744 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.1364995558443041) | > loader_time: 0.0046 (0.004654207430476203)  --> TIME: 2025-02-24 18:32:33 -- STEP: 29582/36938 -- GLOBAL_STEP: 435900 | > loss_text_ce: 0.0355224609375 (0.03861608813949307) | > loss_mel_ce: 2.9345014095306396 (nan) | > loss: 0.011785809881985188 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1364988649635355) | > loader_time: 0.0044 (0.004654133201378893)  --> TIME: 2025-02-24 18:32:43 -- STEP: 29632/36938 -- GLOBAL_STEP: 435950 | > loss_text_ce: 0.043543748557567596 (0.03861273663717783) | > loss_mel_ce: 3.4309933185577393 (nan) | > loss: 0.013787847012281418 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13649494892094935) | > loader_time: 0.0048 (0.004654110765354188)  --> TIME: 2025-02-24 18:32:52 -- STEP: 29682/36938 -- GLOBAL_STEP: 436000 | > loss_text_ce: 0.04295870289206505 (0.03861265920297635) | > loss_mel_ce: 1.8839843273162842 (nan) | > loss: 0.007646599784493446 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1364917321375753) | > loader_time: 0.0045 (0.004654086959080116)  --> TIME: 2025-02-24 18:33:01 -- STEP: 29732/36938 -- GLOBAL_STEP: 436050 | > loss_text_ce: 0.03177674114704132 (0.0386133766836479) | > loss_mel_ce: 2.9730823040008545 (nan) | > loss: 0.011924044229090214 (nan) | > current_lr: 5e-06 | > step_time: 0.1271 (0.1364864570029925) | > loader_time: 0.0047 (0.004654056023864996)  --> TIME: 2025-02-24 18:33:10 -- STEP: 29782/36938 -- GLOBAL_STEP: 436100 | > loss_text_ce: 0.03558048605918884 (0.03861350704524369) | > loss_mel_ce: 2.831949234008789 (nan) | > loss: 0.011379086412489414 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.13649306592863825) | > loader_time: 0.0046 (0.004654088331576976)  --> TIME: 2025-02-24 18:33:20 -- STEP: 29832/36938 -- GLOBAL_STEP: 436150 | > loss_text_ce: 0.03514713793992996 (0.0386113129733128) | > loss_mel_ce: 3.093505620956421 (nan) | > loss: 0.012415289878845215 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13649470248597056) | > loader_time: 0.0045 (0.00465414413948041)  --> TIME: 2025-02-24 18:33:29 -- STEP: 29882/36938 -- GLOBAL_STEP: 436200 | > loss_text_ce: 0.042669959366321564 (0.038611733719652194) | > loss_mel_ce: 2.9860646724700928 (nan) | > loss: 0.012018789537250996 (nan) | > current_lr: 5e-06 | > step_time: 0.1628 (0.1364879483128562) | > loader_time: 0.0047 (0.004654162651834626)  --> TIME: 2025-02-24 18:33:38 -- STEP: 29932/36938 -- GLOBAL_STEP: 436250 | > loss_text_ce: 0.04206300154328346 (0.038611387855795376) | > loss_mel_ce: 2.6431829929351807 (nan) | > loss: 0.010655738413333893 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13647822666722642) | > loader_time: 0.0048 (0.00465411334118711)  --> TIME: 2025-02-24 18:33:47 -- STEP: 29982/36938 -- GLOBAL_STEP: 436300 | > loss_text_ce: 0.03620505332946777 (0.0386116062107386) | > loss_mel_ce: 3.245722532272339 (nan) | > loss: 0.013023522682487965 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.1364754395697725) | > loader_time: 0.0046 (0.004654149091169814)  --> TIME: 2025-02-24 18:33:56 -- STEP: 30032/36938 -- GLOBAL_STEP: 436350 | > loss_text_ce: 0.034703947603702545 (0.03860974384589742) | > loss_mel_ce: 0.006011962890625 (nan) | > loss: 0.00016157107893377542 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13647711683475575) | > loader_time: 0.005 (0.004654080413984527)  --> TIME: 2025-02-24 18:34:06 -- STEP: 30082/36938 -- GLOBAL_STEP: 436400 | > loss_text_ce: 0.043264251202344894 (0.03860857666809242) | > loss_mel_ce: 3.0480387210845947 (nan) | > loss: 0.012267075479030609 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13647368646795097) | > loader_time: 0.0045 (0.0046539870786442014)  --> TIME: 2025-02-24 18:34:15 -- STEP: 30132/36938 -- GLOBAL_STEP: 436450 | > loss_text_ce: 0.038380663841962814 (0.03860956930704258) | > loss_mel_ce: 3.3393971920013428 (nan) | > loss: 0.013403881341218948 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.1364781840237179) | > loader_time: 0.0047 (0.004654049351128982)  --> TIME: 2025-02-24 18:34:24 -- STEP: 30182/36938 -- GLOBAL_STEP: 436500 | > loss_text_ce: 0.05635552108287811 (0.0386102894206852) | > loss_mel_ce: 3.6633458137512207 (nan) | > loss: 0.014760720543563366 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13647901388611627) | > loader_time: 0.0046 (0.004654010945290388)  --> TIME: 2025-02-24 18:34:34 -- STEP: 30232/36938 -- GLOBAL_STEP: 436550 | > loss_text_ce: 0.03684508800506592 (0.038610594024304806) | > loss_mel_ce: 2.903299570083618 (nan) | > loss: 0.011667241342365742 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13648196170020316) | > loader_time: 0.0062 (0.004654378550618587)  --> TIME: 2025-02-24 18:34:43 -- STEP: 30282/36938 -- GLOBAL_STEP: 436600 | > loss_text_ce: 0.04103313386440277 (0.03861091610762802) | > loss_mel_ce: 2.8238525390625 (nan) | > loss: 0.011368594132363796 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.13648267320909235) | > loader_time: 0.0048 (0.004654469629250833)  --> TIME: 2025-02-24 18:34:52 -- STEP: 30332/36938 -- GLOBAL_STEP: 436650 | > loss_text_ce: 0.03718757629394531 (0.038611249626904875) | > loss_mel_ce: 3.793768882751465 (nan) | > loss: 0.01520220935344696 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13647920685582918) | > loader_time: 0.0045 (0.004654409521317483)  --> TIME: 2025-02-24 18:35:02 -- STEP: 30382/36938 -- GLOBAL_STEP: 436700 | > loss_text_ce: 0.02900109626352787 (0.0386076797349275) | > loss_mel_ce: 1.972109317779541 (nan) | > loss: 0.00794091448187828 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13647164017615387) | > loader_time: 0.0049 (0.004654353001285506)  --> TIME: 2025-02-24 18:35:11 -- STEP: 30432/36938 -- GLOBAL_STEP: 436750 | > loss_text_ce: 0.039820875972509384 (0.03860672846172456) | > loss_mel_ce: 2.8884541988372803 (nan) | > loss: 0.011620139703154564 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13647069046362006) | > loader_time: 0.0045 (0.004654293180640211)  --> TIME: 2025-02-24 18:35:20 -- STEP: 30482/36938 -- GLOBAL_STEP: 436800 | > loss_text_ce: 0.05279393494129181 (0.03860650612757523) | > loss_mel_ce: 0.01239013671875 (nan) | > loss: 0.0002586669579613954 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13647188868753268) | > loader_time: 0.0046 (0.0046542674551391435)  --> TIME: 2025-02-24 18:35:29 -- STEP: 30532/36938 -- GLOBAL_STEP: 436850 | > loss_text_ce: 0.027155879884958267 (0.03860593053717746) | > loss_mel_ce: 2.8045694828033447 (nan) | > loss: 0.011237005703151226 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13646632094035321) | > loader_time: 0.0045 (0.00465416343687218)  --> TIME: 2025-02-24 18:35:39 -- STEP: 30582/36938 -- GLOBAL_STEP: 436900 | > loss_text_ce: 0.03916820138692856 (0.03860379414715507) | > loss_mel_ce: 2.4218976497650146 (nan) | > loss: 0.009766134433448315 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13646965925427396) | > loader_time: 0.0045 (0.0046541413209334245)  --> TIME: 2025-02-24 18:35:48 -- STEP: 30632/36938 -- GLOBAL_STEP: 436950 | > loss_text_ce: 0.033880531787872314 (0.03860493756266525) | > loss_mel_ce: 1.9507652521133423 (nan) | > loss: 0.007875579409301281 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13647948897797166) | > loader_time: 0.0046 (0.00465415433325512)  --> TIME: 2025-02-24 18:35:58 -- STEP: 30682/36938 -- GLOBAL_STEP: 437000 | > loss_text_ce: 0.03718024119734764 (0.03860490547293251) | > loss_mel_ce: 3.4854440689086914 (nan) | > loss: 0.013978668488562107 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.13647761901908723) | > loader_time: 0.0048 (0.004654129242603327)  --> TIME: 2025-02-24 18:36:07 -- STEP: 30732/36938 -- GLOBAL_STEP: 437050 | > loss_text_ce: 0.04742946848273277 (0.03860508538938364) | > loss_mel_ce: 3.2365782260894775 (nan) | > loss: 0.013031777925789356 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.13647523291576824) | > loader_time: 0.0047 (0.0046541078333027595)  --> TIME: 2025-02-24 18:36:16 -- STEP: 30782/36938 -- GLOBAL_STEP: 437100 | > loss_text_ce: 0.03619691729545593 (0.03860315780712849) | > loss_mel_ce: 0.040435791015625 (nan) | > loss: 0.00030409806640818715 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13647489356883136) | > loader_time: 0.0045 (0.0046540380693785)  --> TIME: 2025-02-24 18:36:25 -- STEP: 30832/36938 -- GLOBAL_STEP: 437150 | > loss_text_ce: 0.03652257099747658 (0.03860450582038086) | > loss_mel_ce: 2.887448310852051 (nan) | > loss: 0.01160306017845869 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13647559244179927) | > loader_time: 0.0047 (0.004653968593588032)  --> TIME: 2025-02-24 18:36:35 -- STEP: 30882/36938 -- GLOBAL_STEP: 437200 | > loss_text_ce: 0.0399116612970829 (0.038601297040559646) | > loss_mel_ce: 3.52313232421875 (nan) | > loss: 0.014139064587652683 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.1364724194707826) | > loader_time: 0.0046 (0.004653900037597161)  --> TIME: 2025-02-24 18:36:44 -- STEP: 30932/36938 -- GLOBAL_STEP: 437250 | > loss_text_ce: 0.05007423833012581 (0.03860628029159224) | > loss_mel_ce: 2.46875 (nan) | > loss: 0.009995335713028908 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.1364786702140902) | > loader_time: 0.0047 (0.004653939420156725)  --> TIME: 2025-02-24 18:36:53 -- STEP: 30982/36938 -- GLOBAL_STEP: 437300 | > loss_text_ce: 0.03837565332651138 (0.03860697028862683) | > loss_mel_ce: 3.0293402671813965 (nan) | > loss: 0.012173476628959179 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.1364789009402058) | > loader_time: 0.0046 (0.004653925192640093)  --> TIME: 2025-02-24 18:37:03 -- STEP: 31032/36938 -- GLOBAL_STEP: 437350 | > loss_text_ce: 0.03936333209276199 (0.0386055804649272) | > loss_mel_ce: 2.898233413696289 (nan) | > loss: 0.011657130904495716 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13647939682344928) | > loader_time: 0.0046 (0.004653918717011398)  --> TIME: 2025-02-24 18:37:12 -- STEP: 31082/36938 -- GLOBAL_STEP: 437400 | > loss_text_ce: 0.04021713137626648 (0.038601764719624065) | > loss_mel_ce: 3.2830028533935547 (nan) | > loss: 0.013187381438910961 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13648083905564393) | > loader_time: 0.0045 (0.004653979326550967)  --> TIME: 2025-02-24 18:37:22 -- STEP: 31132/36938 -- GLOBAL_STEP: 437450 | > loss_text_ce: 0.03622690215706825 (0.038600519993212644) | > loss_mel_ce: 2.7106773853302 (nan) | > loss: 0.010900414548814297 (nan) | > current_lr: 5e-06 | > step_time: 0.1504 (0.1364856810966774) | > loader_time: 0.0047 (0.004654054621505711)  --> TIME: 2025-02-24 18:37:31 -- STEP: 31182/36938 -- GLOBAL_STEP: 437500 | > loss_text_ce: 0.03854776918888092 (0.03859870418845642) | > loss_mel_ce: 2.589667320251465 (nan) | > loss: 0.010429425165057182 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13648418908121995) | > loader_time: 0.0047 (0.0046540612430021)  --> TIME: 2025-02-24 18:37:40 -- STEP: 31232/36938 -- GLOBAL_STEP: 437550 | > loss_text_ce: 0.039380304515361786 (0.03859702925204103) | > loss_mel_ce: 3.040501832962036 (nan) | > loss: 0.01222175545990467 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.1364841470036843) | > loader_time: 0.0046 (0.00465409828685837)  --> TIME: 2025-02-24 18:37:50 -- STEP: 31282/36938 -- GLOBAL_STEP: 437600 | > loss_text_ce: 0.03477810323238373 (0.03859455797643964) | > loss_mel_ce: 3.200852155685425 (nan) | > loss: 0.012839803472161293 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.1364861817838869) | > loader_time: 0.005 (0.0046540525028006315)  --> TIME: 2025-02-24 18:37:59 -- STEP: 31332/36938 -- GLOBAL_STEP: 437650 | > loss_text_ce: 0.04190105199813843 (0.038595254898393994) | > loss_mel_ce: 3.4976518154144287 (nan) | > loss: 0.014045845717191696 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13648766472500076) | > loader_time: 0.0045 (0.004654086398603563)  --> TIME: 2025-02-24 18:38:08 -- STEP: 31382/36938 -- GLOBAL_STEP: 437700 | > loss_text_ce: 0.046232063323259354 (0.03859422523669008) | > loss_mel_ce: 3.6436386108398438 (nan) | > loss: 0.014642344787716866 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.13648274664167473) | > loader_time: 0.0048 (0.004654123286095999)  --> TIME: 2025-02-24 18:38:18 -- STEP: 31432/36938 -- GLOBAL_STEP: 437750 | > loss_text_ce: 0.037471890449523926 (0.03859511634356138) | > loss_mel_ce: 3.0018672943115234 (nan) | > loss: 0.012060870416462421 (nan) | > current_lr: 5e-06 | > step_time: 0.1434 (0.13648749852672043) | > loader_time: 0.0046 (0.004654108309873243)  --> TIME: 2025-02-24 18:38:27 -- STEP: 31482/36938 -- GLOBAL_STEP: 437800 | > loss_text_ce: 0.037146978080272675 (0.03859617891454889) | > loss_mel_ce: 3.509786605834961 (nan) | > loss: 0.014075133949518204 (nan) | > current_lr: 5e-06 | > step_time: 0.1766 (0.13648954164678198) | > loader_time: 0.0049 (0.004654155859884521)  --> TIME: 2025-02-24 18:38:37 -- STEP: 31532/36938 -- GLOBAL_STEP: 437850 | > loss_text_ce: 0.044632747769355774 (0.03859773715658508) | > loss_mel_ce: 3.385605573654175 (nan) | > loss: 0.013612057082355022 (nan) | > current_lr: 5e-06 | > step_time: 0.1872 (0.1364935902144032) | > loader_time: 0.005 (0.00465412094071935)  --> TIME: 2025-02-24 18:38:46 -- STEP: 31582/36938 -- GLOBAL_STEP: 437900 | > loss_text_ce: 0.05196128413081169 (0.03859758393409171) | > loss_mel_ce: 2.9082818031311035 (nan) | > loss: 0.011746997013688087 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.1364939405378962) | > loader_time: 0.0045 (0.004654060253490958)  --> TIME: 2025-02-24 18:38:55 -- STEP: 31632/36938 -- GLOBAL_STEP: 437950 | > loss_text_ce: 0.04686475545167923 (0.03859957687028077) | > loss_mel_ce: 3.0509908199310303 (nan) | > loss: 0.012293078005313873 (nan) | > current_lr: 5e-06 | > step_time: 0.1558 (0.13650105874015075) | > loader_time: 0.0047 (0.004654078168222646)  --> TIME: 2025-02-24 18:39:05 -- STEP: 31682/36938 -- GLOBAL_STEP: 438000 | > loss_text_ce: 0.034963153302669525 (0.038598263027940984) | > loss_mel_ce: 4.296875 (nan) | > loss: 0.017189834266901016 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13650477240313058) | > loader_time: 0.0046 (0.004654109496810259)  --> TIME: 2025-02-24 18:39:14 -- STEP: 31732/36938 -- GLOBAL_STEP: 438050 | > loss_text_ce: 0.029414484277367592 (0.038598491423580794) | > loss_mel_ce: 2.6528215408325195 (nan) | > loss: 0.01064379420131445 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.1365043424003203) | > loader_time: 0.0048 (0.0046541158945311535)  --> TIME: 2025-02-24 18:39:23 -- STEP: 31782/36938 -- GLOBAL_STEP: 438100 | > loss_text_ce: 0.039795178920030594 (0.03859805898511384) | > loss_mel_ce: 1.3939392566680908 (nan) | > loss: 0.005689422599971294 (nan) | > current_lr: 5e-06 | > step_time: 0.1121 (0.13649724610208508) | > loader_time: 0.0045 (0.0046540756191378475)  --> TIME: 2025-02-24 18:39:33 -- STEP: 31832/36938 -- GLOBAL_STEP: 438150 | > loss_text_ce: 0.03487394005060196 (0.038598845680335565) | > loss_mel_ce: 2.6904826164245605 (nan) | > loss: 0.0108149079605937 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13650651269745226) | > loader_time: 0.0046 (0.004654100767240142)  --> TIME: 2025-02-24 18:39:42 -- STEP: 31882/36938 -- GLOBAL_STEP: 438200 | > loss_text_ce: 0.03021775186061859 (0.0385982069627026) | > loss_mel_ce: 1.6957931518554688 (nan) | > loss: 0.006849250290542841 (nan) | > current_lr: 5e-06 | > step_time: 0.1268 (0.13651641045151577) | > loader_time: 0.0046 (0.004654126023417658)  --> TIME: 2025-02-24 18:39:52 -- STEP: 31932/36938 -- GLOBAL_STEP: 438250 | > loss_text_ce: 0.03483114391565323 (0.038594910660739966) | > loss_mel_ce: 3.2162439823150635 (nan) | > loss: 0.012901091948151588 (nan) | > current_lr: 5e-06 | > step_time: 0.1582 (0.13651599344059812) | > loader_time: 0.0047 (0.004654080478317491)  --> TIME: 2025-02-24 18:40:01 -- STEP: 31982/36938 -- GLOBAL_STEP: 438300 | > loss_text_ce: 0.03442147746682167 (0.03859238986113565) | > loss_mel_ce: 1.9183344841003418 (nan) | > loss: 0.007749031763523817 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.13652149125533924) | > loader_time: 0.0047 (0.004654069546500415)  --> TIME: 2025-02-24 18:40:10 -- STEP: 32032/36938 -- GLOBAL_STEP: 438350 | > loss_text_ce: 0.03060065396130085 (0.03858966205999341) | > loss_mel_ce: 3.1617157459259033 (nan) | > loss: 0.012667922303080559 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1365187640686139) | > loader_time: 0.0048 (0.004654025921335728)  --> TIME: 2025-02-24 18:40:19 -- STEP: 32082/36938 -- GLOBAL_STEP: 438400 | > loss_text_ce: 0.03614256903529167 (0.03858823553802371) | > loss_mel_ce: 3.23471736907959 (nan) | > loss: 0.012979603372514248 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13651266178877627) | > loader_time: 0.0051 (0.0046539767247306)  --> TIME: 2025-02-24 18:40:29 -- STEP: 32132/36938 -- GLOBAL_STEP: 438450 | > loss_text_ce: 0.05023866519331932 (0.03858976929313364) | > loss_mel_ce: 2.930257558822632 (nan) | > loss: 0.011827366426587105 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13651494912039883) | > loader_time: 0.0046 (0.004653961612770771)  --> TIME: 2025-02-24 18:40:38 -- STEP: 32182/36938 -- GLOBAL_STEP: 438500 | > loss_text_ce: 0.03363114222884178 (0.03859080502687687) | > loss_mel_ce: 2.2500159740448 (nan) | > loss: 0.009062091819941998 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.1365134261748088) | > loader_time: 0.0048 (0.004653944228925437)  --> TIME: 2025-02-24 18:40:47 -- STEP: 32232/36938 -- GLOBAL_STEP: 438550 | > loss_text_ce: 0.04200494661927223 (0.038590412439768525) | > loss_mel_ce: 3.24802303314209 (nan) | > loss: 0.013055668212473392 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1365025516892525) | > loader_time: 0.0067 (0.004653935560845202)  --> TIME: 2025-02-24 18:40:56 -- STEP: 32282/36938 -- GLOBAL_STEP: 438600 | > loss_text_ce: 0.04031509906053543 (0.03858950473542998) | > loss_mel_ce: 2.892092704772949 (nan) | > loss: 0.011636539362370968 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13650149467760886) | > loader_time: 0.0047 (0.0046539095858571826)  --> TIME: 2025-02-24 18:41:05 -- STEP: 32332/36938 -- GLOBAL_STEP: 438650 | > loss_text_ce: 0.04792879521846771 (0.038587005756380396) | > loss_mel_ce: 3.8650624752044678 (nan) | > loss: 0.015527743846178055 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.13649382678114197) | > loader_time: 0.0048 (0.004653822117684804)  --> TIME: 2025-02-24 18:41:15 -- STEP: 32382/36938 -- GLOBAL_STEP: 438700 | > loss_text_ce: 0.035077519714832306 (0.038586876211107776) | > loss_mel_ce: 3.4437825679779053 (nan) | > loss: 0.01380500104278326 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13649600083446184) | > loader_time: 0.0048 (0.004653994844618202)  --> TIME: 2025-02-24 18:41:24 -- STEP: 32432/36938 -- GLOBAL_STEP: 438750 | > loss_text_ce: 0.03544214740395546 (0.03858630938656362) | > loss_mel_ce: 2.908216714859009 (nan) | > loss: 0.011681186966598034 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13650125956147227) | > loader_time: 0.0046 (0.004654110588048817)  --> TIME: 2025-02-24 18:41:33 -- STEP: 32482/36938 -- GLOBAL_STEP: 438800 | > loss_text_ce: 0.04991661384701729 (0.038586569845475274) | > loss_mel_ce: 2.8259494304656982 (nan) | > loss: 0.011412166990339756 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13649537298936437) | > loader_time: 0.0045 (0.004654139230774091)  --> TIME: 2025-02-24 18:41:43 -- STEP: 32532/36938 -- GLOBAL_STEP: 438850 | > loss_text_ce: 0.04205140843987465 (0.03858451700879755) | > loss_mel_ce: 3.293820381164551 (nan) | > loss: 0.013237587176263332 (nan) | > current_lr: 5e-06 | > step_time: 0.1667 (0.13649632468165568) | > loader_time: 0.0048 (0.004654124787735313)  --> TIME: 2025-02-24 18:41:52 -- STEP: 32582/36938 -- GLOBAL_STEP: 438900 | > loss_text_ce: 0.03333423659205437 (0.03858407206317595) | > loss_mel_ce: 3.387993574142456 (nan) | > loss: 0.013576698489487171 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.1364943256446084) | > loader_time: 0.0046 (0.004654083621632927)  --> TIME: 2025-02-24 18:42:01 -- STEP: 32632/36938 -- GLOBAL_STEP: 438950 | > loss_text_ce: 0.03825751692056656 (0.03858474743381609) | > loss_mel_ce: 3.2422258853912354 (nan) | > loss: 0.013017792254686356 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.1364895047350067) | > loader_time: 0.0048 (0.004654081560695096)  --> TIME: 2025-02-24 18:42:10 -- STEP: 32682/36938 -- GLOBAL_STEP: 439000 | > loss_text_ce: 0.034366756677627563 (0.03858189492065069) | > loss_mel_ce: 3.211074113845825 (nan) | > loss: 0.01287873461842537 (nan) | > current_lr: 5e-06 | > step_time: 0.1891 (0.13648516449275558) | > loader_time: 0.0049 (0.004654058926575763)  --> TIME: 2025-02-24 18:42:19 -- STEP: 32732/36938 -- GLOBAL_STEP: 439050 | > loss_text_ce: 0.04155298322439194 (0.038581020255070854) | > loss_mel_ce: 3.5707030296325684 (nan) | > loss: 0.014334349893033504 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.13648346563605224) | > loader_time: 0.0049 (0.004654028982954759)  --> TIME: 2025-02-24 18:42:29 -- STEP: 32782/36938 -- GLOBAL_STEP: 439100 | > loss_text_ce: 0.03887971490621567 (0.03858096592304304) | > loss_mel_ce: 3.120943784713745 (nan) | > loss: 0.012538982555270195 (nan) | > current_lr: 5e-06 | > step_time: 0.1929 (0.13648710997921715) | > loader_time: 0.0049 (0.004654078179285507)  --> TIME: 2025-02-24 18:42:38 -- STEP: 32832/36938 -- GLOBAL_STEP: 439150 | > loss_text_ce: 0.03238699212670326 (0.038577269340947476) | > loss_mel_ce: 2.7753188610076904 (nan) | > loss: 0.01114169042557478 (nan) | > current_lr: 5e-06 | > step_time: 0.1523 (0.13649031165515277) | > loader_time: 0.0046 (0.00465407739547728)  --> TIME: 2025-02-24 18:42:48 -- STEP: 32882/36938 -- GLOBAL_STEP: 439200 | > loss_text_ce: 0.04038751497864723 (0.03857841785984918) | > loss_mel_ce: 3.405616283416748 (nan) | > loss: 0.013674619607627392 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.1365040096660349) | > loader_time: 0.0048 (0.004654184780477027)  --> TIME: 2025-02-24 18:42:58 -- STEP: 32932/36938 -- GLOBAL_STEP: 439250 | > loss_text_ce: 0.03888913616538048 (0.038578461176275426) | > loss_mel_ce: 2.4499874114990234 (nan) | > loss: 0.009876495227217674 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13650475972538317) | > loader_time: 0.0046 (0.004654256285112937)  --> TIME: 2025-02-24 18:43:07 -- STEP: 32982/36938 -- GLOBAL_STEP: 439300 | > loss_text_ce: 0.03301708400249481 (0.03857916504465784) | > loss_mel_ce: 0.052490234375 (nan) | > loss: 0.00033931477810256183 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13650382465073208) | > loader_time: 0.0047 (0.004654261400987197)  --> TIME: 2025-02-24 18:43:16 -- STEP: 33032/36938 -- GLOBAL_STEP: 439350 | > loss_text_ce: 0.03510110452771187 (0.038576806737460764) | > loss_mel_ce: 1.6673297882080078 (nan) | > loss: 0.006755678448826075 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13650149410849058) | > loader_time: 0.0045 (0.0046542047458227615)  --> TIME: 2025-02-24 18:43:25 -- STEP: 33082/36938 -- GLOBAL_STEP: 439400 | > loss_text_ce: 0.04694104194641113 (0.038577951647769844) | > loss_mel_ce: 2.6377317905426025 (nan) | > loss: 0.010653464123606682 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13650208783332815) | > loader_time: 0.0047 (0.004654134705741124)  --> TIME: 2025-02-24 18:43:35 -- STEP: 33132/36938 -- GLOBAL_STEP: 439450 | > loss_text_ce: 0.026090890169143677 (0.038579028985195135) | > loss_mel_ce: 2.690683126449585 (nan) | > loss: 0.010780849494040012 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13649865377212725) | > loader_time: 0.0044 (0.00465409843930011)  --> TIME: 2025-02-24 18:43:44 -- STEP: 33182/36938 -- GLOBAL_STEP: 439500 | > loss_text_ce: 0.03208812698721886 (0.03857726737168996) | > loss_mel_ce: 2.775036334991455 (nan) | > loss: 0.011139383539557457 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.13649822973542586) | > loader_time: 0.0062 (0.004654081473765275)  --> TIME: 2025-02-24 18:43:53 -- STEP: 33232/36938 -- GLOBAL_STEP: 439550 | > loss_text_ce: 0.04363589361310005 (0.03857682841761238) | > loss_mel_ce: 3.0719683170318604 (nan) | > loss: 0.012363509275019169 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13648967551794813) | > loader_time: 0.0045 (0.004654083499612311)  --> TIME: 2025-02-24 18:44:02 -- STEP: 33282/36938 -- GLOBAL_STEP: 439600 | > loss_text_ce: 0.05345216020941734 (0.03857677637012162) | > loss_mel_ce: 3.3500983715057373 (nan) | > loss: 0.013506153598427773 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.13648619616549615) | > loader_time: 0.0047 (0.004654036928746207)  --> TIME: 2025-02-24 18:44:11 -- STEP: 33332/36938 -- GLOBAL_STEP: 439650 | > loss_text_ce: 0.04004933312535286 (0.03857522803764658) | > loss_mel_ce: 2.7804183959960938 (nan) | > loss: 0.011192332953214645 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13648854066784014) | > loader_time: 0.0047 (0.004654100014784918)  --> TIME: 2025-02-24 18:44:21 -- STEP: 33382/36938 -- GLOBAL_STEP: 439700 | > loss_text_ce: 0.043500952422618866 (0.03857338095526199) | > loss_mel_ce: 3.448824882507324 (nan) | > loss: 0.013858436606824398 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13649395839041434) | > loader_time: 0.0047 (0.004654141828274067)  --> TIME: 2025-02-24 18:44:30 -- STEP: 33432/36938 -- GLOBAL_STEP: 439750 | > loss_text_ce: 0.024303175508975983 (0.03857247673115353) | > loss_mel_ce: 3.3035714626312256 (nan) | > loss: 0.01320585235953331 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13649548098469935) | > loader_time: 0.0045 (0.004654192373951729)  --> TIME: 2025-02-24 18:44:40 -- STEP: 33482/36938 -- GLOBAL_STEP: 439800 | > loss_text_ce: 0.0363481380045414 (0.03857378905066497) | > loss_mel_ce: 3.3947367668151855 (nan) | > loss: 0.013615417294204235 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13649552735467196) | > loader_time: 0.0046 (0.004654164162169786)  --> TIME: 2025-02-24 18:44:49 -- STEP: 33532/36938 -- GLOBAL_STEP: 439850 | > loss_text_ce: 0.035386040806770325 (0.03857105398735274) | > loss_mel_ce: 3.5621612071990967 (nan) | > loss: 0.014275982044637203 (nan) | > current_lr: 5e-06 | > step_time: 0.1891 (0.1364963449414218) | > loader_time: 0.0051 (0.004654191721456962)  --> TIME: 2025-02-24 18:44:58 -- STEP: 33582/36938 -- GLOBAL_STEP: 439900 | > loss_text_ce: 0.03736155480146408 (0.038571226268797755) | > loss_mel_ce: 3.5246188640594482 (nan) | > loss: 0.014134843833744526 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.1364940489945112) | > loader_time: 0.0049 (0.004654264344999269)  --> TIME: 2025-02-24 18:45:08 -- STEP: 33632/36938 -- GLOBAL_STEP: 439950 | > loss_text_ce: 0.03324294835329056 (0.03856917423557017) | > loss_mel_ce: 2.736196756362915 (nan) | > loss: 0.01098984107375145 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13649609747327732) | > loader_time: 0.0045 (0.004654252726219611)  --> TIME: 2025-02-24 18:45:17 -- STEP: 33682/36938 -- GLOBAL_STEP: 440000 | > loss_text_ce: 0.04043224826455116 (0.03856831722417844) | > loss_mel_ce: 2.7221298217773438 (nan) | > loss: 0.010962548665702343 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13649003574085283) | > loader_time: 0.0047 (0.004654242890328945) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_440000.pth  --> TIME: 2025-02-24 18:45:32 -- STEP: 33732/36938 -- GLOBAL_STEP: 440050 | > loss_text_ce: 0.036193519830703735 (0.038569491149319395) | > loss_mel_ce: 2.486299514770508 (nan) | > loss: 0.010009894147515297 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13648409397735273) | > loader_time: 0.0045 (0.004654322741479219)  --> TIME: 2025-02-24 18:45:41 -- STEP: 33782/36938 -- GLOBAL_STEP: 440100 | > loss_text_ce: 0.03768495097756386 (0.03856968396342135) | > loss_mel_ce: 3.410700798034668 (nan) | > loss: 0.01368407066911459 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13648413336514875) | > loader_time: 0.0046 (0.004654310036281488)  --> TIME: 2025-02-24 18:45:50 -- STEP: 33832/36938 -- GLOBAL_STEP: 440150 | > loss_text_ce: 0.03589611127972603 (0.03857043394735769) | > loss_mel_ce: 3.447791337966919 (nan) | > loss: 0.013824157416820526 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.13647851387387244) | > loader_time: 0.0046 (0.004654283753577992)  --> TIME: 2025-02-24 18:46:00 -- STEP: 33882/36938 -- GLOBAL_STEP: 440200 | > loss_text_ce: 0.050142958760261536 (0.038570275274340406) | > loss_mel_ce: 2.7559990882873535 (nan) | > loss: 0.011135485023260117 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13648152060762359) | > loader_time: 0.0046 (0.004654311231679947)  --> TIME: 2025-02-24 18:46:09 -- STEP: 33932/36938 -- GLOBAL_STEP: 440250 | > loss_text_ce: 0.042467568069696426 (0.03857237247356139) | > loss_mel_ce: 2.7863473892211914 (nan) | > loss: 0.011225457303225994 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13647782175127524) | > loader_time: 0.0045 (0.004654310263371021)  --> TIME: 2025-02-24 18:46:18 -- STEP: 33982/36938 -- GLOBAL_STEP: 440300 | > loss_text_ce: 0.03812882676720619 (0.03857460571469422) | > loss_mel_ce: 3.3296921253204346 (nan) | > loss: 0.013364369980990887 (nan) | > current_lr: 5e-06 | > step_time: 0.1868 (0.1364822364752912) | > loader_time: 0.0051 (0.004654607780685119)  --> TIME: 2025-02-24 18:46:28 -- STEP: 34032/36938 -- GLOBAL_STEP: 440350 | > loss_text_ce: 0.03508329764008522 (0.038573118645487015) | > loss_mel_ce: 3.3547115325927734 (nan) | > loss: 0.013451567851006985 (nan) | > current_lr: 5e-06 | > step_time: 0.145 (0.1364831787897045) | > loader_time: 0.005 (0.004654610239680429)  --> TIME: 2025-02-24 18:46:37 -- STEP: 34082/36938 -- GLOBAL_STEP: 440400 | > loss_text_ce: 0.033484067767858505 (0.03857173659534272) | > loss_mel_ce: 2.5676963329315186 (nan) | > loss: 0.010322144255042076 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13648685836909552) | > loader_time: 0.0046 (0.004654602205296546)  --> TIME: 2025-02-24 18:46:47 -- STEP: 34132/36938 -- GLOBAL_STEP: 440450 | > loss_text_ce: 0.037008605897426605 (0.0385714914523064) | > loss_mel_ce: 3.343776226043701 (nan) | > loss: 0.013415813446044922 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.1364945177175261) | > loader_time: 0.0047 (0.00465463968202322)  --> TIME: 2025-02-24 18:46:57 -- STEP: 34182/36938 -- GLOBAL_STEP: 440500 | > loss_text_ce: 0.037170108407735825 (0.03856993850759677) | > loss_mel_ce: 2.237696886062622 (nan) | > loss: 0.009027251042425632 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13650403091957897) | > loader_time: 0.0046 (0.004654727338677418)  --> TIME: 2025-02-24 18:47:06 -- STEP: 34232/36938 -- GLOBAL_STEP: 440550 | > loss_text_ce: 0.051512978971004486 (0.038569249783446294) | > loss_mel_ce: 3.431631565093994 (nan) | > loss: 0.01382200326770544 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.13650583621218138) | > loader_time: 0.0047 (0.0046547199206587574)  --> TIME: 2025-02-24 18:47:15 -- STEP: 34282/36938 -- GLOBAL_STEP: 440600 | > loss_text_ce: 0.03457583114504814 (0.03856851906934514) | > loss_mel_ce: 0.7285895347595215 (nan) | > loss: 0.0030284342356026173 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13650895117823234) | > loader_time: 0.0046 (0.0046546999642190015)  --> TIME: 2025-02-24 18:47:25 -- STEP: 34332/36938 -- GLOBAL_STEP: 440650 | > loss_text_ce: 0.021282309666275978 (0.038566267407031285) | > loss_mel_ce: 2.102086305618286 (nan) | > loss: 0.008426065556704998 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13650654420754926) | > loader_time: 0.0045 (0.004654825705964172)  --> TIME: 2025-02-24 18:47:34 -- STEP: 34382/36938 -- GLOBAL_STEP: 440700 | > loss_text_ce: 0.05557830631732941 (0.038567701254399095) | > loss_mel_ce: 3.183018445968628 (nan) | > loss: 0.012851574458181858 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13650940532605596) | > loader_time: 0.0046 (0.004654858022326698)  --> TIME: 2025-02-24 18:47:44 -- STEP: 34432/36938 -- GLOBAL_STEP: 440750 | > loss_text_ce: 0.044409845024347305 (0.038566811559968536) | > loss_mel_ce: 3.1378509998321533 (nan) | > loss: 0.012628019787371159 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.13651597560089132) | > loader_time: 0.005 (0.00465487060742974)  --> TIME: 2025-02-24 18:47:53 -- STEP: 34482/36938 -- GLOBAL_STEP: 440800 | > loss_text_ce: 0.04108332842588425 (0.038567964032639825) | > loss_mel_ce: 1.4891024827957153 (nan) | > loss: 0.006072166375815868 (nan) | > current_lr: 5e-06 | > step_time: 0.1617 (0.13651151633181086) | > loader_time: 0.0047 (0.004654875107800564)  --> TIME: 2025-02-24 18:48:02 -- STEP: 34532/36938 -- GLOBAL_STEP: 440850 | > loss_text_ce: 0.0451207309961319 (0.038565953080157825) | > loss_mel_ce: 2.1723098754882812 (nan) | > loss: 0.008799328468739986 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.13650886847883706) | > loader_time: 0.0045 (0.004654823836177818)  --> TIME: 2025-02-24 18:48:11 -- STEP: 34582/36938 -- GLOBAL_STEP: 440900 | > loss_text_ce: 0.02879960462450981 (0.03856494302653651) | > loss_mel_ce: 1.8592398166656494 (nan) | > loss: 0.0074922204948961735 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13650399337212857) | > loader_time: 0.0045 (0.004654812899673338)  --> TIME: 2025-02-24 18:48:21 -- STEP: 34632/36938 -- GLOBAL_STEP: 440950 | > loss_text_ce: 0.029334647580981255 (0.03856605445005761) | > loss_mel_ce: 3.3272342681884766 (nan) | > loss: 0.013319718651473522 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13650766929982072) | > loader_time: 0.0046 (0.004654774058082709)  --> TIME: 2025-02-24 18:48:30 -- STEP: 34682/36938 -- GLOBAL_STEP: 441000 | > loss_text_ce: 0.027742378413677216 (0.03856471366256599) | > loss_mel_ce: 3.5873117446899414 (nan) | > loss: 0.014345454052090645 (nan) | > current_lr: 5e-06 | > step_time: 0.1626 (0.1365082768872569) | > loader_time: 0.0047 (0.0046547229957789775)  --> TIME: 2025-02-24 18:48:40 -- STEP: 34732/36938 -- GLOBAL_STEP: 441050 | > loss_text_ce: 0.03838988393545151 (0.038563065660898) | > loss_mel_ce: 2.9902114868164062 (nan) | > loss: 0.012018260546028614 (nan) | > current_lr: 5e-06 | > step_time: 0.1659 (0.13651846725243585) | > loader_time: 0.0047 (0.00465482271556468)  --> TIME: 2025-02-24 18:48:49 -- STEP: 34782/36938 -- GLOBAL_STEP: 441100 | > loss_text_ce: 0.05654733628034592 (0.03856470512720941) | > loss_mel_ce: 3.213761329650879 (nan) | > loss: 0.01297741662710905 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1365174148802613) | > loader_time: 0.0046 (0.004654811281475471)  --> TIME: 2025-02-24 18:48:59 -- STEP: 34832/36938 -- GLOBAL_STEP: 441150 | > loss_text_ce: 0.04321422427892685 (0.038564823013276905) | > loss_mel_ce: 3.354506015777588 (nan) | > loss: 0.013483017683029175 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13652669883097468) | > loader_time: 0.0048 (0.004654801522968143)  --> TIME: 2025-02-24 18:49:08 -- STEP: 34882/36938 -- GLOBAL_STEP: 441200 | > loss_text_ce: 0.04557080194354057 (0.03856382257503785) | > loss_mel_ce: 3.295973777770996 (nan) | > loss: 0.013260098174214363 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13652088032301424) | > loader_time: 0.0048 (0.004654734870530532)  --> TIME: 2025-02-24 18:49:17 -- STEP: 34932/36938 -- GLOBAL_STEP: 441250 | > loss_text_ce: 0.04141087085008621 (0.03856334489566867) | > loss_mel_ce: 2.6733264923095703 (nan) | > loss: 0.010772768408060074 (nan) | > current_lr: 5e-06 | > step_time: 0.1522 (0.13651387852300026) | > loader_time: 0.0046 (0.004654635013098538)  --> TIME: 2025-02-24 18:49:26 -- STEP: 34982/36938 -- GLOBAL_STEP: 441300 | > loss_text_ce: 0.03983962908387184 (0.038564894975032994) | > loss_mel_ce: 0.0475667305290699 (nan) | > loss: 0.00034685066202655435 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.1365123545828399) | > loader_time: 0.0047 (0.004654651794730222)  --> TIME: 2025-02-24 18:49:35 -- STEP: 35032/36938 -- GLOBAL_STEP: 441350 | > loss_text_ce: 0.038542211055755615 (0.0385646066235153) | > loss_mel_ce: 3.8556933403015137 (nan) | > loss: 0.015453317202627659 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13651376035720392) | > loader_time: 0.0045 (0.004654620078416375)  --> TIME: 2025-02-24 18:49:45 -- STEP: 35082/36938 -- GLOBAL_STEP: 441400 | > loss_text_ce: 0.044828981161117554 (0.038563802927258056) | > loss_mel_ce: 3.153794765472412 (nan) | > loss: 0.01269295159727335 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13651435808528986) | > loader_time: 0.0046 (0.004654564435313089)  --> TIME: 2025-02-24 18:49:54 -- STEP: 35132/36938 -- GLOBAL_STEP: 441450 | > loss_text_ce: 0.035787828266620636 (0.03856089806755078) | > loss_mel_ce: 2.919442653656006 (nan) | > loss: 0.01172710582613945 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13651335679845525) | > loader_time: 0.0044 (0.0046545084280426585)  --> TIME: 2025-02-24 18:50:03 -- STEP: 35182/36938 -- GLOBAL_STEP: 441500 | > loss_text_ce: 0.038576800376176834 (0.03856317887994271) | > loss_mel_ce: 3.0427095890045166 (nan) | > loss: 0.012227327562868595 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13651806442403253) | > loader_time: 0.0046 (0.00465446035286356)  --> TIME: 2025-02-24 18:50:13 -- STEP: 35232/36938 -- GLOBAL_STEP: 441550 | > loss_text_ce: 0.035530734807252884 (0.038563416253176476) | > loss_mel_ce: 3.4560306072235107 (nan) | > loss: 0.013855403289198875 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.1365210414203488) | > loader_time: 0.0047 (0.004654422382080602)  --> TIME: 2025-02-24 18:50:22 -- STEP: 35282/36938 -- GLOBAL_STEP: 441600 | > loss_text_ce: 0.03856952115893364 (0.03856223912217089) | > loss_mel_ce: 2.7779903411865234 (nan) | > loss: 0.011176825501024723 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13652624749967074) | > loader_time: 0.0045 (0.0046544535874604295)  --> TIME: 2025-02-24 18:50:31 -- STEP: 35332/36938 -- GLOBAL_STEP: 441650 | > loss_text_ce: 0.035151418298482895 (0.03856141287143897) | > loss_mel_ce: 1.3674044609069824 (nan) | > loss: 0.005565698258578777 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13652078333141812) | > loader_time: 0.0048 (0.0046543970149059925)  --> TIME: 2025-02-24 18:50:41 -- STEP: 35382/36938 -- GLOBAL_STEP: 441700 | > loss_text_ce: 0.0304107703268528 (0.038562186141617646) | > loss_mel_ce: 3.0721704959869385 (nan) | > loss: 0.01231183111667633 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13651936605836382) | > loader_time: 0.0046 (0.004654425667986)  --> TIME: 2025-02-24 18:50:50 -- STEP: 35432/36938 -- GLOBAL_STEP: 441750 | > loss_text_ce: 0.0378432460129261 (0.03856322687198312) | > loss_mel_ce: 1.653350830078125 (nan) | > loss: 0.006711088120937347 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1365197280253214) | > loader_time: 0.0045 (0.00465441339574192)  --> TIME: 2025-02-24 18:51:00 -- STEP: 35482/36938 -- GLOBAL_STEP: 441800 | > loss_text_ce: 0.04734547808766365 (0.038562236327192294) | > loss_mel_ce: 2.2829277515411377 (nan) | > loss: 0.009247115813195705 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13652712202456538) | > loader_time: 0.0046 (0.00465450174113312)  --> TIME: 2025-02-24 18:51:09 -- STEP: 35532/36938 -- GLOBAL_STEP: 441850 | > loss_text_ce: 0.039899520576000214 (0.03856234857634295) | > loss_mel_ce: 2.8862690925598145 (nan) | > loss: 0.01161178108304739 (nan) | > current_lr: 5e-06 | > step_time: 0.146 (0.13653343580440983) | > loader_time: 0.005 (0.00465452826051285)  --> TIME: 2025-02-24 18:51:19 -- STEP: 35582/36938 -- GLOBAL_STEP: 441900 | > loss_text_ce: 0.04090862348675728 (0.038560593940560994) | > loss_mel_ce: 3.7219502925872803 (nan) | > loss: 0.014931980520486832 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13653495846306105) | > loader_time: 0.0045 (0.004654548715080378)  --> TIME: 2025-02-24 18:51:28 -- STEP: 35632/36938 -- GLOBAL_STEP: 441950 | > loss_text_ce: 0.02710701897740364 (0.038559419503964555) | > loss_mel_ce: 0.008819580078125 (nan) | > loss: 0.00014256588474381715 (nan) | > current_lr: 5e-06 | > step_time: 0.1634 (0.13653751861824318) | > loader_time: 0.0047 (0.004654620828037588)  --> TIME: 2025-02-24 18:51:37 -- STEP: 35682/36938 -- GLOBAL_STEP: 442000 | > loss_text_ce: 0.04580308496952057 (0.03856036782970054) | > loss_mel_ce: 2.599254608154297 (nan) | > loss: 0.010496261529624462 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.1365274934515843) | > loader_time: 0.0045 (0.004654594216336857)  --> TIME: 2025-02-24 18:51:47 -- STEP: 35732/36938 -- GLOBAL_STEP: 442050 | > loss_text_ce: 0.041326768696308136 (0.03855963114680962) | > loss_mel_ce: 0.05634765699505806 (nan) | > loss: 0.00038759695598855615 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1365342838865882) | > loader_time: 0.0046 (0.0046546112499538175)  --> TIME: 2025-02-24 18:51:56 -- STEP: 35782/36938 -- GLOBAL_STEP: 442100 | > loss_text_ce: 0.04336770996451378 (0.038556959843758086) | > loss_mel_ce: 1.5369516611099243 (nan) | > loss: 0.006271108984947205 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13653295085173123) | > loader_time: 0.0048 (0.004654566602412139)  --> TIME: 2025-02-24 18:52:05 -- STEP: 35832/36938 -- GLOBAL_STEP: 442150 | > loss_text_ce: 0.039651982486248016 (0.038556702359901586) | > loss_mel_ce: 3.2362165451049805 (nan) | > loss: 0.012999478727579117 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13653638890396982) | > loader_time: 0.0047 (0.00465457739907734)  --> TIME: 2025-02-24 18:52:14 -- STEP: 35882/36938 -- GLOBAL_STEP: 442200 | > loss_text_ce: 0.033087607473134995 (0.03855698037383652) | > loss_mel_ce: 3.202848196029663 (nan) | > loss: 0.012841015122830868 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13653336972750754) | > loader_time: 0.0045 (0.0046546025510336925)  --> TIME: 2025-02-24 18:52:24 -- STEP: 35932/36938 -- GLOBAL_STEP: 442250 | > loss_text_ce: 0.03666343539953232 (0.03855628989488435) | > loss_mel_ce: 2.994736433029175 (nan) | > loss: 0.012029365636408329 (nan) | > current_lr: 5e-06 | > step_time: 0.1077 (0.13653247289796713) | > loader_time: 0.0044 (0.0046546298159956395)  --> TIME: 2025-02-24 18:52:33 -- STEP: 35982/36938 -- GLOBAL_STEP: 442300 | > loss_text_ce: 0.034110330045223236 (0.03855440240955074) | > loss_mel_ce: 3.1710052490234375 (nan) | > loss: 0.012718712911009789 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.1365376786892538) | > loader_time: 0.0045 (0.004654651273649288)  --> TIME: 2025-02-24 18:52:43 -- STEP: 36032/36938 -- GLOBAL_STEP: 442350 | > loss_text_ce: 0.03602612018585205 (0.03855421496650328) | > loss_mel_ce: 3.115576982498169 (nan) | > loss: 0.012506362982094288 (nan) | > current_lr: 5e-06 | > step_time: 0.1566 (0.13654570884301287) | > loader_time: 0.0046 (0.004654665432910098)  --> TIME: 2025-02-24 18:52:52 -- STEP: 36082/36938 -- GLOBAL_STEP: 442400 | > loss_text_ce: 0.0378371998667717 (0.03855482464484125) | > loss_mel_ce: 3.306330680847168 (nan) | > loss: 0.013270508497953415 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13653973250264673) | > loader_time: 0.0047 (0.004654623936023618)  --> TIME: 2025-02-24 18:53:01 -- STEP: 36132/36938 -- GLOBAL_STEP: 442450 | > loss_text_ce: 0.03355734422802925 (0.038553005284001365) | > loss_mel_ce: 3.110058546066284 (nan) | > loss: 0.0124746672809124 (nan) | > current_lr: 5e-06 | > step_time: 0.1679 (0.13653910136310002) | > loader_time: 0.0047 (0.004654595962225851)  --> TIME: 2025-02-24 18:53:11 -- STEP: 36182/36938 -- GLOBAL_STEP: 442500 | > loss_text_ce: 0.035364553332328796 (0.038550225755800166) | > loss_mel_ce: 3.0330452919006348 (nan) | > loss: 0.012176230549812317 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13654056166569384) | > loader_time: 0.0047 (0.004654628813652244)  --> TIME: 2025-02-24 18:53:20 -- STEP: 36232/36938 -- GLOBAL_STEP: 442550 | > loss_text_ce: 0.040115293115377426 (0.03855187321664942) | > loss_mel_ce: 3.249211072921753 (nan) | > loss: 0.013052883557975292 (nan) | > current_lr: 5e-06 | > step_time: 0.1742 (0.13654120639213396) | > loader_time: 0.005 (0.004654676794716274)  --> TIME: 2025-02-24 18:53:29 -- STEP: 36282/36938 -- GLOBAL_STEP: 442600 | > loss_text_ce: 0.03880855068564415 (0.03855049734304018) | > loss_mel_ce: 2.3275437355041504 (nan) | > loss: 0.009390287101268768 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13653946866485053) | > loader_time: 0.0047 (0.004654678467267492)  --> TIME: 2025-02-24 18:53:39 -- STEP: 36332/36938 -- GLOBAL_STEP: 442650 | > loss_text_ce: 0.03843905031681061 (0.03855105488928482) | > loss_mel_ce: 2.1263511180877686 (nan) | > loss: 0.008590437471866608 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13654346490607222) | > loader_time: 0.0044 (0.004654754596726172)  --> TIME: 2025-02-24 18:53:48 -- STEP: 36382/36938 -- GLOBAL_STEP: 442700 | > loss_text_ce: 0.034836601465940475 (0.03855098379611751) | > loss_mel_ce: 3.0882349014282227 (nan) | > loss: 0.01239314116537571 (nan) | > current_lr: 5e-06 | > step_time: 0.1556 (0.13653623304151666) | > loader_time: 0.0047 (0.004654890930899684)  --> TIME: 2025-02-24 18:53:57 -- STEP: 36432/36938 -- GLOBAL_STEP: 442750 | > loss_text_ce: 0.04713544622063637 (0.0385498330117211) | > loss_mel_ce: 3.573613166809082 (nan) | > loss: 0.01436805073171854 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13653423299143455) | > loader_time: 0.0046 (0.004654950591938828)  --> TIME: 2025-02-24 18:54:06 -- STEP: 36482/36938 -- GLOBAL_STEP: 442800 | > loss_text_ce: 0.043917879462242126 (0.03855038160123996) | > loss_mel_ce: 3.157658576965332 (nan) | > loss: 0.012704669497907162 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.1365324774173254) | > loader_time: 0.0047 (0.004655037511304111)  --> TIME: 2025-02-24 18:54:16 -- STEP: 36532/36938 -- GLOBAL_STEP: 442850 | > loss_text_ce: 0.035838544368743896 (0.038549308864250786) | > loss_mel_ce: 3.452690839767456 (nan) | > loss: 0.013843371532857418 (nan) | > current_lr: 5e-06 | > step_time: 0.1262 (0.13653440370651646) | > loader_time: 0.0045 (0.004655023935811494)  --> TIME: 2025-02-24 18:54:25 -- STEP: 36582/36938 -- GLOBAL_STEP: 442900 | > loss_text_ce: 0.047516826540231705 (0.03855011518790061) | > loss_mel_ce: 3.3056912422180176 (nan) | > loss: 0.013306382112205029 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13653292373612239) | > loader_time: 0.0045 (0.004655002941552674)  --> TIME: 2025-02-24 18:54:34 -- STEP: 36632/36938 -- GLOBAL_STEP: 442950 | > loss_text_ce: 0.034393008798360825 (0.038547831023849115) | > loss_mel_ce: 3.479658365249634 (nan) | > loss: 0.013944649137556553 (nan) | > current_lr: 5e-06 | > step_time: 0.1976 (0.13653601446014327) | > loader_time: 0.0049 (0.004655058505259696)  --> TIME: 2025-02-24 18:54:44 -- STEP: 36682/36938 -- GLOBAL_STEP: 443000 | > loss_text_ce: 0.028687257319688797 (0.03854935538807939) | > loss_mel_ce: 3.066495656967163 (nan) | > loss: 0.012282472103834152 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13653806713621774) | > loader_time: 0.0048 (0.004655048953922042)  --> TIME: 2025-02-24 18:54:53 -- STEP: 36732/36938 -- GLOBAL_STEP: 443050 | > loss_text_ce: 0.037627507001161575 (0.03854869095802107) | > loss_mel_ce: 2.9053127765655518 (nan) | > loss: 0.011678335256874561 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13653396231461204) | > loader_time: 0.0046 (0.004655006189407597)  --> TIME: 2025-02-24 18:55:03 -- STEP: 36782/36938 -- GLOBAL_STEP: 443100 | > loss_text_ce: 0.032124560326337814 (0.03854680335019843) | > loss_mel_ce: 3.054567337036133 (nan) | > loss: 0.012248777784407139 (nan) | > current_lr: 5e-06 | > step_time: 0.1504 (0.13654352018102178) | > loader_time: 0.0047 (0.004655050327802374)  --> TIME: 2025-02-24 18:55:11 -- STEP: 36832/36938 -- GLOBAL_STEP: 443150 | > loss_text_ce: 0.03488491103053093 (0.03854693451044293) | > loss_mel_ce: 3.6489059925079346 (nan) | > loss: 0.014618218876421452 (nan) | > current_lr: 5e-06 | > step_time: 0.1583 (0.13653260416356966) | > loader_time: 0.0046 (0.00465503790060611)  --> TIME: 2025-02-24 18:55:21 -- STEP: 36882/36938 -- GLOBAL_STEP: 443200 | > loss_text_ce: 0.041668668389320374 (0.03854496095421043) | > loss_mel_ce: 2.8083674907684326 (nan) | > loss: 0.011309667490422726 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.1365321360701799) | > loader_time: 0.0046 (0.004655243007009765)  --> TIME: 2025-02-24 18:55:30 -- STEP: 36932/36938 -- GLOBAL_STEP: 443250 | > loss_text_ce: 0.037023987621068954 (0.03854291776709006) | > loss_mel_ce: 2.732140302658081 (nan) | > loss: 0.010988747701048851 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13653622456778602) | > loader_time: 0.0044 (0.004655423375337011)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.020680861897987908 (+0.00011209096058762899) | > avg_loss_text_ce: 0.037509511040505206 (-0.0006441365245102651) | > avg_loss_mel_ce: 2.705216262305137 (-0.026163989659583553) | > avg_loss: 2.7427257783342105 (-0.026808118466103537) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_443256.pth  > EPOCH: 13/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-24 18:55:57)   --> TIME: 2025-02-24 18:56:05 -- STEP: 44/36938 -- GLOBAL_STEP: 443300 | > loss_text_ce: 0.037878409028053284 (0.038622443013909186) | > loss_mel_ce: 3.2042205333709717 (2.8298455568199805) | > loss: 0.01286547351628542 (0.011382810192034494) | > current_lr: 5e-06 | > step_time: 0.1351 (0.1359303972937844) | > loader_time: 0.0047 (0.004906383427706631)  --> TIME: 2025-02-24 18:56:14 -- STEP: 94/36938 -- GLOBAL_STEP: 443350 | > loss_text_ce: 0.03758205473423004 (0.03832476619789575) | > loss_mel_ce: 3.1092240810394287 (2.793631561734575) | > loss: 0.012487326748669147 (0.011237922590225935) | > current_lr: 5e-06 | > step_time: 0.1773 (0.13415197362291045) | > loader_time: 0.0048 (0.004788928843559103)  --> TIME: 2025-02-24 18:56:23 -- STEP: 144/36938 -- GLOBAL_STEP: 443400 | > loss_text_ce: 0.03642730787396431 (0.03816291573457419) | > loss_mel_ce: 3.4524648189544678 (nan) | > loss: 0.013844810426235199 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13457677265008292) | > loader_time: 0.0045 (0.0047346850236256914)  --> TIME: 2025-02-24 18:56:33 -- STEP: 194/36938 -- GLOBAL_STEP: 443450 | > loss_text_ce: 0.02992095798254013 (0.03810097450947332) | > loss_mel_ce: 2.418858528137207 (nan) | > loss: 0.009717379696667194 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13414688946045555) | > loader_time: 0.0046 (0.004715724089710982)  --> TIME: 2025-02-24 18:56:42 -- STEP: 244/36938 -- GLOBAL_STEP: 443500 | > loss_text_ce: 0.027083473280072212 (0.037885960756388844) | > loss_mel_ce: 2.466871976852417 (nan) | > loss: 0.009896649047732353 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.134535503192026) | > loader_time: 0.0046 (0.0047072064681131336)  --> TIME: 2025-02-24 18:56:51 -- STEP: 294/36938 -- GLOBAL_STEP: 443550 | > loss_text_ce: 0.0354531928896904 (0.03786429843301473) | > loss_mel_ce: 2.4497478008270264 (nan) | > loss: 0.009861908853054047 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13420333100013987) | > loader_time: 0.0045 (0.00469089040950853)  --> TIME: 2025-02-24 18:57:00 -- STEP: 344/36938 -- GLOBAL_STEP: 443600 | > loss_text_ce: 0.026647217571735382 (0.038001467251803624) | > loss_mel_ce: 2.678464889526367 (nan) | > loss: 0.010734573006629944 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.1345520435377609) | > loader_time: 0.0046 (0.004679275806560068)  --> TIME: 2025-02-24 18:57:10 -- STEP: 394/36938 -- GLOBAL_STEP: 443650 | > loss_text_ce: 0.03178668022155762 (0.03762067814220481) | > loss_mel_ce: 1.675388216972351 (nan) | > loss: 0.006774503737688065 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13510368862733008) | > loader_time: 0.0045 (0.004671295887322589)  --> TIME: 2025-02-24 18:57:19 -- STEP: 444/36938 -- GLOBAL_STEP: 443700 | > loss_text_ce: 0.023698316887021065 (0.03758664270611228) | > loss_mel_ce: 0.890907883644104 (nan) | > loss: 0.0036293899174779654 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.1354993162928399) | > loader_time: 0.0045 (0.004662682881226405)  --> TIME: 2025-02-24 18:57:28 -- STEP: 494/36938 -- GLOBAL_STEP: 443750 | > loss_text_ce: 0.04475288465619087 (0.0374345938874944) | > loss_mel_ce: 1.686773657798767 (nan) | > loss: 0.006871137302368879 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13562967516632682) | > loader_time: 0.0045 (0.004657571614995171)  --> TIME: 2025-02-24 18:57:38 -- STEP: 544/36938 -- GLOBAL_STEP: 443800 | > loss_text_ce: 0.052345484495162964 (0.03771815052844914) | > loss_mel_ce: 3.2344419956207275 (nan) | > loss: 0.013042808510363102 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.1359369342817978) | > loader_time: 0.0046 (0.004650937283740319)  --> TIME: 2025-02-24 18:57:47 -- STEP: 594/36938 -- GLOBAL_STEP: 443850 | > loss_text_ce: 0.04364882409572601 (0.03788503924516418) | > loss_mel_ce: 11.5 (nan) | > loss: 0.045808132737874985 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13588186185367962) | > loader_time: 0.0045 (0.004646935848274611)  --> TIME: 2025-02-24 18:57:57 -- STEP: 644/36938 -- GLOBAL_STEP: 443900 | > loss_text_ce: 0.03811345621943474 (0.038089652973811505) | > loss_mel_ce: 2.655319929122925 (nan) | > loss: 0.010688227601349354 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.13615235815877474) | > loader_time: 0.0046 (0.004644744144463388)  --> TIME: 2025-02-24 18:58:06 -- STEP: 694/36938 -- GLOBAL_STEP: 443950 | > loss_text_ce: 0.03044901229441166 (0.038058808540404755) | > loss_mel_ce: 2.9767062664031982 (nan) | > loss: 0.011933156289160252 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1359471281942096) | > loader_time: 0.0046 (0.004643442994922994)  --> TIME: 2025-02-24 18:58:15 -- STEP: 744/36938 -- GLOBAL_STEP: 444000 | > loss_text_ce: 0.04399865120649338 (0.03803916039916701) | > loss_mel_ce: 2.7479734420776367 (nan) | > loss: 0.011079255491495132 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13574129983942976) | > loader_time: 0.0047 (0.004640310682276238)  --> TIME: 2025-02-24 18:58:24 -- STEP: 794/36938 -- GLOBAL_STEP: 444050 | > loss_text_ce: 0.02750954031944275 (0.03799879367484408) | > loss_mel_ce: 2.989274740219116 (nan) | > loss: 0.011971366591751575 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13593595394259422) | > loader_time: 0.0046 (0.004637957219813269)  --> TIME: 2025-02-24 18:58:34 -- STEP: 844/36938 -- GLOBAL_STEP: 444100 | > loss_text_ce: 0.036722905933856964 (0.0379631131207703) | > loss_mel_ce: 3.024148941040039 (nan) | > loss: 0.012146317400038242 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13649326817119278) | > loader_time: 0.0045 (0.004636189666404541)  --> TIME: 2025-02-24 18:58:44 -- STEP: 894/36938 -- GLOBAL_STEP: 444150 | > loss_text_ce: 0.03574605658650398 (0.03809117278505027) | > loss_mel_ce: 3.043928384780884 (nan) | > loss: 0.012220931239426136 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.1367439442299622) | > loader_time: 0.0045 (0.004634710232950164)  --> TIME: 2025-02-24 18:58:53 -- STEP: 944/36938 -- GLOBAL_STEP: 444200 | > loss_text_ce: 0.040596842765808105 (0.03807668017531289) | > loss_mel_ce: 2.621553421020508 (nan) | > loss: 0.010564089752733707 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13647487784846335) | > loader_time: 0.0046 (0.004631009899963767)  --> TIME: 2025-02-24 18:59:02 -- STEP: 994/36938 -- GLOBAL_STEP: 444250 | > loss_text_ce: 0.04066358879208565 (0.0380460637128749) | > loss_mel_ce: 3.164119005203247 (nan) | > loss: 0.01271739136427641 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.1363029187354042) | > loader_time: 0.0046 (0.004628285315914653)  --> TIME: 2025-02-24 18:59:11 -- STEP: 1044/36938 -- GLOBAL_STEP: 444300 | > loss_text_ce: 0.03698823228478432 (0.037968996753452285) | > loss_mel_ce: 3.348942756652832 (nan) | > loss: 0.013436234556138515 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.1364193972043149) | > loader_time: 0.0047 (0.004626034776826472)  --> TIME: 2025-02-24 18:59:21 -- STEP: 1094/36938 -- GLOBAL_STEP: 444350 | > loss_text_ce: 0.05294487252831459 (0.038066933687497105) | > loss_mel_ce: 3.3370168209075928 (nan) | > loss: 0.013452230021357536 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13635766658748077) | > loader_time: 0.0045 (0.004623918036438208)  --> TIME: 2025-02-24 18:59:30 -- STEP: 1144/36938 -- GLOBAL_STEP: 444400 | > loss_text_ce: 0.0280331838876009 (0.0381021139999995) | > loss_mel_ce: 3.3617939949035645 (nan) | > loss: 0.013451696373522282 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13639859448779698) | > loader_time: 0.0046 (0.004622700956317924)  --> TIME: 2025-02-24 18:59:39 -- STEP: 1194/36938 -- GLOBAL_STEP: 444450 | > loss_text_ce: 0.04533093050122261 (0.038091274438553833) | > loss_mel_ce: 2.860379457473755 (nan) | > loss: 0.011530597694218159 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13635458978176904) | > loader_time: 0.0045 (0.0046201405213705795)  --> TIME: 2025-02-24 18:59:49 -- STEP: 1244/36938 -- GLOBAL_STEP: 444500 | > loss_text_ce: 0.027413509786128998 (0.03806611367312658) | > loss_mel_ce: 1.3856964111328125 (nan) | > loss: 0.005607579369097948 (nan) | > current_lr: 5e-06 | > step_time: 0.1267 (0.13649700577236043) | > loader_time: 0.0045 (0.004619043548007477)  --> TIME: 2025-02-24 18:59:58 -- STEP: 1294/36938 -- GLOBAL_STEP: 444550 | > loss_text_ce: 0.03779850900173187 (0.03800469351762551) | > loss_mel_ce: 3.39597749710083 (nan) | > loss: 0.013626095838844776 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13634011365899326) | > loader_time: 0.0046 (0.004624046903489357)  --> TIME: 2025-02-24 19:00:07 -- STEP: 1344/36938 -- GLOBAL_STEP: 444600 | > loss_text_ce: 0.03807536140084267 (0.037957585760436625) | > loss_mel_ce: 2.4675092697143555 (nan) | > loss: 0.009942797012627125 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13610329656373849) | > loader_time: 0.0045 (0.004625009816317328)  --> TIME: 2025-02-24 19:00:16 -- STEP: 1394/36938 -- GLOBAL_STEP: 444650 | > loss_text_ce: 0.0426948256790638 (0.037922020910716656) | > loss_mel_ce: 2.966968297958374 (nan) | > loss: 0.011943108402192593 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13619343690584815) | > loader_time: 0.0047 (0.004627053330584267)  --> TIME: 2025-02-24 19:00:26 -- STEP: 1444/36938 -- GLOBAL_STEP: 444700 | > loss_text_ce: 0.03946170583367348 (0.037989786178534046) | > loss_mel_ce: 2.50091552734375 (nan) | > loss: 0.010080861859023571 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13626523774086244) | > loader_time: 0.0045 (0.004628652350724235)  --> TIME: 2025-02-24 19:00:35 -- STEP: 1494/36938 -- GLOBAL_STEP: 444750 | > loss_text_ce: 0.043042294681072235 (0.03805824068600592) | > loss_mel_ce: 0.02978515625 (nan) | > loss: 0.00028899783501401544 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13648388854949794) | > loader_time: 0.0046 (0.004629999598665246)  --> TIME: 2025-02-24 19:00:45 -- STEP: 1544/36938 -- GLOBAL_STEP: 444800 | > loss_text_ce: 0.04943013936281204 (0.038063380069828) | > loss_mel_ce: 3.5676026344299316 (nan) | > loss: 0.014353305101394653 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13649185858859886) | > loader_time: 0.0045 (0.004629238436259131)  --> TIME: 2025-02-24 19:00:54 -- STEP: 1594/36938 -- GLOBAL_STEP: 444850 | > loss_text_ce: 0.0313975028693676 (0.03809350472728871) | > loss_mel_ce: 2.7088067531585693 (nan) | > loss: 0.010873827151954174 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13650900910759184) | > loader_time: 0.0046 (0.004629574127149396)  --> TIME: 2025-02-24 19:01:03 -- STEP: 1644/36938 -- GLOBAL_STEP: 444900 | > loss_text_ce: 0.03424072265625 (0.0381100122701295) | > loss_mel_ce: 0.03657975047826767 (nan) | > loss: 0.00028103363001719117 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13643760243181455) | > loader_time: 0.0044 (0.004628926153020552)  --> TIME: 2025-02-24 19:01:12 -- STEP: 1694/36938 -- GLOBAL_STEP: 444950 | > loss_text_ce: 0.04239959642291069 (0.03815672895183883) | > loss_mel_ce: 2.905485153198242 (nan) | > loss: 0.01169795636087656 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13642094506845812) | > loader_time: 0.0047 (0.0046289627217064155)  --> TIME: 2025-02-24 19:01:21 -- STEP: 1744/36938 -- GLOBAL_STEP: 445000 | > loss_text_ce: 0.03488728031516075 (0.03814155038194134) | > loss_mel_ce: 2.547532320022583 (nan) | > loss: 0.010247697122395039 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.13629728760741128) | > loader_time: 0.0046 (0.004628854743931267)  --> TIME: 2025-02-24 19:01:31 -- STEP: 1794/36938 -- GLOBAL_STEP: 445050 | > loss_text_ce: 0.04134863242506981 (0.03810802083815488) | > loss_mel_ce: 2.7995684146881104 (nan) | > loss: 0.011273480951786041 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13621425482474528) | > loader_time: 0.0046 (0.004627783960323266)  --> TIME: 2025-02-24 19:01:40 -- STEP: 1844/36938 -- GLOBAL_STEP: 445100 | > loss_text_ce: 0.03999242186546326 (0.03809469518778997) | > loss_mel_ce: 1.1387261152267456 (nan) | > loss: 0.004677454940974712 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.1362093544833826) | > loader_time: 0.0046 (0.004630302141648825)  --> TIME: 2025-02-24 19:01:49 -- STEP: 1894/36938 -- GLOBAL_STEP: 445150 | > loss_text_ce: 0.05271388962864876 (0.03809050472360794) | > loss_mel_ce: 3.584789514541626 (nan) | > loss: 0.01443453785032034 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13628236859753579) | > loader_time: 0.0045 (0.00462986410860023)  --> TIME: 2025-02-24 19:01:59 -- STEP: 1944/36938 -- GLOBAL_STEP: 445200 | > loss_text_ce: 0.035127487033605576 (0.03810945748631296) | > loss_mel_ce: 3.001053810119629 (nan) | > loss: 0.012048338539898396 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13631220241632933) | > loader_time: 0.0047 (0.004630008965362734)  --> TIME: 2025-02-24 19:02:08 -- STEP: 1994/36938 -- GLOBAL_STEP: 445250 | > loss_text_ce: 0.035082899034023285 (0.038158857338354675) | > loss_mel_ce: 3.015446662902832 (nan) | > loss: 0.012105276808142662 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13629653281648957) | > loader_time: 0.0047 (0.004630903064188289)  --> TIME: 2025-02-24 19:02:18 -- STEP: 2044/36938 -- GLOBAL_STEP: 445300 | > loss_text_ce: 0.04029238224029541 (0.0381092534723114) | > loss_mel_ce: 3.210688591003418 (nan) | > loss: 0.012900718487799168 (nan) | > current_lr: 5e-06 | > step_time: 0.185 (0.13638909595586513) | > loader_time: 0.0048 (0.004630761953715938)  --> TIME: 2025-02-24 19:02:27 -- STEP: 2094/36938 -- GLOBAL_STEP: 445350 | > loss_text_ce: 0.04705511033535004 (0.03808733698695813) | > loss_mel_ce: 2.7907283306121826 (nan) | > loss: 0.011261045932769775 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13646246744773413) | > loader_time: 0.0045 (0.004632718811290196)  --> TIME: 2025-02-24 19:02:36 -- STEP: 2144/36938 -- GLOBAL_STEP: 445400 | > loss_text_ce: 0.035963334143161774 (0.03808917061672478) | > loss_mel_ce: 1.848076581954956 (nan) | > loss: 0.007476349361240864 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.1365057214427346) | > loader_time: 0.0046 (0.00463488331036781)  --> TIME: 2025-02-24 19:02:46 -- STEP: 2194/36938 -- GLOBAL_STEP: 445450 | > loss_text_ce: 0.04333452135324478 (0.03807314509846807) | > loss_mel_ce: 2.9854722023010254 (nan) | > loss: 0.012019074521958828 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13654706134726557) | > loader_time: 0.0046 (0.004635517664572923)  --> TIME: 2025-02-24 19:02:55 -- STEP: 2244/36938 -- GLOBAL_STEP: 445500 | > loss_text_ce: 0.04531664773821831 (0.03808838683271815) | > loss_mel_ce: 1.9208838939666748 (nan) | > loss: 0.007802383974194527 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13665759244704592) | > loader_time: 0.0045 (0.004636225534632881)  --> TIME: 2025-02-24 19:03:05 -- STEP: 2294/36938 -- GLOBAL_STEP: 445550 | > loss_text_ce: 0.029681669548153877 (0.03808380266344585) | > loss_mel_ce: 3.577031135559082 (nan) | > loss: 0.014312352985143661 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13665317023642246) | > loader_time: 0.0046 (0.004636555312505058)  --> TIME: 2025-02-24 19:03:14 -- STEP: 2344/36938 -- GLOBAL_STEP: 445600 | > loss_text_ce: 0.024723684415221214 (0.03808855886313638) | > loss_mel_ce: 2.8334500789642334 (nan) | > loss: 0.011341960169374943 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13655178192939327) | > loader_time: 0.0046 (0.004635929145910635)  --> TIME: 2025-02-24 19:03:23 -- STEP: 2394/36938 -- GLOBAL_STEP: 445650 | > loss_text_ce: 0.027446826919913292 (0.038105877268041675) | > loss_mel_ce: 2.8862171173095703 (nan) | > loss: 0.011562159284949303 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13651295990971762) | > loader_time: 0.0046 (0.00463560031868561)  --> TIME: 2025-02-24 19:03:33 -- STEP: 2444/36938 -- GLOBAL_STEP: 445700 | > loss_text_ce: 0.03708227723836899 (0.038106230477788675) | > loss_mel_ce: 2.6699330806732178 (nan) | > loss: 0.0107421251013875 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.13673439529250336) | > loader_time: 0.0046 (0.004636816912118814)  --> TIME: 2025-02-24 19:03:43 -- STEP: 2494/36938 -- GLOBAL_STEP: 445750 | > loss_text_ce: 0.02957870438694954 (0.03809844775598881) | > loss_mel_ce: 2.8087871074676514 (nan) | > loss: 0.011263356544077396 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13677321937242862) | > loader_time: 0.0045 (0.004636828002684958)  --> TIME: 2025-02-24 19:03:52 -- STEP: 2544/36938 -- GLOBAL_STEP: 445800 | > loss_text_ce: 0.03324040398001671 (0.03809797258113368) | > loss_mel_ce: 3.664356231689453 (nan) | > loss: 0.014673002995550632 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.1367083917248922) | > loader_time: 0.0054 (0.00463840805884427)  --> TIME: 2025-02-24 19:04:01 -- STEP: 2594/36938 -- GLOBAL_STEP: 445850 | > loss_text_ce: 0.041452839970588684 (0.038127539552311566) | > loss_mel_ce: 3.299280881881714 (nan) | > loss: 0.013256880454719067 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13672795742405486) | > loader_time: 0.0046 (0.0046381476114783346)  --> TIME: 2025-02-24 19:04:11 -- STEP: 2644/36938 -- GLOBAL_STEP: 445900 | > loss_text_ce: 0.03548411652445793 (0.038137729815806175) | > loss_mel_ce: 3.411162853240967 (nan) | > loss: 0.013677171431481838 (nan) | > current_lr: 5e-06 | > step_time: 0.194 (0.13675959559323747) | > loader_time: 0.0049 (0.004638284399722236)  --> TIME: 2025-02-24 19:04:19 -- STEP: 2694/36938 -- GLOBAL_STEP: 445950 | > loss_text_ce: 0.027222558856010437 (0.038157514656694036) | > loss_mel_ce: 1.849255084991455 (nan) | > loss: 0.007446340285241604 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.1366132736737055) | > loader_time: 0.0045 (0.004637292604934933)  --> TIME: 2025-02-24 19:04:28 -- STEP: 2744/36938 -- GLOBAL_STEP: 446000 | > loss_text_ce: 0.04210936278104782 (0.03814271475329622) | > loss_mel_ce: 2.9086315631866455 (nan) | > loss: 0.011709289625287056 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13653807919852604) | > loader_time: 0.0047 (0.004637441899269038)  --> TIME: 2025-02-24 19:04:38 -- STEP: 2794/36938 -- GLOBAL_STEP: 446050 | > loss_text_ce: 0.03493926301598549 (0.038165653239116956) | > loss_mel_ce: 3.576706886291504 (nan) | > loss: 0.014331930316984653 (nan) | > current_lr: 5e-06 | > step_time: 0.1813 (0.13655161430943255) | > loader_time: 0.0048 (0.004643483766079289)  --> TIME: 2025-02-24 19:04:47 -- STEP: 2844/36938 -- GLOBAL_STEP: 446100 | > loss_text_ce: 0.04224944859743118 (0.038194719109940954) | > loss_mel_ce: 1.6540952920913696 (nan) | > loss: 0.006731526926159859 (nan) | > current_lr: 5e-06 | > step_time: 0.1086 (0.13650261777362757) | > loader_time: 0.0044 (0.0046425916977572545)  --> TIME: 2025-02-24 19:04:56 -- STEP: 2894/36938 -- GLOBAL_STEP: 446150 | > loss_text_ce: 0.044492967426776886 (0.03820709752803254) | > loss_mel_ce: 2.8441195487976074 (nan) | > loss: 0.011462748982012272 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.13647579734870796) | > loader_time: 0.0045 (0.0046419300700189705)  --> TIME: 2025-02-24 19:05:05 -- STEP: 2944/36938 -- GLOBAL_STEP: 446200 | > loss_text_ce: 0.029953910037875175 (0.03819791328493487) | > loss_mel_ce: 2.7463083267211914 (nan) | > loss: 0.011016914620995522 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13643726973753917) | > loader_time: 0.0047 (0.004644092496322554)  --> TIME: 2025-02-24 19:05:15 -- STEP: 2994/36938 -- GLOBAL_STEP: 446250 | > loss_text_ce: 0.035221729427576065 (0.03819742972253961) | > loss_mel_ce: 2.5665953159332275 (nan) | > loss: 0.010324671864509583 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13646708652825063) | > loader_time: 0.0045 (0.0046447736864975485)  --> TIME: 2025-02-24 19:05:24 -- STEP: 3044/36938 -- GLOBAL_STEP: 446300 | > loss_text_ce: 0.035690393298864365 (0.03817469360210182) | > loss_mel_ce: 3.013073682785034 (nan) | > loss: 0.01209827046841383 (nan) | > current_lr: 5e-06 | > step_time: 0.1774 (0.13633939870553644) | > loader_time: 0.0048 (0.004644253642424336)  --> TIME: 2025-02-24 19:05:33 -- STEP: 3094/36938 -- GLOBAL_STEP: 446350 | > loss_text_ce: 0.040041785687208176 (0.03818105576673086) | > loss_mel_ce: 3.098726272583008 (nan) | > loss: 0.012455428950488567 (nan) | > current_lr: 5e-06 | > step_time: 0.1694 (0.13637481440092272) | > loader_time: 0.0048 (0.004644849567006462)  --> TIME: 2025-02-24 19:05:42 -- STEP: 3144/36938 -- GLOBAL_STEP: 446400 | > loss_text_ce: 0.036238402128219604 (0.03817421821113112) | > loss_mel_ce: 2.884899616241455 (nan) | > loss: 0.011591818183660507 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13634088711277534) | > loader_time: 0.0045 (0.004644803645956607)  --> TIME: 2025-02-24 19:05:52 -- STEP: 3194/36938 -- GLOBAL_STEP: 446450 | > loss_text_ce: 0.03957850858569145 (0.03818421656062916) | > loss_mel_ce: 3.029646873474121 (nan) | > loss: 0.012179465964436531 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13637085071413232) | > loader_time: 0.0046 (0.004644800367098566)  --> TIME: 2025-02-24 19:06:01 -- STEP: 3244/36938 -- GLOBAL_STEP: 446500 | > loss_text_ce: 0.03310490772128105 (0.03818671982741143) | > loss_mel_ce: 3.4289844036102295 (nan) | > loss: 0.013738450594246387 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.13633212899162211) | > loader_time: 0.0046 (0.004644915369671164)  --> TIME: 2025-02-24 19:06:10 -- STEP: 3294/36938 -- GLOBAL_STEP: 446550 | > loss_text_ce: 0.0422792062163353 (0.03817245726511522) | > loss_mel_ce: 3.6537277698516846 (nan) | > loss: 0.014666695147752762 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13635155136974644) | > loader_time: 0.0045 (0.004645231063972476)  --> TIME: 2025-02-24 19:06:20 -- STEP: 3344/36938 -- GLOBAL_STEP: 446600 | > loss_text_ce: 0.026003355160355568 (0.03817671770759104) | > loss_mel_ce: 2.5955870151519775 (nan) | > loss: 0.010403136722743511 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13641115966025727) | > loader_time: 0.0046 (0.0046450573434099595)  --> TIME: 2025-02-24 19:06:29 -- STEP: 3394/36938 -- GLOBAL_STEP: 446650 | > loss_text_ce: 0.02604805864393711 (0.038165978770966585) | > loss_mel_ce: 0.006622314453125 (nan) | > loss: 0.00012964433699380606 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13639084465305623) | > loader_time: 0.0045 (0.004644613302519689)  --> TIME: 2025-02-24 19:06:39 -- STEP: 3444/36938 -- GLOBAL_STEP: 446700 | > loss_text_ce: 0.049779295921325684 (0.038166750782482085) | > loss_mel_ce: 1.3751088380813599 (nan) | > loss: 0.005654318258166313 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1364714952572992) | > loader_time: 0.0046 (0.004644894226087504)  --> TIME: 2025-02-24 19:06:48 -- STEP: 3494/36938 -- GLOBAL_STEP: 446750 | > loss_text_ce: 0.033312369138002396 (0.038159446679130404) | > loss_mel_ce: 2.625981330871582 (nan) | > loss: 0.010552752763032913 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13653437144154495) | > loader_time: 0.0045 (0.00464494766067216)  --> TIME: 2025-02-24 19:06:58 -- STEP: 3544/36938 -- GLOBAL_STEP: 446800 | > loss_text_ce: 0.04457487538456917 (0.038155719814201974) | > loss_mel_ce: 3.7498605251312256 (nan) | > loss: 0.015057284384965897 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1366101464353352) | > loader_time: 0.0046 (0.004645230269593519)  --> TIME: 2025-02-24 19:07:07 -- STEP: 3594/36938 -- GLOBAL_STEP: 446850 | > loss_text_ce: 0.04405905678868294 (0.03816074583421349) | > loss_mel_ce: 3.0869178771972656 (nan) | > loss: 0.012424511834979057 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13661537995654216) | > loader_time: 0.0047 (0.004645331740445666)  --> TIME: 2025-02-24 19:07:16 -- STEP: 3644/36938 -- GLOBAL_STEP: 446900 | > loss_text_ce: 0.042026057839393616 (0.038158617997807905) | > loss_mel_ce: 2.568875789642334 (nan) | > loss: 0.010360722430050373 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13658969802730825) | > loader_time: 0.0046 (0.004645003052888398)  --> TIME: 2025-02-24 19:07:26 -- STEP: 3694/36938 -- GLOBAL_STEP: 446950 | > loss_text_ce: 0.03818880766630173 (0.03814722443731639) | > loss_mel_ce: 1.9634034633636475 (nan) | > loss: 0.007942826487123966 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13657486800187085) | > loader_time: 0.0046 (0.004645105820445668)  --> TIME: 2025-02-24 19:07:35 -- STEP: 3744/36938 -- GLOBAL_STEP: 447000 | > loss_text_ce: 0.0354042574763298 (0.0381348486335523) | > loss_mel_ce: 3.8071227073669434 (nan) | > loss: 0.015248123556375504 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.13652739794845278) | > loader_time: 0.0048 (0.004645122103711473)  --> TIME: 2025-02-24 19:07:44 -- STEP: 3794/36938 -- GLOBAL_STEP: 447050 | > loss_text_ce: 0.04425673559308052 (0.03813655504026017) | > loss_mel_ce: 3.043487310409546 (nan) | > loss: 0.01225295290350914 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.136536938426491) | > loader_time: 0.0045 (0.004645311273143484)  --> TIME: 2025-02-24 19:07:53 -- STEP: 3844/36938 -- GLOBAL_STEP: 447100 | > loss_text_ce: 0.02858886867761612 (0.03811367365151736) | > loss_mel_ce: 2.1089982986450195 (nan) | > loss: 0.008482488803565502 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.1365319716198513) | > loader_time: 0.0065 (0.004645520578934171)  --> TIME: 2025-02-24 19:08:03 -- STEP: 3894/36938 -- GLOBAL_STEP: 447150 | > loss_text_ce: 0.03832555562257767 (0.038112875019856975) | > loss_mel_ce: 2.8056581020355225 (nan) | > loss: 0.011285649612545967 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.13655794989346373) | > loader_time: 0.0046 (0.004645869679247709)  --> TIME: 2025-02-24 19:08:12 -- STEP: 3944/36938 -- GLOBAL_STEP: 447200 | > loss_text_ce: 0.03229667246341705 (0.03811252784360827) | > loss_mel_ce: 3.067126750946045 (nan) | > loss: 0.01229930017143488 (nan) | > current_lr: 5e-06 | > step_time: 0.1707 (0.1365715357159506) | > loader_time: 0.0049 (0.004645280434200289)  --> TIME: 2025-02-24 19:08:22 -- STEP: 3994/36938 -- GLOBAL_STEP: 447250 | > loss_text_ce: 0.04271431267261505 (0.03810579640348717) | > loss_mel_ce: 2.98345947265625 (nan) | > loss: 0.012008626945316792 (nan) | > current_lr: 5e-06 | > step_time: 0.152 (0.1365958324479175) | > loader_time: 0.0046 (0.004646227009963793)  --> TIME: 2025-02-24 19:08:31 -- STEP: 4044/36938 -- GLOBAL_STEP: 447300 | > loss_text_ce: 0.044734034687280655 (0.03812595177372753) | > loss_mel_ce: 3.5764529705047607 (nan) | > loss: 0.014369790442287922 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13657200890643914) | > loader_time: 0.0046 (0.004645737884778056)  --> TIME: 2025-02-24 19:08:40 -- STEP: 4094/36938 -- GLOBAL_STEP: 447350 | > loss_text_ce: 0.03895801305770874 (0.038129197265479725) | > loss_mel_ce: 2.735970973968506 (nan) | > loss: 0.011011623777449131 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13659576322604944) | > loader_time: 0.0046 (0.004645788558937593)  --> TIME: 2025-02-24 19:08:50 -- STEP: 4144/36938 -- GLOBAL_STEP: 447400 | > loss_text_ce: 0.041881103068590164 (0.03812937537890517) | > loss_mel_ce: 2.999311923980713 (nan) | > loss: 0.012068226933479309 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.1366545605153192) | > loader_time: 0.0047 (0.004646514488463231)  --> TIME: 2025-02-24 19:08:59 -- STEP: 4194/36938 -- GLOBAL_STEP: 447450 | > loss_text_ce: 0.03764839842915535 (0.03812154069628164) | > loss_mel_ce: 3.943734884262085 (nan) | > loss: 0.01579914055764675 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13666809196408514) | > loader_time: 0.0046 (0.004647188716464567)  --> TIME: 2025-02-24 19:09:09 -- STEP: 4244/36938 -- GLOBAL_STEP: 447500 | > loss_text_ce: 0.04440635070204735 (0.03812418878956247) | > loss_mel_ce: 3.341359853744507 (nan) | > loss: 0.013435580767691135 (nan) | > current_lr: 5e-06 | > step_time: 0.1264 (0.13666591112845147) | > loader_time: 0.0046 (0.004648033118719874)  --> TIME: 2025-02-24 19:09:18 -- STEP: 4294/36938 -- GLOBAL_STEP: 447550 | > loss_text_ce: 0.04720563814043999 (0.03810804534458852) | > loss_mel_ce: 2.9760355949401855 (nan) | > loss: 0.011996990069746971 (nan) | > current_lr: 5e-06 | > step_time: 0.1664 (0.13666771343268527) | > loader_time: 0.0048 (0.004648856856848504)  --> TIME: 2025-02-24 19:09:27 -- STEP: 4344/36938 -- GLOBAL_STEP: 447600 | > loss_text_ce: 0.03085121512413025 (0.03807916555555402) | > loss_mel_ce: 3.134101629257202 (nan) | > loss: 0.012559336610138416 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.1366165485610401) | > loader_time: 0.0046 (0.004648767233333968)  --> TIME: 2025-02-24 19:09:36 -- STEP: 4394/36938 -- GLOBAL_STEP: 447650 | > loss_text_ce: 0.047795265913009644 (0.038090771912399256) | > loss_mel_ce: 2.566131591796875 (nan) | > loss: 0.010372726246714592 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.136621676415924) | > loader_time: 0.0046 (0.00464962686687151)  --> TIME: 2025-02-24 19:09:46 -- STEP: 4444/36938 -- GLOBAL_STEP: 447700 | > loss_text_ce: 0.036645084619522095 (0.03810101318609862) | > loss_mel_ce: 3.2181265354156494 (nan) | > loss: 0.012915761210024357 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13661934388424243) | > loader_time: 0.0045 (0.004649908718603559)  --> TIME: 2025-02-24 19:09:55 -- STEP: 4494/36938 -- GLOBAL_STEP: 447750 | > loss_text_ce: 0.03920338675379753 (0.03807955791466663) | > loss_mel_ce: 3.1140594482421875 (nan) | > loss: 0.012512948364019394 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13661610740632762) | > loader_time: 0.0046 (0.004649634089637551)  --> TIME: 2025-02-24 19:10:04 -- STEP: 4544/36938 -- GLOBAL_STEP: 447800 | > loss_text_ce: 0.04888778552412987 (0.0380817527877017) | > loss_mel_ce: 3.1597344875335693 (nan) | > loss: 0.012732628732919693 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13660329338950178) | > loader_time: 0.0053 (0.004650760809300648)  --> TIME: 2025-02-24 19:10:14 -- STEP: 4594/36938 -- GLOBAL_STEP: 447850 | > loss_text_ce: 0.0448094941675663 (0.03808077791698439) | > loss_mel_ce: 3.224757194519043 (nan) | > loss: 0.012974471785128117 (nan) | > current_lr: 5e-06 | > step_time: 0.1943 (0.13662322059526727) | > loader_time: 0.0063 (0.004650993247524156)  --> TIME: 2025-02-24 19:10:23 -- STEP: 4644/36938 -- GLOBAL_STEP: 447900 | > loss_text_ce: 0.04408484324812889 (0.03808665677642781) | > loss_mel_ce: 3.261448383331299 (nan) | > loss: 0.013117196038365364 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13656507156513362) | > loader_time: 0.0046 (0.004650846418894947)  --> TIME: 2025-02-24 19:10:31 -- STEP: 4694/36938 -- GLOBAL_STEP: 447950 | > loss_text_ce: 0.0362008772790432 (0.03807749452417022) | > loss_mel_ce: 3.658923387527466 (nan) | > loss: 0.014663191512227058 (nan) | > current_lr: 5e-06 | > step_time: 0.1629 (0.13649101312179795) | > loader_time: 0.0047 (0.004650509199996575)  --> TIME: 2025-02-24 19:10:41 -- STEP: 4744/36938 -- GLOBAL_STEP: 448000 | > loss_text_ce: 0.03541024774312973 (0.03807657098332449) | > loss_mel_ce: 2.4673941135406494 (nan) | > loss: 0.009931763634085655 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13646093724549915) | > loader_time: 0.0046 (0.004650184768448391)  --> TIME: 2025-02-24 19:10:50 -- STEP: 4794/36938 -- GLOBAL_STEP: 448050 | > loss_text_ce: 0.0404747873544693 (0.038074430775497346) | > loss_mel_ce: 3.4978127479553223 (nan) | > loss: 0.014040824957191944 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.1364751282362923) | > loader_time: 0.0047 (0.004650605784190613)  --> TIME: 2025-02-24 19:11:00 -- STEP: 4844/36938 -- GLOBAL_STEP: 448100 | > loss_text_ce: 0.0409669503569603 (0.03808342997991422) | > loss_mel_ce: 1.3181610107421875 (nan) | > loss: 0.0053933653980493546 (nan) | > current_lr: 5e-06 | > step_time: 0.1567 (0.13653334967859695) | > loader_time: 0.0046 (0.004650254519294846)  --> TIME: 2025-02-24 19:11:09 -- STEP: 4894/36938 -- GLOBAL_STEP: 448150 | > loss_text_ce: 0.04103076830506325 (0.038070086277123244) | > loss_mel_ce: 3.307788133621216 (nan) | > loss: 0.013288965448737144 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.1365101272152743) | > loader_time: 0.0047 (0.004649610094601248)  --> TIME: 2025-02-24 19:11:18 -- STEP: 4944/36938 -- GLOBAL_STEP: 448200 | > loss_text_ce: 0.041966766119003296 (0.0380748281278724) | > loss_mel_ce: 2.650364398956299 (nan) | > loss: 0.01068385411053896 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13650859940592122) | > loader_time: 0.0046 (0.004649707173452408)  --> TIME: 2025-02-24 19:11:28 -- STEP: 4994/36938 -- GLOBAL_STEP: 448250 | > loss_text_ce: 0.040258608758449554 (0.038093124462226804) | > loss_mel_ce: 3.5959086418151855 (nan) | > loss: 0.014429235830903053 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.13654178884633766) | > loader_time: 0.0045 (0.004650181897124824)  --> TIME: 2025-02-24 19:11:37 -- STEP: 5044/36938 -- GLOBAL_STEP: 448300 | > loss_text_ce: 0.04493001848459244 (0.03809669099542825) | > loss_mel_ce: 3.193589448928833 (nan) | > loss: 0.01285126805305481 (nan) | > current_lr: 5e-06 | > step_time: 0.1765 (0.13654832727468533) | > loader_time: 0.0048 (0.004649775449857179)  --> TIME: 2025-02-24 19:11:46 -- STEP: 5094/36938 -- GLOBAL_STEP: 448350 | > loss_text_ce: 0.02991209365427494 (0.038090586911303466) | > loss_mel_ce: 2.664560079574585 (nan) | > loss: 0.010692350566387177 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.13656317073783047) | > loader_time: 0.0046 (0.004649695200596226)  --> TIME: 2025-02-24 19:11:56 -- STEP: 5144/36938 -- GLOBAL_STEP: 448400 | > loss_text_ce: 0.037271976470947266 (0.038085827471769626) | > loss_mel_ce: 3.451801300048828 (nan) | > loss: 0.013845529407262802 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.13654821782052748) | > loader_time: 0.0055 (0.004650288013796418)  --> TIME: 2025-02-24 19:12:05 -- STEP: 5194/36938 -- GLOBAL_STEP: 448450 | > loss_text_ce: 0.029866458848118782 (0.038104102500342574) | > loss_mel_ce: 3.3847343921661377 (nan) | > loss: 0.013550003990530968 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13647951366445077) | > loader_time: 0.0046 (0.0046500267778674985)  --> TIME: 2025-02-24 19:12:14 -- STEP: 5244/36938 -- GLOBAL_STEP: 448500 | > loss_text_ce: 0.023524736985564232 (0.03809694326706494) | > loss_mel_ce: 1.6163454055786133 (nan) | > loss: 0.0065074218437075615 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13647778151509915) | > loader_time: 0.0045 (0.004649829173433609)  --> TIME: 2025-02-24 19:12:23 -- STEP: 5294/36938 -- GLOBAL_STEP: 448550 | > loss_text_ce: 0.03750738874077797 (0.03809616695048063) | > loss_mel_ce: 3.28125 (nan) | > loss: 0.013169672340154648 (nan) | > current_lr: 5e-06 | > step_time: 0.1073 (0.13645460473216553) | > loader_time: 0.0044 (0.004650473189344935)  --> TIME: 2025-02-24 19:12:32 -- STEP: 5344/36938 -- GLOBAL_STEP: 448600 | > loss_text_ce: 0.039549391716718674 (0.038097253093680054) | > loss_mel_ce: 2.415968894958496 (nan) | > loss: 0.009744120761752129 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.1364816705177639) | > loader_time: 0.0054 (0.004651222935693703)  --> TIME: 2025-02-24 19:12:42 -- STEP: 5394/36938 -- GLOBAL_STEP: 448650 | > loss_text_ce: 0.03498118370771408 (0.03811463565021084) | > loss_mel_ce: 3.2621710300445557 (nan) | > loss: 0.013083938509225845 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13644716511578883) | > loader_time: 0.0045 (0.004650778358496243)  --> TIME: 2025-02-24 19:12:51 -- STEP: 5444/36938 -- GLOBAL_STEP: 448700 | > loss_text_ce: 0.05489308759570122 (0.03811439743448905) | > loss_mel_ce: 2.8091161251068115 (nan) | > loss: 0.011365116573870182 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13642167795413876) | > loader_time: 0.0045 (0.004650516557308085)  --> TIME: 2025-02-24 19:13:00 -- STEP: 5494/36938 -- GLOBAL_STEP: 448750 | > loss_text_ce: 0.03228810429573059 (0.038122902625326215) | > loss_mel_ce: 3.333089590072632 (nan) | > loss: 0.013354673981666565 (nan) | > current_lr: 5e-06 | > step_time: 0.1458 (0.1364815853099452) | > loader_time: 0.0048 (0.004651512455671189)  --> TIME: 2025-02-24 19:13:10 -- STEP: 5544/36938 -- GLOBAL_STEP: 448800 | > loss_text_ce: 0.037045564502477646 (0.03812003383765061) | > loss_mel_ce: 3.0958917140960693 (nan) | > loss: 0.01243229117244482 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.1364830952530123) | > loader_time: 0.0047 (0.004651267678190617)  --> TIME: 2025-02-24 19:13:19 -- STEP: 5594/36938 -- GLOBAL_STEP: 448850 | > loss_text_ce: 0.04618089273571968 (0.03811226749688145) | > loss_mel_ce: 2.9563145637512207 (nan) | > loss: 0.011914665810763836 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13649967017666434) | > loader_time: 0.0045 (0.004651903594695202)  --> TIME: 2025-02-24 19:13:29 -- STEP: 5644/36938 -- GLOBAL_STEP: 448900 | > loss_text_ce: 0.04123083874583244 (0.03811436633808472) | > loss_mel_ce: 2.778029441833496 (nan) | > loss: 0.011187541298568249 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13649737627940484) | > loader_time: 0.0047 (0.004652097789553527)  --> TIME: 2025-02-24 19:13:37 -- STEP: 5694/36938 -- GLOBAL_STEP: 448950 | > loss_text_ce: 0.034264229238033295 (0.038084762438171334) | > loss_mel_ce: 3.055351734161377 (nan) | > loss: 0.012260382063686848 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13644071147866402) | > loader_time: 0.0045 (0.004651665771304828)  --> TIME: 2025-02-24 19:13:46 -- STEP: 5744/36938 -- GLOBAL_STEP: 449000 | > loss_text_ce: 0.047921258956193924 (0.03807888583060937) | > loss_mel_ce: 2.9531731605529785 (nan) | > loss: 0.011909104883670807 (nan) | > current_lr: 5e-06 | > step_time: 0.1695 (0.1363989965248237) | > loader_time: 0.0048 (0.004651498620224529)  --> TIME: 2025-02-24 19:13:56 -- STEP: 5794/36938 -- GLOBAL_STEP: 449050 | > loss_text_ce: 0.037723999470472336 (0.038084058981217805) | > loss_mel_ce: 2.656376361846924 (nan) | > loss: 0.010690875351428986 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13640093963887703) | > loader_time: 0.0046 (0.004651431013560924)  --> TIME: 2025-02-24 19:14:05 -- STEP: 5844/36938 -- GLOBAL_STEP: 449100 | > loss_text_ce: 0.045653462409973145 (0.038093234793251836) | > loss_mel_ce: 3.4901578426361084 (nan) | > loss: 0.014030998572707176 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13633459684050786) | > loader_time: 0.0045 (0.004650863982650363)  --> TIME: 2025-02-24 19:14:14 -- STEP: 5894/36938 -- GLOBAL_STEP: 449150 | > loss_text_ce: 0.03662005439400673 (0.0380797536972902) | > loss_mel_ce: 3.029311180114746 (nan) | > loss: 0.012166394852101803 (nan) | > current_lr: 5e-06 | > step_time: 0.1701 (0.1363230745065077) | > loader_time: 0.0047 (0.0046501524536578034)  --> TIME: 2025-02-24 19:14:23 -- STEP: 5944/36938 -- GLOBAL_STEP: 449200 | > loss_text_ce: 0.038057733327150345 (0.038083428967290424) | > loss_mel_ce: 3.8601109981536865 (nan) | > loss: 0.015468924306333065 (nan) | > current_lr: 5e-06 | > step_time: 0.1714 (0.13630195906633935) | > loader_time: 0.0048 (0.0046497427759388645)  --> TIME: 2025-02-24 19:14:32 -- STEP: 5994/36938 -- GLOBAL_STEP: 449250 | > loss_text_ce: 0.03819109499454498 (0.038074434485968964) | > loss_mel_ce: 3.2052700519561768 (nan) | > loss: 0.012870877981185913 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13631424245175613) | > loader_time: 0.0046 (0.004649914738970425)  --> TIME: 2025-02-24 19:14:41 -- STEP: 6044/36938 -- GLOBAL_STEP: 449300 | > loss_text_ce: 0.034464653581380844 (0.03806239406348345) | > loss_mel_ce: 2.9724011421203613 (nan) | > loss: 0.01193200796842575 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13630334842607358) | > loader_time: 0.0052 (0.004650614697123732)  --> TIME: 2025-02-24 19:14:51 -- STEP: 6094/36938 -- GLOBAL_STEP: 449350 | > loss_text_ce: 0.03546620160341263 (0.038072923399155904) | > loss_mel_ce: 1.9403724670410156 (nan) | > loss: 0.007840629667043686 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13628132123418152) | > loader_time: 0.0045 (0.004650394330464459)  --> TIME: 2025-02-24 19:15:00 -- STEP: 6144/36938 -- GLOBAL_STEP: 449400 | > loss_text_ce: 0.0455959253013134 (0.03806914294273159) | > loss_mel_ce: 3.7265625 (nan) | > loss: 0.014968883246183395 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13629388844128665) | > loader_time: 0.0051 (0.004651164985261848)  --> TIME: 2025-02-24 19:15:09 -- STEP: 6194/36938 -- GLOBAL_STEP: 449450 | > loss_text_ce: 0.04057800769805908 (0.038066631408265765) | > loss_mel_ce: 2.8632631301879883 (nan) | > loss: 0.011523179709911346 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.1362694320580169) | > loader_time: 0.0046 (0.004651452173215169)  --> TIME: 2025-02-24 19:15:18 -- STEP: 6244/36938 -- GLOBAL_STEP: 449500 | > loss_text_ce: 0.04211054742336273 (0.038056138516293386) | > loss_mel_ce: 2.9433650970458984 (nan) | > loss: 0.011847126297652721 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.13624607191721158) | > loader_time: 0.0047 (0.004651366404277402)  --> TIME: 2025-02-24 19:15:27 -- STEP: 6294/36938 -- GLOBAL_STEP: 449550 | > loss_text_ce: 0.04925697296857834 (0.038061217545177474) | > loss_mel_ce: 3.2935707569122314 (nan) | > loss: 0.01326519064605236 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13622929143041948) | > loader_time: 0.0045 (0.004651173774045274)  --> TIME: 2025-02-24 19:15:37 -- STEP: 6344/36938 -- GLOBAL_STEP: 449600 | > loss_text_ce: 0.0389053039252758 (0.03805785487326095) | > loss_mel_ce: 3.304598331451416 (nan) | > loss: 0.013267872855067253 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13623037062406826) | > loader_time: 0.0045 (0.004650955317450355)  --> TIME: 2025-02-24 19:15:46 -- STEP: 6394/36938 -- GLOBAL_STEP: 449650 | > loss_text_ce: 0.038273539394140244 (0.03807134502286236) | > loss_mel_ce: 1.5566895008087158 (nan) | > loss: 0.006329218856990337 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13624310008727036) | > loader_time: 0.0046 (0.004651113640788499)  --> TIME: 2025-02-24 19:15:55 -- STEP: 6444/36938 -- GLOBAL_STEP: 449700 | > loss_text_ce: 0.030612485483288765 (0.038063430351920294) | > loss_mel_ce: 1.3546501398086548 (nan) | > loss: 0.005497074220329523 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.1362468685739754) | > loader_time: 0.0046 (0.004650471707740088)  --> TIME: 2025-02-24 19:16:04 -- STEP: 6494/36938 -- GLOBAL_STEP: 449750 | > loss_text_ce: 0.04643621668219566 (0.0380735170459854) | > loss_mel_ce: 2.75215220451355 (nan) | > loss: 0.011105510406196117 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13619912048835622) | > loader_time: 0.0045 (0.004650061887559577)  --> TIME: 2025-02-24 19:16:14 -- STEP: 6544/36938 -- GLOBAL_STEP: 449800 | > loss_text_ce: 0.035158369690179825 (0.03808043703714657) | > loss_mel_ce: 3.5234878063201904 (nan) | > loss: 0.014121612533926964 (nan) | > current_lr: 5e-06 | > step_time: 0.168 (0.13622253284331923) | > loader_time: 0.0048 (0.0046497834413733626)  --> TIME: 2025-02-24 19:16:23 -- STEP: 6594/36938 -- GLOBAL_STEP: 449850 | > loss_text_ce: 0.041147131472826004 (0.03809734096161374) | > loss_mel_ce: 3.0868074893951416 (nan) | > loss: 0.012412519194185734 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13621751019041356) | > loader_time: 0.0048 (0.004649347126538154)  --> TIME: 2025-02-24 19:16:32 -- STEP: 6644/36938 -- GLOBAL_STEP: 449900 | > loss_text_ce: 0.03717203438282013 (0.03810366226527293) | > loss_mel_ce: 3.196897506713867 (nan) | > loss: 0.0128336101770401 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1362394438602348) | > loader_time: 0.0046 (0.0046490083108244006)  --> TIME: 2025-02-24 19:16:42 -- STEP: 6694/36938 -- GLOBAL_STEP: 449950 | > loss_text_ce: 0.03059806115925312 (0.038094129123980185) | > loss_mel_ce: 3.096435785293579 (nan) | > loss: 0.012408865615725517 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.1362577112247453) | > loader_time: 0.0046 (0.004648946134375214)  --> TIME: 2025-02-24 19:16:51 -- STEP: 6744/36938 -- GLOBAL_STEP: 450000 | > loss_text_ce: 0.03943375125527382 (0.03809586993483819) | > loss_mel_ce: 3.1269710063934326 (nan) | > loss: 0.012565098702907562 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.13630161817104916) | > loader_time: 0.0049 (0.004648569639891492) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_450000.pth  --> TIME: 2025-02-24 19:17:07 -- STEP: 6794/36938 -- GLOBAL_STEP: 450050 | > loss_text_ce: 0.047020964324474335 (0.03809901527991351) | > loss_mel_ce: 2.8831043243408203 (nan) | > loss: 0.011627481319010258 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.13629846984160296) | > loader_time: 0.005 (0.004649012447421185)  --> TIME: 2025-02-24 19:17:16 -- STEP: 6844/36938 -- GLOBAL_STEP: 450100 | > loss_text_ce: 0.03996877744793892 (0.03809321406588598) | > loss_mel_ce: 0.0220947265625 (nan) | > loss: 0.00024628377286717296 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13630994417596307) | > loader_time: 0.0045 (0.004649282059148346)  --> TIME: 2025-02-24 19:17:26 -- STEP: 6894/36938 -- GLOBAL_STEP: 450150 | > loss_text_ce: 0.04509744420647621 (0.03809749197338049) | > loss_mel_ce: 3.0061426162719727 (nan) | > loss: 0.012108095921576023 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13630868024607468) | > loader_time: 0.0046 (0.004649648605514892)  --> TIME: 2025-02-24 19:17:35 -- STEP: 6944/36938 -- GLOBAL_STEP: 450200 | > loss_text_ce: 0.04316931590437889 (0.03809413688835661) | > loss_mel_ce: 3.0815343856811523 (nan) | > loss: 0.01239961851388216 (nan) | > current_lr: 5e-06 | > step_time: 0.1961 (0.13631204780070985) | > loader_time: 0.0049 (0.004650223159020945)  --> TIME: 2025-02-24 19:17:44 -- STEP: 6994/36938 -- GLOBAL_STEP: 450250 | > loss_text_ce: 0.037856705486774445 (0.03808653734950365) | > loss_mel_ce: 3.366964340209961 (nan) | > loss: 0.01351119577884674 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.1362825592552076) | > loader_time: 0.0045 (0.004650990588684778)  --> TIME: 2025-02-24 19:17:53 -- STEP: 7044/36938 -- GLOBAL_STEP: 450300 | > loss_text_ce: 0.03549981862306595 (0.03808408536171371) | > loss_mel_ce: 3.175419807434082 (nan) | > loss: 0.012741745449602604 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13621814223191453) | > loader_time: 0.0046 (0.004650676033836679)  --> TIME: 2025-02-24 19:18:02 -- STEP: 7094/36938 -- GLOBAL_STEP: 450350 | > loss_text_ce: 0.04574162885546684 (0.03808515740115821) | > loss_mel_ce: 2.5141875743865967 (nan) | > loss: 0.010158449411392212 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13624612475772405) | > loader_time: 0.0047 (0.004651189993091255)  --> TIME: 2025-02-24 19:18:12 -- STEP: 7144/36938 -- GLOBAL_STEP: 450400 | > loss_text_ce: 0.030344411730766296 (0.038092514843922995) | > loss_mel_ce: 1.539381980895996 (nan) | > loss: 0.006229073274880648 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.1362469456964749) | > loader_time: 0.0044 (0.004651270715012927)  --> TIME: 2025-02-24 19:18:21 -- STEP: 7194/36938 -- GLOBAL_STEP: 450450 | > loss_text_ce: 0.04474164545536041 (0.038102638609199914) | > loss_mel_ce: 3.3743181228637695 (nan) | > loss: 0.013567698188126087 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13622758873708019) | > loader_time: 0.0045 (0.004651006109223352)  --> TIME: 2025-02-24 19:18:30 -- STEP: 7244/36938 -- GLOBAL_STEP: 450500 | > loss_text_ce: 0.031051592901349068 (0.03809953589313258) | > loss_mel_ce: 3.6387135982513428 (nan) | > loss: 0.014562561176717281 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13623272645282544) | > loader_time: 0.0046 (0.0046509801847772776)  --> TIME: 2025-02-24 19:18:39 -- STEP: 7294/36938 -- GLOBAL_STEP: 450550 | > loss_text_ce: 0.03926403075456619 (0.0380974376070065) | > loss_mel_ce: 3.511005163192749 (nan) | > loss: 0.014088369905948639 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13623158360364326) | > loader_time: 0.0046 (0.004650845702850951)  --> TIME: 2025-02-24 19:18:49 -- STEP: 7344/36938 -- GLOBAL_STEP: 450600 | > loss_text_ce: 0.04589655250310898 (0.03810350883250942) | > loss_mel_ce: 2.8725180625915527 (nan) | > loss: 0.01158101111650467 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1362295015704914) | > loader_time: 0.0046 (0.004650718928162567)  --> TIME: 2025-02-24 19:18:58 -- STEP: 7394/36938 -- GLOBAL_STEP: 450650 | > loss_text_ce: 0.033779121935367584 (0.03810916424643031) | > loss_mel_ce: 1.7718771696090698 (nan) | > loss: 0.0071653034538030624 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13624375512027895) | > loader_time: 0.0044 (0.004650524896255916)  --> TIME: 2025-02-24 19:19:07 -- STEP: 7444/36938 -- GLOBAL_STEP: 450700 | > loss_text_ce: 0.038952820003032684 (0.038118091608312116) | > loss_mel_ce: 2.7590484619140625 (nan) | > loss: 0.01110318023711443 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13620341913088985) | > loader_time: 0.0045 (0.004650405566580892)  --> TIME: 2025-02-24 19:19:16 -- STEP: 7494/36938 -- GLOBAL_STEP: 450750 | > loss_text_ce: 0.03610731661319733 (0.0381193709027093) | > loss_mel_ce: 2.8375396728515625 (nan) | > loss: 0.011403361335396767 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13618110099219294) | > loader_time: 0.0045 (0.004650605179832748)  --> TIME: 2025-02-24 19:19:25 -- STEP: 7544/36938 -- GLOBAL_STEP: 450800 | > loss_text_ce: 0.040337156504392624 (0.038120467756496634) | > loss_mel_ce: 0.5349062085151672 (nan) | > loss: 0.00228271191008389 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13617546801096805) | > loader_time: 0.0045 (0.0046502578991721045)  --> TIME: 2025-02-24 19:19:35 -- STEP: 7594/36938 -- GLOBAL_STEP: 450850 | > loss_text_ce: 0.03028547391295433 (0.03811886288399311) | > loss_mel_ce: 3.247980833053589 (nan) | > loss: 0.013008993119001389 (nan) | > current_lr: 5e-06 | > step_time: 0.1842 (0.13615682061044415) | > loader_time: 0.0048 (0.004650328170006795)  --> TIME: 2025-02-24 19:19:44 -- STEP: 7644/36938 -- GLOBAL_STEP: 450900 | > loss_text_ce: 0.04096504673361778 (0.038110774916156664) | > loss_mel_ce: 2.3513383865356445 (nan) | > loss: 0.009493268094956875 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13612365326539555) | > loader_time: 0.0045 (0.004650496862622094)  --> TIME: 2025-02-24 19:19:53 -- STEP: 7694/36938 -- GLOBAL_STEP: 450950 | > loss_text_ce: 0.03904729709029198 (0.03811155611802644) | > loss_mel_ce: 3.049335479736328 (nan) | > loss: 0.012255487963557243 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13615720266486434) | > loader_time: 0.0048 (0.004650636403505479)  --> TIME: 2025-02-24 19:20:03 -- STEP: 7744/36938 -- GLOBAL_STEP: 451000 | > loss_text_ce: 0.03743063658475876 (0.03810515640710468) | > loss_mel_ce: 2.93363881111145 (nan) | > loss: 0.011789957992732525 (nan) | > current_lr: 5e-06 | > step_time: 0.1572 (0.1361737791294893) | > loader_time: 0.0047 (0.0046507569014533475)  --> TIME: 2025-02-24 19:20:12 -- STEP: 7794/36938 -- GLOBAL_STEP: 451050 | > loss_text_ce: 0.03964823856949806 (0.038109096652816955) | > loss_mel_ce: 3.5658626556396484 (nan) | > loss: 0.014307583682239056 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.1362055566086351) | > loader_time: 0.0045 (0.004650775151036167)  --> TIME: 2025-02-24 19:20:22 -- STEP: 7844/36938 -- GLOBAL_STEP: 451100 | > loss_text_ce: 0.0396653488278389 (0.03811050132311188) | > loss_mel_ce: 3.1774916648864746 (nan) | > loss: 0.01276649720966816 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.1362045214286815) | > loader_time: 0.0046 (0.004651521979881024)  --> TIME: 2025-02-24 19:20:31 -- STEP: 7894/36938 -- GLOBAL_STEP: 451150 | > loss_text_ce: 0.03710603713989258 (0.03810379742936137) | > loss_mel_ce: 0.19968314468860626 (nan) | > loss: 0.0009396396926604211 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.13621167838195916) | > loader_time: 0.0047 (0.004651871366624079)  --> TIME: 2025-02-24 19:20:40 -- STEP: 7944/36938 -- GLOBAL_STEP: 451200 | > loss_text_ce: 0.038377344608306885 (0.038107580796525674) | > loss_mel_ce: 2.722194194793701 (nan) | > loss: 0.010954649187624454 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13619312884104087) | > loader_time: 0.0045 (0.0046518972332984165)  --> TIME: 2025-02-24 19:20:49 -- STEP: 7994/36938 -- GLOBAL_STEP: 451250 | > loss_text_ce: 0.04133136570453644 (0.03810988716974071) | > loss_mel_ce: 0.5043553113937378 (nan) | > loss: 0.002165423473343253 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13617825013027798) | > loader_time: 0.0045 (0.004652270174396313)  --> TIME: 2025-02-24 19:20:58 -- STEP: 8044/36938 -- GLOBAL_STEP: 451300 | > loss_text_ce: 0.03335503488779068 (0.03811051429021034) | > loss_mel_ce: 3.19804048538208 (nan) | > loss: 0.012822998687624931 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.13617300210693822) | > loader_time: 0.0047 (0.004652209005683051)  --> TIME: 2025-02-24 19:21:08 -- STEP: 8094/36938 -- GLOBAL_STEP: 451350 | > loss_text_ce: 0.03196115046739578 (0.03810633544784575) | > loss_mel_ce: 2.2743749618530273 (nan) | > loss: 0.009152128361165524 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13617481818928484) | > loader_time: 0.0045 (0.004652337407017153)  --> TIME: 2025-02-24 19:21:17 -- STEP: 8144/36938 -- GLOBAL_STEP: 451400 | > loss_text_ce: 0.03611641749739647 (0.038101895521340044) | > loss_mel_ce: 2.9168074131011963 (nan) | > loss: 0.011717951856553555 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13616132976264045) | > loader_time: 0.0047 (0.004652038421518441)  --> TIME: 2025-02-24 19:21:26 -- STEP: 8194/36938 -- GLOBAL_STEP: 451450 | > loss_text_ce: 0.041772544384002686 (0.0381092026191865) | > loss_mel_ce: 3.660907506942749 (nan) | > loss: 0.014693175442516804 (nan) | > current_lr: 5e-06 | > step_time: 0.1566 (0.1361453942272585) | > loader_time: 0.0046 (0.0046516324881957965)  --> TIME: 2025-02-24 19:21:35 -- STEP: 8244/36938 -- GLOBAL_STEP: 451500 | > loss_text_ce: 0.029398096725344658 (0.03811431770411326) | > loss_mel_ce: 0.10900236666202545 (nan) | > loss: 0.0005492081982083619 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13610617331310929) | > loader_time: 0.0045 (0.004651246777672838)  --> TIME: 2025-02-24 19:21:44 -- STEP: 8294/36938 -- GLOBAL_STEP: 451550 | > loss_text_ce: 0.038711994886398315 (0.038117032299819414) | > loss_mel_ce: 3.592879295349121 (nan) | > loss: 0.014411077834665775 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.13611548611535362) | > loader_time: 0.0048 (0.0046512139743283346)  --> TIME: 2025-02-24 19:21:53 -- STEP: 8344/36938 -- GLOBAL_STEP: 451600 | > loss_text_ce: 0.03880711644887924 (0.038111400534242626) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13611348109071558) | > loader_time: 0.0045 (0.004650985148814814)  --> TIME: 2025-02-24 19:22:03 -- STEP: 8394/36938 -- GLOBAL_STEP: 451650 | > loss_text_ce: 0.0370418056845665 (0.03811271802502497) | > loss_mel_ce: 3.17989444732666 (nan) | > loss: 0.012765620835125446 (nan) | > current_lr: 5e-06 | > step_time: 0.1711 (0.13612299031350092) | > loader_time: 0.0048 (0.004651255115657185)  --> TIME: 2025-02-24 19:22:12 -- STEP: 8444/36938 -- GLOBAL_STEP: 451700 | > loss_text_ce: 0.020444447174668312 (0.03811375503236373) | > loss_mel_ce: 1.3140391111373901 (nan) | > loss: 0.00529556954279542 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13611287158114924) | > loader_time: 0.0045 (0.004652099324763527)  --> TIME: 2025-02-24 19:22:21 -- STEP: 8494/36938 -- GLOBAL_STEP: 451750 | > loss_text_ce: 0.035263024270534515 (0.038106012939314766) | > loss_mel_ce: 3.075144052505493 (nan) | > loss: 0.012342886067926884 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13608764473735674) | > loader_time: 0.0046 (0.004651974110370922)  --> TIME: 2025-02-24 19:22:30 -- STEP: 8544/36938 -- GLOBAL_STEP: 451800 | > loss_text_ce: 0.05097455531358719 (0.038107965205519644) | > loss_mel_ce: 1.575292944908142 (nan) | > loss: 0.006453442852944136 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13609060963218117) | > loader_time: 0.0045 (0.004651973058891656)  --> TIME: 2025-02-24 19:22:40 -- STEP: 8594/36938 -- GLOBAL_STEP: 451850 | > loss_text_ce: 0.03810437023639679 (0.038110461328456066) | > loss_mel_ce: 3.4959957599639893 (nan) | > loss: 0.01402420736849308 (nan) | > current_lr: 5e-06 | > step_time: 0.1864 (0.13610749730737576) | > loader_time: 0.0049 (0.004652235239973065)  --> TIME: 2025-02-24 19:22:49 -- STEP: 8644/36938 -- GLOBAL_STEP: 451900 | > loss_text_ce: 0.04558024927973747 (0.03810699932781004) | > loss_mel_ce: 3.388190269470215 (nan) | > loss: 0.013626073487102985 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13609985674161248) | > loader_time: 0.0046 (0.004652468460900321)  --> TIME: 2025-02-24 19:22:59 -- STEP: 8694/36938 -- GLOBAL_STEP: 451950 | > loss_text_ce: 0.04282902926206589 (0.038111439089037774) | > loss_mel_ce: 2.883131504058838 (nan) | > loss: 0.011610954999923706 (nan) | > current_lr: 5e-06 | > step_time: 0.1813 (0.13612849179810416) | > loader_time: 0.0049 (0.004652433157351093)  --> TIME: 2025-02-24 19:23:07 -- STEP: 8744/36938 -- GLOBAL_STEP: 452000 | > loss_text_ce: 0.04061099514365196 (0.03811355137080039) | > loss_mel_ce: 3.477030038833618 (nan) | > loss: 0.013958893716335297 (nan) | > current_lr: 5e-06 | > step_time: 0.1712 (0.1360999808492511) | > loader_time: 0.0049 (0.004652209627661049)  --> TIME: 2025-02-24 19:23:17 -- STEP: 8794/36938 -- GLOBAL_STEP: 452050 | > loss_text_ce: 0.029776377603411674 (0.03809845359512614) | > loss_mel_ce: 2.9314873218536377 (nan) | > loss: 0.01175104733556509 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13610834500072252) | > loader_time: 0.0045 (0.004652011684593844)  --> TIME: 2025-02-24 19:23:26 -- STEP: 8844/36938 -- GLOBAL_STEP: 452100 | > loss_text_ce: 0.03841432183980942 (0.03809509465760711) | > loss_mel_ce: 3.268490791320801 (nan) | > loss: 0.013122639618813992 (nan) | > current_lr: 5e-06 | > step_time: 0.1883 (0.13610623544397943) | > loader_time: 0.0048 (0.004651754002719918)  --> TIME: 2025-02-24 19:23:36 -- STEP: 8894/36938 -- GLOBAL_STEP: 452150 | > loss_text_ce: 0.038011595606803894 (0.03808814404087908) | > loss_mel_ce: 1.5013657808303833 (nan) | > loss: 0.006108640693128109 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.136136368577175) | > loader_time: 0.0047 (0.004651783127609952)  --> TIME: 2025-02-24 19:23:45 -- STEP: 8944/36938 -- GLOBAL_STEP: 452200 | > loss_text_ce: 0.04460109770298004 (0.03809026778552327) | > loss_mel_ce: 2.1663005352020264 (nan) | > loss: 0.008773420006036758 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.1361206763012461) | > loader_time: 0.0046 (0.004651709458055995)  --> TIME: 2025-02-24 19:23:54 -- STEP: 8994/36938 -- GLOBAL_STEP: 452250 | > loss_text_ce: 0.03493432328104973 (0.0380866885452813) | > loss_mel_ce: 3.1895558834075928 (nan) | > loss: 0.01279559638351202 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13608767473620756) | > loader_time: 0.0045 (0.004651499345935083)  --> TIME: 2025-02-24 19:24:03 -- STEP: 9044/36938 -- GLOBAL_STEP: 452300 | > loss_text_ce: 0.03649844601750374 (0.03809035228267113) | > loss_mel_ce: 1.85321044921875 (nan) | > loss: 0.007498844992369413 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13611341513979988) | > loader_time: 0.0046 (0.004651402330461837)  --> TIME: 2025-02-24 19:24:13 -- STEP: 9094/36938 -- GLOBAL_STEP: 452350 | > loss_text_ce: 0.03620763123035431 (0.03809178317666596) | > loss_mel_ce: 0.10984548181295395 (nan) | > loss: 0.0005795758333988488 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.1360994157409412) | > loader_time: 0.0048 (0.004651148580891345)  --> TIME: 2025-02-24 19:24:22 -- STEP: 9144/36938 -- GLOBAL_STEP: 452400 | > loss_text_ce: 0.031078102067112923 (0.03809304242364551) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.1360849975913837) | > loader_time: 0.0046 (0.004651111645961372)  --> TIME: 2025-02-24 19:24:31 -- STEP: 9194/36938 -- GLOBAL_STEP: 452450 | > loss_text_ce: 0.023752033710479736 (0.03808600979571285) | > loss_mel_ce: 2.5466299057006836 (nan) | > loss: 0.010199928656220436 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13608453845313892) | > loader_time: 0.0045 (0.0046510235081295695)  --> TIME: 2025-02-24 19:24:40 -- STEP: 9244/36938 -- GLOBAL_STEP: 452500 | > loss_text_ce: 0.039261594414711 (0.03808469771420036) | > loss_mel_ce: 2.9776015281677246 (nan) | > loss: 0.011971679516136646 (nan) | > current_lr: 5e-06 | > step_time: 0.1619 (0.13607044381942612) | > loader_time: 0.0047 (0.0046510135957246596)  --> TIME: 2025-02-24 19:24:49 -- STEP: 9294/36938 -- GLOBAL_STEP: 452550 | > loss_text_ce: 0.02409215085208416 (0.03807981297925309) | > loss_mel_ce: 2.1022098064422607 (nan) | > loss: 0.008437707088887691 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.1360670784762884) | > loader_time: 0.0045 (0.00465092860115762)  --> TIME: 2025-02-24 19:24:58 -- STEP: 9344/36938 -- GLOBAL_STEP: 452600 | > loss_text_ce: 0.04181014373898506 (0.03807530839838187) | > loss_mel_ce: 3.24456524848938 (nan) | > loss: 0.013041172176599503 (nan) | > current_lr: 5e-06 | > step_time: 0.145 (0.13602144467606078) | > loader_time: 0.0047 (0.004650597677451288)  --> TIME: 2025-02-24 19:25:08 -- STEP: 9394/36938 -- GLOBAL_STEP: 452650 | > loss_text_ce: 0.03291279822587967 (0.038079123562354496) | > loss_mel_ce: 3.453489303588867 (nan) | > loss: 0.013834929093718529 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.1360254597608039) | > loader_time: 0.0049 (0.0046504772747479155)  --> TIME: 2025-02-24 19:25:17 -- STEP: 9444/36938 -- GLOBAL_STEP: 452700 | > loss_text_ce: 0.029425611719489098 (0.038082288554682174) | > loss_mel_ce: 3.364983320236206 (nan) | > loss: 0.013469877652823925 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13602325369472532) | > loader_time: 0.0045 (0.0046503248986224386)  --> TIME: 2025-02-24 19:25:26 -- STEP: 9494/36938 -- GLOBAL_STEP: 452750 | > loss_text_ce: 0.04135643690824509 (0.03808377737576485) | > loss_mel_ce: 3.5404202938079834 (nan) | > loss: 0.014213399961590767 (nan) | > current_lr: 5e-06 | > step_time: 0.1571 (0.13601449946240124) | > loader_time: 0.0046 (0.004650368749756348)  --> TIME: 2025-02-24 19:25:35 -- STEP: 9544/36938 -- GLOBAL_STEP: 452800 | > loss_text_ce: 0.0373552143573761 (0.038084588224854346) | > loss_mel_ce: 2.9420418739318848 (nan) | > loss: 0.01182300504297018 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.1359888672678784) | > loader_time: 0.0046 (0.004649952241381176)  --> TIME: 2025-02-24 19:25:44 -- STEP: 9594/36938 -- GLOBAL_STEP: 452850 | > loss_text_ce: 0.028849754482507706 (0.03808157532436083) | > loss_mel_ce: 3.1511151790618896 (nan) | > loss: 0.01261890958994627 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13599236972734058) | > loader_time: 0.0045 (0.004649713209874686)  --> TIME: 2025-02-24 19:25:54 -- STEP: 9644/36938 -- GLOBAL_STEP: 452900 | > loss_text_ce: 0.0410848893225193 (0.0380723133073074) | > loss_mel_ce: 3.074984550476074 (nan) | > loss: 0.012365356087684631 (nan) | > current_lr: 5e-06 | > step_time: 0.1872 (0.13600656715283868) | > loader_time: 0.0048 (0.004649702417855445)  --> TIME: 2025-02-24 19:26:03 -- STEP: 9694/36938 -- GLOBAL_STEP: 452950 | > loss_text_ce: 0.044003743678331375 (0.03807607526074893) | > loss_mel_ce: 2.59721302986145 (nan) | > loss: 0.010481019504368305 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13599046867410722) | > loader_time: 0.0045 (0.0046494959407623075)  --> TIME: 2025-02-24 19:26:12 -- STEP: 9744/36938 -- GLOBAL_STEP: 453000 | > loss_text_ce: 0.04188203066587448 (0.03807620835031359) | > loss_mel_ce: 3.5481338500976562 (nan) | > loss: 0.014246095903217793 (nan) | > current_lr: 5e-06 | > step_time: 0.169 (0.13598850105196367) | > loader_time: 0.0048 (0.00464934807986462)  --> TIME: 2025-02-24 19:26:21 -- STEP: 9794/36938 -- GLOBAL_STEP: 453050 | > loss_text_ce: 0.04041682928800583 (0.03808324233795594) | > loss_mel_ce: 3.148353099822998 (nan) | > loss: 0.012653849087655544 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1359787162089799) | > loader_time: 0.0045 (0.004649163680050316)  --> TIME: 2025-02-24 19:26:31 -- STEP: 9844/36938 -- GLOBAL_STEP: 453100 | > loss_text_ce: 0.030344486236572266 (0.03808897727851027) | > loss_mel_ce: 1.4754043817520142 (nan) | > loss: 0.005975194275379181 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.1359862660015077) | > loader_time: 0.0046 (0.004649597108485209)  --> TIME: 2025-02-24 19:26:40 -- STEP: 9894/36938 -- GLOBAL_STEP: 453150 | > loss_text_ce: 0.03586473688483238 (0.03808581313963244) | > loss_mel_ce: 2.3540608882904053 (nan) | > loss: 0.009483832865953445 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13600221144350577) | > loader_time: 0.0045 (0.004650042277286584)  --> TIME: 2025-02-24 19:26:50 -- STEP: 9944/36938 -- GLOBAL_STEP: 453200 | > loss_text_ce: 0.035457856953144073 (0.0380848612912989) | > loss_mel_ce: 2.7980918884277344 (nan) | > loss: 0.011244245804846287 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13600770830628542) | > loader_time: 0.0046 (0.0046501005501490165)  --> TIME: 2025-02-24 19:26:59 -- STEP: 9994/36938 -- GLOBAL_STEP: 453250 | > loss_text_ce: 0.030058320611715317 (0.038079756226805025) | > loss_mel_ce: 3.7109375 (nan) | > loss: 0.014845222234725952 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13599618203596547) | > loader_time: 0.0066 (0.004650334179198626)  --> TIME: 2025-02-24 19:27:08 -- STEP: 10044/36938 -- GLOBAL_STEP: 453300 | > loss_text_ce: 0.03567195683717728 (0.03808006260973183) | > loss_mel_ce: 3.1416513919830322 (nan) | > loss: 0.012608426623046398 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.1360307968218265) | > loader_time: 0.0045 (0.0046507206774004835)  --> TIME: 2025-02-24 19:27:18 -- STEP: 10094/36938 -- GLOBAL_STEP: 453350 | > loss_text_ce: 0.04633069038391113 (0.03808544232287297) | > loss_mel_ce: 3.140469551086426 (nan) | > loss: 0.012646033428609371 (nan) | > current_lr: 5e-06 | > step_time: 0.1447 (0.13603732327033607) | > loader_time: 0.0046 (0.004650892138079603)  --> TIME: 2025-02-24 19:27:27 -- STEP: 10144/36938 -- GLOBAL_STEP: 453400 | > loss_text_ce: 0.04166966304183006 (0.03808741187884271) | > loss_mel_ce: 3.3271570205688477 (nan) | > loss: 0.013368360698223114 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.1360081520977455) | > loader_time: 0.0049 (0.004650705878854924)  --> TIME: 2025-02-24 19:27:37 -- STEP: 10194/36938 -- GLOBAL_STEP: 453450 | > loss_text_ce: 0.04289725050330162 (0.0380836834251984) | > loss_mel_ce: 0.14015960693359375 (nan) | > loss: 0.0007264161831699312 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.1360465384287438) | > loader_time: 0.0046 (0.004650748943099092)  --> TIME: 2025-02-24 19:27:46 -- STEP: 10244/36938 -- GLOBAL_STEP: 453500 | > loss_text_ce: 0.038384001702070236 (0.03808421753385098) | > loss_mel_ce: 2.440642833709717 (nan) | > loss: 0.009837408550083637 (nan) | > current_lr: 5e-06 | > step_time: 0.1695 (0.13608593572540614) | > loader_time: 0.0048 (0.004650802130067969)  --> TIME: 2025-02-24 19:27:56 -- STEP: 10294/36938 -- GLOBAL_STEP: 453550 | > loss_text_ce: 0.04571804031729698 (0.038082431987986796) | > loss_mel_ce: 3.402830123901367 (nan) | > loss: 0.013684715144336224 (nan) | > current_lr: 5e-06 | > step_time: 0.1468 (0.1360889909937704) | > loader_time: 0.0046 (0.004650671690071306)  --> TIME: 2025-02-24 19:28:05 -- STEP: 10344/36938 -- GLOBAL_STEP: 453600 | > loss_text_ce: 0.03288444131612778 (0.038081751562736306) | > loss_mel_ce: 2.837566614151001 (nan) | > loss: 0.011390679515898228 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13609598428356684) | > loader_time: 0.0047 (0.004650598059120979)  --> TIME: 2025-02-24 19:28:15 -- STEP: 10394/36938 -- GLOBAL_STEP: 453650 | > loss_text_ce: 0.04034423828125 (0.038082365186468105) | > loss_mel_ce: 3.634251594543457 (nan) | > loss: 0.014581730589270592 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.1361130050954991) | > loader_time: 0.0046 (0.004650751397956998)  --> TIME: 2025-02-24 19:28:24 -- STEP: 10444/36938 -- GLOBAL_STEP: 453700 | > loss_text_ce: 0.03199407830834389 (0.038082661753954035) | > loss_mel_ce: 2.7387406826019287 (nan) | > loss: 0.010994980111718178 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.1361102220063373) | > loader_time: 0.0049 (0.00465107943444634)  --> TIME: 2025-02-24 19:28:33 -- STEP: 10494/36938 -- GLOBAL_STEP: 453750 | > loss_text_ce: 0.046242497861385345 (0.0380840777404952) | > loss_mel_ce: 3.6435296535491943 (nan) | > loss: 0.014641953632235527 (nan) | > current_lr: 5e-06 | > step_time: 0.1629 (0.1361075796943768) | > loader_time: 0.0046 (0.004651800368839571)  --> TIME: 2025-02-24 19:28:42 -- STEP: 10544/36938 -- GLOBAL_STEP: 453800 | > loss_text_ce: 0.039290644228458405 (0.03807847918273041) | > loss_mel_ce: 3.7012109756469727 (nan) | > loss: 0.014843261800706387 (nan) | > current_lr: 5e-06 | > step_time: 0.1583 (0.13609426110214107) | > loader_time: 0.0047 (0.0046519440159269515)  --> TIME: 2025-02-24 19:28:52 -- STEP: 10594/36938 -- GLOBAL_STEP: 453850 | > loss_text_ce: 0.034233737736940384 (0.03807786381229391) | > loss_mel_ce: 3.1731927394866943 (nan) | > loss: 0.012727883644402027 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13608659630297087) | > loader_time: 0.0045 (0.004652294928968601)  --> TIME: 2025-02-24 19:29:01 -- STEP: 10644/36938 -- GLOBAL_STEP: 453900 | > loss_text_ce: 0.03875967860221863 (0.03807498818942032) | > loss_mel_ce: 2.998412609100342 (nan) | > loss: 0.012052271515130997 (nan) | > current_lr: 5e-06 | > step_time: 0.1796 (0.13607528107993266) | > loader_time: 0.0048 (0.004652684745552045)  --> TIME: 2025-02-24 19:29:10 -- STEP: 10694/36938 -- GLOBAL_STEP: 453950 | > loss_text_ce: 0.02919549122452736 (0.03807050832567268) | > loss_mel_ce: 2.723879098892212 (nan) | > loss: 0.010924899950623512 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13607660500562255) | > loader_time: 0.0045 (0.004652482562139152)  --> TIME: 2025-02-24 19:29:19 -- STEP: 10744/36938 -- GLOBAL_STEP: 454000 | > loss_text_ce: 0.03451450541615486 (0.03807118504087284) | > loss_mel_ce: 2.7291829586029053 (nan) | > loss: 0.010967053472995758 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.136082468930235) | > loader_time: 0.0046 (0.0046523250445268715)  --> TIME: 2025-02-24 19:29:29 -- STEP: 10794/36938 -- GLOBAL_STEP: 454050 | > loss_text_ce: 0.046452101320028305 (0.03807001783817439) | > loss_mel_ce: 2.9965713024139404 (nan) | > loss: 0.012075490318238735 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13609777233303283) | > loader_time: 0.0046 (0.004652909046856595)  --> TIME: 2025-02-24 19:29:38 -- STEP: 10844/36938 -- GLOBAL_STEP: 454100 | > loss_text_ce: 0.032643675804138184 (0.03806549386974661) | > loss_mel_ce: 3.1296234130859375 (nan) | > loss: 0.012548680417239666 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13612600878480144) | > loader_time: 0.0046 (0.00465323227175661)  --> TIME: 2025-02-24 19:29:48 -- STEP: 10894/36938 -- GLOBAL_STEP: 454150 | > loss_text_ce: 0.03519239276647568 (0.03806857599905701) | > loss_mel_ce: 2.822812795639038 (nan) | > loss: 0.011341291479766369 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13612909872465775) | > loader_time: 0.0046 (0.004653191400138926)  --> TIME: 2025-02-24 19:29:57 -- STEP: 10944/36938 -- GLOBAL_STEP: 454200 | > loss_text_ce: 0.04177166521549225 (0.03806890205645913) | > loss_mel_ce: 3.1760809421539307 (nan) | > loss: 0.012769256718456745 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.1361532845910179) | > loader_time: 0.0045 (0.0046531239317523145)  --> TIME: 2025-02-24 19:30:07 -- STEP: 10994/36938 -- GLOBAL_STEP: 454250 | > loss_text_ce: 0.031247815117239952 (0.0380673857807484) | > loss_mel_ce: 2.832366466522217 (nan) | > loss: 0.011363549157977104 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13617135438785452) | > loader_time: 0.0045 (0.004653015678007434)  --> TIME: 2025-02-24 19:30:16 -- STEP: 11044/36938 -- GLOBAL_STEP: 454300 | > loss_text_ce: 0.050666019320487976 (0.0380653125465512) | > loss_mel_ce: 3.1160008907318115 (nan) | > loss: 0.012566139921545982 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.1361688114257445) | > loader_time: 0.0046 (0.004652919025794198)  --> TIME: 2025-02-24 19:30:25 -- STEP: 11094/36938 -- GLOBAL_STEP: 454350 | > loss_text_ce: 0.03011723980307579 (0.03806400708227922) | > loss_mel_ce: 0.53868567943573 (nan) | > loss: 0.002257154555991292 (nan) | > current_lr: 5e-06 | > step_time: 0.1799 (0.13616341589935718) | > loader_time: 0.0048 (0.004652857006788912)  --> TIME: 2025-02-24 19:30:35 -- STEP: 11144/36938 -- GLOBAL_STEP: 454400 | > loss_text_ce: 0.04418444633483887 (0.03806608796515129) | > loss_mel_ce: 2.963803291320801 (nan) | > loss: 0.011936459690332413 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.1361551786059535) | > loader_time: 0.0047 (0.0046527898748059725)  --> TIME: 2025-02-24 19:30:44 -- STEP: 11194/36938 -- GLOBAL_STEP: 454450 | > loss_text_ce: 0.03261556103825569 (0.0380673864050168) | > loss_mel_ce: 0.7752510905265808 (nan) | > loss: 0.00320582021959126 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13615854603915775) | > loader_time: 0.0045 (0.0046527972493147045)  --> TIME: 2025-02-24 19:30:53 -- STEP: 11244/36938 -- GLOBAL_STEP: 454500 | > loss_text_ce: 0.04492773488163948 (0.03806801193484196) | > loss_mel_ce: 2.559377670288086 (nan) | > loss: 0.01033454667776823 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13612011996336712) | > loader_time: 0.0046 (0.004652695187636865)  --> TIME: 2025-02-24 19:31:02 -- STEP: 11294/36938 -- GLOBAL_STEP: 454550 | > loss_text_ce: 0.03321570158004761 (0.03805998904946845) | > loss_mel_ce: 2.99652099609375 (nan) | > loss: 0.012022765353322029 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.13612231582123452) | > loader_time: 0.0047 (0.004652536567679892)  --> TIME: 2025-02-24 19:31:11 -- STEP: 11344/36938 -- GLOBAL_STEP: 454600 | > loss_text_ce: 0.03355386108160019 (0.03806197156125397) | > loss_mel_ce: 3.5471811294555664 (nan) | > loss: 0.014209266752004623 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13611408181183737) | > loader_time: 0.0045 (0.004652266231679455)  --> TIME: 2025-02-24 19:31:20 -- STEP: 11394/36938 -- GLOBAL_STEP: 454650 | > loss_text_ce: 0.06841444969177246 (0.038070412534302725) | > loss_mel_ce: 3.5773797035217285 (nan) | > loss: 0.014467437751591206 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13610321110709805) | > loader_time: 0.0059 (0.004652123399339109)  --> TIME: 2025-02-24 19:31:29 -- STEP: 11444/36938 -- GLOBAL_STEP: 454700 | > loss_text_ce: 0.043781835585832596 (0.038062132034715535) | > loss_mel_ce: 3.0189902782440186 (nan) | > loss: 0.012153858318924904 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13607892113855247) | > loader_time: 0.0046 (0.0046528753847644285)  --> TIME: 2025-02-24 19:31:39 -- STEP: 11494/36938 -- GLOBAL_STEP: 454750 | > loss_text_ce: 0.03262951597571373 (0.03805546113075149) | > loss_mel_ce: 2.8777761459350586 (nan) | > loss: 0.011549229733645916 (nan) | > current_lr: 5e-06 | > step_time: 0.1478 (0.13610349617067413) | > loader_time: 0.0047 (0.004653046748027077)  --> TIME: 2025-02-24 19:31:48 -- STEP: 11544/36938 -- GLOBAL_STEP: 454800 | > loss_text_ce: 0.03678762540221214 (0.038057976824519044) | > loss_mel_ce: 2.4489858150482178 (nan) | > loss: 0.009864180348813534 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.1361115795617619) | > loader_time: 0.0046 (0.004653393520056504)  --> TIME: 2025-02-24 19:31:58 -- STEP: 11594/36938 -- GLOBAL_STEP: 454850 | > loss_text_ce: 0.039243705570697784 (0.03805405639790901) | > loss_mel_ce: 2.7393012046813965 (nan) | > loss: 0.011025972664356232 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.1361113198156783) | > loader_time: 0.0046 (0.004653478318583418)  --> TIME: 2025-02-24 19:32:07 -- STEP: 11644/36938 -- GLOBAL_STEP: 454900 | > loss_text_ce: 0.03684370592236519 (0.038062939121765484) | > loss_mel_ce: 3.0068023204803467 (nan) | > loss: 0.01207796111702919 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13611178548014125) | > loader_time: 0.0047 (0.004653716836750448)  --> TIME: 2025-02-24 19:32:17 -- STEP: 11694/36938 -- GLOBAL_STEP: 454950 | > loss_text_ce: 0.036781128495931625 (0.03806506793260136) | > loss_mel_ce: 3.091050148010254 (nan) | > loss: 0.012412029318511486 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13613035739212295) | > loader_time: 0.0045 (0.004653706782207095)  --> TIME: 2025-02-24 19:32:26 -- STEP: 11744/36938 -- GLOBAL_STEP: 455000 | > loss_text_ce: 0.03363652527332306 (0.03807244309671745) | > loss_mel_ce: 3.0523881912231445 (nan) | > loss: 0.012246130965650082 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.13615561806538426) | > loader_time: 0.0047 (0.004653829157514857)  --> TIME: 2025-02-24 19:32:35 -- STEP: 11794/36938 -- GLOBAL_STEP: 455050 | > loss_text_ce: 0.03646516799926758 (0.038067700614012524) | > loss_mel_ce: 2.2704615592956543 (nan) | > loss: 0.009154471568763256 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1361441783466761) | > loader_time: 0.0045 (0.004653709024135308)  --> TIME: 2025-02-24 19:32:45 -- STEP: 11844/36938 -- GLOBAL_STEP: 455100 | > loss_text_ce: 0.03586394712328911 (0.038067780618141395) | > loss_mel_ce: 2.494080066680908 (nan) | > loss: 0.010039460845291615 (nan) | > current_lr: 5e-06 | > step_time: 0.1118 (0.13616239956124962) | > loader_time: 0.0045 (0.00465407022468466)  --> TIME: 2025-02-24 19:32:55 -- STEP: 11894/36938 -- GLOBAL_STEP: 455150 | > loss_text_ce: 0.040161702781915665 (0.03806408844923891) | > loss_mel_ce: 3.2935922145843506 (nan) | > loss: 0.013229182921350002 (nan) | > current_lr: 5e-06 | > step_time: 0.1693 (0.13619282986509587) | > loader_time: 0.0048 (0.004654225890969595)  --> TIME: 2025-02-24 19:33:04 -- STEP: 11944/36938 -- GLOBAL_STEP: 455200 | > loss_text_ce: 0.03980502858757973 (0.03805814051706814) | > loss_mel_ce: 3.3178515434265137 (nan) | > loss: 0.01332403440028429 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13619724866214875) | > loader_time: 0.0046 (0.004654234136739832)  --> TIME: 2025-02-24 19:33:13 -- STEP: 11994/36938 -- GLOBAL_STEP: 455250 | > loss_text_ce: 0.04345963895320892 (0.03805992082326624) | > loss_mel_ce: 2.428652048110962 (nan) | > loss: 0.009809968061745167 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1361890466526904) | > loader_time: 0.0046 (0.004654026536399253)  --> TIME: 2025-02-24 19:33:23 -- STEP: 12044/36938 -- GLOBAL_STEP: 455300 | > loss_text_ce: 0.039894986897706985 (0.03806643779131265) | > loss_mel_ce: 2.930382251739502 (nan) | > loss: 0.011786815710365772 (nan) | > current_lr: 5e-06 | > step_time: 0.1779 (0.13621444919504466) | > loader_time: 0.0047 (0.004654074182386851)  --> TIME: 2025-02-24 19:33:32 -- STEP: 12094/36938 -- GLOBAL_STEP: 455350 | > loss_text_ce: 0.041531361639499664 (0.03806631440536697) | > loss_mel_ce: 3.5701422691345215 (nan) | > loss: 0.014332039281725883 (nan) | > current_lr: 5e-06 | > step_time: 0.1856 (0.1362053771954993) | > loader_time: 0.0048 (0.0046538794870630475)  --> TIME: 2025-02-24 19:33:41 -- STEP: 12144/36938 -- GLOBAL_STEP: 455400 | > loss_text_ce: 0.03588641807436943 (0.03806054218836615) | > loss_mel_ce: 2.7901179790496826 (nan) | > loss: 0.0112143037840724 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13621264493905202) | > loader_time: 0.0045 (0.00465431285234308)  --> TIME: 2025-02-24 19:33:51 -- STEP: 12194/36938 -- GLOBAL_STEP: 455450 | > loss_text_ce: 0.0402207225561142 (0.03805376760638969) | > loss_mel_ce: 2.5662968158721924 (nan) | > loss: 0.010343324393033981 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13621233540791974) | > loader_time: 0.0046 (0.004654661268215797)  --> TIME: 2025-02-24 19:34:00 -- STEP: 12244/36938 -- GLOBAL_STEP: 455500 | > loss_text_ce: 0.03979134559631348 (0.038049015870904146) | > loss_mel_ce: 3.1885833740234375 (nan) | > loss: 0.012811011634767056 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.13620362471459313) | > loader_time: 0.0049 (0.004654460796847213)  --> TIME: 2025-02-24 19:34:09 -- STEP: 12294/36938 -- GLOBAL_STEP: 455550 | > loss_text_ce: 0.03908292576670647 (0.03804392164133066) | > loss_mel_ce: 3.448944330215454 (nan) | > loss: 0.013841379433870316 (nan) | > current_lr: 5e-06 | > step_time: 0.1823 (0.13618905415898783) | > loader_time: 0.0048 (0.004654596254110763)  --> TIME: 2025-02-24 19:34:18 -- STEP: 12344/36938 -- GLOBAL_STEP: 455600 | > loss_text_ce: 0.03887522593140602 (0.038036547495068254) | > loss_mel_ce: 3.2491822242736816 (nan) | > loss: 0.013047847896814346 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13618547288303767) | > loader_time: 0.0053 (0.004654746065646587)  --> TIME: 2025-02-24 19:34:27 -- STEP: 12394/36938 -- GLOBAL_STEP: 455650 | > loss_text_ce: 0.028097059577703476 (0.0380357139661505) | > loss_mel_ce: 2.6475160121917725 (nan) | > loss: 0.010617513209581375 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13618386315014402) | > loader_time: 0.0046 (0.004654716171909773)  --> TIME: 2025-02-24 19:34:37 -- STEP: 12444/36938 -- GLOBAL_STEP: 455700 | > loss_text_ce: 0.035724736750125885 (0.038034978769036755) | > loss_mel_ce: 2.3542327880859375 (nan) | > loss: 0.009483958594501019 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13619842352264722) | > loader_time: 0.0047 (0.004654764880018196)  --> TIME: 2025-02-24 19:34:46 -- STEP: 12494/36938 -- GLOBAL_STEP: 455750 | > loss_text_ce: 0.04972408339381218 (0.03803648025289981) | > loss_mel_ce: 2.4123048782348633 (nan) | > loss: 0.009769956581294537 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13620591051047604) | > loader_time: 0.0045 (0.004654871953741657)  --> TIME: 2025-02-24 19:34:56 -- STEP: 12544/36938 -- GLOBAL_STEP: 455800 | > loss_text_ce: 0.04085560142993927 (0.03803637864591394) | > loss_mel_ce: 2.820849657058716 (nan) | > loss: 0.011355973780155182 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.1362033358261897) | > loader_time: 0.0045 (0.00465493082848131)  --> TIME: 2025-02-24 19:35:05 -- STEP: 12594/36938 -- GLOBAL_STEP: 455850 | > loss_text_ce: 0.04499564692378044 (0.03803744980684078) | > loss_mel_ce: 3.240873336791992 (nan) | > loss: 0.013039163313806057 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.1362140110149748) | > loader_time: 0.0047 (0.004654982079773533)  --> TIME: 2025-02-24 19:35:14 -- STEP: 12644/36938 -- GLOBAL_STEP: 455900 | > loss_text_ce: 0.04712774232029915 (0.038037063091214025) | > loss_mel_ce: 3.511462688446045 (nan) | > loss: 0.014121390879154205 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.13620863389606808) | > loader_time: 0.0047 (0.004654920146420807)  --> TIME: 2025-02-24 19:35:24 -- STEP: 12694/36938 -- GLOBAL_STEP: 455950 | > loss_text_ce: 0.03864315152168274 (0.038039796050107655) | > loss_mel_ce: 2.728337526321411 (nan) | > loss: 0.010980082675814629 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.13620767241033643) | > loader_time: 0.0047 (0.004654749370999882)  --> TIME: 2025-02-24 19:35:33 -- STEP: 12744/36938 -- GLOBAL_STEP: 456000 | > loss_text_ce: 0.04457122087478638 (0.03804295253676811) | > loss_mel_ce: 2.940490484237671 (nan) | > loss: 0.0118454834446311 (nan) | > current_lr: 5e-06 | > step_time: 0.1504 (0.13619701647563906) | > loader_time: 0.0047 (0.004655259046925744)  --> TIME: 2025-02-24 19:35:42 -- STEP: 12794/36938 -- GLOBAL_STEP: 456050 | > loss_text_ce: 0.036481086164712906 (0.03804607201284307) | > loss_mel_ce: 2.752460479736328 (nan) | > loss: 0.011067229323089123 (nan) | > current_lr: 5e-06 | > step_time: 0.1793 (0.13620939636036658) | > loader_time: 0.0048 (0.004655437002111941)  --> TIME: 2025-02-24 19:35:51 -- STEP: 12844/36938 -- GLOBAL_STEP: 456100 | > loss_text_ce: 0.03969348222017288 (0.0380409944864008) | > loss_mel_ce: 3.389329195022583 (nan) | > loss: 0.013607233762741089 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13619855589229554) | > loader_time: 0.0046 (0.004655741486954786)  --> TIME: 2025-02-24 19:36:01 -- STEP: 12894/36938 -- GLOBAL_STEP: 456150 | > loss_text_ce: 0.030133770778775215 (0.03804211884668219) | > loss_mel_ce: 3.628610849380493 (nan) | > loss: 0.014518828131258488 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.1362052277407757) | > loader_time: 0.0048 (0.004655997642569161)  --> TIME: 2025-02-24 19:36:10 -- STEP: 12944/36938 -- GLOBAL_STEP: 456200 | > loss_text_ce: 0.03814249113202095 (0.03803821160441882) | > loss_mel_ce: 3.003955602645874 (nan) | > loss: 0.012071818113327026 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13621389241463158) | > loader_time: 0.0046 (0.004656178952736964)  --> TIME: 2025-02-24 19:36:19 -- STEP: 12994/36938 -- GLOBAL_STEP: 456250 | > loss_text_ce: 0.039331939071416855 (0.03803532946598496) | > loss_mel_ce: 4.127580165863037 (nan) | > loss: 0.01653536595404148 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13620302342113486) | > loader_time: 0.0046 (0.004656379014065543)  --> TIME: 2025-02-24 19:36:29 -- STEP: 13044/36938 -- GLOBAL_STEP: 456300 | > loss_text_ce: 0.036834217607975006 (0.03803071111374373) | > loss_mel_ce: 1.76783287525177 (nan) | > loss: 0.007161377929151058 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.13621281987134376) | > loader_time: 0.0045 (0.004656446798927611)  --> TIME: 2025-02-24 19:36:38 -- STEP: 13094/36938 -- GLOBAL_STEP: 456350 | > loss_text_ce: 0.03759019449353218 (0.03802762307939029) | > loss_mel_ce: 1.9566866159439087 (nan) | > loss: 0.007913797162473202 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13621803558708515) | > loader_time: 0.0046 (0.0046565592043159845)  --> TIME: 2025-02-24 19:36:47 -- STEP: 13144/36938 -- GLOBAL_STEP: 456400 | > loss_text_ce: 0.023554230108857155 (0.03802621325339624) | > loss_mel_ce: 1.3622775077819824 (nan) | > loss: 0.005499332677572966 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13621511066823333) | > loader_time: 0.0045 (0.004656466582310469)  --> TIME: 2025-02-24 19:36:57 -- STEP: 13194/36938 -- GLOBAL_STEP: 456450 | > loss_text_ce: 0.025394240394234657 (0.03802996667685566) | > loss_mel_ce: 0.2138671875 (nan) | > loss: 0.0009494501864537597 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13624078198890163) | > loader_time: 0.0046 (0.0046565671102267335)  --> TIME: 2025-02-24 19:37:07 -- STEP: 13244/36938 -- GLOBAL_STEP: 456500 | > loss_text_ce: 0.0351702943444252 (0.03802368834072418) | > loss_mel_ce: 3.195796251296997 (nan) | > loss: 0.012821296229958534 (nan) | > current_lr: 5e-06 | > step_time: 0.1774 (0.13625479884033873) | > loader_time: 0.0048 (0.004656622054100898)  --> TIME: 2025-02-24 19:37:16 -- STEP: 13294/36938 -- GLOBAL_STEP: 456550 | > loss_text_ce: 0.04204946756362915 (0.038027677701835474) | > loss_mel_ce: 0.16708984971046448 (nan) | > loss: 0.0008299179608002305 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13625807663148673) | > loader_time: 0.0047 (0.004656573265176683)  --> TIME: 2025-02-24 19:37:25 -- STEP: 13344/36938 -- GLOBAL_STEP: 456600 | > loss_text_ce: 0.04000011831521988 (0.03801892964924239) | > loss_mel_ce: 2.7892372608184814 (nan) | > loss: 0.011227133683860302 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13626765876079366) | > loader_time: 0.0047 (0.004656473688370312)  --> TIME: 2025-02-24 19:37:35 -- STEP: 13394/36938 -- GLOBAL_STEP: 456650 | > loss_text_ce: 0.034124936908483505 (0.03801674118433191) | > loss_mel_ce: 3.236088991165161 (nan) | > loss: 0.012977039441466331 (nan) | > current_lr: 5e-06 | > step_time: 0.169 (0.13626246283583754) | > loader_time: 0.0046 (0.004656561866026589)  --> TIME: 2025-02-24 19:37:44 -- STEP: 13444/36938 -- GLOBAL_STEP: 456700 | > loss_text_ce: 0.03148472309112549 (0.03801914891261302) | > loss_mel_ce: 2.013810873031616 (nan) | > loss: 0.008116253651678562 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.1362749534467616) | > loader_time: 0.0046 (0.0046565909358725135)  --> TIME: 2025-02-24 19:37:54 -- STEP: 13494/36938 -- GLOBAL_STEP: 456750 | > loss_text_ce: 0.03016394004225731 (0.03801951798172374) | > loss_mel_ce: 2.7611348628997803 (nan) | > loss: 0.01107658352702856 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.1362876169095311) | > loader_time: 0.0046 (0.00465655664311385)  --> TIME: 2025-02-24 19:38:03 -- STEP: 13544/36938 -- GLOBAL_STEP: 456800 | > loss_text_ce: 0.033889733254909515 (0.03802264605817117) | > loss_mel_ce: 3.154064416885376 (nan) | > loss: 0.012650612741708755 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.1362949071976821) | > loader_time: 0.0046 (0.004656475197966114)  --> TIME: 2025-02-24 19:38:12 -- STEP: 13594/36938 -- GLOBAL_STEP: 456850 | > loss_text_ce: 0.04280808940529823 (0.038026327891550395) | > loss_mel_ce: 2.838716983795166 (nan) | > loss: 0.011434623971581459 (nan) | > current_lr: 5e-06 | > step_time: 0.144 (0.13628605316364745) | > loader_time: 0.0047 (0.004656387932847417)  --> TIME: 2025-02-24 19:38:21 -- STEP: 13644/36938 -- GLOBAL_STEP: 456900 | > loss_text_ce: 0.027592822909355164 (0.038021642084573634) | > loss_mel_ce: 3.0964181423187256 (nan) | > loss: 0.01239687018096447 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13626888526286351) | > loader_time: 0.0046 (0.004656251383400913)  --> TIME: 2025-02-24 19:38:30 -- STEP: 13694/36938 -- GLOBAL_STEP: 456950 | > loss_text_ce: 0.03431538864970207 (0.03801569406106892) | > loss_mel_ce: 0.1960906982421875 (nan) | > loss: 0.0009143099305219948 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13625694350185116) | > loader_time: 0.0046 (0.004656316416793202)  --> TIME: 2025-02-24 19:38:40 -- STEP: 13744/36938 -- GLOBAL_STEP: 457000 | > loss_text_ce: 0.0448354035615921 (0.03801781429740548) | > loss_mel_ce: 2.6392412185668945 (nan) | > loss: 0.010651097632944584 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13625469151559086) | > loader_time: 0.0047 (0.004656505216958219)  --> TIME: 2025-02-24 19:38:49 -- STEP: 13794/36938 -- GLOBAL_STEP: 457050 | > loss_text_ce: 0.026489367708563805 (0.03801914238652739) | > loss_mel_ce: 0.004052465781569481 (nan) | > loss: 0.00012119775783503428 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13624651272676838) | > loader_time: 0.0045 (0.004656466294219911)  --> TIME: 2025-02-24 19:38:58 -- STEP: 13844/36938 -- GLOBAL_STEP: 457100 | > loss_text_ce: 0.04400699958205223 (0.03802033969462405) | > loss_mel_ce: 2.9405577182769775 (nan) | > loss: 0.011843511834740639 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.1362568848879138) | > loader_time: 0.0046 (0.004656592344693521)  --> TIME: 2025-02-24 19:39:07 -- STEP: 13894/36938 -- GLOBAL_STEP: 457150 | > loss_text_ce: 0.04323114827275276 (0.03801836582254339) | > loss_mel_ce: 2.1844594478607178 (nan) | > loss: 0.008840043097734451 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13624435964378198) | > loader_time: 0.0046 (0.004656511158227267)  --> TIME: 2025-02-24 19:39:17 -- STEP: 13944/36938 -- GLOBAL_STEP: 457200 | > loss_text_ce: 0.04282771423459053 (0.03801666778670963) | > loss_mel_ce: 2.9589221477508545 (nan) | > loss: 0.011911706067621708 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.1362521186924094) | > loader_time: 0.0048 (0.00465654514674407)  --> TIME: 2025-02-24 19:39:26 -- STEP: 13994/36938 -- GLOBAL_STEP: 457250 | > loss_text_ce: 0.03318231552839279 (0.03802066719141087) | > loss_mel_ce: 1.992368221282959 (nan) | > loss: 0.008037899620831013 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13625284155828918) | > loader_time: 0.0045 (0.004656455611201272)  --> TIME: 2025-02-24 19:39:36 -- STEP: 14044/36938 -- GLOBAL_STEP: 457300 | > loss_text_ce: 0.025661420077085495 (0.038019192884234214) | > loss_mel_ce: 2.889007568359375 (nan) | > loss: 0.011566147208213806 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.136268722250468) | > loader_time: 0.0045 (0.004656409392225133)  --> TIME: 2025-02-24 19:39:45 -- STEP: 14094/36938 -- GLOBAL_STEP: 457350 | > loss_text_ce: 0.04872341454029083 (0.03802554527152477) | > loss_mel_ce: 2.9016199111938477 (nan) | > loss: 0.011707712896168232 (nan) | > current_lr: 5e-06 | > step_time: 0.1505 (0.1362614677326893) | > loader_time: 0.0047 (0.004656497935158013)  --> TIME: 2025-02-24 19:39:54 -- STEP: 14144/36938 -- GLOBAL_STEP: 457400 | > loss_text_ce: 0.029369864612817764 (0.038028595193812534) | > loss_mel_ce: 2.6446380615234375 (nan) | > loss: 0.010611142963171005 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13625425914496367) | > loader_time: 0.0046 (0.004656431378956832)  --> TIME: 2025-02-24 19:40:04 -- STEP: 14194/36938 -- GLOBAL_STEP: 457450 | > loss_text_ce: 0.033025648444890976 (0.03802612463238808) | > loss_mel_ce: 3.130061149597168 (nan) | > loss: 0.012551932595670223 (nan) | > current_lr: 5e-06 | > step_time: 0.1932 (0.1362672180831815) | > loader_time: 0.0048 (0.004656525385183263)  --> TIME: 2025-02-24 19:40:13 -- STEP: 14244/36938 -- GLOBAL_STEP: 457500 | > loss_text_ce: 0.03667127341032028 (0.03802474161515303) | > loss_mel_ce: 3.0440914630889893 (nan) | > loss: 0.012225248850882053 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.13627201693676025) | > loader_time: 0.0048 (0.004656346468267999)  --> TIME: 2025-02-24 19:40:22 -- STEP: 14294/36938 -- GLOBAL_STEP: 457550 | > loss_text_ce: 0.03783034533262253 (0.03802361867957521) | > loss_mel_ce: 0.04559326171875 (nan) | > loss: 0.0003310460888314992 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13627448121334168) | > loader_time: 0.0046 (0.004656474824149873)  --> TIME: 2025-02-24 19:40:32 -- STEP: 14344/36938 -- GLOBAL_STEP: 457600 | > loss_text_ce: 0.043102677911520004 (0.038023088646795766) | > loss_mel_ce: 2.014756917953491 (nan) | > loss: 0.008166110143065453 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13626525776717402) | > loader_time: 0.0045 (0.004656673940412835)  --> TIME: 2025-02-24 19:40:41 -- STEP: 14394/36938 -- GLOBAL_STEP: 457650 | > loss_text_ce: 0.03337697312235832 (0.038025426462642294) | > loss_mel_ce: 3.298358201980591 (nan) | > loss: 0.01322117168456316 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.1362738373280567) | > loader_time: 0.0047 (0.004656664858398393)  --> TIME: 2025-02-24 19:40:50 -- STEP: 14444/36938 -- GLOBAL_STEP: 457700 | > loss_text_ce: 0.04054780304431915 (0.038024975653872646) | > loss_mel_ce: 3.233893871307373 (nan) | > loss: 0.01299381721764803 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.13627965557248414) | > loader_time: 0.0046 (0.004656600938943308)  --> TIME: 2025-02-24 19:41:00 -- STEP: 14494/36938 -- GLOBAL_STEP: 457750 | > loss_text_ce: 0.032235730439424515 (0.03801811515436225) | > loss_mel_ce: 1.323301911354065 (nan) | > loss: 0.005379118025302887 (nan) | > current_lr: 5e-06 | > step_time: 0.1807 (0.13627823293661445) | > loader_time: 0.0048 (0.0046563756635341355)  --> TIME: 2025-02-24 19:41:09 -- STEP: 14544/36938 -- GLOBAL_STEP: 457800 | > loss_text_ce: 0.035811323672533035 (0.03801365490304534) | > loss_mel_ce: 0.03353118896484375 (nan) | > loss: 0.00027516871341504157 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.13627692747234127) | > loader_time: 0.0046 (0.004656151838690795)  --> TIME: 2025-02-24 19:41:18 -- STEP: 14594/36938 -- GLOBAL_STEP: 457850 | > loss_text_ce: 0.035911694169044495 (0.03801442174465804) | > loss_mel_ce: 3.2282607555389404 (nan) | > loss: 0.012953066267073154 (nan) | > current_lr: 5e-06 | > step_time: 0.1963 (0.1362798887196349) | > loader_time: 0.0048 (0.004656110460731742)  --> TIME: 2025-02-24 19:41:28 -- STEP: 14644/36938 -- GLOBAL_STEP: 457900 | > loss_text_ce: 0.022590868175029755 (0.03800939738585585) | > loss_mel_ce: 1.9822126626968384 (nan) | > loss: 0.00795556977391243 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1362781473210327) | > loader_time: 0.0045 (0.004656119950317156)  --> TIME: 2025-02-24 19:41:37 -- STEP: 14694/36938 -- GLOBAL_STEP: 457950 | > loss_text_ce: 0.03231852874159813 (0.038010745331349634) | > loss_mel_ce: 2.2081682682037354 (nan) | > loss: 0.008890821598470211 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13627360361002405) | > loader_time: 0.0046 (0.004655943186863661)  --> TIME: 2025-02-24 19:41:46 -- STEP: 14744/36938 -- GLOBAL_STEP: 458000 | > loss_text_ce: 0.0472363717854023 (0.03800839154763252) | > loss_mel_ce: 3.072373867034912 (nan) | > loss: 0.012379406951367855 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13628238662713518) | > loader_time: 0.0046 (0.004655917523286293)  --> TIME: 2025-02-24 19:41:56 -- STEP: 14794/36938 -- GLOBAL_STEP: 458050 | > loss_text_ce: 0.030391590669751167 (0.03801040775125585) | > loss_mel_ce: 3.080477237701416 (nan) | > loss: 0.012344718910753727 (nan) | > current_lr: 5e-06 | > step_time: 0.1435 (0.13629668458693892) | > loader_time: 0.0046 (0.004655858334841332)  --> TIME: 2025-02-24 19:42:05 -- STEP: 14844/36938 -- GLOBAL_STEP: 458100 | > loss_text_ce: 0.03799924999475479 (0.038005162365995306) | > loss_mel_ce: 3.178300619125366 (nan) | > loss: 0.01276309508830309 (nan) | > current_lr: 5e-06 | > step_time: 0.1708 (0.13629141349543195) | > loader_time: 0.0048 (0.004655853945815121)  --> TIME: 2025-02-24 19:42:14 -- STEP: 14894/36938 -- GLOBAL_STEP: 458150 | > loss_text_ce: 0.02337762527167797 (0.03800597599075588) | > loss_mel_ce: 2.506103038787842 (nan) | > loss: 0.010037622414529324 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13629625388499267) | > loader_time: 0.0045 (0.004655852195511283)  --> TIME: 2025-02-24 19:42:24 -- STEP: 14944/36938 -- GLOBAL_STEP: 458200 | > loss_text_ce: 0.03611641004681587 (0.03800301252038149) | > loss_mel_ce: 3.4539291858673096 (nan) | > loss: 0.013849387876689434 (nan) | > current_lr: 5e-06 | > step_time: 0.1434 (0.13631410579290817) | > loader_time: 0.0047 (0.004655930546672745)  --> TIME: 2025-02-24 19:42:33 -- STEP: 14994/36938 -- GLOBAL_STEP: 458250 | > loss_text_ce: 0.04092244431376457 (0.03800120398442436) | > loss_mel_ce: 0.362807035446167 (nan) | > loss: 0.0016021011397242546 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13630931593981563) | > loader_time: 0.0045 (0.004655876636059899)  --> TIME: 2025-02-24 19:42:43 -- STEP: 15044/36938 -- GLOBAL_STEP: 458300 | > loss_text_ce: 0.037058454006910324 (0.03799789523402718) | > loss_mel_ce: 3.178213596343994 (nan) | > loss: 0.012759016826748848 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13630954184705835) | > loader_time: 0.0046 (0.00465589497071002)  --> TIME: 2025-02-24 19:42:52 -- STEP: 15094/36938 -- GLOBAL_STEP: 458350 | > loss_text_ce: 0.04459463059902191 (0.03799335649085224) | > loss_mel_ce: 3.337662935256958 (nan) | > loss: 0.013421657495200634 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.13630499532716492) | > loader_time: 0.0049 (0.004655944064261502)  --> TIME: 2025-02-24 19:43:01 -- STEP: 15144/36938 -- GLOBAL_STEP: 458400 | > loss_text_ce: 0.043372396379709244 (0.03799480341320237) | > loss_mel_ce: 2.9871747493743896 (nan) | > loss: 0.012025981210172176 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.1363099048487796) | > loader_time: 0.0048 (0.004655858857003203)  --> TIME: 2025-02-24 19:43:11 -- STEP: 15194/36938 -- GLOBAL_STEP: 458450 | > loss_text_ce: 0.034093763679265976 (0.037989503045330615) | > loss_mel_ce: 2.2875003814697266 (nan) | > loss: 0.009212676435709 (nan) | > current_lr: 5e-06 | > step_time: 0.181 (0.13634143871148172) | > loader_time: 0.005 (0.004655840115373315)  --> TIME: 2025-02-24 19:43:20 -- STEP: 15244/36938 -- GLOBAL_STEP: 458500 | > loss_text_ce: 0.03950493782758713 (0.03798685225564495) | > loss_mel_ce: 2.988924264907837 (nan) | > loss: 0.012017576955258846 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13633545295549646) | > loader_time: 0.0045 (0.0046557108582051445)  --> TIME: 2025-02-24 19:43:30 -- STEP: 15294/36938 -- GLOBAL_STEP: 458550 | > loss_text_ce: 0.030448703095316887 (0.03798968150185759) | > loss_mel_ce: 2.664240598678589 (nan) | > loss: 0.010693212039768696 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.1363369756326714) | > loader_time: 0.0046 (0.004655786054468284)  --> TIME: 2025-02-24 19:43:39 -- STEP: 15344/36938 -- GLOBAL_STEP: 458600 | > loss_text_ce: 0.05215991288423538 (0.03798916231571574) | > loss_mel_ce: 3.453134298324585 (nan) | > loss: 0.013909898698329926 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.13632808955108508) | > loader_time: 0.0047 (0.004655486351506428)  --> TIME: 2025-02-24 19:43:48 -- STEP: 15394/36938 -- GLOBAL_STEP: 458650 | > loss_text_ce: 0.04644564911723137 (0.03799470952296102) | > loss_mel_ce: 2.9010579586029053 (nan) | > loss: 0.01169644296169281 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13632709761323786) | > loader_time: 0.0046 (0.004655309539344908)  --> TIME: 2025-02-24 19:43:57 -- STEP: 15444/36938 -- GLOBAL_STEP: 458700 | > loss_text_ce: 0.043259840458631516 (0.037997443038461975) | > loss_mel_ce: 1.4377368688583374 (nan) | > loss: 0.005876971408724785 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13630879724992095) | > loader_time: 0.0046 (0.004655118882115779)  --> TIME: 2025-02-24 19:44:06 -- STEP: 15494/36938 -- GLOBAL_STEP: 458750 | > loss_text_ce: 0.03526508808135986 (0.03799522291116099) | > loss_mel_ce: 3.454106569290161 (nan) | > loss: 0.013846714049577713 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.13629923389083207) | > loader_time: 0.0046 (0.0046549456741512224)  --> TIME: 2025-02-24 19:44:15 -- STEP: 15544/36938 -- GLOBAL_STEP: 458800 | > loss_text_ce: 0.03552889823913574 (0.03799704543121441) | > loss_mel_ce: 2.937795877456665 (nan) | > loss: 0.011798908933997154 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13629992822366316) | > loader_time: 0.0046 (0.004654754315585863)  --> TIME: 2025-02-24 19:44:25 -- STEP: 15594/36938 -- GLOBAL_STEP: 458850 | > loss_text_ce: 0.02929861471056938 (0.03800083123135756) | > loss_mel_ce: 2.813950538635254 (nan) | > loss: 0.011282735504209995 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.13630704507011904) | > loader_time: 0.0045 (0.004654523148144667)  --> TIME: 2025-02-24 19:44:34 -- STEP: 15644/36938 -- GLOBAL_STEP: 458900 | > loss_text_ce: 0.040334999561309814 (0.038004857305033675) | > loss_mel_ce: 3.1540229320526123 (nan) | > loss: 0.012676023878157139 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.13631288972055755) | > loader_time: 0.0046 (0.004654355059496674)  --> TIME: 2025-02-24 19:44:44 -- STEP: 15694/36938 -- GLOBAL_STEP: 458950 | > loss_text_ce: 0.03175930678844452 (0.03800449701267224) | > loss_mel_ce: 3.415673017501831 (nan) | > loss: 0.013680287636816502 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.1363161714939578) | > loader_time: 0.0046 (0.004654336890187461)  --> TIME: 2025-02-24 19:44:53 -- STEP: 15744/36938 -- GLOBAL_STEP: 459000 | > loss_text_ce: 0.04363464191555977 (0.03799921070097501) | > loss_mel_ce: 3.3056750297546387 (nan) | > loss: 0.01329091191291809 (nan) | > current_lr: 5e-06 | > step_time: 0.1776 (0.13630643356742872) | > loader_time: 0.005 (0.004654171944754878)  --> TIME: 2025-02-24 19:45:02 -- STEP: 15794/36938 -- GLOBAL_STEP: 459050 | > loss_text_ce: 0.03414413332939148 (0.03800245769796222) | > loss_mel_ce: 2.7156617641448975 (nan) | > loss: 0.010911928489804268 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13629777870465615) | > loader_time: 0.0046 (0.004654070554208916)  --> TIME: 2025-02-24 19:45:11 -- STEP: 15844/36938 -- GLOBAL_STEP: 459100 | > loss_text_ce: 0.03041236288845539 (0.03800162916742728) | > loss_mel_ce: 2.7023284435272217 (nan) | > loss: 0.010844210162758827 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13629792465521967) | > loader_time: 0.0045 (0.004653984099077291)  --> TIME: 2025-02-24 19:45:20 -- STEP: 15894/36938 -- GLOBAL_STEP: 459150 | > loss_text_ce: 0.02772514522075653 (0.037999534642360396) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13629551413555127) | > loader_time: 0.0045 (0.00465392923901122)  --> TIME: 2025-02-24 19:45:30 -- STEP: 15944/36938 -- GLOBAL_STEP: 459200 | > loss_text_ce: 0.04161600396037102 (0.03799929951414079) | > loss_mel_ce: 3.2037253379821777 (nan) | > loss: 0.012878338806331158 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13630905693581016) | > loader_time: 0.0045 (0.004653860292898847)  --> TIME: 2025-02-24 19:45:39 -- STEP: 15994/36938 -- GLOBAL_STEP: 459250 | > loss_text_ce: 0.032090842723846436 (0.03800490376755767) | > loss_mel_ce: 2.4760234355926514 (nan) | > loss: 0.00995283480733633 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13631274313483643) | > loader_time: 0.0047 (0.004653802868484241)  --> TIME: 2025-02-24 19:45:48 -- STEP: 16044/36938 -- GLOBAL_STEP: 459300 | > loss_text_ce: 0.0332835391163826 (0.03800675623116298) | > loss_mel_ce: 3.1273577213287354 (nan) | > loss: 0.012542227283120155 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.136307192068687) | > loader_time: 0.0047 (0.004653816908501352)  --> TIME: 2025-02-24 19:45:58 -- STEP: 16094/36938 -- GLOBAL_STEP: 459350 | > loss_text_ce: 0.036820828914642334 (0.038008975907333604) | > loss_mel_ce: 3.786820888519287 (nan) | > loss: 0.01517318282276392 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13631229731286082) | > loader_time: 0.0045 (0.004653656069382355)  --> TIME: 2025-02-24 19:46:07 -- STEP: 16144/36938 -- GLOBAL_STEP: 459400 | > loss_text_ce: 0.03789205476641655 (0.03800764640005616) | > loss_mel_ce: 3.4142768383026123 (nan) | > loss: 0.013699083589017391 (nan) | > current_lr: 5e-06 | > step_time: 0.1511 (0.13631947195329064) | > loader_time: 0.0048 (0.004653628653411939)  --> TIME: 2025-02-24 19:46:17 -- STEP: 16194/36938 -- GLOBAL_STEP: 459450 | > loss_text_ce: 0.03493141755461693 (0.038011215349015644) | > loss_mel_ce: 3.3172333240509033 (nan) | > loss: 0.013302241452038288 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13632523133104849) | > loader_time: 0.0045 (0.004653816946321052)  --> TIME: 2025-02-24 19:46:26 -- STEP: 16244/36938 -- GLOBAL_STEP: 459500 | > loss_text_ce: 0.05352029204368591 (0.038011545921913886) | > loss_mel_ce: 0.1107177734375 (nan) | > loss: 0.0006517383735626936 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.136332611839808) | > loader_time: 0.0044 (0.0046536382035474045)  --> TIME: 2025-02-24 19:46:36 -- STEP: 16294/36938 -- GLOBAL_STEP: 459550 | > loss_text_ce: 0.03584214299917221 (0.03801054616876052) | > loss_mel_ce: 3.329744815826416 (nan) | > loss: 0.013355504721403122 (nan) | > current_lr: 5e-06 | > step_time: 0.1445 (0.13634735220734326) | > loader_time: 0.0046 (0.0046535824886550035)  --> TIME: 2025-02-24 19:46:46 -- STEP: 16344/36938 -- GLOBAL_STEP: 459600 | > loss_text_ce: 0.044176653027534485 (0.038006252921514355) | > loss_mel_ce: 2.985419273376465 (nan) | > loss: 0.01202220655977726 (nan) | > current_lr: 5e-06 | > step_time: 0.1442 (0.13636936157635454) | > loader_time: 0.0047 (0.004653756197211908)  --> TIME: 2025-02-24 19:46:54 -- STEP: 16394/36938 -- GLOBAL_STEP: 459650 | > loss_text_ce: 0.035432398319244385 (0.03800973329506494) | > loss_mel_ce: 3.1495304107666016 (nan) | > loss: 0.012638742104172707 (nan) | > current_lr: 5e-06 | > step_time: 0.1465 (0.13634433771933613) | > loader_time: 0.0046 (0.004653772333998409)  --> TIME: 2025-02-24 19:47:04 -- STEP: 16444/36938 -- GLOBAL_STEP: 459700 | > loss_text_ce: 0.04157049208879471 (0.03801020261753626) | > loss_mel_ce: 2.9123148918151855 (nan) | > loss: 0.011721767485141754 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.1363404348394753) | > loader_time: 0.0046 (0.004653620418334232)  --> TIME: 2025-02-24 19:47:13 -- STEP: 16494/36938 -- GLOBAL_STEP: 459750 | > loss_text_ce: 0.03203227370977402 (0.0380072929163279) | > loss_mel_ce: 3.294541597366333 (nan) | > loss: 0.013200690969824791 (nan) | > current_lr: 5e-06 | > step_time: 0.187 (0.13635090719472548) | > loader_time: 0.0048 (0.004653678325446312)  --> TIME: 2025-02-24 19:47:22 -- STEP: 16544/36938 -- GLOBAL_STEP: 459800 | > loss_text_ce: 0.04253791272640228 (0.03800663298482953) | > loss_mel_ce: 3.419513463973999 (nan) | > loss: 0.0137382997199893 (nan) | > current_lr: 5e-06 | > step_time: 0.1905 (0.13635373047806976) | > loader_time: 0.0056 (0.004653786984587544)  --> TIME: 2025-02-24 19:47:32 -- STEP: 16594/36938 -- GLOBAL_STEP: 459850 | > loss_text_ce: 0.023620350286364555 (0.03800921294079593) | > loss_mel_ce: 2.7640464305877686 (nan) | > loss: 0.011062170378863811 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13637093232168496) | > loader_time: 0.0046 (0.0046539924310548784)  --> TIME: 2025-02-24 19:47:41 -- STEP: 16644/36938 -- GLOBAL_STEP: 459900 | > loss_text_ce: 0.03944535553455353 (0.038000299276182176) | > loss_mel_ce: 2.9304158687591553 (nan) | > loss: 0.011785164475440979 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1363808154716021) | > loader_time: 0.0045 (0.004653958668648064)  --> TIME: 2025-02-24 19:47:51 -- STEP: 16694/36938 -- GLOBAL_STEP: 459950 | > loss_text_ce: 0.04233693331480026 (0.03799863426032385) | > loss_mel_ce: 2.948781967163086 (nan) | > loss: 0.01186951994895935 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13637656735362724) | > loader_time: 0.0046 (0.004653824736747225)  --> TIME: 2025-02-24 19:48:00 -- STEP: 16744/36938 -- GLOBAL_STEP: 460000 | > loss_text_ce: 0.03271479532122612 (0.03799552907050343) | > loss_mel_ce: 3.1810073852539062 (nan) | > loss: 0.012752866372466087 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13637435326608094) | > loader_time: 0.0046 (0.0046537573463997645) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_460000.pth  --> TIME: 2025-02-24 19:48:16 -- STEP: 16794/36938 -- GLOBAL_STEP: 460050 | > loss_text_ce: 0.036692116409540176 (0.03799589683997247) | > loss_mel_ce: 3.2955949306488037 (nan) | > loss: 0.013223362155258656 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.13637813596735657) | > loader_time: 0.0047 (0.004653709607989186)  --> TIME: 2025-02-24 19:48:25 -- STEP: 16844/36938 -- GLOBAL_STEP: 460100 | > loss_text_ce: 0.03234212473034859 (0.03799869458612831) | > loss_mel_ce: 3.072544574737549 (nan) | > loss: 0.012320979498326778 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13635810758857453) | > loader_time: 0.0045 (0.004653722621063467)  --> TIME: 2025-02-24 19:48:34 -- STEP: 16894/36938 -- GLOBAL_STEP: 460150 | > loss_text_ce: 0.03865136578679085 (0.03799837158237175) | > loss_mel_ce: 3.9191389083862305 (nan) | > loss: 0.015705518424510956 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13635993290331347) | > loader_time: 0.0045 (0.004653714120042269)  --> TIME: 2025-02-24 19:48:43 -- STEP: 16944/36938 -- GLOBAL_STEP: 460200 | > loss_text_ce: 0.029737772420048714 (0.03799595924316792) | > loss_mel_ce: 1.8506461381912231 (nan) | > loss: 0.007461841683834791 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13636123077333565) | > loader_time: 0.0045 (0.0046539164101660965)  --> TIME: 2025-02-24 19:48:53 -- STEP: 16994/36938 -- GLOBAL_STEP: 460250 | > loss_text_ce: 0.032547760754823685 (0.03799360922283543) | > loss_mel_ce: 2.9034714698791504 (nan) | > loss: 0.01165087055414915 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13635989924353717) | > loader_time: 0.0045 (0.0046537956995614405)  --> TIME: 2025-02-24 19:49:02 -- STEP: 17044/36938 -- GLOBAL_STEP: 460300 | > loss_text_ce: 0.039837710559368134 (0.03799271730927264) | > loss_mel_ce: 2.891981840133667 (nan) | > loss: 0.011634204536676407 (nan) | > current_lr: 5e-06 | > step_time: 0.19 (0.13635909876278413) | > loader_time: 0.0049 (0.0046536564771011675)  --> TIME: 2025-02-24 19:49:11 -- STEP: 17094/36938 -- GLOBAL_STEP: 460350 | > loss_text_ce: 0.03236498683691025 (0.037991563247010265) | > loss_mel_ce: 4.1432929039001465 (nan) | > loss: 0.016570070758461952 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13636291649127608) | > loader_time: 0.0046 (0.0046534817219059475)  --> TIME: 2025-02-24 19:49:21 -- STEP: 17144/36938 -- GLOBAL_STEP: 460400 | > loss_text_ce: 0.03170420229434967 (0.03799109403128706) | > loss_mel_ce: 1.2721407413482666 (nan) | > loss: 0.005173988174647093 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13635775407952658) | > loader_time: 0.0045 (0.004653330278685869)  --> TIME: 2025-02-24 19:49:30 -- STEP: 17194/36938 -- GLOBAL_STEP: 460450 | > loss_text_ce: 0.04782407358288765 (0.0379943474163071) | > loss_mel_ce: 2.105929374694824 (nan) | > loss: 0.00854664109647274 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13634690122714171) | > loader_time: 0.0045 (0.004653257201601771)  --> TIME: 2025-02-24 19:49:39 -- STEP: 17244/36938 -- GLOBAL_STEP: 460500 | > loss_text_ce: 0.04940047487616539 (0.037989500528635464) | > loss_mel_ce: 2.502549886703491 (nan) | > loss: 0.010126788169145584 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13634034621414118) | > loader_time: 0.0046 (0.004653408076087886)  --> TIME: 2025-02-24 19:49:48 -- STEP: 17294/36938 -- GLOBAL_STEP: 460550 | > loss_text_ce: 0.04226091876626015 (0.03798552161069461) | > loss_mel_ce: 3.402590751647949 (nan) | > loss: 0.01367004681378603 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13634108285263133) | > loader_time: 0.0046 (0.004653444603935098)  --> TIME: 2025-02-24 19:49:57 -- STEP: 17344/36938 -- GLOBAL_STEP: 460600 | > loss_text_ce: 0.044129762798547745 (0.03798922795273519) | > loss_mel_ce: 2.28940486907959 (nan) | > loss: 0.009260059334337711 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13633393328568133) | > loader_time: 0.0045 (0.004653355897132765)  --> TIME: 2025-02-24 19:50:06 -- STEP: 17394/36938 -- GLOBAL_STEP: 460650 | > loss_text_ce: 0.03502020239830017 (0.03798834203153555) | > loss_mel_ce: 2.8421010971069336 (nan) | > loss: 0.011417148634791374 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13633026534748618) | > loader_time: 0.0045 (0.004653349914563943)  --> TIME: 2025-02-24 19:50:16 -- STEP: 17444/36938 -- GLOBAL_STEP: 460700 | > loss_text_ce: 0.041397176682949066 (0.037991860220274704) | > loss_mel_ce: 2.99174165725708 (nan) | > loss: 0.012036265805363655 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.13633352036980176) | > loader_time: 0.0048 (0.004653398158544249)  --> TIME: 2025-02-24 19:50:25 -- STEP: 17494/36938 -- GLOBAL_STEP: 460750 | > loss_text_ce: 0.03545805439352989 (0.03798830893618517) | > loss_mel_ce: 2.0986249446868896 (nan) | > loss: 0.008468584157526493 (nan) | > current_lr: 5e-06 | > step_time: 0.1262 (0.1363432266050654) | > loader_time: 0.0045 (0.004653498351512616)  --> TIME: 2025-02-24 19:50:34 -- STEP: 17544/36938 -- GLOBAL_STEP: 460800 | > loss_text_ce: 0.03218994289636612 (0.03799038817167486) | > loss_mel_ce: 3.1863551139831543 (nan) | > loss: 0.01277200412005186 (nan) | > current_lr: 5e-06 | > step_time: 0.1515 (0.1363297186509906) | > loader_time: 0.0046 (0.004653440373051508)  --> TIME: 2025-02-24 19:50:43 -- STEP: 17594/36938 -- GLOBAL_STEP: 460850 | > loss_text_ce: 0.04709843173623085 (0.03799039516890155) | > loss_mel_ce: 2.6529009342193604 (nan) | > loss: 0.010714284144341946 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1363187489542644) | > loader_time: 0.0046 (0.004653405219001418)  --> TIME: 2025-02-24 19:50:53 -- STEP: 17644/36938 -- GLOBAL_STEP: 460900 | > loss_text_ce: 0.03720862418413162 (0.03798867519155229) | > loss_mel_ce: 3.1552371978759766 (nan) | > loss: 0.012668436393141747 (nan) | > current_lr: 5e-06 | > step_time: 0.1349 (0.136308070281497) | > loader_time: 0.0047 (0.0046532547303647104)  --> TIME: 2025-02-24 19:51:02 -- STEP: 17694/36938 -- GLOBAL_STEP: 460950 | > loss_text_ce: 0.03987971320748329 (0.03798825895933552) | > loss_mel_ce: 3.26718807220459 (nan) | > loss: 0.013123285956680775 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13631932270840813) | > loader_time: 0.0046 (0.004653282053419141)  --> TIME: 2025-02-24 19:51:12 -- STEP: 17744/36938 -- GLOBAL_STEP: 461000 | > loss_text_ce: 0.04495852068066597 (0.037988193864838135) | > loss_mel_ce: 2.4430184364318848 (nan) | > loss: 0.009872925467789173 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.13632966356303233) | > loader_time: 0.0056 (0.004653488842617619)  --> TIME: 2025-02-24 19:51:21 -- STEP: 17794/36938 -- GLOBAL_STEP: 461050 | > loss_text_ce: 0.04109162837266922 (0.03798699531096186) | > loss_mel_ce: 0.025146484375 (nan) | > loss: 0.0002628496731631458 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.1363326177257249) | > loader_time: 0.0046 (0.004653680762695816)  --> TIME: 2025-02-24 19:51:30 -- STEP: 17844/36938 -- GLOBAL_STEP: 461100 | > loss_text_ce: 0.027833905071020126 (0.0379883528156677) | > loss_mel_ce: 1.6567764282226562 (nan) | > loss: 0.006684962194412947 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13631182959784643) | > loader_time: 0.0045 (0.004653527380719474)  --> TIME: 2025-02-24 19:51:39 -- STEP: 17894/36938 -- GLOBAL_STEP: 461150 | > loss_text_ce: 0.029337555170059204 (0.03798640364703132) | > loss_mel_ce: 2.588418960571289 (nan) | > loss: 0.010387923568487167 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13630924097476607) | > loader_time: 0.0044 (0.004653678215374775)  --> TIME: 2025-02-24 19:51:48 -- STEP: 17944/36938 -- GLOBAL_STEP: 461200 | > loss_text_ce: 0.035317495465278625 (0.03798451788734445) | > loss_mel_ce: 2.8108267784118652 (nan) | > loss: 0.011294223368167877 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13630943315878385) | > loader_time: 0.0046 (0.00465363044851442)  --> TIME: 2025-02-24 19:51:58 -- STEP: 17994/36938 -- GLOBAL_STEP: 461250 | > loss_text_ce: 0.038198452442884445 (0.0379907202312108) | > loss_mel_ce: 3.3568482398986816 (nan) | > loss: 0.013472408056259155 (nan) | > current_lr: 5e-06 | > step_time: 0.1664 (0.13631893693572625) | > loader_time: 0.0047 (0.004653581781123394)  --> TIME: 2025-02-24 19:52:07 -- STEP: 18044/36938 -- GLOBAL_STEP: 461300 | > loss_text_ce: 0.04173688217997551 (0.03799135827131518) | > loss_mel_ce: 1.3994473218917847 (nan) | > loss: 0.005718985106796026 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13632215407861314) | > loader_time: 0.0045 (0.004653447920613851)  --> TIME: 2025-02-24 19:52:17 -- STEP: 18094/36938 -- GLOBAL_STEP: 461350 | > loss_text_ce: 0.03535955771803856 (0.03798991551756961) | > loss_mel_ce: 3.39697265625 (nan) | > loss: 0.013620367273688316 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13632496884083378) | > loader_time: 0.0046 (0.004653323338389388)  --> TIME: 2025-02-24 19:52:26 -- STEP: 18144/36938 -- GLOBAL_STEP: 461400 | > loss_text_ce: 0.019945800304412842 (0.03798838640606078) | > loss_mel_ce: 3.050894260406494 (nan) | > loss: 0.01218587439507246 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13632685709514697) | > loader_time: 0.0046 (0.004653166040016022)  --> TIME: 2025-02-24 19:52:35 -- STEP: 18194/36938 -- GLOBAL_STEP: 461450 | > loss_text_ce: 0.03264234587550163 (0.03798809060556601) | > loss_mel_ce: 3.445319652557373 (nan) | > loss: 0.013801437802612782 (nan) | > current_lr: 5e-06 | > step_time: 0.1824 (0.13632844190774676) | > loader_time: 0.0048 (0.00465304429313305)  --> TIME: 2025-02-24 19:52:45 -- STEP: 18244/36938 -- GLOBAL_STEP: 461500 | > loss_text_ce: 0.03393733128905296 (0.03798666170574016) | > loss_mel_ce: 3.4839823246002197 (nan) | > loss: 0.013959999196231365 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.1363412288295883) | > loader_time: 0.0046 (0.004652933236983382)  --> TIME: 2025-02-24 19:52:54 -- STEP: 18294/36938 -- GLOBAL_STEP: 461550 | > loss_text_ce: 0.045169781893491745 (0.037982715038865335) | > loss_mel_ce: 3.2476274967193604 (nan) | > loss: 0.01306665688753128 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13634343511930244) | > loader_time: 0.0046 (0.004652784524152114)  --> TIME: 2025-02-24 19:53:03 -- STEP: 18344/36938 -- GLOBAL_STEP: 461600 | > loss_text_ce: 0.037507977336645126 (0.03798153128023486) | > loss_mel_ce: 2.2223408222198486 (nan) | > loss: 0.008967654779553413 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13633589965778276) | > loader_time: 0.0045 (0.0046527639024911615)  --> TIME: 2025-02-24 19:53:12 -- STEP: 18394/36938 -- GLOBAL_STEP: 461650 | > loss_text_ce: 0.041695836931467056 (0.03798185143283765) | > loss_mel_ce: 1.346934199333191 (nan) | > loss: 0.005510436836630106 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.1363209884192897) | > loader_time: 0.0045 (0.004652913256781346)  --> TIME: 2025-02-24 19:53:22 -- STEP: 18444/36938 -- GLOBAL_STEP: 461700 | > loss_text_ce: 0.03411359339952469 (0.0379838735961728) | > loss_mel_ce: 2.6461598873138428 (nan) | > loss: 0.010636006481945515 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.1363360242325526) | > loader_time: 0.0046 (0.004652957005191581)  --> TIME: 2025-02-24 19:53:31 -- STEP: 18494/36938 -- GLOBAL_STEP: 461750 | > loss_text_ce: 0.04354136809706688 (0.037984541300682045) | > loss_mel_ce: 3.4032318592071533 (nan) | > loss: 0.013677672483026981 (nan) | > current_lr: 5e-06 | > step_time: 0.1866 (0.13633105676728732) | > loader_time: 0.0048 (0.004653040816413394)  --> TIME: 2025-02-24 19:53:40 -- STEP: 18544/36938 -- GLOBAL_STEP: 461800 | > loss_text_ce: 0.03694567829370499 (0.03798651742259536) | > loss_mel_ce: 3.294916868209839 (nan) | > loss: 0.013221677392721176 (nan) | > current_lr: 5e-06 | > step_time: 0.1956 (0.1363402286976752) | > loader_time: 0.0049 (0.004653145968142612)  --> TIME: 2025-02-24 19:53:50 -- STEP: 18594/36938 -- GLOBAL_STEP: 461850 | > loss_text_ce: 0.04390285536646843 (0.0379900456268635) | > loss_mel_ce: 3.481964588165283 (nan) | > loss: 0.013991538435220718 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13633989583834288) | > loader_time: 0.0046 (0.004653181839440597)  --> TIME: 2025-02-24 19:53:59 -- STEP: 18644/36938 -- GLOBAL_STEP: 461900 | > loss_text_ce: 0.03904185816645622 (0.03799340010440208) | > loss_mel_ce: 3.1126792430877686 (nan) | > loss: 0.012506830506026745 (nan) | > current_lr: 5e-06 | > step_time: 0.175 (0.13634512685854702) | > loader_time: 0.0048 (0.004653195791606653)  --> TIME: 2025-02-24 19:54:09 -- STEP: 18694/36938 -- GLOBAL_STEP: 461950 | > loss_text_ce: 0.03928383067250252 (0.03799279049412319) | > loss_mel_ce: 2.771448850631714 (nan) | > loss: 0.011153701692819595 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13634247842084446) | > loader_time: 0.0045 (0.004653249027206889)  --> TIME: 2025-02-24 19:54:18 -- STEP: 18744/36938 -- GLOBAL_STEP: 462000 | > loss_text_ce: 0.034070439636707306 (0.03799145513132073) | > loss_mel_ce: 2.5459043979644775 (nan) | > loss: 0.010237996466457844 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.13634346194670188) | > loader_time: 0.0055 (0.004653394006024139)  --> TIME: 2025-02-24 19:54:27 -- STEP: 18794/36938 -- GLOBAL_STEP: 462050 | > loss_text_ce: 0.05296493321657181 (0.037994890579977604) | > loss_mel_ce: 3.121774673461914 (nan) | > loss: 0.012598173692822456 (nan) | > current_lr: 5e-06 | > step_time: 0.1355 (0.1363462739210498) | > loader_time: 0.0047 (0.004653525070851307)  --> TIME: 2025-02-24 19:54:37 -- STEP: 18844/36938 -- GLOBAL_STEP: 462100 | > loss_text_ce: 0.05123794451355934 (0.03799733903425677) | > loss_mel_ce: 0.95721435546875 (nan) | > loss: 0.004001794848591089 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13636042596782147) | > loader_time: 0.0045 (0.004653755152359599)  --> TIME: 2025-02-24 19:54:46 -- STEP: 18894/36938 -- GLOBAL_STEP: 462150 | > loss_text_ce: 0.03549150750041008 (0.03799655145969542) | > loss_mel_ce: 2.937049150466919 (nan) | > loss: 0.01179579645395279 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13637710420166918) | > loader_time: 0.0046 (0.004653916770823602)  --> TIME: 2025-02-24 19:54:56 -- STEP: 18944/36938 -- GLOBAL_STEP: 462200 | > loss_text_ce: 0.04241710901260376 (0.03799289019873334) | > loss_mel_ce: 3.333824872970581 (nan) | > loss: 0.01339778583496809 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13637363813767098) | > loader_time: 0.0046 (0.004653872821379345)  --> TIME: 2025-02-24 19:55:05 -- STEP: 18994/36938 -- GLOBAL_STEP: 462250 | > loss_text_ce: 0.03593314811587334 (0.03798871169614383) | > loss_mel_ce: 3.5198428630828857 (nan) | > loss: 0.014110223390161991 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13636985589067813) | > loader_time: 0.0046 (0.00465403280221023)  --> TIME: 2025-02-24 19:55:14 -- STEP: 19044/36938 -- GLOBAL_STEP: 462300 | > loss_text_ce: 0.03641422092914581 (0.037987896033553334) | > loss_mel_ce: 2.0858819484710693 (nan) | > loss: 0.00842181034386158 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13637008339736212) | > loader_time: 0.0045 (0.004653954225486872)  --> TIME: 2025-02-24 19:55:24 -- STEP: 19094/36938 -- GLOBAL_STEP: 462350 | > loss_text_ce: 0.04048248752951622 (0.037983950543809035) | > loss_mel_ce: 0.74395751953125 (nan) | > loss: 0.0031128572300076485 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13637359868772084) | > loader_time: 0.0045 (0.004653834480009136)  --> TIME: 2025-02-24 19:55:33 -- STEP: 19144/36938 -- GLOBAL_STEP: 462400 | > loss_text_ce: 0.033999163657426834 (0.03798355851903255) | > loss_mel_ce: 3.536034107208252 (nan) | > loss: 0.014166799373924732 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.1363631710441246) | > loader_time: 0.0048 (0.004653639378430096)  --> TIME: 2025-02-24 19:55:42 -- STEP: 19194/36938 -- GLOBAL_STEP: 462450 | > loss_text_ce: 0.03419739380478859 (0.03798515358413775) | > loss_mel_ce: 3.205864429473877 (nan) | > loss: 0.012857388705015182 (nan) | > current_lr: 5e-06 | > step_time: 0.181 (0.1363705833516935) | > loader_time: 0.0048 (0.004653581420717173)  --> TIME: 2025-02-24 19:55:52 -- STEP: 19244/36938 -- GLOBAL_STEP: 462500 | > loss_text_ce: 0.03372231870889664 (0.03798585164221694) | > loss_mel_ce: 2.7516934871673584 (nan) | > loss: 0.011053238064050674 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13636822200966175) | > loader_time: 0.0046 (0.00465349349725818)  --> TIME: 2025-02-24 19:56:01 -- STEP: 19294/36938 -- GLOBAL_STEP: 462550 | > loss_text_ce: 0.035035859793424606 (0.03798459584097677) | > loss_mel_ce: 2.7669332027435303 (nan) | > loss: 0.011118925176560879 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13635752488260097) | > loader_time: 0.0045 (0.004653340919754021)  --> TIME: 2025-02-24 19:56:10 -- STEP: 19344/36938 -- GLOBAL_STEP: 462600 | > loss_text_ce: 0.03404423967003822 (0.037987365873310384) | > loss_mel_ce: 1.9137279987335205 (nan) | > loss: 0.007729255594313145 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.1363507510587333) | > loader_time: 0.0046 (0.004653236385807691)  --> TIME: 2025-02-24 19:56:19 -- STEP: 19394/36938 -- GLOBAL_STEP: 462650 | > loss_text_ce: 0.03412887081503868 (0.03798345510306263) | > loss_mel_ce: 2.9896774291992188 (nan) | > loss: 0.01199923176318407 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.13634022380981825) | > loader_time: 0.0048 (0.004653135882193999)  --> TIME: 2025-02-24 19:56:28 -- STEP: 19444/36938 -- GLOBAL_STEP: 462700 | > loss_text_ce: 0.04448534920811653 (0.03798864408259366) | > loss_mel_ce: 2.56716251373291 (nan) | > loss: 0.010363682173192501 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13633966563636143) | > loader_time: 0.0045 (0.004653056850896348)  --> TIME: 2025-02-24 19:56:37 -- STEP: 19494/36938 -- GLOBAL_STEP: 462750 | > loss_text_ce: 0.04314413294196129 (0.03798943151026825) | > loss_mel_ce: 3.7348177433013916 (nan) | > loss: 0.014991912990808487 (nan) | > current_lr: 5e-06 | > step_time: 0.1887 (0.13633936536584781) | > loader_time: 0.0048 (0.004653074649146627)  --> TIME: 2025-02-24 19:56:47 -- STEP: 19544/36938 -- GLOBAL_STEP: 462800 | > loss_text_ce: 0.04883497580885887 (0.03798660598974499) | > loss_mel_ce: 3.02226185798645 (nan) | > loss: 0.01218689326196909 (nan) | > current_lr: 5e-06 | > step_time: 0.182 (0.13635607054620935) | > loader_time: 0.005 (0.004653087464503166)  --> TIME: 2025-02-24 19:56:56 -- STEP: 19594/36938 -- GLOBAL_STEP: 462850 | > loss_text_ce: 0.030400237068533897 (0.03798430623346684) | > loss_mel_ce: 2.4484894275665283 (nan) | > loss: 0.009836864657700062 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.1363619399506346) | > loader_time: 0.0044 (0.004653002055698573)  --> TIME: 2025-02-24 19:57:06 -- STEP: 19644/36938 -- GLOBAL_STEP: 462900 | > loss_text_ce: 0.044986918568611145 (0.037984792793903675) | > loss_mel_ce: 2.484501361846924 (nan) | > loss: 0.010037652216851711 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.13636090837508483) | > loader_time: 0.0047 (0.004653008618682005)  --> TIME: 2025-02-24 19:57:15 -- STEP: 19694/36938 -- GLOBAL_STEP: 462950 | > loss_text_ce: 0.046651653945446014 (0.03798713379965038) | > loss_mel_ce: 2.723968744277954 (nan) | > loss: 0.010994525626301765 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13636009969408103) | > loader_time: 0.0046 (0.00465316917492651)  --> TIME: 2025-02-24 19:57:24 -- STEP: 19744/36938 -- GLOBAL_STEP: 463000 | > loss_text_ce: 0.02461216226220131 (0.037981004138770126) | > loss_mel_ce: 2.506150960922241 (nan) | > loss: 0.010042711161077023 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.1363645599765555) | > loader_time: 0.0046 (0.004653570198452445)  --> TIME: 2025-02-24 19:57:34 -- STEP: 19794/36938 -- GLOBAL_STEP: 463050 | > loss_text_ce: 0.04006873071193695 (0.0379793081448858) | > loss_mel_ce: 3.1734955310821533 (nan) | > loss: 0.012752239592373371 (nan) | > current_lr: 5e-06 | > step_time: 0.1782 (0.13636673468826474) | > loader_time: 0.0048 (0.00465357249416918)  --> TIME: 2025-02-24 19:57:43 -- STEP: 19844/36938 -- GLOBAL_STEP: 463100 | > loss_text_ce: 0.024938272312283516 (0.037974195601174335) | > loss_mel_ce: 2.307783365249634 (nan) | > loss: 0.009256832301616669 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13635190799336683) | > loader_time: 0.0046 (0.004653455136500394)  --> TIME: 2025-02-24 19:57:52 -- STEP: 19894/36938 -- GLOBAL_STEP: 463150 | > loss_text_ce: 0.03361266851425171 (0.037975781168331516) | > loss_mel_ce: 3.399658203125 (nan) | > loss: 0.013624091632664204 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1363578502216488) | > loader_time: 0.0045 (0.004653437084632497)  --> TIME: 2025-02-24 19:58:02 -- STEP: 19944/36938 -- GLOBAL_STEP: 463200 | > loss_text_ce: 0.041014160960912704 (0.03797279811812398) | > loss_mel_ce: 3.4668710231781006 (nan) | > loss: 0.01392018049955368 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.1363598454969257) | > loader_time: 0.0045 (0.004653340224228939)  --> TIME: 2025-02-24 19:58:11 -- STEP: 19994/36938 -- GLOBAL_STEP: 463250 | > loss_text_ce: 0.034410327672958374 (0.03797643457338037) | > loss_mel_ce: 2.261678457260132 (nan) | > loss: 0.009111464023590088 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.1363655275686552) | > loader_time: 0.0044 (0.0046532148240243775)  --> TIME: 2025-02-24 19:58:20 -- STEP: 20044/36938 -- GLOBAL_STEP: 463300 | > loss_text_ce: 0.05452927574515343 (0.037980161885584035) | > loss_mel_ce: 3.1208858489990234 (nan) | > loss: 0.0126008540391922 (nan) | > current_lr: 5e-06 | > step_time: 0.186 (0.13636976129588513) | > loader_time: 0.0049 (0.004653193367334014)  --> TIME: 2025-02-24 19:58:30 -- STEP: 20094/36938 -- GLOBAL_STEP: 463350 | > loss_text_ce: 0.04037003591656685 (0.037980378347728626) | > loss_mel_ce: 3.33782958984375 (nan) | > loss: 0.013405554927885532 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13637012182165248) | > loader_time: 0.0046 (0.004653157933477116)  --> TIME: 2025-02-24 19:58:39 -- STEP: 20144/36938 -- GLOBAL_STEP: 463400 | > loss_text_ce: 0.035325899720191956 (0.03798094277802354) | > loss_mel_ce: 3.11995267868042 (nan) | > loss: 0.012520947493612766 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13637513737767493) | > loader_time: 0.0047 (0.004653115929167257)  --> TIME: 2025-02-24 19:58:49 -- STEP: 20194/36938 -- GLOBAL_STEP: 463450 | > loss_text_ce: 0.03702890872955322 (0.03797668832038496) | > loss_mel_ce: 3.366096019744873 (nan) | > loss: 0.01350446417927742 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.1363880373976587) | > loader_time: 0.0046 (0.004653067875465703)  --> TIME: 2025-02-24 19:58:58 -- STEP: 20244/36938 -- GLOBAL_STEP: 463500 | > loss_text_ce: 0.038000449538230896 (0.03798217107815108) | > loss_mel_ce: 3.4686176776885986 (nan) | > loss: 0.013915151357650757 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13638636928564718) | > loader_time: 0.0047 (0.004652966131124767)  --> TIME: 2025-02-24 19:59:07 -- STEP: 20294/36938 -- GLOBAL_STEP: 463550 | > loss_text_ce: 0.040631793439388275 (0.037987161666463806) | > loss_mel_ce: 1.7956851720809937 (nan) | > loss: 0.007286972366273403 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13638642259074998) | > loader_time: 0.0046 (0.004652902705687955)  --> TIME: 2025-02-24 19:59:17 -- STEP: 20344/36938 -- GLOBAL_STEP: 463600 | > loss_text_ce: 0.02815891243517399 (0.03799105575418056) | > loss_mel_ce: 2.789716958999634 (nan) | > loss: 0.011182047426700592 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.1363892568714072) | > loader_time: 0.0048 (0.004652825212366126)  --> TIME: 2025-02-24 19:59:26 -- STEP: 20394/36938 -- GLOBAL_STEP: 463650 | > loss_text_ce: 0.041196998208761215 (0.03799129095322315) | > loss_mel_ce: 2.445413827896118 (nan) | > loss: 0.009867504239082336 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13640397179766187) | > loader_time: 0.0045 (0.0046528242400572335)  --> TIME: 2025-02-24 19:59:36 -- STEP: 20444/36938 -- GLOBAL_STEP: 463700 | > loss_text_ce: 0.035007327795028687 (0.0379879931935037) | > loss_mel_ce: 3.408198118209839 (nan) | > loss: 0.013663514517247677 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13640464707331437) | > loader_time: 0.0046 (0.00465272995292545)  --> TIME: 2025-02-24 19:59:45 -- STEP: 20494/36938 -- GLOBAL_STEP: 463750 | > loss_text_ce: 0.02935325726866722 (0.037985346485034585) | > loss_mel_ce: 2.789461612701416 (nan) | > loss: 0.011185773648321629 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13640773532517822) | > loader_time: 0.0047 (0.0046526038310487515)  --> TIME: 2025-02-24 19:59:55 -- STEP: 20544/36938 -- GLOBAL_STEP: 463800 | > loss_text_ce: 0.029795799404382706 (0.037985270861360186) | > loss_mel_ce: 2.559669256210327 (nan) | > loss: 0.01027565635740757 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13642127785216557) | > loader_time: 0.0045 (0.004652774187727501)  --> TIME: 2025-02-24 20:00:04 -- STEP: 20594/36938 -- GLOBAL_STEP: 463850 | > loss_text_ce: 0.038564831018447876 (0.037988257667712924) | > loss_mel_ce: 3.682626962661743 (nan) | > loss: 0.014766635373234749 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13642643975716) | > loader_time: 0.0045 (0.004652770662858803)  --> TIME: 2025-02-24 20:00:14 -- STEP: 20644/36938 -- GLOBAL_STEP: 463900 | > loss_text_ce: 0.03019465133547783 (0.03798718713930392) | > loss_mel_ce: 3.2150566577911377 (nan) | > loss: 0.01287798210978508 (nan) | > current_lr: 5e-06 | > step_time: 0.1574 (0.13645335570529205) | > loader_time: 0.0048 (0.004652816700487255)  --> TIME: 2025-02-24 20:00:24 -- STEP: 20694/36938 -- GLOBAL_STEP: 463950 | > loss_text_ce: 0.049009259790182114 (0.037987549102616446) | > loss_mel_ce: 3.39131760597229 (nan) | > loss: 0.013652091845870018 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.13646662209913465) | > loader_time: 0.0046 (0.00465275365234607)  --> TIME: 2025-02-24 20:00:33 -- STEP: 20744/36938 -- GLOBAL_STEP: 464000 | > loss_text_ce: 0.038355812430381775 (0.03799061480322503) | > loss_mel_ce: 3.055933713912964 (nan) | > loss: 0.012278927490115166 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13646677206530755) | > loader_time: 0.0046 (0.004652650382497615)  --> TIME: 2025-02-24 20:00:42 -- STEP: 20794/36938 -- GLOBAL_STEP: 464050 | > loss_text_ce: 0.038335707038640976 (0.03799274762679609) | > loss_mel_ce: 2.6742780208587646 (nan) | > loss: 0.010764340870082378 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.13646377049077596) | > loader_time: 0.0048 (0.00465255020053911)  --> TIME: 2025-02-24 20:00:52 -- STEP: 20844/36938 -- GLOBAL_STEP: 464100 | > loss_text_ce: 0.042638543993234634 (0.03798778020450439) | > loss_mel_ce: 3.2786309719085693 (nan) | > loss: 0.013179641216993332 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13646795893328512) | > loader_time: 0.0046 (0.004652446289937292)  --> TIME: 2025-02-24 20:01:01 -- STEP: 20894/36938 -- GLOBAL_STEP: 464150 | > loss_text_ce: 0.040474288165569305 (0.037988901229191185) | > loss_mel_ce: 1.03448486328125 (nan) | > loss: 0.004265711177140474 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.1364658797107905) | > loader_time: 0.0047 (0.004652303338016605)  --> TIME: 2025-02-24 20:01:10 -- STEP: 20944/36938 -- GLOBAL_STEP: 464200 | > loss_text_ce: 0.03253859654068947 (0.03798421183029149) | > loss_mel_ce: 0.265625 (nan) | > loss: 0.0011831889860332012 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13647154030497496) | > loader_time: 0.0044 (0.004652251955477487)  --> TIME: 2025-02-24 20:01:19 -- STEP: 20994/36938 -- GLOBAL_STEP: 464250 | > loss_text_ce: 0.042681798338890076 (0.03798511922378076) | > loss_mel_ce: 3.2977325916290283 (nan) | > loss: 0.013255612924695015 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13646551074329363) | > loader_time: 0.0046 (0.004652079439304023)  --> TIME: 2025-02-24 20:01:29 -- STEP: 21044/36938 -- GLOBAL_STEP: 464300 | > loss_text_ce: 0.04346802085638046 (0.037985673941970076) | > loss_mel_ce: 3.284703493118286 (nan) | > loss: 0.013207030482590199 (nan) | > current_lr: 5e-06 | > step_time: 0.1923 (0.1364684842584976) | > loader_time: 0.0048 (0.004651998639718537)  --> TIME: 2025-02-24 20:01:38 -- STEP: 21094/36938 -- GLOBAL_STEP: 464350 | > loss_text_ce: 0.04329036921262741 (0.037985390809345765) | > loss_mel_ce: 1.2929575443267822 (nan) | > loss: 0.005302571225911379 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13646941777483673) | > loader_time: 0.0045 (0.004651923761488046)  --> TIME: 2025-02-24 20:01:48 -- STEP: 21144/36938 -- GLOBAL_STEP: 464400 | > loss_text_ce: 0.04223848134279251 (0.03798473389470215) | > loss_mel_ce: 3.228255033493042 (nan) | > loss: 0.012978149577975273 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.1364829454668266) | > loader_time: 0.0046 (0.004651916396920073)  --> TIME: 2025-02-24 20:01:57 -- STEP: 21194/36938 -- GLOBAL_STEP: 464450 | > loss_text_ce: 0.04251600056886673 (0.037986025573251196) | > loss_mel_ce: 2.6343328952789307 (nan) | > loss: 0.010622416622936726 (nan) | > current_lr: 5e-06 | > step_time: 0.1517 (0.13648516458869725) | > loader_time: 0.0046 (0.004651860706174185)  --> TIME: 2025-02-24 20:02:06 -- STEP: 21244/36938 -- GLOBAL_STEP: 464500 | > loss_text_ce: 0.026316210627555847 (0.037984844657135086) | > loss_mel_ce: 3.1866931915283203 (nan) | > loss: 0.01275003794580698 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13648835076515448) | > loader_time: 0.0046 (0.004651775750214259)  --> TIME: 2025-02-24 20:02:16 -- STEP: 21294/36938 -- GLOBAL_STEP: 464550 | > loss_text_ce: 0.044338587671518326 (0.037982023107089785) | > loss_mel_ce: 3.2904067039489746 (nan) | > loss: 0.013233117759227753 (nan) | > current_lr: 5e-06 | > step_time: 0.1977 (0.13648158229549495) | > loader_time: 0.0048 (0.004651650482694377)  --> TIME: 2025-02-24 20:02:25 -- STEP: 21344/36938 -- GLOBAL_STEP: 464600 | > loss_text_ce: 0.035567767918109894 (0.037983208349348936) | > loss_mel_ce: 0.9829804301261902 (nan) | > loss: 0.004041858483105898 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.1364764813249974) | > loader_time: 0.0046 (0.00465152997975942)  --> TIME: 2025-02-24 20:02:34 -- STEP: 21394/36938 -- GLOBAL_STEP: 464650 | > loss_text_ce: 0.03465503454208374 (0.03798433198020917) | > loss_mel_ce: 2.84409761428833 (nan) | > loss: 0.011423622258007526 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.1364780964333845) | > loader_time: 0.0047 (0.0046514468604497245)  --> TIME: 2025-02-24 20:02:43 -- STEP: 21444/36938 -- GLOBAL_STEP: 464700 | > loss_text_ce: 0.03460375592112541 (0.03798465669868924) | > loss_mel_ce: 3.1283719539642334 (nan) | > loss: 0.01255149208009243 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13647921317505593) | > loader_time: 0.0047 (0.004651379549687881)  --> TIME: 2025-02-24 20:02:53 -- STEP: 21494/36938 -- GLOBAL_STEP: 464750 | > loss_text_ce: 0.03337770700454712 (0.03798361309318974) | > loss_mel_ce: 3.650275468826294 (nan) | > loss: 0.014617672190070152 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.1364799984290988) | > loader_time: 0.0047 (0.004651355479410703)  --> TIME: 2025-02-24 20:03:02 -- STEP: 21544/36938 -- GLOBAL_STEP: 464800 | > loss_text_ce: 0.037350572645664215 (0.037981982607329594) | > loss_mel_ce: 2.307790517807007 (nan) | > loss: 0.00930611602962017 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.1364820205032096) | > loader_time: 0.0046 (0.0046513266736939556)  --> TIME: 2025-02-24 20:03:11 -- STEP: 21594/36938 -- GLOBAL_STEP: 464850 | > loss_text_ce: 0.043655265122652054 (0.037985822419071016) | > loss_mel_ce: 3.4994304180145264 (nan) | > loss: 0.014059863984584808 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13648405525102955) | > loader_time: 0.0046 (0.004651411016682749)  --> TIME: 2025-02-24 20:03:20 -- STEP: 21644/36938 -- GLOBAL_STEP: 464900 | > loss_text_ce: 0.04343080520629883 (0.03798778583137279) | > loss_mel_ce: 2.9743711948394775 (nan) | > loss: 0.011975405737757683 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13647446971000454) | > loader_time: 0.0046 (0.0046513100094284915)  --> TIME: 2025-02-24 20:03:30 -- STEP: 21694/36938 -- GLOBAL_STEP: 464950 | > loss_text_ce: 0.024158786982297897 (0.03799003292821456) | > loss_mel_ce: 3.383012533187866 (nan) | > loss: 0.01352052204310894 (nan) | > current_lr: 5e-06 | > step_time: 0.1723 (0.1364862764333377) | > loader_time: 0.0047 (0.004651208555598374)  --> TIME: 2025-02-24 20:03:39 -- STEP: 21744/36938 -- GLOBAL_STEP: 465000 | > loss_text_ce: 0.024378355592489243 (0.037989842022569935) | > loss_mel_ce: 2.8165338039398193 (nan) | > loss: 0.011273461394011974 (nan) | > current_lr: 5e-06 | > step_time: 0.1264 (0.1364920744131033) | > loader_time: 0.0052 (0.004651111954270775)  --> TIME: 2025-02-24 20:03:49 -- STEP: 21794/36938 -- GLOBAL_STEP: 465050 | > loss_text_ce: 0.05157921090722084 (0.03799059466507815) | > loss_mel_ce: 3.4019014835357666 (nan) | > loss: 0.013704289682209492 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13650221768635049) | > loader_time: 0.0045 (0.004651165800575062)  --> TIME: 2025-02-24 20:03:58 -- STEP: 21844/36938 -- GLOBAL_STEP: 465100 | > loss_text_ce: 0.04711178317666054 (0.037990391816484295) | > loss_mel_ce: 2.6339995861053467 (nan) | > loss: 0.010639331303536892 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13650718282767016) | > loader_time: 0.0045 (0.004651118767739548)  --> TIME: 2025-02-24 20:04:08 -- STEP: 21894/36938 -- GLOBAL_STEP: 465150 | > loss_text_ce: 0.03497118130326271 (0.03798993603901734) | > loss_mel_ce: 3.1031928062438965 (nan) | > loss: 0.012453032657504082 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13650468989595554) | > loader_time: 0.0046 (0.004651003693231878)  --> TIME: 2025-02-24 20:04:17 -- STEP: 21944/36938 -- GLOBAL_STEP: 465200 | > loss_text_ce: 0.04488275945186615 (0.037989376474287545) | > loss_mel_ce: 2.7750649452209473 (nan) | > loss: 0.011190269142389297 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13650345982663897) | > loader_time: 0.0047 (0.004651001518414986)  --> TIME: 2025-02-24 20:04:27 -- STEP: 21994/36938 -- GLOBAL_STEP: 465250 | > loss_text_ce: 0.032557614147663116 (0.03798887475135489) | > loss_mel_ce: 2.8921496868133545 (nan) | > loss: 0.011605982668697834 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13651095536271818) | > loader_time: 0.0046 (0.004650919277190796)  --> TIME: 2025-02-24 20:04:36 -- STEP: 22044/36938 -- GLOBAL_STEP: 465300 | > loss_text_ce: 0.026151012629270554 (0.037987022199109384) | > loss_mel_ce: 2.462717056274414 (nan) | > loss: 0.009876460768282413 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.1365067984789174) | > loader_time: 0.0046 (0.004650925642521259)  --> TIME: 2025-02-24 20:04:45 -- STEP: 22094/36938 -- GLOBAL_STEP: 465350 | > loss_text_ce: 0.044012658298015594 (0.037987540526565236) | > loss_mel_ce: 3.1494362354278564 (nan) | > loss: 0.012672416865825653 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13650114417367498) | > loader_time: 0.0046 (0.004650977312455581)  --> TIME: 2025-02-24 20:04:54 -- STEP: 22144/36938 -- GLOBAL_STEP: 465400 | > loss_text_ce: 0.04118453338742256 (0.03798569572859504) | > loss_mel_ce: 2.8227763175964355 (nan) | > loss: 0.011364923790097237 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.1365087033502459) | > loader_time: 0.0047 (0.004650960251085999)  --> TIME: 2025-02-24 20:05:04 -- STEP: 22194/36938 -- GLOBAL_STEP: 465450 | > loss_text_ce: 0.04686766490340233 (0.03798453477584341) | > loss_mel_ce: 3.01918888092041 (nan) | > loss: 0.012166891247034073 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13651523916014743) | > loader_time: 0.0045 (0.004650908955112257)  --> TIME: 2025-02-24 20:05:13 -- STEP: 22244/36938 -- GLOBAL_STEP: 465500 | > loss_text_ce: 0.038403019309043884 (0.03798350914698969) | > loss_mel_ce: 3.059048891067505 (nan) | > loss: 0.01229147706180811 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.13650441952177048) | > loader_time: 0.0046 (0.0046509172693042075)  --> TIME: 2025-02-24 20:05:23 -- STEP: 22294/36938 -- GLOBAL_STEP: 465550 | > loss_text_ce: 0.031564436852931976 (0.03798523629110961) | > loss_mel_ce: 1.4595075845718384 (nan) | > loss: 0.005916953086853027 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13652012868345512) | > loader_time: 0.0047 (0.004651271506597207)  --> TIME: 2025-02-24 20:05:32 -- STEP: 22344/36938 -- GLOBAL_STEP: 465600 | > loss_text_ce: 0.03865700960159302 (0.037984335202964764) | > loss_mel_ce: 2.881842613220215 (nan) | > loss: 0.011589284986257553 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.1365246085381365) | > loader_time: 0.0048 (0.004651277830555475)  --> TIME: 2025-02-24 20:05:42 -- STEP: 22394/36938 -- GLOBAL_STEP: 465650 | > loss_text_ce: 0.04243619367480278 (0.037984293132244794) | > loss_mel_ce: 3.454219102859497 (nan) | > loss: 0.01387561671435833 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1365286525556146) | > loader_time: 0.0045 (0.004651270945859782)  --> TIME: 2025-02-24 20:05:51 -- STEP: 22444/36938 -- GLOBAL_STEP: 465700 | > loss_text_ce: 0.03892816603183746 (0.03798471424011418) | > loss_mel_ce: 3.4977834224700928 (nan) | > loss: 0.014034571126103401 (nan) | > current_lr: 5e-06 | > step_time: 0.1599 (0.13652528264989544) | > loader_time: 0.0046 (0.004651270316811786)  --> TIME: 2025-02-24 20:06:00 -- STEP: 22494/36938 -- GLOBAL_STEP: 465750 | > loss_text_ce: 0.03771368786692619 (0.03798715657640431) | > loss_mel_ce: 2.309022903442383 (nan) | > loss: 0.009312448091804981 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13651386200401272) | > loader_time: 0.0045 (0.004651238910461003)  --> TIME: 2025-02-24 20:06:09 -- STEP: 22544/36938 -- GLOBAL_STEP: 465800 | > loss_text_ce: 0.043224360793828964 (0.03798788995866217) | > loss_mel_ce: 3.2221078872680664 (nan) | > loss: 0.01295766793191433 (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.13650962741050093) | > loader_time: 0.0047 (0.0046513066108214655)  --> TIME: 2025-02-24 20:06:18 -- STEP: 22594/36938 -- GLOBAL_STEP: 465850 | > loss_text_ce: 0.03532663732767105 (0.03798824280161246) | > loss_mel_ce: 2.72359299659729 (nan) | > loss: 0.01094809453934431 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13650425609322894) | > loader_time: 0.0046 (0.004651205195909534)  --> TIME: 2025-02-24 20:06:28 -- STEP: 22644/36938 -- GLOBAL_STEP: 465900 | > loss_text_ce: 0.04845103248953819 (0.037988293094601874) | > loss_mel_ce: 2.2259860038757324 (nan) | > loss: 0.009025543928146362 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13651324545109458) | > loader_time: 0.0046 (0.004651347417078362)  --> TIME: 2025-02-24 20:06:37 -- STEP: 22694/36938 -- GLOBAL_STEP: 465950 | > loss_text_ce: 0.03698374703526497 (0.03798796737949386) | > loss_mel_ce: 3.1945478916168213 (nan) | > loss: 0.012823538854718208 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13649984669704285) | > loader_time: 0.0047 (0.004651293372586799)  --> TIME: 2025-02-24 20:06:46 -- STEP: 22744/36938 -- GLOBAL_STEP: 466000 | > loss_text_ce: 0.03364848718047142 (0.03799060804059675) | > loss_mel_ce: 2.5435962677001953 (nan) | > loss: 0.010227162390947342 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13649431955969507) | > loader_time: 0.0052 (0.004651375998085392)  --> TIME: 2025-02-24 20:06:55 -- STEP: 22794/36938 -- GLOBAL_STEP: 466050 | > loss_text_ce: 0.039439111948013306 (0.037989092010852904) | > loss_mel_ce: 2.7796289920806885 (nan) | > loss: 0.011186779476702213 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.1364950035144919) | > loader_time: 0.0045 (0.004651402876943371)  --> TIME: 2025-02-24 20:07:05 -- STEP: 22844/36938 -- GLOBAL_STEP: 466100 | > loss_text_ce: 0.03059154748916626 (0.03798706097238217) | > loss_mel_ce: 3.275318145751953 (nan) | > loss: 0.013118689879775047 (nan) | > current_lr: 5e-06 | > step_time: 0.1782 (0.13649583173871055) | > loader_time: 0.0048 (0.00465128224945133)  --> TIME: 2025-02-24 20:07:14 -- STEP: 22894/36938 -- GLOBAL_STEP: 466150 | > loss_text_ce: 0.04306497424840927 (0.03798698072589457) | > loss_mel_ce: 3.0777173042297363 (nan) | > loss: 0.012384057976305485 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13650197390022048) | > loader_time: 0.0045 (0.00465119470308218)  --> TIME: 2025-02-24 20:07:23 -- STEP: 22944/36938 -- GLOBAL_STEP: 466200 | > loss_text_ce: 0.044495414942502975 (0.03798711600006646) | > loss_mel_ce: 3.422180652618408 (nan) | > loss: 0.013756651431322098 (nan) | > current_lr: 5e-06 | > step_time: 0.186 (0.1364975576412394) | > loader_time: 0.0071 (0.004651191708017403)  --> TIME: 2025-02-24 20:07:33 -- STEP: 22994/36938 -- GLOBAL_STEP: 466250 | > loss_text_ce: 0.037616364657878876 (0.03798877882248773) | > loss_mel_ce: 3.324427366256714 (nan) | > loss: 0.013341443613171577 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13649892294791555) | > loader_time: 0.0046 (0.004651206985311685)  --> TIME: 2025-02-24 20:07:42 -- STEP: 23044/36938 -- GLOBAL_STEP: 466300 | > loss_text_ce: 0.033319029957056046 (0.037989998563457604) | > loss_mel_ce: 3.0963964462280273 (nan) | > loss: 0.012419506907463074 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13648797953176384) | > loader_time: 0.0046 (0.004651204628407188)  --> TIME: 2025-02-24 20:07:51 -- STEP: 23094/36938 -- GLOBAL_STEP: 466350 | > loss_text_ce: 0.03587993234395981 (0.03799232494927569) | > loss_mel_ce: 2.6387312412261963 (nan) | > loss: 0.010613536462187767 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.1364885023417198) | > loader_time: 0.0046 (0.004651356767532317)  --> TIME: 2025-02-24 20:08:00 -- STEP: 23144/36938 -- GLOBAL_STEP: 466400 | > loss_text_ce: 0.04156121611595154 (0.03799199829542964) | > loss_mel_ce: 0.02392578125 (nan) | > loss: 0.0002598690625745803 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13648128795903963) | > loader_time: 0.0106 (0.004651622585965121)  --> TIME: 2025-02-24 20:08:09 -- STEP: 23194/36938 -- GLOBAL_STEP: 466450 | > loss_text_ce: 0.03370489552617073 (0.037989021715186484) | > loss_mel_ce: 2.79778790473938 (nan) | > loss: 0.011236083693802357 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13648342040835687) | > loader_time: 0.0045 (0.004651769611499722)  --> TIME: 2025-02-24 20:08:19 -- STEP: 23244/36938 -- GLOBAL_STEP: 466500 | > loss_text_ce: 0.0389377661049366 (0.03798786614246777) | > loss_mel_ce: 2.031074285507202 (nan) | > loss: 0.008214334025979042 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13648243733885118) | > loader_time: 0.0046 (0.004651744021870439)  --> TIME: 2025-02-24 20:08:28 -- STEP: 23294/36938 -- GLOBAL_STEP: 466550 | > loss_text_ce: 0.037010688334703445 (0.03799133181006292) | > loss_mel_ce: 2.848745107650757 (nan) | > loss: 0.011451412923634052 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1364825251297531) | > loader_time: 0.0045 (0.00465167075398139)  --> TIME: 2025-02-24 20:08:37 -- STEP: 23344/36938 -- GLOBAL_STEP: 466600 | > loss_text_ce: 0.040096625685691833 (0.0379897195010348) | > loss_mel_ce: 1.857668399810791 (nan) | > loss: 0.007530814036726952 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13648947471825537) | > loader_time: 0.0045 (0.00465163750914856)  --> TIME: 2025-02-24 20:08:46 -- STEP: 23394/36938 -- GLOBAL_STEP: 466650 | > loss_text_ce: 0.03139672055840492 (0.037987598155720545) | > loss_mel_ce: 3.646599292755127 (nan) | > loss: 0.014595222659409046 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.1364743873442587) | > loader_time: 0.0046 (0.004651635663574751)  --> TIME: 2025-02-24 20:08:55 -- STEP: 23444/36938 -- GLOBAL_STEP: 466700 | > loss_text_ce: 0.035055067390203476 (0.0379842503050431) | > loss_mel_ce: 3.153064489364624 (nan) | > loss: 0.01265126932412386 (nan) | > current_lr: 5e-06 | > step_time: 0.1353 (0.13646707006372139) | > loader_time: 0.0046 (0.004651814887312481)  --> TIME: 2025-02-24 20:09:04 -- STEP: 23494/36938 -- GLOBAL_STEP: 466750 | > loss_text_ce: 0.026998771354556084 (0.03798382031434235) | > loss_mel_ce: 1.5578124523162842 (nan) | > loss: 0.00628893356770277 (nan) | > current_lr: 5e-06 | > step_time: 0.107 (0.13645834085372413) | > loader_time: 0.0045 (0.00465185219880068)  --> TIME: 2025-02-24 20:09:13 -- STEP: 23544/36938 -- GLOBAL_STEP: 466800 | > loss_text_ce: 0.03488786518573761 (0.037984791026065216) | > loss_mel_ce: 3.3155014514923096 (nan) | > loss: 0.013295195996761322 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13644343819133764) | > loader_time: 0.0047 (0.004651816835866783)  --> TIME: 2025-02-24 20:09:23 -- STEP: 23594/36938 -- GLOBAL_STEP: 466850 | > loss_text_ce: 0.041323915123939514 (0.03798334669111617) | > loss_mel_ce: 2.2159929275512695 (nan) | > loss: 0.008957606740295887 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13643986028645544) | > loader_time: 0.0045 (0.00465199818416719)  --> TIME: 2025-02-24 20:09:32 -- STEP: 23644/36938 -- GLOBAL_STEP: 466900 | > loss_text_ce: 0.04346127435564995 (0.03798064077363602) | > loss_mel_ce: 0.7407068014144897 (nan) | > loss: 0.0031117782928049564 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13644287780222186) | > loader_time: 0.0046 (0.0046521704867695975)  --> TIME: 2025-02-24 20:09:41 -- STEP: 23694/36938 -- GLOBAL_STEP: 466950 | > loss_text_ce: 0.027409369125962257 (0.03798045075745438) | > loss_mel_ce: 0.1611480712890625 (nan) | > loss: 0.0007482438813894987 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13644022057909103) | > loader_time: 0.0047 (0.0046522813355699836)  --> TIME: 2025-02-24 20:09:51 -- STEP: 23744/36938 -- GLOBAL_STEP: 467000 | > loss_text_ce: 0.04137935861945152 (0.0379793516964971) | > loss_mel_ce: 2.6520304679870605 (nan) | > loss: 0.010688135400414467 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.1364507055527597) | > loader_time: 0.0046 (0.004652479327110538)  --> TIME: 2025-02-24 20:10:00 -- STEP: 23794/36938 -- GLOBAL_STEP: 467050 | > loss_text_ce: 0.02681504376232624 (0.03797624527563699) | > loss_mel_ce: 1.1911840438842773 (nan) | > loss: 0.004833329934626818 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1364500308984707) | > loader_time: 0.0045 (0.0046524090697567)  --> TIME: 2025-02-24 20:10:09 -- STEP: 23844/36938 -- GLOBAL_STEP: 467100 | > loss_text_ce: 0.035162582993507385 (0.03797510369353347) | > loss_mel_ce: 3.289055824279785 (nan) | > loss: 0.01319134421646595 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.13644433080540122) | > loader_time: 0.0048 (0.004652283522051171)  --> TIME: 2025-02-24 20:10:19 -- STEP: 23894/36938 -- GLOBAL_STEP: 467150 | > loss_text_ce: 0.034644387662410736 (0.037974832721175814) | > loss_mel_ce: 2.2210967540740967 (nan) | > loss: 0.008951353840529919 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13645711629233268) | > loader_time: 0.0044 (0.004652335133484245)  --> TIME: 2025-02-24 20:10:28 -- STEP: 23944/36938 -- GLOBAL_STEP: 467200 | > loss_text_ce: 0.03840843215584755 (0.03797077462885393) | > loss_mel_ce: 3.146888256072998 (nan) | > loss: 0.012640067376196384 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13645480201270638) | > loader_time: 0.0047 (0.004652273055505748)  --> TIME: 2025-02-24 20:10:37 -- STEP: 23994/36938 -- GLOBAL_STEP: 467250 | > loss_text_ce: 0.031483471393585205 (0.03797311280029072) | > loss_mel_ce: 2.5867364406585693 (nan) | > loss: 0.010389761999249458 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13645378570710848) | > loader_time: 0.0047 (0.0046524137638603745)  --> TIME: 2025-02-24 20:10:46 -- STEP: 24044/36938 -- GLOBAL_STEP: 467300 | > loss_text_ce: 0.03677111864089966 (0.037975458288317304) | > loss_mel_ce: 2.484426259994507 (nan) | > loss: 0.01000475138425827 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13644420847760283) | > loader_time: 0.0045 (0.004652521412338029)  --> TIME: 2025-02-24 20:10:56 -- STEP: 24094/36938 -- GLOBAL_STEP: 467350 | > loss_text_ce: 0.031827475875616074 (0.037975694278187457) | > loss_mel_ce: 2.524672031402588 (nan) | > loss: 0.010144839994609356 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13645654597638232) | > loader_time: 0.0044 (0.004652676418469367)  --> TIME: 2025-02-24 20:11:05 -- STEP: 24144/36938 -- GLOBAL_STEP: 467400 | > loss_text_ce: 0.02923097461462021 (0.037978438572978716) | > loss_mel_ce: 2.749253749847412 (nan) | > loss: 0.01102573424577713 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13645557603984557) | > loader_time: 0.0046 (0.004652793228507704)  --> TIME: 2025-02-24 20:11:15 -- STEP: 24194/36938 -- GLOBAL_STEP: 467450 | > loss_text_ce: 0.03996020182967186 (0.03797809251133654) | > loss_mel_ce: 3.4512155055999756 (nan) | > loss: 0.013853872194886208 (nan) | > current_lr: 5e-06 | > step_time: 0.1889 (0.13646331881554405) | > loader_time: 0.0048 (0.0046527330920923754)  --> TIME: 2025-02-24 20:11:24 -- STEP: 24244/36938 -- GLOBAL_STEP: 467500 | > loss_text_ce: 0.03360188752412796 (0.03797986289945571) | > loss_mel_ce: 2.657400369644165 (nan) | > loss: 0.010678581893444061 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13646203415206176) | > loader_time: 0.0046 (0.004652656711892381)  --> TIME: 2025-02-24 20:11:33 -- STEP: 24294/36938 -- GLOBAL_STEP: 467550 | > loss_text_ce: 0.043184567242860794 (0.03797664865686556) | > loss_mel_ce: 1.0696760416030884 (nan) | > loss: 0.004416113719344139 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13646624164246823) | > loader_time: 0.0044 (0.004652528740441673)  --> TIME: 2025-02-24 20:11:43 -- STEP: 24344/36938 -- GLOBAL_STEP: 467600 | > loss_text_ce: 0.04120625555515289 (0.03797787507686694) | > loss_mel_ce: 2.8990232944488525 (nan) | > loss: 0.011667578481137753 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13647046465839216) | > loader_time: 0.0046 (0.004652413184259614)  --> TIME: 2025-02-24 20:11:52 -- STEP: 24394/36938 -- GLOBAL_STEP: 467650 | > loss_text_ce: 0.039627913385629654 (0.03797592626361691) | > loss_mel_ce: 0.6758895516395569 (nan) | > loss: 0.0028393552638590336 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13646009513332333) | > loader_time: 0.0045 (0.004652449788935384)  --> TIME: 2025-02-24 20:12:02 -- STEP: 24444/36938 -- GLOBAL_STEP: 467700 | > loss_text_ce: 0.03773823380470276 (0.03797825299290414) | > loss_mel_ce: 3.1192073822021484 (nan) | > loss: 0.01252756267786026 (nan) | > current_lr: 5e-06 | > step_time: 0.1709 (0.13647375084768196) | > loader_time: 0.0048 (0.004652489287005406)  --> TIME: 2025-02-24 20:12:11 -- STEP: 24494/36938 -- GLOBAL_STEP: 467750 | > loss_text_ce: 0.04146639257669449 (0.037977483573765035) | > loss_mel_ce: 3.191824436187744 (nan) | > loss: 0.012830520048737526 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13647732996907488) | > loader_time: 0.0045 (0.004652651697409352)  --> TIME: 2025-02-24 20:12:20 -- STEP: 24544/36938 -- GLOBAL_STEP: 467800 | > loss_text_ce: 0.03947120159864426 (0.037978060679342375) | > loss_mel_ce: 2.9309849739074707 (nan) | > loss: 0.01178752537816763 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1364766427326696) | > loader_time: 0.0052 (0.004652812814696937)  --> TIME: 2025-02-24 20:12:30 -- STEP: 24594/36938 -- GLOBAL_STEP: 467850 | > loss_text_ce: 0.03794805333018303 (0.03798076251272994) | > loss_mel_ce: 2.405320405960083 (nan) | > loss: 0.009695510379970074 (nan) | > current_lr: 5e-06 | > step_time: 0.1825 (0.1364772097215403) | > loader_time: 0.0047 (0.0046528352705784495)  --> TIME: 2025-02-24 20:12:39 -- STEP: 24644/36938 -- GLOBAL_STEP: 467900 | > loss_text_ce: 0.03615988418459892 (0.03798081533505826) | > loss_mel_ce: 3.04707407951355 (nan) | > loss: 0.012235056608915329 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13647529019916227) | > loader_time: 0.0049 (0.004652813006832726)  --> TIME: 2025-02-24 20:12:48 -- STEP: 24694/36938 -- GLOBAL_STEP: 467950 | > loss_text_ce: 0.04729955643415451 (0.03797979789713707) | > loss_mel_ce: 3.14947509765625 (nan) | > loss: 0.012685614638030529 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13648034931146352) | > loader_time: 0.0045 (0.004652760507143765)  --> TIME: 2025-02-24 20:12:58 -- STEP: 24744/36938 -- GLOBAL_STEP: 468000 | > loss_text_ce: 0.03670484200119972 (0.03797913472502524) | > loss_mel_ce: 3.265345811843872 (nan) | > loss: 0.013103376142680645 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13648599123168484) | > loader_time: 0.0046 (0.004652762447712325)  --> TIME: 2025-02-24 20:13:07 -- STEP: 24794/36938 -- GLOBAL_STEP: 468050 | > loss_text_ce: 0.03949806094169617 (0.03797712837990244) | > loss_mel_ce: 2.549067735671997 (nan) | > loss: 0.010272087529301643 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1364839566256848) | > loader_time: 0.0045 (0.0046526527870198)  --> TIME: 2025-02-24 20:13:16 -- STEP: 24844/36938 -- GLOBAL_STEP: 468100 | > loss_text_ce: 0.039864104241132736 (0.03797553388034947) | > loss_mel_ce: 0.9199067950248718 (nan) | > loss: 0.00380861503072083 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13648702337860427) | > loader_time: 0.0046 (0.004652634965991447)  --> TIME: 2025-02-24 20:13:26 -- STEP: 24894/36938 -- GLOBAL_STEP: 468150 | > loss_text_ce: 0.03629138693213463 (0.03797339082970821) | > loss_mel_ce: 2.9067916870117188 (nan) | > loss: 0.011678901500999928 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.1364926758603203) | > loader_time: 0.0045 (0.0046525400614100074)  --> TIME: 2025-02-24 20:13:35 -- STEP: 24944/36938 -- GLOBAL_STEP: 468200 | > loss_text_ce: 0.03877275437116623 (0.03797264701007728) | > loss_mel_ce: 2.8936541080474854 (nan) | > loss: 0.011636615730822086 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.1364932090158715) | > loader_time: 0.0045 (0.004652522614248391)  --> TIME: 2025-02-24 20:13:44 -- STEP: 24994/36938 -- GLOBAL_STEP: 468250 | > loss_text_ce: 0.03914247080683708 (0.03797434485475937) | > loss_mel_ce: 2.6857619285583496 (nan) | > loss: 0.010813113301992416 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13649318975058) | > loader_time: 0.0046 (0.004652420310885544)  --> TIME: 2025-02-24 20:13:54 -- STEP: 25044/36938 -- GLOBAL_STEP: 468300 | > loss_text_ce: 0.04523993283510208 (0.03797250287038347) | > loss_mel_ce: 3.559579849243164 (nan) | > loss: 0.014304840937256813 (nan) | > current_lr: 5e-06 | > step_time: 0.1861 (0.1364884242579643) | > loader_time: 0.005 (0.004652278746227259)  --> TIME: 2025-02-24 20:14:03 -- STEP: 25094/36938 -- GLOBAL_STEP: 468350 | > loss_text_ce: 0.042062778025865555 (0.0379740520274442) | > loss_mel_ce: 2.9684486389160156 (nan) | > loss: 0.011946474201977253 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13648554652090794) | > loader_time: 0.0045 (0.004652191027422867)  --> TIME: 2025-02-24 20:14:12 -- STEP: 25144/36938 -- GLOBAL_STEP: 468400 | > loss_text_ce: 0.023865828290581703 (0.037973678958509866) | > loss_mel_ce: 2.8166258335113525 (nan) | > loss: 0.011271793395280838 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13648125675313927) | > loader_time: 0.0052 (0.004652287421088383)  --> TIME: 2025-02-24 20:14:21 -- STEP: 25194/36938 -- GLOBAL_STEP: 468450 | > loss_text_ce: 0.04186408594250679 (0.037975997021231854) | > loss_mel_ce: 3.492745876312256 (nan) | > loss: 0.01402623113244772 (nan) | > current_lr: 5e-06 | > step_time: 0.1445 (0.1364798642173871) | > loader_time: 0.0046 (0.004652346619880546)  --> TIME: 2025-02-24 20:14:31 -- STEP: 25244/36938 -- GLOBAL_STEP: 468500 | > loss_text_ce: 0.034090083092451096 (0.03797574672989514) | > loss_mel_ce: 2.725497245788574 (nan) | > loss: 0.010950744152069092 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13647978461840796) | > loader_time: 0.0045 (0.004652263396252031)  --> TIME: 2025-02-24 20:14:40 -- STEP: 25294/36938 -- GLOBAL_STEP: 468550 | > loss_text_ce: 0.041305288672447205 (0.03797586601687188) | > loss_mel_ce: 3.2154576778411865 (nan) | > loss: 0.01292366348206997 (nan) | > current_lr: 5e-06 | > step_time: 0.1776 (0.13647148719821323) | > loader_time: 0.0047 (0.004652187005516)  --> TIME: 2025-02-24 20:14:49 -- STEP: 25344/36938 -- GLOBAL_STEP: 468600 | > loss_text_ce: 0.04751180112361908 (0.03797803516488682) | > loss_mel_ce: 3.0912418365478516 (nan) | > loss: 0.012455372139811516 (nan) | > current_lr: 5e-06 | > step_time: 0.1495 (0.13649042544303147) | > loader_time: 0.0047 (0.00465222423651598)  --> TIME: 2025-02-24 20:14:58 -- STEP: 25394/36938 -- GLOBAL_STEP: 468650 | > loss_text_ce: 0.03432336822152138 (0.037973569760764735) | > loss_mel_ce: 2.0370612144470215 (nan) | > loss: 0.008219781331717968 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13648199038946007) | > loader_time: 0.0045 (0.004652443153328406)  --> TIME: 2025-02-24 20:15:07 -- STEP: 25444/36938 -- GLOBAL_STEP: 468700 | > loss_text_ce: 0.03601798787713051 (0.037974165862082486) | > loss_mel_ce: 2.3806560039520264 (nan) | > loss: 0.009589976631104946 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.1364754129366904) | > loader_time: 0.0046 (0.004652363345825726)  --> TIME: 2025-02-24 20:15:17 -- STEP: 25494/36938 -- GLOBAL_STEP: 468750 | > loss_text_ce: 0.034474682062864304 (0.0379752235087382) | > loss_mel_ce: 3.332357168197632 (nan) | > loss: 0.013360444456338882 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13647638380438573) | > loader_time: 0.0046 (0.004652327412744156)  --> TIME: 2025-02-24 20:15:26 -- STEP: 25544/36938 -- GLOBAL_STEP: 468800 | > loss_text_ce: 0.041912741959095 (0.037976334361682504) | > loss_mel_ce: 3.1050264835357666 (nan) | > loss: 0.012487854808568954 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13646090673758107) | > loader_time: 0.0046 (0.004652306012811547)  --> TIME: 2025-02-24 20:15:35 -- STEP: 25594/36938 -- GLOBAL_STEP: 468850 | > loss_text_ce: 0.032613757997751236 (0.037972810658789186) | > loss_mel_ce: 0.02556355856359005 (nan) | > loss: 0.00023086236615199596 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13645323672461845) | > loader_time: 0.0045 (0.004652213284566282)  --> TIME: 2025-02-24 20:15:44 -- STEP: 25644/36938 -- GLOBAL_STEP: 468900 | > loss_text_ce: 0.03164791315793991 (0.037974059014806306) | > loss_mel_ce: 3.1205530166625977 (nan) | > loss: 0.012508735060691833 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.13645645840791573) | > loader_time: 0.0047 (0.004652140618783721)  --> TIME: 2025-02-24 20:15:54 -- STEP: 25694/36938 -- GLOBAL_STEP: 468950 | > loss_text_ce: 0.04375602304935455 (0.0379763117180478) | > loss_mel_ce: 2.9547340869903564 (nan) | > loss: 0.011898770928382874 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13646346466582215) | > loader_time: 0.0046 (0.004652103255340067)  --> TIME: 2025-02-24 20:16:03 -- STEP: 25744/36938 -- GLOBAL_STEP: 469000 | > loss_text_ce: 0.045202452689409256 (0.037978450581930716) | > loss_mel_ce: 2.7303948402404785 (nan) | > loss: 0.011014276184141636 (nan) | > current_lr: 5e-06 | > step_time: 0.1583 (0.13646155556882572) | > loader_time: 0.0047 (0.004652090523463669)  --> TIME: 2025-02-24 20:16:12 -- STEP: 25794/36938 -- GLOBAL_STEP: 469050 | > loss_text_ce: 0.028970059007406235 (0.037978766265692805) | > loss_mel_ce: 2.48868465423584 (nan) | > loss: 0.009990694001317024 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.13646257499525552) | > loader_time: 0.0046 (0.00465210158667595)  --> TIME: 2025-02-24 20:16:21 -- STEP: 25844/36938 -- GLOBAL_STEP: 469100 | > loss_text_ce: 0.04659363254904747 (0.03798047019464384) | > loss_mel_ce: 1.1569252014160156 (nan) | > loss: 0.004775868728756905 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13645899327172814) | > loader_time: 0.0046 (0.004652086933078651)  --> TIME: 2025-02-24 20:16:31 -- STEP: 25894/36938 -- GLOBAL_STEP: 469150 | > loss_text_ce: 0.050022583454847336 (0.03797950920573136) | > loss_mel_ce: 1.7875421047210693 (nan) | > loss: 0.007291923742741346 (nan) | > current_lr: 5e-06 | > step_time: 0.1081 (0.13646236460964) | > loader_time: 0.0045 (0.004652069058207912)  --> TIME: 2025-02-24 20:16:40 -- STEP: 25944/36938 -- GLOBAL_STEP: 469200 | > loss_text_ce: 0.04025011882185936 (0.037977806118012915) | > loss_mel_ce: 3.542191743850708 (nan) | > loss: 0.014216039329767227 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13645132961664222) | > loader_time: 0.0045 (0.004651985518034626)  --> TIME: 2025-02-24 20:16:49 -- STEP: 25994/36938 -- GLOBAL_STEP: 469250 | > loss_text_ce: 0.039267398416996 (0.03797661024376881) | > loss_mel_ce: 2.5593600273132324 (nan) | > loss: 0.010312014259397984 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.1364534024853987) | > loader_time: 0.0047 (0.004651958744113289)  --> TIME: 2025-02-24 20:16:58 -- STEP: 26044/36938 -- GLOBAL_STEP: 469300 | > loss_text_ce: 0.0483601875603199 (0.03797608023126619) | > loss_mel_ce: 3.1730761528015137 (nan) | > loss: 0.012783478014171124 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1364518002217998) | > loader_time: 0.0045 (0.0046519118324985734)  --> TIME: 2025-02-24 20:17:08 -- STEP: 26094/36938 -- GLOBAL_STEP: 469350 | > loss_text_ce: 0.031338728964328766 (0.037977099757277005) | > loss_mel_ce: 1.7139729261398315 (nan) | > loss: 0.006925839930772781 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13644928970382136) | > loader_time: 0.0046 (0.0046519952468362675)  --> TIME: 2025-02-24 20:17:17 -- STEP: 26144/36938 -- GLOBAL_STEP: 469400 | > loss_text_ce: 0.0403212308883667 (0.0379762414531074) | > loss_mel_ce: 3.1084377765655518 (nan) | > loss: 0.012495075352489948 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.1364451728697351) | > loader_time: 0.0047 (0.004651970276779986)  --> TIME: 2025-02-24 20:17:26 -- STEP: 26194/36938 -- GLOBAL_STEP: 469450 | > loss_text_ce: 0.03177468106150627 (0.037975927211974056) | > loss_mel_ce: 3.313671588897705 (nan) | > loss: 0.013275581412017345 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.1364518588869182) | > loader_time: 0.0046 (0.004652096787017406)  --> TIME: 2025-02-24 20:17:36 -- STEP: 26244/36938 -- GLOBAL_STEP: 469500 | > loss_text_ce: 0.04680083319544792 (0.03797596700768027) | > loss_mel_ce: 1.140127182006836 (nan) | > loss: 0.0047100321389734745 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13645327043068103) | > loader_time: 0.0046 (0.00465210496861469)  --> TIME: 2025-02-24 20:17:45 -- STEP: 26294/36938 -- GLOBAL_STEP: 469550 | > loss_text_ce: 0.03824218735098839 (0.03797739683356811) | > loss_mel_ce: 3.085552930831909 (nan) | > loss: 0.012396012432873249 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13644080852259796) | > loader_time: 0.0046 (0.00465200745652284)  --> TIME: 2025-02-24 20:17:54 -- STEP: 26344/36938 -- GLOBAL_STEP: 469600 | > loss_text_ce: 0.044970400631427765 (0.03798053902199151) | > loss_mel_ce: 3.2814714908599854 (nan) | > loss: 0.013200167566537857 (nan) | > current_lr: 5e-06 | > step_time: 0.1832 (0.1364446789560214) | > loader_time: 0.0049 (0.004652023885592482)  --> TIME: 2025-02-24 20:18:03 -- STEP: 26394/36938 -- GLOBAL_STEP: 469650 | > loss_text_ce: 0.039437200874090195 (0.03798146137500957) | > loss_mel_ce: 3.0811524391174316 (nan) | > loss: 0.012383293360471725 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13643830261727558) | > loader_time: 0.0046 (0.004652123311401077)  --> TIME: 2025-02-24 20:18:13 -- STEP: 26444/36938 -- GLOBAL_STEP: 469700 | > loss_text_ce: 0.044216278940439224 (0.03798068929581964) | > loss_mel_ce: 3.426955223083496 (nan) | > loss: 0.013774490915238857 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.1364462296029756) | > loader_time: 0.0047 (0.004652210829784791)  --> TIME: 2025-02-24 20:18:22 -- STEP: 26494/36938 -- GLOBAL_STEP: 469750 | > loss_text_ce: 0.02646050415933132 (0.037978550260182826) | > loss_mel_ce: 3.0774149894714355 (nan) | > loss: 0.01231696642935276 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.1364490588926007) | > loader_time: 0.0047 (0.004652332294893403)  --> TIME: 2025-02-24 20:18:32 -- STEP: 26544/36938 -- GLOBAL_STEP: 469800 | > loss_text_ce: 0.02777041867375374 (0.03797335728629358) | > loss_mel_ce: 0.37729713320732117 (nan) | > loss: 0.0016074110753834248 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1364546652081639) | > loader_time: 0.0045 (0.004652274713018921)  --> TIME: 2025-02-24 20:18:41 -- STEP: 26594/36938 -- GLOBAL_STEP: 469850 | > loss_text_ce: 0.03794271498918533 (0.037972897082534166) | > loss_mel_ce: 1.4823215007781982 (nan) | > loss: 0.006032795179635286 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.13644923369149442) | > loader_time: 0.0046 (0.004652136069827954)  --> TIME: 2025-02-24 20:18:50 -- STEP: 26644/36938 -- GLOBAL_STEP: 469900 | > loss_text_ce: 0.03339584916830063 (0.03797130831127889) | > loss_mel_ce: 3.478630542755127 (nan) | > loss: 0.013936612755060196 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.1364456792615793) | > loader_time: 0.0046 (0.0046519863768357695)  --> TIME: 2025-02-24 20:18:59 -- STEP: 26694/36938 -- GLOBAL_STEP: 469950 | > loss_text_ce: 0.03439927101135254 (0.03796724724021134) | > loss_mel_ce: 3.13459849357605 (nan) | > loss: 0.012575388886034489 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.13644438129494119) | > loader_time: 0.0047 (0.004651837405385262)  --> TIME: 2025-02-24 20:19:08 -- STEP: 26744/36938 -- GLOBAL_STEP: 470000 | > loss_text_ce: 0.037128277122974396 (0.037968136752545324) | > loss_mel_ce: 2.4313511848449707 (nan) | > loss: 0.009795553982257843 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13643933275021852) | > loader_time: 0.0046 (0.004651765890403814) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_470000.pth  --> TIME: 2025-02-24 20:19:24 -- STEP: 26794/36938 -- GLOBAL_STEP: 470050 | > loss_text_ce: 0.023845897987484932 (0.03796775952246274) | > loss_mel_ce: 3.167880058288574 (nan) | > loss: 0.0126655800268054 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13644159802075614) | > loader_time: 0.0046 (0.004651854213889079)  --> TIME: 2025-02-24 20:19:33 -- STEP: 26844/36938 -- GLOBAL_STEP: 470100 | > loss_text_ce: 0.035749197006225586 (0.03796626999710587) | > loss_mel_ce: 2.52410888671875 (nan) | > loss: 0.010158167220652103 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1364351971247844) | > loader_time: 0.0046 (0.004651898119918853)  --> TIME: 2025-02-24 20:19:42 -- STEP: 26894/36938 -- GLOBAL_STEP: 470150 | > loss_text_ce: 0.04699741676449776 (0.03796438285368291) | > loss_mel_ce: 0.17161975800991058 (nan) | > loss: 0.000867528491653502 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13642681231859982) | > loader_time: 0.0045 (0.00465175125371422)  --> TIME: 2025-02-24 20:19:51 -- STEP: 26944/36938 -- GLOBAL_STEP: 470200 | > loss_text_ce: 0.0366913303732872 (0.03796249629885148) | > loss_mel_ce: 3.061511278152466 (nan) | > loss: 0.01229445543140173 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13643599318389002) | > loader_time: 0.0046 (0.004651762580503412)  --> TIME: 2025-02-24 20:20:01 -- STEP: 26994/36938 -- GLOBAL_STEP: 470250 | > loss_text_ce: 0.03335617110133171 (0.037960321217784504) | > loss_mel_ce: 2.617041826248169 (nan) | > loss: 0.010517452843487263 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13643591743297317) | > loader_time: 0.0046 (0.004651738562707157)  --> TIME: 2025-02-24 20:20:10 -- STEP: 27044/36938 -- GLOBAL_STEP: 470300 | > loss_text_ce: 0.029711486771702766 (0.03795865209332528) | > loss_mel_ce: 3.0145413875579834 (nan) | > loss: 0.012080369517207146 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13643535841606183) | > loader_time: 0.0045 (0.004651720787254779)  --> TIME: 2025-02-24 20:20:19 -- STEP: 27094/36938 -- GLOBAL_STEP: 470350 | > loss_text_ce: 0.028365477919578552 (0.037959340103502756) | > loss_mel_ce: 2.6945083141326904 (nan) | > loss: 0.010805054567754269 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13643256805915724) | > loader_time: 0.0045 (0.004651651062489797)  --> TIME: 2025-02-24 20:20:28 -- STEP: 27144/36938 -- GLOBAL_STEP: 470400 | > loss_text_ce: 0.045618195086717606 (0.037958955948448066) | > loss_mel_ce: 3.2217414379119873 (nan) | > loss: 0.012965713627636433 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13643614152452346) | > loader_time: 0.0048 (0.004651647128083463)  --> TIME: 2025-02-24 20:20:37 -- STEP: 27194/36938 -- GLOBAL_STEP: 470450 | > loss_text_ce: 0.033187489956617355 (0.03795679987210208) | > loss_mel_ce: 3.6313583850860596 (nan) | > loss: 0.014541849493980408 (nan) | > current_lr: 5e-06 | > step_time: 0.187 (0.13642959610682998) | > loader_time: 0.0048 (0.0046515547896935384)  --> TIME: 2025-02-24 20:20:47 -- STEP: 27244/36938 -- GLOBAL_STEP: 470500 | > loss_text_ce: 0.04054398834705353 (0.037958386593474135) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13643067664747147) | > loader_time: 0.0046 (0.004651526674232204)  --> TIME: 2025-02-24 20:20:56 -- STEP: 27294/36938 -- GLOBAL_STEP: 470550 | > loss_text_ce: 0.03926345333456993 (0.03795922450089195) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.13643378738676593) | > loader_time: 0.0048 (0.004651437725017604)  --> TIME: 2025-02-24 20:21:05 -- STEP: 27344/36938 -- GLOBAL_STEP: 470600 | > loss_text_ce: 0.04470861330628395 (0.037957579830467734) | > loss_mel_ce: 3.034414529800415 (nan) | > loss: 0.012218743562698364 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.13643160851256933) | > loader_time: 0.0047 (0.004651505401986825)  --> TIME: 2025-02-24 20:21:14 -- STEP: 27394/36938 -- GLOBAL_STEP: 470650 | > loss_text_ce: 0.031390365213155746 (0.03795996232652092) | > loss_mel_ce: 3.160637855529785 (nan) | > loss: 0.012666779570281506 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13642421883284056) | > loader_time: 0.0047 (0.0046515377053310825)  --> TIME: 2025-02-24 20:21:24 -- STEP: 27444/36938 -- GLOBAL_STEP: 470700 | > loss_text_ce: 0.027895187959074974 (0.03795945131116222) | > loss_mel_ce: 2.764082670211792 (nan) | > loss: 0.011079277843236923 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1364206544877894) | > loader_time: 0.0044 (0.004651538642184909)  --> TIME: 2025-02-24 20:21:33 -- STEP: 27494/36938 -- GLOBAL_STEP: 470750 | > loss_text_ce: 0.039106786251068115 (0.03795839297993914) | > loss_mel_ce: 2.8064916133880615 (nan) | > loss: 0.011292058043181896 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13640959821996335) | > loader_time: 0.0046 (0.004651416758931759)  --> TIME: 2025-02-24 20:21:42 -- STEP: 27544/36938 -- GLOBAL_STEP: 470800 | > loss_text_ce: 0.030376659706234932 (0.03795830138563799) | > loss_mel_ce: 7.026041507720947 (nan) | > loss: 0.028001660481095314 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13640755187312054) | > loader_time: 0.0047 (0.004651387997086238)  --> TIME: 2025-02-24 20:21:51 -- STEP: 27594/36938 -- GLOBAL_STEP: 470850 | > loss_text_ce: 0.037939418107271194 (0.037957942610836244) | > loss_mel_ce: 3.063955068588257 (nan) | > loss: 0.012309105135500431 (nan) | > current_lr: 5e-06 | > step_time: 0.182 (0.13640703081505662) | > loader_time: 0.0048 (0.004651374183392615)  --> TIME: 2025-02-24 20:22:01 -- STEP: 27644/36938 -- GLOBAL_STEP: 470900 | > loss_text_ce: 0.03034600429236889 (0.03795548639104051) | > loss_mel_ce: 3.248189926147461 (nan) | > loss: 0.013010063208639622 (nan) | > current_lr: 5e-06 | > step_time: 0.1499 (0.13641743649796012) | > loader_time: 0.0067 (0.0046515070983252456)  --> TIME: 2025-02-24 20:22:10 -- STEP: 27694/36938 -- GLOBAL_STEP: 470950 | > loss_text_ce: 0.0441904291510582 (0.03795334005545024) | > loss_mel_ce: 3.089759111404419 (nan) | > loss: 0.012436307966709137 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.136409647784835) | > loader_time: 0.0047 (0.004651512403716067)  --> TIME: 2025-02-24 20:22:19 -- STEP: 27744/36938 -- GLOBAL_STEP: 471000 | > loss_text_ce: 0.03790079802274704 (0.037951284076848286) | > loss_mel_ce: 1.3576335906982422 (nan) | > loss: 0.005537835415452719 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13641357301565646) | > loader_time: 0.0046 (0.0046516107921643745)  --> TIME: 2025-02-24 20:22:28 -- STEP: 27794/36938 -- GLOBAL_STEP: 471050 | > loss_text_ce: 0.03564198315143585 (0.03795045821633895) | > loss_mel_ce: 2.6573314666748047 (nan) | > loss: 0.010686403140425682 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13641072468182128) | > loader_time: 0.0047 (0.004651634146037073)  --> TIME: 2025-02-24 20:22:38 -- STEP: 27844/36938 -- GLOBAL_STEP: 471100 | > loss_text_ce: 0.037213023751974106 (0.03794900983067751) | > loss_mel_ce: 2.5447819232940674 (nan) | > loss: 0.01024601235985756 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13640963119296584) | > loader_time: 0.0045 (0.004651957948117717)  --> TIME: 2025-02-24 20:22:47 -- STEP: 27894/36938 -- GLOBAL_STEP: 471150 | > loss_text_ce: 0.04480351507663727 (0.03795032481636023) | > loss_mel_ce: 3.371729850769043 (nan) | > loss: 0.013557673431932926 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.13640508189665895) | > loader_time: 0.0049 (0.00465190168725222)  --> TIME: 2025-02-24 20:22:56 -- STEP: 27944/36938 -- GLOBAL_STEP: 471200 | > loss_text_ce: 0.042924605309963226 (0.03795086466945257) | > loss_mel_ce: 2.4493582248687744 (nan) | > loss: 0.009890011511743069 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13639851048004223) | > loader_time: 0.0047 (0.004651846224961985)  --> TIME: 2025-02-24 20:23:05 -- STEP: 27994/36938 -- GLOBAL_STEP: 471250 | > loss_text_ce: 0.03399592638015747 (0.037952351610790386) | > loss_mel_ce: 2.49609375 (nan) | > loss: 0.01004003919661045 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13639561265692013) | > loader_time: 0.0046 (0.004651797578327425)  --> TIME: 2025-02-24 20:23:14 -- STEP: 28044/36938 -- GLOBAL_STEP: 471300 | > loss_text_ce: 0.02684853784739971 (0.03795249997660753) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13639205751159156) | > loader_time: 0.0047 (0.004651824786308991)  --> TIME: 2025-02-24 20:23:24 -- STEP: 28094/36938 -- GLOBAL_STEP: 471350 | > loss_text_ce: 0.03395938500761986 (0.0379552159601542) | > loss_mel_ce: 3.67567777633667 (nan) | > loss: 0.014720783568918705 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13639365702564019) | > loader_time: 0.0046 (0.004651739986509508)  --> TIME: 2025-02-24 20:23:33 -- STEP: 28144/36938 -- GLOBAL_STEP: 471400 | > loss_text_ce: 0.03695181384682655 (0.03795679064453391) | > loss_mel_ce: 0.48021239042282104 (nan) | > loss: 0.002052239142358303 (nan) | > current_lr: 5e-06 | > step_time: 0.1622 (0.13639180646862917) | > loader_time: 0.0047 (0.00465179679067106)  --> TIME: 2025-02-24 20:23:42 -- STEP: 28194/36938 -- GLOBAL_STEP: 471450 | > loss_text_ce: 0.042034849524497986 (0.0379583792540773) | > loss_mel_ce: 3.2190539836883545 (nan) | > loss: 0.012940829619765282 (nan) | > current_lr: 5e-06 | > step_time: 0.1781 (0.13639186223009614) | > loader_time: 0.0048 (0.004651768280105549)  --> TIME: 2025-02-24 20:23:51 -- STEP: 28244/36938 -- GLOBAL_STEP: 471500 | > loss_text_ce: 0.04318295419216156 (0.03795849698583309) | > loss_mel_ce: 3.030407190322876 (nan) | > loss: 0.01219678670167923 (nan) | > current_lr: 5e-06 | > step_time: 0.1662 (0.1363918831844424) | > loader_time: 0.0048 (0.0046517667309823)  --> TIME: 2025-02-24 20:24:01 -- STEP: 28294/36938 -- GLOBAL_STEP: 471550 | > loss_text_ce: 0.027017096057534218 (0.03795495641999694) | > loss_mel_ce: 2.530561685562134 (nan) | > loss: 0.010149123147130013 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13639262107317265) | > loader_time: 0.0046 (0.004651706564373277)  --> TIME: 2025-02-24 20:24:10 -- STEP: 28344/36938 -- GLOBAL_STEP: 471600 | > loss_text_ce: 0.0522092767059803 (0.03795363884975819) | > loss_mel_ce: 3.7390356063842773 (nan) | > loss: 0.015044623985886574 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13639015269723498) | > loader_time: 0.0046 (0.004651652657982264)  --> TIME: 2025-02-24 20:24:19 -- STEP: 28394/36938 -- GLOBAL_STEP: 471650 | > loss_text_ce: 0.03556491807103157 (0.03795408329482915) | > loss_mel_ce: 3.2464687824249268 (nan) | > loss: 0.01302394364029169 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.1363913259464565) | > loader_time: 0.0046 (0.004651629295854214)  --> TIME: 2025-02-24 20:24:29 -- STEP: 28444/36938 -- GLOBAL_STEP: 471700 | > loss_text_ce: 0.02973473258316517 (0.03795464155777603) | > loss_mel_ce: 2.461461305618286 (nan) | > loss: 0.009885699488222599 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13639312371508805) | > loader_time: 0.0047 (0.004651620323992207)  --> TIME: 2025-02-24 20:24:38 -- STEP: 28494/36938 -- GLOBAL_STEP: 471750 | > loss_text_ce: 0.03996508568525314 (0.03795533792895679) | > loss_mel_ce: 2.4613277912139893 (nan) | > loss: 0.009925765916705132 (nan) | > current_lr: 5e-06 | > step_time: 0.1794 (0.1363878190463026) | > loader_time: 0.0048 (0.004651785106033671)  --> TIME: 2025-02-24 20:24:47 -- STEP: 28544/36938 -- GLOBAL_STEP: 471800 | > loss_text_ce: 0.037650447338819504 (0.037954581608924585) | > loss_mel_ce: 3.195749521255493 (nan) | > loss: 0.012830952182412148 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1363893729204408) | > loader_time: 0.0046 (0.004651850749291589)  --> TIME: 2025-02-24 20:24:57 -- STEP: 28594/36938 -- GLOBAL_STEP: 471850 | > loss_text_ce: 0.0358678363263607 (0.037951748780605296) | > loss_mel_ce: 2.1541569232940674 (nan) | > loss: 0.008690575137734413 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.13638947791650197) | > loader_time: 0.0046 (0.004651744665695595)  --> TIME: 2025-02-24 20:25:06 -- STEP: 28644/36938 -- GLOBAL_STEP: 471900 | > loss_text_ce: 0.034537941217422485 (0.03795163475907303) | > loss_mel_ce: 3.221740484237671 (nan) | > loss: 0.012921741232275963 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.1363855355396186) | > loader_time: 0.0046 (0.004651638478011505)  --> TIME: 2025-02-24 20:25:15 -- STEP: 28694/36938 -- GLOBAL_STEP: 471950 | > loss_text_ce: 0.025298114866018295 (0.03795112000842977) | > loss_mel_ce: 0.0128936767578125 (nan) | > loss: 0.0001515547337476164 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13638294598129982) | > loader_time: 0.0048 (0.004651541791992811)  --> TIME: 2025-02-24 20:25:25 -- STEP: 28744/36938 -- GLOBAL_STEP: 472000 | > loss_text_ce: 0.03975551575422287 (0.03795214554054795) | > loss_mel_ce: 2.631452798843384 (nan) | > loss: 0.010600034147500992 (nan) | > current_lr: 5e-06 | > step_time: 0.1521 (0.136389688366539) | > loader_time: 0.0047 (0.004651496080578461)  --> TIME: 2025-02-24 20:25:34 -- STEP: 28794/36938 -- GLOBAL_STEP: 472050 | > loss_text_ce: 0.029024023562669754 (0.037950541449069966) | > loss_mel_ce: 2.608593702316284 (nan) | > loss: 0.010466737672686577 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.1363855788978812) | > loader_time: 0.0045 (0.004651431798222787)  --> TIME: 2025-02-24 20:25:43 -- STEP: 28844/36938 -- GLOBAL_STEP: 472100 | > loss_text_ce: 0.04926092550158501 (0.037951742679418275) | > loss_mel_ce: 2.6241366863250732 (nan) | > loss: 0.01060872059315443 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.136382913369606) | > loader_time: 0.0046 (0.004651341709742094)  --> TIME: 2025-02-24 20:25:53 -- STEP: 28894/36938 -- GLOBAL_STEP: 472150 | > loss_text_ce: 0.030479086562991142 (0.03795222775180847) | > loss_mel_ce: 2.338221311569214 (nan) | > loss: 0.00939960591495037 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13639009630109764) | > loader_time: 0.0045 (0.0046513199583422555)  --> TIME: 2025-02-24 20:26:02 -- STEP: 28944/36938 -- GLOBAL_STEP: 472200 | > loss_text_ce: 0.042783573269844055 (0.03795078947968068) | > loss_mel_ce: 0.018829345703125 (nan) | > loss: 0.0002444957208354026 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.1363895352355269) | > loader_time: 0.0046 (0.004651280917997393)  --> TIME: 2025-02-24 20:26:11 -- STEP: 28994/36938 -- GLOBAL_STEP: 472250 | > loss_text_ce: 0.03147190809249878 (0.037948585610457504) | > loss_mel_ce: 1.30841064453125 (nan) | > loss: 0.005316994618624449 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1363903496391538) | > loader_time: 0.0045 (0.004651309638514399)  --> TIME: 2025-02-24 20:26:21 -- STEP: 29044/36938 -- GLOBAL_STEP: 472300 | > loss_text_ce: 0.03168598935008049 (0.03794546387254665) | > loss_mel_ce: 3.5422797203063965 (nan) | > loss: 0.01418240461498499 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.13639615282607043) | > loader_time: 0.0047 (0.004651364815857314)  --> TIME: 2025-02-24 20:26:30 -- STEP: 29094/36938 -- GLOBAL_STEP: 472350 | > loss_text_ce: 0.03396071493625641 (0.037941700570959386) | > loss_mel_ce: 2.2044320106506348 (nan) | > loss: 0.008882511407136917 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13639485775387608) | > loader_time: 0.0046 (0.0046513072975789055)  --> TIME: 2025-02-24 20:26:40 -- STEP: 29144/36938 -- GLOBAL_STEP: 472400 | > loss_text_ce: 0.03612915053963661 (0.03794137770056319) | > loss_mel_ce: 1.6807303428649902 (nan) | > loss: 0.006812934763729572 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13639823936545484) | > loader_time: 0.0046 (0.0046512636384432)  --> TIME: 2025-02-24 20:26:49 -- STEP: 29194/36938 -- GLOBAL_STEP: 472450 | > loss_text_ce: 0.032166555523872375 (0.037943312058161136) | > loss_mel_ce: 2.8043668270111084 (nan) | > loss: 0.011256084777414799 (nan) | > current_lr: 5e-06 | > step_time: 0.1116 (0.13639935287798946) | > loader_time: 0.0046 (0.004651270901216787)  --> TIME: 2025-02-24 20:26:58 -- STEP: 29244/36938 -- GLOBAL_STEP: 472500 | > loss_text_ce: 0.03628502041101456 (0.037945556571779025) | > loss_mel_ce: 3.2265384197235107 (nan) | > loss: 0.01294771209359169 (nan) | > current_lr: 5e-06 | > step_time: 0.1495 (0.1364025244991961) | > loader_time: 0.0046 (0.00465129524359308)  --> TIME: 2025-02-24 20:27:08 -- STEP: 29294/36938 -- GLOBAL_STEP: 472550 | > loss_text_ce: 0.03744898736476898 (0.03794437716405068) | > loss_mel_ce: 2.7692651748657227 (nan) | > loss: 0.011137754656374454 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13640239162722256) | > loader_time: 0.0046 (0.004651210694994375)  --> TIME: 2025-02-24 20:27:17 -- STEP: 29344/36938 -- GLOBAL_STEP: 472600 | > loss_text_ce: 0.03043297864496708 (0.037942864328429644) | > loss_mel_ce: 3.064337968826294 (nan) | > loss: 0.012280837632715702 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13638997877436942) | > loader_time: 0.0046 (0.004651081072920095)  --> TIME: 2025-02-24 20:27:26 -- STEP: 29394/36938 -- GLOBAL_STEP: 472650 | > loss_text_ce: 0.03691120818257332 (0.037940397663477336) | > loss_mel_ce: 3.2647995948791504 (nan) | > loss: 0.013102027587592602 (nan) | > current_lr: 5e-06 | > step_time: 0.1837 (0.13639714496106492) | > loader_time: 0.0048 (0.004651019481880281)  --> TIME: 2025-02-24 20:27:35 -- STEP: 29444/36938 -- GLOBAL_STEP: 472700 | > loss_text_ce: 0.037879589945077896 (0.037942709486556855) | > loss_mel_ce: 2.754756212234497 (nan) | > loss: 0.011081889271736145 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.136399521377159) | > loader_time: 0.0048 (0.004650987323382104)  --> TIME: 2025-02-24 20:27:45 -- STEP: 29494/36938 -- GLOBAL_STEP: 472750 | > loss_text_ce: 0.044086821377277374 (0.03794398100084481) | > loss_mel_ce: 2.8636651039123535 (nan) | > loss: 0.011538699269294739 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13640105424431875) | > loader_time: 0.0045 (0.004650902415062304)  --> TIME: 2025-02-24 20:27:54 -- STEP: 29544/36938 -- GLOBAL_STEP: 472800 | > loss_text_ce: 0.03168704733252525 (0.03794249604336217) | > loss_mel_ce: 0.8998783826828003 (nan) | > loss: 0.0036966882180422544 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13639323196693726) | > loader_time: 0.0051 (0.004650944000074164)  --> TIME: 2025-02-24 20:28:03 -- STEP: 29594/36938 -- GLOBAL_STEP: 472850 | > loss_text_ce: 0.02988719940185547 (0.0379428295527192) | > loss_mel_ce: 2.9847116470336914 (nan) | > loss: 0.011962694115936756 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13639120624751128) | > loader_time: 0.0044 (0.0046508664930801575)  --> TIME: 2025-02-24 20:28:12 -- STEP: 29644/36938 -- GLOBAL_STEP: 472900 | > loss_text_ce: 0.04280078411102295 (0.03794005102324155) | > loss_mel_ce: 3.1799237728118896 (nan) | > loss: 0.012788590043783188 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.1363894407717953) | > loader_time: 0.0046 (0.004650928531479899)  --> TIME: 2025-02-24 20:28:22 -- STEP: 29694/36938 -- GLOBAL_STEP: 472950 | > loss_text_ce: 0.03207571804523468 (0.037940777852249884) | > loss_mel_ce: 3.34429931640625 (nan) | > loss: 0.013398313894867897 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.1363893825935404) | > loader_time: 0.0046 (0.004650972471932776)  --> TIME: 2025-02-24 20:28:31 -- STEP: 29744/36938 -- GLOBAL_STEP: 473000 | > loss_text_ce: 0.04276835918426514 (0.037939997130060375) | > loss_mel_ce: 2.643718957901001 (nan) | > loss: 0.010660664178431034 (nan) | > current_lr: 5e-06 | > step_time: 0.1917 (0.13639180363025988) | > loader_time: 0.0049 (0.0046509277233838355)  --> TIME: 2025-02-24 20:28:40 -- STEP: 29794/36938 -- GLOBAL_STEP: 473050 | > loss_text_ce: 0.03815407305955887 (0.03793797507276829) | > loss_mel_ce: 3.367140769958496 (nan) | > loss: 0.01351307611912489 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13639311096032214) | > loader_time: 0.0047 (0.004650919279124191)  --> TIME: 2025-02-24 20:28:50 -- STEP: 29844/36938 -- GLOBAL_STEP: 473100 | > loss_text_ce: 0.030978938564658165 (0.03793889366365792) | > loss_mel_ce: 3.2155942916870117 (nan) | > loss: 0.012883227318525314 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13639309198251265) | > loader_time: 0.0047 (0.004650883477457255)  --> TIME: 2025-02-24 20:28:59 -- STEP: 29894/36938 -- GLOBAL_STEP: 473150 | > loss_text_ce: 0.04363584890961647 (0.037941810207684126) | > loss_mel_ce: 2.7972412109375 (nan) | > loss: 0.011273322626948357 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13639648753264463) | > loader_time: 0.0046 (0.004650852094328515)  --> TIME: 2025-02-24 20:29:08 -- STEP: 29944/36938 -- GLOBAL_STEP: 473200 | > loss_text_ce: 0.03941626846790314 (0.037942292958762044) | > loss_mel_ce: 2.737704277038574 (nan) | > loss: 0.01102032046765089 (nan) | > current_lr: 5e-06 | > step_time: 0.1769 (0.1363996839797196) | > loader_time: 0.0049 (0.004650874114628889)  --> TIME: 2025-02-24 20:29:18 -- STEP: 29994/36938 -- GLOBAL_STEP: 473250 | > loss_text_ce: 0.03924277797341347 (0.03794363101927192) | > loss_mel_ce: 2.5541276931762695 (nan) | > loss: 0.010291153565049171 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13640949565855223) | > loader_time: 0.0046 (0.004650934859976091)  --> TIME: 2025-02-24 20:29:27 -- STEP: 30044/36938 -- GLOBAL_STEP: 473300 | > loss_text_ce: 0.04426904395222664 (0.03794273506838725) | > loss_mel_ce: 2.8315255641937256 (nan) | > loss: 0.011411883868277073 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13640856531770504) | > loader_time: 0.0045 (0.004650853854999429)  --> TIME: 2025-02-24 20:29:37 -- STEP: 30094/36938 -- GLOBAL_STEP: 473350 | > loss_text_ce: 0.0283699631690979 (0.037940368779810024) | > loss_mel_ce: 3.109910249710083 (nan) | > loss: 0.012453493662178516 (nan) | > current_lr: 5e-06 | > step_time: 0.1761 (0.13640562814755577) | > loader_time: 0.0048 (0.004650762582321148)  --> TIME: 2025-02-24 20:29:46 -- STEP: 30144/36938 -- GLOBAL_STEP: 473400 | > loss_text_ce: 0.03651119023561478 (0.03793789806768709) | > loss_mel_ce: 3.005121946334839 (nan) | > loss: 0.012069973163306713 (nan) | > current_lr: 5e-06 | > step_time: 0.19 (0.1364038724475968) | > loader_time: 0.0049 (0.004650686450318364)  --> TIME: 2025-02-24 20:29:55 -- STEP: 30194/36938 -- GLOBAL_STEP: 473450 | > loss_text_ce: 0.03733261302113533 (0.037937836761976845) | > loss_mel_ce: 2.7949604988098145 (nan) | > loss: 0.011239258572459221 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.1364005212713678) | > loader_time: 0.0046 (0.004650635893647882)  --> TIME: 2025-02-24 20:30:04 -- STEP: 30244/36938 -- GLOBAL_STEP: 473500 | > loss_text_ce: 0.03968450054526329 (0.03793848029410747) | > loss_mel_ce: 1.2085397243499756 (nan) | > loss: 0.004953271243721247 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13640101099278984) | > loader_time: 0.0044 (0.004650582666199015)  --> TIME: 2025-02-24 20:30:14 -- STEP: 30294/36938 -- GLOBAL_STEP: 473550 | > loss_text_ce: 0.03573957458138466 (0.037941026658176655) | > loss_mel_ce: 3.316504716873169 (nan) | > loss: 0.013302558101713657 (nan) | > current_lr: 5e-06 | > step_time: 0.1793 (0.13639726296914093) | > loader_time: 0.0049 (0.004650495056587478)  --> TIME: 2025-02-24 20:30:23 -- STEP: 30344/36938 -- GLOBAL_STEP: 473600 | > loss_text_ce: 0.027687212452292442 (0.03794061749535299) | > loss_mel_ce: 3.087078094482422 (nan) | > loss: 0.012360180728137493 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.1364034676878661) | > loader_time: 0.0046 (0.004650667926550423)  --> TIME: 2025-02-24 20:30:32 -- STEP: 30394/36938 -- GLOBAL_STEP: 473650 | > loss_text_ce: 0.04487477242946625 (0.0379413351068928) | > loss_mel_ce: 2.5782418251037598 (nan) | > loss: 0.010409193113446236 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.1364045681400863) | > loader_time: 0.0046 (0.004650661111811404)  --> TIME: 2025-02-24 20:30:42 -- STEP: 30444/36938 -- GLOBAL_STEP: 473700 | > loss_text_ce: 0.03429014980792999 (0.037940961218808265) | > loss_mel_ce: 0.1030069962143898 (nan) | > loss: 0.0005448299925774336 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13641077117934788) | > loader_time: 0.0046 (0.004650726062743028)  --> TIME: 2025-02-24 20:30:51 -- STEP: 30494/36938 -- GLOBAL_STEP: 473750 | > loss_text_ce: 0.0389295294880867 (0.037939771832863484) | > loss_mel_ce: 2.4846787452697754 (nan) | > loss: 0.010014318861067295 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13640619593228764) | > loader_time: 0.0046 (0.004650841957390461)  --> TIME: 2025-02-24 20:31:00 -- STEP: 30544/36938 -- GLOBAL_STEP: 473800 | > loss_text_ce: 0.05115548148751259 (0.03793865952323302) | > loss_mel_ce: 3.753502130508423 (nan) | > loss: 0.015097849071025848 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13640636887114313) | > loader_time: 0.0046 (0.004650970352076881)  --> TIME: 2025-02-24 20:31:10 -- STEP: 30594/36938 -- GLOBAL_STEP: 473850 | > loss_text_ce: 0.03993312641978264 (0.037936692369169495) | > loss_mel_ce: 3.241097927093506 (nan) | > loss: 0.013019965961575508 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13640641366577277) | > loader_time: 0.0046 (0.004650998841845687)  --> TIME: 2025-02-24 20:31:19 -- STEP: 30644/36938 -- GLOBAL_STEP: 473900 | > loss_text_ce: 0.038709964603185654 (0.03793682393416375) | > loss_mel_ce: 2.2579197883605957 (nan) | > loss: 0.009113610722124577 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13640383711808363) | > loader_time: 0.0045 (0.004651016696379159)  --> TIME: 2025-02-24 20:31:28 -- STEP: 30694/36938 -- GLOBAL_STEP: 473950 | > loss_text_ce: 0.042886510491371155 (0.037938271477596916) | > loss_mel_ce: 2.994786024093628 (nan) | > loss: 0.012054257094860077 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.1364096781093133) | > loader_time: 0.0046 (0.004651028449553803)  --> TIME: 2025-02-24 20:31:38 -- STEP: 30744/36938 -- GLOBAL_STEP: 474000 | > loss_text_ce: 0.040819548070430756 (0.03793485391807618) | > loss_mel_ce: 3.1370131969451904 (nan) | > loss: 0.012610448524355888 (nan) | > current_lr: 5e-06 | > step_time: 0.151 (0.13641451001787525) | > loader_time: 0.0048 (0.0046510838482052425)  --> TIME: 2025-02-24 20:31:47 -- STEP: 30794/36938 -- GLOBAL_STEP: 474050 | > loss_text_ce: 0.03907657042145729 (0.03793381247956777) | > loss_mel_ce: 3.397545576095581 (nan) | > loss: 0.013637389987707138 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13641314116469933) | > loader_time: 0.0045 (0.004651195516584392)  --> TIME: 2025-02-24 20:31:56 -- STEP: 30844/36938 -- GLOBAL_STEP: 474100 | > loss_text_ce: 0.040728356689214706 (0.03793344054071713) | > loss_mel_ce: 0.5098702311515808 (nan) | > loss: 0.0021849151235073805 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13640832502142078) | > loader_time: 0.0045 (0.004651240439223843)  --> TIME: 2025-02-24 20:32:06 -- STEP: 30894/36938 -- GLOBAL_STEP: 474150 | > loss_text_ce: 0.03983867168426514 (0.037932531847229986) | > loss_mel_ce: 1.9222368001937866 (nan) | > loss: 0.00778601411730051 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13641690491719194) | > loader_time: 0.0045 (0.004651271147797986)  --> TIME: 2025-02-24 20:32:15 -- STEP: 30944/36938 -- GLOBAL_STEP: 474200 | > loss_text_ce: 0.041539162397384644 (0.037933431631222654) | > loss_mel_ce: 2.7718605995178223 (nan) | > loss: 0.011164285242557526 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.1364182497084802) | > loader_time: 0.0052 (0.0046512912169113735)  --> TIME: 2025-02-24 20:32:24 -- STEP: 30994/36938 -- GLOBAL_STEP: 474250 | > loss_text_ce: 0.04722566530108452 (0.037933917255893) | > loss_mel_ce: 2.0544753074645996 (nan) | > loss: 0.008340083993971348 (nan) | > current_lr: 5e-06 | > step_time: 0.1123 (0.13641644718031903) | > loader_time: 0.0044 (0.004651294867207876)  --> TIME: 2025-02-24 20:32:34 -- STEP: 31044/36938 -- GLOBAL_STEP: 474300 | > loss_text_ce: 0.03557753190398216 (0.03793365030917632) | > loss_mel_ce: 3.10587215423584 (nan) | > loss: 0.012466071173548698 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13641671035541808) | > loader_time: 0.0045 (0.004651273330636322)  --> TIME: 2025-02-24 20:32:43 -- STEP: 31094/36938 -- GLOBAL_STEP: 474350 | > loss_text_ce: 0.0364837720990181 (0.037932743061781536) | > loss_mel_ce: 2.257840394973755 (nan) | > loss: 0.009104461409151554 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13641341798227288) | > loader_time: 0.0046 (0.004651275602438669)  --> TIME: 2025-02-24 20:32:52 -- STEP: 31144/36938 -- GLOBAL_STEP: 474400 | > loss_text_ce: 0.0407392643392086 (0.03793143526508747) | > loss_mel_ce: 1.4673129320144653 (nan) | > loss: 0.0059843347407877445 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13641415553628053) | > loader_time: 0.0047 (0.004651283225699567)  --> TIME: 2025-02-24 20:33:02 -- STEP: 31194/36938 -- GLOBAL_STEP: 474450 | > loss_text_ce: 0.044877637177705765 (0.03793332974256334) | > loss_mel_ce: 3.3068933486938477 (nan) | > loss: 0.013300678692758083 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13641876046868498) | > loader_time: 0.0047 (0.0046513549653414915)  --> TIME: 2025-02-24 20:33:11 -- STEP: 31244/36938 -- GLOBAL_STEP: 474500 | > loss_text_ce: 0.04944548010826111 (0.037933608744337134) | > loss_mel_ce: 3.745748281478882 (nan) | > loss: 0.01506029348820448 (nan) | > current_lr: 5e-06 | > step_time: 0.1777 (0.13642400425323634) | > loader_time: 0.0048 (0.004651401675078593)  --> TIME: 2025-02-24 20:33:21 -- STEP: 31294/36938 -- GLOBAL_STEP: 474550 | > loss_text_ce: 0.044527698308229446 (0.03793331266815293) | > loss_mel_ce: 2.840592384338379 (nan) | > loss: 0.011448889970779419 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13642707519838268) | > loader_time: 0.0045 (0.004651324790286045)  --> TIME: 2025-02-24 20:33:30 -- STEP: 31344/36938 -- GLOBAL_STEP: 474600 | > loss_text_ce: 0.044609375298023224 (0.03793571877173609) | > loss_mel_ce: 2.849020481109619 (nan) | > loss: 0.011482658796012402 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13642702388209507) | > loader_time: 0.0045 (0.004651301578942286)  --> TIME: 2025-02-24 20:33:39 -- STEP: 31394/36938 -- GLOBAL_STEP: 474650 | > loss_text_ce: 0.033504366874694824 (0.0379350251404749) | > loss_mel_ce: 2.4881787300109863 (nan) | > loss: 0.010006680153310299 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13641986123481162) | > loader_time: 0.0046 (0.00465137218680203)  --> TIME: 2025-02-24 20:33:48 -- STEP: 31444/36938 -- GLOBAL_STEP: 474700 | > loss_text_ce: 0.033823855221271515 (0.03793478529771062) | > loss_mel_ce: 3.2841577529907227 (nan) | > loss: 0.013166595250368118 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13641790675079515) | > loader_time: 0.0046 (0.004651340132920597)  --> TIME: 2025-02-24 20:33:57 -- STEP: 31494/36938 -- GLOBAL_STEP: 474750 | > loss_text_ce: 0.030918540433049202 (0.03793534126121257) | > loss_mel_ce: 2.454275608062744 (nan) | > loss: 0.009861882776021957 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1364174844075983) | > loader_time: 0.0045 (0.004651343594613595)  --> TIME: 2025-02-24 20:34:07 -- STEP: 31544/36938 -- GLOBAL_STEP: 474800 | > loss_text_ce: 0.04040062427520752 (0.03793733751625342) | > loss_mel_ce: 2.9585840702056885 (nan) | > loss: 0.011900734156370163 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13641271879459393) | > loader_time: 0.0055 (0.004651351754144707)  --> TIME: 2025-02-24 20:34:16 -- STEP: 31594/36938 -- GLOBAL_STEP: 474850 | > loss_text_ce: 0.04263989254832268 (0.03793570239205009) | > loss_mel_ce: 2.2438018321990967 (nan) | > loss: 0.009073182009160519 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13641609333282304) | > loader_time: 0.0046 (0.004651353020694238)  --> TIME: 2025-02-24 20:34:25 -- STEP: 31644/36938 -- GLOBAL_STEP: 474900 | > loss_text_ce: 0.03720797225832939 (0.03793402037444616) | > loss_mel_ce: 3.822542905807495 (nan) | > loss: 0.015316473320126534 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13641695628318531) | > loader_time: 0.0046 (0.0046513800734952334)  --> TIME: 2025-02-24 20:34:35 -- STEP: 31694/36938 -- GLOBAL_STEP: 474950 | > loss_text_ce: 0.04513805732131004 (0.03793468604536876) | > loss_mel_ce: 3.4150197505950928 (nan) | > loss: 0.013730785809457302 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13641298709400299) | > loader_time: 0.0045 (0.004651447918284577)  --> TIME: 2025-02-24 20:34:44 -- STEP: 31744/36938 -- GLOBAL_STEP: 475000 | > loss_text_ce: 0.035130005329847336 (0.03793264418507009) | > loss_mel_ce: 2.813678026199341 (nan) | > loss: 0.011304794810712337 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.1364185777868338) | > loader_time: 0.0046 (0.004651364457282757)  --> TIME: 2025-02-24 20:34:53 -- STEP: 31794/36938 -- GLOBAL_STEP: 475050 | > loss_text_ce: 0.045553211122751236 (0.037930222776728265) | > loss_mel_ce: 3.4981281757354736 (nan) | > loss: 0.014062228612601757 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1364160257667491) | > loader_time: 0.0046 (0.0046514835928778255)  --> TIME: 2025-02-24 20:35:02 -- STEP: 31844/36938 -- GLOBAL_STEP: 475100 | > loss_text_ce: 0.037824612110853195 (0.0379314167875607) | > loss_mel_ce: 3.110823631286621 (nan) | > loss: 0.012494636699557304 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.1364070549335854) | > loader_time: 0.0048 (0.0046515646718761055)  --> TIME: 2025-02-24 20:35:12 -- STEP: 31894/36938 -- GLOBAL_STEP: 475150 | > loss_text_ce: 0.03169998899102211 (0.03793201270510496) | > loss_mel_ce: 2.1489572525024414 (nan) | > loss: 0.008653402328491211 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.1364152202667782) | > loader_time: 0.0066 (0.004651789314778052)  --> TIME: 2025-02-24 20:35:21 -- STEP: 31944/36938 -- GLOBAL_STEP: 475200 | > loss_text_ce: 0.050992969423532486 (0.037932483653639194) | > loss_mel_ce: 0.08556573837995529 (nan) | > loss: 0.0005418997025117278 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1364192614766116) | > loader_time: 0.0045 (0.004651813557235701)  --> TIME: 2025-02-24 20:35:31 -- STEP: 31994/36938 -- GLOBAL_STEP: 475250 | > loss_text_ce: 0.03689763695001602 (0.03793119606640267) | > loss_mel_ce: 3.3460605144500732 (nan) | > loss: 0.013424438424408436 (nan) | > current_lr: 5e-06 | > step_time: 0.1867 (0.13641562559920895) | > loader_time: 0.0049 (0.004651874454720249)  --> TIME: 2025-02-24 20:35:40 -- STEP: 32044/36938 -- GLOBAL_STEP: 475300 | > loss_text_ce: 0.031566787511110306 (0.037929923490061655) | > loss_mel_ce: 3.3560781478881836 (nan) | > loss: 0.013443036936223507 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13641297868568478) | > loader_time: 0.0046 (0.004652019483649327)  --> TIME: 2025-02-24 20:35:49 -- STEP: 32094/36938 -- GLOBAL_STEP: 475350 | > loss_text_ce: 0.033973995596170425 (0.03793051097365735) | > loss_mel_ce: 2.8425400257110596 (nan) | > loss: 0.01141473837196827 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13642081494805747) | > loader_time: 0.0046 (0.004652156520500424)  --> TIME: 2025-02-24 20:35:59 -- STEP: 32144/36938 -- GLOBAL_STEP: 475400 | > loss_text_ce: 0.03263172134757042 (0.037930414681326394) | > loss_mel_ce: 3.260000467300415 (nan) | > loss: 0.013066001236438751 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13641952311238897) | > loader_time: 0.0046 (0.004652238265977868)  --> TIME: 2025-02-24 20:36:08 -- STEP: 32194/36938 -- GLOBAL_STEP: 475450 | > loss_text_ce: 0.050170838832855225 (0.037932390618095994) | > loss_mel_ce: 2.8441617488861084 (nan) | > loss: 0.011485448107123375 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13640990485061852) | > loader_time: 0.0047 (0.00465225655003019)  --> TIME: 2025-02-24 20:36:17 -- STEP: 32244/36938 -- GLOBAL_STEP: 475500 | > loss_text_ce: 0.05049961805343628 (0.03793503282949977) | > loss_mel_ce: 0.1572614461183548 (nan) | > loss: 0.0008244487107731402 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.1364149883816472) | > loader_time: 0.0046 (0.004652346131405521)  --> TIME: 2025-02-24 20:36:26 -- STEP: 32294/36938 -- GLOBAL_STEP: 475550 | > loss_text_ce: 0.030821749940514565 (0.03793716771900035) | > loss_mel_ce: 3.099318504333496 (nan) | > loss: 0.012421192601323128 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.13641823891435018) | > loader_time: 0.0047 (0.004652316373753085)  --> TIME: 2025-02-24 20:36:36 -- STEP: 32344/36938 -- GLOBAL_STEP: 475600 | > loss_text_ce: 0.03644115850329399 (0.03793654799117104) | > loss_mel_ce: 2.2016470432281494 (nan) | > loss: 0.008881302550435066 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13641251705083657) | > loader_time: 0.0045 (0.004652227059231466)  --> TIME: 2025-02-24 20:36:45 -- STEP: 32394/36938 -- GLOBAL_STEP: 475650 | > loss_text_ce: 0.03688148409128189 (0.03793563018770024) | > loss_mel_ce: 3.3571181297302246 (nan) | > loss: 0.013468253426253796 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13640908949270628) | > loader_time: 0.0046 (0.004652177742132937)  --> TIME: 2025-02-24 20:36:54 -- STEP: 32444/36938 -- GLOBAL_STEP: 475700 | > loss_text_ce: 0.0299091637134552 (0.037935684319931676) | > loss_mel_ce: 2.028998613357544 (nan) | > loss: 0.008170269429683685 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13640716623810054) | > loader_time: 0.0047 (0.004652187770163385)  --> TIME: 2025-02-24 20:37:03 -- STEP: 32494/36938 -- GLOBAL_STEP: 475750 | > loss_text_ce: 0.04967161640524864 (0.037935154453109625) | > loss_mel_ce: 2.767058849334717 (nan) | > loss: 0.011177502572536469 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13640679794802552) | > loader_time: 0.0067 (0.004652326140923472)  --> TIME: 2025-02-24 20:37:13 -- STEP: 32544/36938 -- GLOBAL_STEP: 475800 | > loss_text_ce: 0.03800489008426666 (0.03793638552021768) | > loss_mel_ce: 3.0963990688323975 (nan) | > loss: 0.012438111938536167 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13641339969787233) | > loader_time: 0.0046 (0.0046523187452360785)  --> TIME: 2025-02-24 20:37:22 -- STEP: 32594/36938 -- GLOBAL_STEP: 475850 | > loss_text_ce: 0.033840347081422806 (0.03793336618508245) | > loss_mel_ce: 1.15472412109375 (nan) | > loss: 0.004716525785624981 (nan) | > current_lr: 5e-06 | > step_time: 0.1451 (0.13642101255071334) | > loader_time: 0.0049 (0.0046523310271056224)  --> TIME: 2025-02-24 20:37:32 -- STEP: 32644/36938 -- GLOBAL_STEP: 475900 | > loss_text_ce: 0.03427404165267944 (0.03793177975793249) | > loss_mel_ce: 0.00787353515625 (nan) | > loss: 0.00016725229215808213 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13641885533196527) | > loader_time: 0.0047 (0.004652305431422008)  --> TIME: 2025-02-24 20:37:41 -- STEP: 32694/36938 -- GLOBAL_STEP: 475950 | > loss_text_ce: 0.02671934850513935 (0.037932848935526293) | > loss_mel_ce: 2.200791597366333 (nan) | > loss: 0.008839329704642296 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.1364166470575063) | > loader_time: 0.0047 (0.004652360050497496)  --> TIME: 2025-02-24 20:37:50 -- STEP: 32744/36938 -- GLOBAL_STEP: 476000 | > loss_text_ce: 0.03316807001829147 (0.037932659596819844) | > loss_mel_ce: 3.185708522796631 (nan) | > loss: 0.012773321010172367 (nan) | > current_lr: 5e-06 | > step_time: 0.1953 (0.13641689782937017) | > loader_time: 0.0048 (0.004652415114395032)  --> TIME: 2025-02-24 20:37:59 -- STEP: 32794/36938 -- GLOBAL_STEP: 476050 | > loss_text_ce: 0.033083878457546234 (0.03793066976672148) | > loss_mel_ce: 3.112794876098633 (nan) | > loss: 0.012483647093176842 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.1364098812026366) | > loader_time: 0.0046 (0.00465249572504272)  --> TIME: 2025-02-24 20:38:09 -- STEP: 32844/36938 -- GLOBAL_STEP: 476100 | > loss_text_ce: 0.0415409617125988 (0.03792985698726904) | > loss_mel_ce: 2.9273288249969482 (nan) | > loss: 0.011781229637563229 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.13641709296163973) | > loader_time: 0.0046 (0.004652542219195619)  --> TIME: 2025-02-24 20:38:18 -- STEP: 32894/36938 -- GLOBAL_STEP: 476150 | > loss_text_ce: 0.04605216905474663 (0.03793061318107245) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.136423823492369) | > loader_time: 0.0047 (0.004652563776296947)  --> TIME: 2025-02-24 20:38:27 -- STEP: 32944/36938 -- GLOBAL_STEP: 476200 | > loss_text_ce: 0.036902885884046555 (0.0379288218656624) | > loss_mel_ce: 0.10551758110523224 (nan) | > loss: 0.0005651606479659677 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.1364161105932324) | > loader_time: 0.0047 (0.004652612855736589)  --> TIME: 2025-02-24 20:38:37 -- STEP: 32994/36938 -- GLOBAL_STEP: 476250 | > loss_text_ce: 0.04549860954284668 (0.03792511408733867) | > loss_mel_ce: 0.145294189453125 (nan) | > loss: 0.0007571143214590847 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13642269501203527) | > loader_time: 0.0051 (0.004652746368149632)  --> TIME: 2025-02-24 20:38:46 -- STEP: 33044/36938 -- GLOBAL_STEP: 476300 | > loss_text_ce: 0.043300893157720566 (0.037923194619202114) | > loss_mel_ce: 2.924675464630127 (nan) | > loss: 0.011777684092521667 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13641906579720187) | > loader_time: 0.0045 (0.004652793002523943)  --> TIME: 2025-02-24 20:38:56 -- STEP: 33094/36938 -- GLOBAL_STEP: 476350 | > loss_text_ce: 0.03159627690911293 (0.03792170351609253) | > loss_mel_ce: 3.319396495819092 (nan) | > loss: 0.013297590427100658 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13642059339600507) | > loader_time: 0.0046 (0.004652741315999196)  --> TIME: 2025-02-24 20:39:05 -- STEP: 33144/36938 -- GLOBAL_STEP: 476400 | > loss_text_ce: 0.03495535999536514 (0.03791982771581164) | > loss_mel_ce: 2.9346134662628174 (nan) | > loss: 0.011784003116190434 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13642477590681376) | > loader_time: 0.0046 (0.004652770250970898)  --> TIME: 2025-02-24 20:39:14 -- STEP: 33194/36938 -- GLOBAL_STEP: 476450 | > loss_text_ce: 0.03716784715652466 (0.03791765500856747) | > loss_mel_ce: 2.7974331378936768 (nan) | > loss: 0.01124841719865799 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13642527940453128) | > loader_time: 0.0046 (0.004652815834186023)  --> TIME: 2025-02-24 20:39:24 -- STEP: 33244/36938 -- GLOBAL_STEP: 476500 | > loss_text_ce: 0.03037828765809536 (0.03791566663401523) | > loss_mel_ce: 2.9942758083343506 (nan) | > loss: 0.012002596631646156 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13642297712219636) | > loader_time: 0.0046 (0.004652848292194544)  --> TIME: 2025-02-24 20:39:33 -- STEP: 33294/36938 -- GLOBAL_STEP: 476550 | > loss_text_ce: 0.04046434164047241 (0.03791455756649674) | > loss_mel_ce: 1.3368033170700073 (nan) | > loss: 0.005465347785502672 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13642692512210255) | > loader_time: 0.0045 (0.004652874093231259)  --> TIME: 2025-02-24 20:39:42 -- STEP: 33344/36938 -- GLOBAL_STEP: 476600 | > loss_text_ce: 0.03189896047115326 (0.03791530909812311) | > loss_mel_ce: 1.350447177886963 (nan) | > loss: 0.005485501140356064 (nan) | > current_lr: 5e-06 | > step_time: 0.1086 (0.13641898644347866) | > loader_time: 0.0045 (0.004652872645866336)  --> TIME: 2025-02-24 20:39:51 -- STEP: 33394/36938 -- GLOBAL_STEP: 476650 | > loss_text_ce: 0.03495411202311516 (0.03791328681129317) | > loss_mel_ce: 3.5717365741729736 (nan) | > loss: 0.014312265440821648 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1364196969083848) | > loader_time: 0.0046 (0.004652847192484014)  --> TIME: 2025-02-24 20:40:00 -- STEP: 33444/36938 -- GLOBAL_STEP: 476700 | > loss_text_ce: 0.04668405279517174 (0.0379111487176978) | > loss_mel_ce: 3.158113718032837 (nan) | > loss: 0.01271745190024376 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13641233516785803) | > loader_time: 0.0045 (0.004652879110093754)  --> TIME: 2025-02-24 20:40:10 -- STEP: 33494/36938 -- GLOBAL_STEP: 476750 | > loss_text_ce: 0.037217311561107635 (0.03790869485381011) | > loss_mel_ce: 3.446375846862793 (nan) | > loss: 0.01382378302514553 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13641036401017198) | > loader_time: 0.0046 (0.004652839329959175)  --> TIME: 2025-02-24 20:40:19 -- STEP: 33544/36938 -- GLOBAL_STEP: 476800 | > loss_text_ce: 0.032693203538656235 (0.03790898628314886) | > loss_mel_ce: 2.039661169052124 (nan) | > loss: 0.008223628625273705 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13640745745022329) | > loader_time: 0.0065 (0.004652853721995703)  --> TIME: 2025-02-24 20:40:28 -- STEP: 33594/36938 -- GLOBAL_STEP: 476850 | > loss_text_ce: 0.028710821643471718 (0.03791063638670599) | > loss_mel_ce: 3.0200958251953125 (nan) | > loss: 0.01209843996912241 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.1364086271787218) | > loader_time: 0.0046 (0.004652845211566106)  --> TIME: 2025-02-24 20:40:37 -- STEP: 33644/36938 -- GLOBAL_STEP: 476900 | > loss_text_ce: 0.0297652930021286 (0.03790988523930854) | > loss_mel_ce: 3.266094446182251 (nan) | > loss: 0.013078809715807438 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.136399023818594) | > loader_time: 0.0046 (0.00465278038866888)  --> TIME: 2025-02-24 20:40:46 -- STEP: 33694/36938 -- GLOBAL_STEP: 476950 | > loss_text_ce: 0.04013104364275932 (0.037910068111817394) | > loss_mel_ce: 2.448996067047119 (nan) | > loss: 0.009877488948404789 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.1363949274503052) | > loader_time: 0.0045 (0.004652884754149834)  --> TIME: 2025-02-24 20:40:56 -- STEP: 33744/36938 -- GLOBAL_STEP: 477000 | > loss_text_ce: 0.03464947268366814 (0.0379076088900279) | > loss_mel_ce: 2.77939510345459 (nan) | > loss: 0.011166843585669994 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13639875853609806) | > loader_time: 0.0053 (0.004652923842977667)  --> TIME: 2025-02-24 20:41:05 -- STEP: 33794/36938 -- GLOBAL_STEP: 477050 | > loss_text_ce: 0.04191049560904503 (0.037907806877575986) | > loss_mel_ce: 0.034423828125 (nan) | > loss: 0.0003029140061698854 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13640005981356887) | > loader_time: 0.0045 (0.004652904252113604)  --> TIME: 2025-02-24 20:41:14 -- STEP: 33844/36938 -- GLOBAL_STEP: 477100 | > loss_text_ce: 0.03684765100479126 (0.037907223266969596) | > loss_mel_ce: 3.1717331409454346 (nan) | > loss: 0.012732463888823986 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13640164903483468) | > loader_time: 0.0046 (0.004652995827096252)  --> TIME: 2025-02-24 20:41:24 -- STEP: 33894/36938 -- GLOBAL_STEP: 477150 | > loss_text_ce: 0.03745202347636223 (0.037906043051811895) | > loss_mel_ce: 2.540194272994995 (nan) | > loss: 0.010228755883872509 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.13639877450538443) | > loader_time: 0.0048 (0.004653046896037395)  --> TIME: 2025-02-24 20:41:33 -- STEP: 33944/36938 -- GLOBAL_STEP: 477200 | > loss_text_ce: 0.03772599250078201 (0.037905256412702766) | > loss_mel_ce: 2.1756463050842285 (nan) | > loss: 0.008783224038779736 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13640802510077285) | > loader_time: 0.0045 (0.00465304503007057)  --> TIME: 2025-02-24 20:41:43 -- STEP: 33994/36938 -- GLOBAL_STEP: 477250 | > loss_text_ce: 0.04112094268202782 (0.03790879237619588) | > loss_mel_ce: 2.6276917457580566 (nan) | > loss: 0.010590527206659317 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13640438519330841) | > loader_time: 0.0047 (0.0046529369003009005)  --> TIME: 2025-02-24 20:41:52 -- STEP: 34044/36938 -- GLOBAL_STEP: 477300 | > loss_text_ce: 0.0441880077123642 (0.037910758213396935) | > loss_mel_ce: 3.236677885055542 (nan) | > loss: 0.013019310310482979 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.136399389251727) | > loader_time: 0.0045 (0.004652968648877528)  --> TIME: 2025-02-24 20:42:01 -- STEP: 34094/36938 -- GLOBAL_STEP: 477350 | > loss_text_ce: 0.05415286496281624 (0.037911559995604756) | > loss_mel_ce: 3.272522211074829 (nan) | > loss: 0.01320109236985445 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13638885758828828) | > loader_time: 0.0046 (0.004652911731278859)  --> TIME: 2025-02-24 20:42:10 -- STEP: 34144/36938 -- GLOBAL_STEP: 477400 | > loss_text_ce: 0.040248189121484756 (0.03790963017491383) | > loss_mel_ce: 3.387200117111206 (nan) | > loss: 0.013600986450910568 (nan) | > current_lr: 5e-06 | > step_time: 0.1574 (0.13639143129571366) | > loader_time: 0.0047 (0.004652902784197773)  --> TIME: 2025-02-24 20:42:19 -- STEP: 34194/36938 -- GLOBAL_STEP: 477450 | > loss_text_ce: 0.04460977762937546 (0.037910415722366565) | > loss_mel_ce: 3.0529966354370117 (nan) | > loss: 0.012292089872062206 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13639491025680528) | > loader_time: 0.0046 (0.004652919145664267)  --> TIME: 2025-02-24 20:42:28 -- STEP: 34244/36938 -- GLOBAL_STEP: 477500 | > loss_text_ce: 0.03272935375571251 (0.037909381648772805) | > loss_mel_ce: 3.3905866146087646 (nan) | > loss: 0.013584587723016739 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13638655657022267) | > loader_time: 0.0045 (0.004652983659673588)  --> TIME: 2025-02-24 20:42:38 -- STEP: 34294/36938 -- GLOBAL_STEP: 477550 | > loss_text_ce: 0.030982229858636856 (0.037908951437708366) | > loss_mel_ce: 2.4169540405273438 (nan) | > loss: 0.009714033454656601 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.1363885755016674) | > loader_time: 0.0045 (0.004652993257882602)  --> TIME: 2025-02-24 20:42:47 -- STEP: 34344/36938 -- GLOBAL_STEP: 477600 | > loss_text_ce: 0.038617122918367386 (0.037909727283346066) | > loss_mel_ce: 1.5570181608200073 (nan) | > loss: 0.006331886630505323 (nan) | > current_lr: 5e-06 | > step_time: 0.1939 (0.13638218005803524) | > loader_time: 0.0049 (0.0046530477433611795)  --> TIME: 2025-02-24 20:42:56 -- STEP: 34394/36938 -- GLOBAL_STEP: 477650 | > loss_text_ce: 0.03634858876466751 (0.03790971502391901) | > loss_mel_ce: 3.4133336544036865 (nan) | > loss: 0.01368921622633934 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.13638690617109692) | > loader_time: 0.0047 (0.004653090854477696)  --> TIME: 2025-02-24 20:43:06 -- STEP: 34444/36938 -- GLOBAL_STEP: 477700 | > loss_text_ce: 0.03971893712878227 (0.03791056207975827) | > loss_mel_ce: 3.5301318168640137 (nan) | > loss: 0.014166074804961681 (nan) | > current_lr: 5e-06 | > step_time: 0.1699 (0.13639391331068343) | > loader_time: 0.0048 (0.004653214764586679)  --> TIME: 2025-02-24 20:43:15 -- STEP: 34494/36938 -- GLOBAL_STEP: 477750 | > loss_text_ce: 0.03297673538327217 (0.03790979997893775) | > loss_mel_ce: 2.1982009410858154 (nan) | > loss: 0.008853879757225513 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13638525154442926) | > loader_time: 0.0045 (0.004653111923575156)  --> TIME: 2025-02-24 20:43:25 -- STEP: 34544/36938 -- GLOBAL_STEP: 477800 | > loss_text_ce: 0.03976329043507576 (0.03790942212869664) | > loss_mel_ce: 3.16398024559021 (nan) | > loss: 0.012713268399238586 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13639263949525463) | > loader_time: 0.0046 (0.0046531408058928456)  --> TIME: 2025-02-24 20:43:34 -- STEP: 34594/36938 -- GLOBAL_STEP: 477850 | > loss_text_ce: 0.04217871278524399 (0.03790830937037864) | > loss_mel_ce: 3.09926176071167 (nan) | > loss: 0.012466033920645714 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1363930688770239) | > loader_time: 0.0045 (0.004653066853301951)  --> TIME: 2025-02-24 20:43:43 -- STEP: 34644/36938 -- GLOBAL_STEP: 477900 | > loss_text_ce: 0.05443182960152626 (0.03790986778530396) | > loss_mel_ce: 2.70746111869812 (nan) | > loss: 0.010959893465042114 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13639643333095963) | > loader_time: 0.0045 (0.004653052753229981)  --> TIME: 2025-02-24 20:43:53 -- STEP: 34694/36938 -- GLOBAL_STEP: 477950 | > loss_text_ce: 0.028263816609978676 (0.037908088777475465) | > loss_mel_ce: 2.810483455657959 (nan) | > loss: 0.011264870874583721 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13639694634738272) | > loader_time: 0.0045 (0.00465305901442352)  --> TIME: 2025-02-24 20:44:02 -- STEP: 34744/36938 -- GLOBAL_STEP: 478000 | > loss_text_ce: 0.039085861295461655 (0.037907554800165465) | > loss_mel_ce: 2.3184239864349365 (nan) | > loss: 0.009355198591947556 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13639355086659768) | > loader_time: 0.0045 (0.004653076518386139)  --> TIME: 2025-02-24 20:44:11 -- STEP: 34794/36938 -- GLOBAL_STEP: 478050 | > loss_text_ce: 0.03713346645236015 (0.03790713323454955) | > loss_mel_ce: 2.7450928688049316 (nan) | > loss: 0.01104058139026165 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13639467616474937) | > loader_time: 0.0046 (0.004653111287778234)  --> TIME: 2025-02-24 20:44:20 -- STEP: 34844/36938 -- GLOBAL_STEP: 478100 | > loss_text_ce: 0.03506821393966675 (0.037906654609930535) | > loss_mel_ce: 3.4363725185394287 (nan) | > loss: 0.013775559142231941 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.13639517748565078) | > loader_time: 0.0046 (0.004653068972898354)  --> TIME: 2025-02-24 20:44:30 -- STEP: 34894/36938 -- GLOBAL_STEP: 478150 | > loss_text_ce: 0.035254016518592834 (0.03790601824535764) | > loss_mel_ce: 0.009796142578125 (nan) | > loss: 0.00017877048230729997 (nan) | > current_lr: 5e-06 | > step_time: 0.1357 (0.13639876066582587) | > loader_time: 0.0045 (0.004653110725254986)  --> TIME: 2025-02-24 20:44:39 -- STEP: 34944/36938 -- GLOBAL_STEP: 478200 | > loss_text_ce: 0.04104369878768921 (0.037905963561323926) | > loss_mel_ce: 1.4368489980697632 (nan) | > loss: 0.0058646537363529205 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13639793577757345) | > loader_time: 0.0052 (0.004653160136206642)  --> TIME: 2025-02-24 20:44:48 -- STEP: 34994/36938 -- GLOBAL_STEP: 478250 | > loss_text_ce: 0.04408659040927887 (0.03790585398843815) | > loss_mel_ce: 2.6989693641662598 (nan) | > loss: 0.010885143652558327 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13639794230876556) | > loader_time: 0.0046 (0.004653219271368373)  --> TIME: 2025-02-24 20:44:58 -- STEP: 35044/36938 -- GLOBAL_STEP: 478300 | > loss_text_ce: 0.05156604200601578 (0.03790425953181867) | > loss_mel_ce: 3.3105275630950928 (nan) | > loss: 0.01334164198487997 (nan) | > current_lr: 5e-06 | > step_time: 0.1261 (0.13640023548708213) | > loader_time: 0.0046 (0.004653303852610291)  --> TIME: 2025-02-24 20:45:07 -- STEP: 35094/36938 -- GLOBAL_STEP: 478350 | > loss_text_ce: 0.04015112668275833 (0.03790482248814888) | > loss_mel_ce: 2.784966230392456 (nan) | > loss: 0.011210783384740353 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13640402695212214) | > loader_time: 0.0046 (0.004653346771568031)  --> TIME: 2025-02-24 20:45:17 -- STEP: 35144/36938 -- GLOBAL_STEP: 478400 | > loss_text_ce: 0.033593349158763885 (0.03790251786265473) | > loss_mel_ce: 1.5962210893630981 (nan) | > loss: 0.006467517931014299 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1364018198313849) | > loader_time: 0.0045 (0.004653263954971867)  --> TIME: 2025-02-24 20:45:26 -- STEP: 35194/36938 -- GLOBAL_STEP: 478450 | > loss_text_ce: 0.03767533227801323 (0.037902120960907226) | > loss_mel_ce: 3.2570197582244873 (nan) | > loss: 0.013074187561869621 (nan) | > current_lr: 5e-06 | > step_time: 0.1882 (0.13639731912400846) | > loader_time: 0.0048 (0.00465317069722036)  --> TIME: 2025-02-24 20:45:35 -- STEP: 35244/36938 -- GLOBAL_STEP: 478500 | > loss_text_ce: 0.029096469283103943 (0.03789926673656251) | > loss_mel_ce: 3.23294734954834 (nan) | > loss: 0.012944618239998817 (nan) | > current_lr: 5e-06 | > step_time: 0.1971 (0.1363976895301797) | > loader_time: 0.0049 (0.004653162839071429)  --> TIME: 2025-02-24 20:45:44 -- STEP: 35294/36938 -- GLOBAL_STEP: 478550 | > loss_text_ce: 0.034683384001255035 (0.037899630166958194) | > loss_mel_ce: 3.3550567626953125 (nan) | > loss: 0.0134513508528471 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.1363993520850705) | > loader_time: 0.0047 (0.004653201891138749)  --> TIME: 2025-02-24 20:45:53 -- STEP: 35344/36938 -- GLOBAL_STEP: 478600 | > loss_text_ce: 0.05133538693189621 (0.037901296823271) | > loss_mel_ce: 3.293149709701538 (nan) | > loss: 0.01327176671475172 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.1363943404285746) | > loader_time: 0.0045 (0.004653145651463263)  --> TIME: 2025-02-24 20:46:02 -- STEP: 35394/36938 -- GLOBAL_STEP: 478650 | > loss_text_ce: 0.04290561005473137 (0.037900826260682245) | > loss_mel_ce: 3.3649544715881348 (nan) | > loss: 0.013523254543542862 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13638814608309724) | > loader_time: 0.0046 (0.004653089577419977)  --> TIME: 2025-02-24 20:46:12 -- STEP: 35444/36938 -- GLOBAL_STEP: 478700 | > loss_text_ce: 0.029626967385411263 (0.03790112175390564) | > loss_mel_ce: 3.0640339851379395 (nan) | > loss: 0.012276433408260345 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.1363905227933624) | > loader_time: 0.0046 (0.004653037226693792)  --> TIME: 2025-02-24 20:46:21 -- STEP: 35494/36938 -- GLOBAL_STEP: 478750 | > loss_text_ce: 0.030375182628631592 (0.037898626122483084) | > loss_mel_ce: 2.790525197982788 (nan) | > loss: 0.011194050312042236 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13638829021525692) | > loader_time: 0.0045 (0.0046530229484973325)  --> TIME: 2025-02-24 20:46:30 -- STEP: 35544/36938 -- GLOBAL_STEP: 478800 | > loss_text_ce: 0.04403533414006233 (0.037898982919828295) | > loss_mel_ce: 3.4706757068634033 (nan) | > loss: 0.013947267085313797 (nan) | > current_lr: 5e-06 | > step_time: 0.1439 (0.1363825019136232) | > loader_time: 0.0053 (0.004652964003633925)  --> TIME: 2025-02-24 20:46:39 -- STEP: 35594/36938 -- GLOBAL_STEP: 478850 | > loss_text_ce: 0.031655486673116684 (0.037897062458202825) | > loss_mel_ce: 3.2194299697875977 (nan) | > loss: 0.012901133857667446 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13638310768821532) | > loader_time: 0.0045 (0.0046529699699485805)  --> TIME: 2025-02-24 20:46:49 -- STEP: 35644/36938 -- GLOBAL_STEP: 478900 | > loss_text_ce: 0.03463441878557205 (0.03789439033883619) | > loss_mel_ce: 3.411015510559082 (nan) | > loss: 0.013673214241862297 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13638442083181362) | > loader_time: 0.0046 (0.0046528934388967704)  --> TIME: 2025-02-24 20:46:58 -- STEP: 35694/36938 -- GLOBAL_STEP: 478950 | > loss_text_ce: 0.039153803139925 (0.03789371768644694) | > loss_mel_ce: 3.7793240547180176 (nan) | > loss: 0.015152690932154655 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13637740146334124) | > loader_time: 0.0046 (0.004652825244542931)  --> TIME: 2025-02-24 20:47:07 -- STEP: 35744/36938 -- GLOBAL_STEP: 479000 | > loss_text_ce: 0.033910512924194336 (0.0378953229405677) | > loss_mel_ce: 3.550015449523926 (nan) | > loss: 0.014221929013729095 (nan) | > current_lr: 5e-06 | > step_time: 0.1648 (0.13637325062267067) | > loader_time: 0.0046 (0.004652793980274389)  --> TIME: 2025-02-24 20:47:16 -- STEP: 35794/36938 -- GLOBAL_STEP: 479050 | > loss_text_ce: 0.04276799038052559 (0.037896417796331024) | > loss_mel_ce: 2.5874154567718506 (nan) | > loss: 0.010437236167490482 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13637377392241729) | > loader_time: 0.0045 (0.004652692757818698)  --> TIME: 2025-02-24 20:47:25 -- STEP: 35844/36938 -- GLOBAL_STEP: 479100 | > loss_text_ce: 0.038710977882146835 (0.037894892448438064) | > loss_mel_ce: 3.204349994659424 (nan) | > loss: 0.012869291007518768 (nan) | > current_lr: 5e-06 | > step_time: 0.2018 (0.13636687660360777) | > loader_time: 0.0049 (0.0046526424161903905)  --> TIME: 2025-02-24 20:47:34 -- STEP: 35894/36938 -- GLOBAL_STEP: 479150 | > loss_text_ce: 0.03444242104887962 (0.037894480945193244) | > loss_mel_ce: 3.62744140625 (nan) | > loss: 0.014531285502016544 (nan) | > current_lr: 5e-06 | > step_time: 0.1523 (0.1363665324277153) | > loader_time: 0.0046 (0.004652598126456502)  --> TIME: 2025-02-24 20:47:44 -- STEP: 35944/36938 -- GLOBAL_STEP: 479200 | > loss_text_ce: 0.03451506420969963 (0.03789392754560639) | > loss_mel_ce: 2.000471591949463 (nan) | > loss: 0.008075344376266003 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13636206162418762) | > loader_time: 0.0045 (0.004652612005817853)  --> TIME: 2025-02-24 20:47:53 -- STEP: 35994/36938 -- GLOBAL_STEP: 479250 | > loss_text_ce: 0.03381869196891785 (0.03789348994187828) | > loss_mel_ce: 0.0788421630859375 (nan) | > loss: 0.000447066908236593 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13636152100588037) | > loader_time: 0.0044 (0.00465262697929602)  --> TIME: 2025-02-24 20:48:02 -- STEP: 36044/36938 -- GLOBAL_STEP: 479300 | > loss_text_ce: 0.027253849431872368 (0.03789369242256837) | > loss_mel_ce: 3.1625187397003174 (nan) | > loss: 0.012657828629016876 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13635948585122296) | > loader_time: 0.0046 (0.004652619269272242)  --> TIME: 2025-02-24 20:48:12 -- STEP: 36094/36938 -- GLOBAL_STEP: 479350 | > loss_text_ce: 0.043488528579473495 (0.03789122455256485) | > loss_mel_ce: 2.1077077388763428 (nan) | > loss: 0.00853649340569973 (nan) | > current_lr: 5e-06 | > step_time: 0.1556 (0.1363717445595568) | > loader_time: 0.0047 (0.004652744753906184)  --> TIME: 2025-02-24 20:48:21 -- STEP: 36144/36938 -- GLOBAL_STEP: 479400 | > loss_text_ce: 0.043980810791254044 (0.03789033278815795) | > loss_mel_ce: 2.689399003982544 (nan) | > loss: 0.010846746154129505 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13637341131489425) | > loader_time: 0.0046 (0.004652685391369894)  --> TIME: 2025-02-24 20:48:31 -- STEP: 36194/36938 -- GLOBAL_STEP: 479450 | > loss_text_ce: 0.03854789584875107 (0.03789128064588971) | > loss_mel_ce: 3.5764639377593994 (nan) | > loss: 0.014345286414027214 (nan) | > current_lr: 5e-06 | > step_time: 0.1964 (0.13637836003043824) | > loader_time: 0.0048 (0.004652730488622976)  --> TIME: 2025-02-24 20:48:40 -- STEP: 36244/36938 -- GLOBAL_STEP: 479500 | > loss_text_ce: 0.040775638073682785 (0.03789204482577383) | > loss_mel_ce: 2.6585288047790527 (nan) | > loss: 0.010711525566875935 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13638341026897494) | > loader_time: 0.0045 (0.0046526918070769275)  --> TIME: 2025-02-24 20:48:50 -- STEP: 36294/36938 -- GLOBAL_STEP: 479550 | > loss_text_ce: 0.03582457825541496 (0.03789134897327489) | > loss_mel_ce: 2.9325103759765625 (nan) | > loss: 0.01177910715341568 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13638493695744777) | > loader_time: 0.0046 (0.00465271314221672)  --> TIME: 2025-02-24 20:48:59 -- STEP: 36344/36938 -- GLOBAL_STEP: 479600 | > loss_text_ce: 0.027370592579245567 (0.037891043150249576) | > loss_mel_ce: 3.0169544219970703 (nan) | > loss: 0.012080656364560127 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13639211411284935) | > loader_time: 0.0053 (0.004652793045850005)  --> TIME: 2025-02-24 20:49:08 -- STEP: 36394/36938 -- GLOBAL_STEP: 479650 | > loss_text_ce: 0.043643586337566376 (0.037889852776304404) | > loss_mel_ce: 2.14243745803833 (nan) | > loss: 0.008674925193190575 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13638554110319592) | > loader_time: 0.0046 (0.004652942655741186)  --> TIME: 2025-02-24 20:49:17 -- STEP: 36444/36938 -- GLOBAL_STEP: 479700 | > loss_text_ce: 0.036878637969493866 (0.03788827703734685) | > loss_mel_ce: 3.0789380073547363 (nan) | > loss: 0.012364352121949196 (nan) | > current_lr: 5e-06 | > step_time: 0.1819 (0.13637884435008726) | > loader_time: 0.0048 (0.004652972691617238)  --> TIME: 2025-02-24 20:49:27 -- STEP: 36494/36938 -- GLOBAL_STEP: 479750 | > loss_text_ce: 0.039905864745378494 (0.03788792611761617) | > loss_mel_ce: 3.021365165710449 (nan) | > loss: 0.012147901579737663 (nan) | > current_lr: 5e-06 | > step_time: 0.1831 (0.13638055378125608) | > loader_time: 0.0048 (0.004652946388753437)  --> TIME: 2025-02-24 20:49:36 -- STEP: 36544/36938 -- GLOBAL_STEP: 479800 | > loss_text_ce: 0.03841182217001915 (0.03788601240730258) | > loss_mel_ce: 3.241079092025757 (nan) | > loss: 0.013013853691518307 (nan) | > current_lr: 5e-06 | > step_time: 0.1871 (0.13638437515199534) | > loader_time: 0.0048 (0.0046529690955231795)  --> TIME: 2025-02-24 20:49:45 -- STEP: 36594/36938 -- GLOBAL_STEP: 479850 | > loss_text_ce: 0.04962366819381714 (0.03788615261180777) | > loss_mel_ce: 1.9135810136795044 (nan) | > loss: 0.007790495175868273 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13637852845324958) | > loader_time: 0.0046 (0.004652974461834137)  --> TIME: 2025-02-24 20:49:55 -- STEP: 36644/36938 -- GLOBAL_STEP: 479900 | > loss_text_ce: 0.039042308926582336 (0.037886081555929096) | > loss_mel_ce: 3.1052160263061523 (nan) | > loss: 0.012477216310799122 (nan) | > current_lr: 5e-06 | > step_time: 0.1606 (0.1363845387604423) | > loader_time: 0.0047 (0.004653006183724998)  --> TIME: 2025-02-24 20:50:04 -- STEP: 36694/36938 -- GLOBAL_STEP: 479950 | > loss_text_ce: 0.024463335052132607 (0.03788481504068716) | > loss_mel_ce: 3.035996198654175 (nan) | > loss: 0.012144681997597218 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13638751041715547) | > loader_time: 0.0046 (0.004653016591894625)  --> TIME: 2025-02-24 20:50:13 -- STEP: 36744/36938 -- GLOBAL_STEP: 480000 | > loss_text_ce: 0.03266477584838867 (0.037885820968977975) | > loss_mel_ce: 3.2472422122955322 (nan) | > loss: 0.0130155049264431 (nan) | > current_lr: 5e-06 | > step_time: 0.1494 (0.13637956270277538) | > loader_time: 0.0047 (0.004652895044714607) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_480000.pth  --> TIME: 2025-02-24 20:50:29 -- STEP: 36794/36938 -- GLOBAL_STEP: 480050 | > loss_text_ce: 0.04081326723098755 (0.037889062399166715) | > loss_mel_ce: 2.885007858276367 (nan) | > loss: 0.011610401794314384 (nan) | > current_lr: 5e-06 | > step_time: 0.185 (0.1363769691197917) | > loader_time: 0.005 (0.00465303063619429)  --> TIME: 2025-02-24 20:50:38 -- STEP: 36844/36938 -- GLOBAL_STEP: 480100 | > loss_text_ce: 0.04425229877233505 (0.03789018548909499) | > loss_mel_ce: 2.9951233863830566 (nan) | > loss: 0.012061015702784061 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.13637258078942857) | > loader_time: 0.0045 (0.004653086751333794)  --> TIME: 2025-02-24 20:50:47 -- STEP: 36894/36938 -- GLOBAL_STEP: 480150 | > loss_text_ce: 0.03363505005836487 (0.037890348314594625) | > loss_mel_ce: 2.4317829608917236 (nan) | > loss: 0.009783405810594559 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.1363678309512511) | > loader_time: 0.0045 (0.004653202115458637)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.02066935997198124 (-1.1501926006667745e-05) | > avg_loss_text_ce: 0.03687541921724483 (-0.0006340918232603729) | > avg_loss_mel_ce: 2.6770039554869776 (-0.028212306818159227) | > avg_loss: 2.71387938166609 (-0.028846396668120278) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_480194.pth  > EPOCH: 14/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-24 20:51:20)   --> TIME: 2025-02-24 20:51:21 -- STEP: 6/36938 -- GLOBAL_STEP: 480200 | > loss_text_ce: 0.04218185693025589 (0.03381369914859533) | > loss_mel_ce: 3.4776337146759033 (2.3599254599151513) | > loss: 0.013967523351311684 (0.009498965528716022) | > current_lr: 5e-06 | > step_time: 0.1357 (0.12163734436035156) | > loader_time: 0.0042 (0.004339456558227539)  --> TIME: 2025-02-24 20:51:30 -- STEP: 56/36938 -- GLOBAL_STEP: 480250 | > loss_text_ce: 0.031047679483890533 (0.03605178480834835) | > loss_mel_ce: 2.9947102069854736 (2.6828212650393004) | > loss: 0.012006975710391998 (0.010789179379311722) | > current_lr: 5e-06 | > step_time: 0.1352 (0.1299049513680595) | > loader_time: 0.0046 (0.004648029804229736)  --> TIME: 2025-02-24 20:51:39 -- STEP: 106/36938 -- GLOBAL_STEP: 480300 | > loss_text_ce: 0.0376715213060379 (0.03684181589984669) | > loss_mel_ce: 2.2459778785705566 (2.6630991512364797) | > loss: 0.009062101133167744 (0.01071405209126478) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13134069037887286) | > loader_time: 0.0047 (0.0046410785531098)  --> TIME: 2025-02-24 20:51:48 -- STEP: 156/36938 -- GLOBAL_STEP: 480350 | > loss_text_ce: 0.04101792350411415 (0.03732654882165101) | > loss_mel_ce: 0.1053873673081398 (2.607621740257272) | > loss: 0.0005809733993373811 (0.010495827158126466) | > current_lr: 5e-06 | > step_time: 0.1092 (0.1317306130360334) | > loader_time: 0.0045 (0.004649561185103197)  --> TIME: 2025-02-24 20:51:57 -- STEP: 206/36938 -- GLOBAL_STEP: 480400 | > loss_text_ce: 0.0342443585395813 (0.037352882705411844) | > loss_mel_ce: 3.4039857387542725 (2.641243106672016) | > loss: 0.013643770478665829 (0.010629349778823386) | > current_lr: 5e-06 | > step_time: 0.1764 (0.13247222576326545) | > loader_time: 0.0069 (0.004656165548898639)  --> TIME: 2025-02-24 20:52:07 -- STEP: 256/36938 -- GLOBAL_STEP: 480450 | > loss_text_ce: 0.03517000749707222 (0.037290508538717376) | > loss_mel_ce: 3.457023859024048 (2.640853406741371) | > loss: 0.013857913203537464 (0.010627555833025328) | > current_lr: 5e-06 | > step_time: 0.1687 (0.13336131814867264) | > loader_time: 0.0049 (0.004645575769245622)  --> TIME: 2025-02-24 20:52:16 -- STEP: 306/36938 -- GLOBAL_STEP: 480500 | > loss_text_ce: 0.043027155101299286 (0.03747773150383841) | > loss_mel_ce: 3.0159530639648438 (2.6312079566753073) | > loss: 0.012138810940086842 (0.010590023177202635) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13332644085478942) | > loader_time: 0.0047 (0.004648477423424812)  --> TIME: 2025-02-24 20:52:25 -- STEP: 356/36938 -- GLOBAL_STEP: 480550 | > loss_text_ce: 0.04224380850791931 (0.03748858825753581) | > loss_mel_ce: 3.639303684234619 (2.641444187971053) | > loss: 0.014609316363930702 (0.010630686238788729) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13364661409613798) | > loader_time: 0.0047 (0.004662844572174413)  --> TIME: 2025-02-24 20:52:34 -- STEP: 406/36938 -- GLOBAL_STEP: 480600 | > loss_text_ce: 0.042308978736400604 (0.037486339088878044) | > loss_mel_ce: 3.214303970336914 (2.6396232400344126) | > loss: 0.01292306836694479 (0.010623451327890584) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13395625206050035) | > loader_time: 0.0047 (0.004663950116763561)  --> TIME: 2025-02-24 20:52:44 -- STEP: 456/36938 -- GLOBAL_STEP: 480650 | > loss_text_ce: 0.01613997481763363 (0.03735235045876419) | > loss_mel_ce: 2.1878209114074707 (2.647497330370498) | > loss: 0.00874587707221508 (0.010654166018436008) | > current_lr: 5e-06 | > step_time: 0.112 (0.1343955909996702) | > loader_time: 0.0046 (0.004665555661184748)  --> TIME: 2025-02-24 20:52:53 -- STEP: 506/36938 -- GLOBAL_STEP: 480700 | > loss_text_ce: 0.04627705365419388 (0.03739913704293401) | > loss_mel_ce: 3.282660484313965 (nan) | > loss: 0.013210070319473743 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13467156557226376) | > loader_time: 0.0045 (0.004664422495091859)  --> TIME: 2025-02-24 20:53:02 -- STEP: 556/36938 -- GLOBAL_STEP: 480750 | > loss_text_ce: 0.031664006412029266 (0.03726210372636375) | > loss_mel_ce: 3.05822491645813 (nan) | > loss: 0.012261464260518551 (nan) | > current_lr: 5e-06 | > step_time: 0.1918 (0.13438945837158098) | > loader_time: 0.0049 (0.004671553484827496)  --> TIME: 2025-02-24 20:53:12 -- STEP: 606/36938 -- GLOBAL_STEP: 480800 | > loss_text_ce: 0.028481118381023407 (0.037165216337766455) | > loss_mel_ce: 3.2110142707824707 (nan) | > loss: 0.01285514049232006 (nan) | > current_lr: 5e-06 | > step_time: 0.1902 (0.13492468520753076) | > loader_time: 0.0076 (0.004681228017649638)  --> TIME: 2025-02-24 20:53:21 -- STEP: 656/36938 -- GLOBAL_STEP: 480850 | > loss_text_ce: 0.038885001093149185 (0.03709028265155031) | > loss_mel_ce: 2.1393017768859863 (nan) | > loss: 0.008643599227070808 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.134799211853888) | > loader_time: 0.0046 (0.004677191740129054)  --> TIME: 2025-02-24 20:53:30 -- STEP: 706/36938 -- GLOBAL_STEP: 480900 | > loss_text_ce: 0.03485647961497307 (0.037068790861698245) | > loss_mel_ce: 2.9422426223754883 (nan) | > loss: 0.01181388646364212 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.13458400103593357) | > loader_time: 0.0047 (0.004674487343590934)  --> TIME: 2025-02-24 20:53:39 -- STEP: 756/36938 -- GLOBAL_STEP: 480950 | > loss_text_ce: 0.03733633831143379 (0.03702976033042265) | > loss_mel_ce: 2.9930226802825928 (nan) | > loss: 0.01202523522078991 (nan) | > current_lr: 5e-06 | > step_time: 0.1083 (0.13483746594222135) | > loader_time: 0.0045 (0.004672948960904724)  --> TIME: 2025-02-24 20:53:49 -- STEP: 806/36938 -- GLOBAL_STEP: 481000 | > loss_text_ce: 0.03630330413579941 (0.037035828144301085) | > loss_mel_ce: 2.3206284046173096 (nan) | > loss: 0.009352903813123703 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13480544149431936) | > loader_time: 0.0065 (0.004668241399097685)  --> TIME: 2025-02-24 20:53:58 -- STEP: 856/36938 -- GLOBAL_STEP: 481050 | > loss_text_ce: 0.03418375551700592 (0.03698666671848041) | > loss_mel_ce: 1.4555768966674805 (nan) | > loss: 0.005911748856306076 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13492609463005417) | > loader_time: 0.0046 (0.0046699189137075605)  --> TIME: 2025-02-24 20:54:07 -- STEP: 906/36938 -- GLOBAL_STEP: 481100 | > loss_text_ce: 0.03680124506354332 (0.036982875072683) | > loss_mel_ce: 3.2355380058288574 (nan) | > loss: 0.012985474430024624 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13460099802375108) | > loader_time: 0.0046 (0.004665726857469577)  --> TIME: 2025-02-24 20:54:16 -- STEP: 956/36938 -- GLOBAL_STEP: 481150 | > loss_text_ce: 0.0368693545460701 (0.037082777025419615) | > loss_mel_ce: 3.507662296295166 (nan) | > loss: 0.014065602794289589 (nan) | > current_lr: 5e-06 | > step_time: 0.1452 (0.1344480462154085) | > loader_time: 0.0046 (0.004663469152969302)  --> TIME: 2025-02-24 20:54:25 -- STEP: 1006/36938 -- GLOBAL_STEP: 481200 | > loss_text_ce: 0.035192929208278656 (0.03707122498527016) | > loss_mel_ce: 3.533907413482666 (nan) | > loss: 0.014163097366690636 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13434874679648853) | > loader_time: 0.0066 (0.004663983822816888)  --> TIME: 2025-02-24 20:54:34 -- STEP: 1056/36938 -- GLOBAL_STEP: 481250 | > loss_text_ce: 0.02825252152979374 (0.03702423146618948) | > loss_mel_ce: 2.1543538570404053 (nan) | > loss: 0.008661136962473392 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13443717789469342) | > loader_time: 0.0045 (0.004661534545999589)  --> TIME: 2025-02-24 20:54:43 -- STEP: 1106/36938 -- GLOBAL_STEP: 481300 | > loss_text_ce: 0.025301434099674225 (0.03705411142826237) | > loss_mel_ce: 2.9544126987457275 (nan) | > loss: 0.011824263259768486 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13451103965800665) | > loader_time: 0.0045 (0.004658749694203383)  --> TIME: 2025-02-24 20:54:53 -- STEP: 1156/36938 -- GLOBAL_STEP: 481350 | > loss_text_ce: 0.04307643696665764 (0.03708175954064124) | > loss_mel_ce: 2.8167166709899902 (nan) | > loss: 0.01134838629513979 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13456549528973324) | > loader_time: 0.0048 (0.004656505007232347)  --> TIME: 2025-02-24 20:55:02 -- STEP: 1206/36938 -- GLOBAL_STEP: 481400 | > loss_text_ce: 0.04254719614982605 (0.03715677648801261) | > loss_mel_ce: 2.1618850231170654 (nan) | > loss: 0.008747747167944908 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13463330031627443) | > loader_time: 0.0046 (0.004655816068696739)  --> TIME: 2025-02-24 20:55:11 -- STEP: 1256/36938 -- GLOBAL_STEP: 481450 | > loss_text_ce: 0.029431447386741638 (0.03715890403214105) | > loss_mel_ce: 2.0342559814453125 (nan) | > loss: 0.00818923581391573 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13466633201404735) | > loader_time: 0.0046 (0.00465570418698013)  --> TIME: 2025-02-24 20:55:21 -- STEP: 1306/36938 -- GLOBAL_STEP: 481500 | > loss_text_ce: 0.04082841798663139 (0.03720736903228312) | > loss_mel_ce: 2.982837200164795 (nan) | > loss: 0.011998673900961876 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.1347640566584892) | > loader_time: 0.0045 (0.0046581988312749)  --> TIME: 2025-02-24 20:55:30 -- STEP: 1356/36938 -- GLOBAL_STEP: 481550 | > loss_text_ce: 0.03037954680621624 (0.03721315950528889) | > loss_mel_ce: 1.7548072338104248 (nan) | > loss: 0.007084074895828962 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1347233132626807) | > loader_time: 0.0045 (0.00465669920310862)  --> TIME: 2025-02-24 20:55:39 -- STEP: 1406/36938 -- GLOBAL_STEP: 481600 | > loss_text_ce: 0.03968086466193199 (0.03729311125631186) | > loss_mel_ce: 3.5892388820648193 (nan) | > loss: 0.014400476589798927 (nan) | > current_lr: 5e-06 | > step_time: 0.1811 (0.134939895754688) | > loader_time: 0.0049 (0.004656218024100554)  --> TIME: 2025-02-24 20:55:49 -- STEP: 1456/36938 -- GLOBAL_STEP: 481650 | > loss_text_ce: 0.030027097091078758 (0.03731370512856644) | > loss_mel_ce: 3.3547496795654297 (nan) | > loss: 0.013431655243039131 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.13490712544420272) | > loader_time: 0.0047 (0.004655887464900597)  --> TIME: 2025-02-24 20:55:58 -- STEP: 1506/36938 -- GLOBAL_STEP: 481700 | > loss_text_ce: 0.03954395279288292 (0.03731367470149896) | > loss_mel_ce: 3.4145655632019043 (nan) | > loss: 0.013706784695386887 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.13493535559807493) | > loader_time: 0.0047 (0.004657076174519454)  --> TIME: 2025-02-24 20:56:07 -- STEP: 1556/36938 -- GLOBAL_STEP: 481750 | > loss_text_ce: 0.050122737884521484 (0.03732389797980882) | > loss_mel_ce: 2.0379085540771484 (nan) | > loss: 0.00828583911061287 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13504687985908106) | > loader_time: 0.0046 (0.004657931530077233)  --> TIME: 2025-02-24 20:56:17 -- STEP: 1606/36938 -- GLOBAL_STEP: 481800 | > loss_text_ce: 0.04474564269185066 (0.037347630932254475) | > loss_mel_ce: 1.6084914207458496 (nan) | > loss: 0.006560464855283499 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13505241612568616) | > loader_time: 0.0048 (0.004659156573664949)  --> TIME: 2025-02-24 20:56:26 -- STEP: 1656/36938 -- GLOBAL_STEP: 481850 | > loss_text_ce: 0.0485299676656723 (0.0373385350185037) | > loss_mel_ce: 3.282200336456299 (nan) | > loss: 0.013217183761298656 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13514510704123495) | > loader_time: 0.0045 (0.004658323123259245)  --> TIME: 2025-02-24 20:56:35 -- STEP: 1706/36938 -- GLOBAL_STEP: 481900 | > loss_text_ce: 0.05693522468209267 (0.037303529918337956) | > loss_mel_ce: 0.01677227020263672 (nan) | > loss: 0.0002924900618381798 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13514070424217464) | > loader_time: 0.0045 (0.004658173145191331)  --> TIME: 2025-02-24 20:56:45 -- STEP: 1756/36938 -- GLOBAL_STEP: 481950 | > loss_text_ce: 0.031823184341192245 (0.03731379881426949) | > loss_mel_ce: 3.0195443630218506 (nan) | > loss: 0.01210860162973404 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13522892986723445) | > loader_time: 0.0046 (0.004656503982587391)  --> TIME: 2025-02-24 20:56:54 -- STEP: 1806/36938 -- GLOBAL_STEP: 482000 | > loss_text_ce: 0.025341972708702087 (0.03730313394967456) | > loss_mel_ce: 0.009422438219189644 (nan) | > loss: 0.00013795401901006699 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.1352933092106754) | > loader_time: 0.0045 (0.004656145606986178)  --> TIME: 2025-02-24 20:57:03 -- STEP: 1856/36938 -- GLOBAL_STEP: 482050 | > loss_text_ce: 0.03598835691809654 (0.03728084130652235) | > loss_mel_ce: 2.054208517074585 (nan) | > loss: 0.008294432424008846 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13534594545590517) | > loader_time: 0.0045 (0.004656497902911283)  --> TIME: 2025-02-24 20:57:13 -- STEP: 1906/36938 -- GLOBAL_STEP: 482100 | > loss_text_ce: 0.03648219630122185 (0.037277085025934575) | > loss_mel_ce: 2.7977774143218994 (nan) | > loss: 0.011247062124311924 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13532588451881897) | > loader_time: 0.0046 (0.004656406914950166)  --> TIME: 2025-02-24 20:57:22 -- STEP: 1956/36938 -- GLOBAL_STEP: 482150 | > loss_text_ce: 0.028869913890957832 (0.03724333631295659) | > loss_mel_ce: 0.098968505859375 (nan) | > loss: 0.0005072953645139933 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13536119400114172) | > loader_time: 0.0047 (0.004655980259362903)  --> TIME: 2025-02-24 20:57:31 -- STEP: 2006/36938 -- GLOBAL_STEP: 482200 | > loss_text_ce: 0.036837734282016754 (0.03719031821206541) | > loss_mel_ce: 2.8070101737976074 (nan) | > loss: 0.01128511130809784 (nan) | > current_lr: 5e-06 | > step_time: 0.1441 (0.13524858794683042) | > loader_time: 0.0046 (0.004655082584734807)  --> TIME: 2025-02-24 20:57:40 -- STEP: 2056/36938 -- GLOBAL_STEP: 482250 | > loss_text_ce: 0.03583839163184166 (0.037184318605944504) | > loss_mel_ce: 1.4895248413085938 (nan) | > loss: 0.006053029093891382 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.1353384521916687) | > loader_time: 0.0048 (0.004655081938212946)  --> TIME: 2025-02-24 20:57:50 -- STEP: 2106/36938 -- GLOBAL_STEP: 482300 | > loss_text_ce: 0.043601471930742264 (0.037203137751789) | > loss_mel_ce: 2.509014844894409 (nan) | > loss: 0.010129430331289768 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.1353417982516243) | > loader_time: 0.0046 (0.004654742034412176)  --> TIME: 2025-02-24 20:57:59 -- STEP: 2156/36938 -- GLOBAL_STEP: 482350 | > loss_text_ce: 0.04641326144337654 (0.03722597647446223) | > loss_mel_ce: 2.3429126739501953 (nan) | > loss: 0.009481452405452728 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.13520575730831635) | > loader_time: 0.0047 (0.004654233774139176)  --> TIME: 2025-02-24 20:58:08 -- STEP: 2206/36938 -- GLOBAL_STEP: 482400 | > loss_text_ce: 0.03373698517680168 (0.03722834708979686) | > loss_mel_ce: 2.813910961151123 (nan) | > loss: 0.011300190351903439 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.1352763879547746) | > loader_time: 0.0046 (0.004654052575285176)  --> TIME: 2025-02-24 20:58:17 -- STEP: 2256/36938 -- GLOBAL_STEP: 482450 | > loss_text_ce: 0.03183931112289429 (0.03720319751404059) | > loss_mel_ce: 2.338139295578003 (nan) | > loss: 0.009404677897691727 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.1353431418009687) | > loader_time: 0.0046 (0.004652830725866005)  --> TIME: 2025-02-24 20:58:27 -- STEP: 2306/36938 -- GLOBAL_STEP: 482500 | > loss_text_ce: 0.03924740478396416 (0.03720918401279342) | > loss_mel_ce: 2.9306511878967285 (nan) | > loss: 0.011785312555730343 (nan) | > current_lr: 5e-06 | > step_time: 0.1519 (0.13538027016473494) | > loader_time: 0.0047 (0.004653974191478716)  --> TIME: 2025-02-24 20:58:36 -- STEP: 2356/36938 -- GLOBAL_STEP: 482550 | > loss_text_ce: 0.03972085937857628 (0.037207482906425324) | > loss_mel_ce: 3.0742106437683105 (nan) | > loss: 0.012356871739029884 (nan) | > current_lr: 5e-06 | > step_time: 0.1769 (0.1354264953950066) | > loader_time: 0.0048 (0.004653315086720951)  --> TIME: 2025-02-24 20:58:46 -- STEP: 2406/36938 -- GLOBAL_STEP: 482600 | > loss_text_ce: 0.036106206476688385 (0.03718250648003209) | > loss_mel_ce: 2.6453118324279785 (nan) | > loss: 0.01064054761081934 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.1354370569053139) | > loader_time: 0.006 (0.004656427518982536)  --> TIME: 2025-02-24 20:58:55 -- STEP: 2456/36938 -- GLOBAL_STEP: 482650 | > loss_text_ce: 0.049438245594501495 (0.037194502179310716) | > loss_mel_ce: 3.1052684783935547 (nan) | > loss: 0.012518677860498428 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13553636722533668) | > loader_time: 0.0046 (0.004655696864236833)  --> TIME: 2025-02-24 20:59:05 -- STEP: 2506/36938 -- GLOBAL_STEP: 482700 | > loss_text_ce: 0.0313933864235878 (0.0371999031870053) | > loss_mel_ce: 2.9563310146331787 (nan) | > loss: 0.011856049299240112 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13565517898949286) | > loader_time: 0.0048 (0.004658043336982444)  --> TIME: 2025-02-24 20:59:14 -- STEP: 2556/36938 -- GLOBAL_STEP: 482750 | > loss_text_ce: 0.03218352422118187 (0.03719717362882494) | > loss_mel_ce: 3.0668275356292725 (nan) | > loss: 0.012297662906348705 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13568483393330366) | > loader_time: 0.0047 (0.004665534261246795)  --> TIME: 2025-02-24 20:59:24 -- STEP: 2606/36938 -- GLOBAL_STEP: 482800 | > loss_text_ce: 0.040440309792757034 (0.037209084465119155) | > loss_mel_ce: 3.071431875228882 (nan) | > loss: 0.012348700314760208 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13578914031557934) | > loader_time: 0.0047 (0.004669512132452889)  --> TIME: 2025-02-24 20:59:33 -- STEP: 2656/36938 -- GLOBAL_STEP: 482850 | > loss_text_ce: 0.02761540748178959 (0.03723126874166458) | > loss_mel_ce: 2.5681958198547363 (nan) | > loss: 0.010300838388502598 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13590810043984164) | > loader_time: 0.0046 (0.004669725356331787)  --> TIME: 2025-02-24 20:59:42 -- STEP: 2706/36938 -- GLOBAL_STEP: 482900 | > loss_text_ce: 0.027224957942962646 (0.037240305885532204) | > loss_mel_ce: 2.2265655994415283 (nan) | > loss: 0.00894361361861229 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13587522436227992) | > loader_time: 0.0047 (0.004670928252333634)  --> TIME: 2025-02-24 20:59:52 -- STEP: 2756/36938 -- GLOBAL_STEP: 482950 | > loss_text_ce: 0.03705201670527458 (0.03726388715479272) | > loss_mel_ce: 2.943619728088379 (nan) | > loss: 0.011828063055872917 (nan) | > current_lr: 5e-06 | > step_time: 0.1609 (0.13589490887388642) | > loader_time: 0.0048 (0.0046698656933401046)  --> TIME: 2025-02-24 21:00:01 -- STEP: 2806/36938 -- GLOBAL_STEP: 483000 | > loss_text_ce: 0.038852136582136154 (0.03727044096596261) | > loss_mel_ce: 2.737565279006958 (nan) | > loss: 0.011017530225217342 (nan) | > current_lr: 5e-06 | > step_time: 0.1624 (0.1359542626953946) | > loader_time: 0.0048 (0.004670230134078963)  --> TIME: 2025-02-24 21:00:11 -- STEP: 2856/36938 -- GLOBAL_STEP: 483050 | > loss_text_ce: 0.03779977560043335 (0.03727382520528079) | > loss_mel_ce: 3.12412166595459 (nan) | > loss: 0.012547308579087257 (nan) | > current_lr: 5e-06 | > step_time: 0.1798 (0.13599969258829378) | > loader_time: 0.0049 (0.004670417191935512)  --> TIME: 2025-02-24 21:00:20 -- STEP: 2906/36938 -- GLOBAL_STEP: 483100 | > loss_text_ce: 0.03604745492339134 (0.037251192690831526) | > loss_mel_ce: 2.89176869392395 (nan) | > loss: 0.01161831896752119 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.135960259686974) | > loader_time: 0.0047 (0.004670255937333437)  --> TIME: 2025-02-24 21:00:29 -- STEP: 2956/36938 -- GLOBAL_STEP: 483150 | > loss_text_ce: 0.03391240909695625 (0.03726674970056665) | > loss_mel_ce: 2.627037525177002 (nan) | > loss: 0.01055932603776455 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13600515012650816) | > loader_time: 0.0048 (0.00467065246889168)  --> TIME: 2025-02-24 21:00:38 -- STEP: 3006/36938 -- GLOBAL_STEP: 483200 | > loss_text_ce: 0.03928772360086441 (0.037275438757423694) | > loss_mel_ce: 2.6704392433166504 (nan) | > loss: 0.010752885602414608 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13597364562079717) | > loader_time: 0.0048 (0.004670598311814474)  --> TIME: 2025-02-24 21:00:48 -- STEP: 3056/36938 -- GLOBAL_STEP: 483250 | > loss_text_ce: 0.03899124637246132 (0.037287681111492985) | > loss_mel_ce: 3.413363218307495 (nan) | > loss: 0.013699820265173912 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.135909521532933) | > loader_time: 0.0045 (0.004671034154467552)  --> TIME: 2025-02-24 21:00:57 -- STEP: 3106/36938 -- GLOBAL_STEP: 483300 | > loss_text_ce: 0.04977067932486534 (0.03730028965741309) | > loss_mel_ce: 3.112292528152466 (nan) | > loss: 0.01254787016659975 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13595439960936656) | > loader_time: 0.0046 (0.004671567421303354)  --> TIME: 2025-02-24 21:01:06 -- STEP: 3156/36938 -- GLOBAL_STEP: 483350 | > loss_text_ce: 0.034407876431941986 (0.03732536695761835) | > loss_mel_ce: 2.27554988861084 (nan) | > loss: 0.009166499599814415 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13588400545833407) | > loader_time: 0.0045 (0.004671099899084089)  --> TIME: 2025-02-24 21:01:15 -- STEP: 3206/36938 -- GLOBAL_STEP: 483400 | > loss_text_ce: 0.04730323329567909 (0.03731577779606209) | > loss_mel_ce: 0.023223876953125 (nan) | > loss: 0.0002798694767989218 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13591941386108042) | > loader_time: 0.0046 (0.0046724382966886885)  --> TIME: 2025-02-24 21:01:24 -- STEP: 3256/36938 -- GLOBAL_STEP: 483450 | > loss_text_ce: 0.03899303451180458 (0.037315356955035184) | > loss_mel_ce: 2.6149704456329346 (nan) | > loss: 0.010531602427363396 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13587070587811978) | > loader_time: 0.0046 (0.004672649597945898)  --> TIME: 2025-02-24 21:01:33 -- STEP: 3306/36938 -- GLOBAL_STEP: 483500 | > loss_text_ce: 0.028834080323576927 (0.03728613886487522) | > loss_mel_ce: 2.6940929889678955 (nan) | > loss: 0.010805266909301281 (nan) | > current_lr: 5e-06 | > step_time: 0.1831 (0.135813174792946) | > loader_time: 0.0048 (0.0046734296402928255)  --> TIME: 2025-02-24 21:01:42 -- STEP: 3356/36938 -- GLOBAL_STEP: 483550 | > loss_text_ce: 0.038742952048778534 (0.037275609412398165) | > loss_mel_ce: 3.1328320503234863 (nan) | > loss: 0.012585615739226341 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13573684887039258) | > loader_time: 0.0047 (0.0046753100473633435)  --> TIME: 2025-02-24 21:01:52 -- STEP: 3406/36938 -- GLOBAL_STEP: 483600 | > loss_text_ce: 0.03626498207449913 (0.03728256815458202) | > loss_mel_ce: 2.830918550491333 (nan) | > loss: 0.011377712711691856 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13575314621468992) | > loader_time: 0.0048 (0.004675586854998524)  --> TIME: 2025-02-24 21:02:01 -- STEP: 3456/36938 -- GLOBAL_STEP: 483650 | > loss_text_ce: 0.03928106278181076 (0.03727324851341244) | > loss_mel_ce: 2.31368350982666 (nan) | > loss: 0.009337161667644978 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.1357464299157818) | > loader_time: 0.0046 (0.004675244774531427)  --> TIME: 2025-02-24 21:02:10 -- STEP: 3506/36938 -- GLOBAL_STEP: 483700 | > loss_text_ce: 0.040753431618213654 (0.03725872121733245) | > loss_mel_ce: 2.316934585571289 (nan) | > loss: 0.009355905465781689 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13573410051316492) | > loader_time: 0.0048 (0.004675395952519585)  --> TIME: 2025-02-24 21:02:19 -- STEP: 3556/36938 -- GLOBAL_STEP: 483750 | > loss_text_ce: 0.04523717612028122 (0.03727603540808189) | > loss_mel_ce: 2.2656447887420654 (nan) | > loss: 0.009170166216790676 (nan) | > current_lr: 5e-06 | > step_time: 0.1082 (0.13568834103937258) | > loader_time: 0.0045 (0.004674779886067643)  --> TIME: 2025-02-24 21:02:28 -- STEP: 3606/36938 -- GLOBAL_STEP: 483800 | > loss_text_ce: 0.05692550912499428 (0.037296163283662455) | > loss_mel_ce: 3.0941030979156494 (nan) | > loss: 0.012504082173109055 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13562665082187858) | > loader_time: 0.0046 (0.0046748722915310814)  --> TIME: 2025-02-24 21:02:37 -- STEP: 3656/36938 -- GLOBAL_STEP: 483850 | > loss_text_ce: 0.02776271477341652 (0.03729216630455449) | > loss_mel_ce: 2.2177281379699707 (nan) | > loss: 0.008910678327083588 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1355667909203788) | > loader_time: 0.0046 (0.00467496621269626)  --> TIME: 2025-02-24 21:02:47 -- STEP: 3706/36938 -- GLOBAL_STEP: 483900 | > loss_text_ce: 0.03434429690241814 (0.0372627517019981) | > loss_mel_ce: 3.347503185272217 (nan) | > loss: 0.01342003047466278 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.1356557932533064) | > loader_time: 0.0046 (0.004676105200247183)  --> TIME: 2025-02-24 21:02:56 -- STEP: 3756/36938 -- GLOBAL_STEP: 483950 | > loss_text_ce: 0.04392160847783089 (0.037284964954686826) | > loss_mel_ce: 3.4301834106445312 (nan) | > loss: 0.01378613244742155 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13566779753905406) | > loader_time: 0.0046 (0.004677534865114247)  --> TIME: 2025-02-24 21:03:06 -- STEP: 3806/36938 -- GLOBAL_STEP: 484000 | > loss_text_ce: 0.023906445130705833 (0.037297461963201425) | > loss_mel_ce: 2.5319907665252686 (nan) | > loss: 0.010142450220882893 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13567093022547466) | > loader_time: 0.0046 (0.004677791126890671)  --> TIME: 2025-02-24 21:03:15 -- STEP: 3856/36938 -- GLOBAL_STEP: 484050 | > loss_text_ce: 0.03811047226190567 (0.037297713290851686) | > loss_mel_ce: 1.4309779405593872 (nan) | > loss: 0.005829716566950083 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.1356294838348366) | > loader_time: 0.0049 (0.004677638226030271)  --> TIME: 2025-02-24 21:03:24 -- STEP: 3906/36938 -- GLOBAL_STEP: 484100 | > loss_text_ce: 0.03802342712879181 (0.037298170654880454) | > loss_mel_ce: 3.0509636402130127 (nan) | > loss: 0.012257886119186878 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13564747379672154) | > loader_time: 0.0046 (0.004677483196815214)  --> TIME: 2025-02-24 21:03:33 -- STEP: 3956/36938 -- GLOBAL_STEP: 484150 | > loss_text_ce: 0.036851897835731506 (0.03729626038526086) | > loss_mel_ce: 3.374152183532715 (nan) | > loss: 0.013535731472074986 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13552524069563335) | > loader_time: 0.0046 (0.004676580368809553)  --> TIME: 2025-02-24 21:03:43 -- STEP: 4006/36938 -- GLOBAL_STEP: 484200 | > loss_text_ce: 0.03652967885136604 (0.03730725353635769) | > loss_mel_ce: 2.53725266456604 (nan) | > loss: 0.01021342258900404 (nan) | > current_lr: 5e-06 | > step_time: 0.1851 (0.13559632377510253) | > loader_time: 0.0048 (0.004676545255492456)  --> TIME: 2025-02-24 21:03:52 -- STEP: 4056/36938 -- GLOBAL_STEP: 484250 | > loss_text_ce: 0.0340852290391922 (0.03729709296756382) | > loss_mel_ce: 3.20430850982666 (nan) | > loss: 0.01285076979547739 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.13559173885181822) | > loader_time: 0.0046 (0.004676327079945995)  --> TIME: 2025-02-24 21:04:01 -- STEP: 4106/36938 -- GLOBAL_STEP: 484300 | > loss_text_ce: 0.035065095871686935 (0.03730392363308024) | > loss_mel_ce: 3.54110050201416 (nan) | > loss: 0.014191134832799435 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.1355687041312849) | > loader_time: 0.0046 (0.0046770258525842745)  --> TIME: 2025-02-24 21:04:10 -- STEP: 4156/36938 -- GLOBAL_STEP: 484350 | > loss_text_ce: 0.03390439972281456 (0.03730225387249855) | > loss_mel_ce: 2.971454381942749 (nan) | > loss: 0.01192602701485157 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13559918115641553) | > loader_time: 0.0048 (0.004676942416862061)  --> TIME: 2025-02-24 21:04:20 -- STEP: 4206/36938 -- GLOBAL_STEP: 484400 | > loss_text_ce: 0.04365668073296547 (0.03729574312090597) | > loss_mel_ce: 3.6286075115203857 (nan) | > loss: 0.014572476968169212 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13563769752504268) | > loader_time: 0.0046 (0.0046764638839990064)  --> TIME: 2025-02-24 21:04:29 -- STEP: 4256/36938 -- GLOBAL_STEP: 484450 | > loss_text_ce: 0.044776782393455505 (0.037310810783515606) | > loss_mel_ce: 1.9860801696777344 (nan) | > loss: 0.008058955892920494 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13560388741412568) | > loader_time: 0.0046 (0.004676078663284608)  --> TIME: 2025-02-24 21:04:38 -- STEP: 4306/36938 -- GLOBAL_STEP: 484500 | > loss_text_ce: 0.0411631241440773 (0.03732005658650775) | > loss_mel_ce: 3.201246976852417 (nan) | > loss: 0.012866707518696785 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13557732575890974) | > loader_time: 0.0049 (0.004675697793087947)  --> TIME: 2025-02-24 21:04:48 -- STEP: 4356/36938 -- GLOBAL_STEP: 484550 | > loss_text_ce: 0.04511994123458862 (0.037334105350321764) | > loss_mel_ce: 3.008887529373169 (nan) | > loss: 0.012119078077375889 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13560296002607816) | > loader_time: 0.0045 (0.004676497781681718)  --> TIME: 2025-02-24 21:04:57 -- STEP: 4406/36938 -- GLOBAL_STEP: 484600 | > loss_text_ce: 0.030599424615502357 (0.037340743291135556) | > loss_mel_ce: 3.671782970428467 (nan) | > loss: 0.014691994525492191 (nan) | > current_lr: 5e-06 | > step_time: 0.1455 (0.13559941145922463) | > loader_time: 0.0046 (0.004675957651177256)  --> TIME: 2025-02-24 21:05:06 -- STEP: 4456/36938 -- GLOBAL_STEP: 484650 | > loss_text_ce: 0.04143848270177841 (0.03733894995374431) | > loss_mel_ce: 2.95817232131958 (nan) | > loss: 0.011903218924999237 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1356494054332029) | > loader_time: 0.0046 (0.004676650788240404)  --> TIME: 2025-02-24 21:05:16 -- STEP: 4506/36938 -- GLOBAL_STEP: 484700 | > loss_text_ce: 0.032931700348854065 (0.03734754682388334) | > loss_mel_ce: 2.7451891899108887 (nan) | > loss: 0.011024290695786476 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13567400973371244) | > loader_time: 0.0045 (0.004676550745805303)  --> TIME: 2025-02-24 21:05:25 -- STEP: 4556/36938 -- GLOBAL_STEP: 484750 | > loss_text_ce: 0.02934907376766205 (0.037353928963170545) | > loss_mel_ce: 2.47332763671875 (nan) | > loss: 0.009931256994605064 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13571434396730392) | > loader_time: 0.0046 (0.004676568759335447)  --> TIME: 2025-02-24 21:05:35 -- STEP: 4606/36938 -- GLOBAL_STEP: 484800 | > loss_text_ce: 0.03901590034365654 (0.037353905646354504) | > loss_mel_ce: 2.452340602874756 (nan) | > loss: 0.009886336512863636 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13573724738008) | > loader_time: 0.0047 (0.004676364527233145)  --> TIME: 2025-02-24 21:05:44 -- STEP: 4656/36938 -- GLOBAL_STEP: 484850 | > loss_text_ce: 0.03301234170794487 (0.03734372391127841) | > loss_mel_ce: 1.6131103038787842 (nan) | > loss: 0.0065322332084178925 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13574658523720176) | > loader_time: 0.0045 (0.0046762476364771415)  --> TIME: 2025-02-24 21:05:53 -- STEP: 4706/36938 -- GLOBAL_STEP: 484900 | > loss_text_ce: 0.037578485906124115 (0.03736297439572456) | > loss_mel_ce: 3.5458731651306152 (nan) | > loss: 0.014220047742128372 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.13574475508469833) | > loader_time: 0.0049 (0.004675852102368009)  --> TIME: 2025-02-24 21:06:02 -- STEP: 4756/36938 -- GLOBAL_STEP: 484950 | > loss_text_ce: 0.037466857582330704 (0.037374412238517986) | > loss_mel_ce: 3.1070027351379395 (nan) | > loss: 0.012478054501116276 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.1357179731255927) | > loader_time: 0.0048 (0.004675513911989173)  --> TIME: 2025-02-24 21:06:12 -- STEP: 4806/36938 -- GLOBAL_STEP: 485000 | > loss_text_ce: 0.033916838467121124 (0.03739440443786425) | > loss_mel_ce: 2.242750883102417 (nan) | > loss: 0.009034397080540657 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13576439659247255) | > loader_time: 0.0045 (0.004675497971025539)  --> TIME: 2025-02-24 21:06:21 -- STEP: 4856/36938 -- GLOBAL_STEP: 485050 | > loss_text_ce: 0.027458176016807556 (0.037389441212843315) | > loss_mel_ce: 2.5174522399902344 (nan) | > loss: 0.010098851285874844 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1357756183410595) | > loader_time: 0.0045 (0.004674866083622755)  --> TIME: 2025-02-24 21:06:31 -- STEP: 4906/36938 -- GLOBAL_STEP: 485100 | > loss_text_ce: 0.02676633931696415 (0.03739461689352953) | > loss_mel_ce: 2.703125 (nan) | > loss: 0.01083290297538042 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.1358027169036711) | > loader_time: 0.0046 (0.0046742585450835926)  --> TIME: 2025-02-24 21:06:40 -- STEP: 4956/36938 -- GLOBAL_STEP: 485150 | > loss_text_ce: 0.03846573457121849 (0.03739396733896326) | > loss_mel_ce: 2.29050350189209 (nan) | > loss: 0.009241942316293716 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13580358567210918) | > loader_time: 0.0045 (0.004673656674524378)  --> TIME: 2025-02-24 21:06:49 -- STEP: 5006/36938 -- GLOBAL_STEP: 485200 | > loss_text_ce: 0.034497931599617004 (0.037392153565578796) | > loss_mel_ce: 3.1846110820770264 (nan) | > loss: 0.012774243019521236 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.13583423474098852) | > loader_time: 0.0048 (0.004673095402887132)  --> TIME: 2025-02-24 21:06:58 -- STEP: 5056/36938 -- GLOBAL_STEP: 485250 | > loss_text_ce: 0.028825625777244568 (0.0374091676398788) | > loss_mel_ce: 3.2787086963653564 (nan) | > loss: 0.013125136494636536 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13579978878739518) | > loader_time: 0.0044 (0.004672369011972516)  --> TIME: 2025-02-24 21:07:08 -- STEP: 5106/36938 -- GLOBAL_STEP: 485300 | > loss_text_ce: 0.03475327789783478 (0.03740514051191049) | > loss_mel_ce: 3.1156914234161377 (nan) | > loss: 0.012501765973865986 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.1358562851717927) | > loader_time: 0.0046 (0.004672302529244987)  --> TIME: 2025-02-24 21:07:17 -- STEP: 5156/36938 -- GLOBAL_STEP: 485350 | > loss_text_ce: 0.03726254031062126 (0.037413075853672415) | > loss_mel_ce: 3.1886565685272217 (nan) | > loss: 0.01280126627534628 (nan) | > current_lr: 5e-06 | > step_time: 0.1517 (0.13588467510843374) | > loader_time: 0.0047 (0.004671762672110609)  --> TIME: 2025-02-24 21:07:27 -- STEP: 5206/36938 -- GLOBAL_STEP: 485400 | > loss_text_ce: 0.03234289214015007 (0.0373987018240203) | > loss_mel_ce: 2.851043939590454 (nan) | > loss: 0.011442012153565884 (nan) | > current_lr: 5e-06 | > step_time: 0.1582 (0.1359034945678123) | > loader_time: 0.0049 (0.004671078483004127)  --> TIME: 2025-02-24 21:07:36 -- STEP: 5256/36938 -- GLOBAL_STEP: 485450 | > loss_text_ce: 0.03173590078949928 (0.037411447375399025) | > loss_mel_ce: 3.164515972137451 (nan) | > loss: 0.012683539651334286 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13589960980814506) | > loader_time: 0.0046 (0.004670533914362813)  --> TIME: 2025-02-24 21:07:45 -- STEP: 5306/36938 -- GLOBAL_STEP: 485500 | > loss_text_ce: 0.04032165929675102 (0.03740935107269017) | > loss_mel_ce: 0.48310261964797974 (nan) | > loss: 0.0020770805422216654 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13586390643042498) | > loader_time: 0.0045 (0.004669847463240404)  --> TIME: 2025-02-24 21:07:54 -- STEP: 5356/36938 -- GLOBAL_STEP: 485550 | > loss_text_ce: 0.034321289509534836 (0.03742187494350167) | > loss_mel_ce: 1.6109760999679565 (nan) | > loss: 0.006528958212584257 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13587897352712885) | > loader_time: 0.0045 (0.004669321037389338)  --> TIME: 2025-02-24 21:08:03 -- STEP: 5406/36938 -- GLOBAL_STEP: 485600 | > loss_text_ce: 0.040470197796821594 (0.037420215648662755) | > loss_mel_ce: 2.789182186126709 (nan) | > loss: 0.011228779330849648 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.1358577677200687) | > loader_time: 0.0046 (0.004668751029319063)  --> TIME: 2025-02-24 21:08:13 -- STEP: 5456/36938 -- GLOBAL_STEP: 485650 | > loss_text_ce: 0.0319388173520565 (0.03741843544805007) | > loss_mel_ce: 3.239849805831909 (nan) | > loss: 0.012983288615942001 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13587981943161234) | > loader_time: 0.0047 (0.004669039523846231)  --> TIME: 2025-02-24 21:08:23 -- STEP: 5506/36938 -- GLOBAL_STEP: 485700 | > loss_text_ce: 0.036493223160505295 (0.037418930491082754) | > loss_mel_ce: 2.0948615074157715 (nan) | > loss: 0.008457757532596588 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.135963415909021) | > loader_time: 0.0047 (0.004669274237646941)  --> TIME: 2025-02-24 21:08:32 -- STEP: 5556/36938 -- GLOBAL_STEP: 485750 | > loss_text_ce: 0.025387652218341827 (0.03743380630040866) | > loss_mel_ce: 3.0214507579803467 (nan) | > loss: 0.012090629898011684 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.1359430539513591) | > loader_time: 0.0047 (0.004668917091224062)  --> TIME: 2025-02-24 21:08:41 -- STEP: 5606/36938 -- GLOBAL_STEP: 485800 | > loss_text_ce: 0.030989734455943108 (0.03745177065783497) | > loss_mel_ce: 2.8742895126342773 (nan) | > loss: 0.011528885923326015 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.13595874403954541) | > loader_time: 0.0048 (0.004668735411606884)  --> TIME: 2025-02-24 21:08:51 -- STEP: 5656/36938 -- GLOBAL_STEP: 485850 | > loss_text_ce: 0.032589152455329895 (0.037453436474987646) | > loss_mel_ce: 3.244450807571411 (nan) | > loss: 0.013004127889871597 (nan) | > current_lr: 5e-06 | > step_time: 0.1261 (0.13596297781571937) | > loader_time: 0.0047 (0.004668433772455339)  --> TIME: 2025-02-24 21:09:00 -- STEP: 5706/36938 -- GLOBAL_STEP: 485900 | > loss_text_ce: 0.0452030748128891 (0.03744885323163092) | > loss_mel_ce: 2.777398109436035 (nan) | > loss: 0.0112007986754179 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13595257092057014) | > loader_time: 0.0046 (0.004668330460985286)  --> TIME: 2025-02-24 21:09:09 -- STEP: 5756/36938 -- GLOBAL_STEP: 485950 | > loss_text_ce: 0.03774179145693779 (0.03746139834236285) | > loss_mel_ce: 1.9413305521011353 (nan) | > loss: 0.007853462360799313 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.1359464599491405) | > loader_time: 0.0047 (0.004668235530283316)  --> TIME: 2025-02-24 21:09:18 -- STEP: 5806/36938 -- GLOBAL_STEP: 486000 | > loss_text_ce: 0.041429996490478516 (0.03747294146052359) | > loss_mel_ce: 3.536837100982666 (nan) | > loss: 0.014199472963809967 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13592094677791378) | > loader_time: 0.0047 (0.004668160959888315)  --> TIME: 2025-02-24 21:09:28 -- STEP: 5856/36938 -- GLOBAL_STEP: 486050 | > loss_text_ce: 0.03571625053882599 (0.037461624175909376) | > loss_mel_ce: 2.95481538772583 (nan) | > loss: 0.011867189779877663 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13599857704235532) | > loader_time: 0.0047 (0.0046685990658614125)  --> TIME: 2025-02-24 21:09:37 -- STEP: 5906/36938 -- GLOBAL_STEP: 486100 | > loss_text_ce: 0.04209878668189049 (0.03746746936932843) | > loss_mel_ce: 2.187401294708252 (nan) | > loss: 0.008847222663462162 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.1359892803412388) | > loader_time: 0.0046 (0.004668258869805413)  --> TIME: 2025-02-24 21:09:47 -- STEP: 5956/36938 -- GLOBAL_STEP: 486150 | > loss_text_ce: 0.04193204268813133 (0.03746765763267419) | > loss_mel_ce: 3.3196163177490234 (nan) | > loss: 0.013339478522539139 (nan) | > current_lr: 5e-06 | > step_time: 0.1574 (0.13599853154554514) | > loader_time: 0.0047 (0.0046680725365376725)  --> TIME: 2025-02-24 21:09:56 -- STEP: 6006/36938 -- GLOBAL_STEP: 486200 | > loss_text_ce: 0.028318963944911957 (0.03745479823629453) | > loss_mel_ce: 2.6637890338897705 (nan) | > loss: 0.0106829684227705 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13604353207014308) | > loader_time: 0.0046 (0.004667933924849963)  --> TIME: 2025-02-24 21:10:05 -- STEP: 6056/36938 -- GLOBAL_STEP: 486250 | > loss_text_ce: 0.027542978525161743 (0.03745229741114917) | > loss_mel_ce: 1.8004721403121948 (nan) | > loss: 0.00725402869284153 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13604501818886167) | > loader_time: 0.0045 (0.0046676662276884)  --> TIME: 2025-02-24 21:10:15 -- STEP: 6106/36938 -- GLOBAL_STEP: 486300 | > loss_text_ce: 0.04021970555186272 (0.03744383722978721) | > loss_mel_ce: 3.3795559406280518 (nan) | > loss: 0.013570539653301239 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.13607016357093207) | > loader_time: 0.005 (0.004667553205080901)  --> TIME: 2025-02-24 21:10:24 -- STEP: 6156/36938 -- GLOBAL_STEP: 486350 | > loss_text_ce: 0.03221870958805084 (0.03744418500248487) | > loss_mel_ce: 2.3452577590942383 (nan) | > loss: 0.009434430859982967 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13607032787807538) | > loader_time: 0.0045 (0.004666999495594897)  --> TIME: 2025-02-24 21:10:33 -- STEP: 6206/36938 -- GLOBAL_STEP: 486400 | > loss_text_ce: 0.0377979539334774 (0.037433622907811125) | > loss_mel_ce: 3.1401379108428955 (nan) | > loss: 0.012610857374966145 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13604238914590508) | > loader_time: 0.0047 (0.00466674337685204)  --> TIME: 2025-02-24 21:10:43 -- STEP: 6256/36938 -- GLOBAL_STEP: 486450 | > loss_text_ce: 0.03687894344329834 (0.03743517324652953) | > loss_mel_ce: 3.0460879802703857 (nan) | > loss: 0.012233995832502842 (nan) | > current_lr: 5e-06 | > step_time: 0.1438 (0.1360567930485582) | > loader_time: 0.0046 (0.004667066018599683)  --> TIME: 2025-02-24 21:10:52 -- STEP: 6306/36938 -- GLOBAL_STEP: 486500 | > loss_text_ce: 0.033957090228796005 (0.037437815780231816) | > loss_mel_ce: 3.149970769882202 (nan) | > loss: 0.012634634971618652 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13603525781041625) | > loader_time: 0.0046 (0.004666908634983091)  --> TIME: 2025-02-24 21:11:01 -- STEP: 6356/36938 -- GLOBAL_STEP: 486550 | > loss_text_ce: 0.03953234478831291 (0.03743472142872394) | > loss_mel_ce: 3.0678505897521973 (nan) | > loss: 0.012330885976552963 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.13606785861405996) | > loader_time: 0.0048 (0.0046667548153218885)  --> TIME: 2025-02-24 21:11:11 -- STEP: 6406/36938 -- GLOBAL_STEP: 486600 | > loss_text_ce: 0.04095430299639702 (0.03742254109904307) | > loss_mel_ce: 2.2352240085601807 (nan) | > loss: 0.00903245434165001 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13605309786961783) | > loader_time: 0.0046 (0.004667988390688819)  --> TIME: 2025-02-24 21:11:20 -- STEP: 6456/36938 -- GLOBAL_STEP: 486650 | > loss_text_ce: 0.04124050587415695 (0.03741450973496743) | > loss_mel_ce: 2.53462815284729 (nan) | > loss: 0.010221701115369797 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13609836613378515) | > loader_time: 0.0046 (0.0046685865008107225)  --> TIME: 2025-02-24 21:11:30 -- STEP: 6506/36938 -- GLOBAL_STEP: 486700 | > loss_text_ce: 0.04370428994297981 (0.03742294840410443) | > loss_mel_ce: 2.6226072311401367 (nan) | > loss: 0.01058060210198164 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13610664800684333) | > loader_time: 0.0046 (0.00466919641585633)  --> TIME: 2025-02-24 21:11:39 -- STEP: 6556/36938 -- GLOBAL_STEP: 486750 | > loss_text_ce: 0.04896240308880806 (0.03742089527053955) | > loss_mel_ce: 1.0029197931289673 (nan) | > loss: 0.004174135625362396 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13608863842494553) | > loader_time: 0.0047 (0.004669290879210581)  --> TIME: 2025-02-24 21:11:48 -- STEP: 6606/36938 -- GLOBAL_STEP: 486800 | > loss_text_ce: 0.028150657191872597 (0.037427548670426775) | > loss_mel_ce: 2.6119213104248047 (nan) | > loss: 0.010476476512849331 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13610256945332513) | > loader_time: 0.0046 (0.004669345583729474)  --> TIME: 2025-02-24 21:11:58 -- STEP: 6656/36938 -- GLOBAL_STEP: 486850 | > loss_text_ce: 0.05922107398509979 (0.03744291683520155) | > loss_mel_ce: 3.46875 (nan) | > loss: 0.013999885879456997 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13612865402291544) | > loader_time: 0.0045 (0.004670371982054064)  --> TIME: 2025-02-24 21:12:07 -- STEP: 6706/36938 -- GLOBAL_STEP: 486900 | > loss_text_ce: 0.03919876366853714 (0.03744953640789688) | > loss_mel_ce: 3.056208372116089 (nan) | > loss: 0.012283362448215485 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.13612614554716027) | > loader_time: 0.0047 (0.004671113476126076)  --> TIME: 2025-02-24 21:12:16 -- STEP: 6756/36938 -- GLOBAL_STEP: 486950 | > loss_text_ce: 0.026736607775092125 (0.037452701835403966) | > loss_mel_ce: 2.7570254802703857 (nan) | > loss: 0.01104667503386736 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13610090222705912) | > loader_time: 0.0046 (0.004670759994931439)  --> TIME: 2025-02-24 21:12:25 -- STEP: 6806/36938 -- GLOBAL_STEP: 487000 | > loss_text_ce: 0.04999449476599693 (0.03746533517123946) | > loss_mel_ce: 2.2559118270874023 (nan) | > loss: 0.009150422178208828 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13608896028214287) | > loader_time: 0.0047 (0.004670614569880061)  --> TIME: 2025-02-24 21:12:35 -- STEP: 6856/36938 -- GLOBAL_STEP: 487050 | > loss_text_ce: 0.04476862773299217 (0.037454738511781036) | > loss_mel_ce: 2.6140520572662354 (nan) | > loss: 0.010550876148045063 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.1361280456446151) | > loader_time: 0.0046 (0.0046706705624748315)  --> TIME: 2025-02-24 21:12:44 -- STEP: 6906/36938 -- GLOBAL_STEP: 487100 | > loss_text_ce: 0.030322136357426643 (0.0374510376038536) | > loss_mel_ce: 2.5674514770507812 (nan) | > loss: 0.010308626107871532 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13612810691266725) | > loader_time: 0.0045 (0.004670631668031154)  --> TIME: 2025-02-24 21:12:54 -- STEP: 6956/36938 -- GLOBAL_STEP: 487150 | > loss_text_ce: 0.04685774818062782 (0.03744401630360144) | > loss_mel_ce: 3.6244914531707764 (nan) | > loss: 0.014568847604095936 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13616007710408) | > loader_time: 0.0046 (0.00467089070473133)  --> TIME: 2025-02-24 21:13:03 -- STEP: 7006/36938 -- GLOBAL_STEP: 487200 | > loss_text_ce: 0.031659629195928574 (0.03744221426431717) | > loss_mel_ce: 3.2309410572052 (nan) | > loss: 0.012946829199790955 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13614950498580347) | > loader_time: 0.0048 (0.004671511056591289)  --> TIME: 2025-02-24 21:13:12 -- STEP: 7056/36938 -- GLOBAL_STEP: 487250 | > loss_text_ce: 0.03878910839557648 (0.03744102931262546) | > loss_mel_ce: 3.380509853363037 (nan) | > loss: 0.01356864720582962 (nan) | > current_lr: 5e-06 | > step_time: 0.1266 (0.1361474885083645) | > loader_time: 0.0046 (0.00467179938643967)  --> TIME: 2025-02-24 21:13:22 -- STEP: 7106/36938 -- GLOBAL_STEP: 487300 | > loss_text_ce: 0.03834538534283638 (0.037457668991555455) | > loss_mel_ce: 3.1437973976135254 (nan) | > loss: 0.012627551332116127 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13614254644679996) | > loader_time: 0.0046 (0.004671917778990006)  --> TIME: 2025-02-24 21:13:31 -- STEP: 7156/36938 -- GLOBAL_STEP: 487350 | > loss_text_ce: 0.0376017689704895 (0.03745698180963798) | > loss_mel_ce: 3.1882102489471436 (nan) | > loss: 0.012800841592252254 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.1361244754660651) | > loader_time: 0.0045 (0.004671592130015978)  --> TIME: 2025-02-24 21:13:40 -- STEP: 7206/36938 -- GLOBAL_STEP: 487400 | > loss_text_ce: 0.056350935250520706 (0.037450183059140295) | > loss_mel_ce: 1.4681305885314941 (nan) | > loss: 0.006049530114978552 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.1361213042078958) | > loader_time: 0.0045 (0.00467192795817268)  --> TIME: 2025-02-24 21:13:49 -- STEP: 7256/36938 -- GLOBAL_STEP: 487450 | > loss_text_ce: 0.04091167822480202 (0.03745779347792881) | > loss_mel_ce: 3.5325961112976074 (nan) | > loss: 0.014180587604641914 (nan) | > current_lr: 5e-06 | > step_time: 0.1794 (0.1361193967102113) | > loader_time: 0.0047 (0.00467200033457039)  --> TIME: 2025-02-24 21:13:59 -- STEP: 7306/36938 -- GLOBAL_STEP: 487500 | > loss_text_ce: 0.05016027018427849 (0.037458636399226926) | > loss_mel_ce: 2.5348074436187744 (nan) | > loss: 0.010257808491587639 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13612529283805982) | > loader_time: 0.0047 (0.0046717658814357825)  --> TIME: 2025-02-24 21:14:08 -- STEP: 7356/36938 -- GLOBAL_STEP: 487550 | > loss_text_ce: 0.030815808102488518 (0.03745378101347877) | > loss_mel_ce: 2.826282024383545 (nan) | > loss: 0.011337690986692905 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13610784634096929) | > loader_time: 0.0045 (0.004671838732113188)  --> TIME: 2025-02-24 21:14:17 -- STEP: 7406/36938 -- GLOBAL_STEP: 487600 | > loss_text_ce: 0.04234122484922409 (0.03745830950108154) | > loss_mel_ce: 1.9277496337890625 (nan) | > loss: 0.007817821577191353 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1360991288609286) | > loader_time: 0.0045 (0.004671352314620661)  --> TIME: 2025-02-24 21:14:26 -- STEP: 7456/36938 -- GLOBAL_STEP: 487650 | > loss_text_ce: 0.026806838810443878 (0.0374630962559941) | > loss_mel_ce: 2.5179202556610107 (nan) | > loss: 0.010098123922944069 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13607689447285246) | > loader_time: 0.0047 (0.004670991854094631)  --> TIME: 2025-02-24 21:14:36 -- STEP: 7506/36938 -- GLOBAL_STEP: 487700 | > loss_text_ce: 0.03546794503927231 (0.03746233272740687) | > loss_mel_ce: 3.3148982524871826 (nan) | > loss: 0.01329510472714901 (nan) | > current_lr: 5e-06 | > step_time: 0.1618 (0.13609830754425148) | > loader_time: 0.0047 (0.004670571048450943)  --> TIME: 2025-02-24 21:14:45 -- STEP: 7556/36938 -- GLOBAL_STEP: 487750 | > loss_text_ce: 0.043673329055309296 (0.03745543910067204) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13607261936900567) | > loader_time: 0.0045 (0.004669953963944611)  --> TIME: 2025-02-24 21:14:54 -- STEP: 7606/36938 -- GLOBAL_STEP: 487800 | > loss_text_ce: 0.04065905138850212 (0.03746830855302515) | > loss_mel_ce: 2.676801919937134 (nan) | > loss: 0.010783575475215912 (nan) | > current_lr: 5e-06 | > step_time: 0.179 (0.13607385777311165) | > loader_time: 0.0048 (0.004669679173537994)  --> TIME: 2025-02-24 21:15:03 -- STEP: 7656/36938 -- GLOBAL_STEP: 487850 | > loss_text_ce: 0.028381871059536934 (0.037464266334676866) | > loss_mel_ce: 0.1788853257894516 (nan) | > loss: 0.0008224889170378447 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13608020612563174) | > loader_time: 0.0045 (0.0046694129861136)  --> TIME: 2025-02-24 21:15:13 -- STEP: 7706/36938 -- GLOBAL_STEP: 487900 | > loss_text_ce: 0.030904630199074745 (0.037471367530147626) | > loss_mel_ce: 3.3553547859191895 (nan) | > loss: 0.013437537476420403 (nan) | > current_lr: 5e-06 | > step_time: 0.1938 (0.1360878860551349) | > loader_time: 0.0048 (0.004669468216607693)  --> TIME: 2025-02-24 21:15:22 -- STEP: 7756/36938 -- GLOBAL_STEP: 487950 | > loss_text_ce: 0.03728169947862625 (0.03746830415083433) | > loss_mel_ce: 3.0121517181396484 (nan) | > loss: 0.012100927531719208 (nan) | > current_lr: 5e-06 | > step_time: 0.1762 (0.1360897797170894) | > loader_time: 0.0047 (0.0046693469028364605)  --> TIME: 2025-02-24 21:15:31 -- STEP: 7806/36938 -- GLOBAL_STEP: 488000 | > loss_text_ce: 0.03141710162162781 (0.03747206928421791) | > loss_mel_ce: 2.9936702251434326 (nan) | > loss: 0.012004315853118896 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13609562358763316) | > loader_time: 0.0046 (0.004669379430522123)  --> TIME: 2025-02-24 21:15:41 -- STEP: 7856/36938 -- GLOBAL_STEP: 488050 | > loss_text_ce: 0.043603673577308655 (0.037480717680896856) | > loss_mel_ce: 3.935978889465332 (nan) | > loss: 0.0157919954508543 (nan) | > current_lr: 5e-06 | > step_time: 0.1954 (0.13612901311663614) | > loader_time: 0.0048 (0.004669139195848134)  --> TIME: 2025-02-24 21:15:50 -- STEP: 7906/36938 -- GLOBAL_STEP: 488100 | > loss_text_ce: 0.0359935387969017 (0.03748214875729344) | > loss_mel_ce: 3.1704111099243164 (nan) | > loss: 0.012723828665912151 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13613195564666605) | > loader_time: 0.0049 (0.004668829171469979)  --> TIME: 2025-02-24 21:16:00 -- STEP: 7956/36938 -- GLOBAL_STEP: 488150 | > loss_text_ce: 0.034129854291677475 (0.0374865475875808) | > loss_mel_ce: 2.800675868988037 (nan) | > loss: 0.011249229311943054 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13616286335787348) | > loader_time: 0.0046 (0.004668914504600197)  --> TIME: 2025-02-24 21:16:09 -- STEP: 8006/36938 -- GLOBAL_STEP: 488200 | > loss_text_ce: 0.03565659746527672 (0.0374860568151209) | > loss_mel_ce: 3.3500800132751465 (nan) | > loss: 0.013435463421046734 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.1361457599629883) | > loader_time: 0.0046 (0.004668836054014549)  --> TIME: 2025-02-24 21:16:18 -- STEP: 8056/36938 -- GLOBAL_STEP: 488250 | > loss_text_ce: 0.04382992163300514 (0.03749410017178962) | > loss_mel_ce: 2.716430425643921 (nan) | > loss: 0.010953414253890514 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.13612746026333622) | > loader_time: 0.0048 (0.004668264811656909)  --> TIME: 2025-02-24 21:16:28 -- STEP: 8106/36938 -- GLOBAL_STEP: 488300 | > loss_text_ce: 0.03282882645726204 (0.03749258472850048) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1361409056019021) | > loader_time: 0.0045 (0.004668738675535143)  --> TIME: 2025-02-24 21:16:37 -- STEP: 8156/36938 -- GLOBAL_STEP: 488350 | > loss_text_ce: 0.038895606994628906 (0.037501067210668454) | > loss_mel_ce: 3.303906202316284 (nan) | > loss: 0.013265087269246578 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13613626458821396) | > loader_time: 0.0045 (0.004668591740436555)  --> TIME: 2025-02-24 21:16:46 -- STEP: 8206/36938 -- GLOBAL_STEP: 488400 | > loss_text_ce: 0.047135766595602036 (0.037505456781214065) | > loss_mel_ce: 2.6377108097076416 (nan) | > loss: 0.010654154233634472 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.1361236487136882) | > loader_time: 0.0047 (0.004668306641947072)  --> TIME: 2025-02-24 21:16:55 -- STEP: 8256/36938 -- GLOBAL_STEP: 488450 | > loss_text_ce: 0.031370338052511215 (0.0375065704387551) | > loss_mel_ce: 3.170776844024658 (nan) | > loss: 0.012706934474408627 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.1361331648316029) | > loader_time: 0.0046 (0.0046684654184089985)  --> TIME: 2025-02-24 21:17:05 -- STEP: 8306/36938 -- GLOBAL_STEP: 488500 | > loss_text_ce: 0.03869729861617088 (0.03750480460610741) | > loss_mel_ce: 3.4680304527282715 (nan) | > loss: 0.013915587216615677 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13615251184514446) | > loader_time: 0.0046 (0.004668387137003577)  --> TIME: 2025-02-24 21:17:14 -- STEP: 8356/36938 -- GLOBAL_STEP: 488550 | > loss_text_ce: 0.03287026286125183 (0.037505481710685094) | > loss_mel_ce: 3.652928590774536 (nan) | > loss: 0.014626187272369862 (nan) | > current_lr: 5e-06 | > step_time: 0.1531 (0.13616289269016796) | > loader_time: 0.0049 (0.0046685180520258)  --> TIME: 2025-02-24 21:17:24 -- STEP: 8406/36938 -- GLOBAL_STEP: 488600 | > loss_text_ce: 0.046323735266923904 (0.037502200543914115) | > loss_mel_ce: 2.8649396896362305 (nan) | > loss: 0.011552633717656136 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13615704090459638) | > loader_time: 0.0046 (0.004668757712985675)  --> TIME: 2025-02-24 21:17:33 -- STEP: 8456/36938 -- GLOBAL_STEP: 488650 | > loss_text_ce: 0.03723995015025139 (0.03750435171173944) | > loss_mel_ce: 2.8759918212890625 (nan) | > loss: 0.011560444720089436 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13615178805313327) | > loader_time: 0.0045 (0.004668698518368686)  --> TIME: 2025-02-24 21:17:42 -- STEP: 8506/36938 -- GLOBAL_STEP: 488700 | > loss_text_ce: 0.03462797775864601 (0.03750996145074004) | > loss_mel_ce: 3.1309657096862793 (nan) | > loss: 0.012561880052089691 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.1361650833936679) | > loader_time: 0.0047 (0.004668546793686755)  --> TIME: 2025-02-24 21:17:52 -- STEP: 8556/36938 -- GLOBAL_STEP: 488750 | > loss_text_ce: 0.03491067513823509 (0.03750777564456926) | > loss_mel_ce: 3.114520311355591 (nan) | > loss: 0.012497742660343647 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13619926406045293) | > loader_time: 0.0046 (0.0046693046218939755)  --> TIME: 2025-02-24 21:18:01 -- STEP: 8606/36938 -- GLOBAL_STEP: 488800 | > loss_text_ce: 0.03361618146300316 (0.03751078478874243) | > loss_mel_ce: 3.0402212142944336 (nan) | > loss: 0.0121977673843503 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13619762305849867) | > loader_time: 0.0047 (0.00466923527514798)  --> TIME: 2025-02-24 21:18:11 -- STEP: 8656/36938 -- GLOBAL_STEP: 488850 | > loss_text_ce: 0.03959732502698898 (0.03750872480026282) | > loss_mel_ce: 2.907926082611084 (nan) | > loss: 0.011696522124111652 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13619973572477131) | > loader_time: 0.0048 (0.0046691242295580655)  --> TIME: 2025-02-24 21:18:20 -- STEP: 8706/36938 -- GLOBAL_STEP: 488900 | > loss_text_ce: 0.03125550225377083 (0.03751048406976723) | > loss_mel_ce: 3.0166823863983154 (nan) | > loss: 0.012094992212951183 (nan) | > current_lr: 5e-06 | > step_time: 0.1583 (0.1361994388978175) | > loader_time: 0.0047 (0.00466912835596786)  --> TIME: 2025-02-24 21:18:29 -- STEP: 8756/36938 -- GLOBAL_STEP: 488950 | > loss_text_ce: 0.04015468433499336 (0.037514519207851774) | > loss_mel_ce: 2.8230135440826416 (nan) | > loss: 0.011361779645085335 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.1362143406970358) | > loader_time: 0.0047 (0.004669069889856282)  --> TIME: 2025-02-24 21:18:38 -- STEP: 8806/36938 -- GLOBAL_STEP: 489000 | > loss_text_ce: 0.02760457992553711 (0.0375196350685103) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13619880145066607) | > loader_time: 0.0047 (0.004669033205836782)  --> TIME: 2025-02-24 21:18:48 -- STEP: 8856/36938 -- GLOBAL_STEP: 489050 | > loss_text_ce: 0.026515908539295197 (0.03751955888580691) | > loss_mel_ce: 1.6518319845199585 (nan) | > loss: 0.006660111248493195 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13620089658122891) | > loader_time: 0.0046 (0.004668833790540475)  --> TIME: 2025-02-24 21:18:57 -- STEP: 8906/36938 -- GLOBAL_STEP: 489100 | > loss_text_ce: 0.03498121723532677 (0.03750735008817219) | > loss_mel_ce: 2.659970998764038 (nan) | > loss: 0.010694255121052265 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.1362038744505186) | > loader_time: 0.0046 (0.004668759571523468)  --> TIME: 2025-02-24 21:19:07 -- STEP: 8956/36938 -- GLOBAL_STEP: 489150 | > loss_text_ce: 0.039722248911857605 (0.03752149684679519) | > loss_mel_ce: 3.3512096405029297 (nan) | > loss: 0.013456079177558422 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13623914680293486) | > loader_time: 0.0047 (0.004668726858255741)  --> TIME: 2025-02-24 21:19:16 -- STEP: 9006/36938 -- GLOBAL_STEP: 489200 | > loss_text_ce: 0.0336381234228611 (0.037524995881635534) | > loss_mel_ce: 2.382185935974121 (nan) | > loss: 0.009586604312062263 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13623849053184794) | > loader_time: 0.0046 (0.0046686969172973075)  --> TIME: 2025-02-24 21:19:25 -- STEP: 9056/36938 -- GLOBAL_STEP: 489250 | > loss_text_ce: 0.042704563587903976 (0.0375262643466662) | > loss_mel_ce: 0.07958984375 (nan) | > loss: 0.00048529531341046095 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13622078073108929) | > loader_time: 0.0046 (0.00466838858152868)  --> TIME: 2025-02-24 21:19:34 -- STEP: 9106/36938 -- GLOBAL_STEP: 489300 | > loss_text_ce: 0.03179565444588661 (0.037532379831210515) | > loss_mel_ce: 2.998706817626953 (nan) | > loss: 0.012025804258883 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.13623490960368037) | > loader_time: 0.0045 (0.004668345640916865)  --> TIME: 2025-02-24 21:19:44 -- STEP: 9156/36938 -- GLOBAL_STEP: 489350 | > loss_text_ce: 0.03605029359459877 (0.03753057814039639) | > loss_mel_ce: 0.2850273847579956 (nan) | > loss: 0.0012741178506985307 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13623991346921635) | > loader_time: 0.0045 (0.004668110606450607)  --> TIME: 2025-02-24 21:19:53 -- STEP: 9206/36938 -- GLOBAL_STEP: 489400 | > loss_text_ce: 0.03674551099538803 (0.03752485615120967) | > loss_mel_ce: 2.804158926010132 (nan) | > loss: 0.011273430660367012 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13623178077733325) | > loader_time: 0.0046 (0.004668068269426709)  --> TIME: 2025-02-24 21:20:02 -- STEP: 9256/36938 -- GLOBAL_STEP: 489450 | > loss_text_ce: 0.038666050881147385 (0.03753124194430197) | > loss_mel_ce: 2.6812493801116943 (nan) | > loss: 0.01079331524670124 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.13624045010672148) | > loader_time: 0.0049 (0.004668089394540548)  --> TIME: 2025-02-24 21:20:12 -- STEP: 9306/36938 -- GLOBAL_STEP: 489500 | > loss_text_ce: 0.03408759459853172 (0.037533572896479926) | > loss_mel_ce: 1.948981523513794 (nan) | > loss: 0.00786932185292244 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13622248680033514) | > loader_time: 0.0045 (0.004668034124753567)  --> TIME: 2025-02-24 21:20:21 -- STEP: 9356/36938 -- GLOBAL_STEP: 489550 | > loss_text_ce: 0.04271834343671799 (0.037538925857215855) | > loss_mel_ce: 3.1897597312927246 (nan) | > loss: 0.012827294878661633 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.1362296929212648) | > loader_time: 0.0046 (0.0046679545998216596)  --> TIME: 2025-02-24 21:20:31 -- STEP: 9406/36938 -- GLOBAL_STEP: 489600 | > loss_text_ce: 0.036677658557891846 (0.037547620745760284) | > loss_mel_ce: 3.2036759853363037 (nan) | > loss: 0.01285854633897543 (nan) | > current_lr: 5e-06 | > step_time: 0.1758 (0.13626940344689367) | > loader_time: 0.0048 (0.004668009679114193)  --> TIME: 2025-02-24 21:20:40 -- STEP: 9456/36938 -- GLOBAL_STEP: 489650 | > loss_text_ce: 0.035686854273080826 (0.03753971126243352) | > loss_mel_ce: 2.5821824073791504 (nan) | > loss: 0.010388371534645557 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13627780025884725) | > loader_time: 0.0047 (0.004667864258317579)  --> TIME: 2025-02-24 21:20:49 -- STEP: 9506/36938 -- GLOBAL_STEP: 489700 | > loss_text_ce: 0.033330872654914856 (0.03753922379607215) | > loss_mel_ce: 2.34442138671875 (nan) | > loss: 0.00943552516400814 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13626268881635797) | > loader_time: 0.0045 (0.004667497950304492)  --> TIME: 2025-02-24 21:20:58 -- STEP: 9556/36938 -- GLOBAL_STEP: 489750 | > loss_text_ce: 0.04287423565983772 (0.03754477571133116) | > loss_mel_ce: 2.9643073081970215 (nan) | > loss: 0.011933261528611183 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13624165064622812) | > loader_time: 0.0045 (0.004667156907293574)  --> TIME: 2025-02-24 21:21:08 -- STEP: 9606/36938 -- GLOBAL_STEP: 489800 | > loss_text_ce: 0.027232980355620384 (0.037548745693625346) | > loss_mel_ce: 3.338071584701538 (nan) | > loss: 0.013354383409023285 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13625102184127993) | > loader_time: 0.0046 (0.004667229461789057)  --> TIME: 2025-02-24 21:21:17 -- STEP: 9656/36938 -- GLOBAL_STEP: 489850 | > loss_text_ce: 0.04569464921951294 (0.03754709627403697) | > loss_mel_ce: 3.108086109161377 (nan) | > loss: 0.01251500379294157 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13626901977004863) | > loader_time: 0.0046 (0.004667354597962385)  --> TIME: 2025-02-24 21:21:27 -- STEP: 9706/36938 -- GLOBAL_STEP: 489900 | > loss_text_ce: 0.04532294720411301 (0.03754796427013473) | > loss_mel_ce: 3.0901896953582764 (nan) | > loss: 0.012442510575056076 (nan) | > current_lr: 5e-06 | > step_time: 0.1505 (0.136293268243037) | > loader_time: 0.0048 (0.004667233222051178)  --> TIME: 2025-02-24 21:21:36 -- STEP: 9756/36938 -- GLOBAL_STEP: 489950 | > loss_text_ce: 0.039459604769945145 (0.03755228375033147) | > loss_mel_ce: 3.5794875621795654 (nan) | > loss: 0.014360902830958366 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13628800793131945) | > loader_time: 0.0046 (0.004667269934294112)  --> TIME: 2025-02-24 21:21:45 -- STEP: 9806/36938 -- GLOBAL_STEP: 490000 | > loss_text_ce: 0.04018160700798035 (0.037558069919349395) | > loss_mel_ce: 2.5125341415405273 (nan) | > loss: 0.010129825212061405 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13626306909895247) | > loader_time: 0.0045 (0.004667202599214431) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_490000.pth  --> TIME: 2025-02-24 21:22:00 -- STEP: 9856/36938 -- GLOBAL_STEP: 490050 | > loss_text_ce: 0.04039248079061508 (0.03755808531687681) | > loss_mel_ce: 0.0655953511595726 (nan) | > loss: 0.000420586671680212 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.1362666368919913) | > loader_time: 0.0045 (0.004667636902107818)  --> TIME: 2025-02-24 21:22:10 -- STEP: 9906/36938 -- GLOBAL_STEP: 490100 | > loss_text_ce: 0.03210504725575447 (0.03756416114804942) | > loss_mel_ce: 2.9977054595947266 (nan) | > loss: 0.012023057788610458 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.13626850154359832) | > loader_time: 0.0049 (0.004668239750044766)  --> TIME: 2025-02-24 21:22:19 -- STEP: 9956/36938 -- GLOBAL_STEP: 490150 | > loss_text_ce: 0.043975699692964554 (0.037557609293605417) | > loss_mel_ce: 3.553602457046509 (nan) | > loss: 0.014276104047894478 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13628330906982455) | > loader_time: 0.0046 (0.00466831066178336)  --> TIME: 2025-02-24 21:22:29 -- STEP: 10006/36938 -- GLOBAL_STEP: 490200 | > loss_text_ce: 0.03726751729846001 (0.0375574197921349) | > loss_mel_ce: 3.4135591983795166 (nan) | > loss: 0.013693757355213165 (nan) | > current_lr: 5e-06 | > step_time: 0.1503 (0.13629183093952896) | > loader_time: 0.0046 (0.004668172326012648)  --> TIME: 2025-02-24 21:22:38 -- STEP: 10056/36938 -- GLOBAL_STEP: 490250 | > loss_text_ce: 0.04635917395353317 (0.037544804722272264) | > loss_mel_ce: 2.1924307346343994 (nan) | > loss: 0.008884087204933167 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1362965225746628) | > loader_time: 0.0045 (0.004668845481690469)  --> TIME: 2025-02-24 21:22:48 -- STEP: 10106/36938 -- GLOBAL_STEP: 490300 | > loss_text_ce: 0.04036939516663551 (0.037545804727793834) | > loss_mel_ce: 2.748647928237915 (nan) | > loss: 0.011067530140280724 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13631585756434275) | > loader_time: 0.0046 (0.004669228143652378)  --> TIME: 2025-02-24 21:22:57 -- STEP: 10156/36938 -- GLOBAL_STEP: 490350 | > loss_text_ce: 0.038571957498788834 (0.037547851065720346) | > loss_mel_ce: 3.3177850246429443 (nan) | > loss: 0.013318878598511219 (nan) | > current_lr: 5e-06 | > step_time: 0.1688 (0.13629853370951123) | > loader_time: 0.0049 (0.004669277557194246)  --> TIME: 2025-02-24 21:23:06 -- STEP: 10206/36938 -- GLOBAL_STEP: 490400 | > loss_text_ce: 0.03491035848855972 (0.037548556242027964) | > loss_mel_ce: 2.656134605407715 (nan) | > loss: 0.010678750462830067 (nan) | > current_lr: 5e-06 | > step_time: 0.1495 (0.13632213887153452) | > loader_time: 0.0047 (0.0046693414840982194)  --> TIME: 2025-02-24 21:23:16 -- STEP: 10256/36938 -- GLOBAL_STEP: 490450 | > loss_text_ce: 0.05606624856591225 (0.03754483220983274) | > loss_mel_ce: 3.5623629093170166 (nan) | > loss: 0.014358846470713615 (nan) | > current_lr: 5e-06 | > step_time: 0.1507 (0.13632579510464765) | > loader_time: 0.0048 (0.0046693016651454)  --> TIME: 2025-02-24 21:23:25 -- STEP: 10306/36938 -- GLOBAL_STEP: 490500 | > loss_text_ce: 0.03603316470980644 (0.03754786606849124) | > loss_mel_ce: 2.8339016437530518 (nan) | > loss: 0.011388630606234074 (nan) | > current_lr: 5e-06 | > step_time: 0.175 (0.13632422839463887) | > loader_time: 0.0048 (0.004669207867756173)  --> TIME: 2025-02-24 21:23:34 -- STEP: 10356/36938 -- GLOBAL_STEP: 490550 | > loss_text_ce: 0.037920039147138596 (0.037546374627693675) | > loss_mel_ce: 3.3932533264160156 (nan) | > loss: 0.013615768402814865 (nan) | > current_lr: 5e-06 | > step_time: 0.1759 (0.13633967836467573) | > loader_time: 0.0048 (0.004669177918970094)  --> TIME: 2025-02-24 21:23:44 -- STEP: 10406/36938 -- GLOBAL_STEP: 490600 | > loss_text_ce: 0.03496261686086655 (0.03754521519383878) | > loss_mel_ce: 3.2027604579925537 (nan) | > loss: 0.01284810807555914 (nan) | > current_lr: 5e-06 | > step_time: 0.1514 (0.13635501290155744) | > loader_time: 0.0047 (0.00466923671984887)  --> TIME: 2025-02-24 21:23:53 -- STEP: 10456/36938 -- GLOBAL_STEP: 490650 | > loss_text_ce: 0.04142936319112778 (0.037536757112539135) | > loss_mel_ce: 3.250856399536133 (nan) | > loss: 0.013064626604318619 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13634746480097465) | > loader_time: 0.0047 (0.004669547103795729)  --> TIME: 2025-02-24 21:24:03 -- STEP: 10506/36938 -- GLOBAL_STEP: 490700 | > loss_text_ce: 0.04646443575620651 (0.037537771150985354) | > loss_mel_ce: 2.7129921913146973 (nan) | > loss: 0.010950225405395031 (nan) | > current_lr: 5e-06 | > step_time: 0.1798 (0.1363655887693073) | > loader_time: 0.0049 (0.004669695950770817)  --> TIME: 2025-02-24 21:24:12 -- STEP: 10556/36938 -- GLOBAL_STEP: 490750 | > loss_text_ce: 0.035904623568058014 (0.03753141509081396) | > loss_mel_ce: 3.0174548625946045 (nan) | > loss: 0.012116506695747375 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1363558243874025) | > loader_time: 0.0045 (0.004669782563383354)  --> TIME: 2025-02-24 21:24:21 -- STEP: 10606/36938 -- GLOBAL_STEP: 490800 | > loss_text_ce: 0.04368886724114418 (0.037523332756791046) | > loss_mel_ce: 3.297846794128418 (nan) | > loss: 0.013260062783956528 (nan) | > current_lr: 5e-06 | > step_time: 0.1855 (0.1363515076987043) | > loader_time: 0.0051 (0.004670162122698744)  --> TIME: 2025-02-24 21:24:30 -- STEP: 10656/36938 -- GLOBAL_STEP: 490850 | > loss_text_ce: 0.04200252145528793 (0.03752532555358473) | > loss_mel_ce: 2.5751953125 (nan) | > loss: 0.010385706089437008 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13633527228603057) | > loader_time: 0.0045 (0.004670077414662984)  --> TIME: 2025-02-24 21:24:39 -- STEP: 10706/36938 -- GLOBAL_STEP: 490900 | > loss_text_ce: 0.03403991460800171 (0.03752120567903943) | > loss_mel_ce: 3.045474052429199 (nan) | > loss: 0.012220294214785099 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.1363331872281587) | > loader_time: 0.0046 (0.004669881682072573)  --> TIME: 2025-02-24 21:24:49 -- STEP: 10756/36938 -- GLOBAL_STEP: 490950 | > loss_text_ce: 0.042531818151474 (0.03751805290022527) | > loss_mel_ce: 2.8101043701171875 (nan) | > loss: 0.01131998561322689 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13632969461262193) | > loader_time: 0.0046 (0.004669642350894443)  --> TIME: 2025-02-24 21:24:58 -- STEP: 10806/36938 -- GLOBAL_STEP: 491000 | > loss_text_ce: 0.03550133854150772 (0.037516197907701686) | > loss_mel_ce: 2.362758159637451 (nan) | > loss: 0.009516903199255466 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13632045029578851) | > loader_time: 0.0045 (0.004669724361512163)  --> TIME: 2025-02-24 21:25:07 -- STEP: 10856/36938 -- GLOBAL_STEP: 491050 | > loss_text_ce: 0.042553212493658066 (0.037516774951209954) | > loss_mel_ce: 3.34889817237854 (nan) | > loss: 0.01345814112573862 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.1363146144103362) | > loader_time: 0.0046 (0.004669540470456404)  --> TIME: 2025-02-24 21:25:16 -- STEP: 10906/36938 -- GLOBAL_STEP: 491100 | > loss_text_ce: 0.0483330674469471 (0.03751571908432298) | > loss_mel_ce: 3.3982532024383545 (nan) | > loss: 0.013676931150257587 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13629540978670687) | > loader_time: 0.0045 (0.004669291435764965)  --> TIME: 2025-02-24 21:25:25 -- STEP: 10956/36938 -- GLOBAL_STEP: 491150 | > loss_text_ce: 0.030673228204250336 (0.037525138710535415) | > loss_mel_ce: 3.499685525894165 (nan) | > loss: 0.014009361155331135 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1362849574491194) | > loader_time: 0.0046 (0.0046691800521733535)  --> TIME: 2025-02-24 21:25:35 -- STEP: 11006/36938 -- GLOBAL_STEP: 491200 | > loss_text_ce: 0.03742958605289459 (0.03752430805268984) | > loss_mel_ce: 2.9906165599823 (nan) | > loss: 0.012016057036817074 (nan) | > current_lr: 5e-06 | > step_time: 0.1848 (0.13630430691462664) | > loader_time: 0.0049 (0.00466927213927041)  --> TIME: 2025-02-24 21:25:44 -- STEP: 11056/36938 -- GLOBAL_STEP: 491250 | > loss_text_ce: 0.04360023885965347 (0.03751934926113517) | > loss_mel_ce: 0.0940399169921875 (nan) | > loss: 0.0005461911205202341 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13632012862717463) | > loader_time: 0.0049 (0.004669519305746737)  --> TIME: 2025-02-24 21:25:53 -- STEP: 11106/36938 -- GLOBAL_STEP: 491300 | > loss_text_ce: 0.0406070239841938 (0.03752170528623824) | > loss_mel_ce: 0.7557373046875 (nan) | > loss: 0.0031600967049598694 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13631428765022588) | > loader_time: 0.0046 (0.0046693954342402946)  --> TIME: 2025-02-24 21:26:03 -- STEP: 11156/36938 -- GLOBAL_STEP: 491350 | > loss_text_ce: 0.035862863063812256 (0.03751715403923406) | > loss_mel_ce: 3.0844483375549316 (nan) | > loss: 0.012382187880575657 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.1363262159502083) | > loader_time: 0.0048 (0.004669375362341527)  --> TIME: 2025-02-24 21:26:12 -- STEP: 11206/36938 -- GLOBAL_STEP: 491400 | > loss_text_ce: 0.028791705146431923 (0.03751213287761132) | > loss_mel_ce: 1.9747042655944824 (nan) | > loss: 0.007950381375849247 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13630917264630865) | > loader_time: 0.0045 (0.0046691938785278575)  --> TIME: 2025-02-24 21:26:21 -- STEP: 11256/36938 -- GLOBAL_STEP: 491450 | > loss_text_ce: 0.03993012756109238 (0.037517044203327445) | > loss_mel_ce: 2.7853758335113525 (nan) | > loss: 0.011211532168090343 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13631731195490399) | > loader_time: 0.0047 (0.004669531787975341)  --> TIME: 2025-02-24 21:26:30 -- STEP: 11306/36938 -- GLOBAL_STEP: 491500 | > loss_text_ce: 0.043306827545166016 (0.037508644127775756) | > loss_mel_ce: 2.378779411315918 (nan) | > loss: 0.009611453860998154 (nan) | > current_lr: 5e-06 | > step_time: 0.1123 (0.13628597257834366) | > loader_time: 0.0046 (0.004669329180836401)  --> TIME: 2025-02-24 21:26:39 -- STEP: 11356/36938 -- GLOBAL_STEP: 491550 | > loss_text_ce: 0.056773994117975235 (0.03750591048639161) | > loss_mel_ce: 2.6140687465667725 (nan) | > loss: 0.010598582215607166 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13627639308954873) | > loader_time: 0.0045 (0.004669257854651128)  --> TIME: 2025-02-24 21:26:49 -- STEP: 11406/36938 -- GLOBAL_STEP: 491600 | > loss_text_ce: 0.042732059955596924 (0.037500126099623166) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1362586865482791) | > loader_time: 0.0048 (0.004669296225183745)  --> TIME: 2025-02-24 21:26:58 -- STEP: 11456/36938 -- GLOBAL_STEP: 491650 | > loss_text_ce: 0.039263851940631866 (0.03749711219133225) | > loss_mel_ce: 0.1394856721162796 (nan) | > loss: 0.0007093235617503524 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13626527617680192) | > loader_time: 0.0045 (0.004669707913638491)  --> TIME: 2025-02-24 21:27:07 -- STEP: 11506/36938 -- GLOBAL_STEP: 491700 | > loss_text_ce: 0.05478284880518913 (0.03749529490077062) | > loss_mel_ce: 2.596900224685669 (nan) | > loss: 0.010522552765905857 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13627605628453304) | > loader_time: 0.0047 (0.004670154461959284)  --> TIME: 2025-02-24 21:27:17 -- STEP: 11556/36938 -- GLOBAL_STEP: 491750 | > loss_text_ce: 0.04493722692131996 (0.03749592580791276) | > loss_mel_ce: 2.3207366466522217 (nan) | > loss: 0.009387594647705555 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13627195308760476) | > loader_time: 0.0047 (0.004670149750179706)  --> TIME: 2025-02-24 21:27:26 -- STEP: 11606/36938 -- GLOBAL_STEP: 491800 | > loss_text_ce: 0.03892615810036659 (0.03749931851971984) | > loss_mel_ce: 2.1575872898101807 (nan) | > loss: 0.008716323412954807 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13627333232663277) | > loader_time: 0.0047 (0.004670243170391617)  --> TIME: 2025-02-24 21:27:35 -- STEP: 11656/36938 -- GLOBAL_STEP: 491850 | > loss_text_ce: 0.040954895317554474 (0.03749428713074502) | > loss_mel_ce: 2.016432285308838 (nan) | > loss: 0.008164235390722752 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.136257912659366) | > loader_time: 0.0045 (0.004670301425432446)  --> TIME: 2025-02-24 21:27:44 -- STEP: 11706/36938 -- GLOBAL_STEP: 491900 | > loss_text_ce: 0.0367191880941391 (0.03749768724779604) | > loss_mel_ce: 3.2177236080169678 (nan) | > loss: 0.012914455495774746 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13624435684608746) | > loader_time: 0.0047 (0.004670232030320071)  --> TIME: 2025-02-24 21:27:54 -- STEP: 11756/36938 -- GLOBAL_STEP: 491950 | > loss_text_ce: 0.03876003995537758 (0.03750602577197569) | > loss_mel_ce: 3.1981077194213867 (nan) | > loss: 0.012844713404774666 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13624838999478472) | > loader_time: 0.0046 (0.0046701168843620005)  --> TIME: 2025-02-24 21:28:03 -- STEP: 11806/36938 -- GLOBAL_STEP: 492000 | > loss_text_ce: 0.04057242348790169 (0.03749961871101587) | > loss_mel_ce: 2.0110068321228027 (nan) | > loss: 0.008141187950968742 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13625543238287371) | > loader_time: 0.0046 (0.004669998270888126)  --> TIME: 2025-02-24 21:28:12 -- STEP: 11856/36938 -- GLOBAL_STEP: 492050 | > loss_text_ce: 0.023259496316313744 (0.03749973824938659) | > loss_mel_ce: 2.615199089050293 (nan) | > loss: 0.01047007367014885 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13624650288207232) | > loader_time: 0.0045 (0.004669771382683193)  --> TIME: 2025-02-24 21:28:22 -- STEP: 11906/36938 -- GLOBAL_STEP: 492100 | > loss_text_ce: 0.03800486773252487 (0.03749635918084669) | > loss_mel_ce: 3.5832386016845703 (nan) | > loss: 0.014370014891028404 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13626767646639507) | > loader_time: 0.0049 (0.004669707461843638)  --> TIME: 2025-02-24 21:28:31 -- STEP: 11956/36938 -- GLOBAL_STEP: 492150 | > loss_text_ce: 0.03699364885687828 (0.03749539350877972) | > loss_mel_ce: 3.852045774459839 (nan) | > loss: 0.015432697720825672 (nan) | > current_lr: 5e-06 | > step_time: 0.1821 (0.1362997169038361) | > loader_time: 0.0051 (0.004669884607998487)  --> TIME: 2025-02-24 21:28:41 -- STEP: 12006/36938 -- GLOBAL_STEP: 492200 | > loss_text_ce: 0.0297144316136837 (0.03749628288422521) | > loss_mel_ce: 3.4177651405334473 (nan) | > loss: 0.013680474832654 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13630508254532636) | > loader_time: 0.0049 (0.00466980085002766)  --> TIME: 2025-02-24 21:28:50 -- STEP: 12056/36938 -- GLOBAL_STEP: 492250 | > loss_text_ce: 0.037777580320835114 (0.03749480128914603) | > loss_mel_ce: 2.1802124977111816 (nan) | > loss: 0.00880154874175787 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13629923852848927) | > loader_time: 0.0046 (0.004669596342828467)  --> TIME: 2025-02-24 21:28:59 -- STEP: 12106/36938 -- GLOBAL_STEP: 492300 | > loss_text_ce: 0.04800835996866226 (0.03749217411568428) | > loss_mel_ce: 1.0866131782531738 (nan) | > loss: 0.004502466414123774 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13628875063053647) | > loader_time: 0.0046 (0.0046694667481517425)  --> TIME: 2025-02-24 21:29:09 -- STEP: 12156/36938 -- GLOBAL_STEP: 492350 | > loss_text_ce: 0.035932913422584534 (0.03749071614850183) | > loss_mel_ce: 2.6201882362365723 (nan) | > loss: 0.010540164075791836 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.1362983651089009) | > loader_time: 0.0047 (0.004669657738909354)  --> TIME: 2025-02-24 21:29:18 -- STEP: 12206/36938 -- GLOBAL_STEP: 492400 | > loss_text_ce: 0.04399377107620239 (0.03749080714488108) | > loss_mel_ce: 3.147310733795166 (nan) | > loss: 0.012663907371461391 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.13630185134290065) | > loader_time: 0.0049 (0.004669626423634019)  --> TIME: 2025-02-24 21:29:27 -- STEP: 12256/36938 -- GLOBAL_STEP: 492450 | > loss_text_ce: 0.03836099058389664 (0.03749397266317243) | > loss_mel_ce: 3.4487104415893555 (nan) | > loss: 0.013837586157023907 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13629531512138965) | > loader_time: 0.0047 (0.0046696232792291575)  --> TIME: 2025-02-24 21:29:37 -- STEP: 12306/36938 -- GLOBAL_STEP: 492500 | > loss_text_ce: 0.043821707367897034 (0.03749589490011454) | > loss_mel_ce: 2.277048110961914 (nan) | > loss: 0.009209801442921162 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13630115827002068) | > loader_time: 0.0049 (0.004669712846422122)  --> TIME: 2025-02-24 21:29:46 -- STEP: 12356/36938 -- GLOBAL_STEP: 492550 | > loss_text_ce: 0.04976700246334076 (0.03749414041944989) | > loss_mel_ce: 1.907248616218567 (nan) | > loss: 0.007765935733914375 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13629574947814033) | > loader_time: 0.0045 (0.004669696584647812)  --> TIME: 2025-02-24 21:29:55 -- STEP: 12406/36938 -- GLOBAL_STEP: 492600 | > loss_text_ce: 0.03803268074989319 (0.03749553281905441) | > loss_mel_ce: 2.9041247367858887 (nan) | > loss: 0.011675229296088219 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13629923275396025) | > loader_time: 0.0048 (0.0046696243950307)  --> TIME: 2025-02-24 21:30:05 -- STEP: 12456/36938 -- GLOBAL_STEP: 492650 | > loss_text_ce: 0.028026452288031578 (0.037493744507439704) | > loss_mel_ce: 3.161667823791504 (nan) | > loss: 0.012657517567276955 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.1363118907428348) | > loader_time: 0.0047 (0.0046698353806595445)  --> TIME: 2025-02-24 21:30:14 -- STEP: 12506/36938 -- GLOBAL_STEP: 492700 | > loss_text_ce: 0.028419770300388336 (0.037493755668059105) | > loss_mel_ce: 1.6409902572631836 (nan) | > loss: 0.006624643225222826 (nan) | > current_lr: 5e-06 | > step_time: 0.1065 (0.13632459616100503) | > loader_time: 0.0045 (0.004669729336040522)  --> TIME: 2025-02-24 21:30:24 -- STEP: 12556/36938 -- GLOBAL_STEP: 492750 | > loss_text_ce: 0.03175083547830582 (0.03749242104854287) | > loss_mel_ce: 0.091796875 (nan) | > loss: 0.0004902686923742294 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13631822412353437) | > loader_time: 0.0045 (0.004669548277263991)  --> TIME: 2025-02-24 21:30:33 -- STEP: 12606/36938 -- GLOBAL_STEP: 492800 | > loss_text_ce: 0.038506053388118744 (0.037492619207616124) | > loss_mel_ce: 2.89945387840271 (nan) | > loss: 0.011658571660518646 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13632387778654137) | > loader_time: 0.0046 (0.004669313617647585)  --> TIME: 2025-02-24 21:30:42 -- STEP: 12656/36938 -- GLOBAL_STEP: 492850 | > loss_text_ce: 0.03836478292942047 (0.03748837568290587) | > loss_mel_ce: 2.6197917461395264 (nan) | > loss: 0.010548241436481476 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.1363253819693798) | > loader_time: 0.0046 (0.004669063631562901)  --> TIME: 2025-02-24 21:30:51 -- STEP: 12706/36938 -- GLOBAL_STEP: 492900 | > loss_text_ce: 0.025703389197587967 (0.03749082475898) | > loss_mel_ce: 3.0725696086883545 (nan) | > loss: 0.012294734828174114 (nan) | > current_lr: 5e-06 | > step_time: 0.1434 (0.1363173550214418) | > loader_time: 0.0047 (0.004668876259003806)  --> TIME: 2025-02-24 21:31:01 -- STEP: 12756/36938 -- GLOBAL_STEP: 492950 | > loss_text_ce: 0.03835146501660347 (0.03748696277862214) | > loss_mel_ce: 3.0157082080841064 (nan) | > loss: 0.012119285762310028 (nan) | > current_lr: 5e-06 | > step_time: 0.1565 (0.13633404242726907) | > loader_time: 0.0048 (0.004668804686358813)  --> TIME: 2025-02-24 21:31:10 -- STEP: 12806/36938 -- GLOBAL_STEP: 493000 | > loss_text_ce: 0.04293586686253548 (0.03748475049729834) | > loss_mel_ce: 3.4443323612213135 (nan) | > loss: 0.013838366605341434 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.136353028838976) | > loader_time: 0.0048 (0.004668902479520568)  --> TIME: 2025-02-24 21:31:20 -- STEP: 12856/36938 -- GLOBAL_STEP: 493050 | > loss_text_ce: 0.03136502578854561 (0.03748564183135062) | > loss_mel_ce: 3.3381502628326416 (nan) | > loss: 0.013371093198657036 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.1363586160842862) | > loader_time: 0.0048 (0.004668818899171274)  --> TIME: 2025-02-24 21:31:29 -- STEP: 12906/36938 -- GLOBAL_STEP: 493100 | > loss_text_ce: 0.026006240397691727 (0.03749193488942777) | > loss_mel_ce: 2.4420106410980225 (nan) | > loss: 0.009793718345463276 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.1363531657379571) | > loader_time: 0.0046 (0.004668734488552998)  --> TIME: 2025-02-24 21:31:38 -- STEP: 12956/36938 -- GLOBAL_STEP: 493150 | > loss_text_ce: 0.04606868326663971 (0.03749208437765078) | > loss_mel_ce: 3.641002893447876 (nan) | > loss: 0.014631236903369427 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13635025711919244) | > loader_time: 0.0048 (0.004668811030798606)  --> TIME: 2025-02-24 21:31:48 -- STEP: 13006/36938 -- GLOBAL_STEP: 493200 | > loss_text_ce: 0.040201906114816666 (0.03749279399122734) | > loss_mel_ce: 3.056813955307007 (nan) | > loss: 0.012289745733141899 (nan) | > current_lr: 5e-06 | > step_time: 0.1812 (0.13635163121675814) | > loader_time: 0.0048 (0.004669050647536723)  --> TIME: 2025-02-24 21:31:57 -- STEP: 13056/36938 -- GLOBAL_STEP: 493250 | > loss_text_ce: 0.03782154619693756 (0.037497652903802324) | > loss_mel_ce: 2.628810167312622 (nan) | > loss: 0.010581872425973415 (nan) | > current_lr: 5e-06 | > step_time: 0.1531 (0.13635150032738785) | > loader_time: 0.0046 (0.004669153095022122)  --> TIME: 2025-02-24 21:32:07 -- STEP: 13106/36938 -- GLOBAL_STEP: 493300 | > loss_text_ce: 0.04726877436041832 (0.03749685837370893) | > loss_mel_ce: 3.116318464279175 (nan) | > loss: 0.012553919106721878 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.1363828888598709) | > loader_time: 0.0046 (0.0046695143907611765)  --> TIME: 2025-02-24 21:32:16 -- STEP: 13156/36938 -- GLOBAL_STEP: 493350 | > loss_text_ce: 0.03484674170613289 (0.037494185445694266) | > loss_mel_ce: 3.0950686931610107 (nan) | > loss: 0.012420300394296646 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.1363713197086163) | > loader_time: 0.0048 (0.004669511851005971)  --> TIME: 2025-02-24 21:32:25 -- STEP: 13206/36938 -- GLOBAL_STEP: 493400 | > loss_text_ce: 0.040773458778858185 (0.037488431389585) | > loss_mel_ce: 2.4050488471984863 (nan) | > loss: 0.009705644100904465 (nan) | > current_lr: 5e-06 | > step_time: 0.1807 (0.13636842491445064) | > loader_time: 0.0048 (0.004669610467911043)  --> TIME: 2025-02-24 21:32:34 -- STEP: 13256/36938 -- GLOBAL_STEP: 493450 | > loss_text_ce: 0.03663824498653412 (0.03748067870076263) | > loss_mel_ce: 1.8687862157821655 (nan) | > loss: 0.007561208680272102 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.1363602921627479) | > loader_time: 0.0046 (0.0046693933031684036)  --> TIME: 2025-02-24 21:32:44 -- STEP: 13306/36938 -- GLOBAL_STEP: 493500 | > loss_text_ce: 0.030730323866009712 (0.037478184074109605) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.1363468267461432) | > loader_time: 0.0047 (0.004669444266789004)  --> TIME: 2025-02-24 21:32:53 -- STEP: 13356/36938 -- GLOBAL_STEP: 493550 | > loss_text_ce: 0.03382493555545807 (0.037477734002894614) | > loss_mel_ce: 3.338423490524292 (nan) | > loss: 0.013381938450038433 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.1363492773906591) | > loader_time: 0.0049 (0.004669420731285428)  --> TIME: 2025-02-24 21:33:03 -- STEP: 13406/36938 -- GLOBAL_STEP: 493600 | > loss_text_ce: 0.04323916882276535 (0.03747907438383943) | > loss_mel_ce: 3.2288613319396973 (nan) | > loss: 0.012984526343643665 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.13636750643526793) | > loader_time: 0.0047 (0.004669520991178966)  --> TIME: 2025-02-24 21:33:12 -- STEP: 13456/36938 -- GLOBAL_STEP: 493650 | > loss_text_ce: 0.03709075599908829 (0.03747832157256517) | > loss_mel_ce: 3.4160609245300293 (nan) | > loss: 0.01370298396795988 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.13638171590820297) | > loader_time: 0.0047 (0.004669549756469672)  --> TIME: 2025-02-24 21:33:21 -- STEP: 13506/36938 -- GLOBAL_STEP: 493700 | > loss_text_ce: 0.03049074113368988 (0.03748165534616974) | > loss_mel_ce: 3.2571756839752197 (nan) | > loss: 0.013046295382082462 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13638722065129463) | > loader_time: 0.0045 (0.004669715223816532)  --> TIME: 2025-02-24 21:33:31 -- STEP: 13556/36938 -- GLOBAL_STEP: 493750 | > loss_text_ce: 0.03670556843280792 (0.03747442800578576) | > loss_mel_ce: 3.4856762886047363 (nan) | > loss: 0.01397770643234253 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13639606260627174) | > loader_time: 0.0049 (0.0046699542357668935)  --> TIME: 2025-02-24 21:33:40 -- STEP: 13606/36938 -- GLOBAL_STEP: 493800 | > loss_text_ce: 0.02944112941622734 (0.03747178030526251) | > loss_mel_ce: 2.939541816711426 (nan) | > loss: 0.011781678535044193 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13639432623791797) | > loader_time: 0.0048 (0.0046700801320870055)  --> TIME: 2025-02-24 21:33:50 -- STEP: 13656/36938 -- GLOBAL_STEP: 493850 | > loss_text_ce: 0.04023686423897743 (0.03747073312702745) | > loss_mel_ce: 2.6075820922851562 (nan) | > loss: 0.010507219471037388 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13638776666460153) | > loader_time: 0.0045 (0.004669819526946111)  --> TIME: 2025-02-24 21:33:59 -- STEP: 13706/36938 -- GLOBAL_STEP: 493900 | > loss_text_ce: 0.041924022138118744 (0.037466783167305874) | > loss_mel_ce: 2.6151974201202393 (nan) | > loss: 0.010544133372604847 (nan) | > current_lr: 5e-06 | > step_time: 0.1837 (0.1363804546321682) | > loader_time: 0.0049 (0.004669612139032881)  --> TIME: 2025-02-24 21:34:08 -- STEP: 13756/36938 -- GLOBAL_STEP: 493950 | > loss_text_ce: 0.037757180631160736 (0.03746424347076124) | > loss_mel_ce: 3.244839668273926 (nan) | > loss: 0.013026178814470768 (nan) | > current_lr: 5e-06 | > step_time: 0.1628 (0.13639367857309315) | > loader_time: 0.0048 (0.004669466625988974)  --> TIME: 2025-02-24 21:34:18 -- STEP: 13806/36938 -- GLOBAL_STEP: 494000 | > loss_text_ce: 0.030551999807357788 (0.0374606574853032) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1075 (0.1364150987782888) | > loader_time: 0.0045 (0.00466955321155974)  --> TIME: 2025-02-24 21:34:27 -- STEP: 13856/36938 -- GLOBAL_STEP: 494050 | > loss_text_ce: 0.05026677995920181 (0.03745984193797952) | > loss_mel_ce: 1.3649581670761108 (nan) | > loss: 0.0056159719824790955 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13640197579841362) | > loader_time: 0.0045 (0.004669534950140705)  --> TIME: 2025-02-24 21:34:36 -- STEP: 13906/36938 -- GLOBAL_STEP: 494100 | > loss_text_ce: 0.04404938966035843 (0.037456262541816124) | > loss_mel_ce: 2.4653818607330322 (nan) | > loss: 0.009958061389625072 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1364100213430084) | > loader_time: 0.0047 (0.004669681532201558)  --> TIME: 2025-02-24 21:34:46 -- STEP: 13956/36938 -- GLOBAL_STEP: 494150 | > loss_text_ce: 0.039869070053100586 (0.03745625226155952) | > loss_mel_ce: 2.705890655517578 (nan) | > loss: 0.010895872488617897 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13641317389898985) | > loader_time: 0.0045 (0.00466995115914293)  --> TIME: 2025-02-24 21:34:55 -- STEP: 14006/36938 -- GLOBAL_STEP: 494200 | > loss_text_ce: 0.03546702489256859 (0.03745709910047772) | > loss_mel_ce: 3.349437713623047 (nan) | > loss: 0.013432161882519722 (nan) | > current_lr: 5e-06 | > step_time: 0.1956 (0.1364184727111095) | > loader_time: 0.0048 (0.004670026403451897)  --> TIME: 2025-02-24 21:35:04 -- STEP: 14056/36938 -- GLOBAL_STEP: 494250 | > loss_text_ce: 0.033927638083696365 (0.03745556553941285) | > loss_mel_ce: 3.2841806411743164 (nan) | > loss: 0.013167097233235836 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13640811676387352) | > loader_time: 0.0046 (0.004670024579671805)  --> TIME: 2025-02-24 21:35:14 -- STEP: 14106/36938 -- GLOBAL_STEP: 494300 | > loss_text_ce: 0.04124907776713371 (0.03745431470287639) | > loss_mel_ce: 2.680363655090332 (nan) | > loss: 0.01080005057156086 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.1363976546657613) | > loader_time: 0.0046 (0.004670073085856835)  --> TIME: 2025-02-24 21:35:23 -- STEP: 14156/36938 -- GLOBAL_STEP: 494350 | > loss_text_ce: 0.03804624080657959 (0.037448091793639) | > loss_mel_ce: 3.231513023376465 (nan) | > loss: 0.012974442914128304 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13639834286567606) | > loader_time: 0.0045 (0.0046699386459516466)  --> TIME: 2025-02-24 21:35:32 -- STEP: 14206/36938 -- GLOBAL_STEP: 494400 | > loss_text_ce: 0.03215590491890907 (0.03744669266199282) | > loss_mel_ce: 3.3036892414093018 (nan) | > loss: 0.0132374819368124 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.1363985513811721) | > loader_time: 0.0047 (0.004670026905516836)  --> TIME: 2025-02-24 21:35:41 -- STEP: 14256/36938 -- GLOBAL_STEP: 494450 | > loss_text_ce: 0.040422677993774414 (0.03744360913852315) | > loss_mel_ce: 0.356201171875 (nan) | > loss: 0.0015739043010398746 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13639398844019912) | > loader_time: 0.0046 (0.0046698182955066374)  --> TIME: 2025-02-24 21:35:51 -- STEP: 14306/36938 -- GLOBAL_STEP: 494500 | > loss_text_ce: 0.03443603590130806 (0.037444062594532104) | > loss_mel_ce: 3.4045000076293945 (nan) | > loss: 0.013646571896970272 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13639418903504333) | > loader_time: 0.0046 (0.004670069198683049)  --> TIME: 2025-02-24 21:36:00 -- STEP: 14356/36938 -- GLOBAL_STEP: 494550 | > loss_text_ce: 0.0369814857840538 (0.03744633844644906) | > loss_mel_ce: 3.5970780849456787 (nan) | > loss: 0.014420872554183006 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.1363978240608862) | > loader_time: 0.0049 (0.004669950080801419)  --> TIME: 2025-02-24 21:36:09 -- STEP: 14406/36938 -- GLOBAL_STEP: 494600 | > loss_text_ce: 0.04154953733086586 (0.03745028383018245) | > loss_mel_ce: 2.2546684741973877 (nan) | > loss: 0.009111976251006126 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13639919500987716) | > loader_time: 0.0045 (0.004669899346678506)  --> TIME: 2025-02-24 21:36:19 -- STEP: 14456/36938 -- GLOBAL_STEP: 494650 | > loss_text_ce: 0.02998446486890316 (0.03745384393200608) | > loss_mel_ce: 3.0220930576324463 (nan) | > loss: 0.012111419811844826 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13640365850615296) | > loader_time: 0.0045 (0.004669852806311903)  --> TIME: 2025-02-24 21:36:28 -- STEP: 14506/36938 -- GLOBAL_STEP: 494700 | > loss_text_ce: 0.04366481304168701 (0.03745508001400514) | > loss_mel_ce: 3.2048943042755127 (nan) | > loss: 0.01289110817015171 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.1363966549147929) | > loader_time: 0.0048 (0.004669786403545019)  --> TIME: 2025-02-24 21:36:37 -- STEP: 14556/36938 -- GLOBAL_STEP: 494750 | > loss_text_ce: 0.0486447811126709 (0.03745744774140071) | > loss_mel_ce: 3.7297523021698 (nan) | > loss: 0.014993639662861824 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13639606065755272) | > loader_time: 0.0046 (0.004669693905955379)  --> TIME: 2025-02-24 21:36:46 -- STEP: 14606/36938 -- GLOBAL_STEP: 494800 | > loss_text_ce: 0.04465978965163231 (0.037458113655850896) | > loss_mel_ce: 3.132333993911743 (nan) | > loss: 0.012607119046151638 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.1363904036904914) | > loader_time: 0.0047 (0.00466972768233276)  --> TIME: 2025-02-24 21:36:55 -- STEP: 14656/36938 -- GLOBAL_STEP: 494850 | > loss_text_ce: 0.03659025952219963 (0.037455304127616565) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.1363557463422344) | > loader_time: 0.0048 (0.004669613225230992)  --> TIME: 2025-02-24 21:37:04 -- STEP: 14706/36938 -- GLOBAL_STEP: 494900 | > loss_text_ce: 0.035253725945949554 (0.03745169284443563) | > loss_mel_ce: 2.837261438369751 (nan) | > loss: 0.01139887049794197 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13636439773609285) | > loader_time: 0.0047 (0.00466971557577758)  --> TIME: 2025-02-24 21:37:14 -- STEP: 14756/36938 -- GLOBAL_STEP: 494950 | > loss_text_ce: 0.039332035928964615 (0.037449981617383) | > loss_mel_ce: 2.9059102535247803 (nan) | > loss: 0.011687470600008965 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.1363735712456738) | > loader_time: 0.0064 (0.0046699498849497945)  --> TIME: 2025-02-24 21:37:23 -- STEP: 14806/36938 -- GLOBAL_STEP: 495000 | > loss_text_ce: 0.03677665814757347 (0.037455960561214474) | > loss_mel_ce: 2.7425410747528076 (nan) | > loss: 0.011029038578271866 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13638141735589013) | > loader_time: 0.0045 (0.004670054642361551)  --> TIME: 2025-02-24 21:37:32 -- STEP: 14856/36938 -- GLOBAL_STEP: 495050 | > loss_text_ce: 0.03839270398020744 (0.03745989934338165) | > loss_mel_ce: 2.9685120582580566 (nan) | > loss: 0.011932162567973137 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.1363625228084017) | > loader_time: 0.0048 (0.0046698378341841085)  --> TIME: 2025-02-24 21:37:42 -- STEP: 14906/36938 -- GLOBAL_STEP: 495100 | > loss_text_ce: 0.038526106625795364 (0.037458900530573484) | > loss_mel_ce: 3.2661361694335938 (nan) | > loss: 0.013113739900290966 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13637197740634818) | > loader_time: 0.0046 (0.004669812658753429)  --> TIME: 2025-02-24 21:37:51 -- STEP: 14956/36938 -- GLOBAL_STEP: 495150 | > loss_text_ce: 0.04228457435965538 (0.037463064279828664) | > loss_mel_ce: 2.9051034450531006 (nan) | > loss: 0.0116959847509861 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13636641042951053) | > loader_time: 0.0045 (0.0046696280948488825)  --> TIME: 2025-02-24 21:38:00 -- STEP: 15006/36938 -- GLOBAL_STEP: 495200 | > loss_text_ce: 0.029426712542772293 (0.03746774738700913) | > loss_mel_ce: 2.3832366466522217 (nan) | > loss: 0.009574062190949917 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.1363655777274956) | > loader_time: 0.0046 (0.004669502832310261)  --> TIME: 2025-02-24 21:38:10 -- STEP: 15056/36938 -- GLOBAL_STEP: 495250 | > loss_text_ce: 0.03736981004476547 (0.037466899195550345) | > loss_mel_ce: 3.177766799926758 (nan) | > loss: 0.012758479453623295 (nan) | > current_lr: 5e-06 | > step_time: 0.1927 (0.13638146223221168) | > loader_time: 0.0049 (0.0046696232336867136)  --> TIME: 2025-02-24 21:38:19 -- STEP: 15106/36938 -- GLOBAL_STEP: 495300 | > loss_text_ce: 0.03375070542097092 (0.03746976933874729) | > loss_mel_ce: 2.5505175590515137 (nan) | > loss: 0.01025503408163786 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13639079075561547) | > loader_time: 0.0049 (0.004669442755109366)  --> TIME: 2025-02-24 21:38:29 -- STEP: 15156/36938 -- GLOBAL_STEP: 495350 | > loss_text_ce: 0.027693787589669228 (0.03746791387271298) | > loss_mel_ce: 3.193960428237915 (nan) | > loss: 0.012784342281520367 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1364036290524146) | > loader_time: 0.0045 (0.004669351906850667)  --> TIME: 2025-02-24 21:38:38 -- STEP: 15206/36938 -- GLOBAL_STEP: 495400 | > loss_text_ce: 0.03941671922802925 (0.03746940961246516) | > loss_mel_ce: 3.201969623565674 (nan) | > loss: 0.012862645089626312 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.136404969763413) | > loader_time: 0.0045 (0.004669517086600518)  --> TIME: 2025-02-24 21:38:48 -- STEP: 15256/36938 -- GLOBAL_STEP: 495450 | > loss_text_ce: 0.0363534651696682 (0.037467483060427575) | > loss_mel_ce: 3.5796875953674316 (nan) | > loss: 0.014349369332194328 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.13642012189993535) | > loader_time: 0.0046 (0.004669477927728531)  --> TIME: 2025-02-24 21:38:57 -- STEP: 15306/36938 -- GLOBAL_STEP: 495500 | > loss_text_ce: 0.02731543965637684 (0.03746954157428127) | > loss_mel_ce: 3.0939812660217285 (nan) | > loss: 0.012386098504066467 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13643614214322955) | > loader_time: 0.0046 (0.0046694518911375785)  --> TIME: 2025-02-24 21:39:06 -- STEP: 15356/36938 -- GLOBAL_STEP: 495550 | > loss_text_ce: 0.034362733364105225 (0.03747069660751773) | > loss_mel_ce: 3.233258008956909 (nan) | > loss: 0.01296675018966198 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13642686218853844) | > loader_time: 0.0046 (0.004669473813393043)  --> TIME: 2025-02-24 21:39:16 -- STEP: 15406/36938 -- GLOBAL_STEP: 495600 | > loss_text_ce: 0.033508460968732834 (0.03747006699633403) | > loss_mel_ce: 3.2187654972076416 (nan) | > loss: 0.012905850075185299 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.1364280831329413) | > loader_time: 0.0048 (0.004669779355721263)  --> TIME: 2025-02-24 21:39:25 -- STEP: 15456/36938 -- GLOBAL_STEP: 495650 | > loss_text_ce: 0.025845054537057877 (0.03746181066925527) | > loss_mel_ce: 3.0895040035247803 (nan) | > loss: 0.012362496927380562 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.1364391294942376) | > loader_time: 0.0046 (0.00466966428495094)  --> TIME: 2025-02-24 21:39:34 -- STEP: 15506/36938 -- GLOBAL_STEP: 495700 | > loss_text_ce: 0.029941720888018608 (0.037460442261793625) | > loss_mel_ce: 3.0227739810943604 (nan) | > loss: 0.012113952077925205 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13643033398699847) | > loader_time: 0.0046 (0.004669455394550511)  --> TIME: 2025-02-24 21:39:44 -- STEP: 15556/36938 -- GLOBAL_STEP: 495750 | > loss_text_ce: 0.03787246719002724 (0.03745609873758017) | > loss_mel_ce: 3.312469959259033 (nan) | > loss: 0.013295010663568974 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.13645112893059577) | > loader_time: 0.0048 (0.004669635192874026)  --> TIME: 2025-02-24 21:39:53 -- STEP: 15606/36938 -- GLOBAL_STEP: 495800 | > loss_text_ce: 0.02538468688726425 (0.037458459498160436) | > loss_mel_ce: 2.0904018878936768 (nan) | > loss: 0.008395979180932045 (nan) | > current_lr: 5e-06 | > step_time: 0.1081 (0.13645002613705512) | > loader_time: 0.0045 (0.004669469212257407)  --> TIME: 2025-02-24 21:40:03 -- STEP: 15656/36938 -- GLOBAL_STEP: 495850 | > loss_text_ce: 0.03480802848935127 (0.037455353883688) | > loss_mel_ce: 2.52726149559021 (nan) | > loss: 0.010166943073272705 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13644809388176202) | > loader_time: 0.0045 (0.004669272144290327)  --> TIME: 2025-02-24 21:40:12 -- STEP: 15706/36938 -- GLOBAL_STEP: 495900 | > loss_text_ce: 0.027585064992308617 (0.03745636178226111) | > loss_mel_ce: 2.523458957672119 (nan) | > loss: 0.010123191401362419 (nan) | > current_lr: 5e-06 | > step_time: 0.1755 (0.13644295699365871) | > loader_time: 0.0048 (0.004668982745765278)  --> TIME: 2025-02-24 21:40:21 -- STEP: 15756/36938 -- GLOBAL_STEP: 495950 | > loss_text_ce: 0.03640017285943031 (0.03745162495169344) | > loss_mel_ce: 2.213818311691284 (nan) | > loss: 0.008929438889026642 (nan) | > current_lr: 5e-06 | > step_time: 0.1082 (0.13644382258483156) | > loader_time: 0.0044 (0.004668924947110601)  --> TIME: 2025-02-24 21:40:30 -- STEP: 15806/36938 -- GLOBAL_STEP: 496000 | > loss_text_ce: 0.045831579715013504 (0.03745113535693794) | > loss_mel_ce: 2.5690200328826904 (nan) | > loss: 0.010376396588981152 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13643108884337543) | > loader_time: 0.0044 (0.00466890237338328)  --> TIME: 2025-02-24 21:40:40 -- STEP: 15856/36938 -- GLOBAL_STEP: 496050 | > loss_text_ce: 0.029810311272740364 (0.03744841615478415) | > loss_mel_ce: 0.060302734375 (nan) | > loss: 0.0003575914597604424 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.13643912554508886) | > loader_time: 0.0046 (0.0046687380727436795)  --> TIME: 2025-02-24 21:40:48 -- STEP: 15906/36938 -- GLOBAL_STEP: 496100 | > loss_text_ce: 0.0526590570807457 (0.037446688813754916) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13641720063638607) | > loader_time: 0.0045 (0.004668384082179741)  --> TIME: 2025-02-24 21:40:57 -- STEP: 15956/36938 -- GLOBAL_STEP: 496150 | > loss_text_ce: 0.028079688549041748 (0.03744730619403375) | > loss_mel_ce: 2.3152365684509277 (nan) | > loss: 0.009298874996602535 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13640452022987512) | > loader_time: 0.0045 (0.004668289615743937)  --> TIME: 2025-02-24 21:41:06 -- STEP: 16006/36938 -- GLOBAL_STEP: 496200 | > loss_text_ce: 0.03642633929848671 (0.03744029587981435) | > loss_mel_ce: 2.9669084548950195 (nan) | > loss: 0.01191799622029066 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.13637554462322657) | > loader_time: 0.0049 (0.004668113679777414)  --> TIME: 2025-02-24 21:41:15 -- STEP: 16056/36938 -- GLOBAL_STEP: 496250 | > loss_text_ce: 0.03653280436992645 (0.03744231262709477) | > loss_mel_ce: 2.763021230697632 (nan) | > loss: 0.01110934279859066 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13637692590049585) | > loader_time: 0.0045 (0.00466807349202165)  --> TIME: 2025-02-24 21:41:25 -- STEP: 16106/36938 -- GLOBAL_STEP: 496300 | > loss_text_ce: 0.03644094243645668 (0.03744233780294536) | > loss_mel_ce: 2.577065944671631 (nan) | > loss: 0.01037105917930603 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13637926026958472) | > loader_time: 0.0046 (0.00466801176363675)  --> TIME: 2025-02-24 21:41:34 -- STEP: 16156/36938 -- GLOBAL_STEP: 496350 | > loss_text_ce: 0.030031954869627953 (0.03743925914898147) | > loss_mel_ce: 3.174792766571045 (nan) | > loss: 0.01271755900233984 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1363820221849693) | > loader_time: 0.0046 (0.00466811534466498)  --> TIME: 2025-02-24 21:41:44 -- STEP: 16206/36938 -- GLOBAL_STEP: 496400 | > loss_text_ce: 0.03809535875916481 (0.03743956813277949) | > loss_mel_ce: 3.6785264015197754 (nan) | > loss: 0.014748500660061836 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13637796890642848) | > loader_time: 0.0049 (0.004667996374542403)  --> TIME: 2025-02-24 21:41:53 -- STEP: 16256/36938 -- GLOBAL_STEP: 496450 | > loss_text_ce: 0.026180468499660492 (0.03743753900049538) | > loss_mel_ce: 2.835993766784668 (nan) | > loss: 0.011357835493981838 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.1363751124003966) | > loader_time: 0.0046 (0.004668052506259097)  --> TIME: 2025-02-24 21:42:02 -- STEP: 16306/36938 -- GLOBAL_STEP: 496500 | > loss_text_ce: 0.02876036986708641 (0.03743945993328851) | > loss_mel_ce: 2.6255593299865723 (nan) | > loss: 0.010533015243709087 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13638264117000434) | > loader_time: 0.0046 (0.00466814775723087)  --> TIME: 2025-02-24 21:42:12 -- STEP: 16356/36938 -- GLOBAL_STEP: 496550 | > loss_text_ce: 0.04689914733171463 (0.03744511208638197) | > loss_mel_ce: 2.9674246311187744 (nan) | > loss: 0.011961602605879307 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13637924105417198) | > loader_time: 0.0045 (0.004667977754397439)  --> TIME: 2025-02-24 21:42:21 -- STEP: 16406/36938 -- GLOBAL_STEP: 496600 | > loss_text_ce: 0.03501308709383011 (0.03744353685026441) | > loss_mel_ce: 3.341141700744629 (nan) | > loss: 0.013397441245615482 (nan) | > current_lr: 5e-06 | > step_time: 0.1466 (0.1363752130094298) | > loader_time: 0.0046 (0.004667808540736719)  --> TIME: 2025-02-24 21:42:30 -- STEP: 16456/36938 -- GLOBAL_STEP: 496650 | > loss_text_ce: 0.0320141427218914 (0.037441034157426265) | > loss_mel_ce: 2.1813459396362305 (nan) | > loss: 0.008783175610005856 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13636906715232092) | > loader_time: 0.0047 (0.004667746061599528)  --> TIME: 2025-02-24 21:42:39 -- STEP: 16506/36938 -- GLOBAL_STEP: 496700 | > loss_text_ce: 0.04386713355779648 (0.037443327359703016) | > loss_mel_ce: 3.264301300048828 (nan) | > loss: 0.013127652928233147 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13636962952620094) | > loader_time: 0.0047 (0.004667598002609526)  --> TIME: 2025-02-24 21:42:49 -- STEP: 16556/36938 -- GLOBAL_STEP: 496750 | > loss_text_ce: 0.026466261595487595 (0.0374421290617335) | > loss_mel_ce: 3.1773509979248047 (nan) | > loss: 0.01271356176584959 (nan) | > current_lr: 5e-06 | > step_time: 0.181 (0.13636742736383603) | > loader_time: 0.0048 (0.0046674381076612665)  --> TIME: 2025-02-24 21:42:58 -- STEP: 16606/36938 -- GLOBAL_STEP: 496800 | > loss_text_ce: 0.045358046889305115 (0.0374403105132635) | > loss_mel_ce: 1.34881591796875 (nan) | > loss: 0.005532436538487673 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13637710352712287) | > loader_time: 0.0045 (0.0046672598218452585)  --> TIME: 2025-02-24 21:43:07 -- STEP: 16656/36938 -- GLOBAL_STEP: 496850 | > loss_text_ce: 0.034959886223077774 (0.037438791370360144) | > loss_mel_ce: 2.6415276527404785 (nan) | > loss: 0.010620982386171818 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13638369763389813) | > loader_time: 0.0046 (0.004667019308701251)  --> TIME: 2025-02-24 21:43:17 -- STEP: 16706/36938 -- GLOBAL_STEP: 496900 | > loss_text_ce: 0.03467608243227005 (0.03743936165549126) | > loss_mel_ce: 3.4156155586242676 (nan) | > loss: 0.013691633939743042 (nan) | > current_lr: 5e-06 | > step_time: 0.1556 (0.13637369400895677) | > loader_time: 0.0047 (0.0046670275163781735)  --> TIME: 2025-02-24 21:43:26 -- STEP: 16756/36938 -- GLOBAL_STEP: 496950 | > loss_text_ce: 0.03721662983298302 (0.037438803080955804) | > loss_mel_ce: 2.5348904132843018 (nan) | > loss: 0.010206774808466434 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13637912987695067) | > loader_time: 0.0045 (0.004667006591306028)  --> TIME: 2025-02-24 21:43:35 -- STEP: 16806/36938 -- GLOBAL_STEP: 497000 | > loss_text_ce: 0.03444220498204231 (0.03743532377766341) | > loss_mel_ce: 0.0054800850339233875 (nan) | > loss: 0.00015842179709579796 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13637175481120437) | > loader_time: 0.0045 (0.004666840265467658)  --> TIME: 2025-02-24 21:43:44 -- STEP: 16856/36938 -- GLOBAL_STEP: 497050 | > loss_text_ce: 0.04010346904397011 (0.037431991714688015) | > loss_mel_ce: 2.9141995906829834 (nan) | > loss: 0.01172342523932457 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.1363634198946461) | > loader_time: 0.0046 (0.004666738929938629)  --> TIME: 2025-02-24 21:43:53 -- STEP: 16906/36938 -- GLOBAL_STEP: 497100 | > loss_text_ce: 0.038780201226472855 (0.03742655319613766) | > loss_mel_ce: 0.08760172873735428 (nan) | > loss: 0.0005015156348235905 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13636231916015262) | > loader_time: 0.0048 (0.004666804407075292)  --> TIME: 2025-02-24 21:44:03 -- STEP: 16956/36938 -- GLOBAL_STEP: 497150 | > loss_text_ce: 0.03637898340821266 (0.037428164781374645) | > loss_mel_ce: 1.2851613759994507 (nan) | > loss: 0.005244208034127951 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13638211003899836) | > loader_time: 0.0049 (0.0046667848648004615)  --> TIME: 2025-02-24 21:44:13 -- STEP: 17006/36938 -- GLOBAL_STEP: 497200 | > loss_text_ce: 0.04580480977892876 (0.037430247643400166) | > loss_mel_ce: 3.5477540493011475 (nan) | > loss: 0.014260155148804188 (nan) | > current_lr: 5e-06 | > step_time: 0.1499 (0.13638485685090668) | > loader_time: 0.0047 (0.004666711475713712)  --> TIME: 2025-02-24 21:44:22 -- STEP: 17056/36938 -- GLOBAL_STEP: 497250 | > loss_text_ce: 0.028680600225925446 (0.037428461263453366) | > loss_mel_ce: 3.1955792903900146 (nan) | > loss: 0.012794682756066322 (nan) | > current_lr: 5e-06 | > step_time: 0.1791 (0.13639429594629374) | > loader_time: 0.0048 (0.004666562459482251)  --> TIME: 2025-02-24 21:44:31 -- STEP: 17106/36938 -- GLOBAL_STEP: 497300 | > loss_text_ce: 0.0340845100581646 (0.037427303114433146) | > loss_mel_ce: 2.825801134109497 (nan) | > loss: 0.011348753236234188 (nan) | > current_lr: 5e-06 | > step_time: 0.1523 (0.136402930806954) | > loader_time: 0.0046 (0.00466634441673664)  --> TIME: 2025-02-24 21:44:41 -- STEP: 17156/36938 -- GLOBAL_STEP: 497350 | > loss_text_ce: 0.02773953601717949 (0.03742555944136336) | > loss_mel_ce: 2.9629523754119873 (nan) | > loss: 0.011867825873196125 (nan) | > current_lr: 5e-06 | > step_time: 0.1577 (0.13639571877119455) | > loader_time: 0.0046 (0.004666131327663099)  --> TIME: 2025-02-24 21:44:50 -- STEP: 17206/36938 -- GLOBAL_STEP: 497400 | > loss_text_ce: 0.03472384810447693 (0.03742622869222615) | > loss_mel_ce: 2.7944188117980957 (nan) | > loss: 0.011226756498217583 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.13638778371920546) | > loader_time: 0.0046 (0.00466627725733998)  --> TIME: 2025-02-24 21:44:59 -- STEP: 17256/36938 -- GLOBAL_STEP: 497450 | > loss_text_ce: 0.040840763598680496 (0.03742338538712376) | > loss_mel_ce: 2.412971258163452 (nan) | > loss: 0.009737350046634674 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13639585153615405) | > loader_time: 0.0046 (0.004666634605170723)  --> TIME: 2025-02-24 21:45:08 -- STEP: 17306/36938 -- GLOBAL_STEP: 497500 | > loss_text_ce: 0.04325086995959282 (0.037423844888729275) | > loss_mel_ce: 3.040820837020874 (nan) | > loss: 0.012238380499184132 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13639119414529224) | > loader_time: 0.0047 (0.0046666791559464155)  --> TIME: 2025-02-24 21:45:17 -- STEP: 17356/36938 -- GLOBAL_STEP: 497550 | > loss_text_ce: 0.03013339452445507 (0.037424589523218796) | > loss_mel_ce: 2.4305646419525146 (nan) | > loss: 0.009764675982296467 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13637990291065202) | > loader_time: 0.0048 (0.0046665988970692905)  --> TIME: 2025-02-24 21:45:27 -- STEP: 17406/36938 -- GLOBAL_STEP: 497600 | > loss_text_ce: 0.03465086966753006 (0.03742678297082617) | > loss_mel_ce: 3.8493964672088623 (nan) | > loss: 0.015412886627018452 (nan) | > current_lr: 5e-06 | > step_time: 0.1612 (0.13638949476410925) | > loader_time: 0.0046 (0.004666565205055604)  --> TIME: 2025-02-24 21:45:36 -- STEP: 17456/36938 -- GLOBAL_STEP: 497650 | > loss_text_ce: 0.04507288336753845 (0.037423681339031775) | > loss_mel_ce: 1.7938504219055176 (nan) | > loss: 0.007297315169125795 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13639858660077198) | > loader_time: 0.0046 (0.004666547877432328)  --> TIME: 2025-02-24 21:45:46 -- STEP: 17506/36938 -- GLOBAL_STEP: 497700 | > loss_text_ce: 0.03788390010595322 (0.03742613478567026) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13639466643782763) | > loader_time: 0.0046 (0.004666537744419014)  --> TIME: 2025-02-24 21:45:55 -- STEP: 17556/36938 -- GLOBAL_STEP: 497750 | > loss_text_ce: 0.030775882303714752 (0.03742155145140147) | > loss_mel_ce: 1.9136936664581299 (nan) | > loss: 0.007716149557381868 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.1364004022169548) | > loader_time: 0.0045 (0.004666480517762201)  --> TIME: 2025-02-24 21:46:04 -- STEP: 17606/36938 -- GLOBAL_STEP: 497800 | > loss_text_ce: 0.03203628584742546 (0.037420259331434794) | > loss_mel_ce: 2.629836320877075 (nan) | > loss: 0.010562987066805363 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13639638979949742) | > loader_time: 0.0045 (0.004666309268396294)  --> TIME: 2025-02-24 21:46:14 -- STEP: 17656/36938 -- GLOBAL_STEP: 497850 | > loss_text_ce: 0.05270586907863617 (0.037423997517541406) | > loss_mel_ce: 2.5016555786132812 (nan) | > loss: 0.010136354714632034 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.1364094604854938) | > loader_time: 0.0047 (0.00466624970451219)  --> TIME: 2025-02-24 21:46:23 -- STEP: 17706/36938 -- GLOBAL_STEP: 497900 | > loss_text_ce: 0.03505028411746025 (0.03742790942730553) | > loss_mel_ce: 3.070364236831665 (nan) | > loss: 0.012323074974119663 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.1364208667490765) | > loader_time: 0.0046 (0.004666238723603196)  --> TIME: 2025-02-24 21:46:32 -- STEP: 17756/36938 -- GLOBAL_STEP: 497950 | > loss_text_ce: 0.017923396080732346 (0.03742449400470453) | > loss_mel_ce: 2.2189745903015137 (nan) | > loss: 0.008876579813659191 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13640823576949224) | > loader_time: 0.0045 (0.004666223494312079)  --> TIME: 2025-02-24 21:46:41 -- STEP: 17806/36938 -- GLOBAL_STEP: 498000 | > loss_text_ce: 0.027797171846032143 (0.0374258343761867) | > loss_mel_ce: 2.9420325756073 (nan) | > loss: 0.01178503967821598 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.1363956074921926) | > loader_time: 0.0047 (0.004666038300232378)  --> TIME: 2025-02-24 21:46:51 -- STEP: 17856/36938 -- GLOBAL_STEP: 498050 | > loss_text_ce: 0.04088358208537102 (0.037424459716454944) | > loss_mel_ce: 3.508172035217285 (nan) | > loss: 0.014083554968237877 (nan) | > current_lr: 5e-06 | > step_time: 0.1774 (0.1363932223035862) | > loader_time: 0.0048 (0.004665887550747934)  --> TIME: 2025-02-24 21:47:00 -- STEP: 17906/36938 -- GLOBAL_STEP: 498100 | > loss_text_ce: 0.036038875579833984 (0.0374228064318555) | > loss_mel_ce: 2.996391773223877 (nan) | > loss: 0.01203345600515604 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.1364099067455553) | > loader_time: 0.0048 (0.0046659162373938355)  --> TIME: 2025-02-24 21:47:10 -- STEP: 17956/36938 -- GLOBAL_STEP: 498150 | > loss_text_ce: 0.03687858209013939 (0.037421845243272814) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.136403719442515) | > loader_time: 0.0045 (0.004665816512716497)  --> TIME: 2025-02-24 21:47:19 -- STEP: 18006/36938 -- GLOBAL_STEP: 498200 | > loss_text_ce: 0.033373601734638214 (0.037420791662837634) | > loss_mel_ce: 3.067037343978882 (nan) | > loss: 0.012303219176828861 (nan) | > current_lr: 5e-06 | > step_time: 0.1264 (0.13639396730295975) | > loader_time: 0.0047 (0.004665637723369134)  --> TIME: 2025-02-24 21:47:28 -- STEP: 18056/36938 -- GLOBAL_STEP: 498250 | > loss_text_ce: 0.04219271242618561 (0.03741888620765447) | > loss_mel_ce: 2.750945568084717 (nan) | > loss: 0.011083882302045822 (nan) | > current_lr: 5e-06 | > step_time: 0.1558 (0.1363924788884681) | > loader_time: 0.0046 (0.004665590515694456)  --> TIME: 2025-02-24 21:47:37 -- STEP: 18106/36938 -- GLOBAL_STEP: 498300 | > loss_text_ce: 0.03609400615096092 (0.037421974919456195) | > loss_mel_ce: 1.389434814453125 (nan) | > loss: 0.005656860303133726 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13639320247965667) | > loader_time: 0.0045 (0.004665543344894388)  --> TIME: 2025-02-24 21:47:47 -- STEP: 18156/36938 -- GLOBAL_STEP: 498350 | > loss_text_ce: 0.047336895018815994 (0.03742573593977467) | > loss_mel_ce: 3.4552805423736572 (nan) | > loss: 0.013899276033043861 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.13640045306839552) | > loader_time: 0.0048 (0.004665546465560853)  --> TIME: 2025-02-24 21:47:56 -- STEP: 18206/36938 -- GLOBAL_STEP: 498400 | > loss_text_ce: 0.03258493170142174 (0.03742156482922572) | > loss_mel_ce: 3.1031227111816406 (nan) | > loss: 0.012443284504115582 (nan) | > current_lr: 5e-06 | > step_time: 0.1615 (0.13640781664447518) | > loader_time: 0.0046 (0.004665470118052759)  --> TIME: 2025-02-24 21:48:06 -- STEP: 18256/36938 -- GLOBAL_STEP: 498450 | > loss_text_ce: 0.035236671566963196 (0.03742572314290194) | > loss_mel_ce: 2.856259346008301 (nan) | > loss: 0.011474191211163998 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13641017218308865) | > loader_time: 0.0049 (0.0046656115680906735)  --> TIME: 2025-02-24 21:48:15 -- STEP: 18306/36938 -- GLOBAL_STEP: 498500 | > loss_text_ce: 0.037329018115997314 (0.037427623367898975) | > loss_mel_ce: 3.4249629974365234 (nan) | > loss: 0.01373925432562828 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.13641149776562436) | > loader_time: 0.0047 (0.004665641710170286)  --> TIME: 2025-02-24 21:48:24 -- STEP: 18356/36938 -- GLOBAL_STEP: 498550 | > loss_text_ce: 0.03484770655632019 (0.037425047103313534) | > loss_mel_ce: 3.1600139141082764 (nan) | > loss: 0.012678023427724838 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13641729900898691) | > loader_time: 0.0045 (0.004665610226028269)  --> TIME: 2025-02-24 21:48:34 -- STEP: 18406/36938 -- GLOBAL_STEP: 498600 | > loss_text_ce: 0.03771916404366493 (0.03742753121597904) | > loss_mel_ce: 3.4053986072540283 (nan) | > loss: 0.013663167133927345 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.13642044502872938) | > loader_time: 0.0049 (0.0046655387965464)  --> TIME: 2025-02-24 21:48:43 -- STEP: 18456/36938 -- GLOBAL_STEP: 498650 | > loss_text_ce: 0.04773828387260437 (0.0374260140956538) | > loss_mel_ce: 2.893521547317505 (nan) | > loss: 0.011671666987240314 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1364188014194976) | > loader_time: 0.0046 (0.004665401238200448)  --> TIME: 2025-02-24 21:48:52 -- STEP: 18506/36938 -- GLOBAL_STEP: 498700 | > loss_text_ce: 0.03355895355343819 (0.037420850930765456) | > loss_mel_ce: 3.242959499359131 (nan) | > loss: 0.013002057559788227 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13641820890019868) | > loader_time: 0.0047 (0.004665455379628301)  --> TIME: 2025-02-24 21:49:01 -- STEP: 18556/36938 -- GLOBAL_STEP: 498750 | > loss_text_ce: 0.04946473613381386 (0.03742085198040048) | > loss_mel_ce: 3.3551454544067383 (nan) | > loss: 0.013510358519852161 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13640484037160766) | > loader_time: 0.0048 (0.004665537997293264)  --> TIME: 2025-02-24 21:49:11 -- STEP: 18606/36938 -- GLOBAL_STEP: 498800 | > loss_text_ce: 0.037523526698350906 (0.03742044878664867) | > loss_mel_ce: 2.992213249206543 (nan) | > loss: 0.012022765353322029 (nan) | > current_lr: 5e-06 | > step_time: 0.1902 (0.13642142158060483) | > loader_time: 0.0048 (0.004665607267152371)  --> TIME: 2025-02-24 21:49:21 -- STEP: 18656/36938 -- GLOBAL_STEP: 498850 | > loss_text_ce: 0.04991504177451134 (0.03742855304143656) | > loss_mel_ce: 3.3517379760742188 (nan) | > loss: 0.013498623855412006 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.13643529111484265) | > loader_time: 0.0047 (0.004665514655526244)  --> TIME: 2025-02-24 21:49:30 -- STEP: 18706/36938 -- GLOBAL_STEP: 498900 | > loss_text_ce: 0.03894520550966263 (0.03742802150395757) | > loss_mel_ce: 2.805072069168091 (nan) | > loss: 0.011285783722996712 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.13642888428322866) | > loader_time: 0.0049 (0.004665383091460272)  --> TIME: 2025-02-24 21:49:39 -- STEP: 18756/36938 -- GLOBAL_STEP: 498950 | > loss_text_ce: 0.03810779005289078 (0.03743170478841419) | > loss_mel_ce: 3.2157111167907715 (nan) | > loss: 0.012911980971693993 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.13644603349222315) | > loader_time: 0.0048 (0.004665452360115351)  --> TIME: 2025-02-24 21:49:49 -- STEP: 18806/36938 -- GLOBAL_STEP: 499000 | > loss_text_ce: 0.03902854770421982 (0.03743295056875559) | > loss_mel_ce: 3.260007858276367 (nan) | > loss: 0.013091415166854858 (nan) | > current_lr: 5e-06 | > step_time: 0.1876 (0.13644418145422818) | > loader_time: 0.0048 (0.0046653497806067356)  --> TIME: 2025-02-24 21:49:58 -- STEP: 18856/36938 -- GLOBAL_STEP: 499050 | > loss_text_ce: 0.03521862253546715 (0.037435778863289526) | > loss_mel_ce: 2.781144380569458 (nan) | > loss: 0.011176045052707195 (nan) | > current_lr: 5e-06 | > step_time: 0.1708 (0.13644396387574784) | > loader_time: 0.0048 (0.004665390902471871)  --> TIME: 2025-02-24 21:50:07 -- STEP: 18906/36938 -- GLOBAL_STEP: 499100 | > loss_text_ce: 0.04404328390955925 (0.037432676426811036) | > loss_mel_ce: 2.7709736824035645 (nan) | > loss: 0.011170702986419201 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13642440843012105) | > loader_time: 0.0045 (0.004665191622747875)  --> TIME: 2025-02-24 21:50:16 -- STEP: 18956/36938 -- GLOBAL_STEP: 499150 | > loss_text_ce: 0.04296938329935074 (0.03743032906711499) | > loss_mel_ce: 3.287327766418457 (nan) | > loss: 0.013215466402471066 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.1364243609111851) | > loader_time: 0.0047 (0.00466503467356561)  --> TIME: 2025-02-24 21:50:25 -- STEP: 19006/36938 -- GLOBAL_STEP: 499200 | > loss_text_ce: 0.029233640059828758 (0.037434529840058774) | > loss_mel_ce: 3.2156684398651123 (nan) | > loss: 0.012876596301794052 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13641769245249616) | > loader_time: 0.0046 (0.004665097738207971)  --> TIME: 2025-02-24 21:50:34 -- STEP: 19056/36938 -- GLOBAL_STEP: 499250 | > loss_text_ce: 0.03110269457101822 (0.03743243600925288) | > loss_mel_ce: 2.854278087615967 (nan) | > loss: 0.011449924670159817 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13640491400417565) | > loader_time: 0.0046 (0.004665047005381916)  --> TIME: 2025-02-24 21:50:43 -- STEP: 19106/36938 -- GLOBAL_STEP: 499300 | > loss_text_ce: 0.022637778893113136 (0.03742912689370714) | > loss_mel_ce: 3.144442558288574 (nan) | > loss: 0.01256777998059988 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13640657314065693) | > loader_time: 0.0045 (0.004665046340693191)  --> TIME: 2025-02-24 21:50:53 -- STEP: 19156/36938 -- GLOBAL_STEP: 499350 | > loss_text_ce: 0.040524378418922424 (0.03743085710945946) | > loss_mel_ce: 2.6841092109680176 (nan) | > loss: 0.010812039487063885 (nan) | > current_lr: 5e-06 | > step_time: 0.1776 (0.13640761704005744) | > loader_time: 0.005 (0.004664961979070056)  --> TIME: 2025-02-24 21:51:02 -- STEP: 19206/36938 -- GLOBAL_STEP: 499400 | > loss_text_ce: 0.038192469626665115 (0.03743195920062688) | > loss_mel_ce: 3.2203643321990967 (nan) | > loss: 0.012930781580507755 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.1364143460067274) | > loader_time: 0.0046 (0.004665000692179047)  --> TIME: 2025-02-24 21:51:11 -- STEP: 19256/36938 -- GLOBAL_STEP: 499450 | > loss_text_ce: 0.043047718703746796 (0.037432072908777446) | > loss_mel_ce: 1.6574532985687256 (nan) | > loss: 0.0067480201832950115 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1364081502703643) | > loader_time: 0.0046 (0.004664861269397346)  --> TIME: 2025-02-24 21:51:21 -- STEP: 19306/36938 -- GLOBAL_STEP: 499500 | > loss_text_ce: 0.03693738952279091 (0.037435644393508806) | > loss_mel_ce: 2.8175342082977295 (nan) | > loss: 0.011327269487082958 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13640308480157512) | > loader_time: 0.0046 (0.004664760481615955)  --> TIME: 2025-02-24 21:51:30 -- STEP: 19356/36938 -- GLOBAL_STEP: 499550 | > loss_text_ce: 0.03251054883003235 (0.03743730694322895) | > loss_mel_ce: 1.010502815246582 (nan) | > loss: 0.004138941876590252 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1364069317559313) | > loader_time: 0.0045 (0.004664615046068941)  --> TIME: 2025-02-24 21:51:39 -- STEP: 19406/36938 -- GLOBAL_STEP: 499600 | > loss_text_ce: 0.0326722152531147 (0.03743653548495568) | > loss_mel_ce: 3.4221231937408447 (nan) | > loss: 0.013709506019949913 (nan) | > current_lr: 5e-06 | > step_time: 0.1907 (0.136406296530323) | > loader_time: 0.0048 (0.004664516530060985)  --> TIME: 2025-02-24 21:51:49 -- STEP: 19456/36938 -- GLOBAL_STEP: 499650 | > loss_text_ce: 0.035699740052223206 (0.037435467200626474) | > loss_mel_ce: 2.9778761863708496 (nan) | > loss: 0.011958635412156582 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.1364129443628417) | > loader_time: 0.0046 (0.004664515966157357)  --> TIME: 2025-02-24 21:51:58 -- STEP: 19506/36938 -- GLOBAL_STEP: 499700 | > loss_text_ce: 0.03559160605072975 (0.037436315258507546) | > loss_mel_ce: 3.084832191467285 (nan) | > loss: 0.012382634915411472 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13640829605725194) | > loader_time: 0.0046 (0.004664387554312221)  --> TIME: 2025-02-24 21:52:07 -- STEP: 19556/36938 -- GLOBAL_STEP: 499750 | > loss_text_ce: 0.03453715145587921 (0.03743992242706867) | > loss_mel_ce: 3.3456621170043945 (nan) | > loss: 0.013413489796221256 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.13640679568692318) | > loader_time: 0.0048 (0.004664330583429181)  --> TIME: 2025-02-24 21:52:17 -- STEP: 19606/36938 -- GLOBAL_STEP: 499800 | > loss_text_ce: 0.0381406806409359 (0.03744187997772843) | > loss_mel_ce: 3.372265577316284 (nan) | > loss: 0.01353335939347744 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.13640692313568872) | > loader_time: 0.0048 (0.004664176473274113)  --> TIME: 2025-02-24 21:52:26 -- STEP: 19656/36938 -- GLOBAL_STEP: 499850 | > loss_text_ce: 0.03517094627022743 (0.03744050907046916) | > loss_mel_ce: 3.468736410140991 (nan) | > loss: 0.013904395513236523 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.1364023644457414) | > loader_time: 0.0046 (0.004663987643940904)  --> TIME: 2025-02-24 21:52:35 -- STEP: 19706/36938 -- GLOBAL_STEP: 499900 | > loss_text_ce: 0.02185012958943844 (0.03743652985416611) | > loss_mel_ce: 1.7058480978012085 (nan) | > loss: 0.006855945568531752 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1364001365866038) | > loader_time: 0.0045 (0.004663846824999837)  --> TIME: 2025-02-24 21:52:44 -- STEP: 19756/36938 -- GLOBAL_STEP: 499950 | > loss_text_ce: 0.03424433246254921 (0.0374352344978635) | > loss_mel_ce: 0.12664794921875 (nan) | > loss: 0.0006384614389389753 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13640213283022057) | > loader_time: 0.0045 (0.004663718654260264)  --> TIME: 2025-02-24 21:52:54 -- STEP: 19806/36938 -- GLOBAL_STEP: 500000 | > loss_text_ce: 0.040154412388801575 (0.037435034599503605) | > loss_mel_ce: 2.787024974822998 (nan) | > loss: 0.011218966916203499 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.1364041396876502) | > loader_time: 0.0047 (0.004663590396352193) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_500000.pth  --> TIME: 2025-02-24 21:53:09 -- STEP: 19856/36938 -- GLOBAL_STEP: 500050 | > loss_text_ce: 0.0353827029466629 (0.037435671574866575) | > loss_mel_ce: 1.3085289001464844 (nan) | > loss: 0.00533298309892416 (nan) | > current_lr: 5e-06 | > step_time: 0.1069 (0.1363992699685349) | > loader_time: 0.0045 (0.00466358993055355)  --> TIME: 2025-02-24 21:53:18 -- STEP: 19906/36938 -- GLOBAL_STEP: 500100 | > loss_text_ce: 0.024041645228862762 (0.03743954619913391) | > loss_mel_ce: 0.0804850235581398 (nan) | > loss: 0.00041478840284980834 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13639364398976173) | > loader_time: 0.0045 (0.0046636797034846815)  --> TIME: 2025-02-24 21:53:27 -- STEP: 19956/36938 -- GLOBAL_STEP: 500150 | > loss_text_ce: 0.03256932273507118 (0.03743793226638132) | > loss_mel_ce: 1.3815985918045044 (nan) | > loss: 0.005611777771264315 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.1363992018173007) | > loader_time: 0.0046 (0.004663612984014788)  --> TIME: 2025-02-24 21:53:36 -- STEP: 20006/36938 -- GLOBAL_STEP: 500200 | > loss_text_ce: 0.041846297681331635 (0.03744185768191856) | > loss_mel_ce: 3.179424285888672 (nan) | > loss: 0.012782820500433445 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13639059174505283) | > loader_time: 0.0046 (0.004663452570121119)  --> TIME: 2025-02-24 21:53:45 -- STEP: 20056/36938 -- GLOBAL_STEP: 500250 | > loss_text_ce: 0.027264725416898727 (0.037439412211282) | > loss_mel_ce: 3.1065289974212646 (nan) | > loss: 0.012435690499842167 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13638338068590009) | > loader_time: 0.0048 (0.004663432635869144)  --> TIME: 2025-02-24 21:53:54 -- STEP: 20106/36938 -- GLOBAL_STEP: 500300 | > loss_text_ce: 0.030295347794890404 (0.037439969719898936) | > loss_mel_ce: 2.8480212688446045 (nan) | > loss: 0.01142189186066389 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13637833614011588) | > loader_time: 0.0045 (0.004663222229541192)  --> TIME: 2025-02-24 21:54:04 -- STEP: 20156/36938 -- GLOBAL_STEP: 500350 | > loss_text_ce: 0.04046670347452164 (0.03744587525996136) | > loss_mel_ce: 0.5595703125 (nan) | > loss: 0.0023810993880033493 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13636864975204152) | > loader_time: 0.0046 (0.0046631506355680535)  --> TIME: 2025-02-24 21:54:13 -- STEP: 20206/36938 -- GLOBAL_STEP: 500400 | > loss_text_ce: 0.03983965888619423 (0.03744971251305473) | > loss_mel_ce: 3.7490234375 (nan) | > loss: 0.015035171993076801 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.1363657356358672) | > loader_time: 0.0046 (0.004663174817624137)  --> TIME: 2025-02-24 21:54:22 -- STEP: 20256/36938 -- GLOBAL_STEP: 500450 | > loss_text_ce: 0.026441268622875214 (0.03744810435533457) | > loss_mel_ce: 1.012917399406433 (nan) | > loss: 0.00412443932145834 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13636272397086432) | > loader_time: 0.0045 (0.004663090134778074)  --> TIME: 2025-02-24 21:54:31 -- STEP: 20306/36938 -- GLOBAL_STEP: 500500 | > loss_text_ce: 0.040483392775058746 (0.03744531609874749) | > loss_mel_ce: 3.3988969326019287 (nan) | > loss: 0.013648335821926594 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13636456954940568) | > loader_time: 0.0046 (0.004663067616395328)  --> TIME: 2025-02-24 21:54:41 -- STEP: 20356/36938 -- GLOBAL_STEP: 500550 | > loss_text_ce: 0.03547624126076698 (0.03744357694495828) | > loss_mel_ce: 2.5287277698516846 (nan) | > loss: 0.010175413452088833 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13637036108553596) | > loader_time: 0.0047 (0.004662973259071081)  --> TIME: 2025-02-24 21:54:50 -- STEP: 20406/36938 -- GLOBAL_STEP: 500600 | > loss_text_ce: 0.03501977398991585 (0.037441486053288855) | > loss_mel_ce: 2.76149845123291 (nan) | > loss: 0.011097295209765434 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13636749079095795) | > loader_time: 0.0044 (0.0046628260512007235)  --> TIME: 2025-02-24 21:55:00 -- STEP: 20456/36938 -- GLOBAL_STEP: 500650 | > loss_text_ce: 0.03859284892678261 (0.03744314102541382) | > loss_mel_ce: 2.8435065746307373 (nan) | > loss: 0.011436902917921543 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13637045235618853) | > loader_time: 0.0045 (0.004662724213997337)  --> TIME: 2025-02-24 21:55:09 -- STEP: 20506/36938 -- GLOBAL_STEP: 500700 | > loss_text_ce: 0.02623489312827587 (0.037443959002540966) | > loss_mel_ce: 2.5087506771087646 (nan) | > loss: 0.01005946658551693 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.1363741260475724) | > loader_time: 0.0049 (0.00466270409804207)  --> TIME: 2025-02-24 21:55:18 -- STEP: 20556/36938 -- GLOBAL_STEP: 500750 | > loss_text_ce: 0.03775428235530853 (0.03744574595159552) | > loss_mel_ce: 2.887669563293457 (nan) | > loss: 0.01160882506519556 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1363694103317601) | > loader_time: 0.0045 (0.004662565450581084)  --> TIME: 2025-02-24 21:55:27 -- STEP: 20606/36938 -- GLOBAL_STEP: 500800 | > loss_text_ce: 0.046521157026290894 (0.03744602539668516) | > loss_mel_ce: 2.357160806655884 (nan) | > loss: 0.009538421407341957 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.13635779857218883) | > loader_time: 0.0044 (0.004662403166670105)  --> TIME: 2025-02-24 21:55:37 -- STEP: 20656/36938 -- GLOBAL_STEP: 500850 | > loss_text_ce: 0.04385531693696976 (0.03744265495856828) | > loss_mel_ce: 2.298474073410034 (nan) | > loss: 0.009294958785176277 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1363657379593608) | > loader_time: 0.0045 (0.004662290584847109)  --> TIME: 2025-02-24 21:55:46 -- STEP: 20706/36938 -- GLOBAL_STEP: 500900 | > loss_text_ce: 0.04728909954428673 (0.03744152607247812) | > loss_mel_ce: 3.4305009841918945 (nan) | > loss: 0.013800755143165588 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.13637867879374635) | > loader_time: 0.0047 (0.004662226228152979)  --> TIME: 2025-02-24 21:55:56 -- STEP: 20756/36938 -- GLOBAL_STEP: 500950 | > loss_text_ce: 0.01201701257377863 (0.037441627968396625) | > loss_mel_ce: 1.5530599355697632 (nan) | > loss: 0.006210623309016228 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.1363843351367066) | > loader_time: 0.0044 (0.0046621329822529025)  --> TIME: 2025-02-24 21:56:05 -- STEP: 20806/36938 -- GLOBAL_STEP: 501000 | > loss_text_ce: 0.035425808280706406 (0.03743596673006403) | > loss_mel_ce: 2.6508872509002686 (nan) | > loss: 0.01065997313708067 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13638440413072936) | > loader_time: 0.0046 (0.004662020096672965)  --> TIME: 2025-02-24 21:56:14 -- STEP: 20856/36938 -- GLOBAL_STEP: 501050 | > loss_text_ce: 0.03447457402944565 (0.03743510735689184) | > loss_mel_ce: 2.68404221534729 (nan) | > loss: 0.010787765495479107 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.1363678421492212) | > loader_time: 0.0047 (0.004661812583834058)  --> TIME: 2025-02-24 21:56:23 -- STEP: 20906/36938 -- GLOBAL_STEP: 501100 | > loss_text_ce: 0.02766535058617592 (0.03743494220350977) | > loss_mel_ce: 1.401123046875 (nan) | > loss: 0.0056697954423725605 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13636594424005402) | > loader_time: 0.0044 (0.00466170814360519)  --> TIME: 2025-02-24 21:56:33 -- STEP: 20956/36938 -- GLOBAL_STEP: 501150 | > loss_text_ce: 0.035022612661123276 (0.03743422646406127) | > loss_mel_ce: 1.1942535638809204 (nan) | > loss: 0.004878080449998379 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.1363656762720713) | > loader_time: 0.0046 (0.004661577613464293)  --> TIME: 2025-02-24 21:56:42 -- STEP: 21006/36938 -- GLOBAL_STEP: 501200 | > loss_text_ce: 0.03277153521776199 (0.03743316180921397) | > loss_mel_ce: 2.8863890171051025 (nan) | > loss: 0.011583970859646797 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13636225999383306) | > loader_time: 0.0047 (0.0046614213953650886)  --> TIME: 2025-02-24 21:56:51 -- STEP: 21056/36938 -- GLOBAL_STEP: 501250 | > loss_text_ce: 0.03391001373529434 (0.03743814782269215) | > loss_mel_ce: 2.5052669048309326 (nan) | > loss: 0.010076100006699562 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13636465548427157) | > loader_time: 0.0044 (0.004661277797084432)  --> TIME: 2025-02-24 21:57:01 -- STEP: 21106/36938 -- GLOBAL_STEP: 501300 | > loss_text_ce: 0.037543848156929016 (0.03743701665696199) | > loss_mel_ce: 2.998499631881714 (nan) | > loss: 0.012047791853547096 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.13636372887578652) | > loader_time: 0.0047 (0.004661153292052704)  --> TIME: 2025-02-24 21:57:10 -- STEP: 21156/36938 -- GLOBAL_STEP: 501350 | > loss_text_ce: 0.022960031405091286 (0.037436212592357236) | > loss_mel_ce: 1.3313627243041992 (nan) | > loss: 0.00537429703399539 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13635021874124212) | > loader_time: 0.0045 (0.004661197922905468)  --> TIME: 2025-02-24 21:57:19 -- STEP: 21206/36938 -- GLOBAL_STEP: 501400 | > loss_text_ce: 0.03880641609430313 (0.03743764354531068) | > loss_mel_ce: 2.943203926086426 (nan) | > loss: 0.011833375319838524 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13634338943753305) | > loader_time: 0.0046 (0.004661136040943431)  --> TIME: 2025-02-24 21:57:28 -- STEP: 21256/36938 -- GLOBAL_STEP: 501450 | > loss_text_ce: 0.043955497443675995 (0.037441294847107384) | > loss_mel_ce: 3.454291820526123 (nan) | > loss: 0.013881934806704521 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13634642701782373) | > loader_time: 0.0045 (0.004660967937931761)  --> TIME: 2025-02-24 21:57:38 -- STEP: 21306/36938 -- GLOBAL_STEP: 501500 | > loss_text_ce: 0.03717273101210594 (0.03744316178356583) | > loss_mel_ce: 3.2452261447906494 (nan) | > loss: 0.013025393709540367 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13635089586992422) | > loader_time: 0.0045 (0.004660836544482659)  --> TIME: 2025-02-24 21:57:47 -- STEP: 21356/36938 -- GLOBAL_STEP: 501550 | > loss_text_ce: 0.036727141588926315 (0.03744370203327756) | > loss_mel_ce: 0.008874943479895592 (nan) | > loss: 0.000180960661964491 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.1363575202870788) | > loader_time: 0.0045 (0.004660748423965551)  --> TIME: 2025-02-24 21:57:56 -- STEP: 21406/36938 -- GLOBAL_STEP: 501600 | > loss_text_ce: 0.04128400608897209 (0.03744493428956019) | > loss_mel_ce: 2.9972333908081055 (nan) | > loss: 0.012057609856128693 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13635087808492846) | > loader_time: 0.0047 (0.004660536426924459)  --> TIME: 2025-02-24 21:58:05 -- STEP: 21456/36938 -- GLOBAL_STEP: 501650 | > loss_text_ce: 0.03767213225364685 (0.037443561813502974) | > loss_mel_ce: 3.2429776191711426 (nan) | > loss: 0.013018451631069183 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13634201680920374) | > loader_time: 0.0045 (0.004660326418016112)  --> TIME: 2025-02-24 21:58:15 -- STEP: 21506/36938 -- GLOBAL_STEP: 501700 | > loss_text_ce: 0.04018104821443558 (0.03743988199830376) | > loss_mel_ce: 2.6771910190582275 (nan) | > loss: 0.010783223435282707 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13634007248469907) | > loader_time: 0.0047 (0.004660176441501593)  --> TIME: 2025-02-24 21:58:24 -- STEP: 21556/36938 -- GLOBAL_STEP: 501750 | > loss_text_ce: 0.014296566136181355 (0.037439395944596596) | > loss_mel_ce: 1.4463850259780884 (nan) | > loss: 0.005796355661004782 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13634517901150212) | > loader_time: 0.0045 (0.004660076468126667)  --> TIME: 2025-02-24 21:58:33 -- STEP: 21606/36938 -- GLOBAL_STEP: 501800 | > loss_text_ce: 0.03846849501132965 (0.0374384299292625) | > loss_mel_ce: 3.021631956100464 (nan) | > loss: 0.012143257074058056 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1363494293365066) | > loader_time: 0.0044 (0.004660035607594729)  --> TIME: 2025-02-24 21:58:43 -- STEP: 21656/36938 -- GLOBAL_STEP: 501850 | > loss_text_ce: 0.026972366496920586 (0.03743994034680354) | > loss_mel_ce: 2.870861053466797 (nan) | > loss: 0.01149933971464634 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1363533465546069) | > loader_time: 0.0046 (0.0046599196647866435)  --> TIME: 2025-02-24 21:58:52 -- STEP: 21706/36938 -- GLOBAL_STEP: 501900 | > loss_text_ce: 0.03901561722159386 (0.03743518677675783) | > loss_mel_ce: 2.456369638442993 (nan) | > loss: 0.00990232266485691 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.1363575512348167) | > loader_time: 0.0045 (0.004659841546789984)  --> TIME: 2025-02-24 21:59:02 -- STEP: 21756/36938 -- GLOBAL_STEP: 501950 | > loss_text_ce: 0.03559819981455803 (0.03743432225236712) | > loss_mel_ce: 0.2144775390625 (nan) | > loss: 0.0009923640172928572 (nan) | > current_lr: 5e-06 | > step_time: 0.1782 (0.13636814766584035) | > loader_time: 0.0049 (0.004659797157253199)  --> TIME: 2025-02-24 21:59:11 -- STEP: 21806/36938 -- GLOBAL_STEP: 502000 | > loss_text_ce: 0.037500664591789246 (0.037432022078194516) | > loss_mel_ce: 3.4788002967834473 (nan) | > loss: 0.013953575864434242 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13636583185497536) | > loader_time: 0.0048 (0.004659915390249035)  --> TIME: 2025-02-24 21:59:20 -- STEP: 21856/36938 -- GLOBAL_STEP: 502050 | > loss_text_ce: 0.049867287278175354 (0.03742557020733839) | > loss_mel_ce: 0.0522054024040699 (nan) | > loss: 0.00040505037759430707 (nan) | > current_lr: 5e-06 | > step_time: 0.1529 (0.13635680006372308) | > loader_time: 0.0051 (0.004659833345633101)  --> TIME: 2025-02-24 21:59:30 -- STEP: 21906/36938 -- GLOBAL_STEP: 502100 | > loss_text_ce: 0.031979892402887344 (0.03741994867519482) | > loss_mel_ce: 3.230368137359619 (nan) | > loss: 0.012945826165378094 (nan) | > current_lr: 5e-06 | > step_time: 0.1988 (0.13636325931130985) | > loader_time: 0.0048 (0.004659840704305696)  --> TIME: 2025-02-24 21:59:39 -- STEP: 21956/36938 -- GLOBAL_STEP: 502150 | > loss_text_ce: 0.041508764028549194 (0.03741918638566433) | > loss_mel_ce: 3.516018867492676 (nan) | > loss: 0.014117173850536346 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.13636582231973188) | > loader_time: 0.0049 (0.0046597980023907)  --> TIME: 2025-02-24 21:59:48 -- STEP: 22006/36938 -- GLOBAL_STEP: 502200 | > loss_text_ce: 0.028877373784780502 (0.037419838808482944) | > loss_mel_ce: 2.6051368713378906 (nan) | > loss: 0.010452437214553356 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13636210706248342) | > loader_time: 0.0045 (0.004659851334327483)  --> TIME: 2025-02-24 21:59:57 -- STEP: 22056/36938 -- GLOBAL_STEP: 502250 | > loss_text_ce: 0.0447295717895031 (0.03742157465598907) | > loss_mel_ce: 3.5980610847473145 (nan) | > loss: 0.014455518685281277 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13635984320419026) | > loader_time: 0.0045 (0.004659747424487121)  --> TIME: 2025-02-24 22:00:07 -- STEP: 22106/36938 -- GLOBAL_STEP: 502300 | > loss_text_ce: 0.03826526924967766 (0.037422043159634294) | > loss_mel_ce: 2.206977605819702 (nan) | > loss: 0.008909693919122219 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1363556633100854) | > loader_time: 0.0047 (0.004659738463427047)  --> TIME: 2025-02-24 22:00:16 -- STEP: 22156/36938 -- GLOBAL_STEP: 502350 | > loss_text_ce: 0.03735737502574921 (0.03741757739525285) | > loss_mel_ce: 2.6377735137939453 (nan) | > loss: 0.01061559934169054 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1363599695847711) | > loader_time: 0.0045 (0.0046596476415753365)  --> TIME: 2025-02-24 22:00:26 -- STEP: 22206/36938 -- GLOBAL_STEP: 502400 | > loss_text_ce: 0.03891268000006676 (0.03741768488923323) | > loss_mel_ce: 2.919747829437256 (nan) | > loss: 0.011740717105567455 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13637227207049005) | > loader_time: 0.0046 (0.004659613360051321)  --> TIME: 2025-02-24 22:00:35 -- STEP: 22256/36938 -- GLOBAL_STEP: 502450 | > loss_text_ce: 0.027211498469114304 (0.03741321444981246) | > loss_mel_ce: 2.683067798614502 (nan) | > loss: 0.010755077004432678 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1363767648047447) | > loader_time: 0.0045 (0.004659562156750582)  --> TIME: 2025-02-24 22:00:44 -- STEP: 22306/36938 -- GLOBAL_STEP: 502500 | > loss_text_ce: 0.042772360146045685 (0.03741294589513407) | > loss_mel_ce: 0.009033203125 (nan) | > loss: 0.00020557764219120145 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13637094106266168) | > loader_time: 0.0048 (0.004659390007018338)  --> TIME: 2025-02-24 22:00:54 -- STEP: 22356/36938 -- GLOBAL_STEP: 502550 | > loss_text_ce: 0.029646029695868492 (0.037414615841646945) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13637255428941641) | > loader_time: 0.0046 (0.004659251996967039)  --> TIME: 2025-02-24 22:01:03 -- STEP: 22406/36938 -- GLOBAL_STEP: 502600 | > loss_text_ce: 0.043104007840156555 (0.03741401907133538) | > loss_mel_ce: 2.7872390747070312 (nan) | > loss: 0.011231520213186741 (nan) | > current_lr: 5e-06 | > step_time: 0.1846 (0.1363700999645811) | > loader_time: 0.0048 (0.004659068698299068)  --> TIME: 2025-02-24 22:01:12 -- STEP: 22456/36938 -- GLOBAL_STEP: 502650 | > loss_text_ce: 0.02927354909479618 (0.03741245421525051) | > loss_mel_ce: 3.5409903526306152 (nan) | > loss: 0.01416771486401558 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13637516210416223) | > loader_time: 0.0046 (0.004659019280550171)  --> TIME: 2025-02-24 22:01:22 -- STEP: 22506/36938 -- GLOBAL_STEP: 502700 | > loss_text_ce: 0.03643230348825455 (0.037412742402627513) | > loss_mel_ce: 2.0747108459472656 (nan) | > loss: 0.008377552963793278 (nan) | > current_lr: 5e-06 | > step_time: 0.1615 (0.1363714633491787) | > loader_time: 0.0046 (0.004658892897725624)  --> TIME: 2025-02-24 22:01:31 -- STEP: 22556/36938 -- GLOBAL_STEP: 502750 | > loss_text_ce: 0.03568742051720619 (0.037415517904787435) | > loss_mel_ce: 2.9526314735412598 (nan) | > loss: 0.011858409270644188 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.13636568424908263) | > loader_time: 0.0046 (0.004658747657982853)  --> TIME: 2025-02-24 22:01:40 -- STEP: 22606/36938 -- GLOBAL_STEP: 502800 | > loss_text_ce: 0.04158494994044304 (0.03741424140619825) | > loss_mel_ce: 2.621950149536133 (nan) | > loss: 0.010569584555923939 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13636978882493522) | > loader_time: 0.0046 (0.004658749375018033)  --> TIME: 2025-02-24 22:01:50 -- STEP: 22656/36938 -- GLOBAL_STEP: 502850 | > loss_text_ce: 0.035816073417663574 (0.03741445293067768) | > loss_mel_ce: 3.139908790588379 (nan) | > loss: 0.012602084316313267 (nan) | > current_lr: 5e-06 | > step_time: 0.1747 (0.13638832647416355) | > loader_time: 0.0048 (0.004658701782256841)  --> TIME: 2025-02-24 22:01:59 -- STEP: 22706/36938 -- GLOBAL_STEP: 502900 | > loss_text_ce: 0.03428227826952934 (0.03741408327187819) | > loss_mel_ce: 2.747608184814453 (nan) | > loss: 0.011039248667657375 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.13637753175783823) | > loader_time: 0.0049 (0.004658778470003303)  --> TIME: 2025-02-24 22:02:08 -- STEP: 22756/36938 -- GLOBAL_STEP: 502950 | > loss_text_ce: 0.030814001336693764 (0.03741043753088539) | > loss_mel_ce: 2.9250524044036865 (nan) | > loss: 0.011729628778994083 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13636915801928653) | > loader_time: 0.0048 (0.0046588060180716125)  --> TIME: 2025-02-24 22:02:17 -- STEP: 22806/36938 -- GLOBAL_STEP: 503000 | > loss_text_ce: 0.03698509931564331 (0.03741045494409574) | > loss_mel_ce: 2.8803551197052 (nan) | > loss: 0.01157674752175808 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.1363760730859135) | > loader_time: 0.0046 (0.004658698366575441)  --> TIME: 2025-02-24 22:02:27 -- STEP: 22856/36938 -- GLOBAL_STEP: 503050 | > loss_text_ce: 0.04341031610965729 (0.037408683311262736) | > loss_mel_ce: 2.8427529335021973 (nan) | > loss: 0.011453029699623585 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13636816626542875) | > loader_time: 0.0047 (0.004658722047168192)  --> TIME: 2025-02-24 22:02:36 -- STEP: 22906/36938 -- GLOBAL_STEP: 503100 | > loss_text_ce: 0.03899507224559784 (0.037412491684487674) | > loss_mel_ce: 2.054295063018799 (nan) | > loss: 0.008306707255542278 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13636790019897532) | > loader_time: 0.0044 (0.004658695465506869)  --> TIME: 2025-02-24 22:02:45 -- STEP: 22956/36938 -- GLOBAL_STEP: 503150 | > loss_text_ce: 0.03363608568906784 (0.037411607629816435) | > loss_mel_ce: 3.249504327774048 (nan) | > loss: 0.013028335757553577 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1363695746079639) | > loader_time: 0.0045 (0.004658618098373767)  --> TIME: 2025-02-24 22:02:55 -- STEP: 23006/36938 -- GLOBAL_STEP: 503200 | > loss_text_ce: 0.03495292738080025 (0.03740952474559053) | > loss_mel_ce: 0.004852294921875 (nan) | > loss: 0.00015795724175404757 (nan) | > current_lr: 5e-06 | > step_time: 0.1757 (0.13638549181395665) | > loader_time: 0.0048 (0.004658585329278771)  --> TIME: 2025-02-24 22:03:04 -- STEP: 23056/36938 -- GLOBAL_STEP: 503250 | > loss_text_ce: 0.025404345244169235 (0.037408391893601886) | > loss_mel_ce: 0.118408203125 (nan) | > loss: 0.0005706847878172994 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13639255991046376) | > loader_time: 0.0045 (0.004658488620080325)  --> TIME: 2025-02-24 22:03:14 -- STEP: 23106/36938 -- GLOBAL_STEP: 503300 | > loss_text_ce: 0.0521262101829052 (0.03740837920691197) | > loss_mel_ce: 1.574174165725708 (nan) | > loss: 0.006453573238104582 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13638961665982832) | > loader_time: 0.0046 (0.004658454797448041)  --> TIME: 2025-02-24 22:03:23 -- STEP: 23156/36938 -- GLOBAL_STEP: 503350 | > loss_text_ce: 0.029852764680981636 (0.03740736255859794) | > loss_mel_ce: 1.8585216999053955 (nan) | > loss: 0.007493549957871437 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13638717721710567) | > loader_time: 0.0045 (0.004658380059673669)  --> TIME: 2025-02-24 22:03:32 -- STEP: 23206/36938 -- GLOBAL_STEP: 503400 | > loss_text_ce: 0.038165703415870667 (0.037407674021054764) | > loss_mel_ce: 3.04555606842041 (nan) | > loss: 0.01223699189722538 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13638915770487034) | > loader_time: 0.0047 (0.004658253338997941)  --> TIME: 2025-02-24 22:03:41 -- STEP: 23256/36938 -- GLOBAL_STEP: 503450 | > loss_text_ce: 0.03931408375501633 (0.037408526648924294) | > loss_mel_ce: 1.5222020149230957 (nan) | > loss: 0.006196492817252874 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13638726290985426) | > loader_time: 0.0046 (0.0046580903075755554)  --> TIME: 2025-02-24 22:03:51 -- STEP: 23306/36938 -- GLOBAL_STEP: 503500 | > loss_text_ce: 0.04421307146549225 (0.0374069751469604) | > loss_mel_ce: 0.01153564453125 (nan) | > loss: 0.00022122506925370544 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13638312844445083) | > loader_time: 0.0045 (0.004658003554347826)  --> TIME: 2025-02-24 22:04:01 -- STEP: 23356/36938 -- GLOBAL_STEP: 503550 | > loss_text_ce: 0.039737358689308167 (0.03740639988355924) | > loss_mel_ce: 3.322550058364868 (nan) | > loss: 0.013342411257326603 (nan) | > current_lr: 5e-06 | > step_time: 0.1691 (0.13640772315365043) | > loader_time: 0.0049 (0.004658041638980082)  --> TIME: 2025-02-24 22:04:10 -- STEP: 23406/36938 -- GLOBAL_STEP: 503600 | > loss_text_ce: 0.038132380694150925 (0.037403916003995355) | > loss_mel_ce: 2.8620011806488037 (nan) | > loss: 0.011508467607200146 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.1364086663163752) | > loader_time: 0.0047 (0.004657998712900314)  --> TIME: 2025-02-24 22:04:19 -- STEP: 23456/36938 -- GLOBAL_STEP: 503650 | > loss_text_ce: 0.034049272537231445 (0.03740540991172272) | > loss_mel_ce: 3.3336780071258545 (nan) | > loss: 0.01336399745196104 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13640989526841837) | > loader_time: 0.0046 (0.004657936027072754)  --> TIME: 2025-02-24 22:04:29 -- STEP: 23506/36938 -- GLOBAL_STEP: 503700 | > loss_text_ce: 0.03237418457865715 (0.037404985774787056) | > loss_mel_ce: 2.3469817638397217 (nan) | > loss: 0.009441888891160488 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.136403606037965) | > loader_time: 0.0047 (0.004657815053068802)  --> TIME: 2025-02-24 22:04:38 -- STEP: 23556/36938 -- GLOBAL_STEP: 503750 | > loss_text_ce: 0.03047332726418972 (0.03740741583257868) | > loss_mel_ce: 3.028081178665161 (nan) | > loss: 0.012137121520936489 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.1364102409739777) | > loader_time: 0.0047 (0.004657778964218341)  --> TIME: 2025-02-24 22:04:48 -- STEP: 23606/36938 -- GLOBAL_STEP: 503800 | > loss_text_ce: 0.03914114832878113 (0.037406692264637074) | > loss_mel_ce: 2.1751465797424316 (nan) | > loss: 0.00878685712814331 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13641364015098956) | > loader_time: 0.0045 (0.0046576758133176)  --> TIME: 2025-02-24 22:04:57 -- STEP: 23656/36938 -- GLOBAL_STEP: 503850 | > loss_text_ce: 0.04171394556760788 (0.03740492040158482) | > loss_mel_ce: 3.4021120071411133 (nan) | > loss: 0.01366597693413496 (nan) | > current_lr: 5e-06 | > step_time: 0.1457 (0.13641687879942793) | > loader_time: 0.0047 (0.004657515932830686)  --> TIME: 2025-02-24 22:05:07 -- STEP: 23706/36938 -- GLOBAL_STEP: 503900 | > loss_text_ce: 0.03340977057814598 (0.037402662024763375) | > loss_mel_ce: 3.3031983375549316 (nan) | > loss: 0.013240509666502476 (nan) | > current_lr: 5e-06 | > step_time: 0.1879 (0.13642452115762202) | > loader_time: 0.005 (0.004657404891229454)  --> TIME: 2025-02-24 22:05:16 -- STEP: 23756/36938 -- GLOBAL_STEP: 503950 | > loss_text_ce: 0.028065184131264687 (0.03740348281549847) | > loss_mel_ce: 1.9210786819458008 (nan) | > loss: 0.0077346982434391975 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13642491153603523) | > loader_time: 0.0045 (0.004657335836571394)  --> TIME: 2025-02-24 22:05:25 -- STEP: 23806/36938 -- GLOBAL_STEP: 504000 | > loss_text_ce: 0.029118966311216354 (0.03740472380971007) | > loss_mel_ce: 2.5365452766418457 (nan) | > loss: 0.01018120814114809 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13642102201015052) | > loader_time: 0.0045 (0.004657167382053184)  --> TIME: 2025-02-24 22:05:35 -- STEP: 23856/36938 -- GLOBAL_STEP: 504050 | > loss_text_ce: 0.037812214344739914 (0.03740263999307629) | > loss_mel_ce: 2.5318498611450195 (nan) | > loss: 0.01019707228988409 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13642784632307517) | > loader_time: 0.0047 (0.004657166584636573)  --> TIME: 2025-02-24 22:05:44 -- STEP: 23906/36938 -- GLOBAL_STEP: 504100 | > loss_text_ce: 0.04070825129747391 (0.03740092228517105) | > loss_mel_ce: 2.890418529510498 (nan) | > loss: 0.011631456203758717 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1364259251508601) | > loader_time: 0.0045 (0.00465736713886537)  --> TIME: 2025-02-24 22:05:53 -- STEP: 23956/36938 -- GLOBAL_STEP: 504150 | > loss_text_ce: 0.03695252165198326 (0.037398529753332986) | > loss_mel_ce: 3.2801530361175537 (nan) | > loss: 0.013163117691874504 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.13643518650067848) | > loader_time: 0.0047 (0.004657446920105371)  --> TIME: 2025-02-24 22:06:03 -- STEP: 24006/36938 -- GLOBAL_STEP: 504200 | > loss_text_ce: 0.035298850387334824 (0.037398440925596244) | > loss_mel_ce: 3.3235018253326416 (nan) | > loss: 0.01332857459783554 (nan) | > current_lr: 5e-06 | > step_time: 0.1578 (0.13643235031251094) | > loader_time: 0.0048 (0.004657397794195116)  --> TIME: 2025-02-24 22:06:12 -- STEP: 24056/36938 -- GLOBAL_STEP: 504250 | > loss_text_ce: 0.035762641578912735 (0.037394549464032166) | > loss_mel_ce: 2.658369779586792 (nan) | > loss: 0.010691002011299133 (nan) | > current_lr: 5e-06 | > step_time: 0.1572 (0.1364277280720899) | > loader_time: 0.0046 (0.004657327811663248)  --> TIME: 2025-02-24 22:06:21 -- STEP: 24106/36938 -- GLOBAL_STEP: 504300 | > loss_text_ce: 0.04497751593589783 (0.037396473185125054) | > loss_mel_ce: 3.4087133407592773 (nan) | > loss: 0.013705123215913773 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13642392200641382) | > loader_time: 0.0046 (0.004657285367563245)  --> TIME: 2025-02-24 22:06:30 -- STEP: 24156/36938 -- GLOBAL_STEP: 504350 | > loss_text_ce: 0.033737871795892715 (0.03739396925481358) | > loss_mel_ce: 2.3951258659362793 (nan) | > loss: 0.0096383485943079 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.1364204629325444) | > loader_time: 0.0044 (0.00465711909308496)  --> TIME: 2025-02-24 22:06:40 -- STEP: 24206/36938 -- GLOBAL_STEP: 504400 | > loss_text_ce: 0.03391152620315552 (0.03739387990403076) | > loss_mel_ce: 3.5473692417144775 (nan) | > loss: 0.014211432076990604 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.13642438309237656) | > loader_time: 0.0046 (0.004657087449760313)  --> TIME: 2025-02-24 22:06:49 -- STEP: 24256/36938 -- GLOBAL_STEP: 504450 | > loss_text_ce: 0.03297977149486542 (0.03739669076530995) | > loss_mel_ce: 3.070484161376953 (nan) | > loss: 0.01231533382087946 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13641929435777053) | > loader_time: 0.0044 (0.004656999457951868)  --> TIME: 2025-02-24 22:06:58 -- STEP: 24306/36938 -- GLOBAL_STEP: 504500 | > loss_text_ce: 0.033006127923727036 (0.03739668236357045) | > loss_mel_ce: 3.0106194019317627 (nan) | > loss: 0.012077880091965199 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1364256099723518) | > loader_time: 0.0046 (0.004656953153654927)  --> TIME: 2025-02-24 22:07:08 -- STEP: 24356/36938 -- GLOBAL_STEP: 504550 | > loss_text_ce: 0.03463822603225708 (0.03739635099082199) | > loss_mel_ce: 3.29280948638916 (nan) | > loss: 0.013204158283770084 (nan) | > current_lr: 5e-06 | > step_time: 0.1786 (0.13642938983011563) | > loader_time: 0.0049 (0.004656849833109563)  --> TIME: 2025-02-24 22:07:17 -- STEP: 24406/36938 -- GLOBAL_STEP: 504600 | > loss_text_ce: 0.037257492542266846 (0.037394588118604695) | > loss_mel_ce: 2.9463226795196533 (nan) | > loss: 0.011839604005217552 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13643299510495896) | > loader_time: 0.0046 (0.004656815571852542)  --> TIME: 2025-02-24 22:07:26 -- STEP: 24456/36938 -- GLOBAL_STEP: 504650 | > loss_text_ce: 0.04183287173509598 (0.03739444705755715) | > loss_mel_ce: 3.209608316421509 (nan) | > loss: 0.012902545742690563 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1364278202845368) | > loader_time: 0.0046 (0.004656711180770834)  --> TIME: 2025-02-24 22:07:36 -- STEP: 24506/36938 -- GLOBAL_STEP: 504700 | > loss_text_ce: 0.035343192517757416 (0.0373940478135794) | > loss_mel_ce: 3.1386125087738037 (nan) | > loss: 0.01259506307542324 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13643294179292578) | > loader_time: 0.0046 (0.004656716608411545)  --> TIME: 2025-02-24 22:07:45 -- STEP: 24556/36938 -- GLOBAL_STEP: 504750 | > loss_text_ce: 0.029315998777747154 (0.03739371650807766) | > loss_mel_ce: 2.890625 (nan) | > loss: 0.011587067507207394 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13642391554208994) | > loader_time: 0.0047 (0.004656589134136731)  --> TIME: 2025-02-24 22:07:54 -- STEP: 24606/36938 -- GLOBAL_STEP: 504800 | > loss_text_ce: 0.04067521169781685 (0.037393296379309354) | > loss_mel_ce: 2.4002151489257812 (nan) | > loss: 0.009686073288321495 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13642072853960216) | > loader_time: 0.0045 (0.004656453312077949)  --> TIME: 2025-02-24 22:08:03 -- STEP: 24656/36938 -- GLOBAL_STEP: 504850 | > loss_text_ce: 0.03392564132809639 (0.037391853890164285) | > loss_mel_ce: 2.6867897510528564 (nan) | > loss: 0.010796490125358105 (nan) | > current_lr: 5e-06 | > step_time: 0.1796 (0.13641454729068372) | > loader_time: 0.0047 (0.0046562998229848895)  --> TIME: 2025-02-24 22:08:13 -- STEP: 24706/36938 -- GLOBAL_STEP: 504900 | > loss_text_ce: 0.03565481677651405 (0.03738844766850934) | > loss_mel_ce: 2.912273645401001 (nan) | > loss: 0.011698129586875439 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13641765088115498) | > loader_time: 0.0046 (0.004656321576063698)  --> TIME: 2025-02-24 22:08:22 -- STEP: 24756/36938 -- GLOBAL_STEP: 504950 | > loss_text_ce: 0.037644051015377045 (0.03738773717006352) | > loss_mel_ce: 1.802025318145752 (nan) | > loss: 0.007300275377929211 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13642652809263492) | > loader_time: 0.0045 (0.004656353796562721)  --> TIME: 2025-02-24 22:08:31 -- STEP: 24806/36938 -- GLOBAL_STEP: 505000 | > loss_text_ce: 0.03488430380821228 (0.03738823887340641) | > loss_mel_ce: 3.505828619003296 (nan) | > loss: 0.014050448313355446 (nan) | > current_lr: 5e-06 | > step_time: 0.1872 (0.1364309892225363) | > loader_time: 0.0048 (0.00465625937327368)  --> TIME: 2025-02-24 22:08:41 -- STEP: 24856/36938 -- GLOBAL_STEP: 505050 | > loss_text_ce: 0.038704097270965576 (0.03739005748950143) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13643084508182474) | > loader_time: 0.0045 (0.004656125791670584)  --> TIME: 2025-02-24 22:08:50 -- STEP: 24906/36938 -- GLOBAL_STEP: 505100 | > loss_text_ce: 0.04182842746376991 (0.03739017625394243) | > loss_mel_ce: 2.40399169921875 (nan) | > loss: 0.009705635719001293 (nan) | > current_lr: 5e-06 | > step_time: 0.1267 (0.1364254245487716) | > loader_time: 0.0046 (0.004656043127723563)  --> TIME: 2025-02-24 22:08:59 -- STEP: 24956/36938 -- GLOBAL_STEP: 505150 | > loss_text_ce: 0.03937414661049843 (0.0373888566306264) | > loss_mel_ce: 3.2980151176452637 (nan) | > loss: 0.013243609108030796 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13641918533829248) | > loader_time: 0.0046 (0.004655904304830254)  --> TIME: 2025-02-24 22:09:08 -- STEP: 25006/36938 -- GLOBAL_STEP: 505200 | > loss_text_ce: 0.033935874700546265 (0.037383264032614515) | > loss_mel_ce: 3.2416369915008545 (nan) | > loss: 0.012998305261135101 (nan) | > current_lr: 5e-06 | > step_time: 0.1788 (0.13641274614180735) | > loader_time: 0.005 (0.004655751868895304)  --> TIME: 2025-02-24 22:09:17 -- STEP: 25056/36938 -- GLOBAL_STEP: 505250 | > loss_text_ce: 0.04088599979877472 (0.03738242022494317) | > loss_mel_ce: 2.8980445861816406 (nan) | > loss: 0.011662423610687256 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13640988306980878) | > loader_time: 0.0046 (0.00465562526674255)  --> TIME: 2025-02-24 22:09:27 -- STEP: 25106/36938 -- GLOBAL_STEP: 505300 | > loss_text_ce: 0.03782664239406586 (0.037380950850322996) | > loss_mel_ce: 1.9846670627593994 (nan) | > loss: 0.008025769144296646 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.1364189384848356) | > loader_time: 0.0046 (0.004655546233405751)  --> TIME: 2025-02-24 22:09:36 -- STEP: 25156/36938 -- GLOBAL_STEP: 505350 | > loss_text_ce: 0.03877171501517296 (0.03738327862921826) | > loss_mel_ce: 3.1217589378356934 (nan) | > loss: 0.012541788630187511 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.1364157568727921) | > loader_time: 0.0046 (0.004655384623126702)  --> TIME: 2025-02-24 22:09:46 -- STEP: 25206/36938 -- GLOBAL_STEP: 505400 | > loss_text_ce: 0.03464037552475929 (0.0373846394158268) | > loss_mel_ce: 2.8772504329681396 (nan) | > loss: 0.011555122211575508 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13642631269093358) | > loader_time: 0.0045 (0.004655291643878938)  --> TIME: 2025-02-24 22:09:55 -- STEP: 25256/36938 -- GLOBAL_STEP: 505450 | > loss_text_ce: 0.03951173648238182 (0.037385006879746684) | > loss_mel_ce: 2.856870174407959 (nan) | > loss: 0.011493579484522343 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13642381122680897) | > loader_time: 0.0045 (0.004655172166319826)  --> TIME: 2025-02-24 22:10:04 -- STEP: 25306/36938 -- GLOBAL_STEP: 505500 | > loss_text_ce: 0.030631713569164276 (0.037383763111591695) | > loss_mel_ce: 0.05103759840130806 (nan) | > loss: 0.0003240845981054008 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13640853236204492) | > loader_time: 0.0045 (0.00465498215161242)  --> TIME: 2025-02-24 22:10:13 -- STEP: 25356/36938 -- GLOBAL_STEP: 505550 | > loss_text_ce: 0.03230345621705055 (0.03738280936217841) | > loss_mel_ce: 3.1150670051574707 (nan) | > loss: 0.012489566579461098 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13641131558916955) | > loader_time: 0.0045 (0.004654836656543619)  --> TIME: 2025-02-24 22:10:22 -- STEP: 25406/36938 -- GLOBAL_STEP: 505600 | > loss_text_ce: 0.03555983304977417 (0.03738444781051495) | > loss_mel_ce: 0.07934778183698654 (nan) | > loss: 0.0004559826338663697 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13641079405128637) | > loader_time: 0.0045 (0.004654800489287935)  --> TIME: 2025-02-24 22:10:32 -- STEP: 25456/36938 -- GLOBAL_STEP: 505650 | > loss_text_ce: 0.03554564341902733 (0.03738419981181594) | > loss_mel_ce: 3.601391077041626 (nan) | > loss: 0.014432289637625217 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13641506086948252) | > loader_time: 0.0047 (0.004654859603013237)  --> TIME: 2025-02-24 22:10:41 -- STEP: 25506/36938 -- GLOBAL_STEP: 505700 | > loss_text_ce: 0.03758126497268677 (0.03738444158759875) | > loss_mel_ce: 0.08332519233226776 (nan) | > loss: 0.0004797875590156764 (nan) | > current_lr: 5e-06 | > step_time: 0.1891 (0.13642076668080827) | > loader_time: 0.0048 (0.004654826271405863)  --> TIME: 2025-02-24 22:10:50 -- STEP: 25556/36938 -- GLOBAL_STEP: 505750 | > loss_text_ce: 0.05174698680639267 (0.0373828474455256) | > loss_mel_ce: 2.226565361022949 (nan) | > loss: 0.009040922857820988 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13640383145752782) | > loader_time: 0.0045 (0.0046546507243688745)  --> TIME: 2025-02-24 22:10:59 -- STEP: 25606/36938 -- GLOBAL_STEP: 505800 | > loss_text_ce: 0.04178624600172043 (0.03738308379675758) | > loss_mel_ce: 2.17466139793396 (nan) | > loss: 0.008795427158474922 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13640976763140275) | > loader_time: 0.0046 (0.004654574792738694)  --> TIME: 2025-02-24 22:11:10 -- STEP: 25656/36938 -- GLOBAL_STEP: 505850 | > loss_text_ce: 0.04057731851935387 (0.03738194674971866) | > loss_mel_ce: 3.1294522285461426 (nan) | > loss: 0.012579482980072498 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.13643702827475268) | > loader_time: 0.0046 (0.00465469492124845)  --> TIME: 2025-02-24 22:11:19 -- STEP: 25706/36938 -- GLOBAL_STEP: 505900 | > loss_text_ce: 0.03369245305657387 (0.037381164763215874) | > loss_mel_ce: 3.2936460971832275 (nan) | > loss: 0.013203725218772888 (nan) | > current_lr: 5e-06 | > step_time: 0.187 (0.13643658235361852) | > loader_time: 0.0049 (0.004654608857610307)  --> TIME: 2025-02-24 22:11:28 -- STEP: 25756/36938 -- GLOBAL_STEP: 505950 | > loss_text_ce: 0.04397271201014519 (0.03738447352669564) | > loss_mel_ce: 3.4705817699432373 (nan) | > loss: 0.013946645893156528 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13642370623078867) | > loader_time: 0.0046 (0.0046544451024113824)  --> TIME: 2025-02-24 22:11:37 -- STEP: 25806/36938 -- GLOBAL_STEP: 506000 | > loss_text_ce: 0.030285894870758057 (0.03738568186877404) | > loss_mel_ce: 2.7979042530059814 (nan) | > loss: 0.011222977191209793 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13642448039514163) | > loader_time: 0.0045 (0.004654307351743894)  --> TIME: 2025-02-24 22:11:47 -- STEP: 25856/36938 -- GLOBAL_STEP: 506050 | > loss_text_ce: 0.03822745010256767 (0.03738745120387083) | > loss_mel_ce: 3.7462167739868164 (nan) | > loss: 0.015017637051641941 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.1364337336364205) | > loader_time: 0.0046 (0.004654354895326724)  --> TIME: 2025-02-24 22:11:56 -- STEP: 25906/36938 -- GLOBAL_STEP: 506100 | > loss_text_ce: 0.03156742453575134 (0.03738705344908802) | > loss_mel_ce: 3.5452332496643066 (nan) | > loss: 0.01419365406036377 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13644098825071055) | > loader_time: 0.0047 (0.004654307094106563)  --> TIME: 2025-02-24 22:12:05 -- STEP: 25956/36938 -- GLOBAL_STEP: 506150 | > loss_text_ce: 0.0339372418820858 (0.03738856618870264) | > loss_mel_ce: 2.8120062351226807 (nan) | > loss: 0.011293427087366581 (nan) | > current_lr: 5e-06 | > step_time: 0.1057 (0.13644292976808772) | > loader_time: 0.0045 (0.004654411129489228)  --> TIME: 2025-02-24 22:12:15 -- STEP: 26006/36938 -- GLOBAL_STEP: 506200 | > loss_text_ce: 0.03206293657422066 (0.03738805236011046) | > loss_mel_ce: 2.3014309406280518 (nan) | > loss: 0.009259897284209728 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13643532417044707) | > loader_time: 0.0046 (0.004654405887682638)  --> TIME: 2025-02-24 22:12:24 -- STEP: 26056/36938 -- GLOBAL_STEP: 506250 | > loss_text_ce: 0.040683139115571976 (0.037389700781664426) | > loss_mel_ce: 2.1814212799072266 (nan) | > loss: 0.008817874826490879 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13644640583694742) | > loader_time: 0.0046 (0.004654472714960805)  --> TIME: 2025-02-24 22:12:33 -- STEP: 26106/36938 -- GLOBAL_STEP: 506300 | > loss_text_ce: 0.03870752826333046 (0.037390894167915856) | > loss_mel_ce: 2.29229998588562 (nan) | > loss: 0.009250029921531677 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.1364434909466201) | > loader_time: 0.0045 (0.004654487293725277)  --> TIME: 2025-02-24 22:12:43 -- STEP: 26156/36938 -- GLOBAL_STEP: 506350 | > loss_text_ce: 0.037228576838970184 (0.037388751636076335) | > loss_mel_ce: 2.795140027999878 (nan) | > loss: 0.011239558458328247 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.136447366351544) | > loader_time: 0.0046 (0.004654446541852799)  --> TIME: 2025-02-24 22:12:52 -- STEP: 26206/36938 -- GLOBAL_STEP: 506400 | > loss_text_ce: 0.03841949999332428 (0.03738580692939265) | > loss_mel_ce: 1.14862060546875 (nan) | > loss: 0.004710476845502853 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13645313647305884) | > loader_time: 0.0046 (0.0046544225399870204)  --> TIME: 2025-02-24 22:13:01 -- STEP: 26256/36938 -- GLOBAL_STEP: 506450 | > loss_text_ce: 0.035068150609731674 (0.037387156005043616) | > loss_mel_ce: 3.1490590572357178 (nan) | > loss: 0.012635425664484501 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13645006541905008) | > loader_time: 0.0045 (0.0046543262758173215)  --> TIME: 2025-02-24 22:13:11 -- STEP: 26306/36938 -- GLOBAL_STEP: 506500 | > loss_text_ce: 0.045732565224170685 (0.03738575125968282) | > loss_mel_ce: 0.14881204068660736 (nan) | > loss: 0.0007720025023445487 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13645128452269384) | > loader_time: 0.0046 (0.004654259389139018)  --> TIME: 2025-02-24 22:13:20 -- STEP: 26356/36938 -- GLOBAL_STEP: 506550 | > loss_text_ce: 0.04096372053027153 (0.03738622725677579) | > loss_mel_ce: 3.5099711418151855 (nan) | > loss: 0.014091012068092823 (nan) | > current_lr: 5e-06 | > step_time: 0.1506 (0.1364547043998063) | > loader_time: 0.0048 (0.00465435105365906)  --> TIME: 2025-02-24 22:13:30 -- STEP: 26406/36938 -- GLOBAL_STEP: 506600 | > loss_text_ce: 0.0442194864153862 (0.03738722900285702) | > loss_mel_ce: 3.7169911861419678 (nan) | > loss: 0.014925439842045307 (nan) | > current_lr: 5e-06 | > step_time: 0.1869 (0.1364631509361937) | > loader_time: 0.0047 (0.004654234208492168)  --> TIME: 2025-02-24 22:13:39 -- STEP: 26456/36938 -- GLOBAL_STEP: 506650 | > loss_text_ce: 0.03792464733123779 (0.037384373157478674) | > loss_mel_ce: 3.598706007003784 (nan) | > loss: 0.014431074261665344 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.13646245092865472) | > loader_time: 0.0046 (0.004654216331984256)  --> TIME: 2025-02-24 22:13:48 -- STEP: 26506/36938 -- GLOBAL_STEP: 506700 | > loss_text_ce: 0.03826187178492546 (0.03738695735488864) | > loss_mel_ce: 2.9142110347747803 (nan) | > loss: 0.011716162785887718 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.1364648130212937) | > loader_time: 0.0046 (0.004654111110569497)  --> TIME: 2025-02-24 22:13:58 -- STEP: 26556/36938 -- GLOBAL_STEP: 506750 | > loss_text_ce: 0.04048123583197594 (0.03738610356582307) | > loss_mel_ce: 3.1323416233062744 (nan) | > loss: 0.012590567581355572 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.1364685384457611) | > loader_time: 0.0048 (0.004653994874398273)  --> TIME: 2025-02-24 22:14:07 -- STEP: 26606/36938 -- GLOBAL_STEP: 506800 | > loss_text_ce: 0.038318730890750885 (0.03738566261165099) | > loss_mel_ce: 3.209679365158081 (nan) | > loss: 0.012888881377875805 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13647645360176275) | > loader_time: 0.0045 (0.004654037999916805)  --> TIME: 2025-02-24 22:14:17 -- STEP: 26656/36938 -- GLOBAL_STEP: 506850 | > loss_text_ce: 0.03655976802110672 (0.03738726649537021) | > loss_mel_ce: 3.044372081756592 (nan) | > loss: 0.012225921265780926 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.13647872696117466) | > loader_time: 0.0047 (0.0046539046720009865)  --> TIME: 2025-02-24 22:14:26 -- STEP: 26706/36938 -- GLOBAL_STEP: 506900 | > loss_text_ce: 0.031695082783699036 (0.03738624802783901) | > loss_mel_ce: 2.3540849685668945 (nan) | > loss: 0.009467381983995438 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13648144712985896) | > loader_time: 0.0045 (0.004653814856161321)  --> TIME: 2025-02-24 22:14:36 -- STEP: 26756/36938 -- GLOBAL_STEP: 506950 | > loss_text_ce: 0.03765855357050896 (0.03738679886457335) | > loss_mel_ce: 2.7186121940612793 (nan) | > loss: 0.010937582701444626 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13648623445497737) | > loader_time: 0.0046 (0.004653771480720527)  --> TIME: 2025-02-24 22:14:45 -- STEP: 26806/36938 -- GLOBAL_STEP: 507000 | > loss_text_ce: 0.0340358130633831 (0.037386845642137324) | > loss_mel_ce: 2.483527183532715 (nan) | > loss: 0.009990330785512924 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13649372743918073) | > loader_time: 0.0046 (0.004653825881204887)  --> TIME: 2025-02-24 22:14:55 -- STEP: 26856/36938 -- GLOBAL_STEP: 507050 | > loss_text_ce: 0.030905453488230705 (0.03738514623375232) | > loss_mel_ce: 2.937988758087158 (nan) | > loss: 0.011781327426433563 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.1364976449100822) | > loader_time: 0.0046 (0.004653810327309428)  --> TIME: 2025-02-24 22:15:04 -- STEP: 26906/36938 -- GLOBAL_STEP: 507100 | > loss_text_ce: 0.04600302502512932 (0.0373837688674297) | > loss_mel_ce: 2.8411457538604736 (nan) | > loss: 0.011456940323114395 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13651152779105555) | > loader_time: 0.0045 (0.004653882831479031)  --> TIME: 2025-02-24 22:15:14 -- STEP: 26956/36938 -- GLOBAL_STEP: 507150 | > loss_text_ce: 0.031235279515385628 (0.03738428056082483) | > loss_mel_ce: 2.0974650382995605 (nan) | > loss: 0.008447223342955112 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13651566704261694) | > loader_time: 0.0045 (0.004653939402657072)  --> TIME: 2025-02-24 22:15:24 -- STEP: 27006/36938 -- GLOBAL_STEP: 507200 | > loss_text_ce: 0.03571868687868118 (0.03738539127469739) | > loss_mel_ce: 3.411994695663452 (nan) | > loss: 0.013681403361260891 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13652647935592852) | > loader_time: 0.0046 (0.004653879053493947)  --> TIME: 2025-02-24 22:15:33 -- STEP: 27056/36938 -- GLOBAL_STEP: 507250 | > loss_text_ce: 0.03232315555214882 (0.03738489966178049) | > loss_mel_ce: 2.417105197906494 (nan) | > loss: 0.009719953872263432 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.1365293523639282) | > loader_time: 0.0046 (0.004653765200370133)  --> TIME: 2025-02-24 22:15:42 -- STEP: 27106/36938 -- GLOBAL_STEP: 507300 | > loss_text_ce: 0.0429970845580101 (0.03738365059254328) | > loss_mel_ce: 1.4763624668121338 (nan) | > loss: 0.006029204931110144 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13652518218112306) | > loader_time: 0.0045 (0.004653733620861367)  --> TIME: 2025-02-24 22:15:52 -- STEP: 27156/36938 -- GLOBAL_STEP: 507350 | > loss_text_ce: 0.035692375153303146 (0.037383166099487185) | > loss_mel_ce: 2.890748977661133 (nan) | > loss: 0.011612863279879093 (nan) | > current_lr: 5e-06 | > step_time: 0.1707 (0.13653150696451732) | > loader_time: 0.0049 (0.0046536672675463525)  --> TIME: 2025-02-24 22:16:01 -- STEP: 27206/36938 -- GLOBAL_STEP: 507400 | > loss_text_ce: 0.034269824624061584 (0.03738382673295599) | > loss_mel_ce: 2.9955222606658936 (nan) | > loss: 0.012022985145449638 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13653000148790578) | > loader_time: 0.0045 (0.004653718676066125)  --> TIME: 2025-02-24 22:16:10 -- STEP: 27256/36938 -- GLOBAL_STEP: 507450 | > loss_text_ce: 0.030221031978726387 (0.037383590756084314) | > loss_mel_ce: 2.523491859436035 (nan) | > loss: 0.010133781470358372 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1365273771990699) | > loader_time: 0.0047 (0.004653780392827864)  --> TIME: 2025-02-24 22:16:20 -- STEP: 27306/36938 -- GLOBAL_STEP: 507500 | > loss_text_ce: 0.033961016684770584 (0.037380155682547475) | > loss_mel_ce: 2.51772403717041 (nan) | > loss: 0.010125734843313694 (nan) | > current_lr: 5e-06 | > step_time: 0.1263 (0.13653083229330368) | > loader_time: 0.0046 (0.004653786998227098)  --> TIME: 2025-02-24 22:16:29 -- STEP: 27356/36938 -- GLOBAL_STEP: 507550 | > loss_text_ce: 0.04077564552426338 (0.03738005372369021) | > loss_mel_ce: 3.1050353050231934 (nan) | > loss: 0.012483377009630203 (nan) | > current_lr: 5e-06 | > step_time: 0.1612 (0.13653364202784118) | > loader_time: 0.0048 (0.004654112807280942)  --> TIME: 2025-02-24 22:16:38 -- STEP: 27406/36938 -- GLOBAL_STEP: 507600 | > loss_text_ce: 0.03454319015145302 (0.0373825276519459) | > loss_mel_ce: 0.024342792108654976 (nan) | > loss: 0.0002336745528737083 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.136537024411617) | > loader_time: 0.0046 (0.0046541598350698845)  --> TIME: 2025-02-24 22:16:48 -- STEP: 27456/36938 -- GLOBAL_STEP: 507650 | > loss_text_ce: 0.028612516820430756 (0.03738129126650284) | > loss_mel_ce: 0.14711913466453552 (nan) | > loss: 0.0006973479175940156 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13654415834790615) | > loader_time: 0.0046 (0.004654203556773235)  --> TIME: 2025-02-24 22:16:57 -- STEP: 27506/36938 -- GLOBAL_STEP: 507700 | > loss_text_ce: 0.03385498747229576 (0.037379258974530434) | > loss_mel_ce: 3.303757429122925 (nan) | > loss: 0.013244494795799255 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.13653014593450985) | > loader_time: 0.0046 (0.00465422170531276)  --> TIME: 2025-02-24 22:17:06 -- STEP: 27556/36938 -- GLOBAL_STEP: 507750 | > loss_text_ce: 0.03487575426697731 (0.03737969063431226) | > loss_mel_ce: 3.503037691116333 (nan) | > loss: 0.014039340429008007 (nan) | > current_lr: 5e-06 | > step_time: 0.1493 (0.1365345638953197) | > loader_time: 0.0048 (0.004654316082629194)  --> TIME: 2025-02-24 22:17:16 -- STEP: 27606/36938 -- GLOBAL_STEP: 507800 | > loss_text_ce: 0.04190296679735184 (0.037376963328108116) | > loss_mel_ce: 3.38268780708313 (nan) | > loss: 0.01358964666724205 (nan) | > current_lr: 5e-06 | > step_time: 0.1691 (0.13654321980961734) | > loader_time: 0.0046 (0.004654415472688777)  --> TIME: 2025-02-24 22:17:26 -- STEP: 27656/36938 -- GLOBAL_STEP: 507850 | > loss_text_ce: 0.0469508022069931 (0.03737755650377574) | > loss_mel_ce: 2.5136876106262207 (nan) | > loss: 0.0101612638682127 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.1365516353263858) | > loader_time: 0.0046 (0.004654501063443662)  --> TIME: 2025-02-24 22:17:34 -- STEP: 27706/36938 -- GLOBAL_STEP: 507900 | > loss_text_ce: 0.04265930503606796 (0.03737801501254698) | > loss_mel_ce: 2.4770987033843994 (nan) | > loss: 0.009999040514230728 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13653523694958591) | > loader_time: 0.0048 (0.004654371488019433)  --> TIME: 2025-02-24 22:17:44 -- STEP: 27756/36938 -- GLOBAL_STEP: 507950 | > loss_text_ce: 0.02934933640062809 (0.03737851526823163) | > loss_mel_ce: 2.234622001647949 (nan) | > loss: 0.008984013460576534 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1365314052400094) | > loader_time: 0.0046 (0.004654396497682705)  --> TIME: 2025-02-24 22:17:53 -- STEP: 27806/36938 -- GLOBAL_STEP: 508000 | > loss_text_ce: 0.04397096112370491 (0.03738039006250407) | > loss_mel_ce: 2.6171562671661377 (nan) | > loss: 0.010560030117630959 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1365334551805798) | > loader_time: 0.0046 (0.004654356149394799)  --> TIME: 2025-02-24 22:18:02 -- STEP: 27856/36938 -- GLOBAL_STEP: 508050 | > loss_text_ce: 0.029632585123181343 (0.03738171385388075) | > loss_mel_ce: 2.9352707862854004 (nan) | > loss: 0.011765490286052227 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.1365204805584215) | > loader_time: 0.0045 (0.004654279681610524)  --> TIME: 2025-02-24 22:18:11 -- STEP: 27906/36938 -- GLOBAL_STEP: 508100 | > loss_text_ce: 0.03157147020101547 (0.03738102851377289) | > loss_mel_ce: 2.373215913772583 (nan) | > loss: 0.009542807005345821 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13652159358953922) | > loader_time: 0.0046 (0.004654185674373837)  --> TIME: 2025-02-24 22:18:20 -- STEP: 27956/36938 -- GLOBAL_STEP: 508150 | > loss_text_ce: 0.03962097689509392 (0.03738060261808832) | > loss_mel_ce: 2.779346466064453 (nan) | > loss: 0.011186379007995129 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13652353489547345) | > loader_time: 0.0046 (0.00465407200442273)  --> TIME: 2025-02-24 22:18:30 -- STEP: 28006/36938 -- GLOBAL_STEP: 508200 | > loss_text_ce: 0.0201069638133049 (0.03737664808110971) | > loss_mel_ce: 1.7359732389450073 (nan) | > loss: 0.006968572270125151 (nan) | > current_lr: 5e-06 | > step_time: 0.1123 (0.13652926277775504) | > loader_time: 0.0045 (0.00465408385774707)  --> TIME: 2025-02-24 22:18:39 -- STEP: 28056/36938 -- GLOBAL_STEP: 508250 | > loss_text_ce: 0.03691926971077919 (0.037375888640785775) | > loss_mel_ce: 3.276028633117676 (nan) | > loss: 0.013146619312465191 (nan) | > current_lr: 5e-06 | > step_time: 0.1864 (0.13653258362794013) | > loader_time: 0.0049 (0.0046540868904368)  --> TIME: 2025-02-24 22:18:49 -- STEP: 28106/36938 -- GLOBAL_STEP: 508300 | > loss_text_ce: 0.03793218359351158 (0.037374346952135755) | > loss_mel_ce: 3.2449727058410645 (nan) | > loss: 0.013027400709688663 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.13653709158425417) | > loader_time: 0.0047 (0.004654064777694277)  --> TIME: 2025-02-24 22:18:58 -- STEP: 28156/36938 -- GLOBAL_STEP: 508350 | > loss_text_ce: 0.04403029382228851 (0.03737451105519031) | > loss_mel_ce: 3.341149091720581 (nan) | > loss: 0.013433251529932022 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.1365357731736373) | > loader_time: 0.0047 (0.004654030549897072)  --> TIME: 2025-02-24 22:19:07 -- STEP: 28206/36938 -- GLOBAL_STEP: 508400 | > loss_text_ce: 0.04267716407775879 (0.037373537969056056) | > loss_mel_ce: 2.6835577487945557 (nan) | > loss: 0.010818392969667912 (nan) | > current_lr: 5e-06 | > step_time: 0.1629 (0.13653603541228082) | > loader_time: 0.0048 (0.004654066533744115)  --> TIME: 2025-02-24 22:19:17 -- STEP: 28256/36938 -- GLOBAL_STEP: 508450 | > loss_text_ce: 0.03282622620463371 (0.03737164873185852) | > loss_mel_ce: 3.2225935459136963 (nan) | > loss: 0.01291833259165287 (nan) | > current_lr: 5e-06 | > step_time: 0.157 (0.13654705583374221) | > loader_time: 0.0046 (0.004654143912673371)  --> TIME: 2025-02-24 22:19:27 -- STEP: 28306/36938 -- GLOBAL_STEP: 508500 | > loss_text_ce: 0.042180728167295456 (0.037372029124198854) | > loss_mel_ce: 2.6810336112976074 (nan) | > loss: 0.01080640684813261 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13654985945359385) | > loader_time: 0.0046 (0.004654184125939186)  --> TIME: 2025-02-24 22:19:36 -- STEP: 28356/36938 -- GLOBAL_STEP: 508550 | > loss_text_ce: 0.041603732854127884 (0.037372230295198317) | > loss_mel_ce: 2.558192729949951 (nan) | > loss: 0.010316653177142143 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13654651934782092) | > loader_time: 0.0045 (0.004654087583442558)  --> TIME: 2025-02-24 22:19:45 -- STEP: 28406/36938 -- GLOBAL_STEP: 508600 | > loss_text_ce: 0.044041845947504044 (0.0373713367321311) | > loss_mel_ce: 3.7278079986572266 (nan) | > loss: 0.01496765948832035 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.1365487391215302) | > loader_time: 0.0048 (0.004654065342106555)  --> TIME: 2025-02-24 22:19:54 -- STEP: 28456/36938 -- GLOBAL_STEP: 508650 | > loss_text_ce: 0.03666519373655319 (0.03737447535893879) | > loss_mel_ce: 2.1820976734161377 (nan) | > loss: 0.008804614655673504 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13654313444052354) | > loader_time: 0.0046 (0.004654055126671148)  --> TIME: 2025-02-24 22:20:04 -- STEP: 28506/36938 -- GLOBAL_STEP: 508700 | > loss_text_ce: 0.03873484581708908 (0.03737491249563322) | > loss_mel_ce: 3.173130750656128 (nan) | > loss: 0.012745499610900879 (nan) | > current_lr: 5e-06 | > step_time: 0.1512 (0.13654475855106418) | > loader_time: 0.0047 (0.004654111656767576)  --> TIME: 2025-02-24 22:20:13 -- STEP: 28556/36938 -- GLOBAL_STEP: 508750 | > loss_text_ce: 0.0420701764523983 (0.037371607089823296) | > loss_mel_ce: 0.12880107760429382 (nan) | > loss: 0.0006780605763196945 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13653901557926415) | > loader_time: 0.0045 (0.004654108359209515)  --> TIME: 2025-02-24 22:20:22 -- STEP: 28606/36938 -- GLOBAL_STEP: 508800 | > loss_text_ce: 0.054520804435014725 (0.03737130523777422) | > loss_mel_ce: 0.02015852928161621 (nan) | > loss: 0.0002963465522043407 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13653925852384563) | > loader_time: 0.0048 (0.00465403935513043)  --> TIME: 2025-02-24 22:20:31 -- STEP: 28656/36938 -- GLOBAL_STEP: 508850 | > loss_text_ce: 0.033926621079444885 (0.03736910291141764) | > loss_mel_ce: 2.9569239616394043 (nan) | > loss: 0.011868455447256565 (nan) | > current_lr: 5e-06 | > step_time: 0.1811 (0.13653605237877806) | > loader_time: 0.0048 (0.004653933209989508)  --> TIME: 2025-02-24 22:20:41 -- STEP: 28706/36938 -- GLOBAL_STEP: 508900 | > loss_text_ce: 0.03572027385234833 (0.03736654756215499) | > loss_mel_ce: 3.250523567199707 (nan) | > loss: 0.013040651567280293 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13654283786878174) | > loader_time: 0.0046 (0.004653875133280756)  --> TIME: 2025-02-24 22:20:51 -- STEP: 28756/36938 -- GLOBAL_STEP: 508950 | > loss_text_ce: 0.03213292732834816 (0.03736725260939731) | > loss_mel_ce: 2.7058825492858887 (nan) | > loss: 0.010865140706300735 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13655096639892442) | > loader_time: 0.0046 (0.004653942470574386)  --> TIME: 2025-02-24 22:21:00 -- STEP: 28806/36938 -- GLOBAL_STEP: 509000 | > loss_text_ce: 0.03559761866927147 (0.03737008021146659) | > loss_mel_ce: 3.0889811515808105 (nan) | > loss: 0.012399123050272465 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.13655074331517617) | > loader_time: 0.0047 (0.004653962231390829)  --> TIME: 2025-02-24 22:21:09 -- STEP: 28856/36938 -- GLOBAL_STEP: 509050 | > loss_text_ce: 0.045229051262140274 (0.03737271068933172) | > loss_mel_ce: 3.0803866386413574 (nan) | > loss: 0.012403237633407116 (nan) | > current_lr: 5e-06 | > step_time: 0.1489 (0.13654895837795505) | > loader_time: 0.0046 (0.004653926086716587)  --> TIME: 2025-02-24 22:21:18 -- STEP: 28906/36938 -- GLOBAL_STEP: 509100 | > loss_text_ce: 0.03727761283516884 (0.03737281765516817) | > loss_mel_ce: 2.6865453720092773 (nan) | > loss: 0.010808822698891163 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13654005178603223) | > loader_time: 0.0045 (0.00465391561959114)  --> TIME: 2025-02-24 22:21:27 -- STEP: 28956/36938 -- GLOBAL_STEP: 509150 | > loss_text_ce: 0.03510783240199089 (0.03737152741272698) | > loss_mel_ce: 3.025853395462036 (nan) | > loss: 0.01214667223393917 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13653939258395284) | > loader_time: 0.0046 (0.004653908144556536)  --> TIME: 2025-02-24 22:21:37 -- STEP: 29006/36938 -- GLOBAL_STEP: 509200 | > loss_text_ce: 0.04766087606549263 (0.03737273296672734) | > loss_mel_ce: 0.11843039840459824 (nan) | > loss: 0.000659092387650162 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.1365464293347689) | > loader_time: 0.0047 (0.004654010509545384)  --> TIME: 2025-02-24 22:21:46 -- STEP: 29056/36938 -- GLOBAL_STEP: 509250 | > loss_text_ce: 0.03260599821805954 (0.037372931360001485) | > loss_mel_ce: 3.3409385681152344 (nan) | > loss: 0.013387082144618034 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13655025715476155) | > loader_time: 0.0048 (0.0046541294009149565)  --> TIME: 2025-02-24 22:21:56 -- STEP: 29106/36938 -- GLOBAL_STEP: 509300 | > loss_text_ce: 0.027896281331777573 (0.03737164185931649) | > loss_mel_ce: 2.8426289558410645 (nan) | > loss: 0.011390973813831806 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13654706073955805) | > loader_time: 0.0045 (0.004654063036914745)  --> TIME: 2025-02-24 22:22:05 -- STEP: 29156/36938 -- GLOBAL_STEP: 509350 | > loss_text_ce: 0.031511515378952026 (0.03737386366319682) | > loss_mel_ce: 3.0325875282287598 (nan) | > loss: 0.01215912401676178 (nan) | > current_lr: 5e-06 | > step_time: 0.144 (0.1365441035007582) | > loader_time: 0.0047 (0.004653997889989854)  --> TIME: 2025-02-24 22:22:14 -- STEP: 29206/36938 -- GLOBAL_STEP: 509400 | > loss_text_ce: 0.03922904282808304 (0.03737637579289491) | > loss_mel_ce: 2.106170654296875 (nan) | > loss: 0.008513491600751877 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.13653732466957333) | > loader_time: 0.0048 (0.004653907145472921)  --> TIME: 2025-02-24 22:22:23 -- STEP: 29256/36938 -- GLOBAL_STEP: 509450 | > loss_text_ce: 0.032759446650743484 (0.037374910067395635) | > loss_mel_ce: 2.6025233268737793 (nan) | > loss: 0.010457471944391727 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.13652603424798326) | > loader_time: 0.0046 (0.004653802417098452)  --> TIME: 2025-02-24 22:22:32 -- STEP: 29306/36938 -- GLOBAL_STEP: 509500 | > loss_text_ce: 0.03769628331065178 (0.03737671050424619) | > loss_mel_ce: 3.3090779781341553 (nan) | > loss: 0.013280851766467094 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.136524552600275) | > loader_time: 0.0046 (0.004653704245326547)  --> TIME: 2025-02-24 22:22:41 -- STEP: 29356/36938 -- GLOBAL_STEP: 509550 | > loss_text_ce: 0.0362735278904438 (0.03737703697939384) | > loss_mel_ce: 2.746959686279297 (nan) | > loss: 0.01104457676410675 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.1365240187910702) | > loader_time: 0.0046 (0.004653637481330253)  --> TIME: 2025-02-24 22:22:51 -- STEP: 29406/36938 -- GLOBAL_STEP: 509600 | > loss_text_ce: 0.052969206124544144 (0.03737429202018161) | > loss_mel_ce: 2.288517951965332 (nan) | > loss: 0.009291616268455982 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.1365247780406089) | > loader_time: 0.0046 (0.004653579295431493)  --> TIME: 2025-02-24 22:23:00 -- STEP: 29456/36938 -- GLOBAL_STEP: 509650 | > loss_text_ce: 0.03411107137799263 (0.037374716934075715) | > loss_mel_ce: 2.7021048069000244 (nan) | > loss: 0.010858000256121159 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13652600595673975) | > loader_time: 0.0046 (0.004653576144312177)  --> TIME: 2025-02-24 22:23:10 -- STEP: 29506/36938 -- GLOBAL_STEP: 509700 | > loss_text_ce: 0.0465235561132431 (0.03737481168850564) | > loss_mel_ce: 3.0367631912231445 (nan) | > loss: 0.012235265225172043 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.136532772575598) | > loader_time: 0.005 (0.004653701529796641)  --> TIME: 2025-02-24 22:23:19 -- STEP: 29556/36938 -- GLOBAL_STEP: 509750 | > loss_text_ce: 0.03732934221625328 (0.03737537062089475) | > loss_mel_ce: 2.664426803588867 (nan) | > loss: 0.010721255093812943 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.1365331619239854) | > loader_time: 0.0047 (0.004653742678321255)  --> TIME: 2025-02-24 22:23:28 -- STEP: 29606/36938 -- GLOBAL_STEP: 509800 | > loss_text_ce: 0.03855222091078758 (0.03737542794334441) | > loss_mel_ce: 2.0147318840026855 (nan) | > loss: 0.00814795307815075 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1365319097041281) | > loader_time: 0.0045 (0.00465365808444802)  --> TIME: 2025-02-24 22:23:38 -- STEP: 29656/36938 -- GLOBAL_STEP: 509850 | > loss_text_ce: 0.03980470821261406 (0.03737558447460627) | > loss_mel_ce: 2.9221439361572266 (nan) | > loss: 0.011753764934837818 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13653545073977982) | > loader_time: 0.0047 (0.004653604607200028)  --> TIME: 2025-02-24 22:23:47 -- STEP: 29706/36938 -- GLOBAL_STEP: 509900 | > loss_text_ce: 0.02646508626639843 (0.037375409896288465) | > loss_mel_ce: 3.3951146602630615 (nan) | > loss: 0.013577698729932308 (nan) | > current_lr: 5e-06 | > step_time: 0.1949 (0.13653281399884787) | > loader_time: 0.0049 (0.004653515233372615)  --> TIME: 2025-02-24 22:23:56 -- STEP: 29756/36938 -- GLOBAL_STEP: 509950 | > loss_text_ce: 0.031920962035655975 (0.037372618598222274) | > loss_mel_ce: 1.9290087223052979 (nan) | > loss: 0.007781467400491238 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13652762504238708) | > loader_time: 0.0046 (0.004653371154404632)  --> TIME: 2025-02-24 22:24:05 -- STEP: 29806/36938 -- GLOBAL_STEP: 510000 | > loss_text_ce: 0.030425842851400375 (0.03737056959459891) | > loss_mel_ce: 3.2065796852111816 (nan) | > loss: 0.012845260091125965 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13652002393083523) | > loader_time: 0.0045 (0.0046533298742000555) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_510000.pth  --> TIME: 2025-02-24 22:24:20 -- STEP: 29856/36938 -- GLOBAL_STEP: 510050 | > loss_text_ce: 0.04558999836444855 (0.03736894122854633) | > loss_mel_ce: 2.5866281986236572 (nan) | > loss: 0.010445310734212399 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13651650588228775) | > loader_time: 0.0045 (0.0046534841802342245)  --> TIME: 2025-02-24 22:24:29 -- STEP: 29906/36938 -- GLOBAL_STEP: 510100 | > loss_text_ce: 0.03654644638299942 (0.03736963259433726) | > loss_mel_ce: 3.0337471961975098 (nan) | > loss: 0.01218370534479618 (nan) | > current_lr: 5e-06 | > step_time: 0.1757 (0.13650678371750127) | > loader_time: 0.0048 (0.004653424090353073)  --> TIME: 2025-02-24 22:24:39 -- STEP: 29956/36938 -- GLOBAL_STEP: 510150 | > loss_text_ce: 0.04707561805844307 (0.037369120656577506) | > loss_mel_ce: 2.431795358657837 (nan) | > loss: 0.009836790151894093 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13651597506214602) | > loader_time: 0.0049 (0.004653512476473951)  --> TIME: 2025-02-24 22:24:48 -- STEP: 30006/36938 -- GLOBAL_STEP: 510200 | > loss_text_ce: 0.0363122895359993 (0.037367689236042126) | > loss_mel_ce: 3.042044162750244 (nan) | > loss: 0.012215700931847095 (nan) | > current_lr: 5e-06 | > step_time: 0.1619 (0.13651460562913692) | > loader_time: 0.0048 (0.0046534860069764325)  --> TIME: 2025-02-24 22:24:58 -- STEP: 30056/36938 -- GLOBAL_STEP: 510250 | > loss_text_ce: 0.0383792482316494 (0.03736856067373396) | > loss_mel_ce: 0.1549072265625 (nan) | > loss: 0.0007670098566450179 (nan) | > current_lr: 5e-06 | > step_time: 0.1271 (0.13651785787541051) | > loader_time: 0.0046 (0.00465364285787113)  --> TIME: 2025-02-24 22:25:07 -- STEP: 30106/36938 -- GLOBAL_STEP: 510300 | > loss_text_ce: 0.04482005536556244 (0.03736806762307125) | > loss_mel_ce: 2.9472954273223877 (nan) | > loss: 0.011873475275933743 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13652427180030563) | > loader_time: 0.0047 (0.004653757991548729)  --> TIME: 2025-02-24 22:25:16 -- STEP: 30156/36938 -- GLOBAL_STEP: 510350 | > loss_text_ce: 0.03723820298910141 (0.03736683339068759) | > loss_mel_ce: 3.0799992084503174 (nan) | > loss: 0.012369990348815918 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.13652475895529384) | > loader_time: 0.0047 (0.0046538697074619655)  --> TIME: 2025-02-24 22:25:26 -- STEP: 30206/36938 -- GLOBAL_STEP: 510400 | > loss_text_ce: 0.03230130299925804 (0.03736811531954874) | > loss_mel_ce: 2.813542366027832 (nan) | > loss: 0.011293030343949795 (nan) | > current_lr: 5e-06 | > step_time: 0.1983 (0.13652842767679238) | > loader_time: 0.0048 (0.00465394620555033)  --> TIME: 2025-02-24 22:25:35 -- STEP: 30256/36938 -- GLOBAL_STEP: 510450 | > loss_text_ce: 0.04822670668363571 (0.03737237134455578) | > loss_mel_ce: 2.48201847076416 (nan) | > loss: 0.010040655732154846 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13653009468005012) | > loader_time: 0.0045 (0.0046539002076965204)  --> TIME: 2025-02-24 22:25:45 -- STEP: 30306/36938 -- GLOBAL_STEP: 510500 | > loss_text_ce: 0.02650308981537819 (0.037373818269730755) | > loss_mel_ce: 2.9783496856689453 (nan) | > loss: 0.011924020014703274 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13653257583412268) | > loader_time: 0.0045 (0.004653897480878725)  --> TIME: 2025-02-24 22:25:54 -- STEP: 30356/36938 -- GLOBAL_STEP: 510550 | > loss_text_ce: 0.034842804074287415 (0.03737230964742741) | > loss_mel_ce: 2.204439401626587 (nan) | > loss: 0.00888604111969471 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13653548428437096) | > loader_time: 0.0046 (0.004653871617057673)  --> TIME: 2025-02-24 22:26:03 -- STEP: 30406/36938 -- GLOBAL_STEP: 510600 | > loss_text_ce: 0.03436955437064171 (0.03737222747153724) | > loss_mel_ce: 2.5678412914276123 (nan) | > loss: 0.010326233692467213 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13652529454910337) | > loader_time: 0.0047 (0.004653810035521365)  --> TIME: 2025-02-24 22:26:12 -- STEP: 30456/36938 -- GLOBAL_STEP: 510650 | > loss_text_ce: 0.032632943242788315 (0.037368919716070856) | > loss_mel_ce: 3.074068069458008 (nan) | > loss: 0.012328178621828556 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.1365310284043664) | > loader_time: 0.0047 (0.004653863145010569)  --> TIME: 2025-02-24 22:26:22 -- STEP: 30506/36938 -- GLOBAL_STEP: 510700 | > loss_text_ce: 0.035743407905101776 (0.03736804598190848) | > loss_mel_ce: 3.4301388263702393 (nan) | > loss: 0.013753501698374748 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13652417282741514) | > loader_time: 0.0047 (0.004653950530974819)  --> TIME: 2025-02-24 22:26:31 -- STEP: 30556/36938 -- GLOBAL_STEP: 510750 | > loss_text_ce: 0.03192322328686714 (0.03736708911192073) | > loss_mel_ce: 2.2561886310577393 (nan) | > loss: 0.009079809300601482 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13651956622708886) | > loader_time: 0.0046 (0.004654061920684335)  --> TIME: 2025-02-24 22:26:40 -- STEP: 30606/36938 -- GLOBAL_STEP: 510800 | > loss_text_ce: 0.024417854845523834 (0.0373643361346523) | > loss_mel_ce: 2.549591064453125 (nan) | > loss: 0.010214322246611118 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13652717972911074) | > loader_time: 0.0044 (0.004654059431595392)  --> TIME: 2025-02-24 22:26:50 -- STEP: 30656/36938 -- GLOBAL_STEP: 510850 | > loss_text_ce: 0.04935915395617485 (0.03736490933192919) | > loss_mel_ce: 2.9307281970977783 (nan) | > loss: 0.011825744062662125 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13653241846873795) | > loader_time: 0.0047 (0.004654118017388756)  --> TIME: 2025-02-24 22:26:59 -- STEP: 30706/36938 -- GLOBAL_STEP: 510900 | > loss_text_ce: 0.03981520235538483 (0.037362416403385015) | > loss_mel_ce: 3.637938976287842 (nan) | > loss: 0.014594263397157192 (nan) | > current_lr: 5e-06 | > step_time: 0.1825 (0.1365322420389455) | > loader_time: 0.0048 (0.004654108814126919)  --> TIME: 2025-02-24 22:27:09 -- STEP: 30756/36938 -- GLOBAL_STEP: 510950 | > loss_text_ce: 0.038782600313425064 (0.03736233892438105) | > loss_mel_ce: 1.5986318588256836 (nan) | > loss: 0.006497676949948072 (nan) | > current_lr: 5e-06 | > step_time: 0.1444 (0.13654304270100012) | > loader_time: 0.0047 (0.004654131361715384)  --> TIME: 2025-02-24 22:27:18 -- STEP: 30806/36938 -- GLOBAL_STEP: 511000 | > loss_text_ce: 0.05282454192638397 (0.037361349409978595) | > loss_mel_ce: 2.8856332302093506 (nan) | > loss: 0.011660546995699406 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1365436251996395) | > loader_time: 0.0047 (0.004654170816257159)  --> TIME: 2025-02-24 22:27:27 -- STEP: 30856/36938 -- GLOBAL_STEP: 511050 | > loss_text_ce: 0.03637988120317459 (0.03736238147840506) | > loss_mel_ce: 3.3703935146331787 (nan) | > loss: 0.013518942520022392 (nan) | > current_lr: 5e-06 | > step_time: 0.1521 (0.13654248626979515) | > loader_time: 0.0048 (0.004654167490917848)  --> TIME: 2025-02-24 22:27:37 -- STEP: 30906/36938 -- GLOBAL_STEP: 511100 | > loss_text_ce: 0.0335029698908329 (0.037362246238232845) | > loss_mel_ce: 2.353909969329834 (nan) | > loss: 0.009473862126469612 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13654747098300554) | > loader_time: 0.0046 (0.004654186933563467)  --> TIME: 2025-02-24 22:27:46 -- STEP: 30956/36938 -- GLOBAL_STEP: 511150 | > loss_text_ce: 0.03761006146669388 (0.03736021774367853) | > loss_mel_ce: 3.314732074737549 (nan) | > loss: 0.013302945531904697 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.1365434390997332) | > loader_time: 0.0046 (0.004654193189443247)  --> TIME: 2025-02-24 22:27:56 -- STEP: 31006/36938 -- GLOBAL_STEP: 511200 | > loss_text_ce: 0.03401509299874306 (0.03735837553712629) | > loss_mel_ce: 3.139000177383423 (nan) | > loss: 0.012591331265866756 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.1365491857622808) | > loader_time: 0.0046 (0.004654195095995541)  --> TIME: 2025-02-24 22:28:05 -- STEP: 31056/36938 -- GLOBAL_STEP: 511250 | > loss_text_ce: 0.035695016384124756 (0.03735854361198571) | > loss_mel_ce: 3.2030270099639893 (nan) | > loss: 0.012852072715759277 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13654895797026426) | > loader_time: 0.0046 (0.004654273582696273)  --> TIME: 2025-02-24 22:28:15 -- STEP: 31106/36938 -- GLOBAL_STEP: 511300 | > loss_text_ce: 0.03271113708615303 (0.037356448150983455) | > loss_mel_ce: 3.350517749786377 (nan) | > loss: 0.013425512239336967 (nan) | > current_lr: 5e-06 | > step_time: 0.1886 (0.13655406564315092) | > loader_time: 0.0048 (0.0046543232276954)  --> TIME: 2025-02-24 22:28:24 -- STEP: 31156/36938 -- GLOBAL_STEP: 511350 | > loss_text_ce: 0.030234012752771378 (0.03735602686111313) | > loss_mel_ce: 3.9309895038604736 (nan) | > loss: 0.01571914181113243 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13655850848730453) | > loader_time: 0.0045 (0.004654360729672484)  --> TIME: 2025-02-24 22:28:33 -- STEP: 31206/36938 -- GLOBAL_STEP: 511400 | > loss_text_ce: 0.057510193437337875 (0.037353194522667554) | > loss_mel_ce: 2.814359426498413 (nan) | > loss: 0.011396308429539204 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13655473750760583) | > loader_time: 0.0047 (0.004654382517924077)  --> TIME: 2025-02-24 22:28:43 -- STEP: 31256/36938 -- GLOBAL_STEP: 511450 | > loss_text_ce: 0.036114271730184555 (0.03735121609808727) | > loss_mel_ce: 2.640218734741211 (nan) | > loss: 0.01062036957591772 (nan) | > current_lr: 5e-06 | > step_time: 0.1691 (0.13656681653999903) | > loader_time: 0.0048 (0.004654361588710194)  --> TIME: 2025-02-24 22:28:52 -- STEP: 31306/36938 -- GLOBAL_STEP: 511500 | > loss_text_ce: 0.03274201974272728 (0.03734941776693804) | > loss_mel_ce: 3.163764476776123 (nan) | > loss: 0.012684550136327744 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13656200640180122) | > loader_time: 0.0049 (0.0046542963417758225)  --> TIME: 2025-02-24 22:29:01 -- STEP: 31356/36938 -- GLOBAL_STEP: 511550 | > loss_text_ce: 0.04075256362557411 (0.03734910883028874) | > loss_mel_ce: 3.454267978668213 (nan) | > loss: 0.013869130052626133 (nan) | > current_lr: 5e-06 | > step_time: 0.1503 (0.1365591522336146) | > loader_time: 0.0046 (0.004654214582602582)  --> TIME: 2025-02-24 22:29:11 -- STEP: 31406/36938 -- GLOBAL_STEP: 511600 | > loss_text_ce: 0.028753574937582016 (0.03735010368426935) | > loss_mel_ce: 3.427400827407837 (nan) | > loss: 0.013714899308979511 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.1365606196251737) | > loader_time: 0.0045 (0.004654262663034639)  --> TIME: 2025-02-24 22:29:20 -- STEP: 31456/36938 -- GLOBAL_STEP: 511650 | > loss_text_ce: 0.04166252911090851 (0.03735032479768046) | > loss_mel_ce: 3.481377124786377 (nan) | > loss: 0.013980316929519176 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.13655726128046564) | > loader_time: 0.0046 (0.004654237252043293)  --> TIME: 2025-02-24 22:29:29 -- STEP: 31506/36938 -- GLOBAL_STEP: 511700 | > loss_text_ce: 0.03615613654255867 (0.03735221978017977) | > loss_mel_ce: 2.754152774810791 (nan) | > loss: 0.011072655208408833 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.1365596231548235) | > loader_time: 0.0047 (0.00465425103004185)  --> TIME: 2025-02-24 22:29:38 -- STEP: 31556/36938 -- GLOBAL_STEP: 511750 | > loss_text_ce: 0.03476070612668991 (0.03735030618995443) | > loss_mel_ce: 3.4424591064453125 (nan) | > loss: 0.013798492029309273 (nan) | > current_lr: 5e-06 | > step_time: 0.1819 (0.13655359101455675) | > loader_time: 0.0049 (0.004654323605926522)  --> TIME: 2025-02-24 22:29:48 -- STEP: 31606/36938 -- GLOBAL_STEP: 511800 | > loss_text_ce: 0.04216046258807182 (0.03734735056705079) | > loss_mel_ce: 2.8571999073028564 (nan) | > loss: 0.011505398899316788 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.1365534399971364) | > loader_time: 0.0047 (0.004654349371322222)  --> TIME: 2025-02-24 22:29:57 -- STEP: 31656/36938 -- GLOBAL_STEP: 511850 | > loss_text_ce: 0.036278288811445236 (0.037348474387377005) | > loss_mel_ce: 3.0922372341156006 (nan) | > loss: 0.012414745055139065 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.1365553747908711) | > loader_time: 0.0047 (0.004654528382513151)  --> TIME: 2025-02-24 22:30:06 -- STEP: 31706/36938 -- GLOBAL_STEP: 511900 | > loss_text_ce: 0.03675315901637077 (0.037347329418927454) | > loss_mel_ce: 2.0951004028320312 (nan) | > loss: 0.008459736593067646 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.136547193304339) | > loader_time: 0.0046 (0.004654553608361026)  --> TIME: 2025-02-24 22:30:16 -- STEP: 31756/36938 -- GLOBAL_STEP: 511950 | > loss_text_ce: 0.02651720494031906 (0.03734916261671646) | > loss_mel_ce: 1.576209306716919 (nan) | > loss: 0.006360026076436043 (nan) | > current_lr: 5e-06 | > step_time: 0.1082 (0.13655350455740503) | > loader_time: 0.0047 (0.00465465082241132)  --> TIME: 2025-02-24 22:30:25 -- STEP: 31806/36938 -- GLOBAL_STEP: 512000 | > loss_text_ce: 0.030545847490429878 (0.03734719890155642) | > loss_mel_ce: 2.693166971206665 (nan) | > loss: 0.010808384977281094 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13654756358619904) | > loader_time: 0.0047 (0.004654663895268327)  --> TIME: 2025-02-24 22:30:34 -- STEP: 31856/36938 -- GLOBAL_STEP: 512050 | > loss_text_ce: 0.0424678660929203 (0.03734641772511712) | > loss_mel_ce: 3.0138092041015625 (nan) | > loss: 0.01212808396667242 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1365495769897341) | > loader_time: 0.0047 (0.004654660618885958)  --> TIME: 2025-02-24 22:30:43 -- STEP: 31906/36938 -- GLOBAL_STEP: 512100 | > loss_text_ce: 0.02664111740887165 (0.03734510508034584) | > loss_mel_ce: 3.305894136428833 (nan) | > loss: 0.013224347494542599 (nan) | > current_lr: 5e-06 | > step_time: 0.2034 (0.13655027727165245) | > loader_time: 0.0049 (0.0046548556513317674)  --> TIME: 2025-02-24 22:30:53 -- STEP: 31956/36938 -- GLOBAL_STEP: 512150 | > loss_text_ce: 0.03713164106011391 (0.03734568755744111) | > loss_mel_ce: 2.3008248805999756 (nan) | > loss: 0.009277605451643467 (nan) | > current_lr: 5e-06 | > step_time: 0.1343 (0.1365470677625406) | > loader_time: 0.0046 (0.00465488243586487)  --> TIME: 2025-02-24 22:31:02 -- STEP: 32006/36938 -- GLOBAL_STEP: 512200 | > loss_text_ce: 0.030871596187353134 (0.0373425619967514) | > loss_mel_ce: 3.5848605632781982 (nan) | > loss: 0.014348144643008709 (nan) | > current_lr: 5e-06 | > step_time: 0.1989 (0.13654626277672605) | > loader_time: 0.0048 (0.004654902372853041)  --> TIME: 2025-02-24 22:31:11 -- STEP: 32056/36938 -- GLOBAL_STEP: 512250 | > loss_text_ce: 0.04321072995662689 (0.037340395637898545) | > loss_mel_ce: 3.303055763244629 (nan) | > loss: 0.013278836384415627 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.13654782951607053) | > loader_time: 0.0046 (0.004655000029700252)  --> TIME: 2025-02-24 22:31:21 -- STEP: 32106/36938 -- GLOBAL_STEP: 512300 | > loss_text_ce: 0.04033263772726059 (0.03734148041798411) | > loss_mel_ce: 3.0389416217803955 (nan) | > loss: 0.012219342403113842 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13655322020442864) | > loader_time: 0.0046 (0.004654979532042541)  --> TIME: 2025-02-24 22:31:30 -- STEP: 32156/36938 -- GLOBAL_STEP: 512350 | > loss_text_ce: 0.037667129188776016 (0.03734128692894941) | > loss_mel_ce: 3.58009934425354 (nan) | > loss: 0.014356216415762901 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.1365473403950241) | > loader_time: 0.0046 (0.004654968766552789)  --> TIME: 2025-02-24 22:31:39 -- STEP: 32206/36938 -- GLOBAL_STEP: 512400 | > loss_text_ce: 0.0392313115298748 (0.03734250726293453) | > loss_mel_ce: 1.320900559425354 (nan) | > loss: 0.005397349130362272 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.13654551448003582) | > loader_time: 0.0048 (0.004655027636784012)  --> TIME: 2025-02-24 22:31:48 -- STEP: 32256/36938 -- GLOBAL_STEP: 512450 | > loss_text_ce: 0.026866454631090164 (0.037343073352328525) | > loss_mel_ce: 1.9797276258468628 (nan) | > loss: 0.007962675765156746 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1365456354198241) | > loader_time: 0.0045 (0.004655070344193088)  --> TIME: 2025-02-24 22:31:58 -- STEP: 32306/36938 -- GLOBAL_STEP: 512500 | > loss_text_ce: 0.03277003392577171 (0.03734139268002411) | > loss_mel_ce: 3.479473352432251 (nan) | > loss: 0.013937474228441715 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13654429813822644) | > loader_time: 0.0047 (0.004655071982494284)  --> TIME: 2025-02-24 22:32:07 -- STEP: 32356/36938 -- GLOBAL_STEP: 512550 | > loss_text_ce: 0.036133259534835815 (0.03734081241182179) | > loss_mel_ce: 2.9736976623535156 (nan) | > loss: 0.011943774297833443 (nan) | > current_lr: 5e-06 | > step_time: 0.1659 (0.13654748480345646) | > loader_time: 0.0047 (0.004655131289802418)  --> TIME: 2025-02-24 22:32:16 -- STEP: 32406/36938 -- GLOBAL_STEP: 512600 | > loss_text_ce: 0.03767216205596924 (0.03734058261639894) | > loss_mel_ce: 2.971231460571289 (nan) | > loss: 0.011940093711018562 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.1365485535663847) | > loader_time: 0.0046 (0.0046551432247876135)  --> TIME: 2025-02-24 22:32:26 -- STEP: 32456/36938 -- GLOBAL_STEP: 512650 | > loss_text_ce: 0.03602816164493561 (0.03734200342622214) | > loss_mel_ce: 3.3981730937957764 (nan) | > loss: 0.01362778339534998 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.13655324855813403) | > loader_time: 0.0049 (0.004655124644854667)  --> TIME: 2025-02-24 22:32:35 -- STEP: 32506/36938 -- GLOBAL_STEP: 512700 | > loss_text_ce: 0.036229655146598816 (0.037340071074516784) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13655025237436755) | > loader_time: 0.0046 (0.00465512035120846)  --> TIME: 2025-02-24 22:32:44 -- STEP: 32556/36938 -- GLOBAL_STEP: 512750 | > loss_text_ce: 0.01770866848528385 (0.03733913317263301) | > loss_mel_ce: 2.8605780601501465 (nan) | > loss: 0.011421773582696915 (nan) | > current_lr: 5e-06 | > step_time: 0.107 (0.13654315506801218) | > loader_time: 0.0045 (0.004655137286459792)  --> TIME: 2025-02-24 22:32:53 -- STEP: 32606/36938 -- GLOBAL_STEP: 512800 | > loss_text_ce: 0.03470299020409584 (0.03733603060432273) | > loss_mel_ce: 2.789592981338501 (nan) | > loss: 0.011207524687051773 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13653370688124375) | > loader_time: 0.0046 (0.0046551435818403125)  --> TIME: 2025-02-24 22:33:03 -- STEP: 32656/36938 -- GLOBAL_STEP: 512850 | > loss_text_ce: 0.04644450917840004 (0.0373383924527998) | > loss_mel_ce: 3.0487701892852783 (nan) | > loss: 0.0122825987637043 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13653618052039673) | > loader_time: 0.0049 (0.004655241491688259)  --> TIME: 2025-02-24 22:33:12 -- STEP: 32706/36938 -- GLOBAL_STEP: 512900 | > loss_text_ce: 0.049454085528850555 (0.03734041108132812) | > loss_mel_ce: 3.096266746520996 (nan) | > loss: 0.012483019381761551 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13653384214044484) | > loader_time: 0.0045 (0.004655212180327084)  --> TIME: 2025-02-24 22:33:21 -- STEP: 32756/36938 -- GLOBAL_STEP: 512950 | > loss_text_ce: 0.032187700271606445 (0.03733959524692748) | > loss_mel_ce: 2.8362159729003906 (nan) | > loss: 0.011382554657757282 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.1365424081845111) | > loader_time: 0.0047 (0.004655331231281545)  --> TIME: 2025-02-24 22:33:31 -- STEP: 32806/36938 -- GLOBAL_STEP: 513000 | > loss_text_ce: 0.038134459406137466 (0.03733760093062295) | > loss_mel_ce: 2.361539602279663 (nan) | > loss: 0.009522517211735249 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13653702576160603) | > loader_time: 0.0046 (0.00465532031748515)  --> TIME: 2025-02-24 22:33:40 -- STEP: 32856/36938 -- GLOBAL_STEP: 513050 | > loss_text_ce: 0.03479023277759552 (0.037336645838648784) | > loss_mel_ce: 1.8669071197509766 (nan) | > loss: 0.007546418812125921 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.1365389540403293) | > loader_time: 0.0048 (0.004655437593422904)  --> TIME: 2025-02-24 22:33:49 -- STEP: 32906/36938 -- GLOBAL_STEP: 513100 | > loss_text_ce: 0.02952793799340725 (0.037336368977718346) | > loss_mel_ce: 2.8515625 (nan) | > loss: 0.011432899162173271 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13654145604389079) | > loader_time: 0.0045 (0.004655416400310982)  --> TIME: 2025-02-24 22:33:59 -- STEP: 32956/36938 -- GLOBAL_STEP: 513150 | > loss_text_ce: 0.03859136626124382 (0.037334898332209654) | > loss_mel_ce: 3.4458556175231934 (nan) | > loss: 0.013827171176671982 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13654898200918883) | > loader_time: 0.0047 (0.004655429779839354)  --> TIME: 2025-02-24 22:34:08 -- STEP: 33006/36938 -- GLOBAL_STEP: 513200 | > loss_text_ce: 0.03309398144483566 (0.037333737522517195) | > loss_mel_ce: 1.6863281726837158 (nan) | > loss: 0.006823103874921799 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.1365521790323524) | > loader_time: 0.0046 (0.004655409840203126)  --> TIME: 2025-02-24 22:34:18 -- STEP: 33056/36938 -- GLOBAL_STEP: 513250 | > loss_text_ce: 0.04145033285021782 (0.037333761515140035) | > loss_mel_ce: 2.8016436100006104 (nan) | > loss: 0.0112821189686656 (nan) | > current_lr: 5e-06 | > step_time: 0.172 (0.13655727098462453) | > loader_time: 0.0049 (0.004655552762925657)  --> TIME: 2025-02-24 22:34:27 -- STEP: 33106/36938 -- GLOBAL_STEP: 513300 | > loss_text_ce: 0.030217312276363373 (0.03733411260143386) | > loss_mel_ce: 2.704749345779419 (nan) | > loss: 0.010853043757379055 (nan) | > current_lr: 5e-06 | > step_time: 0.1771 (0.13656378652238152) | > loader_time: 0.0052 (0.0046557286120840645)  --> TIME: 2025-02-24 22:34:37 -- STEP: 33156/36938 -- GLOBAL_STEP: 513350 | > loss_text_ce: 0.03696044906973839 (0.03733509607037331) | > loss_mel_ce: 3.0560741424560547 (nan) | > loss: 0.012273946776986122 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13656612895972967) | > loader_time: 0.0048 (0.004655788849110254)  --> TIME: 2025-02-24 22:34:46 -- STEP: 33206/36938 -- GLOBAL_STEP: 513400 | > loss_text_ce: 0.02952778898179531 (0.037334073233065874) | > loss_mel_ce: 2.7950875759124756 (nan) | > loss: 0.011208792217075825 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13656951847230384) | > loader_time: 0.0047 (0.0046558486821528695)  --> TIME: 2025-02-24 22:34:56 -- STEP: 33256/36938 -- GLOBAL_STEP: 513450 | > loss_text_ce: 0.030592726543545723 (0.03733460494010814) | > loss_mel_ce: 3.1676127910614014 (nan) | > loss: 0.012691291980445385 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13657437871791234) | > loader_time: 0.0049 (0.004655961509160393)  --> TIME: 2025-02-24 22:35:05 -- STEP: 33306/36938 -- GLOBAL_STEP: 513500 | > loss_text_ce: 0.04546525329351425 (0.037333517875424646) | > loss_mel_ce: 2.5032272338867188 (nan) | > loss: 0.010113859549164772 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.1365799676231644) | > loader_time: 0.0048 (0.004656098951686621)  --> TIME: 2025-02-24 22:35:15 -- STEP: 33356/36938 -- GLOBAL_STEP: 513550 | > loss_text_ce: 0.034701455384492874 (0.037332185825569575) | > loss_mel_ce: 2.5933327674865723 (nan) | > loss: 0.010428707115352154 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13658149278997356) | > loader_time: 0.0045 (0.004656110311357881)  --> TIME: 2025-02-24 22:35:24 -- STEP: 33406/36938 -- GLOBAL_STEP: 513600 | > loss_text_ce: 0.027660313993692398 (0.037332585238638916) | > loss_mel_ce: 2.4478516578674316 (nan) | > loss: 0.009823461063206196 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.1365880144432494) | > loader_time: 0.0045 (0.00465620319150858)  --> TIME: 2025-02-24 22:35:34 -- STEP: 33456/36938 -- GLOBAL_STEP: 513650 | > loss_text_ce: 0.0408080630004406 (0.03733252063408026) | > loss_mel_ce: 2.723911762237549 (nan) | > loss: 0.010971110314130783 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13659376419799196) | > loader_time: 0.0045 (0.004656341388315607)  --> TIME: 2025-02-24 22:35:43 -- STEP: 33506/36938 -- GLOBAL_STEP: 513700 | > loss_text_ce: 0.0331031009554863 (0.03733040380174244) | > loss_mel_ce: 0.2743293046951294 (nan) | > loss: 0.001219969941303134 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13658945704050135) | > loader_time: 0.0046 (0.004656345902759812)  --> TIME: 2025-02-24 22:35:52 -- STEP: 33556/36938 -- GLOBAL_STEP: 513750 | > loss_text_ce: 0.03260013833642006 (0.03732914753630759) | > loss_mel_ce: 2.378345489501953 (nan) | > loss: 0.009567244909703732 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.1365781871279771) | > loader_time: 0.0047 (0.0046563266585219415)  --> TIME: 2025-02-24 22:36:01 -- STEP: 33606/36938 -- GLOBAL_STEP: 513800 | > loss_text_ce: 0.044280484318733215 (0.0373308838121795) | > loss_mel_ce: 3.660696506500244 (nan) | > loss: 0.014702290296554565 (nan) | > current_lr: 5e-06 | > step_time: 0.1793 (0.13656896943387115) | > loader_time: 0.0049 (0.00465632908146897)  --> TIME: 2025-02-24 22:36:11 -- STEP: 33656/36938 -- GLOBAL_STEP: 513850 | > loss_text_ce: 0.03247366100549698 (0.03732997674163099) | > loss_mel_ce: 2.4621286392211914 (nan) | > loss: 0.009899215772747993 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13657511673320497) | > loader_time: 0.0045 (0.00465651510700861)  --> TIME: 2025-02-24 22:36:20 -- STEP: 33706/36938 -- GLOBAL_STEP: 513900 | > loss_text_ce: 0.024757053703069687 (0.03732967364147996) | > loss_mel_ce: 3.0082030296325684 (nan) | > loss: 0.012035557068884373 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.136581770002768) | > loader_time: 0.0045 (0.004656519563319873)  --> TIME: 2025-02-24 22:36:29 -- STEP: 33756/36938 -- GLOBAL_STEP: 513950 | > loss_text_ce: 0.03342404216527939 (0.037327839328190435) | > loss_mel_ce: 3.2045819759368896 (nan) | > loss: 0.012849231250584126 (nan) | > current_lr: 5e-06 | > step_time: 0.1477 (0.1365815574615148) | > loader_time: 0.0047 (0.00465647655014233)  --> TIME: 2025-02-24 22:36:39 -- STEP: 33806/36938 -- GLOBAL_STEP: 514000 | > loss_text_ce: 0.030043181031942368 (0.03732536062987176) | > loss_mel_ce: 3.179886817932129 (nan) | > loss: 0.01273781806230545 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13658276898673505) | > loader_time: 0.0047 (0.004656547499444911)  --> TIME: 2025-02-24 22:36:48 -- STEP: 33856/36938 -- GLOBAL_STEP: 514050 | > loss_text_ce: 0.058525845408439636 (0.037325223564303664) | > loss_mel_ce: 4.013665199279785 (nan) | > loss: 0.01615948975086212 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13658422025339617) | > loader_time: 0.0047 (0.004656551655783299)  --> TIME: 2025-02-24 22:36:58 -- STEP: 33906/36938 -- GLOBAL_STEP: 514100 | > loss_text_ce: 0.04037544131278992 (0.037324999752223566) | > loss_mel_ce: 3.1505112648010254 (nan) | > loss: 0.012662249617278576 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13658773610776573) | > loader_time: 0.0046 (0.0046565630214724495)  --> TIME: 2025-02-24 22:37:07 -- STEP: 33956/36938 -- GLOBAL_STEP: 514150 | > loss_text_ce: 0.03648658096790314 (0.03732363047409091) | > loss_mel_ce: 1.5897895097732544 (nan) | > loss: 0.0064534773118793964 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13658929335971345) | > loader_time: 0.0046 (0.00465657436773258)  --> TIME: 2025-02-24 22:37:17 -- STEP: 34006/36938 -- GLOBAL_STEP: 514200 | > loss_text_ce: 0.03693298250436783 (0.03732236446324669) | > loss_mel_ce: 3.691486358642578 (nan) | > loss: 0.014795315451920033 (nan) | > current_lr: 5e-06 | > step_time: 0.157 (0.13659709308145082) | > loader_time: 0.0048 (0.004656549384297654)  --> TIME: 2025-02-24 22:37:26 -- STEP: 34056/36938 -- GLOBAL_STEP: 514250 | > loss_text_ce: 0.029808077961206436 (0.03732239761919619) | > loss_mel_ce: 1.7107369899749756 (nan) | > loss: 0.006906925234943628 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13660344141230304) | > loader_time: 0.0045 (0.004656470456200622)  --> TIME: 2025-02-24 22:37:36 -- STEP: 34106/36938 -- GLOBAL_STEP: 514300 | > loss_text_ce: 0.036706723272800446 (0.03732146752534738) | > loss_mel_ce: 3.0855629444122314 (nan) | > loss: 0.012389959767460823 (nan) | > current_lr: 5e-06 | > step_time: 0.1576 (0.13660716360379985) | > loader_time: 0.0046 (0.004656440839944714)  --> TIME: 2025-02-24 22:37:45 -- STEP: 34156/36938 -- GLOBAL_STEP: 514350 | > loss_text_ce: 0.03517785668373108 (0.03732182452764214) | > loss_mel_ce: 3.355193614959717 (nan) | > loss: 0.013453856110572815 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13660910180084412) | > loader_time: 0.0047 (0.004656438826676991)  --> TIME: 2025-02-24 22:37:54 -- STEP: 34206/36938 -- GLOBAL_STEP: 514400 | > loss_text_ce: 0.03557848930358887 (0.037323445060582314) | > loss_mel_ce: 3.1731810569763184 (nan) | > loss: 0.012733173556625843 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.1366073201777875) | > loader_time: 0.0046 (0.004656386028317204)  --> TIME: 2025-02-24 22:38:04 -- STEP: 34256/36938 -- GLOBAL_STEP: 514450 | > loss_text_ce: 0.04109938442707062 (0.03732409507418089) | > loss_mel_ce: 2.8027896881103516 (nan) | > loss: 0.011285274289548397 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13660825125695644) | > loader_time: 0.0046 (0.004656330537484193)  --> TIME: 2025-02-24 22:38:13 -- STEP: 34306/36938 -- GLOBAL_STEP: 514500 | > loss_text_ce: 0.03997593745589256 (0.03732338079439114) | > loss_mel_ce: 3.2641541957855225 (nan) | > loss: 0.013111627660691738 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13660719702141388) | > loader_time: 0.0046 (0.00465634200257863)  --> TIME: 2025-02-24 22:38:22 -- STEP: 34356/36938 -- GLOBAL_STEP: 514550 | > loss_text_ce: 0.038051262497901917 (0.03732660151884657) | > loss_mel_ce: 3.297252655029297 (nan) | > loss: 0.013235334306955338 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13660066163724513) | > loader_time: 0.0046 (0.004656325661826545)  --> TIME: 2025-02-24 22:38:31 -- STEP: 34406/36938 -- GLOBAL_STEP: 514600 | > loss_text_ce: 0.03607405722141266 (0.037323342930084) | > loss_mel_ce: 2.945274829864502 (nan) | > loss: 0.011830750852823257 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13660267658762543) | > loader_time: 0.0046 (0.0046562987039693625)  --> TIME: 2025-02-24 22:38:41 -- STEP: 34456/36938 -- GLOBAL_STEP: 514650 | > loss_text_ce: 0.03772071376442909 (0.03732623369746483) | > loss_mel_ce: 2.833559274673462 (nan) | > loss: 0.01139396894723177 (nan) | > current_lr: 5e-06 | > step_time: 0.1867 (0.1366076157351565) | > loader_time: 0.005 (0.004656289296109221)  --> TIME: 2025-02-24 22:38:50 -- STEP: 34506/36938 -- GLOBAL_STEP: 514700 | > loss_text_ce: 0.038030870258808136 (0.03732658283561356) | > loss_mel_ce: 3.2017698287963867 (nan) | > loss: 0.01285635307431221 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.1366053696253031) | > loader_time: 0.0048 (0.004656281373414158)  --> TIME: 2025-02-24 22:38:59 -- STEP: 34556/36938 -- GLOBAL_STEP: 514750 | > loss_text_ce: 0.039899878203868866 (0.03732513177351615) | > loss_mel_ce: 3.09375262260437 (nan) | > loss: 0.012435129843652248 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.1366013700742115) | > loader_time: 0.0051 (0.004656308405735166)  --> TIME: 2025-02-24 22:39:09 -- STEP: 34606/36938 -- GLOBAL_STEP: 514800 | > loss_text_ce: 0.04198756068944931 (0.03732400283176263) | > loss_mel_ce: 3.233646869659424 (nan) | > loss: 0.012998550198972225 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.13660772936141496) | > loader_time: 0.0046 (0.004656314505377708)  --> TIME: 2025-02-24 22:39:18 -- STEP: 34656/36938 -- GLOBAL_STEP: 514850 | > loss_text_ce: 0.03698474541306496 (0.03732362861454998) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13660618433056373) | > loader_time: 0.0045 (0.004656223083191828)  --> TIME: 2025-02-24 22:39:27 -- STEP: 34706/36938 -- GLOBAL_STEP: 514900 | > loss_text_ce: 0.05051420256495476 (0.03732455703776807) | > loss_mel_ce: 2.695937395095825 (nan) | > loss: 0.010898618027567863 (nan) | > current_lr: 5e-06 | > step_time: 0.108 (0.1366045327789195) | > loader_time: 0.0044 (0.004656172063875084)  --> TIME: 2025-02-24 22:39:37 -- STEP: 34756/36938 -- GLOBAL_STEP: 514950 | > loss_text_ce: 0.0301373228430748 (0.037326411392182623) | > loss_mel_ce: 2.823049306869507 (nan) | > loss: 0.011322169564664364 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13660020695304534) | > loader_time: 0.0046 (0.004656158048959225)  --> TIME: 2025-02-24 22:39:46 -- STEP: 34806/36938 -- GLOBAL_STEP: 515000 | > loss_text_ce: 0.041803028434515 (0.03732766353067144) | > loss_mel_ce: 3.4250404834747314 (nan) | > loss: 0.013757316395640373 (nan) | > current_lr: 5e-06 | > step_time: 0.1519 (0.13659768073591583) | > loader_time: 0.0047 (0.0046561485610117324)  --> TIME: 2025-02-24 22:39:55 -- STEP: 34856/36938 -- GLOBAL_STEP: 515050 | > loss_text_ce: 0.04405805096030235 (0.037327561831981604) | > loss_mel_ce: 3.3032867908477783 (nan) | > loss: 0.013283115811645985 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13659571856431338) | > loader_time: 0.0046 (0.00465611751292421)  --> TIME: 2025-02-24 22:40:04 -- STEP: 34906/36938 -- GLOBAL_STEP: 515100 | > loss_text_ce: 0.033695779740810394 (0.037327671849434106) | > loss_mel_ce: 2.949721574783325 (nan) | > loss: 0.011838957667350769 (nan) | > current_lr: 5e-06 | > step_time: 0.1461 (0.13658831038666588) | > loader_time: 0.0048 (0.004656120582357167)  --> TIME: 2025-02-24 22:40:13 -- STEP: 34956/36938 -- GLOBAL_STEP: 515150 | > loss_text_ce: 0.04160455986857414 (0.03732816386215041) | > loss_mel_ce: 3.0915825366973877 (nan) | > loss: 0.012433283030986786 (nan) | > current_lr: 5e-06 | > step_time: 0.1812 (0.13658797385586505) | > loader_time: 0.0048 (0.004656208858758426)  --> TIME: 2025-02-24 22:40:22 -- STEP: 35006/36938 -- GLOBAL_STEP: 515200 | > loss_text_ce: 0.039188820868730545 (0.03732867930772955) | > loss_mel_ce: 2.6935079097747803 (nan) | > loss: 0.0108440350741148 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13658382461076904) | > loader_time: 0.0046 (0.004656187215614442)  --> TIME: 2025-02-24 22:40:32 -- STEP: 35056/36938 -- GLOBAL_STEP: 515250 | > loss_text_ce: 0.031999699771404266 (0.0373261156718811) | > loss_mel_ce: 1.7693439722061157 (nan) | > loss: 0.007148189470171928 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1365859621412297) | > loader_time: 0.0046 (0.004656283530890787)  --> TIME: 2025-02-24 22:40:41 -- STEP: 35106/36938 -- GLOBAL_STEP: 515300 | > loss_text_ce: 0.03369821235537529 (0.0373257608818001) | > loss_mel_ce: 2.667428493499756 (nan) | > loss: 0.010718758217990398 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13658931044399994) | > loader_time: 0.0046 (0.004656461672939626)  --> TIME: 2025-02-24 22:40:51 -- STEP: 35156/36938 -- GLOBAL_STEP: 515350 | > loss_text_ce: 0.031673360615968704 (0.03732459071666703) | > loss_mel_ce: 3.4226973056793213 (nan) | > loss: 0.013707821257412434 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13658927274735025) | > loader_time: 0.0045 (0.00465645590988437)  --> TIME: 2025-02-24 22:41:00 -- STEP: 35206/36938 -- GLOBAL_STEP: 515400 | > loss_text_ce: 0.03349772095680237 (0.03732221689563418) | > loss_mel_ce: 2.800175666809082 (nan) | > loss: 0.011244736611843109 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13659063721288248) | > loader_time: 0.0047 (0.004656410614130243)  --> TIME: 2025-02-24 22:41:10 -- STEP: 35256/36938 -- GLOBAL_STEP: 515450 | > loss_text_ce: 0.03679924085736275 (0.037322560811318975) | > loss_mel_ce: 2.927130699157715 (nan) | > loss: 0.011761627160012722 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13659751763521266) | > loader_time: 0.0045 (0.004656438617065739)  --> TIME: 2025-02-24 22:41:19 -- STEP: 35306/36938 -- GLOBAL_STEP: 515500 | > loss_text_ce: 0.03186069801449776 (0.037319747526760615) | > loss_mel_ce: 0.9956578016281128 (nan) | > loss: 0.004077454563230276 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.1365960574194655) | > loader_time: 0.0045 (0.004656425260088824)  --> TIME: 2025-02-24 22:41:28 -- STEP: 35356/36938 -- GLOBAL_STEP: 515550 | > loss_text_ce: 0.04349297285079956 (0.037321406590921216) | > loss_mel_ce: 2.2717392444610596 (nan) | > loss: 0.009187430143356323 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13659343404971658) | > loader_time: 0.0048 (0.004656391312922526)  --> TIME: 2025-02-24 22:41:38 -- STEP: 35406/36938 -- GLOBAL_STEP: 515600 | > loss_text_ce: 0.022113697603344917 (0.03732298779275158) | > loss_mel_ce: 2.5559158325195312 (nan) | > loss: 0.010230276733636856 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13659704104591971) | > loader_time: 0.0045 (0.004656409850963878)  --> TIME: 2025-02-24 22:41:47 -- STEP: 35456/36938 -- GLOBAL_STEP: 515650 | > loss_text_ce: 0.03637174144387245 (0.037324908640216765) | > loss_mel_ce: 3.9318106174468994 (nan) | > loss: 0.01574675552546978 (nan) | > current_lr: 5e-06 | > step_time: 0.145 (0.1365981445940285) | > loader_time: 0.0045 (0.004656371563027469)  --> TIME: 2025-02-24 22:41:56 -- STEP: 35506/36938 -- GLOBAL_STEP: 515700 | > loss_text_ce: 0.03328946605324745 (0.037323882169393696) | > loss_mel_ce: 2.719198226928711 (nan) | > loss: 0.010922570712864399 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13659689330130717) | > loader_time: 0.0045 (0.004656340473840938)  --> TIME: 2025-02-24 22:42:06 -- STEP: 35556/36938 -- GLOBAL_STEP: 515750 | > loss_text_ce: 0.031067371368408203 (0.03732244600329683) | > loss_mel_ce: 2.2849388122558594 (nan) | > loss: 0.009190501645207405 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1366039688748923) | > loader_time: 0.0046 (0.004656363598392991)  --> TIME: 2025-02-24 22:42:15 -- STEP: 35606/36938 -- GLOBAL_STEP: 515800 | > loss_text_ce: 0.028851095587015152 (0.037321692214921445) | > loss_mel_ce: 3.14349365234375 (nan) | > loss: 0.01258867047727108 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.1366010631516175) | > loader_time: 0.0045 (0.004656292773067135)  --> TIME: 2025-02-24 22:42:25 -- STEP: 35656/36938 -- GLOBAL_STEP: 515850 | > loss_text_ce: 0.035415876656770706 (0.037320359574602446) | > loss_mel_ce: 3.6961610317230225 (nan) | > loss: 0.01480784546583891 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13661065554576032) | > loader_time: 0.0046 (0.004656260594514965)  --> TIME: 2025-02-24 22:42:34 -- STEP: 35706/36938 -- GLOBAL_STEP: 515900 | > loss_text_ce: 0.034633491188287735 (0.03731987953673436) | > loss_mel_ce: 1.5994924306869507 (nan) | > loss: 0.00648462725803256 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.1366095109552933) | > loader_time: 0.0046 (0.004656252343935196)  --> TIME: 2025-02-24 22:42:43 -- STEP: 35756/36938 -- GLOBAL_STEP: 515950 | > loss_text_ce: 0.04460856318473816 (0.0373212655942451) | > loss_mel_ce: 3.6067070960998535 (nan) | > loss: 0.014489348977804184 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.1366112220539368) | > loader_time: 0.0049 (0.0046562533315122195)  --> TIME: 2025-02-24 22:42:53 -- STEP: 35806/36938 -- GLOBAL_STEP: 516000 | > loss_text_ce: 0.03143935278058052 (0.03732228827643634) | > loss_mel_ce: 2.3549063205718994 (nan) | > loss: 0.009469626471400261 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1366132648205719) | > loader_time: 0.0048 (0.004656283720800902)  --> TIME: 2025-02-24 22:43:02 -- STEP: 35856/36938 -- GLOBAL_STEP: 516050 | > loss_text_ce: 0.024007441475987434 (0.03732156106988212) | > loss_mel_ce: 1.8227523565292358 (nan) | > loss: 0.0073284124955534935 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13660880776985765) | > loader_time: 0.0046 (0.004656256827750611)  --> TIME: 2025-02-24 22:43:11 -- STEP: 35906/36938 -- GLOBAL_STEP: 516100 | > loss_text_ce: 0.031399209052324295 (0.03731927850410687) | > loss_mel_ce: 2.615851640701294 (nan) | > loss: 0.010504964739084244 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13659709485162932) | > loader_time: 0.0045 (0.004656208960556979)  --> TIME: 2025-02-24 22:43:20 -- STEP: 35956/36938 -- GLOBAL_STEP: 516150 | > loss_text_ce: 0.03081515058875084 (0.037319895987639844) | > loss_mel_ce: 2.9369215965270996 (nan) | > loss: 0.01177673414349556 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.1365932304226045) | > loader_time: 0.0046 (0.004656214591512306)  --> TIME: 2025-02-24 22:43:29 -- STEP: 36006/36938 -- GLOBAL_STEP: 516200 | > loss_text_ce: 0.03710697218775749 (0.037321062364861844) | > loss_mel_ce: 2.7717831134796143 (nan) | > loss: 0.01114638987928629 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13659938234743332) | > loader_time: 0.0045 (0.004656635581973589)  --> TIME: 2025-02-24 22:43:39 -- STEP: 36056/36938 -- GLOBAL_STEP: 516250 | > loss_text_ce: 0.041770294308662415 (0.03732205904628723) | > loss_mel_ce: 2.2397682666778564 (nan) | > loss: 0.009053724817931652 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13659667882126839) | > loader_time: 0.0046 (0.004656623446388557)  --> TIME: 2025-02-24 22:43:48 -- STEP: 36106/36938 -- GLOBAL_STEP: 516300 | > loss_text_ce: 0.03347977250814438 (0.03731919334341268) | > loss_mel_ce: 3.3289389610290527 (nan) | > loss: 0.013342931866645813 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.1366001040969982) | > loader_time: 0.0048 (0.0046566504227159395)  --> TIME: 2025-02-24 22:43:57 -- STEP: 36156/36938 -- GLOBAL_STEP: 516350 | > loss_text_ce: 0.03829270601272583 (0.0373187738418749) | > loss_mel_ce: 2.983943462371826 (nan) | > loss: 0.011993001215159893 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.13660116399570213) | > loader_time: 0.0047 (0.0046566358141725385)  --> TIME: 2025-02-24 22:44:07 -- STEP: 36206/36938 -- GLOBAL_STEP: 516400 | > loss_text_ce: 0.036697372794151306 (0.037323320957164466) | > loss_mel_ce: 3.1234395503997803 (nan) | > loss: 0.012540226802229881 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13660008474153024) | > loader_time: 0.0046 (0.004656651240912847)  --> TIME: 2025-02-24 22:44:16 -- STEP: 36256/36938 -- GLOBAL_STEP: 516450 | > loss_text_ce: 0.053304288536310196 (0.03732201762213079) | > loss_mel_ce: 0.14453125 (nan) | > loss: 0.0007850616821087897 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.1365978501980426) | > loader_time: 0.0047 (0.0046566415574838755)  --> TIME: 2025-02-24 22:44:25 -- STEP: 36306/36938 -- GLOBAL_STEP: 516500 | > loss_text_ce: 0.03609316051006317 (0.03732062064796924) | > loss_mel_ce: 3.4197475910186768 (nan) | > loss: 0.013713655062019825 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13659761355598693) | > loader_time: 0.0046 (0.00465663779125339)  --> TIME: 2025-02-24 22:44:35 -- STEP: 36356/36938 -- GLOBAL_STEP: 516550 | > loss_text_ce: 0.03902529180049896 (0.03732095451311604) | > loss_mel_ce: 3.2969985008239746 (nan) | > loss: 0.013238190673291683 (nan) | > current_lr: 5e-06 | > step_time: 0.1622 (0.1366013284300874) | > loader_time: 0.0047 (0.0046566324024680805)  --> TIME: 2025-02-24 22:44:44 -- STEP: 36406/36938 -- GLOBAL_STEP: 516600 | > loss_text_ce: 0.05409817025065422 (0.037323862994360975) | > loss_mel_ce: 1.9736660718917847 (nan) | > loss: 0.008046683855354786 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13660172806960716) | > loader_time: 0.0046 (0.004656765615914533)  --> TIME: 2025-02-24 22:44:53 -- STEP: 36456/36938 -- GLOBAL_STEP: 516650 | > loss_text_ce: 0.03807318955659866 (0.03732509191142525) | > loss_mel_ce: 4.5085225105285645 (nan) | > loss: 0.018042046576738358 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13660083567596482) | > loader_time: 0.0046 (0.004656710337200819)  --> TIME: 2025-02-24 22:45:02 -- STEP: 36506/36938 -- GLOBAL_STEP: 516700 | > loss_text_ce: 0.036357391625642776 (0.0373247435295979) | > loss_mel_ce: 3.401634454727173 (nan) | > loss: 0.013642825186252594 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13658949552689686) | > loader_time: 0.0046 (0.004656646314767349)  --> TIME: 2025-02-24 22:45:11 -- STEP: 36556/36938 -- GLOBAL_STEP: 516750 | > loss_text_ce: 0.03560587763786316 (0.037324924368058475) | > loss_mel_ce: 3.2421875 (nan) | > loss: 0.013007117435336113 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13658420442686964) | > loader_time: 0.0045 (0.004656656622717065)  --> TIME: 2025-02-24 22:45:21 -- STEP: 36606/36938 -- GLOBAL_STEP: 516800 | > loss_text_ce: 0.052781905978918076 (0.03732773307990192) | > loss_mel_ce: 0.319050669670105 (nan) | > loss: 0.0014755261363461614 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13658794200414473) | > loader_time: 0.0046 (0.004656744011111601)  --> TIME: 2025-02-24 22:45:30 -- STEP: 36656/36938 -- GLOBAL_STEP: 516850 | > loss_text_ce: 0.03742043301463127 (0.03732641126097848) | > loss_mel_ce: 1.8066593408584595 (nan) | > loss: 0.007317777257412672 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13658273318926306) | > loader_time: 0.0046 (0.004656751217773397)  --> TIME: 2025-02-24 22:45:39 -- STEP: 36706/36938 -- GLOBAL_STEP: 516900 | > loss_text_ce: 0.03190593793988228 (0.03732486928005305) | > loss_mel_ce: 3.1603026390075684 (nan) | > loss: 0.01266749482601881 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13658952781863426) | > loader_time: 0.0048 (0.004656747765406735)  --> TIME: 2025-02-24 22:45:49 -- STEP: 36756/36938 -- GLOBAL_STEP: 516950 | > loss_text_ce: 0.04020172357559204 (0.03732635817825034) | > loss_mel_ce: 3.453547477722168 (nan) | > loss: 0.013864085078239441 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13659156687514665) | > loader_time: 0.0047 (0.0046567115654998666)  --> TIME: 2025-02-24 22:45:58 -- STEP: 36806/36938 -- GLOBAL_STEP: 517000 | > loss_text_ce: 0.04419679939746857 (0.037325771410117524) | > loss_mel_ce: 1.603865385055542 (nan) | > loss: 0.006539929658174515 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.1365961997345629) | > loader_time: 0.0045 (0.0046568116318795255)  --> TIME: 2025-02-24 22:46:07 -- STEP: 36856/36938 -- GLOBAL_STEP: 517050 | > loss_text_ce: 0.04755032807588577 (0.037328969739024845) | > loss_mel_ce: 2.8083248138427734 (nan) | > loss: 0.011332838796079159 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13658729673648046) | > loader_time: 0.0046 (0.00465674604230001)  --> TIME: 2025-02-24 22:46:16 -- STEP: 36906/36938 -- GLOBAL_STEP: 517100 | > loss_text_ce: 0.04194820672273636 (0.03732942327397081) | > loss_mel_ce: 3.144019365310669 (nan) | > loss: 0.01264273002743721 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13658311679964893) | > loader_time: 0.0046 (0.0046570174695223935)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.02066198670037901 (-7.373271602231779e-06) | > avg_loss_text_ce: 0.03636924698395604 (-0.0005061722332887902) | > avg_loss_mel_ce: 2.6537022599489384 (-0.02330169553803918) | > avg_loss: 2.6900715007640335 (-0.023807880902056766) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_517132.pth  > EPOCH: 15/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-24 22:46:47)   --> TIME: 2025-02-24 22:46:51 -- STEP: 18/36938 -- GLOBAL_STEP: 517150 | > loss_text_ce: 0.03420731797814369 (0.03530457988381386) | > loss_mel_ce: 2.9434988498687744 (2.6197958985964456) | > loss: 0.011816294863820076 (0.01053611361162944) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13631179597642687) | > loader_time: 0.0049 (0.004629691441853841)  --> TIME: 2025-02-24 22:47:00 -- STEP: 68/36938 -- GLOBAL_STEP: 517200 | > loss_text_ce: 0.03850619122385979 (0.036385792239076085) | > loss_mel_ce: 2.8899409770965576 (2.5689427092233124) | > loss: 0.011620823293924332 (0.010338605779586086) | > current_lr: 5e-06 | > step_time: 0.1628 (0.1359301349695991) | > loader_time: 0.0047 (0.00473095739589018)  --> TIME: 2025-02-24 22:47:09 -- STEP: 118/36938 -- GLOBAL_STEP: 517250 | > loss_text_ce: 0.03657227009534836 (0.036482367972385604) | > loss_mel_ce: 3.4258577823638916 (nan) | > loss: 0.013739801943302155 (nan) | > current_lr: 5e-06 | > step_time: 0.1355 (0.1338921118590791) | > loader_time: 0.0047 (0.004754155369128212)  --> TIME: 2025-02-24 22:47:19 -- STEP: 168/36938 -- GLOBAL_STEP: 517300 | > loss_text_ce: 0.04541271924972534 (0.03694144103653907) | > loss_mel_ce: 2.110086441040039 (nan) | > loss: 0.008553569205105305 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13446806300254088) | > loader_time: 0.0045 (0.004755755265553794)  --> TIME: 2025-02-24 22:47:28 -- STEP: 218/36938 -- GLOBAL_STEP: 517350 | > loss_text_ce: 0.0313757099211216 (0.03714213348121954) | > loss_mel_ce: 3.7195518016815186 (nan) | > loss: 0.01488463394343853 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13496052890742571) | > loader_time: 0.0046 (0.004760452366750176)  --> TIME: 2025-02-24 22:47:38 -- STEP: 268/36938 -- GLOBAL_STEP: 517400 | > loss_text_ce: 0.041237711906433105 (0.03716713714468946) | > loss_mel_ce: 3.2426235675811768 (nan) | > loss: 0.013031195849180222 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13623464641286345) | > loader_time: 0.0047 (0.004757261987942371)  --> TIME: 2025-02-24 22:47:47 -- STEP: 318/36938 -- GLOBAL_STEP: 517450 | > loss_text_ce: 0.0392853245139122 (0.03708271997682044) | > loss_mel_ce: 3.0432565212249756 (nan) | > loss: 0.012232310138642788 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.13611853572557536) | > loader_time: 0.0045 (0.00473853777039726)  --> TIME: 2025-02-24 22:47:56 -- STEP: 368/36938 -- GLOBAL_STEP: 517500 | > loss_text_ce: 0.033671483397483826 (0.03712098221750116) | > loss_mel_ce: 3.5378541946411133 (nan) | > loss: 0.014172721654176712 (nan) | > current_lr: 5e-06 | > step_time: 0.1807 (0.13606568134349303) | > loader_time: 0.0049 (0.0047341837831165546)  --> TIME: 2025-02-24 22:48:06 -- STEP: 418/36938 -- GLOBAL_STEP: 517550 | > loss_text_ce: 0.041006412357091904 (0.037114146693596216) | > loss_mel_ce: 2.5383243560791016 (nan) | > loss: 0.010235439985990524 (nan) | > current_lr: 5e-06 | > step_time: 0.1724 (0.13666587697261806) | > loader_time: 0.0048 (0.00473686325493042)  --> TIME: 2025-02-24 22:48:15 -- STEP: 468/36938 -- GLOBAL_STEP: 517600 | > loss_text_ce: 0.03449256718158722 (0.03723975677544674) | > loss_mel_ce: 2.9254744052886963 (nan) | > loss: 0.011745900847017765 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13648567291406463) | > loader_time: 0.0047 (0.004729016723795839)  --> TIME: 2025-02-24 22:48:24 -- STEP: 518/36938 -- GLOBAL_STEP: 517650 | > loss_text_ce: 0.030955173075199127 (0.03739380203439168) | > loss_mel_ce: 3.1962268352508545 (nan) | > loss: 0.012806277722120285 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1364539162072434) | > loader_time: 0.0045 (0.004726019144978767)  --> TIME: 2025-02-24 22:48:34 -- STEP: 568/36938 -- GLOBAL_STEP: 517700 | > loss_text_ce: 0.02901778183877468 (0.03729667023814632) | > loss_mel_ce: 2.5035340785980225 (nan) | > loss: 0.010049808770418167 (nan) | > current_lr: 5e-06 | > step_time: 0.1518 (0.13652295923568822) | > loader_time: 0.0049 (0.0047209896671939955)  --> TIME: 2025-02-24 22:48:43 -- STEP: 618/36938 -- GLOBAL_STEP: 517750 | > loss_text_ce: 0.044376615434885025 (0.03740453251568192) | > loss_mel_ce: 3.2733094692230225 (nan) | > loss: 0.013165421783924103 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.1365226723229615) | > loader_time: 0.0047 (0.004716390930718978)  --> TIME: 2025-02-24 22:48:52 -- STEP: 668/36938 -- GLOBAL_STEP: 517800 | > loss_text_ce: 0.044815581291913986 (0.037342244085805554) | > loss_mel_ce: 1.6709208488464355 (nan) | > loss: 0.00680847791954875 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13633896740610726) | > loader_time: 0.0045 (0.004713622038949751)  --> TIME: 2025-02-24 22:49:01 -- STEP: 718/36938 -- GLOBAL_STEP: 517850 | > loss_text_ce: 0.041089821606874466 (0.037387737274055734) | > loss_mel_ce: 2.6179842948913574 (nan) | > loss: 0.010551881976425648 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.1364418660033711) | > loader_time: 0.0046 (0.004717502089263999)  --> TIME: 2025-02-24 22:49:11 -- STEP: 768/36938 -- GLOBAL_STEP: 517900 | > loss_text_ce: 0.042538393288850784 (0.037397603865732224) | > loss_mel_ce: 3.0025856494903564 (nan) | > loss: 0.012083826586604118 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13640896975994096) | > loader_time: 0.0046 (0.00471962677935759)  --> TIME: 2025-02-24 22:49:20 -- STEP: 818/36938 -- GLOBAL_STEP: 517950 | > loss_text_ce: 0.041243765503168106 (0.0374046636968486) | > loss_mel_ce: 2.5583457946777344 (nan) | > loss: 0.010315832681953907 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.13655761226756635) | > loader_time: 0.0047 (0.004722971496488761)  --> TIME: 2025-02-24 22:49:30 -- STEP: 868/36938 -- GLOBAL_STEP: 518000 | > loss_text_ce: 0.03480909392237663 (0.03733197536774419) | > loss_mel_ce: 3.171360969543457 (nan) | > loss: 0.01272289827466011 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13661410292172746) | > loader_time: 0.0045 (0.004717482674506401)  --> TIME: 2025-02-24 22:49:39 -- STEP: 918/36938 -- GLOBAL_STEP: 518050 | > loss_text_ce: 0.04076879099011421 (0.037335894139552496) | > loss_mel_ce: 3.0556275844573975 (nan) | > loss: 0.012287287972867489 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13663465732582758) | > loader_time: 0.0068 (0.004716074544619889)  --> TIME: 2025-02-24 22:49:48 -- STEP: 968/36938 -- GLOBAL_STEP: 518100 | > loss_text_ce: 0.03853241354227066 (0.03730187202757603) | > loss_mel_ce: 3.400162696838379 (nan) | > loss: 0.013645616360008717 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.13679605648537296) | > loader_time: 0.0046 (0.004717023658358359)  --> TIME: 2025-02-24 22:49:58 -- STEP: 1018/36938 -- GLOBAL_STEP: 518150 | > loss_text_ce: 0.02622894197702408 (0.03727773861860767) | > loss_mel_ce: 2.9074888229370117 (nan) | > loss: 0.011641738004982471 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13689144491446728) | > loader_time: 0.0048 (0.0047158150401237185)  --> TIME: 2025-02-24 22:50:08 -- STEP: 1068/36938 -- GLOBAL_STEP: 518200 | > loss_text_ce: 0.034636806696653366 (0.0372864327128376) | > loss_mel_ce: 2.73046875 (nan) | > loss: 0.010972641408443451 (nan) | > current_lr: 5e-06 | > step_time: 0.1074 (0.13719343960508407) | > loader_time: 0.0045 (0.004721098624811634)  --> TIME: 2025-02-24 22:50:17 -- STEP: 1118/36938 -- GLOBAL_STEP: 518250 | > loss_text_ce: 0.038066036999225616 (0.03722693799613354) | > loss_mel_ce: 2.5840466022491455 (nan) | > loss: 0.010405209846794605 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13713186755376539) | > loader_time: 0.0045 (0.004716457962350041)  --> TIME: 2025-02-24 22:50:26 -- STEP: 1168/36938 -- GLOBAL_STEP: 518300 | > loss_text_ce: 0.035610638558864594 (0.03722607936512369) | > loss_mel_ce: 3.3375051021575928 (nan) | > loss: 0.013385380618274212 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13719087430875584) | > loader_time: 0.0053 (0.004715228325700103)  --> TIME: 2025-02-24 22:50:36 -- STEP: 1218/36938 -- GLOBAL_STEP: 518350 | > loss_text_ce: 0.04176437109708786 (0.037251285815608375) | > loss_mel_ce: 2.2467386722564697 (nan) | > loss: 0.009081361815333366 (nan) | > current_lr: 5e-06 | > step_time: 0.1807 (0.13708338263782557) | > loader_time: 0.0048 (0.004719148715728606)  --> TIME: 2025-02-24 22:50:45 -- STEP: 1268/36938 -- GLOBAL_STEP: 518400 | > loss_text_ce: 0.03374939784407616 (0.037284493729859924) | > loss_mel_ce: 2.848435640335083 (nan) | > loss: 0.011437242850661278 (nan) | > current_lr: 5e-06 | > step_time: 0.1625 (0.13702321466211262) | > loader_time: 0.0048 (0.004714828936459508)  --> TIME: 2025-02-24 22:50:54 -- STEP: 1318/36938 -- GLOBAL_STEP: 518450 | > loss_text_ce: 0.0347631499171257 (0.03731700748188222) | > loss_mel_ce: 3.4538042545318604 (nan) | > loss: 0.013843522407114506 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13708313553394788) | > loader_time: 0.0044 (0.004710728011471362)  --> TIME: 2025-02-24 22:51:04 -- STEP: 1368/36938 -- GLOBAL_STEP: 518500 | > loss_text_ce: 0.05070292949676514 (0.03735100767134047) | > loss_mel_ce: 2.8594775199890137 (nan) | > loss: 0.01154833659529686 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13709904819901225) | > loader_time: 0.0047 (0.004708725458000133)  --> TIME: 2025-02-24 22:51:13 -- STEP: 1418/36938 -- GLOBAL_STEP: 518550 | > loss_text_ce: 0.03483561798930168 (0.03731965903345128) | > loss_mel_ce: 2.5358822345733643 (nan) | > loss: 0.01020126137882471 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13717656327235503) | > loader_time: 0.0044 (0.0047060115044813)  --> TIME: 2025-02-24 22:51:23 -- STEP: 1468/36938 -- GLOBAL_STEP: 518600 | > loss_text_ce: 0.03812333941459656 (0.03735815959650133) | > loss_mel_ce: 0.3285726010799408 (nan) | > loss: 0.0014551427448168397 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13723970515201772) | > loader_time: 0.0046 (0.004705633227117048)  --> TIME: 2025-02-24 22:51:32 -- STEP: 1518/36938 -- GLOBAL_STEP: 518650 | > loss_text_ce: 0.028772583231329918 (0.03739813945289296) | > loss_mel_ce: 1.655259370803833 (nan) | > loss: 0.006682666949927807 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.1372126400549262) | > loader_time: 0.0045 (0.00470255014924663)  --> TIME: 2025-02-24 22:51:41 -- STEP: 1568/36938 -- GLOBAL_STEP: 518700 | > loss_text_ce: 0.039372518658638 (0.0374372378348049) | > loss_mel_ce: 2.8096680641174316 (nan) | > loss: 0.011305716820061207 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13731493299104705) | > loader_time: 0.0045 (0.004703273274460608)  --> TIME: 2025-02-24 22:51:51 -- STEP: 1618/36938 -- GLOBAL_STEP: 518750 | > loss_text_ce: 0.03494200482964516 (0.037416062003510296) | > loss_mel_ce: 3.3558638095855713 (nan) | > loss: 0.013455579057335854 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.13722748617896027) | > loader_time: 0.0047 (0.004702474484484916)  --> TIME: 2025-02-24 22:52:00 -- STEP: 1668/36938 -- GLOBAL_STEP: 518800 | > loss_text_ce: 0.03696431219577789 (0.03739899455746114) | > loss_mel_ce: 3.2734203338623047 (nan) | > loss: 0.013136448338627815 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.13718759856349821) | > loader_time: 0.0046 (0.004702622084308863)  --> TIME: 2025-02-24 22:52:09 -- STEP: 1718/36938 -- GLOBAL_STEP: 518850 | > loss_text_ce: 0.03552497550845146 (0.03739852497811673) | > loss_mel_ce: 0.017669677734375 (nan) | > loss: 0.0002110899076797068 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.1371917398483842) | > loader_time: 0.0046 (0.004700894960840861)  --> TIME: 2025-02-24 22:52:19 -- STEP: 1768/36938 -- GLOBAL_STEP: 518900 | > loss_text_ce: 0.030300090089440346 (0.03741743498717795) | > loss_mel_ce: 3.1382288932800293 (nan) | > loss: 0.012573529034852982 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13724517741354308) | > loader_time: 0.0046 (0.004701887994869796)  --> TIME: 2025-02-24 22:52:28 -- STEP: 1818/36938 -- GLOBAL_STEP: 518950 | > loss_text_ce: 0.036063384264707565 (0.03743941710628686) | > loss_mel_ce: 3.4032223224639893 (nan) | > loss: 0.013647960498929024 (nan) | > current_lr: 5e-06 | > step_time: 0.1651 (0.13716545264975305) | > loader_time: 0.0048 (0.0047014798268233135)  --> TIME: 2025-02-24 22:52:37 -- STEP: 1868/36938 -- GLOBAL_STEP: 519000 | > loss_text_ce: 0.036990515887737274 (0.0373962109348072) | > loss_mel_ce: 1.3064913749694824 (nan) | > loss: 0.005331277847290039 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13719835641297304) | > loader_time: 0.0045 (0.004698855973923737)  --> TIME: 2025-02-24 22:52:47 -- STEP: 1918/36938 -- GLOBAL_STEP: 519050 | > loss_text_ce: 0.04381304234266281 (0.03738174302653016) | > loss_mel_ce: 3.493384838104248 (nan) | > loss: 0.01403649989515543 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13728749553155323) | > loader_time: 0.0045 (0.00469647271789773)  --> TIME: 2025-02-24 22:52:56 -- STEP: 1968/36938 -- GLOBAL_STEP: 519100 | > loss_text_ce: 0.032537929713726044 (0.03736070261953953) | > loss_mel_ce: 3.2046146392822266 (nan) | > loss: 0.012845844030380249 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13738640839975996) | > loader_time: 0.0047 (0.004695112506548558)  --> TIME: 2025-02-24 22:53:06 -- STEP: 2018/36938 -- GLOBAL_STEP: 519150 | > loss_text_ce: 0.030579954385757446 (0.03733739374358685) | > loss_mel_ce: 3.102306842803955 (nan) | > loss: 0.012432091869413853 (nan) | > current_lr: 5e-06 | > step_time: 0.1701 (0.13737834291486456) | > loader_time: 0.0047 (0.004694163149483017)  --> TIME: 2025-02-24 22:53:15 -- STEP: 2068/36938 -- GLOBAL_STEP: 519200 | > loss_text_ce: 0.03037405200302601 (0.037309154687986096) | > loss_mel_ce: 2.681279182434082 (nan) | > loss: 0.010760528966784477 (nan) | > current_lr: 5e-06 | > step_time: 0.1502 (0.13741716713693175) | > loader_time: 0.0046 (0.0046924426431120886)  --> TIME: 2025-02-24 22:53:24 -- STEP: 2118/36938 -- GLOBAL_STEP: 519250 | > loss_text_ce: 0.04348604008555412 (0.03728801763868895) | > loss_mel_ce: 2.72235107421875 (nan) | > loss: 0.010975545272231102 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.1373375151042558) | > loader_time: 0.0046 (0.004692357367676755)  --> TIME: 2025-02-24 22:53:34 -- STEP: 2168/36938 -- GLOBAL_STEP: 519300 | > loss_text_ce: 0.03354303911328316 (0.0372739507356225) | > loss_mel_ce: 2.212136745452881 (nan) | > loss: 0.008911428973078728 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13737681717010422) | > loader_time: 0.0052 (0.004692339039376736)  --> TIME: 2025-02-24 22:53:43 -- STEP: 2218/36938 -- GLOBAL_STEP: 519350 | > loss_text_ce: 0.033574894070625305 (0.03724865117374838) | > loss_mel_ce: 2.7934558391571045 (nan) | > loss: 0.011218376457691193 (nan) | > current_lr: 5e-06 | > step_time: 0.176 (0.13736387917971793) | > loader_time: 0.0047 (0.004692407961467883)  --> TIME: 2025-02-24 22:53:53 -- STEP: 2268/36938 -- GLOBAL_STEP: 519400 | > loss_text_ce: 0.03465894237160683 (0.03722202027466213) | > loss_mel_ce: 2.5347344875335693 (nan) | > loss: 0.010196005925536156 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13735507394481158) | > loader_time: 0.0047 (0.00469277008080188)  --> TIME: 2025-02-24 22:54:02 -- STEP: 2318/36938 -- GLOBAL_STEP: 519450 | > loss_text_ce: 0.03761661797761917 (0.03720812263518464) | > loss_mel_ce: 3.3392562866210938 (nan) | > loss: 0.013400290161371231 (nan) | > current_lr: 5e-06 | > step_time: 0.1648 (0.13735486136313843) | > loader_time: 0.0047 (0.004692788161112378)  --> TIME: 2025-02-24 22:54:11 -- STEP: 2368/36938 -- GLOBAL_STEP: 519500 | > loss_text_ce: 0.037941284477710724 (0.0372058537496077) | > loss_mel_ce: 3.4237871170043945 (nan) | > loss: 0.013737018220126629 (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.13729539674681565) | > loader_time: 0.0046 (0.004691949164545213)  --> TIME: 2025-02-24 22:54:20 -- STEP: 2418/36938 -- GLOBAL_STEP: 519550 | > loss_text_ce: 0.028749214485287666 (0.03723059956683803) | > loss_mel_ce: 2.8997085094451904 (nan) | > loss: 0.011620864272117615 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13721798233398225) | > loader_time: 0.0045 (0.0046911808555041385)  --> TIME: 2025-02-24 22:54:29 -- STEP: 2468/36938 -- GLOBAL_STEP: 519600 | > loss_text_ce: 0.030695781111717224 (0.03725322151958114) | > loss_mel_ce: 2.956868886947632 (nan) | > loss: 0.011855415999889374 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1371074643475116) | > loader_time: 0.0045 (0.004689497344884142)  --> TIME: 2025-02-24 22:54:39 -- STEP: 2518/36938 -- GLOBAL_STEP: 519650 | > loss_text_ce: 0.02912379428744316 (0.03724394481469344) | > loss_mel_ce: 1.033012866973877 (nan) | > loss: 0.004214828368276358 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13714510661633453) | > loader_time: 0.0045 (0.0046881826459840665)  --> TIME: 2025-02-24 22:54:48 -- STEP: 2568/36938 -- GLOBAL_STEP: 519700 | > loss_text_ce: 0.031827617436647415 (0.03722842853673002) | > loss_mel_ce: 2.3794708251953125 (nan) | > loss: 0.009568645618855953 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13710609524049483) | > loader_time: 0.0053 (0.0046893774163314465)  --> TIME: 2025-02-24 22:54:57 -- STEP: 2618/36938 -- GLOBAL_STEP: 519750 | > loss_text_ce: 0.03826339542865753 (0.03721609524955627) | > loss_mel_ce: 2.776787281036377 (nan) | > loss: 0.011170836165547371 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13711048401977557) | > loader_time: 0.0046 (0.004689737710450058)  --> TIME: 2025-02-24 22:55:06 -- STEP: 2668/36938 -- GLOBAL_STEP: 519800 | > loss_text_ce: 0.04312099516391754 (0.03720895456697515) | > loss_mel_ce: 3.348430871963501 (nan) | > loss: 0.01345854066312313 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.13706330997475621) | > loader_time: 0.0047 (0.0046893794318546575)  --> TIME: 2025-02-24 22:55:16 -- STEP: 2718/36938 -- GLOBAL_STEP: 519850 | > loss_text_ce: 0.047228164970874786 (0.037193860664738926) | > loss_mel_ce: 3.609375 (nan) | > loss: 0.014510330744087696 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13709556380995425) | > loader_time: 0.0047 (0.004690671949547998)  --> TIME: 2025-02-24 22:55:25 -- STEP: 2768/36938 -- GLOBAL_STEP: 519900 | > loss_text_ce: 0.025137756019830704 (0.03721193315439742) | > loss_mel_ce: 2.568091630935669 (nan) | > loss: 0.010290592908859253 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1371594968903272) | > loader_time: 0.0045 (0.004689875624083369)  --> TIME: 2025-02-24 22:55:35 -- STEP: 2818/36938 -- GLOBAL_STEP: 519950 | > loss_text_ce: 0.03109123930335045 (0.03721131386950539) | > loss_mel_ce: 2.704519033432007 (nan) | > loss: 0.010855596512556076 (nan) | > current_lr: 5e-06 | > step_time: 0.1507 (0.13721075262728966) | > loader_time: 0.0046 (0.004688435957903891)  --> TIME: 2025-02-24 22:55:44 -- STEP: 2868/36938 -- GLOBAL_STEP: 520000 | > loss_text_ce: 0.041278570890426636 (0.037226571434539855) | > loss_mel_ce: 3.3002147674560547 (nan) | > loss: 0.013259895145893097 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.1371027769902761) | > loader_time: 0.0046 (0.004687696903154272) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_520000.pth  --> TIME: 2025-02-24 22:55:59 -- STEP: 2918/36938 -- GLOBAL_STEP: 520050 | > loss_text_ce: 0.03431147336959839 (0.03725635958995067) | > loss_mel_ce: 1.3962880373001099 (nan) | > loss: 0.005676982458680868 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13708518342337778) | > loader_time: 0.0045 (0.004688541557463655)  --> TIME: 2025-02-24 22:56:09 -- STEP: 2968/36938 -- GLOBAL_STEP: 520100 | > loss_text_ce: 0.04187449812889099 (0.03724972334517183) | > loss_mel_ce: 3.2134690284729004 (nan) | > loss: 0.01291802991181612 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13710768372222112) | > loader_time: 0.0047 (0.004687432571241474)  --> TIME: 2025-02-24 22:56:18 -- STEP: 3018/36938 -- GLOBAL_STEP: 520150 | > loss_text_ce: 0.033660005778074265 (0.037262540447634954) | > loss_mel_ce: 3.3286290168762207 (nan) | > loss: 0.01334241684526205 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13706047065689392) | > loader_time: 0.0046 (0.004686417051938443)  --> TIME: 2025-02-24 22:56:27 -- STEP: 3068/36938 -- GLOBAL_STEP: 520200 | > loss_text_ce: 0.031230835244059563 (0.03727107288345036) | > loss_mel_ce: 2.813059091567993 (nan) | > loss: 0.011286865919828415 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13705359086530336) | > loader_time: 0.0046 (0.004687310663941935)  --> TIME: 2025-02-24 22:56:36 -- STEP: 3118/36938 -- GLOBAL_STEP: 520250 | > loss_text_ce: 0.034522779285907745 (0.03729559161662674) | > loss_mel_ce: 1.95025634765625 (nan) | > loss: 0.007876108400523663 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.136968026717861) | > loader_time: 0.0045 (0.004687153734864136)  --> TIME: 2025-02-24 22:56:45 -- STEP: 3168/36938 -- GLOBAL_STEP: 520300 | > loss_text_ce: 0.038166843354701996 (0.03729516268070471) | > loss_mel_ce: 2.7551472187042236 (nan) | > loss: 0.01108457986265421 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13694867935746638) | > loader_time: 0.0055 (0.004686933424737712)  --> TIME: 2025-02-24 22:56:55 -- STEP: 3218/36938 -- GLOBAL_STEP: 520350 | > loss_text_ce: 0.03185463324189186 (0.03729813584144482) | > loss_mel_ce: 2.943235397338867 (nan) | > loss: 0.011805913411080837 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.13698732341721337) | > loader_time: 0.0046 (0.0046887245142659875)  --> TIME: 2025-02-24 22:57:04 -- STEP: 3268/36938 -- GLOBAL_STEP: 520400 | > loss_text_ce: 0.04528368264436722 (0.03729413321294179) | > loss_mel_ce: 3.069045305252075 (nan) | > loss: 0.012358449399471283 (nan) | > current_lr: 5e-06 | > step_time: 0.1798 (0.13693019999430567) | > loader_time: 0.0048 (0.004688976590169393)  --> TIME: 2025-02-24 22:57:13 -- STEP: 3318/36938 -- GLOBAL_STEP: 520450 | > loss_text_ce: 0.043937262147665024 (0.03727801915734222) | > loss_mel_ce: 1.8124738931655884 (nan) | > loss: 0.007366711273789406 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13687425309308285) | > loader_time: 0.0045 (0.004689059392596813)  --> TIME: 2025-02-24 22:57:23 -- STEP: 3368/36938 -- GLOBAL_STEP: 520500 | > loss_text_ce: 0.030009597539901733 (0.037282993753204034) | > loss_mel_ce: 2.7631709575653076 (nan) | > loss: 0.011084049940109253 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13693203984133742) | > loader_time: 0.0045 (0.004688878594554797)  --> TIME: 2025-02-24 22:57:32 -- STEP: 3418/36938 -- GLOBAL_STEP: 520550 | > loss_text_ce: 0.03278631344437599 (0.037279303130601675) | > loss_mel_ce: 3.739018201828003 (nan) | > loss: 0.01496747974306345 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.1369129539303368) | > loader_time: 0.0047 (0.004689925521908617)  --> TIME: 2025-02-24 22:57:41 -- STEP: 3468/36938 -- GLOBAL_STEP: 520600 | > loss_text_ce: 0.03984957933425903 (0.037291822037654236) | > loss_mel_ce: 2.556997060775757 (nan) | > loss: 0.010304947383701801 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13689207686959978) | > loader_time: 0.0046 (0.004689971880929276)  --> TIME: 2025-02-24 22:57:50 -- STEP: 3518/36938 -- GLOBAL_STEP: 520650 | > loss_text_ce: 0.03353937342762947 (0.03730088146482854) | > loss_mel_ce: 3.318406105041504 (nan) | > loss: 0.013301371596753597 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13685132523298701) | > loader_time: 0.0048 (0.004690376854268168)  --> TIME: 2025-02-24 22:58:00 -- STEP: 3568/36938 -- GLOBAL_STEP: 520700 | > loss_text_ce: 0.04574000835418701 (0.03727528674740998) | > loss_mel_ce: 3.0594685077667236 (nan) | > loss: 0.012322256341576576 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.136924343459275) | > loader_time: 0.0052 (0.004691252793966383)  --> TIME: 2025-02-24 22:58:09 -- STEP: 3618/36938 -- GLOBAL_STEP: 520750 | > loss_text_ce: 0.03092525340616703 (0.03727840023983278) | > loss_mel_ce: 2.504875659942627 (nan) | > loss: 0.010062702931463718 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.13688006560534946) | > loader_time: 0.005 (0.004691171936068892)  --> TIME: 2025-02-24 22:58:19 -- STEP: 3668/36938 -- GLOBAL_STEP: 520800 | > loss_text_ce: 0.03927609696984291 (0.03725052225142476) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.1369054449423582) | > loader_time: 0.0046 (0.00469153085997736)  --> TIME: 2025-02-24 22:58:28 -- STEP: 3718/36938 -- GLOBAL_STEP: 520850 | > loss_text_ce: 0.038440559059381485 (0.03725144588912029) | > loss_mel_ce: 2.9076249599456787 (nan) | > loss: 0.011690736748278141 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13688042710198095) | > loader_time: 0.0046 (0.004692299575046415)  --> TIME: 2025-02-24 22:58:37 -- STEP: 3768/36938 -- GLOBAL_STEP: 520900 | > loss_text_ce: 0.04117429628968239 (0.03723695203306002) | > loss_mel_ce: 3.1968650817871094 (nan) | > loss: 0.012849363498389721 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.13689904778626327) | > loader_time: 0.0046 (0.004692339656712393)  --> TIME: 2025-02-24 22:58:46 -- STEP: 3818/36938 -- GLOBAL_STEP: 520950 | > loss_text_ce: 0.033557381480932236 (0.03723724360634844) | > loss_mel_ce: 3.421215772628784 (nan) | > loss: 0.0137094184756279 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.1368060225026655) | > loader_time: 0.0046 (0.004692188728036772)  --> TIME: 2025-02-24 22:58:55 -- STEP: 3868/36938 -- GLOBAL_STEP: 521000 | > loss_text_ce: 0.03535281494259834 (0.03723972685771853) | > loss_mel_ce: 3.030503511428833 (nan) | > loss: 0.012166096828877926 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.13679175157458248) | > loader_time: 0.0046 (0.004690957846044874)  --> TIME: 2025-02-24 22:59:05 -- STEP: 3918/36938 -- GLOBAL_STEP: 521050 | > loss_text_ce: 0.02069113776087761 (0.03722972837250611) | > loss_mel_ce: 2.631608009338379 (nan) | > loss: 0.010524997487664223 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13677613944530256) | > loader_time: 0.0045 (0.004690259193023164)  --> TIME: 2025-02-24 22:59:14 -- STEP: 3968/36938 -- GLOBAL_STEP: 521100 | > loss_text_ce: 0.043244507163763046 (0.03721781006278172) | > loss_mel_ce: 3.572312593460083 (nan) | > loss: 0.014347449876368046 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13671516050254157) | > loader_time: 0.0046 (0.004690757441905233)  --> TIME: 2025-02-24 22:59:23 -- STEP: 4018/36938 -- GLOBAL_STEP: 521150 | > loss_text_ce: 0.04533548280596733 (0.03721643879642502) | > loss_mel_ce: 2.6869242191314697 (nan) | > loss: 0.010842300951480865 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.1366725042721952) | > loader_time: 0.0046 (0.004691666248961427)  --> TIME: 2025-02-24 22:59:32 -- STEP: 4068/36938 -- GLOBAL_STEP: 521200 | > loss_text_ce: 0.03283056989312172 (0.03720142686558246) | > loss_mel_ce: 0.038818359375 (nan) | > loss: 0.0002843211404979229 (nan) | > current_lr: 5e-06 | > step_time: 0.1507 (0.13669871000413708) | > loader_time: 0.0048 (0.004691603319717599)  --> TIME: 2025-02-24 22:59:41 -- STEP: 4118/36938 -- GLOBAL_STEP: 521250 | > loss_text_ce: 0.04403284937143326 (0.03718896409100437) | > loss_mel_ce: 3.0752811431884766 (nan) | > loss: 0.012378230690956116 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.1366673020149332) | > loader_time: 0.0047 (0.004691360817770748)  --> TIME: 2025-02-24 22:59:51 -- STEP: 4168/36938 -- GLOBAL_STEP: 521300 | > loss_text_ce: 0.032774727791547775 (0.037175249064121334) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13664905106266276) | > loader_time: 0.0046 (0.004691064815374798)  --> TIME: 2025-02-24 23:00:00 -- STEP: 4218/36938 -- GLOBAL_STEP: 521350 | > loss_text_ce: 0.027787433937191963 (0.037173312684877016) | > loss_mel_ce: 1.6689045429229736 (nan) | > loss: 0.00673290528357029 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13665057952622217) | > loader_time: 0.0045 (0.0046911329497314235)  --> TIME: 2025-02-24 23:00:10 -- STEP: 4268/36938 -- GLOBAL_STEP: 521400 | > loss_text_ce: 0.03701207786798477 (0.03717327890494332) | > loss_mel_ce: 2.505009651184082 (nan) | > loss: 0.010087388567626476 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.1366839155820116) | > loader_time: 0.0045 (0.004691041901274715)  --> TIME: 2025-02-24 23:00:19 -- STEP: 4318/36938 -- GLOBAL_STEP: 521450 | > loss_text_ce: 0.03177401423454285 (0.03716343943259992) | > loss_mel_ce: 2.3652727603912354 (nan) | > loss: 0.009512091055512428 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.1366917751850271) | > loader_time: 0.0047 (0.0046924114669015275)  --> TIME: 2025-02-24 23:00:29 -- STEP: 4368/36938 -- GLOBAL_STEP: 521500 | > loss_text_ce: 0.031226634979248047 (0.03716502864457046) | > loss_mel_ce: 2.510402202606201 (nan) | > loss: 0.01008582953363657 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.1367168502384052) | > loader_time: 0.0046 (0.004692209891347218)  --> TIME: 2025-02-24 23:00:38 -- STEP: 4418/36938 -- GLOBAL_STEP: 521550 | > loss_text_ce: 0.05078067258000374 (0.03717351811058944) | > loss_mel_ce: 2.894960880279541 (nan) | > loss: 0.011689451523125172 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13672782712092804) | > loader_time: 0.0045 (0.004692687282415464)  --> TIME: 2025-02-24 23:00:47 -- STEP: 4468/36938 -- GLOBAL_STEP: 521600 | > loss_text_ce: 0.03686780482530594 (0.037179686607886356) | > loss_mel_ce: 2.3130862712860107 (nan) | > loss: 0.009325215592980385 (nan) | > current_lr: 5e-06 | > step_time: 0.1791 (0.13672668371439606) | > loader_time: 0.0048 (0.004692755592861869)  --> TIME: 2025-02-24 23:00:56 -- STEP: 4518/36938 -- GLOBAL_STEP: 521650 | > loss_text_ce: 0.03185627982020378 (0.037164907279329966) | > loss_mel_ce: 2.2369813919067383 (nan) | > loss: 0.009003324434161186 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13669867001576128) | > loader_time: 0.0047 (0.004692441966481938)  --> TIME: 2025-02-24 23:01:06 -- STEP: 4568/36938 -- GLOBAL_STEP: 521700 | > loss_text_ce: 0.03664747253060341 (0.03715906207732722) | > loss_mel_ce: 3.1099002361297607 (nan) | > loss: 0.012486301362514496 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13670334741864626) | > loader_time: 0.0045 (0.004691705029399082)  --> TIME: 2025-02-24 23:01:15 -- STEP: 4618/36938 -- GLOBAL_STEP: 521750 | > loss_text_ce: 0.03838586434721947 (0.03714722878855868) | > loss_mel_ce: 3.853604793548584 (nan) | > loss: 0.015444408170878887 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.1367034144399483) | > loader_time: 0.0048 (0.0046911356303535896)  --> TIME: 2025-02-24 23:01:24 -- STEP: 4668/36938 -- GLOBAL_STEP: 521800 | > loss_text_ce: 0.035209473222494125 (0.037159922193766196) | > loss_mel_ce: 1.6603306531906128 (nan) | > loss: 0.0067283338867127895 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.1366923493305226) | > loader_time: 0.0045 (0.004691230300766027)  --> TIME: 2025-02-24 23:01:33 -- STEP: 4718/36938 -- GLOBAL_STEP: 521850 | > loss_text_ce: 0.03389497473835945 (0.037154163854159594) | > loss_mel_ce: 2.817758321762085 (nan) | > loss: 0.011316085234284401 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13666782762196733) | > loader_time: 0.0046 (0.004690661881120589)  --> TIME: 2025-02-24 23:01:43 -- STEP: 4768/36938 -- GLOBAL_STEP: 521900 | > loss_text_ce: 0.03706527501344681 (0.037149047998995904) | > loss_mel_ce: 3.4909415245056152 (nan) | > loss: 0.014000027440488338 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.13668498466878887) | > loader_time: 0.0047 (0.004691018004145395)  --> TIME: 2025-02-24 23:01:52 -- STEP: 4818/36938 -- GLOBAL_STEP: 521950 | > loss_text_ce: 0.03411516919732094 (0.03714118433645368) | > loss_mel_ce: 1.5382572412490845 (nan) | > loss: 0.00623957347124815 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13669025873237217) | > loader_time: 0.0046 (0.004690520443724316)  --> TIME: 2025-02-24 23:02:02 -- STEP: 4868/36938 -- GLOBAL_STEP: 522000 | > loss_text_ce: 0.04547962546348572 (0.037132874219207725) | > loss_mel_ce: 3.546354293823242 (nan) | > loss: 0.014253309927880764 (nan) | > current_lr: 5e-06 | > step_time: 0.179 (0.13672102966951136) | > loader_time: 0.0049 (0.004690037316343481)  --> TIME: 2025-02-24 23:02:11 -- STEP: 4918/36938 -- GLOBAL_STEP: 522050 | > loss_text_ce: 0.03712770715355873 (0.03715255598781156) | > loss_mel_ce: 2.4969592094421387 (nan) | > loss: 0.010055901482701302 (nan) | > current_lr: 5e-06 | > step_time: 0.1558 (0.13673750173274718) | > loader_time: 0.0047 (0.004690966697280216)  --> TIME: 2025-02-24 23:02:20 -- STEP: 4968/36938 -- GLOBAL_STEP: 522100 | > loss_text_ce: 0.03878910467028618 (0.03716659340999975) | > loss_mel_ce: 3.306846857070923 (nan) | > loss: 0.013276333920657635 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13673686021384002) | > loader_time: 0.0048 (0.004690911554867899)  --> TIME: 2025-02-24 23:02:30 -- STEP: 5018/36938 -- GLOBAL_STEP: 522150 | > loss_text_ce: 0.0425398088991642 (0.0371803634277749) | > loss_mel_ce: 2.4023568630218506 (nan) | > loss: 0.009701971895992756 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13679704859989364) | > loader_time: 0.0046 (0.004692124009084868)  --> TIME: 2025-02-24 23:02:39 -- STEP: 5068/36938 -- GLOBAL_STEP: 522200 | > loss_text_ce: 0.031169675290584564 (0.03718200360035971) | > loss_mel_ce: 3.128422498703003 (nan) | > loss: 0.012538065202534199 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.13678936376850126) | > loader_time: 0.0048 (0.004691721984694671)  --> TIME: 2025-02-24 23:02:49 -- STEP: 5118/36938 -- GLOBAL_STEP: 522250 | > loss_text_ce: 0.04228518530726433 (0.03717946502095839) | > loss_mel_ce: 3.375291585922241 (nan) | > loss: 0.013561813160777092 (nan) | > current_lr: 5e-06 | > step_time: 0.1577 (0.13678576276375914) | > loader_time: 0.0046 (0.004691955818706952)  --> TIME: 2025-02-24 23:02:59 -- STEP: 5168/36938 -- GLOBAL_STEP: 522300 | > loss_text_ce: 0.0320541150867939 (0.037171034564065196) | > loss_mel_ce: 2.9398462772369385 (nan) | > loss: 0.011793256737291813 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13684223749135643) | > loader_time: 0.0045 (0.004692051986422698)  --> TIME: 2025-02-24 23:03:08 -- STEP: 5218/36938 -- GLOBAL_STEP: 522350 | > loss_text_ce: 0.02495008148252964 (0.03716545975447594) | > loss_mel_ce: 2.58280086517334 (nan) | > loss: 0.010348218493163586 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1367984392854068) | > loader_time: 0.0045 (0.004691659147687323)  --> TIME: 2025-02-24 23:03:17 -- STEP: 5268/36938 -- GLOBAL_STEP: 522400 | > loss_text_ce: 0.05129627138376236 (0.03716810764256945) | > loss_mel_ce: 2.823099136352539 (nan) | > loss: 0.011406331323087215 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13676464118646123) | > loader_time: 0.0046 (0.004692646392657165)  --> TIME: 2025-02-24 23:03:26 -- STEP: 5318/36938 -- GLOBAL_STEP: 522450 | > loss_text_ce: 0.038266293704509735 (0.03716996119957632) | > loss_mel_ce: 3.398695230484009 (nan) | > loss: 0.013638736680150032 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13678279004629565) | > loader_time: 0.0047 (0.004694065055976287)  --> TIME: 2025-02-24 23:03:35 -- STEP: 5368/36938 -- GLOBAL_STEP: 522500 | > loss_text_ce: 0.038810476660728455 (0.03715940675769012) | > loss_mel_ce: 3.2651257514953613 (nan) | > loss: 0.01311085931956768 (nan) | > current_lr: 5e-06 | > step_time: 0.1526 (0.13673867758269073) | > loader_time: 0.0046 (0.004693565221195252)  --> TIME: 2025-02-24 23:03:44 -- STEP: 5418/36938 -- GLOBAL_STEP: 522550 | > loss_text_ce: 0.03293352201581001 (0.037161722430672665) | > loss_mel_ce: 3.0069515705108643 (nan) | > loss: 0.012063036672770977 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13669353747728663) | > loader_time: 0.0048 (0.0046931781994104015)  --> TIME: 2025-02-24 23:03:53 -- STEP: 5468/36938 -- GLOBAL_STEP: 522600 | > loss_text_ce: 0.039131708443164825 (0.03714884012248084) | > loss_mel_ce: 2.716416597366333 (nan) | > loss: 0.01093471609055996 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13666868842075366) | > loader_time: 0.0046 (0.004693282548755136)  --> TIME: 2025-02-24 23:04:03 -- STEP: 5518/36938 -- GLOBAL_STEP: 522650 | > loss_text_ce: 0.02826860174536705 (0.03714441965547124) | > loss_mel_ce: 2.4211044311523438 (nan) | > loss: 0.009719735011458397 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13667806688103354) | > loader_time: 0.0046 (0.004694298455230407)  --> TIME: 2025-02-24 23:04:12 -- STEP: 5568/36938 -- GLOBAL_STEP: 522700 | > loss_text_ce: 0.032882463186979294 (0.03714324846721916) | > loss_mel_ce: 3.141606330871582 (nan) | > loss: 0.012597178108990192 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.1366878359567841) | > loader_time: 0.0047 (0.004694017528117403)  --> TIME: 2025-02-24 23:04:22 -- STEP: 5618/36938 -- GLOBAL_STEP: 522750 | > loss_text_ce: 0.029112229123711586 (0.03715219696663427) | > loss_mel_ce: 3.3511149883270264 (nan) | > loss: 0.01341360155493021 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13669110592169492) | > loader_time: 0.0046 (0.004693517442189248)  --> TIME: 2025-02-24 23:04:31 -- STEP: 5668/36938 -- GLOBAL_STEP: 522800 | > loss_text_ce: 0.0434846431016922 (0.03715184983654758) | > loss_mel_ce: 3.5972440242767334 (nan) | > loss: 0.01444733701646328 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13670414782950088) | > loader_time: 0.0046 (0.004693972955056303)  --> TIME: 2025-02-24 23:04:41 -- STEP: 5718/36938 -- GLOBAL_STEP: 522850 | > loss_text_ce: 0.03148028627038002 (0.03714572194274419) | > loss_mel_ce: 2.9029018878936768 (nan) | > loss: 0.011644374579191208 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13675842075841532) | > loader_time: 0.0045 (0.004693656836660164)  --> TIME: 2025-02-24 23:04:50 -- STEP: 5768/36938 -- GLOBAL_STEP: 522900 | > loss_text_ce: 0.032166074961423874 (0.03715183153514728) | > loss_mel_ce: 2.935093402862549 (nan) | > loss: 0.011774839833378792 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13674058654602575) | > loader_time: 0.0046 (0.004694016895743931)  --> TIME: 2025-02-24 23:04:59 -- STEP: 5818/36938 -- GLOBAL_STEP: 522950 | > loss_text_ce: 0.03761940822005272 (0.03715798585976936) | > loss_mel_ce: 3.3218066692352295 (nan) | > loss: 0.013331056572496891 (nan) | > current_lr: 5e-06 | > step_time: 0.1815 (0.136739362757326) | > loader_time: 0.0049 (0.004694241844693933)  --> TIME: 2025-02-24 23:05:08 -- STEP: 5868/36938 -- GLOBAL_STEP: 523000 | > loss_text_ce: 0.040236037224531174 (0.037160492857033336) | > loss_mel_ce: 1.1473115682601929 (nan) | > loss: 0.004712490830570459 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13672392248050946) | > loader_time: 0.0047 (0.0046940684399803265)  --> TIME: 2025-02-24 23:05:17 -- STEP: 5918/36938 -- GLOBAL_STEP: 523050 | > loss_text_ce: 0.03971513733267784 (0.03716792778155132) | > loss_mel_ce: 0.1335042268037796 (nan) | > loss: 0.0006873784586787224 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.1366810313992178) | > loader_time: 0.0046 (0.004693369319125519)  --> TIME: 2025-02-24 23:05:27 -- STEP: 5968/36938 -- GLOBAL_STEP: 523100 | > loss_text_ce: 0.03881482407450676 (0.03717213228716965) | > loss_mel_ce: 2.932245969772339 (nan) | > loss: 0.011789924465119839 (nan) | > current_lr: 5e-06 | > step_time: 0.1501 (0.13672162240855487) | > loader_time: 0.0047 (0.004693057398054636)  --> TIME: 2025-02-24 23:05:36 -- STEP: 6018/36938 -- GLOBAL_STEP: 523150 | > loss_text_ce: 0.043279923498630524 (0.037165470362197535) | > loss_mel_ce: 2.7466909885406494 (nan) | > loss: 0.011071313172578812 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13672159580979945) | > loader_time: 0.0047 (0.004692946965213486)  --> TIME: 2025-02-24 23:05:46 -- STEP: 6068/36938 -- GLOBAL_STEP: 523200 | > loss_text_ce: 0.035501670092344284 (0.03716056106087808) | > loss_mel_ce: 3.078629493713379 (nan) | > loss: 0.012357664294540882 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.13671975365263406) | > loader_time: 0.0047 (0.004693287135585029)  --> TIME: 2025-02-24 23:05:55 -- STEP: 6118/36938 -- GLOBAL_STEP: 523250 | > loss_text_ce: 0.039326876401901245 (0.03715548837139129) | > loss_mel_ce: 3.1950266361236572 (nan) | > loss: 0.012834737077355385 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.1367668722844192) | > loader_time: 0.0048 (0.004693766753242563)  --> TIME: 2025-02-24 23:06:05 -- STEP: 6168/36938 -- GLOBAL_STEP: 523300 | > loss_text_ce: 0.039661578834056854 (0.03715546983632426) | > loss_mel_ce: 3.023074150085449 (nan) | > loss: 0.012153713963925838 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13676709914640364) | > loader_time: 0.0046 (0.004693829866389526)  --> TIME: 2025-02-24 23:06:14 -- STEP: 6218/36938 -- GLOBAL_STEP: 523350 | > loss_text_ce: 0.04058654606342316 (0.037149862158772876) | > loss_mel_ce: 2.3674490451812744 (nan) | > loss: 0.009555697441101074 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13675004556348497) | > loader_time: 0.0046 (0.00469327958383988)  --> TIME: 2025-02-24 23:06:23 -- STEP: 6268/36938 -- GLOBAL_STEP: 523400 | > loss_text_ce: 0.04806148260831833 (0.03714557069718696) | > loss_mel_ce: 1.4074143171310425 (nan) | > loss: 0.005775697994977236 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13676943182564838) | > loader_time: 0.0045 (0.004693359231674997)  --> TIME: 2025-02-24 23:06:33 -- STEP: 6318/36938 -- GLOBAL_STEP: 523450 | > loss_text_ce: 0.03594666346907616 (0.03714161638383352) | > loss_mel_ce: 2.990262746810913 (nan) | > loss: 0.012008767575025558 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13679971001351193) | > loader_time: 0.0049 (0.004692944932414152)  --> TIME: 2025-02-24 23:06:42 -- STEP: 6368/36938 -- GLOBAL_STEP: 523500 | > loss_text_ce: 0.03702172264456749 (0.037130916368154396) | > loss_mel_ce: 2.4917526245117188 (nan) | > loss: 0.010034819133579731 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.1368045247529615) | > loader_time: 0.0046 (0.004692324853722184)  --> TIME: 2025-02-24 23:06:52 -- STEP: 6418/36938 -- GLOBAL_STEP: 523550 | > loss_text_ce: 0.037778884172439575 (0.03712278045467226) | > loss_mel_ce: 3.444469928741455 (nan) | > loss: 0.013818448409438133 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.1368147677519341) | > loader_time: 0.0046 (0.004692435524626372)  --> TIME: 2025-02-24 23:07:01 -- STEP: 6468/36938 -- GLOBAL_STEP: 523600 | > loss_text_ce: 0.03791583329439163 (0.03712291205713704) | > loss_mel_ce: 2.9006707668304443 (nan) | > loss: 0.011661059223115444 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.1368216698230157) | > loader_time: 0.0048 (0.00469203067548348)  --> TIME: 2025-02-24 23:07:11 -- STEP: 6518/36938 -- GLOBAL_STEP: 523650 | > loss_text_ce: 0.038520123809576035 (0.037129779932829156) | > loss_mel_ce: 2.985262393951416 (nan) | > loss: 0.011999137699604034 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13686510473352542) | > loader_time: 0.0046 (0.004691819356021145)  --> TIME: 2025-02-24 23:07:20 -- STEP: 6568/36938 -- GLOBAL_STEP: 523700 | > loss_text_ce: 0.04504772648215294 (0.037126841126367306) | > loss_mel_ce: 0.82135009765625 (nan) | > loss: 0.0034380867145955563 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13688565541997977) | > loader_time: 0.0045 (0.004693144639720297)  --> TIME: 2025-02-24 23:07:30 -- STEP: 6618/36938 -- GLOBAL_STEP: 523750 | > loss_text_ce: 0.03578372672200203 (0.03712489447255582) | > loss_mel_ce: 2.9379310607910156 (nan) | > loss: 0.011800456792116165 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13690033514501768) | > loader_time: 0.0045 (0.004692867213629936)  --> TIME: 2025-02-24 23:07:39 -- STEP: 6668/36938 -- GLOBAL_STEP: 523800 | > loss_text_ce: 0.033057283610105515 (0.037127172803041605) | > loss_mel_ce: 0.010009765625 (nan) | > loss: 0.00017090099572669715 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13690641645764415) | > loader_time: 0.0046 (0.004692985758355239)  --> TIME: 2025-02-24 23:07:49 -- STEP: 6718/36938 -- GLOBAL_STEP: 523850 | > loss_text_ce: 0.045406799763441086 (0.03712867859717176) | > loss_mel_ce: 3.2687127590179443 (nan) | > loss: 0.013151268474757671 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13692719065458037) | > loader_time: 0.0046 (0.0046928349120448866)  --> TIME: 2025-02-24 23:07:58 -- STEP: 6768/36938 -- GLOBAL_STEP: 523900 | > loss_text_ce: 0.02909432351589203 (0.03712166132045985) | > loss_mel_ce: 2.87927508354187 (nan) | > loss: 0.011541148647665977 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1368935147991695) | > loader_time: 0.0048 (0.004693029162167955)  --> TIME: 2025-02-24 23:08:07 -- STEP: 6818/36938 -- GLOBAL_STEP: 523950 | > loss_text_ce: 0.03000510297715664 (0.0371071805142908) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13683944610110013) | > loader_time: 0.0045 (0.004692722045280761)  --> TIME: 2025-02-24 23:08:16 -- STEP: 6868/36938 -- GLOBAL_STEP: 524000 | > loss_text_ce: 0.03406251221895218 (0.03711463845319776) | > loss_mel_ce: 2.56093692779541 (nan) | > loss: 0.010297617875039577 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13682726832691391) | > loader_time: 0.0046 (0.004692533506372342)  --> TIME: 2025-02-24 23:08:25 -- STEP: 6918/36938 -- GLOBAL_STEP: 524050 | > loss_text_ce: 0.02440459281206131 (0.037106379306164256) | > loss_mel_ce: 3.2082595825195312 (nan) | > loss: 0.012828032486140728 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.136822010024016) | > loader_time: 0.0046 (0.004692126299198339)  --> TIME: 2025-02-24 23:08:35 -- STEP: 6968/36938 -- GLOBAL_STEP: 524100 | > loss_text_ce: 0.03830632194876671 (0.037101949656935175) | > loss_mel_ce: 1.041689157485962 (nan) | > loss: 0.004285696893930435 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13684058877137953) | > loader_time: 0.0046 (0.004691907753216291)  --> TIME: 2025-02-24 23:08:44 -- STEP: 7018/36938 -- GLOBAL_STEP: 524150 | > loss_text_ce: 0.04084734246134758 (0.037100775174644926) | > loss_mel_ce: 3.000704526901245 (nan) | > loss: 0.012069650925695896 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.1368254685068983) | > loader_time: 0.0048 (0.004691941577099315)  --> TIME: 2025-02-24 23:08:53 -- STEP: 7068/36938 -- GLOBAL_STEP: 524200 | > loss_text_ce: 0.03339998051524162 (0.03710460775225093) | > loss_mel_ce: 3.0387024879455566 (nan) | > loss: 0.012190883979201317 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.1368487271199333) | > loader_time: 0.0047 (0.00469220888837288)  --> TIME: 2025-02-24 23:09:03 -- STEP: 7118/36938 -- GLOBAL_STEP: 524250 | > loss_text_ce: 0.03933998942375183 (0.037101709295009944) | > loss_mel_ce: 3.3937721252441406 (nan) | > loss: 0.013623462058603764 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13686404066763705) | > loader_time: 0.0046 (0.00469306577755918)  --> TIME: 2025-02-24 23:09:12 -- STEP: 7168/36938 -- GLOBAL_STEP: 524300 | > loss_text_ce: 0.03713029995560646 (0.03709626703403062) | > loss_mel_ce: 3.116044282913208 (nan) | > loss: 0.012512598186731339 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13687567782056081) | > loader_time: 0.0045 (0.004693339345976725)  --> TIME: 2025-02-24 23:09:21 -- STEP: 7218/36938 -- GLOBAL_STEP: 524350 | > loss_text_ce: 0.03415898606181145 (0.03709929034407567) | > loss_mel_ce: 3.5446395874023438 (nan) | > loss: 0.014201582409441471 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.13686091799457212) | > loader_time: 0.0048 (0.00469292792364911)  --> TIME: 2025-02-24 23:09:30 -- STEP: 7268/36938 -- GLOBAL_STEP: 524400 | > loss_text_ce: 0.034640226513147354 (0.03710482498601681) | > loss_mel_ce: 5.39453125 (nan) | > loss: 0.021544333547353745 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13683934607345685) | > loader_time: 0.0046 (0.004692592460766703)  --> TIME: 2025-02-24 23:09:40 -- STEP: 7318/36938 -- GLOBAL_STEP: 524450 | > loss_text_ce: 0.040652334690093994 (0.03710858764464927) | > loss_mel_ce: 3.4113316535949707 (nan) | > loss: 0.013698349706828594 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.1368343351837973) | > loader_time: 0.0047 (0.004692398677532985)  --> TIME: 2025-02-24 23:09:49 -- STEP: 7368/36938 -- GLOBAL_STEP: 524500 | > loss_text_ce: 0.0415663868188858 (0.037105764263768784) | > loss_mel_ce: 3.7605388164520264 (nan) | > loss: 0.01508772000670433 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13680261803889998) | > loader_time: 0.0046 (0.004692454259118624)  --> TIME: 2025-02-24 23:09:58 -- STEP: 7418/36938 -- GLOBAL_STEP: 524550 | > loss_text_ce: 0.06949655711650848 (0.037107868192767304) | > loss_mel_ce: 3.755744457244873 (nan) | > loss: 0.01517952885478735 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13678364967136367) | > loader_time: 0.0044 (0.004692141853532062)  --> TIME: 2025-02-24 23:10:08 -- STEP: 7468/36938 -- GLOBAL_STEP: 524600 | > loss_text_ce: 0.031375471502542496 (0.03711262408653454) | > loss_mel_ce: 3.2206904888153076 (nan) | > loss: 0.012905023992061615 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13682572775878046) | > loader_time: 0.0046 (0.004692298113246252)  --> TIME: 2025-02-24 23:10:17 -- STEP: 7518/36938 -- GLOBAL_STEP: 524650 | > loss_text_ce: 0.03899626061320305 (0.037112709467927996) | > loss_mel_ce: 2.791837692260742 (nan) | > loss: 0.011233468540012836 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13680444381502785) | > loader_time: 0.0045 (0.004692192469609816)  --> TIME: 2025-02-24 23:10:26 -- STEP: 7568/36938 -- GLOBAL_STEP: 524700 | > loss_text_ce: 0.03250117227435112 (0.03710451424473825) | > loss_mel_ce: 2.794720411300659 (nan) | > loss: 0.01121913455426693 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1368074770898698) | > loader_time: 0.0049 (0.004692158789765774)  --> TIME: 2025-02-24 23:10:35 -- STEP: 7618/36938 -- GLOBAL_STEP: 524750 | > loss_text_ce: 0.03725171089172363 (0.03710416426009766) | > loss_mel_ce: 2.7624175548553467 (nan) | > loss: 0.011109799146652222 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13679947988290303) | > loader_time: 0.0046 (0.004692070657545341)  --> TIME: 2025-02-24 23:10:45 -- STEP: 7668/36938 -- GLOBAL_STEP: 524800 | > loss_text_ce: 0.037860751152038574 (0.03710961672594036) | > loss_mel_ce: 0.14453125 (nan) | > loss: 0.0007237778045237064 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.1367977843598021) | > loader_time: 0.0048 (0.004691894034513548)  --> TIME: 2025-02-24 23:10:54 -- STEP: 7718/36938 -- GLOBAL_STEP: 524850 | > loss_text_ce: 0.03696615621447563 (0.03711637869695939) | > loss_mel_ce: 3.217956304550171 (nan) | > loss: 0.01291635911911726 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.13679493147031296) | > loader_time: 0.0047 (0.0046917927577168885)  --> TIME: 2025-02-24 23:11:04 -- STEP: 7768/36938 -- GLOBAL_STEP: 524900 | > loss_text_ce: 0.031379539519548416 (0.03711427573844072) | > loss_mel_ce: 1.9409281015396118 (nan) | > loss: 0.007826617918908596 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13681941349769106) | > loader_time: 0.0048 (0.004691704232955459)  --> TIME: 2025-02-24 23:11:13 -- STEP: 7818/36938 -- GLOBAL_STEP: 524950 | > loss_text_ce: 0.03761019557714462 (0.03710667009842562) | > loss_mel_ce: 3.187119722366333 (nan) | > loss: 0.012796548195183277 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13684555541282725) | > loader_time: 0.0047 (0.0046918557470306665)  --> TIME: 2025-02-24 23:11:23 -- STEP: 7868/36938 -- GLOBAL_STEP: 525000 | > loss_text_ce: 0.03774925693869591 (0.037103429073647914) | > loss_mel_ce: 3.7339093685150146 (nan) | > loss: 0.014966900460422039 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.136852946266888) | > loader_time: 0.0052 (0.004691919882890243)  --> TIME: 2025-02-24 23:11:32 -- STEP: 7918/36938 -- GLOBAL_STEP: 525050 | > loss_text_ce: 0.039482470601797104 (0.037096825911376384) | > loss_mel_ce: 3.146738290786743 (nan) | > loss: 0.012643733061850071 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.13685843913836382) | > loader_time: 0.0046 (0.00469175846054568)  --> TIME: 2025-02-24 23:11:41 -- STEP: 7968/36938 -- GLOBAL_STEP: 525100 | > loss_text_ce: 0.049012959003448486 (0.03709853629113646) | > loss_mel_ce: 2.7961032390594482 (nan) | > loss: 0.011290144175291061 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13683313436536904) | > loader_time: 0.0048 (0.00469132623519286)  --> TIME: 2025-02-24 23:11:50 -- STEP: 8018/36938 -- GLOBAL_STEP: 525150 | > loss_text_ce: 0.025838349014520645 (0.03709540752755354) | > loss_mel_ce: 3.201707601547241 (nan) | > loss: 0.012807723134756088 (nan) | > current_lr: 5e-06 | > step_time: 0.1706 (0.13684626662187013) | > loader_time: 0.0047 (0.004691177634831302)  --> TIME: 2025-02-24 23:12:00 -- STEP: 8068/36938 -- GLOBAL_STEP: 525200 | > loss_text_ce: 0.039488088339567184 (0.037090405579734996) | > loss_mel_ce: 2.257685661315918 (nan) | > loss: 0.009115769527852535 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13685204152961622) | > loader_time: 0.0046 (0.004691127213027506)  --> TIME: 2025-02-24 23:12:09 -- STEP: 8118/36938 -- GLOBAL_STEP: 525250 | > loss_text_ce: 0.03799332678318024 (0.03708908395051519) | > loss_mel_ce: 3.4696733951568604 (nan) | > loss: 0.013919313438236713 (nan) | > current_lr: 5e-06 | > step_time: 0.1763 (0.13686388265502133) | > loader_time: 0.005 (0.0046911209961204535)  --> TIME: 2025-02-24 23:12:19 -- STEP: 8168/36938 -- GLOBAL_STEP: 525300 | > loss_text_ce: 0.035389285534620285 (0.0370889071850144) | > loss_mel_ce: 2.7601912021636963 (nan) | > loss: 0.01109357364475727 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13686239337010436) | > loader_time: 0.0046 (0.004690965639389006)  --> TIME: 2025-02-24 23:12:28 -- STEP: 8218/36938 -- GLOBAL_STEP: 525350 | > loss_text_ce: 0.03201139718294144 (0.03708752158661626) | > loss_mel_ce: 3.1002144813537598 (nan) | > loss: 0.012429469265043736 (nan) | > current_lr: 5e-06 | > step_time: 0.1785 (0.13686230688032427) | > loader_time: 0.0049 (0.004690725950174723)  --> TIME: 2025-02-24 23:12:37 -- STEP: 8268/36938 -- GLOBAL_STEP: 525400 | > loss_text_ce: 0.0378824844956398 (0.03707684582965773) | > loss_mel_ce: 2.349090576171875 (nan) | > loss: 0.009472115896642208 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.1368456665997812) | > loader_time: 0.0046 (0.004690381369846839)  --> TIME: 2025-02-24 23:12:46 -- STEP: 8318/36938 -- GLOBAL_STEP: 525450 | > loss_text_ce: 0.04637116938829422 (0.03707156088245752) | > loss_mel_ce: 3.1938679218292236 (nan) | > loss: 0.012858092784881592 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13680878256513224) | > loader_time: 0.0046 (0.004689968414792779)  --> TIME: 2025-02-24 23:12:55 -- STEP: 8368/36938 -- GLOBAL_STEP: 525500 | > loss_text_ce: 0.03457034006714821 (0.03706609381749673) | > loss_mel_ce: 3.1654744148254395 (nan) | > loss: 0.012698591686785221 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13680613918468557) | > loader_time: 0.0054 (0.004690266817979115)  --> TIME: 2025-02-24 23:13:05 -- STEP: 8418/36938 -- GLOBAL_STEP: 525550 | > loss_text_ce: 0.031392794102430344 (0.03707211771046186) | > loss_mel_ce: 2.6795639991760254 (nan) | > loss: 0.010757765732705593 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13680443185388008) | > loader_time: 0.0045 (0.0046903098895561815)  --> TIME: 2025-02-24 23:13:14 -- STEP: 8468/36938 -- GLOBAL_STEP: 525600 | > loss_text_ce: 0.0396578311920166 (0.037077833419836524) | > loss_mel_ce: 3.173243284225464 (nan) | > loss: 0.012749608606100082 (nan) | > current_lr: 5e-06 | > step_time: 0.1876 (0.13680943054132547) | > loader_time: 0.0048 (0.004690162855102382)  --> TIME: 2025-02-24 23:13:23 -- STEP: 8518/36938 -- GLOBAL_STEP: 525650 | > loss_text_ce: 0.03926974907517433 (0.03707432812915127) | > loss_mel_ce: 3.38433575630188 (nan) | > loss: 0.013585736975073814 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13680029564772614) | > loader_time: 0.0046 (0.004689947907760647)  --> TIME: 2025-02-24 23:13:32 -- STEP: 8568/36938 -- GLOBAL_STEP: 525700 | > loss_text_ce: 0.033412009477615356 (0.03707875500497943) | > loss_mel_ce: 3.2500057220458984 (nan) | > loss: 0.013029435649514198 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.13678785401677326) | > loader_time: 0.0053 (0.004690012845004656)  --> TIME: 2025-02-24 23:13:42 -- STEP: 8618/36938 -- GLOBAL_STEP: 525750 | > loss_text_ce: 0.056689511984586716 (0.03707886954063177) | > loss_mel_ce: 2.588649272918701 (nan) | > loss: 0.010497376322746277 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13677945831973412) | > loader_time: 0.0046 (0.004690193914155409)  --> TIME: 2025-02-24 23:13:51 -- STEP: 8668/36938 -- GLOBAL_STEP: 525800 | > loss_text_ce: 0.03616591915488243 (0.03706827540847559) | > loss_mel_ce: 2.4789347648620605 (nan) | > loss: 0.009980559349060059 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.13677100580740914) | > loader_time: 0.0047 (0.004690529153706802)  --> TIME: 2025-02-24 23:14:00 -- STEP: 8718/36938 -- GLOBAL_STEP: 525850 | > loss_text_ce: 0.04440375044941902 (0.03707727193642254) | > loss_mel_ce: 2.801342725753784 (nan) | > loss: 0.011292645707726479 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.1367801821923742) | > loader_time: 0.0046 (0.004690614335168868)  --> TIME: 2025-02-24 23:14:10 -- STEP: 8768/36938 -- GLOBAL_STEP: 525900 | > loss_text_ce: 0.04274360463023186 (0.0370729288812192) | > loss_mel_ce: 3.147282838821411 (nan) | > loss: 0.01265883632004261 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.1367782370669998) | > loader_time: 0.0046 (0.004690399869297555)  --> TIME: 2025-02-24 23:14:19 -- STEP: 8818/36938 -- GLOBAL_STEP: 525950 | > loss_text_ce: 0.03876882791519165 (0.037079189312267315) | > loss_mel_ce: 3.013662576675415 (nan) | > loss: 0.012112823314964771 (nan) | > current_lr: 5e-06 | > step_time: 0.1934 (0.13676098129002506) | > loader_time: 0.0049 (0.004690037803277876)  --> TIME: 2025-02-24 23:14:29 -- STEP: 8868/36938 -- GLOBAL_STEP: 526000 | > loss_text_ce: 0.03264821320772171 (0.0370836861369217) | > loss_mel_ce: 3.0865509510040283 (nan) | > loss: 0.012377775274217129 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.13679173539014974) | > loader_time: 0.0046 (0.004689885196677198)  --> TIME: 2025-02-24 23:14:38 -- STEP: 8918/36938 -- GLOBAL_STEP: 526050 | > loss_text_ce: 0.03106360323727131 (0.03707713166558397) | > loss_mel_ce: 2.9826340675354004 (nan) | > loss: 0.011959117837250233 (nan) | > current_lr: 5e-06 | > step_time: 0.1811 (0.13679264508226183) | > loader_time: 0.0049 (0.004690208117452335)  --> TIME: 2025-02-24 23:14:47 -- STEP: 8968/36938 -- GLOBAL_STEP: 526100 | > loss_text_ce: 0.03699561208486557 (0.03708027993226273) | > loss_mel_ce: 3.3499605655670166 (nan) | > loss: 0.01344030350446701 (nan) | > current_lr: 5e-06 | > step_time: 0.1738 (0.1367895451908553) | > loader_time: 0.0048 (0.004690076494514691)  --> TIME: 2025-02-24 23:14:56 -- STEP: 9018/36938 -- GLOBAL_STEP: 526150 | > loss_text_ce: 0.03740743547677994 (0.03709010962455939) | > loss_mel_ce: 2.35369873046875 (nan) | > loss: 0.009488516487181187 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13677910614711625) | > loader_time: 0.0048 (0.0046897940478110335)  --> TIME: 2025-02-24 23:15:06 -- STEP: 9068/36938 -- GLOBAL_STEP: 526200 | > loss_text_ce: 0.030472099781036377 (0.037089745057574314) | > loss_mel_ce: 2.6150028705596924 (nan) | > loss: 0.010497916489839554 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1367709680596373) | > loader_time: 0.0046 (0.004689710488338338)  --> TIME: 2025-02-24 23:15:15 -- STEP: 9118/36938 -- GLOBAL_STEP: 526250 | > loss_text_ce: 0.04804181307554245 (0.03709925607267231) | > loss_mel_ce: 2.68115234375 (nan) | > loss: 0.010830136016011238 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13678055733644953) | > loader_time: 0.0045 (0.004689903786423501)  --> TIME: 2025-02-24 23:15:24 -- STEP: 9168/36938 -- GLOBAL_STEP: 526300 | > loss_text_ce: 0.02661609835922718 (0.03708940201748195) | > loss_mel_ce: 0.01983773708343506 (nan) | > loss: 0.00018434062076266855 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13675123089063004) | > loader_time: 0.0046 (0.004689434123913041)  --> TIME: 2025-02-24 23:15:34 -- STEP: 9218/36938 -- GLOBAL_STEP: 526350 | > loss_text_ce: 0.0340004488825798 (0.03709174083532882) | > loss_mel_ce: 3.602320671081543 (nan) | > loss: 0.014429846778512001 (nan) | > current_lr: 5e-06 | > step_time: 0.1484 (0.1367587472642933) | > loader_time: 0.0046 (0.004689355221403021)  --> TIME: 2025-02-24 23:15:43 -- STEP: 9268/36938 -- GLOBAL_STEP: 526400 | > loss_text_ce: 0.03637608140707016 (0.03708980233974466) | > loss_mel_ce: 1.9144338369369507 (nan) | > loss: 0.007741309702396393 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.1367243111622347) | > loader_time: 0.0045 (0.004689291087419358)  --> TIME: 2025-02-24 23:15:52 -- STEP: 9318/36938 -- GLOBAL_STEP: 526450 | > loss_text_ce: 0.04806474968791008 (0.03709550382830107) | > loss_mel_ce: 3.209057092666626 (nan) | > loss: 0.012925087474286556 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13671073035432368) | > loader_time: 0.0045 (0.0046889066286776955)  --> TIME: 2025-02-24 23:16:01 -- STEP: 9368/36938 -- GLOBAL_STEP: 526500 | > loss_text_ce: 0.03521862253546715 (0.03710191252287071) | > loss_mel_ce: 2.7317354679107666 (nan) | > loss: 0.010979977436363697 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.13670912672575453) | > loader_time: 0.0048 (0.004689037112676628)  --> TIME: 2025-02-24 23:16:10 -- STEP: 9418/36938 -- GLOBAL_STEP: 526550 | > loss_text_ce: 0.03708663955330849 (0.03710066509140587) | > loss_mel_ce: 3.1944286823272705 (nan) | > loss: 0.01282347459346056 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13671429444837763) | > loader_time: 0.0045 (0.004689121276716141)  --> TIME: 2025-02-24 23:16:20 -- STEP: 9468/36938 -- GLOBAL_STEP: 526600 | > loss_text_ce: 0.03128715604543686 (0.03709853047347248) | > loss_mel_ce: 2.5107438564300537 (nan) | > loss: 0.010087425820529461 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13674762894143305) | > loader_time: 0.0048 (0.00468907056098555)  --> TIME: 2025-02-24 23:16:29 -- STEP: 9518/36938 -- GLOBAL_STEP: 526650 | > loss_text_ce: 0.03152383863925934 (0.03710422660487357) | > loss_mel_ce: 3.2114555835723877 (nan) | > loss: 0.012868966907262802 (nan) | > current_lr: 5e-06 | > step_time: 0.1583 (0.1367380642094363) | > loader_time: 0.0047 (0.0046896264692844825)  --> TIME: 2025-02-24 23:16:39 -- STEP: 9568/36938 -- GLOBAL_STEP: 526700 | > loss_text_ce: 0.0405498631298542 (0.03711317276633133) | > loss_mel_ce: 2.8578710556030273 (nan) | > loss: 0.011501670815050602 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.13674736802892917) | > loader_time: 0.0047 (0.004689543741602566)  --> TIME: 2025-02-24 23:16:48 -- STEP: 9618/36938 -- GLOBAL_STEP: 526750 | > loss_text_ce: 0.02699253521859646 (0.03710977588575918) | > loss_mel_ce: 2.6074016094207764 (nan) | > loss: 0.010453945957124233 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13672639654837343) | > loader_time: 0.0045 (0.004689330121637462)  --> TIME: 2025-02-24 23:16:57 -- STEP: 9668/36938 -- GLOBAL_STEP: 526800 | > loss_text_ce: 0.047014832496643066 (0.03710812112413984) | > loss_mel_ce: 0.03598785400390625 (nan) | > loss: 0.00032937576179392636 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13673776242670835) | > loader_time: 0.0046 (0.004689535746458006)  --> TIME: 2025-02-24 23:17:06 -- STEP: 9718/36938 -- GLOBAL_STEP: 526850 | > loss_text_ce: 0.040150150656700134 (0.03710676983267579) | > loss_mel_ce: 3.3352243900299072 (nan) | > loss: 0.013394344598054886 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13672626926271364) | > loader_time: 0.0047 (0.004689416391757761)  --> TIME: 2025-02-24 23:17:16 -- STEP: 9768/36938 -- GLOBAL_STEP: 526900 | > loss_text_ce: 0.04010448232293129 (0.03710291916563626) | > loss_mel_ce: 3.563642740249634 (nan) | > loss: 0.01430058479309082 (nan) | > current_lr: 5e-06 | > step_time: 0.1894 (0.13673597042136834) | > loader_time: 0.0049 (0.004689294963855414)  --> TIME: 2025-02-24 23:17:25 -- STEP: 9818/36938 -- GLOBAL_STEP: 526950 | > loss_text_ce: 0.03852194547653198 (0.037108693559392135) | > loss_mel_ce: 2.543090581893921 (nan) | > loss: 0.01024449523538351 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13674710979042023) | > loader_time: 0.0046 (0.0046892124369518895)  --> TIME: 2025-02-24 23:17:35 -- STEP: 9868/36938 -- GLOBAL_STEP: 527000 | > loss_text_ce: 0.029640857130289078 (0.03710986727466994) | > loss_mel_ce: 2.741408109664917 (nan) | > loss: 0.010996227152645588 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13674349042148667) | > loader_time: 0.0046 (0.004689159666810579)  --> TIME: 2025-02-24 23:17:44 -- STEP: 9918/36938 -- GLOBAL_STEP: 527050 | > loss_text_ce: 0.027495959773659706 (0.037106039699198644) | > loss_mel_ce: 2.9783804416656494 (nan) | > loss: 0.011928081512451172 (nan) | > current_lr: 5e-06 | > step_time: 0.1121 (0.1367451507830285) | > loader_time: 0.0047 (0.004689174473441932)  --> TIME: 2025-02-24 23:17:53 -- STEP: 9968/36938 -- GLOBAL_STEP: 527100 | > loss_text_ce: 0.03937878832221031 (0.03710087878473734) | > loss_mel_ce: 3.635892868041992 (nan) | > loss: 0.014584412798285484 (nan) | > current_lr: 5e-06 | > step_time: 0.1862 (0.1367447386775312) | > loader_time: 0.0049 (0.0046890188086281615)  --> TIME: 2025-02-24 23:18:03 -- STEP: 10018/36938 -- GLOBAL_STEP: 527150 | > loss_text_ce: 0.042399898171424866 (0.03710463672646187) | > loss_mel_ce: 2.786954641342163 (nan) | > loss: 0.011227598413825035 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13673620929401936) | > loader_time: 0.0045 (0.004688898849258836)  --> TIME: 2025-02-24 23:18:12 -- STEP: 10068/36938 -- GLOBAL_STEP: 527200 | > loss_text_ce: 0.02363588474690914 (0.037096809123779384) | > loss_mel_ce: 3.107590675354004 (nan) | > loss: 0.012425502762198448 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13675083865899598) | > loader_time: 0.0045 (0.0046889180932482635)  --> TIME: 2025-02-24 23:18:22 -- STEP: 10118/36938 -- GLOBAL_STEP: 527250 | > loss_text_ce: 0.035623643547296524 (0.03710041475885641) | > loss_mel_ce: 3.1569793224334717 (nan) | > loss: 0.012669059447944164 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.13677338626361613) | > loader_time: 0.0047 (0.004688939126401725)  --> TIME: 2025-02-24 23:18:31 -- STEP: 10168/36938 -- GLOBAL_STEP: 527300 | > loss_text_ce: 0.019640333950519562 (0.037093322716501745) | > loss_mel_ce: 2.3444278240203857 (nan) | > loss: 0.00938122346997261 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13676810123899782) | > loader_time: 0.0046 (0.004688964782972959)  --> TIME: 2025-02-24 23:18:40 -- STEP: 10218/36938 -- GLOBAL_STEP: 527350 | > loss_text_ce: 0.04274634271860123 (0.03709590010828792) | > loss_mel_ce: 3.1345856189727783 (nan) | > loss: 0.012608461081981659 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13677979712617852) | > loader_time: 0.0045 (0.004689154897470675)  --> TIME: 2025-02-24 23:18:49 -- STEP: 10268/36938 -- GLOBAL_STEP: 527400 | > loss_text_ce: 0.035918090492486954 (0.037088283866714394) | > loss_mel_ce: 2.6774306297302246 (nan) | > loss: 0.0107672568410635 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.1367588258922726) | > loader_time: 0.0046 (0.004688909952864373)  --> TIME: 2025-02-24 23:18:59 -- STEP: 10318/36938 -- GLOBAL_STEP: 527450 | > loss_text_ce: 0.042063936591148376 (0.0370889387550572) | > loss_mel_ce: 2.7781832218170166 (nan) | > loss: 0.011191457509994507 (nan) | > current_lr: 5e-06 | > step_time: 0.1848 (0.1367819225818156) | > loader_time: 0.0048 (0.004688832204825754)  --> TIME: 2025-02-24 23:19:09 -- STEP: 10368/36938 -- GLOBAL_STEP: 527500 | > loss_text_ce: 0.040866825729608536 (0.037087052035879316) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.1368172988755468) | > loader_time: 0.0048 (0.004689922924211004)  --> TIME: 2025-02-24 23:19:18 -- STEP: 10418/36938 -- GLOBAL_STEP: 527550 | > loss_text_ce: 0.039507996290922165 (0.03708371286532383) | > loss_mel_ce: 2.5466971397399902 (nan) | > loss: 0.010262719355523586 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13679601904022703) | > loader_time: 0.0045 (0.004689760550493295)  --> TIME: 2025-02-24 23:19:27 -- STEP: 10468/36938 -- GLOBAL_STEP: 527600 | > loss_text_ce: 0.03504347428679466 (0.03707620179069618) | > loss_mel_ce: 3.262570381164551 (nan) | > loss: 0.013085770420730114 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.1368167002692131) | > loader_time: 0.0047 (0.004689648468438851)  --> TIME: 2025-02-24 23:19:37 -- STEP: 10518/36938 -- GLOBAL_STEP: 527650 | > loss_text_ce: 0.03239995986223221 (0.037074318642381573) | > loss_mel_ce: 3.17156720161438 (nan) | > loss: 0.012714155949652195 (nan) | > current_lr: 5e-06 | > step_time: 0.1582 (0.13682635788410694) | > loader_time: 0.0047 (0.004689665796367328)  --> TIME: 2025-02-24 23:19:46 -- STEP: 10568/36938 -- GLOBAL_STEP: 527700 | > loss_text_ce: 0.041678451001644135 (0.037073958299671354) | > loss_mel_ce: 3.5772316455841064 (nan) | > loss: 0.014360754750669003 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13681864896387347) | > loader_time: 0.0048 (0.004689849208106983)  --> TIME: 2025-02-24 23:19:55 -- STEP: 10618/36938 -- GLOBAL_STEP: 527750 | > loss_text_ce: 0.04252491891384125 (0.03706433823880224) | > loss_mel_ce: 3.1709792613983154 (nan) | > loss: 0.01275200117379427 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.13679594231496145) | > loader_time: 0.0048 (0.004689607024754177)  --> TIME: 2025-02-24 23:20:04 -- STEP: 10668/36938 -- GLOBAL_STEP: 527800 | > loss_text_ce: 0.038306497037410736 (0.0370616513323794) | > loss_mel_ce: 2.058366537094116 (nan) | > loss: 0.008320131339132786 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13678748343023978) | > loader_time: 0.0045 (0.004689610715121613)  --> TIME: 2025-02-24 23:20:14 -- STEP: 10718/36938 -- GLOBAL_STEP: 527850 | > loss_text_ce: 0.026408420875668526 (0.037060460254475694) | > loss_mel_ce: 3.133641242980957 (nan) | > loss: 0.012539880350232124 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13679637046286525) | > loader_time: 0.0045 (0.004689522411512303)  --> TIME: 2025-02-24 23:20:23 -- STEP: 10768/36938 -- GLOBAL_STEP: 527900 | > loss_text_ce: 0.04194968566298485 (0.03706318503970504) | > loss_mel_ce: 3.098388195037842 (nan) | > loss: 0.012461659498512745 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13678236675475022) | > loader_time: 0.0054 (0.004689798866126024)  --> TIME: 2025-02-24 23:20:32 -- STEP: 10818/36938 -- GLOBAL_STEP: 527950 | > loss_text_ce: 0.04312270134687424 (0.03705575902806665) | > loss_mel_ce: 2.733389377593994 (nan) | > loss: 0.011017905548214912 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13677018760731235) | > loader_time: 0.0049 (0.004690173571906273)  --> TIME: 2025-02-24 23:20:42 -- STEP: 10868/36938 -- GLOBAL_STEP: 528000 | > loss_text_ce: 0.04165368527173996 (0.037052632159789045) | > loss_mel_ce: 2.6264758110046387 (nan) | > loss: 0.010587816126644611 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13678085455176683) | > loader_time: 0.0046 (0.004690523177418251)  --> TIME: 2025-02-24 23:20:51 -- STEP: 10918/36938 -- GLOBAL_STEP: 528050 | > loss_text_ce: 0.029804861173033714 (0.037049086987433255) | > loss_mel_ce: 2.890019655227661 (nan) | > loss: 0.011586606502532959 (nan) | > current_lr: 5e-06 | > step_time: 0.1622 (0.13677785356832226) | > loader_time: 0.0046 (0.004690568882546872)  --> TIME: 2025-02-24 23:21:00 -- STEP: 10968/36938 -- GLOBAL_STEP: 528100 | > loss_text_ce: 0.030730148777365685 (0.03704743197100844) | > loss_mel_ce: 2.053398609161377 (nan) | > loss: 0.008270353078842163 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13676366523006023) | > loader_time: 0.0046 (0.0046904237925962985)  --> TIME: 2025-02-24 23:21:09 -- STEP: 11018/36938 -- GLOBAL_STEP: 528150 | > loss_text_ce: 0.029860204085707664 (0.03704773941269859) | > loss_mel_ce: 2.027493953704834 (nan) | > loss: 0.00816410407423973 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.1367622887856689) | > loader_time: 0.0084 (0.004690988242550472)  --> TIME: 2025-02-24 23:21:18 -- STEP: 11068/36938 -- GLOBAL_STEP: 528200 | > loss_text_ce: 0.048556163907051086 (0.037045370556655265) | > loss_mel_ce: 2.7914814949035645 (nan) | > loss: 0.011269991286098957 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13674633238864448) | > loader_time: 0.0047 (0.004691162672032521)  --> TIME: 2025-02-24 23:21:28 -- STEP: 11118/36938 -- GLOBAL_STEP: 528250 | > loss_text_ce: 0.03247734159231186 (0.037044991174671735) | > loss_mel_ce: 3.103652000427246 (nan) | > loss: 0.01244495902210474 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13675707556142447) | > loader_time: 0.0046 (0.004691234229386864)  --> TIME: 2025-02-24 23:21:37 -- STEP: 11168/36938 -- GLOBAL_STEP: 528300 | > loss_text_ce: 0.04585159569978714 (0.03703896778623689) | > loss_mel_ce: 3.7181055545806885 (nan) | > loss: 0.014936339110136032 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.1367689383277576) | > loader_time: 0.0046 (0.004690918228175388)  --> TIME: 2025-02-24 23:21:47 -- STEP: 11218/36938 -- GLOBAL_STEP: 528350 | > loss_text_ce: 0.044790118932724 (0.037039668751054806) | > loss_mel_ce: 3.3831238746643066 (nan) | > loss: 0.013602834194898605 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.1367935103557392) | > loader_time: 0.0046 (0.004691091317462302)  --> TIME: 2025-02-24 23:21:56 -- STEP: 11268/36938 -- GLOBAL_STEP: 528400 | > loss_text_ce: 0.03972693905234337 (0.03703615536951644) | > loss_mel_ce: 2.8972129821777344 (nan) | > loss: 0.011654524132609367 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13677670515975818) | > loader_time: 0.0048 (0.0046911618580071784)  --> TIME: 2025-02-24 23:22:06 -- STEP: 11318/36938 -- GLOBAL_STEP: 528450 | > loss_text_ce: 0.02851984277367592 (0.03703709772831948) | > loss_mel_ce: 3.1084787845611572 (nan) | > loss: 0.012448407709598541 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13679679770586764) | > loader_time: 0.0047 (0.0046913804340581716)  --> TIME: 2025-02-24 23:22:15 -- STEP: 11368/36938 -- GLOBAL_STEP: 528500 | > loss_text_ce: 0.057703327387571335 (0.03703865092683585) | > loss_mel_ce: 1.9434970617294312 (nan) | > loss: 0.007941272109746933 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13676973327962855) | > loader_time: 0.0046 (0.004691249987845862)  --> TIME: 2025-02-24 23:22:24 -- STEP: 11418/36938 -- GLOBAL_STEP: 528550 | > loss_text_ce: 0.0424773171544075 (0.03704028711066177) | > loss_mel_ce: 1.9821350574493408 (nan) | > loss: 0.008034177124500275 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13675666953745155) | > loader_time: 0.0046 (0.004691191992506634)  --> TIME: 2025-02-24 23:22:33 -- STEP: 11468/36938 -- GLOBAL_STEP: 528600 | > loss_text_ce: 0.031649816781282425 (0.03703743176967485) | > loss_mel_ce: 2.7663445472717285 (nan) | > loss: 0.011103153228759766 (nan) | > current_lr: 5e-06 | > step_time: 0.1084 (0.13676946931156395) | > loader_time: 0.0046 (0.004691375187560169)  --> TIME: 2025-02-24 23:22:43 -- STEP: 11518/36938 -- GLOBAL_STEP: 528650 | > loss_text_ce: 0.03093143366277218 (0.0370396105407633) | > loss_mel_ce: 2.1159934997558594 (nan) | > loss: 0.008519544266164303 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.1367709402782176) | > loader_time: 0.0047 (0.004691430544931876)  --> TIME: 2025-02-24 23:22:52 -- STEP: 11568/36938 -- GLOBAL_STEP: 528700 | > loss_text_ce: 0.03784749284386635 (0.03703819162580742) | > loss_mel_ce: 3.2244861125946045 (nan) | > loss: 0.012945769354701042 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13676502272218757) | > loader_time: 0.0045 (0.004691289420939093)  --> TIME: 2025-02-24 23:23:01 -- STEP: 11618/36938 -- GLOBAL_STEP: 528750 | > loss_text_ce: 0.032809074968099594 (0.03704005773686678) | > loss_mel_ce: 2.298062324523926 (nan) | > loss: 0.0092494897544384 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13675216529348427) | > loader_time: 0.0046 (0.004691315879303)  --> TIME: 2025-02-24 23:23:10 -- STEP: 11668/36938 -- GLOBAL_STEP: 528800 | > loss_text_ce: 0.035794783383607864 (0.03704171003403467) | > loss_mel_ce: 3.393908977508545 (nan) | > loss: 0.013609936460852623 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.1367607907272996) | > loader_time: 0.0046 (0.004691247299267574)  --> TIME: 2025-02-24 23:23:20 -- STEP: 11718/36938 -- GLOBAL_STEP: 528850 | > loss_text_ce: 0.04319440573453903 (0.0370415700205342) | > loss_mel_ce: 3.7210421562194824 (nan) | > loss: 0.014937447383999825 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.13677767163800966) | > loader_time: 0.005 (0.0046912130997803165)  --> TIME: 2025-02-24 23:23:29 -- STEP: 11768/36938 -- GLOBAL_STEP: 528900 | > loss_text_ce: 0.03026490844786167 (0.03704229710458392) | > loss_mel_ce: 3.263573169708252 (nan) | > loss: 0.013070786371827126 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.1367767067039848) | > loader_time: 0.0047 (0.004691059069921986)  --> TIME: 2025-02-24 23:23:38 -- STEP: 11818/36938 -- GLOBAL_STEP: 528950 | > loss_text_ce: 0.03568955510854721 (0.03704088642174464) | > loss_mel_ce: 3.446899652481079 (nan) | > loss: 0.0138197997584939 (nan) | > current_lr: 5e-06 | > step_time: 0.1882 (0.13676357291478725) | > loader_time: 0.0049 (0.004690867407226307)  --> TIME: 2025-02-24 23:23:47 -- STEP: 11868/36938 -- GLOBAL_STEP: 529000 | > loss_text_ce: 0.03257378563284874 (0.03704441434248324) | > loss_mel_ce: 2.3509278297424316 (nan) | > loss: 0.009458339773118496 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13673289084538973) | > loader_time: 0.0047 (0.004691336787345638)  --> TIME: 2025-02-24 23:23:57 -- STEP: 11918/36938 -- GLOBAL_STEP: 529050 | > loss_text_ce: 0.04609401896595955 (0.0370398228234743) | > loss_mel_ce: 0.0662027969956398 (nan) | > loss: 0.00044562233961187303 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13673228980671104) | > loader_time: 0.0045 (0.0046915695118732075)  --> TIME: 2025-02-24 23:24:06 -- STEP: 11968/36938 -- GLOBAL_STEP: 529100 | > loss_text_ce: 0.050337184220552444 (0.03704229540818506) | > loss_mel_ce: 2.696256637573242 (nan) | > loss: 0.01089918240904808 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.1367295535610003) | > loader_time: 0.0047 (0.004691844098867578)  --> TIME: 2025-02-24 23:24:15 -- STEP: 12018/36938 -- GLOBAL_STEP: 529150 | > loss_text_ce: 0.033259160816669464 (0.03704274495755798) | > loss_mel_ce: 2.66910457611084 (nan) | > loss: 0.010723666287958622 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.1367311983740178) | > loader_time: 0.0045 (0.004691648868142433)  --> TIME: 2025-02-24 23:24:25 -- STEP: 12068/36938 -- GLOBAL_STEP: 529200 | > loss_text_ce: 0.052184876054525375 (0.03704145670685417) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.13673277115798144) | > loader_time: 0.0046 (0.004691589755809654)  --> TIME: 2025-02-24 23:24:34 -- STEP: 12118/36938 -- GLOBAL_STEP: 529250 | > loss_text_ce: 0.039283283054828644 (0.03704320788131415) | > loss_mel_ce: 2.8176016807556152 (nan) | > loss: 0.011336845345795155 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13673480884295763) | > loader_time: 0.0046 (0.004691595664451682)  --> TIME: 2025-02-24 23:24:44 -- STEP: 12168/36938 -- GLOBAL_STEP: 529300 | > loss_text_ce: 0.03537444397807121 (0.03704170042062331) | > loss_mel_ce: 3.2847647666931152 (nan) | > loss: 0.013175155967473984 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13674508988426207) | > loader_time: 0.0048 (0.004691551736434141)  --> TIME: 2025-02-24 23:24:53 -- STEP: 12218/36938 -- GLOBAL_STEP: 529350 | > loss_text_ce: 0.03570270165801048 (0.03704065490418923) | > loss_mel_ce: 0.07361755520105362 (nan) | > loss: 0.00043381055002100766 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13676704529518585) | > loader_time: 0.0046 (0.004691574104797392)  --> TIME: 2025-02-24 23:25:03 -- STEP: 12268/36938 -- GLOBAL_STEP: 529400 | > loss_text_ce: 0.03298557549715042 (0.03704217601844457) | > loss_mel_ce: 2.994971752166748 (nan) | > loss: 0.01201570499688387 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13678068161632637) | > loader_time: 0.0046 (0.0046916730364254386)  --> TIME: 2025-02-24 23:25:12 -- STEP: 12318/36938 -- GLOBAL_STEP: 529450 | > loss_text_ce: 0.036513373255729675 (0.03704329149398063) | > loss_mel_ce: 2.27805233001709 (nan) | > loss: 0.009184785187244415 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13679388218217128) | > loader_time: 0.0047 (0.00469200422206779)  --> TIME: 2025-02-24 23:25:22 -- STEP: 12368/36938 -- GLOBAL_STEP: 529500 | > loss_text_ce: 0.03790229931473732 (0.03704545135672466) | > loss_mel_ce: 3.179589033126831 (nan) | > loss: 0.012767823413014412 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.13680864330790793) | > loader_time: 0.0047 (0.0046920721813130395)  --> TIME: 2025-02-24 23:25:31 -- STEP: 12418/36938 -- GLOBAL_STEP: 529550 | > loss_text_ce: 0.03276630491018295 (0.037045692957795) | > loss_mel_ce: 3.6183295249938965 (nan) | > loss: 0.01448847632855177 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13681301123701553) | > loader_time: 0.0046 (0.004692270437441202)  --> TIME: 2025-02-24 23:25:41 -- STEP: 12468/36938 -- GLOBAL_STEP: 529600 | > loss_text_ce: 0.030549315735697746 (0.03704525839120876) | > loss_mel_ce: 3.625 (nan) | > loss: 0.014506149105727673 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13680468132987697) | > loader_time: 0.0046 (0.004692229430645685)  --> TIME: 2025-02-24 23:25:50 -- STEP: 12518/36938 -- GLOBAL_STEP: 529650 | > loss_text_ce: 0.03606259822845459 (0.037048230388572766) | > loss_mel_ce: 1.4097799062728882 (nan) | > loss: 0.0057374704629182816 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.1368180257593371) | > loader_time: 0.0049 (0.004692318569377779)  --> TIME: 2025-02-24 23:25:59 -- STEP: 12568/36938 -- GLOBAL_STEP: 529700 | > loss_text_ce: 0.02914401888847351 (0.037050076583542715) | > loss_mel_ce: 1.9830131530761719 (nan) | > loss: 0.007984750904142857 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13678428883540397) | > loader_time: 0.0045 (0.004692370443113443)  --> TIME: 2025-02-24 23:26:08 -- STEP: 12618/36938 -- GLOBAL_STEP: 529750 | > loss_text_ce: 0.03992842882871628 (0.03705321880136288) | > loss_mel_ce: 2.1247074604034424 (nan) | > loss: 0.008589825592935085 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13679496519046702) | > loader_time: 0.0047 (0.004692418561304483)  --> TIME: 2025-02-24 23:26:18 -- STEP: 12668/36938 -- GLOBAL_STEP: 529800 | > loss_text_ce: 0.03701579570770264 (0.0370620764323805) | > loss_mel_ce: 2.5733354091644287 (nan) | > loss: 0.01035853661596775 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13679628668427643) | > loader_time: 0.0047 (0.004692426795345172)  --> TIME: 2025-02-24 23:26:27 -- STEP: 12718/36938 -- GLOBAL_STEP: 529850 | > loss_text_ce: 0.03533090278506279 (0.037064094173898186) | > loss_mel_ce: 3.3100385665893555 (nan) | > loss: 0.013275276869535446 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.13677669107623378) | > loader_time: 0.0047 (0.004692343575228196)  --> TIME: 2025-02-24 23:26:36 -- STEP: 12768/36938 -- GLOBAL_STEP: 529900 | > loss_text_ce: 0.024187007918953896 (0.03705885960547757) | > loss_mel_ce: 2.6322104930877686 (nan) | > loss: 0.010541261173784733 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13678238507276166) | > loader_time: 0.0046 (0.004692635216509788)  --> TIME: 2025-02-24 23:26:46 -- STEP: 12818/36938 -- GLOBAL_STEP: 529950 | > loss_text_ce: 0.0497402548789978 (0.037058877934948664) | > loss_mel_ce: 1.768535852432251 (nan) | > loss: 0.007215382065623999 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.1367829919028789) | > loader_time: 0.0045 (0.0046928614159422)  --> TIME: 2025-02-24 23:26:55 -- STEP: 12868/36938 -- GLOBAL_STEP: 530000 | > loss_text_ce: 0.04590392857789993 (0.03705806388928771) | > loss_mel_ce: 1.4909577369689941 (nan) | > loss: 0.006098657846450806 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13678887402155307) | > loader_time: 0.0045 (0.0046930207315349455) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_530000.pth  --> TIME: 2025-02-24 23:27:10 -- STEP: 12918/36938 -- GLOBAL_STEP: 530050 | > loss_text_ce: 0.04141097888350487 (0.03705727510739085) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1741 (0.13677810961883796) | > loader_time: 0.0048 (0.004693416900209535)  --> TIME: 2025-02-24 23:27:20 -- STEP: 12968/36938 -- GLOBAL_STEP: 530100 | > loss_text_ce: 0.05002967640757561 (0.037063958311486796) | > loss_mel_ce: 1.1582740545272827 (nan) | > loss: 0.004794856067746878 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13678688548142162) | > loader_time: 0.0052 (0.00469365892477936)  --> TIME: 2025-02-24 23:27:29 -- STEP: 13018/36938 -- GLOBAL_STEP: 530150 | > loss_text_ce: 0.0249195359647274 (0.03705665176494807) | > loss_mel_ce: 3.3656554222106934 (nan) | > loss: 0.013454663567245007 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13678283350023338) | > loader_time: 0.0046 (0.0046939924027370765)  --> TIME: 2025-02-24 23:27:38 -- STEP: 13068/36938 -- GLOBAL_STEP: 530200 | > loss_text_ce: 0.03728365898132324 (0.03705691297720736) | > loss_mel_ce: 2.699720859527588 (nan) | > loss: 0.010861129499971867 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.13676825360052047) | > loader_time: 0.0049 (0.00469386396986066)  --> TIME: 2025-02-24 23:27:47 -- STEP: 13118/36938 -- GLOBAL_STEP: 530250 | > loss_text_ce: 0.03510062023997307 (0.03705410396415847) | > loss_mel_ce: 1.3553951978683472 (nan) | > loss: 0.005517840851098299 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13676471106070784) | > loader_time: 0.0046 (0.004693957159551555)  --> TIME: 2025-02-24 23:27:57 -- STEP: 13168/36938 -- GLOBAL_STEP: 530300 | > loss_text_ce: 0.04158308357000351 (0.03705541438857073) | > loss_mel_ce: 3.449557065963745 (nan) | > loss: 0.01385373156517744 (nan) | > current_lr: 5e-06 | > step_time: 0.1515 (0.13677575087054825) | > loader_time: 0.0054 (0.004694067656921452)  --> TIME: 2025-02-24 23:28:06 -- STEP: 13218/36938 -- GLOBAL_STEP: 530350 | > loss_text_ce: 0.02559478208422661 (0.03705907606190435) | > loss_mel_ce: 2.8603515625 (nan) | > loss: 0.011452168226242065 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.1367814203202589) | > loader_time: 0.0045 (0.004693896060885313)  --> TIME: 2025-02-24 23:28:16 -- STEP: 13268/36938 -- GLOBAL_STEP: 530400 | > loss_text_ce: 0.028816809877753258 (0.03705824685081129) | > loss_mel_ce: 2.934323310852051 (nan) | > loss: 0.011758492328226566 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13679924428480947) | > loader_time: 0.0046 (0.004693861553242746)  --> TIME: 2025-02-24 23:28:25 -- STEP: 13318/36938 -- GLOBAL_STEP: 530450 | > loss_text_ce: 0.041565556079149246 (0.03705896320945106) | > loss_mel_ce: 3.1562142372131348 (nan) | > loss: 0.012689603492617607 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.1367890000038989) | > loader_time: 0.0046 (0.004693599519902912)  --> TIME: 2025-02-24 23:28:34 -- STEP: 13368/36938 -- GLOBAL_STEP: 530500 | > loss_text_ce: 0.028405295684933662 (0.03706456077737116) | > loss_mel_ce: 2.422046184539795 (nan) | > loss: 0.00972401350736618 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.1367837188309499) | > loader_time: 0.0045 (0.004693556338423822)  --> TIME: 2025-02-24 23:28:43 -- STEP: 13418/36938 -- GLOBAL_STEP: 530550 | > loss_text_ce: 0.05123603716492653 (0.03706206046577883) | > loss_mel_ce: 1.206836462020874 (nan) | > loss: 0.004992351401597261 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13678010383542846) | > loader_time: 0.0045 (0.004693342491879982)  --> TIME: 2025-02-24 23:28:53 -- STEP: 13468/36938 -- GLOBAL_STEP: 530600 | > loss_text_ce: 0.020160887390375137 (0.037056284071365915) | > loss_mel_ce: 2.2258877754211426 (nan) | > loss: 0.008912892080843449 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13678770386432731) | > loader_time: 0.0045 (0.004693481134030035)  --> TIME: 2025-02-24 23:29:02 -- STEP: 13518/36938 -- GLOBAL_STEP: 530650 | > loss_text_ce: 0.03472209721803665 (0.03705209378251746) | > loss_mel_ce: 2.529568910598755 (nan) | > loss: 0.010175758972764015 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.1367790134440936) | > loader_time: 0.0048 (0.004693830537097596)  --> TIME: 2025-02-24 23:29:11 -- STEP: 13568/36938 -- GLOBAL_STEP: 530700 | > loss_text_ce: 0.03979877009987831 (0.0370538342976943) | > loss_mel_ce: 3.1417171955108643 (nan) | > loss: 0.012625063769519329 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.13677049410652425) | > loader_time: 0.0047 (0.004693884244364394)  --> TIME: 2025-02-24 23:29:21 -- STEP: 13618/36938 -- GLOBAL_STEP: 530750 | > loss_text_ce: 0.05073601007461548 (0.037058920278775435) | > loss_mel_ce: 2.988398313522339 (nan) | > loss: 0.012060057371854782 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.1367820187065985) | > loader_time: 0.0047 (0.004693743590479441)  --> TIME: 2025-02-24 23:29:30 -- STEP: 13668/36938 -- GLOBAL_STEP: 530800 | > loss_text_ce: 0.028767066076397896 (0.037058103082411066) | > loss_mel_ce: 2.8240485191345215 (nan) | > loss: 0.011320697143673897 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13677176343860897) | > loader_time: 0.0046 (0.00469372870456695)  --> TIME: 2025-02-24 23:29:39 -- STEP: 13718/36938 -- GLOBAL_STEP: 530850 | > loss_text_ce: 0.036779824644327164 (0.03706256027901962) | > loss_mel_ce: 3.1249868869781494 (nan) | > loss: 0.012546693906188011 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13676567878158627) | > loader_time: 0.0045 (0.004694356430305101)  --> TIME: 2025-02-24 23:29:48 -- STEP: 13768/36938 -- GLOBAL_STEP: 530900 | > loss_text_ce: 0.04058138281106949 (0.03706621939185437) | > loss_mel_ce: 3.1471939086914062 (nan) | > loss: 0.012649903073906898 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13677776945600106) | > loader_time: 0.0046 (0.004694361107630628)  --> TIME: 2025-02-24 23:29:58 -- STEP: 13818/36938 -- GLOBAL_STEP: 530950 | > loss_text_ce: 0.02576713263988495 (0.037065049184909334) | > loss_mel_ce: 3.3391170501708984 (nan) | > loss: 0.013352715410292149 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1367707326177888) | > loader_time: 0.0046 (0.004694245178735043)  --> TIME: 2025-02-24 23:30:07 -- STEP: 13868/36938 -- GLOBAL_STEP: 531000 | > loss_text_ce: 0.04055820778012276 (0.03706376616275447) | > loss_mel_ce: 3.0921831130981445 (nan) | > loss: 0.012431513518095016 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13678347407688415) | > loader_time: 0.0045 (0.004694091472565223)  --> TIME: 2025-02-24 23:30:16 -- STEP: 13918/36938 -- GLOBAL_STEP: 531050 | > loss_text_ce: 0.04061572626233101 (0.03705583707746518) | > loss_mel_ce: 3.3524434566497803 (nan) | > loss: 0.013464521616697311 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13678058323666245) | > loader_time: 0.0046 (0.004694261518698489)  --> TIME: 2025-02-24 23:30:25 -- STEP: 13968/36938 -- GLOBAL_STEP: 531100 | > loss_text_ce: 0.03429137542843819 (0.0370559339565711) | > loss_mel_ce: 2.2911932468414307 (nan) | > loss: 0.009228114038705826 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13676136066413824) | > loader_time: 0.0047 (0.0046940658749310615)  --> TIME: 2025-02-24 23:30:35 -- STEP: 14018/36938 -- GLOBAL_STEP: 531150 | > loss_text_ce: 0.04925544187426567 (0.03705456166745157) | > loss_mel_ce: 3.1651198863983154 (nan) | > loss: 0.012755458243191242 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13675835610864712) | > loader_time: 0.0047 (0.004694040992797779)  --> TIME: 2025-02-24 23:30:44 -- STEP: 14068/36938 -- GLOBAL_STEP: 531200 | > loss_text_ce: 0.03376132249832153 (0.037059766758775696) | > loss_mel_ce: 3.057373523712158 (nan) | > loss: 0.012266408652067184 (nan) | > current_lr: 5e-06 | > step_time: 0.1518 (0.1367471566212458) | > loader_time: 0.0047 (0.004694292211112299)  --> TIME: 2025-02-24 23:30:53 -- STEP: 14118/36938 -- GLOBAL_STEP: 531250 | > loss_text_ce: 0.03809051960706711 (0.03706294090850812) | > loss_mel_ce: 3.203040838241577 (nan) | > loss: 0.012861632741987705 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.13673384597609164) | > loader_time: 0.0046 (0.004694080687156907)  --> TIME: 2025-02-24 23:31:02 -- STEP: 14168/36938 -- GLOBAL_STEP: 531300 | > loss_text_ce: 0.032025497406721115 (0.037054206406353776) | > loss_mel_ce: 2.9236810207366943 (nan) | > loss: 0.011728995479643345 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13672877532624034) | > loader_time: 0.0046 (0.004694140408417391)  --> TIME: 2025-02-24 23:31:12 -- STEP: 14218/36938 -- GLOBAL_STEP: 531350 | > loss_text_ce: 0.0384555347263813 (0.037052649381167216) | > loss_mel_ce: 3.455153465270996 (nan) | > loss: 0.01386352814733982 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.13673553142823947) | > loader_time: 0.0046 (0.004694758831222182)  --> TIME: 2025-02-24 23:31:21 -- STEP: 14268/36938 -- GLOBAL_STEP: 531400 | > loss_text_ce: 0.03540070354938507 (0.037048475121421884) | > loss_mel_ce: 2.7512171268463135 (nan) | > loss: 0.01105800736695528 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.13673668496575983) | > loader_time: 0.0046 (0.004694628053798312)  --> TIME: 2025-02-24 23:31:30 -- STEP: 14318/36938 -- GLOBAL_STEP: 531450 | > loss_text_ce: 0.0312038566917181 (0.03704589831520917) | > loss_mel_ce: 3.4854962825775146 (nan) | > loss: 0.013955160044133663 (nan) | > current_lr: 5e-06 | > step_time: 0.1976 (0.1367404424273313) | > loader_time: 0.0049 (0.004694951248328807)  --> TIME: 2025-02-24 23:31:40 -- STEP: 14368/36938 -- GLOBAL_STEP: 531500 | > loss_text_ce: 0.03273378312587738 (0.03704556649592805) | > loss_mel_ce: 2.909285068511963 (nan) | > loss: 0.011674677953124046 (nan) | > current_lr: 5e-06 | > step_time: 0.1583 (0.13674021854100618) | > loader_time: 0.0053 (0.004695200757486523)  --> TIME: 2025-02-24 23:31:49 -- STEP: 14418/36938 -- GLOBAL_STEP: 531550 | > loss_text_ce: 0.035327281802892685 (0.03704608667472389) | > loss_mel_ce: 3.1742637157440186 (nan) | > loss: 0.012736472301185131 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1367364930683241) | > loader_time: 0.0047 (0.004695162212225256)  --> TIME: 2025-02-24 23:31:58 -- STEP: 14468/36938 -- GLOBAL_STEP: 531600 | > loss_text_ce: 0.03172754868865013 (0.037044688643677076) | > loss_mel_ce: 3.7270243167877197 (nan) | > loss: 0.014915683306753635 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.1367265841794139) | > loader_time: 0.0046 (0.00469535963292079)  --> TIME: 2025-02-24 23:32:07 -- STEP: 14518/36938 -- GLOBAL_STEP: 531650 | > loss_text_ce: 0.035712793469429016 (0.03704338549889188) | > loss_mel_ce: 2.499305248260498 (nan) | > loss: 0.010059596039354801 (nan) | > current_lr: 5e-06 | > step_time: 0.1789 (0.13672350643258857) | > loader_time: 0.0048 (0.004695416186561701)  --> TIME: 2025-02-24 23:32:17 -- STEP: 14568/36938 -- GLOBAL_STEP: 531700 | > loss_text_ce: 0.042700257152318954 (0.037043978901756994) | > loss_mel_ce: 3.3286216259002686 (nan) | > loss: 0.013378262519836426 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.13672273468997445) | > loader_time: 0.0049 (0.004695256649302238)  --> TIME: 2025-02-24 23:32:26 -- STEP: 14618/36938 -- GLOBAL_STEP: 531750 | > loss_text_ce: 0.03598994016647339 (0.03704303038394589) | > loss_mel_ce: 2.641298532485962 (nan) | > loss: 0.01062416099011898 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13671520647165614) | > loader_time: 0.0045 (0.004695455961766303)  --> TIME: 2025-02-24 23:32:35 -- STEP: 14668/36938 -- GLOBAL_STEP: 531800 | > loss_text_ce: 0.038513362407684326 (0.03703931216640698) | > loss_mel_ce: 3.438255786895752 (nan) | > loss: 0.013796703889966011 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13671510465664605) | > loader_time: 0.0047 (0.004695455401217485)  --> TIME: 2025-02-24 23:32:45 -- STEP: 14718/36938 -- GLOBAL_STEP: 531850 | > loss_text_ce: 0.034477293491363525 (0.03703577833773858) | > loss_mel_ce: 0.09823913872241974 (nan) | > loss: 0.0005266525549814105 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.13672119093129198) | > loader_time: 0.0046 (0.00469539801798076)  --> TIME: 2025-02-24 23:32:54 -- STEP: 14768/36938 -- GLOBAL_STEP: 531900 | > loss_text_ce: 0.03859302029013634 (0.03704119495074306) | > loss_mel_ce: 3.1234793663024902 (nan) | > loss: 0.012547907419502735 (nan) | > current_lr: 5e-06 | > step_time: 0.182 (0.13671861756855336) | > loader_time: 0.0049 (0.004695451014220139)  --> TIME: 2025-02-24 23:33:03 -- STEP: 14818/36938 -- GLOBAL_STEP: 531950 | > loss_text_ce: 0.03213119879364967 (0.03703594188845599) | > loss_mel_ce: 2.708350419998169 (nan) | > loss: 0.01087492797523737 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13671712238853923) | > loader_time: 0.0049 (0.004695389190011706)  --> TIME: 2025-02-24 23:33:13 -- STEP: 14868/36938 -- GLOBAL_STEP: 532000 | > loss_text_ce: 0.030778929591178894 (0.0370343306266225) | > loss_mel_ce: 2.4190003871917725 (nan) | > loss: 0.009721347130835056 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13672545363834404) | > loader_time: 0.0048 (0.0046957921911254565)  --> TIME: 2025-02-24 23:33:22 -- STEP: 14918/36938 -- GLOBAL_STEP: 532050 | > loss_text_ce: 0.0362178310751915 (0.03703491812948214) | > loss_mel_ce: 3.228713035583496 (nan) | > loss: 0.012956076301634312 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.1367226330839862) | > loader_time: 0.0045 (0.004695661299061502)  --> TIME: 2025-02-24 23:33:32 -- STEP: 14968/36938 -- GLOBAL_STEP: 532100 | > loss_text_ce: 0.031221605837345123 (0.03703668817080352) | > loss_mel_ce: 0.048828125 (nan) | > loss: 0.00031765768653713167 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.13672733287770486) | > loader_time: 0.0049 (0.004695696269427315)  --> TIME: 2025-02-24 23:33:41 -- STEP: 15018/36938 -- GLOBAL_STEP: 532150 | > loss_text_ce: 0.046086106449365616 (0.037034803677862826) | > loss_mel_ce: 3.0975911617279053 (nan) | > loss: 0.012474910356104374 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13672665348540664) | > loader_time: 0.0045 (0.004695700081421262)  --> TIME: 2025-02-24 23:33:50 -- STEP: 15068/36938 -- GLOBAL_STEP: 532200 | > loss_text_ce: 0.043857116252183914 (0.03703529621334271) | > loss_mel_ce: 0.7595062255859375 (nan) | > loss: 0.0031879497691988945 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13670950441294638) | > loader_time: 0.0045 (0.004695565655594141)  --> TIME: 2025-02-24 23:33:59 -- STEP: 15118/36938 -- GLOBAL_STEP: 532250 | > loss_text_ce: 0.035486068576574326 (0.0370332690478599) | > loss_mel_ce: 0.65472412109375 (nan) | > loss: 0.0027389293536543846 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.1366922426419308) | > loader_time: 0.0046 (0.004695310433684626)  --> TIME: 2025-02-24 23:34:08 -- STEP: 15168/36938 -- GLOBAL_STEP: 532300 | > loss_text_ce: 0.03754747286438942 (0.03703189819489017) | > loss_mel_ce: 3.4181885719299316 (nan) | > loss: 0.013713239692151546 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.13669334721401763) | > loader_time: 0.0047 (0.004695472507914416)  --> TIME: 2025-02-24 23:34:18 -- STEP: 15218/36938 -- GLOBAL_STEP: 532350 | > loss_text_ce: 0.044437192380428314 (0.03703059499947717) | > loss_mel_ce: 2.1010727882385254 (nan) | > loss: 0.008513929322361946 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13669318392361665) | > loader_time: 0.0045 (0.004695779892786969)  --> TIME: 2025-02-24 23:34:27 -- STEP: 15268/36938 -- GLOBAL_STEP: 532400 | > loss_text_ce: 0.03661801666021347 (0.0370252712484926) | > loss_mel_ce: 2.818462371826172 (nan) | > loss: 0.011329684406518936 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.13668307903270793) | > loader_time: 0.0047 (0.0046956475286356394)  --> TIME: 2025-02-24 23:34:36 -- STEP: 15318/36938 -- GLOBAL_STEP: 532450 | > loss_text_ce: 0.02555098943412304 (0.03702267374336786) | > loss_mel_ce: 2.4080684185028076 (nan) | > loss: 0.009657220914959908 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.1366673453169494) | > loader_time: 0.0047 (0.004695566722502803)  --> TIME: 2025-02-24 23:34:45 -- STEP: 15368/36938 -- GLOBAL_STEP: 532500 | > loss_text_ce: 0.02838396281003952 (0.03701343296929853) | > loss_mel_ce: 2.8119938373565674 (nan) | > loss: 0.011271340772509575 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13668582695092701) | > loader_time: 0.0047 (0.004695714016496861)  --> TIME: 2025-02-24 23:34:55 -- STEP: 15418/36938 -- GLOBAL_STEP: 532550 | > loss_text_ce: 0.039545897394418716 (0.037015702948445424) | > loss_mel_ce: 1.2562408447265625 (nan) | > loss: 0.00514201121404767 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13669155084417234) | > loader_time: 0.0045 (0.00469566350286922)  --> TIME: 2025-02-24 23:35:04 -- STEP: 15468/36938 -- GLOBAL_STEP: 532600 | > loss_text_ce: 0.04070261865854263 (0.037019189062244426) | > loss_mel_ce: 2.6976206302642822 (nan) | > loss: 0.010866362601518631 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13669780771461654) | > loader_time: 0.0046 (0.0046956941758997586)  --> TIME: 2025-02-24 23:35:13 -- STEP: 15518/36938 -- GLOBAL_STEP: 532650 | > loss_text_ce: 0.03298231214284897 (0.03701722067679372) | > loss_mel_ce: 3.6987345218658447 (nan) | > loss: 0.014808401465415955 (nan) | > current_lr: 5e-06 | > step_time: 0.1827 (0.13668665530709562) | > loader_time: 0.0049 (0.004695538700756182)  --> TIME: 2025-02-24 23:35:22 -- STEP: 15568/36938 -- GLOBAL_STEP: 532700 | > loss_text_ce: 0.03722837194800377 (0.03701777786119605) | > loss_mel_ce: 3.0327770709991455 (nan) | > loss: 0.012182561680674553 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13667225969413557) | > loader_time: 0.0048 (0.004695382080244754)  --> TIME: 2025-02-24 23:35:32 -- STEP: 15618/36938 -- GLOBAL_STEP: 532750 | > loss_text_ce: 0.03825121000409126 (0.037019195926052184) | > loss_mel_ce: 3.0887718200683594 (nan) | > loss: 0.01240882184356451 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.1366676904279239) | > loader_time: 0.0047 (0.004695297340864963)  --> TIME: 2025-02-24 23:35:41 -- STEP: 15668/36938 -- GLOBAL_STEP: 532800 | > loss_text_ce: 0.03861089423298836 (0.037013732124596056) | > loss_mel_ce: 3.0769708156585693 (nan) | > loss: 0.01236342079937458 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.1366730852792966) | > loader_time: 0.0045 (0.004695297900529311)  --> TIME: 2025-02-24 23:35:51 -- STEP: 15718/36938 -- GLOBAL_STEP: 532850 | > loss_text_ce: 0.030306920409202576 (0.037011305327258395) | > loss_mel_ce: 3.122993230819702 (nan) | > loss: 0.012513096444308758 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13668888626033093) | > loader_time: 0.0046 (0.004695259291550214)  --> TIME: 2025-02-24 23:36:00 -- STEP: 15768/36938 -- GLOBAL_STEP: 532900 | > loss_text_ce: 0.029929980635643005 (0.03701018604476158) | > loss_mel_ce: 3.4734418392181396 (nan) | > loss: 0.013902269303798676 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.13669892581570023) | > loader_time: 0.0055 (0.0046955938873891045)  --> TIME: 2025-02-24 23:36:09 -- STEP: 15818/36938 -- GLOBAL_STEP: 532950 | > loss_text_ce: 0.03448724374175072 (0.0370091645196971) | > loss_mel_ce: 2.0241928100585938 (nan) | > loss: 0.008169366046786308 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.13669744974624212) | > loader_time: 0.0047 (0.004695627885004084)  --> TIME: 2025-02-24 23:36:19 -- STEP: 15868/36938 -- GLOBAL_STEP: 533000 | > loss_text_ce: 0.03788457438349724 (0.03701168807937842) | > loss_mel_ce: 3.0116331577301025 (nan) | > loss: 0.01210126094520092 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13668820128982953) | > loader_time: 0.0046 (0.004695879637759315)  --> TIME: 2025-02-24 23:36:28 -- STEP: 15918/36938 -- GLOBAL_STEP: 533050 | > loss_text_ce: 0.040196049958467484 (0.03700991139186355) | > loss_mel_ce: 3.2233219146728516 (nan) | > loss: 0.012950468808412552 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.13669641021808068) | > loader_time: 0.0047 (0.004695838847821171)  --> TIME: 2025-02-24 23:36:38 -- STEP: 15968/36938 -- GLOBAL_STEP: 533100 | > loss_text_ce: 0.03424610197544098 (0.03701048523808447) | > loss_mel_ce: 0.15837478637695312 (nan) | > loss: 0.0007643686258234084 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.1367026507436392) | > loader_time: 0.0046 (0.004696062487925227)  --> TIME: 2025-02-24 23:36:47 -- STEP: 16018/36938 -- GLOBAL_STEP: 533150 | > loss_text_ce: 0.03582288324832916 (0.037004732086295004) | > loss_mel_ce: 3.4220948219299316 (nan) | > loss: 0.013721896335482597 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13669970211486768) | > loader_time: 0.0045 (0.004696142972669203)  --> TIME: 2025-02-24 23:36:56 -- STEP: 16068/36938 -- GLOBAL_STEP: 533200 | > loss_text_ce: 0.037109073251485825 (0.037008159257829125) | > loss_mel_ce: 2.1534790992736816 (nan) | > loss: 0.008692811243236065 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13670272451150992) | > loader_time: 0.0045 (0.004696252246920321)  --> TIME: 2025-02-24 23:37:06 -- STEP: 16118/36938 -- GLOBAL_STEP: 533250 | > loss_text_ce: 0.0336165726184845 (0.037008380295424775) | > loss_mel_ce: 3.3244354724884033 (nan) | > loss: 0.013325603678822517 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.13671410756809912) | > loader_time: 0.0047 (0.004696564456046499)  --> TIME: 2025-02-24 23:37:15 -- STEP: 16168/36938 -- GLOBAL_STEP: 533300 | > loss_text_ce: 0.03582233935594559 (0.0370084804886324) | > loss_mel_ce: 3.4116594791412354 (nan) | > loss: 0.013680484145879745 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.13671948461766165) | > loader_time: 0.0048 (0.004696760715558238)  --> TIME: 2025-02-24 23:37:24 -- STEP: 16218/36938 -- GLOBAL_STEP: 533350 | > loss_text_ce: 0.03529798984527588 (0.03700945312469672) | > loss_mel_ce: 2.764791250228882 (nan) | > loss: 0.011111466214060783 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1367025461807278) | > loader_time: 0.0048 (0.00469674304346369)  --> TIME: 2025-02-24 23:37:33 -- STEP: 16268/36938 -- GLOBAL_STEP: 533400 | > loss_text_ce: 0.03396309167146683 (0.037003727067540904) | > loss_mel_ce: 3.3795571327209473 (nan) | > loss: 0.013545716181397438 (nan) | > current_lr: 5e-06 | > step_time: 0.161 (0.13669266885699283) | > loader_time: 0.0047 (0.004696624311850942)  --> TIME: 2025-02-24 23:37:43 -- STEP: 16318/36938 -- GLOBAL_STEP: 533450 | > loss_text_ce: 0.03362506255507469 (0.0370031949172062) | > loss_mel_ce: 2.9328739643096924 (nan) | > loss: 0.011771822348237038 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.13669314079387518) | > loader_time: 0.0046 (0.004696850362310872)  --> TIME: 2025-02-24 23:37:52 -- STEP: 16368/36938 -- GLOBAL_STEP: 533500 | > loss_text_ce: 0.04977273941040039 (0.03700300084830077) | > loss_mel_ce: 2.9290153980255127 (nan) | > loss: 0.011820588260889053 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.136706304719138) | > loader_time: 0.0047 (0.004696749216300656)  --> TIME: 2025-02-24 23:38:01 -- STEP: 16418/36938 -- GLOBAL_STEP: 533550 | > loss_text_ce: 0.023475561290979385 (0.037002742669481295) | > loss_mel_ce: 2.277127981185913 (nan) | > loss: 0.009129379875957966 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13669141495702664) | > loader_time: 0.0046 (0.004696761927196328)  --> TIME: 2025-02-24 23:38:10 -- STEP: 16468/36938 -- GLOBAL_STEP: 533600 | > loss_text_ce: 0.03642108291387558 (0.03700093212578191) | > loss_mel_ce: 2.512679100036621 (nan) | > loss: 0.010115477256476879 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.1366734443459082) | > loader_time: 0.0046 (0.004696773026271563)  --> TIME: 2025-02-24 23:38:19 -- STEP: 16518/36938 -- GLOBAL_STEP: 533650 | > loss_text_ce: 0.04010558873414993 (0.03700015205333565) | > loss_mel_ce: 2.0903425216674805 (nan) | > loss: 0.008454159833490849 (nan) | > current_lr: 5e-06 | > step_time: 0.1271 (0.13666104120006703) | > loader_time: 0.0047 (0.004696842226628204)  --> TIME: 2025-02-24 23:38:29 -- STEP: 16568/36938 -- GLOBAL_STEP: 533700 | > loss_text_ce: 0.03474079817533493 (0.03700358484922731) | > loss_mel_ce: 3.3212246894836426 (nan) | > loss: 0.013317323289811611 (nan) | > current_lr: 5e-06 | > step_time: 0.1571 (0.1366568050886454) | > loader_time: 0.0047 (0.004696889941903032)  --> TIME: 2025-02-24 23:38:38 -- STEP: 16618/36938 -- GLOBAL_STEP: 533750 | > loss_text_ce: 0.04311291500926018 (0.037004516845633405) | > loss_mel_ce: 3.525379180908203 (nan) | > loss: 0.014160684309899807 (nan) | > current_lr: 5e-06 | > step_time: 0.1829 (0.13666221281727198) | > loader_time: 0.0052 (0.004697029104817003)  --> TIME: 2025-02-24 23:38:47 -- STEP: 16668/36938 -- GLOBAL_STEP: 533800 | > loss_text_ce: 0.03691382706165314 (0.037004786145104364) | > loss_mel_ce: 3.2892863750457764 (nan) | > loss: 0.013199208304286003 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13665982096035578) | > loader_time: 0.0048 (0.0046971198149027265)  --> TIME: 2025-02-24 23:38:57 -- STEP: 16718/36938 -- GLOBAL_STEP: 533850 | > loss_text_ce: 0.032454803586006165 (0.03700319278651245) | > loss_mel_ce: 2.9186158180236816 (nan) | > loss: 0.01171059813350439 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13666595254935254) | > loader_time: 0.0045 (0.004697260552579846)  --> TIME: 2025-02-24 23:39:06 -- STEP: 16768/36938 -- GLOBAL_STEP: 533900 | > loss_text_ce: 0.03933379054069519 (0.037008589141451706) | > loss_mel_ce: 3.1103858947753906 (nan) | > loss: 0.012498888187110424 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.13667168775360083) | > loader_time: 0.0046 (0.004697182407702203)  --> TIME: 2025-02-24 23:39:15 -- STEP: 16818/36938 -- GLOBAL_STEP: 533950 | > loss_text_ce: 0.030383262783288956 (0.03700610864089045) | > loss_mel_ce: 3.403182029724121 (nan) | > loss: 0.013625260442495346 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13666837620017835) | > loader_time: 0.0047 (0.004697226786219364)  --> TIME: 2025-02-24 23:39:25 -- STEP: 16868/36938 -- GLOBAL_STEP: 534000 | > loss_text_ce: 0.038520704954862595 (0.03700489299496626) | > loss_mel_ce: 2.9422388076782227 (nan) | > loss: 0.011828412301838398 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13667721361918325) | > loader_time: 0.0046 (0.004697201982447967)  --> TIME: 2025-02-24 23:39:34 -- STEP: 16918/36938 -- GLOBAL_STEP: 534050 | > loss_text_ce: 0.030912000685930252 (0.03700095036646523) | > loss_mel_ce: 2.843280076980591 (nan) | > loss: 0.011405524797737598 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13668258244552448) | > loader_time: 0.0045 (0.004697460093274442)  --> TIME: 2025-02-24 23:39:44 -- STEP: 16968/36938 -- GLOBAL_STEP: 534100 | > loss_text_ce: 0.03168829530477524 (0.03699693526124886) | > loss_mel_ce: 2.3566031455993652 (nan) | > loss: 0.009477347135543823 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.13668304396541192) | > loader_time: 0.0053 (0.00469787118851942)  --> TIME: 2025-02-24 23:39:53 -- STEP: 17018/36938 -- GLOBAL_STEP: 534150 | > loss_text_ce: 0.039841584861278534 (0.03699569643862251) | > loss_mel_ce: 3.430724620819092 (nan) | > loss: 0.013772089034318924 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.136677822280482) | > loader_time: 0.0048 (0.0046980213034599925)  --> TIME: 2025-02-24 23:40:02 -- STEP: 17068/36938 -- GLOBAL_STEP: 534200 | > loss_text_ce: 0.042140472680330276 (0.03699266268356196) | > loss_mel_ce: 1.4707742929458618 (nan) | > loss: 0.006003630347549915 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13665535203014229) | > loader_time: 0.0045 (0.004697731668786532)  --> TIME: 2025-02-24 23:40:11 -- STEP: 17118/36938 -- GLOBAL_STEP: 534250 | > loss_text_ce: 0.029652748256921768 (0.036983991508420175) | > loss_mel_ce: 2.0968971252441406 (nan) | > loss: 0.008438690565526485 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.1366540618777929) | > loader_time: 0.0047 (0.004698007947193266)  --> TIME: 2025-02-24 23:40:20 -- STEP: 17168/36938 -- GLOBAL_STEP: 534300 | > loss_text_ce: 0.045057229697704315 (0.03698236875559139) | > loss_mel_ce: 2.7345879077911377 (nan) | > loss: 0.011030338704586029 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13665296634501195) | > loader_time: 0.0046 (0.004698018200577475)  --> TIME: 2025-02-24 23:40:29 -- STEP: 17218/36938 -- GLOBAL_STEP: 534350 | > loss_text_ce: 0.03224857151508331 (0.03698016245391725) | > loss_mel_ce: 3.0366532802581787 (nan) | > loss: 0.012178182601928711 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.13664218538631911) | > loader_time: 0.0046 (0.004697806259967933)  --> TIME: 2025-02-24 23:40:39 -- STEP: 17268/36938 -- GLOBAL_STEP: 534400 | > loss_text_ce: 0.022650156170129776 (0.03697623172543002) | > loss_mel_ce: 2.268749475479126 (nan) | > loss: 0.00909285619854927 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.13665303135735574) | > loader_time: 0.0046 (0.004697787394423109)  --> TIME: 2025-02-24 23:40:48 -- STEP: 17318/36938 -- GLOBAL_STEP: 534450 | > loss_text_ce: 0.0352945514023304 (0.036975243597293914) | > loss_mel_ce: 3.2643401622772217 (nan) | > loss: 0.013093789108097553 (nan) | > current_lr: 5e-06 | > step_time: 0.1489 (0.13664612954284316) | > loader_time: 0.0047 (0.0046975509249485336)  --> TIME: 2025-02-24 23:40:57 -- STEP: 17368/36938 -- GLOBAL_STEP: 534500 | > loss_text_ce: 0.03604196757078171 (0.03697534025916366) | > loss_mel_ce: 3.130418539047241 (nan) | > loss: 0.01256532035768032 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.136636373374606) | > loader_time: 0.0046 (0.004697477842244839)  --> TIME: 2025-02-24 23:41:06 -- STEP: 17418/36938 -- GLOBAL_STEP: 534550 | > loss_text_ce: 0.03709246218204498 (0.036976598876410104) | > loss_mel_ce: 3.067728042602539 (nan) | > loss: 0.012320716865360737 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13664053002185675) | > loader_time: 0.0046 (0.004697608090881535)  --> TIME: 2025-02-24 23:41:16 -- STEP: 17468/36938 -- GLOBAL_STEP: 534600 | > loss_text_ce: 0.042937688529491425 (0.03697669752024692) | > loss_mel_ce: 3.1325571537017822 (nan) | > loss: 0.012601170688867569 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.13663727794382544) | > loader_time: 0.0047 (0.004697443255849695)  --> TIME: 2025-02-24 23:41:25 -- STEP: 17518/36938 -- GLOBAL_STEP: 534650 | > loss_text_ce: 0.027618318796157837 (0.03697686471715521) | > loss_mel_ce: 2.569207191467285 (nan) | > loss: 0.01030486449599266 (nan) | > current_lr: 5e-06 | > step_time: 0.1793 (0.13662889567476558) | > loader_time: 0.0048 (0.004697342321254554)  --> TIME: 2025-02-24 23:41:34 -- STEP: 17568/36938 -- GLOBAL_STEP: 534700 | > loss_text_ce: 0.04499818757176399 (0.0369746540411726) | > loss_mel_ce: 3.2986724376678467 (nan) | > loss: 0.013268535025417805 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13661919844986528) | > loader_time: 0.0055 (0.004697749116381666)  --> TIME: 2025-02-24 23:41:43 -- STEP: 17618/36938 -- GLOBAL_STEP: 534750 | > loss_text_ce: 0.03398939222097397 (0.0369755213105757) | > loss_mel_ce: 1.427301049232483 (nan) | > loss: 0.005798771977424622 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.13661527334594978) | > loader_time: 0.0046 (0.0046975577184785655)  --> TIME: 2025-02-24 23:41:53 -- STEP: 17668/36938 -- GLOBAL_STEP: 534800 | > loss_text_ce: 0.04220786318182945 (0.03697683321838565) | > loss_mel_ce: 3.190849542617798 (nan) | > loss: 0.012829594314098358 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13662316579475423) | > loader_time: 0.0047 (0.004697602624290804)  --> TIME: 2025-02-24 23:42:02 -- STEP: 17718/36938 -- GLOBAL_STEP: 534850 | > loss_text_ce: 0.038639772683382034 (0.03697579757258204) | > loss_mel_ce: 2.6558024883270264 (nan) | > loss: 0.0106922322884202 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13661117982428392) | > loader_time: 0.0045 (0.004697457798610009)  --> TIME: 2025-02-24 23:42:11 -- STEP: 17768/36938 -- GLOBAL_STEP: 534900 | > loss_text_ce: 0.03696172684431076 (0.036975492450042995) | > loss_mel_ce: 3.685546875 (nan) | > loss: 0.014771861024200916 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13661112368294692) | > loader_time: 0.0046 (0.004697309722239134)  --> TIME: 2025-02-24 23:42:20 -- STEP: 17818/36938 -- GLOBAL_STEP: 534950 | > loss_text_ce: 0.038577668368816376 (0.03697569619353874) | > loss_mel_ce: 2.9446585178375244 (nan) | > loss: 0.011838238686323166 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.1365816940936405) | > loader_time: 0.0045 (0.004697240312862968)  --> TIME: 2025-02-24 23:42:29 -- STEP: 17868/36938 -- GLOBAL_STEP: 535000 | > loss_text_ce: 0.03724187612533569 (0.036976507976278015) | > loss_mel_ce: 3.4336583614349365 (nan) | > loss: 0.013773414306342602 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.1366007741409738) | > loader_time: 0.0047 (0.00469747382521176)  --> TIME: 2025-02-24 23:42:39 -- STEP: 17918/36938 -- GLOBAL_STEP: 535050 | > loss_text_ce: 0.037602074444293976 (0.036976749627615095) | > loss_mel_ce: 2.4599781036376953 (nan) | > loss: 0.00991103332489729 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13660628253533538) | > loader_time: 0.0045 (0.004697522702979807)  --> TIME: 2025-02-24 23:42:48 -- STEP: 17968/36938 -- GLOBAL_STEP: 535100 | > loss_text_ce: 0.03436454385519028 (0.03697233124612687) | > loss_mel_ce: 2.995619535446167 (nan) | > loss: 0.012023746967315674 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.13661705411287256) | > loader_time: 0.0047 (0.004697954161084347)  --> TIME: 2025-02-24 23:42:58 -- STEP: 18018/36938 -- GLOBAL_STEP: 535150 | > loss_text_ce: 0.03779833763837814 (0.03696847384181902) | > loss_mel_ce: 3.144170045852661 (nan) | > loss: 0.012626859359443188 (nan) | > current_lr: 5e-06 | > step_time: 0.1705 (0.13661418474267045) | > loader_time: 0.005 (0.0046977455759699485)  --> TIME: 2025-02-24 23:43:07 -- STEP: 18068/36938 -- GLOBAL_STEP: 535200 | > loss_text_ce: 0.024765552952885628 (0.03696315585710241) | > loss_mel_ce: 2.500662326812744 (nan) | > loss: 0.010021539404988289 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13661380524864394) | > loader_time: 0.0045 (0.004697849034260195)  --> TIME: 2025-02-24 23:43:16 -- STEP: 18118/36938 -- GLOBAL_STEP: 535250 | > loss_text_ce: 0.03776390105485916 (0.03696114058775033) | > loss_mel_ce: 2.7442739009857178 (nan) | > loss: 0.01103983260691166 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13660195366227473) | > loader_time: 0.0048 (0.004697988635723343)  --> TIME: 2025-02-24 23:43:25 -- STEP: 18168/36938 -- GLOBAL_STEP: 535300 | > loss_text_ce: 0.02554682455956936 (0.03696145397258899) | > loss_mel_ce: 2.6157596111297607 (nan) | > loss: 0.010481375269591808 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.1366024989616707) | > loader_time: 0.0047 (0.0046980007531515905)  --> TIME: 2025-02-24 23:43:35 -- STEP: 18218/36938 -- GLOBAL_STEP: 535350 | > loss_text_ce: 0.04259384423494339 (0.03696017124417029) | > loss_mel_ce: 2.638519287109375 (nan) | > loss: 0.010639338754117489 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13659759930949764) | > loader_time: 0.0045 (0.004698092032635775)  --> TIME: 2025-02-24 23:43:44 -- STEP: 18268/36938 -- GLOBAL_STEP: 535400 | > loss_text_ce: 0.0387037992477417 (0.036958725397361496) | > loss_mel_ce: 2.65028715133667 (nan) | > loss: 0.010670600458979607 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.13659551566764827) | > loader_time: 0.0047 (0.004697892580769483)  --> TIME: 2025-02-24 23:43:53 -- STEP: 18318/36938 -- GLOBAL_STEP: 535450 | > loss_text_ce: 0.03679726645350456 (0.03696146298224095) | > loss_mel_ce: 3.6768057346343994 (nan) | > loss: 0.014736521057784557 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13660211882771897) | > loader_time: 0.0047 (0.004697871515357557)  --> TIME: 2025-02-24 23:44:03 -- STEP: 18368/36938 -- GLOBAL_STEP: 535500 | > loss_text_ce: 0.034761957824230194 (0.036958364076445524) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13661284367018456) | > loader_time: 0.0052 (0.004697904328228292)  --> TIME: 2025-02-24 23:44:13 -- STEP: 18418/36938 -- GLOBAL_STEP: 535550 | > loss_text_ce: 0.031546756625175476 (0.036958290886223676) | > loss_mel_ce: 2.9360921382904053 (nan) | > loss: 0.011776345781981945 (nan) | > current_lr: 5e-06 | > step_time: 0.1571 (0.13663343714507867) | > loader_time: 0.0046 (0.004698017920147056)  --> TIME: 2025-02-24 23:44:22 -- STEP: 18468/36938 -- GLOBAL_STEP: 535600 | > loss_text_ce: 0.03874268755316734 (0.036958712118391135) | > loss_mel_ce: 2.7381696701049805 (nan) | > loss: 0.011019494384527206 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1366349212116872) | > loader_time: 0.0047 (0.004697990683422791)  --> TIME: 2025-02-24 23:44:31 -- STEP: 18518/36938 -- GLOBAL_STEP: 535650 | > loss_text_ce: 0.0341995470225811 (0.03695780018007356) | > loss_mel_ce: 3.0328149795532227 (nan) | > loss: 0.012170692905783653 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13664031676358238) | > loader_time: 0.0046 (0.004698111436989933)  --> TIME: 2025-02-24 23:44:41 -- STEP: 18568/36938 -- GLOBAL_STEP: 535700 | > loss_text_ce: 0.03423219546675682 (0.036956425134414765) | > loss_mel_ce: 3.1459081172943115 (nan) | > loss: 0.012619605287909508 (nan) | > current_lr: 5e-06 | > step_time: 0.1779 (0.13664879785021644) | > loader_time: 0.0048 (0.004698023463261132)  --> TIME: 2025-02-24 23:44:50 -- STEP: 18618/36938 -- GLOBAL_STEP: 535750 | > loss_text_ce: 0.03915746510028839 (0.03695878844990527) | > loss_mel_ce: 2.59112548828125 (nan) | > loss: 0.01043763104826212 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13664066345892223) | > loader_time: 0.0045 (0.004697820863427694)  --> TIME: 2025-02-24 23:45:00 -- STEP: 18668/36938 -- GLOBAL_STEP: 535800 | > loss_text_ce: 0.02998950704932213 (0.03695686948966594) | > loss_mel_ce: 3.4762091636657715 (nan) | > loss: 0.013913487084209919 (nan) | > current_lr: 5e-06 | > step_time: 0.1779 (0.13665513865105317) | > loader_time: 0.0048 (0.004697960820779431)  --> TIME: 2025-02-24 23:45:09 -- STEP: 18718/36938 -- GLOBAL_STEP: 535850 | > loss_text_ce: 0.04817451536655426 (0.03695879888788769) | > loss_mel_ce: 2.4293336868286133 (nan) | > loss: 0.009831382893025875 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13664191811624038) | > loader_time: 0.0046 (0.004697995341757792)  --> TIME: 2025-02-24 23:45:18 -- STEP: 18768/36938 -- GLOBAL_STEP: 535900 | > loss_text_ce: 0.04311300814151764 (0.036963557395303895) | > loss_mel_ce: 2.9567201137542725 (nan) | > loss: 0.011904099956154823 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.13665662004667165) | > loader_time: 0.0048 (0.0046979886847282335)  --> TIME: 2025-02-24 23:45:28 -- STEP: 18818/36938 -- GLOBAL_STEP: 535950 | > loss_text_ce: 0.051090214401483536 (0.03696473076192655) | > loss_mel_ce: 2.758420467376709 (nan) | > loss: 0.011148852296173573 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13665295308575587) | > loader_time: 0.0065 (0.004698176783828189)  --> TIME: 2025-02-24 23:45:37 -- STEP: 18868/36938 -- GLOBAL_STEP: 536000 | > loss_text_ce: 0.03743434697389603 (0.03696480853103395) | > loss_mel_ce: 2.342815399169922 (nan) | > loss: 0.009445436298847198 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.136651903725742) | > loader_time: 0.0046 (0.00469858447500072)  --> TIME: 2025-02-24 23:45:46 -- STEP: 18918/36938 -- GLOBAL_STEP: 536050 | > loss_text_ce: 0.03706168383359909 (0.03696716756969782) | > loss_mel_ce: 2.4367117881774902 (nan) | > loss: 0.009816561825573444 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.1366410211070495) | > loader_time: 0.0046 (0.004698586660530936)  --> TIME: 2025-02-24 23:45:55 -- STEP: 18968/36938 -- GLOBAL_STEP: 536100 | > loss_text_ce: 0.04070413485169411 (0.0369658371829751) | > loss_mel_ce: 2.613276720046997 (nan) | > loss: 0.010531671345233917 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.13664125775245123) | > loader_time: 0.0047 (0.004698486506109361)  --> TIME: 2025-02-24 23:46:05 -- STEP: 19018/36938 -- GLOBAL_STEP: 536150 | > loss_text_ce: 0.03577328473329544 (0.03696712355471913) | > loss_mel_ce: 1.7683274745941162 (nan) | > loss: 0.007159130182117224 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13663830930395138) | > loader_time: 0.0046 (0.004698377689085185)  --> TIME: 2025-02-24 23:46:14 -- STEP: 19068/36938 -- GLOBAL_STEP: 536200 | > loss_text_ce: 0.03206080570816994 (0.0369645606289563) | > loss_mel_ce: 3.2930450439453125 (nan) | > loss: 0.013194865547120571 (nan) | > current_lr: 5e-06 | > step_time: 0.1567 (0.1366332376030774) | > loader_time: 0.0047 (0.004698387301640884)  --> TIME: 2025-02-24 23:46:23 -- STEP: 19118/36938 -- GLOBAL_STEP: 536250 | > loss_text_ce: 0.03245161473751068 (0.03696649674922827) | > loss_mel_ce: 2.0972979068756104 (nan) | > loss: 0.008451387286186218 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13662276423642156) | > loader_time: 0.0046 (0.004698582243777227)  --> TIME: 2025-02-24 23:46:32 -- STEP: 19168/36938 -- GLOBAL_STEP: 536300 | > loss_text_ce: 0.034561216831207275 (0.03696949631105929) | > loss_mel_ce: 2.651116371154785 (nan) | > loss: 0.01065745111554861 (nan) | > current_lr: 5e-06 | > step_time: 0.1439 (0.136630757299807) | > loader_time: 0.0047 (0.004698496405190761)  --> TIME: 2025-02-24 23:46:42 -- STEP: 19218/36938 -- GLOBAL_STEP: 536350 | > loss_text_ce: 0.03680418059229851 (0.03697216782231563) | > loss_mel_ce: 2.947877883911133 (nan) | > loss: 0.011843977496027946 (nan) | > current_lr: 5e-06 | > step_time: 0.179 (0.1366357389976684) | > loader_time: 0.0048 (0.0046983064554623425)  --> TIME: 2025-02-24 23:46:51 -- STEP: 19268/36938 -- GLOBAL_STEP: 536400 | > loss_text_ce: 0.045418236404657364 (0.036970170146522846) | > loss_mel_ce: 2.8415234088897705 (nan) | > loss: 0.01145611796528101 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.13663678293483544) | > loader_time: 0.0047 (0.00469838773558344)  --> TIME: 2025-02-24 23:47:01 -- STEP: 19318/36938 -- GLOBAL_STEP: 536450 | > loss_text_ce: 0.03472461923956871 (0.03696892551714449) | > loss_mel_ce: 2.753119707107544 (nan) | > loss: 0.011062875390052795 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13663902342202242) | > loader_time: 0.0047 (0.004698737633501084)  --> TIME: 2025-02-24 23:47:10 -- STEP: 19368/36938 -- GLOBAL_STEP: 536500 | > loss_text_ce: 0.032726164907217026 (0.03696410491245487) | > loss_mel_ce: 3.4626119136810303 (nan) | > loss: 0.013870389200747013 (nan) | > current_lr: 5e-06 | > step_time: 0.2011 (0.136643699947737) | > loader_time: 0.005 (0.004698835451806817)  --> TIME: 2025-02-24 23:47:20 -- STEP: 19418/36938 -- GLOBAL_STEP: 536550 | > loss_text_ce: 0.033668383955955505 (0.03696707403148192) | > loss_mel_ce: 3.57710599899292 (nan) | > loss: 0.014328470453619957 (nan) | > current_lr: 5e-06 | > step_time: 0.1741 (0.1366554769407354) | > loader_time: 0.0048 (0.0046989441851114176)  --> TIME: 2025-02-24 23:47:29 -- STEP: 19468/36938 -- GLOBAL_STEP: 536600 | > loss_text_ce: 0.03965328261256218 (0.03696877083237664) | > loss_mel_ce: 2.7896788120269775 (nan) | > loss: 0.011227509006857872 (nan) | > current_lr: 5e-06 | > step_time: 0.1563 (0.13665583981413454) | > loader_time: 0.0046 (0.004698977385650881)  --> TIME: 2025-02-24 23:47:39 -- STEP: 19518/36938 -- GLOBAL_STEP: 536650 | > loss_text_ce: 0.040599316358566284 (0.03696493157353087) | > loss_mel_ce: 2.9598605632781982 (nan) | > loss: 0.011906587518751621 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13667510615861952) | > loader_time: 0.0046 (0.004699003575510033)  --> TIME: 2025-02-24 23:47:48 -- STEP: 19568/36938 -- GLOBAL_STEP: 536700 | > loss_text_ce: 0.03492336347699165 (0.03696011422676588) | > loss_mel_ce: 3.3037452697753906 (nan) | > loss: 0.013248685747385025 (nan) | > current_lr: 5e-06 | > step_time: 0.1725 (0.13666717522912403) | > loader_time: 0.0049 (0.004698838377622759)  --> TIME: 2025-02-24 23:47:57 -- STEP: 19618/36938 -- GLOBAL_STEP: 536750 | > loss_text_ce: 0.030235236510634422 (0.03695651011563119) | > loss_mel_ce: 3.121877908706665 (nan) | > loss: 0.012508386746048927 (nan) | > current_lr: 5e-06 | > step_time: 0.1813 (0.13667382918338503) | > loader_time: 0.0049 (0.004698667592843873)  --> TIME: 2025-02-24 23:48:07 -- STEP: 19668/36938 -- GLOBAL_STEP: 536800 | > loss_text_ce: 0.02991340495646 (0.036961724837478074) | > loss_mel_ce: 3.395383596420288 (nan) | > loss: 0.013592449016869068 (nan) | > current_lr: 5e-06 | > step_time: 0.1842 (0.13667626308914801) | > loader_time: 0.0048 (0.004698493651847256)  --> TIME: 2025-02-24 23:48:16 -- STEP: 19718/36938 -- GLOBAL_STEP: 536850 | > loss_text_ce: 0.036324985325336456 (0.03695835313886404) | > loss_mel_ce: 3.2350823879241943 (nan) | > loss: 0.012981776148080826 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13667351312769843) | > loader_time: 0.0048 (0.0046984895947424984)  --> TIME: 2025-02-24 23:48:25 -- STEP: 19768/36938 -- GLOBAL_STEP: 536900 | > loss_text_ce: 0.04024999216198921 (0.03695637623984682) | > loss_mel_ce: 3.71543550491333 (nan) | > loss: 0.01490351464599371 (nan) | > current_lr: 5e-06 | > step_time: 0.151 (0.1366670967378581) | > loader_time: 0.0047 (0.004698777345933709)  --> TIME: 2025-02-24 23:48:34 -- STEP: 19818/36938 -- GLOBAL_STEP: 536950 | > loss_text_ce: 0.03858816623687744 (0.036958259566879474) | > loss_mel_ce: 2.2910943031311035 (nan) | > loss: 0.009244771674275398 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1366673280521723) | > loader_time: 0.0048 (0.004699049353034143)  --> TIME: 2025-02-24 23:48:44 -- STEP: 19868/36938 -- GLOBAL_STEP: 537000 | > loss_text_ce: 0.02818997949361801 (0.03695820345164931) | > loss_mel_ce: 3.4736618995666504 (nan) | > loss: 0.013896238058805466 (nan) | > current_lr: 5e-06 | > step_time: 0.1794 (0.13667088031816793) | > loader_time: 0.0049 (0.004699440748369498)  --> TIME: 2025-02-24 23:48:53 -- STEP: 19918/36938 -- GLOBAL_STEP: 537050 | > loss_text_ce: 0.03069567307829857 (0.03695799248790158) | > loss_mel_ce: 3.3474619388580322 (nan) | > loss: 0.013405388221144676 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.1366790553773906) | > loader_time: 0.0047 (0.0046994134548833675)  --> TIME: 2025-02-24 23:49:03 -- STEP: 19968/36938 -- GLOBAL_STEP: 537100 | > loss_text_ce: 0.03461718186736107 (0.03695586231879119) | > loss_mel_ce: 3.2905778884887695 (nan) | > loss: 0.013195219449698925 (nan) | > current_lr: 5e-06 | > step_time: 0.1776 (0.13667303365535835) | > loader_time: 0.0049 (0.0046996650023338376)  --> TIME: 2025-02-24 23:49:12 -- STEP: 20018/36938 -- GLOBAL_STEP: 537150 | > loss_text_ce: 0.041024260222911835 (0.036954402338541324) | > loss_mel_ce: 2.9795114994049072 (nan) | > loss: 0.011986253783106804 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.13667314801019362) | > loader_time: 0.0048 (0.004699736044521015)  --> TIME: 2025-02-24 23:49:21 -- STEP: 20068/36938 -- GLOBAL_STEP: 537200 | > loss_text_ce: 0.03852534666657448 (0.03695207481999817) | > loss_mel_ce: 2.5524444580078125 (nan) | > loss: 0.01028162706643343 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1366681483534671) | > loader_time: 0.0047 (0.004700080626557899)  --> TIME: 2025-02-24 23:49:30 -- STEP: 20118/36938 -- GLOBAL_STEP: 537250 | > loss_text_ce: 0.03827564790844917 (0.03695356914605432) | > loss_mel_ce: 2.6589787006378174 (nan) | > loss: 0.010703391395509243 (nan) | > current_lr: 5e-06 | > step_time: 0.1855 (0.13665360704735424) | > loader_time: 0.0049 (0.004700052180007241)  --> TIME: 2025-02-24 23:49:40 -- STEP: 20168/36938 -- GLOBAL_STEP: 537300 | > loss_text_ce: 0.03601894527673721 (0.03695563577204739) | > loss_mel_ce: 3.393484354019165 (nan) | > loss: 0.013609140180051327 (nan) | > current_lr: 5e-06 | > step_time: 0.2037 (0.13666484061331546) | > loader_time: 0.005 (0.00470004964565202)  --> TIME: 2025-02-24 23:49:49 -- STEP: 20218/36938 -- GLOBAL_STEP: 537350 | > loss_text_ce: 0.04200610890984535 (0.03695480604194004) | > loss_mel_ce: 2.77401065826416 (nan) | > loss: 0.011174670420587063 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13667548379672576) | > loader_time: 0.0046 (0.0046999027613705046)  --> TIME: 2025-02-24 23:49:58 -- STEP: 20268/36938 -- GLOBAL_STEP: 537400 | > loss_text_ce: 0.03867315128445625 (0.03695845753607885) | > loss_mel_ce: 2.810399055480957 (nan) | > loss: 0.01130584254860878 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.1366696774500793) | > loader_time: 0.0049 (0.004699880786965036)  --> TIME: 2025-02-24 23:50:08 -- STEP: 20318/36938 -- GLOBAL_STEP: 537450 | > loss_text_ce: 0.022763574495911598 (0.03695531046222975) | > loss_mel_ce: 2.4344711303710938 (nan) | > loss: 0.009750931523740292 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13667541136245487) | > loader_time: 0.0045 (0.004700240099426584)  --> TIME: 2025-02-24 23:50:17 -- STEP: 20368/36938 -- GLOBAL_STEP: 537500 | > loss_text_ce: 0.03745220974087715 (0.03695549487776623) | > loss_mel_ce: 3.694596767425537 (nan) | > loss: 0.014809719286859035 (nan) | > current_lr: 5e-06 | > step_time: 0.1769 (0.1366658589780001) | > loader_time: 0.0049 (0.004700319921979751)  --> TIME: 2025-02-24 23:50:27 -- STEP: 20418/36938 -- GLOBAL_STEP: 537550 | > loss_text_ce: 0.04359930008649826 (0.03695508077183632) | > loss_mel_ce: 2.5603058338165283 (nan) | > loss: 0.010332957841455936 (nan) | > current_lr: 5e-06 | > step_time: 0.1528 (0.13667380904366658) | > loader_time: 0.0046 (0.004700494648625813)  --> TIME: 2025-02-24 23:50:36 -- STEP: 20468/36938 -- GLOBAL_STEP: 537600 | > loss_text_ce: 0.04237870126962662 (0.036956077112734416) | > loss_mel_ce: 3.1398870944976807 (nan) | > loss: 0.012628039345145226 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1366615557032261) | > loader_time: 0.0045 (0.004700332652877195)  --> TIME: 2025-02-24 23:50:45 -- STEP: 20518/36938 -- GLOBAL_STEP: 537650 | > loss_text_ce: 0.02593364007771015 (0.03696073203152703) | > loss_mel_ce: 3.1280696392059326 (nan) | > loss: 0.01251588761806488 (nan) | > current_lr: 5e-06 | > step_time: 0.1439 (0.1366644767957745) | > loader_time: 0.0046 (0.004700437253510866)  --> TIME: 2025-02-24 23:50:54 -- STEP: 20568/36938 -- GLOBAL_STEP: 537700 | > loss_text_ce: 0.037370603531599045 (0.0369600727633006) | > loss_mel_ce: 1.0062408447265625 (nan) | > loss: 0.0041413153521716595 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13666986324869645) | > loader_time: 0.0053 (0.004700617943696619)  --> TIME: 2025-02-24 23:51:04 -- STEP: 20618/36938 -- GLOBAL_STEP: 537750 | > loss_text_ce: 0.04109273478388786 (0.036960724774601196) | > loss_mel_ce: 3.266634225845337 (nan) | > loss: 0.013125901110470295 (nan) | > current_lr: 5e-06 | > step_time: 0.1888 (0.1366715512043035) | > loader_time: 0.0061 (0.0047005408255873056)  --> TIME: 2025-02-24 23:51:13 -- STEP: 20668/36938 -- GLOBAL_STEP: 537800 | > loss_text_ce: 0.04059059917926788 (0.03696221850859463) | > loss_mel_ce: 3.141352415084839 (nan) | > loss: 0.01262675877660513 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13667584982512634) | > loader_time: 0.0046 (0.004700471786412666)  --> TIME: 2025-02-24 23:51:23 -- STEP: 20718/36938 -- GLOBAL_STEP: 537850 | > loss_text_ce: 0.03697716444730759 (0.03696015291396939) | > loss_mel_ce: 3.259822130203247 (nan) | > loss: 0.013082536868751049 (nan) | > current_lr: 5e-06 | > step_time: 0.1572 (0.13669043328842825) | > loader_time: 0.0049 (0.004700611728374004)  --> TIME: 2025-02-24 23:51:32 -- STEP: 20768/36938 -- GLOBAL_STEP: 537900 | > loss_text_ce: 0.03814931586384773 (0.03695930378831356) | > loss_mel_ce: 3.4789769649505615 (nan) | > loss: 0.013956851325929165 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.13668623339321634) | > loader_time: 0.0047 (0.004700575557713906)  --> TIME: 2025-02-24 23:51:41 -- STEP: 20818/36938 -- GLOBAL_STEP: 537950 | > loss_text_ce: 0.03633490577340126 (0.03695807124380936) | > loss_mel_ce: 2.6375234127044678 (nan) | > loss: 0.010610549710690975 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13668063727038093) | > loader_time: 0.0048 (0.0047007391896918315)  --> TIME: 2025-02-24 23:51:50 -- STEP: 20868/36938 -- GLOBAL_STEP: 538000 | > loss_text_ce: 0.02637089416384697 (0.036954506303558204) | > loss_mel_ce: 1.369102120399475 (nan) | > loss: 0.0055375914089381695 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13667283101353292) | > loader_time: 0.0045 (0.004700670691086755)  --> TIME: 2025-02-24 23:52:00 -- STEP: 20918/36938 -- GLOBAL_STEP: 538050 | > loss_text_ce: 0.03780008852481842 (0.036951836813558014) | > loss_mel_ce: 3.0564801692962646 (nan) | > loss: 0.012278890237212181 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.1366678915261788) | > loader_time: 0.0046 (0.004700598564917551)  --> TIME: 2025-02-24 23:52:09 -- STEP: 20968/36938 -- GLOBAL_STEP: 538100 | > loss_text_ce: 0.03326726704835892 (0.03695212917000663) | > loss_mel_ce: 3.1205248832702637 (nan) | > loss: 0.012515048496425152 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13666370500821456) | > loader_time: 0.0047 (0.0047004858599688105)  --> TIME: 2025-02-24 23:52:18 -- STEP: 21018/36938 -- GLOBAL_STEP: 538150 | > loss_text_ce: 0.03242828696966171 (0.036956811966577255) | > loss_mel_ce: 2.8514366149902344 (nan) | > loss: 0.011443909257650375 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.1366650708815908) | > loader_time: 0.0046 (0.004700381575012808)  --> TIME: 2025-02-24 23:52:28 -- STEP: 21068/36938 -- GLOBAL_STEP: 538200 | > loss_text_ce: 0.039063919335603714 (0.036954566979715504) | > loss_mel_ce: 3.49884295463562 (nan) | > loss: 0.014039313420653343 (nan) | > current_lr: 5e-06 | > step_time: 0.1915 (0.13667223301652673) | > loader_time: 0.005 (0.004700376465987384)  --> TIME: 2025-02-24 23:52:37 -- STEP: 21118/36938 -- GLOBAL_STEP: 538250 | > loss_text_ce: 0.03989562764763832 (0.03694744078648473) | > loss_mel_ce: 3.335080146789551 (nan) | > loss: 0.01339276134967804 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13666929183939733) | > loader_time: 0.0046 (0.004700155847056136)  --> TIME: 2025-02-24 23:52:46 -- STEP: 21168/36938 -- GLOBAL_STEP: 538300 | > loss_text_ce: 0.03358730301260948 (0.03694653005353438) | > loss_mel_ce: 3.33784818649292 (nan) | > loss: 0.013378712348639965 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13667843510549546) | > loader_time: 0.0047 (0.0046999964606825205)  --> TIME: 2025-02-24 23:52:56 -- STEP: 21218/36938 -- GLOBAL_STEP: 538350 | > loss_text_ce: 0.033275604248046875 (0.03694637142844602) | > loss_mel_ce: 2.822640895843506 (nan) | > loss: 0.011333002708852291 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13668034021406528) | > loader_time: 0.0045 (0.004699897087421521)  --> TIME: 2025-02-24 23:53:05 -- STEP: 21268/36938 -- GLOBAL_STEP: 538400 | > loss_text_ce: 0.044616442173719406 (0.03694374411165781) | > loss_mel_ce: 3.233644723892212 (nan) | > loss: 0.013008973561227322 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1366880639437472) | > loader_time: 0.0045 (0.00469983599341543)  --> TIME: 2025-02-24 23:53:15 -- STEP: 21318/36938 -- GLOBAL_STEP: 538450 | > loss_text_ce: 0.02886885032057762 (0.03694655096003891) | > loss_mel_ce: 1.355508804321289 (nan) | > loss: 0.005493562202900648 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13669369729377853) | > loader_time: 0.0046 (0.0046997191434238195)  --> TIME: 2025-02-24 23:53:24 -- STEP: 21368/36938 -- GLOBAL_STEP: 538500 | > loss_text_ce: 0.03034588322043419 (0.03694679380331916) | > loss_mel_ce: 2.12748646736145 (nan) | > loss: 0.008562827482819557 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13668841449386943) | > loader_time: 0.0046 (0.004699536161626258)  --> TIME: 2025-02-24 23:53:33 -- STEP: 21418/36938 -- GLOBAL_STEP: 538550 | > loss_text_ce: 0.036326080560684204 (0.03694857628856881) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1366827227797916) | > loader_time: 0.0046 (0.0046993559933430195)  --> TIME: 2025-02-24 23:53:42 -- STEP: 21468/36938 -- GLOBAL_STEP: 538600 | > loss_text_ce: 0.03492377698421478 (0.036952620048373575) | > loss_mel_ce: 3.032656192779541 (nan) | > loss: 0.012172937393188477 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1366775921765089) | > loader_time: 0.0046 (0.00469918935819115)  --> TIME: 2025-02-24 23:53:51 -- STEP: 21518/36938 -- GLOBAL_STEP: 538650 | > loss_text_ce: 0.02959553338587284 (0.036955545429763155) | > loss_mel_ce: 2.416013479232788 (nan) | > loss: 0.009704798460006714 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13666923031264924) | > loader_time: 0.0047 (0.004699226681744495)  --> TIME: 2025-02-24 23:54:01 -- STEP: 21568/36938 -- GLOBAL_STEP: 538700 | > loss_text_ce: 0.03435396030545235 (0.036953259301002184) | > loss_mel_ce: 1.277685523033142 (nan) | > loss: 0.005206506233662367 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.1366671884449017) | > loader_time: 0.0051 (0.004699432011970095)  --> TIME: 2025-02-24 23:54:10 -- STEP: 21618/36938 -- GLOBAL_STEP: 538750 | > loss_text_ce: 0.04009528458118439 (0.03694685979059824) | > loss_mel_ce: 2.9004783630371094 (nan) | > loss: 0.011668943800032139 (nan) | > current_lr: 5e-06 | > step_time: 0.1577 (0.1366641861689457) | > loader_time: 0.0049 (0.004699351697352344)  --> TIME: 2025-02-24 23:54:19 -- STEP: 21668/36938 -- GLOBAL_STEP: 538800 | > loss_text_ce: 0.03345106169581413 (0.036945967532375196) | > loss_mel_ce: 2.888150453567505 (nan) | > loss: 0.011593657545745373 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.13666751409162367) | > loader_time: 0.0048 (0.004699337101739716)  --> TIME: 2025-02-24 23:54:29 -- STEP: 21718/36938 -- GLOBAL_STEP: 538850 | > loss_text_ce: 0.032615549862384796 (0.03694700045717474) | > loss_mel_ce: 0.7170767784118652 (nan) | > loss: 0.0029749698005616665 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.13667683838941724) | > loader_time: 0.0046 (0.004699484091545429)  --> TIME: 2025-02-24 23:54:38 -- STEP: 21768/36938 -- GLOBAL_STEP: 538900 | > loss_text_ce: 0.03647911548614502 (0.036946331858819716) | > loss_mel_ce: 3.5074405670166016 (nan) | > loss: 0.014063173905014992 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13667341347171866) | > loader_time: 0.0045 (0.00469936738773027)  --> TIME: 2025-02-24 23:54:47 -- STEP: 21818/36938 -- GLOBAL_STEP: 538950 | > loss_text_ce: 0.046926338225603104 (0.036945922870504105) | > loss_mel_ce: 3.269939422607422 (nan) | > loss: 0.013162165880203247 (nan) | > current_lr: 5e-06 | > step_time: 0.1353 (0.13667629094901504) | > loader_time: 0.0046 (0.0046997358801384986)  --> TIME: 2025-02-24 23:54:57 -- STEP: 21868/36938 -- GLOBAL_STEP: 539000 | > loss_text_ce: 0.02254580147564411 (0.03694452832420414) | > loss_mel_ce: 2.322740316390991 (nan) | > loss: 0.009306691586971283 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13668023631758974) | > loader_time: 0.0045 (0.004700063416219549)  --> TIME: 2025-02-24 23:55:06 -- STEP: 21918/36938 -- GLOBAL_STEP: 539050 | > loss_text_ce: 0.03894525766372681 (0.03694497838698633) | > loss_mel_ce: 3.117511749267578 (nan) | > loss: 0.01252562366425991 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13667820663671484) | > loader_time: 0.0046 (0.004700071784046552)  --> TIME: 2025-02-24 23:55:16 -- STEP: 21968/36938 -- GLOBAL_STEP: 539100 | > loss_text_ce: 0.03688512369990349 (0.03694529745136624) | > loss_mel_ce: 3.223721742630005 (nan) | > loss: 0.012938916683197021 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13668966671019214) | > loader_time: 0.0047 (0.004700300245066256)  --> TIME: 2025-02-24 23:55:25 -- STEP: 22018/36938 -- GLOBAL_STEP: 539150 | > loss_text_ce: 0.04037531837821007 (0.036945615455952124) | > loss_mel_ce: 2.209705352783203 (nan) | > loss: 0.008928891271352768 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13668872437540833) | > loader_time: 0.0046 (0.004700291697103412)  --> TIME: 2025-02-24 23:55:34 -- STEP: 22068/36938 -- GLOBAL_STEP: 539200 | > loss_text_ce: 0.025189485400915146 (0.03694272264934434) | > loss_mel_ce: 2.5739738941192627 (nan) | > loss: 0.01031414046883583 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13669072178280092) | > loader_time: 0.0047 (0.004700367349590424)  --> TIME: 2025-02-24 23:55:44 -- STEP: 22118/36938 -- GLOBAL_STEP: 539250 | > loss_text_ce: 0.03144317865371704 (0.0369450052810551) | > loss_mel_ce: 2.9948995113372803 (nan) | > loss: 0.01200929656624794 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13670671569340653) | > loader_time: 0.0045 (0.004700373618035128)  --> TIME: 2025-02-24 23:55:53 -- STEP: 22168/36938 -- GLOBAL_STEP: 539300 | > loss_text_ce: 0.026814498007297516 (0.03694266502119296) | > loss_mel_ce: 3.0567708015441895 (nan) | > loss: 0.012236449867486954 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13670065804364195) | > loader_time: 0.0045 (0.004700201141064217)  --> TIME: 2025-02-24 23:56:03 -- STEP: 22218/36938 -- GLOBAL_STEP: 539350 | > loss_text_ce: 0.03559749945998192 (0.03694132049512239) | > loss_mel_ce: 2.1430602073669434 (nan) | > loss: 0.008645467460155487 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.1367099203785704) | > loader_time: 0.0046 (0.004700098214564564)  --> TIME: 2025-02-24 23:56:12 -- STEP: 22268/36938 -- GLOBAL_STEP: 539400 | > loss_text_ce: 0.04406876862049103 (0.03693965899714367) | > loss_mel_ce: 3.0597589015960693 (nan) | > loss: 0.012316777370870113 (nan) | > current_lr: 5e-06 | > step_time: 0.1911 (0.13672446290230866) | > loader_time: 0.0048 (0.004700003427043321)  --> TIME: 2025-02-24 23:56:22 -- STEP: 22318/36938 -- GLOBAL_STEP: 539450 | > loss_text_ce: 0.04072216525673866 (0.03693961031601064) | > loss_mel_ce: 2.933701753616333 (nan) | > loss: 0.011803270317614079 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.1367173890014613) | > loader_time: 0.0047 (0.004699959828868792)  --> TIME: 2025-02-24 23:56:31 -- STEP: 22368/36938 -- GLOBAL_STEP: 539500 | > loss_text_ce: 0.03963571786880493 (0.03693898718859735) | > loss_mel_ce: 0.02423095703125 (nan) | > loss: 0.00025343921151943505 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.13671409707511153) | > loader_time: 0.0054 (0.004699904967275316)  --> TIME: 2025-02-24 23:56:40 -- STEP: 22418/36938 -- GLOBAL_STEP: 539550 | > loss_text_ce: 0.035728342831134796 (0.036938046395678194) | > loss_mel_ce: 2.603379964828491 (nan) | > loss: 0.010472653433680534 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13671643701647504) | > loader_time: 0.0045 (0.004699859666786065)  --> TIME: 2025-02-24 23:56:49 -- STEP: 22468/36938 -- GLOBAL_STEP: 539600 | > loss_text_ce: 0.04458499699831009 (0.03694297667672008) | > loss_mel_ce: 2.7401299476623535 (nan) | > loss: 0.011050457134842873 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13670429133723788) | > loader_time: 0.0045 (0.0046998039033885435)  --> TIME: 2025-02-24 23:56:58 -- STEP: 22518/36938 -- GLOBAL_STEP: 539650 | > loss_text_ce: 0.041235148906707764 (0.036942495406037855) | > loss_mel_ce: 3.188481092453003 (nan) | > loss: 0.012816335074603558 (nan) | > current_lr: 5e-06 | > step_time: 0.1855 (0.13670345187684896) | > loader_time: 0.0054 (0.004699899773581206)  --> TIME: 2025-02-24 23:57:08 -- STEP: 22568/36938 -- GLOBAL_STEP: 539700 | > loss_text_ce: 0.036961693316698074 (0.03694751959354155) | > loss_mel_ce: 2.8491768836975098 (nan) | > loss: 0.01145293191075325 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.13671926504878604) | > loader_time: 0.0048 (0.004700329214154897)  --> TIME: 2025-02-24 23:57:17 -- STEP: 22618/36938 -- GLOBAL_STEP: 539750 | > loss_text_ce: 0.024571184068918228 (0.036946640489758736) | > loss_mel_ce: 2.674389600753784 (nan) | > loss: 0.010710162110626698 (nan) | > current_lr: 5e-06 | > step_time: 0.1079 (0.13671803926639484) | > loader_time: 0.0045 (0.004700258952656988)  --> TIME: 2025-02-24 23:57:27 -- STEP: 22668/36938 -- GLOBAL_STEP: 539800 | > loss_text_ce: 0.0447566881775856 (0.036944315094773195) | > loss_mel_ce: 3.292863368988037 (nan) | > loss: 0.013244524598121643 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.13672037375589705) | > loader_time: 0.0047 (0.00470030037461081)  --> TIME: 2025-02-24 23:57:36 -- STEP: 22718/36938 -- GLOBAL_STEP: 539850 | > loss_text_ce: 0.033351216465234756 (0.036941196080310296) | > loss_mel_ce: 2.7536466121673584 (nan) | > loss: 0.011059516109526157 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13672026686186128) | > loader_time: 0.0046 (0.004700205718372186)  --> TIME: 2025-02-24 23:57:46 -- STEP: 22768/36938 -- GLOBAL_STEP: 539900 | > loss_text_ce: 0.04853116348385811 (0.03694012678835832) | > loss_mel_ce: 0.2060546875 (nan) | > loss: 0.0010102613596245646 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.1367225306015665) | > loader_time: 0.0046 (0.004700113184755318)  --> TIME: 2025-02-24 23:57:55 -- STEP: 22818/36938 -- GLOBAL_STEP: 539950 | > loss_text_ce: 0.026226896792650223 (0.03693784383698727) | > loss_mel_ce: 3.5836877822875977 (nan) | > loss: 0.01432505901902914 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13671765442180475) | > loader_time: 0.0046 (0.004700238274821621)  --> TIME: 2025-02-24 23:58:04 -- STEP: 22868/36938 -- GLOBAL_STEP: 540000 | > loss_text_ce: 0.034354254603385925 (0.03693883399551092) | > loss_mel_ce: 2.387831449508667 (nan) | > loss: 0.00961184874176979 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.13672409579744357) | > loader_time: 0.0046 (0.0047001911456588654) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_540000.pth  --> TIME: 2025-02-24 23:58:20 -- STEP: 22918/36938 -- GLOBAL_STEP: 540050 | > loss_text_ce: 0.022922465577721596 (0.03693630154446892) | > loss_mel_ce: 3.1794073581695557 (nan) | > loss: 0.012707659043371677 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.13671985294860767) | > loader_time: 0.0046 (0.004700257803343257)  --> TIME: 2025-02-24 23:58:29 -- STEP: 22968/36938 -- GLOBAL_STEP: 540100 | > loss_text_ce: 0.03956156224012375 (0.03693410986714102) | > loss_mel_ce: 3.410282850265503 (nan) | > loss: 0.0136898597702384 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.13671791898128904) | > loader_time: 0.0057 (0.004700378730545383)  --> TIME: 2025-02-24 23:58:38 -- STEP: 23018/36938 -- GLOBAL_STEP: 540150 | > loss_text_ce: 0.027517126873135567 (0.036932911153883356) | > loss_mel_ce: 2.199373245239258 (nan) | > loss: 0.008836867287755013 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13671859123381855) | > loader_time: 0.0046 (0.004700495900717496)  --> TIME: 2025-02-24 23:58:47 -- STEP: 23068/36938 -- GLOBAL_STEP: 540200 | > loss_text_ce: 0.04094685986638069 (0.03693017256329907) | > loss_mel_ce: 3.387338638305664 (nan) | > loss: 0.013604308478534222 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13671175082097498) | > loader_time: 0.0047 (0.004700459112260287)  --> TIME: 2025-02-24 23:58:57 -- STEP: 23118/36938 -- GLOBAL_STEP: 540250 | > loss_text_ce: 0.046782586723566055 (0.0369298545921577) | > loss_mel_ce: 2.168443441390991 (nan) | > loss: 0.008790579624474049 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13671606955968513) | > loader_time: 0.0046 (0.004700554738315761)  --> TIME: 2025-02-24 23:59:06 -- STEP: 23168/36938 -- GLOBAL_STEP: 540300 | > loss_text_ce: 0.037504784762859344 (0.03692910237602334) | > loss_mel_ce: 3.336757183074951 (nan) | > loss: 0.013389929197728634 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13670802697827578) | > loader_time: 0.0054 (0.0047005638583095)  --> TIME: 2025-02-24 23:59:15 -- STEP: 23218/36938 -- GLOBAL_STEP: 540350 | > loss_text_ce: 0.034866973757743835 (0.03692953839851744) | > loss_mel_ce: 2.4641454219818115 (nan) | > loss: 0.009916717186570168 (nan) | > current_lr: 5e-06 | > step_time: 0.1778 (0.13670402860259595) | > loader_time: 0.0049 (0.0047007363961231495)  --> TIME: 2025-02-24 23:59:24 -- STEP: 23268/36938 -- GLOBAL_STEP: 540400 | > loss_text_ce: 0.029657326638698578 (0.03693050137013024) | > loss_mel_ce: 3.4966647624969482 (nan) | > loss: 0.013993342407047749 (nan) | > current_lr: 5e-06 | > step_time: 0.1628 (0.13669789064908142) | > loader_time: 0.0046 (0.004700638050268211)  --> TIME: 2025-02-24 23:59:34 -- STEP: 23318/36938 -- GLOBAL_STEP: 540450 | > loss_text_ce: 0.04413435980677605 (0.036931306724392265) | > loss_mel_ce: 2.7288448810577393 (nan) | > loss: 0.011003886349499226 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13670445769675157) | > loader_time: 0.0045 (0.0047007155506328035)  --> TIME: 2025-02-24 23:59:43 -- STEP: 23368/36938 -- GLOBAL_STEP: 540500 | > loss_text_ce: 0.03427719697356224 (0.0369292526988208) | > loss_mel_ce: 3.1223576068878174 (nan) | > loss: 0.012526329606771469 (nan) | > current_lr: 5e-06 | > step_time: 0.1994 (0.13670013967910483) | > loader_time: 0.0049 (0.00470072010615556)  --> TIME: 2025-02-24 23:59:52 -- STEP: 23418/36938 -- GLOBAL_STEP: 540550 | > loss_text_ce: 0.02790728770196438 (0.03693109763541537) | > loss_mel_ce: 2.2447102069854736 (nan) | > loss: 0.009018324315547943 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13668589499740236) | > loader_time: 0.0047 (0.004700765478202729)  --> TIME: 2025-02-25 00:00:01 -- STEP: 23468/36938 -- GLOBAL_STEP: 540600 | > loss_text_ce: 0.03560886159539223 (0.0369303775080979) | > loss_mel_ce: 3.3387441635131836 (nan) | > loss: 0.013390290550887585 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13668849479609688) | > loader_time: 0.0046 (0.0047010015604218355)  --> TIME: 2025-02-25 00:00:11 -- STEP: 23518/36938 -- GLOBAL_STEP: 540650 | > loss_text_ce: 0.04036330059170723 (0.036930131053829106) | > loss_mel_ce: 3.697000741958618 (nan) | > loss: 0.014830810949206352 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13669525380746556) | > loader_time: 0.0046 (0.0047011127661701615)  --> TIME: 2025-02-25 00:00:20 -- STEP: 23568/36938 -- GLOBAL_STEP: 540700 | > loss_text_ce: 0.03509724885225296 (0.036929408405682713) | > loss_mel_ce: 1.77295982837677 (nan) | > loss: 0.007174829952418804 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.136704193791078) | > loader_time: 0.0045 (0.004701259169785727)  --> TIME: 2025-02-25 00:00:30 -- STEP: 23618/36938 -- GLOBAL_STEP: 540750 | > loss_text_ce: 0.03828755393624306 (0.036927573922491595) | > loss_mel_ce: 2.2044196128845215 (nan) | > loss: 0.008899632841348648 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13671472297916934) | > loader_time: 0.0046 (0.004701225983116316)  --> TIME: 2025-02-25 00:00:39 -- STEP: 23668/36938 -- GLOBAL_STEP: 540800 | > loss_text_ce: 0.03911292552947998 (0.03692834113747514) | > loss_mel_ce: 3.46805477142334 (nan) | > loss: 0.013917333446443081 (nan) | > current_lr: 5e-06 | > step_time: 0.1619 (0.1367110195485492) | > loader_time: 0.0049 (0.004701113708925457)  --> TIME: 2025-02-25 00:00:49 -- STEP: 23718/36938 -- GLOBAL_STEP: 540850 | > loss_text_ce: 0.03881499171257019 (0.03692699646307404) | > loss_mel_ce: 1.8830077648162842 (nan) | > loss: 0.0076262811198830605 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13672271772222758) | > loader_time: 0.0045 (0.004701305696278491)  --> TIME: 2025-02-25 00:00:58 -- STEP: 23768/36938 -- GLOBAL_STEP: 540900 | > loss_text_ce: 0.040929485112428665 (0.03692348686835125) | > loss_mel_ce: 2.7047548294067383 (nan) | > loss: 0.010895573534071445 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13672081417445248) | > loader_time: 0.0046 (0.004701484196597727)  --> TIME: 2025-02-25 00:01:07 -- STEP: 23818/36938 -- GLOBAL_STEP: 540950 | > loss_text_ce: 0.04700692370533943 (0.03692577231577009) | > loss_mel_ce: 2.298734188079834 (nan) | > loss: 0.009308496490120888 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13671382410078498) | > loader_time: 0.0046 (0.004701537512964126)  --> TIME: 2025-02-25 00:01:16 -- STEP: 23868/36938 -- GLOBAL_STEP: 541000 | > loss_text_ce: 0.03616805374622345 (0.0369230594515419) | > loss_mel_ce: 2.4373223781585693 (nan) | > loss: 0.009815438650548458 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13671007613071384) | > loader_time: 0.0048 (0.004701490166080586)  --> TIME: 2025-02-25 00:01:26 -- STEP: 23918/36938 -- GLOBAL_STEP: 541050 | > loss_text_ce: 0.03492729365825653 (0.03692169019412579) | > loss_mel_ce: 3.151010036468506 (nan) | > loss: 0.012642609886825085 (nan) | > current_lr: 5e-06 | > step_time: 0.161 (0.13670697475578236) | > loader_time: 0.0047 (0.004701524925566848)  --> TIME: 2025-02-25 00:01:35 -- STEP: 23968/36938 -- GLOBAL_STEP: 541100 | > loss_text_ce: 0.03537001833319664 (0.03692151647593762) | > loss_mel_ce: 3.362675428390503 (nan) | > loss: 0.013484308496117592 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.13670164289995007) | > loader_time: 0.0046 (0.0047017190262217574)  --> TIME: 2025-02-25 00:01:44 -- STEP: 24018/36938 -- GLOBAL_STEP: 541150 | > loss_text_ce: 0.03125561773777008 (0.0369191943576783) | > loss_mel_ce: 0.9097900390625 (nan) | > loss: 0.0037343082949519157 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.1367019657707904) | > loader_time: 0.0067 (0.004701783063658076)  --> TIME: 2025-02-25 00:01:54 -- STEP: 24068/36938 -- GLOBAL_STEP: 541200 | > loss_text_ce: 0.03715576231479645 (0.03691800953811831) | > loss_mel_ce: 1.6698194742202759 (nan) | > loss: 0.006773711647838354 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13670784770917466) | > loader_time: 0.0047 (0.004701988114973054)  --> TIME: 2025-02-25 00:02:03 -- STEP: 24118/36938 -- GLOBAL_STEP: 541250 | > loss_text_ce: 0.044427573680877686 (0.03691589196841783) | > loss_mel_ce: 3.1529438495635986 (nan) | > loss: 0.012687982991337776 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13669847543485414) | > loader_time: 0.0046 (0.004702017995114833)  --> TIME: 2025-02-25 00:02:12 -- STEP: 24168/36938 -- GLOBAL_STEP: 541300 | > loss_text_ce: 0.04299101606011391 (0.03691300629667883) | > loss_mel_ce: 1.4843882322311401 (nan) | > loss: 0.006061029154807329 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13670475056375883) | > loader_time: 0.0053 (0.004702138884972743)  --> TIME: 2025-02-25 00:02:21 -- STEP: 24218/36938 -- GLOBAL_STEP: 541350 | > loss_text_ce: 0.037583332508802414 (0.03691234807129672) | > loss_mel_ce: 3.268281936645508 (nan) | > loss: 0.013118513859808445 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13669771767733754) | > loader_time: 0.0046 (0.004702021851391143)  --> TIME: 2025-02-25 00:02:31 -- STEP: 24268/36938 -- GLOBAL_STEP: 541400 | > loss_text_ce: 0.03375726938247681 (0.03691535091841352) | > loss_mel_ce: 2.9256508350372314 (nan) | > loss: 0.011743683367967606 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.13670627821684525) | > loader_time: 0.0047 (0.004702219828282934)  --> TIME: 2025-02-25 00:02:40 -- STEP: 24318/36938 -- GLOBAL_STEP: 541450 | > loss_text_ce: 0.041092582046985626 (0.03691572615020763) | > loss_mel_ce: 2.1001081466674805 (nan) | > loss: 0.008496829308569431 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13670347219078652) | > loader_time: 0.0046 (0.004702228985684549)  --> TIME: 2025-02-25 00:02:49 -- STEP: 24368/36938 -- GLOBAL_STEP: 541500 | > loss_text_ce: 0.037247251719236374 (0.03691283753069144) | > loss_mel_ce: 3.368858814239502 (nan) | > loss: 0.01351629476994276 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.136704519019265) | > loader_time: 0.0047 (0.00470253171610816)  --> TIME: 2025-02-25 00:02:59 -- STEP: 24418/36938 -- GLOBAL_STEP: 541550 | > loss_text_ce: 0.04341711103916168 (0.03691244774198794) | > loss_mel_ce: 2.7051262855529785 (nan) | > loss: 0.010906918905675411 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13671407029575863) | > loader_time: 0.0047 (0.004702644204027143)  --> TIME: 2025-02-25 00:03:08 -- STEP: 24468/36938 -- GLOBAL_STEP: 541600 | > loss_text_ce: 0.03523974493145943 (0.03691263390882063) | > loss_mel_ce: 1.5302079916000366 (nan) | > loss: 0.006212094333022833 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13670904185816007) | > loader_time: 0.0045 (0.004702669090746813)  --> TIME: 2025-02-25 00:03:17 -- STEP: 24518/36938 -- GLOBAL_STEP: 541650 | > loss_text_ce: 0.036034587770700455 (0.03691287102924989) | > loss_mel_ce: 0.01934305764734745 (nan) | > loss: 0.00021975257550366223 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.136704232809604) | > loader_time: 0.0045 (0.004702721434419794)  --> TIME: 2025-02-25 00:03:27 -- STEP: 24568/36938 -- GLOBAL_STEP: 541700 | > loss_text_ce: 0.03326960653066635 (0.036913842360658304) | > loss_mel_ce: 3.455085515975952 (nan) | > loss: 0.013842679560184479 (nan) | > current_lr: 5e-06 | > step_time: 0.1817 (0.1367012915938116) | > loader_time: 0.0049 (0.004702766616660423)  --> TIME: 2025-02-25 00:03:36 -- STEP: 24618/36938 -- GLOBAL_STEP: 541750 | > loss_text_ce: 0.03706441447138786 (0.03691578920506466) | > loss_mel_ce: 3.187617063522339 (nan) | > loss: 0.012796355411410332 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.13670195722785322) | > loader_time: 0.0046 (0.004702859448228297)  --> TIME: 2025-02-25 00:03:46 -- STEP: 24668/36938 -- GLOBAL_STEP: 541800 | > loss_text_ce: 0.03936891630291939 (0.03691953686938853) | > loss_mel_ce: 3.2482452392578125 (nan) | > loss: 0.013046088628470898 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.1367113960123694) | > loader_time: 0.0046 (0.004702831872649638)  --> TIME: 2025-02-25 00:03:55 -- STEP: 24718/36938 -- GLOBAL_STEP: 541850 | > loss_text_ce: 0.032247092574834824 (0.03691777184329644) | > loss_mel_ce: 0.1865452378988266 (nan) | > loss: 0.0008682236075401306 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13670559169231658) | > loader_time: 0.0046 (0.004702825300881976)  --> TIME: 2025-02-25 00:04:04 -- STEP: 24768/36938 -- GLOBAL_STEP: 541900 | > loss_text_ce: 0.036164939403533936 (0.036917707066673425) | > loss_mel_ce: 2.431553363800049 (nan) | > loss: 0.009792533703148365 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.13670344204095647) | > loader_time: 0.0048 (0.004702638237898342)  --> TIME: 2025-02-25 00:04:13 -- STEP: 24818/36938 -- GLOBAL_STEP: 541950 | > loss_text_ce: 0.040109895169734955 (0.036918460232538836) | > loss_mel_ce: 2.65985369682312 (nan) | > loss: 0.01071414165198803 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1366983644237395) | > loader_time: 0.0046 (0.004702493237376979)  --> TIME: 2025-02-25 00:04:23 -- STEP: 24868/36938 -- GLOBAL_STEP: 542000 | > loss_text_ce: 0.03971141576766968 (0.036917815481625565) | > loss_mel_ce: 2.170259952545166 (nan) | > loss: 0.008769728243350983 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13669983984401313) | > loader_time: 0.0046 (0.004702430226247898)  --> TIME: 2025-02-25 00:04:32 -- STEP: 24918/36938 -- GLOBAL_STEP: 542050 | > loss_text_ce: 0.04534875601530075 (0.03691704584048212) | > loss_mel_ce: 3.1575968265533447 (nan) | > loss: 0.012710101902484894 (nan) | > current_lr: 5e-06 | > step_time: 0.1355 (0.13670297477130378) | > loader_time: 0.0047 (0.004702485079405161)  --> TIME: 2025-02-25 00:04:41 -- STEP: 24968/36938 -- GLOBAL_STEP: 542100 | > loss_text_ce: 0.03799691051244736 (0.03691591850602935) | > loss_mel_ce: 2.487031936645508 (nan) | > loss: 0.010019957087934017 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13670455425910927) | > loader_time: 0.0046 (0.0047024537244312)  --> TIME: 2025-02-25 00:04:51 -- STEP: 25018/36938 -- GLOBAL_STEP: 542150 | > loss_text_ce: 0.04502689093351364 (0.03691698676133046) | > loss_mel_ce: 2.9405791759490967 (nan) | > loss: 0.011847643181681633 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13671568657189506) | > loader_time: 0.0047 (0.004702562688875835)  --> TIME: 2025-02-25 00:05:00 -- STEP: 25068/36938 -- GLOBAL_STEP: 542200 | > loss_text_ce: 0.03452056646347046 (0.03691573247538044) | > loss_mel_ce: 2.4253451824188232 (nan) | > loss: 0.009761372581124306 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13670446563798064) | > loader_time: 0.0045 (0.004702507469934451)  --> TIME: 2025-02-25 00:05:10 -- STEP: 25118/36938 -- GLOBAL_STEP: 542250 | > loss_text_ce: 0.043721165508031845 (0.03691825485389399) | > loss_mel_ce: 2.3429954051971436 (nan) | > loss: 0.009471097961068153 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13671261835785548) | > loader_time: 0.0046 (0.004702427915179305)  --> TIME: 2025-02-25 00:05:19 -- STEP: 25168/36938 -- GLOBAL_STEP: 542300 | > loss_text_ce: 0.041808027774095535 (0.03691364083891598) | > loss_mel_ce: 3.3591580390930176 (nan) | > loss: 0.013495897874236107 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13670276856619532) | > loader_time: 0.0046 (0.00470227834834311)  --> TIME: 2025-02-25 00:05:28 -- STEP: 25218/36938 -- GLOBAL_STEP: 542350 | > loss_text_ce: 0.03681027889251709 (0.03691412756681586) | > loss_mel_ce: 2.1655781269073486 (nan) | > loss: 0.008739636279642582 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13670557023078042) | > loader_time: 0.0048 (0.004702581365855927)  --> TIME: 2025-02-25 00:05:37 -- STEP: 25268/36938 -- GLOBAL_STEP: 542400 | > loss_text_ce: 0.029518086463212967 (0.03691280233484746) | > loss_mel_ce: 3.023890733718872 (nan) | > loss: 0.012116702273488045 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13670337911608615) | > loader_time: 0.0045 (0.004702568157990728)  --> TIME: 2025-02-25 00:05:46 -- STEP: 25318/36938 -- GLOBAL_STEP: 542450 | > loss_text_ce: 0.04211237281560898 (0.03691277453019325) | > loss_mel_ce: 2.3348546028137207 (nan) | > loss: 0.009432408958673477 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13670262512376144) | > loader_time: 0.0045 (0.004702503048928187)  --> TIME: 2025-02-25 00:05:56 -- STEP: 25368/36938 -- GLOBAL_STEP: 542500 | > loss_text_ce: 0.0449274517595768 (0.036914339380152865) | > loss_mel_ce: 3.432661533355713 (nan) | > loss: 0.013799956999719143 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.13670306438265967) | > loader_time: 0.0047 (0.0047029701176970725)  --> TIME: 2025-02-25 00:06:05 -- STEP: 25418/36938 -- GLOBAL_STEP: 542550 | > loss_text_ce: 0.04965036362409592 (0.036917873681140016) | > loss_mel_ce: 2.5589842796325684 (nan) | > loss: 0.010351725853979588 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.1366991751292871) | > loader_time: 0.0045 (0.00470298068586626)  --> TIME: 2025-02-25 00:06:14 -- STEP: 25468/36938 -- GLOBAL_STEP: 542600 | > loss_text_ce: 0.042513277381658554 (0.03691755493513938) | > loss_mel_ce: 1.7737759351730347 (nan) | > loss: 0.007207497023046017 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1366958670775677) | > loader_time: 0.0045 (0.004702830925005751)  --> TIME: 2025-02-25 00:06:24 -- STEP: 25518/36938 -- GLOBAL_STEP: 542650 | > loss_text_ce: 0.016047362238168716 (0.03691563502037877) | > loss_mel_ce: 0.004913330078125 (nan) | > loss: 8.317735773744062e-05 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13670192381292864) | > loader_time: 0.0045 (0.004702736305699027)  --> TIME: 2025-02-25 00:06:34 -- STEP: 25568/36938 -- GLOBAL_STEP: 542700 | > loss_text_ce: 0.03945279121398926 (0.03691418345834599) | > loss_mel_ce: 3.001929521560669 (nan) | > loss: 0.012068978510797024 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13671426048901053) | > loader_time: 0.0046 (0.004702656510028421)  --> TIME: 2025-02-25 00:06:43 -- STEP: 25618/36938 -- GLOBAL_STEP: 542750 | > loss_text_ce: 0.048967111855745316 (0.03691601570860249) | > loss_mel_ce: 0.03472900390625 (nan) | > loss: 0.00033212744165211916 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13670827832617793) | > loader_time: 0.0046 (0.004702626239574975)  --> TIME: 2025-02-25 00:06:52 -- STEP: 25668/36938 -- GLOBAL_STEP: 542800 | > loss_text_ce: 0.035898707807064056 (0.03691331973322315) | > loss_mel_ce: 0.0299072265625 (nan) | > loss: 0.00026113467174582183 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13670632255696075) | > loader_time: 0.0046 (0.00470277193793815)  --> TIME: 2025-02-25 00:07:01 -- STEP: 25718/36938 -- GLOBAL_STEP: 542850 | > loss_text_ce: 0.036759957671165466 (0.036914424841321215) | > loss_mel_ce: 2.8207170963287354 (nan) | > loss: 0.01133919507265091 (nan) | > current_lr: 5e-06 | > step_time: 0.1488 (0.13670675905100046) | > loader_time: 0.0047 (0.004702776241695545)  --> TIME: 2025-02-25 00:07:11 -- STEP: 25768/36938 -- GLOBAL_STEP: 542900 | > loss_text_ce: 0.04167284816503525 (0.03691662731023832) | > loss_mel_ce: 3.310302734375 (nan) | > loss: 0.013301491737365723 (nan) | > current_lr: 5e-06 | > step_time: 0.1648 (0.13671359786222032) | > loader_time: 0.0054 (0.004702993438170574)  --> TIME: 2025-02-25 00:07:20 -- STEP: 25818/36938 -- GLOBAL_STEP: 542950 | > loss_text_ce: 0.04680480808019638 (0.03691991645172118) | > loss_mel_ce: 3.052781343460083 (nan) | > loss: 0.012299946509301662 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.1367108147282291) | > loader_time: 0.0045 (0.004703111029472817)  --> TIME: 2025-02-25 00:07:29 -- STEP: 25868/36938 -- GLOBAL_STEP: 543000 | > loss_text_ce: 0.04112442582845688 (0.03691913756205343) | > loss_mel_ce: 2.5658411979675293 (nan) | > loss: 0.010345102287828922 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1367022054759846) | > loader_time: 0.0046 (0.004702990162367048)  --> TIME: 2025-02-25 00:07:38 -- STEP: 25918/36938 -- GLOBAL_STEP: 543050 | > loss_text_ce: 0.031923431903123856 (0.03691788393484069) | > loss_mel_ce: 3.2084195613861084 (nan) | > loss: 0.012858505360782146 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13670035634422495) | > loader_time: 0.0047 (0.004702866790342284)  --> TIME: 2025-02-25 00:07:48 -- STEP: 25968/36938 -- GLOBAL_STEP: 543100 | > loss_text_ce: 0.0500924289226532 (0.036921168521355845) | > loss_mel_ce: 1.7814304828643799 (nan) | > loss: 0.00726794870570302 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13669458579774793) | > loader_time: 0.0045 (0.0047027974567953774)  --> TIME: 2025-02-25 00:07:57 -- STEP: 26018/36938 -- GLOBAL_STEP: 543150 | > loss_text_ce: 0.036812808364629745 (0.03691712953007079) | > loss_mel_ce: 2.57096004486084 (nan) | > loss: 0.010348305106163025 (nan) | > current_lr: 5e-06 | > step_time: 0.186 (0.13669937916397354) | > loader_time: 0.0049 (0.004702761461168631)  --> TIME: 2025-02-25 00:08:06 -- STEP: 26068/36938 -- GLOBAL_STEP: 543200 | > loss_text_ce: 0.03585847467184067 (0.03691663426374645) | > loss_mel_ce: 2.5102946758270264 (nan) | > loss: 0.010103782638907433 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1367011687516699) | > loader_time: 0.0045 (0.004702836352841903)  --> TIME: 2025-02-25 00:08:15 -- STEP: 26118/36938 -- GLOBAL_STEP: 543250 | > loss_text_ce: 0.03419527783989906 (0.03691469933763469) | > loss_mel_ce: 3.059945821762085 (nan) | > loss: 0.012278337962925434 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13669296841461198) | > loader_time: 0.0049 (0.004702721194182471)  --> TIME: 2025-02-25 00:08:25 -- STEP: 26168/36938 -- GLOBAL_STEP: 543300 | > loss_text_ce: 0.04087715968489647 (0.03691353536350738) | > loss_mel_ce: 2.7725467681884766 (nan) | > loss: 0.011164381168782711 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13669445166818242) | > loader_time: 0.0046 (0.004702794209272975)  --> TIME: 2025-02-25 00:08:35 -- STEP: 26218/36938 -- GLOBAL_STEP: 543350 | > loss_text_ce: 0.060530904680490494 (0.036913853452133494) | > loss_mel_ce: 2.39874267578125 (nan) | > loss: 0.009759022854268551 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.1367044907811495) | > loader_time: 0.0045 (0.004702858916135755)  --> TIME: 2025-02-25 00:08:43 -- STEP: 26268/36938 -- GLOBAL_STEP: 543400 | > loss_text_ce: 0.03500518202781677 (0.03691459771342707) | > loss_mel_ce: 3.2703070640563965 (nan) | > loss: 0.013116318732500076 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.1366942908387405) | > loader_time: 0.0047 (0.0047029481279771725)  --> TIME: 2025-02-25 00:08:53 -- STEP: 26318/36938 -- GLOBAL_STEP: 543450 | > loss_text_ce: 0.0331551656126976 (0.036913751822905565) | > loss_mel_ce: 2.5074150562286377 (nan) | > loss: 0.01008162833750248 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.1367006174236537) | > loader_time: 0.0046 (0.004703099970959227)  --> TIME: 2025-02-25 00:09:03 -- STEP: 26368/36938 -- GLOBAL_STEP: 543500 | > loss_text_ce: 0.02979855053126812 (0.036914005577484976) | > loss_mel_ce: 3.582369565963745 (nan) | > loss: 0.014334001578390598 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13670887303475412) | > loader_time: 0.0046 (0.004703293455023187)  --> TIME: 2025-02-25 00:09:12 -- STEP: 26418/36938 -- GLOBAL_STEP: 543550 | > loss_text_ce: 0.044835690408945084 (0.03691344533270165) | > loss_mel_ce: 0.6942397356033325 (nan) | > loss: 0.0029328390955924988 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13671261599661583) | > loader_time: 0.0045 (0.004703413809317748)  --> TIME: 2025-02-25 00:09:22 -- STEP: 26468/36938 -- GLOBAL_STEP: 543600 | > loss_text_ce: 0.033575575798749924 (0.036915686349565234) | > loss_mel_ce: 3.331702470779419 (nan) | > loss: 0.013354278169572353 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13672223896603494) | > loader_time: 0.0047 (0.004703565317283334)  --> TIME: 2025-02-25 00:09:31 -- STEP: 26518/36938 -- GLOBAL_STEP: 543650 | > loss_text_ce: 0.04120394587516785 (0.036913344692119834) | > loss_mel_ce: 2.4447007179260254 (nan) | > loss: 0.009864701889455318 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13670948270273764) | > loader_time: 0.0046 (0.004703682160717333)  --> TIME: 2025-02-25 00:09:40 -- STEP: 26568/36938 -- GLOBAL_STEP: 543700 | > loss_text_ce: 0.03972800076007843 (0.036909445907201986) | > loss_mel_ce: 2.3514626026153564 (nan) | > loss: 0.009488851763308048 (nan) | > current_lr: 5e-06 | > step_time: 0.1078 (0.1366992807736502) | > loader_time: 0.0047 (0.004703771884956104)  --> TIME: 2025-02-25 00:09:49 -- STEP: 26618/36938 -- GLOBAL_STEP: 543750 | > loss_text_ce: 0.036851268261671066 (0.03690700111439744) | > loss_mel_ce: 1.8330209255218506 (nan) | > loss: 0.007420128211379051 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.1366844841948288) | > loader_time: 0.0047 (0.004703643678660419)  --> TIME: 2025-02-25 00:09:58 -- STEP: 26668/36938 -- GLOBAL_STEP: 543800 | > loss_text_ce: 0.031236905604600906 (0.036905211722345455) | > loss_mel_ce: 3.2141988277435303 (nan) | > loss: 0.012878714129328728 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13668092940010035) | > loader_time: 0.0045 (0.004703586312879471)  --> TIME: 2025-02-25 00:10:07 -- STEP: 26718/36938 -- GLOBAL_STEP: 543850 | > loss_text_ce: 0.042244624346494675 (0.03690524489463852) | > loss_mel_ce: 2.5508859157562256 (nan) | > loss: 0.010290201753377914 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.136684524298197) | > loader_time: 0.0045 (0.004703529545518205)  --> TIME: 2025-02-25 00:10:16 -- STEP: 26768/36938 -- GLOBAL_STEP: 543900 | > loss_text_ce: 0.043844129890203476 (0.03690742233090898) | > loss_mel_ce: 1.5118321180343628 (nan) | > loss: 0.006173318717628717 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13667460256220057) | > loader_time: 0.0047 (0.00470356315427814)  --> TIME: 2025-02-25 00:10:25 -- STEP: 26818/36938 -- GLOBAL_STEP: 543950 | > loss_text_ce: 0.04334211349487305 (0.036909508417956406) | > loss_mel_ce: 3.4169111251831055 (nan) | > loss: 0.01373116485774517 (nan) | > current_lr: 5e-06 | > step_time: 0.1689 (0.13666542035655596) | > loader_time: 0.0048 (0.00470373482767815)  --> TIME: 2025-02-25 00:10:35 -- STEP: 26868/36938 -- GLOBAL_STEP: 544000 | > loss_text_ce: 0.03871411085128784 (0.03691140097875552) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13666289420963482) | > loader_time: 0.0045 (0.004703664515226799)  --> TIME: 2025-02-25 00:10:44 -- STEP: 26918/36938 -- GLOBAL_STEP: 544050 | > loss_text_ce: 0.03717955946922302 (0.03691126624682335) | > loss_mel_ce: 3.368703842163086 (nan) | > loss: 0.01351541094481945 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13666292258646107) | > loader_time: 0.005 (0.004703576226974524)  --> TIME: 2025-02-25 00:10:53 -- STEP: 26968/36938 -- GLOBAL_STEP: 544100 | > loss_text_ce: 0.02825314924120903 (0.036910854740130665) | > loss_mel_ce: 2.6677496433258057 (nan) | > loss: 0.010698423720896244 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.1366641859349075) | > loader_time: 0.0048 (0.004703638020342564)  --> TIME: 2025-02-25 00:11:03 -- STEP: 27018/36938 -- GLOBAL_STEP: 544150 | > loss_text_ce: 0.04317418113350868 (0.036911037462782736) | > loss_mel_ce: 3.089287757873535 (nan) | > loss: 0.012430405244231224 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13666356596465662) | > loader_time: 0.0049 (0.004703624877333832)  --> TIME: 2025-02-25 00:11:12 -- STEP: 27068/36938 -- GLOBAL_STEP: 544200 | > loss_text_ce: 0.03809488192200661 (0.03691186324936987) | > loss_mel_ce: 1.771034598350525 (nan) | > loss: 0.00717908563092351 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.1366618798229008) | > loader_time: 0.0047 (0.004703829889902042)  --> TIME: 2025-02-25 00:11:21 -- STEP: 27118/36938 -- GLOBAL_STEP: 544250 | > loss_text_ce: 0.02046874910593033 (0.036910344718445856) | > loss_mel_ce: 3.1455729007720947 (nan) | > loss: 0.012563657946884632 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13666161160455392) | > loader_time: 0.0044 (0.004703994846702668)  --> TIME: 2025-02-25 00:11:31 -- STEP: 27168/36938 -- GLOBAL_STEP: 544300 | > loss_text_ce: 0.032684240490198135 (0.03690936329014347) | > loss_mel_ce: 3.361036539077759 (nan) | > loss: 0.013467147015035152 (nan) | > current_lr: 5e-06 | > step_time: 0.1793 (0.13666623743038211) | > loader_time: 0.0049 (0.004704222320035961)  --> TIME: 2025-02-25 00:11:40 -- STEP: 27218/36938 -- GLOBAL_STEP: 544350 | > loss_text_ce: 0.03270617872476578 (0.036910343221915845) | > loss_mel_ce: 3.313354015350342 (nan) | > loss: 0.01327801775187254 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13667113729819375) | > loader_time: 0.0046 (0.004704390872767762)  --> TIME: 2025-02-25 00:11:49 -- STEP: 27268/36938 -- GLOBAL_STEP: 544400 | > loss_text_ce: 0.03756263107061386 (0.03691120189583835) | > loss_mel_ce: 2.9141032695770264 (nan) | > loss: 0.01171296089887619 (nan) | > current_lr: 5e-06 | > step_time: 0.1266 (0.13666541819105943) | > loader_time: 0.0046 (0.004704493274607546)  --> TIME: 2025-02-25 00:11:58 -- STEP: 27318/36938 -- GLOBAL_STEP: 544450 | > loss_text_ce: 0.03837251663208008 (0.0369110403162985) | > loss_mel_ce: 2.5822203159332275 (nan) | > loss: 0.010399178601801395 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.1366605274725633) | > loader_time: 0.0045 (0.0047045998224550896)  --> TIME: 2025-02-25 00:12:08 -- STEP: 27368/36938 -- GLOBAL_STEP: 544500 | > loss_text_ce: 0.03487398847937584 (0.036910869061780045) | > loss_mel_ce: 2.0960307121276855 (nan) | > loss: 0.008455971255898476 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13666041524254885) | > loader_time: 0.0048 (0.004704611172839132)  --> TIME: 2025-02-25 00:12:17 -- STEP: 27418/36938 -- GLOBAL_STEP: 544550 | > loss_text_ce: 0.033717039972543716 (0.03691170917121032) | > loss_mel_ce: 1.69635009765625 (nan) | > loss: 0.00686534633859992 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.13665210245926798) | > loader_time: 0.0045 (0.004704809152320817)  --> TIME: 2025-02-25 00:12:26 -- STEP: 27468/36938 -- GLOBAL_STEP: 544600 | > loss_text_ce: 0.02524738945066929 (0.03691313347520287) | > loss_mel_ce: 2.577338218688965 (nan) | > loss: 0.010327721014618874 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.13665849364479207) | > loader_time: 0.0048 (0.004705015116944728)  --> TIME: 2025-02-25 00:12:35 -- STEP: 27518/36938 -- GLOBAL_STEP: 544650 | > loss_text_ce: 0.03638410568237305 (0.036912239857236524) | > loss_mel_ce: 3.0910494327545166 (nan) | > loss: 0.012410451658070087 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.1366475002997339) | > loader_time: 0.0046 (0.004704955844253914)  --> TIME: 2025-02-25 00:12:45 -- STEP: 27568/36938 -- GLOBAL_STEP: 544700 | > loss_text_ce: 0.030379118397831917 (0.036911567527753535) | > loss_mel_ce: 1.073390007019043 (nan) | > loss: 0.004380036611109972 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1366435598994994) | > loader_time: 0.0046 (0.004705163096177702)  --> TIME: 2025-02-25 00:12:54 -- STEP: 27618/36938 -- GLOBAL_STEP: 544750 | > loss_text_ce: 0.03405328094959259 (0.03691158527635819) | > loss_mel_ce: 3.2561428546905518 (nan) | > loss: 0.013056335039436817 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13664326167778298) | > loader_time: 0.0046 (0.004705096087074838)  --> TIME: 2025-02-25 00:13:03 -- STEP: 27668/36938 -- GLOBAL_STEP: 544800 | > loss_text_ce: 0.03398244455456734 (0.03691218827079721) | > loss_mel_ce: 1.7237548828125 (nan) | > loss: 0.006975148338824511 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13663364832019056) | > loader_time: 0.0046 (0.004705013929978915)  --> TIME: 2025-02-25 00:13:12 -- STEP: 27718/36938 -- GLOBAL_STEP: 544850 | > loss_text_ce: 0.03601531684398651 (0.03691032282798138) | > loss_mel_ce: 2.695521116256714 (nan) | > loss: 0.0108394306153059 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13662485945955058) | > loader_time: 0.0046 (0.004704925428867065)  --> TIME: 2025-02-25 00:13:21 -- STEP: 27768/36938 -- GLOBAL_STEP: 544900 | > loss_text_ce: 0.03109125979244709 (0.036909701582097734) | > loss_mel_ce: 3.3311829566955566 (nan) | > loss: 0.01334235817193985 (nan) | > current_lr: 5e-06 | > step_time: 0.1866 (0.13662983455221123) | > loader_time: 0.0049 (0.004704834876710247)  --> TIME: 2025-02-25 00:13:31 -- STEP: 27818/36938 -- GLOBAL_STEP: 544950 | > loss_text_ce: 0.02895520068705082 (0.03690882145183209) | > loss_mel_ce: 2.4154880046844482 (nan) | > loss: 0.009700171649456024 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13663665021346086) | > loader_time: 0.0045 (0.004704774364542983)  --> TIME: 2025-02-25 00:13:40 -- STEP: 27868/36938 -- GLOBAL_STEP: 545000 | > loss_text_ce: 0.02984180860221386 (0.03690604688262812) | > loss_mel_ce: 3.1887636184692383 (nan) | > loss: 0.012772244401276112 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13664514933423666) | > loader_time: 0.0046 (0.004704689267753784)  --> TIME: 2025-02-25 00:13:49 -- STEP: 27918/36938 -- GLOBAL_STEP: 545050 | > loss_text_ce: 0.03806906193494797 (0.03690874274954564) | > loss_mel_ce: 3.0112507343292236 (nan) | > loss: 0.01210047584027052 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13663229819175599) | > loader_time: 0.0048 (0.004704494831239026)  --> TIME: 2025-02-25 00:13:59 -- STEP: 27968/36938 -- GLOBAL_STEP: 545100 | > loss_text_ce: 0.038468409329652786 (0.03691296291944341) | > loss_mel_ce: 3.493088722229004 (nan) | > loss: 0.014014116488397121 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.1366304548231364) | > loader_time: 0.0047 (0.004704338249066294)  --> TIME: 2025-02-25 00:14:08 -- STEP: 28018/36938 -- GLOBAL_STEP: 545150 | > loss_text_ce: 0.0307353176176548 (0.03691398839705108) | > loss_mel_ce: 2.771772623062134 (nan) | > loss: 0.011121063493192196 (nan) | > current_lr: 5e-06 | > step_time: 0.1701 (0.13663348431539274) | > loader_time: 0.0047 (0.004704321321698019)  --> TIME: 2025-02-25 00:14:17 -- STEP: 28068/36938 -- GLOBAL_STEP: 545200 | > loss_text_ce: 0.042528972029685974 (0.036911495448868535) | > loss_mel_ce: 3.6854403018951416 (nan) | > loss: 0.01479352917522192 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13663513494451948) | > loader_time: 0.0046 (0.004704224514590856)  --> TIME: 2025-02-25 00:14:26 -- STEP: 28118/36938 -- GLOBAL_STEP: 545250 | > loss_text_ce: 0.0355033278465271 (0.03690996817058398) | > loss_mel_ce: 3.1100714206695557 (nan) | > loss: 0.01248244009912014 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.13662751427805236) | > loader_time: 0.0047 (0.00470432112350748)  --> TIME: 2025-02-25 00:14:36 -- STEP: 28168/36938 -- GLOBAL_STEP: 545300 | > loss_text_ce: 0.027578620240092278 (0.0369066279437718) | > loss_mel_ce: 3.187636137008667 (nan) | > loss: 0.012758789584040642 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13662575687647122) | > loader_time: 0.0053 (0.004704366545215346)  --> TIME: 2025-02-25 00:14:45 -- STEP: 28218/36938 -- GLOBAL_STEP: 545350 | > loss_text_ce: 0.032334428280591965 (0.03690742853304395) | > loss_mel_ce: 1.6589491367340088 (nan) | > loss: 0.006711442954838276 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.1366234305765238) | > loader_time: 0.0046 (0.004704366881738015)  --> TIME: 2025-02-25 00:14:54 -- STEP: 28268/36938 -- GLOBAL_STEP: 545400 | > loss_text_ce: 0.03335609659552574 (0.036906720793034656) | > loss_mel_ce: 3.1744322776794434 (nan) | > loss: 0.012729319743812084 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13661525468602348) | > loader_time: 0.0045 (0.0047042418086277096)  --> TIME: 2025-02-25 00:15:03 -- STEP: 28318/36938 -- GLOBAL_STEP: 545450 | > loss_text_ce: 0.044138744473457336 (0.03690475947383082) | > loss_mel_ce: 2.7321548461914062 (nan) | > loss: 0.011017038486897945 (nan) | > current_lr: 5e-06 | > step_time: 0.1888 (0.13662111474612526) | > loader_time: 0.0049 (0.004704153481340701)  --> TIME: 2025-02-25 00:15:13 -- STEP: 28368/36938 -- GLOBAL_STEP: 545500 | > loss_text_ce: 0.036185987293720245 (0.036905257594651894) | > loss_mel_ce: 3.212500810623169 (nan) | > loss: 0.012891614809632301 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.1366142444837862) | > loader_time: 0.0049 (0.004704036932171736)  --> TIME: 2025-02-25 00:15:22 -- STEP: 28418/36938 -- GLOBAL_STEP: 545550 | > loss_text_ce: 0.025312893092632294 (0.03690269917812154) | > loss_mel_ce: 1.50504469871521 (nan) | > loss: 0.006072848103940487 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13660560279939982) | > loader_time: 0.0046 (0.004704048215935991)  --> TIME: 2025-02-25 00:15:31 -- STEP: 28468/36938 -- GLOBAL_STEP: 545600 | > loss_text_ce: 0.0472828708589077 (0.03690314590169966) | > loss_mel_ce: 3.36098575592041 (nan) | > loss: 0.013524876907467842 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.1366090100267497) | > loader_time: 0.0046 (0.004704156082065382)  --> TIME: 2025-02-25 00:15:40 -- STEP: 28518/36938 -- GLOBAL_STEP: 545650 | > loss_text_ce: 0.03369895741343498 (0.03690365915074921) | > loss_mel_ce: 0.162446528673172 (nan) | > loss: 0.0007783551700413227 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.1366115992580402) | > loader_time: 0.0046 (0.004704390587750821)  --> TIME: 2025-02-25 00:15:50 -- STEP: 28568/36938 -- GLOBAL_STEP: 545700 | > loss_text_ce: 0.0320487804710865 (0.03690064376478533) | > loss_mel_ce: 1.946185827255249 (nan) | > loss: 0.007850137539207935 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.13661316986078742) | > loader_time: 0.0047 (0.004704332575348003)  --> TIME: 2025-02-25 00:15:59 -- STEP: 28618/36938 -- GLOBAL_STEP: 545750 | > loss_text_ce: 0.036337960511446 (0.036901012079362853) | > loss_mel_ce: 2.5742242336273193 (nan) | > loss: 0.01035937387496233 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1366081404997566) | > loader_time: 0.0046 (0.004704228103331611)  --> TIME: 2025-02-25 00:16:08 -- STEP: 28668/36938 -- GLOBAL_STEP: 545800 | > loss_text_ce: 0.03183678165078163 (0.03690201472150539) | > loss_mel_ce: 2.8281702995300293 (nan) | > loss: 0.011349234730005264 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13661194606124838) | > loader_time: 0.0046 (0.004704361657509465)  --> TIME: 2025-02-25 00:16:18 -- STEP: 28718/36938 -- GLOBAL_STEP: 545850 | > loss_text_ce: 0.036825478076934814 (0.0369012559204279) | > loss_mel_ce: 0.14049072563648224 (nan) | > loss: 0.000703635741956532 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13661063192315714) | > loader_time: 0.0048 (0.004704566127751296)  --> TIME: 2025-02-25 00:16:27 -- STEP: 28768/36938 -- GLOBAL_STEP: 545900 | > loss_text_ce: 0.03493240848183632 (0.03689964229684632) | > loss_mel_ce: 1.4072771072387695 (nan) | > loss: 0.005723054055124521 (nan) | > current_lr: 5e-06 | > step_time: 0.1079 (0.13660755942475244) | > loader_time: 0.0045 (0.004704512498269986)  --> TIME: 2025-02-25 00:16:36 -- STEP: 28818/36938 -- GLOBAL_STEP: 545950 | > loss_text_ce: 0.0326896570622921 (0.036899622196293974) | > loss_mel_ce: 2.530198335647583 (nan) | > loss: 0.01017019059509039 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13660930079832798) | > loader_time: 0.0048 (0.004704665902734919)  --> TIME: 2025-02-25 00:16:46 -- STEP: 28868/36938 -- GLOBAL_STEP: 546000 | > loss_text_ce: 0.033339597284793854 (0.03689920264540747) | > loss_mel_ce: 3.0571682453155518 (nan) | > loss: 0.012263920158147812 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.1366157160004182) | > loader_time: 0.0045 (0.0047046069179162995)  --> TIME: 2025-02-25 00:16:55 -- STEP: 28918/36938 -- GLOBAL_STEP: 546050 | > loss_text_ce: 0.04099665954709053 (0.03690040960340573) | > loss_mel_ce: 1.9346123933792114 (nan) | > loss: 0.007839718833565712 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.1366193498250277) | > loader_time: 0.0046 (0.004704651285795826)  --> TIME: 2025-02-25 00:17:05 -- STEP: 28968/36938 -- GLOBAL_STEP: 546100 | > loss_text_ce: 0.02777106501162052 (0.03689930892287854) | > loss_mel_ce: 3.183743715286255 (nan) | > loss: 0.012744106352329254 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13662575788999734) | > loader_time: 0.0046 (0.00470488843783523)  --> TIME: 2025-02-25 00:17:14 -- STEP: 29018/36938 -- GLOBAL_STEP: 546150 | > loss_text_ce: 0.040229447185993195 (0.036899572248917024) | > loss_mel_ce: 3.1377763748168945 (nan) | > loss: 0.012611135840415955 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.1366341685400872) | > loader_time: 0.0047 (0.004704911750405881)  --> TIME: 2025-02-25 00:17:24 -- STEP: 29068/36938 -- GLOBAL_STEP: 546200 | > loss_text_ce: 0.047293975949287415 (0.03690031862313888) | > loss_mel_ce: 3.017730712890625 (nan) | > loss: 0.012162797152996063 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.1366469310290296) | > loader_time: 0.005 (0.004705068783597964)  --> TIME: 2025-02-25 00:17:34 -- STEP: 29118/36938 -- GLOBAL_STEP: 546250 | > loss_text_ce: 0.023832103237509727 (0.03690050626627127) | > loss_mel_ce: 2.27392315864563 (nan) | > loss: 0.009118077345192432 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13664631934781457) | > loader_time: 0.0045 (0.004704993916979267)  --> TIME: 2025-02-25 00:17:43 -- STEP: 29168/36938 -- GLOBAL_STEP: 546300 | > loss_text_ce: 0.036674343049526215 (0.03690228130869147) | > loss_mel_ce: 2.810777425765991 (nan) | > loss: 0.01129941176623106 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13664566861589295) | > loader_time: 0.0048 (0.00470500929435396)  --> TIME: 2025-02-25 00:17:52 -- STEP: 29218/36938 -- GLOBAL_STEP: 546350 | > loss_text_ce: 0.045429714024066925 (0.03690282343411509) | > loss_mel_ce: 0.01963043212890625 (nan) | > loss: 0.00025817519053816795 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13664546521409068) | > loader_time: 0.0046 (0.0047050833057447885)  --> TIME: 2025-02-25 00:18:02 -- STEP: 29268/36938 -- GLOBAL_STEP: 546400 | > loss_text_ce: 0.03813733160495758 (0.03690045220495817) | > loss_mel_ce: 3.6529507637023926 (nan) | > loss: 0.01464717648923397 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.1366463233428501) | > loader_time: 0.0049 (0.004705096579842706)  --> TIME: 2025-02-25 00:18:11 -- STEP: 29318/36938 -- GLOBAL_STEP: 546450 | > loss_text_ce: 0.029028015211224556 (0.036901532331239495) | > loss_mel_ce: 2.653329849243164 (nan) | > loss: 0.010644277557730675 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13664815457425647) | > loader_time: 0.0045 (0.004705017045149382)  --> TIME: 2025-02-25 00:18:20 -- STEP: 29368/36938 -- GLOBAL_STEP: 546500 | > loss_text_ce: 0.041719064116477966 (0.036900216972064914) | > loss_mel_ce: 3.019655227661133 (nan) | > loss: 0.012148311361670494 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.13665192101573997) | > loader_time: 0.0047 (0.004705035907310967)  --> TIME: 2025-02-25 00:18:30 -- STEP: 29418/36938 -- GLOBAL_STEP: 546550 | > loss_text_ce: 0.037731245160102844 (0.03689950326144428) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13665033869905843) | > loader_time: 0.0046 (0.0047049658555729365)  --> TIME: 2025-02-25 00:18:39 -- STEP: 29468/36938 -- GLOBAL_STEP: 546600 | > loss_text_ce: 0.03877310827374458 (0.036896892929975555) | > loss_mel_ce: 2.4375460147857666 (nan) | > loss: 0.009826663881540298 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13665332487711454) | > loader_time: 0.0046 (0.00470492122000811)  --> TIME: 2025-02-25 00:18:48 -- STEP: 29518/36938 -- GLOBAL_STEP: 546650 | > loss_text_ce: 0.034676048904657364 (0.03689380026996578) | > loss_mel_ce: 2.4184186458587646 (nan) | > loss: 0.009734502993524075 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.1366468873060503) | > loader_time: 0.0047 (0.0047048982287076465)  --> TIME: 2025-02-25 00:18:58 -- STEP: 29568/36938 -- GLOBAL_STEP: 546700 | > loss_text_ce: 0.03893113136291504 (0.036895298875905334) | > loss_mel_ce: 2.0527756214141846 (nan) | > loss: 0.008300424553453922 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13665333035446425) | > loader_time: 0.0044 (0.004704885257335419)  --> TIME: 2025-02-25 00:19:07 -- STEP: 29618/36938 -- GLOBAL_STEP: 546750 | > loss_text_ce: 0.023704707622528076 (0.03689375207669167) | > loss_mel_ce: 2.350666046142578 (nan) | > loss: 0.00942210666835308 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13665872790939262) | > loader_time: 0.0049 (0.004705079708360789)  --> TIME: 2025-02-25 00:19:16 -- STEP: 29668/36938 -- GLOBAL_STEP: 546800 | > loss_text_ce: 0.043302591890096664 (0.03689493401216744) | > loss_mel_ce: 2.5060439109802246 (nan) | > loss: 0.010116455145180225 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13665552125299282) | > loader_time: 0.0046 (0.004705024461500597)  --> TIME: 2025-02-25 00:19:26 -- STEP: 29718/36938 -- GLOBAL_STEP: 546850 | > loss_text_ce: 0.056179918348789215 (0.03689550446723441) | > loss_mel_ce: 1.6774808168411255 (nan) | > loss: 0.006879606284201145 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.13665642114336815) | > loader_time: 0.0047 (0.004705056855989646)  --> TIME: 2025-02-25 00:19:35 -- STEP: 29768/36938 -- GLOBAL_STEP: 546900 | > loss_text_ce: 0.03587502986192703 (0.036895550560656167) | > loss_mel_ce: 3.0942533016204834 (nan) | > loss: 0.012421145103871822 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13665164769641003) | > loader_time: 0.0046 (0.004704968266447364)  --> TIME: 2025-02-25 00:19:44 -- STEP: 29818/36938 -- GLOBAL_STEP: 546950 | > loss_text_ce: 0.04851268604397774 (0.03689520675127375) | > loss_mel_ce: 1.8730255365371704 (nan) | > loss: 0.0076251523569226265 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.1366501838454578) | > loader_time: 0.0046 (0.004704882924453874)  --> TIME: 2025-02-25 00:19:54 -- STEP: 29868/36938 -- GLOBAL_STEP: 547000 | > loss_text_ce: 0.04199979081749916 (0.03689539872103751) | > loss_mel_ce: 3.0908098220825195 (nan) | > loss: 0.012431785464286804 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.1366522287977615) | > loader_time: 0.0047 (0.004704804286047398)  --> TIME: 2025-02-25 00:20:03 -- STEP: 29918/36938 -- GLOBAL_STEP: 547050 | > loss_text_ce: 0.050571076571941376 (0.03689542234695868) | > loss_mel_ce: 2.72755765914917 (nan) | > loss: 0.011024320498108864 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13665002531463158) | > loader_time: 0.0047 (0.004704786834012937)  --> TIME: 2025-02-25 00:20:12 -- STEP: 29968/36938 -- GLOBAL_STEP: 547100 | > loss_text_ce: 0.0336168147623539 (0.03689329734144037) | > loss_mel_ce: 2.7397940158843994 (nan) | > loss: 0.011005599051713943 (nan) | > current_lr: 5e-06 | > step_time: 0.1861 (0.13665373778508744) | > loader_time: 0.0052 (0.004704791103781542)  --> TIME: 2025-02-25 00:20:22 -- STEP: 30018/36938 -- GLOBAL_STEP: 547150 | > loss_text_ce: 0.03820075839757919 (0.036891575495118546) | > loss_mel_ce: 2.3575453758239746 (nan) | > loss: 0.00950692966580391 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13665718533973928) | > loader_time: 0.0046 (0.004704759038180067)  --> TIME: 2025-02-25 00:20:31 -- STEP: 30068/36938 -- GLOBAL_STEP: 547200 | > loss_text_ce: 0.03166143596172333 (0.03688995159928383) | > loss_mel_ce: 2.8235087394714355 (nan) | > loss: 0.011330041103065014 (nan) | > current_lr: 5e-06 | > step_time: 0.1531 (0.1366497099169623) | > loader_time: 0.0047 (0.004704714939444214)  --> TIME: 2025-02-25 00:20:40 -- STEP: 30118/36938 -- GLOBAL_STEP: 547250 | > loss_text_ce: 0.03779366984963417 (0.036889875852068765) | > loss_mel_ce: 2.577404499053955 (nan) | > loss: 0.010377771221101284 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.13665413763095852) | > loader_time: 0.0048 (0.004704730366162755)  --> TIME: 2025-02-25 00:20:50 -- STEP: 30168/36938 -- GLOBAL_STEP: 547300 | > loss_text_ce: 0.029999326914548874 (0.03688802955664338) | > loss_mel_ce: 2.3243699073791504 (nan) | > loss: 0.009342735633254051 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.1366712318921532) | > loader_time: 0.005 (0.004704725573215235)  --> TIME: 2025-02-25 00:21:00 -- STEP: 30218/36938 -- GLOBAL_STEP: 547350 | > loss_text_ce: 0.0339684896171093 (0.03688827685083339) | > loss_mel_ce: 3.303696393966675 (nan) | > loss: 0.013244702480733395 (nan) | > current_lr: 5e-06 | > step_time: 0.1776 (0.13667567493945762) | > loader_time: 0.0048 (0.0047046449736871864)  --> TIME: 2025-02-25 00:21:09 -- STEP: 30268/36938 -- GLOBAL_STEP: 547400 | > loss_text_ce: 0.027118243277072906 (0.0368881956515079) | > loss_mel_ce: 2.513871669769287 (nan) | > loss: 0.010083293542265892 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13667285191866338) | > loader_time: 0.0045 (0.0047045380558443615)  --> TIME: 2025-02-25 00:21:19 -- STEP: 30318/36938 -- GLOBAL_STEP: 547450 | > loss_text_ce: 0.04151465371251106 (0.03688998022062322) | > loss_mel_ce: 2.800008773803711 (nan) | > loss: 0.011275887489318848 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13667828622755523) | > loader_time: 0.0046 (0.004704503233274359)  --> TIME: 2025-02-25 00:21:28 -- STEP: 30368/36938 -- GLOBAL_STEP: 547500 | > loss_text_ce: 0.024809326976537704 (0.03688827584129221) | > loss_mel_ce: 2.8986294269561768 (nan) | > loss: 0.011600947938859463 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.1366776136565642) | > loader_time: 0.0045 (0.004704406777410505)  --> TIME: 2025-02-25 00:21:37 -- STEP: 30418/36938 -- GLOBAL_STEP: 547550 | > loss_text_ce: 0.034625496715307236 (0.03688787773468401) | > loss_mel_ce: 2.675187826156616 (nan) | > loss: 0.010753228329122066 (nan) | > current_lr: 5e-06 | > step_time: 0.1343 (0.13667721153128665) | > loader_time: 0.0046 (0.004704316807213302)  --> TIME: 2025-02-25 00:21:46 -- STEP: 30468/36938 -- GLOBAL_STEP: 547600 | > loss_text_ce: 0.03360050916671753 (0.036887130683147834) | > loss_mel_ce: 2.8818247318267822 (nan) | > loss: 0.011569148860871792 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.1366761679706953) | > loader_time: 0.0049 (0.004704222484133853)  --> TIME: 2025-02-25 00:21:56 -- STEP: 30518/36938 -- GLOBAL_STEP: 547650 | > loss_text_ce: 0.04357532411813736 (0.036887350917008455) | > loss_mel_ce: 3.649644374847412 (nan) | > loss: 0.014655634760856628 (nan) | > current_lr: 5e-06 | > step_time: 0.1515 (0.13667858222715493) | > loader_time: 0.0047 (0.00470419331298161)  --> TIME: 2025-02-25 00:22:05 -- STEP: 30568/36938 -- GLOBAL_STEP: 547700 | > loss_text_ce: 0.03374863415956497 (0.03688753648325058) | > loss_mel_ce: 3.230803966522217 (nan) | > loss: 0.012954574078321457 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1366843554134994) | > loader_time: 0.0048 (0.004704119100897607)  --> TIME: 2025-02-25 00:22:15 -- STEP: 30618/36938 -- GLOBAL_STEP: 547750 | > loss_text_ce: 0.04125973582267761 (0.03688923391061557) | > loss_mel_ce: 2.6560895442962646 (nan) | > loss: 0.010703767649829388 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13668690911232728) | > loader_time: 0.0047 (0.004704060393471761)  --> TIME: 2025-02-25 00:22:24 -- STEP: 30668/36938 -- GLOBAL_STEP: 547800 | > loss_text_ce: 0.041064582765102386 (0.03688853301557423) | > loss_mel_ce: 3.3137130737304688 (nan) | > loss: 0.013312609866261482 (nan) | > current_lr: 5e-06 | > step_time: 0.1883 (0.13668311442983125) | > loader_time: 0.0048 (0.004703952317071766)  --> TIME: 2025-02-25 00:22:33 -- STEP: 30718/36938 -- GLOBAL_STEP: 547850 | > loss_text_ce: 0.04612711817026138 (0.03688982008177828) | > loss_mel_ce: 2.7038326263427734 (nan) | > loss: 0.010912539437413216 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1366795083338482) | > loader_time: 0.0046 (0.004703885744120772)  --> TIME: 2025-02-25 00:22:43 -- STEP: 30768/36938 -- GLOBAL_STEP: 547900 | > loss_text_ce: 0.0401558019220829 (0.03689389653096203) | > loss_mel_ce: 2.9762306213378906 (nan) | > loss: 0.011969787999987602 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13667890457392917) | > loader_time: 0.0046 (0.004703761766617675)  --> TIME: 2025-02-25 00:22:52 -- STEP: 30818/36938 -- GLOBAL_STEP: 547950 | > loss_text_ce: 0.03728450462222099 (0.0368913774454387) | > loss_mel_ce: 2.5947070121765137 (nan) | > loss: 0.010444412007927895 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13668307393752605) | > loader_time: 0.0045 (0.004703836976704308)  --> TIME: 2025-02-25 00:23:01 -- STEP: 30868/36938 -- GLOBAL_STEP: 548000 | > loss_text_ce: 0.030206361785531044 (0.03689368436393806) | > loss_mel_ce: 3.0071938037872314 (nan) | > loss: 0.012053175829350948 (nan) | > current_lr: 5e-06 | > step_time: 0.1614 (0.13668334334258483) | > loader_time: 0.0046 (0.0047038873814232406)  --> TIME: 2025-02-25 00:23:11 -- STEP: 30918/36938 -- GLOBAL_STEP: 548050 | > loss_text_ce: 0.022796982899308205 (0.03688967025221439) | > loss_mel_ce: 1.5905287265777588 (nan) | > loss: 0.006402086466550827 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13669035805483207) | > loader_time: 0.0047 (0.004703916347584322)  --> TIME: 2025-02-25 00:23:20 -- STEP: 30968/36938 -- GLOBAL_STEP: 548100 | > loss_text_ce: 0.0338619165122509 (0.03689128795612309) | > loss_mel_ce: 3.2550535202026367 (nan) | > loss: 0.013051251880824566 (nan) | > current_lr: 5e-06 | > step_time: 0.1734 (0.1366900791573118) | > loader_time: 0.0047 (0.004703809265879721)  --> TIME: 2025-02-25 00:23:29 -- STEP: 31018/36938 -- GLOBAL_STEP: 548150 | > loss_text_ce: 0.03552032634615898 (0.03688958388517151) | > loss_mel_ce: 3.492624521255493 (nan) | > loss: 0.014000575989484787 (nan) | > current_lr: 5e-06 | > step_time: 0.144 (0.1366820165594315) | > loader_time: 0.0047 (0.004703681445443028)  --> TIME: 2025-02-25 00:23:39 -- STEP: 31068/36938 -- GLOBAL_STEP: 548200 | > loss_text_ce: 0.03416585922241211 (0.03689092211356331) | > loss_mel_ce: 1.6267746686935425 (nan) | > loss: 0.00659103412181139 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13669238647500537) | > loader_time: 0.0047 (0.004703598016628599)  --> TIME: 2025-02-25 00:23:48 -- STEP: 31118/36938 -- GLOBAL_STEP: 548250 | > loss_text_ce: 0.03394817188382149 (0.03688964004026706) | > loss_mel_ce: 3.259115219116211 (nan) | > loss: 0.013067713007330894 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.13668902627958615) | > loader_time: 0.0048 (0.00470354788575671)  --> TIME: 2025-02-25 00:23:57 -- STEP: 31168/36938 -- GLOBAL_STEP: 548300 | > loss_text_ce: 0.038361821323633194 (0.03689065333359559) | > loss_mel_ce: 3.158041000366211 (nan) | > loss: 0.012684138491749763 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.13668506731732485) | > loader_time: 0.0047 (0.004703450810309522)  --> TIME: 2025-02-25 00:24:07 -- STEP: 31218/36938 -- GLOBAL_STEP: 548350 | > loss_text_ce: 0.035827964544296265 (0.036890702904313656) | > loss_mel_ce: 2.930422782897949 (nan) | > loss: 0.011770837008953094 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1366828853632356) | > loader_time: 0.0046 (0.004703435557818304)  --> TIME: 2025-02-25 00:24:16 -- STEP: 31268/36938 -- GLOBAL_STEP: 548400 | > loss_text_ce: 0.03556422144174576 (0.036887497309666804) | > loss_mel_ce: 2.835153818130493 (nan) | > loss: 0.011391738429665565 (nan) | > current_lr: 5e-06 | > step_time: 0.1824 (0.13667998929851574) | > loader_time: 0.0048 (0.004703390525097064)  --> TIME: 2025-02-25 00:24:25 -- STEP: 31318/36938 -- GLOBAL_STEP: 548450 | > loss_text_ce: 0.030216388404369354 (0.03688904203646855) | > loss_mel_ce: 2.4245564937591553 (nan) | > loss: 0.00974116288125515 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.1366762907022845) | > loader_time: 0.0045 (0.004703261476342877)  --> TIME: 2025-02-25 00:24:34 -- STEP: 31368/36938 -- GLOBAL_STEP: 548500 | > loss_text_ce: 0.03253171592950821 (0.03688592624275624) | > loss_mel_ce: 2.618870258331299 (nan) | > loss: 0.010521437041461468 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.136669052852657) | > loader_time: 0.0048 (0.0047031378022446306)  --> TIME: 2025-02-25 00:24:43 -- STEP: 31418/36938 -- GLOBAL_STEP: 548550 | > loss_text_ce: 0.03744830563664436 (0.03688442187473978) | > loss_mel_ce: 2.820835590362549 (nan) | > loss: 0.011342397890985012 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13667092999295613) | > loader_time: 0.0045 (0.00470321505049544)  --> TIME: 2025-02-25 00:24:53 -- STEP: 31468/36938 -- GLOBAL_STEP: 548600 | > loss_text_ce: 0.03599868342280388 (0.03688524785981094) | > loss_mel_ce: 3.1924166679382324 (nan) | > loss: 0.012811172753572464 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.1366737156564683) | > loader_time: 0.0046 (0.004703295030844776)  --> TIME: 2025-02-25 00:25:02 -- STEP: 31518/36938 -- GLOBAL_STEP: 548650 | > loss_text_ce: 0.02562781237065792 (0.0368839950995209) | > loss_mel_ce: 2.9604122638702393 (nan) | > loss: 0.011849366128444672 (nan) | > current_lr: 5e-06 | > step_time: 0.175 (0.13667650282235516) | > loader_time: 0.0049 (0.004703408464939705)  --> TIME: 2025-02-25 00:25:11 -- STEP: 31568/36938 -- GLOBAL_STEP: 548700 | > loss_text_ce: 0.04716644063591957 (0.03688426622954486) | > loss_mel_ce: 2.2269697189331055 (nan) | > loss: 0.009024349972605705 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.1366682008354427) | > loader_time: 0.0046 (0.004703329969527905)  --> TIME: 2025-02-25 00:25:21 -- STEP: 31618/36938 -- GLOBAL_STEP: 548750 | > loss_text_ce: 0.043932147324085236 (0.036885967751384634) | > loss_mel_ce: 3.4803011417388916 (nan) | > loss: 0.01398505363613367 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.1366784028848913) | > loader_time: 0.0048 (0.004703332844114686)  --> TIME: 2025-02-25 00:25:30 -- STEP: 31668/36938 -- GLOBAL_STEP: 548800 | > loss_text_ce: 0.03793695196509361 (0.036886478836655086) | > loss_mel_ce: 3.439006805419922 (nan) | > loss: 0.013797396793961525 (nan) | > current_lr: 5e-06 | > step_time: 0.1441 (0.13667980236613902) | > loader_time: 0.0046 (0.0047033876500617855)  --> TIME: 2025-02-25 00:25:39 -- STEP: 31718/36938 -- GLOBAL_STEP: 548850 | > loss_text_ce: 0.04516178369522095 (0.03688774840093832) | > loss_mel_ce: 2.13763427734375 (nan) | > loss: 0.008661889471113682 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13667518009085067) | > loader_time: 0.0046 (0.004703480731765156)  --> TIME: 2025-02-25 00:25:49 -- STEP: 31768/36938 -- GLOBAL_STEP: 548900 | > loss_text_ce: 0.03257465735077858 (0.03688986387441026) | > loss_mel_ce: 3.463918447494507 (nan) | > loss: 0.01387497317045927 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13667478968322083) | > loader_time: 0.0046 (0.004703569602858206)  --> TIME: 2025-02-25 00:25:58 -- STEP: 31818/36938 -- GLOBAL_STEP: 548950 | > loss_text_ce: 0.034983258694410324 (0.03689018307475327) | > loss_mel_ce: 3.1236000061035156 (nan) | > loss: 0.012534060515463352 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13667080419458819) | > loader_time: 0.0047 (0.004703535620904986)  --> TIME: 2025-02-25 00:26:07 -- STEP: 31868/36938 -- GLOBAL_STEP: 549000 | > loss_text_ce: 0.013157566078007221 (0.036887069821954346) | > loss_mel_ce: 1.6722534894943237 (nan) | > loss: 0.006688139867037535 (nan) | > current_lr: 5e-06 | > step_time: 0.1079 (0.13667162706311262) | > loader_time: 0.0045 (0.0047034107787375075)  --> TIME: 2025-02-25 00:26:16 -- STEP: 31918/36938 -- GLOBAL_STEP: 549050 | > loss_text_ce: 0.033770106732845306 (0.03688672594713805) | > loss_mel_ce: 2.881504535675049 (nan) | > loss: 0.011568550951778889 (nan) | > current_lr: 5e-06 | > step_time: 0.1868 (0.1366643942913566) | > loader_time: 0.0049 (0.004703449511245629)  --> TIME: 2025-02-25 00:26:26 -- STEP: 31968/36938 -- GLOBAL_STEP: 549100 | > loss_text_ce: 0.02804098092019558 (0.03688453222770947) | > loss_mel_ce: 2.9083352088928223 (nan) | > loss: 0.011652287095785141 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13666845500111283) | > loader_time: 0.0046 (0.004703343623094939)  --> TIME: 2025-02-25 00:26:35 -- STEP: 32018/36938 -- GLOBAL_STEP: 549150 | > loss_text_ce: 0.034729912877082825 (0.036885243649905176) | > loss_mel_ce: 3.7158203125 (nan) | > loss: 0.014883137308061123 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13666995837274634) | > loader_time: 0.0046 (0.004703320795073828)  --> TIME: 2025-02-25 00:26:44 -- STEP: 32068/36938 -- GLOBAL_STEP: 549200 | > loss_text_ce: 0.041313428431749344 (0.03688462678793417) | > loss_mel_ce: 2.435019016265869 (nan) | > loss: 0.009826716966927052 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13666758974309007) | > loader_time: 0.0045 (0.004703237147377006)  --> TIME: 2025-02-25 00:26:54 -- STEP: 32118/36938 -- GLOBAL_STEP: 549250 | > loss_text_ce: 0.03671559691429138 (0.03688324095128719) | > loss_mel_ce: 2.0341272354125977 (nan) | > loss: 0.008217629976570606 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13667228763935982) | > loader_time: 0.0045 (0.004703201543305669)  --> TIME: 2025-02-25 00:27:03 -- STEP: 32168/36938 -- GLOBAL_STEP: 549300 | > loss_text_ce: 0.03368397802114487 (0.036881152326288365) | > loss_mel_ce: 3.275925874710083 (nan) | > loss: 0.013133373111486435 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13666988497526364) | > loader_time: 0.0048 (0.0047032034269644234)  --> TIME: 2025-02-25 00:27:13 -- STEP: 32218/36938 -- GLOBAL_STEP: 549350 | > loss_text_ce: 0.025767499580979347 (0.03687865757799929) | > loss_mel_ce: 2.359619140625 (nan) | > loss: 0.009465821087360382 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13667223673158743) | > loader_time: 0.0045 (0.0047033020101584005)  --> TIME: 2025-02-25 00:27:22 -- STEP: 32268/36938 -- GLOBAL_STEP: 549400 | > loss_text_ce: 0.04493947699666023 (0.03687743051904036) | > loss_mel_ce: 2.745893955230713 (nan) | > loss: 0.011074736714363098 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13666602024524352) | > loader_time: 0.0046 (0.004703362841905796)  --> TIME: 2025-02-25 00:27:31 -- STEP: 32318/36938 -- GLOBAL_STEP: 549450 | > loss_text_ce: 0.03604640066623688 (0.036875044510444405) | > loss_mel_ce: 0.0573170967400074 (nan) | > loss: 0.00037049007369205356 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.136663125546766) | > loader_time: 0.0047 (0.00470347285410856)  --> TIME: 2025-02-25 00:27:40 -- STEP: 32368/36938 -- GLOBAL_STEP: 549500 | > loss_text_ce: 0.027605824172496796 (0.03687355642104395) | > loss_mel_ce: 2.3397538661956787 (nan) | > loss: 0.009394285269081593 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1366608258570655) | > loader_time: 0.0045 (0.0047036062150795565)  --> TIME: 2025-02-25 00:27:49 -- STEP: 32418/36938 -- GLOBAL_STEP: 549550 | > loss_text_ce: 0.029462121427059174 (0.036874674367112836) | > loss_mel_ce: 1.6263021230697632 (nan) | > loss: 0.006570493336766958 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13666078575712579) | > loader_time: 0.0046 (0.004703667002196667)  --> TIME: 2025-02-25 00:27:59 -- STEP: 32468/36938 -- GLOBAL_STEP: 549600 | > loss_text_ce: 0.04407818242907524 (0.03687470311799081) | > loss_mel_ce: 0.06803659349679947 (nan) | > loss: 0.000444899924332276 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.13666041002083282) | > loader_time: 0.0046 (0.004703676478827779)  --> TIME: 2025-02-25 00:28:08 -- STEP: 32518/36938 -- GLOBAL_STEP: 549650 | > loss_text_ce: 0.0403137281537056 (0.03687431184476185) | > loss_mel_ce: 3.041668653488159 (nan) | > loss: 0.012230089865624905 (nan) | > current_lr: 5e-06 | > step_time: 0.1578 (0.13666116757266542) | > loader_time: 0.0049 (0.004703669994107163)  --> TIME: 2025-02-25 00:28:17 -- STEP: 32568/36938 -- GLOBAL_STEP: 549700 | > loss_text_ce: 0.03201265633106232 (0.03687371417636508) | > loss_mel_ce: 2.839982032775879 (nan) | > loss: 0.011396804824471474 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13666285343470078) | > loader_time: 0.0053 (0.0047037082730449695)  --> TIME: 2025-02-25 00:28:27 -- STEP: 32618/36938 -- GLOBAL_STEP: 549750 | > loss_text_ce: 0.0320599339902401 (0.03686975189619684) | > loss_mel_ce: 2.2175421714782715 (nan) | > loss: 0.008926993235945702 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.136660349201664) | > loader_time: 0.0045 (0.004703645009150678)  --> TIME: 2025-02-25 00:28:36 -- STEP: 32668/36938 -- GLOBAL_STEP: 549800 | > loss_text_ce: 0.02717299945652485 (0.03687007576015629) | > loss_mel_ce: 2.3295979499816895 (nan) | > loss: 0.009352266788482666 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13666892777870726) | > loader_time: 0.0045 (0.0047036075046133115)  --> TIME: 2025-02-25 00:28:46 -- STEP: 32718/36938 -- GLOBAL_STEP: 549850 | > loss_text_ce: 0.03758388012647629 (0.03687087350344297) | > loss_mel_ce: 2.9929707050323486 (nan) | > loss: 0.012026011012494564 (nan) | > current_lr: 5e-06 | > step_time: 0.162 (0.13666769295058293) | > loader_time: 0.0048 (0.004703502330317124)  --> TIME: 2025-02-25 00:28:55 -- STEP: 32768/36938 -- GLOBAL_STEP: 549900 | > loss_text_ce: 0.026744134724140167 (0.036871055164482164) | > loss_mel_ce: 0.15552571415901184 (nan) | > loss: 0.0007232930511236191 (nan) | > current_lr: 5e-06 | > step_time: 0.1583 (0.136674608540489) | > loader_time: 0.0047 (0.004703413651441194)  --> TIME: 2025-02-25 00:29:05 -- STEP: 32818/36938 -- GLOBAL_STEP: 549950 | > loss_text_ce: 0.03299150988459587 (0.03687212674996737) | > loss_mel_ce: 2.1178464889526367 (nan) | > loss: 0.008535071276128292 (nan) | > current_lr: 5e-06 | > step_time: 0.1442 (0.13667638082159686) | > loader_time: 0.0047 (0.004703402933323766)  --> TIME: 2025-02-25 00:29:14 -- STEP: 32868/36938 -- GLOBAL_STEP: 550000 | > loss_text_ce: 0.04085506498813629 (0.03687068275036554) | > loss_mel_ce: 3.157270908355713 (nan) | > loss: 0.012690977193415165 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13667164686548228) | > loader_time: 0.0047 (0.004703308139764785) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_550000.pth  --> TIME: 2025-02-25 00:29:29 -- STEP: 32918/36938 -- GLOBAL_STEP: 550050 | > loss_text_ce: 0.031681641936302185 (0.03686956992651014) | > loss_mel_ce: 2.1837799549102783 (nan) | > loss: 0.008791514672338963 (nan) | > current_lr: 5e-06 | > step_time: 0.1071 (0.13666346942431876) | > loader_time: 0.0045 (0.0047034878249561825)  --> TIME: 2025-02-25 00:29:38 -- STEP: 32968/36938 -- GLOBAL_STEP: 550100 | > loss_text_ce: 0.0362975150346756 (0.03686992457297281) | > loss_mel_ce: 2.5113768577575684 (nan) | > loss: 0.01010981947183609 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13666726385931) | > loader_time: 0.0052 (0.004703599665819253)  --> TIME: 2025-02-25 00:29:48 -- STEP: 33018/36938 -- GLOBAL_STEP: 550150 | > loss_text_ce: 0.040989186614751816 (0.036869634387235804) | > loss_mel_ce: 2.9646475315093994 (nan) | > loss: 0.01192713063210249 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13666625103473118) | > loader_time: 0.0046 (0.00470355335591206)  --> TIME: 2025-02-25 00:29:57 -- STEP: 33068/36938 -- GLOBAL_STEP: 550200 | > loss_text_ce: 0.03294989466667175 (0.03686909278074789) | > loss_mel_ce: 2.4427402019500732 (nan) | > loss: 0.009824167937040329 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13666568857100536) | > loader_time: 0.0046 (0.004703528498657297)  --> TIME: 2025-02-25 00:30:06 -- STEP: 33118/36938 -- GLOBAL_STEP: 550250 | > loss_text_ce: 0.02461954951286316 (0.03686634917681106) | > loss_mel_ce: 2.739201307296753 (nan) | > loss: 0.01096754427999258 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13666157711242666) | > loader_time: 0.0046 (0.004703450126631794)  --> TIME: 2025-02-25 00:30:15 -- STEP: 33168/36938 -- GLOBAL_STEP: 550300 | > loss_text_ce: 0.03317956253886223 (0.0368658574170824) | > loss_mel_ce: 0.08233642578125 (nan) | > loss: 0.00045839682570658624 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13665689323843308) | > loader_time: 0.0046 (0.00470332444087926)  --> TIME: 2025-02-25 00:30:25 -- STEP: 33218/36938 -- GLOBAL_STEP: 550350 | > loss_text_ce: 0.038992997258901596 (0.0368648616565821) | > loss_mel_ce: 2.8481240272521973 (nan) | > loss: 0.011456813663244247 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.1366618525307952) | > loader_time: 0.0047 (0.004703403653253846)  --> TIME: 2025-02-25 00:30:34 -- STEP: 33268/36938 -- GLOBAL_STEP: 550400 | > loss_text_ce: 0.03176896274089813 (0.0368627867687113) | > loss_mel_ce: 0.2127583771944046 (nan) | > loss: 0.0009703466203063726 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13666273611761826) | > loader_time: 0.0046 (0.0047034892709664575)  --> TIME: 2025-02-25 00:30:43 -- STEP: 33318/36938 -- GLOBAL_STEP: 550450 | > loss_text_ce: 0.035569384694099426 (0.03686290537100769) | > loss_mel_ce: 1.8106354475021362 (nan) | > loss: 0.007326210383325815 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13666210308947452) | > loader_time: 0.0047 (0.004703491588079342)  --> TIME: 2025-02-25 00:30:53 -- STEP: 33368/36938 -- GLOBAL_STEP: 550500 | > loss_text_ce: 0.03624291345477104 (0.03686444248356687) | > loss_mel_ce: 2.4028728008270264 (nan) | > loss: 0.009679031558334827 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.1366624366648339) | > loader_time: 0.0046 (0.004703379540499411)  --> TIME: 2025-02-25 00:31:02 -- STEP: 33418/36938 -- GLOBAL_STEP: 550550 | > loss_text_ce: 0.03463197126984596 (0.036863297744273146) | > loss_mel_ce: 2.625337839126587 (nan) | > loss: 0.01055543590337038 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13666541262898452) | > loader_time: 0.0045 (0.004703253509397719)  --> TIME: 2025-02-25 00:31:12 -- STEP: 33468/36938 -- GLOBAL_STEP: 550600 | > loss_text_ce: 0.03283118084073067 (0.03686355209551608) | > loss_mel_ce: 2.153742790222168 (nan) | > loss: 0.008676880970597267 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1366687471740951) | > loader_time: 0.0046 (0.004703202967975965)  --> TIME: 2025-02-25 00:31:21 -- STEP: 33518/36938 -- GLOBAL_STEP: 550650 | > loss_text_ce: 0.03349843993782997 (0.03686450289330476) | > loss_mel_ce: 3.3970022201538086 (nan) | > loss: 0.013613099232316017 (nan) | > current_lr: 5e-06 | > step_time: 0.1659 (0.13666805305403634) | > loader_time: 0.0047 (0.004703120703315629)  --> TIME: 2025-02-25 00:31:30 -- STEP: 33568/36938 -- GLOBAL_STEP: 550700 | > loss_text_ce: 0.03226745128631592 (0.036862408745609955) | > loss_mel_ce: 2.7328574657440186 (nan) | > loss: 0.01097271777689457 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.13666984856526507) | > loader_time: 0.0047 (0.004703129354946012)  --> TIME: 2025-02-25 00:31:40 -- STEP: 33618/36938 -- GLOBAL_STEP: 550750 | > loss_text_ce: 0.03833790868520737 (0.03686144031966875) | > loss_mel_ce: 3.158003330230713 (nan) | > loss: 0.012683894485235214 (nan) | > current_lr: 5e-06 | > step_time: 0.1853 (0.13667226434144097) | > loader_time: 0.0049 (0.004703154944885579)  --> TIME: 2025-02-25 00:31:49 -- STEP: 33668/36938 -- GLOBAL_STEP: 550800 | > loss_text_ce: 0.03194517642259598 (0.03686274385265718) | > loss_mel_ce: 3.456472396850586 (nan) | > loss: 0.013842928223311901 (nan) | > current_lr: 5e-06 | > step_time: 0.1886 (0.136670822837765) | > loader_time: 0.0048 (0.004703164277683434)  --> TIME: 2025-02-25 00:31:58 -- STEP: 33718/36938 -- GLOBAL_STEP: 550850 | > loss_text_ce: 0.0501539409160614 (0.036861604273917166) | > loss_mel_ce: 2.7730159759521484 (nan) | > loss: 0.011203056201338768 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.13667519026517128) | > loader_time: 0.0047 (0.004703509177564238)  --> TIME: 2025-02-25 00:32:08 -- STEP: 33768/36938 -- GLOBAL_STEP: 550900 | > loss_text_ce: 0.041416529566049576 (0.03686364379146476) | > loss_mel_ce: 3.0628345012664795 (nan) | > loss: 0.01231845747679472 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13667558761632667) | > loader_time: 0.0049 (0.004703597741735883)  --> TIME: 2025-02-25 00:32:17 -- STEP: 33818/36938 -- GLOBAL_STEP: 550950 | > loss_text_ce: 0.04349087178707123 (0.036863027309286034) | > loss_mel_ce: 2.1366159915924072 (nan) | > loss: 0.008651218377053738 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13666990296253742) | > loader_time: 0.0046 (0.004703549237809335)  --> TIME: 2025-02-25 00:32:26 -- STEP: 33868/36938 -- GLOBAL_STEP: 551000 | > loss_text_ce: 0.03243129700422287 (0.036863803495016424) | > loss_mel_ce: 3.4076056480407715 (nan) | > loss: 0.013650941662490368 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13666555065465064) | > loader_time: 0.0046 (0.0047035716747722035)  --> TIME: 2025-02-25 00:32:35 -- STEP: 33918/36938 -- GLOBAL_STEP: 551050 | > loss_text_ce: 0.03427579998970032 (0.03686491094337509) | > loss_mel_ce: 2.519317865371704 (nan) | > loss: 0.010133308358490467 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13666094230926568) | > loader_time: 0.0045 (0.004703589553884715)  --> TIME: 2025-02-25 00:32:44 -- STEP: 33968/36938 -- GLOBAL_STEP: 551100 | > loss_text_ce: 0.03911218419671059 (0.03686556239024872) | > loss_mel_ce: 2.8629400730133057 (nan) | > loss: 0.011516080237925053 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.13665586624345474) | > loader_time: 0.005 (0.004703513495325732)  --> TIME: 2025-02-25 00:32:54 -- STEP: 34018/36938 -- GLOBAL_STEP: 551150 | > loss_text_ce: 0.032279498875141144 (0.03686544748992944) | > loss_mel_ce: 3.734442710876465 (nan) | > loss: 0.014947311021387577 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13665359857340492) | > loader_time: 0.0046 (0.004703606511558439)  --> TIME: 2025-02-25 00:33:03 -- STEP: 34068/36938 -- GLOBAL_STEP: 551200 | > loss_text_ce: 0.03033585660159588 (0.03686380634084648) | > loss_mel_ce: 1.5912728309631348 (nan) | > loss: 0.006434955634176731 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13665620807164278) | > loader_time: 0.0046 (0.004703681038147352)  --> TIME: 2025-02-25 00:33:13 -- STEP: 34118/36938 -- GLOBAL_STEP: 551250 | > loss_text_ce: 0.03435081988573074 (0.03686148979517587) | > loss_mel_ce: 3.1896934509277344 (nan) | > loss: 0.012793827801942825 (nan) | > current_lr: 5e-06 | > step_time: 0.1754 (0.13666534569179817) | > loader_time: 0.0048 (0.004703816400965433)  --> TIME: 2025-02-25 00:33:22 -- STEP: 34168/36938 -- GLOBAL_STEP: 551300 | > loss_text_ce: 0.04203673452138901 (0.03686131866845784) | > loss_mel_ce: 3.251398801803589 (nan) | > loss: 0.01306918915361166 (nan) | > current_lr: 5e-06 | > step_time: 0.168 (0.13666521824653924) | > loader_time: 0.0055 (0.004703777487008331)  --> TIME: 2025-02-25 00:33:32 -- STEP: 34218/36938 -- GLOBAL_STEP: 551350 | > loss_text_ce: 0.048738960176706314 (0.036861593944496486) | > loss_mel_ce: 2.779461145401001 (nan) | > loss: 0.011223017238080502 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13667368831720433) | > loader_time: 0.0047 (0.004703998203356708)  --> TIME: 2025-02-25 00:33:41 -- STEP: 34268/36938 -- GLOBAL_STEP: 551400 | > loss_text_ce: 0.031071322038769722 (0.03686201906276189) | > loss_mel_ce: 2.556614637374878 (nan) | > loss: 0.01026859600096941 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13667099551279635) | > loader_time: 0.0046 (0.004704108361506548)  --> TIME: 2025-02-25 00:33:50 -- STEP: 34318/36938 -- GLOBAL_STEP: 551450 | > loss_text_ce: 0.04405561834573746 (0.03686060015753355) | > loss_mel_ce: 3.1572864055633545 (nan) | > loss: 0.01270373910665512 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13667252003448443) | > loader_time: 0.0046 (0.004704110528894866)  --> TIME: 2025-02-25 00:34:00 -- STEP: 34368/36938 -- GLOBAL_STEP: 551500 | > loss_text_ce: 0.046415217220783234 (0.036860348196396255) | > loss_mel_ce: 3.1333165168762207 (nan) | > loss: 0.012617983855307102 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.13667111749478192) | > loader_time: 0.0048 (0.004704128888399204)  --> TIME: 2025-02-25 00:34:09 -- STEP: 34418/36938 -- GLOBAL_STEP: 551550 | > loss_text_ce: 0.03330472484230995 (0.0368597200488065) | > loss_mel_ce: 3.1028878688812256 (nan) | > loss: 0.012445209547877312 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13667431529668783) | > loader_time: 0.0047 (0.0047040676362662785)  --> TIME: 2025-02-25 00:34:19 -- STEP: 34468/36938 -- GLOBAL_STEP: 551600 | > loss_text_ce: 0.028541352599859238 (0.03685954388023396) | > loss_mel_ce: 1.233366847038269 (nan) | > loss: 0.0050075724720954895 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13667985472170663) | > loader_time: 0.0045 (0.004704007129042741)  --> TIME: 2025-02-25 00:34:28 -- STEP: 34518/36938 -- GLOBAL_STEP: 551650 | > loss_text_ce: 0.0364050455391407 (0.036859693600957776) | > loss_mel_ce: 0.0361328125 (nan) | > loss: 0.00028784864116460085 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13668451675141258) | > loader_time: 0.0046 (0.004703977512894689)  --> TIME: 2025-02-25 00:34:38 -- STEP: 34568/36938 -- GLOBAL_STEP: 551700 | > loss_text_ce: 0.04022675380110741 (0.036860209365288485) | > loss_mel_ce: 3.088613510131836 (nan) | > loss: 0.012416033074259758 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.13669098825042064) | > loader_time: 0.0048 (0.004703977026064259)  --> TIME: 2025-02-25 00:34:47 -- STEP: 34618/36938 -- GLOBAL_STEP: 551750 | > loss_text_ce: 0.04543069377541542 (0.036858232946815166) | > loss_mel_ce: 2.4725441932678223 (nan) | > loss: 0.0099919643253088 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.13668673940971926) | > loader_time: 0.0046 (0.004703871718559871)  --> TIME: 2025-02-25 00:34:56 -- STEP: 34668/36938 -- GLOBAL_STEP: 551800 | > loss_text_ce: 0.02396332286298275 (0.036858071916347136) | > loss_mel_ce: 2.412998676300049 (nan) | > loss: 0.009670483879745007 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13668876730943275) | > loader_time: 0.0047 (0.00470400548413147)  --> TIME: 2025-02-25 00:35:05 -- STEP: 34718/36938 -- GLOBAL_STEP: 551850 | > loss_text_ce: 0.0401025153696537 (0.03685845672568979) | > loss_mel_ce: 2.966120719909668 (nan) | > loss: 0.011929458007216454 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13667547395821514) | > loader_time: 0.0044 (0.004703885090571515)  --> TIME: 2025-02-25 00:35:14 -- STEP: 34768/36938 -- GLOBAL_STEP: 551900 | > loss_text_ce: 0.03663269430398941 (0.03685640732690655) | > loss_mel_ce: 2.97220516204834 (nan) | > loss: 0.011939833872020245 (nan) | > current_lr: 5e-06 | > step_time: 0.1662 (0.13667548299384422) | > loader_time: 0.0047 (0.0047037810553507845)  --> TIME: 2025-02-25 00:35:24 -- STEP: 34818/36938 -- GLOBAL_STEP: 551950 | > loss_text_ce: 0.03731711208820343 (0.03685351931819256) | > loss_mel_ce: 3.10197377204895 (nan) | > loss: 0.01245750393718481 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13668201999433205) | > loader_time: 0.0047 (0.004703697053614661)  --> TIME: 2025-02-25 00:35:33 -- STEP: 34868/36938 -- GLOBAL_STEP: 552000 | > loss_text_ce: 0.047827303409576416 (0.03685295814568121) | > loss_mel_ce: 3.0926895141601562 (nan) | > loss: 0.012462369166314602 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13668049138744134) | > loader_time: 0.0047 (0.004703637765090315)  --> TIME: 2025-02-25 00:35:42 -- STEP: 34918/36938 -- GLOBAL_STEP: 552050 | > loss_text_ce: 0.03677945211529732 (0.03685101291498773) | > loss_mel_ce: 3.2368850708007812 (nan) | > loss: 0.012990732677280903 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13667268272090577) | > loader_time: 0.0045 (0.004703590178777794)  --> TIME: 2025-02-25 00:35:52 -- STEP: 34968/36938 -- GLOBAL_STEP: 552100 | > loss_text_ce: 0.03552060201764107 (0.03685090661921811) | > loss_mel_ce: 2.3763787746429443 (nan) | > loss: 0.009571029804646969 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13667196575386098) | > loader_time: 0.0046 (0.004703539912636806)  --> TIME: 2025-02-25 00:36:01 -- STEP: 35018/36938 -- GLOBAL_STEP: 552150 | > loss_text_ce: 0.033681463450193405 (0.036850584931165026) | > loss_mel_ce: 2.942125082015991 (nan) | > loss: 0.011808756738901138 (nan) | > current_lr: 5e-06 | > step_time: 0.1449 (0.13666849413048335) | > loader_time: 0.0046 (0.004703420534405674)  --> TIME: 2025-02-25 00:36:10 -- STEP: 35068/36938 -- GLOBAL_STEP: 552200 | > loss_text_ce: 0.03548895940184593 (0.03685151328013691) | > loss_mel_ce: 2.9773013591766357 (nan) | > loss: 0.01195551734417677 (nan) | > current_lr: 5e-06 | > step_time: 0.1591 (0.13666637883812438) | > loader_time: 0.0047 (0.004703361298402358)  --> TIME: 2025-02-25 00:36:19 -- STEP: 35118/36938 -- GLOBAL_STEP: 552250 | > loss_text_ce: 0.027877483516931534 (0.03685268631233689) | > loss_mel_ce: 2.6699328422546387 (nan) | > loss: 0.010705597698688507 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.13666239169866196) | > loader_time: 0.0048 (0.004703276127099867)  --> TIME: 2025-02-25 00:36:28 -- STEP: 35168/36938 -- GLOBAL_STEP: 552300 | > loss_text_ce: 0.03428865224123001 (0.03685147313462944) | > loss_mel_ce: 3.035240888595581 (nan) | > loss: 0.012180673889815807 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.13666210877450694) | > loader_time: 0.0049 (0.004703364493469387)  --> TIME: 2025-02-25 00:36:38 -- STEP: 35218/36938 -- GLOBAL_STEP: 552350 | > loss_text_ce: 0.03955460339784622 (0.036852228741110396) | > loss_mel_ce: 3.512439250946045 (nan) | > loss: 0.014095214195549488 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.13667233236414641) | > loader_time: 0.0048 (0.004703308039991281)  --> TIME: 2025-02-25 00:36:47 -- STEP: 35268/36938 -- GLOBAL_STEP: 552400 | > loss_text_ce: 0.05006974935531616 (0.03685080350910173) | > loss_mel_ce: 1.0737533569335938 (nan) | > loss: 0.004459615796804428 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13667144941150688) | > loader_time: 0.0045 (0.004703174964281302)  --> TIME: 2025-02-25 00:36:56 -- STEP: 35318/36938 -- GLOBAL_STEP: 552450 | > loss_text_ce: 0.0389372780919075 (0.03685099384538316) | > loss_mel_ce: 3.152968406677246 (nan) | > loss: 0.012666293419897556 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13666481709546513) | > loader_time: 0.0046 (0.004703141816833316)  --> TIME: 2025-02-25 00:37:06 -- STEP: 35368/36938 -- GLOBAL_STEP: 552500 | > loss_text_ce: 0.025256123393774033 (0.036849004054864) | > loss_mel_ce: 1.3628458976745605 (nan) | > loss: 0.0055083418264985085 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13666840561966273) | > loader_time: 0.0047 (0.004703097363937681)  --> TIME: 2025-02-25 00:37:15 -- STEP: 35418/36938 -- GLOBAL_STEP: 552550 | > loss_text_ce: 0.03945981711149216 (0.03684808931940995) | > loss_mel_ce: 3.4045863151550293 (nan) | > loss: 0.01366685051470995 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13667003107714526) | > loader_time: 0.0045 (0.004702995091237448)  --> TIME: 2025-02-25 00:37:25 -- STEP: 35468/36938 -- GLOBAL_STEP: 552600 | > loss_text_ce: 0.03559984266757965 (0.03685015184248948) | > loss_mel_ce: 2.457526206970215 (nan) | > loss: 0.009893358685076237 (nan) | > current_lr: 5e-06 | > step_time: 0.1533 (0.1366754911240864) | > loader_time: 0.0047 (0.004703157835672439)  --> TIME: 2025-02-25 00:37:34 -- STEP: 35518/36938 -- GLOBAL_STEP: 552650 | > loss_text_ce: 0.031018078327178955 (0.036848234668513564) | > loss_mel_ce: 2.9937379360198975 (nan) | > loss: 0.01200300082564354 (nan) | > current_lr: 5e-06 | > step_time: 0.1688 (0.13667991887114192) | > loader_time: 0.0048 (0.004703155985098148)  --> TIME: 2025-02-25 00:37:43 -- STEP: 35568/36938 -- GLOBAL_STEP: 552700 | > loss_text_ce: 0.040345434099435806 (0.036848771719593186) | > loss_mel_ce: 3.114234209060669 (nan) | > loss: 0.012518174014985561 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13667579591140258) | > loader_time: 0.0051 (0.004703094307174002)  --> TIME: 2025-02-25 00:37:52 -- STEP: 35618/36938 -- GLOBAL_STEP: 552750 | > loss_text_ce: 0.04053616523742676 (0.03684828566475443) | > loss_mel_ce: 2.2587389945983887 (nan) | > loss: 0.009124108590185642 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.136665352578715) | > loader_time: 0.0047 (0.004703113027207713)  --> TIME: 2025-02-25 00:38:02 -- STEP: 35668/36938 -- GLOBAL_STEP: 552800 | > loss_text_ce: 0.036689870059490204 (0.0368471137474933) | > loss_mel_ce: 3.6011054515838623 (nan) | > loss: 0.01443569641560316 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13666977794953594) | > loader_time: 0.0046 (0.004703142416507923)  --> TIME: 2025-02-25 00:38:11 -- STEP: 35718/36938 -- GLOBAL_STEP: 552850 | > loss_text_ce: 0.043894488364458084 (0.036847565017053795) | > loss_mel_ce: 2.8778512477874756 (nan) | > loss: 0.011594230309128761 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.1366705997465873) | > loader_time: 0.0046 (0.004703279972423489)  --> TIME: 2025-02-25 00:38:20 -- STEP: 35768/36938 -- GLOBAL_STEP: 552900 | > loss_text_ce: 0.031919144093990326 (0.03684407364839701) | > loss_mel_ce: 2.875473737716675 (nan) | > loss: 0.011537274345755577 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.13666926062739054) | > loader_time: 0.0048 (0.0047032048613719285)  --> TIME: 2025-02-25 00:38:30 -- STEP: 35818/36938 -- GLOBAL_STEP: 552950 | > loss_text_ce: 0.038757503032684326 (0.03684581038818461) | > loss_mel_ce: 3.1425974369049072 (nan) | > loss: 0.012624424882233143 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13666686355931273) | > loader_time: 0.0045 (0.0047030740530022395)  --> TIME: 2025-02-25 00:38:39 -- STEP: 35868/36938 -- GLOBAL_STEP: 553000 | > loss_text_ce: 0.04599986597895622 (0.036845074847524764) | > loss_mel_ce: 3.102485418319702 (nan) | > loss: 0.012493989430367947 (nan) | > current_lr: 5e-06 | > step_time: 0.1807 (0.13667148064514187) | > loader_time: 0.0048 (0.004702946427701425)  --> TIME: 2025-02-25 00:38:48 -- STEP: 35918/36938 -- GLOBAL_STEP: 553050 | > loss_text_ce: 0.029434524476528168 (0.036843271517090874) | > loss_mel_ce: 2.430065870285034 (nan) | > loss: 0.00975992251187563 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13666550395841823) | > loader_time: 0.0045 (0.004702756722033527)  --> TIME: 2025-02-25 00:38:58 -- STEP: 35968/36938 -- GLOBAL_STEP: 553100 | > loss_text_ce: 0.03503802791237831 (0.03684305494632114) | > loss_mel_ce: 2.52821683883667 (nan) | > loss: 0.010171647183597088 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.1366662167383903) | > loader_time: 0.0045 (0.00470268711103024)  --> TIME: 2025-02-25 00:39:07 -- STEP: 36018/36938 -- GLOBAL_STEP: 553150 | > loss_text_ce: 0.03434819355607033 (0.0368434996796905) | > loss_mel_ce: 2.8162121772766113 (nan) | > loss: 0.011311748065054417 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.1366654401538024) | > loader_time: 0.0048 (0.004702585522871246)  --> TIME: 2025-02-25 00:39:16 -- STEP: 36068/36938 -- GLOBAL_STEP: 553200 | > loss_text_ce: 0.043577082455158234 (0.03684271890536795) | > loss_mel_ce: 1.2984721660614014 (nan) | > loss: 0.005325592588633299 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13666632262833175) | > loader_time: 0.0045 (0.0047025542321616765)  --> TIME: 2025-02-25 00:39:26 -- STEP: 36118/36938 -- GLOBAL_STEP: 553250 | > loss_text_ce: 0.027739843353629112 (0.036842592459416575) | > loss_mel_ce: 3.0409250259399414 (nan) | > loss: 0.01217724196612835 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13666910931506873) | > loader_time: 0.0045 (0.00470245565064514)  --> TIME: 2025-02-25 00:39:35 -- STEP: 36168/36938 -- GLOBAL_STEP: 553300 | > loss_text_ce: 0.04265766590833664 (0.03684149775082398) | > loss_mel_ce: 2.7395594120025635 (nan) | > loss: 0.011040544137358665 (nan) | > current_lr: 5e-06 | > step_time: 0.1706 (0.13666843454333202) | > loader_time: 0.0048 (0.004702349174236221)  --> TIME: 2025-02-25 00:39:44 -- STEP: 36218/36938 -- GLOBAL_STEP: 553350 | > loss_text_ce: 0.04527727887034416 (0.03684159412302114) | > loss_mel_ce: 2.6490390300750732 (nan) | > loss: 0.010691732168197632 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13667181527057107) | > loader_time: 0.0047 (0.004702366038920718)  --> TIME: 2025-02-25 00:39:54 -- STEP: 36268/36938 -- GLOBAL_STEP: 553400 | > loss_text_ce: 0.0458981916308403 (0.036842481343983734) | > loss_mel_ce: 1.8524373769760132 (nan) | > loss: 0.007533078081905842 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.136673848599022) | > loader_time: 0.0046 (0.004702320899039856)  --> TIME: 2025-02-25 00:40:03 -- STEP: 36318/36938 -- GLOBAL_STEP: 553450 | > loss_text_ce: 0.03352056071162224 (0.03684218882786812) | > loss_mel_ce: 1.6926149129867554 (nan) | > loss: 0.0068497443571686745 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.1366744330029274) | > loader_time: 0.0047 (0.0047023214165534755)  --> TIME: 2025-02-25 00:40:12 -- STEP: 36368/36938 -- GLOBAL_STEP: 553500 | > loss_text_ce: 0.03402825817465782 (0.03684078767810953) | > loss_mel_ce: 3.190702199935913 (nan) | > loss: 0.012796550057828426 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13667758397210666) | > loader_time: 0.0046 (0.0047023392922018875)  --> TIME: 2025-02-25 00:40:22 -- STEP: 36418/36938 -- GLOBAL_STEP: 553550 | > loss_text_ce: 0.040647707879543304 (0.03683931480145802) | > loss_mel_ce: 2.320352792739868 (nan) | > loss: 0.009369050152599812 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.13668195810746994) | > loader_time: 0.0047 (0.004702310041276428)  --> TIME: 2025-02-25 00:40:32 -- STEP: 36468/36938 -- GLOBAL_STEP: 553600 | > loss_text_ce: 0.036381132900714874 (0.03683877287710269) | > loss_mel_ce: 3.0137269496917725 (nan) | > loss: 0.012103605084121227 (nan) | > current_lr: 5e-06 | > step_time: 0.1606 (0.13668799385946606) | > loader_time: 0.0049 (0.004702375347551229)  --> TIME: 2025-02-25 00:40:41 -- STEP: 36518/36938 -- GLOBAL_STEP: 553650 | > loss_text_ce: 0.02729151025414467 (0.036839333794364146) | > loss_mel_ce: 3.1961066722869873 (nan) | > loss: 0.012791263870894909 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13668626522376812) | > loader_time: 0.0046 (0.004702323106823896)  --> TIME: 2025-02-25 00:40:51 -- STEP: 36568/36938 -- GLOBAL_STEP: 553700 | > loss_text_ce: 0.031148063018918037 (0.03683944310106321) | > loss_mel_ce: 1.7197394371032715 (nan) | > loss: 0.006947966758161783 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13669388315996395) | > loader_time: 0.0053 (0.004702373644737586)  --> TIME: 2025-02-25 00:41:00 -- STEP: 36618/36938 -- GLOBAL_STEP: 553750 | > loss_text_ce: 0.039029672741889954 (0.036837905091149536) | > loss_mel_ce: 2.5902464389801025 (nan) | > loss: 0.010433635674417019 (nan) | > current_lr: 5e-06 | > step_time: 0.176 (0.13668936691224864) | > loader_time: 0.0048 (0.004702431760132975)  --> TIME: 2025-02-25 00:41:09 -- STEP: 36668/36938 -- GLOBAL_STEP: 553800 | > loss_text_ce: 0.039736319333314896 (0.03683796940370649) | > loss_mel_ce: 2.791424036026001 (nan) | > loss: 0.011234764009714127 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13669528665466726) | > loader_time: 0.0045 (0.004702389103720995)  --> TIME: 2025-02-25 00:41:19 -- STEP: 36718/36938 -- GLOBAL_STEP: 553850 | > loss_text_ce: 0.02238474227488041 (0.036836750405815404) | > loss_mel_ce: 7.15625 (nan) | > loss: 0.028486646711826324 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13669983895491647) | > loader_time: 0.0046 (0.004702540302115542)  --> TIME: 2025-02-25 00:41:28 -- STEP: 36768/36938 -- GLOBAL_STEP: 553900 | > loss_text_ce: 0.029673149809241295 (0.03683670237678102) | > loss_mel_ce: 2.5998010635375977 (nan) | > loss: 0.010434421710669994 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.1367041859694733) | > loader_time: 0.0048 (0.004702523914858779)  --> TIME: 2025-02-25 00:41:38 -- STEP: 36818/36938 -- GLOBAL_STEP: 553950 | > loss_text_ce: 0.03718458488583565 (0.03683598294878511) | > loss_mel_ce: 3.136073350906372 (nan) | > loss: 0.012592293322086334 (nan) | > current_lr: 5e-06 | > step_time: 0.1869 (0.1367083030726385) | > loader_time: 0.0048 (0.004702509566595118)  --> TIME: 2025-02-25 00:41:48 -- STEP: 36868/36938 -- GLOBAL_STEP: 554000 | > loss_text_ce: 0.036514244973659515 (0.03683583443840161) | > loss_mel_ce: 3.0512871742248535 (nan) | > loss: 0.01225318107753992 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.1367141900021697) | > loader_time: 0.0046 (0.004702467048997191)  --> TIME: 2025-02-25 00:41:57 -- STEP: 36918/36938 -- GLOBAL_STEP: 554050 | > loss_text_ce: 0.03248085081577301 (0.036834217010503364) | > loss_mel_ce: 2.524509906768799 (nan) | > loss: 0.0101467901840806 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13671171898327597) | > loader_time: 0.0047 (0.004702394823255852)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.02115940811610459 (+0.0004974214157255803) | > avg_loss_text_ce: 0.03586672161622806 (-0.0005025253677279837) | > avg_loss_mel_ce: 2.629633758917892 (-0.024068501031046186) | > avg_loss: 2.6655004871363706 (-0.024571013627662897) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_554070.pth  > EPOCH: 16/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-25 00:42:25)   --> TIME: 2025-02-25 00:42:31 -- STEP: 30/36938 -- GLOBAL_STEP: 554100 | > loss_text_ce: 0.035974036902189255 (0.03759342251966398) | > loss_mel_ce: 1.5038686990737915 (2.630461774021387) | > loss: 0.0061104875057935715 (0.010587521341706936) | > current_lr: 5e-06 | > step_time: 0.116 (0.1358549197514852) | > loader_time: 0.0049 (0.004687595367431641)  --> TIME: 2025-02-25 00:42:40 -- STEP: 80/36938 -- GLOBAL_STEP: 554150 | > loss_text_ce: 0.05082932487130165 (0.03733002333901824) | > loss_mel_ce: 2.1550328731536865 (nan) | > loss: 0.008753422647714615 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13510200977325437) | > loader_time: 0.0047 (0.00471092164516449)  --> TIME: 2025-02-25 00:42:49 -- STEP: 130/36938 -- GLOBAL_STEP: 554200 | > loss_text_ce: 0.043372057378292084 (0.03744543176144364) | > loss_mel_ce: 2.4412527084350586 (nan) | > loss: 0.00985962338745594 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13388049419109643) | > loader_time: 0.0046 (0.004697845532343936)  --> TIME: 2025-02-25 00:42:59 -- STEP: 180/36938 -- GLOBAL_STEP: 554250 | > loss_text_ce: 0.03845572471618652 (0.03711015604850322) | > loss_mel_ce: 3.4737188816070557 (nan) | > loss: 0.013937201350927353 (nan) | > current_lr: 5e-06 | > step_time: 0.1579 (0.13426521884070505) | > loader_time: 0.0046 (0.004784112506442597)  --> TIME: 2025-02-25 00:43:08 -- STEP: 230/36938 -- GLOBAL_STEP: 554300 | > loss_text_ce: 0.03825424984097481 (0.03706389080085187) | > loss_mel_ce: 2.4801199436187744 (nan) | > loss: 0.009993549436330795 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13445316086644712) | > loader_time: 0.0045 (0.00476380638454271)  --> TIME: 2025-02-25 00:43:18 -- STEP: 280/36938 -- GLOBAL_STEP: 554350 | > loss_text_ce: 0.0386936329305172 (0.036994457883494265) | > loss_mel_ce: 0.0357666015625 (nan) | > loss: 0.00029547716258093715 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13570971574102128) | > loader_time: 0.0046 (0.004751774242946075)  --> TIME: 2025-02-25 00:43:27 -- STEP: 330/36938 -- GLOBAL_STEP: 554400 | > loss_text_ce: 0.030563808977603912 (0.036765540662136956) | > loss_mel_ce: 3.312225818634033 (nan) | > loss: 0.013265038840472698 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.1360635685198235) | > loader_time: 0.0047 (0.004744055054404514)  --> TIME: 2025-02-25 00:43:36 -- STEP: 380/36938 -- GLOBAL_STEP: 554450 | > loss_text_ce: 0.035138752311468124 (0.036608953200476746) | > loss_mel_ce: 2.745541572570801 (nan) | > loss: 0.011034446768462658 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13510120856134528) | > loader_time: 0.0052 (0.00473225430438393)  --> TIME: 2025-02-25 00:43:45 -- STEP: 430/36938 -- GLOBAL_STEP: 554500 | > loss_text_ce: 0.051654890179634094 (0.03664783468513298) | > loss_mel_ce: 2.1257400512695312 (nan) | > loss: 0.008640456013381481 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13473649579425204) | > loader_time: 0.0048 (0.004726521358933558)  --> TIME: 2025-02-25 00:43:55 -- STEP: 480/36938 -- GLOBAL_STEP: 554550 | > loss_text_ce: 0.03268441930413246 (0.03652875738528872) | > loss_mel_ce: 2.6401820182800293 (nan) | > loss: 0.010606613010168076 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.1357412427663804) | > loader_time: 0.0045 (0.004724974930286407)  --> TIME: 2025-02-25 00:44:04 -- STEP: 530/36938 -- GLOBAL_STEP: 554600 | > loss_text_ce: 0.032011665403842926 (0.03640199347908771) | > loss_mel_ce: 3.0696022510528564 (nan) | > loss: 0.01230799313634634 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13578272000798647) | > loader_time: 0.0066 (0.00472596636358297)  --> TIME: 2025-02-25 00:44:13 -- STEP: 580/36938 -- GLOBAL_STEP: 554650 | > loss_text_ce: 0.027854707092046738 (0.03641285193955593) | > loss_mel_ce: 2.4983108043670654 (nan) | > loss: 0.010024466551840305 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13588851320332498) | > loader_time: 0.0045 (0.004722346108535241)  --> TIME: 2025-02-25 00:44:22 -- STEP: 630/36938 -- GLOBAL_STEP: 554700 | > loss_text_ce: 0.03715074807405472 (0.03658259476106321) | > loss_mel_ce: 3.2948715686798096 (nan) | > loss: 0.01322231162339449 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.13558066496773374) | > loader_time: 0.005 (0.004718680608840216)  --> TIME: 2025-02-25 00:44:32 -- STEP: 680/36938 -- GLOBAL_STEP: 554750 | > loss_text_ce: 0.04083613306283951 (0.03650807891972363) | > loss_mel_ce: 1.5329996347427368 (nan) | > loss: 0.006245380267500877 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13600619154817917) | > loader_time: 0.0046 (0.004722728098140044)  --> TIME: 2025-02-25 00:44:41 -- STEP: 730/36938 -- GLOBAL_STEP: 554800 | > loss_text_ce: 0.04023589566349983 (0.03650759388884043) | > loss_mel_ce: 3.439453125 (nan) | > loss: 0.013808291405439377 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13583626322550318) | > loader_time: 0.0046 (0.004717744213261018)  --> TIME: 2025-02-25 00:44:50 -- STEP: 780/36938 -- GLOBAL_STEP: 554850 | > loss_text_ce: 0.034946270287036896 (0.03652968711864487) | > loss_mel_ce: 3.1328673362731934 (nan) | > loss: 0.012570689432322979 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13564377809182174) | > loader_time: 0.0046 (0.004717081938034453)  --> TIME: 2025-02-25 00:45:00 -- STEP: 830/36938 -- GLOBAL_STEP: 554900 | > loss_text_ce: 0.03360097482800484 (0.03647847757057612) | > loss_mel_ce: 2.6078433990478516 (nan) | > loss: 0.010481922887265682 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13596210508461445) | > loader_time: 0.0045 (0.004709019143897369)  --> TIME: 2025-02-25 00:45:09 -- STEP: 880/36938 -- GLOBAL_STEP: 554950 | > loss_text_ce: 0.038396771997213364 (0.03640156135873192) | > loss_mel_ce: 3.2566401958465576 (nan) | > loss: 0.013075544498860836 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.13613958954811084) | > loader_time: 0.0048 (0.004710694605653939)  --> TIME: 2025-02-25 00:45:19 -- STEP: 930/36938 -- GLOBAL_STEP: 555000 | > loss_text_ce: 0.02742791548371315 (0.036432657101421906) | > loss_mel_ce: 1.1551769971847534 (nan) | > loss: 0.00469287671148777 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13635749278529982) | > loader_time: 0.0046 (0.004709713946106617)  --> TIME: 2025-02-25 00:45:28 -- STEP: 980/36938 -- GLOBAL_STEP: 555050 | > loss_text_ce: 0.03481176495552063 (0.0363941020075688) | > loss_mel_ce: 3.1454403400421143 (nan) | > loss: 0.012620048597455025 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1366172805124398) | > loader_time: 0.0054 (0.00471843481063843)  --> TIME: 2025-02-25 00:45:37 -- STEP: 1030/36938 -- GLOBAL_STEP: 555100 | > loss_text_ce: 0.03714883700013161 (0.03635439142964563) | > loss_mel_ce: 0.43745186924934387 (nan) | > loss: 0.0018833362264558673 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.1364425807323269) | > loader_time: 0.0046 (0.004716302353201563)  --> TIME: 2025-02-25 00:45:47 -- STEP: 1080/36938 -- GLOBAL_STEP: 555150 | > loss_text_ce: 0.03292981535196304 (0.0363334833049319) | > loss_mel_ce: 2.7314133644104004 (nan) | > loss: 0.0109696164727211 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.13667137600757437) | > loader_time: 0.0047 (0.004717636549914326)  --> TIME: 2025-02-25 00:45:56 -- STEP: 1130/36938 -- GLOBAL_STEP: 555200 | > loss_text_ce: 0.03870502486824989 (0.036326427964609054) | > loss_mel_ce: 3.4245548248291016 (nan) | > loss: 0.01374309603124857 (nan) | > current_lr: 5e-06 | > step_time: 0.1911 (0.13674943826894825) | > loader_time: 0.0049 (0.00472180737858325)  --> TIME: 2025-02-25 00:46:06 -- STEP: 1180/36938 -- GLOBAL_STEP: 555250 | > loss_text_ce: 0.03258146718144417 (0.03633649841314026) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13688269611132323) | > loader_time: 0.0046 (0.004726025815737448)  --> TIME: 2025-02-25 00:46:15 -- STEP: 1230/36938 -- GLOBAL_STEP: 555300 | > loss_text_ce: 0.03671373426914215 (0.036332149157770076) | > loss_mel_ce: 3.31103515625 (nan) | > loss: 0.013284718617796898 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13686847473547678) | > loader_time: 0.0045 (0.004727840035911495)  --> TIME: 2025-02-25 00:46:25 -- STEP: 1280/36938 -- GLOBAL_STEP: 555350 | > loss_text_ce: 0.04098314046859741 (0.03634053496789426) | > loss_mel_ce: 2.7157163619995117 (nan) | > loss: 0.010939284227788448 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13693613745272132) | > loader_time: 0.0046 (0.004724240489304063)  --> TIME: 2025-02-25 00:46:34 -- STEP: 1330/36938 -- GLOBAL_STEP: 555400 | > loss_text_ce: 0.028411325067281723 (0.036342352623105985) | > loss_mel_ce: 2.2821877002716064 (nan) | > loss: 0.009169044904410839 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.13696888855525405) | > loader_time: 0.0048 (0.0047254657386837096)  --> TIME: 2025-02-25 00:46:43 -- STEP: 1380/36938 -- GLOBAL_STEP: 555450 | > loss_text_ce: 0.03354990482330322 (0.03633025203278104) | > loss_mel_ce: 2.2031195163726807 (nan) | > loss: 0.00887567363679409 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13691580900247502) | > loader_time: 0.0053 (0.004725995098335151)  --> TIME: 2025-02-25 00:46:53 -- STEP: 1430/36938 -- GLOBAL_STEP: 555500 | > loss_text_ce: 0.04172619804739952 (0.036347868585294825) | > loss_mel_ce: 0.014312744140625 (nan) | > loss: 0.0002223767660325393 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13699853853745855) | > loader_time: 0.0046 (0.004726254356490978)  --> TIME: 2025-02-25 00:47:03 -- STEP: 1480/36938 -- GLOBAL_STEP: 555550 | > loss_text_ce: 0.03732601925730705 (0.03635583762834606) | > loss_mel_ce: 2.9484775066375732 (nan) | > loss: 0.01184842735528946 (nan) | > current_lr: 5e-06 | > step_time: 0.1848 (0.13714677143741258) | > loader_time: 0.0048 (0.004725078312126362)  --> TIME: 2025-02-25 00:47:12 -- STEP: 1530/36938 -- GLOBAL_STEP: 555600 | > loss_text_ce: 0.041750311851501465 (0.03634739408354743) | > loss_mel_ce: 2.9096481800079346 (nan) | > loss: 0.011711899191141129 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13698769628612018) | > loader_time: 0.0044 (0.004721103306689289)  --> TIME: 2025-02-25 00:47:21 -- STEP: 1580/36938 -- GLOBAL_STEP: 555650 | > loss_text_ce: 0.042600952088832855 (0.036356444357410964) | > loss_mel_ce: 3.3526666164398193 (nan) | > loss: 0.013473284430801868 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13699865024301036) | > loader_time: 0.0045 (0.004719864114930355)  --> TIME: 2025-02-25 00:47:30 -- STEP: 1630/36938 -- GLOBAL_STEP: 555700 | > loss_text_ce: 0.03447030112147331 (0.036337782241449686) | > loss_mel_ce: 2.9310154914855957 (nan) | > loss: 0.011767801828682423 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.1370935141674578) | > loader_time: 0.0047 (0.0047170927188147724)  --> TIME: 2025-02-25 00:47:40 -- STEP: 1680/36938 -- GLOBAL_STEP: 555750 | > loss_text_ce: 0.029978618025779724 (0.0363438530270719) | > loss_mel_ce: 2.9128689765930176 (nan) | > loss: 0.011677966453135014 (nan) | > current_lr: 5e-06 | > step_time: 0.1777 (0.1370486898081641) | > loader_time: 0.0048 (0.004713399211565647)  --> TIME: 2025-02-25 00:47:49 -- STEP: 1730/36938 -- GLOBAL_STEP: 555800 | > loss_text_ce: 0.03711118549108505 (0.03637725176192769) | > loss_mel_ce: 2.876178026199341 (nan) | > loss: 0.011560672894120216 (nan) | > current_lr: 5e-06 | > step_time: 0.1895 (0.13711229376710202) | > loader_time: 0.0049 (0.00471150130894831)  --> TIME: 2025-02-25 00:47:58 -- STEP: 1780/36938 -- GLOBAL_STEP: 555850 | > loss_text_ce: 0.036636266857385635 (0.036370924542124275) | > loss_mel_ce: 0.0753173828125 (nan) | > loss: 0.00044426051317714155 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13691305966859435) | > loader_time: 0.0048 (0.004709390709909152)  --> TIME: 2025-02-25 00:48:08 -- STEP: 1830/36938 -- GLOBAL_STEP: 555900 | > loss_text_ce: 0.033543217927217484 (0.03639006378648215) | > loss_mel_ce: 1.1314977407455444 (nan) | > loss: 0.004623178858309984 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1369943352996326) | > loader_time: 0.0054 (0.004712670227217536)  --> TIME: 2025-02-25 00:48:17 -- STEP: 1880/36938 -- GLOBAL_STEP: 555950 | > loss_text_ce: 0.030379844829440117 (0.03639437356081962) | > loss_mel_ce: 2.741649627685547 (nan) | > loss: 0.011000117287039757 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13691052629592568) | > loader_time: 0.0045 (0.004714890870642147)  --> TIME: 2025-02-25 00:48:26 -- STEP: 1930/36938 -- GLOBAL_STEP: 556000 | > loss_text_ce: 0.039886049926280975 (0.036397934881149875) | > loss_mel_ce: 0.06396484375 (nan) | > loss: 0.00041210674680769444 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13684401808625055) | > loader_time: 0.0045 (0.0047115330869052004)  --> TIME: 2025-02-25 00:48:35 -- STEP: 1980/36938 -- GLOBAL_STEP: 556050 | > loss_text_ce: 0.03736131265759468 (0.03640539809976782) | > loss_mel_ce: 3.5889222621917725 (nan) | > loss: 0.014390015043318272 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13685149339714434) | > loader_time: 0.0045 (0.0047084748142897405)  --> TIME: 2025-02-25 00:48:45 -- STEP: 2030/36938 -- GLOBAL_STEP: 556100 | > loss_text_ce: 0.030674714595079422 (0.036408859834056555) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1367952017948545) | > loader_time: 0.0045 (0.004706821065818141)  --> TIME: 2025-02-25 00:48:54 -- STEP: 2080/36938 -- GLOBAL_STEP: 556150 | > loss_text_ce: 0.03664840757846832 (0.03637708964480009) | > loss_mel_ce: 3.06381893157959 (nan) | > loss: 0.012303442694246769 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13683150972311311) | > loader_time: 0.0046 (0.004705602503739864)  --> TIME: 2025-02-25 00:49:04 -- STEP: 2130/36938 -- GLOBAL_STEP: 556200 | > loss_text_ce: 0.04257446154952049 (0.03639798638555149) | > loss_mel_ce: 0.974386990070343 (nan) | > loss: 0.004035561345517635 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13706024935547734) | > loader_time: 0.0045 (0.0047044536877126125)  --> TIME: 2025-02-25 00:49:13 -- STEP: 2180/36938 -- GLOBAL_STEP: 556250 | > loss_text_ce: 0.03938619792461395 (0.03640185559057376) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13706461256797164) | > loader_time: 0.0044 (0.004702994254750935)  --> TIME: 2025-02-25 00:49:23 -- STEP: 2230/36938 -- GLOBAL_STEP: 556300 | > loss_text_ce: 0.01333121582865715 (0.03640191934109062) | > loss_mel_ce: 1.5776883363723755 (nan) | > loss: 0.0063135698437690735 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1371396024130921) | > loader_time: 0.0047 (0.004701006359049015)  --> TIME: 2025-02-25 00:49:32 -- STEP: 2280/36938 -- GLOBAL_STEP: 556350 | > loss_text_ce: 0.04094383120536804 (0.03638522048162192) | > loss_mel_ce: 2.817394971847534 (nan) | > loss: 0.011342614889144897 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.1371413946151736) | > loader_time: 0.0046 (0.004699724285226111)  --> TIME: 2025-02-25 00:49:41 -- STEP: 2330/36938 -- GLOBAL_STEP: 556400 | > loss_text_ce: 0.029556339606642723 (0.036386565873279424) | > loss_mel_ce: 2.977226495742798 (nan) | > loss: 0.011931678280234337 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.1370130059033507) | > loader_time: 0.0044 (0.004697940789578803)  --> TIME: 2025-02-25 00:49:51 -- STEP: 2380/36938 -- GLOBAL_STEP: 556450 | > loss_text_ce: 0.037151090800762177 (0.036368438799106374) | > loss_mel_ce: 3.0957541465759277 (nan) | > loss: 0.012432164512574673 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.1370329145623859) | > loader_time: 0.0046 (0.004695957849005684)  --> TIME: 2025-02-25 00:50:00 -- STEP: 2430/36938 -- GLOBAL_STEP: 556500 | > loss_text_ce: 0.04327504709362984 (0.03636230672805457) | > loss_mel_ce: 3.296109437942505 (nan) | > loss: 0.013251527212560177 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13696914523717332) | > loader_time: 0.0048 (0.004693220377949523)  --> TIME: 2025-02-25 00:50:09 -- STEP: 2480/36938 -- GLOBAL_STEP: 556550 | > loss_text_ce: 0.03665770962834358 (0.03635260767506194) | > loss_mel_ce: 2.939866781234741 (nan) | > loss: 0.011811606585979462 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.1369470616502149) | > loader_time: 0.0045 (0.004694892417999995)  --> TIME: 2025-02-25 00:50:18 -- STEP: 2530/36938 -- GLOBAL_STEP: 556600 | > loss_text_ce: 0.04216308891773224 (0.03634054516239657) | > loss_mel_ce: 3.0404725074768066 (nan) | > loss: 0.01223268173635006 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.1369961446452991) | > loader_time: 0.0045 (0.004693244191497674)  --> TIME: 2025-02-25 00:50:28 -- STEP: 2580/36938 -- GLOBAL_STEP: 556650 | > loss_text_ce: 0.024954641237854958 (0.03634261867327456) | > loss_mel_ce: 2.6279942989349365 (nan) | > loss: 0.010527575388550758 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13703061896701213) | > loader_time: 0.0044 (0.004691879971082814)  --> TIME: 2025-02-25 00:50:37 -- STEP: 2630/36938 -- GLOBAL_STEP: 556700 | > loss_text_ce: 0.04612017422914505 (0.036343922526947424) | > loss_mel_ce: 3.2494773864746094 (nan) | > loss: 0.013077768497169018 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13709583454712274) | > loader_time: 0.0046 (0.004690680848328326)  --> TIME: 2025-02-25 00:50:47 -- STEP: 2680/36938 -- GLOBAL_STEP: 556750 | > loss_text_ce: 0.03507108986377716 (0.036347281086995295) | > loss_mel_ce: 2.1224477291107178 (nan) | > loss: 0.008561583235859871 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.1370807565859895) | > loader_time: 0.0046 (0.0046889316679826374)  --> TIME: 2025-02-25 00:50:56 -- STEP: 2730/36938 -- GLOBAL_STEP: 556800 | > loss_text_ce: 0.037477362900972366 (0.03632632456446959) | > loss_mel_ce: 3.4943013191223145 (nan) | > loss: 0.014014994725584984 (nan) | > current_lr: 5e-06 | > step_time: 0.1502 (0.13703464003272992) | > loader_time: 0.0065 (0.0046885763769184685)  --> TIME: 2025-02-25 00:51:05 -- STEP: 2780/36938 -- GLOBAL_STEP: 556850 | > loss_text_ce: 0.029865939170122147 (0.03634662423282854) | > loss_mel_ce: 0.0079803466796875 (nan) | > loss: 0.00015018368139863014 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13698845552883565) | > loader_time: 0.0045 (0.004690447556886739)  --> TIME: 2025-02-25 00:51:14 -- STEP: 2830/36938 -- GLOBAL_STEP: 556900 | > loss_text_ce: 0.03827085718512535 (0.03635860675017838) | > loss_mel_ce: 5.1015625 (nan) | > loss: 0.020396165549755096 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13700292059895) | > loader_time: 0.0046 (0.004688682573001714)  --> TIME: 2025-02-25 00:51:24 -- STEP: 2880/36938 -- GLOBAL_STEP: 556950 | > loss_text_ce: 0.02641213685274124 (0.03634568130946709) | > loss_mel_ce: 1.636552095413208 (nan) | > loss: 0.006599064916372299 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13693389478656975) | > loader_time: 0.0045 (0.004688938541544814)  --> TIME: 2025-02-25 00:51:33 -- STEP: 2930/36938 -- GLOBAL_STEP: 557000 | > loss_text_ce: 0.028125397861003876 (0.0363617790665845) | > loss_mel_ce: 3.198613166809082 (nan) | > loss: 0.012804518453776836 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.136934849101122) | > loader_time: 0.0045 (0.004690578040816276)  --> TIME: 2025-02-25 00:51:43 -- STEP: 2980/36938 -- GLOBAL_STEP: 557050 | > loss_text_ce: 0.0367269366979599 (0.036370263327438035) | > loss_mel_ce: 2.911259889602661 (nan) | > loss: 0.011698361486196518 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13704416711858475) | > loader_time: 0.0046 (0.004690669927021005)  --> TIME: 2025-02-25 00:51:52 -- STEP: 3030/36938 -- GLOBAL_STEP: 557100 | > loss_text_ce: 0.04175424575805664 (0.03639588212047266) | > loss_mel_ce: 3.517075538635254 (nan) | > loss: 0.014122340828180313 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13713222250293214) | > loader_time: 0.0047 (0.004691687432846224)  --> TIME: 2025-02-25 00:52:02 -- STEP: 3080/36938 -- GLOBAL_STEP: 557150 | > loss_text_ce: 0.042268674820661545 (0.03640794896569044) | > loss_mel_ce: 2.8445699214935303 (nan) | > loss: 0.011455710045993328 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13713405550300325) | > loader_time: 0.0046 (0.004690864411267373)  --> TIME: 2025-02-25 00:52:11 -- STEP: 3130/36938 -- GLOBAL_STEP: 557200 | > loss_text_ce: 0.03844602406024933 (0.03639496329743347) | > loss_mel_ce: 2.930412769317627 (nan) | > loss: 0.011781185865402222 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.1370069039134556) | > loader_time: 0.0047 (0.004691601104248831)  --> TIME: 2025-02-25 00:52:20 -- STEP: 3180/36938 -- GLOBAL_STEP: 557250 | > loss_text_ce: 0.03869802877306938 (0.03639927368739953) | > loss_mel_ce: 2.2004098892211914 (nan) | > loss: 0.00888534914702177 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13704131114407936) | > loader_time: 0.0045 (0.004691962401072187)  --> TIME: 2025-02-25 00:52:29 -- STEP: 3230/36938 -- GLOBAL_STEP: 557300 | > loss_text_ce: 0.029633643105626106 (0.03640442222310655) | > loss_mel_ce: 2.824172258377075 (nan) | > loss: 0.011324627324938774 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13701305308209155) | > loader_time: 0.0045 (0.004692481767282403)  --> TIME: 2025-02-25 00:52:39 -- STEP: 3280/36938 -- GLOBAL_STEP: 557350 | > loss_text_ce: 0.04198380932211876 (0.036387220489871974) | > loss_mel_ce: 3.1418566703796387 (nan) | > loss: 0.012634288519620895 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13700783797880528) | > loader_time: 0.0045 (0.004692450747257333)  --> TIME: 2025-02-25 00:52:48 -- STEP: 3330/36938 -- GLOBAL_STEP: 557400 | > loss_text_ce: 0.03675691410899162 (0.03638588169634379) | > loss_mel_ce: 3.4291772842407227 (nan) | > loss: 0.013753708451986313 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13708332709006027) | > loader_time: 0.0045 (0.004690597723196225)  --> TIME: 2025-02-25 00:52:58 -- STEP: 3380/36938 -- GLOBAL_STEP: 557450 | > loss_text_ce: 0.039125245064496994 (0.03641434778161779) | > loss_mel_ce: 0.04150390625 (nan) | > loss: 0.000319956976454705 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13707885184937008) | > loader_time: 0.0045 (0.004689149814244562)  --> TIME: 2025-02-25 00:53:07 -- STEP: 3430/36938 -- GLOBAL_STEP: 557500 | > loss_text_ce: 0.03385820612311363 (0.03640808742737754) | > loss_mel_ce: 2.637549877166748 (nan) | > loss: 0.01060082670301199 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13700732003148053) | > loader_time: 0.0046 (0.004687605932919696)  --> TIME: 2025-02-25 00:53:16 -- STEP: 3480/36938 -- GLOBAL_STEP: 557550 | > loss_text_ce: 0.02680123783648014 (0.036408800236484615) | > loss_mel_ce: 2.980271100997925 (nan) | > loss: 0.011932827532291412 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13702209132841292) | > loader_time: 0.0045 (0.004687127573736785)  --> TIME: 2025-02-25 00:53:26 -- STEP: 3530/36938 -- GLOBAL_STEP: 557600 | > loss_text_ce: 0.04444022476673126 (0.03642321139832594) | > loss_mel_ce: 2.9826574325561523 (nan) | > loss: 0.012012293562293053 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.1370811000424135) | > loader_time: 0.0046 (0.0046891242181275475)  --> TIME: 2025-02-25 00:53:35 -- STEP: 3580/36938 -- GLOBAL_STEP: 557650 | > loss_text_ce: 0.03539138659834862 (0.03643978615447493) | > loss_mel_ce: 3.2959697246551514 (nan) | > loss: 0.013219687156379223 (nan) | > current_lr: 5e-06 | > step_time: 0.1503 (0.137093550732682) | > loader_time: 0.0048 (0.004688387210142681)  --> TIME: 2025-02-25 00:53:44 -- STEP: 3630/36938 -- GLOBAL_STEP: 557700 | > loss_text_ce: 0.029951199889183044 (0.03641307294168614) | > loss_mel_ce: 1.820200800895691 (nan) | > loss: 0.007341873366385698 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1371127886548845) | > loader_time: 0.0045 (0.004687385388313905)  --> TIME: 2025-02-25 00:53:54 -- STEP: 3680/36938 -- GLOBAL_STEP: 557750 | > loss_text_ce: 0.03983553126454353 (0.03641494911128604) | > loss_mel_ce: 3.5614407062530518 (nan) | > loss: 0.014290778897702694 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13707488516102662) | > loader_time: 0.0046 (0.004688122803750247)  --> TIME: 2025-02-25 00:54:03 -- STEP: 3730/36938 -- GLOBAL_STEP: 557800 | > loss_text_ce: 0.032238371670246124 (0.03642480872876073) | > loss_mel_ce: 3.941053867340088 (nan) | > loss: 0.015767034143209457 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.13703877804426332) | > loader_time: 0.0048 (0.00468781927635459)  --> TIME: 2025-02-25 00:54:12 -- STEP: 3780/36938 -- GLOBAL_STEP: 557850 | > loss_text_ce: 0.03462372347712517 (0.0364143380666121) | > loss_mel_ce: 4.012499809265137 (nan) | > loss: 0.016060015186667442 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13698258040443287) | > loader_time: 0.0045 (0.00468688004862064)  --> TIME: 2025-02-25 00:54:21 -- STEP: 3830/36938 -- GLOBAL_STEP: 557900 | > loss_text_ce: 0.03407163918018341 (0.03638394676202377) | > loss_mel_ce: 3.0106608867645264 (nan) | > loss: 0.01208227314054966 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13694410050195427) | > loader_time: 0.0046 (0.004685352427529917)  --> TIME: 2025-02-25 00:54:30 -- STEP: 3880/36938 -- GLOBAL_STEP: 557950 | > loss_text_ce: 0.040809229016304016 (0.03637223361736902) | > loss_mel_ce: 3.255148410797119 (nan) | > loss: 0.013079197145998478 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.1369699492896954) | > loader_time: 0.0047 (0.0046858182272960245)  --> TIME: 2025-02-25 00:54:39 -- STEP: 3930/36938 -- GLOBAL_STEP: 558000 | > loss_text_ce: 0.03821513056755066 (0.03635552145953046) | > loss_mel_ce: 3.571305513381958 (nan) | > loss: 0.014323495328426361 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.1369175578498354) | > loader_time: 0.0045 (0.0046861691632646965)  --> TIME: 2025-02-25 00:54:49 -- STEP: 3980/36938 -- GLOBAL_STEP: 558050 | > loss_text_ce: 0.03510880097746849 (0.0363548731068663) | > loss_mel_ce: 3.03328013420105 (nan) | > loss: 0.012176147662103176 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.1369335705311453) | > loader_time: 0.0045 (0.004685371545091947)  --> TIME: 2025-02-25 00:54:58 -- STEP: 4030/36938 -- GLOBAL_STEP: 558100 | > loss_text_ce: 0.03730664774775505 (0.03635066365268506) | > loss_mel_ce: 1.669145107269287 (nan) | > loss: 0.006771634332835674 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13689453282368083) | > loader_time: 0.0047 (0.00468382592828339)  --> TIME: 2025-02-25 00:55:08 -- STEP: 4080/36938 -- GLOBAL_STEP: 558150 | > loss_text_ce: 0.02577020600438118 (0.03634621902464825) | > loss_mel_ce: 2.7182323932647705 (nan) | > loss: 0.010888899676501751 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13696093226180345) | > loader_time: 0.0045 (0.004684394948622762)  --> TIME: 2025-02-25 00:55:17 -- STEP: 4130/36938 -- GLOBAL_STEP: 558200 | > loss_text_ce: 0.03348073735833168 (0.0363364049332716) | > loss_mel_ce: 3.072045087814331 (nan) | > loss: 0.012323515489697456 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.13701221198204344) | > loader_time: 0.0047 (0.004685485795970115)  --> TIME: 2025-02-25 00:55:27 -- STEP: 4180/36938 -- GLOBAL_STEP: 558250 | > loss_text_ce: 0.02662678435444832 (0.03634664834717627) | > loss_mel_ce: 2.4660117626190186 (nan) | > loss: 0.009891423396766186 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13700645950992701) | > loader_time: 0.0045 (0.004686504573913284)  --> TIME: 2025-02-25 00:55:36 -- STEP: 4230/36938 -- GLOBAL_STEP: 558300 | > loss_text_ce: 0.0429232120513916 (0.0363536244542304) | > loss_mel_ce: 0.09399516880512238 (nan) | > loss: 0.000543326954357326 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13701016215286066) | > loader_time: 0.0052 (0.004686957048186176)  --> TIME: 2025-02-25 00:55:45 -- STEP: 4280/36938 -- GLOBAL_STEP: 558350 | > loss_text_ce: 0.030419575050473213 (0.03636869096923957) | > loss_mel_ce: 1.7521454095840454 (nan) | > loss: 0.007073671091347933 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13698819658466582) | > loader_time: 0.0046 (0.004686101073416597)  --> TIME: 2025-02-25 00:55:55 -- STEP: 4330/36938 -- GLOBAL_STEP: 558400 | > loss_text_ce: 0.0406029149889946 (0.03635066611752524) | > loss_mel_ce: 3.2247302532196045 (nan) | > loss: 0.012957671657204628 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13699678069724766) | > loader_time: 0.0047 (0.004685362381968161)  --> TIME: 2025-02-25 00:56:04 -- STEP: 4380/36938 -- GLOBAL_STEP: 558450 | > loss_text_ce: 0.035185862332582474 (0.036365585753558) | > loss_mel_ce: 2.711592435836792 (nan) | > loss: 0.010899914428591728 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13704411803859548) | > loader_time: 0.0045 (0.004685149726258023)  --> TIME: 2025-02-25 00:56:13 -- STEP: 4430/36938 -- GLOBAL_STEP: 558500 | > loss_text_ce: 0.03694777935743332 (0.03638755453069333) | > loss_mel_ce: 3.071471691131592 (nan) | > loss: 0.012334998697042465 (nan) | > current_lr: 5e-06 | > step_time: 0.1831 (0.13701781822919304) | > loader_time: 0.0049 (0.004685206553198685)  --> TIME: 2025-02-25 00:56:22 -- STEP: 4480/36938 -- GLOBAL_STEP: 558550 | > loss_text_ce: 0.039456479251384735 (0.036386642321303835) | > loss_mel_ce: 3.0225741863250732 (nan) | > loss: 0.012150915339589119 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13694476208516526) | > loader_time: 0.0048 (0.0046854359230824915)  --> TIME: 2025-02-25 00:56:31 -- STEP: 4530/36938 -- GLOBAL_STEP: 558600 | > loss_text_ce: 0.035784006118774414 (0.036383392611440246) | > loss_mel_ce: 2.5368831157684326 (nan) | > loss: 0.010208996944129467 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13683606320395897) | > loader_time: 0.0046 (0.0046860699622046885)  --> TIME: 2025-02-25 00:56:40 -- STEP: 4580/36938 -- GLOBAL_STEP: 558650 | > loss_text_ce: 0.026305969804525375 (0.036377186297874065) | > loss_mel_ce: 2.6854805946350098 (nan) | > loss: 0.010761057958006859 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13681230331612465) | > loader_time: 0.0045 (0.004688617235708448)  --> TIME: 2025-02-25 00:56:49 -- STEP: 4630/36938 -- GLOBAL_STEP: 558700 | > loss_text_ce: 0.03556813299655914 (0.036378506950623606) | > loss_mel_ce: 2.8639180660247803 (nan) | > loss: 0.011505899019539356 (nan) | > current_lr: 5e-06 | > step_time: 0.1781 (0.1367835671546393) | > loader_time: 0.0048 (0.004689236490566841)  --> TIME: 2025-02-25 00:56:59 -- STEP: 4680/36938 -- GLOBAL_STEP: 558750 | > loss_text_ce: 0.04473115876317024 (0.036379678438926935) | > loss_mel_ce: 3.22027587890625 (nan) | > loss: 0.012956378050148487 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.136819395193687) | > loader_time: 0.0048 (0.004688525964052253)  --> TIME: 2025-02-25 00:57:09 -- STEP: 4730/36938 -- GLOBAL_STEP: 558800 | > loss_text_ce: 0.03640810772776604 (0.03638489792934641) | > loss_mel_ce: 2.946080446243286 (nan) | > loss: 0.01183527335524559 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13688408607660346) | > loader_time: 0.0045 (0.0046883657165107985)  --> TIME: 2025-02-25 00:57:18 -- STEP: 4780/36938 -- GLOBAL_STEP: 558850 | > loss_text_ce: 0.04146651178598404 (0.03640169702123481) | > loss_mel_ce: 3.173177719116211 (nan) | > loss: 0.012756525538861752 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13687000763466187) | > loader_time: 0.0047 (0.00468886003334652)  --> TIME: 2025-02-25 00:57:27 -- STEP: 4830/36938 -- GLOBAL_STEP: 558900 | > loss_text_ce: 0.03673996776342392 (0.036390054625252556) | > loss_mel_ce: 2.587294340133667 (nan) | > loss: 0.010412835516035557 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.13684884583727921) | > loader_time: 0.0055 (0.004689444518237382)  --> TIME: 2025-02-25 00:57:37 -- STEP: 4880/36938 -- GLOBAL_STEP: 558950 | > loss_text_ce: 0.03329537436366081 (0.03638887814471302) | > loss_mel_ce: 1.8508453369140625 (nan) | > loss: 0.007476749364286661 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13686538563400022) | > loader_time: 0.0045 (0.004689721201286945)  --> TIME: 2025-02-25 00:57:46 -- STEP: 4930/36938 -- GLOBAL_STEP: 559000 | > loss_text_ce: 0.02945408783853054 (0.03639491261370298) | > loss_mel_ce: 3.231426954269409 (nan) | > loss: 0.012940004467964172 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13687574863433838) | > loader_time: 0.0068 (0.004690944834123037)  --> TIME: 2025-02-25 00:57:55 -- STEP: 4980/36938 -- GLOBAL_STEP: 559050 | > loss_text_ce: 0.0379682332277298 (0.03640478491001412) | > loss_mel_ce: 3.325551986694336 (nan) | > loss: 0.013347302563488483 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13684436231253136) | > loader_time: 0.0046 (0.004691148378762857)  --> TIME: 2025-02-25 00:58:04 -- STEP: 5030/36938 -- GLOBAL_STEP: 559100 | > loss_text_ce: 0.02839430794119835 (0.03641651800628101) | > loss_mel_ce: 0.25626206398010254 (nan) | > loss: 0.0011295888107270002 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13680148181574) | > loader_time: 0.0045 (0.0046906897372327346)  --> TIME: 2025-02-25 00:58:14 -- STEP: 5080/36938 -- GLOBAL_STEP: 559150 | > loss_text_ce: 0.038823943585157394 (0.03641550603568325) | > loss_mel_ce: 2.994246482849121 (nan) | > loss: 0.012035993859171867 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13681858186646698) | > loader_time: 0.0045 (0.004689789897813574)  --> TIME: 2025-02-25 00:58:23 -- STEP: 5130/36938 -- GLOBAL_STEP: 559200 | > loss_text_ce: 0.03329949826002121 (0.03642908815007639) | > loss_mel_ce: 3.0645673274993896 (nan) | > loss: 0.012293122708797455 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13681452213904519) | > loader_time: 0.0045 (0.0046893398199397505)  --> TIME: 2025-02-25 00:58:32 -- STEP: 5180/36938 -- GLOBAL_STEP: 559250 | > loss_text_ce: 0.04699714481830597 (0.036422283273427336) | > loss_mel_ce: 2.8107242584228516 (nan) | > loss: 0.011340164579451084 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13678053410356994) | > loader_time: 0.0047 (0.004689759131103873)  --> TIME: 2025-02-25 00:58:41 -- STEP: 5230/36938 -- GLOBAL_STEP: 559300 | > loss_text_ce: 0.026933183893561363 (0.03640433029640251) | > loss_mel_ce: 2.6432881355285645 (nan) | > loss: 0.010596117004752159 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13674470658056104) | > loader_time: 0.0067 (0.004690490626012396)  --> TIME: 2025-02-25 00:58:51 -- STEP: 5280/36938 -- GLOBAL_STEP: 559350 | > loss_text_ce: 0.02956083044409752 (0.03640766688463763) | > loss_mel_ce: 3.3413445949554443 (nan) | > loss: 0.013376609422266483 (nan) | > current_lr: 5e-06 | > step_time: 0.1782 (0.13674054696704369) | > loader_time: 0.0047 (0.0046917426766771315)  --> TIME: 2025-02-25 00:59:00 -- STEP: 5330/36938 -- GLOBAL_STEP: 559400 | > loss_text_ce: 0.02829844132065773 (0.03640184652944393) | > loss_mel_ce: 2.722318649291992 (nan) | > loss: 0.010915147140622139 (nan) | > current_lr: 5e-06 | > step_time: 0.1556 (0.13676828479230033) | > loader_time: 0.0046 (0.004693315981923985)  --> TIME: 2025-02-25 00:59:10 -- STEP: 5380/36938 -- GLOBAL_STEP: 559450 | > loss_text_ce: 0.039322033524513245 (0.03638946063501468) | > loss_mel_ce: 3.5521814823150635 (nan) | > loss: 0.01425199955701828 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.13683871970300748) | > loader_time: 0.0048 (0.004693423992639147)  --> TIME: 2025-02-25 00:59:19 -- STEP: 5430/36938 -- GLOBAL_STEP: 559500 | > loss_text_ce: 0.04634268954396248 (0.03638049361294263) | > loss_mel_ce: 3.425485849380493 (nan) | > loss: 0.013777097687125206 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13685560059591292) | > loader_time: 0.0045 (0.004693838597222158)  --> TIME: 2025-02-25 00:59:29 -- STEP: 5480/36938 -- GLOBAL_STEP: 559550 | > loss_text_ce: 0.04129825159907341 (0.036393753705498466) | > loss_mel_ce: 3.027522563934326 (nan) | > loss: 0.012177860364317894 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.1368831145937422) | > loader_time: 0.0045 (0.004694830892729942)  --> TIME: 2025-02-25 00:59:38 -- STEP: 5530/36938 -- GLOBAL_STEP: 559600 | > loss_text_ce: 0.028746332973241806 (0.03641010001251219) | > loss_mel_ce: 2.716099500656128 (nan) | > loss: 0.010892245918512344 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.13691706592739142) | > loader_time: 0.0046 (0.004695267703175337)  --> TIME: 2025-02-25 00:59:48 -- STEP: 5580/36938 -- GLOBAL_STEP: 559650 | > loss_text_ce: 0.03149579092860222 (0.0364186390421575) | > loss_mel_ce: 1.020460605621338 (nan) | > loss: 0.004174430388957262 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13690488680289184) | > loader_time: 0.0045 (0.00469528479388111)  --> TIME: 2025-02-25 00:59:57 -- STEP: 5630/36938 -- GLOBAL_STEP: 559700 | > loss_text_ce: 0.038481488823890686 (0.036412514446941616) | > loss_mel_ce: 2.898782968521118 (nan) | > loss: 0.01165581215173006 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13689077048699735) | > loader_time: 0.0045 (0.0046949279033057945)  --> TIME: 2025-02-25 01:00:06 -- STEP: 5680/36938 -- GLOBAL_STEP: 559750 | > loss_text_ce: 0.018641114234924316 (0.03641737640956662) | > loss_mel_ce: 3.636882781982422 (nan) | > loss: 0.01450604759156704 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.1368860438675945) | > loader_time: 0.0044 (0.004693673110344052)  --> TIME: 2025-02-25 01:00:16 -- STEP: 5730/36938 -- GLOBAL_STEP: 559800 | > loss_text_ce: 0.028702324256300926 (0.036407023110633194) | > loss_mel_ce: 0.005433201789855957 (nan) | > loss: 0.00013545845285989344 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.1369284758393051) | > loader_time: 0.0045 (0.004692820978414335)  --> TIME: 2025-02-25 01:00:25 -- STEP: 5780/36938 -- GLOBAL_STEP: 559850 | > loss_text_ce: 0.027106987312436104 (0.036400198364617234) | > loss_mel_ce: 1.3722867965698242 (nan) | > loss: 0.005553150083869696 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13689580471045407) | > loader_time: 0.0045 (0.004691795494317192)  --> TIME: 2025-02-25 01:00:35 -- STEP: 5830/36938 -- GLOBAL_STEP: 559900 | > loss_text_ce: 0.02841721475124359 (0.03639858982048102) | > loss_mel_ce: 2.8114027976989746 (nan) | > loss: 0.011269127018749714 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13693851901163961) | > loader_time: 0.0047 (0.004692436573231411)  --> TIME: 2025-02-25 01:00:44 -- STEP: 5880/36938 -- GLOBAL_STEP: 559950 | > loss_text_ce: 0.04628689959645271 (0.036406034427111884) | > loss_mel_ce: 2.6826090812683105 (nan) | > loss: 0.010828952305018902 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13695540274081575) | > loader_time: 0.0044 (0.004693069993233201)  --> TIME: 2025-02-25 01:00:53 -- STEP: 5930/36938 -- GLOBAL_STEP: 560000 | > loss_text_ce: 0.04767351225018501 (0.03640950938473116) | > loss_mel_ce: 2.806215286254883 (nan) | > loss: 0.011324956081807613 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13692580954019368) | > loader_time: 0.0046 (0.004692959865872416) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_560000.pth  --> TIME: 2025-02-25 01:01:09 -- STEP: 5980/36938 -- GLOBAL_STEP: 560050 | > loss_text_ce: 0.02023456245660782 (0.036411558802902845) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1072 (0.13690163610770945) | > loader_time: 0.0044 (0.004692889695183491)  --> TIME: 2025-02-25 01:01:18 -- STEP: 6030/36938 -- GLOBAL_STEP: 560100 | > loss_text_ce: 0.03789519518613815 (0.03641314582998037) | > loss_mel_ce: 2.8492674827575684 (nan) | > loss: 0.011456995271146297 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13686969960309192) | > loader_time: 0.0046 (0.004692343851029383)  --> TIME: 2025-02-25 01:01:27 -- STEP: 6080/36938 -- GLOBAL_STEP: 560150 | > loss_text_ce: 0.03139001131057739 (0.03641775635815522) | > loss_mel_ce: 3.0044894218444824 (nan) | > loss: 0.01204714085906744 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.13687814372150536) | > loader_time: 0.0048 (0.004693736606522614)  --> TIME: 2025-02-25 01:01:36 -- STEP: 6130/36938 -- GLOBAL_STEP: 560200 | > loss_text_ce: 0.019223803654313087 (0.03641693099260574) | > loss_mel_ce: 3.1287360191345215 (nan) | > loss: 0.012491904199123383 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13684657833510094) | > loader_time: 0.0046 (0.0046938481183075404)  --> TIME: 2025-02-25 01:01:46 -- STEP: 6180/36938 -- GLOBAL_STEP: 560250 | > loss_text_ce: 0.03807952627539635 (0.036427198474454914) | > loss_mel_ce: 3.2070276737213135 (nan) | > loss: 0.012877410277724266 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.1368553481441487) | > loader_time: 0.0046 (0.00469427633439839)  --> TIME: 2025-02-25 01:01:55 -- STEP: 6230/36938 -- GLOBAL_STEP: 560300 | > loss_text_ce: 0.0207118671387434 (0.036430678728752365) | > loss_mel_ce: 3.4765625 (nan) | > loss: 0.013878073543310165 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1368459950289414) | > loader_time: 0.0047 (0.004693803128997169)  --> TIME: 2025-02-25 01:02:04 -- STEP: 6280/36938 -- GLOBAL_STEP: 560350 | > loss_text_ce: 0.025302840396761894 (0.03643708553915568) | > loss_mel_ce: 2.313406467437744 (nan) | > loss: 0.009280593134462833 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13683792109702062) | > loader_time: 0.0047 (0.004693662133186485)  --> TIME: 2025-02-25 01:02:13 -- STEP: 6330/36938 -- GLOBAL_STEP: 560400 | > loss_text_ce: 0.03860326111316681 (0.036433499978963856) | > loss_mel_ce: 2.2877800464630127 (nan) | > loss: 0.009231680072844028 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13684587851519858) | > loader_time: 0.0046 (0.004694141558184631)  --> TIME: 2025-02-25 01:02:23 -- STEP: 6380/36938 -- GLOBAL_STEP: 560450 | > loss_text_ce: 0.04104529321193695 (0.03644098896711315) | > loss_mel_ce: 3.134342908859253 (nan) | > loss: 0.012600746937096119 (nan) | > current_lr: 5e-06 | > step_time: 0.1922 (0.13687535367415646) | > loader_time: 0.0049 (0.004693555906648546)  --> TIME: 2025-02-25 01:02:32 -- STEP: 6430/36938 -- GLOBAL_STEP: 560500 | > loss_text_ce: 0.02692299336194992 (0.036433043557292835) | > loss_mel_ce: 2.9070675373077393 (nan) | > loss: 0.011642820201814175 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1368914851133786) | > loader_time: 0.0045 (0.004692936796422693)  --> TIME: 2025-02-25 01:02:42 -- STEP: 6480/36938 -- GLOBAL_STEP: 560550 | > loss_text_ce: 0.03035879321396351 (0.036435952453937076) | > loss_mel_ce: 1.8665028810501099 (nan) | > loss: 0.007527229376137257 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13690138396657567) | > loader_time: 0.0045 (0.0046929142357390605)  --> TIME: 2025-02-25 01:02:51 -- STEP: 6530/36938 -- GLOBAL_STEP: 560600 | > loss_text_ce: 0.03997381776571274 (0.03642942348286875) | > loss_mel_ce: 2.543623447418213 (nan) | > loss: 0.010252370499074459 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13691107113182488) | > loader_time: 0.0045 (0.004693020978346353)  --> TIME: 2025-02-25 01:03:01 -- STEP: 6580/36938 -- GLOBAL_STEP: 560650 | > loss_text_ce: 0.02810380794107914 (0.03641987345223846) | > loss_mel_ce: 3.152794599533081 (nan) | > loss: 0.012622613459825516 (nan) | > current_lr: 5e-06 | > step_time: 0.1442 (0.13696417178064174) | > loader_time: 0.0046 (0.004693085226969155)  --> TIME: 2025-02-25 01:03:10 -- STEP: 6630/36938 -- GLOBAL_STEP: 560700 | > loss_text_ce: 0.03100932203233242 (0.036410275457053075) | > loss_mel_ce: 6.890625 (nan) | > loss: 0.027466803789138794 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1369353519845338) | > loader_time: 0.0046 (0.0046923894508212214)  --> TIME: 2025-02-25 01:03:19 -- STEP: 6680/36938 -- GLOBAL_STEP: 560750 | > loss_text_ce: 0.030960964038968086 (0.036416882913124504) | > loss_mel_ce: 1.5266602039337158 (nan) | > loss: 0.006181036587804556 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13693307113504746) | > loader_time: 0.0044 (0.0046924430810048896)  --> TIME: 2025-02-25 01:03:29 -- STEP: 6730/36938 -- GLOBAL_STEP: 560800 | > loss_text_ce: 0.031148336827754974 (0.03641680315839879) | > loss_mel_ce: 2.387843370437622 (nan) | > loss: 0.009599174372851849 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1369133672246613) | > loader_time: 0.0048 (0.004692300113490925)  --> TIME: 2025-02-25 01:03:38 -- STEP: 6780/36938 -- GLOBAL_STEP: 560850 | > loss_text_ce: 0.03495124727487564 (0.03641222183122816) | > loss_mel_ce: 3.3585572242736816 (nan) | > loss: 0.013466304168105125 (nan) | > current_lr: 5e-06 | > step_time: 0.1439 (0.1369403159723879) | > loader_time: 0.0053 (0.0046921681865478276)  --> TIME: 2025-02-25 01:03:48 -- STEP: 6830/36938 -- GLOBAL_STEP: 560900 | > loss_text_ce: 0.039474938064813614 (0.03641435350475492) | > loss_mel_ce: 3.055743455886841 (nan) | > loss: 0.012282613664865494 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.13698628008103889) | > loader_time: 0.0047 (0.00469290337485764)  --> TIME: 2025-02-25 01:03:58 -- STEP: 6880/36938 -- GLOBAL_STEP: 560950 | > loss_text_ce: 0.03153540566563606 (0.0364037082644264) | > loss_mel_ce: 1.8104679584503174 (nan) | > loss: 0.007309537380933762 (nan) | > current_lr: 5e-06 | > step_time: 0.1873 (0.1370424156618678) | > loader_time: 0.0049 (0.004693550737791279)  --> TIME: 2025-02-25 01:04:08 -- STEP: 6930/36938 -- GLOBAL_STEP: 561000 | > loss_text_ce: 0.028533944860100746 (0.03640080994427785) | > loss_mel_ce: 2.6388683319091797 (nan) | > loss: 0.010584929957985878 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.1371054737426851) | > loader_time: 0.0045 (0.004693793975257594)  --> TIME: 2025-02-25 01:04:17 -- STEP: 6980/36938 -- GLOBAL_STEP: 561050 | > loss_text_ce: 0.0336889885365963 (0.036390653849021404) | > loss_mel_ce: 2.90171217918396 (nan) | > loss: 0.011648418381810188 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13712168955871287) | > loader_time: 0.0046 (0.004693776625275267)  --> TIME: 2025-02-25 01:04:27 -- STEP: 7030/36938 -- GLOBAL_STEP: 561100 | > loss_text_ce: 0.0345015749335289 (0.036398314469484355) | > loss_mel_ce: 2.864534616470337 (nan) | > loss: 0.011504112742841244 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13713891397668881) | > loader_time: 0.0046 (0.004694299948842898)  --> TIME: 2025-02-25 01:04:36 -- STEP: 7080/36938 -- GLOBAL_STEP: 561150 | > loss_text_ce: 0.028883416205644608 (0.03639849614152014) | > loss_mel_ce: 1.3758149147033691 (nan) | > loss: 0.005574200302362442 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13712199896068955) | > loader_time: 0.0045 (0.004694537254376596)  --> TIME: 2025-02-25 01:04:45 -- STEP: 7130/36938 -- GLOBAL_STEP: 561200 | > loss_text_ce: 0.034796662628650665 (0.03639229436563025) | > loss_mel_ce: 3.16880202293396 (nan) | > loss: 0.012712693773210049 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13709990777608283) | > loader_time: 0.0045 (0.004693818995193542)  --> TIME: 2025-02-25 01:04:54 -- STEP: 7180/36938 -- GLOBAL_STEP: 561250 | > loss_text_ce: 0.02906849980354309 (0.03640100932057431) | > loss_mel_ce: 2.1220176219940186 (nan) | > loss: 0.008536056615412235 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1370917214656613) | > loader_time: 0.0047 (0.004693713659695592)  --> TIME: 2025-02-25 01:05:03 -- STEP: 7230/36938 -- GLOBAL_STEP: 561300 | > loss_text_ce: 0.04069631174206734 (0.036400675475545576) | > loss_mel_ce: 2.5049920082092285 (nan) | > loss: 0.010101938620209694 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13708322747284662) | > loader_time: 0.0047 (0.004693187651627617)  --> TIME: 2025-02-25 01:05:13 -- STEP: 7280/36938 -- GLOBAL_STEP: 561350 | > loss_text_ce: 0.03436204418540001 (0.03640757309308677) | > loss_mel_ce: 2.0750668048858643 (nan) | > loss: 0.008370749652385712 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.1370601516176062) | > loader_time: 0.0046 (0.004692418529437134)  --> TIME: 2025-02-25 01:05:22 -- STEP: 7330/36938 -- GLOBAL_STEP: 561400 | > loss_text_ce: 0.03577839583158493 (0.036403974552484814) | > loss_mel_ce: 2.8732569217681885 (nan) | > loss: 0.011543790809810162 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.1370346379312623) | > loader_time: 0.0045 (0.004692144380910331)  --> TIME: 2025-02-25 01:05:31 -- STEP: 7380/36938 -- GLOBAL_STEP: 561450 | > loss_text_ce: 0.02600044198334217 (0.03639836829372712) | > loss_mel_ce: 2.7706542015075684 (nan) | > loss: 0.011097836308181286 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13698947481346727) | > loader_time: 0.0047 (0.004692069786350898)  --> TIME: 2025-02-25 01:05:40 -- STEP: 7430/36938 -- GLOBAL_STEP: 561500 | > loss_text_ce: 0.030636737123131752 (0.03640066985301596) | > loss_mel_ce: 3.386836290359497 (nan) | > loss: 0.013561401516199112 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.13703313018239277) | > loader_time: 0.0046 (0.004692773831805178)  --> TIME: 2025-02-25 01:05:50 -- STEP: 7480/36938 -- GLOBAL_STEP: 561550 | > loss_text_ce: 0.04350873455405235 (0.03640709934970287) | > loss_mel_ce: 3.166015625 (nan) | > loss: 0.012736208736896515 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.13703631592944418) | > loader_time: 0.0046 (0.0046928332451193055)  --> TIME: 2025-02-25 01:05:59 -- STEP: 7530/36938 -- GLOBAL_STEP: 561600 | > loss_text_ce: 0.04404687136411667 (0.03640695669840385) | > loss_mel_ce: 0.01153564453125 (nan) | > loss: 0.0002205655473517254 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13700670999676498) | > loader_time: 0.0044 (0.004692598579730959)  --> TIME: 2025-02-25 01:06:08 -- STEP: 7580/36938 -- GLOBAL_STEP: 561650 | > loss_text_ce: 0.041530489921569824 (0.036400033981295765) | > loss_mel_ce: 2.6480085849761963 (nan) | > loss: 0.01067277416586876 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13702782886324025) | > loader_time: 0.0046 (0.004692523869486783)  --> TIME: 2025-02-25 01:06:18 -- STEP: 7630/36938 -- GLOBAL_STEP: 561700 | > loss_text_ce: 0.029628906399011612 (0.03639722874493044) | > loss_mel_ce: 2.19427227973938 (nan) | > loss: 0.00882500596344471 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13704771033120883) | > loader_time: 0.0048 (0.004692837763895053)  --> TIME: 2025-02-25 01:06:27 -- STEP: 7680/36938 -- GLOBAL_STEP: 561750 | > loss_text_ce: 0.02938622422516346 (0.03639888669834639) | > loss_mel_ce: 2.5462522506713867 (nan) | > loss: 0.010220788419246674 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.1370609871732699) | > loader_time: 0.0047 (0.004693275100241101)  --> TIME: 2025-02-25 01:06:37 -- STEP: 7730/36938 -- GLOBAL_STEP: 561800 | > loss_text_ce: 0.02750256471335888 (0.03640105723925053) | > loss_mel_ce: 2.577449083328247 (nan) | > loss: 0.010337110608816147 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.1370547757265954) | > loader_time: 0.0046 (0.004693118011504216)  --> TIME: 2025-02-25 01:06:46 -- STEP: 7780/36938 -- GLOBAL_STEP: 561850 | > loss_text_ce: 0.03539061173796654 (0.03640351026871989) | > loss_mel_ce: 2.7103803157806396 (nan) | > loss: 0.010895917192101479 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13705564099902737) | > loader_time: 0.0046 (0.004693833108426362)  --> TIME: 2025-02-25 01:06:55 -- STEP: 7830/36938 -- GLOBAL_STEP: 561900 | > loss_text_ce: 0.04978899285197258 (0.03640157184843067) | > loss_mel_ce: 2.582186222076416 (nan) | > loss: 0.010444346815347672 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.1370265954603492) | > loader_time: 0.0046 (0.004694629689895022)  --> TIME: 2025-02-25 01:07:04 -- STEP: 7880/36938 -- GLOBAL_STEP: 561950 | > loss_text_ce: 0.029167313128709793 (0.03639901348030367) | > loss_mel_ce: 2.759002447128296 (nan) | > loss: 0.011064167134463787 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.13699833780981027) | > loader_time: 0.0047 (0.0046956573040957345)  --> TIME: 2025-02-25 01:07:14 -- STEP: 7930/36938 -- GLOBAL_STEP: 562000 | > loss_text_ce: 0.03258281946182251 (0.036403011961887985) | > loss_mel_ce: 3.052546501159668 (nan) | > loss: 0.012242577038705349 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13700824742960668) | > loader_time: 0.0045 (0.004696110999599227)  --> TIME: 2025-02-25 01:07:23 -- STEP: 7980/36938 -- GLOBAL_STEP: 562050 | > loss_text_ce: 0.040640782564878464 (0.03640584013819258) | > loss_mel_ce: 3.684699296951294 (nan) | > loss: 0.014783096499741077 (nan) | > current_lr: 5e-06 | > step_time: 0.177 (0.13701645622874928) | > loader_time: 0.0048 (0.004696226538273323)  --> TIME: 2025-02-25 01:07:32 -- STEP: 8030/36938 -- GLOBAL_STEP: 562100 | > loss_text_ce: 0.04532022029161453 (0.03640811985212435) | > loss_mel_ce: 3.047912120819092 (nan) | > loss: 0.012274732813239098 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.1369669677311582) | > loader_time: 0.0047 (0.004695687377141091)  --> TIME: 2025-02-25 01:07:42 -- STEP: 8080/36938 -- GLOBAL_STEP: 562150 | > loss_text_ce: 0.0415494441986084 (0.03641296609807365) | > loss_mel_ce: 3.5019519329071045 (nan) | > loss: 0.014061514288187027 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13698936234606443) | > loader_time: 0.005 (0.004695785311189025)  --> TIME: 2025-02-25 01:07:51 -- STEP: 8130/36938 -- GLOBAL_STEP: 562200 | > loss_text_ce: 0.021754397079348564 (0.03641478995188844) | > loss_mel_ce: 1.606695532798767 (nan) | > loss: 0.006462103221565485 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13696861522224182) | > loader_time: 0.0046 (0.004696575507320011)  --> TIME: 2025-02-25 01:08:00 -- STEP: 8180/36938 -- GLOBAL_STEP: 562250 | > loss_text_ce: 0.03601039573550224 (0.03641265877351723) | > loss_mel_ce: 2.7957658767700195 (nan) | > loss: 0.011237208731472492 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13698830735712655) | > loader_time: 0.0047 (0.004696869616986484)  --> TIME: 2025-02-25 01:08:10 -- STEP: 8230/36938 -- GLOBAL_STEP: 562300 | > loss_text_ce: 0.030998660251498222 (0.03641309073083938) | > loss_mel_ce: 1.279024600982666 (nan) | > loss: 0.005198505707085133 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13697470581546076) | > loader_time: 0.0046 (0.004696942244815939)  --> TIME: 2025-02-25 01:08:19 -- STEP: 8280/36938 -- GLOBAL_STEP: 562350 | > loss_text_ce: 0.050562698394060135 (0.03641523862581513) | > loss_mel_ce: 2.5465242862701416 (nan) | > loss: 0.010305901058018208 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13695417841275595) | > loader_time: 0.0045 (0.004696610008460884)  --> TIME: 2025-02-25 01:08:28 -- STEP: 8330/36938 -- GLOBAL_STEP: 562400 | > loss_text_ce: 0.02743779681622982 (0.03641366849967565) | > loss_mel_ce: 2.977729082107544 (nan) | > loss: 0.011925265192985535 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13696845276158318) | > loader_time: 0.0048 (0.004696678685970234)  --> TIME: 2025-02-25 01:08:38 -- STEP: 8380/36938 -- GLOBAL_STEP: 562450 | > loss_text_ce: 0.03603231906890869 (0.03642051521119657) | > loss_mel_ce: 3.3081963062286377 (nan) | > loss: 0.01327074971050024 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.13696062027127823) | > loader_time: 0.0046 (0.00469641483484418)  --> TIME: 2025-02-25 01:08:47 -- STEP: 8430/36938 -- GLOBAL_STEP: 562500 | > loss_text_ce: 0.03383038565516472 (0.03642136466602106) | > loss_mel_ce: 2.7042670249938965 (nan) | > loss: 0.010865466669201851 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13698260017702787) | > loader_time: 0.0047 (0.0046967614163707545)  --> TIME: 2025-02-25 01:08:56 -- STEP: 8480/36938 -- GLOBAL_STEP: 562550 | > loss_text_ce: 0.011345146223902702 (0.03640380212048675) | > loss_mel_ce: 1.7547578811645508 (nan) | > loss: 0.007008345797657967 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13696809104591115) | > loader_time: 0.0044 (0.0046968326534865)  --> TIME: 2025-02-25 01:09:06 -- STEP: 8530/36938 -- GLOBAL_STEP: 562600 | > loss_text_ce: 0.030769186094403267 (0.0364137429765839) | > loss_mel_ce: 1.8112386465072632 (nan) | > loss: 0.007309555541723967 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.1369778735975874) | > loader_time: 0.0048 (0.004696357403941895)  --> TIME: 2025-02-25 01:09:15 -- STEP: 8580/36938 -- GLOBAL_STEP: 562650 | > loss_text_ce: 0.020215045660734177 (0.036407924370648895) | > loss_mel_ce: 2.4313344955444336 (nan) | > loss: 0.00972837209701538 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13698754769105245) | > loader_time: 0.0045 (0.004696381814552074)  --> TIME: 2025-02-25 01:09:25 -- STEP: 8630/36938 -- GLOBAL_STEP: 562700 | > loss_text_ce: 0.03596866503357887 (0.036405255313237246) | > loss_mel_ce: 3.522672414779663 (nan) | > loss: 0.014121592976152897 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13698777807131907) | > loader_time: 0.0055 (0.004696681822713802)  --> TIME: 2025-02-25 01:09:34 -- STEP: 8680/36938 -- GLOBAL_STEP: 562750 | > loss_text_ce: 0.03964310511946678 (0.03640722743126064) | > loss_mel_ce: 3.0051069259643555 (nan) | > loss: 0.012082342058420181 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13698556678635795) | > loader_time: 0.0046 (0.004697076681022815)  --> TIME: 2025-02-25 01:09:43 -- STEP: 8730/36938 -- GLOBAL_STEP: 562800 | > loss_text_ce: 0.04287021979689598 (0.03641283784930027) | > loss_mel_ce: 3.1097726821899414 (nan) | > loss: 0.012510488741099834 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13698153506712746) | > loader_time: 0.0046 (0.004697484604135269)  --> TIME: 2025-02-25 01:09:52 -- STEP: 8780/36938 -- GLOBAL_STEP: 562850 | > loss_text_ce: 0.034732136875391006 (0.03642356614157923) | > loss_mel_ce: 2.639705181121826 (nan) | > loss: 0.010612847283482552 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13694694870425317) | > loader_time: 0.0045 (0.004697068514204778)  --> TIME: 2025-02-25 01:10:02 -- STEP: 8830/36938 -- GLOBAL_STEP: 562900 | > loss_text_ce: 0.04509982839226723 (0.036428707760114) | > loss_mel_ce: 2.3899881839752197 (nan) | > loss: 0.009663048200309277 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.136953658533799) | > loader_time: 0.0046 (0.004697354328591844)  --> TIME: 2025-02-25 01:10:11 -- STEP: 8880/36938 -- GLOBAL_STEP: 562950 | > loss_text_ce: 0.029019279405474663 (0.03642059257348369) | > loss_mel_ce: 2.7805371284484863 (nan) | > loss: 0.011149033904075623 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13696442786100815) | > loader_time: 0.0046 (0.004697398881654472)  --> TIME: 2025-02-25 01:10:20 -- STEP: 8930/36938 -- GLOBAL_STEP: 563000 | > loss_text_ce: 0.04230093955993652 (0.03643015799814473) | > loss_mel_ce: 0.06281280517578125 (nan) | > loss: 0.00041711804806254804 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13696334775858532) | > loader_time: 0.0046 (0.0046976821820386115)  --> TIME: 2025-02-25 01:10:30 -- STEP: 8980/36938 -- GLOBAL_STEP: 563050 | > loss_text_ce: 0.027336642146110535 (0.036421447473044835) | > loss_mel_ce: 3.0606017112731934 (nan) | > loss: 0.012253724038600922 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13696543230511352) | > loader_time: 0.0048 (0.0046980514558227085)  --> TIME: 2025-02-25 01:10:39 -- STEP: 9030/36938 -- GLOBAL_STEP: 563100 | > loss_text_ce: 0.0397086925804615 (0.036426572450400305) | > loss_mel_ce: 3.6809418201446533 (nan) | > loss: 0.014764486812055111 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.1369500799432551) | > loader_time: 0.0047 (0.00469888814395976)  --> TIME: 2025-02-25 01:10:48 -- STEP: 9080/36938 -- GLOBAL_STEP: 563150 | > loss_text_ce: 0.03461132571101189 (0.03642345022847074) | > loss_mel_ce: 2.3411507606506348 (nan) | > loss: 0.009427627548575401 (nan) | > current_lr: 5e-06 | > step_time: 0.1078 (0.13694214377109232) | > loader_time: 0.0045 (0.004699423213362166)  --> TIME: 2025-02-25 01:10:58 -- STEP: 9130/36938 -- GLOBAL_STEP: 563200 | > loss_text_ce: 0.030916353687644005 (0.03642594659709128) | > loss_mel_ce: 3.1895768642425537 (nan) | > loss: 0.012779735960066319 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1369464945819272) | > loader_time: 0.0046 (0.004699460595748248)  --> TIME: 2025-02-25 01:11:07 -- STEP: 9180/36938 -- GLOBAL_STEP: 563250 | > loss_text_ce: 0.038871970027685165 (0.03642454280838557) | > loss_mel_ce: 2.53387188911438 (nan) | > loss: 0.010209301486611366 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13695624900539516) | > loader_time: 0.0046 (0.004699569226350137)  --> TIME: 2025-02-25 01:11:17 -- STEP: 9230/36938 -- GLOBAL_STEP: 563300 | > loss_text_ce: 0.045991767197847366 (0.03643068034163494) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13697429380075968) | > loader_time: 0.0046 (0.004699521126752294)  --> TIME: 2025-02-25 01:11:26 -- STEP: 9280/36938 -- GLOBAL_STEP: 563350 | > loss_text_ce: 0.028235100209712982 (0.03643303392866427) | > loss_mel_ce: 2.1477034091949463 (nan) | > loss: 0.008634677156805992 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1369525812052453) | > loader_time: 0.0044 (0.004699147055888994)  --> TIME: 2025-02-25 01:11:35 -- STEP: 9330/36938 -- GLOBAL_STEP: 563400 | > loss_text_ce: 0.032702505588531494 (0.03643833633146893) | > loss_mel_ce: 2.7250397205352783 (nan) | > loss: 0.010943422093987465 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13691771947813752) | > loader_time: 0.0045 (0.0046985333116619215)  --> TIME: 2025-02-25 01:11:44 -- STEP: 9380/36938 -- GLOBAL_STEP: 563450 | > loss_text_ce: 0.03928146883845329 (0.03644144913997786) | > loss_mel_ce: 3.123051643371582 (nan) | > loss: 0.012548941187560558 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1369024294271658) | > loader_time: 0.0045 (0.004698203799566982)  --> TIME: 2025-02-25 01:11:53 -- STEP: 9430/36938 -- GLOBAL_STEP: 563500 | > loss_text_ce: 0.038042400032281876 (0.03644110179404493) | > loss_mel_ce: 1.68180251121521 (nan) | > loss: 0.006824782118201256 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.1368984869656614) | > loader_time: 0.0047 (0.004698321068400661)  --> TIME: 2025-02-25 01:12:02 -- STEP: 9480/36938 -- GLOBAL_STEP: 563550 | > loss_text_ce: 0.03711528331041336 (0.036436254251649834) | > loss_mel_ce: 3.2279462814331055 (nan) | > loss: 0.012956594116985798 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13687324604404685) | > loader_time: 0.0046 (0.004698140812322551)  --> TIME: 2025-02-25 01:12:12 -- STEP: 9530/36938 -- GLOBAL_STEP: 563600 | > loss_text_ce: 0.03479202091693878 (0.036434691417023914) | > loss_mel_ce: 2.672304630279541 (nan) | > loss: 0.010742447338998318 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13689142101584106) | > loader_time: 0.0047 (0.004698114104936401)  --> TIME: 2025-02-25 01:12:21 -- STEP: 9580/36938 -- GLOBAL_STEP: 563650 | > loss_text_ce: 0.03681246563792229 (0.03642803848048333) | > loss_mel_ce: 2.6832275390625 (nan) | > loss: 0.010793810710310936 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13689051381232653) | > loader_time: 0.0045 (0.004698497368050015)  --> TIME: 2025-02-25 01:12:30 -- STEP: 9630/36938 -- GLOBAL_STEP: 563700 | > loss_text_ce: 0.02820546366274357 (0.036423905994472094) | > loss_mel_ce: 2.4022152423858643 (nan) | > loss: 0.009644526988267899 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.136904273820444) | > loader_time: 0.0045 (0.004698550589730797)  --> TIME: 2025-02-25 01:12:40 -- STEP: 9680/36938 -- GLOBAL_STEP: 563750 | > loss_text_ce: 0.03680284693837166 (0.036428541443137286) | > loss_mel_ce: 3.3076910972595215 (nan) | > loss: 0.013271802105009556 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13695508315543564) | > loader_time: 0.0046 (0.004699055074660237)  --> TIME: 2025-02-25 01:12:50 -- STEP: 9730/36938 -- GLOBAL_STEP: 563800 | > loss_text_ce: 0.038964226841926575 (0.03642195399059964) | > loss_mel_ce: 1.464813232421875 (nan) | > loss: 0.005967371165752411 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.13695831421345453) | > loader_time: 0.0046 (0.004699303483913996)  --> TIME: 2025-02-25 01:12:59 -- STEP: 9780/36938 -- GLOBAL_STEP: 563850 | > loss_text_ce: 0.03202934190630913 (0.03641775083130082) | > loss_mel_ce: 3.3815419673919678 (nan) | > loss: 0.013545919209718704 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13694124831256288) | > loader_time: 0.0046 (0.0046993228555456895)  --> TIME: 2025-02-25 01:13:08 -- STEP: 9830/36938 -- GLOBAL_STEP: 563900 | > loss_text_ce: 0.028132302686572075 (0.03640978924526465) | > loss_mel_ce: 3.1487598419189453 (nan) | > loss: 0.012606714852154255 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13695013438212264) | > loader_time: 0.0052 (0.004699466162998815)  --> TIME: 2025-02-25 01:13:17 -- STEP: 9880/36938 -- GLOBAL_STEP: 563950 | > loss_text_ce: 0.03622979298233986 (0.03640434796297859) | > loss_mel_ce: 2.746643543243408 (nan) | > loss: 0.011043149046599865 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13693379559497681) | > loader_time: 0.0047 (0.004699556258043287)  --> TIME: 2025-02-25 01:13:27 -- STEP: 9930/36938 -- GLOBAL_STEP: 564000 | > loss_text_ce: 0.029851989820599556 (0.03639986540902709) | > loss_mel_ce: 2.0918543338775635 (nan) | > loss: 0.008419469930231571 (nan) | > current_lr: 5e-06 | > step_time: 0.1074 (0.1369328893563547) | > loader_time: 0.0044 (0.004699402201211696)  --> TIME: 2025-02-25 01:13:37 -- STEP: 9980/36938 -- GLOBAL_STEP: 564050 | > loss_text_ce: 0.027538428083062172 (0.036402112638525344) | > loss_mel_ce: 3.2012524604797363 (nan) | > loss: 0.012812663801014423 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.1369764390832683) | > loader_time: 0.0046 (0.004699151095503073)  --> TIME: 2025-02-25 01:13:46 -- STEP: 10030/36938 -- GLOBAL_STEP: 564100 | > loss_text_ce: 0.033057861030101776 (0.036394078355163506) | > loss_mel_ce: 2.5444424152374268 (nan) | > loss: 0.010228176601231098 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1369591657328588) | > loader_time: 0.0045 (0.004698736646239559)  --> TIME: 2025-02-25 01:13:55 -- STEP: 10080/36938 -- GLOBAL_STEP: 564150 | > loss_text_ce: 0.041224583983421326 (0.0363986113774847) | > loss_mel_ce: 0.031005859375 (nan) | > loss: 0.00028662875411100686 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13694119472352345) | > loader_time: 0.0046 (0.004698269897037071)  --> TIME: 2025-02-25 01:14:04 -- STEP: 10130/36938 -- GLOBAL_STEP: 564200 | > loss_text_ce: 0.03970396891236305 (0.036400466610062306) | > loss_mel_ce: 2.0043716430664062 (nan) | > loss: 0.008111411705613136 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.1369410668756278) | > loader_time: 0.0045 (0.004698355482690769)  --> TIME: 2025-02-25 01:14:14 -- STEP: 10180/36938 -- GLOBAL_STEP: 564250 | > loss_text_ce: 0.0175482165068388 (0.03640491848103392) | > loss_mel_ce: 1.551045298576355 (nan) | > loss: 0.006224577780812979 (nan) | > current_lr: 5e-06 | > step_time: 0.1076 (0.13694172472288713) | > loader_time: 0.0045 (0.004698484960379895)  --> TIME: 2025-02-25 01:14:23 -- STEP: 10230/36938 -- GLOBAL_STEP: 564300 | > loss_text_ce: 0.02735205367207527 (0.036405606597446946) | > loss_mel_ce: 2.5171167850494385 (nan) | > loss: 0.010097099468111992 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13695996716574899) | > loader_time: 0.0044 (0.004698529597717751)  --> TIME: 2025-02-25 01:14:33 -- STEP: 10280/36938 -- GLOBAL_STEP: 564350 | > loss_text_ce: 0.041315775364637375 (0.03641117183402787) | > loss_mel_ce: 3.0544240474700928 (nan) | > loss: 0.01228468306362629 (nan) | > current_lr: 5e-06 | > step_time: 0.1574 (0.13697005841982574) | > loader_time: 0.0046 (0.0046982374172730105)  --> TIME: 2025-02-25 01:14:42 -- STEP: 10330/36938 -- GLOBAL_STEP: 564400 | > loss_text_ce: 0.03109336458146572 (0.03640536220982249) | > loss_mel_ce: 1.3352882862091064 (nan) | > loss: 0.005422149784862995 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13698511167509705) | > loader_time: 0.0047 (0.004698601281423643)  --> TIME: 2025-02-25 01:14:52 -- STEP: 10380/36938 -- GLOBAL_STEP: 564450 | > loss_text_ce: 0.03789375722408295 (0.03640838068899157) | > loss_mel_ce: 3.378530263900757 (nan) | > loss: 0.013557239435613155 (nan) | > current_lr: 5e-06 | > step_time: 0.156 (0.1370153927848985) | > loader_time: 0.0047 (0.004699030891778605)  --> TIME: 2025-02-25 01:15:01 -- STEP: 10430/36938 -- GLOBAL_STEP: 564500 | > loss_text_ce: 0.03602338954806328 (0.03641501994078471) | > loss_mel_ce: 1.787554144859314 (nan) | > loss: 0.0072364192456007 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.1370043074067471) | > loader_time: 0.0047 (0.004699390069407439)  --> TIME: 2025-02-25 01:15:10 -- STEP: 10480/36938 -- GLOBAL_STEP: 564550 | > loss_text_ce: 0.04517245292663574 (0.036411316867629634) | > loss_mel_ce: 1.0977449417114258 (nan) | > loss: 0.0045353868044912815 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.1369852976262122) | > loader_time: 0.0045 (0.004699569599319046)  --> TIME: 2025-02-25 01:15:19 -- STEP: 10530/36938 -- GLOBAL_STEP: 564600 | > loss_text_ce: 0.041789378970861435 (0.03640055016350025) | > loss_mel_ce: 3.1337945461273193 (nan) | > loss: 0.012601523660123348 (nan) | > current_lr: 5e-06 | > step_time: 0.1799 (0.1369770692850683) | > loader_time: 0.0048 (0.004699468816447457)  --> TIME: 2025-02-25 01:15:29 -- STEP: 10580/36938 -- GLOBAL_STEP: 564650 | > loss_text_ce: 0.03398844227194786 (0.03640829447581142) | > loss_mel_ce: 0.4332973062992096 (nan) | > loss: 0.0018543086480349302 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.1369699964000514) | > loader_time: 0.0046 (0.004699637285928415)  --> TIME: 2025-02-25 01:15:38 -- STEP: 10630/36938 -- GLOBAL_STEP: 564700 | > loss_text_ce: 0.045145969837903976 (0.036409381828126024) | > loss_mel_ce: 2.900583267211914 (nan) | > loss: 0.011689402163028717 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13694541413943337) | > loader_time: 0.0052 (0.00469975022919763)  --> TIME: 2025-02-25 01:15:47 -- STEP: 10680/36938 -- GLOBAL_STEP: 564750 | > loss_text_ce: 0.03742373734712601 (0.03641295725634917) | > loss_mel_ce: 2.32720947265625 (nan) | > loss: 0.009383466094732285 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13694171809525146) | > loader_time: 0.0045 (0.004699764872311648)  --> TIME: 2025-02-25 01:15:57 -- STEP: 10730/36938 -- GLOBAL_STEP: 564800 | > loss_text_ce: 0.03509895130991936 (0.03641591417192625) | > loss_mel_ce: 2.711887836456299 (nan) | > loss: 0.010900742374360561 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13695357559111748) | > loader_time: 0.0047 (0.004700146139168001)  --> TIME: 2025-02-25 01:16:06 -- STEP: 10780/36938 -- GLOBAL_STEP: 564850 | > loss_text_ce: 0.03283938020467758 (0.03642140328305262) | > loss_mel_ce: 3.4913623332977295 (nan) | > loss: 0.013984927907586098 (nan) | > current_lr: 5e-06 | > step_time: 0.156 (0.1369343117349453) | > loader_time: 0.0049 (0.0047002272614742645)  --> TIME: 2025-02-25 01:16:14 -- STEP: 10830/36938 -- GLOBAL_STEP: 564900 | > loss_text_ce: 0.0389338880777359 (0.03641652975872902) | > loss_mel_ce: 2.624903440475464 (nan) | > loss: 0.010570784099400043 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13690880095639585) | > loader_time: 0.0053 (0.004700426733945101)  --> TIME: 2025-02-25 01:16:24 -- STEP: 10880/36938 -- GLOBAL_STEP: 564950 | > loss_text_ce: 0.04327654093503952 (0.036418110029069764) | > loss_mel_ce: 0.9215234518051147 (nan) | > loss: 0.003828571643680334 (nan) | > current_lr: 5e-06 | > step_time: 0.1086 (0.13691125536228008) | > loader_time: 0.0045 (0.004700402433381355)  --> TIME: 2025-02-25 01:16:34 -- STEP: 10930/36938 -- GLOBAL_STEP: 565000 | > loss_text_ce: 0.03267430514097214 (0.03642014315665386) | > loss_mel_ce: 2.333604335784912 (nan) | > loss: 0.009389995597302914 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13693152167659747) | > loader_time: 0.0045 (0.004700833488143257)  --> TIME: 2025-02-25 01:16:43 -- STEP: 10980/36938 -- GLOBAL_STEP: 565050 | > loss_text_ce: 0.0380583256483078 (0.036422685781181435) | > loss_mel_ce: 1.3921841382980347 (nan) | > loss: 0.005675565451383591 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13691548958067973) | > loader_time: 0.0049 (0.004701005804517004)  --> TIME: 2025-02-25 01:16:52 -- STEP: 11030/36938 -- GLOBAL_STEP: 565100 | > loss_text_ce: 0.03746645525097847 (0.03641993253013047) | > loss_mel_ce: 3.442145824432373 (nan) | > loss: 0.013807985931634903 (nan) | > current_lr: 5e-06 | > step_time: 0.1728 (0.1369079737261229) | > loader_time: 0.0048 (0.004700649876218426)  --> TIME: 2025-02-25 01:17:01 -- STEP: 11080/36938 -- GLOBAL_STEP: 565150 | > loss_text_ce: 0.03813789412379265 (0.03642405961187944) | > loss_mel_ce: 2.590038537979126 (nan) | > loss: 0.010429272428154945 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13688229638747007) | > loader_time: 0.0047 (0.004700889720813456)  --> TIME: 2025-02-25 01:17:10 -- STEP: 11130/36938 -- GLOBAL_STEP: 565200 | > loss_text_ce: 0.029074108228087425 (0.03642418545102623) | > loss_mel_ce: 2.164080858230591 (nan) | > loss: 0.008702997118234634 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13686443634753026) | > loader_time: 0.0046 (0.004700876631826717)  --> TIME: 2025-02-25 01:17:19 -- STEP: 11180/36938 -- GLOBAL_STEP: 565250 | > loss_text_ce: 0.028312845155596733 (0.03642067352488067) | > loss_mel_ce: 2.2822768688201904 (nan) | > loss: 0.009169007651507854 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13685776063496047) | > loader_time: 0.0046 (0.004700590715425378)  --> TIME: 2025-02-25 01:17:28 -- STEP: 11230/36938 -- GLOBAL_STEP: 565300 | > loss_text_ce: 0.029236795380711555 (0.036419178332589175) | > loss_mel_ce: 3.2359507083892822 (nan) | > loss: 0.012957094237208366 (nan) | > current_lr: 5e-06 | > step_time: 0.176 (0.13685378393528264) | > loader_time: 0.005 (0.004700790869796897)  --> TIME: 2025-02-25 01:17:38 -- STEP: 11280/36938 -- GLOBAL_STEP: 565350 | > loss_text_ce: 0.03995230421423912 (0.03642142000054943) | > loss_mel_ce: 2.583441972732544 (nan) | > loss: 0.010410294868052006 (nan) | > current_lr: 5e-06 | > step_time: 0.1529 (0.13685412990286067) | > loader_time: 0.0046 (0.004700508903949814)  --> TIME: 2025-02-25 01:17:47 -- STEP: 11330/36938 -- GLOBAL_STEP: 565400 | > loss_text_ce: 0.0376063697040081 (0.03643082158051451) | > loss_mel_ce: 3.0349934101104736 (nan) | > loss: 0.012192857451736927 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13684389850047513) | > loader_time: 0.0046 (0.004700484006465895)  --> TIME: 2025-02-25 01:17:56 -- STEP: 11380/36938 -- GLOBAL_STEP: 565450 | > loss_text_ce: 0.03039100393652916 (0.0364270660071181) | > loss_mel_ce: 3.268078088760376 (nan) | > loss: 0.013089164160192013 (nan) | > current_lr: 5e-06 | > step_time: 0.1588 (0.1368520759111555) | > loader_time: 0.0046 (0.004700688402346852)  --> TIME: 2025-02-25 01:18:06 -- STEP: 11430/36938 -- GLOBAL_STEP: 565500 | > loss_text_ce: 0.03608040511608124 (0.03642641599049681) | > loss_mel_ce: 2.4338600635528564 (nan) | > loss: 0.009801351465284824 (nan) | > current_lr: 5e-06 | > step_time: 0.1857 (0.13686391024451935) | > loader_time: 0.0056 (0.0047013803416975425)  --> TIME: 2025-02-25 01:18:15 -- STEP: 11480/36938 -- GLOBAL_STEP: 565550 | > loss_text_ce: 0.039872732013463974 (0.03643190172551181) | > loss_mel_ce: 1.3112691640853882 (nan) | > loss: 0.005361674819141626 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.13685624128019414) | > loader_time: 0.0046 (0.004701393905001635)  --> TIME: 2025-02-25 01:18:24 -- STEP: 11530/36938 -- GLOBAL_STEP: 565600 | > loss_text_ce: 0.02798379398882389 (0.03642870629955858) | > loss_mel_ce: 3.11869215965271 (nan) | > loss: 0.01248681079596281 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13685219041802896) | > loader_time: 0.0045 (0.00470106090965832)  --> TIME: 2025-02-25 01:18:33 -- STEP: 11580/36938 -- GLOBAL_STEP: 565650 | > loss_text_ce: 0.0383182168006897 (0.0364259712081979) | > loss_mel_ce: 2.2187111377716064 (nan) | > loss: 0.008956465870141983 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1368286643209446) | > loader_time: 0.0045 (0.004700516460267152)  --> TIME: 2025-02-25 01:18:43 -- STEP: 11630/36938 -- GLOBAL_STEP: 565700 | > loss_text_ce: 0.04029533639550209 (0.03642954896176121) | > loss_mel_ce: 1.7626303434371948 (nan) | > loss: 0.007154467515647411 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13681074943608143) | > loader_time: 0.0045 (0.004700092724039944)  --> TIME: 2025-02-25 01:18:52 -- STEP: 11680/36938 -- GLOBAL_STEP: 565750 | > loss_text_ce: 0.03991062194108963 (0.0364240608032159) | > loss_mel_ce: 3.223419189453125 (nan) | > loss: 0.01294972188770771 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13679541941375026) | > loader_time: 0.0047 (0.004699684332494852)  --> TIME: 2025-02-25 01:19:01 -- STEP: 11730/36938 -- GLOBAL_STEP: 565800 | > loss_text_ce: 0.029657093808054924 (0.03642494720267862) | > loss_mel_ce: 1.5094960927963257 (nan) | > loss: 0.006107751280069351 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13680479353752767) | > loader_time: 0.0046 (0.00469967018104675)  --> TIME: 2025-02-25 01:19:10 -- STEP: 11780/36938 -- GLOBAL_STEP: 565850 | > loss_text_ce: 0.0405256450176239 (0.03642386793513113) | > loss_mel_ce: 2.8980190753936768 (nan) | > loss: 0.011660892516374588 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13679825632195783) | > loader_time: 0.0045 (0.004699739575993209)  --> TIME: 2025-02-25 01:19:20 -- STEP: 11830/36938 -- GLOBAL_STEP: 565900 | > loss_text_ce: 0.03796207159757614 (0.036425426342962884) | > loss_mel_ce: 2.08203125 (nan) | > loss: 0.008412672206759453 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13679716286776195) | > loader_time: 0.0045 (0.004699979469318648)  --> TIME: 2025-02-25 01:19:29 -- STEP: 11880/36938 -- GLOBAL_STEP: 565950 | > loss_text_ce: 0.03285999223589897 (0.03641731262070605) | > loss_mel_ce: 3.5024654865264893 (nan) | > loss: 0.014029070734977722 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.13680116961700814) | > loader_time: 0.0047 (0.004700144433011896)  --> TIME: 2025-02-25 01:19:39 -- STEP: 11930/36938 -- GLOBAL_STEP: 566000 | > loss_text_ce: 0.04195236414670944 (0.03642029439864936) | > loss_mel_ce: 2.9876952171325684 (nan) | > loss: 0.012022411450743675 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13680021926723274) | > loader_time: 0.0045 (0.004700030824864289)  --> TIME: 2025-02-25 01:19:48 -- STEP: 11980/36938 -- GLOBAL_STEP: 566050 | > loss_text_ce: 0.04623107984662056 (0.036417869968146804) | > loss_mel_ce: 2.805194616317749 (nan) | > loss: 0.011315181851387024 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1367981846026075) | > loader_time: 0.0045 (0.0046999885164238486)  --> TIME: 2025-02-25 01:19:57 -- STEP: 12030/36938 -- GLOBAL_STEP: 566100 | > loss_text_ce: 0.03443547338247299 (0.0364177242911582) | > loss_mel_ce: 3.1163694858551025 (nan) | > loss: 0.01250319555401802 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13678136730035603) | > loader_time: 0.0046 (0.004700114839986664)  --> TIME: 2025-02-25 01:20:06 -- STEP: 12080/36938 -- GLOBAL_STEP: 566150 | > loss_text_ce: 0.04119950160384178 (0.036414087912921805) | > loss_mel_ce: 2.286672830581665 (nan) | > loss: 0.009237589314579964 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13678019419805962) | > loader_time: 0.0047 (0.004699858707307971)  --> TIME: 2025-02-25 01:20:16 -- STEP: 12130/36938 -- GLOBAL_STEP: 566200 | > loss_text_ce: 0.04142681881785393 (0.03641465987345247) | > loss_mel_ce: 1.6850801706314087 (nan) | > loss: 0.006851218640804291 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13678384888594794) | > loader_time: 0.0047 (0.00469954123894045)  --> TIME: 2025-02-25 01:20:25 -- STEP: 12180/36938 -- GLOBAL_STEP: 566250 | > loss_text_ce: 0.03631076589226723 (0.036414653031852616) | > loss_mel_ce: 2.8472797870635986 (nan) | > loss: 0.011442819610238075 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.1367833868623373) | > loader_time: 0.0046 (0.0046992290391906204)  --> TIME: 2025-02-25 01:20:34 -- STEP: 12230/36938 -- GLOBAL_STEP: 566300 | > loss_text_ce: 0.02822115272283554 (0.036418245075832315) | > loss_mel_ce: 1.7558211088180542 (nan) | > loss: 0.00707953330129385 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13678227190296927) | > loader_time: 0.0045 (0.004698967875171067)  --> TIME: 2025-02-25 01:20:43 -- STEP: 12280/36938 -- GLOBAL_STEP: 566350 | > loss_text_ce: 0.036446597427129745 (0.036413159585824625) | > loss_mel_ce: 3.3996236324310303 (nan) | > loss: 0.013635200448334217 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13676098724918248) | > loader_time: 0.0045 (0.004699086715810062)  --> TIME: 2025-02-25 01:20:52 -- STEP: 12330/36938 -- GLOBAL_STEP: 566400 | > loss_text_ce: 0.039821233600378036 (0.0364094220011947) | > loss_mel_ce: 2.5877878665924072 (nan) | > loss: 0.010427020490169525 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.1367625992373544) | > loader_time: 0.0045 (0.004699320514706788)  --> TIME: 2025-02-25 01:21:02 -- STEP: 12380/36938 -- GLOBAL_STEP: 566450 | > loss_text_ce: 0.03660458326339722 (0.036407759421172345) | > loss_mel_ce: 3.23591947555542 (nan) | > loss: 0.012986207380890846 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13677746255871612) | > loader_time: 0.0052 (0.004699697883525831)  --> TIME: 2025-02-25 01:21:11 -- STEP: 12430/36938 -- GLOBAL_STEP: 566500 | > loss_text_ce: 0.033039748668670654 (0.03641099790201075) | > loss_mel_ce: 2.113034248352051 (nan) | > loss: 0.008516167290508747 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13676263860225354) | > loader_time: 0.0045 (0.004699991464806664)  --> TIME: 2025-02-25 01:21:20 -- STEP: 12480/36938 -- GLOBAL_STEP: 566550 | > loss_text_ce: 0.0353088341653347 (0.03641253223550154) | > loss_mel_ce: 2.491722583770752 (nan) | > loss: 0.010027903132140636 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13673013721903213) | > loader_time: 0.0046 (0.004699614701362743)  --> TIME: 2025-02-25 01:21:29 -- STEP: 12530/36938 -- GLOBAL_STEP: 566600 | > loss_text_ce: 0.037616536021232605 (0.03641605441222889) | > loss_mel_ce: 2.419265031814575 (nan) | > loss: 0.00974953081458807 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13673255512073187) | > loader_time: 0.0045 (0.004699531937254386)  --> TIME: 2025-02-25 01:21:38 -- STEP: 12580/36938 -- GLOBAL_STEP: 566650 | > loss_text_ce: 0.03830145299434662 (0.03641410749346384) | > loss_mel_ce: 3.2780489921569824 (nan) | > loss: 0.013160121627151966 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.1367030466304111) | > loader_time: 0.0047 (0.0046991841781688845)  --> TIME: 2025-02-25 01:21:48 -- STEP: 12630/36938 -- GLOBAL_STEP: 566700 | > loss_text_ce: 0.037263404577970505 (0.03641125157634736) | > loss_mel_ce: 2.306049346923828 (nan) | > loss: 0.009298861026763916 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13670619542128734) | > loader_time: 0.0052 (0.004699399333479648)  --> TIME: 2025-02-25 01:21:57 -- STEP: 12680/36938 -- GLOBAL_STEP: 566750 | > loss_text_ce: 0.03810650110244751 (0.036411484073089424) | > loss_mel_ce: 2.5200912952423096 (nan) | > loss: 0.010151579044759274 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13669618229008676) | > loader_time: 0.0046 (0.004700043524853422)  --> TIME: 2025-02-25 01:22:06 -- STEP: 12730/36938 -- GLOBAL_STEP: 566800 | > loss_text_ce: 0.05299268662929535 (0.036411468216012356) | > loss_mel_ce: 0.06127624586224556 (nan) | > loss: 0.0004534481558948755 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13669060814221504) | > loader_time: 0.0045 (0.00470149277329724)  --> TIME: 2025-02-25 01:22:15 -- STEP: 12780/36938 -- GLOBAL_STEP: 566850 | > loss_text_ce: 0.02605252154171467 (0.036409879929979155) | > loss_mel_ce: 2.367133140563965 (nan) | > loss: 0.009496768936514854 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13670116579028974) | > loader_time: 0.0046 (0.004701563860515644)  --> TIME: 2025-02-25 01:22:25 -- STEP: 12830/36938 -- GLOBAL_STEP: 566900 | > loss_text_ce: 0.047565311193466187 (0.03641492209875611) | > loss_mel_ce: 1.99800443649292 (nan) | > loss: 0.008117340505123138 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.1366993123817895) | > loader_time: 0.0047 (0.004701488276338153)  --> TIME: 2025-02-25 01:22:35 -- STEP: 12880/36938 -- GLOBAL_STEP: 566950 | > loss_text_ce: 0.042783912271261215 (0.03641489199796532) | > loss_mel_ce: 2.177734375 (nan) | > loss: 0.008811580948531628 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13673994098761125) | > loader_time: 0.0047 (0.004701843061802533)  --> TIME: 2025-02-25 01:22:44 -- STEP: 12930/36938 -- GLOBAL_STEP: 567000 | > loss_text_ce: 0.036219432950019836 (0.036413424215743985) | > loss_mel_ce: 3.0925118923187256 (nan) | > loss: 0.012415600940585136 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.1367386359165371) | > loader_time: 0.0048 (0.004701793092401636)  --> TIME: 2025-02-25 01:22:53 -- STEP: 12980/36938 -- GLOBAL_STEP: 567050 | > loss_text_ce: 0.036846935749053955 (0.03641207545496316) | > loss_mel_ce: 0.14899088442325592 (nan) | > loss: 0.0007374517153948545 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.1367390490459923) | > loader_time: 0.0047 (0.004701631976569905)  --> TIME: 2025-02-25 01:23:02 -- STEP: 13030/36938 -- GLOBAL_STEP: 567100 | > loss_text_ce: 0.038314152508974075 (0.03641335073392472) | > loss_mel_ce: 2.8728840351104736 (nan) | > loss: 0.011552373878657818 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13672733383735014) | > loader_time: 0.0048 (0.004701410360914487)  --> TIME: 2025-02-25 01:23:12 -- STEP: 13080/36938 -- GLOBAL_STEP: 567150 | > loss_text_ce: 0.033193305134773254 (0.03641311493403179) | > loss_mel_ce: 2.9972879886627197 (nan) | > loss: 0.012025720439851284 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.136723494894279) | > loader_time: 0.0046 (0.004701169368323907)  --> TIME: 2025-02-25 01:23:21 -- STEP: 13130/36938 -- GLOBAL_STEP: 567200 | > loss_text_ce: 0.032755110412836075 (0.03641440261152524) | > loss_mel_ce: 3.4095115661621094 (nan) | > loss: 0.013659789226949215 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13671939621785223) | > loader_time: 0.0047 (0.0047009032460813205)  --> TIME: 2025-02-25 01:23:30 -- STEP: 13180/36938 -- GLOBAL_STEP: 567250 | > loss_text_ce: 0.03227261081337929 (0.036411581080610364) | > loss_mel_ce: 3.448012113571167 (nan) | > loss: 0.013810654170811176 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13671990555226407) | > loader_time: 0.0047 (0.0047006121354689)  --> TIME: 2025-02-25 01:23:40 -- STEP: 13230/36938 -- GLOBAL_STEP: 567300 | > loss_text_ce: 0.03843686357140541 (0.03640696540931046) | > loss_mel_ce: 1.6123589277267456 (nan) | > loss: 0.0065507772378623486 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13672449475243045) | > loader_time: 0.0047 (0.004701035038954527)  --> TIME: 2025-02-25 01:23:49 -- STEP: 13280/36938 -- GLOBAL_STEP: 567350 | > loss_text_ce: 0.044559773057699203 (0.03640865013690613) | > loss_mel_ce: 3.1107699871063232 (nan) | > loss: 0.012521150521934032 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1367236641695707) | > loader_time: 0.0045 (0.004700906054083106)  --> TIME: 2025-02-25 01:23:58 -- STEP: 13330/36938 -- GLOBAL_STEP: 567400 | > loss_text_ce: 0.02974860370159149 (0.03641123426218763) | > loss_mel_ce: 2.7012434005737305 (nan) | > loss: 0.010837270878255367 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13671479291336458) | > loader_time: 0.0066 (0.004701212412955056)  --> TIME: 2025-02-25 01:24:07 -- STEP: 13380/36938 -- GLOBAL_STEP: 567450 | > loss_text_ce: 0.04875402897596359 (0.03641418577749584) | > loss_mel_ce: 1.2296738624572754 (nan) | > loss: 0.005073126871138811 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.136714236885978) | > loader_time: 0.0045 (0.004701617979326211)  --> TIME: 2025-02-25 01:24:17 -- STEP: 13430/36938 -- GLOBAL_STEP: 567500 | > loss_text_ce: 0.03567301854491234 (0.03641547367589752) | > loss_mel_ce: 2.926778793334961 (nan) | > loss: 0.011755761690437794 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13672387317677404) | > loader_time: 0.0047 (0.004701747098966735)  --> TIME: 2025-02-25 01:24:26 -- STEP: 13480/36938 -- GLOBAL_STEP: 567550 | > loss_text_ce: 0.0350407212972641 (0.03641939033610456) | > loss_mel_ce: 2.9398653507232666 (nan) | > loss: 0.011805183254182339 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13672927998293818) | > loader_time: 0.0049 (0.004701901295772272)  --> TIME: 2025-02-25 01:24:36 -- STEP: 13530/36938 -- GLOBAL_STEP: 567600 | > loss_text_ce: 0.06068576127290726 (0.03641815664121391) | > loss_mel_ce: 1.1644757986068726 (nan) | > loss: 0.00486175250262022 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1367365281492015) | > loader_time: 0.0045 (0.004701809784437388)  --> TIME: 2025-02-25 01:24:45 -- STEP: 13580/36938 -- GLOBAL_STEP: 567650 | > loss_text_ce: 0.040905099362134933 (0.03642089692354523) | > loss_mel_ce: 3.064443826675415 (nan) | > loss: 0.012322813272476196 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.1367514923149079) | > loader_time: 0.0048 (0.004702031489500047)  --> TIME: 2025-02-25 01:24:54 -- STEP: 13630/36938 -- GLOBAL_STEP: 567700 | > loss_text_ce: 0.02634654939174652 (0.03642132406916273) | > loss_mel_ce: 2.8273746967315674 (nan) | > loss: 0.011324291117489338 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13674513329850901) | > loader_time: 0.0046 (0.004701853480657394)  --> TIME: 2025-02-25 01:25:04 -- STEP: 13680/36938 -- GLOBAL_STEP: 567750 | > loss_text_ce: 0.027911117300391197 (0.03642703568990396) | > loss_mel_ce: 2.015801429748535 (nan) | > loss: 0.008109970949590206 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13675313961436217) | > loader_time: 0.0045 (0.00470187555976777)  --> TIME: 2025-02-25 01:25:13 -- STEP: 13730/36938 -- GLOBAL_STEP: 567800 | > loss_text_ce: 0.03723912313580513 (0.036423425285657035) | > loss_mel_ce: 2.8628649711608887 (nan) | > loss: 0.011508350260555744 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.13675212434670078) | > loader_time: 0.0047 (0.00470185491696639)  --> TIME: 2025-02-25 01:25:22 -- STEP: 13780/36938 -- GLOBAL_STEP: 567850 | > loss_text_ce: 0.045360248535871506 (0.036422630593164355) | > loss_mel_ce: 2.738661527633667 (nan) | > loss: 0.011047706007957458 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13674000724825733) | > loader_time: 0.0047 (0.004701680749524992)  --> TIME: 2025-02-25 01:25:32 -- STEP: 13830/36938 -- GLOBAL_STEP: 567900 | > loss_text_ce: 0.048755235970020294 (0.036418128953283606) | > loss_mel_ce: 2.8919308185577393 (nan) | > loss: 0.01166938990354538 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13674473010881727) | > loader_time: 0.0045 (0.004701690163546842)  --> TIME: 2025-02-25 01:25:41 -- STEP: 13880/36938 -- GLOBAL_STEP: 567950 | > loss_text_ce: 0.03731239214539528 (0.03641641754301404) | > loss_mel_ce: 3.2341091632843018 (nan) | > loss: 0.012981832958757877 (nan) | > current_lr: 5e-06 | > step_time: 0.1894 (0.13675267491972784) | > loader_time: 0.0048 (0.004701633463675405)  --> TIME: 2025-02-25 01:25:50 -- STEP: 13930/36938 -- GLOBAL_STEP: 568000 | > loss_text_ce: 0.0390026830136776 (0.03641194712888102) | > loss_mel_ce: 2.796027183532715 (nan) | > loss: 0.011250118725001812 (nan) | > current_lr: 5e-06 | > step_time: 0.1081 (0.1367285578523024) | > loader_time: 0.0046 (0.004701415652092278)  --> TIME: 2025-02-25 01:25:59 -- STEP: 13980/36938 -- GLOBAL_STEP: 568050 | > loss_text_ce: 0.03443331643939018 (0.03641619171010579) | > loss_mel_ce: 2.8258628845214844 (nan) | > loss: 0.011350383050739765 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13672898037409778) | > loader_time: 0.0045 (0.004701515550436025)  --> TIME: 2025-02-25 01:26:09 -- STEP: 14030/36938 -- GLOBAL_STEP: 568100 | > loss_text_ce: 0.029229862615466118 (0.036417279164329956) | > loss_mel_ce: 2.790900230407715 (nan) | > loss: 0.011190992780029774 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13675343730325643) | > loader_time: 0.0046 (0.004701715864968642)  --> TIME: 2025-02-25 01:26:18 -- STEP: 14080/36938 -- GLOBAL_STEP: 568150 | > loss_text_ce: 0.0325574167072773 (0.03642085313721584) | > loss_mel_ce: 3.1189253330230713 (nan) | > loss: 0.01250588521361351 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.13674504369158647) | > loader_time: 0.0047 (0.004702147333459406)  --> TIME: 2025-02-25 01:26:28 -- STEP: 14130/36938 -- GLOBAL_STEP: 568200 | > loss_text_ce: 0.029228802770376205 (0.036419215959925796) | > loss_mel_ce: 3.3530468940734863 (nan) | > loss: 0.01342172920703888 (nan) | > current_lr: 5e-06 | > step_time: 0.1871 (0.1367474822566093) | > loader_time: 0.0047 (0.004702217124998345)  --> TIME: 2025-02-25 01:26:37 -- STEP: 14180/36938 -- GLOBAL_STEP: 568250 | > loss_text_ce: 0.032046955078840256 (0.03642130644016002) | > loss_mel_ce: 3.071901559829712 (nan) | > loss: 0.01231725700199604 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13675188280127107) | > loader_time: 0.0046 (0.004702065054889445)  --> TIME: 2025-02-25 01:26:46 -- STEP: 14230/36938 -- GLOBAL_STEP: 568300 | > loss_text_ce: 0.034957244992256165 (0.03641546725885161) | > loss_mel_ce: 2.609724998474121 (nan) | > loss: 0.010494771413505077 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13673717724296788) | > loader_time: 0.0045 (0.004702673910037768)  --> TIME: 2025-02-25 01:26:56 -- STEP: 14280/36938 -- GLOBAL_STEP: 568350 | > loss_text_ce: 0.03005875274538994 (0.036420540715677255) | > loss_mel_ce: 2.963911533355713 (nan) | > loss: 0.011880835518240929 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.1367471964753306) | > loader_time: 0.0047 (0.004702924446565416)  --> TIME: 2025-02-25 01:27:05 -- STEP: 14330/36938 -- GLOBAL_STEP: 568400 | > loss_text_ce: 0.03500409796833992 (0.036418047737049354) | > loss_mel_ce: 1.3844674825668335 (nan) | > loss: 0.0056328242644667625 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13674190157539562) | > loader_time: 0.0045 (0.004703092641624123)  --> TIME: 2025-02-25 01:27:15 -- STEP: 14380/36938 -- GLOBAL_STEP: 568450 | > loss_text_ce: 0.0368528850376606 (0.036422153198651074) | > loss_mel_ce: 3.2563772201538086 (nan) | > loss: 0.013068374246358871 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.1367526465728979) | > loader_time: 0.0048 (0.004703023055664176)  --> TIME: 2025-02-25 01:27:24 -- STEP: 14430/36938 -- GLOBAL_STEP: 568500 | > loss_text_ce: 0.026070863008499146 (0.036420858210521885) | > loss_mel_ce: 1.1807982921600342 (nan) | > loss: 0.0047891633585095406 (nan) | > current_lr: 5e-06 | > step_time: 0.1349 (0.13675501382937433) | > loader_time: 0.0046 (0.004703068799305015)  --> TIME: 2025-02-25 01:27:33 -- STEP: 14480/36938 -- GLOBAL_STEP: 568550 | > loss_text_ce: 0.028903085738420486 (0.03641955506150443) | > loss_mel_ce: 2.529412269592285 (nan) | > loss: 0.010152045637369156 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.13675298289040794) | > loader_time: 0.0048 (0.004703053469815952)  --> TIME: 2025-02-25 01:27:43 -- STEP: 14530/36938 -- GLOBAL_STEP: 568600 | > loss_text_ce: 0.050812169909477234 (0.03642102799064895) | > loss_mel_ce: 3.429201364517212 (nan) | > loss: 0.013809578493237495 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.13675580877972193) | > loader_time: 0.0049 (0.004703378267150361)  --> TIME: 2025-02-25 01:27:52 -- STEP: 14580/36938 -- GLOBAL_STEP: 568650 | > loss_text_ce: 0.04257996380329132 (0.036420701560851705) | > loss_mel_ce: 2.0382211208343506 (nan) | > loss: 0.008257146924734116 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.1367559532896804) | > loader_time: 0.0045 (0.004703361719887593)  --> TIME: 2025-02-25 01:28:01 -- STEP: 14630/36938 -- GLOBAL_STEP: 568700 | > loss_text_ce: 0.03450721502304077 (0.03642030677389567) | > loss_mel_ce: 2.8460655212402344 (nan) | > loss: 0.01143084466457367 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13673769963252042) | > loader_time: 0.0054 (0.004703341048625898)  --> TIME: 2025-02-25 01:28:10 -- STEP: 14680/36938 -- GLOBAL_STEP: 568750 | > loss_text_ce: 0.03751736879348755 (0.036418573701137484) | > loss_mel_ce: 2.91740083694458 (nan) | > loss: 0.011725866235792637 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.13673110217749224) | > loader_time: 0.0049 (0.004703357499042051)  --> TIME: 2025-02-25 01:28:19 -- STEP: 14730/36938 -- GLOBAL_STEP: 568800 | > loss_text_ce: 0.03885940834879875 (0.036418415112224836) | > loss_mel_ce: 3.396437168121338 (nan) | > loss: 0.01363212987780571 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13670897113168118) | > loader_time: 0.0046 (0.004703265780586804)  --> TIME: 2025-02-25 01:28:28 -- STEP: 14780/36938 -- GLOBAL_STEP: 568850 | > loss_text_ce: 0.0312829352915287 (0.03641506573739826) | > loss_mel_ce: 2.3300087451934814 (nan) | > loss: 0.00937020592391491 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.13671430023500464) | > loader_time: 0.0054 (0.00470371431846256)  --> TIME: 2025-02-25 01:28:38 -- STEP: 14830/36938 -- GLOBAL_STEP: 568900 | > loss_text_ce: 0.02988104708492756 (0.03640996040217445) | > loss_mel_ce: 0.13153076171875 (nan) | > loss: 0.0006405230960808694 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.1367135434237507) | > loader_time: 0.0048 (0.004703624877106543)  --> TIME: 2025-02-25 01:28:47 -- STEP: 14880/36938 -- GLOBAL_STEP: 568950 | > loss_text_ce: 0.036728400737047195 (0.036407272336674) | > loss_mel_ce: 2.5392749309539795 (nan) | > loss: 0.010222235694527626 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.1367384201736856) | > loader_time: 0.0046 (0.004703534690923575)  --> TIME: 2025-02-25 01:28:57 -- STEP: 14930/36938 -- GLOBAL_STEP: 569000 | > loss_text_ce: 0.03251264989376068 (0.03640474588719388) | > loss_mel_ce: 2.6162045001983643 (nan) | > loss: 0.010510782711207867 (nan) | > current_lr: 5e-06 | > step_time: 0.1911 (0.13674842227008263) | > loader_time: 0.0049 (0.004703410567649629)  --> TIME: 2025-02-25 01:29:07 -- STEP: 14980/36938 -- GLOBAL_STEP: 569050 | > loss_text_ce: 0.04183398559689522 (0.03640967313366846) | > loss_mel_ce: 2.967726469039917 (nan) | > loss: 0.011942700482904911 (nan) | > current_lr: 5e-06 | > step_time: 0.1531 (0.13676838534218883) | > loader_time: 0.0054 (0.004703527251295782)  --> TIME: 2025-02-25 01:29:16 -- STEP: 15030/36938 -- GLOBAL_STEP: 569100 | > loss_text_ce: 0.0344974659383297 (0.03641144191773126) | > loss_mel_ce: 2.6031908988952637 (nan) | > loss: 0.010467018000781536 (nan) | > current_lr: 5e-06 | > step_time: 0.1764 (0.13678627786680725) | > loader_time: 0.0049 (0.004704098295388173)  --> TIME: 2025-02-25 01:29:26 -- STEP: 15080/36938 -- GLOBAL_STEP: 569150 | > loss_text_ce: 0.04346870630979538 (0.03640919715655214) | > loss_mel_ce: 2.8644213676452637 (nan) | > loss: 0.011539246886968613 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13679576168325872) | > loader_time: 0.0045 (0.004704034043876169)  --> TIME: 2025-02-25 01:29:35 -- STEP: 15130/36938 -- GLOBAL_STEP: 569200 | > loss_text_ce: 0.042754754424095154 (0.03641067030596473) | > loss_mel_ce: 3.4576144218444824 (nan) | > loss: 0.013890353962779045 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13679171763582054) | > loader_time: 0.0046 (0.0047039131888158125)  --> TIME: 2025-02-25 01:29:45 -- STEP: 15180/36938 -- GLOBAL_STEP: 569250 | > loss_text_ce: 0.021068494766950607 (0.03641225323252795) | > loss_mel_ce: 3.4837238788604736 (nan) | > loss: 0.013907907530665398 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.13680764201130277) | > loader_time: 0.0045 (0.004704030449054287)  --> TIME: 2025-02-25 01:29:54 -- STEP: 15230/36938 -- GLOBAL_STEP: 569300 | > loss_text_ce: 0.03953353315591812 (0.03641318985648098) | > loss_mel_ce: 3.160597085952759 (nan) | > loss: 0.01269893255084753 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1368090277507124) | > loader_time: 0.0046 (0.004704131691845626)  --> TIME: 2025-02-25 01:30:03 -- STEP: 15280/36938 -- GLOBAL_STEP: 569350 | > loss_text_ce: 0.020551489666104317 (0.03640774225777115) | > loss_mel_ce: 2.9876835346221924 (nan) | > loss: 0.011937441304326057 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13679813734835938) | > loader_time: 0.0048 (0.004704038338511389)  --> TIME: 2025-02-25 01:30:12 -- STEP: 15330/36938 -- GLOBAL_STEP: 569400 | > loss_text_ce: 0.02744518220424652 (0.0364071980535507) | > loss_mel_ce: 2.5633370876312256 (nan) | > loss: 0.010280882008373737 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13679680054435017) | > loader_time: 0.0045 (0.0047038210037091965)  --> TIME: 2025-02-25 01:30:22 -- STEP: 15380/36938 -- GLOBAL_STEP: 569450 | > loss_text_ce: 0.031059660017490387 (0.03640587962267644) | > loss_mel_ce: 3.252683401107788 (nan) | > loss: 0.01303072739392519 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13679077273382437) | > loader_time: 0.0045 (0.004703637713418658)  --> TIME: 2025-02-25 01:30:31 -- STEP: 15430/36938 -- GLOBAL_STEP: 569500 | > loss_text_ce: 0.02909560687839985 (0.036407535542490654) | > loss_mel_ce: 3.1025495529174805 (nan) | > loss: 0.012427164241671562 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.1367862574249135) | > loader_time: 0.0045 (0.00470337798204501)  --> TIME: 2025-02-25 01:30:40 -- STEP: 15480/36938 -- GLOBAL_STEP: 569550 | > loss_text_ce: 0.044935211539268494 (0.03639916474184914) | > loss_mel_ce: 0.01495361328125 (nan) | > loss: 0.00023765407968312502 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13678180352045852) | > loader_time: 0.0046 (0.004703135222427592)  --> TIME: 2025-02-25 01:30:49 -- STEP: 15530/36938 -- GLOBAL_STEP: 569600 | > loss_text_ce: 0.03339854255318642 (0.036395191058698734) | > loss_mel_ce: 3.4777629375457764 (nan) | > loss: 0.013933181762695312 (nan) | > current_lr: 5e-06 | > step_time: 0.1634 (0.13676340346634155) | > loader_time: 0.0046 (0.004702825586180174)  --> TIME: 2025-02-25 01:30:59 -- STEP: 15580/36938 -- GLOBAL_STEP: 569650 | > loss_text_ce: 0.03589383885264397 (0.03640005864544892) | > loss_mel_ce: 3.079733371734619 (nan) | > loss: 0.012363601475954056 (nan) | > current_lr: 5e-06 | > step_time: 0.1865 (0.13676971897999302) | > loader_time: 0.0049 (0.004702876345643018)  --> TIME: 2025-02-25 01:31:08 -- STEP: 15630/36938 -- GLOBAL_STEP: 569700 | > loss_text_ce: 0.03156588226556778 (0.036401163656872244) | > loss_mel_ce: 2.944399118423462 (nan) | > loss: 0.011809385381639004 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13678476394023598) | > loader_time: 0.0045 (0.004703207238698252)  --> TIME: 2025-02-25 01:31:17 -- STEP: 15680/36938 -- GLOBAL_STEP: 569750 | > loss_text_ce: 0.03635632246732712 (0.03639915059653236) | > loss_mel_ce: 3.1160662174224854 (nan) | > loss: 0.012509613297879696 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13675958734689886) | > loader_time: 0.0046 (0.004703115504615148)  --> TIME: 2025-02-25 01:31:26 -- STEP: 15730/36938 -- GLOBAL_STEP: 569800 | > loss_text_ce: 0.03466305509209633 (0.03640209570200248) | > loss_mel_ce: 2.690049171447754 (nan) | > loss: 0.010812350548803806 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13675218057420208) | > loader_time: 0.0047 (0.0047031480191394946)  --> TIME: 2025-02-25 01:31:36 -- STEP: 15780/36938 -- GLOBAL_STEP: 569850 | > loss_text_ce: 0.035549867898225784 (0.036406750922505546) | > loss_mel_ce: 2.84671950340271 (nan) | > loss: 0.011437577195465565 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.136753480425352) | > loader_time: 0.0049 (0.004703311336962127)  --> TIME: 2025-02-25 01:31:45 -- STEP: 15830/36938 -- GLOBAL_STEP: 569900 | > loss_text_ce: 0.032165706157684326 (0.036408421957649956) | > loss_mel_ce: 2.3860018253326416 (nan) | > loss: 0.009595903567969799 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13675776537053544) | > loader_time: 0.0047 (0.004703186330957936)  --> TIME: 2025-02-25 01:31:54 -- STEP: 15880/36938 -- GLOBAL_STEP: 569950 | > loss_text_ce: 0.032514214515686035 (0.03640746526650007) | > loss_mel_ce: 2.797741174697876 (nan) | > loss: 0.011231173761188984 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.1367485583879481) | > loader_time: 0.0058 (0.004703124194061113)  --> TIME: 2025-02-25 01:32:03 -- STEP: 15930/36938 -- GLOBAL_STEP: 570000 | > loss_text_ce: 0.04343976452946663 (0.036408476916078014) | > loss_mel_ce: 0.011505126953125 (nan) | > loss: 0.00021803528943564743 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13674439807695443) | > loader_time: 0.0046 (0.00470304126943047) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_570000.pth  --> TIME: 2025-02-25 01:32:19 -- STEP: 15980/36938 -- GLOBAL_STEP: 570050 | > loss_text_ce: 0.03826247900724411 (0.03641088283591627) | > loss_mel_ce: 1.9970499277114868 (nan) | > loss: 0.008076637051999569 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.1367488321434418) | > loader_time: 0.0047 (0.004703443996300524)  --> TIME: 2025-02-25 01:32:28 -- STEP: 16030/36938 -- GLOBAL_STEP: 570100 | > loss_text_ce: 0.04038996994495392 (0.036405201241343735) | > loss_mel_ce: 2.4674363136291504 (nan) | > loss: 0.009951692074537277 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13674160645890032) | > loader_time: 0.0051 (0.004703998759026375)  --> TIME: 2025-02-25 01:32:37 -- STEP: 16080/36938 -- GLOBAL_STEP: 570150 | > loss_text_ce: 0.03330552205443382 (0.036407270187231645) | > loss_mel_ce: 3.178516149520874 (nan) | > loss: 0.012745324522256851 (nan) | > current_lr: 5e-06 | > step_time: 0.1837 (0.13674050092104023) | > loader_time: 0.0048 (0.004703881610092227)  --> TIME: 2025-02-25 01:32:46 -- STEP: 16130/36938 -- GLOBAL_STEP: 570200 | > loss_text_ce: 0.051604244858026505 (0.03640689194134285) | > loss_mel_ce: 1.8134223222732544 (nan) | > loss: 0.007400899659842253 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13674179598624372) | > loader_time: 0.0045 (0.004704229578939332)  --> TIME: 2025-02-25 01:32:55 -- STEP: 16180/36938 -- GLOBAL_STEP: 570250 | > loss_text_ce: 0.0364152230322361 (0.036407487125196904) | > loss_mel_ce: 3.4499411582946777 (nan) | > loss: 0.013834748417139053 (nan) | > current_lr: 5e-06 | > step_time: 0.152 (0.13672627363865061) | > loader_time: 0.0049 (0.004704015850284031)  --> TIME: 2025-02-25 01:33:05 -- STEP: 16230/36938 -- GLOBAL_STEP: 570300 | > loss_text_ce: 0.017837990075349808 (0.03640528103343571) | > loss_mel_ce: 2.5625088214874268 (nan) | > loss: 0.010239471681416035 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.136737004563313) | > loader_time: 0.0047 (0.0047041224170010125)  --> TIME: 2025-02-25 01:33:15 -- STEP: 16280/36938 -- GLOBAL_STEP: 570350 | > loss_text_ce: 0.035712551325559616 (0.03640509187341123) | > loss_mel_ce: 3.2834105491638184 (nan) | > loss: 0.013171124272048473 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.136751726159885) | > loader_time: 0.0046 (0.004704360923837373)  --> TIME: 2025-02-25 01:33:24 -- STEP: 16330/36938 -- GLOBAL_STEP: 570400 | > loss_text_ce: 0.03273550793528557 (0.036405039931757484) | > loss_mel_ce: 3.47822904586792 (nan) | > loss: 0.01393240038305521 (nan) | > current_lr: 5e-06 | > step_time: 0.1858 (0.13675272701272734) | > loader_time: 0.0049 (0.0047049312743392985)  --> TIME: 2025-02-25 01:33:33 -- STEP: 16380/36938 -- GLOBAL_STEP: 570450 | > loss_text_ce: 0.029469024389982224 (0.03640138711808731) | > loss_mel_ce: 2.548102617263794 (nan) | > loss: 0.010228459723293781 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13674442248466648) | > loader_time: 0.0046 (0.004704864048404705)  --> TIME: 2025-02-25 01:33:42 -- STEP: 16430/36938 -- GLOBAL_STEP: 570500 | > loss_text_ce: 0.029440047219395638 (0.03640133680444719) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.13672556579657136) | > loader_time: 0.0046 (0.00470468482808665)  --> TIME: 2025-02-25 01:33:51 -- STEP: 16480/36938 -- GLOBAL_STEP: 570550 | > loss_text_ce: 0.03940674290060997 (0.036403345836391966) | > loss_mel_ce: 3.09297776222229 (nan) | > loss: 0.012430097907781601 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13672544844810203) | > loader_time: 0.0046 (0.004704984921274821)  --> TIME: 2025-02-25 01:34:01 -- STEP: 16530/36938 -- GLOBAL_STEP: 570600 | > loss_text_ce: 0.03272714465856552 (0.03640194742292997) | > loss_mel_ce: 3.1509594917297363 (nan) | > loss: 0.012633678503334522 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13673303686916177) | > loader_time: 0.0046 (0.004705072228142645)  --> TIME: 2025-02-25 01:34:10 -- STEP: 16580/36938 -- GLOBAL_STEP: 570650 | > loss_text_ce: 0.03932558000087738 (0.036402580291886526) | > loss_mel_ce: 2.575960397720337 (nan) | > loss: 0.01037811953574419 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13673474608917105) | > loader_time: 0.0047 (0.004705263075696246)  --> TIME: 2025-02-25 01:34:19 -- STEP: 16630/36938 -- GLOBAL_STEP: 570700 | > loss_text_ce: 0.03479970991611481 (0.0364048796401807) | > loss_mel_ce: 2.775362253189087 (nan) | > loss: 0.011151437647640705 (nan) | > current_lr: 5e-06 | > step_time: 0.1854 (0.13673120369340555) | > loader_time: 0.0051 (0.004705724813675768)  --> TIME: 2025-02-25 01:34:29 -- STEP: 16680/36938 -- GLOBAL_STEP: 570750 | > loss_text_ce: 0.036947835236787796 (0.03640332822844254) | > loss_mel_ce: 0.02105712890625 (nan) | > loss: 0.00023017844068817794 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13674331757662064) | > loader_time: 0.0045 (0.004706083155936178)  --> TIME: 2025-02-25 01:34:38 -- STEP: 16730/36938 -- GLOBAL_STEP: 570800 | > loss_text_ce: 0.03329794853925705 (0.0363988700061724) | > loss_mel_ce: 3.252018690109253 (nan) | > loss: 0.013036971911787987 (nan) | > current_lr: 5e-06 | > step_time: 0.1858 (0.13674299507460294) | > loader_time: 0.0049 (0.004706203745142532)  --> TIME: 2025-02-25 01:34:48 -- STEP: 16780/36938 -- GLOBAL_STEP: 570850 | > loss_text_ce: 0.039788782596588135 (0.036391913491668586) | > loss_mel_ce: 2.4644386768341064 (nan) | > loss: 0.009937411174178123 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13674414328369355) | > loader_time: 0.0066 (0.0047068850502496005)  --> TIME: 2025-02-25 01:34:57 -- STEP: 16830/36938 -- GLOBAL_STEP: 570900 | > loss_text_ce: 0.03805413097143173 (0.03639370049325053) | > loss_mel_ce: 3.265509843826294 (nan) | > loss: 0.01310938224196434 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.13676238663481272) | > loader_time: 0.0047 (0.004707027586598648)  --> TIME: 2025-02-25 01:35:07 -- STEP: 16880/36938 -- GLOBAL_STEP: 570950 | > loss_text_ce: 0.04040507599711418 (0.036391352638778726) | > loss_mel_ce: 2.876666307449341 (nan) | > loss: 0.01157568022608757 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13675088393744664) | > loader_time: 0.0046 (0.004706849306115595)  --> TIME: 2025-02-25 01:35:16 -- STEP: 16930/36938 -- GLOBAL_STEP: 571000 | > loss_text_ce: 0.02714131958782673 (0.03638748434849176) | > loss_mel_ce: 3.455684185028076 (nan) | > loss: 0.013820736669003963 (nan) | > current_lr: 5e-06 | > step_time: 0.1782 (0.1367401159380004) | > loader_time: 0.0048 (0.004706776332235578)  --> TIME: 2025-02-25 01:35:24 -- STEP: 16980/36938 -- GLOBAL_STEP: 571050 | > loss_text_ce: 0.03762676566839218 (0.0363888643781826) | > loss_mel_ce: 2.573906660079956 (nan) | > loss: 0.010363228619098663 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13670955976412033) | > loader_time: 0.0045 (0.00470686020643325)  --> TIME: 2025-02-25 01:35:34 -- STEP: 17030/36938 -- GLOBAL_STEP: 571100 | > loss_text_ce: 0.039737436920404434 (0.03639071407195784) | > loss_mel_ce: 3.2110188007354736 (nan) | > loss: 0.01289982721209526 (nan) | > current_lr: 5e-06 | > step_time: 0.185 (0.13670346850307252) | > loader_time: 0.0048 (0.004706850572676766)  --> TIME: 2025-02-25 01:35:43 -- STEP: 17080/36938 -- GLOBAL_STEP: 571150 | > loss_text_ce: 0.040248237550258636 (0.036388146458629804) | > loss_mel_ce: 0.5883331298828125 (nan) | > loss: 0.0024943705648183823 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13670841655072308) | > loader_time: 0.0048 (0.004706978197678456)  --> TIME: 2025-02-25 01:35:52 -- STEP: 17130/36938 -- GLOBAL_STEP: 571200 | > loss_text_ce: 0.03476819396018982 (0.03638580738345532) | > loss_mel_ce: 3.04679536819458 (nan) | > loss: 0.012228427454829216 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13670041474833908) | > loader_time: 0.0046 (0.004707096559710467)  --> TIME: 2025-02-25 01:36:01 -- STEP: 17180/36938 -- GLOBAL_STEP: 571250 | > loss_text_ce: 0.02813643403351307 (0.03639003336028439) | > loss_mel_ce: 2.2395052909851074 (nan) | > loss: 0.00899857934564352 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1367007349367327) | > loader_time: 0.0047 (0.004707624956670532)  --> TIME: 2025-02-25 01:36:11 -- STEP: 17230/36938 -- GLOBAL_STEP: 571300 | > loss_text_ce: 0.03402455151081085 (0.03638986542418776) | > loss_mel_ce: 3.387444496154785 (nan) | > loss: 0.013577258214354515 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.1367090924008942) | > loader_time: 0.005 (0.004708126016086558)  --> TIME: 2025-02-25 01:36:20 -- STEP: 17280/36938 -- GLOBAL_STEP: 571350 | > loss_text_ce: 0.04975621774792671 (0.036395585317600086) | > loss_mel_ce: 0.17608946561813354 (nan) | > loss: 0.000896213052328676 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13670683755642807) | > loader_time: 0.0046 (0.004708304877082495)  --> TIME: 2025-02-25 01:36:30 -- STEP: 17330/36938 -- GLOBAL_STEP: 571400 | > loss_text_ce: 0.036284372210502625 (0.03639441538221626) | > loss_mel_ce: 2.8102197647094727 (nan) | > loss: 0.011295652016997337 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13671231546363422) | > loader_time: 0.0047 (0.004708183107524669)  --> TIME: 2025-02-25 01:36:39 -- STEP: 17380/36938 -- GLOBAL_STEP: 571450 | > loss_text_ce: 0.032865095883607864 (0.036393373252200156) | > loss_mel_ce: 2.8867528438568115 (nan) | > loss: 0.011585786007344723 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.13672338650059196) | > loader_time: 0.0046 (0.004708058252433351)  --> TIME: 2025-02-25 01:36:49 -- STEP: 17430/36938 -- GLOBAL_STEP: 571500 | > loss_text_ce: 0.036028049886226654 (0.036390841272149275) | > loss_mel_ce: 3.1746878623962402 (nan) | > loss: 0.012740937061607838 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13672364260092962) | > loader_time: 0.0054 (0.004708410006997493)  --> TIME: 2025-02-25 01:36:58 -- STEP: 17480/36938 -- GLOBAL_STEP: 571550 | > loss_text_ce: 0.03777517378330231 (0.036391968589310104) | > loss_mel_ce: 2.0716705322265625 (nan) | > loss: 0.008370817638933659 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.1367212442702501) | > loader_time: 0.0046 (0.004708345135507634)  --> TIME: 2025-02-25 01:37:07 -- STEP: 17530/36938 -- GLOBAL_STEP: 571600 | > loss_text_ce: 0.033651113510131836 (0.0363950791903141) | > loss_mel_ce: 2.3423664569854736 (nan) | > loss: 0.009428641758859158 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13671387865824858) | > loader_time: 0.0049 (0.0047083245912280785)  --> TIME: 2025-02-25 01:37:16 -- STEP: 17580/36938 -- GLOBAL_STEP: 571650 | > loss_text_ce: 0.03896300122141838 (0.03639498879254678) | > loss_mel_ce: 2.89888858795166 (nan) | > loss: 0.011658142320811749 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.13671868612334961) | > loader_time: 0.0048 (0.004708511579836848)  --> TIME: 2025-02-25 01:37:26 -- STEP: 17630/36938 -- GLOBAL_STEP: 571700 | > loss_text_ce: 0.03809799253940582 (0.036394405567683565) | > loss_mel_ce: 2.801054000854492 (nan) | > loss: 0.011266477406024933 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13671079834144462) | > loader_time: 0.0046 (0.00470844318565155)  --> TIME: 2025-02-25 01:37:35 -- STEP: 17680/36938 -- GLOBAL_STEP: 571750 | > loss_text_ce: 0.04195195436477661 (0.03639204667705183) | > loss_mel_ce: 3.4436423778533936 (nan) | > loss: 0.013831724412739277 (nan) | > current_lr: 5e-06 | > step_time: 0.1824 (0.13671697771926783) | > loader_time: 0.005 (0.0047083843482565385)  --> TIME: 2025-02-25 01:37:45 -- STEP: 17730/36938 -- GLOBAL_STEP: 571800 | > loss_text_ce: 0.036444395780563354 (0.036390879353192074) | > loss_mel_ce: 3.34922194480896 (nan) | > loss: 0.013435184955596924 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13672933403677784) | > loader_time: 0.0048 (0.004708402491664711)  --> TIME: 2025-02-25 01:37:54 -- STEP: 17780/36938 -- GLOBAL_STEP: 571850 | > loss_text_ce: 0.04226052761077881 (0.03638929643781371) | > loss_mel_ce: 2.233393907546997 (nan) | > loss: 0.009030374698340893 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13672715827772366) | > loader_time: 0.0046 (0.004708468216744783)  --> TIME: 2025-02-25 01:38:03 -- STEP: 17830/36938 -- GLOBAL_STEP: 571900 | > loss_text_ce: 0.04233821481466293 (0.03639126095646047) | > loss_mel_ce: 3.043030261993408 (nan) | > loss: 0.012243526056408882 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.136735728919606) | > loader_time: 0.0046 (0.004708470832880319)  --> TIME: 2025-02-25 01:38:13 -- STEP: 17880/36938 -- GLOBAL_STEP: 571950 | > loss_text_ce: 0.05273027718067169 (0.03639085667413053) | > loss_mel_ce: 3.4803874492645264 (nan) | > loss: 0.014020309783518314 (nan) | > current_lr: 5e-06 | > step_time: 0.1629 (0.13674167804003257) | > loader_time: 0.0048 (0.004708403028897776)  --> TIME: 2025-02-25 01:38:22 -- STEP: 17930/36938 -- GLOBAL_STEP: 572000 | > loss_text_ce: 0.0297254491597414 (0.03639182014020989) | > loss_mel_ce: 2.5740668773651123 (nan) | > loss: 0.010332509875297546 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13673821842610567) | > loader_time: 0.0046 (0.004708398897689068)  --> TIME: 2025-02-25 01:38:31 -- STEP: 17980/36938 -- GLOBAL_STEP: 572050 | > loss_text_ce: 0.03478526696562767 (0.03639418741100737) | > loss_mel_ce: 3.192592144012451 (nan) | > loss: 0.01280705351382494 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13673142538187327) | > loader_time: 0.0047 (0.00470812286232681)  --> TIME: 2025-02-25 01:38:41 -- STEP: 18030/36938 -- GLOBAL_STEP: 572100 | > loss_text_ce: 0.03547593206167221 (0.03639730260982746) | > loss_mel_ce: 2.1720364093780518 (nan) | > loss: 0.008759969845414162 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13673332151411857) | > loader_time: 0.0046 (0.004708009260730871)  --> TIME: 2025-02-25 01:38:50 -- STEP: 18080/36938 -- GLOBAL_STEP: 572150 | > loss_text_ce: 0.03233935683965683 (0.03639643079409598) | > loss_mel_ce: 3.2112009525299072 (nan) | > loss: 0.012871192768216133 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.1367222029419068) | > loader_time: 0.0046 (0.004708012041792394)  --> TIME: 2025-02-25 01:38:59 -- STEP: 18130/36938 -- GLOBAL_STEP: 572200 | > loss_text_ce: 0.0377136692404747 (0.03639853326434242) | > loss_mel_ce: 1.541914463043213 (nan) | > loss: 0.006268365774303675 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13673741976595385) | > loader_time: 0.0046 (0.004707994884505683)  --> TIME: 2025-02-25 01:39:09 -- STEP: 18180/36938 -- GLOBAL_STEP: 572250 | > loss_text_ce: 0.029850231483578682 (0.036399449450163573) | > loss_mel_ce: 2.103386163711548 (nan) | > loss: 0.00846522394567728 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13672973013291362) | > loader_time: 0.0045 (0.00470809105074706)  --> TIME: 2025-02-25 01:39:18 -- STEP: 18230/36938 -- GLOBAL_STEP: 572300 | > loss_text_ce: 0.03593046963214874 (0.03640088663322541) | > loss_mel_ce: 2.6685566902160645 (nan) | > loss: 0.010732091963291168 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.1367209074331124) | > loader_time: 0.0047 (0.004708181497361446)  --> TIME: 2025-02-25 01:39:27 -- STEP: 18280/36938 -- GLOBAL_STEP: 572350 | > loss_text_ce: 0.04118554666638374 (0.03640029962527143) | > loss_mel_ce: 2.405611276626587 (nan) | > loss: 0.009709511883556843 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.136722174462917) | > loader_time: 0.0045 (0.004708032221971493)  --> TIME: 2025-02-25 01:39:36 -- STEP: 18330/36938 -- GLOBAL_STEP: 572400 | > loss_text_ce: 0.043110527098178864 (0.03640094914611275) | > loss_mel_ce: 3.175111770629883 (nan) | > loss: 0.01277072448283434 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13672201678977886) | > loader_time: 0.0047 (0.004707963519973034)  --> TIME: 2025-02-25 01:39:46 -- STEP: 18380/36938 -- GLOBAL_STEP: 572450 | > loss_text_ce: 0.036896225064992905 (0.036398456633897534) | > loss_mel_ce: 3.1200156211853027 (nan) | > loss: 0.012527428567409515 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.1367269953913473) | > loader_time: 0.0047 (0.004708120742482367)  --> TIME: 2025-02-25 01:39:55 -- STEP: 18430/36938 -- GLOBAL_STEP: 572500 | > loss_text_ce: 0.03309023007750511 (0.036399477680642695) | > loss_mel_ce: 3.1681747436523438 (nan) | > loss: 0.012703432701528072 (nan) | > current_lr: 5e-06 | > step_time: 0.1632 (0.1367206972963576) | > loader_time: 0.0048 (0.004708096066442301)  --> TIME: 2025-02-25 01:40:04 -- STEP: 18480/36938 -- GLOBAL_STEP: 572550 | > loss_text_ce: 0.0483408123254776 (0.036400421757404176) | > loss_mel_ce: 3.4362125396728516 (nan) | > loss: 0.013827593065798283 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13672180646683715) | > loader_time: 0.0046 (0.004708071846466547)  --> TIME: 2025-02-25 01:40:14 -- STEP: 18530/36938 -- GLOBAL_STEP: 572600 | > loss_text_ce: 0.04125617817044258 (0.03640105606042169) | > loss_mel_ce: 2.784755229949951 (nan) | > loss: 0.011214331723749638 (nan) | > current_lr: 5e-06 | > step_time: 0.1072 (0.1367270477771498) | > loader_time: 0.0045 (0.004708115602916328)  --> TIME: 2025-02-25 01:40:23 -- STEP: 18580/36938 -- GLOBAL_STEP: 572650 | > loss_text_ce: 0.03487999737262726 (0.036401163492516146) | > loss_mel_ce: 2.9403696060180664 (nan) | > loss: 0.011806546710431576 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.1367222195164376) | > loader_time: 0.0045 (0.0047083337802291455)  --> TIME: 2025-02-25 01:40:32 -- STEP: 18630/36938 -- GLOBAL_STEP: 572700 | > loss_text_ce: 0.033584557473659515 (0.03640153239372667) | > loss_mel_ce: 0.079345703125 (nan) | > loss: 0.0004481359792407602 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13670914320298158) | > loader_time: 0.0045 (0.004708108656264339)  --> TIME: 2025-02-25 01:40:41 -- STEP: 18680/36938 -- GLOBAL_STEP: 572750 | > loss_text_ce: 0.037001244723796844 (0.03639890233780664) | > loss_mel_ce: 2.9825332164764404 (nan) | > loss: 0.01198227982968092 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13669849636723178) | > loader_time: 0.0045 (0.0047078409848427495)  --> TIME: 2025-02-25 01:40:51 -- STEP: 18730/36938 -- GLOBAL_STEP: 572800 | > loss_text_ce: 0.03215052932500839 (0.036399907286010835) | > loss_mel_ce: 2.3807199001312256 (nan) | > loss: 0.009574882686138153 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13671259467638677) | > loader_time: 0.0046 (0.004707704924416926)  --> TIME: 2025-02-25 01:41:01 -- STEP: 18780/36938 -- GLOBAL_STEP: 572850 | > loss_text_ce: 0.03598179668188095 (0.03639881047514418) | > loss_mel_ce: 3.290971040725708 (nan) | > loss: 0.013202195055782795 (nan) | > current_lr: 5e-06 | > step_time: 0.1846 (0.13673380467330543) | > loader_time: 0.0055 (0.004707920119658308)  --> TIME: 2025-02-25 01:41:10 -- STEP: 18830/36938 -- GLOBAL_STEP: 572900 | > loss_text_ce: 0.025256624445319176 (0.03640060960302792) | > loss_mel_ce: 2.0267226696014404 (nan) | > loss: 0.008142775855958462 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13673693537395545) | > loader_time: 0.0052 (0.0047083095103834524)  --> TIME: 2025-02-25 01:41:20 -- STEP: 18880/36938 -- GLOBAL_STEP: 572950 | > loss_text_ce: 0.04051428288221359 (0.03640100965532858) | > loss_mel_ce: 2.7757396697998047 (nan) | > loss: 0.011175611056387424 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.136749475626117) | > loader_time: 0.0048 (0.004708766028032454)  --> TIME: 2025-02-25 01:41:29 -- STEP: 18930/36938 -- GLOBAL_STEP: 573000 | > loss_text_ce: 0.03893952816724777 (0.0364054156509628) | > loss_mel_ce: 3.5803213119506836 (nan) | > loss: 0.014362147077918053 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1367478288800641) | > loader_time: 0.0045 (0.004708877519143038)  --> TIME: 2025-02-25 01:41:38 -- STEP: 18980/36938 -- GLOBAL_STEP: 573050 | > loss_text_ce: 0.035425953567028046 (0.036409335348507696) | > loss_mel_ce: 1.0890251398086548 (nan) | > loss: 0.004462107550352812 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13674804062687548) | > loader_time: 0.0045 (0.004709055690041825)  --> TIME: 2025-02-25 01:41:48 -- STEP: 19030/36938 -- GLOBAL_STEP: 573100 | > loss_text_ce: 0.031265050172805786 (0.036405748027389946) | > loss_mel_ce: 2.874738931655884 (nan) | > loss: 0.011531762778759003 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.13674484489468952) | > loader_time: 0.0053 (0.004709215384687293)  --> TIME: 2025-02-25 01:41:57 -- STEP: 19080/36938 -- GLOBAL_STEP: 573150 | > loss_text_ce: 0.03127988427877426 (0.03640646686254206) | > loss_mel_ce: 2.7538249492645264 (nan) | > loss: 0.011052004061639309 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.13673658246014322) | > loader_time: 0.0046 (0.004709055001380801)  --> TIME: 2025-02-25 01:42:06 -- STEP: 19130/36938 -- GLOBAL_STEP: 573200 | > loss_text_ce: 0.039270494133234024 (0.03640462478561444) | > loss_mel_ce: 2.6522669792175293 (nan) | > loss: 0.01068070437759161 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13674505335214035) | > loader_time: 0.0046 (0.004709111827864293)  --> TIME: 2025-02-25 01:42:16 -- STEP: 19180/36938 -- GLOBAL_STEP: 573250 | > loss_text_ce: 0.035642609000205994 (0.036407116600783204) | > loss_mel_ce: 3.1819400787353516 (nan) | > loss: 0.012768185697495937 (nan) | > current_lr: 5e-06 | > step_time: 0.165 (0.13674325035560586) | > loader_time: 0.0046 (0.004709089287130373)  --> TIME: 2025-02-25 01:42:25 -- STEP: 19230/36938 -- GLOBAL_STEP: 573300 | > loss_text_ce: 0.03986587002873421 (0.036408053119427033) | > loss_mel_ce: 2.766451358795166 (nan) | > loss: 0.011136180721223354 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13673985382572013) | > loader_time: 0.0045 (0.004709116109509795)  --> TIME: 2025-02-25 01:42:34 -- STEP: 19280/36938 -- GLOBAL_STEP: 573350 | > loss_text_ce: 0.041392888873815536 (0.03640992636937267) | > loss_mel_ce: 2.5059151649475098 (nan) | > loss: 0.010108365677297115 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13674467095942894) | > loader_time: 0.0045 (0.004709260654152669)  --> TIME: 2025-02-25 01:42:44 -- STEP: 19330/36938 -- GLOBAL_STEP: 573400 | > loss_text_ce: 0.03318697586655617 (0.03641171144341786) | > loss_mel_ce: 2.6393425464630127 (nan) | > loss: 0.010605276562273502 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13674728828455815) | > loader_time: 0.0047 (0.004709400504102864)  --> TIME: 2025-02-25 01:42:53 -- STEP: 19380/36938 -- GLOBAL_STEP: 573450 | > loss_text_ce: 0.029595762491226196 (0.03641144417455047) | > loss_mel_ce: 2.2881433963775635 (nan) | > loss: 0.009197378531098366 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.1367526918864961) | > loader_time: 0.0047 (0.004709438741268631)  --> TIME: 2025-02-25 01:43:02 -- STEP: 19430/36938 -- GLOBAL_STEP: 573500 | > loss_text_ce: 0.047827839851379395 (0.036411696124766184) | > loss_mel_ce: 3.2473528385162354 (nan) | > loss: 0.013076115399599075 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13675438034982315) | > loader_time: 0.0046 (0.004709499482328105)  --> TIME: 2025-02-25 01:43:12 -- STEP: 19480/36938 -- GLOBAL_STEP: 573550 | > loss_text_ce: 0.04310836270451546 (0.03641066545786926) | > loss_mel_ce: 3.5411906242370605 (nan) | > loss: 0.014223409816622734 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13674663091097505) | > loader_time: 0.0046 (0.004709521945741876)  --> TIME: 2025-02-25 01:43:21 -- STEP: 19530/36938 -- GLOBAL_STEP: 573600 | > loss_text_ce: 0.04247850179672241 (0.03641077955136579) | > loss_mel_ce: 3.1689178943634033 (nan) | > loss: 0.012743637897074223 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13673825718107657) | > loader_time: 0.0045 (0.00470934696705355)  --> TIME: 2025-02-25 01:43:30 -- STEP: 19580/36938 -- GLOBAL_STEP: 573650 | > loss_text_ce: 0.04383911192417145 (0.03641270758190297) | > loss_mel_ce: 2.3657631874084473 (nan) | > loss: 0.009561914950609207 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.13675165511249132) | > loader_time: 0.0046 (0.004709248754659153)  --> TIME: 2025-02-25 01:43:40 -- STEP: 19630/36938 -- GLOBAL_STEP: 573700 | > loss_text_ce: 0.03792612999677658 (0.036413779123380154) | > loss_mel_ce: 2.0277559757232666 (nan) | > loss: 0.008197152055799961 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.1367608041102582) | > loader_time: 0.0053 (0.004709434533568593)  --> TIME: 2025-02-25 01:43:49 -- STEP: 19680/36938 -- GLOBAL_STEP: 573750 | > loss_text_ce: 0.036398883908987045 (0.03641013130640256) | > loss_mel_ce: 2.9091439247131348 (nan) | > loss: 0.011688662692904472 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.13675484597925225) | > loader_time: 0.0047 (0.004709659238171776)  --> TIME: 2025-02-25 01:43:58 -- STEP: 19730/36938 -- GLOBAL_STEP: 573800 | > loss_text_ce: 0.03668096289038658 (0.036412795951700355) | > loss_mel_ce: 2.783637762069702 (nan) | > loss: 0.011191741563379765 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13673990812026102) | > loader_time: 0.0046 (0.004709858829094205)  --> TIME: 2025-02-25 01:44:07 -- STEP: 19780/36938 -- GLOBAL_STEP: 573850 | > loss_text_ce: 0.038316357880830765 (0.03641143157848422) | > loss_mel_ce: 3.0199148654937744 (nan) | > loss: 0.012135838158428669 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13673343225725973) | > loader_time: 0.0048 (0.004709999746933989)  --> TIME: 2025-02-25 01:44:17 -- STEP: 19830/36938 -- GLOBAL_STEP: 573900 | > loss_text_ce: 0.03244955092668533 (0.03641223328726281) | > loss_mel_ce: 2.7143099308013916 (nan) | > loss: 0.010899839922785759 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13674334781192726) | > loader_time: 0.0046 (0.0047099570583587855)  --> TIME: 2025-02-25 01:44:26 -- STEP: 19880/36938 -- GLOBAL_STEP: 573950 | > loss_text_ce: 0.03462378308176994 (0.03641270759536075) | > loss_mel_ce: 2.897568464279175 (nan) | > loss: 0.01163568440824747 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.1367442414554311) | > loader_time: 0.0046 (0.00470988638923922)  --> TIME: 2025-02-25 01:44:35 -- STEP: 19930/36938 -- GLOBAL_STEP: 574000 | > loss_text_ce: 0.035142410546541214 (0.036410737475889646) | > loss_mel_ce: 0.6635799407958984 (nan) | > loss: 0.0027727079577744007 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13674500922850485) | > loader_time: 0.0046 (0.004709644946870175)  --> TIME: 2025-02-25 01:44:45 -- STEP: 19980/36938 -- GLOBAL_STEP: 574050 | > loss_text_ce: 0.035738080739974976 (0.036411076195135304) | > loss_mel_ce: 0.012908935546875 (nan) | > loss: 0.0001930437283590436 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.1367398988257895) | > loader_time: 0.0046 (0.004709373532353466)  --> TIME: 2025-02-25 01:44:54 -- STEP: 20030/36938 -- GLOBAL_STEP: 574100 | > loss_text_ce: 0.03926261514425278 (0.03640871486257847) | > loss_mel_ce: 3.074880838394165 (nan) | > loss: 0.012357712723314762 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13673706064209962) | > loader_time: 0.0046 (0.0047091039609028305)  --> TIME: 2025-02-25 01:45:04 -- STEP: 20080/36938 -- GLOBAL_STEP: 574150 | > loss_text_ce: 0.03396205976605415 (0.036406851069448805) | > loss_mel_ce: 2.6658482551574707 (nan) | > loss: 0.010713533498346806 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13675216412639235) | > loader_time: 0.0045 (0.004709124303908945)  --> TIME: 2025-02-25 01:45:13 -- STEP: 20130/36938 -- GLOBAL_STEP: 574200 | > loss_text_ce: 0.036355968564748764 (0.03640341144286444) | > loss_mel_ce: 3.1424484252929688 (nan) | > loss: 0.012614304199814796 (nan) | > current_lr: 5e-06 | > step_time: 0.1271 (0.13675360578952941) | > loader_time: 0.0046 (0.004709104987086207)  --> TIME: 2025-02-25 01:45:22 -- STEP: 20180/36938 -- GLOBAL_STEP: 574250 | > loss_text_ce: 0.028425192460417747 (0.03640107226053128) | > loss_mel_ce: 1.9692857265472412 (nan) | > loss: 0.00792742520570755 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13674809044251) | > loader_time: 0.0045 (0.004708897075048179)  --> TIME: 2025-02-25 01:45:32 -- STEP: 20230/36938 -- GLOBAL_STEP: 574300 | > loss_text_ce: 0.030228057876229286 (0.03640324252837585) | > loss_mel_ce: 3.1147329807281494 (nan) | > loss: 0.012480005621910095 (nan) | > current_lr: 5e-06 | > step_time: 0.1496 (0.13675516370612087) | > loader_time: 0.0049 (0.004708765051592368)  --> TIME: 2025-02-25 01:45:41 -- STEP: 20280/36938 -- GLOBAL_STEP: 574350 | > loss_text_ce: 0.04103001952171326 (0.03639971680233806) | > loss_mel_ce: 3.4127511978149414 (nan) | > loss: 0.013705481775105 (nan) | > current_lr: 5e-06 | > step_time: 0.1907 (0.13676080690803377) | > loader_time: 0.0048 (0.004708550350200506)  --> TIME: 2025-02-25 01:45:51 -- STEP: 20330/36938 -- GLOBAL_STEP: 574400 | > loss_text_ce: 0.03681665658950806 (0.03640119669041818) | > loss_mel_ce: 2.996066093444824 (nan) | > loss: 0.01203524973243475 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13677631931473813) | > loader_time: 0.0045 (0.00470878428211954)  --> TIME: 2025-02-25 01:46:00 -- STEP: 20380/36938 -- GLOBAL_STEP: 574450 | > loss_text_ce: 0.04913386330008507 (0.03639912376576938) | > loss_mel_ce: 2.3224663734436035 (nan) | > loss: 0.009411112405359745 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13676906374883618) | > loader_time: 0.0045 (0.00470903982007118)  --> TIME: 2025-02-25 01:46:10 -- STEP: 20430/36938 -- GLOBAL_STEP: 574500 | > loss_text_ce: 0.05001424252986908 (0.03639909400533086) | > loss_mel_ce: 2.1338624954223633 (nan) | > loss: 0.008666178211569786 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13679191211946917) | > loader_time: 0.0045 (0.004709596115762098)  --> TIME: 2025-02-25 01:46:19 -- STEP: 20480/36938 -- GLOBAL_STEP: 574550 | > loss_text_ce: 0.031367719173431396 (0.036397923537151565) | > loss_mel_ce: 2.7131433486938477 (nan) | > loss: 0.010890917852520943 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13678447984857503) | > loader_time: 0.0045 (0.0047095181420445525)  --> TIME: 2025-02-25 01:46:28 -- STEP: 20530/36938 -- GLOBAL_STEP: 574600 | > loss_text_ce: 0.045235615223646164 (0.03639896597987224) | > loss_mel_ce: 2.384758949279785 (nan) | > loss: 0.009642836637794971 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13678459362814172) | > loader_time: 0.0045 (0.004709507323914031)  --> TIME: 2025-02-25 01:46:38 -- STEP: 20580/36938 -- GLOBAL_STEP: 574650 | > loss_text_ce: 0.03786797076463699 (0.036397247451409064) | > loss_mel_ce: 2.913475513458252 (nan) | > loss: 0.011711681261658669 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.13679753433047762) | > loader_time: 0.0047 (0.004709728732632722)  --> TIME: 2025-02-25 01:46:47 -- STEP: 20630/36938 -- GLOBAL_STEP: 574700 | > loss_text_ce: 0.04580457881093025 (0.03639518721494139) | > loss_mel_ce: 2.950721263885498 (nan) | > loss: 0.011890975758433342 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13680090086008262) | > loader_time: 0.0046 (0.004709986755123327)  --> TIME: 2025-02-25 01:46:57 -- STEP: 20680/36938 -- GLOBAL_STEP: 574750 | > loss_text_ce: 0.03197147324681282 (0.03639172719894205) | > loss_mel_ce: 2.849592685699463 (nan) | > loss: 0.011434778571128845 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13680919129797764) | > loader_time: 0.0047 (0.004710126107619873)  --> TIME: 2025-02-25 01:47:06 -- STEP: 20730/36938 -- GLOBAL_STEP: 574800 | > loss_text_ce: 0.0385417677462101 (0.03638916179102881) | > loss_mel_ce: 2.9685065746307373 (nan) | > loss: 0.011932732537388802 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13681057911820882) | > loader_time: 0.0046 (0.004710029888659143)  --> TIME: 2025-02-25 01:47:16 -- STEP: 20780/36938 -- GLOBAL_STEP: 574850 | > loss_text_ce: 0.03910670801997185 (0.03638673682685457) | > loss_mel_ce: 2.758699655532837 (nan) | > loss: 0.011102406308054924 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13680321336136744) | > loader_time: 0.0046 (0.004709801499492527)  --> TIME: 2025-02-25 01:47:25 -- STEP: 20830/36938 -- GLOBAL_STEP: 574900 | > loss_text_ce: 0.038558728992938995 (0.03638602221537547) | > loss_mel_ce: 3.2878732681274414 (nan) | > loss: 0.013200127519667149 (nan) | > current_lr: 5e-06 | > step_time: 0.1558 (0.1367911445711074) | > loader_time: 0.0046 (0.004709827768304762)  --> TIME: 2025-02-25 01:47:34 -- STEP: 20880/36938 -- GLOBAL_STEP: 574950 | > loss_text_ce: 0.0332941971719265 (0.03639049832673445) | > loss_mel_ce: 0.080322265625 (nan) | > loss: 0.0004508590209297836 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1367863628371008) | > loader_time: 0.0045 (0.004709880196728487)  --> TIME: 2025-02-25 01:47:43 -- STEP: 20930/36938 -- GLOBAL_STEP: 575000 | > loss_text_ce: 0.031723640859127045 (0.036386378060953316) | > loss_mel_ce: 3.1162829399108887 (nan) | > loss: 0.012492090463638306 (nan) | > current_lr: 5e-06 | > step_time: 0.1438 (0.13677793750908732) | > loader_time: 0.0046 (0.004709720793832069)  --> TIME: 2025-02-25 01:47:52 -- STEP: 20980/36938 -- GLOBAL_STEP: 575050 | > loss_text_ce: 0.03783927485346794 (0.036384506197246674) | > loss_mel_ce: 2.537355422973633 (nan) | > loss: 0.010219026356935501 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.13678294653206116) | > loader_time: 0.0047 (0.00470960217504076)  --> TIME: 2025-02-25 01:48:02 -- STEP: 21030/36938 -- GLOBAL_STEP: 575100 | > loss_text_ce: 0.02911890670657158 (0.03638269712867701) | > loss_mel_ce: 2.5507497787475586 (nan) | > loss: 0.010237575508654118 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13678618176914184) | > loader_time: 0.0045 (0.004709508778421999)  --> TIME: 2025-02-25 01:48:11 -- STEP: 21080/36938 -- GLOBAL_STEP: 575150 | > loss_text_ce: 0.04712708294391632 (0.036381339095797945) | > loss_mel_ce: 2.4791927337646484 (nan) | > loss: 0.010025079362094402 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1367894330332355) | > loader_time: 0.0044 (0.004709432778819937)  --> TIME: 2025-02-25 01:48:20 -- STEP: 21130/36938 -- GLOBAL_STEP: 575200 | > loss_text_ce: 0.039738692343235016 (0.03637890660473297) | > loss_mel_ce: 2.234574317932129 (nan) | > loss: 0.009025052189826965 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13678505629428) | > loader_time: 0.0048 (0.0047094683143915584)  --> TIME: 2025-02-25 01:48:30 -- STEP: 21180/36938 -- GLOBAL_STEP: 575250 | > loss_text_ce: 0.04005858674645424 (0.03638533578598144) | > loss_mel_ce: 3.0732617378234863 (nan) | > loss: 0.012354446575045586 (nan) | > current_lr: 5e-06 | > step_time: 0.1855 (0.1367906261435987) | > loader_time: 0.0049 (0.0047095822094745474)  --> TIME: 2025-02-25 01:48:39 -- STEP: 21230/36938 -- GLOBAL_STEP: 575300 | > loss_text_ce: 0.0347745306789875 (0.03638347999319555) | > loss_mel_ce: 2.917332172393799 (nan) | > loss: 0.011714709922671318 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13678898368867787) | > loader_time: 0.0046 (0.004709418910435796)  --> TIME: 2025-02-25 01:48:48 -- STEP: 21280/36938 -- GLOBAL_STEP: 575350 | > loss_text_ce: 0.032606400549411774 (0.036381433064230105) | > loss_mel_ce: 2.7267329692840576 (nan) | > loss: 0.010949759744107723 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.13676906631405172) | > loader_time: 0.0048 (0.004709196886173776)  --> TIME: 2025-02-25 01:48:57 -- STEP: 21330/36938 -- GLOBAL_STEP: 575400 | > loss_text_ce: 0.04673086851835251 (0.03638359240347508) | > loss_mel_ce: 2.9191184043884277 (nan) | > loss: 0.011769243516027927 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13677151092497083) | > loader_time: 0.0047 (0.004709126912424181)  --> TIME: 2025-02-25 01:49:07 -- STEP: 21380/36938 -- GLOBAL_STEP: 575450 | > loss_text_ce: 0.027494041249155998 (0.03638403605045955) | > loss_mel_ce: 2.7051877975463867 (nan) | > loss: 0.010843975469470024 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13678229741675446) | > loader_time: 0.0046 (0.004708975369513214)  --> TIME: 2025-02-25 01:49:17 -- STEP: 21430/36938 -- GLOBAL_STEP: 575500 | > loss_text_ce: 0.03232515603303909 (0.03638525544072545) | > loss_mel_ce: 4.714749336242676 (nan) | > loss: 0.018837599083781242 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13678784127887333) | > loader_time: 0.0046 (0.004708782056754193)  --> TIME: 2025-02-25 01:49:26 -- STEP: 21480/36938 -- GLOBAL_STEP: 575550 | > loss_text_ce: 0.03399849683046341 (0.03638439538066579) | > loss_mel_ce: 2.9323883056640625 (nan) | > loss: 0.011771377176046371 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.1367936721061192) | > loader_time: 0.0046 (0.004708581052900929)  --> TIME: 2025-02-25 01:49:36 -- STEP: 21530/36938 -- GLOBAL_STEP: 575600 | > loss_text_ce: 0.030975764617323875 (0.03638383864545815) | > loss_mel_ce: 1.9839423894882202 (nan) | > loss: 0.007995706982910633 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13679765465643784) | > loader_time: 0.0046 (0.004708441777723217)  --> TIME: 2025-02-25 01:49:44 -- STEP: 21580/36938 -- GLOBAL_STEP: 575650 | > loss_text_ce: 0.03908057138323784 (0.03638140471670282) | > loss_mel_ce: 2.970623016357422 (nan) | > loss: 0.011943269520998001 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13678552757489423) | > loader_time: 0.0048 (0.004708227567699261)  --> TIME: 2025-02-25 01:49:54 -- STEP: 21630/36938 -- GLOBAL_STEP: 575700 | > loss_text_ce: 0.036344025284051895 (0.03638111489837445) | > loss_mel_ce: 0.08538818359375 (nan) | > loss: 0.00048306433018296957 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13678591178624194) | > loader_time: 0.0046 (0.004708446334260421)  --> TIME: 2025-02-25 01:50:03 -- STEP: 21680/36938 -- GLOBAL_STEP: 575750 | > loss_text_ce: 0.04870505630970001 (0.03638348680404555) | > loss_mel_ce: 3.5576765537261963 (nan) | > loss: 0.014311038888990879 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.1367965160920608) | > loader_time: 0.0047 (0.004708571407627814)  --> TIME: 2025-02-25 01:50:13 -- STEP: 21730/36938 -- GLOBAL_STEP: 575800 | > loss_text_ce: 0.019921284168958664 (0.03638018021596631) | > loss_mel_ce: 2.3442981243133545 (nan) | > loss: 0.009381823241710663 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1368026021324536) | > loader_time: 0.0045 (0.004708921794127735)  --> TIME: 2025-02-25 01:50:22 -- STEP: 21780/36938 -- GLOBAL_STEP: 575850 | > loss_text_ce: 0.031023114919662476 (0.036378182550119195) | > loss_mel_ce: 1.8241373300552368 (nan) | > loss: 0.00736174825578928 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13679989895982586) | > loader_time: 0.0045 (0.004709188646959336)  --> TIME: 2025-02-25 01:50:31 -- STEP: 21830/36938 -- GLOBAL_STEP: 575900 | > loss_text_ce: 0.046133656054735184 (0.036379541426025064) | > loss_mel_ce: 1.7010986804962158 (nan) | > loss: 0.0069334618747234344 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.1367906757958478) | > loader_time: 0.0044 (0.00470900830466761)  --> TIME: 2025-02-25 01:50:41 -- STEP: 21880/36938 -- GLOBAL_STEP: 575950 | > loss_text_ce: 0.044607870280742645 (0.036380329940398276) | > loss_mel_ce: 1.7686192989349365 (nan) | > loss: 0.007195346523076296 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13679788732659404) | > loader_time: 0.0045 (0.004709050511529302)  --> TIME: 2025-02-25 01:50:50 -- STEP: 21930/36938 -- GLOBAL_STEP: 576000 | > loss_text_ce: 0.039860453456640244 (0.036379777771627604) | > loss_mel_ce: 3.636408567428589 (nan) | > loss: 0.014588369987905025 (nan) | > current_lr: 5e-06 | > step_time: 0.1569 (0.13679435128117584) | > loader_time: 0.0047 (0.004708905628811444)  --> TIME: 2025-02-25 01:50:59 -- STEP: 21980/36938 -- GLOBAL_STEP: 576050 | > loss_text_ce: 0.034936752170324326 (0.036379210763162605) | > loss_mel_ce: 2.7377772331237793 (nan) | > loss: 0.01100283395498991 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.1367945412487412) | > loader_time: 0.0047 (0.004708961522828695)  --> TIME: 2025-02-25 01:51:09 -- STEP: 22030/36938 -- GLOBAL_STEP: 576100 | > loss_text_ce: 0.03605300933122635 (0.03637879836659232) | > loss_mel_ce: 1.448197841644287 (nan) | > loss: 0.005889884661883116 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13680931964463852) | > loader_time: 0.0054 (0.004709242129401638)  --> TIME: 2025-02-25 01:51:18 -- STEP: 22080/36938 -- GLOBAL_STEP: 576150 | > loss_text_ce: 0.032958999276161194 (0.036377162511580234) | > loss_mel_ce: 2.1406543254852295 (nan) | > loss: 0.008625450544059277 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13680929020047156) | > loader_time: 0.0045 (0.004709424056868633)  --> TIME: 2025-02-25 01:51:28 -- STEP: 22130/36938 -- GLOBAL_STEP: 576200 | > loss_text_ce: 0.03265247121453285 (0.036375337587421175) | > loss_mel_ce: 3.1985490322113037 (nan) | > loss: 0.012822228483855724 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.13683246680637598) | > loader_time: 0.0048 (0.004709465741788011)  --> TIME: 2025-02-25 01:51:37 -- STEP: 22180/36938 -- GLOBAL_STEP: 576250 | > loss_text_ce: 0.036040790379047394 (0.036375723574839626) | > loss_mel_ce: 3.085977077484131 (nan) | > loss: 0.01238896045833826 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.13682693453712183) | > loader_time: 0.0046 (0.004709277688328279)  --> TIME: 2025-02-25 01:51:48 -- STEP: 22230/36938 -- GLOBAL_STEP: 576300 | > loss_text_ce: 0.031562868505716324 (0.036375541278059316) | > loss_mel_ce: 2.9505176544189453 (nan) | > loss: 0.01183365285396576 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.13685283865737707) | > loader_time: 0.0049 (0.00470926033394112)  --> TIME: 2025-02-25 01:51:57 -- STEP: 22280/36938 -- GLOBAL_STEP: 576350 | > loss_text_ce: 0.0481983944773674 (0.03637541234438916) | > loss_mel_ce: 3.295266628265381 (nan) | > loss: 0.013267719186842442 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13684460171891244) | > loader_time: 0.0046 (0.0047092951927835865)  --> TIME: 2025-02-25 01:52:06 -- STEP: 22330/36938 -- GLOBAL_STEP: 576400 | > loss_text_ce: 0.034794460982084274 (0.036378528255021726) | > loss_mel_ce: 2.7787673473358154 (nan) | > loss: 0.011164928786456585 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13684237595902762) | > loader_time: 0.0047 (0.00470938529874306)  --> TIME: 2025-02-25 01:52:15 -- STEP: 22380/36938 -- GLOBAL_STEP: 576450 | > loss_text_ce: 0.04136350005865097 (0.03637693385498715) | > loss_mel_ce: 2.905395984649658 (nan) | > loss: 0.011693490669131279 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13683340428022475) | > loader_time: 0.0046 (0.004709414055136526)  --> TIME: 2025-02-25 01:52:24 -- STEP: 22430/36938 -- GLOBAL_STEP: 576500 | > loss_text_ce: 0.03801438584923744 (0.03637688291154066) | > loss_mel_ce: 3.01385498046875 (nan) | > loss: 0.012110593728721142 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13683327675078258) | > loader_time: 0.0052 (0.0047097624384607704)  --> TIME: 2025-02-25 01:52:34 -- STEP: 22480/36938 -- GLOBAL_STEP: 576550 | > loss_text_ce: 0.030317004770040512 (0.03637495181283809) | > loss_mel_ce: 0.009368896484375 (nan) | > loss: 0.00015748373698443174 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13684737244322573) | > loader_time: 0.0047 (0.004709811694257218)  --> TIME: 2025-02-25 01:52:43 -- STEP: 22530/36938 -- GLOBAL_STEP: 576600 | > loss_text_ce: 0.03936579078435898 (0.03637373096161207) | > loss_mel_ce: 2.5938050746917725 (nan) | > loss: 0.010449090972542763 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13683889173582997) | > loader_time: 0.0045 (0.004709567782405866)  --> TIME: 2025-02-25 01:52:53 -- STEP: 22580/36938 -- GLOBAL_STEP: 576650 | > loss_text_ce: 0.038453780114650726 (0.03637119237295598) | > loss_mel_ce: 2.881822347640991 (nan) | > loss: 0.01158839836716652 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13683998860332974) | > loader_time: 0.0046 (0.004709645643817122)  --> TIME: 2025-02-25 01:53:02 -- STEP: 22630/36938 -- GLOBAL_STEP: 576700 | > loss_text_ce: 0.040446437895298004 (0.036368775965050906) | > loss_mel_ce: 2.9343016147613525 (nan) | > loss: 0.011804556474089622 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13684303037792572) | > loader_time: 0.0047 (0.0047095777711350165)  --> TIME: 2025-02-25 01:53:11 -- STEP: 22680/36938 -- GLOBAL_STEP: 576750 | > loss_text_ce: 0.044655729085206985 (0.03636646290881343) | > loss_mel_ce: 3.2054121494293213 (nan) | > loss: 0.012897095642983913 (nan) | > current_lr: 5e-06 | > step_time: 0.1625 (0.1368444717329853) | > loader_time: 0.0049 (0.004709595694113053)  --> TIME: 2025-02-25 01:53:21 -- STEP: 22730/36938 -- GLOBAL_STEP: 576800 | > loss_text_ce: 0.03248097747564316 (0.03636398382093219) | > loss_mel_ce: 2.39294171333313 (nan) | > loss: 0.009624693542718887 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13683964492883396) | > loader_time: 0.0045 (0.004709541750436748)  --> TIME: 2025-02-25 01:53:30 -- STEP: 22780/36938 -- GLOBAL_STEP: 576850 | > loss_text_ce: 0.03153986111283302 (0.0363632675329808) | > loss_mel_ce: 8.125 (nan) | > loss: 0.032367222011089325 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13683329669920521) | > loader_time: 0.0044 (0.004709523146983528)  --> TIME: 2025-02-25 01:53:39 -- STEP: 22830/36938 -- GLOBAL_STEP: 576900 | > loss_text_ce: 0.02809128724038601 (0.03636510687595896) | > loss_mel_ce: 2.953035593032837 (nan) | > loss: 0.011829868890345097 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13683511857865244) | > loader_time: 0.0046 (0.004709406855646291)  --> TIME: 2025-02-25 01:53:49 -- STEP: 22880/36938 -- GLOBAL_STEP: 576950 | > loss_text_ce: 0.04474449157714844 (0.036365083329073566) | > loss_mel_ce: 2.561039686203003 (nan) | > loss: 0.010340414009988308 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1368381240463751) | > loader_time: 0.0045 (0.004709273024455677)  --> TIME: 2025-02-25 01:53:58 -- STEP: 22930/36938 -- GLOBAL_STEP: 577000 | > loss_text_ce: 0.04800216481089592 (0.036364621335516385) | > loss_mel_ce: 1.6699923276901245 (nan) | > loss: 0.006817438639700413 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13683648812318486) | > loader_time: 0.0048 (0.004709069603039539)  --> TIME: 2025-02-25 01:54:07 -- STEP: 22980/36938 -- GLOBAL_STEP: 577050 | > loss_text_ce: 0.03284519165754318 (0.036367842722424895) | > loss_mel_ce: 2.7657527923583984 (nan) | > loss: 0.011105548590421677 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13683133173030312) | > loader_time: 0.0047 (0.0047088793301395876)  --> TIME: 2025-02-25 01:54:16 -- STEP: 23030/36938 -- GLOBAL_STEP: 577100 | > loss_text_ce: 0.039405807852745056 (0.036368230321936826) | > loss_mel_ce: 3.1842617988586426 (nan) | > loss: 0.012792333029210567 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13683085323570532) | > loader_time: 0.0047 (0.004708737815611816)  --> TIME: 2025-02-25 01:54:26 -- STEP: 23080/36938 -- GLOBAL_STEP: 577150 | > loss_text_ce: 0.03505367413163185 (0.03636624257151288) | > loss_mel_ce: 1.9117851257324219 (nan) | > loss: 0.007725551258772612 (nan) | > current_lr: 5e-06 | > step_time: 0.1827 (0.13683584946073654) | > loader_time: 0.005 (0.004708721805073391)  --> TIME: 2025-02-25 01:54:35 -- STEP: 23130/36938 -- GLOBAL_STEP: 577200 | > loss_text_ce: 0.024349182844161987 (0.03636437211048003) | > loss_mel_ce: 2.5670995712280273 (nan) | > loss: 0.01028352789580822 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13683638206767185) | > loader_time: 0.0047 (0.0047086798881897235)  --> TIME: 2025-02-25 01:54:45 -- STEP: 23180/36938 -- GLOBAL_STEP: 577250 | > loss_text_ce: 0.03332047536969185 (0.03636573887545856) | > loss_mel_ce: 3.0136783123016357 (nan) | > loss: 0.012091265991330147 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.13683557418834696) | > loader_time: 0.0046 (0.00470849538898554)  --> TIME: 2025-02-25 01:54:54 -- STEP: 23230/36938 -- GLOBAL_STEP: 577300 | > loss_text_ce: 0.03486136719584465 (0.03636276036027059) | > loss_mel_ce: 2.492154359817505 (nan) | > loss: 0.010027840733528137 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.13683409472246635) | > loader_time: 0.0047 (0.004708301040872788)  --> TIME: 2025-02-25 01:55:03 -- STEP: 23280/36938 -- GLOBAL_STEP: 577350 | > loss_text_ce: 0.028999095782637596 (0.03636383327189037) | > loss_mel_ce: 2.386652946472168 (nan) | > loss: 0.00958592165261507 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.1368383310812029) | > loader_time: 0.0046 (0.004708134687643323)  --> TIME: 2025-02-25 01:55:12 -- STEP: 23330/36938 -- GLOBAL_STEP: 577400 | > loss_text_ce: 0.034863296896219254 (0.036361744014072876) | > loss_mel_ce: 2.94484281539917 (nan) | > loss: 0.011824230663478374 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13683513709009784) | > loader_time: 0.0046 (0.004707947586720195)  --> TIME: 2025-02-25 01:55:22 -- STEP: 23380/36938 -- GLOBAL_STEP: 577450 | > loss_text_ce: 0.03587428107857704 (0.036361746132808834) | > loss_mel_ce: 3.3710405826568604 (nan) | > loss: 0.013519504107534885 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.1368298874337817) | > loader_time: 0.0049 (0.004707818157760625)  --> TIME: 2025-02-25 01:55:31 -- STEP: 23430/36938 -- GLOBAL_STEP: 577500 | > loss_text_ce: 0.03646319359540939 (0.03635931188273246) | > loss_mel_ce: 2.699309825897217 (nan) | > loss: 0.01085624285042286 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13682449147887427) | > loader_time: 0.0046 (0.004707705827030059)  --> TIME: 2025-02-25 01:55:40 -- STEP: 23480/36938 -- GLOBAL_STEP: 577550 | > loss_text_ce: 0.034052394330501556 (0.03635696279737006) | > loss_mel_ce: 3.0875091552734375 (nan) | > loss: 0.012387149967253208 (nan) | > current_lr: 5e-06 | > step_time: 0.1457 (0.1368168267115209) | > loader_time: 0.0046 (0.004707539812669686)  --> TIME: 2025-02-25 01:55:49 -- STEP: 23530/36938 -- GLOBAL_STEP: 577600 | > loss_text_ce: 0.04112871736288071 (0.03635846509353201) | > loss_mel_ce: 2.770939588546753 (nan) | > loss: 0.011159001849591732 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13681912779858635) | > loader_time: 0.0046 (0.004707436788553328)  --> TIME: 2025-02-25 01:55:59 -- STEP: 23580/36938 -- GLOBAL_STEP: 577650 | > loss_text_ce: 0.02761564403772354 (0.0363590259028345) | > loss_mel_ce: 2.775686264038086 (nan) | > loss: 0.01112421415746212 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13682298832166112) | > loader_time: 0.0047 (0.004707299560889884)  --> TIME: 2025-02-25 01:56:08 -- STEP: 23630/36938 -- GLOBAL_STEP: 577700 | > loss_text_ce: 0.035153742879629135 (0.036355958260450856) | > loss_mel_ce: 3.470982074737549 (nan) | > loss: 0.013913237489759922 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13682032156131776) | > loader_time: 0.0046 (0.004707078715779285)  --> TIME: 2025-02-25 01:56:17 -- STEP: 23680/36938 -- GLOBAL_STEP: 577750 | > loss_text_ce: 0.03318268805742264 (0.03635712820128861) | > loss_mel_ce: 2.568585157394409 (nan) | > loss: 0.010324476286768913 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.1368127160177031) | > loader_time: 0.0045 (0.004706861642566928)  --> TIME: 2025-02-25 01:56:26 -- STEP: 23730/36938 -- GLOBAL_STEP: 577800 | > loss_text_ce: 0.028261927887797356 (0.036357321583120855) | > loss_mel_ce: 7.328125 (nan) | > loss: 0.029192013666033745 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13680890406227936) | > loader_time: 0.0045 (0.004706691469866915)  --> TIME: 2025-02-25 01:56:36 -- STEP: 23780/36938 -- GLOBAL_STEP: 577850 | > loss_text_ce: 0.037439536303281784 (0.0363536439501956) | > loss_mel_ce: 2.8171072006225586 (nan) | > loss: 0.011327567510306835 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.13681062134100197) | > loader_time: 0.0053 (0.004706996844735283)  --> TIME: 2025-02-25 01:56:45 -- STEP: 23830/36938 -- GLOBAL_STEP: 577900 | > loss_text_ce: 0.03055647388100624 (0.03635045247730087) | > loss_mel_ce: 3.2434895038604736 (nan) | > loss: 0.012992247007787228 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13680632773738247) | > loader_time: 0.0048 (0.004707009102985075)  --> TIME: 2025-02-25 01:56:54 -- STEP: 23880/36938 -- GLOBAL_STEP: 577950 | > loss_text_ce: 0.04752090945839882 (0.03634838799117655) | > loss_mel_ce: 3.0288662910461426 (nan) | > loss: 0.012207886204123497 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1368057457645927) | > loader_time: 0.0069 (0.004707219103472958)  --> TIME: 2025-02-25 01:57:03 -- STEP: 23930/36938 -- GLOBAL_STEP: 578000 | > loss_text_ce: 0.03887864202260971 (0.036346537509456066) | > loss_mel_ce: 3.3764708042144775 (nan) | > loss: 0.013552974909543991 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13679718266856475) | > loader_time: 0.0045 (0.004707498377055984)  --> TIME: 2025-02-25 01:57:12 -- STEP: 23980/36938 -- GLOBAL_STEP: 578050 | > loss_text_ce: 0.04267619922757149 (0.036347522723214405) | > loss_mel_ce: 3.0337727069854736 (nan) | > loss: 0.012208131141960621 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13678799179616347) | > loader_time: 0.0046 (0.004707376115813314)  --> TIME: 2025-02-25 01:57:21 -- STEP: 24030/36938 -- GLOBAL_STEP: 578100 | > loss_text_ce: 0.033448148518800735 (0.03634437747002029) | > loss_mel_ce: 2.837777853012085 (nan) | > loss: 0.011393754743039608 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13678156318537482) | > loader_time: 0.0045 (0.004707447757236817)  --> TIME: 2025-02-25 01:57:31 -- STEP: 24080/36938 -- GLOBAL_STEP: 578150 | > loss_text_ce: 0.03943979740142822 (0.036344684162904345) | > loss_mel_ce: 3.0318193435668945 (nan) | > loss: 0.012187536805868149 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13677894700801183) | > loader_time: 0.0047 (0.004707526051720956)  --> TIME: 2025-02-25 01:57:40 -- STEP: 24130/36938 -- GLOBAL_STEP: 578200 | > loss_text_ce: 0.04067965969443321 (0.03634502978905137) | > loss_mel_ce: 3.4617292881011963 (nan) | > loss: 0.013898449018597603 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13677606478895357) | > loader_time: 0.0046 (0.00470766004466739)  --> TIME: 2025-02-25 01:57:49 -- STEP: 24180/36938 -- GLOBAL_STEP: 578250 | > loss_text_ce: 0.03567805141210556 (0.036346144649077754) | > loss_mel_ce: 2.9712724685668945 (nan) | > loss: 0.011932344175875187 (nan) | > current_lr: 5e-06 | > step_time: 0.1446 (0.13677797811498635) | > loader_time: 0.0053 (0.004707757710818962)  --> TIME: 2025-02-25 01:57:59 -- STEP: 24230/36938 -- GLOBAL_STEP: 578300 | > loss_text_ce: 0.032322242856025696 (0.03634444684969457) | > loss_mel_ce: 2.7982327938079834 (nan) | > loss: 0.011232361197471619 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13678025132705407) | > loader_time: 0.0046 (0.004707646606089555)  --> TIME: 2025-02-25 01:58:08 -- STEP: 24280/36938 -- GLOBAL_STEP: 578350 | > loss_text_ce: 0.029947111383080482 (0.03634445986455805) | > loss_mel_ce: 2.985907554626465 (nan) | > loss: 0.011967677623033524 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13677937314773425) | > loader_time: 0.0048 (0.004707500706787412)  --> TIME: 2025-02-25 01:58:18 -- STEP: 24330/36938 -- GLOBAL_STEP: 578400 | > loss_text_ce: 0.03998224809765816 (0.036348382630559756) | > loss_mel_ce: 3.1500589847564697 (nan) | > loss: 0.012658894993364811 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.13679716456734997) | > loader_time: 0.0046 (0.004707416770784595)  --> TIME: 2025-02-25 01:58:27 -- STEP: 24380/36938 -- GLOBAL_STEP: 578450 | > loss_text_ce: 0.02952309139072895 (0.036348939384162025) | > loss_mel_ce: 2.990994930267334 (nan) | > loss: 0.011986183933913708 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13680350293479374) | > loader_time: 0.0046 (0.004707595801333949)  --> TIME: 2025-02-25 01:58:37 -- STEP: 24430/36938 -- GLOBAL_STEP: 578500 | > loss_text_ce: 0.03229569271206856 (0.03634708691609488) | > loss_mel_ce: 3.557116985321045 (nan) | > loss: 0.01424370240420103 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.13680280430903366) | > loader_time: 0.0055 (0.004707706789476638)  --> TIME: 2025-02-25 01:58:46 -- STEP: 24480/36938 -- GLOBAL_STEP: 578550 | > loss_text_ce: 0.025961631909012794 (0.036348777789290916) | > loss_mel_ce: 2.8727636337280273 (nan) | > loss: 0.011502878740429878 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1367998768301562) | > loader_time: 0.0046 (0.004707874552486826)  --> TIME: 2025-02-25 01:58:56 -- STEP: 24530/36938 -- GLOBAL_STEP: 578600 | > loss_text_ce: 0.04329155012965202 (0.03634842490823705) | > loss_mel_ce: 3.4320766925811768 (nan) | > loss: 0.013791144825518131 (nan) | > current_lr: 5e-06 | > step_time: 0.1788 (0.13681157272881503) | > loader_time: 0.0048 (0.004707906385173769)  --> TIME: 2025-02-25 01:59:05 -- STEP: 24580/36938 -- GLOBAL_STEP: 578650 | > loss_text_ce: 0.03497850522398949 (0.03634954783694179) | > loss_mel_ce: 3.233853340148926 (nan) | > loss: 0.012971554882824421 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13681132126087198) | > loader_time: 0.0047 (0.004707866824672372)  --> TIME: 2025-02-25 01:59:14 -- STEP: 24630/36938 -- GLOBAL_STEP: 578700 | > loss_text_ce: 0.022738004103302956 (0.036345353850454405) | > loss_mel_ce: 2.313664436340332 (nan) | > loss: 0.009271438233554363 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13681359652140845) | > loader_time: 0.0045 (0.004707851905528534)  --> TIME: 2025-02-25 01:59:24 -- STEP: 24680/36938 -- GLOBAL_STEP: 578750 | > loss_text_ce: 0.046792544424533844 (0.03634501274069855) | > loss_mel_ce: 3.195953607559204 (nan) | > loss: 0.012868041172623634 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13682331263149503) | > loader_time: 0.0045 (0.004707892516830227)  --> TIME: 2025-02-25 01:59:33 -- STEP: 24730/36938 -- GLOBAL_STEP: 578800 | > loss_text_ce: 0.04889555647969246 (0.03634971616773894) | > loss_mel_ce: 2.5537872314453125 (nan) | > loss: 0.010328107513487339 (nan) | > current_lr: 5e-06 | > step_time: 0.1887 (0.13682542893262742) | > loader_time: 0.005 (0.004708217051267386)  --> TIME: 2025-02-25 01:59:42 -- STEP: 24780/36938 -- GLOBAL_STEP: 578850 | > loss_text_ce: 0.042436886578798294 (0.036350952675889406) | > loss_mel_ce: 1.011559009552002 (nan) | > loss: 0.0041825235821306705 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13681738256349732) | > loader_time: 0.0045 (0.004708354197941693)  --> TIME: 2025-02-25 01:59:52 -- STEP: 24830/36938 -- GLOBAL_STEP: 578900 | > loss_text_ce: 0.03743542730808258 (0.03635058107302415) | > loss_mel_ce: 2.8802504539489746 (nan) | > loss: 0.011578119359910488 (nan) | > current_lr: 5e-06 | > step_time: 0.1842 (0.13682787406842992) | > loader_time: 0.0049 (0.004708455245532342)  --> TIME: 2025-02-25 02:00:01 -- STEP: 24880/36938 -- GLOBAL_STEP: 578950 | > loss_text_ce: 0.05013386160135269 (0.03634951915120317) | > loss_mel_ce: 3.1152045726776123 (nan) | > loss: 0.012560867704451084 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.136833574265912) | > loader_time: 0.0047 (0.004708541369131578)  --> TIME: 2025-02-25 02:00:11 -- STEP: 24930/36938 -- GLOBAL_STEP: 579000 | > loss_text_ce: 0.03352125361561775 (0.03635114854552099) | > loss_mel_ce: 3.7081668376922607 (nan) | > loss: 0.014847968704998493 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.13684217922189218) | > loader_time: 0.0048 (0.004708518170947066)  --> TIME: 2025-02-25 02:00:20 -- STEP: 24980/36938 -- GLOBAL_STEP: 579050 | > loss_text_ce: 0.037364114075899124 (0.036349595299406776) | > loss_mel_ce: 3.030449867248535 (nan) | > loss: 0.012173864990472794 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13684106484902525) | > loader_time: 0.0046 (0.004708516454582181)  --> TIME: 2025-02-25 02:00:30 -- STEP: 25030/36938 -- GLOBAL_STEP: 579100 | > loss_text_ce: 0.03336566314101219 (0.03634932591299685) | > loss_mel_ce: 2.9598450660705566 (nan) | > loss: 0.011877820827066898 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13684665725272072) | > loader_time: 0.0054 (0.0047084578789571745)  --> TIME: 2025-02-25 02:00:39 -- STEP: 25080/36938 -- GLOBAL_STEP: 579150 | > loss_text_ce: 0.03001844882965088 (0.036348896679294744) | > loss_mel_ce: 3.2610607147216797 (nan) | > loss: 0.01305983867496252 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.136838383594768) | > loader_time: 0.0046 (0.00470833674762434)  --> TIME: 2025-02-25 02:00:48 -- STEP: 25130/36938 -- GLOBAL_STEP: 579200 | > loss_text_ce: 0.030688349157571793 (0.03634879170063485) | > loss_mel_ce: 3.3211731910705566 (nan) | > loss: 0.013301038183271885 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.13684061305057046) | > loader_time: 0.0048 (0.004708259768852296)  --> TIME: 2025-02-25 02:00:58 -- STEP: 25180/36938 -- GLOBAL_STEP: 579250 | > loss_text_ce: 0.04088771715760231 (0.036348820270969036) | > loss_mel_ce: 3.1410834789276123 (nan) | > loss: 0.012626871466636658 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13683659070252016) | > loader_time: 0.0051 (0.004708249423123301)  --> TIME: 2025-02-25 02:01:07 -- STEP: 25230/36938 -- GLOBAL_STEP: 579300 | > loss_text_ce: 0.027199599891901016 (0.03634949823641388) | > loss_mel_ce: 3.305206298828125 (nan) | > loss: 0.013223833404481411 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.13683790230722656) | > loader_time: 0.0046 (0.004708292283002666)  --> TIME: 2025-02-25 02:01:16 -- STEP: 25280/36938 -- GLOBAL_STEP: 579350 | > loss_text_ce: 0.035499196499586105 (0.03634807014045061) | > loss_mel_ce: 3.3034281730651855 (nan) | > loss: 0.013249712064862251 (nan) | > current_lr: 5e-06 | > step_time: 0.1891 (0.1368398626488217) | > loader_time: 0.0049 (0.004708132015753461)  --> TIME: 2025-02-25 02:01:25 -- STEP: 25330/36938 -- GLOBAL_STEP: 579400 | > loss_text_ce: 0.04073683172464371 (0.03634612305781801) | > loss_mel_ce: 2.7062172889709473 (nan) | > loss: 0.01090061292052269 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13682990704795037) | > loader_time: 0.0045 (0.004708057065467581)  --> TIME: 2025-02-25 02:01:35 -- STEP: 25380/36938 -- GLOBAL_STEP: 579450 | > loss_text_ce: 0.05123583599925041 (0.036347681463903804) | > loss_mel_ce: 1.459538459777832 (nan) | > loss: 0.005995136220008135 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13683748251814962) | > loader_time: 0.0045 (0.004708131295287643)  --> TIME: 2025-02-25 02:01:44 -- STEP: 25430/36938 -- GLOBAL_STEP: 579500 | > loss_text_ce: 0.03457968309521675 (0.036346357834067276) | > loss_mel_ce: 2.5174014568328857 (nan) | > loss: 0.010126910172402859 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13683833775912158) | > loader_time: 0.0053 (0.0047080166453318285)  --> TIME: 2025-02-25 02:01:54 -- STEP: 25480/36938 -- GLOBAL_STEP: 579550 | > loss_text_ce: 0.04420200735330582 (0.036344515639715504) | > loss_mel_ce: 2.4741179943084717 (nan) | > loss: 0.009993334300816059 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13683565331965014) | > loader_time: 0.0045 (0.004708063312377875)  --> TIME: 2025-02-25 02:02:03 -- STEP: 25530/36938 -- GLOBAL_STEP: 579600 | > loss_text_ce: 0.04174666479229927 (0.036343514150511305) | > loss_mel_ce: 2.412083148956299 (nan) | > loss: 0.00973741989582777 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.1368369472292972) | > loader_time: 0.0046 (0.004708101391745667)  --> TIME: 2025-02-25 02:02:13 -- STEP: 25580/36938 -- GLOBAL_STEP: 579650 | > loss_text_ce: 0.03520336002111435 (0.036344127740312) | > loss_mel_ce: 3.3717010021209717 (nan) | > loss: 0.013519463129341602 (nan) | > current_lr: 5e-06 | > step_time: 0.1896 (0.13684772345309434) | > loader_time: 0.0056 (0.004708218285588261)  --> TIME: 2025-02-25 02:02:22 -- STEP: 25630/36938 -- GLOBAL_STEP: 579700 | > loss_text_ce: 0.03216039389371872 (0.036343416511406455) | > loss_mel_ce: 3.1048009395599365 (nan) | > loss: 0.012448259629309177 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13685458199474534) | > loader_time: 0.0048 (0.004708225151267471)  --> TIME: 2025-02-25 02:02:31 -- STEP: 25680/36938 -- GLOBAL_STEP: 579750 | > loss_text_ce: 0.03452156111598015 (0.036343827904630026) | > loss_mel_ce: 0.07586669921875 (nan) | > loss: 0.0004380486789159477 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.13684564306171432) | > loader_time: 0.0047 (0.004708188261569907)  --> TIME: 2025-02-25 02:02:40 -- STEP: 25730/36938 -- GLOBAL_STEP: 579800 | > loss_text_ce: 0.03856361657381058 (0.03634490742269592) | > loss_mel_ce: 3.460170269012451 (nan) | > loss: 0.013883865438401699 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13683084709561769) | > loader_time: 0.0045 (0.0047081072878106195)  --> TIME: 2025-02-25 02:02:49 -- STEP: 25780/36938 -- GLOBAL_STEP: 579850 | > loss_text_ce: 0.03134322911500931 (0.036345010385375374) | > loss_mel_ce: 2.5194036960601807 (nan) | > loss: 0.010122012346982956 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13682957586158145) | > loader_time: 0.0046 (0.004708159090105804)  --> TIME: 2025-02-25 02:02:59 -- STEP: 25830/36938 -- GLOBAL_STEP: 579900 | > loss_text_ce: 0.03706071153283119 (0.036347026642386) | > loss_mel_ce: 3.2636327743530273 (nan) | > loss: 0.013097991235554218 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.13683026251384833) | > loader_time: 0.0048 (0.004708130859008804)  --> TIME: 2025-02-25 02:03:08 -- STEP: 25880/36938 -- GLOBAL_STEP: 579950 | > loss_text_ce: 0.03343198075890541 (0.036346030120746504) | > loss_mel_ce: 2.920830726623535 (nan) | > loss: 0.011723265051841736 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.1368333560145427) | > loader_time: 0.0046 (0.0047080649290424485)  --> TIME: 2025-02-25 02:03:17 -- STEP: 25930/36938 -- GLOBAL_STEP: 580000 | > loss_text_ce: 0.03567282110452652 (0.03634744269868182) | > loss_mel_ce: 2.7038373947143555 (nan) | > loss: 0.010871073231101036 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13682444400875537) | > loader_time: 0.0044 (0.00470788784115375) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_580000.pth  --> TIME: 2025-02-25 02:03:32 -- STEP: 25980/36938 -- GLOBAL_STEP: 580050 | > loss_text_ce: 0.034188706427812576 (0.03634471279157436) | > loss_mel_ce: 3.1395816802978516 (nan) | > loss: 0.012594328261911869 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1368146724477001) | > loader_time: 0.0045 (0.004707969868522749)  --> TIME: 2025-02-25 02:03:41 -- STEP: 26030/36938 -- GLOBAL_STEP: 580100 | > loss_text_ce: 0.025733044371008873 (0.036342159344353496) | > loss_mel_ce: 3.086393117904663 (nan) | > loss: 0.012349707074463367 (nan) | > current_lr: 5e-06 | > step_time: 0.1764 (0.13680517640518605) | > loader_time: 0.0048 (0.004708054850663003)  --> TIME: 2025-02-25 02:03:50 -- STEP: 26080/36938 -- GLOBAL_STEP: 580150 | > loss_text_ce: 0.03164076432585716 (0.036340484661842865) | > loss_mel_ce: 2.6887545585632324 (nan) | > loss: 0.01079521980136633 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13680116034175652) | > loader_time: 0.0048 (0.004708223309985029)  --> TIME: 2025-02-25 02:03:59 -- STEP: 26130/36938 -- GLOBAL_STEP: 580200 | > loss_text_ce: 0.027802560478448868 (0.036340209743249634) | > loss_mel_ce: 2.2781925201416016 (nan) | > loss: 0.009150774218142033 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13679160119197878) | > loader_time: 0.0044 (0.004708424610878164)  --> TIME: 2025-02-25 02:04:09 -- STEP: 26180/36938 -- GLOBAL_STEP: 580250 | > loss_text_ce: 0.034340109676122665 (0.03634002999953261) | > loss_mel_ce: 3.0170183181762695 (nan) | > loss: 0.012108566239476204 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13678798418722524) | > loader_time: 0.0053 (0.004708597767526146)  --> TIME: 2025-02-25 02:04:18 -- STEP: 26230/36938 -- GLOBAL_STEP: 580300 | > loss_text_ce: 0.03870712220668793 (0.036341208364712205) | > loss_mel_ce: 2.8654634952545166 (nan) | > loss: 0.011524487286806107 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13678368558603649) | > loader_time: 0.0048 (0.004708689185348723)  --> TIME: 2025-02-25 02:04:27 -- STEP: 26280/36938 -- GLOBAL_STEP: 580350 | > loss_text_ce: 0.032076604664325714 (0.03634227832500739) | > loss_mel_ce: 2.5520081520080566 (nan) | > loss: 0.010254304856061935 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13677474302969692) | > loader_time: 0.0047 (0.004708771999568206)  --> TIME: 2025-02-25 02:04:36 -- STEP: 26330/36938 -- GLOBAL_STEP: 580400 | > loss_text_ce: 0.043515902012586594 (0.03634324563940241) | > loss_mel_ce: 1.4309577941894531 (nan) | > loss: 0.0058510866947472095 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13677673948340788) | > loader_time: 0.0046 (0.0047089081955101475)  --> TIME: 2025-02-25 02:04:46 -- STEP: 26380/36938 -- GLOBAL_STEP: 580450 | > loss_text_ce: 0.029650552198290825 (0.036344193915961484) | > loss_mel_ce: 2.770312547683716 (nan) | > loss: 0.01111096516251564 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13678522694734427) | > loader_time: 0.0044 (0.004708992955899815)  --> TIME: 2025-02-25 02:04:55 -- STEP: 26430/36938 -- GLOBAL_STEP: 580500 | > loss_text_ce: 0.02981981635093689 (0.03634261212826503) | > loss_mel_ce: 1.6706924438476562 (nan) | > loss: 0.006748064886778593 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13678150499042888) | > loader_time: 0.0052 (0.004709031606234758)  --> TIME: 2025-02-25 02:05:04 -- STEP: 26480/36938 -- GLOBAL_STEP: 580550 | > loss_text_ce: 0.038028109818696976 (0.036343700762766384) | > loss_mel_ce: 2.9553780555725098 (nan) | > loss: 0.011878595687448978 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.1367803771510222) | > loader_time: 0.0046 (0.004709102911171291)  --> TIME: 2025-02-25 02:05:14 -- STEP: 26530/36938 -- GLOBAL_STEP: 580600 | > loss_text_ce: 0.02998211607336998 (0.036342539994205496) | > loss_mel_ce: 2.817288398742676 (nan) | > loss: 0.011298692785203457 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13678682234167708) | > loader_time: 0.0047 (0.004709250074128546)  --> TIME: 2025-02-25 02:05:24 -- STEP: 26580/36938 -- GLOBAL_STEP: 580650 | > loss_text_ce: 0.03618225082755089 (0.0363409960040146) | > loss_mel_ce: 2.6071112155914307 (nan) | > loss: 0.010489259846508503 (nan) | > current_lr: 5e-06 | > step_time: 0.1556 (0.13679919874336202) | > loader_time: 0.0055 (0.004709386000335492)  --> TIME: 2025-02-25 02:05:33 -- STEP: 26630/36938 -- GLOBAL_STEP: 580700 | > loss_text_ce: 0.04534822329878807 (0.036343400046044735) | > loss_mel_ce: 3.182694911956787 (nan) | > loss: 0.012809695675969124 (nan) | > current_lr: 5e-06 | > step_time: 0.1521 (0.13679410342296325) | > loader_time: 0.0046 (0.00470940184870749)  --> TIME: 2025-02-25 02:05:42 -- STEP: 26680/36938 -- GLOBAL_STEP: 580750 | > loss_text_ce: 0.04724173992872238 (0.036341069245884136) | > loss_mel_ce: 2.219599723815918 (nan) | > loss: 0.008995402604341507 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13679108462412295) | > loader_time: 0.0045 (0.004709471433773977)  --> TIME: 2025-02-25 02:05:52 -- STEP: 26730/36938 -- GLOBAL_STEP: 580800 | > loss_text_ce: 0.043396420776844025 (0.03634104803451916) | > loss_mel_ce: 2.370361328125 (nan) | > loss: 0.009578404948115349 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13679523156726556) | > loader_time: 0.0046 (0.004709628196499434)  --> TIME: 2025-02-25 02:06:01 -- STEP: 26780/36938 -- GLOBAL_STEP: 580850 | > loss_text_ce: 0.04328537359833717 (0.03634019125579174) | > loss_mel_ce: 1.8974980115890503 (nan) | > loss: 0.00770152173936367 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13679304867203854) | > loader_time: 0.0046 (0.004709703055133256)  --> TIME: 2025-02-25 02:06:10 -- STEP: 26830/36938 -- GLOBAL_STEP: 580900 | > loss_text_ce: 0.03352648392319679 (0.0363383335492495) | > loss_mel_ce: 2.850369930267334 (nan) | > loss: 0.011444034054875374 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13678278877714684) | > loader_time: 0.0045 (0.004709740712311262)  --> TIME: 2025-02-25 02:06:19 -- STEP: 26880/36938 -- GLOBAL_STEP: 580950 | > loss_text_ce: 0.03330063819885254 (0.036336419680412284) | > loss_mel_ce: 0.1734822541475296 (nan) | > loss: 0.0008205670746974647 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13678743902239035) | > loader_time: 0.0045 (0.004709713471432577)  --> TIME: 2025-02-25 02:06:29 -- STEP: 26930/36938 -- GLOBAL_STEP: 581000 | > loss_text_ce: 0.03857271745800972 (0.03633436371063605) | > loss_mel_ce: 3.0866267681121826 (nan) | > loss: 0.012401586398482323 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13679649166754518) | > loader_time: 0.0047 (0.004709850878596932)  --> TIME: 2025-02-25 02:06:38 -- STEP: 26980/36938 -- GLOBAL_STEP: 581050 | > loss_text_ce: 0.032303113490343094 (0.03633325937489885) | > loss_mel_ce: 3.4354248046875 (nan) | > loss: 0.013760825619101524 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13678467457518903) | > loader_time: 0.005 (0.004709880019754246)  --> TIME: 2025-02-25 02:06:47 -- STEP: 27030/36938 -- GLOBAL_STEP: 581100 | > loss_text_ce: 0.03734168782830238 (0.03633209252399247) | > loss_mel_ce: 2.613922119140625 (nan) | > loss: 0.01052088849246502 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13678776019156683) | > loader_time: 0.0046 (0.004709873047750048)  --> TIME: 2025-02-25 02:06:56 -- STEP: 27080/36938 -- GLOBAL_STEP: 581150 | > loss_text_ce: 0.041107501834630966 (0.036333953197034426) | > loss_mel_ce: 3.007100820541382 (nan) | > loss: 0.01209606509655714 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13678189540719307) | > loader_time: 0.0047 (0.0047100176663307704)  --> TIME: 2025-02-25 02:07:06 -- STEP: 27130/36938 -- GLOBAL_STEP: 581200 | > loss_text_ce: 0.034896671772003174 (0.03633508324733651) | > loss_mel_ce: 2.960458517074585 (nan) | > loss: 0.01188633032143116 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.13678249771464027) | > loader_time: 0.0047 (0.004710056937316543)  --> TIME: 2025-02-25 02:07:16 -- STEP: 27180/36938 -- GLOBAL_STEP: 581250 | > loss_text_ce: 0.03472067415714264 (0.036337229904682425) | > loss_mel_ce: 3.630507230758667 (nan) | > loss: 0.01454455591738224 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13679795867823052) | > loader_time: 0.0054 (0.004710465252969664)  --> TIME: 2025-02-25 02:07:25 -- STEP: 27230/36938 -- GLOBAL_STEP: 581300 | > loss_text_ce: 0.035056792199611664 (0.036338345574613906) | > loss_mel_ce: 2.9390554428100586 (nan) | > loss: 0.01180203352123499 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.1367846482298277) | > loader_time: 0.0046 (0.004710213541765715)  --> TIME: 2025-02-25 02:07:34 -- STEP: 27280/36938 -- GLOBAL_STEP: 581350 | > loss_text_ce: 0.03935437649488449 (0.03634126476702473) | > loss_mel_ce: 2.633897542953491 (nan) | > loss: 0.010608143173158169 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1367888468451512) | > loader_time: 0.0044 (0.004710184662572849)  --> TIME: 2025-02-25 02:07:43 -- STEP: 27330/36938 -- GLOBAL_STEP: 581400 | > loss_text_ce: 0.037453971803188324 (0.036340291582373715) | > loss_mel_ce: 2.7467153072357178 (nan) | > loss: 0.011048290878534317 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13678750891498764) | > loader_time: 0.0046 (0.004710084817106853)  --> TIME: 2025-02-25 02:07:53 -- STEP: 27380/36938 -- GLOBAL_STEP: 581450 | > loss_text_ce: 0.031061429530382156 (0.036339663065203934) | > loss_mel_ce: 3.2476069927215576 (nan) | > loss: 0.01301058940589428 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.13679731482567756) | > loader_time: 0.0047 (0.004709923798538158)  --> TIME: 2025-02-25 02:08:03 -- STEP: 27430/36938 -- GLOBAL_STEP: 581500 | > loss_text_ce: 0.04069538414478302 (0.036339374591748894) | > loss_mel_ce: 3.1060118675231934 (nan) | > loss: 0.012486934661865234 (nan) | > current_lr: 5e-06 | > step_time: 0.1882 (0.13680313032472177) | > loader_time: 0.0058 (0.004710184636882702)  --> TIME: 2025-02-25 02:08:12 -- STEP: 27480/36938 -- GLOBAL_STEP: 581550 | > loss_text_ce: 0.039929479360580444 (0.036340891934553074) | > loss_mel_ce: 2.2722229957580566 (nan) | > loss: 0.009175208397209644 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.1368090758590889) | > loader_time: 0.0046 (0.0047102192170061724)  --> TIME: 2025-02-25 02:08:21 -- STEP: 27530/36938 -- GLOBAL_STEP: 581600 | > loss_text_ce: 0.03648659959435463 (0.036338014534819746) | > loss_mel_ce: 2.968017578125 (nan) | > loss: 0.011922636069357395 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13680621607711488) | > loader_time: 0.0046 (0.004710080058107935)  --> TIME: 2025-02-25 02:08:30 -- STEP: 27580/36938 -- GLOBAL_STEP: 581650 | > loss_text_ce: 0.054404739290475845 (0.03633814024127093) | > loss_mel_ce: 3.822265625 (nan) | > loss: 0.015383613295853138 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13679549611072891) | > loader_time: 0.0044 (0.004709899823159094)  --> TIME: 2025-02-25 02:08:40 -- STEP: 27630/36938 -- GLOBAL_STEP: 581700 | > loss_text_ce: 0.02513887546956539 (0.03633778468762007) | > loss_mel_ce: 2.284632921218872 (nan) | > loss: 0.00916576199233532 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.1368008954085773) | > loader_time: 0.0045 (0.004709933281290032)  --> TIME: 2025-02-25 02:08:49 -- STEP: 27680/36938 -- GLOBAL_STEP: 581750 | > loss_text_ce: 0.04614487662911415 (0.0363365463031297) | > loss_mel_ce: 2.7318453788757324 (nan) | > loss: 0.011023771949112415 (nan) | > current_lr: 5e-06 | > step_time: 0.1116 (0.1368074165947859) | > loader_time: 0.0045 (0.00470994231156536)  --> TIME: 2025-02-25 02:08:59 -- STEP: 27730/36938 -- GLOBAL_STEP: 581800 | > loss_text_ce: 0.02705046534538269 (0.03633896557982302) | > loss_mel_ce: 2.510199546813965 (nan) | > loss: 0.010068452917039394 (nan) | > current_lr: 5e-06 | > step_time: 0.1123 (0.13681020130545352) | > loader_time: 0.0045 (0.0047098180338918175)  --> TIME: 2025-02-25 02:09:08 -- STEP: 27780/36938 -- GLOBAL_STEP: 581850 | > loss_text_ce: 0.02624627575278282 (0.03633926213743085) | > loss_mel_ce: 2.3529820442199707 (nan) | > loss: 0.009441383183002472 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13681674798329638) | > loader_time: 0.0048 (0.004709884389172927)  --> TIME: 2025-02-25 02:09:17 -- STEP: 27830/36938 -- GLOBAL_STEP: 581900 | > loss_text_ce: 0.03168773651123047 (0.03633898603102435) | > loss_mel_ce: 0.09521484375 (nan) | > loss: 0.000503581715747714 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13681017960141598) | > loader_time: 0.0052 (0.004709932498267138)  --> TIME: 2025-02-25 02:09:27 -- STEP: 27880/36938 -- GLOBAL_STEP: 581950 | > loss_text_ce: 0.040188729763031006 (0.03633752444039469) | > loss_mel_ce: 3.1297237873077393 (nan) | > loss: 0.01257901918143034 (nan) | > current_lr: 5e-06 | > step_time: 0.1858 (0.13680673308317726) | > loader_time: 0.005 (0.004710183552382202)  --> TIME: 2025-02-25 02:09:36 -- STEP: 27930/36938 -- GLOBAL_STEP: 582000 | > loss_text_ce: 0.0428617000579834 (0.03633679196313871) | > loss_mel_ce: 3.2508347034454346 (nan) | > loss: 0.013070224784314632 (nan) | > current_lr: 5e-06 | > step_time: 0.1502 (0.13680449101474992) | > loader_time: 0.0047 (0.004710112162657341)  --> TIME: 2025-02-25 02:09:45 -- STEP: 27980/36938 -- GLOBAL_STEP: 582050 | > loss_text_ce: 0.03060755878686905 (0.03633665887727459) | > loss_mel_ce: 0.3593466579914093 (nan) | > loss: 0.0015474373940378428 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13680784634132703) | > loader_time: 0.0045 (0.0047100134284433735)  --> TIME: 2025-02-25 02:09:54 -- STEP: 28030/36938 -- GLOBAL_STEP: 582100 | > loss_text_ce: 0.02525620348751545 (0.03633534091618141) | > loss_mel_ce: 2.243572235107422 (nan) | > loss: 0.009003288112580776 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.1368026453896668) | > loader_time: 0.0045 (0.004709935451974417)  --> TIME: 2025-02-25 02:10:04 -- STEP: 28080/36938 -- GLOBAL_STEP: 582150 | > loss_text_ce: 0.03408679738640785 (0.036333891931964984) | > loss_mel_ce: 2.6506128311157227 (nan) | > loss: 0.010653570294380188 (nan) | > current_lr: 5e-06 | > step_time: 0.1623 (0.1368011223454757) | > loader_time: 0.0046 (0.0047097447930578296)  --> TIME: 2025-02-25 02:10:13 -- STEP: 28130/36938 -- GLOBAL_STEP: 582200 | > loss_text_ce: 0.04285106435418129 (0.03633479553510202) | > loss_mel_ce: 3.0799036026000977 (nan) | > loss: 0.012391883879899979 (nan) | > current_lr: 5e-06 | > step_time: 0.176 (0.1367973705478885) | > loader_time: 0.0047 (0.004709612530743281)  --> TIME: 2025-02-25 02:10:22 -- STEP: 28180/36938 -- GLOBAL_STEP: 582250 | > loss_text_ce: 0.034800827503204346 (0.03633512354547098) | > loss_mel_ce: 3.0575740337371826 (nan) | > loss: 0.012271328829228878 (nan) | > current_lr: 5e-06 | > step_time: 0.1469 (0.13679271262109957) | > loader_time: 0.0068 (0.004709572323338251)  --> TIME: 2025-02-25 02:10:31 -- STEP: 28230/36938 -- GLOBAL_STEP: 582300 | > loss_text_ce: 0.0382041297852993 (0.0363334520120177) | > loss_mel_ce: 2.9366862773895264 (nan) | > loss: 0.011805121786892414 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13678852445134151) | > loader_time: 0.0045 (0.0047095622908254545)  --> TIME: 2025-02-25 02:10:41 -- STEP: 28280/36938 -- GLOBAL_STEP: 582350 | > loss_text_ce: 0.033080413937568665 (0.0363330162641074) | > loss_mel_ce: 2.579315662384033 (nan) | > loss: 0.010366651229560375 (nan) | > current_lr: 5e-06 | > step_time: 0.1907 (0.1367942989378363) | > loader_time: 0.0048 (0.004709588334786384)  --> TIME: 2025-02-25 02:10:50 -- STEP: 28330/36938 -- GLOBAL_STEP: 582400 | > loss_text_ce: 0.04001890495419502 (0.036336569230085616) | > loss_mel_ce: 2.9847147464752197 (nan) | > loss: 0.012002911418676376 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13679140382464383) | > loader_time: 0.0049 (0.004709720586325023)  --> TIME: 2025-02-25 02:10:59 -- STEP: 28380/36938 -- GLOBAL_STEP: 582450 | > loss_text_ce: 0.02974594011902809 (0.03633740508966299) | > loss_mel_ce: 2.604041576385498 (nan) | > loss: 0.010451538488268852 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13678315876234193) | > loader_time: 0.0075 (0.004709866384623214)  --> TIME: 2025-02-25 02:11:08 -- STEP: 28430/36938 -- GLOBAL_STEP: 582500 | > loss_text_ce: 0.03292316198348999 (0.0363354572815662) | > loss_mel_ce: 2.788398504257202 (nan) | > loss: 0.011195722036063671 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13678350574400766) | > loader_time: 0.0045 (0.00471013305139429)  --> TIME: 2025-02-25 02:11:18 -- STEP: 28480/36938 -- GLOBAL_STEP: 582550 | > loss_text_ce: 0.03996893763542175 (0.03633708627368029) | > loss_mel_ce: 3.26894474029541 (nan) | > loss: 0.013130610808730125 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13678619075524615) | > loader_time: 0.0045 (0.004710085673278685)  --> TIME: 2025-02-25 02:11:27 -- STEP: 28530/36938 -- GLOBAL_STEP: 582600 | > loss_text_ce: 0.03852788731455803 (0.036333426372649115) | > loss_mel_ce: 1.784468650817871 (nan) | > loss: 0.0072341132909059525 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13678267173419392) | > loader_time: 0.0047 (0.004709976871850202)  --> TIME: 2025-02-25 02:11:36 -- STEP: 28580/36938 -- GLOBAL_STEP: 582650 | > loss_text_ce: 0.04247898980975151 (0.036334360758053755) | > loss_mel_ce: 3.3445208072662354 (nan) | > loss: 0.013440476730465889 (nan) | > current_lr: 5e-06 | > step_time: 0.189 (0.13678337549145453) | > loader_time: 0.0048 (0.004709917703152084)  --> TIME: 2025-02-25 02:11:46 -- STEP: 28630/36938 -- GLOBAL_STEP: 582700 | > loss_text_ce: 0.03855118155479431 (0.036332934620063734) | > loss_mel_ce: 3.29620361328125 (nan) | > loss: 0.013233154080808163 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.1367805801985183) | > loader_time: 0.0046 (0.004709756878177692)  --> TIME: 2025-02-25 02:11:55 -- STEP: 28680/36938 -- GLOBAL_STEP: 582750 | > loss_text_ce: 0.043111104518175125 (0.03632930195932575) | > loss_mel_ce: 3.530470609664917 (nan) | > loss: 0.014180880971252918 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13678239789301622) | > loader_time: 0.0045 (0.004709593396805368)  --> TIME: 2025-02-25 02:12:04 -- STEP: 28730/36938 -- GLOBAL_STEP: 582800 | > loss_text_ce: 0.04000255838036537 (0.03632724342616175) | > loss_mel_ce: 2.7181038856506348 (nan) | > loss: 0.010944867506623268 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.1367747424331799) | > loader_time: 0.0046 (0.0047094699027768645)  --> TIME: 2025-02-25 02:12:13 -- STEP: 28780/36938 -- GLOBAL_STEP: 582850 | > loss_text_ce: 0.038966529071331024 (0.03632595076975151) | > loss_mel_ce: 3.1566877365112305 (nan) | > loss: 0.01268116757273674 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13677858380992924) | > loader_time: 0.0048 (0.004709293744562447)  --> TIME: 2025-02-25 02:12:22 -- STEP: 28830/36938 -- GLOBAL_STEP: 582900 | > loss_text_ce: 0.031358953565359116 (0.03632373231641769) | > loss_mel_ce: 3.173978567123413 (nan) | > loss: 0.012719593942165375 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.1367714688548841) | > loader_time: 0.0045 (0.0047090643857280925)  --> TIME: 2025-02-25 02:12:32 -- STEP: 28880/36938 -- GLOBAL_STEP: 582950 | > loss_text_ce: 0.026075832545757294 (0.03632174654534194) | > loss_mel_ce: 0.006744384765625 (nan) | > loss: 0.00013023897190578282 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.1367646941435295) | > loader_time: 0.0046 (0.0047088722030212155)  --> TIME: 2025-02-25 02:12:41 -- STEP: 28930/36938 -- GLOBAL_STEP: 583000 | > loss_text_ce: 0.04298899695277214 (0.036318637852869526) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.1367713381829569) | > loader_time: 0.0045 (0.004708708836634315)  --> TIME: 2025-02-25 02:12:50 -- STEP: 28980/36938 -- GLOBAL_STEP: 583050 | > loss_text_ce: 0.03824600949883461 (0.03631625745353573) | > loss_mel_ce: 3.55007266998291 (nan) | > loss: 0.014239360578358173 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.13676994498471964) | > loader_time: 0.0047 (0.004708506717115914)  --> TIME: 2025-02-25 02:13:00 -- STEP: 29030/36938 -- GLOBAL_STEP: 583100 | > loss_text_ce: 0.0440356582403183 (0.036317851362044726) | > loss_mel_ce: 2.267064094543457 (nan) | > loss: 0.009171031415462494 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1367663095541911) | > loader_time: 0.0045 (0.0047082892623886066)  --> TIME: 2025-02-25 02:13:09 -- STEP: 29080/36938 -- GLOBAL_STEP: 583150 | > loss_text_ce: 0.049181681126356125 (0.03631983454799737) | > loss_mel_ce: 0.15128707885742188 (nan) | > loss: 0.0007955110049806535 (nan) | > current_lr: 5e-06 | > step_time: 0.1626 (0.13676481568337745) | > loader_time: 0.0048 (0.004708055961246878)  --> TIME: 2025-02-25 02:13:18 -- STEP: 29130/36938 -- GLOBAL_STEP: 583200 | > loss_text_ce: 0.04188144579529762 (0.036318357922667445) | > loss_mel_ce: 2.322603225708008 (nan) | > loss: 0.009382876567542553 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.1367691624667931) | > loader_time: 0.0045 (0.004707983257923638)  --> TIME: 2025-02-25 02:13:28 -- STEP: 29180/36938 -- GLOBAL_STEP: 583250 | > loss_text_ce: 0.034022007137537 (0.036317197986252193) | > loss_mel_ce: 2.9901483058929443 (nan) | > loss: 0.012000677175819874 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13677034577414734) | > loader_time: 0.0052 (0.004707992983812218)  --> TIME: 2025-02-25 02:13:37 -- STEP: 29230/36938 -- GLOBAL_STEP: 583300 | > loss_text_ce: 0.03477628529071808 (0.0363164651832118) | > loss_mel_ce: 3.401655673980713 (nan) | > loss: 0.0136366356164217 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13676791209045625) | > loader_time: 0.0048 (0.004707914209512711)  --> TIME: 2025-02-25 02:13:47 -- STEP: 29280/36938 -- GLOBAL_STEP: 583350 | > loss_text_ce: 0.039342187345027924 (0.03631832868072626) | > loss_mel_ce: 2.740168571472168 (nan) | > loss: 0.011029805056750774 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.13677386786442577) | > loader_time: 0.0045 (0.0047078011547281905)  --> TIME: 2025-02-25 02:13:56 -- STEP: 29330/36938 -- GLOBAL_STEP: 583400 | > loss_text_ce: 0.03645705804228783 (0.036320366487548356) | > loss_mel_ce: 2.2609951496124268 (nan) | > loss: 0.00911687407642603 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13676979875873418) | > loader_time: 0.0045 (0.004707742468960455)  --> TIME: 2025-02-25 02:14:05 -- STEP: 29380/36938 -- GLOBAL_STEP: 583450 | > loss_text_ce: 0.027957700192928314 (0.036321086391592254) | > loss_mel_ce: 0.8768833875656128 (nan) | > loss: 0.0035906394477933645 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13677261012033784) | > loader_time: 0.0052 (0.004707864322493608)  --> TIME: 2025-02-25 02:14:15 -- STEP: 29430/36938 -- GLOBAL_STEP: 583500 | > loss_text_ce: 0.03557277470827103 (0.036321197920379134) | > loss_mel_ce: 3.439865827560425 (nan) | > loss: 0.013791424222290516 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13677755234154748) | > loader_time: 0.0046 (0.00470781891758334)  --> TIME: 2025-02-25 02:14:24 -- STEP: 29480/36938 -- GLOBAL_STEP: 583550 | > loss_text_ce: 0.03430675342679024 (0.03631827395767589) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13677089862131145) | > loader_time: 0.0045 (0.004707771661000188)  --> TIME: 2025-02-25 02:14:33 -- STEP: 29530/36938 -- GLOBAL_STEP: 583600 | > loss_text_ce: 0.032766349613666534 (0.03631833022507012) | > loss_mel_ce: 2.982259511947632 (nan) | > loss: 0.011964389123022556 (nan) | > current_lr: 5e-06 | > step_time: 0.1918 (0.1367719478862304) | > loader_time: 0.0048 (0.0047077607834012615)  --> TIME: 2025-02-25 02:14:43 -- STEP: 29580/36938 -- GLOBAL_STEP: 583650 | > loss_text_ce: 0.03784783557057381 (0.036317414758245806) | > loss_mel_ce: 2.4657437801361084 (nan) | > loss: 0.009934887290000916 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.136775768399641) | > loader_time: 0.0046 (0.004707566228083823)  --> TIME: 2025-02-25 02:14:52 -- STEP: 29630/36938 -- GLOBAL_STEP: 583700 | > loss_text_ce: 0.042160503566265106 (0.036317305412794675) | > loss_mel_ce: 1.00177001953125 (nan) | > loss: 0.004142581485211849 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.1367804786863653) | > loader_time: 0.0053 (0.0047074609377317864)  --> TIME: 2025-02-25 02:15:02 -- STEP: 29680/36938 -- GLOBAL_STEP: 583750 | > loss_text_ce: 0.042859211564064026 (0.03631552254829444) | > loss_mel_ce: 3.137651205062866 (nan) | > loss: 0.012621073983609676 (nan) | > current_lr: 5e-06 | > step_time: 0.1695 (0.13678218762508249) | > loader_time: 0.005 (0.004707386398893849)  --> TIME: 2025-02-25 02:15:11 -- STEP: 29730/36938 -- GLOBAL_STEP: 583800 | > loss_text_ce: 0.04017270356416702 (0.03631414086929132) | > loss_mel_ce: 3.28125 (nan) | > loss: 0.01318025030195713 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13677912040707835) | > loader_time: 0.0045 (0.004707428609486872)  --> TIME: 2025-02-25 02:15:20 -- STEP: 29780/36938 -- GLOBAL_STEP: 583850 | > loss_text_ce: 0.03563747555017471 (0.036313348855726894) | > loss_mel_ce: 2.9339652061462402 (nan) | > loss: 0.011784138157963753 (nan) | > current_lr: 5e-06 | > step_time: 0.1446 (0.13678131630310156) | > loader_time: 0.0053 (0.0047075006047538155)  --> TIME: 2025-02-25 02:15:29 -- STEP: 29830/36938 -- GLOBAL_STEP: 583900 | > loss_text_ce: 0.031139535829424858 (0.036313859163244866) | > loss_mel_ce: 2.7133588790893555 (nan) | > loss: 0.010890867561101913 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13677748171147366) | > loader_time: 0.0046 (0.004707710510366487)  --> TIME: 2025-02-25 02:15:39 -- STEP: 29880/36938 -- GLOBAL_STEP: 583950 | > loss_text_ce: 0.04485516995191574 (0.03631587638921775) | > loss_mel_ce: 0.175935298204422 (nan) | > loss: 0.0008761527133174241 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13678025455679044) | > loader_time: 0.0045 (0.004707609936894229)  --> TIME: 2025-02-25 02:15:48 -- STEP: 29930/36938 -- GLOBAL_STEP: 584000 | > loss_text_ce: 0.03306039422750473 (0.036315509307671454) | > loss_mel_ce: 2.8184726238250732 (nan) | > loss: 0.011315607465803623 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.136784783940708) | > loader_time: 0.0046 (0.004707574326578652)  --> TIME: 2025-02-25 02:15:57 -- STEP: 29980/36938 -- GLOBAL_STEP: 584050 | > loss_text_ce: 0.03868178278207779 (0.036312870843661485) | > loss_mel_ce: 3.110025644302368 (nan) | > loss: 0.012494871392846107 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13678618966936595) | > loader_time: 0.0046 (0.004707579941969109)  --> TIME: 2025-02-25 02:16:07 -- STEP: 30030/36938 -- GLOBAL_STEP: 584100 | > loss_text_ce: 0.03898777440190315 (0.03631275518112873) | > loss_mel_ce: 2.6065890789031982 (nan) | > loss: 0.010498321615159512 (nan) | > current_lr: 5e-06 | > step_time: 0.1566 (0.1367855971628843) | > loader_time: 0.0049 (0.004707615978115253)  --> TIME: 2025-02-25 02:16:16 -- STEP: 30080/36938 -- GLOBAL_STEP: 584150 | > loss_text_ce: 0.0338146835565567 (0.03631246775234523) | > loss_mel_ce: 3.3155152797698975 (nan) | > loss: 0.013290992937982082 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13678144577969822) | > loader_time: 0.0047 (0.004707541039332476)  --> TIME: 2025-02-25 02:16:25 -- STEP: 30130/36938 -- GLOBAL_STEP: 584200 | > loss_text_ce: 0.028048917651176453 (0.03631393811657742) | > loss_mel_ce: 3.159726858139038 (nan) | > loss: 0.012649904936552048 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.1367846627271805) | > loader_time: 0.0048 (0.00470755305988153)  --> TIME: 2025-02-25 02:16:35 -- STEP: 30180/36938 -- GLOBAL_STEP: 584250 | > loss_text_ce: 0.03296364098787308 (0.0363127996044318) | > loss_mel_ce: 2.519536018371582 (nan) | > loss: 0.010128968395292759 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.13679089584912985) | > loader_time: 0.0046 (0.004707408622845391)  --> TIME: 2025-02-25 02:16:44 -- STEP: 30230/36938 -- GLOBAL_STEP: 584300 | > loss_text_ce: 0.03712178021669388 (0.0363130350225659) | > loss_mel_ce: 3.019120216369629 (nan) | > loss: 0.012127945199608803 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13679472803872275) | > loader_time: 0.0046 (0.004707382153565669)  --> TIME: 2025-02-25 02:16:54 -- STEP: 30280/36938 -- GLOBAL_STEP: 584350 | > loss_text_ce: 0.02802950143814087 (0.036312638111861514) | > loss_mel_ce: 2.269526243209839 (nan) | > loss: 0.009117284789681435 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13679080714165226) | > loader_time: 0.0045 (0.004707297529223145)  --> TIME: 2025-02-25 02:17:02 -- STEP: 30330/36938 -- GLOBAL_STEP: 584400 | > loss_text_ce: 0.03054543398320675 (0.0363102432097496) | > loss_mel_ce: 3.2974460124969482 (nan) | > loss: 0.013206316158175468 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13677809330405324) | > loader_time: 0.0048 (0.004707256622767281)  --> TIME: 2025-02-25 02:17:12 -- STEP: 30380/36938 -- GLOBAL_STEP: 584450 | > loss_text_ce: 0.03168226033449173 (0.03630956641424115) | > loss_mel_ce: 2.9222075939178467 (nan) | > loss: 0.011721786111593246 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.13677402363551697) | > loader_time: 0.0048 (0.0047074038704576875)  --> TIME: 2025-02-25 02:17:21 -- STEP: 30430/36938 -- GLOBAL_STEP: 584500 | > loss_text_ce: 0.038583509624004364 (0.03630944440946848) | > loss_mel_ce: 3.425955295562744 (nan) | > loss: 0.01374817080795765 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13677272433871004) | > loader_time: 0.0047 (0.004707389155227149)  --> TIME: 2025-02-25 02:17:30 -- STEP: 30480/36938 -- GLOBAL_STEP: 584550 | > loss_text_ce: 0.027164163067936897 (0.03630905225943264) | > loss_mel_ce: 2.469421863555908 (nan) | > loss: 0.009907088242471218 (nan) | > current_lr: 5e-06 | > step_time: 0.1513 (0.1367694559607286) | > loader_time: 0.0046 (0.004707243373700685)  --> TIME: 2025-02-25 02:17:39 -- STEP: 30530/36938 -- GLOBAL_STEP: 584600 | > loss_text_ce: 0.044594407081604004 (0.03630932970387356) | > loss_mel_ce: 3.299509048461914 (nan) | > loss: 0.013270252384245396 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13676655381537178) | > loader_time: 0.0045 (0.004707235131074792)  --> TIME: 2025-02-25 02:17:49 -- STEP: 30580/36938 -- GLOBAL_STEP: 584650 | > loss_text_ce: 0.03170160576701164 (0.036312792004720096) | > loss_mel_ce: 2.562563180923462 (nan) | > loss: 0.010294701904058456 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13676921483190102) | > loader_time: 0.0048 (0.004707180221849565)  --> TIME: 2025-02-25 02:17:58 -- STEP: 30630/36938 -- GLOBAL_STEP: 584700 | > loss_text_ce: 0.04687708243727684 (0.03631188850889388) | > loss_mel_ce: 2.8137545585632324 (nan) | > loss: 0.01135171391069889 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13676837925190097) | > loader_time: 0.0047 (0.0047070118713877515)  --> TIME: 2025-02-25 02:18:07 -- STEP: 30680/36938 -- GLOBAL_STEP: 584750 | > loss_text_ce: 0.04286394640803337 (0.03631222550371702) | > loss_mel_ce: 2.9954700469970703 (nan) | > loss: 0.012056881561875343 (nan) | > current_lr: 5e-06 | > step_time: 0.1513 (0.13676623574602487) | > loader_time: 0.0046 (0.004706975992531056)  --> TIME: 2025-02-25 02:18:17 -- STEP: 30730/36938 -- GLOBAL_STEP: 584800 | > loss_text_ce: 0.03657016530632973 (0.03631232563073456) | > loss_mel_ce: 2.8668487071990967 (nan) | > loss: 0.011521503329277039 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.13676466202448764) | > loader_time: 0.0046 (0.004707008862084266)  --> TIME: 2025-02-25 02:18:26 -- STEP: 30780/36938 -- GLOBAL_STEP: 584850 | > loss_text_ce: 0.041418708860874176 (0.03631172163634058) | > loss_mel_ce: 3.0566370487213135 (nan) | > loss: 0.012293873354792595 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13677561558865284) | > loader_time: 0.0046 (0.0047070765278415735)  --> TIME: 2025-02-25 02:18:36 -- STEP: 30830/36938 -- GLOBAL_STEP: 584900 | > loss_text_ce: 0.039984330534935 (0.03631074243891849) | > loss_mel_ce: 1.893379807472229 (nan) | > loss: 0.0076720803044736385 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13677655568650207) | > loader_time: 0.0045 (0.004706914263269677)  --> TIME: 2025-02-25 02:18:45 -- STEP: 30880/36938 -- GLOBAL_STEP: 584950 | > loss_text_ce: 0.025249628350138664 (0.03631324719337842) | > loss_mel_ce: 3.2662761211395264 (nan) | > loss: 0.013061610981822014 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13677742372368212) | > loader_time: 0.0045 (0.00470669481884017)  --> TIME: 2025-02-25 02:18:54 -- STEP: 30930/36938 -- GLOBAL_STEP: 585000 | > loss_text_ce: 0.05078252777457237 (0.03631348250574715) | > loss_mel_ce: 1.5954458713531494 (nan) | > loss: 0.006532652769237757 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1367804935361582) | > loader_time: 0.0045 (0.004706582173613609)  --> TIME: 2025-02-25 02:19:04 -- STEP: 30980/36938 -- GLOBAL_STEP: 585050 | > loss_text_ce: 0.04037027433514595 (0.0363136422874801) | > loss_mel_ce: 2.936981201171875 (nan) | > loss: 0.011814887635409832 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13678349334243892) | > loader_time: 0.0045 (0.004706639309249637)  --> TIME: 2025-02-25 02:19:13 -- STEP: 31030/36938 -- GLOBAL_STEP: 585100 | > loss_text_ce: 0.033357247710227966 (0.036312894564712574) | > loss_mel_ce: 3.218052625656128 (nan) | > loss: 0.01290242001414299 (nan) | > current_lr: 5e-06 | > step_time: 0.162 (0.13679161816307092) | > loader_time: 0.0048 (0.004706602583690691)  --> TIME: 2025-02-25 02:19:23 -- STEP: 31080/36938 -- GLOBAL_STEP: 585150 | > loss_text_ce: 0.031354840844869614 (0.03631179142108677) | > loss_mel_ce: 1.9480295181274414 (nan) | > loss: 0.007854700088500977 (nan) | > current_lr: 5e-06 | > step_time: 0.1618 (0.1367898129358248) | > loader_time: 0.0049 (0.004706618408438826)  --> TIME: 2025-02-25 02:19:32 -- STEP: 31130/36938 -- GLOBAL_STEP: 585200 | > loss_text_ce: 0.03237217292189598 (0.036310004552516886) | > loss_mel_ce: 3.003141164779663 (nan) | > loss: 0.012045688927173615 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13680105367121886) | > loader_time: 0.0045 (0.00470674548009723)  --> TIME: 2025-02-25 02:19:42 -- STEP: 31180/36938 -- GLOBAL_STEP: 585250 | > loss_text_ce: 0.03710915148258209 (0.03630805666877442) | > loss_mel_ce: 3.28633975982666 (nan) | > loss: 0.013188290409743786 (nan) | > current_lr: 5e-06 | > step_time: 0.165 (0.13680456359849755) | > loader_time: 0.0047 (0.004706635790195472)  --> TIME: 2025-02-25 02:19:51 -- STEP: 31230/36938 -- GLOBAL_STEP: 585300 | > loss_text_ce: 0.021800795570015907 (0.03630463413660371) | > loss_mel_ce: 0.8024706840515137 (nan) | > loss: 0.0032709187362343073 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13679796116419957) | > loader_time: 0.0045 (0.004706587220398736)  --> TIME: 2025-02-25 02:20:00 -- STEP: 31280/36938 -- GLOBAL_STEP: 585350 | > loss_text_ce: 0.03136143460869789 (0.03630294532531776) | > loss_mel_ce: 2.8000762462615967 (nan) | > loss: 0.011235863901674747 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13679065721114203) | > loader_time: 0.0045 (0.00470660241973374)  --> TIME: 2025-02-25 02:20:09 -- STEP: 31330/36938 -- GLOBAL_STEP: 585400 | > loss_text_ce: 0.034870222210884094 (0.03630141662163229) | > loss_mel_ce: 2.972498893737793 (nan) | > loss: 0.011934004724025726 (nan) | > current_lr: 5e-06 | > step_time: 0.1572 (0.13678409669713482) | > loader_time: 0.0067 (0.004706731346366626)  --> TIME: 2025-02-25 02:20:18 -- STEP: 31380/36938 -- GLOBAL_STEP: 585450 | > loss_text_ce: 0.03719121590256691 (0.036302223474848565) | > loss_mel_ce: 2.753878593444824 (nan) | > loss: 0.011075674556195736 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.1367795956294343) | > loader_time: 0.0045 (0.004706650444320129)  --> TIME: 2025-02-25 02:20:28 -- STEP: 31430/36938 -- GLOBAL_STEP: 585500 | > loss_text_ce: 0.036318350583314896 (0.036301445874651866) | > loss_mel_ce: 2.8658649921417236 (nan) | > loss: 0.011516600847244263 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.1367859614946029) | > loader_time: 0.0046 (0.004706517488684168)  --> TIME: 2025-02-25 02:20:37 -- STEP: 31480/36938 -- GLOBAL_STEP: 585550 | > loss_text_ce: 0.05177401751279831 (0.036302534248052624) | > loss_mel_ce: 3.6928067207336426 (nan) | > loss: 0.014859448187053204 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13678742429807125) | > loader_time: 0.0047 (0.0047063632493098305)  --> TIME: 2025-02-25 02:20:46 -- STEP: 31530/36938 -- GLOBAL_STEP: 585600 | > loss_text_ce: 0.03750813379883766 (0.0363015767492299) | > loss_mel_ce: 3.167468309402466 (nan) | > loss: 0.012718161568045616 (nan) | > current_lr: 5e-06 | > step_time: 0.1868 (0.1367810627032635) | > loader_time: 0.0048 (0.004706224561909707)  --> TIME: 2025-02-25 02:20:56 -- STEP: 31580/36938 -- GLOBAL_STEP: 585650 | > loss_text_ce: 0.02751876786351204 (0.03629872319539219) | > loss_mel_ce: 3.2170732021331787 (nan) | > loss: 0.012875366024672985 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13677811724540104) | > loader_time: 0.0046 (0.004706203061768839)  --> TIME: 2025-02-25 02:21:05 -- STEP: 31630/36938 -- GLOBAL_STEP: 585700 | > loss_text_ce: 0.029922446236014366 (0.03629631682237533) | > loss_mel_ce: 3.228879690170288 (nan) | > loss: 0.012931755743920803 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13677087891452272) | > loader_time: 0.0045 (0.004706232825902499)  --> TIME: 2025-02-25 02:21:14 -- STEP: 31680/36938 -- GLOBAL_STEP: 585750 | > loss_text_ce: 0.041351597756147385 (0.036295812043203736) | > loss_mel_ce: 0.6601890325546265 (nan) | > loss: 0.002783891512081027 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13677245142935435) | > loader_time: 0.0045 (0.004706168423096382)  --> TIME: 2025-02-25 02:21:23 -- STEP: 31730/36938 -- GLOBAL_STEP: 585800 | > loss_text_ce: 0.03268803656101227 (0.03629769723377518) | > loss_mel_ce: 2.6734375953674316 (nan) | > loss: 0.010738594457507133 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13677487056262033) | > loader_time: 0.0045 (0.004706025176194093)  --> TIME: 2025-02-25 02:21:33 -- STEP: 31780/36938 -- GLOBAL_STEP: 585850 | > loss_text_ce: 0.04724448174238205 (0.03629587552404111) | > loss_mel_ce: 3.3710577487945557 (nan) | > loss: 0.013564692810177803 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.13677225136921875) | > loader_time: 0.0046 (0.00470585479460253)  --> TIME: 2025-02-25 02:21:42 -- STEP: 31830/36938 -- GLOBAL_STEP: 585900 | > loss_text_ce: 0.03669695928692818 (0.036295356185526824) | > loss_mel_ce: 2.6235902309417725 (nan) | > loss: 0.010556695982813835 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.1367750049274065) | > loader_time: 0.0046 (0.004705725358860587)  --> TIME: 2025-02-25 02:21:51 -- STEP: 31880/36938 -- GLOBAL_STEP: 585950 | > loss_text_ce: 0.03476693481206894 (0.03629342230572038) | > loss_mel_ce: 2.093777656555176 (nan) | > loss: 0.008446605876088142 (nan) | > current_lr: 5e-06 | > step_time: 0.1271 (0.13677129329671756) | > loader_time: 0.0045 (0.0047056695747854545)  --> TIME: 2025-02-25 02:22:00 -- STEP: 31930/36938 -- GLOBAL_STEP: 586000 | > loss_text_ce: 0.040202073752880096 (0.03629221196706097) | > loss_mel_ce: 3.493975877761841 (nan) | > loss: 0.014024516567587852 (nan) | > current_lr: 5e-06 | > step_time: 0.1807 (0.1367583728381687) | > loader_time: 0.005 (0.004705440098314241)  --> TIME: 2025-02-25 02:22:10 -- STEP: 31980/36938 -- GLOBAL_STEP: 586050 | > loss_text_ce: 0.03691099211573601 (0.036293159315672376) | > loss_mel_ce: 3.4358696937561035 (nan) | > loss: 0.013780876994132996 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13676901949875878) | > loader_time: 0.0045 (0.00470532520328786)  --> TIME: 2025-02-25 02:22:19 -- STEP: 32030/36938 -- GLOBAL_STEP: 586100 | > loss_text_ce: 0.03112238086760044 (0.036293077667726686) | > loss_mel_ce: 2.137122392654419 (nan) | > loss: 0.008604146540164948 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.1367612311217021) | > loader_time: 0.0046 (0.004705115143224858)  --> TIME: 2025-02-25 02:22:28 -- STEP: 32080/36938 -- GLOBAL_STEP: 586150 | > loss_text_ce: 0.026901111006736755 (0.036294587342123076) | > loss_mel_ce: 2.51647686958313 (nan) | > loss: 0.010092769749462605 (nan) | > current_lr: 5e-06 | > step_time: 0.162 (0.13676499751886456) | > loader_time: 0.0046 (0.004704950694134153)  --> TIME: 2025-02-25 02:22:38 -- STEP: 32130/36938 -- GLOBAL_STEP: 586200 | > loss_text_ce: 0.02823506109416485 (0.036295572102824594) | > loss_mel_ce: 2.132993221282959 (nan) | > loss: 0.008576302789151669 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.13676348406319383) | > loader_time: 0.0046 (0.004704911056575586)  --> TIME: 2025-02-25 02:22:47 -- STEP: 32180/36938 -- GLOBAL_STEP: 586250 | > loss_text_ce: 0.03273905813694 (0.03629584243067293) | > loss_mel_ce: 3.0050840377807617 (nan) | > loss: 0.01205485500395298 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13675801392443146) | > loader_time: 0.0045 (0.004704721748940064)  --> TIME: 2025-02-25 02:22:56 -- STEP: 32230/36938 -- GLOBAL_STEP: 586300 | > loss_text_ce: 0.03632133826613426 (0.03629723703815228) | > loss_mel_ce: 3.388551950454712 (nan) | > loss: 0.013590767979621887 (nan) | > current_lr: 5e-06 | > step_time: 0.1642 (0.13675840605203984) | > loader_time: 0.0047 (0.004704508654197666)  --> TIME: 2025-02-25 02:23:05 -- STEP: 32280/36938 -- GLOBAL_STEP: 586350 | > loss_text_ce: 0.042195599526166916 (0.03629882463128237) | > loss_mel_ce: 3.238603115081787 (nan) | > loss: 0.013019043020904064 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13675567981921982) | > loader_time: 0.0045 (0.004704561072506764)  --> TIME: 2025-02-25 02:23:15 -- STEP: 32330/36938 -- GLOBAL_STEP: 586400 | > loss_text_ce: 0.03903963416814804 (0.036296971588278236) | > loss_mel_ce: 2.573399782180786 (nan) | > loss: 0.010366823524236679 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.13675887873030365) | > loader_time: 0.0045 (0.0047044322765762555)  --> TIME: 2025-02-25 02:23:24 -- STEP: 32380/36938 -- GLOBAL_STEP: 586450 | > loss_text_ce: 0.03543080389499664 (0.036297041803731805) | > loss_mel_ce: 2.467641830444336 (nan) | > loss: 0.009932829067111015 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.1367566760341786) | > loader_time: 0.0055 (0.004704442688386522)  --> TIME: 2025-02-25 02:23:33 -- STEP: 32430/36938 -- GLOBAL_STEP: 586500 | > loss_text_ce: 0.03096599318087101 (0.03629653990062507) | > loss_mel_ce: 2.657583475112915 (nan) | > loss: 0.01066884770989418 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13674973121146605) | > loader_time: 0.0046 (0.004704479041968572)  --> TIME: 2025-02-25 02:23:42 -- STEP: 32480/36938 -- GLOBAL_STEP: 586550 | > loss_text_ce: 0.021560074761509895 (0.03629657868866509) | > loss_mel_ce: 1.5739439725875854 (nan) | > loss: 0.006331365555524826 (nan) | > current_lr: 5e-06 | > step_time: 0.1077 (0.13674187670493881) | > loader_time: 0.0045 (0.004704402071501863)  --> TIME: 2025-02-25 02:23:51 -- STEP: 32530/36938 -- GLOBAL_STEP: 586600 | > loss_text_ce: 0.03412420675158501 (0.036295595642321196) | > loss_mel_ce: 1.582599401473999 (nan) | > loss: 0.006415570620447397 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.1367339514821184) | > loader_time: 0.0045 (0.00470447646556475)  --> TIME: 2025-02-25 02:24:00 -- STEP: 32580/36938 -- GLOBAL_STEP: 586650 | > loss_text_ce: 0.034799933433532715 (0.03629462636639619) | > loss_mel_ce: 2.4652578830718994 (nan) | > loss: 0.00992086436599493 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13673393495101815) | > loader_time: 0.0054 (0.004704727205928706)  --> TIME: 2025-02-25 02:24:10 -- STEP: 32630/36938 -- GLOBAL_STEP: 586700 | > loss_text_ce: 0.034482479095458984 (0.036295449717893315) | > loss_mel_ce: 2.9263761043548584 (nan) | > loss: 0.011749439872801304 (nan) | > current_lr: 5e-06 | > step_time: 0.1814 (0.1367362210840446) | > loader_time: 0.0049 (0.00470477840559201)  --> TIME: 2025-02-25 02:24:19 -- STEP: 32680/36938 -- GLOBAL_STEP: 586750 | > loss_text_ce: 0.03352801129221916 (0.036295011546206235) | > loss_mel_ce: 2.9619412422180176 (nan) | > loss: 0.011886783875524998 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.1367297004238038) | > loader_time: 0.0046 (0.004704687083219853)  --> TIME: 2025-02-25 02:24:28 -- STEP: 32730/36938 -- GLOBAL_STEP: 586800 | > loss_text_ce: 0.039899781346321106 (0.03629463999496589) | > loss_mel_ce: 2.8334407806396484 (nan) | > loss: 0.011402145959436893 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13672745562896288) | > loader_time: 0.0045 (0.004704541042606478)  --> TIME: 2025-02-25 02:24:38 -- STEP: 32780/36938 -- GLOBAL_STEP: 586850 | > loss_text_ce: 0.03397415950894356 (0.036292889623239566) | > loss_mel_ce: 2.8644073009490967 (nan) | > loss: 0.011501514352858067 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13672601610803917) | > loader_time: 0.0047 (0.004704569279238432)  --> TIME: 2025-02-25 02:24:47 -- STEP: 32830/36938 -- GLOBAL_STEP: 586900 | > loss_text_ce: 0.030948268249630928 (0.03629387070989657) | > loss_mel_ce: 2.1029324531555176 (nan) | > loss: 0.0084677804261446 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13672274763250736) | > loader_time: 0.0053 (0.004704584742768928)  --> TIME: 2025-02-25 02:24:56 -- STEP: 32880/36938 -- GLOBAL_STEP: 586950 | > loss_text_ce: 0.023488499224185944 (0.036294470857709836) | > loss_mel_ce: 1.6188361644744873 (nan) | > loss: 0.00651716161519289 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13672583964619256) | > loader_time: 0.0045 (0.004704597933159009)  --> TIME: 2025-02-25 02:25:05 -- STEP: 32930/36938 -- GLOBAL_STEP: 587000 | > loss_text_ce: 0.022199597209692 (0.03629208921805254) | > loss_mel_ce: 2.7507903575897217 (nan) | > loss: 0.011003929190337658 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13672329101597186) | > loader_time: 0.0045 (0.004704603872290548)  --> TIME: 2025-02-25 02:25:15 -- STEP: 32980/36938 -- GLOBAL_STEP: 587050 | > loss_text_ce: 0.029913829639554024 (0.036291359092395184) | > loss_mel_ce: 2.1517841815948486 (nan) | > loss: 0.00865753274410963 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13672255798857966) | > loader_time: 0.0046 (0.004704487085197865)  --> TIME: 2025-02-25 02:25:24 -- STEP: 33030/36938 -- GLOBAL_STEP: 587100 | > loss_text_ce: 0.040688756853342056 (0.0362899630447235) | > loss_mel_ce: 3.0661470890045166 (nan) | > loss: 0.01232871413230896 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13671777279420946) | > loader_time: 0.0046 (0.004704327991865605)  --> TIME: 2025-02-25 02:25:34 -- STEP: 33080/36938 -- GLOBAL_STEP: 587150 | > loss_text_ce: 0.03965642675757408 (0.03629196771950384) | > loss_mel_ce: 2.82407546043396 (nan) | > loss: 0.011364015750586987 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13672890573055238) | > loader_time: 0.0045 (0.00470419545559962)  --> TIME: 2025-02-25 02:25:43 -- STEP: 33130/36938 -- GLOBAL_STEP: 587200 | > loss_text_ce: 0.04747587442398071 (0.03629087059874696) | > loss_mel_ce: 1.96875 (nan) | > loss: 0.00800089631229639 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13673155930982772) | > loader_time: 0.0045 (0.004704085808306964)  --> TIME: 2025-02-25 02:25:52 -- STEP: 33180/36938 -- GLOBAL_STEP: 587250 | > loss_text_ce: 0.034404024481773376 (0.03629080044692842) | > loss_mel_ce: 2.0442473888397217 (nan) | > loss: 0.008248616941273212 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13673104539271003) | > loader_time: 0.0046 (0.004704077391828562)  --> TIME: 2025-02-25 02:26:02 -- STEP: 33230/36938 -- GLOBAL_STEP: 587300 | > loss_text_ce: 0.014554671943187714 (0.036290189289812105) | > loss_mel_ce: 2.6147921085357666 (nan) | > loss: 0.010433916933834553 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13673640596123945) | > loader_time: 0.0052 (0.004704072193463777)  --> TIME: 2025-02-25 02:26:11 -- STEP: 33280/36938 -- GLOBAL_STEP: 587350 | > loss_text_ce: 0.030956212431192398 (0.03629085125464296) | > loss_mel_ce: 2.8802082538604736 (nan) | > loss: 0.011552240699529648 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1367358435088618) | > loader_time: 0.0047 (0.004704064553460288)  --> TIME: 2025-02-25 02:26:21 -- STEP: 33330/36938 -- GLOBAL_STEP: 587400 | > loss_text_ce: 0.026908602565526962 (0.03629019602914127) | > loss_mel_ce: 3.0821290016174316 (nan) | > loss: 0.012337451800704002 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13673993365170106) | > loader_time: 0.0046 (0.004704044654209875)  --> TIME: 2025-02-25 02:26:30 -- STEP: 33380/36938 -- GLOBAL_STEP: 587450 | > loss_text_ce: 0.03612040728330612 (0.03628898495975642) | > loss_mel_ce: 3.028778553009033 (nan) | > loss: 0.01216229796409607 (nan) | > current_lr: 5e-06 | > step_time: 0.1599 (0.13673817934284171) | > loader_time: 0.0046 (0.004703895405711673)  --> TIME: 2025-02-25 02:26:39 -- STEP: 33430/36938 -- GLOBAL_STEP: 587500 | > loss_text_ce: 0.030504141002893448 (0.03629121103274124) | > loss_mel_ce: 3.1199679374694824 (nan) | > loss: 0.012501874007284641 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.13673553345554937) | > loader_time: 0.0047 (0.0047037423316711735)  --> TIME: 2025-02-25 02:26:49 -- STEP: 33480/36938 -- GLOBAL_STEP: 587550 | > loss_text_ce: 0.037503521889448166 (0.036290628125550554) | > loss_mel_ce: 1.3787779808044434 (nan) | > loss: 0.005620164796710014 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13674792571164593) | > loader_time: 0.0045 (0.0047036619525372)  --> TIME: 2025-02-25 02:26:59 -- STEP: 33530/36938 -- GLOBAL_STEP: 587600 | > loss_text_ce: 0.031438179314136505 (0.03629197608846181) | > loss_mel_ce: 3.2530786991119385 (nan) | > loss: 0.013033797964453697 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13675937001256425) | > loader_time: 0.0046 (0.004703824334524336)  --> TIME: 2025-02-25 02:27:08 -- STEP: 33580/36938 -- GLOBAL_STEP: 587650 | > loss_text_ce: 0.02721193991601467 (0.036291908548768716) | > loss_mel_ce: 2.0913586616516113 (nan) | > loss: 0.008407026529312134 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.1367628830539954) | > loader_time: 0.0052 (0.004704043934784064)  --> TIME: 2025-02-25 02:27:17 -- STEP: 33630/36938 -- GLOBAL_STEP: 587700 | > loss_text_ce: 0.035785797983407974 (0.03629271728508645) | > loss_mel_ce: 3.122622013092041 (nan) | > loss: 0.012533365748822689 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13676166187749114) | > loader_time: 0.0054 (0.004704069119991417)  --> TIME: 2025-02-25 02:27:27 -- STEP: 33680/36938 -- GLOBAL_STEP: 587750 | > loss_text_ce: 0.04765038937330246 (0.03629369166250514) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13676194532325256) | > loader_time: 0.0047 (0.004704172898641429)  --> TIME: 2025-02-25 02:27:36 -- STEP: 33730/36938 -- GLOBAL_STEP: 587800 | > loss_text_ce: 0.03880004584789276 (0.036294488238102766) | > loss_mel_ce: 2.6712024211883545 (nan) | > loss: 0.010753978043794632 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13676220086360885) | > loader_time: 0.0047 (0.004704086673814646)  --> TIME: 2025-02-25 02:27:45 -- STEP: 33780/36938 -- GLOBAL_STEP: 587850 | > loss_text_ce: 0.039435647428035736 (0.03629532519145107) | > loss_mel_ce: 3.020275354385376 (nan) | > loss: 0.012141711078584194 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13675856777307413) | > loader_time: 0.0053 (0.004704027792770849)  --> TIME: 2025-02-25 02:27:54 -- STEP: 33830/36938 -- GLOBAL_STEP: 587900 | > loss_text_ce: 0.036780569702386856 (0.03629616747871521) | > loss_mel_ce: 1.1910171508789062 (nan) | > loss: 0.0048722135834395885 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13675504962685406) | > loader_time: 0.0047 (0.004704112320160574)  --> TIME: 2025-02-25 02:28:04 -- STEP: 33880/36938 -- GLOBAL_STEP: 587950 | > loss_text_ce: 0.03768935054540634 (0.03629424785593148) | > loss_mel_ce: 3.392214059829712 (nan) | > loss: 0.01361072901636362 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13674844974891573) | > loader_time: 0.0046 (0.004704103116583836)  --> TIME: 2025-02-25 02:28:13 -- STEP: 33930/36938 -- GLOBAL_STEP: 588000 | > loss_text_ce: 0.03833315148949623 (0.036295132215236024) | > loss_mel_ce: 3.2163143157958984 (nan) | > loss: 0.012915268540382385 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13675178962930062) | > loader_time: 0.0047 (0.004704125659008386)  --> TIME: 2025-02-25 02:28:22 -- STEP: 33980/36938 -- GLOBAL_STEP: 588050 | > loss_text_ce: 0.033468447625637054 (0.03629315877484022) | > loss_mel_ce: 2.894423723220825 (nan) | > loss: 0.011618620716035366 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1367491872399039) | > loader_time: 0.0052 (0.004703991156314847)  --> TIME: 2025-02-25 02:28:31 -- STEP: 34030/36938 -- GLOBAL_STEP: 588100 | > loss_text_ce: 0.045494548976421356 (0.03629230152015891) | > loss_mel_ce: 3.3969669342041016 (nan) | > loss: 0.013660562224686146 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13674688773613344) | > loader_time: 0.0045 (0.004703840808661066)  --> TIME: 2025-02-25 02:28:40 -- STEP: 34080/36938 -- GLOBAL_STEP: 588150 | > loss_text_ce: 0.042082007974386215 (0.03629372281843831) | > loss_mel_ce: 1.9553916454315186 (nan) | > loss: 0.00792648270726204 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.1367376963787229) | > loader_time: 0.0045 (0.004703605762669777)  --> TIME: 2025-02-25 02:28:50 -- STEP: 34130/36938 -- GLOBAL_STEP: 588200 | > loss_text_ce: 0.032379526644945145 (0.036292105214138136) | > loss_mel_ce: 2.74418568611145 (nan) | > loss: 0.01101811695843935 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13673773271542164) | > loader_time: 0.0045 (0.004703656641437381)  --> TIME: 2025-02-25 02:28:59 -- STEP: 34180/36938 -- GLOBAL_STEP: 588250 | > loss_text_ce: 0.041030269116163254 (0.03629186257146153) | > loss_mel_ce: 2.429889440536499 (nan) | > loss: 0.00980523694306612 (nan) | > current_lr: 5e-06 | > step_time: 0.1515 (0.1367342619535999) | > loader_time: 0.0048 (0.0047037313317890225)  --> TIME: 2025-02-25 02:29:08 -- STEP: 34230/36938 -- GLOBAL_STEP: 588300 | > loss_text_ce: 0.035482313483953476 (0.03629219078652286) | > loss_mel_ce: 2.6448609828948975 (nan) | > loss: 0.01063628401607275 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13673547014241402) | > loader_time: 0.0046 (0.004703640617406306)  --> TIME: 2025-02-25 02:29:18 -- STEP: 34280/36938 -- GLOBAL_STEP: 588350 | > loss_text_ce: 0.03316793218255043 (0.03629103776520109) | > loss_mel_ce: 2.55000376701355 (nan) | > loss: 0.010250682011246681 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13674129468835441) | > loader_time: 0.0046 (0.004703692439834917)  --> TIME: 2025-02-25 02:29:27 -- STEP: 34330/36938 -- GLOBAL_STEP: 588400 | > loss_text_ce: 0.04389704018831253 (0.03629257891415574) | > loss_mel_ce: 2.581730842590332 (nan) | > loss: 0.010419159196317196 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.1367407247063853) | > loader_time: 0.0045 (0.004703611317786575)  --> TIME: 2025-02-25 02:29:37 -- STEP: 34380/36938 -- GLOBAL_STEP: 588450 | > loss_text_ce: 0.03751255199313164 (0.03629276001722439) | > loss_mel_ce: 4.559659004211426 (nan) | > loss: 0.018242746591567993 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13674366210628985) | > loader_time: 0.0046 (0.004703564772847265)  --> TIME: 2025-02-25 02:29:46 -- STEP: 34430/36938 -- GLOBAL_STEP: 588500 | > loss_text_ce: 0.04002153128385544 (0.03629234723891213) | > loss_mel_ce: 0.3529260754585266 (nan) | > loss: 0.0015593160642310977 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13674147009953364) | > loader_time: 0.0046 (0.004703505309972211)  --> TIME: 2025-02-25 02:29:55 -- STEP: 34480/36938 -- GLOBAL_STEP: 588550 | > loss_text_ce: 0.036237772554159164 (0.03629405715131665) | > loss_mel_ce: 2.5392887592315674 (nan) | > loss: 0.010220344178378582 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.136734872544724) | > loader_time: 0.0046 (0.004703448674795005)  --> TIME: 2025-02-25 02:30:04 -- STEP: 34530/36938 -- GLOBAL_STEP: 588600 | > loss_text_ce: 0.028805486857891083 (0.0362952887374058) | > loss_mel_ce: 3.6400599479675293 (nan) | > loss: 0.014558990485966206 (nan) | > current_lr: 5e-06 | > step_time: 0.173 (0.13673572465022144) | > loader_time: 0.0049 (0.004703593744321628)  --> TIME: 2025-02-25 02:30:14 -- STEP: 34580/36938 -- GLOBAL_STEP: 588650 | > loss_text_ce: 0.043996185064315796 (0.03629316708286858) | > loss_mel_ce: 2.9875972270965576 (nan) | > loss: 0.012030133046209812 (nan) | > current_lr: 5e-06 | > step_time: 0.1751 (0.13673584740447278) | > loader_time: 0.0048 (0.0047035107797388075)  --> TIME: 2025-02-25 02:30:23 -- STEP: 34630/36938 -- GLOBAL_STEP: 588700 | > loss_text_ce: 0.0362626276910305 (0.03629395800386908) | > loss_mel_ce: 3.6083881855010986 (nan) | > loss: 0.014462900348007679 (nan) | > current_lr: 5e-06 | > step_time: 0.1833 (0.1367360981082828) | > loader_time: 0.0051 (0.004703463208217588)  --> TIME: 2025-02-25 02:30:32 -- STEP: 34680/36938 -- GLOBAL_STEP: 588750 | > loss_text_ce: 0.03528021275997162 (0.0362926658033742) | > loss_mel_ce: 3.0000908374786377 (nan) | > loss: 0.012045123614370823 (nan) | > current_lr: 5e-06 | > step_time: 0.1687 (0.13674093781870483) | > loader_time: 0.0047 (0.004703385565932685)  --> TIME: 2025-02-25 02:30:41 -- STEP: 34730/36938 -- GLOBAL_STEP: 588800 | > loss_text_ce: 0.040164608508348465 (0.036294590517167345) | > loss_mel_ce: 1.8965911865234375 (nan) | > loss: 0.007685539312660694 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.13673611780150646) | > loader_time: 0.0047 (0.004703255712968868)  --> TIME: 2025-02-25 02:30:51 -- STEP: 34780/36938 -- GLOBAL_STEP: 588850 | > loss_text_ce: 0.036547161638736725 (0.036293264232064956) | > loss_mel_ce: 3.4850220680236816 (nan) | > loss: 0.01397448219358921 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13673160834309644) | > loader_time: 0.0046 (0.004703166499913907)  --> TIME: 2025-02-25 02:31:00 -- STEP: 34830/36938 -- GLOBAL_STEP: 588900 | > loss_text_ce: 0.04450554400682449 (0.03629089629725735) | > loss_mel_ce: 2.748582124710083 (nan) | > loss: 0.01108368206769228 (nan) | > current_lr: 5e-06 | > step_time: 0.1727 (0.13673040777864048) | > loader_time: 0.0048 (0.004703097346182945)  --> TIME: 2025-02-25 02:31:09 -- STEP: 34880/36938 -- GLOBAL_STEP: 588950 | > loss_text_ce: 0.030679263174533844 (0.03629110096321004) | > loss_mel_ce: 3.0753402709960938 (nan) | > loss: 0.012325474992394447 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1367301424230452) | > loader_time: 0.0046 (0.00470306583500787)  --> TIME: 2025-02-25 02:31:19 -- STEP: 34930/36938 -- GLOBAL_STEP: 589000 | > loss_text_ce: 0.03635988011956215 (0.03629038909075833) | > loss_mel_ce: 3.1545844078063965 (nan) | > loss: 0.012662477791309357 (nan) | > current_lr: 5e-06 | > step_time: 0.1706 (0.13673338230039092) | > loader_time: 0.0047 (0.004703234015376633)  --> TIME: 2025-02-25 02:31:28 -- STEP: 34980/36938 -- GLOBAL_STEP: 589050 | > loss_text_ce: 0.03644280508160591 (0.03628903704135227) | > loss_mel_ce: 2.9632315635681152 (nan) | > loss: 0.011903470382094383 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13673089670412422) | > loader_time: 0.0046 (0.00470330560459557)  --> TIME: 2025-02-25 02:31:37 -- STEP: 35030/36938 -- GLOBAL_STEP: 589100 | > loss_text_ce: 0.0360700786113739 (0.03629058818105539) | > loss_mel_ce: 3.011190176010132 (nan) | > loss: 0.012092303484678268 (nan) | > current_lr: 5e-06 | > step_time: 0.1523 (0.13673017857383002) | > loader_time: 0.0046 (0.004703324997455742)  --> TIME: 2025-02-25 02:31:46 -- STEP: 35080/36938 -- GLOBAL_STEP: 589150 | > loss_text_ce: 0.04031495004892349 (0.03629115316265518) | > loss_mel_ce: 2.716024398803711 (nan) | > loss: 0.010937854647636414 (nan) | > current_lr: 5e-06 | > step_time: 0.1588 (0.13673178113852502) | > loader_time: 0.0048 (0.004703288380069398)  --> TIME: 2025-02-25 02:31:56 -- STEP: 35130/36938 -- GLOBAL_STEP: 589200 | > loss_text_ce: 0.029405349865555763 (0.036291430600240975) | > loss_mel_ce: 0.365846186876297 (nan) | > loss: 0.0015684586251154542 (nan) | > current_lr: 5e-06 | > step_time: 0.1659 (0.1367289611494437) | > loader_time: 0.0047 (0.004703281185683349)  --> TIME: 2025-02-25 02:32:05 -- STEP: 35180/36938 -- GLOBAL_STEP: 589250 | > loss_text_ce: 0.033769138157367706 (0.036292121090675825) | > loss_mel_ce: 3.0878665447235107 (nan) | > loss: 0.012387444265186787 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.13673071062923983) | > loader_time: 0.0046 (0.004703190151841383)  --> TIME: 2025-02-25 02:32:14 -- STEP: 35230/36938 -- GLOBAL_STEP: 589300 | > loss_text_ce: 0.03579411655664444 (0.03629228443405913) | > loss_mel_ce: 3.4939751625061035 (nan) | > loss: 0.0140070216730237 (nan) | > current_lr: 5e-06 | > step_time: 0.1788 (0.13673275511203167) | > loader_time: 0.0049 (0.004703106658214441)  --> TIME: 2025-02-25 02:32:24 -- STEP: 35280/36938 -- GLOBAL_STEP: 589350 | > loss_text_ce: 0.03623557463288307 (0.03629208499253701) | > loss_mel_ce: 3.587394952774048 (nan) | > loss: 0.014379487372934818 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13673970700661828) | > loader_time: 0.0046 (0.004703195781664551)  --> TIME: 2025-02-25 02:32:33 -- STEP: 35330/36938 -- GLOBAL_STEP: 589400 | > loss_text_ce: 0.027295226231217384 (0.03628926603898553) | > loss_mel_ce: 2.9122865200042725 (nan) | > loss: 0.01166500709950924 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13673562628332708) | > loader_time: 0.0045 (0.00470314400411351)  --> TIME: 2025-02-25 02:32:42 -- STEP: 35380/36938 -- GLOBAL_STEP: 589450 | > loss_text_ce: 0.0478532500565052 (0.03628836818181042) | > loss_mel_ce: 3.4746031761169434 (nan) | > loss: 0.013978002592921257 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1367320003517602) | > loader_time: 0.0046 (0.004703098141318232)  --> TIME: 2025-02-25 02:32:51 -- STEP: 35430/36938 -- GLOBAL_STEP: 589500 | > loss_text_ce: 0.033299487084150314 (0.03628853112472333) | > loss_mel_ce: 1.2072689533233643 (nan) | > loss: 0.004922890570014715 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.1367274758373645) | > loader_time: 0.0047 (0.004703203675574014)  --> TIME: 2025-02-25 02:33:01 -- STEP: 35480/36938 -- GLOBAL_STEP: 589550 | > loss_text_ce: 0.04913114011287689 (0.036288934674932) | > loss_mel_ce: 3.090845823287964 (nan) | > loss: 0.012460227124392986 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13672861476358394) | > loader_time: 0.0045 (0.004703223160987817)  --> TIME: 2025-02-25 02:33:10 -- STEP: 35530/36938 -- GLOBAL_STEP: 589600 | > loss_text_ce: 0.031193749979138374 (0.03628915823330806) | > loss_mel_ce: 3.450759172439575 (nan) | > loss: 0.013817274011671543 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13673059373313864) | > loader_time: 0.0048 (0.004703177407262333)  --> TIME: 2025-02-25 02:33:20 -- STEP: 35580/36938 -- GLOBAL_STEP: 589650 | > loss_text_ce: 0.03490070998668671 (0.036287469809200674) | > loss_mel_ce: 1.0054336786270142 (nan) | > loss: 0.0041283112950623035 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13672982915322854) | > loader_time: 0.0045 (0.0047032367519745715)  --> TIME: 2025-02-25 02:33:29 -- STEP: 35630/36938 -- GLOBAL_STEP: 589700 | > loss_text_ce: 0.03760887682437897 (0.03628788146749981) | > loss_mel_ce: 3.608649492263794 (nan) | > loss: 0.014469279907643795 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13674171965944537) | > loader_time: 0.0048 (0.004703309968921102)  --> TIME: 2025-02-25 02:33:39 -- STEP: 35680/36938 -- GLOBAL_STEP: 589750 | > loss_text_ce: 0.03164706751704216 (0.03628718576385153) | > loss_mel_ce: 2.5008397102355957 (nan) | > loss: 0.010049550794064999 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13674014402478188) | > loader_time: 0.0046 (0.004703388232821344)  --> TIME: 2025-02-25 02:33:48 -- STEP: 35730/36938 -- GLOBAL_STEP: 589800 | > loss_text_ce: 0.04653256759047508 (0.03628699228361991) | > loss_mel_ce: 2.9240505695343018 (nan) | > loss: 0.011788029223680496 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13673929848525426) | > loader_time: 0.0045 (0.004703453772881476)  --> TIME: 2025-02-25 02:33:57 -- STEP: 35780/36938 -- GLOBAL_STEP: 589850 | > loss_text_ce: 0.03843072056770325 (0.036286739484043214) | > loss_mel_ce: 3.2319514751434326 (nan) | > loss: 0.012977708131074905 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13673948820507473) | > loader_time: 0.0045 (0.004703502637704341)  --> TIME: 2025-02-25 02:34:07 -- STEP: 35830/36938 -- GLOBAL_STEP: 589900 | > loss_text_ce: 0.040701478719711304 (0.03628963577627891) | > loss_mel_ce: 2.0749878883361816 (nan) | > loss: 0.00839559268206358 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13674435613543234) | > loader_time: 0.0046 (0.0047035534821706845)  --> TIME: 2025-02-25 02:34:17 -- STEP: 35880/36938 -- GLOBAL_STEP: 589950 | > loss_text_ce: 0.034135282039642334 (0.03628985382003433) | > loss_mel_ce: 1.6399058103561401 (nan) | > loss: 0.006643020547926426 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.1367544715486899) | > loader_time: 0.0046 (0.004703459658617462)  --> TIME: 2025-02-25 02:34:26 -- STEP: 35930/36938 -- GLOBAL_STEP: 590000 | > loss_text_ce: 0.03841074928641319 (0.03629036292982222) | > loss_mel_ce: 3.0644161701202393 (nan) | > loss: 0.012312805280089378 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.1367532729704393) | > loader_time: 0.0046 (0.004703432877283429) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_590000.pth  --> TIME: 2025-02-25 02:34:41 -- STEP: 35980/36938 -- GLOBAL_STEP: 590050 | > loss_text_ce: 0.033691540360450745 (0.03628980022160888) | > loss_mel_ce: 2.7591824531555176 (nan) | > loss: 0.01108283456414938 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.1367564952592169) | > loader_time: 0.0048 (0.004703434723625624)  --> TIME: 2025-02-25 02:34:51 -- STEP: 36030/36938 -- GLOBAL_STEP: 590100 | > loss_text_ce: 0.03913387283682823 (0.03628980649178719) | > loss_mel_ce: 0.7603403925895691 (nan) | > loss: 0.003172517055645585 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13675496990469532) | > loader_time: 0.0046 (0.004703349303511063)  --> TIME: 2025-02-25 02:35:00 -- STEP: 36080/36938 -- GLOBAL_STEP: 590150 | > loss_text_ce: 0.028456872329115868 (0.03629023027626966) | > loss_mel_ce: 2.793644666671753 (nan) | > loss: 0.011198816820979118 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13674854221206478) | > loader_time: 0.0046 (0.004703313773064321)  --> TIME: 2025-02-25 02:35:09 -- STEP: 36130/36938 -- GLOBAL_STEP: 590200 | > loss_text_ce: 0.03563392907381058 (0.036289652870850016) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13675034105563205) | > loader_time: 0.0046 (0.004703483171090871)  --> TIME: 2025-02-25 02:35:18 -- STEP: 36180/36938 -- GLOBAL_STEP: 590250 | > loss_text_ce: 0.03271910920739174 (0.03628743416905173) | > loss_mel_ce: 3.0039329528808594 (nan) | > loss: 0.012050207704305649 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13674771985406634) | > loader_time: 0.0046 (0.004703524245815197)  --> TIME: 2025-02-25 02:35:28 -- STEP: 36230/36938 -- GLOBAL_STEP: 590300 | > loss_text_ce: 0.056710246950387955 (0.03628675844967622) | > loss_mel_ce: 3.0594429969787598 (nan) | > loss: 0.012365688569843769 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1367532746422655) | > loader_time: 0.0047 (0.0047035274405589)  --> TIME: 2025-02-25 02:35:38 -- STEP: 36280/36938 -- GLOBAL_STEP: 590350 | > loss_text_ce: 0.03946630284190178 (0.03628744962152882) | > loss_mel_ce: 3.179415225982666 (nan) | > loss: 0.012773340567946434 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13675857602301278) | > loader_time: 0.0045 (0.004703649250911676)  --> TIME: 2025-02-25 02:35:47 -- STEP: 36330/36938 -- GLOBAL_STEP: 590400 | > loss_text_ce: 0.029815251007676125 (0.03628804548983206) | > loss_mel_ce: 2.7787020206451416 (nan) | > loss: 0.011144910007715225 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13675837511534536) | > loader_time: 0.0046 (0.004703734802405729)  --> TIME: 2025-02-25 02:35:56 -- STEP: 36380/36938 -- GLOBAL_STEP: 590450 | > loss_text_ce: 0.044493623077869415 (0.03628726906003365) | > loss_mel_ce: 2.4270055294036865 (nan) | > loss: 0.00980753730982542 (nan) | > current_lr: 5e-06 | > step_time: 0.1491 (0.13675484182154304) | > loader_time: 0.0046 (0.004703743494494167)  --> TIME: 2025-02-25 02:36:05 -- STEP: 36430/36938 -- GLOBAL_STEP: 590500 | > loss_text_ce: 0.041265808045864105 (0.03628686231745337) | > loss_mel_ce: 2.615151882171631 (nan) | > loss: 0.010541340336203575 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13675506400302775) | > loader_time: 0.0047 (0.004703766370978061)  --> TIME: 2025-02-25 02:36:15 -- STEP: 36480/36938 -- GLOBAL_STEP: 590550 | > loss_text_ce: 0.02833687514066696 (0.03628726239625595) | > loss_mel_ce: 2.960831642150879 (nan) | > loss: 0.011861779727041721 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.13675863055937057) | > loader_time: 0.0047 (0.004703809869916851)  --> TIME: 2025-02-25 02:36:24 -- STEP: 36530/36938 -- GLOBAL_STEP: 590600 | > loss_text_ce: 0.03848201781511307 (0.036286896624995846) | > loss_mel_ce: 3.1303579807281494 (nan) | > loss: 0.012574762105941772 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.13676046436856948) | > loader_time: 0.0046 (0.0047038340353163345)  --> TIME: 2025-02-25 02:36:34 -- STEP: 36580/36938 -- GLOBAL_STEP: 590650 | > loss_text_ce: 0.03320803493261337 (0.036285617405563055) | > loss_mel_ce: 3.4735970497131348 (nan) | > loss: 0.0139158945530653 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13676452771635705) | > loader_time: 0.0046 (0.00470391060238399)  --> TIME: 2025-02-25 02:36:43 -- STEP: 36630/36938 -- GLOBAL_STEP: 590700 | > loss_text_ce: 0.03144808113574982 (0.03628716127074392) | > loss_mel_ce: 2.8945024013519287 (nan) | > loss: 0.011610915884375572 (nan) | > current_lr: 5e-06 | > step_time: 0.1798 (0.13676132558311752) | > loader_time: 0.0049 (0.00470386771208079)  --> TIME: 2025-02-25 02:36:52 -- STEP: 36680/36938 -- GLOBAL_STEP: 590750 | > loss_text_ce: 0.045331619679927826 (0.03628761061163063) | > loss_mel_ce: 2.8764209747314453 (nan) | > loss: 0.011594257317483425 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13675969703896812) | > loader_time: 0.0047 (0.004703824106713579)  --> TIME: 2025-02-25 02:37:02 -- STEP: 36730/36938 -- GLOBAL_STEP: 590800 | > loss_text_ce: 0.03714810311794281 (0.03628621793436006) | > loss_mel_ce: 3.3743226528167725 (nan) | > loss: 0.013537582941353321 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.13676456853994778) | > loader_time: 0.0049 (0.004703971991057392)  --> TIME: 2025-02-25 02:37:11 -- STEP: 36780/36938 -- GLOBAL_STEP: 590850 | > loss_text_ce: 0.040551599115133286 (0.036284728938320784) | > loss_mel_ce: 3.334820508956909 (nan) | > loss: 0.013394334353506565 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.13676444001661886) | > loader_time: 0.0048 (0.004704007712183241)  --> TIME: 2025-02-25 02:37:20 -- STEP: 36830/36938 -- GLOBAL_STEP: 590900 | > loss_text_ce: 0.034120190888643265 (0.03628451681071046) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13676391995021056) | > loader_time: 0.0046 (0.004703933837262987)  --> TIME: 2025-02-25 02:37:30 -- STEP: 36880/36938 -- GLOBAL_STEP: 590950 | > loss_text_ce: 0.03612467646598816 (0.03628332860974419) | > loss_mel_ce: 2.231616258621216 (nan) | > loss: 0.008998972363770008 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.136764835542039) | > loader_time: 0.0048 (0.004703875393515845)  --> TIME: 2025-02-25 02:37:39 -- STEP: 36930/36938 -- GLOBAL_STEP: 591000 | > loss_text_ce: 0.030051929876208305 (0.03628285089739721) | > loss_mel_ce: 3.112755060195923 (nan) | > loss: 0.0124714570119977 (nan) | > current_lr: 5e-06 | > step_time: 0.1835 (0.1367603819576209) | > loader_time: 0.0047 (0.00470375748723951)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.020969430999000484 (-0.0001899771171041051) | > avg_loss_text_ce: 0.03539092305789487 (-0.0004757985583331917) | > avg_loss_mel_ce: 2.6151106463800566 (-0.014523112537835559) | > avg_loss: 2.650501563997553 (-0.014998923138817677) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_591008.pth  > EPOCH: 17/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-25 02:38:05)   --> TIME: 2025-02-25 02:38:13 -- STEP: 42/36938 -- GLOBAL_STEP: 591050 | > loss_text_ce: 0.041615456342697144 (0.03616364238162835) | > loss_mel_ce: 3.0789895057678223 (nan) | > loss: 0.012383353896439075 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13687612329210552) | > loader_time: 0.0049 (0.005035315241132462)  --> TIME: 2025-02-25 02:38:23 -- STEP: 92/36938 -- GLOBAL_STEP: 591100 | > loss_text_ce: 0.04037737101316452 (0.03596469456248956) | > loss_mel_ce: 2.740614891052246 (nan) | > loss: 0.011035684496164322 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13586663681527844) | > loader_time: 0.0047 (0.004926342031230094)  --> TIME: 2025-02-25 02:38:32 -- STEP: 142/36938 -- GLOBAL_STEP: 591150 | > loss_text_ce: 0.04128088802099228 (0.03597910157863943) | > loss_mel_ce: 2.4425899982452393 (nan) | > loss: 0.0098566310480237 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13554611844076242) | > loader_time: 0.0048 (0.004895648486177685)  --> TIME: 2025-02-25 02:38:41 -- STEP: 192/36938 -- GLOBAL_STEP: 591200 | > loss_text_ce: 0.037442803382873535 (0.03629728367862602) | > loss_mel_ce: 2.8012807369232178 (nan) | > loss: 0.011264776811003685 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.1355082492033641) | > loader_time: 0.005 (0.004879442354043323)  --> TIME: 2025-02-25 02:38:51 -- STEP: 242/36938 -- GLOBAL_STEP: 591250 | > loss_text_ce: 0.03579653427004814 (0.03639001342321722) | > loss_mel_ce: 3.1200826168060303 (nan) | > loss: 0.012523330748081207 (nan) | > current_lr: 5e-06 | > step_time: 0.1513 (0.13734549530281515) | > loader_time: 0.0048 (0.004876958437202389)  --> TIME: 2025-02-25 02:39:01 -- STEP: 292/36938 -- GLOBAL_STEP: 591300 | > loss_text_ce: 0.03876744583249092 (0.03611654195015968) | > loss_mel_ce: 2.465660572052002 (nan) | > loss: 0.009938206523656845 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.1390360185544785) | > loader_time: 0.0046 (0.004869637423998691)  --> TIME: 2025-02-25 02:39:10 -- STEP: 342/36938 -- GLOBAL_STEP: 591350 | > loss_text_ce: 0.026713751256465912 (0.03606392866359989) | > loss_mel_ce: 2.3058359622955322 (nan) | > loss: 0.00925615057349205 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13902231475763163) | > loader_time: 0.0048 (0.004850955734476012)  --> TIME: 2025-02-25 02:39:20 -- STEP: 392/36938 -- GLOBAL_STEP: 591400 | > loss_text_ce: 0.04423875734210014 (0.035891145767111825) | > loss_mel_ce: 2.916534662246704 (nan) | > loss: 0.011749101802706718 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.13866998893874047) | > loader_time: 0.0047 (0.0048433536169480325)  --> TIME: 2025-02-25 02:39:29 -- STEP: 442/36938 -- GLOBAL_STEP: 591450 | > loss_text_ce: 0.03299269080162048 (0.03587128167626409) | > loss_mel_ce: 3.1338393688201904 (nan) | > loss: 0.012566794641315937 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13844808625959165) | > loader_time: 0.0047 (0.00482144129222335)  --> TIME: 2025-02-25 02:39:38 -- STEP: 492/36938 -- GLOBAL_STEP: 591500 | > loss_text_ce: 0.032902415841817856 (0.03573742856998449) | > loss_mel_ce: 1.279758334159851 (nan) | > loss: 0.005208971444517374 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13781849591712644) | > loader_time: 0.0047 (0.004806566529157687)  --> TIME: 2025-02-25 02:39:47 -- STEP: 542/36938 -- GLOBAL_STEP: 591550 | > loss_text_ce: 0.037174783647060394 (0.03567489107548365) | > loss_mel_ce: 2.8172504901885986 (nan) | > loss: 0.011327085085213184 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13771419023675677) | > loader_time: 0.0045 (0.004793281044907238)  --> TIME: 2025-02-25 02:39:57 -- STEP: 592/36938 -- GLOBAL_STEP: 591600 | > loss_text_ce: 0.0372135192155838 (0.035620185438027537) | > loss_mel_ce: 3.1765458583831787 (nan) | > loss: 0.012753014452755451 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13746101590427198) | > loader_time: 0.0046 (0.004784375429153445)  --> TIME: 2025-02-25 02:40:06 -- STEP: 642/36938 -- GLOBAL_STEP: 591650 | > loss_text_ce: 0.029956771060824394 (0.03564105643760237) | > loss_mel_ce: 2.6875061988830566 (nan) | > loss: 0.010783583857119083 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.137434191792925) | > loader_time: 0.005 (0.0047803385606807365)  --> TIME: 2025-02-25 02:40:16 -- STEP: 692/36938 -- GLOBAL_STEP: 591700 | > loss_text_ce: 0.029921455308794975 (0.03559696446226726) | > loss_mel_ce: 2.79154896736145 (nan) | > loss: 0.011196312494575977 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.137853473252644) | > loader_time: 0.0046 (0.004778833403063649)  --> TIME: 2025-02-25 02:40:25 -- STEP: 742/36938 -- GLOBAL_STEP: 591750 | > loss_text_ce: 0.042877040803432465 (0.03563183063711477) | > loss_mel_ce: 3.099827527999878 (nan) | > loss: 0.012471050024032593 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.13775011889054145) | > loader_time: 0.0047 (0.004777105992052439)  --> TIME: 2025-02-25 02:40:34 -- STEP: 792/36938 -- GLOBAL_STEP: 591800 | > loss_text_ce: 0.043036844581365585 (0.03566912436420379) | > loss_mel_ce: 2.623270034790039 (nan) | > loss: 0.010580583475530148 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13743867055334255) | > loader_time: 0.0046 (0.004767681312079387)  --> TIME: 2025-02-25 02:40:43 -- STEP: 842/36938 -- GLOBAL_STEP: 591850 | > loss_text_ce: 0.040209852159023285 (0.03566287782799778) | > loss_mel_ce: 2.6786587238311768 (nan) | > loss: 0.010789161548018456 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13749290333790928) | > loader_time: 0.0045 (0.0047631648827052234)  --> TIME: 2025-02-25 02:40:53 -- STEP: 892/36938 -- GLOBAL_STEP: 591900 | > loss_text_ce: 0.05325193703174591 (0.035742190260348215) | > loss_mel_ce: 3.1471762657165527 (nan) | > loss: 0.012700112536549568 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13750590096674706) | > loader_time: 0.0045 (0.004756433814095815)  --> TIME: 2025-02-25 02:41:02 -- STEP: 942/36938 -- GLOBAL_STEP: 591950 | > loss_text_ce: 0.03277838975191116 (0.03574559634804503) | > loss_mel_ce: 1.2405226230621338 (nan) | > loss: 0.005052782129496336 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13713591974505468) | > loader_time: 0.0047 (0.004748226983785128)  --> TIME: 2025-02-25 02:41:11 -- STEP: 992/36938 -- GLOBAL_STEP: 592000 | > loss_text_ce: 0.03854300081729889 (0.03575530030197583) | > loss_mel_ce: 2.702491283416748 (nan) | > loss: 0.010877120308578014 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.13699793575271504) | > loader_time: 0.0046 (0.004742218361746885)  --> TIME: 2025-02-25 02:41:21 -- STEP: 1042/36938 -- GLOBAL_STEP: 592050 | > loss_text_ce: 0.04282540827989578 (0.0357346436743501) | > loss_mel_ce: 3.0855605602264404 (nan) | > loss: 0.012414230965077877 (nan) | > current_lr: 5e-06 | > step_time: 0.1853 (0.13733692292746125) | > loader_time: 0.0049 (0.004738369509720761)  --> TIME: 2025-02-25 02:41:30 -- STEP: 1092/36938 -- GLOBAL_STEP: 592100 | > loss_text_ce: 0.03467108681797981 (0.03576472799136769) | > loss_mel_ce: 3.04772686958313 (nan) | > loss: 0.012231738306581974 (nan) | > current_lr: 5e-06 | > step_time: 0.1899 (0.13738903632530816) | > loader_time: 0.005 (0.00473357906271687)  --> TIME: 2025-02-25 02:41:39 -- STEP: 1142/36938 -- GLOBAL_STEP: 592150 | > loss_text_ce: 0.041151173412799835 (0.03579320032361585) | > loss_mel_ce: 3.212749481201172 (nan) | > loss: 0.012912305071949959 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13717737318964232) | > loader_time: 0.0047 (0.004731156362543596)  --> TIME: 2025-02-25 02:41:48 -- STEP: 1192/36938 -- GLOBAL_STEP: 592200 | > loss_text_ce: 0.04272989556193352 (0.035787742263043654) | > loss_mel_ce: 3.454463005065918 (nan) | > loss: 0.013877750374376774 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13688324621859826) | > loader_time: 0.0047 (0.004729694568070796)  --> TIME: 2025-02-25 02:41:57 -- STEP: 1242/36938 -- GLOBAL_STEP: 592250 | > loss_text_ce: 0.03579692170023918 (0.03580380908113071) | > loss_mel_ce: 2.8863680362701416 (nan) | > loss: 0.011595893651247025 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13661097904334338) | > loader_time: 0.0046 (0.004726167270334839)  --> TIME: 2025-02-25 02:42:06 -- STEP: 1292/36938 -- GLOBAL_STEP: 592300 | > loss_text_ce: 0.03596212714910507 (0.03579450945943682) | > loss_mel_ce: 2.8849363327026367 (nan) | > loss: 0.011590867303311825 (nan) | > current_lr: 5e-06 | > step_time: 0.1526 (0.13668065008364236) | > loader_time: 0.0047 (0.004725920711139417)  --> TIME: 2025-02-25 02:42:16 -- STEP: 1342/36938 -- GLOBAL_STEP: 592350 | > loss_text_ce: 0.03959681838750839 (0.03585079220441426) | > loss_mel_ce: 2.6254472732543945 (nan) | > loss: 0.010575572028756142 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13659481906322357) | > loader_time: 0.0046 (0.004726551151133517)  --> TIME: 2025-02-25 02:42:25 -- STEP: 1392/36938 -- GLOBAL_STEP: 592400 | > loss_text_ce: 0.035498395562171936 (0.03590216835420144) | > loss_mel_ce: 2.805349588394165 (nan) | > loss: 0.011273207142949104 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13661594363464705) | > loader_time: 0.0053 (0.004731757887478547)  --> TIME: 2025-02-25 02:42:34 -- STEP: 1442/36938 -- GLOBAL_STEP: 592450 | > loss_text_ce: 0.04122712463140488 (0.035870256053539516) | > loss_mel_ce: 3.636627197265625 (nan) | > loss: 0.014594661071896553 (nan) | > current_lr: 5e-06 | > step_time: 0.171 (0.1366773466792747) | > loader_time: 0.0048 (0.004729420401352304)  --> TIME: 2025-02-25 02:42:44 -- STEP: 1492/36938 -- GLOBAL_STEP: 592500 | > loss_text_ce: 0.035689592361450195 (0.035921225817399614) | > loss_mel_ce: 3.207768678665161 (nan) | > loss: 0.012870866805315018 (nan) | > current_lr: 5e-06 | > step_time: 0.1777 (0.13672195107304047) | > loader_time: 0.0048 (0.004727928312470394)  --> TIME: 2025-02-25 02:42:53 -- STEP: 1542/36938 -- GLOBAL_STEP: 592550 | > loss_text_ce: 0.034766145050525665 (0.03594196374840963) | > loss_mel_ce: 3.2194602489471436 (nan) | > loss: 0.012913597747683525 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.1366286715335573) | > loader_time: 0.0048 (0.004724519571287015)  --> TIME: 2025-02-25 02:43:02 -- STEP: 1592/36938 -- GLOBAL_STEP: 592600 | > loss_text_ce: 0.03172838315367699 (0.03586321636851893) | > loss_mel_ce: 3.458376169204712 (nan) | > loss: 0.013849621638655663 (nan) | > current_lr: 5e-06 | > step_time: 0.1871 (0.13649542996631778) | > loader_time: 0.0048 (0.004720133303397873)  --> TIME: 2025-02-25 02:43:11 -- STEP: 1642/36938 -- GLOBAL_STEP: 592650 | > loss_text_ce: 0.035386670380830765 (0.03585365482455116) | > loss_mel_ce: 2.6561264991760254 (nan) | > loss: 0.010680608451366425 (nan) | > current_lr: 5e-06 | > step_time: 0.1537 (0.13648922942878244) | > loader_time: 0.0047 (0.004718172419521328)  --> TIME: 2025-02-25 02:43:21 -- STEP: 1692/36938 -- GLOBAL_STEP: 592700 | > loss_text_ce: 0.04916086420416832 (0.035865845163099555) | > loss_mel_ce: 2.9956560134887695 (nan) | > loss: 0.012082607485353947 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13657059962586) | > loader_time: 0.0046 (0.004716859078012753)  --> TIME: 2025-02-25 02:43:30 -- STEP: 1742/36938 -- GLOBAL_STEP: 592750 | > loss_text_ce: 0.03641951084136963 (0.03582907453699016) | > loss_mel_ce: 3.0658257007598877 (nan) | > loss: 0.012310498394072056 (nan) | > current_lr: 5e-06 | > step_time: 0.1491 (0.13640871537949792) | > loader_time: 0.0046 (0.004716321877305473)  --> TIME: 2025-02-25 02:43:39 -- STEP: 1792/36938 -- GLOBAL_STEP: 592800 | > loss_text_ce: 0.03954356908798218 (0.03581584467039126) | > loss_mel_ce: 1.5839165449142456 (nan) | > loss: 0.006442302372306585 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13636655120977306) | > loader_time: 0.0045 (0.00471349180276906)  --> TIME: 2025-02-25 02:43:49 -- STEP: 1842/36938 -- GLOBAL_STEP: 592850 | > loss_text_ce: 0.04651765897870064 (0.03585302683976796) | > loss_mel_ce: 3.356764793395996 (nan) | > loss: 0.013505090028047562 (nan) | > current_lr: 5e-06 | > step_time: 0.1532 (0.13645417630737147) | > loader_time: 0.0046 (0.004712636385367837)  --> TIME: 2025-02-25 02:43:58 -- STEP: 1892/36938 -- GLOBAL_STEP: 592900 | > loss_text_ce: 0.03922056406736374 (0.03588513929611484) | > loss_mel_ce: 2.3940749168395996 (nan) | > loss: 0.009655934758484364 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13639783380399528) | > loader_time: 0.0046 (0.004710792739093945)  --> TIME: 2025-02-25 02:44:07 -- STEP: 1942/36938 -- GLOBAL_STEP: 592950 | > loss_text_ce: 0.041535668075084686 (0.03588086414946017) | > loss_mel_ce: 3.13661527633667 (nan) | > loss: 0.012611710466444492 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13645602807203092) | > loader_time: 0.0046 (0.004709093136104826)  --> TIME: 2025-02-25 02:44:16 -- STEP: 1992/36938 -- GLOBAL_STEP: 593000 | > loss_text_ce: 0.03703237697482109 (0.03589702415865392) | > loss_mel_ce: 3.11246919631958 (nan) | > loss: 0.012498022988438606 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.1362770339810703) | > loader_time: 0.0048 (0.004707253960720517)  --> TIME: 2025-02-25 02:44:26 -- STEP: 2042/36938 -- GLOBAL_STEP: 593050 | > loss_text_ce: 0.03917150944471359 (0.03589293528776925) | > loss_mel_ce: 3.420612335205078 (nan) | > loss: 0.013729301281273365 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13638511799224778) | > loader_time: 0.0048 (0.004705789273679559)  --> TIME: 2025-02-25 02:44:35 -- STEP: 2092/36938 -- GLOBAL_STEP: 593100 | > loss_text_ce: 0.026884105056524277 (0.035856138977594314) | > loss_mel_ce: 2.3599023818969727 (nan) | > loss: 0.009471375495195389 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13636476314318552) | > loader_time: 0.0047 (0.004704215558491529)  --> TIME: 2025-02-25 02:44:44 -- STEP: 2142/36938 -- GLOBAL_STEP: 593150 | > loss_text_ce: 0.03326212987303734 (0.03589312695132013) | > loss_mel_ce: 3.019381284713745 (nan) | > loss: 0.012113664299249649 (nan) | > current_lr: 5e-06 | > step_time: 0.1086 (0.1362120514156436) | > loader_time: 0.0053 (0.004702931581313768)  --> TIME: 2025-02-25 02:44:53 -- STEP: 2192/36938 -- GLOBAL_STEP: 593200 | > loss_text_ce: 0.037164654582738876 (0.03594161984876113) | > loss_mel_ce: 2.748913288116455 (nan) | > loss: 0.011055865325033665 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.1362195096529312) | > loader_time: 0.0046 (0.004701672461781198)  --> TIME: 2025-02-25 02:45:03 -- STEP: 2242/36938 -- GLOBAL_STEP: 593250 | > loss_text_ce: 0.023252539336681366 (0.03593623028888664) | > loss_mel_ce: 2.1834676265716553 (nan) | > loss: 0.008756826631724834 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13640937435105896) | > loader_time: 0.0046 (0.00470079948176879)  --> TIME: 2025-02-25 02:45:13 -- STEP: 2292/36938 -- GLOBAL_STEP: 593300 | > loss_text_ce: 0.03888951614499092 (0.035945780117334535) | > loss_mel_ce: 3.11637544631958 (nan) | > loss: 0.012520892545580864 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.13645985374067876) | > loader_time: 0.0048 (0.004699821663568583)  --> TIME: 2025-02-25 02:45:22 -- STEP: 2342/36938 -- GLOBAL_STEP: 593350 | > loss_text_ce: 0.035374198108911514 (0.035927652528892864) | > loss_mel_ce: 0.2039794921875 (nan) | > loss: 0.0009498162544332445 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13660427518220758) | > loader_time: 0.0047 (0.00469905570671154)  --> TIME: 2025-02-25 02:45:31 -- STEP: 2392/36938 -- GLOBAL_STEP: 593400 | > loss_text_ce: 0.03473825752735138 (0.035907906510892124) | > loss_mel_ce: 3.516295909881592 (nan) | > loss: 0.014091406017541885 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13650685588651654) | > loader_time: 0.0046 (0.004697326234351841)  --> TIME: 2025-02-25 02:45:40 -- STEP: 2442/36938 -- GLOBAL_STEP: 593450 | > loss_text_ce: 0.04141790419816971 (0.03593507922467313) | > loss_mel_ce: 2.8515031337738037 (nan) | > loss: 0.011479846201837063 (nan) | > current_lr: 5e-06 | > step_time: 0.1353 (0.13640611040894832) | > loader_time: 0.0048 (0.0046955529633943135)  --> TIME: 2025-02-25 02:45:49 -- STEP: 2492/36938 -- GLOBAL_STEP: 593500 | > loss_text_ce: 0.032941367477178574 (0.035940517974497224) | > loss_mel_ce: 2.7445058822631836 (nan) | > loss: 0.011021616868674755 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13627381338162348) | > loader_time: 0.0046 (0.004694524680056514)  --> TIME: 2025-02-25 02:45:59 -- STEP: 2542/36938 -- GLOBAL_STEP: 593550 | > loss_text_ce: 0.060928720980882645 (0.03594468920407369) | > loss_mel_ce: 2.5737662315368652 (nan) | > loss: 0.01045513991266489 (nan) | > current_lr: 5e-06 | > step_time: 0.1071 (0.13628329028302885) | > loader_time: 0.0045 (0.004694739352240525)  --> TIME: 2025-02-25 02:46:08 -- STEP: 2592/36938 -- GLOBAL_STEP: 593600 | > loss_text_ce: 0.040785662829875946 (0.03592304860128168) | > loss_mel_ce: 3.0819194316864014 (nan) | > loss: 0.012391687370836735 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13629643122355117) | > loader_time: 0.0045 (0.004694020674552456)  --> TIME: 2025-02-25 02:46:18 -- STEP: 2642/36938 -- GLOBAL_STEP: 593650 | > loss_text_ce: 0.03178203105926514 (0.03593289903573048) | > loss_mel_ce: 2.9704864025115967 (nan) | > loss: 0.011913764290511608 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.1363857986347678) | > loader_time: 0.0045 (0.004693493168070685)  --> TIME: 2025-02-25 02:46:27 -- STEP: 2692/36938 -- GLOBAL_STEP: 593700 | > loss_text_ce: 0.02737477794289589 (0.03590666847393453) | > loss_mel_ce: 2.964811086654663 (nan) | > loss: 0.011873753741383553 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13644311208427362) | > loader_time: 0.0046 (0.004692732490720944)  --> TIME: 2025-02-25 02:46:36 -- STEP: 2742/36938 -- GLOBAL_STEP: 593750 | > loss_text_ce: 0.045848045498132706 (0.03588925133120293) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13641406818517843) | > loader_time: 0.0046 (0.004692131372197201)  --> TIME: 2025-02-25 02:46:46 -- STEP: 2792/36938 -- GLOBAL_STEP: 593800 | > loss_text_ce: 0.03152069076895714 (0.035893216306533055) | > loss_mel_ce: 2.9621500968933105 (nan) | > loss: 0.011879646219313145 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1363577385857314) | > loader_time: 0.0046 (0.004690849678564217)  --> TIME: 2025-02-25 02:46:55 -- STEP: 2842/36938 -- GLOBAL_STEP: 593850 | > loss_text_ce: 0.041135381907224655 (0.03591072602055956) | > loss_mel_ce: 3.055285692214966 (nan) | > loss: 0.012287385761737823 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.13636652606209732) | > loader_time: 0.0048 (0.004690002589725769)  --> TIME: 2025-02-25 02:47:04 -- STEP: 2892/36938 -- GLOBAL_STEP: 593900 | > loss_text_ce: 0.047400880604982376 (0.03589707902660986) | > loss_mel_ce: 2.056422233581543 (nan) | > loss: 0.008348505012691021 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1363805386535357) | > loader_time: 0.0047 (0.004689169704996544)  --> TIME: 2025-02-25 02:47:14 -- STEP: 2942/36938 -- GLOBAL_STEP: 593950 | > loss_text_ce: 0.04019724205136299 (0.03589666620394514) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13642628447366015) | > loader_time: 0.0047 (0.004689578212579854)  --> TIME: 2025-02-25 02:47:23 -- STEP: 2992/36938 -- GLOBAL_STEP: 594000 | > loss_text_ce: 0.04324912279844284 (0.03590711740867776) | > loss_mel_ce: 3.290147542953491 (nan) | > loss: 0.013227765448391438 (nan) | > current_lr: 5e-06 | > step_time: 0.1918 (0.136466827462701) | > loader_time: 0.005 (0.004689024651751804)  --> TIME: 2025-02-25 02:47:32 -- STEP: 3042/36938 -- GLOBAL_STEP: 594050 | > loss_text_ce: 0.02912304550409317 (0.035901068010227126) | > loss_mel_ce: 0.021542072296142578 (nan) | > loss: 0.0002010520693147555 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13647802725659605) | > loader_time: 0.0045 (0.004687645026212617)  --> TIME: 2025-02-25 02:47:42 -- STEP: 3092/36938 -- GLOBAL_STEP: 594100 | > loss_text_ce: 0.03345133364200592 (0.03588371134102862) | > loss_mel_ce: 3.3052401542663574 (nan) | > loss: 0.013248776085674763 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13650289829458925) | > loader_time: 0.0046 (0.0046865638600900915)  --> TIME: 2025-02-25 02:47:51 -- STEP: 3142/36938 -- GLOBAL_STEP: 594150 | > loss_text_ce: 0.031127478927373886 (0.03586966879067088) | > loss_mel_ce: 3.1720879077911377 (nan) | > loss: 0.012711172923445702 (nan) | > current_lr: 5e-06 | > step_time: 0.1659 (0.1364968290456458) | > loader_time: 0.0047 (0.004685353731972484)  --> TIME: 2025-02-25 02:48:01 -- STEP: 3192/36938 -- GLOBAL_STEP: 594200 | > loss_text_ce: 0.04013196751475334 (0.03587751422237294) | > loss_mel_ce: 3.02669095993042 (nan) | > loss: 0.012169932946562767 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.13652541643396027) | > loader_time: 0.0047 (0.004684577460277052)  --> TIME: 2025-02-25 02:48:10 -- STEP: 3242/36938 -- GLOBAL_STEP: 594250 | > loss_text_ce: 0.03154297173023224 (0.0359010730041586) | > loss_mel_ce: 1.9639543294906616 (nan) | > loss: 0.007918640971183777 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13648780240900762) | > loader_time: 0.0046 (0.004683262915025886)  --> TIME: 2025-02-25 02:48:19 -- STEP: 3292/36938 -- GLOBAL_STEP: 594300 | > loss_text_ce: 0.039735402911901474 (0.03589190956276404) | > loss_mel_ce: 2.949997663497925 (nan) | > loss: 0.011864020489156246 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.1363653473257292) | > loader_time: 0.0047 (0.004682196301938841)  --> TIME: 2025-02-25 02:48:28 -- STEP: 3342/36938 -- GLOBAL_STEP: 594350 | > loss_text_ce: 0.03385772556066513 (0.035884617920073046) | > loss_mel_ce: 2.8634915351867676 (nan) | > loss: 0.011497418396174908 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.1364272401976341) | > loader_time: 0.0048 (0.004682375241866873)  --> TIME: 2025-02-25 02:48:37 -- STEP: 3392/36938 -- GLOBAL_STEP: 594400 | > loss_text_ce: 0.03809360787272453 (0.0358991605366407) | > loss_mel_ce: 3.51051926612854 (nan) | > loss: 0.014081797562539577 (nan) | > current_lr: 5e-06 | > step_time: 0.1435 (0.13634070177685514) | > loader_time: 0.0046 (0.004681572880385057)  --> TIME: 2025-02-25 02:48:46 -- STEP: 3442/36938 -- GLOBAL_STEP: 594450 | > loss_text_ce: 0.03305454179644585 (0.03591358492893345) | > loss_mel_ce: 2.556009531021118 (nan) | > loss: 0.010274064727127552 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13632809776403648) | > loader_time: 0.0046 (0.0046823441047158625)  --> TIME: 2025-02-25 02:48:56 -- STEP: 3492/36938 -- GLOBAL_STEP: 594500 | > loss_text_ce: 0.042816076427698135 (0.03592967893048153) | > loss_mel_ce: 3.204315423965454 (nan) | > loss: 0.0128854438662529 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.1363283719393803) | > loader_time: 0.0046 (0.004681641361830842)  --> TIME: 2025-02-25 02:49:05 -- STEP: 3542/36938 -- GLOBAL_STEP: 594550 | > loss_text_ce: 0.026530645787715912 (0.03592961402699322) | > loss_mel_ce: 0.02097574807703495 (nan) | > loss: 0.00018851744243875146 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13627600286183253) | > loader_time: 0.0045 (0.004680705097420344)  --> TIME: 2025-02-25 02:49:14 -- STEP: 3592/36938 -- GLOBAL_STEP: 594600 | > loss_text_ce: 0.026963569223880768 (0.03592250128390708) | > loss_mel_ce: 3.186217784881592 (nan) | > loss: 0.012750719673931599 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13632663599897835) | > loader_time: 0.0046 (0.004680223082646494)  --> TIME: 2025-02-25 02:49:24 -- STEP: 3642/36938 -- GLOBAL_STEP: 594650 | > loss_text_ce: 0.0392424650490284 (0.035911787895588874) | > loss_mel_ce: 2.72286319732666 (nan) | > loss: 0.010960737243294716 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13637947368464684) | > loader_time: 0.0045 (0.004679589727172347)  --> TIME: 2025-02-25 02:49:33 -- STEP: 3692/36938 -- GLOBAL_STEP: 594700 | > loss_text_ce: 0.03310288488864899 (0.035922313565115695) | > loss_mel_ce: 3.539473533630371 (nan) | > loss: 0.01417689211666584 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13631458452954162) | > loader_time: 0.0066 (0.004681121941377384)  --> TIME: 2025-02-25 02:49:43 -- STEP: 3742/36938 -- GLOBAL_STEP: 594750 | > loss_text_ce: 0.04197448119521141 (0.03591469870825473) | > loss_mel_ce: 3.253990888595581 (nan) | > loss: 0.013079228810966015 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13637059726286901) | > loader_time: 0.0047 (0.004681412387442166)  --> TIME: 2025-02-25 02:49:52 -- STEP: 3792/36938 -- GLOBAL_STEP: 594800 | > loss_text_ce: 0.033992230892181396 (0.03591505121557566) | > loss_mel_ce: 3.4375 (nan) | > loss: 0.013775764033198357 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13635333272223704) | > loader_time: 0.0049 (0.004681328618073763)  --> TIME: 2025-02-25 02:50:01 -- STEP: 3842/36938 -- GLOBAL_STEP: 594850 | > loss_text_ce: 0.03128322958946228 (0.0359224750840223) | > loss_mel_ce: 2.687776565551758 (nan) | > loss: 0.010789919644594193 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13626736622809837) | > loader_time: 0.0046 (0.004680991048678824)  --> TIME: 2025-02-25 02:50:10 -- STEP: 3892/36938 -- GLOBAL_STEP: 594900 | > loss_text_ce: 0.037429049611091614 (0.03591328612571795) | > loss_mel_ce: 2.2264580726623535 (nan) | > loss: 0.008983680047094822 (nan) | > current_lr: 5e-06 | > step_time: 0.1582 (0.1362431091002788) | > loader_time: 0.0049 (0.004681019106724768)  --> TIME: 2025-02-25 02:50:19 -- STEP: 3942/36938 -- GLOBAL_STEP: 594950 | > loss_text_ce: 0.03499742969870567 (0.03590868386751347) | > loss_mel_ce: 3.0126657485961914 (nan) | > loss: 0.012093902565538883 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.1361804047192126) | > loader_time: 0.0046 (0.0046804840343365)  --> TIME: 2025-02-25 02:50:28 -- STEP: 3992/36938 -- GLOBAL_STEP: 595000 | > loss_text_ce: 0.03039628453552723 (0.03589521372785964) | > loss_mel_ce: 3.1376960277557373 (nan) | > loss: 0.012571794912219048 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.13617653777460748) | > loader_time: 0.0047 (0.004680716262790625)  --> TIME: 2025-02-25 02:50:38 -- STEP: 4042/36938 -- GLOBAL_STEP: 595050 | > loss_text_ce: 0.038934674113988876 (0.035913813827888205) | > loss_mel_ce: 3.1573526859283447 (nan) | > loss: 0.012683681212365627 (nan) | > current_lr: 5e-06 | > step_time: 0.1776 (0.13620439807357645) | > loader_time: 0.0049 (0.0046804000221226)  --> TIME: 2025-02-25 02:50:47 -- STEP: 4092/36938 -- GLOBAL_STEP: 595100 | > loss_text_ce: 0.038081470876932144 (0.03590605808567614) | > loss_mel_ce: 2.852970838546753 (nan) | > loss: 0.0114724300801754 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13612849708875924) | > loader_time: 0.0047 (0.004680077584491329)  --> TIME: 2025-02-25 02:50:56 -- STEP: 4142/36938 -- GLOBAL_STEP: 595150 | > loss_text_ce: 0.030542565509676933 (0.035900139304274416) | > loss_mel_ce: 2.215153932571411 (nan) | > loss: 0.008911495096981525 (nan) | > current_lr: 5e-06 | > step_time: 0.1853 (0.13611295308426663) | > loader_time: 0.0049 (0.004679814966792127)  --> TIME: 2025-02-25 02:51:05 -- STEP: 4192/36938 -- GLOBAL_STEP: 595200 | > loss_text_ce: 0.041330356150865555 (0.03589928825963485) | > loss_mel_ce: 1.0660836696624756 (nan) | > loss: 0.00439450005069375 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13613328895041024) | > loader_time: 0.0045 (0.004679582387436436)  --> TIME: 2025-02-25 02:51:14 -- STEP: 4242/36938 -- GLOBAL_STEP: 595250 | > loss_text_ce: 0.030729390680789948 (0.03589073566079671) | > loss_mel_ce: 1.7685750722885132 (nan) | > loss: 0.007140097673982382 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1360866760994002) | > loader_time: 0.0048 (0.004679360574059528)  --> TIME: 2025-02-25 02:51:24 -- STEP: 4292/36938 -- GLOBAL_STEP: 595300 | > loss_text_ce: 0.04011479765176773 (0.03590743582112435) | > loss_mel_ce: 1.3674606084823608 (nan) | > loss: 0.005585616920143366 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.136063564023678) | > loader_time: 0.0046 (0.004679180313689841)  --> TIME: 2025-02-25 02:51:33 -- STEP: 4342/36938 -- GLOBAL_STEP: 595350 | > loss_text_ce: 0.033596836030483246 (0.035897475499136244) | > loss_mel_ce: 3.218376874923706 (nan) | > loss: 0.012904657982289791 (nan) | > current_lr: 5e-06 | > step_time: 0.1679 (0.13603947218636192) | > loader_time: 0.0048 (0.004678685178629318)  --> TIME: 2025-02-25 02:51:42 -- STEP: 4392/36938 -- GLOBAL_STEP: 595400 | > loss_text_ce: 0.03521766886115074 (0.03588998993860374) | > loss_mel_ce: 3.000053882598877 (nan) | > loss: 0.01204472966492176 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13601834944471386) | > loader_time: 0.0046 (0.004678269933047404)  --> TIME: 2025-02-25 02:51:51 -- STEP: 4442/36938 -- GLOBAL_STEP: 595450 | > loss_text_ce: 0.038230136036872864 (0.035897678173977654) | > loss_mel_ce: 2.341912269592285 (nan) | > loss: 0.0094450106844306 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.1360094349215769) | > loader_time: 0.0046 (0.004677746543901447)  --> TIME: 2025-02-25 02:52:00 -- STEP: 4492/36938 -- GLOBAL_STEP: 595500 | > loss_text_ce: 0.03141885623335838 (0.03591413283596199) | > loss_mel_ce: 2.888328790664673 (nan) | > loss: 0.011586301028728485 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.1360337980496276) | > loader_time: 0.0045 (0.004678353229380032)  --> TIME: 2025-02-25 02:52:10 -- STEP: 4542/36938 -- GLOBAL_STEP: 595550 | > loss_text_ce: 0.046585943549871445 (0.03591802426154957) | > loss_mel_ce: 2.9313693046569824 (nan) | > loss: 0.011817283928394318 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13600097958183854) | > loader_time: 0.0046 (0.004679052801396757)  --> TIME: 2025-02-25 02:52:19 -- STEP: 4592/36938 -- GLOBAL_STEP: 595600 | > loss_text_ce: 0.03624800220131874 (0.035920777332154374) | > loss_mel_ce: 3.3440613746643066 (nan) | > loss: 0.013413926586508751 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13603287779495665) | > loader_time: 0.0045 (0.004678654078822513)  --> TIME: 2025-02-25 02:52:28 -- STEP: 4642/36938 -- GLOBAL_STEP: 595650 | > loss_text_ce: 0.0317775160074234 (0.03591840428118419) | > loss_mel_ce: 2.609245777130127 (nan) | > loss: 0.010480252094566822 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13600531979297456) | > loader_time: 0.0046 (0.004677947817666417)  --> TIME: 2025-02-25 02:52:37 -- STEP: 4692/36938 -- GLOBAL_STEP: 595700 | > loss_text_ce: 0.03819562494754791 (0.03590610354324155) | > loss_mel_ce: 3.5488393306732178 (nan) | > loss: 0.014234266243875027 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13597767107118824) | > loader_time: 0.0048 (0.004678010788110202)  --> TIME: 2025-02-25 02:52:46 -- STEP: 4742/36938 -- GLOBAL_STEP: 595750 | > loss_text_ce: 0.03971659764647484 (0.035898453180093964) | > loss_mel_ce: 3.3643572330474854 (nan) | > loss: 0.01350822951644659 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13594104523700973) | > loader_time: 0.0046 (0.004677674680860265)  --> TIME: 2025-02-25 02:52:56 -- STEP: 4792/36938 -- GLOBAL_STEP: 595800 | > loss_text_ce: 0.033540017902851105 (0.035907965277839224) | > loss_mel_ce: 2.1686768531799316 (nan) | > loss: 0.008738956414163113 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13598457655444957) | > loader_time: 0.0046 (0.004677176475524901)  --> TIME: 2025-02-25 02:53:05 -- STEP: 4842/36938 -- GLOBAL_STEP: 595850 | > loss_text_ce: 0.04568488895893097 (0.03592126572851609) | > loss_mel_ce: 2.7527430057525635 (nan) | > loss: 0.01110487338155508 (nan) | > current_lr: 5e-06 | > step_time: 0.1078 (0.1360093610240427) | > loader_time: 0.0044 (0.004676709535544392)  --> TIME: 2025-02-25 02:53:14 -- STEP: 4892/36938 -- GLOBAL_STEP: 595900 | > loss_text_ce: 0.040332596749067307 (0.03592676255708801) | > loss_mel_ce: 3.6174979209899902 (nan) | > loss: 0.014515201561152935 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13600394117569015) | > loader_time: 0.0046 (0.004676430662050658)  --> TIME: 2025-02-25 02:53:24 -- STEP: 4942/36938 -- GLOBAL_STEP: 595950 | > loss_text_ce: 0.04401012882590294 (0.035937197923657085) | > loss_mel_ce: 3.5772340297698975 (nan) | > loss: 0.014370017684996128 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13599453368972347) | > loader_time: 0.0046 (0.004676889910093926)  --> TIME: 2025-02-25 02:53:33 -- STEP: 4992/36938 -- GLOBAL_STEP: 596000 | > loss_text_ce: 0.04196321964263916 (0.035934325292309305) | > loss_mel_ce: 3.345366954803467 (nan) | > loss: 0.013441787101328373 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13598783533924666) | > loader_time: 0.0046 (0.004677056502073234)  --> TIME: 2025-02-25 02:53:42 -- STEP: 5042/36938 -- GLOBAL_STEP: 596050 | > loss_text_ce: 0.031574275344610214 (0.03595664705829925) | > loss_mel_ce: 2.4613258838653564 (nan) | > loss: 0.009892461821436882 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13598300057335158) | > loader_time: 0.0046 (0.004677746325246585)  --> TIME: 2025-02-25 02:53:51 -- STEP: 5092/36938 -- GLOBAL_STEP: 596100 | > loss_text_ce: 0.04126632213592529 (0.035955367838482286) | > loss_mel_ce: 3.0326333045959473 (nan) | > loss: 0.012198015116155148 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.13598882253022054) | > loader_time: 0.0046 (0.00467769896431598)  --> TIME: 2025-02-25 02:54:01 -- STEP: 5142/36938 -- GLOBAL_STEP: 596150 | > loss_text_ce: 0.028787067160010338 (0.03595350225550587) | > loss_mel_ce: 3.2076523303985596 (nan) | > loss: 0.012843014672398567 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13598884250903484) | > loader_time: 0.0049 (0.004677628552760651)  --> TIME: 2025-02-25 02:54:10 -- STEP: 5192/36938 -- GLOBAL_STEP: 596200 | > loss_text_ce: 0.03148963674902916 (0.035963858319919835) | > loss_mel_ce: 2.4233310222625732 (nan) | > loss: 0.009741351939737797 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13599424268505048) | > loader_time: 0.0046 (0.0046775488897538115)  --> TIME: 2025-02-25 02:54:20 -- STEP: 5242/36938 -- GLOBAL_STEP: 596250 | > loss_text_ce: 0.03936738893389702 (0.03597058499184298) | > loss_mel_ce: 2.780143976211548 (nan) | > loss: 0.011188537813723087 (nan) | > current_lr: 5e-06 | > step_time: 0.145 (0.13603948541653607) | > loader_time: 0.0046 (0.004677193304001486)  --> TIME: 2025-02-25 02:54:29 -- STEP: 5292/36938 -- GLOBAL_STEP: 596300 | > loss_text_ce: 0.03242345154285431 (0.03598393865023986) | > loss_mel_ce: 3.2183876037597656 (nan) | > loss: 0.01290004514157772 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13603356397251068) | > loader_time: 0.0046 (0.004676910980308229)  --> TIME: 2025-02-25 02:54:38 -- STEP: 5342/36938 -- GLOBAL_STEP: 596350 | > loss_text_ce: 0.045071858912706375 (0.035983424052143984) | > loss_mel_ce: 2.699162721633911 (nan) | > loss: 0.010889820754528046 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13603062024503912) | > loader_time: 0.0046 (0.004677470548908973)  --> TIME: 2025-02-25 02:54:48 -- STEP: 5392/36938 -- GLOBAL_STEP: 596400 | > loss_text_ce: 0.03690853714942932 (0.03598016016053992) | > loss_mel_ce: 2.16675066947937 (nan) | > loss: 0.0087446803227067 (nan) | > current_lr: 5e-06 | > step_time: 0.1907 (0.13607002666513007) | > loader_time: 0.0048 (0.004677303245937786)  --> TIME: 2025-02-25 02:54:57 -- STEP: 5442/36938 -- GLOBAL_STEP: 596450 | > loss_text_ce: 0.051037199795246124 (0.03597729214498233) | > loss_mel_ce: 0.3449028730392456 (nan) | > loss: 0.0015711907763034105 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13604138704487098) | > loader_time: 0.0045 (0.0046765550626020795)  --> TIME: 2025-02-25 02:55:06 -- STEP: 5492/36938 -- GLOBAL_STEP: 596500 | > loss_text_ce: 0.044433388859033585 (0.035968235373538854) | > loss_mel_ce: 2.527252197265625 (nan) | > loss: 0.010205102153122425 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13608569796729608) | > loader_time: 0.0047 (0.004676959415638611)  --> TIME: 2025-02-25 02:55:16 -- STEP: 5542/36938 -- GLOBAL_STEP: 596550 | > loss_text_ce: 0.033038169145584106 (0.03595358056635671) | > loss_mel_ce: 2.1269147396087646 (nan) | > loss: 0.008571241982281208 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.13610734912902617) | > loader_time: 0.0046 (0.004677077399742211)  --> TIME: 2025-02-25 02:55:25 -- STEP: 5592/36938 -- GLOBAL_STEP: 596600 | > loss_text_ce: 0.036334410309791565 (0.03596809490877603) | > loss_mel_ce: 2.9634318351745605 (nan) | > loss: 0.011903835460543633 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.13607994775062648) | > loader_time: 0.0047 (0.004677344161553447)  --> TIME: 2025-02-25 02:55:34 -- STEP: 5642/36938 -- GLOBAL_STEP: 596650 | > loss_text_ce: 0.03380665183067322 (0.03595422651537486) | > loss_mel_ce: 2.4034435749053955 (nan) | > loss: 0.009671628475189209 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13603289754455802) | > loader_time: 0.0046 (0.004676737670398108)  --> TIME: 2025-02-25 02:55:44 -- STEP: 5692/36938 -- GLOBAL_STEP: 596700 | > loss_text_ce: 0.03423485904932022 (0.03595330433488705) | > loss_mel_ce: 0.3135742247104645 (nan) | > loss: 0.0013801947934553027 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13604307928641599) | > loader_time: 0.0045 (0.004676333591289617)  --> TIME: 2025-02-25 02:55:53 -- STEP: 5742/36938 -- GLOBAL_STEP: 596750 | > loss_text_ce: 0.04309031739830971 (0.03596149292735226) | > loss_mel_ce: 2.918605327606201 (nan) | > loss: 0.011752760969102383 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.1360821844766632) | > loader_time: 0.0046 (0.004676084740989845)  --> TIME: 2025-02-25 02:56:02 -- STEP: 5792/36938 -- GLOBAL_STEP: 596800 | > loss_text_ce: 0.0442512184381485 (0.03595203446521458) | > loss_mel_ce: 3.6090734004974365 (nan) | > loss: 0.014497321099042892 (nan) | > current_lr: 5e-06 | > step_time: 0.1519 (0.13605479760229272) | > loader_time: 0.0046 (0.004675596252660069)  --> TIME: 2025-02-25 02:56:12 -- STEP: 5842/36938 -- GLOBAL_STEP: 596850 | > loss_text_ce: 0.034174155443906784 (0.035943222856923654) | > loss_mel_ce: 3.0893428325653076 (nan) | > loss: 0.012394909746944904 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.1360513703621483) | > loader_time: 0.0046 (0.004675124533084192)  --> TIME: 2025-02-25 02:56:21 -- STEP: 5892/36938 -- GLOBAL_STEP: 596900 | > loss_text_ce: 0.034545984119176865 (0.035932067579204544) | > loss_mel_ce: 2.821561813354492 (nan) | > loss: 0.011333761736750603 (nan) | > current_lr: 5e-06 | > step_time: 0.1571 (0.13602856208253544) | > loader_time: 0.0047 (0.0046745112617244105)  --> TIME: 2025-02-25 02:56:30 -- STEP: 5942/36938 -- GLOBAL_STEP: 596950 | > loss_text_ce: 0.03370798006653786 (0.035932448587450046) | > loss_mel_ce: 2.6661736965179443 (nan) | > loss: 0.010713817551732063 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13604025919402907) | > loader_time: 0.0046 (0.004674126667753293)  --> TIME: 2025-02-25 02:56:39 -- STEP: 5992/36938 -- GLOBAL_STEP: 597000 | > loss_text_ce: 0.0252508744597435 (0.035922375778873826) | > loss_mel_ce: 2.1634981632232666 (nan) | > loss: 0.008685512468218803 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13605678698249432) | > loader_time: 0.0046 (0.004674164849384449)  --> TIME: 2025-02-25 02:56:49 -- STEP: 6042/36938 -- GLOBAL_STEP: 597050 | > loss_text_ce: 0.03539586812257767 (0.03591810287235639) | > loss_mel_ce: 3.181997060775757 (nan) | > loss: 0.012767433188855648 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.1360598315627557) | > loader_time: 0.0048 (0.004673899581439447)  --> TIME: 2025-02-25 02:56:58 -- STEP: 6092/36938 -- GLOBAL_STEP: 597100 | > loss_text_ce: 0.03606894984841347 (0.03592192904992483) | > loss_mel_ce: 3.6808886528015137 (nan) | > loss: 0.014749832451343536 (nan) | > current_lr: 5e-06 | > step_time: 0.1807 (0.13607518393509654) | > loader_time: 0.0049 (0.004673539183285385)  --> TIME: 2025-02-25 02:57:07 -- STEP: 6142/36938 -- GLOBAL_STEP: 597150 | > loss_text_ce: 0.05466565862298012 (0.03593636338029778) | > loss_mel_ce: 3.1393229961395264 (nan) | > loss: 0.012674558907747269 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13604171650295566) | > loader_time: 0.0045 (0.0046729713105798255)  --> TIME: 2025-02-25 02:57:16 -- STEP: 6192/36938 -- GLOBAL_STEP: 597200 | > loss_text_ce: 0.04212675988674164 (0.03593483110736353) | > loss_mel_ce: 3.3963587284088135 (nan) | > loss: 0.01364478375762701 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13600334193971408) | > loader_time: 0.0047 (0.004672404869582301)  --> TIME: 2025-02-25 02:57:26 -- STEP: 6242/36938 -- GLOBAL_STEP: 597250 | > loss_text_ce: 0.03532997518777847 (0.0359374608965967) | > loss_mel_ce: 2.916583776473999 (nan) | > loss: 0.011713944375514984 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13599953862269362) | > loader_time: 0.0046 (0.004671942610956088)  --> TIME: 2025-02-25 02:57:35 -- STEP: 6292/36938 -- GLOBAL_STEP: 597300 | > loss_text_ce: 0.03223409876227379 (0.03594270004690711) | > loss_mel_ce: 2.983341932296753 (nan) | > loss: 0.01196657307446003 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13601793696451364) | > loader_time: 0.0046 (0.004672210078594117)  --> TIME: 2025-02-25 02:57:44 -- STEP: 6342/36938 -- GLOBAL_STEP: 597350 | > loss_text_ce: 0.04004805162549019 (0.03593635413157177) | > loss_mel_ce: 1.7072887420654297 (nan) | > loss: 0.006933876313269138 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13601624852886912) | > loader_time: 0.0046 (0.004671934048710696)  --> TIME: 2025-02-25 02:57:54 -- STEP: 6392/36938 -- GLOBAL_STEP: 597400 | > loss_text_ce: 0.025197790935635567 (0.035921956372717424) | > loss_mel_ce: 1.4311606884002686 (nan) | > loss: 0.005779200233519077 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.136039518071653) | > loader_time: 0.0046 (0.00467193611572324)  --> TIME: 2025-02-25 02:58:03 -- STEP: 6442/36938 -- GLOBAL_STEP: 597450 | > loss_text_ce: 0.026095042005181313 (0.03591962876659952) | > loss_mel_ce: 2.5443198680877686 (nan) | > loss: 0.010200059972703457 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.1360419041545667) | > loader_time: 0.0046 (0.004671610167810401)  --> TIME: 2025-02-25 02:58:13 -- STEP: 6492/36938 -- GLOBAL_STEP: 597500 | > loss_text_ce: 0.02591750957071781 (0.035915379688483066) | > loss_mel_ce: 2.13470196723938 (nan) | > loss: 0.008573886938393116 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13606153019989087) | > loader_time: 0.0047 (0.004671973427063298)  --> TIME: 2025-02-25 02:58:22 -- STEP: 6542/36938 -- GLOBAL_STEP: 597550 | > loss_text_ce: 0.030311452224850655 (0.03591449892968902) | > loss_mel_ce: 2.9196364879608154 (nan) | > loss: 0.01170614268630743 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13608622798785536) | > loader_time: 0.0046 (0.004671659939007023)  --> TIME: 2025-02-25 02:58:31 -- STEP: 6592/36938 -- GLOBAL_STEP: 597600 | > loss_text_ce: 0.037585318088531494 (0.03591350530860767) | > loss_mel_ce: 3.3785743713378906 (nan) | > loss: 0.013556189835071564 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13608811625722528) | > loader_time: 0.0047 (0.004671273337116525)  --> TIME: 2025-02-25 02:58:40 -- STEP: 6642/36938 -- GLOBAL_STEP: 597650 | > loss_text_ce: 0.045353859663009644 (0.03592024728020869) | > loss_mel_ce: 3.1868810653686523 (nan) | > loss: 0.012826330028474331 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13605025244634997) | > loader_time: 0.0045 (0.004670890258472038)  --> TIME: 2025-02-25 02:58:49 -- STEP: 6692/36938 -- GLOBAL_STEP: 597700 | > loss_text_ce: 0.042758990079164505 (0.03592931167642808) | > loss_mel_ce: 2.1870944499969482 (nan) | > loss: 0.008848625235259533 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13599907439836667) | > loader_time: 0.0046 (0.004670482015296225)  --> TIME: 2025-02-25 02:58:59 -- STEP: 6742/36938 -- GLOBAL_STEP: 597750 | > loss_text_ce: 0.03111284412443638 (0.035921067568561234) | > loss_mel_ce: 3.417679786682129 (nan) | > loss: 0.013685686513781548 (nan) | > current_lr: 5e-06 | > step_time: 0.1609 (0.13602883389285378) | > loader_time: 0.0046 (0.004670101257432862)  --> TIME: 2025-02-25 02:59:08 -- STEP: 6792/36938 -- GLOBAL_STEP: 597800 | > loss_text_ce: 0.018937906250357628 (0.035916318933756616) | > loss_mel_ce: 2.628939628601074 (nan) | > loss: 0.010507450439035892 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13600728561656347) | > loader_time: 0.0045 (0.004669766544032021)  --> TIME: 2025-02-25 02:59:17 -- STEP: 6842/36938 -- GLOBAL_STEP: 597850 | > loss_text_ce: 0.0301507655531168 (0.035918544556752405) | > loss_mel_ce: 2.5144283771514893 (nan) | > loss: 0.010097536258399487 (nan) | > current_lr: 5e-06 | > step_time: 0.1442 (0.13601648218622667) | > loader_time: 0.0048 (0.00466958861781715)  --> TIME: 2025-02-25 02:59:27 -- STEP: 6892/36938 -- GLOBAL_STEP: 597900 | > loss_text_ce: 0.03625719994306564 (0.035924679462949416) | > loss_mel_ce: 2.71395206451416 (nan) | > loss: 0.010913529433310032 (nan) | > current_lr: 5e-06 | > step_time: 0.1516 (0.13607162572724535) | > loader_time: 0.0046 (0.004669399193671313)  --> TIME: 2025-02-25 02:59:36 -- STEP: 6942/36938 -- GLOBAL_STEP: 597950 | > loss_text_ce: 0.03427889198064804 (0.035924582720420664) | > loss_mel_ce: 3.0324625968933105 (nan) | > loss: 0.012169609777629375 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13606598730287828) | > loader_time: 0.0048 (0.004669062722419614)  --> TIME: 2025-02-25 02:59:46 -- STEP: 6992/36938 -- GLOBAL_STEP: 598000 | > loss_text_ce: 0.03464076295495033 (0.03592420961133441) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13611543754827493) | > loader_time: 0.0046 (0.004669005967659603)  --> TIME: 2025-02-25 02:59:55 -- STEP: 7042/36938 -- GLOBAL_STEP: 598050 | > loss_text_ce: 0.024573395028710365 (0.03592938895146011) | > loss_mel_ce: 2.721876859664917 (nan) | > loss: 0.010898612439632416 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.1361444787998085) | > loader_time: 0.0046 (0.004669072376804858)  --> TIME: 2025-02-25 03:00:05 -- STEP: 7092/36938 -- GLOBAL_STEP: 598100 | > loss_text_ce: 0.02722736820578575 (0.03592344891890276) | > loss_mel_ce: 1.9784717559814453 (nan) | > loss: 0.007959123700857162 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13614157341458466) | > loader_time: 0.0045 (0.004669258907832348)  --> TIME: 2025-02-25 03:00:14 -- STEP: 7142/36938 -- GLOBAL_STEP: 598150 | > loss_text_ce: 0.022608017548918724 (0.035921313038115826) | > loss_mel_ce: 1.3129150867462158 (nan) | > loss: 0.005299695301800966 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.1361953197357467) | > loader_time: 0.0045 (0.004669594250617338)  --> TIME: 2025-02-25 03:00:24 -- STEP: 7192/36938 -- GLOBAL_STEP: 598200 | > loss_text_ce: 0.04049309343099594 (0.03592404679501927) | > loss_mel_ce: 2.7455036640167236 (nan) | > loss: 0.011055543087422848 (nan) | > current_lr: 5e-06 | > step_time: 0.1529 (0.13617841887792287) | > loader_time: 0.0047 (0.004669726789992178)  --> TIME: 2025-02-25 03:00:33 -- STEP: 7242/36938 -- GLOBAL_STEP: 598250 | > loss_text_ce: 0.031976696103811264 (0.035919289897588906) | > loss_mel_ce: 3.155538320541382 (nan) | > loss: 0.01264887023717165 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13615705775345455) | > loader_time: 0.0046 (0.00466940248911305)  --> TIME: 2025-02-25 03:00:42 -- STEP: 7292/36938 -- GLOBAL_STEP: 598300 | > loss_text_ce: 0.028366154059767723 (0.03591537660905302) | > loss_mel_ce: 2.4694344997406006 (nan) | > loss: 0.009911907836794853 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.1361649095587068) | > loader_time: 0.0046 (0.00466943114129516)  --> TIME: 2025-02-25 03:00:52 -- STEP: 7342/36938 -- GLOBAL_STEP: 598350 | > loss_text_ce: 0.02715218812227249 (0.035914663729762146) | > loss_mel_ce: 3.733266830444336 (nan) | > loss: 0.014922299422323704 (nan) | > current_lr: 5e-06 | > step_time: 0.1578 (0.1362027038965456) | > loader_time: 0.0047 (0.004669621672041907)  --> TIME: 2025-02-25 03:01:01 -- STEP: 7392/36938 -- GLOBAL_STEP: 598400 | > loss_text_ce: 0.03431900963187218 (0.0359072240527607) | > loss_mel_ce: 2.8624417781829834 (nan) | > loss: 0.011495082639157772 (nan) | > current_lr: 5e-06 | > step_time: 0.1864 (0.13621032637951166) | > loader_time: 0.005 (0.004669331046409949)  --> TIME: 2025-02-25 03:01:11 -- STEP: 7442/36938 -- GLOBAL_STEP: 598450 | > loss_text_ce: 0.027751760557293892 (0.035904331619329606) | > loss_mel_ce: 2.8416388034820557 (nan) | > loss: 0.01138647086918354 (nan) | > current_lr: 5e-06 | > step_time: 0.1456 (0.1362324755091207) | > loader_time: 0.0046 (0.004669224052972557)  --> TIME: 2025-02-25 03:01:20 -- STEP: 7492/36938 -- GLOBAL_STEP: 598500 | > loss_text_ce: 0.02773153968155384 (0.03591131509763483) | > loss_mel_ce: 1.3677096366882324 (nan) | > loss: 0.005537465214729309 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.1362235331624444) | > loader_time: 0.0045 (0.004668793669373239)  --> TIME: 2025-02-25 03:01:30 -- STEP: 7542/36938 -- GLOBAL_STEP: 598550 | > loss_text_ce: 0.03870126232504845 (0.035910798005810755) | > loss_mel_ce: 0.013458251953125 (nan) | > loss: 0.00020698220760095865 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13626258384640863) | > loader_time: 0.0045 (0.004668560308791775)  --> TIME: 2025-02-25 03:01:39 -- STEP: 7592/36938 -- GLOBAL_STEP: 598600 | > loss_text_ce: 0.03903799131512642 (0.03591798481558887) | > loss_mel_ce: 1.752753734588623 (nan) | > loss: 0.007110284641385078 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1362478464873247) | > loader_time: 0.0045 (0.004668219857019933)  --> TIME: 2025-02-25 03:01:48 -- STEP: 7642/36938 -- GLOBAL_STEP: 598650 | > loss_text_ce: 0.027672437950968742 (0.03591480971372721) | > loss_mel_ce: 3.517045497894287 (nan) | > loss: 0.014066342264413834 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.1362356556488313) | > loader_time: 0.0044 (0.00466758750615025)  --> TIME: 2025-02-25 03:01:58 -- STEP: 7692/36938 -- GLOBAL_STEP: 598700 | > loss_text_ce: 0.035681892186403275 (0.035923520667435496) | > loss_mel_ce: 3.4984114170074463 (nan) | > loss: 0.014024181291460991 (nan) | > current_lr: 5e-06 | > step_time: 0.1934 (0.1362627132200538) | > loader_time: 0.0048 (0.004667589426412609)  --> TIME: 2025-02-25 03:02:07 -- STEP: 7742/36938 -- GLOBAL_STEP: 598750 | > loss_text_ce: 0.030738448724150658 (0.035926687294071794) | > loss_mel_ce: 0.5529087781906128 (nan) | > loss: 0.002316060708835721 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.1363016649353138) | > loader_time: 0.0046 (0.004668280247476679)  --> TIME: 2025-02-25 03:02:16 -- STEP: 7792/36938 -- GLOBAL_STEP: 598800 | > loss_text_ce: 0.04058169573545456 (0.03593307667499322) | > loss_mel_ce: 3.7347147464752197 (nan) | > loss: 0.014981335960328579 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13625526256874604) | > loader_time: 0.0046 (0.004667812181938857)  --> TIME: 2025-02-25 03:02:26 -- STEP: 7842/36938 -- GLOBAL_STEP: 598850 | > loss_text_ce: 0.03805375099182129 (0.03592544940075321) | > loss_mel_ce: 2.479759693145752 (nan) | > loss: 0.009991323575377464 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13627166174650499) | > loader_time: 0.0048 (0.004667617994375126)  --> TIME: 2025-02-25 03:02:35 -- STEP: 7892/36938 -- GLOBAL_STEP: 598900 | > loss_text_ce: 0.03451831638813019 (0.03593194561616759) | > loss_mel_ce: 2.515331506729126 (nan) | > loss: 0.010118451900780201 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13628427639747115) | > loader_time: 0.0045 (0.004667294460447452)  --> TIME: 2025-02-25 03:02:44 -- STEP: 7942/36938 -- GLOBAL_STEP: 598950 | > loss_text_ce: 0.03476109355688095 (0.03592973345683284) | > loss_mel_ce: 2.889050245285034 (nan) | > loss: 0.011602426879107952 (nan) | > current_lr: 5e-06 | > step_time: 0.195 (0.13627194529725276) | > loader_time: 0.0049 (0.004666873142479721)  --> TIME: 2025-02-25 03:02:54 -- STEP: 7992/36938 -- GLOBAL_STEP: 599000 | > loss_text_ce: 0.0402575246989727 (0.035932178938275335) | > loss_mel_ce: 2.723175048828125 (nan) | > loss: 0.010966002941131592 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1362949159589258) | > loader_time: 0.0045 (0.004667095832519237)  --> TIME: 2025-02-25 03:03:03 -- STEP: 8042/36938 -- GLOBAL_STEP: 599050 | > loss_text_ce: 0.031843531876802444 (0.035925652299194595) | > loss_mel_ce: 2.829848527908325 (nan) | > loss: 0.011355920694768429 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13629096658987006) | > loader_time: 0.0047 (0.004667033131695968)  --> TIME: 2025-02-25 03:03:13 -- STEP: 8092/36938 -- GLOBAL_STEP: 599100 | > loss_text_ce: 0.034653257578611374 (0.035912526082371485) | > loss_mel_ce: 3.456944227218628 (nan) | > loss: 0.013855545781552792 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13631128481102653) | > loader_time: 0.0046 (0.004666947075122085)  --> TIME: 2025-02-25 03:03:22 -- STEP: 8142/36938 -- GLOBAL_STEP: 599150 | > loss_text_ce: 0.03319917991757393 (0.03590515027146594) | > loss_mel_ce: 3.53324556350708 (nan) | > loss: 0.014152558520436287 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13629939810294422) | > loader_time: 0.0046 (0.0046668316802004105)  --> TIME: 2025-02-25 03:03:31 -- STEP: 8192/36938 -- GLOBAL_STEP: 599200 | > loss_text_ce: 0.03243127092719078 (0.035903707766123866) | > loss_mel_ce: 2.901237726211548 (nan) | > loss: 0.011641544289886951 (nan) | > current_lr: 5e-06 | > step_time: 0.1449 (0.1362794381566346) | > loader_time: 0.0047 (0.00466668128501625)  --> TIME: 2025-02-25 03:03:40 -- STEP: 8242/36938 -- GLOBAL_STEP: 599250 | > loss_text_ce: 0.03655796870589256 (0.0358941220201312) | > loss_mel_ce: 3.3836658000946045 (nan) | > loss: 0.01357231754809618 (nan) | > current_lr: 5e-06 | > step_time: 0.157 (0.1362923631040948) | > loader_time: 0.0046 (0.00466688397844219)  --> TIME: 2025-02-25 03:03:50 -- STEP: 8292/36938 -- GLOBAL_STEP: 599300 | > loss_text_ce: 0.038311924785375595 (0.03589528469544896) | > loss_mel_ce: 3.6155169010162354 (nan) | > loss: 0.014499321579933167 (nan) | > current_lr: 5e-06 | > step_time: 0.1866 (0.13631179237273716) | > loader_time: 0.005 (0.004666834969021432)  --> TIME: 2025-02-25 03:03:59 -- STEP: 8342/36938 -- GLOBAL_STEP: 599350 | > loss_text_ce: 0.03847534582018852 (0.03588763653188807) | > loss_mel_ce: 3.640899658203125 (nan) | > loss: 0.014600695110857487 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.1362919722835785) | > loader_time: 0.0044 (0.004666267182151936)  --> TIME: 2025-02-25 03:04:08 -- STEP: 8392/36938 -- GLOBAL_STEP: 599400 | > loss_text_ce: 0.031282804906368256 (0.035894975775330455) | > loss_mel_ce: 2.25551438331604 (nan) | > loss: 0.009074592962861061 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13628029050545196) | > loader_time: 0.0046 (0.004666026088598242)  --> TIME: 2025-02-25 03:04:18 -- STEP: 8442/36938 -- GLOBAL_STEP: 599450 | > loss_text_ce: 0.0327158160507679 (0.03589552531168414) | > loss_mel_ce: 0.0228271484375 (nan) | > loss: 0.00022040860494598746 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1362940095889289) | > loader_time: 0.0045 (0.004665774831385513)  --> TIME: 2025-02-25 03:04:27 -- STEP: 8492/36938 -- GLOBAL_STEP: 599500 | > loss_text_ce: 0.036624498665332794 (0.035898573739384515) | > loss_mel_ce: 3.5397727489471436 (nan) | > loss: 0.014192053116858006 (nan) | > current_lr: 5e-06 | > step_time: 0.1121 (0.13631761495201195) | > loader_time: 0.0045 (0.004665525943335653)  --> TIME: 2025-02-25 03:04:36 -- STEP: 8542/36938 -- GLOBAL_STEP: 599550 | > loss_text_ce: 0.034603748470544815 (0.035899445000888215) | > loss_mel_ce: 3.172879457473755 (nan) | > loss: 0.012728109024465084 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.1363075172222909) | > loader_time: 0.0045 (0.004664985057872801)  --> TIME: 2025-02-25 03:04:46 -- STEP: 8592/36938 -- GLOBAL_STEP: 599600 | > loss_text_ce: 0.03285320848226547 (0.03589427444929471) | > loss_mel_ce: 2.5117528438568115 (nan) | > loss: 0.010097643360495567 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13635422694505486) | > loader_time: 0.0046 (0.0046649405132680975)  --> TIME: 2025-02-25 03:04:55 -- STEP: 8642/36938 -- GLOBAL_STEP: 599650 | > loss_text_ce: 0.034343354403972626 (0.03589170968133705) | > loss_mel_ce: 2.726003408432007 (nan) | > loss: 0.010953756980597973 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.1363433764174537) | > loader_time: 0.0045 (0.004664600681745473)  --> TIME: 2025-02-25 03:05:04 -- STEP: 8692/36938 -- GLOBAL_STEP: 599700 | > loss_text_ce: 0.039635684341192245 (0.035895031975651345) | > loss_mel_ce: 3.2553915977478027 (nan) | > loss: 0.013075505383312702 (nan) | > current_lr: 5e-06 | > step_time: 0.1842 (0.1363398439040451) | > loader_time: 0.0048 (0.004664545283030114)  --> TIME: 2025-02-25 03:05:14 -- STEP: 8742/36938 -- GLOBAL_STEP: 599750 | > loss_text_ce: 0.03300490975379944 (0.035887263343249175) | > loss_mel_ce: 2.398914098739624 (nan) | > loss: 0.009650473482906818 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13636074647617638) | > loader_time: 0.0047 (0.00466422441757891)  --> TIME: 2025-02-25 03:05:23 -- STEP: 8792/36938 -- GLOBAL_STEP: 599800 | > loss_text_ce: 0.029633788391947746 (0.03588545730793382) | > loss_mel_ce: 4.205729007720947 (nan) | > loss: 0.01680699735879898 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13636060873956665) | > loader_time: 0.0045 (0.004664128861501078)  --> TIME: 2025-02-25 03:05:33 -- STEP: 8842/36938 -- GLOBAL_STEP: 599850 | > loss_text_ce: 0.01806265115737915 (0.03587508414385543) | > loss_mel_ce: 0.9044855833053589 (nan) | > loss: 0.003660905873402953 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13639439112243285) | > loader_time: 0.0047 (0.004664251664676603)  --> TIME: 2025-02-25 03:05:42 -- STEP: 8892/36938 -- GLOBAL_STEP: 599900 | > loss_text_ce: 0.035684872418642044 (0.03587366451349643) | > loss_mel_ce: 3.2783203125 (nan) | > loss: 0.013150814920663834 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.1364085027027174) | > loader_time: 0.0045 (0.004664240766097137)  --> TIME: 2025-02-25 03:05:52 -- STEP: 8942/36938 -- GLOBAL_STEP: 599950 | > loss_text_ce: 0.03377887234091759 (0.03586999072231445) | > loss_mel_ce: 2.123897075653076 (nan) | > loss: 0.008562207221984863 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13639306363524079) | > loader_time: 0.0045 (0.004663991090889306)  --> TIME: 2025-02-25 03:06:01 -- STEP: 8992/36938 -- GLOBAL_STEP: 600000 | > loss_text_ce: 0.03710025176405907 (0.035873640996686995) | > loss_mel_ce: 2.657993793487549 (nan) | > loss: 0.010694818571209908 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13639262138312414) | > loader_time: 0.0046 (0.004663881749657136) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_600000.pth  --> TIME: 2025-02-25 03:06:17 -- STEP: 9042/36938 -- GLOBAL_STEP: 600050 | > loss_text_ce: 0.04006289690732956 (0.03587872667385987) | > loss_mel_ce: 2.697387218475342 (nan) | > loss: 0.010862898081541061 (nan) | > current_lr: 5e-06 | > step_time: 0.1873 (0.13641162075265492) | > loader_time: 0.0049 (0.004664092431999744)  --> TIME: 2025-02-25 03:06:26 -- STEP: 9092/36938 -- GLOBAL_STEP: 600100 | > loss_text_ce: 0.04279019683599472 (0.035875324410002116) | > loss_mel_ce: 2.7174715995788574 (nan) | > loss: 0.010953420773148537 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13638794747313585) | > loader_time: 0.0046 (0.004663885583764292)  --> TIME: 2025-02-25 03:06:35 -- STEP: 9142/36938 -- GLOBAL_STEP: 600150 | > loss_text_ce: 0.03778362274169922 (0.0358777955238419) | > loss_mel_ce: 3.3363728523254395 (nan) | > loss: 0.013389511033892632 (nan) | > current_lr: 5e-06 | > step_time: 0.163 (0.13637180676269367) | > loader_time: 0.0047 (0.004663472336247732)  --> TIME: 2025-02-25 03:06:44 -- STEP: 9192/36938 -- GLOBAL_STEP: 600200 | > loss_text_ce: 0.036130793392658234 (0.035873297003346356) | > loss_mel_ce: 3.2466700077056885 (nan) | > loss: 0.013026988133788109 (nan) | > current_lr: 5e-06 | > step_time: 0.1528 (0.13635931835162113) | > loader_time: 0.0048 (0.0046631484264078385)  --> TIME: 2025-02-25 03:06:53 -- STEP: 9242/36938 -- GLOBAL_STEP: 600250 | > loss_text_ce: 0.040809791535139084 (0.03587857865037321) | > loss_mel_ce: 3.293363571166992 (nan) | > loss: 0.013230848126113415 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13637833444619693) | > loader_time: 0.0046 (0.004662929869038655)  --> TIME: 2025-02-25 03:07:03 -- STEP: 9292/36938 -- GLOBAL_STEP: 600300 | > loss_text_ce: 0.03197565674781799 (0.03588274049420668) | > loss_mel_ce: 3.306018829345703 (nan) | > loss: 0.01324601098895073 (nan) | > current_lr: 5e-06 | > step_time: 0.1718 (0.13640332568374328) | > loader_time: 0.0047 (0.0046627417341471335)  --> TIME: 2025-02-25 03:07:12 -- STEP: 9342/36938 -- GLOBAL_STEP: 600350 | > loss_text_ce: 0.025137154385447502 (0.0358727131454816) | > loss_mel_ce: 3.3301806449890137 (nan) | > loss: 0.013314753770828247 (nan) | > current_lr: 5e-06 | > step_time: 0.1846 (0.13641894690563225) | > loader_time: 0.0048 (0.004662731658006179)  --> TIME: 2025-02-25 03:07:22 -- STEP: 9392/36938 -- GLOBAL_STEP: 600400 | > loss_text_ce: 0.04295416176319122 (0.03586914025114586) | > loss_mel_ce: 3.2941837310791016 (nan) | > loss: 0.013242611661553383 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.1364515203430461) | > loader_time: 0.0046 (0.004662697420818852)  --> TIME: 2025-02-25 03:07:31 -- STEP: 9442/36938 -- GLOBAL_STEP: 600450 | > loss_text_ce: 0.04236676171422005 (0.03587584388973955) | > loss_mel_ce: 2.7989702224731445 (nan) | > loss: 0.011275147087872028 (nan) | > current_lr: 5e-06 | > step_time: 0.1714 (0.13645045013908502) | > loader_time: 0.0048 (0.004662537822226461)  --> TIME: 2025-02-25 03:07:41 -- STEP: 9492/36938 -- GLOBAL_STEP: 600500 | > loss_text_ce: 0.034560028463602066 (0.03587014189683062) | > loss_mel_ce: 1.4373931884765625 (nan) | > loss: 0.005841084755957127 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13647024833950064) | > loader_time: 0.0045 (0.004662432928804711)  --> TIME: 2025-02-25 03:07:50 -- STEP: 9542/36938 -- GLOBAL_STEP: 600550 | > loss_text_ce: 0.03610728681087494 (0.03586884772198469) | > loss_mel_ce: 3.0103657245635986 (nan) | > loss: 0.012089179828763008 (nan) | > current_lr: 5e-06 | > step_time: 0.1466 (0.136489656006008) | > loader_time: 0.0047 (0.004662287707499708)  --> TIME: 2025-02-25 03:08:00 -- STEP: 9592/36938 -- GLOBAL_STEP: 600600 | > loss_text_ce: 0.036690521985292435 (0.03587224516859112) | > loss_mel_ce: 2.6363308429718018 (nan) | > loss: 0.010607227683067322 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13648174798816715) | > loader_time: 0.0046 (0.004662040052461672)  --> TIME: 2025-02-25 03:08:10 -- STEP: 9642/36938 -- GLOBAL_STEP: 600650 | > loss_text_ce: 0.03505752235651016 (0.03586529236629931) | > loss_mel_ce: 2.402055263519287 (nan) | > loss: 0.009671082720160484 (nan) | > current_lr: 5e-06 | > step_time: 0.1606 (0.13653450038417736) | > loader_time: 0.005 (0.004662245172011222)  --> TIME: 2025-02-25 03:08:19 -- STEP: 9692/36938 -- GLOBAL_STEP: 600700 | > loss_text_ce: 0.03164678439497948 (0.03585904556200843) | > loss_mel_ce: 2.7407989501953125 (nan) | > loss: 0.011001769453287125 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13654789816875665) | > loader_time: 0.0046 (0.004662266040310807)  --> TIME: 2025-02-25 03:08:28 -- STEP: 9742/36938 -- GLOBAL_STEP: 600750 | > loss_text_ce: 0.022860469296574593 (0.035862089823445444) | > loss_mel_ce: 2.665708065032959 (nan) | > loss: 0.010668923147022724 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13653468708805025) | > loader_time: 0.0046 (0.004662241345434762)  --> TIME: 2025-02-25 03:08:38 -- STEP: 9792/36938 -- GLOBAL_STEP: 600800 | > loss_text_ce: 0.030315181240439415 (0.03587006712963865) | > loss_mel_ce: 2.0850160121917725 (nan) | > loss: 0.008394171483814716 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13654529176701122) | > loader_time: 0.0047 (0.0046621354576809075)  --> TIME: 2025-02-25 03:08:47 -- STEP: 9842/36938 -- GLOBAL_STEP: 600850 | > loss_text_ce: 0.032933156937360764 (0.03587608473207725) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13652967449805264) | > loader_time: 0.0045 (0.004661870133559466)  --> TIME: 2025-02-25 03:08:56 -- STEP: 9892/36938 -- GLOBAL_STEP: 600900 | > loss_text_ce: 0.041743822395801544 (0.035878883819165125) | > loss_mel_ce: 1.0809122323989868 (nan) | > loss: 0.004454984795302153 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13655562020967796) | > loader_time: 0.0073 (0.004662267673945943)  --> TIME: 2025-02-25 03:09:06 -- STEP: 9942/36938 -- GLOBAL_STEP: 600950 | > loss_text_ce: 0.02655714936554432 (0.0358806672174232) | > loss_mel_ce: 2.514237403869629 (nan) | > loss: 0.010082518681883812 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13654917000141495) | > loader_time: 0.0047 (0.0046628782433107184)  --> TIME: 2025-02-25 03:09:15 -- STEP: 9992/36938 -- GLOBAL_STEP: 601000 | > loss_text_ce: 0.03933820500969887 (0.03587694267136085) | > loss_mel_ce: 3.479038715362549 (nan) | > loss: 0.013961813412606716 (nan) | > current_lr: 5e-06 | > step_time: 0.1853 (0.13658035142694686) | > loader_time: 0.005 (0.004662990307597954)  --> TIME: 2025-02-25 03:09:25 -- STEP: 10042/36938 -- GLOBAL_STEP: 601050 | > loss_text_ce: 0.033264484256505966 (0.03587700279024792) | > loss_mel_ce: 3.141065835952759 (nan) | > loss: 0.012596549466252327 (nan) | > current_lr: 5e-06 | > step_time: 0.1875 (0.1365850136541698) | > loader_time: 0.0048 (0.004663091735330818)  --> TIME: 2025-02-25 03:09:34 -- STEP: 10092/36938 -- GLOBAL_STEP: 601100 | > loss_text_ce: 0.03943812847137451 (0.035879623706592874) | > loss_mel_ce: 3.2393574714660645 (nan) | > loss: 0.013011095114052296 (nan) | > current_lr: 5e-06 | > step_time: 0.1893 (0.13660710379097418) | > loader_time: 0.0051 (0.004663346426098587)  --> TIME: 2025-02-25 03:09:44 -- STEP: 10142/36938 -- GLOBAL_STEP: 601150 | > loss_text_ce: 0.0312040988355875 (0.035881215162201004) | > loss_mel_ce: 2.9784247875213623 (nan) | > loss: 0.011942972429096699 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.13659792192305262) | > loader_time: 0.0046 (0.004663241518434749)  --> TIME: 2025-02-25 03:09:53 -- STEP: 10192/36938 -- GLOBAL_STEP: 601200 | > loss_text_ce: 0.03570672124624252 (0.03588467905781413) | > loss_mel_ce: 2.6023857593536377 (nan) | > loss: 0.010468621738255024 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.1366355370034227) | > loader_time: 0.0047 (0.004663591552379555)  --> TIME: 2025-02-25 03:10:03 -- STEP: 10242/36938 -- GLOBAL_STEP: 601250 | > loss_text_ce: 0.04259403049945831 (0.03588134670096111) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.1366321146197207) | > loader_time: 0.0046 (0.00466361329369749)  --> TIME: 2025-02-25 03:10:12 -- STEP: 10292/36938 -- GLOBAL_STEP: 601300 | > loss_text_ce: 0.03847447410225868 (0.03587802537022652) | > loss_mel_ce: 2.699265956878662 (nan) | > loss: 0.01086405012756586 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13663564977412354) | > loader_time: 0.0045 (0.004663707725370717)  --> TIME: 2025-02-25 03:10:21 -- STEP: 10342/36938 -- GLOBAL_STEP: 601350 | > loss_text_ce: 0.037277109920978546 (0.03587780494645945) | > loss_mel_ce: 2.2404119968414307 (nan) | > loss: 0.009038450196385384 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1366209032410603) | > loader_time: 0.0046 (0.0046637335360292295)  --> TIME: 2025-02-25 03:10:31 -- STEP: 10392/36938 -- GLOBAL_STEP: 601400 | > loss_text_ce: 0.0377684123814106 (0.035879387391649774) | > loss_mel_ce: 3.4214770793914795 (nan) | > loss: 0.013727164827287197 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13661504363077617) | > loader_time: 0.0045 (0.00466361288569174)  --> TIME: 2025-02-25 03:10:40 -- STEP: 10442/36938 -- GLOBAL_STEP: 601450 | > loss_text_ce: 0.033729586750268936 (0.03587928471774231) | > loss_mel_ce: 2.66983699798584 (nan) | > loss: 0.010728439316153526 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.1366394136491303) | > loader_time: 0.0046 (0.004663515698290998)  --> TIME: 2025-02-25 03:10:50 -- STEP: 10492/36938 -- GLOBAL_STEP: 601500 | > loss_text_ce: 0.03026924468576908 (0.03587939542575465) | > loss_mel_ce: 2.651392936706543 (nan) | > loss: 0.010641517117619514 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13665528302878047) | > loader_time: 0.0046 (0.004663477019412898)  --> TIME: 2025-02-25 03:10:59 -- STEP: 10542/36938 -- GLOBAL_STEP: 601550 | > loss_text_ce: 0.044971346855163574 (0.03587934884775042) | > loss_mel_ce: 3.113417863845825 (nan) | > loss: 0.012533290311694145 (nan) | > current_lr: 5e-06 | > step_time: 0.1442 (0.1366407388845628) | > loader_time: 0.0048 (0.004663351409418542)  --> TIME: 2025-02-25 03:11:08 -- STEP: 10592/36938 -- GLOBAL_STEP: 601600 | > loss_text_ce: 0.03469250351190567 (0.03588050419334192) | > loss_mel_ce: 0.18710938096046448 (nan) | > loss: 0.0008801662479527295 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13663737854028404) | > loader_time: 0.0047 (0.004663300073038781)  --> TIME: 2025-02-25 03:11:18 -- STEP: 10642/36938 -- GLOBAL_STEP: 601650 | > loss_text_ce: 0.05567677319049835 (0.035885413963448194) | > loss_mel_ce: 1.0996053218841553 (nan) | > loss: 0.004584453068673611 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13665606282837656) | > loader_time: 0.0046 (0.004663387426553069)  --> TIME: 2025-02-25 03:11:27 -- STEP: 10692/36938 -- GLOBAL_STEP: 601700 | > loss_text_ce: 0.014730011112987995 (0.03588392914806245) | > loss_mel_ce: 1.6697884798049927 (nan) | > loss: 0.006684597581624985 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13664720281874965) | > loader_time: 0.0045 (0.004663292807762072)  --> TIME: 2025-02-25 03:11:37 -- STEP: 10742/36938 -- GLOBAL_STEP: 601750 | > loss_text_ce: 0.04078463837504387 (0.035888993989844464) | > loss_mel_ce: 1.8606215715408325 (nan) | > loss: 0.0075452630408108234 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13666435227760754) | > loader_time: 0.0045 (0.004663275997635598)  --> TIME: 2025-02-25 03:11:46 -- STEP: 10792/36938 -- GLOBAL_STEP: 601800 | > loss_text_ce: 0.04212823137640953 (0.03588451340233945) | > loss_mel_ce: 3.500437021255493 (nan) | > loss: 0.01405780017375946 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13666988892675408) | > loader_time: 0.0046 (0.004663135030695559)  --> TIME: 2025-02-25 03:11:56 -- STEP: 10842/36938 -- GLOBAL_STEP: 601850 | > loss_text_ce: 0.03944201394915581 (0.03588537864020731) | > loss_mel_ce: 3.272796630859375 (nan) | > loss: 0.013143804855644703 (nan) | > current_lr: 5e-06 | > step_time: 0.169 (0.13668625371479448) | > loader_time: 0.0049 (0.004663388769997372)  --> TIME: 2025-02-25 03:12:05 -- STEP: 10892/36938 -- GLOBAL_STEP: 601900 | > loss_text_ce: 0.036430440843105316 (0.035882330438196434) | > loss_mel_ce: 0.12589111924171448 (nan) | > loss: 0.0006441332516260445 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13669497221215773) | > loader_time: 0.0045 (0.0046632940437353314)  --> TIME: 2025-02-25 03:12:14 -- STEP: 10942/36938 -- GLOBAL_STEP: 601950 | > loss_text_ce: 0.04238766059279442 (0.035886214065188674) | > loss_mel_ce: 2.736537456512451 (nan) | > loss: 0.011027481406927109 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1366960715820487) | > loader_time: 0.0046 (0.004663081801889844)  --> TIME: 2025-02-25 03:12:24 -- STEP: 10992/36938 -- GLOBAL_STEP: 602000 | > loss_text_ce: 0.053987011313438416 (0.035888577755838556) | > loss_mel_ce: 2.505051612854004 (nan) | > loss: 0.01015491597354412 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13669730972341396) | > loader_time: 0.0048 (0.004663214195937218)  --> TIME: 2025-02-25 03:12:33 -- STEP: 11042/36938 -- GLOBAL_STEP: 602050 | > loss_text_ce: 0.03169843554496765 (0.03588600596084046) | > loss_mel_ce: 2.2395832538604736 (nan) | > loss: 0.009013023227453232 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13668807106505185) | > loader_time: 0.0047 (0.004663257101397864)  --> TIME: 2025-02-25 03:12:43 -- STEP: 11092/36938 -- GLOBAL_STEP: 602100 | > loss_text_ce: 0.04917578026652336 (0.03588687205516523) | > loss_mel_ce: 2.6511929035186768 (nan) | > loss: 0.010715749114751816 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13671722422426805) | > loader_time: 0.0044 (0.004663718443161048)  --> TIME: 2025-02-25 03:12:52 -- STEP: 11142/36938 -- GLOBAL_STEP: 602150 | > loss_text_ce: 0.03724644333124161 (0.03588889802071656) | > loss_mel_ce: 0.31773144006729126 (nan) | > loss: 0.001408642390742898 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13670242817049671) | > loader_time: 0.0045 (0.004663437792138288)  --> TIME: 2025-02-25 03:13:01 -- STEP: 11192/36938 -- GLOBAL_STEP: 602200 | > loss_text_ce: 0.0383942537009716 (0.03588920329130552) | > loss_mel_ce: 3.0463547706604004 (nan) | > loss: 0.012241068296134472 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.1367002395411062) | > loader_time: 0.0048 (0.00466326435094564)  --> TIME: 2025-02-25 03:13:10 -- STEP: 11242/36938 -- GLOBAL_STEP: 602250 | > loss_text_ce: 0.03535794839262962 (0.03589779548829116) | > loss_mel_ce: 2.970759153366089 (nan) | > loss: 0.011929037049412727 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1366913230833559) | > loader_time: 0.0045 (0.004663004143088333)  --> TIME: 2025-02-25 03:13:20 -- STEP: 11292/36938 -- GLOBAL_STEP: 602300 | > loss_text_ce: 0.023646827787160873 (0.03589760154537469) | > loss_mel_ce: 2.6938440799713135 (nan) | > loss: 0.010783694684505463 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13669638531536554) | > loader_time: 0.0046 (0.004662819927227727)  --> TIME: 2025-02-25 03:13:29 -- STEP: 11342/36938 -- GLOBAL_STEP: 602350 | > loss_text_ce: 0.03175791725516319 (0.0358981293737143) | > loss_mel_ce: 3.0638749599456787 (nan) | > loss: 0.01228425744920969 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.13669064290476007) | > loader_time: 0.0046 (0.004662670926891326)  --> TIME: 2025-02-25 03:13:38 -- STEP: 11392/36938 -- GLOBAL_STEP: 602400 | > loss_text_ce: 0.04087325558066368 (0.035902941440152665) | > loss_mel_ce: 3.153308391571045 (nan) | > loss: 0.012675324454903603 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13668681457220155) | > loader_time: 0.0046 (0.004662804012553067)  --> TIME: 2025-02-25 03:13:48 -- STEP: 11442/36938 -- GLOBAL_STEP: 602450 | > loss_text_ce: 0.03413846716284752 (0.03589999560112705) | > loss_mel_ce: 1.4233747720718384 (nan) | > loss: 0.005783782806247473 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13669779933722068) | > loader_time: 0.0047 (0.004662642860679299)  --> TIME: 2025-02-25 03:13:57 -- STEP: 11492/36938 -- GLOBAL_STEP: 602500 | > loss_text_ce: 0.04472259059548378 (0.0359032928041321) | > loss_mel_ce: 2.895962953567505 (nan) | > loss: 0.01166938804090023 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.1366852160285169) | > loader_time: 0.0045 (0.004662362926727451)  --> TIME: 2025-02-25 03:14:06 -- STEP: 11542/36938 -- GLOBAL_STEP: 602550 | > loss_text_ce: 0.0444134846329689 (0.035899083626384755) | > loss_mel_ce: 3.027374267578125 (nan) | > loss: 0.012189635075628757 (nan) | > current_lr: 5e-06 | > step_time: 0.1648 (0.1366729883816312) | > loader_time: 0.0047 (0.004662099154770949)  --> TIME: 2025-02-25 03:14:16 -- STEP: 11592/36938 -- GLOBAL_STEP: 602600 | > loss_text_ce: 0.04278665408492088 (0.03589820873056198) | > loss_mel_ce: 3.0903587341308594 (nan) | > loss: 0.012433117255568504 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13668196873059346) | > loader_time: 0.0047 (0.004662078051339997)  --> TIME: 2025-02-25 03:14:25 -- STEP: 11642/36938 -- GLOBAL_STEP: 602650 | > loss_text_ce: 0.05232783779501915 (0.035905347676319276) | > loss_mel_ce: 2.7990336418151855 (nan) | > loss: 0.011314927600324154 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.1366748339202963) | > loader_time: 0.0045 (0.004661780066392675)  --> TIME: 2025-02-25 03:14:35 -- STEP: 11692/36938 -- GLOBAL_STEP: 602700 | > loss_text_ce: 0.03630134090781212 (0.03590546778927001) | > loss_mel_ce: 2.9058477878570557 (nan) | > loss: 0.011675195768475533 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13670527219282924) | > loader_time: 0.0045 (0.00466164140068337)  --> TIME: 2025-02-25 03:14:44 -- STEP: 11742/36938 -- GLOBAL_STEP: 602750 | > loss_text_ce: 0.036036502569913864 (0.03590869142762237) | > loss_mel_ce: 2.9661035537719727 (nan) | > loss: 0.011913254857063293 (nan) | > current_lr: 5e-06 | > step_time: 0.1868 (0.13670077392543667) | > loader_time: 0.0048 (0.004661388889404346)  --> TIME: 2025-02-25 03:14:53 -- STEP: 11792/36938 -- GLOBAL_STEP: 602800 | > loss_text_ce: 0.034602053463459015 (0.03590370734827817) | > loss_mel_ce: 0.15253296494483948 (nan) | > loss: 0.000742599309887737 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13670926339914208) | > loader_time: 0.0046 (0.004661334418215279)  --> TIME: 2025-02-25 03:15:03 -- STEP: 11842/36938 -- GLOBAL_STEP: 602850 | > loss_text_ce: 0.041164640337228775 (0.03589485286513344) | > loss_mel_ce: 0.023895263671875 (nan) | > loss: 0.0002581742301117629 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.136706586968071) | > loader_time: 0.0047 (0.0046613799661946445)  --> TIME: 2025-02-25 03:15:12 -- STEP: 11892/36938 -- GLOBAL_STEP: 602900 | > loss_text_ce: 0.03286010026931763 (0.03589591384630995) | > loss_mel_ce: 2.5696332454681396 (nan) | > loss: 0.01032735500484705 (nan) | > current_lr: 5e-06 | > step_time: 0.165 (0.13670722400945615) | > loader_time: 0.0074 (0.004661501777160047)  --> TIME: 2025-02-25 03:15:21 -- STEP: 11942/36938 -- GLOBAL_STEP: 602950 | > loss_text_ce: 0.03721609339118004 (0.0359022321016201) | > loss_mel_ce: 3.498394250869751 (nan) | > loss: 0.014030201360583305 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.13670767549493343) | > loader_time: 0.0046 (0.004661488025916349)  --> TIME: 2025-02-25 03:15:31 -- STEP: 11992/36938 -- GLOBAL_STEP: 603000 | > loss_text_ce: 0.03479514643549919 (0.03590356649029747) | > loss_mel_ce: 0.1717267781496048 (nan) | > loss: 0.0008195315022021532 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13670815319677743) | > loader_time: 0.0045 (0.004661535624268045)  --> TIME: 2025-02-25 03:15:40 -- STEP: 12042/36938 -- GLOBAL_STEP: 603050 | > loss_text_ce: 0.034868884831666946 (0.0359037749043446) | > loss_mel_ce: 3.0050265789031982 (nan) | > loss: 0.012063078582286835 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.13672060010995857) | > loader_time: 0.0049 (0.0046619918532118726)  --> TIME: 2025-02-25 03:15:50 -- STEP: 12092/36938 -- GLOBAL_STEP: 603100 | > loss_text_ce: 0.040861353278160095 (0.03590640612839983) | > loss_mel_ce: 2.704965829849243 (nan) | > loss: 0.010896140709519386 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.1367521670887573) | > loader_time: 0.0048 (0.004662067082556816)  --> TIME: 2025-02-25 03:15:59 -- STEP: 12142/36938 -- GLOBAL_STEP: 603150 | > loss_text_ce: 0.02848196215927601 (0.03590186907730452) | > loss_mel_ce: 0.06689453125 (nan) | > loss: 0.0003784781729336828 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1367466764500339) | > loader_time: 0.0045 (0.0046618008334757075)  --> TIME: 2025-02-25 03:16:09 -- STEP: 12192/36938 -- GLOBAL_STEP: 603200 | > loss_text_ce: 0.03916528448462486 (0.03590386639348524) | > loss_mel_ce: 3.0319721698760986 (nan) | > loss: 0.012187054380774498 (nan) | > current_lr: 5e-06 | > step_time: 0.1441 (0.13676092257415198) | > loader_time: 0.0047 (0.004661968471653487)  --> TIME: 2025-02-25 03:16:18 -- STEP: 12242/36938 -- GLOBAL_STEP: 603250 | > loss_text_ce: 0.043227534741163254 (0.03589678061978427) | > loss_mel_ce: 2.7822630405426025 (nan) | > loss: 0.011212264187633991 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.1367683971732527) | > loader_time: 0.0046 (0.004662117017793626)  --> TIME: 2025-02-25 03:16:28 -- STEP: 12292/36938 -- GLOBAL_STEP: 603300 | > loss_text_ce: 0.038008060306310654 (0.03589459573892441) | > loss_mel_ce: 2.8696532249450684 (nan) | > loss: 0.011538338847458363 (nan) | > current_lr: 5e-06 | > step_time: 0.1859 (0.13677871923589402) | > loader_time: 0.0049 (0.004662142527673656)  --> TIME: 2025-02-25 03:16:37 -- STEP: 12342/36938 -- GLOBAL_STEP: 603350 | > loss_text_ce: 0.033072035759687424 (0.035896615316000566) | > loss_mel_ce: 2.6369049549102783 (nan) | > loss: 0.010595147497951984 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13677517644514062) | > loader_time: 0.0046 (0.004661977571213249)  --> TIME: 2025-02-25 03:16:46 -- STEP: 12392/36938 -- GLOBAL_STEP: 603400 | > loss_text_ce: 0.031739626079797745 (0.035888097497434136) | > loss_mel_ce: 2.7888729572296143 (nan) | > loss: 0.011192908510565758 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.1367737640674225) | > loader_time: 0.0047 (0.004661888499810948)  --> TIME: 2025-02-25 03:16:56 -- STEP: 12442/36938 -- GLOBAL_STEP: 603450 | > loss_text_ce: 0.03864924982190132 (0.03588935300894027) | > loss_mel_ce: 3.1773858070373535 (nan) | > loss: 0.012762044556438923 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13678220074921432) | > loader_time: 0.0046 (0.004661830459218197)  --> TIME: 2025-02-25 03:17:05 -- STEP: 12492/36938 -- GLOBAL_STEP: 603500 | > loss_text_ce: 0.041123442351818085 (0.03589350125432382) | > loss_mel_ce: 3.301379680633545 (nan) | > loss: 0.013263901695609093 (nan) | > current_lr: 5e-06 | > step_time: 0.1918 (0.1367850843775246) | > loader_time: 0.0048 (0.00466175946599962)  --> TIME: 2025-02-25 03:17:15 -- STEP: 12542/36938 -- GLOBAL_STEP: 603550 | > loss_text_ce: 0.03872810676693916 (0.03589471481283623) | > loss_mel_ce: 0.9924493432044983 (nan) | > loss: 0.004091973882168531 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13681232828478793) | > loader_time: 0.0045 (0.00466172099398606)  --> TIME: 2025-02-25 03:17:24 -- STEP: 12592/36938 -- GLOBAL_STEP: 603600 | > loss_text_ce: 0.034822843968868256 (0.03590046296462986) | > loss_mel_ce: 3.061173915863037 (nan) | > loss: 0.012285702861845493 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13682374979216694) | > loader_time: 0.0046 (0.00466164461842332)  --> TIME: 2025-02-25 03:17:34 -- STEP: 12642/36938 -- GLOBAL_STEP: 603650 | > loss_text_ce: 0.029677733778953552 (0.0359024840932036) | > loss_mel_ce: 3.3729333877563477 (nan) | > loss: 0.013502425514161587 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13683196548192772) | > loader_time: 0.0046 (0.004661670800082132)  --> TIME: 2025-02-25 03:17:43 -- STEP: 12692/36938 -- GLOBAL_STEP: 603700 | > loss_text_ce: 0.0338844396173954 (0.03590330356022269) | > loss_mel_ce: 2.8155055046081543 (nan) | > loss: 0.01130710355937481 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.136841334062084) | > loader_time: 0.0048 (0.004661684001690548)  --> TIME: 2025-02-25 03:17:53 -- STEP: 12742/36938 -- GLOBAL_STEP: 603750 | > loss_text_ce: 0.03440656512975693 (0.035902816780774366) | > loss_mel_ce: 3.3795268535614014 (nan) | > loss: 0.013547356240451336 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1368484317816665) | > loader_time: 0.0046 (0.004661840259364561)  --> TIME: 2025-02-25 03:18:02 -- STEP: 12792/36938 -- GLOBAL_STEP: 603800 | > loss_text_ce: 0.031750984489917755 (0.035900068072994534) | > loss_mel_ce: 2.190274715423584 (nan) | > loss: 0.008817562833428383 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.1368405362492135) | > loader_time: 0.0048 (0.00466176796660262)  --> TIME: 2025-02-25 03:18:11 -- STEP: 12842/36938 -- GLOBAL_STEP: 603850 | > loss_text_ce: 0.03774711489677429 (0.03589323579161539) | > loss_mel_ce: 3.2452404499053955 (nan) | > loss: 0.013027729466557503 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.13684230583365017) | > loader_time: 0.0049 (0.004661872497954232)  --> TIME: 2025-02-25 03:18:20 -- STEP: 12892/36938 -- GLOBAL_STEP: 603900 | > loss_text_ce: 0.03495141118764877 (0.03589710799729816) | > loss_mel_ce: 2.595623254776001 (nan) | > loss: 0.010438788682222366 (nan) | > current_lr: 5e-06 | > step_time: 0.1357 (0.13683262686532696) | > loader_time: 0.0046 (0.004661707230099498)  --> TIME: 2025-02-25 03:18:30 -- STEP: 12942/36938 -- GLOBAL_STEP: 603950 | > loss_text_ce: 0.03433757275342941 (0.03590151702265824) | > loss_mel_ce: 3.179321050643921 (nan) | > loss: 0.012752613984048367 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13684180451220582) | > loader_time: 0.0046 (0.00466286517095939)  --> TIME: 2025-02-25 03:18:39 -- STEP: 12992/36938 -- GLOBAL_STEP: 604000 | > loss_text_ce: 0.05473595857620239 (0.035903876676098415) | > loss_mel_ce: 2.1866455078125 (nan) | > loss: 0.00889437086880207 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.1368239942038884) | > loader_time: 0.0048 (0.0046628108752771935)  --> TIME: 2025-02-25 03:18:48 -- STEP: 13042/36938 -- GLOBAL_STEP: 604050 | > loss_text_ce: 0.03651753440499306 (0.035904838028538624) | > loss_mel_ce: 2.9471585750579834 (nan) | > loss: 0.011839985847473145 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.1368257397980876) | > loader_time: 0.0047 (0.004663034535463426)  --> TIME: 2025-02-25 03:18:57 -- STEP: 13092/36938 -- GLOBAL_STEP: 604100 | > loss_text_ce: 0.041863616555929184 (0.035905518924095645) | > loss_mel_ce: 3.03393816947937 (nan) | > loss: 0.012205563485622406 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13681913979221785) | > loader_time: 0.0048 (0.004663162336340724)  --> TIME: 2025-02-25 03:19:07 -- STEP: 13142/36938 -- GLOBAL_STEP: 604150 | > loss_text_ce: 0.04647607356309891 (0.03590652859972277) | > loss_mel_ce: 1.1618458032608032 (nan) | > loss: 0.004794928710907698 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13684205886635922) | > loader_time: 0.0046 (0.004663431903866373)  --> TIME: 2025-02-25 03:19:17 -- STEP: 13192/36938 -- GLOBAL_STEP: 604200 | > loss_text_ce: 0.035500671714544296 (0.03590426250964355) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13683870902850914) | > loader_time: 0.0045 (0.004663282737650939)  --> TIME: 2025-02-25 03:19:26 -- STEP: 13242/36938 -- GLOBAL_STEP: 604250 | > loss_text_ce: 0.036219652742147446 (0.035908989198891135) | > loss_mel_ce: 3.2015745639801025 (nan) | > loss: 0.012848390266299248 (nan) | > current_lr: 5e-06 | > step_time: 0.2017 (0.13685747610624263) | > loader_time: 0.0052 (0.004663549598799833)  --> TIME: 2025-02-25 03:19:35 -- STEP: 13292/36938 -- GLOBAL_STEP: 604300 | > loss_text_ce: 0.040841907262802124 (0.035909012803558814) | > loss_mel_ce: 2.6487159729003906 (nan) | > loss: 0.010672848671674728 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13683777220666046) | > loader_time: 0.0047 (0.0046636783104505135)  --> TIME: 2025-02-25 03:19:45 -- STEP: 13342/36938 -- GLOBAL_STEP: 604350 | > loss_text_ce: 0.03203435242176056 (0.035906152629334255) | > loss_mel_ce: 2.976130485534668 (nan) | > loss: 0.011937162838876247 (nan) | > current_lr: 5e-06 | > step_time: 0.1782 (0.13683892268502668) | > loader_time: 0.0048 (0.004663756540235818)  --> TIME: 2025-02-25 03:19:54 -- STEP: 13392/36938 -- GLOBAL_STEP: 604400 | > loss_text_ce: 0.03744307532906532 (0.03590294320364116) | > loss_mel_ce: 3.271634578704834 (nan) | > loss: 0.013131261803209782 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13683100657152614) | > loader_time: 0.0046 (0.004663734577747484)  --> TIME: 2025-02-25 03:20:03 -- STEP: 13442/36938 -- GLOBAL_STEP: 604450 | > loss_text_ce: 0.034286487847566605 (0.035901942198702096) | > loss_mel_ce: 2.8818359375 (nan) | > loss: 0.0115719148889184 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13683715891472722) | > loader_time: 0.0046 (0.004663748145901583)  --> TIME: 2025-02-25 03:20:13 -- STEP: 13492/36938 -- GLOBAL_STEP: 604500 | > loss_text_ce: 0.03981303796172142 (0.03590157710278341) | > loss_mel_ce: 3.287860155105591 (nan) | > loss: 0.01320505328476429 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13683272904823862) | > loader_time: 0.0046 (0.004663878189785813)  --> TIME: 2025-02-25 03:20:22 -- STEP: 13542/36938 -- GLOBAL_STEP: 604550 | > loss_text_ce: 0.03729093819856644 (0.035897854779200516) | > loss_mel_ce: 3.1915066242218018 (nan) | > loss: 0.012812689878046513 (nan) | > current_lr: 5e-06 | > step_time: 0.1526 (0.13681304935443508) | > loader_time: 0.0047 (0.00466367131831671)  --> TIME: 2025-02-25 03:20:31 -- STEP: 13592/36938 -- GLOBAL_STEP: 604600 | > loss_text_ce: 0.02923743985593319 (0.03590009464854075) | > loss_mel_ce: 2.6027607917785645 (nan) | > loss: 0.010444438084959984 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.1367943504931579) | > loader_time: 0.0045 (0.004663438639828822)  --> TIME: 2025-02-25 03:20:40 -- STEP: 13642/36938 -- GLOBAL_STEP: 604650 | > loss_text_ce: 0.042563047260046005 (0.03589795331296299) | > loss_mel_ce: 2.992875099182129 (nan) | > loss: 0.012045389972627163 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.13680035002576602) | > loader_time: 0.005 (0.0046634434989567645)  --> TIME: 2025-02-25 03:20:50 -- STEP: 13692/36938 -- GLOBAL_STEP: 604700 | > loss_text_ce: 0.04320107772946358 (0.03589800185700223) | > loss_mel_ce: 3.137686252593994 (nan) | > loss: 0.012622568756341934 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.1368081244222958) | > loader_time: 0.0049 (0.004663317376955025)  --> TIME: 2025-02-25 03:20:59 -- STEP: 13742/36938 -- GLOBAL_STEP: 604750 | > loss_text_ce: 0.04031634330749512 (0.03590301663859184) | > loss_mel_ce: 2.754894256591797 (nan) | > loss: 0.011092105880379677 (nan) | > current_lr: 5e-06 | > step_time: 0.1438 (0.1368084335202232) | > loader_time: 0.0046 (0.004663141390378924)  --> TIME: 2025-02-25 03:21:08 -- STEP: 13792/36938 -- GLOBAL_STEP: 604800 | > loss_text_ce: 0.04539833962917328 (0.035908548445470444) | > loss_mel_ce: 3.117053508758545 (nan) | > loss: 0.012549412436783314 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1367919546338079) | > loader_time: 0.0046 (0.004663073166063907)  --> TIME: 2025-02-25 03:21:17 -- STEP: 13842/36938 -- GLOBAL_STEP: 604850 | > loss_text_ce: 0.02999042719602585 (0.03591508516304917) | > loss_mel_ce: 2.5807082653045654 (nan) | > loss: 0.010359915904700756 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13679122470083513) | > loader_time: 0.0045 (0.004663058330412951)  --> TIME: 2025-02-25 03:21:26 -- STEP: 13892/36938 -- GLOBAL_STEP: 604900 | > loss_text_ce: 0.027237005531787872 (0.03591464660500494) | > loss_mel_ce: 3.6744141578674316 (nan) | > loss: 0.01468909252434969 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1367738171060388) | > loader_time: 0.0046 (0.004662903660214852)  --> TIME: 2025-02-25 03:21:36 -- STEP: 13942/36938 -- GLOBAL_STEP: 604950 | > loss_text_ce: 0.03905027359724045 (0.03591524819491965) | > loss_mel_ce: 2.97621750831604 (nan) | > loss: 0.011965349316596985 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13676765499968668) | > loader_time: 0.0046 (0.004662938549603721)  --> TIME: 2025-02-25 03:21:45 -- STEP: 13992/36938 -- GLOBAL_STEP: 605000 | > loss_text_ce: 0.04482513293623924 (0.035914851238819524) | > loss_mel_ce: 3.4689829349517822 (nan) | > loss: 0.013943683356046677 (nan) | > current_lr: 5e-06 | > step_time: 0.1439 (0.13675647190191062) | > loader_time: 0.0048 (0.004662959983922716)  --> TIME: 2025-02-25 03:21:54 -- STEP: 14042/36938 -- GLOBAL_STEP: 605050 | > loss_text_ce: 0.03405404090881348 (0.03591932978298258) | > loss_mel_ce: 2.6002871990203857 (nan) | > loss: 0.01045373547822237 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13673623877192415) | > loader_time: 0.0046 (0.0046628760130142674)  --> TIME: 2025-02-25 03:22:03 -- STEP: 14092/36938 -- GLOBAL_STEP: 605100 | > loss_text_ce: 0.0309651717543602 (0.03591753281260353) | > loss_mel_ce: 2.6322062015533447 (nan) | > loss: 0.010568141005933285 (nan) | > current_lr: 5e-06 | > step_time: 0.1777 (0.136716222360396) | > loader_time: 0.0048 (0.00466274362445518)  --> TIME: 2025-02-25 03:22:12 -- STEP: 14142/36938 -- GLOBAL_STEP: 605150 | > loss_text_ce: 0.04306994378566742 (0.03591902532757404) | > loss_mel_ce: 3.0379137992858887 (nan) | > loss: 0.012226126156747341 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.1367143220939253) | > loader_time: 0.0048 (0.004662889804416567)  --> TIME: 2025-02-25 03:22:22 -- STEP: 14192/36938 -- GLOBAL_STEP: 605200 | > loss_text_ce: 0.032481156289577484 (0.03592316359534838) | > loss_mel_ce: 2.690258741378784 (nan) | > loss: 0.010804523713886738 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13671628055671967) | > loader_time: 0.0047 (0.004662793797462851)  --> TIME: 2025-02-25 03:22:30 -- STEP: 14242/36938 -- GLOBAL_STEP: 605250 | > loss_text_ce: 0.03974524140357971 (0.035924052759252124) | > loss_mel_ce: 2.883047580718994 (nan) | > loss: 0.011598384939134121 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.1366908385200859) | > loader_time: 0.0046 (0.0046627032858922385)  --> TIME: 2025-02-25 03:22:40 -- STEP: 14292/36938 -- GLOBAL_STEP: 605300 | > loss_text_ce: 0.03472648188471794 (0.03592369944283878) | > loss_mel_ce: 3.10752010345459 (nan) | > loss: 0.012469233013689518 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.13670467609722592) | > loader_time: 0.0048 (0.004662754470281739)  --> TIME: 2025-02-25 03:22:49 -- STEP: 14342/36938 -- GLOBAL_STEP: 605350 | > loss_text_ce: 0.03590743616223335 (0.03592348273648968) | > loss_mel_ce: 2.5302517414093018 (nan) | > loss: 0.010183172300457954 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.1367056069056899) | > loader_time: 0.0045 (0.0046627504724851545)  --> TIME: 2025-02-25 03:22:59 -- STEP: 14392/36938 -- GLOBAL_STEP: 605400 | > loss_text_ce: 0.03169335052371025 (0.03591830140631224) | > loss_mel_ce: 0.18299102783203125 (nan) | > loss: 0.0008519222028553486 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13672556497442354) | > loader_time: 0.0046 (0.004662789474797926)  --> TIME: 2025-02-25 03:23:08 -- STEP: 14442/36938 -- GLOBAL_STEP: 605450 | > loss_text_ce: 0.033157821744680405 (0.035917306449981305) | > loss_mel_ce: 2.8463292121887207 (nan) | > loss: 0.011426536366343498 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13671828689807602) | > loader_time: 0.0046 (0.004662847142522618)  --> TIME: 2025-02-25 03:23:18 -- STEP: 14492/36938 -- GLOBAL_STEP: 605500 | > loss_text_ce: 0.041683945804834366 (0.03592106773083075) | > loss_mel_ce: 2.9694502353668213 (nan) | > loss: 0.011948945932090282 (nan) | > current_lr: 5e-06 | > step_time: 0.1872 (0.13671680882659476) | > loader_time: 0.005 (0.004662873071759479)  --> TIME: 2025-02-25 03:23:27 -- STEP: 14542/36938 -- GLOBAL_STEP: 605550 | > loss_text_ce: 0.04008674994111061 (0.03592062560972595) | > loss_mel_ce: 2.761319637298584 (nan) | > loss: 0.011116692796349525 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13671626565610803) | > loader_time: 0.0045 (0.004663319211274365)  --> TIME: 2025-02-25 03:23:37 -- STEP: 14592/36938 -- GLOBAL_STEP: 605600 | > loss_text_ce: 0.039514701813459396 (0.03592019663170624) | > loss_mel_ce: 2.682504653930664 (nan) | > loss: 0.010801665484905243 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1367254406377147) | > loader_time: 0.0048 (0.004663385271111054)  --> TIME: 2025-02-25 03:23:46 -- STEP: 14642/36938 -- GLOBAL_STEP: 605650 | > loss_text_ce: 0.041965220123529434 (0.03591737539748254) | > loss_mel_ce: 3.134089469909668 (nan) | > loss: 0.012603392824530602 (nan) | > current_lr: 5e-06 | > step_time: 0.1875 (0.13671426432101105) | > loader_time: 0.0049 (0.004663374690699465)  --> TIME: 2025-02-25 03:23:55 -- STEP: 14692/36938 -- GLOBAL_STEP: 605700 | > loss_text_ce: 0.03465646132826805 (0.035920557456259335) | > loss_mel_ce: 3.2005460262298584 (nan) | > loss: 0.012838105671107769 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13670441015779936) | > loader_time: 0.0046 (0.004663257192533257)  --> TIME: 2025-02-25 03:24:04 -- STEP: 14742/36938 -- GLOBAL_STEP: 605750 | > loss_text_ce: 0.04018017649650574 (0.035918448728160066) | > loss_mel_ce: 3.040010690689087 (nan) | > loss: 0.01222298014909029 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13669044719818166) | > loader_time: 0.0046 (0.004663689442816857)  --> TIME: 2025-02-25 03:24:13 -- STEP: 14792/36938 -- GLOBAL_STEP: 605800 | > loss_text_ce: 0.03290051966905594 (0.035917615074663775) | > loss_mel_ce: 2.2873642444610596 (nan) | > loss: 0.009207400493323803 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1366956930601511) | > loader_time: 0.0046 (0.0046638808681102)  --> TIME: 2025-02-25 03:24:23 -- STEP: 14842/36938 -- GLOBAL_STEP: 605850 | > loss_text_ce: 0.04247704893350601 (0.03591816341594806) | > loss_mel_ce: 3.2446248531341553 (nan) | > loss: 0.013044056482613087 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.1366976264544434) | > loader_time: 0.0047 (0.004663980612776002)  --> TIME: 2025-02-25 03:24:32 -- STEP: 14892/36938 -- GLOBAL_STEP: 605900 | > loss_text_ce: 0.03221060708165169 (0.03592031838825371) | > loss_mel_ce: 2.6798510551452637 (nan) | > loss: 0.010762150399386883 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13669939534881215) | > loader_time: 0.0046 (0.0046639807468263446)  --> TIME: 2025-02-25 03:24:42 -- STEP: 14942/36938 -- GLOBAL_STEP: 605950 | > loss_text_ce: 0.03415132313966751 (0.035918071825085114) | > loss_mel_ce: 2.8336734771728516 (nan) | > loss: 0.01138025801628828 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.1367110797095497) | > loader_time: 0.0046 (0.004664001351873338)  --> TIME: 2025-02-25 03:24:51 -- STEP: 14992/36938 -- GLOBAL_STEP: 606000 | > loss_text_ce: 0.049243297427892685 (0.035914097319288935) | > loss_mel_ce: 2.949104070663452 (nan) | > loss: 0.011898203752934933 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13671285618521972) | > loader_time: 0.0046 (0.0046639433538137424)  --> TIME: 2025-02-25 03:25:00 -- STEP: 15042/36938 -- GLOBAL_STEP: 606050 | > loss_text_ce: 0.03250694274902344 (0.0359103329440192) | > loss_mel_ce: 2.1635682582855225 (nan) | > loss: 0.008714584633708 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13671633985159523) | > loader_time: 0.0048 (0.004663903604493981)  --> TIME: 2025-02-25 03:25:10 -- STEP: 15092/36938 -- GLOBAL_STEP: 606100 | > loss_text_ce: 0.036605168133974075 (0.035915346587424256) | > loss_mel_ce: 2.635493755340576 (nan) | > loss: 0.010603567585349083 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13671249291829182) | > loader_time: 0.0046 (0.004663872554515145)  --> TIME: 2025-02-25 03:25:19 -- STEP: 15142/36938 -- GLOBAL_STEP: 606150 | > loss_text_ce: 0.040059201419353485 (0.0359131375976839) | > loss_mel_ce: 3.4400882720947266 (nan) | > loss: 0.013810109347105026 (nan) | > current_lr: 5e-06 | > step_time: 0.1504 (0.13672136528270568) | > loader_time: 0.0046 (0.004663860147592311)  --> TIME: 2025-02-25 03:25:29 -- STEP: 15192/36938 -- GLOBAL_STEP: 606200 | > loss_text_ce: 0.02821091189980507 (0.03591535209950048) | > loss_mel_ce: 1.9506371021270752 (nan) | > loss: 0.00785257201641798 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13671957406010837) | > loader_time: 0.0045 (0.004663763453045139)  --> TIME: 2025-02-25 03:25:38 -- STEP: 15242/36938 -- GLOBAL_STEP: 606250 | > loss_text_ce: 0.046038951724767685 (0.03592298240325687) | > loss_mel_ce: 3.7820703983306885 (nan) | > loss: 0.015190910547971725 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13671847901433748) | > loader_time: 0.0046 (0.004663694923184618)  --> TIME: 2025-02-25 03:25:47 -- STEP: 15292/36938 -- GLOBAL_STEP: 606300 | > loss_text_ce: 0.03677492216229439 (0.035923352511014923) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1366923527040312) | > loader_time: 0.0044 (0.004663493428715457)  --> TIME: 2025-02-25 03:25:56 -- STEP: 15342/36938 -- GLOBAL_STEP: 606350 | > loss_text_ce: 0.03852425143122673 (0.03592212183953146) | > loss_mel_ce: 3.8299453258514404 (nan) | > loss: 0.01535107009112835 (nan) | > current_lr: 5e-06 | > step_time: 0.1851 (0.13668801257389937) | > loader_time: 0.0048 (0.004663390700532077)  --> TIME: 2025-02-25 03:26:05 -- STEP: 15392/36938 -- GLOBAL_STEP: 606400 | > loss_text_ce: 0.03712238743901253 (0.035919361871598394) | > loss_mel_ce: 1.4521830081939697 (nan) | > loss: 0.005909942090511322 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1366925933659694) | > loader_time: 0.0047 (0.004663308141385172)  --> TIME: 2025-02-25 03:26:15 -- STEP: 15442/36938 -- GLOBAL_STEP: 606450 | > loss_text_ce: 0.03193480521440506 (0.03591845259531165) | > loss_mel_ce: 2.832021474838257 (nan) | > loss: 0.011364906094968319 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13669585828073352) | > loader_time: 0.0046 (0.004663159587565126)  --> TIME: 2025-02-25 03:26:24 -- STEP: 15492/36938 -- GLOBAL_STEP: 606500 | > loss_text_ce: 0.028508776798844337 (0.035914776171058566) | > loss_mel_ce: 2.797034740447998 (nan) | > loss: 0.011212474666535854 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13670357627028962) | > loader_time: 0.0049 (0.004663199624933978)  --> TIME: 2025-02-25 03:26:33 -- STEP: 15542/36938 -- GLOBAL_STEP: 606550 | > loss_text_ce: 0.03421958163380623 (0.03591137296079013) | > loss_mel_ce: 2.4402129650115967 (nan) | > loss: 0.009819176979362965 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13670368052529094) | > loader_time: 0.0047 (0.004663216716426149)  --> TIME: 2025-02-25 03:26:43 -- STEP: 15592/36938 -- GLOBAL_STEP: 606600 | > loss_text_ce: 0.047072794288396835 (0.035909837226513766) | > loss_mel_ce: 2.3976070880889893 (nan) | > loss: 0.009701111353933811 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.1366981033210938) | > loader_time: 0.0047 (0.004663617473923263)  --> TIME: 2025-02-25 03:26:52 -- STEP: 15642/36938 -- GLOBAL_STEP: 606650 | > loss_text_ce: 0.03855477273464203 (0.03590959039029823) | > loss_mel_ce: 3.6081960201263428 (nan) | > loss: 0.014471233822405338 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13670236778417774) | > loader_time: 0.0046 (0.004663506549210374)  --> TIME: 2025-02-25 03:27:01 -- STEP: 15692/36938 -- GLOBAL_STEP: 606700 | > loss_text_ce: 0.028020020574331284 (0.035914212292933345) | > loss_mel_ce: 3.0849626064300537 (nan) | > loss: 0.01235310547053814 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13670148168508392) | > loader_time: 0.0046 (0.0046636987455699885)  --> TIME: 2025-02-25 03:27:10 -- STEP: 15742/36938 -- GLOBAL_STEP: 606750 | > loss_text_ce: 0.035822272300720215 (0.03591648629272432) | > loss_mel_ce: 1.3441572189331055 (nan) | > loss: 0.005476109217852354 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1366940983640329) | > loader_time: 0.0046 (0.0046639564515613885)  --> TIME: 2025-02-25 03:27:20 -- STEP: 15792/36938 -- GLOBAL_STEP: 606800 | > loss_text_ce: 0.03881847858428955 (0.035919197923961386) | > loss_mel_ce: 0.029672661796212196 (nan) | > loss: 0.00027179025346413255 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13668957878028706) | > loader_time: 0.0045 (0.004663859200574201)  --> TIME: 2025-02-25 03:27:29 -- STEP: 15842/36938 -- GLOBAL_STEP: 606850 | > loss_text_ce: 0.046613771468400955 (0.035915519299568474) | > loss_mel_ce: 3.1341545581817627 (nan) | > loss: 0.012622096575796604 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13667591911511423) | > loader_time: 0.0045 (0.004663772722067736)  --> TIME: 2025-02-25 03:27:38 -- STEP: 15892/36938 -- GLOBAL_STEP: 606900 | > loss_text_ce: 0.03479215130209923 (0.035918255202381184) | > loss_mel_ce: 0.15796661376953125 (nan) | > loss: 0.000764915777835995 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.1366794393790507) | > loader_time: 0.0046 (0.004663728539480998)  --> TIME: 2025-02-25 03:27:47 -- STEP: 15942/36938 -- GLOBAL_STEP: 606950 | > loss_text_ce: 0.037151165306568146 (0.03592023171303129) | > loss_mel_ce: 1.9197744131088257 (nan) | > loss: 0.00776557857170701 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1366638480128912) | > loader_time: 0.0048 (0.0046636764684056435)  --> TIME: 2025-02-25 03:27:56 -- STEP: 15992/36938 -- GLOBAL_STEP: 607000 | > loss_text_ce: 0.03873199224472046 (0.03591965836332568) | > loss_mel_ce: 3.1935043334960938 (nan) | > loss: 0.012826335616409779 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13666035405631755) | > loader_time: 0.0047 (0.004663858415485301)  --> TIME: 2025-02-25 03:28:06 -- STEP: 16042/36938 -- GLOBAL_STEP: 607050 | > loss_text_ce: 0.03530485928058624 (0.035913321312195) | > loss_mel_ce: 3.639146566390991 (nan) | > loss: 0.014581156894564629 (nan) | > current_lr: 5e-06 | > step_time: 0.1707 (0.1366593517310123) | > loader_time: 0.0046 (0.004663813026479564)  --> TIME: 2025-02-25 03:28:15 -- STEP: 16092/36938 -- GLOBAL_STEP: 607100 | > loss_text_ce: 0.03296313062310219 (0.0359112217205384) | > loss_mel_ce: 2.5125210285186768 (nan) | > loss: 0.010101127438247204 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.1366546788442023) | > loader_time: 0.0047 (0.004663899277917705)  --> TIME: 2025-02-25 03:28:24 -- STEP: 16142/36938 -- GLOBAL_STEP: 607150 | > loss_text_ce: 0.03137940168380737 (0.03591058334134205) | > loss_mel_ce: 2.6196353435516357 (nan) | > loss: 0.010519900359213352 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.1366567321610026) | > loader_time: 0.0045 (0.0046639389123870325)  --> TIME: 2025-02-25 03:28:34 -- STEP: 16192/36938 -- GLOBAL_STEP: 607200 | > loss_text_ce: 0.038518037647008896 (0.035909593816231794) | > loss_mel_ce: 2.8178982734680176 (nan) | > loss: 0.011334985494613647 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.1366613942673317) | > loader_time: 0.0046 (0.004664024212968186)  --> TIME: 2025-02-25 03:28:43 -- STEP: 16242/36938 -- GLOBAL_STEP: 607250 | > loss_text_ce: 0.024814549833536148 (0.035904322491343746) | > loss_mel_ce: 2.7315237522125244 (nan) | > loss: 0.010937850922346115 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1366743066307651) | > loader_time: 0.0047 (0.0046642434639584665)  --> TIME: 2025-02-25 03:28:52 -- STEP: 16292/36938 -- GLOBAL_STEP: 607300 | > loss_text_ce: 0.04266979917883873 (0.03590453315225327) | > loss_mel_ce: 2.8064725399017334 (nan) | > loss: 0.01130612101405859 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13666247717145302) | > loader_time: 0.0046 (0.004664146121643442)  --> TIME: 2025-02-25 03:29:02 -- STEP: 16342/36938 -- GLOBAL_STEP: 607350 | > loss_text_ce: 0.03814735263586044 (0.035899460619230404) | > loss_mel_ce: 2.6682229042053223 (nan) | > loss: 0.010739565826952457 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13666563036391885) | > loader_time: 0.0046 (0.004664020677803904)  --> TIME: 2025-02-25 03:29:11 -- STEP: 16392/36938 -- GLOBAL_STEP: 607400 | > loss_text_ce: 0.03771395981311798 (0.03589915597321356) | > loss_mel_ce: 2.2157695293426514 (nan) | > loss: 0.008942395448684692 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13667261351778057) | > loader_time: 0.0047 (0.004663929670481849)  --> TIME: 2025-02-25 03:29:20 -- STEP: 16442/36938 -- GLOBAL_STEP: 607450 | > loss_text_ce: 0.03510523587465286 (0.03590120719703833) | > loss_mel_ce: 3.059939384460449 (nan) | > loss: 0.012281923554837704 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13666822995572242) | > loader_time: 0.0046 (0.004663979611305704)  --> TIME: 2025-02-25 03:29:30 -- STEP: 16492/36938 -- GLOBAL_STEP: 607500 | > loss_text_ce: 0.032564662396907806 (0.03589641166028246) | > loss_mel_ce: 2.7581701278686523 (nan) | > loss: 0.011074344627559185 (nan) | > current_lr: 5e-06 | > step_time: 0.1837 (0.13666547466590784) | > loader_time: 0.0048 (0.004664046235839726)  --> TIME: 2025-02-25 03:29:39 -- STEP: 16542/36938 -- GLOBAL_STEP: 607550 | > loss_text_ce: 0.029256777837872505 (0.03589669261868618) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13667558969371518) | > loader_time: 0.0046 (0.004664214702892593)  --> TIME: 2025-02-25 03:29:48 -- STEP: 16592/36938 -- GLOBAL_STEP: 607600 | > loss_text_ce: 0.025406839326024055 (0.035897362263169946) | > loss_mel_ce: 0.01904296875 (nan) | > loss: 0.00017638813005760312 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.136658005193666) | > loader_time: 0.0045 (0.004664196543891245)  --> TIME: 2025-02-25 03:29:57 -- STEP: 16642/36938 -- GLOBAL_STEP: 607650 | > loss_text_ce: 0.03670506179332733 (0.035892723443154646) | > loss_mel_ce: 2.9338722229003906 (nan) | > loss: 0.01178800594061613 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.1366500693750901) | > loader_time: 0.0048 (0.004664114054228398)  --> TIME: 2025-02-25 03:30:07 -- STEP: 16692/36938 -- GLOBAL_STEP: 607700 | > loss_text_ce: 0.03851683810353279 (0.03589119766333343) | > loss_mel_ce: 2.5385725498199463 (nan) | > loss: 0.010226545855402946 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.1366480255823981) | > loader_time: 0.0046 (0.004664249223626355)  --> TIME: 2025-02-25 03:30:16 -- STEP: 16742/36938 -- GLOBAL_STEP: 607750 | > loss_text_ce: 0.034540899097919464 (0.03589201683222817) | > loss_mel_ce: 2.8793511390686035 (nan) | > loss: 0.011563064530491829 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13665728369090657) | > loader_time: 0.0046 (0.004664436646678949)  --> TIME: 2025-02-25 03:30:25 -- STEP: 16792/36938 -- GLOBAL_STEP: 607800 | > loss_text_ce: 0.036810632795095444 (0.035893106872034765) | > loss_mel_ce: 2.898716688156128 (nan) | > loss: 0.011648918502032757 (nan) | > current_lr: 5e-06 | > step_time: 0.144 (0.13665601311552328) | > loader_time: 0.0047 (0.004664597140998692)  --> TIME: 2025-02-25 03:30:35 -- STEP: 16842/36938 -- GLOBAL_STEP: 607850 | > loss_text_ce: 0.03190646320581436 (0.03589122020769422) | > loss_mel_ce: 3.1351020336151123 (nan) | > loss: 0.012567494064569473 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13665139240545277) | > loader_time: 0.0047 (0.004664587275232454)  --> TIME: 2025-02-25 03:30:44 -- STEP: 16892/36938 -- GLOBAL_STEP: 607900 | > loss_text_ce: 0.027967464178800583 (0.03589345684917665) | > loss_mel_ce: 3.417001724243164 (nan) | > loss: 0.013670513406395912 (nan) | > current_lr: 5e-06 | > step_time: 0.162 (0.1366690759435241) | > loader_time: 0.0046 (0.0046645016035648385)  --> TIME: 2025-02-25 03:30:54 -- STEP: 16942/36938 -- GLOBAL_STEP: 607950 | > loss_text_ce: 0.03811143711209297 (0.03589325112191395) | > loss_mel_ce: 3.1736061573028564 (nan) | > loss: 0.012744911946356297 (nan) | > current_lr: 5e-06 | > step_time: 0.1558 (0.13666674630896067) | > loader_time: 0.0047 (0.004664541304062207)  --> TIME: 2025-02-25 03:31:03 -- STEP: 16992/36938 -- GLOBAL_STEP: 608000 | > loss_text_ce: 0.03236084058880806 (0.035894863758448296) | > loss_mel_ce: 2.7711856365203857 (nan) | > loss: 0.01112518459558487 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13667873528885957) | > loader_time: 0.0046 (0.004664564017634365)  --> TIME: 2025-02-25 03:31:12 -- STEP: 17042/36938 -- GLOBAL_STEP: 608050 | > loss_text_ce: 0.03612026944756508 (0.03589187862612529) | > loss_mel_ce: 0.9642232060432434 (nan) | > loss: 0.003969617187976837 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1366666045327124) | > loader_time: 0.0045 (0.004664708437347126)  --> TIME: 2025-02-25 03:31:22 -- STEP: 17092/36938 -- GLOBAL_STEP: 608100 | > loss_text_ce: 0.04087018966674805 (0.0358916147492268) | > loss_mel_ce: 2.1053059101104736 (nan) | > loss: 0.008516572415828705 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13666465382594936) | > loader_time: 0.0046 (0.004664745845254549)  --> TIME: 2025-02-25 03:31:31 -- STEP: 17142/36938 -- GLOBAL_STEP: 608150 | > loss_text_ce: 0.034747205674648285 (0.03589367898663086) | > loss_mel_ce: 2.7275407314300537 (nan) | > loss: 0.010961459949612617 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13667209361762517) | > loader_time: 0.0046 (0.004664687636908048)  --> TIME: 2025-02-25 03:31:40 -- STEP: 17192/36938 -- GLOBAL_STEP: 608200 | > loss_text_ce: 0.025159448385238647 (0.03589631200157202) | > loss_mel_ce: 3.0991530418395996 (nan) | > loss: 0.012398065999150276 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.1366628865659816) | > loader_time: 0.0047 (0.004664772274661676)  --> TIME: 2025-02-25 03:31:49 -- STEP: 17242/36938 -- GLOBAL_STEP: 608250 | > loss_text_ce: 0.029584869742393494 (0.03589803913663063) | > loss_mel_ce: 2.7813003063201904 (nan) | > loss: 0.011154307052493095 (nan) | > current_lr: 5e-06 | > step_time: 0.1718 (0.13665651686021993) | > loader_time: 0.0049 (0.0046647978055754874)  --> TIME: 2025-02-25 03:31:59 -- STEP: 17292/36938 -- GLOBAL_STEP: 608300 | > loss_text_ce: 0.03232244774699211 (0.0359008000226354) | > loss_mel_ce: 2.247751235961914 (nan) | > loss: 0.009047911502420902 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13665108416826632) | > loader_time: 0.0048 (0.004664806147126868)  --> TIME: 2025-02-25 03:32:08 -- STEP: 17342/36938 -- GLOBAL_STEP: 608350 | > loss_text_ce: 0.04878563806414604 (0.03590759018634982) | > loss_mel_ce: 2.426459550857544 (nan) | > loss: 0.009822402149438858 (nan) | > current_lr: 5e-06 | > step_time: 0.1078 (0.13666560072178982) | > loader_time: 0.0045 (0.004664837001118173)  --> TIME: 2025-02-25 03:32:18 -- STEP: 17392/36938 -- GLOBAL_STEP: 608400 | > loss_text_ce: 0.024702465161681175 (0.035904027245726194) | > loss_mel_ce: 2.721931219100952 (nan) | > loss: 0.010899340733885765 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13665804649397295) | > loader_time: 0.0046 (0.004664898439624909)  --> TIME: 2025-02-25 03:32:27 -- STEP: 17442/36938 -- GLOBAL_STEP: 608450 | > loss_text_ce: 0.03782828897237778 (0.035901805685388526) | > loss_mel_ce: 2.7683472633361816 (nan) | > loss: 0.011135617271065712 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13667151764190674) | > loader_time: 0.0047 (0.004664923767200926)  --> TIME: 2025-02-25 03:32:36 -- STEP: 17492/36938 -- GLOBAL_STEP: 608500 | > loss_text_ce: 0.03329802304506302 (0.03590088897518225) | > loss_mel_ce: 3.817838430404663 (nan) | > loss: 0.015282288193702698 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.1366611141622021) | > loader_time: 0.0048 (0.004664855310853091)  --> TIME: 2025-02-25 03:32:46 -- STEP: 17542/36938 -- GLOBAL_STEP: 608550 | > loss_text_ce: 0.042926691472530365 (0.035905004176372136) | > loss_mel_ce: 2.9418904781341553 (nan) | > loss: 0.011844513937830925 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13666696490562932) | > loader_time: 0.0046 (0.004664988192113672)  --> TIME: 2025-02-25 03:32:55 -- STEP: 17592/36938 -- GLOBAL_STEP: 608600 | > loss_text_ce: 0.028438586741685867 (0.03590453522677533) | > loss_mel_ce: 3.6066040992736816 (nan) | > loss: 0.014424772933125496 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.1366712108911733) | > loader_time: 0.0046 (0.004664903556612073)  --> TIME: 2025-02-25 03:33:05 -- STEP: 17642/36938 -- GLOBAL_STEP: 608650 | > loss_text_ce: 0.03202996030449867 (0.03590615599859304) | > loss_mel_ce: 3.520263671875 (nan) | > loss: 0.014096403494477272 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1366943071899652) | > loader_time: 0.0046 (0.004665024885176081)  --> TIME: 2025-02-25 03:33:14 -- STEP: 17692/36938 -- GLOBAL_STEP: 608700 | > loss_text_ce: 0.027278823778033257 (0.035904564465817745) | > loss_mel_ce: 2.52226185798645 (nan) | > loss: 0.010117226280272007 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13669114978826036) | > loader_time: 0.0046 (0.004665126041564992)  --> TIME: 2025-02-25 03:33:23 -- STEP: 17742/36938 -- GLOBAL_STEP: 608750 | > loss_text_ce: 0.04198582470417023 (0.03590682479489394) | > loss_mel_ce: 3.0224902629852295 (nan) | > loss: 0.012160619720816612 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13668018233452203) | > loader_time: 0.0048 (0.00466502715280527)  --> TIME: 2025-02-25 03:33:33 -- STEP: 17792/36938 -- GLOBAL_STEP: 608800 | > loss_text_ce: 0.03515380620956421 (0.03590645236471946) | > loss_mel_ce: 2.6486353874206543 (nan) | > loss: 0.010649957694113255 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13667733555914213) | > loader_time: 0.0047 (0.004664932719344697)  --> TIME: 2025-02-25 03:33:42 -- STEP: 17842/36938 -- GLOBAL_STEP: 608850 | > loss_text_ce: 0.040994513779878616 (0.03590234406246491) | > loss_mel_ce: 3.1042184829711914 (nan) | > loss: 0.012481003999710083 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.13668464598748134) | > loader_time: 0.0048 (0.004664876418000908)  --> TIME: 2025-02-25 03:33:51 -- STEP: 17892/36938 -- GLOBAL_STEP: 608900 | > loss_text_ce: 0.04088214412331581 (0.0359041020141948) | > loss_mel_ce: 5.34375 (nan) | > loss: 0.021367589011788368 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13668500698841332) | > loader_time: 0.0045 (0.004664901263383881)  --> TIME: 2025-02-25 03:34:00 -- STEP: 17942/36938 -- GLOBAL_STEP: 608950 | > loss_text_ce: 0.04398787021636963 (0.035902362925409104) | > loss_mel_ce: 3.120992422103882 (nan) | > loss: 0.012559446506202221 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13666669016976127) | > loader_time: 0.0046 (0.004664673559137584)  --> TIME: 2025-02-25 03:34:09 -- STEP: 17992/36938 -- GLOBAL_STEP: 609000 | > loss_text_ce: 0.038146067410707474 (0.03590578650898324) | > loss_mel_ce: 2.018007516860962 (nan) | > loss: 0.008159340359270573 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.1366622990875995) | > loader_time: 0.0046 (0.004664507149165854)  --> TIME: 2025-02-25 03:34:19 -- STEP: 18042/36938 -- GLOBAL_STEP: 609050 | > loss_text_ce: 0.04084242880344391 (0.035906342865256756) | > loss_mel_ce: 3.4016501903533936 (nan) | > loss: 0.01366068609058857 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.13666390209792087) | > loader_time: 0.0048 (0.004664458677200724)  --> TIME: 2025-02-25 03:34:28 -- STEP: 18092/36938 -- GLOBAL_STEP: 609100 | > loss_text_ce: 0.045093778520822525 (0.03590030701883066) | > loss_mel_ce: 2.5600407123565674 (nan) | > loss: 0.010337836109101772 (nan) | > current_lr: 5e-06 | > step_time: 0.1499 (0.1366522937729743) | > loader_time: 0.0047 (0.004664398415173506)  --> TIME: 2025-02-25 03:34:37 -- STEP: 18142/36938 -- GLOBAL_STEP: 609150 | > loss_text_ce: 0.03802359104156494 (0.03589677754872742) | > loss_mel_ce: 2.9655940532684326 (nan) | > loss: 0.011919118463993073 (nan) | > current_lr: 5e-06 | > step_time: 0.1793 (0.13666700877897248) | > loader_time: 0.0048 (0.004664415286008098)  --> TIME: 2025-02-25 03:34:47 -- STEP: 18192/36938 -- GLOBAL_STEP: 609200 | > loss_text_ce: 0.03753647580742836 (0.03589782658572238) | > loss_mel_ce: 2.670600652694702 (nan) | > loss: 0.010746575891971588 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13667132939386445) | > loader_time: 0.0046 (0.004664371450322694)  --> TIME: 2025-02-25 03:34:56 -- STEP: 18242/36938 -- GLOBAL_STEP: 609250 | > loss_text_ce: 0.040078386664390564 (0.03589490910284613) | > loss_mel_ce: 3.2190968990325928 (nan) | > loss: 0.012933235615491867 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.13667043412598195) | > loader_time: 0.0046 (0.004664257892507841)  --> TIME: 2025-02-25 03:35:05 -- STEP: 18292/36938 -- GLOBAL_STEP: 609300 | > loss_text_ce: 0.0358198881149292 (0.035895885091298076) | > loss_mel_ce: 3.8043477535247803 (nan) | > loss: 0.015238760970532894 (nan) | > current_lr: 5e-06 | > step_time: 0.1935 (0.13666502610354) | > loader_time: 0.0049 (0.0046643378461916525)  --> TIME: 2025-02-25 03:35:15 -- STEP: 18342/36938 -- GLOBAL_STEP: 609350 | > loss_text_ce: 0.030932709574699402 (0.035896520472207004) | > loss_mel_ce: 3.2912163734436035 (nan) | > loss: 0.01318313181400299 (nan) | > current_lr: 5e-06 | > step_time: 0.1952 (0.13666875901054243) | > loader_time: 0.0049 (0.004664386089567443)  --> TIME: 2025-02-25 03:35:24 -- STEP: 18392/36938 -- GLOBAL_STEP: 609400 | > loss_text_ce: 0.03682935610413551 (0.03589692966632326) | > loss_mel_ce: 1.82012939453125 (nan) | > loss: 0.007368884515017271 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13665095529072968) | > loader_time: 0.0046 (0.004664340956203825)  --> TIME: 2025-02-25 03:35:33 -- STEP: 18442/36938 -- GLOBAL_STEP: 609450 | > loss_text_ce: 0.03746727108955383 (0.035897699705927) | > loss_mel_ce: 2.7245771884918213 (nan) | > loss: 0.010960494168102741 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13666009784016256) | > loader_time: 0.0046 (0.004664361173089676)  --> TIME: 2025-02-25 03:35:42 -- STEP: 18492/36938 -- GLOBAL_STEP: 609500 | > loss_text_ce: 0.02491091936826706 (0.03589388591087474) | > loss_mel_ce: 1.5476943254470825 (nan) | > loss: 0.006240497343242168 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13664226231058213) | > loader_time: 0.0045 (0.004664233693719653)  --> TIME: 2025-02-25 03:35:51 -- STEP: 18542/36938 -- GLOBAL_STEP: 609550 | > loss_text_ce: 0.02420387603342533 (0.03589705507771348) | > loss_mel_ce: 3.4297614097595215 (nan) | > loss: 0.013706211932003498 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13663038112188547) | > loader_time: 0.0046 (0.004664237195009703)  --> TIME: 2025-02-25 03:36:00 -- STEP: 18592/36938 -- GLOBAL_STEP: 609600 | > loss_text_ce: 0.03458530083298683 (0.03589530237642802) | > loss_mel_ce: 2.889749050140381 (nan) | > loss: 0.011604501865804195 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.13662314318800098) | > loader_time: 0.0047 (0.004664375698402292)  --> TIME: 2025-02-25 03:36:09 -- STEP: 18642/36938 -- GLOBAL_STEP: 609650 | > loss_text_ce: 0.030463261529803276 (0.035894046944887965) | > loss_mel_ce: 2.955911159515381 (nan) | > loss: 0.011850692331790924 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13661334170840841) | > loader_time: 0.0049 (0.004664541505818303)  --> TIME: 2025-02-25 03:36:18 -- STEP: 18692/36938 -- GLOBAL_STEP: 609700 | > loss_text_ce: 0.03877422213554382 (0.03589962264569732) | > loss_mel_ce: 1.8079833984375 (nan) | > loss: 0.007328403647989035 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.1365985470743621) | > loader_time: 0.0046 (0.004664531005109981)  --> TIME: 2025-02-25 03:36:28 -- STEP: 18742/36938 -- GLOBAL_STEP: 609750 | > loss_text_ce: 0.03556525334715843 (0.035897658811718025) | > loss_mel_ce: 2.548037052154541 (nan) | > loss: 0.010252390056848526 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13659624074645407) | > loader_time: 0.0046 (0.004664736844320175)  --> TIME: 2025-02-25 03:36:37 -- STEP: 18792/36938 -- GLOBAL_STEP: 609800 | > loss_text_ce: 0.0398649126291275 (0.03589835448527363) | > loss_mel_ce: 2.223480224609375 (nan) | > loss: 0.008981529623270035 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13660494620975555) | > loader_time: 0.0046 (0.004664702865102026)  --> TIME: 2025-02-25 03:36:47 -- STEP: 18842/36938 -- GLOBAL_STEP: 609850 | > loss_text_ce: 0.04557930305600166 (0.035899958122448714) | > loss_mel_ce: 2.856356620788574 (nan) | > loss: 0.011515619233250618 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13660426155521924) | > loader_time: 0.0045 (0.004664824123076854)  --> TIME: 2025-02-25 03:36:56 -- STEP: 18892/36938 -- GLOBAL_STEP: 609900 | > loss_text_ce: 0.04650261625647545 (0.035898493138344256) | > loss_mel_ce: 0.01104736328125 (nan) | > loss: 0.00022837294091004878 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13660201736357663) | > loader_time: 0.0046 (0.004664822261312765)  --> TIME: 2025-02-25 03:37:05 -- STEP: 18942/36938 -- GLOBAL_STEP: 609950 | > loss_text_ce: 0.03147433325648308 (0.03589801355515323) | > loss_mel_ce: 2.8077895641326904 (nan) | > loss: 0.01126692071557045 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13659684087366375) | > loader_time: 0.0046 (0.004664795084798137)  --> TIME: 2025-02-25 03:37:14 -- STEP: 18992/36938 -- GLOBAL_STEP: 610000 | > loss_text_ce: 0.03615088760852814 (0.03589861892720214) | > loss_mel_ce: 3.285470724105835 (nan) | > loss: 0.01318103913217783 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.13660012506946495) | > loader_time: 0.0046 (0.004664766457066778) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_610000.pth  --> TIME: 2025-02-25 03:37:30 -- STEP: 19042/36938 -- GLOBAL_STEP: 610050 | > loss_text_ce: 0.032828133553266525 (0.03589491273775652) | > loss_mel_ce: 2.529653549194336 (nan) | > loss: 0.01016857847571373 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.1366010786617838) | > loader_time: 0.0047 (0.004665063191531971)  --> TIME: 2025-02-25 03:37:39 -- STEP: 19092/36938 -- GLOBAL_STEP: 610100 | > loss_text_ce: 0.03685689717531204 (0.03589371028929894) | > loss_mel_ce: 2.9244918823242188 (nan) | > loss: 0.011751384474337101 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.1365975312858956) | > loader_time: 0.0048 (0.004665035310342029)  --> TIME: 2025-02-25 03:37:48 -- STEP: 19142/36938 -- GLOBAL_STEP: 610150 | > loss_text_ce: 0.0365535244345665 (0.03589341184153522) | > loss_mel_ce: 2.9866318702697754 (nan) | > loss: 0.011996768414974213 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13659928286166445) | > loader_time: 0.0046 (0.0046649869239862125)  --> TIME: 2025-02-25 03:37:58 -- STEP: 19192/36938 -- GLOBAL_STEP: 610200 | > loss_text_ce: 0.030501021072268486 (0.035896553887236976) | > loss_mel_ce: 3.1423373222351074 (nan) | > loss: 0.012590629048645496 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13659872851952085) | > loader_time: 0.0046 (0.004664964244086427)  --> TIME: 2025-02-25 03:38:07 -- STEP: 19242/36938 -- GLOBAL_STEP: 610250 | > loss_text_ce: 0.02742563933134079 (0.0358980378896796) | > loss_mel_ce: 1.6340593099594116 (nan) | > loss: 0.006593194790184498 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.1366023984994693) | > loader_time: 0.0045 (0.0046648796178882955)  --> TIME: 2025-02-25 03:38:16 -- STEP: 19292/36938 -- GLOBAL_STEP: 610300 | > loss_text_ce: 0.0294233076274395 (0.035896349865674575) | > loss_mel_ce: 2.6190521717071533 (nan) | > loss: 0.010509823448956013 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13659552652361698) | > loader_time: 0.0046 (0.0046648698403779455)  --> TIME: 2025-02-25 03:38:26 -- STEP: 19342/36938 -- GLOBAL_STEP: 610350 | > loss_text_ce: 0.022837558761239052 (0.035895234565868135) | > loss_mel_ce: 1.8215713500976562 (nan) | > loss: 0.00731908343732357 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13660164658920407) | > loader_time: 0.0045 (0.0046650490658740095)  --> TIME: 2025-02-25 03:38:35 -- STEP: 19392/36938 -- GLOBAL_STEP: 610400 | > loss_text_ce: 0.04900111258029938 (0.03589690793489075) | > loss_mel_ce: 1.3123672008514404 (nan) | > loss: 0.005402255337685347 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13658745237702907) | > loader_time: 0.0046 (0.004665238543017081)  --> TIME: 2025-02-25 03:38:44 -- STEP: 19442/36938 -- GLOBAL_STEP: 610450 | > loss_text_ce: 0.04162262752652168 (0.0358979591726431) | > loss_mel_ce: 3.0473685264587402 (nan) | > loss: 0.012257901951670647 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13658173028339377) | > loader_time: 0.0045 (0.004665122161173012)  --> TIME: 2025-02-25 03:38:53 -- STEP: 19492/36938 -- GLOBAL_STEP: 610500 | > loss_text_ce: 0.04638664051890373 (0.035899177594179) | > loss_mel_ce: 3.3174080848693848 (nan) | > loss: 0.013348393142223358 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13657080091149146) | > loader_time: 0.0046 (0.004665311445036195)  --> TIME: 2025-02-25 03:39:02 -- STEP: 19542/36938 -- GLOBAL_STEP: 610550 | > loss_text_ce: 0.030934922397136688 (0.03589585844099255) | > loss_mel_ce: 3.169585704803467 (nan) | > loss: 0.012700478546321392 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.13656576967400358) | > loader_time: 0.0052 (0.0046653108750037715)  --> TIME: 2025-02-25 03:39:11 -- STEP: 19592/36938 -- GLOBAL_STEP: 610600 | > loss_text_ce: 0.030446507036685944 (0.03589389138827834) | > loss_mel_ce: 2.7698466777801514 (nan) | > loss: 0.011112275533378124 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.13656870832780021) | > loader_time: 0.0048 (0.004665292321832486)  --> TIME: 2025-02-25 03:39:21 -- STEP: 19642/36938 -- GLOBAL_STEP: 610650 | > loss_text_ce: 0.026504376903176308 (0.03589216159345964) | > loss_mel_ce: 3.132960557937622 (nan) | > loss: 0.012537559494376183 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13657017094505547) | > loader_time: 0.0047 (0.004665223732340322)  --> TIME: 2025-02-25 03:39:30 -- STEP: 19692/36938 -- GLOBAL_STEP: 610700 | > loss_text_ce: 0.029536275193095207 (0.03589093934415599) | > loss_mel_ce: 0.8327111601829529 (nan) | > loss: 0.0034216169733554125 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.13654637213235596) | > loader_time: 0.0046 (0.004665101189548442)  --> TIME: 2025-02-25 03:39:39 -- STEP: 19742/36938 -- GLOBAL_STEP: 610750 | > loss_text_ce: 0.040555741637945175 (0.03589056439417698) | > loss_mel_ce: 3.176978349685669 (nan) | > loss: 0.012767992913722992 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.13654960154665793) | > loader_time: 0.0049 (0.00466522641752449)  --> TIME: 2025-02-25 03:39:48 -- STEP: 19792/36938 -- GLOBAL_STEP: 610800 | > loss_text_ce: 0.03637170419096947 (0.03589160425442411) | > loss_mel_ce: 3.180344820022583 (nan) | > loss: 0.012764749117195606 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.13654453692266574) | > loader_time: 0.0048 (0.004665182353800686)  --> TIME: 2025-02-25 03:39:58 -- STEP: 19842/36938 -- GLOBAL_STEP: 610850 | > loss_text_ce: 0.030456161126494408 (0.035889010767429425) | > loss_mel_ce: 2.746006727218628 (nan) | > loss: 0.011017709970474243 (nan) | > current_lr: 5e-06 | > step_time: 0.1118 (0.13655476758297644) | > loader_time: 0.0046 (0.0046654862629863705)  --> TIME: 2025-02-25 03:40:07 -- STEP: 19892/36938 -- GLOBAL_STEP: 610900 | > loss_text_ce: 0.03417612612247467 (0.035885696021456616) | > loss_mel_ce: 3.11948299407959 (nan) | > loss: 0.01251452136784792 (nan) | > current_lr: 5e-06 | > step_time: 0.1782 (0.13655606210051965) | > loader_time: 0.0048 (0.004665502019506704)  --> TIME: 2025-02-25 03:40:17 -- STEP: 19942/36938 -- GLOBAL_STEP: 610950 | > loss_text_ce: 0.03924313560128212 (0.03588666083012251) | > loss_mel_ce: 2.494178533554077 (nan) | > loss: 0.010053261183202267 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.13656010328617432) | > loader_time: 0.0048 (0.00466551340495483)  --> TIME: 2025-02-25 03:40:26 -- STEP: 19992/36938 -- GLOBAL_STEP: 611000 | > loss_text_ce: 0.048783715814352036 (0.03588400398190595) | > loss_mel_ce: 3.200753688812256 (nan) | > loss: 0.012894990853965282 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13656528794369502) | > loader_time: 0.0049 (0.004665592244836305)  --> TIME: 2025-02-25 03:40:36 -- STEP: 20042/36938 -- GLOBAL_STEP: 611050 | > loss_text_ce: 0.036764830350875854 (0.03588046845444182) | > loss_mel_ce: 2.920733690261841 (nan) | > loss: 0.011736106127500534 (nan) | > current_lr: 5e-06 | > step_time: 0.185 (0.13656848575668898) | > loader_time: 0.0048 (0.004665504885628683)  --> TIME: 2025-02-25 03:40:45 -- STEP: 20092/36938 -- GLOBAL_STEP: 611100 | > loss_text_ce: 0.04293463006615639 (0.035883140481196854) | > loss_mel_ce: 3.3999128341674805 (nan) | > loss: 0.013662094250321388 (nan) | > current_lr: 5e-06 | > step_time: 0.1773 (0.13657437926764576) | > loader_time: 0.0048 (0.004665509557524629)  --> TIME: 2025-02-25 03:40:55 -- STEP: 20142/36938 -- GLOBAL_STEP: 611150 | > loss_text_ce: 0.041999753564596176 (0.035886157173472634) | > loss_mel_ce: 3.0286335945129395 (nan) | > loss: 0.012185053899884224 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.136585196900919) | > loader_time: 0.0047 (0.004665650638185998)  --> TIME: 2025-02-25 03:41:04 -- STEP: 20192/36938 -- GLOBAL_STEP: 611200 | > loss_text_ce: 0.04314544051885605 (0.03588502849800868) | > loss_mel_ce: 3.090675115585327 (nan) | > loss: 0.012435796670615673 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.1365933825399151) | > loader_time: 0.0047 (0.004665702262601069)  --> TIME: 2025-02-25 03:41:13 -- STEP: 20242/36938 -- GLOBAL_STEP: 611250 | > loss_text_ce: 0.05029474198818207 (0.03588408629569418) | > loss_mel_ce: 1.6648445129394531 (nan) | > loss: 0.006806108634918928 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13659216926310086) | > loader_time: 0.0045 (0.0046656343520199905)  --> TIME: 2025-02-25 03:41:23 -- STEP: 20292/36938 -- GLOBAL_STEP: 611300 | > loss_text_ce: 0.04143010079860687 (0.035882612312145935) | > loss_mel_ce: 2.977876901626587 (nan) | > loss: 0.011981377378106117 (nan) | > current_lr: 5e-06 | > step_time: 0.1788 (0.1365913758963114) | > loader_time: 0.0049 (0.004665635956501589)  --> TIME: 2025-02-25 03:41:32 -- STEP: 20342/36938 -- GLOBAL_STEP: 611350 | > loss_text_ce: 0.03492379188537598 (0.035882945837444176) | > loss_mel_ce: 2.821780204772949 (nan) | > loss: 0.011336127296090126 (nan) | > current_lr: 5e-06 | > step_time: 0.1872 (0.13660411179071524) | > loader_time: 0.0049 (0.0046657019455680565)  --> TIME: 2025-02-25 03:41:42 -- STEP: 20392/36938 -- GLOBAL_STEP: 611400 | > loss_text_ce: 0.040521591901779175 (0.03588265811669313) | > loss_mel_ce: 0.9771175384521484 (nan) | > loss: 0.004038251005113125 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13660159802941885) | > loader_time: 0.0045 (0.004665640802091589)  --> TIME: 2025-02-25 03:41:51 -- STEP: 20442/36938 -- GLOBAL_STEP: 611450 | > loss_text_ce: 0.03397972881793976 (0.03588312873524761) | > loss_mel_ce: 3.448436737060547 (nan) | > loss: 0.013819113373756409 (nan) | > current_lr: 5e-06 | > step_time: 0.1628 (0.13661662680378908) | > loader_time: 0.0049 (0.00466565850919128)  --> TIME: 2025-02-25 03:42:00 -- STEP: 20492/36938 -- GLOBAL_STEP: 611500 | > loss_text_ce: 0.03196866810321808 (0.03588081611736291) | > loss_mel_ce: 2.970431327819824 (nan) | > loss: 0.011914285831153393 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13661138319815563) | > loader_time: 0.0046 (0.004665676118246442)  --> TIME: 2025-02-25 03:42:10 -- STEP: 20542/36938 -- GLOBAL_STEP: 611550 | > loss_text_ce: 0.03827250748872757 (0.035882892895287656) | > loss_mel_ce: 4.266528606414795 (nan) | > loss: 0.017082544043660164 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13661130884120543) | > loader_time: 0.0046 (0.004665727683137793)  --> TIME: 2025-02-25 03:42:19 -- STEP: 20592/36938 -- GLOBAL_STEP: 611600 | > loss_text_ce: 0.03314964473247528 (0.035879835148020435) | > loss_mel_ce: 3.188183546066284 (nan) | > loss: 0.012783069163560867 (nan) | > current_lr: 5e-06 | > step_time: 0.1472 (0.13661234270674147) | > loader_time: 0.0047 (0.00466575001734768)  --> TIME: 2025-02-25 03:42:28 -- STEP: 20642/36938 -- GLOBAL_STEP: 611650 | > loss_text_ce: 0.03795682266354561 (0.0358817172974415) | > loss_mel_ce: 3.349266529083252 (nan) | > loss: 0.013441362418234348 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.13661543808522972) | > loader_time: 0.0046 (0.004665728595272743)  --> TIME: 2025-02-25 03:42:38 -- STEP: 20692/36938 -- GLOBAL_STEP: 611700 | > loss_text_ce: 0.044838059693574905 (0.03588009901262298) | > loss_mel_ce: 3.32863450050354 (nan) | > loss: 0.013386796228587627 (nan) | > current_lr: 5e-06 | > step_time: 0.15 (0.13662553100873492) | > loader_time: 0.0047 (0.00466570849808553)  --> TIME: 2025-02-25 03:42:47 -- STEP: 20742/36938 -- GLOBAL_STEP: 611750 | > loss_text_ce: 0.02616729587316513 (0.035879239491779055) | > loss_mel_ce: 3.307662010192871 (nan) | > loss: 0.013229482807219028 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13662843163523092) | > loader_time: 0.0047 (0.004665721946736905)  --> TIME: 2025-02-25 03:42:56 -- STEP: 20792/36938 -- GLOBAL_STEP: 611800 | > loss_text_ce: 0.04105447605252266 (0.035878511643085025) | > loss_mel_ce: 3.2275164127349854 (nan) | > loss: 0.012970520183444023 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.136617080575459) | > loader_time: 0.0048 (0.004665739389968498)  --> TIME: 2025-02-25 03:43:06 -- STEP: 20842/36938 -- GLOBAL_STEP: 611850 | > loss_text_ce: 0.03445804491639137 (0.03587776455460173) | > loss_mel_ce: 1.6650949716567993 (nan) | > loss: 0.00674425857141614 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13661113926755972) | > loader_time: 0.0046 (0.004665701886466156)  --> TIME: 2025-02-25 03:43:15 -- STEP: 20892/36938 -- GLOBAL_STEP: 611900 | > loss_text_ce: 0.03215514495968819 (0.03587973621719694) | > loss_mel_ce: 3.0933854579925537 (nan) | > loss: 0.012402939610183239 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13661018791678903) | > loader_time: 0.0048 (0.004665649601401161)  --> TIME: 2025-02-25 03:43:24 -- STEP: 20942/36938 -- GLOBAL_STEP: 611950 | > loss_text_ce: 0.04128099977970123 (0.03587710141542178) | > loss_mel_ce: 2.352517604827881 (nan) | > loss: 0.00949920155107975 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13660533027905714) | > loader_time: 0.0046 (0.004665560793391657)  --> TIME: 2025-02-25 03:43:33 -- STEP: 20992/36938 -- GLOBAL_STEP: 612000 | > loss_text_ce: 0.03887025639414787 (0.035875693856235756) | > loss_mel_ce: 2.9112844467163086 (nan) | > loss: 0.011706964112818241 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1366069643281213) | > loader_time: 0.0046 (0.004665554909989599)  --> TIME: 2025-02-25 03:43:43 -- STEP: 21042/36938 -- GLOBAL_STEP: 612050 | > loss_text_ce: 0.03308285027742386 (0.03587661912596947) | > loss_mel_ce: 3.575927734375 (nan) | > loss: 0.014321471564471722 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13660815780510152) | > loader_time: 0.0045 (0.0046655553770251134)  --> TIME: 2025-02-25 03:43:52 -- STEP: 21092/36938 -- GLOBAL_STEP: 612100 | > loss_text_ce: 0.034519851207733154 (0.035881986529205004) | > loss_mel_ce: 3.592343807220459 (nan) | > loss: 0.014392317272722721 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.13660479653340504) | > loader_time: 0.0047 (0.004665539214038177)  --> TIME: 2025-02-25 03:44:01 -- STEP: 21142/36938 -- GLOBAL_STEP: 612150 | > loss_text_ce: 0.039144404232501984 (0.0358838196590128) | > loss_mel_ce: 2.7869410514831543 (nan) | > loss: 0.011214626021683216 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.1366029967423607) | > loader_time: 0.0046 (0.004665518233277873)  --> TIME: 2025-02-25 03:44:10 -- STEP: 21192/36938 -- GLOBAL_STEP: 612200 | > loss_text_ce: 0.030204584822058678 (0.035883530963664895) | > loss_mel_ce: 2.6529345512390137 (nan) | > loss: 0.01064737793058157 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.1365996712713966) | > loader_time: 0.0047 (0.004665463825314392)  --> TIME: 2025-02-25 03:44:20 -- STEP: 21242/36938 -- GLOBAL_STEP: 612250 | > loss_text_ce: 0.037372834980487823 (0.035883088942177496) | > loss_mel_ce: 2.9057865142822266 (nan) | > loss: 0.011679204180836678 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13660884911411658) | > loader_time: 0.0046 (0.004665506188001417)  --> TIME: 2025-02-25 03:44:29 -- STEP: 21292/36938 -- GLOBAL_STEP: 612300 | > loss_text_ce: 0.0362001471221447 (0.035880906909785175) | > loss_mel_ce: 2.985896348953247 (nan) | > loss: 0.011992447078227997 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13662127649651884) | > loader_time: 0.0047 (0.0046654635637567495)  --> TIME: 2025-02-25 03:44:39 -- STEP: 21342/36938 -- GLOBAL_STEP: 612350 | > loss_text_ce: 0.03258185461163521 (0.03587844127334586) | > loss_mel_ce: 2.8270716667175293 (nan) | > loss: 0.011347832158207893 (nan) | > current_lr: 5e-06 | > step_time: 0.186 (0.13662415292110813) | > loader_time: 0.0051 (0.0046654462188736485)  --> TIME: 2025-02-25 03:44:48 -- STEP: 21392/36938 -- GLOBAL_STEP: 612400 | > loss_text_ce: 0.03795943781733513 (0.03587634983074063) | > loss_mel_ce: 2.9168362617492676 (nan) | > loss: 0.011725380085408688 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.1366225293793596) | > loader_time: 0.0047 (0.004665438684850115)  --> TIME: 2025-02-25 03:44:57 -- STEP: 21442/36938 -- GLOBAL_STEP: 612450 | > loss_text_ce: 0.0301799438893795 (0.035875887711307286) | > loss_mel_ce: 3.0227010250091553 (nan) | > loss: 0.012114607729017735 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13661959179968106) | > loader_time: 0.0046 (0.004665508920516387)  --> TIME: 2025-02-25 03:45:07 -- STEP: 21492/36938 -- GLOBAL_STEP: 612500 | > loss_text_ce: 0.036872100085020065 (0.03587570750383677) | > loss_mel_ce: 0.011932373046875 (nan) | > loss: 0.00019366855849511921 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13661545801269162) | > loader_time: 0.0046 (0.004665803214857182)  --> TIME: 2025-02-25 03:45:16 -- STEP: 21542/36938 -- GLOBAL_STEP: 612550 | > loss_text_ce: 0.03206106647849083 (0.035874812964110395) | > loss_mel_ce: 3.051717758178711 (nan) | > loss: 0.012237218208611012 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1366138616629529) | > loader_time: 0.0048 (0.0046656867408805675)  --> TIME: 2025-02-25 03:45:25 -- STEP: 21592/36938 -- GLOBAL_STEP: 612600 | > loss_text_ce: 0.0417000949382782 (0.03587588089170225) | > loss_mel_ce: 3.228013515472412 (nan) | > loss: 0.012975054793059826 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13661548811093435) | > loader_time: 0.0046 (0.004665721396951164)  --> TIME: 2025-02-25 03:45:35 -- STEP: 21642/36938 -- GLOBAL_STEP: 612650 | > loss_text_ce: 0.03857109323143959 (0.03587525408401182) | > loss_mel_ce: 2.900506019592285 (nan) | > loss: 0.011663004755973816 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13661126130478318) | > loader_time: 0.0046 (0.004665787113582069)  --> TIME: 2025-02-25 03:45:44 -- STEP: 21692/36938 -- GLOBAL_STEP: 612700 | > loss_text_ce: 0.04422374069690704 (0.03587724746698795) | > loss_mel_ce: 2.8130319118499756 (nan) | > loss: 0.011338316835463047 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.13661048475191948) | > loader_time: 0.0046 (0.004665959019865831)  --> TIME: 2025-02-25 03:45:53 -- STEP: 21742/36938 -- GLOBAL_STEP: 612750 | > loss_text_ce: 0.03074961155653 (0.035877577039320235) | > loss_mel_ce: 2.1128437519073486 (nan) | > loss: 0.008506323210895061 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.1366116087018436) | > loader_time: 0.0046 (0.004665831032129174)  --> TIME: 2025-02-25 03:46:02 -- STEP: 21792/36938 -- GLOBAL_STEP: 612800 | > loss_text_ce: 0.038709308952093124 (0.03588024833022993) | > loss_mel_ce: 3.0692803859710693 (nan) | > loss: 0.012333293445408344 (nan) | > current_lr: 5e-06 | > step_time: 0.1691 (0.13659800583614262) | > loader_time: 0.0051 (0.004665852183812517)  --> TIME: 2025-02-25 03:46:12 -- STEP: 21842/36938 -- GLOBAL_STEP: 612850 | > loss_text_ce: 0.04542258009314537 (0.03588074144773631) | > loss_mel_ce: 1.8279470205307007 (nan) | > loss: 0.007434006780385971 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13659809535375544) | > loader_time: 0.0045 (0.004665867169581649)  --> TIME: 2025-02-25 03:46:21 -- STEP: 21892/36938 -- GLOBAL_STEP: 612900 | > loss_text_ce: 0.04196795076131821 (0.035880043324283184) | > loss_mel_ce: 3.19397234916687 (nan) | > loss: 0.01284103374928236 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.13660198751760047) | > loader_time: 0.0047 (0.0046659374550711875)  --> TIME: 2025-02-25 03:46:30 -- STEP: 21942/36938 -- GLOBAL_STEP: 612950 | > loss_text_ce: 0.024229908362030983 (0.03587760667881689) | > loss_mel_ce: 3.0517499446868896 (nan) | > loss: 0.01220627035945654 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13660713190063561) | > loader_time: 0.0046 (0.004665885081581103)  --> TIME: 2025-02-25 03:46:40 -- STEP: 21992/36938 -- GLOBAL_STEP: 613000 | > loss_text_ce: 0.027255214750766754 (0.03588145075374502) | > loss_mel_ce: 2.1322543621063232 (nan) | > loss: 0.008569483645260334 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13661688658357615) | > loader_time: 0.0045 (0.004665960524896576)  --> TIME: 2025-02-25 03:46:49 -- STEP: 22042/36938 -- GLOBAL_STEP: 613050 | > loss_text_ce: 0.03787523880600929 (0.03588079268713736) | > loss_mel_ce: 2.6330742835998535 (nan) | > loss: 0.010599005967378616 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.1366061199685962) | > loader_time: 0.0046 (0.004666021705030719)  --> TIME: 2025-02-25 03:46:59 -- STEP: 22092/36938 -- GLOBAL_STEP: 613100 | > loss_text_ce: 0.0358106791973114 (0.035879571294707056) | > loss_mel_ce: 3.377002716064453 (nan) | > loss: 0.013542911037802696 (nan) | > current_lr: 5e-06 | > step_time: 0.1828 (0.13661827275702465) | > loader_time: 0.0048 (0.004666124816047055)  --> TIME: 2025-02-25 03:47:08 -- STEP: 22142/36938 -- GLOBAL_STEP: 613150 | > loss_text_ce: 0.03182660788297653 (0.03587564454689746) | > loss_mel_ce: 2.971251964569092 (nan) | > loss: 0.011916978284716606 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.1366150409653515) | > loader_time: 0.0046 (0.004666043193604544)  --> TIME: 2025-02-25 03:47:17 -- STEP: 22192/36938 -- GLOBAL_STEP: 613200 | > loss_text_ce: 0.042294859886169434 (0.03587226372732684) | > loss_mel_ce: 3.509810447692871 (nan) | > loss: 0.014095657505095005 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.1366154955674519) | > loader_time: 0.0046 (0.004666092826379835)  --> TIME: 2025-02-25 03:47:26 -- STEP: 22242/36938 -- GLOBAL_STEP: 613250 | > loss_text_ce: 0.034910887479782104 (0.03587264217821158) | > loss_mel_ce: 1.4497884511947632 (nan) | > loss: 0.005891664419323206 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13660996050920793) | > loader_time: 0.0047 (0.004665914590319503)  --> TIME: 2025-02-25 03:47:36 -- STEP: 22292/36938 -- GLOBAL_STEP: 613300 | > loss_text_ce: 0.030465995892882347 (0.03586807264641376) | > loss_mel_ce: 2.944812059402466 (nan) | > loss: 0.011806660331785679 (nan) | > current_lr: 5e-06 | > step_time: 0.1769 (0.1366238533413086) | > loader_time: 0.0047 (0.004665932160358167)  --> TIME: 2025-02-25 03:47:45 -- STEP: 22342/36938 -- GLOBAL_STEP: 613350 | > loss_text_ce: 0.012562145479023457 (0.03586497101983573) | > loss_mel_ce: 1.8956856727600098 (nan) | > loss: 0.0075724124908447266 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1366238075962697) | > loader_time: 0.0045 (0.004665976191322547)  --> TIME: 2025-02-25 03:47:55 -- STEP: 22392/36938 -- GLOBAL_STEP: 613400 | > loss_text_ce: 0.041663773357868195 (0.03586731237052689) | > loss_mel_ce: 2.5027236938476562 (nan) | > loss: 0.0100967762991786 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13662792623532788) | > loader_time: 0.0046 (0.004666024295293079)  --> TIME: 2025-02-25 03:48:04 -- STEP: 22442/36938 -- GLOBAL_STEP: 613450 | > loss_text_ce: 0.03328819200396538 (0.035865559447836856) | > loss_mel_ce: 3.0929157733917236 (nan) | > loss: 0.012405572459101677 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.1366303678863917) | > loader_time: 0.0047 (0.004665944880315644)  --> TIME: 2025-02-25 03:48:14 -- STEP: 22492/36938 -- GLOBAL_STEP: 613500 | > loss_text_ce: 0.04155714437365532 (0.03586678698378328) | > loss_mel_ce: 2.5571792125701904 (nan) | > loss: 0.010312446393072605 (nan) | > current_lr: 5e-06 | > step_time: 0.1507 (0.13663229737950827) | > loader_time: 0.0047 (0.004666027587311947)  --> TIME: 2025-02-25 03:48:23 -- STEP: 22542/36938 -- GLOBAL_STEP: 613550 | > loss_text_ce: 0.035705938935279846 (0.03586804237836063) | > loss_mel_ce: 3.4273927211761475 (nan) | > loss: 0.013742456212639809 (nan) | > current_lr: 5e-06 | > step_time: 0.1697 (0.1366395596317295) | > loader_time: 0.0048 (0.0046659682004541285)  --> TIME: 2025-02-25 03:48:33 -- STEP: 22592/36938 -- GLOBAL_STEP: 613600 | > loss_text_ce: 0.033401668071746826 (0.035872513052948286) | > loss_mel_ce: 2.88730788230896 (nan) | > loss: 0.011590118519961834 (nan) | > current_lr: 5e-06 | > step_time: 0.1582 (0.13663936975193028) | > loader_time: 0.0047 (0.0046659190703721674)  --> TIME: 2025-02-25 03:48:42 -- STEP: 22642/36938 -- GLOBAL_STEP: 613650 | > loss_text_ce: 0.054256290197372437 (0.03587514300425132) | > loss_mel_ce: 2.892756223678589 (nan) | > loss: 0.011694494634866714 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13663052075238316) | > loader_time: 0.0046 (0.00466586203870394)  --> TIME: 2025-02-25 03:48:51 -- STEP: 22692/36938 -- GLOBAL_STEP: 613700 | > loss_text_ce: 0.035089872777462006 (0.03587138205428449) | > loss_mel_ce: 2.769228935241699 (nan) | > loss: 0.011128250509500504 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.1366310130135625) | > loader_time: 0.0047 (0.0046657435628822816)  --> TIME: 2025-02-25 03:49:01 -- STEP: 22742/36938 -- GLOBAL_STEP: 613750 | > loss_text_ce: 0.03466572239995003 (0.035870572361167856) | > loss_mel_ce: 3.403757333755493 (nan) | > loss: 0.013644536957144737 (nan) | > current_lr: 5e-06 | > step_time: 0.1922 (0.13664051977125174) | > loader_time: 0.0048 (0.004665670708567501)  --> TIME: 2025-02-25 03:49:10 -- STEP: 22792/36938 -- GLOBAL_STEP: 613800 | > loss_text_ce: 0.03543779253959656 (0.03587267049396984) | > loss_mel_ce: 1.3687540292739868 (nan) | > loss: 0.0055721900425851345 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.1366362826641751) | > loader_time: 0.0045 (0.0046655983517319794)  --> TIME: 2025-02-25 03:49:19 -- STEP: 22842/36938 -- GLOBAL_STEP: 613850 | > loss_text_ce: 0.04085307568311691 (0.0358702674451498) | > loss_mel_ce: 3.5149312019348145 (nan) | > loss: 0.014110255986452103 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13662546346509238) | > loader_time: 0.0046 (0.0046655406739809745)  --> TIME: 2025-02-25 03:49:28 -- STEP: 22892/36938 -- GLOBAL_STEP: 613900 | > loss_text_ce: 0.03144320100545883 (0.03586951773236102) | > loss_mel_ce: 2.9250335693359375 (nan) | > loss: 0.011732051149010658 (nan) | > current_lr: 5e-06 | > step_time: 0.1533 (0.1366193047483001) | > loader_time: 0.0046 (0.00466554148847483)  --> TIME: 2025-02-25 03:49:37 -- STEP: 22942/36938 -- GLOBAL_STEP: 613950 | > loss_text_ce: 0.024913344532251358 (0.03586588079942256) | > loss_mel_ce: 2.8464975357055664 (nan) | > loss: 0.01139448769390583 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13661207189203234) | > loader_time: 0.0045 (0.004665429533312543)  --> TIME: 2025-02-25 03:49:47 -- STEP: 22992/36938 -- GLOBAL_STEP: 614000 | > loss_text_ce: 0.0307175200432539 (0.03586482285920783) | > loss_mel_ce: 3.519864797592163 (nan) | > loss: 0.01408961322158575 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.1366119706431607) | > loader_time: 0.0047 (0.004665381682773564)  --> TIME: 2025-02-25 03:49:55 -- STEP: 23042/36938 -- GLOBAL_STEP: 614050 | > loss_text_ce: 0.039147622883319855 (0.03586400875621344) | > loss_mel_ce: 2.070187568664551 (nan) | > loss: 0.00837037805467844 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13659481991497258) | > loader_time: 0.0045 (0.004665309393036526)  --> TIME: 2025-02-25 03:50:05 -- STEP: 23092/36938 -- GLOBAL_STEP: 614100 | > loss_text_ce: 0.033085018396377563 (0.03586246548380977) | > loss_mel_ce: 2.5619208812713623 (nan) | > loss: 0.010297643020749092 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13659462423066854) | > loader_time: 0.0046 (0.00466525263499702)  --> TIME: 2025-02-25 03:50:14 -- STEP: 23142/36938 -- GLOBAL_STEP: 614150 | > loss_text_ce: 0.03871433809399605 (0.035865616423625686) | > loss_mel_ce: 3.390625 (nan) | > loss: 0.013608490116894245 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13659208905324918) | > loader_time: 0.0048 (0.004665379402250319)  --> TIME: 2025-02-25 03:50:23 -- STEP: 23192/36938 -- GLOBAL_STEP: 614200 | > loss_text_ce: 0.04046608880162239 (0.035862277307088956) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1071 (0.1365805070188051) | > loader_time: 0.0044 (0.004665224037732948)  --> TIME: 2025-02-25 03:50:32 -- STEP: 23242/36938 -- GLOBAL_STEP: 614250 | > loss_text_ce: 0.034682825207710266 (0.03586395071959135) | > loss_mel_ce: 2.5141398906707764 (nan) | > loss: 0.010114376433193684 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13656902667136572) | > loader_time: 0.0045 (0.004665169327304819)  --> TIME: 2025-02-25 03:50:41 -- STEP: 23292/36938 -- GLOBAL_STEP: 614300 | > loss_text_ce: 0.03879648447036743 (0.03586187512820421) | > loss_mel_ce: 2.869152307510376 (nan) | > loss: 0.011539479717612267 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13657219250669325) | > loader_time: 0.0046 (0.004665170372218261)  --> TIME: 2025-02-25 03:50:51 -- STEP: 23342/36938 -- GLOBAL_STEP: 614350 | > loss_text_ce: 0.023037435486912727 (0.03585924359920026) | > loss_mel_ce: 1.8796950578689575 (nan) | > loss: 0.007550526410341263 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13657519458018624) | > loader_time: 0.0046 (0.0046651089123002)  --> TIME: 2025-02-25 03:51:01 -- STEP: 23392/36938 -- GLOBAL_STEP: 614400 | > loss_text_ce: 0.0343235544860363 (0.03585862550377521) | > loss_mel_ce: 0.027587890625 (nan) | > loss: 0.0002456803631503135 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.1365907230043701) | > loader_time: 0.0046 (0.004665162979985721)  --> TIME: 2025-02-25 03:51:10 -- STEP: 23442/36938 -- GLOBAL_STEP: 614450 | > loss_text_ce: 0.025144658982753754 (0.03585768157335556) | > loss_mel_ce: 0.0103759765625 (nan) | > loss: 0.0001409549149684608 (nan) | > current_lr: 5e-06 | > step_time: 0.1267 (0.1365913173035845) | > loader_time: 0.0046 (0.00466507761239009)  --> TIME: 2025-02-25 03:51:19 -- STEP: 23492/36938 -- GLOBAL_STEP: 614500 | > loss_text_ce: 0.02879338525235653 (0.035855111521916366) | > loss_mel_ce: 1.7613844871520996 (nan) | > loss: 0.007103880867362022 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13659245108656845) | > loader_time: 0.0045 (0.004664898618977095)  --> TIME: 2025-02-25 03:51:29 -- STEP: 23542/36938 -- GLOBAL_STEP: 614550 | > loss_text_ce: 0.0313011035323143 (0.03585473170908066) | > loss_mel_ce: 3.2502541542053223 (nan) | > loss: 0.01302204467356205 (nan) | > current_lr: 5e-06 | > step_time: 0.1632 (0.13659464669869628) | > loader_time: 0.0047 (0.004664805850767155)  --> TIME: 2025-02-25 03:51:38 -- STEP: 23592/36938 -- GLOBAL_STEP: 614600 | > loss_text_ce: 0.04580782726407051 (0.035857216191380256) | > loss_mel_ce: 3.1149394512176514 (nan) | > loss: 0.012542648240923882 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13658801598361284) | > loader_time: 0.0047 (0.004664722419504633)  --> TIME: 2025-02-25 03:51:47 -- STEP: 23642/36938 -- GLOBAL_STEP: 614650 | > loss_text_ce: 0.04245714470744133 (0.0358562702789318) | > loss_mel_ce: 1.2148350477218628 (nan) | > loss: 0.004989254754036665 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.1365943804235787) | > loader_time: 0.0046 (0.004664596038142616)  --> TIME: 2025-02-25 03:51:57 -- STEP: 23692/36938 -- GLOBAL_STEP: 614700 | > loss_text_ce: 0.028801264241337776 (0.0358589293914237) | > loss_mel_ce: 2.9879143238067627 (nan) | > loss: 0.011971093714237213 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.13660464538586722) | > loader_time: 0.0046 (0.004664516974276395)  --> TIME: 2025-02-25 03:52:06 -- STEP: 23742/36938 -- GLOBAL_STEP: 614750 | > loss_text_ce: 0.03305021673440933 (0.03585729778352504) | > loss_mel_ce: 2.7014262676239014 (nan) | > loss: 0.010851098224520683 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.1366165331305549) | > loader_time: 0.0045 (0.004664452221969799)  --> TIME: 2025-02-25 03:52:16 -- STEP: 23792/36938 -- GLOBAL_STEP: 614800 | > loss_text_ce: 0.028308846056461334 (0.03585641892224731) | > loss_mel_ce: 3.1387789249420166 (nan) | > loss: 0.012567809782922268 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.1366168548576603) | > loader_time: 0.0047 (0.004664334470422461)  --> TIME: 2025-02-25 03:52:25 -- STEP: 23842/36938 -- GLOBAL_STEP: 614850 | > loss_text_ce: 0.027199923992156982 (0.035855963102511365) | > loss_mel_ce: 3.4507334232330322 (nan) | > loss: 0.013801324181258678 (nan) | > current_lr: 5e-06 | > step_time: 0.1265 (0.13662209486603302) | > loader_time: 0.0045 (0.004664210642797116)  --> TIME: 2025-02-25 03:52:34 -- STEP: 23892/36938 -- GLOBAL_STEP: 614900 | > loss_text_ce: 0.028904857113957405 (0.03585469417506889) | > loss_mel_ce: 0.1956380158662796 (nan) | > loss: 0.0008910431643016636 (nan) | > current_lr: 5e-06 | > step_time: 0.1482 (0.1366172128639362) | > loader_time: 0.0046 (0.00466409266224543)  --> TIME: 2025-02-25 03:52:44 -- STEP: 23942/36938 -- GLOBAL_STEP: 614950 | > loss_text_ce: 0.029191583395004272 (0.03585261591220666) | > loss_mel_ce: 2.78389048576355 (nan) | > loss: 0.011163024231791496 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13661526165832227) | > loader_time: 0.0046 (0.0046640258316965446)  --> TIME: 2025-02-25 03:52:53 -- STEP: 23992/36938 -- GLOBAL_STEP: 615000 | > loss_text_ce: 0.04495280608534813 (0.035850705009617186) | > loss_mel_ce: 2.924809694290161 (nan) | > loss: 0.011784772388637066 (nan) | > current_lr: 5e-06 | > step_time: 0.1612 (0.1366210698723668) | > loader_time: 0.0047 (0.004664074196025269)  --> TIME: 2025-02-25 03:53:02 -- STEP: 24042/36938 -- GLOBAL_STEP: 615050 | > loss_text_ce: 0.028447212651371956 (0.03585065035298926) | > loss_mel_ce: 2.5971872806549072 (nan) | > loss: 0.010419185273349285 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13661423543459233) | > loader_time: 0.0048 (0.00466395750308372)  --> TIME: 2025-02-25 03:53:12 -- STEP: 24092/36938 -- GLOBAL_STEP: 615100 | > loss_text_ce: 0.021786538884043694 (0.03585077844559784) | > loss_mel_ce: 0.1486990749835968 (nan) | > loss: 0.0006765302387066185 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13661298292846533) | > loader_time: 0.0045 (0.004663944778778061)  --> TIME: 2025-02-25 03:53:21 -- STEP: 24142/36938 -- GLOBAL_STEP: 615150 | > loss_text_ce: 0.03614553064107895 (0.0358515739023974) | > loss_mel_ce: 2.6512722969055176 (nan) | > loss: 0.010664356872439384 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13660229927325546) | > loader_time: 0.0046 (0.004663792070087466)  --> TIME: 2025-02-25 03:53:30 -- STEP: 24192/36938 -- GLOBAL_STEP: 615200 | > loss_text_ce: 0.039625562727451324 (0.035852927580846676) | > loss_mel_ce: 3.077833652496338 (nan) | > loss: 0.012370871379971504 (nan) | > current_lr: 5e-06 | > step_time: 0.1502 (0.1366003357347038) | > loader_time: 0.0046 (0.004663649601516904)  --> TIME: 2025-02-25 03:53:40 -- STEP: 24242/36938 -- GLOBAL_STEP: 615250 | > loss_text_ce: 0.029825065284967422 (0.03585174095022914) | > loss_mel_ce: 2.1685638427734375 (nan) | > loss: 0.00872376561164856 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.1366034756890122) | > loader_time: 0.0044 (0.0046635399398980044)  --> TIME: 2025-02-25 03:53:49 -- STEP: 24292/36938 -- GLOBAL_STEP: 615300 | > loss_text_ce: 0.03674020990729332 (0.03585340459890045) | > loss_mel_ce: 2.966203451156616 (nan) | > loss: 0.011916443705558777 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.1365967204381144) | > loader_time: 0.0045 (0.00466333100259079)  --> TIME: 2025-02-25 03:53:58 -- STEP: 24342/36938 -- GLOBAL_STEP: 615350 | > loss_text_ce: 0.03903777897357941 (0.035853330579969644) | > loss_mel_ce: 2.9734489917755127 (nan) | > loss: 0.011954313144087791 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13660595062656358) | > loader_time: 0.0046 (0.004663252289847029)  --> TIME: 2025-02-25 03:54:08 -- STEP: 24392/36938 -- GLOBAL_STEP: 615400 | > loss_text_ce: 0.027191119268536568 (0.03585329997794881) | > loss_mel_ce: 2.773594856262207 (nan) | > loss: 0.011114231310784817 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13661062600144194) | > loader_time: 0.0045 (0.004663439677089273)  --> TIME: 2025-02-25 03:54:17 -- STEP: 24442/36938 -- GLOBAL_STEP: 615450 | > loss_text_ce: 0.038063157349824905 (0.03584864769621967) | > loss_mel_ce: 2.641745090484619 (nan) | > loss: 0.010634159669280052 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13661000723924985) | > loader_time: 0.0046 (0.0046633535238736895)  --> TIME: 2025-02-25 03:54:26 -- STEP: 24492/36938 -- GLOBAL_STEP: 615500 | > loss_text_ce: 0.032109566032886505 (0.035850510762073906) | > loss_mel_ce: 2.1812825202941895 (nan) | > loss: 0.008783302269876003 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1366109205834245) | > loader_time: 0.0047 (0.004663321048279286)  --> TIME: 2025-02-25 03:54:36 -- STEP: 24542/36938 -- GLOBAL_STEP: 615550 | > loss_text_ce: 0.037689950317144394 (0.03584815068407927) | > loss_mel_ce: 2.9753737449645996 (nan) | > loss: 0.011956602334976196 (nan) | > current_lr: 5e-06 | > step_time: 0.1703 (0.13660907653071458) | > loader_time: 0.0048 (0.0046632233443970215)  --> TIME: 2025-02-25 03:54:45 -- STEP: 24592/36938 -- GLOBAL_STEP: 615600 | > loss_text_ce: 0.027811547741293907 (0.03584855715279086) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.1366103661471394) | > loader_time: 0.0046 (0.004663205042083935)  --> TIME: 2025-02-25 03:54:54 -- STEP: 24642/36938 -- GLOBAL_STEP: 615650 | > loss_text_ce: 0.039697788655757904 (0.03585024500638613) | > loss_mel_ce: 0.562915027141571 (nan) | > loss: 0.002391320653259754 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13660205971177744) | > loader_time: 0.0045 (0.004663122453992223)  --> TIME: 2025-02-25 03:55:03 -- STEP: 24692/36938 -- GLOBAL_STEP: 615700 | > loss_text_ce: 0.027918867766857147 (0.03585027928257211) | > loss_mel_ce: 2.914175033569336 (nan) | > loss: 0.011674975976347923 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.136598106742233) | > loader_time: 0.0047 (0.004663074362245159)  --> TIME: 2025-02-25 03:55:13 -- STEP: 24742/36938 -- GLOBAL_STEP: 615750 | > loss_text_ce: 0.041458610445261 (0.03585094481966433) | > loss_mel_ce: 3.335244655609131 (nan) | > loss: 0.013399616815149784 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1366118296839924) | > loader_time: 0.0045 (0.004663018553560146)  --> TIME: 2025-02-25 03:55:23 -- STEP: 24792/36938 -- GLOBAL_STEP: 615800 | > loss_text_ce: 0.044211871922016144 (0.03584989551588452) | > loss_mel_ce: 3.78619384765625 (nan) | > loss: 0.015200023539364338 (nan) | > current_lr: 5e-06 | > step_time: 0.1813 (0.13662012089218112) | > loader_time: 0.0048 (0.0046629574114986335)  --> TIME: 2025-02-25 03:55:32 -- STEP: 24842/36938 -- GLOBAL_STEP: 615850 | > loss_text_ce: 0.02971595898270607 (0.0358496588901418) | > loss_mel_ce: 1.7038224935531616 (nan) | > loss: 0.006879121530801058 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13662042801875912) | > loader_time: 0.0045 (0.004662874969400735)  --> TIME: 2025-02-25 03:55:41 -- STEP: 24892/36938 -- GLOBAL_STEP: 615900 | > loss_text_ce: 0.034891869872808456 (0.03585345098160882) | > loss_mel_ce: 3.3427343368530273 (nan) | > loss: 0.013403279706835747 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.13662500275535672) | > loader_time: 0.0048 (0.004662853315599524)  --> TIME: 2025-02-25 03:55:51 -- STEP: 24942/36938 -- GLOBAL_STEP: 615950 | > loss_text_ce: 0.03610438108444214 (0.03585414718773752) | > loss_mel_ce: 1.9125165939331055 (nan) | > loss: 0.0077326237224042416 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13662262789163626) | > loader_time: 0.0048 (0.004662769902403448)  --> TIME: 2025-02-25 03:56:00 -- STEP: 24992/36938 -- GLOBAL_STEP: 616000 | > loss_text_ce: 0.04555971175432205 (0.0358537510877365) | > loss_mel_ce: 2.692235231399536 (nan) | > loss: 0.010864266194403172 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13661338968939396) | > loader_time: 0.0046 (0.004662636195270126)  --> TIME: 2025-02-25 03:56:09 -- STEP: 25042/36938 -- GLOBAL_STEP: 616050 | > loss_text_ce: 0.04252956435084343 (0.035856418790607934) | > loss_mel_ce: 3.651918888092041 (nan) | > loss: 0.01466051023453474 (nan) | > current_lr: 5e-06 | > step_time: 0.1558 (0.1366101318479257) | > loader_time: 0.0046 (0.004662540114903735)  --> TIME: 2025-02-25 03:56:18 -- STEP: 25092/36938 -- GLOBAL_STEP: 616100 | > loss_text_ce: 0.0266506876796484 (0.03585717125776267) | > loss_mel_ce: 2.946044921875 (nan) | > loss: 0.01179641205817461 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13659957183826915) | > loader_time: 0.0047 (0.004662591799506934)  --> TIME: 2025-02-25 03:56:27 -- STEP: 25142/36938 -- GLOBAL_STEP: 616150 | > loss_text_ce: 0.038889575749635696 (0.035857555145931565) | > loss_mel_ce: 2.6331803798675537 (nan) | > loss: 0.010603453032672405 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13659818540769875) | > loader_time: 0.0046 (0.0046625054069411856)  --> TIME: 2025-02-25 03:56:37 -- STEP: 25192/36938 -- GLOBAL_STEP: 616200 | > loss_text_ce: 0.03283290937542915 (0.03585796252749329) | > loss_mel_ce: 2.7126662731170654 (nan) | > loss: 0.010894838720560074 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13660704846342753) | > loader_time: 0.0046 (0.004662483845411258)  --> TIME: 2025-02-25 03:56:46 -- STEP: 25242/36938 -- GLOBAL_STEP: 616250 | > loss_text_ce: 0.031101113185286522 (0.03585836693856834) | > loss_mel_ce: 2.7317278385162354 (nan) | > loss: 0.01096360757946968 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13660650903700103) | > loader_time: 0.0047 (0.004662400077464396)  --> TIME: 2025-02-25 03:56:56 -- STEP: 25292/36938 -- GLOBAL_STEP: 616300 | > loss_text_ce: 0.038027483969926834 (0.03585932538393873) | > loss_mel_ce: 3.1336631774902344 (nan) | > loss: 0.012586074881255627 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13661394897149015) | > loader_time: 0.0045 (0.004662339123257298)  --> TIME: 2025-02-25 03:57:05 -- STEP: 25342/36938 -- GLOBAL_STEP: 616350 | > loss_text_ce: 0.027902306988835335 (0.035852398044330554) | > loss_mel_ce: 2.5161800384521484 (nan) | > loss: 0.0100955655798316 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1366105635101627) | > loader_time: 0.0045 (0.004662248059235836)  --> TIME: 2025-02-25 03:57:14 -- STEP: 25392/36938 -- GLOBAL_STEP: 616400 | > loss_text_ce: 0.031588103622198105 (0.035849373869028985) | > loss_mel_ce: 2.64935302734375 (nan) | > loss: 0.010638656094670296 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.13660893639492083) | > loader_time: 0.0048 (0.004662124913070836)  --> TIME: 2025-02-25 03:57:23 -- STEP: 25442/36938 -- GLOBAL_STEP: 616450 | > loss_text_ce: 0.046121224761009216 (0.03585487738129073) | > loss_mel_ce: 1.727149248123169 (nan) | > loss: 0.007036787923425436 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13659814116096566) | > loader_time: 0.0045 (0.004662079206100402)  --> TIME: 2025-02-25 03:57:33 -- STEP: 25492/36938 -- GLOBAL_STEP: 616500 | > loss_text_ce: 0.040418900549411774 (0.03585762261255446) | > loss_mel_ce: 2.41976261138916 (nan) | > loss: 0.009762625209987164 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13659736863414149) | > loader_time: 0.0045 (0.0046624036892493545)  --> TIME: 2025-02-25 03:57:42 -- STEP: 25542/36938 -- GLOBAL_STEP: 616550 | > loss_text_ce: 0.03452328220009804 (0.03585559717835159) | > loss_mel_ce: 2.2821950912475586 (nan) | > loss: 0.009193327277898788 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.1365992896624619) | > loader_time: 0.0046 (0.004662373493276142)  --> TIME: 2025-02-25 03:57:51 -- STEP: 25592/36938 -- GLOBAL_STEP: 616600 | > loss_text_ce: 0.041384875774383545 (0.03585629587124385) | > loss_mel_ce: 2.692455530166626 (nan) | > loss: 0.010848574340343475 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13659862458016486) | > loader_time: 0.0046 (0.004662517552526362)  --> TIME: 2025-02-25 03:58:01 -- STEP: 25642/36938 -- GLOBAL_STEP: 616650 | > loss_text_ce: 0.03259196877479553 (0.0358560911176937) | > loss_mel_ce: 2.647282838821411 (nan) | > loss: 0.010634425096213818 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.1366051669779454) | > loader_time: 0.0047 (0.004662517340523026)  --> TIME: 2025-02-25 03:58:10 -- STEP: 25692/36938 -- GLOBAL_STEP: 616700 | > loss_text_ce: 0.04233100637793541 (0.03585847151838591) | > loss_mel_ce: 3.002913236618042 (nan) | > loss: 0.012084303423762321 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13660283771126147) | > loader_time: 0.0048 (0.004662504879908291)  --> TIME: 2025-02-25 03:58:19 -- STEP: 25742/36938 -- GLOBAL_STEP: 616750 | > loss_text_ce: 0.03373703360557556 (0.035859004867501106) | > loss_mel_ce: 0.178955078125 (nan) | > loss: 0.0008440163801424205 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1366020171720656) | > loader_time: 0.0046 (0.004662565552968843)  --> TIME: 2025-02-25 03:58:28 -- STEP: 25792/36938 -- GLOBAL_STEP: 616800 | > loss_text_ce: 0.032016489654779434 (0.03586129947528896) | > loss_mel_ce: 3.547003746032715 (nan) | > loss: 0.014202462509274483 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1365977837600124) | > loader_time: 0.0045 (0.004662531906410746)  --> TIME: 2025-02-25 03:58:38 -- STEP: 25842/36938 -- GLOBAL_STEP: 616850 | > loss_text_ce: 0.024210967123508453 (0.03585938683080035) | > loss_mel_ce: 0.9913718700408936 (nan) | > loss: 0.004030090756714344 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13659535117880378) | > loader_time: 0.0047 (0.004662492531536654)  --> TIME: 2025-02-25 03:58:47 -- STEP: 25892/36938 -- GLOBAL_STEP: 616900 | > loss_text_ce: 0.04335751757025719 (0.035859737890810366) | > loss_mel_ce: 2.7419791221618652 (nan) | > loss: 0.011052924208343029 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.13659358350567208) | > loader_time: 0.005 (0.004662517784515884)  --> TIME: 2025-02-25 03:58:56 -- STEP: 25942/36938 -- GLOBAL_STEP: 616950 | > loss_text_ce: 0.03763516992330551 (0.03586000490867686) | > loss_mel_ce: 2.480537176132202 (nan) | > loss: 0.009992747567594051 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13658703907390207) | > loader_time: 0.0045 (0.004662569004259144)  --> TIME: 2025-02-25 03:59:05 -- STEP: 25992/36938 -- GLOBAL_STEP: 617000 | > loss_text_ce: 0.0333796851336956 (0.035860886285962346) | > loss_mel_ce: 2.8292579650878906 (nan) | > loss: 0.01135967392474413 (nan) | > current_lr: 5e-06 | > step_time: 0.179 (0.1365881910522957) | > loader_time: 0.0049 (0.004662641913166589)  --> TIME: 2025-02-25 03:59:15 -- STEP: 26042/36938 -- GLOBAL_STEP: 617050 | > loss_text_ce: 0.035565417259931564 (0.035862098023800894) | > loss_mel_ce: 3.4724318981170654 (nan) | > loss: 0.013920624740421772 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.1365820816901497) | > loader_time: 0.0046 (0.004662740817405848)  --> TIME: 2025-02-25 03:59:24 -- STEP: 26092/36938 -- GLOBAL_STEP: 617100 | > loss_text_ce: 0.034045975655317307 (0.03586141556762183) | > loss_mel_ce: 3.273560047149658 (nan) | > loss: 0.013125421479344368 (nan) | > current_lr: 5e-06 | > step_time: 0.1832 (0.13656996479981753) | > loader_time: 0.0049 (0.004662743388519129)  --> TIME: 2025-02-25 03:59:33 -- STEP: 26142/36938 -- GLOBAL_STEP: 617150 | > loss_text_ce: 0.03565661609172821 (0.035864066363892944) | > loss_mel_ce: 3.0573391914367676 (nan) | > loss: 0.012273794040083885 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.13656634280661661) | > loader_time: 0.0049 (0.004662798664179726)  --> TIME: 2025-02-25 03:59:42 -- STEP: 26192/36938 -- GLOBAL_STEP: 617200 | > loss_text_ce: 0.03851446136832237 (0.03586500840095542) | > loss_mel_ce: 2.5988903045654297 (nan) | > loss: 0.010465892031788826 (nan) | > current_lr: 5e-06 | > step_time: 0.1466 (0.13656023638089618) | > loader_time: 0.0047 (0.0046627501306842935)  --> TIME: 2025-02-25 03:59:52 -- STEP: 26242/36938 -- GLOBAL_STEP: 617250 | > loss_text_ce: 0.04012678563594818 (0.035865986163211856) | > loss_mel_ce: 1.7475031614303589 (nan) | > loss: 0.0070937699638307095 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1365699595444151) | > loader_time: 0.0045 (0.004662796860651238)  --> TIME: 2025-02-25 04:00:01 -- STEP: 26292/36938 -- GLOBAL_STEP: 617300 | > loss_text_ce: 0.045749127864837646 (0.03586804907614894) | > loss_mel_ce: 1.2169053554534912 (nan) | > loss: 0.005010534077882767 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1365575000301401) | > loader_time: 0.0047 (0.004662674864042553)  --> TIME: 2025-02-25 04:00:10 -- STEP: 26342/36938 -- GLOBAL_STEP: 617350 | > loss_text_ce: 0.0359485000371933 (0.03586888734050247) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13655111226918884) | > loader_time: 0.0046 (0.004662615591641086)  --> TIME: 2025-02-25 04:00:19 -- STEP: 26392/36938 -- GLOBAL_STEP: 617400 | > loss_text_ce: 0.04436367005109787 (0.03586713104242973) | > loss_mel_ce: 2.6841166019439697 (nan) | > loss: 0.010827303864061832 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13653696796575301) | > loader_time: 0.0045 (0.0046625260007349244)  --> TIME: 2025-02-25 04:00:28 -- STEP: 26442/36938 -- GLOBAL_STEP: 617450 | > loss_text_ce: 0.03898247703909874 (0.03586758483908035) | > loss_mel_ce: 2.1401021480560303 (nan) | > loss: 0.008647161535918713 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13654011551528217) | > loader_time: 0.0045 (0.004662540963229717)  --> TIME: 2025-02-25 04:00:37 -- STEP: 26492/36938 -- GLOBAL_STEP: 617500 | > loss_text_ce: 0.02509753592312336 (0.035869174798577104) | > loss_mel_ce: 0.1580439954996109 (nan) | > loss: 0.0007267521577887237 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.1365380882406884) | > loader_time: 0.0045 (0.0046624524183329855)  --> TIME: 2025-02-25 04:00:46 -- STEP: 26542/36938 -- GLOBAL_STEP: 617550 | > loss_text_ce: 0.043613024055957794 (0.03586903502598965) | > loss_mel_ce: 2.6930761337280273 (nan) | > loss: 0.010859877802431583 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13653106146359584) | > loader_time: 0.0045 (0.00466234204674101)  --> TIME: 2025-02-25 04:00:56 -- STEP: 26592/36938 -- GLOBAL_STEP: 617600 | > loss_text_ce: 0.039187923073768616 (0.03587117320306632) | > loss_mel_ce: 2.3682572841644287 (nan) | > loss: 0.009553354233503342 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13654089385529228) | > loader_time: 0.0046 (0.0046622785420194325)  --> TIME: 2025-02-25 04:01:05 -- STEP: 26642/36938 -- GLOBAL_STEP: 617650 | > loss_text_ce: 0.03237251564860344 (0.03586981792659213) | > loss_mel_ce: 2.7890892028808594 (nan) | > loss: 0.011196277104318142 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13654072727135727) | > loader_time: 0.0045 (0.004662198943782553)  --> TIME: 2025-02-25 04:01:14 -- STEP: 26692/36938 -- GLOBAL_STEP: 617700 | > loss_text_ce: 0.034494586288928986 (0.035871639733520255) | > loss_mel_ce: 2.7351131439208984 (nan) | > loss: 0.010990507900714874 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13653636474436381) | > loader_time: 0.0047 (0.0046620769477866315)  --> TIME: 2025-02-25 04:01:24 -- STEP: 26742/36938 -- GLOBAL_STEP: 617750 | > loss_text_ce: 0.039402082562446594 (0.035872680728996915) | > loss_mel_ce: 1.8506730794906616 (nan) | > loss: 0.007500298786908388 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13654022372461624) | > loader_time: 0.0048 (0.00466204104146432)  --> TIME: 2025-02-25 04:01:33 -- STEP: 26792/36938 -- GLOBAL_STEP: 617800 | > loss_text_ce: 0.03875074163079262 (0.035870950705131785) | > loss_mel_ce: 2.6074767112731934 (nan) | > loss: 0.010500903241336346 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13653473250365405) | > loader_time: 0.0045 (0.004661891247913665)  --> TIME: 2025-02-25 04:01:42 -- STEP: 26842/36938 -- GLOBAL_STEP: 617850 | > loss_text_ce: 0.019387686625123024 (0.035866782464190256) | > loss_mel_ce: 2.308851480484009 (nan) | > loss: 0.009239044971764088 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13653564307455293) | > loader_time: 0.0044 (0.004661775551967877)  --> TIME: 2025-02-25 04:01:52 -- STEP: 26892/36938 -- GLOBAL_STEP: 617900 | > loss_text_ce: 0.029905948787927628 (0.03586643664196748) | > loss_mel_ce: 3.173311471939087 (nan) | > loss: 0.012711181305348873 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.13654274607491937) | > loader_time: 0.0045 (0.004661706468099032)  --> TIME: 2025-02-25 04:02:01 -- STEP: 26942/36938 -- GLOBAL_STEP: 617950 | > loss_text_ce: 0.035984113812446594 (0.03586596613757409) | > loss_mel_ce: 3.1049492359161377 (nan) | > loss: 0.01246402133256197 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13653642782061592) | > loader_time: 0.0046 (0.004661590420636127)  --> TIME: 2025-02-25 04:02:11 -- STEP: 26992/36938 -- GLOBAL_STEP: 618000 | > loss_text_ce: 0.03332268074154854 (0.035864624997355224) | > loss_mel_ce: 2.6474077701568604 (nan) | > loss: 0.010637819766998291 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.136551456182541) | > loader_time: 0.0046 (0.0046617947143426045)  --> TIME: 2025-02-25 04:02:20 -- STEP: 27042/36938 -- GLOBAL_STEP: 618050 | > loss_text_ce: 0.04405386373400688 (0.035866697636455676) | > loss_mel_ce: 2.358628034591675 (nan) | > loss: 0.009534452110528946 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.1365542054731771) | > loader_time: 0.0046 (0.004661782915868528)  --> TIME: 2025-02-25 04:02:30 -- STEP: 27092/36938 -- GLOBAL_STEP: 618100 | > loss_text_ce: 0.03310004994273186 (0.0358689509682262) | > loss_mel_ce: 3.0557780265808105 (nan) | > loss: 0.012257453985512257 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.13656767595118738) | > loader_time: 0.0049 (0.004661793557787608)  --> TIME: 2025-02-25 04:02:39 -- STEP: 27142/36938 -- GLOBAL_STEP: 618150 | > loss_text_ce: 0.031681936234235764 (0.03586769253711321) | > loss_mel_ce: 3.132589340209961 (nan) | > loss: 0.012556632980704308 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13656335244717369) | > loader_time: 0.0046 (0.004661736399798384)  --> TIME: 2025-02-25 04:02:48 -- STEP: 27192/36938 -- GLOBAL_STEP: 618200 | > loss_text_ce: 0.04270419478416443 (0.035865747208439776) | > loss_mel_ce: 0.5557632446289062 (nan) | > loss: 0.0023748709354549646 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13655562433610016) | > loader_time: 0.0047 (0.004661737417045456)  --> TIME: 2025-02-25 04:02:58 -- STEP: 27242/36938 -- GLOBAL_STEP: 618250 | > loss_text_ce: 0.034708984196186066 (0.035864542051205574) | > loss_mel_ce: 2.7009758949279785 (nan) | > loss: 0.010855892673134804 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13655440126971713) | > loader_time: 0.0047 (0.00466163477335076)  --> TIME: 2025-02-25 04:03:07 -- STEP: 27292/36938 -- GLOBAL_STEP: 618300 | > loss_text_ce: 0.02915184199810028 (0.03586303715308907) | > loss_mel_ce: 2.9656360149383545 (nan) | > loss: 0.011884079314768314 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13655333888158538) | > loader_time: 0.0046 (0.004661592756847537)  --> TIME: 2025-02-25 04:03:16 -- STEP: 27342/36938 -- GLOBAL_STEP: 618350 | > loss_text_ce: 0.02981707453727722 (0.0358648315889132) | > loss_mel_ce: 3.2936015129089355 (nan) | > loss: 0.01318817026913166 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.13655031222674824) | > loader_time: 0.0047 (0.004661537046867124)  --> TIME: 2025-02-25 04:03:25 -- STEP: 27392/36938 -- GLOBAL_STEP: 618400 | > loss_text_ce: 0.04224062338471413 (0.03586625227035196) | > loss_mel_ce: 2.0578057765960693 (nan) | > loss: 0.008333518169820309 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13654738079721632) | > loader_time: 0.0047 (0.004661505389088256)  --> TIME: 2025-02-25 04:03:35 -- STEP: 27442/36938 -- GLOBAL_STEP: 618450 | > loss_text_ce: 0.03740951046347618 (0.03586618570747526) | > loss_mel_ce: 0.8076555728912354 (nan) | > loss: 0.003353432985022664 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13655276428322968) | > loader_time: 0.0045 (0.004661481926596046)  --> TIME: 2025-02-25 04:03:44 -- STEP: 27492/36938 -- GLOBAL_STEP: 618500 | > loss_text_ce: 0.03613903746008873 (0.03586548488381388) | > loss_mel_ce: 1.3559198379516602 (nan) | > loss: 0.005524043459445238 (nan) | > current_lr: 5e-06 | > step_time: 0.1121 (0.13655868459264683) | > loader_time: 0.0046 (0.004661478001391852)  --> TIME: 2025-02-25 04:03:54 -- STEP: 27542/36938 -- GLOBAL_STEP: 618550 | > loss_text_ce: 0.026827378198504448 (0.035867637131333084) | > loss_mel_ce: 1.2667056322097778 (nan) | > loss: 0.005133067723363638 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13657488392533515) | > loader_time: 0.0045 (0.004661500285146367)  --> TIME: 2025-02-25 04:04:04 -- STEP: 27592/36938 -- GLOBAL_STEP: 618600 | > loss_text_ce: 0.0317310206592083 (0.03586870977165168) | > loss_mel_ce: 2.627924680709839 (nan) | > loss: 0.01055419072508812 (nan) | > current_lr: 5e-06 | > step_time: 0.1558 (0.13658006592535094) | > loader_time: 0.0047 (0.004661544358153138)  --> TIME: 2025-02-25 04:04:13 -- STEP: 27642/36938 -- GLOBAL_STEP: 618650 | > loss_text_ce: 0.04283548891544342 (0.03586973164407884) | > loss_mel_ce: 2.5364444255828857 (nan) | > loss: 0.010235237888991833 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.1365789687544669) | > loader_time: 0.0047 (0.004661533950025699)  --> TIME: 2025-02-25 04:04:22 -- STEP: 27692/36938 -- GLOBAL_STEP: 618700 | > loss_text_ce: 0.0511600524187088 (0.03586836219308475) | > loss_mel_ce: 1.9551496505737305 (nan) | > loss: 0.007961547002196312 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.1365751469994522) | > loader_time: 0.0046 (0.004661484104213281)  --> TIME: 2025-02-25 04:04:32 -- STEP: 27742/36938 -- GLOBAL_STEP: 618750 | > loss_text_ce: 0.04559701308608055 (0.03586653568023492) | > loss_mel_ce: 2.5403034687042236 (nan) | > loss: 0.010261510498821735 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.1365749017548241) | > loader_time: 0.0047 (0.00466153377771948)  --> TIME: 2025-02-25 04:04:41 -- STEP: 27792/36938 -- GLOBAL_STEP: 618800 | > loss_text_ce: 0.037495069205760956 (0.03586871918913024) | > loss_mel_ce: 2.81628155708313 (nan) | > loss: 0.011324510909616947 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13657532232584538) | > loader_time: 0.0048 (0.004661458221124275)  --> TIME: 2025-02-25 04:04:50 -- STEP: 27842/36938 -- GLOBAL_STEP: 618850 | > loss_text_ce: 0.038557808846235275 (0.035866526787539835) | > loss_mel_ce: 1.8830515146255493 (nan) | > loss: 0.007625434082001448 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13657540008002864) | > loader_time: 0.0045 (0.004661398101457565)  --> TIME: 2025-02-25 04:05:00 -- STEP: 27892/36938 -- GLOBAL_STEP: 618900 | > loss_text_ce: 0.04453779011964798 (0.03586800604196877) | > loss_mel_ce: 2.3417701721191406 (nan) | > loss: 0.009469476528465748 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.1365841432313876) | > loader_time: 0.0047 (0.004661376133003414)  --> TIME: 2025-02-25 04:05:09 -- STEP: 27942/36938 -- GLOBAL_STEP: 618950 | > loss_text_ce: 0.03709841147065163 (0.03586801601842146) | > loss_mel_ce: 2.5986075401306152 (nan) | > loss: 0.01045915111899376 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.1365774760871886) | > loader_time: 0.0045 (0.0046612863576145445)  --> TIME: 2025-02-25 04:05:19 -- STEP: 27992/36938 -- GLOBAL_STEP: 619000 | > loss_text_ce: 0.030782679095864296 (0.03586642307622682) | > loss_mel_ce: 1.3931121826171875 (nan) | > loss: 0.0056503769010305405 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13659186455548636) | > loader_time: 0.0045 (0.004661268167885514)  --> TIME: 2025-02-25 04:05:28 -- STEP: 28042/36938 -- GLOBAL_STEP: 619050 | > loss_text_ce: 0.030738243833184242 (0.035865150190393505) | > loss_mel_ce: 2.9807214736938477 (nan) | > loss: 0.011950237676501274 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.13659370947084798) | > loader_time: 0.0046 (0.004661146656319634)  --> TIME: 2025-02-25 04:05:37 -- STEP: 28092/36938 -- GLOBAL_STEP: 619100 | > loss_text_ce: 0.032617922872304916 (0.03586327547358462) | > loss_mel_ce: 3.301663875579834 (nan) | > loss: 0.013231277465820312 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13659509894651972) | > loader_time: 0.0048 (0.004661061681269205)  --> TIME: 2025-02-25 04:05:47 -- STEP: 28142/36938 -- GLOBAL_STEP: 619150 | > loss_text_ce: 0.031734295189380646 (0.035861552462650256) | > loss_mel_ce: 2.863718032836914 (nan) | > loss: 0.01148989051580429 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.13659697611375468) | > loader_time: 0.0048 (0.00466098659845588)  --> TIME: 2025-02-25 04:05:56 -- STEP: 28192/36938 -- GLOBAL_STEP: 619200 | > loss_text_ce: 0.02736235037446022 (0.03586374431805266) | > loss_mel_ce: 2.985898017883301 (nan) | > loss: 0.011957382783293724 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13660048341609005) | > loader_time: 0.0046 (0.004660892660785646)  --> TIME: 2025-02-25 04:06:06 -- STEP: 28242/36938 -- GLOBAL_STEP: 619250 | > loss_text_ce: 0.028451036661863327 (0.035864608921433255) | > loss_mel_ce: 2.9242498874664307 (nan) | > loss: 0.01171706710010767 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.13659860842812616) | > loader_time: 0.0047 (0.0046607901156687625)  --> TIME: 2025-02-25 04:06:15 -- STEP: 28292/36938 -- GLOBAL_STEP: 619300 | > loss_text_ce: 0.03667595237493515 (0.0358617145769389) | > loss_mel_ce: 3.4645047187805176 (nan) | > loss: 0.013893574476242065 (nan) | > current_lr: 5e-06 | > step_time: 0.1905 (0.13659906362047897) | > loader_time: 0.0048 (0.004660696140967818)  --> TIME: 2025-02-25 04:06:24 -- STEP: 28342/36938 -- GLOBAL_STEP: 619350 | > loss_text_ce: 0.04454367607831955 (0.03586299275644435) | > loss_mel_ce: 3.605369806289673 (nan) | > loss: 0.014483784325420856 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13660632323195826) | > loader_time: 0.0046 (0.004660619313829138)  --> TIME: 2025-02-25 04:06:34 -- STEP: 28392/36938 -- GLOBAL_STEP: 619400 | > loss_text_ce: 0.042405061423778534 (0.03586487653759003) | > loss_mel_ce: 2.7662277221679688 (nan) | > loss: 0.01114536914974451 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.13659938409920308) | > loader_time: 0.0045 (0.004660484538208666)  --> TIME: 2025-02-25 04:06:42 -- STEP: 28442/36938 -- GLOBAL_STEP: 619450 | > loss_text_ce: 0.029309464618563652 (0.0358618274371363) | > loss_mel_ce: 1.1702059507369995 (nan) | > loss: 0.004759982228279114 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13658643090752615) | > loader_time: 0.0045 (0.004660318742932768)  --> TIME: 2025-02-25 04:06:52 -- STEP: 28492/36938 -- GLOBAL_STEP: 619500 | > loss_text_ce: 0.036193929612636566 (0.03586158147428811) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13658157448311395) | > loader_time: 0.0046 (0.004660187218776028)  --> TIME: 2025-02-25 04:07:01 -- STEP: 28542/36938 -- GLOBAL_STEP: 619550 | > loss_text_ce: 0.03980575129389763 (0.03586189490537677) | > loss_mel_ce: 3.145224094390869 (nan) | > loss: 0.012639007531106472 (nan) | > current_lr: 5e-06 | > step_time: 0.178 (0.13659214919153567) | > loader_time: 0.005 (0.004660151424291712)  --> TIME: 2025-02-25 04:07:11 -- STEP: 28592/36938 -- GLOBAL_STEP: 619600 | > loss_text_ce: 0.035791393369436264 (0.03585885235933248) | > loss_mel_ce: 3.5206737518310547 (nan) | > loss: 0.014112957753241062 (nan) | > current_lr: 5e-06 | > step_time: 0.1832 (0.1365938725488842) | > loader_time: 0.0048 (0.0046601977322311364)  --> TIME: 2025-02-25 04:07:20 -- STEP: 28642/36938 -- GLOBAL_STEP: 619650 | > loss_text_ce: 0.037679485976696014 (0.03585873882795194) | > loss_mel_ce: 1.1423135995864868 (nan) | > loss: 0.004682512488216162 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13658841765382254) | > loader_time: 0.0045 (0.004660157690866673)  --> TIME: 2025-02-25 04:07:29 -- STEP: 28692/36938 -- GLOBAL_STEP: 619700 | > loss_text_ce: 0.025546975433826447 (0.035856457186912064) | > loss_mel_ce: 1.9357401132583618 (nan) | > loss: 0.007782885804772377 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13658912982399551) | > loader_time: 0.0045 (0.004660096225688034)  --> TIME: 2025-02-25 04:07:38 -- STEP: 28742/36938 -- GLOBAL_STEP: 619750 | > loss_text_ce: 0.03846921771764755 (0.03585499334803863) | > loss_mel_ce: 3.174656867980957 (nan) | > loss: 0.012750501744449139 (nan) | > current_lr: 5e-06 | > step_time: 0.177 (0.13658393861090976) | > loader_time: 0.0049 (0.004660017529705902)  --> TIME: 2025-02-25 04:07:48 -- STEP: 28792/36938 -- GLOBAL_STEP: 619800 | > loss_text_ce: 0.03869291767477989 (0.03585405874449106) | > loss_mel_ce: 2.9651103019714355 (nan) | > loss: 0.011919855140149593 (nan) | > current_lr: 5e-06 | > step_time: 0.1699 (0.13658727373007654) | > loader_time: 0.0048 (0.004659963874692383)  --> TIME: 2025-02-25 04:07:57 -- STEP: 28842/36938 -- GLOBAL_STEP: 619850 | > loss_text_ce: 0.051866158843040466 (0.03585520472611131) | > loss_mel_ce: 2.68991756439209 (nan) | > loss: 0.010880094021558762 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13658523527331487) | > loader_time: 0.0045 (0.0046598683960287135)  --> TIME: 2025-02-25 04:08:06 -- STEP: 28892/36938 -- GLOBAL_STEP: 619900 | > loss_text_ce: 0.03238704428076744 (0.03585637727683027) | > loss_mel_ce: 3.0367953777313232 (nan) | > loss: 0.012179295532405376 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.1365890437160752) | > loader_time: 0.0045 (0.004659850586158818)  --> TIME: 2025-02-25 04:08:15 -- STEP: 28942/36938 -- GLOBAL_STEP: 619950 | > loss_text_ce: 0.0327426977455616 (0.03585458404308099) | > loss_mel_ce: 2.0255563259124756 (nan) | > loss: 0.008167853578925133 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13658760044736346) | > loader_time: 0.0045 (0.004659811650188498)  --> TIME: 2025-02-25 04:08:25 -- STEP: 28992/36938 -- GLOBAL_STEP: 620000 | > loss_text_ce: 0.03320179879665375 (0.03585469128210453) | > loss_mel_ce: 3.233652114868164 (nan) | > loss: 0.012963706627488136 (nan) | > current_lr: 5e-06 | > step_time: 0.1814 (0.13659133680707553) | > loader_time: 0.005 (0.004659999630703862) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_620000.pth  --> TIME: 2025-02-25 04:08:41 -- STEP: 29042/36938 -- GLOBAL_STEP: 620050 | > loss_text_ce: 0.017080433666706085 (0.0358551106538568) | > loss_mel_ce: 2.314107656478882 (nan) | > loss: 0.009250747039914131 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13659787447309185) | > loader_time: 0.0045 (0.004659996389989737)  --> TIME: 2025-02-25 04:08:50 -- STEP: 29092/36938 -- GLOBAL_STEP: 620100 | > loss_text_ce: 0.041663073003292084 (0.03585457564938497) | > loss_mel_ce: 2.9986472129821777 (nan) | > loss: 0.01206472422927618 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13659118831559225) | > loader_time: 0.0045 (0.004660030613079335)  --> TIME: 2025-02-25 04:08:59 -- STEP: 29142/36938 -- GLOBAL_STEP: 620150 | > loss_text_ce: 0.039064183831214905 (0.03585484194721413) | > loss_mel_ce: 3.0592200756073 (nan) | > loss: 0.012294779531657696 (nan) | > current_lr: 5e-06 | > step_time: 0.179 (0.13658972344788883) | > loader_time: 0.0049 (0.0046600948503518255)  --> TIME: 2025-02-25 04:09:08 -- STEP: 29192/36938 -- GLOBAL_STEP: 620200 | > loss_text_ce: 0.03590409457683563 (0.03585558763703729) | > loss_mel_ce: 3.0777587890625 (nan) | > loss: 0.012355806306004524 (nan) | > current_lr: 5e-06 | > step_time: 0.1775 (0.1365808284455827) | > loader_time: 0.0048 (0.004660042059457505)  --> TIME: 2025-02-25 04:09:17 -- STEP: 29242/36938 -- GLOBAL_STEP: 620250 | > loss_text_ce: 0.037877943366765976 (0.035856510327520526) | > loss_mel_ce: 2.782658576965332 (nan) | > loss: 0.011192605830729008 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13657661893529016) | > loader_time: 0.0045 (0.004659950337750159)  --> TIME: 2025-02-25 04:09:27 -- STEP: 29292/36938 -- GLOBAL_STEP: 620300 | > loss_text_ce: 0.022119024768471718 (0.03585502339917259) | > loss_mel_ce: 2.6734867095947266 (nan) | > loss: 0.01069684885442257 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1365896531683521) | > loader_time: 0.0046 (0.004659981207005688)  --> TIME: 2025-02-25 04:09:36 -- STEP: 29342/36938 -- GLOBAL_STEP: 620350 | > loss_text_ce: 0.03520231321454048 (0.03585291122142847) | > loss_mel_ce: 3.0461573600769043 (nan) | > loss: 0.012227618135511875 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.13658328357304134) | > loader_time: 0.0049 (0.0046599234924280305)  --> TIME: 2025-02-25 04:09:45 -- STEP: 29392/36938 -- GLOBAL_STEP: 620400 | > loss_text_ce: 0.03867549076676369 (0.03585383769681223) | > loss_mel_ce: 3.545977830886841 (nan) | > loss: 0.014224816113710403 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.136588433934867) | > loader_time: 0.0045 (0.004659895581853893)  --> TIME: 2025-02-25 04:09:54 -- STEP: 29442/36938 -- GLOBAL_STEP: 620450 | > loss_text_ce: 0.027848074212670326 (0.03585316797711513) | > loss_mel_ce: 0.00830688513815403 (nan) | > loss: 0.00014347207616083324 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.1365849388074097) | > loader_time: 0.0047 (0.004659808481300809)  --> TIME: 2025-02-25 04:10:04 -- STEP: 29492/36938 -- GLOBAL_STEP: 620500 | > loss_text_ce: 0.03277907148003578 (0.0358551646079473) | > loss_mel_ce: 2.4513792991638184 (nan) | > loss: 0.009857771918177605 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1365863332168936) | > loader_time: 0.0045 (0.004659783301773215)  --> TIME: 2025-02-25 04:10:13 -- STEP: 29542/36938 -- GLOBAL_STEP: 620550 | > loss_text_ce: 0.03454915061593056 (0.03585773777465916) | > loss_mel_ce: 1.9045759439468384 (nan) | > loss: 0.007694941014051437 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13658854167419882) | > loader_time: 0.0046 (0.004659780651526687)  --> TIME: 2025-02-25 04:10:22 -- STEP: 29592/36938 -- GLOBAL_STEP: 620600 | > loss_text_ce: 0.032108552753925323 (0.03585769515342887) | > loss_mel_ce: 3.1294617652893066 (nan) | > loss: 0.012545914389193058 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.136585429234258) | > loader_time: 0.0046 (0.004659741101763864)  --> TIME: 2025-02-25 04:10:32 -- STEP: 29642/36938 -- GLOBAL_STEP: 620650 | > loss_text_ce: 0.024167487397789955 (0.03585720332437373) | > loss_mel_ce: 1.2595443725585938 (nan) | > loss: 0.005094095133244991 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13658881373238438) | > loader_time: 0.0046 (0.004659724657001992)  --> TIME: 2025-02-25 04:10:41 -- STEP: 29692/36938 -- GLOBAL_STEP: 620700 | > loss_text_ce: 0.032450053840875626 (0.03585800967930962) | > loss_mel_ce: 0.5659657120704651 (nan) | > loss: 0.0023746660444885492 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.13658960075467036) | > loader_time: 0.0047 (0.004659721966334561)  --> TIME: 2025-02-25 04:10:51 -- STEP: 29742/36938 -- GLOBAL_STEP: 620750 | > loss_text_ce: 0.036073893308639526 (0.035858201919861055) | > loss_mel_ce: 2.7368574142456055 (nan) | > loss: 0.011003696359694004 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.136596637393894) | > loader_time: 0.0047 (0.0046597881601234)  --> TIME: 2025-02-25 04:11:01 -- STEP: 29792/36938 -- GLOBAL_STEP: 620800 | > loss_text_ce: 0.037799984216690063 (0.03585859541931864) | > loss_mel_ce: 3.389836072921753 (nan) | > loss: 0.01360173150897026 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13660772673777652) | > loader_time: 0.0046 (0.004659791550049878)  --> TIME: 2025-02-25 04:11:10 -- STEP: 29842/36938 -- GLOBAL_STEP: 620850 | > loss_text_ce: 0.031204991042613983 (0.03585811993546555) | > loss_mel_ce: 2.684582233428955 (nan) | > loss: 0.010776934213936329 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.13661196197327957) | > loader_time: 0.0046 (0.004659756052375037)  --> TIME: 2025-02-25 04:11:19 -- STEP: 29892/36938 -- GLOBAL_STEP: 620900 | > loss_text_ce: 0.037633370608091354 (0.03585706815541242) | > loss_mel_ce: 0.0398305244743824 (nan) | > loss: 0.00030739643261767924 (nan) | > current_lr: 5e-06 | > step_time: 0.1071 (0.13661467634210214) | > loader_time: 0.0046 (0.00465978001489008)  --> TIME: 2025-02-25 04:11:29 -- STEP: 29942/36938 -- GLOBAL_STEP: 620950 | > loss_text_ce: 0.029610252007842064 (0.03585734243484924) | > loss_mel_ce: 4.1015625 (nan) | > loss: 0.016393544152379036 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.13661299471020333) | > loader_time: 0.0047 (0.004659873626568361)  --> TIME: 2025-02-25 04:11:38 -- STEP: 29992/36938 -- GLOBAL_STEP: 621000 | > loss_text_ce: 0.034464750438928604 (0.03585635232233559) | > loss_mel_ce: 2.4251441955566406 (nan) | > loss: 0.009760353714227676 (nan) | > current_lr: 5e-06 | > step_time: 0.1496 (0.13662295531959437) | > loader_time: 0.0046 (0.004659901701249798)  --> TIME: 2025-02-25 04:11:48 -- STEP: 30042/36938 -- GLOBAL_STEP: 621050 | > loss_text_ce: 0.04361135512590408 (0.03585486191371766) | > loss_mel_ce: 2.8708815574645996 (nan) | > loss: 0.011565448716282845 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13663363432917525) | > loader_time: 0.0047 (0.004659989179986204)  --> TIME: 2025-02-25 04:11:58 -- STEP: 30092/36938 -- GLOBAL_STEP: 621100 | > loss_text_ce: 0.04541325569152832 (0.03585665409200485) | > loss_mel_ce: 2.3862366676330566 (nan) | > loss: 0.009649405255913734 (nan) | > current_lr: 5e-06 | > step_time: 0.1785 (0.13664387800141098) | > loader_time: 0.0049 (0.004660111062786289)  --> TIME: 2025-02-25 04:12:07 -- STEP: 30142/36938 -- GLOBAL_STEP: 621150 | > loss_text_ce: 0.04048791900277138 (0.03585704625610389) | > loss_mel_ce: 1.4169137477874756 (nan) | > loss: 0.005783339962363243 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13664255897760383) | > loader_time: 0.0046 (0.00466019369597049)  --> TIME: 2025-02-25 04:12:16 -- STEP: 30192/36938 -- GLOBAL_STEP: 621200 | > loss_text_ce: 0.02933448553085327 (0.03585742332944889) | > loss_mel_ce: 2.2934019565582275 (nan) | > loss: 0.009217209182679653 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.136639660337834) | > loader_time: 0.0046 (0.004660205498029802)  --> TIME: 2025-02-25 04:12:26 -- STEP: 30242/36938 -- GLOBAL_STEP: 621250 | > loss_text_ce: 0.03191157802939415 (0.03585679112542351) | > loss_mel_ce: 3.150702953338623 (nan) | > loss: 0.012629423290491104 (nan) | > current_lr: 5e-06 | > step_time: 0.1786 (0.13664337041303734) | > loader_time: 0.0048 (0.004660244010426821)  --> TIME: 2025-02-25 04:12:35 -- STEP: 30292/36938 -- GLOBAL_STEP: 621300 | > loss_text_ce: 0.03514362871646881 (0.035856880580066995) | > loss_mel_ce: 1.8890336751937866 (nan) | > loss: 0.00763562461361289 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13664181708943218) | > loader_time: 0.0045 (0.004660148507582561)  --> TIME: 2025-02-25 04:12:44 -- STEP: 30342/36938 -- GLOBAL_STEP: 621350 | > loss_text_ce: 0.03283106908202171 (0.0358563438208602) | > loss_mel_ce: 2.874136447906494 (nan) | > loss: 0.0115355858579278 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13664067233137595) | > loader_time: 0.0048 (0.004660219847900629)  --> TIME: 2025-02-25 04:12:54 -- STEP: 30392/36938 -- GLOBAL_STEP: 621400 | > loss_text_ce: 0.03695833310484886 (0.0358570937781446) | > loss_mel_ce: 2.7954769134521484 (nan) | > loss: 0.011239822953939438 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13664072501750368) | > loader_time: 0.0047 (0.004660213039121584)  --> TIME: 2025-02-25 04:13:03 -- STEP: 30442/36938 -- GLOBAL_STEP: 621450 | > loss_text_ce: 0.017031723633408546 (0.035857782576579615) | > loss_mel_ce: 2.555272102355957 (nan) | > loss: 0.010207555256783962 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13664764689440523) | > loader_time: 0.0045 (0.004660204592346978)  --> TIME: 2025-02-25 04:13:13 -- STEP: 30492/36938 -- GLOBAL_STEP: 621500 | > loss_text_ce: 0.031151866540312767 (0.0358570472663623) | > loss_mel_ce: 3.6694207191467285 (nan) | > loss: 0.014684812165796757 (nan) | > current_lr: 5e-06 | > step_time: 0.1447 (0.13664363539470653) | > loader_time: 0.0046 (0.004660154333518742)  --> TIME: 2025-02-25 04:13:22 -- STEP: 30542/36938 -- GLOBAL_STEP: 621550 | > loss_text_ce: 0.040597785264253616 (0.035856930547000586) | > loss_mel_ce: 3.0075128078460693 (nan) | > loss: 0.0120956776663661 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.1366456898730547) | > loader_time: 0.0046 (0.00466009605048755)  --> TIME: 2025-02-25 04:13:31 -- STEP: 30592/36938 -- GLOBAL_STEP: 621600 | > loss_text_ce: 0.03762107342481613 (0.03585508788192234) | > loss_mel_ce: 0.4128892719745636 (nan) | > loss: 0.001787739573046565 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13664731917178008) | > loader_time: 0.0045 (0.004660005038253462)  --> TIME: 2025-02-25 04:13:41 -- STEP: 30642/36938 -- GLOBAL_STEP: 621650 | > loss_text_ce: 0.03650391846895218 (0.03585563423354518) | > loss_mel_ce: 1.2754987478256226 (nan) | > loss: 0.005206360016018152 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13666010937797463) | > loader_time: 0.0045 (0.004660015193034676)  --> TIME: 2025-02-25 04:13:50 -- STEP: 30692/36938 -- GLOBAL_STEP: 621700 | > loss_text_ce: 0.03399769589304924 (0.03585624618466148) | > loss_mel_ce: 3.0501952171325684 (nan) | > loss: 0.012238861992955208 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.1366565189600206) | > loader_time: 0.0047 (0.004660138923217989)  --> TIME: 2025-02-25 04:14:00 -- STEP: 30742/36938 -- GLOBAL_STEP: 621750 | > loss_text_ce: 0.04345250874757767 (0.035855827755594386) | > loss_mel_ce: 3.1341969966888428 (nan) | > loss: 0.012609721161425114 (nan) | > current_lr: 5e-06 | > step_time: 0.1499 (0.13665595219662788) | > loader_time: 0.0047 (0.004660124730973908)  --> TIME: 2025-02-25 04:14:09 -- STEP: 30792/36938 -- GLOBAL_STEP: 621800 | > loss_text_ce: 0.04429540038108826 (0.03585533863912207) | > loss_mel_ce: 2.809183120727539 (nan) | > loss: 0.011323328129947186 (nan) | > current_lr: 5e-06 | > step_time: 0.1629 (0.13665007560802553) | > loader_time: 0.0048 (0.00466018286455018)  --> TIME: 2025-02-25 04:14:18 -- STEP: 30842/36938 -- GLOBAL_STEP: 621850 | > loss_text_ce: 0.03158757463097572 (0.035855795074586826) | > loss_mel_ce: 2.828875780105591 (nan) | > loss: 0.011351046152412891 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.13665147625776133) | > loader_time: 0.0046 (0.004660141946555032)  --> TIME: 2025-02-25 04:14:27 -- STEP: 30892/36938 -- GLOBAL_STEP: 621900 | > loss_text_ce: 0.034225404262542725 (0.035857432156187564) | > loss_mel_ce: 0.4244907796382904 (nan) | > loss: 0.0018203024519607425 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13664446775582584) | > loader_time: 0.0046 (0.004660171161549394)  --> TIME: 2025-02-25 04:14:37 -- STEP: 30942/36938 -- GLOBAL_STEP: 621950 | > loss_text_ce: 0.0351589061319828 (0.035858093114922815) | > loss_mel_ce: 2.6047258377075195 (nan) | > loss: 0.010475733317434788 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13664441603848632) | > loader_time: 0.0044 (0.004660172211577591)  --> TIME: 2025-02-25 04:14:46 -- STEP: 30992/36938 -- GLOBAL_STEP: 622000 | > loss_text_ce: 0.03175121173262596 (0.03585681009020216) | > loss_mel_ce: 2.7822179794311523 (nan) | > loss: 0.011166544631123543 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.13663888689963277) | > loader_time: 0.0048 (0.00466012064642557)  --> TIME: 2025-02-25 04:14:55 -- STEP: 31042/36938 -- GLOBAL_STEP: 622050 | > loss_text_ce: 0.03400389850139618 (0.0358583778107521) | > loss_mel_ce: 2.370473861694336 (nan) | > loss: 0.009541579522192478 (nan) | > current_lr: 5e-06 | > step_time: 0.1357 (0.13664406781097954) | > loader_time: 0.0047 (0.004660162942002937)  --> TIME: 2025-02-25 04:15:04 -- STEP: 31092/36938 -- GLOBAL_STEP: 622100 | > loss_text_ce: 0.033705245703458786 (0.035858313512117844) | > loss_mel_ce: 2.6974546909332275 (nan) | > loss: 0.010837936773896217 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1366428008843889) | > loader_time: 0.0046 (0.004660189496831359)  --> TIME: 2025-02-25 04:15:14 -- STEP: 31142/36938 -- GLOBAL_STEP: 622150 | > loss_text_ce: 0.030854474753141403 (0.03585787489573582) | > loss_mel_ce: 2.7519114017486572 (nan) | > loss: 0.011042722500860691 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.1366470134003328) | > loader_time: 0.0046 (0.004660252798699188)  --> TIME: 2025-02-25 04:15:23 -- STEP: 31192/36938 -- GLOBAL_STEP: 622200 | > loss_text_ce: 0.04364848509430885 (0.035860201241421676) | > loss_mel_ce: 2.36083722114563 (nan) | > loss: 0.00954161025583744 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13664948055914752) | > loader_time: 0.0047 (0.004660258532548841)  --> TIME: 2025-02-25 04:15:33 -- STEP: 31242/36938 -- GLOBAL_STEP: 622250 | > loss_text_ce: 0.03887872397899628 (0.035859843111121205) | > loss_mel_ce: 3.4364075660705566 (nan) | > loss: 0.013790818862617016 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13665351552653707) | > loader_time: 0.0048 (0.004660217109207915)  --> TIME: 2025-02-25 04:15:42 -- STEP: 31292/36938 -- GLOBAL_STEP: 622300 | > loss_text_ce: 0.03230563923716545 (0.03585765081124756) | > loss_mel_ce: 3.0988070964813232 (nan) | > loss: 0.012425052002072334 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.1366538936549646) | > loader_time: 0.0049 (0.004660250508814229)  --> TIME: 2025-02-25 04:15:52 -- STEP: 31342/36938 -- GLOBAL_STEP: 622350 | > loss_text_ce: 0.04038678854703903 (0.035859255328161525) | > loss_mel_ce: 0.9286103844642639 (nan) | > loss: 0.0038452271837741137 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13665445837093027) | > loader_time: 0.0047 (0.004660304394003243)  --> TIME: 2025-02-25 04:16:01 -- STEP: 31392/36938 -- GLOBAL_STEP: 622400 | > loss_text_ce: 0.02961379662156105 (0.03585849092981534) | > loss_mel_ce: 3.3239355087280273 (nan) | > loss: 0.01330773625522852 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.13666509580964561) | > loader_time: 0.0047 (0.004660347573436392)  --> TIME: 2025-02-25 04:16:10 -- STEP: 31442/36938 -- GLOBAL_STEP: 622450 | > loss_text_ce: 0.032071612775325775 (0.03585706079609351) | > loss_mel_ce: 2.4571547508239746 (nan) | > loss: 0.009877882897853851 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13666204781820715) | > loader_time: 0.0047 (0.004660431062225104)  --> TIME: 2025-02-25 04:16:20 -- STEP: 31492/36938 -- GLOBAL_STEP: 622500 | > loss_text_ce: 0.042790450155735016 (0.03585930424970059) | > loss_mel_ce: 3.3257596492767334 (nan) | > loss: 0.013367262668907642 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13665972378449845) | > loader_time: 0.0047 (0.0046604354893648205)  --> TIME: 2025-02-25 04:16:29 -- STEP: 31542/36938 -- GLOBAL_STEP: 622550 | > loss_text_ce: 0.039656102657318115 (0.035858475522142164) | > loss_mel_ce: 3.1239395141601562 (nan) | > loss: 0.01255395170301199 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13665582665658732) | > loader_time: 0.0047 (0.004660511929944594)  --> TIME: 2025-02-25 04:16:38 -- STEP: 31592/36938 -- GLOBAL_STEP: 622600 | > loss_text_ce: 0.029216162860393524 (0.035857181941981814) | > loss_mel_ce: 2.094228982925415 (nan) | > loss: 0.008426370099186897 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13665515896820582) | > loader_time: 0.005 (0.0046604831072613664)  --> TIME: 2025-02-25 04:16:48 -- STEP: 31642/36938 -- GLOBAL_STEP: 622650 | > loss_text_ce: 0.03727918490767479 (0.03585563730228269) | > loss_mel_ce: 2.7845263481140137 (nan) | > loss: 0.011197641491889954 (nan) | > current_lr: 5e-06 | > step_time: 0.1915 (0.1366598305166628) | > loader_time: 0.0049 (0.004660442123958036)  --> TIME: 2025-02-25 04:16:57 -- STEP: 31692/36938 -- GLOBAL_STEP: 622700 | > loss_text_ce: 0.04335011541843414 (0.03585628273045897) | > loss_mel_ce: 2.7615132331848145 (nan) | > loss: 0.011130411177873611 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13666251211113398) | > loader_time: 0.0049 (0.004660441533011315)  --> TIME: 2025-02-25 04:17:06 -- STEP: 31742/36938 -- GLOBAL_STEP: 622750 | > loss_text_ce: 0.032201848924160004 (0.03585409292094783) | > loss_mel_ce: 3.061912775039673 (nan) | > loss: 0.012278232723474503 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1366636419287249) | > loader_time: 0.0046 (0.004660484944179491)  --> TIME: 2025-02-25 04:17:16 -- STEP: 31792/36938 -- GLOBAL_STEP: 622800 | > loss_text_ce: 0.032496366649866104 (0.035855199552209005) | > loss_mel_ce: 2.2987303733825684 (nan) | > loss: 0.009250900708138943 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.1366645706527844) | > loader_time: 0.0044 (0.004660554436444664)  --> TIME: 2025-02-25 04:17:25 -- STEP: 31842/36938 -- GLOBAL_STEP: 622850 | > loss_text_ce: 0.0322498083114624 (0.035855643156975946) | > loss_mel_ce: 2.824504852294922 (nan) | > loss: 0.011336329393088818 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13665981239862482) | > loader_time: 0.0045 (0.004660491967020062)  --> TIME: 2025-02-25 04:17:35 -- STEP: 31892/36938 -- GLOBAL_STEP: 622900 | > loss_text_ce: 0.036054838448762894 (0.03585374075539703) | > loss_mel_ce: 3.1305320262908936 (nan) | > loss: 0.012565821409225464 (nan) | > current_lr: 5e-06 | > step_time: 0.1896 (0.13666747545008803) | > loader_time: 0.0048 (0.004660560677166101)  --> TIME: 2025-02-25 04:17:44 -- STEP: 31942/36938 -- GLOBAL_STEP: 622950 | > loss_text_ce: 0.04482051730155945 (0.03585497927459179) | > loss_mel_ce: 2.7142748832702637 (nan) | > loss: 0.010948792099952698 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13666377883542324) | > loader_time: 0.0045 (0.004660529608449163)  --> TIME: 2025-02-25 04:17:53 -- STEP: 31992/36938 -- GLOBAL_STEP: 623000 | > loss_text_ce: 0.03607536107301712 (0.03585433776904411) | > loss_mel_ce: 3.523923873901367 (nan) | > loss: 0.014126981608569622 (nan) | > current_lr: 5e-06 | > step_time: 0.1708 (0.13666748465463835) | > loader_time: 0.0052 (0.004660494552907091)  --> TIME: 2025-02-25 04:18:03 -- STEP: 32042/36938 -- GLOBAL_STEP: 623050 | > loss_text_ce: 0.03684850037097931 (0.03585216107200195) | > loss_mel_ce: 3.271822690963745 (nan) | > loss: 0.013129648752510548 (nan) | > current_lr: 5e-06 | > step_time: 0.1869 (0.13667151785977233) | > loader_time: 0.0048 (0.00466043233618475)  --> TIME: 2025-02-25 04:18:12 -- STEP: 32092/36938 -- GLOBAL_STEP: 623100 | > loss_text_ce: 0.03768496587872505 (0.03585071011318555) | > loss_mel_ce: 3.1385185718536377 (nan) | > loss: 0.012603983283042908 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.1366618594504583) | > loader_time: 0.0046 (0.004660527939126657)  --> TIME: 2025-02-25 04:18:21 -- STEP: 32142/36938 -- GLOBAL_STEP: 623150 | > loss_text_ce: 0.02983381226658821 (0.03585120002770312) | > loss_mel_ce: 2.4441077709198 (nan) | > loss: 0.009817228652536869 (nan) | > current_lr: 5e-06 | > step_time: 0.1511 (0.1366568145098603) | > loader_time: 0.0048 (0.004660581282899993)  --> TIME: 2025-02-25 04:18:30 -- STEP: 32192/36938 -- GLOBAL_STEP: 623200 | > loss_text_ce: 0.027340875938534737 (0.03584975247418501) | > loss_mel_ce: 1.837799310684204 (nan) | > loss: 0.007401350419968367 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13666149759091134) | > loader_time: 0.0046 (0.004660711270084939)  --> TIME: 2025-02-25 04:18:40 -- STEP: 32242/36938 -- GLOBAL_STEP: 623250 | > loss_text_ce: 0.03754788637161255 (0.035849366599455215) | > loss_mel_ce: 3.168660879135132 (nan) | > loss: 0.012723051011562347 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.13666075069121505) | > loader_time: 0.0047 (0.004660713946979797)  --> TIME: 2025-02-25 04:18:49 -- STEP: 32292/36938 -- GLOBAL_STEP: 623300 | > loss_text_ce: 0.03406611457467079 (0.035850860891526524) | > loss_mel_ce: 2.8549180030822754 (nan) | > loss: 0.011464223265647888 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.1366558797028992) | > loader_time: 0.0046 (0.0046606843878785175)  --> TIME: 2025-02-25 04:18:58 -- STEP: 32342/36938 -- GLOBAL_STEP: 623350 | > loss_text_ce: 0.03643055260181427 (0.03585132827020348) | > loss_mel_ce: 2.7258260250091553 (nan) | > loss: 0.010961336083710194 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.1366565926723255) | > loader_time: 0.0046 (0.00466072716375916)  --> TIME: 2025-02-25 04:19:07 -- STEP: 32392/36938 -- GLOBAL_STEP: 623400 | > loss_text_ce: 0.03353442996740341 (0.03585055428466188) | > loss_mel_ce: 2.687121868133545 (nan) | > loss: 0.010796256363391876 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13665266282383534) | > loader_time: 0.0045 (0.004660647705650027)  --> TIME: 2025-02-25 04:19:17 -- STEP: 32442/36938 -- GLOBAL_STEP: 623450 | > loss_text_ce: 0.04166766256093979 (0.03585084154553072) | > loss_mel_ce: 3.0966010093688965 (nan) | > loss: 0.01245344802737236 (nan) | > current_lr: 5e-06 | > step_time: 0.1505 (0.13665949939674613) | > loader_time: 0.0048 (0.00466062363254829)  --> TIME: 2025-02-25 04:19:26 -- STEP: 32492/36938 -- GLOBAL_STEP: 623500 | > loss_text_ce: 0.039202697575092316 (0.035849914302991344) | > loss_mel_ce: 2.974428415298462 (nan) | > loss: 0.011958854272961617 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13665489238808026) | > loader_time: 0.0046 (0.004660634032966465)  --> TIME: 2025-02-25 04:19:36 -- STEP: 32542/36938 -- GLOBAL_STEP: 623550 | > loss_text_ce: 0.03572758659720421 (0.03584966585889453) | > loss_mel_ce: 3.273315906524658 (nan) | > loss: 0.013131125830113888 (nan) | > current_lr: 5e-06 | > step_time: 0.2007 (0.13666100218039456) | > loader_time: 0.0048 (0.004660623638157089)  --> TIME: 2025-02-25 04:19:45 -- STEP: 32592/36938 -- GLOBAL_STEP: 623600 | > loss_text_ce: 0.031178852543234825 (0.035847229894618596) | > loss_mel_ce: 3.181781768798828 (nan) | > loss: 0.012749845162034035 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13666497652999995) | > loader_time: 0.0049 (0.0046606046359329795)  --> TIME: 2025-02-25 04:19:54 -- STEP: 32642/36938 -- GLOBAL_STEP: 623650 | > loss_text_ce: 0.03758644312620163 (0.03584612438668699) | > loss_mel_ce: 2.3017678260803223 (nan) | > loss: 0.009283152408897877 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13665969961524083) | > loader_time: 0.0045 (0.004660858885051819)  --> TIME: 2025-02-25 04:20:04 -- STEP: 32692/36938 -- GLOBAL_STEP: 623700 | > loss_text_ce: 0.0382893942296505 (0.035847313914567816) | > loss_mel_ce: 0.09365234524011612 (nan) | > loss: 0.0005235783755779266 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13666485503241152) | > loader_time: 0.0046 (0.004660878773055342)  --> TIME: 2025-02-25 04:20:13 -- STEP: 32742/36938 -- GLOBAL_STEP: 623750 | > loss_text_ce: 0.034275177866220474 (0.03584805590589603) | > loss_mel_ce: 3.023711681365967 (nan) | > loss: 0.012134869582951069 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13666177189901332) | > loader_time: 0.0046 (0.004660855579824802)  --> TIME: 2025-02-25 04:20:23 -- STEP: 32792/36938 -- GLOBAL_STEP: 623800 | > loss_text_ce: 0.028126660734415054 (0.03584807824767585) | > loss_mel_ce: 2.7969231605529785 (nan) | > loss: 0.011210516095161438 (nan) | > current_lr: 5e-06 | > step_time: 0.1793 (0.13667288384777224) | > loader_time: 0.0048 (0.004660936289224514)  --> TIME: 2025-02-25 04:20:32 -- STEP: 32842/36938 -- GLOBAL_STEP: 623850 | > loss_text_ce: 0.03932136297225952 (0.035847191825153724) | > loss_mel_ce: 2.950136661529541 (nan) | > loss: 0.011862929910421371 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.1366741068506491) | > loader_time: 0.0047 (0.004660986792655773)  --> TIME: 2025-02-25 04:20:42 -- STEP: 32892/36938 -- GLOBAL_STEP: 623900 | > loss_text_ce: 0.03731530159711838 (0.03584782266530601) | > loss_mel_ce: 3.30309796333313 (nan) | > loss: 0.013255609199404716 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.13667527870816995) | > loader_time: 0.0047 (0.004661052661642022)  --> TIME: 2025-02-25 04:20:51 -- STEP: 32942/36938 -- GLOBAL_STEP: 623950 | > loss_text_ce: 0.02860204689204693 (0.035847594934948185) | > loss_mel_ce: 1.2063645124435425 (nan) | > loss: 0.004900660831481218 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13667889983484205) | > loader_time: 0.0046 (0.0046610561530886045)  --> TIME: 2025-02-25 04:21:01 -- STEP: 32992/36938 -- GLOBAL_STEP: 624000 | > loss_text_ce: 0.04566962644457817 (0.03584820680714509) | > loss_mel_ce: 2.6701297760009766 (nan) | > loss: 0.010776981711387634 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1366901246814559) | > loader_time: 0.0045 (0.004661102689780993)  --> TIME: 2025-02-25 04:21:10 -- STEP: 33042/36938 -- GLOBAL_STEP: 624050 | > loss_text_ce: 0.023595362901687622 (0.03584765053167728) | > loss_mel_ce: 2.3289361000061035 (nan) | > loss: 0.009335442446172237 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13668697460909487) | > loader_time: 0.0047 (0.0046610885970738945)  --> TIME: 2025-02-25 04:21:19 -- STEP: 33092/36938 -- GLOBAL_STEP: 624100 | > loss_text_ce: 0.03947807103395462 (0.03584748696728049) | > loss_mel_ce: 3.2168467044830322 (nan) | > loss: 0.012921924702823162 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1366841524684497) | > loader_time: 0.0046 (0.004661116521649297)  --> TIME: 2025-02-25 04:21:29 -- STEP: 33142/36938 -- GLOBAL_STEP: 624150 | > loss_text_ce: 0.040619511157274246 (0.03584850822967009) | > loss_mel_ce: 3.4671473503112793 (nan) | > loss: 0.013919711112976074 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13668657413871912) | > loader_time: 0.0046 (0.004661176583221813)  --> TIME: 2025-02-25 04:21:38 -- STEP: 33192/36938 -- GLOBAL_STEP: 624200 | > loss_text_ce: 0.03414568677544594 (0.03584763009728939) | > loss_mel_ce: 3.0970067977905273 (nan) | > loss: 0.01242520846426487 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13669141076880467) | > loader_time: 0.0047 (0.0046612997461899855)  --> TIME: 2025-02-25 04:21:48 -- STEP: 33242/36938 -- GLOBAL_STEP: 624250 | > loss_text_ce: 0.02280859462916851 (0.03584439519670142) | > loss_mel_ce: 7.40625 (nan) | > loss: 0.02948039211332798 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1366963379865487) | > loader_time: 0.0047 (0.004661321553948165)  --> TIME: 2025-02-25 04:21:57 -- STEP: 33292/36938 -- GLOBAL_STEP: 624300 | > loss_text_ce: 0.034193091094493866 (0.0358437649423622) | > loss_mel_ce: 2.3398053646087646 (nan) | > loss: 0.00942062959074974 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13670499860760538) | > loader_time: 0.0046 (0.0046613918006184075)  --> TIME: 2025-02-25 04:22:06 -- STEP: 33342/36938 -- GLOBAL_STEP: 624350 | > loss_text_ce: 0.03900082781910896 (0.035845081391613895) | > loss_mel_ce: 3.057626724243164 (nan) | > loss: 0.012288205325603485 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.136701812956373) | > loader_time: 0.0046 (0.004661356299448808)  --> TIME: 2025-02-25 04:22:16 -- STEP: 33392/36938 -- GLOBAL_STEP: 624400 | > loss_text_ce: 0.042876482009887695 (0.03584777577186622) | > loss_mel_ce: 2.7261126041412354 (nan) | > loss: 0.010988052934408188 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.1367013599664669) | > loader_time: 0.0047 (0.004661356268970968)  --> TIME: 2025-02-25 04:22:25 -- STEP: 33442/36938 -- GLOBAL_STEP: 624450 | > loss_text_ce: 0.028394998982548714 (0.03584759685836708) | > loss_mel_ce: 2.9492621421813965 (nan) | > loss: 0.011816100217401981 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.1366986469192711) | > loader_time: 0.0048 (0.0046613289404340555)  --> TIME: 2025-02-25 04:22:34 -- STEP: 33492/36938 -- GLOBAL_STEP: 624500 | > loss_text_ce: 0.031235193833708763 (0.03584741965175572) | > loss_mel_ce: 2.8623483180999756 (nan) | > loss: 0.011482475325465202 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13670270550169816) | > loader_time: 0.0047 (0.0046613515669138295)  --> TIME: 2025-02-25 04:22:43 -- STEP: 33542/36938 -- GLOBAL_STEP: 624550 | > loss_text_ce: 0.04559679329395294 (0.03584723401491232) | > loss_mel_ce: 3.1798648834228516 (nan) | > loss: 0.012799452058970928 (nan) | > current_lr: 5e-06 | > step_time: 0.1556 (0.13669506911834498) | > loader_time: 0.0048 (0.004661332657509087)  --> TIME: 2025-02-25 04:22:53 -- STEP: 33592/36938 -- GLOBAL_STEP: 624600 | > loss_text_ce: 0.03667750209569931 (0.035848380983415175) | > loss_mel_ce: 3.154210329055786 (nan) | > loss: 0.012662254273891449 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13669929189408295) | > loader_time: 0.0046 (0.004661303143979393)  --> TIME: 2025-02-25 04:23:02 -- STEP: 33642/36938 -- GLOBAL_STEP: 624650 | > loss_text_ce: 0.028872495517134666 (0.03584628379692459) | > loss_mel_ce: 2.7880284786224365 (nan) | > loss: 0.01117817871272564 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13670080671638657) | > loader_time: 0.0045 (0.004661240544247458)  --> TIME: 2025-02-25 04:23:12 -- STEP: 33692/36938 -- GLOBAL_STEP: 624700 | > loss_text_ce: 0.02963675931096077 (0.035844949130092216) | > loss_mel_ce: 2.8107399940490723 (nan) | > loss: 0.011271337047219276 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13670512539579244) | > loader_time: 0.0046 (0.004661210561526127)  --> TIME: 2025-02-25 04:23:21 -- STEP: 33742/36938 -- GLOBAL_STEP: 624750 | > loss_text_ce: 0.035971518605947495 (0.035843856758350036) | > loss_mel_ce: 3.082998275756836 (nan) | > loss: 0.012376864440739155 (nan) | > current_lr: 5e-06 | > step_time: 0.1811 (0.13671561448690334) | > loader_time: 0.0048 (0.00466120868407236)  --> TIME: 2025-02-25 04:23:30 -- STEP: 33792/36938 -- GLOBAL_STEP: 624800 | > loss_text_ce: 0.04697585850954056 (0.035844392242958205) | > loss_mel_ce: 3.8246092796325684 (nan) | > loss: 0.015363434329628944 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13670932240501915) | > loader_time: 0.0046 (0.004661134846339196)  --> TIME: 2025-02-25 04:23:40 -- STEP: 33842/36938 -- GLOBAL_STEP: 624850 | > loss_text_ce: 0.03830965235829353 (0.0358442926519488) | > loss_mel_ce: 2.688100576400757 (nan) | > loss: 0.01081908866763115 (nan) | > current_lr: 5e-06 | > step_time: 0.1504 (0.13671270834132185) | > loader_time: 0.0046 (0.004661237007830086)  --> TIME: 2025-02-25 04:23:49 -- STEP: 33892/36938 -- GLOBAL_STEP: 624900 | > loss_text_ce: 0.03592465817928314 (0.03584465036679976) | > loss_mel_ce: 0.0045928955078125 (nan) | > loss: 0.00016078395128715783 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.1367096951308078) | > loader_time: 0.0047 (0.004661234811249127)  --> TIME: 2025-02-25 04:23:58 -- STEP: 33942/36938 -- GLOBAL_STEP: 624950 | > loss_text_ce: 0.04718984663486481 (0.03584662838593917) | > loss_mel_ce: 3.2548062801361084 (nan) | > loss: 0.013103160075843334 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1367011083316647) | > loader_time: 0.0062 (0.004661291210774065)  --> TIME: 2025-02-25 04:24:07 -- STEP: 33992/36938 -- GLOBAL_STEP: 625000 | > loss_text_ce: 0.0387115404009819 (0.03584887429100346) | > loss_mel_ce: 3.3099048137664795 (nan) | > loss: 0.013288160786032677 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13669349736088848) | > loader_time: 0.0046 (0.004661198285473879)  --> TIME: 2025-02-25 04:24:17 -- STEP: 34042/36938 -- GLOBAL_STEP: 625050 | > loss_text_ce: 0.03363421931862831 (0.0358487384976642) | > loss_mel_ce: 2.5850088596343994 (nan) | > loss: 0.01039144117385149 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13669169032470704) | > loader_time: 0.0047 (0.004661144132262857)  --> TIME: 2025-02-25 04:24:26 -- STEP: 34092/36938 -- GLOBAL_STEP: 625100 | > loss_text_ce: 0.03158959373831749 (0.03584581130699072) | > loss_mel_ce: 3.117133617401123 (nan) | > loss: 0.012494933791458607 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.13669235913613317) | > loader_time: 0.0047 (0.004661104824010687)  --> TIME: 2025-02-25 04:24:35 -- STEP: 34142/36938 -- GLOBAL_STEP: 625150 | > loss_text_ce: 0.03050943836569786 (0.03584500908492925) | > loss_mel_ce: 0.007232666015625 (nan) | > loss: 0.00014977026148699224 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1366956074553038) | > loader_time: 0.0045 (0.004661305201688413)  --> TIME: 2025-02-25 04:24:45 -- STEP: 34192/36938 -- GLOBAL_STEP: 625200 | > loss_text_ce: 0.0285942405462265 (0.03584337962136365) | > loss_mel_ce: 3.198072671890259 (nan) | > loss: 0.012804234400391579 (nan) | > current_lr: 5e-06 | > step_time: 0.1837 (0.1367022248632612) | > loader_time: 0.0048 (0.004661358554730332)  --> TIME: 2025-02-25 04:24:54 -- STEP: 34242/36938 -- GLOBAL_STEP: 625250 | > loss_text_ce: 0.03490903973579407 (0.03584453032062212) | > loss_mel_ce: 2.1312005519866943 (nan) | > loss: 0.008595673367381096 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13670392201037415) | > loader_time: 0.0046 (0.004661389269231975)  --> TIME: 2025-02-25 04:25:04 -- STEP: 34292/36938 -- GLOBAL_STEP: 625300 | > loss_text_ce: 0.02621546946465969 (0.03584297283129517) | > loss_mel_ce: 3.2730278968811035 (nan) | > loss: 0.01309223659336567 (nan) | > current_lr: 5e-06 | > step_time: 0.172 (0.13670406686540734) | > loader_time: 0.0048 (0.004661409972805048)  --> TIME: 2025-02-25 04:25:13 -- STEP: 34342/36938 -- GLOBAL_STEP: 625350 | > loss_text_ce: 0.03919047862291336 (0.03584136324404435) | > loss_mel_ce: 2.961021661758423 (nan) | > loss: 0.011905604973435402 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13670113698193867) | > loader_time: 0.0046 (0.004661568590899226)  --> TIME: 2025-02-25 04:25:22 -- STEP: 34392/36938 -- GLOBAL_STEP: 625400 | > loss_text_ce: 0.04232041910290718 (0.03584155874213798) | > loss_mel_ce: 2.3484089374542236 (nan) | > loss: 0.009487021714448929 (nan) | > current_lr: 5e-06 | > step_time: 0.1121 (0.1366940844361239) | > loader_time: 0.0048 (0.004661459830173893)  --> TIME: 2025-02-25 04:25:31 -- STEP: 34442/36938 -- GLOBAL_STEP: 625450 | > loss_text_ce: 0.028377218171954155 (0.03584126965560508) | > loss_mel_ce: 3.109375 (nan) | > loss: 0.012451399117708206 (nan) | > current_lr: 5e-06 | > step_time: 0.1582 (0.1366981231725335) | > loader_time: 0.0046 (0.004661401592821964)  --> TIME: 2025-02-25 04:25:40 -- STEP: 34492/36938 -- GLOBAL_STEP: 625500 | > loss_text_ce: 0.03669695928692818 (0.03584149104207519) | > loss_mel_ce: 3.1903796195983887 (nan) | > loss: 0.012805859558284283 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13669320409070904) | > loader_time: 0.0046 (0.004661408769383502)  --> TIME: 2025-02-25 04:25:49 -- STEP: 34542/36938 -- GLOBAL_STEP: 625550 | > loss_text_ce: 0.04183739423751831 (0.03584042453546694) | > loss_mel_ce: 2.9595751762390137 (nan) | > loss: 0.01191036868840456 (nan) | > current_lr: 5e-06 | > step_time: 0.1723 (0.13668465040049052) | > loader_time: 0.0048 (0.004661268306091894)  --> TIME: 2025-02-25 04:25:58 -- STEP: 34592/36938 -- GLOBAL_STEP: 625600 | > loss_text_ce: 0.03535990044474602 (0.035841385676176334) | > loss_mel_ce: 2.6377692222595215 (nan) | > loss: 0.010607656091451645 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13668131962018762) | > loader_time: 0.0046 (0.004661182746304968)  --> TIME: 2025-02-25 04:26:08 -- STEP: 34642/36938 -- GLOBAL_STEP: 625650 | > loss_text_ce: 0.03389016166329384 (0.035841595896464754) | > loss_mel_ce: 3.2763943672180176 (nan) | > loss: 0.013136050663888454 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1366930311584116) | > loader_time: 0.0045 (0.004661124549983791)  --> TIME: 2025-02-25 04:26:18 -- STEP: 34692/36938 -- GLOBAL_STEP: 625700 | > loss_text_ce: 0.03669566288590431 (0.0358419829914729) | > loss_mel_ce: 3.399152994155884 (nan) | > loss: 0.01363432127982378 (nan) | > current_lr: 5e-06 | > step_time: 0.177 (0.1366962684516217) | > loader_time: 0.0048 (0.004661103075905622)  --> TIME: 2025-02-25 04:26:27 -- STEP: 34742/36938 -- GLOBAL_STEP: 625750 | > loss_text_ce: 0.02852211706340313 (0.03584033717614605) | > loss_mel_ce: 0.1148664653301239 (nan) | > loss: 0.0005690023535862565 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13669640407421663) | > loader_time: 0.0046 (0.004661000301290176)  --> TIME: 2025-02-25 04:26:37 -- STEP: 34792/36938 -- GLOBAL_STEP: 625800 | > loss_text_ce: 0.028487741947174072 (0.0358417570354818) | > loss_mel_ce: 2.7318131923675537 (nan) | > loss: 0.010953575372695923 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13669844286610058) | > loader_time: 0.0048 (0.00466098547475207)  --> TIME: 2025-02-25 04:26:46 -- STEP: 34842/36938 -- GLOBAL_STEP: 625850 | > loss_text_ce: 0.04598795250058174 (0.0358428953904411) | > loss_mel_ce: 3.3911783695220947 (nan) | > loss: 0.013639548793435097 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1366963255563491) | > loader_time: 0.0047 (0.004660854540231994)  --> TIME: 2025-02-25 04:26:55 -- STEP: 34892/36938 -- GLOBAL_STEP: 625900 | > loss_text_ce: 0.033660583198070526 (0.03584291227447436) | > loss_mel_ce: 3.460482120513916 (nan) | > loss: 0.013865646906197071 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.13670094436452832) | > loader_time: 0.0046 (0.004660750582009647)  --> TIME: 2025-02-25 04:27:05 -- STEP: 34942/36938 -- GLOBAL_STEP: 625950 | > loss_text_ce: 0.03782714158296585 (0.03584222689171345) | > loss_mel_ce: 2.6619298458099365 (nan) | > loss: 0.010713322088122368 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.1367038920164364) | > loader_time: 0.0045 (0.0046607061677271045)  --> TIME: 2025-02-25 04:27:14 -- STEP: 34992/36938 -- GLOBAL_STEP: 626000 | > loss_text_ce: 0.04384768754243851 (0.03584171589589524) | > loss_mel_ce: 3.037013053894043 (nan) | > loss: 0.012225639075040817 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13670245326704522) | > loader_time: 0.0046 (0.004660678968671602)  --> TIME: 2025-02-25 04:27:23 -- STEP: 35042/36938 -- GLOBAL_STEP: 626050 | > loss_text_ce: 0.0379873663187027 (0.03584341781797363) | > loss_mel_ce: 2.532827615737915 (nan) | > loss: 0.010201647877693176 (nan) | > current_lr: 5e-06 | > step_time: 0.1823 (0.13670310033292304) | > loader_time: 0.0047 (0.004660641131257658)  --> TIME: 2025-02-25 04:27:33 -- STEP: 35092/36938 -- GLOBAL_STEP: 626100 | > loss_text_ce: 0.03899689391255379 (0.03584463125200668) | > loss_mel_ce: 3.075617551803589 (nan) | > loss: 0.012359581887722015 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13670239024051917) | > loader_time: 0.0045 (0.004660510261341732)  --> TIME: 2025-02-25 04:27:42 -- STEP: 35142/36938 -- GLOBAL_STEP: 626150 | > loss_text_ce: 0.030436133965849876 (0.03584523153117118) | > loss_mel_ce: 2.872006893157959 (nan) | > loss: 0.011517630890011787 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.1367089162953756) | > loader_time: 0.0047 (0.004660474705198148)  --> TIME: 2025-02-25 04:27:51 -- STEP: 35192/36938 -- GLOBAL_STEP: 626200 | > loss_text_ce: 0.038790106773376465 (0.03584425151803244) | > loss_mel_ce: 3.599588632583618 (nan) | > loss: 0.014438011683523655 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.1367038037999698) | > loader_time: 0.0065 (0.004660445963910957)  --> TIME: 2025-02-25 04:28:01 -- STEP: 35242/36938 -- GLOBAL_STEP: 626250 | > loss_text_ce: 0.04179748520255089 (0.0358435849230872) | > loss_mel_ce: 2.034475803375244 (nan) | > loss: 0.008239179849624634 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13670258163884857) | > loader_time: 0.0045 (0.004660353217599335)  --> TIME: 2025-02-25 04:28:10 -- STEP: 35292/36938 -- GLOBAL_STEP: 626300 | > loss_text_ce: 0.0406547412276268 (0.03584443678830039) | > loss_mel_ce: 2.902742624282837 (nan) | > loss: 0.011680148541927338 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.1367037469306037) | > loader_time: 0.0048 (0.0046603187038746574)  --> TIME: 2025-02-25 04:28:19 -- STEP: 35342/36938 -- GLOBAL_STEP: 626350 | > loss_text_ce: 0.03448968008160591 (0.03584488392678103) | > loss_mel_ce: 2.9350242614746094 (nan) | > loss: 0.011783786118030548 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.1367071022587187) | > loader_time: 0.0046 (0.004660277663194166)  --> TIME: 2025-02-25 04:28:29 -- STEP: 35392/36938 -- GLOBAL_STEP: 626400 | > loss_text_ce: 0.03843158110976219 (0.03584540146166132) | > loss_mel_ce: 2.13555908203125 (nan) | > loss: 0.008626948110759258 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13670785927212012) | > loader_time: 0.0045 (0.0046603048513315275)  --> TIME: 2025-02-25 04:28:38 -- STEP: 35442/36938 -- GLOBAL_STEP: 626450 | > loss_text_ce: 0.03589188680052757 (0.035843667882591274) | > loss_mel_ce: 3.292280673980713 (nan) | > loss: 0.013207034207880497 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.1367027981745037) | > loader_time: 0.0046 (0.0046602635020078715)  --> TIME: 2025-02-25 04:28:47 -- STEP: 35492/36938 -- GLOBAL_STEP: 626500 | > loss_text_ce: 0.03148844465613365 (0.035841036699264625) | > loss_mel_ce: 1.3073018789291382 (nan) | > loss: 0.005312660243362188 (nan) | > current_lr: 5e-06 | > step_time: 0.1075 (0.13669665574113848) | > loader_time: 0.0044 (0.004660241582086543)  --> TIME: 2025-02-25 04:28:56 -- STEP: 35542/36938 -- GLOBAL_STEP: 626550 | > loss_text_ce: 0.03387284278869629 (0.03584025876867883) | > loss_mel_ce: 2.559812545776367 (nan) | > loss: 0.01029240246862173 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13669298085033482) | > loader_time: 0.0045 (0.00466018762567574)  --> TIME: 2025-02-25 04:29:06 -- STEP: 35592/36938 -- GLOBAL_STEP: 626600 | > loss_text_ce: 0.034500863403081894 (0.03583865027962999) | > loss_mel_ce: 3.1596481800079346 (nan) | > loss: 0.012675195001065731 (nan) | > current_lr: 5e-06 | > step_time: 0.1899 (0.13669434842915404) | > loader_time: 0.0048 (0.004660172197464551)  --> TIME: 2025-02-25 04:29:14 -- STEP: 35642/36938 -- GLOBAL_STEP: 626650 | > loss_text_ce: 0.034155312925577164 (0.035839305922777794) | > loss_mel_ce: 0.0121162636205554 (nan) | > loss: 0.0001836173760239035 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13668594472864853) | > loader_time: 0.0044 (0.004660080735594224)  --> TIME: 2025-02-25 04:29:24 -- STEP: 35692/36938 -- GLOBAL_STEP: 626700 | > loss_text_ce: 0.04136132448911667 (0.03583925980843861) | > loss_mel_ce: 3.0083138942718506 (nan) | > loss: 0.012101886793971062 (nan) | > current_lr: 5e-06 | > step_time: 0.1501 (0.13669034410829548) | > loader_time: 0.0046 (0.004660050063126309)  --> TIME: 2025-02-25 04:29:33 -- STEP: 35742/36938 -- GLOBAL_STEP: 626750 | > loss_text_ce: 0.03718274086713791 (0.03583899270944244) | > loss_mel_ce: 3.165712356567383 (nan) | > loss: 0.012709902599453926 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13669026609059662) | > loader_time: 0.0047 (0.00465994293198377)  --> TIME: 2025-02-25 04:29:43 -- STEP: 35792/36938 -- GLOBAL_STEP: 626800 | > loss_text_ce: 0.03138717636466026 (0.03584253377574386) | > loss_mel_ce: 2.655388355255127 (nan) | > loss: 0.010661807842552662 (nan) | > current_lr: 5e-06 | > step_time: 0.1625 (0.13669369555260344) | > loader_time: 0.0047 (0.004659880337354927)  --> TIME: 2025-02-25 04:29:52 -- STEP: 35842/36938 -- GLOBAL_STEP: 626850 | > loss_text_ce: 0.03931136429309845 (0.03584174899203795) | > loss_mel_ce: 3.1727004051208496 (nan) | > loss: 0.012746079824864864 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13669605597967538) | > loader_time: 0.0047 (0.004659832937431021)  --> TIME: 2025-02-25 04:30:02 -- STEP: 35892/36938 -- GLOBAL_STEP: 626900 | > loss_text_ce: 0.04250513017177582 (0.03584182434995003) | > loss_mel_ce: 3.1255862712860107 (nan) | > loss: 0.012571791186928749 (nan) | > current_lr: 5e-06 | > step_time: 0.1783 (0.1367016114492975) | > loader_time: 0.0048 (0.004659776110772944)  --> TIME: 2025-02-25 04:30:11 -- STEP: 35942/36938 -- GLOBAL_STEP: 626950 | > loss_text_ce: 0.04336697980761528 (0.03584352679692392) | > loss_mel_ce: 3.0790226459503174 (nan) | > loss: 0.012390435673296452 (nan) | > current_lr: 5e-06 | > step_time: 0.1613 (0.13670599720632048) | > loader_time: 0.0046 (0.004659716052540996)  --> TIME: 2025-02-25 04:30:21 -- STEP: 35992/36938 -- GLOBAL_STEP: 627000 | > loss_text_ce: 0.03884146735072136 (0.03584250023434897) | > loss_mel_ce: 2.810257911682129 (nan) | > loss: 0.011305950582027435 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13670342377647568) | > loader_time: 0.0045 (0.004659616720626095)  --> TIME: 2025-02-25 04:30:30 -- STEP: 36042/36938 -- GLOBAL_STEP: 627050 | > loss_text_ce: 0.036422837525606155 (0.035840609298540924) | > loss_mel_ce: 2.311501979827881 (nan) | > loss: 0.009317162446677685 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13670338148097222) | > loader_time: 0.0045 (0.004659576266780772)  --> TIME: 2025-02-25 04:30:40 -- STEP: 36092/36938 -- GLOBAL_STEP: 627100 | > loss_text_ce: 0.037162791937589645 (0.03583866566656752) | > loss_mel_ce: 3.353125810623169 (nan) | > loss: 0.013453527353703976 (nan) | > current_lr: 5e-06 | > step_time: 0.1435 (0.13670888594984892) | > loader_time: 0.0046 (0.004659578922541898)  --> TIME: 2025-02-25 04:30:49 -- STEP: 36142/36938 -- GLOBAL_STEP: 627150 | > loss_text_ce: 0.03792576491832733 (0.035837111265377176) | > loss_mel_ce: 3.379673957824707 (nan) | > loss: 0.013561904430389404 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.1367045272320832) | > loader_time: 0.0046 (0.004659484843280017)  --> TIME: 2025-02-25 04:30:58 -- STEP: 36192/36938 -- GLOBAL_STEP: 627200 | > loss_text_ce: 0.04033609479665756 (0.035835584485923576) | > loss_mel_ce: 3.033743381500244 (nan) | > loss: 0.012198728509247303 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13670406219813763) | > loader_time: 0.0048 (0.004659411840793011)  --> TIME: 2025-02-25 04:31:07 -- STEP: 36242/36938 -- GLOBAL_STEP: 627250 | > loss_text_ce: 0.03782562166452408 (0.035837205515116426) | > loss_mel_ce: 2.7363369464874268 (nan) | > loss: 0.011008582077920437 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13670232663597423) | > loader_time: 0.0045 (0.004659382286901215)  --> TIME: 2025-02-25 04:31:17 -- STEP: 36292/36938 -- GLOBAL_STEP: 627300 | > loss_text_ce: 0.03433990478515625 (0.035838186875189346) | > loss_mel_ce: 2.5277726650238037 (nan) | > loss: 0.01016711350530386 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13670822615057404) | > loader_time: 0.0045 (0.004659369954145262)  --> TIME: 2025-02-25 04:31:26 -- STEP: 36342/36938 -- GLOBAL_STEP: 627350 | > loss_text_ce: 0.03542632609605789 (0.0358382288748286) | > loss_mel_ce: 2.1841537952423096 (nan) | > loss: 0.00880785845220089 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13671000362931743) | > loader_time: 0.0045 (0.00465940423426662)  --> TIME: 2025-02-25 04:31:35 -- STEP: 36392/36938 -- GLOBAL_STEP: 627400 | > loss_text_ce: 0.035697516053915024 (0.03583892462622545) | > loss_mel_ce: 2.7266151905059814 (nan) | > loss: 0.010961558669805527 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13670306153731346) | > loader_time: 0.0044 (0.004659295874811802)  --> TIME: 2025-02-25 04:31:45 -- STEP: 36442/36938 -- GLOBAL_STEP: 627450 | > loss_text_ce: 0.03658241033554077 (0.03583913452829821) | > loss_mel_ce: 2.6722328662872314 (nan) | > loss: 0.010749267414212227 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.13670919740457124) | > loader_time: 0.0046 (0.004659244077448135)  --> TIME: 2025-02-25 04:31:54 -- STEP: 36492/36938 -- GLOBAL_STEP: 627500 | > loss_text_ce: 0.03075900301337242 (0.03583864597493384) | > loss_mel_ce: 2.4419007301330566 (nan) | > loss: 0.009812142699956894 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.136715358160546) | > loader_time: 0.0046 (0.004659192859767172)  --> TIME: 2025-02-25 04:32:04 -- STEP: 36542/36938 -- GLOBAL_STEP: 627550 | > loss_text_ce: 0.0334075465798378 (0.03583818446733857) | > loss_mel_ce: 2.995244264602661 (nan) | > loss: 0.012018459849059582 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13671136396310013) | > loader_time: 0.0045 (0.004659132491347645)  --> TIME: 2025-02-25 04:32:13 -- STEP: 36592/36938 -- GLOBAL_STEP: 627600 | > loss_text_ce: 0.04608333483338356 (0.0358384931728278) | > loss_mel_ce: 3.074683666229248 (nan) | > loss: 0.012383997440338135 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13670394777808542) | > loader_time: 0.0045 (0.004659027675639932)  --> TIME: 2025-02-25 04:32:22 -- STEP: 36642/36938 -- GLOBAL_STEP: 627650 | > loss_text_ce: 0.025462452322244644 (0.035839701379127856) | > loss_mel_ce: 3.4187562465667725 (nan) | > loss: 0.013667535036802292 (nan) | > current_lr: 5e-06 | > step_time: 0.1966 (0.13671525984402244) | > loader_time: 0.0049 (0.00465896116464617)  --> TIME: 2025-02-25 04:32:32 -- STEP: 36692/36938 -- GLOBAL_STEP: 627700 | > loss_text_ce: 0.03373359516263008 (0.03583899678480241) | > loss_mel_ce: 0.003631591796875 (nan) | > loss: 0.00014827455743215978 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13672156168488428) | > loader_time: 0.0045 (0.004658854359903638)  --> TIME: 2025-02-25 04:32:41 -- STEP: 36742/36938 -- GLOBAL_STEP: 627750 | > loss_text_ce: 0.03164971247315407 (0.03584020927313757) | > loss_mel_ce: 2.788431167602539 (nan) | > loss: 0.011190798133611679 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13672285218560928) | > loader_time: 0.0045 (0.004658764256512715)  --> TIME: 2025-02-25 04:32:51 -- STEP: 36792/36938 -- GLOBAL_STEP: 627800 | > loss_text_ce: 0.037450533360242844 (0.03584098380019888) | > loss_mel_ce: 3.65439510345459 (nan) | > loss: 0.014650181867182255 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13672416703056614) | > loader_time: 0.0045 (0.004658667347591382)  --> TIME: 2025-02-25 04:33:00 -- STEP: 36842/36938 -- GLOBAL_STEP: 627850 | > loss_text_ce: 0.040511708706617355 (0.03584134949782828) | > loss_mel_ce: 3.0159199237823486 (nan) | > loss: 0.012128697708249092 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.1367253643003478) | > loader_time: 0.0045 (0.004658557422437698)  --> TIME: 2025-02-25 04:33:09 -- STEP: 36892/36938 -- GLOBAL_STEP: 627900 | > loss_text_ce: 0.05313432961702347 (0.035840421960191285) | > loss_mel_ce: 5.1160712242126465 (nan) | > loss: 0.020512722432613373 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.1367221132840118) | > loader_time: 0.0046 (0.004658449003756683)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.021116197699367397 (+0.00014676670036691333) | > avg_loss_text_ce: 0.03498688088453346 (-0.00040404217336140885) | > avg_loss_mel_ce: 2.5961150568310565 (-0.018995589549000158) | > avg_loss: 2.6311019349216256 (-0.019399629075927294) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_627946.pth  > EPOCH: 18/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-25 04:33:43)   --> TIME: 2025-02-25 04:33:44 -- STEP: 4/36938 -- GLOBAL_STEP: 627950 | > loss_text_ce: 0.03466479480266571 (0.035014549270272255) | > loss_mel_ce: 2.9776437282562256 (2.9510980248451233) | > loss: 0.011953606270253658 (0.011849653674289584) | > current_lr: 5e-06 | > step_time: 0.1186 (0.1420615315437317) | > loader_time: 0.0044 (0.006362318992614746)  --> TIME: 2025-02-25 04:33:53 -- STEP: 54/36938 -- GLOBAL_STEP: 628000 | > loss_text_ce: 0.046675972640514374 (0.03686000668892154) | > loss_mel_ce: 1.2753959894180298 (2.749530916136724) | > loss: 0.005246317479759455 (0.011057107424238455) | > current_lr: 5e-06 | > step_time: 0.1354 (0.1379331217871773) | > loader_time: 0.0048 (0.004970091360586659)  --> TIME: 2025-02-25 04:34:03 -- STEP: 104/36938 -- GLOBAL_STEP: 628050 | > loss_text_ce: 0.04425044357776642 (0.03576956078625069) | > loss_mel_ce: 2.1063232421875 (2.6566380176406628) | > loss: 0.00853402353823185 (0.010684157676923158) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13825723070364743) | > loader_time: 0.0047 (0.004903016182092522)  --> TIME: 2025-02-25 04:34:12 -- STEP: 154/36938 -- GLOBAL_STEP: 628100 | > loss_text_ce: 0.03523861989378929 (0.03606025910900012) | > loss_mel_ce: 2.7159087657928467 (2.7195754426640333) | > loss: 0.01091725192964077 (0.01093506293867219) | > current_lr: 5e-06 | > step_time: 0.128 (0.1373657545486055) | > loader_time: 0.0047 (0.004884362220764163)  --> TIME: 2025-02-25 04:34:21 -- STEP: 204/36938 -- GLOBAL_STEP: 628150 | > loss_text_ce: 0.033645886927843094 (0.03617869699191229) | > loss_mel_ce: 3.374567985534668 (nan) | > loss: 0.013524658977985382 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.13641931730158185) | > loader_time: 0.0049 (0.00484663481805839)  --> TIME: 2025-02-25 04:34:30 -- STEP: 254/36938 -- GLOBAL_STEP: 628200 | > loss_text_ce: 0.03408105671405792 (0.03595243257243097) | > loss_mel_ce: 1.2221035957336426 (nan) | > loss: 0.004984860308468342 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.1361378393773957) | > loader_time: 0.0047 (0.0048400539112842)  --> TIME: 2025-02-25 04:34:40 -- STEP: 304/36938 -- GLOBAL_STEP: 628250 | > loss_text_ce: 0.03906167298555374 (0.035630105755684) | > loss_mel_ce: 2.854189872741699 (nan) | > loss: 0.011481158435344696 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1357691774242803) | > loader_time: 0.0047 (0.004820736615281357)  --> TIME: 2025-02-25 04:34:49 -- STEP: 354/36938 -- GLOBAL_STEP: 628300 | > loss_text_ce: 0.03603934869170189 (0.035700453645260304) | > loss_mel_ce: 3.106815814971924 (nan) | > loss: 0.012471647933125496 (nan) | > current_lr: 5e-06 | > step_time: 0.1352 (0.13577853757782843) | > loader_time: 0.0046 (0.0048137249919654275)  --> TIME: 2025-02-25 04:34:58 -- STEP: 404/36938 -- GLOBAL_STEP: 628350 | > loss_text_ce: 0.032504357397556305 (0.03578810782937128) | > loss_mel_ce: 3.177678108215332 (nan) | > loss: 0.012738820165395737 (nan) | > current_lr: 5e-06 | > step_time: 0.186 (0.1358143905601879) | > loader_time: 0.0049 (0.004809224369502302)  --> TIME: 2025-02-25 04:35:08 -- STEP: 454/36938 -- GLOBAL_STEP: 628400 | > loss_text_ce: 0.028054289519786835 (0.03574549679394137) | > loss_mel_ce: 0.0322469063103199 (nan) | > loss: 0.00023929047165438533 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.1361107910257079) | > loader_time: 0.0046 (0.004799823403883609)  --> TIME: 2025-02-25 04:35:17 -- STEP: 504/36938 -- GLOBAL_STEP: 628450 | > loss_text_ce: 0.03583342581987381 (0.03568787379227282) | > loss_mel_ce: 2.980985641479492 (nan) | > loss: 0.011971504427492619 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13626627080024228) | > loader_time: 0.0046 (0.004783740119328573)  --> TIME: 2025-02-25 04:35:27 -- STEP: 554/36938 -- GLOBAL_STEP: 628500 | > loss_text_ce: 0.04281909018754959 (0.03576003899125362) | > loss_mel_ce: 2.802324056625366 (nan) | > loss: 0.011290251277387142 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13685407879550535) | > loader_time: 0.0065 (0.004789618378511834)  --> TIME: 2025-02-25 04:35:36 -- STEP: 604/36938 -- GLOBAL_STEP: 628550 | > loss_text_ce: 0.04040759056806564 (0.03591516032808813) | > loss_mel_ce: 3.5977258682250977 (nan) | > loss: 0.014437038451433182 (nan) | > current_lr: 5e-06 | > step_time: 0.1917 (0.1365855786184601) | > loader_time: 0.005 (0.0047842733907383795)  --> TIME: 2025-02-25 04:35:45 -- STEP: 654/36938 -- GLOBAL_STEP: 628600 | > loss_text_ce: 0.03427605703473091 (0.03586941178096317) | > loss_mel_ce: 2.9449069499969482 (nan) | > loss: 0.011822155676782131 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1366995824586361) | > loader_time: 0.0046 (0.004781264413022849)  --> TIME: 2025-02-25 04:35:55 -- STEP: 704/36938 -- GLOBAL_STEP: 628650 | > loss_text_ce: 0.024552935734391212 (0.03582047838162083) | > loss_mel_ce: 1.005712866783142 (nan) | > loss: 0.004088356625288725 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13704651594161987) | > loader_time: 0.0046 (0.004776319319551643)  --> TIME: 2025-02-25 04:36:04 -- STEP: 754/36938 -- GLOBAL_STEP: 628700 | > loss_text_ce: 0.03783098980784416 (0.03583754740027954) | > loss_mel_ce: 2.696516990661621 (nan) | > loss: 0.010850587859749794 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13683214111732872) | > loader_time: 0.0048 (0.004773447937295039)  --> TIME: 2025-02-25 04:36:13 -- STEP: 804/36938 -- GLOBAL_STEP: 628750 | > loss_text_ce: 0.034535061568021774 (0.035772403643063444) | > loss_mel_ce: 2.8287925720214844 (nan) | > loss: 0.011362412013113499 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13691432114264274) | > loader_time: 0.0046 (0.004768201367772046)  --> TIME: 2025-02-25 04:36:23 -- STEP: 854/36938 -- GLOBAL_STEP: 628800 | > loss_text_ce: 0.03476031497120857 (0.035718286779667686) | > loss_mel_ce: 3.12398099899292 (nan) | > loss: 0.012534688226878643 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.13701919063192908) | > loader_time: 0.0046 (0.004762557407173681)  --> TIME: 2025-02-25 04:36:32 -- STEP: 904/36938 -- GLOBAL_STEP: 628850 | > loss_text_ce: 0.03968453407287598 (0.035677892661785296) | > loss_mel_ce: 2.6886837482452393 (nan) | > loss: 0.010826858691871166 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13679906066540076) | > loader_time: 0.0046 (0.004757184634166487)  --> TIME: 2025-02-25 04:36:41 -- STEP: 954/36938 -- GLOBAL_STEP: 628900 | > loss_text_ce: 0.04314795881509781 (0.0357421513937757) | > loss_mel_ce: 2.7577621936798096 (nan) | > loss: 0.011114723980426788 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13681773569598898) | > loader_time: 0.0046 (0.004754923674545454)  --> TIME: 2025-02-25 04:36:51 -- STEP: 1004/36938 -- GLOBAL_STEP: 628950 | > loss_text_ce: 0.034045930951833725 (0.03569858003151693) | > loss_mel_ce: 3.198296546936035 (nan) | > loss: 0.012826756574213505 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13683613861699515) | > loader_time: 0.0046 (0.004749473585075596)  --> TIME: 2025-02-25 04:37:00 -- STEP: 1054/36938 -- GLOBAL_STEP: 629000 | > loss_text_ce: 0.02667493000626564 (0.0357135839758536) | > loss_mel_ce: 2.816866159439087 (nan) | > loss: 0.011283894069492817 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.1369819480508736) | > loader_time: 0.0046 (0.004746092564693217)  --> TIME: 2025-02-25 04:37:09 -- STEP: 1104/36938 -- GLOBAL_STEP: 629050 | > loss_text_ce: 0.0324077270925045 (0.035705446711896585) | > loss_mel_ce: 2.7852284908294678 (nan) | > loss: 0.01118109654635191 (nan) | > current_lr: 5e-06 | > step_time: 0.157 (0.13685383680074115) | > loader_time: 0.0048 (0.004742080966631578)  --> TIME: 2025-02-25 04:37:19 -- STEP: 1154/36938 -- GLOBAL_STEP: 629100 | > loss_text_ce: 0.047191571444272995 (0.03571423099003408) | > loss_mel_ce: 0.009368896484375 (nan) | > loss: 0.00022444631031248719 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.136911451506656) | > loader_time: 0.0046 (0.004739939649968671)  --> TIME: 2025-02-25 04:37:28 -- STEP: 1204/36938 -- GLOBAL_STEP: 629150 | > loss_text_ce: 0.038386810570955276 (0.03569359815079234) | > loss_mel_ce: 2.7694573402404785 (nan) | > loss: 0.011142238974571228 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13691232806424378) | > loader_time: 0.0052 (0.004738160542079384)  --> TIME: 2025-02-25 04:37:37 -- STEP: 1254/36938 -- GLOBAL_STEP: 629200 | > loss_text_ce: 0.043435610830783844 (0.03571201768737919) | > loss_mel_ce: 2.283694267272949 (nan) | > loss: 0.00923464260995388 (nan) | > current_lr: 5e-06 | > step_time: 0.1079 (0.13679644593781853) | > loader_time: 0.0044 (0.004734933661501946)  --> TIME: 2025-02-25 04:37:47 -- STEP: 1304/36938 -- GLOBAL_STEP: 629250 | > loss_text_ce: 0.03329446539282799 (0.035733014911533134) | > loss_mel_ce: 2.7476563453674316 (nan) | > loss: 0.011035519652068615 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13676578281847274) | > loader_time: 0.0047 (0.004734466412316074)  --> TIME: 2025-02-25 04:37:56 -- STEP: 1354/36938 -- GLOBAL_STEP: 629300 | > loss_text_ce: 0.0526009276509285 (0.0357708264014338) | > loss_mel_ce: 1.371851921081543 (nan) | > loss: 0.005652591120451689 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13680316919414143) | > loader_time: 0.0046 (0.004739242607240668)  --> TIME: 2025-02-25 04:38:05 -- STEP: 1404/36938 -- GLOBAL_STEP: 629350 | > loss_text_ce: 0.030993368476629257 (0.03573255499153583) | > loss_mel_ce: 2.8519275188446045 (nan) | > loss: 0.011440163478255272 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13665247048407905) | > loader_time: 0.0047 (0.004737066577302771)  --> TIME: 2025-02-25 04:38:15 -- STEP: 1454/36938 -- GLOBAL_STEP: 629400 | > loss_text_ce: 0.03398563340306282 (0.035679894445036116) | > loss_mel_ce: 2.34464168548584 (nan) | > loss: 0.009438998065888882 (nan) | > current_lr: 5e-06 | > step_time: 0.1571 (0.13671152414613139) | > loader_time: 0.005 (0.004736784235811307)  --> TIME: 2025-02-25 04:38:24 -- STEP: 1504/36938 -- GLOBAL_STEP: 629450 | > loss_text_ce: 0.03214915469288826 (0.035688823435153325) | > loss_mel_ce: 2.4297075271606445 (nan) | > loss: 0.009769272990524769 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.13666196421105833) | > loader_time: 0.0049 (0.004735753415746897)  --> TIME: 2025-02-25 04:38:34 -- STEP: 1554/36938 -- GLOBAL_STEP: 629500 | > loss_text_ce: 0.037573251873254776 (0.03572080064831025) | > loss_mel_ce: 2.346832275390625 (nan) | > loss: 0.009461927227675915 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.1368417856469272) | > loader_time: 0.0046 (0.004735592556122503)  --> TIME: 2025-02-25 04:38:43 -- STEP: 1604/36938 -- GLOBAL_STEP: 629550 | > loss_text_ce: 0.03753110393881798 (0.03563695441306723) | > loss_mel_ce: 2.952775478363037 (nan) | > loss: 0.011866296641528606 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.136946713835224) | > loader_time: 0.0047 (0.004735569555563235)  --> TIME: 2025-02-25 04:38:52 -- STEP: 1654/36938 -- GLOBAL_STEP: 629600 | > loss_text_ce: 0.02554526925086975 (0.03563067241388274) | > loss_mel_ce: 3.0300869941711426 (nan) | > loss: 0.012125525623559952 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13691574486452276) | > loader_time: 0.0045 (0.004736384716979548)  --> TIME: 2025-02-25 04:39:02 -- STEP: 1704/36938 -- GLOBAL_STEP: 629650 | > loss_text_ce: 0.033308837562799454 (0.035632163521772425) | > loss_mel_ce: 3.4495656490325928 (nan) | > loss: 0.013820931315422058 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13682782957811324) | > loader_time: 0.0046 (0.004736251534430644)  --> TIME: 2025-02-25 04:39:11 -- STEP: 1754/36938 -- GLOBAL_STEP: 629700 | > loss_text_ce: 0.035124894231557846 (0.03566330774584216) | > loss_mel_ce: 1.6132625341415405 (nan) | > loss: 0.006541220471262932 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1368093379139221) | > loader_time: 0.0045 (0.004734291318069172)  --> TIME: 2025-02-25 04:39:20 -- STEP: 1804/36938 -- GLOBAL_STEP: 629750 | > loss_text_ce: 0.04369356483221054 (0.035666262737667334) | > loss_mel_ce: 0.5700018405914307 (nan) | > loss: 0.0024352993350476027 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.1368281633785189) | > loader_time: 0.0047 (0.0047324223158893555)  --> TIME: 2025-02-25 04:39:30 -- STEP: 1854/36938 -- GLOBAL_STEP: 629800 | > loss_text_ce: 0.029046332463622093 (0.03564787180310647) | > loss_mel_ce: 2.7466001510620117 (nan) | > loss: 0.01101447083055973 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.13678594417427983) | > loader_time: 0.005 (0.004730578765128432)  --> TIME: 2025-02-25 04:39:39 -- STEP: 1904/36938 -- GLOBAL_STEP: 629850 | > loss_text_ce: 0.03959415480494499 (0.0356447070992567) | > loss_mel_ce: 2.3407986164093018 (nan) | > loss: 0.00944600347429514 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13679858153106778) | > loader_time: 0.0046 (0.004729811133456841)  --> TIME: 2025-02-25 04:39:48 -- STEP: 1954/36938 -- GLOBAL_STEP: 629900 | > loss_text_ce: 0.04008375480771065 (0.035653052656618625) | > loss_mel_ce: 0.0708325058221817 (nan) | > loss: 0.0004401438927743584 (nan) | > current_lr: 5e-06 | > step_time: 0.1537 (0.13662189251324772) | > loader_time: 0.0046 (0.004727823585424364)  --> TIME: 2025-02-25 04:39:58 -- STEP: 2004/36938 -- GLOBAL_STEP: 629950 | > loss_text_ce: 0.038474928587675095 (0.03565236013206071) | > loss_mel_ce: 3.306129217147827 (nan) | > loss: 0.013272239826619625 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13675598077431403) | > loader_time: 0.0045 (0.004726247278278231)  --> TIME: 2025-02-25 04:40:07 -- STEP: 2054/36938 -- GLOBAL_STEP: 630000 | > loss_text_ce: 0.03352530673146248 (0.035684375996051176) | > loss_mel_ce: 3.275038242340088 (nan) | > loss: 0.0131292212754488 (nan) | > current_lr: 5e-06 | > step_time: 0.2075 (0.13677554690036933) | > loader_time: 0.0049 (0.004724040932177113) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_630000.pth  --> TIME: 2025-02-25 04:40:22 -- STEP: 2104/36938 -- GLOBAL_STEP: 630050 | > loss_text_ce: 0.022845907136797905 (0.03570140771498766) | > loss_mel_ce: 3.1591806411743164 (nan) | > loss: 0.012627090327441692 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13675534827174332) | > loader_time: 0.0046 (0.004723620392070533)  --> TIME: 2025-02-25 04:40:32 -- STEP: 2154/36938 -- GLOBAL_STEP: 630100 | > loss_text_ce: 0.03662458807229996 (0.03568257533313015) | > loss_mel_ce: 2.8292219638824463 (nan) | > loss: 0.011372407898306847 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.1366949364778169) | > loader_time: 0.0045 (0.004721120539481935)  --> TIME: 2025-02-25 04:40:41 -- STEP: 2204/36938 -- GLOBAL_STEP: 630150 | > loss_text_ce: 0.04674425348639488 (0.03570041833944046) | > loss_mel_ce: 2.896082878112793 (nan) | > loss: 0.011677886359393597 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13673023952546431) | > loader_time: 0.0046 (0.00471853387766872)  --> TIME: 2025-02-25 04:40:50 -- STEP: 2254/36938 -- GLOBAL_STEP: 630200 | > loss_text_ce: 0.04455224424600601 (0.03569478141028222) | > loss_mel_ce: 2.5598549842834473 (nan) | > loss: 0.010334949940443039 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.1367574096677994) | > loader_time: 0.0045 (0.004716663309823345)  --> TIME: 2025-02-25 04:41:00 -- STEP: 2304/36938 -- GLOBAL_STEP: 630250 | > loss_text_ce: 0.036205943673849106 (0.03571552694443701) | > loss_mel_ce: 2.8773958683013916 (nan) | > loss: 0.01156191248446703 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13671883019722156) | > loader_time: 0.0045 (0.004715259186923514)  --> TIME: 2025-02-25 04:41:09 -- STEP: 2354/36938 -- GLOBAL_STEP: 630300 | > loss_text_ce: 0.028095876798033714 (0.03573238599084841) | > loss_mel_ce: 0.247314453125 (nan) | > loss: 0.0010928981937468052 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.1366780365333782) | > loader_time: 0.0044 (0.004713165628373278)  --> TIME: 2025-02-25 04:41:18 -- STEP: 2404/36938 -- GLOBAL_STEP: 630350 | > loss_text_ce: 0.03486216068267822 (0.03572841664774652) | > loss_mel_ce: 1.3790664672851562 (nan) | > loss: 0.005610828287899494 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13657523818103307) | > loader_time: 0.0049 (0.004711172445839942)  --> TIME: 2025-02-25 04:41:27 -- STEP: 2454/36938 -- GLOBAL_STEP: 630400 | > loss_text_ce: 0.03461911529302597 (0.035726413075730906) | > loss_mel_ce: 2.978741407394409 (nan) | > loss: 0.011957780458033085 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.13666118454330786) | > loader_time: 0.0046 (0.0047102776016757775)  --> TIME: 2025-02-25 04:41:36 -- STEP: 2504/36938 -- GLOBAL_STEP: 630450 | > loss_text_ce: 0.03979106247425079 (0.035707452176391645) | > loss_mel_ce: 3.0706264972686768 (nan) | > loss: 0.012342927977442741 (nan) | > current_lr: 5e-06 | > step_time: 0.1459 (0.13660554059397267) | > loader_time: 0.0047 (0.004709210353918358)  --> TIME: 2025-02-25 04:41:46 -- STEP: 2554/36938 -- GLOBAL_STEP: 630500 | > loss_text_ce: 0.034515250474214554 (0.035686746437668476) | > loss_mel_ce: 2.7330191135406494 (nan) | > loss: 0.010982279665768147 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13653043442250384) | > loader_time: 0.0045 (0.00470679638996289)  --> TIME: 2025-02-25 04:41:54 -- STEP: 2604/36938 -- GLOBAL_STEP: 630550 | > loss_text_ce: 0.028122417628765106 (0.03568310356418526) | > loss_mel_ce: 3.3830714225769043 (nan) | > loss: 0.013536483980715275 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1363946590372308) | > loader_time: 0.0046 (0.004704660534309356)  --> TIME: 2025-02-25 04:42:03 -- STEP: 2654/36938 -- GLOBAL_STEP: 630600 | > loss_text_ce: 0.029242051765322685 (0.03568187985146736) | > loss_mel_ce: 1.7330485582351685 (nan) | > loss: 0.0069932169280946255 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13625634974254708) | > loader_time: 0.0046 (0.004703371167272764)  --> TIME: 2025-02-25 04:42:13 -- STEP: 2704/36938 -- GLOBAL_STEP: 630650 | > loss_text_ce: 0.03679484874010086 (0.035666873043791486) | > loss_mel_ce: 2.819828987121582 (nan) | > loss: 0.011335809715092182 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13634456429608446) | > loader_time: 0.0046 (0.0047030710786052105)  --> TIME: 2025-02-25 04:42:22 -- STEP: 2754/36938 -- GLOBAL_STEP: 630700 | > loss_text_ce: 0.03023655153810978 (0.03565228155008584) | > loss_mel_ce: 2.596602439880371 (nan) | > loss: 0.010423964820802212 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13641103273210234) | > loader_time: 0.0046 (0.0047024135236387)  --> TIME: 2025-02-25 04:42:32 -- STEP: 2804/36938 -- GLOBAL_STEP: 630750 | > loss_text_ce: 0.04511341452598572 (0.03565368902724708) | > loss_mel_ce: 2.4608571529388428 (nan) | > loss: 0.009944328106939793 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.136433017917094) | > loader_time: 0.0046 (0.0047029556119323095)  --> TIME: 2025-02-25 04:42:41 -- STEP: 2854/36938 -- GLOBAL_STEP: 630800 | > loss_text_ce: 0.04706137999892235 (0.035657884072076415) | > loss_mel_ce: 2.825230598449707 (nan) | > loss: 0.011397984810173512 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.1365184866638555) | > loader_time: 0.0047 (0.004702318692090394)  --> TIME: 2025-02-25 04:42:51 -- STEP: 2904/36938 -- GLOBAL_STEP: 630850 | > loss_text_ce: 0.04444243013858795 (0.03566751143569254) | > loss_mel_ce: 3.02158784866333 (nan) | > loss: 0.01216678787022829 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13651490523467008) | > loader_time: 0.0047 (0.004701955916139386)  --> TIME: 2025-02-25 04:43:00 -- STEP: 2954/36938 -- GLOBAL_STEP: 630900 | > loss_text_ce: 0.03343954309821129 (0.035644222579306224) | > loss_mel_ce: 2.594438314437866 (nan) | > loss: 0.010428087785840034 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.13654704972062406) | > loader_time: 0.0046 (0.004701225355660998)  --> TIME: 2025-02-25 04:43:09 -- STEP: 3004/36938 -- GLOBAL_STEP: 630950 | > loss_text_ce: 0.03415258601307869 (0.03563492169874048) | > loss_mel_ce: 2.9619967937469482 (nan) | > loss: 0.011889481917023659 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13650193639823768) | > loader_time: 0.0053 (0.004700378397015862)  --> TIME: 2025-02-25 04:43:19 -- STEP: 3054/36938 -- GLOBAL_STEP: 631000 | > loss_text_ce: 0.027768554165959358 (0.0356293087837377) | > loss_mel_ce: 3.577373743057251 (nan) | > loss: 0.01430612150579691 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.1364815887002746) | > loader_time: 0.0057 (0.004699562137706195)  --> TIME: 2025-02-25 04:43:28 -- STEP: 3104/36938 -- GLOBAL_STEP: 631050 | > loss_text_ce: 0.05131617933511734 (0.03561357561422067) | > loss_mel_ce: 3.0158908367156982 (nan) | > loss: 0.012171457521617413 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13652350853398845) | > loader_time: 0.0046 (0.004698887774624781)  --> TIME: 2025-02-25 04:43:37 -- STEP: 3154/36938 -- GLOBAL_STEP: 631100 | > loss_text_ce: 0.03571191057562828 (0.03560257136890248) | > loss_mel_ce: 2.8546206951141357 (nan) | > loss: 0.011469574645161629 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13648171022713837) | > loader_time: 0.0045 (0.0046984524989929465)  --> TIME: 2025-02-25 04:43:47 -- STEP: 3204/36938 -- GLOBAL_STEP: 631150 | > loss_text_ce: 0.024797597900032997 (0.03557716492103056) | > loss_mel_ce: 2.034231662750244 (nan) | > loss: 0.008170751854777336 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13649271028616236) | > loader_time: 0.0046 (0.004697845744134189)  --> TIME: 2025-02-25 04:43:56 -- STEP: 3254/36938 -- GLOBAL_STEP: 631200 | > loss_text_ce: 0.027390209957957268 (0.03556867438519043) | > loss_mel_ce: 0.26470947265625 (nan) | > loss: 0.001159125822596252 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13649855307735476) | > loader_time: 0.0046 (0.004697941239069192)  --> TIME: 2025-02-25 04:44:06 -- STEP: 3304/36938 -- GLOBAL_STEP: 631250 | > loss_text_ce: 0.03749062493443489 (0.03557178738569842) | > loss_mel_ce: 3.2329604625701904 (nan) | > loss: 0.012977981008589268 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13654279961424287) | > loader_time: 0.0046 (0.004697925314198972)  --> TIME: 2025-02-25 04:44:15 -- STEP: 3354/36938 -- GLOBAL_STEP: 631300 | > loss_text_ce: 0.022916028276085854 (0.03556556778653935) | > loss_mel_ce: 1.2576172351837158 (nan) | > loss: 0.005081481765955687 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13658326476830074) | > loader_time: 0.0046 (0.004697684738418496)  --> TIME: 2025-02-25 04:44:24 -- STEP: 3404/36938 -- GLOBAL_STEP: 631350 | > loss_text_ce: 0.03768205642700195 (0.03557047600497863) | > loss_mel_ce: 3.4866321086883545 (nan) | > loss: 0.013985374942421913 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.13656403931551572) | > loader_time: 0.0046 (0.00469619785156429)  --> TIME: 2025-02-25 04:44:34 -- STEP: 3454/36938 -- GLOBAL_STEP: 631400 | > loss_text_ce: 0.03277198597788811 (0.03558318129377728) | > loss_mel_ce: 3.1279616355895996 (nan) | > loss: 0.012542595155537128 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.13656714201387618) | > loader_time: 0.0048 (0.004695543748626971)  --> TIME: 2025-02-25 04:44:43 -- STEP: 3504/36938 -- GLOBAL_STEP: 631450 | > loss_text_ce: 0.039909105747938156 (0.03560712446456514) | > loss_mel_ce: 1.9717448949813843 (nan) | > loss: 0.007982754148542881 (nan) | > current_lr: 5e-06 | > step_time: 0.1069 (0.13652778086869088) | > loader_time: 0.0044 (0.004694816932830633)  --> TIME: 2025-02-25 04:44:52 -- STEP: 3554/36938 -- GLOBAL_STEP: 631500 | > loss_text_ce: 0.04137524589896202 (0.035608330736807824) | > loss_mel_ce: 2.3094608783721924 (nan) | > loss: 0.009328714571893215 (nan) | > current_lr: 5e-06 | > step_time: 0.1452 (0.1364512736031222) | > loader_time: 0.0067 (0.004694640804155526)  --> TIME: 2025-02-25 04:45:01 -- STEP: 3604/36938 -- GLOBAL_STEP: 631550 | > loss_text_ce: 0.032954078167676926 (0.03561092446326641) | > loss_mel_ce: 3.535658359527588 (nan) | > loss: 0.01416116114705801 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13640547744706488) | > loader_time: 0.0046 (0.0046938701422180075)  --> TIME: 2025-02-25 04:45:10 -- STEP: 3654/36938 -- GLOBAL_STEP: 631600 | > loss_text_ce: 0.030623052269220352 (0.03561169572901461) | > loss_mel_ce: 3.17694091796875 (nan) | > loss: 0.012728428468108177 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13633145017576884) | > loader_time: 0.0046 (0.004692932850817347)  --> TIME: 2025-02-25 04:45:19 -- STEP: 3704/36938 -- GLOBAL_STEP: 631650 | > loss_text_ce: 0.025787893682718277 (0.03559477292665043) | > loss_mel_ce: 0.8631853461265564 (nan) | > loss: 0.0035276717972010374 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.13634992315238773) | > loader_time: 0.0046 (0.004692023953917729)  --> TIME: 2025-02-25 04:45:29 -- STEP: 3754/36938 -- GLOBAL_STEP: 631700 | > loss_text_ce: 0.031628191471099854 (0.035607348825855985) | > loss_mel_ce: 3.2977864742279053 (nan) | > loss: 0.013211963698267937 (nan) | > current_lr: 5e-06 | > step_time: 0.1809 (0.13634588436427453) | > loader_time: 0.0049 (0.004691169181909086)  --> TIME: 2025-02-25 04:45:38 -- STEP: 3804/36938 -- GLOBAL_STEP: 631750 | > loss_text_ce: 0.02505401521921158 (0.03562155980068397) | > loss_mel_ce: 2.086862564086914 (nan) | > loss: 0.008380621671676636 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13635039887343003) | > loader_time: 0.0045 (0.004690221180800506)  --> TIME: 2025-02-25 04:45:48 -- STEP: 3854/36938 -- GLOBAL_STEP: 631800 | > loss_text_ce: 0.03872368857264519 (0.03567094768851332) | > loss_mel_ce: 2.9825024604797363 (nan) | > loss: 0.011988993734121323 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.13641865190348562) | > loader_time: 0.0046 (0.004689569354490348)  --> TIME: 2025-02-25 04:45:57 -- STEP: 3904/36938 -- GLOBAL_STEP: 631850 | > loss_text_ce: 0.03445034474134445 (0.03566583323571069) | > loss_mel_ce: 3.371971368789673 (nan) | > loss: 0.013517547398805618 (nan) | > current_lr: 5e-06 | > step_time: 0.1827 (0.1364490974266994) | > loader_time: 0.0049 (0.004688580993746145)  --> TIME: 2025-02-25 04:46:06 -- STEP: 3954/36938 -- GLOBAL_STEP: 631900 | > loss_text_ce: 0.04250764101743698 (0.035670137920849825) | > loss_mel_ce: 3.6037580966949463 (nan) | > loss: 0.014469309709966183 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.13644495351663027) | > loader_time: 0.0048 (0.004688423814710847)  --> TIME: 2025-02-25 04:46:16 -- STEP: 4004/36938 -- GLOBAL_STEP: 631950 | > loss_text_ce: 0.02712531015276909 (0.035662395181366956) | > loss_mel_ce: 2.8292672634124756 (nan) | > loss: 0.011334892362356186 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.1364390783972074) | > loader_time: 0.0044 (0.0046880736455812535)  --> TIME: 2025-02-25 04:46:25 -- STEP: 4054/36938 -- GLOBAL_STEP: 632000 | > loss_text_ce: 0.03499828651547432 (0.0356811404767615) | > loss_mel_ce: 1.711822509765625 (nan) | > loss: 0.006931828800588846 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13648634246257785) | > loader_time: 0.0045 (0.004687490696147207)  --> TIME: 2025-02-25 04:46:34 -- STEP: 4104/36938 -- GLOBAL_STEP: 632050 | > loss_text_ce: 0.03956882655620575 (0.03568999689584327) | > loss_mel_ce: 2.728703498840332 (nan) | > loss: 0.010985208675265312 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13646994290063683) | > loader_time: 0.0045 (0.004686408688915168)  --> TIME: 2025-02-25 04:46:44 -- STEP: 4154/36938 -- GLOBAL_STEP: 632100 | > loss_text_ce: 0.0344758965075016 (0.035693492625154415) | > loss_mel_ce: 0.007354736328125 (nan) | > loss: 0.0001659945846768096 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.1365307998817929) | > loader_time: 0.0046 (0.004685726714467105)  --> TIME: 2025-02-25 04:46:53 -- STEP: 4204/36938 -- GLOBAL_STEP: 632150 | > loss_text_ce: 0.03089541383087635 (0.035708605961787286) | > loss_mel_ce: 3.02827787399292 (nan) | > loss: 0.01213957741856575 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13654044171268853) | > loader_time: 0.0046 (0.004684973398239468)  --> TIME: 2025-02-25 04:47:03 -- STEP: 4254/36938 -- GLOBAL_STEP: 632200 | > loss_text_ce: 0.02498091384768486 (0.03570231179683546) | > loss_mel_ce: 1.3887939453125 (nan) | > loss: 0.005610217805951834 (nan) | > current_lr: 5e-06 | > step_time: 0.1083 (0.1365982286796692) | > loader_time: 0.0045 (0.004684815229694775)  --> TIME: 2025-02-25 04:47:12 -- STEP: 4304/36938 -- GLOBAL_STEP: 632250 | > loss_text_ce: 0.030563434585928917 (0.035680846559995136) | > loss_mel_ce: 2.9931640625 (nan) | > loss: 0.011998918838799 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13659783710334353) | > loader_time: 0.0044 (0.004683912409725683)  --> TIME: 2025-02-25 04:47:22 -- STEP: 4354/36938 -- GLOBAL_STEP: 632300 | > loss_text_ce: 0.04260506108403206 (0.03568200319988303) | > loss_mel_ce: 0.26434326171875 (nan) | > loss: 0.0012180489720776677 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13660773860789432) | > loader_time: 0.0059 (0.004683516854587145)  --> TIME: 2025-02-25 04:47:31 -- STEP: 4404/36938 -- GLOBAL_STEP: 632350 | > loss_text_ce: 0.03677210211753845 (0.03568155772825) | > loss_mel_ce: 3.084132432937622 (nan) | > loss: 0.012384542264044285 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13661117666315511) | > loader_time: 0.0046 (0.004682770975062676)  --> TIME: 2025-02-25 04:47:40 -- STEP: 4454/36938 -- GLOBAL_STEP: 632400 | > loss_text_ce: 0.038818392902612686 (0.03567805190934063) | > loss_mel_ce: 2.993105888366699 (nan) | > loss: 0.01203144621104002 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.13659569582277506) | > loader_time: 0.0047 (0.0046822722310381855)  --> TIME: 2025-02-25 04:47:49 -- STEP: 4504/36938 -- GLOBAL_STEP: 632450 | > loss_text_ce: 0.04056642949581146 (0.03567979540378119) | > loss_mel_ce: 3.328911066055298 (nan) | > loss: 0.013370943255722523 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13656002378379029) | > loader_time: 0.0045 (0.004681253941402132)  --> TIME: 2025-02-25 04:47:59 -- STEP: 4554/36938 -- GLOBAL_STEP: 632500 | > loss_text_ce: 0.034740932285785675 (0.03568095292038739) | > loss_mel_ce: 3.594254970550537 (nan) | > loss: 0.014400778338313103 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.1365710769665046) | > loader_time: 0.0046 (0.004680636424373315)  --> TIME: 2025-02-25 04:48:08 -- STEP: 4604/36938 -- GLOBAL_STEP: 632550 | > loss_text_ce: 0.03929973393678665 (0.03565752273425459) | > loss_mel_ce: 0.4506492614746094 (nan) | > loss: 0.0019442421616986394 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13657370352102913) | > loader_time: 0.0047 (0.004679876343257317)  --> TIME: 2025-02-25 04:48:18 -- STEP: 4654/36938 -- GLOBAL_STEP: 632600 | > loss_text_ce: 0.033801015466451645 (0.035656063733394285) | > loss_mel_ce: 3.2230544090270996 (nan) | > loss: 0.012924030423164368 (nan) | > current_lr: 5e-06 | > step_time: 0.1885 (0.13661573181144135) | > loader_time: 0.0048 (0.004679903740221002)  --> TIME: 2025-02-25 04:48:27 -- STEP: 4704/36938 -- GLOBAL_STEP: 632650 | > loss_text_ce: 0.024598659947514534 (0.03565137178500873) | > loss_mel_ce: 3.2733395099639893 (nan) | > loss: 0.013087056577205658 (nan) | > current_lr: 5e-06 | > step_time: 0.1662 (0.13662007364894238) | > loader_time: 0.0047 (0.004679870341910801)  --> TIME: 2025-02-25 04:48:37 -- STEP: 4754/36938 -- GLOBAL_STEP: 632700 | > loss_text_ce: 0.03728296235203743 (0.03565470270928489) | > loss_mel_ce: 3.37947416305542 (nan) | > loss: 0.013558560982346535 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.13665648328189708) | > loader_time: 0.0046 (0.004679392755257269)  --> TIME: 2025-02-25 04:48:46 -- STEP: 4804/36938 -- GLOBAL_STEP: 632750 | > loss_text_ce: 0.03471069410443306 (0.03565576112442201) | > loss_mel_ce: 1.7251403331756592 (nan) | > loss: 0.006983535829931498 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.13666904846099268) | > loader_time: 0.0047 (0.004678777761006734)  --> TIME: 2025-02-25 04:48:55 -- STEP: 4854/36938 -- GLOBAL_STEP: 632800 | > loss_text_ce: 0.027658244594931602 (0.03564301460308305) | > loss_mel_ce: 3.254021167755127 (nan) | > loss: 0.013022538274526596 (nan) | > current_lr: 5e-06 | > step_time: 0.1782 (0.13667117724165162) | > loader_time: 0.0048 (0.0046792288203764)  --> TIME: 2025-02-25 04:49:05 -- STEP: 4904/36938 -- GLOBAL_STEP: 632850 | > loss_text_ce: 0.03792154788970947 (0.035650655988650984) | > loss_mel_ce: 0.8621930480003357 (nan) | > loss: 0.0035718835424631834 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13664461820588386) | > loader_time: 0.0046 (0.004678430847009282)  --> TIME: 2025-02-25 04:49:14 -- STEP: 4954/36938 -- GLOBAL_STEP: 632900 | > loss_text_ce: 0.03529505804181099 (0.03566224985993988) | > loss_mel_ce: 2.914193868637085 (nan) | > loss: 0.011704321950674057 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13667369841759247) | > loader_time: 0.0045 (0.00467789389695378)  --> TIME: 2025-02-25 04:49:23 -- STEP: 5004/36938 -- GLOBAL_STEP: 632950 | > loss_text_ce: 0.029245667159557343 (0.03565957696180051) | > loss_mel_ce: 3.144449234008789 (nan) | > loss: 0.012594027444720268 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.13666683397323565) | > loader_time: 0.0046 (0.004677187624595149)  --> TIME: 2025-02-25 04:49:33 -- STEP: 5054/36938 -- GLOBAL_STEP: 633000 | > loss_text_ce: 0.0354025736451149 (0.03565816365003292) | > loss_mel_ce: 1.752197265625 (nan) | > loss: 0.007093650288879871 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13667597933913755) | > loader_time: 0.0045 (0.004676762899014616)  --> TIME: 2025-02-25 04:49:42 -- STEP: 5104/36938 -- GLOBAL_STEP: 633050 | > loss_text_ce: 0.038465652614831924 (0.03567002874161898) | > loss_mel_ce: 1.8585984706878662 (nan) | > loss: 0.0075280326418578625 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1366420309278284) | > loader_time: 0.0045 (0.00467602656365936)  --> TIME: 2025-02-25 04:49:51 -- STEP: 5154/36938 -- GLOBAL_STEP: 633100 | > loss_text_ce: 0.03732650727033615 (0.03567132031831278) | > loss_mel_ce: 2.566721200942993 (nan) | > loss: 0.010333523154258728 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13664537115433806) | > loader_time: 0.0046 (0.004675671163343584)  --> TIME: 2025-02-25 04:50:01 -- STEP: 5204/36938 -- GLOBAL_STEP: 633150 | > loss_text_ce: 0.03485948592424393 (0.03567601512093175) | > loss_mel_ce: 2.9893574714660645 (nan) | > loss: 0.012000861577689648 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.13667701286686829) | > loader_time: 0.0048 (0.004675204940798837)  --> TIME: 2025-02-25 04:50:10 -- STEP: 5254/36938 -- GLOBAL_STEP: 633200 | > loss_text_ce: 0.043499018996953964 (0.035671749829552575) | > loss_mel_ce: 2.715211868286133 (nan) | > loss: 0.010947265662252903 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13671177446411087) | > loader_time: 0.0046 (0.004675253017528644)  --> TIME: 2025-02-25 04:50:20 -- STEP: 5304/36938 -- GLOBAL_STEP: 633250 | > loss_text_ce: 0.036611512303352356 (0.03567579857703741) | > loss_mel_ce: 2.613260269165039 (nan) | > loss: 0.010515364818274975 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13671220917687327) | > loader_time: 0.0046 (0.00467502931484091)  --> TIME: 2025-02-25 04:50:29 -- STEP: 5354/36938 -- GLOBAL_STEP: 633300 | > loss_text_ce: 0.028332559391856194 (0.03567560838791488) | > loss_mel_ce: 0.8356667757034302 (nan) | > loss: 0.0034285688307136297 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13667750331862968) | > loader_time: 0.0045 (0.004674465521797132)  --> TIME: 2025-02-25 04:50:38 -- STEP: 5404/36938 -- GLOBAL_STEP: 633350 | > loss_text_ce: 0.051989611238241196 (0.035676211545238416) | > loss_mel_ce: 3.2218966484069824 (nan) | > loss: 0.012991612777113914 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13666671646867815) | > loader_time: 0.0046 (0.00467398786968342)  --> TIME: 2025-02-25 04:50:48 -- STEP: 5454/36938 -- GLOBAL_STEP: 633400 | > loss_text_ce: 0.04456477612257004 (0.03566951088441337) | > loss_mel_ce: 1.9675334692001343 (nan) | > loss: 0.007984517142176628 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13667657102557543) | > loader_time: 0.0045 (0.004673944185800422)  --> TIME: 2025-02-25 04:50:57 -- STEP: 5504/36938 -- GLOBAL_STEP: 633450 | > loss_text_ce: 0.03211808577179909 (0.03566906381033632) | > loss_mel_ce: 3.3616862297058105 (nan) | > loss: 0.013467478565871716 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13674995935586975) | > loader_time: 0.0046 (0.004673869890529059)  --> TIME: 2025-02-25 04:51:07 -- STEP: 5554/36938 -- GLOBAL_STEP: 633500 | > loss_text_ce: 0.02889331988990307 (0.035672142154439576) | > loss_mel_ce: 0.10579223930835724 (nan) | > loss: 0.0005344665260054171 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13676111510128752) | > loader_time: 0.0045 (0.004673625008854715)  --> TIME: 2025-02-25 04:51:17 -- STEP: 5604/36938 -- GLOBAL_STEP: 633550 | > loss_text_ce: 0.02560504525899887 (0.03566424459698269) | > loss_mel_ce: 2.711052894592285 (nan) | > loss: 0.01085975393652916 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.1368172829445557) | > loader_time: 0.005 (0.004675602121577786)  --> TIME: 2025-02-25 04:51:26 -- STEP: 5654/36938 -- GLOBAL_STEP: 633600 | > loss_text_ce: 0.03294975310564041 (0.03565969864245169) | > loss_mel_ce: 3.233441114425659 (nan) | > loss: 0.01296186912804842 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13685163011135928) | > loader_time: 0.0046 (0.004675365185880986)  --> TIME: 2025-02-25 04:51:36 -- STEP: 5704/36938 -- GLOBAL_STEP: 633650 | > loss_text_ce: 0.04219089448451996 (0.03566962448036929) | > loss_mel_ce: 2.6925904750823975 (nan) | > loss: 0.010852307081222534 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13686575574426763) | > loader_time: 0.0045 (0.00467495555463045)  --> TIME: 2025-02-25 04:51:45 -- STEP: 5754/36938 -- GLOBAL_STEP: 633700 | > loss_text_ce: 0.030972076579928398 (0.03566882680816057) | > loss_mel_ce: 3.2583749294281006 (nan) | > loss: 0.013052964583039284 (nan) | > current_lr: 5e-06 | > step_time: 0.157 (0.1368787412971596) | > loader_time: 0.0046 (0.004674570362394707)  --> TIME: 2025-02-25 04:51:54 -- STEP: 5804/36938 -- GLOBAL_STEP: 633750 | > loss_text_ce: 0.03419150412082672 (0.03567257650542521) | > loss_mel_ce: 3.4114246368408203 (nan) | > loss: 0.013673081062734127 (nan) | > current_lr: 5e-06 | > step_time: 0.1737 (0.13684884414107576) | > loader_time: 0.0048 (0.0046740210853224405)  --> TIME: 2025-02-25 04:52:03 -- STEP: 5854/36938 -- GLOBAL_STEP: 633800 | > loss_text_ce: 0.03528328239917755 (0.035674339590029844) | > loss_mel_ce: 2.804405450820923 (nan) | > loss: 0.011268607340753078 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13685109316386296) | > loader_time: 0.0044 (0.004673776954035733)  --> TIME: 2025-02-25 04:52:13 -- STEP: 5904/36938 -- GLOBAL_STEP: 633850 | > loss_text_ce: 0.035096731036901474 (0.03566845341602624) | > loss_mel_ce: 2.2553117275238037 (nan) | > loss: 0.009088922291994095 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13683861623288465) | > loader_time: 0.0044 (0.004673370177829813)  --> TIME: 2025-02-25 04:52:22 -- STEP: 5954/36938 -- GLOBAL_STEP: 633900 | > loss_text_ce: 0.02940785512328148 (0.03566324812722073) | > loss_mel_ce: 2.463939666748047 (nan) | > loss: 0.009894235990941525 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13679485650201872) | > loader_time: 0.0046 (0.0046728726878116555)  --> TIME: 2025-02-25 04:52:31 -- STEP: 6004/36938 -- GLOBAL_STEP: 633950 | > loss_text_ce: 0.034582190215587616 (0.035661273915910195) | > loss_mel_ce: 3.388908863067627 (nan) | > loss: 0.013585282489657402 (nan) | > current_lr: 5e-06 | > step_time: 0.1632 (0.13677401390971486) | > loader_time: 0.0047 (0.004672417237232876)  --> TIME: 2025-02-25 04:52:40 -- STEP: 6054/36938 -- GLOBAL_STEP: 634000 | > loss_text_ce: 0.03518904000520706 (0.035651372730403495) | > loss_mel_ce: 3.1979691982269287 (nan) | > loss: 0.012829993851482868 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.1367885556046146) | > loader_time: 0.0047 (0.004672540900019633)  --> TIME: 2025-02-25 04:52:50 -- STEP: 6104/36938 -- GLOBAL_STEP: 634050 | > loss_text_ce: 0.039088550955057144 (0.035653317356476165) | > loss_mel_ce: 3.382416248321533 (nan) | > loss: 0.013577400706708431 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13681654762940448) | > loader_time: 0.0045 (0.004672428649187403)  --> TIME: 2025-02-25 04:52:59 -- STEP: 6154/36938 -- GLOBAL_STEP: 634100 | > loss_text_ce: 0.030933108180761337 (0.03565506203109782) | > loss_mel_ce: 3.1359808444976807 (nan) | > loss: 0.012567119672894478 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13679409732204847) | > loader_time: 0.0044 (0.004671863080775697)  --> TIME: 2025-02-25 04:53:08 -- STEP: 6204/36938 -- GLOBAL_STEP: 634150 | > loss_text_ce: 0.036016371101140976 (0.035653747063818395) | > loss_mel_ce: 2.9979186058044434 (nan) | > loss: 0.012039425782859325 (nan) | > current_lr: 5e-06 | > step_time: 0.1702 (0.13678823925463396) | > loader_time: 0.0047 (0.004671384756369875)  --> TIME: 2025-02-25 04:53:18 -- STEP: 6254/36938 -- GLOBAL_STEP: 634200 | > loss_text_ce: 0.033892735838890076 (0.03565017745535575) | > loss_mel_ce: 3.2063167095184326 (nan) | > loss: 0.012857974506914616 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.1368012348226278) | > loader_time: 0.0047 (0.004671437236345501)  --> TIME: 2025-02-25 04:53:27 -- STEP: 6304/36938 -- GLOBAL_STEP: 634250 | > loss_text_ce: 0.0324774868786335 (0.035652452693787785) | > loss_mel_ce: 3.1716930866241455 (nan) | > loss: 0.012714962475001812 (nan) | > current_lr: 5e-06 | > step_time: 0.1579 (0.136804127262025) | > loader_time: 0.0046 (0.0046715654319313)  --> TIME: 2025-02-25 04:53:36 -- STEP: 6354/36938 -- GLOBAL_STEP: 634300 | > loss_text_ce: 0.02938198111951351 (0.03565888813112578) | > loss_mel_ce: 3.4884955883026123 (nan) | > loss: 0.013959832489490509 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.136805212681571) | > loader_time: 0.0046 (0.004671117289095585)  --> TIME: 2025-02-25 04:53:46 -- STEP: 6404/36938 -- GLOBAL_STEP: 634350 | > loss_text_ce: 0.033367156982421875 (0.03564877620157099) | > loss_mel_ce: 3.2194271087646484 (nan) | > loss: 0.012907914817333221 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1368081549939917) | > loader_time: 0.0045 (0.004670670373524677)  --> TIME: 2025-02-25 04:53:54 -- STEP: 6454/36938 -- GLOBAL_STEP: 634400 | > loss_text_ce: 0.028586577624082565 (0.03563381160479859) | > loss_mel_ce: 3.0890443325042725 (nan) | > loss: 0.012371552176773548 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13676113900585593) | > loader_time: 0.0045 (0.004670085609975084)  --> TIME: 2025-02-25 04:54:04 -- STEP: 6504/36938 -- GLOBAL_STEP: 634450 | > loss_text_ce: 0.036188628524541855 (0.035634497663998364) | > loss_mel_ce: 3.105194330215454 (nan) | > loss: 0.012465805746614933 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.13673901865842947) | > loader_time: 0.0046 (0.004669426698672964)  --> TIME: 2025-02-25 04:54:13 -- STEP: 6554/36938 -- GLOBAL_STEP: 634500 | > loss_text_ce: 0.03725256770849228 (0.0356340692064541) | > loss_mel_ce: 2.781315565109253 (nan) | > loss: 0.011184794828295708 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13673427179887637) | > loader_time: 0.0047 (0.004670255206727682)  --> TIME: 2025-02-25 04:54:22 -- STEP: 6604/36938 -- GLOBAL_STEP: 634550 | > loss_text_ce: 0.038856975734233856 (0.03564538143779404) | > loss_mel_ce: 3.768749237060547 (nan) | > loss: 0.015109549276530743 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13675332491792003) | > loader_time: 0.0047 (0.004670589711000387)  --> TIME: 2025-02-25 04:54:32 -- STEP: 6654/36938 -- GLOBAL_STEP: 634600 | > loss_text_ce: 0.02448897808790207 (0.035641676562115844) | > loss_mel_ce: 3.4062139987945557 (nan) | > loss: 0.013613901101052761 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13678103848409393) | > loader_time: 0.0048 (0.004670915354254872)  --> TIME: 2025-02-25 04:54:41 -- STEP: 6704/36938 -- GLOBAL_STEP: 634650 | > loss_text_ce: 0.021114502102136612 (0.03564653832349835) | > loss_mel_ce: 0.080078125 (nan) | > loss: 0.0004015580634586513 (nan) | > current_lr: 5e-06 | > step_time: 0.1083 (0.13677101768997732) | > loader_time: 0.0045 (0.004670655172593839)  --> TIME: 2025-02-25 04:54:51 -- STEP: 6754/36938 -- GLOBAL_STEP: 634700 | > loss_text_ce: 0.02909895032644272 (0.03564625186445781) | > loss_mel_ce: 2.4588050842285156 (nan) | > loss: 0.009872635826468468 (nan) | > current_lr: 5e-06 | > step_time: 0.1537 (0.13681501321337863) | > loader_time: 0.0046 (0.004671196313450808)  --> TIME: 2025-02-25 04:55:00 -- STEP: 6804/36938 -- GLOBAL_STEP: 634750 | > loss_text_ce: 0.03603074699640274 (0.03564041139594105) | > loss_mel_ce: 1.3860551118850708 (nan) | > loss: 0.005643198266625404 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13683552334828555) | > loader_time: 0.0045 (0.004670879512867601)  --> TIME: 2025-02-25 04:55:09 -- STEP: 6854/36938 -- GLOBAL_STEP: 634800 | > loss_text_ce: 0.035036541521549225 (0.03563777109630351) | > loss_mel_ce: 2.4120945930480957 (nan) | > loss: 0.009710838086903095 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13680439597842983) | > loader_time: 0.0046 (0.004670859426738293)  --> TIME: 2025-02-25 04:55:19 -- STEP: 6904/36938 -- GLOBAL_STEP: 634850 | > loss_text_ce: 0.05160693824291229 (0.03564679674747027) | > loss_mel_ce: 1.1022409200668335 (nan) | > loss: 0.004578761290758848 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13681604726151472) | > loader_time: 0.0046 (0.004670869468399437)  --> TIME: 2025-02-25 04:55:28 -- STEP: 6954/36938 -- GLOBAL_STEP: 634900 | > loss_text_ce: 0.041971754282712936 (0.035649177960104445) | > loss_mel_ce: 2.806452989578247 (nan) | > loss: 0.011303273029625416 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13680503098896732) | > loader_time: 0.0046 (0.00467075998494431)  --> TIME: 2025-02-25 04:55:38 -- STEP: 7004/36938 -- GLOBAL_STEP: 634950 | > loss_text_ce: 0.022226201370358467 (0.03564390342236896) | > loss_mel_ce: 0.0453287772834301 (nan) | > loss: 0.00026807532412931323 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13684507307496974) | > loader_time: 0.0046 (0.004670527238698771)  --> TIME: 2025-02-25 04:55:47 -- STEP: 7054/36938 -- GLOBAL_STEP: 635000 | > loss_text_ce: 0.04089374095201492 (0.035640819938518135) | > loss_mel_ce: 3.0851547718048096 (nan) | > loss: 0.012404954992234707 (nan) | > current_lr: 5e-06 | > step_time: 0.1687 (0.13685237029675545) | > loader_time: 0.0047 (0.004670117237360375)  --> TIME: 2025-02-25 04:55:57 -- STEP: 7104/36938 -- GLOBAL_STEP: 635050 | > loss_text_ce: 0.026923857629299164 (0.03564181791797822) | > loss_mel_ce: 3.008054494857788 (nan) | > loss: 0.012043565511703491 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13685891102697326) | > loader_time: 0.0045 (0.004669568291655532)  --> TIME: 2025-02-25 04:56:06 -- STEP: 7154/36938 -- GLOBAL_STEP: 635100 | > loss_text_ce: 0.029116149991750717 (0.03563181897674806) | > loss_mel_ce: 0.06005859375 (nan) | > loss: 0.00035386806121096015 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1368248503002194) | > loader_time: 0.0045 (0.00466930949557984)  --> TIME: 2025-02-25 04:56:14 -- STEP: 7204/36938 -- GLOBAL_STEP: 635150 | > loss_text_ce: 0.028874212875962257 (0.035624205247798565) | > loss_mel_ce: 4.794642925262451 (nan) | > loss: 0.0191409420222044 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13676463578026593) | > loader_time: 0.0045 (0.0046690519090363345)  --> TIME: 2025-02-25 04:56:23 -- STEP: 7254/36938 -- GLOBAL_STEP: 635200 | > loss_text_ce: 0.016897473484277725 (0.03561942556604388) | > loss_mel_ce: 0.54296875 (nan) | > loss: 0.002221691654995084 (nan) | > current_lr: 5e-06 | > step_time: 0.1075 (0.13670979227807972) | > loader_time: 0.0044 (0.004668332211137114)  --> TIME: 2025-02-25 04:56:32 -- STEP: 7304/36938 -- GLOBAL_STEP: 635250 | > loss_text_ce: 0.03211294487118721 (0.03562090858569235) | > loss_mel_ce: 2.482377052307129 (nan) | > loss: 0.009978135116398335 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.136701364735261) | > loader_time: 0.0045 (0.004667868064502244)  --> TIME: 2025-02-25 04:56:42 -- STEP: 7354/36938 -- GLOBAL_STEP: 635300 | > loss_text_ce: 0.0375913567841053 (0.03563128158714565) | > loss_mel_ce: 3.0795094966888428 (nan) | > loss: 0.012369449250400066 (nan) | > current_lr: 5e-06 | > step_time: 0.1818 (0.13670381178245017) | > loader_time: 0.0049 (0.004667955538096361)  --> TIME: 2025-02-25 04:56:51 -- STEP: 7404/36938 -- GLOBAL_STEP: 635350 | > loss_text_ce: 0.05264531821012497 (0.035639608908305885) | > loss_mel_ce: 3.3466796875 (nan) | > loss: 0.01348938513547182 (nan) | > current_lr: 5e-06 | > step_time: 0.1075 (0.1366924102856357) | > loader_time: 0.0045 (0.004667898630979961)  --> TIME: 2025-02-25 04:57:01 -- STEP: 7454/36938 -- GLOBAL_STEP: 635400 | > loss_text_ce: 0.032166820019483566 (0.035637714743148516) | > loss_mel_ce: 2.5228989124298096 (nan) | > loss: 0.010139149613678455 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.13672662110759753) | > loader_time: 0.0046 (0.004668496555126734)  --> TIME: 2025-02-25 04:57:10 -- STEP: 7504/36938 -- GLOBAL_STEP: 635450 | > loss_text_ce: 0.033322256058454514 (0.03563473892655345) | > loss_mel_ce: 2.7073895931243896 (nan) | > loss: 0.010875841602683067 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13673257008035988) | > loader_time: 0.0046 (0.004668377419270432)  --> TIME: 2025-02-25 04:57:19 -- STEP: 7554/36938 -- GLOBAL_STEP: 635500 | > loss_text_ce: 0.031257834285497665 (0.0356374206078142) | > loss_mel_ce: 2.8784284591674805 (nan) | > loss: 0.01154637522995472 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.136731435389691) | > loader_time: 0.0047 (0.004668264910440645)  --> TIME: 2025-02-25 04:57:29 -- STEP: 7604/36938 -- GLOBAL_STEP: 635550 | > loss_text_ce: 0.037509966641664505 (0.035642749392307584) | > loss_mel_ce: 2.748440980911255 (nan) | > loss: 0.011055361479520798 (nan) | > current_lr: 5e-06 | > step_time: 0.1783 (0.13671850355470316) | > loader_time: 0.0048 (0.004667688018080437)  --> TIME: 2025-02-25 04:57:38 -- STEP: 7654/36938 -- GLOBAL_STEP: 635600 | > loss_text_ce: 0.042268700897693634 (0.03563804341821331) | > loss_mel_ce: 3.129399538040161 (nan) | > loss: 0.012585986405611038 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13676056068810083) | > loader_time: 0.0045 (0.0046682515093262895)  --> TIME: 2025-02-25 04:57:47 -- STEP: 7704/36938 -- GLOBAL_STEP: 635650 | > loss_text_ce: 0.034102506935596466 (0.03563546768857732) | > loss_mel_ce: 2.691411256790161 (nan) | > loss: 0.010815531015396118 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13673429495944095) | > loader_time: 0.0046 (0.004667676621632165)  --> TIME: 2025-02-25 04:57:57 -- STEP: 7754/36938 -- GLOBAL_STEP: 635700 | > loss_text_ce: 0.04479716718196869 (0.03564021812172015) | > loss_mel_ce: 3.2479147911071777 (nan) | > loss: 0.01306631788611412 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.1367125427227884) | > loader_time: 0.0046 (0.004667199577349254)  --> TIME: 2025-02-25 04:58:06 -- STEP: 7804/36938 -- GLOBAL_STEP: 635750 | > loss_text_ce: 0.03757455572485924 (0.03564980701910488) | > loss_mel_ce: 3.0208306312561035 (nan) | > loss: 0.012136529199779034 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13672806940464827) | > loader_time: 0.0045 (0.00466701032442047)  --> TIME: 2025-02-25 04:58:15 -- STEP: 7854/36938 -- GLOBAL_STEP: 635800 | > loss_text_ce: 0.034096620976924896 (0.03564209074931249) | > loss_mel_ce: 2.64117169380188 (nan) | > loss: 0.010616145096719265 (nan) | > current_lr: 5e-06 | > step_time: 0.1523 (0.13672091877730583) | > loader_time: 0.0046 (0.004666726978370971)  --> TIME: 2025-02-25 04:58:25 -- STEP: 7904/36938 -- GLOBAL_STEP: 635850 | > loss_text_ce: 0.042467936873435974 (0.03564101113310482) | > loss_mel_ce: 2.6529195308685303 (nan) | > loss: 0.010695981793105602 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13671995337434073) | > loader_time: 0.0045 (0.004666490683912748)  --> TIME: 2025-02-25 04:58:34 -- STEP: 7954/36938 -- GLOBAL_STEP: 635900 | > loss_text_ce: 0.03846001625061035 (0.035642853326957914) | > loss_mel_ce: 3.6225485801696777 (nan) | > loss: 0.014527812600135803 (nan) | > current_lr: 5e-06 | > step_time: 0.1831 (0.13669634587746088) | > loader_time: 0.0049 (0.004666802449714367)  --> TIME: 2025-02-25 04:58:43 -- STEP: 8004/36938 -- GLOBAL_STEP: 635950 | > loss_text_ce: 0.04026614502072334 (0.03564121605987126) | > loss_mel_ce: 3.153233289718628 (nan) | > loss: 0.012672617100179195 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.13670483444524542) | > loader_time: 0.0049 (0.0046667517095372296)  --> TIME: 2025-02-25 04:58:52 -- STEP: 8054/36938 -- GLOBAL_STEP: 636000 | > loss_text_ce: 0.04926491156220436 (0.035640513331881285) | > loss_mel_ce: 2.842306613922119 (nan) | > loss: 0.011474491097033024 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.1366982076965545) | > loader_time: 0.0046 (0.0046665303784681365)  --> TIME: 2025-02-25 04:59:01 -- STEP: 8104/36938 -- GLOBAL_STEP: 636050 | > loss_text_ce: 0.026807639747858047 (0.03563332676357505) | > loss_mel_ce: 2.3504812717437744 (nan) | > loss: 0.00943368673324585 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1366599056233534) | > loader_time: 0.0045 (0.004666561965057433)  --> TIME: 2025-02-25 04:59:11 -- STEP: 8154/36938 -- GLOBAL_STEP: 636100 | > loss_text_ce: 0.04211810603737831 (0.035637683779351295) | > loss_mel_ce: 3.1051273345947266 (nan) | > loss: 0.012489070184528828 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13664356475663464) | > loader_time: 0.0045 (0.004666683455732251)  --> TIME: 2025-02-25 04:59:20 -- STEP: 8204/36938 -- GLOBAL_STEP: 636150 | > loss_text_ce: 0.03543912619352341 (0.03563350634020502) | > loss_mel_ce: 2.640990972518921 (nan) | > loss: 0.010620754212141037 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.1366487804533265) | > loader_time: 0.0046 (0.004666513899138705)  --> TIME: 2025-02-25 04:59:29 -- STEP: 8254/36938 -- GLOBAL_STEP: 636200 | > loss_text_ce: 0.03873142972588539 (0.035634631614298126) | > loss_mel_ce: 2.5931477546691895 (nan) | > loss: 0.01044396497309208 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13666233026810864) | > loader_time: 0.0046 (0.004666674937235983)  --> TIME: 2025-02-25 04:59:39 -- STEP: 8304/36938 -- GLOBAL_STEP: 636250 | > loss_text_ce: 0.034922923892736435 (0.03563211949296334) | > loss_mel_ce: 3.4187910556793213 (nan) | > loss: 0.013705214485526085 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13665273845884776) | > loader_time: 0.0047 (0.004666596852056325)  --> TIME: 2025-02-25 04:59:48 -- STEP: 8354/36938 -- GLOBAL_STEP: 636300 | > loss_text_ce: 0.02822193130850792 (0.035638049510208485) | > loss_mel_ce: 2.3277499675750732 (nan) | > loss: 0.009349094703793526 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13669002181695347) | > loader_time: 0.0045 (0.004667138551050379)  --> TIME: 2025-02-25 04:59:57 -- STEP: 8404/36938 -- GLOBAL_STEP: 636350 | > loss_text_ce: 0.038124747574329376 (0.035631954352788625) | > loss_mel_ce: 3.1674907207489014 (nan) | > loss: 0.012720697559416294 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13666400087838554) | > loader_time: 0.0047 (0.004667117305167343)  --> TIME: 2025-02-25 05:00:07 -- STEP: 8454/36938 -- GLOBAL_STEP: 636400 | > loss_text_ce: 0.015179043635725975 (0.03563121633451272) | > loss_mel_ce: 0.9171441197395325 (nan) | > loss: 0.003699695225805044 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.136659427712311) | > loader_time: 0.0045 (0.004667589110010532)  --> TIME: 2025-02-25 05:00:16 -- STEP: 8504/36938 -- GLOBAL_STEP: 636450 | > loss_text_ce: 0.028405122458934784 (0.035631136795477955) | > loss_mel_ce: 2.6488020420074463 (nan) | > loss: 0.010623838752508163 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.1366524916152732) | > loader_time: 0.0046 (0.004667435181331544)  --> TIME: 2025-02-25 05:00:25 -- STEP: 8554/36938 -- GLOBAL_STEP: 636500 | > loss_text_ce: 0.0376904271543026 (0.03562450035666173) | > loss_mel_ce: 2.4992239475250244 (nan) | > loss: 0.010067121125757694 (nan) | > current_lr: 5e-06 | > step_time: 0.1832 (0.13661219621055776) | > loader_time: 0.0048 (0.0046679873819910865)  --> TIME: 2025-02-25 05:00:34 -- STEP: 8604/36938 -- GLOBAL_STEP: 636550 | > loss_text_ce: 0.030139803886413574 (0.035617728120556975) | > loss_mel_ce: 2.4532642364501953 (nan) | > loss: 0.00985477864742279 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13663369256471322) | > loader_time: 0.0046 (0.004667645800895106)  --> TIME: 2025-02-25 05:00:43 -- STEP: 8654/36938 -- GLOBAL_STEP: 636600 | > loss_text_ce: 0.039568252861499786 (0.03561761788399703) | > loss_mel_ce: 3.556208848953247 (nan) | > loss: 0.014268957078456879 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.13662315539022116) | > loader_time: 0.0048 (0.004667489060855282)  --> TIME: 2025-02-25 05:00:53 -- STEP: 8704/36938 -- GLOBAL_STEP: 636650 | > loss_text_ce: 0.04627767577767372 (0.03561854698658445) | > loss_mel_ce: 3.202514886856079 (nan) | > loss: 0.012892034836113453 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13663893250529324) | > loader_time: 0.0046 (0.004667343243080028)  --> TIME: 2025-02-25 05:01:02 -- STEP: 8754/36938 -- GLOBAL_STEP: 636700 | > loss_text_ce: 0.027791041880846024 (0.03562174564913036) | > loss_mel_ce: 2.177776336669922 (nan) | > loss: 0.008752251975238323 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.1366334866518726) | > loader_time: 0.0046 (0.004667106599657404)  --> TIME: 2025-02-25 05:01:12 -- STEP: 8804/36938 -- GLOBAL_STEP: 636750 | > loss_text_ce: 0.035078540444374084 (0.035618282067312365) | > loss_mel_ce: 2.9489657878875732 (nan) | > loss: 0.011841446161270142 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.1366492316485638) | > loader_time: 0.0048 (0.00466686010577364)  --> TIME: 2025-02-25 05:01:21 -- STEP: 8854/36938 -- GLOBAL_STEP: 636800 | > loss_text_ce: 0.029903151094913483 (0.03561795405224763) | > loss_mel_ce: 2.721738338470459 (nan) | > loss: 0.01091921329498291 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13665528221487921) | > loader_time: 0.0046 (0.004666586640674435)  --> TIME: 2025-02-25 05:01:30 -- STEP: 8904/36938 -- GLOBAL_STEP: 636850 | > loss_text_ce: 0.03562309965491295 (0.03562308792321908) | > loss_mel_ce: 3.471576690673828 (nan) | > loss: 0.013917460106313229 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13666302774365868) | > loader_time: 0.0047 (0.004666801652497045)  --> TIME: 2025-02-25 05:01:40 -- STEP: 8954/36938 -- GLOBAL_STEP: 636900 | > loss_text_ce: 0.037507880479097366 (0.03562673172849637) | > loss_mel_ce: 2.660273790359497 (nan) | > loss: 0.01070548314601183 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.136667469795564) | > loader_time: 0.0045 (0.004666754462928261)  --> TIME: 2025-02-25 05:01:49 -- STEP: 9004/36938 -- GLOBAL_STEP: 636950 | > loss_text_ce: 0.037591353058815 (0.03562203459776797) | > loss_mel_ce: 2.830162525177002 (nan) | > loss: 0.011379976756870747 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13665425300809936) | > loader_time: 0.0045 (0.004666786731904698)  --> TIME: 2025-02-25 05:01:58 -- STEP: 9054/36938 -- GLOBAL_STEP: 637000 | > loss_text_ce: 0.021487964317202568 (0.03562461869840118) | > loss_mel_ce: 3.017958879470825 (nan) | > loss: 0.01206129789352417 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1366406065858905) | > loader_time: 0.0046 (0.0046667513111081724)  --> TIME: 2025-02-25 05:02:07 -- STEP: 9104/36938 -- GLOBAL_STEP: 637050 | > loss_text_ce: 0.03225001320242882 (0.03562554606500044) | > loss_mel_ce: 3.4134597778320312 (nan) | > loss: 0.013673451729118824 (nan) | > current_lr: 5e-06 | > step_time: 0.1876 (0.1366348883336789) | > loader_time: 0.0048 (0.0046668041150473615)  --> TIME: 2025-02-25 05:02:16 -- STEP: 9154/36938 -- GLOBAL_STEP: 637100 | > loss_text_ce: 0.03685388341546059 (0.03562216647620457) | > loss_mel_ce: 3.392617702484131 (nan) | > loss: 0.013609014451503754 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13659864014247755) | > loader_time: 0.0046 (0.004666418025971302)  --> TIME: 2025-02-25 05:02:25 -- STEP: 9204/36938 -- GLOBAL_STEP: 637150 | > loss_text_ce: 0.03575987368822098 (0.03562189453463201) | > loss_mel_ce: 2.523683547973633 (nan) | > loss: 0.010156522504985332 (nan) | > current_lr: 5e-06 | > step_time: 0.1511 (0.1365869784033953) | > loader_time: 0.0046 (0.004666130965505159)  --> TIME: 2025-02-25 05:02:34 -- STEP: 9254/36938 -- GLOBAL_STEP: 637200 | > loss_text_ce: 0.03480430692434311 (0.03563013980766962) | > loss_mel_ce: 2.270341396331787 (nan) | > loss: 0.009147404693067074 (nan) | > current_lr: 5e-06 | > step_time: 0.1574 (0.1365605339984384) | > loader_time: 0.0046 (0.00466599259851612)  --> TIME: 2025-02-25 05:02:44 -- STEP: 9304/36938 -- GLOBAL_STEP: 637250 | > loss_text_ce: 0.03301384299993515 (0.03562324812829182) | > loss_mel_ce: 2.9780728816986084 (nan) | > loss: 0.01194875780493021 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13658624779839784) | > loader_time: 0.0045 (0.004666092317881312)  --> TIME: 2025-02-25 05:02:54 -- STEP: 9354/36938 -- GLOBAL_STEP: 637300 | > loss_text_ce: 0.03052476979792118 (0.03561509066847394) | > loss_mel_ce: 2.2665743827819824 (nan) | > loss: 0.009115473367273808 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13661396638537024) | > loader_time: 0.0046 (0.004666226756911729)  --> TIME: 2025-02-25 05:03:03 -- STEP: 9404/36938 -- GLOBAL_STEP: 637350 | > loss_text_ce: 0.031538594514131546 (0.03561608382191331) | > loss_mel_ce: 3.5299479961395264 (nan) | > loss: 0.01413288339972496 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13662080017367276) | > loader_time: 0.0045 (0.004666437194074594)  --> TIME: 2025-02-25 05:03:13 -- STEP: 9454/36938 -- GLOBAL_STEP: 637400 | > loss_text_ce: 0.0377364419400692 (0.035620777938662376) | > loss_mel_ce: 2.6247708797454834 (nan) | > loss: 0.010565505363047123 (nan) | > current_lr: 5e-06 | > step_time: 0.1519 (0.1366396528080194) | > loader_time: 0.0046 (0.004666401060333166)  --> TIME: 2025-02-25 05:03:22 -- STEP: 9504/36938 -- GLOBAL_STEP: 637450 | > loss_text_ce: 0.03722608461976051 (0.03562007037909421) | > loss_mel_ce: 2.9400382041931152 (nan) | > loss: 0.01181454211473465 (nan) | > current_lr: 5e-06 | > step_time: 0.1851 (0.13666968847866384) | > loader_time: 0.0048 (0.004666442195775144)  --> TIME: 2025-02-25 05:03:32 -- STEP: 9554/36938 -- GLOBAL_STEP: 637500 | > loss_text_ce: 0.03834503889083862 (0.035624926089218946) | > loss_mel_ce: 2.5890870094299316 (nan) | > loss: 0.010426318272948265 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13666965765810285) | > loader_time: 0.0046 (0.0046662595298516485)  --> TIME: 2025-02-25 05:03:41 -- STEP: 9604/36938 -- GLOBAL_STEP: 637550 | > loss_text_ce: 0.03799538314342499 (0.03562179527151211) | > loss_mel_ce: 3.0391438007354736 (nan) | > loss: 0.012210870161652565 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.13670667952569682) | > loader_time: 0.0048 (0.0046661136449252325)  --> TIME: 2025-02-25 05:03:50 -- STEP: 9654/36938 -- GLOBAL_STEP: 637600 | > loss_text_ce: 0.03160202130675316 (0.03562085713260461) | > loss_mel_ce: 3.623192310333252 (nan) | > loss: 0.01450315210968256 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13667596255816672) | > loader_time: 0.0046 (0.004666168323727861)  --> TIME: 2025-02-25 05:04:00 -- STEP: 9704/36938 -- GLOBAL_STEP: 637650 | > loss_text_ce: 0.037596333771944046 (0.03562520961879905) | > loss_mel_ce: 3.174663543701172 (nan) | > loss: 0.012747063301503658 (nan) | > current_lr: 5e-06 | > step_time: 0.1984 (0.13670231734831642) | > loader_time: 0.0049 (0.004666093451271161)  --> TIME: 2025-02-25 05:04:10 -- STEP: 9754/36938 -- GLOBAL_STEP: 637700 | > loss_text_ce: 0.04420776292681694 (0.03561922064330223) | > loss_mel_ce: 3.4197049140930176 (nan) | > loss: 0.013745686039328575 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13671114653966088) | > loader_time: 0.0047 (0.004666399413111376)  --> TIME: 2025-02-25 05:04:19 -- STEP: 9804/36938 -- GLOBAL_STEP: 637750 | > loss_text_ce: 0.022255834192037582 (0.035616289895683) | > loss_mel_ce: 0.1482747346162796 (nan) | > loss: 0.0006767086451873183 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13671585575797895) | > loader_time: 0.0047 (0.004666218656464437)  --> TIME: 2025-02-25 05:04:28 -- STEP: 9854/36938 -- GLOBAL_STEP: 637800 | > loss_text_ce: 0.03293377161026001 (0.03561543583036721) | > loss_mel_ce: 1.8712451457977295 (nan) | > loss: 0.007556265685707331 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13670049215490573) | > loader_time: 0.0045 (0.004666407620759245)  --> TIME: 2025-02-25 05:04:37 -- STEP: 9904/36938 -- GLOBAL_STEP: 637850 | > loss_text_ce: 0.036013346165418625 (0.035614170834064075) | > loss_mel_ce: 3.328066110610962 (nan) | > loss: 0.013349522836506367 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.1366890206668217) | > loader_time: 0.0047 (0.004666133992891517)  --> TIME: 2025-02-25 05:04:47 -- STEP: 9954/36938 -- GLOBAL_STEP: 637900 | > loss_text_ce: 0.03716767951846123 (0.03561133168703628) | > loss_mel_ce: 2.9448018074035645 (nan) | > loss: 0.01183321326971054 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13670657278424586) | > loader_time: 0.0045 (0.004666114203154336)  --> TIME: 2025-02-25 05:04:56 -- STEP: 10004/36938 -- GLOBAL_STEP: 637950 | > loss_text_ce: 0.03295324742794037 (0.0356157139284028) | > loss_mel_ce: 3.2359907627105713 (nan) | > loss: 0.012972000986337662 (nan) | > current_lr: 5e-06 | > step_time: 0.176 (0.13669314841087776) | > loader_time: 0.0049 (0.004665927904122161)  --> TIME: 2025-02-25 05:05:05 -- STEP: 10054/36938 -- GLOBAL_STEP: 638000 | > loss_text_ce: 0.03432334214448929 (0.0356212028515121) | > loss_mel_ce: 2.8252835273742676 (nan) | > loss: 0.01134764775633812 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13667499412000106) | > loader_time: 0.0047 (0.0046658353016346575)  --> TIME: 2025-02-25 05:05:14 -- STEP: 10104/36938 -- GLOBAL_STEP: 638050 | > loss_text_ce: 0.031773630529642105 (0.03562087674462254) | > loss_mel_ce: 2.79280686378479 (nan) | > loss: 0.011208653450012207 (nan) | > current_lr: 5e-06 | > step_time: 0.1699 (0.13668680599258692) | > loader_time: 0.0047 (0.004665583702466451)  --> TIME: 2025-02-25 05:05:24 -- STEP: 10154/36938 -- GLOBAL_STEP: 638100 | > loss_text_ce: 0.027882346883416176 (0.03561883545004641) | > loss_mel_ce: 2.5848941802978516 (nan) | > loss: 0.010368161834776402 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13669011643112747) | > loader_time: 0.0045 (0.004665380297201173)  --> TIME: 2025-02-25 05:05:33 -- STEP: 10204/36938 -- GLOBAL_STEP: 638150 | > loss_text_ce: 0.03798341378569603 (0.03562193074450185) | > loss_mel_ce: 2.857218027114868 (nan) | > loss: 0.011488894931972027 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.13669540675562156) | > loader_time: 0.0047 (0.004665139935522253)  --> TIME: 2025-02-25 05:05:43 -- STEP: 10254/36938 -- GLOBAL_STEP: 638200 | > loss_text_ce: 0.04187947139143944 (0.03562230705144616) | > loss_mel_ce: 3.2255470752716064 (nan) | > loss: 0.012965979054570198 (nan) | > current_lr: 5e-06 | > step_time: 0.1775 (0.13671208481261288) | > loader_time: 0.0048 (0.0046650314972789915)  --> TIME: 2025-02-25 05:05:52 -- STEP: 10304/36938 -- GLOBAL_STEP: 638250 | > loss_text_ce: 0.030561285093426704 (0.03562638201528105) | > loss_mel_ce: 3.3706369400024414 (nan) | > loss: 0.013496818952262402 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13670629184253447) | > loader_time: 0.0045 (0.004664848749496924)  --> TIME: 2025-02-25 05:06:01 -- STEP: 10354/36938 -- GLOBAL_STEP: 638300 | > loss_text_ce: 0.03499230742454529 (0.03563335723095597) | > loss_mel_ce: 2.6234264373779297 (nan) | > loss: 0.010549280792474747 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13669122543393886) | > loader_time: 0.0046 (0.00466471356684254)  --> TIME: 2025-02-25 05:06:10 -- STEP: 10404/36938 -- GLOBAL_STEP: 638350 | > loss_text_ce: 0.03721170499920845 (0.035633289736975944) | > loss_mel_ce: 3.350200891494751 (nan) | > loss: 0.013442113995552063 (nan) | > current_lr: 5e-06 | > step_time: 0.1833 (0.1366891124669242) | > loader_time: 0.0049 (0.004664490631753231)  --> TIME: 2025-02-25 05:06:19 -- STEP: 10454/36938 -- GLOBAL_STEP: 638400 | > loss_text_ce: 0.046836547553539276 (0.03563341591079573) | > loss_mel_ce: 3.095670700073242 (nan) | > loss: 0.012470267713069916 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13667560161950357) | > loader_time: 0.0046 (0.004664301621382521)  --> TIME: 2025-02-25 05:06:29 -- STEP: 10504/36938 -- GLOBAL_STEP: 638450 | > loss_text_ce: 0.030323190614581108 (0.03563257694416034) | > loss_mel_ce: 2.5360214710235596 (nan) | > loss: 0.0101839080452919 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13667118469904327) | > loader_time: 0.0046 (0.004664149682937648)  --> TIME: 2025-02-25 05:06:38 -- STEP: 10554/36938 -- GLOBAL_STEP: 638500 | > loss_text_ce: 0.03798961639404297 (0.03563566200462691) | > loss_mel_ce: 3.2922403812408447 (nan) | > loss: 0.013215199112892151 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13665584147100723) | > loader_time: 0.0046 (0.0046637234688708794)  --> TIME: 2025-02-25 05:06:47 -- STEP: 10604/36938 -- GLOBAL_STEP: 638550 | > loss_text_ce: 0.03447682783007622 (0.03563735869132408) | > loss_mel_ce: 2.617697238922119 (nan) | > loss: 0.010524500161409378 (nan) | > current_lr: 5e-06 | > step_time: 0.1776 (0.1366682480029995) | > loader_time: 0.0048 (0.00466339269344513)  --> TIME: 2025-02-25 05:06:56 -- STEP: 10654/36938 -- GLOBAL_STEP: 638600 | > loss_text_ce: 0.029844719916582108 (0.035633068680882084) | > loss_mel_ce: 0.012603759765625 (nan) | > loss: 0.000168446364114061 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13664941550957138) | > loader_time: 0.0045 (0.004662967721807686)  --> TIME: 2025-02-25 05:07:06 -- STEP: 10704/36938 -- GLOBAL_STEP: 638650 | > loss_text_ce: 0.038463059812784195 (0.03562997151093108) | > loss_mel_ce: 3.263366222381592 (nan) | > loss: 0.013102497905492783 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.13665003299445846) | > loader_time: 0.0048 (0.004662715622484246)  --> TIME: 2025-02-25 05:07:15 -- STEP: 10754/36938 -- GLOBAL_STEP: 638700 | > loss_text_ce: 0.03510018810629845 (0.035634011939869016) | > loss_mel_ce: 3.042252779006958 (nan) | > loss: 0.01221171859651804 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13666676479532577) | > loader_time: 0.0046 (0.004662620992449468)  --> TIME: 2025-02-25 05:07:24 -- STEP: 10804/36938 -- GLOBAL_STEP: 638750 | > loss_text_ce: 0.030950693413615227 (0.03563260658041063) | > loss_mel_ce: 3.1978237628936768 (nan) | > loss: 0.0128125986084342 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.1366430644810708) | > loader_time: 0.0045 (0.0046623523303289385)  --> TIME: 2025-02-25 05:07:33 -- STEP: 10854/36938 -- GLOBAL_STEP: 638800 | > loss_text_ce: 0.039351992309093475 (0.035629209597092634) | > loss_mel_ce: 2.4534597396850586 (nan) | > loss: 0.009892110712826252 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.136606975266875) | > loader_time: 0.0045 (0.004662090141250002)  --> TIME: 2025-02-25 05:07:42 -- STEP: 10904/36938 -- GLOBAL_STEP: 638850 | > loss_text_ce: 0.041820090264081955 (0.03562264331744227) | > loss_mel_ce: 2.209871530532837 (nan) | > loss: 0.008935284800827503 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.1365794766194141) | > loader_time: 0.0046 (0.004661872119301842)  --> TIME: 2025-02-25 05:07:51 -- STEP: 10954/36938 -- GLOBAL_STEP: 638900 | > loss_text_ce: 0.04418281838297844 (0.03562751090450257) | > loss_mel_ce: 3.100778341293335 (nan) | > loss: 0.012480005621910095 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1365883895596471) | > loader_time: 0.0046 (0.004661776842330099)  --> TIME: 2025-02-25 05:08:01 -- STEP: 11004/36938 -- GLOBAL_STEP: 638950 | > loss_text_ce: 0.026642460376024246 (0.03562322475191471) | > loss_mel_ce: 3.4501953125 (nan) | > loss: 0.013796976767480373 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1365690170656938) | > loader_time: 0.0045 (0.004661689711154409)  --> TIME: 2025-02-25 05:08:10 -- STEP: 11054/36938 -- GLOBAL_STEP: 639000 | > loss_text_ce: 0.03106509894132614 (0.03562272455787938) | > loss_mel_ce: 2.6941802501678467 (nan) | > loss: 0.010814465582370758 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13656533564627452) | > loader_time: 0.0045 (0.004661400818104392)  --> TIME: 2025-02-25 05:08:19 -- STEP: 11104/36938 -- GLOBAL_STEP: 639050 | > loss_text_ce: 0.041823506355285645 (0.03562323471798984) | > loss_mel_ce: 2.7023680210113525 (nan) | > loss: 0.010889650322496891 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1365477956767042) | > loader_time: 0.0045 (0.004660921241089671)  --> TIME: 2025-02-25 05:08:29 -- STEP: 11154/36938 -- GLOBAL_STEP: 639100 | > loss_text_ce: 0.03057905100286007 (0.03562017153916471) | > loss_mel_ce: 2.9504809379577637 (nan) | > loss: 0.011829604394733906 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13657633089476312) | > loader_time: 0.0046 (0.004660893773357147)  --> TIME: 2025-02-25 05:08:38 -- STEP: 11204/36938 -- GLOBAL_STEP: 639150 | > loss_text_ce: 0.038454990833997726 (0.03561729319863773) | > loss_mel_ce: 3.3712804317474365 (nan) | > loss: 0.013530696742236614 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.13658272622100648) | > loader_time: 0.0048 (0.004660743213219127)  --> TIME: 2025-02-25 05:08:47 -- STEP: 11254/36938 -- GLOBAL_STEP: 639200 | > loss_text_ce: 0.03595878928899765 (0.03561809085844418) | > loss_mel_ce: 3.165611743927002 (nan) | > loss: 0.012704645283520222 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.13657643391646662) | > loader_time: 0.0048 (0.004660600410042479)  --> TIME: 2025-02-25 05:08:57 -- STEP: 11304/36938 -- GLOBAL_STEP: 639250 | > loss_text_ce: 0.039468273520469666 (0.03561615267052031) | > loss_mel_ce: 2.5134530067443848 (nan) | > loss: 0.010130641050636768 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.1365787867780959) | > loader_time: 0.0047 (0.00466047207345714)  --> TIME: 2025-02-25 05:09:06 -- STEP: 11354/36938 -- GLOBAL_STEP: 639300 | > loss_text_ce: 0.02367164008319378 (0.0356095409844648) | > loss_mel_ce: 3.355804443359375 (nan) | > loss: 0.013410620391368866 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.13657347455040705) | > loader_time: 0.0047 (0.004660344594209919)  --> TIME: 2025-02-25 05:09:15 -- STEP: 11404/36938 -- GLOBAL_STEP: 639350 | > loss_text_ce: 0.0312834270298481 (0.03560588100953813) | > loss_mel_ce: 2.7390031814575195 (nan) | > loss: 0.010993201285600662 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13655837177603117) | > loader_time: 0.0045 (0.0046607288549925735)  --> TIME: 2025-02-25 05:09:24 -- STEP: 11454/36938 -- GLOBAL_STEP: 639400 | > loss_text_ce: 0.05032713711261749 (0.03560732729576525) | > loss_mel_ce: 2.0442707538604736 (nan) | > loss: 0.008311896584928036 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1365557316608076) | > loader_time: 0.0045 (0.004661475610874587)  --> TIME: 2025-02-25 05:09:33 -- STEP: 11504/36938 -- GLOBAL_STEP: 639450 | > loss_text_ce: 0.03560204058885574 (0.03560211907779792) | > loss_mel_ce: 2.2775423526763916 (nan) | > loss: 0.009179145097732544 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13651841657135214) | > loader_time: 0.0045 (0.0046610947237227095)  --> TIME: 2025-02-25 05:09:43 -- STEP: 11554/36938 -- GLOBAL_STEP: 639500 | > loss_text_ce: 0.04052305966615677 (0.035601750988885676) | > loss_mel_ce: 3.34554386138916 (nan) | > loss: 0.013436773791909218 (nan) | > current_lr: 5e-06 | > step_time: 0.1873 (0.13654672111093935) | > loader_time: 0.0048 (0.004661171312911598)  --> TIME: 2025-02-25 05:09:52 -- STEP: 11604/36938 -- GLOBAL_STEP: 639550 | > loss_text_ce: 0.03934058919548988 (0.03559621014765167) | > loss_mel_ce: 3.31182861328125 (nan) | > loss: 0.01329829078167677 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.1365443880251468) | > loader_time: 0.0048 (0.00466111323949347)  --> TIME: 2025-02-25 05:10:02 -- STEP: 11654/36938 -- GLOBAL_STEP: 639600 | > loss_text_ce: 0.033118441700935364 (0.035594598017201155) | > loss_mel_ce: 3.4499735832214355 (nan) | > loss: 0.013821794651448727 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.1365660092874443) | > loader_time: 0.005 (0.004661056094008413)  --> TIME: 2025-02-25 05:10:11 -- STEP: 11704/36938 -- GLOBAL_STEP: 639650 | > loss_text_ce: 0.033347271382808685 (0.03559404733421743) | > loss_mel_ce: 2.9216182231903076 (nan) | > loss: 0.01172605436295271 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13656735322447788) | > loader_time: 0.0046 (0.004660982101320496)  --> TIME: 2025-02-25 05:10:21 -- STEP: 11754/36938 -- GLOBAL_STEP: 639700 | > loss_text_ce: 0.03326641768217087 (0.03559412770833739) | > loss_mel_ce: 2.21875 (nan) | > loss: 0.008936572819948196 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13657028565139923) | > loader_time: 0.0045 (0.0046608549448753055)  --> TIME: 2025-02-25 05:10:30 -- STEP: 11804/36938 -- GLOBAL_STEP: 639750 | > loss_text_ce: 0.03916531428694725 (0.03559623546013094) | > loss_mel_ce: 2.6226704120635986 (nan) | > loss: 0.010562840849161148 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.136553366561536) | > loader_time: 0.0046 (0.004660636176500683)  --> TIME: 2025-02-25 05:10:39 -- STEP: 11854/36938 -- GLOBAL_STEP: 639800 | > loss_text_ce: 0.030499085783958435 (0.03558956983726886) | > loss_mel_ce: 3.0872628688812256 (nan) | > loss: 0.012372071854770184 (nan) | > current_lr: 5e-06 | > step_time: 0.1814 (0.13656376764588135) | > loader_time: 0.0048 (0.004660777223438051)  --> TIME: 2025-02-25 05:10:48 -- STEP: 11904/36938 -- GLOBAL_STEP: 639850 | > loss_text_ce: 0.04895607754588127 (0.03559782175587836) | > loss_mel_ce: 0.04443359375 (nan) | > loss: 0.0003705939743667841 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.1365523492336599) | > loader_time: 0.0047 (0.004660568770862405)  --> TIME: 2025-02-25 05:10:57 -- STEP: 11954/36938 -- GLOBAL_STEP: 639900 | > loss_text_ce: 0.031806934624910355 (0.03559489114791494) | > loss_mel_ce: 3.11328125 (nan) | > loss: 0.012480509467422962 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13655489087802908) | > loader_time: 0.0047 (0.004660493597171377)  --> TIME: 2025-02-25 05:11:07 -- STEP: 12004/36938 -- GLOBAL_STEP: 639950 | > loss_text_ce: 0.027965698391199112 (0.03559879305330534) | > loss_mel_ce: 2.8970980644226074 (nan) | > loss: 0.011607397347688675 (nan) | > current_lr: 5e-06 | > step_time: 0.1579 (0.13656780116759124) | > loader_time: 0.0047 (0.00466047366910361)  --> TIME: 2025-02-25 05:11:16 -- STEP: 12054/36938 -- GLOBAL_STEP: 640000 | > loss_text_ce: 0.03406193479895592 (0.03559822635184277) | > loss_mel_ce: 2.049497365951538 (nan) | > loss: 0.008268092758953571 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13655743583905844) | > loader_time: 0.0046 (0.004660282835984919) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_640000.pth  --> TIME: 2025-02-25 05:11:31 -- STEP: 12104/36938 -- GLOBAL_STEP: 640050 | > loss_text_ce: 0.04440790414810181 (0.0355975869429798) | > loss_mel_ce: 2.4782698154449463 (nan) | > loss: 0.010010626167058945 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13654499633455683) | > loader_time: 0.0046 (0.004660567915053538)  --> TIME: 2025-02-25 05:11:40 -- STEP: 12154/36938 -- GLOBAL_STEP: 640100 | > loss_text_ce: 0.034489747136831284 (0.035594281381949394) | > loss_mel_ce: 2.986802816390991 (nan) | > loss: 0.011989256367087364 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13652539282659637) | > loader_time: 0.0045 (0.004660354744197316)  --> TIME: 2025-02-25 05:11:49 -- STEP: 12204/36938 -- GLOBAL_STEP: 640150 | > loss_text_ce: 0.03392082080245018 (0.035588702422519776) | > loss_mel_ce: 3.108492136001587 (nan) | > loss: 0.01246989332139492 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1365091366206813) | > loader_time: 0.0045 (0.004660232287475385)  --> TIME: 2025-02-25 05:11:58 -- STEP: 12254/36938 -- GLOBAL_STEP: 640200 | > loss_text_ce: 0.03409004211425781 (0.035590069666179364) | > loss_mel_ce: 3.4610073566436768 (nan) | > loss: 0.013869434595108032 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13649224157101528) | > loader_time: 0.0046 (0.004660378725080837)  --> TIME: 2025-02-25 05:12:07 -- STEP: 12304/36938 -- GLOBAL_STEP: 640250 | > loss_text_ce: 0.02382461540400982 (0.035589068332430675) | > loss_mel_ce: 3.196223020553589 (nan) | > loss: 0.012777968309819698 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13646356794397058) | > loader_time: 0.0046 (0.004660289003108047)  --> TIME: 2025-02-25 05:12:17 -- STEP: 12354/36938 -- GLOBAL_STEP: 640300 | > loss_text_ce: 0.034545231610536575 (0.035591985980509666) | > loss_mel_ce: 3.0685477256774902 (nan) | > loss: 0.012313861399888992 (nan) | > current_lr: 5e-06 | > step_time: 0.1853 (0.1364634100704191) | > loader_time: 0.0049 (0.004660151258378779)  --> TIME: 2025-02-25 05:12:26 -- STEP: 12404/36938 -- GLOBAL_STEP: 640350 | > loss_text_ce: 0.04084058105945587 (0.03559465990693107) | > loss_mel_ce: 3.269090414047241 (nan) | > loss: 0.013134648092091084 (nan) | > current_lr: 5e-06 | > step_time: 0.1533 (0.13643518064299692) | > loader_time: 0.0046 (0.004660129681666874)  --> TIME: 2025-02-25 05:12:35 -- STEP: 12454/36938 -- GLOBAL_STEP: 640400 | > loss_text_ce: 0.026616020128130913 (0.03559817002921072) | > loss_mel_ce: 1.5604568719863892 (nan) | > loss: 0.00629790872335434 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.1364421087914077) | > loader_time: 0.0046 (0.004660067386758909)  --> TIME: 2025-02-25 05:12:44 -- STEP: 12504/36938 -- GLOBAL_STEP: 640450 | > loss_text_ce: 0.034628480672836304 (0.03560267858809318) | > loss_mel_ce: 1.6549072265625 (nan) | > loss: 0.0067045073956251144 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13643454128705973) | > loader_time: 0.0044 (0.004660118411766438)  --> TIME: 2025-02-25 05:12:54 -- STEP: 12554/36938 -- GLOBAL_STEP: 640500 | > loss_text_ce: 0.017315246164798737 (0.035600393920705194) | > loss_mel_ce: 1.68130362033844 (nan) | > loss: 0.006740551441907883 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13643891246945208) | > loader_time: 0.0045 (0.004659900434347256)  --> TIME: 2025-02-25 05:13:03 -- STEP: 12604/36938 -- GLOBAL_STEP: 640550 | > loss_text_ce: 0.028828302398324013 (0.03560178236740582) | > loss_mel_ce: 1.717699646949768 (nan) | > loss: 0.006930666510015726 (nan) | > current_lr: 5e-06 | > step_time: 0.1615 (0.13644055248025685) | > loader_time: 0.0046 (0.00465983192113346)  --> TIME: 2025-02-25 05:13:12 -- STEP: 12654/36938 -- GLOBAL_STEP: 640600 | > loss_text_ce: 0.030107133090496063 (0.035601406685918886) | > loss_mel_ce: 2.947906017303467 (nan) | > loss: 0.011817513033747673 (nan) | > current_lr: 5e-06 | > step_time: 0.1704 (0.13643084392740576) | > loader_time: 0.0048 (0.0046597794181120755)  --> TIME: 2025-02-25 05:13:21 -- STEP: 12704/36938 -- GLOBAL_STEP: 640650 | > loss_text_ce: 0.036599189043045044 (0.0356014517314938) | > loss_mel_ce: 0.0843573659658432 (nan) | > loss: 0.00047998636728152633 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.1364223711586722) | > loader_time: 0.0046 (0.00465949598502753)  --> TIME: 2025-02-25 05:13:31 -- STEP: 12754/36938 -- GLOBAL_STEP: 640700 | > loss_text_ce: 0.034954629838466644 (0.03559578739444585) | > loss_mel_ce: 2.375742197036743 (nan) | > loss: 0.009566257707774639 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13641462395964019) | > loader_time: 0.0046 (0.004659256124563765)  --> TIME: 2025-02-25 05:13:40 -- STEP: 12804/36938 -- GLOBAL_STEP: 640750 | > loss_text_ce: 0.032728396356105804 (0.0355969689272474) | > loss_mel_ce: 2.3591573238372803 (nan) | > loss: 0.00949161034077406 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.1363935847090247) | > loader_time: 0.0046 (0.004658991118886634)  --> TIME: 2025-02-25 05:13:49 -- STEP: 12854/36938 -- GLOBAL_STEP: 640800 | > loss_text_ce: 0.013623052276670933 (0.03559614481601193) | > loss_mel_ce: 2.3127527236938477 (nan) | > loss: 0.00923165027052164 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13641943639630924) | > loader_time: 0.0045 (0.004658894904649931)  --> TIME: 2025-02-25 05:13:59 -- STEP: 12904/36938 -- GLOBAL_STEP: 640850 | > loss_text_ce: 0.02741856314241886 (0.03559593464506534) | > loss_mel_ce: 2.5802786350250244 (nan) | > loss: 0.010348005220293999 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13642083839164074) | > loader_time: 0.0046 (0.0046586822406593864)  --> TIME: 2025-02-25 05:14:08 -- STEP: 12954/36938 -- GLOBAL_STEP: 640900 | > loss_text_ce: 0.04500013217329979 (0.035596282788625075) | > loss_mel_ce: 2.492875099182129 (nan) | > loss: 0.01007093396037817 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13640726634118808) | > loader_time: 0.0045 (0.0046585905012643475)  --> TIME: 2025-02-25 05:14:17 -- STEP: 13004/36938 -- GLOBAL_STEP: 640950 | > loss_text_ce: 0.025582339614629745 (0.03559999771746187) | > loss_mel_ce: 2.9620518684387207 (nan) | > loss: 0.01185569167137146 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13641399959167053) | > loader_time: 0.0046 (0.004658628668795366)  --> TIME: 2025-02-25 05:14:27 -- STEP: 13054/36938 -- GLOBAL_STEP: 641000 | > loss_text_ce: 0.034635722637176514 (0.03559651563875214) | > loss_mel_ce: 3.0970263481140137 (nan) | > loss: 0.012427231296896935 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.1364318256047964) | > loader_time: 0.0045 (0.004658815377482199)  --> TIME: 2025-02-25 05:14:36 -- STEP: 13104/36938 -- GLOBAL_STEP: 641050 | > loss_text_ce: 0.043948959559202194 (0.035595152527377986) | > loss_mel_ce: 3.5902557373046875 (nan) | > loss: 0.014421448111534119 (nan) | > current_lr: 5e-06 | > step_time: 0.1439 (0.1364332341631112) | > loader_time: 0.0046 (0.0046587224335606445)  --> TIME: 2025-02-25 05:14:46 -- STEP: 13154/36938 -- GLOBAL_STEP: 641100 | > loss_text_ce: 0.03747599944472313 (0.03559132827927578) | > loss_mel_ce: 3.1760687828063965 (nan) | > loss: 0.012752163223922253 (nan) | > current_lr: 5e-06 | > step_time: 0.1501 (0.1364373258132788) | > loader_time: 0.0047 (0.004658727111543806)  --> TIME: 2025-02-25 05:14:55 -- STEP: 13204/36938 -- GLOBAL_STEP: 641150 | > loss_text_ce: 0.03437185660004616 (0.03559311758684495) | > loss_mel_ce: 1.9635974168777466 (nan) | > loss: 0.007928449660539627 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13642749671537377) | > loader_time: 0.0046 (0.004658658679273408)  --> TIME: 2025-02-25 05:15:04 -- STEP: 13254/36938 -- GLOBAL_STEP: 641200 | > loss_text_ce: 0.04281678423285484 (0.035592146415836017) | > loss_mel_ce: 2.994774103164673 (nan) | > loss: 0.01205393299460411 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.1364212696307997) | > loader_time: 0.0046 (0.004658472831205704)  --> TIME: 2025-02-25 05:15:13 -- STEP: 13304/36938 -- GLOBAL_STEP: 641250 | > loss_text_ce: 0.032103583216667175 (0.03559328935258024) | > loss_mel_ce: 2.890244245529175 (nan) | > loss: 0.01159661915153265 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.1364124806545191) | > loader_time: 0.0046 (0.004658221840069944)  --> TIME: 2025-02-25 05:15:22 -- STEP: 13354/36938 -- GLOBAL_STEP: 641300 | > loss_text_ce: 0.04100274294614792 (0.03559509069198923) | > loss_mel_ce: 3.6249308586120605 (nan) | > loss: 0.014547356404364109 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13639407770637665) | > loader_time: 0.0045 (0.00465789227959137)  --> TIME: 2025-02-25 05:15:32 -- STEP: 13404/36938 -- GLOBAL_STEP: 641350 | > loss_text_ce: 0.039891477674245834 (0.035591153182273655) | > loss_mel_ce: 3.081458330154419 (nan) | > loss: 0.012386309914290905 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1364224341441471) | > loader_time: 0.0046 (0.004657879156057452)  --> TIME: 2025-02-25 05:15:41 -- STEP: 13454/36938 -- GLOBAL_STEP: 641400 | > loss_text_ce: 0.03403465077280998 (0.03559055882078948) | > loss_mel_ce: 2.3302595615386963 (nan) | > loss: 0.009382120333611965 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13642125337390076) | > loader_time: 0.0045 (0.004657654682877897)  --> TIME: 2025-02-25 05:15:50 -- STEP: 13504/36938 -- GLOBAL_STEP: 641450 | > loss_text_ce: 0.03704860806465149 (0.035591359973842975) | > loss_mel_ce: 3.0377938747406006 (nan) | > loss: 0.012201756238937378 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.13642936111626489) | > loader_time: 0.0046 (0.004657523468207401)  --> TIME: 2025-02-25 05:15:59 -- STEP: 13554/36938 -- GLOBAL_STEP: 641500 | > loss_text_ce: 0.03403211757540703 (0.035591594436063766) | > loss_mel_ce: 1.410386085510254 (nan) | > loss: 0.0057318187318742275 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13641025623565287) | > loader_time: 0.0044 (0.004657219429714089)  --> TIME: 2025-02-25 05:16:09 -- STEP: 13604/36938 -- GLOBAL_STEP: 641550 | > loss_text_ce: 0.03975224122405052 (0.0355901833484262) | > loss_mel_ce: 3.430363416671753 (nan) | > loss: 0.013770300894975662 (nan) | > current_lr: 5e-06 | > step_time: 0.1349 (0.13641139368070163) | > loader_time: 0.0047 (0.004657055044552739)  --> TIME: 2025-02-25 05:16:18 -- STEP: 13654/36938 -- GLOBAL_STEP: 641600 | > loss_text_ce: 0.040214672684669495 (0.03559179378459777) | > loss_mel_ce: 3.110628604888916 (nan) | > loss: 0.012503347359597683 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13639668461796992) | > loader_time: 0.0045 (0.004656736770765096)  --> TIME: 2025-02-25 05:16:27 -- STEP: 13704/36938 -- GLOBAL_STEP: 641650 | > loss_text_ce: 0.03679122030735016 (0.035584342008254055) | > loss_mel_ce: 3.168247699737549 (nan) | > loss: 0.012718409299850464 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13641109067448678) | > loader_time: 0.0044 (0.0046565788257073235)  --> TIME: 2025-02-25 05:16:37 -- STEP: 13754/36938 -- GLOBAL_STEP: 641700 | > loss_text_ce: 0.02601209655404091 (0.035584845854261964) | > loss_mel_ce: 1.5478057861328125 (nan) | > loss: 0.006245309486985207 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.1364295129011899) | > loader_time: 0.0045 (0.004656516883337545)  --> TIME: 2025-02-25 05:16:46 -- STEP: 13804/36938 -- GLOBAL_STEP: 641750 | > loss_text_ce: 0.032544661313295364 (0.035588453416406006) | > loss_mel_ce: 2.0128462314605713 (nan) | > loss: 0.008116630837321281 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13642591129210158) | > loader_time: 0.0045 (0.004656364540541142)  --> TIME: 2025-02-25 05:16:56 -- STEP: 13854/36938 -- GLOBAL_STEP: 641800 | > loss_text_ce: 0.0318264476954937 (0.03558131787814069) | > loss_mel_ce: 2.736456871032715 (nan) | > loss: 0.010985252447426319 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13644139524385254) | > loader_time: 0.0046 (0.004656268350142182)  --> TIME: 2025-02-25 05:17:05 -- STEP: 13904/36938 -- GLOBAL_STEP: 641850 | > loss_text_ce: 0.03797394037246704 (0.03557881890095195) | > loss_mel_ce: 3.313981056213379 (nan) | > loss: 0.013301408849656582 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13644571053625143) | > loader_time: 0.0045 (0.004656236074324712)  --> TIME: 2025-02-25 05:17:15 -- STEP: 13954/36938 -- GLOBAL_STEP: 641900 | > loss_text_ce: 0.03650140389800072 (0.03557720630722923) | > loss_mel_ce: 2.8686811923980713 (nan) | > loss: 0.011528503149747849 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.13645359677709404) | > loader_time: 0.0046 (0.00465605758537411)  --> TIME: 2025-02-25 05:17:24 -- STEP: 14004/36938 -- GLOBAL_STEP: 641950 | > loss_text_ce: 0.039720747619867325 (0.03557989744882353) | > loss_mel_ce: 0.0048220157623291016 (nan) | > loss: 0.000176757006556727 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13645697019672937) | > loader_time: 0.0045 (0.00465582585682088)  --> TIME: 2025-02-25 05:17:33 -- STEP: 14054/36938 -- GLOBAL_STEP: 642000 | > loss_text_ce: 0.04371457174420357 (0.03557652696570556) | > loss_mel_ce: 3.3512070178985596 (nan) | > loss: 0.013471911661326885 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13645267035317557) | > loader_time: 0.0046 (0.004655594945853029)  --> TIME: 2025-02-25 05:17:43 -- STEP: 14104/36938 -- GLOBAL_STEP: 642050 | > loss_text_ce: 0.027573194354772568 (0.03556667664999533) | > loss_mel_ce: 2.9251163005828857 (nan) | > loss: 0.011717022396624088 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.1364625347857444) | > loader_time: 0.0046 (0.004655528528340075)  --> TIME: 2025-02-25 05:17:52 -- STEP: 14154/36938 -- GLOBAL_STEP: 642100 | > loss_text_ce: 0.047395966947078705 (0.03556757517552705) | > loss_mel_ce: 1.6031086444854736 (nan) | > loss: 0.006549621932208538 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1364523524599295) | > loader_time: 0.0045 (0.004655320495809215)  --> TIME: 2025-02-25 05:18:01 -- STEP: 14204/36938 -- GLOBAL_STEP: 642150 | > loss_text_ce: 0.043078579008579254 (0.03556737549979583) | > loss_mel_ce: 2.8557965755462646 (nan) | > loss: 0.011503473855555058 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13647237242997093) | > loader_time: 0.0044 (0.0046552269937487805)  --> TIME: 2025-02-25 05:18:10 -- STEP: 14254/36938 -- GLOBAL_STEP: 642200 | > loss_text_ce: 0.03449038043618202 (0.035566930502566096) | > loss_mel_ce: 2.8689019680023193 (nan) | > loss: 0.011521399021148682 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13645590719876335) | > loader_time: 0.0046 (0.004655011375639197)  --> TIME: 2025-02-25 05:18:20 -- STEP: 14304/36938 -- GLOBAL_STEP: 642250 | > loss_text_ce: 0.03023179993033409 (0.03556305727967238) | > loss_mel_ce: 0.0038299560546875 (nan) | > loss: 0.00013516571198124439 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13647477368780478) | > loader_time: 0.0046 (0.004654874087580092)  --> TIME: 2025-02-25 05:18:29 -- STEP: 14354/36938 -- GLOBAL_STEP: 642300 | > loss_text_ce: 0.03941812366247177 (0.03555888747356491) | > loss_mel_ce: 3.6037392616271973 (nan) | > loss: 0.014456975273787975 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13647385731171333) | > loader_time: 0.0046 (0.004654696364081862)  --> TIME: 2025-02-25 05:18:39 -- STEP: 14404/36938 -- GLOBAL_STEP: 642350 | > loss_text_ce: 0.03161514922976494 (0.03556291450340545) | > loss_mel_ce: 3.3739798069000244 (nan) | > loss: 0.013514267280697823 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.13648507104122637) | > loader_time: 0.0048 (0.004654617549246856)  --> TIME: 2025-02-25 05:18:49 -- STEP: 14454/36938 -- GLOBAL_STEP: 642400 | > loss_text_ce: 0.03013753332197666 (0.03556228041111723) | > loss_mel_ce: 2.7487053871154785 (nan) | > loss: 0.011027155444025993 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13651511814974707) | > loader_time: 0.0045 (0.004654611230836754)  --> TIME: 2025-02-25 05:18:58 -- STEP: 14504/36938 -- GLOBAL_STEP: 642450 | > loss_text_ce: 0.024618875235319138 (0.03556129482206934) | > loss_mel_ce: 2.653979539871216 (nan) | > loss: 0.010629359632730484 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13650631247077083) | > loader_time: 0.0046 (0.004654455106033513)  --> TIME: 2025-02-25 05:19:07 -- STEP: 14554/36938 -- GLOBAL_STEP: 642500 | > loss_text_ce: 0.04161924868822098 (0.03555717242384295) | > loss_mel_ce: 2.5545735359191895 (nan) | > loss: 0.010302352719008923 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13650385649040178) | > loader_time: 0.0046 (0.004654290012004721)  --> TIME: 2025-02-25 05:19:16 -- STEP: 14604/36938 -- GLOBAL_STEP: 642550 | > loss_text_ce: 0.03797605261206627 (0.03555755946383171) | > loss_mel_ce: 3.505173683166504 (nan) | > loss: 0.014060119166970253 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13650773672560032) | > loader_time: 0.0046 (0.004654155614048898)  --> TIME: 2025-02-25 05:19:25 -- STEP: 14654/36938 -- GLOBAL_STEP: 642600 | > loss_text_ce: 0.032815515995025635 (0.03555832783178934) | > loss_mel_ce: 2.947042465209961 (nan) | > loss: 0.011824834160506725 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13650461357385424) | > loader_time: 0.0046 (0.004654028494751462)  --> TIME: 2025-02-25 05:19:35 -- STEP: 14704/36938 -- GLOBAL_STEP: 642650 | > loss_text_ce: 0.02884749136865139 (0.035558283856488994) | > loss_mel_ce: 3.2473032474517822 (nan) | > loss: 0.01300059910863638 (nan) | > current_lr: 5e-06 | > step_time: 0.1812 (0.13651512843740207) | > loader_time: 0.0048 (0.004654058758919329)  --> TIME: 2025-02-25 05:19:45 -- STEP: 14754/36938 -- GLOBAL_STEP: 642700 | > loss_text_ce: 0.032913077622652054 (0.03555572893742186) | > loss_mel_ce: 2.8065946102142334 (nan) | > loss: 0.0112678874284029 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.13652973761965187) | > loader_time: 0.0047 (0.004653974892858143)  --> TIME: 2025-02-25 05:19:54 -- STEP: 14804/36938 -- GLOBAL_STEP: 642750 | > loss_text_ce: 0.028599653393030167 (0.03556012476172091) | > loss_mel_ce: 2.9115149974823 (nan) | > loss: 0.011667123064398766 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.1365353846285874) | > loader_time: 0.0045 (0.004653929697761849)  --> TIME: 2025-02-25 05:20:03 -- STEP: 14854/36938 -- GLOBAL_STEP: 642800 | > loss_text_ce: 0.031087636947631836 (0.035555725728685685) | > loss_mel_ce: 2.6436331272125244 (nan) | > loss: 0.010613972321152687 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.136530579845855) | > loader_time: 0.0045 (0.004654014240576657)  --> TIME: 2025-02-25 05:20:13 -- STEP: 14904/36938 -- GLOBAL_STEP: 642850 | > loss_text_ce: 0.032261162996292114 (0.0355541296528852) | > loss_mel_ce: 2.6276016235351562 (nan) | > loss: 0.010555011220276356 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.1365263133649223) | > loader_time: 0.0045 (0.004653931207958877)  --> TIME: 2025-02-25 05:20:22 -- STEP: 14954/36938 -- GLOBAL_STEP: 642900 | > loss_text_ce: 0.039064355194568634 (0.035552985749909845) | > loss_mel_ce: 1.2399592399597168 (nan) | > loss: 0.0050754910334944725 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13651356565389566) | > loader_time: 0.0046 (0.004653913285685389)  --> TIME: 2025-02-25 05:20:31 -- STEP: 15004/36938 -- GLOBAL_STEP: 642950 | > loss_text_ce: 0.04680107533931732 (0.035549820202952) | > loss_mel_ce: 0.078460693359375 (nan) | > loss: 0.0004970705485902727 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13650586544053203) | > loader_time: 0.0045 (0.004653837371908541)  --> TIME: 2025-02-25 05:20:40 -- STEP: 15054/36938 -- GLOBAL_STEP: 643000 | > loss_text_ce: 0.02539457008242607 (0.03555202286278922) | > loss_mel_ce: 2.361645460128784 (nan) | > loss: 0.009472382254898548 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.1364988538077655) | > loader_time: 0.0045 (0.004653839740649274)  --> TIME: 2025-02-25 05:20:49 -- STEP: 15104/36938 -- GLOBAL_STEP: 643050 | > loss_text_ce: 0.03569009155035019 (0.03555195285736164) | > loss_mel_ce: 2.7770609855651855 (nan) | > loss: 0.011161711066961288 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13650504423905294) | > loader_time: 0.0046 (0.004653792875676836)  --> TIME: 2025-02-25 05:20:59 -- STEP: 15154/36938 -- GLOBAL_STEP: 643100 | > loss_text_ce: 0.03467169776558876 (0.03554851158103032) | > loss_mel_ce: 2.767643928527832 (nan) | > loss: 0.011120300740003586 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13650305919035582) | > loader_time: 0.0046 (0.004653749482304745)  --> TIME: 2025-02-25 05:21:08 -- STEP: 15204/36938 -- GLOBAL_STEP: 643150 | > loss_text_ce: 0.028796019032597542 (0.0355550171560548) | > loss_mel_ce: 2.4331588745117188 (nan) | > loss: 0.009769662283360958 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13650827254473805) | > loader_time: 0.0045 (0.004653611376987442)  --> TIME: 2025-02-25 05:21:18 -- STEP: 15254/36938 -- GLOBAL_STEP: 643200 | > loss_text_ce: 0.032784946262836456 (0.035553885165111355) | > loss_mel_ce: 2.5751330852508545 (nan) | > loss: 0.010348881594836712 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13652038314762382) | > loader_time: 0.0046 (0.004653636352941183)  --> TIME: 2025-02-25 05:21:27 -- STEP: 15304/36938 -- GLOBAL_STEP: 643250 | > loss_text_ce: 0.03998124599456787 (0.0355567641204398) | > loss_mel_ce: 3.2849783897399902 (nan) | > loss: 0.013194285333156586 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13650827250802264) | > loader_time: 0.0045 (0.004653476494117607)  --> TIME: 2025-02-25 05:21:36 -- STEP: 15354/36938 -- GLOBAL_STEP: 643300 | > loss_text_ce: 0.034606680274009705 (0.035557371169273634) | > loss_mel_ce: 2.3686676025390625 (nan) | > loss: 0.00953680370002985 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.1365273527386637) | > loader_time: 0.0046 (0.0046533878479709445)  --> TIME: 2025-02-25 05:21:45 -- STEP: 15404/36938 -- GLOBAL_STEP: 643350 | > loss_text_ce: 0.03597487136721611 (0.03556479828223473) | > loss_mel_ce: 2.688929319381714 (nan) | > loss: 0.010813113301992416 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13651401689844486) | > loader_time: 0.0046 (0.004653188817626741)  --> TIME: 2025-02-25 05:21:55 -- STEP: 15454/36938 -- GLOBAL_STEP: 643400 | > loss_text_ce: 0.031079590320587158 (0.035563023212088735) | > loss_mel_ce: 2.7194924354553223 (nan) | > loss: 0.010914968326687813 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.1365298630735528) | > loader_time: 0.0046 (0.004653066824829957)  --> TIME: 2025-02-25 05:22:04 -- STEP: 15504/36938 -- GLOBAL_STEP: 643450 | > loss_text_ce: 0.03086589090526104 (0.03556582144413735) | > loss_mel_ce: 2.7435247898101807 (nan) | > loss: 0.011009487323462963 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13653334827615077) | > loader_time: 0.0045 (0.004652906020851445)  --> TIME: 2025-02-25 05:22:13 -- STEP: 15554/36938 -- GLOBAL_STEP: 643500 | > loss_text_ce: 0.028886372223496437 (0.03556715209527793) | > loss_mel_ce: 2.7628941535949707 (nan) | > loss: 0.011078494600951672 (nan) | > current_lr: 5e-06 | > step_time: 0.1529 (0.1365289235001627) | > loader_time: 0.0046 (0.0046527405025506155)  --> TIME: 2025-02-25 05:22:23 -- STEP: 15604/36938 -- GLOBAL_STEP: 643550 | > loss_text_ce: 0.03129824250936508 (0.035563853749815584) | > loss_mel_ce: 2.6738295555114746 (nan) | > loss: 0.010734634473919868 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13653213653586518) | > loader_time: 0.0045 (0.004652722390362368)  --> TIME: 2025-02-25 05:22:32 -- STEP: 15654/36938 -- GLOBAL_STEP: 643600 | > loss_text_ce: 0.019811710342764854 (0.035559481751019785) | > loss_mel_ce: 2.2795138359069824 (nan) | > loss: 0.00912430789321661 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.13653110623192208) | > loader_time: 0.0046 (0.004653058259807292)  --> TIME: 2025-02-25 05:22:42 -- STEP: 15704/36938 -- GLOBAL_STEP: 643650 | > loss_text_ce: 0.045422300696372986 (0.035559072004800796) | > loss_mel_ce: 3.094663619995117 (nan) | > loss: 0.012460659258067608 (nan) | > current_lr: 5e-06 | > step_time: 0.1793 (0.13653246508043093) | > loader_time: 0.0049 (0.004653165990821689)  --> TIME: 2025-02-25 05:22:51 -- STEP: 15754/36938 -- GLOBAL_STEP: 643700 | > loss_text_ce: 0.03614761307835579 (0.03556021120859378) | > loss_mel_ce: 0.7804107666015625 (nan) | > loss: 0.0032403110526502132 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.1365170541020906) | > loader_time: 0.0046 (0.004652931058529409)  --> TIME: 2025-02-25 05:23:00 -- STEP: 15804/36938 -- GLOBAL_STEP: 643750 | > loss_text_ce: 0.04258062317967415 (0.03555906265613375) | > loss_mel_ce: 2.9962942600250244 (nan) | > loss: 0.012059028260409832 (nan) | > current_lr: 5e-06 | > step_time: 0.1778 (0.1365191043102123) | > loader_time: 0.0048 (0.004652969612709567)  --> TIME: 2025-02-25 05:23:09 -- STEP: 15854/36938 -- GLOBAL_STEP: 643800 | > loss_text_ce: 0.03239399939775467 (0.03555777004726188) | > loss_mel_ce: 3.17098331451416 (nan) | > loss: 0.012711815536022186 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.1365312879708437) | > loader_time: 0.0045 (0.004653093853047349)  --> TIME: 2025-02-25 05:23:19 -- STEP: 15904/36938 -- GLOBAL_STEP: 643850 | > loss_text_ce: 0.04234709218144417 (0.03555344933443878) | > loss_mel_ce: 2.2659573554992676 (nan) | > loss: 0.009159939363598824 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.13654787103894794) | > loader_time: 0.0048 (0.004653199472777555)  --> TIME: 2025-02-25 05:23:28 -- STEP: 15954/36938 -- GLOBAL_STEP: 643900 | > loss_text_ce: 0.04832941293716431 (0.03555463784232656) | > loss_mel_ce: 1.6329498291015625 (nan) | > loss: 0.006671743001788855 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13654678346400367) | > loader_time: 0.0046 (0.004653272733990819)  --> TIME: 2025-02-25 05:23:38 -- STEP: 16004/36938 -- GLOBAL_STEP: 643950 | > loss_text_ce: 0.035378411412239075 (0.0355510307235934) | > loss_mel_ce: 1.8908507823944092 (nan) | > loss: 0.007643767166882753 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.1365688115887218) | > loader_time: 0.0047 (0.004653258685379195)  --> TIME: 2025-02-25 05:23:47 -- STEP: 16054/36938 -- GLOBAL_STEP: 644000 | > loss_text_ce: 0.03936523199081421 (0.03555100538582331) | > loss_mel_ce: 2.9069724082946777 (nan) | > loss: 0.011691817082464695 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13656138354998087) | > loader_time: 0.0045 (0.00465312149035495)  --> TIME: 2025-02-25 05:23:57 -- STEP: 16104/36938 -- GLOBAL_STEP: 644050 | > loss_text_ce: 0.028980683535337448 (0.03555679117408974) | > loss_mel_ce: 3.087169647216797 (nan) | > loss: 0.012365676462650299 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13655973269876712) | > loader_time: 0.0046 (0.0046531813421832055)  --> TIME: 2025-02-25 05:24:06 -- STEP: 16154/36938 -- GLOBAL_STEP: 644100 | > loss_text_ce: 0.030207190662622452 (0.03555257128367323) | > loss_mel_ce: 2.9963033199310303 (nan) | > loss: 0.01200996246188879 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.1365590442949353) | > loader_time: 0.0045 (0.004653391322095421)  --> TIME: 2025-02-25 05:24:15 -- STEP: 16204/36938 -- GLOBAL_STEP: 644150 | > loss_text_ce: 0.03230460733175278 (0.03555209062059194) | > loss_mel_ce: 2.564826726913452 (nan) | > loss: 0.01030607707798481 (nan) | > current_lr: 5e-06 | > step_time: 0.1264 (0.13654580821934528) | > loader_time: 0.0046 (0.004653304881503388)  --> TIME: 2025-02-25 05:24:24 -- STEP: 16254/36938 -- GLOBAL_STEP: 644200 | > loss_text_ce: 0.038966186344623566 (0.03555244734226331) | > loss_mel_ce: 2.0412192344665527 (nan) | > loss: 0.0082547040656209 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13655087675601837) | > loader_time: 0.0045 (0.004653308405359244)  --> TIME: 2025-02-25 05:24:34 -- STEP: 16304/36938 -- GLOBAL_STEP: 644250 | > loss_text_ce: 0.03635522723197937 (0.03555136906694585) | > loss_mel_ce: 2.4849696159362793 (nan) | > loss: 0.010005258023738861 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.1365501975697326) | > loader_time: 0.0047 (0.004653372155672429)  --> TIME: 2025-02-25 05:24:43 -- STEP: 16354/36938 -- GLOBAL_STEP: 644300 | > loss_text_ce: 0.04430624842643738 (0.035547352878412396) | > loss_mel_ce: 2.783154249191284 (nan) | > loss: 0.011220081709325314 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13656500968487645) | > loader_time: 0.0046 (0.004653465344005765)  --> TIME: 2025-02-25 05:24:53 -- STEP: 16404/36938 -- GLOBAL_STEP: 644350 | > loss_text_ce: 0.027271350845694542 (0.03554680477589316) | > loss_mel_ce: 0.0116424560546875 (nan) | > loss: 0.00015441988944076002 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13656546784098458) | > loader_time: 0.0046 (0.00465351699242153)  --> TIME: 2025-02-25 05:25:02 -- STEP: 16454/36938 -- GLOBAL_STEP: 644400 | > loss_text_ce: 0.04329974204301834 (0.03554282329717982) | > loss_mel_ce: 4.045572757720947 (nan) | > loss: 0.01622568443417549 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.13656414173868633) | > loader_time: 0.0044 (0.004653392273353542)  --> TIME: 2025-02-25 05:25:11 -- STEP: 16504/36938 -- GLOBAL_STEP: 644450 | > loss_text_ce: 0.04797983914613724 (0.03554392319524272) | > loss_mel_ce: 3.5180788040161133 (nan) | > loss: 0.014151027426123619 (nan) | > current_lr: 5e-06 | > step_time: 0.1829 (0.13656480281721722) | > loader_time: 0.0048 (0.004653378432662632)  --> TIME: 2025-02-25 05:25:21 -- STEP: 16554/36938 -- GLOBAL_STEP: 644500 | > loss_text_ce: 0.03486783057451248 (0.035548790600260886) | > loss_mel_ce: 3.2414374351501465 (nan) | > loss: 0.013001211918890476 (nan) | > current_lr: 5e-06 | > step_time: 0.1825 (0.13656392549134835) | > loader_time: 0.0049 (0.00465337288499209)  --> TIME: 2025-02-25 05:25:30 -- STEP: 16604/36938 -- GLOBAL_STEP: 644550 | > loss_text_ce: 0.035587869584560394 (0.03555300204571064) | > loss_mel_ce: 2.882521390914917 (nan) | > loss: 0.01157979853451252 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13658950355247718) | > loader_time: 0.0046 (0.004653561347882387)  --> TIME: 2025-02-25 05:25:40 -- STEP: 16654/36938 -- GLOBAL_STEP: 644600 | > loss_text_ce: 0.03876223787665367 (0.03555275272789142) | > loss_mel_ce: 3.101078748703003 (nan) | > loss: 0.012459687888622284 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13659495425221505) | > loader_time: 0.0047 (0.004653624974623347)  --> TIME: 2025-02-25 05:25:49 -- STEP: 16704/36938 -- GLOBAL_STEP: 644650 | > loss_text_ce: 0.03815989941358566 (0.03555036554718728) | > loss_mel_ce: 3.01265811920166 (nan) | > loss: 0.012106421403586864 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13658990652453917) | > loader_time: 0.0046 (0.004653596387055636)  --> TIME: 2025-02-25 05:25:59 -- STEP: 16754/36938 -- GLOBAL_STEP: 644700 | > loss_text_ce: 0.03847157210111618 (0.03555197953135893) | > loss_mel_ce: 3.2770066261291504 (nan) | > loss: 0.01315665990114212 (nan) | > current_lr: 5e-06 | > step_time: 0.2037 (0.13660222501847033) | > loader_time: 0.0049 (0.004653718500842019)  --> TIME: 2025-02-25 05:26:08 -- STEP: 16804/36938 -- GLOBAL_STEP: 644750 | > loss_text_ce: 0.04170724377036095 (0.035554490222089306) | > loss_mel_ce: 4.942708492279053 (nan) | > loss: 0.019779426977038383 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13658878527150775) | > loader_time: 0.0048 (0.0046536637271934606)  --> TIME: 2025-02-25 05:26:17 -- STEP: 16854/36938 -- GLOBAL_STEP: 644800 | > loss_text_ce: 0.03014952316880226 (0.035549975371767635) | > loss_mel_ce: 3.475151300430298 (nan) | > loss: 0.013909924775362015 (nan) | > current_lr: 5e-06 | > step_time: 0.1881 (0.13658128758547441) | > loader_time: 0.0048 (0.004653639650237329)  --> TIME: 2025-02-25 05:26:26 -- STEP: 16904/36938 -- GLOBAL_STEP: 644850 | > loss_text_ce: 0.03421500697731972 (0.03554845636536848) | > loss_mel_ce: 2.5030925273895264 (nan) | > loss: 0.010068681091070175 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13656448937116897) | > loader_time: 0.0046 (0.004653576294280194)  --> TIME: 2025-02-25 05:26:35 -- STEP: 16954/36938 -- GLOBAL_STEP: 644900 | > loss_text_ce: 0.027444519102573395 (0.03555206608136407) | > loss_mel_ce: 2.142388105392456 (nan) | > loss: 0.008610447868704796 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13656115111447484) | > loader_time: 0.0045 (0.004653580320659095)  --> TIME: 2025-02-25 05:26:44 -- STEP: 17004/36938 -- GLOBAL_STEP: 644950 | > loss_text_ce: 0.04621679335832596 (0.0355511258780307) | > loss_mel_ce: 3.4074294567108154 (nan) | > loss: 0.013704946264624596 (nan) | > current_lr: 5e-06 | > step_time: 0.1514 (0.13654670522398682) | > loader_time: 0.0047 (0.004653478322155857)  --> TIME: 2025-02-25 05:26:53 -- STEP: 17054/36938 -- GLOBAL_STEP: 645000 | > loss_text_ce: 0.03614463657140732 (0.035548928183426504) | > loss_mel_ce: 2.6691839694976807 (nan) | > loss: 0.01073543168604374 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13653725740064385) | > loader_time: 0.0046 (0.00465342768390306)  --> TIME: 2025-02-25 05:27:02 -- STEP: 17104/36938 -- GLOBAL_STEP: 645050 | > loss_text_ce: 0.032300274819135666 (0.03554807315920083) | > loss_mel_ce: 3.05918550491333 (nan) | > loss: 0.012267800979316235 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.1365296665031967) | > loader_time: 0.0046 (0.0046537583598028846)  --> TIME: 2025-02-25 05:27:12 -- STEP: 17154/36938 -- GLOBAL_STEP: 645100 | > loss_text_ce: 0.04071902483701706 (0.03554809782683347) | > loss_mel_ce: 3.1312663555145264 (nan) | > loss: 0.012587244622409344 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13653403304459125) | > loader_time: 0.0046 (0.004653672384217822)  --> TIME: 2025-02-25 05:27:21 -- STEP: 17204/36938 -- GLOBAL_STEP: 645150 | > loss_text_ce: 0.013273444026708603 (0.035548995578270744) | > loss_mel_ce: 2.202176570892334 (nan) | > loss: 0.008791469037532806 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.1365165106642116) | > loader_time: 0.0044 (0.004653538307237826)  --> TIME: 2025-02-25 05:27:30 -- STEP: 17254/36938 -- GLOBAL_STEP: 645200 | > loss_text_ce: 0.034979622811079025 (0.03554926923501725) | > loss_mel_ce: 0.07106590270996094 (nan) | > loss: 0.0004208156024105847 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.1365151299072807) | > loader_time: 0.0045 (0.004653474471239351)  --> TIME: 2025-02-25 05:27:39 -- STEP: 17304/36938 -- GLOBAL_STEP: 645250 | > loss_text_ce: 0.03264288604259491 (0.03554687367176718) | > loss_mel_ce: 3.1700592041015625 (nan) | > loss: 0.012709136120975018 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.1365179390996575) | > loader_time: 0.0047 (0.004653436424993436)  --> TIME: 2025-02-25 05:27:49 -- STEP: 17354/36938 -- GLOBAL_STEP: 645300 | > loss_text_ce: 0.040188174694776535 (0.035543687059610904) | > loss_mel_ce: 2.8508002758026123 (nan) | > loss: 0.011472176760435104 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.1365138635089136) | > loader_time: 0.0048 (0.0046533996421125445)  --> TIME: 2025-02-25 05:27:58 -- STEP: 17404/36938 -- GLOBAL_STEP: 645350 | > loss_text_ce: 0.031706299632787704 (0.03554475223242067) | > loss_mel_ce: 2.9783825874328613 (nan) | > loss: 0.011944797821342945 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.1365035561768938) | > loader_time: 0.0046 (0.00465316214634889)  --> TIME: 2025-02-25 05:28:07 -- STEP: 17454/36938 -- GLOBAL_STEP: 645400 | > loss_text_ce: 0.03318442404270172 (0.03554468598500409) | > loss_mel_ce: 2.2918527126312256 (nan) | > loss: 0.009226338937878609 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13650841521440957) | > loader_time: 0.0045 (0.004653064153087059)  --> TIME: 2025-02-25 05:28:17 -- STEP: 17504/36938 -- GLOBAL_STEP: 645450 | > loss_text_ce: 0.033420853316783905 (0.03554865937826625) | > loss_mel_ce: 3.226276397705078 (nan) | > loss: 0.012935306876897812 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13651898714426056) | > loader_time: 0.0045 (0.0046530000225281564)  --> TIME: 2025-02-25 05:28:26 -- STEP: 17554/36938 -- GLOBAL_STEP: 645500 | > loss_text_ce: 0.037094954401254654 (0.03554566171951043) | > loss_mel_ce: 3.264277935028076 (nan) | > loss: 0.01310068741440773 (nan) | > current_lr: 5e-06 | > step_time: 0.1578 (0.136511535569559) | > loader_time: 0.0046 (0.00465285859537706)  --> TIME: 2025-02-25 05:28:35 -- STEP: 17604/36938 -- GLOBAL_STEP: 645550 | > loss_text_ce: 0.02501852437853813 (0.03554715321258951) | > loss_mel_ce: 2.56390643119812 (nan) | > loss: 0.010273512452840805 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13651784242106954) | > loader_time: 0.0045 (0.004652669052729904)  --> TIME: 2025-02-25 05:28:44 -- STEP: 17654/36938 -- GLOBAL_STEP: 645600 | > loss_text_ce: 0.03704023361206055 (0.03554801853874584) | > loss_mel_ce: 2.540796995162964 (nan) | > loss: 0.010229513049125671 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13651584655171436) | > loader_time: 0.0045 (0.0046526557985675085)  --> TIME: 2025-02-25 05:28:54 -- STEP: 17704/36938 -- GLOBAL_STEP: 645650 | > loss_text_ce: 0.03592650964856148 (0.035548423476093265) | > loss_mel_ce: 2.2731478214263916 (nan) | > loss: 0.009162994101643562 (nan) | > current_lr: 5e-06 | > step_time: 0.1778 (0.13653429519844712) | > loader_time: 0.0049 (0.004652770945673555)  --> TIME: 2025-02-25 05:29:03 -- STEP: 17754/36938 -- GLOBAL_STEP: 645700 | > loss_text_ce: 0.04480941593647003 (0.03554199582450075) | > loss_mel_ce: 2.7249088287353516 (nan) | > loss: 0.010990945622324944 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.13652897763389282) | > loader_time: 0.0045 (0.004652718293367375)  --> TIME: 2025-02-25 05:29:13 -- STEP: 17804/36938 -- GLOBAL_STEP: 645750 | > loss_text_ce: 0.04136344790458679 (0.03554172383576753) | > loss_mel_ce: 2.707662343978882 (nan) | > loss: 0.01090883370488882 (nan) | > current_lr: 5e-06 | > step_time: 0.1886 (0.1365373199694726) | > loader_time: 0.0048 (0.004652635150211792)  --> TIME: 2025-02-25 05:29:22 -- STEP: 17854/36938 -- GLOBAL_STEP: 645800 | > loss_text_ce: 0.04035992920398712 (0.0355384453034758) | > loss_mel_ce: 3.697801351547241 (nan) | > loss: 0.014833974651992321 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13653568583452522) | > loader_time: 0.0046 (0.00465262599871142)  --> TIME: 2025-02-25 05:29:31 -- STEP: 17904/36938 -- GLOBAL_STEP: 645850 | > loss_text_ce: 0.03267023339867592 (0.03553407548837703) | > loss_mel_ce: 3.0786774158477783 (nan) | > loss: 0.012346618808805943 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13652274621129626) | > loader_time: 0.0045 (0.004652588946996151)  --> TIME: 2025-02-25 05:29:41 -- STEP: 17954/36938 -- GLOBAL_STEP: 645900 | > loss_text_ce: 0.030817613005638123 (0.035535327291247344) | > loss_mel_ce: 2.4073140621185303 (nan) | > loss: 0.00967512559145689 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13652928439628317) | > loader_time: 0.0046 (0.004652452426377952)  --> TIME: 2025-02-25 05:29:50 -- STEP: 18004/36938 -- GLOBAL_STEP: 645950 | > loss_text_ce: 0.0387955978512764 (0.03553597113325309) | > loss_mel_ce: 0.03612598776817322 (nan) | > loss: 0.0002973079099319875 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1365277134850412) | > loader_time: 0.0045 (0.004652253536880351)  --> TIME: 2025-02-25 05:29:59 -- STEP: 18054/36938 -- GLOBAL_STEP: 646000 | > loss_text_ce: 0.0302626620978117 (0.03553526624787512) | > loss_mel_ce: 2.022873640060425 (nan) | > loss: 0.008147367276251316 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13652561663577703) | > loader_time: 0.0045 (0.004652167576233841)  --> TIME: 2025-02-25 05:30:08 -- STEP: 18104/36938 -- GLOBAL_STEP: 646050 | > loss_text_ce: 0.0390150211751461 (0.03553489205961234) | > loss_mel_ce: 3.0073111057281494 (nan) | > loss: 0.012088596820831299 (nan) | > current_lr: 5e-06 | > step_time: 0.1699 (0.1365181432589992) | > loader_time: 0.0048 (0.004652122112169661)  --> TIME: 2025-02-25 05:30:18 -- STEP: 18154/36938 -- GLOBAL_STEP: 646100 | > loss_text_ce: 0.042009562253952026 (0.03553362133142909) | > loss_mel_ce: 3.3616700172424316 (nan) | > loss: 0.013506665825843811 (nan) | > current_lr: 5e-06 | > step_time: 0.186 (0.13653171709932124) | > loader_time: 0.0048 (0.004652144560352146)  --> TIME: 2025-02-25 05:30:27 -- STEP: 18204/36938 -- GLOBAL_STEP: 646150 | > loss_text_ce: 0.038417961448431015 (0.035530445122241176) | > loss_mel_ce: 3.112166404724121 (nan) | > loss: 0.012502320110797882 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.1365413026766739) | > loader_time: 0.0046 (0.00465209293931271)  --> TIME: 2025-02-25 05:30:37 -- STEP: 18254/36938 -- GLOBAL_STEP: 646200 | > loss_text_ce: 0.04021989554166794 (0.03552845027848746) | > loss_mel_ce: 0.0346272774040699 (nan) | > loss: 0.0002970126224681735 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.136538204650988) | > loader_time: 0.0046 (0.004651965206288766)  --> TIME: 2025-02-25 05:30:46 -- STEP: 18304/36938 -- GLOBAL_STEP: 646250 | > loss_text_ce: 0.031220274046063423 (0.035528216786836) | > loss_mel_ce: 2.6186797618865967 (nan) | > loss: 0.010515476576983929 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13653491256380876) | > loader_time: 0.0046 (0.004651798391258804)  --> TIME: 2025-02-25 05:30:55 -- STEP: 18354/36938 -- GLOBAL_STEP: 646300 | > loss_text_ce: 0.03606881573796272 (0.035529543257958644) | > loss_mel_ce: 1.8265554904937744 (nan) | > loss: 0.007391366641968489 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13652737908105142) | > loader_time: 0.0044 (0.004651657828607113)  --> TIME: 2025-02-25 05:31:04 -- STEP: 18404/36938 -- GLOBAL_STEP: 646350 | > loss_text_ce: 0.03441024199128151 (0.03552719072928214) | > loss_mel_ce: 2.3785018920898438 (nan) | > loss: 0.009575048461556435 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13652735764605697) | > loader_time: 0.0046 (0.0046515603916356466)  --> TIME: 2025-02-25 05:31:13 -- STEP: 18454/36938 -- GLOBAL_STEP: 646400 | > loss_text_ce: 0.050205592066049576 (0.03552844371511178) | > loss_mel_ce: 2.8925678730010986 (nan) | > loss: 0.011677674017846584 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13651744375201014) | > loader_time: 0.0045 (0.004651347722911084)  --> TIME: 2025-02-25 05:31:22 -- STEP: 18504/36938 -- GLOBAL_STEP: 646450 | > loss_text_ce: 0.0392938107252121 (0.035531603283119106) | > loss_mel_ce: 3.0278685092926025 (nan) | > loss: 0.012171279639005661 (nan) | > current_lr: 5e-06 | > step_time: 0.1692 (0.13650467070572564) | > loader_time: 0.0048 (0.004651253505851924)  --> TIME: 2025-02-25 05:31:32 -- STEP: 18554/36938 -- GLOBAL_STEP: 646500 | > loss_text_ce: 0.0268937349319458 (0.03553170675248778) | > loss_mel_ce: 2.2099318504333496 (nan) | > loss: 0.008876292034983635 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13650194131391308) | > loader_time: 0.0048 (0.0046512402888836)  --> TIME: 2025-02-25 05:31:41 -- STEP: 18604/36938 -- GLOBAL_STEP: 646550 | > loss_text_ce: 0.04555671662092209 (0.03553502188135035) | > loss_mel_ce: 2.732618570327759 (nan) | > loss: 0.011024505831301212 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.13650451352893125) | > loader_time: 0.0047 (0.004651182122293067)  --> TIME: 2025-02-25 05:31:50 -- STEP: 18654/36938 -- GLOBAL_STEP: 646600 | > loss_text_ce: 0.04564197361469269 (0.035540074354419623) | > loss_mel_ce: 3.1070051193237305 (nan) | > loss: 0.012510504573583603 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13649521213233773) | > loader_time: 0.0046 (0.004651096315262233)  --> TIME: 2025-02-25 05:32:00 -- STEP: 18704/36938 -- GLOBAL_STEP: 646650 | > loss_text_ce: 0.03229983150959015 (0.03554285212352802) | > loss_mel_ce: 2.865234375 (nan) | > loss: 0.011498152278363705 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13649978002388344) | > loader_time: 0.0045 (0.0046514895122412225)  --> TIME: 2025-02-25 05:32:09 -- STEP: 18754/36938 -- GLOBAL_STEP: 646700 | > loss_text_ce: 0.03704488277435303 (0.035542013507353816) | > loss_mel_ce: 2.6228926181793213 (nan) | > loss: 0.010555307380855083 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13650329476342593) | > loader_time: 0.0048 (0.004651549948905596)  --> TIME: 2025-02-25 05:32:18 -- STEP: 18804/36938 -- GLOBAL_STEP: 646750 | > loss_text_ce: 0.03586340695619583 (0.035538305099896986) | > loss_mel_ce: 2.8137807846069336 (nan) | > loss: 0.011308112181723118 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13649144084675244) | > loader_time: 0.0046 (0.004651376248116045)  --> TIME: 2025-02-25 05:32:27 -- STEP: 18854/36938 -- GLOBAL_STEP: 646800 | > loss_text_ce: 0.024582425132393837 (0.03553594514257131) | > loss_mel_ce: 1.5408624410629272 (nan) | > loss: 0.0062120831571519375 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13649290741292974) | > loader_time: 0.0045 (0.004651235385905708)  --> TIME: 2025-02-25 05:32:37 -- STEP: 18904/36938 -- GLOBAL_STEP: 646850 | > loss_text_ce: 0.034274596720933914 (0.035537458431017736) | > loss_mel_ce: 2.9254963397979736 (nan) | > loss: 0.011745123192667961 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13648746878072687) | > loader_time: 0.0045 (0.004651158682330411)  --> TIME: 2025-02-25 05:32:46 -- STEP: 18954/36938 -- GLOBAL_STEP: 646900 | > loss_text_ce: 0.0304340161383152 (0.035535438398652516) | > loss_mel_ce: 2.0684421062469482 (nan) | > loss: 0.008328874595463276 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13649245569873789) | > loader_time: 0.0046 (0.004651288763812876)  --> TIME: 2025-02-25 05:32:55 -- STEP: 19004/36938 -- GLOBAL_STEP: 646950 | > loss_text_ce: 0.03535471111536026 (0.035534650935220694) | > loss_mel_ce: 3.1455399990081787 (nan) | > loss: 0.012622598558664322 (nan) | > current_lr: 5e-06 | > step_time: 0.1766 (0.1364798851827151) | > loader_time: 0.005 (0.004651219599925965)  --> TIME: 2025-02-25 05:33:04 -- STEP: 19054/36938 -- GLOBAL_STEP: 647000 | > loss_text_ce: 0.049534983932971954 (0.03553218284625343) | > loss_mel_ce: 0.107177734375 (nan) | > loss: 0.0006218758644536138 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1364654684862587) | > loader_time: 0.0046 (0.004651387490842818)  --> TIME: 2025-02-25 05:33:13 -- STEP: 19104/36938 -- GLOBAL_STEP: 647050 | > loss_text_ce: 0.038244448602199554 (0.03553098949360562) | > loss_mel_ce: 3.257157325744629 (nan) | > loss: 0.013076991774141788 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.13646201459906213) | > loader_time: 0.0047 (0.00465133435512509)  --> TIME: 2025-02-25 05:33:23 -- STEP: 19154/36938 -- GLOBAL_STEP: 647100 | > loss_text_ce: 0.04282684251666069 (0.03552933978974863) | > loss_mel_ce: 2.961944580078125 (nan) | > loss: 0.011923696845769882 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.1364525444786172) | > loader_time: 0.0046 (0.004651273704712847)  --> TIME: 2025-02-25 05:33:32 -- STEP: 19204/36938 -- GLOBAL_STEP: 647150 | > loss_text_ce: 0.03236444666981697 (0.03552994225111138) | > loss_mel_ce: 3.0161337852478027 (nan) | > loss: 0.01209721527993679 (nan) | > current_lr: 5e-06 | > step_time: 0.1811 (0.13646805590576133) | > loader_time: 0.0048 (0.004651540494218437)  --> TIME: 2025-02-25 05:33:41 -- STEP: 19254/36938 -- GLOBAL_STEP: 647200 | > loss_text_ce: 0.03202824294567108 (0.03553141973135109) | > loss_mel_ce: 2.741806745529175 (nan) | > loss: 0.011007281951606274 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.13646164113293677) | > loader_time: 0.0046 (0.004651524114920836)  --> TIME: 2025-02-25 05:33:51 -- STEP: 19304/36938 -- GLOBAL_STEP: 647250 | > loss_text_ce: 0.034512270241975784 (0.03553635494970127) | > loss_mel_ce: 2.7556116580963135 (nan) | > loss: 0.011071921326220036 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13645896421790948) | > loader_time: 0.0046 (0.00465151526796539)  --> TIME: 2025-02-25 05:34:00 -- STEP: 19354/36938 -- GLOBAL_STEP: 647300 | > loss_text_ce: 0.029124727472662926 (0.03553495101941156) | > loss_mel_ce: 3.1336729526519775 (nan) | > loss: 0.012550785206258297 (nan) | > current_lr: 5e-06 | > step_time: 0.1758 (0.1364595380661667) | > loader_time: 0.0047 (0.004651422723334771)  --> TIME: 2025-02-25 05:34:09 -- STEP: 19404/36938 -- GLOBAL_STEP: 647350 | > loss_text_ce: 0.03757338225841522 (0.035530910585077465) | > loss_mel_ce: 3.390500545501709 (nan) | > loss: 0.013603468425571918 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13645870857350828) | > loader_time: 0.0046 (0.004651343065594895)  --> TIME: 2025-02-25 05:34:18 -- STEP: 19454/36938 -- GLOBAL_STEP: 647400 | > loss_text_ce: 0.04111811891198158 (0.035529171099176374) | > loss_mel_ce: 2.5210280418395996 (nan) | > loss: 0.010167247615754604 (nan) | > current_lr: 5e-06 | > step_time: 0.1751 (0.13645395915677636) | > loader_time: 0.0048 (0.004651628161207137)  --> TIME: 2025-02-25 05:34:28 -- STEP: 19504/36938 -- GLOBAL_STEP: 647450 | > loss_text_ce: 0.03944778814911842 (0.035529209906493144) | > loss_mel_ce: 2.5352623462677 (nan) | > loss: 0.010217104107141495 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.1364577790696274) | > loader_time: 0.0046 (0.004651526944181609)  --> TIME: 2025-02-25 05:34:37 -- STEP: 19554/36938 -- GLOBAL_STEP: 647500 | > loss_text_ce: 0.033024948090314865 (0.03552965663095885) | > loss_mel_ce: 2.598177194595337 (nan) | > loss: 0.010441279038786888 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13645110795306956) | > loader_time: 0.0046 (0.004651498536067714)  --> TIME: 2025-02-25 05:34:46 -- STEP: 19604/36938 -- GLOBAL_STEP: 647550 | > loss_text_ce: 0.042041029781103134 (0.03552826937097821) | > loss_mel_ce: 1.1992835998535156 (nan) | > loss: 0.004925891757011414 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13644623338045123) | > loader_time: 0.0045 (0.004651404295471656)  --> TIME: 2025-02-25 05:34:56 -- STEP: 19654/36938 -- GLOBAL_STEP: 647600 | > loss_text_ce: 0.03649231046438217 (0.035529375570481504) | > loss_mel_ce: 2.7607052326202393 (nan) | > loss: 0.011099991388618946 (nan) | > current_lr: 5e-06 | > step_time: 0.1766 (0.13645156922897914) | > loader_time: 0.0048 (0.0046513771081584795)  --> TIME: 2025-02-25 05:35:05 -- STEP: 19704/36938 -- GLOBAL_STEP: 647650 | > loss_text_ce: 0.03499653562903404 (0.0355271438294658) | > loss_mel_ce: 3.3918282985687256 (nan) | > loss: 0.013598511926829815 (nan) | > current_lr: 5e-06 | > step_time: 0.184 (0.13644403083702905) | > loader_time: 0.0048 (0.004651375686643206)  --> TIME: 2025-02-25 05:35:14 -- STEP: 19754/36938 -- GLOBAL_STEP: 647700 | > loss_text_ce: 0.04222733899950981 (0.035527405509664996) | > loss_mel_ce: 3.3954405784606934 (nan) | > loss: 0.013641539961099625 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13644378204824606) | > loader_time: 0.0046 (0.004651445373055492)  --> TIME: 2025-02-25 05:35:24 -- STEP: 19804/36938 -- GLOBAL_STEP: 647750 | > loss_text_ce: 0.04910551384091377 (0.03552968927901371) | > loss_mel_ce: 3.5259366035461426 (nan) | > loss: 0.014186675660312176 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13645600703142544) | > loader_time: 0.0046 (0.004651278456012038)  --> TIME: 2025-02-25 05:35:33 -- STEP: 19854/36938 -- GLOBAL_STEP: 647800 | > loss_text_ce: 0.03635461628437042 (0.035530323994316736) | > loss_mel_ce: 2.7677981853485107 (nan) | > loss: 0.011127591133117676 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1364558995568621) | > loader_time: 0.0045 (0.004651273342853876)  --> TIME: 2025-02-25 05:35:42 -- STEP: 19904/36938 -- GLOBAL_STEP: 647850 | > loss_text_ce: 0.037107840180397034 (0.03553467487600599) | > loss_mel_ce: 1.56103515625 (nan) | > loss: 0.00634183781221509 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13645985127214072) | > loader_time: 0.0045 (0.004651275753879107)  --> TIME: 2025-02-25 05:35:52 -- STEP: 19954/36938 -- GLOBAL_STEP: 647900 | > loss_text_ce: 0.038774553686380386 (0.0355403458950593) | > loss_mel_ce: 2.948410987854004 (nan) | > loss: 0.011853910982608795 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.1364627665050095) | > loader_time: 0.0046 (0.004651308967771588)  --> TIME: 2025-02-25 05:36:01 -- STEP: 20004/36938 -- GLOBAL_STEP: 647950 | > loss_text_ce: 0.0390470027923584 (0.03554038315795583) | > loss_mel_ce: 0.025279998779296875 (nan) | > loss: 0.0002552658843342215 (nan) | > current_lr: 5e-06 | > step_time: 0.1699 (0.13646058450100837) | > loader_time: 0.0048 (0.004651286058057878)  --> TIME: 2025-02-25 05:36:10 -- STEP: 20054/36938 -- GLOBAL_STEP: 648000 | > loss_text_ce: 0.03390960022807121 (0.03554072851451485) | > loss_mel_ce: 3.5128350257873535 (nan) | > loss: 0.014074384234845638 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.13645981319216735) | > loader_time: 0.005 (0.0046511736089239215)  --> TIME: 2025-02-25 05:36:19 -- STEP: 20104/36938 -- GLOBAL_STEP: 648050 | > loss_text_ce: 0.04001621529459953 (0.03554221145647293) | > loss_mel_ce: 2.8844265937805176 (nan) | > loss: 0.011604932136833668 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13644856477798908) | > loader_time: 0.0045 (0.004650999446148627)  --> TIME: 2025-02-25 05:36:29 -- STEP: 20154/36938 -- GLOBAL_STEP: 648100 | > loss_text_ce: 0.044761862605810165 (0.0355453904584797) | > loss_mel_ce: 2.218693733215332 (nan) | > loss: 0.008981967344880104 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.1364443920618535) | > loader_time: 0.0045 (0.0046508309149583755)  --> TIME: 2025-02-25 05:36:38 -- STEP: 20204/36938 -- GLOBAL_STEP: 648150 | > loss_text_ce: 0.03803529962897301 (0.03554452890944163) | > loss_mel_ce: 3.1997811794281006 (nan) | > loss: 0.012848478741943836 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13643881516323503) | > loader_time: 0.0046 (0.0046506195204302015)  --> TIME: 2025-02-25 05:36:47 -- STEP: 20254/36938 -- GLOBAL_STEP: 648200 | > loss_text_ce: 0.03103986196219921 (0.035543406417379395) | > loss_mel_ce: 3.3916094303131104 (nan) | > loss: 0.013581942766904831 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.13644182787980244) | > loader_time: 0.0048 (0.004650752295084166)  --> TIME: 2025-02-25 05:36:56 -- STEP: 20304/36938 -- GLOBAL_STEP: 648250 | > loss_text_ce: 0.032488804310560226 (0.035544548893134145) | > loss_mel_ce: 3.1882412433624268 (nan) | > loss: 0.012780675664544106 (nan) | > current_lr: 5e-06 | > step_time: 0.1774 (0.1364381242517039) | > loader_time: 0.0048 (0.004650821875548922)  --> TIME: 2025-02-25 05:37:06 -- STEP: 20354/36938 -- GLOBAL_STEP: 648300 | > loss_text_ce: 0.04757299646735191 (0.03554588767111561) | > loss_mel_ce: 2.5923514366149902 (nan) | > loss: 0.010475891642272472 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1364472594583996) | > loader_time: 0.0045 (0.004651415848223637)  --> TIME: 2025-02-25 05:37:15 -- STEP: 20404/36938 -- GLOBAL_STEP: 648350 | > loss_text_ce: 0.029749780893325806 (0.03554376494335698) | > loss_mel_ce: 3.30078125 (nan) | > loss: 0.013216393999755383 (nan) | > current_lr: 5e-06 | > step_time: 0.1266 (0.13644558440187698) | > loader_time: 0.0047 (0.004651670513422307)  --> TIME: 2025-02-25 05:37:24 -- STEP: 20454/36938 -- GLOBAL_STEP: 648400 | > loss_text_ce: 0.034851934760808945 (0.03554702795424727) | > loss_mel_ce: 2.0536091327667236 (nan) | > loss: 0.00828754436224699 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13642779012760073) | > loader_time: 0.0046 (0.004651635905637237)  --> TIME: 2025-02-25 05:37:33 -- STEP: 20504/36938 -- GLOBAL_STEP: 648450 | > loss_text_ce: 0.041881151497364044 (0.035547289146185636) | > loss_mel_ce: 2.7723066806793213 (nan) | > loss: 0.011167412623763084 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13641315762577153) | > loader_time: 0.0045 (0.0046515878968717075)  --> TIME: 2025-02-25 05:37:43 -- STEP: 20554/36938 -- GLOBAL_STEP: 648500 | > loss_text_ce: 0.0357171893119812 (0.035547490095261335) | > loss_mel_ce: 3.3957254886627197 (nan) | > loss: 0.013616836629807949 (nan) | > current_lr: 5e-06 | > step_time: 0.197 (0.13641885313598467) | > loader_time: 0.0049 (0.004651644506655846)  --> TIME: 2025-02-25 05:37:52 -- STEP: 20604/36938 -- GLOBAL_STEP: 648550 | > loss_text_ce: 0.03483486548066139 (0.0355430430642898) | > loss_mel_ce: 2.102402687072754 (nan) | > loss: 0.0084811020642519 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.1364251987676302) | > loader_time: 0.0046 (0.004651637175457625)  --> TIME: 2025-02-25 05:38:01 -- STEP: 20654/36938 -- GLOBAL_STEP: 648600 | > loss_text_ce: 0.031811442226171494 (0.03554457577827055) | > loss_mel_ce: 3.110243082046509 (nan) | > loss: 0.012468471191823483 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.1364161781458055) | > loader_time: 0.0046 (0.0046516261512178)  --> TIME: 2025-02-25 05:38:11 -- STEP: 20704/36938 -- GLOBAL_STEP: 648650 | > loss_text_ce: 0.0389857180416584 (0.03554036580121514) | > loss_mel_ce: 3.3601818084716797 (nan) | > loss: 0.013488761149346828 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.13642990267857177) | > loader_time: 0.0047 (0.004651643692802251)  --> TIME: 2025-02-25 05:38:20 -- STEP: 20754/36938 -- GLOBAL_STEP: 648700 | > loss_text_ce: 0.027794117107987404 (0.03554137050988829) | > loss_mel_ce: 3.0225346088409424 (nan) | > loss: 0.012104479596018791 (nan) | > current_lr: 5e-06 | > step_time: 0.1642 (0.13644163366387443) | > loader_time: 0.0048 (0.00465182821748574)  --> TIME: 2025-02-25 05:38:30 -- STEP: 20804/36938 -- GLOBAL_STEP: 648750 | > loss_text_ce: 0.03717504069209099 (0.03554072183637616) | > loss_mel_ce: 3.2727761268615723 (nan) | > loss: 0.013134727254509926 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13644748420766706) | > loader_time: 0.0046 (0.0046517573579231974)  --> TIME: 2025-02-25 05:38:39 -- STEP: 20854/36938 -- GLOBAL_STEP: 648800 | > loss_text_ce: 0.04050229862332344 (0.03553994836567421) | > loss_mel_ce: 0.1355387419462204 (nan) | > loss: 0.0006985755753703415 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.1364399148872082) | > loader_time: 0.0046 (0.004651781021148173)  --> TIME: 2025-02-25 05:38:48 -- STEP: 20904/36938 -- GLOBAL_STEP: 648850 | > loss_text_ce: 0.031329818069934845 (0.035539622916499605) | > loss_mel_ce: 2.6500649452209473 (nan) | > loss: 0.010640456341207027 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.1364447447334817) | > loader_time: 0.0047 (0.004651836084306727)  --> TIME: 2025-02-25 05:38:58 -- STEP: 20954/36938 -- GLOBAL_STEP: 648900 | > loss_text_ce: 0.037220485508441925 (0.03553705005949632) | > loss_mel_ce: 0.9063906073570251 (nan) | > loss: 0.0037444885820150375 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13643833179333734) | > loader_time: 0.0046 (0.00465182641586344)  --> TIME: 2025-02-25 05:39:07 -- STEP: 21004/36938 -- GLOBAL_STEP: 648950 | > loss_text_ce: 0.026572708040475845 (0.03553494976859531) | > loss_mel_ce: 2.9768989086151123 (nan) | > loss: 0.011918539181351662 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13643789760182456) | > loader_time: 0.0045 (0.004651938863219234)  --> TIME: 2025-02-25 05:39:16 -- STEP: 21054/36938 -- GLOBAL_STEP: 649000 | > loss_text_ce: 0.043868500739336014 (0.0355338569103003) | > loss_mel_ce: 0.0267333984375 (nan) | > loss: 0.0002801662776619196 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13644114208321115) | > loader_time: 0.0047 (0.004651986319443284)  --> TIME: 2025-02-25 05:39:26 -- STEP: 21104/36938 -- GLOBAL_STEP: 649050 | > loss_text_ce: 0.03145907446742058 (0.03552673734269803) | > loss_mel_ce: 2.744276523590088 (nan) | > loss: 0.011014824733138084 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13644548495941966) | > loader_time: 0.0046 (0.004651967269439794)  --> TIME: 2025-02-25 05:39:35 -- STEP: 21154/36938 -- GLOBAL_STEP: 649100 | > loss_text_ce: 0.04406207054853439 (0.03552551598535454) | > loss_mel_ce: 0.12996311485767365 (nan) | > loss: 0.0006905761547386646 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.1364517292160841) | > loader_time: 0.0046 (0.004651928270337634)  --> TIME: 2025-02-25 05:39:44 -- STEP: 21204/36938 -- GLOBAL_STEP: 649150 | > loss_text_ce: 0.0416569784283638 (0.03552345517455122) | > loss_mel_ce: 3.5626823902130127 (nan) | > loss: 0.014302934519946575 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.1364485551064659) | > loader_time: 0.0048 (0.004651954456941756)  --> TIME: 2025-02-25 05:39:54 -- STEP: 21254/36938 -- GLOBAL_STEP: 649200 | > loss_text_ce: 0.03640533238649368 (0.03552212973550402) | > loss_mel_ce: 2.6298155784606934 (nan) | > loss: 0.010580242611467838 (nan) | > current_lr: 5e-06 | > step_time: 0.1773 (0.1364517353704357) | > loader_time: 0.0048 (0.0046521006045992675)  --> TIME: 2025-02-25 05:40:03 -- STEP: 21304/36938 -- GLOBAL_STEP: 649250 | > loss_text_ce: 0.04313343018293381 (0.035521863633041016) | > loss_mel_ce: 3.5885229110717773 (nan) | > loss: 0.014411335811018944 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.13644514937083924) | > loader_time: 0.0048 (0.004652062910103603)  --> TIME: 2025-02-25 05:40:12 -- STEP: 21354/36938 -- GLOBAL_STEP: 649300 | > loss_text_ce: 0.03669731691479683 (0.035520562577303176) | > loss_mel_ce: 3.038325786590576 (nan) | > loss: 0.012202473357319832 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13644508136684297) | > loader_time: 0.0046 (0.0046521559004548355)  --> TIME: 2025-02-25 05:40:22 -- STEP: 21404/36938 -- GLOBAL_STEP: 649350 | > loss_text_ce: 0.03256913274526596 (0.035516316503614566) | > loss_mel_ce: 3.0573337078094482 (nan) | > loss: 0.012261520139873028 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.1364500734142629) | > loader_time: 0.0046 (0.004652147191948952)  --> TIME: 2025-02-25 05:40:31 -- STEP: 21454/36938 -- GLOBAL_STEP: 649400 | > loss_text_ce: 0.029428791254758835 (0.03551533822138826) | > loss_mel_ce: 3.112318754196167 (nan) | > loss: 0.012467253021895885 (nan) | > current_lr: 5e-06 | > step_time: 0.1513 (0.13645458165870544) | > loader_time: 0.0046 (0.004652202501649953)  --> TIME: 2025-02-25 05:40:41 -- STEP: 21504/36938 -- GLOBAL_STEP: 649450 | > loss_text_ce: 0.031856030225753784 (0.035513363749790444) | > loss_mel_ce: 3.113489866256714 (nan) | > loss: 0.01248153205960989 (nan) | > current_lr: 5e-06 | > step_time: 0.1708 (0.13646042843659775) | > loader_time: 0.0048 (0.004652111303238647)  --> TIME: 2025-02-25 05:40:50 -- STEP: 21554/36938 -- GLOBAL_STEP: 649500 | > loss_text_ce: 0.038854315876960754 (0.035513055780052996) | > loss_mel_ce: 2.8776051998138428 (nan) | > loss: 0.011573253199458122 (nan) | > current_lr: 5e-06 | > step_time: 0.1532 (0.1364543500012525) | > loader_time: 0.0047 (0.004652006701125819)  --> TIME: 2025-02-25 05:40:59 -- STEP: 21604/36938 -- GLOBAL_STEP: 649550 | > loss_text_ce: 0.03090997226536274 (0.035513223795212656) | > loss_mel_ce: 2.5569729804992676 (nan) | > loss: 0.010269377380609512 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13646079892031596) | > loader_time: 0.0046 (0.004651902605264074)  --> TIME: 2025-02-25 05:41:08 -- STEP: 21654/36938 -- GLOBAL_STEP: 649600 | > loss_text_ce: 0.03694988787174225 (0.03551384999809097) | > loss_mel_ce: 0.36199951171875 (nan) | > loss: 0.0015831325436010957 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13644719557649015) | > loader_time: 0.0045 (0.004651758769239613)  --> TIME: 2025-02-25 05:41:17 -- STEP: 21704/36938 -- GLOBAL_STEP: 649650 | > loss_text_ce: 0.03535328805446625 (0.035512567064954884) | > loss_mel_ce: 2.923255205154419 (nan) | > loss: 0.01174051035195589 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13644584302659718) | > loader_time: 0.0046 (0.0046517160098754045)  --> TIME: 2025-02-25 05:41:27 -- STEP: 21754/36938 -- GLOBAL_STEP: 649700 | > loss_text_ce: 0.03444216027855873 (0.035511383506852755) | > loss_mel_ce: 2.6378750801086426 (nan) | > loss: 0.01060443464666605 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13644890680222863) | > loader_time: 0.0045 (0.004651613847904929)  --> TIME: 2025-02-25 05:41:36 -- STEP: 21804/36938 -- GLOBAL_STEP: 649750 | > loss_text_ce: 0.02986738830804825 (0.03550975247599946) | > loss_mel_ce: 3.0910913944244385 (nan) | > loss: 0.012384758330881596 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13645857746416157) | > loader_time: 0.0046 (0.00465154991174397)  --> TIME: 2025-02-25 05:41:45 -- STEP: 21854/36938 -- GLOBAL_STEP: 649800 | > loss_text_ce: 0.0281898844987154 (0.03550788156099775) | > loss_mel_ce: 2.487071990966797 (nan) | > loss: 0.009981198236346245 (nan) | > current_lr: 5e-06 | > step_time: 0.107 (0.1364440647301376) | > loader_time: 0.0045 (0.0046515178514594)  --> TIME: 2025-02-25 05:41:54 -- STEP: 21904/36938 -- GLOBAL_STEP: 649850 | > loss_text_ce: 0.03297964856028557 (0.035509206788815736) | > loss_mel_ce: 3.4895832538604736 (nan) | > loss: 0.013978425413370132 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13644032117178073) | > loader_time: 0.0044 (0.004651584008734462)  --> TIME: 2025-02-25 05:42:04 -- STEP: 21954/36938 -- GLOBAL_STEP: 649900 | > loss_text_ce: 0.03381875529885292 (0.035511577528448926) | > loss_mel_ce: 2.670691728591919 (nan) | > loss: 0.01073218509554863 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.136435693227128) | > loader_time: 0.0046 (0.004651612028719866)  --> TIME: 2025-02-25 05:42:13 -- STEP: 22004/36938 -- GLOBAL_STEP: 649950 | > loss_text_ce: 0.030395010486245155 (0.03551418606447178) | > loss_mel_ce: 2.17596435546875 (nan) | > loss: 0.00875539518892765 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13644480100655013) | > loader_time: 0.0045 (0.00465157340383817)  --> TIME: 2025-02-25 05:42:23 -- STEP: 22054/36938 -- GLOBAL_STEP: 650000 | > loss_text_ce: 0.031980954110622406 (0.0355136448867437) | > loss_mel_ce: 3.149684190750122 (nan) | > loss: 0.012625656090676785 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13645554796419407) | > loader_time: 0.0047 (0.004651675060414411) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_650000.pth  --> TIME: 2025-02-25 05:42:38 -- STEP: 22104/36938 -- GLOBAL_STEP: 650050 | > loss_text_ce: 0.04193650931119919 (0.035513570458468736) | > loss_mel_ce: 3.4693336486816406 (nan) | > loss: 0.013933612033724785 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13644829017072801) | > loader_time: 0.0046 (0.004652232611218197)  --> TIME: 2025-02-25 05:42:47 -- STEP: 22154/36938 -- GLOBAL_STEP: 650100 | > loss_text_ce: 0.031810324639081955 (0.03551387625613665) | > loss_mel_ce: 2.7574634552001953 (nan) | > loss: 0.011068547144532204 (nan) | > current_lr: 5e-06 | > step_time: 0.151 (0.1364402395299043) | > loader_time: 0.0047 (0.004652228727338126)  --> TIME: 2025-02-25 05:42:56 -- STEP: 22204/36938 -- GLOBAL_STEP: 650150 | > loss_text_ce: 0.03548841550946236 (0.03551748897806488) | > loss_mel_ce: 2.2184245586395264 (nan) | > loss: 0.008944099768996239 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13642659177223576) | > loader_time: 0.0045 (0.0046521037940484315)  --> TIME: 2025-02-25 05:43:05 -- STEP: 22254/36938 -- GLOBAL_STEP: 650200 | > loss_text_ce: 0.03195205703377724 (0.03551838262246184) | > loss_mel_ce: 2.8886520862579346 (nan) | > loss: 0.011589700356125832 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13640979914006618) | > loader_time: 0.0047 (0.004651999467143681)  --> TIME: 2025-02-25 05:43:14 -- STEP: 22304/36938 -- GLOBAL_STEP: 650250 | > loss_text_ce: 0.03302006050944328 (0.03551706780680856) | > loss_mel_ce: 2.6446914672851562 (nan) | > loss: 0.010625840164721012 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1364189725224913) | > loader_time: 0.0045 (0.004651904416135612)  --> TIME: 2025-02-25 05:43:24 -- STEP: 22354/36938 -- GLOBAL_STEP: 650300 | > loss_text_ce: 0.02981662191450596 (0.03551766037600345) | > loss_mel_ce: 2.816762685775757 (nan) | > loss: 0.011295950040221214 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13641875079442495) | > loader_time: 0.0051 (0.004652057413318105)  --> TIME: 2025-02-25 05:43:33 -- STEP: 22404/36938 -- GLOBAL_STEP: 650350 | > loss_text_ce: 0.03508449345827103 (0.03552091847160733) | > loss_mel_ce: 1.9822838306427002 (nan) | > loss: 0.008005429990589619 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.1364149959613934) | > loader_time: 0.0046 (0.00465211383446353)  --> TIME: 2025-02-25 05:43:42 -- STEP: 22454/36938 -- GLOBAL_STEP: 650400 | > loss_text_ce: 0.03118966333568096 (0.03552493412884464) | > loss_mel_ce: 2.94000244140625 (nan) | > loss: 0.011790446005761623 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13641463183354402) | > loader_time: 0.0045 (0.00465206412074722)  --> TIME: 2025-02-25 05:43:51 -- STEP: 22504/36938 -- GLOBAL_STEP: 650450 | > loss_text_ce: 0.03844556212425232 (0.03552449268887829) | > loss_mel_ce: 3.2560737133026123 (nan) | > loss: 0.013073490001261234 (nan) | > current_lr: 5e-06 | > step_time: 0.1785 (0.1364086154852316) | > loader_time: 0.0048 (0.004651968202852)  --> TIME: 2025-02-25 05:44:00 -- STEP: 22554/36938 -- GLOBAL_STEP: 650500 | > loss_text_ce: 0.03825001418590546 (0.0355219156229851) | > loss_mel_ce: 1.7856004238128662 (nan) | > loss: 0.007237501908093691 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13640304756384491) | > loader_time: 0.0045 (0.00465192291196848)  --> TIME: 2025-02-25 05:44:10 -- STEP: 22604/36938 -- GLOBAL_STEP: 650550 | > loss_text_ce: 0.04137140139937401 (0.035521634851121364) | > loss_mel_ce: 2.948667287826538 (nan) | > loss: 0.011865233071148396 (nan) | > current_lr: 5e-06 | > step_time: 0.1507 (0.13641817402869272) | > loader_time: 0.0047 (0.004651959333519379)  --> TIME: 2025-02-25 05:44:20 -- STEP: 22654/36938 -- GLOBAL_STEP: 650600 | > loss_text_ce: 0.03586791828274727 (0.035519807970015936) | > loss_mel_ce: 3.389404296875 (nan) | > loss: 0.013592351227998734 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13642159408764065) | > loader_time: 0.0049 (0.004651939289035438)  --> TIME: 2025-02-25 05:44:29 -- STEP: 22704/36938 -- GLOBAL_STEP: 650650 | > loss_text_ce: 0.03906942903995514 (0.035521101763636334) | > loss_mel_ce: 2.654416084289551 (nan) | > loss: 0.010688435286283493 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13640911492812524) | > loader_time: 0.0046 (0.00465180587818961)  --> TIME: 2025-02-25 05:44:38 -- STEP: 22754/36938 -- GLOBAL_STEP: 650700 | > loss_text_ce: 0.033019013702869415 (0.03551857101631049) | > loss_mel_ce: 3.2097373008728027 (nan) | > loss: 0.012868081219494343 (nan) | > current_lr: 5e-06 | > step_time: 0.1688 (0.1364176081177687) | > loader_time: 0.0048 (0.00465183339995346)  --> TIME: 2025-02-25 05:44:48 -- STEP: 22804/36938 -- GLOBAL_STEP: 650750 | > loss_text_ce: 0.035692766308784485 (0.03551729904966214) | > loss_mel_ce: 2.4781723022460938 (nan) | > loss: 0.009975655004382133 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13642134235273762) | > loader_time: 0.0046 (0.0046518345377650604)  --> TIME: 2025-02-25 05:44:57 -- STEP: 22854/36938 -- GLOBAL_STEP: 650800 | > loss_text_ce: 0.04349059984087944 (0.035518605013419476) | > loss_mel_ce: 1.7792794704437256 (nan) | > loss: 0.007233215030282736 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13642538431551093) | > loader_time: 0.0046 (0.00465180499979108)  --> TIME: 2025-02-25 05:45:06 -- STEP: 22904/36938 -- GLOBAL_STEP: 650850 | > loss_text_ce: 0.0362171046435833 (0.03551523827667435) | > loss_mel_ce: 2.053070068359375 (nan) | > loss: 0.008290822617709637 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13641905866841864) | > loader_time: 0.0045 (0.004651763588658511)  --> TIME: 2025-02-25 05:45:16 -- STEP: 22954/36938 -- GLOBAL_STEP: 650900 | > loss_text_ce: 0.041771385818719864 (0.035514421592940215) | > loss_mel_ce: 3.6211400032043457 (nan) | > loss: 0.014535363763570786 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.1364177150718849) | > loader_time: 0.0045 (0.004651722794180712)  --> TIME: 2025-02-25 05:45:25 -- STEP: 23004/36938 -- GLOBAL_STEP: 650950 | > loss_text_ce: 0.030167793855071068 (0.03551398809568306) | > loss_mel_ce: 2.937654972076416 (nan) | > loss: 0.011777075007557869 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.1364095077641514) | > loader_time: 0.0046 (0.004651616571509646)  --> TIME: 2025-02-25 05:45:34 -- STEP: 23054/36938 -- GLOBAL_STEP: 651000 | > loss_text_ce: 0.026763638481497765 (0.03551487969190959) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13640424393817513) | > loader_time: 0.0045 (0.004651510737202294)  --> TIME: 2025-02-25 05:45:43 -- STEP: 23104/36938 -- GLOBAL_STEP: 651050 | > loss_text_ce: 0.028147539123892784 (0.035516109928036284) | > loss_mel_ce: 2.308882236480713 (nan) | > loss: 0.00927392765879631 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13639476413418136) | > loader_time: 0.0045 (0.004651405763427979)  --> TIME: 2025-02-25 05:45:52 -- STEP: 23154/36938 -- GLOBAL_STEP: 651100 | > loss_text_ce: 0.030382953584194183 (0.035516857105734684) | > loss_mel_ce: 1.6966116428375244 (nan) | > loss: 0.006853153929114342 (nan) | > current_lr: 5e-06 | > step_time: 0.1556 (0.13639024868729865) | > loader_time: 0.0046 (0.004651295208937158)  --> TIME: 2025-02-25 05:46:01 -- STEP: 23204/36938 -- GLOBAL_STEP: 651150 | > loss_text_ce: 0.043364133685827255 (0.035522096102397135) | > loss_mel_ce: 2.9736344814300537 (nan) | > loss: 0.0119722168892622 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1363866437798723) | > loader_time: 0.0045 (0.004651266168960317)  --> TIME: 2025-02-25 05:46:11 -- STEP: 23254/36938 -- GLOBAL_STEP: 651200 | > loss_text_ce: 0.03393079340457916 (0.03552345469693488) | > loss_mel_ce: 2.66154408454895 (nan) | > loss: 0.010696329176425934 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13638840446962722) | > loader_time: 0.0046 (0.004651191341832905)  --> TIME: 2025-02-25 05:46:20 -- STEP: 23304/36938 -- GLOBAL_STEP: 651250 | > loss_text_ce: 0.03370372951030731 (0.03552298223727841) | > loss_mel_ce: 2.9354255199432373 (nan) | > loss: 0.011782259680330753 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.13639720965405788) | > loader_time: 0.0048 (0.004651372656967899)  --> TIME: 2025-02-25 05:46:30 -- STEP: 23354/36938 -- GLOBAL_STEP: 651300 | > loss_text_ce: 0.027345599606633186 (0.03552300447580241) | > loss_mel_ce: 2.9483604431152344 (nan) | > loss: 0.011808358132839203 (nan) | > current_lr: 5e-06 | > step_time: 0.1869 (0.1364046085000277) | > loader_time: 0.0048 (0.004651421266150546)  --> TIME: 2025-02-25 05:46:39 -- STEP: 23404/36938 -- GLOBAL_STEP: 651350 | > loss_text_ce: 0.04526442289352417 (0.03552426826171432) | > loss_mel_ce: 1.092936396598816 (nan) | > loss: 0.004516670014709234 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1364045079254409) | > loader_time: 0.0045 (0.004651420851117269)  --> TIME: 2025-02-25 05:46:49 -- STEP: 23454/36938 -- GLOBAL_STEP: 651400 | > loss_text_ce: 0.03267320245504379 (0.03552429543223921) | > loss_mel_ce: 2.4402780532836914 (nan) | > loss: 0.009813298471271992 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13641476749082854) | > loader_time: 0.0046 (0.004651536475547784)  --> TIME: 2025-02-25 05:46:58 -- STEP: 23504/36938 -- GLOBAL_STEP: 651450 | > loss_text_ce: 0.042694225907325745 (0.03552706969761341) | > loss_mel_ce: 3.5582032203674316 (nan) | > loss: 0.01428927667438984 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13640893839548213) | > loader_time: 0.0045 (0.004651481416291985)  --> TIME: 2025-02-25 05:47:07 -- STEP: 23554/36938 -- GLOBAL_STEP: 651500 | > loss_text_ce: 0.037930313497781754 (0.03553011193350368) | > loss_mel_ce: 3.6452112197875977 (nan) | > loss: 0.01461564190685749 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.13640673752499932) | > loader_time: 0.0048 (0.004651435822251002)  --> TIME: 2025-02-25 05:47:17 -- STEP: 23604/36938 -- GLOBAL_STEP: 651550 | > loss_text_ce: 0.034552812576293945 (0.03552739998159841) | > loss_mel_ce: 3.0568652153015137 (nan) | > loss: 0.012267532758414745 (nan) | > current_lr: 5e-06 | > step_time: 0.1514 (0.13640696626460216) | > loader_time: 0.0046 (0.004651593153930453)  --> TIME: 2025-02-25 05:47:26 -- STEP: 23654/36938 -- GLOBAL_STEP: 651600 | > loss_text_ce: 0.030932562425732613 (0.03552645547416279) | > loss_mel_ce: 3.2734484672546387 (nan) | > loss: 0.013112624175846577 (nan) | > current_lr: 5e-06 | > step_time: 0.1873 (0.13640910777616672) | > loader_time: 0.0069 (0.004651662593179387)  --> TIME: 2025-02-25 05:47:36 -- STEP: 23704/36938 -- GLOBAL_STEP: 651650 | > loss_text_ce: 0.03855486214160919 (0.03552895581403646) | > loss_mel_ce: 3.209047317504883 (nan) | > loss: 0.012887311168015003 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.1364292524631788) | > loader_time: 0.0047 (0.0046515704468622)  --> TIME: 2025-02-25 05:47:45 -- STEP: 23754/36938 -- GLOBAL_STEP: 651700 | > loss_text_ce: 0.055010560899972916 (0.03552859698184588) | > loss_mel_ce: 3.5834763050079346 (nan) | > loss: 0.014438441023230553 (nan) | > current_lr: 5e-06 | > step_time: 0.1625 (0.1364245215499702) | > loader_time: 0.0048 (0.004651557870251262)  --> TIME: 2025-02-25 05:47:54 -- STEP: 23804/36938 -- GLOBAL_STEP: 651750 | > loss_text_ce: 0.0366930216550827 (0.03553080948524638) | > loss_mel_ce: 2.419661045074463 (nan) | > loss: 0.009747437201440334 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.1364148778187047) | > loader_time: 0.0045 (0.004651580973044918)  --> TIME: 2025-02-25 05:48:03 -- STEP: 23854/36938 -- GLOBAL_STEP: 651800 | > loss_text_ce: 0.04105158895254135 (0.03553317500283983) | > loss_mel_ce: 2.978100538253784 (nan) | > loss: 0.011980763636529446 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13641281618479845) | > loader_time: 0.0046 (0.004651524389520779)  --> TIME: 2025-02-25 05:48:13 -- STEP: 23904/36938 -- GLOBAL_STEP: 651850 | > loss_text_ce: 0.04314872995018959 (0.035534934658260235) | > loss_mel_ce: 3.0522093772888184 (nan) | > loss: 0.01228316780179739 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.1364168322668813) | > loader_time: 0.0045 (0.004651444683592022)  --> TIME: 2025-02-25 05:48:22 -- STEP: 23954/36938 -- GLOBAL_STEP: 651900 | > loss_text_ce: 0.025796109810471535 (0.03553002497339625) | > loss_mel_ce: 2.726120710372925 (nan) | > loss: 0.010920305736362934 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.1364145771911884) | > loader_time: 0.0046 (0.0046514997779303)  --> TIME: 2025-02-25 05:48:32 -- STEP: 24004/36938 -- GLOBAL_STEP: 651950 | > loss_text_ce: 0.035215191543102264 (0.03552766461633041) | > loss_mel_ce: 2.88511061668396 (nan) | > loss: 0.011588594876229763 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13642013321278787) | > loader_time: 0.0046 (0.004651555030112718)  --> TIME: 2025-02-25 05:48:41 -- STEP: 24054/36938 -- GLOBAL_STEP: 652000 | > loss_text_ce: 0.031916890293359756 (0.035529114605572995) | > loss_mel_ce: 1.853479027748108 (nan) | > loss: 0.007481730077415705 (nan) | > current_lr: 5e-06 | > step_time: 0.1077 (0.1364268076789535) | > loader_time: 0.0045 (0.004651598356663575)  --> TIME: 2025-02-25 05:48:50 -- STEP: 24104/36938 -- GLOBAL_STEP: 652050 | > loss_text_ce: 0.03135085105895996 (0.035528801088714514) | > loss_mel_ce: 0.012939453125 (nan) | > loss: 0.00017575517995283008 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13642018591293284) | > loader_time: 0.0045 (0.004651563956102117)  --> TIME: 2025-02-25 05:49:00 -- STEP: 24154/36938 -- GLOBAL_STEP: 652100 | > loss_text_ce: 0.0355047807097435 (0.03552827637514932) | > loss_mel_ce: 0.1120724156498909 (nan) | > loss: 0.0005856238421984017 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.13641943997318853) | > loader_time: 0.0048 (0.004651557217669282)  --> TIME: 2025-02-25 05:49:09 -- STEP: 24204/36938 -- GLOBAL_STEP: 652150 | > loss_text_ce: 0.03574647009372711 (0.03552696171491105) | > loss_mel_ce: 3.5988264083862305 (nan) | > loss: 0.01442290935665369 (nan) | > current_lr: 5e-06 | > step_time: 0.168 (0.1364173718945596) | > loader_time: 0.0048 (0.004651617430553158)  --> TIME: 2025-02-25 05:49:18 -- STEP: 24254/36938 -- GLOBAL_STEP: 652200 | > loss_text_ce: 0.04014629125595093 (0.035525584352117834) | > loss_mel_ce: 3.1897149085998535 (nan) | > loss: 0.01281691063195467 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.13641005585885008) | > loader_time: 0.0047 (0.004651615092177744)  --> TIME: 2025-02-25 05:49:27 -- STEP: 24304/36938 -- GLOBAL_STEP: 652250 | > loss_text_ce: 0.037661343812942505 (0.03552453303795479) | > loss_mel_ce: 2.874577045440674 (nan) | > loss: 0.011556502431631088 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.1364075553856681) | > loader_time: 0.0046 (0.004651545673863853)  --> TIME: 2025-02-25 05:49:36 -- STEP: 24354/36938 -- GLOBAL_STEP: 652300 | > loss_text_ce: 0.03792630508542061 (0.03552458520735186) | > loss_mel_ce: 2.985872745513916 (nan) | > loss: 0.011999202892184258 (nan) | > current_lr: 5e-06 | > step_time: 0.1441 (0.13640251038862894) | > loader_time: 0.0047 (0.004651560007649684)  --> TIME: 2025-02-25 05:49:46 -- STEP: 24404/36938 -- GLOBAL_STEP: 652350 | > loss_text_ce: 0.0280911847949028 (0.0355239579670983) | > loss_mel_ce: 3.210069417953491 (nan) | > loss: 0.012849844060838223 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13640384222909696) | > loader_time: 0.0045 (0.004651472326615404)  --> TIME: 2025-02-25 05:49:55 -- STEP: 24454/36938 -- GLOBAL_STEP: 652400 | > loss_text_ce: 0.03638630732893944 (0.03552362239843681) | > loss_mel_ce: 3.1012635231018066 (nan) | > loss: 0.0124509921297431 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13639825974928674) | > loader_time: 0.0045 (0.004651297656788013)  --> TIME: 2025-02-25 05:50:04 -- STEP: 24504/36938 -- GLOBAL_STEP: 652450 | > loss_text_ce: 0.03616529330611229 (0.03552293130062865) | > loss_mel_ce: 2.5790019035339355 (nan) | > loss: 0.010377648286521435 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13640123741945037) | > loader_time: 0.0045 (0.0046511872547185945)  --> TIME: 2025-02-25 05:50:13 -- STEP: 24554/36938 -- GLOBAL_STEP: 652500 | > loss_text_ce: 0.04195879399776459 (0.035523666889859286) | > loss_mel_ce: 0.23698286712169647 (nan) | > loss: 0.0011069114552810788 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13639809351805507) | > loader_time: 0.0046 (0.0046510686992463045)  --> TIME: 2025-02-25 05:50:23 -- STEP: 24604/36938 -- GLOBAL_STEP: 652550 | > loss_text_ce: 0.028413811698555946 (0.03552199032374499) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13640106028608087) | > loader_time: 0.0045 (0.004650977322230062)  --> TIME: 2025-02-25 05:50:32 -- STEP: 24654/36938 -- GLOBAL_STEP: 652600 | > loss_text_ce: 0.04057598114013672 (0.0355224973809451) | > loss_mel_ce: 3.127079486846924 (nan) | > loss: 0.012570061720907688 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13640693621381228) | > loader_time: 0.0048 (0.004650945838296323)  --> TIME: 2025-02-25 05:50:42 -- STEP: 24704/36938 -- GLOBAL_STEP: 652650 | > loss_text_ce: 0.030015453696250916 (0.035521553530357394) | > loss_mel_ce: 3.019425630569458 (nan) | > loss: 0.012100957334041595 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.13640578294780842) | > loader_time: 0.0047 (0.004650895372805218)  --> TIME: 2025-02-25 05:50:51 -- STEP: 24754/36938 -- GLOBAL_STEP: 652700 | > loss_text_ce: 0.045679789036512375 (0.03552462840632647) | > loss_mel_ce: 2.9063894748687744 (nan) | > loss: 0.01171456091105938 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13640001885983558) | > loader_time: 0.0046 (0.004650876057247199)  --> TIME: 2025-02-25 05:51:00 -- STEP: 24804/36938 -- GLOBAL_STEP: 652750 | > loss_text_ce: 0.03889252245426178 (0.035523560730636373) | > loss_mel_ce: 2.6244008541107178 (nan) | > loss: 0.010568625293672085 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1364031993002872) | > loader_time: 0.0045 (0.004650817794427636)  --> TIME: 2025-02-25 05:51:09 -- STEP: 24854/36938 -- GLOBAL_STEP: 652800 | > loss_text_ce: 0.03579147160053253 (0.035522188579568924) | > loss_mel_ce: 2.601552724838257 (nan) | > loss: 0.010465651750564575 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13640242679280032) | > loader_time: 0.0045 (0.004650876548348394)  --> TIME: 2025-02-25 05:51:19 -- STEP: 24904/36938 -- GLOBAL_STEP: 652850 | > loss_text_ce: 0.03674277663230896 (0.03552501849442593) | > loss_mel_ce: 2.7187867164611816 (nan) | > loss: 0.01093464158475399 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13640021842237485) | > loader_time: 0.0046 (0.004650775638761694)  --> TIME: 2025-02-25 05:51:28 -- STEP: 24954/36938 -- GLOBAL_STEP: 652900 | > loss_text_ce: 0.03731045499444008 (0.03552382462503121) | > loss_mel_ce: 2.3451650142669678 (nan) | > loss: 0.009454268030822277 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.13640135359934122) | > loader_time: 0.0048 (0.004650708755220456)  --> TIME: 2025-02-25 05:51:37 -- STEP: 25004/36938 -- GLOBAL_STEP: 652950 | > loss_text_ce: 0.028555935248732567 (0.03552213140339933) | > loss_mel_ce: 2.5122523307800293 (nan) | > loss: 0.01008257269859314 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1364011907268568) | > loader_time: 0.0045 (0.00465061133088275)  --> TIME: 2025-02-25 05:51:47 -- STEP: 25054/36938 -- GLOBAL_STEP: 653000 | > loss_text_ce: 0.034031108021736145 (0.03551978725651394) | > loss_mel_ce: 2.394976854324341 (nan) | > loss: 0.009638921357691288 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13640153366179253) | > loader_time: 0.0044 (0.004650537533934042)  --> TIME: 2025-02-25 05:51:56 -- STEP: 25104/36938 -- GLOBAL_STEP: 653050 | > loss_text_ce: 0.026165660470724106 (0.035517677501708686) | > loss_mel_ce: 2.977048397064209 (nan) | > loss: 0.011917516589164734 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13639769553568298) | > loader_time: 0.0046 (0.004650453546003157)  --> TIME: 2025-02-25 05:52:05 -- STEP: 25154/36938 -- GLOBAL_STEP: 653100 | > loss_text_ce: 0.05436183512210846 (0.03551566126448559) | > loss_mel_ce: 2.3203225135803223 (nan) | > loss: 0.009423350915312767 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13638998045682038) | > loader_time: 0.0045 (0.004650370327971601)  --> TIME: 2025-02-25 05:52:14 -- STEP: 25204/36938 -- GLOBAL_STEP: 653150 | > loss_text_ce: 0.03168381378054619 (0.03551598683025785) | > loss_mel_ce: 3.212578058242798 (nan) | > loss: 0.012874056585133076 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13637837744425046) | > loader_time: 0.0045 (0.004650281991415278)  --> TIME: 2025-02-25 05:52:23 -- STEP: 25254/36938 -- GLOBAL_STEP: 653200 | > loss_text_ce: 0.033352382481098175 (0.035515720768723155) | > loss_mel_ce: 2.7160561084747314 (nan) | > loss: 0.010910351760685444 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13637320013673843) | > loader_time: 0.0046 (0.004650204370676462)  --> TIME: 2025-02-25 05:52:32 -- STEP: 25304/36938 -- GLOBAL_STEP: 653250 | > loss_text_ce: 0.02666211500763893 (0.03551436902371747) | > loss_mel_ce: 2.5500552654266357 (nan) | > loss: 0.010225069709122181 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13636863940206034) | > loader_time: 0.0045 (0.004650096359261994)  --> TIME: 2025-02-25 05:52:41 -- STEP: 25354/36938 -- GLOBAL_STEP: 653300 | > loss_text_ce: 0.02166960947215557 (0.035511438074925385) | > loss_mel_ce: 2.5838088989257812 (nan) | > loss: 0.010339201427996159 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13636197503681075) | > loader_time: 0.0045 (0.004649975749890963)  --> TIME: 2025-02-25 05:52:50 -- STEP: 25404/36938 -- GLOBAL_STEP: 653350 | > loss_text_ce: 0.02994050458073616 (0.035512249489587595) | > loss_mel_ce: 3.4794538021087646 (nan) | > loss: 0.013926168903708458 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13636372242738143) | > loader_time: 0.0045 (0.004649955782246898)  --> TIME: 2025-02-25 05:53:00 -- STEP: 25454/36938 -- GLOBAL_STEP: 653400 | > loss_text_ce: 0.029891060665249825 (0.03551323072221739) | > loss_mel_ce: 2.763671875 (nan) | > loss: 0.011085567995905876 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13636108639759673) | > loader_time: 0.0045 (0.004649877885417011)  --> TIME: 2025-02-25 05:53:09 -- STEP: 25504/36938 -- GLOBAL_STEP: 653450 | > loss_text_ce: 0.032716479152441025 (0.03550936853171489) | > loss_mel_ce: 3.054455041885376 (nan) | > loss: 0.012250681407749653 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13636607284116561) | > loader_time: 0.0045 (0.004649795582482948)  --> TIME: 2025-02-25 05:53:19 -- STEP: 25554/36938 -- GLOBAL_STEP: 653500 | > loss_text_ce: 0.04088142141699791 (0.035507290520687004) | > loss_mel_ce: 2.0453126430511475 (nan) | > loss: 0.008278547786176205 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13637691071459423) | > loader_time: 0.0046 (0.004649777073549317)  --> TIME: 2025-02-25 05:53:28 -- STEP: 25604/36938 -- GLOBAL_STEP: 653550 | > loss_text_ce: 0.032092854380607605 (0.03550406741257004) | > loss_mel_ce: 3.6307339668273926 (nan) | > loss: 0.01453502755612135 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13637591742701885) | > loader_time: 0.0045 (0.0046496142299636235)  --> TIME: 2025-02-25 05:53:37 -- STEP: 25654/36938 -- GLOBAL_STEP: 653600 | > loss_text_ce: 0.033150456845760345 (0.03550290913585601) | > loss_mel_ce: 1.8720703125 (nan) | > loss: 0.007560400292277336 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13636786239576668) | > loader_time: 0.0044 (0.004649482708686604)  --> TIME: 2025-02-25 05:53:46 -- STEP: 25704/36938 -- GLOBAL_STEP: 653650 | > loss_text_ce: 0.04356607794761658 (0.03550644843923043) | > loss_mel_ce: 1.1347424983978271 (nan) | > loss: 0.004675827920436859 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.136370815325109) | > loader_time: 0.0045 (0.004649385424963326)  --> TIME: 2025-02-25 05:53:56 -- STEP: 25754/36938 -- GLOBAL_STEP: 653700 | > loss_text_ce: 0.031594451516866684 (0.03550287648968791) | > loss_mel_ce: 0.8261865973472595 (nan) | > loss: 0.003403893206268549 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13636944136967938) | > loader_time: 0.0045 (0.004649308061640483)  --> TIME: 2025-02-25 05:54:05 -- STEP: 25804/36938 -- GLOBAL_STEP: 653750 | > loss_text_ce: 0.03922493755817413 (0.03550417047239273) | > loss_mel_ce: 3.635058641433716 (nan) | > loss: 0.014580490998923779 (nan) | > current_lr: 5e-06 | > step_time: 0.1859 (0.13636073219260497) | > loader_time: 0.0049 (0.004649263438356529)  --> TIME: 2025-02-25 05:54:14 -- STEP: 25854/36938 -- GLOBAL_STEP: 653800 | > loss_text_ce: 0.03210916742682457 (0.03550380526930131) | > loss_mel_ce: 3.3626036643981934 (nan) | > loss: 0.013471083715558052 (nan) | > current_lr: 5e-06 | > step_time: 0.1875 (0.1363684845568262) | > loader_time: 0.0049 (0.004649290031869273)  --> TIME: 2025-02-25 05:54:23 -- STEP: 25904/36938 -- GLOBAL_STEP: 653850 | > loss_text_ce: 0.032912783324718475 (0.03550256554990391) | > loss_mel_ce: 3.2420551776885986 (nan) | > loss: 0.012995905242860317 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13636668287838802) | > loader_time: 0.0047 (0.004649243765660771)  --> TIME: 2025-02-25 05:54:33 -- STEP: 25954/36938 -- GLOBAL_STEP: 653900 | > loss_text_ce: 0.04371105507016182 (0.03550519222496135) | > loss_mel_ce: 2.996582508087158 (nan) | > loss: 0.012064657174050808 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.13636529881992507) | > loader_time: 0.0049 (0.004649225860969627)  --> TIME: 2025-02-25 05:54:42 -- STEP: 26004/36938 -- GLOBAL_STEP: 653950 | > loss_text_ce: 0.04254300147294998 (0.03550408263458713) | > loss_mel_ce: 2.9442827701568604 (nan) | > loss: 0.01185248326510191 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.1363692042772952) | > loader_time: 0.0046 (0.004649294777808424)  --> TIME: 2025-02-25 05:54:52 -- STEP: 26054/36938 -- GLOBAL_STEP: 654000 | > loss_text_ce: 0.04015723615884781 (0.0355057774695075) | > loss_mel_ce: 3.060913324356079 (nan) | > loss: 0.01230583619326353 (nan) | > current_lr: 5e-06 | > step_time: 0.1506 (0.13637878917317126) | > loader_time: 0.0046 (0.004649277219130315)  --> TIME: 2025-02-25 05:55:01 -- STEP: 26104/36938 -- GLOBAL_STEP: 654050 | > loss_text_ce: 0.03659616783261299 (0.03550563563721621) | > loss_mel_ce: 2.835252523422241 (nan) | > loss: 0.011396225541830063 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1363765987614118) | > loader_time: 0.0046 (0.004649176321400844)  --> TIME: 2025-02-25 05:55:11 -- STEP: 26154/36938 -- GLOBAL_STEP: 654100 | > loss_text_ce: 0.03168359398841858 (0.03550299231939129) | > loss_mel_ce: 2.887347936630249 (nan) | > loss: 0.011583459563553333 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13638467133077) | > loader_time: 0.0046 (0.004649153094486039)  --> TIME: 2025-02-25 05:55:20 -- STEP: 26204/36938 -- GLOBAL_STEP: 654150 | > loss_text_ce: 0.03784564882516861 (0.035503187144918576) | > loss_mel_ce: 2.533527374267578 (nan) | > loss: 0.010203861631453037 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13637746413997548) | > loader_time: 0.0046 (0.004649249474850523)  --> TIME: 2025-02-25 05:55:29 -- STEP: 26254/36938 -- GLOBAL_STEP: 654200 | > loss_text_ce: 0.03081698901951313 (0.03550142889480292) | > loss_mel_ce: 1.699837565422058 (nan) | > loss: 0.006867677439004183 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.13638784805128332) | > loader_time: 0.0045 (0.004649207417106034)  --> TIME: 2025-02-25 05:55:39 -- STEP: 26304/36938 -- GLOBAL_STEP: 654250 | > loss_text_ce: 0.03158121928572655 (0.035500060296990064) | > loss_mel_ce: 2.910068988800049 (nan) | > loss: 0.0116732157766819 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13639176138416015) | > loader_time: 0.0046 (0.004649285027653055)  --> TIME: 2025-02-25 05:55:48 -- STEP: 26354/36938 -- GLOBAL_STEP: 654300 | > loss_text_ce: 0.025284627452492714 (0.03550032049472214) | > loss_mel_ce: 2.987359046936035 (nan) | > loss: 0.011954935267567635 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.136390222976228) | > loader_time: 0.0048 (0.004649325776662864)  --> TIME: 2025-02-25 05:55:57 -- STEP: 26404/36938 -- GLOBAL_STEP: 654350 | > loss_text_ce: 0.03246143087744713 (0.035499111427656875) | > loss_mel_ce: 3.0078125 (nan) | > loss: 0.012064579874277115 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.13638028401783733) | > loader_time: 0.0045 (0.004649173976832886)  --> TIME: 2025-02-25 05:56:06 -- STEP: 26454/36938 -- GLOBAL_STEP: 654400 | > loss_text_ce: 0.032853592187166214 (0.03549843269139857) | > loss_mel_ce: 3.6761324405670166 (nan) | > loss: 0.014718199148774147 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.13637766194035228) | > loader_time: 0.0047 (0.004649127494940057)  --> TIME: 2025-02-25 05:56:15 -- STEP: 26504/36938 -- GLOBAL_STEP: 654450 | > loss_text_ce: 0.04283733293414116 (0.03549920574411527) | > loss_mel_ce: 2.28167724609375 (nan) | > loss: 0.00922426488250494 (nan) | > current_lr: 5e-06 | > step_time: 0.1782 (0.136377386966932) | > loader_time: 0.0048 (0.0046491771621571636)  --> TIME: 2025-02-25 05:56:25 -- STEP: 26554/36938 -- GLOBAL_STEP: 654500 | > loss_text_ce: 0.0412704236805439 (0.0354989460998301) | > loss_mel_ce: 2.954228401184082 (nan) | > loss: 0.0118869012221694 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1363726338232592) | > loader_time: 0.0045 (0.004649090283856359)  --> TIME: 2025-02-25 05:56:34 -- STEP: 26604/36938 -- GLOBAL_STEP: 654550 | > loss_text_ce: 0.03653431683778763 (0.035495882708832256) | > loss_mel_ce: 2.9123482704162598 (nan) | > loss: 0.01170191541314125 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13636628850351157) | > loader_time: 0.0045 (0.004648963870222161)  --> TIME: 2025-02-25 05:56:43 -- STEP: 26654/36938 -- GLOBAL_STEP: 654600 | > loss_text_ce: 0.038500428199768066 (0.035493537776739366) | > loss_mel_ce: 3.3317618370056152 (nan) | > loss: 0.013374056667089462 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13637311851091743) | > loader_time: 0.0046 (0.004649011534367903)  --> TIME: 2025-02-25 05:56:52 -- STEP: 26704/36938 -- GLOBAL_STEP: 654650 | > loss_text_ce: 0.04509805887937546 (0.03549294418709326) | > loss_mel_ce: 1.0013359785079956 (nan) | > loss: 0.00415251636877656 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.1363701502172574) | > loader_time: 0.0045 (0.0046489677292062045)  --> TIME: 2025-02-25 05:57:02 -- STEP: 26754/36938 -- GLOBAL_STEP: 654700 | > loss_text_ce: 0.04876825585961342 (0.03549412114477756) | > loss_mel_ce: 3.1809792518615723 (nan) | > loss: 0.012816458940505981 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13636735664262625) | > loader_time: 0.0045 (0.004648954395828303)  --> TIME: 2025-02-25 05:57:11 -- STEP: 26804/36938 -- GLOBAL_STEP: 654750 | > loss_text_ce: 0.03504139557480812 (0.035491731585537256) | > loss_mel_ce: 2.9325151443481445 (nan) | > loss: 0.011776018887758255 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.1363622553756628) | > loader_time: 0.0046 (0.004648889904323924)  --> TIME: 2025-02-25 05:57:20 -- STEP: 26854/36938 -- GLOBAL_STEP: 654800 | > loss_text_ce: 0.033013343811035156 (0.035491502204835136) | > loss_mel_ce: 3.499393939971924 (nan) | > loss: 0.014017489738762379 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13635682879462244) | > loader_time: 0.0045 (0.004648789207445421)  --> TIME: 2025-02-25 05:57:29 -- STEP: 26904/36938 -- GLOBAL_STEP: 654850 | > loss_text_ce: 0.02569623850286007 (0.035489207728293164) | > loss_mel_ce: 2.9671921730041504 (nan) | > loss: 0.011876542121171951 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.1363602838911816) | > loader_time: 0.0044 (0.004648690887621875)  --> TIME: 2025-02-25 05:57:39 -- STEP: 26954/36938 -- GLOBAL_STEP: 654900 | > loss_text_ce: 0.02896081656217575 (0.03548991972969059) | > loss_mel_ce: 2.8828585147857666 (nan) | > loss: 0.011554839089512825 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13636019817753264) | > loader_time: 0.0046 (0.0046486894976644185)  --> TIME: 2025-02-25 05:57:48 -- STEP: 27004/36938 -- GLOBAL_STEP: 654950 | > loss_text_ce: 0.024965906515717506 (0.03548988650139482) | > loss_mel_ce: 2.645132303237915 (nan) | > loss: 0.01059562899172306 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13635509882197763) | > loader_time: 0.0045 (0.00464864065692682)  --> TIME: 2025-02-25 05:57:57 -- STEP: 27054/36938 -- GLOBAL_STEP: 655000 | > loss_text_ce: 0.06008443236351013 (0.03548910640106579) | > loss_mel_ce: 0.1098361536860466 (nan) | > loss: 0.0006742880796082318 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1363451569961787) | > loader_time: 0.0046 (0.004648610300681811)  --> TIME: 2025-02-25 05:58:06 -- STEP: 27104/36938 -- GLOBAL_STEP: 655050 | > loss_text_ce: 0.03398464247584343 (0.03548630073800166) | > loss_mel_ce: 2.826221466064453 (nan) | > loss: 0.011350025422871113 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.1363436416442436) | > loader_time: 0.0045 (0.0046486095596795325)  --> TIME: 2025-02-25 05:58:15 -- STEP: 27154/36938 -- GLOBAL_STEP: 655100 | > loss_text_ce: 0.024409959092736244 (0.03548293947012061) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13634628899943554) | > loader_time: 0.0046 (0.0046485947554650705)  --> TIME: 2025-02-25 05:58:24 -- STEP: 27204/36938 -- GLOBAL_STEP: 655150 | > loss_text_ce: 0.03479830175638199 (0.03548331255384349) | > loss_mel_ce: 2.615558385848999 (nan) | > loss: 0.01051728893071413 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13634052916039943) | > loader_time: 0.0046 (0.0046484643808972536)  --> TIME: 2025-02-25 05:58:34 -- STEP: 27254/36938 -- GLOBAL_STEP: 655200 | > loss_text_ce: 0.035378631204366684 (0.035482932488011855) | > loss_mel_ce: 3.5037739276885986 (nan) | > loss: 0.014044256880879402 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.1363392800974048) | > loader_time: 0.0048 (0.004648368846926618)  --> TIME: 2025-02-25 05:58:43 -- STEP: 27304/36938 -- GLOBAL_STEP: 655250 | > loss_text_ce: 0.030125536024570465 (0.035482699154714345) | > loss_mel_ce: 3.263766050338745 (nan) | > loss: 0.013070999644696712 (nan) | > current_lr: 5e-06 | > step_time: 0.1862 (0.1363411233060481) | > loader_time: 0.0049 (0.004648470988624292)  --> TIME: 2025-02-25 05:58:53 -- STEP: 27354/36938 -- GLOBAL_STEP: 655300 | > loss_text_ce: 0.03886348381638527 (0.03548165766385144) | > loss_mel_ce: 1.7673029899597168 (nan) | > loss: 0.007167327683418989 (nan) | > current_lr: 5e-06 | > step_time: 0.1778 (0.13634795436404387) | > loader_time: 0.0048 (0.004648574953357433)  --> TIME: 2025-02-25 05:59:02 -- STEP: 27404/36938 -- GLOBAL_STEP: 655350 | > loss_text_ce: 0.036355938762426376 (0.03548044027313245) | > loss_mel_ce: 3.119413375854492 (nan) | > loss: 0.012522894889116287 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13634388726256194) | > loader_time: 0.0048 (0.004648593999472619)  --> TIME: 2025-02-25 05:59:11 -- STEP: 27454/36938 -- GLOBAL_STEP: 655400 | > loss_text_ce: 0.034227002412080765 (0.03547991476598195) | > loss_mel_ce: 1.6639608144760132 (nan) | > loss: 0.006738841067999601 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.1363428679472699) | > loader_time: 0.0046 (0.004648675459701052)  --> TIME: 2025-02-25 05:59:21 -- STEP: 27504/36938 -- GLOBAL_STEP: 655450 | > loss_text_ce: 0.033261872828006744 (0.03548085027328413) | > loss_mel_ce: 2.9348504543304443 (nan) | > loss: 0.011778224259614944 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.13634459333686) | > loader_time: 0.0048 (0.00464873546393129)  --> TIME: 2025-02-25 05:59:30 -- STEP: 27554/36938 -- GLOBAL_STEP: 655500 | > loss_text_ce: 0.0272549856454134 (0.03547894288677935) | > loss_mel_ce: 3.129035472869873 (nan) | > loss: 0.012524963356554508 (nan) | > current_lr: 5e-06 | > step_time: 0.1703 (0.13633877213309026) | > loader_time: 0.0049 (0.00464875986053971)  --> TIME: 2025-02-25 05:59:39 -- STEP: 27604/36938 -- GLOBAL_STEP: 655550 | > loss_text_ce: 0.035513680428266525 (0.035480828963968036) | > loss_mel_ce: 2.4490692615509033 (nan) | > loss: 0.009859456680715084 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13633619372593436) | > loader_time: 0.0047 (0.004648771092194153)  --> TIME: 2025-02-25 05:59:48 -- STEP: 27654/36938 -- GLOBAL_STEP: 655600 | > loss_text_ce: 0.038266442716121674 (0.0354815953907706) | > loss_mel_ce: 2.6612870693206787 (nan) | > loss: 0.010712514631450176 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13633719995567123) | > loader_time: 0.0046 (0.004648843021603868)  --> TIME: 2025-02-25 05:59:58 -- STEP: 27704/36938 -- GLOBAL_STEP: 655650 | > loss_text_ce: 0.03348330780863762 (0.035481024271600985) | > loss_mel_ce: 3.131850481033325 (nan) | > loss: 0.012560849077999592 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13633511028914838) | > loader_time: 0.0045 (0.004648765154804681)  --> TIME: 2025-02-25 06:00:07 -- STEP: 27754/36938 -- GLOBAL_STEP: 655700 | > loss_text_ce: 0.029541442170739174 (0.035478582927357095) | > loss_mel_ce: 0.0006561279296875 (nan) | > loss: 0.00011983163130935282 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13632550730591417) | > loader_time: 0.0046 (0.004648720040362232)  --> TIME: 2025-02-25 06:00:16 -- STEP: 27804/36938 -- GLOBAL_STEP: 655750 | > loss_text_ce: 0.011978879570960999 (0.03547714600886055) | > loss_mel_ce: 2.290900945663452 (nan) | > loss: 0.009138412773609161 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13633332686739022) | > loader_time: 0.0045 (0.004648725457576304)  --> TIME: 2025-02-25 06:00:25 -- STEP: 27854/36938 -- GLOBAL_STEP: 655800 | > loss_text_ce: 0.03734808415174484 (0.035475382827340876) | > loss_mel_ce: 2.187279224395752 (nan) | > loss: 0.008827886544167995 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13632777868447302) | > loader_time: 0.0045 (0.0046486667954349)  --> TIME: 2025-02-25 06:00:35 -- STEP: 27904/36938 -- GLOBAL_STEP: 655850 | > loss_text_ce: 0.019621558487415314 (0.035477885127716664) | > loss_mel_ce: 2.4348440170288086 (nan) | > loss: 0.009739943780004978 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13633338281662535) | > loader_time: 0.0045 (0.00464863150696688)  --> TIME: 2025-02-25 06:00:44 -- STEP: 27954/36938 -- GLOBAL_STEP: 655900 | > loss_text_ce: 0.038687508553266525 (0.035476041043104874) | > loss_mel_ce: 2.532489538192749 (nan) | > loss: 0.010203083977103233 (nan) | > current_lr: 5e-06 | > step_time: 0.1854 (0.136328565543016) | > loader_time: 0.0048 (0.00464852462470245)  --> TIME: 2025-02-25 06:00:53 -- STEP: 28004/36938 -- GLOBAL_STEP: 655950 | > loss_text_ce: 0.038002341985702515 (0.03547664963557261) | > loss_mel_ce: 3.3404948711395264 (nan) | > loss: 0.013406735844910145 (nan) | > current_lr: 5e-06 | > step_time: 0.1758 (0.13631961925798264) | > loader_time: 0.0048 (0.00464839853400622)  --> TIME: 2025-02-25 06:01:02 -- STEP: 28054/36938 -- GLOBAL_STEP: 656000 | > loss_text_ce: 0.036410141736269 (0.03547617663775623) | > loss_mel_ce: 3.534844398498535 (nan) | > loss: 0.014171645976603031 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.1363270103765694) | > loader_time: 0.0046 (0.004648363971825778)  --> TIME: 2025-02-25 06:01:12 -- STEP: 28104/36938 -- GLOBAL_STEP: 656050 | > loss_text_ce: 0.034757424145936966 (0.03547595320561751) | > loss_mel_ce: 2.696619749069214 (nan) | > loss: 0.010838799178600311 (nan) | > current_lr: 5e-06 | > step_time: 0.163 (0.1363262693884692) | > loader_time: 0.0047 (0.004648329116936385)  --> TIME: 2025-02-25 06:01:21 -- STEP: 28154/36938 -- GLOBAL_STEP: 656100 | > loss_text_ce: 0.03473583236336708 (0.03547878229932123) | > loss_mel_ce: 3.2162227630615234 (nan) | > loss: 0.01290063001215458 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13632914951733816) | > loader_time: 0.0046 (0.00464832598981662)  --> TIME: 2025-02-25 06:01:30 -- STEP: 28204/36938 -- GLOBAL_STEP: 656150 | > loss_text_ce: 0.03705364093184471 (0.03547828594226505) | > loss_mel_ce: 3.1554839611053467 (nan) | > loss: 0.012668800540268421 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13632220128735148) | > loader_time: 0.0047 (0.00464834467500105)  --> TIME: 2025-02-25 06:01:40 -- STEP: 28254/36938 -- GLOBAL_STEP: 656200 | > loss_text_ce: 0.03179246932268143 (0.0354755332201592) | > loss_mel_ce: 2.744140625 (nan) | > loss: 0.011015607975423336 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13633169762992098) | > loader_time: 0.0046 (0.004648365606174446)  --> TIME: 2025-02-25 06:01:49 -- STEP: 28304/36938 -- GLOBAL_STEP: 656250 | > loss_text_ce: 0.03774857148528099 (0.03547274647033801) | > loss_mel_ce: 3.12520694732666 (nan) | > loss: 0.012551411986351013 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.13633595973910187) | > loader_time: 0.0048 (0.004648306490742863)  --> TIME: 2025-02-25 06:01:59 -- STEP: 28354/36938 -- GLOBAL_STEP: 656300 | > loss_text_ce: 0.032674942165613174 (0.035472635808829554) | > loss_mel_ce: 1.1728602647781372 (nan) | > loss: 0.0047838701866567135 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.1363361682896546) | > loader_time: 0.0045 (0.004648194113260022)  --> TIME: 2025-02-25 06:02:08 -- STEP: 28404/36938 -- GLOBAL_STEP: 656350 | > loss_text_ce: 0.03635066747665405 (0.0354698562489396) | > loss_mel_ce: 2.6892082691192627 (nan) | > loss: 0.010815710760653019 (nan) | > current_lr: 5e-06 | > step_time: 0.1533 (0.13633783120300427) | > loader_time: 0.0046 (0.004648069607810284)  --> TIME: 2025-02-25 06:02:17 -- STEP: 28454/36938 -- GLOBAL_STEP: 656400 | > loss_text_ce: 0.03563934937119484 (0.035471240279736235) | > loss_mel_ce: 2.867419481277466 (nan) | > loss: 0.011520074680447578 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13634036872998007) | > loader_time: 0.0045 (0.004648078985288632)  --> TIME: 2025-02-25 06:02:26 -- STEP: 28504/36938 -- GLOBAL_STEP: 656450 | > loss_text_ce: 0.030986253172159195 (0.03547288575329925) | > loss_mel_ce: 0.0185546875 (nan) | > loss: 0.00019659104873426259 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13633408260050936) | > loader_time: 0.0045 (0.004648217793900136)  --> TIME: 2025-02-25 06:02:36 -- STEP: 28554/36938 -- GLOBAL_STEP: 656500 | > loss_text_ce: 0.032730020582675934 (0.03547195708135843) | > loss_mel_ce: 2.921412944793701 (nan) | > loss: 0.011722791008651257 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1363390200260155) | > loader_time: 0.0045 (0.004648209837226622)  --> TIME: 2025-02-25 06:02:45 -- STEP: 28604/36938 -- GLOBAL_STEP: 656550 | > loss_text_ce: 0.040201473981142044 (0.03547257862076357) | > loss_mel_ce: 0.05037689208984375 (nan) | > loss: 0.00035943795228376985 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.1363350069701219) | > loader_time: 0.0045 (0.004648291844618828)  --> TIME: 2025-02-25 06:02:54 -- STEP: 28654/36938 -- GLOBAL_STEP: 656600 | > loss_text_ce: 0.026632381603121758 (0.035470066941822816) | > loss_mel_ce: 0.138519287109375 (nan) | > loss: 0.0006553638377226889 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13633316620084887) | > loader_time: 0.0046 (0.004648336697217175)  --> TIME: 2025-02-25 06:03:03 -- STEP: 28704/36938 -- GLOBAL_STEP: 656650 | > loss_text_ce: 0.026287728920578957 (0.03547098788621241) | > loss_mel_ce: 2.2484114170074463 (nan) | > loss: 0.009026585146784782 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.1363232215096044) | > loader_time: 0.0047 (0.004648257366723167)  --> TIME: 2025-02-25 06:03:13 -- STEP: 28754/36938 -- GLOBAL_STEP: 656700 | > loss_text_ce: 0.04292336851358414 (0.035469980083067766) | > loss_mel_ce: 3.1006977558135986 (nan) | > loss: 0.01247468777000904 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13631870438659954) | > loader_time: 0.0045 (0.0046482978033291385)  --> TIME: 2025-02-25 06:03:22 -- STEP: 28804/36938 -- GLOBAL_STEP: 656750 | > loss_text_ce: 0.034368645399808884 (0.03546803499932963) | > loss_mel_ce: 2.5187506675720215 (nan) | > loss: 0.010131427086889744 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13631600715967943) | > loader_time: 0.0045 (0.004648325840908022)  --> TIME: 2025-02-25 06:03:31 -- STEP: 28854/36938 -- GLOBAL_STEP: 656800 | > loss_text_ce: 0.029233697801828384 (0.03546697381203993) | > loss_mel_ce: 2.8780109882354736 (nan) | > loss: 0.01153668574988842 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1363203657419564) | > loader_time: 0.0046 (0.004648305492754108)  --> TIME: 2025-02-25 06:03:41 -- STEP: 28904/36938 -- GLOBAL_STEP: 656850 | > loss_text_ce: 0.03303520381450653 (0.035465241086337175) | > loss_mel_ce: 3.326904296875 (nan) | > loss: 0.01333309430629015 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.1363218191783239) | > loader_time: 0.0045 (0.004648231582367079)  --> TIME: 2025-02-25 06:03:50 -- STEP: 28954/36938 -- GLOBAL_STEP: 656900 | > loss_text_ce: 0.028887391090393066 (0.03546622906395137) | > loss_mel_ce: 2.3450772762298584 (nan) | > loss: 0.009420495480298996 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.1363185524586402) | > loader_time: 0.0046 (0.0046481148778482265)  --> TIME: 2025-02-25 06:03:59 -- STEP: 29004/36938 -- GLOBAL_STEP: 656950 | > loss_text_ce: 0.034615352749824524 (0.03546583158616674) | > loss_mel_ce: 0.026611328125 (nan) | > loss: 0.00024296304036397487 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13632180705232833) | > loader_time: 0.0045 (0.004648158910635858)  --> TIME: 2025-02-25 06:04:08 -- STEP: 29054/36938 -- GLOBAL_STEP: 657000 | > loss_text_ce: 0.0255288016051054 (0.035464166479464015) | > loss_mel_ce: 2.7357611656188965 (nan) | > loss: 0.010957500897347927 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.13631799989617369) | > loader_time: 0.0046 (0.0046482123847410425)  --> TIME: 2025-02-25 06:04:17 -- STEP: 29104/36938 -- GLOBAL_STEP: 657050 | > loss_text_ce: 0.023364994674921036 (0.03546552923600165) | > loss_mel_ce: 3.4027798175811768 (nan) | > loss: 0.013595813885331154 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13631276400486309) | > loader_time: 0.0046 (0.004648206062274944)  --> TIME: 2025-02-25 06:04:26 -- STEP: 29154/36938 -- GLOBAL_STEP: 657100 | > loss_text_ce: 0.04344915598630905 (0.03546647932726099) | > loss_mel_ce: 3.003371238708496 (nan) | > loss: 0.012090558186173439 (nan) | > current_lr: 5e-06 | > step_time: 0.1505 (0.1363050225847915) | > loader_time: 0.0047 (0.0046482746383736005)  --> TIME: 2025-02-25 06:04:35 -- STEP: 29204/36938 -- GLOBAL_STEP: 657150 | > loss_text_ce: 0.03845410421490669 (0.03546592411370867) | > loss_mel_ce: 3.559518575668335 (nan) | > loss: 0.014277669601142406 (nan) | > current_lr: 5e-06 | > step_time: 0.1747 (0.13629769749909745) | > loader_time: 0.0048 (0.004648416022081397)  --> TIME: 2025-02-25 06:04:45 -- STEP: 29254/36938 -- GLOBAL_STEP: 657200 | > loss_text_ce: 0.032902102917432785 (0.035464820324914555) | > loss_mel_ce: 3.093195915222168 (nan) | > loss: 0.01240515150129795 (nan) | > current_lr: 5e-06 | > step_time: 0.1866 (0.13630554826340846) | > loader_time: 0.0049 (0.004648394241379944)  --> TIME: 2025-02-25 06:04:54 -- STEP: 29304/36938 -- GLOBAL_STEP: 657250 | > loss_text_ce: 0.05150358006358147 (0.03546670035979952) | > loss_mel_ce: 1.422876000404358 (nan) | > loss: 0.0058507127687335014 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13629071626414468) | > loader_time: 0.0047 (0.004648241886440398)  --> TIME: 2025-02-25 06:05:03 -- STEP: 29354/36938 -- GLOBAL_STEP: 657300 | > loss_text_ce: 0.028765248134732246 (0.03546722307140702) | > loss_mel_ce: 2.2686026096343994 (nan) | > loss: 0.009116539731621742 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13629529063071405) | > loader_time: 0.0045 (0.004648244095862274)  --> TIME: 2025-02-25 06:05:13 -- STEP: 29404/36938 -- GLOBAL_STEP: 657350 | > loss_text_ce: 0.032115738838911057 (0.03546689861268354) | > loss_mel_ce: 3.0679774284362793 (nan) | > loss: 0.012301957234740257 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13630455237734768) | > loader_time: 0.0049 (0.004648316516274248)  --> TIME: 2025-02-25 06:05:22 -- STEP: 29454/36938 -- GLOBAL_STEP: 657400 | > loss_text_ce: 0.030839940533041954 (0.03546552397372093) | > loss_mel_ce: 2.998591661453247 (nan) | > loss: 0.012021554633975029 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13629624066838741) | > loader_time: 0.0045 (0.004648191150000352)  --> TIME: 2025-02-25 06:05:31 -- STEP: 29504/36938 -- GLOBAL_STEP: 657450 | > loss_text_ce: 0.03425228223204613 (0.03546545540888325) | > loss_mel_ce: 3.4854071140289307 (nan) | > loss: 0.01396690309047699 (nan) | > current_lr: 5e-06 | > step_time: 0.1859 (0.13629874107862136) | > loader_time: 0.005 (0.004648169288472335)  --> TIME: 2025-02-25 06:05:41 -- STEP: 29554/36938 -- GLOBAL_STEP: 657500 | > loss_text_ce: 0.04191847890615463 (0.035466552050795117) | > loss_mel_ce: 2.5290274620056152 (nan) | > loss: 0.010202167555689812 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.1363056684981138) | > loader_time: 0.0046 (0.004648223784535061)  --> TIME: 2025-02-25 06:05:50 -- STEP: 29604/36938 -- GLOBAL_STEP: 657550 | > loss_text_ce: 0.04311903938651085 (0.0354653938065392) | > loss_mel_ce: 2.4853549003601074 (nan) | > loss: 0.010033627040684223 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.13630361967578916) | > loader_time: 0.0046 (0.004648242105005424)  --> TIME: 2025-02-25 06:05:59 -- STEP: 29654/36938 -- GLOBAL_STEP: 657600 | > loss_text_ce: 0.03208741173148155 (0.035466376358333426) | > loss_mel_ce: 2.9350528717041016 (nan) | > loss: 0.011774366721510887 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13630849633943887) | > loader_time: 0.0045 (0.004648280688850743)  --> TIME: 2025-02-25 06:06:09 -- STEP: 29704/36938 -- GLOBAL_STEP: 657650 | > loss_text_ce: 0.02921408787369728 (0.03546533092089895) | > loss_mel_ce: 3.084630250930786 (nan) | > loss: 0.012356526218354702 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13630749339399734) | > loader_time: 0.0045 (0.004648262459795775)  --> TIME: 2025-02-25 06:06:18 -- STEP: 29754/36938 -- GLOBAL_STEP: 657700 | > loss_text_ce: 0.02818065509200096 (0.035467575152839916) | > loss_mel_ce: 2.0701801776885986 (nan) | > loss: 0.00832682941108942 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13630733230689684) | > loader_time: 0.0046 (0.004648406018234505)  --> TIME: 2025-02-25 06:06:27 -- STEP: 29804/36938 -- GLOBAL_STEP: 657750 | > loss_text_ce: 0.02542475424706936 (0.035466497333765744) | > loss_mel_ce: 2.952759027481079 (nan) | > loss: 0.011818190105259418 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13631102759923436) | > loader_time: 0.0045 (0.004648433461507324)  --> TIME: 2025-02-25 06:06:37 -- STEP: 29854/36938 -- GLOBAL_STEP: 657800 | > loss_text_ce: 0.03261467441916466 (0.03546538142242972) | > loss_mel_ce: 3.050055980682373 (nan) | > loss: 0.012232820503413677 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13631215114367237) | > loader_time: 0.0046 (0.004648431335968503)  --> TIME: 2025-02-25 06:06:46 -- STEP: 29904/36938 -- GLOBAL_STEP: 657850 | > loss_text_ce: 0.04372972995042801 (0.035467712944119524) | > loss_mel_ce: 2.7093396186828613 (nan) | > loss: 0.01092487946152687 (nan) | > current_lr: 5e-06 | > step_time: 0.1825 (0.136303910641761) | > loader_time: 0.0048 (0.0046484465424296234)  --> TIME: 2025-02-25 06:06:55 -- STEP: 29954/36938 -- GLOBAL_STEP: 657900 | > loss_text_ce: 0.04539762809872627 (0.03546788463905175) | > loss_mel_ce: 2.6342852115631104 (nan) | > loss: 0.010633662343025208 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13629596697565297) | > loader_time: 0.0045 (0.004648398189349822)  --> TIME: 2025-02-25 06:07:04 -- STEP: 30004/36938 -- GLOBAL_STEP: 657950 | > loss_text_ce: 0.04403657466173172 (0.035466322509004417) | > loss_mel_ce: 0.0708182230591774 (nan) | > loss: 0.00045577302807942033 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13629403159294376) | > loader_time: 0.0045 (0.0046484404811063165)  --> TIME: 2025-02-25 06:07:14 -- STEP: 30054/36938 -- GLOBAL_STEP: 658000 | > loss_text_ce: 0.03512483835220337 (0.0354676209286974) | > loss_mel_ce: 3.238764524459839 (nan) | > loss: 0.012991624884307384 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13629528803764007) | > loader_time: 0.0047 (0.004648432828728411)  --> TIME: 2025-02-25 06:07:23 -- STEP: 30104/36938 -- GLOBAL_STEP: 658050 | > loss_text_ce: 0.035303954035043716 (0.03546686821465365) | > loss_mel_ce: 2.896054744720459 (nan) | > loss: 0.011632377281785011 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13628759105060317) | > loader_time: 0.0046 (0.004648415341581265)  --> TIME: 2025-02-25 06:07:32 -- STEP: 30154/36938 -- GLOBAL_STEP: 658100 | > loss_text_ce: 0.03143158555030823 (0.03546586949691448) | > loss_mel_ce: 3.0984060764312744 (nan) | > loss: 0.012419992126524448 (nan) | > current_lr: 5e-06 | > step_time: 0.1814 (0.13628598953768659) | > loader_time: 0.0048 (0.004648449772460751)  --> TIME: 2025-02-25 06:07:41 -- STEP: 30204/36938 -- GLOBAL_STEP: 658150 | > loss_text_ce: 0.06048591062426567 (0.03546565472739903) | > loss_mel_ce: 1.0526381731033325 (nan) | > loss: 0.004417159594595432 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13628172085878976) | > loader_time: 0.0045 (0.00464833038498306)  --> TIME: 2025-02-25 06:07:50 -- STEP: 30254/36938 -- GLOBAL_STEP: 658200 | > loss_text_ce: 0.03280008211731911 (0.0354648703951389) | > loss_mel_ce: 2.5858826637268066 (nan) | > loss: 0.010391599498689175 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13627989188751652) | > loader_time: 0.0045 (0.004648448266111119)  --> TIME: 2025-02-25 06:07:59 -- STEP: 30304/36938 -- GLOBAL_STEP: 658250 | > loss_text_ce: 0.03982643038034439 (0.035463593720940254) | > loss_mel_ce: 3.8939902782440186 (nan) | > loss: 0.01561038475483656 (nan) | > current_lr: 5e-06 | > step_time: 0.1823 (0.1362810426302155) | > loader_time: 0.0048 (0.004648339432106595)  --> TIME: 2025-02-25 06:08:09 -- STEP: 30354/36938 -- GLOBAL_STEP: 658300 | > loss_text_ce: 0.025857197120785713 (0.0354633506588073) | > loss_mel_ce: 2.7683684825897217 (nan) | > loss: 0.01108819805085659 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13627985526491518) | > loader_time: 0.0046 (0.004648338855314351)  --> TIME: 2025-02-25 06:08:18 -- STEP: 30404/36938 -- GLOBAL_STEP: 658350 | > loss_text_ce: 0.04082471877336502 (0.035460897448943714) | > loss_mel_ce: 1.796240210533142 (nan) | > loss: 0.00728994095697999 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13628321412424746) | > loader_time: 0.0045 (0.004648288493563566)  --> TIME: 2025-02-25 06:08:27 -- STEP: 30454/36938 -- GLOBAL_STEP: 658400 | > loss_text_ce: 0.03436289727687836 (0.035461532634934435) | > loss_mel_ce: 3.078415632247925 (nan) | > loss: 0.012352296151220798 (nan) | > current_lr: 5e-06 | > step_time: 0.1629 (0.13628150200651096) | > loader_time: 0.0046 (0.00464839721419645)  --> TIME: 2025-02-25 06:08:37 -- STEP: 30504/36938 -- GLOBAL_STEP: 658450 | > loss_text_ce: 0.03402859717607498 (0.03546070689080852) | > loss_mel_ce: 2.8778223991394043 (nan) | > loss: 0.011554964818060398 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.13628597240606694) | > loader_time: 0.0046 (0.004648397069198697)  --> TIME: 2025-02-25 06:08:46 -- STEP: 30554/36938 -- GLOBAL_STEP: 658500 | > loss_text_ce: 0.03980812057852745 (0.03545882344142169) | > loss_mel_ce: 3.4077022075653076 (nan) | > loss: 0.013680596835911274 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13628211132696663) | > loader_time: 0.0045 (0.004648388942015088)  --> TIME: 2025-02-25 06:08:55 -- STEP: 30604/36938 -- GLOBAL_STEP: 658550 | > loss_text_ce: 0.026659773662686348 (0.03545906356136121) | > loss_mel_ce: 2.2662901878356934 (nan) | > loss: 0.00909900851547718 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13627925157983137) | > loader_time: 0.0045 (0.004648560364251659)  --> TIME: 2025-02-25 06:09:05 -- STEP: 30654/36938 -- GLOBAL_STEP: 658600 | > loss_text_ce: 0.041614387184381485 (0.03545571889734036) | > loss_mel_ce: 3.287407398223877 (nan) | > loss: 0.01321040466427803 (nan) | > current_lr: 5e-06 | > step_time: 0.145 (0.1362799983937068) | > loader_time: 0.0045 (0.00464859797139919)  --> TIME: 2025-02-25 06:09:14 -- STEP: 30704/36938 -- GLOBAL_STEP: 658650 | > loss_text_ce: 0.03493437543511391 (0.035453885395756786) | > loss_mel_ce: 3.0997884273529053 (nan) | > loss: 0.012439376674592495 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.1362852496194479) | > loader_time: 0.0047 (0.00464870442537791)  --> TIME: 2025-02-25 06:09:24 -- STEP: 30754/36938 -- GLOBAL_STEP: 658700 | > loss_text_ce: 0.03438931331038475 (0.03545546584554363) | > loss_mel_ce: 2.7580668926239014 (nan) | > loss: 0.01108117587864399 (nan) | > current_lr: 5e-06 | > step_time: 0.1512 (0.13629260750502706) | > loader_time: 0.0046 (0.004648698270107693)  --> TIME: 2025-02-25 06:09:33 -- STEP: 30804/36938 -- GLOBAL_STEP: 658750 | > loss_text_ce: 0.03427008539438248 (0.03545436140332098) | > loss_mel_ce: 3.1967601776123047 (nan) | > loss: 0.01282154954969883 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13629271319710826) | > loader_time: 0.0047 (0.004648626740761128)  --> TIME: 2025-02-25 06:09:42 -- STEP: 30854/36938 -- GLOBAL_STEP: 658800 | > loss_text_ce: 0.034222736954689026 (0.03545305266738637) | > loss_mel_ce: 2.5280723571777344 (nan) | > loss: 0.010167839005589485 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.1362893529462811) | > loader_time: 0.0046 (0.004648588964339007)  --> TIME: 2025-02-25 06:09:52 -- STEP: 30904/36938 -- GLOBAL_STEP: 658850 | > loss_text_ce: 0.03521177917718887 (0.03545397342398606) | > loss_mel_ce: 3.3787477016448975 (nan) | > loss: 0.013547458685934544 (nan) | > current_lr: 5e-06 | > step_time: 0.1775 (0.13629475886297635) | > loader_time: 0.0048 (0.004648616716338962)  --> TIME: 2025-02-25 06:10:01 -- STEP: 30954/36938 -- GLOBAL_STEP: 658900 | > loss_text_ce: 0.0352700911462307 (0.035452259353261635) | > loss_mel_ce: 2.2094132900238037 (nan) | > loss: 0.008907473646104336 (nan) | > current_lr: 5e-06 | > step_time: 0.1786 (0.1362950744697907) | > loader_time: 0.0049 (0.004648569542639022)  --> TIME: 2025-02-25 06:10:11 -- STEP: 31004/36938 -- GLOBAL_STEP: 658950 | > loss_text_ce: 0.038340236991643906 (0.035452951916102295) | > loss_mel_ce: 3.011859178543091 (nan) | > loss: 0.012103967368602753 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13629212394527707) | > loader_time: 0.0046 (0.004648482002853309)  --> TIME: 2025-02-25 06:10:20 -- STEP: 31054/36938 -- GLOBAL_STEP: 659000 | > loss_text_ce: 0.03184268996119499 (0.035452030296124204) | > loss_mel_ce: 3.302241325378418 (nan) | > loss: 0.013230493292212486 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.13629040750631294) | > loader_time: 0.0047 (0.004648403151878143)  --> TIME: 2025-02-25 06:10:29 -- STEP: 31104/36938 -- GLOBAL_STEP: 659050 | > loss_text_ce: 0.03552522137761116 (0.03545144061749475) | > loss_mel_ce: 3.1813807487487793 (nan) | > loss: 0.012765500694513321 (nan) | > current_lr: 5e-06 | > step_time: 0.1265 (0.13629316148413184) | > loader_time: 0.0045 (0.0046483049008213375)  --> TIME: 2025-02-25 06:10:39 -- STEP: 31154/36938 -- GLOBAL_STEP: 659100 | > loss_text_ce: 0.03374577313661575 (0.03545188796934008) | > loss_mel_ce: 2.9972031116485596 (nan) | > loss: 0.012027575634419918 (nan) | > current_lr: 5e-06 | > step_time: 0.1868 (0.13630630856432) | > loader_time: 0.0048 (0.004648291292488701)  --> TIME: 2025-02-25 06:10:48 -- STEP: 31204/36938 -- GLOBAL_STEP: 659150 | > loss_text_ce: 0.04391847923398018 (0.035453497199282126) | > loss_mel_ce: 0.02855682373046875 (nan) | > loss: 0.00028760041459463537 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13630046323848985) | > loader_time: 0.0045 (0.004648169666173404)  --> TIME: 2025-02-25 06:10:57 -- STEP: 31254/36938 -- GLOBAL_STEP: 659200 | > loss_text_ce: 0.03043276071548462 (0.035452128045930696) | > loss_mel_ce: 1.9234042167663574 (nan) | > loss: 0.007753321435302496 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13629939111079792) | > loader_time: 0.0045 (0.00464818342912751)  --> TIME: 2025-02-25 06:11:07 -- STEP: 31304/36938 -- GLOBAL_STEP: 659250 | > loss_text_ce: 0.03388892114162445 (0.035455268259786815) | > loss_mel_ce: 3.0162909030914307 (nan) | > loss: 0.01210388820618391 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13629743015098839) | > loader_time: 0.0045 (0.004648215800272415)  --> TIME: 2025-02-25 06:11:16 -- STEP: 31354/36938 -- GLOBAL_STEP: 659300 | > loss_text_ce: 0.02767559140920639 (0.035453828434118616) | > loss_mel_ce: 1.5124484300613403 (nan) | > loss: 0.006111603695899248 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13629567226965314) | > loader_time: 0.0045 (0.004648179524922885)  --> TIME: 2025-02-25 06:11:25 -- STEP: 31404/36938 -- GLOBAL_STEP: 659350 | > loss_text_ce: 0.03546079248189926 (0.035451903423289126) | > loss_mel_ce: 3.2533771991729736 (nan) | > loss: 0.013050944544374943 (nan) | > current_lr: 5e-06 | > step_time: 0.1502 (0.13629920745166088) | > loader_time: 0.0046 (0.004648198444909183)  --> TIME: 2025-02-25 06:11:35 -- STEP: 31454/36938 -- GLOBAL_STEP: 659400 | > loss_text_ce: 0.03541148826479912 (0.035451337979072566) | > loss_mel_ce: 2.352768898010254 (nan) | > loss: 0.009476907551288605 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13630855140314801) | > loader_time: 0.0047 (0.0046482807865143964)  --> TIME: 2025-02-25 06:11:45 -- STEP: 31504/36938 -- GLOBAL_STEP: 659450 | > loss_text_ce: 0.03054063767194748 (0.035452941715149414) | > loss_mel_ce: 3.2598583698272705 (nan) | > loss: 0.013057139702141285 (nan) | > current_lr: 5e-06 | > step_time: 0.1667 (0.1363134612024212) | > loader_time: 0.0047 (0.004648301249049283)  --> TIME: 2025-02-25 06:11:54 -- STEP: 31554/36938 -- GLOBAL_STEP: 659500 | > loss_text_ce: 0.022058062255382538 (0.035452416583299874) | > loss_mel_ce: 2.172532320022583 (nan) | > loss: 0.008708692155778408 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13631793709469694) | > loader_time: 0.0046 (0.004648261056046383)  --> TIME: 2025-02-25 06:12:03 -- STEP: 31604/36938 -- GLOBAL_STEP: 659550 | > loss_text_ce: 0.029215456917881966 (0.03545065494715135) | > loss_mel_ce: 3.47036075592041 (nan) | > loss: 0.013887208886444569 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.13631729976449702) | > loader_time: 0.0049 (0.004648191795184958)  --> TIME: 2025-02-25 06:12:13 -- STEP: 31654/36938 -- GLOBAL_STEP: 659600 | > loss_text_ce: 0.03675522282719612 (0.035452663581770245) | > loss_mel_ce: 2.7095398902893066 (nan) | > loss: 0.01089799776673317 (nan) | > current_lr: 5e-06 | > step_time: 0.1788 (0.1363236480091517) | > loader_time: 0.005 (0.004648179732865172)  --> TIME: 2025-02-25 06:12:22 -- STEP: 31704/36938 -- GLOBAL_STEP: 659650 | > loss_text_ce: 0.02660517208278179 (0.03545153168846269) | > loss_mel_ce: 0.030853271484375 (nan) | > loss: 0.00022800971055403352 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1363230767031664) | > loader_time: 0.0045 (0.004648178056307335)  --> TIME: 2025-02-25 06:12:31 -- STEP: 31754/36938 -- GLOBAL_STEP: 659700 | > loss_text_ce: 0.041347768157720566 (0.03544844563039412) | > loss_mel_ce: 2.7619242668151855 (nan) | > loss: 0.011124095879495144 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.13632343284217344) | > loader_time: 0.0045 (0.004648111768593865)  --> TIME: 2025-02-25 06:12:41 -- STEP: 31804/36938 -- GLOBAL_STEP: 659750 | > loss_text_ce: 0.035594966262578964 (0.0354454721297324) | > loss_mel_ce: 2.293774127960205 (nan) | > loss: 0.009243528358638287 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13632159850204745) | > loader_time: 0.0046 (0.004648029586831326)  --> TIME: 2025-02-25 06:12:50 -- STEP: 31854/36938 -- GLOBAL_STEP: 659800 | > loss_text_ce: 0.03437450900673866 (0.035443384366402146) | > loss_mel_ce: 2.537583589553833 (nan) | > loss: 0.010206183418631554 (nan) | > current_lr: 5e-06 | > step_time: 0.1434 (0.136326973838096) | > loader_time: 0.0046 (0.004648030062452903)  --> TIME: 2025-02-25 06:12:59 -- STEP: 31904/36938 -- GLOBAL_STEP: 659850 | > loss_text_ce: 0.03440965712070465 (0.03544150721233909) | > loss_mel_ce: 2.713780403137207 (nan) | > loss: 0.010905517265200615 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13632445280237326) | > loader_time: 0.0046 (0.004647954319460813)  --> TIME: 2025-02-25 06:13:09 -- STEP: 31954/36938 -- GLOBAL_STEP: 659900 | > loss_text_ce: 0.043588198721408844 (0.035440853147882416) | > loss_mel_ce: 2.9674692153930664 (nan) | > loss: 0.011948641389608383 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.13633001716188892) | > loader_time: 0.0046 (0.004647948479722534)  --> TIME: 2025-02-25 06:13:18 -- STEP: 32004/36938 -- GLOBAL_STEP: 659950 | > loss_text_ce: 0.031083757057785988 (0.03543994816170536) | > loss_mel_ce: 0.3940151333808899 (nan) | > loss: 0.0016869005048647523 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13632730109559493) | > loader_time: 0.0048 (0.004647860242759946)  --> TIME: 2025-02-25 06:13:27 -- STEP: 32054/36938 -- GLOBAL_STEP: 660000 | > loss_text_ce: 0.027882955968379974 (0.03543883234942597) | > loss_mel_ce: 2.855123996734619 (nan) | > loss: 0.011440505273640156 (nan) | > current_lr: 5e-06 | > step_time: 0.1084 (0.13632536139737722) | > loader_time: 0.0044 (0.004647795658798244) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_660000.pth  --> TIME: 2025-02-25 06:13:42 -- STEP: 32104/36938 -- GLOBAL_STEP: 660050 | > loss_text_ce: 0.034051474183797836 (0.03543734745855405) | > loss_mel_ce: 2.863821506500244 (nan) | > loss: 0.011499496176838875 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13631610284563256) | > loader_time: 0.0047 (0.004647783313105291)  --> TIME: 2025-02-25 06:13:51 -- STEP: 32154/36938 -- GLOBAL_STEP: 660100 | > loss_text_ce: 0.03167223185300827 (0.035438401268935125) | > loss_mel_ce: 0.004241943359375 (nan) | > loss: 0.00014251658285502344 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13631512663914033) | > loader_time: 0.0046 (0.0046477967874026605)  --> TIME: 2025-02-25 06:14:01 -- STEP: 32204/36938 -- GLOBAL_STEP: 660150 | > loss_text_ce: 0.030040888115763664 (0.03543513846725848) | > loss_mel_ce: 2.8304622173309326 (nan) | > loss: 0.011351203545928001 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13631394864194632) | > loader_time: 0.0047 (0.004647900555951442)  --> TIME: 2025-02-25 06:14:10 -- STEP: 32254/36938 -- GLOBAL_STEP: 660200 | > loss_text_ce: 0.029757380485534668 (0.03543430020167738) | > loss_mel_ce: 1.8900498151779175 (nan) | > loss: 0.007618282921612263 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.1363121109915912) | > loader_time: 0.0046 (0.004647809625181971)  --> TIME: 2025-02-25 06:14:19 -- STEP: 32304/36938 -- GLOBAL_STEP: 660250 | > loss_text_ce: 0.038637544959783554 (0.035433235686178434) | > loss_mel_ce: 2.6653788089752197 (nan) | > loss: 0.010730224661529064 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13630351610204852) | > loader_time: 0.0046 (0.004647686826586421)  --> TIME: 2025-02-25 06:14:28 -- STEP: 32354/36938 -- GLOBAL_STEP: 660300 | > loss_text_ce: 0.04043962433934212 (0.035431122753893436) | > loss_mel_ce: 2.08742356300354 (nan) | > loss: 0.008443902246654034 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13629871389807838) | > loader_time: 0.0045 (0.004647655186992402)  --> TIME: 2025-02-25 06:14:37 -- STEP: 32404/36938 -- GLOBAL_STEP: 660350 | > loss_text_ce: 0.03670996055006981 (0.0354313776491391) | > loss_mel_ce: 2.9301769733428955 (nan) | > loss: 0.011773361824452877 (nan) | > current_lr: 5e-06 | > step_time: 0.1968 (0.13630014991277473) | > loader_time: 0.0049 (0.004647712592150949)  --> TIME: 2025-02-25 06:14:47 -- STEP: 32454/36938 -- GLOBAL_STEP: 660400 | > loss_text_ce: 0.040562961250543594 (0.035430702711536886) | > loss_mel_ce: 2.115753650665283 (nan) | > loss: 0.008556812070310116 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13629959540115175) | > loader_time: 0.0045 (0.004647748170722065)  --> TIME: 2025-02-25 06:14:56 -- STEP: 32504/36938 -- GLOBAL_STEP: 660450 | > loss_text_ce: 0.036405425518751144 (0.03543144810713069) | > loss_mel_ce: 2.882479429244995 (nan) | > loss: 0.011582876555621624 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13629906598699407) | > loader_time: 0.0046 (0.004647734230913723)  --> TIME: 2025-02-25 06:15:06 -- STEP: 32554/36938 -- GLOBAL_STEP: 660500 | > loss_text_ce: 0.03747783973813057 (0.03542791033812409) | > loss_mel_ce: 2.9512555599212646 (nan) | > loss: 0.011860053054988384 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.1363122291556727) | > loader_time: 0.0049 (0.004647868091339275)  --> TIME: 2025-02-25 06:15:15 -- STEP: 32604/36938 -- GLOBAL_STEP: 660550 | > loss_text_ce: 0.021177999675273895 (0.03542912447685237) | > loss_mel_ce: 2.455939769744873 (nan) | > loss: 0.009829833172261715 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13632315649545643) | > loader_time: 0.0045 (0.004647933556366334)  --> TIME: 2025-02-25 06:15:25 -- STEP: 32654/36938 -- GLOBAL_STEP: 660600 | > loss_text_ce: 0.03267543017864227 (0.035429369558066366) | > loss_mel_ce: 2.096778392791748 (nan) | > loss: 0.008450214751064777 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13631730081987198) | > loader_time: 0.0045 (0.004647904334024775)  --> TIME: 2025-02-25 06:15:34 -- STEP: 32704/36938 -- GLOBAL_STEP: 660650 | > loss_text_ce: 0.03718261793255806 (0.03542826461090645) | > loss_mel_ce: 2.3151042461395264 (nan) | > loss: 0.009334472008049488 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13632090617911574) | > loader_time: 0.0045 (0.004647879910504041)  --> TIME: 2025-02-25 06:15:43 -- STEP: 32754/36938 -- GLOBAL_STEP: 660700 | > loss_text_ce: 0.026238402351737022 (0.03542641334580529) | > loss_mel_ce: 2.2573463916778564 (nan) | > loss: 0.009061845019459724 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.1363183171668888) | > loader_time: 0.0045 (0.004647881074681659)  --> TIME: 2025-02-25 06:15:52 -- STEP: 32804/36938 -- GLOBAL_STEP: 660750 | > loss_text_ce: 0.0391533188521862 (0.03542479001466686) | > loss_mel_ce: 1.3587902784347534 (nan) | > loss: 0.005547395441681147 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13631383283387305) | > loader_time: 0.0047 (0.004647894365556959)  --> TIME: 2025-02-25 06:16:02 -- STEP: 32854/36938 -- GLOBAL_STEP: 660800 | > loss_text_ce: 0.03734998404979706 (0.03542353166173745) | > loss_mel_ce: 2.5207788944244385 (nan) | > loss: 0.010151305235922337 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13630942847104952) | > loader_time: 0.0046 (0.00464796146226319)  --> TIME: 2025-02-25 06:16:11 -- STEP: 32904/36938 -- GLOBAL_STEP: 660850 | > loss_text_ce: 0.030764762312173843 (0.035424204818293635) | > loss_mel_ce: 2.44588303565979 (nan) | > loss: 0.009827968664467335 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.1363214088593426) | > loader_time: 0.0046 (0.004648004363929728)  --> TIME: 2025-02-25 06:16:21 -- STEP: 32954/36938 -- GLOBAL_STEP: 660900 | > loss_text_ce: 0.028922686353325844 (0.03542135131661308) | > loss_mel_ce: 3.132535219192505 (nan) | > loss: 0.012545469217002392 (nan) | > current_lr: 5e-06 | > step_time: 0.1704 (0.13632494284630917) | > loader_time: 0.0047 (0.004648032578810208)  --> TIME: 2025-02-25 06:16:30 -- STEP: 33004/36938 -- GLOBAL_STEP: 660950 | > loss_text_ce: 0.025231190025806427 (0.035420842694966224) | > loss_mel_ce: 2.991805076599121 (nan) | > loss: 0.011972366832196712 (nan) | > current_lr: 5e-06 | > step_time: 0.1886 (0.13633072197820034) | > loader_time: 0.005 (0.004647990397692673)  --> TIME: 2025-02-25 06:16:40 -- STEP: 33054/36938 -- GLOBAL_STEP: 661000 | > loss_text_ce: 0.026739828288555145 (0.03542220700381078) | > loss_mel_ce: 1.3306593894958496 (nan) | > loss: 0.005386505275964737 (nan) | > current_lr: 5e-06 | > step_time: 0.1086 (0.13633324297570568) | > loader_time: 0.0044 (0.00464798896060402)  --> TIME: 2025-02-25 06:16:49 -- STEP: 33104/36938 -- GLOBAL_STEP: 661050 | > loss_text_ce: 0.03323263302445412 (0.03542293314851854) | > loss_mel_ce: 3.080130100250244 (nan) | > loss: 0.012354615144431591 (nan) | > current_lr: 5e-06 | > step_time: 0.1511 (0.13632917958016783) | > loader_time: 0.0046 (0.004647999908279938)  --> TIME: 2025-02-25 06:16:58 -- STEP: 33154/36938 -- GLOBAL_STEP: 661100 | > loss_text_ce: 0.04273809865117073 (0.03542307344528058) | > loss_mel_ce: 1.2398579120635986 (nan) | > loss: 0.005089667160063982 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1363324647632349) | > loader_time: 0.0046 (0.00464803151215072)  --> TIME: 2025-02-25 06:17:08 -- STEP: 33204/36938 -- GLOBAL_STEP: 661150 | > loss_text_ce: 0.03819546476006508 (0.0354233730331443) | > loss_mel_ce: 3.14978289604187 (nan) | > loss: 0.012650707736611366 (nan) | > current_lr: 5e-06 | > step_time: 0.1343 (0.1363354834386076) | > loader_time: 0.0046 (0.0046480552516289676)  --> TIME: 2025-02-25 06:17:17 -- STEP: 33254/36938 -- GLOBAL_STEP: 661200 | > loss_text_ce: 0.03987301141023636 (0.03542333318106766) | > loss_mel_ce: 2.929748058319092 (nan) | > loss: 0.011784211732447147 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.1363400309200692) | > loader_time: 0.0046 (0.004648143969690857)  --> TIME: 2025-02-25 06:17:27 -- STEP: 33304/36938 -- GLOBAL_STEP: 661250 | > loss_text_ce: 0.0312601663172245 (0.03542506823230885) | > loss_mel_ce: 2.9500985145568848 (nan) | > loss: 0.011830789037048817 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13633911060708376) | > loader_time: 0.0045 (0.004648062856673272)  --> TIME: 2025-02-25 06:17:36 -- STEP: 33354/36938 -- GLOBAL_STEP: 661300 | > loss_text_ce: 0.04713481664657593 (0.03542533088926148) | > loss_mel_ce: 2.8830349445343018 (nan) | > loss: 0.011627658270299435 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13633702855219557) | > loader_time: 0.0045 (0.004647987004828089)  --> TIME: 2025-02-25 06:17:45 -- STEP: 33404/36938 -- GLOBAL_STEP: 661350 | > loss_text_ce: 0.0385407879948616 (0.035425206223681796) | > loss_mel_ce: 3.180903911590576 (nan) | > loss: 0.012775574810802937 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13634573576793105) | > loader_time: 0.0046 (0.004647967109079485)  --> TIME: 2025-02-25 06:17:55 -- STEP: 33454/36938 -- GLOBAL_STEP: 661400 | > loss_text_ce: 0.04408523812890053 (0.03542535153379708) | > loss_mel_ce: 2.6444900035858154 (nan) | > loss: 0.010668950155377388 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13634643125228907) | > loader_time: 0.0045 (0.004647931900384738)  --> TIME: 2025-02-25 06:18:04 -- STEP: 33504/36938 -- GLOBAL_STEP: 661450 | > loss_text_ce: 0.03541257604956627 (0.03542317951085536) | > loss_mel_ce: 3.2290661334991455 (nan) | > loss: 0.012954281643033028 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.1363459260993711) | > loader_time: 0.0047 (0.00464793259087583)  --> TIME: 2025-02-25 06:18:14 -- STEP: 33554/36938 -- GLOBAL_STEP: 661500 | > loss_text_ce: 0.03827420994639397 (0.03542214099411842) | > loss_mel_ce: 2.8585057258605957 (nan) | > loss: 0.01149515900760889 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.1363461849664886) | > loader_time: 0.0045 (0.0046478814303381084)  --> TIME: 2025-02-25 06:18:23 -- STEP: 33604/36938 -- GLOBAL_STEP: 661550 | > loss_text_ce: 0.031479399651288986 (0.03542020964073077) | > loss_mel_ce: 2.843691825866699 (nan) | > loss: 0.011409410275518894 (nan) | > current_lr: 5e-06 | > step_time: 0.1648 (0.13634816665278077) | > loader_time: 0.0047 (0.004647869493915637)  --> TIME: 2025-02-25 06:18:32 -- STEP: 33654/36938 -- GLOBAL_STEP: 661600 | > loss_text_ce: 0.02680220827460289 (0.03541973159254592) | > loss_mel_ce: 2.7565135955810547 (nan) | > loss: 0.011044904589653015 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13634357327281843) | > loader_time: 0.0049 (0.004647857323753826)  --> TIME: 2025-02-25 06:18:41 -- STEP: 33704/36938 -- GLOBAL_STEP: 661650 | > loss_text_ce: 0.039631154388189316 (0.03541809317429625) | > loss_mel_ce: 3.6173880100250244 (nan) | > loss: 0.01451198197901249 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.1363412381039064) | > loader_time: 0.0046 (0.00464783545602055)  --> TIME: 2025-02-25 06:18:51 -- STEP: 33754/36938 -- GLOBAL_STEP: 661700 | > loss_text_ce: 0.03741544485092163 (0.035415989500128225) | > loss_mel_ce: 1.2046390771865845 (nan) | > loss: 0.00492878770455718 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.1363405470049338) | > loader_time: 0.0045 (0.0046478501426853586)  --> TIME: 2025-02-25 06:19:00 -- STEP: 33804/36938 -- GLOBAL_STEP: 661750 | > loss_text_ce: 0.0260106660425663 (0.03541792005886487) | > loss_mel_ce: 2.288792371749878 (nan) | > loss: 0.009185726754367352 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.1363414299514855) | > loader_time: 0.0046 (0.004647826022783572)  --> TIME: 2025-02-25 06:19:09 -- STEP: 33854/36938 -- GLOBAL_STEP: 661800 | > loss_text_ce: 0.028838548809289932 (0.035420241824521935) | > loss_mel_ce: 2.7865514755249023 (nan) | > loss: 0.01117218378931284 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1363435523919983) | > loader_time: 0.0045 (0.0046477860368318545)  --> TIME: 2025-02-25 06:19:19 -- STEP: 33904/36938 -- GLOBAL_STEP: 661850 | > loss_text_ce: 0.04712961986660957 (0.03542253765973218) | > loss_mel_ce: 4.412089824676514 (nan) | > loss: 0.017695317044854164 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13634143139658783) | > loader_time: 0.0045 (0.004647881425512793)  --> TIME: 2025-02-25 06:19:28 -- STEP: 33954/36938 -- GLOBAL_STEP: 661900 | > loss_text_ce: 0.03103708289563656 (0.03542152833635825) | > loss_mel_ce: 2.9558441638946533 (nan) | > loss: 0.011852703988552094 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13634191407902738) | > loader_time: 0.0047 (0.004647896056279991)  --> TIME: 2025-02-25 06:19:37 -- STEP: 34004/36938 -- GLOBAL_STEP: 661950 | > loss_text_ce: 0.0321212001144886 (0.03542435968077514) | > loss_mel_ce: 2.5417797565460205 (nan) | > loss: 0.01021389290690422 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13634455005584814) | > loader_time: 0.0044 (0.0046478739529129355)  --> TIME: 2025-02-25 06:19:47 -- STEP: 34054/36938 -- GLOBAL_STEP: 662000 | > loss_text_ce: 0.032121218740940094 (0.03542557136806742) | > loss_mel_ce: 2.669055938720703 (nan) | > loss: 0.010718957521021366 (nan) | > current_lr: 5e-06 | > step_time: 0.1501 (0.13634388267717243) | > loader_time: 0.0045 (0.004647812483741545)  --> TIME: 2025-02-25 06:19:56 -- STEP: 34104/36938 -- GLOBAL_STEP: 662050 | > loss_text_ce: 0.03168352320790291 (0.035425468993562616) | > loss_mel_ce: 2.3416547775268555 (nan) | > loss: 0.009418009780347347 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.13634747212240042) | > loader_time: 0.0046 (0.004647781108987735)  --> TIME: 2025-02-25 06:20:05 -- STEP: 34154/36938 -- GLOBAL_STEP: 662100 | > loss_text_ce: 0.03489862009882927 (0.035426781402278514) | > loss_mel_ce: 2.783247470855713 (nan) | > loss: 0.011183119378983974 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13635308589506265) | > loader_time: 0.0045 (0.004647746929109051)  --> TIME: 2025-02-25 06:20:15 -- STEP: 34204/36938 -- GLOBAL_STEP: 662150 | > loss_text_ce: 0.0315750390291214 (0.035427037867558735) | > loss_mel_ce: 3.1868762969970703 (nan) | > loss: 0.01277163252234459 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.1363616943317538) | > loader_time: 0.0049 (0.004647792410312396)  --> TIME: 2025-02-25 06:20:25 -- STEP: 34254/36938 -- GLOBAL_STEP: 662200 | > loss_text_ce: 0.037189994007349014 (0.03542699760516457) | > loss_mel_ce: 2.775054931640625 (nan) | > loss: 0.011159702204167843 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.13636570122416253) | > loader_time: 0.0046 (0.004647829009625326)  --> TIME: 2025-02-25 06:20:34 -- STEP: 34304/36938 -- GLOBAL_STEP: 662250 | > loss_text_ce: 0.0339052751660347 (0.03542705077967175) | > loss_mel_ce: 3.347442626953125 (nan) | > loss: 0.013418047688901424 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13636754408700996) | > loader_time: 0.0046 (0.004647807002679174)  --> TIME: 2025-02-25 06:20:43 -- STEP: 34354/36938 -- GLOBAL_STEP: 662300 | > loss_text_ce: 0.040902260690927505 (0.03542864818320409) | > loss_mel_ce: 2.6912729740142822 (nan) | > loss: 0.010841966606676579 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13637087815936344) | > loader_time: 0.0046 (0.004647826186546156)  --> TIME: 2025-02-25 06:20:52 -- STEP: 34404/36938 -- GLOBAL_STEP: 662350 | > loss_text_ce: 0.026678146794438362 (0.03542798514335643) | > loss_mel_ce: 2.643585205078125 (nan) | > loss: 0.010596283711493015 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13636752212897055) | > loader_time: 0.0046 (0.004647956069382572)  --> TIME: 2025-02-25 06:21:02 -- STEP: 34454/36938 -- GLOBAL_STEP: 662400 | > loss_text_ce: 0.03618631139397621 (0.03542549676363115) | > loss_mel_ce: 2.82049822807312 (nan) | > loss: 0.011336049996316433 (nan) | > current_lr: 5e-06 | > step_time: 0.1714 (0.13636810351782558) | > loader_time: 0.0048 (0.00464793564847433)  --> TIME: 2025-02-25 06:21:11 -- STEP: 34504/36938 -- GLOBAL_STEP: 662450 | > loss_text_ce: 0.03471735492348671 (0.03542381133754108) | > loss_mel_ce: 3.149627923965454 (nan) | > loss: 0.012636291794478893 (nan) | > current_lr: 5e-06 | > step_time: 0.1842 (0.1363777415591535) | > loader_time: 0.0049 (0.004647956718404577)  --> TIME: 2025-02-25 06:21:20 -- STEP: 34554/36938 -- GLOBAL_STEP: 662500 | > loss_text_ce: 0.03915807977318764 (0.03542219034006446) | > loss_mel_ce: 3.6078827381134033 (nan) | > loss: 0.014472384937107563 (nan) | > current_lr: 5e-06 | > step_time: 0.1566 (0.1363719148879278) | > loader_time: 0.0047 (0.0046479634514987635)  --> TIME: 2025-02-25 06:21:29 -- STEP: 34604/36938 -- GLOBAL_STEP: 662550 | > loss_text_ce: 0.039921361953020096 (0.03542348759228092) | > loss_mel_ce: 3.359241485595703 (nan) | > loss: 0.013488741591572762 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.13636350075008694) | > loader_time: 0.0047 (0.004647956812482335)  --> TIME: 2025-02-25 06:21:38 -- STEP: 34654/36938 -- GLOBAL_STEP: 662600 | > loss_text_ce: 0.03245547041296959 (0.03542371773406485) | > loss_mel_ce: 3.283414363861084 (nan) | > loss: 0.013158214278519154 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.13635971709641118) | > loader_time: 0.0048 (0.004647957512915026)  --> TIME: 2025-02-25 06:21:48 -- STEP: 34704/36938 -- GLOBAL_STEP: 662650 | > loss_text_ce: 0.042408786714076996 (0.03542168597162534) | > loss_mel_ce: 2.826125144958496 (nan) | > loss: 0.011383071541786194 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.136362659644013) | > loader_time: 0.0046 (0.004648020089964206)  --> TIME: 2025-02-25 06:21:57 -- STEP: 34754/36938 -- GLOBAL_STEP: 662700 | > loss_text_ce: 0.03897085413336754 (0.03542003008482505) | > loss_mel_ce: 2.5768325328826904 (nan) | > loss: 0.010380173102021217 (nan) | > current_lr: 5e-06 | > step_time: 0.144 (0.13636843905169155) | > loader_time: 0.0046 (0.004648063093236898)  --> TIME: 2025-02-25 06:22:06 -- STEP: 34804/36938 -- GLOBAL_STEP: 662750 | > loss_text_ce: 0.038840070366859436 (0.03541959998838588) | > loss_mel_ce: 2.809182643890381 (nan) | > loss: 0.0113016776740551 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.1363606447941146) | > loader_time: 0.0045 (0.004648031427755856)  --> TIME: 2025-02-25 06:22:16 -- STEP: 34854/36938 -- GLOBAL_STEP: 662800 | > loss_text_ce: 0.04244714975357056 (0.035419497855351756) | > loss_mel_ce: 1.488916039466858 (nan) | > loss: 0.006076838821172714 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13636271322341836) | > loader_time: 0.0045 (0.004648130588648599)  --> TIME: 2025-02-25 06:22:25 -- STEP: 34904/36938 -- GLOBAL_STEP: 662850 | > loss_text_ce: 0.04459960758686066 (0.035419319629027474) | > loss_mel_ce: 0.0481974296271801 (nan) | > loss: 0.00036824223934672773 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13636198847522682) | > loader_time: 0.0045 (0.004648242587210412)  --> TIME: 2025-02-25 06:22:34 -- STEP: 34954/36938 -- GLOBAL_STEP: 662900 | > loss_text_ce: 0.04113386943936348 (0.03541734414151722) | > loss_mel_ce: 3.64225697517395 (nan) | > loss: 0.014616630971431732 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.1363648866334319) | > loader_time: 0.0045 (0.004648240703790773)  --> TIME: 2025-02-25 06:22:44 -- STEP: 35004/36938 -- GLOBAL_STEP: 662950 | > loss_text_ce: 0.04320550709962845 (0.035414460556132225) | > loss_mel_ce: 2.7515056133270264 (nan) | > loss: 0.011090124025940895 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13636674793525758) | > loader_time: 0.0046 (0.004648305718360144)  --> TIME: 2025-02-25 06:22:53 -- STEP: 35054/36938 -- GLOBAL_STEP: 663000 | > loss_text_ce: 0.03693500533699989 (0.035413512265769054) | > loss_mel_ce: 3.261962652206421 (nan) | > loss: 0.01309086475521326 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.13636325086088674) | > loader_time: 0.0047 (0.004648263696433406)  --> TIME: 2025-02-25 06:23:02 -- STEP: 35104/36938 -- GLOBAL_STEP: 663050 | > loss_text_ce: 0.03415164723992348 (0.035413750224610435) | > loss_mel_ce: 2.961216449737549 (nan) | > loss: 0.011886381544172764 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13636445183452833) | > loader_time: 0.0046 (0.004648216910925145)  --> TIME: 2025-02-25 06:23:11 -- STEP: 35154/36938 -- GLOBAL_STEP: 663100 | > loss_text_ce: 0.04786796495318413 (0.03541584973202699) | > loss_mel_ce: 2.7533533573150635 (nan) | > loss: 0.011115958914160728 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13636060872500758) | > loader_time: 0.0045 (0.00464819143227684)  --> TIME: 2025-02-25 06:23:21 -- STEP: 35204/36938 -- GLOBAL_STEP: 663150 | > loss_text_ce: 0.03299471735954285 (0.03541433746944385) | > loss_mel_ce: 2.4084646701812744 (nan) | > loss: 0.009688331745564938 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13635736619893746) | > loader_time: 0.0047 (0.004648242893713114)  --> TIME: 2025-02-25 06:23:30 -- STEP: 35254/36938 -- GLOBAL_STEP: 663200 | > loss_text_ce: 0.029914995655417442 (0.035418310490133574) | > loss_mel_ce: 2.828993320465088 (nan) | > loss: 0.01134487520903349 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13635272024682274) | > loader_time: 0.0048 (0.004648252461916498)  --> TIME: 2025-02-25 06:23:39 -- STEP: 35304/36938 -- GLOBAL_STEP: 663250 | > loss_text_ce: 0.0356299951672554 (0.03541864055066846) | > loss_mel_ce: 3.037351608276367 (nan) | > loss: 0.012194371782243252 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.1363461231823376) | > loader_time: 0.0048 (0.004648260152612771)  --> TIME: 2025-02-25 06:23:48 -- STEP: 35354/36938 -- GLOBAL_STEP: 663300 | > loss_text_ce: 0.04073169082403183 (0.03541780597206671) | > loss_mel_ce: 1.8855952024459839 (nan) | > loss: 0.007644154597073793 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13634867168979287) | > loader_time: 0.0046 (0.004648300548980109)  --> TIME: 2025-02-25 06:23:58 -- STEP: 35404/36938 -- GLOBAL_STEP: 663350 | > loss_text_ce: 0.031266435980796814 (0.03541883920503427) | > loss_mel_ce: 2.273177146911621 (nan) | > loss: 0.009144618175923824 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13634530829381142) | > loader_time: 0.0045 (0.0046483191941743725)  --> TIME: 2025-02-25 06:24:07 -- STEP: 35454/36938 -- GLOBAL_STEP: 663400 | > loss_text_ce: 0.03848456218838692 (0.03541792410201996) | > loss_mel_ce: 3.1416656970977783 (nan) | > loss: 0.012619644403457642 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.13635149508178274) | > loader_time: 0.0046 (0.004648336784794042)  --> TIME: 2025-02-25 06:24:16 -- STEP: 35504/36938 -- GLOBAL_STEP: 663450 | > loss_text_ce: 0.031941842287778854 (0.03541853458162307) | > loss_mel_ce: 3.0019938945770264 (nan) | > loss: 0.01203942857682705 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13634883910215992) | > loader_time: 0.0047 (0.00464833603350093)  --> TIME: 2025-02-25 06:24:26 -- STEP: 35554/36938 -- GLOBAL_STEP: 663500 | > loss_text_ce: 0.032073911279439926 (0.035415856920768755) | > loss_mel_ce: 2.4379384517669678 (nan) | > loss: 0.009801637381315231 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13635189229325959) | > loader_time: 0.0045 (0.004648351384984936)  --> TIME: 2025-02-25 06:24:35 -- STEP: 35604/36938 -- GLOBAL_STEP: 663550 | > loss_text_ce: 0.04100222513079643 (0.03541679350000508) | > loss_mel_ce: 2.5521321296691895 (nan) | > loss: 0.010290216654539108 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13634680971082944) | > loader_time: 0.0045 (0.004648291573151093)  --> TIME: 2025-02-25 06:24:44 -- STEP: 35654/36938 -- GLOBAL_STEP: 663600 | > loss_text_ce: 0.027053227648139 (0.035416529968403235) | > loss_mel_ce: 1.376495361328125 (nan) | > loss: 0.005569637753069401 (nan) | > current_lr: 5e-06 | > step_time: 0.1121 (0.1363490659710309) | > loader_time: 0.0046 (0.00464829377052247)  --> TIME: 2025-02-25 06:24:54 -- STEP: 35704/36938 -- GLOBAL_STEP: 663650 | > loss_text_ce: 0.034565407782793045 (0.03541738308538179) | > loss_mel_ce: 1.6919894218444824 (nan) | > loss: 0.006851408630609512 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1363513273974305) | > loader_time: 0.0045 (0.004648250667284412)  --> TIME: 2025-02-25 06:25:03 -- STEP: 35754/36938 -- GLOBAL_STEP: 663700 | > loss_text_ce: 0.0314963199198246 (0.03541868231820114) | > loss_mel_ce: 1.0780140161514282 (nan) | > loss: 0.004402819089591503 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13635064323669713) | > loader_time: 0.0046 (0.004648333982299193)  --> TIME: 2025-02-25 06:25:12 -- STEP: 35804/36938 -- GLOBAL_STEP: 663750 | > loss_text_ce: 0.03277262672781944 (0.03541789951822386) | > loss_mel_ce: 0.068267822265625 (nan) | > loss: 0.00040095418808050454 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.1363484661761043) | > loader_time: 0.0046 (0.004648322067424832)  --> TIME: 2025-02-25 06:25:21 -- STEP: 35854/36938 -- GLOBAL_STEP: 663800 | > loss_text_ce: 0.031027887016534805 (0.0354160385460071) | > loss_mel_ce: 3.340834379196167 (nan) | > loss: 0.01338040642440319 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13634585539588448) | > loader_time: 0.0046 (0.00464831325129684)  --> TIME: 2025-02-25 06:25:31 -- STEP: 35904/36938 -- GLOBAL_STEP: 663850 | > loss_text_ce: 0.027014121413230896 (0.03541239185173472) | > loss_mel_ce: 1.3852452039718628 (nan) | > loss: 0.005604204256087542 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.1363531230698175) | > loader_time: 0.0044 (0.0046483692239950155)  --> TIME: 2025-02-25 06:25:40 -- STEP: 35954/36938 -- GLOBAL_STEP: 663900 | > loss_text_ce: 0.04143386706709862 (0.03541228646227971) | > loss_mel_ce: 2.9859070777893066 (nan) | > loss: 0.012013258412480354 (nan) | > current_lr: 5e-06 | > step_time: 0.184 (0.13635530088915465) | > loader_time: 0.0048 (0.004648425664348437)  --> TIME: 2025-02-25 06:25:50 -- STEP: 36004/36938 -- GLOBAL_STEP: 663950 | > loss_text_ce: 0.04064534977078438 (0.035413208740158526) | > loss_mel_ce: 2.978149890899658 (nan) | > loss: 0.011979347094893456 (nan) | > current_lr: 5e-06 | > step_time: 0.181 (0.13636111148978822) | > loader_time: 0.0049 (0.0046484374017294935)  --> TIME: 2025-02-25 06:25:59 -- STEP: 36054/36938 -- GLOBAL_STEP: 664000 | > loss_text_ce: 0.04746195301413536 (0.03541498645843983) | > loss_mel_ce: 2.5665557384490967 (nan) | > loss: 0.010373086668550968 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13635724107444122) | > loader_time: 0.0048 (0.0046484003237891956)  --> TIME: 2025-02-25 06:26:09 -- STEP: 36104/36938 -- GLOBAL_STEP: 664050 | > loss_text_ce: 0.03602536395192146 (0.03541498963640423) | > loss_mel_ce: 3.333650588989258 (nan) | > loss: 0.013371730223298073 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13635785285686278) | > loader_time: 0.0045 (0.004648414830688158)  --> TIME: 2025-02-25 06:26:18 -- STEP: 36154/36938 -- GLOBAL_STEP: 664100 | > loss_text_ce: 0.03321840614080429 (0.035413778078613026) | > loss_mel_ce: 2.8026514053344727 (nan) | > loss: 0.01125345192849636 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13635623862589338) | > loader_time: 0.0045 (0.0046484162402956365)  --> TIME: 2025-02-25 06:26:27 -- STEP: 36204/36938 -- GLOBAL_STEP: 664150 | > loss_text_ce: 0.030097799375653267 (0.03541383543167556) | > loss_mel_ce: 1.4283853769302368 (nan) | > loss: 0.0057876319624483585 (nan) | > current_lr: 5e-06 | > step_time: 0.1075 (0.136355058269175) | > loader_time: 0.0045 (0.004648419753344291)  --> TIME: 2025-02-25 06:26:36 -- STEP: 36254/36938 -- GLOBAL_STEP: 664200 | > loss_text_ce: 0.031152641400694847 (0.035413283182318234) | > loss_mel_ce: 3.2100486755371094 (nan) | > loss: 0.012861911207437515 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13634622656004042) | > loader_time: 0.0047 (0.004648370573654435)  --> TIME: 2025-02-25 06:26:45 -- STEP: 36304/36938 -- GLOBAL_STEP: 664250 | > loss_text_ce: 0.03759884834289551 (0.03541315464908235) | > loss_mel_ce: 2.880452871322632 (nan) | > loss: 0.011579571291804314 (nan) | > current_lr: 5e-06 | > step_time: 0.1766 (0.13634738612406033) | > loader_time: 0.0048 (0.004648368984605323)  --> TIME: 2025-02-25 06:26:54 -- STEP: 36354/36938 -- GLOBAL_STEP: 664300 | > loss_text_ce: 0.03478006273508072 (0.03541278922480527) | > loss_mel_ce: 0.20356445014476776 (nan) | > loss: 0.0009458116255700588 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.13634124480701357) | > loader_time: 0.0047 (0.004648337415613462)  --> TIME: 2025-02-25 06:27:04 -- STEP: 36404/36938 -- GLOBAL_STEP: 664350 | > loss_text_ce: 0.029401611536741257 (0.03541320068844106) | > loss_mel_ce: 2.831731081008911 (nan) | > loss: 0.01135370135307312 (nan) | > current_lr: 5e-06 | > step_time: 0.1606 (0.13634652273299316) | > loader_time: 0.0047 (0.004648427847466264)  --> TIME: 2025-02-25 06:27:14 -- STEP: 36454/36938 -- GLOBAL_STEP: 664400 | > loss_text_ce: 0.024857843294739723 (0.03541256775695319) | > loss_mel_ce: 2.066818952560425 (nan) | > loss: 0.008300305344164371 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.136350545578663) | > loader_time: 0.0046 (0.004648550660597002)  --> TIME: 2025-02-25 06:27:23 -- STEP: 36504/36938 -- GLOBAL_STEP: 664450 | > loss_text_ce: 0.013983692973852158 (0.035411025892863573) | > loss_mel_ce: 0.48994141817092896 (nan) | > loss: 0.0019997027702629566 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13634692864305034) | > loader_time: 0.0045 (0.004648499953336644)  --> TIME: 2025-02-25 06:27:32 -- STEP: 36554/36938 -- GLOBAL_STEP: 664500 | > loss_text_ce: 0.05024949461221695 (0.03540946182203752) | > loss_mel_ce: 3.1254689693450928 (nan) | > loss: 0.012602058239281178 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.13634348055227918) | > loader_time: 0.0048 (0.004648455926728494)  --> TIME: 2025-02-25 06:27:41 -- STEP: 36604/36938 -- GLOBAL_STEP: 664550 | > loss_text_ce: 0.036863770335912704 (0.03540972497515947) | > loss_mel_ce: 2.4502463340759277 (nan) | > loss: 0.009869485162198544 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.1363438905268949) | > loader_time: 0.0046 (0.004648378606364199)  --> TIME: 2025-02-25 06:27:50 -- STEP: 36654/36938 -- GLOBAL_STEP: 664600 | > loss_text_ce: 0.042122021317481995 (0.03540915773959233) | > loss_mel_ce: 2.1274516582489014 (nan) | > loss: 0.008609420619904995 (nan) | > current_lr: 5e-06 | > step_time: 0.1123 (0.13634321828413137) | > loader_time: 0.0045 (0.0046484100582518196)  --> TIME: 2025-02-25 06:28:00 -- STEP: 36704/36938 -- GLOBAL_STEP: 664650 | > loss_text_ce: 0.019313715398311615 (0.03540629266065817) | > loss_mel_ce: 1.43450129032135 (nan) | > loss: 0.005769107490777969 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.13634943951054326) | > loader_time: 0.0045 (0.004648423892524641)  --> TIME: 2025-02-25 06:28:09 -- STEP: 36754/36938 -- GLOBAL_STEP: 664700 | > loss_text_ce: 0.053855050355196 (0.035406321530463844) | > loss_mel_ce: 1.7787312269210815 (nan) | > loss: 0.0072721680626273155 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13634715789535942) | > loader_time: 0.0045 (0.00464845146079148)  --> TIME: 2025-02-25 06:28:18 -- STEP: 36804/36938 -- GLOBAL_STEP: 664750 | > loss_text_ce: 0.0426139160990715 (0.03540504212227966) | > loss_mel_ce: 2.7641470432281494 (nan) | > loss: 0.011137940920889378 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13634329753755778) | > loader_time: 0.0045 (0.004648466334888694)  --> TIME: 2025-02-25 06:28:28 -- STEP: 36854/36938 -- GLOBAL_STEP: 664800 | > loss_text_ce: 0.031332261860370636 (0.035401547122364846) | > loss_mel_ce: 2.7595102787017822 (nan) | > loss: 0.011074773035943508 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13634632377983547) | > loader_time: 0.0046 (0.004648447502530829)  --> TIME: 2025-02-25 06:28:37 -- STEP: 36904/36938 -- GLOBAL_STEP: 664850 | > loss_text_ce: 0.03458482772111893 (0.03540018515508653) | > loss_mel_ce: 2.9610595703125 (nan) | > loss: 0.01188747864216566 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13634550838906526) | > loader_time: 0.0045 (0.0046483755899876505)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.02117228862082605 (+5.609092145865413e-05) | > avg_loss_text_ce: 0.034583430617383795 (-0.00040345026714966314) | > avg_loss_mel_ce: 2.5839344457824636 (-0.012180611048592915) | > avg_loss: 2.618517877736896 (-0.012584057184729769) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_664884.pth  > EPOCH: 19/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-25 06:29:09)   --> TIME: 2025-02-25 06:29:12 -- STEP: 16/36938 -- GLOBAL_STEP: 664900 | > loss_text_ce: 0.033917129039764404 (0.037097336957231164) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1303490549325943) | > loader_time: 0.0042 (0.004700228571891785)  --> TIME: 2025-02-25 06:29:21 -- STEP: 66/36938 -- GLOBAL_STEP: 664950 | > loss_text_ce: 0.034635309129953384 (0.035019120342577946) | > loss_mel_ce: 2.4804847240448 (nan) | > loss: 0.009980635717511177 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13487583579439105) | > loader_time: 0.0045 (0.004708904208558978)  --> TIME: 2025-02-25 06:29:30 -- STEP: 116/36938 -- GLOBAL_STEP: 665000 | > loss_text_ce: 0.03889181464910507 (0.03420667570277022) | > loss_mel_ce: 1.19635808467865 (nan) | > loss: 0.0049017854034900665 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13439239715707713) | > loader_time: 0.0045 (0.004679013942850047)  --> TIME: 2025-02-25 06:29:39 -- STEP: 166/36938 -- GLOBAL_STEP: 665050 | > loss_text_ce: 0.0456576943397522 (0.034781681822545575) | > loss_mel_ce: 2.457855224609375 (nan) | > loss: 0.00993457529693842 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13329713459474496) | > loader_time: 0.0046 (0.004676938056945799)  --> TIME: 2025-02-25 06:29:48 -- STEP: 216/36938 -- GLOBAL_STEP: 665100 | > loss_text_ce: 0.043011464178562164 (0.03489525914537133) | > loss_mel_ce: 2.875566005706787 (nan) | > loss: 0.011581657454371452 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13377903677799072) | > loader_time: 0.0046 (0.004670121051647042)  --> TIME: 2025-02-25 06:29:58 -- STEP: 266/36938 -- GLOBAL_STEP: 665150 | > loss_text_ce: 0.03192982077598572 (0.035002766121038816) | > loss_mel_ce: 3.1455564498901367 (nan) | > loss: 0.012609072960913181 (nan) | > current_lr: 5e-06 | > step_time: 0.1842 (0.13480448453946212) | > loader_time: 0.0048 (0.004695053387405278)  --> TIME: 2025-02-25 06:30:07 -- STEP: 316/36938 -- GLOBAL_STEP: 665200 | > loss_text_ce: 0.040262673050165176 (0.03524555570056922) | > loss_mel_ce: 2.212167263031006 (nan) | > loss: 0.008938214741647243 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.1345908573911159) | > loader_time: 0.0046 (0.004693862758105314)  --> TIME: 2025-02-25 06:30:16 -- STEP: 366/36938 -- GLOBAL_STEP: 665250 | > loss_text_ce: 0.030992135405540466 (0.035364208910919646) | > loss_mel_ce: 3.49010968208313 (nan) | > loss: 0.013972626999020576 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.13429276278761562) | > loader_time: 0.0045 (0.004676265143305875)  --> TIME: 2025-02-25 06:30:25 -- STEP: 416/36938 -- GLOBAL_STEP: 665300 | > loss_text_ce: 0.04038175940513611 (0.03534829036708784) | > loss_mel_ce: 2.6908090114593506 (nan) | > loss: 0.010838058777153492 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13451110571622854) | > loader_time: 0.0045 (0.0046663158215009245)  --> TIME: 2025-02-25 06:30:35 -- STEP: 466/36938 -- GLOBAL_STEP: 665350 | > loss_text_ce: 0.023799628019332886 (0.03527481371912584) | > loss_mel_ce: 2.538325309753418 (nan) | > loss: 0.010167162865400314 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13531641182469725) | > loader_time: 0.0045 (0.004662644709640305)  --> TIME: 2025-02-25 06:30:44 -- STEP: 516/36938 -- GLOBAL_STEP: 665400 | > loss_text_ce: 0.027422793209552765 (0.03508967448877855) | > loss_mel_ce: 2.8877334594726562 (nan) | > loss: 0.011568081565201283 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.1353590211202933) | > loader_time: 0.0047 (0.004661862702332725)  --> TIME: 2025-02-25 06:30:54 -- STEP: 566/36938 -- GLOBAL_STEP: 665450 | > loss_text_ce: 0.03183866664767265 (0.03519018247144389) | > loss_mel_ce: 3.1855764389038086 (nan) | > loss: 0.012767520733177662 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13547741566445734) | > loader_time: 0.0047 (0.0046622121291952515)  --> TIME: 2025-02-25 06:31:03 -- STEP: 616/36938 -- GLOBAL_STEP: 665500 | > loss_text_ce: 0.0307159423828125 (0.03528517302028937) | > loss_mel_ce: 3.4351744651794434 (nan) | > loss: 0.01375353429466486 (nan) | > current_lr: 5e-06 | > step_time: 0.1791 (0.13568567068545864) | > loader_time: 0.0049 (0.004659440997359041)  --> TIME: 2025-02-25 06:31:12 -- STEP: 666/36938 -- GLOBAL_STEP: 665550 | > loss_text_ce: 0.03766974061727524 (0.035318906846884185) | > loss_mel_ce: 3.118743658065796 (nan) | > loss: 0.012525450438261032 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.1358751237571421) | > loader_time: 0.0046 (0.004655594940300105)  --> TIME: 2025-02-25 06:31:21 -- STEP: 716/36938 -- GLOBAL_STEP: 665600 | > loss_text_ce: 0.037374649196863174 (0.03531554683668325) | > loss_mel_ce: 3.311814785003662 (nan) | > loss: 0.013290435075759888 (nan) | > current_lr: 5e-06 | > step_time: 0.1516 (0.13552417368862227) | > loader_time: 0.0046 (0.004655245629102825)  --> TIME: 2025-02-25 06:31:30 -- STEP: 766/36938 -- GLOBAL_STEP: 665650 | > loss_text_ce: 0.04303203895688057 (0.0353974343810046) | > loss_mel_ce: 2.6340978145599365 (nan) | > loss: 0.01062353141605854 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13521250670012228) | > loader_time: 0.0048 (0.004669269133485641)  --> TIME: 2025-02-25 06:31:40 -- STEP: 816/36938 -- GLOBAL_STEP: 665700 | > loss_text_ce: 0.036384981125593185 (0.03532164710386678) | > loss_mel_ce: 2.848594903945923 (nan) | > loss: 0.011448333971202374 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13553671246650167) | > loader_time: 0.0045 (0.004667941553919924)  --> TIME: 2025-02-25 06:31:49 -- STEP: 866/36938 -- GLOBAL_STEP: 665750 | > loss_text_ce: 0.043370168656110764 (0.035352195644273735) | > loss_mel_ce: 3.1147873401641846 (nan) | > loss: 0.012532372027635574 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13566855993336938) | > loader_time: 0.0045 (0.004666999911601219)  --> TIME: 2025-02-25 06:31:58 -- STEP: 916/36938 -- GLOBAL_STEP: 665800 | > loss_text_ce: 0.038358189165592194 (0.03548048718376942) | > loss_mel_ce: 2.2179958820343018 (nan) | > loss: 0.008953786455094814 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13516136131952933) | > loader_time: 0.0045 (0.004662460375040379)  --> TIME: 2025-02-25 06:32:07 -- STEP: 966/36938 -- GLOBAL_STEP: 665850 | > loss_text_ce: 0.050383299589157104 (0.03543158343030956) | > loss_mel_ce: 2.55426025390625 (nan) | > loss: 0.010335887782275677 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.1352106406821968) | > loader_time: 0.0044 (0.004658582541266337)  --> TIME: 2025-02-25 06:32:17 -- STEP: 1016/36938 -- GLOBAL_STEP: 665900 | > loss_text_ce: 0.024359652772545815 (0.03542480031376722) | > loss_mel_ce: 2.662334680557251 (nan) | > loss: 0.01066148653626442 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13527720978879557) | > loader_time: 0.0046 (0.004658419316209208)  --> TIME: 2025-02-25 06:32:26 -- STEP: 1066/36938 -- GLOBAL_STEP: 665950 | > loss_text_ce: 0.03462299332022667 (0.03539497752046638) | > loss_mel_ce: 2.9862639904022217 (nan) | > loss: 0.011987647041678429 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.1354796859605228) | > loader_time: 0.0048 (0.004654042716321535)  --> TIME: 2025-02-25 06:32:35 -- STEP: 1116/36938 -- GLOBAL_STEP: 666000 | > loss_text_ce: 0.023963412269949913 (0.03549794754844602) | > loss_mel_ce: 2.1655120849609375 (nan) | > loss: 0.008688395842909813 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13525886027189143) | > loader_time: 0.0045 (0.00465216901567247)  --> TIME: 2025-02-25 06:32:44 -- STEP: 1166/36938 -- GLOBAL_STEP: 666050 | > loss_text_ce: 0.036425232887268066 (0.0354610926321489) | > loss_mel_ce: 2.587794542312622 (nan) | > loss: 0.010413571260869503 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1352299431380441) | > loader_time: 0.0045 (0.00464841104903311)  --> TIME: 2025-02-25 06:32:53 -- STEP: 1216/36938 -- GLOBAL_STEP: 666100 | > loss_text_ce: 0.02775009535253048 (0.03542907879837962) | > loss_mel_ce: 3.0935583114624023 (nan) | > loss: 0.012386145070195198 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13525153186760455) | > loader_time: 0.0047 (0.004647431208899143)  --> TIME: 2025-02-25 06:33:03 -- STEP: 1266/36938 -- GLOBAL_STEP: 666150 | > loss_text_ce: 0.030674492940306664 (0.03539822896571736) | > loss_mel_ce: 3.0003867149353027 (nan) | > loss: 0.012028020806610584 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.1352944709301749) | > loader_time: 0.0046 (0.004646203905861895)  --> TIME: 2025-02-25 06:33:12 -- STEP: 1316/36938 -- GLOBAL_STEP: 666200 | > loss_text_ce: 0.043281517922878265 (0.03542111841551285) | > loss_mel_ce: 3.3620052337646484 (nan) | > loss: 0.013513043522834778 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.13513161020076037) | > loader_time: 0.0047 (0.004645743449770567)  --> TIME: 2025-02-25 06:33:21 -- STEP: 1366/36938 -- GLOBAL_STEP: 666250 | > loss_text_ce: 0.039709948003292084 (0.03538436686761932) | > loss_mel_ce: 2.8420002460479736 (nan) | > loss: 0.01143535878509283 (nan) | > current_lr: 5e-06 | > step_time: 0.1867 (0.13505730621985793) | > loader_time: 0.0051 (0.004648451225649379)  --> TIME: 2025-02-25 06:33:30 -- STEP: 1416/36938 -- GLOBAL_STEP: 666300 | > loss_text_ce: 0.04086628183722496 (0.03537160146121191) | > loss_mel_ce: 3.3328840732574463 (nan) | > loss: 0.013387898914515972 (nan) | > current_lr: 5e-06 | > step_time: 0.1799 (0.13500157436408586) | > loader_time: 0.0048 (0.0046488107261011095)  --> TIME: 2025-02-25 06:33:40 -- STEP: 1466/36938 -- GLOBAL_STEP: 666350 | > loss_text_ce: 0.04377506673336029 (0.03535270507003997) | > loss_mel_ce: 2.2916886806488037 (nan) | > loss: 0.009267713874578476 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13524042990815924) | > loader_time: 0.0046 (0.004648943552378896)  --> TIME: 2025-02-25 06:33:49 -- STEP: 1516/36938 -- GLOBAL_STEP: 666400 | > loss_text_ce: 0.03306787088513374 (0.03534763746668827) | > loss_mel_ce: 3.070115327835083 (nan) | > loss: 0.01231421995908022 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13538145484270103) | > loader_time: 0.0045 (0.00464927992908809)  --> TIME: 2025-02-25 06:33:59 -- STEP: 1566/36938 -- GLOBAL_STEP: 666450 | > loss_text_ce: 0.032350804656744 (0.03539132910553434) | > loss_mel_ce: 4.0576171875 (nan) | > loss: 0.016230033710598946 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.1354924680933948) | > loader_time: 0.0046 (0.004650307341096946)  --> TIME: 2025-02-25 06:34:08 -- STEP: 1616/36938 -- GLOBAL_STEP: 666500 | > loss_text_ce: 0.037404075264930725 (0.0353689038230448) | > loss_mel_ce: 3.477076530456543 (nan) | > loss: 0.013946352526545525 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1355381109336816) | > loader_time: 0.0045 (0.004649851581837862)  --> TIME: 2025-02-25 06:34:18 -- STEP: 1666/36938 -- GLOBAL_STEP: 666550 | > loss_text_ce: 0.03951047733426094 (0.035378720974135996) | > loss_mel_ce: 0.0291748046875 (nan) | > loss: 0.0002725606318563223 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13563953284599073) | > loader_time: 0.0047 (0.004648230943073022)  --> TIME: 2025-02-25 06:34:27 -- STEP: 1716/36938 -- GLOBAL_STEP: 666600 | > loss_text_ce: 0.025826966390013695 (0.03534714630555303) | > loss_mel_ce: 2.6997196674346924 (nan) | > loss: 0.010815661400556564 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1355276589904911) | > loader_time: 0.0044 (0.0046460125551912815)  --> TIME: 2025-02-25 06:34:36 -- STEP: 1766/36938 -- GLOBAL_STEP: 666650 | > loss_text_ce: 0.034160736948251724 (0.03530109821821938) | > loss_mel_ce: 2.1169426441192627 (nan) | > loss: 0.008536125533282757 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13544447654235942) | > loader_time: 0.0046 (0.004646760016038484)  --> TIME: 2025-02-25 06:34:45 -- STEP: 1816/36938 -- GLOBAL_STEP: 666700 | > loss_text_ce: 0.03519409894943237 (0.03525027426755644) | > loss_mel_ce: 3.151182174682617 (nan) | > loss: 0.012644351460039616 (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.1354591997446994) | > loader_time: 0.0047 (0.004645856991738468)  --> TIME: 2025-02-25 06:34:55 -- STEP: 1866/36938 -- GLOBAL_STEP: 666750 | > loss_text_ce: 0.03332425281405449 (0.035214237956627696) | > loss_mel_ce: 3.1495673656463623 (nan) | > loss: 0.012630523182451725 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.13562508091921466) | > loader_time: 0.0046 (0.0046450653515700165)  --> TIME: 2025-02-25 06:35:04 -- STEP: 1916/36938 -- GLOBAL_STEP: 666800 | > loss_text_ce: 0.02631925605237484 (0.03520617976741937) | > loss_mel_ce: 3.0669500827789307 (nan) | > loss: 0.012274879030883312 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.13565762841875711) | > loader_time: 0.0045 (0.004642749131348032)  --> TIME: 2025-02-25 06:35:13 -- STEP: 1966/36938 -- GLOBAL_STEP: 666850 | > loss_text_ce: 0.024846196174621582 (0.03522199450301377) | > loss_mel_ce: 2.5128250122070312 (nan) | > loss: 0.010070123709738255 (nan) | > current_lr: 5e-06 | > step_time: 0.1576 (0.13577676122574203) | > loader_time: 0.0047 (0.0046414130823051875)  --> TIME: 2025-02-25 06:35:23 -- STEP: 2016/36938 -- GLOBAL_STEP: 666900 | > loss_text_ce: 0.040750984102487564 (0.03523088121021704) | > loss_mel_ce: 2.65179705619812 (nan) | > loss: 0.010684714652597904 (nan) | > current_lr: 5e-06 | > step_time: 0.1615 (0.13582684087847902) | > loader_time: 0.0048 (0.004640604531954199)  --> TIME: 2025-02-25 06:35:32 -- STEP: 2066/36938 -- GLOBAL_STEP: 666950 | > loss_text_ce: 0.03470762073993683 (0.03521175024037444) | > loss_mel_ce: 1.3502023220062256 (nan) | > loss: 0.005495674908161163 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13592215671428517) | > loader_time: 0.0044 (0.004641370067060968)  --> TIME: 2025-02-25 06:35:42 -- STEP: 2116/36938 -- GLOBAL_STEP: 667000 | > loss_text_ce: 0.03603310137987137 (0.035219085850842734) | > loss_mel_ce: 3.2791008949279785 (nan) | > loss: 0.013155294582247734 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.13593587825789566) | > loader_time: 0.0048 (0.004641184509363871)  --> TIME: 2025-02-25 06:35:51 -- STEP: 2166/36938 -- GLOBAL_STEP: 667050 | > loss_text_ce: 0.021152006462216377 (0.03519702850929566) | > loss_mel_ce: 2.800961494445801 (nan) | > loss: 0.011198863387107849 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13600140276157732) | > loader_time: 0.0046 (0.004640529250394158)  --> TIME: 2025-02-25 06:36:00 -- STEP: 2216/36938 -- GLOBAL_STEP: 667100 | > loss_text_ce: 0.029270019382238388 (0.03520567015415258) | > loss_mel_ce: 2.2553138732910156 (nan) | > loss: 0.009065809659659863 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13600192431508423) | > loader_time: 0.0045 (0.0046408146297027865)  --> TIME: 2025-02-25 06:36:09 -- STEP: 2266/36938 -- GLOBAL_STEP: 667150 | > loss_text_ce: 0.02878381684422493 (0.03519948556261358) | > loss_mel_ce: 1.8847708702087402 (nan) | > loss: 0.007593471556901932 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13592824100501125) | > loader_time: 0.0045 (0.004641063215452695)  --> TIME: 2025-02-25 06:36:19 -- STEP: 2316/36938 -- GLOBAL_STEP: 667200 | > loss_text_ce: 0.032870445400476456 (0.03515675759251798) | > loss_mel_ce: 1.581326961517334 (nan) | > loss: 0.006405545398592949 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.1359448467923364) | > loader_time: 0.0049 (0.004641046589932082)  --> TIME: 2025-02-25 06:36:28 -- STEP: 2366/36938 -- GLOBAL_STEP: 667250 | > loss_text_ce: 0.04389378800988197 (0.03517792903718213) | > loss_mel_ce: 2.9840052127838135 (nan) | > loss: 0.01201547309756279 (nan) | > current_lr: 5e-06 | > step_time: 0.1443 (0.13611892603860504) | > loader_time: 0.0046 (0.004642833921733989)  --> TIME: 2025-02-25 06:36:38 -- STEP: 2416/36938 -- GLOBAL_STEP: 667300 | > loss_text_ce: 0.044668443500995636 (0.03518168124931541) | > loss_mel_ce: 2.653618812561035 (nan) | > loss: 0.010707490146160126 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13608047247722457) | > loader_time: 0.0046 (0.0046418966441754425)  --> TIME: 2025-02-25 06:36:47 -- STEP: 2466/36938 -- GLOBAL_STEP: 667350 | > loss_text_ce: 0.04001885652542114 (0.03521086515017335) | > loss_mel_ce: 3.1528258323669434 (nan) | > loss: 0.012670018710196018 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13613351224789474) | > loader_time: 0.0046 (0.0046411904492034)  --> TIME: 2025-02-25 06:36:56 -- STEP: 2516/36938 -- GLOBAL_STEP: 667400 | > loss_text_ce: 0.03352373465895653 (0.035211592401112886) | > loss_mel_ce: 3.3711137771606445 (nan) | > loss: 0.013510467484593391 (nan) | > current_lr: 5e-06 | > step_time: 0.1599 (0.13613196691003646) | > loader_time: 0.0046 (0.004640358430592546)  --> TIME: 2025-02-25 06:37:06 -- STEP: 2566/36938 -- GLOBAL_STEP: 667450 | > loss_text_ce: 0.032693494111299515 (0.035186209490875565) | > loss_mel_ce: 3.2061522006988525 (nan) | > loss: 0.012852562591433525 (nan) | > current_lr: 5e-06 | > step_time: 0.1796 (0.13611720459403623) | > loader_time: 0.0048 (0.00464093387823781)  --> TIME: 2025-02-25 06:37:15 -- STEP: 2616/36938 -- GLOBAL_STEP: 667500 | > loss_text_ce: 0.03963522985577583 (0.03517310977244143) | > loss_mel_ce: 1.808202862739563 (nan) | > loss: 0.007332690991461277 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13616152157841857) | > loader_time: 0.0045 (0.004641195958542893)  --> TIME: 2025-02-25 06:37:24 -- STEP: 2666/36938 -- GLOBAL_STEP: 667550 | > loss_text_ce: 0.03932211548089981 (0.035188855437753346) | > loss_mel_ce: 3.102759838104248 (nan) | > loss: 0.012468580156564713 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.1361041074396523) | > loader_time: 0.0046 (0.004641035402617055)  --> TIME: 2025-02-25 06:37:33 -- STEP: 2716/36938 -- GLOBAL_STEP: 667600 | > loss_text_ce: 0.03307924419641495 (0.03517737524962499) | > loss_mel_ce: 3.5093276500701904 (nan) | > loss: 0.01405717059969902 (nan) | > current_lr: 5e-06 | > step_time: 0.1784 (0.13604282244315735) | > loader_time: 0.0049 (0.004640376796076325)  --> TIME: 2025-02-25 06:37:42 -- STEP: 2766/36938 -- GLOBAL_STEP: 667650 | > loss_text_ce: 0.03473832458257675 (0.0351715385018217) | > loss_mel_ce: 0.22589111328125 (nan) | > loss: 0.0010342438472434878 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.1359817765367609) | > loader_time: 0.0046 (0.004641568152178737)  --> TIME: 2025-02-25 06:37:51 -- STEP: 2816/36938 -- GLOBAL_STEP: 667700 | > loss_text_ce: 0.036756597459316254 (0.035155550220885984) | > loss_mel_ce: 1.7306408882141113 (nan) | > loss: 0.007013482507318258 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.135994734665887) | > loader_time: 0.0045 (0.004641213454306125)  --> TIME: 2025-02-25 06:38:01 -- STEP: 2866/36938 -- GLOBAL_STEP: 667750 | > loss_text_ce: 0.03147243708372116 (0.035155555291851326) | > loss_mel_ce: 2.7957160472869873 (nan) | > loss: 0.011219002306461334 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13602265850780557) | > loader_time: 0.0049 (0.0046415315669313265)  --> TIME: 2025-02-25 06:38:10 -- STEP: 2916/36938 -- GLOBAL_STEP: 667800 | > loss_text_ce: 0.0469060018658638 (0.035198698633456434) | > loss_mel_ce: 3.0614819526672363 (nan) | > loss: 0.012334873899817467 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1360806059117833) | > loader_time: 0.0045 (0.004641978910757512)  --> TIME: 2025-02-25 06:38:20 -- STEP: 2966/36938 -- GLOBAL_STEP: 667850 | > loss_text_ce: 0.03373175859451294 (0.03520835163881863) | > loss_mel_ce: 1.9359130859375 (nan) | > loss: 0.007816051132977009 (nan) | > current_lr: 5e-06 | > step_time: 0.1079 (0.13602845056267573) | > loader_time: 0.0045 (0.004641187970733121)  --> TIME: 2025-02-25 06:38:29 -- STEP: 3016/36938 -- GLOBAL_STEP: 667900 | > loss_text_ce: 0.03000449575483799 (0.03521858065559543) | > loss_mel_ce: 3.094630479812622 (nan) | > loss: 0.012399345636367798 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.13608974850778544) | > loader_time: 0.0046 (0.004640724282682093)  --> TIME: 2025-02-25 06:38:38 -- STEP: 3066/36938 -- GLOBAL_STEP: 667950 | > loss_text_ce: 0.035342346876859665 (0.03521334279063702) | > loss_mel_ce: 3.0668609142303467 (nan) | > loss: 0.0123103316873312 (nan) | > current_lr: 5e-06 | > step_time: 0.1629 (0.1360675209266875) | > loader_time: 0.0047 (0.004640445395069649)  --> TIME: 2025-02-25 06:38:48 -- STEP: 3116/36938 -- GLOBAL_STEP: 668000 | > loss_text_ce: 0.032449934631586075 (0.0352136506645226) | > loss_mel_ce: 1.6571197509765625 (nan) | > loss: 0.006704641971737146 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13604528920794307) | > loader_time: 0.0045 (0.0046400091920492615)  --> TIME: 2025-02-25 06:38:57 -- STEP: 3166/36938 -- GLOBAL_STEP: 668050 | > loss_text_ce: 0.032549336552619934 (0.03521823961480702) | > loss_mel_ce: 3.070768117904663 (nan) | > loss: 0.0123147526755929 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13600624044409873) | > loader_time: 0.0046 (0.004639242769266618)  --> TIME: 2025-02-25 06:39:06 -- STEP: 3216/36938 -- GLOBAL_STEP: 668100 | > loss_text_ce: 0.03313417732715607 (0.03521083389433203) | > loss_mel_ce: 2.9356119632720947 (nan) | > loss: 0.01178073976188898 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.13608976620346708) | > loader_time: 0.0047 (0.004639370895143751)  --> TIME: 2025-02-25 06:39:16 -- STEP: 3266/36938 -- GLOBAL_STEP: 668150 | > loss_text_ce: 0.03817197307944298 (0.03523538392168318) | > loss_mel_ce: 3.011115074157715 (nan) | > loss: 0.012100346386432648 (nan) | > current_lr: 5e-06 | > step_time: 0.1782 (0.13612977982443342) | > loader_time: 0.0047 (0.004640160713616268)  --> TIME: 2025-02-25 06:39:25 -- STEP: 3316/36938 -- GLOBAL_STEP: 668200 | > loss_text_ce: 0.041174501180648804 (0.03522658468767841) | > loss_mel_ce: 2.750765085220337 (nan) | > loss: 0.011079126037657261 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.1361521294091085) | > loader_time: 0.0047 (0.004641233149138505)  --> TIME: 2025-02-25 06:39:34 -- STEP: 3366/36938 -- GLOBAL_STEP: 668250 | > loss_text_ce: 0.036332517862319946 (0.035230768036287566) | > loss_mel_ce: 2.566492795944214 (nan) | > loss: 0.01032867282629013 (nan) | > current_lr: 5e-06 | > step_time: 0.1488 (0.1361452843982829) | > loader_time: 0.0047 (0.004640933805750138)  --> TIME: 2025-02-25 06:39:44 -- STEP: 3416/36938 -- GLOBAL_STEP: 668300 | > loss_text_ce: 0.03547298163175583 (0.03524915497358536) | > loss_mel_ce: 2.9290378093719482 (nan) | > loss: 0.011763932183384895 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13614567402375047) | > loader_time: 0.0046 (0.004639931459337526)  --> TIME: 2025-02-25 06:39:53 -- STEP: 3466/36938 -- GLOBAL_STEP: 668350 | > loss_text_ce: 0.042578328400850296 (0.03524099265999627) | > loss_mel_ce: 3.078389883041382 (nan) | > loss: 0.012384794652462006 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1362079825624174) | > loader_time: 0.0045 (0.004639193768179338)  --> TIME: 2025-02-25 06:40:02 -- STEP: 3516/36938 -- GLOBAL_STEP: 668400 | > loss_text_ce: 0.03335178643465042 (0.035247874898365554) | > loss_mel_ce: 2.5498206615448 (nan) | > loss: 0.01025068573653698 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13618058162880117) | > loader_time: 0.0045 (0.004638700355035029)  --> TIME: 2025-02-25 06:40:12 -- STEP: 3566/36938 -- GLOBAL_STEP: 668450 | > loss_text_ce: 0.029399890452623367 (0.03524786182530369) | > loss_mel_ce: 2.057490110397339 (nan) | > loss: 0.008281310088932514 (nan) | > current_lr: 5e-06 | > step_time: 0.1352 (0.13619034730244764) | > loader_time: 0.0047 (0.004638873521223923)  --> TIME: 2025-02-25 06:40:21 -- STEP: 3616/36938 -- GLOBAL_STEP: 668500 | > loss_text_ce: 0.04047410562634468 (0.03523582460016704) | > loss_mel_ce: 2.243096113204956 (nan) | > loss: 0.009061787277460098 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.1362254436300915) | > loader_time: 0.0046 (0.004640471223181311)  --> TIME: 2025-02-25 06:40:30 -- STEP: 3666/36938 -- GLOBAL_STEP: 668550 | > loss_text_ce: 0.036718547344207764 (0.035242310819656585) | > loss_mel_ce: 2.8814549446105957 (nan) | > loss: 0.011580054648220539 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13617817229152804) | > loader_time: 0.0047 (0.0046397013802119795)  --> TIME: 2025-02-25 06:40:39 -- STEP: 3716/36938 -- GLOBAL_STEP: 668600 | > loss_text_ce: 0.040911778807640076 (0.035253986681182944) | > loss_mel_ce: 1.895021915435791 (nan) | > loss: 0.007682276889681816 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13617256960648086) | > loader_time: 0.0047 (0.004639638188837403)  --> TIME: 2025-02-25 06:40:49 -- STEP: 3766/36938 -- GLOBAL_STEP: 668650 | > loss_text_ce: 0.03664982318878174 (0.035243863408541305) | > loss_mel_ce: 3.321352005004883 (nan) | > loss: 0.01332540437579155 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13616951393182097) | > loader_time: 0.0048 (0.0046396344124605944)  --> TIME: 2025-02-25 06:40:58 -- STEP: 3816/36938 -- GLOBAL_STEP: 668700 | > loss_text_ce: 0.04054490104317665 (0.0352317980286593) | > loss_mel_ce: 2.837769031524658 (nan) | > loss: 0.01142188161611557 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13619657657431314) | > loader_time: 0.0045 (0.004639944815285785)  --> TIME: 2025-02-25 06:41:08 -- STEP: 3866/36938 -- GLOBAL_STEP: 668750 | > loss_text_ce: 0.04164623096585274 (0.03521748773890475) | > loss_mel_ce: 3.077683687210083 (nan) | > loss: 0.01237829402089119 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1362016158466562) | > loader_time: 0.0046 (0.004639781514289631)  --> TIME: 2025-02-25 06:41:17 -- STEP: 3916/36938 -- GLOBAL_STEP: 668800 | > loss_text_ce: 0.036388907581567764 (0.03520908375224051) | > loss_mel_ce: 2.7037007808685303 (nan) | > loss: 0.010873372666537762 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13622947623220683) | > loader_time: 0.0046 (0.004639807042111157)  --> TIME: 2025-02-25 06:41:27 -- STEP: 3966/36938 -- GLOBAL_STEP: 668850 | > loss_text_ce: 0.034946683794260025 (0.03522045458786742) | > loss_mel_ce: 3.1840484142303467 (nan) | > loss: 0.012773790396749973 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.13630103361588522) | > loader_time: 0.0048 (0.004640282733357424)  --> TIME: 2025-02-25 06:41:36 -- STEP: 4016/36938 -- GLOBAL_STEP: 668900 | > loss_text_ce: 0.03378139063715935 (0.03522232215549144) | > loss_mel_ce: 3.1493492126464844 (nan) | > loss: 0.012631471268832684 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.1362697191209903) | > loader_time: 0.0046 (0.00464053582622709)  --> TIME: 2025-02-25 06:41:45 -- STEP: 4066/36938 -- GLOBAL_STEP: 668950 | > loss_text_ce: 0.036822717636823654 (0.03523250489435683) | > loss_mel_ce: 2.9831416606903076 (nan) | > loss: 0.01198398694396019 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.13624815705135623) | > loader_time: 0.0046 (0.004640121619453922)  --> TIME: 2025-02-25 06:41:55 -- STEP: 4116/36938 -- GLOBAL_STEP: 669000 | > loss_text_ce: 0.030862268060445786 (0.035224766809895156) | > loss_mel_ce: 2.68593430519104 (nan) | > loss: 0.01078093983232975 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13629059466027388) | > loader_time: 0.0045 (0.004640175487603562)  --> TIME: 2025-02-25 06:42:04 -- STEP: 4166/36938 -- GLOBAL_STEP: 669050 | > loss_text_ce: 0.03786564618349075 (0.035221604936171824) | > loss_mel_ce: 2.8291218280792236 (nan) | > loss: 0.011376935057342052 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13636883822874632) | > loader_time: 0.0046 (0.004640526858534412)  --> TIME: 2025-02-25 06:42:14 -- STEP: 4216/36938 -- GLOBAL_STEP: 669100 | > loss_text_ce: 0.03479459509253502 (0.03522900201290125) | > loss_mel_ce: 3.028775930404663 (nan) | > loss: 0.012157026678323746 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1364022581581821) | > loader_time: 0.0045 (0.00464060212674583)  --> TIME: 2025-02-25 06:42:23 -- STEP: 4266/36938 -- GLOBAL_STEP: 669150 | > loss_text_ce: 0.03359473496675491 (0.035232870981002896) | > loss_mel_ce: 3.1047255992889404 (nan) | > loss: 0.012453652918338776 (nan) | > current_lr: 5e-06 | > step_time: 0.1817 (0.1364127619412933) | > loader_time: 0.0048 (0.004640607503004817)  --> TIME: 2025-02-25 06:42:33 -- STEP: 4316/36938 -- GLOBAL_STEP: 669200 | > loss_text_ce: 0.03267503157258034 (0.03523103468529224) | > loss_mel_ce: 2.536881446838379 (nan) | > loss: 0.010196653194725513 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13643175330396232) | > loader_time: 0.0045 (0.004640325061472828)  --> TIME: 2025-02-25 06:42:42 -- STEP: 4366/36938 -- GLOBAL_STEP: 669250 | > loss_text_ce: 0.03175794705748558 (0.03519555371070379) | > loss_mel_ce: 2.643882989883423 (nan) | > loss: 0.01061762310564518 (nan) | > current_lr: 5e-06 | > step_time: 0.1761 (0.13635996531522732) | > loader_time: 0.0048 (0.00463994669903323)  --> TIME: 2025-02-25 06:42:51 -- STEP: 4416/36938 -- GLOBAL_STEP: 669300 | > loss_text_ce: 0.03459250181913376 (0.035184205553588845) | > loss_mel_ce: 1.3287773132324219 (nan) | > loss: 0.005410198122262955 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13634720412285395) | > loader_time: 0.0045 (0.004640015409044587)  --> TIME: 2025-02-25 06:43:00 -- STEP: 4466/36938 -- GLOBAL_STEP: 669350 | > loss_text_ce: 0.031707823276519775 (0.03518350447656937) | > loss_mel_ce: 2.560856342315674 (nan) | > loss: 0.010287953540682793 (nan) | > current_lr: 5e-06 | > step_time: 0.1491 (0.1364061092508246) | > loader_time: 0.0046 (0.004640034373651096)  --> TIME: 2025-02-25 06:43:10 -- STEP: 4516/36938 -- GLOBAL_STEP: 669400 | > loss_text_ce: 0.033742379397153854 (0.03518865440755198) | > loss_mel_ce: 2.76304030418396 (nan) | > loss: 0.011098344810307026 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.136368793074909) | > loader_time: 0.0045 (0.004640019921943922)  --> TIME: 2025-02-25 06:43:19 -- STEP: 4566/36938 -- GLOBAL_STEP: 669450 | > loss_text_ce: 0.02707737311720848 (0.03518273044019394) | > loss_mel_ce: 1.491898536682129 (nan) | > loss: 0.006027682218700647 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13632723990627063) | > loader_time: 0.0046 (0.004639455638222136)  --> TIME: 2025-02-25 06:43:28 -- STEP: 4616/36938 -- GLOBAL_STEP: 669500 | > loss_text_ce: 0.04668794944882393 (0.035184398977258655) | > loss_mel_ce: 2.724729299545288 (nan) | > loss: 0.010997687466442585 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13635607324605145) | > loader_time: 0.0047 (0.0046390302776257225)  --> TIME: 2025-02-25 06:43:37 -- STEP: 4666/36938 -- GLOBAL_STEP: 669550 | > loss_text_ce: 0.032567933201789856 (0.035192617254137706) | > loss_mel_ce: 2.5678861141204834 (nan) | > loss: 0.010319262742996216 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13630319111754907) | > loader_time: 0.0047 (0.004638511992502343)  --> TIME: 2025-02-25 06:43:47 -- STEP: 4716/36938 -- GLOBAL_STEP: 669600 | > loss_text_ce: 0.0466267354786396 (0.03518976506925616) | > loss_mel_ce: 2.877840995788574 (nan) | > loss: 0.011605031788349152 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13634955589805592) | > loader_time: 0.0045 (0.004638242408115026)  --> TIME: 2025-02-25 06:43:56 -- STEP: 4766/36938 -- GLOBAL_STEP: 669650 | > loss_text_ce: 0.03476772829890251 (0.03519388077861899) | > loss_mel_ce: 3.2910122871398926 (nan) | > loss: 0.013197540305554867 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.13635570678798917) | > loader_time: 0.0046 (0.004638164372590208)  --> TIME: 2025-02-25 06:44:05 -- STEP: 4816/36938 -- GLOBAL_STEP: 669700 | > loss_text_ce: 0.032609861344099045 (0.03518990188256139) | > loss_mel_ce: 1.570626974105835 (nan) | > loss: 0.006362051237374544 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13633689432841145) | > loader_time: 0.0047 (0.004637653992025539)  --> TIME: 2025-02-25 06:44:15 -- STEP: 4866/36938 -- GLOBAL_STEP: 669750 | > loss_text_ce: 0.03667081147432327 (0.03518847488743468) | > loss_mel_ce: 3.164682149887085 (nan) | > loss: 0.012703782878816128 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13632093752748473) | > loader_time: 0.0045 (0.004637406776826449)  --> TIME: 2025-02-25 06:44:24 -- STEP: 4916/36938 -- GLOBAL_STEP: 669800 | > loss_text_ce: 0.04249782860279083 (0.03521378447742722) | > loss_mel_ce: 0.008909404277801514 (nan) | > loss: 0.00020399696950335056 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13635206639621142) | > loader_time: 0.0045 (0.004638263566208795)  --> TIME: 2025-02-25 06:44:33 -- STEP: 4966/36938 -- GLOBAL_STEP: 669850 | > loss_text_ce: 0.0409734770655632 (0.035216388358812224) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13633159894017544) | > loader_time: 0.0046 (0.0046378858522498675)  --> TIME: 2025-02-25 06:44:42 -- STEP: 5016/36938 -- GLOBAL_STEP: 669900 | > loss_text_ce: 0.048715345561504364 (0.035217461287433194) | > loss_mel_ce: 2.595950126647949 (nan) | > loss: 0.01049470528960228 (nan) | > current_lr: 5e-06 | > step_time: 0.1499 (0.13632909076635868) | > loader_time: 0.0047 (0.004637461197243141)  --> TIME: 2025-02-25 06:44:52 -- STEP: 5066/36938 -- GLOBAL_STEP: 669950 | > loss_text_ce: 0.032934367656707764 (0.03521552236906619) | > loss_mel_ce: 2.9007225036621094 (nan) | > loss: 0.011641496792435646 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13630098473917432) | > loader_time: 0.0045 (0.004637466981106932)  --> TIME: 2025-02-25 06:45:01 -- STEP: 5116/36938 -- GLOBAL_STEP: 670000 | > loss_text_ce: 0.033067792654037476 (0.035218208320285965) | > loss_mel_ce: 1.288275957107544 (nan) | > loss: 0.00524342805147171 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13632560004658542) | > loader_time: 0.0046 (0.00463783326234733) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_670000.pth  --> TIME: 2025-02-25 06:45:17 -- STEP: 5166/36938 -- GLOBAL_STEP: 670050 | > loss_text_ce: 0.0432824045419693 (0.03521614447052177) | > loss_mel_ce: 1.7686896324157715 (nan) | > loss: 0.007190365809947252 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13633813878338832) | > loader_time: 0.0045 (0.004638646676288362)  --> TIME: 2025-02-25 06:45:26 -- STEP: 5216/36938 -- GLOBAL_STEP: 670100 | > loss_text_ce: 0.02624211460351944 (0.035210536649854934) | > loss_mel_ce: 2.6116700172424316 (nan) | > loss: 0.010467905551195145 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13635487936757057) | > loader_time: 0.0046 (0.004639414373350993)  --> TIME: 2025-02-25 06:45:36 -- STEP: 5266/36938 -- GLOBAL_STEP: 670150 | > loss_text_ce: 0.03328828141093254 (0.035217832376528944) | > loss_mel_ce: 3.1919727325439453 (nan) | > loss: 0.012798655778169632 (nan) | > current_lr: 5e-06 | > step_time: 0.1959 (0.13631829566209394) | > loader_time: 0.0048 (0.004639640716468682)  --> TIME: 2025-02-25 06:45:45 -- STEP: 5316/36938 -- GLOBAL_STEP: 670200 | > loss_text_ce: 0.0448489785194397 (0.035217510675941366) | > loss_mel_ce: 2.5334134101867676 (nan) | > loss: 0.010231200605630875 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.136347679646953) | > loader_time: 0.0046 (0.004640117769765148)  --> TIME: 2025-02-25 06:45:54 -- STEP: 5366/36938 -- GLOBAL_STEP: 670250 | > loss_text_ce: 0.0364844873547554 (0.03523525270014484) | > loss_mel_ce: 2.44534969329834 (nan) | > loss: 0.009848549030721188 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13633882306291187) | > loader_time: 0.0048 (0.004640200268720161)  --> TIME: 2025-02-25 06:46:04 -- STEP: 5416/36938 -- GLOBAL_STEP: 670300 | > loss_text_ce: 0.02079641819000244 (0.03523431806340182) | > loss_mel_ce: 2.2609775066375732 (nan) | > loss: 0.009054659865796566 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13632299822577656) | > loader_time: 0.0045 (0.004640425105566797)  --> TIME: 2025-02-25 06:46:13 -- STEP: 5466/36938 -- GLOBAL_STEP: 670350 | > loss_text_ce: 0.03197598084807396 (0.035238116963753265) | > loss_mel_ce: 2.492952585220337 (nan) | > loss: 0.010019558481872082 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13633691178456847) | > loader_time: 0.0045 (0.004640303991061602)  --> TIME: 2025-02-25 06:46:22 -- STEP: 5516/36938 -- GLOBAL_STEP: 670400 | > loss_text_ce: 0.030363136902451515 (0.035231350409339814) | > loss_mel_ce: 1.613303780555725 (nan) | > loss: 0.006522487848997116 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.136337294623504) | > loader_time: 0.0046 (0.004640839383427497)  --> TIME: 2025-02-25 06:46:31 -- STEP: 5566/36938 -- GLOBAL_STEP: 670450 | > loss_text_ce: 0.0387912280857563 (0.03523864948578609) | > loss_mel_ce: 0.90692138671875 (nan) | > loss: 0.003752828110009432 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13628621594836443) | > loader_time: 0.0045 (0.004641392956609288)  --> TIME: 2025-02-25 06:46:40 -- STEP: 5616/36938 -- GLOBAL_STEP: 670500 | > loss_text_ce: 0.050943970680236816 (0.03524608502812568) | > loss_mel_ce: 1.7211205959320068 (nan) | > loss: 0.007032002788037062 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13627487514433334) | > loader_time: 0.0045 (0.004641709164676481)  --> TIME: 2025-02-25 06:46:49 -- STEP: 5666/36938 -- GLOBAL_STEP: 670550 | > loss_text_ce: 0.04036834463477135 (0.03524454530283153) | > loss_mel_ce: 3.1292529106140137 (nan) | > loss: 0.012577862478792667 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13623503926698272) | > loader_time: 0.0046 (0.004641980700723427)  --> TIME: 2025-02-25 06:46:59 -- STEP: 5716/36938 -- GLOBAL_STEP: 670600 | > loss_text_ce: 0.030093258246779442 (0.035237875013489114) | > loss_mel_ce: 1.5697861909866333 (nan) | > loss: 0.00634872866794467 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13624258380073007) | > loader_time: 0.0045 (0.00464225282528917)  --> TIME: 2025-02-25 06:47:08 -- STEP: 5766/36938 -- GLOBAL_STEP: 670650 | > loss_text_ce: 0.03998972102999687 (0.035243676881526596) | > loss_mel_ce: 3.142256736755371 (nan) | > loss: 0.012627962976694107 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13627753413223842) | > loader_time: 0.0046 (0.004642391882958259)  --> TIME: 2025-02-25 06:47:18 -- STEP: 5816/36938 -- GLOBAL_STEP: 670700 | > loss_text_ce: 0.024815930053591728 (0.03524751906962806) | > loss_mel_ce: 2.5697314739227295 (nan) | > loss: 0.010295824147760868 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.1362672134557483) | > loader_time: 0.0047 (0.004642306487530735)  --> TIME: 2025-02-25 06:47:26 -- STEP: 5866/36938 -- GLOBAL_STEP: 670750 | > loss_text_ce: 0.03867139667272568 (0.0352451193625565) | > loss_mel_ce: 2.6876535415649414 (nan) | > loss: 0.010818750597536564 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13622657317087733) | > loader_time: 0.0047 (0.004642137845465624)  --> TIME: 2025-02-25 06:47:35 -- STEP: 5916/36938 -- GLOBAL_STEP: 670800 | > loss_text_ce: 0.0266281571239233 (0.03524608102244886) | > loss_mel_ce: 1.4343928098678589 (nan) | > loss: 0.005797702353447676 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13615577804469983) | > loader_time: 0.0045 (0.00464162716114334)  --> TIME: 2025-02-25 06:47:44 -- STEP: 5966/36938 -- GLOBAL_STEP: 670850 | > loss_text_ce: 0.03400922566652298 (0.0352432367823048) | > loss_mel_ce: 2.5541093349456787 (nan) | > loss: 0.010270312428474426 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13611342125277845) | > loader_time: 0.0045 (0.004641023570964515)  --> TIME: 2025-02-25 06:47:54 -- STEP: 6016/36938 -- GLOBAL_STEP: 670900 | > loss_text_ce: 0.030156126245856285 (0.03522981406422347) | > loss_mel_ce: 3.0643413066864014 (nan) | > loss: 0.012279752641916275 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.13613044458659404) | > loader_time: 0.0046 (0.004641044290458887)  --> TIME: 2025-02-25 06:48:03 -- STEP: 6066/36938 -- GLOBAL_STEP: 670950 | > loss_text_ce: 0.031148353591561317 (0.03522047513688748) | > loss_mel_ce: 2.9880545139312744 (nan) | > loss: 0.011980964802205563 (nan) | > current_lr: 5e-06 | > step_time: 0.1855 (0.1361708351774181) | > loader_time: 0.0049 (0.004641728121338978)  --> TIME: 2025-02-25 06:48:13 -- STEP: 6116/36938 -- GLOBAL_STEP: 671000 | > loss_text_ce: 0.04116884991526604 (0.03522051649443459) | > loss_mel_ce: 3.1797146797180176 (nan) | > loss: 0.012781284749507904 (nan) | > current_lr: 5e-06 | > step_time: 0.1576 (0.13618047542241432) | > loader_time: 0.0047 (0.00464192377811477)  --> TIME: 2025-02-25 06:48:22 -- STEP: 6166/36938 -- GLOBAL_STEP: 671050 | > loss_text_ce: 0.029056450352072716 (0.03522647935576914) | > loss_mel_ce: 2.8759515285491943 (nan) | > loss: 0.011527810245752335 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.1362053853058789) | > loader_time: 0.0046 (0.00464181744584469)  --> TIME: 2025-02-25 06:48:31 -- STEP: 6216/36938 -- GLOBAL_STEP: 671100 | > loss_text_ce: 0.033744096755981445 (0.0352322598068258) | > loss_mel_ce: 2.7623586654663086 (nan) | > loss: 0.011095646768808365 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13619723394417219) | > loader_time: 0.0047 (0.004641837479985338)  --> TIME: 2025-02-25 06:48:41 -- STEP: 6266/36938 -- GLOBAL_STEP: 671150 | > loss_text_ce: 0.03368513286113739 (0.03523596873530099) | > loss_mel_ce: 1.8224283456802368 (nan) | > loss: 0.007365529891103506 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13623131086298945) | > loader_time: 0.0045 (0.004642215659357339)  --> TIME: 2025-02-25 06:48:50 -- STEP: 6316/36938 -- GLOBAL_STEP: 671200 | > loss_text_ce: 0.03075926937162876 (0.03523846883875311) | > loss_mel_ce: 3.302474021911621 (nan) | > loss: 0.013227117247879505 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13626941111814064) | > loader_time: 0.0045 (0.004643184312768473)  --> TIME: 2025-02-25 06:49:00 -- STEP: 6366/36938 -- GLOBAL_STEP: 671250 | > loss_text_ce: 0.03244321420788765 (0.035247551213494745) | > loss_mel_ce: 0.028076171875 (nan) | > loss: 0.00024015631061047316 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13630014828380224) | > loader_time: 0.0045 (0.004643027601922173)  --> TIME: 2025-02-25 06:49:09 -- STEP: 6416/36938 -- GLOBAL_STEP: 671300 | > loss_text_ce: 0.04746987670660019 (0.035255695672582764) | > loss_mel_ce: 1.2278691530227661 (nan) | > loss: 0.0050608692690730095 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1362895126131704) | > loader_time: 0.0046 (0.004643331256293281)  --> TIME: 2025-02-25 06:49:18 -- STEP: 6466/36938 -- GLOBAL_STEP: 671350 | > loss_text_ce: 0.028392869979143143 (0.03525034919498684) | > loss_mel_ce: 2.987764358520508 (nan) | > loss: 0.011968878097832203 (nan) | > current_lr: 5e-06 | > step_time: 0.1515 (0.1362833911112282) | > loader_time: 0.0047 (0.00464375159927022)  --> TIME: 2025-02-25 06:49:28 -- STEP: 6516/36938 -- GLOBAL_STEP: 671400 | > loss_text_ce: 0.03280934318900108 (0.03525973908875336) | > loss_mel_ce: 3.0639960765838623 (nan) | > loss: 0.012288910336792469 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.136285115964139) | > loader_time: 0.0047 (0.004643712474057234)  --> TIME: 2025-02-25 06:49:37 -- STEP: 6566/36938 -- GLOBAL_STEP: 671450 | > loss_text_ce: 0.03777259588241577 (0.03526599694723104) | > loss_mel_ce: 3.405094861984253 (nan) | > loss: 0.01366217341274023 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13626129980171522) | > loader_time: 0.0047 (0.0046433687500571546)  --> TIME: 2025-02-25 06:49:46 -- STEP: 6616/36938 -- GLOBAL_STEP: 671500 | > loss_text_ce: 0.035742662847042084 (0.03527002758656195) | > loss_mel_ce: 0.5341084599494934 (nan) | > loss: 0.0022613140754401684 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13627887518114742) | > loader_time: 0.0047 (0.004643577041764343)  --> TIME: 2025-02-25 06:49:56 -- STEP: 6666/36938 -- GLOBAL_STEP: 671550 | > loss_text_ce: 0.03973497450351715 (0.03527342983902382) | > loss_mel_ce: 2.438638210296631 (nan) | > loss: 0.009834814816713333 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1363135519737314) | > loader_time: 0.0045 (0.004643447614452898)  --> TIME: 2025-02-25 06:50:05 -- STEP: 6716/36938 -- GLOBAL_STEP: 671600 | > loss_text_ce: 0.028732148930430412 (0.03527017759365558) | > loss_mel_ce: 2.483989715576172 (nan) | > loss: 0.009971118532121181 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13633173168096038) | > loader_time: 0.0047 (0.004644247021541615)  --> TIME: 2025-02-25 06:50:15 -- STEP: 6766/36938 -- GLOBAL_STEP: 671650 | > loss_text_ce: 0.031307779252529144 (0.03528692718857355) | > loss_mel_ce: 2.719844341278076 (nan) | > loss: 0.010917270556092262 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1363813072980385) | > loader_time: 0.0047 (0.004645209146116795)  --> TIME: 2025-02-25 06:50:25 -- STEP: 6816/36938 -- GLOBAL_STEP: 671700 | > loss_text_ce: 0.030664412304759026 (0.03527391179078903) | > loss_mel_ce: 2.612731456756592 (nan) | > loss: 0.01048966683447361 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.1363960972204455) | > loader_time: 0.0046 (0.004645547011928368)  --> TIME: 2025-02-25 06:50:34 -- STEP: 6866/36938 -- GLOBAL_STEP: 671750 | > loss_text_ce: 0.028237806633114815 (0.03527257203792105) | > loss_mel_ce: 2.7618861198425293 (nan) | > loss: 0.011071921326220036 (nan) | > current_lr: 5e-06 | > step_time: 0.1848 (0.13643867161126033) | > loader_time: 0.0048 (0.0046455327810992425)  --> TIME: 2025-02-25 06:50:44 -- STEP: 6916/36938 -- GLOBAL_STEP: 671800 | > loss_text_ce: 0.024115558713674545 (0.035279800437909105) | > loss_mel_ce: 3.135956287384033 (nan) | > loss: 0.012539968825876713 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.1364392728526998) | > loader_time: 0.0046 (0.0046453170861861315)  --> TIME: 2025-02-25 06:50:53 -- STEP: 6966/36938 -- GLOBAL_STEP: 671850 | > loss_text_ce: 0.04256409406661987 (0.03527471586511602) | > loss_mel_ce: 2.787930488586426 (nan) | > loss: 0.011232121847569942 (nan) | > current_lr: 5e-06 | > step_time: 0.1774 (0.13646103981203161) | > loader_time: 0.0048 (0.004645233622509061)  --> TIME: 2025-02-25 06:51:02 -- STEP: 7016/36938 -- GLOBAL_STEP: 671900 | > loss_text_ce: 0.03683954477310181 (0.03527599285893522) | > loss_mel_ce: 0.10911254584789276 (nan) | > loss: 0.0005791750154457986 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13647710162394405) | > loader_time: 0.0046 (0.004645170650297305)  --> TIME: 2025-02-25 06:51:12 -- STEP: 7066/36938 -- GLOBAL_STEP: 671950 | > loss_text_ce: 0.03132130950689316 (0.03526943953847007) | > loss_mel_ce: 3.190300703048706 (nan) | > loss: 0.012784214690327644 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13647573330973956) | > loader_time: 0.0044 (0.004645577105995339)  --> TIME: 2025-02-25 06:51:21 -- STEP: 7116/36938 -- GLOBAL_STEP: 672000 | > loss_text_ce: 0.034757643938064575 (0.035266539520762634) | > loss_mel_ce: 3.4512484073638916 (nan) | > loss: 0.013833357952535152 (nan) | > current_lr: 5e-06 | > step_time: 0.1453 (0.13648954531394616) | > loader_time: 0.0046 (0.004646242401987187)  --> TIME: 2025-02-25 06:51:31 -- STEP: 7166/36938 -- GLOBAL_STEP: 672050 | > loss_text_ce: 0.028914105147123337 (0.03527567536353619) | > loss_mel_ce: 1.7032028436660767 (nan) | > loss: 0.006873480509966612 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13648827858487556) | > loader_time: 0.0046 (0.004646048111986113)  --> TIME: 2025-02-25 06:51:40 -- STEP: 7216/36938 -- GLOBAL_STEP: 672100 | > loss_text_ce: 0.039007578045129776 (0.035271901438687046) | > loss_mel_ce: 3.1523165702819824 (nan) | > loss: 0.01266398560255766 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.13652081975915756) | > loader_time: 0.0049 (0.00464613316064398)  --> TIME: 2025-02-25 06:51:50 -- STEP: 7266/36938 -- GLOBAL_STEP: 672150 | > loss_text_ce: 0.03811049833893776 (0.03527298940990634) | > loss_mel_ce: 0.9719128608703613 (nan) | > loss: 0.004008029587566853 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13655340694508364) | > loader_time: 0.0046 (0.004646367059408903)  --> TIME: 2025-02-25 06:51:59 -- STEP: 7316/36938 -- GLOBAL_STEP: 672200 | > loss_text_ce: 0.043164562433958054 (0.03528352053909833) | > loss_mel_ce: 3.0385444164276123 (nan) | > loss: 0.012229003943502903 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13656159960030878) | > loader_time: 0.0046 (0.004646767254668279)  --> TIME: 2025-02-25 06:52:08 -- STEP: 7366/36938 -- GLOBAL_STEP: 672250 | > loss_text_ce: 0.03490940108895302 (0.035285832354318064) | > loss_mel_ce: 0.1318359375 (nan) | > loss: 0.0006616878672502935 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13654720961254604) | > loader_time: 0.0046 (0.004647311068992388)  --> TIME: 2025-02-25 06:52:18 -- STEP: 7416/36938 -- GLOBAL_STEP: 672300 | > loss_text_ce: 0.03772774711251259 (0.035282544805763844) | > loss_mel_ce: 2.5853769779205322 (nan) | > loss: 0.010409146547317505 (nan) | > current_lr: 5e-06 | > step_time: 0.165 (0.1365464566337769) | > loader_time: 0.0046 (0.004647305160921616)  --> TIME: 2025-02-25 06:52:27 -- STEP: 7466/36938 -- GLOBAL_STEP: 672350 | > loss_text_ce: 0.0305688064545393 (0.035284100721363044) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13653581138593499) | > loader_time: 0.0046 (0.004647379709637702)  --> TIME: 2025-02-25 06:52:36 -- STEP: 7516/36938 -- GLOBAL_STEP: 672400 | > loss_text_ce: 0.0391734316945076 (0.03529840881305303) | > loss_mel_ce: 2.287677764892578 (nan) | > loss: 0.009233537130057812 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13654033702507282) | > loader_time: 0.0046 (0.004647448413101763)  --> TIME: 2025-02-25 06:52:46 -- STEP: 7566/36938 -- GLOBAL_STEP: 672450 | > loss_text_ce: 0.03211679309606552 (0.03530440566986967) | > loss_mel_ce: 2.8916449546813965 (nan) | > loss: 0.011602230370044708 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13653723776986518) | > loader_time: 0.0045 (0.004647201153137295)  --> TIME: 2025-02-25 06:52:55 -- STEP: 7616/36938 -- GLOBAL_STEP: 672500 | > loss_text_ce: 0.03852449357509613 (0.03530212149083695) | > loss_mel_ce: 2.8442981243133545 (nan) | > loss: 0.011439772322773933 (nan) | > current_lr: 5e-06 | > step_time: 0.163 (0.13651302747991945) | > loader_time: 0.0047 (0.004647211555172406)  --> TIME: 2025-02-25 06:53:04 -- STEP: 7666/36938 -- GLOBAL_STEP: 672550 | > loss_text_ce: 0.04738268256187439 (0.03529289217110362) | > loss_mel_ce: 2.546787977218628 (nan) | > loss: 0.010294328443706036 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.1365330762910436) | > loader_time: 0.0047 (0.004647250994048593)  --> TIME: 2025-02-25 06:53:13 -- STEP: 7716/36938 -- GLOBAL_STEP: 672600 | > loss_text_ce: 0.034478072077035904 (0.035288753402706756) | > loss_mel_ce: 2.7260220050811768 (nan) | > loss: 0.010954366065561771 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13651729977273064) | > loader_time: 0.0046 (0.004647539927589878)  --> TIME: 2025-02-25 06:53:22 -- STEP: 7766/36938 -- GLOBAL_STEP: 672650 | > loss_text_ce: 0.03608083352446556 (0.03528942361410728) | > loss_mel_ce: 2.9381256103515625 (nan) | > loss: 0.01180240698158741 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13648406156952075) | > loader_time: 0.0047 (0.004647450013594185)  --> TIME: 2025-02-25 06:53:32 -- STEP: 7816/36938 -- GLOBAL_STEP: 672700 | > loss_text_ce: 0.04355909675359726 (0.03528913558354019) | > loss_mel_ce: 2.902822256088257 (nan) | > loss: 0.011691990308463573 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13646912516810672) | > loader_time: 0.0046 (0.004647676172373595)  --> TIME: 2025-02-25 06:53:41 -- STEP: 7866/36938 -- GLOBAL_STEP: 672750 | > loss_text_ce: 0.039927564561367035 (0.03529655555996769) | > loss_mel_ce: 2.68579363822937 (nan) | > loss: 0.010816354304552078 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13646954220728907) | > loader_time: 0.0047 (0.004647681405760685)  --> TIME: 2025-02-25 06:53:51 -- STEP: 7916/36938 -- GLOBAL_STEP: 672800 | > loss_text_ce: 0.03458353132009506 (0.035295012968433855) | > loss_mel_ce: 1.9194012880325317 (nan) | > loss: 0.007753908634185791 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13648121949697553) | > loader_time: 0.0045 (0.004648411737782227)  --> TIME: 2025-02-25 06:54:00 -- STEP: 7966/36938 -- GLOBAL_STEP: 672850 | > loss_text_ce: 0.04552527144551277 (0.03529305677433554) | > loss_mel_ce: 2.723247766494751 (nan) | > loss: 0.010987195186316967 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13649966173246258) | > loader_time: 0.0053 (0.004649052570849656)  --> TIME: 2025-02-25 06:54:09 -- STEP: 8016/36938 -- GLOBAL_STEP: 672900 | > loss_text_ce: 0.03884732350707054 (0.03529322828852141) | > loss_mel_ce: 2.9223430156707764 (nan) | > loss: 0.011750755831599236 (nan) | > current_lr: 5e-06 | > step_time: 0.1526 (0.1365030799618742) | > loader_time: 0.0047 (0.004649248576449768)  --> TIME: 2025-02-25 06:54:19 -- STEP: 8066/36938 -- GLOBAL_STEP: 672950 | > loss_text_ce: 0.03349152207374573 (0.035288755109493586) | > loss_mel_ce: 3.2305710315704346 (nan) | > loss: 0.012952630408108234 (nan) | > current_lr: 5e-06 | > step_time: 0.1811 (0.1364802508445231) | > loader_time: 0.0048 (0.004649038206044176)  --> TIME: 2025-02-25 06:54:28 -- STEP: 8116/36938 -- GLOBAL_STEP: 673000 | > loss_text_ce: 0.04501353204250336 (0.0352915637890505) | > loss_mel_ce: 2.449939489364624 (nan) | > loss: 0.009900607168674469 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13649524400949867) | > loader_time: 0.0047 (0.004649104274159399)  --> TIME: 2025-02-25 06:54:37 -- STEP: 8166/36938 -- GLOBAL_STEP: 673050 | > loss_text_ce: 0.042486369609832764 (0.03528852718365631) | > loss_mel_ce: 3.5287156105041504 (nan) | > loss: 0.014171437360346317 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13648603956926555) | > loader_time: 0.0045 (0.00464932106302089)  --> TIME: 2025-02-25 06:54:46 -- STEP: 8216/36938 -- GLOBAL_STEP: 673100 | > loss_text_ce: 0.03620687499642372 (0.03528853962873504) | > loss_mel_ce: 3.2684757709503174 (nan) | > loss: 0.013113820925354958 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.1364680412855681) | > loader_time: 0.0056 (0.004649345343161688)  --> TIME: 2025-02-25 06:54:56 -- STEP: 8266/36938 -- GLOBAL_STEP: 673150 | > loss_text_ce: 0.0451173298060894 (0.035286719688842484) | > loss_mel_ce: 2.3975119590759277 (nan) | > loss: 0.009692974388599396 (nan) | > current_lr: 5e-06 | > step_time: 0.151 (0.1364892551289213) | > loader_time: 0.0047 (0.0046494403417307074)  --> TIME: 2025-02-25 06:55:06 -- STEP: 8316/36938 -- GLOBAL_STEP: 673200 | > loss_text_ce: 0.0334574431180954 (0.03527984753122967) | > loss_mel_ce: 3.058483362197876 (nan) | > loss: 0.012269607745110989 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13652551343316044) | > loader_time: 0.0046 (0.004649962812389754)  --> TIME: 2025-02-25 06:55:15 -- STEP: 8366/36938 -- GLOBAL_STEP: 673250 | > loss_text_ce: 0.038319334387779236 (0.035278640852222806) | > loss_mel_ce: 2.905130386352539 (nan) | > loss: 0.011680357158184052 (nan) | > current_lr: 5e-06 | > step_time: 0.1612 (0.13654274938210878) | > loader_time: 0.0051 (0.0046501847337143175)  --> TIME: 2025-02-25 06:55:25 -- STEP: 8416/36938 -- GLOBAL_STEP: 673300 | > loss_text_ce: 0.02698233723640442 (0.03527373345984871) | > loss_mel_ce: 1.592535376548767 (nan) | > loss: 0.0064266580156981945 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13654291825262801) | > loader_time: 0.0045 (0.004650393876286053)  --> TIME: 2025-02-25 06:55:34 -- STEP: 8466/36938 -- GLOBAL_STEP: 673350 | > loss_text_ce: 0.028407670557498932 (0.03526980404542079) | > loss_mel_ce: 2.6404337882995605 (nan) | > loss: 0.01059064082801342 (nan) | > current_lr: 5e-06 | > step_time: 0.1507 (0.13652811021126707) | > loader_time: 0.0046 (0.004650427775266962)  --> TIME: 2025-02-25 06:55:43 -- STEP: 8516/36938 -- GLOBAL_STEP: 673400 | > loss_text_ce: 0.032557982951402664 (0.03526771688302784) | > loss_mel_ce: 2.989675283432007 (nan) | > loss: 0.011992990039288998 (nan) | > current_lr: 5e-06 | > step_time: 0.1881 (0.13652419937558477) | > loader_time: 0.0048 (0.004650982823826544)  --> TIME: 2025-02-25 06:55:53 -- STEP: 8566/36938 -- GLOBAL_STEP: 673450 | > loss_text_ce: 0.037194233387708664 (0.0352713725973855) | > loss_mel_ce: 1.3973926305770874 (nan) | > loss: 0.005692805629223585 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13655705780952945) | > loader_time: 0.0047 (0.004651281618220261)  --> TIME: 2025-02-25 06:56:02 -- STEP: 8616/36938 -- GLOBAL_STEP: 673500 | > loss_text_ce: 0.0457054041326046 (0.035272131393168184) | > loss_mel_ce: 2.4040353298187256 (nan) | > loss: 0.009721193462610245 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13659517343883315) | > loader_time: 0.0045 (0.004651411053419328)  --> TIME: 2025-02-25 06:56:12 -- STEP: 8666/36938 -- GLOBAL_STEP: 673550 | > loss_text_ce: 0.03545817360281944 (0.035263643137072156) | > loss_mel_ce: 3.113881826400757 (nan) | > loss: 0.012497381307184696 (nan) | > current_lr: 5e-06 | > step_time: 0.1862 (0.13661176575652592) | > loader_time: 0.0049 (0.0046514887031715345)  --> TIME: 2025-02-25 06:56:21 -- STEP: 8716/36938 -- GLOBAL_STEP: 673600 | > loss_text_ce: 0.03416994959115982 (0.03526217959119855) | > loss_mel_ce: 2.5705761909484863 (nan) | > loss: 0.010336294770240784 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13661943873225604) | > loader_time: 0.0047 (0.004651788261408317)  --> TIME: 2025-02-25 06:56:31 -- STEP: 8766/36938 -- GLOBAL_STEP: 673650 | > loss_text_ce: 0.04114323854446411 (0.035266440017193534) | > loss_mel_ce: 3.1822926998138428 (nan) | > loss: 0.012791412882506847 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13662165358489242) | > loader_time: 0.0046 (0.004651989317799762)  --> TIME: 2025-02-25 06:56:40 -- STEP: 8816/36938 -- GLOBAL_STEP: 673700 | > loss_text_ce: 0.03969993442296982 (0.035255924163546025) | > loss_mel_ce: 2.9481232166290283 (nan) | > loss: 0.0118564423173666 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.13663605842637955) | > loader_time: 0.0047 (0.004651987860942702)  --> TIME: 2025-02-25 06:56:49 -- STEP: 8866/36938 -- GLOBAL_STEP: 673750 | > loss_text_ce: 0.038500092923641205 (0.0352615527832827) | > loss_mel_ce: 3.0649573802948 (nan) | > loss: 0.01231530774384737 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.1366025846751529) | > loader_time: 0.0046 (0.0046516929822663965)  --> TIME: 2025-02-25 06:56:59 -- STEP: 8916/36938 -- GLOBAL_STEP: 673800 | > loss_text_ce: 0.036282751709222794 (0.03526635045871335) | > loss_mel_ce: 2.7445180416107178 (nan) | > loss: 0.011034924536943436 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13660733878425352) | > loader_time: 0.0046 (0.00465199791192473)  --> TIME: 2025-02-25 06:57:08 -- STEP: 8966/36938 -- GLOBAL_STEP: 673850 | > loss_text_ce: 0.03600446134805679 (0.03527082835702003) | > loss_mel_ce: 2.791717052459717 (nan) | > loss: 0.01122111827135086 (nan) | > current_lr: 5e-06 | > step_time: 0.1532 (0.13660783177902747) | > loader_time: 0.0054 (0.00465206006475785)  --> TIME: 2025-02-25 06:57:17 -- STEP: 9016/36938 -- GLOBAL_STEP: 673900 | > loss_text_ce: 0.044881824404001236 (0.03527513145997598) | > loss_mel_ce: 2.1227214336395264 (nan) | > loss: 0.008601600304245949 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13660155957318468) | > loader_time: 0.0046 (0.004652406065043028)  --> TIME: 2025-02-25 06:57:27 -- STEP: 9066/36938 -- GLOBAL_STEP: 673950 | > loss_text_ce: 0.030846761539578438 (0.03527342609639169) | > loss_mel_ce: 2.4061787128448486 (nan) | > loss: 0.009670737199485302 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1366295248316058) | > loader_time: 0.0046 (0.0046523897005658005)  --> TIME: 2025-02-25 06:57:36 -- STEP: 9116/36938 -- GLOBAL_STEP: 674000 | > loss_text_ce: 0.046760715544223785 (0.03527796444889304) | > loss_mel_ce: 3.1875 (nan) | > loss: 0.012834369204938412 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13664269329634304) | > loader_time: 0.0046 (0.004652585728014699)  --> TIME: 2025-02-25 06:57:46 -- STEP: 9166/36938 -- GLOBAL_STEP: 674050 | > loss_text_ce: 0.03212895616889 (0.035273288134196924) | > loss_mel_ce: 1.8873530626296997 (nan) | > loss: 0.007616992574185133 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13665710228019598) | > loader_time: 0.0046 (0.004652838532175005)  --> TIME: 2025-02-25 06:57:55 -- STEP: 9216/36938 -- GLOBAL_STEP: 674100 | > loss_text_ce: 0.028505031019449234 (0.035274062357226896) | > loss_mel_ce: 0.6092773675918579 (nan) | > loss: 0.002530882600694895 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13667356564352928) | > loader_time: 0.0046 (0.004653355960423742)  --> TIME: 2025-02-25 06:58:04 -- STEP: 9266/36938 -- GLOBAL_STEP: 674150 | > loss_text_ce: 0.02548273093998432 (0.03528112221847737) | > loss_mel_ce: 2.65858793258667 (nan) | > loss: 0.010651074349880219 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13665920872938425) | > loader_time: 0.0045 (0.004653343263054259)  --> TIME: 2025-02-25 06:58:14 -- STEP: 9316/36938 -- GLOBAL_STEP: 674200 | > loss_text_ce: 0.042684540152549744 (0.035275635864552075) | > loss_mel_ce: 2.815145492553711 (nan) | > loss: 0.011340595781803131 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.13665798051821026) | > loader_time: 0.0048 (0.0046534651387346885)  --> TIME: 2025-02-25 06:58:23 -- STEP: 9366/36938 -- GLOBAL_STEP: 674250 | > loss_text_ce: 0.028766686096787453 (0.03527278055067205) | > loss_mel_ce: 2.5280606746673584 (nan) | > loss: 0.010146141052246094 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13667391865217332) | > loader_time: 0.0046 (0.004653654749159778)  --> TIME: 2025-02-25 06:58:32 -- STEP: 9416/36938 -- GLOBAL_STEP: 674300 | > loss_text_ce: 0.027589546516537666 (0.035270117042260206) | > loss_mel_ce: 2.5545527935028076 (nan) | > loss: 0.010246597230434418 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13664973772169722) | > loader_time: 0.0045 (0.004653533890577523)  --> TIME: 2025-02-25 06:58:41 -- STEP: 9466/36938 -- GLOBAL_STEP: 674350 | > loss_text_ce: 0.026722542941570282 (0.035270938580509816) | > loss_mel_ce: 1.4277011156082153 (nan) | > loss: 0.005771522875875235 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13663756295789956) | > loader_time: 0.0045 (0.004653190171116627)  --> TIME: 2025-02-25 06:58:51 -- STEP: 9516/36938 -- GLOBAL_STEP: 674400 | > loss_text_ce: 0.04085538163781166 (0.03527145186136154) | > loss_mel_ce: 2.58097767829895 (nan) | > loss: 0.010404099710285664 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.1366302959555189) | > loader_time: 0.0046 (0.004653287039127796)  --> TIME: 2025-02-25 06:59:00 -- STEP: 9566/36938 -- GLOBAL_STEP: 674450 | > loss_text_ce: 0.03733493760228157 (0.03527301278517521) | > loss_mel_ce: 1.0771687030792236 (nan) | > loss: 0.004422633908689022 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13663236266919215) | > loader_time: 0.0047 (0.004653300322824348)  --> TIME: 2025-02-25 06:59:09 -- STEP: 9616/36938 -- GLOBAL_STEP: 674500 | > loss_text_ce: 0.035255394876003265 (0.03527289664178326) | > loss_mel_ce: 2.850219964981079 (nan) | > loss: 0.011450299993157387 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.1366161716212453) | > loader_time: 0.0048 (0.00465310237272804)  --> TIME: 2025-02-25 06:59:19 -- STEP: 9666/36938 -- GLOBAL_STEP: 674550 | > loss_text_ce: 0.033520687371492386 (0.03527313481573884) | > loss_mel_ce: 3.5700600147247314 (nan) | > loss: 0.014299924485385418 (nan) | > current_lr: 5e-06 | > step_time: 0.1986 (0.13665157769892095) | > loader_time: 0.0048 (0.004653140671278351)  --> TIME: 2025-02-25 06:59:28 -- STEP: 9716/36938 -- GLOBAL_STEP: 674600 | > loss_text_ce: 0.03486265987157822 (0.035268028498887584) | > loss_mel_ce: 3.199930429458618 (nan) | > loss: 0.01283648144453764 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.13664711270130184) | > loader_time: 0.0046 (0.004653070924114218)  --> TIME: 2025-02-25 06:59:38 -- STEP: 9766/36938 -- GLOBAL_STEP: 674650 | > loss_text_ce: 0.041094448417425156 (0.03527029425408534) | > loss_mel_ce: 3.338036298751831 (nan) | > loss: 0.013409250415861607 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13667756564397981) | > loader_time: 0.0045 (0.004653192215856397)  --> TIME: 2025-02-25 06:59:47 -- STEP: 9816/36938 -- GLOBAL_STEP: 674700 | > loss_text_ce: 0.03548381105065346 (0.03526200154191303) | > loss_mel_ce: 3.4469313621520996 (nan) | > loss: 0.013819108717143536 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.13668756146885702) | > loader_time: 0.0053 (0.004653401338869496)  --> TIME: 2025-02-25 06:59:57 -- STEP: 9866/36938 -- GLOBAL_STEP: 674750 | > loss_text_ce: 0.04115169122815132 (0.035267955542196784) | > loss_mel_ce: 3.08317232131958 (nan) | > loss: 0.012398112565279007 (nan) | > current_lr: 5e-06 | > step_time: 0.1968 (0.1366938774275924) | > loader_time: 0.0048 (0.004653401580708453)  --> TIME: 2025-02-25 07:00:06 -- STEP: 9916/36938 -- GLOBAL_STEP: 674800 | > loss_text_ce: 0.0460108257830143 (0.03527133645788111) | > loss_mel_ce: 2.0255889892578125 (nan) | > loss: 0.008220634423196316 (nan) | > current_lr: 5e-06 | > step_time: 0.1121 (0.13670314490771124) | > loader_time: 0.0045 (0.00465342170435268)  --> TIME: 2025-02-25 07:00:15 -- STEP: 9966/36938 -- GLOBAL_STEP: 674850 | > loss_text_ce: 0.0323655866086483 (0.03526357794320694) | > loss_mel_ce: 1.5895742177963257 (nan) | > loss: 0.006436269264668226 (nan) | > current_lr: 5e-06 | > step_time: 0.1827 (0.13670250411591556) | > loader_time: 0.0049 (0.004653303134687769)  --> TIME: 2025-02-25 07:00:25 -- STEP: 10016/36938 -- GLOBAL_STEP: 674900 | > loss_text_ce: 0.04045204073190689 (0.03526804894867236) | > loss_mel_ce: 0.4589146077632904 (nan) | > loss: 0.0019816136918962 (nan) | > current_lr: 5e-06 | > step_time: 0.1861 (0.13669356265768867) | > loader_time: 0.0056 (0.00465387762925875)  --> TIME: 2025-02-25 07:00:34 -- STEP: 10066/36938 -- GLOBAL_STEP: 674950 | > loss_text_ce: 0.03150634467601776 (0.03526492678102707) | > loss_mel_ce: 2.0258336067199707 (nan) | > loss: 0.00816404726356268 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.13669505999004467) | > loader_time: 0.0047 (0.004653940588300975)  --> TIME: 2025-02-25 07:00:43 -- STEP: 10116/36938 -- GLOBAL_STEP: 675000 | > loss_text_ce: 0.035784199833869934 (0.03525991903797688) | > loss_mel_ce: 1.4796243906021118 (nan) | > loss: 0.006013526581227779 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13668598754927327) | > loader_time: 0.0045 (0.004653703794030613)  --> TIME: 2025-02-25 07:00:52 -- STEP: 10166/36938 -- GLOBAL_STEP: 675050 | > loss_text_ce: 0.03164123371243477 (0.035263247446607957) | > loss_mel_ce: 2.555636405944824 (nan) | > loss: 0.010266975499689579 (nan) | > current_lr: 5e-06 | > step_time: 0.1454 (0.1366789139515535) | > loader_time: 0.0046 (0.004653636756760372)  --> TIME: 2025-02-25 07:01:02 -- STEP: 10216/36938 -- GLOBAL_STEP: 675100 | > loss_text_ce: 0.036037079989910126 (0.035270369813333396) | > loss_mel_ce: 3.4829976558685303 (nan) | > loss: 0.013964423909783363 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13667740498377007) | > loader_time: 0.0046 (0.004653507480419688)  --> TIME: 2025-02-25 07:01:11 -- STEP: 10266/36938 -- GLOBAL_STEP: 675150 | > loss_text_ce: 0.04048973321914673 (0.035271686819862307) | > loss_mel_ce: 2.6433889865875244 (nan) | > loss: 0.010650312528014183 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13668320322046043) | > loader_time: 0.0046 (0.004653301012346476)  --> TIME: 2025-02-25 07:01:20 -- STEP: 10316/36938 -- GLOBAL_STEP: 675200 | > loss_text_ce: 0.035478536039590836 (0.035276459099904754) | > loss_mel_ce: 3.564383029937744 (nan) | > loss: 0.014285165816545486 (nan) | > current_lr: 5e-06 | > step_time: 0.1634 (0.13668356779796678) | > loader_time: 0.0047 (0.004653068557641618)  --> TIME: 2025-02-25 07:01:30 -- STEP: 10366/36938 -- GLOBAL_STEP: 675250 | > loss_text_ce: 0.03277146816253662 (0.035278089166610474) | > loss_mel_ce: 2.4567184448242188 (nan) | > loss: 0.009878929704427719 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1366849110298077) | > loader_time: 0.0046 (0.004652876686502447)  --> TIME: 2025-02-25 07:01:39 -- STEP: 10416/36938 -- GLOBAL_STEP: 675300 | > loss_text_ce: 0.04952966049313545 (0.03527687711712011) | > loss_mel_ce: 2.8871970176696777 (nan) | > loss: 0.011653677560389042 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.13667393058034882) | > loader_time: 0.0049 (0.004652783434878094)  --> TIME: 2025-02-25 07:01:48 -- STEP: 10466/36938 -- GLOBAL_STEP: 675350 | > loss_text_ce: 0.04133148491382599 (0.03527503767823737) | > loss_mel_ce: 2.5176849365234375 (nan) | > loss: 0.010154827497899532 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.1366858089856339) | > loader_time: 0.0046 (0.004652885526857185)  --> TIME: 2025-02-25 07:01:58 -- STEP: 10516/36938 -- GLOBAL_STEP: 675400 | > loss_text_ce: 0.035458095371723175 (0.03527912880800964) | > loss_mel_ce: 3.031052589416504 (nan) | > loss: 0.012168694287538528 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13670721234092706) | > loader_time: 0.0046 (0.004653009274650659)  --> TIME: 2025-02-25 07:02:07 -- STEP: 10566/36938 -- GLOBAL_STEP: 675450 | > loss_text_ce: 0.029150046408176422 (0.03527862726393958) | > loss_mel_ce: 2.994356155395508 (nan) | > loss: 0.011998040601611137 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13668473861715277) | > loader_time: 0.0046 (0.004652860104620217)  --> TIME: 2025-02-25 07:02:16 -- STEP: 10616/36938 -- GLOBAL_STEP: 675500 | > loss_text_ce: 0.03765947371721268 (0.03527129147733063) | > loss_mel_ce: 2.3730850219726562 (nan) | > loss: 0.009566446766257286 (nan) | > current_lr: 5e-06 | > step_time: 0.1782 (0.13667401491742262) | > loader_time: 0.0049 (0.0046526960798376855)  --> TIME: 2025-02-25 07:02:26 -- STEP: 10666/36938 -- GLOBAL_STEP: 675550 | > loss_text_ce: 0.042904291301965714 (0.03527476189597976) | > loss_mel_ce: 2.7998197078704834 (nan) | > loss: 0.01128065213561058 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13671319034965548) | > loader_time: 0.0046 (0.004652829257851335)  --> TIME: 2025-02-25 07:02:36 -- STEP: 10716/36938 -- GLOBAL_STEP: 675600 | > loss_text_ce: 0.0277030598372221 (0.03527474728866247) | > loss_mel_ce: 2.560868501663208 (nan) | > loss: 0.010272109881043434 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.1367281583228545) | > loader_time: 0.0046 (0.004652849303648134)  --> TIME: 2025-02-25 07:02:45 -- STEP: 10766/36938 -- GLOBAL_STEP: 675650 | > loss_text_ce: 0.03671454265713692 (0.03527693954027721) | > loss_mel_ce: 3.173288106918335 (nan) | > loss: 0.012738106772303581 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.1367004599332945) | > loader_time: 0.0047 (0.004652641573825669)  --> TIME: 2025-02-25 07:02:54 -- STEP: 10816/36938 -- GLOBAL_STEP: 675700 | > loss_text_ce: 0.029951617121696472 (0.035276172154210354) | > loss_mel_ce: 2.861445665359497 (nan) | > loss: 0.011473799124360085 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13668389693343455) | > loader_time: 0.0045 (0.004652535774298657)  --> TIME: 2025-02-25 07:03:03 -- STEP: 10866/36938 -- GLOBAL_STEP: 675750 | > loss_text_ce: 0.03346370533108711 (0.03527868026747514) | > loss_mel_ce: 3.1282482147216797 (nan) | > loss: 0.012546476908028126 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.1366818560523575) | > loader_time: 0.0046 (0.004652454952673188)  --> TIME: 2025-02-25 07:03:12 -- STEP: 10916/36938 -- GLOBAL_STEP: 675800 | > loss_text_ce: 0.03388898819684982 (0.03527753978543467) | > loss_mel_ce: 3.0567409992218018 (nan) | > loss: 0.012264405377209187 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13667207161294306) | > loader_time: 0.0045 (0.00465225408958749)  --> TIME: 2025-02-25 07:03:22 -- STEP: 10966/36938 -- GLOBAL_STEP: 675850 | > loss_text_ce: 0.043144531548023224 (0.03527475075239794) | > loss_mel_ce: 3.1640625 (nan) | > loss: 0.012727012857794762 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.1366704376253058) | > loader_time: 0.0045 (0.004652174724054954)  --> TIME: 2025-02-25 07:03:31 -- STEP: 11016/36938 -- GLOBAL_STEP: 675900 | > loss_text_ce: 0.03371131047606468 (0.035279335607382244) | > loss_mel_ce: 3.3175811767578125 (nan) | > loss: 0.013298779726028442 (nan) | > current_lr: 5e-06 | > step_time: 0.1842 (0.13665756156848013) | > loader_time: 0.0048 (0.004652014355254312)  --> TIME: 2025-02-25 07:03:40 -- STEP: 11066/36938 -- GLOBAL_STEP: 675950 | > loss_text_ce: 0.037308018654584885 (0.03528275903856008) | > loss_mel_ce: 2.619753360748291 (nan) | > loss: 0.010543894954025745 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13668634235244911) | > loader_time: 0.0044 (0.004651877777972431)  --> TIME: 2025-02-25 07:03:50 -- STEP: 11116/36938 -- GLOBAL_STEP: 676000 | > loss_text_ce: 0.039099305868148804 (0.03528172928548693) | > loss_mel_ce: 3.1672565937042236 (nan) | > loss: 0.012723634950816631 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.13667323823968677) | > loader_time: 0.0046 (0.004651688487034713)  --> TIME: 2025-02-25 07:03:59 -- STEP: 11166/36938 -- GLOBAL_STEP: 676050 | > loss_text_ce: 0.026186613366007805 (0.0352726072371079) | > loss_mel_ce: 2.8313229084014893 (nan) | > loss: 0.011339324526488781 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13666400302167508) | > loader_time: 0.0045 (0.00465140926908382)  --> TIME: 2025-02-25 07:04:08 -- STEP: 11216/36938 -- GLOBAL_STEP: 676100 | > loss_text_ce: 0.03376208245754242 (0.0352697514219923) | > loss_mel_ce: 1.6643338203430176 (nan) | > loss: 0.006738476455211639 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.1366726305447362) | > loader_time: 0.0055 (0.004651686561940895)  --> TIME: 2025-02-25 07:04:18 -- STEP: 11266/36938 -- GLOBAL_STEP: 676150 | > loss_text_ce: 0.024886149913072586 (0.03527371416704326) | > loss_mel_ce: 3.3104240894317627 (nan) | > loss: 0.013235358521342278 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13669455085282398) | > loader_time: 0.0045 (0.004651769934846717)  --> TIME: 2025-02-25 07:04:27 -- STEP: 11316/36938 -- GLOBAL_STEP: 676200 | > loss_text_ce: 0.0374881848692894 (0.03526680228384671) | > loss_mel_ce: 0.03271484375 (nan) | > loss: 0.00027858346584253013 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.1366867143680538) | > loader_time: 0.0046 (0.004651591250229811)  --> TIME: 2025-02-25 07:04:36 -- STEP: 11366/36938 -- GLOBAL_STEP: 676250 | > loss_text_ce: 0.0334450863301754 (0.03526796471449143) | > loss_mel_ce: 3.4240670204162598 (nan) | > loss: 0.013720287010073662 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.1366995471307936) | > loader_time: 0.0046 (0.004651576013143841)  --> TIME: 2025-02-25 07:04:46 -- STEP: 11416/36938 -- GLOBAL_STEP: 676300 | > loss_text_ce: 0.04064326360821724 (0.0352674193528384) | > loss_mel_ce: 3.17362642288208 (nan) | > loss: 0.012755039148032665 (nan) | > current_lr: 5e-06 | > step_time: 0.1883 (0.1367068649925549) | > loader_time: 0.0048 (0.0046514102271493505)  --> TIME: 2025-02-25 07:04:55 -- STEP: 11466/36938 -- GLOBAL_STEP: 676350 | > loss_text_ce: 0.03176644816994667 (0.03526236451556333) | > loss_mel_ce: 0.16412353515625 (nan) | > loss: 0.000777341250795871 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1367060590243645) | > loader_time: 0.0045 (0.004651299180856392)  --> TIME: 2025-02-25 07:05:05 -- STEP: 11516/36938 -- GLOBAL_STEP: 676400 | > loss_text_ce: 0.043774474412202835 (0.03525791400935168) | > loss_mel_ce: 2.773923873901367 (nan) | > loss: 0.011181342415511608 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.1367340551850366) | > loader_time: 0.0045 (0.004651299364329116)  --> TIME: 2025-02-25 07:05:14 -- STEP: 11566/36938 -- GLOBAL_STEP: 676450 | > loss_text_ce: 0.03802051395177841 (0.03525378327559379) | > loss_mel_ce: 3.4197378158569336 (nan) | > loss: 0.013721263967454433 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13674127045498993) | > loader_time: 0.0048 (0.004651254752546808)  --> TIME: 2025-02-25 07:05:24 -- STEP: 11616/36938 -- GLOBAL_STEP: 676500 | > loss_text_ce: 0.03611593320965767 (0.035255543076155554) | > loss_mel_ce: 3.0752413272857666 (nan) | > loss: 0.012346656061708927 (nan) | > current_lr: 5e-06 | > step_time: 0.1829 (0.13673839172138003) | > loader_time: 0.0049 (0.004651189835603559)  --> TIME: 2025-02-25 07:05:33 -- STEP: 11666/36938 -- GLOBAL_STEP: 676550 | > loss_text_ce: 0.03897930309176445 (0.03525394069324029) | > loss_mel_ce: 0.0620252825319767 (nan) | > loss: 0.0004008118703495711 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13671903102982003) | > loader_time: 0.0046 (0.004651084274038237)  --> TIME: 2025-02-25 07:05:42 -- STEP: 11716/36938 -- GLOBAL_STEP: 676600 | > loss_text_ce: 0.03012770600616932 (0.03525105021032613) | > loss_mel_ce: 2.4375672340393066 (nan) | > loss: 0.009792440570890903 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.1367313417813687) | > loader_time: 0.0046 (0.00465104318633751)  --> TIME: 2025-02-25 07:05:52 -- STEP: 11766/36938 -- GLOBAL_STEP: 676650 | > loss_text_ce: 0.03283130005002022 (0.03524714324678124) | > loss_mel_ce: 2.987671136856079 (nan) | > loss: 0.011986120603978634 (nan) | > current_lr: 5e-06 | > step_time: 0.1896 (0.13676262927180974) | > loader_time: 0.0048 (0.00465099750358636)  --> TIME: 2025-02-25 07:06:01 -- STEP: 11816/36938 -- GLOBAL_STEP: 676700 | > loss_text_ce: 0.03552170842885971 (0.0352503524469111) | > loss_mel_ce: 3.095768690109253 (nan) | > loss: 0.012425756081938744 (nan) | > current_lr: 5e-06 | > step_time: 0.1449 (0.13675917474175892) | > loader_time: 0.0047 (0.004650826258843365)  --> TIME: 2025-02-25 07:06:11 -- STEP: 11866/36938 -- GLOBAL_STEP: 676750 | > loss_text_ce: 0.04127730429172516 (0.035250591527552604) | > loss_mel_ce: 3.056626558303833 (nan) | > loss: 0.012293270789086819 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.1367506090593682) | > loader_time: 0.0046 (0.004650736184620872)  --> TIME: 2025-02-25 07:06:20 -- STEP: 11916/36938 -- GLOBAL_STEP: 676800 | > loss_text_ce: 0.034415826201438904 (0.03524388167782045) | > loss_mel_ce: 2.9888784885406494 (nan) | > loss: 0.01199719961732626 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.13673869566694907) | > loader_time: 0.0046 (0.0046509175182149025)  --> TIME: 2025-02-25 07:06:29 -- STEP: 11966/36938 -- GLOBAL_STEP: 676850 | > loss_text_ce: 0.03480624035000801 (0.035237354639224644) | > loss_mel_ce: 3.0259668827056885 (nan) | > loss: 0.012145926244556904 (nan) | > current_lr: 5e-06 | > step_time: 0.1576 (0.13673901278976985) | > loader_time: 0.0046 (0.004650653414795581)  --> TIME: 2025-02-25 07:06:38 -- STEP: 12016/36938 -- GLOBAL_STEP: 676900 | > loss_text_ce: 0.030663976445794106 (0.03523852688151252) | > loss_mel_ce: 2.863631248474121 (nan) | > loss: 0.011485299095511436 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13672721562547444) | > loader_time: 0.0045 (0.004650347282026157)  --> TIME: 2025-02-25 07:06:47 -- STEP: 12066/36938 -- GLOBAL_STEP: 676950 | > loss_text_ce: 0.029119551181793213 (0.03524348423250497) | > loss_mel_ce: 2.9223484992980957 (nan) | > loss: 0.011712174862623215 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13670494883249718) | > loader_time: 0.0045 (0.0046499507375445505)  --> TIME: 2025-02-25 07:06:57 -- STEP: 12116/36938 -- GLOBAL_STEP: 677000 | > loss_text_ce: 0.03541494160890579 (0.035243872743125834) | > loss_mel_ce: 3.2884111404418945 (nan) | > loss: 0.013189787045121193 (nan) | > current_lr: 5e-06 | > step_time: 0.1701 (0.13671343479663364) | > loader_time: 0.0046 (0.004649876957877932)  --> TIME: 2025-02-25 07:07:06 -- STEP: 12166/36938 -- GLOBAL_STEP: 677050 | > loss_text_ce: 0.03754306584596634 (0.03523944424348644) | > loss_mel_ce: 2.935410976409912 (nan) | > loss: 0.011797437444329262 (nan) | > current_lr: 5e-06 | > step_time: 0.1788 (0.13671857883874117) | > loader_time: 0.0049 (0.004649784853833481)  --> TIME: 2025-02-25 07:07:15 -- STEP: 12216/36938 -- GLOBAL_STEP: 677100 | > loss_text_ce: 0.03833393380045891 (0.03524434793442887) | > loss_mel_ce: 3.36328125 (nan) | > loss: 0.013498473912477493 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.13671418846863548) | > loader_time: 0.0067 (0.004649803618160652)  --> TIME: 2025-02-25 07:07:25 -- STEP: 12266/36938 -- GLOBAL_STEP: 677150 | > loss_text_ce: 0.04854869097471237 (0.0352455549458091) | > loss_mel_ce: 3.210057020187378 (nan) | > loss: 0.01293097622692585 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13672577419308962) | > loader_time: 0.0045 (0.004649564976134725)  --> TIME: 2025-02-25 07:07:34 -- STEP: 12316/36938 -- GLOBAL_STEP: 677200 | > loss_text_ce: 0.035245075821876526 (0.03524589080338717) | > loss_mel_ce: 1.1121695041656494 (nan) | > loss: 0.004553232807666063 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13670567592733568) | > loader_time: 0.0045 (0.004649516106735916)  --> TIME: 2025-02-25 07:07:43 -- STEP: 12366/36938 -- GLOBAL_STEP: 677250 | > loss_text_ce: 0.03056875243782997 (0.0352437918080605) | > loss_mel_ce: 0.1827581524848938 (nan) | > loss: 0.0008465353748761117 (nan) | > current_lr: 5e-06 | > step_time: 0.1451 (0.1367158471005218) | > loader_time: 0.0046 (0.004649595845657576)  --> TIME: 2025-02-25 07:07:53 -- STEP: 12416/36938 -- GLOBAL_STEP: 677300 | > loss_text_ce: 0.04499391093850136 (0.035247761769383476) | > loss_mel_ce: 2.808131456375122 (nan) | > loss: 0.01132192648947239 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13672241015532555) | > loader_time: 0.0045 (0.004649452461870659)  --> TIME: 2025-02-25 07:08:02 -- STEP: 12466/36938 -- GLOBAL_STEP: 677350 | > loss_text_ce: 0.032070230692625046 (0.03525085016602759) | > loss_mel_ce: 3.296097755432129 (nan) | > loss: 0.01320701651275158 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13672752355890405) | > loader_time: 0.0045 (0.004649800587174741)  --> TIME: 2025-02-25 07:08:12 -- STEP: 12516/36938 -- GLOBAL_STEP: 677400 | > loss_text_ce: 0.03254326432943344 (0.03524966228330088) | > loss_mel_ce: 3.309579849243164 (nan) | > loss: 0.01326239388436079 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.13674186437264801) | > loader_time: 0.0046 (0.0046499756320515576)  --> TIME: 2025-02-25 07:08:21 -- STEP: 12566/36938 -- GLOBAL_STEP: 677450 | > loss_text_ce: 0.04323714226484299 (0.03525484806291628) | > loss_mel_ce: 3.1346538066864014 (nan) | > loss: 0.012610679492354393 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.1367371908403702) | > loader_time: 0.0046 (0.0046498621798388776)  --> TIME: 2025-02-25 07:08:30 -- STEP: 12616/36938 -- GLOBAL_STEP: 677500 | > loss_text_ce: 0.029491638764739037 (0.03525667494344823) | > loss_mel_ce: 3.0931336879730225 (nan) | > loss: 0.012391370721161366 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13673500179034923) | > loader_time: 0.0045 (0.004649654474905484)  --> TIME: 2025-02-25 07:08:39 -- STEP: 12666/36938 -- GLOBAL_STEP: 677550 | > loss_text_ce: 0.03956771269440651 (0.03526198213689714) | > loss_mel_ce: 2.8006982803344727 (nan) | > loss: 0.011270897462964058 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13671465417361042) | > loader_time: 0.0046 (0.004649669077843253)  --> TIME: 2025-02-25 07:08:49 -- STEP: 12716/36938 -- GLOBAL_STEP: 677600 | > loss_text_ce: 0.045796893537044525 (0.03527046264433138) | > loss_mel_ce: 1.8121541738510132 (nan) | > loss: 0.007372822146862745 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13673040599858952) | > loader_time: 0.0045 (0.0046499054287169085)  --> TIME: 2025-02-25 07:08:58 -- STEP: 12766/36938 -- GLOBAL_STEP: 677650 | > loss_text_ce: 0.032974787056446075 (0.035274335611001816) | > loss_mel_ce: 2.931278705596924 (nan) | > loss: 0.011762911453843117 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.136698146682911) | > loader_time: 0.0046 (0.004649670860264222)  --> TIME: 2025-02-25 07:09:07 -- STEP: 12816/36938 -- GLOBAL_STEP: 677700 | > loss_text_ce: 0.038618456572294235 (0.03527521958386305) | > loss_mel_ce: 3.253354072570801 (nan) | > loss: 0.013063384220004082 (nan) | > current_lr: 5e-06 | > step_time: 0.1809 (0.13670455990882366) | > loader_time: 0.0049 (0.004649526580293817)  --> TIME: 2025-02-25 07:09:16 -- STEP: 12866/36938 -- GLOBAL_STEP: 677750 | > loss_text_ce: 0.03388788551092148 (0.03527620606606214) | > loss_mel_ce: 3.2855398654937744 (nan) | > loss: 0.013172333128750324 (nan) | > current_lr: 5e-06 | > step_time: 0.1622 (0.13670513569873177) | > loader_time: 0.0046 (0.004649353716693279)  --> TIME: 2025-02-25 07:09:26 -- STEP: 12916/36938 -- GLOBAL_STEP: 677800 | > loss_text_ce: 0.038577839732170105 (0.03528229922401454) | > loss_mel_ce: 3.1438708305358887 (nan) | > loss: 0.012628764845430851 (nan) | > current_lr: 5e-06 | > step_time: 0.161 (0.13669162229990992) | > loader_time: 0.0046 (0.004649715255236386)  --> TIME: 2025-02-25 07:09:35 -- STEP: 12966/36938 -- GLOBAL_STEP: 677850 | > loss_text_ce: 0.039789192378520966 (0.03527937564326854) | > loss_mel_ce: 0.0087890625 (nan) | > loss: 0.0001927708653965965 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13669900156439146) | > loader_time: 0.0046 (0.004649445062872977)  --> TIME: 2025-02-25 07:09:44 -- STEP: 13016/36938 -- GLOBAL_STEP: 677900 | > loss_text_ce: 0.04101181775331497 (0.03527759144232364) | > loss_mel_ce: 1.7318826913833618 (nan) | > loss: 0.007035295944660902 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13667731105218842) | > loader_time: 0.0045 (0.004649407506795026)  --> TIME: 2025-02-25 07:09:53 -- STEP: 13066/36938 -- GLOBAL_STEP: 677950 | > loss_text_ce: 0.031323615461587906 (0.035278609608049655) | > loss_mel_ce: 3.203089714050293 (nan) | > loss: 0.012834974564611912 (nan) | > current_lr: 5e-06 | > step_time: 0.1355 (0.13667826050002602) | > loader_time: 0.0046 (0.004649290424675656)  --> TIME: 2025-02-25 07:10:02 -- STEP: 13116/36938 -- GLOBAL_STEP: 678000 | > loss_text_ce: 0.038569316267967224 (0.035277470323371134) | > loss_mel_ce: 2.5526766777038574 (nan) | > loss: 0.010282722301781178 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13666874628381098) | > loader_time: 0.0045 (0.004649477911708039)  --> TIME: 2025-02-25 07:10:12 -- STEP: 13166/36938 -- GLOBAL_STEP: 678050 | > loss_text_ce: 0.028390798717737198 (0.03528010390918769) | > loss_mel_ce: 2.1715171337127686 (nan) | > loss: 0.008729794062674046 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13665177888028512) | > loader_time: 0.0046 (0.004649580765436902)  --> TIME: 2025-02-25 07:10:21 -- STEP: 13216/36938 -- GLOBAL_STEP: 678100 | > loss_text_ce: 0.03752099350094795 (0.03528312344807909) | > loss_mel_ce: 2.743269920349121 (nan) | > loss: 0.011034884490072727 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.13664356186156085) | > loader_time: 0.0047 (0.0046494214211480036)  --> TIME: 2025-02-25 07:10:30 -- STEP: 13266/36938 -- GLOBAL_STEP: 678150 | > loss_text_ce: 0.035876188427209854 (0.03528224543060234) | > loss_mel_ce: 3.1156387329101562 (nan) | > loss: 0.012506012804806232 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13662938630978058) | > loader_time: 0.0047 (0.00464971002978148)  --> TIME: 2025-02-25 07:10:39 -- STEP: 13316/36938 -- GLOBAL_STEP: 678200 | > loss_text_ce: 0.032365813851356506 (0.03527796863268856) | > loss_mel_ce: 1.228833794593811 (nan) | > loss: 0.005004760809242725 (nan) | > current_lr: 5e-06 | > step_time: 0.1814 (0.13661609096332267) | > loader_time: 0.0048 (0.004649810996561226)  --> TIME: 2025-02-25 07:10:49 -- STEP: 13366/36938 -- GLOBAL_STEP: 678250 | > loss_text_ce: 0.03682751953601837 (0.0352767235635692) | > loss_mel_ce: 2.5792479515075684 (nan) | > loss: 0.010381252504885197 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13662595686826176) | > loader_time: 0.0048 (0.004649781296036649)  --> TIME: 2025-02-25 07:10:58 -- STEP: 13416/36938 -- GLOBAL_STEP: 678300 | > loss_text_ce: 0.033830270171165466 (0.035279012912649545) | > loss_mel_ce: 2.7119524478912354 (nan) | > loss: 0.01089596375823021 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13661800068122804) | > loader_time: 0.0046 (0.0046497920331460135)  --> TIME: 2025-02-25 07:11:07 -- STEP: 13466/36938 -- GLOBAL_STEP: 678350 | > loss_text_ce: 0.029753824695944786 (0.0352801678790901) | > loss_mel_ce: 2.9410834312438965 (nan) | > loss: 0.011789037846028805 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13662219213975502) | > loader_time: 0.0046 (0.00464970127498343)  --> TIME: 2025-02-25 07:11:16 -- STEP: 13516/36938 -- GLOBAL_STEP: 678400 | > loss_text_ce: 0.035185087472200394 (0.03527937012134211) | > loss_mel_ce: 4.03819465637207 (nan) | > loss: 0.016164205968379974 (nan) | > current_lr: 5e-06 | > step_time: 0.1458 (0.13662847605778922) | > loader_time: 0.0046 (0.004649725070399049)  --> TIME: 2025-02-25 07:11:26 -- STEP: 13566/36938 -- GLOBAL_STEP: 678450 | > loss_text_ce: 0.03276439756155014 (0.03527451325302699) | > loss_mel_ce: 2.608790159225464 (nan) | > loss: 0.010482359677553177 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13661546292741533) | > loader_time: 0.0046 (0.004649955474502576)  --> TIME: 2025-02-25 07:11:35 -- STEP: 13616/36938 -- GLOBAL_STEP: 678500 | > loss_text_ce: 0.0364416241645813 (0.035272654171469034) | > loss_mel_ce: 3.1252059936523438 (nan) | > loss: 0.012546220794320107 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.13661860887368316) | > loader_time: 0.0048 (0.004650149008502123)  --> TIME: 2025-02-25 07:11:45 -- STEP: 13666/36938 -- GLOBAL_STEP: 678550 | > loss_text_ce: 0.03170940279960632 (0.03527289680877926) | > loss_mel_ce: 1.4728337526321411 (nan) | > loss: 0.005970409605652094 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13663656738096444) | > loader_time: 0.0046 (0.0046500200131758315)  --> TIME: 2025-02-25 07:11:54 -- STEP: 13716/36938 -- GLOBAL_STEP: 678600 | > loss_text_ce: 0.026003854349255562 (0.03526854864538945) | > loss_mel_ce: 1.7994791269302368 (nan) | > loss: 0.0072439806535840034 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13664936981106598) | > loader_time: 0.0045 (0.004649890393896532)  --> TIME: 2025-02-25 07:12:04 -- STEP: 13766/36938 -- GLOBAL_STEP: 678650 | > loss_text_ce: 0.02183966152369976 (0.03527094263424583) | > loss_mel_ce: 2.32285213470459 (nan) | > loss: 0.009304332546889782 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.1366563850617867) | > loader_time: 0.0047 (0.00464980406209027)  --> TIME: 2025-02-25 07:12:13 -- STEP: 13816/36938 -- GLOBAL_STEP: 678700 | > loss_text_ce: 0.03129599615931511 (0.03527011260931989) | > loss_mel_ce: 0.08447265625 (nan) | > loss: 0.0004593994526658207 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.13665872391207595) | > loader_time: 0.0046 (0.004649834544519221)  --> TIME: 2025-02-25 07:12:22 -- STEP: 13866/36938 -- GLOBAL_STEP: 678750 | > loss_text_ce: 0.03008432500064373 (0.035262169911433945) | > loss_mel_ce: 3.017061710357666 (nan) | > loss: 0.01209184993058443 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13665137255877835) | > loader_time: 0.0048 (0.004649866131086906)  --> TIME: 2025-02-25 07:12:32 -- STEP: 13916/36938 -- GLOBAL_STEP: 678800 | > loss_text_ce: 0.04396480321884155 (0.035266754721187336) | > loss_mel_ce: 2.8431434631347656 (nan) | > loss: 0.011456780135631561 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13665801387780263) | > loader_time: 0.0046 (0.004649959648090651)  --> TIME: 2025-02-25 07:12:41 -- STEP: 13966/36938 -- GLOBAL_STEP: 678850 | > loss_text_ce: 0.03582718223333359 (0.03527111181288864) | > loss_mel_ce: 3.513540029525757 (nan) | > loss: 0.014084790833294392 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.13666922904761192) | > loader_time: 0.0046 (0.00464994928206206)  --> TIME: 2025-02-25 07:12:51 -- STEP: 14016/36938 -- GLOBAL_STEP: 678900 | > loss_text_ce: 0.04301641508936882 (0.0352689708449765) | > loss_mel_ce: 3.05851411819458 (nan) | > loss: 0.012307661585509777 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13667808792906783) | > loader_time: 0.0046 (0.004649868651748242)  --> TIME: 2025-02-25 07:13:00 -- STEP: 14066/36938 -- GLOBAL_STEP: 678950 | > loss_text_ce: 0.038720738142728806 (0.03527150697735758) | > loss_mel_ce: 2.7143940925598145 (nan) | > loss: 0.010925060138106346 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13669312544369358) | > loader_time: 0.0046 (0.004649788899762719)  --> TIME: 2025-02-25 07:13:10 -- STEP: 14116/36938 -- GLOBAL_STEP: 679000 | > loss_text_ce: 0.03439316526055336 (0.03527417805524312) | > loss_mel_ce: 2.6674296855926514 (nan) | > loss: 0.010721519589424133 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13667708017157887) | > loader_time: 0.0044 (0.004649596279827446)  --> TIME: 2025-02-25 07:13:19 -- STEP: 14166/36938 -- GLOBAL_STEP: 679050 | > loss_text_ce: 0.04788382351398468 (0.035274393664225766) | > loss_mel_ce: 2.128622055053711 (nan) | > loss: 0.008636928163468838 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13667457692454477) | > loader_time: 0.0045 (0.00464946003800375)  --> TIME: 2025-02-25 07:13:28 -- STEP: 14216/36938 -- GLOBAL_STEP: 679100 | > loss_text_ce: 0.035904817283153534 (0.035276063436264514) | > loss_mel_ce: 2.5993103981018066 (nan) | > loss: 0.010457203723490238 (nan) | > current_lr: 5e-06 | > step_time: 0.1824 (0.13667474417922584) | > loader_time: 0.0057 (0.004649513094493621)  --> TIME: 2025-02-25 07:13:37 -- STEP: 14266/36938 -- GLOBAL_STEP: 679150 | > loss_text_ce: 0.03992592170834541 (0.03527322673731971) | > loss_mel_ce: 0.02239990234375 (nan) | > loss: 0.0002473247004672885 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.1366654479886343) | > loader_time: 0.0046 (0.004649630572908504)  --> TIME: 2025-02-25 07:13:47 -- STEP: 14316/36938 -- GLOBAL_STEP: 679200 | > loss_text_ce: 0.03613671287894249 (0.03527511674180884) | > loss_mel_ce: 0.873870849609375 (nan) | > loss: 0.0036111411172896624 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13666754404770373) | > loader_time: 0.0045 (0.004649682113597359)  --> TIME: 2025-02-25 07:13:56 -- STEP: 14366/36938 -- GLOBAL_STEP: 679250 | > loss_text_ce: 0.035745181143283844 (0.035268565834792154) | > loss_mel_ce: 2.647006034851074 (nan) | > loss: 0.01064583845436573 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13666936603741556) | > loader_time: 0.0046 (0.00464961541391059)  --> TIME: 2025-02-25 07:14:05 -- STEP: 14416/36938 -- GLOBAL_STEP: 679300 | > loss_text_ce: 0.038543034344911575 (0.03526210742409547) | > loss_mel_ce: 2.0639989376068115 (nan) | > loss: 0.008343420922756195 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.1366511767070644) | > loader_time: 0.0047 (0.004649492416874015)  --> TIME: 2025-02-25 07:14:14 -- STEP: 14466/36938 -- GLOBAL_STEP: 679350 | > loss_text_ce: 0.03508514165878296 (0.03526534085779661) | > loss_mel_ce: 3.0334689617156982 (nan) | > loss: 0.012176803313195705 (nan) | > current_lr: 5e-06 | > step_time: 0.1842 (0.136633868750144) | > loader_time: 0.0048 (0.0046495137398591966)  --> TIME: 2025-02-25 07:14:24 -- STEP: 14516/36938 -- GLOBAL_STEP: 679400 | > loss_text_ce: 0.04034165292978287 (0.03526522524067958) | > loss_mel_ce: 2.75 (nan) | > loss: 0.011072784662246704 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13665156262280298) | > loader_time: 0.0045 (0.0046494367629182805)  --> TIME: 2025-02-25 07:14:33 -- STEP: 14566/36938 -- GLOBAL_STEP: 679450 | > loss_text_ce: 0.03468308597803116 (0.03526069513143204) | > loss_mel_ce: 3.3368208408355713 (nan) | > loss: 0.013378984294831753 (nan) | > current_lr: 5e-06 | > step_time: 0.1885 (0.13666586834667777) | > loader_time: 0.007 (0.004649557943578502)  --> TIME: 2025-02-25 07:14:42 -- STEP: 14616/36938 -- GLOBAL_STEP: 679500 | > loss_text_ce: 0.03738570958375931 (0.03526159362798925) | > loss_mel_ce: 3.386979103088379 (nan) | > loss: 0.013588749803602695 (nan) | > current_lr: 5e-06 | > step_time: 0.1496 (0.13664787064893172) | > loader_time: 0.0046 (0.004649462762528838)  --> TIME: 2025-02-25 07:14:51 -- STEP: 14666/36938 -- GLOBAL_STEP: 679550 | > loss_text_ce: 0.02953185886144638 (0.03525795470081159) | > loss_mel_ce: 0.0693766251206398 (nan) | > loss: 0.0003924939956050366 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13663478666904036) | > loader_time: 0.0045 (0.004649507565370244)  --> TIME: 2025-02-25 07:15:01 -- STEP: 14716/36938 -- GLOBAL_STEP: 679600 | > loss_text_ce: 0.04567848518490791 (0.035266035343562235) | > loss_mel_ce: 3.239894390106201 (nan) | > loss: 0.013037987984716892 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13664667269042335) | > loader_time: 0.0046 (0.004649518608042449)  --> TIME: 2025-02-25 07:15:10 -- STEP: 14766/36938 -- GLOBAL_STEP: 679650 | > loss_text_ce: 0.03502300754189491 (0.035265923685279685) | > loss_mel_ce: 2.598646879196167 (nan) | > loss: 0.01045107189565897 (nan) | > current_lr: 5e-06 | > step_time: 0.1623 (0.13664000406992494) | > loader_time: 0.0046 (0.004649419408656381)  --> TIME: 2025-02-25 07:15:19 -- STEP: 14816/36938 -- GLOBAL_STEP: 679700 | > loss_text_ce: 0.038606688380241394 (0.03526819454998617) | > loss_mel_ce: 2.602236270904541 (nan) | > loss: 0.010479535907506943 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13663478247161556) | > loader_time: 0.0045 (0.004649335892617568)  --> TIME: 2025-02-25 07:15:29 -- STEP: 14866/36938 -- GLOBAL_STEP: 679750 | > loss_text_ce: 0.02874966710805893 (0.035265680993599675) | > loss_mel_ce: 3.1873104572296143 (nan) | > loss: 0.012762144207954407 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13663652024037518) | > loader_time: 0.0046 (0.004649228737265423)  --> TIME: 2025-02-25 07:15:38 -- STEP: 14916/36938 -- GLOBAL_STEP: 679800 | > loss_text_ce: 0.0440744012594223 (0.03526810707805901) | > loss_mel_ce: 0.16828003525733948 (nan) | > loss: 0.0008426763815805316 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.13662587076257718) | > loader_time: 0.0046 (0.004649065492875003)  --> TIME: 2025-02-25 07:15:47 -- STEP: 14966/36938 -- GLOBAL_STEP: 679850 | > loss_text_ce: 0.0362214669585228 (0.03526955982707538) | > loss_mel_ce: 2.371887445449829 (nan) | > loss: 0.009555988013744354 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13662372710120776) | > loader_time: 0.0045 (0.004649208602971539)  --> TIME: 2025-02-25 07:15:56 -- STEP: 15016/36938 -- GLOBAL_STEP: 679900 | > loss_text_ce: 0.03271383047103882 (0.03527355261615177) | > loss_mel_ce: 2.822793960571289 (nan) | > loss: 0.011331381276249886 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13662200469124738) | > loader_time: 0.0045 (0.004649313050632339)  --> TIME: 2025-02-25 07:16:06 -- STEP: 15066/36938 -- GLOBAL_STEP: 679950 | > loss_text_ce: 0.029573509469628334 (0.03527068758310957) | > loss_mel_ce: 0.6269391179084778 (nan) | > loss: 0.002605208894237876 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13661815916638945) | > loader_time: 0.0046 (0.004649247683868545)  --> TIME: 2025-02-25 07:16:15 -- STEP: 15116/36938 -- GLOBAL_STEP: 680000 | > loss_text_ce: 0.037021659314632416 (0.03527029269895168) | > loss_mel_ce: 3.561056137084961 (nan) | > loss: 0.014278086833655834 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.13662255998009817) | > loader_time: 0.0048 (0.004649276470311511) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_680000.pth  --> TIME: 2025-02-25 07:16:31 -- STEP: 15166/36938 -- GLOBAL_STEP: 680050 | > loss_text_ce: 0.04473311826586723 (0.035272782019628364) | > loss_mel_ce: 3.9309895038604736 (nan) | > loss: 0.015776677057147026 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13662517848939829) | > loader_time: 0.0046 (0.004649316857393409)  --> TIME: 2025-02-25 07:16:40 -- STEP: 15216/36938 -- GLOBAL_STEP: 680100 | > loss_text_ce: 0.032851457595825195 (0.03527013454600969) | > loss_mel_ce: 3.283978223800659 (nan) | > loss: 0.013162023387849331 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13661512229532846) | > loader_time: 0.0046 (0.004649251386068842)  --> TIME: 2025-02-25 07:16:49 -- STEP: 15266/36938 -- GLOBAL_STEP: 680150 | > loss_text_ce: 0.034695759415626526 (0.035264457155887406) | > loss_mel_ce: 3.376696825027466 (nan) | > loss: 0.013537273742258549 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.13661505601891177) | > loader_time: 0.0046 (0.004649215579798156)  --> TIME: 2025-02-25 07:16:58 -- STEP: 15316/36938 -- GLOBAL_STEP: 680200 | > loss_text_ce: 0.027417927980422974 (0.035262531819555236) | > loss_mel_ce: 0.9773356318473816 (nan) | > loss: 0.003987117670476437 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13662170087366507) | > loader_time: 0.0046 (0.004649242429516583)  --> TIME: 2025-02-25 07:17:08 -- STEP: 15366/36938 -- GLOBAL_STEP: 680250 | > loss_text_ce: 0.03685139864683151 (0.03526428554289042) | > loss_mel_ce: 1.7784432172775269 (nan) | > loss: 0.007203550543636084 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13661873981416742) | > loader_time: 0.0046 (0.004649242292883551)  --> TIME: 2025-02-25 07:17:17 -- STEP: 15416/36938 -- GLOBAL_STEP: 680300 | > loss_text_ce: 0.03231409564614296 (0.03526440456054815) | > loss_mel_ce: 2.510988712310791 (nan) | > loss: 0.010092471726238728 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13660564234545022) | > loader_time: 0.0045 (0.004649289435652414)  --> TIME: 2025-02-25 07:17:26 -- STEP: 15466/36938 -- GLOBAL_STEP: 680350 | > loss_text_ce: 0.04115498438477516 (0.03526262059248008) | > loss_mel_ce: 2.560746669769287 (nan) | > loss: 0.010325008071959019 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13660215036566592) | > loader_time: 0.0045 (0.004649180205475997)  --> TIME: 2025-02-25 07:17:35 -- STEP: 15516/36938 -- GLOBAL_STEP: 680400 | > loss_text_ce: 0.04155896231532097 (0.03526270674151692) | > loss_mel_ce: 2.7990562915802 (nan) | > loss: 0.011272283270955086 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.13659908593884135) | > loader_time: 0.0049 (0.004649159895387247)  --> TIME: 2025-02-25 07:17:44 -- STEP: 15566/36938 -- GLOBAL_STEP: 680450 | > loss_text_ce: 0.036635659635066986 (0.0352620138073195) | > loss_mel_ce: 2.732236623764038 (nan) | > loss: 0.01098758913576603 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13659345144847507) | > loader_time: 0.0045 (0.004649134263057377)  --> TIME: 2025-02-25 07:17:54 -- STEP: 15616/36938 -- GLOBAL_STEP: 680500 | > loss_text_ce: 0.03096882998943329 (0.035260132886564904) | > loss_mel_ce: 3.1300926208496094 (nan) | > loss: 0.012543895281851292 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13660697834413557) | > loader_time: 0.0046 (0.004649136322321457)  --> TIME: 2025-02-25 07:18:04 -- STEP: 15666/36938 -- GLOBAL_STEP: 680550 | > loss_text_ce: 0.030021730810403824 (0.03525931344956236) | > loss_mel_ce: 1.4543899297714233 (nan) | > loss: 0.005890523083508015 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.1366137209088352) | > loader_time: 0.0046 (0.004649137181370064)  --> TIME: 2025-02-25 07:18:13 -- STEP: 15716/36938 -- GLOBAL_STEP: 680600 | > loss_text_ce: 0.03205035254359245 (0.035256046422189144) | > loss_mel_ce: 2.717447519302368 (nan) | > loss: 0.010910706594586372 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13659825588067573) | > loader_time: 0.0048 (0.004649123471333612)  --> TIME: 2025-02-25 07:18:22 -- STEP: 15766/36938 -- GLOBAL_STEP: 680650 | > loss_text_ce: 0.03391991928219795 (0.03525260517064866) | > loss_mel_ce: 2.282377243041992 (nan) | > loss: 0.009191655553877354 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13660235057457226) | > loader_time: 0.0045 (0.004649085410579147)  --> TIME: 2025-02-25 07:18:31 -- STEP: 15816/36938 -- GLOBAL_STEP: 680700 | > loss_text_ce: 0.026701273396611214 (0.035249844415218966) | > loss_mel_ce: 2.893263101577759 (nan) | > loss: 0.011587160639464855 (nan) | > current_lr: 5e-06 | > step_time: 0.1121 (0.13659551100533115) | > loader_time: 0.0046 (0.004649113164625804)  --> TIME: 2025-02-25 07:18:40 -- STEP: 15866/36938 -- GLOBAL_STEP: 680750 | > loss_text_ce: 0.03219602257013321 (0.035248605873011804) | > loss_mel_ce: 2.5495564937591553 (nan) | > loss: 0.010245050303637981 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13657982076204664) | > loader_time: 0.0046 (0.004649042978001825)  --> TIME: 2025-02-25 07:18:50 -- STEP: 15916/36938 -- GLOBAL_STEP: 680800 | > loss_text_ce: 0.0345599539577961 (0.0352500834128722) | > loss_mel_ce: 2.8937196731567383 (nan) | > loss: 0.011620158329606056 (nan) | > current_lr: 5e-06 | > step_time: 0.1518 (0.136578279333697) | > loader_time: 0.0047 (0.004649012928843103)  --> TIME: 2025-02-25 07:18:59 -- STEP: 15966/36938 -- GLOBAL_STEP: 680850 | > loss_text_ce: 0.04082200303673744 (0.03525034043707873) | > loss_mel_ce: 3.395280599594116 (nan) | > loss: 0.013635328970849514 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13658563512527117) | > loader_time: 0.0046 (0.004649175791816272)  --> TIME: 2025-02-25 07:19:09 -- STEP: 16016/36938 -- GLOBAL_STEP: 680900 | > loss_text_ce: 0.0491340234875679 (0.035248754454867935) | > loss_mel_ce: 1.8320611715316772 (nan) | > loss: 0.00746506080031395 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13659747406378847) | > loader_time: 0.0046 (0.00464923633681191)  --> TIME: 2025-02-25 07:19:18 -- STEP: 16066/36938 -- GLOBAL_STEP: 680950 | > loss_text_ce: 0.02720101736485958 (0.03524859512096579) | > loss_mel_ce: 2.2101662158966064 (nan) | > loss: 0.008878441527485847 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13658638598834918) | > loader_time: 0.0046 (0.0046490603420485934)  --> TIME: 2025-02-25 07:19:27 -- STEP: 16116/36938 -- GLOBAL_STEP: 681000 | > loss_text_ce: 0.03539332002401352 (0.03524875400150352) | > loss_mel_ce: 0.01129150390625 (nan) | > loss: 0.00018525724590290338 (nan) | > current_lr: 5e-06 | > step_time: 0.1357 (0.13659206316056277) | > loader_time: 0.0046 (0.0046489766141740415)  --> TIME: 2025-02-25 07:19:36 -- STEP: 16166/36938 -- GLOBAL_STEP: 681050 | > loss_text_ce: 0.029302086681127548 (0.03524726333919766) | > loss_mel_ce: 2.4590508937835693 (nan) | > loss: 0.009874417446553707 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13658234128618124) | > loader_time: 0.0045 (0.0046487888250996276)  --> TIME: 2025-02-25 07:19:45 -- STEP: 16216/36938 -- GLOBAL_STEP: 681100 | > loss_text_ce: 0.04082804173231125 (0.035250638224118926) | > loss_mel_ce: 0.013519287109375 (nan) | > loss: 0.00021566401119343936 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13657446425420988) | > loader_time: 0.0046 (0.004648715184131273)  --> TIME: 2025-02-25 07:19:55 -- STEP: 16266/36938 -- GLOBAL_STEP: 681150 | > loss_text_ce: 0.03181718662381172 (0.03524745035359823) | > loss_mel_ce: 3.0201127529144287 (nan) | > loss: 0.012110834009945393 (nan) | > current_lr: 5e-06 | > step_time: 0.1634 (0.13659011558399975) | > loader_time: 0.0046 (0.004648555018402586)  --> TIME: 2025-02-25 07:20:04 -- STEP: 16316/36938 -- GLOBAL_STEP: 681200 | > loss_text_ce: 0.029626736417412758 (0.03524755796441122) | > loss_mel_ce: 1.4066903591156006 (nan) | > loss: 0.0056996713392436504 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13657398864199463) | > loader_time: 0.0046 (0.004648291383721304)  --> TIME: 2025-02-25 07:20:14 -- STEP: 16366/36938 -- GLOBAL_STEP: 681250 | > loss_text_ce: 0.03796451538801193 (0.03524627512219367) | > loss_mel_ce: 2.8084075450897217 (nan) | > loss: 0.011295127682387829 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.1365897247148262) | > loader_time: 0.0046 (0.004648418541891593)  --> TIME: 2025-02-25 07:20:23 -- STEP: 16416/36938 -- GLOBAL_STEP: 681300 | > loss_text_ce: 0.0355670228600502 (0.03524776858434893) | > loss_mel_ce: 2.8978333473205566 (nan) | > loss: 0.011640478856861591 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13657605444711085) | > loader_time: 0.0046 (0.004648366140575664)  --> TIME: 2025-02-25 07:20:32 -- STEP: 16466/36938 -- GLOBAL_STEP: 681350 | > loss_text_ce: 0.04032273218035698 (0.03524849365281985) | > loss_mel_ce: 2.723705291748047 (nan) | > loss: 0.010968365706503391 (nan) | > current_lr: 5e-06 | > step_time: 0.1448 (0.13658159313146298) | > loader_time: 0.0046 (0.00464851465576201)  --> TIME: 2025-02-25 07:20:42 -- STEP: 16516/36938 -- GLOBAL_STEP: 681400 | > loss_text_ce: 0.045519642531871796 (0.03524938685131345) | > loss_mel_ce: 3.476519823074341 (nan) | > loss: 0.013976347632706165 (nan) | > current_lr: 5e-06 | > step_time: 0.1527 (0.13659370045131844) | > loader_time: 0.0046 (0.00464874481456162)  --> TIME: 2025-02-25 07:20:51 -- STEP: 16566/36938 -- GLOBAL_STEP: 681450 | > loss_text_ce: 0.03890175372362137 (0.035251468779099016) | > loss_mel_ce: 2.477637767791748 (nan) | > loss: 0.009986268356442451 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13658558215609096) | > loader_time: 0.0047 (0.004648992509552922)  --> TIME: 2025-02-25 07:21:00 -- STEP: 16616/36938 -- GLOBAL_STEP: 681500 | > loss_text_ce: 0.033527251332998276 (0.035249093986389084) | > loss_mel_ce: 3.0209410190582275 (nan) | > loss: 0.012120906263589859 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13658404073350178) | > loader_time: 0.0047 (0.004649164932647686)  --> TIME: 2025-02-25 07:21:10 -- STEP: 16666/36938 -- GLOBAL_STEP: 681550 | > loss_text_ce: 0.03719743713736534 (0.03525479951393746) | > loss_mel_ce: 2.8081753253936768 (nan) | > loss: 0.011291162110865116 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.13658438357759836) | > loader_time: 0.0046 (0.004649135097369528)  --> TIME: 2025-02-25 07:21:19 -- STEP: 16716/36938 -- GLOBAL_STEP: 681600 | > loss_text_ce: 0.04183831065893173 (0.035254016097893213) | > loss_mel_ce: 2.9315297603607178 (nan) | > loss: 0.011799080297350883 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13658137791006722) | > loader_time: 0.0046 (0.0046491896059274)  --> TIME: 2025-02-25 07:21:28 -- STEP: 16766/36938 -- GLOBAL_STEP: 681650 | > loss_text_ce: 0.03223417326807976 (0.03525409268357069) | > loss_mel_ce: 2.4492762088775635 (nan) | > loss: 0.009847263805568218 (nan) | > current_lr: 5e-06 | > step_time: 0.1948 (0.13658799076228217) | > loader_time: 0.0049 (0.004649216671142428)  --> TIME: 2025-02-25 07:21:38 -- STEP: 16816/36938 -- GLOBAL_STEP: 681700 | > loss_text_ce: 0.04318004474043846 (0.03525620900218801) | > loss_mel_ce: 2.708503007888794 (nan) | > loss: 0.010919377207756042 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13658516772523824) | > loader_time: 0.0045 (0.00464916354015143)  --> TIME: 2025-02-25 07:21:47 -- STEP: 16866/36938 -- GLOBAL_STEP: 681750 | > loss_text_ce: 0.039375029504299164 (0.035258061966681) | > loss_mel_ce: 2.49996280670166 (nan) | > loss: 0.010076737962663174 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.13659457757240318) | > loader_time: 0.0045 (0.0046492264418645495)  --> TIME: 2025-02-25 07:21:57 -- STEP: 16916/36938 -- GLOBAL_STEP: 681800 | > loss_text_ce: 0.039824217557907104 (0.035258348895387585) | > loss_mel_ce: 2.5522966384887695 (nan) | > loss: 0.010286194272339344 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13660285912047168) | > loader_time: 0.0045 (0.004649251537363586)  --> TIME: 2025-02-25 07:22:06 -- STEP: 16966/36938 -- GLOBAL_STEP: 681850 | > loss_text_ce: 0.029755154624581337 (0.03525665867771796) | > loss_mel_ce: 2.5299127101898193 (nan) | > loss: 0.010157412849366665 (nan) | > current_lr: 5e-06 | > step_time: 0.196 (0.13661198465382596) | > loader_time: 0.005 (0.004649400978061396)  --> TIME: 2025-02-25 07:22:16 -- STEP: 17016/36938 -- GLOBAL_STEP: 681900 | > loss_text_ce: 0.030522027984261513 (0.035254620637871284) | > loss_mel_ce: 2.426360845565796 (nan) | > loss: 0.009749536402523518 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1366133068613341) | > loader_time: 0.0046 (0.004649350227859023)  --> TIME: 2025-02-25 07:22:25 -- STEP: 17066/36938 -- GLOBAL_STEP: 681950 | > loss_text_ce: 0.03624042496085167 (0.035254553479319266) | > loss_mel_ce: 2.1815574169158936 (nan) | > loss: 0.008800785057246685 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.136618446538195) | > loader_time: 0.0045 (0.004649254776424781)  --> TIME: 2025-02-25 07:22:34 -- STEP: 17116/36938 -- GLOBAL_STEP: 682000 | > loss_text_ce: 0.040160976350307465 (0.03525771558195312) | > loss_mel_ce: 2.185698986053467 (nan) | > loss: 0.008832777850329876 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13660703250865402) | > loader_time: 0.0045 (0.004649222231435005)  --> TIME: 2025-02-25 07:22:43 -- STEP: 17166/36938 -- GLOBAL_STEP: 682050 | > loss_text_ce: 0.03577538579702377 (0.03525671701524362) | > loss_mel_ce: 2.8153634071350098 (nan) | > loss: 0.011314043775200844 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.13660292297369064) | > loader_time: 0.0048 (0.0046493749052970166)  --> TIME: 2025-02-25 07:22:53 -- STEP: 17216/36938 -- GLOBAL_STEP: 682100 | > loss_text_ce: 0.04777922481298447 (0.03525893304181735) | > loss_mel_ce: 2.512795925140381 (nan) | > loss: 0.010161013342440128 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13659959231056698) | > loader_time: 0.0046 (0.0046494021928664905)  --> TIME: 2025-02-25 07:23:02 -- STEP: 17266/36938 -- GLOBAL_STEP: 682150 | > loss_text_ce: 0.03363729268312454 (0.035259741385698574) | > loss_mel_ce: 3.213130474090576 (nan) | > loss: 0.01288399938493967 (nan) | > current_lr: 5e-06 | > step_time: 0.1789 (0.1366036813725579) | > loader_time: 0.0047 (0.004649586298092068)  --> TIME: 2025-02-25 07:23:11 -- STEP: 17316/36938 -- GLOBAL_STEP: 682200 | > loss_text_ce: 0.04155217111110687 (0.03525947249822794) | > loss_mel_ce: 2.69887113571167 (nan) | > loss: 0.01087469607591629 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13659304116058094) | > loader_time: 0.0045 (0.004649343777540016)  --> TIME: 2025-02-25 07:23:20 -- STEP: 17366/36938 -- GLOBAL_STEP: 682250 | > loss_text_ce: 0.033768124878406525 (0.03525968820222828) | > loss_mel_ce: 3.126325845718384 (nan) | > loss: 0.012540056370198727 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13660041264203737) | > loader_time: 0.0046 (0.00464928441232736)  --> TIME: 2025-02-25 07:23:30 -- STEP: 17416/36938 -- GLOBAL_STEP: 682300 | > loss_text_ce: 0.03818398714065552 (0.035258113565045386) | > loss_mel_ce: 2.398468017578125 (nan) | > loss: 0.009669254533946514 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13660385099966488) | > loader_time: 0.0045 (0.004649285047383938)  --> TIME: 2025-02-25 07:23:39 -- STEP: 17466/36938 -- GLOBAL_STEP: 682350 | > loss_text_ce: 0.030976196750998497 (0.03525813018026954) | > loss_mel_ce: 3.090132236480713 (nan) | > loss: 0.012385351583361626 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.1366007013390813) | > loader_time: 0.0046 (0.00464930316501776)  --> TIME: 2025-02-25 07:23:48 -- STEP: 17516/36938 -- GLOBAL_STEP: 682400 | > loss_text_ce: 0.02940543182194233 (0.03525911494178206) | > loss_mel_ce: 2.2896807193756104 (nan) | > loss: 0.009202723391354084 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13659819499845777) | > loader_time: 0.0045 (0.004649252863228661)  --> TIME: 2025-02-25 07:23:58 -- STEP: 17566/36938 -- GLOBAL_STEP: 682450 | > loss_text_ce: 0.03466382995247841 (0.03525959178501032) | > loss_mel_ce: 3.038909673690796 (nan) | > loss: 0.012196721509099007 (nan) | > current_lr: 5e-06 | > step_time: 0.1828 (0.13659832976503125) | > loader_time: 0.0048 (0.004649132541055819)  --> TIME: 2025-02-25 07:24:07 -- STEP: 17616/36938 -- GLOBAL_STEP: 682500 | > loss_text_ce: 0.03281813859939575 (0.03525790389449377) | > loss_mel_ce: 3.022348403930664 (nan) | > loss: 0.01212367694824934 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13661217232057962) | > loader_time: 0.0053 (0.004649198983713888)  --> TIME: 2025-02-25 07:24:16 -- STEP: 17666/36938 -- GLOBAL_STEP: 682550 | > loss_text_ce: 0.020451709628105164 (0.035253769104205315) | > loss_mel_ce: 1.0793787240982056 (nan) | > loss: 0.004364406690001488 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13659284535369884) | > loader_time: 0.0045 (0.004649053448042535)  --> TIME: 2025-02-25 07:24:26 -- STEP: 17716/36938 -- GLOBAL_STEP: 682600 | > loss_text_ce: 0.03369014337658882 (0.03525258166828852) | > loss_mel_ce: 3.277994394302368 (nan) | > loss: 0.013141606003046036 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.1365927153393762) | > loader_time: 0.0046 (0.004648870177235605)  --> TIME: 2025-02-25 07:24:35 -- STEP: 17766/36938 -- GLOBAL_STEP: 682650 | > loss_text_ce: 0.03587360307574272 (0.035252427397788486) | > loss_mel_ce: 2.937258243560791 (nan) | > loss: 0.01179814338684082 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13659203954340418) | > loader_time: 0.0046 (0.004648693681742897)  --> TIME: 2025-02-25 07:24:44 -- STEP: 17816/36938 -- GLOBAL_STEP: 682700 | > loss_text_ce: 0.0347520187497139 (0.035251662680086025) | > loss_mel_ce: 2.701975107192993 (nan) | > loss: 0.010860029608011246 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13660263502656667) | > loader_time: 0.0045 (0.004648608547398257)  --> TIME: 2025-02-25 07:24:54 -- STEP: 17866/36938 -- GLOBAL_STEP: 682750 | > loss_text_ce: 0.0378301665186882 (0.035250970452753355) | > loss_mel_ce: 2.4240329265594482 (nan) | > loss: 0.009769298136234283 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1366110969254352) | > loader_time: 0.0045 (0.00464853169629011)  --> TIME: 2025-02-25 07:25:04 -- STEP: 17916/36938 -- GLOBAL_STEP: 682800 | > loss_text_ce: 0.03683166578412056 (0.035252763358310645) | > loss_mel_ce: 1.7502551078796387 (nan) | > loss: 0.007091614417731762 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13662492777785412) | > loader_time: 0.0045 (0.0046484770852557485)  --> TIME: 2025-02-25 07:25:13 -- STEP: 17966/36938 -- GLOBAL_STEP: 682850 | > loss_text_ce: 0.035317350178956985 (0.035251959336870455) | > loss_mel_ce: 2.7235474586486816 (nan) | > loss: 0.010947877541184425 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.1366350093159126) | > loader_time: 0.0045 (0.0046483609241883385)  --> TIME: 2025-02-25 07:25:23 -- STEP: 18016/36938 -- GLOBAL_STEP: 682900 | > loss_text_ce: 0.04800955206155777 (0.03525400962160542) | > loss_mel_ce: 3.4888856410980225 (nan) | > loss: 0.01403529942035675 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13664800742538508) | > loader_time: 0.0046 (0.0046482710812909035)  --> TIME: 2025-02-25 07:25:32 -- STEP: 18066/36938 -- GLOBAL_STEP: 682950 | > loss_text_ce: 0.036591969430446625 (0.03525675587207837) | > loss_mel_ce: 2.8724799156188965 (nan) | > loss: 0.011543937027454376 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13663468996472575) | > loader_time: 0.0046 (0.00464805163881066)  --> TIME: 2025-02-25 07:25:41 -- STEP: 18116/36938 -- GLOBAL_STEP: 683000 | > loss_text_ce: 0.04725136235356331 (0.035255735841135415) | > loss_mel_ce: 3.328277587890625 (nan) | > loss: 0.013394957408308983 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.13662898370724244) | > loader_time: 0.0046 (0.004647830775516335)  --> TIME: 2025-02-25 07:25:50 -- STEP: 18166/36938 -- GLOBAL_STEP: 683050 | > loss_text_ce: 0.03402181714773178 (0.035257572040259616) | > loss_mel_ce: 2.426098585128784 (nan) | > loss: 0.009762383066117764 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13661788193176547) | > loader_time: 0.0045 (0.004647803821057629)  --> TIME: 2025-02-25 07:25:59 -- STEP: 18216/36938 -- GLOBAL_STEP: 683100 | > loss_text_ce: 0.020181605592370033 (0.03525582922867564) | > loss_mel_ce: 2.814117431640625 (nan) | > loss: 0.011247219517827034 (nan) | > current_lr: 5e-06 | > step_time: 0.1265 (0.1366073431469233) | > loader_time: 0.0045 (0.004647593763663035)  --> TIME: 2025-02-25 07:26:08 -- STEP: 18266/36938 -- GLOBAL_STEP: 683150 | > loss_text_ce: 0.0359317921102047 (0.035249685910328024) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13658895970800525) | > loader_time: 0.0045 (0.004647371673145402)  --> TIME: 2025-02-25 07:26:17 -- STEP: 18316/36938 -- GLOBAL_STEP: 683200 | > loss_text_ce: 0.03244612738490105 (0.035246165854000984) | > loss_mel_ce: 2.641724109649658 (nan) | > loss: 0.01061178743839264 (nan) | > current_lr: 5e-06 | > step_time: 0.1355 (0.13657542019854654) | > loader_time: 0.0045 (0.004647211961856583)  --> TIME: 2025-02-25 07:26:27 -- STEP: 18366/36938 -- GLOBAL_STEP: 683250 | > loss_text_ce: 0.03544418886303902 (0.03524039070817983) | > loss_mel_ce: 3.101168632507324 (nan) | > loss: 0.012446877546608448 (nan) | > current_lr: 5e-06 | > step_time: 0.1775 (0.1365813517848298) | > loader_time: 0.0048 (0.004647098620395091)  --> TIME: 2025-02-25 07:26:36 -- STEP: 18416/36938 -- GLOBAL_STEP: 683300 | > loss_text_ce: 0.04406992346048355 (0.03524275952361475) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.1365699899988935) | > loader_time: 0.0044 (0.004646866853396637)  --> TIME: 2025-02-25 07:26:45 -- STEP: 18466/36938 -- GLOBAL_STEP: 683350 | > loss_text_ce: 0.03696554899215698 (0.03524576183541652) | > loss_mel_ce: 2.9661543369293213 (nan) | > loss: 0.011917143128812313 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.13656606400992904) | > loader_time: 0.0046 (0.004646817434264007)  --> TIME: 2025-02-25 07:26:54 -- STEP: 18516/36938 -- GLOBAL_STEP: 683400 | > loss_text_ce: 0.02052488550543785 (0.03524291933721212) | > loss_mel_ce: 2.516045570373535 (nan) | > loss: 0.010065756738185883 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.1365721193805164) | > loader_time: 0.0045 (0.004646922746779039)  --> TIME: 2025-02-25 07:27:04 -- STEP: 18566/36938 -- GLOBAL_STEP: 683450 | > loss_text_ce: 0.03362320736050606 (0.03524255575669381) | > loss_mel_ce: 3.3589465618133545 (nan) | > loss: 0.013462578877806664 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13657643432107616) | > loader_time: 0.0046 (0.004647001667448005)  --> TIME: 2025-02-25 07:27:13 -- STEP: 18616/36938 -- GLOBAL_STEP: 683500 | > loss_text_ce: 0.03583066537976265 (0.03524140892244183) | > loss_mel_ce: 3.1686952114105225 (nan) | > loss: 0.012716373428702354 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.1365730777594168) | > loader_time: 0.0057 (0.004647086772685083)  --> TIME: 2025-02-25 07:27:22 -- STEP: 18666/36938 -- GLOBAL_STEP: 683550 | > loss_text_ce: 0.040090736001729965 (0.03524377096610422) | > loss_mel_ce: 1.4853661060333252 (nan) | > loss: 0.0060534002259373665 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.1365688764102509) | > loader_time: 0.0044 (0.004647121927074752)  --> TIME: 2025-02-25 07:27:31 -- STEP: 18716/36938 -- GLOBAL_STEP: 683600 | > loss_text_ce: 0.030898001044988632 (0.035240859942334425) | > loss_mel_ce: 1.5354772806167603 (nan) | > loss: 0.0062157753854990005 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13656970047752007) | > loader_time: 0.0045 (0.004647121428933793)  --> TIME: 2025-02-25 07:27:40 -- STEP: 18766/36938 -- GLOBAL_STEP: 683650 | > loss_text_ce: 0.02162454091012478 (0.03524093149081204) | > loss_mel_ce: 1.9213099479675293 (nan) | > loss: 0.007710058242082596 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13655582308248312) | > loader_time: 0.0044 (0.004646968361310669)  --> TIME: 2025-02-25 07:27:50 -- STEP: 18816/36938 -- GLOBAL_STEP: 683700 | > loss_text_ce: 0.026130899786949158 (0.035238691425164045) | > loss_mel_ce: 2.184988260269165 (nan) | > loss: 0.008774283342063427 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13655371929765947) | > loader_time: 0.0046 (0.004647113205421524)  --> TIME: 2025-02-25 07:27:59 -- STEP: 18866/36938 -- GLOBAL_STEP: 683750 | > loss_text_ce: 0.03140222653746605 (0.035242167750600045) | > loss_mel_ce: 1.3595889806747437 (nan) | > loss: 0.005519806873053312 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13653771864398576) | > loader_time: 0.0045 (0.004647007198812936)  --> TIME: 2025-02-25 07:28:08 -- STEP: 18916/36938 -- GLOBAL_STEP: 683800 | > loss_text_ce: 0.028661226853728294 (0.035243695344323825) | > loss_mel_ce: 2.472517967224121 (nan) | > loss: 0.009925315156579018 (nan) | > current_lr: 5e-06 | > step_time: 0.1508 (0.13653661524626806) | > loader_time: 0.0047 (0.004646999245089221)  --> TIME: 2025-02-25 07:28:17 -- STEP: 18966/36938 -- GLOBAL_STEP: 683850 | > loss_text_ce: 0.02571929804980755 (0.03524096738627624) | > loss_mel_ce: 2.9776809215545654 (nan) | > loss: 0.011918256059288979 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13653728682369032) | > loader_time: 0.0045 (0.004646872413142058)  --> TIME: 2025-02-25 07:28:26 -- STEP: 19016/36938 -- GLOBAL_STEP: 683900 | > loss_text_ce: 0.04394775256514549 (0.035243967093811814) | > loss_mel_ce: 2.65592885017395 (nan) | > loss: 0.010713796131312847 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13653520913148123) | > loader_time: 0.0046 (0.00464680028603664)  --> TIME: 2025-02-25 07:28:36 -- STEP: 19066/36938 -- GLOBAL_STEP: 683950 | > loss_text_ce: 0.02565709315240383 (0.03523980988097554) | > loss_mel_ce: 2.90327787399292 (nan) | > loss: 0.011622758582234383 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13654620210008697) | > loader_time: 0.0046 (0.004646824162266211)  --> TIME: 2025-02-25 07:28:45 -- STEP: 19116/36938 -- GLOBAL_STEP: 684000 | > loss_text_ce: 0.02649317868053913 (0.03523762024879434) | > loss_mel_ce: 2.9556539058685303 (nan) | > loss: 0.01183391734957695 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13654974944223805) | > loader_time: 0.0046 (0.004646806742861103)  --> TIME: 2025-02-25 07:28:54 -- STEP: 19166/36938 -- GLOBAL_STEP: 684050 | > loss_text_ce: 0.04197539761662483 (0.03524014316841328) | > loss_mel_ce: 1.9957411289215088 (nan) | > loss: 0.00808617752045393 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13654023731698986) | > loader_time: 0.0045 (0.004646607670868074)  --> TIME: 2025-02-25 07:29:03 -- STEP: 19216/36938 -- GLOBAL_STEP: 684100 | > loss_text_ce: 0.026736686006188393 (0.03523566052470774) | > loss_mel_ce: 3.046696662902832 (nan) | > loss: 0.012196164578199387 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13652228914530024) | > loader_time: 0.0046 (0.004646378529756676)  --> TIME: 2025-02-25 07:29:12 -- STEP: 19266/36938 -- GLOBAL_STEP: 684150 | > loss_text_ce: 0.0336364321410656 (0.03523620109582456) | > loss_mel_ce: 2.67982816696167 (nan) | > loss: 0.01076771691441536 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.1365059504538174) | > loader_time: 0.0046 (0.004646188185913679)  --> TIME: 2025-02-25 07:29:21 -- STEP: 19316/36938 -- GLOBAL_STEP: 684200 | > loss_text_ce: 0.04326494783163071 (0.03523779535682697) | > loss_mel_ce: 2.9792439937591553 (nan) | > loss: 0.011994083411991596 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1364925165037158) | > loader_time: 0.0046 (0.004646040201433978)  --> TIME: 2025-02-25 07:29:30 -- STEP: 19366/36938 -- GLOBAL_STEP: 684250 | > loss_text_ce: 0.039841603487730026 (0.0352391108022573) | > loss_mel_ce: 2.7018356323242188 (nan) | > loss: 0.01087967213243246 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13647574992092673) | > loader_time: 0.0046 (0.0046459346298687206)  --> TIME: 2025-02-25 07:29:40 -- STEP: 19416/36938 -- GLOBAL_STEP: 684300 | > loss_text_ce: 0.04275564104318619 (0.035235280205860614) | > loss_mel_ce: 2.6787047386169434 (nan) | > loss: 0.010799446143209934 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13647401384429042) | > loader_time: 0.0048 (0.004645788379790202)  --> TIME: 2025-02-25 07:29:49 -- STEP: 19466/36938 -- GLOBAL_STEP: 684350 | > loss_text_ce: 0.037446528673172 (0.03523203061194495) | > loss_mel_ce: 2.8957228660583496 (nan) | > loss: 0.011639561504125595 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13646604946702387) | > loader_time: 0.0046 (0.004645568597210172)  --> TIME: 2025-02-25 07:29:58 -- STEP: 19516/36938 -- GLOBAL_STEP: 684400 | > loss_text_ce: 0.03566742315888405 (0.03523547501797837) | > loss_mel_ce: 1.1522235870361328 (nan) | > loss: 0.004713853355497122 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13646143205734135) | > loader_time: 0.0045 (0.004645388093144209)  --> TIME: 2025-02-25 07:30:07 -- STEP: 19566/36938 -- GLOBAL_STEP: 684450 | > loss_text_ce: 0.03245358541607857 (0.03523829105238459) | > loss_mel_ce: 2.81748366355896 (nan) | > loss: 0.011309275403618813 (nan) | > current_lr: 5e-06 | > step_time: 0.1954 (0.13645465020861405) | > loader_time: 0.0048 (0.004645354748015914)  --> TIME: 2025-02-25 07:30:16 -- STEP: 19616/36938 -- GLOBAL_STEP: 684500 | > loss_text_ce: 0.03289906680583954 (0.0352397198809902) | > loss_mel_ce: 2.7521350383758545 (nan) | > loss: 0.01105172373354435 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13645064372563212) | > loader_time: 0.0055 (0.004645458928523591)  --> TIME: 2025-02-25 07:30:26 -- STEP: 19666/36938 -- GLOBAL_STEP: 684550 | > loss_text_ce: 0.028647832572460175 (0.03524223085834579) | > loss_mel_ce: 2.621222734451294 (nan) | > loss: 0.010515360161662102 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13645850282155203) | > loader_time: 0.0045 (0.004645592366450068)  --> TIME: 2025-02-25 07:30:34 -- STEP: 19716/36938 -- GLOBAL_STEP: 684600 | > loss_text_ce: 0.032275907695293427 (0.03523876947186608) | > loss_mel_ce: 2.9830925464630127 (nan) | > loss: 0.011965748853981495 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13644001443527784) | > loader_time: 0.0046 (0.004645381527175327)  --> TIME: 2025-02-25 07:30:44 -- STEP: 19766/36938 -- GLOBAL_STEP: 684650 | > loss_text_ce: 0.03209828585386276 (0.0352408339110297) | > loss_mel_ce: 2.7380483150482178 (nan) | > loss: 0.010992646217346191 (nan) | > current_lr: 5e-06 | > step_time: 0.1438 (0.13644154057574626) | > loader_time: 0.0046 (0.0046453041235686235)  --> TIME: 2025-02-25 07:30:53 -- STEP: 19816/36938 -- GLOBAL_STEP: 684700 | > loss_text_ce: 0.04651714861392975 (0.03523982396012055) | > loss_mel_ce: 2.2282369136810303 (nan) | > loss: 0.009026802144944668 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13644030888714903) | > loader_time: 0.0046 (0.004645279905321518)  --> TIME: 2025-02-25 07:31:03 -- STEP: 19866/36938 -- GLOBAL_STEP: 684750 | > loss_text_ce: 0.035674311220645905 (0.0352348830475151) | > loss_mel_ce: 2.9929962158203125 (nan) | > loss: 0.012018534354865551 (nan) | > current_lr: 5e-06 | > step_time: 0.1579 (0.1364524912572538) | > loader_time: 0.0046 (0.00464524856017433)  --> TIME: 2025-02-25 07:31:12 -- STEP: 19916/36938 -- GLOBAL_STEP: 684800 | > loss_text_ce: 0.0442415215075016 (0.03523565459740957) | > loss_mel_ce: 0.5497233271598816 (nan) | > loss: 0.0023570035118609667 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13644579043946695) | > loader_time: 0.0045 (0.004645196662224014)  --> TIME: 2025-02-25 07:31:21 -- STEP: 19966/36938 -- GLOBAL_STEP: 684850 | > loss_text_ce: 0.031523656100034714 (0.035234416761310325) | > loss_mel_ce: 2.4407272338867188 (nan) | > loss: 0.009810520336031914 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.1364562020062515) | > loader_time: 0.0044 (0.004645134874160648)  --> TIME: 2025-02-25 07:31:31 -- STEP: 20016/36938 -- GLOBAL_STEP: 684900 | > loss_text_ce: 0.04104430601000786 (0.035235870020951) | > loss_mel_ce: 2.6034607887268066 (nan) | > loss: 0.010494068264961243 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.1364522968574486) | > loader_time: 0.0047 (0.004645058374515447)  --> TIME: 2025-02-25 07:31:40 -- STEP: 20066/36938 -- GLOBAL_STEP: 684950 | > loss_text_ce: 0.03228234499692917 (0.035232093310733494) | > loss_mel_ce: 3.2144570350646973 (nan) | > loss: 0.012883887626230717 (nan) | > current_lr: 5e-06 | > step_time: 0.1797 (0.13645112723614086) | > loader_time: 0.0048 (0.004645102546636386)  --> TIME: 2025-02-25 07:31:49 -- STEP: 20116/36938 -- GLOBAL_STEP: 685000 | > loss_text_ce: 0.02786373533308506 (0.03522680241844775) | > loss_mel_ce: 2.9527227878570557 (nan) | > loss: 0.011827724985778332 (nan) | > current_lr: 5e-06 | > step_time: 0.1709 (0.13645804923958874) | > loader_time: 0.0047 (0.004645002767707366)  --> TIME: 2025-02-25 07:31:59 -- STEP: 20166/36938 -- GLOBAL_STEP: 685050 | > loss_text_ce: 0.03661080822348595 (0.03522407842271139) | > loss_mel_ce: 2.733934164047241 (nan) | > loss: 0.010994226671755314 (nan) | > current_lr: 5e-06 | > step_time: 0.1691 (0.13645409046864668) | > loader_time: 0.0048 (0.004644858308648535)  --> TIME: 2025-02-25 07:32:08 -- STEP: 20216/36938 -- GLOBAL_STEP: 685100 | > loss_text_ce: 0.03325299173593521 (0.03522190971939076) | > loss_mel_ce: 2.9396047592163086 (nan) | > loss: 0.011797054670751095 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13644241136810173) | > loader_time: 0.0045 (0.00464475353008492)  --> TIME: 2025-02-25 07:32:17 -- STEP: 20266/36938 -- GLOBAL_STEP: 685150 | > loss_text_ce: 0.03506339341402054 (0.03522026573020966) | > loss_mel_ce: 2.9241089820861816 (nan) | > loss: 0.011742748320102692 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13643949665699032) | > loader_time: 0.0046 (0.004644625986668265)  --> TIME: 2025-02-25 07:32:26 -- STEP: 20316/36938 -- GLOBAL_STEP: 685200 | > loss_text_ce: 0.043236393481492996 (0.035218556745201184) | > loss_mel_ce: 2.995090961456299 (nan) | > loss: 0.012056855484843254 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13643318561381967) | > loader_time: 0.0045 (0.004644577675481599)  --> TIME: 2025-02-25 07:32:35 -- STEP: 20366/36938 -- GLOBAL_STEP: 685250 | > loss_text_ce: 0.03814307600259781 (0.03521886857977001) | > loss_mel_ce: 3.2104344367980957 (nan) | > loss: 0.012891181744635105 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13642924207490528) | > loader_time: 0.0046 (0.004644777350325979)  --> TIME: 2025-02-25 07:32:45 -- STEP: 20416/36938 -- GLOBAL_STEP: 685300 | > loss_text_ce: 0.037197910249233246 (0.0352171306802342) | > loss_mel_ce: 3.182368040084839 (nan) | > loss: 0.012776055373251438 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.136429009170835) | > loader_time: 0.0045 (0.004644785088057803)  --> TIME: 2025-02-25 07:32:54 -- STEP: 20466/36938 -- GLOBAL_STEP: 685350 | > loss_text_ce: 0.03687823936343193 (0.03521686081854531) | > loss_mel_ce: 2.950932264328003 (nan) | > loss: 0.01185639202594757 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.1364309724260493) | > loader_time: 0.0046 (0.004644755963925964)  --> TIME: 2025-02-25 07:33:03 -- STEP: 20516/36938 -- GLOBAL_STEP: 685400 | > loss_text_ce: 0.03220172971487045 (0.035213716037888795) | > loss_mel_ce: 2.626058578491211 (nan) | > loss: 0.010548653081059456 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13643036479721257) | > loader_time: 0.0045 (0.004644677162077438)  --> TIME: 2025-02-25 07:33:13 -- STEP: 20566/36938 -- GLOBAL_STEP: 685450 | > loss_text_ce: 0.04380283132195473 (0.03521009362669976) | > loss_mel_ce: 3.2166330814361572 (nan) | > loss: 0.012938237749040127 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.13643545433436932) | > loader_time: 0.0046 (0.004644562701219989)  --> TIME: 2025-02-25 07:33:22 -- STEP: 20616/36938 -- GLOBAL_STEP: 685500 | > loss_text_ce: 0.02691624127328396 (0.035210080597503385) | > loss_mel_ce: 1.79529869556427 (nan) | > loss: 0.007231011986732483 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.1364448698375593) | > loader_time: 0.0045 (0.004644540677387019)  --> TIME: 2025-02-25 07:33:32 -- STEP: 20666/36938 -- GLOBAL_STEP: 685550 | > loss_text_ce: 0.0394265241920948 (0.0352142236279236) | > loss_mel_ce: 0.30369871854782104 (nan) | > loss: 0.0013616082724183798 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.1364384546038097) | > loader_time: 0.0046 (0.0046445004282207126)  --> TIME: 2025-02-25 07:33:41 -- STEP: 20716/36938 -- GLOBAL_STEP: 685600 | > loss_text_ce: 0.042145658284425735 (0.03521139707085774) | > loss_mel_ce: 2.9878389835357666 (nan) | > loss: 0.012023749761283398 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13643253451494852) | > loader_time: 0.0046 (0.004644457979491372)  --> TIME: 2025-02-25 07:33:50 -- STEP: 20766/36938 -- GLOBAL_STEP: 685650 | > loss_text_ce: 0.027637336403131485 (0.035209095093060964) | > loss_mel_ce: 2.1761715412139893 (nan) | > loss: 0.00874527357518673 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13643345407561105) | > loader_time: 0.0045 (0.0046444931414204695)  --> TIME: 2025-02-25 07:33:59 -- STEP: 20816/36938 -- GLOBAL_STEP: 685700 | > loss_text_ce: 0.043548859655857086 (0.03520633755611484) | > loss_mel_ce: 3.3647913932800293 (nan) | > loss: 0.013525160029530525 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.1364311045128781) | > loader_time: 0.0047 (0.004644549151826687)  --> TIME: 2025-02-25 07:34:09 -- STEP: 20866/36938 -- GLOBAL_STEP: 685750 | > loss_text_ce: 0.03960292413830757 (0.03520739267250017) | > loss_mel_ce: 2.827044725418091 (nan) | > loss: 0.01137558650225401 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.13644580295807346) | > loader_time: 0.0048 (0.004644696051832419)  --> TIME: 2025-02-25 07:34:19 -- STEP: 20916/36938 -- GLOBAL_STEP: 685800 | > loss_text_ce: 0.04032549262046814 (0.035211094158711785) | > loss_mel_ce: 3.704393148422241 (nan) | > loss: 0.014859994873404503 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13645422323002215) | > loader_time: 0.0046 (0.00464469300421639)  --> TIME: 2025-02-25 07:34:28 -- STEP: 20966/36938 -- GLOBAL_STEP: 685850 | > loss_text_ce: 0.03585882484912872 (0.03520864085074428) | > loss_mel_ce: 2.3552775382995605 (nan) | > loss: 0.009488636627793312 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.1364520900544916) | > loader_time: 0.0046 (0.004644621422667652)  --> TIME: 2025-02-25 07:34:38 -- STEP: 21016/36938 -- GLOBAL_STEP: 685900 | > loss_text_ce: 0.034167494624853134 (0.035208390897326414) | > loss_mel_ce: 2.670377492904663 (nan) | > loss: 0.010732321999967098 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13647356656598716) | > loader_time: 0.0048 (0.004644738944891873)  --> TIME: 2025-02-25 07:34:47 -- STEP: 21066/36938 -- GLOBAL_STEP: 685950 | > loss_text_ce: 0.04018949717283249 (0.0352097868821758) | > loss_mel_ce: 3.0195152759552 (nan) | > loss: 0.01214168593287468 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.13647653878182173) | > loader_time: 0.0047 (0.004644715196328266)  --> TIME: 2025-02-25 07:34:56 -- STEP: 21116/36938 -- GLOBAL_STEP: 686000 | > loss_text_ce: 0.02583405002951622 (0.03520776632397687) | > loss_mel_ce: 2.6784939765930176 (nan) | > loss: 0.010731461457908154 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13647984439935118) | > loader_time: 0.0045 (0.004644709975684667)  --> TIME: 2025-02-25 07:35:06 -- STEP: 21166/36938 -- GLOBAL_STEP: 686050 | > loss_text_ce: 0.028773225843906403 (0.03520409372360357) | > loss_mel_ce: 5.546875 (nan) | > loss: 0.022125590592622757 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13648158311280645) | > loader_time: 0.0046 (0.004644630514673137)  --> TIME: 2025-02-25 07:35:15 -- STEP: 21216/36938 -- GLOBAL_STEP: 686100 | > loss_text_ce: 0.03640802204608917 (0.0352025334126292) | > loss_mel_ce: 2.8204939365386963 (nan) | > loss: 0.011336913332343102 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13647681117327493) | > loader_time: 0.0047 (0.004644497296299504)  --> TIME: 2025-02-25 07:35:24 -- STEP: 21266/36938 -- GLOBAL_STEP: 686150 | > loss_text_ce: 0.04037231206893921 (0.03520276029159367) | > loss_mel_ce: 2.247708320617676 (nan) | > loss: 0.009079685434699059 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13647836317272066) | > loader_time: 0.0044 (0.004644418170846077)  --> TIME: 2025-02-25 07:35:33 -- STEP: 21316/36938 -- GLOBAL_STEP: 686200 | > loss_text_ce: 0.02976398728787899 (0.035203773264704516) | > loss_mel_ce: 2.3827664852142334 (nan) | > loss: 0.00957353413105011 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1364730599226748) | > loader_time: 0.0045 (0.004644341206188199)  --> TIME: 2025-02-25 07:35:42 -- STEP: 21366/36938 -- GLOBAL_STEP: 686250 | > loss_text_ce: 0.044286418706178665 (0.035205793647292774) | > loss_mel_ce: 1.2317731380462646 (nan) | > loss: 0.005063728429377079 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13646217445013595) | > loader_time: 0.0045 (0.004644317059194185)  --> TIME: 2025-02-25 07:35:52 -- STEP: 21416/36938 -- GLOBAL_STEP: 686300 | > loss_text_ce: 0.03467750549316406 (0.035204811909433975) | > loss_mel_ce: 2.875432014465332 (nan) | > loss: 0.011548054404556751 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13646694296413636) | > loader_time: 0.0045 (0.004644413158258569)  --> TIME: 2025-02-25 07:36:01 -- STEP: 21466/36938 -- GLOBAL_STEP: 686350 | > loss_text_ce: 0.043118566274642944 (0.03520116995043722) | > loss_mel_ce: 2.9082319736480713 (nan) | > loss: 0.011711709201335907 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13647220547463812) | > loader_time: 0.0046 (0.004644509587118629)  --> TIME: 2025-02-25 07:36:11 -- STEP: 21516/36938 -- GLOBAL_STEP: 686400 | > loss_text_ce: 0.03485774248838425 (0.03519934808586411) | > loss_mel_ce: 3.1996095180511475 (nan) | > loss: 0.012835188768804073 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.1364709321041714) | > loader_time: 0.0046 (0.004644415484504812)  --> TIME: 2025-02-25 07:36:20 -- STEP: 21566/36938 -- GLOBAL_STEP: 686450 | > loss_text_ce: 0.03000631183385849 (0.03519842768726944) | > loss_mel_ce: 3.2994987964630127 (nan) | > loss: 0.013212323188781738 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.13647116186843677) | > loader_time: 0.0048 (0.004644257951775535)  --> TIME: 2025-02-25 07:36:29 -- STEP: 21616/36938 -- GLOBAL_STEP: 686500 | > loss_text_ce: 0.026483900845050812 (0.03519796997025749) | > loss_mel_ce: 1.3176676034927368 (nan) | > loss: 0.005333934910595417 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13647246442398786) | > loader_time: 0.0045 (0.004644414623696031)  --> TIME: 2025-02-25 07:36:39 -- STEP: 21666/36938 -- GLOBAL_STEP: 686550 | > loss_text_ce: 0.029527155682444572 (0.03519732482936979) | > loss_mel_ce: 1.0555444955825806 (nan) | > loss: 0.004305840469896793 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13647474840298535) | > loader_time: 0.0046 (0.0046444175250640405)  --> TIME: 2025-02-25 07:36:48 -- STEP: 21716/36938 -- GLOBAL_STEP: 686600 | > loss_text_ce: 0.02933095209300518 (0.03519540143021365) | > loss_mel_ce: 2.787329912185669 (nan) | > loss: 0.011177225969731808 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.1364845587944191) | > loader_time: 0.0045 (0.004644355253085035)  --> TIME: 2025-02-25 07:36:57 -- STEP: 21766/36938 -- GLOBAL_STEP: 686650 | > loss_text_ce: 0.03648789972066879 (0.03519373073745017) | > loss_mel_ce: 0.006805419921875 (nan) | > loss: 0.00017179889255203307 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13648039314508728) | > loader_time: 0.0045 (0.004644166368405444)  --> TIME: 2025-02-25 07:37:06 -- STEP: 21816/36938 -- GLOBAL_STEP: 686700 | > loss_text_ce: 0.0397438146173954 (0.035191556940367116) | > loss_mel_ce: 0.02813720703125 (nan) | > loss: 0.00026936913491226733 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13646134010679706) | > loader_time: 0.0045 (0.0046441492620824)  --> TIME: 2025-02-25 07:37:16 -- STEP: 21866/36938 -- GLOBAL_STEP: 686750 | > loss_text_ce: 0.023145334795117378 (0.03518735857078694) | > loss_mel_ce: 1.604075312614441 (nan) | > loss: 0.006457224953919649 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13647012659106203) | > loader_time: 0.0066 (0.004644418846576271)  --> TIME: 2025-02-25 07:37:25 -- STEP: 21916/36938 -- GLOBAL_STEP: 686800 | > loss_text_ce: 0.034773945808410645 (0.03519146909246632) | > loss_mel_ce: 2.7396016120910645 (nan) | > loss: 0.011009426787495613 (nan) | > current_lr: 5e-06 | > step_time: 0.1782 (0.1364668099674393) | > loader_time: 0.0048 (0.0046443260810435545)  --> TIME: 2025-02-25 07:37:34 -- STEP: 21966/36938 -- GLOBAL_STEP: 686850 | > loss_text_ce: 0.034518759697675705 (0.03518785290857775) | > loss_mel_ce: 3.0298843383789062 (nan) | > loss: 0.012160330079495907 (nan) | > current_lr: 5e-06 | > step_time: 0.1979 (0.13647214506511032) | > loader_time: 0.0048 (0.0046443090101935)  --> TIME: 2025-02-25 07:37:44 -- STEP: 22016/36938 -- GLOBAL_STEP: 686900 | > loss_text_ce: 0.03661840781569481 (0.0351867817952193) | > loss_mel_ce: 1.5618866682052612 (nan) | > loss: 0.006343274377286434 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.13647327664156633) | > loader_time: 0.0049 (0.004644293013180394)  --> TIME: 2025-02-25 07:37:53 -- STEP: 22066/36938 -- GLOBAL_STEP: 686950 | > loss_text_ce: 0.038455475121736526 (0.03518433698975439) | > loss_mel_ce: 2.6753227710723877 (nan) | > loss: 0.01076896209269762 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13646961188776624) | > loader_time: 0.0046 (0.0046442422972275904)  --> TIME: 2025-02-25 07:38:02 -- STEP: 22116/36938 -- GLOBAL_STEP: 687000 | > loss_text_ce: 0.048597969114780426 (0.03518158033078171) | > loss_mel_ce: 3.244297742843628 (nan) | > loss: 0.013067047111690044 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.1364615142162161) | > loader_time: 0.0046 (0.00464417259998013)  --> TIME: 2025-02-25 07:38:12 -- STEP: 22166/36938 -- GLOBAL_STEP: 687050 | > loss_text_ce: 0.033360596746206284 (0.03517947739265312) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13647192463293542) | > loader_time: 0.0045 (0.004644342377912274)  --> TIME: 2025-02-25 07:38:21 -- STEP: 22216/36938 -- GLOBAL_STEP: 687100 | > loss_text_ce: 0.0342036634683609 (0.03517564521317043) | > loss_mel_ce: 1.4649314880371094 (nan) | > loss: 0.005948949139565229 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13646096817486375) | > loader_time: 0.0045 (0.004644178672363991)  --> TIME: 2025-02-25 07:38:30 -- STEP: 22266/36938 -- GLOBAL_STEP: 687150 | > loss_text_ce: 0.05639907345175743 (0.035174053868712724) | > loss_mel_ce: 3.4898293018341064 (nan) | > loss: 0.014072335325181484 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13645287523969485) | > loader_time: 0.0045 (0.00464403236328853)  --> TIME: 2025-02-25 07:38:39 -- STEP: 22316/36938 -- GLOBAL_STEP: 687200 | > loss_text_ce: 0.029082490131258965 (0.035172963794496165) | > loss_mel_ce: 2.519148349761963 (nan) | > loss: 0.010112027637660503 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.13646309668284004) | > loader_time: 0.0048 (0.0046440542203633325)  --> TIME: 2025-02-25 07:38:49 -- STEP: 22366/36938 -- GLOBAL_STEP: 687250 | > loss_text_ce: 0.031117672100663185 (0.03517194916893524) | > loss_mel_ce: 1.8074209690093994 (nan) | > loss: 0.007295788731426001 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13646808975787322) | > loader_time: 0.0045 (0.004643980147511745)  --> TIME: 2025-02-25 07:38:58 -- STEP: 22416/36938 -- GLOBAL_STEP: 687300 | > loss_text_ce: 0.029813343659043312 (0.035172543546401334) | > loss_mel_ce: 2.807570457458496 (nan) | > loss: 0.011259459890425205 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13646865169717756) | > loader_time: 0.0045 (0.004643998545548624)  --> TIME: 2025-02-25 07:39:07 -- STEP: 22466/36938 -- GLOBAL_STEP: 687350 | > loss_text_ce: 0.028259895741939545 (0.03517317937477782) | > loss_mel_ce: 3.2451984882354736 (nan) | > loss: 0.01298991497606039 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.13647053295428538) | > loader_time: 0.0046 (0.004643997748728181)  --> TIME: 2025-02-25 07:39:17 -- STEP: 22516/36938 -- GLOBAL_STEP: 687400 | > loss_text_ce: 0.04159083589911461 (0.035174968919092636) | > loss_mel_ce: 3.036231279373169 (nan) | > loss: 0.01221358124166727 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13647322047771104) | > loader_time: 0.0046 (0.004643917549649673)  --> TIME: 2025-02-25 07:39:26 -- STEP: 22566/36938 -- GLOBAL_STEP: 687450 | > loss_text_ce: 0.047379277646541595 (0.03517410883083713) | > loss_mel_ce: 3.345663547515869 (nan) | > loss: 0.013464456424117088 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.13647013424580268) | > loader_time: 0.0046 (0.0046438333424631585)  --> TIME: 2025-02-25 07:39:35 -- STEP: 22616/36938 -- GLOBAL_STEP: 687500 | > loss_text_ce: 0.029757799580693245 (0.03517117700008321) | > loss_mel_ce: 2.892763614654541 (nan) | > loss: 0.01159730739891529 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13646409483735158) | > loader_time: 0.0046 (0.00464371224153069)  --> TIME: 2025-02-25 07:39:44 -- STEP: 22666/36938 -- GLOBAL_STEP: 687550 | > loss_text_ce: 0.03385885804891586 (0.03516995567706149) | > loss_mel_ce: 3.503530263900757 (nan) | > loss: 0.014037258923053741 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13644906969195772) | > loader_time: 0.0045 (0.0046435707845934185)  --> TIME: 2025-02-25 07:39:53 -- STEP: 22716/36938 -- GLOBAL_STEP: 687600 | > loss_text_ce: 0.043227419257164 (0.035169735465275624) | > loss_mel_ce: 2.719649076461792 (nan) | > loss: 0.010963796637952328 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13644915469663776) | > loader_time: 0.0046 (0.004643707486142714)  --> TIME: 2025-02-25 07:40:02 -- STEP: 22766/36938 -- GLOBAL_STEP: 687650 | > loss_text_ce: 0.026655253022909164 (0.035175190842119425) | > loss_mel_ce: 2.905785322189331 (nan) | > loss: 0.011636669747531414 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.13644807799298267) | > loader_time: 0.0045 (0.004643603702489172)  --> TIME: 2025-02-25 07:40:12 -- STEP: 22816/36938 -- GLOBAL_STEP: 687700 | > loss_text_ce: 0.027183357626199722 (0.03517299878786862) | > loss_mel_ce: 2.9255640506744385 (nan) | > loss: 0.011717252433300018 (nan) | > current_lr: 5e-06 | > step_time: 0.161 (0.13644779532060972) | > loader_time: 0.0054 (0.004643685770904171)  --> TIME: 2025-02-25 07:40:21 -- STEP: 22866/36938 -- GLOBAL_STEP: 687750 | > loss_text_ce: 0.030886666849255562 (0.035173906529494835) | > loss_mel_ce: 2.1140196323394775 (nan) | > loss: 0.00851153302937746 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1364365577656039) | > loader_time: 0.0045 (0.004643746918813638)  --> TIME: 2025-02-25 07:40:30 -- STEP: 22916/36938 -- GLOBAL_STEP: 687800 | > loss_text_ce: 0.030184870585799217 (0.035174985346250284) | > loss_mel_ce: 2.563555955886841 (nan) | > loss: 0.010292623192071915 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.13644204123687195) | > loader_time: 0.0046 (0.004643634760186249)  --> TIME: 2025-02-25 07:40:40 -- STEP: 22966/36938 -- GLOBAL_STEP: 687850 | > loss_text_ce: 0.03925010934472084 (0.035173723900348945) | > loss_mel_ce: 2.270355701446533 (nan) | > loss: 0.009165102615952492 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13643920042680202) | > loader_time: 0.0045 (0.004643706694864301)  --> TIME: 2025-02-25 07:40:49 -- STEP: 23016/36938 -- GLOBAL_STEP: 687900 | > loss_text_ce: 0.027711791917681694 (0.03516805349045715) | > loss_mel_ce: 2.9263641834259033 (nan) | > loss: 0.011722524650394917 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13642807607854532) | > loader_time: 0.0053 (0.004643794901469591)  --> TIME: 2025-02-25 07:40:58 -- STEP: 23066/36938 -- GLOBAL_STEP: 687950 | > loss_text_ce: 0.037787582725286484 (0.035169893567080814) | > loss_mel_ce: 3.0602495670318604 (nan) | > loss: 0.012293798848986626 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.1364202563991917) | > loader_time: 0.0045 (0.004643955472997244)  --> TIME: 2025-02-25 07:41:07 -- STEP: 23116/36938 -- GLOBAL_STEP: 688000 | > loss_text_ce: 0.03408513590693474 (0.03516862183774802) | > loss_mel_ce: 2.518949031829834 (nan) | > loss: 0.010131088085472584 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1364198613381331) | > loader_time: 0.0045 (0.004643914845600747)  --> TIME: 2025-02-25 07:41:16 -- STEP: 23166/36938 -- GLOBAL_STEP: 688050 | > loss_text_ce: 0.035684049129486084 (0.035168290748432675) | > loss_mel_ce: 2.762590169906616 (nan) | > loss: 0.011104263365268707 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13641484142080415) | > loader_time: 0.0045 (0.004643771373205677)  --> TIME: 2025-02-25 07:41:25 -- STEP: 23216/36938 -- GLOBAL_STEP: 688100 | > loss_text_ce: 0.02874431572854519 (0.03516967384426127) | > loss_mel_ce: 3.279810667037964 (nan) | > loss: 0.013129186816513538 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13640259845062827) | > loader_time: 0.0046 (0.0046436239282974506)  --> TIME: 2025-02-25 07:41:34 -- STEP: 23266/36938 -- GLOBAL_STEP: 688150 | > loss_text_ce: 0.03130507096648216 (0.035168762424947875) | > loss_mel_ce: 2.3684895038604736 (nan) | > loss: 0.009522994980216026 (nan) | > current_lr: 5e-06 | > step_time: 0.1353 (0.136410363673712) | > loader_time: 0.0047 (0.004643569252491587)  --> TIME: 2025-02-25 07:41:44 -- STEP: 23316/36938 -- GLOBAL_STEP: 688200 | > loss_text_ce: 0.034776218235492706 (0.035168928788426326) | > loss_mel_ce: 2.1289875507354736 (nan) | > loss: 0.008586364798247814 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.1364119740554197) | > loader_time: 0.0046 (0.004643775950903157)  --> TIME: 2025-02-25 07:41:53 -- STEP: 23366/36938 -- GLOBAL_STEP: 688250 | > loss_text_ce: 0.0370229110121727 (0.035169700067036076) | > loss_mel_ce: 3.015059471130371 (nan) | > loss: 0.012111438438296318 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13641700401377888) | > loader_time: 0.0049 (0.004643759774006667)  --> TIME: 2025-02-25 07:42:03 -- STEP: 23416/36938 -- GLOBAL_STEP: 688300 | > loss_text_ce: 0.036930397152900696 (0.03516720334756184) | > loss_mel_ce: 2.7633216381073 (nan) | > loss: 0.011112111620604992 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13643045366563758) | > loader_time: 0.0045 (0.004643749999315756)  --> TIME: 2025-02-25 07:42:13 -- STEP: 23466/36938 -- GLOBAL_STEP: 688350 | > loss_text_ce: 0.028621474280953407 (0.03516720849331674) | > loss_mel_ce: 3.5927112102508545 (nan) | > loss: 0.014370368793606758 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13644298888996198) | > loader_time: 0.0045 (0.004643673889880612)  --> TIME: 2025-02-25 07:42:22 -- STEP: 23516/36938 -- GLOBAL_STEP: 688400 | > loss_text_ce: 0.03274019807577133 (0.0351687540077777) | > loss_mel_ce: 3.583251953125 (nan) | > loss: 0.014349175617098808 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.1364418775030771) | > loader_time: 0.0045 (0.004643592375803788)  --> TIME: 2025-02-25 07:42:31 -- STEP: 23566/36938 -- GLOBAL_STEP: 688450 | > loss_text_ce: 0.02707085572183132 (0.035168402144821614) | > loss_mel_ce: 2.930321455001831 (nan) | > loss: 0.011735684238374233 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.136434029047652) | > loader_time: 0.0046 (0.004643739731720704)  --> TIME: 2025-02-25 07:42:41 -- STEP: 23616/36938 -- GLOBAL_STEP: 688500 | > loss_text_ce: 0.037636902183294296 (0.03516922971834113) | > loss_mel_ce: 2.984858989715576 (nan) | > loss: 0.011994032189249992 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13644245235617816) | > loader_time: 0.0045 (0.004643985966275997)  --> TIME: 2025-02-25 07:42:50 -- STEP: 23666/36938 -- GLOBAL_STEP: 688550 | > loss_text_ce: 0.030102262273430824 (0.03516913919811087) | > loss_mel_ce: 1.5384374856948853 (nan) | > loss: 0.006224364507943392 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13644126968090328) | > loader_time: 0.0045 (0.004643882528862904)  --> TIME: 2025-02-25 07:42:59 -- STEP: 23716/36938 -- GLOBAL_STEP: 688600 | > loss_text_ce: 0.032875556498765945 (0.03516762220630769) | > loss_mel_ce: 1.5170027017593384 (nan) | > loss: 0.006150310859084129 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.136437761319147) | > loader_time: 0.0045 (0.004643876901624312)  --> TIME: 2025-02-25 07:43:08 -- STEP: 23766/36938 -- GLOBAL_STEP: 688650 | > loss_text_ce: 0.0319158136844635 (0.03516652207911907) | > loss_mel_ce: 3.2435319423675537 (nan) | > loss: 0.012997809797525406 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13643115354107851) | > loader_time: 0.0046 (0.004643913381962222)  --> TIME: 2025-02-25 07:43:17 -- STEP: 23816/36938 -- GLOBAL_STEP: 688700 | > loss_text_ce: 0.03404957056045532 (0.035167811277993934) | > loss_mel_ce: 0.0107879638671875 (nan) | > loss: 0.0001779267331585288 (nan) | > current_lr: 5e-06 | > step_time: 0.1439 (0.13642019869338495) | > loader_time: 0.0046 (0.0046438149629841885)  --> TIME: 2025-02-25 07:43:26 -- STEP: 23866/36938 -- GLOBAL_STEP: 688750 | > loss_text_ce: 0.03885210305452347 (0.03517127016690058) | > loss_mel_ce: 2.843370199203491 (nan) | > loss: 0.011437390930950642 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13642119087617788) | > loader_time: 0.0046 (0.004643790731683372)  --> TIME: 2025-02-25 07:43:36 -- STEP: 23916/36938 -- GLOBAL_STEP: 688800 | > loss_text_ce: 0.04086841642856598 (0.035175277081845285) | > loss_mel_ce: 2.9796907901763916 (nan) | > loss: 0.011986346915364265 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13642108664701572) | > loader_time: 0.0047 (0.0046438873262783145)  --> TIME: 2025-02-25 07:43:45 -- STEP: 23966/36938 -- GLOBAL_STEP: 688850 | > loss_text_ce: 0.033071693032979965 (0.03517588570691156) | > loss_mel_ce: 3.0030901432037354 (nan) | > loss: 0.012048262171447277 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13642154580154162) | > loader_time: 0.0045 (0.00464387754959044)  --> TIME: 2025-02-25 07:43:54 -- STEP: 24016/36938 -- GLOBAL_STEP: 688900 | > loss_text_ce: 0.028365327045321465 (0.03517513005218839) | > loss_mel_ce: 1.3283264636993408 (nan) | > loss: 0.005383698269724846 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.1364171678606144) | > loader_time: 0.0045 (0.004643895620508696)  --> TIME: 2025-02-25 07:44:04 -- STEP: 24066/36938 -- GLOBAL_STEP: 688950 | > loss_text_ce: 0.036162544041872025 (0.03517408038891301) | > loss_mel_ce: 0.042236328125 (nan) | > loss: 0.00031110664713196456 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13642927314994743) | > loader_time: 0.0046 (0.004643968619244591)  --> TIME: 2025-02-25 07:44:13 -- STEP: 24116/36938 -- GLOBAL_STEP: 689000 | > loss_text_ce: 0.028483673930168152 (0.035173393025749386) | > loss_mel_ce: 3.0147242546081543 (nan) | > loss: 0.012076222337782383 (nan) | > current_lr: 5e-06 | > step_time: 0.1472 (0.13642436292638457) | > loader_time: 0.0047 (0.004644022462063094)  --> TIME: 2025-02-25 07:44:22 -- STEP: 24166/36938 -- GLOBAL_STEP: 689050 | > loss_text_ce: 0.034909024834632874 (0.03517267465165817) | > loss_mel_ce: 3.396691083908081 (nan) | > loss: 0.013617461547255516 (nan) | > current_lr: 5e-06 | > step_time: 0.172 (0.13642462664543123) | > loader_time: 0.0048 (0.00464407108008223)  --> TIME: 2025-02-25 07:44:32 -- STEP: 24216/36938 -- GLOBAL_STEP: 689100 | > loss_text_ce: 0.02994801662862301 (0.03517291107822492) | > loss_mel_ce: 2.6800830364227295 (nan) | > loss: 0.010754091665148735 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1364168627824485) | > loader_time: 0.0046 (0.0046441831189411465)  --> TIME: 2025-02-25 07:44:41 -- STEP: 24266/36938 -- GLOBAL_STEP: 689150 | > loss_text_ce: 0.045747771859169006 (0.03517316516414647) | > loss_mel_ce: 1.965388298034668 (nan) | > loss: 0.007980698719620705 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13641646797930304) | > loader_time: 0.0048 (0.004644160974962048)  --> TIME: 2025-02-25 07:44:50 -- STEP: 24316/36938 -- GLOBAL_STEP: 689200 | > loss_text_ce: 0.033378683030605316 (0.0351737329699507) | > loss_mel_ce: 2.556966781616211 (nan) | > loss: 0.010279148817062378 (nan) | > current_lr: 5e-06 | > step_time: 0.1609 (0.13641503932229299) | > loader_time: 0.0048 (0.004644444250005446)  --> TIME: 2025-02-25 07:44:59 -- STEP: 24366/36938 -- GLOBAL_STEP: 689250 | > loss_text_ce: 0.0383705198764801 (0.03517277979856982) | > loss_mel_ce: 3.277773380279541 (nan) | > loss: 0.013159302063286304 (nan) | > current_lr: 5e-06 | > step_time: 0.1624 (0.13641032749508825) | > loader_time: 0.0046 (0.004644441436314164)  --> TIME: 2025-02-25 07:45:09 -- STEP: 24416/36938 -- GLOBAL_STEP: 689300 | > loss_text_ce: 0.031031673774123192 (0.03517520485594931) | > loss_mel_ce: 3.0863335132598877 (nan) | > loss: 0.012370496988296509 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.13641204269966598) | > loader_time: 0.0046 (0.00464448830793911)  --> TIME: 2025-02-25 07:45:18 -- STEP: 24466/36938 -- GLOBAL_STEP: 689350 | > loss_text_ce: 0.02968491055071354 (0.0351724935337938) | > loss_mel_ce: 2.1420199871063232 (nan) | > loss: 0.008617877960205078 (nan) | > current_lr: 5e-06 | > step_time: 0.1565 (0.13642121923434608) | > loader_time: 0.0046 (0.004644494955960467)  --> TIME: 2025-02-25 07:45:28 -- STEP: 24516/36938 -- GLOBAL_STEP: 689400 | > loss_text_ce: 0.030426548793911934 (0.03517373033738763) | > loss_mel_ce: 1.914136290550232 (nan) | > loss: 0.007716519292443991 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13642725571661765) | > loader_time: 0.0045 (0.004644488020187504)  --> TIME: 2025-02-25 07:45:37 -- STEP: 24566/36938 -- GLOBAL_STEP: 689450 | > loss_text_ce: 0.02208900637924671 (0.03517374650961241) | > loss_mel_ce: 1.3708349466323853 (nan) | > loss: 0.005527476314455271 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13642093110994868) | > loader_time: 0.0045 (0.004644426413994675)  --> TIME: 2025-02-25 07:45:46 -- STEP: 24616/36938 -- GLOBAL_STEP: 689500 | > loss_text_ce: 0.03051207959651947 (0.03517467112131068) | > loss_mel_ce: 2.3166542053222656 (nan) | > loss: 0.009314152412116528 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.13643144934565574) | > loader_time: 0.0053 (0.004644713968978029)  --> TIME: 2025-02-25 07:45:56 -- STEP: 24666/36938 -- GLOBAL_STEP: 689550 | > loss_text_ce: 0.03697871044278145 (0.03517844598690656) | > loss_mel_ce: 3.2048447132110596 (nan) | > loss: 0.012864379212260246 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13643420716904836) | > loader_time: 0.0046 (0.004644733811878525)  --> TIME: 2025-02-25 07:46:05 -- STEP: 24716/36938 -- GLOBAL_STEP: 689600 | > loss_text_ce: 0.0409364216029644 (0.0351772253430743) | > loss_mel_ce: 2.335221767425537 (nan) | > loss: 0.009429199621081352 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.1364319211052619) | > loader_time: 0.0044 (0.004644802732169744)  --> TIME: 2025-02-25 07:46:14 -- STEP: 24766/36938 -- GLOBAL_STEP: 689650 | > loss_text_ce: 0.040379904210567474 (0.0351797431813955) | > loss_mel_ce: 3.2019848823547363 (nan) | > loss: 0.012866527773439884 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13643351781833207) | > loader_time: 0.0047 (0.004644868332090288)  --> TIME: 2025-02-25 07:46:24 -- STEP: 24816/36938 -- GLOBAL_STEP: 689700 | > loss_text_ce: 0.030980655923485756 (0.03517790123413344) | > loss_mel_ce: 3.71875 (nan) | > loss: 0.014879884198307991 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13643346041882873) | > loader_time: 0.0046 (0.00464482913856577)  --> TIME: 2025-02-25 07:46:33 -- STEP: 24866/36938 -- GLOBAL_STEP: 689750 | > loss_text_ce: 0.03435772657394409 (0.0351815117462175) | > loss_mel_ce: 3.7113442420959473 (nan) | > loss: 0.014863898046314716 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.13644499895914544) | > loader_time: 0.0047 (0.004644917461864976)  --> TIME: 2025-02-25 07:46:43 -- STEP: 24916/36938 -- GLOBAL_STEP: 689800 | > loss_text_ce: 0.038160987198352814 (0.035182447638773894) | > loss_mel_ce: 3.012026071548462 (nan) | > loss: 0.012103918008506298 (nan) | > current_lr: 5e-06 | > step_time: 0.187 (0.1364494969592578) | > loader_time: 0.0048 (0.004644867571614691)  --> TIME: 2025-02-25 07:46:52 -- STEP: 24966/36938 -- GLOBAL_STEP: 689850 | > loss_text_ce: 0.02823879010975361 (0.03518140727785713) | > loss_mel_ce: 3.1441662311553955 (nan) | > loss: 0.012588909827172756 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.1364407088047394) | > loader_time: 0.0045 (0.004644736250098323)  --> TIME: 2025-02-25 07:47:01 -- STEP: 25016/36938 -- GLOBAL_STEP: 689900 | > loss_text_ce: 0.030216645449399948 (0.035182589651764566) | > loss_mel_ce: 2.8642783164978027 (nan) | > loss: 0.011486091651022434 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13644466610392442) | > loader_time: 0.0046 (0.004644629299202213)  --> TIME: 2025-02-25 07:47:11 -- STEP: 25066/36938 -- GLOBAL_STEP: 689950 | > loss_text_ce: 0.036264896392822266 (0.03518070498282843) | > loss_mel_ce: 3.440664768218994 (nan) | > loss: 0.013797340914607048 (nan) | > current_lr: 5e-06 | > step_time: 0.1513 (0.13645453024279527) | > loader_time: 0.0045 (0.004644735407526781)  --> TIME: 2025-02-25 07:47:21 -- STEP: 25116/36938 -- GLOBAL_STEP: 690000 | > loss_text_ce: 0.03684244677424431 (0.03518030511806107) | > loss_mel_ce: 3.188849687576294 (nan) | > loss: 0.012800365686416626 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13646623753764672) | > loader_time: 0.0045 (0.0046447954809690035) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_690000.pth  --> TIME: 2025-02-25 07:47:36 -- STEP: 25166/36938 -- GLOBAL_STEP: 690050 | > loss_text_ce: 0.03837617486715317 (0.03518250727357538) | > loss_mel_ce: 2.810504674911499 (nan) | > loss: 0.011305083520710468 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13645963405261066) | > loader_time: 0.0046 (0.004644826079441847)  --> TIME: 2025-02-25 07:47:45 -- STEP: 25216/36938 -- GLOBAL_STEP: 690100 | > loss_text_ce: 0.04303901642560959 (0.0351829231005865) | > loss_mel_ce: 1.0717601776123047 (nan) | > loss: 0.004423806443810463 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13646329319113965) | > loader_time: 0.0046 (0.004644883616926669)  --> TIME: 2025-02-25 07:47:55 -- STEP: 25266/36938 -- GLOBAL_STEP: 690150 | > loss_text_ce: 0.03306888788938522 (0.03518573348101171) | > loss_mel_ce: 3.1497952938079834 (nan) | > loss: 0.012630414217710495 (nan) | > current_lr: 5e-06 | > step_time: 0.1825 (0.136468981937851) | > loader_time: 0.0048 (0.004644929669130352)  --> TIME: 2025-02-25 07:48:04 -- STEP: 25316/36938 -- GLOBAL_STEP: 690200 | > loss_text_ce: 0.030995003879070282 (0.03518527168332497) | > loss_mel_ce: 2.453190803527832 (nan) | > loss: 0.00985787995159626 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13646746355128458) | > loader_time: 0.0046 (0.0046449619308431556)  --> TIME: 2025-02-25 07:48:13 -- STEP: 25366/36938 -- GLOBAL_STEP: 690250 | > loss_text_ce: 0.039349883794784546 (0.035182515747172925) | > loss_mel_ce: 2.638523578643799 (nan) | > loss: 0.010626481845974922 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13647414526427668) | > loader_time: 0.0046 (0.004645093329686077)  --> TIME: 2025-02-25 07:48:22 -- STEP: 25416/36938 -- GLOBAL_STEP: 690300 | > loss_text_ce: 0.028069959953427315 (0.035179140450303084) | > loss_mel_ce: 2.5810546875 (nan) | > loss: 0.01035366952419281 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13646661068307223) | > loader_time: 0.0046 (0.004645066110075408)  --> TIME: 2025-02-25 07:48:32 -- STEP: 25466/36938 -- GLOBAL_STEP: 690350 | > loss_text_ce: 0.03628801181912422 (0.03517930875214529) | > loss_mel_ce: 2.6361987590789795 (nan) | > loss: 0.010605107061564922 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.13646192991807476) | > loader_time: 0.0046 (0.004645020769086726)  --> TIME: 2025-02-25 07:48:42 -- STEP: 25516/36938 -- GLOBAL_STEP: 690400 | > loss_text_ce: 0.020879238843917847 (0.03517849049557206) | > loss_mel_ce: 2.9944844245910645 (nan) | > loss: 0.011965729296207428 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.1364810557495481) | > loader_time: 0.0047 (0.004645189300452764)  --> TIME: 2025-02-25 07:48:51 -- STEP: 25566/36938 -- GLOBAL_STEP: 690450 | > loss_text_ce: 0.0376146174967289 (0.03517720279273842) | > loss_mel_ce: 1.7661527395248413 (nan) | > loss: 0.0071578072383999825 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13648198873124015) | > loader_time: 0.0046 (0.004645257220715478)  --> TIME: 2025-02-25 07:49:00 -- STEP: 25616/36938 -- GLOBAL_STEP: 690500 | > loss_text_ce: 0.02543957717716694 (0.03517668883121788) | > loss_mel_ce: 1.6251307725906372 (nan) | > loss: 0.00654988270252943 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13647594676585106) | > loader_time: 0.0047 (0.004645303133723859)  --> TIME: 2025-02-25 07:49:09 -- STEP: 25666/36938 -- GLOBAL_STEP: 690550 | > loss_text_ce: 0.03801310062408447 (0.035176589982521066) | > loss_mel_ce: 2.6836345195770264 (nan) | > loss: 0.010800190269947052 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13646899514973734) | > loader_time: 0.0047 (0.004645272119840581)  --> TIME: 2025-02-25 07:49:18 -- STEP: 25716/36938 -- GLOBAL_STEP: 690600 | > loss_text_ce: 0.030786797404289246 (0.0351751793216788) | > loss_mel_ce: 2.697812795639038 (nan) | > loss: 0.010827776975929737 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13646835254684586) | > loader_time: 0.0046 (0.004645231714291655)  --> TIME: 2025-02-25 07:49:27 -- STEP: 25766/36938 -- GLOBAL_STEP: 690650 | > loss_text_ce: 0.039326343685388565 (0.03517436234299708) | > loss_mel_ce: 2.951399087905884 (nan) | > loss: 0.011867959052324295 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.13646106543049016) | > loader_time: 0.0049 (0.004645155701847849)  --> TIME: 2025-02-25 07:49:37 -- STEP: 25816/36938 -- GLOBAL_STEP: 690700 | > loss_text_ce: 0.04082553833723068 (0.0351720246305846) | > loss_mel_ce: 2.8366384506225586 (nan) | > loss: 0.011418508365750313 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1364584693071599) | > loader_time: 0.0045 (0.004645113748110647)  --> TIME: 2025-02-25 07:49:46 -- STEP: 25866/36938 -- GLOBAL_STEP: 690750 | > loss_text_ce: 0.04197591170668602 (0.035172547801129786) | > loss_mel_ce: 3.4938693046569824 (nan) | > loss: 0.01403113268315792 (nan) | > current_lr: 5e-06 | > step_time: 0.1858 (0.13645335205338807) | > loader_time: 0.0048 (0.004645150304900624)  --> TIME: 2025-02-25 07:49:55 -- STEP: 25916/36938 -- GLOBAL_STEP: 690800 | > loss_text_ce: 0.03631419688463211 (0.03517322482500682) | > loss_mel_ce: 2.81479811668396 (nan) | > loss: 0.011313938535749912 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13645669219486062) | > loader_time: 0.0045 (0.004645195184325796)  --> TIME: 2025-02-25 07:50:04 -- STEP: 25966/36938 -- GLOBAL_STEP: 690850 | > loss_text_ce: 0.04288869723677635 (0.03517358687129639) | > loss_mel_ce: 2.5100820064544678 (nan) | > loss: 0.010130836628377438 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13645247812402528) | > loader_time: 0.0046 (0.004645122756743496)  --> TIME: 2025-02-25 07:50:14 -- STEP: 26016/36938 -- GLOBAL_STEP: 690900 | > loss_text_ce: 0.03409072011709213 (0.03517417713634912) | > loss_mel_ce: 3.0031986236572266 (nan) | > loss: 0.01205273624509573 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.1364591490811813) | > loader_time: 0.0046 (0.004645063254299475)  --> TIME: 2025-02-25 07:50:23 -- STEP: 26066/36938 -- GLOBAL_STEP: 690950 | > loss_text_ce: 0.029790448024868965 (0.03517555391224032) | > loss_mel_ce: 3.080420970916748 (nan) | > loss: 0.012342109344899654 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13645039426661124) | > loader_time: 0.0046 (0.004644919546697026)  --> TIME: 2025-02-25 07:50:32 -- STEP: 26116/36938 -- GLOBAL_STEP: 691000 | > loss_text_ce: 0.0418393649160862 (0.035174588262430175) | > loss_mel_ce: 1.7370116710662842 (nan) | > loss: 0.007058932911604643 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13644824637481462) | > loader_time: 0.0045 (0.004645024950498072)  --> TIME: 2025-02-25 07:50:41 -- STEP: 26166/36938 -- GLOBAL_STEP: 691050 | > loss_text_ce: 0.03691514581441879 (0.03517388772440965) | > loss_mel_ce: 3.0077474117279053 (nan) | > loss: 0.01208199467509985 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13645293693845648) | > loader_time: 0.0046 (0.0046452267184713255)  --> TIME: 2025-02-25 07:50:51 -- STEP: 26216/36938 -- GLOBAL_STEP: 691100 | > loss_text_ce: 0.031479205936193466 (0.03517152337067678) | > loss_mel_ce: 2.4460647106170654 (nan) | > loss: 0.009831523522734642 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13644592317095694) | > loader_time: 0.0054 (0.00464541429348531)  --> TIME: 2025-02-25 07:51:00 -- STEP: 26266/36938 -- GLOBAL_STEP: 691150 | > loss_text_ce: 0.032319534569978714 (0.035167724690172264) | > loss_mel_ce: 2.2460851669311523 (nan) | > loss: 0.009041288867592812 (nan) | > current_lr: 5e-06 | > step_time: 0.179 (0.13644426478786192) | > loader_time: 0.0048 (0.004645553663081372)  --> TIME: 2025-02-25 07:51:09 -- STEP: 26316/36938 -- GLOBAL_STEP: 691200 | > loss_text_ce: 0.03265916556119919 (0.03516743131659872) | > loss_mel_ce: 2.699838876724243 (nan) | > loss: 0.0108432462438941 (nan) | > current_lr: 5e-06 | > step_time: 0.1844 (0.1364485564012979) | > loader_time: 0.0048 (0.004645619507998472)  --> TIME: 2025-02-25 07:51:19 -- STEP: 26366/36938 -- GLOBAL_STEP: 691250 | > loss_text_ce: 0.032802026718854904 (0.035169433745390666) | > loss_mel_ce: 2.764787435531616 (nan) | > loss: 0.011101546697318554 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13644893536715544) | > loader_time: 0.0046 (0.004645577803065337)  --> TIME: 2025-02-25 07:51:28 -- STEP: 26416/36938 -- GLOBAL_STEP: 691300 | > loss_text_ce: 0.03450487181544304 (0.0351701908087208) | > loss_mel_ce: 3.062664031982422 (nan) | > loss: 0.012290353886783123 (nan) | > current_lr: 5e-06 | > step_time: 0.1793 (0.13644966794497082) | > loader_time: 0.005 (0.004645730936837706)  --> TIME: 2025-02-25 07:51:37 -- STEP: 26466/36938 -- GLOBAL_STEP: 691350 | > loss_text_ce: 0.02554522454738617 (0.03516915238618015) | > loss_mel_ce: 1.0241321325302124 (nan) | > loss: 0.0041653867810964584 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13644772582896153) | > loader_time: 0.0044 (0.004645713528907167)  --> TIME: 2025-02-25 07:51:46 -- STEP: 26516/36938 -- GLOBAL_STEP: 691400 | > loss_text_ce: 0.03187107294797897 (0.03516939683118538) | > loss_mel_ce: 2.1781113147735596 (nan) | > loss: 0.00876977201551199 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13643615229454473) | > loader_time: 0.0047 (0.004645862421454962)  --> TIME: 2025-02-25 07:51:55 -- STEP: 26566/36938 -- GLOBAL_STEP: 691450 | > loss_text_ce: 0.03178052231669426 (0.03517015163762821) | > loss_mel_ce: 1.4452511072158813 (nan) | > loss: 0.005861236713826656 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.1364309445370171) | > loader_time: 0.0046 (0.0046459467737895105)  --> TIME: 2025-02-25 07:52:05 -- STEP: 26616/36938 -- GLOBAL_STEP: 691500 | > loss_text_ce: 0.03597274050116539 (0.03517208281835338) | > loss_mel_ce: 2.9260635375976562 (nan) | > loss: 0.011754113249480724 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13643500588661747) | > loader_time: 0.0052 (0.004646063119683619)  --> TIME: 2025-02-25 07:52:14 -- STEP: 26666/36938 -- GLOBAL_STEP: 691550 | > loss_text_ce: 0.029990319162607193 (0.03517327868501703) | > loss_mel_ce: 2.908318042755127 (nan) | > loss: 0.01165995467454195 (nan) | > current_lr: 5e-06 | > step_time: 0.1072 (0.13642525857989543) | > loader_time: 0.0046 (0.004646020140176151)  --> TIME: 2025-02-25 07:52:23 -- STEP: 26716/36938 -- GLOBAL_STEP: 691600 | > loss_text_ce: 0.03965726122260094 (0.0351728785960562) | > loss_mel_ce: 2.6727826595306396 (nan) | > loss: 0.010763651691377163 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.1364114419627561) | > loader_time: 0.0045 (0.004646172056228091)  --> TIME: 2025-02-25 07:52:32 -- STEP: 26766/36938 -- GLOBAL_STEP: 691650 | > loss_text_ce: 0.04874921217560768 (0.03517402526914538) | > loss_mel_ce: 3.438516855239868 (nan) | > loss: 0.013838358223438263 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.1364178381131499) | > loader_time: 0.0048 (0.004646165412093876)  --> TIME: 2025-02-25 07:52:42 -- STEP: 26816/36938 -- GLOBAL_STEP: 691700 | > loss_text_ce: 0.04015813767910004 (0.035174216023968204) | > loss_mel_ce: 2.71566104888916 (nan) | > loss: 0.010935790836811066 (nan) | > current_lr: 5e-06 | > step_time: 0.146 (0.13642742791544168) | > loader_time: 0.0046 (0.004646110630689759)  --> TIME: 2025-02-25 07:52:51 -- STEP: 26866/36938 -- GLOBAL_STEP: 691750 | > loss_text_ce: 0.0499907024204731 (0.03517616970412888) | > loss_mel_ce: 3.4451181888580322 (nan) | > loss: 0.013869480229914188 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.13642816426259935) | > loader_time: 0.0046 (0.004646136942991033)  --> TIME: 2025-02-25 07:53:00 -- STEP: 26916/36938 -- GLOBAL_STEP: 691800 | > loss_text_ce: 0.0376509353518486 (0.03517568331171138) | > loss_mel_ce: 2.2492237091064453 (nan) | > loss: 0.00907489936798811 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13641817848932788) | > loader_time: 0.0045 (0.004646230477389608)  --> TIME: 2025-02-25 07:53:09 -- STEP: 26966/36938 -- GLOBAL_STEP: 691850 | > loss_text_ce: 0.0337236262857914 (0.03517830270539208) | > loss_mel_ce: 3.0535130500793457 (nan) | > loss: 0.012250939384102821 (nan) | > current_lr: 5e-06 | > step_time: 0.1759 (0.13642417610040736) | > loader_time: 0.0048 (0.004646365060603267)  --> TIME: 2025-02-25 07:53:19 -- STEP: 27016/36938 -- GLOBAL_STEP: 691900 | > loss_text_ce: 0.03011062741279602 (0.035176299388490595) | > loss_mel_ce: 2.420842170715332 (nan) | > loss: 0.009726003743708134 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13642141693330864) | > loader_time: 0.0046 (0.004646268793207172)  --> TIME: 2025-02-25 07:53:28 -- STEP: 27066/36938 -- GLOBAL_STEP: 691950 | > loss_text_ce: 0.03784957155585289 (0.03517116588838177) | > loss_mel_ce: 3.1172516345977783 (nan) | > loss: 0.012520244345068932 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13641991893124375) | > loader_time: 0.0046 (0.004646191591347239)  --> TIME: 2025-02-25 07:53:38 -- STEP: 27116/36938 -- GLOBAL_STEP: 692000 | > loss_text_ce: 0.026389680802822113 (0.03517244116468167) | > loss_mel_ce: 2.228381633758545 (nan) | > loss: 0.008947505615651608 (nan) | > current_lr: 5e-06 | > step_time: 0.1788 (0.1364370116197096) | > loader_time: 0.0047 (0.004646190650555602)  --> TIME: 2025-02-25 07:53:47 -- STEP: 27166/36938 -- GLOBAL_STEP: 692050 | > loss_text_ce: 0.04223022237420082 (0.035172858376195736) | > loss_mel_ce: 1.9287220239639282 (nan) | > loss: 0.007821239531040192 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13643480654906423) | > loader_time: 0.0045 (0.004646109523631366)  --> TIME: 2025-02-25 07:53:56 -- STEP: 27216/36938 -- GLOBAL_STEP: 692100 | > loss_text_ce: 0.018191633746027946 (0.03517331139401916) | > loss_mel_ce: 1.7861579656600952 (nan) | > loss: 0.0071601178497076035 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13642822997945542) | > loader_time: 0.0044 (0.004646055238307187)  --> TIME: 2025-02-25 07:54:06 -- STEP: 27266/36938 -- GLOBAL_STEP: 692150 | > loss_text_ce: 0.02607792057096958 (0.03517137865878119) | > loss_mel_ce: 2.410291910171509 (nan) | > loss: 0.009668135084211826 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13643063428090008) | > loader_time: 0.0046 (0.004646054841291092)  --> TIME: 2025-02-25 07:54:15 -- STEP: 27316/36938 -- GLOBAL_STEP: 692200 | > loss_text_ce: 0.0387539267539978 (0.03517166471106361) | > loss_mel_ce: 3.2521564960479736 (nan) | > loss: 0.013059169054031372 (nan) | > current_lr: 5e-06 | > step_time: 0.1883 (0.1364373338573363) | > loader_time: 0.0048 (0.004646125597736639)  --> TIME: 2025-02-25 07:54:24 -- STEP: 27366/36938 -- GLOBAL_STEP: 692250 | > loss_text_ce: 0.038383178412914276 (0.03517066803714905) | > loss_mel_ce: 3.677281141281128 (nan) | > loss: 0.014744700863957405 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.13643654597865726) | > loader_time: 0.0049 (0.004646096785044292)  --> TIME: 2025-02-25 07:54:34 -- STEP: 27416/36938 -- GLOBAL_STEP: 692300 | > loss_text_ce: 0.042740095406770706 (0.03517083050529619) | > loss_mel_ce: 3.555384397506714 (nan) | > loss: 0.014278273098170757 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13644085037906428) | > loader_time: 0.0046 (0.004646085870137253)  --> TIME: 2025-02-25 07:54:43 -- STEP: 27466/36938 -- GLOBAL_STEP: 692350 | > loss_text_ce: 0.022002441808581352 (0.035171757115413885) | > loss_mel_ce: 1.217016577720642 (nan) | > loss: 0.004916742444038391 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1364381897174307) | > loader_time: 0.0046 (0.004646143301814709)  --> TIME: 2025-02-25 07:54:52 -- STEP: 27516/36938 -- GLOBAL_STEP: 692400 | > loss_text_ce: 0.030840985476970673 (0.035170811384487036) | > loss_mel_ce: 1.5548269748687744 (nan) | > loss: 0.006292333360761404 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13643257358701716) | > loader_time: 0.0047 (0.004646132047447738)  --> TIME: 2025-02-25 07:55:01 -- STEP: 27566/36938 -- GLOBAL_STEP: 692450 | > loss_text_ce: 0.030521031469106674 (0.03517088990551509) | > loss_mel_ce: 1.4740962982177734 (nan) | > loss: 0.005970703903585672 (nan) | > current_lr: 5e-06 | > step_time: 0.1081 (0.13642367835419386) | > loader_time: 0.0044 (0.004646109988050603)  --> TIME: 2025-02-25 07:55:11 -- STEP: 27616/36938 -- GLOBAL_STEP: 692500 | > loss_text_ce: 0.03524817153811455 (0.03517047949631741) | > loss_mel_ce: 3.089813470840454 (nan) | > loss: 0.012401039712131023 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.13642373440197825) | > loader_time: 0.0049 (0.004646159112798791)  --> TIME: 2025-02-25 07:55:20 -- STEP: 27666/36938 -- GLOBAL_STEP: 692550 | > loss_text_ce: 0.03295839577913284 (0.03517018062894935) | > loss_mel_ce: 2.8452913761138916 (nan) | > loss: 0.0114216273650527 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13642753894973708) | > loader_time: 0.0046 (0.004646098045829505)  --> TIME: 2025-02-25 07:55:30 -- STEP: 27716/36938 -- GLOBAL_STEP: 692600 | > loss_text_ce: 0.038248274475336075 (0.035168839859022236) | > loss_mel_ce: 2.568204641342163 (nan) | > loss: 0.010343068279325962 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.1364378951589912) | > loader_time: 0.0048 (0.00464608273923887)  --> TIME: 2025-02-25 07:55:39 -- STEP: 27766/36938 -- GLOBAL_STEP: 692650 | > loss_text_ce: 0.037770647555589676 (0.03517090019951179) | > loss_mel_ce: 3.1583714485168457 (nan) | > loss: 0.01268310472369194 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13643537834742714) | > loader_time: 0.0046 (0.004646005835200317)  --> TIME: 2025-02-25 07:55:48 -- STEP: 27816/36938 -- GLOBAL_STEP: 692700 | > loss_text_ce: 0.04002397507429123 (0.03516861636812773) | > loss_mel_ce: 2.396984338760376 (nan) | > loss: 0.00967066828161478 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13644003536394556) | > loader_time: 0.0047 (0.004646066270890779)  --> TIME: 2025-02-25 07:55:58 -- STEP: 27866/36938 -- GLOBAL_STEP: 692750 | > loss_text_ce: 0.03537628799676895 (0.035173280215132885) | > loss_mel_ce: 2.6186912059783936 (nan) | > loss: 0.01053201500326395 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13644718552306387) | > loader_time: 0.0046 (0.004646086473779135)  --> TIME: 2025-02-25 07:56:07 -- STEP: 27916/36938 -- GLOBAL_STEP: 692800 | > loss_text_ce: 0.03335391357541084 (0.03517350273768395) | > loss_mel_ce: 3.2244977951049805 (nan) | > loss: 0.012927982956171036 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13644751262862798) | > loader_time: 0.0045 (0.004646139323224579)  --> TIME: 2025-02-25 07:56:16 -- STEP: 27966/36938 -- GLOBAL_STEP: 692850 | > loss_text_ce: 0.02470249868929386 (0.03517310159996377) | > loss_mel_ce: 2.884486675262451 (nan) | > loss: 0.011544402688741684 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13644312950347753) | > loader_time: 0.0046 (0.004646124113768043)  --> TIME: 2025-02-25 07:56:26 -- STEP: 28016/36938 -- GLOBAL_STEP: 692900 | > loss_text_ce: 0.03981538861989975 (0.03517446732683089) | > loss_mel_ce: 3.165102958679199 (nan) | > loss: 0.012717930600047112 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13643993478989483) | > loader_time: 0.0046 (0.004646178885979488)  --> TIME: 2025-02-25 07:56:35 -- STEP: 28066/36938 -- GLOBAL_STEP: 692950 | > loss_text_ce: 0.04251641035079956 (0.03517369411203248) | > loss_mel_ce: 3.506235361099243 (nan) | > loss: 0.01408234890550375 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.1364411255982519) | > loader_time: 0.0046 (0.004646204231997252)  --> TIME: 2025-02-25 07:56:44 -- STEP: 28116/36938 -- GLOBAL_STEP: 693000 | > loss_text_ce: 0.0306028351187706 (0.0351710469545254) | > loss_mel_ce: 2.891357421875 (nan) | > loss: 0.01159508153796196 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.1364329025321049) | > loader_time: 0.0046 (0.004646168526451247)  --> TIME: 2025-02-25 07:56:54 -- STEP: 28166/36938 -- GLOBAL_STEP: 693050 | > loss_text_ce: 0.03347871080040932 (0.03517197668468433) | > loss_mel_ce: 1.3236815929412842 (nan) | > loss: 0.005385557189583778 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13643598241629182) | > loader_time: 0.0046 (0.004646091140197737)  --> TIME: 2025-02-25 07:57:03 -- STEP: 28216/36938 -- GLOBAL_STEP: 693100 | > loss_text_ce: 0.04277796298265457 (0.03517156394533507) | > loss_mel_ce: 2.0963900089263916 (nan) | > loss: 0.008488762192428112 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13643447422325458) | > loader_time: 0.0045 (0.0046459771196329085)  --> TIME: 2025-02-25 07:57:12 -- STEP: 28266/36938 -- GLOBAL_STEP: 693150 | > loss_text_ce: 0.0321761853992939 (0.03516938829912366) | > loss_mel_ce: 3.0050177574157715 (nan) | > loss: 0.012052358128130436 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13642999318335383) | > loader_time: 0.0046 (0.004645943321139491)  --> TIME: 2025-02-25 07:57:22 -- STEP: 28316/36938 -- GLOBAL_STEP: 693200 | > loss_text_ce: 0.03267095237970352 (0.03516693870403594) | > loss_mel_ce: 2.4051311016082764 (nan) | > loss: 0.00967381801456213 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13643374592423577) | > loader_time: 0.0045 (0.004646059196704958)  --> TIME: 2025-02-25 07:57:31 -- STEP: 28366/36938 -- GLOBAL_STEP: 693250 | > loss_text_ce: 0.04092099517583847 (0.035165199178134474) | > loss_mel_ce: 3.013127088546753 (nan) | > loss: 0.012119239196181297 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.13643414689448355) | > loader_time: 0.0046 (0.004646185817313713)  --> TIME: 2025-02-25 07:57:40 -- STEP: 28416/36938 -- GLOBAL_STEP: 693300 | > loss_text_ce: 0.0285625122487545 (0.03516491437842792) | > loss_mel_ce: 2.4787752628326416 (nan) | > loss: 0.009949753992259502 (nan) | > current_lr: 5e-06 | > step_time: 0.1784 (0.13644317002245343) | > loader_time: 0.0048 (0.004646128538559682)  --> TIME: 2025-02-25 07:57:50 -- STEP: 28466/36938 -- GLOBAL_STEP: 693350 | > loss_text_ce: 0.028641968965530396 (0.035164468439112376) | > loss_mel_ce: 1.798034429550171 (nan) | > loss: 0.007248715963214636 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.1364450387874967) | > loader_time: 0.0044 (0.004646057029940675)  --> TIME: 2025-02-25 07:57:59 -- STEP: 28516/36938 -- GLOBAL_STEP: 693400 | > loss_text_ce: 0.03281121701002121 (0.035160586084962364) | > loss_mel_ce: 0.0592041015625 (nan) | > loss: 0.0003651401784736663 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13644330761412152) | > loader_time: 0.0045 (0.0046460132542691865)  --> TIME: 2025-02-25 07:58:08 -- STEP: 28566/36938 -- GLOBAL_STEP: 693450 | > loss_text_ce: 0.03667256981134415 (0.035159123245653394) | > loss_mel_ce: 2.065452814102173 (nan) | > loss: 0.008341767825186253 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.1364445160694376) | > loader_time: 0.0049 (0.004646017964894649)  --> TIME: 2025-02-25 07:58:17 -- STEP: 28616/36938 -- GLOBAL_STEP: 693500 | > loss_text_ce: 0.027431944385170937 (0.035156439342832954) | > loss_mel_ce: 1.4783110618591309 (nan) | > loss: 0.0059751709923148155 (nan) | > current_lr: 5e-06 | > step_time: 0.1079 (0.13643894174394142) | > loader_time: 0.0045 (0.004646010936422987)  --> TIME: 2025-02-25 07:58:27 -- STEP: 28666/36938 -- GLOBAL_STEP: 693550 | > loss_text_ce: 0.03918986767530441 (0.035156712676448265) | > loss_mel_ce: 3.3132922649383545 (nan) | > loss: 0.013303501531481743 (nan) | > current_lr: 5e-06 | > step_time: 0.1766 (0.13643727609120246) | > loader_time: 0.0047 (0.004646135409512253)  --> TIME: 2025-02-25 07:58:36 -- STEP: 28716/36938 -- GLOBAL_STEP: 693600 | > loss_text_ce: 0.03288789093494415 (0.035155683123155305) | > loss_mel_ce: 2.4186580181121826 (nan) | > loss: 0.009728357195854187 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13643683988182115) | > loader_time: 0.0047 (0.004646128159509855)  --> TIME: 2025-02-25 07:58:46 -- STEP: 28766/36938 -- GLOBAL_STEP: 693650 | > loss_text_ce: 0.033265676349401474 (0.03515460473294093) | > loss_mel_ce: 3.25007700920105 (nan) | > loss: 0.013029137626290321 (nan) | > current_lr: 5e-06 | > step_time: 0.1775 (0.13644085478767642) | > loader_time: 0.0049 (0.004646221694450319)  --> TIME: 2025-02-25 07:58:55 -- STEP: 28816/36938 -- GLOBAL_STEP: 693700 | > loss_text_ce: 0.0309775210916996 (0.035154105139256986) | > loss_mel_ce: 3.3186466693878174 (nan) | > loss: 0.013292159885168076 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13645248667754215) | > loader_time: 0.0047 (0.004646311992410151)  --> TIME: 2025-02-25 07:59:04 -- STEP: 28866/36938 -- GLOBAL_STEP: 693750 | > loss_text_ce: 0.03383760526776314 (0.03515547326358345) | > loss_mel_ce: 2.4188029766082764 (nan) | > loss: 0.009732700884342194 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.1364461833570211) | > loader_time: 0.0047 (0.004646301352126195)  --> TIME: 2025-02-25 07:59:14 -- STEP: 28916/36938 -- GLOBAL_STEP: 693800 | > loss_text_ce: 0.042317185550928116 (0.03515497386039549) | > loss_mel_ce: 3.1952879428863525 (nan) | > loss: 0.012847639620304108 (nan) | > current_lr: 5e-06 | > step_time: 0.1512 (0.13644529413856948) | > loader_time: 0.0046 (0.004646487025934918)  --> TIME: 2025-02-25 07:59:23 -- STEP: 28966/36938 -- GLOBAL_STEP: 693850 | > loss_text_ce: 0.02260417677462101 (0.03515340504395684) | > loss_mel_ce: 1.6271830797195435 (nan) | > loss: 0.006546775344759226 (nan) | > current_lr: 5e-06 | > step_time: 0.1084 (0.13644122958043287) | > loader_time: 0.0046 (0.0046464845240795135)  --> TIME: 2025-02-25 07:59:32 -- STEP: 29016/36938 -- GLOBAL_STEP: 693900 | > loss_text_ce: 0.027600791305303574 (0.03515285635913977) | > loss_mel_ce: 2.5815541744232178 (nan) | > loss: 0.010353789664804935 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13643965353510792) | > loader_time: 0.0046 (0.004646463000744799)  --> TIME: 2025-02-25 07:59:42 -- STEP: 29066/36938 -- GLOBAL_STEP: 693950 | > loss_text_ce: 0.02664853073656559 (0.03515384237490968) | > loss_mel_ce: 2.1897287368774414 (nan) | > loss: 0.008795148693025112 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13644432798432907) | > loader_time: 0.0045 (0.004646655435876456)  --> TIME: 2025-02-25 07:59:51 -- STEP: 29116/36938 -- GLOBAL_STEP: 694000 | > loss_text_ce: 0.02432072162628174 (0.03515484054180224) | > loss_mel_ce: 3.1269423961639404 (nan) | > loss: 0.01250501349568367 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.13644489667657572) | > loader_time: 0.0046 (0.004646619051789035)  --> TIME: 2025-02-25 08:00:00 -- STEP: 29166/36938 -- GLOBAL_STEP: 694050 | > loss_text_ce: 0.03823991119861603 (0.03515245709222134) | > loss_mel_ce: 3.3569235801696777 (nan) | > loss: 0.013472871854901314 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13644570755155722) | > loader_time: 0.0047 (0.004646593100542489)  --> TIME: 2025-02-25 08:00:10 -- STEP: 29216/36938 -- GLOBAL_STEP: 694100 | > loss_text_ce: 0.04394759610295296 (0.035153135322218095) | > loss_mel_ce: 2.9479990005493164 (nan) | > loss: 0.01187280472368002 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13644935224135793) | > loader_time: 0.0046 (0.004646550819098228)  --> TIME: 2025-02-25 08:00:19 -- STEP: 29266/36938 -- GLOBAL_STEP: 694150 | > loss_text_ce: 0.030868342146277428 (0.03515245383301785) | > loss_mel_ce: 3.3789076805114746 (nan) | > loss: 0.013530857861042023 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.1364496172407233) | > loader_time: 0.0046 (0.004646592087081877)  --> TIME: 2025-02-25 08:00:28 -- STEP: 29316/36938 -- GLOBAL_STEP: 694200 | > loss_text_ce: 0.042506132274866104 (0.03515237884367016) | > loss_mel_ce: 3.6292026042938232 (nan) | > loss: 0.014570274390280247 (nan) | > current_lr: 5e-06 | > step_time: 0.1877 (0.13644696990665067) | > loader_time: 0.0048 (0.00464682697615465)  --> TIME: 2025-02-25 08:00:37 -- STEP: 29366/36938 -- GLOBAL_STEP: 694250 | > loss_text_ce: 0.03237320855259895 (0.03515206148990918) | > loss_mel_ce: 3.0498857498168945 (nan) | > loss: 0.012231186963617802 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13643958642779314) | > loader_time: 0.0046 (0.004646787199817849)  --> TIME: 2025-02-25 08:00:47 -- STEP: 29416/36938 -- GLOBAL_STEP: 694300 | > loss_text_ce: 0.03479549288749695 (0.035153154125776666) | > loss_mel_ce: 2.5936362743377686 (nan) | > loss: 0.010430285707116127 (nan) | > current_lr: 5e-06 | > step_time: 0.1264 (0.13643859659552238) | > loader_time: 0.0046 (0.004646780700310108)  --> TIME: 2025-02-25 08:00:56 -- STEP: 29466/36938 -- GLOBAL_STEP: 694350 | > loss_text_ce: 0.038756608963012695 (0.035152684237805834) | > loss_mel_ce: 3.245919942855835 (nan) | > loss: 0.013034431263804436 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13644682328602026) | > loader_time: 0.0046 (0.00464676200497994)  --> TIME: 2025-02-25 08:01:05 -- STEP: 29516/36938 -- GLOBAL_STEP: 694400 | > loss_text_ce: 0.026867453008890152 (0.03515086436279707) | > loss_mel_ce: 2.892362117767334 (nan) | > loss: 0.011584244668483734 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13644417703983583) | > loader_time: 0.0045 (0.004646686748980761)  --> TIME: 2025-02-25 08:01:15 -- STEP: 29566/36938 -- GLOBAL_STEP: 694450 | > loss_text_ce: 0.038445111364126205 (0.035149662177219855) | > loss_mel_ce: 2.2754125595092773 (nan) | > loss: 0.0091819753870368 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1364433658644742) | > loader_time: 0.0046 (0.004646732343805177)  --> TIME: 2025-02-25 08:01:24 -- STEP: 29616/36938 -- GLOBAL_STEP: 694500 | > loss_text_ce: 0.044739626348018646 (0.0351507209093215) | > loss_mel_ce: 3.050097703933716 (nan) | > loss: 0.012281101197004318 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13645227982796831) | > loader_time: 0.0046 (0.004646848764437585)  --> TIME: 2025-02-25 08:01:34 -- STEP: 29666/36938 -- GLOBAL_STEP: 694550 | > loss_text_ce: 0.02949867956340313 (0.035150425159207695) | > loss_mel_ce: 1.997037649154663 (nan) | > loss: 0.008041811175644398 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13646302547569159) | > loader_time: 0.0046 (0.004646956257590917)  --> TIME: 2025-02-25 08:01:43 -- STEP: 29716/36938 -- GLOBAL_STEP: 694600 | > loss_text_ce: 0.022704560309648514 (0.0351507361363513) | > loss_mel_ce: 1.9017802476882935 (nan) | > loss: 0.007636845111846924 (nan) | > current_lr: 5e-06 | > step_time: 0.1355 (0.13645832455002463) | > loader_time: 0.0047 (0.004646943281116962)  --> TIME: 2025-02-25 08:01:53 -- STEP: 29766/36938 -- GLOBAL_STEP: 694650 | > loss_text_ce: 0.03462551534175873 (0.035152281297473914) | > loss_mel_ce: 2.420785665512085 (nan) | > loss: 0.009743696078658104 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13646465932009985) | > loader_time: 0.0047 (0.0046470458009490795)  --> TIME: 2025-02-25 08:02:02 -- STEP: 29816/36938 -- GLOBAL_STEP: 694700 | > loss_text_ce: 0.036761969327926636 (0.035151606498234775) | > loss_mel_ce: 2.3627047538757324 (nan) | > loss: 0.009521693922579288 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.1364702648968731) | > loader_time: 0.0047 (0.004647204782868759)  --> TIME: 2025-02-25 08:02:12 -- STEP: 29866/36938 -- GLOBAL_STEP: 694750 | > loss_text_ce: 0.033022090792655945 (0.03515319736290386) | > loss_mel_ce: 3.163257360458374 (nan) | > loss: 0.01268364954739809 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13648847460563032) | > loader_time: 0.0046 (0.004647346556102851)  --> TIME: 2025-02-25 08:02:21 -- STEP: 29916/36938 -- GLOBAL_STEP: 694800 | > loss_text_ce: 0.03731240704655647 (0.03515245057678584) | > loss_mel_ce: 2.9004993438720703 (nan) | > loss: 0.011657983995974064 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.13648717293271864) | > loader_time: 0.0068 (0.00464737028788079)  --> TIME: 2025-02-25 08:02:31 -- STEP: 29966/36938 -- GLOBAL_STEP: 694850 | > loss_text_ce: 0.03884970769286156 (0.035151669119509024) | > loss_mel_ce: 2.851727247238159 (nan) | > loss: 0.011470543220639229 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13649673999919168) | > loader_time: 0.0045 (0.004647417403600929)  --> TIME: 2025-02-25 08:02:40 -- STEP: 30016/36938 -- GLOBAL_STEP: 694900 | > loss_text_ce: 0.034509267657995224 (0.035150167970312084) | > loss_mel_ce: 3.2015371322631836 (nan) | > loss: 0.012841454707086086 (nan) | > current_lr: 5e-06 | > step_time: 0.1752 (0.13649702362859065) | > loader_time: 0.0048 (0.004647370531106551)  --> TIME: 2025-02-25 08:02:50 -- STEP: 30066/36938 -- GLOBAL_STEP: 694950 | > loss_text_ce: 0.03380490839481354 (0.03515170664990368) | > loss_mel_ce: 3.256108522415161 (nan) | > loss: 0.013055212795734406 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.13649739045816928) | > loader_time: 0.0047 (0.0046473819323105585)  --> TIME: 2025-02-25 08:02:59 -- STEP: 30116/36938 -- GLOBAL_STEP: 695000 | > loss_text_ce: 0.04033535346388817 (0.0351500600049572) | > loss_mel_ce: 2.6562399864196777 (nan) | > loss: 0.01070069707930088 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.1364977859709733) | > loader_time: 0.0046 (0.004647432039863541)  --> TIME: 2025-02-25 08:03:09 -- STEP: 30166/36938 -- GLOBAL_STEP: 695050 | > loss_text_ce: 0.046525049954652786 (0.03514780224152309) | > loss_mel_ce: 2.982675313949585 (nan) | > loss: 0.012020637281239033 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1365015169578804) | > loader_time: 0.0045 (0.0046474485650884806)  --> TIME: 2025-02-25 08:03:18 -- STEP: 30216/36938 -- GLOBAL_STEP: 695100 | > loss_text_ce: 0.03009914606809616 (0.03514758119424228) | > loss_mel_ce: 3.145540237426758 (nan) | > loss: 0.012601744383573532 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13650359566450831) | > loader_time: 0.0046 (0.004647476950239683)  --> TIME: 2025-02-25 08:03:27 -- STEP: 30266/36938 -- GLOBAL_STEP: 695150 | > loss_text_ce: 0.03681067377328873 (0.03514677090910673) | > loss_mel_ce: 2.7051713466644287 (nan) | > loss: 0.010880881920456886 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13650047862584) | > loader_time: 0.0066 (0.004647487887606412)  --> TIME: 2025-02-25 08:03:37 -- STEP: 30316/36938 -- GLOBAL_STEP: 695200 | > loss_text_ce: 0.031290777027606964 (0.035147064223763756) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13650363892894746) | > loader_time: 0.0046 (0.004647469674712007)  --> TIME: 2025-02-25 08:03:46 -- STEP: 30366/36938 -- GLOBAL_STEP: 695250 | > loss_text_ce: 0.03170933946967125 (0.0351442236459798) | > loss_mel_ce: 0.1685791015625 (nan) | > loss: 0.0007947954582050443 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1364950617020953) | > loader_time: 0.0046 (0.004647489538747375)  --> TIME: 2025-02-25 08:03:55 -- STEP: 30416/36938 -- GLOBAL_STEP: 695300 | > loss_text_ce: 0.027343828231096268 (0.0351415645270029) | > loss_mel_ce: 1.3046565055847168 (nan) | > loss: 0.005285716149955988 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.136492891869125) | > loader_time: 0.0045 (0.004647488259504115)  --> TIME: 2025-02-25 08:04:04 -- STEP: 30466/36938 -- GLOBAL_STEP: 695350 | > loss_text_ce: 0.045223820954561234 (0.03514049490863016) | > loss_mel_ce: 1.929701328277588 (nan) | > loss: 0.007837004959583282 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.1364865846457978) | > loader_time: 0.0046 (0.004647449194027212)  --> TIME: 2025-02-25 08:04:13 -- STEP: 30516/36938 -- GLOBAL_STEP: 695400 | > loss_text_ce: 0.03970145061612129 (0.035139942813644284) | > loss_mel_ce: 2.46994686126709 (nan) | > loss: 0.009958922863006592 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13648370316556951) | > loader_time: 0.0046 (0.004647448938254492)  --> TIME: 2025-02-25 08:04:23 -- STEP: 30566/36938 -- GLOBAL_STEP: 695450 | > loss_text_ce: 0.03616507351398468 (0.0351386706117545) | > loss_mel_ce: 2.6986637115478516 (nan) | > loss: 0.010852495208382607 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.1364836772371309) | > loader_time: 0.0048 (0.004647452559980007)  --> TIME: 2025-02-25 08:04:32 -- STEP: 30616/36938 -- GLOBAL_STEP: 695500 | > loss_text_ce: 0.03562166541814804 (0.035140137237659595) | > loss_mel_ce: 2.287203073501587 (nan) | > loss: 0.009217559359967709 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13647757541862224) | > loader_time: 0.0047 (0.004647453740211884)  --> TIME: 2025-02-25 08:04:41 -- STEP: 30666/36938 -- GLOBAL_STEP: 695550 | > loss_text_ce: 0.03592107817530632 (0.03514122392356394) | > loss_mel_ce: 3.166410446166992 (nan) | > loss: 0.0127076655626297 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13647064844362863) | > loader_time: 0.0046 (0.004647460358876624)  --> TIME: 2025-02-25 08:04:50 -- STEP: 30716/36938 -- GLOBAL_STEP: 695600 | > loss_text_ce: 0.036812927573919296 (0.03514134511909493) | > loss_mel_ce: 2.7465996742248535 (nan) | > loss: 0.011045289225876331 (nan) | > current_lr: 5e-06 | > step_time: 0.1573 (0.13647633707139795) | > loader_time: 0.0047 (0.004647466397127112)  --> TIME: 2025-02-25 08:05:00 -- STEP: 30766/36938 -- GLOBAL_STEP: 695650 | > loss_text_ce: 0.03629562631249428 (0.03514266633616629) | > loss_mel_ce: 3.501481771469116 (nan) | > loss: 0.01403880026191473 (nan) | > current_lr: 5e-06 | > step_time: 0.1609 (0.1364771704412446) | > loader_time: 0.0047 (0.004647480289159415)  --> TIME: 2025-02-25 08:05:09 -- STEP: 30816/36938 -- GLOBAL_STEP: 695700 | > loss_text_ce: 0.028383085504174232 (0.035140174244052386) | > loss_mel_ce: 3.278641700744629 (nan) | > loss: 0.013123114593327045 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.1364778507489271) | > loader_time: 0.0046 (0.00464748559463556)  --> TIME: 2025-02-25 08:05:18 -- STEP: 30866/36938 -- GLOBAL_STEP: 695750 | > loss_text_ce: 0.03586526960134506 (0.035139888960829375) | > loss_mel_ce: 2.6171672344207764 (nan) | > loss: 0.010527907870709896 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13647935770971859) | > loader_time: 0.0045 (0.0046474690771934005)  --> TIME: 2025-02-25 08:05:27 -- STEP: 30916/36938 -- GLOBAL_STEP: 695800 | > loss_text_ce: 0.04527783766388893 (0.035139282113817276) | > loss_mel_ce: 2.4664816856384277 (nan) | > loss: 0.009967300109565258 (nan) | > current_lr: 5e-06 | > step_time: 0.108 (0.13646786329215257) | > loader_time: 0.0045 (0.004647521325480442)  --> TIME: 2025-02-25 08:05:36 -- STEP: 30966/36938 -- GLOBAL_STEP: 695850 | > loss_text_ce: 0.031133592128753662 (0.035140893385886274) | > loss_mel_ce: 2.630535125732422 (nan) | > loss: 0.010562178678810596 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13646214469369253) | > loader_time: 0.0048 (0.00464745913104386)  --> TIME: 2025-02-25 08:05:46 -- STEP: 31016/36938 -- GLOBAL_STEP: 695900 | > loss_text_ce: 0.036913998425006866 (0.03514059799952626) | > loss_mel_ce: 2.9119699001312256 (nan) | > loss: 0.011701921001076698 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1364629048568724) | > loader_time: 0.0045 (0.004647447548178255)  --> TIME: 2025-02-25 08:05:55 -- STEP: 31066/36938 -- GLOBAL_STEP: 695950 | > loss_text_ce: 0.03884579986333847 (0.035141899869780675) | > loss_mel_ce: 2.9595794677734375 (nan) | > loss: 0.01189851388335228 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13646772902739163) | > loader_time: 0.0046 (0.004647660699022992)  --> TIME: 2025-02-25 08:06:05 -- STEP: 31116/36938 -- GLOBAL_STEP: 696000 | > loss_text_ce: 0.034707844257354736 (0.03514196231558635) | > loss_mel_ce: 2.327867269515991 (nan) | > loss: 0.009375298395752907 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.13647780292428496) | > loader_time: 0.0046 (0.004647732416972627)  --> TIME: 2025-02-25 08:06:14 -- STEP: 31166/36938 -- GLOBAL_STEP: 696050 | > loss_text_ce: 0.033012934029102325 (0.03514019600223604) | > loss_mel_ce: 1.2662674188613892 (nan) | > loss: 0.005155874881893396 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13648501604074342) | > loader_time: 0.0048 (0.004647774528969022)  --> TIME: 2025-02-25 08:06:24 -- STEP: 31216/36938 -- GLOBAL_STEP: 696100 | > loss_text_ce: 0.03562004491686821 (0.03513626437617534) | > loss_mel_ce: 2.711386203765869 (nan) | > loss: 0.01090081874281168 (nan) | > current_lr: 5e-06 | > step_time: 0.1797 (0.13648568462677718) | > loader_time: 0.0048 (0.004647743184098455)  --> TIME: 2025-02-25 08:06:33 -- STEP: 31266/36938 -- GLOBAL_STEP: 696150 | > loss_text_ce: 0.03995955362915993 (0.03513630167340175) | > loss_mel_ce: 2.3665263652801514 (nan) | > loss: 0.009549547918140888 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13648927954484122) | > loader_time: 0.0045 (0.004647878190419236)  --> TIME: 2025-02-25 08:06:43 -- STEP: 31316/36938 -- GLOBAL_STEP: 696200 | > loss_text_ce: 0.04477463662624359 (0.03513393457277123) | > loss_mel_ce: 2.8802897930145264 (nan) | > loss: 0.011607399210333824 (nan) | > current_lr: 5e-06 | > step_time: 0.1528 (0.13649997812134096) | > loader_time: 0.0047 (0.004648078293434302)  --> TIME: 2025-02-25 08:06:52 -- STEP: 31366/36938 -- GLOBAL_STEP: 696250 | > loss_text_ce: 0.03812785819172859 (0.035132005523496296) | > loss_mel_ce: 2.893634557723999 (nan) | > loss: 0.011633978225290775 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.1364961162419483) | > loader_time: 0.0049 (0.004648166583644438)  --> TIME: 2025-02-25 08:07:02 -- STEP: 31416/36938 -- GLOBAL_STEP: 696300 | > loss_text_ce: 0.03834127262234688 (0.035132162817417205) | > loss_mel_ce: 2.628511667251587 (nan) | > loss: 0.010582750663161278 (nan) | > current_lr: 5e-06 | > step_time: 0.1625 (0.13650206889485614) | > loader_time: 0.0046 (0.004648198274246205)  --> TIME: 2025-02-25 08:07:11 -- STEP: 31466/36938 -- GLOBAL_STEP: 696350 | > loss_text_ce: 0.02742382138967514 (0.03513322377333313) | > loss_mel_ce: 3.2286665439605713 (nan) | > loss: 0.012920994311571121 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.1365013272280541) | > loader_time: 0.0046 (0.004648263445496366)  --> TIME: 2025-02-25 08:07:20 -- STEP: 31516/36938 -- GLOBAL_STEP: 696400 | > loss_text_ce: 0.03875696659088135 (0.03513266667012675) | > loss_mel_ce: 3.471546173095703 (nan) | > loss: 0.013929774984717369 (nan) | > current_lr: 5e-06 | > step_time: 0.1848 (0.1364981108683733) | > loader_time: 0.0048 (0.004648270953774282)  --> TIME: 2025-02-25 08:07:29 -- STEP: 31566/36938 -- GLOBAL_STEP: 696450 | > loss_text_ce: 0.03642725571990013 (0.035132455292840016) | > loss_mel_ce: 3.2124617099761963 (nan) | > loss: 0.01289241760969162 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.1364956031331744) | > loader_time: 0.0049 (0.004648307253031463)  --> TIME: 2025-02-25 08:07:39 -- STEP: 31616/36938 -- GLOBAL_STEP: 696500 | > loss_text_ce: 0.03789859637618065 (0.03513261282346755) | > loss_mel_ce: 3.250347852706909 (nan) | > loss: 0.013048597611486912 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.1364944348496333) | > loader_time: 0.0046 (0.004648387756364544)  --> TIME: 2025-02-25 08:07:48 -- STEP: 31666/36938 -- GLOBAL_STEP: 696550 | > loss_text_ce: 0.03880387544631958 (0.035131927714725286) | > loss_mel_ce: 2.856473445892334 (nan) | > loss: 0.011489195749163628 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13648888600690906) | > loader_time: 0.0045 (0.004648273911093951)  --> TIME: 2025-02-25 08:07:57 -- STEP: 31716/36938 -- GLOBAL_STEP: 696600 | > loss_text_ce: 0.042896732687950134 (0.035131975435720664) | > loss_mel_ce: 2.9554686546325684 (nan) | > loss: 0.011898276396095753 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13648819612936425) | > loader_time: 0.0047 (0.004648369879210814)  --> TIME: 2025-02-25 08:08:06 -- STEP: 31766/36938 -- GLOBAL_STEP: 696650 | > loss_text_ce: 0.038799673318862915 (0.03513179601626971) | > loss_mel_ce: 2.666050910949707 (nan) | > loss: 0.010733534581959248 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13648408073806348) | > loader_time: 0.0045 (0.004648503080045123)  --> TIME: 2025-02-25 08:08:15 -- STEP: 31816/36938 -- GLOBAL_STEP: 696700 | > loss_text_ce: 0.040111616253852844 (0.035130457842292) | > loss_mel_ce: 2.2281854152679443 (nan) | > loss: 0.009001178666949272 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13647930787560944) | > loader_time: 0.0046 (0.004648520924321421)  --> TIME: 2025-02-25 08:08:25 -- STEP: 31866/36938 -- GLOBAL_STEP: 696750 | > loss_text_ce: 0.028481069952249527 (0.03513049985550022) | > loss_mel_ce: 3.155813217163086 (nan) | > loss: 0.012636088766157627 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.13648008427092806) | > loader_time: 0.0045 (0.004648424366552529)  --> TIME: 2025-02-25 08:08:33 -- STEP: 31916/36938 -- GLOBAL_STEP: 696800 | > loss_text_ce: 0.027402261272072792 (0.03513268523363763) | > loss_mel_ce: 2.4299161434173584 (nan) | > loss: 0.00975126400589943 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.13646927733277198) | > loader_time: 0.0045 (0.00464840406821117)  --> TIME: 2025-02-25 08:08:43 -- STEP: 31966/36938 -- GLOBAL_STEP: 696850 | > loss_text_ce: 0.03985132277011871 (0.035131141897521805) | > loss_mel_ce: 2.615358352661133 (nan) | > loss: 0.010536547750234604 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13646888477083324) | > loader_time: 0.0047 (0.004648403546198996)  --> TIME: 2025-02-25 08:08:52 -- STEP: 32016/36938 -- GLOBAL_STEP: 696900 | > loss_text_ce: 0.03747697174549103 (0.03512962522215688) | > loss_mel_ce: 0.01260200422257185 (nan) | > loss: 0.00019872610573656857 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1364705316472913) | > loader_time: 0.0045 (0.004648330716834191)  --> TIME: 2025-02-25 08:09:01 -- STEP: 32066/36938 -- GLOBAL_STEP: 696950 | > loss_text_ce: 0.033882301300764084 (0.03512922983674341) | > loss_mel_ce: 2.6781704425811768 (nan) | > loss: 0.010762115009129047 (nan) | > current_lr: 5e-06 | > step_time: 0.1458 (0.13646299052519414) | > loader_time: 0.005 (0.004648236530075303)  --> TIME: 2025-02-25 08:09:10 -- STEP: 32116/36938 -- GLOBAL_STEP: 697000 | > loss_text_ce: 0.03814476728439331 (0.035131147582108876) | > loss_mel_ce: 2.971391201019287 (nan) | > loss: 0.011942603625357151 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13646045868743195) | > loader_time: 0.0046 (0.004648210511199857)  --> TIME: 2025-02-25 08:09:20 -- STEP: 32166/36938 -- GLOBAL_STEP: 697050 | > loss_text_ce: 0.028224589303135872 (0.03512931598416663) | > loss_mel_ce: 3.12998628616333 (nan) | > loss: 0.012532584369182587 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13645868615517576) | > loader_time: 0.0046 (0.0046481969588832986)  --> TIME: 2025-02-25 08:09:29 -- STEP: 32216/36938 -- GLOBAL_STEP: 697100 | > loss_text_ce: 0.03250563144683838 (0.03512843968539258) | > loss_mel_ce: 2.8356270790100098 (nan) | > loss: 0.011381478980183601 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13645966708112378) | > loader_time: 0.0045 (0.004648114741218569)  --> TIME: 2025-02-25 08:09:38 -- STEP: 32266/36938 -- GLOBAL_STEP: 697150 | > loss_text_ce: 0.03978198394179344 (0.0351287658576983) | > loss_mel_ce: 2.604102611541748 (nan) | > loss: 0.010491606779396534 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.1364579774217188) | > loader_time: 0.0046 (0.004648075923660758)  --> TIME: 2025-02-25 08:09:48 -- STEP: 32316/36938 -- GLOBAL_STEP: 697200 | > loss_text_ce: 0.03459351137280464 (0.035126861903875595) | > loss_mel_ce: 3.236715316772461 (nan) | > loss: 0.012981384992599487 (nan) | > current_lr: 5e-06 | > step_time: 0.1625 (0.1364586719679843) | > loader_time: 0.0046 (0.004648085579834944)  --> TIME: 2025-02-25 08:09:57 -- STEP: 32366/36938 -- GLOBAL_STEP: 697250 | > loss_text_ce: 0.0294382032006979 (0.035125674758085174) | > loss_mel_ce: 2.8781654834747314 (nan) | > loss: 0.011538110673427582 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13646102727273382) | > loader_time: 0.0046 (0.00464813619242502)  --> TIME: 2025-02-25 08:10:06 -- STEP: 32416/36938 -- GLOBAL_STEP: 697300 | > loss_text_ce: 0.038539331406354904 (0.035124454096484736) | > loss_mel_ce: 3.323084831237793 (nan) | > loss: 0.01333977933973074 (nan) | > current_lr: 5e-06 | > step_time: 0.2003 (0.1364654749117024) | > loader_time: 0.0049 (0.00464821234713184)  --> TIME: 2025-02-25 08:10:16 -- STEP: 32466/36938 -- GLOBAL_STEP: 697350 | > loss_text_ce: 0.035986389964818954 (0.0351257970346521) | > loss_mel_ce: 2.5370090007781982 (nan) | > loss: 0.010210299864411354 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13646094577116882) | > loader_time: 0.0046 (0.004648219178316734)  --> TIME: 2025-02-25 08:10:25 -- STEP: 32516/36938 -- GLOBAL_STEP: 697400 | > loss_text_ce: 0.02664695493876934 (0.03512487045825749) | > loss_mel_ce: 3.682175874710083 (nan) | > loss: 0.014717550948262215 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.1364658411350726) | > loader_time: 0.0045 (0.004648180835903135)  --> TIME: 2025-02-25 08:10:35 -- STEP: 32566/36938 -- GLOBAL_STEP: 697450 | > loss_text_ce: 0.034494005143642426 (0.03512582377339211) | > loss_mel_ce: 2.957606554031372 (nan) | > loss: 0.011873415671288967 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.13647252077816982) | > loader_time: 0.0048 (0.004648357338762716)  --> TIME: 2025-02-25 08:10:44 -- STEP: 32616/36938 -- GLOBAL_STEP: 697500 | > loss_text_ce: 0.03488114848732948 (0.03512419609695379) | > loss_mel_ce: 2.8319900035858154 (nan) | > loss: 0.011376473121345043 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.13647615583556305) | > loader_time: 0.005 (0.004648346306794055)  --> TIME: 2025-02-25 08:10:53 -- STEP: 32666/36938 -- GLOBAL_STEP: 697550 | > loss_text_ce: 0.022791091352701187 (0.03512169736763271) | > loss_mel_ce: 2.3123040199279785 (nan) | > loss: 0.009266250766813755 (nan) | > current_lr: 5e-06 | > step_time: 0.1041 (0.13647659379036103) | > loader_time: 0.0045 (0.004648337607680635)  --> TIME: 2025-02-25 08:11:03 -- STEP: 32716/36938 -- GLOBAL_STEP: 697600 | > loss_text_ce: 0.03686331585049629 (0.03512122007062972) | > loss_mel_ce: 2.842932939529419 (nan) | > loss: 0.011427763849496841 (nan) | > current_lr: 5e-06 | > step_time: 0.1556 (0.13647994408908795) | > loader_time: 0.0046 (0.004648332724668586)  --> TIME: 2025-02-25 08:11:12 -- STEP: 32766/36938 -- GLOBAL_STEP: 697650 | > loss_text_ce: 0.03850338235497475 (0.035120782641809914) | > loss_mel_ce: 2.6536865234375 (nan) | > loss: 0.010683294385671616 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.1364827082138643) | > loader_time: 0.0048 (0.004648473282724203)  --> TIME: 2025-02-25 08:11:21 -- STEP: 32816/36938 -- GLOBAL_STEP: 697700 | > loss_text_ce: 0.03334479033946991 (0.03512300679108038) | > loss_mel_ce: 2.765238046646118 (nan) | > loss: 0.011105488054454327 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13647689680858932) | > loader_time: 0.0053 (0.004648518432994283)  --> TIME: 2025-02-25 08:11:31 -- STEP: 32866/36938 -- GLOBAL_STEP: 697750 | > loss_text_ce: 0.029069675132632256 (0.03512339619704308) | > loss_mel_ce: 3.3752052783966064 (nan) | > loss: 0.01350902859121561 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.1364793918343581) | > loader_time: 0.0048 (0.004648470054516893)  --> TIME: 2025-02-25 08:11:40 -- STEP: 32916/36938 -- GLOBAL_STEP: 697800 | > loss_text_ce: 0.035915080457925797 (0.035123324084537944) | > loss_mel_ce: 2.9595324993133545 (nan) | > loss: 0.011886697262525558 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13647322043249466) | > loader_time: 0.0045 (0.004648352476208765)  --> TIME: 2025-02-25 08:11:49 -- STEP: 32966/36938 -- GLOBAL_STEP: 697850 | > loss_text_ce: 0.02642429806292057 (0.035122712272296484) | > loss_mel_ce: 2.3391060829162598 (nan) | > loss: 0.00938702654093504 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13647093812954786) | > loader_time: 0.0046 (0.004648287117079838)  --> TIME: 2025-02-25 08:11:58 -- STEP: 33016/36938 -- GLOBAL_STEP: 697900 | > loss_text_ce: 0.03622720018029213 (0.03512155151283859) | > loss_mel_ce: 3.1237740516662598 (nan) | > loss: 0.012539688497781754 (nan) | > current_lr: 5e-06 | > step_time: 0.1734 (0.13646299587140587) | > loader_time: 0.0048 (0.004648151793721397)  --> TIME: 2025-02-25 08:12:08 -- STEP: 33066/36938 -- GLOBAL_STEP: 697950 | > loss_text_ce: 0.03465842828154564 (0.035119915780095615) | > loss_mel_ce: 2.4785022735595703 (nan) | > loss: 0.009972860105335712 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.1364738363640806) | > loader_time: 0.0046 (0.004648232628744971)  --> TIME: 2025-02-25 08:12:17 -- STEP: 33116/36938 -- GLOBAL_STEP: 698000 | > loss_text_ce: 0.03659011796116829 (0.0351199549974654) | > loss_mel_ce: 3.1577444076538086 (nan) | > loss: 0.012675931677222252 (nan) | > current_lr: 5e-06 | > step_time: 0.1799 (0.13647447165579987) | > loader_time: 0.0047 (0.004648204168874234)  --> TIME: 2025-02-25 08:12:26 -- STEP: 33166/36938 -- GLOBAL_STEP: 698050 | > loss_text_ce: 0.03620743006467819 (0.03511938861791022) | > loss_mel_ce: 3.593404531478882 (nan) | > loss: 0.014403223060071468 (nan) | > current_lr: 5e-06 | > step_time: 0.1833 (0.1364708052090974) | > loader_time: 0.0047 (0.004648069014690438)  --> TIME: 2025-02-25 08:12:36 -- STEP: 33216/36938 -- GLOBAL_STEP: 698100 | > loss_text_ce: 0.03445877134799957 (0.03511943962258007) | > loss_mel_ce: 1.390857458114624 (nan) | > loss: 0.005656016990542412 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.1364743743228191) | > loader_time: 0.0052 (0.00464805026102617)  --> TIME: 2025-02-25 08:12:45 -- STEP: 33266/36938 -- GLOBAL_STEP: 698150 | > loss_text_ce: 0.030176792293787003 (0.035119744117638865) | > loss_mel_ce: 2.7638490200042725 (nan) | > loss: 0.011087404564023018 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13647620210513894) | > loader_time: 0.0045 (0.004648045697129365)  --> TIME: 2025-02-25 08:12:55 -- STEP: 33316/36938 -- GLOBAL_STEP: 698200 | > loss_text_ce: 0.036794956773519516 (0.035120249418130405) | > loss_mel_ce: 3.48823881149292 (nan) | > loss: 0.01398822944611311 (nan) | > current_lr: 5e-06 | > step_time: 0.1852 (0.13648371501678166) | > loader_time: 0.0048 (0.004647973012002462)  --> TIME: 2025-02-25 08:13:04 -- STEP: 33366/36938 -- GLOBAL_STEP: 698250 | > loss_text_ce: 0.03816532343626022 (0.03511682752452755) | > loss_mel_ce: 1.9714752435684204 (nan) | > loss: 0.007974764332175255 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13648730488923233) | > loader_time: 0.0045 (0.004647913942632573)  --> TIME: 2025-02-25 08:13:14 -- STEP: 33416/36938 -- GLOBAL_STEP: 698300 | > loss_text_ce: 0.04709094390273094 (0.03511717597991778) | > loss_mel_ce: 2.9706220626831055 (nan) | > loss: 0.011975052766501904 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13649827389589148) | > loader_time: 0.0046 (0.004647994315421603)  --> TIME: 2025-02-25 08:13:23 -- STEP: 33466/36938 -- GLOBAL_STEP: 698350 | > loss_text_ce: 0.051205139607191086 (0.03511721110664911) | > loss_mel_ce: 2.3931710720062256 (nan) | > loss: 0.009699906222522259 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.1364987678823619) | > loader_time: 0.0044 (0.004647914588055989)  --> TIME: 2025-02-25 08:13:33 -- STEP: 33516/36938 -- GLOBAL_STEP: 698400 | > loss_text_ce: 0.031947553157806396 (0.03511588883204217) | > loss_mel_ce: 3.263566017150879 (nan) | > loss: 0.01307743601500988 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.13649659035014816) | > loader_time: 0.0047 (0.004647807661510191)  --> TIME: 2025-02-25 08:13:42 -- STEP: 33566/36938 -- GLOBAL_STEP: 698450 | > loss_text_ce: 0.034502699971199036 (0.035116499855334056) | > loss_mel_ce: 2.8510820865631104 (nan) | > loss: 0.011450733989477158 (nan) | > current_lr: 5e-06 | > step_time: 0.1445 (0.13649562019873554) | > loader_time: 0.0045 (0.004647695122533392)  --> TIME: 2025-02-25 08:13:51 -- STEP: 33616/36938 -- GLOBAL_STEP: 698500 | > loss_text_ce: 0.04194694384932518 (0.0351192433361443) | > loss_mel_ce: 2.9685776233673096 (nan) | > loss: 0.011946526356041431 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.1364964346823826) | > loader_time: 0.0064 (0.0046476785169789325)  --> TIME: 2025-02-25 08:14:00 -- STEP: 33666/36938 -- GLOBAL_STEP: 698550 | > loss_text_ce: 0.04040825366973877 (0.03511777522852061) | > loss_mel_ce: 2.666748523712158 (nan) | > loss: 0.01074268575757742 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.13649554303755596) | > loader_time: 0.0047 (0.004647645792816365)  --> TIME: 2025-02-25 08:14:10 -- STEP: 33716/36938 -- GLOBAL_STEP: 698600 | > loss_text_ce: 0.041864316910505295 (0.03511896920750083) | > loss_mel_ce: 3.6875 (nan) | > loss: 0.014799065887928009 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13650104925009815) | > loader_time: 0.0045 (0.004647635638551194)  --> TIME: 2025-02-25 08:14:19 -- STEP: 33766/36938 -- GLOBAL_STEP: 698650 | > loss_text_ce: 0.04093218222260475 (0.035117429751676296) | > loss_mel_ce: 3.3351588249206543 (nan) | > loss: 0.013397186994552612 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.1364979708935412) | > loader_time: 0.0045 (0.004647541595481128)  --> TIME: 2025-02-25 08:14:29 -- STEP: 33816/36938 -- GLOBAL_STEP: 698700 | > loss_text_ce: 0.03672277182340622 (0.035116982869477904) | > loss_mel_ce: 3.1840150356292725 (nan) | > loss: 0.012780705466866493 (nan) | > current_lr: 5e-06 | > step_time: 0.1506 (0.13650480820857877) | > loader_time: 0.0046 (0.00464753295079681)  --> TIME: 2025-02-25 08:14:38 -- STEP: 33866/36938 -- GLOBAL_STEP: 698750 | > loss_text_ce: 0.03285258635878563 (0.03511553580352914) | > loss_mel_ce: 3.6453065872192383 (nan) | > loss: 0.014595870859920979 (nan) | > current_lr: 5e-06 | > step_time: 0.1599 (0.13650503189827096) | > loader_time: 0.0045 (0.004647459851754212)  --> TIME: 2025-02-25 08:14:47 -- STEP: 33916/36938 -- GLOBAL_STEP: 698800 | > loss_text_ce: 0.03895043209195137 (0.035115090787166105) | > loss_mel_ce: 1.6981765031814575 (nan) | > loss: 0.0068933614529669285 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13650341423520243) | > loader_time: 0.0046 (0.00464740678490523)  --> TIME: 2025-02-25 08:14:57 -- STEP: 33966/36938 -- GLOBAL_STEP: 698850 | > loss_text_ce: 0.03512544184923172 (0.03511290741488956) | > loss_mel_ce: 3.737060546875 (nan) | > loss: 0.0149689931422472 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13651176645359364) | > loader_time: 0.0045 (0.0046473508910759975)  --> TIME: 2025-02-25 08:15:06 -- STEP: 34016/36938 -- GLOBAL_STEP: 698900 | > loss_text_ce: 0.03179355710744858 (0.035114107466062416) | > loss_mel_ce: 2.321495532989502 (nan) | > loss: 0.00933844968676567 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.1365099290731027) | > loader_time: 0.0047 (0.0046472326061238935)  --> TIME: 2025-02-25 08:15:15 -- STEP: 34066/36938 -- GLOBAL_STEP: 698950 | > loss_text_ce: 0.01637118123471737 (0.03511488728881591) | > loss_mel_ce: 1.3103293180465698 (nan) | > loss: 0.005264684557914734 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13650047952651548) | > loader_time: 0.0045 (0.00464709339926823)  --> TIME: 2025-02-25 08:15:25 -- STEP: 34116/36938 -- GLOBAL_STEP: 699000 | > loss_text_ce: 0.03980028256773949 (0.03511483814682647) | > loss_mel_ce: 2.7879960536956787 (nan) | > loss: 0.011221414431929588 (nan) | > current_lr: 5e-06 | > step_time: 0.1842 (0.1365052831104447) | > loader_time: 0.0047 (0.004646988620315271)  --> TIME: 2025-02-25 08:15:34 -- STEP: 34166/36938 -- GLOBAL_STEP: 699050 | > loss_text_ce: 0.031621333211660385 (0.03511476810752916) | > loss_mel_ce: 0.988616943359375 (nan) | > loss: 0.004048564936965704 (nan) | > current_lr: 5e-06 | > step_time: 0.1267 (0.1365063554026548) | > loader_time: 0.0046 (0.004646935521848016)  --> TIME: 2025-02-25 08:15:44 -- STEP: 34216/36938 -- GLOBAL_STEP: 699100 | > loss_text_ce: 0.0350283719599247 (0.03511382445731896) | > loss_mel_ce: 3.373110771179199 (nan) | > loss: 0.013524362817406654 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13651899414723537) | > loader_time: 0.0046 (0.004647041052781956)  --> TIME: 2025-02-25 08:15:54 -- STEP: 34266/36938 -- GLOBAL_STEP: 699150 | > loss_text_ce: 0.049493879079818726 (0.03511291315896734) | > loss_mel_ce: 2.9853360652923584 (nan) | > loss: 0.01204297598451376 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13653111736835438) | > loader_time: 0.0046 (0.004647159829682705)  --> TIME: 2025-02-25 08:16:03 -- STEP: 34316/36938 -- GLOBAL_STEP: 699200 | > loss_text_ce: 0.03421575203537941 (0.035113430297211194) | > loss_mel_ce: 3.58203125 (nan) | > loss: 0.01435018703341484 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13653405823064402) | > loader_time: 0.0045 (0.004647135456648585)  --> TIME: 2025-02-25 08:16:13 -- STEP: 34366/36938 -- GLOBAL_STEP: 699250 | > loss_text_ce: 0.03960754722356796 (0.03511300719772025) | > loss_mel_ce: 1.4886106252670288 (nan) | > loss: 0.006064358167350292 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1365446540423134) | > loader_time: 0.0045 (0.004647160030136995)  --> TIME: 2025-02-25 08:16:22 -- STEP: 34416/36938 -- GLOBAL_STEP: 699300 | > loss_text_ce: 0.027253801003098488 (0.0351104057844639) | > loss_mel_ce: 3.3493287563323975 (nan) | > loss: 0.013399138115346432 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.1365433262357143) | > loader_time: 0.0046 (0.004647126029059812)  --> TIME: 2025-02-25 08:16:31 -- STEP: 34466/36938 -- GLOBAL_STEP: 699350 | > loss_text_ce: 0.03860781714320183 (0.035112514705515935) | > loss_mel_ce: 1.9741950035095215 (nan) | > loss: 0.007987312972545624 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13654244330688378) | > loader_time: 0.0045 (0.004647073304114295)  --> TIME: 2025-02-25 08:16:40 -- STEP: 34516/36938 -- GLOBAL_STEP: 699400 | > loss_text_ce: 0.034486379474401474 (0.03511380207044279) | > loss_mel_ce: 2.6627120971679688 (nan) | > loss: 0.01070316880941391 (nan) | > current_lr: 5e-06 | > step_time: 0.1086 (0.136533131963035) | > loader_time: 0.0044 (0.004646943464852773)  --> TIME: 2025-02-25 08:16:50 -- STEP: 34566/36938 -- GLOBAL_STEP: 699450 | > loss_text_ce: 0.03623759746551514 (0.03511448491795069) | > loss_mel_ce: 1.7157788276672363 (nan) | > loss: 0.006952446419745684 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13653125655614431) | > loader_time: 0.0045 (0.004647046860421912)  --> TIME: 2025-02-25 08:16:59 -- STEP: 34616/36938 -- GLOBAL_STEP: 699500 | > loss_text_ce: 0.047814417630434036 (0.03511598804691313) | > loss_mel_ce: 2.766918659210205 (nan) | > loss: 0.011169576086103916 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13653440919846907) | > loader_time: 0.0053 (0.004647035665701632)  --> TIME: 2025-02-25 08:17:08 -- STEP: 34666/36938 -- GLOBAL_STEP: 699550 | > loss_text_ce: 0.02635769359767437 (0.035114066370515395) | > loss_mel_ce: 3.0637123584747314 (nan) | > loss: 0.012262183241546154 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.1365346966272316) | > loader_time: 0.0047 (0.004647100266819909)  --> TIME: 2025-02-25 08:17:17 -- STEP: 34716/36938 -- GLOBAL_STEP: 699600 | > loss_text_ce: 0.03294232115149498 (0.035115789704132046) | > loss_mel_ce: 2.067706346511841 (nan) | > loss: 0.008335907943546772 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13653143143953195) | > loader_time: 0.0045 (0.004647105269504046)  --> TIME: 2025-02-25 08:17:27 -- STEP: 34766/36938 -- GLOBAL_STEP: 699650 | > loss_text_ce: 0.03312086686491966 (0.03511266474254343) | > loss_mel_ce: 0.7877976298332214 (nan) | > loss: 0.0032576133962720633 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13653826771853284) | > loader_time: 0.0048 (0.004647222609279393)  --> TIME: 2025-02-25 08:17:36 -- STEP: 34816/36938 -- GLOBAL_STEP: 699700 | > loss_text_ce: 0.04076952859759331 (0.03511350200843124) | > loss_mel_ce: 1.6440719366073608 (nan) | > loss: 0.006685879547148943 (nan) | > current_lr: 5e-06 | > step_time: 0.1121 (0.13654061314403254) | > loader_time: 0.0044 (0.004647294572992806)  --> TIME: 2025-02-25 08:17:46 -- STEP: 34866/36938 -- GLOBAL_STEP: 699750 | > loss_text_ce: 0.04330912604928017 (0.035115110008184576) | > loss_mel_ce: 3.4238617420196533 (nan) | > loss: 0.013758615590631962 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.1365490335130657) | > loader_time: 0.0046 (0.004647302147526073)  --> TIME: 2025-02-25 08:17:55 -- STEP: 34916/36938 -- GLOBAL_STEP: 699800 | > loss_text_ce: 0.02390921302139759 (0.03511487404878393) | > loss_mel_ce: 2.3508152961730957 (nan) | > loss: 0.009423511102795601 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13654607481147366) | > loader_time: 0.0046 (0.0046473120834590095)  --> TIME: 2025-02-25 08:18:05 -- STEP: 34966/36938 -- GLOBAL_STEP: 699850 | > loss_text_ce: 0.03500058874487877 (0.03511465254445858) | > loss_mel_ce: 2.7884163856506348 (nan) | > loss: 0.011204035952687263 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13654736952139812) | > loader_time: 0.0046 (0.004647374084951377)  --> TIME: 2025-02-25 08:18:14 -- STEP: 35016/36938 -- GLOBAL_STEP: 699900 | > loss_text_ce: 0.03130204975605011 (0.03511442841946785) | > loss_mel_ce: 3.0701069831848145 (nan) | > loss: 0.012307179160416126 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.13654981538234534) | > loader_time: 0.0054 (0.004647426431462842)  --> TIME: 2025-02-25 08:18:23 -- STEP: 35066/36938 -- GLOBAL_STEP: 699950 | > loss_text_ce: 0.039874833077192307 (0.035112601174748806) | > loss_mel_ce: 4.302455425262451 (nan) | > loss: 0.017231471836566925 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13655085867037117) | > loader_time: 0.0046 (0.004647372249378523)  --> TIME: 2025-02-25 08:18:33 -- STEP: 35116/36938 -- GLOBAL_STEP: 700000 | > loss_text_ce: 0.03256342187523842 (0.035110647906217936) | > loss_mel_ce: 2.732063055038452 (nan) | > loss: 0.01097074057906866 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.136547688505537) | > loader_time: 0.0046 (0.004647321161423752) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_700000.pth  --> TIME: 2025-02-25 08:18:48 -- STEP: 35166/36938 -- GLOBAL_STEP: 700050 | > loss_text_ce: 0.02898029051721096 (0.0351118002194562) | > loss_mel_ce: 1.8653088808059692 (nan) | > loss: 0.007517020683735609 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13655321334860895) | > loader_time: 0.0045 (0.004647387041544165)  --> TIME: 2025-02-25 08:18:57 -- STEP: 35216/36938 -- GLOBAL_STEP: 700100 | > loss_text_ce: 0.043489862233400345 (0.03511174127213807) | > loss_mel_ce: 2.897632598876953 (nan) | > loss: 0.011671122163534164 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13654680158105031) | > loader_time: 0.0045 (0.004647349232112972)  --> TIME: 2025-02-25 08:19:06 -- STEP: 35266/36938 -- GLOBAL_STEP: 700150 | > loss_text_ce: 0.03792340308427811 (0.03511065738601017) | > loss_mel_ce: 3.5722877979278564 (nan) | > loss: 0.01432623527944088 (nan) | > current_lr: 5e-06 | > step_time: 0.1773 (0.136537778622941) | > loader_time: 0.0048 (0.004647307656082146)  --> TIME: 2025-02-25 08:19:15 -- STEP: 35316/36938 -- GLOBAL_STEP: 700200 | > loss_text_ce: 0.041522808372974396 (0.0351097563234769) | > loss_mel_ce: 3.366424083709717 (nan) | > loss: 0.01352359913289547 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13653320765384486) | > loader_time: 0.0046 (0.004647269330244542)  --> TIME: 2025-02-25 08:19:25 -- STEP: 35366/36938 -- GLOBAL_STEP: 700250 | > loss_text_ce: 0.030817624181509018 (0.035109813199415835) | > loss_mel_ce: 3.056133270263672 (nan) | > loss: 0.01224980503320694 (nan) | > current_lr: 5e-06 | > step_time: 0.1797 (0.1365354515231644) | > loader_time: 0.0048 (0.004647260525778998)  --> TIME: 2025-02-25 08:19:34 -- STEP: 35416/36938 -- GLOBAL_STEP: 700300 | > loss_text_ce: 0.03687276318669319 (0.035110078531122774) | > loss_mel_ce: 3.120502233505249 (nan) | > loss: 0.012529266998171806 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13653973174439735) | > loader_time: 0.0047 (0.0046472754762812065)  --> TIME: 2025-02-25 08:19:44 -- STEP: 35466/36938 -- GLOBAL_STEP: 700350 | > loss_text_ce: 0.026803787797689438 (0.03510762393255169) | > loss_mel_ce: 2.186230421066284 (nan) | > loss: 0.008781882002949715 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13654440580564892) | > loader_time: 0.0046 (0.004647274990207642)  --> TIME: 2025-02-25 08:19:53 -- STEP: 35516/36938 -- GLOBAL_STEP: 700400 | > loss_text_ce: 0.02742709591984749 (0.035107219218877515) | > loss_mel_ce: 2.4993696212768555 (nan) | > loss: 0.010026971809566021 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13654602300960902) | > loader_time: 0.0044 (0.004647225910162719)  --> TIME: 2025-02-25 08:20:02 -- STEP: 35566/36938 -- GLOBAL_STEP: 700450 | > loss_text_ce: 0.04019327461719513 (0.035107353148437534) | > loss_mel_ce: 2.8393471240997314 (nan) | > loss: 0.01142674870789051 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13654472226870695) | > loader_time: 0.0045 (0.004647145360860755)  --> TIME: 2025-02-25 08:20:12 -- STEP: 35616/36938 -- GLOBAL_STEP: 700500 | > loss_text_ce: 0.03439487889409065 (0.035107279606110306) | > loss_mel_ce: 2.9009060859680176 (nan) | > loss: 0.011648020707070827 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.13654131475109635) | > loader_time: 0.0048 (0.004647103107307597)  --> TIME: 2025-02-25 08:20:21 -- STEP: 35666/36938 -- GLOBAL_STEP: 700550 | > loss_text_ce: 0.04412778839468956 (0.03510595690009373) | > loss_mel_ce: 3.1655054092407227 (nan) | > loss: 0.012736639939248562 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.13653725898720984) | > loader_time: 0.0046 (0.004647025890614533)  --> TIME: 2025-02-25 08:20:30 -- STEP: 35716/36938 -- GLOBAL_STEP: 700600 | > loss_text_ce: 0.03590629622340202 (0.03510690145665473) | > loss_mel_ce: 3.0504679679870605 (nan) | > loss: 0.012247517704963684 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1365358473165441) | > loader_time: 0.0045 (0.004646989069349204)  --> TIME: 2025-02-25 08:20:40 -- STEP: 35766/36938 -- GLOBAL_STEP: 700650 | > loss_text_ce: 0.033411331474781036 (0.03510583609887643) | > loss_mel_ce: 2.7061190605163574 (nan) | > loss: 0.010871152393519878 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13654820188302424) | > loader_time: 0.0045 (0.004647057361599592)  --> TIME: 2025-02-25 08:20:49 -- STEP: 35816/36938 -- GLOBAL_STEP: 700700 | > loss_text_ce: 0.038983408361673355 (0.03510666072395364) | > loss_mel_ce: 2.73539662361145 (nan) | > loss: 0.011009445413947105 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13655416626522215) | > loader_time: 0.0049 (0.004647080616570764)  --> TIME: 2025-02-25 08:20:59 -- STEP: 35866/36938 -- GLOBAL_STEP: 700750 | > loss_text_ce: 0.03469302877783775 (0.03510805607850701) | > loss_mel_ce: 2.8501391410827637 (nan) | > loss: 0.011447747237980366 (nan) | > current_lr: 5e-06 | > step_time: 0.1582 (0.13655933991854724) | > loader_time: 0.0046 (0.00464702896270685)  --> TIME: 2025-02-25 08:21:08 -- STEP: 35916/36938 -- GLOBAL_STEP: 700800 | > loss_text_ce: 0.03940092772245407 (0.03510863300617135) | > loss_mel_ce: 3.357133388519287 (nan) | > loss: 0.013478310778737068 (nan) | > current_lr: 5e-06 | > step_time: 0.1688 (0.13656058076734595) | > loader_time: 0.0048 (0.004647064466547616)  --> TIME: 2025-02-25 08:21:18 -- STEP: 35966/36938 -- GLOBAL_STEP: 700850 | > loss_text_ce: 0.028191596269607544 (0.035107129459231) | > loss_mel_ce: 2.507467746734619 (nan) | > loss: 0.01006214041262865 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13656359640833168) | > loader_time: 0.0045 (0.004647042895426707)  --> TIME: 2025-02-25 08:21:27 -- STEP: 36016/36938 -- GLOBAL_STEP: 700900 | > loss_text_ce: 0.044815097004175186 (0.03510430364658852) | > loss_mel_ce: 0.0479736328125 (nan) | > loss: 0.00036820926470682025 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13656265655658995) | > loader_time: 0.0046 (0.004647047313940583)  --> TIME: 2025-02-25 08:21:36 -- STEP: 36066/36938 -- GLOBAL_STEP: 700950 | > loss_text_ce: 0.032997604459524155 (0.03510400041767307) | > loss_mel_ce: 3.2152717113494873 (nan) | > loss: 0.01288995798677206 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13656316150320807) | > loader_time: 0.0046 (0.004647090425376583)  --> TIME: 2025-02-25 08:21:46 -- STEP: 36116/36938 -- GLOBAL_STEP: 701000 | > loss_text_ce: 0.04469534382224083 (0.03510458079033251) | > loss_mel_ce: 2.8432083129882812 (nan) | > loss: 0.011459936387836933 (nan) | > current_lr: 5e-06 | > step_time: 0.1732 (0.13657167407018944) | > loader_time: 0.0047 (0.004647096442627039)  --> TIME: 2025-02-25 08:21:56 -- STEP: 36166/36938 -- GLOBAL_STEP: 701050 | > loss_text_ce: 0.035399749875068665 (0.03510381508773014) | > loss_mel_ce: 1.7533531188964844 (nan) | > loss: 0.0070982263423502445 (nan) | > current_lr: 5e-06 | > step_time: 0.1622 (0.13657614887205619) | > loader_time: 0.0048 (0.0046470659875978)  --> TIME: 2025-02-25 08:22:05 -- STEP: 36216/36938 -- GLOBAL_STEP: 701100 | > loss_text_ce: 0.029629644006490707 (0.0351040243862243) | > loss_mel_ce: 1.5979598760604858 (nan) | > loss: 0.006458688527345657 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13657165038446023) | > loader_time: 0.0045 (0.004647155760770779)  --> TIME: 2025-02-25 08:22:14 -- STEP: 36266/36938 -- GLOBAL_STEP: 701150 | > loss_text_ce: 0.03456205874681473 (0.035102916297240584) | > loss_mel_ce: 2.127622604370117 (nan) | > loss: 0.008580098859965801 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13657167265131512) | > loader_time: 0.0045 (0.004647160275906926)  --> TIME: 2025-02-25 08:22:23 -- STEP: 36316/36938 -- GLOBAL_STEP: 701200 | > loss_text_ce: 0.03436652570962906 (0.03510385322768612) | > loss_mel_ce: 2.7245140075683594 (nan) | > loss: 0.010947939939796925 (nan) | > current_lr: 5e-06 | > step_time: 0.1886 (0.1365726615563529) | > loader_time: 0.0049 (0.004647168225293551)  --> TIME: 2025-02-25 08:22:33 -- STEP: 36366/36938 -- GLOBAL_STEP: 701250 | > loss_text_ce: 0.04720349982380867 (0.035103903950328905) | > loss_mel_ce: 5.40625 (nan) | > loss: 0.02164069004356861 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13657371363073054) | > loader_time: 0.0045 (0.004647156405893426)  --> TIME: 2025-02-25 08:22:42 -- STEP: 36416/36938 -- GLOBAL_STEP: 701300 | > loss_text_ce: 0.0331244133412838 (0.03510329525054695) | > loss_mel_ce: 2.2455530166625977 (nan) | > loss: 0.009042371064424515 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13657558194127492) | > loader_time: 0.0047 (0.004647345601307705)  --> TIME: 2025-02-25 08:22:51 -- STEP: 36466/36938 -- GLOBAL_STEP: 701350 | > loss_text_ce: 0.03127063438296318 (0.035103882818666304) | > loss_mel_ce: 2.9617011547088623 (nan) | > loss: 0.011876873672008514 (nan) | > current_lr: 5e-06 | > step_time: 0.1892 (0.13657717904708158) | > loader_time: 0.0048 (0.00464736660816271)  --> TIME: 2025-02-25 08:23:01 -- STEP: 36516/36938 -- GLOBAL_STEP: 701400 | > loss_text_ce: 0.03167428821325302 (0.035104010898414174) | > loss_mel_ce: 2.805738687515259 (nan) | > loss: 0.011259576305747032 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13657714578483462) | > loader_time: 0.0045 (0.004647334135953753)  --> TIME: 2025-02-25 08:23:10 -- STEP: 36566/36938 -- GLOBAL_STEP: 701450 | > loss_text_ce: 0.04020871967077255 (0.03510304938936915) | > loss_mel_ce: 1.4789448976516724 (nan) | > loss: 0.006028387695550919 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13657788907465027) | > loader_time: 0.0046 (0.004647280998670234)  --> TIME: 2025-02-25 08:23:20 -- STEP: 36616/36938 -- GLOBAL_STEP: 701500 | > loss_text_ce: 0.035999659448862076 (0.03510205723476639) | > loss_mel_ce: 2.798846483230591 (nan) | > loss: 0.011249390430748463 (nan) | > current_lr: 5e-06 | > step_time: 0.1779 (0.136583606788417) | > loader_time: 0.0049 (0.004647278046019355)  --> TIME: 2025-02-25 08:23:29 -- STEP: 36666/36938 -- GLOBAL_STEP: 701550 | > loss_text_ce: 0.044166143983602524 (0.0351023747028842) | > loss_mel_ce: 2.063103199005127 (nan) | > loss: 0.008362180553376675 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13658282713629125) | > loader_time: 0.0044 (0.004647209751868042)  --> TIME: 2025-02-25 08:23:38 -- STEP: 36716/36938 -- GLOBAL_STEP: 701600 | > loss_text_ce: 0.02354101836681366 (0.0351027020018408) | > loss_mel_ce: 1.47548508644104 (nan) | > loss: 0.0059485165402293205 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1365836696085001) | > loader_time: 0.0045 (0.004647178105220186)  --> TIME: 2025-02-25 08:23:47 -- STEP: 36766/36938 -- GLOBAL_STEP: 701650 | > loss_text_ce: 0.04214620962738991 (0.03510158550591958) | > loss_mel_ce: 3.1121599674224854 (nan) | > loss: 0.012517089024186134 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.1365755385817669) | > loader_time: 0.0047 (0.004647074680570127)  --> TIME: 2025-02-25 08:23:57 -- STEP: 36816/36938 -- GLOBAL_STEP: 701700 | > loss_text_ce: 0.04411402717232704 (0.0351020114645147) | > loss_mel_ce: 2.765819549560547 (nan) | > loss: 0.011150531470775604 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13657298233393977) | > loader_time: 0.0047 (0.004647045731803538)  --> TIME: 2025-02-25 08:24:06 -- STEP: 36866/36938 -- GLOBAL_STEP: 701750 | > loss_text_ce: 0.06758951395750046 (0.0351023626161017) | > loss_mel_ce: 1.472049355506897 (nan) | > loss: 0.006109678652137518 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.1365662435504085) | > loader_time: 0.0045 (0.004646993127058307)  --> TIME: 2025-02-25 08:24:15 -- STEP: 36916/36938 -- GLOBAL_STEP: 701800 | > loss_text_ce: 0.035747773945331573 (0.03510245574979449) | > loss_mel_ce: 2.68121337890625 (nan) | > loss: 0.010781592689454556 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13656826588401927) | > loader_time: 0.0046 (0.00464696720886727)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.020373691426645413 (-0.0007985971941806384) | > avg_loss_text_ce: 0.03421285008956301 (-0.00037058052782078243) | > avg_loss_mel_ce: 2.5687388158080617 (-0.015195629974401825) | > avg_loss: 2.602951672112587 (-0.015566205624308882) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_701822.pth  > EPOCH: 20/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-25 08:24:44)   --> TIME: 2025-02-25 08:24:50 -- STEP: 28/36938 -- GLOBAL_STEP: 701850 | > loss_text_ce: 0.03785647079348564 (0.03228148492053151) | > loss_mel_ce: 3.0897116661071777 (2.401511298126674) | > loss: 0.012410986237227917 (0.009657908382775656) | > current_lr: 5e-06 | > step_time: 0.1358 (0.1361553839274815) | > loader_time: 0.0049 (0.0048131006104605535)  --> TIME: 2025-02-25 08:24:59 -- STEP: 78/36938 -- GLOBAL_STEP: 701900 | > loss_text_ce: 0.043273020535707474 (0.035396139137446894) | > loss_mel_ce: 3.063710927963257 (nan) | > loss: 0.012329301796853542 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13569845297397712) | > loader_time: 0.0047 (0.0047707649377676156)  --> TIME: 2025-02-25 08:25:08 -- STEP: 128/36938 -- GLOBAL_STEP: 701950 | > loss_text_ce: 0.03791733831167221 (0.03585120636853391) | > loss_mel_ce: 2.3342630863189697 (nan) | > loss: 0.009413414634764194 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13682295568287375) | > loader_time: 0.0046 (0.004761677235364914)  --> TIME: 2025-02-25 08:25:17 -- STEP: 178/36938 -- GLOBAL_STEP: 702000 | > loss_text_ce: 0.03367428481578827 (0.035810099569348136) | > loss_mel_ce: 2.9927456378936768 (nan) | > loss: 0.012009602971374989 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13613906871067014) | > loader_time: 0.0046 (0.004756675677353079)  --> TIME: 2025-02-25 08:25:26 -- STEP: 228/36938 -- GLOBAL_STEP: 702050 | > loss_text_ce: 0.020435016602277756 (0.035493544997287935) | > loss_mel_ce: 2.766900062561035 (nan) | > loss: 0.01106085442006588 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13490111995161627) | > loader_time: 0.0045 (0.004736119195034629)  --> TIME: 2025-02-25 08:25:36 -- STEP: 278/36938 -- GLOBAL_STEP: 702100 | > loss_text_ce: 0.038329534232616425 (0.03568770156221852) | > loss_mel_ce: 3.2547779083251953 (nan) | > loss: 0.013067888095974922 (nan) | > current_lr: 5e-06 | > step_time: 0.1729 (0.1350582889515719) | > loader_time: 0.0048 (0.004723873927438858)  --> TIME: 2025-02-25 08:25:45 -- STEP: 328/36938 -- GLOBAL_STEP: 702150 | > loss_text_ce: 0.03342416137456894 (0.03586517015426622) | > loss_mel_ce: 3.3609538078308105 (nan) | > loss: 0.013469754718244076 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13625462389573817) | > loader_time: 0.0046 (0.004714273098038464)  --> TIME: 2025-02-25 08:25:54 -- STEP: 378/36938 -- GLOBAL_STEP: 702200 | > loss_text_ce: 0.03425493463873863 (0.03599712649784074) | > loss_mel_ce: 2.852794885635376 (nan) | > loss: 0.011456548236310482 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13571242995993804) | > loader_time: 0.0045 (0.004694882524076592)  --> TIME: 2025-02-25 08:26:04 -- STEP: 428/36938 -- GLOBAL_STEP: 702250 | > loss_text_ce: 0.025887927040457726 (0.03584031473462269) | > loss_mel_ce: 2.694120168685913 (nan) | > loss: 0.010793684050440788 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13572084068137907) | > loader_time: 0.0044 (0.004692207430010644)  --> TIME: 2025-02-25 08:26:13 -- STEP: 478/36938 -- GLOBAL_STEP: 702300 | > loss_text_ce: 0.0418131947517395 (0.03571807555308776) | > loss_mel_ce: 2.8599677085876465 (nan) | > loss: 0.011515003629028797 (nan) | > current_lr: 5e-06 | > step_time: 0.1824 (0.1361874731015959) | > loader_time: 0.005 (0.004685194422510378)  --> TIME: 2025-02-25 08:26:22 -- STEP: 528/36938 -- GLOBAL_STEP: 702350 | > loss_text_ce: 0.03374508023262024 (0.03570062452292239) | > loss_mel_ce: 3.374847412109375 (nan) | > loss: 0.013526161201298237 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13574479971871226) | > loader_time: 0.0047 (0.004678243940526789)  --> TIME: 2025-02-25 08:26:31 -- STEP: 578/36938 -- GLOBAL_STEP: 702400 | > loss_text_ce: 0.0349726676940918 (0.03554738570765229) | > loss_mel_ce: 3.2520265579223633 (nan) | > loss: 0.013043648563325405 (nan) | > current_lr: 5e-06 | > step_time: 0.1573 (0.1354965041665469) | > loader_time: 0.0046 (0.004675686978138857)  --> TIME: 2025-02-25 08:26:41 -- STEP: 628/36938 -- GLOBAL_STEP: 702450 | > loss_text_ce: 0.045165497809648514 (0.035527068768645746) | > loss_mel_ce: 2.442078113555908 (nan) | > loss: 0.0098700150847435 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13581094506439884) | > loader_time: 0.0047 (0.00467537922464359)  --> TIME: 2025-02-25 08:26:51 -- STEP: 678/36938 -- GLOBAL_STEP: 702500 | > loss_text_ce: 0.03739120811223984 (0.03553800646127768) | > loss_mel_ce: 3.2142655849456787 (nan) | > loss: 0.01290340069681406 (nan) | > current_lr: 5e-06 | > step_time: 0.163 (0.1360844458098959) | > loader_time: 0.0046 (0.004675806447819627)  --> TIME: 2025-02-25 08:27:00 -- STEP: 728/36938 -- GLOBAL_STEP: 702550 | > loss_text_ce: 0.03768261522054672 (0.03557833158513434) | > loss_mel_ce: 3.324009895324707 (nan) | > loss: 0.013340050354599953 (nan) | > current_lr: 5e-06 | > step_time: 0.1798 (0.13637335582093876) | > loader_time: 0.0048 (0.0046739895920177126)  --> TIME: 2025-02-25 08:27:09 -- STEP: 778/36938 -- GLOBAL_STEP: 702600 | > loss_text_ce: 0.04577503725886345 (0.03567448037659757) | > loss_mel_ce: 2.7623331546783447 (nan) | > loss: 0.01114328671246767 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13598242302480268) | > loader_time: 0.0046 (0.0046732940526731815)  --> TIME: 2025-02-25 08:27:18 -- STEP: 828/36938 -- GLOBAL_STEP: 702650 | > loss_text_ce: 0.03492385894060135 (0.03569403863714446) | > loss_mel_ce: 3.0271100997924805 (nan) | > loss: 0.012150928378105164 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1354801260330826) | > loader_time: 0.0046 (0.004667382885292533)  --> TIME: 2025-02-25 08:27:27 -- STEP: 878/36938 -- GLOBAL_STEP: 702700 | > loss_text_ce: 0.02553778886795044 (0.035592979085387735) | > loss_mel_ce: 2.330024480819702 (nan) | > loss: 0.009347469545900822 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.1356149982482805) | > loader_time: 0.0046 (0.004665551804735886)  --> TIME: 2025-02-25 08:27:37 -- STEP: 928/36938 -- GLOBAL_STEP: 702750 | > loss_text_ce: 0.03616856038570404 (0.035596279979378434) | > loss_mel_ce: 2.5064921379089355 (nan) | > loss: 0.01008992362767458 (nan) | > current_lr: 5e-06 | > step_time: 0.1825 (0.13587305052527052) | > loader_time: 0.005 (0.004665846711602707)  --> TIME: 2025-02-25 08:27:46 -- STEP: 978/36938 -- GLOBAL_STEP: 702800 | > loss_text_ce: 0.03946100175380707 (0.03553058481480025) | > loss_mel_ce: 3.460463762283325 (nan) | > loss: 0.013888590969145298 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13595640634954087) | > loader_time: 0.0048 (0.004666093180759799)  --> TIME: 2025-02-25 08:27:56 -- STEP: 1028/36938 -- GLOBAL_STEP: 702850 | > loss_text_ce: 0.02797950990498066 (0.03545580370213972) | > loss_mel_ce: 2.961435556411743 (nan) | > loss: 0.011862759478390217 (nan) | > current_lr: 5e-06 | > step_time: 0.1699 (0.1360346930036284) | > loader_time: 0.0048 (0.004665381249750637)  --> TIME: 2025-02-25 08:28:05 -- STEP: 1078/36938 -- GLOBAL_STEP: 702900 | > loss_text_ce: 0.042928047478199005 (0.03541037154510096) | > loss_mel_ce: 3.2382731437683105 (nan) | > loss: 0.01302064023911953 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13629334269295368) | > loader_time: 0.0047 (0.00466503614839685)  --> TIME: 2025-02-25 08:28:15 -- STEP: 1128/36938 -- GLOBAL_STEP: 702950 | > loss_text_ce: 0.03267401084303856 (0.03537913609183801) | > loss_mel_ce: 3.373737096786499 (nan) | > loss: 0.01351750548928976 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.13650077306632447) | > loader_time: 0.0047 (0.004664749118453228)  --> TIME: 2025-02-25 08:28:24 -- STEP: 1178/36938 -- GLOBAL_STEP: 703000 | > loss_text_ce: 0.03839758783578873 (0.03536451596399881) | > loss_mel_ce: 3.3200955390930176 (nan) | > loss: 0.013327354565262794 (nan) | > current_lr: 5e-06 | > step_time: 0.1782 (0.13658050992897686) | > loader_time: 0.0048 (0.004663873002160793)  --> TIME: 2025-02-25 08:28:34 -- STEP: 1228/36938 -- GLOBAL_STEP: 703050 | > loss_text_ce: 0.03169676288962364 (0.03538047380636206) | > loss_mel_ce: 3.2334563732147217 (nan) | > loss: 0.012956957332789898 (nan) | > current_lr: 5e-06 | > step_time: 0.1882 (0.13675958461792528) | > loader_time: 0.0048 (0.004662667112940688)  --> TIME: 2025-02-25 08:28:43 -- STEP: 1278/36938 -- GLOBAL_STEP: 703100 | > loss_text_ce: 0.03517564758658409 (0.03539316896465795) | > loss_mel_ce: 2.3801841735839844 (nan) | > loss: 0.0095847612246871 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13670740926023178) | > loader_time: 0.0045 (0.004660371323706401)  --> TIME: 2025-02-25 08:28:52 -- STEP: 1328/36938 -- GLOBAL_STEP: 703150 | > loss_text_ce: 0.03509186580777168 (0.0353972603895152) | > loss_mel_ce: 3.2880594730377197 (nan) | > loss: 0.013187109492719173 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13677574891641897) | > loader_time: 0.0046 (0.00465983457594033)  --> TIME: 2025-02-25 08:29:02 -- STEP: 1378/36938 -- GLOBAL_STEP: 703200 | > loss_text_ce: 0.02844061516225338 (0.03539591174943075) | > loss_mel_ce: 2.274364709854126 (nan) | > loss: 0.009138117544353008 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.1368755043331526) | > loader_time: 0.0047 (0.004659049915817552)  --> TIME: 2025-02-25 08:29:11 -- STEP: 1428/36938 -- GLOBAL_STEP: 703250 | > loss_text_ce: 0.035241782665252686 (0.03541581936282334) | > loss_mel_ce: 3.4374783039093018 (nan) | > loss: 0.013780636712908745 (nan) | > current_lr: 5e-06 | > step_time: 0.1824 (0.1368855556854011) | > loader_time: 0.0049 (0.0046583504236045025)  --> TIME: 2025-02-25 08:29:21 -- STEP: 1478/36938 -- GLOBAL_STEP: 703300 | > loss_text_ce: 0.0321979783475399 (0.03536821338530829) | > loss_mel_ce: 3.1884117126464844 (nan) | > loss: 0.012780197896063328 (nan) | > current_lr: 5e-06 | > step_time: 0.1886 (0.13697661727626512) | > loader_time: 0.0048 (0.004658322534315964)  --> TIME: 2025-02-25 08:29:30 -- STEP: 1528/36938 -- GLOBAL_STEP: 703350 | > loss_text_ce: 0.034776028245687485 (0.03536085611762708) | > loss_mel_ce: 2.5099806785583496 (nan) | > loss: 0.01009824126958847 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.1369735423807072) | > loader_time: 0.0046 (0.004657946331962867)  --> TIME: 2025-02-25 08:29:40 -- STEP: 1578/36938 -- GLOBAL_STEP: 703400 | > loss_text_ce: 0.03849704563617706 (0.03537133105243213) | > loss_mel_ce: 2.902411937713623 (nan) | > loss: 0.01167027372866869 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13708540831228597) | > loader_time: 0.0046 (0.004659358690597864)  --> TIME: 2025-02-25 08:29:49 -- STEP: 1628/36938 -- GLOBAL_STEP: 703450 | > loss_text_ce: 0.031907159835100174 (0.03539479677382339) | > loss_mel_ce: 2.8485629558563232 (nan) | > loss: 0.011430437676608562 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13713291016491946) | > loader_time: 0.0047 (0.004658843873466846)  --> TIME: 2025-02-25 08:29:58 -- STEP: 1678/36938 -- GLOBAL_STEP: 703500 | > loss_text_ce: 0.03002002462744713 (0.03539560813205911) | > loss_mel_ce: 4.115885257720947 (nan) | > loss: 0.016452006995677948 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13704831821841718) | > loader_time: 0.0045 (0.004657495718036418)  --> TIME: 2025-02-25 08:30:08 -- STEP: 1728/36938 -- GLOBAL_STEP: 703550 | > loss_text_ce: 0.038824521005153656 (0.03540802252299968) | > loss_mel_ce: 1.6991854906082153 (nan) | > loss: 0.006896865554153919 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.137201343145635) | > loader_time: 0.0046 (0.004657007615875313)  --> TIME: 2025-02-25 08:30:17 -- STEP: 1778/36938 -- GLOBAL_STEP: 703600 | > loss_text_ce: 0.028971528634428978 (0.03540883991915498) | > loss_mel_ce: 2.590170383453369 (nan) | > loss: 0.010393420234322548 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13716587253308782) | > loader_time: 0.0046 (0.004655948505895074)  --> TIME: 2025-02-25 08:30:27 -- STEP: 1828/36938 -- GLOBAL_STEP: 703650 | > loss_text_ce: 0.030202601104974747 (0.03541030780681263) | > loss_mel_ce: 3.1015474796295166 (nan) | > loss: 0.012427580542862415 (nan) | > current_lr: 5e-06 | > step_time: 0.1702 (0.13717445020341754) | > loader_time: 0.0048 (0.0046545118419555375)  --> TIME: 2025-02-25 08:30:36 -- STEP: 1878/36938 -- GLOBAL_STEP: 703700 | > loss_text_ce: 0.042555876076221466 (0.03539909289840362) | > loss_mel_ce: 2.9386448860168457 (nan) | > loss: 0.0118301622569561 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.13707903462350038) | > loader_time: 0.0047 (0.004654608341578602)  --> TIME: 2025-02-25 08:30:45 -- STEP: 1928/36938 -- GLOBAL_STEP: 703750 | > loss_text_ce: 0.03798418119549751 (0.03538012679683673) | > loss_mel_ce: 2.9185009002685547 (nan) | > loss: 0.01173208374530077 (nan) | > current_lr: 5e-06 | > step_time: 0.1884 (0.13715398361079414) | > loader_time: 0.0048 (0.004654879391935355)  --> TIME: 2025-02-25 08:30:55 -- STEP: 1978/36938 -- GLOBAL_STEP: 703800 | > loss_text_ce: 0.03689509630203247 (0.035386684018726834) | > loss_mel_ce: 2.957552671432495 (nan) | > loss: 0.011882729828357697 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13714389504508845) | > loader_time: 0.0046 (0.004654621089553446)  --> TIME: 2025-02-25 08:31:04 -- STEP: 2028/36938 -- GLOBAL_STEP: 703850 | > loss_text_ce: 0.04177035763859749 (0.035386549943095535) | > loss_mel_ce: 2.1513571739196777 (nan) | > loss: 0.008702888153493404 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13717076729035224) | > loader_time: 0.0047 (0.004655453933061226)  --> TIME: 2025-02-25 08:31:14 -- STEP: 2078/36938 -- GLOBAL_STEP: 703900 | > loss_text_ce: 0.03580731153488159 (0.03533850988208232) | > loss_mel_ce: 1.7673461437225342 (nan) | > loss: 0.007155371364206076 (nan) | > current_lr: 5e-06 | > step_time: 0.1343 (0.1372537646188083) | > loader_time: 0.0045 (0.004655776859132914)  --> TIME: 2025-02-25 08:31:23 -- STEP: 2128/36938 -- GLOBAL_STEP: 703950 | > loss_text_ce: 0.028930004686117172 (0.035353596639535884) | > loss_mel_ce: 2.7468574047088623 (nan) | > loss: 0.0110150296241045 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.13723035347192797) | > loader_time: 0.0047 (0.004655203536937109)  --> TIME: 2025-02-25 08:31:32 -- STEP: 2178/36938 -- GLOBAL_STEP: 704000 | > loss_text_ce: 0.03514271229505539 (0.035369146354855704) | > loss_mel_ce: 2.854912042617798 (nan) | > loss: 0.011468471959233284 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.1371930145801372) | > loader_time: 0.0046 (0.004654448879651785)  --> TIME: 2025-02-25 08:31:41 -- STEP: 2228/36938 -- GLOBAL_STEP: 704050 | > loss_text_ce: 0.03670942783355713 (0.03538238407610928) | > loss_mel_ce: 2.134232997894287 (nan) | > loss: 0.008614851161837578 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13706886169084306) | > loader_time: 0.0046 (0.004654312904262027)  --> TIME: 2025-02-25 08:31:51 -- STEP: 2278/36938 -- GLOBAL_STEP: 704100 | > loss_text_ce: 0.026130348443984985 (0.03538806310505714) | > loss_mel_ce: 2.461789846420288 (nan) | > loss: 0.009872700087726116 (nan) | > current_lr: 5e-06 | > step_time: 0.1617 (0.1371882569905013) | > loader_time: 0.0048 (0.004655045098228553)  --> TIME: 2025-02-25 08:32:00 -- STEP: 2328/36938 -- GLOBAL_STEP: 704150 | > loss_text_ce: 0.04035715013742447 (0.035386851564277025) | > loss_mel_ce: 2.8902957439422607 (nan) | > loss: 0.011629575863480568 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13717685573289484) | > loader_time: 0.0047 (0.004654254597896561)  --> TIME: 2025-02-25 08:32:10 -- STEP: 2378/36938 -- GLOBAL_STEP: 704200 | > loss_text_ce: 0.0400681234896183 (0.03536012393827387) | > loss_mel_ce: 3.4380974769592285 (nan) | > loss: 0.013802245259284973 (nan) | > current_lr: 5e-06 | > step_time: 0.1885 (0.13730034886336906) | > loader_time: 0.0048 (0.004653801930060999)  --> TIME: 2025-02-25 08:32:19 -- STEP: 2428/36938 -- GLOBAL_STEP: 704250 | > loss_text_ce: 0.028225021436810493 (0.0353405270522194) | > loss_mel_ce: 3.3846192359924316 (nan) | > loss: 0.01354303304105997 (nan) | > current_lr: 5e-06 | > step_time: 0.1609 (0.13729765751020123) | > loader_time: 0.0045 (0.004653856624684966)  --> TIME: 2025-02-25 08:32:29 -- STEP: 2478/36938 -- GLOBAL_STEP: 704300 | > loss_text_ce: 0.0356396920979023 (0.03534314094962223) | > loss_mel_ce: 3.4164044857025146 (nan) | > loss: 0.01369858905673027 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13730771787903592) | > loader_time: 0.0045 (0.004652897420672273)  --> TIME: 2025-02-25 08:32:38 -- STEP: 2528/36938 -- GLOBAL_STEP: 704350 | > loss_text_ce: 0.036954496055841446 (0.03531730387734162) | > loss_mel_ce: 2.60951828956604 (nan) | > loss: 0.010501876473426819 (nan) | > current_lr: 5e-06 | > step_time: 0.1785 (0.13722779762141302) | > loader_time: 0.005 (0.004652711881112443)  --> TIME: 2025-02-25 08:32:47 -- STEP: 2578/36938 -- GLOBAL_STEP: 704400 | > loss_text_ce: 0.043604809790849686 (0.035313671334849736) | > loss_mel_ce: 3.437185049057007 (nan) | > loss: 0.013812659308314323 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.1372364646617158) | > loader_time: 0.0046 (0.004653824864661993)  --> TIME: 2025-02-25 08:32:57 -- STEP: 2628/36938 -- GLOBAL_STEP: 704450 | > loss_text_ce: 0.032930225133895874 (0.03531210736373001) | > loss_mel_ce: 2.818183422088623 (nan) | > loss: 0.011313943192362785 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13725308976398384) | > loader_time: 0.0046 (0.004654258534969978)  --> TIME: 2025-02-25 08:33:06 -- STEP: 2678/36938 -- GLOBAL_STEP: 704500 | > loss_text_ce: 0.03619019314646721 (0.03530835968187006) | > loss_mel_ce: 3.071298599243164 (nan) | > loss: 0.012331306003034115 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13719286653334917) | > loader_time: 0.0045 (0.004654851842941491)  --> TIME: 2025-02-25 08:33:15 -- STEP: 2728/36938 -- GLOBAL_STEP: 704550 | > loss_text_ce: 0.04513442516326904 (0.03530699035010781) | > loss_mel_ce: 1.7300293445587158 (nan) | > loss: 0.007044300902634859 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.1371373594506394) | > loader_time: 0.0055 (0.004655529676644323)  --> TIME: 2025-02-25 08:33:24 -- STEP: 2778/36938 -- GLOBAL_STEP: 704600 | > loss_text_ce: 0.03265269845724106 (0.0353075037727132) | > loss_mel_ce: 2.413771867752075 (nan) | > loss: 0.009708033874630928 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13711741272711414) | > loader_time: 0.0046 (0.004655422625394062)  --> TIME: 2025-02-25 08:33:34 -- STEP: 2828/36938 -- GLOBAL_STEP: 704650 | > loss_text_ce: 0.03150951862335205 (0.035321694946303925) | > loss_mel_ce: 3.145814895629883 (nan) | > loss: 0.012608430348336697 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13728744010776903) | > loader_time: 0.0046 (0.004656165037330503)  --> TIME: 2025-02-25 08:33:43 -- STEP: 2878/36938 -- GLOBAL_STEP: 704700 | > loss_text_ce: 0.0371280200779438 (0.035330746440660976) | > loss_mel_ce: 2.8782474994659424 (nan) | > loss: 0.011568950489163399 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13729683801148648) | > loader_time: 0.0045 (0.004658512890214631)  --> TIME: 2025-02-25 08:33:52 -- STEP: 2928/36938 -- GLOBAL_STEP: 704750 | > loss_text_ce: 0.032422471791505814 (0.035331877653965504) | > loss_mel_ce: 3.117288112640381 (nan) | > loss: 0.012498852796852589 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13721053709423597) | > loader_time: 0.0051 (0.004657863674919456)  --> TIME: 2025-02-25 08:34:02 -- STEP: 2978/36938 -- GLOBAL_STEP: 704800 | > loss_text_ce: 0.034397535026073456 (0.03535277174149013) | > loss_mel_ce: 2.4020731449127197 (nan) | > loss: 0.009668535552918911 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.137210518502004) | > loader_time: 0.0045 (0.004657969048873456)  --> TIME: 2025-02-25 08:34:11 -- STEP: 3028/36938 -- GLOBAL_STEP: 704850 | > loss_text_ce: 0.029645903035998344 (0.035338697206386514) | > loss_mel_ce: 2.9163613319396973 (nan) | > loss: 0.011690505780279636 (nan) | > current_lr: 5e-06 | > step_time: 0.1767 (0.13717013731342936) | > loader_time: 0.0047 (0.004657337734847273)  --> TIME: 2025-02-25 08:34:20 -- STEP: 3078/36938 -- GLOBAL_STEP: 704900 | > loss_text_ce: 0.03797508776187897 (0.03533614302432873) | > loss_mel_ce: 2.1845035552978516 (nan) | > loss: 0.00881936028599739 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13704987452199033) | > loader_time: 0.0046 (0.0046571806857460376)  --> TIME: 2025-02-25 08:34:29 -- STEP: 3128/36938 -- GLOBAL_STEP: 704950 | > loss_text_ce: 0.0378793366253376 (0.035334998979935896) | > loss_mel_ce: 3.2409415245056152 (nan) | > loss: 0.013011193834245205 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13703733126220796) | > loader_time: 0.0046 (0.0046575551142777885)  --> TIME: 2025-02-25 08:34:39 -- STEP: 3178/36938 -- GLOBAL_STEP: 705000 | > loss_text_ce: 0.029051564633846283 (0.03530671367980897) | > loss_mel_ce: 2.829031229019165 (nan) | > loss: 0.011341598816215992 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13707268448128052) | > loader_time: 0.0048 (0.004657870947451915)  --> TIME: 2025-02-25 08:34:48 -- STEP: 3228/36938 -- GLOBAL_STEP: 705050 | > loss_text_ce: 0.031205322593450546 (0.035297821069891086) | > loss_mel_ce: 3.3559019565582275 (nan) | > loss: 0.013440903276205063 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.1370886645500251) | > loader_time: 0.0049 (0.00465734186668821)  --> TIME: 2025-02-25 08:34:57 -- STEP: 3278/36938 -- GLOBAL_STEP: 705100 | > loss_text_ce: 0.0372869037091732 (0.03529419863961364) | > loss_mel_ce: 3.096639633178711 (nan) | > loss: 0.0124362176284194 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13705562570605667) | > loader_time: 0.0046 (0.0046568400544171745)  --> TIME: 2025-02-25 08:35:07 -- STEP: 3328/36938 -- GLOBAL_STEP: 705150 | > loss_text_ce: 0.02814059890806675 (0.03528924920213358) | > loss_mel_ce: 5.25 (nan) | > loss: 0.020945003256201744 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13703970045137892) | > loader_time: 0.0046 (0.00465703103691339)  --> TIME: 2025-02-25 08:35:16 -- STEP: 3378/36938 -- GLOBAL_STEP: 705200 | > loss_text_ce: 0.03221555054187775 (0.03527539822478633) | > loss_mel_ce: 3.107679605484009 (nan) | > loss: 0.012459902092814445 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.13698254957080547) | > loader_time: 0.0046 (0.004656851679845411)  --> TIME: 2025-02-25 08:35:25 -- STEP: 3428/36938 -- GLOBAL_STEP: 705250 | > loss_text_ce: 0.03341163694858551 (0.03526335251935986) | > loss_mel_ce: 3.0161571502685547 (nan) | > loss: 0.012101464904844761 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13703431096906046) | > loader_time: 0.0046 (0.004656907766535783)  --> TIME: 2025-02-25 08:35:34 -- STEP: 3478/36938 -- GLOBAL_STEP: 705300 | > loss_text_ce: 0.03821353241801262 (0.03526303268671298) | > loss_mel_ce: 2.377739191055298 (nan) | > loss: 0.009587114676833153 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13701385064259472) | > loader_time: 0.0045 (0.00465655971217799)  --> TIME: 2025-02-25 08:35:44 -- STEP: 3528/36938 -- GLOBAL_STEP: 705350 | > loss_text_ce: 0.03388522192835808 (0.035240707997139804) | > loss_mel_ce: 2.908231496810913 (nan) | > loss: 0.011675067245960236 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.1370487952448619) | > loader_time: 0.0047 (0.004656152727922877)  --> TIME: 2025-02-25 08:35:54 -- STEP: 3578/36938 -- GLOBAL_STEP: 705400 | > loss_text_ce: 0.033218350261449814 (0.035235339924047614) | > loss_mel_ce: 2.7382094860076904 (nan) | > loss: 0.010997730307281017 (nan) | > current_lr: 5e-06 | > step_time: 0.1502 (0.13717257716657114) | > loader_time: 0.0046 (0.00465683526603651)  --> TIME: 2025-02-25 08:36:03 -- STEP: 3628/36938 -- GLOBAL_STEP: 705450 | > loss_text_ce: 0.0337098129093647 (0.03523381996043481) | > loss_mel_ce: 3.0820260047912598 (nan) | > loss: 0.012364031746983528 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.13721913572446046) | > loader_time: 0.005 (0.0046572465092239385)  --> TIME: 2025-02-25 08:36:13 -- STEP: 3678/36938 -- GLOBAL_STEP: 705500 | > loss_text_ce: 0.0411686971783638 (0.03525171650025658) | > loss_mel_ce: 2.4307072162628174 (nan) | > loss: 0.009809032082557678 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13719901953008065) | > loader_time: 0.0045 (0.004657108670930093)  --> TIME: 2025-02-25 08:36:22 -- STEP: 3728/36938 -- GLOBAL_STEP: 705550 | > loss_text_ce: 0.03767610341310501 (0.03524497974226294) | > loss_mel_ce: 2.7289249897003174 (nan) | > loss: 0.010978576727211475 (nan) | > current_lr: 5e-06 | > step_time: 0.1679 (0.1371794348750509) | > loader_time: 0.0048 (0.004657237250917452)  --> TIME: 2025-02-25 08:36:31 -- STEP: 3778/36938 -- GLOBAL_STEP: 705600 | > loss_text_ce: 0.03400591015815735 (0.03521361634506989) | > loss_mel_ce: 2.051548480987549 (nan) | > loss: 0.008276009932160378 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.1371197731422967) | > loader_time: 0.0048 (0.004656688380708379)  --> TIME: 2025-02-25 08:36:40 -- STEP: 3828/36938 -- GLOBAL_STEP: 705650 | > loss_text_ce: 0.030688263475894928 (0.03521137808492389) | > loss_mel_ce: 2.8436646461486816 (nan) | > loss: 0.01140616275370121 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.13713490639111642) | > loader_time: 0.0047 (0.004656540562740306)  --> TIME: 2025-02-25 08:36:49 -- STEP: 3878/36938 -- GLOBAL_STEP: 705700 | > loss_text_ce: 0.03687792271375656 (0.03519597255167097) | > loss_mel_ce: 2.591094970703125 (nan) | > loss: 0.01042846404016018 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1370834114011757) | > loader_time: 0.0045 (0.004655873425047436)  --> TIME: 2025-02-25 08:36:59 -- STEP: 3928/36938 -- GLOBAL_STEP: 705750 | > loss_text_ce: 0.02974802441895008 (0.03518838104303519) | > loss_mel_ce: 2.8912036418914795 (nan) | > loss: 0.011591078713536263 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13709801125914942) | > loader_time: 0.0047 (0.004656322801185958)  --> TIME: 2025-02-25 08:37:08 -- STEP: 3978/36938 -- GLOBAL_STEP: 705800 | > loss_text_ce: 0.03204748034477234 (0.035179100458474194) | > loss_mel_ce: 3.0749337673187256 (nan) | > loss: 0.012329291552305222 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1371474388917532) | > loader_time: 0.0046 (0.004656861810123212)  --> TIME: 2025-02-25 08:37:18 -- STEP: 4028/36938 -- GLOBAL_STEP: 705850 | > loss_text_ce: 0.04032158851623535 (0.035158211177366686) | > loss_mel_ce: 0.1209208145737648 (nan) | > loss: 0.0006398507975973189 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1371856565035093) | > loader_time: 0.0045 (0.004657481194963041)  --> TIME: 2025-02-25 08:37:27 -- STEP: 4078/36938 -- GLOBAL_STEP: 705900 | > loss_text_ce: 0.038898807018995285 (0.035156621196905165) | > loss_mel_ce: 2.571988105773926 (nan) | > loss: 0.010360662825405598 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13717680593633771) | > loader_time: 0.0047 (0.004657280860197426)  --> TIME: 2025-02-25 08:37:37 -- STEP: 4128/36938 -- GLOBAL_STEP: 705950 | > loss_text_ce: 0.03182012587785721 (0.03514547236584122) | > loss_mel_ce: 0.05731964111328125 (nan) | > loss: 0.000353729265043512 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13718323236288077) | > loader_time: 0.0045 (0.0046571735148281945)  --> TIME: 2025-02-25 08:37:46 -- STEP: 4178/36938 -- GLOBAL_STEP: 706000 | > loss_text_ce: 0.035318780690431595 (0.0351287349378677) | > loss_mel_ce: 3.14367413520813 (nan) | > loss: 0.012615052051842213 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13721640490070428) | > loader_time: 0.0046 (0.0046571699154325545)  --> TIME: 2025-02-25 08:37:55 -- STEP: 4228/36938 -- GLOBAL_STEP: 706050 | > loss_text_ce: 0.025759655982255936 (0.03512705075766017) | > loss_mel_ce: 2.7004799842834473 (nan) | > loss: 0.010818411596119404 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13719916907592938) | > loader_time: 0.0047 (0.004656804262069219)  --> TIME: 2025-02-25 08:38:05 -- STEP: 4278/36938 -- GLOBAL_STEP: 706100 | > loss_text_ce: 0.038029395043849945 (0.03512086926982536) | > loss_mel_ce: 3.1260228157043457 (nan) | > loss: 0.012555764056742191 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13720310286084583) | > loader_time: 0.0046 (0.004656531644681164)  --> TIME: 2025-02-25 08:38:14 -- STEP: 4328/36938 -- GLOBAL_STEP: 706150 | > loss_text_ce: 0.038164518773555756 (0.03511124044375295) | > loss_mel_ce: 3.1432411670684814 (nan) | > loss: 0.012624626979231834 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.13718936199842288) | > loader_time: 0.0046 (0.004656909408939519)  --> TIME: 2025-02-25 08:38:24 -- STEP: 4378/36938 -- GLOBAL_STEP: 706200 | > loss_text_ce: 0.02462267316877842 (0.03510147238311068) | > loss_mel_ce: 1.1257563829421997 (nan) | > loss: 0.004564996343106031 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13722469965689527) | > loader_time: 0.0045 (0.004657168593130025)  --> TIME: 2025-02-25 08:38:33 -- STEP: 4428/36938 -- GLOBAL_STEP: 706250 | > loss_text_ce: 0.038199253380298615 (0.03511002076900797) | > loss_mel_ce: 3.0745508670806885 (nan) | > loss: 0.01235218346118927 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13725354559484051) | > loader_time: 0.0046 (0.004657300668680244)  --> TIME: 2025-02-25 08:38:43 -- STEP: 4478/36938 -- GLOBAL_STEP: 706300 | > loss_text_ce: 0.0347299724817276 (0.0351014293700311) | > loss_mel_ce: 3.3124706745147705 (nan) | > loss: 0.013282543048262596 (nan) | > current_lr: 5e-06 | > step_time: 0.1471 (0.13730951535377017) | > loader_time: 0.0047 (0.004657446512850121)  --> TIME: 2025-02-25 08:38:52 -- STEP: 4528/36938 -- GLOBAL_STEP: 706350 | > loss_text_ce: 0.03375692293047905 (0.035102224324635785) | > loss_mel_ce: 1.7784748077392578 (nan) | > loss: 0.007191396318376064 (nan) | > current_lr: 5e-06 | > step_time: 0.1528 (0.13729634175452285) | > loader_time: 0.005 (0.004657267629047163)  --> TIME: 2025-02-25 08:39:01 -- STEP: 4578/36938 -- GLOBAL_STEP: 706400 | > loss_text_ce: 0.038106705993413925 (0.035103170409906084) | > loss_mel_ce: 2.7008440494537354 (nan) | > loss: 0.010868852958083153 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13726285244503042) | > loader_time: 0.0045 (0.004657031615887629)  --> TIME: 2025-02-25 08:39:10 -- STEP: 4628/36938 -- GLOBAL_STEP: 706450 | > loss_text_ce: 0.03053390420973301 (0.03509605865420047) | > loss_mel_ce: 2.765268325805664 (nan) | > loss: 0.011094453744590282 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13721801148680898) | > loader_time: 0.0046 (0.004656932481734423)  --> TIME: 2025-02-25 08:39:20 -- STEP: 4678/36938 -- GLOBAL_STEP: 706500 | > loss_text_ce: 0.026049159467220306 (0.03508114062174807) | > loss_mel_ce: 2.90625 (nan) | > loss: 0.011636108160018921 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13719979995027926) | > loader_time: 0.0044 (0.004656650307546469)  --> TIME: 2025-02-25 08:39:29 -- STEP: 4728/36938 -- GLOBAL_STEP: 706550 | > loss_text_ce: 0.034617360681295395 (0.035079593956549325) | > loss_mel_ce: 3.451904296875 (nan) | > loss: 0.013835404068231583 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13713057039151177) | > loader_time: 0.0045 (0.004656129127428212)  --> TIME: 2025-02-25 08:39:38 -- STEP: 4778/36938 -- GLOBAL_STEP: 706600 | > loss_text_ce: 0.042292360216379166 (0.03507429312145852) | > loss_mel_ce: 2.8880245685577393 (nan) | > loss: 0.011628242209553719 (nan) | > current_lr: 5e-06 | > step_time: 0.149 (0.13709770754921569) | > loader_time: 0.0047 (0.004656054425708618)  --> TIME: 2025-02-25 08:39:47 -- STEP: 4828/36938 -- GLOBAL_STEP: 706650 | > loss_text_ce: 0.02748560905456543 (0.035074000583626766) | > loss_mel_ce: 2.8569605350494385 (nan) | > loss: 0.011446215212345123 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13712411506569117) | > loader_time: 0.0046 (0.004656068332548862)  --> TIME: 2025-02-25 08:39:57 -- STEP: 4878/36938 -- GLOBAL_STEP: 706700 | > loss_text_ce: 0.034737106412649155 (0.03506297680047962) | > loss_mel_ce: 2.590625047683716 (nan) | > loss: 0.010418104007840157 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.1370848960297773) | > loader_time: 0.0045 (0.004655704091640225)  --> TIME: 2025-02-25 08:40:06 -- STEP: 4928/36938 -- GLOBAL_STEP: 706750 | > loss_text_ce: 0.035038236528635025 (0.03505343041275388) | > loss_mel_ce: 2.8845102787017822 (nan) | > loss: 0.011585510335862637 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13705528392033192) | > loader_time: 0.0045 (0.0046559172597798425)  --> TIME: 2025-02-25 08:40:15 -- STEP: 4978/36938 -- GLOBAL_STEP: 706800 | > loss_text_ce: 0.036982372403144836 (0.03506609939664376) | > loss_mel_ce: 2.8333957195281982 (nan) | > loss: 0.011390389874577522 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13706949615440253) | > loader_time: 0.0047 (0.004655300205900471)  --> TIME: 2025-02-25 08:40:25 -- STEP: 5028/36938 -- GLOBAL_STEP: 706850 | > loss_text_ce: 0.03974289447069168 (0.035070667038512394) | > loss_mel_ce: 2.812445640563965 (nan) | > loss: 0.011318209581077099 (nan) | > current_lr: 5e-06 | > step_time: 0.1797 (0.1370995233906772) | > loader_time: 0.0048 (0.004654769681234064)  --> TIME: 2025-02-25 08:40:34 -- STEP: 5078/36938 -- GLOBAL_STEP: 706900 | > loss_text_ce: 0.03458340838551521 (0.03507166427595732) | > loss_mel_ce: 2.092139482498169 (nan) | > loss: 0.008439376950263977 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13705210340168647) | > loader_time: 0.0046 (0.00465393169699057)  --> TIME: 2025-02-25 08:40:43 -- STEP: 5128/36938 -- GLOBAL_STEP: 706950 | > loss_text_ce: 0.030395779758691788 (0.035054276999545254) | > loss_mel_ce: 3.002331256866455 (nan) | > loss: 0.012034631334245205 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13707563564669553) | > loader_time: 0.0046 (0.0046533801347342744)  --> TIME: 2025-02-25 08:40:52 -- STEP: 5178/36938 -- GLOBAL_STEP: 707000 | > loss_text_ce: 0.029719237238168716 (0.03506583372305812) | > loss_mel_ce: 2.9041032791137695 (nan) | > loss: 0.011642154306173325 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13704722610943695) | > loader_time: 0.0046 (0.004652620052545403)  --> TIME: 2025-02-25 08:41:01 -- STEP: 5228/36938 -- GLOBAL_STEP: 707050 | > loss_text_ce: 0.030935293063521385 (0.03506242109828875) | > loss_mel_ce: 1.5503638982772827 (nan) | > loss: 0.006274997256696224 (nan) | > current_lr: 5e-06 | > step_time: 0.1613 (0.13700931676034306) | > loader_time: 0.0046 (0.004651867485994956)  --> TIME: 2025-02-25 08:41:10 -- STEP: 5278/36938 -- GLOBAL_STEP: 707100 | > loss_text_ce: 0.025108685716986656 (0.035043650914114814) | > loss_mel_ce: 2.202446937561035 (nan) | > loss: 0.008839506655931473 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1369834215454013) | > loader_time: 0.0046 (0.004651694868766436)  --> TIME: 2025-02-25 08:41:20 -- STEP: 5328/36938 -- GLOBAL_STEP: 707150 | > loss_text_ce: 0.034987058490514755 (0.035025633692212774) | > loss_mel_ce: 3.340672492980957 (nan) | > loss: 0.013395475223660469 (nan) | > current_lr: 5e-06 | > step_time: 0.1626 (0.1369612740503779) | > loader_time: 0.0047 (0.004651112733660516)  --> TIME: 2025-02-25 08:41:29 -- STEP: 5378/36938 -- GLOBAL_STEP: 707200 | > loss_text_ce: 0.03971923142671585 (0.03502331850042387) | > loss_mel_ce: 2.990010976791382 (nan) | > loss: 0.012022740207612514 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.13699189168718) | > loader_time: 0.0048 (0.004651055366144426)  --> TIME: 2025-02-25 08:41:38 -- STEP: 5428/36938 -- GLOBAL_STEP: 707250 | > loss_text_ce: 0.036798544228076935 (0.035030075767219504) | > loss_mel_ce: 3.254368782043457 (nan) | > loss: 0.013060187920928001 (nan) | > current_lr: 5e-06 | > step_time: 0.1507 (0.13697053534895012) | > loader_time: 0.0048 (0.00465138523387135)  --> TIME: 2025-02-25 08:41:47 -- STEP: 5478/36938 -- GLOBAL_STEP: 707300 | > loss_text_ce: 0.03966967388987541 (0.035010719843896906) | > loss_mel_ce: 2.7799670696258545 (nan) | > loss: 0.011189036071300507 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13692860087150688) | > loader_time: 0.0045 (0.004651272980215422)  --> TIME: 2025-02-25 08:41:57 -- STEP: 5528/36938 -- GLOBAL_STEP: 707350 | > loss_text_ce: 0.028216449543833733 (0.03501378529935571) | > loss_mel_ce: 2.151587963104248 (nan) | > loss: 0.008650017902255058 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13695881542697183) | > loader_time: 0.0045 (0.004650862534380851)  --> TIME: 2025-02-25 08:42:07 -- STEP: 5578/36938 -- GLOBAL_STEP: 707400 | > loss_text_ce: 0.031541626900434494 (0.03500801547640486) | > loss_mel_ce: 2.632429599761963 (nan) | > loss: 0.010571314953267574 (nan) | > current_lr: 5e-06 | > step_time: 0.1511 (0.13700599378050374) | > loader_time: 0.0046 (0.0046509024745548565)  --> TIME: 2025-02-25 08:42:16 -- STEP: 5628/36938 -- GLOBAL_STEP: 707450 | > loss_text_ce: 0.025323450565338135 (0.03500240542757097) | > loss_mel_ce: 2.783891201019287 (nan) | > loss: 0.011147677898406982 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13701215858737417) | > loader_time: 0.0046 (0.004650904764583449)  --> TIME: 2025-02-25 08:42:26 -- STEP: 5678/36938 -- GLOBAL_STEP: 707500 | > loss_text_ce: 0.03229950740933418 (0.03498835616698135) | > loss_mel_ce: 2.723588705062866 (nan) | > loss: 0.010936064645648003 (nan) | > current_lr: 5e-06 | > step_time: 0.1774 (0.13704394752519958) | > loader_time: 0.0048 (0.004651006488323041)  --> TIME: 2025-02-25 08:42:35 -- STEP: 5728/36938 -- GLOBAL_STEP: 707550 | > loss_text_ce: 0.04548536613583565 (0.03499262510537418) | > loss_mel_ce: 2.959134578704834 (nan) | > loss: 0.01192309521138668 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13709148968564774) | > loader_time: 0.0045 (0.004651018153022789)  --> TIME: 2025-02-25 08:42:45 -- STEP: 5778/36938 -- GLOBAL_STEP: 707600 | > loss_text_ce: 0.022559914737939835 (0.03498791485113189) | > loss_mel_ce: 0.007539340294897556 (nan) | > loss: 0.0001194414944620803 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13709948161103108) | > loader_time: 0.0046 (0.004650820205268582)  --> TIME: 2025-02-25 08:42:54 -- STEP: 5828/36938 -- GLOBAL_STEP: 707650 | > loss_text_ce: 0.029131941497325897 (0.03498891235777284) | > loss_mel_ce: 3.02215313911438 (nan) | > loss: 0.01210827473551035 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13705984565928628) | > loader_time: 0.0046 (0.004650560485719405)  --> TIME: 2025-02-25 08:43:03 -- STEP: 5878/36938 -- GLOBAL_STEP: 707700 | > loss_text_ce: 0.02958075888454914 (0.034982939657012595) | > loss_mel_ce: 3.3269271850585938 (nan) | > loss: 0.013319477438926697 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13708802427958391) | > loader_time: 0.0047 (0.004650556988438684)  --> TIME: 2025-02-25 08:43:13 -- STEP: 5928/36938 -- GLOBAL_STEP: 707750 | > loss_text_ce: 0.03229707479476929 (0.03498697947388985) | > loss_mel_ce: 2.2172608375549316 (nan) | > loss: 0.008926818147301674 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13710424199438945) | > loader_time: 0.0045 (0.004650610500340839)  --> TIME: 2025-02-25 08:43:22 -- STEP: 5978/36938 -- GLOBAL_STEP: 707800 | > loss_text_ce: 0.03388482332229614 (0.034992412558369344) | > loss_mel_ce: 1.5526834726333618 (nan) | > loss: 0.006295906379818916 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13709957063616862) | > loader_time: 0.0046 (0.004650946922149088)  --> TIME: 2025-02-25 08:43:32 -- STEP: 6028/36938 -- GLOBAL_STEP: 707850 | > loss_text_ce: 0.03665197640657425 (0.034983559434352066) | > loss_mel_ce: 2.9814372062683105 (nan) | > loss: 0.011976545676589012 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13708889211497732) | > loader_time: 0.0046 (0.004651334796746033)  --> TIME: 2025-02-25 08:43:41 -- STEP: 6078/36938 -- GLOBAL_STEP: 707900 | > loss_text_ce: 0.03732750937342644 (0.03497845121095751) | > loss_mel_ce: 3.081465005874634 (nan) | > loss: 0.01237616129219532 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.1371110718514663) | > loader_time: 0.0046 (0.004651133188019842)  --> TIME: 2025-02-25 08:43:50 -- STEP: 6128/36938 -- GLOBAL_STEP: 707950 | > loss_text_ce: 0.037746164947748184 (0.03496730607398104) | > loss_mel_ce: 2.0982208251953125 (nan) | > loss: 0.008476059883832932 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13708825658413523) | > loader_time: 0.0045 (0.004651214762082601)  --> TIME: 2025-02-25 08:43:59 -- STEP: 6178/36938 -- GLOBAL_STEP: 708000 | > loss_text_ce: 0.03543185815215111 (0.03497028797708123) | > loss_mel_ce: 3.0443029403686523 (nan) | > loss: 0.012221170589327812 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13705213662681945) | > loader_time: 0.0046 (0.0046516402943229136)  --> TIME: 2025-02-25 08:44:09 -- STEP: 6228/36938 -- GLOBAL_STEP: 708050 | > loss_text_ce: 0.035757895559072495 (0.03497203600915306) | > loss_mel_ce: 3.218252658843994 (nan) | > loss: 0.012912740930914879 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.13710009898600406) | > loader_time: 0.0046 (0.004651908853073944)  --> TIME: 2025-02-25 08:44:19 -- STEP: 6278/36938 -- GLOBAL_STEP: 708100 | > loss_text_ce: 0.03672482818365097 (0.034965764349323435) | > loss_mel_ce: 3.3232548236846924 (nan) | > loss: 0.013333253562450409 (nan) | > current_lr: 5e-06 | > step_time: 0.1984 (0.13711658648982783) | > loader_time: 0.0049 (0.004652042569685744)  --> TIME: 2025-02-25 08:44:28 -- STEP: 6328/36938 -- GLOBAL_STEP: 708150 | > loss_text_ce: 0.03886483982205391 (0.03496937150448815) | > loss_mel_ce: 3.5744850635528564 (nan) | > loss: 0.014338690787553787 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13709607916294247) | > loader_time: 0.0048 (0.004651822046142449)  --> TIME: 2025-02-25 08:44:37 -- STEP: 6378/36938 -- GLOBAL_STEP: 708200 | > loss_text_ce: 0.03376419097185135 (0.034972695370346904) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13704037258863533) | > loader_time: 0.0044 (0.004651355421928135)  --> TIME: 2025-02-25 08:44:46 -- STEP: 6428/36938 -- GLOBAL_STEP: 708250 | > loss_text_ce: 0.04086144268512726 (0.034960497043042736) | > loss_mel_ce: 2.541749954223633 (nan) | > loss: 0.0102484580129385 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1370022494598584) | > loader_time: 0.0046 (0.004651419591221207)  --> TIME: 2025-02-25 08:44:55 -- STEP: 6478/36938 -- GLOBAL_STEP: 708300 | > loss_text_ce: 0.0458102710545063 (0.034962380875543736) | > loss_mel_ce: 3.2279787063598633 (nan) | > loss: 0.012991226278245449 (nan) | > current_lr: 5e-06 | > step_time: 0.1829 (0.13704819479044608) | > loader_time: 0.0049 (0.004651389912566706)  --> TIME: 2025-02-25 08:45:04 -- STEP: 6528/36938 -- GLOBAL_STEP: 708350 | > loss_text_ce: 0.034748781472444534 (0.03495331710006454) | > loss_mel_ce: 3.6103107929229736 (nan) | > loss: 0.01446452271193266 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.13701919079119104) | > loader_time: 0.0045 (0.004651067121063955)  --> TIME: 2025-02-25 08:45:14 -- STEP: 6578/36938 -- GLOBAL_STEP: 708400 | > loss_text_ce: 0.03569205105304718 (0.03494368775657945) | > loss_mel_ce: 3.009183645248413 (nan) | > loss: 0.0120828403159976 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1370013621702281) | > loader_time: 0.0045 (0.004650806865999485)  --> TIME: 2025-02-25 08:45:23 -- STEP: 6628/36938 -- GLOBAL_STEP: 708450 | > loss_text_ce: 0.030255377292633057 (0.03493671410146016) | > loss_mel_ce: 3.0654492378234863 (nan) | > loss: 0.012284542433917522 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13698001248187622) | > loader_time: 0.0046 (0.004650440069301051)  --> TIME: 2025-02-25 08:45:32 -- STEP: 6678/36938 -- GLOBAL_STEP: 708500 | > loss_text_ce: 0.03361308574676514 (0.034916771101779466) | > loss_mel_ce: 2.8105201721191406 (nan) | > loss: 0.011286243796348572 (nan) | > current_lr: 5e-06 | > step_time: 0.1606 (0.13697689266467614) | > loader_time: 0.0046 (0.004650130783165591)  --> TIME: 2025-02-25 08:45:41 -- STEP: 6728/36938 -- GLOBAL_STEP: 708550 | > loss_text_ce: 0.04489501938223839 (0.03491356139340931) | > loss_mel_ce: 3.3107151985168457 (nan) | > loss: 0.013315914198756218 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13697487739547118) | > loader_time: 0.0046 (0.0046499296402676025)  --> TIME: 2025-02-25 08:45:51 -- STEP: 6778/36938 -- GLOBAL_STEP: 708600 | > loss_text_ce: 0.042996227741241455 (0.03490969942147266) | > loss_mel_ce: 2.5282530784606934 (nan) | > loss: 0.010203370824456215 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13698712795946447) | > loader_time: 0.0048 (0.004650031651462831)  --> TIME: 2025-02-25 08:46:00 -- STEP: 6828/36938 -- GLOBAL_STEP: 708650 | > loss_text_ce: 0.046059269458055496 (0.034902569576394) | > loss_mel_ce: 2.442866086959839 (nan) | > loss: 0.009876688942313194 (nan) | > current_lr: 5e-06 | > step_time: 0.1785 (0.13698015404362116) | > loader_time: 0.0048 (0.0046496795295062725)  --> TIME: 2025-02-25 08:46:10 -- STEP: 6878/36938 -- GLOBAL_STEP: 708700 | > loss_text_ce: 0.04639243334531784 (0.034896600572405775) | > loss_mel_ce: 3.306980848312378 (nan) | > loss: 0.013307037763297558 (nan) | > current_lr: 5e-06 | > step_time: 0.1912 (0.13699414245470146) | > loader_time: 0.0048 (0.0046497995196888965)  --> TIME: 2025-02-25 08:46:19 -- STEP: 6928/36938 -- GLOBAL_STEP: 708750 | > loss_text_ce: 0.03229242563247681 (0.03489593665508595) | > loss_mel_ce: 3.156001567840576 (nan) | > loss: 0.012651960365474224 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.1369942952179746) | > loader_time: 0.0046 (0.0046495749479628515)  --> TIME: 2025-02-25 08:46:28 -- STEP: 6978/36938 -- GLOBAL_STEP: 708800 | > loss_text_ce: 0.022669441998004913 (0.03489789486293879) | > loss_mel_ce: 3.351289987564087 (nan) | > loss: 0.013388728722929955 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13697092093721658) | > loader_time: 0.0046 (0.004649386873351833)  --> TIME: 2025-02-25 08:46:38 -- STEP: 7028/36938 -- GLOBAL_STEP: 708850 | > loss_text_ce: 0.02755488082766533 (0.034901352922204636) | > loss_mel_ce: 2.9559178352355957 (nan) | > loss: 0.011839178390800953 (nan) | > current_lr: 5e-06 | > step_time: 0.1878 (0.13700566864447916) | > loader_time: 0.0048 (0.004649751859424998)  --> TIME: 2025-02-25 08:46:48 -- STEP: 7078/36938 -- GLOBAL_STEP: 708900 | > loss_text_ce: 0.03926355391740799 (0.034908331345238255) | > loss_mel_ce: 2.7914750576019287 (nan) | > loss: 0.011233090423047543 (nan) | > current_lr: 5e-06 | > step_time: 0.1868 (0.1370219490089537) | > loader_time: 0.0048 (0.004649369923466838)  --> TIME: 2025-02-25 08:46:57 -- STEP: 7128/36938 -- GLOBAL_STEP: 708950 | > loss_text_ce: 0.041110895574092865 (0.03490921167521409) | > loss_mel_ce: 2.676393747329712 (nan) | > loss: 0.010783749632537365 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.1369997655301787) | > loader_time: 0.0045 (0.0046487830236437)  --> TIME: 2025-02-25 08:47:06 -- STEP: 7178/36938 -- GLOBAL_STEP: 709000 | > loss_text_ce: 0.04004082456231117 (0.034905770381091966) | > loss_mel_ce: 2.2269604206085205 (nan) | > loss: 0.00899603683501482 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1369662594084425) | > loader_time: 0.0045 (0.004648272391314218)  --> TIME: 2025-02-25 08:47:15 -- STEP: 7228/36938 -- GLOBAL_STEP: 709050 | > loss_text_ce: 0.03346690535545349 (0.034901117798385145) | > loss_mel_ce: 2.294342041015625 (nan) | > loss: 0.009237336926162243 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13697329152006463) | > loader_time: 0.0069 (0.004648404518543313)  --> TIME: 2025-02-25 08:47:24 -- STEP: 7278/36938 -- GLOBAL_STEP: 709100 | > loss_text_ce: 0.03210768103599548 (0.03489759953704832) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13696235685303976) | > loader_time: 0.0047 (0.004648182542155678)  --> TIME: 2025-02-25 08:47:34 -- STEP: 7328/36938 -- GLOBAL_STEP: 709150 | > loss_text_ce: 0.03480113297700882 (0.03490893424457585) | > loss_mel_ce: 2.9375088214874268 (nan) | > loss: 0.011794881895184517 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.1369489375745888) | > loader_time: 0.0046 (0.0046479606667460234)  --> TIME: 2025-02-25 08:47:43 -- STEP: 7378/36938 -- GLOBAL_STEP: 709200 | > loss_text_ce: 0.029619114473462105 (0.034908315467383794) | > loss_mel_ce: 2.6104772090911865 (nan) | > loss: 0.010476573370397091 (nan) | > current_lr: 5e-06 | > step_time: 0.1523 (0.13697958115349962) | > loader_time: 0.0047 (0.004648159725874792)  --> TIME: 2025-02-25 08:47:52 -- STEP: 7428/36938 -- GLOBAL_STEP: 709250 | > loss_text_ce: 0.033852867782115936 (0.034910728887508805) | > loss_mel_ce: 2.4767885208129883 (nan) | > loss: 0.00996286328881979 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1369729817523455) | > loader_time: 0.0046 (0.0046478928772428615)  --> TIME: 2025-02-25 08:48:02 -- STEP: 7478/36938 -- GLOBAL_STEP: 709300 | > loss_text_ce: 0.03221208602190018 (0.03490824254372131) | > loss_mel_ce: 2.0872368812561035 (nan) | > loss: 0.008410511538386345 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13697347034582916) | > loader_time: 0.0045 (0.004647746415149714)  --> TIME: 2025-02-25 08:48:11 -- STEP: 7528/36938 -- GLOBAL_STEP: 709350 | > loss_text_ce: 0.0444817878305912 (0.03491170157880813) | > loss_mel_ce: 3.1429433822631836 (nan) | > loss: 0.012648513540625572 (nan) | > current_lr: 5e-06 | > step_time: 0.1842 (0.13695812538883784) | > loader_time: 0.0048 (0.004647768550927522)  --> TIME: 2025-02-25 08:48:20 -- STEP: 7578/36938 -- GLOBAL_STEP: 709400 | > loss_text_ce: 0.03142299875617027 (0.03490988556841292) | > loss_mel_ce: 0.028354432433843613 (nan) | > loss: 0.00023721203615423292 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13693770902331148) | > loader_time: 0.0046 (0.004647523219807532)  --> TIME: 2025-02-25 08:48:29 -- STEP: 7628/36938 -- GLOBAL_STEP: 709450 | > loss_text_ce: 0.039022430777549744 (0.034917578281366775) | > loss_mel_ce: 0.2832275331020355 (nan) | > loss: 0.0012787696905434132 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13691947034471766) | > loader_time: 0.0045 (0.004647277322938188)  --> TIME: 2025-02-25 08:48:39 -- STEP: 7678/36938 -- GLOBAL_STEP: 709500 | > loss_text_ce: 0.03597019240260124 (0.03490786867273958) | > loss_mel_ce: 0.1328887939453125 (nan) | > loss: 0.0006700754165649414 (nan) | > current_lr: 5e-06 | > step_time: 0.1513 (0.13688863482007033) | > loader_time: 0.0048 (0.0046469096126392365)  --> TIME: 2025-02-25 08:48:47 -- STEP: 7728/36938 -- GLOBAL_STEP: 709550 | > loss_text_ce: 0.023688284680247307 (0.03491250126343983) | > loss_mel_ce: 2.7975308895111084 (nan) | > loss: 0.011195315048098564 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13684204615668005) | > loader_time: 0.0045 (0.004646791712097492)  --> TIME: 2025-02-25 08:48:57 -- STEP: 7778/36938 -- GLOBAL_STEP: 709600 | > loss_text_ce: 0.039164695888757706 (0.03491260327770435) | > loss_mel_ce: 2.777118444442749 (nan) | > loss: 0.011175727471709251 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13682687353064038) | > loader_time: 0.0046 (0.0046463175155902455)  --> TIME: 2025-02-25 08:49:06 -- STEP: 7828/36938 -- GLOBAL_STEP: 709650 | > loss_text_ce: 0.034030936658382416 (0.034910722381083566) | > loss_mel_ce: 3.4483747482299805 (nan) | > loss: 0.013819070532917976 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13681346485672052) | > loader_time: 0.0045 (0.004646125684054937)  --> TIME: 2025-02-25 08:49:15 -- STEP: 7878/36938 -- GLOBAL_STEP: 709700 | > loss_text_ce: 0.03391805291175842 (0.034910640570657264) | > loss_mel_ce: 3.2337100505828857 (nan) | > loss: 0.012966779060661793 (nan) | > current_lr: 5e-06 | > step_time: 0.1889 (0.13680899203746705) | > loader_time: 0.0048 (0.004645634738586062)  --> TIME: 2025-02-25 08:49:25 -- STEP: 7928/36938 -- GLOBAL_STEP: 709750 | > loss_text_ce: 0.029881572350859642 (0.034905908547654685) | > loss_mel_ce: 2.5574889183044434 (nan) | > loss: 0.010267343372106552 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.1368165263674937) | > loader_time: 0.0046 (0.004645575608543135)  --> TIME: 2025-02-25 08:49:34 -- STEP: 7978/36938 -- GLOBAL_STEP: 709800 | > loss_text_ce: 0.03623737022280693 (0.03489775965004258) | > loss_mel_ce: 2.5825679302215576 (nan) | > loss: 0.010392085649073124 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13682264847156564) | > loader_time: 0.0045 (0.004645554814370952)  --> TIME: 2025-02-25 08:49:44 -- STEP: 8028/36938 -- GLOBAL_STEP: 709850 | > loss_text_ce: 0.047945890575647354 (0.03489519309613977) | > loss_mel_ce: 3.1115074157714844 (nan) | > loss: 0.012537513859570026 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13684551411145487) | > loader_time: 0.0046 (0.004645687166469626)  --> TIME: 2025-02-25 08:49:53 -- STEP: 8078/36938 -- GLOBAL_STEP: 709900 | > loss_text_ce: 0.03717799484729767 (0.03488685849071275) | > loss_mel_ce: 2.7589797973632812 (nan) | > loss: 0.011095864698290825 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13680997512280943) | > loader_time: 0.0045 (0.004645562638266005)  --> TIME: 2025-02-25 08:50:02 -- STEP: 8128/36938 -- GLOBAL_STEP: 709950 | > loss_text_ce: 0.04153284803032875 (0.03489490007643862) | > loss_mel_ce: 3.5170094966888428 (nan) | > loss: 0.01412119995802641 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.1368464138505498) | > loader_time: 0.0048 (0.004646725571296339)  --> TIME: 2025-02-25 08:50:12 -- STEP: 8178/36938 -- GLOBAL_STEP: 710000 | > loss_text_ce: 0.049514804035425186 (0.03489998169801346) | > loss_mel_ce: 4.293749809265137 (nan) | > loss: 0.017235178500413895 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.13684395199796937) | > loader_time: 0.0046 (0.004646557267763462) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_710000.pth  --> TIME: 2025-02-25 08:50:27 -- STEP: 8228/36938 -- GLOBAL_STEP: 710050 | > loss_text_ce: 0.03146132826805115 (0.034900786482249466) | > loss_mel_ce: 2.9385905265808105 (nan) | > loss: 0.011785920709371567 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13683541622565235) | > loader_time: 0.0045 (0.004646463393005972)  --> TIME: 2025-02-25 08:50:36 -- STEP: 8278/36938 -- GLOBAL_STEP: 710100 | > loss_text_ce: 0.027772804722189903 (0.034907505816574515) | > loss_mel_ce: 2.7620303630828857 (nan) | > loss: 0.011070648208260536 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13683200622705155) | > loader_time: 0.0046 (0.004646811602800524)  --> TIME: 2025-02-25 08:50:45 -- STEP: 8328/36938 -- GLOBAL_STEP: 710150 | > loss_text_ce: 0.04131443798542023 (0.03491106120018257) | > loss_mel_ce: 1.8523355722427368 (nan) | > loss: 0.007514484692364931 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13680860354394223) | > loader_time: 0.0045 (0.004646611288118311)  --> TIME: 2025-02-25 08:50:54 -- STEP: 8378/36938 -- GLOBAL_STEP: 710200 | > loss_text_ce: 0.03001645766198635 (0.0349164450087371) | > loss_mel_ce: 3.1046652793884277 (nan) | > loss: 0.012439213693141937 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1367992725313124) | > loader_time: 0.0045 (0.004646536728509937)  --> TIME: 2025-02-25 08:51:03 -- STEP: 8428/36938 -- GLOBAL_STEP: 710250 | > loss_text_ce: 0.030807550996541977 (0.03491245208281129) | > loss_mel_ce: 2.675586462020874 (nan) | > loss: 0.010739658959209919 (nan) | > current_lr: 5e-06 | > step_time: 0.1513 (0.13677370893372495) | > loader_time: 0.0046 (0.00464654240384167)  --> TIME: 2025-02-25 08:51:13 -- STEP: 8478/36938 -- GLOBAL_STEP: 710300 | > loss_text_ce: 0.034401796758174896 (0.034917240180351916) | > loss_mel_ce: 3.1880195140838623 (nan) | > loss: 0.01278738770633936 (nan) | > current_lr: 5e-06 | > step_time: 0.1623 (0.13678387097801695) | > loader_time: 0.0047 (0.004646575909286317)  --> TIME: 2025-02-25 08:51:22 -- STEP: 8528/36938 -- GLOBAL_STEP: 710350 | > loss_text_ce: 0.037300221621990204 (0.03492246598988591) | > loss_mel_ce: 3.4079060554504395 (nan) | > loss: 0.013671454042196274 (nan) | > current_lr: 5e-06 | > step_time: 0.165 (0.1368217363701978) | > loader_time: 0.0047 (0.004646623671390322)  --> TIME: 2025-02-25 08:51:32 -- STEP: 8578/36938 -- GLOBAL_STEP: 710400 | > loss_text_ce: 0.03604113683104515 (0.03491644247031793) | > loss_mel_ce: 2.7260398864746094 (nan) | > loss: 0.010960639454424381 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13681243160271842) | > loader_time: 0.0048 (0.004646870605760473)  --> TIME: 2025-02-25 08:51:41 -- STEP: 8628/36938 -- GLOBAL_STEP: 710450 | > loss_text_ce: 0.0320143960416317 (0.034918189913921226) | > loss_mel_ce: 1.0338233709335327 (nan) | > loss: 0.0042295148596167564 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.1367874700104579) | > loader_time: 0.0045 (0.004646585752525202)  --> TIME: 2025-02-25 08:51:50 -- STEP: 8678/36938 -- GLOBAL_STEP: 710500 | > loss_text_ce: 0.03458555042743683 (0.03491696454191801) | > loss_mel_ce: 3.132227897644043 (nan) | > loss: 0.012566720135509968 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.1367791560312165) | > loader_time: 0.0046 (0.004646673815732287)  --> TIME: 2025-02-25 08:51:59 -- STEP: 8728/36938 -- GLOBAL_STEP: 710550 | > loss_text_ce: 0.03355597332119942 (0.03491962847364315) | > loss_mel_ce: 1.5164718627929688 (nan) | > loss: 0.006150904577225447 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.1367877285268525) | > loader_time: 0.0047 (0.004646530100012128)  --> TIME: 2025-02-25 08:52:08 -- STEP: 8778/36938 -- GLOBAL_STEP: 710600 | > loss_text_ce: 0.027397938072681427 (0.03492337783737454) | > loss_mel_ce: 2.6794471740722656 (nan) | > loss: 0.01074144896119833 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13673843908864042) | > loader_time: 0.0046 (0.004646207890279145)  --> TIME: 2025-02-25 08:52:18 -- STEP: 8828/36938 -- GLOBAL_STEP: 710650 | > loss_text_ce: 0.03606230765581131 (0.03492580033620572) | > loss_mel_ce: 3.0287322998046875 (nan) | > loss: 0.012161883525550365 (nan) | > current_lr: 5e-06 | > step_time: 0.1769 (0.13673088882214707) | > loader_time: 0.0048 (0.004646055126924775)  --> TIME: 2025-02-25 08:52:27 -- STEP: 8878/36938 -- GLOBAL_STEP: 710700 | > loss_text_ce: 0.04510392248630524 (0.0349207917048118) | > loss_mel_ce: 2.948918581008911 (nan) | > loss: 0.011881043203175068 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13672493454249357) | > loader_time: 0.0045 (0.004646374046547396)  --> TIME: 2025-02-25 08:52:36 -- STEP: 8928/36938 -- GLOBAL_STEP: 710750 | > loss_text_ce: 0.029977886006236076 (0.03492048692563621) | > loss_mel_ce: 2.3711671829223633 (nan) | > loss: 0.009528353810310364 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.13672762548410794) | > loader_time: 0.0046 (0.004646343703124682)  --> TIME: 2025-02-25 08:52:45 -- STEP: 8978/36938 -- GLOBAL_STEP: 710800 | > loss_text_ce: 0.03497801348567009 (0.03491664021067423) | > loss_mel_ce: 2.9932291507720947 (nan) | > loss: 0.012016694992780685 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.13670715122027474) | > loader_time: 0.0048 (0.004646163657025468)  --> TIME: 2025-02-25 08:52:55 -- STEP: 9028/36938 -- GLOBAL_STEP: 710850 | > loss_text_ce: 0.033854562789201736 (0.03491184817130658) | > loss_mel_ce: 3.0550341606140137 (nan) | > loss: 0.01225749496370554 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.1367240080170049) | > loader_time: 0.0048 (0.0046460382643662874)  --> TIME: 2025-02-25 08:53:04 -- STEP: 9078/36938 -- GLOBAL_STEP: 710900 | > loss_text_ce: 0.03166070580482483 (0.03491622268651707) | > loss_mel_ce: 2.6778857707977295 (nan) | > loss: 0.010752169415354729 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.1367358687061078) | > loader_time: 0.0046 (0.004645829107247145)  --> TIME: 2025-02-25 08:53:14 -- STEP: 9128/36938 -- GLOBAL_STEP: 710950 | > loss_text_ce: 0.03652849420905113 (0.03492299057830988) | > loss_mel_ce: 3.014723300933838 (nan) | > loss: 0.012108143419027328 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13672421892205452) | > loader_time: 0.0045 (0.004645429087980705)  --> TIME: 2025-02-25 08:53:23 -- STEP: 9178/36938 -- GLOBAL_STEP: 711000 | > loss_text_ce: 0.037620026618242264 (0.034921992695278375) | > loss_mel_ce: 3.2947065830230713 (nan) | > loss: 0.013223519548773766 (nan) | > current_lr: 5e-06 | > step_time: 0.1705 (0.13672177945899955) | > loader_time: 0.0047 (0.004645068366482357)  --> TIME: 2025-02-25 08:53:32 -- STEP: 9228/36938 -- GLOBAL_STEP: 711050 | > loss_text_ce: 0.03541698679327965 (0.03493114521815363) | > loss_mel_ce: 0.08071645349264145 (nan) | > loss: 0.0004608469898812473 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13669155043960426) | > loader_time: 0.0046 (0.004644837351552222)  --> TIME: 2025-02-25 08:53:41 -- STEP: 9278/36938 -- GLOBAL_STEP: 711100 | > loss_text_ce: 0.03471257910132408 (0.034933194010669945) | > loss_mel_ce: 2.36997127532959 (nan) | > loss: 0.009542396292090416 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13667613590398398) | > loader_time: 0.0045 (0.004644514980386058)  --> TIME: 2025-02-25 08:53:50 -- STEP: 9328/36938 -- GLOBAL_STEP: 711150 | > loss_text_ce: 0.0363764762878418 (0.034926500455072874) | > loss_mel_ce: 2.1648943424224854 (nan) | > loss: 0.008735202252864838 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.13666565966749308) | > loader_time: 0.0047 (0.004644489896645062)  --> TIME: 2025-02-25 08:54:00 -- STEP: 9378/36938 -- GLOBAL_STEP: 711200 | > loss_text_ce: 0.033074039965867996 (0.03493062225667964) | > loss_mel_ce: 2.728952169418335 (nan) | > loss: 0.010960422456264496 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13667822664646923) | > loader_time: 0.0046 (0.004644479469898084)  --> TIME: 2025-02-25 08:54:09 -- STEP: 9428/36938 -- GLOBAL_STEP: 711250 | > loss_text_ce: 0.027358559891581535 (0.03493337032300602) | > loss_mel_ce: 2.746882200241089 (nan) | > loss: 0.011008892208337784 (nan) | > current_lr: 5e-06 | > step_time: 0.1734 (0.13670453382438694) | > loader_time: 0.0046 (0.004644379228366143)  --> TIME: 2025-02-25 08:54:19 -- STEP: 9478/36938 -- GLOBAL_STEP: 711300 | > loss_text_ce: 0.039883941411972046 (0.03492722387892727) | > loss_mel_ce: 2.341395378112793 (nan) | > loss: 0.009449521079659462 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.1367047309724562) | > loader_time: 0.0047 (0.004644144886932084)  --> TIME: 2025-02-25 08:54:28 -- STEP: 9528/36938 -- GLOBAL_STEP: 711350 | > loss_text_ce: 0.041391193866729736 (0.03492599646715276) | > loss_mel_ce: 2.6286470890045166 (nan) | > loss: 0.010595390573143959 (nan) | > current_lr: 5e-06 | > step_time: 0.1609 (0.1367325361139331) | > loader_time: 0.0049 (0.004644155752548745)  --> TIME: 2025-02-25 08:54:38 -- STEP: 9578/36938 -- GLOBAL_STEP: 711400 | > loss_text_ce: 0.032608676701784134 (0.034934155774934895) | > loss_mel_ce: 3.4350316524505615 (nan) | > loss: 0.013760478235781193 (nan) | > current_lr: 5e-06 | > step_time: 0.178 (0.13674156298391482) | > loader_time: 0.0048 (0.004644405321578042)  --> TIME: 2025-02-25 08:54:47 -- STEP: 9628/36938 -- GLOBAL_STEP: 711450 | > loss_text_ce: 0.031201228499412537 (0.03493201651352732) | > loss_mel_ce: 3.3547887802124023 (nan) | > loss: 0.013436468318104744 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.13675543080239988) | > loader_time: 0.0048 (0.004644225507442219)  --> TIME: 2025-02-25 08:54:57 -- STEP: 9678/36938 -- GLOBAL_STEP: 711500 | > loss_text_ce: 0.043921906501054764 (0.03493202708060679) | > loss_mel_ce: 2.900843620300293 (nan) | > loss: 0.011685578152537346 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13676898416395836) | > loader_time: 0.0045 (0.004644454498629997)  --> TIME: 2025-02-25 08:55:06 -- STEP: 9728/36938 -- GLOBAL_STEP: 711550 | > loss_text_ce: 0.03713943064212799 (0.03493438303590461) | > loss_mel_ce: 2.7395522594451904 (nan) | > loss: 0.011018618009984493 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13676882454341213) | > loader_time: 0.0046 (0.004644556363162247)  --> TIME: 2025-02-25 08:55:15 -- STEP: 9778/36938 -- GLOBAL_STEP: 711600 | > loss_text_ce: 0.04160166531801224 (0.03493005408636648) | > loss_mel_ce: 3.619403600692749 (nan) | > loss: 0.014527799561619759 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13675589840893515) | > loader_time: 0.0047 (0.004644310506093874)  --> TIME: 2025-02-25 08:55:24 -- STEP: 9828/36938 -- GLOBAL_STEP: 711650 | > loss_text_ce: 0.0387311615049839 (0.03492777149932501) | > loss_mel_ce: 1.614652395248413 (nan) | > loss: 0.006561046466231346 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13672929000311135) | > loader_time: 0.0045 (0.004643899010699806)  --> TIME: 2025-02-25 08:55:33 -- STEP: 9878/36938 -- GLOBAL_STEP: 711700 | > loss_text_ce: 0.03085583820939064 (0.034924088659356554) | > loss_mel_ce: 3.487898111343384 (nan) | > loss: 0.013963310047984123 (nan) | > current_lr: 5e-06 | > step_time: 0.1811 (0.1367026820166392) | > loader_time: 0.0049 (0.004643903832899999)  --> TIME: 2025-02-25 08:55:42 -- STEP: 9928/36938 -- GLOBAL_STEP: 711750 | > loss_text_ce: 0.02653687819838524 (0.03492746330342882) | > loss_mel_ce: 2.363006353378296 (nan) | > loss: 0.009482314810156822 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13668526827952862) | > loader_time: 0.005 (0.004644148826214896)  --> TIME: 2025-02-25 08:55:52 -- STEP: 9978/36938 -- GLOBAL_STEP: 711800 | > loss_text_ce: 0.024825148284435272 (0.03492372200967488) | > loss_mel_ce: 2.093560218811035 (nan) | > loss: 0.008406291715800762 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13670582450160224) | > loader_time: 0.0046 (0.004644028455658178)  --> TIME: 2025-02-25 08:56:01 -- STEP: 10028/36938 -- GLOBAL_STEP: 711850 | > loss_text_ce: 0.021857857704162598 (0.03492097002924024) | > loss_mel_ce: 2.010329246520996 (nan) | > loss: 0.008064234629273415 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13667485734881024) | > loader_time: 0.0045 (0.004643797922001263)  --> TIME: 2025-02-25 08:56:10 -- STEP: 10078/36938 -- GLOBAL_STEP: 711900 | > loss_text_ce: 0.026522358879446983 (0.03491968469293917) | > loss_mel_ce: 3.3012826442718506 (nan) | > loss: 0.013205576688051224 (nan) | > current_lr: 5e-06 | > step_time: 0.177 (0.13668156976050713) | > loader_time: 0.005 (0.004643974807627221)  --> TIME: 2025-02-25 08:56:20 -- STEP: 10128/36938 -- GLOBAL_STEP: 711950 | > loss_text_ce: 0.026629481464624405 (0.03492122483208003) | > loss_mel_ce: 2.276013135910034 (nan) | > loss: 0.009137471206486225 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13668105578253092) | > loader_time: 0.0045 (0.0046441921549385794)  --> TIME: 2025-02-25 08:56:29 -- STEP: 10178/36938 -- GLOBAL_STEP: 712000 | > loss_text_ce: 0.033938731998205185 (0.034926429313495176) | > loss_mel_ce: 0.1162109375 (nan) | > loss: 0.000595832068938762 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13666854733496941) | > loader_time: 0.0045 (0.004643897055455201)  --> TIME: 2025-02-25 08:56:38 -- STEP: 10228/36938 -- GLOBAL_STEP: 712050 | > loss_text_ce: 0.037492819130420685 (0.03492207890179753) | > loss_mel_ce: 0.0064722695387899876 (nan) | > loss: 0.0001744646579027176 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.1366455381532927) | > loader_time: 0.0045 (0.004643777850833225)  --> TIME: 2025-02-25 08:56:47 -- STEP: 10278/36938 -- GLOBAL_STEP: 712100 | > loss_text_ce: 0.034093406051397324 (0.03492220430188177) | > loss_mel_ce: 1.8870849609375 (nan) | > loss: 0.007623724173754454 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1366602310667621) | > loader_time: 0.0045 (0.004643658831741688)  --> TIME: 2025-02-25 08:56:56 -- STEP: 10328/36938 -- GLOBAL_STEP: 712150 | > loss_text_ce: 0.027339503169059753 (0.03491649666576002) | > loss_mel_ce: 3.2082955837249756 (nan) | > loss: 0.012839822098612785 (nan) | > current_lr: 5e-06 | > step_time: 0.1851 (0.13663639946560144) | > loader_time: 0.0048 (0.004643487801023646)  --> TIME: 2025-02-25 08:57:06 -- STEP: 10378/36938 -- GLOBAL_STEP: 712200 | > loss_text_ce: 0.037799153476953506 (0.03491366907629698) | > loss_mel_ce: 3.0235397815704346 (nan) | > loss: 0.012148170731961727 (nan) | > current_lr: 5e-06 | > step_time: 0.1756 (0.1366410899607984) | > loader_time: 0.005 (0.004643338795777901)  --> TIME: 2025-02-25 08:57:15 -- STEP: 10428/36938 -- GLOBAL_STEP: 712250 | > loss_text_ce: 0.03880951181054115 (0.03490889978805475) | > loss_mel_ce: 2.794586658477783 (nan) | > loss: 0.011243636719882488 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13663917633597372) | > loader_time: 0.0046 (0.004643064831059515)  --> TIME: 2025-02-25 08:57:24 -- STEP: 10478/36938 -- GLOBAL_STEP: 712300 | > loss_text_ce: 0.029505908489227295 (0.03491086040760885) | > loss_mel_ce: 1.8175970315933228 (nan) | > loss: 0.007329773623496294 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13662817999706414) | > loader_time: 0.0045 (0.004642769634598518)  --> TIME: 2025-02-25 08:57:34 -- STEP: 10528/36938 -- GLOBAL_STEP: 712350 | > loss_text_ce: 0.036584533751010895 (0.034908657875669116) | > loss_mel_ce: 3.001314878463745 (nan) | > loss: 0.01205515768378973 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13664875587159805) | > loader_time: 0.0046 (0.004642646657840828)  --> TIME: 2025-02-25 08:57:43 -- STEP: 10578/36938 -- GLOBAL_STEP: 712400 | > loss_text_ce: 0.03976510092616081 (0.034913482709537064) | > loss_mel_ce: 2.7264742851257324 (nan) | > loss: 0.010977141559123993 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13664543085500439) | > loader_time: 0.0045 (0.0046424231472437395)  --> TIME: 2025-02-25 08:57:53 -- STEP: 10628/36938 -- GLOBAL_STEP: 712450 | > loss_text_ce: 0.03507513552904129 (0.03491031493748691) | > loss_mel_ce: 2.9551873207092285 (nan) | > loss: 0.011866121552884579 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.1366549368679957) | > loader_time: 0.0047 (0.004642522240833639)  --> TIME: 2025-02-25 08:58:02 -- STEP: 10678/36938 -- GLOBAL_STEP: 712500 | > loss_text_ce: 0.027870334684848785 (0.03490758608856753) | > loss_mel_ce: 1.3719055652618408 (nan) | > loss: 0.00555466627702117 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13663721812488208) | > loader_time: 0.0046 (0.0046423410605705585)  --> TIME: 2025-02-25 08:58:11 -- STEP: 10728/36938 -- GLOBAL_STEP: 712550 | > loss_text_ce: 0.03505716845393181 (0.034907415673038636) | > loss_mel_ce: 2.271106719970703 (nan) | > loss: 0.009151443839073181 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.1366217107936403) | > loader_time: 0.0046 (0.0046423977653452285)  --> TIME: 2025-02-25 08:58:20 -- STEP: 10778/36938 -- GLOBAL_STEP: 712600 | > loss_text_ce: 0.03354652225971222 (0.034910099026911945) | > loss_mel_ce: 2.4348013401031494 (nan) | > loss: 0.009795031510293484 (nan) | > current_lr: 5e-06 | > step_time: 0.1086 (0.13662195199052293) | > loader_time: 0.0045 (0.0046427973702834816)  --> TIME: 2025-02-25 08:58:30 -- STEP: 10828/36938 -- GLOBAL_STEP: 712650 | > loss_text_ce: 0.034552644938230515 (0.03490668952082814) | > loss_mel_ce: 3.2621288299560547 (nan) | > loss: 0.013082070276141167 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.1366456419857048) | > loader_time: 0.0048 (0.004642758657450525)  --> TIME: 2025-02-25 08:58:39 -- STEP: 10878/36938 -- GLOBAL_STEP: 712700 | > loss_text_ce: 0.03838922083377838 (0.03490486080039268) | > loss_mel_ce: 2.996922731399536 (nan) | > loss: 0.01204488892108202 (nan) | > current_lr: 5e-06 | > step_time: 0.1527 (0.13666304309790345) | > loader_time: 0.0047 (0.00464275269456638)  --> TIME: 2025-02-25 08:58:49 -- STEP: 10928/36938 -- GLOBAL_STEP: 712750 | > loss_text_ce: 0.029733756557106972 (0.03490705289722836) | > loss_mel_ce: 0.15532226860523224 (nan) | > loss: 0.0007343493634834886 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.13665326242914733) | > loader_time: 0.0046 (0.004642633707596668)  --> TIME: 2025-02-25 08:58:58 -- STEP: 10978/36938 -- GLOBAL_STEP: 712800 | > loss_text_ce: 0.03254086524248123 (0.03490749268047197) | > loss_mel_ce: 1.9862587451934814 (nan) | > loss: 0.008011110126972198 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13666436856378147) | > loader_time: 0.0046 (0.004642729356132989)  --> TIME: 2025-02-25 08:59:07 -- STEP: 11028/36938 -- GLOBAL_STEP: 712850 | > loss_text_ce: 0.03770611062645912 (0.0349112920086265) | > loss_mel_ce: 3.585627317428589 (nan) | > loss: 0.014378308318555355 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.1366512660622214) | > loader_time: 0.0046 (0.004642775299153914)  --> TIME: 2025-02-25 08:59:17 -- STEP: 11078/36938 -- GLOBAL_STEP: 712900 | > loss_text_ce: 0.05129607021808624 (0.03491416313741116) | > loss_mel_ce: 3.454663038253784 (nan) | > loss: 0.013912536203861237 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13666230962043652) | > loader_time: 0.0046 (0.004642651343220834)  --> TIME: 2025-02-25 08:59:26 -- STEP: 11128/36938 -- GLOBAL_STEP: 712950 | > loss_text_ce: 0.03999935835599899 (0.03491268199870161) | > loss_mel_ce: 2.5255064964294434 (nan) | > loss: 0.01018057856708765 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13665568706459716) | > loader_time: 0.0044 (0.004642445736013684)  --> TIME: 2025-02-25 08:59:35 -- STEP: 11178/36938 -- GLOBAL_STEP: 713000 | > loss_text_ce: 0.03566985949873924 (0.03490183621954569) | > loss_mel_ce: 3.3941519260406494 (nan) | > loss: 0.013610404916107655 (nan) | > current_lr: 5e-06 | > step_time: 0.1441 (0.13665593809423748) | > loader_time: 0.0047 (0.004642425271943911)  --> TIME: 2025-02-25 08:59:44 -- STEP: 11228/36938 -- GLOBAL_STEP: 713050 | > loss_text_ce: 0.04787640646100044 (0.034903235002502) | > loss_mel_ce: 1.614904522895813 (nan) | > loss: 0.006598337087780237 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13664443509027518) | > loader_time: 0.0045 (0.004642214879558164)  --> TIME: 2025-02-25 08:59:53 -- STEP: 11278/36938 -- GLOBAL_STEP: 713100 | > loss_text_ce: 0.0406990572810173 (0.03490651699062473) | > loss_mel_ce: 2.7520344257354736 (nan) | > loss: 0.01108227577060461 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.136623554014953) | > loader_time: 0.0047 (0.00464189164624569)  --> TIME: 2025-02-25 09:00:03 -- STEP: 11328/36938 -- GLOBAL_STEP: 713150 | > loss_text_ce: 0.01995563879609108 (0.03490134106099195) | > loss_mel_ce: 2.673567056655884 (nan) | > loss: 0.010688582435250282 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13662647178112505) | > loader_time: 0.0044 (0.0046417779673290985)  --> TIME: 2025-02-25 09:00:12 -- STEP: 11378/36938 -- GLOBAL_STEP: 713200 | > loss_text_ce: 0.038584884256124496 (0.03490129635697271) | > loss_mel_ce: 2.9383597373962402 (nan) | > loss: 0.011813273653388023 (nan) | > current_lr: 5e-06 | > step_time: 0.156 (0.13663049317512702) | > loader_time: 0.0047 (0.004641682407226959)  --> TIME: 2025-02-25 09:00:21 -- STEP: 11428/36938 -- GLOBAL_STEP: 713250 | > loss_text_ce: 0.03537583723664284 (0.034903284136693385) | > loss_mel_ce: 1.3857421875 (nan) | > loss: 0.00563935749232769 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13661972008512613) | > loader_time: 0.0046 (0.004641571326985177)  --> TIME: 2025-02-25 09:00:31 -- STEP: 11478/36938 -- GLOBAL_STEP: 713300 | > loss_text_ce: 0.03823692351579666 (0.0349049370620339) | > loss_mel_ce: 2.865419387817383 (nan) | > loss: 0.01152244582772255 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13663557691635655) | > loader_time: 0.0045 (0.004641614634933794)  --> TIME: 2025-02-25 09:00:40 -- STEP: 11528/36938 -- GLOBAL_STEP: 713350 | > loss_text_ce: 0.050816912204027176 (0.03490416373893357) | > loss_mel_ce: 2.762075424194336 (nan) | > loss: 0.011162272654473782 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1366221531036419) | > loader_time: 0.0044 (0.004641385375419331)  --> TIME: 2025-02-25 09:00:49 -- STEP: 11578/36938 -- GLOBAL_STEP: 713400 | > loss_text_ce: 0.028179394081234932 (0.034905252141318344) | > loss_mel_ce: 3.4191040992736816 (nan) | > loss: 0.013679697178304195 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13661523976460327) | > loader_time: 0.0045 (0.004641346989939895)  --> TIME: 2025-02-25 09:00:59 -- STEP: 11628/36938 -- GLOBAL_STEP: 713450 | > loss_text_ce: 0.03605619817972183 (0.03490406279761291) | > loss_mel_ce: 2.7206525802612305 (nan) | > loss: 0.010939321480691433 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.1366328008608766) | > loader_time: 0.0047 (0.004641387177202613)  --> TIME: 2025-02-25 09:01:08 -- STEP: 11678/36938 -- GLOBAL_STEP: 713500 | > loss_text_ce: 0.03562043607234955 (0.034905905336891724) | > loss_mel_ce: 3.0139265060424805 (nan) | > loss: 0.012101377360522747 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13664154092586572) | > loader_time: 0.0045 (0.00464132042816393)  --> TIME: 2025-02-25 09:01:17 -- STEP: 11728/36938 -- GLOBAL_STEP: 713550 | > loss_text_ce: 0.0506979301571846 (0.03490636481345704) | > loss_mel_ce: 2.722200870513916 (nan) | > loss: 0.011003567837178707 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13663508054910947) | > loader_time: 0.0047 (0.004641223978378283)  --> TIME: 2025-02-25 09:01:27 -- STEP: 11778/36938 -- GLOBAL_STEP: 713600 | > loss_text_ce: 0.04263942316174507 (0.03490219147071561) | > loss_mel_ce: 3.246140241622925 (nan) | > loss: 0.013050713576376438 (nan) | > current_lr: 5e-06 | > step_time: 0.1817 (0.1366272214556582) | > loader_time: 0.0049 (0.004641105250564097)  --> TIME: 2025-02-25 09:01:36 -- STEP: 11828/36938 -- GLOBAL_STEP: 713650 | > loss_text_ce: 0.03729237616062164 (0.034903263558384995) | > loss_mel_ce: 3.0480234622955322 (nan) | > loss: 0.012243318371474743 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13662594278640097) | > loader_time: 0.0045 (0.004641030682959518)  --> TIME: 2025-02-25 09:01:46 -- STEP: 11878/36938 -- GLOBAL_STEP: 713700 | > loss_text_ce: 0.04036153852939606 (0.034909038702541834) | > loss_mel_ce: 2.828117609024048 (nan) | > loss: 0.011382854543626308 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13664366653379506) | > loader_time: 0.0046 (0.004641034523231224)  --> TIME: 2025-02-25 09:01:55 -- STEP: 11928/36938 -- GLOBAL_STEP: 713750 | > loss_text_ce: 0.03181620314717293 (0.03490466013204702) | > loss_mel_ce: 3.083895206451416 (nan) | > loss: 0.012363934889435768 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13662478809465498) | > loader_time: 0.0045 (0.004640846265233833)  --> TIME: 2025-02-25 09:02:04 -- STEP: 11978/36938 -- GLOBAL_STEP: 713800 | > loss_text_ce: 0.03764040768146515 (0.03489822913842949) | > loss_mel_ce: 2.295466423034668 (nan) | > loss: 0.009258360601961613 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13660924114992365) | > loader_time: 0.0049 (0.004640631612821358)  --> TIME: 2025-02-25 09:02:13 -- STEP: 12028/36938 -- GLOBAL_STEP: 713850 | > loss_text_ce: 0.04042484611272812 (0.03490064159913103) | > loss_mel_ce: 3.178201675415039 (nan) | > loss: 0.012772328220307827 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13660203272772348) | > loader_time: 0.0045 (0.004640538687717109)  --> TIME: 2025-02-25 09:02:22 -- STEP: 12078/36938 -- GLOBAL_STEP: 713900 | > loss_text_ce: 0.034688059240579605 (0.034898547260968076) | > loss_mel_ce: 2.9617526531219482 (nan) | > loss: 0.011890638619661331 (nan) | > current_lr: 5e-06 | > step_time: 0.1521 (0.136585609252376) | > loader_time: 0.0046 (0.004640355215184768)  --> TIME: 2025-02-25 09:02:31 -- STEP: 12128/36938 -- GLOBAL_STEP: 713950 | > loss_text_ce: 0.03788597136735916 (0.03489492329964491) | > loss_mel_ce: 3.3775577545166016 (nan) | > loss: 0.013553348369896412 (nan) | > current_lr: 5e-06 | > step_time: 0.1556 (0.1365843578426073) | > loader_time: 0.0046 (0.004640196098658541)  --> TIME: 2025-02-25 09:02:40 -- STEP: 12178/36938 -- GLOBAL_STEP: 714000 | > loss_text_ce: 0.03487968444824219 (0.03489202645315606) | > loss_mel_ce: 2.5566530227661133 (nan) | > loss: 0.010283860377967358 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.1365783713999051) | > loader_time: 0.0047 (0.00464004672675871)  --> TIME: 2025-02-25 09:02:50 -- STEP: 12228/36938 -- GLOBAL_STEP: 714050 | > loss_text_ce: 0.03812846913933754 (0.03488763161044708) | > loss_mel_ce: 2.701564073562622 (nan) | > loss: 0.010871795937418938 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13658086568029484) | > loader_time: 0.0045 (0.004639864864792367)  --> TIME: 2025-02-25 09:02:59 -- STEP: 12278/36938 -- GLOBAL_STEP: 714100 | > loss_text_ce: 0.026323873549699783 (0.03488360268592832) | > loss_mel_ce: 3.110588788986206 (nan) | > loss: 0.012448066845536232 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13656581327802256) | > loader_time: 0.0045 (0.004639650696087876)  --> TIME: 2025-02-25 09:03:08 -- STEP: 12328/36938 -- GLOBAL_STEP: 714150 | > loss_text_ce: 0.01882193796336651 (0.03487676736243882) | > loss_mel_ce: 0.00738525390625 (nan) | > loss: 0.0001039967974065803 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13656730521506652) | > loader_time: 0.0045 (0.004639641117693785)  --> TIME: 2025-02-25 09:03:18 -- STEP: 12378/36938 -- GLOBAL_STEP: 714200 | > loss_text_ce: 0.0409269705414772 (0.03487587767383949) | > loss_mel_ce: 2.7467336654663086 (nan) | > loss: 0.011062146164476871 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13657362966896588) | > loader_time: 0.0046 (0.004639507380146699)  --> TIME: 2025-02-25 09:03:27 -- STEP: 12428/36938 -- GLOBAL_STEP: 714250 | > loss_text_ce: 0.03807181864976883 (0.03487885803921548) | > loss_mel_ce: 3.14406681060791 (nan) | > loss: 0.012627535499632359 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13654229237713378) | > loader_time: 0.0046 (0.004639347112942336)  --> TIME: 2025-02-25 09:03:36 -- STEP: 12478/36938 -- GLOBAL_STEP: 714300 | > loss_text_ce: 0.036572523415088654 (0.03488008636050666) | > loss_mel_ce: 3.655980348587036 (nan) | > loss: 0.014652987942099571 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.1365405669634999) | > loader_time: 0.0046 (0.0046392794239309)  --> TIME: 2025-02-25 09:03:45 -- STEP: 12528/36938 -- GLOBAL_STEP: 714350 | > loss_text_ce: 0.036398082971572876 (0.034878830221379865) | > loss_mel_ce: 2.8495702743530273 (nan) | > loss: 0.011452256701886654 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.13655310637040538) | > loader_time: 0.0046 (0.004639136018576451)  --> TIME: 2025-02-25 09:03:55 -- STEP: 12578/36938 -- GLOBAL_STEP: 714400 | > loss_text_ce: 0.03101206198334694 (0.03488195790608757) | > loss_mel_ce: 2.7359659671783447 (nan) | > loss: 0.010980072431266308 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.1365498793221831) | > loader_time: 0.0046 (0.004638940602953248)  --> TIME: 2025-02-25 09:04:04 -- STEP: 12628/36938 -- GLOBAL_STEP: 714450 | > loss_text_ce: 0.033091891556978226 (0.03488830742317064) | > loss_mel_ce: 2.7601370811462402 (nan) | > loss: 0.011084241792559624 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13656322111009603) | > loader_time: 0.0047 (0.004638822991749065)  --> TIME: 2025-02-25 09:04:13 -- STEP: 12678/36938 -- GLOBAL_STEP: 714500 | > loss_text_ce: 0.03048231452703476 (0.03488745151051203) | > loss_mel_ce: 3.3405208587646484 (nan) | > loss: 0.013376997783780098 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.136571944671995) | > loader_time: 0.0046 (0.004638666252097721)  --> TIME: 2025-02-25 09:04:23 -- STEP: 12728/36938 -- GLOBAL_STEP: 714550 | > loss_text_ce: 0.03363746404647827 (0.03488696243803905) | > loss_mel_ce: 3.387152671813965 (nan) | > loss: 0.013574564829468727 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13657851117276215) | > loader_time: 0.0044 (0.004638436734489047)  --> TIME: 2025-02-25 09:04:32 -- STEP: 12778/36938 -- GLOBAL_STEP: 714600 | > loss_text_ce: 0.03341206535696983 (0.03489189593548217) | > loss_mel_ce: 2.6896824836730957 (nan) | > loss: 0.010805930942296982 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.1365771372938856) | > loader_time: 0.0048 (0.004638267339609994)  --> TIME: 2025-02-25 09:04:41 -- STEP: 12828/36938 -- GLOBAL_STEP: 714650 | > loss_text_ce: 0.03129606693983078 (0.03489196296244922) | > loss_mel_ce: 3.4293363094329834 (nan) | > loss: 0.013732668943703175 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.1365690514199049) | > loader_time: 0.0046 (0.004638324599419203)  --> TIME: 2025-02-25 09:04:51 -- STEP: 12878/36938 -- GLOBAL_STEP: 714700 | > loss_text_ce: 0.036347948014736176 (0.03488806750063304) | > loss_mel_ce: 3.1059436798095703 (nan) | > loss: 0.012469411827623844 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13657664419572077) | > loader_time: 0.0047 (0.004638334482533351)  --> TIME: 2025-02-25 09:05:00 -- STEP: 12928/36938 -- GLOBAL_STEP: 714750 | > loss_text_ce: 0.031000753864645958 (0.03488691646202594) | > loss_mel_ce: 2.445413112640381 (nan) | > loss: 0.009827040135860443 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13657203427340794) | > loader_time: 0.0046 (0.004638267146183718)  --> TIME: 2025-02-25 09:05:09 -- STEP: 12978/36938 -- GLOBAL_STEP: 714800 | > loss_text_ce: 0.035871800035238266 (0.034883587022875584) | > loss_mel_ce: 2.4477388858795166 (nan) | > loss: 0.009855598211288452 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13656178287189313) | > loader_time: 0.0047 (0.004638304492140945)  --> TIME: 2025-02-25 09:05:19 -- STEP: 13028/36938 -- GLOBAL_STEP: 714850 | > loss_text_ce: 0.03519178554415703 (0.034881771196638926) | > loss_mel_ce: 2.610137939453125 (nan) | > loss: 0.010497340932488441 (nan) | > current_lr: 5e-06 | > step_time: 0.1519 (0.13658288804746233) | > loader_time: 0.0046 (0.004638302004114052)  --> TIME: 2025-02-25 09:05:28 -- STEP: 13078/36938 -- GLOBAL_STEP: 714900 | > loss_text_ce: 0.03224337846040726 (0.03487837958434601) | > loss_mel_ce: 2.863227605819702 (nan) | > loss: 0.01148996502161026 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13658162539004254) | > loader_time: 0.0047 (0.004638168293687077)  --> TIME: 2025-02-25 09:05:37 -- STEP: 13128/36938 -- GLOBAL_STEP: 714950 | > loss_text_ce: 0.03416883945465088 (0.03488254084685697) | > loss_mel_ce: 3.2160308361053467 (nan) | > loss: 0.012897619046270847 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.1365652239264554) | > loader_time: 0.0046 (0.004638063115800477)  --> TIME: 2025-02-25 09:05:47 -- STEP: 13178/36938 -- GLOBAL_STEP: 715000 | > loss_text_ce: 0.04454272240400314 (0.03488102599381685) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13657253683020515) | > loader_time: 0.0046 (0.0046378886469906075)  --> TIME: 2025-02-25 09:05:56 -- STEP: 13228/36938 -- GLOBAL_STEP: 715050 | > loss_text_ce: 0.04112618789076805 (0.034877235020727954) | > loss_mel_ce: 3.1976873874664307 (nan) | > loss: 0.012852435931563377 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13656512878796975) | > loader_time: 0.0045 (0.004637637039596386)  --> TIME: 2025-02-25 09:06:05 -- STEP: 13278/36938 -- GLOBAL_STEP: 715100 | > loss_text_ce: 0.023532718420028687 (0.03488181240908266) | > loss_mel_ce: 1.3722498416900635 (nan) | > loss: 0.005538820289075375 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1365682072516654) | > loader_time: 0.0046 (0.0046374743773754485)  --> TIME: 2025-02-25 09:06:14 -- STEP: 13328/36938 -- GLOBAL_STEP: 715150 | > loss_text_ce: 0.03537733107805252 (0.034886043442895046) | > loss_mel_ce: 2.859480381011963 (nan) | > loss: 0.011487530544400215 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13654955416595876) | > loader_time: 0.0045 (0.004637204101678131)  --> TIME: 2025-02-25 09:06:24 -- STEP: 13378/36938 -- GLOBAL_STEP: 715200 | > loss_text_ce: 0.03696802631020546 (0.0348834690477352) | > loss_mel_ce: 3.12404727935791 (nan) | > loss: 0.012543711811304092 (nan) | > current_lr: 5e-06 | > step_time: 0.1507 (0.13655939040153434) | > loader_time: 0.0046 (0.004637253838708923)  --> TIME: 2025-02-25 09:06:33 -- STEP: 13428/36938 -- GLOBAL_STEP: 715250 | > loss_text_ce: 0.03472282737493515 (0.03488284298228491) | > loss_mel_ce: 2.788753032684326 (nan) | > loss: 0.011204269714653492 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1365665188672332) | > loader_time: 0.0045 (0.004637074143821928)  --> TIME: 2025-02-25 09:06:42 -- STEP: 13478/36938 -- GLOBAL_STEP: 715300 | > loss_text_ce: 0.03643958643078804 (0.03487776603485035) | > loss_mel_ce: 3.6911613941192627 (nan) | > loss: 0.014792068861424923 (nan) | > current_lr: 5e-06 | > step_time: 0.149 (0.13656885897411036) | > loader_time: 0.0046 (0.004636865603957402)  --> TIME: 2025-02-25 09:06:52 -- STEP: 13528/36938 -- GLOBAL_STEP: 715350 | > loss_text_ce: 0.03644677996635437 (0.03487980226440305) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13657480721499135) | > loader_time: 0.0048 (0.004636747924482089)  --> TIME: 2025-02-25 09:07:01 -- STEP: 13578/36938 -- GLOBAL_STEP: 715400 | > loss_text_ce: 0.04872232675552368 (0.034880841321944606) | > loss_mel_ce: 2.4105355739593506 (nan) | > loss: 0.009758960455656052 (nan) | > current_lr: 5e-06 | > step_time: 0.1068 (0.1365790794516972) | > loader_time: 0.0044 (0.00463688387613019)  --> TIME: 2025-02-25 09:07:10 -- STEP: 13628/36938 -- GLOBAL_STEP: 715450 | > loss_text_ce: 0.0420646034181118 (0.03487987706649823) | > loss_mel_ce: 2.2574706077575684 (nan) | > loss: 0.009125140495598316 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13656905094899208) | > loader_time: 0.0046 (0.004636846471819521)  --> TIME: 2025-02-25 09:07:20 -- STEP: 13678/36938 -- GLOBAL_STEP: 715500 | > loss_text_ce: 0.03158453106880188 (0.03488439414772753) | > loss_mel_ce: 2.9248299598693848 (nan) | > loss: 0.011731804348528385 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13657306510988385) | > loader_time: 0.0045 (0.004636837683464451)  --> TIME: 2025-02-25 09:07:29 -- STEP: 13728/36938 -- GLOBAL_STEP: 715550 | > loss_text_ce: 0.035622917115688324 (0.03488501869407983) | > loss_mel_ce: 2.7166078090667725 (nan) | > loss: 0.010921550914645195 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.13655913994584662) | > loader_time: 0.0048 (0.004636602055081667)  --> TIME: 2025-02-25 09:07:38 -- STEP: 13778/36938 -- GLOBAL_STEP: 715600 | > loss_text_ce: 0.030845006927847862 (0.034885288742061905) | > loss_mel_ce: 2.8831756114959717 (nan) | > loss: 0.011563573963940144 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13654889773519652) | > loader_time: 0.0045 (0.004636640004659003)  --> TIME: 2025-02-25 09:07:47 -- STEP: 13828/36938 -- GLOBAL_STEP: 715650 | > loss_text_ce: 0.04021300747990608 (0.03488112056206515) | > loss_mel_ce: 3.2225797176361084 (nan) | > loss: 0.01294759102165699 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.1365709973960941) | > loader_time: 0.0046 (0.004636693007690137)  --> TIME: 2025-02-25 09:07:57 -- STEP: 13878/36938 -- GLOBAL_STEP: 715700 | > loss_text_ce: 0.03712226822972298 (0.034884245703964485) | > loss_mel_ce: 2.300973892211914 (nan) | > loss: 0.009278159588575363 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13656439029065864) | > loader_time: 0.0046 (0.004636661259752465)  --> TIME: 2025-02-25 09:08:06 -- STEP: 13928/36938 -- GLOBAL_STEP: 715750 | > loss_text_ce: 0.036389634013175964 (0.03488210786058037) | > loss_mel_ce: 2.6715409755706787 (nan) | > loss: 0.010745757259428501 (nan) | > current_lr: 5e-06 | > step_time: 0.1817 (0.13655794620308562) | > loader_time: 0.0048 (0.004636611680338114)  --> TIME: 2025-02-25 09:08:15 -- STEP: 13978/36938 -- GLOBAL_STEP: 715800 | > loss_text_ce: 0.03273749351501465 (0.034879915866910687) | > loss_mel_ce: 1.896673560142517 (nan) | > loss: 0.007656393572688103 (nan) | > current_lr: 5e-06 | > step_time: 0.1867 (0.13657129926390932) | > loader_time: 0.005 (0.004636704350322888)  --> TIME: 2025-02-25 09:08:24 -- STEP: 14028/36938 -- GLOBAL_STEP: 715850 | > loss_text_ce: 0.03602857142686844 (0.03487995746238768) | > loss_mel_ce: 2.243049383163452 (nan) | > loss: 0.009043960832059383 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13654928039817885) | > loader_time: 0.0045 (0.004636589876429991)  --> TIME: 2025-02-25 09:08:34 -- STEP: 14078/36938 -- GLOBAL_STEP: 715900 | > loss_text_ce: 0.03330298140645027 (0.034880725561695) | > loss_mel_ce: 2.827698230743408 (nan) | > loss: 0.011353179812431335 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13655089913723412) | > loader_time: 0.0046 (0.004636720883471997)  --> TIME: 2025-02-25 09:08:43 -- STEP: 14128/36938 -- GLOBAL_STEP: 715950 | > loss_text_ce: 0.03296650946140289 (0.03488078216760022) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1815 (0.13654790516617973) | > loader_time: 0.0047 (0.004636516252433447)  --> TIME: 2025-02-25 09:08:52 -- STEP: 14178/36938 -- GLOBAL_STEP: 716000 | > loss_text_ce: 0.030482526868581772 (0.034880716527596334) | > loss_mel_ce: 2.1897192001342773 (nan) | > loss: 0.008810324594378471 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.1365613790909762) | > loader_time: 0.0045 (0.004636509476671224)  --> TIME: 2025-02-25 09:09:02 -- STEP: 14228/36938 -- GLOBAL_STEP: 716050 | > loss_text_ce: 0.04722283408045769 (0.034882421998292966) | > loss_mel_ce: 1.8429896831512451 (nan) | > loss: 0.007500843610614538 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13656365330178488) | > loader_time: 0.0045 (0.004636449645602495)  --> TIME: 2025-02-25 09:09:11 -- STEP: 14278/36938 -- GLOBAL_STEP: 716100 | > loss_text_ce: 0.03621619939804077 (0.034881096807340965) | > loss_mel_ce: 1.83966064453125 (nan) | > loss: 0.0074439565651118755 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1365456029069353) | > loader_time: 0.0045 (0.004636183525103607)  --> TIME: 2025-02-25 09:09:20 -- STEP: 14328/36938 -- GLOBAL_STEP: 716150 | > loss_text_ce: 0.02825837954878807 (0.03487714084145045) | > loss_mel_ce: 2.9554522037506104 (nan) | > loss: 0.011840121820569038 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13653814313399842) | > loader_time: 0.0046 (0.0046359648057563724)  --> TIME: 2025-02-25 09:09:30 -- STEP: 14378/36938 -- GLOBAL_STEP: 716200 | > loss_text_ce: 0.03076855279505253 (0.03487941291220022) | > loss_mel_ce: 2.6788692474365234 (nan) | > loss: 0.010752531699836254 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13656207637638404) | > loader_time: 0.0045 (0.004636074177063717)  --> TIME: 2025-02-25 09:09:39 -- STEP: 14428/36938 -- GLOBAL_STEP: 716250 | > loss_text_ce: 0.033536147326231 (0.034884297430447096) | > loss_mel_ce: 0.0453694649040699 (nan) | > loss: 0.00031311751808971167 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13654553660534852) | > loader_time: 0.0046 (0.004635936770374361)  --> TIME: 2025-02-25 09:09:48 -- STEP: 14478/36938 -- GLOBAL_STEP: 716300 | > loss_text_ce: 0.030995700508356094 (0.03488587947671177) | > loss_mel_ce: 2.582258701324463 (nan) | > loss: 0.010370057076215744 (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.13652511370216738) | > loader_time: 0.0046 (0.004635738806099426)  --> TIME: 2025-02-25 09:09:57 -- STEP: 14528/36938 -- GLOBAL_STEP: 716350 | > loss_text_ce: 0.027827493846416473 (0.03488375330336154) | > loss_mel_ce: 2.500802516937256 (nan) | > loss: 0.010034246370196342 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13654695193607497) | > loader_time: 0.0045 (0.004635814626573981)  --> TIME: 2025-02-25 09:10:07 -- STEP: 14578/36938 -- GLOBAL_STEP: 716400 | > loss_text_ce: 0.03053892031311989 (0.034881817011985804) | > loss_mel_ce: 3.2987632751464844 (nan) | > loss: 0.013211517594754696 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.1365428877099266) | > loader_time: 0.0046 (0.004635805389586413)  --> TIME: 2025-02-25 09:10:16 -- STEP: 14628/36938 -- GLOBAL_STEP: 716450 | > loss_text_ce: 0.03689032420516014 (0.03488561494868293) | > loss_mel_ce: 3.2764830589294434 (nan) | > loss: 0.013148307800292969 (nan) | > current_lr: 5e-06 | > step_time: 0.1521 (0.13654550146398214) | > loader_time: 0.0046 (0.004635797666336355)  --> TIME: 2025-02-25 09:10:25 -- STEP: 14678/36938 -- GLOBAL_STEP: 716500 | > loss_text_ce: 0.035167939960956573 (0.03487927452012008) | > loss_mel_ce: 2.784242630004883 (nan) | > loss: 0.011188137345016003 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13652309708349136) | > loader_time: 0.0045 (0.004635807375993003)  --> TIME: 2025-02-25 09:10:34 -- STEP: 14728/36938 -- GLOBAL_STEP: 716550 | > loss_text_ce: 0.031099911779165268 (0.03488009393056846) | > loss_mel_ce: 3.0456595420837402 (nan) | > loss: 0.012209363281726837 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.13649549434393635) | > loader_time: 0.0047 (0.0046356230860880475)  --> TIME: 2025-02-25 09:10:43 -- STEP: 14778/36938 -- GLOBAL_STEP: 716600 | > loss_text_ce: 0.02681225910782814 (0.03487631149092409) | > loss_mel_ce: 2.8334178924560547 (nan) | > loss: 0.011350120417773724 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1364991200407469) | > loader_time: 0.0046 (0.004635524888502621)  --> TIME: 2025-02-25 09:10:52 -- STEP: 14828/36938 -- GLOBAL_STEP: 716650 | > loss_text_ce: 0.028427544981241226 (0.03487478921391903) | > loss_mel_ce: 3.21972918510437 (nan) | > loss: 0.01288951188325882 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.13650053717037305) | > loader_time: 0.0046 (0.00463538680206516)  --> TIME: 2025-02-25 09:11:02 -- STEP: 14878/36938 -- GLOBAL_STEP: 716700 | > loss_text_ce: 0.03528163209557533 (0.03487273850313608) | > loss_mel_ce: 3.2914669513702393 (nan) | > loss: 0.013201383873820305 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.13651127830630586) | > loader_time: 0.0048 (0.0046352854594217605)  --> TIME: 2025-02-25 09:11:12 -- STEP: 14928/36938 -- GLOBAL_STEP: 716750 | > loss_text_ce: 0.04511116445064545 (0.034873606356001646) | > loss_mel_ce: 2.4846017360687256 (nan) | > loss: 0.010038543492555618 (nan) | > current_lr: 5e-06 | > step_time: 0.165 (0.13652263065882014) | > loader_time: 0.0048 (0.00463599012915088)  --> TIME: 2025-02-25 09:11:21 -- STEP: 14978/36938 -- GLOBAL_STEP: 716800 | > loss_text_ce: 0.03689925745129585 (0.034872450790941256) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13651880232319133) | > loader_time: 0.0044 (0.004635924012531399)  --> TIME: 2025-02-25 09:11:30 -- STEP: 15028/36938 -- GLOBAL_STEP: 716850 | > loss_text_ce: 0.038341324776411057 (0.034871913040404434) | > loss_mel_ce: 3.274268865585327 (nan) | > loss: 0.01314527913928032 (nan) | > current_lr: 5e-06 | > step_time: 0.1692 (0.1365181594603178) | > loader_time: 0.0048 (0.004635836902310745)  --> TIME: 2025-02-25 09:11:39 -- STEP: 15078/36938 -- GLOBAL_STEP: 716900 | > loss_text_ce: 0.035234399139881134 (0.03486864429979235) | > loss_mel_ce: 2.600559711456299 (nan) | > loss: 0.010459501296281815 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13652253721765772) | > loader_time: 0.0045 (0.004635607663382417)  --> TIME: 2025-02-25 09:11:48 -- STEP: 15128/36938 -- GLOBAL_STEP: 716950 | > loss_text_ce: 0.03913934528827667 (0.03486351634162504) | > loss_mel_ce: 1.582269310951233 (nan) | > loss: 0.006434161681681871 (nan) | > current_lr: 5e-06 | > step_time: 0.1853 (0.13651429180554348) | > loader_time: 0.0048 (0.004635459386375696)  --> TIME: 2025-02-25 09:11:58 -- STEP: 15178/36938 -- GLOBAL_STEP: 717000 | > loss_text_ce: 0.031155668199062347 (0.034863363254762886) | > loss_mel_ce: 2.8914947509765625 (nan) | > loss: 0.01159781962633133 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.136506360075757) | > loader_time: 0.0046 (0.004635386590132011)  --> TIME: 2025-02-25 09:12:07 -- STEP: 15228/36938 -- GLOBAL_STEP: 717050 | > loss_text_ce: 0.025341305881738663 (0.03486289940764662) | > loss_mel_ce: 2.544161796569824 (nan) | > loss: 0.010196441784501076 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13650777642420733) | > loader_time: 0.0045 (0.004635332731052046)  --> TIME: 2025-02-25 09:12:16 -- STEP: 15278/36938 -- GLOBAL_STEP: 717100 | > loss_text_ce: 0.03166044503450394 (0.03486597198748653) | > loss_mel_ce: 2.966095209121704 (nan) | > loss: 0.011895856820046902 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.1365237494915759) | > loader_time: 0.0046 (0.004635333031755347)  --> TIME: 2025-02-25 09:12:26 -- STEP: 15328/36938 -- GLOBAL_STEP: 717150 | > loss_text_ce: 0.0372195765376091 (0.034864006746299865) | > loss_mel_ce: 1.800979495048523 (nan) | > loss: 0.007294441107660532 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1365214039208006) | > loader_time: 0.0047 (0.004635272839248544)  --> TIME: 2025-02-25 09:12:35 -- STEP: 15378/36938 -- GLOBAL_STEP: 717200 | > loss_text_ce: 0.037805769592523575 (0.034869473250087483) | > loss_mel_ce: 3.0036439895629883 (nan) | > loss: 0.012069245800375938 (nan) | > current_lr: 5e-06 | > step_time: 0.187 (0.13651799625407224) | > loader_time: 0.0048 (0.004635299007144382)  --> TIME: 2025-02-25 09:12:45 -- STEP: 15428/36938 -- GLOBAL_STEP: 717250 | > loss_text_ce: 0.032709989696741104 (0.03486990889339742) | > loss_mel_ce: 3.4752566814422607 (nan) | > loss: 0.013920503668487072 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13652948558036992) | > loader_time: 0.0048 (0.0046353076973633875)  --> TIME: 2025-02-25 09:12:54 -- STEP: 15478/36938 -- GLOBAL_STEP: 717300 | > loss_text_ce: 0.029199568554759026 (0.03487377120635612) | > loss_mel_ce: 2.498727321624756 (nan) | > loss: 0.010031457059085369 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13653506043807975) | > loader_time: 0.0048 (0.004635347539347437)  --> TIME: 2025-02-25 09:13:03 -- STEP: 15528/36938 -- GLOBAL_STEP: 717350 | > loss_text_ce: 0.030705947428941727 (0.03487450860228179) | > loss_mel_ce: 3.094616413116455 (nan) | > loss: 0.012402073480188847 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.13653191683035745) | > loader_time: 0.0046 (0.004635323973306355)  --> TIME: 2025-02-25 09:13:12 -- STEP: 15578/36938 -- GLOBAL_STEP: 717400 | > loss_text_ce: 0.03433388099074364 (0.03487761324363156) | > loss_mel_ce: 3.2125892639160156 (nan) | > loss: 0.01288461685180664 (nan) | > current_lr: 5e-06 | > step_time: 0.1915 (0.13651726596054622) | > loader_time: 0.0048 (0.004635186323321871)  --> TIME: 2025-02-25 09:13:22 -- STEP: 15628/36938 -- GLOBAL_STEP: 717450 | > loss_text_ce: 0.033374376595020294 (0.03487434722890782) | > loss_mel_ce: 2.8735060691833496 (nan) | > loss: 0.011535240337252617 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13652082956566555) | > loader_time: 0.0045 (0.0046352981178541365)  --> TIME: 2025-02-25 09:13:31 -- STEP: 15678/36938 -- GLOBAL_STEP: 717500 | > loss_text_ce: 0.02577121928334236 (0.03487322139688578) | > loss_mel_ce: 3.2095184326171875 (nan) | > loss: 0.012838451191782951 (nan) | > current_lr: 5e-06 | > step_time: 0.1629 (0.13651204055966748) | > loader_time: 0.0047 (0.004635098166331318)  --> TIME: 2025-02-25 09:13:40 -- STEP: 15728/36938 -- GLOBAL_STEP: 717550 | > loss_text_ce: 0.03317515552043915 (0.034873447283967025) | > loss_mel_ce: 2.555352210998535 (nan) | > loss: 0.010271934792399406 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.13651276342684548) | > loader_time: 0.0046 (0.0046350737523935485)  --> TIME: 2025-02-25 09:13:49 -- STEP: 15778/36938 -- GLOBAL_STEP: 717600 | > loss_text_ce: 0.04282423481345177 (0.034873994879045925) | > loss_mel_ce: 1.8030977249145508 (nan) | > loss: 0.007325087673962116 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13650138930595107) | > loader_time: 0.0046 (0.0046349533581283315)  --> TIME: 2025-02-25 09:13:59 -- STEP: 15828/36938 -- GLOBAL_STEP: 717650 | > loss_text_ce: 0.030469775199890137 (0.03486813713344786) | > loss_mel_ce: 2.872324228286743 (nan) | > loss: 0.011519024148583412 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13650802426305733) | > loader_time: 0.0045 (0.0046350511150104546)  --> TIME: 2025-02-25 09:14:07 -- STEP: 15878/36938 -- GLOBAL_STEP: 717700 | > loss_text_ce: 0.02930322103202343 (0.034869207641632385) | > loss_mel_ce: 0.01621246337890625 (nan) | > loss: 0.0001806178188417107 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.13648959280697145) | > loader_time: 0.0046 (0.004634804457767669)  --> TIME: 2025-02-25 09:14:17 -- STEP: 15928/36938 -- GLOBAL_STEP: 717750 | > loss_text_ce: 0.03645266592502594 (0.03486974050474721) | > loss_mel_ce: 3.1891093254089355 (nan) | > loss: 0.012799850665032864 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.13647988187135207) | > loader_time: 0.0048 (0.004634671433375497)  --> TIME: 2025-02-25 09:14:26 -- STEP: 15978/36938 -- GLOBAL_STEP: 717800 | > loss_text_ce: 0.0372009240090847 (0.034869785231277865) | > loss_mel_ce: 2.202765941619873 (nan) | > loss: 0.008888757787644863 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.1364666987217514) | > loader_time: 0.0046 (0.004634638858656207)  --> TIME: 2025-02-25 09:14:35 -- STEP: 16028/36938 -- GLOBAL_STEP: 717850 | > loss_text_ce: 0.03769690915942192 (0.03487078589764933) | > loss_mel_ce: 2.375640869140625 (nan) | > loss: 0.009576737880706787 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13645198364819572) | > loader_time: 0.0045 (0.004634591388916586)  --> TIME: 2025-02-25 09:14:44 -- STEP: 16078/36938 -- GLOBAL_STEP: 717900 | > loss_text_ce: 0.03057917207479477 (0.03487059191883725) | > loss_mel_ce: 2.937695264816284 (nan) | > loss: 0.01177886687219143 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13645340429786085) | > loader_time: 0.0046 (0.00463468279021787)  --> TIME: 2025-02-25 09:14:53 -- STEP: 16128/36938 -- GLOBAL_STEP: 717950 | > loss_text_ce: 0.039899278432130814 (0.03487145258698135) | > loss_mel_ce: 2.2753357887268066 (nan) | > loss: 0.009187441319227219 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13644897896382618) | > loader_time: 0.0047 (0.004634739890221557)  --> TIME: 2025-02-25 09:15:03 -- STEP: 16178/36938 -- GLOBAL_STEP: 718000 | > loss_text_ce: 0.046298444271087646 (0.03486844445678352) | > loss_mel_ce: 1.0291475057601929 (nan) | > loss: 0.004267642740160227 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13644999534624927) | > loader_time: 0.0045 (0.0046348674053579226)  --> TIME: 2025-02-25 09:15:12 -- STEP: 16228/36938 -- GLOBAL_STEP: 718050 | > loss_text_ce: 0.050908077508211136 (0.03487124170310065) | > loss_mel_ce: 1.9665940999984741 (nan) | > loss: 0.00800596084445715 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13645312365622966) | > loader_time: 0.0047 (0.004634946856499885)  --> TIME: 2025-02-25 09:15:21 -- STEP: 16278/36938 -- GLOBAL_STEP: 718100 | > loss_text_ce: 0.027987856417894363 (0.03487259105725799) | > loss_mel_ce: 3.057904005050659 (nan) | > loss: 0.01224560383707285 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13645204508798262) | > loader_time: 0.0045 (0.004635105277765783)  --> TIME: 2025-02-25 09:15:31 -- STEP: 16328/36938 -- GLOBAL_STEP: 718150 | > loss_text_ce: 0.04227631166577339 (0.03487355367984175) | > loss_mel_ce: 3.074044942855835 (nan) | > loss: 0.012366355396807194 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.136454454024945) | > loader_time: 0.0047 (0.004635370169008312)  --> TIME: 2025-02-25 09:15:40 -- STEP: 16378/36938 -- GLOBAL_STEP: 718200 | > loss_text_ce: 0.03885134682059288 (0.03487005067705143) | > loss_mel_ce: 2.6890175342559814 (nan) | > loss: 0.010824876837432384 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13645983652137533) | > loader_time: 0.0046 (0.004635613208318421)  --> TIME: 2025-02-25 09:15:50 -- STEP: 16428/36938 -- GLOBAL_STEP: 718250 | > loss_text_ce: 0.035060327500104904 (0.034866797336177036) | > loss_mel_ce: 2.392313241958618 (nan) | > loss: 0.009632435627281666 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13646419187053105) | > loader_time: 0.0047 (0.0046359591162462984)  --> TIME: 2025-02-25 09:15:58 -- STEP: 16478/36938 -- GLOBAL_STEP: 718300 | > loss_text_ce: 0.023900775238871574 (0.03486331237602358) | > loss_mel_ce: 3.0597023963928223 (nan) | > loss: 0.012236521579325199 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13644114563890286) | > loader_time: 0.0046 (0.0046358752241989955)  --> TIME: 2025-02-25 09:16:08 -- STEP: 16528/36938 -- GLOBAL_STEP: 718350 | > loss_text_ce: 0.03588363900780678 (0.0348595070890679) | > loss_mel_ce: 2.5844099521636963 (nan) | > loss: 0.01039799116551876 (nan) | > current_lr: 5e-06 | > step_time: 0.185 (0.13644627309318738) | > loader_time: 0.0049 (0.004635996359246275)  --> TIME: 2025-02-25 09:16:17 -- STEP: 16578/36938 -- GLOBAL_STEP: 718400 | > loss_text_ce: 0.04466312378644943 (0.03485640186262767) | > loss_mel_ce: 2.512875556945801 (nan) | > loss: 0.010148963890969753 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1364388760417035) | > loader_time: 0.0046 (0.004636018148793395)  --> TIME: 2025-02-25 09:16:27 -- STEP: 16628/36938 -- GLOBAL_STEP: 718450 | > loss_text_ce: 0.02580970712006092 (0.03485715269403023) | > loss_mel_ce: 3.267378091812134 (nan) | > loss: 0.01306820660829544 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13645152208866526) | > loader_time: 0.0045 (0.004636042933066009)  --> TIME: 2025-02-25 09:16:36 -- STEP: 16678/36938 -- GLOBAL_STEP: 718500 | > loss_text_ce: 0.027814416214823723 (0.03485861958730144) | > loss_mel_ce: 3.2450647354125977 (nan) | > loss: 0.01298761647194624 (nan) | > current_lr: 5e-06 | > step_time: 0.1516 (0.13644004282669525) | > loader_time: 0.0048 (0.004636035161076116)  --> TIME: 2025-02-25 09:16:45 -- STEP: 16728/36938 -- GLOBAL_STEP: 718550 | > loss_text_ce: 0.03703662380576134 (0.03485793084559131) | > loss_mel_ce: 0.20294189453125 (nan) | > loss: 0.0009522957843728364 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13642969914226158) | > loader_time: 0.0046 (0.004635936774345509)  --> TIME: 2025-02-25 09:16:55 -- STEP: 16778/36938 -- GLOBAL_STEP: 718600 | > loss_text_ce: 0.0434182807803154 (0.03485599663101934) | > loss_mel_ce: 2.5195512771606445 (nan) | > loss: 0.010170514695346355 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.1364482148979095) | > loader_time: 0.0046 (0.004635947241460846)  --> TIME: 2025-02-25 09:17:04 -- STEP: 16828/36938 -- GLOBAL_STEP: 718650 | > loss_text_ce: 0.03709854185581207 (0.03485252768255552) | > loss_mel_ce: 2.857112407684326 (nan) | > loss: 0.01148496475070715 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.13644536704158017) | > loader_time: 0.0046 (0.004635845081863412)  --> TIME: 2025-02-25 09:17:13 -- STEP: 16878/36938 -- GLOBAL_STEP: 718700 | > loss_text_ce: 0.04137139394879341 (0.03485306555449188) | > loss_mel_ce: 2.580258369445801 (nan) | > loss: 0.010403293184936047 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.1364473626573789) | > loader_time: 0.0044 (0.004635757385151738)  --> TIME: 2025-02-25 09:17:23 -- STEP: 16928/36938 -- GLOBAL_STEP: 718750 | > loss_text_ce: 0.030219560489058495 (0.03485276735383787) | > loss_mel_ce: 3.0245978832244873 (nan) | > loss: 0.012122292071580887 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.1364517746130557) | > loader_time: 0.0046 (0.004635680826039295)  --> TIME: 2025-02-25 09:17:32 -- STEP: 16978/36938 -- GLOBAL_STEP: 718800 | > loss_text_ce: 0.04192106053233147 (0.034853706739198735) | > loss_mel_ce: 3.0345282554626465 (nan) | > loss: 0.01220813300460577 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13644851022199805) | > loader_time: 0.0046 (0.004635662461832526)  --> TIME: 2025-02-25 09:17:41 -- STEP: 17028/36938 -- GLOBAL_STEP: 718850 | > loss_text_ce: 0.035070132464170456 (0.03485607969287385) | > loss_mel_ce: 2.7843170166015625 (nan) | > loss: 0.011188045144081116 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13645089644346337) | > loader_time: 0.0045 (0.004635823691482123)  --> TIME: 2025-02-25 09:17:50 -- STEP: 17078/36938 -- GLOBAL_STEP: 718900 | > loss_text_ce: 0.03498423099517822 (0.03485145890361722) | > loss_mel_ce: 2.6328861713409424 (nan) | > loss: 0.010586787946522236 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13644482888694123) | > loader_time: 0.0046 (0.004635783559140471)  --> TIME: 2025-02-25 09:18:00 -- STEP: 17128/36938 -- GLOBAL_STEP: 718950 | > loss_text_ce: 0.028343424201011658 (0.03485011077207151) | > loss_mel_ce: 2.8661839962005615 (nan) | > loss: 0.011486221104860306 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13644148390574926) | > loader_time: 0.0048 (0.00463578859426773)  --> TIME: 2025-02-25 09:18:09 -- STEP: 17178/36938 -- GLOBAL_STEP: 719000 | > loss_text_ce: 0.02845115400850773 (0.03484433003683815) | > loss_mel_ce: 3.0880987644195557 (nan) | > loss: 0.01236726250499487 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13644650596195013) | > loader_time: 0.0045 (0.004635697347152654)  --> TIME: 2025-02-25 09:18:19 -- STEP: 17228/36938 -- GLOBAL_STEP: 719050 | > loss_text_ce: 0.0374528206884861 (0.034844666758899495) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13645557026479) | > loader_time: 0.0046 (0.004635620510215565)  --> TIME: 2025-02-25 09:18:28 -- STEP: 17278/36938 -- GLOBAL_STEP: 719100 | > loss_text_ce: 0.034246526658535004 (0.03484428233396648) | > loss_mel_ce: 3.1745989322662354 (nan) | > loss: 0.012733514420688152 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13646032827045992) | > loader_time: 0.0046 (0.004635660829664385)  --> TIME: 2025-02-25 09:18:37 -- STEP: 17328/36938 -- GLOBAL_STEP: 719150 | > loss_text_ce: 0.032424189150333405 (0.034840733093707915) | > loss_mel_ce: 0.008056640625 (nan) | > loss: 0.00016063822840806097 (nan) | > current_lr: 5e-06 | > step_time: 0.1353 (0.13645387393450875) | > loader_time: 0.0046 (0.004635582326301676)  --> TIME: 2025-02-25 09:18:47 -- STEP: 17378/36938 -- GLOBAL_STEP: 719200 | > loss_text_ce: 0.03890540078282356 (0.03483961348603336) | > loss_mel_ce: 3.431436777114868 (nan) | > loss: 0.013771199621260166 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13646866684199302) | > loader_time: 0.0045 (0.004635607884824436)  --> TIME: 2025-02-25 09:18:56 -- STEP: 17428/36938 -- GLOBAL_STEP: 719250 | > loss_text_ce: 0.03732253611087799 (0.034837058837826305) | > loss_mel_ce: 2.4062817096710205 (nan) | > loss: 0.009696843102574348 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1364791005719884) | > loader_time: 0.0045 (0.0046354541681161284)  --> TIME: 2025-02-25 09:19:06 -- STEP: 17478/36938 -- GLOBAL_STEP: 719300 | > loss_text_ce: 0.043814919888973236 (0.034839421499772906) | > loss_mel_ce: 3.5688045024871826 (nan) | > loss: 0.014335792511701584 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13648603381001922) | > loader_time: 0.0046 (0.00463540518034211)  --> TIME: 2025-02-25 09:19:15 -- STEP: 17528/36938 -- GLOBAL_STEP: 719350 | > loss_text_ce: 0.0321645550429821 (0.034836951368156836) | > loss_mel_ce: 3.5857765674591064 (nan) | > loss: 0.01435691025108099 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13648873101027315) | > loader_time: 0.0045 (0.004635440601381941)  --> TIME: 2025-02-25 09:19:25 -- STEP: 17578/36938 -- GLOBAL_STEP: 719400 | > loss_text_ce: 0.033487461507320404 (0.034836157362667396) | > loss_mel_ce: 2.0873005390167236 (nan) | > loss: 0.008415826596319675 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13649735553555747) | > loader_time: 0.0047 (0.00463548253482891)  --> TIME: 2025-02-25 09:19:34 -- STEP: 17628/36938 -- GLOBAL_STEP: 719450 | > loss_text_ce: 0.03751160204410553 (0.03483621667666286) | > loss_mel_ce: 2.809016466140747 (nan) | > loss: 0.011295747011899948 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13650557619929768) | > loader_time: 0.0046 (0.004635318799275276)  --> TIME: 2025-02-25 09:19:43 -- STEP: 17678/36938 -- GLOBAL_STEP: 719500 | > loss_text_ce: 0.034869175404310226 (0.03483121300721822) | > loss_mel_ce: 1.3165740966796875 (nan) | > loss: 0.005362870637327433 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13648870959920156) | > loader_time: 0.0044 (0.004635151242600221)  --> TIME: 2025-02-25 09:19:52 -- STEP: 17728/36938 -- GLOBAL_STEP: 719550 | > loss_text_ce: 0.04267854243516922 (0.03483060907387799) | > loss_mel_ce: 2.911813259124756 (nan) | > loss: 0.011724174953997135 (nan) | > current_lr: 5e-06 | > step_time: 0.1578 (0.1364774629204717) | > loader_time: 0.0073 (0.004635360804705826)  --> TIME: 2025-02-25 09:20:01 -- STEP: 17778/36938 -- GLOBAL_STEP: 719600 | > loss_text_ce: 0.028182396665215492 (0.03483121118822912) | > loss_mel_ce: 2.543682336807251 (nan) | > loss: 0.010205813683569431 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13647327750052932) | > loader_time: 0.0047 (0.004635337327379288)  --> TIME: 2025-02-25 09:20:11 -- STEP: 17828/36938 -- GLOBAL_STEP: 719650 | > loss_text_ce: 0.03170202299952507 (0.03483160331527462) | > loss_mel_ce: 2.6675076484680176 (nan) | > loss: 0.010711150243878365 (nan) | > current_lr: 5e-06 | > step_time: 0.1788 (0.13647187323160534) | > loader_time: 0.0048 (0.004635390276220938)  --> TIME: 2025-02-25 09:20:20 -- STEP: 17878/36938 -- GLOBAL_STEP: 719700 | > loss_text_ce: 0.029183370992541313 (0.03483000362670339) | > loss_mel_ce: 2.5206332206726074 (nan) | > loss: 0.01011832058429718 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13647351652760578) | > loader_time: 0.0046 (0.004635532079146776)  --> TIME: 2025-02-25 09:20:29 -- STEP: 17928/36938 -- GLOBAL_STEP: 719750 | > loss_text_ce: 0.0318741612136364 (0.034829124149351944) | > loss_mel_ce: 2.5438787937164307 (nan) | > loss: 0.010221242904663086 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.1364639799997034) | > loader_time: 0.0047 (0.004635447612142828)  --> TIME: 2025-02-25 09:20:38 -- STEP: 17978/36938 -- GLOBAL_STEP: 719800 | > loss_text_ce: 0.042127035558223724 (0.03483245081283485) | > loss_mel_ce: 2.9941177368164062 (nan) | > loss: 0.012048591859638691 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13645855277620922) | > loader_time: 0.0045 (0.004635385284568103)  --> TIME: 2025-02-25 09:20:47 -- STEP: 18028/36938 -- GLOBAL_STEP: 719850 | > loss_text_ce: 0.017150865867733955 (0.03483163008768266) | > loss_mel_ce: 2.634427547454834 (nan) | > loss: 0.010522137396037579 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13644562025199292) | > loader_time: 0.0047 (0.0046352342594376)  --> TIME: 2025-02-25 09:20:56 -- STEP: 18078/36938 -- GLOBAL_STEP: 719900 | > loss_text_ce: 0.03419727087020874 (0.03482650133428453) | > loss_mel_ce: 2.9488365650177 (nan) | > loss: 0.011837436817586422 (nan) | > current_lr: 5e-06 | > step_time: 0.184 (0.1364346580545467) | > loader_time: 0.0048 (0.004635181425946878)  --> TIME: 2025-02-25 09:21:06 -- STEP: 18128/36938 -- GLOBAL_STEP: 719950 | > loss_text_ce: 0.038725364953279495 (0.03482492475614032) | > loss_mel_ce: 3.0934829711914062 (nan) | > loss: 0.012429398484528065 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13642615552034854) | > loader_time: 0.0046 (0.0046352265239919185)  --> TIME: 2025-02-25 09:21:14 -- STEP: 18178/36938 -- GLOBAL_STEP: 720000 | > loss_text_ce: 0.02742253802716732 (0.03482269008131284) | > loss_mel_ce: 2.653597354888916 (nan) | > loss: 0.010638968087732792 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13640828184567905) | > loader_time: 0.0044 (0.00463507374543038) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_720000.pth  --> TIME: 2025-02-25 09:21:30 -- STEP: 18228/36938 -- GLOBAL_STEP: 720050 | > loss_text_ce: 0.04503102973103523 (0.03482469784508514) | > loss_mel_ce: 2.4672136306762695 (nan) | > loss: 0.009969225153326988 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13640818817981257) | > loader_time: 0.0045 (0.004635071110929439)  --> TIME: 2025-02-25 09:21:39 -- STEP: 18278/36938 -- GLOBAL_STEP: 720100 | > loss_text_ce: 0.037294790148735046 (0.034822650753470824) | > loss_mel_ce: 1.4336134195327759 (nan) | > loss: 0.005836937576532364 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13641222772380204) | > loader_time: 0.0045 (0.004635084261060963)  --> TIME: 2025-02-25 09:21:48 -- STEP: 18328/36938 -- GLOBAL_STEP: 720150 | > loss_text_ce: 0.03980034589767456 (0.03482173063487615) | > loss_mel_ce: 2.2234222888946533 (nan) | > loss: 0.008981042541563511 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.1364081639980243) | > loader_time: 0.0047 (0.004635061267056033)  --> TIME: 2025-02-25 09:21:58 -- STEP: 18378/36938 -- GLOBAL_STEP: 720200 | > loss_text_ce: 0.03636239096522331 (0.034822272828990315) | > loss_mel_ce: 3.2371554374694824 (nan) | > loss: 0.012990150600671768 (nan) | > current_lr: 5e-06 | > step_time: 0.1784 (0.13640970109090023) | > loader_time: 0.0048 (0.004635093131435352)  --> TIME: 2025-02-25 09:22:07 -- STEP: 18428/36938 -- GLOBAL_STEP: 720250 | > loss_text_ce: 0.034875575453042984 (0.03482265129592621) | > loss_mel_ce: 3.452780246734619 (nan) | > loss: 0.0138399051502347 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.13639848671379726) | > loader_time: 0.0048 (0.004634993775485516)  --> TIME: 2025-02-25 09:22:16 -- STEP: 18478/36938 -- GLOBAL_STEP: 720300 | > loss_text_ce: 0.03866835683584213 (0.034823336296232354) | > loss_mel_ce: 3.2080438137054443 (nan) | > loss: 0.012883779592812061 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13640637509504697) | > loader_time: 0.005 (0.004634941730009986)  --> TIME: 2025-02-25 09:22:26 -- STEP: 18528/36938 -- GLOBAL_STEP: 720350 | > loss_text_ce: 0.040423206984996796 (0.03481848859149786) | > loss_mel_ce: 2.8903965950012207 (nan) | > loss: 0.01163023803383112 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13641286838239988) | > loader_time: 0.0045 (0.00463489365855646)  --> TIME: 2025-02-25 09:22:35 -- STEP: 18578/36938 -- GLOBAL_STEP: 720400 | > loss_text_ce: 0.03723474591970444 (0.03482073735901329) | > loss_mel_ce: 3.5253729820251465 (nan) | > loss: 0.014137333258986473 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13640783016360095) | > loader_time: 0.0046 (0.00463477874010085)  --> TIME: 2025-02-25 09:22:44 -- STEP: 18628/36938 -- GLOBAL_STEP: 720450 | > loss_text_ce: 0.041715800762176514 (0.0348217477515022) | > loss_mel_ce: 2.343447685241699 (nan) | > loss: 0.009464935399591923 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13639827055661838) | > loader_time: 0.0046 (0.004634756642092386)  --> TIME: 2025-02-25 09:22:53 -- STEP: 18678/36938 -- GLOBAL_STEP: 720500 | > loss_text_ce: 0.027356918901205063 (0.03481881118834578) | > loss_mel_ce: 3.2874581813812256 (nan) | > loss: 0.013154028914868832 (nan) | > current_lr: 5e-06 | > step_time: 0.1998 (0.136408948262635) | > loader_time: 0.0049 (0.004634752111703497)  --> TIME: 2025-02-25 09:23:03 -- STEP: 18728/36938 -- GLOBAL_STEP: 720550 | > loss_text_ce: 0.04726798087358475 (0.03481771382809546) | > loss_mel_ce: 0.046661376953125 (nan) | > loss: 0.00037273557973094285 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1364062782003048) | > loader_time: 0.0045 (0.004634719865539032)  --> TIME: 2025-02-25 09:23:12 -- STEP: 18778/36938 -- GLOBAL_STEP: 720600 | > loss_text_ce: 0.03413061797618866 (0.034814933396952075) | > loss_mel_ce: 2.515625 (nan) | > loss: 0.01011807844042778 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13640831131081435) | > loader_time: 0.0046 (0.004634704106353515)  --> TIME: 2025-02-25 09:23:21 -- STEP: 18828/36938 -- GLOBAL_STEP: 720650 | > loss_text_ce: 0.03586743399500847 (0.034817313122881055) | > loss_mel_ce: 3.116243839263916 (nan) | > loss: 0.01250837929546833 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1364091354518631) | > loader_time: 0.0045 (0.004634674881480665)  --> TIME: 2025-02-25 09:23:31 -- STEP: 18878/36938 -- GLOBAL_STEP: 720700 | > loss_text_ce: 0.03721686825156212 (0.0348154134149134) | > loss_mel_ce: 2.0414936542510986 (nan) | > loss: 0.008248851634562016 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.1364157730675009) | > loader_time: 0.0045 (0.004634735126820689)  --> TIME: 2025-02-25 09:23:40 -- STEP: 18928/36938 -- GLOBAL_STEP: 720750 | > loss_text_ce: 0.038794949650764465 (0.03481309806811136) | > loss_mel_ce: 2.516284465789795 (nan) | > loss: 0.010139204561710358 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13641719152286536) | > loader_time: 0.0047 (0.004634602976268557)  --> TIME: 2025-02-25 09:23:49 -- STEP: 18978/36938 -- GLOBAL_STEP: 720800 | > loss_text_ce: 0.026888985186815262 (0.03481000003309032) | > loss_mel_ce: 2.941453695297241 (nan) | > loss: 0.011779138818383217 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.13642653181249134) | > loader_time: 0.0047 (0.00463452367061984)  --> TIME: 2025-02-25 09:23:59 -- STEP: 19028/36938 -- GLOBAL_STEP: 720850 | > loss_text_ce: 0.030167000368237495 (0.034812427533679705) | > loss_mel_ce: 1.6422022581100464 (nan) | > loss: 0.006636386271566153 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13642119885192647) | > loader_time: 0.0044 (0.004634380628763425)  --> TIME: 2025-02-25 09:24:08 -- STEP: 19078/36938 -- GLOBAL_STEP: 720900 | > loss_text_ce: 0.03511432185769081 (0.03481264074026158) | > loss_mel_ce: 3.1678476333618164 (nan) | > loss: 0.012710167095065117 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13643530802552295) | > loader_time: 0.0045 (0.0046343903632043385)  --> TIME: 2025-02-25 09:24:18 -- STEP: 19128/36938 -- GLOBAL_STEP: 720950 | > loss_text_ce: 0.040788404643535614 (0.03481305513613947) | > loss_mel_ce: 2.9011669158935547 (nan) | > loss: 0.0116744264960289 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13643827878596265) | > loader_time: 0.0046 (0.004634321658295186)  --> TIME: 2025-02-25 09:24:27 -- STEP: 19178/36938 -- GLOBAL_STEP: 721000 | > loss_text_ce: 0.034182820469141006 (0.03481202542349445) | > loss_mel_ce: 3.108444929122925 (nan) | > loss: 0.012470745481550694 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.13643278579839244) | > loader_time: 0.0046 (0.00463419897189066)  --> TIME: 2025-02-25 09:24:36 -- STEP: 19228/36938 -- GLOBAL_STEP: 721050 | > loss_text_ce: 0.037487663328647614 (0.03481289705560738) | > loss_mel_ce: 1.9443689584732056 (nan) | > loss: 0.007864510640501976 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13641502579706327) | > loader_time: 0.0046 (0.0046341836340888015)  --> TIME: 2025-02-25 09:24:45 -- STEP: 19278/36938 -- GLOBAL_STEP: 721100 | > loss_text_ce: 0.0276152603328228 (0.034809786717046445) | > loss_mel_ce: 3.132328987121582 (nan) | > loss: 0.012539462186396122 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13640355696531278) | > loader_time: 0.0045 (0.004634058590511111)  --> TIME: 2025-02-25 09:24:54 -- STEP: 19328/36938 -- GLOBAL_STEP: 721150 | > loss_text_ce: 0.029145387932658195 (0.034805344451276414) | > loss_mel_ce: 1.32373046875 (nan) | > loss: 0.005368554964661598 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13640610955112856) | > loader_time: 0.0046 (0.004634030644368635)  --> TIME: 2025-02-25 09:25:03 -- STEP: 19378/36938 -- GLOBAL_STEP: 721200 | > loss_text_ce: 0.03310840204358101 (0.034806088753348854) | > loss_mel_ce: 2.211026668548584 (nan) | > loss: 0.008905299007892609 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13639893482831156) | > loader_time: 0.0047 (0.004634008821976992)  --> TIME: 2025-02-25 09:25:13 -- STEP: 19428/36938 -- GLOBAL_STEP: 721250 | > loss_text_ce: 0.03733198717236519 (0.03480831168921604) | > loss_mel_ce: 3.215146064758301 (nan) | > loss: 0.012906660325825214 (nan) | > current_lr: 5e-06 | > step_time: 0.1522 (0.13640455846706045) | > loader_time: 0.0046 (0.004633947694549478)  --> TIME: 2025-02-25 09:25:22 -- STEP: 19478/36938 -- GLOBAL_STEP: 721300 | > loss_text_ce: 0.04161625728011131 (0.034808624591024406) | > loss_mel_ce: 3.0964109897613525 (nan) | > loss: 0.012452489696443081 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.136402142000145) | > loader_time: 0.0046 (0.004633881739980548)  --> TIME: 2025-02-25 09:25:31 -- STEP: 19528/36938 -- GLOBAL_STEP: 721350 | > loss_text_ce: 0.035537779331207275 (0.03481086960480364) | > loss_mel_ce: 2.864478588104248 (nan) | > loss: 0.011508001945912838 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1363930211405151) | > loader_time: 0.0045 (0.004633726533052107)  --> TIME: 2025-02-25 09:25:40 -- STEP: 19578/36938 -- GLOBAL_STEP: 721400 | > loss_text_ce: 0.03460617735981941 (0.03481093248876071) | > loss_mel_ce: 2.893261671066284 (nan) | > loss: 0.011618523858487606 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13638260126089585) | > loader_time: 0.0045 (0.004633575029399451)  --> TIME: 2025-02-25 09:25:50 -- STEP: 19628/36938 -- GLOBAL_STEP: 721450 | > loss_text_ce: 0.03845934197306633 (0.03481358534273855) | > loss_mel_ce: 2.948589324951172 (nan) | > loss: 0.011853368021547794 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.13637313756287847) | > loader_time: 0.0046 (0.004633494457888539)  --> TIME: 2025-02-25 09:25:59 -- STEP: 19678/36938 -- GLOBAL_STEP: 721500 | > loss_text_ce: 0.04491032660007477 (0.034815408752822234) | > loss_mel_ce: 2.856797695159912 (nan) | > loss: 0.011514715850353241 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13636900661606965) | > loader_time: 0.0047 (0.004633567175471143)  --> TIME: 2025-02-25 09:26:08 -- STEP: 19728/36938 -- GLOBAL_STEP: 721550 | > loss_text_ce: 0.03501078858971596 (0.03481342553921908) | > loss_mel_ce: 2.4395201206207275 (nan) | > loss: 0.009819568134844303 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13637403867057207) | > loader_time: 0.0045 (0.00463354452199694)  --> TIME: 2025-02-25 09:26:17 -- STEP: 19778/36938 -- GLOBAL_STEP: 721600 | > loss_text_ce: 0.0315520353615284 (0.03481493597878097) | > loss_mel_ce: 2.2899961471557617 (nan) | > loss: 0.009212493896484375 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.1363758556955401) | > loader_time: 0.0046 (0.004633490532253885)  --> TIME: 2025-02-25 09:26:27 -- STEP: 19828/36938 -- GLOBAL_STEP: 721650 | > loss_text_ce: 0.03410407528281212 (0.03481266594233392) | > loss_mel_ce: 3.245271682739258 (nan) | > loss: 0.01301339641213417 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.13637266633778547) | > loader_time: 0.0051 (0.004633674644187767)  --> TIME: 2025-02-25 09:26:36 -- STEP: 19878/36938 -- GLOBAL_STEP: 721700 | > loss_text_ce: 0.031900275498628616 (0.03481163292299981) | > loss_mel_ce: 2.897810220718384 (nan) | > loss: 0.011625835672020912 (nan) | > current_lr: 5e-06 | > step_time: 0.1715 (0.1363687463444314) | > loader_time: 0.0047 (0.004633638098129773)  --> TIME: 2025-02-25 09:26:45 -- STEP: 19928/36938 -- GLOBAL_STEP: 721750 | > loss_text_ce: 0.030210474506020546 (0.03481013644794197) | > loss_mel_ce: 2.185673236846924 (nan) | > loss: 0.008793190121650696 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13637539649861605) | > loader_time: 0.0045 (0.004633629719256759)  --> TIME: 2025-02-25 09:26:54 -- STEP: 19978/36938 -- GLOBAL_STEP: 721800 | > loss_text_ce: 0.03622540459036827 (0.034811556142212326) | > loss_mel_ce: 3.2352383136749268 (nan) | > loss: 0.012981999665498734 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13636208797840235) | > loader_time: 0.0046 (0.004633477755955265)  --> TIME: 2025-02-25 09:27:04 -- STEP: 20028/36938 -- GLOBAL_STEP: 721850 | > loss_text_ce: 0.03250880539417267 (0.03481163777767575) | > loss_mel_ce: 2.7404961585998535 (nan) | > loss: 0.011003988794982433 (nan) | > current_lr: 5e-06 | > step_time: 0.1835 (0.13637409115923388) | > loader_time: 0.0048 (0.0046334650098908275)  --> TIME: 2025-02-25 09:27:13 -- STEP: 20078/36938 -- GLOBAL_STEP: 721900 | > loss_text_ce: 0.04171375557780266 (0.03481429181860366) | > loss_mel_ce: 2.4987034797668457 (nan) | > loss: 0.01008102111518383 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13636869172530364) | > loader_time: 0.0046 (0.004633432520446398)  --> TIME: 2025-02-25 09:27:22 -- STEP: 20128/36938 -- GLOBAL_STEP: 721950 | > loss_text_ce: 0.047490689903497696 (0.034816720108314894) | > loss_mel_ce: 3.049992322921753 (nan) | > loss: 0.012291599996387959 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13635810411317603) | > loader_time: 0.0046 (0.0046333278542671575)  --> TIME: 2025-02-25 09:27:32 -- STEP: 20178/36938 -- GLOBAL_STEP: 722000 | > loss_text_ce: 0.04526926949620247 (0.034813706771350726) | > loss_mel_ce: 3.5323779582977295 (nan) | > loss: 0.014197013340890408 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13636211742080168) | > loader_time: 0.0046 (0.0046333831133366365)  --> TIME: 2025-02-25 09:27:41 -- STEP: 20228/36938 -- GLOBAL_STEP: 722050 | > loss_text_ce: 0.027927182614803314 (0.034811324670747865) | > loss_mel_ce: 2.614942789077759 (nan) | > loss: 0.010487579740583897 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13635555794287668) | > loader_time: 0.0046 (0.004633363702444761)  --> TIME: 2025-02-25 09:27:50 -- STEP: 20278/36938 -- GLOBAL_STEP: 722100 | > loss_text_ce: 0.04743855074048042 (0.03480821530633109) | > loss_mel_ce: 2.84173583984375 (nan) | > loss: 0.011464978568255901 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.1363699392097127) | > loader_time: 0.0048 (0.004633481362148392)  --> TIME: 2025-02-25 09:28:00 -- STEP: 20328/36938 -- GLOBAL_STEP: 722150 | > loss_text_ce: 0.03302547335624695 (0.03480719358454867) | > loss_mel_ce: 0.1220703125 (nan) | > loss: 0.0006154595175758004 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13636400502809967) | > loader_time: 0.0046 (0.004633405355429451)  --> TIME: 2025-02-25 09:28:09 -- STEP: 20378/36938 -- GLOBAL_STEP: 722200 | > loss_text_ce: 0.022288551554083824 (0.03480546490596134) | > loss_mel_ce: 2.8838400840759277 (nan) | > loss: 0.011532257311046124 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.1363666238348628) | > loader_time: 0.0045 (0.004633290597554524)  --> TIME: 2025-02-25 09:28:19 -- STEP: 20428/36938 -- GLOBAL_STEP: 722250 | > loss_text_ce: 0.03784370794892311 (0.03480837587982952) | > loss_mel_ce: 3.1929965019226074 (nan) | > loss: 0.012820795178413391 (nan) | > current_lr: 5e-06 | > step_time: 0.1271 (0.1363763259025343) | > loader_time: 0.0045 (0.0046331891113469284)  --> TIME: 2025-02-25 09:28:28 -- STEP: 20478/36938 -- GLOBAL_STEP: 722300 | > loss_text_ce: 0.03647167235612869 (0.034812292610974314) | > loss_mel_ce: 2.846001386642456 (nan) | > loss: 0.011438385583460331 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13638486183647933) | > loader_time: 0.0046 (0.004633106725711172)  --> TIME: 2025-02-25 09:28:37 -- STEP: 20528/36938 -- GLOBAL_STEP: 722350 | > loss_text_ce: 0.04082159325480461 (0.034813526660828015) | > loss_mel_ce: 3.251824140548706 (nan) | > loss: 0.013066055253148079 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13638713838574737) | > loader_time: 0.0047 (0.0046330229760331125)  --> TIME: 2025-02-25 09:28:46 -- STEP: 20578/36938 -- GLOBAL_STEP: 722400 | > loss_text_ce: 0.0392487570643425 (0.03481513347618126) | > loss_mel_ce: 3.347766876220703 (nan) | > loss: 0.013440539129078388 (nan) | > current_lr: 5e-06 | > step_time: 0.1624 (0.1363767091311946) | > loader_time: 0.0048 (0.004632884170724259)  --> TIME: 2025-02-25 09:28:56 -- STEP: 20628/36938 -- GLOBAL_STEP: 722450 | > loss_text_ce: 0.03068063035607338 (0.034815187484038064) | > loss_mel_ce: 2.839726448059082 (nan) | > loss: 0.011390505358576775 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.1363839756640355) | > loader_time: 0.0045 (0.004632893368234814)  --> TIME: 2025-02-25 09:29:06 -- STEP: 20678/36938 -- GLOBAL_STEP: 722500 | > loss_text_ce: 0.03788762167096138 (0.03481704808307004) | > loss_mel_ce: 2.633934497833252 (nan) | > loss: 0.010602469556033611 (nan) | > current_lr: 5e-06 | > step_time: 0.1848 (0.13639854076164912) | > loader_time: 0.0048 (0.004632895015196668)  --> TIME: 2025-02-25 09:29:15 -- STEP: 20728/36938 -- GLOBAL_STEP: 722550 | > loss_text_ce: 0.028729716315865517 (0.034815619545140626) | > loss_mel_ce: 2.330676317214966 (nan) | > loss: 0.009362722747027874 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.13640418554401418) | > loader_time: 0.0048 (0.004632802163252524)  --> TIME: 2025-02-25 09:29:24 -- STEP: 20778/36938 -- GLOBAL_STEP: 722600 | > loss_text_ce: 0.03611224889755249 (0.03481445270079074) | > loss_mel_ce: 2.8369524478912354 (nan) | > loss: 0.01140105165541172 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13640201359123857) | > loader_time: 0.0046 (0.004632770630769181)  --> TIME: 2025-02-25 09:29:33 -- STEP: 20828/36938 -- GLOBAL_STEP: 722650 | > loss_text_ce: 0.03277136757969856 (0.034815840926999866) | > loss_mel_ce: 2.6827261447906494 (nan) | > loss: 0.010775784030556679 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.13639407303733067) | > loader_time: 0.0048 (0.00463279006301222)  --> TIME: 2025-02-25 09:29:43 -- STEP: 20878/36938 -- GLOBAL_STEP: 722700 | > loss_text_ce: 0.03282858431339264 (0.03481715083594582) | > loss_mel_ce: 3.0491786003112793 (nan) | > loss: 0.01223018765449524 (nan) | > current_lr: 5e-06 | > step_time: 0.1823 (0.13639728000239684) | > loader_time: 0.0048 (0.004633115207866877)  --> TIME: 2025-02-25 09:29:52 -- STEP: 20928/36938 -- GLOBAL_STEP: 722750 | > loss_text_ce: 0.03750448673963547 (0.034814389812500735) | > loss_mel_ce: 2.517895460128784 (nan) | > loss: 0.010140476748347282 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13638428010798428) | > loader_time: 0.0046 (0.004633164995945181)  --> TIME: 2025-02-25 09:30:01 -- STEP: 20978/36938 -- GLOBAL_STEP: 722800 | > loss_text_ce: 0.03509439527988434 (0.034811137026613446) | > loss_mel_ce: 4.065104007720947 (nan) | > loss: 0.016270628198981285 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13639204295714238) | > loader_time: 0.0045 (0.004633211932699187)  --> TIME: 2025-02-25 09:30:11 -- STEP: 21028/36938 -- GLOBAL_STEP: 722850 | > loss_text_ce: 0.038844890892505646 (0.034809129561941064) | > loss_mel_ce: 2.779174566268921 (nan) | > loss: 0.011182617396116257 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.13638483813581392) | > loader_time: 0.0046 (0.004633209234592859)  --> TIME: 2025-02-25 09:30:20 -- STEP: 21078/36938 -- GLOBAL_STEP: 722900 | > loss_text_ce: 0.03488181531429291 (0.03481134255902299) | > loss_mel_ce: 2.973569869995117 (nan) | > loss: 0.011938300915062428 (nan) | > current_lr: 5e-06 | > step_time: 0.1857 (0.13639324406358655) | > loader_time: 0.005 (0.004633310307405818)  --> TIME: 2025-02-25 09:30:29 -- STEP: 21128/36938 -- GLOBAL_STEP: 722950 | > loss_text_ce: 0.028195343911647797 (0.03480755324079295) | > loss_mel_ce: 2.7858316898345947 (nan) | > loss: 0.011166774667799473 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13639264767344222) | > loader_time: 0.0046 (0.004633347889276565)  --> TIME: 2025-02-25 09:30:39 -- STEP: 21178/36938 -- GLOBAL_STEP: 723000 | > loss_text_ce: 0.03808850795030594 (0.03480799790729159) | > loss_mel_ce: 3.3620622158050537 (nan) | > loss: 0.013492662459611893 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13638999667245608) | > loader_time: 0.0049 (0.00463331396400291)  --> TIME: 2025-02-25 09:30:48 -- STEP: 21228/36938 -- GLOBAL_STEP: 723050 | > loss_text_ce: 0.03878709673881531 (0.03481000667917545) | > loss_mel_ce: 3.101179599761963 (nan) | > loss: 0.012460186146199703 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1363892263110043) | > loader_time: 0.0046 (0.004633330593500867)  --> TIME: 2025-02-25 09:30:57 -- STEP: 21278/36938 -- GLOBAL_STEP: 723100 | > loss_text_ce: 0.03928155452013016 (0.03481265350995777) | > loss_mel_ce: 2.4412600994110107 (nan) | > loss: 0.00984342023730278 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13638512684176463) | > loader_time: 0.0046 (0.00463328838617306)  --> TIME: 2025-02-25 09:31:07 -- STEP: 21328/36938 -- GLOBAL_STEP: 723150 | > loss_text_ce: 0.03623849153518677 (0.034814606762125604) | > loss_mel_ce: 3.205181837081909 (nan) | > loss: 0.012862779200077057 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13638725607402385) | > loader_time: 0.0046 (0.004633387563287617)  --> TIME: 2025-02-25 09:31:16 -- STEP: 21378/36938 -- GLOBAL_STEP: 723200 | > loss_text_ce: 0.03152678906917572 (0.034814646054966966) | > loss_mel_ce: 2.5982604026794434 (nan) | > loss: 0.010435664094984531 (nan) | > current_lr: 5e-06 | > step_time: 0.1583 (0.1363933109369462) | > loader_time: 0.0046 (0.004633392305344689)  --> TIME: 2025-02-25 09:31:25 -- STEP: 21428/36938 -- GLOBAL_STEP: 723250 | > loss_text_ce: 0.033661846071481705 (0.03481516152758889) | > loss_mel_ce: 2.683467388153076 (nan) | > loss: 0.01078225951641798 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13639015915101488) | > loader_time: 0.0046 (0.00463352973100206)  --> TIME: 2025-02-25 09:31:34 -- STEP: 21478/36938 -- GLOBAL_STEP: 723300 | > loss_text_ce: 0.03641783446073532 (0.034816605283370966) | > loss_mel_ce: 1.1685712337493896 (nan) | > loss: 0.004781702999025583 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13638105176216545) | > loader_time: 0.0045 (0.004633617063664944)  --> TIME: 2025-02-25 09:31:44 -- STEP: 21528/36938 -- GLOBAL_STEP: 723350 | > loss_text_ce: 0.03351699188351631 (0.0348172334577168) | > loss_mel_ce: 3.069822311401367 (nan) | > loss: 0.012314838357269764 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.1363807288920831) | > loader_time: 0.0046 (0.004633587849091424)  --> TIME: 2025-02-25 09:31:53 -- STEP: 21578/36938 -- GLOBAL_STEP: 723400 | > loss_text_ce: 0.036152444779872894 (0.03481731373351127) | > loss_mel_ce: 2.668896436691284 (nan) | > loss: 0.010734321549534798 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13637746798007774) | > loader_time: 0.0046 (0.004633521766301938)  --> TIME: 2025-02-25 09:32:02 -- STEP: 21628/36938 -- GLOBAL_STEP: 723450 | > loss_text_ce: 0.03447779268026352 (0.0348204514337714) | > loss_mel_ce: 2.332570791244507 (nan) | > loss: 0.009393050335347652 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.13638464642117926) | > loader_time: 0.005 (0.004633613582545455)  --> TIME: 2025-02-25 09:32:12 -- STEP: 21678/36938 -- GLOBAL_STEP: 723500 | > loss_text_ce: 0.034345950931310654 (0.034817115835089896) | > loss_mel_ce: 2.945772647857666 (nan) | > loss: 0.011825867928564548 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13638219514206903) | > loader_time: 0.0045 (0.004633514959661034)  --> TIME: 2025-02-25 09:32:21 -- STEP: 21728/36938 -- GLOBAL_STEP: 723550 | > loss_text_ce: 0.030890485271811485 (0.034819848317986096) | > loss_mel_ce: 2.6063499450683594 (nan) | > loss: 0.010465240105986595 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13637260694347636) | > loader_time: 0.0046 (0.004633433447495109)  --> TIME: 2025-02-25 09:32:30 -- STEP: 21778/36938 -- GLOBAL_STEP: 723600 | > loss_text_ce: 0.0372154600918293 (0.03482214280181197) | > loss_mel_ce: 3.561718702316284 (nan) | > loss: 0.014281485229730606 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13636257676056257) | > loader_time: 0.0045 (0.0046333300639297805)  --> TIME: 2025-02-25 09:32:39 -- STEP: 21828/36938 -- GLOBAL_STEP: 723650 | > loss_text_ce: 0.03805235028266907 (0.034824365743342064) | > loss_mel_ce: 3.21393084526062 (nan) | > loss: 0.01290469616651535 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.13636272910839786) | > loader_time: 0.0046 (0.004633256503027028)  --> TIME: 2025-02-25 09:32:48 -- STEP: 21878/36938 -- GLOBAL_STEP: 723700 | > loss_text_ce: 0.03981292247772217 (0.03482416435564164) | > loss_mel_ce: 2.940279006958008 (nan) | > loss: 0.011825762689113617 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.1363594755632532) | > loader_time: 0.0046 (0.004633153484207416)  --> TIME: 2025-02-25 09:32:57 -- STEP: 21928/36938 -- GLOBAL_STEP: 723750 | > loss_text_ce: 0.026094941422343254 (0.03481811565558202) | > loss_mel_ce: 1.5329861640930176 (nan) | > loss: 0.0061868298798799515 (nan) | > current_lr: 5e-06 | > step_time: 0.1118 (0.13635293482125055) | > loader_time: 0.0044 (0.004633229085886735)  --> TIME: 2025-02-25 09:33:07 -- STEP: 21978/36938 -- GLOBAL_STEP: 723800 | > loss_text_ce: 0.029371658340096474 (0.03481660444776436) | > loss_mel_ce: 2.6867966651916504 (nan) | > loss: 0.010778446681797504 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.13634924412815191) | > loader_time: 0.0045 (0.0046332810311052975)  --> TIME: 2025-02-25 09:33:16 -- STEP: 22028/36938 -- GLOBAL_STEP: 723850 | > loss_text_ce: 0.042897071689367294 (0.03481689120488282) | > loss_mel_ce: 1.7834404706954956 (nan) | > loss: 0.007247371599078178 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1363587654154302) | > loader_time: 0.0045 (0.004633255082378571)  --> TIME: 2025-02-25 09:33:25 -- STEP: 22078/36938 -- GLOBAL_STEP: 723900 | > loss_text_ce: 0.03717280924320221 (0.03481421065723904) | > loss_mel_ce: 3.05369234085083 (nan) | > loss: 0.012265338562428951 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13634737823652984) | > loader_time: 0.0048 (0.004633250330677804)  --> TIME: 2025-02-25 09:33:34 -- STEP: 22128/36938 -- GLOBAL_STEP: 723950 | > loss_text_ce: 0.04366659000515938 (0.03481295891228196) | > loss_mel_ce: 2.188674211502075 (nan) | > loss: 0.008858495391905308 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.1363472413387501) | > loader_time: 0.0051 (0.004633253541271993)  --> TIME: 2025-02-25 09:33:44 -- STEP: 22178/36938 -- GLOBAL_STEP: 724000 | > loss_text_ce: 0.03347136080265045 (0.03481422571355532) | > loss_mel_ce: 1.6024227142333984 (nan) | > loss: 0.006491643376648426 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13635288019951822) | > loader_time: 0.0045 (0.004633425150477455)  --> TIME: 2025-02-25 09:33:53 -- STEP: 22228/36938 -- GLOBAL_STEP: 724050 | > loss_text_ce: 0.02635032869875431 (0.03481179293068615) | > loss_mel_ce: 1.9817681312561035 (nan) | > loss: 0.007968723773956299 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13634939584797343) | > loader_time: 0.0045 (0.004633426762794171)  --> TIME: 2025-02-25 09:34:03 -- STEP: 22278/36938 -- GLOBAL_STEP: 724100 | > loss_text_ce: 0.03271931782364845 (0.03480899217983771) | > loss_mel_ce: 2.9143576622009277 (nan) | > loss: 0.011694750748574734 (nan) | > current_lr: 5e-06 | > step_time: 0.1599 (0.13635984803992732) | > loader_time: 0.0046 (0.004633603259534446)  --> TIME: 2025-02-25 09:34:13 -- STEP: 22328/36938 -- GLOBAL_STEP: 724150 | > loss_text_ce: 0.03253922611474991 (0.034811053212139294) | > loss_mel_ce: 2.5594992637634277 (nan) | > loss: 0.010285867378115654 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.1363757193430235) | > loader_time: 0.0047 (0.004633781069369762)  --> TIME: 2025-02-25 09:34:22 -- STEP: 22378/36938 -- GLOBAL_STEP: 724200 | > loss_text_ce: 0.036615218967199326 (0.03480880393408873) | > loss_mel_ce: 3.078819751739502 (nan) | > loss: 0.012362837791442871 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13638431790026617) | > loader_time: 0.0046 (0.004633878551397558)  --> TIME: 2025-02-25 09:34:32 -- STEP: 22428/36938 -- GLOBAL_STEP: 724250 | > loss_text_ce: 0.03364768996834755 (0.03480897322113576) | > loss_mel_ce: 2.691293239593506 (nan) | > loss: 0.010813258588314056 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13638831505232116) | > loader_time: 0.0045 (0.004633889386248645)  --> TIME: 2025-02-25 09:34:41 -- STEP: 22478/36938 -- GLOBAL_STEP: 724300 | > loss_text_ce: 0.03332408890128136 (0.03480926560414032) | > loss_mel_ce: 2.7950730323791504 (nan) | > loss: 0.011223798617720604 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13639411797715784) | > loader_time: 0.0046 (0.004634034040718995)  --> TIME: 2025-02-25 09:34:50 -- STEP: 22528/36938 -- GLOBAL_STEP: 724350 | > loss_text_ce: 0.027121545746922493 (0.03481020102510087) | > loss_mel_ce: 2.3105199337005615 (nan) | > loss: 0.009276355616748333 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13638444890437562) | > loader_time: 0.0048 (0.004633897672068639)  --> TIME: 2025-02-25 09:34:59 -- STEP: 22578/36938 -- GLOBAL_STEP: 724400 | > loss_text_ce: 0.031071139499545097 (0.03481138195247386) | > loss_mel_ce: 0.9466598629951477 (nan) | > loss: 0.0038798851892352104 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13637951272764695) | > loader_time: 0.0046 (0.0046339585831995265)  --> TIME: 2025-02-25 09:35:09 -- STEP: 22628/36938 -- GLOBAL_STEP: 724450 | > loss_text_ce: 0.020029297098517418 (0.03481052510192767) | > loss_mel_ce: 2.130101203918457 (nan) | > loss: 0.008532264269888401 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13637804998743858) | > loader_time: 0.0045 (0.004633981715416912)  --> TIME: 2025-02-25 09:35:18 -- STEP: 22678/36938 -- GLOBAL_STEP: 724500 | > loss_text_ce: 0.03572608903050423 (0.03481158515761576) | > loss_mel_ce: 3.1166818141937256 (nan) | > loss: 0.01250955555588007 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13637355255149244) | > loader_time: 0.0045 (0.004634012987988131)  --> TIME: 2025-02-25 09:35:27 -- STEP: 22728/36938 -- GLOBAL_STEP: 724550 | > loss_text_ce: 0.03678053990006447 (0.034810364143762505) | > loss_mel_ce: 3.1224329471588135 (nan) | > loss: 0.01253656204789877 (nan) | > current_lr: 5e-06 | > step_time: 0.1514 (0.13637077696136352) | > loader_time: 0.0047 (0.004634007082485612)  --> TIME: 2025-02-25 09:35:36 -- STEP: 22778/36938 -- GLOBAL_STEP: 724600 | > loss_text_ce: 0.03063061833381653 (0.03480913260738559) | > loss_mel_ce: 3.254582405090332 (nan) | > loss: 0.013036560267210007 (nan) | > current_lr: 5e-06 | > step_time: 0.1755 (0.1363728667008937) | > loader_time: 0.0048 (0.004634109181048959)  --> TIME: 2025-02-25 09:35:46 -- STEP: 22828/36938 -- GLOBAL_STEP: 724650 | > loss_text_ce: 0.04430263862013817 (0.03481101827845798) | > loss_mel_ce: 0.06667963415384293 (nan) | > loss: 0.00044040585635229945 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13638454301774713) | > loader_time: 0.0045 (0.004634143425951634)  --> TIME: 2025-02-25 09:35:55 -- STEP: 22878/36938 -- GLOBAL_STEP: 724700 | > loss_text_ce: 0.03934654965996742 (0.03481036223986065) | > loss_mel_ce: 2.7686586380004883 (nan) | > loss: 0.011142877861857414 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13639114125853224) | > loader_time: 0.0047 (0.00463412119402459)  --> TIME: 2025-02-25 09:36:05 -- STEP: 22928/36938 -- GLOBAL_STEP: 724750 | > loss_text_ce: 0.04049236327409744 (0.0348087692475183) | > loss_mel_ce: 2.996335506439209 (nan) | > loss: 0.012050904333591461 (nan) | > current_lr: 5e-06 | > step_time: 0.1266 (0.13639715720495918) | > loader_time: 0.0046 (0.004634148701869415)  --> TIME: 2025-02-25 09:36:14 -- STEP: 22978/36938 -- GLOBAL_STEP: 724800 | > loss_text_ce: 0.04734022170305252 (0.03481210521011772) | > loss_mel_ce: 2.484628915786743 (nan) | > loss: 0.010047497227787971 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13638583715657923) | > loader_time: 0.0047 (0.0046343475629501045)  --> TIME: 2025-02-25 09:36:23 -- STEP: 23028/36938 -- GLOBAL_STEP: 724850 | > loss_text_ce: 0.04051445424556732 (0.03481209479048013) | > loss_mel_ce: 2.5670502185821533 (nan) | > loss: 0.010347479954361916 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13638239379390604) | > loader_time: 0.0045 (0.004634378373426163)  --> TIME: 2025-02-25 09:36:33 -- STEP: 23078/36938 -- GLOBAL_STEP: 724900 | > loss_text_ce: 0.032611601054668427 (0.03481248876253305) | > loss_mel_ce: 2.8994390964508057 (nan) | > loss: 0.011635122820734978 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13639180441644197) | > loader_time: 0.0046 (0.0046346028184382285)  --> TIME: 2025-02-25 09:36:42 -- STEP: 23128/36938 -- GLOBAL_STEP: 724950 | > loss_text_ce: 0.03900010138750076 (0.0348176481350031) | > loss_mel_ce: 2.0774624347686768 (nan) | > loss: 0.008398661389946938 (nan) | > current_lr: 5e-06 | > step_time: 0.1075 (0.13637975544840572) | > loader_time: 0.0045 (0.004634593554493117)  --> TIME: 2025-02-25 09:36:51 -- STEP: 23178/36938 -- GLOBAL_STEP: 725000 | > loss_text_ce: 0.03187748044729233 (0.034816307626433056) | > loss_mel_ce: 3.4661760330200195 (nan) | > loss: 0.013881165534257889 (nan) | > current_lr: 5e-06 | > step_time: 0.189 (0.1363793572438116) | > loader_time: 0.0048 (0.004634765309741864)  --> TIME: 2025-02-25 09:37:00 -- STEP: 23228/36938 -- GLOBAL_STEP: 725050 | > loss_text_ce: 0.0295379851013422 (0.03481979704566576) | > loss_mel_ce: 3.213008403778076 (nan) | > loss: 0.012867247685790062 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13637495071686498) | > loader_time: 0.0047 (0.004634845332761033)  --> TIME: 2025-02-25 09:37:09 -- STEP: 23278/36938 -- GLOBAL_STEP: 725100 | > loss_text_ce: 0.03246292844414711 (0.034818111705740465) | > loss_mel_ce: 3.183764696121216 (nan) | > loss: 0.012762808240950108 (nan) | > current_lr: 5e-06 | > step_time: 0.1439 (0.1363685643798935) | > loader_time: 0.0046 (0.004634843145882248)  --> TIME: 2025-02-25 09:37:19 -- STEP: 23328/36938 -- GLOBAL_STEP: 725150 | > loss_text_ce: 0.03386899456381798 (0.03481998000882775) | > loss_mel_ce: 2.859616994857788 (nan) | > loss: 0.011482087895274162 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.136375317194172) | > loader_time: 0.0046 (0.004634845117809663)  --> TIME: 2025-02-25 09:37:28 -- STEP: 23378/36938 -- GLOBAL_STEP: 725200 | > loss_text_ce: 0.030720429494976997 (0.034816677748112784) | > loss_mel_ce: 2.551574230194092 (nan) | > loss: 0.010247201658785343 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13637656932595632) | > loader_time: 0.0046 (0.004634967963131949)  --> TIME: 2025-02-25 09:37:38 -- STEP: 23428/36938 -- GLOBAL_STEP: 725250 | > loss_text_ce: 0.03220059722661972 (0.03481489427613375) | > loss_mel_ce: 3.1559102535247803 (nan) | > loss: 0.012651233933866024 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13638657781274768) | > loader_time: 0.0047 (0.0046349532758495435)  --> TIME: 2025-02-25 09:37:47 -- STEP: 23478/36938 -- GLOBAL_STEP: 725300 | > loss_text_ce: 0.04405279457569122 (0.03481613897300175) | > loss_mel_ce: 0.0037511189002543688 (nan) | > loss: 0.00018969806842505932 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13639313131329753) | > loader_time: 0.0045 (0.00463495225879562)  --> TIME: 2025-02-25 09:37:56 -- STEP: 23528/36938 -- GLOBAL_STEP: 725350 | > loss_text_ce: 0.03880270570516586 (0.03481700459291119) | > loss_mel_ce: 2.0041537284851074 (nan) | > loss: 0.008106970228254795 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13638542007321885) | > loader_time: 0.0046 (0.0046348364448028445)  --> TIME: 2025-02-25 09:38:06 -- STEP: 23578/36938 -- GLOBAL_STEP: 725400 | > loss_text_ce: 0.039140745997428894 (0.03481934408458437) | > loss_mel_ce: 2.56201171875 (nan) | > loss: 0.010322034358978271 (nan) | > current_lr: 5e-06 | > step_time: 0.1563 (0.1363875667439526) | > loader_time: 0.0047 (0.0046348091663927105)  --> TIME: 2025-02-25 09:38:15 -- STEP: 23628/36938 -- GLOBAL_STEP: 725450 | > loss_text_ce: 0.030282560735940933 (0.03482237150385752) | > loss_mel_ce: 3.5123424530029297 (nan) | > loss: 0.014058036729693413 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13638826766271733) | > loader_time: 0.0046 (0.0046348934632096704)  --> TIME: 2025-02-25 09:38:24 -- STEP: 23678/36938 -- GLOBAL_STEP: 725500 | > loss_text_ce: 0.011423885822296143 (0.03481969757460918) | > loss_mel_ce: 2.1136860847473145 (nan) | > loss: 0.008432975970208645 (nan) | > current_lr: 5e-06 | > step_time: 0.1079 (0.13639071191300828) | > loader_time: 0.0045 (0.004634949371353349)  --> TIME: 2025-02-25 09:38:34 -- STEP: 23728/36938 -- GLOBAL_STEP: 725550 | > loss_text_ce: 0.03427352011203766 (0.0348193018319385) | > loss_mel_ce: 3.128934144973755 (nan) | > loss: 0.01255241222679615 (nan) | > current_lr: 5e-06 | > step_time: 0.19 (0.1363911510759817) | > loader_time: 0.0048 (0.0046348835436460165)  --> TIME: 2025-02-25 09:38:43 -- STEP: 23778/36938 -- GLOBAL_STEP: 725600 | > loss_text_ce: 0.03702308237552643 (0.03481547266858435) | > loss_mel_ce: 2.641573905944824 (nan) | > loss: 0.010629354044795036 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.1363920530766345) | > loader_time: 0.0048 (0.004634834557147504)  --> TIME: 2025-02-25 09:38:52 -- STEP: 23828/36938 -- GLOBAL_STEP: 725650 | > loss_text_ce: 0.03350694105029106 (0.03481636494251184) | > loss_mel_ce: 3.3313615322113037 (nan) | > loss: 0.01335265301167965 (nan) | > current_lr: 5e-06 | > step_time: 0.1833 (0.1363964299137459) | > loader_time: 0.0048 (0.00463493753444113)  --> TIME: 2025-02-25 09:39:02 -- STEP: 23878/36938 -- GLOBAL_STEP: 725700 | > loss_text_ce: 0.041860345751047134 (0.034814356632711305) | > loss_mel_ce: 1.3281892538070679 (nan) | > loss: 0.005436704959720373 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13640301949874736) | > loader_time: 0.0048 (0.004635005612720419)  --> TIME: 2025-02-25 09:39:11 -- STEP: 23928/36938 -- GLOBAL_STEP: 725750 | > loss_text_ce: 0.03414182737469673 (0.0348159306318652) | > loss_mel_ce: 3.0381548404693604 (nan) | > loss: 0.012191654182970524 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13639490583874891) | > loader_time: 0.0046 (0.004634958232616903)  --> TIME: 2025-02-25 09:39:20 -- STEP: 23978/36938 -- GLOBAL_STEP: 725800 | > loss_text_ce: 0.033234111964702606 (0.03481454206326883) | > loss_mel_ce: 2.9819600582122803 (nan) | > loss: 0.011965056881308556 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.1363958930619247) | > loader_time: 0.0046 (0.004635037309145624)  --> TIME: 2025-02-25 09:39:29 -- STEP: 24028/36938 -- GLOBAL_STEP: 725850 | > loss_text_ce: 0.04159203916788101 (0.03481483119881806) | > loss_mel_ce: 2.6028614044189453 (nan) | > loss: 0.010493863373994827 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13639011902203552) | > loader_time: 0.0044 (0.004634933382773802)  --> TIME: 2025-02-25 09:39:39 -- STEP: 24078/36938 -- GLOBAL_STEP: 725900 | > loss_text_ce: 0.032264064997434616 (0.034816669018767496) | > loss_mel_ce: 2.904052734375 (nan) | > loss: 0.011652051471173763 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.13638465145476186) | > loader_time: 0.0051 (0.004634922867112799)  --> TIME: 2025-02-25 09:39:48 -- STEP: 24128/36938 -- GLOBAL_STEP: 725950 | > loss_text_ce: 0.030281580984592438 (0.03481767134840095) | > loss_mel_ce: 3.0864245891571045 (nan) | > loss: 0.012367881834506989 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.13638379382320445) | > loader_time: 0.0047 (0.0046347512786995455)  --> TIME: 2025-02-25 09:39:57 -- STEP: 24178/36938 -- GLOBAL_STEP: 726000 | > loss_text_ce: 0.027982648462057114 (0.0348169287950646) | > loss_mel_ce: 2.3466885089874268 (nan) | > loss: 0.009423298761248589 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.13639153663273396) | > loader_time: 0.0045 (0.004634779542302435)  --> TIME: 2025-02-25 09:40:07 -- STEP: 24228/36938 -- GLOBAL_STEP: 726050 | > loss_text_ce: 0.02812224254012108 (0.03481616238618024) | > loss_mel_ce: 2.5358052253723145 (nan) | > loss: 0.010174315422773361 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13639235958046217) | > loader_time: 0.0045 (0.004634730302600625)  --> TIME: 2025-02-25 09:40:16 -- STEP: 24278/36938 -- GLOBAL_STEP: 726100 | > loss_text_ce: 0.040841441601514816 (0.034817183489460904) | > loss_mel_ce: 2.219996929168701 (nan) | > loss: 0.008971581235527992 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13639177817647916) | > loader_time: 0.0065 (0.004634792186624866)  --> TIME: 2025-02-25 09:40:25 -- STEP: 24328/36938 -- GLOBAL_STEP: 726150 | > loss_text_ce: 0.039025645703077316 (0.03481730895442499) | > loss_mel_ce: 2.3640854358673096 (nan) | > loss: 0.009536155499517918 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13639228798103653) | > loader_time: 0.0047 (0.004634747866618316)  --> TIME: 2025-02-25 09:40:35 -- STEP: 24378/36938 -- GLOBAL_STEP: 726200 | > loss_text_ce: 0.034962236881256104 (0.034819401493018924) | > loss_mel_ce: 3.4855668544769287 (nan) | > loss: 0.013970354571938515 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13639046769506422) | > loader_time: 0.0045 (0.004634676089933456)  --> TIME: 2025-02-25 09:40:44 -- STEP: 24428/36938 -- GLOBAL_STEP: 726250 | > loss_text_ce: 0.04037385806441307 (0.03481536703384048) | > loss_mel_ce: 3.1950602531433105 (nan) | > loss: 0.012839024886488914 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.1363857608780804) | > loader_time: 0.0046 (0.004634628997451052)  --> TIME: 2025-02-25 09:40:53 -- STEP: 24478/36938 -- GLOBAL_STEP: 726300 | > loss_text_ce: 0.03725632652640343 (0.03481719178966493) | > loss_mel_ce: 2.503391981124878 (nan) | > loss: 0.010081937536597252 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13638029590191886) | > loader_time: 0.0045 (0.004634555847740836)  --> TIME: 2025-02-25 09:41:03 -- STEP: 24528/36938 -- GLOBAL_STEP: 726350 | > loss_text_ce: 0.03695419803261757 (0.03481868806664793) | > loss_mel_ce: 2.066589832305908 (nan) | > loss: 0.008347397670149803 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.13639048191938125) | > loader_time: 0.0045 (0.004634578867201227)  --> TIME: 2025-02-25 09:41:12 -- STEP: 24578/36938 -- GLOBAL_STEP: 726400 | > loss_text_ce: 0.03206796944141388 (0.034819305152858) | > loss_mel_ce: 2.9899396896362305 (nan) | > loss: 0.011992095038294792 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.1363880882446335) | > loader_time: 0.0046 (0.004634588328726317)  --> TIME: 2025-02-25 09:41:21 -- STEP: 24628/36938 -- GLOBAL_STEP: 726450 | > loss_text_ce: 0.035414304584264755 (0.03482085045943521) | > loss_mel_ce: 2.7138078212738037 (nan) | > loss: 0.0109096122905612 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13639650917758048) | > loader_time: 0.0046 (0.004634523742168344)  --> TIME: 2025-02-25 09:41:31 -- STEP: 24678/36938 -- GLOBAL_STEP: 726500 | > loss_text_ce: 0.02825990319252014 (0.03482132952627744) | > loss_mel_ce: 2.7624151706695557 (nan) | > loss: 0.011074108071625233 (nan) | > current_lr: 5e-06 | > step_time: 0.1613 (0.13640111612823472) | > loader_time: 0.0046 (0.004634464054693517)  --> TIME: 2025-02-25 09:41:40 -- STEP: 24728/36938 -- GLOBAL_STEP: 726550 | > loss_text_ce: 0.038452159613370895 (0.034820029617804844) | > loss_mel_ce: 3.2092514038085938 (nan) | > loss: 0.012887713499367237 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.1363994607509672) | > loader_time: 0.0045 (0.004634409130525909)  --> TIME: 2025-02-25 09:41:49 -- STEP: 24778/36938 -- GLOBAL_STEP: 726600 | > loss_text_ce: 0.028946971520781517 (0.03482115798950012) | > loss_mel_ce: 2.0291545391082764 (nan) | > loss: 0.008167069405317307 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13640121495101218) | > loader_time: 0.0048 (0.004634318518017014)  --> TIME: 2025-02-25 09:41:59 -- STEP: 24828/36938 -- GLOBAL_STEP: 726650 | > loss_text_ce: 0.03661878779530525 (0.034819490901885415) | > loss_mel_ce: 3.030250072479248 (nan) | > loss: 0.012170114554464817 (nan) | > current_lr: 5e-06 | > step_time: 0.1687 (0.13641081463989124) | > loader_time: 0.0048 (0.004634404376410851)  --> TIME: 2025-02-25 09:42:08 -- STEP: 24878/36938 -- GLOBAL_STEP: 726700 | > loss_text_ce: 0.03737986832857132 (0.034819687513837155) | > loss_mel_ce: 2.3688619136810303 (nan) | > loss: 0.009548579342663288 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13641010826543884) | > loader_time: 0.0045 (0.00463446749302244)  --> TIME: 2025-02-25 09:42:17 -- STEP: 24928/36938 -- GLOBAL_STEP: 726750 | > loss_text_ce: 0.04082103818655014 (0.034820515501632385) | > loss_mel_ce: 0.019287109375 (nan) | > loss: 0.0002385244151810184 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.1364137977552967) | > loader_time: 0.0045 (0.004634554592344969)  --> TIME: 2025-02-25 09:42:27 -- STEP: 24978/36938 -- GLOBAL_STEP: 726800 | > loss_text_ce: 0.044732823967933655 (0.03482205361560097) | > loss_mel_ce: 2.878800630569458 (nan) | > loss: 0.011601324193179607 (nan) | > current_lr: 5e-06 | > step_time: 0.1574 (0.13642544045403648) | > loader_time: 0.0047 (0.004634630681038737)  --> TIME: 2025-02-25 09:42:37 -- STEP: 25028/36938 -- GLOBAL_STEP: 726850 | > loss_text_ce: 0.03352484852075577 (0.03482113393592669) | > loss_mel_ce: 2.9326870441436768 (nan) | > loss: 0.011770682409405708 (nan) | > current_lr: 5e-06 | > step_time: 0.1816 (0.13643115815099732) | > loader_time: 0.0048 (0.004634749190159943)  --> TIME: 2025-02-25 09:42:46 -- STEP: 25078/36938 -- GLOBAL_STEP: 726900 | > loss_text_ce: 0.031859517097473145 (0.03482138843100572) | > loss_mel_ce: 2.7873103618621826 (nan) | > loss: 0.011187182739377022 (nan) | > current_lr: 5e-06 | > step_time: 0.1582 (0.13642683572219305) | > loader_time: 0.0047 (0.004634790428518169)  --> TIME: 2025-02-25 09:42:55 -- STEP: 25128/36938 -- GLOBAL_STEP: 726950 | > loss_text_ce: 0.051100991666316986 (0.0348238231112326) | > loss_mel_ce: 3.1929078102111816 (nan) | > loss: 0.012873051688075066 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13642617680585478) | > loader_time: 0.0045 (0.0046348015391420505)  --> TIME: 2025-02-25 09:43:04 -- STEP: 25178/36938 -- GLOBAL_STEP: 727000 | > loss_text_ce: 0.031570326536893845 (0.034823459339945344) | > loss_mel_ce: 0.8973772525787354 (nan) | > loss: 0.0036863000132143497 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13643160792622083) | > loader_time: 0.0045 (0.004634850416638752)  --> TIME: 2025-02-25 09:43:14 -- STEP: 25228/36938 -- GLOBAL_STEP: 727050 | > loss_text_ce: 0.0329909548163414 (0.03482313289080277) | > loss_mel_ce: 3.23323917388916 (nan) | > loss: 0.012961231172084808 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13642956707960335) | > loader_time: 0.0047 (0.004634874774666121)  --> TIME: 2025-02-25 09:43:23 -- STEP: 25278/36938 -- GLOBAL_STEP: 727100 | > loss_text_ce: 0.0420781746506691 (0.0348233709291092) | > loss_mel_ce: 1.1754271984100342 (nan) | > loss: 0.004831370431929827 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13641708961477472) | > loader_time: 0.0045 (0.004634932236483181)  --> TIME: 2025-02-25 09:43:32 -- STEP: 25328/36938 -- GLOBAL_STEP: 727150 | > loss_text_ce: 0.03566723316907883 (0.03482062777770429) | > loss_mel_ce: 3.288032054901123 (nan) | > loss: 0.013189283199608326 (nan) | > current_lr: 5e-06 | > step_time: 0.1624 (0.13641193963761444) | > loader_time: 0.0046 (0.004634910061324755)  --> TIME: 2025-02-25 09:43:41 -- STEP: 25378/36938 -- GLOBAL_STEP: 727200 | > loss_text_ce: 0.04082917422056198 (0.034818118914575466) | > loss_mel_ce: 0.15374755859375 (nan) | > loss: 0.0007721299771219492 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1364104625923019) | > loader_time: 0.0045 (0.004634947658045475)  --> TIME: 2025-02-25 09:43:51 -- STEP: 25428/36938 -- GLOBAL_STEP: 727250 | > loss_text_ce: 0.03581012412905693 (0.034815688450989965) | > loss_mel_ce: 0.062255859375 (nan) | > loss: 0.00038915075128898025 (nan) | > current_lr: 5e-06 | > step_time: 0.1533 (0.13642376167173764) | > loader_time: 0.0047 (0.0046349262711160456)  --> TIME: 2025-02-25 09:44:00 -- STEP: 25478/36938 -- GLOBAL_STEP: 727300 | > loss_text_ce: 0.03575636446475983 (0.03481558522569833) | > loss_mel_ce: 2.678110361099243 (nan) | > loss: 0.01076931320130825 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.13642144261204545) | > loader_time: 0.0047 (0.004635004806728158)  --> TIME: 2025-02-25 09:44:09 -- STEP: 25528/36938 -- GLOBAL_STEP: 727350 | > loss_text_ce: 0.04042375460267067 (0.03481457693785005) | > loss_mel_ce: 3.256977081298828 (nan) | > loss: 0.013084924779832363 (nan) | > current_lr: 5e-06 | > step_time: 0.1625 (0.13641740883715447) | > loader_time: 0.0046 (0.004634985390295236)  --> TIME: 2025-02-25 09:44:19 -- STEP: 25578/36938 -- GLOBAL_STEP: 727400 | > loss_text_ce: 0.03791226074099541 (0.0348115048722771) | > loss_mel_ce: 2.9527933597564697 (nan) | > loss: 0.011867880821228027 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13642164014541433) | > loader_time: 0.0046 (0.00463516736983208)  --> TIME: 2025-02-25 09:44:28 -- STEP: 25628/36938 -- GLOBAL_STEP: 727450 | > loss_text_ce: 0.04266416281461716 (0.03481226713535167) | > loss_mel_ce: 3.6566154956817627 (nan) | > loss: 0.0146796815097332 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13642522416844927) | > loader_time: 0.0045 (0.004635176625883624)  --> TIME: 2025-02-25 09:44:38 -- STEP: 25678/36938 -- GLOBAL_STEP: 727500 | > loss_text_ce: 0.035878803580999374 (0.03481040808512009) | > loss_mel_ce: 3.285770893096924 (nan) | > loss: 0.013181150890886784 (nan) | > current_lr: 5e-06 | > step_time: 0.1865 (0.13643139956831077) | > loader_time: 0.005 (0.00463521772107283)  --> TIME: 2025-02-25 09:44:47 -- STEP: 25728/36938 -- GLOBAL_STEP: 727550 | > loss_text_ce: 0.03889595344662666 (0.034811250375907245) | > loss_mel_ce: 2.7304575443267822 (nan) | > loss: 0.010989498347043991 (nan) | > current_lr: 5e-06 | > step_time: 0.1494 (0.13643156292278374) | > loader_time: 0.0046 (0.004635128567912676)  --> TIME: 2025-02-25 09:44:56 -- STEP: 25778/36938 -- GLOBAL_STEP: 727600 | > loss_text_ce: 0.030550913885235786 (0.03480816596893706) | > loss_mel_ce: 2.8740155696868896 (nan) | > loss: 0.011526058427989483 (nan) | > current_lr: 5e-06 | > step_time: 0.1693 (0.13643768017948135) | > loader_time: 0.0048 (0.004635266312821069)  --> TIME: 2025-02-25 09:45:06 -- STEP: 25828/36938 -- GLOBAL_STEP: 727650 | > loss_text_ce: 0.035061538219451904 (0.03480807305134106) | > loss_mel_ce: 2.556993007659912 (nan) | > loss: 0.010285931639373302 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13644042772203294) | > loader_time: 0.0046 (0.004635556629979049)  --> TIME: 2025-02-25 09:45:15 -- STEP: 25878/36938 -- GLOBAL_STEP: 727700 | > loss_text_ce: 0.04779158905148506 (0.03480650945566265) | > loss_mel_ce: 2.7233288288116455 (nan) | > loss: 0.010996509343385696 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13644855523995747) | > loader_time: 0.0046 (0.004635493513402556)  --> TIME: 2025-02-25 09:45:25 -- STEP: 25928/36938 -- GLOBAL_STEP: 727750 | > loss_text_ce: 0.03889593854546547 (0.034806335744480214) | > loss_mel_ce: 3.5259337425231934 (nan) | > loss: 0.014146150089800358 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13644902050881347) | > loader_time: 0.0046 (0.004635386814934272)  --> TIME: 2025-02-25 09:45:34 -- STEP: 25978/36938 -- GLOBAL_STEP: 727800 | > loss_text_ce: 0.03804674372076988 (0.034804003112529236) | > loss_mel_ce: 2.6864404678344727 (nan) | > loss: 0.010811458341777325 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13645095429268853) | > loader_time: 0.0047 (0.0046355295093168285)  --> TIME: 2025-02-25 09:45:44 -- STEP: 26028/36938 -- GLOBAL_STEP: 727850 | > loss_text_ce: 0.036976344883441925 (0.03480249932151054) | > loss_mel_ce: 2.772611379623413 (nan) | > loss: 0.011149158701300621 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.1364572182107635) | > loader_time: 0.0046 (0.004635518901960365)  --> TIME: 2025-02-25 09:45:53 -- STEP: 26078/36938 -- GLOBAL_STEP: 727900 | > loss_text_ce: 0.03844957426190376 (0.034803890732612815) | > loss_mel_ce: 1.924259066581726 (nan) | > loss: 0.007788526825606823 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1364571257725168) | > loader_time: 0.0046 (0.004635583075362392)  --> TIME: 2025-02-25 09:46:02 -- STEP: 26128/36938 -- GLOBAL_STEP: 727950 | > loss_text_ce: 0.03824934363365173 (0.03480433578846385) | > loss_mel_ce: 2.962364673614502 (nan) | > loss: 0.011907198466360569 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.1364643322731627) | > loader_time: 0.0048 (0.0046356027467953035)  --> TIME: 2025-02-25 09:46:12 -- STEP: 26178/36938 -- GLOBAL_STEP: 728000 | > loss_text_ce: 0.03440304473042488 (0.03480398850754593) | > loss_mel_ce: 2.992642641067505 (nan) | > loss: 0.012012086808681488 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.1364594532740527) | > loader_time: 0.0047 (0.00463555174101955)  --> TIME: 2025-02-25 09:46:21 -- STEP: 26228/36938 -- GLOBAL_STEP: 728050 | > loss_text_ce: 0.03532539680600166 (0.034804023514766705) | > loss_mel_ce: 1.8725875616073608 (nan) | > loss: 0.00757108349353075 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13645760794320047) | > loader_time: 0.0045 (0.0046355836871776746)  --> TIME: 2025-02-25 09:46:30 -- STEP: 26278/36938 -- GLOBAL_STEP: 728100 | > loss_text_ce: 0.031047767028212547 (0.03480431360978503) | > loss_mel_ce: 2.474754810333252 (nan) | > loss: 0.009943662211298943 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13646018926987893) | > loader_time: 0.0045 (0.004635666329273303)  --> TIME: 2025-02-25 09:46:40 -- STEP: 26328/36938 -- GLOBAL_STEP: 728150 | > loss_text_ce: 0.03231310471892357 (0.03480414465401331) | > loss_mel_ce: 2.55896258354187 (nan) | > loss: 0.010282840579748154 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13646298732956066) | > loader_time: 0.0045 (0.0046356572491968355)  --> TIME: 2025-02-25 09:46:49 -- STEP: 26378/36938 -- GLOBAL_STEP: 728200 | > loss_text_ce: 0.03493451699614525 (0.03479863397878943) | > loss_mel_ce: 2.8078525066375732 (nan) | > loss: 0.011280901730060577 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13644853647126343) | > loader_time: 0.0046 (0.0046355580802719095)  --> TIME: 2025-02-25 09:46:58 -- STEP: 26428/36938 -- GLOBAL_STEP: 728250 | > loss_text_ce: 0.03617286682128906 (0.034797595741664064) | > loss_mel_ce: 2.85221004486084 (nan) | > loss: 0.011461837217211723 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13645346742616726) | > loader_time: 0.0047 (0.004635525242326408)  --> TIME: 2025-02-25 09:47:08 -- STEP: 26478/36938 -- GLOBAL_STEP: 728300 | > loss_text_ce: 0.01888280175626278 (0.03479825050441163) | > loss_mel_ce: 2.7323176860809326 (nan) | > loss: 0.010917462408542633 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13646119957703542) | > loader_time: 0.0064 (0.004635699863781046)  --> TIME: 2025-02-25 09:47:17 -- STEP: 26528/36938 -- GLOBAL_STEP: 728350 | > loss_text_ce: 0.027686020359396935 (0.034799908293054536) | > loss_mel_ce: 3.157740831375122 (nan) | > loss: 0.012640584260225296 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13645350357879346) | > loader_time: 0.0045 (0.004635737487648593)  --> TIME: 2025-02-25 09:47:26 -- STEP: 26578/36938 -- GLOBAL_STEP: 728400 | > loss_text_ce: 0.03015032969415188 (0.03480022037427059) | > loss_mel_ce: 2.6627557277679443 (nan) | > loss: 0.010686135850846767 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13644414945043087) | > loader_time: 0.0047 (0.004635840257427493)  --> TIME: 2025-02-25 09:47:35 -- STEP: 26628/36938 -- GLOBAL_STEP: 728450 | > loss_text_ce: 0.04718232527375221 (0.03479622556943834) | > loss_mel_ce: 3.40846848487854 (nan) | > loss: 0.013712900690734386 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.1364475903755304) | > loader_time: 0.0047 (0.004635774616509633)  --> TIME: 2025-02-25 09:47:44 -- STEP: 26678/36938 -- GLOBAL_STEP: 728500 | > loss_text_ce: 0.03243236243724823 (0.03479780504939169) | > loss_mel_ce: 2.5139827728271484 (nan) | > loss: 0.010104821994900703 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13643562620355865) | > loader_time: 0.0045 (0.0046357219209734605)  --> TIME: 2025-02-25 09:47:54 -- STEP: 26728/36938 -- GLOBAL_STEP: 728550 | > loss_text_ce: 0.03319444879889488 (0.034798772899015874) | > loss_mel_ce: 3.3296210765838623 (nan) | > loss: 0.013344506733119488 (nan) | > current_lr: 5e-06 | > step_time: 0.182 (0.13644447661345332) | > loader_time: 0.0047 (0.004635635597268391)  --> TIME: 2025-02-25 09:48:03 -- STEP: 26778/36938 -- GLOBAL_STEP: 728600 | > loss_text_ce: 0.04026403650641441 (0.03479949353393306) | > loss_mel_ce: 2.511275053024292 (nan) | > loss: 0.010125156491994858 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13643836022420547) | > loader_time: 0.0046 (0.004635579574098327)  --> TIME: 2025-02-25 09:48:12 -- STEP: 26828/36938 -- GLOBAL_STEP: 728650 | > loss_text_ce: 0.03472934290766716 (0.03479967489803222) | > loss_mel_ce: 2.686235189437866 (nan) | > loss: 0.010797478258609772 (nan) | > current_lr: 5e-06 | > step_time: 0.1629 (0.13643334097454884) | > loader_time: 0.0046 (0.004635517059005953)  --> TIME: 2025-02-25 09:48:21 -- STEP: 26878/36938 -- GLOBAL_STEP: 728700 | > loss_text_ce: 0.035272348672151566 (0.03479725089669452) | > loss_mel_ce: 2.22344970703125 (nan) | > loss: 0.008963183499872684 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13643863846196777) | > loader_time: 0.0044 (0.0046354741139713095)  --> TIME: 2025-02-25 09:48:31 -- STEP: 26928/36938 -- GLOBAL_STEP: 728750 | > loss_text_ce: 0.038053713738918304 (0.03479500936213006) | > loss_mel_ce: 0.03826904296875 (nan) | > loss: 0.00030286810942925513 (nan) | > current_lr: 5e-06 | > step_time: 0.1262 (0.13644126850051413) | > loader_time: 0.0045 (0.004635452852317137)  --> TIME: 2025-02-25 09:48:40 -- STEP: 26978/36938 -- GLOBAL_STEP: 728800 | > loss_text_ce: 0.02968866005539894 (0.03479335395006474) | > loss_mel_ce: 2.816070318222046 (nan) | > loss: 0.011292695067822933 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13644904229204233) | > loader_time: 0.0046 (0.004635562022874663)  --> TIME: 2025-02-25 09:48:50 -- STEP: 27028/36938 -- GLOBAL_STEP: 728850 | > loss_text_ce: 0.03114030510187149 (0.034791153690604176) | > loss_mel_ce: 2.157334804534912 (nan) | > loss: 0.008684425614774227 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13644730960968202) | > loader_time: 0.0047 (0.004635474086248853)  --> TIME: 2025-02-25 09:48:59 -- STEP: 27078/36938 -- GLOBAL_STEP: 728900 | > loss_text_ce: 0.043830156326293945 (0.034792045984677236) | > loss_mel_ce: 2.740516424179077 (nan) | > loss: 0.011048994958400726 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13645266032005884) | > loader_time: 0.0046 (0.004635408970848106)  --> TIME: 2025-02-25 09:49:09 -- STEP: 27128/36938 -- GLOBAL_STEP: 728950 | > loss_text_ce: 0.030845405533909798 (0.03479183886167401) | > loss_mel_ce: 2.3594436645507812 (nan) | > loss: 0.00948527455329895 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.136457450329748) | > loader_time: 0.0046 (0.0046354109331837945)  --> TIME: 2025-02-25 09:49:18 -- STEP: 27178/36938 -- GLOBAL_STEP: 729000 | > loss_text_ce: 0.039984095841646194 (0.03479077348964541) | > loss_mel_ce: 2.9308080673217773 (nan) | > loss: 0.011788858100771904 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.1364540431101038) | > loader_time: 0.0047 (0.004635486743844746)  --> TIME: 2025-02-25 09:49:27 -- STEP: 27228/36938 -- GLOBAL_STEP: 729050 | > loss_text_ce: 0.036660145968198776 (0.03479086221220414) | > loss_mel_ce: 2.5382251739501953 (nan) | > loss: 0.010217799805104733 (nan) | > current_lr: 5e-06 | > step_time: 0.1591 (0.1364564854650051) | > loader_time: 0.0046 (0.004635498853652364)  --> TIME: 2025-02-25 09:49:37 -- STEP: 27278/36938 -- GLOBAL_STEP: 729100 | > loss_text_ce: 0.03164321184158325 (0.03478981604249563) | > loss_mel_ce: 3.106548309326172 (nan) | > loss: 0.012453141622245312 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13646509789137493) | > loader_time: 0.0046 (0.004635453801043556)  --> TIME: 2025-02-25 09:49:46 -- STEP: 27328/36938 -- GLOBAL_STEP: 729150 | > loss_text_ce: 0.028272759169340134 (0.03478631604199391) | > loss_mel_ce: 3.0171775817871094 (nan) | > loss: 0.012085121124982834 (nan) | > current_lr: 5e-06 | > step_time: 0.178 (0.13646165973373858) | > loader_time: 0.0048 (0.004635470463464625)  --> TIME: 2025-02-25 09:49:55 -- STEP: 27378/36938 -- GLOBAL_STEP: 729200 | > loss_text_ce: 0.02804717794060707 (0.034782839621243976) | > loss_mel_ce: 2.517744541168213 (nan) | > loss: 0.010102348402142525 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1364568674117623) | > loader_time: 0.0046 (0.004635422857990826)  --> TIME: 2025-02-25 09:50:04 -- STEP: 27428/36938 -- GLOBAL_STEP: 729250 | > loss_text_ce: 0.03998929262161255 (0.03478254122393794) | > loss_mel_ce: 2.4597585201263428 (nan) | > loss: 0.009919634088873863 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1364486040779745) | > loader_time: 0.0045 (0.004635298332076939)  --> TIME: 2025-02-25 09:50:14 -- STEP: 27478/36938 -- GLOBAL_STEP: 729300 | > loss_text_ce: 0.03657911345362663 (0.034783849630513855) | > loss_mel_ce: 2.955728530883789 (nan) | > loss: 0.011874237097799778 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.13645785200568963) | > loader_time: 0.0048 (0.00463537650473106)  --> TIME: 2025-02-25 09:50:23 -- STEP: 27528/36938 -- GLOBAL_STEP: 729350 | > loss_text_ce: 0.0372110940515995 (0.03478339864952002) | > loss_mel_ce: 3.1007730960845947 (nan) | > loss: 0.012452319264411926 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13645140692890467) | > loader_time: 0.0046 (0.0046352782383868585)  --> TIME: 2025-02-25 09:50:32 -- STEP: 27578/36938 -- GLOBAL_STEP: 729400 | > loss_text_ce: 0.028145067393779755 (0.03478079401984639) | > loss_mel_ce: 4.359375 (nan) | > loss: 0.017410794273018837 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13644833625793718) | > loader_time: 0.0046 (0.004635290961570217)  --> TIME: 2025-02-25 09:50:42 -- STEP: 27628/36938 -- GLOBAL_STEP: 729450 | > loss_text_ce: 0.0383576862514019 (0.03478268242817481) | > loss_mel_ce: 2.708923578262329 (nan) | > loss: 0.01090191025286913 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.1364562665137259) | > loader_time: 0.0047 (0.004635322675598635)  --> TIME: 2025-02-25 09:50:51 -- STEP: 27678/36938 -- GLOBAL_STEP: 729500 | > loss_text_ce: 0.029592327773571014 (0.03478002375183868) | > loss_mel_ce: 1.5285905599594116 (nan) | > loss: 0.006183265708386898 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13646146801305503) | > loader_time: 0.0045 (0.004635270839740499)  --> TIME: 2025-02-25 09:51:01 -- STEP: 27728/36938 -- GLOBAL_STEP: 729550 | > loss_text_ce: 0.027862166985869408 (0.034782149636243875) | > loss_mel_ce: 0.8657006621360779 (nan) | > loss: 0.003545884508639574 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13646875489535362) | > loader_time: 0.0045 (0.004635174719501762)  --> TIME: 2025-02-25 09:51:10 -- STEP: 27778/36938 -- GLOBAL_STEP: 729600 | > loss_text_ce: 0.035858821123838425 (0.03478400342455319) | > loss_mel_ce: 2.7957301139831543 (nan) | > loss: 0.011236464604735374 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13646913330072055) | > loader_time: 0.0046 (0.004635138434067389)  --> TIME: 2025-02-25 09:51:19 -- STEP: 27828/36938 -- GLOBAL_STEP: 729650 | > loss_text_ce: 0.030043330043554306 (0.03478461917167433) | > loss_mel_ce: 1.842421054840088 (nan) | > loss: 0.007430414669215679 (nan) | > current_lr: 5e-06 | > step_time: 0.1065 (0.1364662353893418) | > loader_time: 0.0044 (0.004635114470689272)  --> TIME: 2025-02-25 09:51:29 -- STEP: 27878/36938 -- GLOBAL_STEP: 729700 | > loss_text_ce: 0.02130313776433468 (0.034782549564659586) | > loss_mel_ce: 0.02436319924890995 (nan) | > loss: 0.00018121564062312245 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13647308394251897) | > loader_time: 0.0045 (0.0046351780909549105)  --> TIME: 2025-02-25 09:51:38 -- STEP: 27928/36938 -- GLOBAL_STEP: 729750 | > loss_text_ce: 0.035820797085762024 (0.03478003400807463) | > loss_mel_ce: 2.8696115016937256 (nan) | > loss: 0.01152949407696724 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.13646286273542138) | > loader_time: 0.0047 (0.004635003355095082)  --> TIME: 2025-02-25 09:51:47 -- STEP: 27978/36938 -- GLOBAL_STEP: 729800 | > loss_text_ce: 0.040895674377679825 (0.03477965287512178) | > loss_mel_ce: 3.095703125 (nan) | > loss: 0.012446821667253971 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13646616050501495) | > loader_time: 0.0045 (0.004635030857377686)  --> TIME: 2025-02-25 09:51:56 -- STEP: 28028/36938 -- GLOBAL_STEP: 729850 | > loss_text_ce: 0.0391179621219635 (0.03478176460885978) | > loss_mel_ce: 2.6758615970611572 (nan) | > loss: 0.010773729532957077 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13645909299383602) | > loader_time: 0.0045 (0.004634936934142165)  --> TIME: 2025-02-25 09:52:05 -- STEP: 28078/36938 -- GLOBAL_STEP: 729900 | > loss_text_ce: 0.031965840607881546 (0.03477948935090431) | > loss_mel_ce: 2.742558479309082 (nan) | > loss: 0.01101001724600792 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13645478344955397) | > loader_time: 0.0045 (0.0046349934800006196)  --> TIME: 2025-02-25 09:52:15 -- STEP: 28128/36938 -- GLOBAL_STEP: 729950 | > loss_text_ce: 0.03944946825504303 (0.034781095285678215) | > loss_mel_ce: 3.2668869495391846 (nan) | > loss: 0.013120383024215698 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.13646452026325132) | > loader_time: 0.0047 (0.00463496196900618)  --> TIME: 2025-02-25 09:52:24 -- STEP: 28178/36938 -- GLOBAL_STEP: 730000 | > loss_text_ce: 0.030018901452422142 (0.034778145327979244) | > loss_mel_ce: 2.8951566219329834 (nan) | > loss: 0.011607839725911617 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13646216525018845) | > loader_time: 0.0046 (0.004634864758934132) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_730000.pth  --> TIME: 2025-02-25 09:52:40 -- STEP: 28228/36938 -- GLOBAL_STEP: 730050 | > loss_text_ce: 0.044282082468271255 (0.034778095840068746) | > loss_mel_ce: 3.152031183242798 (nan) | > loss: 0.012683783657848835 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13645789642517753) | > loader_time: 0.0045 (0.004634901097835382)  --> TIME: 2025-02-25 09:52:49 -- STEP: 28278/36938 -- GLOBAL_STEP: 730100 | > loss_text_ce: 0.04701413959264755 (0.034777915456437665) | > loss_mel_ce: 2.0826330184936523 (nan) | > loss: 0.008450981229543686 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13645225792408675) | > loader_time: 0.0044 (0.004634843772064212)  --> TIME: 2025-02-25 09:52:58 -- STEP: 28328/36938 -- GLOBAL_STEP: 730150 | > loss_text_ce: 0.06281447410583496 (0.03478344803905112) | > loss_mel_ce: 2.562678098678589 (nan) | > loss: 0.010418621823191643 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13644612480172713) | > loader_time: 0.0045 (0.004634901228950236)  --> TIME: 2025-02-25 09:53:07 -- STEP: 28378/36938 -- GLOBAL_STEP: 730200 | > loss_text_ce: 0.03913336247205734 (0.03478295712292883) | > loss_mel_ce: 3.104703664779663 (nan) | > loss: 0.012475544586777687 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13644823096117556) | > loader_time: 0.0047 (0.004634959096678028)  --> TIME: 2025-02-25 09:53:17 -- STEP: 28428/36938 -- GLOBAL_STEP: 730250 | > loss_text_ce: 0.030043022707104683 (0.03478244909887199) | > loss_mel_ce: 2.874727487564087 (nan) | > loss: 0.011526867747306824 (nan) | > current_lr: 5e-06 | > step_time: 0.1812 (0.13645484293330715) | > loader_time: 0.0048 (0.004634954967259921)  --> TIME: 2025-02-25 09:53:26 -- STEP: 28478/36938 -- GLOBAL_STEP: 730300 | > loss_text_ce: 0.02919868379831314 (0.03478040222037501) | > loss_mel_ce: 1.391870141029358 (nan) | > loss: 0.005639162380248308 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13645152458272877) | > loader_time: 0.0045 (0.004634921073042879)  --> TIME: 2025-02-25 09:53:35 -- STEP: 28528/36938 -- GLOBAL_STEP: 730350 | > loss_text_ce: 0.03406329080462456 (0.03477986308404449) | > loss_mel_ce: 3.1800975799560547 (nan) | > loss: 0.012754607945680618 (nan) | > current_lr: 5e-06 | > step_time: 0.1515 (0.13645855076214064) | > loader_time: 0.0047 (0.004634903828481305)  --> TIME: 2025-02-25 09:53:45 -- STEP: 28578/36938 -- GLOBAL_STEP: 730400 | > loss_text_ce: 0.032860372215509415 (0.03477629959482011) | > loss_mel_ce: 2.671743631362915 (nan) | > loss: 0.010732555761933327 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13646909417044217) | > loader_time: 0.0047 (0.004634944492761625)  --> TIME: 2025-02-25 09:53:55 -- STEP: 28628/36938 -- GLOBAL_STEP: 730450 | > loss_text_ce: 0.02643420919775963 (0.03477658759588533) | > loss_mel_ce: 2.0759665966033936 (nan) | > loss: 0.008342860266566277 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13647308950236314) | > loader_time: 0.0045 (0.004634973538793027)  --> TIME: 2025-02-25 09:54:04 -- STEP: 28678/36938 -- GLOBAL_STEP: 730500 | > loss_text_ce: 0.03536112606525421 (0.03477716221509783) | > loss_mel_ce: 3.104774236679077 (nan) | > loss: 0.012460854835808277 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.1364704946633937) | > loader_time: 0.0046 (0.004635127171519674)  --> TIME: 2025-02-25 09:54:14 -- STEP: 28728/36938 -- GLOBAL_STEP: 730550 | > loss_text_ce: 0.03275398910045624 (0.03477702043726622) | > loss_mel_ce: 3.182605743408203 (nan) | > loss: 0.01275936421006918 (nan) | > current_lr: 5e-06 | > step_time: 0.1623 (0.13648364049948153) | > loader_time: 0.0048 (0.004635184098671017)  --> TIME: 2025-02-25 09:54:23 -- STEP: 28778/36938 -- GLOBAL_STEP: 730600 | > loss_text_ce: 0.03745928406715393 (0.03477815713999984) | > loss_mel_ce: 0.0308837890625 (nan) | > loss: 0.00027120267623104155 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13647461843023348) | > loader_time: 0.0047 (0.004635171186386297)  --> TIME: 2025-02-25 09:54:32 -- STEP: 28828/36938 -- GLOBAL_STEP: 730650 | > loss_text_ce: 0.05447819083929062 (0.03477785808562266) | > loss_mel_ce: 3.1065056324005127 (nan) | > loss: 0.012543587014079094 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1364758604688025) | > loader_time: 0.0048 (0.004635207494585127)  --> TIME: 2025-02-25 09:54:42 -- STEP: 28878/36938 -- GLOBAL_STEP: 730700 | > loss_text_ce: 0.02850397303700447 (0.03477727880689051) | > loss_mel_ce: 3.3875436782836914 (nan) | > loss: 0.013555744662880898 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13649020222912828) | > loader_time: 0.0044 (0.004635935328708341)  --> TIME: 2025-02-25 09:54:51 -- STEP: 28928/36938 -- GLOBAL_STEP: 730750 | > loss_text_ce: 0.03189459070563316 (0.034777189156178966) | > loss_mel_ce: 3.069620132446289 (nan) | > loss: 0.012307599186897278 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13649071619508021) | > loader_time: 0.0046 (0.004635941484465005)  --> TIME: 2025-02-25 09:55:01 -- STEP: 28978/36938 -- GLOBAL_STEP: 730800 | > loss_text_ce: 0.03542288392782211 (0.03477773347112847) | > loss_mel_ce: 3.213242769241333 (nan) | > loss: 0.012891530990600586 (nan) | > current_lr: 5e-06 | > step_time: 0.1957 (0.13650042514259983) | > loader_time: 0.005 (0.004636086500392818)  --> TIME: 2025-02-25 09:55:10 -- STEP: 29028/36938 -- GLOBAL_STEP: 730850 | > loss_text_ce: 0.03663529455661774 (0.03477964664926715) | > loss_mel_ce: 3.366692066192627 (nan) | > loss: 0.013505268841981888 (nan) | > current_lr: 5e-06 | > step_time: 0.1642 (0.13649604561835815) | > loader_time: 0.0049 (0.0046361535068909845)  --> TIME: 2025-02-25 09:55:19 -- STEP: 29078/36938 -- GLOBAL_STEP: 730900 | > loss_text_ce: 0.04518931731581688 (0.034779497781604995) | > loss_mel_ce: 2.897916078567505 (nan) | > loss: 0.01167899090796709 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.13649094913215562) | > loader_time: 0.0049 (0.004636127516331109)  --> TIME: 2025-02-25 09:55:29 -- STEP: 29128/36938 -- GLOBAL_STEP: 730950 | > loss_text_ce: 0.02674809843301773 (0.034779796475078964) | > loss_mel_ce: 2.5430819988250732 (nan) | > loss: 0.010197739116847515 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13649760259929775) | > loader_time: 0.0046 (0.0046361168067358995)  --> TIME: 2025-02-25 09:55:38 -- STEP: 29178/36938 -- GLOBAL_STEP: 731000 | > loss_text_ce: 0.03807499259710312 (0.03477764669488512) | > loss_mel_ce: 3.602522611618042 (nan) | > loss: 0.01444681640714407 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13649446895299197) | > loader_time: 0.0045 (0.004636099172002919)  --> TIME: 2025-02-25 09:55:47 -- STEP: 29228/36938 -- GLOBAL_STEP: 731050 | > loss_text_ce: 0.028980199247598648 (0.03477533348631201) | > loss_mel_ce: 3.1080665588378906 (nan) | > loss: 0.012448599562048912 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.1364855807816525) | > loader_time: 0.0046 (0.0046360484549100925)  --> TIME: 2025-02-25 09:55:56 -- STEP: 29278/36938 -- GLOBAL_STEP: 731100 | > loss_text_ce: 0.038963835686445236 (0.03477478657801246) | > loss_mel_ce: 2.840919256210327 (nan) | > loss: 0.01142810843884945 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13648182137822437) | > loader_time: 0.0049 (0.004636098260525769)  --> TIME: 2025-02-25 09:56:05 -- STEP: 29328/36938 -- GLOBAL_STEP: 731150 | > loss_text_ce: 0.03266729786992073 (0.03477207758685339) | > loss_mel_ce: 2.3728761672973633 (nan) | > loss: 0.009545808658003807 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.1364776061911827) | > loader_time: 0.0046 (0.004636087186077674)  --> TIME: 2025-02-25 09:56:14 -- STEP: 29378/36938 -- GLOBAL_STEP: 731200 | > loss_text_ce: 0.033580031245946884 (0.034771069529403696) | > loss_mel_ce: 2.579224109649658 (nan) | > loss: 0.010368270799517632 (nan) | > current_lr: 5e-06 | > step_time: 0.1084 (0.13646345272780994) | > loader_time: 0.0044 (0.004636005560286525)  --> TIME: 2025-02-25 09:56:24 -- STEP: 29428/36938 -- GLOBAL_STEP: 731250 | > loss_text_ce: 0.027861613780260086 (0.03477086334465762) | > loss_mel_ce: 2.990255355834961 (nan) | > loss: 0.011976655572652817 (nan) | > current_lr: 5e-06 | > step_time: 0.1776 (0.13646242961913385) | > loader_time: 0.0048 (0.004636310827834164)  --> TIME: 2025-02-25 09:56:33 -- STEP: 29478/36938 -- GLOBAL_STEP: 731300 | > loss_text_ce: 0.029457487165927887 (0.0347703997885458) | > loss_mel_ce: 1.8213341236114502 (nan) | > loss: 0.0073444112204015255 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13646312948563813) | > loader_time: 0.0046 (0.004636364234206355)  --> TIME: 2025-02-25 09:56:42 -- STEP: 29528/36938 -- GLOBAL_STEP: 731350 | > loss_text_ce: 0.03551487252116203 (0.034766514031868964) | > loss_mel_ce: 1.9760103225708008 (nan) | > loss: 0.007982242852449417 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13646682037534683) | > loader_time: 0.0045 (0.004636345767806904)  --> TIME: 2025-02-25 09:56:51 -- STEP: 29578/36938 -- GLOBAL_STEP: 731400 | > loss_text_ce: 0.038161277770996094 (0.03476638470229345) | > loss_mel_ce: 3.1746232509613037 (nan) | > loss: 0.012749145738780499 (nan) | > current_lr: 5e-06 | > step_time: 0.1522 (0.13646275703431002) | > loader_time: 0.0047 (0.004636287068537178)  --> TIME: 2025-02-25 09:57:01 -- STEP: 29628/36938 -- GLOBAL_STEP: 731450 | > loss_text_ce: 0.024117538705468178 (0.03476410944754007) | > loss_mel_ce: 2.4725732803344727 (nan) | > loss: 0.009907503612339497 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.1364617020121086) | > loader_time: 0.0047 (0.00463632876145239)  --> TIME: 2025-02-25 09:57:10 -- STEP: 29678/36938 -- GLOBAL_STEP: 731500 | > loss_text_ce: 0.032748833298683167 (0.03476572092208939) | > loss_mel_ce: 2.3368473052978516 (nan) | > loss: 0.009403159841895103 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13646027999904722) | > loader_time: 0.0046 (0.004636391144780841)  --> TIME: 2025-02-25 09:57:20 -- STEP: 29728/36938 -- GLOBAL_STEP: 731550 | > loss_text_ce: 0.03153837099671364 (0.034765250792916806) | > loss_mel_ce: 3.548076868057251 (nan) | > loss: 0.014204823412001133 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.1364690619084761) | > loader_time: 0.0047 (0.004636441432621298)  --> TIME: 2025-02-25 09:57:29 -- STEP: 29778/36938 -- GLOBAL_STEP: 731600 | > loss_text_ce: 0.025924457237124443 (0.03476406096737454) | > loss_mel_ce: 3.3566014766693115 (nan) | > loss: 0.013422722928225994 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13646824520504935) | > loader_time: 0.0047 (0.004636485122338877)  --> TIME: 2025-02-25 09:57:38 -- STEP: 29828/36938 -- GLOBAL_STEP: 731650 | > loss_text_ce: 0.033480219542980194 (0.03476105409513131) | > loss_mel_ce: 3.227412700653076 (nan) | > loss: 0.012940051965415478 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.1364683077484673) | > loader_time: 0.0046 (0.004636612273547862)  --> TIME: 2025-02-25 09:57:47 -- STEP: 29878/36938 -- GLOBAL_STEP: 731700 | > loss_text_ce: 0.032405927777290344 (0.03476111751211367) | > loss_mel_ce: 2.641228675842285 (nan) | > loss: 0.010609661228954792 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1364634236112262) | > loader_time: 0.0048 (0.004636583210785339)  --> TIME: 2025-02-25 09:57:57 -- STEP: 29928/36938 -- GLOBAL_STEP: 731750 | > loss_text_ce: 0.03156060352921486 (0.034760916520451025) | > loss_mel_ce: 2.988075017929077 (nan) | > loss: 0.011982682161033154 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.13647024610309647) | > loader_time: 0.0048 (0.004636559017008554)  --> TIME: 2025-02-25 09:58:07 -- STEP: 29978/36938 -- GLOBAL_STEP: 731800 | > loss_text_ce: 0.029553160071372986 (0.03476107754029513) | > loss_mel_ce: 2.950322151184082 (nan) | > loss: 0.011824903078377247 (nan) | > current_lr: 5e-06 | > step_time: 0.185 (0.136477743523937) | > loader_time: 0.0047 (0.004636653150899116)  --> TIME: 2025-02-25 09:58:16 -- STEP: 30028/36938 -- GLOBAL_STEP: 731850 | > loss_text_ce: 0.03695579618215561 (0.03475706919976144) | > loss_mel_ce: 1.9793701171875 (nan) | > loss: 0.008001293987035751 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.1364751566864929) | > loader_time: 0.0045 (0.004636600099487118)  --> TIME: 2025-02-25 09:58:25 -- STEP: 30078/36938 -- GLOBAL_STEP: 731900 | > loss_text_ce: 0.037697792053222656 (0.03475760002960209) | > loss_mel_ce: 1.6228435039520264 (nan) | > loss: 0.006589449942111969 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13647892124134983) | > loader_time: 0.0045 (0.004636604494693087)  --> TIME: 2025-02-25 09:58:35 -- STEP: 30128/36938 -- GLOBAL_STEP: 731950 | > loss_text_ce: 0.04109920933842659 (0.03475857274821993) | > loss_mel_ce: 1.7269376516342163 (nan) | > loss: 0.007016019430011511 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.1364774600312364) | > loader_time: 0.0045 (0.0046365777593436385)  --> TIME: 2025-02-25 09:58:44 -- STEP: 30178/36938 -- GLOBAL_STEP: 732000 | > loss_text_ce: 0.0383233018219471 (0.03475802032237136) | > loss_mel_ce: 2.3776912689208984 (nan) | > loss: 0.009587360545992851 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13647349578848117) | > loader_time: 0.0048 (0.004636563105409062)  --> TIME: 2025-02-25 09:58:53 -- STEP: 30228/36938 -- GLOBAL_STEP: 732050 | > loss_text_ce: 0.02797596901655197 (0.0347564015371745) | > loss_mel_ce: 1.8572067022323608 (nan) | > loss: 0.0074808839708566666 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13647351605529096) | > loader_time: 0.0045 (0.004636581035238951)  --> TIME: 2025-02-25 09:59:02 -- STEP: 30278/36938 -- GLOBAL_STEP: 732100 | > loss_text_ce: 0.030477000400424004 (0.03475628253270609) | > loss_mel_ce: 1.9035683870315552 (nan) | > loss: 0.007674783933907747 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1364679421905456) | > loader_time: 0.0045 (0.004636535116007069)  --> TIME: 2025-02-25 09:59:12 -- STEP: 30328/36938 -- GLOBAL_STEP: 732150 | > loss_text_ce: 0.03587502986192703 (0.03475830159742141) | > loss_mel_ce: 3.5108261108398438 (nan) | > loss: 0.014074211940169334 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13646872543276142) | > loader_time: 0.0046 (0.004636519960223823)  --> TIME: 2025-02-25 09:59:21 -- STEP: 30378/36938 -- GLOBAL_STEP: 732200 | > loss_text_ce: 0.031049184501171112 (0.03475713760844255) | > loss_mel_ce: 2.6449618339538574 (nan) | > loss: 0.010619091801345348 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13647247676954896) | > loader_time: 0.0046 (0.00463653025958966)  --> TIME: 2025-02-25 09:59:30 -- STEP: 30428/36938 -- GLOBAL_STEP: 732250 | > loss_text_ce: 0.02315702848136425 (0.03475645708268037) | > loss_mel_ce: 2.933809518814087 (nan) | > loss: 0.01173399481922388 (nan) | > current_lr: 5e-06 | > step_time: 0.1491 (0.13646975446032636) | > loader_time: 0.0046 (0.00463650886185366)  --> TIME: 2025-02-25 09:59:40 -- STEP: 30478/36938 -- GLOBAL_STEP: 732300 | > loss_text_ce: 0.047551315277814865 (0.03475604476299629) | > loss_mel_ce: 2.6402180194854736 (nan) | > loss: 0.010665751993656158 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.1364789107503119) | > loader_time: 0.0045 (0.004636540024273817)  --> TIME: 2025-02-25 09:59:49 -- STEP: 30528/36938 -- GLOBAL_STEP: 732350 | > loss_text_ce: 0.048019688576459885 (0.03475655718282275) | > loss_mel_ce: 2.15283203125 (nan) | > loss: 0.008733538910746574 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13647477654447393) | > loader_time: 0.0045 (0.004636493228391527)  --> TIME: 2025-02-25 09:59:59 -- STEP: 30578/36938 -- GLOBAL_STEP: 732400 | > loss_text_ce: 0.03808804228901863 (0.03475718808910659) | > loss_mel_ce: 2.178579807281494 (nan) | > loss: 0.008796301670372486 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13647984033071034) | > loader_time: 0.0046 (0.004636526864103151)  --> TIME: 2025-02-25 10:00:08 -- STEP: 30628/36938 -- GLOBAL_STEP: 732450 | > loss_text_ce: 0.030817095190286636 (0.03475708553577401) | > loss_mel_ce: 0.1538955718278885 (nan) | > loss: 0.0007329867803491652 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13648058768117136) | > loader_time: 0.0048 (0.004636525267080019)  --> TIME: 2025-02-25 10:00:17 -- STEP: 30678/36938 -- GLOBAL_STEP: 732500 | > loss_text_ce: 0.03770081326365471 (0.03475946156021253) | > loss_mel_ce: 2.9208288192749023 (nan) | > loss: 0.011740198358893394 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.1364765557373) | > loader_time: 0.0046 (0.004636455626723676)  --> TIME: 2025-02-25 10:00:26 -- STEP: 30728/36938 -- GLOBAL_STEP: 732550 | > loss_text_ce: 0.031038880348205566 (0.03475877162588834) | > loss_mel_ce: 2.5426700115203857 (nan) | > loss: 0.01021313201636076 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13647525025295532) | > loader_time: 0.0046 (0.0046364188473598895)  --> TIME: 2025-02-25 10:00:36 -- STEP: 30778/36938 -- GLOBAL_STEP: 732600 | > loss_text_ce: 0.027368448674678802 (0.03476238050158276) | > loss_mel_ce: 3.392439842224121 (nan) | > loss: 0.01357066910713911 (nan) | > current_lr: 5e-06 | > step_time: 0.1819 (0.13647786672681325) | > loader_time: 0.0048 (0.004636382838192251)  --> TIME: 2025-02-25 10:00:45 -- STEP: 30828/36938 -- GLOBAL_STEP: 732650 | > loss_text_ce: 0.03280866518616676 (0.03476232269824667) | > loss_mel_ce: 2.14267897605896 (nan) | > loss: 0.008632888086140156 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13648133499390067) | > loader_time: 0.0047 (0.00463635812895267)  --> TIME: 2025-02-25 10:00:54 -- STEP: 30878/36938 -- GLOBAL_STEP: 732700 | > loss_text_ce: 0.031589213758707047 (0.03476021962230224) | > loss_mel_ce: 3.1355669498443604 (nan) | > loss: 0.012568080797791481 (nan) | > current_lr: 5e-06 | > step_time: 0.1706 (0.13647993900853325) | > loader_time: 0.0046 (0.004636293866256176)  --> TIME: 2025-02-25 10:01:04 -- STEP: 30928/36938 -- GLOBAL_STEP: 732750 | > loss_text_ce: 0.03621656447649002 (0.034758509020230935) | > loss_mel_ce: 3.1782286167144775 (nan) | > loss: 0.012755735777318478 (nan) | > current_lr: 5e-06 | > step_time: 0.1483 (0.1364821758963446) | > loader_time: 0.0046 (0.004636295436579389)  --> TIME: 2025-02-25 10:01:13 -- STEP: 30978/36938 -- GLOBAL_STEP: 732800 | > loss_text_ce: 0.03430888056755066 (0.03476003235346968) | > loss_mel_ce: 2.3783977031707764 (nan) | > loss: 0.009574233554303646 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13648569262953664) | > loader_time: 0.0045 (0.004636314180162214)  --> TIME: 2025-02-25 10:01:23 -- STEP: 31028/36938 -- GLOBAL_STEP: 732850 | > loss_text_ce: 0.03693890571594238 (0.034760393541690894) | > loss_mel_ce: 2.1759791374206543 (nan) | > loss: 0.00878142099827528 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13649568170312956) | > loader_time: 0.0045 (0.0046362891337975096)  --> TIME: 2025-02-25 10:01:32 -- STEP: 31078/36938 -- GLOBAL_STEP: 732900 | > loss_text_ce: 0.035820987075567245 (0.03476072138970599) | > loss_mel_ce: 2.6606640815734863 (nan) | > loss: 0.010700338520109653 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.1365009198448701) | > loader_time: 0.0048 (0.004636274340591707)  --> TIME: 2025-02-25 10:01:42 -- STEP: 31128/36938 -- GLOBAL_STEP: 732950 | > loss_text_ce: 0.03305177390575409 (0.03476006040841852) | > loss_mel_ce: 3.1040027141571045 (nan) | > loss: 0.0124486293643713 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13650898231359582) | > loader_time: 0.0046 (0.0046362474779027526)  --> TIME: 2025-02-25 10:01:52 -- STEP: 31178/36938 -- GLOBAL_STEP: 733000 | > loss_text_ce: 0.03847246617078781 (0.034760308232636795) | > loss_mel_ce: 3.433018684387207 (nan) | > loss: 0.013775759376585484 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.1365142446781985) | > loader_time: 0.0048 (0.00463622521311073)  --> TIME: 2025-02-25 10:02:01 -- STEP: 31228/36938 -- GLOBAL_STEP: 733050 | > loss_text_ce: 0.03376389667391777 (0.034759298579766826) | > loss_mel_ce: 2.4424543380737305 (nan) | > loss: 0.009826263412833214 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13650492076090803) | > loader_time: 0.0046 (0.0046361763360971515)  --> TIME: 2025-02-25 10:02:10 -- STEP: 31278/36938 -- GLOBAL_STEP: 733100 | > loss_text_ce: 0.035810936242341995 (0.034759402591064775) | > loss_mel_ce: 2.6089940071105957 (nan) | > loss: 0.010495258495211601 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13650671032751283) | > loader_time: 0.0046 (0.004636329163583586)  --> TIME: 2025-02-25 10:02:20 -- STEP: 31328/36938 -- GLOBAL_STEP: 733150 | > loss_text_ce: 0.0329710878431797 (0.03476063315616322) | > loss_mel_ce: 3.532724142074585 (nan) | > loss: 0.014149585738778114 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13652075228183383) | > loader_time: 0.0046 (0.0046363654218241155)  --> TIME: 2025-02-25 10:02:30 -- STEP: 31378/36938 -- GLOBAL_STEP: 733200 | > loss_text_ce: 0.03423940762877464 (0.034760334333505415) | > loss_mel_ce: 2.030548095703125 (nan) | > loss: 0.008193600922822952 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.13652975927446928) | > loader_time: 0.0047 (0.004636458961858338)  --> TIME: 2025-02-25 10:02:39 -- STEP: 31428/36938 -- GLOBAL_STEP: 733250 | > loss_text_ce: 0.03842100128531456 (0.034761746520568126) | > loss_mel_ce: 2.3375000953674316 (nan) | > loss: 0.00942825898528099 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13653412260626313) | > loader_time: 0.0046 (0.004636450739053726)  --> TIME: 2025-02-25 10:02:49 -- STEP: 31478/36938 -- GLOBAL_STEP: 733300 | > loss_text_ce: 0.02533169649541378 (0.03476185981943807) | > loss_mel_ce: 3.0864107608795166 (nan) | > loss: 0.012348185293376446 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.1365429596371058) | > loader_time: 0.0046 (0.004636433029259632)  --> TIME: 2025-02-25 10:02:58 -- STEP: 31528/36938 -- GLOBAL_STEP: 733350 | > loss_text_ce: 0.03317936509847641 (0.034760333229715765) | > loss_mel_ce: 2.4591972827911377 (nan) | > loss: 0.009890384040772915 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1365455437420412) | > loader_time: 0.0046 (0.004636382344288739)  --> TIME: 2025-02-25 10:03:07 -- STEP: 31578/36938 -- GLOBAL_STEP: 733400 | > loss_text_ce: 0.026003416627645493 (0.034760863995038756) | > loss_mel_ce: 1.2240374088287354 (nan) | > loss: 0.0049604796804487705 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13654118939782164) | > loader_time: 0.0044 (0.0046363128161021104)  --> TIME: 2025-02-25 10:03:17 -- STEP: 31628/36938 -- GLOBAL_STEP: 733450 | > loss_text_ce: 0.02731364406645298 (0.034759649330394826) | > loss_mel_ce: 0.9209277033805847 (nan) | > loss: 0.0037628626450896263 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13654042382570214) | > loader_time: 0.0045 (0.004636273283685247)  --> TIME: 2025-02-25 10:03:26 -- STEP: 31678/36938 -- GLOBAL_STEP: 733500 | > loss_text_ce: 0.032209429889917374 (0.03475949671421699) | > loss_mel_ce: 3.3866477012634277 (nan) | > loss: 0.013566894456744194 (nan) | > current_lr: 5e-06 | > step_time: 0.1865 (0.13654526165544947) | > loader_time: 0.0048 (0.004636316951518329)  --> TIME: 2025-02-25 10:03:36 -- STEP: 31728/36938 -- GLOBAL_STEP: 733550 | > loss_text_ce: 0.043457888066768646 (0.03475882334738475) | > loss_mel_ce: 3.431939125061035 (nan) | > loss: 0.013791259378194809 (nan) | > current_lr: 5e-06 | > step_time: 0.1489 (0.1365580937655603) | > loader_time: 0.0046 (0.0046363935378002956)  --> TIME: 2025-02-25 10:03:45 -- STEP: 31778/36938 -- GLOBAL_STEP: 733600 | > loss_text_ce: 0.032389622181653976 (0.03475822564745533) | > loss_mel_ce: 2.3476788997650146 (nan) | > loss: 0.009444717317819595 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13655612284004776) | > loader_time: 0.0065 (0.004636380946901039)  --> TIME: 2025-02-25 10:03:54 -- STEP: 31828/36938 -- GLOBAL_STEP: 733650 | > loss_text_ce: 0.03354382514953613 (0.03475778972960872) | > loss_mel_ce: 2.798367977142334 (nan) | > loss: 0.011237746104598045 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13655390264671277) | > loader_time: 0.0046 (0.004636368680213041)  --> TIME: 2025-02-25 10:04:04 -- STEP: 31878/36938 -- GLOBAL_STEP: 733700 | > loss_text_ce: 0.03294478729367256 (0.03475835270545374) | > loss_mel_ce: 2.6548469066619873 (nan) | > loss: 0.010665840469300747 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13655572329764804) | > loader_time: 0.0046 (0.004636388320427433)  --> TIME: 2025-02-25 10:04:13 -- STEP: 31928/36938 -- GLOBAL_STEP: 733750 | > loss_text_ce: 0.03513677418231964 (0.034759199196826254) | > loss_mel_ce: 3.0233218669891357 (nan) | > loss: 0.012136740610003471 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13655907979766682) | > loader_time: 0.0047 (0.004636472596527188)  --> TIME: 2025-02-25 10:04:22 -- STEP: 31978/36938 -- GLOBAL_STEP: 733800 | > loss_text_ce: 0.026820702478289604 (0.03475875800646862) | > loss_mel_ce: 2.471552848815918 (nan) | > loss: 0.00991418119519949 (nan) | > current_lr: 5e-06 | > step_time: 0.161 (0.13655737507476795) | > loader_time: 0.0049 (0.004636455084728963)  --> TIME: 2025-02-25 10:04:32 -- STEP: 32028/36938 -- GLOBAL_STEP: 733850 | > loss_text_ce: 0.029219049960374832 (0.03475569710813587) | > loss_mel_ce: 2.5478920936584473 (nan) | > loss: 0.010226632468402386 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.136553238650692) | > loader_time: 0.0047 (0.004636436965085299)  --> TIME: 2025-02-25 10:04:41 -- STEP: 32078/36938 -- GLOBAL_STEP: 733900 | > loss_text_ce: 0.038919847458601 (0.03475375818243706) | > loss_mel_ce: 3.1547129154205322 (nan) | > loss: 0.012673147022724152 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.13655013317582926) | > loader_time: 0.0049 (0.004636432295227903)  --> TIME: 2025-02-25 10:04:50 -- STEP: 32128/36938 -- GLOBAL_STEP: 733950 | > loss_text_ce: 0.033954914659261703 (0.034751858185678705) | > loss_mel_ce: 1.9200439453125 (nan) | > loss: 0.007753964047878981 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13654695376308598) | > loader_time: 0.0045 (0.0046365387084712035)  --> TIME: 2025-02-25 10:04:59 -- STEP: 32178/36938 -- GLOBAL_STEP: 734000 | > loss_text_ce: 0.02830168418586254 (0.03475090438529356) | > loss_mel_ce: 1.2004643678665161 (nan) | > loss: 0.004876056220382452 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13654325349075028) | > loader_time: 0.0045 (0.004636566614791997)  --> TIME: 2025-02-25 10:05:08 -- STEP: 32228/36938 -- GLOBAL_STEP: 734050 | > loss_text_ce: 0.027808930724859238 (0.034751062041577116) | > loss_mel_ce: 3.3578693866729736 (nan) | > loss: 0.01343523245304823 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.1365325676236365) | > loader_time: 0.0045 (0.004636458091764244)  --> TIME: 2025-02-25 10:05:17 -- STEP: 32278/36938 -- GLOBAL_STEP: 734100 | > loss_text_ce: 0.026107782498002052 (0.034751911821371786) | > loss_mel_ce: 0.07666015625 (nan) | > loss: 0.0004078093043062836 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.1365347560050804) | > loader_time: 0.0046 (0.004636485039342088)  --> TIME: 2025-02-25 10:05:27 -- STEP: 32328/36938 -- GLOBAL_STEP: 734150 | > loss_text_ce: 0.03699339181184769 (0.03475248891510184) | > loss_mel_ce: 2.6056530475616455 (nan) | > loss: 0.010486693121492863 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.1365334168205582) | > loader_time: 0.0046 (0.00463644964392307)  --> TIME: 2025-02-25 10:05:37 -- STEP: 32378/36938 -- GLOBAL_STEP: 734200 | > loss_text_ce: 0.04205487295985222 (0.03475121508909177) | > loss_mel_ce: 2.678521156311035 (nan) | > loss: 0.010795936919748783 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13654579788739174) | > loader_time: 0.0045 (0.004636461286034057)  --> TIME: 2025-02-25 10:05:46 -- STEP: 32428/36938 -- GLOBAL_STEP: 734250 | > loss_text_ce: 0.04120923578739166 (0.034752339399067814) | > loss_mel_ce: 3.3701958656311035 (nan) | > loss: 0.013537322171032429 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13654327921939763) | > loader_time: 0.0047 (0.004636639854360097)  --> TIME: 2025-02-25 10:05:55 -- STEP: 32478/36938 -- GLOBAL_STEP: 734300 | > loss_text_ce: 0.03913295641541481 (0.03475136844006249) | > loss_mel_ce: 2.2988526821136475 (nan) | > loss: 0.009277721866965294 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.13654396249725473) | > loader_time: 0.0044 (0.004636580526068977)  --> TIME: 2025-02-25 10:06:05 -- STEP: 32528/36938 -- GLOBAL_STEP: 734350 | > loss_text_ce: 0.027487603947520256 (0.03474966563318545) | > loss_mel_ce: 1.006830096244812 (nan) | > loss: 0.004104435909539461 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13654895457386487) | > loader_time: 0.0045 (0.004636569623869739)  --> TIME: 2025-02-25 10:06:14 -- STEP: 32578/36938 -- GLOBAL_STEP: 734400 | > loss_text_ce: 0.034652743488550186 (0.03474804495923678) | > loss_mel_ce: 3.092740535736084 (nan) | > loss: 0.012410291470587254 (nan) | > current_lr: 5e-06 | > step_time: 0.1531 (0.13655318738318112) | > loader_time: 0.0046 (0.004636573296780594)  --> TIME: 2025-02-25 10:06:23 -- STEP: 32628/36938 -- GLOBAL_STEP: 734450 | > loss_text_ce: 0.03550035133957863 (0.03474752987623444) | > loss_mel_ce: 3.460925579071045 (nan) | > loss: 0.01387470681220293 (nan) | > current_lr: 5e-06 | > step_time: 0.178 (0.136552886371302) | > loader_time: 0.0048 (0.004636537740814254)  --> TIME: 2025-02-25 10:06:32 -- STEP: 32678/36938 -- GLOBAL_STEP: 734500 | > loss_text_ce: 0.05084017291665077 (0.03474590580933719) | > loss_mel_ce: 2.2421727180480957 (nan) | > loss: 0.009099258109927177 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13654984543727613) | > loader_time: 0.0045 (0.004636515462928944)  --> TIME: 2025-02-25 10:06:42 -- STEP: 32728/36938 -- GLOBAL_STEP: 734550 | > loss_text_ce: 0.03580847755074501 (0.034746326766002304) | > loss_mel_ce: 3.262368679046631 (nan) | > loss: 0.013088005594909191 (nan) | > current_lr: 5e-06 | > step_time: 0.1779 (0.1365509963187113) | > loader_time: 0.0048 (0.004636530413134583)  --> TIME: 2025-02-25 10:06:51 -- STEP: 32778/36938 -- GLOBAL_STEP: 734600 | > loss_text_ce: 0.032800767570734024 (0.03474503363787772) | > loss_mel_ce: 3.119967222213745 (nan) | > loss: 0.01251098420470953 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.13655330449755557) | > loader_time: 0.0046 (0.0046365469834157)  --> TIME: 2025-02-25 10:07:01 -- STEP: 32828/36938 -- GLOBAL_STEP: 734650 | > loss_text_ce: 0.04244336113333702 (0.034745676429363465) | > loss_mel_ce: 2.8199100494384766 (nan) | > loss: 0.011358545161783695 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13655138200944927) | > loader_time: 0.0047 (0.004636549014215476)  --> TIME: 2025-02-25 10:07:10 -- STEP: 32878/36938 -- GLOBAL_STEP: 734700 | > loss_text_ce: 0.03737661615014076 (0.034743917236814435) | > loss_mel_ce: 3.3504157066345215 (nan) | > loss: 0.01344362087547779 (nan) | > current_lr: 5e-06 | > step_time: 0.178 (0.13655190590676866) | > loader_time: 0.005 (0.004636538711093955)  --> TIME: 2025-02-25 10:07:20 -- STEP: 32928/36938 -- GLOBAL_STEP: 734750 | > loss_text_ce: 0.03901505842804909 (0.03474422620731154) | > loss_mel_ce: 2.2589519023895264 (nan) | > loss: 0.009118917398154736 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.1365597910664987) | > loader_time: 0.0047 (0.0046365636865182075)  --> TIME: 2025-02-25 10:07:29 -- STEP: 32978/36938 -- GLOBAL_STEP: 734800 | > loss_text_ce: 0.03618241846561432 (0.0347460081722424) | > loss_mel_ce: 3.0416557788848877 (nan) | > loss: 0.012213644571602345 (nan) | > current_lr: 5e-06 | > step_time: 0.1809 (0.136556376421298) | > loader_time: 0.0048 (0.004636558576034755)  --> TIME: 2025-02-25 10:07:38 -- STEP: 33028/36938 -- GLOBAL_STEP: 734850 | > loss_text_ce: 0.030853433534502983 (0.034745293391991144) | > loss_mel_ce: 0.4411708414554596 (nan) | > loss: 0.0018731122836470604 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13656457640348904) | > loader_time: 0.0045 (0.00463657103685543)  --> TIME: 2025-02-25 10:07:48 -- STEP: 33078/36938 -- GLOBAL_STEP: 734900 | > loss_text_ce: 0.028175339102745056 (0.03474636805357612) | > loss_mel_ce: 2.9133963584899902 (nan) | > loss: 0.011672904714941978 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13656510612798745) | > loader_time: 0.0045 (0.00463654696707123)  --> TIME: 2025-02-25 10:07:57 -- STEP: 33128/36938 -- GLOBAL_STEP: 734950 | > loss_text_ce: 0.033347468823194504 (0.03474753581736848) | > loss_mel_ce: 2.6638314723968506 (nan) | > loss: 0.010703091509640217 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13656728586060138) | > loader_time: 0.0045 (0.004636527957388869)  --> TIME: 2025-02-25 10:08:07 -- STEP: 33178/36938 -- GLOBAL_STEP: 735000 | > loss_text_ce: 0.028883907943964005 (0.03474820321533043) | > loss_mel_ce: 2.604408025741577 (nan) | > loss: 0.010449571534991264 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13656741020311441) | > loader_time: 0.0045 (0.004636510003862718)  --> TIME: 2025-02-25 10:08:16 -- STEP: 33228/36938 -- GLOBAL_STEP: 735050 | > loss_text_ce: 0.04472266882658005 (0.034748434267440106) | > loss_mel_ce: 2.895966053009033 (nan) | > loss: 0.0116694001480937 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.13656429103153578) | > loader_time: 0.0048 (0.004636548616488053)  --> TIME: 2025-02-25 10:08:25 -- STEP: 33278/36938 -- GLOBAL_STEP: 735100 | > loss_text_ce: 0.034825604408979416 (0.034748526913030625) | > loss_mel_ce: 3.145573139190674 (nan) | > loss: 0.01262063067406416 (nan) | > current_lr: 5e-06 | > step_time: 0.1857 (0.1365608355697946) | > loader_time: 0.0049 (0.004636579683547879)  --> TIME: 2025-02-25 10:08:34 -- STEP: 33328/36938 -- GLOBAL_STEP: 735150 | > loss_text_ce: 0.039160460233688354 (0.034749560717599) | > loss_mel_ce: 2.947970151901245 (nan) | > loss: 0.01185369398444891 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13655679960177633) | > loader_time: 0.0048 (0.0046365713764141185)  --> TIME: 2025-02-25 10:08:44 -- STEP: 33378/36938 -- GLOBAL_STEP: 735200 | > loss_text_ce: 0.03656230494379997 (0.03474789657213464) | > loss_mel_ce: 2.361499071121216 (nan) | > loss: 0.009516117163002491 (nan) | > current_lr: 5e-06 | > step_time: 0.1084 (0.13656062284533985) | > loader_time: 0.0045 (0.004636638559813092)  --> TIME: 2025-02-25 10:08:53 -- STEP: 33428/36938 -- GLOBAL_STEP: 735250 | > loss_text_ce: 0.031303536146879196 (0.03474781326361199) | > loss_mel_ce: 2.752551555633545 (nan) | > loss: 0.011047044768929482 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13655397806268058) | > loader_time: 0.0046 (0.004636660965347422)  --> TIME: 2025-02-25 10:09:02 -- STEP: 33478/36938 -- GLOBAL_STEP: 735300 | > loss_text_ce: 0.028271226212382317 (0.03474841508233229) | > loss_mel_ce: 3.1458563804626465 (nan) | > loss: 0.01259574480354786 (nan) | > current_lr: 5e-06 | > step_time: 0.184 (0.1365516660560868) | > loader_time: 0.0049 (0.004636831590939719)  --> TIME: 2025-02-25 10:09:12 -- STEP: 33528/36938 -- GLOBAL_STEP: 735350 | > loss_text_ce: 0.03267230466008186 (0.03474838389120426) | > loss_mel_ce: 2.8833930492401123 (nan) | > loss: 0.011571688577532768 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.13655913752919052) | > loader_time: 0.0046 (0.004636917200159038)  --> TIME: 2025-02-25 10:09:21 -- STEP: 33578/36938 -- GLOBAL_STEP: 735400 | > loss_text_ce: 0.034252993762493134 (0.03474819320175552) | > loss_mel_ce: 3.443953275680542 (nan) | > loss: 0.013802406378090382 (nan) | > current_lr: 5e-06 | > step_time: 0.1694 (0.13655391290577115) | > loader_time: 0.0046 (0.004636924193964665)  --> TIME: 2025-02-25 10:09:30 -- STEP: 33628/36938 -- GLOBAL_STEP: 735450 | > loss_text_ce: 0.03337410092353821 (0.034746540937594134) | > loss_mel_ce: 3.4229331016540527 (nan) | > loss: 0.013715505599975586 (nan) | > current_lr: 5e-06 | > step_time: 0.1977 (0.1365567500276207) | > loader_time: 0.0049 (0.004637051489373877)  --> TIME: 2025-02-25 10:09:40 -- STEP: 33678/36938 -- GLOBAL_STEP: 735500 | > loss_text_ce: 0.04382622241973877 (0.03474560123410632) | > loss_mel_ce: 2.6666460037231445 (nan) | > loss: 0.010755842551589012 (nan) | > current_lr: 5e-06 | > step_time: 0.1858 (0.13656091115927674) | > loader_time: 0.0048 (0.004637143540972798)  --> TIME: 2025-02-25 10:09:49 -- STEP: 33728/36938 -- GLOBAL_STEP: 735550 | > loss_text_ce: 0.03298170119524002 (0.03474267630165116) | > loss_mel_ce: 2.006678342819214 (nan) | > loss: 0.008093888871371746 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13655922419092947) | > loader_time: 0.0045 (0.004637177736698105)  --> TIME: 2025-02-25 10:09:58 -- STEP: 33778/36938 -- GLOBAL_STEP: 735600 | > loss_text_ce: 0.03070938028395176 (0.03474285972060757) | > loss_mel_ce: 2.464625835418701 (nan) | > loss: 0.009902124293148518 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13655812636991016) | > loader_time: 0.0048 (0.004637173595842929)  --> TIME: 2025-02-25 10:10:07 -- STEP: 33828/36938 -- GLOBAL_STEP: 735650 | > loss_text_ce: 0.03650778532028198 (0.03474341631848065) | > loss_mel_ce: 2.983663320541382 (nan) | > loss: 0.011984806507825851 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.1365523905044786) | > loader_time: 0.0048 (0.004637166295644161)  --> TIME: 2025-02-25 10:10:16 -- STEP: 33878/36938 -- GLOBAL_STEP: 735700 | > loss_text_ce: 0.028735319152474403 (0.03474338750107381) | > loss_mel_ce: 2.9257731437683105 (nan) | > loss: 0.011724241077899933 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.1365494667143082) | > loader_time: 0.0046 (0.004637172177237789)  --> TIME: 2025-02-25 10:10:26 -- STEP: 33928/36938 -- GLOBAL_STEP: 735750 | > loss_text_ce: 0.021424805745482445 (0.03474234509677877) | > loss_mel_ce: 0.8517493605613708 (nan) | > loss: 0.0034649772569537163 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.1365459413636384) | > loader_time: 0.0047 (0.004637145716411493)  --> TIME: 2025-02-25 10:10:35 -- STEP: 33978/36938 -- GLOBAL_STEP: 735800 | > loss_text_ce: 0.03253365680575371 (0.03474383022430836) | > loss_mel_ce: 2.9040396213531494 (nan) | > loss: 0.011653069406747818 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13654397345675487) | > loader_time: 0.0046 (0.004637173370234682)  --> TIME: 2025-02-25 10:10:44 -- STEP: 34028/36938 -- GLOBAL_STEP: 735850 | > loss_text_ce: 0.035310786217451096 (0.03474478650467789) | > loss_mel_ce: 3.5976901054382324 (nan) | > loss: 0.014416670426726341 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.1365437430837709) | > loader_time: 0.0046 (0.004637278617304188)  --> TIME: 2025-02-25 10:10:54 -- STEP: 34078/36938 -- GLOBAL_STEP: 735900 | > loss_text_ce: 0.025118250399827957 (0.034744727807530094) | > loss_mel_ce: 3.014366626739502 (nan) | > loss: 0.012061448767781258 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.13654797697327564) | > loader_time: 0.0047 (0.004637281766932109)  --> TIME: 2025-02-25 10:11:03 -- STEP: 34128/36938 -- GLOBAL_STEP: 735950 | > loss_text_ce: 0.041868433356285095 (0.03474396235658091) | > loss_mel_ce: 3.072237491607666 (nan) | > loss: 0.012357563711702824 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.1365482271891275) | > loader_time: 0.0046 (0.004637332293442919)  --> TIME: 2025-02-25 10:11:12 -- STEP: 34178/36938 -- GLOBAL_STEP: 736000 | > loss_text_ce: 0.02458694390952587 (0.03474176663999024) | > loss_mel_ce: 2.25732421875 (nan) | > loss: 0.009055203758180141 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.1365361006572857) | > loader_time: 0.0045 (0.004637256103371417)  --> TIME: 2025-02-25 10:11:21 -- STEP: 34228/36938 -- GLOBAL_STEP: 736050 | > loss_text_ce: 0.03889719024300575 (0.03474374105902989) | > loss_mel_ce: 3.3365654945373535 (nan) | > loss: 0.013394693844020367 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.1365297730453557) | > loader_time: 0.0047 (0.004637270660841727)  --> TIME: 2025-02-25 10:11:30 -- STEP: 34278/36938 -- GLOBAL_STEP: 736100 | > loss_text_ce: 0.0299180056899786 (0.03474523877513965) | > loss_mel_ce: 2.8500564098358154 (nan) | > loss: 0.011428470723330975 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13652879771416232) | > loader_time: 0.0046 (0.004637305889144101)  --> TIME: 2025-02-25 10:11:40 -- STEP: 34328/36938 -- GLOBAL_STEP: 736150 | > loss_text_ce: 0.03279513120651245 (0.034744295668195875) | > loss_mel_ce: 0.10761718451976776 (nan) | > loss: 0.0005571917863562703 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.1365282079315613) | > loader_time: 0.0046 (0.004637464071814753)  --> TIME: 2025-02-25 10:11:49 -- STEP: 34378/36938 -- GLOBAL_STEP: 736200 | > loss_text_ce: 0.025304066017270088 (0.03474137958342809) | > loss_mel_ce: 3.104649543762207 (nan) | > loss: 0.012420451268553734 (nan) | > current_lr: 5e-06 | > step_time: 0.1776 (0.13652346182593597) | > loader_time: 0.0048 (0.00463741696992603)  --> TIME: 2025-02-25 10:11:58 -- STEP: 34428/36938 -- GLOBAL_STEP: 736250 | > loss_text_ce: 0.027185462415218353 (0.034739459475905676) | > loss_mel_ce: 0.17032471299171448 (nan) | > loss: 0.000783770636189729 (nan) | > current_lr: 5e-06 | > step_time: 0.1268 (0.13652127663322225) | > loader_time: 0.0045 (0.004637353329122773)  --> TIME: 2025-02-25 10:12:07 -- STEP: 34478/36938 -- GLOBAL_STEP: 736300 | > loss_text_ce: 0.03836699575185776 (0.03473875041465684) | > loss_mel_ce: 2.5839788913726807 (nan) | > loss: 0.010406135581433773 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13651533426669532) | > loader_time: 0.0046 (0.004637312554408609)  --> TIME: 2025-02-25 10:12:16 -- STEP: 34528/36938 -- GLOBAL_STEP: 736350 | > loss_text_ce: 0.03063815087080002 (0.034737171491943046) | > loss_mel_ce: 2.432657241821289 (nan) | > loss: 0.009774982929229736 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13651688412121912) | > loader_time: 0.0046 (0.0046372885666476835)  --> TIME: 2025-02-25 10:12:25 -- STEP: 34578/36938 -- GLOBAL_STEP: 736400 | > loss_text_ce: 0.044138792902231216 (0.034737646123158016) | > loss_mel_ce: 2.8569538593292236 (nan) | > loss: 0.0115122739225626 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1365106170825189) | > loader_time: 0.0045 (0.0046372495480026055)  --> TIME: 2025-02-25 10:12:35 -- STEP: 34628/36938 -- GLOBAL_STEP: 736450 | > loss_text_ce: 0.03603927418589592 (0.034738532156273) | > loss_mel_ce: 2.619929313659668 (nan) | > loss: 0.010539558716118336 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.13651999081063843) | > loader_time: 0.0046 (0.004637236419995873)  --> TIME: 2025-02-25 10:12:45 -- STEP: 34678/36938 -- GLOBAL_STEP: 736500 | > loss_text_ce: 0.03933218866586685 (0.03473689372900677) | > loss_mel_ce: 2.7009925842285156 (nan) | > loss: 0.010874304920434952 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13652716789754968) | > loader_time: 0.0047 (0.004637206100566381)  --> TIME: 2025-02-25 10:12:54 -- STEP: 34728/36938 -- GLOBAL_STEP: 736550 | > loss_text_ce: 0.03526097536087036 (0.03473618297676352) | > loss_mel_ce: 2.9498796463012695 (nan) | > loss: 0.01184579636901617 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.1365188594664666) | > loader_time: 0.0046 (0.004637129630561351)  --> TIME: 2025-02-25 10:13:03 -- STEP: 34778/36938 -- GLOBAL_STEP: 736600 | > loss_text_ce: 0.035355616360902786 (0.03473303363593106) | > loss_mel_ce: 3.0790858268737793 (nan) | > loss: 0.012358895502984524 (nan) | > current_lr: 5e-06 | > step_time: 0.1687 (0.13651947588785993) | > loader_time: 0.0048 (0.004637101498787593)  --> TIME: 2025-02-25 10:13:13 -- STEP: 34828/36938 -- GLOBAL_STEP: 736650 | > loss_text_ce: 0.0407230369746685 (0.034731887528719786) | > loss_mel_ce: 3.0416736602783203 (nan) | > loss: 0.012231733649969101 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.13652114626788225) | > loader_time: 0.0046 (0.004637036440470214)  --> TIME: 2025-02-25 10:13:21 -- STEP: 34878/36938 -- GLOBAL_STEP: 736700 | > loss_text_ce: 0.041286300867795944 (0.0347349647471232) | > loss_mel_ce: 2.9619269371032715 (nan) | > loss: 0.011917512863874435 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13651466277345825) | > loader_time: 0.0045 (0.004637021647659525)  --> TIME: 2025-02-25 10:13:31 -- STEP: 34928/36938 -- GLOBAL_STEP: 736750 | > loss_text_ce: 0.04696110263466835 (0.03473475114078723) | > loss_mel_ce: 2.3597300052642822 (nan) | > loss: 0.009550361894071102 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13651465906143445) | > loader_time: 0.0045 (0.004637033027134733)  --> TIME: 2025-02-25 10:13:40 -- STEP: 34978/36938 -- GLOBAL_STEP: 736800 | > loss_text_ce: 0.04057307541370392 (0.03473472371716609) | > loss_mel_ce: 2.7675869464874268 (nan) | > loss: 0.01114349253475666 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13651331901495803) | > loader_time: 0.0045 (0.004637028328638565)  --> TIME: 2025-02-25 10:13:49 -- STEP: 35028/36938 -- GLOBAL_STEP: 736850 | > loss_text_ce: 0.037812910974025726 (0.03473337853614129) | > loss_mel_ce: 3.0663397312164307 (nan) | > loss: 0.012318067252635956 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13651011740084598) | > loader_time: 0.0045 (0.004637037862364089)  --> TIME: 2025-02-25 10:13:58 -- STEP: 35078/36938 -- GLOBAL_STEP: 736900 | > loss_text_ce: 0.041606511920690536 (0.03473416018576476) | > loss_mel_ce: 2.521226644515991 (nan) | > loss: 0.01016997266560793 (nan) | > current_lr: 5e-06 | > step_time: 0.1872 (0.13650482220445584) | > loader_time: 0.0049 (0.004637047654377079)  --> TIME: 2025-02-25 10:14:08 -- STEP: 35128/36938 -- GLOBAL_STEP: 736950 | > loss_text_ce: 0.03476988896727562 (0.03473451356611399) | > loss_mel_ce: 3.217167377471924 (nan) | > loss: 0.012904513627290726 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13650335390158966) | > loader_time: 0.0048 (0.004637066961230679)  --> TIME: 2025-02-25 10:14:17 -- STEP: 35178/36938 -- GLOBAL_STEP: 737000 | > loss_text_ce: 0.0410761758685112 (0.03473333653080174) | > loss_mel_ce: 3.0395476818084717 (nan) | > loss: 0.012224698439240456 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13651122467535115) | > loader_time: 0.0046 (0.0046370766094961805)  --> TIME: 2025-02-25 10:14:26 -- STEP: 35228/36938 -- GLOBAL_STEP: 737050 | > loss_text_ce: 0.04618005082011223 (0.03473420737186896) | > loss_mel_ce: 3.0461552143096924 (nan) | > loss: 0.012271172367036343 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13650384144329897) | > loader_time: 0.0045 (0.00463701520156167)  --> TIME: 2025-02-25 10:14:36 -- STEP: 35278/36938 -- GLOBAL_STEP: 737100 | > loss_text_ce: 0.03610683232545853 (0.03473685029202458) | > loss_mel_ce: 2.8881399631500244 (nan) | > loss: 0.011604154482483864 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13650638118164143) | > loader_time: 0.0045 (0.004637019840644677)  --> TIME: 2025-02-25 10:14:45 -- STEP: 35328/36938 -- GLOBAL_STEP: 737150 | > loss_text_ce: 0.0325494222342968 (0.03473788925196246) | > loss_mel_ce: 2.706932783126831 (nan) | > loss: 0.010870961472392082 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13650508017778673) | > loader_time: 0.0045 (0.004636982671808504)  --> TIME: 2025-02-25 10:14:54 -- STEP: 35378/36938 -- GLOBAL_STEP: 737200 | > loss_text_ce: 0.04190343618392944 (0.03473700104210029) | > loss_mel_ce: 2.2821221351623535 (nan) | > loss: 0.009222324006259441 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13649842585317284) | > loader_time: 0.0044 (0.00463700727823665)  --> TIME: 2025-02-25 10:15:03 -- STEP: 35428/36938 -- GLOBAL_STEP: 737250 | > loss_text_ce: 0.04100378602743149 (0.03473453131724448) | > loss_mel_ce: 2.7087700366973877 (nan) | > loss: 0.010911800898611546 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.13649832884048962) | > loader_time: 0.0046 (0.004637008167166496)  --> TIME: 2025-02-25 10:15:13 -- STEP: 35478/36938 -- GLOBAL_STEP: 737300 | > loss_text_ce: 0.02709793671965599 (0.03473418445041683) | > loss_mel_ce: 1.908062219619751 (nan) | > loss: 0.0076792072504758835 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1364989529347385) | > loader_time: 0.0045 (0.004636940601827784)  --> TIME: 2025-02-25 10:15:22 -- STEP: 35528/36938 -- GLOBAL_STEP: 737350 | > loss_text_ce: 0.0423005037009716 (0.034733906990989716) | > loss_mel_ce: 2.859700918197632 (nan) | > loss: 0.011515879072248936 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13649745901219953) | > loader_time: 0.0046 (0.004636959519855506)  --> TIME: 2025-02-25 10:15:31 -- STEP: 35578/36938 -- GLOBAL_STEP: 737400 | > loss_text_ce: 0.042432207614183426 (0.034734160174600646) | > loss_mel_ce: 2.650583267211914 (nan) | > loss: 0.010686570778489113 (nan) | > current_lr: 5e-06 | > step_time: 0.1712 (0.13650369539624943) | > loader_time: 0.005 (0.004636930202417869)  --> TIME: 2025-02-25 10:15:41 -- STEP: 35628/36938 -- GLOBAL_STEP: 737450 | > loss_text_ce: 0.030005915090441704 (0.03473374759025552) | > loss_mel_ce: 2.798597812652588 (nan) | > loss: 0.01122461911290884 (nan) | > current_lr: 5e-06 | > step_time: 0.1262 (0.13651273008199716) | > loader_time: 0.0045 (0.004636910944872597)  --> TIME: 2025-02-25 10:15:50 -- STEP: 35678/36938 -- GLOBAL_STEP: 737500 | > loss_text_ce: 0.04006532207131386 (0.03473251758725842) | > loss_mel_ce: 4.483915328979492 (nan) | > loss: 0.01795230433344841 (nan) | > current_lr: 5e-06 | > step_time: 0.1116 (0.13650816226929133) | > loader_time: 0.0045 (0.0046368336168344)  --> TIME: 2025-02-25 10:16:00 -- STEP: 35728/36938 -- GLOBAL_STEP: 737550 | > loss_text_ce: 0.029801836237311363 (0.034731632483982644) | > loss_mel_ce: 3.458627223968506 (nan) | > loss: 0.013842972926795483 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13651641090248343) | > loader_time: 0.0045 (0.004636861046920477)  --> TIME: 2025-02-25 10:16:09 -- STEP: 35778/36938 -- GLOBAL_STEP: 737600 | > loss_text_ce: 0.03928989917039871 (0.03473058113437539) | > loss_mel_ce: 3.745518922805786 (nan) | > loss: 0.015019083395600319 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.136521211757476) | > loader_time: 0.0048 (0.00463685120949304)  --> TIME: 2025-02-25 10:16:19 -- STEP: 35828/36938 -- GLOBAL_STEP: 737650 | > loss_text_ce: 0.030632413923740387 (0.03473139263641075) | > loss_mel_ce: 2.8198447227478027 (nan) | > loss: 0.011311418376863003 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13652062499923817) | > loader_time: 0.0046 (0.004636821342762969)  --> TIME: 2025-02-25 10:16:28 -- STEP: 35878/36938 -- GLOBAL_STEP: 737700 | > loss_text_ce: 0.03356607258319855 (0.03473212018380004) | > loss_mel_ce: 2.5956203937530518 (nan) | > loss: 0.010433279909193516 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13652232319384122) | > loader_time: 0.0046 (0.004636767370537753)  --> TIME: 2025-02-25 10:16:37 -- STEP: 35928/36938 -- GLOBAL_STEP: 737750 | > loss_text_ce: 0.028813127428293228 (0.034730899159535755) | > loss_mel_ce: 2.530380964279175 (nan) | > loss: 0.010155532509088516 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.1365203543737672) | > loader_time: 0.0046 (0.004636724139608317)  --> TIME: 2025-02-25 10:16:47 -- STEP: 35978/36938 -- GLOBAL_STEP: 737800 | > loss_text_ce: 0.035486605018377304 (0.03473080236016301) | > loss_mel_ce: 2.518906593322754 (nan) | > loss: 0.010136482305824757 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.13651991883858497) | > loader_time: 0.0048 (0.004636714692921327)  --> TIME: 2025-02-25 10:16:56 -- STEP: 36028/36938 -- GLOBAL_STEP: 737850 | > loss_text_ce: 0.04453602805733681 (0.03473062990951423) | > loss_mel_ce: 3.6176798343658447 (nan) | > loss: 0.0145326042547822 (nan) | > current_lr: 5e-06 | > step_time: 0.1789 (0.13651753610149578) | > loader_time: 0.005 (0.0046366810586882)  --> TIME: 2025-02-25 10:17:05 -- STEP: 36078/36938 -- GLOBAL_STEP: 737900 | > loss_text_ce: 0.047220904380083084 (0.03473218329022313) | > loss_mel_ce: 2.457101583480835 (nan) | > loss: 0.009937788359820843 (nan) | > current_lr: 5e-06 | > step_time: 0.1086 (0.13652030666583345) | > loader_time: 0.0045 (0.0046367350528136895)  --> TIME: 2025-02-25 10:17:14 -- STEP: 36128/36938 -- GLOBAL_STEP: 737950 | > loss_text_ce: 0.044549234211444855 (0.034732760333754595) | > loss_mel_ce: 2.577221393585205 (nan) | > loss: 0.010403851978480816 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1365146993059731) | > loader_time: 0.0045 (0.004636661056346651)  --> TIME: 2025-02-25 10:17:23 -- STEP: 36178/36938 -- GLOBAL_STEP: 738000 | > loss_text_ce: 0.03505592420697212 (0.034730910800085044) | > loss_mel_ce: 2.5980417728424072 (nan) | > loss: 0.010448800399899483 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13650517086990932) | > loader_time: 0.0046 (0.004636566531790625)  --> TIME: 2025-02-25 10:17:32 -- STEP: 36228/36938 -- GLOBAL_STEP: 738050 | > loss_text_ce: 0.03401542082428932 (0.03472798449690455) | > loss_mel_ce: 3.0716710090637207 (nan) | > loss: 0.012324153445661068 (nan) | > current_lr: 5e-06 | > step_time: 0.1807 (0.13650580027435927) | > loader_time: 0.0048 (0.004636503811162648)  --> TIME: 2025-02-25 10:17:42 -- STEP: 36278/36938 -- GLOBAL_STEP: 738100 | > loss_text_ce: 0.030446786433458328 (0.034728096189973745) | > loss_mel_ce: 1.2401829957962036 (nan) | > loss: 0.005042181815952063 (nan) | > current_lr: 5e-06 | > step_time: 0.108 (0.13650297576402326) | > loader_time: 0.0044 (0.004636504209924204)  --> TIME: 2025-02-25 10:17:51 -- STEP: 36328/36938 -- GLOBAL_STEP: 738150 | > loss_text_ce: 0.028723591938614845 (0.03472532821133976) | > loss_mel_ce: 2.476254463195801 (nan) | > loss: 0.009940389543771744 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13650383081790063) | > loader_time: 0.0045 (0.004636482923625709)  --> TIME: 2025-02-25 10:18:00 -- STEP: 36378/36938 -- GLOBAL_STEP: 738200 | > loss_text_ce: 0.03621496260166168 (0.03472335913447281) | > loss_mel_ce: 2.8614747524261475 (nan) | > loss: 0.011498769745230675 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.1365018647121714) | > loader_time: 0.0046 (0.004636477916798683)  --> TIME: 2025-02-25 10:18:10 -- STEP: 36428/36938 -- GLOBAL_STEP: 738250 | > loss_text_ce: 0.03268597647547722 (0.03472124491113136) | > loss_mel_ce: 2.0053696632385254 (nan) | > loss: 0.00808752328157425 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.1365070161003178) | > loader_time: 0.0046 (0.004636580103430923)  --> TIME: 2025-02-25 10:18:19 -- STEP: 36478/36938 -- GLOBAL_STEP: 738300 | > loss_text_ce: 0.03916899859905243 (0.03472036890806271) | > loss_mel_ce: 2.9568378925323486 (nan) | > loss: 0.011888917535543442 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.1365114617775172) | > loader_time: 0.0048 (0.004636609774552458)  --> TIME: 2025-02-25 10:18:29 -- STEP: 36528/36938 -- GLOBAL_STEP: 738350 | > loss_text_ce: 0.03480072319507599 (0.03472052428982965) | > loss_mel_ce: 3.1179709434509277 (nan) | > loss: 0.012511000037193298 (nan) | > current_lr: 5e-06 | > step_time: 0.1902 (0.13651578849572366) | > loader_time: 0.0048 (0.004636647281707294)  --> TIME: 2025-02-25 10:18:38 -- STEP: 36578/36938 -- GLOBAL_STEP: 738400 | > loss_text_ce: 0.035730715841054916 (0.03472025017474294) | > loss_mel_ce: 2.9636385440826416 (nan) | > loss: 0.011902258731424809 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.1365114061520476) | > loader_time: 0.0046 (0.004636558483125235)  --> TIME: 2025-02-25 10:18:47 -- STEP: 36628/36938 -- GLOBAL_STEP: 738450 | > loss_text_ce: 0.027707137167453766 (0.03472007391379347) | > loss_mel_ce: 2.827448844909668 (nan) | > loss: 0.011329984292387962 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13651346676055717) | > loader_time: 0.0045 (0.004636485392810135)  --> TIME: 2025-02-25 10:18:57 -- STEP: 36678/36938 -- GLOBAL_STEP: 738500 | > loss_text_ce: 0.0295222457498312 (0.03472025635630861) | > loss_mel_ce: 2.05122709274292 (nan) | > loss: 0.0082569420337677 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13651138348098155) | > loader_time: 0.0045 (0.004636478590480344)  --> TIME: 2025-02-25 10:19:06 -- STEP: 36728/36938 -- GLOBAL_STEP: 738550 | > loss_text_ce: 0.0405937097966671 (0.034721322763117644) | > loss_mel_ce: 0.041259765625 (nan) | > loss: 0.00032481542439199984 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13651297629801373) | > loader_time: 0.0044 (0.004636490800702157)  --> TIME: 2025-02-25 10:19:15 -- STEP: 36778/36938 -- GLOBAL_STEP: 738600 | > loss_text_ce: 0.03200314939022064 (0.0347203257112769) | > loss_mel_ce: 0.5758668184280396 (nan) | > loss: 0.0024121825117617846 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13650913869027118) | > loader_time: 0.0045 (0.0046364234292336605)  --> TIME: 2025-02-25 10:19:24 -- STEP: 36828/36938 -- GLOBAL_STEP: 738650 | > loss_text_ce: 0.035881832242012024 (0.03471923027739077) | > loss_mel_ce: 1.7609121799468994 (nan) | > loss: 0.007130135782063007 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13651239709485408) | > loader_time: 0.0045 (0.00463639499310679)  --> TIME: 2025-02-25 10:19:34 -- STEP: 36878/36938 -- GLOBAL_STEP: 738700 | > loss_text_ce: 0.043464627116918564 (0.034720835458333536) | > loss_mel_ce: 2.4453747272491455 (nan) | > loss: 0.00987634714692831 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13651075941911617) | > loader_time: 0.0046 (0.004636345829517073)  --> TIME: 2025-02-25 10:19:44 -- STEP: 36928/36938 -- GLOBAL_STEP: 738750 | > loss_text_ce: 0.036107372492551804 (0.034721028290382705) | > loss_mel_ce: 3.0920333862304688 (nan) | > loss: 0.012413257732987404 (nan) | > current_lr: 5e-06 | > step_time: 0.1814 (0.13652173801610987) | > loader_time: 0.0049 (0.004636325691043807)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.021163055212190834 (+0.0007893637855454216) | > avg_loss_text_ce: 0.03390969807226764 (-0.00030315201729537444) | > avg_loss_mel_ce: 2.5616747140884395 (-0.00706410171962224) | > avg_loss: 2.595584405530796 (-0.007367266581790766) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_738760.pth  > EPOCH: 21/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-25 10:20:12)   --> TIME: 2025-02-25 10:20:19 -- STEP: 40/36938 -- GLOBAL_STEP: 738800 | > loss_text_ce: 0.030671434476971626 (0.03346551796421409) | > loss_mel_ce: 3.1630609035491943 (2.453424990363419) | > loss: 0.012673541903495789 (0.009868613666913006) | > current_lr: 5e-06 | > step_time: 0.1312 (0.12968588471412662) | > loader_time: 0.0048 (0.004778963327407837)  --> TIME: 2025-02-25 10:20:28 -- STEP: 90/36938 -- GLOBAL_STEP: 738850 | > loss_text_ce: 0.038958657532930374 (0.033831188765664884) | > loss_mel_ce: 3.10612154006958 (2.6504169884241278) | > loss: 0.01248047687113285 (0.010651779056125734) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13437407546573224) | > loader_time: 0.0046 (0.004792234632703995)  --> TIME: 2025-02-25 10:20:38 -- STEP: 140/36938 -- GLOBAL_STEP: 738900 | > loss_text_ce: 0.041513100266456604 (0.034174150308328) | > loss_mel_ce: 3.0557024478912354 (2.6969423935881673) | > loss: 0.012290539219975471 (0.010837764668810581) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13543167965752742) | > loader_time: 0.0047 (0.004794231482914518)  --> TIME: 2025-02-25 10:20:47 -- STEP: 190/36938 -- GLOBAL_STEP: 738950 | > loss_text_ce: 0.03933410719037056 (0.03477496160684445) | > loss_mel_ce: 2.8702449798583984 (nan) | > loss: 0.01154594961553812 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.1344088403802169) | > loader_time: 0.0045 (0.004763148960314298)  --> TIME: 2025-02-25 10:20:56 -- STEP: 240/36938 -- GLOBAL_STEP: 739000 | > loss_text_ce: 0.028260542079806328 (0.03477910331760845) | > loss_mel_ce: 2.649571180343628 (nan) | > loss: 0.010626317001879215 (nan) | > current_lr: 5e-06 | > step_time: 0.1521 (0.1333046764135359) | > loader_time: 0.0052 (0.004735261201858521)  --> TIME: 2025-02-25 10:21:05 -- STEP: 290/36938 -- GLOBAL_STEP: 739050 | > loss_text_ce: 0.038649845868349075 (0.03458766393235019) | > loss_mel_ce: 3.135221004486084 (nan) | > loss: 0.01259472593665123 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.13357002981777832) | > loader_time: 0.0046 (0.004721981903602337)  --> TIME: 2025-02-25 10:21:14 -- STEP: 340/36938 -- GLOBAL_STEP: 739100 | > loss_text_ce: 0.03503541275858879 (0.034668119821478334) | > loss_mel_ce: 2.5579946041107178 (nan) | > loss: 0.010289802215993404 (nan) | > current_lr: 5e-06 | > step_time: 0.1756 (0.134208400810466) | > loader_time: 0.0048 (0.004721505501691035)  --> TIME: 2025-02-25 10:21:23 -- STEP: 390/36938 -- GLOBAL_STEP: 739150 | > loss_text_ce: 0.037525638937950134 (0.03466606547769446) | > loss_mel_ce: 3.3053250312805176 (nan) | > loss: 0.013265280984342098 (nan) | > current_lr: 5e-06 | > step_time: 0.1788 (0.13397237398685552) | > loader_time: 0.0049 (0.004707094339224009)  --> TIME: 2025-02-25 10:21:33 -- STEP: 440/36938 -- GLOBAL_STEP: 739200 | > loss_text_ce: 0.02786301262676716 (0.03473859022329138) | > loss_mel_ce: 2.305234670639038 (nan) | > loss: 0.009258324280381203 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13464817513119093) | > loader_time: 0.0045 (0.004706523093310272)  --> TIME: 2025-02-25 10:21:42 -- STEP: 490/36938 -- GLOBAL_STEP: 739250 | > loss_text_ce: 0.04533061385154724 (0.034853790322204634) | > loss_mel_ce: 1.6063385009765625 (nan) | > loss: 0.00655424315482378 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13461966076675733) | > loader_time: 0.0045 (0.004703678403581894)  --> TIME: 2025-02-25 10:21:52 -- STEP: 540/36938 -- GLOBAL_STEP: 739300 | > loss_text_ce: 0.03718600422143936 (0.034801925234151654) | > loss_mel_ce: 2.9497334957122803 (nan) | > loss: 0.011852855794131756 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13488219932273587) | > loader_time: 0.0046 (0.004702546861436634)  --> TIME: 2025-02-25 10:22:01 -- STEP: 590/36938 -- GLOBAL_STEP: 739350 | > loss_text_ce: 0.034965880215168 (0.03464770096875096) | > loss_mel_ce: 1.6327636241912842 (nan) | > loss: 0.006617974489927292 (nan) | > current_lr: 5e-06 | > step_time: 0.1692 (0.13515844749192063) | > loader_time: 0.0047 (0.004698805485741568)  --> TIME: 2025-02-25 10:22:10 -- STEP: 640/36938 -- GLOBAL_STEP: 739400 | > loss_text_ce: 0.03506310284137726 (0.03466145846905419) | > loss_mel_ce: 3.1351749897003174 (nan) | > loss: 0.012580309994518757 (nan) | > current_lr: 5e-06 | > step_time: 0.173 (0.1351107031106949) | > loader_time: 0.0047 (0.004694180563092232)  --> TIME: 2025-02-25 10:22:20 -- STEP: 690/36938 -- GLOBAL_STEP: 739450 | > loss_text_ce: 0.026971690356731415 (0.03464434116241941) | > loss_mel_ce: 1.4240646362304688 (nan) | > loss: 0.005758081097155809 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13539281133292375) | > loader_time: 0.0045 (0.004688789187998012)  --> TIME: 2025-02-25 10:22:29 -- STEP: 740/36938 -- GLOBAL_STEP: 739500 | > loss_text_ce: 0.032665517181158066 (0.03467303008923459) | > loss_mel_ce: 3.0665838718414307 (nan) | > loss: 0.012298609130084515 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13528409584148496) | > loader_time: 0.0045 (0.004689008158606454)  --> TIME: 2025-02-25 10:22:38 -- STEP: 790/36938 -- GLOBAL_STEP: 739550 | > loss_text_ce: 0.035627447068691254 (0.034671917459868536) | > loss_mel_ce: 3.3325388431549072 (nan) | > loss: 0.013365739956498146 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.13535115235968473) | > loader_time: 0.0048 (0.004685890825488899)  --> TIME: 2025-02-25 10:22:47 -- STEP: 840/36938 -- GLOBAL_STEP: 739600 | > loss_text_ce: 0.029084626585245132 (0.034716906860869895) | > loss_mel_ce: 3.4223039150238037 (nan) | > loss: 0.013695986941456795 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.13546697639283672) | > loader_time: 0.0046 (0.00468553134373256)  --> TIME: 2025-02-25 10:22:57 -- STEP: 890/36938 -- GLOBAL_STEP: 739650 | > loss_text_ce: 0.042312756180763245 (0.03464209908263732) | > loss_mel_ce: 3.372455596923828 (nan) | > loss: 0.013550668954849243 (nan) | > current_lr: 5e-06 | > step_time: 0.1667 (0.13552448535233397) | > loader_time: 0.0047 (0.004681534713573669)  --> TIME: 2025-02-25 10:23:06 -- STEP: 940/36938 -- GLOBAL_STEP: 739700 | > loss_text_ce: 0.03828974813222885 (0.034700419315870076) | > loss_mel_ce: 3.695868492126465 (nan) | > loss: 0.014818089082837105 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13534351054658286) | > loader_time: 0.0046 (0.00468022493605918)  --> TIME: 2025-02-25 10:23:15 -- STEP: 990/36938 -- GLOBAL_STEP: 739750 | > loss_text_ce: 0.024636056274175644 (0.03474308906329997) | > loss_mel_ce: 2.6859002113342285 (nan) | > loss: 0.010756096802651882 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.1356227474983293) | > loader_time: 0.0048 (0.004678306675920582)  --> TIME: 2025-02-25 10:23:25 -- STEP: 1040/36938 -- GLOBAL_STEP: 739800 | > loss_text_ce: 0.04834417253732681 (0.03474172867942023) | > loss_mel_ce: 3.2310261726379395 (nan) | > loss: 0.013013374991714954 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13555076443232028) | > loader_time: 0.0045 (0.004675159316796523)  --> TIME: 2025-02-25 10:23:34 -- STEP: 1090/36938 -- GLOBAL_STEP: 739850 | > loss_text_ce: 0.025683388113975525 (0.03479666050967814) | > loss_mel_ce: 2.423027753829956 (nan) | > loss: 0.009717108681797981 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13570966064383133) | > loader_time: 0.0045 (0.0046732530681365125)  --> TIME: 2025-02-25 10:23:43 -- STEP: 1140/36938 -- GLOBAL_STEP: 739900 | > loss_text_ce: 0.0334661565721035 (0.034710820415465665) | > loss_mel_ce: 3.005657196044922 (nan) | > loss: 0.012060013599693775 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13561917982603375) | > loader_time: 0.0045 (0.004671801182261683)  --> TIME: 2025-02-25 10:23:52 -- STEP: 1190/36938 -- GLOBAL_STEP: 739950 | > loss_text_ce: 0.03687326982617378 (0.034751689118654026) | > loss_mel_ce: 1.782458782196045 (nan) | > loss: 0.007219572085887194 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13534561726225527) | > loader_time: 0.0046 (0.004670375936171587)  --> TIME: 2025-02-25 10:24:01 -- STEP: 1240/36938 -- GLOBAL_STEP: 740000 | > loss_text_ce: 0.035630762577056885 (0.03473324423445568) | > loss_mel_ce: 2.866563320159912 (nan) | > loss: 0.011516643688082695 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13532039330851647) | > loader_time: 0.0046 (0.004668683390463553) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_740000.pth  --> TIME: 2025-02-25 10:24:17 -- STEP: 1290/36938 -- GLOBAL_STEP: 740050 | > loss_text_ce: 0.026069946587085724 (0.034763325179039045) | > loss_mel_ce: 2.111772060394287 (nan) | > loss: 0.008483500219881535 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13540191391641782) | > loader_time: 0.0046 (0.004669332689093066)  --> TIME: 2025-02-25 10:24:26 -- STEP: 1340/36938 -- GLOBAL_STEP: 740100 | > loss_text_ce: 0.03842630237340927 (0.03475842735549404) | > loss_mel_ce: 2.694521903991699 (nan) | > loss: 0.010845033451914787 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13539714581930826) | > loader_time: 0.0045 (0.004672879069598756)  --> TIME: 2025-02-25 10:24:35 -- STEP: 1390/36938 -- GLOBAL_STEP: 740150 | > loss_text_ce: 0.03202082961797714 (0.034723321970203766) | > loss_mel_ce: 3.2174930572509766 (nan) | > loss: 0.012894896790385246 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13536457741003255) | > loader_time: 0.0045 (0.0046725285139015285)  --> TIME: 2025-02-25 10:24:45 -- STEP: 1440/36938 -- GLOBAL_STEP: 740200 | > loss_text_ce: 0.02882704883813858 (0.03472593101000205) | > loss_mel_ce: 3.1023941040039062 (nan) | > loss: 0.012425481341779232 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.135485732720958) | > loader_time: 0.0048 (0.00467326790094376)  --> TIME: 2025-02-25 10:24:54 -- STEP: 1490/36938 -- GLOBAL_STEP: 740250 | > loss_text_ce: 0.03362707048654556 (0.034732671264534026) | > loss_mel_ce: 2.794278860092163 (nan) | > loss: 0.011221849359571934 (nan) | > current_lr: 5e-06 | > step_time: 0.1528 (0.13544190137978385) | > loader_time: 0.0046 (0.004673822294145621)  --> TIME: 2025-02-25 10:25:03 -- STEP: 1540/36938 -- GLOBAL_STEP: 740300 | > loss_text_ce: 0.04243132472038269 (0.03469744954596869) | > loss_mel_ce: 2.973219394683838 (nan) | > loss: 0.011966868303716183 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13541308440171287) | > loader_time: 0.0047 (0.004673817095818457)  --> TIME: 2025-02-25 10:25:13 -- STEP: 1590/36938 -- GLOBAL_STEP: 740350 | > loss_text_ce: 0.0353822335600853 (0.034700880975187205) | > loss_mel_ce: 2.7465388774871826 (nan) | > loss: 0.01103937067091465 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13552302369531613) | > loader_time: 0.0047 (0.004676283980315581)  --> TIME: 2025-02-25 10:25:22 -- STEP: 1640/36938 -- GLOBAL_STEP: 740400 | > loss_text_ce: 0.03282878175377846 (0.03470814699057217) | > loss_mel_ce: 3.2057409286499023 (nan) | > loss: 0.012851468287408352 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.1355009509295953) | > loader_time: 0.0046 (0.004676324855990523)  --> TIME: 2025-02-25 10:25:31 -- STEP: 1690/36938 -- GLOBAL_STEP: 740450 | > loss_text_ce: 0.03544846177101135 (0.03469709781433882) | > loss_mel_ce: 2.6930925846099854 (nan) | > loss: 0.010827545076608658 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.1355250798738921) | > loader_time: 0.0047 (0.004676052522377147)  --> TIME: 2025-02-25 10:25:41 -- STEP: 1740/36938 -- GLOBAL_STEP: 740500 | > loss_text_ce: 0.044947121292352676 (0.03469328235717084) | > loss_mel_ce: 3.125154495239258 (nan) | > loss: 0.012579768896102905 (nan) | > current_lr: 5e-06 | > step_time: 0.1441 (0.13560190611872197) | > loader_time: 0.0048 (0.0046767851402019596)  --> TIME: 2025-02-25 10:25:50 -- STEP: 1790/36938 -- GLOBAL_STEP: 740550 | > loss_text_ce: 0.04281090199947357 (0.03470849007961221) | > loss_mel_ce: 2.4420573711395264 (nan) | > loss: 0.009860589168965816 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13576443754760942) | > loader_time: 0.0047 (0.004677895727104311)  --> TIME: 2025-02-25 10:26:00 -- STEP: 1840/36938 -- GLOBAL_STEP: 740600 | > loss_text_ce: 0.03293211758136749 (0.03471149481407813) | > loss_mel_ce: 3.26416015625 (nan) | > loss: 0.013083700090646744 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13576459029446503) | > loader_time: 0.0046 (0.004677135659300759)  --> TIME: 2025-02-25 10:26:09 -- STEP: 1890/36938 -- GLOBAL_STEP: 740650 | > loss_text_ce: 0.027089674025774002 (0.0347158013345309) | > loss_mel_ce: 3.2239983081817627 (nan) | > loss: 0.012901143170893192 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.13583447567369608) | > loader_time: 0.0048 (0.004676832093132863)  --> TIME: 2025-02-25 10:26:18 -- STEP: 1940/36938 -- GLOBAL_STEP: 740700 | > loss_text_ce: 0.027749206870794296 (0.034719643816735944) | > loss_mel_ce: 3.078775644302368 (nan) | > loss: 0.012327481061220169 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.13588616344117663) | > loader_time: 0.0047 (0.004676946168093334)  --> TIME: 2025-02-25 10:26:27 -- STEP: 1990/36938 -- GLOBAL_STEP: 740750 | > loss_text_ce: 0.031239237636327744 (0.03473319468373929) | > loss_mel_ce: 2.1871676445007324 (nan) | > loss: 0.008803202770650387 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13575710531455223) | > loader_time: 0.0045 (0.004676371962580845)  --> TIME: 2025-02-25 10:26:37 -- STEP: 2040/36938 -- GLOBAL_STEP: 740800 | > loss_text_ce: 0.035932909697294235 (0.03472674506369467) | > loss_mel_ce: 3.2768685817718506 (nan) | > loss: 0.013146039098501205 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.1357863874996413) | > loader_time: 0.0048 (0.00467468652070737)  --> TIME: 2025-02-25 10:26:46 -- STEP: 2090/36938 -- GLOBAL_STEP: 740850 | > loss_text_ce: 0.05943719670176506 (0.03475306466506999) | > loss_mel_ce: 3.3273000717163086 (nan) | > loss: 0.013439434580504894 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13574356177206848) | > loader_time: 0.0045 (0.004672732079428346)  --> TIME: 2025-02-25 10:26:55 -- STEP: 2140/36938 -- GLOBAL_STEP: 740900 | > loss_text_ce: 0.0339869000017643 (0.034752988398388446) | > loss_mel_ce: 2.553955078125 (nan) | > loss: 0.01026961114257574 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13576973609835224) | > loader_time: 0.0045 (0.004672065627909148)  --> TIME: 2025-02-25 10:27:04 -- STEP: 2190/36938 -- GLOBAL_STEP: 740950 | > loss_text_ce: 0.028650173917412758 (0.03478734897849359) | > loss_mel_ce: 1.3080512285232544 (nan) | > loss: 0.005304371006786823 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.1358215261267748) | > loader_time: 0.0046 (0.004671774498403886)  --> TIME: 2025-02-25 10:27:14 -- STEP: 2240/36938 -- GLOBAL_STEP: 741000 | > loss_text_ce: 0.03092818520963192 (0.03482216738214317) | > loss_mel_ce: 3.238523244857788 (nan) | > loss: 0.01297401450574398 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.1357522799500401) | > loader_time: 0.0047 (0.004672557967049733)  --> TIME: 2025-02-25 10:27:23 -- STEP: 2290/36938 -- GLOBAL_STEP: 741050 | > loss_text_ce: 0.04628881439566612 (0.03480088013797739) | > loss_mel_ce: 3.242520809173584 (nan) | > loss: 0.013050831854343414 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13577577276521421) | > loader_time: 0.0046 (0.004671871818309266)  --> TIME: 2025-02-25 10:27:32 -- STEP: 2340/36938 -- GLOBAL_STEP: 741100 | > loss_text_ce: 0.03626624122262001 (0.034766888248328236) | > loss_mel_ce: 0.046142578125 (nan) | > loss: 0.00032701913733035326 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13569680044793725) | > loader_time: 0.0049 (0.004671932081890919)  --> TIME: 2025-02-25 10:27:42 -- STEP: 2390/36938 -- GLOBAL_STEP: 741150 | > loss_text_ce: 0.02535269968211651 (0.03472968798606155) | > loss_mel_ce: 2.838623046875 (nan) | > loss: 0.011364983394742012 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.1358590970977085) | > loader_time: 0.0048 (0.004672627668500445)  --> TIME: 2025-02-25 10:27:52 -- STEP: 2440/36938 -- GLOBAL_STEP: 741200 | > loss_text_ce: 0.038891687989234924 (0.034741349051492794) | > loss_mel_ce: 2.352555751800537 (nan) | > loss: 0.009489871561527252 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.1360185166851422) | > loader_time: 0.0047 (0.004672727623923882)  --> TIME: 2025-02-25 10:28:01 -- STEP: 2490/36938 -- GLOBAL_STEP: 741250 | > loss_text_ce: 0.04556355997920036 (0.034730837652347586) | > loss_mel_ce: 2.2309467792510986 (nan) | > loss: 0.009033771231770515 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13601923572969268) | > loader_time: 0.0045 (0.004672776072858329)  --> TIME: 2025-02-25 10:28:10 -- STEP: 2540/36938 -- GLOBAL_STEP: 741300 | > loss_text_ce: 0.03644216060638428 (0.03471011043174239) | > loss_mel_ce: 3.0138888359069824 (nan) | > loss: 0.012104488909244537 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1359870193511484) | > loader_time: 0.0046 (0.004673042259817049)  --> TIME: 2025-02-25 10:28:19 -- STEP: 2590/36938 -- GLOBAL_STEP: 741350 | > loss_text_ce: 0.02977851592004299 (0.03470772771222309) | > loss_mel_ce: 2.245981216430664 (nan) | > loss: 0.009030792862176895 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13597582399154742) | > loader_time: 0.0046 (0.004672831458014408)  --> TIME: 2025-02-25 10:28:29 -- STEP: 2640/36938 -- GLOBAL_STEP: 741400 | > loss_text_ce: 0.030457763001322746 (0.03467830472632847) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1118 (0.1360487498117217) | > loader_time: 0.0044 (0.0046732289321494815)  --> TIME: 2025-02-25 10:28:38 -- STEP: 2690/36938 -- GLOBAL_STEP: 741450 | > loss_text_ce: 0.019090736284852028 (0.03467463025490864) | > loss_mel_ce: 2.547572374343872 (nan) | > loss: 0.01018517091870308 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13592546393880187) | > loader_time: 0.0045 (0.004671611395910327)  --> TIME: 2025-02-25 10:28:47 -- STEP: 2740/36938 -- GLOBAL_STEP: 741500 | > loss_text_ce: 0.02769305929541588 (0.0346819240847997) | > loss_mel_ce: 3.1105518341064453 (nan) | > loss: 0.01245335303246975 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1359778782747089) | > loader_time: 0.0045 (0.004671028669733198)  --> TIME: 2025-02-25 10:28:56 -- STEP: 2790/36938 -- GLOBAL_STEP: 741550 | > loss_text_ce: 0.03522537276148796 (0.034679774221994246) | > loss_mel_ce: 2.556820869445801 (nan) | > loss: 0.010285898111760616 (nan) | > current_lr: 5e-06 | > step_time: 0.1822 (0.13598781768566406) | > loader_time: 0.0049 (0.004670218413021401)  --> TIME: 2025-02-25 10:29:06 -- STEP: 2840/36938 -- GLOBAL_STEP: 741600 | > loss_text_ce: 0.03585335239768028 (0.03468787659740104) | > loss_mel_ce: 2.936465263366699 (nan) | > loss: 0.011794916354119778 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.1360581729613562) | > loader_time: 0.0046 (0.004669541624230398)  --> TIME: 2025-02-25 10:29:15 -- STEP: 2890/36938 -- GLOBAL_STEP: 741650 | > loss_text_ce: 0.0310211181640625 (0.0347193146366175) | > loss_mel_ce: 2.490722179412842 (nan) | > loss: 0.0100069185718894 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.1360864428912895) | > loader_time: 0.0046 (0.0046687923913183525)  --> TIME: 2025-02-25 10:29:24 -- STEP: 2940/36938 -- GLOBAL_STEP: 741700 | > loss_text_ce: 0.04870165139436722 (0.03473967939621589) | > loss_mel_ce: 3.0040411949157715 (nan) | > loss: 0.01211406011134386 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.13603980338492386) | > loader_time: 0.0048 (0.004667436671094827)  --> TIME: 2025-02-25 10:29:33 -- STEP: 2990/36938 -- GLOBAL_STEP: 741750 | > loss_text_ce: 0.038476672023534775 (0.034733544287262504) | > loss_mel_ce: 3.4347102642059326 (nan) | > loss: 0.013782489113509655 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13601348129004576) | > loader_time: 0.0047 (0.004666660541674759)  --> TIME: 2025-02-25 10:29:43 -- STEP: 3040/36938 -- GLOBAL_STEP: 741800 | > loss_text_ce: 0.03113687038421631 (0.03472356607763741) | > loss_mel_ce: 1.2162710428237915 (nan) | > loss: 0.004950031638145447 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13595683590362) | > loader_time: 0.0046 (0.0046652581346662385)  --> TIME: 2025-02-25 10:29:52 -- STEP: 3090/36938 -- GLOBAL_STEP: 741850 | > loss_text_ce: 0.030737096443772316 (0.03471128945675921) | > loss_mel_ce: 2.5899598598480225 (nan) | > loss: 0.010399592109024525 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13603081031910472) | > loader_time: 0.0046 (0.0046656262141601176)  --> TIME: 2025-02-25 10:30:01 -- STEP: 3140/36938 -- GLOBAL_STEP: 741900 | > loss_text_ce: 0.01818801835179329 (0.034696452083825484) | > loss_mel_ce: 0.37446197867393494 (nan) | > loss: 0.0015581350307911634 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13605368600529522) | > loader_time: 0.0045 (0.004665959868461463)  --> TIME: 2025-02-25 10:30:11 -- STEP: 3190/36938 -- GLOBAL_STEP: 741950 | > loss_text_ce: 0.025270413607358932 (0.03470374743270237) | > loss_mel_ce: 3.2438812255859375 (nan) | > loss: 0.012972825206816196 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.13606348022771908) | > loader_time: 0.0048 (0.0046659243144211715)  --> TIME: 2025-02-25 10:30:20 -- STEP: 3240/36938 -- GLOBAL_STEP: 742000 | > loss_text_ce: 0.039377957582473755 (0.0347025324202454) | > loss_mel_ce: 2.9305367469787598 (nan) | > loss: 0.011785376816987991 (nan) | > current_lr: 5e-06 | > step_time: 0.1763 (0.1360225646584128) | > loader_time: 0.0048 (0.004665747983956043)  --> TIME: 2025-02-25 10:30:29 -- STEP: 3290/36938 -- GLOBAL_STEP: 742050 | > loss_text_ce: 0.0419561043381691 (0.03469747696071871) | > loss_mel_ce: 2.684216260910034 (nan) | > loss: 0.010818145237863064 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13601871080311656) | > loader_time: 0.0046 (0.004665350261792589)  --> TIME: 2025-02-25 10:30:39 -- STEP: 3340/36938 -- GLOBAL_STEP: 742100 | > loss_text_ce: 0.04005356878042221 (0.03466631798129425) | > loss_mel_ce: 2.8179068565368652 (nan) | > loss: 0.011341113597154617 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13607253857001642) | > loader_time: 0.0046 (0.004664650434505442)  --> TIME: 2025-02-25 10:30:48 -- STEP: 3390/36938 -- GLOBAL_STEP: 742150 | > loss_text_ce: 0.03919921815395355 (0.034651337358496244) | > loss_mel_ce: 0.004644562024623156 (nan) | > loss: 0.000173983265995048 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13605140563661083) | > loader_time: 0.0045 (0.004664142715544124)  --> TIME: 2025-02-25 10:30:57 -- STEP: 3440/36938 -- GLOBAL_STEP: 742200 | > loss_text_ce: 0.03246934339404106 (0.03463864704637339) | > loss_mel_ce: 3.3159141540527344 (nan) | > loss: 0.013287236914038658 (nan) | > current_lr: 5e-06 | > step_time: 0.1693 (0.1360498897558042) | > loader_time: 0.0048 (0.004663215958794882)  --> TIME: 2025-02-25 10:31:06 -- STEP: 3490/36938 -- GLOBAL_STEP: 742250 | > loss_text_ce: 0.03155144676566124 (0.03463381431772671) | > loss_mel_ce: 3.185210943222046 (nan) | > loss: 0.012764930725097656 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13605730916162695) | > loader_time: 0.0047 (0.004662535525324693)  --> TIME: 2025-02-25 10:31:16 -- STEP: 3540/36938 -- GLOBAL_STEP: 742300 | > loss_text_ce: 0.03181074187159538 (0.0346372063995187) | > loss_mel_ce: 2.9375 (nan) | > loss: 0.011782979592680931 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1361464742213327) | > loader_time: 0.0044 (0.004663059401646846)  --> TIME: 2025-02-25 10:31:25 -- STEP: 3590/36938 -- GLOBAL_STEP: 742350 | > loss_text_ce: 0.030165182426571846 (0.03465020343386328) | > loss_mel_ce: 3.1656782627105713 (nan) | > loss: 0.012681919150054455 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.13607876194552876) | > loader_time: 0.0046 (0.0046623906053208375)  --> TIME: 2025-02-25 10:31:34 -- STEP: 3640/36938 -- GLOBAL_STEP: 742400 | > loss_text_ce: 0.03720696270465851 (0.03464832441180254) | > loss_mel_ce: 2.701789379119873 (nan) | > loss: 0.010869033634662628 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13602774077719476) | > loader_time: 0.0046 (0.0046617722118293846)  --> TIME: 2025-02-25 10:31:44 -- STEP: 3690/36938 -- GLOBAL_STEP: 742450 | > loss_text_ce: 0.022749708965420723 (0.034631558190930306) | > loss_mel_ce: 0.022705078125 (nan) | > loss: 0.00018037614063359797 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13607035174279442) | > loader_time: 0.0045 (0.004661556311093049)  --> TIME: 2025-02-25 10:31:53 -- STEP: 3740/36938 -- GLOBAL_STEP: 742500 | > loss_text_ce: 0.0333346389234066 (0.03464164947916604) | > loss_mel_ce: 1.5960235595703125 (nan) | > loss: 0.006465707439929247 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1361054738575133) | > loader_time: 0.0045 (0.004661554066255131)  --> TIME: 2025-02-25 10:32:02 -- STEP: 3790/36938 -- GLOBAL_STEP: 742550 | > loss_text_ce: 0.03372127562761307 (0.03464470453250735) | > loss_mel_ce: 1.975658893585205 (nan) | > loss: 0.007973730564117432 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13606096377158938) | > loader_time: 0.0046 (0.004661289808932584)  --> TIME: 2025-02-25 10:32:12 -- STEP: 3840/36938 -- GLOBAL_STEP: 742600 | > loss_text_ce: 0.033903103321790695 (0.034664640709767554) | > loss_mel_ce: 3.2589712142944336 (nan) | > loss: 0.013066962361335754 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.1361539556334424) | > loader_time: 0.0048 (0.004661506476501624)  --> TIME: 2025-02-25 10:32:21 -- STEP: 3890/36938 -- GLOBAL_STEP: 742650 | > loss_text_ce: 0.03702770918607712 (0.03464712768931971) | > loss_mel_ce: 2.114985704421997 (nan) | > loss: 0.008539736270904541 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13608008742638988) | > loader_time: 0.0045 (0.004660807170720823)  --> TIME: 2025-02-25 10:32:30 -- STEP: 3940/36938 -- GLOBAL_STEP: 742700 | > loss_text_ce: 0.034531064331531525 (0.034623324359308615) | > loss_mel_ce: 3.137640953063965 (nan) | > loss: 0.012587985023856163 (nan) | > current_lr: 5e-06 | > step_time: 0.1531 (0.13605852562764048) | > loader_time: 0.0047 (0.004662246691999096)  --> TIME: 2025-02-25 10:32:39 -- STEP: 3990/36938 -- GLOBAL_STEP: 742750 | > loss_text_ce: 0.027390340343117714 (0.034597954000194685) | > loss_mel_ce: 1.8654837608337402 (nan) | > loss: 0.007511405739933252 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13602815941162952) | > loader_time: 0.0047 (0.004662788959971646)  --> TIME: 2025-02-25 10:32:48 -- STEP: 4040/36938 -- GLOBAL_STEP: 742800 | > loss_text_ce: 0.035762231796979904 (0.03460489137739058) | > loss_mel_ce: 3.460052967071533 (nan) | > loss: 0.013872283510863781 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13601527863209867) | > loader_time: 0.0046 (0.004662943358468537)  --> TIME: 2025-02-25 10:32:57 -- STEP: 4090/36938 -- GLOBAL_STEP: 742850 | > loss_text_ce: 0.0305717084556818 (0.034597113097091146) | > loss_mel_ce: 2.818531036376953 (nan) | > loss: 0.01130596362054348 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.1359521702624191) | > loader_time: 0.0046 (0.004663533336667385)  --> TIME: 2025-02-25 10:33:07 -- STEP: 4140/36938 -- GLOBAL_STEP: 742900 | > loss_text_ce: 0.02815118059515953 (0.03460278382619334) | > loss_mel_ce: 2.9138684272766113 (nan) | > loss: 0.011674681678414345 (nan) | > current_lr: 5e-06 | > step_time: 0.1526 (0.13593590017678156) | > loader_time: 0.0048 (0.0046638848125070765)  --> TIME: 2025-02-25 10:33:16 -- STEP: 4190/36938 -- GLOBAL_STEP: 742950 | > loss_text_ce: 0.023441174998879433 (0.03460469104611899) | > loss_mel_ce: 3.1094229221343994 (nan) | > loss: 0.01243200059980154 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13591825637726412) | > loader_time: 0.0047 (0.004664481966478444)  --> TIME: 2025-02-25 10:33:25 -- STEP: 4240/36938 -- GLOBAL_STEP: 743000 | > loss_text_ce: 0.03733929619193077 (0.034600712747954496) | > loss_mel_ce: 2.661442279815674 (nan) | > loss: 0.010709450580179691 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.1359565165245315) | > loader_time: 0.0048 (0.004665456684130544)  --> TIME: 2025-02-25 10:33:35 -- STEP: 4290/36938 -- GLOBAL_STEP: 743050 | > loss_text_ce: 0.034740421921014786 (0.03461803950799202) | > loss_mel_ce: 3.159381628036499 (nan) | > loss: 0.01267508789896965 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13597277550152753) | > loader_time: 0.0046 (0.0046651267227315165)  --> TIME: 2025-02-25 10:33:44 -- STEP: 4340/36938 -- GLOBAL_STEP: 743100 | > loss_text_ce: 0.02781422808766365 (0.03461447293904488) | > loss_mel_ce: 2.9501993656158447 (nan) | > loss: 0.011817514896392822 (nan) | > current_lr: 5e-06 | > step_time: 0.1705 (0.1359672259625208) | > loader_time: 0.0049 (0.004665051352593211)  --> TIME: 2025-02-25 10:33:53 -- STEP: 4390/36938 -- GLOBAL_STEP: 743150 | > loss_text_ce: 0.03589819371700287 (0.034623119401149385) | > loss_mel_ce: 3.0068554878234863 (nan) | > loss: 0.012074420228600502 (nan) | > current_lr: 5e-06 | > step_time: 0.1883 (0.1359987365900797) | > loader_time: 0.0049 (0.004664821613893969)  --> TIME: 2025-02-25 10:34:02 -- STEP: 4440/36938 -- GLOBAL_STEP: 743200 | > loss_text_ce: 0.031047746539115906 (0.034602537144789385) | > loss_mel_ce: 2.302929639816284 (nan) | > loss: 0.009261815808713436 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13596840645815927) | > loader_time: 0.0046 (0.0046642101562774955)  --> TIME: 2025-02-25 10:34:12 -- STEP: 4490/36938 -- GLOBAL_STEP: 743250 | > loss_text_ce: 0.03390243276953697 (0.034593047659412776) | > loss_mel_ce: 2.4761080741882324 (nan) | > loss: 0.009960359893739223 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.1359652776760624) | > loader_time: 0.0045 (0.004663529321717263)  --> TIME: 2025-02-25 10:34:21 -- STEP: 4540/36938 -- GLOBAL_STEP: 743300 | > loss_text_ce: 0.029987258836627007 (0.03460680597240218) | > loss_mel_ce: 2.5955810546875 (nan) | > loss: 0.010418922640383244 (nan) | > current_lr: 5e-06 | > step_time: 0.1086 (0.13600416530071416) | > loader_time: 0.0046 (0.0046649055333914755)  --> TIME: 2025-02-25 10:34:30 -- STEP: 4590/36938 -- GLOBAL_STEP: 743350 | > loss_text_ce: 0.03619345650076866 (0.034613788766433465) | > loss_mel_ce: 2.2783138751983643 (nan) | > loss: 0.009184553287923336 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13596051404159354) | > loader_time: 0.0046 (0.004665301827823418)  --> TIME: 2025-02-25 10:34:40 -- STEP: 4640/36938 -- GLOBAL_STEP: 743400 | > loss_text_ce: 0.030704142525792122 (0.03463157163390993) | > loss_mel_ce: 2.746786594390869 (nan) | > loss: 0.011021790094673634 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1359486900013072) | > loader_time: 0.0046 (0.004665108795823726)  --> TIME: 2025-02-25 10:34:49 -- STEP: 4690/36938 -- GLOBAL_STEP: 743450 | > loss_text_ce: 0.03552708029747009 (0.034644601378303976) | > loss_mel_ce: 2.7789883613586426 (nan) | > loss: 0.011168712750077248 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13597155588267987) | > loader_time: 0.0046 (0.0046648026529405705)  --> TIME: 2025-02-25 10:34:58 -- STEP: 4740/36938 -- GLOBAL_STEP: 743500 | > loss_text_ce: 0.03511424735188484 (0.03464652235493983) | > loss_mel_ce: 2.7813363075256348 (nan) | > loss: 0.011176391504704952 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.136002735397484) | > loader_time: 0.0047 (0.004664824280557759)  --> TIME: 2025-02-25 10:35:08 -- STEP: 4790/36938 -- GLOBAL_STEP: 743550 | > loss_text_ce: 0.03627936914563179 (0.03465283404207469) | > loss_mel_ce: 3.3504037857055664 (nan) | > loss: 0.013439220376312733 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13608248831086) | > loader_time: 0.0046 (0.0046648913485023355)  --> TIME: 2025-02-25 10:35:17 -- STEP: 4840/36938 -- GLOBAL_STEP: 743600 | > loss_text_ce: 0.03709777817130089 (0.03463887154251684) | > loss_mel_ce: 2.4390852451324463 (nan) | > loss: 0.009826123714447021 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13606658740477165) | > loader_time: 0.0046 (0.004664335172038436)  --> TIME: 2025-02-25 10:35:26 -- STEP: 4890/36938 -- GLOBAL_STEP: 743650 | > loss_text_ce: 0.028560496866703033 (0.03463616348205024) | > loss_mel_ce: 2.9349589347839355 (nan) | > loss: 0.01175999827682972 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13602217794196042) | > loader_time: 0.0045 (0.004663812748493594)  --> TIME: 2025-02-25 10:35:35 -- STEP: 4940/36938 -- GLOBAL_STEP: 743700 | > loss_text_ce: 0.03229841589927673 (0.03463134407442871) | > loss_mel_ce: 2.978178024291992 (nan) | > loss: 0.011946335434913635 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13594628866867534) | > loader_time: 0.0045 (0.004663152009369392)  --> TIME: 2025-02-25 10:35:45 -- STEP: 4990/36938 -- GLOBAL_STEP: 743750 | > loss_text_ce: 0.03979833051562309 (0.0346290724749541) | > loss_mel_ce: 3.4024577140808105 (nan) | > loss: 0.013659746386110783 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13597694566111332) | > loader_time: 0.0045 (0.00466287537423786)  --> TIME: 2025-02-25 10:35:54 -- STEP: 5040/36938 -- GLOBAL_STEP: 743800 | > loss_text_ce: 0.03661741316318512 (0.034624044103954504) | > loss_mel_ce: 2.6177475452423096 (nan) | > loss: 0.010533194988965988 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.1359896758246049) | > loader_time: 0.0045 (0.004662308806464791)  --> TIME: 2025-02-25 10:36:03 -- STEP: 5090/36938 -- GLOBAL_STEP: 743850 | > loss_text_ce: 0.030937498435378075 (0.03462980608849231) | > loss_mel_ce: 1.4207637310028076 (nan) | > loss: 0.005760719534009695 (nan) | > current_lr: 5e-06 | > step_time: 0.1781 (0.13600410646914546) | > loader_time: 0.0048 (0.004661687652348072)  --> TIME: 2025-02-25 10:36:12 -- STEP: 5140/36938 -- GLOBAL_STEP: 743900 | > loss_text_ce: 0.030417928472161293 (0.03463137857706548) | > loss_mel_ce: 1.9126474857330322 (nan) | > loss: 0.007710577454417944 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13599925852935188) | > loader_time: 0.0053 (0.004661527728292271)  --> TIME: 2025-02-25 10:36:22 -- STEP: 5190/36938 -- GLOBAL_STEP: 743950 | > loss_text_ce: 0.03275114297866821 (0.03463689188539325) | > loss_mel_ce: 3.018834114074707 (nan) | > loss: 0.012109465897083282 (nan) | > current_lr: 5e-06 | > step_time: 0.1864 (0.13607225464028946) | > loader_time: 0.005 (0.0046617557547685064)  --> TIME: 2025-02-25 10:36:31 -- STEP: 5240/36938 -- GLOBAL_STEP: 744000 | > loss_text_ce: 0.03302836790680885 (0.03463786832671899) | > loss_mel_ce: 3.5373620986938477 (nan) | > loss: 0.014168216846883297 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.136067523028105) | > loader_time: 0.0048 (0.004661473836607608)  --> TIME: 2025-02-25 10:36:41 -- STEP: 5290/36938 -- GLOBAL_STEP: 744050 | > loss_text_ce: 0.03828049451112747 (0.03465296200481868) | > loss_mel_ce: 3.1622397899627686 (nan) | > loss: 0.012700477614998817 (nan) | > current_lr: 5e-06 | > step_time: 0.1457 (0.13604839573735844) | > loader_time: 0.0046 (0.004660784815119436)  --> TIME: 2025-02-25 10:36:50 -- STEP: 5340/36938 -- GLOBAL_STEP: 744100 | > loss_text_ce: 0.03507029265165329 (0.03466792191197388) | > loss_mel_ce: 2.930346727371216 (nan) | > loss: 0.011767528019845486 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13607830679818542) | > loader_time: 0.0046 (0.00466055954886733)  --> TIME: 2025-02-25 10:36:59 -- STEP: 5390/36938 -- GLOBAL_STEP: 744150 | > loss_text_ce: 0.021609775722026825 (0.034674739520582115) | > loss_mel_ce: 2.523052453994751 (nan) | > loss: 0.010097866877913475 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13607363532779448) | > loader_time: 0.0047 (0.004660090179301811)  --> TIME: 2025-02-25 10:37:09 -- STEP: 5440/36938 -- GLOBAL_STEP: 744200 | > loss_text_ce: 0.03043847717344761 (0.03466442080526201) | > loss_mel_ce: 2.8138434886932373 (nan) | > loss: 0.011286833323538303 (nan) | > current_lr: 5e-06 | > step_time: 0.1513 (0.13606552841032318) | > loader_time: 0.0046 (0.00465985111453954)  --> TIME: 2025-02-25 10:37:18 -- STEP: 5490/36938 -- GLOBAL_STEP: 744250 | > loss_text_ce: 0.040332820266485214 (0.034657412689884884) | > loss_mel_ce: 3.2843329906463623 (nan) | > loss: 0.013193119317293167 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13605588641974511) | > loader_time: 0.0046 (0.004659451465571943)  --> TIME: 2025-02-25 10:37:27 -- STEP: 5540/36938 -- GLOBAL_STEP: 744300 | > loss_text_ce: 0.02950412593781948 (0.03466539405713305) | > loss_mel_ce: 1.816575288772583 (nan) | > loss: 0.007325712591409683 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13606257387016732) | > loader_time: 0.0045 (0.004659233712977874)  --> TIME: 2025-02-25 10:37:37 -- STEP: 5590/36938 -- GLOBAL_STEP: 744350 | > loss_text_ce: 0.03231656551361084 (0.03465680633396682) | > loss_mel_ce: 3.0481207370758057 (nan) | > loss: 0.012223958037793636 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13608194809267132) | > loader_time: 0.0045 (0.004658956263274322)  --> TIME: 2025-02-25 10:37:46 -- STEP: 5640/36938 -- GLOBAL_STEP: 744400 | > loss_text_ce: 0.032546497881412506 (0.03466790843712071) | > loss_mel_ce: 1.4607980251312256 (nan) | > loss: 0.005925970617681742 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1361149322902065) | > loader_time: 0.0044 (0.004658768320759979)  --> TIME: 2025-02-25 10:37:56 -- STEP: 5690/36938 -- GLOBAL_STEP: 744450 | > loss_text_ce: 0.029484834522008896 (0.03467332379440719) | > loss_mel_ce: 2.335782766342163 (nan) | > loss: 0.009385982528328896 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13612079536558727) | > loader_time: 0.0046 (0.004658403673155231)  --> TIME: 2025-02-25 10:38:05 -- STEP: 5740/36938 -- GLOBAL_STEP: 744500 | > loss_text_ce: 0.043914325535297394 (0.03467109779411933) | > loss_mel_ce: 2.9628214836120605 (nan) | > loss: 0.011931492015719414 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.1360890982873769) | > loader_time: 0.0045 (0.004658012730734686)  --> TIME: 2025-02-25 10:38:14 -- STEP: 5790/36938 -- GLOBAL_STEP: 744550 | > loss_text_ce: 0.0293718408793211 (0.03467870952066075) | > loss_mel_ce: 2.602358818054199 (nan) | > loss: 0.010443375445902348 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13609226931768031) | > loader_time: 0.0046 (0.004657546761534167)  --> TIME: 2025-02-25 10:38:23 -- STEP: 5840/36938 -- GLOBAL_STEP: 744600 | > loss_text_ce: 0.02623269520699978 (0.03468161781672514) | > loss_mel_ce: 2.866842746734619 (nan) | > loss: 0.01148045901209116 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.1360574156045921) | > loader_time: 0.0045 (0.004657278607969413)  --> TIME: 2025-02-25 10:38:32 -- STEP: 5890/36938 -- GLOBAL_STEP: 744650 | > loss_text_ce: 0.03368251770734787 (0.03466765145725348) | > loss_mel_ce: 2.5763626098632812 (nan) | > loss: 0.010357323102653027 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13606298487943452) | > loader_time: 0.0047 (0.004657093616414352)  --> TIME: 2025-02-25 10:38:41 -- STEP: 5940/36938 -- GLOBAL_STEP: 744700 | > loss_text_ce: 0.040145549923181534 (0.03468219317505227) | > loss_mel_ce: 1.5458838939666748 (nan) | > loss: 0.006293768063187599 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1360470799886023) | > loader_time: 0.0046 (0.0046568908675351135)  --> TIME: 2025-02-25 10:38:51 -- STEP: 5990/36938 -- GLOBAL_STEP: 744750 | > loss_text_ce: 0.03796187788248062 (0.034688236197932446) | > loss_mel_ce: 2.9678046703338623 (nan) | > loss: 0.011927645653486252 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.1360697731947867) | > loader_time: 0.0046 (0.004656690150151071)  --> TIME: 2025-02-25 10:39:00 -- STEP: 6040/36938 -- GLOBAL_STEP: 744800 | > loss_text_ce: 0.03754213824868202 (0.03468536352538284) | > loss_mel_ce: 3.3323919773101807 (nan) | > loss: 0.01337275467813015 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.13607572621067704) | > loader_time: 0.0047 (0.004656445664285827)  --> TIME: 2025-02-25 10:39:09 -- STEP: 6090/36938 -- GLOBAL_STEP: 744850 | > loss_text_ce: 0.03640923276543617 (0.034685866140618235) | > loss_mel_ce: 1.9565800428390503 (nan) | > loss: 0.007908687926828861 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13608856549599774) | > loader_time: 0.0045 (0.004656101917398394)  --> TIME: 2025-02-25 10:39:19 -- STEP: 6140/36938 -- GLOBAL_STEP: 744900 | > loss_text_ce: 0.031181950122117996 (0.034670701002131786) | > loss_mel_ce: 2.3112988471984863 (nan) | > loss: 0.009295559488236904 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.13611269117566668) | > loader_time: 0.0048 (0.004656034960420597)  --> TIME: 2025-02-25 10:39:28 -- STEP: 6190/36938 -- GLOBAL_STEP: 744950 | > loss_text_ce: 0.03195691108703613 (0.03465757592263749) | > loss_mel_ce: 2.7470147609710693 (nan) | > loss: 0.011027665808796883 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.136098587840363) | > loader_time: 0.0046 (0.004655599247465617)  --> TIME: 2025-02-25 10:39:37 -- STEP: 6240/36938 -- GLOBAL_STEP: 745000 | > loss_text_ce: 0.03345757722854614 (0.03465437371573313) | > loss_mel_ce: 0.06610107421875 (nan) | > loss: 0.00039507405017502606 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13604427087001394) | > loader_time: 0.0047 (0.004655068157575073)  --> TIME: 2025-02-25 10:39:46 -- STEP: 6290/36938 -- GLOBAL_STEP: 745050 | > loss_text_ce: 0.024676131084561348 (0.03465349577583477) | > loss_mel_ce: 2.899008274078369 (nan) | > loss: 0.011601923033595085 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13604888836416495) | > loader_time: 0.0045 (0.004654825662391556)  --> TIME: 2025-02-25 10:39:56 -- STEP: 6340/36938 -- GLOBAL_STEP: 745100 | > loss_text_ce: 0.011428650468587875 (0.0346504309545168) | > loss_mel_ce: 1.851339340209961 (nan) | > loss: 0.007391936611384153 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1360667161008921) | > loader_time: 0.0044 (0.004654396633244464)  --> TIME: 2025-02-25 10:40:05 -- STEP: 6390/36938 -- GLOBAL_STEP: 745150 | > loss_text_ce: 0.039631832391023636 (0.03465275296778159) | > loss_mel_ce: 2.4541878700256348 (nan) | > loss: 0.009896110743284225 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13608362476963853) | > loader_time: 0.0046 (0.0046539928991469965)  --> TIME: 2025-02-25 10:40:14 -- STEP: 6440/36938 -- GLOBAL_STEP: 745200 | > loss_text_ce: 0.033816199749708176 (0.034647933947448806) | > loss_mel_ce: 3.3523690700531006 (nan) | > loss: 0.013437243178486824 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.13610721382295196) | > loader_time: 0.0047 (0.004653840109428262)  --> TIME: 2025-02-25 10:40:24 -- STEP: 6490/36938 -- GLOBAL_STEP: 745250 | > loss_text_ce: 0.03842959925532341 (0.034666825997541256) | > loss_mel_ce: 3.3952574729919434 (nan) | > loss: 0.013625742867588997 (nan) | > current_lr: 5e-06 | > step_time: 0.1884 (0.1361065795866117) | > loader_time: 0.0048 (0.004653427560450665)  --> TIME: 2025-02-25 10:40:33 -- STEP: 6540/36938 -- GLOBAL_STEP: 745300 | > loss_text_ce: 0.03424838185310364 (0.03466283381939362) | > loss_mel_ce: 3.556079387664795 (nan) | > loss: 0.014247332699596882 (nan) | > current_lr: 5e-06 | > step_time: 0.1618 (0.13611884977474928) | > loader_time: 0.0046 (0.004653129701585216)  --> TIME: 2025-02-25 10:40:42 -- STEP: 6590/36938 -- GLOBAL_STEP: 745350 | > loss_text_ce: 0.03218197822570801 (0.03466691886949629) | > loss_mel_ce: 2.3259973526000977 (nan) | > loss: 0.009357854723930359 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13613136892796346) | > loader_time: 0.0045 (0.004652729888244538)  --> TIME: 2025-02-25 10:40:52 -- STEP: 6640/36938 -- GLOBAL_STEP: 745400 | > loss_text_ce: 0.02965353988111019 (0.03466000701258715) | > loss_mel_ce: 2.642430067062378 (nan) | > loss: 0.010603507049381733 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13614005004785215) | > loader_time: 0.0046 (0.004652358501790518)  --> TIME: 2025-02-25 10:41:01 -- STEP: 6690/36938 -- GLOBAL_STEP: 745450 | > loss_text_ce: 0.029940027743577957 (0.03466120094982471) | > loss_mel_ce: 2.518841505050659 (nan) | > loss: 0.010114213451743126 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13614646244476797) | > loader_time: 0.0046 (0.004652011162853381)  --> TIME: 2025-02-25 10:41:10 -- STEP: 6740/36938 -- GLOBAL_STEP: 745500 | > loss_text_ce: 0.022918909788131714 (0.034659699475370565) | > loss_mel_ce: 1.1088440418243408 (nan) | > loss: 0.004491123370826244 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13614425347891404) | > loader_time: 0.0046 (0.00465217421953331)  --> TIME: 2025-02-25 10:41:20 -- STEP: 6790/36938 -- GLOBAL_STEP: 745550 | > loss_text_ce: 0.025336844846606255 (0.03465560174482669) | > loss_mel_ce: 2.19325590133667 (nan) | > loss: 0.008803939446806908 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.1361716699881132) | > loader_time: 0.0046 (0.004652484913462211)  --> TIME: 2025-02-25 10:41:29 -- STEP: 6840/36938 -- GLOBAL_STEP: 745600 | > loss_text_ce: 0.03402423486113548 (0.03465226268251452) | > loss_mel_ce: 3.0859251022338867 (nan) | > loss: 0.0123807517811656 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.1361635304333878) | > loader_time: 0.0046 (0.00465259060525057)  --> TIME: 2025-02-25 10:41:38 -- STEP: 6890/36938 -- GLOBAL_STEP: 745650 | > loss_text_ce: 0.029925845563411713 (0.03465604451140295) | > loss_mel_ce: 3.1166677474975586 (nan) | > loss: 0.012486482970416546 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13616042074859658) | > loader_time: 0.0046 (0.00465238699892262)  --> TIME: 2025-02-25 10:41:47 -- STEP: 6940/36938 -- GLOBAL_STEP: 745700 | > loss_text_ce: 0.03723523020744324 (0.03466523421072115) | > loss_mel_ce: 2.2047348022460938 (nan) | > loss: 0.00889670755714178 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.1361393350345609) | > loader_time: 0.0046 (0.004652467687810181)  --> TIME: 2025-02-25 10:41:56 -- STEP: 6990/36938 -- GLOBAL_STEP: 745750 | > loss_text_ce: 0.02370183914899826 (0.034662646968024284) | > loss_mel_ce: 3.191875696182251 (nan) | > loss: 0.012760229408740997 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13610867614909908) | > loader_time: 0.0045 (0.004652118375884608)  --> TIME: 2025-02-25 10:42:06 -- STEP: 7040/36938 -- GLOBAL_STEP: 745800 | > loss_text_ce: 0.03045554645359516 (0.0346586811206759) | > loss_mel_ce: 3.1493306159973145 (nan) | > loss: 0.012618199922144413 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.13614124086770238) | > loader_time: 0.0046 (0.004652065004814747)  --> TIME: 2025-02-25 10:42:15 -- STEP: 7090/36938 -- GLOBAL_STEP: 745850 | > loss_text_ce: 0.027973409742116928 (0.03466058341570842) | > loss_mel_ce: 2.72973895072937 (nan) | > loss: 0.010943303816020489 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13612686471306995) | > loader_time: 0.0046 (0.004651874917518597)  --> TIME: 2025-02-25 10:42:24 -- STEP: 7140/36938 -- GLOBAL_STEP: 745900 | > loss_text_ce: 0.045269593596458435 (0.03465282277866721) | > loss_mel_ce: 2.6099507808685303 (nan) | > loss: 0.010536588728427887 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13612692910416224) | > loader_time: 0.0046 (0.0046520030131193026)  --> TIME: 2025-02-25 10:42:33 -- STEP: 7190/36938 -- GLOBAL_STEP: 745950 | > loss_text_ce: 0.04237930104136467 (0.03464484857136234) | > loss_mel_ce: 2.5023722648620605 (nan) | > loss: 0.010098221711814404 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13609415662471155) | > loader_time: 0.0045 (0.0046515993017480425)  --> TIME: 2025-02-25 10:42:42 -- STEP: 7240/36938 -- GLOBAL_STEP: 746000 | > loss_text_ce: 0.03104436956346035 (0.03463302319547813) | > loss_mel_ce: 0.012786865234375 (nan) | > loss: 0.00017393348389305174 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13609124155334446) | > loader_time: 0.0047 (0.004651356533746026)  --> TIME: 2025-02-25 10:42:52 -- STEP: 7290/36938 -- GLOBAL_STEP: 746050 | > loss_text_ce: 0.04144294559955597 (0.03463234487289901) | > loss_mel_ce: 2.953031539916992 (nan) | > loss: 0.011882835999131203 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.1361164392921342) | > loader_time: 0.0046 (0.004651230581830387)  --> TIME: 2025-02-25 10:43:01 -- STEP: 7340/36938 -- GLOBAL_STEP: 746100 | > loss_text_ce: 0.03827203810214996 (0.034627761263324223) | > loss_mel_ce: 3.1012308597564697 (nan) | > loss: 0.012458345852792263 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13610327087890955) | > loader_time: 0.0045 (0.004651661919637979)  --> TIME: 2025-02-25 10:43:11 -- STEP: 7390/36938 -- GLOBAL_STEP: 746150 | > loss_text_ce: 0.037009723484516144 (0.03462394566309211) | > loss_mel_ce: 2.2499871253967285 (nan) | > loss: 0.009075384587049484 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13611355905442582) | > loader_time: 0.0045 (0.004651209175667349)  --> TIME: 2025-02-25 10:43:20 -- STEP: 7440/36938 -- GLOBAL_STEP: 746200 | > loss_text_ce: 0.043440040200948715 (0.034630041634689024) | > loss_mel_ce: 3.485408067703247 (nan) | > loss: 0.014003366231918335 (nan) | > current_lr: 5e-06 | > step_time: 0.1621 (0.13611553176756902) | > loader_time: 0.0047 (0.004651101782757741)  --> TIME: 2025-02-25 10:43:29 -- STEP: 7490/36938 -- GLOBAL_STEP: 746250 | > loss_text_ce: 0.024798668920993805 (0.03463060116091964) | > loss_mel_ce: 2.472654104232788 (nan) | > loss: 0.009910527616739273 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13611656463989816) | > loader_time: 0.0045 (0.004650952914687431)  --> TIME: 2025-02-25 10:43:38 -- STEP: 7540/36938 -- GLOBAL_STEP: 746300 | > loss_text_ce: 0.029826849699020386 (0.03462183467082842) | > loss_mel_ce: 3.3286259174346924 (nan) | > loss: 0.01332719437777996 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13611001636368583) | > loader_time: 0.0046 (0.004650956249995945)  --> TIME: 2025-02-25 10:43:48 -- STEP: 7590/36938 -- GLOBAL_STEP: 746350 | > loss_text_ce: 0.03866579011082649 (0.03462237800546338) | > loss_mel_ce: 1.82666015625 (nan) | > loss: 0.007402087561786175 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.1361263794075993) | > loader_time: 0.0047 (0.004650807317851867)  --> TIME: 2025-02-25 10:43:57 -- STEP: 7640/36938 -- GLOBAL_STEP: 746400 | > loss_text_ce: 0.032754186540842056 (0.0346228325689524) | > loss_mel_ce: 3.07521653175354 (nan) | > loss: 0.0123332180082798 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.1361118569736087) | > loader_time: 0.0046 (0.004650928493569652)  --> TIME: 2025-02-25 10:44:07 -- STEP: 7690/36938 -- GLOBAL_STEP: 746450 | > loss_text_ce: 0.02960592694580555 (0.03461647132091878) | > loss_mel_ce: 3.272940158843994 (nan) | > loss: 0.013105342164635658 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13611780174066926) | > loader_time: 0.0045 (0.004650698216590824)  --> TIME: 2025-02-25 10:44:16 -- STEP: 7740/36938 -- GLOBAL_STEP: 746500 | > loss_text_ce: 0.03419296443462372 (0.034619547516584095) | > loss_mel_ce: 2.815857410430908 (nan) | > loss: 0.011309724301099777 (nan) | > current_lr: 5e-06 | > step_time: 0.179 (0.13612300348528344) | > loader_time: 0.0048 (0.00465049620438607)  --> TIME: 2025-02-25 10:44:25 -- STEP: 7790/36938 -- GLOBAL_STEP: 746550 | > loss_text_ce: 0.03134946525096893 (0.034622716033216774) | > loss_mel_ce: 2.770305633544922 (nan) | > loss: 0.011117679066956043 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1361185630876696) | > loader_time: 0.0045 (0.004650337613439981)  --> TIME: 2025-02-25 10:44:34 -- STEP: 7840/36938 -- GLOBAL_STEP: 746600 | > loss_text_ce: 0.02950623258948326 (0.03462856920752014) | > loss_mel_ce: 2.843515634536743 (nan) | > loss: 0.011400881223380566 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13612208868167894) | > loader_time: 0.0046 (0.004650322150210938)  --> TIME: 2025-02-25 10:44:44 -- STEP: 7890/36938 -- GLOBAL_STEP: 746650 | > loss_text_ce: 0.027726415544748306 (0.034638522881353544) | > loss_mel_ce: 0.068603515625 (nan) | > loss: 0.0003822616417892277 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13611933650172764) | > loader_time: 0.0046 (0.004650301897027186)  --> TIME: 2025-02-25 10:44:53 -- STEP: 7940/36938 -- GLOBAL_STEP: 746700 | > loss_text_ce: 0.03702230006456375 (0.03463284839542882) | > loss_mel_ce: 3.131805419921875 (nan) | > loss: 0.012574713677167892 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.13614274017156275) | > loader_time: 0.0047 (0.004650566319674622)  --> TIME: 2025-02-25 10:45:03 -- STEP: 7990/36938 -- GLOBAL_STEP: 746750 | > loss_text_ce: 0.04191194102168083 (0.03463692376996221) | > loss_mel_ce: 2.6885430812835693 (nan) | > loss: 0.010835139080882072 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1361504601596747) | > loader_time: 0.0046 (0.004650773154629926)  --> TIME: 2025-02-25 10:45:12 -- STEP: 8040/36938 -- GLOBAL_STEP: 746800 | > loss_text_ce: 0.029547065496444702 (0.034637861928569) | > loss_mel_ce: 3.4517624378204346 (nan) | > loss: 0.013814720325171947 (nan) | > current_lr: 5e-06 | > step_time: 0.1438 (0.1361567189740903) | > loader_time: 0.0046 (0.004650602204289595)  --> TIME: 2025-02-25 10:45:21 -- STEP: 8090/36938 -- GLOBAL_STEP: 746850 | > loss_text_ce: 0.03043230250477791 (0.03462524588343004) | > loss_mel_ce: 2.769399642944336 (nan) | > loss: 0.01111044455319643 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.13614415712792788) | > loader_time: 0.0046 (0.0046502436636699515)  --> TIME: 2025-02-25 10:45:31 -- STEP: 8140/36938 -- GLOBAL_STEP: 746900 | > loss_text_ce: 0.03324764966964722 (0.03462915618043553) | > loss_mel_ce: 3.330521821975708 (nan) | > loss: 0.0133482925593853 (nan) | > current_lr: 5e-06 | > step_time: 0.1822 (0.13618164856252324) | > loader_time: 0.0049 (0.004650331275761852)  --> TIME: 2025-02-25 10:45:40 -- STEP: 8190/36938 -- GLOBAL_STEP: 746950 | > loss_text_ce: 0.03666137531399727 (0.0346242876924684) | > loss_mel_ce: 2.7008814811706543 (nan) | > loss: 0.010863265953958035 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.1362222593782591) | > loader_time: 0.0046 (0.004650180855076821)  --> TIME: 2025-02-25 10:45:50 -- STEP: 8240/36938 -- GLOBAL_STEP: 747000 | > loss_text_ce: 0.02839016169309616 (0.03461849824875796) | > loss_mel_ce: 2.0098955631256104 (nan) | > loss: 0.008088435977697372 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13623342814954792) | > loader_time: 0.0047 (0.004650211826111499)  --> TIME: 2025-02-25 10:45:59 -- STEP: 8290/36938 -- GLOBAL_STEP: 747050 | > loss_text_ce: 0.02926497720181942 (0.034611090576777276) | > loss_mel_ce: 2.737516403198242 (nan) | > loss: 0.01097929198294878 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.1362354432429283) | > loader_time: 0.0045 (0.004650413601763709)  --> TIME: 2025-02-25 10:46:09 -- STEP: 8340/36938 -- GLOBAL_STEP: 747100 | > loss_text_ce: 0.030444879084825516 (0.03461215572969934) | > loss_mel_ce: 3.2255685329437256 (nan) | > loss: 0.012920688837766647 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13625556679366638) | > loader_time: 0.0045 (0.004650421451321601)  --> TIME: 2025-02-25 10:46:18 -- STEP: 8390/36938 -- GLOBAL_STEP: 747150 | > loss_text_ce: 0.034363679587841034 (0.03460282038273439) | > loss_mel_ce: 1.954176425933838 (nan) | > loss: 0.007891032844781876 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13629234519135588) | > loader_time: 0.0046 (0.004650346485451654)  --> TIME: 2025-02-25 10:46:28 -- STEP: 8440/36938 -- GLOBAL_STEP: 747200 | > loss_text_ce: 0.0381791926920414 (0.03459805922165642) | > loss_mel_ce: 2.4556076526641846 (nan) | > loss: 0.00989598035812378 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.136286306465972) | > loader_time: 0.0045 (0.004650171277647327)  --> TIME: 2025-02-25 10:46:37 -- STEP: 8490/36938 -- GLOBAL_STEP: 747250 | > loss_text_ce: 0.03493981435894966 (0.03459995732016349) | > loss_mel_ce: 1.9358267784118652 (nan) | > loss: 0.00782050285488367 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13629479352099644) | > loader_time: 0.0045 (0.004650012960984366)  --> TIME: 2025-02-25 10:46:46 -- STEP: 8540/36938 -- GLOBAL_STEP: 747300 | > loss_text_ce: 0.02630169503390789 (0.03459519251463481) | > loss_mel_ce: 0.0186767578125 (nan) | > loss: 0.00017848594870883971 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1363119977419499) | > loader_time: 0.0045 (0.004649752113244007)  --> TIME: 2025-02-25 10:46:56 -- STEP: 8590/36938 -- GLOBAL_STEP: 747350 | > loss_text_ce: 0.04682515189051628 (0.034594024253122295) | > loss_mel_ce: 3.5907633304595947 (nan) | > loss: 0.014434875920414925 (nan) | > current_lr: 5e-06 | > step_time: 0.1526 (0.13633707588293553) | > loader_time: 0.0046 (0.0046496012435664315)  --> TIME: 2025-02-25 10:47:05 -- STEP: 8640/36938 -- GLOBAL_STEP: 747400 | > loss_text_ce: 0.03333347663283348 (0.03458889245521282) | > loss_mel_ce: 2.0307400226593018 (nan) | > loss: 0.008190768770873547 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13634300336793664) | > loader_time: 0.0047 (0.0046497832569811015)  --> TIME: 2025-02-25 10:47:15 -- STEP: 8690/36938 -- GLOBAL_STEP: 747450 | > loss_text_ce: 0.034682102501392365 (0.03458471400202423) | > loss_mel_ce: 3.2548177242279053 (nan) | > loss: 0.013053570874035358 (nan) | > current_lr: 5e-06 | > step_time: 0.186 (0.13635010612299078) | > loader_time: 0.0048 (0.004649905258547444)  --> TIME: 2025-02-25 10:47:24 -- STEP: 8740/36938 -- GLOBAL_STEP: 747500 | > loss_text_ce: 0.037725720554590225 (0.0345775451287372) | > loss_mel_ce: 2.9039502143859863 (nan) | > loss: 0.011673317290842533 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13635332046984586) | > loader_time: 0.0047 (0.004650038139770992)  --> TIME: 2025-02-25 10:47:33 -- STEP: 8790/36938 -- GLOBAL_STEP: 747550 | > loss_text_ce: 0.032043762505054474 (0.034565704662259625) | > loss_mel_ce: 3.1726577281951904 (nan) | > loss: 0.012717070057988167 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.1363478043787313) | > loader_time: 0.0046 (0.00464999499445753)  --> TIME: 2025-02-25 10:47:43 -- STEP: 8840/36938 -- GLOBAL_STEP: 747600 | > loss_text_ce: 0.03455190360546112 (0.03456866592642906) | > loss_mel_ce: 2.684093952178955 (nan) | > loss: 0.010788277722895145 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.13636259495942343) | > loader_time: 0.0047 (0.004649962181419264)  --> TIME: 2025-02-25 10:47:52 -- STEP: 8890/36938 -- GLOBAL_STEP: 747650 | > loss_text_ce: 0.023200038820505142 (0.03457162202345332) | > loss_mel_ce: 2.615461587905884 (nan) | > loss: 0.010470880195498466 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13637051523335236) | > loader_time: 0.0045 (0.004649906807460715)  --> TIME: 2025-02-25 10:48:02 -- STEP: 8940/36938 -- GLOBAL_STEP: 747700 | > loss_text_ce: 0.04087303951382637 (0.034573309882637834) | > loss_mel_ce: 3.136049270629883 (nan) | > loss: 0.01260683499276638 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13639528996725783) | > loader_time: 0.0046 (0.004649847919242222)  --> TIME: 2025-02-25 10:48:11 -- STEP: 8990/36938 -- GLOBAL_STEP: 747750 | > loss_text_ce: 0.036319609731435776 (0.03457663741016623) | > loss_mel_ce: 3.157750368118286 (nan) | > loss: 0.012674881145358086 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.1363984183554393) | > loader_time: 0.0046 (0.00464967251884791)  --> TIME: 2025-02-25 10:48:20 -- STEP: 9040/36938 -- GLOBAL_STEP: 747800 | > loss_text_ce: 0.0311890859156847 (0.03456330550154597) | > loss_mel_ce: 3.034457206726074 (nan) | > loss: 0.01216526422649622 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13638673182082425) | > loader_time: 0.0045 (0.004649857820662772)  --> TIME: 2025-02-25 10:48:29 -- STEP: 9090/36938 -- GLOBAL_STEP: 747850 | > loss_text_ce: 0.024170875549316406 (0.0345631289264092) | > loss_mel_ce: 0.0643717423081398 (nan) | > loss: 0.00035135960206389427 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13633980415310706) | > loader_time: 0.0045 (0.004649745040994266)  --> TIME: 2025-02-25 10:48:38 -- STEP: 9140/36938 -- GLOBAL_STEP: 747900 | > loss_text_ce: 0.03988794609904289 (0.0345553254040216) | > loss_mel_ce: 2.6689188480377197 (nan) | > loss: 0.01074923388659954 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13634784552931037) | > loader_time: 0.0046 (0.004649837220523753)  --> TIME: 2025-02-25 10:48:48 -- STEP: 9190/36938 -- GLOBAL_STEP: 747950 | > loss_text_ce: 0.03815507888793945 (0.03454583369365725) | > loss_mel_ce: 2.1770737171173096 (nan) | > loss: 0.008790590800344944 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13634400894385076) | > loader_time: 0.0045 (0.0046495567078948365)  --> TIME: 2025-02-25 10:48:57 -- STEP: 9240/36938 -- GLOBAL_STEP: 748000 | > loss_text_ce: 0.030672874301671982 (0.03454314911439704) | > loss_mel_ce: 3.1653919219970703 (nan) | > loss: 0.012682797387242317 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.136373923918902) | > loader_time: 0.0046 (0.004649806538701571)  --> TIME: 2025-02-25 10:49:07 -- STEP: 9290/36938 -- GLOBAL_STEP: 748050 | > loss_text_ce: 0.030655350536108017 (0.034542268635334314) | > loss_mel_ce: 2.6692850589752197 (nan) | > loss: 0.010714050382375717 (nan) | > current_lr: 5e-06 | > step_time: 0.1512 (0.13638227281067455) | > loader_time: 0.0046 (0.004649575097953538)  --> TIME: 2025-02-25 10:49:16 -- STEP: 9340/36938 -- GLOBAL_STEP: 748100 | > loss_text_ce: 0.026812605559825897 (0.0345384652245735) | > loss_mel_ce: 1.7591241598129272 (nan) | > loss: 0.007087050937116146 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13639359428234266) | > loader_time: 0.0046 (0.004649444769996134)  --> TIME: 2025-02-25 10:49:25 -- STEP: 9390/36938 -- GLOBAL_STEP: 748150 | > loss_text_ce: 0.03795766830444336 (0.03453877403086441) | > loss_mel_ce: 0.1076311394572258 (nan) | > loss: 0.0005777334445156157 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13639905886909112) | > loader_time: 0.0046 (0.004649210052368357)  --> TIME: 2025-02-25 10:49:35 -- STEP: 9440/36938 -- GLOBAL_STEP: 748200 | > loss_text_ce: 0.029587160795927048 (0.03453812650108911) | > loss_mel_ce: 1.9558885097503662 (nan) | > loss: 0.007878872565925121 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13639106927281752) | > loader_time: 0.0045 (0.004649018382622023)  --> TIME: 2025-02-25 10:49:44 -- STEP: 9490/36938 -- GLOBAL_STEP: 748250 | > loss_text_ce: 0.030238138511776924 (0.0345313579646834) | > loss_mel_ce: 2.633505344390869 (nan) | > loss: 0.010570411570370197 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13637651836909812) | > loader_time: 0.0046 (0.004648849182812257)  --> TIME: 2025-02-25 10:49:53 -- STEP: 9540/36938 -- GLOBAL_STEP: 748300 | > loss_text_ce: 0.03694690763950348 (0.034530228119835596) | > loss_mel_ce: 3.7443172931671143 (nan) | > loss: 0.015005017630755901 (nan) | > current_lr: 5e-06 | > step_time: 0.1996 (0.13637898600576348) | > loader_time: 0.0048 (0.004648613030055784)  --> TIME: 2025-02-25 10:50:02 -- STEP: 9590/36938 -- GLOBAL_STEP: 748350 | > loss_text_ce: 0.03581363707780838 (0.03452758678313782) | > loss_mel_ce: 3.2674529552459717 (nan) | > loss: 0.013108201324939728 (nan) | > current_lr: 5e-06 | > step_time: 0.1718 (0.13635354469665317) | > loader_time: 0.0062 (0.004648746464622901)  --> TIME: 2025-02-25 10:50:11 -- STEP: 9640/36938 -- GLOBAL_STEP: 748400 | > loss_text_ce: 0.029711784794926643 (0.034523445251880884) | > loss_mel_ce: 3.180713176727295 (nan) | > loss: 0.012739782221615314 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13634122795583842) | > loader_time: 0.0046 (0.004648428537044284)  --> TIME: 2025-02-25 10:50:21 -- STEP: 9690/36938 -- GLOBAL_STEP: 748450 | > loss_text_ce: 0.025031276047229767 (0.034515615165525815) | > loss_mel_ce: 2.046874761581421 (nan) | > loss: 0.008221849799156189 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.13634639384207753) | > loader_time: 0.0047 (0.0046484793917928495)  --> TIME: 2025-02-25 10:50:30 -- STEP: 9740/36938 -- GLOBAL_STEP: 748500 | > loss_text_ce: 0.027625037357211113 (0.03450948494587867) | > loss_mel_ce: 3.095710277557373 (nan) | > loss: 0.012394188903272152 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.1363509126022859) | > loader_time: 0.0046 (0.0046484060845580575)  --> TIME: 2025-02-25 10:50:39 -- STEP: 9790/36938 -- GLOBAL_STEP: 748550 | > loss_text_ce: 0.029011573642492294 (0.03449882226040067) | > loss_mel_ce: 3.1857733726501465 (nan) | > loss: 0.012757083401083946 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13635089049178592) | > loader_time: 0.0046 (0.004648156282972376)  --> TIME: 2025-02-25 10:50:49 -- STEP: 9840/36938 -- GLOBAL_STEP: 748600 | > loss_text_ce: 0.033274225890636444 (0.03449936808518522) | > loss_mel_ce: 3.096686363220215 (nan) | > loss: 0.012420479208230972 (nan) | > current_lr: 5e-06 | > step_time: 0.1664 (0.13636117870730144) | > loader_time: 0.0047 (0.004648193547396141)  --> TIME: 2025-02-25 10:50:58 -- STEP: 9890/36938 -- GLOBAL_STEP: 748650 | > loss_text_ce: 0.04240240529179573 (0.03450244305725045) | > loss_mel_ce: 3.3857221603393555 (nan) | > loss: 0.01360367052257061 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13635593309199992) | > loader_time: 0.0046 (0.0046479150667952185)  --> TIME: 2025-02-25 10:51:07 -- STEP: 9940/36938 -- GLOBAL_STEP: 748700 | > loss_text_ce: 0.025197142735123634 (0.03449695504830787) | > loss_mel_ce: 2.5704867839813232 (nan) | > loss: 0.010300333611667156 (nan) | > current_lr: 5e-06 | > step_time: 0.1266 (0.13636329857636276) | > loader_time: 0.0045 (0.004647964083212948)  --> TIME: 2025-02-25 10:51:16 -- STEP: 9990/36938 -- GLOBAL_STEP: 748750 | > loss_text_ce: 0.03690049424767494 (0.0344923642263547) | > loss_mel_ce: 2.811739206314087 (nan) | > loss: 0.011304126121103764 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.1363496651281944) | > loader_time: 0.0046 (0.004647588204812465)  --> TIME: 2025-02-25 10:51:26 -- STEP: 10040/36938 -- GLOBAL_STEP: 748800 | > loss_text_ce: 0.02880486100912094 (0.03448785448918094) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13635650324631507) | > loader_time: 0.0046 (0.004647640165579733)  --> TIME: 2025-02-25 10:51:35 -- STEP: 10090/36938 -- GLOBAL_STEP: 748850 | > loss_text_ce: 0.035371437668800354 (0.03448028170953183) | > loss_mel_ce: 3.3404860496520996 (nan) | > loss: 0.013396261259913445 (nan) | > current_lr: 5e-06 | > step_time: 0.169 (0.13632506927248278) | > loader_time: 0.0048 (0.004647843617986507)  --> TIME: 2025-02-25 10:51:44 -- STEP: 10140/36938 -- GLOBAL_STEP: 748900 | > loss_text_ce: 0.04086517170071602 (0.03448254538946132) | > loss_mel_ce: 2.5036158561706543 (nan) | > loss: 0.010097147896885872 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13632749784627624) | > loader_time: 0.0045 (0.00464767716104931)  --> TIME: 2025-02-25 10:51:53 -- STEP: 10190/36938 -- GLOBAL_STEP: 748950 | > loss_text_ce: 0.03283347189426422 (0.03448497370740909) | > loss_mel_ce: 3.3156368732452393 (nan) | > loss: 0.013287581503391266 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13628594018994186) | > loader_time: 0.0066 (0.004647608617571068)  --> TIME: 2025-02-25 10:52:02 -- STEP: 10240/36938 -- GLOBAL_STEP: 749000 | > loss_text_ce: 0.03315752372145653 (0.03448581219290649) | > loss_mel_ce: 2.557133674621582 (nan) | > loss: 0.010278934612870216 (nan) | > current_lr: 5e-06 | > step_time: 0.1807 (0.13629473489709204) | > loader_time: 0.0049 (0.004647708777338252)  --> TIME: 2025-02-25 10:52:12 -- STEP: 10290/36938 -- GLOBAL_STEP: 749050 | > loss_text_ce: 0.04620489105582237 (0.034487286237664984) | > loss_mel_ce: 2.559537172317505 (nan) | > loss: 0.010340246371924877 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13629890866831046) | > loader_time: 0.0046 (0.0046475558748977274)  --> TIME: 2025-02-25 10:52:21 -- STEP: 10340/36938 -- GLOBAL_STEP: 749100 | > loss_text_ce: 0.03351848945021629 (0.03448943631476738) | > loss_mel_ce: 2.5849342346191406 (nan) | > loss: 0.010390685871243477 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.1362983016488169) | > loader_time: 0.0047 (0.004647476691572519)  --> TIME: 2025-02-25 10:52:30 -- STEP: 10390/36938 -- GLOBAL_STEP: 749150 | > loss_text_ce: 0.0361512117087841 (0.03448756772736176) | > loss_mel_ce: 0.7216832041740417 (nan) | > loss: 0.003007279708981514 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13627437533267536) | > loader_time: 0.0045 (0.004647216783106833)  --> TIME: 2025-02-25 10:52:39 -- STEP: 10440/36938 -- GLOBAL_STEP: 749200 | > loss_text_ce: 0.03271135315299034 (0.03448398261611141) | > loss_mel_ce: 0.14311523735523224 (nan) | > loss: 0.0006977246375754476 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13627568645495564) | > loader_time: 0.0046 (0.004647002457658897)  --> TIME: 2025-02-25 10:52:48 -- STEP: 10490/36938 -- GLOBAL_STEP: 749250 | > loss_text_ce: 0.03491690382361412 (0.03448215962278431) | > loss_mel_ce: 3.083570957183838 (nan) | > loss: 0.01237495243549347 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13627073337511306) | > loader_time: 0.0046 (0.0046467359686488685)  --> TIME: 2025-02-25 10:52:58 -- STEP: 10540/36938 -- GLOBAL_STEP: 749300 | > loss_text_ce: 0.04136492684483528 (0.03448059231907043) | > loss_mel_ce: 0.1361083984375 (nan) | > loss: 0.0007042592624202371 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13627490359420355) | > loader_time: 0.0062 (0.004646700179554246)  --> TIME: 2025-02-25 10:53:07 -- STEP: 10590/36938 -- GLOBAL_STEP: 749350 | > loss_text_ce: 0.027547422796487808 (0.03448647785803441) | > loss_mel_ce: 3.1028659343719482 (nan) | > loss: 0.012422275729477406 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1362815464972551) | > loader_time: 0.0045 (0.004646492657287706)  --> TIME: 2025-02-25 10:53:16 -- STEP: 10640/36938 -- GLOBAL_STEP: 749400 | > loss_text_ce: 0.035837579518556595 (0.03448349960757561) | > loss_mel_ce: 3.2048144340515137 (nan) | > loss: 0.01285973098129034 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.1362627349849933) | > loader_time: 0.0046 (0.0046461701393127365)  --> TIME: 2025-02-25 10:53:26 -- STEP: 10690/36938 -- GLOBAL_STEP: 749450 | > loss_text_ce: 0.031440213322639465 (0.034482802370854264) | > loss_mel_ce: 3.261465311050415 (nan) | > loss: 0.013067086227238178 (nan) | > current_lr: 5e-06 | > step_time: 0.1752 (0.1362648305768051) | > loader_time: 0.0048 (0.0046459520498762785)  --> TIME: 2025-02-25 10:53:35 -- STEP: 10740/36938 -- GLOBAL_STEP: 749500 | > loss_text_ce: 0.02838737703859806 (0.034481403767868485) | > loss_mel_ce: 2.8790698051452637 (nan) | > loss: 0.011537528596818447 (nan) | > current_lr: 5e-06 | > step_time: 0.1453 (0.1362882410569789) | > loader_time: 0.0046 (0.004645941066564341)  --> TIME: 2025-02-25 10:53:45 -- STEP: 10790/36938 -- GLOBAL_STEP: 749550 | > loss_text_ce: 0.03829961642622948 (0.03448382702598273) | > loss_mel_ce: 2.7460663318634033 (nan) | > loss: 0.011049071326851845 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.1363015821181151) | > loader_time: 0.0046 (0.004645757701686392)  --> TIME: 2025-02-25 10:53:54 -- STEP: 10840/36938 -- GLOBAL_STEP: 749600 | > loss_text_ce: 0.03043031133711338 (0.03448663310481186) | > loss_mel_ce: 3.376093864440918 (nan) | > loss: 0.01351795345544815 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13631498010835982) | > loader_time: 0.0045 (0.0046456175096800285)  --> TIME: 2025-02-25 10:54:04 -- STEP: 10890/36938 -- GLOBAL_STEP: 749650 | > loss_text_ce: 0.03864524886012077 (0.03448707917234827) | > loss_mel_ce: 2.7302019596099854 (nan) | > loss: 0.010987489484250546 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.13632979454306768) | > loader_time: 0.0048 (0.004645399635707258)  --> TIME: 2025-02-25 10:54:13 -- STEP: 10940/36938 -- GLOBAL_STEP: 749700 | > loss_text_ce: 0.038764648139476776 (0.034483416617713585) | > loss_mel_ce: 2.9006335735321045 (nan) | > loss: 0.011664279736578465 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13632283470094544) | > loader_time: 0.0046 (0.004645118220850571)  --> TIME: 2025-02-25 10:54:22 -- STEP: 10990/36938 -- GLOBAL_STEP: 749750 | > loss_text_ce: 0.03625968471169472 (0.03448168686637708) | > loss_mel_ce: 2.3132481575012207 (nan) | > loss: 0.009323444217443466 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.1363314593022689) | > loader_time: 0.0046 (0.00464495783832747)  --> TIME: 2025-02-25 10:54:31 -- STEP: 11040/36938 -- GLOBAL_STEP: 749800 | > loss_text_ce: 0.04165007546544075 (0.03447910613748188) | > loss_mel_ce: 2.0299508571624756 (nan) | > loss: 0.008220639079809189 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13633575873530443) | > loader_time: 0.0046 (0.004644869635070571)  --> TIME: 2025-02-25 10:54:41 -- STEP: 11090/36938 -- GLOBAL_STEP: 749850 | > loss_text_ce: 0.03966105356812477 (0.03447364972042718) | > loss_mel_ce: 2.517646312713623 (nan) | > loss: 0.010148046538233757 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13631633572797736) | > loader_time: 0.0046 (0.004644870048605499)  --> TIME: 2025-02-25 10:54:50 -- STEP: 11140/36938 -- GLOBAL_STEP: 749900 | > loss_text_ce: 0.04059286415576935 (0.03447977924448596) | > loss_mel_ce: 3.392069101333618 (nan) | > loss: 0.013621675781905651 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1363185391177826) | > loader_time: 0.0045 (0.004644930148681147)  --> TIME: 2025-02-25 10:54:59 -- STEP: 11190/36938 -- GLOBAL_STEP: 749950 | > loss_text_ce: 0.03638046234846115 (0.034481479705438835) | > loss_mel_ce: 2.439587354660034 (nan) | > loss: 0.009825269691646099 (nan) | > current_lr: 5e-06 | > step_time: 0.1491 (0.13630464402983747) | > loader_time: 0.0047 (0.004644834686327023)  --> TIME: 2025-02-25 10:55:08 -- STEP: 11240/36938 -- GLOBAL_STEP: 750000 | > loss_text_ce: 0.03485638275742531 (0.03448164351019372) | > loss_mel_ce: 3.775792121887207 (nan) | > loss: 0.015121622011065483 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13630060801726676) | > loader_time: 0.0047 (0.0046448111746234845) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_750000.pth  --> TIME: 2025-02-25 10:55:23 -- STEP: 11290/36938 -- GLOBAL_STEP: 750050 | > loss_text_ce: 0.03800646588206291 (0.034481301628205056) | > loss_mel_ce: 2.9625797271728516 (nan) | > loss: 0.011907089501619339 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.13628262314150275) | > loader_time: 0.0047 (0.004644753829476263)  --> TIME: 2025-02-25 10:55:33 -- STEP: 11340/36938 -- GLOBAL_STEP: 750100 | > loss_text_ce: 0.02879462204873562 (0.03448199622474321) | > loss_mel_ce: 1.6267324686050415 (nan) | > loss: 0.006569552235305309 (nan) | > current_lr: 5e-06 | > step_time: 0.1056 (0.1362799433383575) | > loader_time: 0.0044 (0.00464471982269691)  --> TIME: 2025-02-25 10:55:42 -- STEP: 11390/36938 -- GLOBAL_STEP: 750150 | > loss_text_ce: 0.03828795254230499 (0.03447804066603581) | > loss_mel_ce: 2.7958896160125732 (nan) | > loss: 0.011246737092733383 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13627625706951516) | > loader_time: 0.0045 (0.0046446220198674)  --> TIME: 2025-02-25 10:55:51 -- STEP: 11440/36938 -- GLOBAL_STEP: 750200 | > loss_text_ce: 0.032374508678913116 (0.0344782237240303) | > loss_mel_ce: 1.916832685470581 (nan) | > loss: 0.007734949700534344 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13627351676667512) | > loader_time: 0.0048 (0.0046444941650737445)  --> TIME: 2025-02-25 10:56:01 -- STEP: 11490/36938 -- GLOBAL_STEP: 750250 | > loss_text_ce: 0.021281113848090172 (0.03448492704320858) | > loss_mel_ce: 2.606707811355591 (nan) | > loss: 0.010428527370095253 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13627428796832944) | > loader_time: 0.0044 (0.0046442882821703165)  --> TIME: 2025-02-25 10:56:10 -- STEP: 11540/36938 -- GLOBAL_STEP: 750300 | > loss_text_ce: 0.029966793954372406 (0.034485275615200456) | > loss_mel_ce: 3.137997627258301 (nan) | > loss: 0.012571288272738457 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13628254715656712) | > loader_time: 0.0046 (0.004644117776814538)  --> TIME: 2025-02-25 10:56:19 -- STEP: 11590/36938 -- GLOBAL_STEP: 750350 | > loss_text_ce: 0.03617333993315697 (0.0344802827894933) | > loss_mel_ce: 3.123112916946411 (nan) | > loss: 0.0125368507578969 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13627932065514498) | > loader_time: 0.0046 (0.004643827949339615)  --> TIME: 2025-02-25 10:56:29 -- STEP: 11640/36938 -- GLOBAL_STEP: 750400 | > loss_text_ce: 0.03699536994099617 (0.03447635261697187) | > loss_mel_ce: 3.0292911529541016 (nan) | > loss: 0.01216780487447977 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.13629909913564497) | > loader_time: 0.0047 (0.004643739621663836)  --> TIME: 2025-02-25 10:56:38 -- STEP: 11690/36938 -- GLOBAL_STEP: 750450 | > loss_text_ce: 0.03125813603401184 (0.034475830841532364) | > loss_mel_ce: 3.0181682109832764 (nan) | > loss: 0.012100898660719395 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13629060244335933) | > loader_time: 0.0045 (0.004643484299972183)  --> TIME: 2025-02-25 10:56:48 -- STEP: 11740/36938 -- GLOBAL_STEP: 750500 | > loss_text_ce: 0.027087010443210602 (0.034476156780429736) | > loss_mel_ce: 1.0311509370803833 (nan) | > loss: 0.004199357237666845 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13630548297241998) | > loader_time: 0.0045 (0.004643472658511525)  --> TIME: 2025-02-25 10:56:57 -- STEP: 11790/36938 -- GLOBAL_STEP: 750550 | > loss_text_ce: 0.032145608216524124 (0.034472890035190176) | > loss_mel_ce: 3.4054040908813477 (nan) | > loss: 0.013641070574522018 (nan) | > current_lr: 5e-06 | > step_time: 0.156 (0.1362942488947799) | > loader_time: 0.0046 (0.004643490963613105)  --> TIME: 2025-02-25 10:57:06 -- STEP: 11840/36938 -- GLOBAL_STEP: 750600 | > loss_text_ce: 0.02814684994518757 (0.03447089905542261) | > loss_mel_ce: 2.4320068359375 (nan) | > loss: 0.009762514382600784 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.13629521268445116) | > loader_time: 0.0046 (0.004643795296952523)  --> TIME: 2025-02-25 10:57:16 -- STEP: 11890/36938 -- GLOBAL_STEP: 750650 | > loss_text_ce: 0.0390639565885067 (0.034474443797965326) | > loss_mel_ce: 1.3141251802444458 (nan) | > loss: 0.00536979828029871 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.1363099714404301) | > loader_time: 0.0045 (0.0046440806280574)  --> TIME: 2025-02-25 10:57:26 -- STEP: 11940/36938 -- GLOBAL_STEP: 750700 | > loss_text_ce: 0.033950820565223694 (0.03447910484497514) | > loss_mel_ce: 2.934145450592041 (nan) | > loss: 0.011778159998357296 (nan) | > current_lr: 5e-06 | > step_time: 0.1885 (0.13634490805055632) | > loader_time: 0.0049 (0.004644254923266181)  --> TIME: 2025-02-25 10:57:35 -- STEP: 11990/36938 -- GLOBAL_STEP: 750750 | > loss_text_ce: 0.027696926146745682 (0.03447477654371433) | > loss_mel_ce: 2.8128347396850586 (nan) | > loss: 0.011271951720118523 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.13636448796934855) | > loader_time: 0.0046 (0.004644236711783643)  --> TIME: 2025-02-25 10:57:45 -- STEP: 12040/36938 -- GLOBAL_STEP: 750800 | > loss_text_ce: 0.03898102045059204 (0.03447559456204848) | > loss_mel_ce: 2.085613250732422 (nan) | > loss: 0.008430929854512215 (nan) | > current_lr: 5e-06 | > step_time: 0.1628 (0.13637812713848016) | > loader_time: 0.0046 (0.004644164512323774)  --> TIME: 2025-02-25 10:57:54 -- STEP: 12090/36938 -- GLOBAL_STEP: 750850 | > loss_text_ce: 0.027088621631264687 (0.03447022682981782) | > loss_mel_ce: 2.166553497314453 (nan) | > loss: 0.008704929612576962 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.1363608882484408) | > loader_time: 0.0046 (0.004643969362288879)  --> TIME: 2025-02-25 10:58:03 -- STEP: 12140/36938 -- GLOBAL_STEP: 750900 | > loss_text_ce: 0.039799291640520096 (0.034475901456246176) | > loss_mel_ce: 3.2967002391815186 (nan) | > loss: 0.013240077532827854 (nan) | > current_lr: 5e-06 | > step_time: 0.1697 (0.1363710645788784) | > loader_time: 0.0046 (0.004643928101270942)  --> TIME: 2025-02-25 10:58:12 -- STEP: 12190/36938 -- GLOBAL_STEP: 750950 | > loss_text_ce: 0.03458905965089798 (0.034464072168845415) | > loss_mel_ce: 1.8866841793060303 (nan) | > loss: 0.007624100428074598 (nan) | > current_lr: 5e-06 | > step_time: 0.1591 (0.13636570817823818) | > loader_time: 0.0047 (0.0046437091373634385)  --> TIME: 2025-02-25 10:58:22 -- STEP: 12240/36938 -- GLOBAL_STEP: 751000 | > loss_text_ce: 0.03144681826233864 (0.034459367384403854) | > loss_mel_ce: 2.205705404281616 (nan) | > loss: 0.008877589367330074 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13636994835208482) | > loader_time: 0.0046 (0.0046435354971418085)  --> TIME: 2025-02-25 10:58:31 -- STEP: 12290/36938 -- GLOBAL_STEP: 751050 | > loss_text_ce: 0.02924611046910286 (0.03445445202560804) | > loss_mel_ce: 2.7563302516937256 (nan) | > loss: 0.011053875088691711 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.1363890678927222) | > loader_time: 0.0046 (0.004643441197152432)  --> TIME: 2025-02-25 10:58:41 -- STEP: 12340/36938 -- GLOBAL_STEP: 751100 | > loss_text_ce: 0.03354800119996071 (0.034450949211808864) | > loss_mel_ce: 2.4417498111724854 (nan) | > loss: 0.009822611697018147 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13639968871297536) | > loader_time: 0.0046 (0.004643573250917395)  --> TIME: 2025-02-25 10:58:50 -- STEP: 12390/36938 -- GLOBAL_STEP: 751150 | > loss_text_ce: 0.03875427320599556 (0.03444783506851853) | > loss_mel_ce: 3.4691684246063232 (nan) | > loss: 0.013920328579843044 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.136414605268459) | > loader_time: 0.0046 (0.004643588281620682)  --> TIME: 2025-02-25 10:59:00 -- STEP: 12440/36938 -- GLOBAL_STEP: 751200 | > loss_text_ce: 0.03439252823591232 (0.03444552800118035) | > loss_mel_ce: 2.9817280769348145 (nan) | > loss: 0.011968733742833138 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13642066195846816) | > loader_time: 0.0049 (0.004643558401770136)  --> TIME: 2025-02-25 10:59:09 -- STEP: 12490/36938 -- GLOBAL_STEP: 751250 | > loss_text_ce: 0.023759888485074043 (0.034441272025589595) | > loss_mel_ce: 1.1614640951156616 (nan) | > loss: 0.0047032698057591915 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13641223399709815) | > loader_time: 0.0045 (0.004643375898381048)  --> TIME: 2025-02-25 10:59:18 -- STEP: 12540/36938 -- GLOBAL_STEP: 751300 | > loss_text_ce: 0.030090665444731712 (0.03443688607716768) | > loss_mel_ce: 3.000483989715576 (nan) | > loss: 0.012026090174913406 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13641053588386604) | > loader_time: 0.0046 (0.004643238313270323)  --> TIME: 2025-02-25 10:59:27 -- STEP: 12590/36938 -- GLOBAL_STEP: 751350 | > loss_text_ce: 0.03357125073671341 (0.03443466395027279) | > loss_mel_ce: 2.20428204536438 (nan) | > loss: 0.008880370296537876 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13640731104410725) | > loader_time: 0.0046 (0.004643086273581181)  --> TIME: 2025-02-25 10:59:36 -- STEP: 12640/36938 -- GLOBAL_STEP: 751400 | > loss_text_ce: 0.03574776649475098 (0.03444236750976807) | > loss_mel_ce: 0.6902748346328735 (nan) | > loss: 0.002881042193621397 (nan) | > current_lr: 5e-06 | > step_time: 0.1679 (0.136389962593212) | > loader_time: 0.0049 (0.004643028653875171)  --> TIME: 2025-02-25 10:59:46 -- STEP: 12690/36938 -- GLOBAL_STEP: 751450 | > loss_text_ce: 0.030890299007296562 (0.03444504664727959) | > loss_mel_ce: 2.7613282203674316 (nan) | > loss: 0.011080232448875904 (nan) | > current_lr: 5e-06 | > step_time: 0.1084 (0.1363841561467001) | > loader_time: 0.0045 (0.004643160926722643)  --> TIME: 2025-02-25 10:59:55 -- STEP: 12740/36938 -- GLOBAL_STEP: 751500 | > loss_text_ce: 0.03398172929883003 (0.0344497786723723) | > loss_mel_ce: 3.2603702545166016 (nan) | > loss: 0.013072825968265533 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13636668791389223) | > loader_time: 0.0046 (0.004643225127149777)  --> TIME: 2025-02-25 11:00:04 -- STEP: 12790/36938 -- GLOBAL_STEP: 751550 | > loss_text_ce: 0.04267444461584091 (0.03444772284962933) | > loss_mel_ce: 3.0973095893859863 (nan) | > loss: 0.012460255064070225 (nan) | > current_lr: 5e-06 | > step_time: 0.1651 (0.13634902908990362) | > loader_time: 0.0048 (0.004643324635008318)  --> TIME: 2025-02-25 11:00:13 -- STEP: 12840/36938 -- GLOBAL_STEP: 751600 | > loss_text_ce: 0.02697211131453514 (0.03444782204025184) | > loss_mel_ce: 2.88537859916687 (nan) | > loss: 0.011556947603821754 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1363391363732175) | > loader_time: 0.0045 (0.004643479964443442)  --> TIME: 2025-02-25 11:00:22 -- STEP: 12890/36938 -- GLOBAL_STEP: 751650 | > loss_text_ce: 0.04533880949020386 (0.034450589165834544) | > loss_mel_ce: 1.8701578378677368 (nan) | > loss: 0.0076011773198843 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13635503589135972) | > loader_time: 0.0045 (0.0046433628761463)  --> TIME: 2025-02-25 11:00:32 -- STEP: 12940/36938 -- GLOBAL_STEP: 751700 | > loss_text_ce: 0.03797392547130585 (0.03444774732284467) | > loss_mel_ce: 3.1067705154418945 (nan) | > loss: 0.01247914507985115 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13636157895903395) | > loader_time: 0.0045 (0.004643313501496575)  --> TIME: 2025-02-25 11:00:41 -- STEP: 12990/36938 -- GLOBAL_STEP: 751750 | > loss_text_ce: 0.02962452732026577 (0.03444866657291517) | > loss_mel_ce: 2.351813554763794 (nan) | > loss: 0.009450151585042477 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13635206101397807) | > loader_time: 0.0046 (0.004643089758422933)  --> TIME: 2025-02-25 11:00:50 -- STEP: 13040/36938 -- GLOBAL_STEP: 751800 | > loss_text_ce: 0.03308006748557091 (0.034449065640066154) | > loss_mel_ce: 3.3376402854919434 (nan) | > loss: 0.013375875540077686 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13634153743463148) | > loader_time: 0.0046 (0.00464315286443276)  --> TIME: 2025-02-25 11:00:59 -- STEP: 13090/36938 -- GLOBAL_STEP: 751850 | > loss_text_ce: 0.035711467266082764 (0.0344442002224824) | > loss_mel_ce: 3.2957606315612793 (nan) | > loss: 0.01322012860327959 (nan) | > current_lr: 5e-06 | > step_time: 0.1829 (0.13633452651152916) | > loader_time: 0.0047 (0.0046431586430945105)  --> TIME: 2025-02-25 11:01:09 -- STEP: 13140/36938 -- GLOBAL_STEP: 751900 | > loss_text_ce: 0.03723875805735588 (0.034443200075831834) | > loss_mel_ce: 4.147641658782959 (nan) | > loss: 0.016606668010354042 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13634873864494795) | > loader_time: 0.0045 (0.004643155124089475)  --> TIME: 2025-02-25 11:01:18 -- STEP: 13190/36938 -- GLOBAL_STEP: 751950 | > loss_text_ce: 0.03139078617095947 (0.034440959861094154) | > loss_mel_ce: 2.975282669067383 (nan) | > loss: 0.011931244283914566 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13634929147247596) | > loader_time: 0.0046 (0.0046431509448869035)  --> TIME: 2025-02-25 11:01:28 -- STEP: 13240/36938 -- GLOBAL_STEP: 752000 | > loss_text_ce: 0.03926477208733559 (0.034442855045340295) | > loss_mel_ce: 2.8874573707580566 (nan) | > loss: 0.011613977141678333 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13635820609565355) | > loader_time: 0.0046 (0.00464326539428571)  --> TIME: 2025-02-25 11:01:37 -- STEP: 13290/36938 -- GLOBAL_STEP: 752050 | > loss_text_ce: 0.0340026319026947 (0.03444299459306051) | > loss_mel_ce: 1.6721420288085938 (nan) | > loss: 0.00677041569724679 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13635271770600849) | > loader_time: 0.0046 (0.00464322041352947)  --> TIME: 2025-02-25 11:01:46 -- STEP: 13340/36938 -- GLOBAL_STEP: 752100 | > loss_text_ce: 0.03271399438381195 (0.03444028612801537) | > loss_mel_ce: 2.916618824005127 (nan) | > loss: 0.011703701689839363 (nan) | > current_lr: 5e-06 | > step_time: 0.188 (0.1363481438499532) | > loader_time: 0.0048 (0.004643485392408911)  --> TIME: 2025-02-25 11:01:55 -- STEP: 13390/36938 -- GLOBAL_STEP: 752150 | > loss_text_ce: 0.035752296447753906 (0.03443599940834782) | > loss_mel_ce: 1.3781415224075317 (nan) | > loss: 0.005610689986497164 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13635039970534837) | > loader_time: 0.0045 (0.004643433587243058)  --> TIME: 2025-02-25 11:02:05 -- STEP: 13440/36938 -- GLOBAL_STEP: 752200 | > loss_text_ce: 0.032061830163002014 (0.034434510622703154) | > loss_mel_ce: 3.3521692752838135 (nan) | > loss: 0.013429488986730576 (nan) | > current_lr: 5e-06 | > step_time: 0.177 (0.1363584798893773) | > loader_time: 0.0047 (0.004643525537990369)  --> TIME: 2025-02-25 11:02:14 -- STEP: 13490/36938 -- GLOBAL_STEP: 752250 | > loss_text_ce: 0.03540317341685295 (0.03443057206228088) | > loss_mel_ce: 3.299663543701172 (nan) | > loss: 0.013234392739832401 (nan) | > current_lr: 5e-06 | > step_time: 0.1892 (0.13637190409463243) | > loader_time: 0.0048 (0.004643366812245748)  --> TIME: 2025-02-25 11:02:24 -- STEP: 13540/36938 -- GLOBAL_STEP: 752300 | > loss_text_ce: 0.03498195484280586 (0.03442372746065426) | > loss_mel_ce: 2.7178194522857666 (nan) | > loss: 0.01092381589114666 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.1363805846942594) | > loader_time: 0.0046 (0.004643291384642003)  --> TIME: 2025-02-25 11:02:33 -- STEP: 13590/36938 -- GLOBAL_STEP: 752350 | > loss_text_ce: 0.03115224465727806 (0.03442186597451001) | > loss_mel_ce: 2.665740489959717 (nan) | > loss: 0.01070195622742176 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.1363935588650019) | > loader_time: 0.0046 (0.004643376633206896)  --> TIME: 2025-02-25 11:02:43 -- STEP: 13640/36938 -- GLOBAL_STEP: 752400 | > loss_text_ce: 0.04401513561606407 (0.03442252438516815) | > loss_mel_ce: 0.15421295166015625 (nan) | > loss: 0.0007866194355301559 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.1363871307666715) | > loader_time: 0.0046 (0.004643260593638024)  --> TIME: 2025-02-25 11:02:52 -- STEP: 13690/36938 -- GLOBAL_STEP: 752450 | > loss_text_ce: 0.03421669453382492 (0.03443172086292859) | > loss_mel_ce: 2.75543475151062 (nan) | > loss: 0.01107004564255476 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13639644841541904) | > loader_time: 0.0046 (0.0046432250254862814)  --> TIME: 2025-02-25 11:03:01 -- STEP: 13740/36938 -- GLOBAL_STEP: 752500 | > loss_text_ce: 0.03491470590233803 (0.034431246152687396) | > loss_mel_ce: 3.040424108505249 (nan) | > loss: 0.01220372598618269 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13638503051741302) | > loader_time: 0.0046 (0.004642978783431162)  --> TIME: 2025-02-25 11:03:11 -- STEP: 13790/36938 -- GLOBAL_STEP: 752550 | > loss_text_ce: 0.048624347895383835 (0.034433460206960635) | > loss_mel_ce: 1.2381714582443237 (nan) | > loss: 0.005106333177536726 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13639695921006592) | > loader_time: 0.0046 (0.004642925788043954)  --> TIME: 2025-02-25 11:03:20 -- STEP: 13840/36938 -- GLOBAL_STEP: 752600 | > loss_text_ce: 0.03502705693244934 (0.034433542924389625) | > loss_mel_ce: 2.253427505493164 (nan) | > loss: 0.009081169031560421 (nan) | > current_lr: 5e-06 | > step_time: 0.152 (0.13641217785074677) | > loader_time: 0.0046 (0.004642872658768117)  --> TIME: 2025-02-25 11:03:29 -- STEP: 13890/36938 -- GLOBAL_STEP: 752650 | > loss_text_ce: 0.0381784662604332 (0.03443487031013866) | > loss_mel_ce: 2.7164552211761475 (nan) | > loss: 0.010931086726486683 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13641545904721333) | > loader_time: 0.0045 (0.004642823379274897)  --> TIME: 2025-02-25 11:03:39 -- STEP: 13940/36938 -- GLOBAL_STEP: 752700 | > loss_text_ce: 0.02972293458878994 (0.03443175557355446) | > loss_mel_ce: 3.054976224899292 (nan) | > loss: 0.012240869924426079 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13642687927530858) | > loader_time: 0.0045 (0.004642783398265623)  --> TIME: 2025-02-25 11:03:48 -- STEP: 13990/36938 -- GLOBAL_STEP: 752750 | > loss_text_ce: 0.03094881772994995 (0.034434423895035024) | > loss_mel_ce: 2.70229434967041 (nan) | > loss: 0.010846204124391079 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13643483832702386) | > loader_time: 0.0045 (0.004642813660742277)  --> TIME: 2025-02-25 11:03:58 -- STEP: 14040/36938 -- GLOBAL_STEP: 752800 | > loss_text_ce: 0.03681138530373573 (0.03443535900033997) | > loss_mel_ce: 2.8722925186157227 (nan) | > loss: 0.011544063687324524 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13644500030072162) | > loader_time: 0.0046 (0.004642743059033318)  --> TIME: 2025-02-25 11:04:07 -- STEP: 14090/36938 -- GLOBAL_STEP: 752850 | > loss_text_ce: 0.03866332769393921 (0.034432860891578765) | > loss_mel_ce: 2.8741610050201416 (nan) | > loss: 0.011558827944099903 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13644804600877564) | > loader_time: 0.0044 (0.004642641958909454)  --> TIME: 2025-02-25 11:04:17 -- STEP: 14140/36938 -- GLOBAL_STEP: 752900 | > loss_text_ce: 0.03539225086569786 (0.03443175287755244) | > loss_mel_ce: 3.092249631881714 (nan) | > loss: 0.012411277741193771 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13645712272112734) | > loader_time: 0.0045 (0.0046426140435851205)  --> TIME: 2025-02-25 11:04:26 -- STEP: 14190/36938 -- GLOBAL_STEP: 752950 | > loss_text_ce: 0.03352651372551918 (0.034431827347069455) | > loss_mel_ce: 3.0333101749420166 (nan) | > loss: 0.012169986963272095 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13644520824773954) | > loader_time: 0.0046 (0.004642532239756651)  --> TIME: 2025-02-25 11:04:35 -- STEP: 14240/36938 -- GLOBAL_STEP: 753000 | > loss_text_ce: 0.03287971764802933 (0.03443181613686487) | > loss_mel_ce: 3.3745760917663574 (nan) | > loss: 0.013521650806069374 (nan) | > current_lr: 5e-06 | > step_time: 0.1876 (0.1364623157998166) | > loader_time: 0.0048 (0.004642531911978537)  --> TIME: 2025-02-25 11:04:45 -- STEP: 14290/36938 -- GLOBAL_STEP: 753050 | > loss_text_ce: 0.039734143763780594 (0.034436268280107235) | > loss_mel_ce: 2.0550484657287598 (nan) | > loss: 0.008312629535794258 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13645925937289705) | > loader_time: 0.0045 (0.004642382896054927)  --> TIME: 2025-02-25 11:04:54 -- STEP: 14340/36938 -- GLOBAL_STEP: 753100 | > loss_text_ce: 0.03527399152517319 (0.03443673005557504) | > loss_mel_ce: 2.9317033290863037 (nan) | > loss: 0.011773720383644104 (nan) | > current_lr: 5e-06 | > step_time: 0.1832 (0.1364509990524564) | > loader_time: 0.0047 (0.004642286759540103)  --> TIME: 2025-02-25 11:05:03 -- STEP: 14390/36938 -- GLOBAL_STEP: 753150 | > loss_text_ce: 0.036121662706136703 (0.03443614524796287) | > loss_mel_ce: 3.093374013900757 (nan) | > loss: 0.012418634258210659 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13645731715215084) | > loader_time: 0.0047 (0.0046423534621290825)  --> TIME: 2025-02-25 11:05:13 -- STEP: 14440/36938 -- GLOBAL_STEP: 753200 | > loss_text_ce: 0.03679085895419121 (0.03443570586013004) | > loss_mel_ce: 2.7398693561553955 (nan) | > loss: 0.011018493212759495 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13646633780233908) | > loader_time: 0.0046 (0.0046423975780729865)  --> TIME: 2025-02-25 11:05:22 -- STEP: 14490/36938 -- GLOBAL_STEP: 753250 | > loss_text_ce: 0.04830296337604523 (0.0344337827101829) | > loss_mel_ce: 0.4891328811645508 (nan) | > loss: 0.0021326818969100714 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.1364638035506376) | > loader_time: 0.0046 (0.004642319810892296)  --> TIME: 2025-02-25 11:05:32 -- STEP: 14540/36938 -- GLOBAL_STEP: 753300 | > loss_text_ce: 0.032974351197481155 (0.03442943958415723) | > loss_mel_ce: 3.4699018001556396 (nan) | > loss: 0.013900302350521088 (nan) | > current_lr: 5e-06 | > step_time: 0.1699 (0.13648752444712148) | > loader_time: 0.0047 (0.0046423253855645994)  --> TIME: 2025-02-25 11:05:41 -- STEP: 14590/36938 -- GLOBAL_STEP: 753350 | > loss_text_ce: 0.0388660654425621 (0.03442630341399097) | > loss_mel_ce: 1.907122015953064 (nan) | > loss: 0.0077221752144396305 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13648779527874833) | > loader_time: 0.0046 (0.004642253677023014)  --> TIME: 2025-02-25 11:05:50 -- STEP: 14640/36938 -- GLOBAL_STEP: 753400 | > loss_text_ce: 0.04175637662410736 (0.034427520670467615) | > loss_mel_ce: 2.300492763519287 (nan) | > loss: 0.009294640272855759 (nan) | > current_lr: 5e-06 | > step_time: 0.1864 (0.1364924463743737) | > loader_time: 0.0048 (0.004642186138799258)  --> TIME: 2025-02-25 11:06:00 -- STEP: 14690/36938 -- GLOBAL_STEP: 753450 | > loss_text_ce: 0.03483981266617775 (0.03442733484428767) | > loss_mel_ce: 3.3904128074645996 (nan) | > loss: 0.013592272996902466 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13649338846550896) | > loader_time: 0.0046 (0.004642060259071779)  --> TIME: 2025-02-25 11:06:09 -- STEP: 14740/36938 -- GLOBAL_STEP: 753500 | > loss_text_ce: 0.020604034885764122 (0.03442198669656698) | > loss_mel_ce: 3.3937089443206787 (nan) | > loss: 0.01354886218905449 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13649565706097927) | > loader_time: 0.0044 (0.004641925237233774)  --> TIME: 2025-02-25 11:06:18 -- STEP: 14790/36938 -- GLOBAL_STEP: 753550 | > loss_text_ce: 0.03282320499420166 (0.03442153293309471) | > loss_mel_ce: 2.702336072921753 (nan) | > loss: 0.01085380744189024 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13649044773561086) | > loader_time: 0.0045 (0.004641716588260816)  --> TIME: 2025-02-25 11:06:27 -- STEP: 14840/36938 -- GLOBAL_STEP: 753600 | > loss_text_ce: 0.029246825724840164 (0.034420322614779834) | > loss_mel_ce: 2.800898551940918 (nan) | > loss: 0.011230736039578915 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.1364959597908922) | > loader_time: 0.0044 (0.004641630032634455)  --> TIME: 2025-02-25 11:06:37 -- STEP: 14890/36938 -- GLOBAL_STEP: 753650 | > loss_text_ce: 0.03501531481742859 (0.03441977009247271) | > loss_mel_ce: 0.012115478515625 (nan) | > loss: 0.00018702696252148598 (nan) | > current_lr: 5e-06 | > step_time: 0.1566 (0.13650983958696025) | > loader_time: 0.0048 (0.004641544602716242)  --> TIME: 2025-02-25 11:06:47 -- STEP: 14940/36938 -- GLOBAL_STEP: 753700 | > loss_text_ce: 0.030610162764787674 (0.03441897614122226) | > loss_mel_ce: 2.975318193435669 (nan) | > loss: 0.011928287334740162 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.13653016893101025) | > loader_time: 0.0047 (0.004641463782094705)  --> TIME: 2025-02-25 11:06:56 -- STEP: 14990/36938 -- GLOBAL_STEP: 753750 | > loss_text_ce: 0.026498781517148018 (0.03441720696077944) | > loss_mel_ce: 2.513134002685547 (nan) | > loss: 0.010077908635139465 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.1365358349162011) | > loader_time: 0.0046 (0.004641479583801287)  --> TIME: 2025-02-25 11:07:05 -- STEP: 15040/36938 -- GLOBAL_STEP: 753800 | > loss_text_ce: 0.027886921539902687 (0.034419019971082605) | > loss_mel_ce: 3.241302013397217 (nan) | > loss: 0.012972972355782986 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13652829643576872) | > loader_time: 0.0045 (0.004641389989472428)  --> TIME: 2025-02-25 11:07:15 -- STEP: 15090/36938 -- GLOBAL_STEP: 753850 | > loss_text_ce: 0.04140472784638405 (0.03441994828526916) | > loss_mel_ce: 2.908571720123291 (nan) | > loss: 0.011706256307661533 (nan) | > current_lr: 5e-06 | > step_time: 0.1865 (0.13653556294122238) | > loader_time: 0.0048 (0.004641412961870073)  --> TIME: 2025-02-25 11:07:24 -- STEP: 15140/36938 -- GLOBAL_STEP: 753900 | > loss_text_ce: 0.0365462563931942 (0.03442164851756115) | > loss_mel_ce: 3.0898098945617676 (nan) | > loss: 0.012406175956130028 (nan) | > current_lr: 5e-06 | > step_time: 0.1605 (0.13654014634330344) | > loader_time: 0.0046 (0.00464160149447037)  --> TIME: 2025-02-25 11:07:33 -- STEP: 15190/36938 -- GLOBAL_STEP: 753950 | > loss_text_ce: 0.03647993505001068 (0.03442795344795785) | > loss_mel_ce: 2.9694366455078125 (nan) | > loss: 0.011928240768611431 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.1365432119275349) | > loader_time: 0.0046 (0.004641786792546665)  --> TIME: 2025-02-25 11:07:43 -- STEP: 15240/36938 -- GLOBAL_STEP: 754000 | > loss_text_ce: 0.04915287718176842 (0.034431886069080715) | > loss_mel_ce: 3.432421922683716 (nan) | > loss: 0.013815773651003838 (nan) | > current_lr: 5e-06 | > step_time: 0.1072 (0.13655004413735064) | > loader_time: 0.0044 (0.004641889524584971)  --> TIME: 2025-02-25 11:07:52 -- STEP: 15290/36938 -- GLOBAL_STEP: 754050 | > loss_text_ce: 0.023104678839445114 (0.03443168300460423) | > loss_mel_ce: 3.1949462890625 (nan) | > loss: 0.01277004461735487 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13654239483328887) | > loader_time: 0.0047 (0.004641901487770832)  --> TIME: 2025-02-25 11:08:01 -- STEP: 15340/36938 -- GLOBAL_STEP: 754100 | > loss_text_ce: 0.026308419182896614 (0.034425634216405605) | > loss_mel_ce: 2.361797332763672 (nan) | > loss: 0.009476611390709877 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13652215841696358) | > loader_time: 0.0045 (0.0046419070628040396)  --> TIME: 2025-02-25 11:08:10 -- STEP: 15390/36938 -- GLOBAL_STEP: 754150 | > loss_text_ce: 0.030948085710406303 (0.034429845005235195) | > loss_mel_ce: 3.1024153232574463 (nan) | > loss: 0.012433982454240322 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.13652764737722403) | > loader_time: 0.0048 (0.004641808047551758)  --> TIME: 2025-02-25 11:08:20 -- STEP: 15440/36938 -- GLOBAL_STEP: 754200 | > loss_text_ce: 0.033419169485569 (0.03442642477249322) | > loss_mel_ce: 3.390115737915039 (nan) | > loss: 0.013585456646978855 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.136526923846706) | > loader_time: 0.0046 (0.004641812283140368)  --> TIME: 2025-02-25 11:08:29 -- STEP: 15490/36938 -- GLOBAL_STEP: 754250 | > loss_text_ce: 0.041266292333602905 (0.03442508248627927) | > loss_mel_ce: 2.1478679180145264 (nan) | > loss: 0.008687040768563747 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13652716882002758) | > loader_time: 0.0045 (0.004641849121945816)  --> TIME: 2025-02-25 11:08:38 -- STEP: 15540/36938 -- GLOBAL_STEP: 754300 | > loss_text_ce: 0.03724091872572899 (0.03442011796018332) | > loss_mel_ce: 2.8704473972320557 (nan) | > loss: 0.011538446880877018 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.13652989503806634) | > loader_time: 0.0047 (0.004641782777189726)  --> TIME: 2025-02-25 11:08:48 -- STEP: 15590/36938 -- GLOBAL_STEP: 754350 | > loss_text_ce: 0.032182879745960236 (0.03441960788717249) | > loss_mel_ce: 3.091829299926758 (nan) | > loss: 0.012396874837577343 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.1365324553807185) | > loader_time: 0.0046 (0.004641719060191878)  --> TIME: 2025-02-25 11:08:57 -- STEP: 15640/36938 -- GLOBAL_STEP: 754400 | > loss_text_ce: 0.02877170220017433 (0.034419636662437414) | > loss_mel_ce: 2.990091562271118 (nan) | > loss: 0.01197961624711752 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13653316982567015) | > loader_time: 0.0045 (0.004641628387334075)  --> TIME: 2025-02-25 11:09:06 -- STEP: 15690/36938 -- GLOBAL_STEP: 754450 | > loss_text_ce: 0.034101780503988266 (0.03442043789613164) | > loss_mel_ce: 2.986395835876465 (nan) | > loss: 0.011986101977527142 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13653558150000458) | > loader_time: 0.0046 (0.004641823406961005)  --> TIME: 2025-02-25 11:09:16 -- STEP: 15740/36938 -- GLOBAL_STEP: 754500 | > loss_text_ce: 0.03720635920763016 (0.03442026481838266) | > loss_mel_ce: 3.03947114944458 (nan) | > loss: 0.012209038250148296 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.13653487747173157) | > loader_time: 0.0047 (0.004641732630311588)  --> TIME: 2025-02-25 11:09:25 -- STEP: 15790/36938 -- GLOBAL_STEP: 754550 | > loss_text_ce: 0.026350507512688637 (0.034421089245036035) | > loss_mel_ce: 0.11572265625 (nan) | > loss: 0.0005637824651785195 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13654358275258452) | > loader_time: 0.0046 (0.004641621893459344)  --> TIME: 2025-02-25 11:09:35 -- STEP: 15840/36938 -- GLOBAL_STEP: 754600 | > loss_text_ce: 0.026758866384625435 (0.03442078541224654) | > loss_mel_ce: 2.6193785667419434 (nan) | > loss: 0.010500546544790268 (nan) | > current_lr: 5e-06 | > step_time: 0.1862 (0.13654933802106325) | > loader_time: 0.0049 (0.004641673345156369)  --> TIME: 2025-02-25 11:09:44 -- STEP: 15890/36938 -- GLOBAL_STEP: 754650 | > loss_text_ce: 0.05241915211081505 (0.03441883666145457) | > loss_mel_ce: 2.653273820877075 (nan) | > loss: 0.010736878030002117 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13655462979220243) | > loader_time: 0.0046 (0.00464175244109894)  --> TIME: 2025-02-25 11:09:54 -- STEP: 15940/36938 -- GLOBAL_STEP: 754700 | > loss_text_ce: 0.03091263771057129 (0.034416813537223805) | > loss_mel_ce: 2.99910831451416 (nan) | > loss: 0.012023893184959888 (nan) | > current_lr: 5e-06 | > step_time: 0.1727 (0.13656680277330374) | > loader_time: 0.0049 (0.004641840628428899)  --> TIME: 2025-02-25 11:10:03 -- STEP: 15990/36938 -- GLOBAL_STEP: 754750 | > loss_text_ce: 0.0408085472881794 (0.03442171319970886) | > loss_mel_ce: 2.271937847137451 (nan) | > loss: 0.009177565574645996 (nan) | > current_lr: 5e-06 | > step_time: 0.1514 (0.13658795353768552) | > loader_time: 0.0047 (0.004641972071234304)  --> TIME: 2025-02-25 11:10:13 -- STEP: 16040/36938 -- GLOBAL_STEP: 754800 | > loss_text_ce: 0.032065849751234055 (0.03441940766748775) | > loss_mel_ce: 0.9991666078567505 (nan) | > loss: 0.004092192742973566 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13658243484628033) | > loader_time: 0.0045 (0.004642222320052432)  --> TIME: 2025-02-25 11:10:22 -- STEP: 16090/36938 -- GLOBAL_STEP: 754850 | > loss_text_ce: 0.037826087325811386 (0.03442171418696716) | > loss_mel_ce: 3.1322994232177734 (nan) | > loss: 0.012579863891005516 (nan) | > current_lr: 5e-06 | > step_time: 0.1626 (0.1365995133864073) | > loader_time: 0.0049 (0.004642244612229672)  --> TIME: 2025-02-25 11:10:32 -- STEP: 16140/36938 -- GLOBAL_STEP: 754900 | > loss_text_ce: 0.046048033982515335 (0.034420766674450785) | > loss_mel_ce: 2.8472983837127686 (nan) | > loss: 0.011481533758342266 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13660256196427534) | > loader_time: 0.0045 (0.004642223233479381)  --> TIME: 2025-02-25 11:10:41 -- STEP: 16190/36938 -- GLOBAL_STEP: 754950 | > loss_text_ce: 0.029985031113028526 (0.034421204125371856) | > loss_mel_ce: 1.939324975013733 (nan) | > loss: 0.007814723066985607 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13659732353547543) | > loader_time: 0.0046 (0.0046421082392380745)  --> TIME: 2025-02-25 11:10:50 -- STEP: 16240/36938 -- GLOBAL_STEP: 755000 | > loss_text_ce: 0.03412271663546562 (0.034420873323545015) | > loss_mel_ce: 2.8687617778778076 (nan) | > loss: 0.011519383639097214 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13660110430470984) | > loader_time: 0.0046 (0.004642048580893134)  --> TIME: 2025-02-25 11:11:00 -- STEP: 16290/36938 -- GLOBAL_STEP: 755050 | > loss_text_ce: 0.024492017924785614 (0.03442116510037267) | > loss_mel_ce: 2.9548850059509277 (nan) | > loss: 0.011822924949228764 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13660445495790624) | > loader_time: 0.0046 (0.004642040250929219)  --> TIME: 2025-02-25 11:11:09 -- STEP: 16340/36938 -- GLOBAL_STEP: 755100 | > loss_text_ce: 0.02789752371609211 (0.03441794225119629) | > loss_mel_ce: 1.354516625404358 (nan) | > loss: 0.005485770758241415 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13659850786482408) | > loader_time: 0.0045 (0.004641974293324743)  --> TIME: 2025-02-25 11:11:19 -- STEP: 16390/36938 -- GLOBAL_STEP: 755150 | > loss_text_ce: 0.037084877490997314 (0.034417291409424206) | > loss_mel_ce: 2.2642934322357178 (nan) | > loss: 0.009132454171776772 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13661420916987466) | > loader_time: 0.0046 (0.004642128231614135)  --> TIME: 2025-02-25 11:11:28 -- STEP: 16440/36938 -- GLOBAL_STEP: 755200 | > loss_text_ce: 0.02756776288151741 (0.034418394232628886) | > loss_mel_ce: 3.073129892349243 (nan) | > loss: 0.012304357253015041 (nan) | > current_lr: 5e-06 | > step_time: 0.176 (0.1366390486268247) | > loader_time: 0.0049 (0.004642140212720307)  --> TIME: 2025-02-25 11:11:38 -- STEP: 16490/36938 -- GLOBAL_STEP: 755250 | > loss_text_ce: 0.02820708602666855 (0.03441800846073847) | > loss_mel_ce: 2.898200035095215 (nan) | > loss: 0.0116127273067832 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13664205963500428) | > loader_time: 0.0045 (0.004642040112150135)  --> TIME: 2025-02-25 11:11:47 -- STEP: 16540/36938 -- GLOBAL_STEP: 755300 | > loss_text_ce: 0.03862253949046135 (0.034419705119815173) | > loss_mel_ce: 1.5019292831420898 (nan) | > loss: 0.006113301031291485 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13662796597094617) | > loader_time: 0.0045 (0.004641872147118434)  --> TIME: 2025-02-25 11:11:56 -- STEP: 16590/36938 -- GLOBAL_STEP: 755350 | > loss_text_ce: 0.030778123065829277 (0.03442305576324589) | > loss_mel_ce: 2.886495590209961 (nan) | > loss: 0.011576483957469463 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.13663200889458788) | > loader_time: 0.0046 (0.004641853505549028)  --> TIME: 2025-02-25 11:12:06 -- STEP: 16640/36938 -- GLOBAL_STEP: 755400 | > loss_text_ce: 0.03527374193072319 (0.03442521632381482) | > loss_mel_ce: 2.954833984375 (nan) | > loss: 0.011865507811307907 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.13663111615639617) | > loader_time: 0.0048 (0.004641748363008845)  --> TIME: 2025-02-25 11:12:15 -- STEP: 16690/36938 -- GLOBAL_STEP: 755450 | > loss_text_ce: 0.03510981425642967 (0.034428911290141084) | > loss_mel_ce: 1.0201040506362915 (nan) | > loss: 0.004187356680631638 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.1366346712381002) | > loader_time: 0.0044 (0.004641804929405271)  --> TIME: 2025-02-25 11:12:25 -- STEP: 16740/36938 -- GLOBAL_STEP: 755500 | > loss_text_ce: 0.03149198740720749 (0.034431606314335395) | > loss_mel_ce: 1.5363168716430664 (nan) | > loss: 0.006221463903784752 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13663664844016318) | > loader_time: 0.0045 (0.004641735069498205)  --> TIME: 2025-02-25 11:12:34 -- STEP: 16790/36938 -- GLOBAL_STEP: 755550 | > loss_text_ce: 0.035158559679985046 (0.034433827128820545) | > loss_mel_ce: 3.2540557384490967 (nan) | > loss: 0.013052438385784626 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.13663816234765994) | > loader_time: 0.0046 (0.0046417360862427304)  --> TIME: 2025-02-25 11:12:43 -- STEP: 16840/36938 -- GLOBAL_STEP: 755600 | > loss_text_ce: 0.03437291458249092 (0.03443722817278087) | > loss_mel_ce: 3.21893310546875 (nan) | > loss: 0.012909945100545883 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13664738190429102) | > loader_time: 0.0046 (0.0046417114711996875)  --> TIME: 2025-02-25 11:12:53 -- STEP: 16890/36938 -- GLOBAL_STEP: 755650 | > loss_text_ce: 0.030027054250240326 (0.034439904008241616) | > loss_mel_ce: 2.7333731651306152 (nan) | > loss: 0.01096587534993887 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.1366476417787702) | > loader_time: 0.0046 (0.004641587385558161)  --> TIME: 2025-02-25 11:13:02 -- STEP: 16940/36938 -- GLOBAL_STEP: 755700 | > loss_text_ce: 0.030942045152187347 (0.034439970679645) | > loss_mel_ce: 2.7048096656799316 (nan) | > loss: 0.01085615810006857 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.136657941594739) | > loader_time: 0.0046 (0.004641531743856906)  --> TIME: 2025-02-25 11:13:11 -- STEP: 16990/36938 -- GLOBAL_STEP: 755750 | > loss_text_ce: 0.027099886909127235 (0.034440007655135364) | > loss_mel_ce: 2.711836814880371 (nan) | > loss: 0.010868797078728676 (nan) | > current_lr: 5e-06 | > step_time: 0.168 (0.13664370514913057) | > loader_time: 0.0048 (0.004641386338582506)  --> TIME: 2025-02-25 11:13:20 -- STEP: 17040/36938 -- GLOBAL_STEP: 755800 | > loss_text_ce: 0.03039911948144436 (0.034438726786926395) | > loss_mel_ce: 7.375 (nan) | > loss: 0.029386507347226143 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13663627815918328) | > loader_time: 0.0045 (0.0046413052389879)  --> TIME: 2025-02-25 11:13:29 -- STEP: 17090/36938 -- GLOBAL_STEP: 755850 | > loss_text_ce: 0.0363396592438221 (0.034437756031760434) | > loss_mel_ce: 2.040428638458252 (nan) | > loss: 0.008241144940257072 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13662332834173604) | > loader_time: 0.0045 (0.00464116723443973)  --> TIME: 2025-02-25 11:13:39 -- STEP: 17140/36938 -- GLOBAL_STEP: 755900 | > loss_text_ce: 0.02748998999595642 (0.0344332585588333) | > loss_mel_ce: 2.891880989074707 (nan) | > loss: 0.011584805324673653 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13662659521002898) | > loader_time: 0.0046 (0.004641097178954346)  --> TIME: 2025-02-25 11:13:48 -- STEP: 17190/36938 -- GLOBAL_STEP: 755950 | > loss_text_ce: 0.01932915486395359 (0.03443272818647085) | > loss_mel_ce: 2.768782377243042 (nan) | > loss: 0.011063935235142708 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13662569744771555) | > loader_time: 0.0045 (0.00464093054082934)  --> TIME: 2025-02-25 11:13:57 -- STEP: 17240/36938 -- GLOBAL_STEP: 756000 | > loss_text_ce: 0.038628045469522476 (0.03443000792631595) | > loss_mel_ce: 3.1287732124328613 (nan) | > loss: 0.01256905309855938 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13663176085169113) | > loader_time: 0.0049 (0.004640956793473087)  --> TIME: 2025-02-25 11:14:07 -- STEP: 17290/36938 -- GLOBAL_STEP: 756050 | > loss_text_ce: 0.031459297984838486 (0.03442542433330422) | > loss_mel_ce: 3.2201404571533203 (nan) | > loss: 0.012903174385428429 (nan) | > current_lr: 5e-06 | > step_time: 0.1697 (0.13663617688979496) | > loader_time: 0.0048 (0.004640895855640507)  --> TIME: 2025-02-25 11:14:16 -- STEP: 17340/36938 -- GLOBAL_STEP: 756100 | > loss_text_ce: 0.03411805257201195 (0.03442525851589687) | > loss_mel_ce: 0.12890625 (nan) | > loss: 0.0006469218642450869 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1366415090626926) | > loader_time: 0.0044 (0.0046408267994652785)  --> TIME: 2025-02-25 11:14:26 -- STEP: 17390/36938 -- GLOBAL_STEP: 756150 | > loss_text_ce: 0.02465089038014412 (0.03442919094810646) | > loss_mel_ce: 3.0587825775146484 (nan) | > loss: 0.012235847301781178 (nan) | > current_lr: 5e-06 | > step_time: 0.1761 (0.13665484049458837) | > loader_time: 0.0048 (0.004640816709925329)  --> TIME: 2025-02-25 11:14:35 -- STEP: 17440/36938 -- GLOBAL_STEP: 756200 | > loss_text_ce: 0.04241310805082321 (0.03442623809416817) | > loss_mel_ce: 1.439697265625 (nan) | > loss: 0.005881390534341335 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13665203357781994) | > loader_time: 0.0046 (0.004640713976610674)  --> TIME: 2025-02-25 11:14:44 -- STEP: 17490/36938 -- GLOBAL_STEP: 756250 | > loss_text_ce: 0.018374698236584663 (0.03442237538548074) | > loss_mel_ce: 3.316100835800171 (nan) | > loss: 0.013232046738266945 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1366614948892145) | > loader_time: 0.0044 (0.004640655206775156)  --> TIME: 2025-02-25 11:14:54 -- STEP: 17540/36938 -- GLOBAL_STEP: 756300 | > loss_text_ce: 0.04047529026865959 (0.03442244484851795) | > loss_mel_ce: 2.566617965698242 (nan) | > loss: 0.010345608927309513 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13665881703354088) | > loader_time: 0.0045 (0.004640561362628636)  --> TIME: 2025-02-25 11:15:03 -- STEP: 17590/36938 -- GLOBAL_STEP: 756350 | > loss_text_ce: 0.030222997069358826 (0.03442059327631712) | > loss_mel_ce: 2.5004615783691406 (nan) | > loss: 0.010042399168014526 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13666421423722977) | > loader_time: 0.0045 (0.0046404813757804616)  --> TIME: 2025-02-25 11:15:12 -- STEP: 17640/36938 -- GLOBAL_STEP: 756400 | > loss_text_ce: 0.03298429399728775 (0.03441893249880024) | > loss_mel_ce: 3.1415936946868896 (nan) | > loss: 0.012597532011568546 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.13665584659630628) | > loader_time: 0.0048 (0.004640332736125591)  --> TIME: 2025-02-25 11:15:21 -- STEP: 17690/36938 -- GLOBAL_STEP: 756450 | > loss_text_ce: 0.03763090446591377 (0.03442192974301887) | > loss_mel_ce: 3.0985500812530518 (nan) | > loss: 0.012445162981748581 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1366542285414603) | > loader_time: 0.0045 (0.004640218280276835)  --> TIME: 2025-02-25 11:15:31 -- STEP: 17740/36938 -- GLOBAL_STEP: 756500 | > loss_text_ce: 0.03425505757331848 (0.03442027145600857) | > loss_mel_ce: 2.9443933963775635 (nan) | > loss: 0.011820034123957157 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.13665855639144972) | > loader_time: 0.0046 (0.004640094900615035)  --> TIME: 2025-02-25 11:15:40 -- STEP: 17790/36938 -- GLOBAL_STEP: 756550 | > loss_text_ce: 0.03452722355723381 (0.03442180134212587) | > loss_mel_ce: 3.377228260040283 (nan) | > loss: 0.013538713566958904 (nan) | > current_lr: 5e-06 | > step_time: 0.1697 (0.13665900344323423) | > loader_time: 0.0046 (0.004640103994694351)  --> TIME: 2025-02-25 11:15:50 -- STEP: 17840/36938 -- GLOBAL_STEP: 756600 | > loss_text_ce: 0.02710442803800106 (0.034421807401364676) | > loss_mel_ce: 3.1812167167663574 (nan) | > loss: 0.012731433846056461 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.1366714892900593) | > loader_time: 0.0048 (0.004640074548699915)  --> TIME: 2025-02-25 11:15:59 -- STEP: 17890/36938 -- GLOBAL_STEP: 756650 | > loss_text_ce: 0.03446885198354721 (0.0344237917963918) | > loss_mel_ce: 2.937401294708252 (nan) | > loss: 0.011793136596679688 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1366647043062911) | > loader_time: 0.0046 (0.004640025463498708)  --> TIME: 2025-02-25 11:16:08 -- STEP: 17940/36938 -- GLOBAL_STEP: 756700 | > loss_text_ce: 0.04019150137901306 (0.03442421502354817) | > loss_mel_ce: 2.636049509048462 (nan) | > loss: 0.010620004497468472 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13666075704621516) | > loader_time: 0.0066 (0.0046401912419160585)  --> TIME: 2025-02-25 11:16:17 -- STEP: 17990/36938 -- GLOBAL_STEP: 756750 | > loss_text_ce: 0.03905671834945679 (0.03442254464099065) | > loss_mel_ce: 1.9922558069229126 (nan) | > loss: 0.008060764521360397 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.1366540573781714) | > loader_time: 0.0065 (0.0046401784445193675)  --> TIME: 2025-02-25 11:16:26 -- STEP: 18040/36938 -- GLOBAL_STEP: 756800 | > loss_text_ce: 0.038498032838106155 (0.03442316914246759) | > loss_mel_ce: 4.0445556640625 (nan) | > loss: 0.016202595084905624 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13664109903798816) | > loader_time: 0.0046 (0.004640150810292433)  --> TIME: 2025-02-25 11:16:36 -- STEP: 18090/36938 -- GLOBAL_STEP: 756850 | > loss_text_ce: 0.03637794032692909 (0.03442150943785414) | > loss_mel_ce: 2.523104667663574 (nan) | > loss: 0.010156677104532719 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13664936531703886) | > loader_time: 0.0047 (0.004640160192106873)  --> TIME: 2025-02-25 11:16:45 -- STEP: 18140/36938 -- GLOBAL_STEP: 756900 | > loss_text_ce: 0.025434071198105812 (0.03442002317966759) | > loss_mel_ce: 2.379866361618042 (nan) | > loss: 0.00954484287649393 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.1366574344324844) | > loader_time: 0.0047 (0.004640200803140543)  --> TIME: 2025-02-25 11:16:55 -- STEP: 18190/36938 -- GLOBAL_STEP: 756950 | > loss_text_ce: 0.02631986141204834 (0.034422054322238944) | > loss_mel_ce: 1.2160917520523071 (nan) | > loss: 0.004930205177515745 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13665258977490663) | > loader_time: 0.0045 (0.0046404683504738895)  --> TIME: 2025-02-25 11:17:04 -- STEP: 18240/36938 -- GLOBAL_STEP: 757000 | > loss_text_ce: 0.031108954921364784 (0.03442178442589174) | > loss_mel_ce: 3.039915084838867 (nan) | > loss: 0.012186603620648384 (nan) | > current_lr: 5e-06 | > step_time: 0.1705 (0.13665117441039493) | > loader_time: 0.0049 (0.0046404800786260855)  --> TIME: 2025-02-25 11:17:14 -- STEP: 18290/36938 -- GLOBAL_STEP: 757050 | > loss_text_ce: 0.04350268095731735 (0.034422702851113186) | > loss_mel_ce: 2.7243690490722656 (nan) | > loss: 0.010983617976307869 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.1366742429086733) | > loader_time: 0.0048 (0.004640536363037501)  --> TIME: 2025-02-25 11:17:23 -- STEP: 18340/36938 -- GLOBAL_STEP: 757100 | > loss_text_ce: 0.03841444104909897 (0.03442202419679394) | > loss_mel_ce: 2.477084159851074 (nan) | > loss: 0.009982137940824032 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13667772739248318) | > loader_time: 0.0048 (0.004640636761289898)  --> TIME: 2025-02-25 11:17:32 -- STEP: 18390/36938 -- GLOBAL_STEP: 757150 | > loss_text_ce: 0.031229548156261444 (0.034424479093803875) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13667571731337286) | > loader_time: 0.0045 (0.00464067499296635)  --> TIME: 2025-02-25 11:17:42 -- STEP: 18440/36938 -- GLOBAL_STEP: 757200 | > loss_text_ce: 0.044540610164403915 (0.03442708927518835) | > loss_mel_ce: 2.87017822265625 (nan) | > loss: 0.0115663455799222 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.136683136767784) | > loader_time: 0.0046 (0.004640783767120952)  --> TIME: 2025-02-25 11:17:51 -- STEP: 18490/36938 -- GLOBAL_STEP: 757250 | > loss_text_ce: 0.031020386144518852 (0.034423917617242006) | > loss_mel_ce: 2.4174771308898926 (nan) | > loss: 0.009716260246932507 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.13667697581553884) | > loader_time: 0.0046 (0.004640748308696505)  --> TIME: 2025-02-25 11:18:01 -- STEP: 18540/36938 -- GLOBAL_STEP: 757300 | > loss_text_ce: 0.028304120525717735 (0.034427176010689606) | > loss_mel_ce: 2.9934093952178955 (nan) | > loss: 0.011990927159786224 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.13669682566273586) | > loader_time: 0.0048 (0.004640880603234739)  --> TIME: 2025-02-25 11:18:10 -- STEP: 18590/36938 -- GLOBAL_STEP: 757350 | > loss_text_ce: 0.030632974579930305 (0.034422944090216574) | > loss_mel_ce: 3.002037525177002 (nan) | > loss: 0.012034407816827297 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13668834697565527) | > loader_time: 0.0047 (0.004641093497253215)  --> TIME: 2025-02-25 11:18:19 -- STEP: 18640/36938 -- GLOBAL_STEP: 757400 | > loss_text_ce: 0.03337942436337471 (0.03442401712887253) | > loss_mel_ce: 2.2886428833007812 (nan) | > loss: 0.009214374236762524 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13667316911302563) | > loader_time: 0.0045 (0.004641028343352119)  --> TIME: 2025-02-25 11:18:28 -- STEP: 18690/36938 -- GLOBAL_STEP: 757450 | > loss_text_ce: 0.03491101786494255 (0.03442366277358531) | > loss_mel_ce: 2.827608585357666 (nan) | > loss: 0.011359205469489098 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.136676548163863) | > loader_time: 0.0045 (0.004641149183082471)  --> TIME: 2025-02-25 11:18:38 -- STEP: 18740/36938 -- GLOBAL_STEP: 757500 | > loss_text_ce: 0.041033390909433365 (0.03442279564066666) | > loss_mel_ce: 1.954827904701233 (nan) | > loss: 0.00792008452117443 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13667653267259042) | > loader_time: 0.0045 (0.004641219747740781)  --> TIME: 2025-02-25 11:18:47 -- STEP: 18790/36938 -- GLOBAL_STEP: 757550 | > loss_text_ce: 0.022785980254411697 (0.034422516830497664) | > loss_mel_ce: 3.085536003112793 (nan) | > loss: 0.012334611266851425 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.1366602931268817) | > loader_time: 0.0046 (0.0046411680248964905)  --> TIME: 2025-02-25 11:18:56 -- STEP: 18840/36938 -- GLOBAL_STEP: 757600 | > loss_text_ce: 0.033121999353170395 (0.03442150052165288) | > loss_mel_ce: 2.3930437564849854 (nan) | > loss: 0.009627643041312695 (nan) | > current_lr: 5e-06 | > step_time: 0.1268 (0.13666479529342473) | > loader_time: 0.0046 (0.004641230635329158)  --> TIME: 2025-02-25 11:19:05 -- STEP: 18890/36938 -- GLOBAL_STEP: 757650 | > loss_text_ce: 0.032596245408058167 (0.034425535801080445) | > loss_mel_ce: 2.8822386264801025 (nan) | > loss: 0.011566806584596634 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.1366631359571891) | > loader_time: 0.0046 (0.004641500927898349)  --> TIME: 2025-02-25 11:19:15 -- STEP: 18940/36938 -- GLOBAL_STEP: 757700 | > loss_text_ce: 0.03668701648712158 (0.03442435718240845) | > loss_mel_ce: 2.203125 (nan) | > loss: 0.008888143114745617 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13666567653889775) | > loader_time: 0.0046 (0.0046415617372820945)  --> TIME: 2025-02-25 11:19:24 -- STEP: 18990/36938 -- GLOBAL_STEP: 757750 | > loss_text_ce: 0.031850021332502365 (0.03442413994307802) | > loss_mel_ce: 2.4331305027008057 (nan) | > loss: 0.009781669825315475 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13667162174046202) | > loader_time: 0.0046 (0.00464159141659046)  --> TIME: 2025-02-25 11:19:34 -- STEP: 19040/36938 -- GLOBAL_STEP: 757800 | > loss_text_ce: 0.037494055926799774 (0.034426587953882024) | > loss_mel_ce: 0.0699056014418602 (nan) | > loss: 0.0004261891299393028 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13669137068155554) | > loader_time: 0.0047 (0.004641916058143652)  --> TIME: 2025-02-25 11:19:43 -- STEP: 19090/36938 -- GLOBAL_STEP: 757850 | > loss_text_ce: 0.03199266642332077 (0.03442594920743168) | > loss_mel_ce: 1.3739256858825684 (nan) | > loss: 0.005579041317105293 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13669618468686875) | > loader_time: 0.0045 (0.004641988778251701)  --> TIME: 2025-02-25 11:19:53 -- STEP: 19140/36938 -- GLOBAL_STEP: 757900 | > loss_text_ce: 0.041631683707237244 (0.034429988791911365) | > loss_mel_ce: 3.0950145721435547 (nan) | > loss: 0.012447009794414043 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13668979847319188) | > loader_time: 0.0045 (0.00464194665508211)  --> TIME: 2025-02-25 11:20:02 -- STEP: 19190/36938 -- GLOBAL_STEP: 757950 | > loss_text_ce: 0.03767373412847519 (0.03443172363144109) | > loss_mel_ce: 2.04376220703125 (nan) | > loss: 0.008259667083621025 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13669076541117536) | > loader_time: 0.0047 (0.004641941315560002)  --> TIME: 2025-02-25 11:20:11 -- STEP: 19240/36938 -- GLOBAL_STEP: 758000 | > loss_text_ce: 0.029911324381828308 (0.034433070143874686) | > loss_mel_ce: 2.823427438735962 (nan) | > loss: 0.011322773061692715 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.13668703545899633) | > loader_time: 0.0047 (0.004641988607057666)  --> TIME: 2025-02-25 11:20:20 -- STEP: 19290/36938 -- GLOBAL_STEP: 758050 | > loss_text_ce: 0.036694277077913284 (0.03442807710734868) | > loss_mel_ce: 1.560638427734375 (nan) | > loss: 0.00633862242102623 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.1366803135508526) | > loader_time: 0.0046 (0.0046420176576739995)  --> TIME: 2025-02-25 11:20:30 -- STEP: 19340/36938 -- GLOBAL_STEP: 758100 | > loss_text_ce: 0.03324165195226669 (0.03442749133176594) | > loss_mel_ce: 2.8386425971984863 (nan) | > loss: 0.011396367102861404 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13668172427269296) | > loader_time: 0.0047 (0.004642107247566611)  --> TIME: 2025-02-25 11:20:39 -- STEP: 19390/36938 -- GLOBAL_STEP: 758150 | > loss_text_ce: 0.026690026745200157 (0.03443106901964024) | > loss_mel_ce: 2.7565371990203857 (nan) | > loss: 0.011044553481042385 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13667641654956603) | > loader_time: 0.0047 (0.004642090347392344)  --> TIME: 2025-02-25 11:20:48 -- STEP: 19440/36938 -- GLOBAL_STEP: 758200 | > loss_text_ce: 0.02972114086151123 (0.03443134536563311) | > loss_mel_ce: 3.1015396118164062 (nan) | > loss: 0.012425638735294342 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13666756171503328) | > loader_time: 0.0045 (0.004642170961992245)  --> TIME: 2025-02-25 11:20:57 -- STEP: 19490/36938 -- GLOBAL_STEP: 758250 | > loss_text_ce: 0.03202059119939804 (0.034434618143887666) | > loss_mel_ce: 2.9208199977874756 (nan) | > loss: 0.01171762216836214 (nan) | > current_lr: 5e-06 | > step_time: 0.1797 (0.1366633244329029) | > loader_time: 0.0048 (0.004642329587882578)  --> TIME: 2025-02-25 11:21:06 -- STEP: 19540/36938 -- GLOBAL_STEP: 758300 | > loss_text_ce: 0.042779892683029175 (0.0344351071342465) | > loss_mel_ce: 1.3838119506835938 (nan) | > loss: 0.005661079194396734 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13664974114546036) | > loader_time: 0.0046 (0.004642370645197051)  --> TIME: 2025-02-25 11:21:16 -- STEP: 19590/36938 -- GLOBAL_STEP: 758350 | > loss_text_ce: 0.03261615335941315 (0.03443432530091597) | > loss_mel_ce: 3.322732448577881 (nan) | > loss: 0.013314875774085522 (nan) | > current_lr: 5e-06 | > step_time: 0.1736 (0.13664043896535524) | > loader_time: 0.0048 (0.004642441809451237)  --> TIME: 2025-02-25 11:21:25 -- STEP: 19640/36938 -- GLOBAL_STEP: 758400 | > loss_text_ce: 0.03405049443244934 (0.034434603060910715) | > loss_mel_ce: 2.25 (nan) | > loss: 0.009063692763447762 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13665202005089314) | > loader_time: 0.0045 (0.0046425204655789086)  --> TIME: 2025-02-25 11:21:35 -- STEP: 19690/36938 -- GLOBAL_STEP: 758450 | > loss_text_ce: 0.03726319596171379 (0.03443480683954181) | > loss_mel_ce: 2.7339043617248535 (nan) | > loss: 0.010996697470545769 (nan) | > current_lr: 5e-06 | > step_time: 0.1896 (0.13666947861502537) | > loader_time: 0.0049 (0.004642661275326028)  --> TIME: 2025-02-25 11:21:44 -- STEP: 19740/36938 -- GLOBAL_STEP: 758500 | > loss_text_ce: 0.03693196550011635 (0.03443886600430078) | > loss_mel_ce: 2.23765230178833 (nan) | > loss: 0.00902612879872322 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13666032849836618) | > loader_time: 0.0045 (0.0046426495037059584)  --> TIME: 2025-02-25 11:21:53 -- STEP: 19790/36938 -- GLOBAL_STEP: 758550 | > loss_text_ce: 0.028414960950613022 (0.03444134528786826) | > loss_mel_ce: 2.257375717163086 (nan) | > loss: 0.009070598520338535 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13665315792860672) | > loader_time: 0.0047 (0.004642742459613789)  --> TIME: 2025-02-25 11:22:02 -- STEP: 19840/36938 -- GLOBAL_STEP: 758600 | > loss_text_ce: 0.02855067141354084 (0.034442888540375555) | > loss_mel_ce: 1.9887826442718506 (nan) | > loss: 0.008005291223526001 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13664721387528722) | > loader_time: 0.0046 (0.0046427374285075)  --> TIME: 2025-02-25 11:22:12 -- STEP: 19890/36938 -- GLOBAL_STEP: 758650 | > loss_text_ce: 0.04485969990491867 (0.034441122635140944) | > loss_mel_ce: 0.832463264465332 (nan) | > loss: 0.0034814404789358377 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.1366544601244817) | > loader_time: 0.0047 (0.004642837535441973)  --> TIME: 2025-02-25 11:22:21 -- STEP: 19940/36938 -- GLOBAL_STEP: 758700 | > loss_text_ce: 0.0460442453622818 (0.034441086946899895) | > loss_mel_ce: 2.958671808242798 (nan) | > loss: 0.011923477053642273 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.1366582356699743) | > loader_time: 0.0047 (0.004642948236245452)  --> TIME: 2025-02-25 11:22:31 -- STEP: 19990/36938 -- GLOBAL_STEP: 758750 | > loss_text_ce: 0.0323251336812973 (0.03444190009605673) | > loss_mel_ce: 3.0920562744140625 (nan) | > loss: 0.012398338876664639 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.13666842440595267) | > loader_time: 0.0048 (0.004642997305652037)  --> TIME: 2025-02-25 11:22:41 -- STEP: 20040/36938 -- GLOBAL_STEP: 758800 | > loss_text_ce: 0.030347933992743492 (0.034437593370592504) | > loss_mel_ce: 2.34836745262146 (nan) | > loss: 0.009439347311854362 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1366758671825292) | > loader_time: 0.0046 (0.0046430463562468615)  --> TIME: 2025-02-25 11:22:50 -- STEP: 20090/36938 -- GLOBAL_STEP: 758850 | > loss_text_ce: 0.045324619859457016 (0.034434546551432216) | > loss_mel_ce: 0.1664777547121048 (nan) | > loss: 0.0008404856780543923 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13667019667941233) | > loader_time: 0.0046 (0.004643004328650353)  --> TIME: 2025-02-25 11:22:59 -- STEP: 20140/36938 -- GLOBAL_STEP: 758900 | > loss_text_ce: 0.037559349089860916 (0.03443271330975808) | > loss_mel_ce: 2.8883278369903564 (nan) | > loss: 0.011610663495957851 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.1366654767653924) | > loader_time: 0.0045 (0.004642986943492122)  --> TIME: 2025-02-25 11:23:09 -- STEP: 20190/36938 -- GLOBAL_STEP: 758950 | > loss_text_ce: 0.04073375463485718 (0.03443517083193292) | > loss_mel_ce: 2.513582468032837 (nan) | > loss: 0.010136175900697708 (nan) | > current_lr: 5e-06 | > step_time: 0.1771 (0.13667573641880607) | > loader_time: 0.0048 (0.004643014600337176)  --> TIME: 2025-02-25 11:23:18 -- STEP: 20240/36938 -- GLOBAL_STEP: 759000 | > loss_text_ce: 0.03562026843428612 (0.03443255751939478) | > loss_mel_ce: 2.151611328125 (nan) | > loss: 0.00867949053645134 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13666318260398438) | > loader_time: 0.0046 (0.004642914618428054)  --> TIME: 2025-02-25 11:23:27 -- STEP: 20290/36938 -- GLOBAL_STEP: 759050 | > loss_text_ce: 0.03732002526521683 (0.034435696656755735) | > loss_mel_ce: 2.484998941421509 (nan) | > loss: 0.010009203106164932 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13665848390523896) | > loader_time: 0.0045 (0.004642967874631327)  --> TIME: 2025-02-25 11:23:36 -- STEP: 20340/36938 -- GLOBAL_STEP: 759100 | > loss_text_ce: 0.03144920617341995 (0.03443654715944367) | > loss_mel_ce: 2.83734130859375 (nan) | > loss: 0.011384090408682823 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13665592636447985) | > loader_time: 0.0046 (0.0046429786475827755)  --> TIME: 2025-02-25 11:23:45 -- STEP: 20390/36938 -- GLOBAL_STEP: 759150 | > loss_text_ce: 0.032214317470788956 (0.034434356725198914) | > loss_mel_ce: 2.5311617851257324 (nan) | > loss: 0.010172128677368164 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.13665414361640482) | > loader_time: 0.0048 (0.0046429551776102146)  --> TIME: 2025-02-25 11:23:55 -- STEP: 20440/36938 -- GLOBAL_STEP: 759200 | > loss_text_ce: 0.04433557763695717 (0.03443700783615383) | > loss_mel_ce: 2.547323703765869 (nan) | > loss: 0.01028436329215765 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13664394056027304) | > loader_time: 0.0045 (0.004642908412183105)  --> TIME: 2025-02-25 11:24:04 -- STEP: 20490/36938 -- GLOBAL_STEP: 759250 | > loss_text_ce: 0.04532841220498085 (0.0344370476279742) | > loss_mel_ce: 2.8949732780456543 (nan) | > loss: 0.01166786439716816 (nan) | > current_lr: 5e-06 | > step_time: 0.1881 (0.13663678280954786) | > loader_time: 0.005 (0.004642836823004989)  --> TIME: 2025-02-25 11:24:13 -- STEP: 20540/36938 -- GLOBAL_STEP: 759300 | > loss_text_ce: 0.03732917085289955 (0.03443834817584729) | > loss_mel_ce: 3.1922850608825684 (nan) | > loss: 0.0128159299492836 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1366414179588376) | > loader_time: 0.0045 (0.004642896259887312)  --> TIME: 2025-02-25 11:24:22 -- STEP: 20590/36938 -- GLOBAL_STEP: 759350 | > loss_text_ce: 0.0358758419752121 (0.0344381497582153) | > loss_mel_ce: 3.68257474899292 (nan) | > loss: 0.014755756594240665 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13663842906183166) | > loader_time: 0.0046 (0.00464290956549994)  --> TIME: 2025-02-25 11:24:31 -- STEP: 20640/36938 -- GLOBAL_STEP: 759400 | > loss_text_ce: 0.03335675224661827 (0.03443682645925854) | > loss_mel_ce: 2.8036017417907715 (nan) | > loss: 0.011257772333920002 (nan) | > current_lr: 5e-06 | > step_time: 0.1791 (0.13662657157857422) | > loader_time: 0.0048 (0.004642875769803691)  --> TIME: 2025-02-25 11:24:41 -- STEP: 20690/36938 -- GLOBAL_STEP: 759450 | > loss_text_ce: 0.020140251144766808 (0.03443743585221344) | > loss_mel_ce: 0.04295751824975014 (nan) | > loss: 0.00025038799503818154 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13662109516737614) | > loader_time: 0.0045 (0.004643158281177872)  --> TIME: 2025-02-25 11:24:50 -- STEP: 20740/36938 -- GLOBAL_STEP: 759500 | > loss_text_ce: 0.031225845217704773 (0.03444181922073293) | > loss_mel_ce: 2.9690141677856445 (nan) | > loss: 0.011905714869499207 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13662260555830275) | > loader_time: 0.0045 (0.004643252443094718)  --> TIME: 2025-02-25 11:24:59 -- STEP: 20790/36938 -- GLOBAL_STEP: 759550 | > loss_text_ce: 0.011710102669894695 (0.03444125000002194) | > loss_mel_ce: 0.09267578274011612 (nan) | > loss: 0.0004142297257203609 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13660399922962316) | > loader_time: 0.0045 (0.004643300956920565)  --> TIME: 2025-02-25 11:25:08 -- STEP: 20840/36938 -- GLOBAL_STEP: 759600 | > loss_text_ce: 0.03790375590324402 (0.034442844979050424) | > loss_mel_ce: 3.0740966796875 (nan) | > loss: 0.012349208816885948 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.1365887762260083) | > loader_time: 0.005 (0.004643324217732255)  --> TIME: 2025-02-25 11:25:17 -- STEP: 20890/36938 -- GLOBAL_STEP: 759650 | > loss_text_ce: 0.03397657349705696 (0.03443954023360938) | > loss_mel_ce: 1.2225226163864136 (nan) | > loss: 0.004986108280718327 (nan) | > current_lr: 5e-06 | > step_time: 0.1073 (0.13657310877535786) | > loader_time: 0.0044 (0.0046433411927357835)  --> TIME: 2025-02-25 11:25:26 -- STEP: 20940/36938 -- GLOBAL_STEP: 759700 | > loss_text_ce: 0.04547573998570442 (0.034443006429460216) | > loss_mel_ce: 2.934537649154663 (nan) | > loss: 0.01182545069605112 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13657289001069647) | > loader_time: 0.0047 (0.004643456801530163)  --> TIME: 2025-02-25 11:25:35 -- STEP: 20990/36938 -- GLOBAL_STEP: 759750 | > loss_text_ce: 0.03351534903049469 (0.034444797236255945) | > loss_mel_ce: 0.02197265625 (nan) | > loss: 0.00022019051539245993 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.1365618619877944) | > loader_time: 0.0045 (0.004643541724980579)  --> TIME: 2025-02-25 11:25:44 -- STEP: 21040/36938 -- GLOBAL_STEP: 759800 | > loss_text_ce: 0.029656177386641502 (0.03444866270438634) | > loss_mel_ce: 2.954789400100708 (nan) | > loss: 0.011843038722872734 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.13655992996783425) | > loader_time: 0.0047 (0.004643663356059391)  --> TIME: 2025-02-25 11:25:53 -- STEP: 21090/36938 -- GLOBAL_STEP: 759850 | > loss_text_ce: 0.04421341419219971 (0.03444751160450719) | > loss_mel_ce: 3.0774099826812744 (nan) | > loss: 0.012387395836412907 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13654125519079305) | > loader_time: 0.0046 (0.004643652664083263)  --> TIME: 2025-02-25 11:26:02 -- STEP: 21140/36938 -- GLOBAL_STEP: 759900 | > loss_text_ce: 0.03770661726593971 (0.03444545941437054) | > loss_mel_ce: 3.008861780166626 (nan) | > loss: 0.012089557945728302 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13652962949215186) | > loader_time: 0.0046 (0.004643632977665188)  --> TIME: 2025-02-25 11:26:11 -- STEP: 21190/36938 -- GLOBAL_STEP: 759950 | > loss_text_ce: 0.03762110695242882 (0.03444131154214532) | > loss_mel_ce: 3.1489531993865967 (nan) | > loss: 0.012645136564970016 (nan) | > current_lr: 5e-06 | > step_time: 0.1784 (0.13651787291837908) | > loader_time: 0.0048 (0.004643649726388795)  --> TIME: 2025-02-25 11:26:21 -- STEP: 21240/36938 -- GLOBAL_STEP: 760000 | > loss_text_ce: 0.03019426204264164 (0.034438921060244904) | > loss_mel_ce: 3.194075107574463 (nan) | > loss: 0.012794720940291882 (nan) | > current_lr: 5e-06 | > step_time: 0.1565 (0.13651957422102154) | > loader_time: 0.0046 (0.004643759877892746) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_760000.pth  --> TIME: 2025-02-25 11:26:36 -- STEP: 21290/36938 -- GLOBAL_STEP: 760050 | > loss_text_ce: 0.019381629303097725 (0.0344342950119968) | > loss_mel_ce: 2.350940465927124 (nan) | > loss: 0.009406040422618389 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13651878351110452) | > loader_time: 0.0047 (0.004643972561694229)  --> TIME: 2025-02-25 11:26:45 -- STEP: 21340/36938 -- GLOBAL_STEP: 760100 | > loss_text_ce: 0.022539351135492325 (0.03443324654271464) | > loss_mel_ce: 2.686129093170166 (nan) | > loss: 0.010748685337603092 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13650242109218394) | > loader_time: 0.0047 (0.004643994899363619)  --> TIME: 2025-02-25 11:26:54 -- STEP: 21390/36938 -- GLOBAL_STEP: 760150 | > loss_text_ce: 0.027050994336605072 (0.03443344186324436) | > loss_mel_ce: 1.9144636392593384 (nan) | > loss: 0.007704423274844885 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13648684838488168) | > loader_time: 0.0046 (0.004643883912459641)  --> TIME: 2025-02-25 11:27:03 -- STEP: 21440/36938 -- GLOBAL_STEP: 760200 | > loss_text_ce: 0.03832457587122917 (0.03443253622468532) | > loss_mel_ce: 2.9479622840881348 (nan) | > loss: 0.011850345879793167 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.136494158263973) | > loader_time: 0.0046 (0.00464394292724664)  --> TIME: 2025-02-25 11:27:13 -- STEP: 21490/36938 -- GLOBAL_STEP: 760250 | > loss_text_ce: 0.03560430556535721 (0.03443280606023139) | > loss_mel_ce: 2.7629449367523193 (nan) | > loss: 0.011105354875326157 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.13649599167179136) | > loader_time: 0.0045 (0.004643939405665815)  --> TIME: 2025-02-25 11:27:22 -- STEP: 21540/36938 -- GLOBAL_STEP: 760300 | > loss_text_ce: 0.0329725407063961 (0.03443296510369955) | > loss_mel_ce: 3.153447151184082 (nan) | > loss: 0.01264452375471592 (nan) | > current_lr: 5e-06 | > step_time: 0.1267 (0.13647934632270486) | > loader_time: 0.0045 (0.0046439142612361446)  --> TIME: 2025-02-25 11:27:31 -- STEP: 21590/36938 -- GLOBAL_STEP: 760350 | > loss_text_ce: 0.028682874515652657 (0.03443520376704654) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13648164123227866) | > loader_time: 0.0045 (0.004643975092670091)  --> TIME: 2025-02-25 11:27:41 -- STEP: 21640/36938 -- GLOBAL_STEP: 760400 | > loss_text_ce: 0.04853622987866402 (0.034435020242309954) | > loss_mel_ce: 2.9543869495391846 (nan) | > loss: 0.011916362680494785 (nan) | > current_lr: 5e-06 | > step_time: 0.1859 (0.1364936651782511) | > loader_time: 0.005 (0.004644004860118203)  --> TIME: 2025-02-25 11:27:50 -- STEP: 21690/36938 -- GLOBAL_STEP: 760450 | > loss_text_ce: 0.028905827552080154 (0.034435022738966245) | > loss_mel_ce: 3.2495265007019043 (nan) | > loss: 0.013009653426706791 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.136488251908138) | > loader_time: 0.0045 (0.004643893945376497)  --> TIME: 2025-02-25 11:27:59 -- STEP: 21740/36938 -- GLOBAL_STEP: 760500 | > loss_text_ce: 0.026288630440831184 (0.03443586282687409) | > loss_mel_ce: 2.2398037910461426 (nan) | > loss: 0.008992431685328484 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.1364903670116217) | > loader_time: 0.0046 (0.004643818535931939)  --> TIME: 2025-02-25 11:28:08 -- STEP: 21790/36938 -- GLOBAL_STEP: 760550 | > loss_text_ce: 0.03066411428153515 (0.034436918708302254) | > loss_mel_ce: 2.725712537765503 (nan) | > loss: 0.010938003659248352 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13649358729694827) | > loader_time: 0.0046 (0.004643797228893236)  --> TIME: 2025-02-25 11:28:18 -- STEP: 21840/36938 -- GLOBAL_STEP: 760600 | > loss_text_ce: 0.028576860204339027 (0.03443530109039798) | > loss_mel_ce: 2.76865816116333 (nan) | > loss: 0.01110013946890831 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.13649284073284843) | > loader_time: 0.0046 (0.0046437466755891174)  --> TIME: 2025-02-25 11:28:27 -- STEP: 21890/36938 -- GLOBAL_STEP: 760650 | > loss_text_ce: 0.027552323415875435 (0.034435869302420284) | > loss_mel_ce: 2.193021535873413 (nan) | > loss: 0.008811801671981812 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13650816615284758) | > loader_time: 0.0046 (0.004643745071651596)  --> TIME: 2025-02-25 11:28:36 -- STEP: 21940/36938 -- GLOBAL_STEP: 760700 | > loss_text_ce: 0.03723473846912384 (0.034436302104329845) | > loss_mel_ce: 1.9277304410934448 (nan) | > loss: 0.007797481492161751 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13649883094220924) | > loader_time: 0.0045 (0.004643646912240047)  --> TIME: 2025-02-25 11:28:46 -- STEP: 21990/36938 -- GLOBAL_STEP: 760750 | > loss_text_ce: 0.03336767479777336 (0.03443718797820758) | > loss_mel_ce: 3.494802951812744 (nan) | > loss: 0.01400067750364542 (nan) | > current_lr: 5e-06 | > step_time: 0.1883 (0.1365093713350991) | > loader_time: 0.0048 (0.004643635361668387)  --> TIME: 2025-02-25 11:28:55 -- STEP: 22040/36938 -- GLOBAL_STEP: 760800 | > loss_text_ce: 0.0318811796605587 (0.03444106284577146) | > loss_mel_ce: 2.6847012042999268 (nan) | > loss: 0.010780089534819126 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1364963063527379) | > loader_time: 0.0046 (0.004643568142356087)  --> TIME: 2025-02-25 11:29:04 -- STEP: 22090/36938 -- GLOBAL_STEP: 760850 | > loss_text_ce: 0.03824490308761597 (0.03444147325150806) | > loss_mel_ce: 1.4390767812728882 (nan) | > loss: 0.005862387828528881 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13649982252289494) | > loader_time: 0.0045 (0.0046436435229748885)  --> TIME: 2025-02-25 11:29:13 -- STEP: 22140/36938 -- GLOBAL_STEP: 760900 | > loss_text_ce: 0.03204810246825218 (0.03444094295149074) | > loss_mel_ce: 3.1128501892089844 (nan) | > loss: 0.012479755096137524 (nan) | > current_lr: 5e-06 | > step_time: 0.1705 (0.1364896737678493) | > loader_time: 0.0047 (0.0046436194481888175)  --> TIME: 2025-02-25 11:29:22 -- STEP: 22190/36938 -- GLOBAL_STEP: 760950 | > loss_text_ce: 0.027711588889360428 (0.034439036658276335) | > loss_mel_ce: 2.331490993499756 (nan) | > loss: 0.009361915290355682 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13647911693664255) | > loader_time: 0.0046 (0.004643678031240597)  --> TIME: 2025-02-25 11:29:32 -- STEP: 22240/36938 -- GLOBAL_STEP: 761000 | > loss_text_ce: 0.03145204111933708 (0.03444123089823558) | > loss_mel_ce: 3.12556791305542 (nan) | > loss: 0.012527856975793839 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13647868476754607) | > loader_time: 0.0045 (0.004643638571389267)  --> TIME: 2025-02-25 11:29:41 -- STEP: 22290/36938 -- GLOBAL_STEP: 761050 | > loss_text_ce: 0.035954974591732025 (0.034437399293421815) | > loss_mel_ce: 3.243817090988159 (nan) | > loss: 0.013014969415962696 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.13648304544579903) | > loader_time: 0.0048 (0.004643553765486452)  --> TIME: 2025-02-25 11:29:50 -- STEP: 22340/36938 -- GLOBAL_STEP: 761100 | > loss_text_ce: 0.03255865350365639 (0.034439422778873595) | > loss_mel_ce: 2.8843817710876465 (nan) | > loss: 0.01157516147941351 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13648241891750107) | > loader_time: 0.0045 (0.004643536884014138)  --> TIME: 2025-02-25 11:30:00 -- STEP: 22390/36938 -- GLOBAL_STEP: 761150 | > loss_text_ce: 0.038429662585258484 (0.03443969351157556) | > loss_mel_ce: 2.5609660148620605 (nan) | > loss: 0.010315063409507275 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13649168502225068) | > loader_time: 0.0046 (0.00464362368726366)  --> TIME: 2025-02-25 11:30:09 -- STEP: 22440/36938 -- GLOBAL_STEP: 761200 | > loss_text_ce: 0.024011705070734024 (0.03443650410957994) | > loss_mel_ce: 2.3872692584991455 (nan) | > loss: 0.009568575769662857 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13649186848956768) | > loader_time: 0.0046 (0.00464356721191268)  --> TIME: 2025-02-25 11:30:19 -- STEP: 22490/36938 -- GLOBAL_STEP: 761250 | > loss_text_ce: 0.030289091169834137 (0.03443748781601647) | > loss_mel_ce: 3.2640340328216553 (nan) | > loss: 0.01307271234691143 (nan) | > current_lr: 5e-06 | > step_time: 0.1822 (0.13649238111072914) | > loader_time: 0.0048 (0.004643475368002752)  --> TIME: 2025-02-25 11:30:28 -- STEP: 22540/36938 -- GLOBAL_STEP: 761300 | > loss_text_ce: 0.027686545625329018 (0.03443690620014848) | > loss_mel_ce: 3.325666666030884 (nan) | > loss: 0.013306958600878716 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13650347844837327) | > loader_time: 0.0045 (0.004643407551292388)  --> TIME: 2025-02-25 11:30:37 -- STEP: 22590/36938 -- GLOBAL_STEP: 761350 | > loss_text_ce: 0.037165895104408264 (0.03443840160836459) | > loss_mel_ce: 2.8330118656158447 (nan) | > loss: 0.0113895945250988 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13650463545623348) | > loader_time: 0.0046 (0.00464328103700849)  --> TIME: 2025-02-25 11:30:46 -- STEP: 22640/36938 -- GLOBAL_STEP: 761400 | > loss_text_ce: 0.03942001238465309 (0.03443691044467473) | > loss_mel_ce: 2.843108654022217 (nan) | > loss: 0.01143860723823309 (nan) | > current_lr: 5e-06 | > step_time: 0.1866 (0.13649622344086246) | > loader_time: 0.0049 (0.004643170572422394)  --> TIME: 2025-02-25 11:30:56 -- STEP: 22690/36938 -- GLOBAL_STEP: 761450 | > loss_text_ce: 0.03914914280176163 (0.034439193174520384) | > loss_mel_ce: 3.1763246059417725 (nan) | > loss: 0.012759816832840443 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.1365094433066195) | > loader_time: 0.0047 (0.004643464992097821)  --> TIME: 2025-02-25 11:31:06 -- STEP: 22740/36938 -- GLOBAL_STEP: 761500 | > loss_text_ce: 0.03158095106482506 (0.03443990636100081) | > loss_mel_ce: 2.791158437728882 (nan) | > loss: 0.011201347224414349 (nan) | > current_lr: 5e-06 | > step_time: 0.1516 (0.13650949244553864) | > loader_time: 0.0047 (0.004643517224119732)  --> TIME: 2025-02-25 11:31:15 -- STEP: 22790/36938 -- GLOBAL_STEP: 761550 | > loss_text_ce: 0.0425085574388504 (0.03443730907439491) | > loss_mel_ce: 2.8496601581573486 (nan) | > loss: 0.011476860381662846 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13651138898206028) | > loader_time: 0.0045 (0.004643537518014592)  --> TIME: 2025-02-25 11:31:24 -- STEP: 22840/36938 -- GLOBAL_STEP: 761600 | > loss_text_ce: 0.03656800091266632 (0.03443926103407388) | > loss_mel_ce: 0.0066986083984375 (nan) | > loss: 0.00017169291095342487 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13651283389632765) | > loader_time: 0.0047 (0.004643596513050013)  --> TIME: 2025-02-25 11:31:34 -- STEP: 22890/36938 -- GLOBAL_STEP: 761650 | > loss_text_ce: 0.033585552126169205 (0.0344417131622677) | > loss_mel_ce: 3.2566680908203125 (nan) | > loss: 0.013056563213467598 (nan) | > current_lr: 5e-06 | > step_time: 0.1809 (0.13652366830460141) | > loader_time: 0.0049 (0.004643719661907612)  --> TIME: 2025-02-25 11:31:43 -- STEP: 22940/36938 -- GLOBAL_STEP: 761700 | > loss_text_ce: 0.03511027246713638 (0.03444518172600294) | > loss_mel_ce: 3.5961132049560547 (nan) | > loss: 0.014409617520868778 (nan) | > current_lr: 5e-06 | > step_time: 0.2029 (0.13652431945127444) | > loader_time: 0.0049 (0.004643912375648837)  --> TIME: 2025-02-25 11:31:53 -- STEP: 22990/36938 -- GLOBAL_STEP: 761750 | > loss_text_ce: 0.03524397686123848 (0.0344472059237082) | > loss_mel_ce: 2.6602978706359863 (nan) | > loss: 0.010696595534682274 (nan) | > current_lr: 5e-06 | > step_time: 0.1116 (0.1365390789306608) | > loader_time: 0.0045 (0.0046440132393324556)  --> TIME: 2025-02-25 11:32:02 -- STEP: 23040/36938 -- GLOBAL_STEP: 761800 | > loss_text_ce: 0.0349186472594738 (0.034445309888946514) | > loss_mel_ce: 3.1782450675964355 (nan) | > loss: 0.012750649824738503 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13653200549177147) | > loader_time: 0.0045 (0.004644019974188663)  --> TIME: 2025-02-25 11:32:11 -- STEP: 23090/36938 -- GLOBAL_STEP: 761850 | > loss_text_ce: 0.017646698281168938 (0.03444590487583152) | > loss_mel_ce: 1.7099401950836182 (nan) | > loss: 0.006855503655970097 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1365251724520732) | > loader_time: 0.0045 (0.0046440619212200435)  --> TIME: 2025-02-25 11:32:20 -- STEP: 23140/36938 -- GLOBAL_STEP: 761900 | > loss_text_ce: 0.04492051154375076 (0.03444785936373335) | > loss_mel_ce: 2.3522989749908447 (nan) | > loss: 0.00951277557760477 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13651966137832622) | > loader_time: 0.0047 (0.004644115051526693)  --> TIME: 2025-02-25 11:32:29 -- STEP: 23190/36938 -- GLOBAL_STEP: 761950 | > loss_text_ce: 0.03521573171019554 (0.03444637274072398) | > loss_mel_ce: 1.9435220956802368 (nan) | > loss: 0.007852134294807911 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1365048222009265) | > loader_time: 0.0046 (0.004644163758038131)  --> TIME: 2025-02-25 11:32:39 -- STEP: 23240/36938 -- GLOBAL_STEP: 762000 | > loss_text_ce: 0.02605155110359192 (0.03444735994269954) | > loss_mel_ce: 2.9347000122070312 (nan) | > loss: 0.011749014258384705 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.136504791084544) | > loader_time: 0.0046 (0.0046443064528775444)  --> TIME: 2025-02-25 11:32:48 -- STEP: 23290/36938 -- GLOBAL_STEP: 762050 | > loss_text_ce: 0.03592052310705185 (0.03444540385463169) | > loss_mel_ce: 2.8533811569213867 (nan) | > loss: 0.011465484276413918 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.1365062535704863) | > loader_time: 0.0046 (0.0046443247293494316)  --> TIME: 2025-02-25 11:32:57 -- STEP: 23340/36938 -- GLOBAL_STEP: 762100 | > loss_text_ce: 0.03808670863509178 (0.03444596404427592) | > loss_mel_ce: 3.1231045722961426 (nan) | > loss: 0.012544410303235054 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.13650320013466472) | > loader_time: 0.0047 (0.0046442905142456345)  --> TIME: 2025-02-25 11:33:06 -- STEP: 23390/36938 -- GLOBAL_STEP: 762150 | > loss_text_ce: 0.03213432431221008 (0.03444755947748501) | > loss_mel_ce: 1.8945175409317017 (nan) | > loss: 0.0076454440131783485 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13649706356339053) | > loader_time: 0.0046 (0.004644250757625944)  --> TIME: 2025-02-25 11:33:16 -- STEP: 23440/36938 -- GLOBAL_STEP: 762200 | > loss_text_ce: 0.02554602175951004 (0.03444956079119042) | > loss_mel_ce: 2.8735134601593018 (nan) | > loss: 0.01150420494377613 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13649783291100806) | > loader_time: 0.0045 (0.004644242447797751)  --> TIME: 2025-02-25 11:33:25 -- STEP: 23490/36938 -- GLOBAL_STEP: 762250 | > loss_text_ce: 0.02812761813402176 (0.03444929681147419) | > loss_mel_ce: 3.4154052734375 (nan) | > loss: 0.013664813712239265 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13649371066160607) | > loader_time: 0.0045 (0.0046441596941828155)  --> TIME: 2025-02-25 11:33:35 -- STEP: 23540/36938 -- GLOBAL_STEP: 762300 | > loss_text_ce: 0.033631257712841034 (0.03444760930673004) | > loss_mel_ce: 1.501704216003418 (nan) | > loss: 0.006092601455748081 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13650256870354194) | > loader_time: 0.0045 (0.004644246838714675)  --> TIME: 2025-02-25 11:33:44 -- STEP: 23590/36938 -- GLOBAL_STEP: 762350 | > loss_text_ce: 0.031085984781384468 (0.03444928611680425) | > loss_mel_ce: 3.5104548931121826 (nan) | > loss: 0.01405373401939869 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13651067383868604) | > loader_time: 0.0044 (0.00464425428583744)  --> TIME: 2025-02-25 11:33:53 -- STEP: 23640/36938 -- GLOBAL_STEP: 762400 | > loss_text_ce: 0.027924325317144394 (0.03444970141916629) | > loss_mel_ce: 2.0835931301116943 (nan) | > loss: 0.008379037491977215 (nan) | > current_lr: 5e-06 | > step_time: 0.1529 (0.13650165783169033) | > loader_time: 0.0046 (0.004644179041615593)  --> TIME: 2025-02-25 11:34:02 -- STEP: 23690/36938 -- GLOBAL_STEP: 762450 | > loss_text_ce: 0.038054946810007095 (0.034451839594432224) | > loss_mel_ce: 3.139254570007324 (nan) | > loss: 0.012608371675014496 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13649202768890348) | > loader_time: 0.0046 (0.004644197429368741)  --> TIME: 2025-02-25 11:34:12 -- STEP: 23740/36938 -- GLOBAL_STEP: 762500 | > loss_text_ce: 0.03362403064966202 (0.03445207844468702) | > loss_mel_ce: 2.8258094787597656 (nan) | > loss: 0.011346958577632904 (nan) | > current_lr: 5e-06 | > step_time: 0.1814 (0.13649573396411316) | > loader_time: 0.0048 (0.004644343154010592)  --> TIME: 2025-02-25 11:34:21 -- STEP: 23790/36938 -- GLOBAL_STEP: 762550 | > loss_text_ce: 0.03005746379494667 (0.03445354736871626) | > loss_mel_ce: 3.325742721557617 (nan) | > loss: 0.013316667638719082 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.13649708249580095) | > loader_time: 0.0049 (0.004644746287323033)  --> TIME: 2025-02-25 11:34:30 -- STEP: 23840/36938 -- GLOBAL_STEP: 762600 | > loss_text_ce: 0.03783204406499863 (0.03445065287348103) | > loss_mel_ce: 1.8680908679962158 (nan) | > loss: 0.007563186343759298 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13649892298967384) | > loader_time: 0.0045 (0.0046448300448840045)  --> TIME: 2025-02-25 11:34:40 -- STEP: 23890/36938 -- GLOBAL_STEP: 762650 | > loss_text_ce: 0.03709015995264053 (0.034448092658027424) | > loss_mel_ce: 1.9837144613265991 (nan) | > loss: 0.008019066415727139 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1365078008539173) | > loader_time: 0.0044 (0.004644884151013236)  --> TIME: 2025-02-25 11:34:50 -- STEP: 23940/36938 -- GLOBAL_STEP: 762700 | > loss_text_ce: 0.04156055301427841 (0.034447675151011235) | > loss_mel_ce: 2.078068733215332 (nan) | > loss: 0.008411228656768799 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.1365154859017821) | > loader_time: 0.0047 (0.004644980944487681)  --> TIME: 2025-02-25 11:34:59 -- STEP: 23990/36938 -- GLOBAL_STEP: 762750 | > loss_text_ce: 0.04236766695976257 (0.03445202857240405) | > loss_mel_ce: 2.6534388065338135 (nan) | > loss: 0.010697645135223866 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13651156043847462) | > loader_time: 0.0047 (0.004644947779481736)  --> TIME: 2025-02-25 11:35:08 -- STEP: 24040/36938 -- GLOBAL_STEP: 762800 | > loss_text_ce: 0.037541378289461136 (0.03444965167417375) | > loss_mel_ce: 2.6044301986694336 (nan) | > loss: 0.010484014637768269 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13651280008417715) | > loader_time: 0.0045 (0.004644946944892111)  --> TIME: 2025-02-25 11:35:17 -- STEP: 24090/36938 -- GLOBAL_STEP: 762850 | > loss_text_ce: 0.04218688979744911 (0.034447963455991125) | > loss_mel_ce: 3.510849952697754 (nan) | > loss: 0.014099353924393654 (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.136517177898888) | > loader_time: 0.0048 (0.00464500278395308)  --> TIME: 2025-02-25 11:35:27 -- STEP: 24140/36938 -- GLOBAL_STEP: 762900 | > loss_text_ce: 0.04725850373506546 (0.03444734570719205) | > loss_mel_ce: 1.4919390678405762 (nan) | > loss: 0.0061079273000359535 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1365125930003136) | > loader_time: 0.0046 (0.004645130934553298)  --> TIME: 2025-02-25 11:35:36 -- STEP: 24190/36938 -- GLOBAL_STEP: 762950 | > loss_text_ce: 0.03480980172753334 (0.0344480045589473) | > loss_mel_ce: 0.8279440999031067 (nan) | > loss: 0.0034236270003020763 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13650489057475565) | > loader_time: 0.0045 (0.004645145387874254)  --> TIME: 2025-02-25 11:35:45 -- STEP: 24240/36938 -- GLOBAL_STEP: 763000 | > loss_text_ce: 0.02677149884402752 (0.03444639816165596) | > loss_mel_ce: 2.407013416290283 (nan) | > loss: 0.009657877497375011 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13651019594456998) | > loader_time: 0.0045 (0.004645186298751202)  --> TIME: 2025-02-25 11:35:55 -- STEP: 24290/36938 -- GLOBAL_STEP: 763050 | > loss_text_ce: 0.031846385449171066 (0.03444512560081769) | > loss_mel_ce: 2.2041468620300293 (nan) | > loss: 0.008872989565134048 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13651277069792342) | > loader_time: 0.0044 (0.004645142873259129)  --> TIME: 2025-02-25 11:36:04 -- STEP: 24340/36938 -- GLOBAL_STEP: 763100 | > loss_text_ce: 0.03564021363854408 (0.03444453791264466) | > loss_mel_ce: 2.9859814643859863 (nan) | > loss: 0.01199056301265955 (nan) | > current_lr: 5e-06 | > step_time: 0.1887 (0.1365169780021245) | > loader_time: 0.0048 (0.004645277969937103)  --> TIME: 2025-02-25 11:36:13 -- STEP: 24390/36938 -- GLOBAL_STEP: 763150 | > loss_text_ce: 0.023396238684654236 (0.034440709988908524) | > loss_mel_ce: 2.9214353561401367 (nan) | > loss: 0.011685840785503387 (nan) | > current_lr: 5e-06 | > step_time: 0.1434 (0.13651261617043578) | > loader_time: 0.0049 (0.004645451760380032)  --> TIME: 2025-02-25 11:36:22 -- STEP: 24440/36938 -- GLOBAL_STEP: 763200 | > loss_text_ce: 0.03374886140227318 (0.034443711009525924) | > loss_mel_ce: 1.6548137664794922 (nan) | > loss: 0.006700645666569471 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.1365037733033526) | > loader_time: 0.0046 (0.004645478013299295)  --> TIME: 2025-02-25 11:36:32 -- STEP: 24490/36938 -- GLOBAL_STEP: 763250 | > loss_text_ce: 0.03635505586862564 (0.03444208725115607) | > loss_mel_ce: 3.2901124954223633 (nan) | > loss: 0.01320026908069849 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.1364989521845166) | > loader_time: 0.0046 (0.004645421447535835)  --> TIME: 2025-02-25 11:36:41 -- STEP: 24540/36938 -- GLOBAL_STEP: 763300 | > loss_text_ce: 0.049511026591062546 (0.03444296501268737) | > loss_mel_ce: 1.8118469715118408 (nan) | > loss: 0.007386341691017151 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13649432200577236) | > loader_time: 0.0045 (0.004645417935496532)  --> TIME: 2025-02-25 11:36:50 -- STEP: 24590/36938 -- GLOBAL_STEP: 763350 | > loss_text_ce: 0.03329474478960037 (0.03444380596122541) | > loss_mel_ce: 3.5137360095977783 (nan) | > loss: 0.014075519517064095 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13649756646631472) | > loader_time: 0.0046 (0.00464537704121829)  --> TIME: 2025-02-25 11:37:00 -- STEP: 24640/36938 -- GLOBAL_STEP: 763400 | > loss_text_ce: 0.024073852226138115 (0.03444041936634128) | > loss_mel_ce: 2.7199809551239014 (nan) | > loss: 0.01088910736143589 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.13649882472954858) | > loader_time: 0.0047 (0.004645301198417491)  --> TIME: 2025-02-25 11:37:09 -- STEP: 24690/36938 -- GLOBAL_STEP: 763450 | > loss_text_ce: 0.02900519035756588 (0.03443898621964672) | > loss_mel_ce: 3.060800552368164 (nan) | > loss: 0.012261134572327137 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.1365041468393574) | > loader_time: 0.0046 (0.004645246646335704)  --> TIME: 2025-02-25 11:37:19 -- STEP: 24740/36938 -- GLOBAL_STEP: 763500 | > loss_text_ce: 0.02854064479470253 (0.03443849611245903) | > loss_mel_ce: 2.851423501968384 (nan) | > loss: 0.011428429745137691 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13651215831599386) | > loader_time: 0.0045 (0.004645252998740612)  --> TIME: 2025-02-25 11:37:28 -- STEP: 24790/36938 -- GLOBAL_STEP: 763550 | > loss_text_ce: 0.03163033723831177 (0.03443854594736169) | > loss_mel_ce: 2.6690826416015625 (nan) | > loss: 0.010717115364968777 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13651574362954552) | > loader_time: 0.0047 (0.0046452153541715726)  --> TIME: 2025-02-25 11:37:38 -- STEP: 24840/36938 -- GLOBAL_STEP: 763600 | > loss_text_ce: 0.018041091039776802 (0.03443645915626164) | > loss_mel_ce: 1.270583152770996 (nan) | > loss: 0.005113588646054268 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13652203943994384) | > loader_time: 0.0047 (0.0046451850789756555)  --> TIME: 2025-02-25 11:37:47 -- STEP: 24890/36938 -- GLOBAL_STEP: 763650 | > loss_text_ce: 0.03457483649253845 (0.0344346929038098) | > loss_mel_ce: 3.1205623149871826 (nan) | > loss: 0.012520385906100273 (nan) | > current_lr: 5e-06 | > step_time: 0.1818 (0.13652393371525393) | > loader_time: 0.0048 (0.004645119138681632)  --> TIME: 2025-02-25 11:37:56 -- STEP: 24940/36938 -- GLOBAL_STEP: 763700 | > loss_text_ce: 0.023089736700057983 (0.03443605015177575) | > loss_mel_ce: 2.2665622234344482 (nan) | > loss: 0.00908592063933611 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.13651422270795574) | > loader_time: 0.0047 (0.004645063901960898)  --> TIME: 2025-02-25 11:38:05 -- STEP: 24990/36938 -- GLOBAL_STEP: 763750 | > loss_text_ce: 0.03872308135032654 (0.03443723836490342) | > loss_mel_ce: 2.678962469100952 (nan) | > loss: 0.010784466750919819 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13650977446490162) | > loader_time: 0.0045 (0.0046451529391816565)  --> TIME: 2025-02-25 11:38:15 -- STEP: 25040/36938 -- GLOBAL_STEP: 763800 | > loss_text_ce: 0.03305351734161377 (0.034435851631586425) | > loss_mel_ce: 3.379141330718994 (nan) | > loss: 0.01354045607149601 (nan) | > current_lr: 5e-06 | > step_time: 0.1788 (0.13651377205460313) | > loader_time: 0.0049 (0.0046452898948718185)  --> TIME: 2025-02-25 11:38:24 -- STEP: 25090/36938 -- GLOBAL_STEP: 763850 | > loss_text_ce: 0.030860131606459618 (0.034437183386450604) | > loss_mel_ce: 0.004217758774757385 (nan) | > loss: 0.00013919798948336393 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13651246059078778) | > loader_time: 0.0045 (0.004645267564873147)  --> TIME: 2025-02-25 11:38:34 -- STEP: 25140/36938 -- GLOBAL_STEP: 763900 | > loss_text_ce: 0.03574007377028465 (0.03443835746825136) | > loss_mel_ce: 2.686145305633545 (nan) | > loss: 0.010801133699715137 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13652110762698658) | > loader_time: 0.0048 (0.0046454549878954254)  --> TIME: 2025-02-25 11:38:43 -- STEP: 25190/36938 -- GLOBAL_STEP: 763950 | > loss_text_ce: 0.0387033149600029 (0.0344362366484251) | > loss_mel_ce: 1.9512224197387695 (nan) | > loss: 0.007896531373262405 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13651047501596297) | > loader_time: 0.0046 (0.004645433412652586)  --> TIME: 2025-02-25 11:38:52 -- STEP: 25240/36938 -- GLOBAL_STEP: 764000 | > loss_text_ce: 0.04780155047774315 (0.034436013576096394) | > loss_mel_ce: 2.4611330032348633 (nan) | > loss: 0.009956089779734612 (nan) | > current_lr: 5e-06 | > step_time: 0.1507 (0.13651454030239538) | > loader_time: 0.0047 (0.004645469345494796)  --> TIME: 2025-02-25 11:39:01 -- STEP: 25290/36938 -- GLOBAL_STEP: 764050 | > loss_text_ce: 0.03143809735774994 (0.03443657427318407) | > loss_mel_ce: 2.68121600151062 (nan) | > loss: 0.010764501057565212 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13650418783469276) | > loader_time: 0.0047 (0.0046454394085221565)  --> TIME: 2025-02-25 11:39:10 -- STEP: 25340/36938 -- GLOBAL_STEP: 764100 | > loss_text_ce: 0.029822073876857758 (0.034435149030789335) | > loss_mel_ce: 2.7893600463867188 (nan) | > loss: 0.011187231168150902 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.1364995016012977) | > loader_time: 0.0046 (0.004645493497223772)  --> TIME: 2025-02-25 11:39:19 -- STEP: 25390/36938 -- GLOBAL_STEP: 764150 | > loss_text_ce: 0.03377682343125343 (0.03443451720099517) | > loss_mel_ce: 2.3408517837524414 (nan) | > loss: 0.009423129260540009 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13649648531725686) | > loader_time: 0.0046 (0.004645530273239181)  --> TIME: 2025-02-25 11:39:29 -- STEP: 25440/36938 -- GLOBAL_STEP: 764200 | > loss_text_ce: 0.03904271125793457 (0.03443446298896445) | > loss_mel_ce: 1.6604697704315186 (nan) | > loss: 0.006744097452610731 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13650100167442528) | > loader_time: 0.0044 (0.004645457338986912)  --> TIME: 2025-02-25 11:39:39 -- STEP: 25490/36938 -- GLOBAL_STEP: 764250 | > loss_text_ce: 0.030050231143832207 (0.03443323904005379) | > loss_mel_ce: 1.4144883155822754 (nan) | > loss: 0.005732296500355005 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13651057898460545) | > loader_time: 0.0045 (0.00464537702106223)  --> TIME: 2025-02-25 11:39:48 -- STEP: 25540/36938 -- GLOBAL_STEP: 764300 | > loss_text_ce: 0.03386906906962395 (0.03443344659062606) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.107 (0.13650666162570102) | > loader_time: 0.0045 (0.004645329653143229)  --> TIME: 2025-02-25 11:39:57 -- STEP: 25590/36938 -- GLOBAL_STEP: 764350 | > loss_text_ce: 0.03397684171795845 (0.034432037001502644) | > loss_mel_ce: 3.004591941833496 (nan) | > loss: 0.012057812884449959 (nan) | > current_lr: 5e-06 | > step_time: 0.1577 (0.13650684339919186) | > loader_time: 0.0046 (0.00464527164413017)  --> TIME: 2025-02-25 11:40:07 -- STEP: 25640/36938 -- GLOBAL_STEP: 764400 | > loss_text_ce: 0.03607422858476639 (0.034434227936017796) | > loss_mel_ce: 0.005859375 (nan) | > loss: 0.00016640320245642215 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.1365122886418189) | > loader_time: 0.0046 (0.004645390508699344)  --> TIME: 2025-02-25 11:40:15 -- STEP: 25690/36938 -- GLOBAL_STEP: 764450 | > loss_text_ce: 0.026838388293981552 (0.034435182677327425) | > loss_mel_ce: 2.7463948726654053 (nan) | > loss: 0.01100489404052496 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13649847339498059) | > loader_time: 0.0045 (0.0046453901838820096)  --> TIME: 2025-02-25 11:40:25 -- STEP: 25740/36938 -- GLOBAL_STEP: 764500 | > loss_text_ce: 0.03659750148653984 (0.034435252021027075) | > loss_mel_ce: 3.2533118724823 (nan) | > loss: 0.013055196963250637 (nan) | > current_lr: 5e-06 | > step_time: 0.1837 (0.13651665629762685) | > loader_time: 0.0048 (0.00464574004108811)  --> TIME: 2025-02-25 11:40:35 -- STEP: 25790/36938 -- GLOBAL_STEP: 764550 | > loss_text_ce: 0.030723005533218384 (0.03443589197253624) | > loss_mel_ce: 3.046865463256836 (nan) | > loss: 0.012212653644382954 (nan) | > current_lr: 5e-06 | > step_time: 0.1764 (0.1365156641945731) | > loader_time: 0.0048 (0.004645708495676632)  --> TIME: 2025-02-25 11:40:44 -- STEP: 25840/36938 -- GLOBAL_STEP: 764600 | > loss_text_ce: 0.025882383808493614 (0.03443294546034205) | > loss_mel_ce: 3.116583824157715 (nan) | > loss: 0.012470104731619358 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13650931454849036) | > loader_time: 0.0045 (0.00464567652796813)  --> TIME: 2025-02-25 11:40:53 -- STEP: 25890/36938 -- GLOBAL_STEP: 764650 | > loss_text_ce: 0.030032595619559288 (0.03443413755772638) | > loss_mel_ce: 3.4136462211608887 (nan) | > loss: 0.013665392994880676 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.13651107343848395) | > loader_time: 0.0047 (0.004645720252018262)  --> TIME: 2025-02-25 11:41:02 -- STEP: 25940/36938 -- GLOBAL_STEP: 764700 | > loss_text_ce: 0.035540852695703506 (0.03443496419449315) | > loss_mel_ce: 2.608525276184082 (nan) | > loss: 0.010492326691746712 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13651301629192297) | > loader_time: 0.0046 (0.004645715020855483)  --> TIME: 2025-02-25 11:41:12 -- STEP: 25990/36938 -- GLOBAL_STEP: 764750 | > loss_text_ce: 0.027719080448150635 (0.03443285633633204) | > loss_mel_ce: 2.8352973461151123 (nan) | > loss: 0.011361177079379559 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13651195123774587) | > loader_time: 0.0047 (0.004645784463548532)  --> TIME: 2025-02-25 11:41:21 -- STEP: 26040/36938 -- GLOBAL_STEP: 764800 | > loss_text_ce: 0.034099604934453964 (0.03443231540213493) | > loss_mel_ce: 3.770810604095459 (nan) | > loss: 0.015098851174116135 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13651556875299287) | > loader_time: 0.0046 (0.004645819808664045)  --> TIME: 2025-02-25 11:41:31 -- STEP: 26090/36938 -- GLOBAL_STEP: 764850 | > loss_text_ce: 0.02659807913005352 (0.03443080989312664) | > loss_mel_ce: 2.7585463523864746 (nan) | > loss: 0.011052160523831844 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13652319123161122) | > loader_time: 0.0045 (0.004645819406301047)  --> TIME: 2025-02-25 11:41:40 -- STEP: 26140/36938 -- GLOBAL_STEP: 764900 | > loss_text_ce: 0.025539027526974678 (0.034428804101530694) | > loss_mel_ce: 2.906257390975952 (nan) | > loss: 0.011634113267064095 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.1365275114934329) | > loader_time: 0.0046 (0.004645882988122007)  --> TIME: 2025-02-25 11:41:50 -- STEP: 26190/36938 -- GLOBAL_STEP: 764950 | > loss_text_ce: 0.035656385123729706 (0.03442661145704843) | > loss_mel_ce: 3.2714273929595947 (nan) | > loss: 0.013123349286615849 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13652990214831168) | > loader_time: 0.0048 (0.004645880300063769)  --> TIME: 2025-02-25 11:41:59 -- STEP: 26240/36938 -- GLOBAL_STEP: 765000 | > loss_text_ce: 0.03541797399520874 (0.03442584960852751) | > loss_mel_ce: 2.5290651321411133 (nan) | > loss: 0.01017652079463005 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13652534572998318) | > loader_time: 0.0046 (0.004645896621230173)  --> TIME: 2025-02-25 11:42:09 -- STEP: 26290/36938 -- GLOBAL_STEP: 765050 | > loss_text_ce: 0.03234734386205673 (0.03442473402268084) | > loss_mel_ce: 2.330277919769287 (nan) | > loss: 0.009375497698783875 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13653285736455564) | > loader_time: 0.0045 (0.004645947695413261)  --> TIME: 2025-02-25 11:42:18 -- STEP: 26340/36938 -- GLOBAL_STEP: 765100 | > loss_text_ce: 0.03097725845873356 (0.0344279826824159) | > loss_mel_ce: 2.3835113048553467 (nan) | > loss: 0.009581304155290127 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13652680229760647) | > loader_time: 0.0045 (0.004646034908729154)  --> TIME: 2025-02-25 11:42:27 -- STEP: 26390/36938 -- GLOBAL_STEP: 765150 | > loss_text_ce: 0.026445647701621056 (0.034426481617694356) | > loss_mel_ce: 2.380584716796875 (nan) | > loss: 0.009551708586513996 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1365219317949749) | > loader_time: 0.0044 (0.004646035160427522)  --> TIME: 2025-02-25 11:42:36 -- STEP: 26440/36938 -- GLOBAL_STEP: 765200 | > loss_text_ce: 0.03381624445319176 (0.034428056956652055) | > loss_mel_ce: 3.2405316829681396 (nan) | > loss: 0.012993445619940758 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13653385294728268) | > loader_time: 0.0046 (0.0046460435989224115)  --> TIME: 2025-02-25 11:42:46 -- STEP: 26490/36938 -- GLOBAL_STEP: 765250 | > loss_text_ce: 0.038115959614515305 (0.03442890783659367) | > loss_mel_ce: 2.716317892074585 (nan) | > loss: 0.010930294170975685 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13654125324507174) | > loader_time: 0.0046 (0.004646161683508484)  --> TIME: 2025-02-25 11:42:55 -- STEP: 26540/36938 -- GLOBAL_STEP: 765300 | > loss_text_ce: 0.03900045529007912 (0.034430064418835296) | > loss_mel_ce: 3.2054646015167236 (nan) | > loss: 0.012874862179160118 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13652907241621653) | > loader_time: 0.0046 (0.004646140089947754)  --> TIME: 2025-02-25 11:43:04 -- STEP: 26590/36938 -- GLOBAL_STEP: 765350 | > loss_text_ce: 0.03461623564362526 (0.034430498076478834) | > loss_mel_ce: 4.166015625 (nan) | > loss: 0.016669176518917084 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.1365189238301179) | > loader_time: 0.0047 (0.004646091507807323)  --> TIME: 2025-02-25 11:43:13 -- STEP: 26640/36938 -- GLOBAL_STEP: 765400 | > loss_text_ce: 0.03337733447551727 (0.03443015861316578) | > loss_mel_ce: 2.8741261959075928 (nan) | > loss: 0.011537712998688221 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13652277405376714) | > loader_time: 0.0046 (0.004646119690156204)  --> TIME: 2025-02-25 11:43:23 -- STEP: 26690/36938 -- GLOBAL_STEP: 765450 | > loss_text_ce: 0.019130665808916092 (0.03442942813498035) | > loss_mel_ce: 3.414407253265381 (nan) | > loss: 0.013625151477754116 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.13652773361699316) | > loader_time: 0.0046 (0.004646127998449245)  --> TIME: 2025-02-25 11:43:32 -- STEP: 26740/36938 -- GLOBAL_STEP: 765500 | > loss_text_ce: 0.023927858099341393 (0.0344292109771566) | > loss_mel_ce: 1.8507047891616821 (nan) | > loss: 0.007439018692821264 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1365262404518886) | > loader_time: 0.0046 (0.004646107645826552)  --> TIME: 2025-02-25 11:43:41 -- STEP: 26790/36938 -- GLOBAL_STEP: 765550 | > loss_text_ce: 0.03322789818048477 (0.03442887951296275) | > loss_mel_ce: 3.023386001586914 (nan) | > loss: 0.012129421345889568 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13652601921100216) | > loader_time: 0.0047 (0.0046461388351936)  --> TIME: 2025-02-25 11:43:51 -- STEP: 26840/36938 -- GLOBAL_STEP: 765600 | > loss_text_ce: 0.03446519747376442 (0.03442946685410082) | > loss_mel_ce: 2.9628119468688965 (nan) | > loss: 0.01189395785331726 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13653465796334352) | > loader_time: 0.0046 (0.004646188606981378)  --> TIME: 2025-02-25 11:44:00 -- STEP: 26890/36938 -- GLOBAL_STEP: 765650 | > loss_text_ce: 0.029101833701133728 (0.03442950776022089) | > loss_mel_ce: 0.15020751953125 (nan) | > loss: 0.0007115451153367758 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13653800222411897) | > loader_time: 0.0045 (0.004646141434212349)  --> TIME: 2025-02-25 11:44:10 -- STEP: 26940/36938 -- GLOBAL_STEP: 765700 | > loss_text_ce: 0.032483212649822235 (0.03442750016618699) | > loss_mel_ce: 2.93310546875 (nan) | > loss: 0.011768209747970104 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13653570225969958) | > loader_time: 0.0047 (0.004646155474888642)  --> TIME: 2025-02-25 11:44:19 -- STEP: 26990/36938 -- GLOBAL_STEP: 765750 | > loss_text_ce: 0.03080308996140957 (0.03442837579452573) | > loss_mel_ce: 3.181150197982788 (nan) | > loss: 0.012745847925543785 (nan) | > current_lr: 5e-06 | > step_time: 0.1878 (0.13652825325495732) | > loader_time: 0.005 (0.004646124889250828)  --> TIME: 2025-02-25 11:44:28 -- STEP: 27040/36938 -- GLOBAL_STEP: 765800 | > loss_text_ce: 0.02468707226216793 (0.034426706374868166) | > loss_mel_ce: 2.6222307682037354 (nan) | > loss: 0.010503643192350864 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.13653590278865468) | > loader_time: 0.0047 (0.004646191785674137)  --> TIME: 2025-02-25 11:44:38 -- STEP: 27090/36938 -- GLOBAL_STEP: 765850 | > loss_text_ce: 0.04101952910423279 (0.034428599699932956) | > loss_mel_ce: 3.495800733566284 (nan) | > loss: 0.014035001397132874 (nan) | > current_lr: 5e-06 | > step_time: 0.1732 (0.136547180734117) | > loader_time: 0.0047 (0.004646224304951188)  --> TIME: 2025-02-25 11:44:47 -- STEP: 27140/36938 -- GLOBAL_STEP: 765900 | > loss_text_ce: 0.031797196716070175 (0.03442911373014866) | > loss_mel_ce: 2.8793067932128906 (nan) | > loss: 0.011552000418305397 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13654721556293292) | > loader_time: 0.0046 (0.004646209951877256)  --> TIME: 2025-02-25 11:44:57 -- STEP: 27190/36938 -- GLOBAL_STEP: 765950 | > loss_text_ce: 0.04111010953783989 (0.03443154421179082) | > loss_mel_ce: 2.7739453315734863 (nan) | > loss: 0.011170855723321438 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.1365527438871925) | > loader_time: 0.0045 (0.004646174983971272)  --> TIME: 2025-02-25 11:45:06 -- STEP: 27240/36938 -- GLOBAL_STEP: 766000 | > loss_text_ce: 0.03844958171248436 (0.03442898902402057) | > loss_mel_ce: 2.04896879196167 (nan) | > loss: 0.008283406496047974 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13655134042979475) | > loader_time: 0.0047 (0.004646142175019065)  --> TIME: 2025-02-25 11:45:15 -- STEP: 27290/36938 -- GLOBAL_STEP: 766050 | > loss_text_ce: 0.030377868562936783 (0.03443153103852442) | > loss_mel_ce: 2.531205415725708 (nan) | > loss: 0.010165013372898102 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13654688907307508) | > loader_time: 0.0046 (0.004646094249866538)  --> TIME: 2025-02-25 11:45:25 -- STEP: 27340/36938 -- GLOBAL_STEP: 766100 | > loss_text_ce: 0.04449125751852989 (0.03443338454141675) | > loss_mel_ce: 2.2655510902404785 (nan) | > loss: 0.009166835807263851 (nan) | > current_lr: 5e-06 | > step_time: 0.1506 (0.1365456927755769) | > loader_time: 0.0046 (0.00464601765257998)  --> TIME: 2025-02-25 11:45:34 -- STEP: 27390/36938 -- GLOBAL_STEP: 766150 | > loss_text_ce: 0.03170491009950638 (0.034431768177086035) | > loss_mel_ce: 2.87884783744812 (nan) | > loss: 0.011549812741577625 (nan) | > current_lr: 5e-06 | > step_time: 0.1687 (0.13653960437746934) | > loader_time: 0.0048 (0.004645942483952825)  --> TIME: 2025-02-25 11:45:43 -- STEP: 27440/36938 -- GLOBAL_STEP: 766200 | > loss_text_ce: 0.03444690629839897 (0.0344312788897911) | > loss_mel_ce: 2.2553365230560303 (nan) | > loss: 0.009086443111300468 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13653942045074297) | > loader_time: 0.0047 (0.004645936047718072)  --> TIME: 2025-02-25 11:45:52 -- STEP: 27490/36938 -- GLOBAL_STEP: 766250 | > loss_text_ce: 0.03416561335325241 (0.034429906156773285) | > loss_mel_ce: 2.8814263343811035 (nan) | > loss: 0.011569810099899769 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.1365370678962396) | > loader_time: 0.0046 (0.004645901708092952)  --> TIME: 2025-02-25 11:46:02 -- STEP: 27540/36938 -- GLOBAL_STEP: 766300 | > loss_text_ce: 0.03978604078292847 (0.03443162611262311) | > loss_mel_ce: 3.3543598651885986 (nan) | > loss: 0.013468833640217781 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13654074573655486) | > loader_time: 0.0045 (0.004645821402362072)  --> TIME: 2025-02-25 11:46:11 -- STEP: 27590/36938 -- GLOBAL_STEP: 766350 | > loss_text_ce: 0.03206302225589752 (0.03443322773579271) | > loss_mel_ce: 2.715909242630005 (nan) | > loss: 0.010904652066528797 (nan) | > current_lr: 5e-06 | > step_time: 0.1794 (0.13654167884586196) | > loader_time: 0.0048 (0.004645723180089941)  --> TIME: 2025-02-25 11:46:20 -- STEP: 27640/36938 -- GLOBAL_STEP: 766400 | > loss_text_ce: 0.027518756687641144 (0.03443406474262387) | > loss_mel_ce: 1.6447219848632812 (nan) | > loss: 0.006635876372456551 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13653913644909466) | > loader_time: 0.0045 (0.004645602023377276)  --> TIME: 2025-02-25 11:46:29 -- STEP: 27690/36938 -- GLOBAL_STEP: 766450 | > loss_text_ce: 0.039470307528972626 (0.03443059287464157) | > loss_mel_ce: 3.080151319503784 (nan) | > loss: 0.012379450723528862 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13653685841675864) | > loader_time: 0.0047 (0.004645659192540939)  --> TIME: 2025-02-25 11:46:39 -- STEP: 27740/36938 -- GLOBAL_STEP: 766500 | > loss_text_ce: 0.03519389405846596 (0.034429802366706334) | > loss_mel_ce: 3.557421922683716 (nan) | > loss: 0.014256413094699383 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.1365369028983235) | > loader_time: 0.0045 (0.004645659894331307)  --> TIME: 2025-02-25 11:46:48 -- STEP: 27790/36938 -- GLOBAL_STEP: 766550 | > loss_text_ce: 0.03149609640240669 (0.03443157614908175) | > loss_mel_ce: 2.403609037399292 (nan) | > loss: 0.009663116186857224 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13653828626744688) | > loader_time: 0.0046 (0.004645642714347732)  --> TIME: 2025-02-25 11:46:57 -- STEP: 27840/36938 -- GLOBAL_STEP: 766600 | > loss_text_ce: 0.03194218501448631 (0.03443073374951858) | > loss_mel_ce: 0.6276900172233582 (nan) | > loss: 0.0026175882667303085 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13653964716298939) | > loader_time: 0.0046 (0.004645656700106889)  --> TIME: 2025-02-25 11:47:07 -- STEP: 27890/36938 -- GLOBAL_STEP: 766650 | > loss_text_ce: 0.02273787185549736 (0.03442751149633251) | > loss_mel_ce: 1.7799649238586426 (nan) | > loss: 0.007153582759201527 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1365344822641864) | > loader_time: 0.0044 (0.004645702667072212)  --> TIME: 2025-02-25 11:47:16 -- STEP: 27940/36938 -- GLOBAL_STEP: 766700 | > loss_text_ce: 0.050631895661354065 (0.03442740050836489) | > loss_mel_ce: 3.391200065612793 (nan) | > loss: 0.013658064417541027 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13654624511279223) | > loader_time: 0.0045 (0.0046456620193159395)  --> TIME: 2025-02-25 11:47:26 -- STEP: 27990/36938 -- GLOBAL_STEP: 766750 | > loss_text_ce: 0.04266183450818062 (0.03442639664468325) | > loss_mel_ce: 3.000378131866455 (nan) | > loss: 0.012075556442141533 (nan) | > current_lr: 5e-06 | > step_time: 0.17 (0.13654294836815128) | > loader_time: 0.0048 (0.004645636346605106)  --> TIME: 2025-02-25 11:47:35 -- STEP: 28040/36938 -- GLOBAL_STEP: 766800 | > loss_text_ce: 0.035860754549503326 (0.03442646623021142) | > loss_mel_ce: 3.3837618827819824 (nan) | > loss: 0.01356993243098259 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.1365466075684998) | > loader_time: 0.0047 (0.004645604856004065)  --> TIME: 2025-02-25 11:47:44 -- STEP: 28090/36938 -- GLOBAL_STEP: 766850 | > loss_text_ce: 0.03970196470618248 (0.034428239834616625) | > loss_mel_ce: 2.282144784927368 (nan) | > loss: 0.009213677607476711 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.136540412266258) | > loader_time: 0.0045 (0.004645564862526638)  --> TIME: 2025-02-25 11:47:53 -- STEP: 28140/36938 -- GLOBAL_STEP: 766900 | > loss_text_ce: 0.03417326137423515 (0.03442925208389855) | > loss_mel_ce: 2.5991086959838867 (nan) | > loss: 0.01044953241944313 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.13654064249297715) | > loader_time: 0.0047 (0.00464554008390351)  --> TIME: 2025-02-25 11:48:03 -- STEP: 28190/36938 -- GLOBAL_STEP: 766950 | > loss_text_ce: 0.03550694137811661 (0.03442600087985224) | > loss_mel_ce: 2.7677605152130127 (nan) | > loss: 0.011124078184366226 (nan) | > current_lr: 5e-06 | > step_time: 0.1514 (0.1365422956230548) | > loader_time: 0.0046 (0.004645532409787067)  --> TIME: 2025-02-25 11:48:12 -- STEP: 28240/36938 -- GLOBAL_STEP: 767000 | > loss_text_ce: 0.03743623569607735 (0.03442416513841951) | > loss_mel_ce: 3.1112685203552246 (nan) | > loss: 0.012494861148297787 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.13653541100092811) | > loader_time: 0.0047 (0.004645478911170219)  --> TIME: 2025-02-25 11:48:22 -- STEP: 28290/36938 -- GLOBAL_STEP: 767050 | > loss_text_ce: 0.036761749535799026 (0.034421414034042856) | > loss_mel_ce: 3.1896793842315674 (nan) | > loss: 0.012803338468074799 (nan) | > current_lr: 5e-06 | > step_time: 0.1724 (0.13654444213468145) | > loader_time: 0.005 (0.004645479580841135)  --> TIME: 2025-02-25 11:48:31 -- STEP: 28340/36938 -- GLOBAL_STEP: 767100 | > loss_text_ce: 0.037926867604255676 (0.0344231139115291) | > loss_mel_ce: 3.6953125 (nan) | > loss: 0.014814442954957485 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.1365390973605998) | > loader_time: 0.0045 (0.004645422275586303)  --> TIME: 2025-02-25 11:48:40 -- STEP: 28390/36938 -- GLOBAL_STEP: 767150 | > loss_text_ce: 0.041693124920129776 (0.03442313171694853) | > loss_mel_ce: 2.0360660552978516 (nan) | > loss: 0.008245076984167099 (nan) | > current_lr: 5e-06 | > step_time: 0.156 (0.1365369107446953) | > loader_time: 0.0049 (0.004645619572111752)  --> TIME: 2025-02-25 11:48:49 -- STEP: 28440/36938 -- GLOBAL_STEP: 767200 | > loss_text_ce: 0.034509774297475815 (0.03442391032659568) | > loss_mel_ce: 3.411052942276001 (nan) | > loss: 0.01367286778986454 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13653393091867172) | > loader_time: 0.0047 (0.004645501293042893)  --> TIME: 2025-02-25 11:48:58 -- STEP: 28490/36938 -- GLOBAL_STEP: 767250 | > loss_text_ce: 0.03614852949976921 (0.03442409827895992) | > loss_mel_ce: 2.799955129623413 (nan) | > loss: 0.011254380457103252 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.13652932902477152) | > loader_time: 0.0047 (0.004645453080163599)  --> TIME: 2025-02-25 11:49:08 -- STEP: 28540/36938 -- GLOBAL_STEP: 767300 | > loss_text_ce: 0.03574730455875397 (0.03442675353464519) | > loss_mel_ce: 2.0791382789611816 (nan) | > loss: 0.008392403833568096 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13653242279786648) | > loader_time: 0.0045 (0.004645413473593094)  --> TIME: 2025-02-25 11:49:17 -- STEP: 28590/36938 -- GLOBAL_STEP: 767350 | > loss_text_ce: 0.032594047486782074 (0.0344242816004577) | > loss_mel_ce: 2.6497323513031006 (nan) | > loss: 0.010644152760505676 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13652955400814626) | > loader_time: 0.0045 (0.004645285818200729)  --> TIME: 2025-02-25 11:49:26 -- STEP: 28640/36938 -- GLOBAL_STEP: 767400 | > loss_text_ce: 0.03499592840671539 (0.03442260399694941) | > loss_mel_ce: 2.7210352420806885 (nan) | > loss: 0.010936632752418518 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13653011736590218) | > loader_time: 0.0046 (0.0046451887271923885)  --> TIME: 2025-02-25 11:49:36 -- STEP: 28690/36938 -- GLOBAL_STEP: 767450 | > loss_text_ce: 0.02852593921124935 (0.03442122423940228) | > loss_mel_ce: 2.7752511501312256 (nan) | > loss: 0.011126100085675716 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.1365323215878864) | > loader_time: 0.0046 (0.004645174087319667)  --> TIME: 2025-02-25 11:49:45 -- STEP: 28740/36938 -- GLOBAL_STEP: 767500 | > loss_text_ce: 0.043211936950683594 (0.03442195298256491) | > loss_mel_ce: 1.758636474609375 (nan) | > loss: 0.007150192745029926 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13654018015851663) | > loader_time: 0.0046 (0.004645157134109863)  --> TIME: 2025-02-25 11:49:54 -- STEP: 28790/36938 -- GLOBAL_STEP: 767550 | > loss_text_ce: 0.036284103989601135 (0.03442331492453162) | > loss_mel_ce: 2.5946173667907715 (nan) | > loss: 0.010440086014568806 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13653141531359717) | > loader_time: 0.0046 (0.004645143775900207)  --> TIME: 2025-02-25 11:50:04 -- STEP: 28840/36938 -- GLOBAL_STEP: 767600 | > loss_text_ce: 0.03858036547899246 (0.03442144159643559) | > loss_mel_ce: 2.1889760494232178 (nan) | > loss: 0.008839510381221771 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13653744193783518) | > loader_time: 0.0049 (0.0046451596049759975)  --> TIME: 2025-02-25 11:50:13 -- STEP: 28890/36938 -- GLOBAL_STEP: 767650 | > loss_text_ce: 0.036511488258838654 (0.03442225865258957) | > loss_mel_ce: 2.1792259216308594 (nan) | > loss: 0.008792608976364136 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.1365308624025293) | > loader_time: 0.0045 (0.004645053339318132)  --> TIME: 2025-02-25 11:50:23 -- STEP: 28940/36938 -- GLOBAL_STEP: 767700 | > loss_text_ce: 0.03559718653559685 (0.03442218908312819) | > loss_mel_ce: 3.628676414489746 (nan) | > loss: 0.014540769159793854 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.1365416694774916) | > loader_time: 0.0046 (0.004645011700180059)  --> TIME: 2025-02-25 11:50:32 -- STEP: 28990/36938 -- GLOBAL_STEP: 767750 | > loss_text_ce: 0.03596067801117897 (0.03442200387565606) | > loss_mel_ce: 3.0254030227661133 (nan) | > loss: 0.012148269452154636 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13654605139120263) | > loader_time: 0.0049 (0.00464501445891325)  --> TIME: 2025-02-25 11:50:42 -- STEP: 29040/36938 -- GLOBAL_STEP: 767800 | > loss_text_ce: 0.0326237753033638 (0.03442275273689339) | > loss_mel_ce: 2.6638855934143066 (nan) | > loss: 0.010700434446334839 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.1365458204151852) | > loader_time: 0.0046 (0.004644954557589607)  --> TIME: 2025-02-25 11:50:51 -- STEP: 29090/36938 -- GLOBAL_STEP: 767850 | > loss_text_ce: 0.023185137659311295 (0.0344232984879669) | > loss_mel_ce: 0.884888768196106 (nan) | > loss: 0.003603468183428049 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13655175691455876) | > loader_time: 0.0043 (0.004644955200360557)  --> TIME: 2025-02-25 11:51:00 -- STEP: 29140/36938 -- GLOBAL_STEP: 767900 | > loss_text_ce: 0.025625526905059814 (0.03442144060939537) | > loss_mel_ce: 1.9385963678359985 (nan) | > loss: 0.007794531993567944 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13654670279240497) | > loader_time: 0.0045 (0.004644848879838382)  --> TIME: 2025-02-25 11:51:10 -- STEP: 29190/36938 -- GLOBAL_STEP: 767950 | > loss_text_ce: 0.029574807733297348 (0.03442211960553727) | > loss_mel_ce: 2.5035667419433594 (nan) | > loss: 0.01005215011537075 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13655713098675742) | > loader_time: 0.0046 (0.004644840798503791)  --> TIME: 2025-02-25 11:51:20 -- STEP: 29240/36938 -- GLOBAL_STEP: 768000 | > loss_text_ce: 0.03333744406700134 (0.034423723551744195) | > loss_mel_ce: 2.9594357013702393 (nan) | > loss: 0.01187608391046524 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.13656987141812824) | > loader_time: 0.0045 (0.004644870236210245)  --> TIME: 2025-02-25 11:51:29 -- STEP: 29290/36938 -- GLOBAL_STEP: 768050 | > loss_text_ce: 0.019792484119534492 (0.034424125386250115) | > loss_mel_ce: 1.4565788507461548 (nan) | > loss: 0.005858616437762976 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13657516866627417) | > loader_time: 0.0045 (0.004644861999490896)  --> TIME: 2025-02-25 11:51:39 -- STEP: 29340/36938 -- GLOBAL_STEP: 768100 | > loss_text_ce: 0.00996752455830574 (0.03442289149807676) | > loss_mel_ce: 2.0210156440734863 (nan) | > loss: 0.008059457875788212 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1365763205773353) | > loader_time: 0.0044 (0.004644816264703164)  --> TIME: 2025-02-25 11:51:48 -- STEP: 29390/36938 -- GLOBAL_STEP: 768150 | > loss_text_ce: 0.02563663199543953 (0.03442430971478704) | > loss_mel_ce: 2.5872910022735596 (nan) | > loss: 0.010368761606514454 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13657898141154345) | > loader_time: 0.0045 (0.0046447953304980145)  --> TIME: 2025-02-25 11:51:57 -- STEP: 29440/36938 -- GLOBAL_STEP: 768200 | > loss_text_ce: 0.035027358680963516 (0.0344246216885159) | > loss_mel_ce: 2.627878189086914 (nan) | > loss: 0.010567085817456245 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.1365730044472489) | > loader_time: 0.0045 (0.004644667276221788)  --> TIME: 2025-02-25 11:52:06 -- STEP: 29490/36938 -- GLOBAL_STEP: 768250 | > loss_text_ce: 0.03535788506269455 (0.03442793816502381) | > loss_mel_ce: 3.3951058387756348 (nan) | > loss: 0.013612952083349228 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13657453119977245) | > loader_time: 0.0047 (0.0046445989172189586)  --> TIME: 2025-02-25 11:52:15 -- STEP: 29540/36938 -- GLOBAL_STEP: 768300 | > loss_text_ce: 0.04064815118908882 (0.03443032635631865) | > loss_mel_ce: 2.7965755462646484 (nan) | > loss: 0.011258824728429317 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.1365679268426979) | > loader_time: 0.0046 (0.004644529296485023)  --> TIME: 2025-02-25 11:52:24 -- STEP: 29590/36938 -- GLOBAL_STEP: 768350 | > loss_text_ce: 0.03309652581810951 (0.034432926823502484) | > loss_mel_ce: 2.7511074542999268 (nan) | > loss: 0.011048429645597935 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13656283287069096) | > loader_time: 0.0045 (0.0046444990780757376)  --> TIME: 2025-02-25 11:52:34 -- STEP: 29640/36938 -- GLOBAL_STEP: 768400 | > loss_text_ce: 0.04522829130291939 (0.034431198550521465) | > loss_mel_ce: 1.997322916984558 (nan) | > loss: 0.008105362765491009 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13656375107334176) | > loader_time: 0.0046 (0.004644555979572986)  --> TIME: 2025-02-25 11:52:43 -- STEP: 29690/36938 -- GLOBAL_STEP: 768450 | > loss_text_ce: 0.02858336828649044 (0.034431159558730155) | > loss_mel_ce: 2.9384400844573975 (nan) | > loss: 0.011773902922868729 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.1365583504760025) | > loader_time: 0.0046 (0.004644492998312698)  --> TIME: 2025-02-25 11:52:52 -- STEP: 29740/36938 -- GLOBAL_STEP: 768500 | > loss_text_ce: 0.02414095029234886 (0.03442999795730595) | > loss_mel_ce: 1.6006280183792114 (nan) | > loss: 0.006447496358305216 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13656140004698797) | > loader_time: 0.0044 (0.004644454223001748)  --> TIME: 2025-02-25 11:53:02 -- STEP: 29790/36938 -- GLOBAL_STEP: 768550 | > loss_text_ce: 0.038448840379714966 (0.0344303876349153) | > loss_mel_ce: 1.4958572387695312 (nan) | > loss: 0.00608851620927453 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13655968958517695) | > loader_time: 0.0045 (0.004644477427425957)  --> TIME: 2025-02-25 11:53:11 -- STEP: 29840/36938 -- GLOBAL_STEP: 768600 | > loss_text_ce: 0.04232453554868698 (0.03443123157705424) | > loss_mel_ce: 2.731255531311035 (nan) | > loss: 0.011006270535290241 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13655490454176597) | > loader_time: 0.0045 (0.004644399761514449)  --> TIME: 2025-02-25 11:53:20 -- STEP: 29890/36938 -- GLOBAL_STEP: 768650 | > loss_text_ce: 0.04136718809604645 (0.03442994508962094) | > loss_mel_ce: 1.9441444873809814 (nan) | > loss: 0.007879015058279037 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13655735361572202) | > loader_time: 0.0045 (0.004644335125871803)  --> TIME: 2025-02-25 11:53:29 -- STEP: 29940/36938 -- GLOBAL_STEP: 768700 | > loss_text_ce: 0.03604836016893387 (0.034431248273558646) | > loss_mel_ce: 1.9428280591964722 (nan) | > loss: 0.007852684706449509 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13655445506434277) | > loader_time: 0.0048 (0.004644276575000602)  --> TIME: 2025-02-25 11:53:38 -- STEP: 29990/36938 -- GLOBAL_STEP: 768750 | > loss_text_ce: 0.053481265902519226 (0.03443321023948763) | > loss_mel_ce: 1.9523228406906128 (nan) | > loss: 0.007959540002048016 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13654962796773898) | > loader_time: 0.0046 (0.004644156162164035)  --> TIME: 2025-02-25 11:53:48 -- STEP: 30040/36938 -- GLOBAL_STEP: 768800 | > loss_text_ce: 0.02985469251871109 (0.03443132500876035) | > loss_mel_ce: 2.5138118267059326 (nan) | > loss: 0.010093915276229382 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13655113968328655) | > loader_time: 0.0045 (0.004644051888652568)  --> TIME: 2025-02-25 11:53:57 -- STEP: 30090/36938 -- GLOBAL_STEP: 768850 | > loss_text_ce: 0.040182407945394516 (0.0344314068553356) | > loss_mel_ce: 3.077373743057251 (nan) | > loss: 0.012371255084872246 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.1365482740364201) | > loader_time: 0.0045 (0.004643970686322091)  --> TIME: 2025-02-25 11:54:06 -- STEP: 30140/36938 -- GLOBAL_STEP: 768900 | > loss_text_ce: 0.021517790853977203 (0.034430320896010544) | > loss_mel_ce: 2.9047858715057373 (nan) | > loss: 0.011612316593527794 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13654175969885254) | > loader_time: 0.0045 (0.004643888867447228)  --> TIME: 2025-02-25 11:54:16 -- STEP: 30190/36938 -- GLOBAL_STEP: 768950 | > loss_text_ce: 0.03309621661901474 (0.03443009576872215) | > loss_mel_ce: 2.0584654808044434 (nan) | > loss: 0.00829984899610281 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13654622253735244) | > loader_time: 0.0048 (0.004643940443864694)  --> TIME: 2025-02-25 11:54:25 -- STEP: 30240/36938 -- GLOBAL_STEP: 769000 | > loss_text_ce: 0.027731968089938164 (0.03442912614841518) | > loss_mel_ce: 3.1011929512023926 (nan) | > loss: 0.012416369281709194 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.1365460546086086) | > loader_time: 0.0047 (0.004643989192745688)  --> TIME: 2025-02-25 11:54:34 -- STEP: 30290/36938 -- GLOBAL_STEP: 769050 | > loss_text_ce: 0.038350775837898254 (0.034431230289490765) | > loss_mel_ce: 3.2394564151763916 (nan) | > loss: 0.013007172383368015 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13653994744742695) | > loader_time: 0.0045 (0.004643953850104824)  --> TIME: 2025-02-25 11:54:43 -- STEP: 30340/36938 -- GLOBAL_STEP: 769100 | > loss_text_ce: 0.027747131884098053 (0.03443075471220634) | > loss_mel_ce: 2.2673895359039307 (nan) | > loss: 0.009107685647904873 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13654174971282013) | > loader_time: 0.0046 (0.004643986856929566)  --> TIME: 2025-02-25 11:54:53 -- STEP: 30390/36938 -- GLOBAL_STEP: 769150 | > loss_text_ce: 0.04294406622648239 (0.03443099910760494) | > loss_mel_ce: 2.7899813652038574 (nan) | > loss: 0.011241767555475235 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13654157110404522) | > loader_time: 0.0044 (0.0046439319505782685)  --> TIME: 2025-02-25 11:55:02 -- STEP: 30440/36938 -- GLOBAL_STEP: 769200 | > loss_text_ce: 0.04121693968772888 (0.03443277680636942) | > loss_mel_ce: 1.7782481908798218 (nan) | > loss: 0.007220100145787001 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.1365430426268646) | > loader_time: 0.0046 (0.0046439149298273304)  --> TIME: 2025-02-25 11:55:12 -- STEP: 30490/36938 -- GLOBAL_STEP: 769250 | > loss_text_ce: 0.037797875702381134 (0.03443172625689068) | > loss_mel_ce: 3.275066614151001 (nan) | > loss: 0.013146288692951202 (nan) | > current_lr: 5e-06 | > step_time: 0.1858 (0.13655066412759753) | > loader_time: 0.0048 (0.0046438699395745095)  --> TIME: 2025-02-25 11:55:21 -- STEP: 30540/36938 -- GLOBAL_STEP: 769300 | > loss_text_ce: 0.025969836860895157 (0.03443135814296026) | > loss_mel_ce: 2.4188055992126465 (nan) | > loss: 0.00970149040222168 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13654870155518206) | > loader_time: 0.0046 (0.004643905818579488)  --> TIME: 2025-02-25 11:55:30 -- STEP: 30590/36938 -- GLOBAL_STEP: 769350 | > loss_text_ce: 0.029690558090806007 (0.03443199907255675) | > loss_mel_ce: 1.7480360269546509 (nan) | > loss: 0.00705447094514966 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13654756885839198) | > loader_time: 0.0046 (0.004643859548403508)  --> TIME: 2025-02-25 11:55:39 -- STEP: 30640/36938 -- GLOBAL_STEP: 769400 | > loss_text_ce: 0.02724628709256649 (0.034431003235972334) | > loss_mel_ce: 2.1299233436584473 (nan) | > loss: 0.008560197427868843 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.1365379569658112) | > loader_time: 0.0045 (0.004643805344484509)  --> TIME: 2025-02-25 11:55:48 -- STEP: 30690/36938 -- GLOBAL_STEP: 769450 | > loss_text_ce: 0.033477120101451874 (0.03443032084897759) | > loss_mel_ce: 3.4237465858459473 (nan) | > loss: 0.01371914241462946 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.13653437514847483) | > loader_time: 0.0048 (0.004643746259819237)  --> TIME: 2025-02-25 11:55:58 -- STEP: 30740/36938 -- GLOBAL_STEP: 769500 | > loss_text_ce: 0.035736531019210815 (0.03442938461216264) | > loss_mel_ce: 0.9382104277610779 (nan) | > loss: 0.0038648690097033978 (nan) | > current_lr: 5e-06 | > step_time: 0.1861 (0.13653539495958436) | > loader_time: 0.0049 (0.004643724665831045)  --> TIME: 2025-02-25 11:56:07 -- STEP: 30790/36938 -- GLOBAL_STEP: 769550 | > loss_text_ce: 0.03698958829045296 (0.034432201836283594) | > loss_mel_ce: 3.0485198497772217 (nan) | > loss: 0.012244085781276226 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13653404149733608) | > loader_time: 0.0046 (0.004643700834449617)  --> TIME: 2025-02-25 11:56:17 -- STEP: 30840/36938 -- GLOBAL_STEP: 769600 | > loss_text_ce: 0.03144014626741409 (0.03443288684317701) | > loss_mel_ce: 2.5198776721954346 (nan) | > loss: 0.010124278254806995 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13654410878508125) | > loader_time: 0.0046 (0.00464373662314064)  --> TIME: 2025-02-25 11:56:26 -- STEP: 30890/36938 -- GLOBAL_STEP: 769650 | > loss_text_ce: 0.03949545696377754 (0.03443093625711042) | > loss_mel_ce: 1.8799675703048706 (nan) | > loss: 0.007616917137056589 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13654577925815867) | > loader_time: 0.0045 (0.004643779018620697)  --> TIME: 2025-02-25 11:56:36 -- STEP: 30940/36938 -- GLOBAL_STEP: 769700 | > loss_text_ce: 0.04193456098437309 (0.034429041169880006) | > loss_mel_ce: 3.0151138305664062 (nan) | > loss: 0.012131145223975182 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13654813368087867) | > loader_time: 0.0046 (0.004643710251384669)  --> TIME: 2025-02-25 11:56:45 -- STEP: 30990/36938 -- GLOBAL_STEP: 769750 | > loss_text_ce: 0.025802068412303925 (0.03442878058176007) | > loss_mel_ce: 2.155893087387085 (nan) | > loss: 0.008657521568238735 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13655078358787268) | > loader_time: 0.0045 (0.004643620372241694)  --> TIME: 2025-02-25 11:56:55 -- STEP: 31040/36938 -- GLOBAL_STEP: 769800 | > loss_text_ce: 0.03549375385046005 (0.03442994871695249) | > loss_mel_ce: 2.745849847793579 (nan) | > loss: 0.011037078686058521 (nan) | > current_lr: 5e-06 | > step_time: 0.1831 (0.13655433707784248) | > loader_time: 0.0048 (0.004643581385157787)  --> TIME: 2025-02-25 11:57:04 -- STEP: 31090/36938 -- GLOBAL_STEP: 769850 | > loss_text_ce: 0.035802122205495834 (0.03443217357250945) | > loss_mel_ce: 3.1626312732696533 (nan) | > loss: 0.012692197225987911 (nan) | > current_lr: 5e-06 | > step_time: 0.1445 (0.13655402522334145) | > loader_time: 0.0047 (0.004643586587583815)  --> TIME: 2025-02-25 11:57:13 -- STEP: 31140/36938 -- GLOBAL_STEP: 769900 | > loss_text_ce: 0.03414484113454819 (0.03443213914010816) | > loss_mel_ce: 3.175903558731079 (nan) | > loss: 0.012738288380205631 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1365406065777329) | > loader_time: 0.0046 (0.004643522223066517)  --> TIME: 2025-02-25 11:57:22 -- STEP: 31190/36938 -- GLOBAL_STEP: 769950 | > loss_text_ce: 0.0394853875041008 (0.034433658338686675) | > loss_mel_ce: 3.3076043128967285 (nan) | > loss: 0.013282103464007378 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.1365451667659682) | > loader_time: 0.0048 (0.004643492669621978)  --> TIME: 2025-02-25 11:57:32 -- STEP: 31240/36938 -- GLOBAL_STEP: 770000 | > loss_text_ce: 0.035832446068525314 (0.03443327028353231) | > loss_mel_ce: 2.1809470653533936 (nan) | > loss: 0.008796744048595428 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13655135261417314) | > loader_time: 0.0045 (0.004643577108309927) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_770000.pth  --> TIME: 2025-02-25 11:57:47 -- STEP: 31290/36938 -- GLOBAL_STEP: 770050 | > loss_text_ce: 0.039412472397089005 (0.034432879768338184) | > loss_mel_ce: 3.101762533187866 (nan) | > loss: 0.012464980594813824 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13655107055620289) | > loader_time: 0.0048 (0.0046436364218692796)  --> TIME: 2025-02-25 11:57:56 -- STEP: 31340/36938 -- GLOBAL_STEP: 770100 | > loss_text_ce: 0.03128824010491371 (0.03443222365246394) | > loss_mel_ce: 2.712453842163086 (nan) | > loss: 0.010887865908443928 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.13654517848246825) | > loader_time: 0.0046 (0.004643599372345258)  --> TIME: 2025-02-25 11:58:05 -- STEP: 31390/36938 -- GLOBAL_STEP: 770150 | > loss_text_ce: 0.033324457705020905 (0.03442888072967488) | > loss_mel_ce: 2.540278911590576 (nan) | > loss: 0.010212712921202183 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13653527828264064) | > loader_time: 0.0046 (0.004643624259545279)  --> TIME: 2025-02-25 11:58:14 -- STEP: 31440/36938 -- GLOBAL_STEP: 770200 | > loss_text_ce: 0.03756276145577431 (0.034429028216404645) | > loss_mel_ce: 0.07130178064107895 (nan) | > loss: 0.0004320021835155785 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13653297295400624) | > loader_time: 0.0045 (0.004643653079146965)  --> TIME: 2025-02-25 11:58:24 -- STEP: 31490/36938 -- GLOBAL_STEP: 770250 | > loss_text_ce: 0.030632654204964638 (0.03442916754682285) | > loss_mel_ce: 2.4846670627593994 (nan) | > loss: 0.009981349110603333 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.1365364269666662) | > loader_time: 0.0045 (0.004643742362065116)  --> TIME: 2025-02-25 11:58:33 -- STEP: 31540/36938 -- GLOBAL_STEP: 770300 | > loss_text_ce: 0.0368867963552475 (0.034430369267419714) | > loss_mel_ce: 2.1681265830993652 (nan) | > loss: 0.008750053122639656 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13653332358610162) | > loader_time: 0.0047 (0.004643724164255238)  --> TIME: 2025-02-25 11:58:42 -- STEP: 31590/36938 -- GLOBAL_STEP: 770350 | > loss_text_ce: 0.03373291343450546 (0.03443126696465292) | > loss_mel_ce: 1.1714990139007568 (nan) | > loss: 0.004782666452229023 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13652735594217935) | > loader_time: 0.0045 (0.004643740477385359)  --> TIME: 2025-02-25 11:58:52 -- STEP: 31640/36938 -- GLOBAL_STEP: 770400 | > loss_text_ce: 0.02836698479950428 (0.03443199800737745) | > loss_mel_ce: 3.206329345703125 (nan) | > loss: 0.012836097739636898 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.13653639619028035) | > loader_time: 0.0047 (0.004643901982892192)  --> TIME: 2025-02-25 11:59:01 -- STEP: 31690/36938 -- GLOBAL_STEP: 770450 | > loss_text_ce: 0.03584124147891998 (0.034431554558670766) | > loss_mel_ce: 2.8251562118530273 (nan) | > loss: 0.011353164911270142 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13653221836824492) | > loader_time: 0.0045 (0.004643946899228699)  --> TIME: 2025-02-25 11:59:10 -- STEP: 31740/36938 -- GLOBAL_STEP: 770500 | > loss_text_ce: 0.024181080982089043 (0.034431867877305476) | > loss_mel_ce: 2.559213638305664 (nan) | > loss: 0.010251566767692566 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13653605605493913) | > loader_time: 0.0046 (0.0046439793174493805)  --> TIME: 2025-02-25 11:59:20 -- STEP: 31790/36938 -- GLOBAL_STEP: 770550 | > loss_text_ce: 0.03577360883355141 (0.03443232938777932) | > loss_mel_ce: 2.2606098651885986 (nan) | > loss: 0.00911263283342123 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13653377845400685) | > loader_time: 0.0046 (0.004644018001022425)  --> TIME: 2025-02-25 11:59:29 -- STEP: 31840/36938 -- GLOBAL_STEP: 770600 | > loss_text_ce: 0.028849594295024872 (0.03443081144669946) | > loss_mel_ce: 3.2439398765563965 (nan) | > loss: 0.012987260706722736 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13652943312822388) | > loader_time: 0.0045 (0.004643989672612919)  --> TIME: 2025-02-25 11:59:38 -- STEP: 31890/36938 -- GLOBAL_STEP: 770650 | > loss_text_ce: 0.04242946207523346 (0.03443005503006059) | > loss_mel_ce: 3.4424617290496826 (nan) | > loss: 0.013828934170305729 (nan) | > current_lr: 5e-06 | > step_time: 0.1512 (0.1365299378192036) | > loader_time: 0.0047 (0.004644164952337663)  --> TIME: 2025-02-25 11:59:47 -- STEP: 31940/36938 -- GLOBAL_STEP: 770700 | > loss_text_ce: 0.03306308388710022 (0.03442791864242693) | > loss_mel_ce: 2.7123935222625732 (nan) | > loss: 0.010894670151174068 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.136529079712252) | > loader_time: 0.0048 (0.004644255550040568)  --> TIME: 2025-02-25 11:59:56 -- STEP: 31990/36938 -- GLOBAL_STEP: 770750 | > loss_text_ce: 0.0365423820912838 (0.034429233266746975) | > loss_mel_ce: 2.9420101642608643 (nan) | > loss: 0.011819654144346714 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13652412318259302) | > loader_time: 0.0045 (0.004644319950770362)  --> TIME: 2025-02-25 12:00:06 -- STEP: 32040/36938 -- GLOBAL_STEP: 770800 | > loss_text_ce: 0.030904468148946762 (0.03443033897225513) | > loss_mel_ce: 2.9101521968841553 (nan) | > loss: 0.011670860461890697 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.1365266260359629) | > loader_time: 0.0047 (0.004644334710641463)  --> TIME: 2025-02-25 12:00:15 -- STEP: 32090/36938 -- GLOBAL_STEP: 770850 | > loss_text_ce: 0.03541882336139679 (0.03442988885855655) | > loss_mel_ce: 3.0391857624053955 (nan) | > loss: 0.012200811877846718 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.1365264518567813) | > loader_time: 0.0047 (0.00464434939479859)  --> TIME: 2025-02-25 12:00:25 -- STEP: 32140/36938 -- GLOBAL_STEP: 770900 | > loss_text_ce: 0.033127572387456894 (0.03442997313689908) | > loss_mel_ce: 2.5719680786132812 (nan) | > loss: 0.010337681509554386 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13652529332974683) | > loader_time: 0.0046 (0.004644359463701815)  --> TIME: 2025-02-25 12:00:34 -- STEP: 32190/36938 -- GLOBAL_STEP: 770950 | > loss_text_ce: 0.03710735961794853 (0.03442881939638756) | > loss_mel_ce: 0.7665552496910095 (nan) | > loss: 0.003189137438312173 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13653347824033985) | > loader_time: 0.0045 (0.004644393528348446)  --> TIME: 2025-02-25 12:00:43 -- STEP: 32240/36938 -- GLOBAL_STEP: 771000 | > loss_text_ce: 0.0415620356798172 (0.034430183441102026) | > loss_mel_ce: 2.174093008041382 (nan) | > loss: 0.00879228301346302 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.13653157567120183) | > loader_time: 0.0047 (0.004644441501082631)  --> TIME: 2025-02-25 12:00:53 -- STEP: 32290/36938 -- GLOBAL_STEP: 771050 | > loss_text_ce: 0.040759991854429245 (0.03443101047641767) | > loss_mel_ce: 3.1051747798919678 (nan) | > loss: 0.0124838687479496 (nan) | > current_lr: 5e-06 | > step_time: 0.1866 (0.13653593248684626) | > loader_time: 0.0049 (0.004644524855449805)  --> TIME: 2025-02-25 12:01:03 -- STEP: 32340/36938 -- GLOBAL_STEP: 771100 | > loss_text_ce: 0.04124452546238899 (0.034431030516280596) | > loss_mel_ce: 3.100714683532715 (nan) | > loss: 0.012468093074858189 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13655165708868722) | > loader_time: 0.0048 (0.0046446423067601425)  --> TIME: 2025-02-25 12:01:12 -- STEP: 32390/36938 -- GLOBAL_STEP: 771150 | > loss_text_ce: 0.02460809424519539 (0.03443001393840068) | > loss_mel_ce: 3.325169086456299 (nan) | > loss: 0.013292767107486725 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13655522239911297) | > loader_time: 0.0046 (0.004644644102147803)  --> TIME: 2025-02-25 12:01:22 -- STEP: 32440/36938 -- GLOBAL_STEP: 771200 | > loss_text_ce: 0.03413954749703407 (0.034430882435736436) | > loss_mel_ce: 2.8984375 (nan) | > loss: 0.01163721177726984 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13654985168418338) | > loader_time: 0.0045 (0.00464456777390365)  --> TIME: 2025-02-25 12:01:31 -- STEP: 32490/36938 -- GLOBAL_STEP: 771250 | > loss_text_ce: 0.027536265552043915 (0.03442829438342169) | > loss_mel_ce: 3.0991623401641846 (nan) | > loss: 0.012407534755766392 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13655144738285205) | > loader_time: 0.0046 (0.004644569971995134)  --> TIME: 2025-02-25 12:01:40 -- STEP: 32540/36938 -- GLOBAL_STEP: 771300 | > loss_text_ce: 0.04529779031872749 (0.03442645395496895) | > loss_mel_ce: 2.2232208251953125 (nan) | > loss: 0.009002058766782284 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1365528875285314) | > loader_time: 0.0045 (0.004644501795414128)  --> TIME: 2025-02-25 12:01:50 -- STEP: 32590/36938 -- GLOBAL_STEP: 771350 | > loss_text_ce: 0.030726369470357895 (0.03442674323041411) | > loss_mel_ce: 3.112583875656128 (nan) | > loss: 0.01247345469892025 (nan) | > current_lr: 5e-06 | > step_time: 0.1442 (0.13655289096604137) | > loader_time: 0.0046 (0.004644436915252222)  --> TIME: 2025-02-25 12:01:59 -- STEP: 32640/36938 -- GLOBAL_STEP: 771400 | > loss_text_ce: 0.03045056015253067 (0.0344282718546239) | > loss_mel_ce: 3.0576295852661133 (nan) | > loss: 0.012254287488758564 (nan) | > current_lr: 5e-06 | > step_time: 0.1943 (0.13655624488259174) | > loader_time: 0.0049 (0.0046444394497894914)  --> TIME: 2025-02-25 12:02:08 -- STEP: 32690/36938 -- GLOBAL_STEP: 771450 | > loss_text_ce: 0.025164805352687836 (0.03442758167864521) | > loss_mel_ce: 1.913559913635254 (nan) | > loss: 0.007693352643400431 (nan) | > current_lr: 5e-06 | > step_time: 0.1071 (0.1365558083249229) | > loader_time: 0.0045 (0.004644515318183527)  --> TIME: 2025-02-25 12:02:18 -- STEP: 32740/36938 -- GLOBAL_STEP: 771500 | > loss_text_ce: 0.023105168715119362 (0.03442686009994512) | > loss_mel_ce: 2.6752259731292725 (nan) | > loss: 0.010707663372159004 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13655301170809467) | > loader_time: 0.0046 (0.00464454253563157)  --> TIME: 2025-02-25 12:02:26 -- STEP: 32790/36938 -- GLOBAL_STEP: 771550 | > loss_text_ce: 0.030267102643847466 (0.03442421074892467) | > loss_mel_ce: 2.966123342514038 (nan) | > loss: 0.011890438385307789 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13653954959344816) | > loader_time: 0.0046 (0.004644555825945748)  --> TIME: 2025-02-25 12:02:36 -- STEP: 32840/36938 -- GLOBAL_STEP: 771600 | > loss_text_ce: 0.032127924263477325 (0.03442384289119884) | > loss_mel_ce: 0.9549357295036316 (nan) | > loss: 0.003916919231414795 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13654910442890228) | > loader_time: 0.0046 (0.0046447114227169605)  --> TIME: 2025-02-25 12:02:45 -- STEP: 32890/36938 -- GLOBAL_STEP: 771650 | > loss_text_ce: 0.03517564758658409 (0.034422164354541325) | > loss_mel_ce: 3.0419700145721436 (nan) | > loss: 0.012210896238684654 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13654560165660548) | > loader_time: 0.0046 (0.004644738652851864)  --> TIME: 2025-02-25 12:02:54 -- STEP: 32940/36938 -- GLOBAL_STEP: 771700 | > loss_text_ce: 0.03285994380712509 (0.03442062259683953) | > loss_mel_ce: 1.6481558084487915 (nan) | > loss: 0.006670697592198849 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13653718159281383) | > loader_time: 0.0045 (0.0046447816369891235)  --> TIME: 2025-02-25 12:03:04 -- STEP: 32990/36938 -- GLOBAL_STEP: 771750 | > loss_text_ce: 0.03092627041041851 (0.034422649780729314) | > loss_mel_ce: 2.570370674133301 (nan) | > loss: 0.01032260712236166 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.1365389111909996) | > loader_time: 0.0046 (0.00464485396974628)  --> TIME: 2025-02-25 12:03:13 -- STEP: 33040/36938 -- GLOBAL_STEP: 771800 | > loss_text_ce: 0.03722091019153595 (0.034423190899493215) | > loss_mel_ce: 3.3570196628570557 (nan) | > loss: 0.013469209894537926 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1365324527674679) | > loader_time: 0.0045 (0.004644923435285088)  --> TIME: 2025-02-25 12:03:22 -- STEP: 33090/36938 -- GLOBAL_STEP: 771850 | > loss_text_ce: 0.029240669682621956 (0.03442450983995271) | > loss_mel_ce: 2.8751163482666016 (nan) | > loss: 0.011525226756930351 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13653082237318692) | > loader_time: 0.0047 (0.004644978381456958)  --> TIME: 2025-02-25 12:03:32 -- STEP: 33140/36938 -- GLOBAL_STEP: 771900 | > loss_text_ce: 0.019442524760961533 (0.03442349581584093) | > loss_mel_ce: 2.9039084911346436 (nan) | > loss: 0.011600599624216557 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13654245733135748) | > loader_time: 0.0045 (0.004645096601026937)  --> TIME: 2025-02-25 12:03:41 -- STEP: 33190/36938 -- GLOBAL_STEP: 771950 | > loss_text_ce: 0.029421018436551094 (0.03442209117612897) | > loss_mel_ce: 1.747841238975525 (nan) | > loss: 0.007052628323435783 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13654414993267852) | > loader_time: 0.0046 (0.004645111848877739)  --> TIME: 2025-02-25 12:03:51 -- STEP: 33240/36938 -- GLOBAL_STEP: 772000 | > loss_text_ce: 0.04378615319728851 (0.03442223694519546) | > loss_mel_ce: 1.8318208456039429 (nan) | > loss: 0.007442885544151068 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13654784311313023) | > loader_time: 0.0045 (0.004645121965454244)  --> TIME: 2025-02-25 12:04:00 -- STEP: 33290/36938 -- GLOBAL_STEP: 772050 | > loss_text_ce: 0.034067172557115555 (0.03442162899280874) | > loss_mel_ce: 1.8287876844406128 (nan) | > loss: 0.007392281666398048 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13654599650266505) | > loader_time: 0.0046 (0.004645090784959241)  --> TIME: 2025-02-25 12:04:10 -- STEP: 33340/36938 -- GLOBAL_STEP: 772100 | > loss_text_ce: 0.03525863215327263 (0.03442430619556004) | > loss_mel_ce: 2.95833683013916 (nan) | > loss: 0.011879347264766693 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.1365536084844466) | > loader_time: 0.0048 (0.004645140169620813)  --> TIME: 2025-02-25 12:04:19 -- STEP: 33390/36938 -- GLOBAL_STEP: 772150 | > loss_text_ce: 0.029329728335142136 (0.03442341298719533) | > loss_mel_ce: 1.7158106565475464 (nan) | > loss: 0.006925160996615887 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13654475110703332) | > loader_time: 0.0045 (0.004645156946036598)  --> TIME: 2025-02-25 12:04:28 -- STEP: 33440/36938 -- GLOBAL_STEP: 772200 | > loss_text_ce: 0.03563116863369942 (0.034421900812977635) | > loss_mel_ce: 2.8676905632019043 (nan) | > loss: 0.011521118693053722 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13655168744936053) | > loader_time: 0.0046 (0.004645118887344627)  --> TIME: 2025-02-25 12:04:38 -- STEP: 33490/36938 -- GLOBAL_STEP: 772250 | > loss_text_ce: 0.033925898373126984 (0.034420684205796984) | > loss_mel_ce: 3.5282938480377197 (nan) | > loss: 0.014135793782770634 (nan) | > current_lr: 5e-06 | > step_time: 0.1797 (0.13655336138809965) | > loader_time: 0.0048 (0.0046450487924497895)  --> TIME: 2025-02-25 12:04:47 -- STEP: 33540/36938 -- GLOBAL_STEP: 772300 | > loss_text_ce: 0.026975177228450775 (0.034419719369545555) | > loss_mel_ce: 3.4040322303771973 (nan) | > loss: 0.013615109026432037 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13655367464129353) | > loader_time: 0.0046 (0.004645017008022259)  --> TIME: 2025-02-25 12:04:56 -- STEP: 33590/36938 -- GLOBAL_STEP: 772350 | > loss_text_ce: 0.03547075763344765 (0.03441746606868304) | > loss_mel_ce: 2.6416783332824707 (nan) | > loss: 0.010623607784509659 (nan) | > current_lr: 5e-06 | > step_time: 0.1624 (0.13654853387686816) | > loader_time: 0.0047 (0.004645048155959227)  --> TIME: 2025-02-25 12:05:05 -- STEP: 33640/36938 -- GLOBAL_STEP: 772400 | > loss_text_ce: 0.029045380651950836 (0.034416694453597386) | > loss_mel_ce: 2.2176549434661865 (nan) | > loss: 0.008915477432310581 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1365472859491502) | > loader_time: 0.0045 (0.004645166846432965)  --> TIME: 2025-02-25 12:05:15 -- STEP: 33690/36938 -- GLOBAL_STEP: 772450 | > loss_text_ce: 0.03940306603908539 (0.03441648544408058) | > loss_mel_ce: 3.3021254539489746 (nan) | > loss: 0.013260033912956715 (nan) | > current_lr: 5e-06 | > step_time: 0.1621 (0.1365531203866405) | > loader_time: 0.0047 (0.004645197863534932)  --> TIME: 2025-02-25 12:05:24 -- STEP: 33740/36938 -- GLOBAL_STEP: 772500 | > loss_text_ce: 0.04523462802171707 (0.034418068164556524) | > loss_mel_ce: 2.997231960296631 (nan) | > loss: 0.012073281221091747 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.13655668229271092) | > loader_time: 0.0047 (0.004645214274428491)  --> TIME: 2025-02-25 12:05:34 -- STEP: 33790/36938 -- GLOBAL_STEP: 772550 | > loss_text_ce: 0.02947131171822548 (0.03441772251074333) | > loss_mel_ce: 2.3408429622650146 (nan) | > loss: 0.009406009688973427 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1365573376725866) | > loader_time: 0.0045 (0.004645323626350897)  --> TIME: 2025-02-25 12:05:43 -- STEP: 33840/36938 -- GLOBAL_STEP: 772600 | > loss_text_ce: 0.035619430243968964 (0.034417603157095175) | > loss_mel_ce: 2.799034833908081 (nan) | > loss: 0.011248628608882427 (nan) | > current_lr: 5e-06 | > step_time: 0.169 (0.13656336202266467) | > loader_time: 0.0048 (0.004645332560189804)  --> TIME: 2025-02-25 12:05:53 -- STEP: 33890/36938 -- GLOBAL_STEP: 772650 | > loss_text_ce: 0.042306821793317795 (0.03441799678481117) | > loss_mel_ce: 2.944531202316284 (nan) | > loss: 0.011852532625198364 (nan) | > current_lr: 5e-06 | > step_time: 0.1511 (0.136562419226122) | > loader_time: 0.0047 (0.004645443638013379)  --> TIME: 2025-02-25 12:06:02 -- STEP: 33940/36938 -- GLOBAL_STEP: 772700 | > loss_text_ce: 0.028858449310064316 (0.034417407169200456) | > loss_mel_ce: 3.69921875 (nan) | > loss: 0.014793957583606243 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13656303244755513) | > loader_time: 0.0046 (0.004645419816234647)  --> TIME: 2025-02-25 12:06:11 -- STEP: 33990/36938 -- GLOBAL_STEP: 772750 | > loss_text_ce: 0.03424198925495148 (0.03441593445904349) | > loss_mel_ce: 2.9373185634613037 (nan) | > loss: 0.011791907250881195 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13655260763788496) | > loader_time: 0.0045 (0.004645362809392498)  --> TIME: 2025-02-25 12:06:20 -- STEP: 34040/36938 -- GLOBAL_STEP: 772800 | > loss_text_ce: 0.03534547984600067 (0.034416453889817324) | > loss_mel_ce: 2.74186635017395 (nan) | > loss: 0.011020682752132416 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13655793784508083) | > loader_time: 0.0048 (0.00464547670125682)  --> TIME: 2025-02-25 12:06:30 -- STEP: 34090/36938 -- GLOBAL_STEP: 772850 | > loss_text_ce: 0.03199891746044159 (0.0344143969698293) | > loss_mel_ce: 3.305222988128662 (nan) | > loss: 0.013242945075035095 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13655849367546335) | > loader_time: 0.0047 (0.004645533581289185)  --> TIME: 2025-02-25 12:06:39 -- STEP: 34140/36938 -- GLOBAL_STEP: 772900 | > loss_text_ce: 0.023975331336259842 (0.034410854546335704) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13656586556666572) | > loader_time: 0.0047 (0.004645708058406969)  --> TIME: 2025-02-25 12:06:49 -- STEP: 34190/36938 -- GLOBAL_STEP: 772950 | > loss_text_ce: 0.036882661283016205 (0.034412193701894744) | > loss_mel_ce: 2.726146936416626 (nan) | > loss: 0.010964403860270977 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13657154637353836) | > loader_time: 0.0046 (0.004645748436886799)  --> TIME: 2025-02-25 12:06:58 -- STEP: 34240/36938 -- GLOBAL_STEP: 773000 | > loss_text_ce: 0.028158366680145264 (0.034411405657160166) | > loss_mel_ce: 1.6077256202697754 (nan) | > loss: 0.006491603795439005 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13656789922964999) | > loader_time: 0.0046 (0.004645774095693489)  --> TIME: 2025-02-25 12:07:08 -- STEP: 34290/36938 -- GLOBAL_STEP: 773050 | > loss_text_ce: 0.044452086091041565 (0.03441008199123102) | > loss_mel_ce: 3.248518466949463 (nan) | > loss: 0.013067344203591347 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.1365686195589922) | > loader_time: 0.0047 (0.004645744709205134)  --> TIME: 2025-02-25 12:07:17 -- STEP: 34340/36938 -- GLOBAL_STEP: 773100 | > loss_text_ce: 0.033850014209747314 (0.034409186007329104) | > loss_mel_ce: 2.973536252975464 (nan) | > loss: 0.011934072710573673 (nan) | > current_lr: 5e-06 | > step_time: 0.1716 (0.13657092836337884) | > loader_time: 0.0048 (0.0046457801853839855)  --> TIME: 2025-02-25 12:07:26 -- STEP: 34390/36938 -- GLOBAL_STEP: 773150 | > loss_text_ce: 0.03282556310296059 (0.034409726043747646) | > loss_mel_ce: 2.257477045059204 (nan) | > loss: 0.009088502265512943 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13656629316680016) | > loader_time: 0.0048 (0.004645769996121729)  --> TIME: 2025-02-25 12:07:35 -- STEP: 34440/36938 -- GLOBAL_STEP: 773200 | > loss_text_ce: 0.0303338672965765 (0.034407359427495524) | > loss_mel_ce: 3.092851161956787 (nan) | > loss: 0.012393591925501823 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13655832214638008) | > loader_time: 0.0045 (0.004645757288882966)  --> TIME: 2025-02-25 12:07:44 -- STEP: 34490/36938 -- GLOBAL_STEP: 773250 | > loss_text_ce: 0.04206184297800064 (0.03440777082193319) | > loss_mel_ce: 3.0745818614959717 (nan) | > loss: 0.01236763410270214 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.13655868390222486) | > loader_time: 0.0047 (0.00464573711822192)  --> TIME: 2025-02-25 12:07:54 -- STEP: 34540/36938 -- GLOBAL_STEP: 773300 | > loss_text_ce: 0.037918634712696075 (0.03440703563389462) | > loss_mel_ce: 3.054126024246216 (nan) | > loss: 0.012270019389688969 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1365577128627921) | > loader_time: 0.0046 (0.004645788828347835)  --> TIME: 2025-02-25 12:08:03 -- STEP: 34590/36938 -- GLOBAL_STEP: 773350 | > loss_text_ce: 0.03746409714221954 (0.03440706099957829) | > loss_mel_ce: 3.086270809173584 (nan) | > loss: 0.012395774014294147 (nan) | > current_lr: 5e-06 | > step_time: 0.1782 (0.13656013077413137) | > loader_time: 0.0049 (0.004645914285642702)  --> TIME: 2025-02-25 12:08:13 -- STEP: 34640/36938 -- GLOBAL_STEP: 773400 | > loss_text_ce: 0.034598205238580704 (0.0344065703725061) | > loss_mel_ce: 3.08725643157959 (nan) | > loss: 0.012388312257826328 (nan) | > current_lr: 5e-06 | > step_time: 0.1759 (0.13656732608622352) | > loader_time: 0.0049 (0.004646025388125191)  --> TIME: 2025-02-25 12:08:22 -- STEP: 34690/36938 -- GLOBAL_STEP: 773450 | > loss_text_ce: 0.035621318966150284 (0.034406485975741134) | > loss_mel_ce: 2.5351831912994385 (nan) | > loss: 0.010201605968177319 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13656309790486257) | > loader_time: 0.0046 (0.004645979868365701)  --> TIME: 2025-02-25 12:08:31 -- STEP: 34740/36938 -- GLOBAL_STEP: 773500 | > loss_text_ce: 0.0354643389582634 (0.034407811926754975) | > loss_mel_ce: 3.1663084030151367 (nan) | > loss: 0.01270544808357954 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13656600378132114) | > loader_time: 0.0046 (0.004646146990611907)  --> TIME: 2025-02-25 12:08:41 -- STEP: 34790/36938 -- GLOBAL_STEP: 773550 | > loss_text_ce: 0.03567314147949219 (0.03440674529720085) | > loss_mel_ce: 0.0289306640625 (nan) | > loss: 0.0002563643211033195 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13657089091128616) | > loader_time: 0.0045 (0.004646149302529487)  --> TIME: 2025-02-25 12:08:50 -- STEP: 34840/36938 -- GLOBAL_STEP: 773600 | > loss_text_ce: 0.034980498254299164 (0.03440561284683748) | > loss_mel_ce: 2.428135871887207 (nan) | > loss: 0.009774272330105305 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13657018874187085) | > loader_time: 0.0046 (0.004646166676631896)  --> TIME: 2025-02-25 12:09:00 -- STEP: 34890/36938 -- GLOBAL_STEP: 773650 | > loss_text_ce: 0.03490261733531952 (0.034405763287473023) | > loss_mel_ce: 2.897101640701294 (nan) | > loss: 0.011634938418865204 (nan) | > current_lr: 5e-06 | > step_time: 0.161 (0.1365752373564452) | > loader_time: 0.0047 (0.004646226942248801)  --> TIME: 2025-02-25 12:09:08 -- STEP: 34940/36938 -- GLOBAL_STEP: 773700 | > loss_text_ce: 0.03962036222219467 (0.03440633640324978) | > loss_mel_ce: 2.731290817260742 (nan) | > loss: 0.010995680466294289 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13656664439863592) | > loader_time: 0.0064 (0.004646309075794979)  --> TIME: 2025-02-25 12:09:18 -- STEP: 34990/36938 -- GLOBAL_STEP: 773750 | > loss_text_ce: 0.030444154515862465 (0.03440456269335536) | > loss_mel_ce: 2.6565496921539307 (nan) | > loss: 0.010662674903869629 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.1365660070793811) | > loader_time: 0.0048 (0.004646314209139191)  --> TIME: 2025-02-25 12:09:27 -- STEP: 35040/36938 -- GLOBAL_STEP: 773800 | > loss_text_ce: 0.0373506061732769 (0.03440598297577673) | > loss_mel_ce: 2.8924720287323 (nan) | > loss: 0.011626280844211578 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13657044502289953) | > loader_time: 0.0048 (0.004646318633806763)  --> TIME: 2025-02-25 12:09:37 -- STEP: 35090/36938 -- GLOBAL_STEP: 773850 | > loss_text_ce: 0.025436392053961754 (0.03440575584619022) | > loss_mel_ce: 2.475383758544922 (nan) | > loss: 0.009923890233039856 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.13656884627860041) | > loader_time: 0.0047 (0.004646358744104081)  --> TIME: 2025-02-25 12:09:46 -- STEP: 35140/36938 -- GLOBAL_STEP: 773900 | > loss_text_ce: 0.030456606298685074 (0.03440600871044859) | > loss_mel_ce: 2.702648162841797 (nan) | > loss: 0.010845654644072056 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13656875018579626) | > loader_time: 0.0047 (0.004646342785845581)  --> TIME: 2025-02-25 12:09:55 -- STEP: 35190/36938 -- GLOBAL_STEP: 773950 | > loss_text_ce: 0.03973189741373062 (0.03440490674693791) | > loss_mel_ce: 3.2171130180358887 (nan) | > loss: 0.01292398851364851 (nan) | > current_lr: 5e-06 | > step_time: 0.1905 (0.1365638930745415) | > loader_time: 0.0048 (0.004646370552520447)  --> TIME: 2025-02-25 12:10:04 -- STEP: 35240/36938 -- GLOBAL_STEP: 774000 | > loss_text_ce: 0.0350615531206131 (0.03440501841271942) | > loss_mel_ce: 1.2846883535385132 (nan) | > loss: 0.0052371034398674965 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1365572934321195) | > loader_time: 0.0045 (0.004646292054829075)  --> TIME: 2025-02-25 12:10:13 -- STEP: 35290/36938 -- GLOBAL_STEP: 774050 | > loss_text_ce: 0.03821658343076706 (0.03440535330852514) | > loss_mel_ce: 3.272592544555664 (nan) | > loss: 0.01313813216984272 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13655779065812948) | > loader_time: 0.0047 (0.004646314977417255)  --> TIME: 2025-02-25 12:10:22 -- STEP: 35340/36938 -- GLOBAL_STEP: 774100 | > loss_text_ce: 0.0393272303044796 (0.034405270296037675) | > loss_mel_ce: 3.249821662902832 (nan) | > loss: 0.013052178546786308 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13655392107615522) | > loader_time: 0.0047 (0.0046462611890338455)  --> TIME: 2025-02-25 12:10:32 -- STEP: 35390/36938 -- GLOBAL_STEP: 774150 | > loss_text_ce: 0.03551330417394638 (0.034407246358956776) | > loss_mel_ce: 0.1379547119140625 (nan) | > loss: 0.0006883651949465275 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13656171294517191) | > loader_time: 0.0046 (0.004646245407229534)  --> TIME: 2025-02-25 12:10:41 -- STEP: 35440/36938 -- GLOBAL_STEP: 774200 | > loss_text_ce: 0.035986073315143585 (0.03440687724788039) | > loss_mel_ce: 1.8275680541992188 (nan) | > loss: 0.007395056542009115 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.1365624972781002) | > loader_time: 0.0046 (0.004646201710937798)  --> TIME: 2025-02-25 12:10:50 -- STEP: 35490/36938 -- GLOBAL_STEP: 774250 | > loss_text_ce: 0.03131426125764847 (0.034406580309317605) | > loss_mel_ce: 0.9304874539375305 (nan) | > loss: 0.0038166737649589777 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.1365594439051065) | > loader_time: 0.0046 (0.004646224463681302)  --> TIME: 2025-02-25 12:10:59 -- STEP: 35540/36938 -- GLOBAL_STEP: 774300 | > loss_text_ce: 0.03397779539227486 (0.03440694530938936) | > loss_mel_ce: 2.774381637573242 (nan) | > loss: 0.011144284158945084 (nan) | > current_lr: 5e-06 | > step_time: 0.1574 (0.1365520328501343) | > loader_time: 0.0046 (0.004646176029338425)  --> TIME: 2025-02-25 12:11:09 -- STEP: 35590/36938 -- GLOBAL_STEP: 774350 | > loss_text_ce: 0.038222841918468475 (0.034407882513015124) | > loss_mel_ce: 0.113525390625 (nan) | > loss: 0.0006021755980327725 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13655008384772196) | > loader_time: 0.0045 (0.004646222086962979)  --> TIME: 2025-02-25 12:11:18 -- STEP: 35640/36938 -- GLOBAL_STEP: 774400 | > loss_text_ce: 0.04378372058272362 (0.034408091078669066) | > loss_mel_ce: 1.8662670850753784 (nan) | > loss: 0.007579566910862923 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13654716253013136) | > loader_time: 0.0046 (0.004646190636353332)  --> TIME: 2025-02-25 12:11:27 -- STEP: 35690/36938 -- GLOBAL_STEP: 774450 | > loss_text_ce: 0.03922896832227707 (0.03440874187723545) | > loss_mel_ce: 2.9802310466766357 (nan) | > loss: 0.011981984600424767 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13654601050211573) | > loader_time: 0.0046 (0.004646215688684606)  --> TIME: 2025-02-25 12:11:37 -- STEP: 35740/36938 -- GLOBAL_STEP: 774500 | > loss_text_ce: 0.03853844851255417 (0.03441090943923946) | > loss_mel_ce: 2.511620044708252 (nan) | > loss: 0.010119677521288395 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13654539944676042) | > loader_time: 0.0045 (0.004646264539461401)  --> TIME: 2025-02-25 12:11:46 -- STEP: 35790/36938 -- GLOBAL_STEP: 774550 | > loss_text_ce: 0.03702809661626816 (0.03441198430817484) | > loss_mel_ce: 3.333658218383789 (nan) | > loss: 0.013375739566981792 (nan) | > current_lr: 5e-06 | > step_time: 0.1794 (0.13655179189083608) | > loader_time: 0.0048 (0.004646239789646744)  --> TIME: 2025-02-25 12:11:55 -- STEP: 35840/36938 -- GLOBAL_STEP: 774600 | > loss_text_ce: 0.03532903641462326 (0.03441146043533069) | > loss_mel_ce: 2.8973402976989746 (nan) | > loss: 0.011637577787041664 (nan) | > current_lr: 5e-06 | > step_time: 0.1507 (0.13654734156360662) | > loader_time: 0.0046 (0.004646247432434136)  --> TIME: 2025-02-25 12:12:05 -- STEP: 35890/36938 -- GLOBAL_STEP: 774650 | > loss_text_ce: 0.03511800616979599 (0.03441181006325216) | > loss_mel_ce: 3.2038824558258057 (nan) | > loss: 0.012853177264332771 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.1365461906240199) | > loader_time: 0.0047 (0.0046462261301456805)  --> TIME: 2025-02-25 12:12:14 -- STEP: 35940/36938 -- GLOBAL_STEP: 774700 | > loss_text_ce: 0.041305769234895706 (0.03441202017669352) | > loss_mel_ce: 2.684215545654297 (nan) | > loss: 0.01081556174904108 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13654755191399717) | > loader_time: 0.0046 (0.004646212814513621)  --> TIME: 2025-02-25 12:12:23 -- STEP: 35990/36938 -- GLOBAL_STEP: 774750 | > loss_text_ce: 0.027092019096016884 (0.0344126970281036) | > loss_mel_ce: 2.5632970333099365 (nan) | > loss: 0.010279322043061256 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.1365421446670133) | > loader_time: 0.0045 (0.004646177933129317)  --> TIME: 2025-02-25 12:12:32 -- STEP: 36040/36938 -- GLOBAL_STEP: 774800 | > loss_text_ce: 0.03690207377076149 (0.034411724916861765) | > loss_mel_ce: 2.238267421722412 (nan) | > loss: 0.009028451517224312 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.1365381714308045) | > loader_time: 0.0046 (0.004646190250091891)  --> TIME: 2025-02-25 12:12:42 -- STEP: 36090/36938 -- GLOBAL_STEP: 774850 | > loss_text_ce: 0.03499304875731468 (0.0344126815996427) | > loss_mel_ce: 3.098576545715332 (nan) | > loss: 0.012434800155460835 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.1365378987263967) | > loader_time: 0.0047 (0.004646241615339805)  --> TIME: 2025-02-25 12:12:51 -- STEP: 36140/36938 -- GLOBAL_STEP: 774900 | > loss_text_ce: 0.03506922349333763 (0.03441177446636867) | > loss_mel_ce: 2.456298828125 (nan) | > loss: 0.009886382147669792 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13653725368513558) | > loader_time: 0.0046 (0.004646161503731118)  --> TIME: 2025-02-25 12:13:00 -- STEP: 36190/36938 -- GLOBAL_STEP: 774950 | > loss_text_ce: 0.03619390353560448 (0.034410890673718636) | > loss_mel_ce: 3.272890567779541 (nan) | > loss: 0.013131287880241871 (nan) | > current_lr: 5e-06 | > step_time: 0.1571 (0.13653307888645613) | > loader_time: 0.0046 (0.004646082944255996)  --> TIME: 2025-02-25 12:13:09 -- STEP: 36240/36938 -- GLOBAL_STEP: 775000 | > loss_text_ce: 0.03451566398143768 (0.034408545777305316) | > loss_mel_ce: 3.066685438156128 (nan) | > loss: 0.012306354008615017 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.13653369713698082) | > loader_time: 0.0047 (0.004646031318384552)  --> TIME: 2025-02-25 12:13:19 -- STEP: 36290/36938 -- GLOBAL_STEP: 775050 | > loss_text_ce: 0.031686004251241684 (0.03440852134736671) | > loss_mel_ce: 3.1006288528442383 (nan) | > loss: 0.012429822236299515 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13653310504817753) | > loader_time: 0.0046 (0.004645971563375496)  --> TIME: 2025-02-25 12:13:28 -- STEP: 36340/36938 -- GLOBAL_STEP: 775100 | > loss_text_ce: 0.03434405103325844 (0.034407061485042474) | > loss_mel_ce: 1.1008895635604858 (nan) | > loss: 0.004504895769059658 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13653236094843843) | > loader_time: 0.0044 (0.0046458917524897225)  --> TIME: 2025-02-25 12:13:37 -- STEP: 36390/36938 -- GLOBAL_STEP: 775150 | > loss_text_ce: 0.03165409341454506 (0.034405281654410216) | > loss_mel_ce: 2.9744720458984375 (nan) | > loss: 0.011929072439670563 (nan) | > current_lr: 5e-06 | > step_time: 0.1522 (0.13653317062564604) | > loader_time: 0.0047 (0.0046459193870437435)  --> TIME: 2025-02-25 12:13:47 -- STEP: 36440/36938 -- GLOBAL_STEP: 775200 | > loss_text_ce: 0.03540172427892685 (0.034402958863786796) | > loss_mel_ce: 2.9401097297668457 (nan) | > loss: 0.011807586066424847 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13653422946725885) | > loader_time: 0.0046 (0.004645947423384294)  --> TIME: 2025-02-25 12:13:56 -- STEP: 36490/36938 -- GLOBAL_STEP: 775250 | > loss_text_ce: 0.03472495451569557 (0.034401163261763394) | > loss_mel_ce: 3.277331590652466 (nan) | > loss: 0.013143082149326801 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13653273014017467) | > loader_time: 0.0045 (0.004645882021338681)  --> TIME: 2025-02-25 12:14:05 -- STEP: 36540/36938 -- GLOBAL_STEP: 775300 | > loss_text_ce: 0.03423361852765083 (0.03440003981230102) | > loss_mel_ce: 2.4456660747528076 (nan) | > loss: 0.009840872138738632 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13653292380595594) | > loader_time: 0.0046 (0.004645831498802193)  --> TIME: 2025-02-25 12:14:15 -- STEP: 36590/36938 -- GLOBAL_STEP: 775350 | > loss_text_ce: 0.030329471454024315 (0.034401310549070876) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13653557187841797) | > loader_time: 0.0046 (0.004645795651425684)  --> TIME: 2025-02-25 12:14:24 -- STEP: 36640/36938 -- GLOBAL_STEP: 775400 | > loss_text_ce: 0.03684057667851448 (0.03440256270660806) | > loss_mel_ce: 2.081103563308716 (nan) | > loss: 0.008404540829360485 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13653366512346563) | > loader_time: 0.0047 (0.00464574163006903)  --> TIME: 2025-02-25 12:14:33 -- STEP: 36690/36938 -- GLOBAL_STEP: 775450 | > loss_text_ce: 0.024565009400248528 (0.034403682281901375) | > loss_mel_ce: 1.964202880859375 (nan) | > loss: 0.007891936227679253 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.1365308752824132) | > loader_time: 0.0046 (0.004645683980501454)  --> TIME: 2025-02-25 12:14:43 -- STEP: 36740/36938 -- GLOBAL_STEP: 775500 | > loss_text_ce: 0.030974827706813812 (0.03440290279394915) | > loss_mel_ce: 2.70033597946167 (nan) | > loss: 0.010838535614311695 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13653467963685487) | > loader_time: 0.0045 (0.0046456737043290355)  --> TIME: 2025-02-25 12:14:52 -- STEP: 36790/36938 -- GLOBAL_STEP: 775550 | > loss_text_ce: 0.04697432368993759 (0.0344027012232947) | > loss_mel_ce: 0.09130859375 (nan) | > loss: 0.0005487417802214622 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13653513640610948) | > loader_time: 0.0046 (0.004645639504061728)  --> TIME: 2025-02-25 12:15:01 -- STEP: 36840/36938 -- GLOBAL_STEP: 775600 | > loss_text_ce: 0.0357184112071991 (0.03440358190689553) | > loss_mel_ce: 3.113201379776001 (nan) | > loss: 0.012495714239776134 (nan) | > current_lr: 5e-06 | > step_time: 0.1447 (0.1365344393525148) | > loader_time: 0.0046 (0.004645612263135873)  --> TIME: 2025-02-25 12:15:11 -- STEP: 36890/36938 -- GLOBAL_STEP: 775650 | > loss_text_ce: 0.03588443994522095 (0.03440386714927747) | > loss_mel_ce: 3.2572028636932373 (nan) | > loss: 0.01306780707091093 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13654118099972767) | > loader_time: 0.0046 (0.004645586679478775)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.02086423765314688 (-0.0002988175590439533) | > avg_loss_text_ce: 0.03355918868933572 (-0.00035050938293192113) | > avg_loss_mel_ce: 2.548962063423478 (-0.012712650664961522) | > avg_loss: 2.5825212636206425 (-0.013063141910153675) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_775698.pth  > EPOCH: 22/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-25 12:15:47)   --> TIME: 2025-02-25 12:15:48 -- STEP: 2/36938 -- GLOBAL_STEP: 775700 | > loss_text_ce: 0.03592142090201378 (0.032717762514948845) | > loss_mel_ce: 2.201279640197754 (2.75480318069458) | > loss: 0.00887778215110302 (0.011061591561883688) | > current_lr: 5e-06 | > step_time: 0.1614 (0.16177165508270264) | > loader_time: 0.0061 (0.005473494529724121)  --> TIME: 2025-02-25 12:15:57 -- STEP: 52/36938 -- GLOBAL_STEP: 775750 | > loss_text_ce: 0.033434707671403885 (0.03415896249218629) | > loss_mel_ce: 2.890861988067627 (2.56181007824265) | > loss: 0.011604351922869682 (0.010301465009643625) | > current_lr: 5e-06 | > step_time: 0.1609 (0.13448738134824312) | > loader_time: 0.0049 (0.004758105828211857)  --> TIME: 2025-02-25 12:16:06 -- STEP: 102/36938 -- GLOBAL_STEP: 775800 | > loss_text_ce: 0.033291496336460114 (0.03449743967868534) | > loss_mel_ce: 2.939377546310425 (2.5980984781887004) | > loss: 0.011796306818723679 (0.010446809814739565) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13155622809540993) | > loader_time: 0.0047 (0.004737061612746296)  --> TIME: 2025-02-25 12:16:15 -- STEP: 152/36938 -- GLOBAL_STEP: 775850 | > loss_text_ce: 0.03249296173453331 (0.03423649712318653) | > loss_mel_ce: 3.0618631839752197 (2.6034863385322837) | > loss: 0.012279191054403782 (0.010467154737757245) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1311427292070891) | > loader_time: 0.0046 (0.004710343323255839)  --> TIME: 2025-02-25 12:16:24 -- STEP: 202/36938 -- GLOBAL_STEP: 775900 | > loss_text_ce: 0.04066767543554306 (0.03421660112032646) | > loss_mel_ce: 2.043696880340576 (2.596743730699072) | > loss: 0.008271289058029652 (0.010440319366992272) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1317297833980901) | > loader_time: 0.0045 (0.00469353765544325)  --> TIME: 2025-02-25 12:16:33 -- STEP: 252/36938 -- GLOBAL_STEP: 775950 | > loss_text_ce: 0.032899145036935806 (0.0342884466591631) | > loss_mel_ce: 2.8676843643188477 (nan) | > loss: 0.011510252952575684 (nan) | > current_lr: 5e-06 | > step_time: 0.1503 (0.13283957659252107) | > loader_time: 0.0048 (0.0046810025260562)  --> TIME: 2025-02-25 12:16:43 -- STEP: 302/36938 -- GLOBAL_STEP: 776000 | > loss_text_ce: 0.03325001522898674 (0.03425688418985301) | > loss_mel_ce: 3.2193758487701416 (nan) | > loss: 0.012907246127724648 (nan) | > current_lr: 5e-06 | > step_time: 0.1942 (0.1335141714045544) | > loader_time: 0.005 (0.004679854342479582)  --> TIME: 2025-02-25 12:16:52 -- STEP: 352/36938 -- GLOBAL_STEP: 776050 | > loss_text_ce: 0.029491132125258446 (0.03423778231477959) | > loss_mel_ce: 1.56903874874115 (nan) | > loss: 0.006343373097479343 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13346530429341577) | > loader_time: 0.0044 (0.004684096032922917)  --> TIME: 2025-02-25 12:17:01 -- STEP: 402/36938 -- GLOBAL_STEP: 776100 | > loss_text_ce: 0.040800776332616806 (0.03434230928165625) | > loss_mel_ce: 1.8348032236099243 (nan) | > loss: 0.007442873436957598 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.1336646595997599) | > loader_time: 0.0045 (0.004678002044336118)  --> TIME: 2025-02-25 12:17:10 -- STEP: 452/36938 -- GLOBAL_STEP: 776150 | > loss_text_ce: 0.033694539219141006 (0.03439005659851945) | > loss_mel_ce: 1.6019799709320068 (nan) | > loss: 0.006490772124379873 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1340242028236391) | > loader_time: 0.0046 (0.004680159872612066)  --> TIME: 2025-02-25 12:17:20 -- STEP: 502/36938 -- GLOBAL_STEP: 776200 | > loss_text_ce: 0.03191560506820679 (0.03426764431628453) | > loss_mel_ce: 2.629297971725464 (nan) | > loss: 0.010560371913015842 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13452657142958332) | > loader_time: 0.0049 (0.004683795203250717)  --> TIME: 2025-02-25 12:17:29 -- STEP: 552/36938 -- GLOBAL_STEP: 776250 | > loss_text_ce: 0.02511654980480671 (0.034248328330737626) | > loss_mel_ce: 2.255581855773926 (nan) | > loss: 0.009050390683114529 (nan) | > current_lr: 5e-06 | > step_time: 0.1569 (0.13452422964400154) | > loader_time: 0.0047 (0.004684065563091334)  --> TIME: 2025-02-25 12:17:38 -- STEP: 602/36938 -- GLOBAL_STEP: 776300 | > loss_text_ce: 0.029889898374676704 (0.034229647778160247) | > loss_mel_ce: 2.7968602180480957 (nan) | > loss: 0.011217262595891953 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13471765058777274) | > loader_time: 0.0046 (0.004688015411858542)  --> TIME: 2025-02-25 12:17:48 -- STEP: 652/36938 -- GLOBAL_STEP: 776350 | > loss_text_ce: 0.02947237901389599 (0.03420694007972496) | > loss_mel_ce: 2.9751858711242676 (nan) | > loss: 0.011923247948288918 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.13481988314470641) | > loader_time: 0.0046 (0.004685438118097971)  --> TIME: 2025-02-25 12:17:57 -- STEP: 702/36938 -- GLOBAL_STEP: 776400 | > loss_text_ce: 0.0406574010848999 (0.034233799356797386) | > loss_mel_ce: 3.216412305831909 (nan) | > loss: 0.012924879789352417 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.1349250118956609) | > loader_time: 0.0047 (0.0046822824369468575)  --> TIME: 2025-02-25 12:18:06 -- STEP: 752/36938 -- GLOBAL_STEP: 776450 | > loss_text_ce: 0.04121178388595581 (0.034177657507825615) | > loss_mel_ce: 2.6200692653656006 (nan) | > loss: 0.010560640133917332 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13491959394292652) | > loader_time: 0.0046 (0.004678546748262769)  --> TIME: 2025-02-25 12:18:15 -- STEP: 802/36938 -- GLOBAL_STEP: 776500 | > loss_text_ce: 0.028221696615219116 (0.034147937272319955) | > loss_mel_ce: 3.1061768531799316 (nan) | > loss: 0.012438089586794376 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13497073721707337) | > loader_time: 0.0046 (0.004675358310899229)  --> TIME: 2025-02-25 12:18:25 -- STEP: 852/36938 -- GLOBAL_STEP: 776550 | > loss_text_ce: 0.03246114030480385 (0.03422357342337109) | > loss_mel_ce: 0.6589469909667969 (nan) | > loss: 0.0027436832897365093 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13492818813368762) | > loader_time: 0.0045 (0.0046709333786942)  --> TIME: 2025-02-25 12:18:34 -- STEP: 902/36938 -- GLOBAL_STEP: 776600 | > loss_text_ce: 0.03567495942115784 (0.03424127323619041) | > loss_mel_ce: 2.8786189556121826 (nan) | > loss: 0.01156465895473957 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.13533522548802415) | > loader_time: 0.0047 (0.004674853876794782)  --> TIME: 2025-02-25 12:18:44 -- STEP: 952/36938 -- GLOBAL_STEP: 776650 | > loss_text_ce: 0.03837088122963905 (0.03418719489225223) | > loss_mel_ce: 2.8693020343780518 (nan) | > loss: 0.011538385413587093 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1354434938490892) | > loader_time: 0.0046 (0.00467686367635967)  --> TIME: 2025-02-25 12:18:53 -- STEP: 1002/36938 -- GLOBAL_STEP: 776700 | > loss_text_ce: 0.03043212741613388 (0.03413461189728205) | > loss_mel_ce: 2.9843833446502686 (nan) | > loss: 0.0119635546579957 (nan) | > current_lr: 5e-06 | > step_time: 0.1606 (0.13548320186828192) | > loader_time: 0.0047 (0.004676144042176874)  --> TIME: 2025-02-25 12:19:03 -- STEP: 1052/36938 -- GLOBAL_STEP: 776750 | > loss_text_ce: 0.03332752734422684 (0.034164628073282716) | > loss_mel_ce: 2.231034517288208 (nan) | > loss: 0.008985564112663269 (nan) | > current_lr: 5e-06 | > step_time: 0.1489 (0.1357789594864211) | > loader_time: 0.0047 (0.004677689120796694)  --> TIME: 2025-02-25 12:19:12 -- STEP: 1102/36938 -- GLOBAL_STEP: 776800 | > loss_text_ce: 0.03988315910100937 (0.03415395164044195) | > loss_mel_ce: 3.017345666885376 (nan) | > loss: 0.012131861411035061 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13583539228906655) | > loader_time: 0.0047 (0.004680754268667007)  --> TIME: 2025-02-25 12:19:21 -- STEP: 1152/36938 -- GLOBAL_STEP: 776850 | > loss_text_ce: 0.02831350825726986 (0.03413486278198733) | > loss_mel_ce: 2.0809760093688965 (nan) | > loss: 0.00837019644677639 (nan) | > current_lr: 5e-06 | > step_time: 0.108 (0.13558756497999033) | > loader_time: 0.0045 (0.0046789695819218925)  --> TIME: 2025-02-25 12:19:30 -- STEP: 1202/36938 -- GLOBAL_STEP: 776900 | > loss_text_ce: 0.03786596655845642 (0.03408766356366567) | > loss_mel_ce: 3.157682418823242 (nan) | > loss: 0.012680747546255589 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.135769626860214) | > loader_time: 0.0046 (0.004681969442700777)  --> TIME: 2025-02-25 12:19:40 -- STEP: 1252/36938 -- GLOBAL_STEP: 776950 | > loss_text_ce: 0.035340629518032074 (0.03401093637517178) | > loss_mel_ce: 2.0505483150482178 (nan) | > loss: 0.008277337066829205 (nan) | > current_lr: 5e-06 | > step_time: 0.1518 (0.1357785215773903) | > loader_time: 0.0047 (0.004683173312165859)  --> TIME: 2025-02-25 12:19:49 -- STEP: 1302/36938 -- GLOBAL_STEP: 777000 | > loss_text_ce: 0.03500379994511604 (0.03403071790201139) | > loss_mel_ce: 3.2309136390686035 (nan) | > loss: 0.01295999065041542 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.1359635955910163) | > loader_time: 0.0046 (0.004684177594983257)  --> TIME: 2025-02-25 12:19:59 -- STEP: 1352/36938 -- GLOBAL_STEP: 777050 | > loss_text_ce: 0.03748662769794464 (0.034030210152378636) | > loss_mel_ce: 2.7959272861480713 (nan) | > loss: 0.011243706569075584 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13598400671806554) | > loader_time: 0.0046 (0.00468290822040399)  --> TIME: 2025-02-25 12:20:08 -- STEP: 1402/36938 -- GLOBAL_STEP: 777100 | > loss_text_ce: 0.04214362055063248 (0.034000825528825995) | > loss_mel_ce: 2.703979015350342 (nan) | > loss: 0.010897312313318253 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13599380974082562) | > loader_time: 0.0046 (0.004683424674835412)  --> TIME: 2025-02-25 12:20:17 -- STEP: 1452/36938 -- GLOBAL_STEP: 777150 | > loss_text_ce: 0.03985242173075676 (0.034020362062682455) | > loss_mel_ce: 2.7065789699554443 (nan) | > loss: 0.010898537933826447 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13597485496978123) | > loader_time: 0.0046 (0.0046822637237464645)  --> TIME: 2025-02-25 12:20:26 -- STEP: 1502/36938 -- GLOBAL_STEP: 777200 | > loss_text_ce: 0.030850805342197418 (0.03394994706959621) | > loss_mel_ce: 2.6922242641448975 (nan) | > loss: 0.010805854573845863 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.1357930612945048) | > loader_time: 0.0047 (0.004684627770425153)  --> TIME: 2025-02-25 12:20:35 -- STEP: 1552/36938 -- GLOBAL_STEP: 777250 | > loss_text_ce: 0.035403020679950714 (0.033956206217048104) | > loss_mel_ce: 3.211900472640991 (nan) | > loss: 0.012886125594377518 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.13580643301157605) | > loader_time: 0.0047 (0.004686622582760055)  --> TIME: 2025-02-25 12:20:44 -- STEP: 1602/36938 -- GLOBAL_STEP: 777300 | > loss_text_ce: 0.035132817924022675 (0.033959615287881356) | > loss_mel_ce: 2.6429708003997803 (nan) | > loss: 0.010627396404743195 (nan) | > current_lr: 5e-06 | > step_time: 0.1506 (0.13569049739956696) | > loader_time: 0.0047 (0.004686325379227583)  --> TIME: 2025-02-25 12:20:54 -- STEP: 1652/36938 -- GLOBAL_STEP: 777350 | > loss_text_ce: 0.039319563657045364 (0.03397313749798757) | > loss_mel_ce: 3.6185302734375 (nan) | > loss: 0.014515277929604053 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13582013190225578) | > loader_time: 0.0046 (0.0046884504126578855)  --> TIME: 2025-02-25 12:21:03 -- STEP: 1702/36938 -- GLOBAL_STEP: 777400 | > loss_text_ce: 0.038869574666023254 (0.03396748092636183) | > loss_mel_ce: 3.129791259765625 (nan) | > loss: 0.01257405150681734 (nan) | > current_lr: 5e-06 | > step_time: 0.1462 (0.13580117452578855) | > loader_time: 0.0047 (0.004687836671688029)  --> TIME: 2025-02-25 12:21:13 -- STEP: 1752/36938 -- GLOBAL_STEP: 777450 | > loss_text_ce: 0.029200758785009384 (0.03397122800050378) | > loss_mel_ce: 2.667510509490967 (nan) | > loss: 0.010701236315071583 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13589582595650973) | > loader_time: 0.0046 (0.004687689074642588)  --> TIME: 2025-02-25 12:21:22 -- STEP: 1802/36938 -- GLOBAL_STEP: 777500 | > loss_text_ce: 0.037801988422870636 (0.03399647688947941) | > loss_mel_ce: 2.958294153213501 (nan) | > loss: 0.011889271438121796 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.135899255992834) | > loader_time: 0.0047 (0.004687997927014762)  --> TIME: 2025-02-25 12:21:31 -- STEP: 1852/36938 -- GLOBAL_STEP: 777550 | > loss_text_ce: 0.02815631777048111 (0.03399195836014143) | > loss_mel_ce: 1.9987772703170776 (nan) | > loss: 0.008043387904763222 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13588689663497683) | > loader_time: 0.0044 (0.004688779427217362)  --> TIME: 2025-02-25 12:21:41 -- STEP: 1902/36938 -- GLOBAL_STEP: 777600 | > loss_text_ce: 0.03374814987182617 (0.03400999897714904) | > loss_mel_ce: 2.8960957527160645 (nan) | > loss: 0.011626365594565868 (nan) | > current_lr: 5e-06 | > step_time: 0.1701 (0.13601256707990667) | > loader_time: 0.0048 (0.004688785279210815)  --> TIME: 2025-02-25 12:21:50 -- STEP: 1952/36938 -- GLOBAL_STEP: 777650 | > loss_text_ce: 0.02743830531835556 (0.03402364603482324) | > loss_mel_ce: 3.181781768798828 (nan) | > loss: 0.012735001742839813 (nan) | > current_lr: 5e-06 | > step_time: 0.1574 (0.13602979029299742) | > loader_time: 0.0047 (0.0046893392429977225)  --> TIME: 2025-02-25 12:22:00 -- STEP: 2002/36938 -- GLOBAL_STEP: 777700 | > loss_text_ce: 0.02841813676059246 (0.03405093746892475) | > loss_mel_ce: 2.5363223552703857 (nan) | > loss: 0.01017754152417183 (nan) | > current_lr: 5e-06 | > step_time: 0.1687 (0.1359972219009853) | > loader_time: 0.0048 (0.004689133726990788)  --> TIME: 2025-02-25 12:22:09 -- STEP: 2052/36938 -- GLOBAL_STEP: 777750 | > loss_text_ce: 0.04245617613196373 (0.03404371440410614) | > loss_mel_ce: 2.5199778079986572 (nan) | > loss: 0.010168389417231083 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.1359319976198739) | > loader_time: 0.0047 (0.004687151248924454)  --> TIME: 2025-02-25 12:22:18 -- STEP: 2102/36938 -- GLOBAL_STEP: 777800 | > loss_text_ce: 0.03600898012518883 (0.03402881367445758) | > loss_mel_ce: 2.4399242401123047 (nan) | > loss: 0.00982513278722763 (nan) | > current_lr: 5e-06 | > step_time: 0.178 (0.13608023600845953) | > loader_time: 0.0048 (0.004687104533673698)  --> TIME: 2025-02-25 12:22:28 -- STEP: 2152/36938 -- GLOBAL_STEP: 777850 | > loss_text_ce: 0.03326048329472542 (0.03400998380118254) | > loss_mel_ce: 3.5137810707092285 (nan) | > loss: 0.014075563289225101 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.13605843190810019) | > loader_time: 0.0046 (0.004686256102026621)  --> TIME: 2025-02-25 12:22:37 -- STEP: 2202/36938 -- GLOBAL_STEP: 777900 | > loss_text_ce: 0.04390651732683182 (0.034007174646368515) | > loss_mel_ce: 0.5708996653556824 (nan) | > loss: 0.0024397072847932577 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1361420765234922) | > loader_time: 0.0045 (0.004685083808517806)  --> TIME: 2025-02-25 12:22:46 -- STEP: 2252/36938 -- GLOBAL_STEP: 777950 | > loss_text_ce: 0.03567982092499733 (0.03400081193544378) | > loss_mel_ce: 2.9804134368896484 (nan) | > loss: 0.011968624778091908 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13615928545303055) | > loader_time: 0.0046 (0.004685670825558488)  --> TIME: 2025-02-25 12:22:56 -- STEP: 2302/36938 -- GLOBAL_STEP: 778000 | > loss_text_ce: 0.034633003175258636 (0.03397211654728308) | > loss_mel_ce: 3.089898109436035 (nan) | > loss: 0.012398933060467243 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1361822079410767) | > loader_time: 0.0047 (0.0046848375625345206)  --> TIME: 2025-02-25 12:23:05 -- STEP: 2352/36938 -- GLOBAL_STEP: 778050 | > loss_text_ce: 0.03732655197381973 (0.03396149678472554) | > loss_mel_ce: 0.0383097343146801 (nan) | > loss: 0.00030014399089850485 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13620098219031368) | > loader_time: 0.0046 (0.004684155591491126)  --> TIME: 2025-02-25 12:23:14 -- STEP: 2402/36938 -- GLOBAL_STEP: 778100 | > loss_text_ce: 0.031521208584308624 (0.03396745926415939) | > loss_mel_ce: 2.902623414993286 (nan) | > loss: 0.011643432080745697 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13614823141264767) | > loader_time: 0.0046 (0.004685790711497385)  --> TIME: 2025-02-25 12:23:23 -- STEP: 2452/36938 -- GLOBAL_STEP: 778150 | > loss_text_ce: 0.03545082360506058 (0.033947954324285275) | > loss_mel_ce: 2.581987142562866 (nan) | > loss: 0.010386659763753414 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.13605717646161647) | > loader_time: 0.0046 (0.004684507652363612)  --> TIME: 2025-02-25 12:23:33 -- STEP: 2502/36938 -- GLOBAL_STEP: 778200 | > loss_text_ce: 0.03743816167116165 (0.033959926278006536) | > loss_mel_ce: 2.545171022415161 (nan) | > loss: 0.010248449631035328 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13605887426746827) | > loader_time: 0.0046 (0.004684709054198292)  --> TIME: 2025-02-25 12:23:42 -- STEP: 2552/36938 -- GLOBAL_STEP: 778250 | > loss_text_ce: 0.0334157794713974 (0.033968632884592155) | > loss_mel_ce: 2.2821879386901855 (nan) | > loss: 0.00918890442699194 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13598411705426655) | > loader_time: 0.0046 (0.0046839280561967326)  --> TIME: 2025-02-25 12:23:51 -- STEP: 2602/36938 -- GLOBAL_STEP: 778300 | > loss_text_ce: 0.03125359117984772 (0.0339864688022756) | > loss_mel_ce: 3.13909912109375 (nan) | > loss: 0.012580765411257744 (nan) | > current_lr: 5e-06 | > step_time: 0.1517 (0.13596839642726302) | > loader_time: 0.0048 (0.004683452968318858)  --> TIME: 2025-02-25 12:24:00 -- STEP: 2652/36938 -- GLOBAL_STEP: 778350 | > loss_text_ce: 0.04795846715569496 (0.034013572870999466) | > loss_mel_ce: 2.7682642936706543 (nan) | > loss: 0.011175487190485 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13596829183141207) | > loader_time: 0.0045 (0.004683570804279491)  --> TIME: 2025-02-25 12:24:09 -- STEP: 2702/36938 -- GLOBAL_STEP: 778400 | > loss_text_ce: 0.03599679097533226 (0.034004804287395274) | > loss_mel_ce: 2.766089677810669 (nan) | > loss: 0.011119390837848186 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13594034581074968) | > loader_time: 0.0049 (0.004683551128311208)  --> TIME: 2025-02-25 12:24:19 -- STEP: 2752/36938 -- GLOBAL_STEP: 778450 | > loss_text_ce: 0.024893956258893013 (0.034044910002321085) | > loss_mel_ce: 2.3099234104156494 (nan) | > loss: 0.009265149012207985 (nan) | > current_lr: 5e-06 | > step_time: 0.1084 (0.13600114585701806) | > loader_time: 0.0045 (0.004684356062911273)  --> TIME: 2025-02-25 12:24:28 -- STEP: 2802/36938 -- GLOBAL_STEP: 778500 | > loss_text_ce: 0.03132283687591553 (0.034075645403504384) | > loss_mel_ce: 3.3324673175811768 (nan) | > loss: 0.013348373584449291 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13599675285399251) | > loader_time: 0.0046 (0.0046840757237937404)  --> TIME: 2025-02-25 12:24:38 -- STEP: 2852/36938 -- GLOBAL_STEP: 778550 | > loss_text_ce: 0.028473777696490288 (0.03407987969866097) | > loss_mel_ce: 3.328533887863159 (nan) | > loss: 0.013321460224688053 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13603969961195805) | > loader_time: 0.0047 (0.004683516303085176)  --> TIME: 2025-02-25 12:24:47 -- STEP: 2902/36938 -- GLOBAL_STEP: 778600 | > loss_text_ce: 0.03484410420060158 (0.034070782711938634) | > loss_mel_ce: 3.4340944290161133 (nan) | > loss: 0.013765630312263966 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13607272668019582) | > loader_time: 0.0046 (0.00468393985687824)  --> TIME: 2025-02-25 12:24:56 -- STEP: 2952/36938 -- GLOBAL_STEP: 778650 | > loss_text_ce: 0.044650834053754807 (0.03410398266858702) | > loss_mel_ce: 3.2907047271728516 (nan) | > loss: 0.013235538266599178 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.1360709903039908) | > loader_time: 0.0046 (0.004683053590417883)  --> TIME: 2025-02-25 12:25:06 -- STEP: 3002/36938 -- GLOBAL_STEP: 778700 | > loss_text_ce: 0.04087284579873085 (0.034127742527356744) | > loss_mel_ce: 3.48287296295166 (nan) | > loss: 0.013983119279146194 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.13612242836224728) | > loader_time: 0.0049 (0.004683313569889167)  --> TIME: 2025-02-25 12:25:15 -- STEP: 3052/36938 -- GLOBAL_STEP: 778750 | > loss_text_ce: 0.028912359848618507 (0.03415705517683844) | > loss_mel_ce: 2.2668142318725586 (nan) | > loss: 0.009110026061534882 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13609409590067434) | > loader_time: 0.0045 (0.00468220013934695)  --> TIME: 2025-02-25 12:25:24 -- STEP: 3102/36938 -- GLOBAL_STEP: 778800 | > loss_text_ce: 0.030844366177916527 (0.03414941410912977) | > loss_mel_ce: 1.5134943723678589 (nan) | > loss: 0.006128328386694193 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13608107353163718) | > loader_time: 0.0046 (0.004680967192585284)  --> TIME: 2025-02-25 12:25:34 -- STEP: 3152/36938 -- GLOBAL_STEP: 778850 | > loss_text_ce: 0.03873717412352562 (0.034148461172395006) | > loss_mel_ce: 3.1822938919067383 (nan) | > loss: 0.012781869620084763 (nan) | > current_lr: 5e-06 | > step_time: 0.1755 (0.13614816653546938) | > loader_time: 0.0048 (0.0046810575850724035)  --> TIME: 2025-02-25 12:25:43 -- STEP: 3202/36938 -- GLOBAL_STEP: 778900 | > loss_text_ce: 0.04208065941929817 (0.03414688736484207) | > loss_mel_ce: 3.435138702392578 (nan) | > loss: 0.013798490166664124 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13615813060226759) | > loader_time: 0.0046 (0.00468007979133888)  --> TIME: 2025-02-25 12:25:52 -- STEP: 3252/36938 -- GLOBAL_STEP: 778950 | > loss_text_ce: 0.04996423050761223 (0.034154396839233644) | > loss_mel_ce: 1.5902634859085083 (nan) | > loss: 0.006508840247988701 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.13608384858198255) | > loader_time: 0.0045 (0.00467873660694043)  --> TIME: 2025-02-25 12:26:01 -- STEP: 3302/36938 -- GLOBAL_STEP: 779000 | > loss_text_ce: 0.02963300794363022 (0.03418392713589516) | > loss_mel_ce: 0.145416259765625 (nan) | > loss: 0.0006946400390006602 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13611381065477235) | > loader_time: 0.0046 (0.0046786659779077915)  --> TIME: 2025-02-25 12:26:11 -- STEP: 3352/36938 -- GLOBAL_STEP: 779050 | > loss_text_ce: 0.03354981541633606 (0.03416651596041808) | > loss_mel_ce: 2.8221499919891357 (nan) | > loss: 0.011332142166793346 (nan) | > current_lr: 5e-06 | > step_time: 0.1485 (0.13607131766248703) | > loader_time: 0.0046 (0.004677421295557501)  --> TIME: 2025-02-25 12:26:20 -- STEP: 3402/36938 -- GLOBAL_STEP: 779100 | > loss_text_ce: 0.04490390047430992 (0.03417439627319328) | > loss_mel_ce: 3.0145299434661865 (nan) | > loss: 0.012140611186623573 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13607729343299912) | > loader_time: 0.0046 (0.004676723816617671)  --> TIME: 2025-02-25 12:26:29 -- STEP: 3452/36938 -- GLOBAL_STEP: 779150 | > loss_text_ce: 0.03121388517320156 (0.034186974857412594) | > loss_mel_ce: 3.121381998062134 (nan) | > loss: 0.012510302476584911 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13600529220952345) | > loader_time: 0.0046 (0.004675910964625463)  --> TIME: 2025-02-25 12:26:38 -- STEP: 3502/36938 -- GLOBAL_STEP: 779200 | > loss_text_ce: 0.03590145334601402 (0.03418444242963472) | > loss_mel_ce: 2.4060590267181396 (nan) | > loss: 0.009690320119261742 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13596316074249323) | > loader_time: 0.0045 (0.004674569393552286)  --> TIME: 2025-02-25 12:26:47 -- STEP: 3552/36938 -- GLOBAL_STEP: 779250 | > loss_text_ce: 0.04568227753043175 (0.03418941954035429) | > loss_mel_ce: 2.745302677154541 (nan) | > loss: 0.011075337417423725 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.1359552224879865) | > loader_time: 0.0046 (0.004673760753494129)  --> TIME: 2025-02-25 12:26:57 -- STEP: 3602/36938 -- GLOBAL_STEP: 779300 | > loss_text_ce: 0.034654028713703156 (0.034186427002624774) | > loss_mel_ce: 3.289858341217041 (nan) | > loss: 0.013192510232329369 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.1359972822738976) | > loader_time: 0.0047 (0.004673631372615939)  --> TIME: 2025-02-25 12:27:06 -- STEP: 3652/36938 -- GLOBAL_STEP: 779350 | > loss_text_ce: 0.035512104630470276 (0.0341741021443056) | > loss_mel_ce: 2.222810983657837 (nan) | > loss: 0.008961600251495838 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13595205858790438) | > loader_time: 0.0046 (0.00467313132437532)  --> TIME: 2025-02-25 12:27:15 -- STEP: 3702/36938 -- GLOBAL_STEP: 779400 | > loss_text_ce: 0.041779909282922745 (0.03417116298975729) | > loss_mel_ce: 2.9518511295318604 (nan) | > loss: 0.011879488825798035 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13594243611470191) | > loader_time: 0.0046 (0.004672404755005513)  --> TIME: 2025-02-25 12:27:24 -- STEP: 3752/36938 -- GLOBAL_STEP: 779450 | > loss_text_ce: 0.0382460281252861 (0.034166123267255176) | > loss_mel_ce: 2.842677593231201 (nan) | > loss: 0.01143223699182272 (nan) | > current_lr: 5e-06 | > step_time: 0.1724 (0.13597894896830556) | > loader_time: 0.0048 (0.004671408868293522)  --> TIME: 2025-02-25 12:27:34 -- STEP: 3802/36938 -- GLOBAL_STEP: 779500 | > loss_text_ce: 0.0323420986533165 (0.03415272992764688) | > loss_mel_ce: 3.0926852226257324 (nan) | > loss: 0.012400902807712555 (nan) | > current_lr: 5e-06 | > step_time: 0.1825 (0.1360175824052466) | > loader_time: 0.0048 (0.004670721678154401)  --> TIME: 2025-02-25 12:27:43 -- STEP: 3852/36938 -- GLOBAL_STEP: 779550 | > loss_text_ce: 0.03257826715707779 (0.034150074208532395) | > loss_mel_ce: 2.922860860824585 (nan) | > loss: 0.011727933771908283 (nan) | > current_lr: 5e-06 | > step_time: 0.1858 (0.13596965658825605) | > loader_time: 0.0048 (0.004669951501293721)  --> TIME: 2025-02-25 12:27:52 -- STEP: 3902/36938 -- GLOBAL_STEP: 779600 | > loss_text_ce: 0.037302885204553604 (0.03413082300606577) | > loss_mel_ce: 3.1347732543945312 (nan) | > loss: 0.01258760504424572 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.1359505989440337) | > loader_time: 0.0045 (0.004669098717197283)  --> TIME: 2025-02-25 12:28:01 -- STEP: 3952/36938 -- GLOBAL_STEP: 779650 | > loss_text_ce: 0.032791584730148315 (0.03411869102380109) | > loss_mel_ce: 1.1682565212249756 (nan) | > loss: 0.00476606423035264 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1359638861919702) | > loader_time: 0.0045 (0.004668379300519043)  --> TIME: 2025-02-25 12:28:11 -- STEP: 4002/36938 -- GLOBAL_STEP: 779700 | > loss_text_ce: 0.025949226692318916 (0.0341130945862255) | > loss_mel_ce: 2.360037326812744 (nan) | > loss: 0.0094682015478611 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13597202974221256) | > loader_time: 0.0046 (0.004667927836370972)  --> TIME: 2025-02-25 12:28:20 -- STEP: 4052/36938 -- GLOBAL_STEP: 779750 | > loss_text_ce: 0.028813375160098076 (0.034124981014835735) | > loss_mel_ce: 3.0098307132720947 (nan) | > loss: 0.01205811183899641 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13602433853064663) | > loader_time: 0.0045 (0.004667167543541176)  --> TIME: 2025-02-25 12:28:30 -- STEP: 4102/36938 -- GLOBAL_STEP: 779800 | > loss_text_ce: 0.03280704841017723 (0.03412405429258102) | > loss_mel_ce: 2.5640571117401123 (nan) | > loss: 0.010305017232894897 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13607036125828476) | > loader_time: 0.0046 (0.004666706342804092)  --> TIME: 2025-02-25 12:28:39 -- STEP: 4152/36938 -- GLOBAL_STEP: 779850 | > loss_text_ce: 0.03792111203074455 (0.03414561733718318) | > loss_mel_ce: 2.6306612491607666 (nan) | > loss: 0.01058961357921362 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13610209958172007) | > loader_time: 0.0046 (0.0046661103391922995)  --> TIME: 2025-02-25 12:28:49 -- STEP: 4202/36938 -- GLOBAL_STEP: 779900 | > loss_text_ce: 0.03161684051156044 (0.03415204763283034) | > loss_mel_ce: 2.6623787879943848 (nan) | > loss: 0.010690459981560707 (nan) | > current_lr: 5e-06 | > step_time: 0.1818 (0.13608469893851752) | > loader_time: 0.0048 (0.004666260399743526)  --> TIME: 2025-02-25 12:28:58 -- STEP: 4252/36938 -- GLOBAL_STEP: 779950 | > loss_text_ce: 0.03958301246166229 (0.03416172091727362) | > loss_mel_ce: 3.2671635150909424 (nan) | > loss: 0.013122010976076126 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13606645257991076) | > loader_time: 0.0046 (0.004666141654002926)  --> TIME: 2025-02-25 12:29:07 -- STEP: 4302/36938 -- GLOBAL_STEP: 780000 | > loss_text_ce: 0.030127648264169693 (0.03415466067114922) | > loss_mel_ce: 2.7847306728363037 (nan) | > loss: 0.01117007341235876 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.13606622511705405) | > loader_time: 0.0049 (0.0046662153171307) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_780000.pth  --> TIME: 2025-02-25 12:29:22 -- STEP: 4352/36938 -- GLOBAL_STEP: 780050 | > loss_text_ce: 0.036041148006916046 (0.03416507463707065) | > loss_mel_ce: 2.1396355628967285 (nan) | > loss: 0.008633638732135296 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.13604735462542816) | > loader_time: 0.0046 (0.004666190649218421)  --> TIME: 2025-02-25 12:29:31 -- STEP: 4402/36938 -- GLOBAL_STEP: 780100 | > loss_text_ce: 0.03823511675000191 (0.034167611212814454) | > loss_mel_ce: 3.3593907356262207 (nan) | > loss: 0.01348264329135418 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.13596984756258254) | > loader_time: 0.0047 (0.004666162100449199)  --> TIME: 2025-02-25 12:29:40 -- STEP: 4452/36938 -- GLOBAL_STEP: 780150 | > loss_text_ce: 0.02781636454164982 (0.03418376595066371) | > loss_mel_ce: 1.2064908742904663 (nan) | > loss: 0.004898045212030411 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.13584267155929156) | > loader_time: 0.0046 (0.00466533260953823)  --> TIME: 2025-02-25 12:29:49 -- STEP: 4502/36938 -- GLOBAL_STEP: 780200 | > loss_text_ce: 0.030280234292149544 (0.03416497160704171) | > loss_mel_ce: 0.00762939453125 (nan) | > loss: 0.00015043503663036972 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.1358041287209921) | > loader_time: 0.0047 (0.004664748417224323)  --> TIME: 2025-02-25 12:29:58 -- STEP: 4552/36938 -- GLOBAL_STEP: 780250 | > loss_text_ce: 0.03183411806821823 (0.0341572720080332) | > loss_mel_ce: 3.342355728149414 (nan) | > loss: 0.013389643281698227 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13580802837239395) | > loader_time: 0.0046 (0.004664372580751179)  --> TIME: 2025-02-25 12:30:08 -- STEP: 4602/36938 -- GLOBAL_STEP: 780300 | > loss_text_ce: 0.048939961940050125 (0.034172265525293635) | > loss_mel_ce: 0.8611692190170288 (nan) | > loss: 0.0036115446127951145 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13583629139606326) | > loader_time: 0.0045 (0.0046643514003198334)  --> TIME: 2025-02-25 12:30:17 -- STEP: 4652/36938 -- GLOBAL_STEP: 780350 | > loss_text_ce: 0.03847187012434006 (0.03417893265184211) | > loss_mel_ce: 3.2764246463775635 (nan) | > loss: 0.013154352083802223 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13582185943477204) | > loader_time: 0.0044 (0.0046639539820309515)  --> TIME: 2025-02-25 12:30:26 -- STEP: 4702/36938 -- GLOBAL_STEP: 780400 | > loss_text_ce: 0.026831844821572304 (0.034169193087458576) | > loss_mel_ce: 2.6695425510406494 (nan) | > loss: 0.010699898935854435 (nan) | > current_lr: 5e-06 | > step_time: 0.1827 (0.135841808893081) | > loader_time: 0.0049 (0.004663579923353119)  --> TIME: 2025-02-25 12:30:35 -- STEP: 4752/36938 -- GLOBAL_STEP: 780450 | > loss_text_ce: 0.021965267136693 (0.03416273083893721) | > loss_mel_ce: 3.0423176288604736 (nan) | > loss: 0.012159853242337704 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1358544480318974) | > loader_time: 0.0045 (0.004663422302364913)  --> TIME: 2025-02-25 12:30:45 -- STEP: 4802/36938 -- GLOBAL_STEP: 780500 | > loss_text_ce: 0.03053751401603222 (0.0341690695049029) | > loss_mel_ce: 2.4661357402801514 (nan) | > loss: 0.009907434694468975 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.13589887686542335) | > loader_time: 0.0046 (0.004663184353432435)  --> TIME: 2025-02-25 12:30:54 -- STEP: 4852/36938 -- GLOBAL_STEP: 780550 | > loss_text_ce: 0.03421814367175102 (0.03417517398219163) | > loss_mel_ce: 3.200608491897583 (nan) | > loss: 0.012836613692343235 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13592286883359675) | > loader_time: 0.0047 (0.004662742176433302)  --> TIME: 2025-02-25 12:31:04 -- STEP: 4902/36938 -- GLOBAL_STEP: 780600 | > loss_text_ce: 0.03481849655508995 (0.03415781944824489) | > loss_mel_ce: 2.3548295497894287 (nan) | > loss: 0.0094827301800251 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13591211812927098) | > loader_time: 0.0046 (0.004662111407637561)  --> TIME: 2025-02-25 12:31:13 -- STEP: 4952/36938 -- GLOBAL_STEP: 780650 | > loss_text_ce: 0.034870874136686325 (0.03414305478713575) | > loss_mel_ce: 3.481870174407959 (nan) | > loss: 0.013955322094261646 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.1359415310939024) | > loader_time: 0.0046 (0.004661817639247678)  --> TIME: 2025-02-25 12:31:22 -- STEP: 5002/36938 -- GLOBAL_STEP: 780700 | > loss_text_ce: 0.03797019645571709 (0.03415669265483579) | > loss_mel_ce: 0.00677490234375 (nan) | > loss: 0.00017755992303136736 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13595797170976873) | > loader_time: 0.0047 (0.0046615737383483005)  --> TIME: 2025-02-25 12:31:32 -- STEP: 5052/36938 -- GLOBAL_STEP: 780750 | > loss_text_ce: 0.030990585684776306 (0.03416166130987214) | > loss_mel_ce: 2.1383590698242188 (nan) | > loss: 0.008608531206846237 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1359519184060447) | > loader_time: 0.0045 (0.004661126591719445)  --> TIME: 2025-02-25 12:31:41 -- STEP: 5102/36938 -- GLOBAL_STEP: 780800 | > loss_text_ce: 0.05020059645175934 (0.03415831364253154) | > loss_mel_ce: 3.3835608959198 (nan) | > loss: 0.013626039028167725 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.13592981432335652) | > loader_time: 0.0046 (0.004660553205064688)  --> TIME: 2025-02-25 12:31:50 -- STEP: 5152/36938 -- GLOBAL_STEP: 780850 | > loss_text_ce: 0.038386039435863495 (0.03415335880690798) | > loss_mel_ce: 3.375217914581299 (nan) | > loss: 0.013546048663556576 (nan) | > current_lr: 5e-06 | > step_time: 0.1074 (0.13590930198660553) | > loader_time: 0.0056 (0.004660812131366383)  --> TIME: 2025-02-25 12:31:59 -- STEP: 5202/36938 -- GLOBAL_STEP: 780900 | > loss_text_ce: 0.03452726826071739 (0.03415701361799155) | > loss_mel_ce: 3.0456032752990723 (nan) | > loss: 0.012222740799188614 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13591748968723508) | > loader_time: 0.0066 (0.004660985736560939)  --> TIME: 2025-02-25 12:32:08 -- STEP: 5252/36938 -- GLOBAL_STEP: 780950 | > loss_text_ce: 0.030012432485818863 (0.034162850932041466) | > loss_mel_ce: 0.8688877820968628 (nan) | > loss: 0.0035670646466314793 (nan) | > current_lr: 5e-06 | > step_time: 0.1494 (0.13591792868551003) | > loader_time: 0.0046 (0.004660923019766545)  --> TIME: 2025-02-25 12:32:18 -- STEP: 5302/36938 -- GLOBAL_STEP: 781000 | > loss_text_ce: 0.03240591660141945 (0.034155518989567306) | > loss_mel_ce: 2.971076488494873 (nan) | > loss: 0.01191858109086752 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13592105413643393) | > loader_time: 0.0048 (0.0046615883432573274)  --> TIME: 2025-02-25 12:32:27 -- STEP: 5352/36938 -- GLOBAL_STEP: 781050 | > loss_text_ce: 0.037803273648023605 (0.034143228486517165) | > loss_mel_ce: 3.146573305130005 (nan) | > loss: 0.012636415660381317 (nan) | > current_lr: 5e-06 | > step_time: 0.1844 (0.13595266103388143) | > loader_time: 0.0048 (0.004661514263694781)  --> TIME: 2025-02-25 12:32:37 -- STEP: 5402/36938 -- GLOBAL_STEP: 781100 | > loss_text_ce: 0.03269946947693825 (0.03415002261406729) | > loss_mel_ce: 2.3525431156158447 (nan) | > loss: 0.009465249255299568 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13598449774823) | > loader_time: 0.0045 (0.0046616395903004435)  --> TIME: 2025-02-25 12:32:46 -- STEP: 5452/36938 -- GLOBAL_STEP: 781150 | > loss_text_ce: 0.045257557183504105 (0.034156899855513775) | > loss_mel_ce: 2.8389341831207275 (nan) | > loss: 0.01144520565867424 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.13598640128609396) | > loader_time: 0.0047 (0.004661047887207393)  --> TIME: 2025-02-25 12:32:55 -- STEP: 5502/36938 -- GLOBAL_STEP: 781200 | > loss_text_ce: 0.03435598313808441 (0.03415601924420625) | > loss_mel_ce: 3.0012378692626953 (nan) | > loss: 0.012046007439494133 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1359730302442513) | > loader_time: 0.0045 (0.004660811869719388)  --> TIME: 2025-02-25 12:33:05 -- STEP: 5552/36938 -- GLOBAL_STEP: 781250 | > loss_text_ce: 0.0341903455555439 (0.034161708483094) | > loss_mel_ce: 2.72314453125 (nan) | > loss: 0.010941806249320507 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13598636239337675) | > loader_time: 0.0045 (0.004660884353200715)  --> TIME: 2025-02-25 12:33:14 -- STEP: 5602/36938 -- GLOBAL_STEP: 781300 | > loss_text_ce: 0.02590709924697876 (0.03414830899478511) | > loss_mel_ce: 1.4048011302947998 (nan) | > loss: 0.005677413661032915 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.1360116035586725) | > loader_time: 0.0045 (0.004660708186712411)  --> TIME: 2025-02-25 12:33:23 -- STEP: 5652/36938 -- GLOBAL_STEP: 781350 | > loss_text_ce: 0.03219335898756981 (0.03413916374896155) | > loss_mel_ce: 3.2708921432495117 (nan) | > loss: 0.0131074832752347 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13596054221920695) | > loader_time: 0.0067 (0.004660827971820877)  --> TIME: 2025-02-25 12:33:32 -- STEP: 5702/36938 -- GLOBAL_STEP: 781400 | > loss_text_ce: 0.013665920123457909 (0.0341309810368267) | > loss_mel_ce: 1.9914164543151855 (nan) | > loss: 0.007956676185131073 (nan) | > current_lr: 5e-06 | > step_time: 0.1081 (0.13593959992327517) | > loader_time: 0.0044 (0.004660404928522508)  --> TIME: 2025-02-25 12:33:41 -- STEP: 5752/36938 -- GLOBAL_STEP: 781450 | > loss_text_ce: 0.036122262477874756 (0.03412071719567478) | > loss_mel_ce: 3.1080079078674316 (nan) | > loss: 0.012476707808673382 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13589780930020434) | > loader_time: 0.0046 (0.004659769225021072)  --> TIME: 2025-02-25 12:33:51 -- STEP: 5802/36938 -- GLOBAL_STEP: 781500 | > loss_text_ce: 0.03029637783765793 (0.034130845674124465) | > loss_mel_ce: 2.8685429096221924 (nan) | > loss: 0.011503331363201141 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13594700857837758) | > loader_time: 0.0045 (0.004659482299439136)  --> TIME: 2025-02-25 12:34:00 -- STEP: 5852/36938 -- GLOBAL_STEP: 781550 | > loss_text_ce: 0.032952483743429184 (0.03412970306316129) | > loss_mel_ce: 1.6639516353607178 (nan) | > loss: 0.006733746733516455 (nan) | > current_lr: 5e-06 | > step_time: 0.1699 (0.1359591088063577) | > loader_time: 0.0048 (0.004659163080008172)  --> TIME: 2025-02-25 12:34:10 -- STEP: 5902/36938 -- GLOBAL_STEP: 781600 | > loss_text_ce: 0.028254229575395584 (0.03413302748079335) | > loss_mel_ce: 2.8334577083587646 (nan) | > loss: 0.011356000788509846 (nan) | > current_lr: 5e-06 | > step_time: 0.188 (0.13597808749503712) | > loader_time: 0.0048 (0.004658773566698159)  --> TIME: 2025-02-25 12:34:19 -- STEP: 5952/36938 -- GLOBAL_STEP: 781650 | > loss_text_ce: 0.031864844262599945 (0.034129013667435135) | > loss_mel_ce: 2.9608514308929443 (nan) | > loss: 0.011875859461724758 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13598962088105548) | > loader_time: 0.0045 (0.004658318895806564)  --> TIME: 2025-02-25 12:34:29 -- STEP: 6002/36938 -- GLOBAL_STEP: 781700 | > loss_text_ce: 0.03378162905573845 (0.03413432102041521) | > loss_mel_ce: 3.308239698410034 (nan) | > loss: 0.013261989690363407 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13602471065616617) | > loader_time: 0.0046 (0.0046586499377832035)  --> TIME: 2025-02-25 12:34:38 -- STEP: 6052/36938 -- GLOBAL_STEP: 781750 | > loss_text_ce: 0.03500477597117424 (0.03413964262443318) | > loss_mel_ce: 3.22244930267334 (nan) | > loss: 0.012926406227052212 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.1360321298625721) | > loader_time: 0.0048 (0.004658167990957451)  --> TIME: 2025-02-25 12:34:47 -- STEP: 6102/36938 -- GLOBAL_STEP: 781800 | > loss_text_ce: 0.03789537027478218 (0.03413580399588416) | > loss_mel_ce: 2.7921462059020996 (nan) | > loss: 0.011230324395000935 (nan) | > current_lr: 5e-06 | > step_time: 0.1953 (0.1360397430296144) | > loader_time: 0.0048 (0.0046576880814794015)  --> TIME: 2025-02-25 12:34:57 -- STEP: 6152/36938 -- GLOBAL_STEP: 781850 | > loss_text_ce: 0.025662412866950035 (0.034137095429302364) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13605114383567318) | > loader_time: 0.0046 (0.00465716694824408)  --> TIME: 2025-02-25 12:35:06 -- STEP: 6202/36938 -- GLOBAL_STEP: 781900 | > loss_text_ce: 0.030941670760512352 (0.03414268347367722) | > loss_mel_ce: 3.120948553085327 (nan) | > loss: 0.012507501989603043 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.136013211107608) | > loader_time: 0.0048 (0.004656550615920219)  --> TIME: 2025-02-25 12:35:15 -- STEP: 6252/36938 -- GLOBAL_STEP: 781950 | > loss_text_ce: 0.03929247707128525 (0.034136957489847804) | > loss_mel_ce: 2.796090841293335 (nan) | > loss: 0.011251522228121758 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13603442072181962) | > loader_time: 0.0045 (0.004656623398273795)  --> TIME: 2025-02-25 12:35:25 -- STEP: 6302/36938 -- GLOBAL_STEP: 782000 | > loss_text_ce: 0.04027598351240158 (0.03413698877796399) | > loss_mel_ce: 2.4524898529052734 (nan) | > loss: 0.009891929104924202 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13604206321580747) | > loader_time: 0.0045 (0.004656459860709398)  --> TIME: 2025-02-25 12:35:34 -- STEP: 6352/36938 -- GLOBAL_STEP: 782050 | > loss_text_ce: 0.036947377026081085 (0.034142314520915464) | > loss_mel_ce: 6.169034004211426 (nan) | > loss: 0.02462691068649292 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13600644210273763) | > loader_time: 0.0048 (0.004656460633506114)  --> TIME: 2025-02-25 12:35:42 -- STEP: 6402/36938 -- GLOBAL_STEP: 782100 | > loss_text_ce: 0.02486378885805607 (0.034143050423666436) | > loss_mel_ce: 3.0839271545410156 (nan) | > loss: 0.012336472049355507 (nan) | > current_lr: 5e-06 | > step_time: 0.1471 (0.1359526617979121) | > loader_time: 0.0046 (0.004657246067687081)  --> TIME: 2025-02-25 12:35:52 -- STEP: 6452/36938 -- GLOBAL_STEP: 782150 | > loss_text_ce: 0.0267158355563879 (0.034143298889047796) | > loss_mel_ce: 0.01531982421875 (nan) | > loss: 0.00016680819680914283 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.1359980964025063) | > loader_time: 0.0045 (0.004657824366120772)  --> TIME: 2025-02-25 12:36:01 -- STEP: 6502/36938 -- GLOBAL_STEP: 782200 | > loss_text_ce: 0.036595042794942856 (0.03414363537160762) | > loss_mel_ce: 3.5810630321502686 (nan) | > loss: 0.014355787076056004 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13599367685884206) | > loader_time: 0.0046 (0.004657688231807023)  --> TIME: 2025-02-25 12:36:11 -- STEP: 6552/36938 -- GLOBAL_STEP: 782250 | > loss_text_ce: 0.0374177023768425 (0.03415076600081225) | > loss_mel_ce: 2.9945614337921143 (nan) | > loss: 0.01203166414052248 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.136016792830385) | > loader_time: 0.0048 (0.004657653079655879)  --> TIME: 2025-02-25 12:36:20 -- STEP: 6602/36938 -- GLOBAL_STEP: 782300 | > loss_text_ce: 0.02670948952436447 (0.034151401590122504) | > loss_mel_ce: 3.7927989959716797 (nan) | > loss: 0.015156781300902367 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1359915576245493) | > loader_time: 0.0046 (0.00465750795391538)  --> TIME: 2025-02-25 12:36:29 -- STEP: 6652/36938 -- GLOBAL_STEP: 782350 | > loss_text_ce: 0.03221028670668602 (0.0341383660131235) | > loss_mel_ce: 2.649665355682373 (nan) | > loss: 0.010642364621162415 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13598278231170677) | > loader_time: 0.0046 (0.004657921881397595)  --> TIME: 2025-02-25 12:36:38 -- STEP: 6702/36938 -- GLOBAL_STEP: 782400 | > loss_text_ce: 0.03777386248111725 (0.03414388690924783) | > loss_mel_ce: 2.923828125 (nan) | > loss: 0.011752389371395111 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13596416754140647) | > loader_time: 0.0045 (0.004657957170373186)  --> TIME: 2025-02-25 12:36:48 -- STEP: 6752/36938 -- GLOBAL_STEP: 782450 | > loss_text_ce: 0.02879018522799015 (0.03414601295834986) | > loss_mel_ce: 1.2523388862609863 (nan) | > loss: 0.005083845928311348 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13602557284007183) | > loader_time: 0.0046 (0.004658122551384702)  --> TIME: 2025-02-25 12:36:58 -- STEP: 6802/36938 -- GLOBAL_STEP: 782500 | > loss_text_ce: 0.03321385756134987 (0.03413930054854237) | > loss_mel_ce: 5.794270992279053 (nan) | > loss: 0.023124942556023598 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13604079057664925) | > loader_time: 0.0046 (0.004658169201001537)  --> TIME: 2025-02-25 12:37:07 -- STEP: 6852/36938 -- GLOBAL_STEP: 782550 | > loss_text_ce: 0.033002886921167374 (0.03413949306563329) | > loss_mel_ce: 2.9979746341705322 (nan) | > loss: 0.012027689255774021 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13598629896639436) | > loader_time: 0.0046 (0.004657929707742623)  --> TIME: 2025-02-25 12:37:16 -- STEP: 6902/36938 -- GLOBAL_STEP: 782600 | > loss_text_ce: 0.03534354269504547 (0.03414276770702779) | > loss_mel_ce: 3.436016321182251 (nan) | > loss: 0.013775238767266273 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.1359936303244027) | > loader_time: 0.0047 (0.004658082366784192)  --> TIME: 2025-02-25 12:37:25 -- STEP: 6952/36938 -- GLOBAL_STEP: 782650 | > loss_text_ce: 0.04384174197912216 (0.03414642073546815) | > loss_mel_ce: 1.9997851848602295 (nan) | > loss: 0.008109631948173046 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13597935709086392) | > loader_time: 0.0044 (0.004657845056784037)  --> TIME: 2025-02-25 12:37:35 -- STEP: 7002/36938 -- GLOBAL_STEP: 782700 | > loss_text_ce: 0.030488742515444756 (0.034147680398306506) | > loss_mel_ce: 2.332993745803833 (nan) | > loss: 0.00937889888882637 (nan) | > current_lr: 5e-06 | > step_time: 0.1872 (0.13601935522721847) | > loader_time: 0.0049 (0.004658203743349251)  --> TIME: 2025-02-25 12:37:44 -- STEP: 7052/36938 -- GLOBAL_STEP: 782750 | > loss_text_ce: 0.04025780409574509 (0.034145806295220266) | > loss_mel_ce: 2.8528435230255127 (nan) | > loss: 0.011480561457574368 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.1360031512401627) | > loader_time: 0.0046 (0.004658344146276299)  --> TIME: 2025-02-25 12:37:53 -- STEP: 7102/36938 -- GLOBAL_STEP: 782800 | > loss_text_ce: 0.030808553099632263 (0.03414683103688855) | > loss_mel_ce: 3.3897595405578613 (nan) | > loss: 0.013573682866990566 (nan) | > current_lr: 5e-06 | > step_time: 0.1702 (0.1360267932164577) | > loader_time: 0.0048 (0.004658754259322746)  --> TIME: 2025-02-25 12:38:02 -- STEP: 7152/36938 -- GLOBAL_STEP: 782850 | > loss_text_ce: 0.03560436889529228 (0.03414537845623733) | > loss_mel_ce: 2.447279453277588 (nan) | > loss: 0.009852714836597443 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13600935558611435) | > loader_time: 0.0045 (0.004658470084469865)  --> TIME: 2025-02-25 12:38:11 -- STEP: 7202/36938 -- GLOBAL_STEP: 782900 | > loss_text_ce: 0.03246963769197464 (0.03414512068714439) | > loss_mel_ce: 3.2188706398010254 (nan) | > loss: 0.012902145273983479 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.13597088252726805) | > loader_time: 0.0048 (0.004658463662679056)  --> TIME: 2025-02-25 12:38:21 -- STEP: 7252/36938 -- GLOBAL_STEP: 782950 | > loss_text_ce: 0.03410559520125389 (0.03414085247580587) | > loss_mel_ce: 2.542588710784912 (nan) | > loss: 0.010224977508187294 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13595450687198096) | > loader_time: 0.005 (0.004658354722657924)  --> TIME: 2025-02-25 12:38:30 -- STEP: 7302/36938 -- GLOBAL_STEP: 783000 | > loss_text_ce: 0.035234805196523666 (0.034139221887555846) | > loss_mel_ce: 0.015869140625 (nan) | > loss: 0.00020279345335438848 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13593521910933637) | > loader_time: 0.0046 (0.004658384833783858)  --> TIME: 2025-02-25 12:38:39 -- STEP: 7352/36938 -- GLOBAL_STEP: 783050 | > loss_text_ce: 0.03105984441936016 (0.03413150836452649) | > loss_mel_ce: 3.0628581047058105 (nan) | > loss: 0.012277452275156975 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13593200984146483) | > loader_time: 0.0048 (0.004658338975595061)  --> TIME: 2025-02-25 12:38:48 -- STEP: 7402/36938 -- GLOBAL_STEP: 783100 | > loss_text_ce: 0.02696812152862549 (0.0341321481365364) | > loss_mel_ce: 2.7104218006134033 (nan) | > loss: 0.01086265966296196 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13591972520885487) | > loader_time: 0.0046 (0.004658166571779215)  --> TIME: 2025-02-25 12:38:57 -- STEP: 7452/36938 -- GLOBAL_STEP: 783150 | > loss_text_ce: 0.030412882566452026 (0.03412743452054067) | > loss_mel_ce: 2.6536941528320312 (nan) | > loss: 0.010651218704879284 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13591250855629664) | > loader_time: 0.0045 (0.004657882103277553)  --> TIME: 2025-02-25 12:39:07 -- STEP: 7502/36938 -- GLOBAL_STEP: 783200 | > loss_text_ce: 0.04128972813487053 (0.03412591405195485) | > loss_mel_ce: 3.322497844696045 (nan) | > loss: 0.013348364271223545 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13590330732437647) | > loader_time: 0.0047 (0.004658407511567479)  --> TIME: 2025-02-25 12:39:16 -- STEP: 7552/36938 -- GLOBAL_STEP: 783250 | > loss_text_ce: 0.044492706656455994 (0.034134918585211345) | > loss_mel_ce: 3.396282196044922 (nan) | > loss: 0.013653869740664959 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13587328422246367) | > loader_time: 0.0046 (0.004658304060162132)  --> TIME: 2025-02-25 12:39:25 -- STEP: 7602/36938 -- GLOBAL_STEP: 783300 | > loss_text_ce: 0.03478856384754181 (0.03413469612981127) | > loss_mel_ce: 3.577389717102051 (nan) | > loss: 0.014334041625261307 (nan) | > current_lr: 5e-06 | > step_time: 0.1775 (0.13586616356163983) | > loader_time: 0.0048 (0.00465822865918448)  --> TIME: 2025-02-25 12:39:34 -- STEP: 7652/36938 -- GLOBAL_STEP: 783350 | > loss_text_ce: 0.04282805323600769 (0.03413533997272604) | > loss_mel_ce: 0.1075846329331398 (nan) | > loss: 0.0005968757322989404 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13587973657479438) | > loader_time: 0.0045 (0.004658192287112063)  --> TIME: 2025-02-25 12:39:44 -- STEP: 7702/36938 -- GLOBAL_STEP: 783400 | > loss_text_ce: 0.033684227615594864 (0.03413601522799446) | > loss_mel_ce: 2.9692513942718506 (nan) | > loss: 0.01191641204059124 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.1359012261888204) | > loader_time: 0.0045 (0.004658044885889465)  --> TIME: 2025-02-25 12:39:53 -- STEP: 7752/36938 -- GLOBAL_STEP: 783450 | > loss_text_ce: 0.033714670687913895 (0.03414023511584592) | > loss_mel_ce: 2.6729164123535156 (nan) | > loss: 0.010740600526332855 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13586784888470282) | > loader_time: 0.0046 (0.004657602224064569)  --> TIME: 2025-02-25 12:40:02 -- STEP: 7802/36938 -- GLOBAL_STEP: 783500 | > loss_text_ce: 0.03319934010505676 (0.034137796981356015) | > loss_mel_ce: 3.4793970584869385 (nan) | > loss: 0.013938874937593937 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13583842163604234) | > loader_time: 0.0047 (0.0046573051945486684)  --> TIME: 2025-02-25 12:40:11 -- STEP: 7852/36938 -- GLOBAL_STEP: 783550 | > loss_text_ce: 0.038197584450244904 (0.0341378321442505) | > loss_mel_ce: 3.278273344039917 (nan) | > loss: 0.013160599395632744 (nan) | > current_lr: 5e-06 | > step_time: 0.1769 (0.13583215820625394) | > loader_time: 0.0048 (0.004657033658185549)  --> TIME: 2025-02-25 12:40:20 -- STEP: 7902/36938 -- GLOBAL_STEP: 783600 | > loss_text_ce: 0.03487086296081543 (0.034137306080447856) | > loss_mel_ce: 2.599795341491699 (nan) | > loss: 0.010455025359988213 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1358388737466361) | > loader_time: 0.0045 (0.004656715864955375)  --> TIME: 2025-02-25 12:40:30 -- STEP: 7952/36938 -- GLOBAL_STEP: 783650 | > loss_text_ce: 0.035360950976610184 (0.03414525814160927) | > loss_mel_ce: 2.9006547927856445 (nan) | > loss: 0.011650857515633106 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.1358258763189529) | > loader_time: 0.0046 (0.004656489466277653)  --> TIME: 2025-02-25 12:40:39 -- STEP: 8002/36938 -- GLOBAL_STEP: 783700 | > loss_text_ce: 0.03560337796807289 (0.03413944538918717) | > loss_mel_ce: 2.9430935382843018 (nan) | > loss: 0.011820225976407528 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.1358072473179906) | > loader_time: 0.0048 (0.004656311304263556)  --> TIME: 2025-02-25 12:40:48 -- STEP: 8052/36938 -- GLOBAL_STEP: 783750 | > loss_text_ce: 0.03171085566282272 (0.03414176799316673) | > loss_mel_ce: 2.3149642944335938 (nan) | > loss: 0.009312203153967857 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.1357998771271895) | > loader_time: 0.0044 (0.004655986417305163)  --> TIME: 2025-02-25 12:40:57 -- STEP: 8102/36938 -- GLOBAL_STEP: 783800 | > loss_text_ce: 0.03535855561494827 (0.034135579502716125) | > loss_mel_ce: 2.7713894844055176 (nan) | > loss: 0.011137889698147774 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.1358069576354301) | > loader_time: 0.0047 (0.00465563911474714)  --> TIME: 2025-02-25 12:41:07 -- STEP: 8152/36938 -- GLOBAL_STEP: 783850 | > loss_text_ce: 0.03669018670916557 (0.03413493380051502) | > loss_mel_ce: 2.935905933380127 (nan) | > loss: 0.011796017177402973 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13580444654023938) | > loader_time: 0.0046 (0.004655303325690519)  --> TIME: 2025-02-25 12:41:15 -- STEP: 8202/36938 -- GLOBAL_STEP: 783900 | > loss_text_ce: 0.038345854729413986 (0.034139575505180764) | > loss_mel_ce: 0.17780303955078125 (nan) | > loss: 0.000857733772136271 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.1357669328021238) | > loader_time: 0.0046 (0.004655001745430973)  --> TIME: 2025-02-25 12:41:25 -- STEP: 8252/36938 -- GLOBAL_STEP: 783950 | > loss_text_ce: 0.02683345042169094 (0.034140252521427564) | > loss_mel_ce: 2.9260380268096924 (nan) | > loss: 0.01171774510294199 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13575406847101845) | > loader_time: 0.0046 (0.004654594289407076)  --> TIME: 2025-02-25 12:41:34 -- STEP: 8302/36938 -- GLOBAL_STEP: 784000 | > loss_text_ce: 0.03488703817129135 (0.03412980902874956) | > loss_mel_ce: 3.268573522567749 (nan) | > loss: 0.013108971528708935 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.13573674993668963) | > loader_time: 0.0045 (0.004654199466507748)  --> TIME: 2025-02-25 12:41:43 -- STEP: 8352/36938 -- GLOBAL_STEP: 784050 | > loss_text_ce: 0.03511142358183861 (0.03411999409958097) | > loss_mel_ce: 3.3000876903533936 (nan) | > loss: 0.013234918005764484 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.135719680609146) | > loader_time: 0.0046 (0.0046540244386113995)  --> TIME: 2025-02-25 12:41:52 -- STEP: 8402/36938 -- GLOBAL_STEP: 784100 | > loss_text_ce: 0.03024115413427353 (0.03412779766329771) | > loss_mel_ce: 2.703253746032715 (nan) | > loss: 0.010847203433513641 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13572116969057973) | > loader_time: 0.0045 (0.004653855608463635)  --> TIME: 2025-02-25 12:42:01 -- STEP: 8452/36938 -- GLOBAL_STEP: 784150 | > loss_text_ce: 0.03458232432603836 (0.03411809846962018) | > loss_mel_ce: 1.8122153282165527 (nan) | > loss: 0.007328562904149294 (nan) | > current_lr: 5e-06 | > step_time: 0.1741 (0.1357091618723855) | > loader_time: 0.0048 (0.004653705164994671)  --> TIME: 2025-02-25 12:42:11 -- STEP: 8502/36938 -- GLOBAL_STEP: 784200 | > loss_text_ce: 0.03902358561754227 (0.03412318208917149) | > loss_mel_ce: 3.130007028579712 (nan) | > loss: 0.012575519271194935 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13572529951731382) | > loader_time: 0.0047 (0.004653599115854939)  --> TIME: 2025-02-25 12:42:20 -- STEP: 8552/36938 -- GLOBAL_STEP: 784250 | > loss_text_ce: 0.03321893513202667 (0.03411706690983598) | > loss_mel_ce: 0.0753173828125 (nan) | > loss: 0.0004306996997911483 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13575818855925312) | > loader_time: 0.0045 (0.004653531998758347)  --> TIME: 2025-02-25 12:42:30 -- STEP: 8602/36938 -- GLOBAL_STEP: 784300 | > loss_text_ce: 0.03112003579735756 (0.034115421137400126) | > loss_mel_ce: 2.6600735187530518 (nan) | > loss: 0.010679339990019798 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.13578357100071017) | > loader_time: 0.0047 (0.004653635121478342)  --> TIME: 2025-02-25 12:42:39 -- STEP: 8652/36938 -- GLOBAL_STEP: 784350 | > loss_text_ce: 0.04555508866906166 (0.03411973634499213) | > loss_mel_ce: 3.271256446838379 (nan) | > loss: 0.013161951676011086 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.13581872388713634) | > loader_time: 0.0048 (0.004653776761177127)  --> TIME: 2025-02-25 12:42:49 -- STEP: 8702/36938 -- GLOBAL_STEP: 784400 | > loss_text_ce: 0.04115656390786171 (0.034113049284638125) | > loss_mel_ce: 3.0494771003723145 (nan) | > loss: 0.01226442027837038 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13582829970432367) | > loader_time: 0.0045 (0.004653847099747833)  --> TIME: 2025-02-25 12:42:58 -- STEP: 8752/36938 -- GLOBAL_STEP: 784450 | > loss_text_ce: 0.03305411338806152 (0.03411656511290717) | > loss_mel_ce: 3.090064525604248 (nan) | > loss: 0.012393328361213207 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13584240366597508) | > loader_time: 0.0045 (0.0046536586020938684)  --> TIME: 2025-02-25 12:43:08 -- STEP: 8802/36938 -- GLOBAL_STEP: 784500 | > loss_text_ce: 0.027207080274820328 (0.034112691549680156) | > loss_mel_ce: 2.6155495643615723 (nan) | > loss: 0.010487129911780357 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.13585318976763955) | > loader_time: 0.0047 (0.004653437791480665)  --> TIME: 2025-02-25 12:43:17 -- STEP: 8852/36938 -- GLOBAL_STEP: 784550 | > loss_text_ce: 0.03606804087758064 (0.03410988089723609) | > loss_mel_ce: 2.8303327560424805 (nan) | > loss: 0.011374606750905514 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.1358514720640516) | > loader_time: 0.0045 (0.004653153110293853)  --> TIME: 2025-02-25 12:43:26 -- STEP: 8902/36938 -- GLOBAL_STEP: 784600 | > loss_text_ce: 0.03430713713169098 (0.03411296228829008) | > loss_mel_ce: 2.822589159011841 (nan) | > loss: 0.01133689098060131 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13586013656175705) | > loader_time: 0.0046 (0.004653011057195278)  --> TIME: 2025-02-25 12:43:35 -- STEP: 8952/36938 -- GLOBAL_STEP: 784650 | > loss_text_ce: 0.038111425936222076 (0.034113147441348336) | > loss_mel_ce: 2.471407890319824 (nan) | > loss: 0.009958410635590553 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1358432581213781) | > loader_time: 0.0044 (0.004652570037569076)  --> TIME: 2025-02-25 12:43:45 -- STEP: 9002/36938 -- GLOBAL_STEP: 784700 | > loss_text_ce: 0.02267475239932537 (0.03411991823676906) | > loss_mel_ce: 2.587679386138916 (nan) | > loss: 0.01035854872316122 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13586966378560114) | > loader_time: 0.0045 (0.004652592373593497)  --> TIME: 2025-02-25 12:43:54 -- STEP: 9052/36938 -- GLOBAL_STEP: 784750 | > loss_text_ce: 0.04193143919110298 (0.03412664014859648) | > loss_mel_ce: 3.191453695297241 (nan) | > loss: 0.012830893509089947 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.1358449343712102) | > loader_time: 0.0045 (0.004652417343483532)  --> TIME: 2025-02-25 12:44:03 -- STEP: 9102/36938 -- GLOBAL_STEP: 784800 | > loss_text_ce: 0.035207632929086685 (0.03412708484376753) | > loss_mel_ce: 2.880950689315796 (nan) | > loss: 0.01157205831259489 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13584915351616236) | > loader_time: 0.0045 (0.004652122119525537)  --> TIME: 2025-02-25 12:44:13 -- STEP: 9152/36938 -- GLOBAL_STEP: 784850 | > loss_text_ce: 0.03216772526502609 (0.03412647744190088) | > loss_mel_ce: 3.109757661819458 (nan) | > loss: 0.01246795803308487 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13587749048650702) | > loader_time: 0.0047 (0.0046522223449253575)  --> TIME: 2025-02-25 12:44:22 -- STEP: 9202/36938 -- GLOBAL_STEP: 784900 | > loss_text_ce: 0.040658608078956604 (0.03412148708006369) | > loss_mel_ce: 2.81748104095459 (nan) | > loss: 0.011341825127601624 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.13589592400749637) | > loader_time: 0.0048 (0.004652358479822132)  --> TIME: 2025-02-25 12:44:32 -- STEP: 9252/36938 -- GLOBAL_STEP: 784950 | > loss_text_ce: 0.03442972153425217 (0.034120536786900496) | > loss_mel_ce: 3.688354969024658 (nan) | > loss: 0.014772956259548664 (nan) | > current_lr: 5e-06 | > step_time: 0.1871 (0.13589645434497927) | > loader_time: 0.0049 (0.004652677291510585)  --> TIME: 2025-02-25 12:44:40 -- STEP: 9302/36938 -- GLOBAL_STEP: 785000 | > loss_text_ce: 0.03460855036973953 (0.034115362539066246) | > loss_mel_ce: 2.99576473236084 (nan) | > loss: 0.012025292031466961 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13586633309475263) | > loader_time: 0.0046 (0.004652664318466109)  --> TIME: 2025-02-25 12:44:50 -- STEP: 9352/36938 -- GLOBAL_STEP: 785050 | > loss_text_ce: 0.028996089473366737 (0.034118903065008484) | > loss_mel_ce: 1.7863363027572632 (nan) | > loss: 0.007203700486570597 (nan) | > current_lr: 5e-06 | > step_time: 0.1629 (0.13588953683578234) | > loader_time: 0.0047 (0.0046533763510636355)  --> TIME: 2025-02-25 12:44:59 -- STEP: 9402/36938 -- GLOBAL_STEP: 785100 | > loss_text_ce: 0.03595619276165962 (0.03411792456577522) | > loss_mel_ce: 2.388486385345459 (nan) | > loss: 0.009620804339647293 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1358766619942591) | > loader_time: 0.0046 (0.004653336392694576)  --> TIME: 2025-02-25 12:45:08 -- STEP: 9452/36938 -- GLOBAL_STEP: 785150 | > loss_text_ce: 0.03907465934753418 (0.03412225559078702) | > loss_mel_ce: 2.103178024291992 (nan) | > loss: 0.008501003496348858 (nan) | > current_lr: 5e-06 | > step_time: 0.1811 (0.135863382433917) | > loader_time: 0.0049 (0.00465327952809118)  --> TIME: 2025-02-25 12:45:18 -- STEP: 9502/36938 -- GLOBAL_STEP: 785200 | > loss_text_ce: 0.025856276974081993 (0.03412285535803094) | > loss_mel_ce: 0.0698627308011055 (nan) | > loss: 0.0003798373509198427 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13586593075717962) | > loader_time: 0.0046 (0.004653423315749571)  --> TIME: 2025-02-25 12:45:27 -- STEP: 9552/36938 -- GLOBAL_STEP: 785250 | > loss_text_ce: 0.0343853123486042 (0.03412799695873267) | > loss_mel_ce: 2.8093271255493164 (nan) | > loss: 0.011284573003649712 (nan) | > current_lr: 5e-06 | > step_time: 0.1791 (0.13587498210582732) | > loader_time: 0.0048 (0.004653572037791087)  --> TIME: 2025-02-25 12:45:36 -- STEP: 9602/36938 -- GLOBAL_STEP: 785300 | > loss_text_ce: 0.03416221961379051 (0.0341233188849754) | > loss_mel_ce: 3.2142512798309326 (nan) | > loss: 0.012890530750155449 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13583831450909914) | > loader_time: 0.0046 (0.004653701531939198)  --> TIME: 2025-02-25 12:45:45 -- STEP: 9652/36938 -- GLOBAL_STEP: 785350 | > loss_text_ce: 0.030743157491087914 (0.034123404618658086) | > loss_mel_ce: 2.550004243850708 (nan) | > loss: 0.010241061449050903 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13584844531530796) | > loader_time: 0.0046 (0.004653796757398875)  --> TIME: 2025-02-25 12:45:54 -- STEP: 9702/36938 -- GLOBAL_STEP: 785400 | > loss_text_ce: 0.04763247072696686 (0.034118726950540176) | > loss_mel_ce: 2.5796971321105957 (nan) | > loss: 0.010425911284983158 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13584747168974085) | > loader_time: 0.0046 (0.004654004091676841)  --> TIME: 2025-02-25 12:46:04 -- STEP: 9752/36938 -- GLOBAL_STEP: 785450 | > loss_text_ce: 0.03781622275710106 (0.03412012972982026) | > loss_mel_ce: 2.93768572807312 (nan) | > loss: 0.011807548813521862 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13586845864420388) | > loader_time: 0.0047 (0.004654225044508662)  --> TIME: 2025-02-25 12:46:13 -- STEP: 9802/36938 -- GLOBAL_STEP: 785500 | > loss_text_ce: 0.02969302609562874 (0.034118482841901006) | > loss_mel_ce: 3.0998973846435547 (nan) | > loss: 0.012419010512530804 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13582909199637438) | > loader_time: 0.0046 (0.00465398794581661)  --> TIME: 2025-02-25 12:46:22 -- STEP: 9852/36938 -- GLOBAL_STEP: 785550 | > loss_text_ce: 0.03177661448717117 (0.034124282858940376) | > loss_mel_ce: 2.9567058086395264 (nan) | > loss: 0.011859058402478695 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13584234142903656) | > loader_time: 0.0045 (0.0046541183594035)  --> TIME: 2025-02-25 12:46:32 -- STEP: 9902/36938 -- GLOBAL_STEP: 785600 | > loss_text_ce: 0.02365349978208542 (0.03411691143455635) | > loss_mel_ce: 3.309027671813965 (nan) | > loss: 0.013224925845861435 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13585021940777256) | > loader_time: 0.0046 (0.004654586592049966)  --> TIME: 2025-02-25 12:46:41 -- STEP: 9952/36938 -- GLOBAL_STEP: 785650 | > loss_text_ce: 0.035161860287189484 (0.034116169554555466) | > loss_mel_ce: 0.8859651684761047 (nan) | > loss: 0.003655266249552369 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13584597617196165) | > loader_time: 0.0045 (0.0046542753552317245)  --> TIME: 2025-02-25 12:46:50 -- STEP: 10002/36938 -- GLOBAL_STEP: 785700 | > loss_text_ce: 0.023936426267027855 (0.03411483892102355) | > loss_mel_ce: 2.4363296031951904 (nan) | > loss: 0.0097629614174366 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1358412983560061) | > loader_time: 0.0047 (0.004654288649487508)  --> TIME: 2025-02-25 12:46:59 -- STEP: 10052/36938 -- GLOBAL_STEP: 785750 | > loss_text_ce: 0.033074405044317245 (0.03411577933423798) | > loss_mel_ce: 2.6062495708465576 (nan) | > loss: 0.010473508387804031 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.13585186678244718) | > loader_time: 0.0047 (0.00465418329003604)  --> TIME: 2025-02-25 12:47:09 -- STEP: 10102/36938 -- GLOBAL_STEP: 785800 | > loss_text_ce: 0.03538866341114044 (0.03411279563314288) | > loss_mel_ce: 1.7231580018997192 (nan) | > loss: 0.006978360470384359 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1358342890313679) | > loader_time: 0.0046 (0.00465412669501855)  --> TIME: 2025-02-25 12:47:18 -- STEP: 10152/36938 -- GLOBAL_STEP: 785850 | > loss_text_ce: 0.029442638158798218 (0.03411111875746884) | > loss_mel_ce: 2.5573368072509766 (nan) | > loss: 0.01026499830186367 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13583652756174724) | > loader_time: 0.0046 (0.004654125001507542)  --> TIME: 2025-02-25 12:47:27 -- STEP: 10202/36938 -- GLOBAL_STEP: 785900 | > loss_text_ce: 0.030226273462176323 (0.03411211882139298) | > loss_mel_ce: 2.705986738204956 (nan) | > loss: 0.010857988148927689 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13584469711282798) | > loader_time: 0.0045 (0.0046538869391887965)  --> TIME: 2025-02-25 12:47:36 -- STEP: 10252/36938 -- GLOBAL_STEP: 785950 | > loss_text_ce: 0.03971468284726143 (0.03411440515743166) | > loss_mel_ce: 3.1179986000061035 (nan) | > loss: 0.012530608102679253 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13584828953514166) | > loader_time: 0.0046 (0.0046537911757053504)  --> TIME: 2025-02-25 12:47:46 -- STEP: 10302/36938 -- GLOBAL_STEP: 786000 | > loss_text_ce: 0.028903638944029808 (0.0341143223423226) | > loss_mel_ce: 2.7825872898101807 (nan) | > loss: 0.011156710796058178 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.1358690228977846) | > loader_time: 0.0045 (0.004653865840684506)  --> TIME: 2025-02-25 12:47:55 -- STEP: 10352/36938 -- GLOBAL_STEP: 786050 | > loss_text_ce: 0.03032754361629486 (0.034111192925574574) | > loss_mel_ce: 2.9432146549224854 (nan) | > loss: 0.011799771338701248 (nan) | > current_lr: 5e-06 | > step_time: 0.1959 (0.13587262243078527) | > loader_time: 0.0049 (0.004653756133924127)  --> TIME: 2025-02-25 12:48:05 -- STEP: 10402/36938 -- GLOBAL_STEP: 786100 | > loss_text_ce: 0.03270028904080391 (0.03410996391431302) | > loss_mel_ce: 1.759000539779663 (nan) | > loss: 0.007109924219548702 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13589533682718488) | > loader_time: 0.0045 (0.004653640353571931)  --> TIME: 2025-02-25 12:48:14 -- STEP: 10452/36938 -- GLOBAL_STEP: 786150 | > loss_text_ce: 0.03421230614185333 (0.034103295431228725) | > loss_mel_ce: 2.702178478240967 (nan) | > loss: 0.010858694091439247 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13590138669311436) | > loader_time: 0.0048 (0.0046535025736038306)  --> TIME: 2025-02-25 12:48:23 -- STEP: 10502/36938 -- GLOBAL_STEP: 786200 | > loss_text_ce: 0.04126719757914543 (0.03410718226049754) | > loss_mel_ce: 2.5088629722595215 (nan) | > loss: 0.010119564831256866 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13590808113151176) | > loader_time: 0.0045 (0.0046533377278171195)  --> TIME: 2025-02-25 12:48:33 -- STEP: 10552/36938 -- GLOBAL_STEP: 786250 | > loss_text_ce: 0.03622308745980263 (0.03411389199935286) | > loss_mel_ce: 3.148465871810913 (nan) | > loss: 0.01263765525072813 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.13592602702322287) | > loader_time: 0.0048 (0.004653394787125369)  --> TIME: 2025-02-25 12:48:42 -- STEP: 10602/36938 -- GLOBAL_STEP: 786300 | > loss_text_ce: 0.0384456068277359 (0.03411905547193541) | > loss_mel_ce: 2.843031406402588 (nan) | > loss: 0.011434433981776237 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.1359452128162971) | > loader_time: 0.0046 (0.0046532253930308244)  --> TIME: 2025-02-25 12:48:52 -- STEP: 10652/36938 -- GLOBAL_STEP: 786350 | > loss_text_ce: 0.03645903617143631 (0.03411902685494398) | > loss_mel_ce: 2.7899210453033447 (nan) | > loss: 0.011215793900191784 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.13594689068828267) | > loader_time: 0.0047 (0.00465296902156977)  --> TIME: 2025-02-25 12:49:01 -- STEP: 10702/36938 -- GLOBAL_STEP: 786400 | > loss_text_ce: 0.03409014269709587 (0.03411663230534655) | > loss_mel_ce: 2.2891273498535156 (nan) | > loss: 0.00921911746263504 (nan) | > current_lr: 5e-06 | > step_time: 0.1513 (0.135965540283886) | > loader_time: 0.0048 (0.00465285426054106)  --> TIME: 2025-02-25 12:49:10 -- STEP: 10752/36938 -- GLOBAL_STEP: 786450 | > loss_text_ce: 0.030118903145194054 (0.03411699614618883) | > loss_mel_ce: 2.752744436264038 (nan) | > loss: 0.011043108999729156 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13594621187075964) | > loader_time: 0.0045 (0.004652503256996474)  --> TIME: 2025-02-25 12:49:20 -- STEP: 10802/36938 -- GLOBAL_STEP: 786500 | > loss_text_ce: 0.03914465382695198 (0.0341196461857278) | > loss_mel_ce: 2.6152148246765137 (nan) | > loss: 0.010533173568546772 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13596131938008374) | > loader_time: 0.0046 (0.0046523848058470626)  --> TIME: 2025-02-25 12:49:29 -- STEP: 10852/36938 -- GLOBAL_STEP: 786550 | > loss_text_ce: 0.03204181417822838 (0.034120440356336414) | > loss_mel_ce: 2.185917854309082 (nan) | > loss: 0.00880142766982317 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.13596528315957152) | > loader_time: 0.0048 (0.004652311869578038)  --> TIME: 2025-02-25 12:49:39 -- STEP: 10902/36938 -- GLOBAL_STEP: 786600 | > loss_text_ce: 0.04043472185730934 (0.034122798319084366) | > loss_mel_ce: 2.6576550006866455 (nan) | > loss: 0.0107067059725523 (nan) | > current_lr: 5e-06 | > step_time: 0.186 (0.1359871253685608) | > loader_time: 0.0048 (0.004652339763497892)  --> TIME: 2025-02-25 12:49:48 -- STEP: 10952/36938 -- GLOBAL_STEP: 786650 | > loss_text_ce: 0.04934169724583626 (0.034122017247842334) | > loss_mel_ce: 2.3051910400390625 (nan) | > loss: 0.009343384765088558 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.1359797467969472) | > loader_time: 0.0046 (0.00465207802195719)  --> TIME: 2025-02-25 12:49:57 -- STEP: 11002/36938 -- GLOBAL_STEP: 786700 | > loss_text_ce: 0.03633437678217888 (0.034123765011709786) | > loss_mel_ce: 2.504316568374634 (nan) | > loss: 0.010081948712468147 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13598691439806296) | > loader_time: 0.0046 (0.0046518539173172595)  --> TIME: 2025-02-25 12:50:07 -- STEP: 11052/36938 -- GLOBAL_STEP: 786750 | > loss_text_ce: 0.041982196271419525 (0.03412840978702203) | > loss_mel_ce: 2.127924680709839 (nan) | > loss: 0.008610742166638374 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13599117159023694) | > loader_time: 0.0046 (0.004651728700650926)  --> TIME: 2025-02-25 12:50:16 -- STEP: 11102/36938 -- GLOBAL_STEP: 786800 | > loss_text_ce: 0.03607306256890297 (0.03413171461137233) | > loss_mel_ce: 3.582103967666626 (nan) | > loss: 0.014357846230268478 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13598974552955137) | > loader_time: 0.0046 (0.004651610560512521)  --> TIME: 2025-02-25 12:50:25 -- STEP: 11152/36938 -- GLOBAL_STEP: 786850 | > loss_text_ce: 0.027414076030254364 (0.03413669733052334) | > loss_mel_ce: 3.1993520259857178 (nan) | > loss: 0.012804628349840641 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13597017466765393) | > loader_time: 0.0045 (0.0046514904225402704)  --> TIME: 2025-02-25 12:50:34 -- STEP: 11202/36938 -- GLOBAL_STEP: 786900 | > loss_text_ce: 0.03170806169509888 (0.03414422328499248) | > loss_mel_ce: 3.345302104949951 (nan) | > loss: 0.013400834985077381 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.13598507392663997) | > loader_time: 0.0046 (0.0046512432086469775)  --> TIME: 2025-02-25 12:50:44 -- STEP: 11252/36938 -- GLOBAL_STEP: 786950 | > loss_text_ce: 0.04140453785657883 (0.03414244454516596) | > loss_mel_ce: 2.7520203590393066 (nan) | > loss: 0.011085019446909428 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13599164777068196) | > loader_time: 0.0046 (0.004651154757308817)  --> TIME: 2025-02-25 12:50:53 -- STEP: 11302/36938 -- GLOBAL_STEP: 787000 | > loss_text_ce: 0.034154340624809265 (0.03413874958472886) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.1359831062255127) | > loader_time: 0.0045 (0.0046508023853197335)  --> TIME: 2025-02-25 12:51:02 -- STEP: 11352/36938 -- GLOBAL_STEP: 787050 | > loss_text_ce: 0.03528877720236778 (0.03414107316478458) | > loss_mel_ce: 2.0171163082122803 (nan) | > loss: 0.008144465275108814 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.1359714949903897) | > loader_time: 0.0045 (0.004650546388780668)  --> TIME: 2025-02-25 12:51:11 -- STEP: 11402/36938 -- GLOBAL_STEP: 787100 | > loss_text_ce: 0.030854303389787674 (0.034147867371197814) | > loss_mel_ce: 0.20279541611671448 (nan) | > loss: 0.0009271814487874508 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.13599000251454185) | > loader_time: 0.0048 (0.004650321263552666)  --> TIME: 2025-02-25 12:51:21 -- STEP: 11452/36938 -- GLOBAL_STEP: 787150 | > loss_text_ce: 0.025299496948719025 (0.03414586704743829) | > loss_mel_ce: 3.2632994651794434 (nan) | > loss: 0.013049996457993984 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13600373853139847) | > loader_time: 0.0046 (0.004650122691316693)  --> TIME: 2025-02-25 12:51:30 -- STEP: 11502/36938 -- GLOBAL_STEP: 787200 | > loss_text_ce: 0.03698447346687317 (0.03414476910311927) | > loss_mel_ce: 1.727746605873108 (nan) | > loss: 0.0070029012858867645 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13601809355553873) | > loader_time: 0.0046 (0.004649952502276366)  --> TIME: 2025-02-25 12:51:40 -- STEP: 11552/36938 -- GLOBAL_STEP: 787250 | > loss_text_ce: 0.032020892947912216 (0.03414720584358811) | > loss_mel_ce: 2.3134231567382812 (nan) | > loss: 0.009307317435741425 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13603161007083367) | > loader_time: 0.0045 (0.004650135924446273)  --> TIME: 2025-02-25 12:51:49 -- STEP: 11602/36938 -- GLOBAL_STEP: 787300 | > loss_text_ce: 0.033864445984363556 (0.034152642263854276) | > loss_mel_ce: 3.0931596755981445 (nan) | > loss: 0.012408826500177383 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.13602896880084273) | > loader_time: 0.0048 (0.004649986564815082)  --> TIME: 2025-02-25 12:51:58 -- STEP: 11652/36938 -- GLOBAL_STEP: 787350 | > loss_text_ce: 0.028777889907360077 (0.03415026611308311) | > loss_mel_ce: 2.894472599029541 (nan) | > loss: 0.011600201018154621 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.1360313173753065) | > loader_time: 0.0047 (0.004649726111906059)  --> TIME: 2025-02-25 12:52:08 -- STEP: 11702/36938 -- GLOBAL_STEP: 787400 | > loss_text_ce: 0.031833283603191376 (0.03415018887575966) | > loss_mel_ce: 3.1994359493255615 (nan) | > loss: 0.012822497636079788 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.13604709931345751) | > loader_time: 0.0047 (0.0046498354348009936)  --> TIME: 2025-02-25 12:52:17 -- STEP: 11752/36938 -- GLOBAL_STEP: 787450 | > loss_text_ce: 0.03630789741873741 (0.03415729003456593) | > loss_mel_ce: 2.7571611404418945 (nan) | > loss: 0.011085194535553455 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.13606153630415094) | > loader_time: 0.0048 (0.0046500528524163675)  --> TIME: 2025-02-25 12:52:27 -- STEP: 11802/36938 -- GLOBAL_STEP: 787500 | > loss_text_ce: 0.03827384486794472 (0.03415884284669994) | > loss_mel_ce: 3.187283515930176 (nan) | > loss: 0.012799831107258797 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.13604967979188123) | > loader_time: 0.0046 (0.004650020534720295)  --> TIME: 2025-02-25 12:52:36 -- STEP: 11852/36938 -- GLOBAL_STEP: 787550 | > loss_text_ce: 0.038532186299562454 (0.034156135412673455) | > loss_mel_ce: 3.017426013946533 (nan) | > loss: 0.012126819230616093 (nan) | > current_lr: 5e-06 | > step_time: 0.1765 (0.1360380110650545) | > loader_time: 0.0048 (0.004649829590920041)  --> TIME: 2025-02-25 12:52:45 -- STEP: 11902/36938 -- GLOBAL_STEP: 787600 | > loss_text_ce: 0.0333915613591671 (0.034155142945748285) | > loss_mel_ce: 2.8302018642425537 (nan) | > loss: 0.011363466270267963 (nan) | > current_lr: 5e-06 | > step_time: 0.185 (0.13605232063290937) | > loader_time: 0.0049 (0.004649821459155295)  --> TIME: 2025-02-25 12:52:55 -- STEP: 11952/36938 -- GLOBAL_STEP: 787650 | > loss_text_ce: 0.03285893425345421 (0.03415275335589587) | > loss_mel_ce: 2.518170118331909 (nan) | > loss: 0.010123131796717644 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13604731883709095) | > loader_time: 0.0046 (0.004650057858252609)  --> TIME: 2025-02-25 12:53:04 -- STEP: 12002/36938 -- GLOBAL_STEP: 787700 | > loss_text_ce: 0.036797888576984406 (0.03415394810198607) | > loss_mel_ce: 2.987064838409424 (nan) | > loss: 0.011999456211924553 (nan) | > current_lr: 5e-06 | > step_time: 0.1588 (0.13606390341622257) | > loader_time: 0.0047 (0.0046500511078849515)  --> TIME: 2025-02-25 12:53:13 -- STEP: 12052/36938 -- GLOBAL_STEP: 787750 | > loss_text_ce: 0.019110625609755516 (0.03414737749892683) | > loss_mel_ce: 2.3343775272369385 (nan) | > loss: 0.009339239448308945 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13605351452806913) | > loader_time: 0.0046 (0.004649869931958804)  --> TIME: 2025-02-25 12:53:22 -- STEP: 12102/36938 -- GLOBAL_STEP: 787800 | > loss_text_ce: 0.039038971066474915 (0.034144852397917994) | > loss_mel_ce: 2.1285576820373535 (nan) | > loss: 0.00860157422721386 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13604355417900016) | > loader_time: 0.0045 (0.0046499231948751545)  --> TIME: 2025-02-25 12:53:31 -- STEP: 12152/36938 -- GLOBAL_STEP: 787850 | > loss_text_ce: 0.030357850715517998 (0.0341476701190657) | > loss_mel_ce: 1.3339334726333618 (nan) | > loss: 0.005413854494690895 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13602388485863967) | > loader_time: 0.0045 (0.004649836131033719)  --> TIME: 2025-02-25 12:53:40 -- STEP: 12202/36938 -- GLOBAL_STEP: 787900 | > loss_text_ce: 0.02736690640449524 (0.03414956754372072) | > loss_mel_ce: 1.482038974761963 (nan) | > loss: 0.005989706143736839 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13600948854970718) | > loader_time: 0.0046 (0.004649609683908553)  --> TIME: 2025-02-25 12:53:50 -- STEP: 12252/36938 -- GLOBAL_STEP: 787950 | > loss_text_ce: 0.03957527503371239 (0.03415610341434549) | > loss_mel_ce: 3.6710851192474365 (nan) | > loss: 0.014724844135344028 (nan) | > current_lr: 5e-06 | > step_time: 0.1874 (0.13601065291091208) | > loader_time: 0.0048 (0.004649701166728937)  --> TIME: 2025-02-25 12:53:59 -- STEP: 12302/36938 -- GLOBAL_STEP: 788000 | > loss_text_ce: 0.03265335038304329 (0.03415919811499812) | > loss_mel_ce: 3.054065704345703 (nan) | > loss: 0.012248885817825794 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.136034380282993) | > loader_time: 0.0045 (0.004649709035004838)  --> TIME: 2025-02-25 12:54:09 -- STEP: 12352/36938 -- GLOBAL_STEP: 788050 | > loss_text_ce: 0.03018653765320778 (0.03416051899577261) | > loss_mel_ce: 2.6696560382843018 (nan) | > loss: 0.010713662020862103 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1360488673720335) | > loader_time: 0.0046 (0.004650299722048882)  --> TIME: 2025-02-25 12:54:18 -- STEP: 12402/36938 -- GLOBAL_STEP: 788100 | > loss_text_ce: 0.0355999730527401 (0.034160400779475894) | > loss_mel_ce: 1.8825581073760986 (nan) | > loss: 0.00761173851788044 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13605640129088745) | > loader_time: 0.0045 (0.004650516445416419)  --> TIME: 2025-02-25 12:54:28 -- STEP: 12452/36938 -- GLOBAL_STEP: 788150 | > loss_text_ce: 0.026854820549488068 (0.034158854607933574) | > loss_mel_ce: 2.7928624153137207 (nan) | > loss: 0.011189354583621025 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13606310336872424) | > loader_time: 0.0047 (0.00465065646776613)  --> TIME: 2025-02-25 12:54:37 -- STEP: 12502/36938 -- GLOBAL_STEP: 788200 | > loss_text_ce: 0.0292509738355875 (0.034158665737278056) | > loss_mel_ce: 3.1802453994750977 (nan) | > loss: 0.012736096978187561 (nan) | > current_lr: 5e-06 | > step_time: 0.1793 (0.13606980298542254) | > loader_time: 0.0048 (0.00465078331760129)  --> TIME: 2025-02-25 12:54:47 -- STEP: 12552/36938 -- GLOBAL_STEP: 788250 | > loss_text_ce: 0.027700917795300484 (0.03415460914034753) | > loss_mel_ce: 2.7755725383758545 (nan) | > loss: 0.011124101467430592 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13608506261124753) | > loader_time: 0.0046 (0.004650674860062589)  --> TIME: 2025-02-25 12:54:56 -- STEP: 12602/36938 -- GLOBAL_STEP: 788300 | > loss_text_ce: 0.03214099630713463 (0.03415495511228637) | > loss_mel_ce: 2.9659135341644287 (nan) | > loss: 0.011897042393684387 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13609692679948437) | > loader_time: 0.0045 (0.004650610455481348)  --> TIME: 2025-02-25 12:55:05 -- STEP: 12652/36938 -- GLOBAL_STEP: 788350 | > loss_text_ce: 0.03675068914890289 (0.03415223171153023) | > loss_mel_ce: 3.6270878314971924 (nan) | > loss: 0.014539043419063091 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.1360998302933692) | > loader_time: 0.0046 (0.004650505818485478)  --> TIME: 2025-02-25 12:55:15 -- STEP: 12702/36938 -- GLOBAL_STEP: 788400 | > loss_text_ce: 0.03276737034320831 (0.0341489858603609) | > loss_mel_ce: 2.949213981628418 (nan) | > loss: 0.011833259835839272 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.13611094781947414) | > loader_time: 0.0046 (0.004650413586462285)  --> TIME: 2025-02-25 12:55:24 -- STEP: 12752/36938 -- GLOBAL_STEP: 788450 | > loss_text_ce: 0.03626397252082825 (0.03414718804508788) | > loss_mel_ce: 2.35262131690979 (nan) | > loss: 0.009479704312980175 (nan) | > current_lr: 5e-06 | > step_time: 0.1079 (0.13610867408093746) | > loader_time: 0.0045 (0.00465032787364877)  --> TIME: 2025-02-25 12:55:33 -- STEP: 12802/36938 -- GLOBAL_STEP: 788500 | > loss_text_ce: 0.03237944468855858 (0.03414626705270076) | > loss_mel_ce: 2.114877462387085 (nan) | > loss: 0.008520861156284809 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13610561645286148) | > loader_time: 0.0047 (0.004650530880679367)  --> TIME: 2025-02-25 12:55:42 -- STEP: 12852/36938 -- GLOBAL_STEP: 788550 | > loss_text_ce: 0.031426336616277695 (0.03415114164220227) | > loss_mel_ce: 2.6456243991851807 (nan) | > loss: 0.01062321849167347 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13610414560064546) | > loader_time: 0.0046 (0.004650663260986147)  --> TIME: 2025-02-25 12:55:52 -- STEP: 12902/36938 -- GLOBAL_STEP: 788600 | > loss_text_ce: 0.03646206855773926 (0.03414980325414165) | > loss_mel_ce: 2.8817508220672607 (nan) | > loss: 0.0115802101790905 (nan) | > current_lr: 5e-06 | > step_time: 0.1662 (0.13610986158323396) | > loader_time: 0.0047 (0.004650550578993567)  --> TIME: 2025-02-25 12:56:01 -- STEP: 12952/36938 -- GLOBAL_STEP: 788650 | > loss_text_ce: 0.027266651391983032 (0.034149806077895084) | > loss_mel_ce: 3.044421911239624 (nan) | > loss: 0.012189241126179695 (nan) | > current_lr: 5e-06 | > step_time: 0.1688 (0.1360930042034173) | > loader_time: 0.0069 (0.004650603480512807)  --> TIME: 2025-02-25 12:56:10 -- STEP: 13002/36938 -- GLOBAL_STEP: 788700 | > loss_text_ce: 0.03013923577964306 (0.034149477484242886) | > loss_mel_ce: 0.8747965693473816 (nan) | > loss: 0.003591015236452222 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13606614621671054) | > loader_time: 0.0046 (0.004650478142258777)  --> TIME: 2025-02-25 12:56:19 -- STEP: 13052/36938 -- GLOBAL_STEP: 788750 | > loss_text_ce: 0.04607011377811432 (0.03414921567898314) | > loss_mel_ce: 3.2043933868408203 (nan) | > loss: 0.01289866492152214 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13606635398233474) | > loader_time: 0.0046 (0.004650325724730037)  --> TIME: 2025-02-25 12:56:28 -- STEP: 13102/36938 -- GLOBAL_STEP: 788800 | > loss_text_ce: 0.04075099527835846 (0.034146398081983054) | > loss_mel_ce: 0.10742797702550888 (nan) | > loss: 0.0005880118114873767 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13607751125780607) | > loader_time: 0.0046 (0.004650416728686295)  --> TIME: 2025-02-25 12:56:38 -- STEP: 13152/36938 -- GLOBAL_STEP: 788850 | > loss_text_ce: 0.033907510340213776 (0.034149984690902205) | > loss_mel_ce: 3.2603001594543457 (nan) | > loss: 0.013072253204882145 (nan) | > current_lr: 5e-06 | > step_time: 0.1857 (0.13608224287520385) | > loader_time: 0.005 (0.004650506641887965)  --> TIME: 2025-02-25 12:56:47 -- STEP: 13202/36938 -- GLOBAL_STEP: 788900 | > loss_text_ce: 0.04695628583431244 (0.03415558271566046) | > loss_mel_ce: 3.1380255222320557 (nan) | > loss: 0.01263881754130125 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.13609197049082436) | > loader_time: 0.0048 (0.00465066404780407)  --> TIME: 2025-02-25 12:56:57 -- STEP: 13252/36938 -- GLOBAL_STEP: 788950 | > loss_text_ce: 0.03910364955663681 (0.03415661856167766) | > loss_mel_ce: 3.1482269763946533 (nan) | > loss: 0.012648138217628002 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13610168633920147) | > loader_time: 0.0046 (0.00465063940100607)  --> TIME: 2025-02-25 12:57:06 -- STEP: 13302/36938 -- GLOBAL_STEP: 789000 | > loss_text_ce: 0.037851519882678986 (0.03416194402408434) | > loss_mel_ce: 3.1725642681121826 (nan) | > loss: 0.012739745900034904 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.13610998165731142) | > loader_time: 0.0047 (0.004650590724827485)  --> TIME: 2025-02-25 12:57:15 -- STEP: 13352/36938 -- GLOBAL_STEP: 789050 | > loss_text_ce: 0.033424995839595795 (0.03416453064723435) | > loss_mel_ce: 2.9864501953125 (nan) | > loss: 0.011983633041381836 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13609028775010606) | > loader_time: 0.0045 (0.00465059414318466)  --> TIME: 2025-02-25 12:57:24 -- STEP: 13402/36938 -- GLOBAL_STEP: 789100 | > loss_text_ce: 0.03507604822516441 (0.034161335269011044) | > loss_mel_ce: 2.7853384017944336 (nan) | > loss: 0.011192121542990208 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13609354653619632) | > loader_time: 0.0046 (0.00465039503501433)  --> TIME: 2025-02-25 12:57:34 -- STEP: 13452/36938 -- GLOBAL_STEP: 789150 | > loss_text_ce: 0.04256928339600563 (0.03416287264122843) | > loss_mel_ce: 3.285443067550659 (nan) | > loss: 0.01320639904588461 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.1360909292430635) | > loader_time: 0.0047 (0.004650391020166566)  --> TIME: 2025-02-25 12:57:43 -- STEP: 13502/36938 -- GLOBAL_STEP: 789200 | > loss_text_ce: 0.02791016362607479 (0.03416272139414273) | > loss_mel_ce: 2.9089949131011963 (nan) | > loss: 0.011654386296868324 (nan) | > current_lr: 5e-06 | > step_time: 0.1817 (0.13609713574653745) | > loader_time: 0.0048 (0.004650366975543875)  --> TIME: 2025-02-25 12:57:52 -- STEP: 13552/36938 -- GLOBAL_STEP: 789250 | > loss_text_ce: 0.029427727684378624 (0.03415869936853341) | > loss_mel_ce: 2.5768542289733887 (nan) | > loss: 0.010342389345169067 (nan) | > current_lr: 5e-06 | > step_time: 0.1487 (0.1360914295686603) | > loader_time: 0.0046 (0.004650250464290774)  --> TIME: 2025-02-25 12:58:01 -- STEP: 13602/36938 -- GLOBAL_STEP: 789300 | > loss_text_ce: 0.029455427080392838 (0.03415690629580481) | > loss_mel_ce: 2.3643834590911865 (nan) | > loss: 0.00949936080724001 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13607264192573076) | > loader_time: 0.0045 (0.004650172933447529)  --> TIME: 2025-02-25 12:58:11 -- STEP: 13652/36938 -- GLOBAL_STEP: 789350 | > loss_text_ce: 0.03542438521981239 (0.03415813430714792) | > loss_mel_ce: 2.7794854640960693 (nan) | > loss: 0.011170278303325176 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13608897285939262) | > loader_time: 0.0047 (0.004650386414358848)  --> TIME: 2025-02-25 12:58:20 -- STEP: 13702/36938 -- GLOBAL_STEP: 789400 | > loss_text_ce: 0.034359004348516464 (0.03415410123466935) | > loss_mel_ce: 2.2147316932678223 (nan) | > loss: 0.008924963884055614 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13609072783100876) | > loader_time: 0.0046 (0.004650448764471955)  --> TIME: 2025-02-25 12:58:30 -- STEP: 13752/36938 -- GLOBAL_STEP: 789450 | > loss_text_ce: 0.03619764745235443 (0.03415675931477419) | > loss_mel_ce: 2.8887557983398438 (nan) | > loss: 0.011606958694756031 (nan) | > current_lr: 5e-06 | > step_time: 0.1642 (0.1361213279644235) | > loader_time: 0.0049 (0.004650565948924616)  --> TIME: 2025-02-25 12:58:39 -- STEP: 13802/36938 -- GLOBAL_STEP: 789500 | > loss_text_ce: 0.040160171687603 (0.03415492997001242) | > loss_mel_ce: 2.0813777446746826 (nan) | > loss: 0.008418802171945572 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13611661800524955) | > loader_time: 0.0046 (0.0046506271623490735)  --> TIME: 2025-02-25 12:58:49 -- STEP: 13852/36938 -- GLOBAL_STEP: 789550 | > loss_text_ce: 0.029703857377171516 (0.034153446156911696) | > loss_mel_ce: 2.138671875 (nan) | > loss: 0.008604666218161583 (nan) | > current_lr: 5e-06 | > step_time: 0.1075 (0.13612854697612775) | > loader_time: 0.0045 (0.004650530342142815)  --> TIME: 2025-02-25 12:58:58 -- STEP: 13902/36938 -- GLOBAL_STEP: 789600 | > loss_text_ce: 0.04526606947183609 (0.03415465349539961) | > loss_mel_ce: 3.0725302696228027 (nan) | > loss: 0.012372208759188652 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13614210304193872) | > loader_time: 0.0045 (0.004650457285064849)  --> TIME: 2025-02-25 12:59:08 -- STEP: 13952/36938 -- GLOBAL_STEP: 789650 | > loss_text_ce: 0.028622960671782494 (0.03414946123019087) | > loss_mel_ce: 2.091398239135742 (nan) | > loss: 0.008412783034145832 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13613768248314723) | > loader_time: 0.0046 (0.004650310655936195)  --> TIME: 2025-02-25 12:59:17 -- STEP: 14002/36938 -- GLOBAL_STEP: 789700 | > loss_text_ce: 0.04392581433057785 (0.03415013527962697) | > loss_mel_ce: 3.4321627616882324 (nan) | > loss: 0.013794003054499626 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13614226863854575) | > loader_time: 0.0046 (0.004650125859754634)  --> TIME: 2025-02-25 12:59:27 -- STEP: 14052/36938 -- GLOBAL_STEP: 789750 | > loss_text_ce: 0.032359007745981216 (0.03414921947211965) | > loss_mel_ce: 2.4974260330200195 (nan) | > loss: 0.0100388303399086 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13615748836822494) | > loader_time: 0.0045 (0.00465008250811214)  --> TIME: 2025-02-25 12:59:36 -- STEP: 14102/36938 -- GLOBAL_STEP: 789800 | > loss_text_ce: 0.034827426075935364 (0.03414748294739671) | > loss_mel_ce: 1.386440634727478 (nan) | > loss: 0.005639953073114157 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1361445338484959) | > loader_time: 0.0045 (0.004650017941628704)  --> TIME: 2025-02-25 12:59:45 -- STEP: 14152/36938 -- GLOBAL_STEP: 789850 | > loss_text_ce: 0.02869187854230404 (0.03415114272020188) | > loss_mel_ce: 1.95703125 (nan) | > loss: 0.007879854179918766 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13613262090257158) | > loader_time: 0.0045 (0.004649980938185271)  --> TIME: 2025-02-25 12:59:54 -- STEP: 14202/36938 -- GLOBAL_STEP: 789900 | > loss_text_ce: 0.03412419930100441 (0.03415039026429827) | > loss_mel_ce: 2.981032133102417 (nan) | > loss: 0.011964906938374043 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13613331222211925) | > loader_time: 0.0046 (0.004649932057802792)  --> TIME: 2025-02-25 13:00:04 -- STEP: 14252/36938 -- GLOBAL_STEP: 789950 | > loss_text_ce: 0.03864695504307747 (0.034149693848524315) | > loss_mel_ce: 3.097698211669922 (nan) | > loss: 0.012445814907550812 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.136148342633228) | > loader_time: 0.0046 (0.0046500011404478105)  --> TIME: 2025-02-25 13:00:13 -- STEP: 14302/36938 -- GLOBAL_STEP: 790000 | > loss_text_ce: 0.034503135830163956 (0.0341494120246546) | > loss_mel_ce: 2.9085333347320557 (nan) | > loss: 0.011678717099130154 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1361671701009056) | > loader_time: 0.0045 (0.004650011210654308) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_790000.pth  --> TIME: 2025-02-25 13:00:29 -- STEP: 14352/36938 -- GLOBAL_STEP: 790050 | > loss_text_ce: 0.03564959391951561 (0.03414796346914295) | > loss_mel_ce: 3.0561728477478027 (nan) | > loss: 0.012269137427210808 (nan) | > current_lr: 5e-06 | > step_time: 0.163 (0.1361705599188677) | > loader_time: 0.0046 (0.004649974912438244)  --> TIME: 2025-02-25 13:00:38 -- STEP: 14402/36938 -- GLOBAL_STEP: 790100 | > loss_text_ce: 0.04617660492658615 (0.03414213382806462) | > loss_mel_ce: 3.555039644241333 (nan) | > loss: 0.014290541410446167 (nan) | > current_lr: 5e-06 | > step_time: 0.1778 (0.13617395821618944) | > loader_time: 0.0048 (0.004649876968808651)  --> TIME: 2025-02-25 13:00:47 -- STEP: 14452/36938 -- GLOBAL_STEP: 790150 | > loss_text_ce: 0.03623693436384201 (0.03414104915892399) | > loss_mel_ce: 0.020294189453125 (nan) | > loss: 0.0002243298658868298 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.1361611042199561) | > loader_time: 0.0046 (0.004649782820880304)  --> TIME: 2025-02-25 13:00:57 -- STEP: 14502/36938 -- GLOBAL_STEP: 790200 | > loss_text_ce: 0.031635794788599014 (0.03413937691395664) | > loss_mel_ce: 3.078115940093994 (nan) | > loss: 0.012340284883975983 (nan) | > current_lr: 5e-06 | > step_time: 0.1651 (0.13617967676416098) | > loader_time: 0.0048 (0.004650557243385052)  --> TIME: 2025-02-25 13:01:06 -- STEP: 14552/36938 -- GLOBAL_STEP: 790250 | > loss_text_ce: 0.02677266299724579 (0.034137332737327906) | > loss_mel_ce: 1.9010082483291626 (nan) | > loss: 0.007649924606084824 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13617823858258551) | > loader_time: 0.0046 (0.004650585971866884)  --> TIME: 2025-02-25 13:01:15 -- STEP: 14602/36938 -- GLOBAL_STEP: 790300 | > loss_text_ce: 0.049399618059396744 (0.034141011100883224) | > loss_mel_ce: 3.1991546154022217 (nan) | > loss: 0.012891088612377644 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13618744103782238) | > loader_time: 0.0047 (0.004650603727255277)  --> TIME: 2025-02-25 13:01:25 -- STEP: 14652/36938 -- GLOBAL_STEP: 790350 | > loss_text_ce: 0.04613643139600754 (0.0341395650016015) | > loss_mel_ce: 2.433480739593506 (nan) | > loss: 0.009839750826358795 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13618833334303487) | > loader_time: 0.0046 (0.004650594593884357)  --> TIME: 2025-02-25 13:01:34 -- STEP: 14702/36938 -- GLOBAL_STEP: 790400 | > loss_text_ce: 0.03560185059905052 (0.03413798018743659) | > loss_mel_ce: 3.4400596618652344 (nan) | > loss: 0.013792308047413826 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.1361967345157175) | > loader_time: 0.0046 (0.004650524336859385)  --> TIME: 2025-02-25 13:01:44 -- STEP: 14752/36938 -- GLOBAL_STEP: 790450 | > loss_text_ce: 0.03147317096590996 (0.03414022835507401) | > loss_mel_ce: 1.8312337398529053 (nan) | > loss: 0.007391694467514753 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13620907983009384) | > loader_time: 0.0045 (0.004650703463662995)  --> TIME: 2025-02-25 13:01:53 -- STEP: 14802/36938 -- GLOBAL_STEP: 790500 | > loss_text_ce: 0.03271934390068054 (0.034140637862474454) | > loss_mel_ce: 3.181591510772705 (nan) | > loss: 0.012755203060805798 (nan) | > current_lr: 5e-06 | > step_time: 0.1746 (0.13620511967819232) | > loader_time: 0.0048 (0.004650721548699224)  --> TIME: 2025-02-25 13:02:02 -- STEP: 14852/36938 -- GLOBAL_STEP: 790550 | > loss_text_ce: 0.029031962156295776 (0.034141479812034664) | > loss_mel_ce: 2.873741626739502 (nan) | > loss: 0.011518944054841995 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13620279374997238) | > loader_time: 0.0045 (0.004650636163003169)  --> TIME: 2025-02-25 13:02:12 -- STEP: 14902/36938 -- GLOBAL_STEP: 790600 | > loss_text_ce: 0.03637394309043884 (0.034139174185548224) | > loss_mel_ce: 3.00378155708313 (nan) | > loss: 0.012064109556376934 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13621052707254405) | > loader_time: 0.0045 (0.00465061385876988)  --> TIME: 2025-02-25 13:02:21 -- STEP: 14952/36938 -- GLOBAL_STEP: 790650 | > loss_text_ce: 0.03567180410027504 (0.034141430934620014) | > loss_mel_ce: 2.605088472366333 (nan) | > loss: 0.010479208081960678 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.1362112872960295) | > loader_time: 0.0045 (0.004650503269423138)  --> TIME: 2025-02-25 13:02:30 -- STEP: 15002/36938 -- GLOBAL_STEP: 790700 | > loss_text_ce: 0.03352101147174835 (0.0341432693909415) | > loss_mel_ce: 2.9436652660369873 (nan) | > loss: 0.011814231984317303 (nan) | > current_lr: 5e-06 | > step_time: 0.1623 (0.13621022572216773) | > loader_time: 0.0047 (0.00465055599703406)  --> TIME: 2025-02-25 13:02:40 -- STEP: 15052/36938 -- GLOBAL_STEP: 790750 | > loss_text_ce: 0.03473950922489166 (0.034142577831482154) | > loss_mel_ce: 2.612194299697876 (nan) | > loss: 0.010503706522285938 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13622238941405565) | > loader_time: 0.0045 (0.004650532534044542)  --> TIME: 2025-02-25 13:02:49 -- STEP: 15102/36938 -- GLOBAL_STEP: 790800 | > loss_text_ce: 0.03477180004119873 (0.03413971313974103) | > loss_mel_ce: 3.490938186645508 (nan) | > loss: 0.013990913517773151 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13621556321574707) | > loader_time: 0.0048 (0.00465046483275907)  --> TIME: 2025-02-25 13:02:58 -- STEP: 15152/36938 -- GLOBAL_STEP: 790850 | > loss_text_ce: 0.04252657666802406 (0.03413906686955524) | > loss_mel_ce: 3.2526042461395264 (nan) | > loss: 0.013075916096568108 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13621207267643964) | > loader_time: 0.0045 (0.004650435814635938)  --> TIME: 2025-02-25 13:03:08 -- STEP: 15202/36938 -- GLOBAL_STEP: 790900 | > loss_text_ce: 0.05226049944758415 (0.03414450453158377) | > loss_mel_ce: 2.9249775409698486 (nan) | > loss: 0.011814436875283718 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13622218945233158) | > loader_time: 0.0046 (0.004650465627513962)  --> TIME: 2025-02-25 13:03:17 -- STEP: 15252/36938 -- GLOBAL_STEP: 790950 | > loss_text_ce: 0.021714579313993454 (0.03414213524666292) | > loss_mel_ce: 2.700639486312866 (nan) | > loss: 0.010802993550896645 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13622546386693793) | > loader_time: 0.0046 (0.004650648688194547)  --> TIME: 2025-02-25 13:03:26 -- STEP: 15302/36938 -- GLOBAL_STEP: 791000 | > loss_text_ce: 0.03608357533812523 (0.03414514299711924) | > loss_mel_ce: 2.6336262226104736 (nan) | > loss: 0.010594086721539497 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.1362321964511221) | > loader_time: 0.0047 (0.004651096783000935)  --> TIME: 2025-02-25 13:03:36 -- STEP: 15352/36938 -- GLOBAL_STEP: 791050 | > loss_text_ce: 0.031818974763154984 (0.03415172355790165) | > loss_mel_ce: 3.3237600326538086 (nan) | > loss: 0.013315790332853794 (nan) | > current_lr: 5e-06 | > step_time: 0.1866 (0.1362439379398874) | > loader_time: 0.0051 (0.004651217317133915)  --> TIME: 2025-02-25 13:03:45 -- STEP: 15402/36938 -- GLOBAL_STEP: 791100 | > loss_text_ce: 0.02936805970966816 (0.03415122219372126) | > loss_mel_ce: 1.9939631223678589 (nan) | > loss: 0.008029092103242874 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.1362391264310772) | > loader_time: 0.0046 (0.004651243478319059)  --> TIME: 2025-02-25 13:03:54 -- STEP: 15452/36938 -- GLOBAL_STEP: 791150 | > loss_text_ce: 0.04079888388514519 (0.03415404164645537) | > loss_mel_ce: 2.730577230453491 (nan) | > loss: 0.010997525416314602 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13622061861570522) | > loader_time: 0.0045 (0.0046511540103076)  --> TIME: 2025-02-25 13:04:04 -- STEP: 15502/36938 -- GLOBAL_STEP: 791200 | > loss_text_ce: 0.04468448832631111 (0.03415731073716067) | > loss_mel_ce: 2.2709853649139404 (nan) | > loss: 0.009189166128635406 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.13623573366034056) | > loader_time: 0.0049 (0.004651239111783079)  --> TIME: 2025-02-25 13:04:13 -- STEP: 15552/36938 -- GLOBAL_STEP: 791250 | > loss_text_ce: 0.0346469022333622 (0.03415386479388904) | > loss_mel_ce: 2.617220163345337 (nan) | > loss: 0.010523282922804356 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13622370975498363) | > loader_time: 0.0046 (0.004651137263563931)  --> TIME: 2025-02-25 13:04:22 -- STEP: 15602/36938 -- GLOBAL_STEP: 791300 | > loss_text_ce: 0.04348117113113403 (0.03415693194988515) | > loss_mel_ce: 2.106489658355713 (nan) | > loss: 0.008531630970537663 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13621762817814653) | > loader_time: 0.0045 (0.004651171185239561)  --> TIME: 2025-02-25 13:04:32 -- STEP: 15652/36938 -- GLOBAL_STEP: 791350 | > loss_text_ce: 0.035678040236234665 (0.034157949815324065) | > loss_mel_ce: 2.657960891723633 (nan) | > loss: 0.010689044371247292 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13622982919962598) | > loader_time: 0.0046 (0.004651187951687263)  --> TIME: 2025-02-25 13:04:41 -- STEP: 15702/36938 -- GLOBAL_STEP: 791400 | > loss_text_ce: 0.030643245205283165 (0.034155209329730846) | > loss_mel_ce: 1.6458536386489868 (nan) | > loss: 0.006652765907347202 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13623685934576762) | > loader_time: 0.0045 (0.004651427010854576)  --> TIME: 2025-02-25 13:04:50 -- STEP: 15752/36938 -- GLOBAL_STEP: 791450 | > loss_text_ce: 0.035229381173849106 (0.034154501334180375) | > loss_mel_ce: 3.4527251720428467 (nan) | > loss: 0.013841090723872185 (nan) | > current_lr: 5e-06 | > step_time: 0.1864 (0.1362381172912752) | > loader_time: 0.0049 (0.004651380030257626)  --> TIME: 2025-02-25 13:05:00 -- STEP: 15802/36938 -- GLOBAL_STEP: 791500 | > loss_text_ce: 0.045431435108184814 (0.034153340522335644) | > loss_mel_ce: 3.106563091278076 (nan) | > loss: 0.012507915496826172 (nan) | > current_lr: 5e-06 | > step_time: 0.1866 (0.1362569210816304) | > loader_time: 0.0052 (0.004651367626617479)  --> TIME: 2025-02-25 13:05:09 -- STEP: 15852/36938 -- GLOBAL_STEP: 791550 | > loss_text_ce: 0.034794338047504425 (0.03415177689380886) | > loss_mel_ce: 2.7587642669677734 (nan) | > loss: 0.011085550300776958 (nan) | > current_lr: 5e-06 | > step_time: 0.1773 (0.13626372454715735) | > loader_time: 0.0048 (0.004651631561157055)  --> TIME: 2025-02-25 13:05:19 -- STEP: 15902/36938 -- GLOBAL_STEP: 791600 | > loss_text_ce: 0.057821791619062424 (0.03415003725698978) | > loss_mel_ce: 0.01029004156589508 (nan) | > loss: 0.00027028509066440165 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13627105506767004) | > loader_time: 0.0045 (0.0046517807138054664)  --> TIME: 2025-02-25 13:05:28 -- STEP: 15952/36938 -- GLOBAL_STEP: 791650 | > loss_text_ce: 0.030319515615701675 (0.034151250122798314) | > loss_mel_ce: 3.118583917617798 (nan) | > loss: 0.012495649047195911 (nan) | > current_lr: 5e-06 | > step_time: 0.1693 (0.13627337720413316) | > loader_time: 0.0048 (0.004651688405479811)  --> TIME: 2025-02-25 13:05:37 -- STEP: 16002/36938 -- GLOBAL_STEP: 791700 | > loss_text_ce: 0.03316326066851616 (0.03414964281380363) | > loss_mel_ce: 2.9203941822052 (nan) | > loss: 0.011720467358827591 (nan) | > current_lr: 5e-06 | > step_time: 0.1452 (0.13627635756219592) | > loader_time: 0.0046 (0.004651669635875218)  --> TIME: 2025-02-25 13:05:47 -- STEP: 16052/36938 -- GLOBAL_STEP: 791750 | > loss_text_ce: 0.030983811244368553 (0.03415077620145529) | > loss_mel_ce: 2.639174222946167 (nan) | > loss: 0.010595866478979588 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1362822293908471) | > loader_time: 0.0045 (0.004651589581356249)  --> TIME: 2025-02-25 13:05:56 -- STEP: 16102/36938 -- GLOBAL_STEP: 791800 | > loss_text_ce: 0.02171209454536438 (0.03414787720837259) | > loss_mel_ce: 2.740772008895874 (nan) | > loss: 0.01096223946660757 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13630335081617212) | > loader_time: 0.0045 (0.004651651547097555)  --> TIME: 2025-02-25 13:06:06 -- STEP: 16152/36938 -- GLOBAL_STEP: 791850 | > loss_text_ce: 0.0346585251390934 (0.03414543737398359) | > loss_mel_ce: 0.02897135354578495 (nan) | > loss: 0.0002524995361454785 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.1363144625880571) | > loader_time: 0.0045 (0.004651618423055699)  --> TIME: 2025-02-25 13:06:15 -- STEP: 16202/36938 -- GLOBAL_STEP: 791900 | > loss_text_ce: 0.04222562164068222 (0.03414682884312132) | > loss_mel_ce: 1.6247845888137817 (nan) | > loss: 0.006615119986236095 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13631727995364573) | > loader_time: 0.0045 (0.00465169433840498)  --> TIME: 2025-02-25 13:06:25 -- STEP: 16252/36938 -- GLOBAL_STEP: 791950 | > loss_text_ce: 0.0386904813349247 (0.03414552667512415) | > loss_mel_ce: 2.5279719829559326 (nan) | > loss: 0.01018516905605793 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.1363115319911118) | > loader_time: 0.0046 (0.004651700822467326)  --> TIME: 2025-02-25 13:06:34 -- STEP: 16302/36938 -- GLOBAL_STEP: 792000 | > loss_text_ce: 0.028834663331508636 (0.034145231004333615) | > loss_mel_ce: 2.343473434448242 (nan) | > loss: 0.009413921274244785 (nan) | > current_lr: 5e-06 | > step_time: 0.1523 (0.13631659296447893) | > loader_time: 0.0047 (0.00465168701097463)  --> TIME: 2025-02-25 13:06:43 -- STEP: 16352/36938 -- GLOBAL_STEP: 792050 | > loss_text_ce: 0.03339659795165062 (0.03414107178213254) | > loss_mel_ce: 0.9581283330917358 (nan) | > loss: 0.003934623207896948 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13632353026339736) | > loader_time: 0.0045 (0.004651734652817837)  --> TIME: 2025-02-25 13:06:53 -- STEP: 16402/36938 -- GLOBAL_STEP: 792100 | > loss_text_ce: 0.03956814482808113 (0.03414534153595044) | > loss_mel_ce: 2.2021484375 (nan) | > loss: 0.008895701728761196 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13631410221401785) | > loader_time: 0.0047 (0.0046517377567791346)  --> TIME: 2025-02-25 13:07:02 -- STEP: 16452/36938 -- GLOBAL_STEP: 792150 | > loss_text_ce: 0.0406453013420105 (0.034142903928003454) | > loss_mel_ce: 2.8405582904815674 (nan) | > loss: 0.011433348059654236 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13632040630429199) | > loader_time: 0.0066 (0.0046517631447057315)  --> TIME: 2025-02-25 13:07:12 -- STEP: 16502/36938 -- GLOBAL_STEP: 792200 | > loss_text_ce: 0.04083985835313797 (0.03414158347699261) | > loss_mel_ce: 1.17457115650177 (nan) | > loss: 0.004823060240596533 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13633474600328147) | > loader_time: 0.0046 (0.004651791557313866)  --> TIME: 2025-02-25 13:07:21 -- STEP: 16552/36938 -- GLOBAL_STEP: 792250 | > loss_text_ce: 0.03608169034123421 (0.03413743732702805) | > loss_mel_ce: 2.864576816558838 (nan) | > loss: 0.01151055097579956 (nan) | > current_lr: 5e-06 | > step_time: 0.1694 (0.13635995891890523) | > loader_time: 0.0048 (0.004651780114650963)  --> TIME: 2025-02-25 13:07:31 -- STEP: 16602/36938 -- GLOBAL_STEP: 792300 | > loss_text_ce: 0.03389735519886017 (0.034137237370516806) | > loss_mel_ce: 0.051239013671875 (nan) | > loss: 0.00033784276456572115 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13635174561201782) | > loader_time: 0.0046 (0.004651758717048779)  --> TIME: 2025-02-25 13:07:40 -- STEP: 16652/36938 -- GLOBAL_STEP: 792350 | > loss_text_ce: 0.038488078862428665 (0.03413775581651735) | > loss_mel_ce: 2.9102721214294434 (nan) | > loss: 0.011701430194079876 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13637382917440877) | > loader_time: 0.0046 (0.004651803438290655)  --> TIME: 2025-02-25 13:07:50 -- STEP: 16702/36938 -- GLOBAL_STEP: 792400 | > loss_text_ce: 0.03152291849255562 (0.034138371467807994) | > loss_mel_ce: 3.0116477012634277 (nan) | > loss: 0.012076075188815594 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.1363722225560817) | > loader_time: 0.0046 (0.004651710382122839)  --> TIME: 2025-02-25 13:07:59 -- STEP: 16752/36938 -- GLOBAL_STEP: 792450 | > loss_text_ce: 0.03398186340928078 (0.03413920171517492) | > loss_mel_ce: 2.688708782196045 (nan) | > loss: 0.010804328136146069 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13637552603211126) | > loader_time: 0.0047 (0.004651904988539372)  --> TIME: 2025-02-25 13:08:08 -- STEP: 16802/36938 -- GLOBAL_STEP: 792500 | > loss_text_ce: 0.030230293050408363 (0.03414051750824058) | > loss_mel_ce: 2.194369316101074 (nan) | > loss: 0.00882777664810419 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13637111354705778) | > loader_time: 0.0047 (0.004651958694657806)  --> TIME: 2025-02-25 13:08:17 -- STEP: 16852/36938 -- GLOBAL_STEP: 792550 | > loss_text_ce: 0.031708501279354095 (0.0341408718912315) | > loss_mel_ce: 2.0284667015075684 (nan) | > loss: 0.008175298571586609 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13636610516736763) | > loader_time: 0.0046 (0.004651842450065629)  --> TIME: 2025-02-25 13:08:27 -- STEP: 16902/36938 -- GLOBAL_STEP: 792600 | > loss_text_ce: 0.03674311190843582 (0.03413820602489058) | > loss_mel_ce: 3.1954195499420166 (nan) | > loss: 0.01282604318112135 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13636435119234883) | > loader_time: 0.0046 (0.004651845143315586)  --> TIME: 2025-02-25 13:08:36 -- STEP: 16952/36938 -- GLOBAL_STEP: 792650 | > loss_text_ce: 0.03908369317650795 (0.03413724622741047) | > loss_mel_ce: 3.2658846378326416 (nan) | > loss: 0.013114954344928265 (nan) | > current_lr: 5e-06 | > step_time: 0.1531 (0.13636811297701615) | > loader_time: 0.0046 (0.004651688696810417)  --> TIME: 2025-02-25 13:08:45 -- STEP: 17002/36938 -- GLOBAL_STEP: 792700 | > loss_text_ce: 0.03838956728577614 (0.03413557186030368) | > loss_mel_ce: 2.4163482189178467 (nan) | > loss: 0.009741024114191532 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13636490897899023) | > loader_time: 0.0047 (0.0046516319903580315)  --> TIME: 2025-02-25 13:08:55 -- STEP: 17052/36938 -- GLOBAL_STEP: 792750 | > loss_text_ce: 0.035028114914894104 (0.0341389154722167) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.13636022659358174) | > loader_time: 0.0047 (0.004651580426213672)  --> TIME: 2025-02-25 13:09:04 -- STEP: 17102/36938 -- GLOBAL_STEP: 792800 | > loss_text_ce: 0.025219641625881195 (0.034138296681913996) | > loss_mel_ce: 3.2042059898376465 (nan) | > loss: 0.012815182097256184 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.1363590313055243) | > loader_time: 0.0046 (0.0046515110681913325)  --> TIME: 2025-02-25 13:09:13 -- STEP: 17152/36938 -- GLOBAL_STEP: 792850 | > loss_text_ce: 0.03625752031803131 (0.034137130391987264) | > loss_mel_ce: 2.9108293056488037 (nan) | > loss: 0.011694789864122868 (nan) | > current_lr: 5e-06 | > step_time: 0.1874 (0.13636921689525958) | > loader_time: 0.0048 (0.0046514420311397665)  --> TIME: 2025-02-25 13:09:23 -- STEP: 17202/36938 -- GLOBAL_STEP: 792900 | > loss_text_ce: 0.038504745811223984 (0.03413443955242086) | > loss_mel_ce: 3.0787932872772217 (nan) | > loss: 0.012370231561362743 (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.13638582998563425) | > loader_time: 0.0046 (0.004651398232420893)  --> TIME: 2025-02-25 13:09:32 -- STEP: 17252/36938 -- GLOBAL_STEP: 792950 | > loss_text_ce: 0.03960712254047394 (0.03412988084376393) | > loss_mel_ce: 2.994877576828003 (nan) | > loss: 0.0120416060090065 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.1363787185750542) | > loader_time: 0.0046 (0.00465126148708565)  --> TIME: 2025-02-25 13:09:41 -- STEP: 17302/36938 -- GLOBAL_STEP: 793000 | > loss_text_ce: 0.03655161336064339 (0.03412810793147994) | > loss_mel_ce: 3.170912265777588 (nan) | > loss: 0.012728032656013966 (nan) | > current_lr: 5e-06 | > step_time: 0.1578 (0.13636195892516587) | > loader_time: 0.0047 (0.004651269889569627)  --> TIME: 2025-02-25 13:09:50 -- STEP: 17352/36938 -- GLOBAL_STEP: 793050 | > loss_text_ce: 0.05276528745889664 (0.03412797263352847) | > loss_mel_ce: 2.2423360347747803 (nan) | > loss: 0.00910754594951868 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13635170415778977) | > loader_time: 0.0045 (0.004651367595636564)  --> TIME: 2025-02-25 13:10:00 -- STEP: 17402/36938 -- GLOBAL_STEP: 793100 | > loss_text_ce: 0.03445546329021454 (0.03412789548096994) | > loss_mel_ce: 0.2178548127412796 (nan) | > loss: 0.001001231255941093 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13636337970950274) | > loader_time: 0.0044 (0.004651434023398651)  --> TIME: 2025-02-25 13:10:09 -- STEP: 17452/36938 -- GLOBAL_STEP: 793150 | > loss_text_ce: 0.03497409075498581 (0.034124174432791844) | > loss_mel_ce: 2.6517529487609863 (nan) | > loss: 0.010661615990102291 (nan) | > current_lr: 5e-06 | > step_time: 0.1505 (0.13636959381888286) | > loader_time: 0.0046 (0.004651474469084558)  --> TIME: 2025-02-25 13:10:19 -- STEP: 17502/36938 -- GLOBAL_STEP: 793200 | > loss_text_ce: 0.03735078126192093 (0.03412904878140663) | > loss_mel_ce: 2.3035242557525635 (nan) | > loss: 0.009289187379181385 (nan) | > current_lr: 5e-06 | > step_time: 0.1613 (0.13637026156470325) | > loader_time: 0.0048 (0.004651534517837689)  --> TIME: 2025-02-25 13:10:28 -- STEP: 17552/36938 -- GLOBAL_STEP: 793250 | > loss_text_ce: 0.036673884838819504 (0.034131919582261444) | > loss_mel_ce: 3.0123679637908936 (nan) | > loss: 0.012099373154342175 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13637569006074052) | > loader_time: 0.0046 (0.004651789528843273)  --> TIME: 2025-02-25 13:10:37 -- STEP: 17602/36938 -- GLOBAL_STEP: 793300 | > loss_text_ce: 0.03262465074658394 (0.03413501625228529) | > loss_mel_ce: 4.520089149475098 (nan) | > loss: 0.018066326156258583 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13638785804839182) | > loader_time: 0.0047 (0.0046521057230331075)  --> TIME: 2025-02-25 13:10:47 -- STEP: 17652/36938 -- GLOBAL_STEP: 793350 | > loss_text_ce: 0.031993407756090164 (0.03413220170858726) | > loss_mel_ce: 0.0792715921998024 (nan) | > loss: 0.00044152780901640654 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13639478125108706) | > loader_time: 0.0046 (0.004652327794810429)  --> TIME: 2025-02-25 13:10:56 -- STEP: 17702/36938 -- GLOBAL_STEP: 793400 | > loss_text_ce: 0.03346126899123192 (0.034135876317724065) | > loss_mel_ce: 3.1500039100646973 (nan) | > loss: 0.012632799334824085 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.13638953872487197) | > loader_time: 0.0047 (0.004652177596574405)  --> TIME: 2025-02-25 13:11:05 -- STEP: 17752/36938 -- GLOBAL_STEP: 793450 | > loss_text_ce: 0.03757001459598541 (0.0341376769068932) | > loss_mel_ce: 3.1492700576782227 (nan) | > loss: 0.012646191753447056 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.1363889360653493) | > loader_time: 0.0045 (0.004652152973663283)  --> TIME: 2025-02-25 13:11:15 -- STEP: 17802/36938 -- GLOBAL_STEP: 793500 | > loss_text_ce: 0.04164355620741844 (0.03413920991556728) | > loss_mel_ce: 2.6176397800445557 (nan) | > loss: 0.010552712716162205 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13637899594445738) | > loader_time: 0.0049 (0.004652329916740296)  --> TIME: 2025-02-25 13:11:24 -- STEP: 17852/36938 -- GLOBAL_STEP: 793550 | > loss_text_ce: 0.03969096019864082 (0.03413870936900676) | > loss_mel_ce: 3.533231496810913 (nan) | > loss: 0.014178264886140823 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13637787906305732) | > loader_time: 0.0045 (0.0046523633009645455)  --> TIME: 2025-02-25 13:11:33 -- STEP: 17902/36938 -- GLOBAL_STEP: 793600 | > loss_text_ce: 0.036817312240600586 (0.03413489056473738) | > loss_mel_ce: 3.0879857540130615 (nan) | > loss: 0.012400012463331223 (nan) | > current_lr: 5e-06 | > step_time: 0.1755 (0.13638827033901177) | > loader_time: 0.0048 (0.004652325207528401)  --> TIME: 2025-02-25 13:11:43 -- STEP: 17952/36938 -- GLOBAL_STEP: 793650 | > loss_text_ce: 0.035373054444789886 (0.03413421803099265) | > loss_mel_ce: 3.4256787300109863 (nan) | > loss: 0.013734333217144012 (nan) | > current_lr: 5e-06 | > step_time: 0.1777 (0.13640384066604302) | > loader_time: 0.0049 (0.004652310647535244)  --> TIME: 2025-02-25 13:11:52 -- STEP: 18002/36938 -- GLOBAL_STEP: 793700 | > loss_text_ce: 0.03768818825483322 (0.03413330480601618) | > loss_mel_ce: 2.7926833629608154 (nan) | > loss: 0.011231633834540844 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13640855378726413) | > loader_time: 0.0045 (0.004652500046635853)  --> TIME: 2025-02-25 13:12:02 -- STEP: 18052/36938 -- GLOBAL_STEP: 793750 | > loss_text_ce: 0.025073649361729622 (0.03413052951333863) | > loss_mel_ce: 2.707789182662964 (nan) | > loss: 0.010844694450497627 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.1364056718420781) | > loader_time: 0.0046 (0.004652507403407747)  --> TIME: 2025-02-25 13:12:11 -- STEP: 18102/36938 -- GLOBAL_STEP: 793800 | > loss_text_ce: 0.04723268002271652 (0.03413250580326717) | > loss_mel_ce: 2.9471540451049805 (nan) | > loss: 0.011882487684488297 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13640809798027337) | > loader_time: 0.0046 (0.004652513244404694)  --> TIME: 2025-02-25 13:12:21 -- STEP: 18152/36938 -- GLOBAL_STEP: 793850 | > loss_text_ce: 0.0327356718480587 (0.03413266733181218) | > loss_mel_ce: 2.8215785026550293 (nan) | > loss: 0.011326643638312817 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13642704704880707) | > loader_time: 0.0046 (0.004652522993592143)  --> TIME: 2025-02-25 13:12:30 -- STEP: 18202/36938 -- GLOBAL_STEP: 793900 | > loss_text_ce: 0.047391995787620544 (0.03413299652492485) | > loss_mel_ce: 3.077522039413452 (nan) | > loss: 0.012400452978909016 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13641777778376546) | > loader_time: 0.0047 (0.00465239308559166)  --> TIME: 2025-02-25 13:12:39 -- STEP: 18252/36938 -- GLOBAL_STEP: 793950 | > loss_text_ce: 0.03839674964547157 (0.03412981368913596) | > loss_mel_ce: 1.671295166015625 (nan) | > loss: 0.006784492172300816 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13641515189347886) | > loader_time: 0.0046 (0.004652299654737219)  --> TIME: 2025-02-25 13:12:48 -- STEP: 18302/36938 -- GLOBAL_STEP: 794000 | > loss_text_ce: 0.04531283304095268 (0.03412889273210895) | > loss_mel_ce: 2.34529185295105 (nan) | > loss: 0.009486527182161808 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13641617865057537) | > loader_time: 0.0046 (0.004652187975622018)  --> TIME: 2025-02-25 13:12:58 -- STEP: 18352/36938 -- GLOBAL_STEP: 794050 | > loss_text_ce: 0.03906414657831192 (0.03412727422386229) | > loss_mel_ce: 2.099008321762085 (nan) | > loss: 0.008484414778649807 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13640805712552742) | > loader_time: 0.0045 (0.004652055586122295)  --> TIME: 2025-02-25 13:13:07 -- STEP: 18402/36938 -- GLOBAL_STEP: 794100 | > loss_text_ce: 0.022046424448490143 (0.03412934477512777) | > loss_mel_ce: 2.2971808910369873 (nan) | > loss: 0.009203283116221428 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.1364044526545541) | > loader_time: 0.0048 (0.004652017498130385)  --> TIME: 2025-02-25 13:13:16 -- STEP: 18452/36938 -- GLOBAL_STEP: 794150 | > loss_text_ce: 0.04250463843345642 (0.0341334130339213) | > loss_mel_ce: 2.8707330226898193 (nan) | > loss: 0.011560467071831226 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13640237065165797) | > loader_time: 0.0045 (0.00465200246091059)  --> TIME: 2025-02-25 13:13:25 -- STEP: 18502/36938 -- GLOBAL_STEP: 794200 | > loss_text_ce: 0.03432055562734604 (0.03413372326971674) | > loss_mel_ce: 2.762317657470703 (nan) | > loss: 0.011097771115601063 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13639106393285677) | > loader_time: 0.0046 (0.004651879132882866)  --> TIME: 2025-02-25 13:13:35 -- STEP: 18552/36938 -- GLOBAL_STEP: 794250 | > loss_text_ce: 0.03864066302776337 (0.03413276484480294) | > loss_mel_ce: 3.2494823932647705 (nan) | > loss: 0.013048108667135239 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.1364005712560152) | > loader_time: 0.0046 (0.004651882490139178)  --> TIME: 2025-02-25 13:13:44 -- STEP: 18602/36938 -- GLOBAL_STEP: 794300 | > loss_text_ce: 0.03656858578324318 (0.03413431753686105) | > loss_mel_ce: 2.756633996963501 (nan) | > loss: 0.011084138415753841 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13640310364171487) | > loader_time: 0.0047 (0.004651778988294262)  --> TIME: 2025-02-25 13:13:53 -- STEP: 18652/36938 -- GLOBAL_STEP: 794350 | > loss_text_ce: 0.02965681068599224 (0.034134271962339646) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13639410939718724) | > loader_time: 0.0045 (0.004651596010332082)  --> TIME: 2025-02-25 13:14:03 -- STEP: 18702/36938 -- GLOBAL_STEP: 794400 | > loss_text_ce: 0.032002273947000504 (0.03413237460966905) | > loss_mel_ce: 3.6192193031311035 (nan) | > loss: 0.014488975517451763 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13640940392059445) | > loader_time: 0.0045 (0.004651562375393099)  --> TIME: 2025-02-25 13:14:12 -- STEP: 18752/36938 -- GLOBAL_STEP: 794450 | > loss_text_ce: 0.034500155597925186 (0.0341363446443813) | > loss_mel_ce: 3.1849887371063232 (nan) | > loss: 0.012775749899446964 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13642528874373688) | > loader_time: 0.0044 (0.004651437008136793)  --> TIME: 2025-02-25 13:14:22 -- STEP: 18802/36938 -- GLOBAL_STEP: 794500 | > loss_text_ce: 0.0297952052205801 (0.034136314835215564) | > loss_mel_ce: 2.568817615509033 (nan) | > loss: 0.010311956517398357 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1364335596986229) | > loader_time: 0.0045 (0.004651380363848236)  --> TIME: 2025-02-25 13:14:31 -- STEP: 18852/36938 -- GLOBAL_STEP: 794550 | > loss_text_ce: 0.033718280494213104 (0.03413669959574414) | > loss_mel_ce: 2.7233386039733887 (nan) | > loss: 0.010940702632069588 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.1364329707817291) | > loader_time: 0.0045 (0.004651321844768319)  --> TIME: 2025-02-25 13:14:40 -- STEP: 18902/36938 -- GLOBAL_STEP: 794600 | > loss_text_ce: 0.033106446266174316 (0.03413492952498194) | > loss_mel_ce: 2.6978678703308105 (nan) | > loss: 0.010837200097739697 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.13641772628675616) | > loader_time: 0.0044 (0.004651153318441293)  --> TIME: 2025-02-25 13:14:50 -- STEP: 18952/36938 -- GLOBAL_STEP: 794650 | > loss_text_ce: 0.03603335842490196 (0.03413495445188998) | > loss_mel_ce: 2.733567237854004 (nan) | > loss: 0.010990479029715061 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13642601760040624) | > loader_time: 0.0046 (0.004651113495438264)  --> TIME: 2025-02-25 13:14:59 -- STEP: 19002/36938 -- GLOBAL_STEP: 794700 | > loss_text_ce: 0.03648993745446205 (0.03413523207233846) | > loss_mel_ce: 3.3878173828125 (nan) | > loss: 0.013588521629571915 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13642567105700312) | > loader_time: 0.0046 (0.004651038813071575)  --> TIME: 2025-02-25 13:15:08 -- STEP: 19052/36938 -- GLOBAL_STEP: 794750 | > loss_text_ce: 0.0331016443669796 (0.034136365805966656) | > loss_mel_ce: 2.4479641914367676 (nan) | > loss: 0.009845499880611897 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.1364149500060779) | > loader_time: 0.0046 (0.0046512802158262204)  --> TIME: 2025-02-25 13:15:17 -- STEP: 19102/36938 -- GLOBAL_STEP: 794800 | > loss_text_ce: 0.03323614224791527 (0.0341299570871745) | > loss_mel_ce: 2.585033893585205 (nan) | > loss: 0.010389960370957851 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13641476983138387) | > loader_time: 0.0046 (0.004651189312212475)  --> TIME: 2025-02-25 13:15:27 -- STEP: 19152/36938 -- GLOBAL_STEP: 794850 | > loss_text_ce: 0.02892261929810047 (0.03412749244795125) | > loss_mel_ce: 3.0458648204803467 (nan) | > loss: 0.012201538309454918 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.13641462048691538) | > loader_time: 0.0047 (0.004651245006742153)  --> TIME: 2025-02-25 13:15:36 -- STEP: 19202/36938 -- GLOBAL_STEP: 794900 | > loss_text_ce: 0.0405920036137104 (0.03412745390600506) | > loss_mel_ce: 3.113862991333008 (nan) | > loss: 0.01251767948269844 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.13641365703574437) | > loader_time: 0.0049 (0.004651153463831115)  --> TIME: 2025-02-25 13:15:45 -- STEP: 19252/36938 -- GLOBAL_STEP: 794950 | > loss_text_ce: 0.03467169031500816 (0.03412752091767928) | > loss_mel_ce: 2.432905912399292 (nan) | > loss: 0.00979197584092617 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13641136420285815) | > loader_time: 0.0047 (0.004651201445026522)  --> TIME: 2025-02-25 13:15:54 -- STEP: 19302/36938 -- GLOBAL_STEP: 795000 | > loss_text_ce: 0.034112367779016495 (0.034131589439769815) | > loss_mel_ce: 2.9180424213409424 (nan) | > loss: 0.011714900843799114 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1364019610896382) | > loader_time: 0.0046 (0.004651134378019996)  --> TIME: 2025-02-25 13:16:03 -- STEP: 19352/36938 -- GLOBAL_STEP: 795050 | > loss_text_ce: 0.038275107741355896 (0.0341322226921957) | > loss_mel_ce: 3.154477119445801 (nan) | > loss: 0.01266965176910162 (nan) | > current_lr: 5e-06 | > step_time: 0.1779 (0.13639906344457137) | > loader_time: 0.0048 (0.004651092556499676)  --> TIME: 2025-02-25 13:16:12 -- STEP: 19402/36938 -- GLOBAL_STEP: 795100 | > loss_text_ce: 0.03383888676762581 (0.03413078342466958) | > loss_mel_ce: 0.0224609375 (nan) | > loss: 0.00022341201838571578 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13639125085690515) | > loader_time: 0.0045 (0.004651012709184786)  --> TIME: 2025-02-25 13:16:22 -- STEP: 19452/36938 -- GLOBAL_STEP: 795150 | > loss_text_ce: 0.04485369101166725 (0.03412774386354181) | > loss_mel_ce: 2.5837466716766357 (nan) | > loss: 0.010430954396724701 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13638044445450465) | > loader_time: 0.0046 (0.004651027355276474)  --> TIME: 2025-02-25 13:16:31 -- STEP: 19502/36938 -- GLOBAL_STEP: 795200 | > loss_text_ce: 0.029136961326003075 (0.034130280842048846) | > loss_mel_ce: 0.82275390625 (nan) | > loss: 0.0033805195707827806 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1363754355348743) | > loader_time: 0.0045 (0.004651312888701671)  --> TIME: 2025-02-25 13:16:40 -- STEP: 19552/36938 -- GLOBAL_STEP: 795250 | > loss_text_ce: 0.028261227533221245 (0.03412836273089344) | > loss_mel_ce: 2.84590220451355 (nan) | > loss: 0.011405411176383495 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13637188903731337) | > loader_time: 0.0045 (0.004651269404470644)  --> TIME: 2025-02-25 13:16:49 -- STEP: 19602/36938 -- GLOBAL_STEP: 795300 | > loss_text_ce: 0.038572583347558975 (0.03412664315144732) | > loss_mel_ce: 2.8669869899749756 (nan) | > loss: 0.011529998853802681 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13637840595796108) | > loader_time: 0.0046 (0.004651518552087561)  --> TIME: 2025-02-25 13:16:59 -- STEP: 19652/36938 -- GLOBAL_STEP: 795350 | > loss_text_ce: 0.03817982226610184 (0.03412795025873688) | > loss_mel_ce: 3.125548839569092 (nan) | > loss: 0.012554479762911797 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.13637301939071977) | > loader_time: 0.0046 (0.004651614029393449)  --> TIME: 2025-02-25 13:17:08 -- STEP: 19702/36938 -- GLOBAL_STEP: 795400 | > loss_text_ce: 0.033547528088092804 (0.03412629612017639) | > loss_mel_ce: 3.597346305847168 (nan) | > loss: 0.014408309943974018 (nan) | > current_lr: 5e-06 | > step_time: 0.1867 (0.13638112712252548) | > loader_time: 0.0048 (0.00465150817785516)  --> TIME: 2025-02-25 13:17:17 -- STEP: 19752/36938 -- GLOBAL_STEP: 795450 | > loss_text_ce: 0.028507044538855553 (0.03412245802300092) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13637422616059808) | > loader_time: 0.0045 (0.0046514824437148975)  --> TIME: 2025-02-25 13:17:27 -- STEP: 19802/36938 -- GLOBAL_STEP: 795500 | > loss_text_ce: 0.038014933466911316 (0.03411970688502315) | > loss_mel_ce: 2.9884073734283447 (nan) | > loss: 0.01200961321592331 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13638491171103007) | > loader_time: 0.0045 (0.004651491214092224)  --> TIME: 2025-02-25 13:17:36 -- STEP: 19852/36938 -- GLOBAL_STEP: 795550 | > loss_text_ce: 0.03292068466544151 (0.034122074128021734) | > loss_mel_ce: 1.6233224868774414 (nan) | > loss: 0.006572394166141748 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.1363860760248168) | > loader_time: 0.0045 (0.004651375314580725)  --> TIME: 2025-02-25 13:17:45 -- STEP: 19902/36938 -- GLOBAL_STEP: 795600 | > loss_text_ce: 0.044788558036088943 (0.03412146203047862) | > loss_mel_ce: 3.641139030456543 (nan) | > loss: 0.014626697637140751 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13637424999595887) | > loader_time: 0.0045 (0.004651180955827252)  --> TIME: 2025-02-25 13:17:54 -- STEP: 19952/36938 -- GLOBAL_STEP: 795650 | > loss_text_ce: 0.028962664306163788 (0.034120410942495204) | > loss_mel_ce: 2.3173322677612305 (nan) | > loss: 0.00931069441139698 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.1363658555981248) | > loader_time: 0.0046 (0.004651125481631557)  --> TIME: 2025-02-25 13:18:03 -- STEP: 20002/36938 -- GLOBAL_STEP: 795700 | > loss_text_ce: 0.04101285710930824 (0.034120611503428565) | > loss_mel_ce: 1.424285888671875 (nan) | > loss: 0.005814678035676479 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13636035187794399) | > loader_time: 0.0045 (0.00465089020139279)  --> TIME: 2025-02-25 13:18:12 -- STEP: 20052/36938 -- GLOBAL_STEP: 795750 | > loss_text_ce: 0.0293800700455904 (0.03411905209761634) | > loss_mel_ce: 2.675565004348755 (nan) | > loss: 0.010733909904956818 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13634989359363134) | > loader_time: 0.0046 (0.004650774269362757)  --> TIME: 2025-02-25 13:18:21 -- STEP: 20102/36938 -- GLOBAL_STEP: 795800 | > loss_text_ce: 0.0307539664208889 (0.03411770506475922) | > loss_mel_ce: 3.0983049869537354 (nan) | > loss: 0.0124169010668993 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.13633840147647894) | > loader_time: 0.0046 (0.0046506218501743985)  --> TIME: 2025-02-25 13:18:30 -- STEP: 20152/36938 -- GLOBAL_STEP: 795850 | > loss_text_ce: 0.0399722121655941 (0.03411586513546294) | > loss_mel_ce: 1.72509765625 (nan) | > loss: 0.007004245650023222 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1363296951387269) | > loader_time: 0.0045 (0.004650564101916953)  --> TIME: 2025-02-25 13:18:39 -- STEP: 20202/36938 -- GLOBAL_STEP: 795900 | > loss_text_ce: 0.028065498918294907 (0.034113671816175466) | > loss_mel_ce: 2.6708984375 (nan) | > loss: 0.010710175149142742 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13631863720805787) | > loader_time: 0.0044 (0.004650468873972971)  --> TIME: 2025-02-25 13:18:49 -- STEP: 20252/36938 -- GLOBAL_STEP: 795950 | > loss_text_ce: 0.032757919281721115 (0.03411375935442901) | > loss_mel_ce: 3.0433144569396973 (nan) | > loss: 0.012206637300550938 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13631991327173903) | > loader_time: 0.0046 (0.0046503964253101346)  --> TIME: 2025-02-25 13:18:58 -- STEP: 20302/36938 -- GLOBAL_STEP: 796000 | > loss_text_ce: 0.03849907964468002 (0.03411675354120137) | > loss_mel_ce: 2.2875454425811768 (nan) | > loss: 0.009230336174368858 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13632033896697557) | > loader_time: 0.0044 (0.004650304886099534)  --> TIME: 2025-02-25 13:19:07 -- STEP: 20352/36938 -- GLOBAL_STEP: 796050 | > loss_text_ce: 0.025419430807232857 (0.034113291453278294) | > loss_mel_ce: 2.439096689224243 (nan) | > loss: 0.009779826737940311 (nan) | > current_lr: 5e-06 | > step_time: 0.1496 (0.13632550056578613) | > loader_time: 0.0047 (0.004650225687139471)  --> TIME: 2025-02-25 13:19:17 -- STEP: 20402/36938 -- GLOBAL_STEP: 796100 | > loss_text_ce: 0.031239280477166176 (0.03411328520766603) | > loss_mel_ce: 3.114323854446411 (nan) | > loss: 0.012482394464313984 (nan) | > current_lr: 5e-06 | > step_time: 0.1621 (0.13631667155703436) | > loader_time: 0.0047 (0.00465001338674165)  --> TIME: 2025-02-25 13:19:26 -- STEP: 20452/36938 -- GLOBAL_STEP: 796150 | > loss_text_ce: 0.032353468239307404 (0.03411192129868458) | > loss_mel_ce: 0.8015018105506897 (nan) | > loss: 0.0033089497592300177 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13631201289345) | > loader_time: 0.0045 (0.004650014360287256)  --> TIME: 2025-02-25 13:19:35 -- STEP: 20502/36938 -- GLOBAL_STEP: 796200 | > loss_text_ce: 0.042729344218969345 (0.03411223202373792) | > loss_mel_ce: 2.652637243270874 (nan) | > loss: 0.010695899836719036 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13631158142063796) | > loader_time: 0.0046 (0.004649952718332689)  --> TIME: 2025-02-25 13:19:44 -- STEP: 20552/36938 -- GLOBAL_STEP: 796250 | > loss_text_ce: 0.03408624231815338 (0.03411211050319441) | > loss_mel_ce: 2.234628677368164 (nan) | > loss: 0.009002837352454662 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13630670959868746) | > loader_time: 0.0045 (0.004649813929714117)  --> TIME: 2025-02-25 13:19:54 -- STEP: 20602/36938 -- GLOBAL_STEP: 796300 | > loss_text_ce: 0.032869551330804825 (0.03411155406975316) | > loss_mel_ce: 3.2415390014648438 (nan) | > loss: 0.012993685901165009 (nan) | > current_lr: 5e-06 | > step_time: 0.1959 (0.13631131221757165) | > loader_time: 0.0048 (0.004649701910960498)  --> TIME: 2025-02-25 13:20:03 -- STEP: 20652/36938 -- GLOBAL_STEP: 796350 | > loss_text_ce: 0.03424869850277901 (0.03411070760150994) | > loss_mel_ce: 2.2340915203094482 (nan) | > loss: 0.009001350030303001 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13631100248694764) | > loader_time: 0.0044 (0.0046496030182056475)  --> TIME: 2025-02-25 13:20:12 -- STEP: 20702/36938 -- GLOBAL_STEP: 796400 | > loss_text_ce: 0.024616237729787827 (0.034108276650976786) | > loss_mel_ce: 3.0078125 (nan) | > loss: 0.012033447623252869 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13630491743317183) | > loader_time: 0.0045 (0.004649718986388169)  --> TIME: 2025-02-25 13:20:22 -- STEP: 20752/36938 -- GLOBAL_STEP: 796450 | > loss_text_ce: 0.031416911631822586 (0.03411125255882907) | > loss_mel_ce: 1.359440565109253 (nan) | > loss: 0.0055192760191857815 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13631038020782132) | > loader_time: 0.0045 (0.004649700331246779)  --> TIME: 2025-02-25 13:20:31 -- STEP: 20802/36938 -- GLOBAL_STEP: 796500 | > loss_text_ce: 0.03349480405449867 (0.034110125256065646) | > loss_mel_ce: 1.672593593597412 (nan) | > loss: 0.0067701926454901695 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13630901697839343) | > loader_time: 0.0046 (0.004649657914757972)  --> TIME: 2025-02-25 13:20:40 -- STEP: 20852/36938 -- GLOBAL_STEP: 796550 | > loss_text_ce: 0.036016978323459625 (0.03410648923394421) | > loss_mel_ce: 2.519716739654541 (nan) | > loss: 0.010141801089048386 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13631777549096413) | > loader_time: 0.0046 (0.004649575820427469)  --> TIME: 2025-02-25 13:20:50 -- STEP: 20902/36938 -- GLOBAL_STEP: 796600 | > loss_text_ce: 0.029659053310751915 (0.034107957642253474) | > loss_mel_ce: 3.19905686378479 (nan) | > loss: 0.012812365777790546 (nan) | > current_lr: 5e-06 | > step_time: 0.1875 (0.13632662605867465) | > loader_time: 0.0048 (0.004649665638335367)  --> TIME: 2025-02-25 13:20:59 -- STEP: 20952/36938 -- GLOBAL_STEP: 796650 | > loss_text_ce: 0.04400131106376648 (0.034109291028919145) | > loss_mel_ce: 1.1961709260940552 (nan) | > loss: 0.004921318963170052 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13632550645575534) | > loader_time: 0.0045 (0.004649585499113525)  --> TIME: 2025-02-25 13:21:08 -- STEP: 21002/36938 -- GLOBAL_STEP: 796700 | > loss_text_ce: 0.03245273604989052 (0.034106527056849065) | > loss_mel_ce: 2.6305174827575684 (nan) | > loss: 0.01056734286248684 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.1363218085741545) | > loader_time: 0.0046 (0.004649582266546476)  --> TIME: 2025-02-25 13:21:18 -- STEP: 21052/36938 -- GLOBAL_STEP: 796750 | > loss_text_ce: 0.02960929274559021 (0.034102950526945355) | > loss_mel_ce: 3.203063488006592 (nan) | > loss: 0.012828066945075989 (nan) | > current_lr: 5e-06 | > step_time: 0.1516 (0.13633678801132657) | > loader_time: 0.0046 (0.0046495206338381416)  --> TIME: 2025-02-25 13:21:28 -- STEP: 21102/36938 -- GLOBAL_STEP: 796800 | > loss_text_ce: 0.047691039741039276 (0.034108056590252664) | > loss_mel_ce: 3.466655969619751 (nan) | > loss: 0.013945822604000568 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13635132425984503) | > loader_time: 0.0045 (0.004649579078707593)  --> TIME: 2025-02-25 13:21:37 -- STEP: 21152/36938 -- GLOBAL_STEP: 796850 | > loss_text_ce: 0.037839435040950775 (0.03411030399892406) | > loss_mel_ce: 2.9054365158081055 (nan) | > loss: 0.011679667048156261 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13636742273004612) | > loader_time: 0.0045 (0.004649578961409902)  --> TIME: 2025-02-25 13:21:47 -- STEP: 21202/36938 -- GLOBAL_STEP: 796900 | > loss_text_ce: 0.02906234748661518 (0.034111072957928944) | > loss_mel_ce: 2.548828125 (nan) | > loss: 0.010229724459350109 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1363664153454134) | > loader_time: 0.0046 (0.004649572311263204)  --> TIME: 2025-02-25 13:21:56 -- STEP: 21252/36938 -- GLOBAL_STEP: 796950 | > loss_text_ce: 0.03336092084646225 (0.034110153168871074) | > loss_mel_ce: 2.8569958209991455 (nan) | > loss: 0.011469670571386814 (nan) | > current_lr: 5e-06 | > step_time: 0.1811 (0.13636820847629572) | > loader_time: 0.0048 (0.004649617589846114)  --> TIME: 2025-02-25 13:22:05 -- STEP: 21302/36938 -- GLOBAL_STEP: 797000 | > loss_text_ce: 0.03591129556298256 (0.03410895736100216) | > loss_mel_ce: 2.598492383956909 (nan) | > loss: 0.010453983210027218 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.1363659238224353) | > loader_time: 0.0045 (0.004649529691235417)  --> TIME: 2025-02-25 13:22:15 -- STEP: 21352/36938 -- GLOBAL_STEP: 797050 | > loss_text_ce: 0.03327218443155289 (0.03410956737944425) | > loss_mel_ce: 1.4254882335662842 (nan) | > loss: 0.005788731854408979 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13635943103658427) | > loader_time: 0.0045 (0.004649426538250178)  --> TIME: 2025-02-25 13:22:24 -- STEP: 21402/36938 -- GLOBAL_STEP: 797100 | > loss_text_ce: 0.03872723504900932 (0.034112648668115586) | > loss_mel_ce: 2.8596630096435547 (nan) | > loss: 0.011501549743115902 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.13635621131240686) | > loader_time: 0.0046 (0.0046492706291163975)  --> TIME: 2025-02-25 13:22:33 -- STEP: 21452/36938 -- GLOBAL_STEP: 797150 | > loss_text_ce: 0.03607846796512604 (0.034114275538174435) | > loss_mel_ce: 3.705524444580078 (nan) | > loss: 0.014847631566226482 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13635157803068304) | > loader_time: 0.0047 (0.004649208815889594)  --> TIME: 2025-02-25 13:22:42 -- STEP: 21502/36938 -- GLOBAL_STEP: 797200 | > loss_text_ce: 0.03137057647109032 (0.03411091319990032) | > loss_mel_ce: 2.5410983562469482 (nan) | > loss: 0.010208210907876492 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13634227373557628) | > loader_time: 0.0045 (0.004648997388699054)  --> TIME: 2025-02-25 13:22:52 -- STEP: 21552/36938 -- GLOBAL_STEP: 797250 | > loss_text_ce: 0.04132876172661781 (0.03410966576470444) | > loss_mel_ce: 3.968385696411133 (nan) | > loss: 0.015911566093564034 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13634986472997102) | > loader_time: 0.0047 (0.004648967736636457)  --> TIME: 2025-02-25 13:23:01 -- STEP: 21602/36938 -- GLOBAL_STEP: 797300 | > loss_text_ce: 0.02785971388220787 (0.03410662094089025) | > loss_mel_ce: 2.6902318000793457 (nan) | > loss: 0.010786077938973904 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13634378406120962) | > loader_time: 0.0046 (0.00464888376589674)  --> TIME: 2025-02-25 13:23:10 -- STEP: 21652/36938 -- GLOBAL_STEP: 797350 | > loss_text_ce: 0.026381803676486015 (0.03410420483820414) | > loss_mel_ce: 0.6984601616859436 (nan) | > loss: 0.002876357175409794 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.13634370944866847) | > loader_time: 0.0046 (0.004649036883779337)  --> TIME: 2025-02-25 13:23:19 -- STEP: 21702/36938 -- GLOBAL_STEP: 797400 | > loss_text_ce: 0.03301427885890007 (0.03410478814775992) | > loss_mel_ce: 0.5301689505577087 (nan) | > loss: 0.002234854269772768 (nan) | > current_lr: 5e-06 | > step_time: 0.1693 (0.13634132175508812) | > loader_time: 0.0047 (0.00464902650528863)  --> TIME: 2025-02-25 13:23:29 -- STEP: 21752/36938 -- GLOBAL_STEP: 797450 | > loss_text_ce: 0.03802671656012535 (0.03410750044280783) | > loss_mel_ce: 3.1586594581604004 (nan) | > loss: 0.012685263529419899 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.1363538381982348) | > loader_time: 0.0046 (0.004648983182605484)  --> TIME: 2025-02-25 13:23:38 -- STEP: 21802/36938 -- GLOBAL_STEP: 797500 | > loss_text_ce: 0.04902619123458862 (0.034109767946598195) | > loss_mel_ce: 1.1861350536346436 (nan) | > loss: 0.004901433829218149 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13635797653053006) | > loader_time: 0.0045 (0.004648944760952399)  --> TIME: 2025-02-25 13:23:48 -- STEP: 21852/36938 -- GLOBAL_STEP: 797550 | > loss_text_ce: 0.03765368461608887 (0.034109242046974686) | > loss_mel_ce: 3.4848392009735107 (nan) | > loss: 0.013978146947920322 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.1363550844306846) | > loader_time: 0.0046 (0.004648806672163489)  --> TIME: 2025-02-25 13:23:57 -- STEP: 21902/36938 -- GLOBAL_STEP: 797600 | > loss_text_ce: 0.03545548394322395 (0.03410976281930136) | > loss_mel_ce: 3.0015220642089844 (nan) | > loss: 0.012051498517394066 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1363527918610872) | > loader_time: 0.0046 (0.0046487314582861925)  --> TIME: 2025-02-25 13:24:06 -- STEP: 21952/36938 -- GLOBAL_STEP: 797650 | > loss_text_ce: 0.047689903527498245 (0.03410645707067657) | > loss_mel_ce: 2.2394704818725586 (nan) | > loss: 0.009076033718883991 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13633533765944703) | > loader_time: 0.0045 (0.004648549531100474)  --> TIME: 2025-02-25 13:24:15 -- STEP: 22002/36938 -- GLOBAL_STEP: 797700 | > loss_text_ce: 0.03649836778640747 (0.03410474944814196) | > loss_mel_ce: 2.721879482269287 (nan) | > loss: 0.010945944115519524 (nan) | > current_lr: 5e-06 | > step_time: 0.1632 (0.13633295971960824) | > loader_time: 0.0047 (0.004648410215257344)  --> TIME: 2025-02-25 13:24:24 -- STEP: 22052/36938 -- GLOBAL_STEP: 797750 | > loss_text_ce: 0.03639694303274155 (0.03410584585419004) | > loss_mel_ce: 2.882841110229492 (nan) | > loss: 0.011584279127418995 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13634000167469548) | > loader_time: 0.0046 (0.004648313545259603)  --> TIME: 2025-02-25 13:24:34 -- STEP: 22102/36938 -- GLOBAL_STEP: 797800 | > loss_text_ce: 0.03069494478404522 (0.03410646355699203) | > loss_mel_ce: 3.010157346725464 (nan) | > loss: 0.012066874653100967 (nan) | > current_lr: 5e-06 | > step_time: 0.1851 (0.1363346050059823) | > loader_time: 0.0048 (0.004648186396542839)  --> TIME: 2025-02-25 13:24:43 -- STEP: 22152/36938 -- GLOBAL_STEP: 797850 | > loss_text_ce: 0.025300247594714165 (0.03410802475467173) | > loss_mel_ce: 2.92136812210083 (nan) | > loss: 0.011693129315972328 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13633733030991135) | > loader_time: 0.0045 (0.004648062361840898)  --> TIME: 2025-02-25 13:24:52 -- STEP: 22202/36938 -- GLOBAL_STEP: 797900 | > loss_text_ce: 0.03028172254562378 (0.03410796087053825) | > loss_mel_ce: 3.011221170425415 (nan) | > loss: 0.012069457210600376 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.13634049766783796) | > loader_time: 0.0047 (0.0046480866383136294)  --> TIME: 2025-02-25 13:25:02 -- STEP: 22252/36938 -- GLOBAL_STEP: 797950 | > loss_text_ce: 0.035598572343587875 (0.034109148860545876) | > loss_mel_ce: 2.357496738433838 (nan) | > loss: 0.009496410377323627 (nan) | > current_lr: 5e-06 | > step_time: 0.1626 (0.13633322966478054) | > loader_time: 0.0048 (0.004647938506157125)  --> TIME: 2025-02-25 13:25:11 -- STEP: 22302/36938 -- GLOBAL_STEP: 798000 | > loss_text_ce: 0.043236713856458664 (0.03411098064570104) | > loss_mel_ce: 3.3225741386413574 (nan) | > loss: 0.013356393203139305 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13633025896280862) | > loader_time: 0.0045 (0.004647779257326837)  --> TIME: 2025-02-25 13:25:20 -- STEP: 22352/36938 -- GLOBAL_STEP: 798050 | > loss_text_ce: 0.024348802864551544 (0.034109688272178695) | > loss_mel_ce: 1.5272247791290283 (nan) | > loss: 0.0061570387333631516 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13633159561377364) | > loader_time: 0.0045 (0.004647654949893448)  --> TIME: 2025-02-25 13:25:29 -- STEP: 22402/36938 -- GLOBAL_STEP: 798100 | > loss_text_ce: 0.03792142495512962 (0.03411166575781281) | > loss_mel_ce: 3.0140771865844727 (nan) | > loss: 0.01211110595613718 (nan) | > current_lr: 5e-06 | > step_time: 0.1891 (0.1363294938453317) | > loader_time: 0.0048 (0.00464753555023357)  --> TIME: 2025-02-25 13:25:39 -- STEP: 22452/36938 -- GLOBAL_STEP: 798150 | > loss_text_ce: 0.0328790545463562 (0.03411203092758065) | > loss_mel_ce: 2.711649179458618 (nan) | > loss: 0.01089098583906889 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13634265347340932) | > loader_time: 0.0046 (0.004647430041119682)  --> TIME: 2025-02-25 13:25:48 -- STEP: 22502/36938 -- GLOBAL_STEP: 798200 | > loss_text_ce: 0.04100925475358963 (0.034112240763081635) | > loss_mel_ce: 3.317257881164551 (nan) | > loss: 0.01332645770162344 (nan) | > current_lr: 5e-06 | > step_time: 0.1848 (0.1363388522234336) | > loader_time: 0.0048 (0.004647274280524733)  --> TIME: 2025-02-25 13:25:58 -- STEP: 22552/36938 -- GLOBAL_STEP: 798250 | > loss_text_ce: 0.03631710261106491 (0.03410936141337567) | > loss_mel_ce: 3.49565052986145 (nan) | > loss: 0.014015745371580124 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.136337928503022) | > loader_time: 0.0045 (0.004647193563115596)  --> TIME: 2025-02-25 13:26:07 -- STEP: 22602/36938 -- GLOBAL_STEP: 798300 | > loss_text_ce: 0.03969818726181984 (0.034112656178697305) | > loss_mel_ce: 3.2615280151367188 (nan) | > loss: 0.01310010440647602 (nan) | > current_lr: 5e-06 | > step_time: 0.1573 (0.1363394832243991) | > loader_time: 0.0046 (0.004647065829243405)  --> TIME: 2025-02-25 13:26:16 -- STEP: 22652/36938 -- GLOBAL_STEP: 798350 | > loss_text_ce: 0.02753392793238163 (0.034112577378605395) | > loss_mel_ce: 2.544053077697754 (nan) | > loss: 0.010204710997641087 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13634637915454448) | > loader_time: 0.0045 (0.0046469608991785965)  --> TIME: 2025-02-25 13:26:26 -- STEP: 22702/36938 -- GLOBAL_STEP: 798400 | > loss_text_ce: 0.029952041804790497 (0.03411337123092861) | > loss_mel_ce: 2.922370433807373 (nan) | > loss: 0.01171556580811739 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13634648362756963) | > loader_time: 0.0045 (0.004646867122453582)  --> TIME: 2025-02-25 13:26:35 -- STEP: 22752/36938 -- GLOBAL_STEP: 798450 | > loss_text_ce: 0.030120229348540306 (0.0341120649191125) | > loss_mel_ce: 2.955500602722168 (nan) | > loss: 0.011847701855003834 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.13634716368341868) | > loader_time: 0.0047 (0.004646731579857069)  --> TIME: 2025-02-25 13:26:44 -- STEP: 22802/36938 -- GLOBAL_STEP: 798500 | > loss_text_ce: 0.040775593370199203 (0.03411357791384638) | > loss_mel_ce: 2.219099283218384 (nan) | > loss: 0.008967758156359196 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13634896722762913) | > loader_time: 0.0046 (0.004646671444645569)  --> TIME: 2025-02-25 13:26:54 -- STEP: 22852/36938 -- GLOBAL_STEP: 798550 | > loss_text_ce: 0.023490237072110176 (0.03411319165328522) | > loss_mel_ce: 3.061082124710083 (nan) | > loss: 0.012240367010235786 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1363529512557965) | > loader_time: 0.0045 (0.004646558227836231)  --> TIME: 2025-02-25 13:27:03 -- STEP: 22902/36938 -- GLOBAL_STEP: 798600 | > loss_text_ce: 0.040334682911634445 (0.034115630342820134) | > loss_mel_ce: 2.026186466217041 (nan) | > loss: 0.008200481534004211 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.1363535303495277) | > loader_time: 0.0047 (0.004646491841992597)  --> TIME: 2025-02-25 13:27:12 -- STEP: 22952/36938 -- GLOBAL_STEP: 798650 | > loss_text_ce: 0.03549031540751457 (0.034116590148415414) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13634449769163295) | > loader_time: 0.0045 (0.004646620525221858)  --> TIME: 2025-02-25 13:27:21 -- STEP: 23002/36938 -- GLOBAL_STEP: 798700 | > loss_text_ce: 0.038383059203624725 (0.0341168313463567) | > loss_mel_ce: 2.411137819290161 (nan) | > loss: 0.009720321744680405 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13634019126664826) | > loader_time: 0.0046 (0.0046465005390375486)  --> TIME: 2025-02-25 13:27:30 -- STEP: 23052/36938 -- GLOBAL_STEP: 798750 | > loss_text_ce: 0.034364622086286545 (0.03411746809534425) | > loss_mel_ce: 3.33546781539917 (nan) | > loss: 0.013372352346777916 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.1363260416071917) | > loader_time: 0.0046 (0.004646491294889335)  --> TIME: 2025-02-25 13:27:40 -- STEP: 23102/36938 -- GLOBAL_STEP: 798800 | > loss_text_ce: 0.030215833336114883 (0.034113268521742124) | > loss_mel_ce: 0.03192138671875 (nan) | > loss: 0.0002465762954670936 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13633107700427094) | > loader_time: 0.0046 (0.004646499965440877)  --> TIME: 2025-02-25 13:27:50 -- STEP: 23152/36938 -- GLOBAL_STEP: 798850 | > loss_text_ce: 0.03326723724603653 (0.03411584366115358) | > loss_mel_ce: 2.77116060256958 (nan) | > loss: 0.011128682643175125 (nan) | > current_lr: 5e-06 | > step_time: 0.1651 (0.13635169286808582) | > loader_time: 0.0048 (0.004646520554483913)  --> TIME: 2025-02-25 13:27:59 -- STEP: 23202/36938 -- GLOBAL_STEP: 798900 | > loss_text_ce: 0.029658660292625427 (0.034116019549542) | > loss_mel_ce: 3.274705171585083 (nan) | > loss: 0.013112555257976055 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13634729516905886) | > loader_time: 0.0045 (0.004646550837328479)  --> TIME: 2025-02-25 13:28:08 -- STEP: 23252/36938 -- GLOBAL_STEP: 798950 | > loss_text_ce: 0.030446551740169525 (0.03411798067853129) | > loss_mel_ce: 3.4266765117645264 (nan) | > loss: 0.01371874287724495 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13634866533479534) | > loader_time: 0.0045 (0.00464656062612597)  --> TIME: 2025-02-25 13:28:17 -- STEP: 23302/36938 -- GLOBAL_STEP: 799000 | > loss_text_ce: 0.02561742439866066 (0.03411639768094058) | > loss_mel_ce: 2.4150524139404297 (nan) | > loss: 0.009685197845101357 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13633937073842511) | > loader_time: 0.0044 (0.004646465426585739)  --> TIME: 2025-02-25 13:28:26 -- STEP: 23352/36938 -- GLOBAL_STEP: 799050 | > loss_text_ce: 0.04064634069800377 (0.03411801911339493) | > loss_mel_ce: 3.1434857845306396 (nan) | > loss: 0.012635445222258568 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13633789854655692) | > loader_time: 0.0045 (0.0046464191311310155)  --> TIME: 2025-02-25 13:28:36 -- STEP: 23402/36938 -- GLOBAL_STEP: 799100 | > loss_text_ce: 0.03861192241311073 (0.03411902805938187) | > loss_mel_ce: 3.166734457015991 (nan) | > loss: 0.01271962933242321 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.1363447122559178) | > loader_time: 0.0045 (0.004646377485640592)  --> TIME: 2025-02-25 13:28:45 -- STEP: 23452/36938 -- GLOBAL_STEP: 799150 | > loss_text_ce: 0.038239650428295135 (0.03411928379157487) | > loss_mel_ce: 2.7078754901885986 (nan) | > loss: 0.01089728344231844 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.1363358529033297) | > loader_time: 0.0046 (0.004646276169093279)  --> TIME: 2025-02-25 13:28:54 -- STEP: 23502/36938 -- GLOBAL_STEP: 799200 | > loss_text_ce: 0.031488195061683655 (0.03411835360208178) | > loss_mel_ce: 3.1637370586395264 (nan) | > loss: 0.012679466046392918 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1363385993768258) | > loader_time: 0.0045 (0.004646166762172736)  --> TIME: 2025-02-25 13:29:04 -- STEP: 23552/36938 -- GLOBAL_STEP: 799250 | > loss_text_ce: 0.037199899554252625 (0.03411573104740055) | > loss_mel_ce: 3.0956099033355713 (nan) | > loss: 0.012431786395609379 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.1363454478914319) | > loader_time: 0.0048 (0.004646254612294899)  --> TIME: 2025-02-25 13:29:13 -- STEP: 23602/36938 -- GLOBAL_STEP: 799300 | > loss_text_ce: 0.036327261477708817 (0.03411797617925376) | > loss_mel_ce: 2.8754634857177734 (nan) | > loss: 0.011554726399481297 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13633213811138462) | > loader_time: 0.0047 (0.004646268368535081)  --> TIME: 2025-02-25 13:29:22 -- STEP: 23652/36938 -- GLOBAL_STEP: 799350 | > loss_text_ce: 0.04057405889034271 (0.0341169024707707) | > loss_mel_ce: 2.6486597061157227 (nan) | > loss: 0.010671563446521759 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.1363274994465815) | > loader_time: 0.0045 (0.00464621496224757)  --> TIME: 2025-02-25 13:29:32 -- STEP: 23702/36938 -- GLOBAL_STEP: 799400 | > loss_text_ce: 0.04237339273095131 (0.03411633133284384) | > loss_mel_ce: 2.1802165508270264 (nan) | > loss: 0.008819801732897758 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13634305680897546) | > loader_time: 0.0045 (0.004646269191358603)  --> TIME: 2025-02-25 13:29:41 -- STEP: 23752/36938 -- GLOBAL_STEP: 799450 | > loss_text_ce: 0.038348324596881866 (0.0341159547311333) | > loss_mel_ce: 2.8821747303009033 (nan) | > loss: 0.011589378118515015 (nan) | > current_lr: 5e-06 | > step_time: 0.1622 (0.13634316837518579) | > loader_time: 0.0048 (0.004646352070997631)  --> TIME: 2025-02-25 13:29:51 -- STEP: 23802/36938 -- GLOBAL_STEP: 799500 | > loss_text_ce: 0.03368256241083145 (0.03411732449010569) | > loss_mel_ce: 1.6682902574539185 (nan) | > loss: 0.0067538609728217125 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1363480895038971) | > loader_time: 0.0045 (0.004646266872026839)  --> TIME: 2025-02-25 13:30:00 -- STEP: 23852/36938 -- GLOBAL_STEP: 799550 | > loss_text_ce: 0.04090101644396782 (0.03411681315131945) | > loss_mel_ce: 2.6983273029327393 (nan) | > loss: 0.010869953781366348 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13635369126236546) | > loader_time: 0.0046 (0.004646287975186557)  --> TIME: 2025-02-25 13:30:10 -- STEP: 23902/36938 -- GLOBAL_STEP: 799600 | > loss_text_ce: 0.028651170432567596 (0.03411236237409589) | > loss_mel_ce: 2.885249137878418 (nan) | > loss: 0.01156309712678194 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.13636432503189622) | > loader_time: 0.0046 (0.00464620802474517)  --> TIME: 2025-02-25 13:30:19 -- STEP: 23952/36938 -- GLOBAL_STEP: 799650 | > loss_text_ce: 0.03693204000592232 (0.03410853473743345) | > loss_mel_ce: 2.292736768722534 (nan) | > loss: 0.00924471765756607 (nan) | > current_lr: 5e-06 | > step_time: 0.1074 (0.13635718984729697) | > loader_time: 0.0044 (0.004646188729432102)  --> TIME: 2025-02-25 13:30:28 -- STEP: 24002/36938 -- GLOBAL_STEP: 799700 | > loss_text_ce: 0.03479241952300072 (0.03410685678566678) | > loss_mel_ce: 0.992810070514679 (nan) | > loss: 0.004077787511050701 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13635198693108172) | > loader_time: 0.0045 (0.004646291068053027)  --> TIME: 2025-02-25 13:30:37 -- STEP: 24052/36938 -- GLOBAL_STEP: 799750 | > loss_text_ce: 0.039977822452783585 (0.03410488992381779) | > loss_mel_ce: 3.377110481262207 (nan) | > loss: 0.013559875078499317 (nan) | > current_lr: 5e-06 | > step_time: 0.1814 (0.13635187563990545) | > loader_time: 0.0047 (0.0046463671092203855)  --> TIME: 2025-02-25 13:30:46 -- STEP: 24102/36938 -- GLOBAL_STEP: 799800 | > loss_text_ce: 0.03358578309416771 (0.034107503572401665) | > loss_mel_ce: 3.188053846359253 (nan) | > loss: 0.01278428453952074 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.13635050851807481) | > loader_time: 0.0046 (0.0046463184718343676)  --> TIME: 2025-02-25 13:30:55 -- STEP: 24152/36938 -- GLOBAL_STEP: 799850 | > loss_text_ce: 0.04058384150266647 (0.034105569486694055) | > loss_mel_ce: 2.0388681888580322 (nan) | > loss: 0.0082517946138978 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13633530333011334) | > loader_time: 0.0045 (0.0046461839457149195)  --> TIME: 2025-02-25 13:31:05 -- STEP: 24202/36938 -- GLOBAL_STEP: 799900 | > loss_text_ce: 0.037136856466531754 (0.03410708034629176) | > loss_mel_ce: 1.9475719928741455 (nan) | > loss: 0.007875829003751278 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.13633128035493086) | > loader_time: 0.0049 (0.004646116736979267)  --> TIME: 2025-02-25 13:31:14 -- STEP: 24252/36938 -- GLOBAL_STEP: 799950 | > loss_text_ce: 0.030087510123848915 (0.0341055525028096) | > loss_mel_ce: 3.2901930809020996 (nan) | > loss: 0.013175717554986477 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13632658381047613) | > loader_time: 0.0044 (0.004646037103869103)  --> TIME: 2025-02-25 13:31:23 -- STEP: 24302/36938 -- GLOBAL_STEP: 800000 | > loss_text_ce: 0.0345693975687027 (0.03410506257755916) | > loss_mel_ce: 2.4672904014587402 (nan) | > loss: 0.009928015992045403 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13632680674990766) | > loader_time: 0.0045 (0.004645997914215485) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_800000.pth  --> TIME: 2025-02-25 13:31:38 -- STEP: 24352/36938 -- GLOBAL_STEP: 800050 | > loss_text_ce: 0.03135059401392937 (0.03410493512706745) | > loss_mel_ce: 3.2799742221832275 (nan) | > loss: 0.01314017828553915 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13632271877452015) | > loader_time: 0.0046 (0.004646030424772232)  --> TIME: 2025-02-25 13:31:47 -- STEP: 24402/36938 -- GLOBAL_STEP: 800100 | > loss_text_ce: 0.047842610627412796 (0.03410457175606059) | > loss_mel_ce: 2.8847954273223877 (nan) | > loss: 0.011637452989816666 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.13631461544238496) | > loader_time: 0.0046 (0.004646026133592111)  --> TIME: 2025-02-25 13:31:57 -- STEP: 24452/36938 -- GLOBAL_STEP: 800150 | > loss_text_ce: 0.03945377096533775 (0.03410562411361692) | > loss_mel_ce: 2.7694180011749268 (nan) | > loss: 0.01114631723612547 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13632006527611887) | > loader_time: 0.0046 (0.004646076950136923)  --> TIME: 2025-02-25 13:32:06 -- STEP: 24502/36938 -- GLOBAL_STEP: 800200 | > loss_text_ce: 0.02824532240629196 (0.034105849209660394) | > loss_mel_ce: 1.9624074697494507 (nan) | > loss: 0.007899416610598564 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.136313273978519) | > loader_time: 0.0046 (0.004646076814329086)  --> TIME: 2025-02-25 13:32:16 -- STEP: 24552/36938 -- GLOBAL_STEP: 800250 | > loss_text_ce: 0.028998402878642082 (0.034106123390352225) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.13632836978689508) | > loader_time: 0.0046 (0.00464611874608664)  --> TIME: 2025-02-25 13:32:25 -- STEP: 24602/36938 -- GLOBAL_STEP: 800300 | > loss_text_ce: 0.03222222998738289 (0.034105126873269165) | > loss_mel_ce: 1.8934184312820435 (nan) | > loss: 0.007641431875526905 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13632188616860272) | > loader_time: 0.0046 (0.004646059594768026)  --> TIME: 2025-02-25 13:32:35 -- STEP: 24652/36938 -- GLOBAL_STEP: 800350 | > loss_text_ce: 0.03624768555164337 (0.03410543238176086) | > loss_mel_ce: 2.8492953777313232 (nan) | > loss: 0.011450568214058876 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1363334716594625) | > loader_time: 0.0045 (0.004646110269672501)  --> TIME: 2025-02-25 13:32:44 -- STEP: 24702/36938 -- GLOBAL_STEP: 800400 | > loss_text_ce: 0.03184529393911362 (0.034104520932804104) | > loss_mel_ce: 3.0536980628967285 (nan) | > loss: 0.012244220823049545 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.13633239526959431) | > loader_time: 0.0047 (0.0046460835250917805)  --> TIME: 2025-02-25 13:32:53 -- STEP: 24752/36938 -- GLOBAL_STEP: 800450 | > loss_text_ce: 0.05304303020238876 (0.03410471783778284) | > loss_mel_ce: 2.783569574356079 (nan) | > loss: 0.011256400495767593 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13634117112363137) | > loader_time: 0.0045 (0.004646156582820041)  --> TIME: 2025-02-25 13:33:03 -- STEP: 24802/36938 -- GLOBAL_STEP: 800500 | > loss_text_ce: 0.03288167715072632 (0.034107765348787414) | > loss_mel_ce: 0.07584762573242188 (nan) | > loss: 0.00043146550888195634 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.1363427960942822) | > loader_time: 0.0045 (0.004646190461896362)  --> TIME: 2025-02-25 13:33:12 -- STEP: 24852/36938 -- GLOBAL_STEP: 800550 | > loss_text_ce: 0.031623780727386475 (0.03410787548675422) | > loss_mel_ce: 3.0585806369781494 (nan) | > loss: 0.012262716889381409 (nan) | > current_lr: 5e-06 | > step_time: 0.1844 (0.13634990208701156) | > loader_time: 0.0048 (0.004646142832269336)  --> TIME: 2025-02-25 13:33:22 -- STEP: 24902/36938 -- GLOBAL_STEP: 800600 | > loss_text_ce: 0.0251303780823946 (0.034106893211167166) | > loss_mel_ce: 2.4433774948120117 (nan) | > loss: 0.009795666672289371 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13635073535538678) | > loader_time: 0.0046 (0.00464605504791861)  --> TIME: 2025-02-25 13:33:31 -- STEP: 24952/36938 -- GLOBAL_STEP: 800650 | > loss_text_ce: 0.02956300415098667 (0.034105252476449945) | > loss_mel_ce: 2.8546619415283203 (nan) | > loss: 0.011445337906479836 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13635276783043943) | > loader_time: 0.0046 (0.0046460799258961055)  --> TIME: 2025-02-25 13:33:40 -- STEP: 25002/36938 -- GLOBAL_STEP: 800700 | > loss_text_ce: 0.027921905741095543 (0.03410462966205472) | > loss_mel_ce: 3.026690721511841 (nan) | > loss: 0.012121479026973248 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13635459301729386) | > loader_time: 0.0048 (0.004646084764634878)  --> TIME: 2025-02-25 13:33:49 -- STEP: 25052/36938 -- GLOBAL_STEP: 800750 | > loss_text_ce: 0.034454990178346634 (0.03410696025839499) | > loss_mel_ce: 2.6759912967681885 (nan) | > loss: 0.010755740106105804 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.13634546766565883) | > loader_time: 0.0046 (0.0046460538288855555)  --> TIME: 2025-02-25 13:33:59 -- STEP: 25102/36938 -- GLOBAL_STEP: 800800 | > loss_text_ce: 0.038318198174238205 (0.034107205460025956) | > loss_mel_ce: 3.4444265365600586 (nan) | > loss: 0.013820416294038296 (nan) | > current_lr: 5e-06 | > step_time: 0.1634 (0.13634685416324588) | > loader_time: 0.0046 (0.0046459919959366365)  --> TIME: 2025-02-25 13:34:08 -- STEP: 25152/36938 -- GLOBAL_STEP: 800850 | > loss_text_ce: 0.030967406928539276 (0.034106492814742664) | > loss_mel_ce: 2.037200927734375 (nan) | > loss: 0.008207017555832863 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13634342406390582) | > loader_time: 0.0044 (0.004646181756910197)  --> TIME: 2025-02-25 13:34:17 -- STEP: 25202/36938 -- GLOBAL_STEP: 800900 | > loss_text_ce: 0.03435470536351204 (0.03410411700094155) | > loss_mel_ce: 1.17941415309906 (nan) | > loss: 0.004816543310880661 (nan) | > current_lr: 5e-06 | > step_time: 0.1116 (0.13634290024265666) | > loader_time: 0.0046 (0.004646213023816417)  --> TIME: 2025-02-25 13:34:26 -- STEP: 25252/36938 -- GLOBAL_STEP: 800950 | > loss_text_ce: 0.03936031460762024 (0.03410581557348494) | > loss_mel_ce: 3.4873595237731934 (nan) | > loss: 0.013994920998811722 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13633695866516649) | > loader_time: 0.0049 (0.004646328687856704)  --> TIME: 2025-02-25 13:34:36 -- STEP: 25302/36938 -- GLOBAL_STEP: 801000 | > loss_text_ce: 0.04195959120988846 (0.03410506044714637) | > loss_mel_ce: 2.961460590362549 (nan) | > loss: 0.011918334290385246 (nan) | > current_lr: 5e-06 | > step_time: 0.1727 (0.13633653911410865) | > loader_time: 0.0047 (0.004646275855931254)  --> TIME: 2025-02-25 13:34:45 -- STEP: 25352/36938 -- GLOBAL_STEP: 801050 | > loss_text_ce: 0.04345171898603439 (0.03410733900590289) | > loss_mel_ce: 1.9838765859603882 (nan) | > loss: 0.00804495345801115 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.1363392775744886) | > loader_time: 0.0045 (0.004646333451149207)  --> TIME: 2025-02-25 13:34:55 -- STEP: 25402/36938 -- GLOBAL_STEP: 801100 | > loss_text_ce: 0.03842807561159134 (0.03410786710965944) | > loss_mel_ce: 3.384456157684326 (nan) | > loss: 0.013582875020802021 (nan) | > current_lr: 5e-06 | > step_time: 0.1971 (0.13635936742429236) | > loader_time: 0.0048 (0.0046465338876741195)  --> TIME: 2025-02-25 13:35:04 -- STEP: 25452/36938 -- GLOBAL_STEP: 801150 | > loss_text_ce: 0.03187043219804764 (0.03410598749446948) | > loss_mel_ce: 2.895376443862915 (nan) | > loss: 0.011616059578955173 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13636064207181034) | > loader_time: 0.0046 (0.004646528868785422)  --> TIME: 2025-02-25 13:35:14 -- STEP: 25502/36938 -- GLOBAL_STEP: 801200 | > loss_text_ce: 0.03608511760830879 (0.03410587483332194) | > loss_mel_ce: 3.512397289276123 (nan) | > loss: 0.014081280678510666 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.13635724031750368) | > loader_time: 0.0047 (0.004646447506823036)  --> TIME: 2025-02-25 13:35:23 -- STEP: 25552/36938 -- GLOBAL_STEP: 801250 | > loss_text_ce: 0.027052883058786392 (0.0341056163432458) | > loss_mel_ce: 1.0884805917739868 (nan) | > loss: 0.004426720552146435 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.1363656004761335) | > loader_time: 0.0046 (0.004646390181970235)  --> TIME: 2025-02-25 13:35:32 -- STEP: 25602/36938 -- GLOBAL_STEP: 801300 | > loss_text_ce: 0.03667309507727623 (0.03410540798449961) | > loss_mel_ce: 2.8762152194976807 (nan) | > loss: 0.011559081263840199 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13636171683679552) | > loader_time: 0.0045 (0.004646362238457041)  --> TIME: 2025-02-25 13:35:42 -- STEP: 25652/36938 -- GLOBAL_STEP: 801350 | > loss_text_ce: 0.03421086072921753 (0.034107114167424464) | > loss_mel_ce: 3.059154748916626 (nan) | > loss: 0.012275260873138905 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.13635762932518788) | > loader_time: 0.0046 (0.004646328213842086)  --> TIME: 2025-02-25 13:35:51 -- STEP: 25702/36938 -- GLOBAL_STEP: 801400 | > loss_text_ce: 0.033161938190460205 (0.03410816048312171) | > loss_mel_ce: 1.6848580837249756 (nan) | > loss: 0.006817540153861046 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.1363640749986026) | > loader_time: 0.0046 (0.004646433558355364)  --> TIME: 2025-02-25 13:36:00 -- STEP: 25752/36938 -- GLOBAL_STEP: 801450 | > loss_text_ce: 0.03766970708966255 (0.03410948097255075) | > loss_mel_ce: 3.7767720222473145 (nan) | > loss: 0.015136674046516418 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.13636042963240044) | > loader_time: 0.0046 (0.004646465881302654)  --> TIME: 2025-02-25 13:36:10 -- STEP: 25802/36938 -- GLOBAL_STEP: 801500 | > loss_text_ce: 0.03800297528505325 (0.034108325815869645) | > loss_mel_ce: 3.6157267093658447 (nan) | > loss: 0.014498928561806679 (nan) | > current_lr: 5e-06 | > step_time: 0.1789 (0.1363679331944398) | > loader_time: 0.0048 (0.004646469701972948)  --> TIME: 2025-02-25 13:36:19 -- STEP: 25852/36938 -- GLOBAL_STEP: 801550 | > loss_text_ce: 0.05725536122918129 (0.034110009463473055) | > loss_mel_ce: 2.9176394939422607 (nan) | > loss: 0.011805138550698757 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13637250306785917) | > loader_time: 0.0046 (0.004646501101411678)  --> TIME: 2025-02-25 13:36:29 -- STEP: 25902/36938 -- GLOBAL_STEP: 801600 | > loss_text_ce: 0.02856316789984703 (0.03410943223336211) | > loss_mel_ce: 2.814453125 (nan) | > loss: 0.011281811632215977 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13637552195263197) | > loader_time: 0.0045 (0.004646457886476826)  --> TIME: 2025-02-25 13:36:38 -- STEP: 25952/36938 -- GLOBAL_STEP: 801650 | > loss_text_ce: 0.024479255080223083 (0.03410599168858563) | > loss_mel_ce: 2.1083428859710693 (nan) | > loss: 0.008463580161333084 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13637375476070734) | > loader_time: 0.0045 (0.004646514626231386)  --> TIME: 2025-02-25 13:36:47 -- STEP: 26002/36938 -- GLOBAL_STEP: 801700 | > loss_text_ce: 0.021335376426577568 (0.034104032365038) | > loss_mel_ce: 3.1134140491485596 (nan) | > loss: 0.012439482845366001 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.1363708160261163) | > loader_time: 0.0047 (0.004646447482232705)  --> TIME: 2025-02-25 13:36:56 -- STEP: 26052/36938 -- GLOBAL_STEP: 801750 | > loss_text_ce: 0.034896381199359894 (0.034106843752702724) | > loss_mel_ce: 2.9973113536834717 (nan) | > loss: 0.012032571248710155 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.1363722593685859) | > loader_time: 0.0047 (0.004646491477259982)  --> TIME: 2025-02-25 13:37:06 -- STEP: 26102/36938 -- GLOBAL_STEP: 801800 | > loss_text_ce: 0.024314898997545242 (0.034104856344149836) | > loss_mel_ce: 1.7539124488830566 (nan) | > loss: 0.007056457921862602 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13636961423357855) | > loader_time: 0.0046 (0.004646496081861955)  --> TIME: 2025-02-25 13:37:15 -- STEP: 26152/36938 -- GLOBAL_STEP: 801850 | > loss_text_ce: 0.03287384286522865 (0.03410459437111122) | > loss_mel_ce: 2.463796377182007 (nan) | > loss: 0.009907422587275505 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13636049538002942) | > loader_time: 0.0045 (0.004646412738786664)  --> TIME: 2025-02-25 13:37:24 -- STEP: 26202/36938 -- GLOBAL_STEP: 801900 | > loss_text_ce: 0.019585255533456802 (0.03410183453232048) | > loss_mel_ce: 1.6526211500167847 (nan) | > loss: 0.006635739933699369 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13636659268304324) | > loader_time: 0.0045 (0.00464638060590222)  --> TIME: 2025-02-25 13:37:34 -- STEP: 26252/36938 -- GLOBAL_STEP: 801950 | > loss_text_ce: 0.031993694603443146 (0.03410134821489219) | > loss_mel_ce: 1.7258050441741943 (nan) | > loss: 0.00697539234533906 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13636606061638432) | > loader_time: 0.0044 (0.004646330958329668)  --> TIME: 2025-02-25 13:37:43 -- STEP: 26302/36938 -- GLOBAL_STEP: 802000 | > loss_text_ce: 0.03394370898604393 (0.034102670699475555) | > loss_mel_ce: 2.6969985961914062 (nan) | > loss: 0.010837073437869549 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13636945906759412) | > loader_time: 0.0045 (0.004646291053664103)  --> TIME: 2025-02-25 13:37:53 -- STEP: 26352/36938 -- GLOBAL_STEP: 802050 | > loss_text_ce: 0.03990991786122322 (0.03410201680295326) | > loss_mel_ce: 3.0584826469421387 (nan) | > loss: 0.012295208871364594 (nan) | > current_lr: 5e-06 | > step_time: 0.175 (0.1363759172201591) | > loader_time: 0.0047 (0.004646292023030537)  --> TIME: 2025-02-25 13:38:01 -- STEP: 26402/36938 -- GLOBAL_STEP: 802100 | > loss_text_ce: 0.02487305738031864 (0.034099463331763225) | > loss_mel_ce: 1.955085039138794 (nan) | > loss: 0.007856977172195911 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13636678668118174) | > loader_time: 0.0044 (0.00464618125654302)  --> TIME: 2025-02-25 13:38:11 -- STEP: 26452/36938 -- GLOBAL_STEP: 802150 | > loss_text_ce: 0.034534893929958344 (0.03409955300779841) | > loss_mel_ce: 3.1577212810516357 (nan) | > loss: 0.012667683884501457 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.1363680697183251) | > loader_time: 0.0048 (0.004646135218363785)  --> TIME: 2025-02-25 13:38:20 -- STEP: 26502/36938 -- GLOBAL_STEP: 802200 | > loss_text_ce: 0.033022839576005936 (0.034097067064800146) | > loss_mel_ce: 2.8330743312835693 (nan) | > loss: 0.011373402550816536 (nan) | > current_lr: 5e-06 | > step_time: 0.192 (0.13637073916603018) | > loader_time: 0.0048 (0.004646096343985545)  --> TIME: 2025-02-25 13:38:30 -- STEP: 26552/36938 -- GLOBAL_STEP: 802250 | > loss_text_ce: 0.037110015749931335 (0.034097402021559683) | > loss_mel_ce: 3.4939301013946533 (nan) | > loss: 0.014012064784765244 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13637185718108838) | > loader_time: 0.0045 (0.004646039971676071)  --> TIME: 2025-02-25 13:38:39 -- STEP: 26602/36938 -- GLOBAL_STEP: 802300 | > loss_text_ce: 0.03578030690550804 (0.03409870878245262) | > loss_mel_ce: 2.9083364009857178 (nan) | > loss: 0.011683003045618534 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13637583319938837) | > loader_time: 0.0045 (0.004646001046030704)  --> TIME: 2025-02-25 13:38:48 -- STEP: 26652/36938 -- GLOBAL_STEP: 802350 | > loss_text_ce: 0.0361102819442749 (0.034102889062958656) | > loss_mel_ce: 2.3891091346740723 (nan) | > loss: 0.009623887948691845 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.1363788355082056) | > loader_time: 0.0046 (0.0046460442351236204)  --> TIME: 2025-02-25 13:38:58 -- STEP: 26702/36938 -- GLOBAL_STEP: 802400 | > loss_text_ce: 0.024711983278393745 (0.03410666154875559) | > loss_mel_ce: 2.02414608001709 (nan) | > loss: 0.00813039019703865 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13637215545066586) | > loader_time: 0.0045 (0.0046460048668965906)  --> TIME: 2025-02-25 13:39:07 -- STEP: 26752/36938 -- GLOBAL_STEP: 802450 | > loss_text_ce: 0.042251285165548325 (0.034108310094601496) | > loss_mel_ce: 3.277932643890381 (nan) | > loss: 0.013175333850085735 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13637062700456437) | > loader_time: 0.0045 (0.004645904820216373)  --> TIME: 2025-02-25 13:39:16 -- STEP: 26802/36938 -- GLOBAL_STEP: 802500 | > loss_text_ce: 0.03557826951146126 (0.034105541362907346) | > loss_mel_ce: 1.370507836341858 (nan) | > loss: 0.005579707212746143 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13636996383836386) | > loader_time: 0.0045 (0.004646066168993008)  --> TIME: 2025-02-25 13:39:26 -- STEP: 26852/36938 -- GLOBAL_STEP: 802550 | > loss_text_ce: 0.02979814074933529 (0.03410653266694351) | > loss_mel_ce: 0.9226589202880859 (nan) | > loss: 0.0037795917596668005 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13637406511170402) | > loader_time: 0.0045 (0.004646013981003604)  --> TIME: 2025-02-25 13:39:35 -- STEP: 26902/36938 -- GLOBAL_STEP: 802600 | > loss_text_ce: 0.038746584206819534 (0.034106709950097916) | > loss_mel_ce: 2.527596950531006 (nan) | > loss: 0.010183903388679028 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13637855386886466) | > loader_time: 0.0047 (0.004645898035313366)  --> TIME: 2025-02-25 13:39:44 -- STEP: 26952/36938 -- GLOBAL_STEP: 802650 | > loss_text_ce: 0.030118511989712715 (0.034103219161252744) | > loss_mel_ce: 2.5583243370056152 (nan) | > loss: 0.01027159858494997 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13638575916900322) | > loader_time: 0.0045 (0.0046458760667321755)  --> TIME: 2025-02-25 13:39:54 -- STEP: 27002/36938 -- GLOBAL_STEP: 802700 | > loss_text_ce: 0.04535230994224548 (0.03410229804668945) | > loss_mel_ce: 2.9496285915374756 (nan) | > loss: 0.011884844861924648 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13639570063956988) | > loader_time: 0.0045 (0.004645828970820224)  --> TIME: 2025-02-25 13:40:04 -- STEP: 27052/36938 -- GLOBAL_STEP: 802750 | > loss_text_ce: 0.031373199075460434 (0.03410230456335296) | > loss_mel_ce: 2.9867849349975586 (nan) | > loss: 0.011976818554103374 (nan) | > current_lr: 5e-06 | > step_time: 0.1766 (0.13641149833254207) | > loader_time: 0.0049 (0.004645867741144993)  --> TIME: 2025-02-25 13:40:13 -- STEP: 27102/36938 -- GLOBAL_STEP: 802800 | > loss_text_ce: 0.030352815985679626 (0.03410182130764881) | > loss_mel_ce: 2.9640262126922607 (nan) | > loss: 0.011882456950843334 (nan) | > current_lr: 5e-06 | > step_time: 0.1516 (0.1364045368948597) | > loader_time: 0.0047 (0.00464581756185023)  --> TIME: 2025-02-25 13:40:22 -- STEP: 27152/36938 -- GLOBAL_STEP: 802850 | > loss_text_ce: 0.04149683937430382 (0.03410236740243503) | > loss_mel_ce: 2.926427125930786 (nan) | > loss: 0.011777476407587528 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.13640818249216924) | > loader_time: 0.0049 (0.0046458597403101235)  --> TIME: 2025-02-25 13:40:31 -- STEP: 27202/36938 -- GLOBAL_STEP: 802900 | > loss_text_ce: 0.022287555038928986 (0.03410259505863135) | > loss_mel_ce: 2.5610439777374268 (nan) | > loss: 0.010251316241919994 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.1364024049455161) | > loader_time: 0.0045 (0.004645856809339158)  --> TIME: 2025-02-25 13:40:41 -- STEP: 27252/36938 -- GLOBAL_STEP: 802950 | > loss_text_ce: 0.04654693603515625 (0.034102386029783924) | > loss_mel_ce: 3.1561472415924072 (nan) | > loss: 0.01270910445600748 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1364154179820962) | > loader_time: 0.0046 (0.004645896258900837)  --> TIME: 2025-02-25 13:40:50 -- STEP: 27302/36938 -- GLOBAL_STEP: 803000 | > loss_text_ce: 0.02603505738079548 (0.034100663439117586) | > loss_mel_ce: 2.1907055377960205 (nan) | > loss: 0.008796590380370617 (nan) | > current_lr: 5e-06 | > step_time: 0.1869 (0.1364089884205046) | > loader_time: 0.0048 (0.0046458782516428385)  --> TIME: 2025-02-25 13:40:59 -- STEP: 27352/36938 -- GLOBAL_STEP: 803050 | > loss_text_ce: 0.03391820192337036 (0.03410113982750184) | > loss_mel_ce: 2.6323134899139404 (nan) | > loss: 0.010580284520983696 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13639839075921964) | > loader_time: 0.0045 (0.00464589395659752)  --> TIME: 2025-02-25 13:41:09 -- STEP: 27402/36938 -- GLOBAL_STEP: 803100 | > loss_text_ce: 0.02452189102768898 (0.03410129615422255) | > loss_mel_ce: 2.457247018814087 (nan) | > loss: 0.00984828919172287 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13639580839768395) | > loader_time: 0.0045 (0.004645885781522821)  --> TIME: 2025-02-25 13:41:18 -- STEP: 27452/36938 -- GLOBAL_STEP: 803150 | > loss_text_ce: 0.032295215874910355 (0.03410182382605984) | > loss_mel_ce: 3.019056797027588 (nan) | > loss: 0.012108540162444115 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.13639798701790878) | > loader_time: 0.0047 (0.004645935651531027)  --> TIME: 2025-02-25 13:41:27 -- STEP: 27502/36938 -- GLOBAL_STEP: 803200 | > loss_text_ce: 0.02923765406012535 (0.03410091663897506) | > loss_mel_ce: 3.0213518142700195 (nan) | > loss: 0.012105515226721764 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13639470807214826) | > loader_time: 0.0046 (0.004645945150092299)  --> TIME: 2025-02-25 13:41:37 -- STEP: 27552/36938 -- GLOBAL_STEP: 803250 | > loss_text_ce: 0.03135043755173683 (0.034101943979730764) | > loss_mel_ce: 2.9936366081237793 (nan) | > loss: 0.012003917247056961 (nan) | > current_lr: 5e-06 | > step_time: 0.1979 (0.13639655383033342) | > loader_time: 0.0049 (0.004645993969858623)  --> TIME: 2025-02-25 13:41:46 -- STEP: 27602/36938 -- GLOBAL_STEP: 803300 | > loss_text_ce: 0.03802301734685898 (0.03410321944288292) | > loss_mel_ce: 2.736610174179077 (nan) | > loss: 0.011010449379682541 (nan) | > current_lr: 5e-06 | > step_time: 0.1824 (0.13640700982157858) | > loader_time: 0.0048 (0.004646147975765492)  --> TIME: 2025-02-25 13:41:56 -- STEP: 27652/36938 -- GLOBAL_STEP: 803350 | > loss_text_ce: 0.0384492352604866 (0.03410316515597677) | > loss_mel_ce: 2.772696018218994 (nan) | > loss: 0.01115533895790577 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13641192985308687) | > loader_time: 0.0045 (0.0046462302837016215)  --> TIME: 2025-02-25 13:42:06 -- STEP: 27702/36938 -- GLOBAL_STEP: 803400 | > loss_text_ce: 0.03717730939388275 (0.03410310728958774) | > loss_mel_ce: 2.992755174636841 (nan) | > loss: 0.012023542076349258 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13642714670974573) | > loader_time: 0.0047 (0.004646367290357225)  --> TIME: 2025-02-25 13:42:15 -- STEP: 27752/36938 -- GLOBAL_STEP: 803450 | > loss_text_ce: 0.03704391419887543 (0.03410116481315138) | > loss_mel_ce: 3.1476852893829346 (nan) | > loss: 0.01263781450688839 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.1364220600026454) | > loader_time: 0.0046 (0.004646300324516734)  --> TIME: 2025-02-25 13:42:24 -- STEP: 27802/36938 -- GLOBAL_STEP: 803500 | > loss_text_ce: 0.03657037019729614 (0.0341005667545434) | > loss_mel_ce: 2.3925604820251465 (nan) | > loss: 0.009639408439397812 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13642002934046352) | > loader_time: 0.0046 (0.004646560621127848)  --> TIME: 2025-02-25 13:42:33 -- STEP: 27852/36938 -- GLOBAL_STEP: 803550 | > loss_text_ce: 0.03217913210391998 (0.03410009120625232) | > loss_mel_ce: 2.957200050354004 (nan) | > loss: 0.011862616054713726 (nan) | > current_lr: 5e-06 | > step_time: 0.1626 (0.13641671825889928) | > loader_time: 0.0046 (0.004646549044404614)  --> TIME: 2025-02-25 13:42:42 -- STEP: 27902/36938 -- GLOBAL_STEP: 803600 | > loss_text_ce: 0.029569977894425392 (0.03409969050140953) | > loss_mel_ce: 1.130214810371399 (nan) | > loss: 0.004602320957928896 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13640866422472026) | > loader_time: 0.0046 (0.004646506867317524)  --> TIME: 2025-02-25 13:42:52 -- STEP: 27952/36938 -- GLOBAL_STEP: 803650 | > loss_text_ce: 0.030736509710550308 (0.03409771997398031) | > loss_mel_ce: 1.2625732421875 (nan) | > loss: 0.005132182035595179 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13641300630282832) | > loader_time: 0.0046 (0.004646487683313703)  --> TIME: 2025-02-25 13:43:01 -- STEP: 28002/36938 -- GLOBAL_STEP: 803700 | > loss_text_ce: 0.02465144731104374 (0.0340998816962707) | > loss_mel_ce: 2.2449023723602295 (nan) | > loss: 0.009006166830658913 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13641638019648383) | > loader_time: 0.0045 (0.004646570322709611)  --> TIME: 2025-02-25 13:43:11 -- STEP: 28052/36938 -- GLOBAL_STEP: 803750 | > loss_text_ce: 0.03791191056370735 (0.03409762061542343) | > loss_mel_ce: 2.589792013168335 (nan) | > loss: 0.01042739674448967 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1364200581383061) | > loader_time: 0.0045 (0.00464678339317017)  --> TIME: 2025-02-25 13:43:20 -- STEP: 28102/36938 -- GLOBAL_STEP: 803800 | > loss_text_ce: 0.03805917501449585 (0.0340970603436109) | > loss_mel_ce: 2.881035804748535 (nan) | > loss: 0.01158371102064848 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13642483369776892) | > loader_time: 0.0047 (0.004646818338021848)  --> TIME: 2025-02-25 13:43:30 -- STEP: 28152/36938 -- GLOBAL_STEP: 803850 | > loss_text_ce: 0.03212838992476463 (0.03409588689323417) | > loss_mel_ce: 3.3146586418151855 (nan) | > loss: 0.013280901126563549 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13643304230849065) | > loader_time: 0.0046 (0.004646842860470949)  --> TIME: 2025-02-25 13:43:39 -- STEP: 28202/36938 -- GLOBAL_STEP: 803900 | > loss_text_ce: 0.03549174964427948 (0.034095126531882756) | > loss_mel_ce: 2.7737038135528564 (nan) | > loss: 0.011147601529955864 (nan) | > current_lr: 5e-06 | > step_time: 0.1078 (0.1364382745762948) | > loader_time: 0.0045 (0.004646946061512227)  --> TIME: 2025-02-25 13:43:49 -- STEP: 28252/36938 -- GLOBAL_STEP: 803950 | > loss_text_ce: 0.03372662886977196 (0.03409864693788711) | > loss_mel_ce: 3.254411458969116 (nan) | > loss: 0.013048168271780014 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13643769715437515) | > loader_time: 0.0046 (0.004646946295924988)  --> TIME: 2025-02-25 13:43:58 -- STEP: 28302/36938 -- GLOBAL_STEP: 804000 | > loss_text_ce: 0.03584463894367218 (0.03409631578318859) | > loss_mel_ce: 3.108877658843994 (nan) | > loss: 0.012479057535529137 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1364261483937197) | > loader_time: 0.0046 (0.0046470149920843085)  --> TIME: 2025-02-25 13:44:07 -- STEP: 28352/36938 -- GLOBAL_STEP: 804050 | > loss_text_ce: 0.025389689952135086 (0.034096232643102735) | > loss_mel_ce: 2.2722599506378174 (nan) | > loss: 0.00911765731871128 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1364299556152139) | > loader_time: 0.0045 (0.00464709689231008)  --> TIME: 2025-02-25 13:44:16 -- STEP: 28402/36938 -- GLOBAL_STEP: 804100 | > loss_text_ce: 0.03595886379480362 (0.03409622081497529) | > loss_mel_ce: 2.595371723175049 (nan) | > loss: 0.010441788472235203 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13642882468893275) | > loader_time: 0.0044 (0.004647085225277435)  --> TIME: 2025-02-25 13:44:26 -- STEP: 28452/36938 -- GLOBAL_STEP: 804150 | > loss_text_ce: 0.036194466054439545 (0.03409963792763324) | > loss_mel_ce: 1.7784886360168457 (nan) | > loss: 0.007201123517006636 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.13643332930803084) | > loader_time: 0.0048 (0.004647180725045665)  --> TIME: 2025-02-25 13:44:35 -- STEP: 28502/36938 -- GLOBAL_STEP: 804200 | > loss_text_ce: 0.037436872720718384 (0.034099852718115296) | > loss_mel_ce: 2.11785888671875 (nan) | > loss: 0.008552761748433113 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.1364373612525993) | > loader_time: 0.0046 (0.004647290720855013)  --> TIME: 2025-02-25 13:44:45 -- STEP: 28552/36938 -- GLOBAL_STEP: 804250 | > loss_text_ce: 0.03355426341295242 (0.03410189334748986) | > loss_mel_ce: 2.8840439319610596 (nan) | > loss: 0.011577771045267582 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13644181948878448) | > loader_time: 0.0048 (0.004647369535405581)  --> TIME: 2025-02-25 13:44:54 -- STEP: 28602/36938 -- GLOBAL_STEP: 804300 | > loss_text_ce: 0.03057284839451313 (0.03410240038870585) | > loss_mel_ce: 2.836275100708008 (nan) | > loss: 0.01137638185173273 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13643986859877363) | > loader_time: 0.0046 (0.00464739774325607)  --> TIME: 2025-02-25 13:45:03 -- STEP: 28652/36938 -- GLOBAL_STEP: 804350 | > loss_text_ce: 0.04038872942328453 (0.034100017977348684) | > loss_mel_ce: 2.6090993881225586 (nan) | > loss: 0.010513843037188053 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13644067762318363) | > loader_time: 0.0045 (0.004647329693048302)  --> TIME: 2025-02-25 13:45:12 -- STEP: 28702/36938 -- GLOBAL_STEP: 804400 | > loss_text_ce: 0.03611791133880615 (0.03409990972266476) | > loss_mel_ce: 2.6758110523223877 (nan) | > loss: 0.01076162327080965 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13643601782836368) | > loader_time: 0.0047 (0.004647396979917787)  --> TIME: 2025-02-25 13:45:22 -- STEP: 28752/36938 -- GLOBAL_STEP: 804450 | > loss_text_ce: 0.03962891176342964 (0.03409946987211218) | > loss_mel_ce: 1.7177985906600952 (nan) | > loss: 0.006973918993026018 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13644412387127128) | > loader_time: 0.0045 (0.00464756319140754)  --> TIME: 2025-02-25 13:45:31 -- STEP: 28802/36938 -- GLOBAL_STEP: 804500 | > loss_text_ce: 0.029544338583946228 (0.03409984535100175) | > loss_mel_ce: 1.9867051839828491 (nan) | > loss: 0.008000990375876427 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.13644780227603318) | > loader_time: 0.0044 (0.004647552954654791)  --> TIME: 2025-02-25 13:45:41 -- STEP: 28852/36938 -- GLOBAL_STEP: 804550 | > loss_text_ce: 0.037318117916584015 (0.03409913722669097) | > loss_mel_ce: 2.741706371307373 (nan) | > loss: 0.011027876287698746 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13645019223682361) | > loader_time: 0.0047 (0.004647551578803114)  --> TIME: 2025-02-25 13:45:50 -- STEP: 28902/36938 -- GLOBAL_STEP: 804600 | > loss_text_ce: 0.024615546688437462 (0.03409625026829537) | > loss_mel_ce: 2.9006166458129883 (nan) | > loss: 0.011608065105974674 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13645249006698804) | > loader_time: 0.0045 (0.0046475385763300485)  --> TIME: 2025-02-25 13:45:59 -- STEP: 28952/36938 -- GLOBAL_STEP: 804650 | > loss_text_ce: 0.03914627805352211 (0.03409689016630291) | > loss_mel_ce: 3.076474905014038 (nan) | > loss: 0.012363576330244541 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13644045941340135) | > loader_time: 0.0046 (0.004647445533775661)  --> TIME: 2025-02-25 13:46:09 -- STEP: 29002/36938 -- GLOBAL_STEP: 804700 | > loss_text_ce: 0.02831937186419964 (0.03409743367423752) | > loss_mel_ce: 1.8823140859603882 (nan) | > loss: 0.007581879384815693 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.13644500172850885) | > loader_time: 0.0046 (0.004647406838892375)  --> TIME: 2025-02-25 13:46:18 -- STEP: 29052/36938 -- GLOBAL_STEP: 804750 | > loss_text_ce: 0.03342345729470253 (0.03409770926380235) | > loss_mel_ce: 3.3683104515075684 (nan) | > loss: 0.013498945161700249 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1364445188198794) | > loader_time: 0.0046 (0.004647405641919008)  --> TIME: 2025-02-25 13:46:27 -- STEP: 29102/36938 -- GLOBAL_STEP: 804800 | > loss_text_ce: 0.029381362721323967 (0.034097997488619365) | > loss_mel_ce: 3.7113871574401855 (nan) | > loss: 0.01484431978315115 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13644982935120412) | > loader_time: 0.0046 (0.004647425765983376)  --> TIME: 2025-02-25 13:46:36 -- STEP: 29152/36938 -- GLOBAL_STEP: 804850 | > loss_text_ce: 0.03141060471534729 (0.034098132864338916) | > loss_mel_ce: 3.206721305847168 (nan) | > loss: 0.01284973043948412 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.1364397995427457) | > loader_time: 0.0046 (0.004647362032653763)  --> TIME: 2025-02-25 13:46:46 -- STEP: 29202/36938 -- GLOBAL_STEP: 804900 | > loss_text_ce: 0.03924040123820305 (0.034095614295581576) | > loss_mel_ce: 2.8222804069519043 (nan) | > loss: 0.011355241760611534 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.1364439733713731) | > loader_time: 0.0047 (0.004647348884131609)  --> TIME: 2025-02-25 13:46:55 -- STEP: 29252/36938 -- GLOBAL_STEP: 804950 | > loss_text_ce: 0.034963611513376236 (0.03409572909518652) | > loss_mel_ce: 2.8972816467285156 (nan) | > loss: 0.011635894887149334 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13643583794323427) | > loader_time: 0.0046 (0.004647291547766691)  --> TIME: 2025-02-25 13:47:04 -- STEP: 29302/36938 -- GLOBAL_STEP: 805000 | > loss_text_ce: 0.028154222294688225 (0.034098467494734205) | > loss_mel_ce: 0.23291015625 (nan) | > loss: 0.001035969820804894 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.1364444104423046) | > loader_time: 0.0046 (0.0046472560992445264)  --> TIME: 2025-02-25 13:47:14 -- STEP: 29352/36938 -- GLOBAL_STEP: 805050 | > loss_text_ce: 0.026078445836901665 (0.03409951320714461) | > loss_mel_ce: 2.2718963623046875 (nan) | > loss: 0.009118948131799698 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13644180956767946) | > loader_time: 0.0045 (0.004647190124405697)  --> TIME: 2025-02-25 13:47:23 -- STEP: 29402/36938 -- GLOBAL_STEP: 805100 | > loss_text_ce: 0.03616411238908768 (0.03409902925347361) | > loss_mel_ce: 3.16007661819458 (nan) | > loss: 0.012683495879173279 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13644904788193477) | > loader_time: 0.0049 (0.004647184882745603)  --> TIME: 2025-02-25 13:47:33 -- STEP: 29452/36938 -- GLOBAL_STEP: 805150 | > loss_text_ce: 0.029776975512504578 (0.034096697152262434) | > loss_mel_ce: 3.0800774097442627 (nan) | > loss: 0.012340692803263664 (nan) | > current_lr: 5e-06 | > step_time: 0.179 (0.1364522706259366) | > loader_time: 0.0048 (0.004647132933633635)  --> TIME: 2025-02-25 13:47:41 -- STEP: 29502/36938 -- GLOBAL_STEP: 805200 | > loss_text_ce: 0.03569592535495758 (0.03409630329209958) | > loss_mel_ce: 2.9081130027770996 (nan) | > loss: 0.011681783013045788 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13644140269035243) | > loader_time: 0.0046 (0.0046470201942413625)  --> TIME: 2025-02-25 13:47:51 -- STEP: 29552/36938 -- GLOBAL_STEP: 805250 | > loss_text_ce: 0.035625990480184555 (0.0340951897227434) | > loss_mel_ce: 2.071155548095703 (nan) | > loss: 0.008360244333744049 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.13644065020145008) | > loader_time: 0.0048 (0.004646929345004447)  --> TIME: 2025-02-25 13:48:00 -- STEP: 29602/36938 -- GLOBAL_STEP: 805300 | > loss_text_ce: 0.03246273845434189 (0.03409561518098127) | > loss_mel_ce: 2.7354817390441895 (nan) | > loss: 0.010983907617628574 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13643097179208677) | > loader_time: 0.0046 (0.004646840808150136)  --> TIME: 2025-02-25 13:48:09 -- STEP: 29652/36938 -- GLOBAL_STEP: 805350 | > loss_text_ce: 0.025760168209671974 (0.03409573464424744) | > loss_mel_ce: 2.5985097885131836 (nan) | > loss: 0.01041377056390047 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13642604914204226) | > loader_time: 0.0045 (0.004646769897281373)  --> TIME: 2025-02-25 13:48:18 -- STEP: 29702/36938 -- GLOBAL_STEP: 805400 | > loss_text_ce: 0.03368628770112991 (0.034097158982147165) | > loss_mel_ce: 2.409747838973999 (nan) | > loss: 0.009696167893707752 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.1364220991157758) | > loader_time: 0.0045 (0.004646737634449369)  --> TIME: 2025-02-25 13:48:28 -- STEP: 29752/36938 -- GLOBAL_STEP: 805450 | > loss_text_ce: 0.02262573130428791 (0.03409787592736426) | > loss_mel_ce: 3.3090596199035645 (nan) | > loss: 0.013220974244177341 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13642861463107464) | > loader_time: 0.0047 (0.0046467516940907755)  --> TIME: 2025-02-25 13:48:37 -- STEP: 29802/36938 -- GLOBAL_STEP: 805500 | > loss_text_ce: 0.0372873991727829 (0.03409753367757949) | > loss_mel_ce: 3.064725875854492 (nan) | > loss: 0.012309577316045761 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.13642142359166062) | > loader_time: 0.0047 (0.004646692585764356)  --> TIME: 2025-02-25 13:48:46 -- STEP: 29852/36938 -- GLOBAL_STEP: 805550 | > loss_text_ce: 0.036112383008003235 (0.03409563036939153) | > loss_mel_ce: 2.6771676540374756 (nan) | > loss: 0.010766984894871712 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13642935810053325) | > loader_time: 0.0046 (0.004646677003219267)  --> TIME: 2025-02-25 13:48:56 -- STEP: 29902/36938 -- GLOBAL_STEP: 805600 | > loss_text_ce: 0.035581864416599274 (0.03409576512795075) | > loss_mel_ce: 3.0741279125213623 (nan) | > loss: 0.012340119108557701 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.136433042998968) | > loader_time: 0.0046 (0.004646654990467159)  --> TIME: 2025-02-25 13:49:05 -- STEP: 29952/36938 -- GLOBAL_STEP: 805650 | > loss_text_ce: 0.032606638967990875 (0.034095324471643575) | > loss_mel_ce: 3.0011491775512695 (nan) | > loss: 0.01203871425241232 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.13642966237046503) | > loader_time: 0.0046 (0.004646579727021066)  --> TIME: 2025-02-25 13:49:14 -- STEP: 30002/36938 -- GLOBAL_STEP: 805700 | > loss_text_ce: 0.00966715533286333 (0.03409665577800545) | > loss_mel_ce: 1.7810885906219482 (nan) | > loss: 0.007106174249202013 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13642682987692709) | > loader_time: 0.0045 (0.004646509434809579)  --> TIME: 2025-02-25 13:49:23 -- STEP: 30052/36938 -- GLOBAL_STEP: 805750 | > loss_text_ce: 0.036568257957696915 (0.03409658889805018) | > loss_mel_ce: 3.0505120754241943 (nan) | > loss: 0.012250319123268127 (nan) | > current_lr: 5e-06 | > step_time: 0.1825 (0.13641912380547494) | > loader_time: 0.0048 (0.004646441946965214)  --> TIME: 2025-02-25 13:49:32 -- STEP: 30102/36938 -- GLOBAL_STEP: 805800 | > loss_text_ce: 0.024337751790881157 (0.034095946396489295) | > loss_mel_ce: 3.1056385040283203 (nan) | > loss: 0.012420541606843472 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.13641568161975348) | > loader_time: 0.0046 (0.004646394595094947)  --> TIME: 2025-02-25 13:49:42 -- STEP: 30152/36938 -- GLOBAL_STEP: 805850 | > loss_text_ce: 0.03777315467596054 (0.03409569181228097) | > loss_mel_ce: 2.7750933170318604 (nan) | > loss: 0.011162169277668 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.13641473556045608) | > loader_time: 0.0046 (0.004646316293254424)  --> TIME: 2025-02-25 13:49:51 -- STEP: 30202/36938 -- GLOBAL_STEP: 805900 | > loss_text_ce: 0.0394173189997673 (0.03409543646153302) | > loss_mel_ce: 3.546518564224243 (nan) | > loss: 0.014229904860258102 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.1364091311564593) | > loader_time: 0.0048 (0.0046462410215147324)  --> TIME: 2025-02-25 13:50:00 -- STEP: 30252/36938 -- GLOBAL_STEP: 805950 | > loss_text_ce: 0.03837050497531891 (0.03409533946567178) | > loss_mel_ce: 2.511046886444092 (nan) | > loss: 0.01011673640459776 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13639949957056807) | > loader_time: 0.0048 (0.004646133252683206)  --> TIME: 2025-02-25 13:50:09 -- STEP: 30302/36938 -- GLOBAL_STEP: 806000 | > loss_text_ce: 0.03707889840006828 (0.034092368685598354) | > loss_mel_ce: 0.1431012600660324 (nan) | > loss: 0.0007150006713345647 (nan) | > current_lr: 5e-06 | > step_time: 0.1352 (0.136396258831433) | > loader_time: 0.0047 (0.00464603860152769)  --> TIME: 2025-02-25 13:50:18 -- STEP: 30352/36938 -- GLOBAL_STEP: 806050 | > loss_text_ce: 0.03479050472378731 (0.03409145797980142) | > loss_mel_ce: 2.711503267288208 (nan) | > loss: 0.010897992178797722 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1363991207522217) | > loader_time: 0.0045 (0.00464600109400473)  --> TIME: 2025-02-25 13:50:28 -- STEP: 30402/36938 -- GLOBAL_STEP: 806100 | > loss_text_ce: 0.044190485030412674 (0.03409328115779922) | > loss_mel_ce: 2.5598435401916504 (nan) | > loss: 0.010333468206226826 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.1364054415196336) | > loader_time: 0.0046 (0.004645998944860436)  --> TIME: 2025-02-25 13:50:37 -- STEP: 30452/36938 -- GLOBAL_STEP: 806150 | > loss_text_ce: 0.0426357202231884 (0.034092335722230895) | > loss_mel_ce: 3.275883913040161 (nan) | > loss: 0.013168728910386562 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.136409942359691) | > loader_time: 0.0047 (0.004646018764027872)  --> TIME: 2025-02-25 13:50:47 -- STEP: 30502/36938 -- GLOBAL_STEP: 806200 | > loss_text_ce: 0.03544452786445618 (0.03409222249520832) | > loss_mel_ce: 2.805330514907837 (nan) | > loss: 0.011272917501628399 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.1364174572712207) | > loader_time: 0.0046 (0.004646136841846864)  --> TIME: 2025-02-25 13:50:56 -- STEP: 30552/36938 -- GLOBAL_STEP: 806250 | > loss_text_ce: 0.03287985175848007 (0.03409209520286336) | > loss_mel_ce: 2.6132900714874268 (nan) | > loss: 0.010500675067305565 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13641361123949267) | > loader_time: 0.0047 (0.004646092887385502)  --> TIME: 2025-02-25 13:51:05 -- STEP: 30602/36938 -- GLOBAL_STEP: 806300 | > loss_text_ce: 0.037992123514413834 (0.034094054967459285) | > loss_mel_ce: 2.691762685775757 (nan) | > loss: 0.010832361876964569 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13640911958083743) | > loader_time: 0.0045 (0.004646127032778679)  --> TIME: 2025-02-25 13:51:15 -- STEP: 30652/36938 -- GLOBAL_STEP: 806350 | > loss_text_ce: 0.04345850273966789 (0.03409363546405207) | > loss_mel_ce: 2.817477226257324 (nan) | > loss: 0.011352919973433018 (nan) | > current_lr: 5e-06 | > step_time: 0.1566 (0.13641295486869817) | > loader_time: 0.0047 (0.004646111643844865)  --> TIME: 2025-02-25 13:51:24 -- STEP: 30702/36938 -- GLOBAL_STEP: 806400 | > loss_text_ce: 0.033755477517843246 (0.03409417409645915) | > loss_mel_ce: 2.2285821437835693 (nan) | > loss: 0.00897753145545721 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13640406408043393) | > loader_time: 0.0046 (0.004646070531102484)  --> TIME: 2025-02-25 13:51:33 -- STEP: 30752/36938 -- GLOBAL_STEP: 806450 | > loss_text_ce: 0.03021930903196335 (0.034095110180550496) | > loss_mel_ce: 2.4365651607513428 (nan) | > loss: 0.00978882797062397 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.1364042790573665) | > loader_time: 0.0046 (0.004646086257094551)  --> TIME: 2025-02-25 13:51:42 -- STEP: 30802/36938 -- GLOBAL_STEP: 806500 | > loss_text_ce: 0.03372808173298836 (0.034093611122085414) | > loss_mel_ce: 0.8514232635498047 (nan) | > loss: 0.003512505441904068 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.1364064496969618) | > loader_time: 0.0066 (0.004646138830329245)  --> TIME: 2025-02-25 13:51:52 -- STEP: 30852/36938 -- GLOBAL_STEP: 806550 | > loss_text_ce: 0.0286362636834383 (0.03409424725471767) | > loss_mel_ce: 2.880563735961914 (nan) | > loss: 0.011544444598257542 (nan) | > current_lr: 5e-06 | > step_time: 0.151 (0.13640473194349792) | > loader_time: 0.0047 (0.004646170283050149)  --> TIME: 2025-02-25 13:52:01 -- STEP: 30902/36938 -- GLOBAL_STEP: 806600 | > loss_text_ce: 0.037730488926172256 (0.03409392399513661) | > loss_mel_ce: 2.7886159420013428 (nan) | > loss: 0.011215660721063614 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.1364079648781326) | > loader_time: 0.0048 (0.004646198740746808)  --> TIME: 2025-02-25 13:52:10 -- STEP: 30952/36938 -- GLOBAL_STEP: 806650 | > loss_text_ce: 0.03748540207743645 (0.034094670388771224) | > loss_mel_ce: 0.1062796488404274 (nan) | > loss: 0.0005704962532036006 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.13640408722828667) | > loader_time: 0.0047 (0.004646122340054218)  --> TIME: 2025-02-25 13:52:20 -- STEP: 31002/36938 -- GLOBAL_STEP: 806700 | > loss_text_ce: 0.03757372498512268 (0.03409257066829604) | > loss_mel_ce: 2.347459316253662 (nan) | > loss: 0.009464417584240437 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13641678962420792) | > loader_time: 0.0045 (0.004646106701759803)  --> TIME: 2025-02-25 13:52:29 -- STEP: 31052/36938 -- GLOBAL_STEP: 806750 | > loss_text_ce: 0.03758363798260689 (0.034092099122727496) | > loss_mel_ce: 3.242072343826294 (nan) | > loss: 0.013014508411288261 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.13642006709405785) | > loader_time: 0.0048 (0.004646059157813681)  --> TIME: 2025-02-25 13:52:39 -- STEP: 31102/36938 -- GLOBAL_STEP: 806800 | > loss_text_ce: 0.03465842828154564 (0.034091712839492715) | > loss_mel_ce: 2.275941848754883 (nan) | > loss: 0.009169049561023712 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.1364258244968088) | > loader_time: 0.0046 (0.004646174747164611)  --> TIME: 2025-02-25 13:52:48 -- STEP: 31152/36938 -- GLOBAL_STEP: 806850 | > loss_text_ce: 0.03391202911734581 (0.03409149304541056) | > loss_mel_ce: 2.2613043785095215 (nan) | > loss: 0.009108001366257668 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13641941290239679) | > loader_time: 0.0045 (0.00464608336205233)  --> TIME: 2025-02-25 13:52:57 -- STEP: 31202/36938 -- GLOBAL_STEP: 806900 | > loss_text_ce: 0.03309018909931183 (0.03408975001392572) | > loss_mel_ce: 2.4536054134368896 (nan) | > loss: 0.009867839515209198 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13642072302432728) | > loader_time: 0.0046 (0.004646129122484018)  --> TIME: 2025-02-25 13:53:07 -- STEP: 31252/36938 -- GLOBAL_STEP: 806950 | > loss_text_ce: 0.04837293550372124 (0.03409165884965936) | > loss_mel_ce: 2.5367605686187744 (nan) | > loss: 0.010258466936647892 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13642372992559598) | > loader_time: 0.0047 (0.0046461430460150045)  --> TIME: 2025-02-25 13:53:16 -- STEP: 31302/36938 -- GLOBAL_STEP: 807000 | > loss_text_ce: 0.0371057465672493 (0.03409271112010477) | > loss_mel_ce: 2.6532340049743652 (nan) | > loss: 0.010675951838493347 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13642843375456706) | > loader_time: 0.0047 (0.004646262538508508)  --> TIME: 2025-02-25 13:53:26 -- STEP: 31352/36938 -- GLOBAL_STEP: 807050 | > loss_text_ce: 0.034207459539175034 (0.0340922650904046) | > loss_mel_ce: 1.6234445571899414 (nan) | > loss: 0.006577984429895878 (nan) | > current_lr: 5e-06 | > step_time: 0.1118 (0.13642132141572447) | > loader_time: 0.0045 (0.004646270813215594)  --> TIME: 2025-02-25 13:53:35 -- STEP: 31402/36938 -- GLOBAL_STEP: 807100 | > loss_text_ce: 0.035081058740615845 (0.03409188131073222) | > loss_mel_ce: 2.4914164543151855 (nan) | > loss: 0.010025784373283386 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13642750221577005) | > loader_time: 0.0046 (0.004646390799875993)  --> TIME: 2025-02-25 13:53:44 -- STEP: 31452/36938 -- GLOBAL_STEP: 807150 | > loss_text_ce: 0.03943224251270294 (0.03409217848865002) | > loss_mel_ce: 3.0574088096618652 (nan) | > loss: 0.012289052829146385 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.13642910133862943) | > loader_time: 0.0049 (0.004646417188929459)  --> TIME: 2025-02-25 13:53:54 -- STEP: 31502/36938 -- GLOBAL_STEP: 807200 | > loss_text_ce: 0.03304765745997429 (0.03408981602253228) | > loss_mel_ce: 2.7827320098876953 (nan) | > loss: 0.011173729784786701 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13642215746166972) | > loader_time: 0.0047 (0.004646430658269399)  --> TIME: 2025-02-25 13:54:03 -- STEP: 31552/36938 -- GLOBAL_STEP: 807250 | > loss_text_ce: 0.034803517162799835 (0.03409135256499367) | > loss_mel_ce: 3.1168062686920166 (nan) | > loss: 0.012506389059126377 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13642250670169206) | > loader_time: 0.0045 (0.0046464100359178)  --> TIME: 2025-02-25 13:54:12 -- STEP: 31602/36938 -- GLOBAL_STEP: 807300 | > loss_text_ce: 0.03433928266167641 (0.03409035381285257) | > loss_mel_ce: 3.5317492485046387 (nan) | > loss: 0.014151145704090595 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13641929464259908) | > loader_time: 0.0046 (0.004646439883054277)  --> TIME: 2025-02-25 13:54:22 -- STEP: 31652/36938 -- GLOBAL_STEP: 807350 | > loss_text_ce: 0.03012779727578163 (0.03408929021351371) | > loss_mel_ce: 2.651890516281128 (nan) | > loss: 0.010642929933965206 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.13642557248384632) | > loader_time: 0.0046 (0.0046464532376480905)  --> TIME: 2025-02-25 13:54:31 -- STEP: 31702/36938 -- GLOBAL_STEP: 807400 | > loss_text_ce: 0.03174302354454994 (0.03408839205313689) | > loss_mel_ce: 1.7451859712600708 (nan) | > loss: 0.007051305845379829 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13642338243360166) | > loader_time: 0.0046 (0.00464645414109955)  --> TIME: 2025-02-25 13:54:40 -- STEP: 31752/36938 -- GLOBAL_STEP: 807450 | > loss_text_ce: 0.033514562994241714 (0.034089627067266974) | > loss_mel_ce: 0.017852783203125 (nan) | > loss: 0.0002038386883214116 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.1364050630238829) | > loader_time: 0.0045 (0.004646361564983222)  --> TIME: 2025-02-25 13:54:49 -- STEP: 31802/36938 -- GLOBAL_STEP: 807500 | > loss_text_ce: 0.04284042865037918 (0.03409044268452557) | > loss_mel_ce: 3.2017996311187744 (nan) | > loss: 0.012875556945800781 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13640225336211426) | > loader_time: 0.0045 (0.004646580524095548)  --> TIME: 2025-02-25 13:54:58 -- STEP: 31852/36938 -- GLOBAL_STEP: 807550 | > loss_text_ce: 0.029038218781352043 (0.03408823230122677) | > loss_mel_ce: 2.7815845012664795 (nan) | > loss: 0.0111532649025321 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13639458662508752) | > loader_time: 0.0048 (0.0046465961414861034)  --> TIME: 2025-02-25 13:55:07 -- STEP: 31902/36938 -- GLOBAL_STEP: 807600 | > loss_text_ce: 0.03304394334554672 (0.034087470562724854) | > loss_mel_ce: 2.7970194816589355 (nan) | > loss: 0.011230411008000374 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.13640026922592421) | > loader_time: 0.0049 (0.004646733482611497)  --> TIME: 2025-02-25 13:55:17 -- STEP: 31952/36938 -- GLOBAL_STEP: 807650 | > loss_text_ce: 0.020640920847654343 (0.03408794845400781) | > loss_mel_ce: 2.8611302375793457 (nan) | > loss: 0.011435599997639656 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13639932738791452) | > loader_time: 0.0046 (0.00464687694533804)  --> TIME: 2025-02-25 13:55:26 -- STEP: 32002/36938 -- GLOBAL_STEP: 807700 | > loss_text_ce: 0.032523058354854584 (0.03408927924000501) | > loss_mel_ce: 2.0666496753692627 (nan) | > loss: 0.00833005178719759 (nan) | > current_lr: 5e-06 | > step_time: 0.1527 (0.13641062369756943) | > loader_time: 0.0047 (0.0046470305612851745)  --> TIME: 2025-02-25 13:55:35 -- STEP: 32052/36938 -- GLOBAL_STEP: 807750 | > loss_text_ce: 0.035535600036382675 (0.034088157089414504) | > loss_mel_ce: 3.0440304279327393 (nan) | > loss: 0.012220500968396664 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13640479693084864) | > loader_time: 0.0046 (0.004647083471705993)  --> TIME: 2025-02-25 13:55:45 -- STEP: 32102/36938 -- GLOBAL_STEP: 807800 | > loss_text_ce: 0.03811264783143997 (0.03408825293566991) | > loss_mel_ce: 3.058814764022827 (nan) | > loss: 0.012289395555853844 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13640973230929562) | > loader_time: 0.0048 (0.004647222517970281)  --> TIME: 2025-02-25 13:55:54 -- STEP: 32152/36938 -- GLOBAL_STEP: 807850 | > loss_text_ce: 0.02394803985953331 (0.03408676924674786) | > loss_mel_ce: 0.08935546875 (nan) | > loss: 0.0004496171313803643 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.13640588612424734) | > loader_time: 0.0044 (0.004647405060347911)  --> TIME: 2025-02-25 13:56:03 -- STEP: 32202/36938 -- GLOBAL_STEP: 807900 | > loss_text_ce: 0.038001898676157 (0.03408759968867454) | > loss_mel_ce: 3.1976518630981445 (nan) | > loss: 0.01283989753574133 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.13640445132824439) | > loader_time: 0.0048 (0.004647495657618373)  --> TIME: 2025-02-25 13:56:13 -- STEP: 32252/36938 -- GLOBAL_STEP: 807950 | > loss_text_ce: 0.025744756683707237 (0.034084775086546074) | > loss_mel_ce: 2.6033921241760254 (nan) | > loss: 0.010433083400130272 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13640664108896824) | > loader_time: 0.0046 (0.004647552464390691)  --> TIME: 2025-02-25 13:56:22 -- STEP: 32302/36938 -- GLOBAL_STEP: 808000 | > loss_text_ce: 0.03933651000261307 (0.034085894491458865) | > loss_mel_ce: 2.4683420658111572 (nan) | > loss: 0.00995110534131527 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13640651060150044) | > loader_time: 0.0048 (0.004647551302673393)  --> TIME: 2025-02-25 13:56:32 -- STEP: 32352/36938 -- GLOBAL_STEP: 808050 | > loss_text_ce: 0.030688762664794922 (0.03408667233625894) | > loss_mel_ce: 3.0924439430236816 (nan) | > loss: 0.012393384240567684 (nan) | > current_lr: 5e-06 | > step_time: 0.1711 (0.13641403684011172) | > loader_time: 0.0048 (0.00464754669561351)  --> TIME: 2025-02-25 13:56:41 -- STEP: 32402/36938 -- GLOBAL_STEP: 808100 | > loss_text_ce: 0.042431555688381195 (0.03408911800847044) | > loss_mel_ce: 2.952263116836548 (nan) | > loss: 0.011883709579706192 (nan) | > current_lr: 5e-06 | > step_time: 0.1891 (0.1364149790061532) | > loader_time: 0.0049 (0.00464755877632618)  --> TIME: 2025-02-25 13:56:51 -- STEP: 32452/36938 -- GLOBAL_STEP: 808150 | > loss_text_ce: 0.03640878573060036 (0.03409016538615959) | > loss_mel_ce: 2.615333318710327 (nan) | > loss: 0.01052278745919466 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13642245540632975) | > loader_time: 0.0049 (0.004647754159379852)  --> TIME: 2025-02-25 13:57:00 -- STEP: 32502/36938 -- GLOBAL_STEP: 808200 | > loss_text_ce: 0.03622390329837799 (0.0340901602298123) | > loss_mel_ce: 3.125694990158081 (nan) | > loss: 0.012547297403216362 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13642377532391833) | > loader_time: 0.0046 (0.004647831199103313)  --> TIME: 2025-02-25 13:57:09 -- STEP: 32552/36938 -- GLOBAL_STEP: 808250 | > loss_text_ce: 0.04138711467385292 (0.034093460103869704) | > loss_mel_ce: 0.1064453125 (nan) | > loss: 0.000586636655498296 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1364223669219231) | > loader_time: 0.0046 (0.004647945546201562)  --> TIME: 2025-02-25 13:57:19 -- STEP: 32602/36938 -- GLOBAL_STEP: 808300 | > loss_text_ce: 0.040613483637571335 (0.03409229426729145) | > loss_mel_ce: 3.0997097492218018 (nan) | > loss: 0.012461600825190544 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.13643147464062055) | > loader_time: 0.0047 (0.0046479601149515114)  --> TIME: 2025-02-25 13:57:28 -- STEP: 32652/36938 -- GLOBAL_STEP: 808350 | > loss_text_ce: 0.03776906058192253 (0.03409306959979391) | > loss_mel_ce: 2.9464879035949707 (nan) | > loss: 0.011842289939522743 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.1364309983219596) | > loader_time: 0.0045 (0.004647941021567037)  --> TIME: 2025-02-25 13:57:38 -- STEP: 32702/36938 -- GLOBAL_STEP: 808400 | > loss_text_ce: 0.03829052671790123 (0.034092229964537216) | > loss_mel_ce: 3.711444139480591 (nan) | > loss: 0.01487990003079176 (nan) | > current_lr: 5e-06 | > step_time: 0.1263 (0.1364368574312063) | > loader_time: 0.0046 (0.004647995760576312)  --> TIME: 2025-02-25 13:57:47 -- STEP: 32752/36938 -- GLOBAL_STEP: 808450 | > loss_text_ce: 0.030935341492295265 (0.03409205864470935) | > loss_mel_ce: 0.09616699069738388 (nan) | > loss: 0.0005043743294663727 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.13643720546581686) | > loader_time: 0.0046 (0.004648007310537486)  --> TIME: 2025-02-25 13:57:56 -- STEP: 32802/36938 -- GLOBAL_STEP: 808500 | > loss_text_ce: 0.03509138897061348 (0.034091547046475386) | > loss_mel_ce: 2.9531497955322266 (nan) | > loss: 0.01185810100287199 (nan) | > current_lr: 5e-06 | > step_time: 0.1927 (0.1364372710208833) | > loader_time: 0.0049 (0.004648134276806029)  --> TIME: 2025-02-25 13:58:06 -- STEP: 32852/36938 -- GLOBAL_STEP: 808550 | > loss_text_ce: 0.031168583780527115 (0.03409043279455276) | > loss_mel_ce: 0.02734375 (nan) | > loss: 0.00023219181457534432 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13643922232310027) | > loader_time: 0.0046 (0.004648107051501038)  --> TIME: 2025-02-25 13:58:15 -- STEP: 32902/36938 -- GLOBAL_STEP: 808600 | > loss_text_ce: 0.03443014249205589 (0.03409092606885273) | > loss_mel_ce: 2.6059985160827637 (nan) | > loss: 0.010477891191840172 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13643993132672785) | > loader_time: 0.0045 (0.004648194038558559)  --> TIME: 2025-02-25 13:58:24 -- STEP: 32952/36938 -- GLOBAL_STEP: 808650 | > loss_text_ce: 0.0281230416148901 (0.034091298063910994) | > loss_mel_ce: 2.5291073322296143 (nan) | > loss: 0.010147740133106709 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13643174215293824) | > loader_time: 0.0045 (0.0046481194210214825)  --> TIME: 2025-02-25 13:58:33 -- STEP: 33002/36938 -- GLOBAL_STEP: 808700 | > loss_text_ce: 0.029859988018870354 (0.03409058879005493) | > loss_mel_ce: 2.053157329559326 (nan) | > loss: 0.008265942335128784 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.1364303371283682) | > loader_time: 0.0046 (0.004648257021426921)  --> TIME: 2025-02-25 13:58:42 -- STEP: 33052/36938 -- GLOBAL_STEP: 808750 | > loss_text_ce: 0.03618117794394493 (0.034090832413399684) | > loss_mel_ce: 2.7954957485198975 (nan) | > loss: 0.011236813850700855 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.1364218785098282) | > loader_time: 0.0046 (0.0046482567750814946)  --> TIME: 2025-02-25 13:58:51 -- STEP: 33102/36938 -- GLOBAL_STEP: 808800 | > loss_text_ce: 0.03777690231800079 (0.03409048878573113) | > loss_mel_ce: 2.0429303646087646 (nan) | > loss: 0.008256775327026844 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13641399005757107) | > loader_time: 0.0045 (0.004648391807643301)  --> TIME: 2025-02-25 13:59:01 -- STEP: 33152/36938 -- GLOBAL_STEP: 808850 | > loss_text_ce: 0.03802007436752319 (0.03409040753866638) | > loss_mel_ce: 2.352020263671875 (nan) | > loss: 0.009484288282692432 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13641584607768906) | > loader_time: 0.0048 (0.004648408403037625)  --> TIME: 2025-02-25 13:59:10 -- STEP: 33202/36938 -- GLOBAL_STEP: 808900 | > loss_text_ce: 0.02937481366097927 (0.03408862947476877) | > loss_mel_ce: 0.031005859375 (nan) | > loss: 0.00023960586986504495 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13641540341819283) | > loader_time: 0.0046 (0.0046484120803657715)  --> TIME: 2025-02-25 13:59:19 -- STEP: 33252/36938 -- GLOBAL_STEP: 808950 | > loss_text_ce: 0.027625393122434616 (0.03408935567571709) | > loss_mel_ce: 2.8712215423583984 (nan) | > loss: 0.011503361165523529 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13641508644014036) | > loader_time: 0.0046 (0.004648296515835243)  --> TIME: 2025-02-25 13:59:29 -- STEP: 33302/36938 -- GLOBAL_STEP: 809000 | > loss_text_ce: 0.03154304251074791 (0.03408885931817221) | > loss_mel_ce: 3.1701691150665283 (nan) | > loss: 0.01270520780235529 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13641806864021078) | > loader_time: 0.0046 (0.004648380677311033)  --> TIME: 2025-02-25 13:59:38 -- STEP: 33352/36938 -- GLOBAL_STEP: 809050 | > loss_text_ce: 0.03150935098528862 (0.03408746948300077) | > loss_mel_ce: 2.704935312271118 (nan) | > loss: 0.010858908295631409 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.1364181945080319) | > loader_time: 0.0047 (0.004648396360642589)  --> TIME: 2025-02-25 13:59:48 -- STEP: 33402/36938 -- GLOBAL_STEP: 809100 | > loss_text_ce: 0.021380765363574028 (0.034086154132958256) | > loss_mel_ce: 2.51950740814209 (nan) | > loss: 0.010082889348268509 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13642955458768766) | > loader_time: 0.0045 (0.004648767604877015)  --> TIME: 2025-02-25 13:59:57 -- STEP: 33452/36938 -- GLOBAL_STEP: 809150 | > loss_text_ce: 0.044245678931474686 (0.03408622164004731) | > loss_mel_ce: 0.1511077880859375 (nan) | > loss: 0.0007752121891826391 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13642427513762723) | > loader_time: 0.0045 (0.004648689781523341)  --> TIME: 2025-02-25 14:00:06 -- STEP: 33502/36938 -- GLOBAL_STEP: 809200 | > loss_text_ce: 0.04037175700068474 (0.03408312258581895) | > loss_mel_ce: 2.5864975452423096 (nan) | > loss: 0.010424084961414337 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13641669294612221) | > loader_time: 0.0045 (0.004648631561706997)  --> TIME: 2025-02-25 14:00:15 -- STEP: 33552/36938 -- GLOBAL_STEP: 809250 | > loss_text_ce: 0.04234383627772331 (0.03408135046563907) | > loss_mel_ce: 2.427737236022949 (nan) | > loss: 0.009801909327507019 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13641356602559818) | > loader_time: 0.0046 (0.004648553540607743)  --> TIME: 2025-02-25 14:00:24 -- STEP: 33602/36938 -- GLOBAL_STEP: 809300 | > loss_text_ce: 0.03464820608496666 (0.03408253925673057) | > loss_mel_ce: 2.3290421962738037 (nan) | > loss: 0.00937972404062748 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13641381969012537) | > loader_time: 0.0045 (0.0046485194095039414)  --> TIME: 2025-02-25 14:00:33 -- STEP: 33652/36938 -- GLOBAL_STEP: 809350 | > loss_text_ce: 0.028638694435358047 (0.034081057626865995) | > loss_mel_ce: 1.1078027486801147 (nan) | > loss: 0.004509688820689917 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.136413315990996) | > loader_time: 0.0045 (0.0046485289585912245)  --> TIME: 2025-02-25 14:00:43 -- STEP: 33702/36938 -- GLOBAL_STEP: 809400 | > loss_text_ce: 0.04934631288051605 (0.03408141288840359) | > loss_mel_ce: 2.5686471462249756 (nan) | > loss: 0.010388863272964954 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13641600764177633) | > loader_time: 0.0045 (0.004648476324409126)  --> TIME: 2025-02-25 14:00:52 -- STEP: 33752/36938 -- GLOBAL_STEP: 809450 | > loss_text_ce: 0.030312946066260338 (0.03408195593483865) | > loss_mel_ce: 3.0990076065063477 (nan) | > loss: 0.012417939491569996 (nan) | > current_lr: 5e-06 | > step_time: 0.1821 (0.13641106034800107) | > loader_time: 0.0048 (0.004648404039175905)  --> TIME: 2025-02-25 14:01:02 -- STEP: 33802/36938 -- GLOBAL_STEP: 809500 | > loss_text_ce: 0.02873270772397518 (0.03408042021552803) | > loss_mel_ce: 2.167869806289673 (nan) | > loss: 0.008716677315533161 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13641364124708416) | > loader_time: 0.0045 (0.004648379190215989)  --> TIME: 2025-02-25 14:01:11 -- STEP: 33852/36938 -- GLOBAL_STEP: 809550 | > loss_text_ce: 0.03940790146589279 (0.03407965938165172) | > loss_mel_ce: 2.7608282566070557 (nan) | > loss: 0.011112049221992493 (nan) | > current_lr: 5e-06 | > step_time: 0.181 (0.1364137719289694) | > loader_time: 0.0048 (0.004648325193383314)  --> TIME: 2025-02-25 14:01:20 -- STEP: 33902/36938 -- GLOBAL_STEP: 809600 | > loss_text_ce: 0.017264869064092636 (0.034079977285893265) | > loss_mel_ce: 2.711930274963379 (nan) | > loss: 0.010830139741301537 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.13641026576889845) | > loader_time: 0.0046 (0.004648441973284484)  --> TIME: 2025-02-25 14:01:29 -- STEP: 33952/36938 -- GLOBAL_STEP: 809650 | > loss_text_ce: 0.030855119228363037 (0.03408122873308459) | > loss_mel_ce: 2.989990234375 (nan) | > loss: 0.011987482197582722 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13640455510484842) | > loader_time: 0.0044 (0.0046484798656444976)  --> TIME: 2025-02-25 14:01:38 -- STEP: 34002/36938 -- GLOBAL_STEP: 809700 | > loss_text_ce: 0.02767404168844223 (0.03408115228785315) | > loss_mel_ce: 2.6882286071777344 (nan) | > loss: 0.010777391493320465 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.1364010890423077) | > loader_time: 0.0046 (0.00464844579563712)  --> TIME: 2025-02-25 14:01:47 -- STEP: 34052/36938 -- GLOBAL_STEP: 809750 | > loss_text_ce: 0.043061792850494385 (0.034080971776110025) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13639974996708365) | > loader_time: 0.0045 (0.004648554637375269)  --> TIME: 2025-02-25 14:01:57 -- STEP: 34102/36938 -- GLOBAL_STEP: 809800 | > loss_text_ce: 0.03806526958942413 (0.034079847539441536) | > loss_mel_ce: 3.026245594024658 (nan) | > loss: 0.012159964069724083 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1364025002413498) | > loader_time: 0.0045 (0.004648531338286403)  --> TIME: 2025-02-25 14:02:06 -- STEP: 34152/36938 -- GLOBAL_STEP: 809850 | > loss_text_ce: 0.03475905582308769 (0.03408067336358456) | > loss_mel_ce: 2.79670786857605 (nan) | > loss: 0.011235980316996574 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13640072616460747) | > loader_time: 0.0047 (0.00464849496898622)  --> TIME: 2025-02-25 14:02:16 -- STEP: 34202/36938 -- GLOBAL_STEP: 809900 | > loss_text_ce: 0.03876619413495064 (0.03408200531358952) | > loss_mel_ce: 2.8429813385009766 (nan) | > loss: 0.011435506865382195 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.1364060222499586) | > loader_time: 0.0045 (0.004648476216913121)  --> TIME: 2025-02-25 14:02:25 -- STEP: 34252/36938 -- GLOBAL_STEP: 809950 | > loss_text_ce: 0.03266274183988571 (0.03408138172913207) | > loss_mel_ce: 2.740967273712158 (nan) | > loss: 0.011006468906998634 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.1363999047360659) | > loader_time: 0.0045 (0.004648390202464942)  --> TIME: 2025-02-25 14:02:34 -- STEP: 34302/36938 -- GLOBAL_STEP: 810000 | > loss_text_ce: 0.03347557783126831 (0.0340811091935774) | > loss_mel_ce: 3.282888412475586 (nan) | > loss: 0.013160175643861294 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13639472033579797) | > loader_time: 0.0046 (0.004648404846760818) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_810000.pth  --> TIME: 2025-02-25 14:02:49 -- STEP: 34352/36938 -- GLOBAL_STEP: 810050 | > loss_text_ce: 0.038935545831918716 (0.03407998034355023) | > loss_mel_ce: 2.885371446609497 (nan) | > loss: 0.01160439383238554 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13639299902573926) | > loader_time: 0.0048 (0.0046484689755277855)  --> TIME: 2025-02-25 14:02:58 -- STEP: 34402/36938 -- GLOBAL_STEP: 810100 | > loss_text_ce: 0.02782539464533329 (0.03408115603283457) | > loss_mel_ce: 2.843202590942383 (nan) | > loss: 0.011392968706786633 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13637919194767217) | > loader_time: 0.0046 (0.004648410617650309)  --> TIME: 2025-02-25 14:03:07 -- STEP: 34452/36938 -- GLOBAL_STEP: 810150 | > loss_text_ce: 0.02753959596157074 (0.03408299044548719) | > loss_mel_ce: 3.290498733520508 (nan) | > loss: 0.013166818767786026 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13637678521886099) | > loader_time: 0.0044 (0.004648417355526253)  --> TIME: 2025-02-25 14:03:16 -- STEP: 34502/36938 -- GLOBAL_STEP: 810200 | > loss_text_ce: 0.030452929437160492 (0.03408130756048394) | > loss_mel_ce: 2.573188066482544 (nan) | > loss: 0.01033190917223692 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13638298549926303) | > loader_time: 0.0046 (0.004648426437189989)  --> TIME: 2025-02-25 14:03:26 -- STEP: 34552/36938 -- GLOBAL_STEP: 810250 | > loss_text_ce: 0.037491533905267715 (0.03408027490533355) | > loss_mel_ce: 3.522216320037842 (nan) | > loss: 0.014125825837254524 (nan) | > current_lr: 5e-06 | > step_time: 0.1813 (0.13638837189413824) | > loader_time: 0.0049 (0.004648443524499124)  --> TIME: 2025-02-25 14:03:36 -- STEP: 34602/36938 -- GLOBAL_STEP: 810300 | > loss_text_ce: 0.03953058645129204 (0.03407913504856788) | > loss_mel_ce: 2.878788948059082 (nan) | > loss: 0.011580632999539375 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13639946304561842) | > loader_time: 0.0048 (0.004648498893232007)  --> TIME: 2025-02-25 14:03:45 -- STEP: 34652/36938 -- GLOBAL_STEP: 810350 | > loss_text_ce: 0.03803257644176483 (0.034077548177585716) | > loss_mel_ce: 1.2340855598449707 (nan) | > loss: 0.005048087798058987 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13640217226921889) | > loader_time: 0.0047 (0.00464854317615233)  --> TIME: 2025-02-25 14:03:55 -- STEP: 34702/36938 -- GLOBAL_STEP: 810400 | > loss_text_ce: 0.031186386942863464 (0.03407699305627621) | > loss_mel_ce: 2.5152945518493652 (nan) | > loss: 0.01010508369654417 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1364128644456369) | > loader_time: 0.0045 (0.004648595321804352)  --> TIME: 2025-02-25 14:04:05 -- STEP: 34752/36938 -- GLOBAL_STEP: 810450 | > loss_text_ce: 0.029948431998491287 (0.03407799189475191) | > loss_mel_ce: 2.3819844722747803 (nan) | > loss: 0.009571162983775139 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.136421723857275) | > loader_time: 0.0046 (0.004648626790575794)  --> TIME: 2025-02-25 14:04:14 -- STEP: 34802/36938 -- GLOBAL_STEP: 810500 | > loss_text_ce: 0.03495914116501808 (0.03407725448388924) | > loss_mel_ce: 0.8932444453239441 (nan) | > loss: 0.0036833477206528187 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13641628085398871) | > loader_time: 0.0045 (0.0046485900111626335)  --> TIME: 2025-02-25 14:04:23 -- STEP: 34852/36938 -- GLOBAL_STEP: 810550 | > loss_text_ce: 0.030214982107281685 (0.03407832377925452) | > loss_mel_ce: 2.189502477645874 (nan) | > loss: 0.00880840327590704 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13641097984671874) | > loader_time: 0.0056 (0.004648601428712414)  --> TIME: 2025-02-25 14:04:33 -- STEP: 34902/36938 -- GLOBAL_STEP: 810600 | > loss_text_ce: 0.031331948935985565 (0.034078082803295974) | > loss_mel_ce: 0.7728942632675171 (nan) | > loss: 0.003191374009475112 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13641987155142446) | > loader_time: 0.0044 (0.004648618025667984)  --> TIME: 2025-02-25 14:04:42 -- STEP: 34952/36938 -- GLOBAL_STEP: 810650 | > loss_text_ce: 0.03176826238632202 (0.03407763845127825) | > loss_mel_ce: 2.825061798095703 (nan) | > loss: 0.01133662834763527 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.1364204894557756) | > loader_time: 0.0046 (0.004648677167425347)  --> TIME: 2025-02-25 14:04:51 -- STEP: 35002/36938 -- GLOBAL_STEP: 810700 | > loss_text_ce: 0.030174683779478073 (0.03407640451008017) | > loss_mel_ce: 3.442366361618042 (nan) | > loss: 0.013779926113784313 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1364133467191179) | > loader_time: 0.0044 (0.004648611604278494)  --> TIME: 2025-02-25 14:05:01 -- STEP: 35052/36938 -- GLOBAL_STEP: 810750 | > loss_text_ce: 0.026824695989489555 (0.03407578098141436) | > loss_mel_ce: 2.3972513675689697 (nan) | > loss: 0.009619350545108318 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13642301098826698) | > loader_time: 0.0045 (0.004648608975278251)  --> TIME: 2025-02-25 14:05:10 -- STEP: 35102/36938 -- GLOBAL_STEP: 810800 | > loss_text_ce: 0.0281312745064497 (0.03407476558920737) | > loss_mel_ce: 3.0186290740966797 (nan) | > loss: 0.012090319767594337 (nan) | > current_lr: 5e-06 | > step_time: 0.1467 (0.13641759825733593) | > loader_time: 0.0048 (0.00464866419584554)  --> TIME: 2025-02-25 14:05:19 -- STEP: 35152/36938 -- GLOBAL_STEP: 810850 | > loss_text_ce: 0.04852893576025963 (0.03407478330320526) | > loss_mel_ce: 1.397536277770996 (nan) | > loss: 0.005738354288041592 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.1364152358304819) | > loader_time: 0.0046 (0.004648672046311958)  --> TIME: 2025-02-25 14:05:28 -- STEP: 35202/36938 -- GLOBAL_STEP: 810900 | > loss_text_ce: 0.03406497463583946 (0.03407415868284277) | > loss_mel_ce: 3.342365264892578 (nan) | > loss: 0.013398533686995506 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13641036398660983) | > loader_time: 0.0046 (0.004648700369182609)  --> TIME: 2025-02-25 14:05:37 -- STEP: 35252/36938 -- GLOBAL_STEP: 810950 | > loss_text_ce: 0.033744122833013535 (0.03407455242097235) | > loss_mel_ce: 2.449727773666382 (nan) | > loss: 0.009855047799646854 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13641395804073975) | > loader_time: 0.0045 (0.004648672395459688)  --> TIME: 2025-02-25 14:05:47 -- STEP: 35302/36938 -- GLOBAL_STEP: 811000 | > loss_text_ce: 0.03123980015516281 (0.03407429954546033) | > loss_mel_ce: 2.951781749725342 (nan) | > loss: 0.011837387457489967 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.1364088072721954) | > loader_time: 0.0047 (0.004648670137965267)  --> TIME: 2025-02-25 14:05:56 -- STEP: 35352/36938 -- GLOBAL_STEP: 811050 | > loss_text_ce: 0.031975336372852325 (0.03407457832226584) | > loss_mel_ce: 2.524308443069458 (nan) | > loss: 0.01014398317784071 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.1364099312878) | > loader_time: 0.0045 (0.004648711480930359)  --> TIME: 2025-02-25 14:06:05 -- STEP: 35402/36938 -- GLOBAL_STEP: 811100 | > loss_text_ce: 0.03624056279659271 (0.03407371356765749) | > loss_mel_ce: 2.6199851036071777 (nan) | > loss: 0.01054057851433754 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.13641331637756238) | > loader_time: 0.005 (0.004648686324086665)  --> TIME: 2025-02-25 14:06:15 -- STEP: 35452/36938 -- GLOBAL_STEP: 811150 | > loss_text_ce: 0.03130253404378891 (0.034074038257941204) | > loss_mel_ce: 3.2227230072021484 (nan) | > loss: 0.012912800535559654 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13641324894538268) | > loader_time: 0.0046 (0.004648725193993832)  --> TIME: 2025-02-25 14:06:24 -- STEP: 35502/36938 -- GLOBAL_STEP: 811200 | > loss_text_ce: 0.05142870917916298 (0.034074180611325826) | > loss_mel_ce: 1.3440150022506714 (nan) | > loss: 0.00553747545927763 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13641220060216025) | > loader_time: 0.0046 (0.004648744814846267)  --> TIME: 2025-02-25 14:06:33 -- STEP: 35552/36938 -- GLOBAL_STEP: 811250 | > loss_text_ce: 0.037321969866752625 (0.03407268369543974) | > loss_mel_ce: 3.4349615573883057 (nan) | > loss: 0.013778903521597385 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.136406162869383) | > loader_time: 0.0045 (0.004648707102913785)  --> TIME: 2025-02-25 14:06:42 -- STEP: 35602/36938 -- GLOBAL_STEP: 811300 | > loss_text_ce: 0.036489810794591904 (0.03407141370966162) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13640334796894804) | > loader_time: 0.0046 (0.004648635597833332)  --> TIME: 2025-02-25 14:06:52 -- STEP: 35652/36938 -- GLOBAL_STEP: 811350 | > loss_text_ce: 0.04312610626220703 (0.03407102422725006) | > loss_mel_ce: 2.6110920906066895 (nan) | > loss: 0.010532612912356853 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13640351256283123) | > loader_time: 0.0047 (0.004648720671081213)  --> TIME: 2025-02-25 14:07:02 -- STEP: 35702/36938 -- GLOBAL_STEP: 811400 | > loss_text_ce: 0.03688036650419235 (0.03407015192350767) | > loss_mel_ce: 2.6744425296783447 (nan) | > loss: 0.010759217664599419 (nan) | > current_lr: 5e-06 | > step_time: 0.1502 (0.13642205656306036) | > loader_time: 0.0047 (0.004648763147377324)  --> TIME: 2025-02-25 14:07:11 -- STEP: 35752/36938 -- GLOBAL_STEP: 811450 | > loss_text_ce: 0.028646472841501236 (0.034070379370144205) | > loss_mel_ce: 2.892052173614502 (nan) | > loss: 0.011590074747800827 (nan) | > current_lr: 5e-06 | > step_time: 0.1456 (0.1364213957445657) | > loader_time: 0.0046 (0.004648730635616343)  --> TIME: 2025-02-25 14:07:20 -- STEP: 35802/36938 -- GLOBAL_STEP: 811500 | > loss_text_ce: 0.03321801498532295 (0.03407039117781029) | > loss_mel_ce: 2.1341264247894287 (nan) | > loss: 0.008600573055446148 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13641927179814203) | > loader_time: 0.0045 (0.0046486694328905336)  --> TIME: 2025-02-25 14:07:29 -- STEP: 35852/36938 -- GLOBAL_STEP: 811550 | > loss_text_ce: 0.03281982243061066 (0.03407071875604688) | > loss_mel_ce: 2.625765085220337 (nan) | > loss: 0.010549940168857574 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.13641621568938994) | > loader_time: 0.0046 (0.004648596011779688)  --> TIME: 2025-02-25 14:07:39 -- STEP: 35902/36938 -- GLOBAL_STEP: 811600 | > loss_text_ce: 0.03526369854807854 (0.03406930377170547) | > loss_mel_ce: 3.1896376609802246 (nan) | > loss: 0.012797228991985321 (nan) | > current_lr: 5e-06 | > step_time: 0.1444 (0.13642362763732785) | > loader_time: 0.0046 (0.0046485817722607325)  --> TIME: 2025-02-25 14:07:49 -- STEP: 35952/36938 -- GLOBAL_STEP: 811650 | > loss_text_ce: 0.029624124988913536 (0.034069296838887785) | > loss_mel_ce: 3.0163211822509766 (nan) | > loss: 0.012087085284292698 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13643228243762873) | > loader_time: 0.0046 (0.004648624710045667)  --> TIME: 2025-02-25 14:07:58 -- STEP: 36002/36938 -- GLOBAL_STEP: 811700 | > loss_text_ce: 0.029677744954824448 (0.03406893006245994) | > loss_mel_ce: 1.2380988597869873 (nan) | > loss: 0.005030859727412462 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.1364306968209823) | > loader_time: 0.0046 (0.004648592662668247)  --> TIME: 2025-02-25 14:08:07 -- STEP: 36052/36938 -- GLOBAL_STEP: 811750 | > loss_text_ce: 0.02478334866464138 (0.034068980702741294) | > loss_mel_ce: 1.9223906993865967 (nan) | > loss: 0.00772688165307045 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13643013123572906) | > loader_time: 0.0046 (0.00464851707038216)  --> TIME: 2025-02-25 14:08:17 -- STEP: 36102/36938 -- GLOBAL_STEP: 811800 | > loss_text_ce: 0.04501470923423767 (0.03406949239646737) | > loss_mel_ce: 3.417147636413574 (nan) | > loss: 0.013738740235567093 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1364322899517331) | > loader_time: 0.0046 (0.004648474694407828)  --> TIME: 2025-02-25 14:08:26 -- STEP: 36152/36938 -- GLOBAL_STEP: 811850 | > loss_text_ce: 0.03478052839636803 (0.03406793155059682) | > loss_mel_ce: 2.083531379699707 (nan) | > loss: 0.008406000211834908 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.1364358181274421) | > loader_time: 0.0045 (0.004648403200488049)  --> TIME: 2025-02-25 14:08:35 -- STEP: 36202/36938 -- GLOBAL_STEP: 811900 | > loss_text_ce: 0.030276404693722725 (0.034068139879647664) | > loss_mel_ce: 2.6794073581695557 (nan) | > loss: 0.010752713307738304 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13643199413838655) | > loader_time: 0.0049 (0.004648317421913737)  --> TIME: 2025-02-25 14:08:44 -- STEP: 36252/36938 -- GLOBAL_STEP: 811950 | > loss_text_ce: 0.044282060116529465 (0.03406987285288145) | > loss_mel_ce: 2.1745200157165527 (nan) | > loss: 0.008804770186543465 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13642470191859607) | > loader_time: 0.0046 (0.004648218917278462)  --> TIME: 2025-02-25 14:08:54 -- STEP: 36302/36938 -- GLOBAL_STEP: 812000 | > loss_text_ce: 0.036240704357624054 (0.034070113989309675) | > loss_mel_ce: 2.1993415355682373 (nan) | > loss: 0.008871358819305897 (nan) | > current_lr: 5e-06 | > step_time: 0.1078 (0.13642993198466452) | > loader_time: 0.0045 (0.004648207153579669)  --> TIME: 2025-02-25 14:09:03 -- STEP: 36352/36938 -- GLOBAL_STEP: 812050 | > loss_text_ce: 0.02834174409508705 (0.03407000969888599) | > loss_mel_ce: 2.9527182579040527 (nan) | > loss: 0.011829604394733906 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13643009271997392) | > loader_time: 0.0045 (0.004648248822441417)  --> TIME: 2025-02-25 14:09:12 -- STEP: 36402/36938 -- GLOBAL_STEP: 812100 | > loss_text_ce: 0.025350071489810944 (0.0340696006690856) | > loss_mel_ce: 2.684291362762451 (nan) | > loss: 0.010752545669674873 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13642568216501488) | > loader_time: 0.0045 (0.004648182118478044)  --> TIME: 2025-02-25 14:09:21 -- STEP: 36452/36938 -- GLOBAL_STEP: 812150 | > loss_text_ce: 0.04082879424095154 (0.034069989036366474) | > loss_mel_ce: 2.710031509399414 (nan) | > loss: 0.010916112922132015 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13642309484583334) | > loader_time: 0.0046 (0.004648119986260058)  --> TIME: 2025-02-25 14:09:31 -- STEP: 36502/36938 -- GLOBAL_STEP: 812200 | > loss_text_ce: 0.03594040870666504 (0.034069386059202504) | > loss_mel_ce: 3.085808515548706 (nan) | > loss: 0.012387893162667751 (nan) | > current_lr: 5e-06 | > step_time: 0.1451 (0.1364196210970626) | > loader_time: 0.0046 (0.004648060695706103)  --> TIME: 2025-02-25 14:09:40 -- STEP: 36552/36938 -- GLOBAL_STEP: 812250 | > loss_text_ce: 0.03712683916091919 (0.03406803080144336) | > loss_mel_ce: 1.0135070085525513 (nan) | > loss: 0.004169182386249304 (nan) | > current_lr: 5e-06 | > step_time: 0.1123 (0.13641577009352185) | > loader_time: 0.0045 (0.004648010229537371)  --> TIME: 2025-02-25 14:09:49 -- STEP: 36602/36938 -- GLOBAL_STEP: 812300 | > loss_text_ce: 0.02657039649784565 (0.034068185145281815) | > loss_mel_ce: 3.562328338623047 (nan) | > loss: 0.014241661876440048 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13641726026821152) | > loader_time: 0.0045 (0.00464806299536329)  --> TIME: 2025-02-25 14:09:58 -- STEP: 36652/36938 -- GLOBAL_STEP: 812350 | > loss_text_ce: 0.04035187512636185 (0.034068992413084656) | > loss_mel_ce: 3.3822360038757324 (nan) | > loss: 0.013581698760390282 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13641607935715125) | > loader_time: 0.0046 (0.004647984887709815)  --> TIME: 2025-02-25 14:10:08 -- STEP: 36702/36938 -- GLOBAL_STEP: 812400 | > loss_text_ce: 0.025851108133792877 (0.034068445283357306) | > loss_mel_ce: 2.2977969646453857 (nan) | > loss: 0.009220825508236885 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.13641734265104072) | > loader_time: 0.0048 (0.004647924499767417)  --> TIME: 2025-02-25 14:10:17 -- STEP: 36752/36938 -- GLOBAL_STEP: 812450 | > loss_text_ce: 0.03933704271912575 (0.03406719115916396) | > loss_mel_ce: 2.5914628505706787 (nan) | > loss: 0.010439682751893997 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13641369872810463) | > loader_time: 0.0045 (0.004647908603361407)  --> TIME: 2025-02-25 14:10:26 -- STEP: 36802/36938 -- GLOBAL_STEP: 812500 | > loss_text_ce: 0.044534068554639816 (0.03406679670756977) | > loss_mel_ce: 0.36268284916877747 (nan) | > loss: 0.0016159402439370751 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1364154580960905) | > loader_time: 0.0045 (0.004647903556143767)  --> TIME: 2025-02-25 14:10:36 -- STEP: 36852/36938 -- GLOBAL_STEP: 812550 | > loss_text_ce: 0.042184650897979736 (0.03406510238455241) | > loss_mel_ce: 2.4017629623413086 (nan) | > loss: 0.009698204696178436 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13641401715507942) | > loader_time: 0.0045 (0.004647933051004672)  --> TIME: 2025-02-25 14:10:45 -- STEP: 36902/36938 -- GLOBAL_STEP: 812600 | > loss_text_ce: 0.03762892633676529 (0.034067214668356605) | > loss_mel_ce: 3.13515567779541 (nan) | > loss: 0.01259041577577591 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13641080879065492) | > loader_time: 0.0046 (0.00464784752727557)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.020884183373781715 (+1.9945720634833436e-05) | > avg_loss_text_ce: 0.033252646888394154 (-0.0003065418009415635) | > avg_loss_mel_ce: 2.5374204631489112 (-0.011541600274566743) | > avg_loss: 2.5706731113466907 (-0.01184815227395175) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_812636.pth  > EPOCH: 23/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-25 14:11:16)   --> TIME: 2025-02-25 14:11:19 -- STEP: 14/36938 -- GLOBAL_STEP: 812650 | > loss_text_ce: 0.0306432843208313 (0.03599878813007048) | > loss_mel_ce: 2.3269081115722656 (nan) | > loss: 0.009355363436043262 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13446571145738875) | > loader_time: 0.0042 (0.004478199141366141)  --> TIME: 2025-02-25 14:11:29 -- STEP: 64/36938 -- GLOBAL_STEP: 812700 | > loss_text_ce: 0.03827707841992378 (0.03414063998206984) | > loss_mel_ce: 1.4663512706756592 (nan) | > loss: 0.005970747675746679 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.1385484337806702) | > loader_time: 0.0045 (0.004730597138404847)  --> TIME: 2025-02-25 14:11:37 -- STEP: 114/36938 -- GLOBAL_STEP: 812750 | > loss_text_ce: 0.031251851469278336 (0.03501295160273448) | > loss_mel_ce: 3.169691324234009 (nan) | > loss: 0.01270215678960085 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.13457189108196063) | > loader_time: 0.005 (0.004731703222843639)  --> TIME: 2025-02-25 14:11:47 -- STEP: 164/36938 -- GLOBAL_STEP: 812800 | > loss_text_ce: 0.027425019070506096 (0.03390783600213869) | > loss_mel_ce: 2.183603525161743 (nan) | > loss: 0.008773923851549625 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13585225983363833) | > loader_time: 0.0045 (0.004720218297911853)  --> TIME: 2025-02-25 14:11:56 -- STEP: 214/36938 -- GLOBAL_STEP: 812850 | > loss_text_ce: 0.039500217884778976 (0.03366233060728303) | > loss_mel_ce: 3.1036736965179443 (nan) | > loss: 0.012472913600504398 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13599377703443882) | > loader_time: 0.0045 (0.004700313104647342)  --> TIME: 2025-02-25 14:12:06 -- STEP: 264/36938 -- GLOBAL_STEP: 812900 | > loss_text_ce: 0.029793810099363327 (0.03361503817634938) | > loss_mel_ce: 2.4262266159057617 (nan) | > loss: 0.009746112860739231 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13633239811117015) | > loader_time: 0.0044 (0.004681904207576405)  --> TIME: 2025-02-25 14:12:15 -- STEP: 314/36938 -- GLOBAL_STEP: 812950 | > loss_text_ce: 0.03441238030791283 (0.0336073467086787) | > loss_mel_ce: 2.6373777389526367 (nan) | > loss: 0.010602341964840889 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13604579078164086) | > loader_time: 0.0046 (0.0046630725739108545)  --> TIME: 2025-02-25 14:12:24 -- STEP: 364/36938 -- GLOBAL_STEP: 813000 | > loss_text_ce: 0.02895735204219818 (0.033705237114310754) | > loss_mel_ce: 2.8466238975524902 (nan) | > loss: 0.011411037296056747 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13632288214924593) | > loader_time: 0.0047 (0.004652646216717394)  --> TIME: 2025-02-25 14:12:34 -- STEP: 414/36938 -- GLOBAL_STEP: 813050 | > loss_text_ce: 0.038037125021219254 (0.03369453059653371) | > loss_mel_ce: 2.90948748588562 (nan) | > loss: 0.011696526780724525 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.1370500763833236) | > loader_time: 0.0047 (0.0046532234707892224)  --> TIME: 2025-02-25 14:12:43 -- STEP: 464/36938 -- GLOBAL_STEP: 813100 | > loss_text_ce: 0.028284145519137383 (0.033804152435864) | > loss_mel_ce: 2.617647171020508 (nan) | > loss: 0.010499727912247181 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.1370981551449875) | > loader_time: 0.0046 (0.004647129568560369)  --> TIME: 2025-02-25 14:12:53 -- STEP: 514/36938 -- GLOBAL_STEP: 813150 | > loss_text_ce: 0.03522040322422981 (0.03387275454678834) | > loss_mel_ce: 2.8698647022247314 (nan) | > loss: 0.011528115719556808 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.137439937443121) | > loader_time: 0.0045 (0.004646032236893353)  --> TIME: 2025-02-25 14:13:02 -- STEP: 564/36938 -- GLOBAL_STEP: 813200 | > loss_text_ce: 0.029141904786229134 (0.03390341817982604) | > loss_mel_ce: 2.03702974319458 (nan) | > loss: 0.008199093863368034 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1371501265688146) | > loader_time: 0.0046 (0.004644817917059497)  --> TIME: 2025-02-25 14:13:11 -- STEP: 614/36938 -- GLOBAL_STEP: 813250 | > loss_text_ce: 0.03658817335963249 (0.03386331794947292) | > loss_mel_ce: 3.5970544815063477 (nan) | > loss: 0.014419217593967915 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.13725440471102443) | > loader_time: 0.0047 (0.004643776517737573)  --> TIME: 2025-02-25 14:13:21 -- STEP: 664/36938 -- GLOBAL_STEP: 813300 | > loss_text_ce: 0.03694521635770798 (0.03388839080952202) | > loss_mel_ce: 3.056673765182495 (nan) | > loss: 0.012276265770196915 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.137046668184809) | > loader_time: 0.0046 (0.00464152715292322)  --> TIME: 2025-02-25 14:13:30 -- STEP: 714/36938 -- GLOBAL_STEP: 813350 | > loss_text_ce: 0.04024628922343254 (0.03393168967603306) | > loss_mel_ce: 3.2861409187316895 (nan) | > loss: 0.013199949637055397 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13739462827100146) | > loader_time: 0.0045 (0.004640670049758188)  --> TIME: 2025-02-25 14:13:40 -- STEP: 764/36938 -- GLOBAL_STEP: 813400 | > loss_text_ce: 0.03247673436999321 (0.033899531911824585) | > loss_mel_ce: 2.7357680797576904 (nan) | > loss: 0.010985098779201508 (nan) | > current_lr: 5e-06 | > step_time: 0.1997 (0.137413517030746) | > loader_time: 0.005 (0.004638897810930987)  --> TIME: 2025-02-25 14:13:49 -- STEP: 814/36938 -- GLOBAL_STEP: 813450 | > loss_text_ce: 0.038976240903139114 (0.03401670872442046) | > loss_mel_ce: 0.00933837890625 (nan) | > loss: 0.00019172469910699874 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13726151223850488) | > loader_time: 0.0046 (0.004643979470911429)  --> TIME: 2025-02-25 14:13:58 -- STEP: 864/36938 -- GLOBAL_STEP: 813500 | > loss_text_ce: 0.036163002252578735 (0.034007426512996826) | > loss_mel_ce: 2.1337547302246094 (nan) | > loss: 0.008610785938799381 (nan) | > current_lr: 5e-06 | > step_time: 0.1505 (0.1372274516357315) | > loader_time: 0.0047 (0.004644161573162792)  --> TIME: 2025-02-25 14:14:07 -- STEP: 914/36938 -- GLOBAL_STEP: 813550 | > loss_text_ce: 0.03299066424369812 (0.034056530916896324) | > loss_mel_ce: 3.0402352809906006 (nan) | > loss: 0.012195342220366001 (nan) | > current_lr: 5e-06 | > step_time: 0.1605 (0.1370367933415218) | > loader_time: 0.0047 (0.004646334919418241)  --> TIME: 2025-02-25 14:14:16 -- STEP: 964/36938 -- GLOBAL_STEP: 813600 | > loss_text_ce: 0.03710394352674484 (0.03412527121464375) | > loss_mel_ce: 2.924891948699951 (nan) | > loss: 0.011753952130675316 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.13678367330820232) | > loader_time: 0.0046 (0.0046458612833775)  --> TIME: 2025-02-25 14:14:26 -- STEP: 1014/36938 -- GLOBAL_STEP: 813650 | > loss_text_ce: 0.027981698513031006 (0.03408282794447757) | > loss_mel_ce: 0.01312255859375 (nan) | > loss: 0.0001631121413083747 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13701525430472408) | > loader_time: 0.0045 (0.004647137380445738)  --> TIME: 2025-02-25 14:14:36 -- STEP: 1064/36938 -- GLOBAL_STEP: 813700 | > loss_text_ce: 0.03158889710903168 (0.034154868346842095) | > loss_mel_ce: 2.9756789207458496 (nan) | > loss: 0.011933602392673492 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.13722508473503855) | > loader_time: 0.0047 (0.004650129411453596)  --> TIME: 2025-02-25 14:14:45 -- STEP: 1114/36938 -- GLOBAL_STEP: 813750 | > loss_text_ce: 0.03620953857898712 (0.034149124073048626) | > loss_mel_ce: 1.693395733833313 (nan) | > loss: 0.006863513495773077 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13706073833753216) | > loader_time: 0.0047 (0.004649632709030624)  --> TIME: 2025-02-25 14:14:54 -- STEP: 1164/36938 -- GLOBAL_STEP: 813800 | > loss_text_ce: 0.03151385858654976 (0.03409369055204149) | > loss_mel_ce: 2.8597164154052734 (nan) | > loss: 0.011473136954009533 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13692774887347137) | > loader_time: 0.0045 (0.004647692044576015)  --> TIME: 2025-02-25 14:15:03 -- STEP: 1214/36938 -- GLOBAL_STEP: 813850 | > loss_text_ce: 0.03518456965684891 (0.03409223779468282) | > loss_mel_ce: 2.6523280143737793 (nan) | > loss: 0.010664733126759529 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1368505552141041) | > loader_time: 0.0045 (0.004645844071465238)  --> TIME: 2025-02-25 14:15:13 -- STEP: 1264/36938 -- GLOBAL_STEP: 813900 | > loss_text_ce: 0.026146147400140762 (0.03412800977121499) | > loss_mel_ce: 1.999853253364563 (nan) | > loss: 0.008039680309593678 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13683680756182598) | > loader_time: 0.0046 (0.004644645543038102)  --> TIME: 2025-02-25 14:15:22 -- STEP: 1314/36938 -- GLOBAL_STEP: 813950 | > loss_text_ce: 0.024510497227311134 (0.03410762325822289) | > loss_mel_ce: 1.3057149648666382 (nan) | > loss: 0.005278673022985458 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13668948838336994) | > loader_time: 0.0044 (0.004643725115051741)  --> TIME: 2025-02-25 14:15:31 -- STEP: 1364/36938 -- GLOBAL_STEP: 814000 | > loss_text_ce: 0.028606774285435677 (0.03410608546543442) | > loss_mel_ce: 2.616598129272461 (nan) | > loss: 0.010496845468878746 (nan) | > current_lr: 5e-06 | > step_time: 0.1942 (0.13689884709472872) | > loader_time: 0.0049 (0.004644530085174923)  --> TIME: 2025-02-25 14:15:40 -- STEP: 1414/36938 -- GLOBAL_STEP: 814050 | > loss_text_ce: 0.031146444380283356 (0.034091057853254225) | > loss_mel_ce: 1.6390026807785034 (nan) | > loss: 0.00662757596001029 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13683634448422388) | > loader_time: 0.0045 (0.004644324621014416)  --> TIME: 2025-02-25 14:15:50 -- STEP: 1464/36938 -- GLOBAL_STEP: 814100 | > loss_text_ce: 0.022579407319426537 (0.0341465486286086) | > loss_mel_ce: 1.579826831817627 (nan) | > loss: 0.00635875528678298 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.13693172732988984) | > loader_time: 0.0045 (0.0046445520523467326)  --> TIME: 2025-02-25 14:15:59 -- STEP: 1514/36938 -- GLOBAL_STEP: 814150 | > loss_text_ce: 0.045485325157642365 (0.03415668485169464) | > loss_mel_ce: 2.984539270401001 (nan) | > loss: 0.012023907154798508 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13679323297195736) | > loader_time: 0.0048 (0.0046434503250424395)  --> TIME: 2025-02-25 14:16:08 -- STEP: 1564/36938 -- GLOBAL_STEP: 814200 | > loss_text_ce: 0.03678234666585922 (0.034145438040266055) | > loss_mel_ce: 2.8252954483032227 (nan) | > loss: 0.011357451789081097 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1367196823324998) | > loader_time: 0.0045 (0.004642071473933846)  --> TIME: 2025-02-25 14:16:18 -- STEP: 1614/36938 -- GLOBAL_STEP: 814250 | > loss_text_ce: 0.03861142694950104 (0.03412126766545453) | > loss_mel_ce: 3.182292938232422 (nan) | > loss: 0.012781367637217045 (nan) | > current_lr: 5e-06 | > step_time: 0.1692 (0.1367063046683461) | > loader_time: 0.0047 (0.004643720912815172)  --> TIME: 2025-02-25 14:16:27 -- STEP: 1664/36938 -- GLOBAL_STEP: 814300 | > loss_text_ce: 0.0310409776866436 (0.034129547471820815) | > loss_mel_ce: 2.7689504623413086 (nan) | > loss: 0.011111077852547169 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13679776441019315) | > loader_time: 0.0046 (0.004644618584559511)  --> TIME: 2025-02-25 14:16:36 -- STEP: 1714/36938 -- GLOBAL_STEP: 814350 | > loss_text_ce: 0.032928865402936935 (0.03409921062584256) | > loss_mel_ce: 3.15337872505188 (nan) | > loss: 0.012644078582525253 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13685542143350998) | > loader_time: 0.0051 (0.004643916110035419)  --> TIME: 2025-02-25 14:16:46 -- STEP: 1764/36938 -- GLOBAL_STEP: 814400 | > loss_text_ce: 0.029646819457411766 (0.03407141046288112) | > loss_mel_ce: 3.063451051712036 (nan) | > loss: 0.012274199165403843 (nan) | > current_lr: 5e-06 | > step_time: 0.168 (0.13691792736788722) | > loader_time: 0.0047 (0.004642766349169673)  --> TIME: 2025-02-25 14:16:56 -- STEP: 1814/36938 -- GLOBAL_STEP: 814450 | > loss_text_ce: 0.023589346557855606 (0.03400990438350817) | > loss_mel_ce: 0.012102622538805008 (nan) | > loss: 0.00014163480955176055 (nan) | > current_lr: 5e-06 | > step_time: 0.1123 (0.13703580241744814) | > loader_time: 0.0044 (0.004642389443638306)  --> TIME: 2025-02-25 14:17:05 -- STEP: 1864/36938 -- GLOBAL_STEP: 814500 | > loss_text_ce: 0.045122187584638596 (0.034021521987925074) | > loss_mel_ce: 2.4974091053009033 (nan) | > loss: 0.010089410468935966 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13693761838352214) | > loader_time: 0.0047 (0.004642447688548861)  --> TIME: 2025-02-25 14:17:14 -- STEP: 1914/36938 -- GLOBAL_STEP: 814550 | > loss_text_ce: 0.03919222950935364 (0.03402563672495249) | > loss_mel_ce: 3.3905766010284424 (nan) | > loss: 0.013610194437205791 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.13677322316443777) | > loader_time: 0.0047 (0.004641917058292975)  --> TIME: 2025-02-25 14:17:23 -- STEP: 1964/36938 -- GLOBAL_STEP: 814600 | > loss_text_ce: 0.034085508435964584 (0.03400989987151485) | > loss_mel_ce: 1.70611572265625 (nan) | > loss: 0.006905560847371817 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13688130971127754) | > loader_time: 0.0046 (0.004642314677811446)  --> TIME: 2025-02-25 14:17:33 -- STEP: 2014/36938 -- GLOBAL_STEP: 814650 | > loss_text_ce: 0.026818441227078438 (0.03397491059100479) | > loss_mel_ce: 2.7333366870880127 (nan) | > loss: 0.010952997021377087 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13690097152062178) | > loader_time: 0.0046 (0.004642298702213473)  --> TIME: 2025-02-25 14:17:42 -- STEP: 2064/36938 -- GLOBAL_STEP: 814700 | > loss_text_ce: 0.027908949181437492 (0.03401230012342807) | > loss_mel_ce: 0.05000203475356102 (nan) | > loss: 0.00030917057301849127 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.136728940083999) | > loader_time: 0.0046 (0.004641738626383997)  --> TIME: 2025-02-25 14:17:51 -- STEP: 2114/36938 -- GLOBAL_STEP: 814750 | > loss_text_ce: 0.03048282116651535 (0.03400368647396125) | > loss_mel_ce: 3.2521729469299316 (nan) | > loss: 0.013026412576436996 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.13660779141432391) | > loader_time: 0.0047 (0.004641361809498653)  --> TIME: 2025-02-25 14:18:00 -- STEP: 2164/36938 -- GLOBAL_STEP: 814800 | > loss_text_ce: 0.028365230187773705 (0.034000630890296375) | > loss_mel_ce: 2.3677732944488525 (nan) | > loss: 0.009508486837148666 (nan) | > current_lr: 5e-06 | > step_time: 0.1829 (0.13668367558618558) | > loader_time: 0.0048 (0.004642295418737558)  --> TIME: 2025-02-25 14:18:09 -- STEP: 2214/36938 -- GLOBAL_STEP: 814850 | > loss_text_ce: 0.03313982114195824 (0.03400790009962252) | > loss_mel_ce: 2.987978458404541 (nan) | > loss: 0.011988564394414425 (nan) | > current_lr: 5e-06 | > step_time: 0.1823 (0.13666912436808337) | > loader_time: 0.0047 (0.004642034180011423)  --> TIME: 2025-02-25 14:18:19 -- STEP: 2264/36938 -- GLOBAL_STEP: 814900 | > loss_text_ce: 0.02845487743616104 (0.033985668534574194) | > loss_mel_ce: 4.006728649139404 (nan) | > loss: 0.016012633219361305 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13669003772651345) | > loader_time: 0.0046 (0.0046410410016669845)  --> TIME: 2025-02-25 14:18:28 -- STEP: 2314/36938 -- GLOBAL_STEP: 814950 | > loss_text_ce: 0.034462884068489075 (0.03401138488881049) | > loss_mel_ce: 2.996492624282837 (nan) | > loss: 0.012027601711452007 (nan) | > current_lr: 5e-06 | > step_time: 0.163 (0.13670684225928084) | > loader_time: 0.0046 (0.0046406248672933815)  --> TIME: 2025-02-25 14:18:37 -- STEP: 2364/36938 -- GLOBAL_STEP: 815000 | > loss_text_ce: 0.032467518001794815 (0.03402685406208099) | > loss_mel_ce: 3.0468811988830566 (nan) | > loss: 0.01221963856369257 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.13670379679820252) | > loader_time: 0.0048 (0.0046403347538207474)  --> TIME: 2025-02-25 14:18:46 -- STEP: 2414/36938 -- GLOBAL_STEP: 815050 | > loss_text_ce: 0.04176773130893707 (0.034026227296926455) | > loss_mel_ce: 2.42415714263916 (nan) | > loss: 0.009785417467355728 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.1365877276328148) | > loader_time: 0.0044 (0.004639475426598835)  --> TIME: 2025-02-25 14:18:56 -- STEP: 2464/36938 -- GLOBAL_STEP: 815100 | > loss_text_ce: 0.0317906029522419 (0.03402120047951926) | > loss_mel_ce: 3.323699951171875 (nan) | > loss: 0.013315439224243164 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13658586541166534) | > loader_time: 0.0046 (0.004639710989091311)  --> TIME: 2025-02-25 14:19:05 -- STEP: 2514/36938 -- GLOBAL_STEP: 815150 | > loss_text_ce: 0.03594483062624931 (0.03402592503712434) | > loss_mel_ce: 0.10570144653320312 (nan) | > loss: 0.0005620884476229548 (nan) | > current_lr: 5e-06 | > step_time: 0.1818 (0.1365867851267944) | > loader_time: 0.0047 (0.004639522651878736)  --> TIME: 2025-02-25 14:19:14 -- STEP: 2564/36938 -- GLOBAL_STEP: 815200 | > loss_text_ce: 0.03344324603676796 (0.03402248659540203) | > loss_mel_ce: 3.143404722213745 (nan) | > loss: 0.012606539763510227 (nan) | > current_lr: 5e-06 | > step_time: 0.1266 (0.13656117466050247) | > loader_time: 0.0047 (0.004639026899233632)  --> TIME: 2025-02-25 14:19:24 -- STEP: 2614/36938 -- GLOBAL_STEP: 815250 | > loss_text_ce: 0.035500604659318924 (0.03402292930756922) | > loss_mel_ce: 2.240997314453125 (nan) | > loss: 0.00903372187167406 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.1365329922841717) | > loader_time: 0.0047 (0.0046405015141342)  --> TIME: 2025-02-25 14:19:32 -- STEP: 2664/36938 -- GLOBAL_STEP: 815300 | > loss_text_ce: 0.03759297356009483 (0.03403190696121162) | > loss_mel_ce: 3.1073782444000244 (nan) | > loss: 0.012480044737458229 (nan) | > current_lr: 5e-06 | > step_time: 0.1496 (0.13644940403846612) | > loader_time: 0.0047 (0.004640965221880439)  --> TIME: 2025-02-25 14:19:42 -- STEP: 2714/36938 -- GLOBAL_STEP: 815350 | > loss_text_ce: 0.04367493838071823 (0.034020469666375536) | > loss_mel_ce: 2.4661080837249756 (nan) | > loss: 0.009959457442164421 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13652694067185434) | > loader_time: 0.0046 (0.004640858930617288)  --> TIME: 2025-02-25 14:19:52 -- STEP: 2764/36938 -- GLOBAL_STEP: 815400 | > loss_text_ce: 0.04058460518717766 (0.03402207947710813) | > loss_mel_ce: 3.404700756072998 (nan) | > loss: 0.01367176789790392 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.1366267000714532) | > loader_time: 0.0048 (0.004644488719023778)  --> TIME: 2025-02-25 14:20:01 -- STEP: 2814/36938 -- GLOBAL_STEP: 815450 | > loss_text_ce: 0.025394074618816376 (0.03400331003427249) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13666059925103716) | > loader_time: 0.0046 (0.004644003588846586)  --> TIME: 2025-02-25 14:20:10 -- STEP: 2864/36938 -- GLOBAL_STEP: 815500 | > loss_text_ce: 0.028328269720077515 (0.03400424445033655) | > loss_mel_ce: 2.5782582759857178 (nan) | > loss: 0.010343597270548344 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13664984295155058) | > loader_time: 0.0045 (0.004643836334430976)  --> TIME: 2025-02-25 14:20:20 -- STEP: 2914/36938 -- GLOBAL_STEP: 815550 | > loss_text_ce: 0.03176978603005409 (0.03400237070762154) | > loss_mel_ce: 2.842411518096924 (nan) | > loss: 0.011405481956899166 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13660871581664685) | > loader_time: 0.0046 (0.004644173011439421)  --> TIME: 2025-02-25 14:20:29 -- STEP: 2964/36938 -- GLOBAL_STEP: 815600 | > loss_text_ce: 0.038109805434942245 (0.03399957215329229) | > loss_mel_ce: 2.949887752532959 (nan) | > loss: 0.011857133358716965 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.1365955179197745) | > loader_time: 0.0047 (0.004644334798882364)  --> TIME: 2025-02-25 14:20:38 -- STEP: 3014/36938 -- GLOBAL_STEP: 815650 | > loss_text_ce: 0.04332771152257919 (0.03400171881197989) | > loss_mel_ce: 1.9181994199752808 (nan) | > loss: 0.007783838082104921 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.1366340014502631) | > loader_time: 0.0044 (0.004645600489772701)  --> TIME: 2025-02-25 14:20:48 -- STEP: 3064/36938 -- GLOBAL_STEP: 815700 | > loss_text_ce: 0.04483981803059578 (0.03401702722350931) | > loss_mel_ce: 3.0061728954315186 (nan) | > loss: 0.012107194401323795 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13659717333223415) | > loader_time: 0.0046 (0.004645574886556083)  --> TIME: 2025-02-25 14:20:57 -- STEP: 3114/36938 -- GLOBAL_STEP: 815750 | > loss_text_ce: 0.03782961145043373 (0.034022737290248246) | > loss_mel_ce: 2.279203176498413 (nan) | > loss: 0.009194575250148773 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1366283951007791) | > loader_time: 0.0045 (0.004647111540508331)  --> TIME: 2025-02-25 14:21:06 -- STEP: 3164/36938 -- GLOBAL_STEP: 815800 | > loss_text_ce: 0.02773214317858219 (0.03401216672787566) | > loss_mel_ce: 2.9769489765167236 (nan) | > loss: 0.011923338286578655 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.1366293816138459) | > loader_time: 0.0045 (0.004646969730724427)  --> TIME: 2025-02-25 14:21:15 -- STEP: 3214/36938 -- GLOBAL_STEP: 815850 | > loss_text_ce: 0.03469698503613472 (0.03402520226420119) | > loss_mel_ce: 1.9735498428344727 (nan) | > loss: 0.007969234138727188 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13652662327368215) | > loader_time: 0.0045 (0.004647339584668673)  --> TIME: 2025-02-25 14:21:24 -- STEP: 3264/36938 -- GLOBAL_STEP: 815900 | > loss_text_ce: 0.03929009288549423 (0.03402047632912642) | > loss_mel_ce: 2.670699119567871 (nan) | > loss: 0.010753926821053028 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.13641323835826388) | > loader_time: 0.0047 (0.0046462959372529766)  --> TIME: 2025-02-25 14:21:34 -- STEP: 3314/36938 -- GLOBAL_STEP: 815950 | > loss_text_ce: 0.02405986562371254 (0.03400000850133254) | > loss_mel_ce: 2.9772579669952393 (nan) | > loss: 0.01190999150276184 (nan) | > current_lr: 5e-06 | > step_time: 0.1691 (0.1364522763743196) | > loader_time: 0.0047 (0.0046459809799977)  --> TIME: 2025-02-25 14:21:43 -- STEP: 3364/36938 -- GLOBAL_STEP: 816000 | > loss_text_ce: 0.03338634595274925 (0.03399936789392018) | > loss_mel_ce: 3.019791603088379 (nan) | > loss: 0.012115785852074623 (nan) | > current_lr: 5e-06 | > step_time: 0.1468 (0.13646017236460117) | > loader_time: 0.0046 (0.00464544178614576)  --> TIME: 2025-02-25 14:21:52 -- STEP: 3414/36938 -- GLOBAL_STEP: 816050 | > loss_text_ce: 0.03748732805252075 (0.03400202950612166) | > loss_mel_ce: 3.3883326053619385 (nan) | > loss: 0.013594524003565311 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.1363880205517564) | > loader_time: 0.0047 (0.00464437862691225)  --> TIME: 2025-02-25 14:22:01 -- STEP: 3464/36938 -- GLOBAL_STEP: 816100 | > loss_text_ce: 0.03799363598227501 (0.033989975258479764) | > loss_mel_ce: 3.191890001296997 (nan) | > loss: 0.012816999107599258 (nan) | > current_lr: 5e-06 | > step_time: 0.1715 (0.13635384795572134) | > loader_time: 0.0047 (0.004643661802415201)  --> TIME: 2025-02-25 14:22:11 -- STEP: 3514/36938 -- GLOBAL_STEP: 816150 | > loss_text_ce: 0.030846836045384407 (0.03399256484208771) | > loss_mel_ce: 0.6202392578125 (nan) | > loss: 0.0025836750864982605 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13636364873051304) | > loader_time: 0.0045 (0.004643053783227851)  --> TIME: 2025-02-25 14:22:21 -- STEP: 3564/36938 -- GLOBAL_STEP: 816200 | > loss_text_ce: 0.038438670337200165 (0.033994395776591954) | > loss_mel_ce: 2.863837957382202 (nan) | > loss: 0.011516970582306385 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13648729706987917) | > loader_time: 0.0046 (0.004643140617845829)  --> TIME: 2025-02-25 14:22:30 -- STEP: 3614/36938 -- GLOBAL_STEP: 816250 | > loss_text_ce: 0.03380734845995903 (0.03401371405738743) | > loss_mel_ce: 3.0420823097229004 (nan) | > loss: 0.012205911800265312 (nan) | > current_lr: 5e-06 | > step_time: 0.179 (0.13652056399536971) | > loader_time: 0.0049 (0.004642712720270365)  --> TIME: 2025-02-25 14:22:39 -- STEP: 3664/36938 -- GLOBAL_STEP: 816300 | > loss_text_ce: 0.041578516364097595 (0.03399113640667006) | > loss_mel_ce: 2.7844107151031494 (nan) | > loss: 0.01121424324810505 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13653707393652478) | > loader_time: 0.0045 (0.0046421377289243065)  --> TIME: 2025-02-25 14:22:49 -- STEP: 3714/36938 -- GLOBAL_STEP: 816350 | > loss_text_ce: 0.027828238904476166 (0.033977778463969696) | > loss_mel_ce: 3.151728391647339 (nan) | > loss: 0.012617289088666439 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.13653364377953897) | > loader_time: 0.0046 (0.0046419557474862445)  --> TIME: 2025-02-25 14:22:58 -- STEP: 3764/36938 -- GLOBAL_STEP: 816400 | > loss_text_ce: 0.040900059044361115 (0.03397251810832964) | > loss_mel_ce: 2.589216470718384 (nan) | > loss: 0.010436970740556717 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13657768164126957) | > loader_time: 0.0046 (0.0046415330879746464)  --> TIME: 2025-02-25 14:23:08 -- STEP: 3814/36938 -- GLOBAL_STEP: 816450 | > loss_text_ce: 0.020213622599840164 (0.03397613712481439) | > loss_mel_ce: 1.4706776142120361 (nan) | > loss: 0.005916235502809286 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13664356088163218) | > loader_time: 0.0045 (0.004641414876376756)  --> TIME: 2025-02-25 14:23:17 -- STEP: 3864/36938 -- GLOBAL_STEP: 816500 | > loss_text_ce: 0.018390625715255737 (0.033968688752386275) | > loss_mel_ce: 3.2667410373687744 (nan) | > loss: 0.01303623802959919 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13661876610840823) | > loader_time: 0.0046 (0.004641149786935329)  --> TIME: 2025-02-25 14:23:27 -- STEP: 3914/36938 -- GLOBAL_STEP: 816550 | > loss_text_ce: 0.04137013852596283 (0.03397325578197532) | > loss_mel_ce: 2.5277862548828125 (nan) | > loss: 0.010195065289735794 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.13664717325825562) | > loader_time: 0.0049 (0.004640833114444684)  --> TIME: 2025-02-25 14:23:36 -- STEP: 3964/36938 -- GLOBAL_STEP: 816600 | > loss_text_ce: 0.020444035530090332 (0.03397164293390974) | > loss_mel_ce: 2.4282889366149902 (nan) | > loss: 0.009717194363474846 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13668551177959268) | > loader_time: 0.0045 (0.004640837911399888)  --> TIME: 2025-02-25 14:23:45 -- STEP: 4014/36938 -- GLOBAL_STEP: 816650 | > loss_text_ce: 0.042330045253038406 (0.03398254719030555) | > loss_mel_ce: 1.7940605878829956 (nan) | > loss: 0.007287264801561832 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.1366229664940827) | > loader_time: 0.0044 (0.004640375255172259)  --> TIME: 2025-02-25 14:23:55 -- STEP: 4064/36938 -- GLOBAL_STEP: 816700 | > loss_text_ce: 0.04108849912881851 (0.03398962810820709) | > loss_mel_ce: 3.2141575813293457 (nan) | > loss: 0.01291764434427023 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.13666678636562168) | > loader_time: 0.0048 (0.0046404542064103485)  --> TIME: 2025-02-25 14:24:04 -- STEP: 4114/36938 -- GLOBAL_STEP: 816750 | > loss_text_ce: 0.02884269692003727 (0.03397676217386465) | > loss_mel_ce: 0.66064453125 (nan) | > loss: 0.0027360606472939253 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.136687178604801) | > loader_time: 0.0045 (0.004640284822470473)  --> TIME: 2025-02-25 14:24:14 -- STEP: 4164/36938 -- GLOBAL_STEP: 816800 | > loss_text_ce: 0.032828424125909805 (0.03398653916839227) | > loss_mel_ce: 2.6633927822113037 (nan) | > loss: 0.010699290782213211 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13672482119971574) | > loader_time: 0.0049 (0.004640667227800931)  --> TIME: 2025-02-25 14:24:23 -- STEP: 4214/36938 -- GLOBAL_STEP: 816850 | > loss_text_ce: 0.033045727759599686 (0.03398385946428904) | > loss_mel_ce: 2.951913833618164 (nan) | > loss: 0.011845078319311142 (nan) | > current_lr: 5e-06 | > step_time: 0.1854 (0.13666597270377165) | > loader_time: 0.0049 (0.00464056162795694)  --> TIME: 2025-02-25 14:24:32 -- STEP: 4264/36938 -- GLOBAL_STEP: 816900 | > loss_text_ce: 0.03856220841407776 (0.033967852624793515) | > loss_mel_ce: 1.5933603048324585 (nan) | > loss: 0.006475883070379496 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13665371121206163) | > loader_time: 0.0045 (0.004640287742382137)  --> TIME: 2025-02-25 14:24:41 -- STEP: 4314/36938 -- GLOBAL_STEP: 816950 | > loss_text_ce: 0.02601807750761509 (0.03397514415869454) | > loss_mel_ce: 3.6349284648895264 (nan) | > loss: 0.014527566730976105 (nan) | > current_lr: 5e-06 | > step_time: 0.1074 (0.13660416229488573) | > loader_time: 0.0044 (0.004640487205335588)  --> TIME: 2025-02-25 14:24:51 -- STEP: 4364/36938 -- GLOBAL_STEP: 817000 | > loss_text_ce: 0.032018959522247314 (0.033981244321775746) | > loss_mel_ce: 2.9679031372070312 (nan) | > loss: 0.011904452927410603 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1366683277485024) | > loader_time: 0.0046 (0.004640668657916492)  --> TIME: 2025-02-25 14:25:00 -- STEP: 4414/36938 -- GLOBAL_STEP: 817050 | > loss_text_ce: 0.03416552394628525 (0.03398753618243362) | > loss_mel_ce: 3.5072648525238037 (nan) | > loss: 0.01405329629778862 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.1366527192885379) | > loader_time: 0.005 (0.004640681580499007)  --> TIME: 2025-02-25 14:25:09 -- STEP: 4464/36938 -- GLOBAL_STEP: 817100 | > loss_text_ce: 0.039347875863313675 (0.03399074988858587) | > loss_mel_ce: 0.013844807632267475 (nan) | > loss: 0.00021108207874931395 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13665134927064299) | > loader_time: 0.0046 (0.004640581146363285)  --> TIME: 2025-02-25 14:25:19 -- STEP: 4514/36938 -- GLOBAL_STEP: 817150 | > loss_text_ce: 0.04031369090080261 (0.03399677353543424) | > loss_mel_ce: 3.269566774368286 (nan) | > loss: 0.013134446926414967 (nan) | > current_lr: 5e-06 | > step_time: 0.1767 (0.1366560408432082) | > loader_time: 0.0048 (0.004640028124949967)  --> TIME: 2025-02-25 14:25:28 -- STEP: 4564/36938 -- GLOBAL_STEP: 817200 | > loss_text_ce: 0.03059447929263115 (0.03398767959548502) | > loss_mel_ce: 3.1547486782073975 (nan) | > loss: 0.012640251778066158 (nan) | > current_lr: 5e-06 | > step_time: 0.1522 (0.13663115112746022) | > loader_time: 0.0046 (0.00463919661527762)  --> TIME: 2025-02-25 14:25:37 -- STEP: 4614/36938 -- GLOBAL_STEP: 817250 | > loss_text_ce: 0.03147239610552788 (0.03397408462891286) | > loss_mel_ce: 1.7044390439987183 (nan) | > loss: 0.006888538133352995 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13664983762617922) | > loader_time: 0.0045 (0.004638850146369287)  --> TIME: 2025-02-25 14:25:47 -- STEP: 4664/36938 -- GLOBAL_STEP: 817300 | > loss_text_ce: 0.03144212067127228 (0.033991061743487636) | > loss_mel_ce: 2.9058008193969727 (nan) | > loss: 0.011655726470053196 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1366424466999279) | > loader_time: 0.0044 (0.00463819053937804)  --> TIME: 2025-02-25 14:25:55 -- STEP: 4714/36938 -- GLOBAL_STEP: 817350 | > loss_text_ce: 0.039290573447942734 (0.033989913877981504) | > loss_mel_ce: 2.7770721912384033 (nan) | > loss: 0.01117604412138462 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13656884434367858) | > loader_time: 0.0045 (0.004637183099163251)  --> TIME: 2025-02-25 14:26:05 -- STEP: 4764/36938 -- GLOBAL_STEP: 817400 | > loss_text_ce: 0.03314777836203575 (0.03398857855607092) | > loss_mel_ce: 3.1580255031585693 (nan) | > loss: 0.012663386762142181 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13655507964911764) | > loader_time: 0.0045 (0.004636513496225969)  --> TIME: 2025-02-25 14:26:14 -- STEP: 4814/36938 -- GLOBAL_STEP: 817450 | > loss_text_ce: 0.03531236946582794 (0.033990001977054755) | > loss_mel_ce: 3.2418997287750244 (nan) | > loss: 0.01300481054931879 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.1365186110739401) | > loader_time: 0.0046 (0.004635888261323555)  --> TIME: 2025-02-25 14:26:23 -- STEP: 4864/36938 -- GLOBAL_STEP: 817500 | > loss_text_ce: 0.034650638699531555 (0.03398520429465462) | > loss_mel_ce: 3.053231716156006 (nan) | > loss: 0.012253501452505589 (nan) | > current_lr: 5e-06 | > step_time: 0.1579 (0.13648594759012567) | > loader_time: 0.0046 (0.004637114850706177)  --> TIME: 2025-02-25 14:26:32 -- STEP: 4914/36938 -- GLOBAL_STEP: 817550 | > loss_text_ce: 0.050200946629047394 (0.0339912742658338) | > loss_mel_ce: 2.423262357711792 (nan) | > loss: 0.009815331548452377 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1365175197703792) | > loader_time: 0.0044 (0.004636955930782038)  --> TIME: 2025-02-25 14:26:42 -- STEP: 4964/36938 -- GLOBAL_STEP: 817600 | > loss_text_ce: 0.037236910313367844 (0.034002589865975485) | > loss_mel_ce: 1.9982802867889404 (nan) | > loss: 0.008077450096607208 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.1365243050957958) | > loader_time: 0.0046 (0.004636506511740494)  --> TIME: 2025-02-25 14:26:51 -- STEP: 5014/36938 -- GLOBAL_STEP: 817650 | > loss_text_ce: 0.033774178475141525 (0.034011455508350816) | > loss_mel_ce: 2.8958096504211426 (nan) | > loss: 0.01162533275783062 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.13653914629818684) | > loader_time: 0.0046 (0.004636662577936077)  --> TIME: 2025-02-25 14:27:00 -- STEP: 5064/36938 -- GLOBAL_STEP: 817700 | > loss_text_ce: 0.04199672117829323 (0.03401887063400464) | > loss_mel_ce: 2.9986400604248047 (nan) | > loss: 0.01206601969897747 (nan) | > current_lr: 5e-06 | > step_time: 0.152 (0.13651368831759572) | > loader_time: 0.0046 (0.004636122952516986)  --> TIME: 2025-02-25 14:27:09 -- STEP: 5114/36938 -- GLOBAL_STEP: 817750 | > loss_text_ce: 0.042539339512586594 (0.03401680052386769) | > loss_mel_ce: 2.2914302349090576 (nan) | > loss: 0.009261785075068474 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13646168386185586) | > loader_time: 0.0045 (0.004635540125283305)  --> TIME: 2025-02-25 14:27:19 -- STEP: 5164/36938 -- GLOBAL_STEP: 817800 | > loss_text_ce: 0.030647169798612595 (0.034002607767661244) | > loss_mel_ce: 2.662944793701172 (nan) | > loss: 0.010688858106732368 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13648080281007197) | > loader_time: 0.0046 (0.004635794600435778)  --> TIME: 2025-02-25 14:27:28 -- STEP: 5214/36938 -- GLOBAL_STEP: 817850 | > loss_text_ce: 0.03595872223377228 (0.03399310820447689) | > loss_mel_ce: 2.770336627960205 (nan) | > loss: 0.01113609317690134 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.1364722209823602) | > loader_time: 0.0049 (0.00463629742897832)  --> TIME: 2025-02-25 14:27:37 -- STEP: 5264/36938 -- GLOBAL_STEP: 817900 | > loss_text_ce: 0.03481471538543701 (0.033993486132781185) | > loss_mel_ce: 2.3969454765319824 (nan) | > loss: 0.009649842977523804 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13646433685869422) | > loader_time: 0.0053 (0.004636038581651025)  --> TIME: 2025-02-25 14:27:47 -- STEP: 5314/36938 -- GLOBAL_STEP: 817950 | > loss_text_ce: 0.03738570585846901 (0.033988871212766746) | > loss_mel_ce: 3.0233922004699707 (nan) | > loss: 0.012145944871008396 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13651367924381205) | > loader_time: 0.0046 (0.0046360305165903445)  --> TIME: 2025-02-25 14:27:56 -- STEP: 5364/36938 -- GLOBAL_STEP: 818000 | > loss_text_ce: 0.02837255969643593 (0.03397393053485028) | > loss_mel_ce: 2.12371563911438 (nan) | > loss: 0.008540032431483269 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13649447807885207) | > loader_time: 0.0044 (0.004635982376528164)  --> TIME: 2025-02-25 14:28:05 -- STEP: 5414/36938 -- GLOBAL_STEP: 818050 | > loss_text_ce: 0.03183849900960922 (0.033979108978212685) | > loss_mel_ce: 3.18676495552063 (nan) | > loss: 0.01277223601937294 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.1365160926224165) | > loader_time: 0.0046 (0.004636233611260111)  --> TIME: 2025-02-25 14:28:15 -- STEP: 5464/36938 -- GLOBAL_STEP: 818100 | > loss_text_ce: 0.04299388825893402 (0.03398272702040876) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13651512979937505) | > loader_time: 0.0046 (0.0046357600615628365)  --> TIME: 2025-02-25 14:28:24 -- STEP: 5514/36938 -- GLOBAL_STEP: 818150 | > loss_text_ce: 0.032306551933288574 (0.03397633603277085) | > loss_mel_ce: 2.8249919414520264 (nan) | > loss: 0.011338486336171627 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13651764405824413) | > loader_time: 0.0046 (0.004635936979543557)  --> TIME: 2025-02-25 14:28:34 -- STEP: 5564/36938 -- GLOBAL_STEP: 818200 | > loss_text_ce: 0.03770093992352486 (0.03397386463420579) | > loss_mel_ce: 2.8060483932495117 (nan) | > loss: 0.0112847201526165 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.13653094112058925) | > loader_time: 0.0048 (0.004636338466710091)  --> TIME: 2025-02-25 14:28:43 -- STEP: 5614/36938 -- GLOBAL_STEP: 818250 | > loss_text_ce: 0.03385365754365921 (0.033983691032122385) | > loss_mel_ce: 3.5116922855377197 (nan) | > loss: 0.014069627970457077 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13660122289069832) | > loader_time: 0.0045 (0.004636156758592765)  --> TIME: 2025-02-25 14:28:53 -- STEP: 5664/36938 -- GLOBAL_STEP: 818300 | > loss_text_ce: 0.028717881068587303 (0.033990547412185414) | > loss_mel_ce: 2.6362717151641846 (nan) | > loss: 0.010575356893241405 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13662721713384) | > loader_time: 0.0047 (0.0046361483170487645)  --> TIME: 2025-02-25 14:29:02 -- STEP: 5714/36938 -- GLOBAL_STEP: 818350 | > loss_text_ce: 0.03040313720703125 (0.0340056052310198) | > loss_mel_ce: 1.440948486328125 (nan) | > loss: 0.0058386973105371 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13661619450272408) | > loader_time: 0.0046 (0.004636702439064811)  --> TIME: 2025-02-25 14:29:11 -- STEP: 5764/36938 -- GLOBAL_STEP: 818400 | > loss_text_ce: 0.033559512346982956 (0.03400792696078255) | > loss_mel_ce: 2.12544846534729 (nan) | > loss: 0.008567492477595806 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13661817665483955) | > loader_time: 0.0045 (0.00463659067603963)  --> TIME: 2025-02-25 14:29:21 -- STEP: 5814/36938 -- GLOBAL_STEP: 818450 | > loss_text_ce: 0.034352075308561325 (0.03400724482638305) | > loss_mel_ce: 2.9319446086883545 (nan) | > loss: 0.011771019548177719 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.1366390277316657) | > loader_time: 0.0046 (0.004636459552343648)  --> TIME: 2025-02-25 14:29:30 -- STEP: 5864/36938 -- GLOBAL_STEP: 818500 | > loss_text_ce: 0.031922075897455215 (0.03399238054828615) | > loss_mel_ce: 1.8605821132659912 (nan) | > loss: 0.007509937975555658 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13661542908881913) | > loader_time: 0.0046 (0.004636271669952466)  --> TIME: 2025-02-25 14:29:39 -- STEP: 5914/36938 -- GLOBAL_STEP: 818550 | > loss_text_ce: 0.0336790531873703 (0.03399576521064407) | > loss_mel_ce: 2.2269515991210938 (nan) | > loss: 0.008970757015049458 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.1366149895006556) | > loader_time: 0.0045 (0.004636455194991214)  --> TIME: 2025-02-25 14:29:48 -- STEP: 5964/36938 -- GLOBAL_STEP: 818600 | > loss_text_ce: 0.03103426843881607 (0.03400427304438743) | > loss_mel_ce: 1.2657548189163208 (nan) | > loss: 0.005145988427102566 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13659036163513044) | > loader_time: 0.0045 (0.004636314433345364)  --> TIME: 2025-02-25 14:29:58 -- STEP: 6014/36938 -- GLOBAL_STEP: 818650 | > loss_text_ce: 0.032378461211919785 (0.03400446802590144) | > loss_mel_ce: 3.103428363800049 (nan) | > loss: 0.012443678453564644 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.13661341077273684) | > loader_time: 0.0047 (0.004636423390372304)  --> TIME: 2025-02-25 14:30:08 -- STEP: 6064/36938 -- GLOBAL_STEP: 818700 | > loss_text_ce: 0.03178349509835243 (0.03400949104937839) | > loss_mel_ce: 3.1463239192962646 (nan) | > loss: 0.012611538171768188 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.13662786278529682) | > loader_time: 0.0047 (0.004636944126642462)  --> TIME: 2025-02-25 14:30:17 -- STEP: 6114/36938 -- GLOBAL_STEP: 818750 | > loss_text_ce: 0.03149697929620743 (0.03400492484461626) | > loss_mel_ce: 2.5849194526672363 (nan) | > loss: 0.010382605716586113 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13663066763249285) | > loader_time: 0.0045 (0.004637225219382141)  --> TIME: 2025-02-25 14:30:26 -- STEP: 6164/36938 -- GLOBAL_STEP: 818800 | > loss_text_ce: 0.037497106939554214 (0.034013144460843116) | > loss_mel_ce: 2.397977828979492 (nan) | > loss: 0.009664583019912243 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13660689277512805) | > loader_time: 0.0046 (0.004637329358701796)  --> TIME: 2025-02-25 14:30:36 -- STEP: 6214/36938 -- GLOBAL_STEP: 818850 | > loss_text_ce: 0.030268320813775063 (0.03401593210134842) | > loss_mel_ce: 2.52118182182312 (nan) | > loss: 0.010124803520739079 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13662057292012517) | > loader_time: 0.0046 (0.004637682403365534)  --> TIME: 2025-02-25 14:30:45 -- STEP: 6264/36938 -- GLOBAL_STEP: 818900 | > loss_text_ce: 0.039285413920879364 (0.03401382877874982) | > loss_mel_ce: 3.1615233421325684 (nan) | > loss: 0.012701623141765594 (nan) | > current_lr: 5e-06 | > step_time: 0.1521 (0.13665427284679207) | > loader_time: 0.0046 (0.0046381135096494955)  --> TIME: 2025-02-25 14:30:55 -- STEP: 6314/36938 -- GLOBAL_STEP: 818950 | > loss_text_ce: 0.03316137567162514 (0.03401281765570957) | > loss_mel_ce: 2.9894015789031982 (nan) | > loss: 0.011994298547506332 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.1366581356823729) | > loader_time: 0.0045 (0.00463788355306841)  --> TIME: 2025-02-25 14:31:04 -- STEP: 6364/36938 -- GLOBAL_STEP: 819000 | > loss_text_ce: 0.038525599986314774 (0.03401338632480768) | > loss_mel_ce: 1.575054407119751 (nan) | > loss: 0.006403095554560423 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13662973849897794) | > loader_time: 0.0047 (0.004637969544217069)  --> TIME: 2025-02-25 14:31:13 -- STEP: 6414/36938 -- GLOBAL_STEP: 819050 | > loss_text_ce: 0.03720920532941818 (0.03402378907604903) | > loss_mel_ce: 3.451202630996704 (nan) | > loss: 0.013842904940247536 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13659122278655397) | > loader_time: 0.0046 (0.004637787042468493)  --> TIME: 2025-02-25 14:31:22 -- STEP: 6464/36938 -- GLOBAL_STEP: 819100 | > loss_text_ce: 0.023402830585837364 (0.034025914658876284) | > loss_mel_ce: 1.595800757408142 (nan) | > loss: 0.006425411440432072 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13660126872877093) | > loader_time: 0.0045 (0.004637763604964335)  --> TIME: 2025-02-25 14:31:31 -- STEP: 6514/36938 -- GLOBAL_STEP: 819150 | > loss_text_ce: 0.03130844235420227 (0.0340255305271378) | > loss_mel_ce: 2.720090866088867 (nan) | > loss: 0.010918252170085907 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13661329011299303) | > loader_time: 0.0046 (0.004638240386417074)  --> TIME: 2025-02-25 14:31:41 -- STEP: 6564/36938 -- GLOBAL_STEP: 819200 | > loss_text_ce: 0.03543354198336601 (0.03401402638709223) | > loss_mel_ce: 2.6525421142578125 (nan) | > loss: 0.010666570626199245 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13662027796908624) | > loader_time: 0.0047 (0.004638044773115732)  --> TIME: 2025-02-25 14:31:50 -- STEP: 6614/36938 -- GLOBAL_STEP: 819250 | > loss_text_ce: 0.03198198974132538 (0.0340157268428906) | > loss_mel_ce: 3.371725559234619 (nan) | > loss: 0.01350677665323019 (nan) | > current_lr: 5e-06 | > step_time: 0.1705 (0.1366382835853632) | > loader_time: 0.0047 (0.00463803502271135)  --> TIME: 2025-02-25 14:31:59 -- STEP: 6664/36938 -- GLOBAL_STEP: 819300 | > loss_text_ce: 0.02878696098923683 (0.034009340489903554) | > loss_mel_ce: 1.9835273027420044 (nan) | > loss: 0.007985374890267849 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13661937513986866) | > loader_time: 0.0046 (0.004638014399275469)  --> TIME: 2025-02-25 14:32:08 -- STEP: 6714/36938 -- GLOBAL_STEP: 819350 | > loss_text_ce: 0.03177466616034508 (0.03400246137536722) | > loss_mel_ce: 0.0035858154296875 (nan) | > loss: 0.0001403193746227771 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13658469927818775) | > loader_time: 0.0045 (0.004638364991866142)  --> TIME: 2025-02-25 14:32:18 -- STEP: 6764/36938 -- GLOBAL_STEP: 819400 | > loss_text_ce: 0.03478310629725456 (0.03400057989132969) | > loss_mel_ce: 3.2805817127227783 (nan) | > loss: 0.013156210072338581 (nan) | > current_lr: 5e-06 | > step_time: 0.1667 (0.13662158195116647) | > loader_time: 0.0046 (0.004638300676588315)  --> TIME: 2025-02-25 14:32:28 -- STEP: 6814/36938 -- GLOBAL_STEP: 819450 | > loss_text_ce: 0.03619055077433586 (0.0339983748393703) | > loss_mel_ce: 2.4617202281951904 (nan) | > loss: 0.009912344627082348 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.1366407939126005) | > loader_time: 0.0046 (0.004637949796306567)  --> TIME: 2025-02-25 14:32:37 -- STEP: 6864/36938 -- GLOBAL_STEP: 819500 | > loss_text_ce: 0.023016300052404404 (0.03398090586001649) | > loss_mel_ce: 0.15747681260108948 (nan) | > loss: 0.0007162425899878144 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13663167299785275) | > loader_time: 0.0046 (0.004637730774623789)  --> TIME: 2025-02-25 14:32:46 -- STEP: 6914/36938 -- GLOBAL_STEP: 819550 | > loss_text_ce: 0.04152843356132507 (0.0339879581228163) | > loss_mel_ce: 2.8489696979522705 (nan) | > loss: 0.011470231227576733 (nan) | > current_lr: 5e-06 | > step_time: 0.1506 (0.13664037763365522) | > loader_time: 0.0046 (0.004638022896455186)  --> TIME: 2025-02-25 14:32:55 -- STEP: 6964/36938 -- GLOBAL_STEP: 819600 | > loss_text_ce: 0.030404988676309586 (0.03399484603754558) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13663559022270227) | > loader_time: 0.0044 (0.004637844462561763)  --> TIME: 2025-02-25 14:33:05 -- STEP: 7014/36938 -- GLOBAL_STEP: 819650 | > loss_text_ce: 0.03857165202498436 (0.03398802571223321) | > loss_mel_ce: 1.55804443359375 (nan) | > loss: 0.006335778161883354 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13662412719302375) | > loader_time: 0.0045 (0.004637458707180169)  --> TIME: 2025-02-25 14:33:14 -- STEP: 7064/36938 -- GLOBAL_STEP: 819700 | > loss_text_ce: 0.03285769745707512 (0.033985859664876224) | > loss_mel_ce: 2.193293333053589 (nan) | > loss: 0.0088339326903224 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13662333295407433) | > loader_time: 0.0045 (0.00463726900577003)  --> TIME: 2025-02-25 14:33:24 -- STEP: 7114/36938 -- GLOBAL_STEP: 819750 | > loss_text_ce: 0.029996467754244804 (0.03398554868831699) | > loss_mel_ce: 2.473376512527466 (nan) | > loss: 0.009934020228683949 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.13664755802460207) | > loader_time: 0.0046 (0.00463720945819963)  --> TIME: 2025-02-25 14:33:33 -- STEP: 7164/36938 -- GLOBAL_STEP: 819800 | > loss_text_ce: 0.03883476182818413 (0.03398769283443591) | > loss_mel_ce: 2.610572576522827 (nan) | > loss: 0.01051352173089981 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13667666918734897) | > loader_time: 0.0047 (0.004637184388007862)  --> TIME: 2025-02-25 14:33:42 -- STEP: 7214/36938 -- GLOBAL_STEP: 819850 | > loss_text_ce: 0.035924799740314484 (0.03399445876734236) | > loss_mel_ce: 2.9028661251068115 (nan) | > loss: 0.01166186947375536 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13666952661044207) | > loader_time: 0.0046 (0.004636897120675366)  --> TIME: 2025-02-25 14:33:51 -- STEP: 7264/36938 -- GLOBAL_STEP: 819900 | > loss_text_ce: 0.0386790931224823 (0.033993027178111014) | > loss_mel_ce: 2.824878215789795 (nan) | > loss: 0.011363323777914047 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13666663158044962) | > loader_time: 0.0046 (0.004636817862021219)  --> TIME: 2025-02-25 14:34:01 -- STEP: 7314/36938 -- GLOBAL_STEP: 819950 | > loss_text_ce: 0.03962668031454086 (0.03399544547034109) | > loss_mel_ce: 3.0288150310516357 (nan) | > loss: 0.01217635627835989 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13668288169482193) | > loader_time: 0.0045 (0.004637239896635014)  --> TIME: 2025-02-25 14:34:10 -- STEP: 7364/36938 -- GLOBAL_STEP: 820000 | > loss_text_ce: 0.030233779922127724 (0.034000263224866205) | > loss_mel_ce: 2.605394124984741 (nan) | > loss: 0.010458841919898987 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13667906667796387) | > loader_time: 0.0046 (0.00463727872569814) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_820000.pth  --> TIME: 2025-02-25 14:34:26 -- STEP: 7414/36938 -- GLOBAL_STEP: 820050 | > loss_text_ce: 0.02372359298169613 (0.03399083916211425) | > loss_mel_ce: 1.27215576171875 (nan) | > loss: 0.005142378620803356 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1366593024786638) | > loader_time: 0.0044 (0.004637229207843835)  --> TIME: 2025-02-25 14:34:35 -- STEP: 7464/36938 -- GLOBAL_STEP: 820100 | > loss_text_ce: 0.03761856630444527 (0.03398778017571373) | > loss_mel_ce: 2.2692673206329346 (nan) | > loss: 0.009154309518635273 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13666223936653357) | > loader_time: 0.0045 (0.004637466717635172)  --> TIME: 2025-02-25 14:34:44 -- STEP: 7514/36938 -- GLOBAL_STEP: 820150 | > loss_text_ce: 0.03717762976884842 (0.03399010787482246) | > loss_mel_ce: 3.344731330871582 (nan) | > loss: 0.01342027448117733 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.1366547717355117) | > loader_time: 0.0048 (0.004637429934408468)  --> TIME: 2025-02-25 14:34:53 -- STEP: 7564/36938 -- GLOBAL_STEP: 820200 | > loss_text_ce: 0.028443574905395508 (0.03399326005681655) | > loss_mel_ce: 3.086825370788574 (nan) | > loss: 0.012362179346382618 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13658364732576916) | > loader_time: 0.0046 (0.004637739886561736)  --> TIME: 2025-02-25 14:35:02 -- STEP: 7614/36938 -- GLOBAL_STEP: 820250 | > loss_text_ce: 0.04343525320291519 (0.03398735111538681) | > loss_mel_ce: 2.1513631343841553 (nan) | > loss: 0.008709518238902092 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.1365640854190461) | > loader_time: 0.0046 (0.0046378984393677485)  --> TIME: 2025-02-25 14:35:12 -- STEP: 7664/36938 -- GLOBAL_STEP: 820300 | > loss_text_ce: 0.03429155796766281 (0.03398833270325649) | > loss_mel_ce: 3.1518142223358154 (nan) | > loss: 0.012643277645111084 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.1366199258722694) | > loader_time: 0.0046 (0.004638229413171879)  --> TIME: 2025-02-25 14:35:21 -- STEP: 7714/36938 -- GLOBAL_STEP: 820350 | > loss_text_ce: 0.033439233899116516 (0.03398583541168598) | > loss_mel_ce: 3.0626707077026367 (nan) | > loss: 0.012286150828003883 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.1366313504900748) | > loader_time: 0.0047 (0.0046384881004509345)  --> TIME: 2025-02-25 14:35:30 -- STEP: 7764/36938 -- GLOBAL_STEP: 820400 | > loss_text_ce: 0.03190429136157036 (0.03397994595908468) | > loss_mel_ce: 3.155986785888672 (nan) | > loss: 0.012650362215936184 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.13660377736929358) | > loader_time: 0.0047 (0.004638800695960739)  --> TIME: 2025-02-25 14:35:40 -- STEP: 7814/36938 -- GLOBAL_STEP: 820450 | > loss_text_ce: 0.03185560926795006 (0.033980643993378405) | > loss_mel_ce: 0.004058837890625 (nan) | > loss: 0.00014251765969675034 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13659468150547735) | > loader_time: 0.0046 (0.004638831573341805)  --> TIME: 2025-02-25 14:35:49 -- STEP: 7864/36938 -- GLOBAL_STEP: 820500 | > loss_text_ce: 0.028361553326249123 (0.03398709683650613) | > loss_mel_ce: 1.8408777713775635 (nan) | > loss: 0.007417616900056601 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13658317650686347) | > loader_time: 0.0045 (0.004639110542144057)  --> TIME: 2025-02-25 14:35:58 -- STEP: 7914/36938 -- GLOBAL_STEP: 820550 | > loss_text_ce: 0.039516497403383255 (0.03398968420507571) | > loss_mel_ce: 2.7475028038024902 (nan) | > loss: 0.011059600859880447 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.1365983671752801) | > loader_time: 0.0046 (0.004639627326760936)  --> TIME: 2025-02-25 14:36:08 -- STEP: 7964/36938 -- GLOBAL_STEP: 820600 | > loss_text_ce: 0.029501795768737793 (0.03398327528986003) | > loss_mel_ce: 2.779203176498413 (nan) | > loss: 0.011145655065774918 (nan) | > current_lr: 5e-06 | > step_time: 0.1659 (0.13662201070354452) | > loader_time: 0.0049 (0.004639958688687085)  --> TIME: 2025-02-25 14:36:17 -- STEP: 8014/36938 -- GLOBAL_STEP: 820650 | > loss_text_ce: 0.029995283111929893 (0.03398443799812663) | > loss_mel_ce: 2.2496676445007324 (nan) | > loss: 0.009046281687915325 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13661861458353095) | > loader_time: 0.0045 (0.004640054815809755)  --> TIME: 2025-02-25 14:36:27 -- STEP: 8064/36938 -- GLOBAL_STEP: 820700 | > loss_text_ce: 0.030110105872154236 (0.03398178698343035) | > loss_mel_ce: 2.2387137413024902 (nan) | > loss: 0.009003269486129284 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1366129043024213) | > loader_time: 0.0045 (0.004640243415321603)  --> TIME: 2025-02-25 14:36:36 -- STEP: 8114/36938 -- GLOBAL_STEP: 820750 | > loss_text_ce: 0.03427222743630409 (0.0339769539044513) | > loss_mel_ce: 3.039290428161621 (nan) | > loss: 0.012196677736938 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13660382633876847) | > loader_time: 0.0045 (0.004640124365411879)  --> TIME: 2025-02-25 14:36:45 -- STEP: 8164/36938 -- GLOBAL_STEP: 820800 | > loss_text_ce: 0.04000712186098099 (0.033976788541652446) | > loss_mel_ce: 3.4970703125 (nan) | > loss: 0.014036022126674652 (nan) | > current_lr: 5e-06 | > step_time: 0.1832 (0.13659255747582524) | > loader_time: 0.0048 (0.00464027173842709)  --> TIME: 2025-02-25 14:36:54 -- STEP: 8214/36938 -- GLOBAL_STEP: 820850 | > loss_text_ce: 0.0406220480799675 (0.033975877588542626) | > loss_mel_ce: 2.7346835136413574 (nan) | > loss: 0.011013117618858814 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.136599261945897) | > loader_time: 0.0045 (0.00464013283462281)  --> TIME: 2025-02-25 14:37:04 -- STEP: 8264/36938 -- GLOBAL_STEP: 820900 | > loss_text_ce: 0.04155883565545082 (0.03398377712311563) | > loss_mel_ce: 3.2605011463165283 (nan) | > loss: 0.01310341339558363 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.13658351874536165) | > loader_time: 0.0046 (0.0046399363820573405)  --> TIME: 2025-02-25 14:37:12 -- STEP: 8314/36938 -- GLOBAL_STEP: 820950 | > loss_text_ce: 0.02765168994665146 (0.03398341648780216) | > loss_mel_ce: 3.0858848094940186 (nan) | > loss: 0.012355304323136806 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1365437924589481) | > loader_time: 0.0045 (0.0046397538778154826)  --> TIME: 2025-02-25 14:37:22 -- STEP: 8364/36938 -- GLOBAL_STEP: 821000 | > loss_text_ce: 0.04027332738041878 (0.03397721465018277) | > loss_mel_ce: 2.6495096683502197 (nan) | > loss: 0.01067374274134636 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1365343333715918) | > loader_time: 0.0046 (0.004639735836437687)  --> TIME: 2025-02-25 14:37:31 -- STEP: 8414/36938 -- GLOBAL_STEP: 821050 | > loss_text_ce: 0.023696817457675934 (0.03397567677387969) | > loss_mel_ce: 3.01348614692688 (nan) | > loss: 0.012052314355969429 (nan) | > current_lr: 5e-06 | > step_time: 0.1461 (0.136520520961282) | > loader_time: 0.0046 (0.004639708035230789)  --> TIME: 2025-02-25 14:37:40 -- STEP: 8464/36938 -- GLOBAL_STEP: 821100 | > loss_text_ce: 0.043220214545726776 (0.03397464162539352) | > loss_mel_ce: 2.651068687438965 (nan) | > loss: 0.010691623203456402 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13651953761541574) | > loader_time: 0.0045 (0.004639554339230382)  --> TIME: 2025-02-25 14:37:49 -- STEP: 8514/36938 -- GLOBAL_STEP: 821150 | > loss_text_ce: 0.04127135127782822 (0.0339762551266116) | > loss_mel_ce: 2.8680195808410645 (nan) | > loss: 0.011544805951416492 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13650636186279394) | > loader_time: 0.0045 (0.004639362740018192)  --> TIME: 2025-02-25 14:37:58 -- STEP: 8564/36938 -- GLOBAL_STEP: 821200 | > loss_text_ce: 0.03387751430273056 (0.03397482436875445) | > loss_mel_ce: 0.05535888671875 (nan) | > loss: 0.0003541127371136099 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13647213846876607) | > loader_time: 0.0046 (0.0046398695636608355)  --> TIME: 2025-02-25 14:38:08 -- STEP: 8614/36938 -- GLOBAL_STEP: 821250 | > loss_text_ce: 0.03886973112821579 (0.03397678141801117) | > loss_mel_ce: 3.111907720565796 (nan) | > loss: 0.012503085657954216 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.1364809265816493) | > loader_time: 0.0048 (0.004639783618564738)  --> TIME: 2025-02-25 14:38:17 -- STEP: 8664/36938 -- GLOBAL_STEP: 821300 | > loss_text_ce: 0.03474712371826172 (0.03398155972459595) | > loss_mel_ce: 2.76755690574646 (nan) | > loss: 0.01112025510519743 (nan) | > current_lr: 5e-06 | > step_time: 0.1721 (0.13648121695989626) | > loader_time: 0.0047 (0.00463979539228387)  --> TIME: 2025-02-25 14:38:26 -- STEP: 8714/36938 -- GLOBAL_STEP: 821350 | > loss_text_ce: 0.04032183066010475 (0.033975286512715364) | > loss_mel_ce: 2.011543035507202 (nan) | > loss: 0.00814232137054205 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13647728990135008) | > loader_time: 0.0045 (0.004639516052971023)  --> TIME: 2025-02-25 14:38:35 -- STEP: 8764/36938 -- GLOBAL_STEP: 821400 | > loss_text_ce: 0.02348647266626358 (0.033969768562529715) | > loss_mel_ce: 2.127437114715576 (nan) | > loss: 0.008535411208868027 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13648445488709077) | > loader_time: 0.0045 (0.004639520159816686)  --> TIME: 2025-02-25 14:38:45 -- STEP: 8814/36938 -- GLOBAL_STEP: 821450 | > loss_text_ce: 0.042884137481451035 (0.033968347106792494) | > loss_mel_ce: 0.3177446722984314 (nan) | > loss: 0.0014310667756944895 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.1364639178138216) | > loader_time: 0.0044 (0.004639369656446154)  --> TIME: 2025-02-25 14:38:54 -- STEP: 8864/36938 -- GLOBAL_STEP: 821500 | > loss_text_ce: 0.0365586094558239 (0.03397174179322836) | > loss_mel_ce: 3.0245256423950195 (nan) | > loss: 0.01214716024696827 (nan) | > current_lr: 5e-06 | > step_time: 0.1761 (0.13645861499576367) | > loader_time: 0.0048 (0.004639323625969102)  --> TIME: 2025-02-25 14:39:03 -- STEP: 8914/36938 -- GLOBAL_STEP: 821550 | > loss_text_ce: 0.0327899307012558 (0.0339667126396368) | > loss_mel_ce: 0.09663085639476776 (nan) | > loss: 0.0005135745741426945 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13646261283360137) | > loader_time: 0.0046 (0.004639111507733293)  --> TIME: 2025-02-25 14:39:12 -- STEP: 8964/36938 -- GLOBAL_STEP: 821600 | > loss_text_ce: 0.028098199516534805 (0.03396835845085971) | > loss_mel_ce: 2.957681179046631 (nan) | > loss: 0.011848331429064274 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.1364268947530678) | > loader_time: 0.0046 (0.004638807654433982)  --> TIME: 2025-02-25 14:39:21 -- STEP: 9014/36938 -- GLOBAL_STEP: 821650 | > loss_text_ce: 0.03605029359459877 (0.033964301962244266) | > loss_mel_ce: 0.1259562224149704 (nan) | > loss: 0.0006428830092772841 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13641647364682227) | > loader_time: 0.0045 (0.004638662141470248)  --> TIME: 2025-02-25 14:39:30 -- STEP: 9064/36938 -- GLOBAL_STEP: 821700 | > loss_text_ce: 0.03896879032254219 (0.03396228608771472) | > loss_mel_ce: 3.2497053146362305 (nan) | > loss: 0.013050294481217861 (nan) | > current_lr: 5e-06 | > step_time: 0.1577 (0.1363977707021552) | > loader_time: 0.0047 (0.004638547220802467)  --> TIME: 2025-02-25 14:39:40 -- STEP: 9114/36938 -- GLOBAL_STEP: 821750 | > loss_text_ce: 0.026397190988063812 (0.03396636000855859) | > loss_mel_ce: 0.5118719935417175 (nan) | > loss: 0.0021359887905418873 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13639192573852088) | > loader_time: 0.0046 (0.004638669834551323)  --> TIME: 2025-02-25 14:39:49 -- STEP: 9164/36938 -- GLOBAL_STEP: 821800 | > loss_text_ce: 0.03396453335881233 (0.03396255077366536) | > loss_mel_ce: 3.105164051055908 (nan) | > loss: 0.012456860393285751 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.1363913879962741) | > loader_time: 0.0045 (0.004638601395304357)  --> TIME: 2025-02-25 14:39:58 -- STEP: 9214/36938 -- GLOBAL_STEP: 821850 | > loss_text_ce: 0.037545789033174515 (0.03396034069943116) | > loss_mel_ce: 2.423121452331543 (nan) | > loss: 0.009764553047716618 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13639432376859922) | > loader_time: 0.0047 (0.004638467172440918)  --> TIME: 2025-02-25 14:40:08 -- STEP: 9264/36938 -- GLOBAL_STEP: 821900 | > loss_text_ce: 0.03178364783525467 (0.033961198060042996) | > loss_mel_ce: 3.2134530544281006 (nan) | > loss: 0.01287792343646288 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13640595977685202) | > loader_time: 0.0046 (0.004638499881113528)  --> TIME: 2025-02-25 14:40:17 -- STEP: 9314/36938 -- GLOBAL_STEP: 821950 | > loss_text_ce: 0.03713062405586243 (0.03396159036587763) | > loss_mel_ce: 0.0692138671875 (nan) | > loss: 0.0004220019618514925 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13644059676583398) | > loader_time: 0.0046 (0.004638721150903438)  --> TIME: 2025-02-25 14:40:26 -- STEP: 9364/36938 -- GLOBAL_STEP: 822000 | > loss_text_ce: 0.018707560375332832 (0.03395628779036509) | > loss_mel_ce: 1.2170149087905884 (nan) | > loss: 0.004903660621494055 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13641774562728143) | > loader_time: 0.0046 (0.004638830880128825)  --> TIME: 2025-02-25 14:40:36 -- STEP: 9414/36938 -- GLOBAL_STEP: 822050 | > loss_text_ce: 0.027698533609509468 (0.033959274801708725) | > loss_mel_ce: 2.361950635910034 (nan) | > loss: 0.009482734836637974 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13644378238963248) | > loader_time: 0.0048 (0.004639066732028536)  --> TIME: 2025-02-25 14:40:45 -- STEP: 9464/36938 -- GLOBAL_STEP: 822100 | > loss_text_ce: 0.033930547535419464 (0.03396356295060523) | > loss_mel_ce: 2.7671661376953125 (nan) | > loss: 0.011115463450551033 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.13643359896094742) | > loader_time: 0.0045 (0.004639076486242481)  --> TIME: 2025-02-25 14:40:54 -- STEP: 9514/36938 -- GLOBAL_STEP: 822150 | > loss_text_ce: 0.022832421585917473 (0.03395542292038298) | > loss_mel_ce: 1.0204471349716187 (nan) | > loss: 0.004139998462051153 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13643448481119722) | > loader_time: 0.0046 (0.004639098793111568)  --> TIME: 2025-02-25 14:41:04 -- STEP: 9564/36938 -- GLOBAL_STEP: 822200 | > loss_text_ce: 0.03430189564824104 (0.03396249630744362) | > loss_mel_ce: 2.3033926486968994 (nan) | > loss: 0.00927656702697277 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1364482592359742) | > loader_time: 0.0046 (0.004639222576048811)  --> TIME: 2025-02-25 14:41:13 -- STEP: 9614/36938 -- GLOBAL_STEP: 822250 | > loss_text_ce: 0.03700300306081772 (0.033964242716615314) | > loss_mel_ce: 3.0668554306030273 (nan) | > loss: 0.012316899374127388 (nan) | > current_lr: 5e-06 | > step_time: 0.1799 (0.13647246457495338) | > loader_time: 0.0049 (0.0046394295372039355)  --> TIME: 2025-02-25 14:41:23 -- STEP: 9664/36938 -- GLOBAL_STEP: 822300 | > loss_text_ce: 0.0335719995200634 (0.03396188428441205) | > loss_mel_ce: 2.375401020050049 (nan) | > loss: 0.009559417143464088 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.136471698615725) | > loader_time: 0.0046 (0.004639467409509689)  --> TIME: 2025-02-25 14:41:32 -- STEP: 9714/36938 -- GLOBAL_STEP: 822350 | > loss_text_ce: 0.025685658678412437 (0.033961893006198965) | > loss_mel_ce: 3.302273988723755 (nan) | > loss: 0.01320618949830532 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13650529885797694) | > loader_time: 0.0045 (0.004639775070212416)  --> TIME: 2025-02-25 14:41:42 -- STEP: 9764/36938 -- GLOBAL_STEP: 822400 | > loss_text_ce: 0.03224298357963562 (0.033962883676346214) | > loss_mel_ce: 2.284994125366211 (nan) | > loss: 0.009195386432111263 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13652532220814703) | > loader_time: 0.0045 (0.004639996582736051)  --> TIME: 2025-02-25 14:41:51 -- STEP: 9814/36938 -- GLOBAL_STEP: 822450 | > loss_text_ce: 0.036717116832733154 (0.03396229742222797) | > loss_mel_ce: 3.4198036193847656 (nan) | > loss: 0.013716353103518486 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13652211594100613) | > loader_time: 0.0046 (0.004640171623579804)  --> TIME: 2025-02-25 14:42:00 -- STEP: 9864/36938 -- GLOBAL_STEP: 822500 | > loss_text_ce: 0.038797199726104736 (0.03395645225130549) | > loss_mel_ce: 2.961980104446411 (nan) | > loss: 0.011907846666872501 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13651703246125174) | > loader_time: 0.0046 (0.0046402003498262935)  --> TIME: 2025-02-25 14:42:10 -- STEP: 9914/36938 -- GLOBAL_STEP: 822550 | > loss_text_ce: 0.033742647618055344 (0.03395798844893105) | > loss_mel_ce: 2.674201726913452 (nan) | > loss: 0.010745811276137829 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13652919700892702) | > loader_time: 0.0046 (0.004640255336057323)  --> TIME: 2025-02-25 14:42:19 -- STEP: 9964/36938 -- GLOBAL_STEP: 822600 | > loss_text_ce: 0.03209199383854866 (0.03395527536356063) | > loss_mel_ce: 3.395022392272949 (nan) | > loss: 0.01359966117888689 (nan) | > current_lr: 5e-06 | > step_time: 0.1784 (0.13652027646301765) | > loader_time: 0.005 (0.0046403052241196265)  --> TIME: 2025-02-25 14:42:28 -- STEP: 10014/36938 -- GLOBAL_STEP: 822650 | > loss_text_ce: 0.037472035735845566 (0.033954984997273584) | > loss_mel_ce: 1.6059901714324951 (nan) | > loss: 0.006521675735712051 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13651377249554503) | > loader_time: 0.0045 (0.004640312758610104)  --> TIME: 2025-02-25 14:42:38 -- STEP: 10064/36938 -- GLOBAL_STEP: 822700 | > loss_text_ce: 0.04013397917151451 (0.03395374536526126) | > loss_mel_ce: 1.298160195350647 (nan) | > loss: 0.005310691427439451 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13652665422717059) | > loader_time: 0.0046 (0.004640272837754073)  --> TIME: 2025-02-25 14:42:47 -- STEP: 10114/36938 -- GLOBAL_STEP: 822750 | > loss_text_ce: 0.027701659128069878 (0.03395299703863996) | > loss_mel_ce: 3.19913387298584 (nan) | > loss: 0.012804903090000153 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13652259287024224) | > loader_time: 0.0046 (0.004640271005032668)  --> TIME: 2025-02-25 14:42:56 -- STEP: 10164/36938 -- GLOBAL_STEP: 822800 | > loss_text_ce: 0.03611430898308754 (0.033952542319428326) | > loss_mel_ce: 3.7439236640930176 (nan) | > loss: 0.015000151470303535 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13652903409231315) | > loader_time: 0.0045 (0.00464030742551434)  --> TIME: 2025-02-25 14:43:06 -- STEP: 10214/36938 -- GLOBAL_STEP: 822850 | > loss_text_ce: 0.03725215047597885 (0.033948591577765606) | > loss_mel_ce: 3.1290974617004395 (nan) | > loss: 0.012564879842102528 (nan) | > current_lr: 5e-06 | > step_time: 0.1818 (0.13654269954101816) | > loader_time: 0.0049 (0.004640495914849876)  --> TIME: 2025-02-25 14:43:15 -- STEP: 10264/36938 -- GLOBAL_STEP: 822900 | > loss_text_ce: 0.03546249493956566 (0.033943290169719545) | > loss_mel_ce: 0.024504471570253372 (nan) | > loss: 0.0002379641664447263 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13653241020111268) | > loader_time: 0.0046 (0.0046405686546096645)  --> TIME: 2025-02-25 14:43:25 -- STEP: 10314/36938 -- GLOBAL_STEP: 822950 | > loss_text_ce: 0.02673274278640747 (0.03394048280039841) | > loss_mel_ce: 3.1518750190734863 (nan) | > loss: 0.012613523751497269 (nan) | > current_lr: 5e-06 | > step_time: 0.1915 (0.13654138943888786) | > loader_time: 0.0049 (0.004640864151634161)  --> TIME: 2025-02-25 14:43:34 -- STEP: 10364/36938 -- GLOBAL_STEP: 823000 | > loss_text_ce: 0.023152103647589684 (0.03393942940739725) | > loss_mel_ce: 3.127046823501587 (nan) | > loss: 0.012500789947807789 (nan) | > current_lr: 5e-06 | > step_time: 0.1605 (0.13654664946725092) | > loader_time: 0.0047 (0.004641045018633734)  --> TIME: 2025-02-25 14:43:43 -- STEP: 10414/36938 -- GLOBAL_STEP: 823050 | > loss_text_ce: 0.03286723047494888 (0.03394361040366243) | > loss_mel_ce: 2.518782615661621 (nan) | > loss: 0.010125595144927502 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13653613437222598) | > loader_time: 0.0047 (0.004640960844460054)  --> TIME: 2025-02-25 14:43:53 -- STEP: 10464/36938 -- GLOBAL_STEP: 823100 | > loss_text_ce: 0.030229680240154266 (0.033942900161491714) | > loss_mel_ce: 2.670701742172241 (nan) | > loss: 0.010717982426285744 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.1365419613932256) | > loader_time: 0.0046 (0.004641041752030712)  --> TIME: 2025-02-25 14:44:02 -- STEP: 10514/36938 -- GLOBAL_STEP: 823150 | > loss_text_ce: 0.04184460639953613 (0.033948739536977496) | > loss_mel_ce: 2.497959613800049 (nan) | > loss: 0.010078588500618935 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13653129601129407) | > loader_time: 0.0046 (0.0046410149940184385)  --> TIME: 2025-02-25 14:44:11 -- STEP: 10564/36938 -- GLOBAL_STEP: 823200 | > loss_text_ce: 0.030142327770590782 (0.03394468966809786) | > loss_mel_ce: 2.4199235439300537 (nan) | > loss: 0.009722484275698662 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.13652729771436092) | > loader_time: 0.0047 (0.004641096617226695)  --> TIME: 2025-02-25 14:44:20 -- STEP: 10614/36938 -- GLOBAL_STEP: 823250 | > loss_text_ce: 0.035309355705976486 (0.033946926387815546) | > loss_mel_ce: 3.292663097381592 (nan) | > loss: 0.013206240721046925 (nan) | > current_lr: 5e-06 | > step_time: 0.204 (0.13652606241353113) | > loader_time: 0.0049 (0.004641287875350673)  --> TIME: 2025-02-25 14:44:30 -- STEP: 10664/36938 -- GLOBAL_STEP: 823300 | > loss_text_ce: 0.030653947964310646 (0.033942504328066295) | > loss_mel_ce: 2.5547475814819336 (nan) | > loss: 0.010259530507028103 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13652565239160877) | > loader_time: 0.0046 (0.004641283904352491)  --> TIME: 2025-02-25 14:44:39 -- STEP: 10714/36938 -- GLOBAL_STEP: 823350 | > loss_text_ce: 0.03046475537121296 (0.03394582267930259) | > loss_mel_ce: 2.5252459049224854 (nan) | > loss: 0.010141708888113499 (nan) | > current_lr: 5e-06 | > step_time: 0.1447 (0.1365383980054791) | > loader_time: 0.0047 (0.004641626783331446)  --> TIME: 2025-02-25 14:44:49 -- STEP: 10764/36938 -- GLOBAL_STEP: 823400 | > loss_text_ce: 0.03468593209981918 (0.03393426556020782) | > loss_mel_ce: 2.825920581817627 (nan) | > loss: 0.011351613327860832 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13654264627181692) | > loader_time: 0.0046 (0.004641736142921514)  --> TIME: 2025-02-25 14:44:58 -- STEP: 10814/36938 -- GLOBAL_STEP: 823450 | > loss_text_ce: 0.026560267433524132 (0.03392678832401719) | > loss_mel_ce: 2.754706621170044 (nan) | > loss: 0.011036774143576622 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13653040355557536) | > loader_time: 0.0046 (0.004641927278524669)  --> TIME: 2025-02-25 14:45:07 -- STEP: 10864/36938 -- GLOBAL_STEP: 823500 | > loss_text_ce: 0.03755391016602516 (0.03392579230873887) | > loss_mel_ce: 3.018657922744751 (nan) | > loss: 0.012127825990319252 (nan) | > current_lr: 5e-06 | > step_time: 0.1859 (0.136551309623845) | > loader_time: 0.0049 (0.0046421745476561155)  --> TIME: 2025-02-25 14:45:16 -- STEP: 10914/36938 -- GLOBAL_STEP: 823550 | > loss_text_ce: 0.022279715165495872 (0.03392740844970607) | > loss_mel_ce: 2.6547319889068604 (nan) | > loss: 0.010623062960803509 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.136544012512199) | > loader_time: 0.0046 (0.00464238488283134)  --> TIME: 2025-02-25 14:45:26 -- STEP: 10964/36938 -- GLOBAL_STEP: 823600 | > loss_text_ce: 0.04106738418340683 (0.03393211315653738) | > loss_mel_ce: 2.5607388019561768 (nan) | > loss: 0.010324628092348576 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.1365490021256803) | > loader_time: 0.0047 (0.004642610087180733)  --> TIME: 2025-02-25 14:45:35 -- STEP: 11014/36938 -- GLOBAL_STEP: 823650 | > loss_text_ce: 0.03542126342654228 (0.03392919734235704) | > loss_mel_ce: 3.266019105911255 (nan) | > loss: 0.013100954703986645 (nan) | > current_lr: 5e-06 | > step_time: 0.1493 (0.13655676161158423) | > loader_time: 0.0047 (0.004642635221292557)  --> TIME: 2025-02-25 14:45:45 -- STEP: 11064/36938 -- GLOBAL_STEP: 823700 | > loss_text_ce: 0.034398119896650314 (0.03393342383638556) | > loss_mel_ce: 2.8416852951049805 (nan) | > loss: 0.011413030326366425 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13656080210889884) | > loader_time: 0.0047 (0.004642788538963825)  --> TIME: 2025-02-25 14:45:54 -- STEP: 11114/36938 -- GLOBAL_STEP: 823750 | > loss_text_ce: 0.04118463769555092 (0.03393395229815747) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1365411037460861) | > loader_time: 0.0045 (0.0046429807856204295)  --> TIME: 2025-02-25 14:46:03 -- STEP: 11164/36938 -- GLOBAL_STEP: 823800 | > loss_text_ce: 0.04000120609998703 (0.03393076873946346) | > loss_mel_ce: 3.1675806045532227 (nan) | > loss: 0.012728500179946423 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13654957010685387) | > loader_time: 0.0047 (0.0046429914712307905)  --> TIME: 2025-02-25 14:46:12 -- STEP: 11214/36938 -- GLOBAL_STEP: 823850 | > loss_text_ce: 0.05172329768538475 (0.033930223449032855) | > loss_mel_ce: 3.274956464767456 (nan) | > loss: 0.013201110996305943 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.1365370937097049) | > loader_time: 0.0046 (0.004642863164765321)  --> TIME: 2025-02-25 14:46:22 -- STEP: 11264/36938 -- GLOBAL_STEP: 823900 | > loss_text_ce: 0.0319482646882534 (0.0339295399952872) | > loss_mel_ce: 2.7283565998077393 (nan) | > loss: 0.010953591205179691 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13656294064342325) | > loader_time: 0.0045 (0.004643036899241528)  --> TIME: 2025-02-25 14:46:31 -- STEP: 11314/36938 -- GLOBAL_STEP: 823950 | > loss_text_ce: 0.03231118246912956 (0.03393036058858775) | > loss_mel_ce: 1.5099376440048218 (nan) | > loss: 0.006120035424828529 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.1365745586832235) | > loader_time: 0.0047 (0.004643406529971936)  --> TIME: 2025-02-25 14:46:40 -- STEP: 11364/36938 -- GLOBAL_STEP: 824000 | > loss_text_ce: 0.03472330793738365 (0.03393212980108876) | > loss_mel_ce: 2.861229419708252 (nan) | > loss: 0.011491876095533371 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.1365560746159367) | > loader_time: 0.0046 (0.004643357207094508)  --> TIME: 2025-02-25 14:46:50 -- STEP: 11414/36938 -- GLOBAL_STEP: 824050 | > loss_text_ce: 0.027097096666693687 (0.033933497218335884) | > loss_mel_ce: 0.18701171875 (nan) | > loss: 0.000849638192448765 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13657940946361424) | > loader_time: 0.0046 (0.004643391388946091)  --> TIME: 2025-02-25 14:46:59 -- STEP: 11464/36938 -- GLOBAL_STEP: 824100 | > loss_text_ce: 0.02946745790541172 (0.0339321595679518) | > loss_mel_ce: 0.2027130126953125 (nan) | > loss: 0.0009213511366397142 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13657879315717028) | > loader_time: 0.0045 (0.004643412274409234)  --> TIME: 2025-02-25 14:47:08 -- STEP: 11514/36938 -- GLOBAL_STEP: 824150 | > loss_text_ce: 0.030899500474333763 (0.03393053231532442) | > loss_mel_ce: 2.776219367980957 (nan) | > loss: 0.011139361187815666 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.1365878922214149) | > loader_time: 0.0046 (0.004643364356005998)  --> TIME: 2025-02-25 14:47:17 -- STEP: 11564/36938 -- GLOBAL_STEP: 824200 | > loss_text_ce: 0.03094695508480072 (0.03392516022581264) | > loss_mel_ce: 2.5990090370178223 (nan) | > loss: 0.01043633371591568 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13657127021300036) | > loader_time: 0.0046 (0.00464329534812354)  --> TIME: 2025-02-25 14:47:27 -- STEP: 11614/36938 -- GLOBAL_STEP: 824250 | > loss_text_ce: 0.03325236961245537 (0.033924135404592645) | > loss_mel_ce: 1.8972852230072021 (nan) | > loss: 0.007660863921046257 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.13656518490934058) | > loader_time: 0.0047 (0.004643511809107148)  --> TIME: 2025-02-25 14:47:36 -- STEP: 11664/36938 -- GLOBAL_STEP: 824300 | > loss_text_ce: 0.03570755943655968 (0.033921999600125124) | > loss_mel_ce: 3.1579103469848633 (nan) | > loss: 0.012673087418079376 (nan) | > current_lr: 5e-06 | > step_time: 0.1628 (0.13654568595160205) | > loader_time: 0.0047 (0.0046438245493688685)  --> TIME: 2025-02-25 14:47:45 -- STEP: 11714/36938 -- GLOBAL_STEP: 824350 | > loss_text_ce: 0.019639503210783005 (0.03392079881635235) | > loss_mel_ce: 2.481273889541626 (nan) | > loss: 0.009924259968101978 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1365416120583236) | > loader_time: 0.0044 (0.004643814950871021)  --> TIME: 2025-02-25 14:47:54 -- STEP: 11764/36938 -- GLOBAL_STEP: 824400 | > loss_text_ce: 0.04637707769870758 (0.03392261571446564) | > loss_mel_ce: 3.187410831451416 (nan) | > loss: 0.012832492589950562 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13654132981318226) | > loader_time: 0.0046 (0.004643731622297071)  --> TIME: 2025-02-25 14:48:04 -- STEP: 11814/36938 -- GLOBAL_STEP: 824450 | > loss_text_ce: 0.02640918828547001 (0.03392082006166489) | > loss_mel_ce: 2.5430679321289062 (nan) | > loss: 0.010196338407695293 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13653686144680124) | > loader_time: 0.0047 (0.004643856601221058)  --> TIME: 2025-02-25 14:48:13 -- STEP: 11864/36938 -- GLOBAL_STEP: 824500 | > loss_text_ce: 0.024734802544116974 (0.03391341723864986) | > loss_mel_ce: 1.5113027095794678 (nan) | > loss: 0.006095387507230043 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1365659134217558) | > loader_time: 0.0046 (0.0046444092139310295)  --> TIME: 2025-02-25 14:48:23 -- STEP: 11914/36938 -- GLOBAL_STEP: 824550 | > loss_text_ce: 0.02892044000327587 (0.03391023124961371) | > loss_mel_ce: 0.7302383780479431 (nan) | > loss: 0.003012535162270069 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1365636289369868) | > loader_time: 0.0046 (0.00464449425921937)  --> TIME: 2025-02-25 14:48:32 -- STEP: 11964/36938 -- GLOBAL_STEP: 824600 | > loss_text_ce: 0.03870544210076332 (0.03390912441868406) | > loss_mel_ce: 2.396301507949829 (nan) | > loss: 0.00966272596269846 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.1365620106687838) | > loader_time: 0.0046 (0.004644498881668756)  --> TIME: 2025-02-25 14:48:41 -- STEP: 12014/36938 -- GLOBAL_STEP: 824650 | > loss_text_ce: 0.03192658722400665 (0.033908514685873706) | > loss_mel_ce: 2.327000379562378 (nan) | > loss: 0.009360821917653084 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13656020218071438) | > loader_time: 0.0046 (0.004644665560111918)  --> TIME: 2025-02-25 14:48:50 -- STEP: 12064/36938 -- GLOBAL_STEP: 824700 | > loss_text_ce: 0.03358973562717438 (0.03390645619970633) | > loss_mel_ce: 0.5084304809570312 (nan) | > loss: 0.002150873886421323 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13654601722165213) | > loader_time: 0.0046 (0.004644785955824964)  --> TIME: 2025-02-25 14:49:00 -- STEP: 12114/36938 -- GLOBAL_STEP: 824750 | > loss_text_ce: 0.04244155436754227 (0.033900915271811334) | > loss_mel_ce: 3.105515718460083 (nan) | > loss: 0.012491894885897636 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13654648033486608) | > loader_time: 0.0045 (0.004644599865426396)  --> TIME: 2025-02-25 14:49:09 -- STEP: 12164/36938 -- GLOBAL_STEP: 824800 | > loss_text_ce: 0.039575543254613876 (0.03390674554300039) | > loss_mel_ce: 2.816016912460327 (nan) | > loss: 0.011331716552376747 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13655482799510615) | > loader_time: 0.0046 (0.004644438394077979)  --> TIME: 2025-02-25 14:49:18 -- STEP: 12214/36938 -- GLOBAL_STEP: 824850 | > loss_text_ce: 0.037637729197740555 (0.03390740602346924) | > loss_mel_ce: 2.736023426055908 (nan) | > loss: 0.011006592772901058 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.13655620293158954) | > loader_time: 0.0046 (0.00464424556809008)  --> TIME: 2025-02-25 14:49:28 -- STEP: 12264/36938 -- GLOBAL_STEP: 824900 | > loss_text_ce: 0.03929274156689644 (0.03390926539732988) | > loss_mel_ce: 0.16254836320877075 (nan) | > loss: 0.0008009567973203957 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.13655247462005807) | > loader_time: 0.0046 (0.004643974277760097)  --> TIME: 2025-02-25 14:49:37 -- STEP: 12314/36938 -- GLOBAL_STEP: 824950 | > loss_text_ce: 0.03285271301865578 (0.033907055410615436) | > loss_mel_ce: 2.849647045135498 (nan) | > loss: 0.011438491754233837 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13655708926982962) | > loader_time: 0.0046 (0.0046438539068826955)  --> TIME: 2025-02-25 14:49:46 -- STEP: 12364/36938 -- GLOBAL_STEP: 825000 | > loss_text_ce: 0.028937654569745064 (0.03390440956384509) | > loss_mel_ce: 2.789407730102539 (nan) | > loss: 0.011183911003172398 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13653690322862444) | > loader_time: 0.0046 (0.004643748644215113)  --> TIME: 2025-02-25 14:49:55 -- STEP: 12414/36938 -- GLOBAL_STEP: 825050 | > loss_text_ce: 0.03600846603512764 (0.03390160075330188) | > loss_mel_ce: 3.2993764877319336 (nan) | > loss: 0.013235654681921005 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13653613654084937) | > loader_time: 0.0045 (0.004643655119069773)  --> TIME: 2025-02-25 14:50:05 -- STEP: 12464/36938 -- GLOBAL_STEP: 825100 | > loss_text_ce: 0.031233321875333786 (0.033903641045581974) | > loss_mel_ce: 2.698406934738159 (nan) | > loss: 0.010831906460225582 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.13655203609549246) | > loader_time: 0.0048 (0.004643783164575243)  --> TIME: 2025-02-25 14:50:14 -- STEP: 12514/36938 -- GLOBAL_STEP: 825150 | > loss_text_ce: 0.020432038232684135 (0.033904367270234256) | > loss_mel_ce: 2.4063074588775635 (nan) | > loss: 0.009629919193685055 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13656841212994095) | > loader_time: 0.0046 (0.004643896774150538)  --> TIME: 2025-02-25 14:50:23 -- STEP: 12564/36938 -- GLOBAL_STEP: 825200 | > loss_text_ce: 0.02141742594540119 (0.03389996110799077) | > loss_mel_ce: 1.492449164390564 (nan) | > loss: 0.006007407326251268 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13655199102443083) | > loader_time: 0.0045 (0.004644151118787663)  --> TIME: 2025-02-25 14:50:33 -- STEP: 12614/36938 -- GLOBAL_STEP: 825250 | > loss_text_ce: 0.033162057399749756 (0.033898801171958114) | > loss_mel_ce: 2.9643404483795166 (nan) | > loss: 0.011894851922988892 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13654610463135344) | > loader_time: 0.0046 (0.00464410845746096)  --> TIME: 2025-02-25 14:50:42 -- STEP: 12664/36938 -- GLOBAL_STEP: 825300 | > loss_text_ce: 0.035243816673755646 (0.03390116175183338) | > loss_mel_ce: 2.4450676441192627 (nan) | > loss: 0.009842505678534508 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13653242328237808) | > loader_time: 0.0045 (0.00464406078628399)  --> TIME: 2025-02-25 14:50:51 -- STEP: 12714/36938 -- GLOBAL_STEP: 825350 | > loss_text_ce: 0.03445449098944664 (0.03390254331732141) | > loss_mel_ce: 3.244389295578003 (nan) | > loss: 0.013011286035180092 (nan) | > current_lr: 5e-06 | > step_time: 0.1855 (0.1365427908065548) | > loader_time: 0.0048 (0.004644414548587352)  --> TIME: 2025-02-25 14:51:01 -- STEP: 12764/36938 -- GLOBAL_STEP: 825400 | > loss_text_ce: 0.02571253664791584 (0.033905155120349695) | > loss_mel_ce: 2.8037543296813965 (nan) | > loss: 0.011228043586015701 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13655284660731537) | > loader_time: 0.0045 (0.004644511224931391)  --> TIME: 2025-02-25 14:51:10 -- STEP: 12814/36938 -- GLOBAL_STEP: 825450 | > loss_text_ce: 0.03288921341300011 (0.03389660754001636) | > loss_mel_ce: 4.0 (nan) | > loss: 0.016003530472517014 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.1365400635174668) | > loader_time: 0.0046 (0.004644357062212212)  --> TIME: 2025-02-25 14:51:19 -- STEP: 12864/36938 -- GLOBAL_STEP: 825500 | > loss_text_ce: 0.03199327364563942 (0.033895192939767206) | > loss_mel_ce: 3.38671875 (nan) | > loss: 0.013566318899393082 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.1365563226910083) | > loader_time: 0.0045 (0.004644318154795247)  --> TIME: 2025-02-25 14:51:29 -- STEP: 12914/36938 -- GLOBAL_STEP: 825550 | > loss_text_ce: 0.022358814254403114 (0.03389601454494139) | > loss_mel_ce: 1.555932641029358 (nan) | > loss: 0.006263061426579952 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13657952459009576) | > loader_time: 0.0046 (0.004644296478335283)  --> TIME: 2025-02-25 14:51:38 -- STEP: 12964/36938 -- GLOBAL_STEP: 825600 | > loss_text_ce: 0.03436056151986122 (0.033892946802634154) | > loss_mel_ce: 2.128200054168701 (nan) | > loss: 0.008581590838730335 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13656978298579864) | > loader_time: 0.0046 (0.004644119198842384)  --> TIME: 2025-02-25 14:51:48 -- STEP: 13014/36938 -- GLOBAL_STEP: 825650 | > loss_text_ce: 0.03019482083618641 (0.03389381595845871) | > loss_mel_ce: 1.6211141347885132 (nan) | > loss: 0.006552814040333033 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13658367808887117) | > loader_time: 0.0046 (0.004644037868563652)  --> TIME: 2025-02-25 14:51:57 -- STEP: 13064/36938 -- GLOBAL_STEP: 825700 | > loss_text_ce: 0.033779822289943695 (0.03389517491660826) | > loss_mel_ce: 2.3648324012756348 (nan) | > loss: 0.009518302977085114 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13659874688827184) | > loader_time: 0.0045 (0.004643935589285179)  --> TIME: 2025-02-25 14:52:07 -- STEP: 13114/36938 -- GLOBAL_STEP: 825750 | > loss_text_ce: 0.03210371360182762 (0.033902789956742355) | > loss_mel_ce: 2.7651615142822266 (nan) | > loss: 0.011100259609520435 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13662378444747333) | > loader_time: 0.0045 (0.004643946972419812)  --> TIME: 2025-02-25 14:52:17 -- STEP: 13164/36938 -- GLOBAL_STEP: 825800 | > loss_text_ce: 0.01709100604057312 (0.033903362501349725) | > loss_mel_ce: 1.5832797288894653 (nan) | > loss: 0.00635067792609334 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13663747036431773) | > loader_time: 0.0046 (0.004644002316027775)  --> TIME: 2025-02-25 14:52:26 -- STEP: 13214/36938 -- GLOBAL_STEP: 825850 | > loss_text_ce: 0.037276651710271835 (0.03390944614972328) | > loss_mel_ce: 3.5870535373687744 (nan) | > loss: 0.014382263645529747 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13665116607321973) | > loader_time: 0.0046 (0.004644114563031001)  --> TIME: 2025-02-25 14:52:35 -- STEP: 13264/36938 -- GLOBAL_STEP: 825900 | > loss_text_ce: 0.030344950035214424 (0.03391344222515099) | > loss_mel_ce: 2.7810654640197754 (nan) | > loss: 0.011156391352415085 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.13664608566479244) | > loader_time: 0.0049 (0.004644145741968871)  --> TIME: 2025-02-25 14:52:45 -- STEP: 13314/36938 -- GLOBAL_STEP: 825950 | > loss_text_ce: 0.034030620008707047 (0.033911315236278015) | > loss_mel_ce: 3.1912291049957275 (nan) | > loss: 0.012798651121556759 (nan) | > current_lr: 5e-06 | > step_time: 0.1697 (0.13664436202445446) | > loader_time: 0.0048 (0.004644603526536733)  --> TIME: 2025-02-25 14:52:54 -- STEP: 13364/36938 -- GLOBAL_STEP: 826000 | > loss_text_ce: 0.0330473929643631 (0.033911711607988) | > loss_mel_ce: 3.508355140686035 (nan) | > loss: 0.01405318547040224 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13664408921481905) | > loader_time: 0.0046 (0.0046447332267167424)  --> TIME: 2025-02-25 14:53:03 -- STEP: 13414/36938 -- GLOBAL_STEP: 826050 | > loss_text_ce: 0.035885997116565704 (0.03391066729809689) | > loss_mel_ce: 2.6187500953674316 (nan) | > loss: 0.010534270666539669 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13664311694018308) | > loader_time: 0.0045 (0.004644813455126928)  --> TIME: 2025-02-25 14:53:13 -- STEP: 13464/36938 -- GLOBAL_STEP: 826100 | > loss_text_ce: 0.038331322371959686 (0.03391056711092663) | > loss_mel_ce: 3.208502769470215 (nan) | > loss: 0.012884262949228287 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13665812502853739) | > loader_time: 0.0047 (0.004645180698667891)  --> TIME: 2025-02-25 14:53:22 -- STEP: 13514/36938 -- GLOBAL_STEP: 826150 | > loss_text_ce: 0.050812963396310806 (0.03390900618830168) | > loss_mel_ce: 3.5288355350494385 (nan) | > loss: 0.014204954728484154 (nan) | > current_lr: 5e-06 | > step_time: 0.177 (0.13667374047475198) | > loader_time: 0.0048 (0.004645225033880731)  --> TIME: 2025-02-25 14:53:31 -- STEP: 13564/36938 -- GLOBAL_STEP: 826200 | > loss_text_ce: 0.03286813572049141 (0.03390982836722196) | > loss_mel_ce: 3.1081533432006836 (nan) | > loss: 0.012464371509850025 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.13667199902392538) | > loader_time: 0.0049 (0.004645232217777996)  --> TIME: 2025-02-25 14:53:41 -- STEP: 13614/36938 -- GLOBAL_STEP: 826250 | > loss_text_ce: 0.029594669118523598 (0.033909656528005275) | > loss_mel_ce: 2.8768789768218994 (nan) | > loss: 0.011533626355230808 (nan) | > current_lr: 5e-06 | > step_time: 0.1725 (0.13667299032246402) | > loader_time: 0.0048 (0.004645490604331691)  --> TIME: 2025-02-25 14:53:50 -- STEP: 13664/36938 -- GLOBAL_STEP: 826300 | > loss_text_ce: 0.028116341680288315 (0.03391117243798538) | > loss_mel_ce: 2.6671550273895264 (nan) | > loss: 0.010695521719753742 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1366774433879168) | > loader_time: 0.0045 (0.004645627890593543)  --> TIME: 2025-02-25 14:54:00 -- STEP: 13714/36938 -- GLOBAL_STEP: 826350 | > loss_text_ce: 0.03273152559995651 (0.03390982387560534) | > loss_mel_ce: 2.255581855773926 (nan) | > loss: 0.009080609306693077 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13669014048329572) | > loader_time: 0.0047 (0.0046456376647782205)  --> TIME: 2025-02-25 14:54:09 -- STEP: 13764/36938 -- GLOBAL_STEP: 826400 | > loss_text_ce: 0.04408032447099686 (0.03391000763122047) | > loss_mel_ce: 2.734114408493042 (nan) | > loss: 0.01102458219975233 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13668171103975582) | > loader_time: 0.0046 (0.004645628019392185)  --> TIME: 2025-02-25 14:54:18 -- STEP: 13814/36938 -- GLOBAL_STEP: 826450 | > loss_text_ce: 0.03477973863482475 (0.03390536899459293) | > loss_mel_ce: 2.1876091957092285 (nan) | > loss: 0.008819004520773888 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.1366644291727944) | > loader_time: 0.0047 (0.004645716251711835)  --> TIME: 2025-02-25 14:54:28 -- STEP: 13864/36938 -- GLOBAL_STEP: 826500 | > loss_text_ce: 0.0376383401453495 (0.03390970799977366) | > loss_mel_ce: 3.5869619846343994 (nan) | > loss: 0.01438333559781313 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.1366857181927137) | > loader_time: 0.0047 (0.00464596219717664)  --> TIME: 2025-02-25 14:54:37 -- STEP: 13914/36938 -- GLOBAL_STEP: 826550 | > loss_text_ce: 0.040576525032520294 (0.03391018778699356) | > loss_mel_ce: 2.6354544162750244 (nan) | > loss: 0.01061917096376419 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13669126390536993) | > loader_time: 0.0048 (0.004646073474749681)  --> TIME: 2025-02-25 14:54:47 -- STEP: 13964/36938 -- GLOBAL_STEP: 826600 | > loss_text_ce: 0.03473994508385658 (0.033910208868982716) | > loss_mel_ce: 1.820274829864502 (nan) | > loss: 0.007361170370131731 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.13670579989637405) | > loader_time: 0.0046 (0.00464631801893156)  --> TIME: 2025-02-25 14:54:56 -- STEP: 14014/36938 -- GLOBAL_STEP: 826650 | > loss_text_ce: 0.03617456555366516 (0.033911841061427125) | > loss_mel_ce: 0.18596191704273224 (nan) | > loss: 0.0008814940229058266 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13670491088520448) | > loader_time: 0.0046 (0.004646349501061975)  --> TIME: 2025-02-25 14:55:05 -- STEP: 14064/36938 -- GLOBAL_STEP: 826700 | > loss_text_ce: 0.031003449112176895 (0.0339117705681868) | > loss_mel_ce: 3.2960267066955566 (nan) | > loss: 0.013202501460909843 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13669649263223066) | > loader_time: 0.0046 (0.004646306236584969)  --> TIME: 2025-02-25 14:55:15 -- STEP: 14114/36938 -- GLOBAL_STEP: 826750 | > loss_text_ce: 0.03512176498770714 (0.033914571665718235) | > loss_mel_ce: 2.9326930046081543 (nan) | > loss: 0.011777043342590332 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13671132608397263) | > loader_time: 0.0047 (0.004646276724951712)  --> TIME: 2025-02-25 14:55:24 -- STEP: 14164/36938 -- GLOBAL_STEP: 826800 | > loss_text_ce: 0.03259725868701935 (0.03390888145981952) | > loss_mel_ce: 2.9734346866607666 (nan) | > loss: 0.011928698979318142 (nan) | > current_lr: 5e-06 | > step_time: 0.108 (0.13669886110327925) | > loader_time: 0.0045 (0.004646141325070731)  --> TIME: 2025-02-25 14:55:33 -- STEP: 14214/36938 -- GLOBAL_STEP: 826850 | > loss_text_ce: 0.03592407703399658 (0.03390677721702588) | > loss_mel_ce: 3.249444007873535 (nan) | > loss: 0.013037174940109253 (nan) | > current_lr: 5e-06 | > step_time: 0.1892 (0.13670193654199028) | > loader_time: 0.0048 (0.004646464425400024)  --> TIME: 2025-02-25 14:55:43 -- STEP: 14264/36938 -- GLOBAL_STEP: 826900 | > loss_text_ce: 0.02936253324151039 (0.03390395326280786) | > loss_mel_ce: 2.670461416244507 (nan) | > loss: 0.010713587515056133 (nan) | > current_lr: 5e-06 | > step_time: 0.1689 (0.13670491608955646) | > loader_time: 0.0048 (0.0046465479786049575)  --> TIME: 2025-02-25 14:55:52 -- STEP: 14314/36938 -- GLOBAL_STEP: 826950 | > loss_text_ce: 0.029240993782877922 (0.03389870611966618) | > loss_mel_ce: 2.56904673576355 (nan) | > loss: 0.010310666635632515 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1366823266567562) | > loader_time: 0.0045 (0.004646436851984353)  --> TIME: 2025-02-25 14:56:01 -- STEP: 14364/36938 -- GLOBAL_STEP: 827000 | > loss_text_ce: 0.01603352092206478 (0.033897599256855426) | > loss_mel_ce: 0.8546133041381836 (nan) | > loss: 0.003454947844147682 (nan) | > current_lr: 5e-06 | > step_time: 0.1079 (0.13665255521605973) | > loader_time: 0.0045 (0.004646296920765732)  --> TIME: 2025-02-25 14:56:10 -- STEP: 14414/36938 -- GLOBAL_STEP: 827050 | > loss_text_ce: 0.034626901149749756 (0.03389693522109664) | > loss_mel_ce: 3.090825080871582 (nan) | > loss: 0.012402588501572609 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.136661400407268) | > loader_time: 0.0045 (0.004646428120653975)  --> TIME: 2025-02-25 14:56:20 -- STEP: 14464/36938 -- GLOBAL_STEP: 827100 | > loss_text_ce: 0.03734944388270378 (0.03389725834260009) | > loss_mel_ce: 3.060189723968506 (nan) | > loss: 0.012291822582483292 (nan) | > current_lr: 5e-06 | > step_time: 0.1828 (0.13666863414763925) | > loader_time: 0.0049 (0.004646697584374805)  --> TIME: 2025-02-25 14:56:29 -- STEP: 14514/36938 -- GLOBAL_STEP: 827150 | > loss_text_ce: 0.04924958199262619 (0.03389577372162596) | > loss_mel_ce: 2.7776269912719727 (nan) | > loss: 0.011217765510082245 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13666190965717434) | > loader_time: 0.0045 (0.004646956764569165)  --> TIME: 2025-02-25 14:56:38 -- STEP: 14564/36938 -- GLOBAL_STEP: 827200 | > loss_text_ce: 0.032608821988105774 (0.03389947673985488) | > loss_mel_ce: 3.0696139335632324 (nan) | > loss: 0.012310408055782318 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.1366577331622339) | > loader_time: 0.0046 (0.004647074411020789)  --> TIME: 2025-02-25 14:56:47 -- STEP: 14614/36938 -- GLOBAL_STEP: 827250 | > loss_text_ce: 0.03859446197748184 (0.0338995852993258) | > loss_mel_ce: 3.292590856552124 (nan) | > loss: 0.01321899052709341 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13665152488466856) | > loader_time: 0.0047 (0.004647208627278186)  --> TIME: 2025-02-25 14:56:57 -- STEP: 14664/36938 -- GLOBAL_STEP: 827300 | > loss_text_ce: 0.03867833688855171 (0.033897180844210506) | > loss_mel_ce: 2.7300522327423096 (nan) | > loss: 0.010987026616930962 (nan) | > current_lr: 5e-06 | > step_time: 0.1527 (0.1366541589373009) | > loader_time: 0.0047 (0.004647341798188176)  --> TIME: 2025-02-25 14:57:06 -- STEP: 14714/36938 -- GLOBAL_STEP: 827350 | > loss_text_ce: 0.03824378550052643 (0.033896395378409626) | > loss_mel_ce: 3.424325704574585 (nan) | > loss: 0.01374035608023405 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13664744644688798) | > loader_time: 0.0046 (0.004647496003600349)  --> TIME: 2025-02-25 14:57:15 -- STEP: 14764/36938 -- GLOBAL_STEP: 827400 | > loss_text_ce: 0.030175112187862396 (0.033897241919087634) | > loss_mel_ce: 2.896583080291748 (nan) | > loss: 0.011614120565354824 (nan) | > current_lr: 5e-06 | > step_time: 0.1519 (0.13666644888627036) | > loader_time: 0.0046 (0.0046475956802347375)  --> TIME: 2025-02-25 14:57:24 -- STEP: 14814/36938 -- GLOBAL_STEP: 827450 | > loss_text_ce: 0.024506613612174988 (0.03389632511893773) | > loss_mel_ce: 0.005950292106717825 (nan) | > loss: 0.00012086074275430292 (nan) | > current_lr: 5e-06 | > step_time: 0.1118 (0.13664424223054636) | > loader_time: 0.0045 (0.004647664040773673)  --> TIME: 2025-02-25 14:57:34 -- STEP: 14864/36938 -- GLOBAL_STEP: 827500 | > loss_text_ce: 0.02878284454345703 (0.0338941990984708) | > loss_mel_ce: 2.5592916011810303 (nan) | > loss: 0.010270137339830399 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13664310605970006) | > loader_time: 0.005 (0.004648050495831383)  --> TIME: 2025-02-25 14:57:43 -- STEP: 14914/36938 -- GLOBAL_STEP: 827550 | > loss_text_ce: 0.04689338058233261 (0.033896728394101114) | > loss_mel_ce: 1.8385241031646729 (nan) | > loss: 0.007481815759092569 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13663607470484437) | > loader_time: 0.0046 (0.004648266712110441)  --> TIME: 2025-02-25 14:57:52 -- STEP: 14964/36938 -- GLOBAL_STEP: 827600 | > loss_text_ce: 0.02922692708671093 (0.033894997132484915) | > loss_mel_ce: 2.2514255046844482 (nan) | > loss: 0.009050209075212479 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13662343090850754) | > loader_time: 0.0046 (0.004648231864134842)  --> TIME: 2025-02-25 14:58:02 -- STEP: 15014/36938 -- GLOBAL_STEP: 827650 | > loss_text_ce: 0.03658777102828026 (0.03389201960698222) | > loss_mel_ce: 3.0304059982299805 (nan) | > loss: 0.012170610949397087 (nan) | > current_lr: 5e-06 | > step_time: 0.1816 (0.13663051411682262) | > loader_time: 0.0048 (0.00464837357701577)  --> TIME: 2025-02-25 14:58:11 -- STEP: 15064/36938 -- GLOBAL_STEP: 827700 | > loss_text_ce: 0.031837522983551025 (0.03388550560609826) | > loss_mel_ce: 2.0197343826293945 (nan) | > loss: 0.008141158148646355 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.1366425764035247) | > loader_time: 0.0045 (0.0046483897745007744)  --> TIME: 2025-02-25 14:58:20 -- STEP: 15114/36938 -- GLOBAL_STEP: 827750 | > loss_text_ce: 0.03403247892856598 (0.03388452792296907) | > loss_mel_ce: 3.2121050357818604 (nan) | > loss: 0.012881498783826828 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.1366225369821806) | > loader_time: 0.0047 (0.004648300852743382)  --> TIME: 2025-02-25 14:58:30 -- STEP: 15164/36938 -- GLOBAL_STEP: 827800 | > loss_text_ce: 0.032509695738554 (0.033886574628826424) | > loss_mel_ce: 1.8054028749465942 (nan) | > loss: 0.007293304428458214 (nan) | > current_lr: 5e-06 | > step_time: 0.1071 (0.13662674780163708) | > loader_time: 0.0046 (0.004648654591719592)  --> TIME: 2025-02-25 14:58:39 -- STEP: 15214/36938 -- GLOBAL_STEP: 827850 | > loss_text_ce: 0.03378701210021973 (0.033885671009605885) | > loss_mel_ce: 3.1471869945526123 (nan) | > loss: 0.012622913345694542 (nan) | > current_lr: 5e-06 | > step_time: 0.1748 (0.13661880638590987) | > loader_time: 0.0048 (0.004648532239964829)  --> TIME: 2025-02-25 14:58:48 -- STEP: 15264/36938 -- GLOBAL_STEP: 827900 | > loss_text_ce: 0.02781841717660427 (0.033886461273061136) | > loss_mel_ce: 3.012805223464966 (nan) | > loss: 0.012065967544913292 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13662339759447842) | > loader_time: 0.0048 (0.004648696357848998)  --> TIME: 2025-02-25 14:58:57 -- STEP: 15314/36938 -- GLOBAL_STEP: 827950 | > loss_text_ce: 0.03715749830007553 (0.03389019310750243) | > loss_mel_ce: 1.9183028936386108 (nan) | > loss: 0.0077597638592123985 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13662103090551597) | > loader_time: 0.0047 (0.0046487991688720625)  --> TIME: 2025-02-25 14:59:07 -- STEP: 15364/36938 -- GLOBAL_STEP: 828000 | > loss_text_ce: 0.026510175317525864 (0.03388644102633482) | > loss_mel_ce: 1.2868194580078125 (nan) | > loss: 0.005211625713855028 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13662360655519737) | > loader_time: 0.0046 (0.0046488291830545925)  --> TIME: 2025-02-25 14:59:16 -- STEP: 15414/36938 -- GLOBAL_STEP: 828050 | > loss_text_ce: 0.0314549021422863 (0.03388860057415876) | > loss_mel_ce: 2.3782260417938232 (nan) | > loss: 0.009562226012349129 (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.136625308570996) | > loader_time: 0.0046 (0.0046489505865577796)  --> TIME: 2025-02-25 14:59:25 -- STEP: 15464/36938 -- GLOBAL_STEP: 828100 | > loss_text_ce: 0.038555413484573364 (0.033890165800765336) | > loss_mel_ce: 2.878434658050537 (nan) | > loss: 0.011575357988476753 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13660486346998232) | > loader_time: 0.0047 (0.004648818432575537)  --> TIME: 2025-02-25 14:59:34 -- STEP: 15514/36938 -- GLOBAL_STEP: 828150 | > loss_text_ce: 0.04008009657263756 (0.03388699854641136) | > loss_mel_ce: 2.8634822368621826 (nan) | > loss: 0.011522073298692703 (nan) | > current_lr: 5e-06 | > step_time: 0.1511 (0.13659279312963957) | > loader_time: 0.0047 (0.004648907199661423)  --> TIME: 2025-02-25 14:59:44 -- STEP: 15564/36938 -- GLOBAL_STEP: 828200 | > loss_text_ce: 0.02905658818781376 (0.033887603728517506) | > loss_mel_ce: 3.1150567531585693 (nan) | > loss: 0.01247664075344801 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13659648284358242) | > loader_time: 0.0045 (0.004648939927786662)  --> TIME: 2025-02-25 14:59:53 -- STEP: 15614/36938 -- GLOBAL_STEP: 828250 | > loss_text_ce: 0.03649376332759857 (0.03388968257352068) | > loss_mel_ce: 1.9509249925613403 (nan) | > loss: 0.007886582985520363 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13661745872023645) | > loader_time: 0.0045 (0.004649035631655964)  --> TIME: 2025-02-25 15:00:03 -- STEP: 15664/36938 -- GLOBAL_STEP: 828300 | > loss_text_ce: 0.03876519948244095 (0.03388873696069397) | > loss_mel_ce: 2.6776654720306396 (nan) | > loss: 0.01077948696911335 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13661876290397384) | > loader_time: 0.0047 (0.004649088106311245)  --> TIME: 2025-02-25 15:00:12 -- STEP: 15714/36938 -- GLOBAL_STEP: 828350 | > loss_text_ce: 0.03267389535903931 (0.0338837784144017) | > loss_mel_ce: 2.864438772201538 (nan) | > loss: 0.011496479623019695 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13661769015409472) | > loader_time: 0.0046 (0.004649311087886163)  --> TIME: 2025-02-25 15:00:22 -- STEP: 15764/36938 -- GLOBAL_STEP: 828400 | > loss_text_ce: 0.03371606394648552 (0.033888710565360164) | > loss_mel_ce: 1.3946400880813599 (nan) | > loss: 0.005668080411851406 (nan) | > current_lr: 5e-06 | > step_time: 0.1081 (0.13663074674415204) | > loader_time: 0.0045 (0.00464936702936864)  --> TIME: 2025-02-25 15:00:31 -- STEP: 15814/36938 -- GLOBAL_STEP: 828450 | > loss_text_ce: 0.030478917062282562 (0.03388869081225377) | > loss_mel_ce: 1.8829631805419922 (nan) | > loss: 0.007593024987727404 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13663492886323933) | > loader_time: 0.0045 (0.004649528574216848)  --> TIME: 2025-02-25 15:00:40 -- STEP: 15864/36938 -- GLOBAL_STEP: 828500 | > loss_text_ce: 0.03304799646139145 (0.03389109641633377) | > loss_mel_ce: 2.6602694988250732 (nan) | > loss: 0.010687768459320068 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13662571824083944) | > loader_time: 0.0046 (0.00464953113191366)  --> TIME: 2025-02-25 15:00:50 -- STEP: 15914/36938 -- GLOBAL_STEP: 828550 | > loss_text_ce: 0.037531521171331406 (0.033897172318450915) | > loss_mel_ce: 2.836016893386841 (nan) | > loss: 0.011402971111238003 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13665453815664008) | > loader_time: 0.0046 (0.0046497204652215405)  --> TIME: 2025-02-25 15:00:59 -- STEP: 15964/36938 -- GLOBAL_STEP: 828600 | > loss_text_ce: 0.02726629190146923 (0.03389428067936983) | > loss_mel_ce: 2.8844733238220215 (nan) | > loss: 0.011554522439837456 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.13664414499732605) | > loader_time: 0.0047 (0.004649849903968833)  --> TIME: 2025-02-25 15:01:09 -- STEP: 16014/36938 -- GLOBAL_STEP: 828650 | > loss_text_ce: 0.030103228986263275 (0.03389443562727677) | > loss_mel_ce: 2.6284289360046387 (nan) | > loss: 0.01054973155260086 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13665332643223824) | > loader_time: 0.0069 (0.004650207082177412)  --> TIME: 2025-02-25 15:01:18 -- STEP: 16064/36938 -- GLOBAL_STEP: 828700 | > loss_text_ce: 0.028226453810930252 (0.03389159771956983) | > loss_mel_ce: 3.0794894695281982 (nan) | > loss: 0.012332206591963768 (nan) | > current_lr: 5e-06 | > step_time: 0.1945 (0.1366725228133197) | > loader_time: 0.0048 (0.004650466678389504)  --> TIME: 2025-02-25 15:01:28 -- STEP: 16114/36938 -- GLOBAL_STEP: 828750 | > loss_text_ce: 0.0299314446747303 (0.03389300222870286) | > loss_mel_ce: 2.4013755321502686 (nan) | > loss: 0.009648044593632221 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1366704556491313) | > loader_time: 0.0046 (0.0046503994236132)  --> TIME: 2025-02-25 15:01:37 -- STEP: 16164/36938 -- GLOBAL_STEP: 828800 | > loss_text_ce: 0.03916584327816963 (0.03389243177359278) | > loss_mel_ce: 2.837963342666626 (nan) | > loss: 0.011417179368436337 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.1366702435601326) | > loader_time: 0.0046 (0.0046505160598405496)  --> TIME: 2025-02-25 15:01:46 -- STEP: 16214/36938 -- GLOBAL_STEP: 828850 | > loss_text_ce: 0.028005292639136314 (0.03389327029535723) | > loss_mel_ce: 0.03662109375 (nan) | > loss: 0.0002564539317972958 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.1366690837073834) | > loader_time: 0.0046 (0.004650607655488612)  --> TIME: 2025-02-25 15:01:56 -- STEP: 16264/36938 -- GLOBAL_STEP: 828900 | > loss_text_ce: 0.037072692066431046 (0.03389687413501828) | > loss_mel_ce: 3.302039861679077 (nan) | > loss: 0.013250446878373623 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.1366661666240594) | > loader_time: 0.0047 (0.0046506124034164705)  --> TIME: 2025-02-25 15:02:05 -- STEP: 16314/36938 -- GLOBAL_STEP: 828950 | > loss_text_ce: 0.03340035304427147 (0.03389033401774047) | > loss_mel_ce: 2.8549644947052 (nan) | > loss: 0.011461765505373478 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1366714651245285) | > loader_time: 0.0046 (0.00465074374100821)  --> TIME: 2025-02-25 15:02:14 -- STEP: 16364/36938 -- GLOBAL_STEP: 829000 | > loss_text_ce: 0.02387821674346924 (0.03388563587750972) | > loss_mel_ce: 2.3007102012634277 (nan) | > loss: 0.00922455731779337 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13666684685520417) | > loader_time: 0.0045 (0.00465064400188265)  --> TIME: 2025-02-25 15:02:24 -- STEP: 16414/36938 -- GLOBAL_STEP: 829050 | > loss_text_ce: 0.02341175451874733 (0.03388409915351687) | > loss_mel_ce: 3.156471014022827 (nan) | > loss: 0.01261858269572258 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.13667567384596027) | > loader_time: 0.0049 (0.004650619181933601)  --> TIME: 2025-02-25 15:02:34 -- STEP: 16464/36938 -- GLOBAL_STEP: 829100 | > loss_text_ce: 0.04062496870756149 (0.03388834329211325) | > loss_mel_ce: 0.10844726860523224 (nan) | > loss: 0.0005915564834140241 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13668806613418455) | > loader_time: 0.0046 (0.004650627196819825)  --> TIME: 2025-02-25 15:02:43 -- STEP: 16514/36938 -- GLOBAL_STEP: 829150 | > loss_text_ce: 0.03701263293623924 (0.03388528286366499) | > loss_mel_ce: 2.6327950954437256 (nan) | > loss: 0.010594475083053112 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13667534274599918) | > loader_time: 0.0045 (0.004650624335151432)  --> TIME: 2025-02-25 15:02:52 -- STEP: 16564/36938 -- GLOBAL_STEP: 829200 | > loss_text_ce: 0.028160003945231438 (0.03388259461491641) | > loss_mel_ce: 3.223060131072998 (nan) | > loss: 0.012901668436825275 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.13666507585185178) | > loader_time: 0.0048 (0.004650540669682338)  --> TIME: 2025-02-25 15:03:01 -- STEP: 16614/36938 -- GLOBAL_STEP: 829250 | > loss_text_ce: 0.03113568387925625 (0.03388106415958142) | > loss_mel_ce: 2.558640241622925 (nan) | > loss: 0.01027688942849636 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13666577990396386) | > loader_time: 0.0046 (0.0046506341045852995)  --> TIME: 2025-02-25 15:03:10 -- STEP: 16664/36938 -- GLOBAL_STEP: 829300 | > loss_text_ce: 0.03533147647976875 (0.033879079016703846) | > loss_mel_ce: 2.5733957290649414 (nan) | > loss: 0.010352092795073986 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13666171527973395) | > loader_time: 0.0046 (0.004650662752852887)  --> TIME: 2025-02-25 15:03:20 -- STEP: 16714/36938 -- GLOBAL_STEP: 829350 | > loss_text_ce: 0.026645855978131294 (0.03387988087859013) | > loss_mel_ce: 2.701326608657837 (nan) | > loss: 0.010825288482010365 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13666258794604974) | > loader_time: 0.0046 (0.004650721057003096)  --> TIME: 2025-02-25 15:03:29 -- STEP: 16764/36938 -- GLOBAL_STEP: 829400 | > loss_text_ce: 0.03441416844725609 (0.033880887997701704) | > loss_mel_ce: 3.2153923511505127 (nan) | > loss: 0.012896058149635792 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.1366631137360487) | > loader_time: 0.0046 (0.004650717190073709)  --> TIME: 2025-02-25 15:03:38 -- STEP: 16814/36938 -- GLOBAL_STEP: 829450 | > loss_text_ce: 0.02881115861237049 (0.03387879924955015) | > loss_mel_ce: 1.69757080078125 (nan) | > loss: 0.006850722711533308 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13666470684363746) | > loader_time: 0.0046 (0.004650693494470725)  --> TIME: 2025-02-25 15:03:48 -- STEP: 16864/36938 -- GLOBAL_STEP: 829500 | > loss_text_ce: 0.02236432582139969 (0.033876564835104914) | > loss_mel_ce: 3.306690216064453 (nan) | > loss: 0.013210535049438477 (nan) | > current_lr: 5e-06 | > step_time: 0.1938 (0.13667736403234138) | > loader_time: 0.0049 (0.0046507323291541435)  --> TIME: 2025-02-25 15:03:57 -- STEP: 16914/36938 -- GLOBAL_STEP: 829550 | > loss_text_ce: 0.026499493047595024 (0.033875054045438946) | > loss_mel_ce: 3.3126633167266846 (nan) | > loss: 0.013250647112727165 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1366629573037842) | > loader_time: 0.0045 (0.004650695309566235)  --> TIME: 2025-02-25 15:04:06 -- STEP: 16964/36938 -- GLOBAL_STEP: 829600 | > loss_text_ce: 0.0295002032071352 (0.03387332995017233) | > loss_mel_ce: 2.9752936363220215 (nan) | > loss: 0.011923786252737045 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13665656556051026) | > loader_time: 0.0045 (0.004650718885838656)  --> TIME: 2025-02-25 15:04:15 -- STEP: 17014/36938 -- GLOBAL_STEP: 829650 | > loss_text_ce: 0.031812041997909546 (0.03387455968295727) | > loss_mel_ce: 2.536707878112793 (nan) | > loss: 0.010192539542913437 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13665163727194574) | > loader_time: 0.0047 (0.004650770966281079)  --> TIME: 2025-02-25 15:04:25 -- STEP: 17064/36938 -- GLOBAL_STEP: 829700 | > loss_text_ce: 0.03018886409699917 (0.0338707234471074) | > loss_mel_ce: 2.246288299560547 (nan) | > loss: 0.009033639915287495 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13665365530077567) | > loader_time: 0.0046 (0.004650658723953675)  --> TIME: 2025-02-25 15:04:34 -- STEP: 17114/36938 -- GLOBAL_STEP: 829750 | > loss_text_ce: 0.0225131344050169 (0.03386805731596596) | > loss_mel_ce: 1.431793212890625 (nan) | > loss: 0.005771057214587927 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13664082546330014) | > loader_time: 0.0046 (0.00465055270995772)  --> TIME: 2025-02-25 15:04:43 -- STEP: 17164/36938 -- GLOBAL_STEP: 829800 | > loss_text_ce: 0.029767600819468498 (0.033870294443844866) | > loss_mel_ce: 2.636530637741089 (nan) | > loss: 0.010580549016594887 (nan) | > current_lr: 5e-06 | > step_time: 0.1349 (0.13662825342492643) | > loader_time: 0.0047 (0.004650656436879262)  --> TIME: 2025-02-25 15:04:52 -- STEP: 17214/36938 -- GLOBAL_STEP: 829850 | > loss_text_ce: 0.03225579857826233 (0.03387361635748635) | > loss_mel_ce: 2.796522378921509 (nan) | > loss: 0.01122531108558178 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13662267174470943) | > loader_time: 0.0046 (0.004650728328864821)  --> TIME: 2025-02-25 15:05:01 -- STEP: 17264/36938 -- GLOBAL_STEP: 829900 | > loss_text_ce: 0.0303909033536911 (0.03387701070240188) | > loss_mel_ce: 2.672382354736328 (nan) | > loss: 0.010725291445851326 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13662755005993846) | > loader_time: 0.0047 (0.00465071724729475)  --> TIME: 2025-02-25 15:05:10 -- STEP: 17314/36938 -- GLOBAL_STEP: 829950 | > loss_text_ce: 0.043201711028814316 (0.03388127008477961) | > loss_mel_ce: 2.842991352081299 (nan) | > loss: 0.011453147977590561 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13661641679102643) | > loader_time: 0.0046 (0.004650980712663238)  --> TIME: 2025-02-25 15:05:20 -- STEP: 17364/36938 -- GLOBAL_STEP: 830000 | > loss_text_ce: 0.034438103437423706 (0.033878462763324596) | > loss_mel_ce: 3.1395068168640137 (nan) | > loss: 0.012595020234584808 (nan) | > current_lr: 5e-06 | > step_time: 0.185 (0.1366172288727913) | > loader_time: 0.0048 (0.00465102666425583) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_830000.pth  --> TIME: 2025-02-25 15:05:35 -- STEP: 17414/36938 -- GLOBAL_STEP: 830050 | > loss_text_ce: 0.029651746153831482 (0.03388115135468067) | > loss_mel_ce: 3.2020766735076904 (nan) | > loss: 0.012824319303035736 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.1366143329992109) | > loader_time: 0.0048 (0.004651007893795494)  --> TIME: 2025-02-25 15:05:44 -- STEP: 17464/36938 -- GLOBAL_STEP: 830100 | > loss_text_ce: 0.016009483486413956 (0.03388170717226265) | > loss_mel_ce: 2.153981924057007 (nan) | > loss: 0.008611077442765236 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13661449060929398) | > loader_time: 0.0046 (0.004650955810712871)  --> TIME: 2025-02-25 15:05:53 -- STEP: 17514/36938 -- GLOBAL_STEP: 830150 | > loss_text_ce: 0.041385989636182785 (0.03388098374302014) | > loss_mel_ce: 2.919114112854004 (nan) | > loss: 0.011748016811907291 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13658904145811382) | > loader_time: 0.0045 (0.004650718942276064)  --> TIME: 2025-02-25 15:06:02 -- STEP: 17564/36938 -- GLOBAL_STEP: 830200 | > loss_text_ce: 0.025254560634493828 (0.033881863832239914) | > loss_mel_ce: 1.2615203857421875 (nan) | > loss: 0.005106250289827585 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.13657315458531477) | > loader_time: 0.0045 (0.004650706176306049)  --> TIME: 2025-02-25 15:06:12 -- STEP: 17614/36938 -- GLOBAL_STEP: 830250 | > loss_text_ce: 0.0334094762802124 (0.033882408605843344) | > loss_mel_ce: 3.9464962482452393 (nan) | > loss: 0.015793276950716972 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13658147130554538) | > loader_time: 0.0046 (0.004650752688152656)  --> TIME: 2025-02-25 15:06:21 -- STEP: 17664/36938 -- GLOBAL_STEP: 830300 | > loss_text_ce: 0.03598811849951744 (0.03388389931379874) | > loss_mel_ce: 1.3060836791992188 (nan) | > loss: 0.005325681995600462 (nan) | > current_lr: 5e-06 | > step_time: 0.1355 (0.13659191057355993) | > loader_time: 0.0047 (0.00465063719699779)  --> TIME: 2025-02-25 15:06:31 -- STEP: 17714/36938 -- GLOBAL_STEP: 830350 | > loss_text_ce: 0.03627685457468033 (0.033885145615689484) | > loss_mel_ce: 2.912310838699341 (nan) | > loss: 0.011700745671987534 (nan) | > current_lr: 5e-06 | > step_time: 0.163 (0.13660885626297772) | > loader_time: 0.0047 (0.004650871506310122)  --> TIME: 2025-02-25 15:06:40 -- STEP: 17764/36938 -- GLOBAL_STEP: 830400 | > loss_text_ce: 0.03155225142836571 (0.03388270443514863) | > loss_mel_ce: 2.121795654296875 (nan) | > loss: 0.008545031771063805 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13660946230017654) | > loader_time: 0.0049 (0.004650846106918552)  --> TIME: 2025-02-25 15:06:50 -- STEP: 17814/36938 -- GLOBAL_STEP: 830450 | > loss_text_ce: 0.03981703519821167 (0.03388610121434242) | > loss_mel_ce: 3.1810834407806396 (nan) | > loss: 0.012781351804733276 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.1366180162364504) | > loader_time: 0.0045 (0.004650980250875827)  --> TIME: 2025-02-25 15:06:59 -- STEP: 17864/36938 -- GLOBAL_STEP: 830500 | > loss_text_ce: 0.026965972036123276 (0.03388325884087198) | > loss_mel_ce: 1.2770715951919556 (nan) | > loss: 0.005174752790480852 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13662074372152405) | > loader_time: 0.0045 (0.0046509160517151345)  --> TIME: 2025-02-25 15:07:09 -- STEP: 17914/36938 -- GLOBAL_STEP: 830550 | > loss_text_ce: 0.0334850549697876 (0.0338856850043076) | > loss_mel_ce: 2.5810203552246094 (nan) | > loss: 0.01037502195686102 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13664013032230776) | > loader_time: 0.0045 (0.004650897328656891)  --> TIME: 2025-02-25 15:07:18 -- STEP: 17964/36938 -- GLOBAL_STEP: 830600 | > loss_text_ce: 0.030310431495308876 (0.03388551960376165) | > loss_mel_ce: 1.6087254285812378 (nan) | > loss: 0.0065041109919548035 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13663381776096248) | > loader_time: 0.0046 (0.004650923078189143)  --> TIME: 2025-02-25 15:07:27 -- STEP: 18014/36938 -- GLOBAL_STEP: 830650 | > loss_text_ce: 0.024927429854869843 (0.03388661031302197) | > loss_mel_ce: 3.3378379344940186 (nan) | > loss: 0.01334430743008852 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.13664033106671283) | > loader_time: 0.0047 (0.004650964646409819)  --> TIME: 2025-02-25 15:07:37 -- STEP: 18064/36938 -- GLOBAL_STEP: 830700 | > loss_text_ce: 0.03166060149669647 (0.03388643613207026) | > loss_mel_ce: 0.02325439453125 (nan) | > loss: 0.00021791666222270578 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13663206058756508) | > loader_time: 0.0046 (0.004650845041414395)  --> TIME: 2025-02-25 15:07:46 -- STEP: 18114/36938 -- GLOBAL_STEP: 830750 | > loss_text_ce: 0.030890315771102905 (0.03388949481797539) | > loss_mel_ce: 2.7135443687438965 (nan) | > loss: 0.010890614241361618 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13662828866832938) | > loader_time: 0.0053 (0.004650936572115001)  --> TIME: 2025-02-25 15:07:55 -- STEP: 18164/36938 -- GLOBAL_STEP: 830800 | > loss_text_ce: 0.030758334323763847 (0.033889392209186964) | > loss_mel_ce: 2.4713473320007324 (nan) | > loss: 0.009928991086781025 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1366441466162004) | > loader_time: 0.0045 (0.0046510460276457674)  --> TIME: 2025-02-25 15:08:04 -- STEP: 18214/36938 -- GLOBAL_STEP: 830850 | > loss_text_ce: 0.033815715461969376 (0.03388832978845136) | > loss_mel_ce: 3.158414602279663 (nan) | > loss: 0.01266758143901825 (nan) | > current_lr: 5e-06 | > step_time: 0.1862 (0.1366263074174313) | > loader_time: 0.0051 (0.004650925299306329)  --> TIME: 2025-02-25 15:08:13 -- STEP: 18264/36938 -- GLOBAL_STEP: 830900 | > loss_text_ce: 0.03893055394291878 (0.03388979474886838) | > loss_mel_ce: 2.9697868824005127 (nan) | > loss: 0.011939356103539467 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13661083704382027) | > loader_time: 0.0045 (0.004650754621558368)  --> TIME: 2025-02-25 15:08:23 -- STEP: 18314/36938 -- GLOBAL_STEP: 830950 | > loss_text_ce: 0.0300734993070364 (0.033885085253134266) | > loss_mel_ce: 2.0594093799591064 (nan) | > loss: 0.008291598409414291 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13661589243142053) | > loader_time: 0.0046 (0.004650624529742846)  --> TIME: 2025-02-25 15:08:32 -- STEP: 18364/36938 -- GLOBAL_STEP: 831000 | > loss_text_ce: 0.04023037478327751 (0.03388658783909161) | > loss_mel_ce: 3.4600882530212402 (nan) | > loss: 0.013890153728425503 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13661120417486308) | > loader_time: 0.0047 (0.004650584092407122)  --> TIME: 2025-02-25 15:08:42 -- STEP: 18414/36938 -- GLOBAL_STEP: 831050 | > loss_text_ce: 0.02734457701444626 (0.033884196681188646) | > loss_mel_ce: 2.58550763130188 (nan) | > loss: 0.010368461720645428 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13662891590182247) | > loader_time: 0.0045 (0.00465057634745382)  --> TIME: 2025-02-25 15:08:51 -- STEP: 18464/36938 -- GLOBAL_STEP: 831100 | > loss_text_ce: 0.041178517043590546 (0.03388166328972005) | > loss_mel_ce: 2.314986228942871 (nan) | > loss: 0.00934986025094986 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13662537130522387) | > loader_time: 0.0044 (0.004650425366037219)  --> TIME: 2025-02-25 15:09:00 -- STEP: 18514/36938 -- GLOBAL_STEP: 831150 | > loss_text_ce: 0.033490002155303955 (0.03388213063059521) | > loss_mel_ce: 0.00696563720703125 (nan) | > loss: 0.0001605382567504421 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13662840537843624) | > loader_time: 0.0046 (0.004650355454228842)  --> TIME: 2025-02-25 15:09:10 -- STEP: 18564/36938 -- GLOBAL_STEP: 831200 | > loss_text_ce: 0.03322353586554527 (0.033880596405926194) | > loss_mel_ce: 3.181493043899536 (nan) | > loss: 0.012756813317537308 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.13663746304513097) | > loader_time: 0.0047 (0.004650350583829998)  --> TIME: 2025-02-25 15:09:19 -- STEP: 18614/36938 -- GLOBAL_STEP: 831250 | > loss_text_ce: 0.03612827509641647 (0.033877620095351224) | > loss_mel_ce: 2.83723521232605 (nan) | > loss: 0.011402236297726631 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1366214986722046) | > loader_time: 0.0045 (0.004650200810713703)  --> TIME: 2025-02-25 15:09:28 -- STEP: 18664/36938 -- GLOBAL_STEP: 831300 | > loss_text_ce: 0.038857657462358475 (0.03387676221384405) | > loss_mel_ce: 2.9298534393310547 (nan) | > loss: 0.011780600063502789 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13662707888666212) | > loader_time: 0.0046 (0.004650081578519172)  --> TIME: 2025-02-25 15:09:38 -- STEP: 18714/36938 -- GLOBAL_STEP: 831350 | > loss_text_ce: 0.03100758232176304 (0.03387719942512602) | > loss_mel_ce: 2.887364387512207 (nan) | > loss: 0.01158084161579609 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1366277459060921) | > loader_time: 0.0046 (0.004650010516835385)  --> TIME: 2025-02-25 15:09:47 -- STEP: 18764/36938 -- GLOBAL_STEP: 831400 | > loss_text_ce: 0.035326309502124786 (0.033878380700050836) | > loss_mel_ce: 2.8332927227020264 (nan) | > loss: 0.01138340961188078 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13663663120316596) | > loader_time: 0.0046 (0.004649961561415647)  --> TIME: 2025-02-25 15:09:57 -- STEP: 18814/36938 -- GLOBAL_STEP: 831450 | > loss_text_ce: 0.03118371218442917 (0.03387418899127161) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.1366407753012831) | > loader_time: 0.0045 (0.004649839569027719)  --> TIME: 2025-02-25 15:10:06 -- STEP: 18864/36938 -- GLOBAL_STEP: 831500 | > loss_text_ce: 0.041143253445625305 (0.033875787480788795) | > loss_mel_ce: 3.217890501022339 (nan) | > loss: 0.012932674027979374 (nan) | > current_lr: 5e-06 | > step_time: 0.1456 (0.13665746984034993) | > loader_time: 0.0068 (0.004649903015427101)  --> TIME: 2025-02-25 15:10:15 -- STEP: 18914/36938 -- GLOBAL_STEP: 831550 | > loss_text_ce: 0.02642248570919037 (0.03387970355385339) | > loss_mel_ce: 2.0140206813812256 (nan) | > loss: 0.008096997626125813 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13664160455976299) | > loader_time: 0.0045 (0.00465005108679123)  --> TIME: 2025-02-25 15:10:24 -- STEP: 18964/36938 -- GLOBAL_STEP: 831600 | > loss_text_ce: 0.031774431467056274 (0.033877143288254) | > loss_mel_ce: 2.8256285190582275 (nan) | > loss: 0.011338901706039906 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13662848087132648) | > loader_time: 0.0045 (0.004650054023728991)  --> TIME: 2025-02-25 15:10:33 -- STEP: 19014/36938 -- GLOBAL_STEP: 831650 | > loss_text_ce: 0.03010009229183197 (0.03387737371730055) | > loss_mel_ce: 3.1632919311523438 (nan) | > loss: 0.012672191485762596 (nan) | > current_lr: 5e-06 | > step_time: 0.1717 (0.13660753308704868) | > loader_time: 0.0048 (0.004650142524625338)  --> TIME: 2025-02-25 15:10:42 -- STEP: 19064/36938 -- GLOBAL_STEP: 831700 | > loss_text_ce: 0.034902095794677734 (0.03387914026717679) | > loss_mel_ce: 0.1622314453125 (nan) | > loss: 0.000782275979872793 (nan) | > current_lr: 5e-06 | > step_time: 0.1614 (0.13660588163915408) | > loader_time: 0.0048 (0.004650824694287303)  --> TIME: 2025-02-25 15:10:52 -- STEP: 19114/36938 -- GLOBAL_STEP: 831750 | > loss_text_ce: 0.0444585457444191 (0.03387906571151083) | > loss_mel_ce: 2.015625 (nan) | > loss: 0.008174935355782509 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.1366095770299724) | > loader_time: 0.0045 (0.004651220445791591)  --> TIME: 2025-02-25 15:11:01 -- STEP: 19164/36938 -- GLOBAL_STEP: 831800 | > loss_text_ce: 0.02698548696935177 (0.03388015151794044) | > loss_mel_ce: 2.4801273345947266 (nan) | > loss: 0.009948860853910446 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13660473207471052) | > loader_time: 0.0044 (0.0046511443242029426)  --> TIME: 2025-02-25 15:11:10 -- STEP: 19214/36938 -- GLOBAL_STEP: 831850 | > loss_text_ce: 0.03194744512438774 (0.033879502476435924) | > loss_mel_ce: 2.793179750442505 (nan) | > loss: 0.011210822500288486 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.13661114035628608) | > loader_time: 0.0047 (0.004651039550291253)  --> TIME: 2025-02-25 15:11:20 -- STEP: 19264/36938 -- GLOBAL_STEP: 831900 | > loss_text_ce: 0.03955551236867905 (0.03387756405669569) | > loss_mel_ce: 2.2303833961486816 (nan) | > loss: 0.009007695131003857 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.1366061417268369) | > loader_time: 0.0046 (0.004651087265078009)  --> TIME: 2025-02-25 15:11:29 -- STEP: 19314/36938 -- GLOBAL_STEP: 831950 | > loss_text_ce: 0.03385952487587929 (0.03387540170413291) | > loss_mel_ce: 3.4089114665985107 (nan) | > loss: 0.013661790639162064 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.1366111838300377) | > loader_time: 0.0045 (0.004651034249891035)  --> TIME: 2025-02-25 15:11:38 -- STEP: 19364/36938 -- GLOBAL_STEP: 832000 | > loss_text_ce: 0.033589430153369904 (0.03387574232518147) | > loss_mel_ce: 2.759934663772583 (nan) | > loss: 0.01108541339635849 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1366103359476353) | > loader_time: 0.0045 (0.00465102494686404)  --> TIME: 2025-02-25 15:11:48 -- STEP: 19414/36938 -- GLOBAL_STEP: 832050 | > loss_text_ce: 0.03398005664348602 (0.033876064032048764) | > loss_mel_ce: 2.311448335647583 (nan) | > loss: 0.0093072559684515 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13661108598338803) | > loader_time: 0.0045 (0.0046508601314316275)  --> TIME: 2025-02-25 15:11:57 -- STEP: 19464/36938 -- GLOBAL_STEP: 832100 | > loss_text_ce: 0.045038726180791855 (0.033874388046825306) | > loss_mel_ce: 3.207728385925293 (nan) | > loss: 0.012907806783914566 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13660767332693294) | > loader_time: 0.0045 (0.004650853944814496)  --> TIME: 2025-02-25 15:12:06 -- STEP: 19514/36938 -- GLOBAL_STEP: 832150 | > loss_text_ce: 0.039827607572078705 (0.03387379395759559) | > loss_mel_ce: 2.0796990394592285 (nan) | > loss: 0.008410820737481117 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.1365816995258698) | > loader_time: 0.0044 (0.004650658816847502)  --> TIME: 2025-02-25 15:12:15 -- STEP: 19564/36938 -- GLOBAL_STEP: 832200 | > loss_text_ce: 0.03707221522927284 (0.03387186574477168) | > loss_mel_ce: 1.95989990234375 (nan) | > loss: 0.007924492470920086 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13659716988116904) | > loader_time: 0.0044 (0.004650639466434084)  --> TIME: 2025-02-25 15:12:25 -- STEP: 19614/36938 -- GLOBAL_STEP: 832250 | > loss_text_ce: 0.03443436697125435 (0.03386931353011914) | > loss_mel_ce: 0.0864461362361908 (nan) | > loss: 0.00047968456055969 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13659776810587504) | > loader_time: 0.0045 (0.004650574376170536)  --> TIME: 2025-02-25 15:12:34 -- STEP: 19664/36938 -- GLOBAL_STEP: 832300 | > loss_text_ce: 0.03641523793339729 (0.03387037916100446) | > loss_mel_ce: 2.9156863689422607 (nan) | > loss: 0.011714690364897251 (nan) | > current_lr: 5e-06 | > step_time: 0.1556 (0.13660718094439392) | > loader_time: 0.0047 (0.004650549919164886)  --> TIME: 2025-02-25 15:12:43 -- STEP: 19714/36938 -- GLOBAL_STEP: 832350 | > loss_text_ce: 0.042309172451496124 (0.03386614529228517) | > loss_mel_ce: 3.2137584686279297 (nan) | > loss: 0.012920904904603958 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.1366040005189994) | > loader_time: 0.0045 (0.004650487284927855)  --> TIME: 2025-02-25 15:12:53 -- STEP: 19764/36938 -- GLOBAL_STEP: 832400 | > loss_text_ce: 0.04172316938638687 (0.03386696550780808) | > loss_mel_ce: 0.21844482421875 (nan) | > loss: 0.001032412750646472 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13660465424853702) | > loader_time: 0.0046 (0.004650398922989518)  --> TIME: 2025-02-25 15:13:02 -- STEP: 19814/36938 -- GLOBAL_STEP: 832450 | > loss_text_ce: 0.031192926689982414 (0.0338661098384414) | > loss_mel_ce: 2.6760478019714355 (nan) | > loss: 0.010743020102381706 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13660820791287534) | > loader_time: 0.0046 (0.004650386152058564)  --> TIME: 2025-02-25 15:13:12 -- STEP: 19864/36938 -- GLOBAL_STEP: 832500 | > loss_text_ce: 0.03411642089486122 (0.03386660042922899) | > loss_mel_ce: 3.0463926792144775 (nan) | > loss: 0.012224243953824043 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.1366271692396806) | > loader_time: 0.0046 (0.00465038950482636)  --> TIME: 2025-02-25 15:13:21 -- STEP: 19914/36938 -- GLOBAL_STEP: 832550 | > loss_text_ce: 0.0391741506755352 (0.03386408008234352) | > loss_mel_ce: 2.868370294570923 (nan) | > loss: 0.011537875048816204 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13661975003131788) | > loader_time: 0.0046 (0.004650437150648513)  --> TIME: 2025-02-25 15:13:30 -- STEP: 19964/36938 -- GLOBAL_STEP: 832600 | > loss_text_ce: 0.03812390938401222 (0.03386341452781823) | > loss_mel_ce: 2.5171890258789062 (nan) | > loss: 0.0101401312276721 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13662038350291703) | > loader_time: 0.0046 (0.004650372394553359)  --> TIME: 2025-02-25 15:13:39 -- STEP: 20014/36938 -- GLOBAL_STEP: 832650 | > loss_text_ce: 0.030948564410209656 (0.03386392954082138) | > loss_mel_ce: 3.4407548904418945 (nan) | > loss: 0.01377660222351551 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13661284489363504) | > loader_time: 0.0047 (0.004650316003010539)  --> TIME: 2025-02-25 15:13:49 -- STEP: 20064/36938 -- GLOBAL_STEP: 832700 | > loss_text_ce: 0.03610948473215103 (0.0338610264995698) | > loss_mel_ce: 2.944045066833496 (nan) | > loss: 0.011826010420918465 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13660920300837476) | > loader_time: 0.0047 (0.004650427797954811)  --> TIME: 2025-02-25 15:13:58 -- STEP: 20114/36938 -- GLOBAL_STEP: 832750 | > loss_text_ce: 0.035095106810331345 (0.03385825175430831) | > loss_mel_ce: 3.0572471618652344 (nan) | > loss: 0.012271200306713581 (nan) | > current_lr: 5e-06 | > step_time: 0.1962 (0.1366095186951184) | > loader_time: 0.0049 (0.004650339900564226)  --> TIME: 2025-02-25 15:14:08 -- STEP: 20164/36938 -- GLOBAL_STEP: 832800 | > loss_text_ce: 0.035535942763090134 (0.0338607883315288) | > loss_mel_ce: 2.7472167015075684 (nan) | > loss: 0.011042670346796513 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13662110868840796) | > loader_time: 0.0046 (0.004650439092314123)  --> TIME: 2025-02-25 15:14:17 -- STEP: 20214/36938 -- GLOBAL_STEP: 832850 | > loss_text_ce: 0.02739792689681053 (0.033859305513061336) | > loss_mel_ce: 2.65862774848938 (nan) | > loss: 0.010658832266926765 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13663667328954407) | > loader_time: 0.0045 (0.004650460101499681)  --> TIME: 2025-02-25 15:14:27 -- STEP: 20264/36938 -- GLOBAL_STEP: 832900 | > loss_text_ce: 0.031014716252684593 (0.03385587495946923) | > loss_mel_ce: 2.793973207473755 (nan) | > loss: 0.01121027022600174 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.1366413312729211) | > loader_time: 0.0048 (0.0046504456395459115)  --> TIME: 2025-02-25 15:14:36 -- STEP: 20314/36938 -- GLOBAL_STEP: 832950 | > loss_text_ce: 0.025383174419403076 (0.033851504485320344) | > loss_mel_ce: 3.048413038253784 (nan) | > loss: 0.012197605334222317 (nan) | > current_lr: 5e-06 | > step_time: 0.1998 (0.13664801733177298) | > loader_time: 0.0049 (0.004650302180697437)  --> TIME: 2025-02-25 15:14:46 -- STEP: 20364/36938 -- GLOBAL_STEP: 833000 | > loss_text_ce: 0.027674227952957153 (0.03384942472792557) | > loss_mel_ce: 3.100304126739502 (nan) | > loss: 0.01241261325776577 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13665167286924898) | > loader_time: 0.0045 (0.004650314297710742)  --> TIME: 2025-02-25 15:14:55 -- STEP: 20414/36938 -- GLOBAL_STEP: 833050 | > loss_text_ce: 0.039792947471141815 (0.03384656161338135) | > loss_mel_ce: 2.185331344604492 (nan) | > loss: 0.008829859085381031 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13665339385446043) | > loader_time: 0.0045 (0.004650432238724081)  --> TIME: 2025-02-25 15:15:04 -- STEP: 20464/36938 -- GLOBAL_STEP: 833100 | > loss_text_ce: 0.020246295258402824 (0.033842878564908335) | > loss_mel_ce: 2.6425251960754395 (nan) | > loss: 0.01056655403226614 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13665942209274118) | > loader_time: 0.0046 (0.0046503670729166596)  --> TIME: 2025-02-25 15:15:14 -- STEP: 20514/36938 -- GLOBAL_STEP: 833150 | > loss_text_ce: 0.04342426732182503 (0.033842843812330645) | > loss_mel_ce: 2.6636404991149902 (nan) | > loss: 0.010742321610450745 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13666164882004006) | > loader_time: 0.0044 (0.004650516678299759)  --> TIME: 2025-02-25 15:15:23 -- STEP: 20564/36938 -- GLOBAL_STEP: 833200 | > loss_text_ce: 0.03753533959388733 (0.03384578683930048) | > loss_mel_ce: 1.977685570716858 (nan) | > loss: 0.007996908389031887 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13665443253039625) | > loader_time: 0.0045 (0.004650354339000055)  --> TIME: 2025-02-25 15:15:32 -- STEP: 20614/36938 -- GLOBAL_STEP: 833250 | > loss_text_ce: 0.033049292862415314 (0.033847127707485956) | > loss_mel_ce: 2.5781097412109375 (nan) | > loss: 0.010361743159592152 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13665164330078192) | > loader_time: 0.0045 (0.004650277773138272)  --> TIME: 2025-02-25 15:15:42 -- STEP: 20664/36938 -- GLOBAL_STEP: 833300 | > loss_text_ce: 0.023203816264867783 (0.03384672546744113) | > loss_mel_ce: 0.1106211319565773 (nan) | > loss: 0.0005310514243319631 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13666122421432053) | > loader_time: 0.0046 (0.004650320337116828)  --> TIME: 2025-02-25 15:15:51 -- STEP: 20714/36938 -- GLOBAL_STEP: 833350 | > loss_text_ce: 0.042176950722932816 (0.03384774834562548) | > loss_mel_ce: 3.0773987770080566 (nan) | > loss: 0.012379269115626812 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13665275799557233) | > loader_time: 0.0045 (0.0046502217669117255)  --> TIME: 2025-02-25 15:16:00 -- STEP: 20764/36938 -- GLOBAL_STEP: 833400 | > loss_text_ce: 0.03423145040869713 (0.033848333353525904) | > loss_mel_ce: 1.875244140625 (nan) | > loss: 0.00757728423923254 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13665760261570334) | > loader_time: 0.0045 (0.004650350366569254)  --> TIME: 2025-02-25 15:16:10 -- STEP: 20814/36938 -- GLOBAL_STEP: 833450 | > loss_text_ce: 0.027092237025499344 (0.03384696579488804) | > loss_mel_ce: 3.109710693359375 (nan) | > loss: 0.012447630986571312 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1366514488589346) | > loader_time: 0.0045 (0.004650275072882379)  --> TIME: 2025-02-25 15:16:19 -- STEP: 20864/36938 -- GLOBAL_STEP: 833500 | > loss_text_ce: 0.04138796404004097 (0.03384888540875558) | > loss_mel_ce: 2.947925090789795 (nan) | > loss: 0.011862354353070259 (nan) | > current_lr: 5e-06 | > step_time: 0.1679 (0.13665106451493256) | > loader_time: 0.0049 (0.004650264132790767)  --> TIME: 2025-02-25 15:16:28 -- STEP: 20914/36938 -- GLOBAL_STEP: 833550 | > loss_text_ce: 0.03313305601477623 (0.03384772992778488) | > loss_mel_ce: 2.552734375 (nan) | > loss: 0.010261379182338715 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.1366396869039465) | > loader_time: 0.0045 (0.0046502031650234935)  --> TIME: 2025-02-25 15:16:37 -- STEP: 20964/36938 -- GLOBAL_STEP: 833600 | > loss_text_ce: 0.03065633773803711 (0.03385032519294419) | > loss_mel_ce: 2.2657063007354736 (nan) | > loss: 0.009112550877034664 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.13663448165605485) | > loader_time: 0.0048 (0.004650248618690912)  --> TIME: 2025-02-25 15:16:46 -- STEP: 21014/36938 -- GLOBAL_STEP: 833650 | > loss_text_ce: 0.03369888290762901 (0.03384704999480326) | > loss_mel_ce: 2.735581159591675 (nan) | > loss: 0.010989206843078136 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.1366327311430595) | > loader_time: 0.005 (0.0046501830198949256)  --> TIME: 2025-02-25 15:16:56 -- STEP: 21064/36938 -- GLOBAL_STEP: 833700 | > loss_text_ce: 0.028590062633156776 (0.033846101467966426) | > loss_mel_ce: 1.8373500108718872 (nan) | > loss: 0.0074045248329639435 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.1366374722755881) | > loader_time: 0.0044 (0.004650167150276491)  --> TIME: 2025-02-25 15:17:05 -- STEP: 21114/36938 -- GLOBAL_STEP: 833750 | > loss_text_ce: 0.047982048243284225 (0.03384803220083806) | > loss_mel_ce: 1.833741307258606 (nan) | > loss: 0.007467156741768122 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13664881731622275) | > loader_time: 0.0046 (0.004650198251356211)  --> TIME: 2025-02-25 15:17:15 -- STEP: 21164/36938 -- GLOBAL_STEP: 833800 | > loss_text_ce: 0.04511706903576851 (0.03384647424119999) | > loss_mel_ce: 0.017998475581407547 (nan) | > loss: 0.0002504585136193782 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13665386156287046) | > loader_time: 0.0046 (0.004650165308763253)  --> TIME: 2025-02-25 15:17:24 -- STEP: 21214/36938 -- GLOBAL_STEP: 833850 | > loss_text_ce: 0.04837485030293465 (0.03384227780109832) | > loss_mel_ce: 3.1197409629821777 (nan) | > loss: 0.012571888975799084 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.1366465804578461) | > loader_time: 0.0045 (0.004650025865837666)  --> TIME: 2025-02-25 15:17:33 -- STEP: 21264/36938 -- GLOBAL_STEP: 833900 | > loss_text_ce: 0.02719806879758835 (0.03384217657658659) | > loss_mel_ce: 2.041377067565918 (nan) | > loss: 0.008208631537854671 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13663458157025357) | > loader_time: 0.0045 (0.004649981071484362)  --> TIME: 2025-02-25 15:17:42 -- STEP: 21314/36938 -- GLOBAL_STEP: 833950 | > loss_text_ce: 0.032343119382858276 (0.03384415486739586) | > loss_mel_ce: 1.7089660167694092 (nan) | > loss: 0.006909957621246576 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13663865919910628) | > loader_time: 0.0045 (0.004650093292758214)  --> TIME: 2025-02-25 15:17:52 -- STEP: 21364/36938 -- GLOBAL_STEP: 834000 | > loss_text_ce: 0.04223937913775444 (0.033846453566068004) | > loss_mel_ce: 0.8915871977806091 (nan) | > loss: 0.003705661278218031 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13664423918326898) | > loader_time: 0.0066 (0.004650155863184257)  --> TIME: 2025-02-25 15:18:01 -- STEP: 21414/36938 -- GLOBAL_STEP: 834050 | > loss_text_ce: 0.02127721533179283 (0.033845989495457474) | > loss_mel_ce: 2.879650115966797 (nan) | > loss: 0.011511617340147495 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1366441681837752) | > loader_time: 0.0045 (0.004650096293422934)  --> TIME: 2025-02-25 15:18:11 -- STEP: 21464/36938 -- GLOBAL_STEP: 834100 | > loss_text_ce: 0.03204556927084923 (0.03384692837323473) | > loss_mel_ce: 2.6955888271331787 (nan) | > loss: 0.010823946446180344 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1366539910752263) | > loader_time: 0.0046 (0.004650145280374489)  --> TIME: 2025-02-25 15:18:20 -- STEP: 21514/36938 -- GLOBAL_STEP: 834150 | > loss_text_ce: 0.03498659282922745 (0.03384612806938254) | > loss_mel_ce: 3.0813612937927246 (nan) | > loss: 0.012366460636258125 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13665395834925498) | > loader_time: 0.0046 (0.0046501034330322)  --> TIME: 2025-02-25 15:18:30 -- STEP: 21564/36938 -- GLOBAL_STEP: 834200 | > loss_text_ce: 0.037130650132894516 (0.03384663379643876) | > loss_mel_ce: 2.5036704540252686 (nan) | > loss: 0.010082544758915901 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.1366601146410726) | > loader_time: 0.0066 (0.004650217740262857)  --> TIME: 2025-02-25 15:18:39 -- STEP: 21614/36938 -- GLOBAL_STEP: 834250 | > loss_text_ce: 0.03547053039073944 (0.033844693448917544) | > loss_mel_ce: 3.0695531368255615 (nan) | > loss: 0.012321523390710354 (nan) | > current_lr: 5e-06 | > step_time: 0.1809 (0.1366606147611169) | > loader_time: 0.0048 (0.004650135402974175)  --> TIME: 2025-02-25 15:18:48 -- STEP: 21664/36938 -- GLOBAL_STEP: 834300 | > loss_text_ce: 0.03233019635081291 (0.03383978638645654) | > loss_mel_ce: 1.6136474609375 (nan) | > loss: 0.00653165765106678 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13665824380195804) | > loader_time: 0.0045 (0.004650054953475106)  --> TIME: 2025-02-25 15:18:58 -- STEP: 21714/36938 -- GLOBAL_STEP: 834350 | > loss_text_ce: 0.039667095988988876 (0.0338377134494112) | > loss_mel_ce: 2.9330599308013916 (nan) | > loss: 0.011796536855399609 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13665886944012665) | > loader_time: 0.0046 (0.00465000950522631)  --> TIME: 2025-02-25 15:19:07 -- STEP: 21764/36938 -- GLOBAL_STEP: 834400 | > loss_text_ce: 0.02538641355931759 (0.033836955177091) | > loss_mel_ce: 3.4795260429382324 (nan) | > loss: 0.013908383436501026 (nan) | > current_lr: 5e-06 | > step_time: 0.1786 (0.13666903878415287) | > loader_time: 0.0048 (0.004650023081705327)  --> TIME: 2025-02-25 15:19:16 -- STEP: 21814/36938 -- GLOBAL_STEP: 834450 | > loss_text_ce: 0.03751154989004135 (0.033837841361366014) | > loss_mel_ce: 3.571498155593872 (nan) | > loss: 0.014321467839181423 (nan) | > current_lr: 5e-06 | > step_time: 0.1648 (0.1366705077153767) | > loader_time: 0.0047 (0.004650035601352096)  --> TIME: 2025-02-25 15:19:26 -- STEP: 21864/36938 -- GLOBAL_STEP: 834500 | > loss_text_ce: 0.02534589171409607 (0.03383651066083673) | > loss_mel_ce: 2.6977055072784424 (nan) | > loss: 0.010805759578943253 (nan) | > current_lr: 5e-06 | > step_time: 0.1833 (0.13666463691020314) | > loader_time: 0.0049 (0.004650003343896887)  --> TIME: 2025-02-25 15:19:35 -- STEP: 21914/36938 -- GLOBAL_STEP: 834550 | > loss_text_ce: 0.035209428519010544 (0.03383438152688539) | > loss_mel_ce: 3.3953301906585693 (nan) | > loss: 0.013613252900540829 (nan) | > current_lr: 5e-06 | > step_time: 0.188 (0.13667644287663788) | > loader_time: 0.0049 (0.004650212317756814)  --> TIME: 2025-02-25 15:19:45 -- STEP: 21964/36938 -- GLOBAL_STEP: 834600 | > loss_text_ce: 0.031460318714380264 (0.03383665486473641) | > loss_mel_ce: 2.644521713256836 (nan) | > loss: 0.01061897724866867 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13667861053232122) | > loader_time: 0.0046 (0.004650201558241773)  --> TIME: 2025-02-25 15:19:54 -- STEP: 22014/36938 -- GLOBAL_STEP: 834650 | > loss_text_ce: 0.03478148579597473 (0.03383709654399672) | > loss_mel_ce: 3.2862610816955566 (nan) | > loss: 0.013178740628063679 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.13668725426021522) | > loader_time: 0.0045 (0.004650148739333437)  --> TIME: 2025-02-25 15:20:03 -- STEP: 22064/36938 -- GLOBAL_STEP: 834700 | > loss_text_ce: 0.04225265979766846 (0.0338340880254053) | > loss_mel_ce: 2.808159112930298 (nan) | > loss: 0.011311158537864685 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13667562160913937) | > loader_time: 0.0047 (0.004650095554691345)  --> TIME: 2025-02-25 15:20:13 -- STEP: 22114/36938 -- GLOBAL_STEP: 834750 | > loss_text_ce: 0.03452974557876587 (0.03383438788821706) | > loss_mel_ce: 2.9026081562042236 (nan) | > loss: 0.011655309237539768 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13668420698173453) | > loader_time: 0.0076 (0.0046502950017486685)  --> TIME: 2025-02-25 15:20:22 -- STEP: 22164/36938 -- GLOBAL_STEP: 834800 | > loss_text_ce: 0.02699887938797474 (0.03383460808674777) | > loss_mel_ce: 2.531294822692871 (nan) | > loss: 0.010151959955692291 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.13668332207915163) | > loader_time: 0.0047 (0.004650434299273453)  --> TIME: 2025-02-25 15:20:32 -- STEP: 22214/36938 -- GLOBAL_STEP: 834850 | > loss_text_ce: 0.04320568963885307 (0.03383619562992155) | > loss_mel_ce: 0.3524373471736908 (nan) | > loss: 0.0015700120711699128 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13669055124487148) | > loader_time: 0.0045 (0.004650759527382075)  --> TIME: 2025-02-25 15:20:41 -- STEP: 22264/36938 -- GLOBAL_STEP: 834900 | > loss_text_ce: 0.03345341235399246 (0.03383586745188137) | > loss_mel_ce: 1.985498070716858 (nan) | > loss: 0.008011712692677975 (nan) | > current_lr: 5e-06 | > step_time: 0.1832 (0.13669643714151372) | > loader_time: 0.0049 (0.004650775987040493)  --> TIME: 2025-02-25 15:20:50 -- STEP: 22314/36938 -- GLOBAL_STEP: 834950 | > loss_text_ce: 0.03431229293346405 (0.03383606977377147) | > loss_mel_ce: 2.9533116817474365 (nan) | > loss: 0.011855651624500751 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13669405423089523) | > loader_time: 0.0046 (0.004650776313818645)  --> TIME: 2025-02-25 15:20:59 -- STEP: 22364/36938 -- GLOBAL_STEP: 835000 | > loss_text_ce: 0.04197516292333603 (0.03383735635219087) | > loss_mel_ce: 2.500103712081909 (nan) | > loss: 0.010087615810334682 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13668921125572783) | > loader_time: 0.0047 (0.004650721735912476)  --> TIME: 2025-02-25 15:21:09 -- STEP: 22414/36938 -- GLOBAL_STEP: 835050 | > loss_text_ce: 0.02398972027003765 (0.03383617048925722) | > loss_mel_ce: 2.908458709716797 (nan) | > loss: 0.011636700481176376 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13668785312636034) | > loader_time: 0.0045 (0.004651039729761232)  --> TIME: 2025-02-25 15:21:18 -- STEP: 22464/36938 -- GLOBAL_STEP: 835100 | > loss_text_ce: 0.02772452123463154 (0.033834257128531095) | > loss_mel_ce: 2.5478882789611816 (nan) | > loss: 0.010220685973763466 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13668761972306248) | > loader_time: 0.0046 (0.004651165649592008)  --> TIME: 2025-02-25 15:21:27 -- STEP: 22514/36938 -- GLOBAL_STEP: 835150 | > loss_text_ce: 0.034174442291259766 (0.03383511217150363) | > loss_mel_ce: 3.2984917163848877 (nan) | > loss: 0.01322486624121666 (nan) | > current_lr: 5e-06 | > step_time: 0.1441 (0.13668617702795005) | > loader_time: 0.0046 (0.00465127145078111)  --> TIME: 2025-02-25 15:21:36 -- STEP: 22564/36938 -- GLOBAL_STEP: 835200 | > loss_text_ce: 0.03169337660074234 (0.033834264176671375) | > loss_mel_ce: 2.732410430908203 (nan) | > loss: 0.010968666523694992 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13668158693191995) | > loader_time: 0.0046 (0.004651336557073491)  --> TIME: 2025-02-25 15:21:46 -- STEP: 22614/36938 -- GLOBAL_STEP: 835250 | > loss_text_ce: 0.0401487872004509 (0.03383471331217055) | > loss_mel_ce: 3.073667049407959 (nan) | > loss: 0.012356412597000599 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13668282027585005) | > loader_time: 0.0047 (0.004651353099232911)  --> TIME: 2025-02-25 15:21:55 -- STEP: 22664/36938 -- GLOBAL_STEP: 835300 | > loss_text_ce: 0.039392679929733276 (0.03383433145745089) | > loss_mel_ce: 2.118321418762207 (nan) | > loss: 0.008562358096241951 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13668774765286085) | > loader_time: 0.0046 (0.004651496036289171)  --> TIME: 2025-02-25 15:22:05 -- STEP: 22714/36938 -- GLOBAL_STEP: 835350 | > loss_text_ce: 0.038874249905347824 (0.033835863990666064) | > loss_mel_ce: 2.4054744243621826 (nan) | > loss: 0.009699796326458454 (nan) | > current_lr: 5e-06 | > step_time: 0.163 (0.13670496895687698) | > loader_time: 0.0047 (0.004651519638599604)  --> TIME: 2025-02-25 15:22:15 -- STEP: 22764/36938 -- GLOBAL_STEP: 835400 | > loss_text_ce: 0.03433149680495262 (0.033836679987575335) | > loss_mel_ce: 2.75249981880188 (nan) | > loss: 0.01105885487049818 (nan) | > current_lr: 5e-06 | > step_time: 0.156 (0.13671237753975665) | > loader_time: 0.0048 (0.0046517259101418445)  --> TIME: 2025-02-25 15:22:24 -- STEP: 22814/36938 -- GLOBAL_STEP: 835450 | > loss_text_ce: 0.02934587560594082 (0.03383553585936826) | > loss_mel_ce: 2.370859384536743 (nan) | > loss: 0.009524624794721603 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.13671750775497296) | > loader_time: 0.0046 (0.0046517035185094004)  --> TIME: 2025-02-25 15:22:34 -- STEP: 22864/36938 -- GLOBAL_STEP: 835500 | > loss_text_ce: 0.04007963091135025 (0.03383525902562774) | > loss_mel_ce: 2.8245790004730225 (nan) | > loss: 0.011367693543434143 (nan) | > current_lr: 5e-06 | > step_time: 0.1619 (0.13672483139433625) | > loader_time: 0.0054 (0.004651738232962298)  --> TIME: 2025-02-25 15:22:43 -- STEP: 22914/36938 -- GLOBAL_STEP: 835550 | > loss_text_ce: 0.030412346124649048 (0.03383642611266719) | > loss_mel_ce: 2.24609375 (nan) | > loss: 0.009033755399286747 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13671927874735793) | > loader_time: 0.0046 (0.004652069305222793)  --> TIME: 2025-02-25 15:22:52 -- STEP: 22964/36938 -- GLOBAL_STEP: 835600 | > loss_text_ce: 0.047714345157146454 (0.033835405128816734) | > loss_mel_ce: 1.6143537759780884 (nan) | > loss: 0.00659550866112113 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13671715312294067) | > loader_time: 0.0047 (0.004652240325756284)  --> TIME: 2025-02-25 15:23:01 -- STEP: 23014/36938 -- GLOBAL_STEP: 835650 | > loss_text_ce: 0.04167405143380165 (0.03383569215329909) | > loss_mel_ce: 0.02618916891515255 (nan) | > loss: 0.00026929849991574883 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.1367172374291901) | > loader_time: 0.0046 (0.0046524202791523635)  --> TIME: 2025-02-25 15:23:10 -- STEP: 23064/36938 -- GLOBAL_STEP: 835700 | > loss_text_ce: 0.03937981650233269 (0.03383761464632361) | > loss_mel_ce: 3.349452257156372 (nan) | > loss: 0.013447747565805912 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13670806368109442) | > loader_time: 0.0046 (0.004652366553831533)  --> TIME: 2025-02-25 15:23:20 -- STEP: 23114/36938 -- GLOBAL_STEP: 835750 | > loss_text_ce: 0.03501853346824646 (0.03384051110693455) | > loss_mel_ce: 1.1994781494140625 (nan) | > loss: 0.0048987967893481255 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13671107949028166) | > loader_time: 0.0046 (0.004652323138603452)  --> TIME: 2025-02-25 15:23:29 -- STEP: 23164/36938 -- GLOBAL_STEP: 835800 | > loss_text_ce: 0.031814638525247574 (0.033838359693808606) | > loss_mel_ce: 2.1359822750091553 (nan) | > loss: 0.008602368645370007 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.1367101310922009) | > loader_time: 0.0047 (0.004652446857327881)  --> TIME: 2025-02-25 15:23:39 -- STEP: 23214/36938 -- GLOBAL_STEP: 835850 | > loss_text_ce: 0.03572187200188637 (0.03383676266483688) | > loss_mel_ce: 2.87591552734375 (nan) | > loss: 0.011554117314517498 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.1367189790827271) | > loader_time: 0.0047 (0.004652591693244188)  --> TIME: 2025-02-25 15:23:48 -- STEP: 23264/36938 -- GLOBAL_STEP: 835900 | > loss_text_ce: 0.037176694720983505 (0.03383971872011535) | > loss_mel_ce: 3.1623756885528564 (nan) | > loss: 0.012696636840701103 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.1367178022082611) | > loader_time: 0.0047 (0.004652557082402484)  --> TIME: 2025-02-25 15:23:58 -- STEP: 23314/36938 -- GLOBAL_STEP: 835950 | > loss_text_ce: 0.032710228115320206 (0.03384172048110046) | > loss_mel_ce: 2.552323341369629 (nan) | > loss: 0.010258070193231106 (nan) | > current_lr: 5e-06 | > step_time: 0.1835 (0.13672685239678545) | > loader_time: 0.0048 (0.004652622992253842)  --> TIME: 2025-02-25 15:24:07 -- STEP: 23364/36938 -- GLOBAL_STEP: 836000 | > loss_text_ce: 0.03973783925175667 (0.03384340414625925) | > loss_mel_ce: 2.6402437686920166 (nan) | > loss: 0.010634848847985268 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.1367225035415333) | > loader_time: 0.0046 (0.004652663547481248)  --> TIME: 2025-02-25 15:24:16 -- STEP: 23414/36938 -- GLOBAL_STEP: 836050 | > loss_text_ce: 0.03403211757540703 (0.03384160553337869) | > loss_mel_ce: 2.583369493484497 (nan) | > loss: 0.010386514477431774 (nan) | > current_lr: 5e-06 | > step_time: 0.1909 (0.1367261447435652) | > loader_time: 0.0049 (0.004652681741317751)  --> TIME: 2025-02-25 15:24:26 -- STEP: 23464/36938 -- GLOBAL_STEP: 836100 | > loss_text_ce: 0.03175905719399452 (0.033838140066415286) | > loss_mel_ce: 2.5611345767974854 (nan) | > loss: 0.010289261117577553 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13672562214322762) | > loader_time: 0.0046 (0.0046529297510836945)  --> TIME: 2025-02-25 15:24:35 -- STEP: 23514/36938 -- GLOBAL_STEP: 836150 | > loss_text_ce: 0.03312825411558151 (0.033835431180843416) | > loss_mel_ce: 2.3626606464385986 (nan) | > loss: 0.00950709916651249 (nan) | > current_lr: 5e-06 | > step_time: 0.1352 (0.13672663026433246) | > loader_time: 0.0046 (0.004652972072122078)  --> TIME: 2025-02-25 15:24:44 -- STEP: 23564/36938 -- GLOBAL_STEP: 836200 | > loss_text_ce: 0.03424620255827904 (0.033838798418731886) | > loss_mel_ce: 3.343851327896118 (nan) | > loss: 0.013405149802565575 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13672207834926686) | > loader_time: 0.0046 (0.004653096057384805)  --> TIME: 2025-02-25 15:24:53 -- STEP: 23614/36938 -- GLOBAL_STEP: 836250 | > loss_text_ce: 0.03667115420103073 (0.033838133764849) | > loss_mel_ce: 2.501293897628784 (nan) | > loss: 0.010071290656924248 (nan) | > current_lr: 5e-06 | > step_time: 0.1357 (0.1367140221658369) | > loader_time: 0.0047 (0.004653087849236153)  --> TIME: 2025-02-25 15:25:02 -- STEP: 23664/36938 -- GLOBAL_STEP: 836300 | > loss_text_ce: 0.028471026569604874 (0.03383523288293587) | > loss_mel_ce: 2.4361815452575684 (nan) | > loss: 0.009780367836356163 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13670441684608517) | > loader_time: 0.0046 (0.004653169959279782)  --> TIME: 2025-02-25 15:25:11 -- STEP: 23714/36938 -- GLOBAL_STEP: 836350 | > loss_text_ce: 0.025061067193746567 (0.03383377737854146) | > loss_mel_ce: 2.396200180053711 (nan) | > loss: 0.00960818026214838 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13670153258733064) | > loader_time: 0.0066 (0.004653346079079697)  --> TIME: 2025-02-25 15:25:21 -- STEP: 23764/36938 -- GLOBAL_STEP: 836400 | > loss_text_ce: 0.030977437272667885 (0.033834080960944285) | > loss_mel_ce: 3.3606934547424316 (nan) | > loss: 0.01345901284366846 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.13669779275206545) | > loader_time: 0.0047 (0.004653295359413248)  --> TIME: 2025-02-25 15:25:30 -- STEP: 23814/36938 -- GLOBAL_STEP: 836450 | > loss_text_ce: 0.037868138402700424 (0.03383558006521356) | > loss_mel_ce: 2.5709068775177 (nan) | > loss: 0.010352281853556633 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.1367003582101976) | > loader_time: 0.0047 (0.004653344198809834)  --> TIME: 2025-02-25 15:25:39 -- STEP: 23864/36938 -- GLOBAL_STEP: 836500 | > loss_text_ce: 0.033017609268426895 (0.03383506338003256) | > loss_mel_ce: 3.239915132522583 (nan) | > loss: 0.012987828813493252 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.13670117822901962) | > loader_time: 0.0049 (0.004653483389449354)  --> TIME: 2025-02-25 15:25:49 -- STEP: 23914/36938 -- GLOBAL_STEP: 836550 | > loss_text_ce: 0.03984071686863899 (0.03383281673815303) | > loss_mel_ce: 2.106260299682617 (nan) | > loss: 0.008516274392604828 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13670522542345628) | > loader_time: 0.0048 (0.004653577253433064)  --> TIME: 2025-02-25 15:25:58 -- STEP: 23964/36938 -- GLOBAL_STEP: 836600 | > loss_text_ce: 0.03060617670416832 (0.03383572111207048) | > loss_mel_ce: 2.5400846004486084 (nan) | > loss: 0.010201155208051205 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13670520953793067) | > loader_time: 0.0046 (0.00465371587443203)  --> TIME: 2025-02-25 15:26:08 -- STEP: 24014/36938 -- GLOBAL_STEP: 836650 | > loss_text_ce: 0.04179062694311142 (0.03383385254928965) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13670709455421373) | > loader_time: 0.0045 (0.004653701429175631)  --> TIME: 2025-02-25 15:26:17 -- STEP: 24064/36938 -- GLOBAL_STEP: 836700 | > loss_text_ce: 0.04111446812748909 (0.033832665164191776) | > loss_mel_ce: 3.0817739963531494 (nan) | > loss: 0.012392415665090084 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.136709456341936) | > loader_time: 0.0048 (0.004654035418036754)  --> TIME: 2025-02-25 15:26:26 -- STEP: 24114/36938 -- GLOBAL_STEP: 836750 | > loss_text_ce: 0.035270486027002335 (0.033832816882092207) | > loss_mel_ce: 3.4635417461395264 (nan) | > loss: 0.01388417650014162 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13670970247269815) | > loader_time: 0.0045 (0.004654087899306054)  --> TIME: 2025-02-25 15:26:35 -- STEP: 24164/36938 -- GLOBAL_STEP: 836800 | > loss_text_ce: 0.03296014666557312 (0.03383537928151061) | > loss_mel_ce: 2.0349104404449463 (nan) | > loss: 0.00820583663880825 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13670894028512584) | > loader_time: 0.0045 (0.00465412847136566)  --> TIME: 2025-02-25 15:26:45 -- STEP: 24214/36938 -- GLOBAL_STEP: 836850 | > loss_text_ce: 0.024304358288645744 (0.03383374875725241) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13670272351374718) | > loader_time: 0.0045 (0.004654133596536065)  --> TIME: 2025-02-25 15:26:54 -- STEP: 24264/36938 -- GLOBAL_STEP: 836900 | > loss_text_ce: 0.03306111320853233 (0.033834209538723854) | > loss_mel_ce: 2.7969448566436768 (nan) | > loss: 0.011230182833969593 (nan) | > current_lr: 5e-06 | > step_time: 0.144 (0.13670435166579087) | > loader_time: 0.0047 (0.004654153311877518)  --> TIME: 2025-02-25 15:27:04 -- STEP: 24314/36938 -- GLOBAL_STEP: 836950 | > loss_text_ce: 0.041092418134212494 (0.03383268113219323) | > loss_mel_ce: 3.37292742729187 (nan) | > loss: 0.01354769803583622 (nan) | > current_lr: 5e-06 | > step_time: 0.1708 (0.13672300326232936) | > loader_time: 0.0049 (0.004654299852986696)  --> TIME: 2025-02-25 15:27:13 -- STEP: 24364/36938 -- GLOBAL_STEP: 837000 | > loss_text_ce: 0.021469878032803535 (0.033830134025979036) | > loss_mel_ce: 2.9658267498016357 (nan) | > loss: 0.011854352429509163 (nan) | > current_lr: 5e-06 | > step_time: 0.146 (0.13672511545509172) | > loader_time: 0.0047 (0.004654333120182286)  --> TIME: 2025-02-25 15:27:23 -- STEP: 24414/36938 -- GLOBAL_STEP: 837050 | > loss_text_ce: 0.03023594617843628 (0.03383096639777085) | > loss_mel_ce: 2.915724277496338 (nan) | > loss: 0.011690319515764713 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13672666156322277) | > loader_time: 0.0045 (0.0046543677354938945)  --> TIME: 2025-02-25 15:27:33 -- STEP: 24464/36938 -- GLOBAL_STEP: 837100 | > loss_text_ce: 0.02842039428651333 (0.03382946817954169) | > loss_mel_ce: 1.7097574472427368 (nan) | > loss: 0.006897531915456057 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.1367453899919173) | > loader_time: 0.0046 (0.004654431056555297)  --> TIME: 2025-02-25 15:27:42 -- STEP: 24514/36938 -- GLOBAL_STEP: 837150 | > loss_text_ce: 0.03732801601290703 (0.03383028512900994) | > loss_mel_ce: 3.045943260192871 (nan) | > loss: 0.012235203757882118 (nan) | > current_lr: 5e-06 | > step_time: 0.1724 (0.136740531512514) | > loader_time: 0.0048 (0.004654427089008012)  --> TIME: 2025-02-25 15:27:51 -- STEP: 24564/36938 -- GLOBAL_STEP: 837200 | > loss_text_ce: 0.03167858347296715 (0.033829053738311465) | > loss_mel_ce: 2.262563467025757 (nan) | > loss: 0.009104136377573013 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13674228358513377) | > loader_time: 0.0046 (0.004654499077482456)  --> TIME: 2025-02-25 15:28:00 -- STEP: 24614/36938 -- GLOBAL_STEP: 837250 | > loss_text_ce: 0.017743878066539764 (0.03382943811045979) | > loss_mel_ce: 2.35257625579834 (nan) | > loss: 0.009406032972037792 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1367444206893986) | > loader_time: 0.0045 (0.004654406377609939)  --> TIME: 2025-02-25 15:28:10 -- STEP: 24664/36938 -- GLOBAL_STEP: 837300 | > loss_text_ce: 0.0433044470846653 (0.03382727811546488) | > loss_mel_ce: 3.001772165298462 (nan) | > loss: 0.012083638459444046 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13675559528880152) | > loader_time: 0.0045 (0.00465445621153614)  --> TIME: 2025-02-25 15:28:20 -- STEP: 24714/36938 -- GLOBAL_STEP: 837350 | > loss_text_ce: 0.0441729873418808 (0.033829481725547755) | > loss_mel_ce: 3.0631864070892334 (nan) | > loss: 0.012330791912972927 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13676492465760876) | > loader_time: 0.0046 (0.004654455669221236)  --> TIME: 2025-02-25 15:28:29 -- STEP: 24764/36938 -- GLOBAL_STEP: 837400 | > loss_text_ce: 0.03128562122583389 (0.03383107341635217) | > loss_mel_ce: 0.7473170161247253 (nan) | > loss: 0.0030896933749318123 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.13675678044947537) | > loader_time: 0.0045 (0.004654366680078089)  --> TIME: 2025-02-25 15:28:38 -- STEP: 24814/36938 -- GLOBAL_STEP: 837450 | > loss_text_ce: 0.02492348849773407 (0.03383056528474759) | > loss_mel_ce: 2.6260921955108643 (nan) | > loss: 0.01051990408450365 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13675987603269166) | > loader_time: 0.0046 (0.0046543385333573575)  --> TIME: 2025-02-25 15:28:48 -- STEP: 24864/36938 -- GLOBAL_STEP: 837500 | > loss_text_ce: 0.025250496342778206 (0.03383175411457596) | > loss_mel_ce: 3.2095580101013184 (nan) | > loss: 0.01283654198050499 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13676052982589343) | > loader_time: 0.0046 (0.004654275135950992)  --> TIME: 2025-02-25 15:28:57 -- STEP: 24914/36938 -- GLOBAL_STEP: 837550 | > loss_text_ce: 0.03861059993505478 (0.03383087834574038) | > loss_mel_ce: 2.6427886486053467 (nan) | > loss: 0.010640474036335945 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.13675854107900517) | > loader_time: 0.0046 (0.004654235438683477)  --> TIME: 2025-02-25 15:29:06 -- STEP: 24964/36938 -- GLOBAL_STEP: 837600 | > loss_text_ce: 0.033381812274456024 (0.03383105544457264) | > loss_mel_ce: 3.3471248149871826 (nan) | > loss: 0.013414708897471428 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13676091379661715) | > loader_time: 0.0047 (0.00465435387518634)  --> TIME: 2025-02-25 15:29:16 -- STEP: 25014/36938 -- GLOBAL_STEP: 837650 | > loss_text_ce: 0.03918199986219406 (0.033830032814593514) | > loss_mel_ce: 3.1356139183044434 (nan) | > loss: 0.012598397210240364 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.13677253371054465) | > loader_time: 0.0047 (0.004654480731009743)  --> TIME: 2025-02-25 15:29:25 -- STEP: 25064/36938 -- GLOBAL_STEP: 837700 | > loss_text_ce: 0.03308164328336716 (0.03383074075065651) | > loss_mel_ce: 2.3429014682769775 (nan) | > loss: 0.009428504854440689 (nan) | > current_lr: 5e-06 | > step_time: 0.1537 (0.13677128053609527) | > loader_time: 0.0048 (0.00465449218052931)  --> TIME: 2025-02-25 15:29:35 -- STEP: 25114/36938 -- GLOBAL_STEP: 837750 | > loss_text_ce: 0.037186987698078156 (0.033832676700020756) | > loss_mel_ce: 3.0009639263153076 (nan) | > loss: 0.012056155130267143 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.1367737078083797) | > loader_time: 0.0046 (0.004654453838764879)  --> TIME: 2025-02-25 15:29:44 -- STEP: 25164/36938 -- GLOBAL_STEP: 837800 | > loss_text_ce: 0.03140342980623245 (0.0338336546482536) | > loss_mel_ce: 3.155592203140259 (nan) | > loss: 0.012646809220314026 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13676913537616903) | > loader_time: 0.0046 (0.004654334148946042)  --> TIME: 2025-02-25 15:29:53 -- STEP: 25214/36938 -- GLOBAL_STEP: 837850 | > loss_text_ce: 0.039035432040691376 (0.03383520820848943) | > loss_mel_ce: 2.6003825664520264 (nan) | > loss: 0.010473880916833878 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13677282402212343) | > loader_time: 0.0045 (0.004654264765897007)  --> TIME: 2025-02-25 15:30:03 -- STEP: 25264/36938 -- GLOBAL_STEP: 837900 | > loss_text_ce: 0.03577698767185211 (0.033834228903454014) | > loss_mel_ce: 1.5148446559906006 (nan) | > loss: 0.006153260823339224 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13678305794763976) | > loader_time: 0.0046 (0.004654243437454167)  --> TIME: 2025-02-25 15:30:12 -- STEP: 25314/36938 -- GLOBAL_STEP: 837950 | > loss_text_ce: 0.030757201835513115 (0.03383357578368617) | > loss_mel_ce: 3.0311224460601807 (nan) | > loss: 0.012150316499173641 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.1367860683626676) | > loader_time: 0.0046 (0.004654259819964465)  --> TIME: 2025-02-25 15:30:22 -- STEP: 25364/36938 -- GLOBAL_STEP: 838000 | > loss_text_ce: 0.033222366124391556 (0.033831611615831224) | > loss_mel_ce: 2.6526072025299072 (nan) | > loss: 0.010658054612576962 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13678443006979704) | > loader_time: 0.0046 (0.004654157003794641)  --> TIME: 2025-02-25 15:30:31 -- STEP: 25414/36938 -- GLOBAL_STEP: 838050 | > loss_text_ce: 0.03569035604596138 (0.03382897734100561) | > loss_mel_ce: 3.181453227996826 (nan) | > loss: 0.012766443192958832 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13678251865800115) | > loader_time: 0.0046 (0.004654130553283594)  --> TIME: 2025-02-25 15:30:40 -- STEP: 25464/36938 -- GLOBAL_STEP: 838100 | > loss_text_ce: 0.03131318837404251 (0.033829942888525116) | > loss_mel_ce: 2.7334530353546143 (nan) | > loss: 0.010971294716000557 (nan) | > current_lr: 5e-06 | > step_time: 0.1842 (0.136791445467033) | > loader_time: 0.0048 (0.004654131162627251)  --> TIME: 2025-02-25 15:30:50 -- STEP: 25514/36938 -- GLOBAL_STEP: 838150 | > loss_text_ce: 0.03116379678249359 (0.03383018616903781) | > loss_mel_ce: 2.8936007022857666 (nan) | > loss: 0.011606208980083466 (nan) | > current_lr: 5e-06 | > step_time: 0.1625 (0.13679160886380187) | > loader_time: 0.0047 (0.004654230738429983)  --> TIME: 2025-02-25 15:30:59 -- STEP: 25564/36938 -- GLOBAL_STEP: 838200 | > loss_text_ce: 0.040463197976350784 (0.03383325971386138) | > loss_mel_ce: 3.332611560821533 (nan) | > loss: 0.013385218568146229 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13679271759216205) | > loader_time: 0.0046 (0.004654248020791078)  --> TIME: 2025-02-25 15:31:08 -- STEP: 25614/36938 -- GLOBAL_STEP: 838250 | > loss_text_ce: 0.040203746408224106 (0.03383251806222102) | > loss_mel_ce: 3.028033494949341 (nan) | > loss: 0.012175546027719975 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13678258102225238) | > loader_time: 0.0046 (0.004654328689238604)  --> TIME: 2025-02-25 15:31:17 -- STEP: 25664/36938 -- GLOBAL_STEP: 838300 | > loss_text_ce: 0.03550848737359047 (0.033831437726774036) | > loss_mel_ce: 2.737562894821167 (nan) | > loss: 0.011004251427948475 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.1367820883332352) | > loader_time: 0.0047 (0.004654528178330364)  --> TIME: 2025-02-25 15:31:27 -- STEP: 25714/36938 -- GLOBAL_STEP: 838350 | > loss_text_ce: 0.03343057259917259 (0.033830107057517475) | > loss_mel_ce: 2.6954598426818848 (nan) | > loss: 0.010828930884599686 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.13677432066620857) | > loader_time: 0.0046 (0.004654436540926294)  --> TIME: 2025-02-25 15:31:36 -- STEP: 25764/36938 -- GLOBAL_STEP: 838400 | > loss_text_ce: 0.031758930534124374 (0.03383093685382531) | > loss_mel_ce: 2.4583899974823 (nan) | > loss: 0.009881543926894665 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.1367716268896416) | > loader_time: 0.0045 (0.004654353495211239)  --> TIME: 2025-02-25 15:31:45 -- STEP: 25814/36938 -- GLOBAL_STEP: 838450 | > loss_text_ce: 0.03219810128211975 (0.0338309540932414) | > loss_mel_ce: 2.283609390258789 (nan) | > loss: 0.009189712814986706 (nan) | > current_lr: 5e-06 | > step_time: 0.1864 (0.13677805499139495) | > loader_time: 0.0049 (0.004654391208884223)  --> TIME: 2025-02-25 15:31:55 -- STEP: 25864/36938 -- GLOBAL_STEP: 838500 | > loss_text_ce: 0.03614033758640289 (0.03383343411841291) | > loss_mel_ce: 1.9840706586837769 (nan) | > loss: 0.008016711100935936 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13678056598078436) | > loader_time: 0.0046 (0.0046545504657174155)  --> TIME: 2025-02-25 15:32:04 -- STEP: 25914/36938 -- GLOBAL_STEP: 838550 | > loss_text_ce: 0.034438882023096085 (0.03383288320387276) | > loss_mel_ce: 1.9736984968185425 (nan) | > loss: 0.007968800142407417 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13677858307174534) | > loader_time: 0.0044 (0.004654592815221629)  --> TIME: 2025-02-25 15:32:14 -- STEP: 25964/36938 -- GLOBAL_STEP: 838600 | > loss_text_ce: 0.03947357088327408 (0.03383301995885331) | > loss_mel_ce: 2.9944655895233154 (nan) | > loss: 0.012039441615343094 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13678755756152555) | > loader_time: 0.0046 (0.004654648031811546)  --> TIME: 2025-02-25 15:32:23 -- STEP: 26014/36938 -- GLOBAL_STEP: 838650 | > loss_text_ce: 0.021394308656454086 (0.033834319314801735) | > loss_mel_ce: 2.7771151065826416 (nan) | > loss: 0.011105196550488472 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.1367872761774556) | > loader_time: 0.0048 (0.00465460545627912)  --> TIME: 2025-02-25 15:32:32 -- STEP: 26064/36938 -- GLOBAL_STEP: 838700 | > loss_text_ce: 0.03165126591920853 (0.033832285312275195) | > loss_mel_ce: 4.00390625 (nan) | > loss: 0.01601411961019039 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.136782112920702) | > loader_time: 0.0067 (0.004654601207168779)  --> TIME: 2025-02-25 15:32:41 -- STEP: 26114/36938 -- GLOBAL_STEP: 838750 | > loss_text_ce: 0.036352429538965225 (0.0338304625196627) | > loss_mel_ce: 1.6124842166900635 (nan) | > loss: 0.006543003022670746 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13678721927104723) | > loader_time: 0.0047 (0.004654633904832723)  --> TIME: 2025-02-25 15:32:51 -- STEP: 26164/36938 -- GLOBAL_STEP: 838800 | > loss_text_ce: 0.038645822554826736 (0.03382977849204163) | > loss_mel_ce: 2.7570698261260986 (nan) | > loss: 0.01109411008656025 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13677823396837932) | > loader_time: 0.0045 (0.004654589349606432)  --> TIME: 2025-02-25 15:33:00 -- STEP: 26214/36938 -- GLOBAL_STEP: 838850 | > loss_text_ce: 0.023341631516814232 (0.03383049777700224) | > loss_mel_ce: 2.2786622047424316 (nan) | > loss: 0.009134936146438122 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.13677245176128672) | > loader_time: 0.0046 (0.00465455554193222)  --> TIME: 2025-02-25 15:33:09 -- STEP: 26264/36938 -- GLOBAL_STEP: 838900 | > loss_text_ce: 0.043570518493652344 (0.033830080055370866) | > loss_mel_ce: 1.4918464422225952 (nan) | > loss: 0.006092924624681473 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13678309039992675) | > loader_time: 0.0045 (0.004654531921150831)  --> TIME: 2025-02-25 15:33:19 -- STEP: 26314/36938 -- GLOBAL_STEP: 838950 | > loss_text_ce: 0.02742638811469078 (0.033831753950759325) | > loss_mel_ce: 1.1468058824539185 (nan) | > loss: 0.00465965224429965 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.1367902119869869) | > loader_time: 0.0045 (0.0046545341582594265)  --> TIME: 2025-02-25 15:33:28 -- STEP: 26364/36938 -- GLOBAL_STEP: 839000 | > loss_text_ce: 0.04674999788403511 (0.03383156162679032) | > loss_mel_ce: 2.958224058151245 (nan) | > loss: 0.01192450150847435 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13679363362761382) | > loader_time: 0.0046 (0.004654564231322133)  --> TIME: 2025-02-25 15:33:38 -- STEP: 26414/36938 -- GLOBAL_STEP: 839050 | > loss_text_ce: 0.030788855627179146 (0.033831260942435895) | > loss_mel_ce: 2.2111666202545166 (nan) | > loss: 0.008896649815142155 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.1367965339305526) | > loader_time: 0.0046 (0.004654593703116211)  --> TIME: 2025-02-25 15:33:48 -- STEP: 26464/36938 -- GLOBAL_STEP: 839100 | > loss_text_ce: 0.03215279057621956 (0.033830551366287265) | > loss_mel_ce: 3.7579102516174316 (nan) | > loss: 0.01503993384540081 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13680716476965216) | > loader_time: 0.0045 (0.0046546356763776196)  --> TIME: 2025-02-25 15:33:57 -- STEP: 26514/36938 -- GLOBAL_STEP: 839150 | > loss_text_ce: 0.03456807881593704 (0.033829355844437636) | > loss_mel_ce: 2.4919769763946533 (nan) | > loss: 0.010025973431766033 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13679953810945203) | > loader_time: 0.0045 (0.004654688281945307)  --> TIME: 2025-02-25 15:34:06 -- STEP: 26564/36938 -- GLOBAL_STEP: 839200 | > loss_text_ce: 0.03353588655591011 (0.03382958933776136) | > loss_mel_ce: 3.2231664657592773 (nan) | > loss: 0.012923423200845718 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.136789923887479) | > loader_time: 0.0046 (0.004654675583005781)  --> TIME: 2025-02-25 15:34:15 -- STEP: 26614/36938 -- GLOBAL_STEP: 839250 | > loss_text_ce: 0.035239897668361664 (0.03382703380093317) | > loss_mel_ce: 2.9391958713531494 (nan) | > loss: 0.011803317815065384 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13678793727281138) | > loader_time: 0.0046 (0.0046546073091408915)  --> TIME: 2025-02-25 15:34:24 -- STEP: 26664/36938 -- GLOBAL_STEP: 839300 | > loss_text_ce: 0.037943948060274124 (0.03382603678439626) | > loss_mel_ce: 2.969010829925537 (nan) | > loss: 0.01193236093968153 (nan) | > current_lr: 5e-06 | > step_time: 0.1648 (0.13679081082630265) | > loader_time: 0.0047 (0.004654600541226095)  --> TIME: 2025-02-25 15:34:34 -- STEP: 26714/36938 -- GLOBAL_STEP: 839350 | > loss_text_ce: 0.04021438583731651 (0.03382625430153025) | > loss_mel_ce: 2.4913017749786377 (nan) | > loss: 0.010045698843896389 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13680033607201697) | > loader_time: 0.0045 (0.004654615548517883)  --> TIME: 2025-02-25 15:34:43 -- STEP: 26764/36938 -- GLOBAL_STEP: 839400 | > loss_text_ce: 0.030758969485759735 (0.0338280218289539) | > loss_mel_ce: 2.58591890335083 (nan) | > loss: 0.01038364227861166 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1367849507882027) | > loader_time: 0.0046 (0.004654701462313636)  --> TIME: 2025-02-25 15:34:52 -- STEP: 26814/36938 -- GLOBAL_STEP: 839450 | > loss_text_ce: 0.03669716417789459 (0.03382550223388126) | > loss_mel_ce: 3.230736017227173 (nan) | > loss: 0.012966005131602287 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1367808375181115) | > loader_time: 0.0045 (0.0046548557075209536)  --> TIME: 2025-02-25 15:35:01 -- STEP: 26864/36938 -- GLOBAL_STEP: 839500 | > loss_text_ce: 0.03422398120164871 (0.03382722233712497) | > loss_mel_ce: 2.2298583984375 (nan) | > loss: 0.008984454907476902 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13678304996605575) | > loader_time: 0.0045 (0.004654862931826895)  --> TIME: 2025-02-25 15:35:11 -- STEP: 26914/36938 -- GLOBAL_STEP: 839550 | > loss_text_ce: 0.036200471222400665 (0.033828748432848825) | > loss_mel_ce: 3.0422019958496094 (nan) | > loss: 0.01221588347107172 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.1367848621414339) | > loader_time: 0.0046 (0.00465490164795397)  --> TIME: 2025-02-25 15:35:20 -- STEP: 26964/36938 -- GLOBAL_STEP: 839600 | > loss_text_ce: 0.04001595824956894 (0.03382642793094326) | > loss_mel_ce: 2.2120471000671387 (nan) | > loss: 0.008936758153140545 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.136773491751068) | > loader_time: 0.0048 (0.004655025467252247)  --> TIME: 2025-02-25 15:35:29 -- STEP: 27014/36938 -- GLOBAL_STEP: 839650 | > loss_text_ce: 0.02938428521156311 (0.033824850851645924) | > loss_mel_ce: 0.44532471895217896 (nan) | > loss: 0.0018837660318240523 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13677129932059665) | > loader_time: 0.0046 (0.004654995507417995)  --> TIME: 2025-02-25 15:35:38 -- STEP: 27064/36938 -- GLOBAL_STEP: 839700 | > loss_text_ce: 0.03211518004536629 (0.03382417800513634) | > loss_mel_ce: 2.8391506671905518 (nan) | > loss: 0.011393913067877293 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1367650645268313) | > loader_time: 0.0046 (0.00465494579300525)  --> TIME: 2025-02-25 15:35:47 -- STEP: 27114/36938 -- GLOBAL_STEP: 839750 | > loss_text_ce: 0.027490882202982903 (0.03382544366024164) | > loss_mel_ce: 1.9877716302871704 (nan) | > loss: 0.007997074164450169 (nan) | > current_lr: 5e-06 | > step_time: 0.1579 (0.13675597039895518) | > loader_time: 0.0046 (0.004654960812760478)  --> TIME: 2025-02-25 15:35:56 -- STEP: 27164/36938 -- GLOBAL_STEP: 839800 | > loss_text_ce: 0.03311939164996147 (0.033824056466763695) | > loss_mel_ce: 2.5287973880767822 (nan) | > loss: 0.01016633678227663 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13674829188230248) | > loader_time: 0.0046 (0.004654896617409675)  --> TIME: 2025-02-25 15:36:05 -- STEP: 27214/36938 -- GLOBAL_STEP: 839850 | > loss_text_ce: 0.02820475585758686 (0.033822376691415676) | > loss_mel_ce: 1.4354209899902344 (nan) | > loss: 0.005808039102703333 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13673867676096446) | > loader_time: 0.0045 (0.004654773618381734)  --> TIME: 2025-02-25 15:36:15 -- STEP: 27264/36938 -- GLOBAL_STEP: 839900 | > loss_text_ce: 0.033198680728673935 (0.03382494211147184) | > loss_mel_ce: 2.557791233062744 (nan) | > loss: 0.010281706228852272 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13673361994221264) | > loader_time: 0.0047 (0.004655153512464989)  --> TIME: 2025-02-25 15:36:24 -- STEP: 27314/36938 -- GLOBAL_STEP: 839950 | > loss_text_ce: 0.03278971463441849 (0.033824635843074276) | > loss_mel_ce: 2.511552333831787 (nan) | > loss: 0.010096595622599125 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13673310620444235) | > loader_time: 0.0045 (0.004655212576361196)  --> TIME: 2025-02-25 15:36:34 -- STEP: 27364/36938 -- GLOBAL_STEP: 840000 | > loss_text_ce: 0.03276559337973595 (0.03382674115134802) | > loss_mel_ce: 2.7931668758392334 (nan) | > loss: 0.011214018799364567 (nan) | > current_lr: 5e-06 | > step_time: 0.1566 (0.1367413351679045) | > loader_time: 0.0048 (0.0046553600254401716) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_840000.pth  --> TIME: 2025-02-25 15:36:49 -- STEP: 27414/36938 -- GLOBAL_STEP: 840050 | > loss_text_ce: 0.03242857754230499 (0.033828060020463076) | > loss_mel_ce: 2.2860679626464844 (nan) | > loss: 0.009200382977724075 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13674148863059646) | > loader_time: 0.005 (0.004655399146483202)  --> TIME: 2025-02-25 15:36:58 -- STEP: 27464/36938 -- GLOBAL_STEP: 840100 | > loss_text_ce: 0.021787848323583603 (0.033827695194396074) | > loss_mel_ce: 1.758501648902893 (nan) | > loss: 0.0070646414533257484 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13673171518219204) | > loader_time: 0.0047 (0.004655459515387964)  --> TIME: 2025-02-25 15:37:08 -- STEP: 27514/36938 -- GLOBAL_STEP: 840150 | > loss_text_ce: 0.040098097175359726 (0.033826570447895825) | > loss_mel_ce: 3.35115647315979 (nan) | > loss: 0.013457360677421093 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.13673824082715919) | > loader_time: 0.0048 (0.004655521085999367)  --> TIME: 2025-02-25 15:37:17 -- STEP: 27564/36938 -- GLOBAL_STEP: 840200 | > loss_text_ce: 0.04251363128423691 (0.03382391418952633) | > loss_mel_ce: 1.0449142456054688 (nan) | > loss: 0.004315190017223358 (nan) | > current_lr: 5e-06 | > step_time: 0.1515 (0.13673927614952242) | > loader_time: 0.0048 (0.004655702187454134)  --> TIME: 2025-02-25 15:37:26 -- STEP: 27614/36938 -- GLOBAL_STEP: 840250 | > loss_text_ce: 0.031134461984038353 (0.03382107764896012) | > loss_mel_ce: 2.968744993209839 (nan) | > loss: 0.011904283426702023 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.1367390606399386) | > loader_time: 0.0046 (0.004655665583585316)  --> TIME: 2025-02-25 15:37:35 -- STEP: 27664/36938 -- GLOBAL_STEP: 840300 | > loss_text_ce: 0.02867293357849121 (0.033822259366341) | > loss_mel_ce: 2.11138916015625 (nan) | > loss: 0.008492310531437397 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13673252605026825) | > loader_time: 0.0046 (0.004655604170470509)  --> TIME: 2025-02-25 15:37:45 -- STEP: 27714/36938 -- GLOBAL_STEP: 840350 | > loss_text_ce: 0.025633057579398155 (0.033824359966753995) | > loss_mel_ce: 3.2940592765808105 (nan) | > loss: 0.013173382729291916 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.1367302113232478) | > loader_time: 0.0046 (0.004655530719931882)  --> TIME: 2025-02-25 15:37:54 -- STEP: 27764/36938 -- GLOBAL_STEP: 840400 | > loss_text_ce: 0.032682131975889206 (0.033822505721275785) | > loss_mel_ce: 0.0133056640625 (nan) | > loss: 0.0001824912615120411 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.1367314719291648) | > loader_time: 0.0047 (0.004655560418725127)  --> TIME: 2025-02-25 15:38:03 -- STEP: 27814/36938 -- GLOBAL_STEP: 840450 | > loss_text_ce: 0.0391111895442009 (0.03382212754259051) | > loss_mel_ce: 2.256169557571411 (nan) | > loss: 0.009108257479965687 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13673377674797116) | > loader_time: 0.0046 (0.0046555928994692535)  --> TIME: 2025-02-25 15:38:13 -- STEP: 27864/36938 -- GLOBAL_STEP: 840500 | > loss_text_ce: 0.034370481967926025 (0.033821195908607174) | > loss_mel_ce: 2.5697221755981445 (nan) | > loss: 0.01033370103687048 (nan) | > current_lr: 5e-06 | > step_time: 0.1956 (0.1367428736494364) | > loader_time: 0.005 (0.004655753020887015)  --> TIME: 2025-02-25 15:38:22 -- STEP: 27914/36938 -- GLOBAL_STEP: 840550 | > loss_text_ce: 0.043838705867528915 (0.03382053816061294) | > loss_mel_ce: 3.553525447845459 (nan) | > loss: 0.014275255613029003 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13673761237391144) | > loader_time: 0.0045 (0.00465573559537131)  --> TIME: 2025-02-25 15:38:32 -- STEP: 27964/36938 -- GLOBAL_STEP: 840600 | > loss_text_ce: 0.03406061977148056 (0.03382029788975758) | > loss_mel_ce: 3.410618305206299 (nan) | > loss: 0.013669362291693687 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.13673974776742123) | > loader_time: 0.0047 (0.004655719792411446)  --> TIME: 2025-02-25 15:38:41 -- STEP: 28014/36938 -- GLOBAL_STEP: 840650 | > loss_text_ce: 0.03055436722934246 (0.03381998787432708) | > loss_mel_ce: 2.6295580863952637 (nan) | > loss: 0.010556002147495747 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.13674123584054007) | > loader_time: 0.0047 (0.004655781161260009)  --> TIME: 2025-02-25 15:38:51 -- STEP: 28064/36938 -- GLOBAL_STEP: 840700 | > loss_text_ce: 0.03542690351605415 (0.03382120099520828) | > loss_mel_ce: 2.7729177474975586 (nan) | > loss: 0.011144225485622883 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.1367483081778216) | > loader_time: 0.0046 (0.004655833807407511)  --> TIME: 2025-02-25 15:39:00 -- STEP: 28114/36938 -- GLOBAL_STEP: 840750 | > loss_text_ce: 0.031932197511196136 (0.033819186612312056) | > loss_mel_ce: 4.963541507720947 (nan) | > loss: 0.019823310896754265 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.1367412173765833) | > loader_time: 0.0045 (0.004655832670029557)  --> TIME: 2025-02-25 15:39:09 -- STEP: 28164/36938 -- GLOBAL_STEP: 840800 | > loss_text_ce: 0.035112667828798294 (0.03381785646165921) | > loss_mel_ce: 3.0172300338745117 (nan) | > loss: 0.012112471275031567 (nan) | > current_lr: 5e-06 | > step_time: 0.1532 (0.13673556590855773) | > loader_time: 0.0048 (0.0046559113566297375)  --> TIME: 2025-02-25 15:39:18 -- STEP: 28214/36938 -- GLOBAL_STEP: 840850 | > loss_text_ce: 0.03804273158311844 (0.03381842333599539) | > loss_mel_ce: 3.2400498390197754 (nan) | > loss: 0.013008304871618748 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1367349419712462) | > loader_time: 0.0046 (0.00465590794791047)  --> TIME: 2025-02-25 15:39:27 -- STEP: 28264/36938 -- GLOBAL_STEP: 840900 | > loss_text_ce: 0.027733931317925453 (0.033817123716271436) | > loss_mel_ce: 3.1590144634246826 (nan) | > loss: 0.012645828537642956 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.136730091820822) | > loader_time: 0.0046 (0.004655998580826257)  --> TIME: 2025-02-25 15:39:37 -- STEP: 28314/36938 -- GLOBAL_STEP: 840950 | > loss_text_ce: 0.028752364218235016 (0.0338141628082335) | > loss_mel_ce: 3.400416851043701 (nan) | > loss: 0.013607814908027649 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1367315531383261) | > loader_time: 0.0045 (0.004656011214353621)  --> TIME: 2025-02-25 15:39:46 -- STEP: 28364/36938 -- GLOBAL_STEP: 841000 | > loss_text_ce: 0.02773761749267578 (0.03381267038132373) | > loss_mel_ce: 2.7962286472320557 (nan) | > loss: 0.011206216178834438 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13672675459026226) | > loader_time: 0.0046 (0.004656036588372667)  --> TIME: 2025-02-25 15:39:55 -- STEP: 28414/36938 -- GLOBAL_STEP: 841050 | > loss_text_ce: 0.03369607403874397 (0.03381028362843095) | > loss_mel_ce: 2.4854087829589844 (nan) | > loss: 0.009996448643505573 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13671974129065023) | > loader_time: 0.0047 (0.004656023149160755)  --> TIME: 2025-02-25 15:40:04 -- STEP: 28464/36938 -- GLOBAL_STEP: 841100 | > loss_text_ce: 0.026806239038705826 (0.033808384147790235) | > loss_mel_ce: 2.4971935749053955 (nan) | > loss: 0.010015873238444328 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13671092505599824) | > loader_time: 0.0046 (0.004655967105835451)  --> TIME: 2025-02-25 15:40:13 -- STEP: 28514/36938 -- GLOBAL_STEP: 841150 | > loss_text_ce: 0.03585163876414299 (0.03381113832291439) | > loss_mel_ce: 2.9838531017303467 (nan) | > loss: 0.01198295596987009 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13670859986036613) | > loader_time: 0.0046 (0.004655958651793969)  --> TIME: 2025-02-25 15:40:23 -- STEP: 28564/36938 -- GLOBAL_STEP: 841200 | > loss_text_ce: 0.030405232682824135 (0.03381081564915877) | > loss_mel_ce: 2.713412284851074 (nan) | > loss: 0.010888165794312954 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13670513322249603) | > loader_time: 0.0049 (0.004656013863508375)  --> TIME: 2025-02-25 15:40:32 -- STEP: 28614/36938 -- GLOBAL_STEP: 841250 | > loss_text_ce: 0.035509273409843445 (0.03381373878647078) | > loss_mel_ce: 0.568206787109375 (nan) | > loss: 0.0023956988006830215 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1367000506476993) | > loader_time: 0.0045 (0.004656114776220972)  --> TIME: 2025-02-25 15:40:41 -- STEP: 28664/36938 -- GLOBAL_STEP: 841300 | > loss_text_ce: 0.03349471092224121 (0.03381408500771595) | > loss_mel_ce: 3.3226511478424072 (nan) | > loss: 0.01331803947687149 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.136698244887626) | > loader_time: 0.0046 (0.004656213939505977)  --> TIME: 2025-02-25 15:40:50 -- STEP: 28714/36938 -- GLOBAL_STEP: 841350 | > loss_text_ce: 0.032929740846157074 (0.03381525942832762) | > loss_mel_ce: 2.1400344371795654 (nan) | > loss: 0.008622873574495316 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13669306349105081) | > loader_time: 0.0046 (0.004656279635909301)  --> TIME: 2025-02-25 15:41:00 -- STEP: 28764/36938 -- GLOBAL_STEP: 841400 | > loss_text_ce: 0.035205237567424774 (0.03381491732240862) | > loss_mel_ce: 2.62876296043396 (nan) | > loss: 0.010571302846074104 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.1366926139227596) | > loader_time: 0.0049 (0.004656298595548349)  --> TIME: 2025-02-25 15:41:09 -- STEP: 28814/36938 -- GLOBAL_STEP: 841450 | > loss_text_ce: 0.0344383604824543 (0.033813414249314555) | > loss_mel_ce: 3.1903076171875 (nan) | > loss: 0.012796611525118351 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.13669290779746934) | > loader_time: 0.0047 (0.004656365282325211)  --> TIME: 2025-02-25 15:41:18 -- STEP: 28864/36938 -- GLOBAL_STEP: 841500 | > loss_text_ce: 0.037353515625 (0.03381367857183259) | > loss_mel_ce: 1.8592778444290161 (nan) | > loss: 0.0075263152830302715 (nan) | > current_lr: 5e-06 | > step_time: 0.1065 (0.13669389009707064) | > loader_time: 0.0046 (0.004656430821196725)  --> TIME: 2025-02-25 15:41:27 -- STEP: 28914/36938 -- GLOBAL_STEP: 841550 | > loss_text_ce: 0.024578014388680458 (0.03381498094160236) | > loss_mel_ce: 2.5988094806671143 (nan) | > loss: 0.010410268791019917 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.1366921270290276) | > loader_time: 0.0045 (0.004656490955048012)  --> TIME: 2025-02-25 15:41:37 -- STEP: 28964/36938 -- GLOBAL_STEP: 841600 | > loss_text_ce: 0.03495466336607933 (0.033814962063261046) | > loss_mel_ce: 2.587757110595703 (nan) | > loss: 0.010407586582005024 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13669529560109198) | > loader_time: 0.0047 (0.0046565668998774035)  --> TIME: 2025-02-25 15:41:46 -- STEP: 29014/36938 -- GLOBAL_STEP: 841650 | > loss_text_ce: 0.042532578110694885 (0.033812540620579874) | > loss_mel_ce: 2.590566873550415 (nan) | > loss: 0.010448808781802654 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13669878154419962) | > loader_time: 0.0045 (0.0046565436623349)  --> TIME: 2025-02-25 15:41:55 -- STEP: 29064/36938 -- GLOBAL_STEP: 841700 | > loss_text_ce: 0.03045520931482315 (0.03381468187672207) | > loss_mel_ce: 0.02001953125 (nan) | > loss: 0.00020029660663567483 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13669337627975062) | > loader_time: 0.0046 (0.0046566001252631856)  --> TIME: 2025-02-25 15:42:05 -- STEP: 29114/36938 -- GLOBAL_STEP: 841750 | > loss_text_ce: 0.02657269686460495 (0.033814010616977475) | > loss_mel_ce: 2.585766553878784 (nan) | > loss: 0.010366425849497318 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13669150682434858) | > loader_time: 0.0046 (0.004656611173831141)  --> TIME: 2025-02-25 15:42:14 -- STEP: 29164/36938 -- GLOBAL_STEP: 841800 | > loss_text_ce: 0.028899526223540306 (0.03381305030544431) | > loss_mel_ce: 1.2366071939468384 (nan) | > loss: 0.005021852441132069 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.1366849967750274) | > loader_time: 0.0046 (0.0046566465463090175)  --> TIME: 2025-02-25 15:42:23 -- STEP: 29214/36938 -- GLOBAL_STEP: 841850 | > loss_text_ce: 0.03195314109325409 (0.03381425279987791) | > loss_mel_ce: 2.183969259262085 (nan) | > loss: 0.008793342858552933 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.13668024307356028) | > loader_time: 0.0046 (0.00465671545411148)  --> TIME: 2025-02-25 15:42:32 -- STEP: 29264/36938 -- GLOBAL_STEP: 841900 | > loss_text_ce: 0.03652508929371834 (0.03381544791225442) | > loss_mel_ce: 2.6276447772979736 (nan) | > loss: 0.010572102852165699 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13667639401553516) | > loader_time: 0.0047 (0.0046567484011344985)  --> TIME: 2025-02-25 15:42:41 -- STEP: 29314/36938 -- GLOBAL_STEP: 841950 | > loss_text_ce: 0.02776913158595562 (0.033814222110080984) | > loss_mel_ce: 0.0267333984375 (nan) | > loss: 0.00021627990645356476 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.13666696693071487) | > loader_time: 0.0045 (0.004656855102091921)  --> TIME: 2025-02-25 15:42:51 -- STEP: 29364/36938 -- GLOBAL_STEP: 842000 | > loss_text_ce: 0.024332556873559952 (0.03381392371011991) | > loss_mel_ce: 2.6204631328582764 (nan) | > loss: 0.010495221242308617 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13666715490089) | > loader_time: 0.0048 (0.0046569775729587886)  --> TIME: 2025-02-25 15:43:00 -- STEP: 29414/36938 -- GLOBAL_STEP: 842050 | > loss_text_ce: 0.037774622440338135 (0.033815948972851784) | > loss_mel_ce: 2.7252604961395264 (nan) | > loss: 0.010964425280690193 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13665988158019063) | > loader_time: 0.0052 (0.004656982031184319)  --> TIME: 2025-02-25 15:43:09 -- STEP: 29464/36938 -- GLOBAL_STEP: 842100 | > loss_text_ce: 0.037721846252679825 (0.03381443305019225) | > loss_mel_ce: 2.8261947631835938 (nan) | > loss: 0.011364749632775784 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13665251969126016) | > loader_time: 0.0047 (0.004657005700539409)  --> TIME: 2025-02-25 15:43:18 -- STEP: 29514/36938 -- GLOBAL_STEP: 842150 | > loss_text_ce: 0.03593222051858902 (0.03381571428609365) | > loss_mel_ce: 2.785306692123413 (nan) | > loss: 0.011195393279194832 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1366567054948108) | > loader_time: 0.0046 (0.004656993479249496)  --> TIME: 2025-02-25 15:43:27 -- STEP: 29564/36938 -- GLOBAL_STEP: 842200 | > loss_text_ce: 0.02038414217531681 (0.033814689359965525) | > loss_mel_ce: 2.31668758392334 (nan) | > loss: 0.009274094365537167 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13665008002988252) | > loader_time: 0.0045 (0.004656967355794981)  --> TIME: 2025-02-25 15:43:37 -- STEP: 29614/36938 -- GLOBAL_STEP: 842250 | > loss_text_ce: 0.02952297404408455 (0.033813720087912245) | > loss_mel_ce: 3.0517916679382324 (nan) | > loss: 0.012227439321577549 (nan) | > current_lr: 5e-06 | > step_time: 0.1797 (0.1366596740802659) | > loader_time: 0.0048 (0.004656923447613643)  --> TIME: 2025-02-25 15:43:47 -- STEP: 29664/36938 -- GLOBAL_STEP: 842300 | > loss_text_ce: 0.02896801196038723 (0.03381273623137991) | > loss_mel_ce: 0.0844268798828125 (nan) | > loss: 0.0004499797651078552 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13666660300061248) | > loader_time: 0.0046 (0.004656935217182478)  --> TIME: 2025-02-25 15:43:56 -- STEP: 29714/36938 -- GLOBAL_STEP: 842350 | > loss_text_ce: 0.03620576858520508 (0.03381502426202434) | > loss_mel_ce: 2.2034265995025635 (nan) | > loss: 0.008887430652976036 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.1366729050535715) | > loader_time: 0.0047 (0.004657047645572436)  --> TIME: 2025-02-25 15:44:05 -- STEP: 29764/36938 -- GLOBAL_STEP: 842400 | > loss_text_ce: 0.026811378076672554 (0.03381379636335279) | > loss_mel_ce: 2.5668249130249023 (nan) | > loss: 0.010292207822203636 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13666755738013017) | > loader_time: 0.0046 (0.00465718966276314)  --> TIME: 2025-02-25 15:44:15 -- STEP: 29814/36938 -- GLOBAL_STEP: 842450 | > loss_text_ce: 0.03595230355858803 (0.033813474445892236) | > loss_mel_ce: 3.2100770473480225 (nan) | > loss: 0.01288106944411993 (nan) | > current_lr: 5e-06 | > step_time: 0.1899 (0.13667041349996456) | > loader_time: 0.005 (0.004657215368998007)  --> TIME: 2025-02-25 15:44:25 -- STEP: 29864/36938 -- GLOBAL_STEP: 842500 | > loss_text_ce: 0.041664063930511475 (0.03381344705369553) | > loss_mel_ce: 2.5975453853607178 (nan) | > loss: 0.010473053902387619 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.136684158181749) | > loader_time: 0.0046 (0.00465734241125636)  --> TIME: 2025-02-25 15:44:34 -- STEP: 29914/36938 -- GLOBAL_STEP: 842550 | > loss_text_ce: 0.028121644631028175 (0.033811846308516635) | > loss_mel_ce: 2.5234320163726807 (nan) | > loss: 0.010125214233994484 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.13667577849023038) | > loader_time: 0.0048 (0.0046573525532893735)  --> TIME: 2025-02-25 15:44:43 -- STEP: 29964/36938 -- GLOBAL_STEP: 842600 | > loss_text_ce: 0.035101838409900665 (0.03381030339791961) | > loss_mel_ce: 1.7028175592422485 (nan) | > loss: 0.006896506063640118 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13667698955045832) | > loader_time: 0.0046 (0.004657480502189067)  --> TIME: 2025-02-25 15:44:53 -- STEP: 30014/36938 -- GLOBAL_STEP: 842650 | > loss_text_ce: 0.03986264392733574 (0.0338105510815037) | > loss_mel_ce: 3.1711039543151855 (nan) | > loss: 0.012741931714117527 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13668476545477948) | > loader_time: 0.0047 (0.00465745480109668)  --> TIME: 2025-02-25 15:45:02 -- STEP: 30064/36938 -- GLOBAL_STEP: 842700 | > loss_text_ce: 0.02636473998427391 (0.03381081096971257) | > loss_mel_ce: 3.2694437503814697 (nan) | > loss: 0.013078605756163597 (nan) | > current_lr: 5e-06 | > step_time: 0.1811 (0.13669355981301815) | > loader_time: 0.0047 (0.0046574778620901034)  --> TIME: 2025-02-25 15:45:12 -- STEP: 30114/36938 -- GLOBAL_STEP: 842750 | > loss_text_ce: 0.03382980823516846 (0.03380801372315194) | > loss_mel_ce: 3.4623794555664062 (nan) | > loss: 0.013873846270143986 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.13669284420706426) | > loader_time: 0.0047 (0.004657457634423632)  --> TIME: 2025-02-25 15:45:21 -- STEP: 30164/36938 -- GLOBAL_STEP: 842800 | > loss_text_ce: 0.04174055531620979 (0.033809094538312524) | > loss_mel_ce: 3.123173713684082 (nan) | > loss: 0.012559184804558754 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.13669431050692346) | > loader_time: 0.0047 (0.004657598479864936)  --> TIME: 2025-02-25 15:45:31 -- STEP: 30214/36938 -- GLOBAL_STEP: 842850 | > loss_text_ce: 0.0366300567984581 (0.033805793529596076) | > loss_mel_ce: 2.9737331867218018 (nan) | > loss: 0.01194588653743267 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.13670168639002772) | > loader_time: 0.005 (0.004657652894626923)  --> TIME: 2025-02-25 15:45:40 -- STEP: 30264/36938 -- GLOBAL_STEP: 842900 | > loss_text_ce: 0.04601262882351875 (0.03380591716210435) | > loss_mel_ce: 2.862301826477051 (nan) | > loss: 0.011540930718183517 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13670426399941696) | > loader_time: 0.0046 (0.0046576398518108734)  --> TIME: 2025-02-25 15:45:49 -- STEP: 30314/36938 -- GLOBAL_STEP: 842950 | > loss_text_ce: 0.03096708469092846 (0.03380475329860618) | > loss_mel_ce: 2.4539148807525635 (nan) | > loss: 0.009860643185675144 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.1366997570666544) | > loader_time: 0.0046 (0.004657596241572898)  --> TIME: 2025-02-25 15:45:59 -- STEP: 30364/36938 -- GLOBAL_STEP: 843000 | > loss_text_ce: 0.028257323428988457 (0.033803589397960986) | > loss_mel_ce: 2.332289934158325 (nan) | > loss: 0.009367251768708229 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.13670141956674667) | > loader_time: 0.0047 (0.004657577202577952)  --> TIME: 2025-02-25 15:46:08 -- STEP: 30414/36938 -- GLOBAL_STEP: 843050 | > loss_text_ce: 0.04186767339706421 (0.03380643631171629) | > loss_mel_ce: 2.0900180339813232 (nan) | > loss: 0.008459864184260368 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13669817873482693) | > loader_time: 0.0045 (0.004657525396005255)  --> TIME: 2025-02-25 15:46:17 -- STEP: 30464/36938 -- GLOBAL_STEP: 843100 | > loss_text_ce: 0.03905283287167549 (0.033808001970269515) | > loss_mel_ce: 3.2772176265716553 (nan) | > loss: 0.013159804046154022 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.13670595657831813) | > loader_time: 0.0048 (0.004657562094263162)  --> TIME: 2025-02-25 15:46:27 -- STEP: 30514/36938 -- GLOBAL_STEP: 843150 | > loss_text_ce: 0.03516305983066559 (0.03380848620151159) | > loss_mel_ce: 2.981032609939575 (nan) | > loss: 0.011969031766057014 (nan) | > current_lr: 5e-06 | > step_time: 0.1629 (0.1367167022130269) | > loader_time: 0.0047 (0.004657604555756447)  --> TIME: 2025-02-25 15:46:37 -- STEP: 30564/36938 -- GLOBAL_STEP: 843200 | > loss_text_ce: 0.034832533448934555 (0.03380665956043154) | > loss_mel_ce: 3.3705623149871826 (nan) | > loss: 0.013513471931219101 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.1367228000970002) | > loader_time: 0.0049 (0.004657628593637025)  --> TIME: 2025-02-25 15:46:46 -- STEP: 30614/36938 -- GLOBAL_STEP: 843250 | > loss_text_ce: 0.02079704962670803 (0.033805717341650866) | > loss_mel_ce: 2.9769537448883057 (nan) | > loss: 0.011895837262272835 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.136719472007842) | > loader_time: 0.0049 (0.004657576434124901)  --> TIME: 2025-02-25 15:46:55 -- STEP: 30664/36938 -- GLOBAL_STEP: 843300 | > loss_text_ce: 0.03138694539666176 (0.03380461579861536) | > loss_mel_ce: 2.42519474029541 (nan) | > loss: 0.009748340584337711 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.1367202144113256) | > loader_time: 0.0046 (0.004657578039131984)  --> TIME: 2025-02-25 15:47:05 -- STEP: 30714/36938 -- GLOBAL_STEP: 843350 | > loss_text_ce: 0.02949540503323078 (0.03380339078123932) | > loss_mel_ce: 2.6115565299987793 (nan) | > loss: 0.010480365715920925 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13671733578985512) | > loader_time: 0.0046 (0.00465749979345191)  --> TIME: 2025-02-25 15:47:14 -- STEP: 30764/36938 -- GLOBAL_STEP: 843400 | > loss_text_ce: 0.04303234443068504 (0.0338042147342323) | > loss_mel_ce: 2.7947590351104736 (nan) | > loss: 0.011261077597737312 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13670980379309047) | > loader_time: 0.0046 (0.004657428033050265)  --> TIME: 2025-02-25 15:47:23 -- STEP: 30814/36938 -- GLOBAL_STEP: 843450 | > loss_text_ce: 0.03512291982769966 (0.03380262254627759) | > loss_mel_ce: 0.08050537109375 (nan) | > loss: 0.0004588424344547093 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13671082287239616) | > loader_time: 0.0045 (0.004657408964736842)  --> TIME: 2025-02-25 15:47:32 -- STEP: 30864/36938 -- GLOBAL_STEP: 843500 | > loss_text_ce: 0.03784727305173874 (0.03380443443400127) | > loss_mel_ce: 2.8343424797058105 (nan) | > loss: 0.01139757875353098 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.1367124030941081) | > loader_time: 0.0046 (0.00465735915165728)  --> TIME: 2025-02-25 15:47:41 -- STEP: 30914/36938 -- GLOBAL_STEP: 843550 | > loss_text_ce: 0.027089063078165054 (0.03380266961392138) | > loss_mel_ce: 3.0467355251312256 (nan) | > loss: 0.01219771709293127 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.13670956873296886) | > loader_time: 0.0047 (0.004657296689552942)  --> TIME: 2025-02-25 15:47:51 -- STEP: 30964/36938 -- GLOBAL_STEP: 843600 | > loss_text_ce: 0.046074651181697845 (0.03380368903188865) | > loss_mel_ce: 2.632760524749756 (nan) | > loss: 0.010630298405885696 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13671084419576304) | > loader_time: 0.0046 (0.004657275846740278)  --> TIME: 2025-02-25 15:48:00 -- STEP: 31014/36938 -- GLOBAL_STEP: 843650 | > loss_text_ce: 0.03432869538664818 (0.03380411128220645) | > loss_mel_ce: 2.424872636795044 (nan) | > loss: 0.00975873600691557 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13671212855318407) | > loader_time: 0.0046 (0.004657285782495513)  --> TIME: 2025-02-25 15:48:09 -- STEP: 31064/36938 -- GLOBAL_STEP: 843700 | > loss_text_ce: 0.03111773170530796 (0.03380396025224896) | > loss_mel_ce: 3.23319673538208 (nan) | > loss: 0.012953629717230797 (nan) | > current_lr: 5e-06 | > step_time: 0.151 (0.13671241469245785) | > loader_time: 0.0047 (0.004657259344778317)  --> TIME: 2025-02-25 15:48:19 -- STEP: 31114/36938 -- GLOBAL_STEP: 843750 | > loss_text_ce: 0.024654271081089973 (0.0338010186104514) | > loss_mel_ce: 2.072662115097046 (nan) | > loss: 0.008322685025632381 (nan) | > current_lr: 5e-06 | > step_time: 0.1121 (0.13671602258463486) | > loader_time: 0.0045 (0.00465727155861606)  --> TIME: 2025-02-25 15:48:28 -- STEP: 31164/36938 -- GLOBAL_STEP: 843800 | > loss_text_ce: 0.03738119825720787 (0.033801352381926525) | > loss_mel_ce: 2.3962368965148926 (nan) | > loss: 0.00965721532702446 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.1367138106485438) | > loader_time: 0.0046 (0.004657221221446434)  --> TIME: 2025-02-25 15:48:37 -- STEP: 31214/36938 -- GLOBAL_STEP: 843850 | > loss_text_ce: 0.043820254504680634 (0.03380299909844166) | > loss_mel_ce: 3.0006837844848633 (nan) | > loss: 0.01208136510103941 (nan) | > current_lr: 5e-06 | > step_time: 0.1266 (0.13671362566567635) | > loader_time: 0.0046 (0.004657232219835114)  --> TIME: 2025-02-25 15:48:47 -- STEP: 31264/36938 -- GLOBAL_STEP: 843900 | > loss_text_ce: 0.03805273026227951 (0.03380300567812866) | > loss_mel_ce: 3.1333301067352295 (nan) | > loss: 0.012584853917360306 (nan) | > current_lr: 5e-06 | > step_time: 0.1454 (0.13670865475307914) | > loader_time: 0.0046 (0.004657159846356111)  --> TIME: 2025-02-25 15:48:56 -- STEP: 31314/36938 -- GLOBAL_STEP: 843950 | > loss_text_ce: 0.03663330525159836 (0.03380079996827046) | > loss_mel_ce: 2.7080843448638916 (nan) | > loss: 0.010891737416386604 (nan) | > current_lr: 5e-06 | > step_time: 0.1651 (0.13671216829500646) | > loader_time: 0.0048 (0.004657277318110609)  --> TIME: 2025-02-25 15:49:05 -- STEP: 31364/36938 -- GLOBAL_STEP: 844000 | > loss_text_ce: 0.03123351000249386 (0.03380033510840846) | > loss_mel_ce: 2.9692325592041016 (nan) | > loss: 0.011906612664461136 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.13671069178041906) | > loader_time: 0.0047 (0.004657363324517447)  --> TIME: 2025-02-25 15:49:15 -- STEP: 31414/36938 -- GLOBAL_STEP: 844050 | > loss_text_ce: 0.04515537992119789 (0.03380044056660619) | > loss_mel_ce: 1.3424890041351318 (nan) | > loss: 0.005506525747478008 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13670860753369427) | > loader_time: 0.0047 (0.0046574983665411355)  --> TIME: 2025-02-25 15:49:24 -- STEP: 31464/36938 -- GLOBAL_STEP: 844100 | > loss_text_ce: 0.031606510281562805 (0.03379841687489536) | > loss_mel_ce: 2.621372938156128 (nan) | > loss: 0.010527696460485458 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.13670324113058088) | > loader_time: 0.0049 (0.004657583149704586)  --> TIME: 2025-02-25 15:49:33 -- STEP: 31514/36938 -- GLOBAL_STEP: 844150 | > loss_text_ce: 0.026462916284799576 (0.033798139351845106) | > loss_mel_ce: 0.0081787109375 (nan) | > loss: 0.00013746677723247558 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13670552394229346) | > loader_time: 0.0046 (0.00465773709982249)  --> TIME: 2025-02-25 15:49:42 -- STEP: 31564/36938 -- GLOBAL_STEP: 844200 | > loss_text_ce: 0.03833712264895439 (0.03379749675562157) | > loss_mel_ce: 1.2711724042892456 (nan) | > loss: 0.005196466576308012 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.1367038032761131) | > loader_time: 0.0047 (0.004657800841763536)  --> TIME: 2025-02-25 15:49:51 -- STEP: 31614/36938 -- GLOBAL_STEP: 844250 | > loss_text_ce: 0.04128814488649368 (0.03379764185485823) | > loss_mel_ce: 2.706749200820923 (nan) | > loss: 0.01090491097420454 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13669779886366593) | > loader_time: 0.0046 (0.0046578999329482005)  --> TIME: 2025-02-25 15:50:01 -- STEP: 31664/36938 -- GLOBAL_STEP: 844300 | > loss_text_ce: 0.037273865193128586 (0.03379656897099913) | > loss_mel_ce: 1.2588151693344116 (nan) | > loss: 0.005143210757523775 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.1366985487465074) | > loader_time: 0.0047 (0.004658032579515961)  --> TIME: 2025-02-25 15:50:10 -- STEP: 31714/36938 -- GLOBAL_STEP: 844350 | > loss_text_ce: 0.03726806864142418 (0.03379597038699269) | > loss_mel_ce: 2.8029913902282715 (nan) | > loss: 0.011270872317254543 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.13670022900215717) | > loader_time: 0.0046 (0.004658093065739484)  --> TIME: 2025-02-25 15:50:19 -- STEP: 31764/36938 -- GLOBAL_STEP: 844400 | > loss_text_ce: 0.036148443818092346 (0.0337945440919676) | > loss_mel_ce: 3.2534632682800293 (nan) | > loss: 0.013054016046226025 (nan) | > current_lr: 5e-06 | > step_time: 0.1818 (0.13669074182170587) | > loader_time: 0.0048 (0.00465812823916844)  --> TIME: 2025-02-25 15:50:28 -- STEP: 31814/36938 -- GLOBAL_STEP: 844450 | > loss_text_ce: 0.03553837910294533 (0.03379509588719354) | > loss_mel_ce: 2.558161973953247 (nan) | > loss: 0.010292462073266506 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1366893732473576) | > loader_time: 0.0047 (0.004658301291642915)  --> TIME: 2025-02-25 15:50:37 -- STEP: 31864/36938 -- GLOBAL_STEP: 844500 | > loss_text_ce: 0.03795819357037544 (0.03379521469445102) | > loss_mel_ce: 2.278378486633301 (nan) | > loss: 0.009191812947392464 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13668759997364588) | > loader_time: 0.0045 (0.0046583219161741105)  --> TIME: 2025-02-25 15:50:47 -- STEP: 31914/36938 -- GLOBAL_STEP: 844550 | > loss_text_ce: 0.03867090865969658 (0.03379596206154447) | > loss_mel_ce: 0.6239123344421387 (nan) | > loss: 0.0026292987167835236 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13669838042174728) | > loader_time: 0.0046 (0.004658393388614169)  --> TIME: 2025-02-25 15:50:57 -- STEP: 31964/36938 -- GLOBAL_STEP: 844600 | > loss_text_ce: 0.05663055181503296 (0.033795680354839486) | > loss_mel_ce: 2.5102579593658447 (nan) | > loss: 0.010186065919697285 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13670205542597053) | > loader_time: 0.0046 (0.004658419876160685)  --> TIME: 2025-02-25 15:51:06 -- STEP: 32014/36938 -- GLOBAL_STEP: 844650 | > loss_text_ce: 0.03812273591756821 (0.033796057724239756) | > loss_mel_ce: 2.711843490600586 (nan) | > loss: 0.01091256458312273 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.1367008726728062) | > loader_time: 0.0046 (0.004658430507540987)  --> TIME: 2025-02-25 15:51:15 -- STEP: 32064/36938 -- GLOBAL_STEP: 844700 | > loss_text_ce: 0.03229954093694687 (0.03379633836541172) | > loss_mel_ce: 2.446307420730591 (nan) | > loss: 0.00983574241399765 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13669078823870606) | > loader_time: 0.0047 (0.004658428680694502)  --> TIME: 2025-02-25 15:51:24 -- STEP: 32114/36938 -- GLOBAL_STEP: 844750 | > loss_text_ce: 0.0329703688621521 (0.03379576216917778) | > loss_mel_ce: 2.143483877182007 (nan) | > loss: 0.008636724203824997 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.136688470959241) | > loader_time: 0.0045 (0.00465852146525233)  --> TIME: 2025-02-25 15:51:34 -- STEP: 32164/36938 -- GLOBAL_STEP: 844800 | > loss_text_ce: 0.03078465349972248 (0.03379533587918024) | > loss_mel_ce: 2.931947708129883 (nan) | > loss: 0.01175687462091446 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13668734706941127) | > loader_time: 0.0046 (0.004658543956826798)  --> TIME: 2025-02-25 15:51:43 -- STEP: 32214/36938 -- GLOBAL_STEP: 844850 | > loss_text_ce: 0.03395490720868111 (0.03379281537216822) | > loss_mel_ce: 1.1179238557815552 (nan) | > loss: 0.004570947494357824 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.1366829558399327) | > loader_time: 0.0046 (0.004658516991136232)  --> TIME: 2025-02-25 15:51:52 -- STEP: 32264/36938 -- GLOBAL_STEP: 844900 | > loss_text_ce: 0.025649067014455795 (0.033792951665130855) | > loss_mel_ce: 2.386474609375 (nan) | > loss: 0.009571920149028301 (nan) | > current_lr: 5e-06 | > step_time: 0.1123 (0.1366728736734741) | > loader_time: 0.0046 (0.0046585582782339525)  --> TIME: 2025-02-25 15:52:01 -- STEP: 32314/36938 -- GLOBAL_STEP: 844950 | > loss_text_ce: 0.02546432986855507 (0.03379255357634973) | > loss_mel_ce: 2.444895029067993 (nan) | > loss: 0.009803013876080513 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1366710733082099) | > loader_time: 0.0047 (0.004658651579176716)  --> TIME: 2025-02-25 15:52:10 -- STEP: 32364/36938 -- GLOBAL_STEP: 845000 | > loss_text_ce: 0.03291289135813713 (0.03379265702907556) | > loss_mel_ce: 2.4205689430236816 (nan) | > loss: 0.00973603967577219 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.13666891908781256) | > loader_time: 0.0047 (0.004658807666233063)  --> TIME: 2025-02-25 15:52:20 -- STEP: 32414/36938 -- GLOBAL_STEP: 845050 | > loss_text_ce: 0.048391811549663544 (0.03379148382576564) | > loss_mel_ce: 2.7402446269989014 (nan) | > loss: 0.011066018603742123 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13666779886692898) | > loader_time: 0.0046 (0.004658948722726841)  --> TIME: 2025-02-25 15:52:29 -- STEP: 32464/36938 -- GLOBAL_STEP: 845100 | > loss_text_ce: 0.03539752960205078 (0.03379117297270299) | > loss_mel_ce: 2.507967233657837 (nan) | > loss: 0.010092717595398426 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13666999851206205) | > loader_time: 0.0047 (0.004659035255487116)  --> TIME: 2025-02-25 15:52:38 -- STEP: 32514/36938 -- GLOBAL_STEP: 845150 | > loss_text_ce: 0.03163844347000122 (0.03379036348836504) | > loss_mel_ce: 2.9766719341278076 (nan) | > loss: 0.011937740258872509 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13666921108640373) | > loader_time: 0.0045 (0.00465908326690264)  --> TIME: 2025-02-25 15:52:48 -- STEP: 32564/36938 -- GLOBAL_STEP: 845200 | > loss_text_ce: 0.03235141560435295 (0.03379028643654323) | > loss_mel_ce: 2.017155170440674 (nan) | > loss: 0.008132963441312313 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13667093397665858) | > loader_time: 0.0047 (0.004659170498793483)  --> TIME: 2025-02-25 15:52:57 -- STEP: 32614/36938 -- GLOBAL_STEP: 845250 | > loss_text_ce: 0.03162560239434242 (0.033789982237600595) | > loss_mel_ce: 3.1831979751586914 (nan) | > loss: 0.012757237069308758 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.136672571492443) | > loader_time: 0.0046 (0.0046591545413066275)  --> TIME: 2025-02-25 15:53:06 -- STEP: 32664/36938 -- GLOBAL_STEP: 845300 | > loss_text_ce: 0.032274436205625534 (0.03378892545306646) | > loss_mel_ce: 3.380622148513794 (nan) | > loss: 0.013543241657316685 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13667353391355452) | > loader_time: 0.0046 (0.004659132340828513)  --> TIME: 2025-02-25 15:53:16 -- STEP: 32714/36938 -- GLOBAL_STEP: 845350 | > loss_text_ce: 0.04362468793988228 (0.03379004084100178) | > loss_mel_ce: 2.8113772869110107 (nan) | > loss: 0.011329373344779015 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13668298109124258) | > loader_time: 0.0046 (0.004659169984124733)  --> TIME: 2025-02-25 15:53:26 -- STEP: 32764/36938 -- GLOBAL_STEP: 845400 | > loss_text_ce: 0.03223574161529541 (0.033790295196973956) | > loss_mel_ce: 3.0778849124908447 (nan) | > loss: 0.012341749854385853 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13668796376800196) | > loader_time: 0.0046 (0.004659212075111086)  --> TIME: 2025-02-25 15:53:35 -- STEP: 32814/36938 -- GLOBAL_STEP: 845450 | > loss_text_ce: 0.027251044288277626 (0.03378723765971326) | > loss_mel_ce: 2.4019477367401123 (nan) | > loss: 0.00963967852294445 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13668417363873991) | > loader_time: 0.0045 (0.004659196834514215)  --> TIME: 2025-02-25 15:53:44 -- STEP: 32864/36938 -- GLOBAL_STEP: 845500 | > loss_text_ce: 0.03237423673272133 (0.033785219851200546) | > loss_mel_ce: 3.33449387550354 (nan) | > loss: 0.013360587880015373 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.13668556075394522) | > loader_time: 0.0047 (0.004659202657168419)  --> TIME: 2025-02-25 15:53:53 -- STEP: 32914/36938 -- GLOBAL_STEP: 845550 | > loss_text_ce: 0.03020547330379486 (0.03378300136699421) | > loss_mel_ce: 2.64837646484375 (nan) | > loss: 0.010629294440150261 (nan) | > current_lr: 5e-06 | > step_time: 0.1471 (0.13668348281143247) | > loader_time: 0.0046 (0.004659185847354087)  --> TIME: 2025-02-25 15:54:03 -- STEP: 32964/36938 -- GLOBAL_STEP: 845600 | > loss_text_ce: 0.037565916776657104 (0.03378288523540314) | > loss_mel_ce: 2.551807403564453 (nan) | > loss: 0.01027529127895832 (nan) | > current_lr: 5e-06 | > step_time: 0.1075 (0.13669123292012772) | > loader_time: 0.0045 (0.004659283495313863)  --> TIME: 2025-02-25 15:54:13 -- STEP: 33014/36938 -- GLOBAL_STEP: 845650 | > loss_text_ce: 0.027202734723687172 (0.03378122201592381) | > loss_mel_ce: 2.891512870788574 (nan) | > loss: 0.0115822060033679 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.1366918392774165) | > loader_time: 0.005 (0.004659449107398364)  --> TIME: 2025-02-25 15:54:22 -- STEP: 33064/36938 -- GLOBAL_STEP: 845700 | > loss_text_ce: 0.03413068875670433 (0.033780701291591546) | > loss_mel_ce: 3.6750290393829346 (nan) | > loss: 0.014718888327479362 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13669203491122175) | > loader_time: 0.0046 (0.004659436803564384)  --> TIME: 2025-02-25 15:54:31 -- STEP: 33114/36938 -- GLOBAL_STEP: 845750 | > loss_text_ce: 0.031744834035634995 (0.03377862292008193) | > loss_mel_ce: 2.3008499145507812 (nan) | > loss: 0.009256328456103802 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13669456831245613) | > loader_time: 0.0047 (0.00465950542813288)  --> TIME: 2025-02-25 15:54:41 -- STEP: 33164/36938 -- GLOBAL_STEP: 845800 | > loss_text_ce: 0.033520761877298355 (0.03377920982625403) | > loss_mel_ce: 2.6752281188964844 (nan) | > loss: 0.01074900384992361 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.136698495484768) | > loader_time: 0.0047 (0.0046595908767020095)  --> TIME: 2025-02-25 15:54:50 -- STEP: 33214/36938 -- GLOBAL_STEP: 845850 | > loss_text_ce: 0.024003906175494194 (0.033779199709270985) | > loss_mel_ce: 1.9038461446762085 (nan) | > loss: 0.007650198880583048 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13670552752410986) | > loader_time: 0.0044 (0.004659642897285624)  --> TIME: 2025-02-25 15:55:00 -- STEP: 33264/36938 -- GLOBAL_STEP: 845900 | > loss_text_ce: 0.029115932062268257 (0.03378134371262966) | > loss_mel_ce: 3.065312385559082 (nan) | > loss: 0.012279477901756763 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13670329366124856) | > loader_time: 0.0046 (0.004659751570815354)  --> TIME: 2025-02-25 15:55:09 -- STEP: 33314/36938 -- GLOBAL_STEP: 845950 | > loss_text_ce: 0.03583788126707077 (0.03378121429561645) | > loss_mel_ce: 2.9681131839752197 (nan) | > loss: 0.011920441873371601 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.136710906605579) | > loader_time: 0.0048 (0.00465978598218222)  --> TIME: 2025-02-25 15:55:18 -- STEP: 33364/36938 -- GLOBAL_STEP: 846000 | > loss_text_ce: 0.034359708428382874 (0.03378154669027259) | > loss_mel_ce: 2.349207639694214 (nan) | > loss: 0.009458601474761963 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.13670731423327415) | > loader_time: 0.0046 (0.004659723276543733)  --> TIME: 2025-02-25 15:55:28 -- STEP: 33414/36938 -- GLOBAL_STEP: 846050 | > loss_text_ce: 0.029410481452941895 (0.03378333302098603) | > loss_mel_ce: 2.4719011783599854 (nan) | > loss: 0.009925841353833675 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13670323307041762) | > loader_time: 0.0045 (0.004659734095078343)  --> TIME: 2025-02-25 15:55:37 -- STEP: 33464/36938 -- GLOBAL_STEP: 846100 | > loss_text_ce: 0.037483423948287964 (0.03378537397892605) | > loss_mel_ce: 2.949488639831543 (nan) | > loss: 0.01185306441038847 (nan) | > current_lr: 5e-06 | > step_time: 0.1521 (0.13670525767716057) | > loader_time: 0.0046 (0.004659714017391997)  --> TIME: 2025-02-25 15:55:46 -- STEP: 33514/36938 -- GLOBAL_STEP: 846150 | > loss_text_ce: 0.026753045618534088 (0.033782123237327974) | > loss_mel_ce: 2.9763455390930176 (nan) | > loss: 0.011917058378458023 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13669578886945624) | > loader_time: 0.0045 (0.004659751573211923)  --> TIME: 2025-02-25 15:55:56 -- STEP: 33564/36938 -- GLOBAL_STEP: 846200 | > loss_text_ce: 0.025729728862643242 (0.03378230761741107) | > loss_mel_ce: 3.0764000415802 (nan) | > loss: 0.01231003925204277 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13669828126348474) | > loader_time: 0.0046 (0.0046599373432723535)  --> TIME: 2025-02-25 15:56:05 -- STEP: 33614/36938 -- GLOBAL_STEP: 846250 | > loss_text_ce: 0.03184874355792999 (0.03378064411880705) | > loss_mel_ce: 2.8953003883361816 (nan) | > loss: 0.011615671217441559 (nan) | > current_lr: 5e-06 | > step_time: 0.1609 (0.13670078823764684) | > loader_time: 0.0047 (0.004659942338813762)  --> TIME: 2025-02-25 15:56:14 -- STEP: 33664/36938 -- GLOBAL_STEP: 846300 | > loss_text_ce: 0.032639432698488235 (0.033779404642728716) | > loss_mel_ce: 3.0819997787475586 (nan) | > loss: 0.012359680607914925 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.13669333262758634) | > loader_time: 0.0046 (0.004659976505677956)  --> TIME: 2025-02-25 15:56:23 -- STEP: 33714/36938 -- GLOBAL_STEP: 846350 | > loss_text_ce: 0.028887156397104263 (0.03377801472481521) | > loss_mel_ce: 3.465200424194336 (nan) | > loss: 0.013865428045392036 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13669531666568543) | > loader_time: 0.0049 (0.004660024446065489)  --> TIME: 2025-02-25 15:56:33 -- STEP: 33764/36938 -- GLOBAL_STEP: 846400 | > loss_text_ce: 0.039066508412361145 (0.033778190374197783) | > loss_mel_ce: 3.172588586807251 (nan) | > loss: 0.012744664214551449 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.13669174456254676) | > loader_time: 0.0047 (0.004660024100352113)  --> TIME: 2025-02-25 15:56:42 -- STEP: 33814/36938 -- GLOBAL_STEP: 846450 | > loss_text_ce: 0.026234567165374756 (0.033775462574742475) | > loss_mel_ce: 2.465346574783325 (nan) | > loss: 0.009887227788567543 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13669339146881795) | > loader_time: 0.0046 (0.004660013799813445)  --> TIME: 2025-02-25 15:56:51 -- STEP: 33864/36938 -- GLOBAL_STEP: 846500 | > loss_text_ce: 0.03128783032298088 (0.03377489489481239) | > loss_mel_ce: 3.188034772872925 (nan) | > loss: 0.012775090523064137 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13669230882760525) | > loader_time: 0.0047 (0.004660015540739161)  --> TIME: 2025-02-25 15:57:01 -- STEP: 33914/36938 -- GLOBAL_STEP: 846550 | > loss_text_ce: 0.03372638300061226 (0.033773668851331824) | > loss_mel_ce: 0.00439453125 (nan) | > loss: 0.00015127347433008254 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.1366970891511594) | > loader_time: 0.0046 (0.004660058451787103)  --> TIME: 2025-02-25 15:57:10 -- STEP: 33964/36938 -- GLOBAL_STEP: 846600 | > loss_text_ce: 0.03229738771915436 (0.033774865236258285) | > loss_mel_ce: 1.263604998588562 (nan) | > loss: 0.0051424698904156685 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13668685290883995) | > loader_time: 0.0045 (0.00466003270273339)  --> TIME: 2025-02-25 15:57:19 -- STEP: 34014/36938 -- GLOBAL_STEP: 846650 | > loss_text_ce: 0.03038916550576687 (0.03377569399100767) | > loss_mel_ce: 2.9397621154785156 (nan) | > loss: 0.011786314658820629 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13668120158316627) | > loader_time: 0.0046 (0.004660060286991535)  --> TIME: 2025-02-25 15:57:28 -- STEP: 34064/36938 -- GLOBAL_STEP: 846700 | > loss_text_ce: 0.03105429746210575 (0.0337757957480154) | > loss_mel_ce: 2.8589439392089844 (nan) | > loss: 0.011468247510492802 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.136683420578658) | > loader_time: 0.0046 (0.004660069893480403)  --> TIME: 2025-02-25 15:57:38 -- STEP: 34114/36938 -- GLOBAL_STEP: 846750 | > loss_text_ce: 0.03286142647266388 (0.03377570824413396) | > loss_mel_ce: 2.575284004211426 (nan) | > loss: 0.010349784046411514 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13668439895248413) | > loader_time: 0.0046 (0.004660102737783684)  --> TIME: 2025-02-25 15:57:47 -- STEP: 34164/36938 -- GLOBAL_STEP: 846800 | > loss_text_ce: 0.027820302173495293 (0.03377549676071962) | > loss_mel_ce: 3.214214563369751 (nan) | > loss: 0.012865218333899975 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.1366881496550935) | > loader_time: 0.0049 (0.0046603516217012695)  --> TIME: 2025-02-25 15:57:57 -- STEP: 34214/36938 -- GLOBAL_STEP: 846850 | > loss_text_ce: 0.03386355936527252 (0.03377730658237287) | > loss_mel_ce: 2.74666690826416 (nan) | > loss: 0.011033851653337479 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.1366931684056072) | > loader_time: 0.0046 (0.00466040881475741)  --> TIME: 2025-02-25 15:58:06 -- STEP: 34264/36938 -- GLOBAL_STEP: 846900 | > loss_text_ce: 0.0225985124707222 (0.033775261183135466) | > loss_mel_ce: 2.4385733604431152 (nan) | > loss: 0.00976655539125204 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.1367056863123704) | > loader_time: 0.0049 (0.004660483487098251)  --> TIME: 2025-02-25 15:58:16 -- STEP: 34314/36938 -- GLOBAL_STEP: 846950 | > loss_text_ce: 0.024986641481518745 (0.03377441385526541) | > loss_mel_ce: 2.950235605239868 (nan) | > loss: 0.01180643867701292 (nan) | > current_lr: 5e-06 | > step_time: 0.1884 (0.13670428561297246) | > loader_time: 0.005 (0.004660492399991324)  --> TIME: 2025-02-25 15:58:25 -- STEP: 34364/36938 -- GLOBAL_STEP: 847000 | > loss_text_ce: 0.027113519608974457 (0.03377687442850924) | > loss_mel_ce: 2.88779878616333 (nan) | > loss: 0.011567112989723682 (nan) | > current_lr: 5e-06 | > step_time: 0.1614 (0.13671017694273513) | > loader_time: 0.0047 (0.004660531904488992)  --> TIME: 2025-02-25 15:58:35 -- STEP: 34414/36938 -- GLOBAL_STEP: 847050 | > loss_text_ce: 0.0345035083591938 (0.03377544756162796) | > loss_mel_ce: 2.8638007640838623 (nan) | > loss: 0.011501207947731018 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.13671578256369124) | > loader_time: 0.0047 (0.004660467354113876)  --> TIME: 2025-02-25 15:58:44 -- STEP: 34464/36938 -- GLOBAL_STEP: 847100 | > loss_text_ce: 0.07170330733060837 (0.03377631395516303) | > loss_mel_ce: 2.6255886554718018 (nan) | > loss: 0.010703539475798607 (nan) | > current_lr: 5e-06 | > step_time: 0.1078 (0.13671119527038705) | > loader_time: 0.0045 (0.004660465362842365)  --> TIME: 2025-02-25 15:58:53 -- STEP: 34514/36938 -- GLOBAL_STEP: 847150 | > loss_text_ce: 0.03379131108522415 (0.033777172444725855) | > loss_mel_ce: 3.112072467803955 (nan) | > loss: 0.012483587488532066 (nan) | > current_lr: 5e-06 | > step_time: 0.1691 (0.13670742706551547) | > loader_time: 0.0048 (0.00466046013063659)  --> TIME: 2025-02-25 15:59:02 -- STEP: 34564/36938 -- GLOBAL_STEP: 847200 | > loss_text_ce: 0.041701510548591614 (0.03377686633034696) | > loss_mel_ce: 3.0901577472686768 (nan) | > loss: 0.012428013607859612 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13669780266118067) | > loader_time: 0.0046 (0.004660497204516025)  --> TIME: 2025-02-25 15:59:11 -- STEP: 34614/36938 -- GLOBAL_STEP: 847250 | > loss_text_ce: 0.04596788436174393 (0.03377610734229292) | > loss_mel_ce: 2.384962558746338 (nan) | > loss: 0.009646549820899963 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.13669413480738335) | > loader_time: 0.0048 (0.004660467447975288)  --> TIME: 2025-02-25 15:59:20 -- STEP: 34664/36938 -- GLOBAL_STEP: 847300 | > loss_text_ce: 0.03586685284972191 (0.033773431516984156) | > loss_mel_ce: 1.5341390371322632 (nan) | > loss: 0.006230182480067015 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13668632473622133) | > loader_time: 0.0046 (0.004660569442603311)  --> TIME: 2025-02-25 15:59:30 -- STEP: 34714/36938 -- GLOBAL_STEP: 847350 | > loss_text_ce: 0.028617404401302338 (0.03377432737074756) | > loss_mel_ce: 3.025935411453247 (nan) | > loss: 0.01212124153971672 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.13668075880916758) | > loader_time: 0.0047 (0.004660775799242211)  --> TIME: 2025-02-25 15:59:39 -- STEP: 34764/36938 -- GLOBAL_STEP: 847400 | > loss_text_ce: 0.030550828203558922 (0.033774655860688355) | > loss_mel_ce: 2.672257661819458 (nan) | > loss: 0.010725431144237518 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13667854514800754) | > loader_time: 0.0047 (0.004660792344459489)  --> TIME: 2025-02-25 15:59:48 -- STEP: 34814/36938 -- GLOBAL_STEP: 847450 | > loss_text_ce: 0.03707318753004074 (0.033777649419302974) | > loss_mel_ce: 2.8835184574127197 (nan) | > loss: 0.011589650064706802 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13667610585740972) | > loader_time: 0.0046 (0.004660863676918712)  --> TIME: 2025-02-25 15:59:57 -- STEP: 34864/36938 -- GLOBAL_STEP: 847500 | > loss_text_ce: 0.03408081829547882 (0.03377861862972502) | > loss_mel_ce: 2.8400797843933105 (nan) | > loss: 0.0114054000005126 (nan) | > current_lr: 5e-06 | > step_time: 0.175 (0.13667763068677158) | > loader_time: 0.0048 (0.004660846539843562)  --> TIME: 2025-02-25 16:00:07 -- STEP: 34914/36938 -- GLOBAL_STEP: 847550 | > loss_text_ce: 0.026409901678562164 (0.033776586011911094) | > loss_mel_ce: 2.891240358352661 (nan) | > loss: 0.011577977798879147 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.13667617586051475) | > loader_time: 0.0046 (0.004660802218842595)  --> TIME: 2025-02-25 16:00:16 -- STEP: 34964/36938 -- GLOBAL_STEP: 847600 | > loss_text_ce: 0.046669382601976395 (0.03377666303302197) | > loss_mel_ce: 2.54384183883667 (nan) | > loss: 0.010279807262122631 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13668053473532776) | > loader_time: 0.0046 (0.0046608733675351875)  --> TIME: 2025-02-25 16:00:25 -- STEP: 35014/36938 -- GLOBAL_STEP: 847650 | > loss_text_ce: 0.038428645581007004 (0.033776809444657975) | > loss_mel_ce: 2.3868799209594727 (nan) | > loss: 0.009624240919947624 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1366797619165035) | > loader_time: 0.0045 (0.004660887408244264)  --> TIME: 2025-02-25 16:00:35 -- STEP: 35064/36938 -- GLOBAL_STEP: 847700 | > loss_text_ce: 0.03227796405553818 (0.03377744380178386) | > loss_mel_ce: 3.0258798599243164 (nan) | > loss: 0.012135547585785389 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13667948623919915) | > loader_time: 0.0046 (0.004660873585249924)  --> TIME: 2025-02-25 16:00:44 -- STEP: 35114/36938 -- GLOBAL_STEP: 847750 | > loss_text_ce: 0.024286014959216118 (0.033778003622030234) | > loss_mel_ce: 2.7345099449157715 (nan) | > loss: 0.010947603732347488 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13668326755900065) | > loader_time: 0.0047 (0.004660882893880776)  --> TIME: 2025-02-25 16:00:54 -- STEP: 35164/36938 -- GLOBAL_STEP: 847800 | > loss_text_ce: 0.04651165008544922 (0.03377773820637886) | > loss_mel_ce: 3.310500383377075 (nan) | > loss: 0.013321476988494396 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13668554354115445) | > loader_time: 0.0046 (0.004660920937602899)  --> TIME: 2025-02-25 16:01:03 -- STEP: 35214/36938 -- GLOBAL_STEP: 847850 | > loss_text_ce: 0.032386720180511475 (0.03377808718888442) | > loss_mel_ce: 3.091865301132202 (nan) | > loss: 0.0123978266492486 (nan) | > current_lr: 5e-06 | > step_time: 0.1813 (0.13668696831728) | > loader_time: 0.0048 (0.00466093774236445)  --> TIME: 2025-02-25 16:01:12 -- STEP: 35264/36938 -- GLOBAL_STEP: 847900 | > loss_text_ce: 0.031045394018292427 (0.03377794280574624) | > loss_mel_ce: 2.3237266540527344 (nan) | > loss: 0.009344333782792091 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1366904327703372) | > loader_time: 0.0046 (0.004660917003174213)  --> TIME: 2025-02-25 16:01:22 -- STEP: 35314/36938 -- GLOBAL_STEP: 847950 | > loss_text_ce: 0.05151219666004181 (0.03377768192518887) | > loss_mel_ce: 3.7606682777404785 (nan) | > loss: 0.015127701684832573 (nan) | > current_lr: 5e-06 | > step_time: 0.1783 (0.13669592230885946) | > loader_time: 0.005 (0.004660980472034296)  --> TIME: 2025-02-25 16:01:32 -- STEP: 35364/36938 -- GLOBAL_STEP: 848000 | > loss_text_ce: 0.040924131870269775 (0.033778938374674806) | > loss_mel_ce: 3.251223564147949 (nan) | > loss: 0.013064078986644745 (nan) | > current_lr: 5e-06 | > step_time: 0.1664 (0.13670181638177087) | > loader_time: 0.0048 (0.0046610125399407015)  --> TIME: 2025-02-25 16:01:41 -- STEP: 35414/36938 -- GLOBAL_STEP: 848050 | > loss_text_ce: 0.025672107934951782 (0.033777818712420554) | > loss_mel_ce: 2.6448376178741455 (nan) | > loss: 0.010597261600196362 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.1366990243667556) | > loader_time: 0.0046 (0.004661057625133571)  --> TIME: 2025-02-25 16:01:50 -- STEP: 35464/36938 -- GLOBAL_STEP: 848100 | > loss_text_ce: 0.03222566470503807 (0.03378051862103529) | > loss_mel_ce: 3.0872879028320312 (nan) | > loss: 0.012379022315144539 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13669376345861933) | > loader_time: 0.0047 (0.004661191015353234)  --> TIME: 2025-02-25 16:02:00 -- STEP: 35514/36938 -- GLOBAL_STEP: 848150 | > loss_text_ce: 0.039375804364681244 (0.0337811342710446) | > loss_mel_ce: 2.387622594833374 (nan) | > loss: 0.009630946442484856 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.1366988710638257) | > loader_time: 0.0045 (0.0046612332115209725)  --> TIME: 2025-02-25 16:02:09 -- STEP: 35564/36938 -- GLOBAL_STEP: 848200 | > loss_text_ce: 0.032794345170259476 (0.03378001525658228) | > loss_mel_ce: 2.6634581089019775 (nan) | > loss: 0.010699414648115635 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13669809083973009) | > loader_time: 0.0046 (0.004661250678911842)  --> TIME: 2025-02-25 16:02:18 -- STEP: 35614/36938 -- GLOBAL_STEP: 848250 | > loss_text_ce: 0.023033589124679565 (0.033778802043270426) | > loss_mel_ce: 3.8567707538604736 (nan) | > loss: 0.015396050177514553 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.1366935752032221) | > loader_time: 0.0045 (0.004661236117541785)  --> TIME: 2025-02-25 16:02:27 -- STEP: 35664/36938 -- GLOBAL_STEP: 848300 | > loss_text_ce: 0.03596140071749687 (0.033777703560986985) | > loss_mel_ce: 2.8584036827087402 (nan) | > loss: 0.011485576629638672 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.1366905325153803) | > loader_time: 0.0047 (0.0046611917612530045)  --> TIME: 2025-02-25 16:02:37 -- STEP: 35714/36938 -- GLOBAL_STEP: 848350 | > loss_text_ce: 0.032723475247621536 (0.03377723037444418) | > loss_mel_ce: 3.072648048400879 (nan) | > loss: 0.012322903610765934 (nan) | > current_lr: 5e-06 | > step_time: 0.1835 (0.13669383975795615) | > loader_time: 0.0048 (0.004661177363195136)  --> TIME: 2025-02-25 16:02:46 -- STEP: 35764/36938 -- GLOBAL_STEP: 848400 | > loss_text_ce: 0.03455200046300888 (0.033775160240875096) | > loss_mel_ce: 0.02998860739171505 (nan) | > loss: 0.00025611353339627385 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13669096752242996) | > loader_time: 0.0046 (0.004661177584901231)  --> TIME: 2025-02-25 16:02:55 -- STEP: 35814/36938 -- GLOBAL_STEP: 848450 | > loss_text_ce: 0.033781588077545166 (0.0337745113411003) | > loss_mel_ce: 2.6717076301574707 (nan) | > loss: 0.010736068710684776 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.1366873690981753) | > loader_time: 0.0046 (0.004661141604496312)  --> TIME: 2025-02-25 16:03:04 -- STEP: 35864/36938 -- GLOBAL_STEP: 848500 | > loss_text_ce: 0.030240796506404877 (0.03377490495902399) | > loss_mel_ce: 7.28125 (nan) | > loss: 0.029013855382800102 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13668247441774373) | > loader_time: 0.0047 (0.004661133618804709)  --> TIME: 2025-02-25 16:03:14 -- STEP: 35914/36938 -- GLOBAL_STEP: 848550 | > loss_text_ce: 0.030805496498942375 (0.03377656355768806) | > loss_mel_ce: 2.8978912830352783 (nan) | > loss: 0.011621813289821148 (nan) | > current_lr: 5e-06 | > step_time: 0.1634 (0.1366830705474461) | > loader_time: 0.0047 (0.004661256322856982)  --> TIME: 2025-02-25 16:03:23 -- STEP: 35964/36938 -- GLOBAL_STEP: 848600 | > loss_text_ce: 0.04210488870739937 (0.03377472406073325) | > loss_mel_ce: 2.906456708908081 (nan) | > loss: 0.011700642295181751 (nan) | > current_lr: 5e-06 | > step_time: 0.1447 (0.13668375203665512) | > loader_time: 0.0047 (0.004661310018733095)  --> TIME: 2025-02-25 16:03:32 -- STEP: 36014/36938 -- GLOBAL_STEP: 848650 | > loss_text_ce: 0.0441642701625824 (0.033774800582459186) | > loss_mel_ce: 3.0761373043060303 (nan) | > loss: 0.012382149696350098 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13668060677435742) | > loader_time: 0.0046 (0.0046613666571284715)  --> TIME: 2025-02-25 16:03:42 -- STEP: 36064/36938 -- GLOBAL_STEP: 848700 | > loss_text_ce: 0.03011280857026577 (0.03377666156367681) | > loss_mel_ce: 2.6432974338531494 (nan) | > loss: 0.01060877088457346 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.1366830388845979) | > loader_time: 0.0048 (0.004661456755335163)  --> TIME: 2025-02-25 16:03:52 -- STEP: 36114/36938 -- GLOBAL_STEP: 848750 | > loss_text_ce: 0.04424672573804855 (0.033776270747679214) | > loss_mel_ce: 1.6674591302871704 (nan) | > loss: 0.006792483851313591 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.1366914448933786) | > loader_time: 0.0046 (0.004661460476552222)  --> TIME: 2025-02-25 16:04:01 -- STEP: 36164/36938 -- GLOBAL_STEP: 848800 | > loss_text_ce: 0.03537394851446152 (0.03377661305137124) | > loss_mel_ce: 2.738835573196411 (nan) | > loss: 0.01100876834243536 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.13668957844783752) | > loader_time: 0.0049 (0.004661592844115077)  --> TIME: 2025-02-25 16:04:10 -- STEP: 36214/36938 -- GLOBAL_STEP: 848850 | > loss_text_ce: 0.04009965807199478 (0.03377819760902818) | > loss_mel_ce: 2.6496505737304688 (nan) | > loss: 0.010673612356185913 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.13668927476415002) | > loader_time: 0.0047 (0.004661548886220376)  --> TIME: 2025-02-25 16:04:19 -- STEP: 36264/36938 -- GLOBAL_STEP: 848900 | > loss_text_ce: 0.03241124749183655 (0.03377804914033827) | > loss_mel_ce: 2.9038784503936768 (nan) | > loss: 0.011651944369077682 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.13668611813349027) | > loader_time: 0.0046 (0.004661710339089022)  --> TIME: 2025-02-25 16:04:28 -- STEP: 36314/36938 -- GLOBAL_STEP: 848950 | > loss_text_ce: 0.04386645555496216 (0.03377963033708808) | > loss_mel_ce: 1.4804949760437012 (nan) | > loss: 0.006049053277820349 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13668104489679475) | > loader_time: 0.0046 (0.004661700842026891)  --> TIME: 2025-02-25 16:04:37 -- STEP: 36364/36938 -- GLOBAL_STEP: 849000 | > loss_text_ce: 0.03556949272751808 (0.03377977043048451) | > loss_mel_ce: 2.8936312198638916 (nan) | > loss: 0.011623812839388847 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13667377477798484) | > loader_time: 0.0047 (0.004661737371102071)  --> TIME: 2025-02-25 16:04:47 -- STEP: 36414/36938 -- GLOBAL_STEP: 849050 | > loss_text_ce: 0.03750919550657272 (0.03377920477522447) | > loss_mel_ce: 2.4839634895324707 (nan) | > loss: 0.010005844756960869 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.1366728074128851) | > loader_time: 0.0047 (0.004661763127529604)  --> TIME: 2025-02-25 16:04:56 -- STEP: 36464/36938 -- GLOBAL_STEP: 849100 | > loss_text_ce: 0.03268205001950264 (0.03377976082135155) | > loss_mel_ce: 2.7447385787963867 (nan) | > loss: 0.011021510697901249 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.13667562631412136) | > loader_time: 0.0048 (0.004661804263714152)  --> TIME: 2025-02-25 16:05:06 -- STEP: 36514/36938 -- GLOBAL_STEP: 849150 | > loss_text_ce: 0.032969653606414795 (0.03377945294769836) | > loss_mel_ce: 3.2549195289611816 (nan) | > loss: 0.013047180138528347 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1366795043717572) | > loader_time: 0.0045 (0.004661784406075779)  --> TIME: 2025-02-25 16:05:15 -- STEP: 36564/36938 -- GLOBAL_STEP: 849200 | > loss_text_ce: 0.027659637853503227 (0.033779555860088656) | > loss_mel_ce: 2.8192646503448486 (nan) | > loss: 0.011297319084405899 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.1366789003989205) | > loader_time: 0.0047 (0.004661839269937054)  --> TIME: 2025-02-25 16:05:24 -- STEP: 36614/36938 -- GLOBAL_STEP: 849250 | > loss_text_ce: 0.030336007475852966 (0.03377863029164594) | > loss_mel_ce: 2.643333673477173 (nan) | > loss: 0.010609800927340984 (nan) | > current_lr: 5e-06 | > step_time: 0.1879 (0.13668024928694777) | > loader_time: 0.0049 (0.004661829922066004)  --> TIME: 2025-02-25 16:05:34 -- STEP: 36664/36938 -- GLOBAL_STEP: 849300 | > loss_text_ce: 0.02873041294515133 (0.033777885887953676) | > loss_mel_ce: 3.0376484394073486 (nan) | > loss: 0.012168170884251595 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13667744023555978) | > loader_time: 0.0047 (0.004661958764636187)  --> TIME: 2025-02-25 16:05:43 -- STEP: 36714/36938 -- GLOBAL_STEP: 849350 | > loss_text_ce: 0.03455729782581329 (0.03377824827968137) | > loss_mel_ce: 3.1873230934143066 (nan) | > loss: 0.01278524100780487 (nan) | > current_lr: 5e-06 | > step_time: 0.152 (0.1366727132322411) | > loader_time: 0.0047 (0.0046619225114611414)  --> TIME: 2025-02-25 16:05:52 -- STEP: 36764/36938 -- GLOBAL_STEP: 849400 | > loss_text_ce: 0.0249848160892725 (0.03377755540015786) | > loss_mel_ce: 2.8238418102264404 (nan) | > loss: 0.011304868385195732 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13667313227805578) | > loader_time: 0.0046 (0.004661901908190083)  --> TIME: 2025-02-25 16:06:02 -- STEP: 36814/36938 -- GLOBAL_STEP: 849450 | > loss_text_ce: 0.03720009699463844 (0.03377808739458106) | > loss_mel_ce: 2.9181573390960693 (nan) | > loss: 0.011727609671652317 (nan) | > current_lr: 5e-06 | > step_time: 0.1827 (0.13667741555992036) | > loader_time: 0.0048 (0.004661886561355329)  --> TIME: 2025-02-25 16:06:11 -- STEP: 36864/36938 -- GLOBAL_STEP: 849500 | > loss_text_ce: 0.026928871870040894 (0.03377748072224106) | > loss_mel_ce: 2.277547836303711 (nan) | > loss: 0.009144749492406845 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13667766250566543) | > loader_time: 0.0046 (0.004661918520772194)  --> TIME: 2025-02-25 16:06:20 -- STEP: 36914/36938 -- GLOBAL_STEP: 849550 | > loss_text_ce: 0.030694011598825455 (0.03377742653714885) | > loss_mel_ce: 2.282665967941284 (nan) | > loss: 0.009180000051856041 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13667413855371538) | > loader_time: 0.0046 (0.004661940117727632)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.020697486282575235 (-0.00018669709120647943) | > avg_loss_text_ce: 0.03294894587027262 (-0.0003037010181215355) | > avg_loss_mel_ce: 2.5311445815138307 (-0.006275881635080527) | > avg_loss: 2.564093521916039 (-0.006579589430651733) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_849574.pth  > EPOCH: 24/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-25 16:06:50)   --> TIME: 2025-02-25 16:06:55 -- STEP: 26/36938 -- GLOBAL_STEP: 849600 | > loss_text_ce: 0.03161356598138809 (0.03179447531986695) | > loss_mel_ce: 0.2645517885684967 (2.398585728727854) | > loss: 0.0011752593563869596 (0.00964436652765681) | > current_lr: 5e-06 | > step_time: 0.1532 (0.13332603527949402) | > loader_time: 0.0051 (0.004796138176551232)  --> TIME: 2025-02-25 16:07:04 -- STEP: 76/36938 -- GLOBAL_STEP: 849650 | > loss_text_ce: 0.03568791598081589 (0.03199304748130471) | > loss_mel_ce: 3.0040056705474854 (2.433352016255651) | > loss: 0.01206227671355009 (0.009783115931847868) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13249027415325768) | > loader_time: 0.0047 (0.0048743862854807)  --> TIME: 2025-02-25 16:07:14 -- STEP: 126/36938 -- GLOBAL_STEP: 849700 | > loss_text_ce: 0.029903260990977287 (0.032601426946856644) | > loss_mel_ce: 2.7609915733337402 (2.487957282512936) | > loss: 0.011074979789555073 (0.010002217671763711) | > current_lr: 5e-06 | > step_time: 0.1614 (0.13638182859572148) | > loader_time: 0.0047 (0.004875606960720485)  --> TIME: 2025-02-25 16:07:23 -- STEP: 176/36938 -- GLOBAL_STEP: 849750 | > loss_text_ce: 0.028554117307066917 (0.03316186601296065) | > loss_mel_ce: 2.630934476852417 (2.5210984224501214) | > loss: 0.010553526692092419 (0.010135954093460563) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13522966747934165) | > loader_time: 0.0047 (0.004832588813521644)  --> TIME: 2025-02-25 16:07:32 -- STEP: 226/36938 -- GLOBAL_STEP: 849800 | > loss_text_ce: 0.042796600610017776 (0.0333370115789296) | > loss_mel_ce: 2.678307056427002 (2.507774655907046) | > loss: 0.010798031464219093 (0.01008377702462701) | > current_lr: 5e-06 | > step_time: 0.1275 (0.1350766663002757) | > loader_time: 0.0048 (0.0048202086338954685)  --> TIME: 2025-02-25 16:07:41 -- STEP: 276/36938 -- GLOBAL_STEP: 849850 | > loss_text_ce: 0.036400195211172104 (0.033307438938999914) | > loss_mel_ce: 2.892876625061035 (2.5068495194039633) | > loss: 0.011624114587903023 (0.010079988489451349) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13556544072386148) | > loader_time: 0.0046 (0.0048050647196562395)  --> TIME: 2025-02-25 16:07:51 -- STEP: 326/36938 -- GLOBAL_STEP: 849900 | > loss_text_ce: 0.03255132585763931 (0.03349272902780696) | > loss_mel_ce: 0.18198852241039276 (2.5251802900660394) | > loss: 0.0008513486827723682 (0.010153464929138566) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13594123337166442) | > loader_time: 0.0049 (0.004790159090896326)  --> TIME: 2025-02-25 16:08:00 -- STEP: 376/36938 -- GLOBAL_STEP: 849950 | > loss_text_ce: 0.01958976686000824 (0.033345400722340685) | > loss_mel_ce: 2.4484500885009766 (2.521978696675707) | > loss: 0.009793808683753014 (0.01014017555551501) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13597186385317045) | > loader_time: 0.0059 (0.0047906966919594625)  --> TIME: 2025-02-25 16:08:09 -- STEP: 426/36938 -- GLOBAL_STEP: 850000 | > loss_text_ce: 0.031039562076330185 (0.033363450978808294) | > loss_mel_ce: 2.536309003829956 (2.5328618361376374) | > loss: 0.010187891311943531 (0.010183434248664146) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13564422936506657) | > loader_time: 0.0047 (0.00479215299579459) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_850000.pth  --> TIME: 2025-02-25 16:08:24 -- STEP: 476/36938 -- GLOBAL_STEP: 850050 | > loss_text_ce: 0.03457099571824074 (0.03349878772420164) | > loss_mel_ce: 2.875932216644287 (2.5198687172755077) | > loss: 0.011549616232514381 (0.010132411303261079) | > current_lr: 5e-06 | > step_time: 0.1361 (0.1355382083844737) | > loader_time: 0.005 (0.004798836066943255)  --> TIME: 2025-02-25 16:08:33 -- STEP: 526/36938 -- GLOBAL_STEP: 850100 | > loss_text_ce: 0.03397376090288162 (0.03360909216474215) | > loss_mel_ce: 2.457303524017334 (nan) | > loss: 0.009886021725833416 (nan) | > current_lr: 5e-06 | > step_time: 0.149 (0.13526657053726246) | > loader_time: 0.0048 (0.004793589106077475)  --> TIME: 2025-02-25 16:08:43 -- STEP: 576/36938 -- GLOBAL_STEP: 850150 | > loss_text_ce: 0.03097199648618698 (0.03367461430348664) | > loss_mel_ce: 3.0135018825531006 (nan) | > loss: 0.012081245891749859 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1356162205338478) | > loader_time: 0.0047 (0.004795240859190619)  --> TIME: 2025-02-25 16:08:52 -- STEP: 626/36938 -- GLOBAL_STEP: 850200 | > loss_text_ce: 0.027438415214419365 (0.03381392133002656) | > loss_mel_ce: 3.060708999633789 (nan) | > loss: 0.012254553847014904 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13532541468501472) | > loader_time: 0.0047 (0.004788652395668877)  --> TIME: 2025-02-25 16:09:01 -- STEP: 676/36938 -- GLOBAL_STEP: 850250 | > loss_text_ce: 0.030125731602311134 (0.0338388673724199) | > loss_mel_ce: 2.751145124435425 (nan) | > loss: 0.011036789044737816 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.1356144103072805) | > loader_time: 0.0049 (0.00478549328076063)  --> TIME: 2025-02-25 16:09:11 -- STEP: 726/36938 -- GLOBAL_STEP: 850300 | > loss_text_ce: 0.03868931904435158 (0.03382848011016766) | > loss_mel_ce: 2.8339788913726807 (nan) | > loss: 0.01139947772026062 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13574396316013404) | > loader_time: 0.0047 (0.004782459295814022)  --> TIME: 2025-02-25 16:09:20 -- STEP: 776/36938 -- GLOBAL_STEP: 850350 | > loss_text_ce: 0.029593070968985558 (0.03371210773859523) | > loss_mel_ce: 2.7105371952056885 (nan) | > loss: 0.010873532854020596 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13573358507500483) | > loader_time: 0.0045 (0.004777674515222753)  --> TIME: 2025-02-25 16:09:30 -- STEP: 826/36938 -- GLOBAL_STEP: 850400 | > loss_text_ce: 0.03560015931725502 (0.033730235940733995) | > loss_mel_ce: 2.5021278858184814 (nan) | > loss: 0.010070350021123886 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.1362803539410053) | > loader_time: 0.0046 (0.004774696601793773)  --> TIME: 2025-02-25 16:09:39 -- STEP: 876/36938 -- GLOBAL_STEP: 850450 | > loss_text_ce: 0.02574189193546772 (0.03372839709305846) | > loss_mel_ce: 2.935872793197632 (nan) | > loss: 0.01175243966281414 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13621214753416575) | > loader_time: 0.0047 (0.0047697101009490745)  --> TIME: 2025-02-25 16:09:48 -- STEP: 926/36938 -- GLOBAL_STEP: 850500 | > loss_text_ce: 0.03505997732281685 (0.0337705007697945) | > loss_mel_ce: 3.235178232192993 (nan) | > loss: 0.012977136299014091 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13633382655116955) | > loader_time: 0.0046 (0.004769883197018181)  --> TIME: 2025-02-25 16:09:58 -- STEP: 976/36938 -- GLOBAL_STEP: 850550 | > loss_text_ce: 0.03451744467020035 (0.033690741826726746) | > loss_mel_ce: 2.686906576156616 (nan) | > loss: 0.010799302719533443 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13656631893798948) | > loader_time: 0.0047 (0.004769883683470422)  --> TIME: 2025-02-25 16:10:07 -- STEP: 1026/36938 -- GLOBAL_STEP: 850600 | > loss_text_ce: 0.03587697818875313 (0.03379622242571409) | > loss_mel_ce: 3.4514801502227783 (nan) | > loss: 0.013838719576597214 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13671035492396955) | > loader_time: 0.0047 (0.0047676049710249505)  --> TIME: 2025-02-25 16:10:17 -- STEP: 1076/36938 -- GLOBAL_STEP: 850650 | > loss_text_ce: 0.036098092794418335 (0.03380206228081544) | > loss_mel_ce: 3.146054744720459 (nan) | > loss: 0.012627591378986835 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13672564552619107) | > loader_time: 0.0046 (0.00476530670676532)  --> TIME: 2025-02-25 16:10:26 -- STEP: 1126/36938 -- GLOBAL_STEP: 850700 | > loss_text_ce: 0.04552999511361122 (0.03375553494636932) | > loss_mel_ce: 2.3091914653778076 (nan) | > loss: 0.009344133548438549 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1364730317046333) | > loader_time: 0.0046 (0.004765040709324453)  --> TIME: 2025-02-25 16:10:35 -- STEP: 1176/36938 -- GLOBAL_STEP: 850750 | > loss_text_ce: 0.037567827850580215 (0.033687040098106566) | > loss_mel_ce: 3.020796298980713 (nan) | > loss: 0.012136366218328476 (nan) | > current_lr: 5e-06 | > step_time: 0.1769 (0.13627716006875842) | > loader_time: 0.0048 (0.004762281568682914)  --> TIME: 2025-02-25 16:10:44 -- STEP: 1226/36938 -- GLOBAL_STEP: 850800 | > loss_text_ce: 0.03527265414595604 (0.03368507286527641) | > loss_mel_ce: 3.052100419998169 (nan) | > loss: 0.012251480482518673 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13636986622024702) | > loader_time: 0.0047 (0.004761853381432278)  --> TIME: 2025-02-25 16:10:53 -- STEP: 1276/36938 -- GLOBAL_STEP: 850850 | > loss_text_ce: 0.028213204815983772 (0.03373374397206626) | > loss_mel_ce: 3.1749722957611084 (nan) | > loss: 0.012711054645478725 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.13636697497113745) | > loader_time: 0.0047 (0.0047605312356380726)  --> TIME: 2025-02-25 16:11:03 -- STEP: 1326/36938 -- GLOBAL_STEP: 850900 | > loss_text_ce: 0.02550683729350567 (0.03377830146335711) | > loss_mel_ce: 2.6084375381469727 (nan) | > loss: 0.010452160611748695 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13633546390504686) | > loader_time: 0.0045 (0.004758263245607022)  --> TIME: 2025-02-25 16:11:12 -- STEP: 1376/36938 -- GLOBAL_STEP: 850950 | > loss_text_ce: 0.032684292644262314 (0.03383905093691881) | > loss_mel_ce: 2.933541774749756 (nan) | > loss: 0.01177073922008276 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.1364880344548892) | > loader_time: 0.0048 (0.004758841942909142)  --> TIME: 2025-02-25 16:11:21 -- STEP: 1426/36938 -- GLOBAL_STEP: 851000 | > loss_text_ce: 0.030505917966365814 (0.033838127180019675) | > loss_mel_ce: 2.5602915287017822 (nan) | > loss: 0.010280942544341087 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.1364404959564959) | > loader_time: 0.0047 (0.004758392778051514)  --> TIME: 2025-02-25 16:11:31 -- STEP: 1476/36938 -- GLOBAL_STEP: 851050 | > loss_text_ce: 0.03908183053135872 (0.03382234067686396) | > loss_mel_ce: 2.76475191116333 (nan) | > loss: 0.011126324534416199 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.13647091097948025) | > loader_time: 0.0046 (0.004755100745172678)  --> TIME: 2025-02-25 16:11:41 -- STEP: 1526/36938 -- GLOBAL_STEP: 851100 | > loss_text_ce: 0.035376667976379395 (0.033784204204768256) | > loss_mel_ce: 1.8252743482589722 (nan) | > loss: 0.007383536081761122 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13671263407910936) | > loader_time: 0.0047 (0.004753511198851254)  --> TIME: 2025-02-25 16:11:50 -- STEP: 1576/36938 -- GLOBAL_STEP: 851150 | > loss_text_ce: 0.041607312858104706 (0.03378414055498013) | > loss_mel_ce: 2.499941110610962 (nan) | > loss: 0.010085510089993477 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1365703491087491) | > loader_time: 0.0046 (0.004752094219178709)  --> TIME: 2025-02-25 16:11:59 -- STEP: 1626/36938 -- GLOBAL_STEP: 851200 | > loss_text_ce: 0.035103656351566315 (0.033785628347324355) | > loss_mel_ce: 2.967999219894409 (nan) | > loss: 0.011917075142264366 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.13664186777957105) | > loader_time: 0.0047 (0.004750863359129873)  --> TIME: 2025-02-25 16:12:09 -- STEP: 1676/36938 -- GLOBAL_STEP: 851250 | > loss_text_ce: 0.03656410798430443 (0.033788333428014586) | > loss_mel_ce: 1.8115938901901245 (nan) | > loss: 0.007333960849791765 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13677634615317724) | > loader_time: 0.0045 (0.004749105625334662)  --> TIME: 2025-02-25 16:12:18 -- STEP: 1726/36938 -- GLOBAL_STEP: 851300 | > loss_text_ce: 0.032684534788131714 (0.03378066106948703) | > loss_mel_ce: 3.215341329574585 (nan) | > loss: 0.012888992205262184 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.13661038806513032) | > loader_time: 0.0047 (0.004746331953090687)  --> TIME: 2025-02-25 16:12:27 -- STEP: 1776/36938 -- GLOBAL_STEP: 851350 | > loss_text_ce: 0.035934366285800934 (0.03378455014165231) | > loss_mel_ce: 2.8464794158935547 (nan) | > loss: 0.011438150890171528 (nan) | > current_lr: 5e-06 | > step_time: 0.181 (0.13679825225928877) | > loader_time: 0.0049 (0.004748000217987598)  --> TIME: 2025-02-25 16:12:37 -- STEP: 1826/36938 -- GLOBAL_STEP: 851400 | > loss_text_ce: 0.03166547790169716 (0.03375799811944) | > loss_mel_ce: 3.449719190597534 (nan) | > loss: 0.013815019279718399 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13682494915589302) | > loader_time: 0.0046 (0.004746708394663737)  --> TIME: 2025-02-25 16:12:46 -- STEP: 1876/36938 -- GLOBAL_STEP: 851450 | > loss_text_ce: 0.01894468627870083 (0.03373851824198368) | > loss_mel_ce: 2.3445310592651367 (nan) | > loss: 0.009378872811794281 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.136779860646994) | > loader_time: 0.0046 (0.004746967795561106)  --> TIME: 2025-02-25 16:12:55 -- STEP: 1926/36938 -- GLOBAL_STEP: 851500 | > loss_text_ce: 0.03209042549133301 (0.033723955024285365) | > loss_mel_ce: 1.99017333984375 (nan) | > loss: 0.008024856448173523 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.136787629201776) | > loader_time: 0.0046 (0.004745991430549967)  --> TIME: 2025-02-25 16:13:05 -- STEP: 1976/36938 -- GLOBAL_STEP: 851550 | > loss_text_ce: 0.03499816358089447 (0.03370659846646596) | > loss_mel_ce: 3.0589840412139893 (nan) | > loss: 0.012277708388864994 (nan) | > current_lr: 5e-06 | > step_time: 0.1529 (0.13679477415586758) | > loader_time: 0.0046 (0.004744292994742441)  --> TIME: 2025-02-25 16:13:14 -- STEP: 2026/36938 -- GLOBAL_STEP: 851600 | > loss_text_ce: 0.03205900266766548 (0.033693109734603655) | > loss_mel_ce: 2.787832260131836 (nan) | > loss: 0.011190044693648815 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13672398285239715) | > loader_time: 0.0046 (0.004742158012352469)  --> TIME: 2025-02-25 16:13:23 -- STEP: 2076/36938 -- GLOBAL_STEP: 851650 | > loss_text_ce: 0.03474478796124458 (0.033697881216366465) | > loss_mel_ce: 2.860396385192871 (nan) | > loss: 0.01148865558207035 (nan) | > current_lr: 5e-06 | > step_time: 0.1848 (0.1367768495987834) | > loader_time: 0.0048 (0.004739253622029319)  --> TIME: 2025-02-25 16:13:33 -- STEP: 2126/36938 -- GLOBAL_STEP: 851700 | > loss_text_ce: 0.030669761821627617 (0.03368404205811422) | > loss_mel_ce: 2.8064606189727783 (nan) | > loss: 0.01125845406204462 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13681587652563476) | > loader_time: 0.0047 (0.004736667742509375)  --> TIME: 2025-02-25 16:13:42 -- STEP: 2176/36938 -- GLOBAL_STEP: 851750 | > loss_text_ce: 0.024080362170934677 (0.033681415756719346) | > loss_mel_ce: 3.046398639678955 (nan) | > loss: 0.012184441089630127 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1368452010347562) | > loader_time: 0.0053 (0.004738029969089163)  --> TIME: 2025-02-25 16:13:51 -- STEP: 2226/36938 -- GLOBAL_STEP: 851800 | > loss_text_ce: 0.030705004930496216 (0.033670313335848134) | > loss_mel_ce: 3.1584088802337646 (nan) | > loss: 0.01265521440654993 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13680721518164365) | > loader_time: 0.0047 (0.004737273702081635)  --> TIME: 2025-02-25 16:14:01 -- STEP: 2276/36938 -- GLOBAL_STEP: 851850 | > loss_text_ce: 0.02973288670182228 (0.033673997452852586) | > loss_mel_ce: 1.8638417720794678 (nan) | > loss: 0.0075141857378184795 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.136862637917061) | > loader_time: 0.0046 (0.004736555691134104)  --> TIME: 2025-02-25 16:14:10 -- STEP: 2326/36938 -- GLOBAL_STEP: 851900 | > loss_text_ce: 0.033100999891757965 (0.033673793977560575) | > loss_mel_ce: 2.8174526691436768 (nan) | > loss: 0.011311721988022327 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13677368118961813) | > loader_time: 0.0047 (0.004736755802420217)  --> TIME: 2025-02-25 16:14:19 -- STEP: 2376/36938 -- GLOBAL_STEP: 851950 | > loss_text_ce: 0.03211979940533638 (0.03365640776460802) | > loss_mel_ce: 2.694714069366455 (nan) | > loss: 0.01082076970487833 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13673359898204346) | > loader_time: 0.0046 (0.004735557519225555)  --> TIME: 2025-02-25 16:14:29 -- STEP: 2426/36938 -- GLOBAL_STEP: 852000 | > loss_text_ce: 0.04126780107617378 (0.03363798912123208) | > loss_mel_ce: 2.0956873893737793 (nan) | > loss: 0.008479981683194637 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13674138974082048) | > loader_time: 0.0047 (0.004735003398256855)  --> TIME: 2025-02-25 16:14:38 -- STEP: 2476/36938 -- GLOBAL_STEP: 852050 | > loss_text_ce: 0.03753598779439926 (0.033636249139941135) | > loss_mel_ce: 2.7977993488311768 (nan) | > loss: 0.011251331306993961 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.136792974152357) | > loader_time: 0.0046 (0.004732961897318499)  --> TIME: 2025-02-25 16:14:48 -- STEP: 2526/36938 -- GLOBAL_STEP: 852100 | > loss_text_ce: 0.033704616129398346 (0.03365117482551966) | > loss_mel_ce: 2.763899803161621 (nan) | > loss: 0.011101605370640755 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13682662015478472) | > loader_time: 0.0046 (0.004731059546330946)  --> TIME: 2025-02-25 16:14:57 -- STEP: 2576/36938 -- GLOBAL_STEP: 852150 | > loss_text_ce: 0.03203081712126732 (0.03365134927791245) | > loss_mel_ce: 2.9375803470611572 (nan) | > loss: 0.011784171685576439 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.13683576770820965) | > loader_time: 0.0048 (0.004729180787661058)  --> TIME: 2025-02-25 16:15:06 -- STEP: 2626/36938 -- GLOBAL_STEP: 852200 | > loss_text_ce: 0.0336727574467659 (0.033643362573394196) | > loss_mel_ce: 2.0823938846588135 (nan) | > loss: 0.008397090248763561 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1368661082072377) | > loader_time: 0.0045 (0.004727673185570767)  --> TIME: 2025-02-25 16:15:15 -- STEP: 2676/36938 -- GLOBAL_STEP: 852250 | > loss_text_ce: 0.045295652002096176 (0.033661056692769184) | > loss_mel_ce: 2.907160520553589 (nan) | > loss: 0.01171609666198492 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13681105095531165) | > loader_time: 0.0047 (0.0047273974425767725)  --> TIME: 2025-02-25 16:15:25 -- STEP: 2726/36938 -- GLOBAL_STEP: 852300 | > loss_text_ce: 0.03681641072034836 (0.03364653538663827) | > loss_mel_ce: 0.0890590101480484 (nan) | > loss: 0.0004995056660845876 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1368411318484421) | > loader_time: 0.0048 (0.004727316532428877)  --> TIME: 2025-02-25 16:15:34 -- STEP: 2776/36938 -- GLOBAL_STEP: 852350 | > loss_text_ce: 0.03905831649899483 (0.03363217717380157) | > loss_mel_ce: 2.925846576690674 (nan) | > loss: 0.011765495873987675 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.13685754637209732) | > loader_time: 0.0047 (0.004727793272359226)  --> TIME: 2025-02-25 16:15:44 -- STEP: 2826/36938 -- GLOBAL_STEP: 852400 | > loss_text_ce: 0.03512953594326973 (0.03360962456715766) | > loss_mel_ce: 2.169588565826416 (nan) | > loss: 0.00874888151884079 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.13694263719449348) | > loader_time: 0.0048 (0.00472838298192871)  --> TIME: 2025-02-25 16:15:54 -- STEP: 2876/36938 -- GLOBAL_STEP: 852450 | > loss_text_ce: 0.03706175088882446 (0.033618702017605084) | > loss_mel_ce: 3.056617498397827 (nan) | > loss: 0.01227650512009859 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13709176846770782) | > loader_time: 0.0048 (0.00473146362331215)  --> TIME: 2025-02-25 16:16:03 -- STEP: 2926/36938 -- GLOBAL_STEP: 852500 | > loss_text_ce: 0.03227876499295235 (0.03361480151158613) | > loss_mel_ce: 2.6944186687469482 (nan) | > loss: 0.010820228606462479 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.1370681377607748) | > loader_time: 0.0048 (0.004731085393335912)  --> TIME: 2025-02-25 16:16:12 -- STEP: 2976/36938 -- GLOBAL_STEP: 852550 | > loss_text_ce: 0.029167817905545235 (0.03362689849605125) | > loss_mel_ce: 2.68994140625 (nan) | > loss: 0.010790117084980011 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1370768895553002) | > loader_time: 0.0047 (0.0047304500495233825)  --> TIME: 2025-02-25 16:16:22 -- STEP: 3026/36938 -- GLOBAL_STEP: 852600 | > loss_text_ce: 0.028732558712363243 (0.03360906256250764) | > loss_mel_ce: 0.08642578125 (nan) | > loss: 0.0004569775774143636 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13703546071793152) | > loader_time: 0.0045 (0.004730003760968776)  --> TIME: 2025-02-25 16:16:31 -- STEP: 3076/36938 -- GLOBAL_STEP: 852650 | > loss_text_ce: 0.033332791179418564 (0.033612624156532205) | > loss_mel_ce: 2.72214674949646 (nan) | > loss: 0.010934443213045597 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13704521556824453) | > loader_time: 0.0047 (0.0047308223610271535)  --> TIME: 2025-02-25 16:16:40 -- STEP: 3126/36938 -- GLOBAL_STEP: 852700 | > loss_text_ce: 0.03407077491283417 (0.033611450045204515) | > loss_mel_ce: 2.904961585998535 (nan) | > loss: 0.011662826873362064 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.13702547595963113) | > loader_time: 0.0046 (0.004730535941633436)  --> TIME: 2025-02-25 16:16:50 -- STEP: 3176/36938 -- GLOBAL_STEP: 852750 | > loss_text_ce: 0.03906167298555374 (0.03361006525183279) | > loss_mel_ce: 2.7096073627471924 (nan) | > loss: 0.010907418094575405 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.13703344173034998) | > loader_time: 0.0063 (0.004731298318737997)  --> TIME: 2025-02-25 16:16:59 -- STEP: 3226/36938 -- GLOBAL_STEP: 852800 | > loss_text_ce: 0.03443894162774086 (0.0336010706918724) | > loss_mel_ce: 3.7928755283355713 (nan) | > loss: 0.015187756158411503 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13695450200106263) | > loader_time: 0.0046 (0.00473114266132465)  --> TIME: 2025-02-25 16:17:08 -- STEP: 3276/36938 -- GLOBAL_STEP: 852850 | > loss_text_ce: 0.03211001679301262 (0.03360299518336435) | > loss_mel_ce: 2.92629075050354 (nan) | > loss: 0.011739686131477356 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13692966558412145) | > loader_time: 0.0046 (0.004730992701464076)  --> TIME: 2025-02-25 16:17:17 -- STEP: 3326/36938 -- GLOBAL_STEP: 852900 | > loss_text_ce: 0.03479474410414696 (0.0336096666941461) | > loss_mel_ce: 2.2059037685394287 (nan) | > loss: 0.008891661651432514 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.1369225864874905) | > loader_time: 0.0049 (0.004730283676395188)  --> TIME: 2025-02-25 16:17:27 -- STEP: 3376/36938 -- GLOBAL_STEP: 852950 | > loss_text_ce: 0.035291630774736404 (0.0335980996448697) | > loss_mel_ce: 3.4381003379821777 (nan) | > loss: 0.013783302158117294 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.13692588503891792) | > loader_time: 0.005 (0.00472938025732175)  --> TIME: 2025-02-25 16:17:36 -- STEP: 3426/36938 -- GLOBAL_STEP: 853000 | > loss_text_ce: 0.02632058411836624 (0.03358410751491669) | > loss_mel_ce: 2.7999331951141357 (nan) | > loss: 0.011215293779969215 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.1369715959651383) | > loader_time: 0.0047 (0.0047290784465034786)  --> TIME: 2025-02-25 16:17:46 -- STEP: 3476/36938 -- GLOBAL_STEP: 853050 | > loss_text_ce: 0.023406028747558594 (0.03357699053239845) | > loss_mel_ce: 2.992978811264038 (nan) | > loss: 0.01196978148072958 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1369912082498719) | > loader_time: 0.0045 (0.004728517268963313)  --> TIME: 2025-02-25 16:17:55 -- STEP: 3526/36938 -- GLOBAL_STEP: 853100 | > loss_text_ce: 0.03165954351425171 (0.0335833323565833) | > loss_mel_ce: 3.6757571697235107 (nan) | > loss: 0.014711972326040268 (nan) | > current_lr: 5e-06 | > step_time: 0.1449 (0.13691349869347758) | > loader_time: 0.0047 (0.004728160118150622)  --> TIME: 2025-02-25 16:18:04 -- STEP: 3576/36938 -- GLOBAL_STEP: 853150 | > loss_text_ce: 0.03736754506826401 (0.03358479347225847) | > loss_mel_ce: 3.417464256286621 (nan) | > loss: 0.013709651306271553 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13691832741901588) | > loader_time: 0.0046 (0.004728823300176011)  --> TIME: 2025-02-25 16:18:13 -- STEP: 3626/36938 -- GLOBAL_STEP: 853200 | > loss_text_ce: 0.033729903399944305 (0.03358835672319634) | > loss_mel_ce: 2.9164674282073975 (nan) | > loss: 0.011707132682204247 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13689455404770767) | > loader_time: 0.0046 (0.004727948060432843)  --> TIME: 2025-02-25 16:18:22 -- STEP: 3676/36938 -- GLOBAL_STEP: 853250 | > loss_text_ce: 0.03460288792848587 (0.03358936424532324) | > loss_mel_ce: 3.0247817039489746 (nan) | > loss: 0.012140415608882904 (nan) | > current_lr: 5e-06 | > step_time: 0.1759 (0.13688067824329422) | > loader_time: 0.0048 (0.004727623278479833)  --> TIME: 2025-02-25 16:18:31 -- STEP: 3726/36938 -- GLOBAL_STEP: 853300 | > loss_text_ce: 0.03143591433763504 (0.03360138528480916) | > loss_mel_ce: 2.8601560592651367 (nan) | > loss: 0.011474572122097015 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13682286462921991) | > loader_time: 0.0046 (0.004726450736331983)  --> TIME: 2025-02-25 16:18:41 -- STEP: 3776/36938 -- GLOBAL_STEP: 853350 | > loss_text_ce: 0.03435012325644493 (0.03360738376425822) | > loss_mel_ce: 3.5272107124328613 (nan) | > loss: 0.014133178628981113 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.13681700813063083) | > loader_time: 0.0046 (0.004725404715133918)  --> TIME: 2025-02-25 16:18:50 -- STEP: 3826/36938 -- GLOBAL_STEP: 853400 | > loss_text_ce: 0.029040442779660225 (0.03359642943052867) | > loss_mel_ce: 2.4493000507354736 (nan) | > loss: 0.009834684431552887 (nan) | > current_lr: 5e-06 | > step_time: 0.1667 (0.13690486660940848) | > loader_time: 0.0047 (0.004724298916870046)  --> TIME: 2025-02-25 16:18:59 -- STEP: 3876/36938 -- GLOBAL_STEP: 853450 | > loss_text_ce: 0.03468425199389458 (0.033621824586281525) | > loss_mel_ce: 2.2486588954925537 (nan) | > loss: 0.00906088575720787 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1368475714458151) | > loader_time: 0.0045 (0.004725034453428437)  --> TIME: 2025-02-25 16:19:09 -- STEP: 3926/36938 -- GLOBAL_STEP: 853500 | > loss_text_ce: 0.02777879685163498 (0.033639390162945246) | > loss_mel_ce: 2.967053174972534 (nan) | > loss: 0.011884254403412342 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13685633245925752) | > loader_time: 0.0046 (0.004723962143982051)  --> TIME: 2025-02-25 16:19:18 -- STEP: 3976/36938 -- GLOBAL_STEP: 853550 | > loss_text_ce: 0.03981005772948265 (0.03363880792387222) | > loss_mel_ce: 2.669908046722412 (nan) | > loss: 0.010752850212156773 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.13680861221712565) | > loader_time: 0.0047 (0.004722729415241134)  --> TIME: 2025-02-25 16:19:27 -- STEP: 4026/36938 -- GLOBAL_STEP: 853600 | > loss_text_ce: 0.04106130078434944 (0.033636767516663106) | > loss_mel_ce: 2.8684682846069336 (nan) | > loss: 0.011545753106474876 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13672231751416813) | > loader_time: 0.0046 (0.004721366109625349)  --> TIME: 2025-02-25 16:19:36 -- STEP: 4076/36938 -- GLOBAL_STEP: 853650 | > loss_text_ce: 0.03905528038740158 (0.03364586875814183) | > loss_mel_ce: 3.004220962524414 (nan) | > loss: 0.01207649428397417 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.1367820130480163) | > loader_time: 0.0047 (0.004721345857034364)  --> TIME: 2025-02-25 16:19:46 -- STEP: 4126/36938 -- GLOBAL_STEP: 853700 | > loss_text_ce: 0.03482695296406746 (0.03363685398301635) | > loss_mel_ce: 2.8675129413604736 (nan) | > loss: 0.011517222970724106 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13675798470686168) | > loader_time: 0.0046 (0.0047213226282301814)  --> TIME: 2025-02-25 16:19:55 -- STEP: 4176/36938 -- GLOBAL_STEP: 853750 | > loss_text_ce: 0.033103909343481064 (0.03362739474358311) | > loss_mel_ce: 0.2373046875 (nan) | > loss: 0.0010730500798672438 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.13679261362872328) | > loader_time: 0.0047 (0.004721812589871923)  --> TIME: 2025-02-25 16:20:05 -- STEP: 4226/36938 -- GLOBAL_STEP: 853800 | > loss_text_ce: 0.04065608233213425 (0.03362712481421) | > loss_mel_ce: 1.8143784999847412 (nan) | > loss: 0.007361248601227999 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1368066785575874) | > loader_time: 0.0046 (0.004720725891322693)  --> TIME: 2025-02-25 16:20:14 -- STEP: 4276/36938 -- GLOBAL_STEP: 853850 | > loss_text_ce: 0.042362891137599945 (0.03361369369327642) | > loss_mel_ce: 2.154008388519287 (nan) | > loss: 0.008715759962797165 (nan) | > current_lr: 5e-06 | > step_time: 0.1352 (0.13683201365653744) | > loader_time: 0.0046 (0.004721116863532633)  --> TIME: 2025-02-25 16:20:23 -- STEP: 4326/36938 -- GLOBAL_STEP: 853900 | > loss_text_ce: 0.03937938064336777 (0.03361248790247303) | > loss_mel_ce: 2.3466312885284424 (nan) | > loss: 0.00946829654276371 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13683181976313052) | > loader_time: 0.0047 (0.004720953014566004)  --> TIME: 2025-02-25 16:20:33 -- STEP: 4376/36938 -- GLOBAL_STEP: 853950 | > loss_text_ce: 0.034709472209215164 (0.03361976807456045) | > loss_mel_ce: 3.1688199043273926 (nan) | > loss: 0.012712419033050537 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.1368329849713897) | > loader_time: 0.0047 (0.004721094855878635)  --> TIME: 2025-02-25 16:20:42 -- STEP: 4426/36938 -- GLOBAL_STEP: 854000 | > loss_text_ce: 0.0394778810441494 (0.0335959042111051) | > loss_mel_ce: 1.8519495725631714 (nan) | > loss: 0.007505664601922035 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.1367849567278012) | > loader_time: 0.0047 (0.004720108411024439)  --> TIME: 2025-02-25 16:20:51 -- STEP: 4476/36938 -- GLOBAL_STEP: 854050 | > loss_text_ce: 0.04024248197674751 (0.03361134820907373) | > loss_mel_ce: 2.3577845096588135 (nan) | > loss: 0.009515980258584023 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13675821631800575) | > loader_time: 0.0047 (0.004719226726791058)  --> TIME: 2025-02-25 16:21:00 -- STEP: 4526/36938 -- GLOBAL_STEP: 854100 | > loss_text_ce: 0.0347873792052269 (0.03359628482518037) | > loss_mel_ce: 2.8806312084198 (nan) | > loss: 0.011569121852517128 (nan) | > current_lr: 5e-06 | > step_time: 0.1798 (0.13678215538249075) | > loader_time: 0.0048 (0.004718863010617196)  --> TIME: 2025-02-25 16:21:10 -- STEP: 4576/36938 -- GLOBAL_STEP: 854150 | > loss_text_ce: 0.03768705576658249 (0.03359184034964094) | > loss_mel_ce: 2.5788912773132324 (nan) | > loss: 0.010383248329162598 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13675407111853133) | > loader_time: 0.0045 (0.004718464467075314)  --> TIME: 2025-02-25 16:21:19 -- STEP: 4626/36938 -- GLOBAL_STEP: 854200 | > loss_text_ce: 0.035775814205408096 (0.033597949658143285) | > loss_mel_ce: 2.7802977561950684 (nan) | > loss: 0.01117489580065012 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13675211654099026) | > loader_time: 0.0047 (0.004718764385927605)  --> TIME: 2025-02-25 16:21:29 -- STEP: 4676/36938 -- GLOBAL_STEP: 854250 | > loss_text_ce: 0.03171500563621521 (0.03360152974947046) | > loss_mel_ce: 3.2530665397644043 (nan) | > loss: 0.013034847564995289 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.1368265926481005) | > loader_time: 0.0047 (0.004719782732204039)  --> TIME: 2025-02-25 16:21:38 -- STEP: 4726/36938 -- GLOBAL_STEP: 854300 | > loss_text_ce: 0.042263489216566086 (0.03360680994919416) | > loss_mel_ce: 0.11773681640625 (nan) | > loss: 0.0006349218892864883 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13681307062979603) | > loader_time: 0.0047 (0.004719128207135279)  --> TIME: 2025-02-25 16:21:47 -- STEP: 4776/36938 -- GLOBAL_STEP: 854350 | > loss_text_ce: 0.03254293277859688 (0.03361402654188096) | > loss_mel_ce: 2.663490056991577 (nan) | > loss: 0.01069854386150837 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13685112527666765) | > loader_time: 0.0045 (0.0047183877459323065)  --> TIME: 2025-02-25 16:21:57 -- STEP: 4826/36938 -- GLOBAL_STEP: 854400 | > loss_text_ce: 0.03139551728963852 (0.03360724728245723) | > loss_mel_ce: 2.6843388080596924 (nan) | > loss: 0.010776723735034466 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13680774514427116) | > loader_time: 0.0046 (0.004717908654527143)  --> TIME: 2025-02-25 16:22:06 -- STEP: 4876/36938 -- GLOBAL_STEP: 854450 | > loss_text_ce: 0.031991325318813324 (0.03361540159914981) | > loss_mel_ce: 0.018310546875 (nan) | > loss: 0.0001996106147998944 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13682258354043036) | > loader_time: 0.0044 (0.004717398560174272)  --> TIME: 2025-02-25 16:22:15 -- STEP: 4926/36938 -- GLOBAL_STEP: 854500 | > loss_text_ce: 0.03871523588895798 (0.033616427781375725) | > loss_mel_ce: 2.8034415245056152 (nan) | > loss: 0.01127840019762516 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.1367818522734026) | > loader_time: 0.0045 (0.0047162401932698125)  --> TIME: 2025-02-25 16:22:24 -- STEP: 4976/36938 -- GLOBAL_STEP: 854550 | > loss_text_ce: 0.032465286552906036 (0.03359781900926876) | > loss_mel_ce: 3.4200167655944824 (nan) | > loss: 0.013700325973331928 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.13676591873935548) | > loader_time: 0.0049 (0.0047160333854975745)  --> TIME: 2025-02-25 16:22:33 -- STEP: 5026/36938 -- GLOBAL_STEP: 854600 | > loss_text_ce: 0.030229369178414345 (0.03359813190747606) | > loss_mel_ce: 2.360147476196289 (nan) | > loss: 0.009485622867941856 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1367437108503219) | > loader_time: 0.0045 (0.004715080950016328)  --> TIME: 2025-02-25 16:22:43 -- STEP: 5076/36938 -- GLOBAL_STEP: 854650 | > loss_text_ce: 0.037649355828762054 (0.033606224375945036) | > loss_mel_ce: 3.034182548522949 (nan) | > loss: 0.012189810164272785 (nan) | > current_lr: 5e-06 | > step_time: 0.1897 (0.13676214410715695) | > loader_time: 0.0049 (0.004714755301779882)  --> TIME: 2025-02-25 16:22:52 -- STEP: 5126/36938 -- GLOBAL_STEP: 854700 | > loss_text_ce: 0.04076719284057617 (0.033611609415891405) | > loss_mel_ce: 2.6219375133514404 (nan) | > loss: 0.01056628953665495 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.1367248334877198) | > loader_time: 0.0048 (0.004714242238994684)  --> TIME: 2025-02-25 16:23:01 -- STEP: 5176/36938 -- GLOBAL_STEP: 854750 | > loss_text_ce: 0.03789535537362099 (0.033611363074216895) | > loss_mel_ce: 3.122468948364258 (nan) | > loss: 0.012541129253804684 (nan) | > current_lr: 5e-06 | > step_time: 0.1692 (0.13670790536326233) | > loader_time: 0.0047 (0.004714001367412716)  --> TIME: 2025-02-25 16:23:11 -- STEP: 5226/36938 -- GLOBAL_STEP: 854800 | > loss_text_ce: 0.018733391538262367 (0.0336054237014117) | > loss_mel_ce: 1.4759657382965088 (nan) | > loss: 0.005931346211582422 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.13676475534209118) | > loader_time: 0.0045 (0.004713355787768466)  --> TIME: 2025-02-25 16:23:20 -- STEP: 5276/36938 -- GLOBAL_STEP: 854850 | > loss_text_ce: 0.034976694732904434 (0.03360614433271741) | > loss_mel_ce: 3.442723512649536 (nan) | > loss: 0.01380039844661951 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.1367879198571781) | > loader_time: 0.0047 (0.0047125064847684354)  --> TIME: 2025-02-25 16:23:29 -- STEP: 5326/36938 -- GLOBAL_STEP: 854900 | > loss_text_ce: 0.030771957710385323 (0.03360758344053525) | > loss_mel_ce: 2.1496779918670654 (nan) | > loss: 0.008652579970657825 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.1367490297862561) | > loader_time: 0.0046 (0.004712797939173038)  --> TIME: 2025-02-25 16:23:39 -- STEP: 5376/36938 -- GLOBAL_STEP: 854950 | > loss_text_ce: 0.028037237003445625 (0.03361441696324615) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13673323814180605) | > loader_time: 0.0046 (0.004712017918271671)  --> TIME: 2025-02-25 16:23:48 -- STEP: 5426/36938 -- GLOBAL_STEP: 855000 | > loss_text_ce: 0.025223664939403534 (0.03360683503335871) | > loss_mel_ce: 0.03978881984949112 (nan) | > loss: 0.0002579860738478601 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.1366801587497443) | > loader_time: 0.0045 (0.004711799721765281)  --> TIME: 2025-02-25 16:23:57 -- STEP: 5476/36938 -- GLOBAL_STEP: 855050 | > loss_text_ce: 0.03789275512099266 (0.03360792093041541) | > loss_mel_ce: 3.3880512714385986 (nan) | > loss: 0.013595016673207283 (nan) | > current_lr: 5e-06 | > step_time: 0.19 (0.13664942061143487) | > loader_time: 0.0049 (0.004711018721494811)  --> TIME: 2025-02-25 16:24:06 -- STEP: 5526/36938 -- GLOBAL_STEP: 855100 | > loss_text_ce: 0.02118084393441677 (0.03359003880149179) | > loss_mel_ce: 1.5834609270095825 (nan) | > loss: 0.00636762659996748 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.1366656638555148) | > loader_time: 0.0045 (0.0047105173248335125)  --> TIME: 2025-02-25 16:24:15 -- STEP: 5576/36938 -- GLOBAL_STEP: 855150 | > loss_text_ce: 0.049366243183612823 (0.033584323606802524) | > loss_mel_ce: 1.3960037231445312 (nan) | > loss: 0.005735595244914293 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13664785549151484) | > loader_time: 0.0045 (0.004711101951352827)  --> TIME: 2025-02-25 16:24:25 -- STEP: 5626/36938 -- GLOBAL_STEP: 855200 | > loss_text_ce: 0.030447274446487427 (0.033582984709531444) | > loss_mel_ce: 2.8932085037231445 (nan) | > loss: 0.011601809412240982 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13664188573168012) | > loader_time: 0.0045 (0.004710969957134024)  --> TIME: 2025-02-25 16:24:34 -- STEP: 5676/36938 -- GLOBAL_STEP: 855250 | > loss_text_ce: 0.031339652836322784 (0.03357998877100857) | > loss_mel_ce: 2.6695210933685303 (nan) | > loss: 0.010717702098190784 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1366297826538458) | > loader_time: 0.0062 (0.004711630690509443)  --> TIME: 2025-02-25 16:24:43 -- STEP: 5726/36938 -- GLOBAL_STEP: 855300 | > loss_text_ce: 0.022472256794571877 (0.03357868495849751) | > loss_mel_ce: 3.282987356185913 (nan) | > loss: 0.013116903603076935 (nan) | > current_lr: 5e-06 | > step_time: 0.1951 (0.13664908755545882) | > loader_time: 0.0049 (0.004711503712730383)  --> TIME: 2025-02-25 16:24:53 -- STEP: 5776/36938 -- GLOBAL_STEP: 855350 | > loss_text_ce: 0.0350072979927063 (0.03358588695759592) | > loss_mel_ce: 3.3814215660095215 (nan) | > loss: 0.013557258062064648 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.13669635797141283) | > loader_time: 0.0047 (0.00471148955689902)  --> TIME: 2025-02-25 16:25:02 -- STEP: 5826/36938 -- GLOBAL_STEP: 855400 | > loss_text_ce: 0.02888026088476181 (0.0335716212381264) | > loss_mel_ce: 2.9327123165130615 (nan) | > loss: 0.011752352118492126 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13666979504580956) | > loader_time: 0.0046 (0.00471096938310354)  --> TIME: 2025-02-25 16:25:11 -- STEP: 5876/36938 -- GLOBAL_STEP: 855450 | > loss_text_ce: 0.038241464644670486 (0.03357241980140611) | > loss_mel_ce: 1.6298828125 (nan) | > loss: 0.006619541440159082 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13664475823844457) | > loader_time: 0.0045 (0.004710454247775239)  --> TIME: 2025-02-25 16:25:21 -- STEP: 5926/36938 -- GLOBAL_STEP: 855500 | > loss_text_ce: 0.033372338861227036 (0.0335683238846911) | > loss_mel_ce: 3.13138747215271 (nan) | > loss: 0.012558571994304657 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13668611932356017) | > loader_time: 0.0048 (0.00471091004911403)  --> TIME: 2025-02-25 16:25:30 -- STEP: 5976/36938 -- GLOBAL_STEP: 855550 | > loss_text_ce: 0.032817065715789795 (0.033569321037585496) | > loss_mel_ce: 3.33549165725708 (nan) | > loss: 0.013366305269300938 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.13665955397019866) | > loader_time: 0.0049 (0.004710558866720436)  --> TIME: 2025-02-25 16:25:39 -- STEP: 6026/36938 -- GLOBAL_STEP: 855600 | > loss_text_ce: 0.03527134656906128 (0.03355694630199037) | > loss_mel_ce: 2.7607421875 (nan) | > loss: 0.011095292866230011 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13662124748369203) | > loader_time: 0.0047 (0.004710516540307041)  --> TIME: 2025-02-25 16:25:48 -- STEP: 6076/36938 -- GLOBAL_STEP: 855650 | > loss_text_ce: 0.026289556175470352 (0.03354551699754381) | > loss_mel_ce: 0.02701568603515625 (nan) | > loss: 0.00021152874978724867 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.13661737434953466) | > loader_time: 0.0047 (0.004709796461327594)  --> TIME: 2025-02-25 16:25:57 -- STEP: 6126/36938 -- GLOBAL_STEP: 855700 | > loss_text_ce: 0.03169866278767586 (0.03354654804971563) | > loss_mel_ce: 2.3355212211608887 (nan) | > loss: 0.009393730200827122 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.1365937402814262) | > loader_time: 0.0045 (0.004709342512242201)  --> TIME: 2025-02-25 16:26:06 -- STEP: 6176/36938 -- GLOBAL_STEP: 855750 | > loss_text_ce: 0.03337918594479561 (0.033540887942258894) | > loss_mel_ce: 2.8183815479278564 (nan) | > loss: 0.011316510848701 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.1365624698255342) | > loader_time: 0.0054 (0.004708916913968912)  --> TIME: 2025-02-25 16:26:16 -- STEP: 6226/36938 -- GLOBAL_STEP: 855800 | > loss_text_ce: 0.023662609979510307 (0.03353799734287939) | > loss_mel_ce: 2.5834712982177734 (nan) | > loss: 0.010345770046114922 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13654405102636308) | > loader_time: 0.0046 (0.004708668023797815)  --> TIME: 2025-02-25 16:26:26 -- STEP: 6276/36938 -- GLOBAL_STEP: 855850 | > loss_text_ce: 0.03734317794442177 (0.033532119957537784) | > loss_mel_ce: 2.7856907844543457 (nan) | > loss: 0.011202516965568066 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.13663635203001015) | > loader_time: 0.0049 (0.004709725360189605)  --> TIME: 2025-02-25 16:26:35 -- STEP: 6326/36938 -- GLOBAL_STEP: 855900 | > loss_text_ce: 0.032876819372177124 (0.03353820691607151) | > loss_mel_ce: 2.96565318107605 (nan) | > loss: 0.011898929253220558 (nan) | > current_lr: 5e-06 | > step_time: 0.1764 (0.13664929624829408) | > loader_time: 0.0048 (0.004709758224873246)  --> TIME: 2025-02-25 16:26:44 -- STEP: 6376/36938 -- GLOBAL_STEP: 855950 | > loss_text_ce: 0.032550785690546036 (0.03354505156888409) | > loss_mel_ce: 3.000545024871826 (nan) | > loss: 0.0120360953733325 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13663388599864765) | > loader_time: 0.0044 (0.004709924927619362)  --> TIME: 2025-02-25 16:26:54 -- STEP: 6426/36938 -- GLOBAL_STEP: 856000 | > loss_text_ce: 0.047160904854536057 (0.0335559954368495) | > loss_mel_ce: 3.4914774894714355 (nan) | > loss: 0.01404221635311842 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13665135734889317) | > loader_time: 0.0046 (0.0047099714964823165)  --> TIME: 2025-02-25 16:27:03 -- STEP: 6476/36938 -- GLOBAL_STEP: 856050 | > loss_text_ce: 0.02832046151161194 (0.03354650877840075) | > loss_mel_ce: 2.658395528793335 (nan) | > loss: 0.010661572217941284 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.13664777169189268) | > loader_time: 0.0067 (0.004710489507808313)  --> TIME: 2025-02-25 16:27:12 -- STEP: 6526/36938 -- GLOBAL_STEP: 856100 | > loss_text_ce: 0.027063952758908272 (0.03355902437376967) | > loss_mel_ce: 1.4106935262680054 (nan) | > loss: 0.005705387331545353 (nan) | > current_lr: 5e-06 | > step_time: 0.1071 (0.1366274163579044) | > loader_time: 0.0045 (0.004710457641363088)  --> TIME: 2025-02-25 16:27:21 -- STEP: 6576/36938 -- GLOBAL_STEP: 856150 | > loss_text_ce: 0.036755070090293884 (0.033562001628475925) | > loss_mel_ce: 3.319790840148926 (nan) | > loss: 0.01331962738186121 (nan) | > current_lr: 5e-06 | > step_time: 0.1874 (0.13661207447231777) | > loader_time: 0.005 (0.0047098247746771744)  --> TIME: 2025-02-25 16:27:31 -- STEP: 6626/36938 -- GLOBAL_STEP: 856200 | > loss_text_ce: 0.035383693873882294 (0.03356659879464924) | > loss_mel_ce: 2.3947975635528564 (nan) | > loss: 0.00964357703924179 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13661182739926744) | > loader_time: 0.0046 (0.0047097448996104505)  --> TIME: 2025-02-25 16:27:40 -- STEP: 6676/36938 -- GLOBAL_STEP: 856250 | > loss_text_ce: 0.029755260795354843 (0.03356579662531652) | > loss_mel_ce: 2.9988949298858643 (nan) | > loss: 0.012018454261124134 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13662634705125773) | > loader_time: 0.0046 (0.004709687398678549)  --> TIME: 2025-02-25 16:27:50 -- STEP: 6726/36938 -- GLOBAL_STEP: 856300 | > loss_text_ce: 0.04583724960684776 (0.03357647260104871) | > loss_mel_ce: 2.7104499340057373 (nan) | > loss: 0.010937647894024849 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13663697501502095) | > loader_time: 0.0047 (0.0047095028587724914)  --> TIME: 2025-02-25 16:27:59 -- STEP: 6776/36938 -- GLOBAL_STEP: 856350 | > loss_text_ce: 0.028095632791519165 (0.03358098187683406) | > loss_mel_ce: 2.521592140197754 (nan) | > loss: 0.010117809288203716 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13664259714950716) | > loader_time: 0.0046 (0.004709386698892874)  --> TIME: 2025-02-25 16:28:08 -- STEP: 6826/36938 -- GLOBAL_STEP: 856400 | > loss_text_ce: 0.03629818558692932 (0.0335929522332029) | > loss_mel_ce: 2.8635201454162598 (nan) | > loss: 0.011507216840982437 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.1366061979537475) | > loader_time: 0.0046 (0.004708932950075461)  --> TIME: 2025-02-25 16:28:17 -- STEP: 6876/36938 -- GLOBAL_STEP: 856450 | > loss_text_ce: 0.03270089626312256 (0.03358525324706577) | > loss_mel_ce: 2.616346597671509 (nan) | > loss: 0.01051209308207035 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13659063870994154) | > loader_time: 0.0047 (0.004709089995124041)  --> TIME: 2025-02-25 16:28:26 -- STEP: 6926/36938 -- GLOBAL_STEP: 856500 | > loss_text_ce: 0.04168599471449852 (0.033582516608788186) | > loss_mel_ce: 3.0980224609375 (nan) | > loss: 0.012459161691367626 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1365848969697189) | > loader_time: 0.0045 (0.004708843977507109)  --> TIME: 2025-02-25 16:28:36 -- STEP: 6976/36938 -- GLOBAL_STEP: 856550 | > loss_text_ce: 0.028549253940582275 (0.03358764167479338) | > loss_mel_ce: 2.8333652019500732 (nan) | > loss: 0.011356803588569164 (nan) | > current_lr: 5e-06 | > step_time: 0.1529 (0.13657719918757336) | > loader_time: 0.0047 (0.004708934096841656)  --> TIME: 2025-02-25 16:28:45 -- STEP: 7026/36938 -- GLOBAL_STEP: 856600 | > loss_text_ce: 0.030838314443826675 (0.03358929309664711) | > loss_mel_ce: 3.2214162349700928 (nan) | > loss: 0.012905772775411606 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13658110097097262) | > loader_time: 0.0047 (0.004708681865326215)  --> TIME: 2025-02-25 16:28:54 -- STEP: 7076/36938 -- GLOBAL_STEP: 856650 | > loss_text_ce: 0.043441079556941986 (0.03358068747681265) | > loss_mel_ce: 1.5367507934570312 (nan) | > loss: 0.006270602811127901 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13655341406067936) | > loader_time: 0.0046 (0.004708327736269644)  --> TIME: 2025-02-25 16:29:03 -- STEP: 7126/36938 -- GLOBAL_STEP: 856700 | > loss_text_ce: 0.032600417733192444 (0.033581517592993514) | > loss_mel_ce: 3.471701145172119 (nan) | > loss: 0.013905959203839302 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.13652420448781732) | > loader_time: 0.0048 (0.004708071388021953)  --> TIME: 2025-02-25 16:29:13 -- STEP: 7176/36938 -- GLOBAL_STEP: 856750 | > loss_text_ce: 0.029844289645552635 (0.03357694672232167) | > loss_mel_ce: 2.3505046367645264 (nan) | > loss: 0.009445829316973686 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.1365442979030114) | > loader_time: 0.0047 (0.004708357113792497)  --> TIME: 2025-02-25 16:29:22 -- STEP: 7226/36938 -- GLOBAL_STEP: 856800 | > loss_text_ce: 0.030297258868813515 (0.033579795424270364) | > loss_mel_ce: 3.1031887531280518 (nan) | > loss: 0.012434469535946846 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.1365428187660381) | > loader_time: 0.0046 (0.004708319630215069)  --> TIME: 2025-02-25 16:29:31 -- STEP: 7276/36938 -- GLOBAL_STEP: 856850 | > loss_text_ce: 0.029245220124721527 (0.0335805366513866) | > loss_mel_ce: 2.941082715988159 (nan) | > loss: 0.011787015944719315 (nan) | > current_lr: 5e-06 | > step_time: 0.1899 (0.13654992028603843) | > loader_time: 0.0048 (0.004708016882105295)  --> TIME: 2025-02-25 16:29:41 -- STEP: 7326/36938 -- GLOBAL_STEP: 856900 | > loss_text_ce: 0.0219362024217844 (0.03358446771722385) | > loss_mel_ce: 2.745842695236206 (nan) | > loss: 0.010983250103890896 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.1365402821729061) | > loader_time: 0.0047 (0.004707887398555522)  --> TIME: 2025-02-25 16:29:50 -- STEP: 7376/36938 -- GLOBAL_STEP: 856950 | > loss_text_ce: 0.0328749418258667 (0.0335883427376721) | > loss_mel_ce: 2.77142071723938 (nan) | > loss: 0.011128157377243042 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13655233270046183) | > loader_time: 0.0046 (0.004707387141447023)  --> TIME: 2025-02-25 16:30:00 -- STEP: 7426/36938 -- GLOBAL_STEP: 857000 | > loss_text_ce: 0.027672862634062767 (0.033587665537509116) | > loss_mel_ce: 3.3184571266174316 (nan) | > loss: 0.0132782943546772 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13658792382457774) | > loader_time: 0.0045 (0.004707268136484506)  --> TIME: 2025-02-25 16:30:09 -- STEP: 7476/36938 -- GLOBAL_STEP: 857050 | > loss_text_ce: 0.039931848645210266 (0.03358574227655148) | > loss_mel_ce: 2.6371641159057617 (nan) | > loss: 0.010623397305607796 (nan) | > current_lr: 5e-06 | > step_time: 0.1868 (0.13660932878812604) | > loader_time: 0.0049 (0.004707540944019333)  --> TIME: 2025-02-25 16:30:19 -- STEP: 7526/36938 -- GLOBAL_STEP: 857100 | > loss_text_ce: 0.028561823070049286 (0.03358827150678498) | > loss_mel_ce: 2.8566620349884033 (nan) | > loss: 0.011449301615357399 (nan) | > current_lr: 5e-06 | > step_time: 0.1828 (0.13662410802483782) | > loader_time: 0.0048 (0.004707868511677439)  --> TIME: 2025-02-25 16:30:28 -- STEP: 7576/36938 -- GLOBAL_STEP: 857150 | > loss_text_ce: 0.03433326259255409 (0.033593958288105574) | > loss_mel_ce: 0.01287841796875 (nan) | > loss: 0.0001873479486675933 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.1366489995369306) | > loader_time: 0.0046 (0.00470849412169856)  --> TIME: 2025-02-25 16:30:38 -- STEP: 7626/36938 -- GLOBAL_STEP: 857200 | > loss_text_ce: 0.03502863645553589 (0.03358981713914928) | > loss_mel_ce: 2.8332080841064453 (nan) | > loss: 0.01138189248740673 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13666409421335957) | > loader_time: 0.0046 (0.004708471055609847)  --> TIME: 2025-02-25 16:30:47 -- STEP: 7676/36938 -- GLOBAL_STEP: 857250 | > loss_text_ce: 0.035404108464717865 (0.03358428695903034) | > loss_mel_ce: 2.644404649734497 (nan) | > loss: 0.010634162463247776 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.13667122652407612) | > loader_time: 0.0051 (0.004708462608798602)  --> TIME: 2025-02-25 16:30:56 -- STEP: 7726/36938 -- GLOBAL_STEP: 857300 | > loss_text_ce: 0.03806915134191513 (0.033589094586370424) | > loss_mel_ce: 1.7927182912826538 (nan) | > loss: 0.007265029940754175 (nan) | > current_lr: 5e-06 | > step_time: 0.1076 (0.13666997380494827) | > loader_time: 0.0053 (0.004708225264809724)  --> TIME: 2025-02-25 16:31:06 -- STEP: 7776/36938 -- GLOBAL_STEP: 857350 | > loss_text_ce: 0.04656226187944412 (0.03358409181105469) | > loss_mel_ce: 2.4049479961395264 (nan) | > loss: 0.009728215634822845 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13669651044616915) | > loader_time: 0.0045 (0.004708101045447629)  --> TIME: 2025-02-25 16:31:15 -- STEP: 7826/36938 -- GLOBAL_STEP: 857400 | > loss_text_ce: 0.029469532892107964 (0.03357643479197629) | > loss_mel_ce: 2.7219021320343018 (nan) | > loss: 0.010918142274022102 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.1366951802853511) | > loader_time: 0.0045 (0.00470790225101378)  --> TIME: 2025-02-25 16:31:25 -- STEP: 7876/36938 -- GLOBAL_STEP: 857450 | > loss_text_ce: 0.03168562427163124 (0.033579567647735314) | > loss_mel_ce: 2.5105528831481934 (nan) | > loss: 0.010088248178362846 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13675402184317295) | > loader_time: 0.0046 (0.004708200466094026)  --> TIME: 2025-02-25 16:31:34 -- STEP: 7926/36938 -- GLOBAL_STEP: 857500 | > loss_text_ce: 0.04088253527879715 (0.033581332525075246) | > loss_mel_ce: 1.89023756980896 (nan) | > loss: 0.007663175463676453 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.136717948413033) | > loader_time: 0.0046 (0.004708293499201816)  --> TIME: 2025-02-25 16:31:43 -- STEP: 7976/36938 -- GLOBAL_STEP: 857550 | > loss_text_ce: 0.03158167004585266 (0.03358799852596622) | > loss_mel_ce: 2.7648606300354004 (nan) | > loss: 0.011096993461251259 (nan) | > current_lr: 5e-06 | > step_time: 0.1927 (0.13671930709717328) | > loader_time: 0.0049 (0.004708186942811253)  --> TIME: 2025-02-25 16:31:53 -- STEP: 8026/36938 -- GLOBAL_STEP: 857600 | > loss_text_ce: 0.02714724652469158 (0.03359378060212721) | > loss_mel_ce: 2.14227294921875 (nan) | > loss: 0.008608810603618622 (nan) | > current_lr: 5e-06 | > step_time: 0.1072 (0.13668983457333783) | > loader_time: 0.0044 (0.004707706203313369)  --> TIME: 2025-02-25 16:32:02 -- STEP: 8076/36938 -- GLOBAL_STEP: 857650 | > loss_text_ce: 0.02657422237098217 (0.0335880182221035) | > loss_mel_ce: 2.722231864929199 (nan) | > loss: 0.010907961055636406 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.1367078004289577) | > loader_time: 0.0046 (0.004707461155659511)  --> TIME: 2025-02-25 16:32:11 -- STEP: 8126/36938 -- GLOBAL_STEP: 857700 | > loss_text_ce: 0.021504880860447884 (0.03358865163497366) | > loss_mel_ce: 2.725959300994873 (nan) | > loss: 0.01090263668447733 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13666560609969078) | > loader_time: 0.0045 (0.0047073404747308)  --> TIME: 2025-02-25 16:32:20 -- STEP: 8176/36938 -- GLOBAL_STEP: 857750 | > loss_text_ce: 0.027942528948187828 (0.03358462077933185) | > loss_mel_ce: 0.0076310476288199425 (nan) | > loss: 0.0001411649864166975 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13665878589022593) | > loader_time: 0.0046 (0.004707602080533665)  --> TIME: 2025-02-25 16:32:29 -- STEP: 8226/36938 -- GLOBAL_STEP: 857800 | > loss_text_ce: 0.035104427486658096 (0.033583792633542935) | > loss_mel_ce: 2.9534783363342285 (nan) | > loss: 0.011859456077218056 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13663828743944548) | > loader_time: 0.0045 (0.004707414768283005)  --> TIME: 2025-02-25 16:32:39 -- STEP: 8276/36938 -- GLOBAL_STEP: 857850 | > loss_text_ce: 0.048390500247478485 (0.03357699979445498) | > loss_mel_ce: 3.0005252361297607 (nan) | > loss: 0.012098872102797031 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.136625255289257) | > loader_time: 0.0046 (0.004707322540117282)  --> TIME: 2025-02-25 16:32:48 -- STEP: 8326/36938 -- GLOBAL_STEP: 857900 | > loss_text_ce: 0.028670458123087883 (0.03357281877531516) | > loss_mel_ce: 2.080765962600708 (nan) | > loss: 0.008370780386030674 (nan) | > current_lr: 5e-06 | > step_time: 0.1612 (0.13665198423585787) | > loader_time: 0.0047 (0.004707211718489409)  --> TIME: 2025-02-25 16:32:57 -- STEP: 8376/36938 -- GLOBAL_STEP: 857950 | > loss_text_ce: 0.03759836032986641 (0.033573909392140035) | > loss_mel_ce: 3.738133192062378 (nan) | > loss: 0.014983062632381916 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13664790156805243) | > loader_time: 0.0047 (0.004707552243554275)  --> TIME: 2025-02-25 16:33:07 -- STEP: 8426/36938 -- GLOBAL_STEP: 858000 | > loss_text_ce: 0.03777952492237091 (0.03356625240549454) | > loss_mel_ce: 0.04931640625 (nan) | > loss: 0.0003456187841948122 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13662147188084728) | > loader_time: 0.0046 (0.004707561884983527)  --> TIME: 2025-02-25 16:33:16 -- STEP: 8476/36938 -- GLOBAL_STEP: 858050 | > loss_text_ce: 0.035692401230335236 (0.03357600708098292) | > loss_mel_ce: 3.1852729320526123 (nan) | > loss: 0.012781609781086445 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13662184027356267) | > loader_time: 0.0047 (0.004707804092994115)  --> TIME: 2025-02-25 16:33:25 -- STEP: 8526/36938 -- GLOBAL_STEP: 858100 | > loss_text_ce: 0.030596081167459488 (0.03357336307211012) | > loss_mel_ce: 2.264159917831421 (nan) | > loss: 0.009106175042688847 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13661523413943313) | > loader_time: 0.0047 (0.0047074619548689465)  --> TIME: 2025-02-25 16:33:34 -- STEP: 8576/36938 -- GLOBAL_STEP: 858150 | > loss_text_ce: 0.031556639820337296 (0.033563684425929156) | > loss_mel_ce: 0.0453898124396801 (nan) | > loss: 0.0003053430700674653 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13659494748311204) | > loader_time: 0.0046 (0.004707873785006481)  --> TIME: 2025-02-25 16:33:44 -- STEP: 8626/36938 -- GLOBAL_STEP: 858200 | > loss_text_ce: 0.033663928508758545 (0.03356552975126589) | > loss_mel_ce: 2.501844644546509 (nan) | > loss: 0.010061542503535748 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13661021335534532) | > loader_time: 0.0046 (0.0047080519579138615)  --> TIME: 2025-02-25 16:33:54 -- STEP: 8676/36938 -- GLOBAL_STEP: 858250 | > loss_text_ce: 0.03145651891827583 (0.03355859104986605) | > loss_mel_ce: 2.235504150390625 (nan) | > loss: 0.008995875716209412 (nan) | > current_lr: 5e-06 | > step_time: 0.1858 (0.13665667905847015) | > loader_time: 0.0048 (0.004708199470259742)  --> TIME: 2025-02-25 16:34:03 -- STEP: 8726/36938 -- GLOBAL_STEP: 858300 | > loss_text_ce: 0.03624681010842323 (0.03355470162338403) | > loss_mel_ce: 2.092758893966675 (nan) | > loss: 0.008448435924947262 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13665943249032014) | > loader_time: 0.0046 (0.004707994167066472)  --> TIME: 2025-02-25 16:34:12 -- STEP: 8776/36938 -- GLOBAL_STEP: 858350 | > loss_text_ce: 0.04223895072937012 (0.03354915207758735) | > loss_mel_ce: 0.0213623046875 (nan) | > loss: 0.00025238594389520586 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.136673041496259) | > loader_time: 0.0046 (0.00470831704987325)  --> TIME: 2025-02-25 16:34:22 -- STEP: 8826/36938 -- GLOBAL_STEP: 858400 | > loss_text_ce: 0.028970986604690552 (0.0335490129146882) | > loss_mel_ce: 2.560249090194702 (nan) | > loss: 0.010274683125317097 (nan) | > current_lr: 5e-06 | > step_time: 0.1582 (0.13669795485383449) | > loader_time: 0.0047 (0.00470815760030949)  --> TIME: 2025-02-25 16:34:31 -- STEP: 8876/36938 -- GLOBAL_STEP: 858450 | > loss_text_ce: 0.03613666445016861 (0.03355330767453299) | > loss_mel_ce: 0.0605061836540699 (nan) | > loss: 0.0003835034149233252 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13665568251070376) | > loader_time: 0.0045 (0.00470814529223828)  --> TIME: 2025-02-25 16:34:40 -- STEP: 8926/36938 -- GLOBAL_STEP: 858500 | > loss_text_ce: 0.034556105732917786 (0.03355544154018277) | > loss_mel_ce: 2.62398624420166 (nan) | > loss: 0.010549771599471569 (nan) | > current_lr: 5e-06 | > step_time: 0.1888 (0.13664539798396327) | > loader_time: 0.0048 (0.004707634996571896)  --> TIME: 2025-02-25 16:34:49 -- STEP: 8976/36938 -- GLOBAL_STEP: 858550 | > loss_text_ce: 0.034971121698617935 (0.03355982739714078) | > loss_mel_ce: 3.088628053665161 (nan) | > loss: 0.01239523570984602 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.1366499929345222) | > loader_time: 0.0047 (0.004707120611280905)  --> TIME: 2025-02-25 16:34:59 -- STEP: 9026/36938 -- GLOBAL_STEP: 858600 | > loss_text_ce: 0.033336613327264786 (0.03355822012133009) | > loss_mel_ce: 1.8428280353546143 (nan) | > loss: 0.007445098366588354 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1366351632883822) | > loader_time: 0.0045 (0.00470654139781829)  --> TIME: 2025-02-25 16:35:08 -- STEP: 9076/36938 -- GLOBAL_STEP: 858650 | > loss_text_ce: 0.03978419303894043 (0.03356324678694075) | > loss_mel_ce: 2.810269594192505 (nan) | > loss: 0.011309738270938396 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13663190285399762) | > loader_time: 0.0046 (0.0047059067811877795)  --> TIME: 2025-02-25 16:35:17 -- STEP: 9126/36938 -- GLOBAL_STEP: 858700 | > loss_text_ce: 0.03890077397227287 (0.033559012416474174) | > loss_mel_ce: 2.645812511444092 (nan) | > loss: 0.01065362524241209 (nan) | > current_lr: 5e-06 | > step_time: 0.1265 (0.13662453252130613) | > loader_time: 0.0045 (0.004705904817048235)  --> TIME: 2025-02-25 16:35:27 -- STEP: 9176/36938 -- GLOBAL_STEP: 858750 | > loss_text_ce: 0.04899638146162033 (0.033556996688507305) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.13665224075213406) | > loader_time: 0.0046 (0.0047060514702003025)  --> TIME: 2025-02-25 16:35:36 -- STEP: 9226/36938 -- GLOBAL_STEP: 858800 | > loss_text_ce: 0.034364279359579086 (0.03355390204273362) | > loss_mel_ce: 2.8997185230255127 (nan) | > loss: 0.011643186211585999 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.1366264012276979) | > loader_time: 0.0046 (0.004705606327639897)  --> TIME: 2025-02-25 16:35:45 -- STEP: 9276/36938 -- GLOBAL_STEP: 858850 | > loss_text_ce: 0.041673097759485245 (0.03355909646014824) | > loss_mel_ce: 4.205078125 (nan) | > loss: 0.016852188855409622 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13661795075775535) | > loader_time: 0.0045 (0.004705253193326029)  --> TIME: 2025-02-25 16:35:54 -- STEP: 9326/36938 -- GLOBAL_STEP: 858900 | > loss_text_ce: 0.03891237452626228 (0.03355676562535506) | > loss_mel_ce: 2.6097707748413086 (nan) | > loss: 0.010510647669434547 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.1366270036368106) | > loader_time: 0.0046 (0.004704753574884285)  --> TIME: 2025-02-25 16:36:04 -- STEP: 9376/36938 -- GLOBAL_STEP: 858950 | > loss_text_ce: 0.027323517948389053 (0.03355782882081184) | > loss_mel_ce: 3.1547577381134033 (nan) | > loss: 0.012627307325601578 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.13663657370687698) | > loader_time: 0.0046 (0.004704311744344942)  --> TIME: 2025-02-25 16:36:13 -- STEP: 9426/36938 -- GLOBAL_STEP: 859000 | > loss_text_ce: 0.03434513136744499 (0.033550752106810446) | > loss_mel_ce: 2.860874652862549 (nan) | > loss: 0.011488968506455421 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13664113438207773) | > loader_time: 0.0046 (0.004704188926190111)  --> TIME: 2025-02-25 16:36:22 -- STEP: 9476/36938 -- GLOBAL_STEP: 859050 | > loss_text_ce: 0.02733052708208561 (0.03355373473593829) | > loss_mel_ce: 3.158161163330078 (nan) | > loss: 0.01264084130525589 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13660354813526973) | > loader_time: 0.0047 (0.0047042850655008)  --> TIME: 2025-02-25 16:36:31 -- STEP: 9526/36938 -- GLOBAL_STEP: 859100 | > loss_text_ce: 0.028846442699432373 (0.033557370011664304) | > loss_mel_ce: 4.6328125 (nan) | > loss: 0.018498646095395088 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13660349511509487) | > loader_time: 0.0046 (0.0047038377094689434)  --> TIME: 2025-02-25 16:36:40 -- STEP: 9576/36938 -- GLOBAL_STEP: 859150 | > loss_text_ce: 0.03244508057832718 (0.03355271670838558) | > loss_mel_ce: 2.2906155586242676 (nan) | > loss: 0.009218495339155197 (nan) | > current_lr: 5e-06 | > step_time: 0.1353 (0.13655986707212464) | > loader_time: 0.0054 (0.004703528301458921)  --> TIME: 2025-02-25 16:36:50 -- STEP: 9626/36938 -- GLOBAL_STEP: 859200 | > loss_text_ce: 0.03179934248328209 (0.03355231842276878) | > loss_mel_ce: 2.9089248180389404 (nan) | > loss: 0.011669540777802467 (nan) | > current_lr: 5e-06 | > step_time: 0.196 (0.13660330220564607) | > loader_time: 0.0048 (0.004703513802695432)  --> TIME: 2025-02-25 16:37:00 -- STEP: 9676/36938 -- GLOBAL_STEP: 859250 | > loss_text_ce: 0.039083223789930344 (0.033549671368858114) | > loss_mel_ce: 2.0466675758361816 (nan) | > loss: 0.00827678944915533 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13661308471404446) | > loader_time: 0.0045 (0.004703534319658045)  --> TIME: 2025-02-25 16:37:09 -- STEP: 9726/36938 -- GLOBAL_STEP: 859300 | > loss_text_ce: 0.027170410379767418 (0.03355357922246288) | > loss_mel_ce: 2.5836539268493652 (nan) | > loss: 0.01036041509360075 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13662060947543658) | > loader_time: 0.0045 (0.004703561415918432)  --> TIME: 2025-02-25 16:37:18 -- STEP: 9776/36938 -- GLOBAL_STEP: 859350 | > loss_text_ce: 0.04721522703766823 (0.03355493347620452) | > loss_mel_ce: 2.7604620456695557 (nan) | > loss: 0.011141576804220676 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1366076726472508) | > loader_time: 0.0046 (0.004704026758378159)  --> TIME: 2025-02-25 16:37:28 -- STEP: 9826/36938 -- GLOBAL_STEP: 859400 | > loss_text_ce: 0.026895148679614067 (0.033562346464036164) | > loss_mel_ce: 2.5044524669647217 (nan) | > loss: 0.010045030154287815 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13660209464755035) | > loader_time: 0.0046 (0.004703700627374143)  --> TIME: 2025-02-25 16:37:37 -- STEP: 9876/36938 -- GLOBAL_STEP: 859450 | > loss_text_ce: 0.027399348095059395 (0.033558435092780774) | > loss_mel_ce: 2.9619393348693848 (nan) | > loss: 0.011862455867230892 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.13660023769129084) | > loader_time: 0.0046 (0.004703540317246482)  --> TIME: 2025-02-25 16:37:46 -- STEP: 9926/36938 -- GLOBAL_STEP: 859500 | > loss_text_ce: 0.03618451580405235 (0.03356483422826613) | > loss_mel_ce: 2.4872500896453857 (nan) | > loss: 0.01001362968236208 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13660100833220992) | > loader_time: 0.0047 (0.004703535323610909)  --> TIME: 2025-02-25 16:37:56 -- STEP: 9976/36938 -- GLOBAL_STEP: 859550 | > loss_text_ce: 0.04184919595718384 (0.03355756795391172) | > loss_mel_ce: 0.20878295600414276 (nan) | > loss: 0.0009945721831172705 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13661267614976397) | > loader_time: 0.0053 (0.004703601958183834)  --> TIME: 2025-02-25 16:38:05 -- STEP: 10026/36938 -- GLOBAL_STEP: 859600 | > loss_text_ce: 0.029085572808980942 (0.033559665883767256) | > loss_mel_ce: 3.0697693824768066 (nan) | > loss: 0.01229704450815916 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13660038511552805) | > loader_time: 0.0045 (0.004703220887358216)  --> TIME: 2025-02-25 16:38:14 -- STEP: 10076/36938 -- GLOBAL_STEP: 859650 | > loss_text_ce: 0.022155720740556717 (0.033554550004921795) | > loss_mel_ce: 1.7795991897583008 (nan) | > loss: 0.007149821612983942 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13657892178618322) | > loader_time: 0.0046 (0.004702757965623976)  --> TIME: 2025-02-25 16:38:23 -- STEP: 10126/36938 -- GLOBAL_STEP: 859700 | > loss_text_ce: 0.03500926494598389 (0.033552501548475895) | > loss_mel_ce: 1.1248748302459717 (nan) | > loss: 0.004602714907377958 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.1365744005943879) | > loader_time: 0.0044 (0.00470231094074042)  --> TIME: 2025-02-25 16:38:33 -- STEP: 10176/36938 -- GLOBAL_STEP: 859750 | > loss_text_ce: 0.039853114634752274 (0.03355007265438532) | > loss_mel_ce: 2.37959623336792 (nan) | > loss: 0.009600989520549774 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.13658200103625515) | > loader_time: 0.0046 (0.004701928874044301)  --> TIME: 2025-02-25 16:38:42 -- STEP: 10226/36938 -- GLOBAL_STEP: 859800 | > loss_text_ce: 0.02256006933748722 (0.03355402441942313) | > loss_mel_ce: 1.5103684663772583 (nan) | > loss: 0.006083050277084112 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13656586456410708) | > loader_time: 0.0045 (0.004701842819667484)  --> TIME: 2025-02-25 16:38:51 -- STEP: 10276/36938 -- GLOBAL_STEP: 859850 | > loss_text_ce: 0.029022080823779106 (0.0335493877590094) | > loss_mel_ce: 2.169036865234375 (nan) | > loss: 0.00872245617210865 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1365742488286055) | > loader_time: 0.0045 (0.004701518093671904)  --> TIME: 2025-02-25 16:39:00 -- STEP: 10326/36938 -- GLOBAL_STEP: 859900 | > loss_text_ce: 0.034685514867305756 (0.03354819598103531) | > loss_mel_ce: 3.1449272632598877 (nan) | > loss: 0.012617511674761772 (nan) | > current_lr: 5e-06 | > step_time: 0.1449 (0.13654830163471343) | > loader_time: 0.0046 (0.0047012985433784946)  --> TIME: 2025-02-25 16:39:09 -- STEP: 10376/36938 -- GLOBAL_STEP: 859950 | > loss_text_ce: 0.024984393268823624 (0.03353905710242687) | > loss_mel_ce: 2.554987668991089 (nan) | > loss: 0.010237985290586948 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.1365455406715801) | > loader_time: 0.0047 (0.004701285129707409)  --> TIME: 2025-02-25 16:39:19 -- STEP: 10426/36938 -- GLOBAL_STEP: 860000 | > loss_text_ce: 0.027080407366156578 (0.033541156381454926) | > loss_mel_ce: 2.244978189468384 (nan) | > loss: 0.009016105905175209 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13654836841354923) | > loader_time: 0.0046 (0.0047011734907397955) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_860000.pth  --> TIME: 2025-02-25 16:39:34 -- STEP: 10476/36938 -- GLOBAL_STEP: 860050 | > loss_text_ce: 0.03460744395852089 (0.03354370957405799) | > loss_mel_ce: 1.1128594875335693 (nan) | > loss: 0.0045534404926002026 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13656008473422132) | > loader_time: 0.0045 (0.004700876456628821)  --> TIME: 2025-02-25 16:39:44 -- STEP: 10526/36938 -- GLOBAL_STEP: 860100 | > loss_text_ce: 0.026919040828943253 (0.03354462868900842) | > loss_mel_ce: 2.846470594406128 (nan) | > loss: 0.011402340605854988 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13654630949305582) | > loader_time: 0.0046 (0.004700439252170255)  --> TIME: 2025-02-25 16:39:53 -- STEP: 10576/36938 -- GLOBAL_STEP: 860150 | > loss_text_ce: 0.0359589084982872 (0.033547215076968444) | > loss_mel_ce: 2.8742549419403076 (nan) | > loss: 0.011548468843102455 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.1365712461556319) | > loader_time: 0.0057 (0.004700663298052851)  --> TIME: 2025-02-25 16:40:02 -- STEP: 10626/36938 -- GLOBAL_STEP: 860200 | > loss_text_ce: 0.03832782804965973 (0.03354600866283287) | > loss_mel_ce: 2.291529655456543 (nan) | > loss: 0.009245467372238636 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13656232656787243) | > loader_time: 0.0045 (0.004700429982975698)  --> TIME: 2025-02-25 16:40:12 -- STEP: 10676/36938 -- GLOBAL_STEP: 860250 | > loss_text_ce: 0.036668673157691956 (0.03354631925433362) | > loss_mel_ce: 2.9734134674072266 (nan) | > loss: 0.011944771744310856 (nan) | > current_lr: 5e-06 | > step_time: 0.1813 (0.1365985853835499) | > loader_time: 0.0047 (0.0047006132719712435)  --> TIME: 2025-02-25 16:40:21 -- STEP: 10726/36938 -- GLOBAL_STEP: 860300 | > loss_text_ce: 0.037834975868463516 (0.033550620174812196) | > loss_mel_ce: 2.6782028675079346 (nan) | > loss: 0.010777927935123444 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13657927848766827) | > loader_time: 0.0046 (0.004699994018223231)  --> TIME: 2025-02-25 16:40:30 -- STEP: 10776/36938 -- GLOBAL_STEP: 860350 | > loss_text_ce: 0.030193481594324112 (0.033547710497073814) | > loss_mel_ce: 2.5623538494110107 (nan) | > loss: 0.010287887416779995 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13657464358040183) | > loader_time: 0.0045 (0.004699784468613299)  --> TIME: 2025-02-25 16:40:40 -- STEP: 10826/36938 -- GLOBAL_STEP: 860400 | > loss_text_ce: 0.0338883250951767 (0.033548857177024764) | > loss_mel_ce: 2.742356300354004 (nan) | > loss: 0.011016844771802425 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.1365712395926486) | > loader_time: 0.0045 (0.004699453174704532)  --> TIME: 2025-02-25 16:40:49 -- STEP: 10876/36938 -- GLOBAL_STEP: 860450 | > loss_text_ce: 0.04224791377782822 (0.033544687164019656) | > loss_mel_ce: 2.8426105976104736 (nan) | > loss: 0.011447852477431297 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.136569676459734) | > loader_time: 0.0045 (0.004699004511922636)  --> TIME: 2025-02-25 16:40:58 -- STEP: 10926/36938 -- GLOBAL_STEP: 860500 | > loss_text_ce: 0.028399214148521423 (0.03354431129642511) | > loss_mel_ce: 0.1934102326631546 (nan) | > loss: 0.0008801962831057608 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.1365637319899291) | > loader_time: 0.0046 (0.004698782771929446)  --> TIME: 2025-02-25 16:41:08 -- STEP: 10976/36938 -- GLOBAL_STEP: 860550 | > loss_text_ce: 0.02612745203077793 (0.0335396815030447) | > loss_mel_ce: 1.6841034889221191 (nan) | > loss: 0.00678663095459342 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13658386375483508) | > loader_time: 0.0044 (0.004699034089597257)  --> TIME: 2025-02-25 16:41:17 -- STEP: 11026/36938 -- GLOBAL_STEP: 860600 | > loss_text_ce: 0.032621119171381 (0.03354216523428945) | > loss_mel_ce: 2.1496622562408447 (nan) | > loss: 0.008659855462610722 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13659840563821302) | > loader_time: 0.0046 (0.004698956032186812)  --> TIME: 2025-02-25 16:41:27 -- STEP: 11076/36938 -- GLOBAL_STEP: 860650 | > loss_text_ce: 0.03425940126180649 (0.03354169514207446) | > loss_mel_ce: 3.1386637687683105 (nan) | > loss: 0.012590965256094933 (nan) | > current_lr: 5e-06 | > step_time: 0.169 (0.1365878489870762) | > loader_time: 0.0047 (0.004698913056052265)  --> TIME: 2025-02-25 16:41:36 -- STEP: 11126/36938 -- GLOBAL_STEP: 860700 | > loss_text_ce: 0.03558822721242905 (0.0335467488370343) | > loss_mel_ce: 2.2300212383270264 (nan) | > loss: 0.00899051409214735 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.13657590262049385) | > loader_time: 0.0047 (0.004698655855177807)  --> TIME: 2025-02-25 16:41:45 -- STEP: 11176/36938 -- GLOBAL_STEP: 860750 | > loss_text_ce: 0.04158028960227966 (0.03354358156059984) | > loss_mel_ce: 2.5862624645233154 (nan) | > loss: 0.010427948087453842 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13656895736241698) | > loader_time: 0.0045 (0.004698541860710153)  --> TIME: 2025-02-25 16:41:54 -- STEP: 11226/36938 -- GLOBAL_STEP: 860800 | > loss_text_ce: 0.02927776426076889 (0.03354992567303199) | > loss_mel_ce: 3.119760513305664 (nan) | > loss: 0.012496184557676315 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.1365519613160307) | > loader_time: 0.0046 (0.004698747112940881)  --> TIME: 2025-02-25 16:42:04 -- STEP: 11276/36938 -- GLOBAL_STEP: 860850 | > loss_text_ce: 0.025697801262140274 (0.03355235546246163) | > loss_mel_ce: 2.9066834449768066 (nan) | > loss: 0.011636434122920036 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1365691922200657) | > loader_time: 0.0065 (0.004698856200847109)  --> TIME: 2025-02-25 16:42:13 -- STEP: 11326/36938 -- GLOBAL_STEP: 860900 | > loss_text_ce: 0.0394546277821064 (0.03355373518884158) | > loss_mel_ce: 3.1735143661499023 (nan) | > loss: 0.012749877758324146 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.1365821285549243) | > loader_time: 0.0046 (0.0046985864891888945)  --> TIME: 2025-02-25 16:42:22 -- STEP: 11376/36938 -- GLOBAL_STEP: 860950 | > loss_text_ce: 0.02755426988005638 (0.03355148736066217) | > loss_mel_ce: 2.54510498046875 (nan) | > loss: 0.010208966210484505 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13658336947152017) | > loader_time: 0.0046 (0.004698431567300728)  --> TIME: 2025-02-25 16:42:32 -- STEP: 11426/36938 -- GLOBAL_STEP: 861000 | > loss_text_ce: 0.033523064106702805 (0.03355164971711527) | > loss_mel_ce: 1.1463351249694824 (nan) | > loss: 0.004681977443397045 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13657145816933822) | > loader_time: 0.0045 (0.004698960622096051)  --> TIME: 2025-02-25 16:42:41 -- STEP: 11476/36938 -- GLOBAL_STEP: 861050 | > loss_text_ce: 0.034891486167907715 (0.03354543082423492) | > loss_mel_ce: 3.486829996109009 (nan) | > loss: 0.013975085690617561 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13657522754163906) | > loader_time: 0.0046 (0.0046989059938541845)  --> TIME: 2025-02-25 16:42:51 -- STEP: 11526/36938 -- GLOBAL_STEP: 861100 | > loss_text_ce: 0.03296817094087601 (0.03354648239553429) | > loss_mel_ce: 3.208245038986206 (nan) | > loss: 0.01286195870488882 (nan) | > current_lr: 5e-06 | > step_time: 0.1566 (0.13661638469553705) | > loader_time: 0.0047 (0.004699206612868464)  --> TIME: 2025-02-25 16:43:00 -- STEP: 11576/36938 -- GLOBAL_STEP: 861150 | > loss_text_ce: 0.03544779121875763 (0.0335494974592206) | > loss_mel_ce: 3.182772159576416 (nan) | > loss: 0.012770715169608593 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.1366368690975466) | > loader_time: 0.0063 (0.004699381285893481)  --> TIME: 2025-02-25 16:43:10 -- STEP: 11626/36938 -- GLOBAL_STEP: 861200 | > loss_text_ce: 0.030852116644382477 (0.03354920659516395) | > loss_mel_ce: 2.850959300994873 (nan) | > loss: 0.01143576018512249 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.13661497297127825) | > loader_time: 0.0046 (0.004699196705571267)  --> TIME: 2025-02-25 16:43:18 -- STEP: 11676/36938 -- GLOBAL_STEP: 861250 | > loss_text_ce: 0.037532560527324677 (0.03355038494744916) | > loss_mel_ce: 1.802001953125 (nan) | > loss: 0.007299740798771381 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13659009650865173) | > loader_time: 0.0045 (0.004698910607830957)  --> TIME: 2025-02-25 16:43:28 -- STEP: 11726/36938 -- GLOBAL_STEP: 861300 | > loss_text_ce: 0.02528863213956356 (0.03354424179097793) | > loss_mel_ce: 3.424175977706909 (nan) | > loss: 0.013688351958990097 (nan) | > current_lr: 5e-06 | > step_time: 0.1268 (0.13660235478571772) | > loader_time: 0.0046 (0.004699216510337421)  --> TIME: 2025-02-25 16:43:37 -- STEP: 11776/36938 -- GLOBAL_STEP: 861350 | > loss_text_ce: 0.03342815488576889 (0.033548369386508965) | > loss_mel_ce: 2.679692506790161 (nan) | > loss: 0.010766352526843548 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13659254022185527) | > loader_time: 0.0046 (0.004698841731347458)  --> TIME: 2025-02-25 16:43:46 -- STEP: 11826/36938 -- GLOBAL_STEP: 861400 | > loss_text_ce: 0.03564521670341492 (0.03355744520163505) | > loss_mel_ce: 2.408090829849243 (nan) | > loss: 0.009697365574538708 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13657404639364848) | > loader_time: 0.0046 (0.004699043910676354)  --> TIME: 2025-02-25 16:43:56 -- STEP: 11876/36938 -- GLOBAL_STEP: 861450 | > loss_text_ce: 0.03032945841550827 (0.03355333979692283) | > loss_mel_ce: 2.4553403854370117 (nan) | > loss: 0.009863769635558128 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13658197219623106) | > loader_time: 0.0048 (0.004698901495254162)  --> TIME: 2025-02-25 16:44:06 -- STEP: 11926/36938 -- GLOBAL_STEP: 861500 | > loss_text_ce: 0.025900835171341896 (0.0335512569261006) | > loss_mel_ce: 2.89416241645813 (nan) | > loss: 0.011587553657591343 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.13662738490580442) | > loader_time: 0.0047 (0.004699432867876365)  --> TIME: 2025-02-25 16:44:15 -- STEP: 11976/36938 -- GLOBAL_STEP: 861550 | > loss_text_ce: 0.0330304317176342 (0.033549286115450244) | > loss_mel_ce: 2.941915512084961 (nan) | > loss: 0.011805341579020023 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13663423807921973) | > loader_time: 0.0049 (0.004699426029392604)  --> TIME: 2025-02-25 16:44:24 -- STEP: 12026/36938 -- GLOBAL_STEP: 861600 | > loss_text_ce: 0.03390416130423546 (0.03355222870018782) | > loss_mel_ce: 3.132826566696167 (nan) | > loss: 0.012566392309963703 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.13663175727372195) | > loader_time: 0.0048 (0.004699232136967283)  --> TIME: 2025-02-25 16:44:34 -- STEP: 12076/36938 -- GLOBAL_STEP: 861650 | > loss_text_ce: 0.029230300337076187 (0.033553617314008646) | > loss_mel_ce: 2.0462777614593506 (nan) | > loss: 0.008236143738031387 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.1366584563973957) | > loader_time: 0.0047 (0.004699494871093394)  --> TIME: 2025-02-25 16:44:44 -- STEP: 12126/36938 -- GLOBAL_STEP: 861700 | > loss_text_ce: 0.03990664333105087 (0.033560639219494995) | > loss_mel_ce: 2.7927086353302 (nan) | > loss: 0.011240538209676743 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13669006525241276) | > loader_time: 0.0047 (0.004699355026646516)  --> TIME: 2025-02-25 16:44:53 -- STEP: 12176/36938 -- GLOBAL_STEP: 861750 | > loss_text_ce: 0.033881351351737976 (0.03355885827323362) | > loss_mel_ce: 2.6810309886932373 (nan) | > loss: 0.010773462243378162 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.1366751874049291) | > loader_time: 0.0046 (0.004699575642561924)  --> TIME: 2025-02-25 16:45:02 -- STEP: 12226/36938 -- GLOBAL_STEP: 861800 | > loss_text_ce: 0.038743969053030014 (0.03355975393888384) | > loss_mel_ce: 2.2182395458221436 (nan) | > loss: 0.008956284262239933 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.13665921097591116) | > loader_time: 0.0047 (0.004699346088220321)  --> TIME: 2025-02-25 16:45:11 -- STEP: 12276/36938 -- GLOBAL_STEP: 861850 | > loss_text_ce: 0.03383198752999306 (0.03355200922373945) | > loss_mel_ce: 3.2014951705932617 (nan) | > loss: 0.012838601134717464 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13666198594673432) | > loader_time: 0.0046 (0.00469917816383979)  --> TIME: 2025-02-25 16:45:21 -- STEP: 12326/36938 -- GLOBAL_STEP: 861900 | > loss_text_ce: 0.033074717968702316 (0.03355831844675018) | > loss_mel_ce: 2.7729012966156006 (nan) | > loss: 0.011134825646877289 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.136658761846773) | > loader_time: 0.0046 (0.004698889123606195)  --> TIME: 2025-02-25 16:45:30 -- STEP: 12376/36938 -- GLOBAL_STEP: 861950 | > loss_text_ce: 0.035116057842969894 (0.03356265894777847) | > loss_mel_ce: 3.085716724395752 (nan) | > loss: 0.012384257279336452 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13664292289352226) | > loader_time: 0.0046 (0.004698592015983353)  --> TIME: 2025-02-25 16:45:39 -- STEP: 12426/36938 -- GLOBAL_STEP: 862000 | > loss_text_ce: 0.031388454139232635 (0.033562398132499574) | > loss_mel_ce: 2.680670738220215 (nan) | > loss: 0.010762140154838562 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.13664107635058353) | > loader_time: 0.0047 (0.004698458317299676)  --> TIME: 2025-02-25 16:45:48 -- STEP: 12476/36938 -- GLOBAL_STEP: 862050 | > loss_text_ce: 0.034903302788734436 (0.03356316681085627) | > loss_mel_ce: 3.733154296875 (nan) | > loss: 0.014952610246837139 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.13664158579629684) | > loader_time: 0.0068 (0.004698464812357927)  --> TIME: 2025-02-25 16:45:57 -- STEP: 12526/36938 -- GLOBAL_STEP: 862100 | > loss_text_ce: 0.04261112958192825 (0.033564460807249646) | > loss_mel_ce: 2.937221050262451 (nan) | > loss: 0.011824731715023518 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13663694871113333) | > loader_time: 0.0045 (0.004698244295389748)  --> TIME: 2025-02-25 16:46:07 -- STEP: 12576/36938 -- GLOBAL_STEP: 862150 | > loss_text_ce: 0.040873151272535324 (0.033565927103517676) | > loss_mel_ce: 3.116163730621338 (nan) | > loss: 0.01252792403101921 (nan) | > current_lr: 5e-06 | > step_time: 0.1775 (0.1366462114885556) | > loader_time: 0.0048 (0.0046980134365516)  --> TIME: 2025-02-25 16:46:16 -- STEP: 12626/36938 -- GLOBAL_STEP: 862200 | > loss_text_ce: 0.03279247507452965 (0.03356194620628612) | > loss_mel_ce: 2.664741277694702 (nan) | > loss: 0.010704499669373035 (nan) | > current_lr: 5e-06 | > step_time: 0.1819 (0.13664078997598791) | > loader_time: 0.0048 (0.004697934961462466)  --> TIME: 2025-02-25 16:46:25 -- STEP: 12676/36938 -- GLOBAL_STEP: 862250 | > loss_text_ce: 0.04184317961335182 (0.03356103020124749) | > loss_mel_ce: 2.920217990875244 (nan) | > loss: 0.011754211969673634 (nan) | > current_lr: 5e-06 | > step_time: 0.1892 (0.13664315060208906) | > loader_time: 0.0048 (0.00469779528139064)  --> TIME: 2025-02-25 16:46:35 -- STEP: 12726/36938 -- GLOBAL_STEP: 862300 | > loss_text_ce: 0.02930423431098461 (0.03356211062565507) | > loss_mel_ce: 3.169365167617798 (nan) | > loss: 0.0126931332051754 (nan) | > current_lr: 5e-06 | > step_time: 0.187 (0.13662842665943079) | > loader_time: 0.0049 (0.0046982181047879705)  --> TIME: 2025-02-25 16:46:44 -- STEP: 12776/36938 -- GLOBAL_STEP: 862350 | > loss_text_ce: 0.035232510417699814 (0.03356459466604878) | > loss_mel_ce: 2.4507720470428467 (nan) | > loss: 0.00986509770154953 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13662836578911905) | > loader_time: 0.0046 (0.004698097425919419)  --> TIME: 2025-02-25 16:46:53 -- STEP: 12826/36938 -- GLOBAL_STEP: 862400 | > loss_text_ce: 0.04208320006728172 (0.03356196379821106) | > loss_mel_ce: 2.4953646659851074 (nan) | > loss: 0.010069238021969795 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1366128155468736) | > loader_time: 0.0045 (0.004697954768084611)  --> TIME: 2025-02-25 16:47:02 -- STEP: 12876/36938 -- GLOBAL_STEP: 862450 | > loss_text_ce: 0.023422829806804657 (0.03355730358233725) | > loss_mel_ce: 2.6027650833129883 (nan) | > loss: 0.010421380400657654 (nan) | > current_lr: 5e-06 | > step_time: 0.1798 (0.136601455780466) | > loader_time: 0.0048 (0.004698297017551624)  --> TIME: 2025-02-25 16:47:12 -- STEP: 12926/36938 -- GLOBAL_STEP: 862500 | > loss_text_ce: 0.038025084882974625 (0.03355510475208493) | > loss_mel_ce: 0.2086012065410614 (nan) | > loss: 0.0009786757873371243 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1366106861980078) | > loader_time: 0.0046 (0.004698323664900559)  --> TIME: 2025-02-25 16:47:22 -- STEP: 12976/36938 -- GLOBAL_STEP: 862550 | > loss_text_ce: 0.03494919091463089 (0.03355131703776282) | > loss_mel_ce: 2.8326058387756348 (nan) | > loss: 0.011379187926650047 (nan) | > current_lr: 5e-06 | > step_time: 0.1825 (0.13664606195195778) | > loader_time: 0.0049 (0.004698744574251639)  --> TIME: 2025-02-25 16:47:31 -- STEP: 13026/36938 -- GLOBAL_STEP: 862600 | > loss_text_ce: 0.03343278169631958 (0.03355307657553011) | > loss_mel_ce: 2.2235171794891357 (nan) | > loss: 0.008956151083111763 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13666821739630736) | > loader_time: 0.0046 (0.00469872985551675)  --> TIME: 2025-02-25 16:47:40 -- STEP: 13076/36938 -- GLOBAL_STEP: 862650 | > loss_text_ce: 0.03018864244222641 (0.03355115588096855) | > loss_mel_ce: 3.2050957679748535 (nan) | > loss: 0.01283843070268631 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13666311567367148) | > loader_time: 0.0047 (0.00469844382849989)  --> TIME: 2025-02-25 16:47:50 -- STEP: 13126/36938 -- GLOBAL_STEP: 862700 | > loss_text_ce: 0.036856286227703094 (0.03355770704791815) | > loss_mel_ce: 2.868356704711914 (nan) | > loss: 0.011528623290359974 (nan) | > current_lr: 5e-06 | > step_time: 0.1789 (0.13665238375636826) | > loader_time: 0.0049 (0.004698284094097137)  --> TIME: 2025-02-25 16:47:59 -- STEP: 13176/36938 -- GLOBAL_STEP: 862750 | > loss_text_ce: 0.029558436945080757 (0.033556722991906776) | > loss_mel_ce: 2.7070083618164062 (nan) | > loss: 0.010859392583370209 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13663815435092194) | > loader_time: 0.0047 (0.004698144517382039)  --> TIME: 2025-02-25 16:48:08 -- STEP: 13226/36938 -- GLOBAL_STEP: 862800 | > loss_text_ce: 0.028903478756546974 (0.03355384275441664) | > loss_mel_ce: 2.8965978622436523 (nan) | > loss: 0.011609133332967758 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13663744605580785) | > loader_time: 0.0046 (0.0046983304731102625)  --> TIME: 2025-02-25 16:48:17 -- STEP: 13276/36938 -- GLOBAL_STEP: 862850 | > loss_text_ce: 0.04672699794173241 (0.03354651873617554) | > loss_mel_ce: 2.6139490604400635 (nan) | > loss: 0.010558238252997398 (nan) | > current_lr: 5e-06 | > step_time: 0.145 (0.1366350105384199) | > loader_time: 0.0046 (0.004698126457584432)  --> TIME: 2025-02-25 16:48:27 -- STEP: 13326/36938 -- GLOBAL_STEP: 862900 | > loss_text_ce: 0.022288059815764427 (0.033542779533847294) | > loss_mel_ce: 4.498161792755127 (nan) | > loss: 0.017938293516635895 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.13664268212831512) | > loader_time: 0.0047 (0.004698157095791022)  --> TIME: 2025-02-25 16:48:36 -- STEP: 13376/36938 -- GLOBAL_STEP: 862950 | > loss_text_ce: 0.033331096172332764 (0.03354048721451052) | > loss_mel_ce: 2.547494411468506 (nan) | > loss: 0.01024137157946825 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.136634405434987) | > loader_time: 0.0047 (0.004698082626436303)  --> TIME: 2025-02-25 16:48:45 -- STEP: 13426/36938 -- GLOBAL_STEP: 863000 | > loss_text_ce: 0.03797479346394539 (0.03353833075006219) | > loss_mel_ce: 2.773352861404419 (nan) | > loss: 0.011156062595546246 (nan) | > current_lr: 5e-06 | > step_time: 0.1749 (0.13663856034553734) | > loader_time: 0.0049 (0.0046982354633360165)  --> TIME: 2025-02-25 16:48:54 -- STEP: 13476/36938 -- GLOBAL_STEP: 863050 | > loss_text_ce: 0.03251764923334122 (0.03353510156957178) | > loss_mel_ce: 2.397376537322998 (nan) | > loss: 0.009642438031733036 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.13662488972195927) | > loader_time: 0.0046 (0.004698306437096335)  --> TIME: 2025-02-25 16:49:04 -- STEP: 13526/36938 -- GLOBAL_STEP: 863100 | > loss_text_ce: 0.03167051449418068 (0.03353269281091335) | > loss_mel_ce: 3.2779788970947266 (nan) | > loss: 0.013133530505001545 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13662378866961908) | > loader_time: 0.0046 (0.004698066779817313)  --> TIME: 2025-02-25 16:49:13 -- STEP: 13576/36938 -- GLOBAL_STEP: 863150 | > loss_text_ce: 0.04042944312095642 (0.03352792489922185) | > loss_mel_ce: 2.26483154296875 (nan) | > loss: 0.009147861041128635 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13662572439516257) | > loader_time: 0.0046 (0.004698266211016836)  --> TIME: 2025-02-25 16:49:22 -- STEP: 13626/36938 -- GLOBAL_STEP: 863200 | > loss_text_ce: 0.037571899592876434 (0.03352737200187908) | > loss_mel_ce: 2.9899473190307617 (nan) | > loss: 0.012013966217637062 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13662509550419036) | > loader_time: 0.0045 (0.004698374154860607)  --> TIME: 2025-02-25 16:49:32 -- STEP: 13676/36938 -- GLOBAL_STEP: 863250 | > loss_text_ce: 0.025831451639533043 (0.03352849226632614) | > loss_mel_ce: 1.32830810546875 (nan) | > loss: 0.0053735701367259026 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.1366209436243962) | > loader_time: 0.0046 (0.004698243849665895)  --> TIME: 2025-02-25 16:49:41 -- STEP: 13726/36938 -- GLOBAL_STEP: 863300 | > loss_text_ce: 0.04048342630267143 (0.033528986399642684) | > loss_mel_ce: 3.0187265872955322 (nan) | > loss: 0.01213972270488739 (nan) | > current_lr: 5e-06 | > step_time: 0.1434 (0.13663109308009588) | > loader_time: 0.0046 (0.004698151422106229)  --> TIME: 2025-02-25 16:49:50 -- STEP: 13776/36938 -- GLOBAL_STEP: 863350 | > loss_text_ce: 0.04248046875 (0.033527941803291356) | > loss_mel_ce: 2.7697482109069824 (nan) | > loss: 0.011159637942910194 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13663559385362237) | > loader_time: 0.0046 (0.004697921609075488)  --> TIME: 2025-02-25 16:50:00 -- STEP: 13826/36938 -- GLOBAL_STEP: 863400 | > loss_text_ce: 0.030934352427721024 (0.03352924994174536) | > loss_mel_ce: 0.14277362823486328 (nan) | > loss: 0.0006893174140714109 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13663388524647968) | > loader_time: 0.0045 (0.004697688457401259)  --> TIME: 2025-02-25 16:50:09 -- STEP: 13876/36938 -- GLOBAL_STEP: 863450 | > loss_text_ce: 0.03481505438685417 (0.0335261484179976) | > loss_mel_ce: 2.577861785888672 (nan) | > loss: 0.01036776602268219 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.1366405378813423) | > loader_time: 0.0046 (0.004697423601191734)  --> TIME: 2025-02-25 16:50:19 -- STEP: 13926/36938 -- GLOBAL_STEP: 863500 | > loss_text_ce: 0.02785535529255867 (0.033528000339916236) | > loss_mel_ce: 2.976337194442749 (nan) | > loss: 0.01192140020430088 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.1366446225033327) | > loader_time: 0.0046 (0.004697134093137081)  --> TIME: 2025-02-25 16:50:28 -- STEP: 13976/36938 -- GLOBAL_STEP: 863550 | > loss_text_ce: 0.020607465878129005 (0.03352677053871841) | > loss_mel_ce: 1.6608201265335083 (nan) | > loss: 0.006672332063317299 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13664758212237443) | > loader_time: 0.0046 (0.004696910867161508)  --> TIME: 2025-02-25 16:50:37 -- STEP: 14026/36938 -- GLOBAL_STEP: 863600 | > loss_text_ce: 0.022748302668333054 (0.03352033120448384) | > loss_mel_ce: 2.904350757598877 (nan) | > loss: 0.011615472845733166 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13665841826864925) | > loader_time: 0.0045 (0.004697176081876196)  --> TIME: 2025-02-25 16:50:47 -- STEP: 14076/36938 -- GLOBAL_STEP: 863650 | > loss_text_ce: 0.026658393442630768 (0.03352137020528107) | > loss_mel_ce: 3.1411538124084473 (nan) | > loss: 0.012570683844387531 (nan) | > current_lr: 5e-06 | > step_time: 0.1796 (0.13664995921067977) | > loader_time: 0.0048 (0.004696861828345727)  --> TIME: 2025-02-25 16:50:56 -- STEP: 14126/36938 -- GLOBAL_STEP: 863700 | > loss_text_ce: 0.03991317003965378 (0.033520962735939706) | > loss_mel_ce: 0.996002197265625 (nan) | > loss: 0.004110775422304869 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13665522655447604) | > loader_time: 0.0046 (0.004696589209583433)  --> TIME: 2025-02-25 16:51:05 -- STEP: 14176/36938 -- GLOBAL_STEP: 863750 | > loss_text_ce: 0.029388397932052612 (0.03352085589916044) | > loss_mel_ce: 2.3743393421173096 (nan) | > loss: 0.009538603015244007 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.1366614741241687) | > loader_time: 0.0052 (0.004696505938910992)  --> TIME: 2025-02-25 16:51:15 -- STEP: 14226/36938 -- GLOBAL_STEP: 863800 | > loss_text_ce: 0.03789423778653145 (0.03352106415693976) | > loss_mel_ce: 2.59716796875 (nan) | > loss: 0.010456596501171589 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13666073434315654) | > loader_time: 0.0046 (0.0046973297838317895)  --> TIME: 2025-02-25 16:51:24 -- STEP: 14276/36938 -- GLOBAL_STEP: 863850 | > loss_text_ce: 0.037769198417663574 (0.03351835179094086) | > loss_mel_ce: 3.265611410140991 (nan) | > loss: 0.013108653947710991 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.1366710949926089) | > loader_time: 0.0048 (0.004697476975822273)  --> TIME: 2025-02-25 16:51:33 -- STEP: 14326/36938 -- GLOBAL_STEP: 863900 | > loss_text_ce: 0.026300985366106033 (0.033516299065917785) | > loss_mel_ce: 2.982154369354248 (nan) | > loss: 0.011938314884901047 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13666619910685243) | > loader_time: 0.0046 (0.004697394757135661)  --> TIME: 2025-02-25 16:51:43 -- STEP: 14376/36938 -- GLOBAL_STEP: 863950 | > loss_text_ce: 0.03865302354097366 (0.03351672784119253) | > loss_mel_ce: 3.2600958347320557 (nan) | > loss: 0.013090274296700954 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13665612876979105) | > loader_time: 0.0047 (0.004697395170404952)  --> TIME: 2025-02-25 16:51:52 -- STEP: 14426/36938 -- GLOBAL_STEP: 864000 | > loss_text_ce: 0.039973583072423935 (0.03351762826741281) | > loss_mel_ce: 3.166050434112549 (nan) | > loss: 0.012722318060696125 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13665186927501338) | > loader_time: 0.0065 (0.004697516327110538)  --> TIME: 2025-02-25 16:52:01 -- STEP: 14476/36938 -- GLOBAL_STEP: 864050 | > loss_text_ce: 0.029661132022738457 (0.03351897071668929) | > loss_mel_ce: 3.00111722946167 (nan) | > loss: 0.012026899494230747 (nan) | > current_lr: 5e-06 | > step_time: 0.1493 (0.13666430882374586) | > loader_time: 0.0047 (0.00469754863160852)  --> TIME: 2025-02-25 16:52:11 -- STEP: 14526/36938 -- GLOBAL_STEP: 864100 | > loss_text_ce: 0.0247561763972044 (0.03351602541027826) | > loss_mel_ce: 2.411651611328125 (nan) | > loss: 0.009668285958468914 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13666652361092332) | > loader_time: 0.0046 (0.004697473075753235)  --> TIME: 2025-02-25 16:52:20 -- STEP: 14576/36938 -- GLOBAL_STEP: 864150 | > loss_text_ce: 0.038163743913173676 (0.033511227586203625) | > loss_mel_ce: 3.1611530780792236 (nan) | > loss: 0.01269570179283619 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13664955679331311) | > loader_time: 0.0047 (0.004697349196469598)  --> TIME: 2025-02-25 16:52:29 -- STEP: 14626/36938 -- GLOBAL_STEP: 864200 | > loss_text_ce: 0.031495217233896255 (0.03351433935488434) | > loss_mel_ce: 2.6127007007598877 (nan) | > loss: 0.010492841713130474 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13665512019561254) | > loader_time: 0.0045 (0.0046974409951828396)  --> TIME: 2025-02-25 16:52:38 -- STEP: 14676/36938 -- GLOBAL_STEP: 864250 | > loss_text_ce: 0.03314487263560295 (0.0335124506826164) | > loss_mel_ce: 2.5376718044281006 (nan) | > loss: 0.010201654396951199 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13664560132469825) | > loader_time: 0.0045 (0.004697242479098188)  --> TIME: 2025-02-25 16:52:48 -- STEP: 14726/36938 -- GLOBAL_STEP: 864300 | > loss_text_ce: 0.031065361574292183 (0.03351258626966607) | > loss_mel_ce: 0.1210479736328125 (nan) | > loss: 0.0006036243867129087 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13665496663491564) | > loader_time: 0.0046 (0.004697141821546642)  --> TIME: 2025-02-25 16:52:57 -- STEP: 14776/36938 -- GLOBAL_STEP: 864350 | > loss_text_ce: 0.034344349056482315 (0.033515873009933236) | > loss_mel_ce: 3.219372272491455 (nan) | > loss: 0.01291157491505146 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.13664606975166593) | > loader_time: 0.0048 (0.0046968238219481494)  --> TIME: 2025-02-25 16:53:06 -- STEP: 14826/36938 -- GLOBAL_STEP: 864400 | > loss_text_ce: 0.030270840972661972 (0.033519992554158294) | > loss_mel_ce: 2.9973278045654297 (nan) | > loss: 0.012014281004667282 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13663885105513324) | > loader_time: 0.0046 (0.004696912664513526)  --> TIME: 2025-02-25 16:53:16 -- STEP: 14876/36938 -- GLOBAL_STEP: 864450 | > loss_text_ce: 0.02690030448138714 (0.033515077909759) | > loss_mel_ce: 1.4884916543960571 (nan) | > loss: 0.006013460457324982 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.1366489927401365) | > loader_time: 0.0045 (0.004697109573340651)  --> TIME: 2025-02-25 16:53:25 -- STEP: 14926/36938 -- GLOBAL_STEP: 864500 | > loss_text_ce: 0.035599078983068466 (0.03351548416879436) | > loss_mel_ce: 2.494145154953003 (nan) | > loss: 0.010038668289780617 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13663516344277182) | > loader_time: 0.0046 (0.004696957821918193)  --> TIME: 2025-02-25 16:53:34 -- STEP: 14976/36938 -- GLOBAL_STEP: 864550 | > loss_text_ce: 0.032821934670209885 (0.03351796075531792) | > loss_mel_ce: 2.748215436935425 (nan) | > loss: 0.011035863310098648 (nan) | > current_lr: 5e-06 | > step_time: 0.1343 (0.13662254864461382) | > loader_time: 0.0046 (0.004696861912424732)  --> TIME: 2025-02-25 16:53:43 -- STEP: 15026/36938 -- GLOBAL_STEP: 864600 | > loss_text_ce: 0.04466739669442177 (0.03351616749885056) | > loss_mel_ce: 1.7137545347213745 (nan) | > loss: 0.006977865006774664 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13663272770079776) | > loader_time: 0.0045 (0.004696664949176173)  --> TIME: 2025-02-25 16:53:53 -- STEP: 15076/36938 -- GLOBAL_STEP: 864650 | > loss_text_ce: 0.038628049194812775 (0.033516673745861966) | > loss_mel_ce: 3.1920266151428223 (nan) | > loss: 0.012820059433579445 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.1366311083677497) | > loader_time: 0.0047 (0.004696570346702708)  --> TIME: 2025-02-25 16:54:02 -- STEP: 15126/36938 -- GLOBAL_STEP: 864700 | > loss_text_ce: 0.03976237028837204 (0.03351621486500971) | > loss_mel_ce: 2.942779302597046 (nan) | > loss: 0.011835482902824879 (nan) | > current_lr: 5e-06 | > step_time: 0.1078 (0.13663537428521846) | > loader_time: 0.0045 (0.004696762058984867)  --> TIME: 2025-02-25 16:54:11 -- STEP: 15176/36938 -- GLOBAL_STEP: 864750 | > loss_text_ce: 0.03500114753842354 (0.033521634878742164) | > loss_mel_ce: 2.5830211639404297 (nan) | > loss: 0.010388977825641632 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.13663598272947475) | > loader_time: 0.0047 (0.004696564119491804)  --> TIME: 2025-02-25 16:54:21 -- STEP: 15226/36938 -- GLOBAL_STEP: 864800 | > loss_text_ce: 0.029403163120150566 (0.033522467531958935) | > loss_mel_ce: 0.02783203125 (nan) | > loss: 0.00022712380450684577 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13662990337756636) | > loader_time: 0.0046 (0.004696449766205393)  --> TIME: 2025-02-25 16:54:30 -- STEP: 15276/36938 -- GLOBAL_STEP: 864850 | > loss_text_ce: 0.0401243157684803 (0.033525065398141006) | > loss_mel_ce: 2.750631093978882 (nan) | > loss: 0.011074427515268326 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13663266472368205) | > loader_time: 0.0045 (0.004696467185588541)  --> TIME: 2025-02-25 16:54:39 -- STEP: 15326/36938 -- GLOBAL_STEP: 864900 | > loss_text_ce: 0.051257312297821045 (0.033528843668484805) | > loss_mel_ce: 2.527458667755127 (nan) | > loss: 0.010233000852167606 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13664420194263682) | > loader_time: 0.0046 (0.00469625190641274)  --> TIME: 2025-02-25 16:54:49 -- STEP: 15376/36938 -- GLOBAL_STEP: 864950 | > loss_text_ce: 0.029913030564785004 (0.03352909093209288) | > loss_mel_ce: 1.4601949453353882 (nan) | > loss: 0.005913127213716507 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.1366607859405095) | > loader_time: 0.0046 (0.004696058634664697)  --> TIME: 2025-02-25 16:54:58 -- STEP: 15426/36938 -- GLOBAL_STEP: 865000 | > loss_text_ce: 0.027266787365078926 (0.03352728044470726) | > loss_mel_ce: 2.4131407737731934 (nan) | > loss: 0.009684157557785511 (nan) | > current_lr: 5e-06 | > step_time: 0.1506 (0.13665220385996185) | > loader_time: 0.0046 (0.00469582248998339)  --> TIME: 2025-02-25 16:55:07 -- STEP: 15476/36938 -- GLOBAL_STEP: 865050 | > loss_text_ce: 0.03154964745044708 (0.0335277248081636) | > loss_mel_ce: 2.9556660652160645 (nan) | > loss: 0.01185403112322092 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13663792990689175) | > loader_time: 0.0046 (0.004695542039227859)  --> TIME: 2025-02-25 16:55:16 -- STEP: 15526/36938 -- GLOBAL_STEP: 865100 | > loss_text_ce: 0.03297480568289757 (0.03352005128531187) | > loss_mel_ce: 3.1458590030670166 (nan) | > loss: 0.012614420615136623 (nan) | > current_lr: 5e-06 | > step_time: 0.1628 (0.13662584004874495) | > loader_time: 0.0047 (0.004695619794183903)  --> TIME: 2025-02-25 16:55:26 -- STEP: 15576/36938 -- GLOBAL_STEP: 865150 | > loss_text_ce: 0.028821203857660294 (0.03351321989237717) | > loss_mel_ce: 2.9151151180267334 (nan) | > loss: 0.01168228778988123 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13663228876356823) | > loader_time: 0.0053 (0.004695521282059861)  --> TIME: 2025-02-25 16:55:35 -- STEP: 15626/36938 -- GLOBAL_STEP: 865200 | > loss_text_ce: 0.033600736409425735 (0.033513926320708945) | > loss_mel_ce: 2.555649995803833 (nan) | > loss: 0.010274805128574371 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13663676453967183) | > loader_time: 0.0045 (0.0046954195706621115)  --> TIME: 2025-02-25 16:55:44 -- STEP: 15676/36938 -- GLOBAL_STEP: 865250 | > loss_text_ce: 0.03730783984065056 (0.03351668901445617) | > loss_mel_ce: 2.6287312507629395 (nan) | > loss: 0.010579520836472511 (nan) | > current_lr: 5e-06 | > step_time: 0.1508 (0.13663804115887698) | > loader_time: 0.0047 (0.004695403983502326)  --> TIME: 2025-02-25 16:55:54 -- STEP: 15726/36938 -- GLOBAL_STEP: 865300 | > loss_text_ce: 0.0315285325050354 (0.03351586150516166) | > loss_mel_ce: 3.0592939853668213 (nan) | > loss: 0.012265169061720371 (nan) | > current_lr: 5e-06 | > step_time: 0.1891 (0.1366334584595272) | > loader_time: 0.0048 (0.004695396333587954)  --> TIME: 2025-02-25 16:56:03 -- STEP: 15776/36938 -- GLOBAL_STEP: 865350 | > loss_text_ce: 0.03046869859099388 (0.033516384251795685) | > loss_mel_ce: 2.6884889602661133 (nan) | > loss: 0.01078951545059681 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13662235969280842) | > loader_time: 0.0046 (0.004695137286403827)  --> TIME: 2025-02-25 16:56:12 -- STEP: 15826/36938 -- GLOBAL_STEP: 865400 | > loss_text_ce: 0.024237550795078278 (0.033517804590632576) | > loss_mel_ce: 3.0164568424224854 (nan) | > loss: 0.012066248804330826 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.136623322805532) | > loader_time: 0.0046 (0.004695063879889709)  --> TIME: 2025-02-25 16:56:21 -- STEP: 15876/36938 -- GLOBAL_STEP: 865450 | > loss_text_ce: 0.03628559783101082 (0.033519129635346846) | > loss_mel_ce: 2.954620361328125 (nan) | > loss: 0.011868675239384174 (nan) | > current_lr: 5e-06 | > step_time: 0.1527 (0.1366259452977687) | > loader_time: 0.0047 (0.004694943074826827)  --> TIME: 2025-02-25 16:56:31 -- STEP: 15926/36938 -- GLOBAL_STEP: 865500 | > loss_text_ce: 0.029557012021541595 (0.03351541681979571) | > loss_mel_ce: 2.4153518676757812 (nan) | > loss: 0.009702019393444061 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.136618389910255) | > loader_time: 0.0046 (0.004694897416217909)  --> TIME: 2025-02-25 16:56:40 -- STEP: 15976/36938 -- GLOBAL_STEP: 865550 | > loss_text_ce: 0.032900985330343246 (0.03351552150920842) | > loss_mel_ce: 2.8469910621643066 (nan) | > loss: 0.011428143829107285 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.1366292473430565) | > loader_time: 0.0046 (0.004694781872888283)  --> TIME: 2025-02-25 16:56:49 -- STEP: 16026/36938 -- GLOBAL_STEP: 865600 | > loss_text_ce: 0.02564845234155655 (0.03351585463770981) | > loss_mel_ce: 2.2064995765686035 (nan) | > loss: 0.008857730776071548 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13662161104660941) | > loader_time: 0.0046 (0.004694612511501872)  --> TIME: 2025-02-25 16:56:59 -- STEP: 16076/36938 -- GLOBAL_STEP: 865650 | > loss_text_ce: 0.025438683107495308 (0.03351510665576413) | > loss_mel_ce: 2.9761791229248047 (nan) | > loss: 0.011911182664334774 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13662601982547604) | > loader_time: 0.0046 (0.0046945025624846705)  --> TIME: 2025-02-25 16:57:08 -- STEP: 16126/36938 -- GLOBAL_STEP: 865700 | > loss_text_ce: 0.03208319470286369 (0.0335118841316004) | > loss_mel_ce: 0.1050567626953125 (nan) | > loss: 0.0005442062392830849 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.13662933667540222) | > loader_time: 0.0047 (0.004694923253977801)  --> TIME: 2025-02-25 16:57:18 -- STEP: 16176/36938 -- GLOBAL_STEP: 865750 | > loss_text_ce: 0.03052743338048458 (0.03351265158022674) | > loss_mel_ce: 2.5345871448516846 (nan) | > loss: 0.010179026052355766 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13664332831059184) | > loader_time: 0.0053 (0.00469531861716985)  --> TIME: 2025-02-25 16:57:27 -- STEP: 16226/36938 -- GLOBAL_STEP: 865800 | > loss_text_ce: 0.032398756593465805 (0.0335083442896145) | > loss_mel_ce: 3.0613887310028076 (nan) | > loss: 0.012276935391128063 (nan) | > current_lr: 5e-06 | > step_time: 0.1468 (0.13664838141612126) | > loader_time: 0.0046 (0.004695167924125598)  --> TIME: 2025-02-25 16:57:36 -- STEP: 16276/36938 -- GLOBAL_STEP: 865850 | > loss_text_ce: 0.04294230416417122 (0.03351186073031992) | > loss_mel_ce: 2.903933048248291 (nan) | > loss: 0.011693950742483139 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13664791942316573) | > loader_time: 0.0045 (0.00469492803952358)  --> TIME: 2025-02-25 16:57:46 -- STEP: 16326/36938 -- GLOBAL_STEP: 865900 | > loss_text_ce: 0.03964896872639656 (0.033514322801095286) | > loss_mel_ce: 2.864100694656372 (nan) | > loss: 0.011522816494107246 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13663976472571637) | > loader_time: 0.0046 (0.004694961835601131)  --> TIME: 2025-02-25 16:57:55 -- STEP: 16376/36938 -- GLOBAL_STEP: 865950 | > loss_text_ce: 0.03193715587258339 (0.033516310026862596) | > loss_mel_ce: 2.2013232707977295 (nan) | > loss: 0.008862145245075226 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.1366318806998601) | > loader_time: 0.0047 (0.00469485870150745)  --> TIME: 2025-02-25 16:58:04 -- STEP: 16426/36938 -- GLOBAL_STEP: 866000 | > loss_text_ce: 0.027386052533984184 (0.033515299431075445) | > loss_mel_ce: 2.480142116546631 (nan) | > loss: 0.009950508363544941 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13661389128525783) | > loader_time: 0.0046 (0.00469476590562385)  --> TIME: 2025-02-25 16:58:13 -- STEP: 16476/36938 -- GLOBAL_STEP: 866050 | > loss_text_ce: 0.03763047978281975 (0.0335169344851381) | > loss_mel_ce: 2.3073322772979736 (nan) | > loss: 0.009305408224463463 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13660459738097205) | > loader_time: 0.0044 (0.004694679562517019)  --> TIME: 2025-02-25 16:58:22 -- STEP: 16526/36938 -- GLOBAL_STEP: 866100 | > loss_text_ce: 0.029051443561911583 (0.033517433430672) | > loss_mel_ce: 2.331785202026367 (nan) | > loss: 0.00936840008944273 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.13661841725547116) | > loader_time: 0.0046 (0.004694518375869918)  --> TIME: 2025-02-25 16:58:32 -- STEP: 16576/36938 -- GLOBAL_STEP: 866150 | > loss_text_ce: 0.02824016660451889 (0.03351494070425884) | > loss_mel_ce: 2.499753713607788 (nan) | > loss: 0.010031722486019135 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13661089568294596) | > loader_time: 0.0053 (0.0046946061307628975)  --> TIME: 2025-02-25 16:58:41 -- STEP: 16626/36938 -- GLOBAL_STEP: 866200 | > loss_text_ce: 0.039182133972644806 (0.03351281226800709) | > loss_mel_ce: 3.2817962169647217 (nan) | > loss: 0.013178486377000809 (nan) | > current_lr: 5e-06 | > step_time: 0.1591 (0.13662173744287914) | > loader_time: 0.0047 (0.004694615806552961)  --> TIME: 2025-02-25 16:58:51 -- STEP: 16676/36938 -- GLOBAL_STEP: 866250 | > loss_text_ce: 0.04206378757953644 (0.03351485739772796) | > loss_mel_ce: 3.75272798538208 (nan) | > loss: 0.015058698132634163 (nan) | > current_lr: 5e-06 | > step_time: 0.1731 (0.1366403744100064) | > loader_time: 0.0048 (0.0046947950023231)  --> TIME: 2025-02-25 16:59:00 -- STEP: 16726/36938 -- GLOBAL_STEP: 866300 | > loss_text_ce: 0.03832864761352539 (0.033515932277075514) | > loss_mel_ce: 2.1963675022125244 (nan) | > loss: 0.008867842145264149 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.13663811321878874) | > loader_time: 0.0046 (0.004694734109470055)  --> TIME: 2025-02-25 16:59:10 -- STEP: 16776/36938 -- GLOBAL_STEP: 866350 | > loss_text_ce: 0.032835014164447784 (0.033517334653525396) | > loss_mel_ce: 2.585526943206787 (nan) | > loss: 0.0103903254494071 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13664722294823334) | > loader_time: 0.0053 (0.004694774128677836)  --> TIME: 2025-02-25 16:59:19 -- STEP: 16826/36938 -- GLOBAL_STEP: 866400 | > loss_text_ce: 0.031114839017391205 (0.03351892952099962) | > loss_mel_ce: 3.0580039024353027 (nan) | > loss: 0.012258408591151237 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13664994798941613) | > loader_time: 0.0045 (0.004694763763642648)  --> TIME: 2025-02-25 16:59:28 -- STEP: 16876/36938 -- GLOBAL_STEP: 866450 | > loss_text_ce: 0.029313120990991592 (0.03352130466661966) | > loss_mel_ce: 2.5757336616516113 (nan) | > loss: 0.010337487794458866 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.1366522387323515) | > loader_time: 0.0046 (0.004694813982969654)  --> TIME: 2025-02-25 16:59:38 -- STEP: 16926/36938 -- GLOBAL_STEP: 866500 | > loss_text_ce: 0.03682732582092285 (0.03352030746528268) | > loss_mel_ce: 2.8599960803985596 (nan) | > loss: 0.011495331302285194 (nan) | > current_lr: 5e-06 | > step_time: 0.1625 (0.13665897854955178) | > loader_time: 0.0049 (0.00469474427372966)  --> TIME: 2025-02-25 16:59:47 -- STEP: 16976/36938 -- GLOBAL_STEP: 866550 | > loss_text_ce: 0.02188093401491642 (0.03351879764003063) | > loss_mel_ce: 5.095312595367432 (nan) | > loss: 0.02030632458627224 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1366676249726551) | > loader_time: 0.0045 (0.004694779606274441)  --> TIME: 2025-02-25 16:59:57 -- STEP: 17026/36938 -- GLOBAL_STEP: 866600 | > loss_text_ce: 0.03468532860279083 (0.03351744386580482) | > loss_mel_ce: 3.107130527496338 (nan) | > loss: 0.012467524036765099 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13667710910150932) | > loader_time: 0.0047 (0.004694773211793408)  --> TIME: 2025-02-25 17:00:07 -- STEP: 17076/36938 -- GLOBAL_STEP: 866650 | > loss_text_ce: 0.03657933697104454 (0.033514837721256914) | > loss_mel_ce: 3.033581495285034 (nan) | > loss: 0.012183179147541523 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13668857273661164) | > loader_time: 0.0047 (0.0046949538505448645)  --> TIME: 2025-02-25 17:00:16 -- STEP: 17126/36938 -- GLOBAL_STEP: 866700 | > loss_text_ce: 0.04116891324520111 (0.03351221105904297) | > loss_mel_ce: 2.3530099391937256 (nan) | > loss: 0.009500710293650627 (nan) | > current_lr: 5e-06 | > step_time: 0.1729 (0.1367006553857942) | > loader_time: 0.0049 (0.004695149597324768)  --> TIME: 2025-02-25 17:00:26 -- STEP: 17176/36938 -- GLOBAL_STEP: 866750 | > loss_text_ce: 0.03383754938840866 (0.03350719963789225) | > loss_mel_ce: 3.1957485675811768 (nan) | > loss: 0.01281581912189722 (nan) | > current_lr: 5e-06 | > step_time: 0.168 (0.13671630616792124) | > loader_time: 0.0048 (0.004695124538987515)  --> TIME: 2025-02-25 17:00:35 -- STEP: 17226/36938 -- GLOBAL_STEP: 866800 | > loss_text_ce: 0.030183618888258934 (0.03350674732854533) | > loss_mel_ce: 3.219724655151367 (nan) | > loss: 0.012896462343633175 (nan) | > current_lr: 5e-06 | > step_time: 0.1459 (0.13671122618174406) | > loader_time: 0.0046 (0.004695087820066595)  --> TIME: 2025-02-25 17:00:44 -- STEP: 17276/36938 -- GLOBAL_STEP: 866850 | > loss_text_ce: 0.030865399166941643 (0.0335055255646677) | > loss_mel_ce: 0.0033416748046875 (nan) | > loss: 0.0001357423752779141 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13671698794372636) | > loader_time: 0.0046 (0.004695273861616774)  --> TIME: 2025-02-25 17:00:53 -- STEP: 17326/36938 -- GLOBAL_STEP: 866900 | > loss_text_ce: 0.03235917165875435 (0.03350482272720828) | > loss_mel_ce: 2.7523891925811768 (nan) | > loss: 0.011050589382648468 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.13670811327833296) | > loader_time: 0.0047 (0.004695343624481784)  --> TIME: 2025-02-25 17:01:03 -- STEP: 17376/36938 -- GLOBAL_STEP: 866950 | > loss_text_ce: 0.04111641272902489 (0.033507621434225844) | > loss_mel_ce: 3.14890193939209 (nan) | > loss: 0.012658803723752499 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13670770796513218) | > loader_time: 0.0047 (0.004695305686519498)  --> TIME: 2025-02-25 17:01:13 -- STEP: 17426/36938 -- GLOBAL_STEP: 867000 | > loss_text_ce: 0.044317055493593216 (0.03350852069015346) | > loss_mel_ce: 2.508711338043213 (nan) | > loss: 0.010131065733730793 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13673242840744435) | > loader_time: 0.0046 (0.0046953829205242615)  --> TIME: 2025-02-25 17:01:22 -- STEP: 17476/36938 -- GLOBAL_STEP: 867050 | > loss_text_ce: 0.03722488880157471 (0.03351122382641349) | > loss_mel_ce: 2.132586717605591 (nan) | > loss: 0.00861036404967308 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13672801042536212) | > loader_time: 0.0046 (0.004695388238854021)  --> TIME: 2025-02-25 17:01:31 -- STEP: 17526/36938 -- GLOBAL_STEP: 867100 | > loss_text_ce: 0.032559897750616074 (0.033509700642348955) | > loss_mel_ce: 2.4571471214294434 (nan) | > loss: 0.009879790246486664 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.1367193498962967) | > loader_time: 0.0046 (0.004695248878615678)  --> TIME: 2025-02-25 17:01:40 -- STEP: 17576/36938 -- GLOBAL_STEP: 867150 | > loss_text_ce: 0.043403759598731995 (0.03350614192576574) | > loss_mel_ce: 2.643040657043457 (nan) | > loss: 0.010660494677722454 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.1367268565436844) | > loader_time: 0.0046 (0.004695269672991077)  --> TIME: 2025-02-25 17:01:49 -- STEP: 17626/36938 -- GLOBAL_STEP: 867200 | > loss_text_ce: 0.027108510956168175 (0.033506661565648806) | > loss_mel_ce: 1.9489611387252808 (nan) | > loss: 0.007841547019779682 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13671137368680378) | > loader_time: 0.0045 (0.004695064983911131)  --> TIME: 2025-02-25 17:01:59 -- STEP: 17676/36938 -- GLOBAL_STEP: 867250 | > loss_text_ce: 0.03837971016764641 (0.03350753877523724) | > loss_mel_ce: 3.0584943294525146 (nan) | > loss: 0.012289183214306831 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.13672828169855264) | > loader_time: 0.0047 (0.004695052797369714)  --> TIME: 2025-02-25 17:02:08 -- STEP: 17726/36938 -- GLOBAL_STEP: 867300 | > loss_text_ce: 0.03079662099480629 (0.033511575339951236) | > loss_mel_ce: 2.3424370288848877 (nan) | > loss: 0.009417594410479069 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13672412008057183) | > loader_time: 0.0048 (0.004694882047696867)  --> TIME: 2025-02-25 17:02:18 -- STEP: 17776/36938 -- GLOBAL_STEP: 867350 | > loss_text_ce: 0.026245879009366035 (0.03351076898070922) | > loss_mel_ce: 1.3177490234375 (nan) | > loss: 0.005333313252776861 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13671445091440254) | > loader_time: 0.0046 (0.004694642165456114)  --> TIME: 2025-02-25 17:02:27 -- STEP: 17826/36938 -- GLOBAL_STEP: 867400 | > loss_text_ce: 0.032522838562726974 (0.03351158789660666) | > loss_mel_ce: 3.085350751876831 (nan) | > loss: 0.0123725151643157 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13673406404022084) | > loader_time: 0.0046 (0.00469448901340907)  --> TIME: 2025-02-25 17:02:37 -- STEP: 17876/36938 -- GLOBAL_STEP: 867450 | > loss_text_ce: 0.022580362856388092 (0.03351074980164838) | > loss_mel_ce: 2.8345468044281006 (nan) | > loss: 0.011337807402014732 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.1367406087530942) | > loader_time: 0.0047 (0.004694356670793613)  --> TIME: 2025-02-25 17:02:46 -- STEP: 17926/36938 -- GLOBAL_STEP: 867500 | > loss_text_ce: 0.029593756422400475 (0.03351558983951659) | > loss_mel_ce: 2.675809860229492 (nan) | > loss: 0.010735728777945042 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.13672893350310297) | > loader_time: 0.0047 (0.004694273904619306)  --> TIME: 2025-02-25 17:02:55 -- STEP: 17976/36938 -- GLOBAL_STEP: 867550 | > loss_text_ce: 0.027871377766132355 (0.03351640729224267) | > loss_mel_ce: 1.8569438457489014 (nan) | > loss: 0.007479425985366106 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13671976558675125) | > loader_time: 0.0048 (0.0046943420429042875)  --> TIME: 2025-02-25 17:03:05 -- STEP: 18026/36938 -- GLOBAL_STEP: 867600 | > loss_text_ce: 0.037249233573675156 (0.033518067521152245) | > loss_mel_ce: 3.5206878185272217 (nan) | > loss: 0.01411879900842905 (nan) | > current_lr: 5e-06 | > step_time: 0.1815 (0.13673645150154912) | > loader_time: 0.0048 (0.004694295844769317)  --> TIME: 2025-02-25 17:03:14 -- STEP: 18076/36938 -- GLOBAL_STEP: 867650 | > loss_text_ce: 0.03405908867716789 (0.03352095391280824) | > loss_mel_ce: 2.638552188873291 (nan) | > loss: 0.010605600662529469 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13673894821898452) | > loader_time: 0.0045 (0.004694513275866534)  --> TIME: 2025-02-25 17:03:23 -- STEP: 18126/36938 -- GLOBAL_STEP: 867700 | > loss_text_ce: 0.03078106977045536 (0.03351980599125753) | > loss_mel_ce: 1.8267509937286377 (nan) | > loss: 0.007371159270405769 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13672530783941897) | > loader_time: 0.0045 (0.004694569627799086)  --> TIME: 2025-02-25 17:03:33 -- STEP: 18176/36938 -- GLOBAL_STEP: 867750 | > loss_text_ce: 0.03337056189775467 (0.03351746341819202) | > loss_mel_ce: 2.5019869804382324 (nan) | > loss: 0.010060942731797695 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13673484396241917) | > loader_time: 0.0045 (0.004694491637949367)  --> TIME: 2025-02-25 17:03:42 -- STEP: 18226/36938 -- GLOBAL_STEP: 867800 | > loss_text_ce: 0.039493005722761154 (0.033519897695845534) | > loss_mel_ce: 0.211273193359375 (nan) | > loss: 0.000995103968307376 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13673141400847807) | > loader_time: 0.0046 (0.004694270993461592)  --> TIME: 2025-02-25 17:03:51 -- STEP: 18276/36938 -- GLOBAL_STEP: 867850 | > loss_text_ce: 0.027840351685881615 (0.03352480540254843) | > loss_mel_ce: 2.8438937664031982 (nan) | > loss: 0.01139577105641365 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.1367276780687241) | > loader_time: 0.0046 (0.00469405546989866)  --> TIME: 2025-02-25 17:04:01 -- STEP: 18326/36938 -- GLOBAL_STEP: 867900 | > loss_text_ce: 0.03627653419971466 (0.03352525454238319) | > loss_mel_ce: 2.608483076095581 (nan) | > loss: 0.0104950787499547 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1367321741134724) | > loader_time: 0.0045 (0.004693945344327302)  --> TIME: 2025-02-25 17:04:10 -- STEP: 18376/36938 -- GLOBAL_STEP: 867950 | > loss_text_ce: 0.02781061828136444 (0.03352418929550477) | > loss_mel_ce: 1.7143206596374512 (nan) | > loss: 0.006913219578564167 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13673016017657225) | > loader_time: 0.0046 (0.004693769791046323)  --> TIME: 2025-02-25 17:04:19 -- STEP: 18426/36938 -- GLOBAL_STEP: 868000 | > loss_text_ce: 0.03692201152443886 (0.033525024189643864) | > loss_mel_ce: 3.1730806827545166 (nan) | > loss: 0.012738106772303581 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13673257171738915) | > loader_time: 0.0047 (0.004693646119392146)  --> TIME: 2025-02-25 17:04:29 -- STEP: 18476/36938 -- GLOBAL_STEP: 868050 | > loss_text_ce: 0.04318573325872421 (0.03352663476448728) | > loss_mel_ce: 2.654301643371582 (nan) | > loss: 0.010704315267503262 (nan) | > current_lr: 5e-06 | > step_time: 0.171 (0.13673139597900386) | > loader_time: 0.0048 (0.004693426180515584)  --> TIME: 2025-02-25 17:04:38 -- STEP: 18526/36938 -- GLOBAL_STEP: 868100 | > loss_text_ce: 0.03456415235996246 (0.033525339468511356) | > loss_mel_ce: 2.7845652103424072 (nan) | > loss: 0.011187022551894188 (nan) | > current_lr: 5e-06 | > step_time: 0.1905 (0.13674351092639894) | > loader_time: 0.0049 (0.004693281916939868)  --> TIME: 2025-02-25 17:04:47 -- STEP: 18576/36938 -- GLOBAL_STEP: 868150 | > loss_text_ce: 0.028903311118483543 (0.03352142637234521) | > loss_mel_ce: 3.304379463195801 (nan) | > loss: 0.013227312825620174 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1367304143106803) | > loader_time: 0.0046 (0.00469326748353052)  --> TIME: 2025-02-25 17:04:56 -- STEP: 18626/36938 -- GLOBAL_STEP: 868200 | > loss_text_ce: 0.032500553876161575 (0.03352002900423367) | > loss_mel_ce: 3.1440744400024414 (nan) | > loss: 0.01260545663535595 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.13672730870985714) | > loader_time: 0.0048 (0.00469321251222734)  --> TIME: 2025-02-25 17:05:06 -- STEP: 18676/36938 -- GLOBAL_STEP: 868250 | > loss_text_ce: 0.037008460611104965 (0.03351717258682432) | > loss_mel_ce: 2.8530898094177246 (nan) | > loss: 0.011468645185232162 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13672343933702302) | > loader_time: 0.0045 (0.004693224461233361)  --> TIME: 2025-02-25 17:05:15 -- STEP: 18726/36938 -- GLOBAL_STEP: 868300 | > loss_text_ce: 0.025615781545639038 (0.033517103320146256) | > loss_mel_ce: 0.339599609375 (nan) | > loss: 0.001449267496354878 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13670979224246477) | > loader_time: 0.0045 (0.004692980408681239)  --> TIME: 2025-02-25 17:05:24 -- STEP: 18776/36938 -- GLOBAL_STEP: 868350 | > loss_text_ce: 0.03353924676775932 (0.03351867248843427) | > loss_mel_ce: 2.589508056640625 (nan) | > loss: 0.010408918373286724 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13669914661136434) | > loader_time: 0.0045 (0.004692672680020575)  --> TIME: 2025-02-25 17:05:33 -- STEP: 18826/36938 -- GLOBAL_STEP: 868400 | > loss_text_ce: 0.03338169679045677 (0.033518054838093427) | > loss_mel_ce: 2.651353120803833 (nan) | > loss: 0.01065370999276638 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.13669392955704646) | > loader_time: 0.0046 (0.0046924456873267215)  --> TIME: 2025-02-25 17:05:43 -- STEP: 18876/36938 -- GLOBAL_STEP: 868450 | > loss_text_ce: 0.03183429315686226 (0.03351738641782934) | > loss_mel_ce: 2.5991764068603516 (nan) | > loss: 0.010440519079566002 (nan) | > current_lr: 5e-06 | > step_time: 0.1848 (0.1366971023230767) | > loader_time: 0.0048 (0.004692427294566019)  --> TIME: 2025-02-25 17:05:52 -- STEP: 18926/36938 -- GLOBAL_STEP: 868500 | > loss_text_ce: 0.033243849873542786 (0.033519459196662216) | > loss_mel_ce: 2.845770835876465 (nan) | > loss: 0.011424662545323372 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.13669682058007634) | > loader_time: 0.0046 (0.004692472212790525)  --> TIME: 2025-02-25 17:06:01 -- STEP: 18976/36938 -- GLOBAL_STEP: 868550 | > loss_text_ce: 0.023186538368463516 (0.03352009788317291) | > loss_mel_ce: 2.628720283508301 (nan) | > loss: 0.010523440316319466 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.1367026808315145) | > loader_time: 0.0046 (0.004692597154521611)  --> TIME: 2025-02-25 17:06:11 -- STEP: 19026/36938 -- GLOBAL_STEP: 868600 | > loss_text_ce: 0.02425926923751831 (0.03351912330544285) | > loss_mel_ce: 2.78470778465271 (nan) | > loss: 0.011146695353090763 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.1367088844256611) | > loader_time: 0.0046 (0.004692584135223349)  --> TIME: 2025-02-25 17:06:20 -- STEP: 19076/36938 -- GLOBAL_STEP: 868650 | > loss_text_ce: 0.03747791051864624 (0.0335200127872502) | > loss_mel_ce: 2.8984375 (nan) | > loss: 0.01165045890957117 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13669960021122776) | > loader_time: 0.0045 (0.00469236263665894)  --> TIME: 2025-02-25 17:06:29 -- STEP: 19126/36938 -- GLOBAL_STEP: 868700 | > loss_text_ce: 0.04112948849797249 (0.033517601755862735) | > loss_mel_ce: 2.714080333709717 (nan) | > loss: 0.010933373123407364 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.1366953443037372) | > loader_time: 0.0047 (0.00469237178934925)  --> TIME: 2025-02-25 17:06:38 -- STEP: 19176/36938 -- GLOBAL_STEP: 868750 | > loss_text_ce: 0.028679238632321358 (0.03351734696637184) | > loss_mel_ce: 2.846050500869751 (nan) | > loss: 0.011407657526433468 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13668455512363534) | > loader_time: 0.0052 (0.00469220526326831)  --> TIME: 2025-02-25 17:06:47 -- STEP: 19226/36938 -- GLOBAL_STEP: 868800 | > loss_text_ce: 0.03776064142584801 (0.03351355476310086) | > loss_mel_ce: 2.421328544616699 (nan) | > loss: 0.009758291766047478 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13666699751847317) | > loader_time: 0.0045 (0.004692101210718481)  --> TIME: 2025-02-25 17:06:57 -- STEP: 19276/36938 -- GLOBAL_STEP: 868850 | > loss_text_ce: 0.034154899418354034 (0.03351368533683513) | > loss_mel_ce: 1.6965781450271606 (nan) | > loss: 0.006867988500744104 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13667675090246786) | > loader_time: 0.0046 (0.004691947518257853)  --> TIME: 2025-02-25 17:07:06 -- STEP: 19326/36938 -- GLOBAL_STEP: 868900 | > loss_text_ce: 0.022938085719943047 (0.03351389276322308) | > loss_mel_ce: 2.5633764266967773 (nan) | > loss: 0.010263153351843357 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1366772904080735) | > loader_time: 0.0045 (0.004691794645733235)  --> TIME: 2025-02-25 17:07:15 -- STEP: 19376/36938 -- GLOBAL_STEP: 868950 | > loss_text_ce: 0.04183207079768181 (0.03351362974384435) | > loss_mel_ce: 1.3455698490142822 (nan) | > loss: 0.0055055636912584305 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13667362562936794) | > loader_time: 0.0045 (0.00469155699433028)  --> TIME: 2025-02-25 17:07:25 -- STEP: 19426/36938 -- GLOBAL_STEP: 869000 | > loss_text_ce: 0.03222086653113365 (0.033511166051578216) | > loss_mel_ce: 2.6893417835235596 (nan) | > loss: 0.010799852199852467 (nan) | > current_lr: 5e-06 | > step_time: 0.1271 (0.13668215822194166) | > loader_time: 0.0046 (0.004691337625999764)  --> TIME: 2025-02-25 17:07:34 -- STEP: 19476/36938 -- GLOBAL_STEP: 869050 | > loss_text_ce: 0.031070005148649216 (0.033511267163866695) | > loss_mel_ce: 2.6638896465301514 (nan) | > loss: 0.010694284923374653 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.1366889838394777) | > loader_time: 0.0045 (0.0046911426064512205)  --> TIME: 2025-02-25 17:07:44 -- STEP: 19526/36938 -- GLOBAL_STEP: 869100 | > loss_text_ce: 0.036008115857839584 (0.033510001042681294) | > loss_mel_ce: 1.402746558189392 (nan) | > loss: 0.005709344521164894 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13668523820882778) | > loader_time: 0.0045 (0.004690897693085057)  --> TIME: 2025-02-25 17:07:53 -- STEP: 19576/36938 -- GLOBAL_STEP: 869150 | > loss_text_ce: 0.02162293903529644 (0.033513033766855556) | > loss_mel_ce: 2.3206424713134766 (nan) | > loss: 0.009294704534113407 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.1366746477268162) | > loader_time: 0.0046 (0.004690871233349681)  --> TIME: 2025-02-25 17:08:02 -- STEP: 19626/36938 -- GLOBAL_STEP: 869200 | > loss_text_ce: 0.030588623136281967 (0.03351031929874431) | > loss_mel_ce: 2.9749672412872314 (nan) | > loss: 0.011926809325814247 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.1366836368812892) | > loader_time: 0.0046 (0.004690733911259076)  --> TIME: 2025-02-25 17:08:11 -- STEP: 19676/36938 -- GLOBAL_STEP: 869250 | > loss_text_ce: 0.036487728357315063 (0.03350841234229636) | > loss_mel_ce: 2.822322130203247 (nan) | > loss: 0.011344484984874725 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.1366799543678922) | > loader_time: 0.0047 (0.004690763692977977)  --> TIME: 2025-02-25 17:08:20 -- STEP: 19726/36938 -- GLOBAL_STEP: 869300 | > loss_text_ce: 0.032512035220861435 (0.03351181396219443) | > loss_mel_ce: 2.9094841480255127 (nan) | > loss: 0.011674588546156883 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13666888331467741) | > loader_time: 0.0047 (0.004690922407691551)  --> TIME: 2025-02-25 17:08:30 -- STEP: 19776/36938 -- GLOBAL_STEP: 869350 | > loss_text_ce: 0.0221087783575058 (0.03351192305717457) | > loss_mel_ce: 2.0679092407226562 (nan) | > loss: 0.00829372275620699 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13667109526669738) | > loader_time: 0.0045 (0.004690910632266963)  --> TIME: 2025-02-25 17:08:39 -- STEP: 19826/36938 -- GLOBAL_STEP: 869400 | > loss_text_ce: 0.026932332664728165 (0.03351180463661752) | > loss_mel_ce: 1.6874923706054688 (nan) | > loss: 0.006803273223340511 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.1366722002816375) | > loader_time: 0.0047 (0.004690851403283239)  --> TIME: 2025-02-25 17:08:48 -- STEP: 19876/36938 -- GLOBAL_STEP: 869450 | > loss_text_ce: 0.026141613721847534 (0.03351334022411637) | > loss_mel_ce: 3.5961215496063232 (nan) | > loss: 0.014374061487615108 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13666998493189242) | > loader_time: 0.0046 (0.004690954744707866)  --> TIME: 2025-02-25 17:08:58 -- STEP: 19926/36938 -- GLOBAL_STEP: 869500 | > loss_text_ce: 0.03358965367078781 (0.033510788443494206) | > loss_mel_ce: 2.510964870452881 (nan) | > loss: 0.010097439400851727 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13667285335862037) | > loader_time: 0.0046 (0.0046910717941294885)  --> TIME: 2025-02-25 17:09:07 -- STEP: 19976/36938 -- GLOBAL_STEP: 869550 | > loss_text_ce: 0.030467087402939796 (0.03351129759481606) | > loss_mel_ce: 1.5291231870651245 (nan) | > loss: 0.00618885038420558 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13668101924536102) | > loader_time: 0.0046 (0.004691027418154925)  --> TIME: 2025-02-25 17:09:17 -- STEP: 20026/36938 -- GLOBAL_STEP: 869600 | > loss_text_ce: 0.01489650085568428 (0.03351117976152683) | > loss_mel_ce: 0.025917870923876762 (nan) | > loss: 0.00016196179785765707 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.1366770209149861) | > loader_time: 0.0046 (0.0046910637446267904)  --> TIME: 2025-02-25 17:09:26 -- STEP: 20076/36938 -- GLOBAL_STEP: 869650 | > loss_text_ce: 0.03636136278510094 (0.03351216260605397) | > loss_mel_ce: 3.4970486164093018 (nan) | > loss: 0.01402146928012371 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13668418739345084) | > loader_time: 0.0046 (0.004691010073471594)  --> TIME: 2025-02-25 17:09:35 -- STEP: 20126/36938 -- GLOBAL_STEP: 869700 | > loss_text_ce: 0.03124920278787613 (0.03351044078042539) | > loss_mel_ce: 2.6587536334991455 (nan) | > loss: 0.010674615390598774 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.13667917906687163) | > loader_time: 0.005 (0.004691008626851999)  --> TIME: 2025-02-25 17:09:45 -- STEP: 20176/36938 -- GLOBAL_STEP: 869750 | > loss_text_ce: 0.036672234535217285 (0.03351129900496403) | > loss_mel_ce: 3.2217867374420166 (nan) | > loss: 0.012930394150316715 (nan) | > current_lr: 5e-06 | > step_time: 0.1857 (0.13669067037474264) | > loader_time: 0.0049 (0.00469093714839971)  --> TIME: 2025-02-25 17:09:54 -- STEP: 20226/36938 -- GLOBAL_STEP: 869800 | > loss_text_ce: 0.027911843731999397 (0.03350875812693976) | > loss_mel_ce: 2.6194348335266113 (nan) | > loss: 0.010505344718694687 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.136687397886122) | > loader_time: 0.0046 (0.004690787481717062)  --> TIME: 2025-02-25 17:10:03 -- STEP: 20276/36938 -- GLOBAL_STEP: 869850 | > loss_text_ce: 0.03550656884908676 (0.03351107890941617) | > loss_mel_ce: 2.104001522064209 (nan) | > loss: 0.00849011167883873 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13668252086470098) | > loader_time: 0.0046 (0.004690673170674827)  --> TIME: 2025-02-25 17:10:13 -- STEP: 20326/36938 -- GLOBAL_STEP: 869900 | > loss_text_ce: 0.036915265023708344 (0.03351022250705443) | > loss_mel_ce: 0.10948486626148224 (nan) | > loss: 0.0005809529102407396 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13667869658190054) | > loader_time: 0.0067 (0.004690617836097623)  --> TIME: 2025-02-25 17:10:22 -- STEP: 20376/36938 -- GLOBAL_STEP: 869950 | > loss_text_ce: 0.04457993805408478 (0.033509167103212764) | > loss_mel_ce: 2.8080978393554688 (nan) | > loss: 0.011320150457322598 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13668022979040642) | > loader_time: 0.0052 (0.004690604323165105)  --> TIME: 2025-02-25 17:10:31 -- STEP: 20426/36938 -- GLOBAL_STEP: 870000 | > loss_text_ce: 0.04942620173096657 (0.03350743103876689) | > loss_mel_ce: 1.5627790689468384 (nan) | > loss: 0.006397640332579613 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13666901049879615) | > loader_time: 0.0044 (0.004690736675197086) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_870000.pth  --> TIME: 2025-02-25 17:10:47 -- STEP: 20476/36938 -- GLOBAL_STEP: 870050 | > loss_text_ce: 0.03313043341040611 (0.03350789730643926) | > loss_mel_ce: 2.997093677520752 (nan) | > loss: 0.012024699710309505 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.1366800162027907) | > loader_time: 0.0048 (0.004690792405016139)  --> TIME: 2025-02-25 17:10:56 -- STEP: 20526/36938 -- GLOBAL_STEP: 870100 | > loss_text_ce: 0.03971828147768974 (0.03350614464584849) | > loss_mel_ce: 3.0743305683135986 (nan) | > loss: 0.012357337400317192 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.13666889284898315) | > loader_time: 0.0047 (0.004690958198418015)  --> TIME: 2025-02-25 17:11:05 -- STEP: 20576/36938 -- GLOBAL_STEP: 870150 | > loss_text_ce: 0.048293787986040115 (0.033507122009172816) | > loss_mel_ce: 2.0278124809265137 (nan) | > loss: 0.008238517679274082 (nan) | > current_lr: 5e-06 | > step_time: 0.1076 (0.13665775305839198) | > loader_time: 0.0045 (0.004690934627274892)  --> TIME: 2025-02-25 17:11:14 -- STEP: 20626/36938 -- GLOBAL_STEP: 870200 | > loss_text_ce: 0.0324249304831028 (0.03350774067380228) | > loss_mel_ce: 1.984099268913269 (nan) | > loss: 0.008002080954611301 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.1366476771395271) | > loader_time: 0.0046 (0.004690946159891743)  --> TIME: 2025-02-25 17:11:23 -- STEP: 20676/36938 -- GLOBAL_STEP: 870250 | > loss_text_ce: 0.02585858292877674 (0.033510352606364285) | > loss_mel_ce: 0.7118170857429504 (nan) | > loss: 0.0029272844549268484 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1366362875332152) | > loader_time: 0.0045 (0.0046908166911759355)  --> TIME: 2025-02-25 17:11:33 -- STEP: 20726/36938 -- GLOBAL_STEP: 870300 | > loss_text_ce: 0.023935982957482338 (0.03351019003388852) | > loss_mel_ce: 2.9696338176727295 (nan) | > loss: 0.01187924575060606 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13664854684069752) | > loader_time: 0.0045 (0.004690733860556623)  --> TIME: 2025-02-25 17:11:42 -- STEP: 20776/36938 -- GLOBAL_STEP: 870350 | > loss_text_ce: 0.04044259339570999 (0.03350895822420573) | > loss_mel_ce: 2.8241019248962402 (nan) | > loss: 0.01136724092066288 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13663603650591769) | > loader_time: 0.0053 (0.004690635075787655)  --> TIME: 2025-02-25 17:11:51 -- STEP: 20826/36938 -- GLOBAL_STEP: 870400 | > loss_text_ce: 0.018725259229540825 (0.0335076175258741) | > loss_mel_ce: 2.442793846130371 (nan) | > loss: 0.009767932817339897 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13663780018379576) | > loader_time: 0.0045 (0.004690532220448296)  --> TIME: 2025-02-25 17:12:01 -- STEP: 20876/36938 -- GLOBAL_STEP: 870450 | > loss_text_ce: 0.03822477161884308 (0.03350950388361288) | > loss_mel_ce: 3.048574686050415 (nan) | > loss: 0.012249204330146313 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13664503373001513) | > loader_time: 0.0046 (0.004690353967239407)  --> TIME: 2025-02-25 17:12:10 -- STEP: 20926/36938 -- GLOBAL_STEP: 870500 | > loss_text_ce: 0.023971574380993843 (0.03350796521218746) | > loss_mel_ce: 2.4839601516723633 (nan) | > loss: 0.00995211023837328 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13665035039166543) | > loader_time: 0.0045 (0.004690161230321898)  --> TIME: 2025-02-25 17:12:19 -- STEP: 20976/36938 -- GLOBAL_STEP: 870550 | > loss_text_ce: 0.04054208844900131 (0.03350988539473156) | > loss_mel_ce: 2.7759127616882324 (nan) | > loss: 0.01117640919983387 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.1366521002885317) | > loader_time: 0.0046 (0.004689966923039349)  --> TIME: 2025-02-25 17:12:29 -- STEP: 21026/36938 -- GLOBAL_STEP: 870600 | > loss_text_ce: 0.035418801009655 (0.0335080348281049) | > loss_mel_ce: 3.6304423809051514 (nan) | > loss: 0.014547068625688553 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.13665373274548198) | > loader_time: 0.0048 (0.00468973979434697)  --> TIME: 2025-02-25 17:12:38 -- STEP: 21076/36938 -- GLOBAL_STEP: 870650 | > loss_text_ce: 0.043090056627988815 (0.03350742706024273) | > loss_mel_ce: 0.022979736328125 (nan) | > loss: 0.00026218174025416374 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13665388709206086) | > loader_time: 0.0046 (0.004690134009457019)  --> TIME: 2025-02-25 17:12:48 -- STEP: 21126/36938 -- GLOBAL_STEP: 870700 | > loss_text_ce: 0.03332233428955078 (0.033511186424248496) | > loss_mel_ce: 0.10371752083301544 (nan) | > loss: 0.0005438089719973505 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13666183578576496) | > loader_time: 0.0045 (0.0046901453695863335)  --> TIME: 2025-02-25 17:12:57 -- STEP: 21176/36938 -- GLOBAL_STEP: 870750 | > loss_text_ce: 0.03857814520597458 (0.0335116875113393) | > loss_mel_ce: 1.5425889492034912 (nan) | > loss: 0.006274472922086716 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.1366639212815326) | > loader_time: 0.0046 (0.00468998166764508)  --> TIME: 2025-02-25 17:13:06 -- STEP: 21226/36938 -- GLOBAL_STEP: 870800 | > loss_text_ce: 0.04677506163716316 (0.033510054662787496) | > loss_mel_ce: 2.4284377098083496 (nan) | > loss: 0.009822273626923561 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13665303344451638) | > loader_time: 0.0047 (0.004689904484955075)  --> TIME: 2025-02-25 17:13:15 -- STEP: 21276/36938 -- GLOBAL_STEP: 870850 | > loss_text_ce: 0.019589735195040703 (0.03350998038736966) | > loss_mel_ce: 3.0659427642822266 (nan) | > loss: 0.012244177050888538 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13664852109343617) | > loader_time: 0.0046 (0.004689852833680632)  --> TIME: 2025-02-25 17:13:24 -- STEP: 21326/36938 -- GLOBAL_STEP: 870900 | > loss_text_ce: 0.03874059394001961 (0.03351239093334669) | > loss_mel_ce: 2.93088436126709 (nan) | > loss: 0.011784226633608341 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.13664102487243057) | > loader_time: 0.0046 (0.004689771574768624)  --> TIME: 2025-02-25 17:13:34 -- STEP: 21376/36938 -- GLOBAL_STEP: 870950 | > loss_text_ce: 0.03582600876688957 (0.033511327319207536) | > loss_mel_ce: 2.3109958171844482 (nan) | > loss: 0.009312785230576992 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.13664542157537607) | > loader_time: 0.0053 (0.004689713527342518)  --> TIME: 2025-02-25 17:13:44 -- STEP: 21426/36938 -- GLOBAL_STEP: 871000 | > loss_text_ce: 0.03383450210094452 (0.03351416164146026) | > loss_mel_ce: 3.2280449867248535 (nan) | > loss: 0.012943967245519161 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1366643570033176) | > loader_time: 0.0045 (0.004689807330127706)  --> TIME: 2025-02-25 17:13:53 -- STEP: 21476/36938 -- GLOBAL_STEP: 871050 | > loss_text_ce: 0.03581387549638748 (0.033512536465504145) | > loss_mel_ce: 3.0840044021606445 (nan) | > loss: 0.012380232103168964 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.1366570333369644) | > loader_time: 0.0046 (0.0046896733680813225)  --> TIME: 2025-02-25 17:14:02 -- STEP: 21526/36938 -- GLOBAL_STEP: 871100 | > loss_text_ce: 0.03247654438018799 (0.03351383657461758) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1366533876783421) | > loader_time: 0.0047 (0.004689533338552962)  --> TIME: 2025-02-25 17:14:11 -- STEP: 21576/36938 -- GLOBAL_STEP: 871150 | > loss_text_ce: 0.033464401960372925 (0.033514151760807576) | > loss_mel_ce: 3.1353113651275635 (nan) | > loss: 0.012574507854878902 (nan) | > current_lr: 5e-06 | > step_time: 0.1599 (0.1366584791954157) | > loader_time: 0.0049 (0.00468961311615438)  --> TIME: 2025-02-25 17:14:21 -- STEP: 21626/36938 -- GLOBAL_STEP: 871200 | > loss_text_ce: 0.03425999730825424 (0.03351161617269395) | > loss_mel_ce: 2.54476261138916 (nan) | > loss: 0.010234217159450054 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13665647878464302) | > loader_time: 0.0045 (0.004689492383784225)  --> TIME: 2025-02-25 17:14:30 -- STEP: 21676/36938 -- GLOBAL_STEP: 871250 | > loss_text_ce: 0.029338886961340904 (0.03351113870541401) | > loss_mel_ce: 3.1449766159057617 (nan) | > loss: 0.012596490792930126 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.1366686651046305) | > loader_time: 0.0047 (0.004689462159827839)  --> TIME: 2025-02-25 17:14:40 -- STEP: 21726/36938 -- GLOBAL_STEP: 871300 | > loss_text_ce: 0.03715991973876953 (0.03350866925504401) | > loss_mel_ce: 3.138575315475464 (nan) | > loss: 0.012602124363183975 (nan) | > current_lr: 5e-06 | > step_time: 0.147 (0.13667317601316037) | > loader_time: 0.0046 (0.004689360338713398)  --> TIME: 2025-02-25 17:14:49 -- STEP: 21776/36938 -- GLOBAL_STEP: 871350 | > loss_text_ce: 0.0332302451133728 (0.03351083499042242) | > loss_mel_ce: 2.9499223232269287 (nan) | > loss: 0.011837908066809177 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1366762168086864) | > loader_time: 0.0046 (0.004689341078428202)  --> TIME: 2025-02-25 17:14:58 -- STEP: 21826/36938 -- GLOBAL_STEP: 871400 | > loss_text_ce: 0.03426875174045563 (0.03351248411127491) | > loss_mel_ce: 2.831422805786133 (nan) | > loss: 0.0113717932254076 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13666765894995136) | > loader_time: 0.0045 (0.004689275601230688)  --> TIME: 2025-02-25 17:15:07 -- STEP: 21876/36938 -- GLOBAL_STEP: 871450 | > loss_text_ce: 0.037510745227336884 (0.033513935969142254) | > loss_mel_ce: 2.4315378665924072 (nan) | > loss: 0.009797812439501286 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.1366664986109948) | > loader_time: 0.0047 (0.004689364486473923)  --> TIME: 2025-02-25 17:15:17 -- STEP: 21926/36938 -- GLOBAL_STEP: 871500 | > loss_text_ce: 0.030328327789902687 (0.03351257363234709) | > loss_mel_ce: 3.1957907676696777 (nan) | > loss: 0.012802060693502426 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.1366639367901018) | > loader_time: 0.0047 (0.004689361724411603)  --> TIME: 2025-02-25 17:15:26 -- STEP: 21976/36938 -- GLOBAL_STEP: 871550 | > loss_text_ce: 0.040041275322437286 (0.033516322252032486) | > loss_mel_ce: 2.8018038272857666 (nan) | > loss: 0.01127716340124607 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13666040065464743) | > loader_time: 0.0046 (0.004689212751510056)  --> TIME: 2025-02-25 17:15:35 -- STEP: 22026/36938 -- GLOBAL_STEP: 871600 | > loss_text_ce: 0.03716081380844116 (0.0335140079500459) | > loss_mel_ce: 2.857785940170288 (nan) | > loss: 0.011487884446978569 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.13665193198628456) | > loader_time: 0.0046 (0.00468911229887591)  --> TIME: 2025-02-25 17:15:45 -- STEP: 22076/36938 -- GLOBAL_STEP: 871650 | > loss_text_ce: 0.027315758168697357 (0.03351592662139063) | > loss_mel_ce: 2.8845503330230713 (nan) | > loss: 0.011555025354027748 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13667275995680314) | > loader_time: 0.0048 (0.0046892142053919575)  --> TIME: 2025-02-25 17:15:54 -- STEP: 22126/36938 -- GLOBAL_STEP: 871700 | > loss_text_ce: 0.03161781281232834 (0.03351843666262855) | > loss_mel_ce: 2.8296847343444824 (nan) | > loss: 0.011354376561939716 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13667179684111852) | > loader_time: 0.0046 (0.004689168393434103)  --> TIME: 2025-02-25 17:16:03 -- STEP: 22176/36938 -- GLOBAL_STEP: 871750 | > loss_text_ce: 0.03939991444349289 (0.03351958405027711) | > loss_mel_ce: 2.793173313140869 (nan) | > loss: 0.011240370571613312 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.1366703732619211) | > loader_time: 0.0047 (0.0046890225761380445)  --> TIME: 2025-02-25 17:16:12 -- STEP: 22226/36938 -- GLOBAL_STEP: 871800 | > loss_text_ce: 0.026944750919938087 (0.033516427696840025) | > loss_mel_ce: 2.1536076068878174 (nan) | > loss: 0.008652986027300358 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13665813999816218) | > loader_time: 0.0044 (0.004688921932010028)  --> TIME: 2025-02-25 17:16:22 -- STEP: 22276/36938 -- GLOBAL_STEP: 871850 | > loss_text_ce: 0.02322300150990486 (0.03351148265097172) | > loss_mel_ce: 2.6859331130981445 (nan) | > loss: 0.010750619694590569 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13665922554161328) | > loader_time: 0.0045 (0.004688902439816792)  --> TIME: 2025-02-25 17:16:31 -- STEP: 22326/36938 -- GLOBAL_STEP: 871900 | > loss_text_ce: 0.028991639614105225 (0.03351429943416599) | > loss_mel_ce: 2.8109257221221924 (nan) | > loss: 0.011269514448940754 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.1366494032448789) | > loader_time: 0.0045 (0.00468882337155162)  --> TIME: 2025-02-25 17:16:40 -- STEP: 22376/36938 -- GLOBAL_STEP: 871950 | > loss_text_ce: 0.03699081018567085 (0.033515202936632515) | > loss_mel_ce: 2.9233386516571045 (nan) | > loss: 0.011747339740395546 (nan) | > current_lr: 5e-06 | > step_time: 0.1692 (0.13665175008526592) | > loader_time: 0.0047 (0.004688706213037664)  --> TIME: 2025-02-25 17:16:49 -- STEP: 22426/36938 -- GLOBAL_STEP: 872000 | > loss_text_ce: 0.03620890900492668 (0.033516594777304544) | > loss_mel_ce: 2.9120535850524902 (nan) | > loss: 0.011699454858899117 (nan) | > current_lr: 5e-06 | > step_time: 0.1077 (0.13664511571993168) | > loader_time: 0.0045 (0.004688595934491673)  --> TIME: 2025-02-25 17:16:59 -- STEP: 22476/36938 -- GLOBAL_STEP: 872050 | > loss_text_ce: 0.03559814393520355 (0.033514568003725945) | > loss_mel_ce: 2.7409400939941406 (nan) | > loss: 0.011018008925020695 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13664112392959474) | > loader_time: 0.0046 (0.004688576683282718)  --> TIME: 2025-02-25 17:17:08 -- STEP: 22526/36938 -- GLOBAL_STEP: 872100 | > loss_text_ce: 0.02591165527701378 (0.033515074058062415) | > loss_mel_ce: 2.4560906887054443 (nan) | > loss: 0.009849215857684612 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13663541922441824) | > loader_time: 0.0046 (0.004688520166068048)  --> TIME: 2025-02-25 17:17:17 -- STEP: 22576/36938 -- GLOBAL_STEP: 872150 | > loss_text_ce: 0.0327596478164196 (0.03351833578875668) | > loss_mel_ce: 3.2516069412231445 (nan) | > loss: 0.013033201918005943 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.1366386214504842) | > loader_time: 0.0047 (0.00468844201740518)  --> TIME: 2025-02-25 17:17:26 -- STEP: 22626/36938 -- GLOBAL_STEP: 872200 | > loss_text_ce: 0.027084769681096077 (0.033517026724678024) | > loss_mel_ce: 3.103649139404297 (nan) | > loss: 0.01242354791611433 (nan) | > current_lr: 5e-06 | > step_time: 0.1537 (0.13663210835537334) | > loader_time: 0.0047 (0.004688434150673533)  --> TIME: 2025-02-25 17:17:36 -- STEP: 22676/36938 -- GLOBAL_STEP: 872250 | > loss_text_ce: 0.02931911125779152 (0.033519541596821696) | > loss_mel_ce: 2.1125779151916504 (nan) | > loss: 0.00849959161132574 (nan) | > current_lr: 5e-06 | > step_time: 0.1741 (0.13662702638986418) | > loader_time: 0.0048 (0.004688550690364773)  --> TIME: 2025-02-25 17:17:45 -- STEP: 22726/36938 -- GLOBAL_STEP: 872300 | > loss_text_ce: 0.022613219916820526 (0.03351990023868322) | > loss_mel_ce: 2.2674238681793213 (nan) | > loss: 0.009087449871003628 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.1366294369483132) | > loader_time: 0.0046 (0.004688527491125609)  --> TIME: 2025-02-25 17:17:54 -- STEP: 22776/36938 -- GLOBAL_STEP: 872350 | > loss_text_ce: 0.03227260708808899 (0.03352322937524186) | > loss_mel_ce: 2.9816012382507324 (nan) | > loss: 0.011959817260503769 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13662754889534418) | > loader_time: 0.0046 (0.004688387079158484)  --> TIME: 2025-02-25 17:18:04 -- STEP: 22826/36938 -- GLOBAL_STEP: 872400 | > loss_text_ce: 0.04163092374801636 (0.03352409954841548) | > loss_mel_ce: 2.9309563636779785 (nan) | > loss: 0.011795981787145138 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13663149009523406) | > loader_time: 0.0046 (0.00468831767148476)  --> TIME: 2025-02-25 17:18:13 -- STEP: 22876/36938 -- GLOBAL_STEP: 872450 | > loss_text_ce: 0.04126528650522232 (0.03352493959340424) | > loss_mel_ce: 3.515246868133545 (nan) | > loss: 0.014113144017755985 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13664486698388079) | > loader_time: 0.0048 (0.004688339521885319)  --> TIME: 2025-02-25 17:18:23 -- STEP: 22926/36938 -- GLOBAL_STEP: 872500 | > loss_text_ce: 0.03224189206957817 (0.033521792978091304) | > loss_mel_ce: 2.4275426864624023 (nan) | > loss: 0.009761050343513489 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13664170207127024) | > loader_time: 0.0045 (0.004688075790332042)  --> TIME: 2025-02-25 17:18:32 -- STEP: 22976/36938 -- GLOBAL_STEP: 872550 | > loss_text_ce: 0.02539605088531971 (0.03351974753410145) | > loss_mel_ce: 2.3856041431427 (nan) | > loss: 0.009567461907863617 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.1366407023664968) | > loader_time: 0.0046 (0.004687861832569548)  --> TIME: 2025-02-25 17:18:41 -- STEP: 23026/36938 -- GLOBAL_STEP: 872600 | > loss_text_ce: 0.01775851845741272 (0.03351871429957977) | > loss_mel_ce: 1.330531358718872 (nan) | > loss: 0.005350356921553612 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1366446178918045) | > loader_time: 0.0045 (0.004687660959979816)  --> TIME: 2025-02-25 17:18:50 -- STEP: 23076/36938 -- GLOBAL_STEP: 872650 | > loss_text_ce: 0.03692943602800369 (0.0335222991451139) | > loss_mel_ce: 1.320275902748108 (nan) | > loss: 0.0053857360035181046 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13663609030135537) | > loader_time: 0.0045 (0.004687484049635851)  --> TIME: 2025-02-25 17:19:00 -- STEP: 23126/36938 -- GLOBAL_STEP: 872700 | > loss_text_ce: 0.03390942141413689 (0.033523665376345324) | > loss_mel_ce: 2.272214889526367 (nan) | > loss: 0.009151287376880646 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13663342355145677) | > loader_time: 0.0046 (0.004687292573980431)  --> TIME: 2025-02-25 17:19:09 -- STEP: 23176/36938 -- GLOBAL_STEP: 872750 | > loss_text_ce: 0.0370819978415966 (0.03352174549872272) | > loss_mel_ce: 2.721273422241211 (nan) | > loss: 0.010945855639874935 (nan) | > current_lr: 5e-06 | > step_time: 0.1831 (0.13664035989285966) | > loader_time: 0.0049 (0.004687188656773663)  --> TIME: 2025-02-25 17:19:19 -- STEP: 23226/36938 -- GLOBAL_STEP: 872800 | > loss_text_ce: 0.040252044796943665 (0.03352114538479506) | > loss_mel_ce: 3.429778575897217 (nan) | > loss: 0.013769962824881077 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13664321984409533) | > loader_time: 0.0066 (0.0046871668052623746)  --> TIME: 2025-02-25 17:19:28 -- STEP: 23276/36938 -- GLOBAL_STEP: 872850 | > loss_text_ce: 0.033354490995407104 (0.03352408228038887) | > loss_mel_ce: 1.765625 (nan) | > loss: 0.007138808257877827 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13665094680240344) | > loader_time: 0.0045 (0.00468725450548194)  --> TIME: 2025-02-25 17:19:38 -- STEP: 23326/36938 -- GLOBAL_STEP: 872900 | > loss_text_ce: 0.023137444630265236 (0.033526452796157034) | > loss_mel_ce: 2.5139055252075195 (nan) | > loss: 0.010067630559206009 (nan) | > current_lr: 5e-06 | > step_time: 0.1268 (0.13665721360870295) | > loader_time: 0.0046 (0.004687270056799659)  --> TIME: 2025-02-25 17:19:47 -- STEP: 23376/36938 -- GLOBAL_STEP: 872950 | > loss_text_ce: 0.027049846947193146 (0.03352410193332044) | > loss_mel_ce: 2.8636372089385986 (nan) | > loss: 0.011470980942249298 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13666378343088978) | > loader_time: 0.0047 (0.004687334181999651)  --> TIME: 2025-02-25 17:19:57 -- STEP: 23426/36938 -- GLOBAL_STEP: 873000 | > loss_text_ce: 0.04412934184074402 (0.033523501139461544) | > loss_mel_ce: 0.11931152641773224 (nan) | > loss: 0.0006485749036073685 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.1366654318260165) | > loader_time: 0.0046 (0.004687224985095122)  --> TIME: 2025-02-25 17:20:06 -- STEP: 23476/36938 -- GLOBAL_STEP: 873050 | > loss_text_ce: 0.03333287686109543 (0.033525415901898975) | > loss_mel_ce: 2.990589141845703 (nan) | > loss: 0.011999690905213356 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13666477672975316) | > loader_time: 0.0046 (0.004687230100337558)  --> TIME: 2025-02-25 17:20:15 -- STEP: 23526/36938 -- GLOBAL_STEP: 873100 | > loss_text_ce: 0.02414081245660782 (0.033522998259552135) | > loss_mel_ce: 1.485789179801941 (nan) | > loss: 0.005991786252707243 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13666504353454564) | > loader_time: 0.0045 (0.004687291043738849)  --> TIME: 2025-02-25 17:20:25 -- STEP: 23576/36938 -- GLOBAL_STEP: 873150 | > loss_text_ce: 0.03610832989215851 (0.033524715528041996) | > loss_mel_ce: 2.522473096847534 (nan) | > loss: 0.010153100825846195 (nan) | > current_lr: 5e-06 | > step_time: 0.1807 (0.13667444724489689) | > loader_time: 0.0048 (0.004687303288491771)  --> TIME: 2025-02-25 17:20:34 -- STEP: 23626/36938 -- GLOBAL_STEP: 873200 | > loss_text_ce: 0.033540111035108566 (0.033525439687516076) | > loss_mel_ce: 2.8588900566101074 (nan) | > loss: 0.011477897875010967 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.1366813625236229) | > loader_time: 0.0049 (0.00468752128470609)  --> TIME: 2025-02-25 17:20:44 -- STEP: 23676/36938 -- GLOBAL_STEP: 873250 | > loss_text_ce: 0.03253483027219772 (0.03352239217221606) | > loss_mel_ce: 0.517134428024292 (nan) | > loss: 0.002181227318942547 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13668433278371037) | > loader_time: 0.0045 (0.004687368869781454)  --> TIME: 2025-02-25 17:20:53 -- STEP: 23726/36938 -- GLOBAL_STEP: 873300 | > loss_text_ce: 0.035052284598350525 (0.03352197199972099) | > loss_mel_ce: 2.634847640991211 (nan) | > loss: 0.01059484202414751 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.13668120480162518) | > loader_time: 0.0047 (0.004687425540167398)  --> TIME: 2025-02-25 17:21:02 -- STEP: 23776/36938 -- GLOBAL_STEP: 873350 | > loss_text_ce: 0.026161639019846916 (0.03352258984270691) | > loss_mel_ce: 2.35482120513916 (nan) | > loss: 0.009448345750570297 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1366838691091789) | > loader_time: 0.0046 (0.004687397629223118)  --> TIME: 2025-02-25 17:21:11 -- STEP: 23826/36938 -- GLOBAL_STEP: 873400 | > loss_text_ce: 0.045081477612257004 (0.03352147878034214) | > loss_mel_ce: 2.8396215438842773 (nan) | > loss: 0.011447234079241753 (nan) | > current_lr: 5e-06 | > step_time: 0.1703 (0.13667542698460736) | > loader_time: 0.0048 (0.004687321753443021)  --> TIME: 2025-02-25 17:21:21 -- STEP: 23876/36938 -- GLOBAL_STEP: 873450 | > loss_text_ce: 0.03754155710339546 (0.03352057684761658) | > loss_mel_ce: 2.94521164894104 (nan) | > loss: 0.011836322955787182 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13667247823223702) | > loader_time: 0.0047 (0.004687337794282293)  --> TIME: 2025-02-25 17:21:30 -- STEP: 23926/36938 -- GLOBAL_STEP: 873500 | > loss_text_ce: 0.031142350286245346 (0.03351985573908578) | > loss_mel_ce: 1.2865725755691528 (nan) | > loss: 0.005229027941823006 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13666497612740844) | > loader_time: 0.0046 (0.004687305179555416)  --> TIME: 2025-02-25 17:21:39 -- STEP: 23976/36938 -- GLOBAL_STEP: 873550 | > loss_text_ce: 0.02790401130914688 (0.03352072827343484) | > loss_mel_ce: 3.1139326095581055 (nan) | > loss: 0.012467605993151665 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.1366734124041411) | > loader_time: 0.0048 (0.004687326120304002)  --> TIME: 2025-02-25 17:21:49 -- STEP: 24026/36938 -- GLOBAL_STEP: 873600 | > loss_text_ce: 0.029758909717202187 (0.03352397317223378) | > loss_mel_ce: 2.8673934936523438 (nan) | > loss: 0.011496637016534805 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13667677833725705) | > loader_time: 0.0046 (0.004687345703704177)  --> TIME: 2025-02-25 17:21:58 -- STEP: 24076/36938 -- GLOBAL_STEP: 873650 | > loss_text_ce: 0.034387655556201935 (0.033524199748630745) | > loss_mel_ce: 2.892493724822998 (nan) | > loss: 0.011614609509706497 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13667513298659514) | > loader_time: 0.0047 (0.004687412491389749)  --> TIME: 2025-02-25 17:22:08 -- STEP: 24126/36938 -- GLOBAL_STEP: 873700 | > loss_text_ce: 0.021438684314489365 (0.0335236553119021) | > loss_mel_ce: 2.3789894580841064 (nan) | > loss: 0.009525508619844913 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13668496886622258) | > loader_time: 0.0047 (0.004687482609259052)  --> TIME: 2025-02-25 17:22:17 -- STEP: 24176/36938 -- GLOBAL_STEP: 873750 | > loss_text_ce: 0.0340118408203125 (0.033525283910784846) | > loss_mel_ce: 2.797386646270752 (nan) | > loss: 0.011235709302127361 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.1366904391842431) | > loader_time: 0.0048 (0.004687474282430049)  --> TIME: 2025-02-25 17:22:27 -- STEP: 24226/36938 -- GLOBAL_STEP: 873800 | > loss_text_ce: 0.027169303968548775 (0.03352382227881907) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1083 (0.13669903259848867) | > loader_time: 0.0047 (0.004687443708966016)  --> TIME: 2025-02-25 17:22:36 -- STEP: 24276/36938 -- GLOBAL_STEP: 873850 | > loss_text_ce: 0.032161686569452286 (0.033526431951936525) | > loss_mel_ce: 2.602053642272949 (nan) | > loss: 0.010453236289322376 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13670874792026028) | > loader_time: 0.0046 (0.004687478709680492)  --> TIME: 2025-02-25 17:22:46 -- STEP: 24326/36938 -- GLOBAL_STEP: 873900 | > loss_text_ce: 0.030120696872472763 (0.03352474725877869) | > loss_mel_ce: 0.0093994140625 (nan) | > loss: 0.00015682584489695728 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13670443740710742) | > loader_time: 0.0045 (0.004687524141767794)  --> TIME: 2025-02-25 17:22:55 -- STEP: 24376/36938 -- GLOBAL_STEP: 873950 | > loss_text_ce: 0.03498871996998787 (0.03352620625171687) | > loss_mel_ce: 3.2848663330078125 (nan) | > loss: 0.013174028135836124 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.136701951231923) | > loader_time: 0.0046 (0.004687438255300014)  --> TIME: 2025-02-25 17:23:04 -- STEP: 24426/36938 -- GLOBAL_STEP: 874000 | > loss_text_ce: 0.02849637158215046 (0.033525823196155845) | > loss_mel_ce: 2.621249198913574 (nan) | > loss: 0.010514863766729832 (nan) | > current_lr: 5e-06 | > step_time: 0.1558 (0.13670761247705265) | > loader_time: 0.0046 (0.004687328816123914)  --> TIME: 2025-02-25 17:23:14 -- STEP: 24476/36938 -- GLOBAL_STEP: 874050 | > loss_text_ce: 0.0372106097638607 (0.033529176349896224) | > loss_mel_ce: 1.8371034860610962 (nan) | > loss: 0.007437754888087511 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13670986695974535) | > loader_time: 0.0047 (0.00468722447049169)  --> TIME: 2025-02-25 17:23:23 -- STEP: 24526/36938 -- GLOBAL_STEP: 874100 | > loss_text_ce: 0.0392436757683754 (0.033527511017219014) | > loss_mel_ce: 3.12261700630188 (nan) | > loss: 0.012547067366540432 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.13671394254977884) | > loader_time: 0.0049 (0.0046872788673997245)  --> TIME: 2025-02-25 17:23:32 -- STEP: 24576/36938 -- GLOBAL_STEP: 874150 | > loss_text_ce: 0.03428465873003006 (0.033529277564184494) | > loss_mel_ce: 2.9506428241729736 (nan) | > loss: 0.01184495072811842 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13670722154589912) | > loader_time: 0.0046 (0.004687245527747993)  --> TIME: 2025-02-25 17:23:42 -- STEP: 24626/36938 -- GLOBAL_STEP: 874200 | > loss_text_ce: 0.03163657337427139 (0.03353042243235961) | > loss_mel_ce: 3.2922487258911133 (nan) | > loss: 0.013190021738409996 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13670677039099016) | > loader_time: 0.0046 (0.00468714614988368)  --> TIME: 2025-02-25 17:23:51 -- STEP: 24676/36938 -- GLOBAL_STEP: 874250 | > loss_text_ce: 0.03862619400024414 (0.03353156906930471) | > loss_mel_ce: 3.1818017959594727 (nan) | > loss: 0.012779477052390575 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.1367216826084263) | > loader_time: 0.0046 (0.0046870639769024695)  --> TIME: 2025-02-25 17:24:01 -- STEP: 24726/36938 -- GLOBAL_STEP: 874300 | > loss_text_ce: 0.027140261605381966 (0.03353365361508204) | > loss_mel_ce: 1.9088518619537354 (nan) | > loss: 0.007682508789002895 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13672462261235055) | > loader_time: 0.0045 (0.004686992116164548)  --> TIME: 2025-02-25 17:24:10 -- STEP: 24776/36938 -- GLOBAL_STEP: 874350 | > loss_text_ce: 0.04624756798148155 (0.03353642590084117) | > loss_mel_ce: 1.6697460412979126 (nan) | > loss: 0.006809499114751816 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13671627073931486) | > loader_time: 0.0045 (0.004686915551149556)  --> TIME: 2025-02-25 17:24:19 -- STEP: 24826/36938 -- GLOBAL_STEP: 874400 | > loss_text_ce: 0.02577887661755085 (0.03353788136956021) | > loss_mel_ce: 3.0486137866973877 (nan) | > loss: 0.01219997089356184 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13671591262107427) | > loader_time: 0.0046 (0.004686891105877402)  --> TIME: 2025-02-25 17:24:29 -- STEP: 24876/36938 -- GLOBAL_STEP: 874450 | > loss_text_ce: 0.035641107708215714 (0.03353683002409004) | > loss_mel_ce: 3.258146047592163 (nan) | > loss: 0.013070585206151009 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13671382610428634) | > loader_time: 0.0046 (0.004686825163093343)  --> TIME: 2025-02-25 17:24:38 -- STEP: 24926/36938 -- GLOBAL_STEP: 874500 | > loss_text_ce: 0.03336462751030922 (0.03353430906689229) | > loss_mel_ce: 1.8890502452850342 (nan) | > loss: 0.007628630846738815 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1367084884743026) | > loader_time: 0.0046 (0.004686764420432255)  --> TIME: 2025-02-25 17:24:47 -- STEP: 24976/36938 -- GLOBAL_STEP: 874550 | > loss_text_ce: 0.033827148377895355 (0.033532463642440266) | > loss_mel_ce: 2.9164326190948486 (nan) | > loss: 0.011707380414009094 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1367069681644587) | > loader_time: 0.0045 (0.004686703844608034)  --> TIME: 2025-02-25 17:24:56 -- STEP: 25026/36938 -- GLOBAL_STEP: 874600 | > loss_text_ce: 0.03282919153571129 (0.033530271287897484) | > loss_mel_ce: 2.740208864212036 (nan) | > loss: 0.011004120111465454 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.13670082709814585) | > loader_time: 0.0046 (0.004686609490506673)  --> TIME: 2025-02-25 17:25:06 -- STEP: 25076/36938 -- GLOBAL_STEP: 874650 | > loss_text_ce: 0.02693505398929119 (0.0335289806707376) | > loss_mel_ce: 2.687410831451416 (nan) | > loss: 0.010771214962005615 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.1366990412093553) | > loader_time: 0.0047 (0.004686530392446408)  --> TIME: 2025-02-25 17:25:15 -- STEP: 25126/36938 -- GLOBAL_STEP: 874700 | > loss_text_ce: 0.033970899879932404 (0.03352577522793481) | > loss_mel_ce: 3.254946231842041 (nan) | > loss: 0.013051259331405163 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.1366999143913823) | > loader_time: 0.0046 (0.004686390993977115)  --> TIME: 2025-02-25 17:25:24 -- STEP: 25176/36938 -- GLOBAL_STEP: 874750 | > loss_text_ce: 0.039068132638931274 (0.03352592928109547) | > loss_mel_ce: 2.060737371444702 (nan) | > loss: 0.008332562632858753 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.1367052552584125) | > loader_time: 0.0047 (0.004686384815468162)  --> TIME: 2025-02-25 17:25:34 -- STEP: 25226/36938 -- GLOBAL_STEP: 874800 | > loss_text_ce: 0.04029197618365288 (0.03352670383102884) | > loss_mel_ce: 2.7135009765625 (nan) | > loss: 0.01092775072902441 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13670215525030163) | > loader_time: 0.0045 (0.004686315016374865)  --> TIME: 2025-02-25 17:25:43 -- STEP: 25276/36938 -- GLOBAL_STEP: 874850 | > loss_text_ce: 0.02893766760826111 (0.03352669284468656) | > loss_mel_ce: 2.0356993675231934 (nan) | > loss: 0.008193003945052624 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.1367011497548568) | > loader_time: 0.0045 (0.004686277026635072)  --> TIME: 2025-02-25 17:25:52 -- STEP: 25326/36938 -- GLOBAL_STEP: 874900 | > loss_text_ce: 0.03120150789618492 (0.033526687219228045) | > loss_mel_ce: 1.3434032201766968 (nan) | > loss: 0.005454780999571085 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.13670482766728975) | > loader_time: 0.0045 (0.004686251180314838)  --> TIME: 2025-02-25 17:26:02 -- STEP: 25376/36938 -- GLOBAL_STEP: 874950 | > loss_text_ce: 0.03158587962388992 (0.03352858735567576) | > loss_mel_ce: 2.2305824756622314 (nan) | > loss: 0.008976859040558338 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13670473853020437) | > loader_time: 0.0048 (0.004686172530148966)  --> TIME: 2025-02-25 17:26:11 -- STEP: 25426/36938 -- GLOBAL_STEP: 875000 | > loss_text_ce: 0.03261835500597954 (0.03352732823271255) | > loss_mel_ce: 3.1962294578552246 (nan) | > loss: 0.012812888249754906 (nan) | > current_lr: 5e-06 | > step_time: 0.1762 (0.13671393734103665) | > loader_time: 0.0048 (0.00468606894653708)  --> TIME: 2025-02-25 17:26:21 -- STEP: 25476/36938 -- GLOBAL_STEP: 875050 | > loss_text_ce: 0.040762823075056076 (0.03352692375706204) | > loss_mel_ce: 2.422264814376831 (nan) | > loss: 0.0097739202901721 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1367137463262379) | > loader_time: 0.0045 (0.00468606832993609)  --> TIME: 2025-02-25 17:26:30 -- STEP: 25526/36938 -- GLOBAL_STEP: 875100 | > loss_text_ce: 0.03599042817950249 (0.03352868906169358) | > loss_mel_ce: 3.1995043754577637 (nan) | > loss: 0.01283926609903574 (nan) | > current_lr: 5e-06 | > step_time: 0.1458 (0.1367135598238642) | > loader_time: 0.0046 (0.004686042609226328)  --> TIME: 2025-02-25 17:26:39 -- STEP: 25576/36938 -- GLOBAL_STEP: 875150 | > loss_text_ce: 0.028877820819616318 (0.03352460556886113) | > loss_mel_ce: 2.3379247188568115 (nan) | > loss: 0.00939207337796688 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13671740331163507) | > loader_time: 0.0047 (0.004685948742975959)  --> TIME: 2025-02-25 17:26:49 -- STEP: 25626/36938 -- GLOBAL_STEP: 875200 | > loss_text_ce: 0.034239932894706726 (0.03352429724284513) | > loss_mel_ce: 2.5357842445373535 (nan) | > loss: 0.010198509320616722 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.13671822661604716) | > loader_time: 0.0048 (0.00468601099754264)  --> TIME: 2025-02-25 17:26:58 -- STEP: 25676/36938 -- GLOBAL_STEP: 875250 | > loss_text_ce: 0.03559572994709015 (0.03352337880638958) | > loss_mel_ce: 3.003974199295044 (nan) | > loss: 0.012061785906553268 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.13672135979016056) | > loader_time: 0.0048 (0.0046859069913509285)  --> TIME: 2025-02-25 17:27:07 -- STEP: 25726/36938 -- GLOBAL_STEP: 875300 | > loss_text_ce: 0.026785189285874367 (0.03352220474626414) | > loss_mel_ce: 2.9587361812591553 (nan) | > loss: 0.011847307905554771 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13670582727928826) | > loader_time: 0.0045 (0.004685731546917515)  --> TIME: 2025-02-25 17:27:16 -- STEP: 25776/36938 -- GLOBAL_STEP: 875350 | > loss_text_ce: 0.035462554544210434 (0.03352441128554621) | > loss_mel_ce: 2.9554824829101562 (nan) | > loss: 0.011868830770254135 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.1367056594640109) | > loader_time: 0.0046 (0.004685677925597617)  --> TIME: 2025-02-25 17:27:26 -- STEP: 25826/36938 -- GLOBAL_STEP: 875400 | > loss_text_ce: 0.04863263666629791 (0.03352325744127788) | > loss_mel_ce: 2.148599624633789 (nan) | > loss: 0.008719176054000854 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.13670333216674635) | > loader_time: 0.0045 (0.004685617514254059)  --> TIME: 2025-02-25 17:27:35 -- STEP: 25876/36938 -- GLOBAL_STEP: 875450 | > loss_text_ce: 0.032744016498327255 (0.033524243685985805) | > loss_mel_ce: 2.8667171001434326 (nan) | > loss: 0.011505798436701298 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13670975003033903) | > loader_time: 0.0046 (0.004685532200887412)  --> TIME: 2025-02-25 17:27:44 -- STEP: 25926/36938 -- GLOBAL_STEP: 875500 | > loss_text_ce: 0.04466986656188965 (0.03352336893670698) | > loss_mel_ce: 2.7832112312316895 (nan) | > loss: 0.011221750639379025 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13671225671239237) | > loader_time: 0.0046 (0.004685581489118013)  --> TIME: 2025-02-25 17:27:54 -- STEP: 25976/36938 -- GLOBAL_STEP: 875550 | > loss_text_ce: 0.024478236213326454 (0.03352328242132017) | > loss_mel_ce: 2.9931178092956543 (nan) | > loss: 0.01197458803653717 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.1367120096178323) | > loader_time: 0.0048 (0.004685435436085446)  --> TIME: 2025-02-25 17:28:03 -- STEP: 26026/36938 -- GLOBAL_STEP: 875600 | > loss_text_ce: 0.02947787195444107 (0.033523272636647644) | > loss_mel_ce: 2.6828830242156982 (nan) | > loss: 0.010763336904346943 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13670369021982154) | > loader_time: 0.0045 (0.004685276908437695)  --> TIME: 2025-02-25 17:28:12 -- STEP: 26076/36938 -- GLOBAL_STEP: 875650 | > loss_text_ce: 0.03201589360833168 (0.03352534480018334) | > loss_mel_ce: 2.734830141067505 (nan) | > loss: 0.0109795480966568 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13670689481078982) | > loader_time: 0.0045 (0.004685177103034673)  --> TIME: 2025-02-25 17:28:22 -- STEP: 26126/36938 -- GLOBAL_STEP: 875700 | > loss_text_ce: 0.03915350139141083 (0.03352515517334658) | > loss_mel_ce: 0.029585838317871094 (nan) | > loss: 0.00027277518529444933 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13670856021113678) | > loader_time: 0.0045 (0.004685072094706209)  --> TIME: 2025-02-25 17:28:31 -- STEP: 26176/36938 -- GLOBAL_STEP: 875750 | > loss_text_ce: 0.03544573858380318 (0.03352625332468873) | > loss_mel_ce: 0.20553311705589294 (nan) | > loss: 0.0009562653140164912 (nan) | > current_lr: 5e-06 | > step_time: 0.1617 (0.13670171416549062) | > loader_time: 0.0048 (0.004684949735484643)  --> TIME: 2025-02-25 17:28:40 -- STEP: 26226/36938 -- GLOBAL_STEP: 875800 | > loss_text_ce: 0.027148611843585968 (0.03352835615213319) | > loss_mel_ce: 0.6709268689155579 (nan) | > loss: 0.002770140999928117 (nan) | > current_lr: 5e-06 | > step_time: 0.1077 (0.13670013033261008) | > loader_time: 0.0045 (0.004684922997524027)  --> TIME: 2025-02-25 17:28:49 -- STEP: 26276/36938 -- GLOBAL_STEP: 875850 | > loss_text_ce: 0.02042502537369728 (0.033529721076235336) | > loss_mel_ce: 2.570858955383301 (nan) | > loss: 0.01028287410736084 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13670234726269814) | > loader_time: 0.0045 (0.00468487868589938)  --> TIME: 2025-02-25 17:28:59 -- STEP: 26326/36938 -- GLOBAL_STEP: 875900 | > loss_text_ce: 0.03203464671969414 (0.033529721480612784) | > loss_mel_ce: 3.003444194793701 (nan) | > loss: 0.012045552022755146 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13670001693439554) | > loader_time: 0.0046 (0.004684715115950636)  --> TIME: 2025-02-25 17:29:08 -- STEP: 26376/36938 -- GLOBAL_STEP: 875950 | > loss_text_ce: 0.026239832863211632 (0.033531368409864114) | > loss_mel_ce: 2.993558645248413 (nan) | > loss: 0.011983328498899937 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13669632194003886) | > loader_time: 0.0052 (0.004684739440797033)  --> TIME: 2025-02-25 17:29:17 -- STEP: 26426/36938 -- GLOBAL_STEP: 876000 | > loss_text_ce: 0.0310503039509058 (0.033530551092687086) | > loss_mel_ce: 2.6352994441986084 (nan) | > loss: 0.010580752976238728 (nan) | > current_lr: 5e-06 | > step_time: 0.1882 (0.13669220233571674) | > loader_time: 0.0048 (0.0046846373007406105)  --> TIME: 2025-02-25 17:29:26 -- STEP: 26476/36938 -- GLOBAL_STEP: 876050 | > loss_text_ce: 0.03277340903878212 (0.03353069310866862) | > loss_mel_ce: 2.755922555923462 (nan) | > loss: 0.011066255159676075 (nan) | > current_lr: 5e-06 | > step_time: 0.1457 (0.13668951823967515) | > loader_time: 0.0046 (0.004684583273407965)  --> TIME: 2025-02-25 17:29:35 -- STEP: 26526/36938 -- GLOBAL_STEP: 876100 | > loss_text_ce: 0.03762088343501091 (0.033529429684457546) | > loss_mel_ce: 2.8314852714538574 (nan) | > loss: 0.011385342106223106 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.1366842639021123) | > loader_time: 0.0047 (0.004684384058101172)  --> TIME: 2025-02-25 17:29:45 -- STEP: 26576/36938 -- GLOBAL_STEP: 876150 | > loss_text_ce: 0.032645754516124725 (0.03352862889775415) | > loss_mel_ce: 3.041698455810547 (nan) | > loss: 0.01219977904111147 (nan) | > current_lr: 5e-06 | > step_time: 0.1576 (0.13668340916449892) | > loader_time: 0.0046 (0.004684240585849038)  --> TIME: 2025-02-25 17:29:54 -- STEP: 26626/36938 -- GLOBAL_STEP: 876200 | > loss_text_ce: 0.03171210363507271 (0.03353021945674933) | > loss_mel_ce: 2.966104030609131 (nan) | > loss: 0.011896096169948578 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.13668386262641394) | > loader_time: 0.0048 (0.004684081095842543)  --> TIME: 2025-02-25 17:30:03 -- STEP: 26676/36938 -- GLOBAL_STEP: 876250 | > loss_text_ce: 0.03993357717990875 (0.03353027020775732) | > loss_mel_ce: 2.1455883979797363 (nan) | > loss: 0.00867270678281784 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1366657953084293) | > loader_time: 0.0046 (0.004684175834035118)  --> TIME: 2025-02-25 17:30:12 -- STEP: 26726/36938 -- GLOBAL_STEP: 876300 | > loss_text_ce: 0.031393442302942276 (0.033531336854624165) | > loss_mel_ce: 0.0322265625 (nan) | > loss: 0.00025246033328585327 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13666402632574798) | > loader_time: 0.0046 (0.004684076412329083)  --> TIME: 2025-02-25 17:30:21 -- STEP: 26776/36938 -- GLOBAL_STEP: 876350 | > loss_text_ce: 0.03287970647215843 (0.03352993023817619) | > loss_mel_ce: 3.004021167755127 (nan) | > loss: 0.012051193974912167 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.1366659189235784) | > loader_time: 0.0053 (0.0046840632161276145)  --> TIME: 2025-02-25 17:30:31 -- STEP: 26826/36938 -- GLOBAL_STEP: 876400 | > loss_text_ce: 0.027726396918296814 (0.03353143573640653) | > loss_mel_ce: 1.6232638359069824 (nan) | > loss: 0.0065515488386154175 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13666277469284344) | > loader_time: 0.0044 (0.004683922701017737)  --> TIME: 2025-02-25 17:30:40 -- STEP: 26876/36938 -- GLOBAL_STEP: 876450 | > loss_text_ce: 0.030677031725645065 (0.03353210891212184) | > loss_mel_ce: 2.857872486114502 (nan) | > loss: 0.011462499387562275 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1366667884910261) | > loader_time: 0.0045 (0.004683949626579665)  --> TIME: 2025-02-25 17:30:49 -- STEP: 26926/36938 -- GLOBAL_STEP: 876500 | > loss_text_ce: 0.04020443186163902 (0.03352988381676877) | > loss_mel_ce: 2.6913018226623535 (nan) | > loss: 0.01083931140601635 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13667165340520482) | > loader_time: 0.0045 (0.004683801069360918)  --> TIME: 2025-02-25 17:30:59 -- STEP: 26976/36938 -- GLOBAL_STEP: 876550 | > loss_text_ce: 0.03725212439894676 (0.033529712955105824) | > loss_mel_ce: 2.438040256500244 (nan) | > loss: 0.009822589345276356 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13667332834408366) | > loader_time: 0.0047 (0.004683665604212364)  --> TIME: 2025-02-25 17:31:08 -- STEP: 27026/36938 -- GLOBAL_STEP: 876600 | > loss_text_ce: 0.03687372803688049 (0.0335276278973572) | > loss_mel_ce: 2.757525682449341 (nan) | > loss: 0.011088887229561806 (nan) | > current_lr: 5e-06 | > step_time: 0.1882 (0.1366763029092864) | > loader_time: 0.0049 (0.004683620587607712)  --> TIME: 2025-02-25 17:31:18 -- STEP: 27076/36938 -- GLOBAL_STEP: 876650 | > loss_text_ce: 0.023543905466794968 (0.03352813811323738) | > loss_mel_ce: 1.8805768489837646 (nan) | > loss: 0.007556035183370113 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13668706184969912) | > loader_time: 0.0047 (0.004683572708160289)  --> TIME: 2025-02-25 17:31:28 -- STEP: 27126/36938 -- GLOBAL_STEP: 876700 | > loss_text_ce: 0.03385261446237564 (0.03352879468237747) | > loss_mel_ce: 2.496091604232788 (nan) | > loss: 0.01003946177661419 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13669077252107645) | > loader_time: 0.0046 (0.0046834799073169845)  --> TIME: 2025-02-25 17:31:37 -- STEP: 27176/36938 -- GLOBAL_STEP: 876750 | > loss_text_ce: 0.03277193382382393 (0.03353165507434886) | > loss_mel_ce: 3.125591993331909 (nan) | > loss: 0.012533190660178661 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.1366929597966207) | > loader_time: 0.0048 (0.004683423821351271)  --> TIME: 2025-02-25 17:31:47 -- STEP: 27226/36938 -- GLOBAL_STEP: 876800 | > loss_text_ce: 0.03940792381763458 (0.03353383833603217) | > loss_mel_ce: 1.446810007095337 (nan) | > loss: 0.005897690542042255 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.13669809025862742) | > loader_time: 0.0047 (0.004683319628917321)  --> TIME: 2025-02-25 17:31:56 -- STEP: 27276/36938 -- GLOBAL_STEP: 876850 | > loss_text_ce: 0.034967247396707535 (0.033533355297601604) | > loss_mel_ce: 3.5032544136047363 (nan) | > loss: 0.014040562324225903 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.13669709044653813) | > loader_time: 0.0046 (0.004683160392013256)  --> TIME: 2025-02-25 17:32:05 -- STEP: 27326/36938 -- GLOBAL_STEP: 876900 | > loss_text_ce: 0.029299048706889153 (0.033531950225891775) | > loss_mel_ce: 2.7313122749328613 (nan) | > loss: 0.010954807512462139 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.1366877690822481) | > loader_time: 0.0046 (0.004682985631541651)  --> TIME: 2025-02-25 17:32:14 -- STEP: 27376/36938 -- GLOBAL_STEP: 876950 | > loss_text_ce: 0.04605967551469803 (0.03353351522387431) | > loss_mel_ce: 2.9691243171691895 (nan) | > loss: 0.011965015903115273 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13668381291687098) | > loader_time: 0.0045 (0.004682843567403947)  --> TIME: 2025-02-25 17:32:23 -- STEP: 27426/36938 -- GLOBAL_STEP: 877000 | > loss_text_ce: 0.031408682465553284 (0.03353055267487654) | > loss_mel_ce: 1.8050332069396973 (nan) | > loss: 0.007287468295544386 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13668401849997475) | > loader_time: 0.0045 (0.004682814667211469)  --> TIME: 2025-02-25 17:32:33 -- STEP: 27476/36938 -- GLOBAL_STEP: 877050 | > loss_text_ce: 0.03643711656332016 (0.03352973070239015) | > loss_mel_ce: 2.7844834327697754 (nan) | > loss: 0.011194129474461079 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13668752586950908) | > loader_time: 0.0046 (0.0046831029421863955)  --> TIME: 2025-02-25 17:32:42 -- STEP: 27526/36938 -- GLOBAL_STEP: 877100 | > loss_text_ce: 0.029763396829366684 (0.03352905586660979) | > loss_mel_ce: 1.985248327255249 (nan) | > loss: 0.007996078580617905 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13669228276601478) | > loader_time: 0.0047 (0.004683107525285614)  --> TIME: 2025-02-25 17:32:51 -- STEP: 27576/36938 -- GLOBAL_STEP: 877150 | > loss_text_ce: 0.03969062492251396 (0.033529102469883325) | > loss_mel_ce: 1.8191065788269043 (nan) | > loss: 0.007376179564744234 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13668635011168873) | > loader_time: 0.0047 (0.004683011133912172)  --> TIME: 2025-02-25 17:33:01 -- STEP: 27626/36938 -- GLOBAL_STEP: 877200 | > loss_text_ce: 0.03453696519136429 (0.0335267542429284) | > loss_mel_ce: 2.762939453125 (nan) | > loss: 0.01110109779983759 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13668356707791166) | > loader_time: 0.0047 (0.004683028570274701)  --> TIME: 2025-02-25 17:33:10 -- STEP: 27676/36938 -- GLOBAL_STEP: 877250 | > loss_text_ce: 0.033746156841516495 (0.03352202212577033) | > loss_mel_ce: 3.2999632358551025 (nan) | > loss: 0.0132290069013834 (nan) | > current_lr: 5e-06 | > step_time: 0.1529 (0.13668417427508442) | > loader_time: 0.0047 (0.004682997270959815)  --> TIME: 2025-02-25 17:33:20 -- STEP: 27726/36938 -- GLOBAL_STEP: 877300 | > loss_text_ce: 0.025009019300341606 (0.03352019998119245) | > loss_mel_ce: 2.8372080326080322 (nan) | > loss: 0.011358004063367844 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.136689979414536) | > loader_time: 0.0046 (0.0046830772880626014)  --> TIME: 2025-02-25 17:33:29 -- STEP: 27776/36938 -- GLOBAL_STEP: 877350 | > loss_text_ce: 0.03466418385505676 (0.033518348451009594) | > loss_mel_ce: 2.637436866760254 (nan) | > loss: 0.010603575967252254 (nan) | > current_lr: 5e-06 | > step_time: 0.1622 (0.13668923045632983) | > loader_time: 0.0047 (0.004683001095660793)  --> TIME: 2025-02-25 17:33:38 -- STEP: 27826/36938 -- GLOBAL_STEP: 877400 | > loss_text_ce: 0.0356895886361599 (0.03351674876300665) | > loss_mel_ce: 0.05216878280043602 (nan) | > loss: 0.0003486443602014333 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13668768244660834) | > loader_time: 0.0046 (0.004682958970035242)  --> TIME: 2025-02-25 17:33:47 -- STEP: 27876/36938 -- GLOBAL_STEP: 877450 | > loss_text_ce: 0.023504789918661118 (0.03351437065915806) | > loss_mel_ce: 1.453271508216858 (nan) | > loss: 0.005860223434865475 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13668182389194558) | > loader_time: 0.0045 (0.004682872708994279)  --> TIME: 2025-02-25 17:33:57 -- STEP: 27926/36938 -- GLOBAL_STEP: 877500 | > loss_text_ce: 0.025632398203015327 (0.03351373405718763) | > loss_mel_ce: 2.8311448097229004 (nan) | > loss: 0.011336417868733406 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13667933617144917) | > loader_time: 0.0045 (0.004682791187813899)  --> TIME: 2025-02-25 17:34:06 -- STEP: 27976/36938 -- GLOBAL_STEP: 877550 | > loss_text_ce: 0.03250844404101372 (0.03351444927834387) | > loss_mel_ce: 3.2279627323150635 (nan) | > loss: 0.012938378378748894 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.13667212121139585) | > loader_time: 0.0049 (0.004682729269458022)  --> TIME: 2025-02-25 17:34:15 -- STEP: 28026/36938 -- GLOBAL_STEP: 877600 | > loss_text_ce: 0.03084648959338665 (0.03351369426168821) | > loss_mel_ce: 2.7197210788726807 (nan) | > loss: 0.01091495156288147 (nan) | > current_lr: 5e-06 | > step_time: 0.1785 (0.1366725336481938) | > loader_time: 0.0049 (0.004682737346856851)  --> TIME: 2025-02-25 17:34:25 -- STEP: 28076/36938 -- GLOBAL_STEP: 877650 | > loss_text_ce: 0.03813173621892929 (0.0335135353521608) | > loss_mel_ce: 3.380573272705078 (nan) | > loss: 0.013566290028393269 (nan) | > current_lr: 5e-06 | > step_time: 0.177 (0.1366794041664036) | > loader_time: 0.0048 (0.004682826501621013)  --> TIME: 2025-02-25 17:34:34 -- STEP: 28126/36938 -- GLOBAL_STEP: 877700 | > loss_text_ce: 0.03658510744571686 (0.03351566852936944) | > loss_mel_ce: 2.1933486461639404 (nan) | > loss: 0.008848943747580051 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.1366825909140875) | > loader_time: 0.0047 (0.004682908354515532)  --> TIME: 2025-02-25 17:34:43 -- STEP: 28176/36938 -- GLOBAL_STEP: 877750 | > loss_text_ce: 0.02676825225353241 (0.0335151750877582) | > loss_mel_ce: 2.3534250259399414 (nan) | > loss: 0.009445211850106716 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13667943713206002) | > loader_time: 0.0046 (0.004682953844417001)  --> TIME: 2025-02-25 17:34:52 -- STEP: 28226/36938 -- GLOBAL_STEP: 877800 | > loss_text_ce: 0.03403691202402115 (0.03351270653212937) | > loss_mel_ce: 2.4450058937072754 (nan) | > loss: 0.009837471880018711 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13667555307417856) | > loader_time: 0.0045 (0.0046828939517198015)  --> TIME: 2025-02-25 17:35:02 -- STEP: 28276/36938 -- GLOBAL_STEP: 877850 | > loss_text_ce: 0.03748365491628647 (0.03351164217488931) | > loss_mel_ce: 2.553515672683716 (nan) | > loss: 0.010281744413077831 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13667014943168823) | > loader_time: 0.0045 (0.004682754522537548)  --> TIME: 2025-02-25 17:35:11 -- STEP: 28326/36938 -- GLOBAL_STEP: 877900 | > loss_text_ce: 0.03951239213347435 (0.03350978163431651) | > loss_mel_ce: 3.309809684753418 (nan) | > loss: 0.013290961273014545 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13666113843051172) | > loader_time: 0.0046 (0.004682629077959562)  --> TIME: 2025-02-25 17:35:20 -- STEP: 28376/36938 -- GLOBAL_STEP: 877950 | > loss_text_ce: 0.029743008315563202 (0.03350949624483999) | > loss_mel_ce: 2.300581455230713 (nan) | > loss: 0.009247319772839546 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.13666923666356556) | > loader_time: 0.0046 (0.004682730924320221)  --> TIME: 2025-02-25 17:35:30 -- STEP: 28426/36938 -- GLOBAL_STEP: 878000 | > loss_text_ce: 0.037402525544166565 (0.033508867252226696) | > loss_mel_ce: 2.5962066650390625 (nan) | > loss: 0.010450830683112144 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.1366729707391788) | > loader_time: 0.0047 (0.004682719208442147)  --> TIME: 2025-02-25 17:35:39 -- STEP: 28476/36938 -- GLOBAL_STEP: 878050 | > loss_text_ce: 0.0353703610599041 (0.033506059145439984) | > loss_mel_ce: 3.0484719276428223 (nan) | > loss: 0.012237469665706158 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13667892478694166) | > loader_time: 0.0046 (0.004682840365240898)  --> TIME: 2025-02-25 17:35:49 -- STEP: 28526/36938 -- GLOBAL_STEP: 878100 | > loss_text_ce: 0.04706815630197525 (0.03350809393419922) | > loss_mel_ce: 1.6133335828781128 (nan) | > loss: 0.006588895805180073 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.13667529962227087) | > loader_time: 0.0045 (0.0046827791700487925)  --> TIME: 2025-02-25 17:35:58 -- STEP: 28576/36938 -- GLOBAL_STEP: 878150 | > loss_text_ce: 0.03885086253285408 (0.03350877122690712) | > loss_mel_ce: 3.264197587966919 (nan) | > loss: 0.013107335194945335 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.1366789974564261) | > loader_time: 0.0046 (0.004682843630751703)  --> TIME: 2025-02-25 17:36:07 -- STEP: 28626/36938 -- GLOBAL_STEP: 878200 | > loss_text_ce: 0.03258325532078743 (0.033510281626674576) | > loss_mel_ce: 3.3116698265075684 (nan) | > loss: 0.013270846568048 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13667542179167294) | > loader_time: 0.0046 (0.0046828650898514375)  --> TIME: 2025-02-25 17:36:17 -- STEP: 28676/36938 -- GLOBAL_STEP: 878250 | > loss_text_ce: 0.03314848616719246 (0.033511432527615966) | > loss_mel_ce: 2.8357937335968018 (nan) | > loss: 0.011384692043066025 (nan) | > current_lr: 5e-06 | > step_time: 0.1614 (0.13668071033066911) | > loader_time: 0.0046 (0.004683135119188186)  --> TIME: 2025-02-25 17:36:26 -- STEP: 28726/36938 -- GLOBAL_STEP: 878300 | > loss_text_ce: 0.032130248844623566 (0.03350911030293851) | > loss_mel_ce: 3.0739455223083496 (nan) | > loss: 0.012325698509812355 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13667764478113825) | > loader_time: 0.0046 (0.004683067271912427)  --> TIME: 2025-02-25 17:36:35 -- STEP: 28776/36938 -- GLOBAL_STEP: 878350 | > loss_text_ce: 0.03516872972249985 (0.03351203183690637) | > loss_mel_ce: 2.6105496883392334 (nan) | > loss: 0.010498883202672005 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.1366608908831287) | > loader_time: 0.0045 (0.004682846232722566)  --> TIME: 2025-02-25 17:36:44 -- STEP: 28826/36938 -- GLOBAL_STEP: 878400 | > loss_text_ce: 0.02513321302831173 (0.03351344729183163) | > loss_mel_ce: 2.3603973388671875 (nan) | > loss: 0.009466391056776047 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.136661051893303) | > loader_time: 0.0045 (0.004682742026663888)  --> TIME: 2025-02-25 17:36:53 -- STEP: 28876/36938 -- GLOBAL_STEP: 878450 | > loss_text_ce: 0.02477429248392582 (0.03351278168642493) | > loss_mel_ce: 2.5862679481506348 (nan) | > loss: 0.010361279360949993 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13665535188576564) | > loader_time: 0.0045 (0.004682613007000771)  --> TIME: 2025-02-25 17:37:02 -- STEP: 28926/36938 -- GLOBAL_STEP: 878500 | > loss_text_ce: 0.03210754320025444 (0.03351061919865445) | > loss_mel_ce: 1.885048747062683 (nan) | > loss: 0.007607763633131981 (nan) | > current_lr: 5e-06 | > step_time: 0.1076 (0.13665091729317358) | > loader_time: 0.0045 (0.004682510058876364)  --> TIME: 2025-02-25 17:37:11 -- STEP: 28976/36938 -- GLOBAL_STEP: 878550 | > loss_text_ce: 0.031559064984321594 (0.03351091215150937) | > loss_mel_ce: 1.585938572883606 (nan) | > loss: 0.006418642122298479 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.136641351764271) | > loader_time: 0.0045 (0.004682426777483652)  --> TIME: 2025-02-25 17:37:21 -- STEP: 29026/36938 -- GLOBAL_STEP: 878600 | > loss_text_ce: 0.02873893268406391 (0.03351143900123291) | > loss_mel_ce: 2.429443359375 (nan) | > loss: 0.009754692204296589 (nan) | > current_lr: 5e-06 | > step_time: 0.1116 (0.13664632950645147) | > loader_time: 0.0057 (0.00468236759529855)  --> TIME: 2025-02-25 17:37:30 -- STEP: 29076/36938 -- GLOBAL_STEP: 878650 | > loss_text_ce: 0.029980890452861786 (0.03351241438335042) | > loss_mel_ce: 2.584845542907715 (nan) | > loss: 0.010376296006143093 (nan) | > current_lr: 5e-06 | > step_time: 0.1829 (0.13664629474468726) | > loader_time: 0.0049 (0.004682459321535645)  --> TIME: 2025-02-25 17:37:40 -- STEP: 29126/36938 -- GLOBAL_STEP: 878700 | > loss_text_ce: 0.03784703090786934 (0.033510955620521414) | > loss_mel_ce: 2.751229763031006 (nan) | > loss: 0.011067765764892101 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13664860002763443) | > loader_time: 0.0045 (0.004682445905942497)  --> TIME: 2025-02-25 17:37:49 -- STEP: 29176/36938 -- GLOBAL_STEP: 878750 | > loss_text_ce: 0.033228423446416855 (0.03350941548362077) | > loss_mel_ce: 2.5914504528045654 (nan) | > loss: 0.010415392927825451 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13665278896194924) | > loader_time: 0.0046 (0.004682418252135853)  --> TIME: 2025-02-25 17:37:58 -- STEP: 29226/36938 -- GLOBAL_STEP: 878800 | > loss_text_ce: 0.044752221554517746 (0.03351079719599992) | > loss_mel_ce: 2.39182710647583 (nan) | > loss: 0.009668965823948383 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13664437937620572) | > loader_time: 0.0045 (0.004682283018727316)  --> TIME: 2025-02-25 17:38:08 -- STEP: 29276/36938 -- GLOBAL_STEP: 878850 | > loss_text_ce: 0.029964305460453033 (0.033510908787402725) | > loss_mel_ce: 2.6803338527679443 (nan) | > loss: 0.010755151510238647 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13665273502593148) | > loader_time: 0.0046 (0.004682342224978449)  --> TIME: 2025-02-25 17:38:17 -- STEP: 29326/36938 -- GLOBAL_STEP: 878900 | > loss_text_ce: 0.0352066233754158 (0.03351123137510064) | > loss_mel_ce: 2.7332661151885986 (nan) | > loss: 0.01098600309342146 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.1366559112087426) | > loader_time: 0.0046 (0.004682326775362006)  --> TIME: 2025-02-25 17:38:27 -- STEP: 29376/36938 -- GLOBAL_STEP: 878950 | > loss_text_ce: 0.033730119466781616 (0.033511690508219366) | > loss_mel_ce: 3.3222784996032715 (nan) | > loss: 0.01331749465316534 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.1366549960148867) | > loader_time: 0.0054 (0.00468240307404582)  --> TIME: 2025-02-25 17:38:36 -- STEP: 29426/36938 -- GLOBAL_STEP: 879000 | > loss_text_ce: 0.030795639380812645 (0.03351177389122711) | > loss_mel_ce: 3.4393787384033203 (nan) | > loss: 0.013770533725619316 (nan) | > current_lr: 5e-06 | > step_time: 0.1531 (0.136659801609938) | > loader_time: 0.0047 (0.004682430248406962)  --> TIME: 2025-02-25 17:38:45 -- STEP: 29476/36938 -- GLOBAL_STEP: 879050 | > loss_text_ce: 0.03443467617034912 (0.03351401486341773) | > loss_mel_ce: 2.853254556655884 (nan) | > loss: 0.01145908422768116 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13665807863510498) | > loader_time: 0.0046 (0.004682259662194342)  --> TIME: 2025-02-25 17:38:55 -- STEP: 29526/36938 -- GLOBAL_STEP: 879100 | > loss_text_ce: 0.040549516677856445 (0.033514831990614405) | > loss_mel_ce: 2.739626407623291 (nan) | > loss: 0.011032444424927235 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.13665673270551945) | > loader_time: 0.0046 (0.004682122461923202)  --> TIME: 2025-02-25 17:39:04 -- STEP: 29576/36938 -- GLOBAL_STEP: 879150 | > loss_text_ce: 0.04021437093615532 (0.03351467188564465) | > loss_mel_ce: 1.2306462526321411 (nan) | > loss: 0.005043098237365484 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.13666240183733658) | > loader_time: 0.0046 (0.004682138824192008)  --> TIME: 2025-02-25 17:39:14 -- STEP: 29626/36938 -- GLOBAL_STEP: 879200 | > loss_text_ce: 0.031961262226104736 (0.03351471447549618) | > loss_mel_ce: 2.271728515625 (nan) | > loss: 0.00914162676781416 (nan) | > current_lr: 5e-06 | > step_time: 0.1531 (0.13666073331824397) | > loader_time: 0.0047 (0.0046821699388390845)  --> TIME: 2025-02-25 17:39:23 -- STEP: 29676/36938 -- GLOBAL_STEP: 879250 | > loss_text_ce: 0.03269379958510399 (0.03351313291280851) | > loss_mel_ce: 2.1446001529693604 (nan) | > loss: 0.008640056475996971 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1366547231308472) | > loader_time: 0.0045 (0.004682146542026315)  --> TIME: 2025-02-25 17:39:32 -- STEP: 29726/36938 -- GLOBAL_STEP: 879300 | > loss_text_ce: 0.02552608586847782 (0.03351351028251361) | > loss_mel_ce: 2.792177200317383 (nan) | > loss: 0.01118136290460825 (nan) | > current_lr: 5e-06 | > step_time: 0.1689 (0.1366538240395768) | > loader_time: 0.0047 (0.004682020320390543)  --> TIME: 2025-02-25 17:39:41 -- STEP: 29776/36938 -- GLOBAL_STEP: 879350 | > loss_text_ce: 0.03569071367383003 (0.03350885657949281) | > loss_mel_ce: 1.8577792644500732 (nan) | > loss: 0.007513769902288914 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13664906324237092) | > loader_time: 0.0054 (0.0046821307793283385)  --> TIME: 2025-02-25 17:39:51 -- STEP: 29826/36938 -- GLOBAL_STEP: 879400 | > loss_text_ce: 0.034141361713409424 (0.0335088353819843) | > loss_mel_ce: 2.3529207706451416 (nan) | > loss: 0.009472468867897987 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.13665377487800748) | > loader_time: 0.0046 (0.0046821468466205505)  --> TIME: 2025-02-25 17:40:00 -- STEP: 29876/36938 -- GLOBAL_STEP: 879450 | > loss_text_ce: 0.040056340396404266 (0.03350973300838921) | > loss_mel_ce: 2.8091044425964355 (nan) | > loss: 0.01130619365721941 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13664970058468362) | > loader_time: 0.0046 (0.004682072826714387)  --> TIME: 2025-02-25 17:40:09 -- STEP: 29926/36938 -- GLOBAL_STEP: 879500 | > loss_text_ce: 0.026483681052923203 (0.03351000896561955) | > loss_mel_ce: 1.8516292572021484 (nan) | > loss: 0.007452829275280237 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13664864991857206) | > loader_time: 0.0045 (0.004682017011629046)  --> TIME: 2025-02-25 17:40:19 -- STEP: 29976/36938 -- GLOBAL_STEP: 879550 | > loss_text_ce: 0.0331895686686039 (0.033507524595435656) | > loss_mel_ce: 0.1473795622587204 (nan) | > loss: 0.000716544222086668 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13665619596723497) | > loader_time: 0.0046 (0.004681997245745789)  --> TIME: 2025-02-25 17:40:28 -- STEP: 30026/36938 -- GLOBAL_STEP: 879600 | > loss_text_ce: 0.02476920373737812 (0.03350655700956689) | > loss_mel_ce: 2.9300148487091064 (nan) | > loss: 0.011725334450602531 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.1366510258233567) | > loader_time: 0.0046 (0.004681888025573835)  --> TIME: 2025-02-25 17:40:37 -- STEP: 30076/36938 -- GLOBAL_STEP: 879650 | > loss_text_ce: 0.03929053619503975 (0.03350723267672586) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.1366484146959069) | > loader_time: 0.0044 (0.004681824202993484)  --> TIME: 2025-02-25 17:40:46 -- STEP: 30126/36938 -- GLOBAL_STEP: 879700 | > loss_text_ce: 0.03691911697387695 (0.03350850336122179) | > loss_mel_ce: 2.7346410751342773 (nan) | > loss: 0.0109982555732131 (nan) | > current_lr: 5e-06 | > step_time: 0.1833 (0.13665015241788298) | > loader_time: 0.0068 (0.004681822765018306)  --> TIME: 2025-02-25 17:40:55 -- STEP: 30176/36938 -- GLOBAL_STEP: 879750 | > loss_text_ce: 0.03845422342419624 (0.03350848949014701) | > loss_mel_ce: 2.539853572845459 (nan) | > loss: 0.01023138128221035 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13664047623078746) | > loader_time: 0.0046 (0.0046818044159091035)  --> TIME: 2025-02-25 17:41:04 -- STEP: 30226/36938 -- GLOBAL_STEP: 879800 | > loss_text_ce: 0.03484165668487549 (0.03350917120412179) | > loss_mel_ce: 1.9120631217956543 (nan) | > loss: 0.007725812960416079 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1366401777713581) | > loader_time: 0.0045 (0.004681682875365721)  --> TIME: 2025-02-25 17:41:13 -- STEP: 30276/36938 -- GLOBAL_STEP: 879850 | > loss_text_ce: 0.035533927381038666 (0.03350854038859104) | > loss_mel_ce: 2.2134292125701904 (nan) | > loss: 0.008924457244575024 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13662734761202064) | > loader_time: 0.0045 (0.004681616167140778)  --> TIME: 2025-02-25 17:41:23 -- STEP: 30326/36938 -- GLOBAL_STEP: 879900 | > loss_text_ce: 0.03350043296813965 (0.03350815761440416) | > loss_mel_ce: 2.9755239486694336 (nan) | > loss: 0.01194057334214449 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.13663223641517522) | > loader_time: 0.0048 (0.004681563861670541)  --> TIME: 2025-02-25 17:41:32 -- STEP: 30376/36938 -- GLOBAL_STEP: 879950 | > loss_text_ce: 0.03574903681874275 (0.03350656699835732) | > loss_mel_ce: 2.3787713050842285 (nan) | > loss: 0.0095814298838377 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13663422447585732) | > loader_time: 0.0046 (0.004681429911325353)  --> TIME: 2025-02-25 17:41:42 -- STEP: 30426/36938 -- GLOBAL_STEP: 880000 | > loss_text_ce: 0.03467116132378578 (0.033504550159412005) | > loss_mel_ce: 2.653327703475952 (nan) | > loss: 0.010666662827134132 (nan) | > current_lr: 5e-06 | > step_time: 0.1865 (0.13664379792516412) | > loader_time: 0.0049 (0.004681346645755149) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_880000.pth  --> TIME: 2025-02-25 17:41:57 -- STEP: 30476/36938 -- GLOBAL_STEP: 880050 | > loss_text_ce: 0.03008211962878704 (0.03350419492273992) | > loss_mel_ce: 1.5008844137191772 (nan) | > loss: 0.006075264420360327 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13664566279865684) | > loader_time: 0.0046 (0.004681368131683861)  --> TIME: 2025-02-25 17:42:06 -- STEP: 30526/36938 -- GLOBAL_STEP: 880100 | > loss_text_ce: 0.041723474860191345 (0.03350292663345276) | > loss_mel_ce: 1.711761474609375 (nan) | > loss: 0.00695827417075634 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.13663234895329782) | > loader_time: 0.0046 (0.0046812633632996314)  --> TIME: 2025-02-25 17:42:15 -- STEP: 30576/36938 -- GLOBAL_STEP: 880150 | > loss_text_ce: 0.04078831151127815 (0.033505016874100534) | > loss_mel_ce: 1.5804548263549805 (nan) | > loss: 0.00643350463360548 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13663205158205072) | > loader_time: 0.0052 (0.004681259417084709)  --> TIME: 2025-02-25 17:42:25 -- STEP: 30626/36938 -- GLOBAL_STEP: 880200 | > loss_text_ce: 0.028462115675210953 (0.03350532655456077) | > loss_mel_ce: 3.0556442737579346 (nan) | > loss: 0.012238518334925175 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.13663196504555777) | > loader_time: 0.0046 (0.004681274875797039)  --> TIME: 2025-02-25 17:42:34 -- STEP: 30676/36938 -- GLOBAL_STEP: 880250 | > loss_text_ce: 0.04442150518298149 (0.03350530338988753) | > loss_mel_ce: 2.849705696105957 (nan) | > loss: 0.011484632268548012 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13662707762886575) | > loader_time: 0.0046 (0.0046812313090155155)  --> TIME: 2025-02-25 17:42:43 -- STEP: 30726/36938 -- GLOBAL_STEP: 880300 | > loss_text_ce: 0.01515371073037386 (0.03350622514287638) | > loss_mel_ce: 1.3778722286224365 (nan) | > loss: 0.0055278814397752285 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13662246590688498) | > loader_time: 0.0045 (0.004681132946723288)  --> TIME: 2025-02-25 17:42:53 -- STEP: 30776/36938 -- GLOBAL_STEP: 880350 | > loss_text_ce: 0.025902334600687027 (0.03350585256939224) | > loss_mel_ce: 2.342158317565918 (nan) | > loss: 0.009397066198289394 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13663275633225044) | > loader_time: 0.0053 (0.0046812491367970496)  --> TIME: 2025-02-25 17:43:02 -- STEP: 30826/36938 -- GLOBAL_STEP: 880400 | > loss_text_ce: 0.0358794741332531 (0.033507975710334995) | > loss_mel_ce: 0.9688720703125 (nan) | > loss: 0.003987109754234552 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13663618886972628) | > loader_time: 0.0046 (0.004681341050856512)  --> TIME: 2025-02-25 17:43:11 -- STEP: 30876/36938 -- GLOBAL_STEP: 880450 | > loss_text_ce: 0.03734593465924263 (0.0335064923854898) | > loss_mel_ce: 2.9707283973693848 (nan) | > loss: 0.011936803348362446 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.1366326719356445) | > loader_time: 0.0046 (0.004681213028088992)  --> TIME: 2025-02-25 17:43:21 -- STEP: 30926/36938 -- GLOBAL_STEP: 880500 | > loss_text_ce: 0.03102451004087925 (0.03350612950412308) | > loss_mel_ce: 3.2981700897216797 (nan) | > loss: 0.013211090117692947 (nan) | > current_lr: 5e-06 | > step_time: 0.1699 (0.13663325373432317) | > loader_time: 0.0047 (0.0046811481346436526)  --> TIME: 2025-02-25 17:43:30 -- STEP: 30976/36938 -- GLOBAL_STEP: 880550 | > loss_text_ce: 0.026367971673607826 (0.03350735019488596) | > loss_mel_ce: 1.9692198038101196 (nan) | > loss: 0.007918999530375004 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.1366427421169574) | > loader_time: 0.0047 (0.004681270515684722)  --> TIME: 2025-02-25 17:43:40 -- STEP: 31026/36938 -- GLOBAL_STEP: 880600 | > loss_text_ce: 0.034763153642416 (0.033506635455275845) | > loss_mel_ce: 1.3944648504257202 (nan) | > loss: 0.005671539809554815 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13664555964583208) | > loader_time: 0.0045 (0.004681232388519182)  --> TIME: 2025-02-25 17:43:49 -- STEP: 31076/36938 -- GLOBAL_STEP: 880650 | > loss_text_ce: 0.027801938354969025 (0.03350437820137056) | > loss_mel_ce: 3.110391139984131 (nan) | > loss: 0.012453148141503334 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13664117245650073) | > loader_time: 0.0046 (0.004681174390517608)  --> TIME: 2025-02-25 17:43:58 -- STEP: 31126/36938 -- GLOBAL_STEP: 880700 | > loss_text_ce: 0.031515661627054214 (0.03350273266921532) | > loss_mel_ce: 3.0446255207061768 (nan) | > loss: 0.0122069101780653 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13664202381281715) | > loader_time: 0.0047 (0.004681241831711743)  --> TIME: 2025-02-25 17:44:08 -- STEP: 31176/36938 -- GLOBAL_STEP: 880750 | > loss_text_ce: 0.04101348668336868 (0.0335014431145517) | > loss_mel_ce: 3.05712890625 (nan) | > loss: 0.012294216081500053 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.1366490646757026) | > loader_time: 0.0046 (0.0046812979447587505)  --> TIME: 2025-02-25 17:44:17 -- STEP: 31226/36938 -- GLOBAL_STEP: 880800 | > loss_text_ce: 0.032692041248083115 (0.03350243426982008) | > loss_mel_ce: 2.947892427444458 (nan) | > loss: 0.01182771660387516 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13664582505901277) | > loader_time: 0.0045 (0.004681212381497149)  --> TIME: 2025-02-25 17:44:26 -- STEP: 31276/36938 -- GLOBAL_STEP: 880850 | > loss_text_ce: 0.0245329812169075 (0.03350098581010544) | > loss_mel_ce: 2.2060956954956055 (nan) | > loss: 0.008851701393723488 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13664603997167105) | > loader_time: 0.0045 (0.00468123428716973)  --> TIME: 2025-02-25 17:44:35 -- STEP: 31326/36938 -- GLOBAL_STEP: 880900 | > loss_text_ce: 0.03266681730747223 (0.033498589789701314) | > loss_mel_ce: 2.292903184890747 (nan) | > loss: 0.009228453040122986 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13663836070604146) | > loader_time: 0.0045 (0.004681152333277398)  --> TIME: 2025-02-25 17:44:45 -- STEP: 31376/36938 -- GLOBAL_STEP: 880950 | > loss_text_ce: 0.030344059690833092 (0.033500149563052854) | > loss_mel_ce: 2.5347769260406494 (nan) | > loss: 0.010179052129387856 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13663769551742205) | > loader_time: 0.0047 (0.00468100883230025)  --> TIME: 2025-02-25 17:44:54 -- STEP: 31426/36938 -- GLOBAL_STEP: 881000 | > loss_text_ce: 0.03208950161933899 (0.0334995434520977) | > loss_mel_ce: 2.410945177078247 (nan) | > loss: 0.009694582782685757 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13663595561681255) | > loader_time: 0.0046 (0.004681006604070013)  --> TIME: 2025-02-25 17:45:03 -- STEP: 31476/36938 -- GLOBAL_STEP: 881050 | > loss_text_ce: 0.046093642711639404 (0.033500583280366106) | > loss_mel_ce: 0.12967953085899353 (nan) | > loss: 0.0006975126452744007 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.1366339636085315) | > loader_time: 0.0045 (0.004680901762036033)  --> TIME: 2025-02-25 17:45:12 -- STEP: 31526/36938 -- GLOBAL_STEP: 881100 | > loss_text_ce: 0.024296840652823448 (0.03349926682125994) | > loss_mel_ce: 2.36379337310791 (nan) | > loss: 0.009476548992097378 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.1366264242391726) | > loader_time: 0.0046 (0.004680749676232537)  --> TIME: 2025-02-25 17:45:22 -- STEP: 31576/36938 -- GLOBAL_STEP: 881150 | > loss_text_ce: 0.03572958707809448 (0.033499943695188134) | > loss_mel_ce: 3.2156031131744385 (nan) | > loss: 0.012902114540338516 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.1366310508033116) | > loader_time: 0.0047 (0.004680638732202185)  --> TIME: 2025-02-25 17:45:31 -- STEP: 31626/36938 -- GLOBAL_STEP: 881200 | > loss_text_ce: 0.032895419746637344 (0.03350053301246193) | > loss_mel_ce: 2.878321886062622 (nan) | > loss: 0.011552450247108936 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.1366245468208751) | > loader_time: 0.0047 (0.004680558859129681)  --> TIME: 2025-02-25 17:45:40 -- STEP: 31676/36938 -- GLOBAL_STEP: 881250 | > loss_text_ce: 0.028291387483477592 (0.03350060096881191) | > loss_mel_ce: 2.8621952533721924 (nan) | > loss: 0.011470185592770576 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13662214782114748) | > loader_time: 0.0046 (0.004680485861788231)  --> TIME: 2025-02-25 17:45:50 -- STEP: 31726/36938 -- GLOBAL_STEP: 881300 | > loss_text_ce: 0.035485390573740005 (0.0335018579831788) | > loss_mel_ce: 2.4606266021728516 (nan) | > loss: 0.009905206970870495 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13662914748691607) | > loader_time: 0.0046 (0.004680437442897591)  --> TIME: 2025-02-25 17:45:59 -- STEP: 31776/36938 -- GLOBAL_STEP: 881350 | > loss_text_ce: 0.030846575275063515 (0.03350311916901001) | > loss_mel_ce: 2.8797168731689453 (nan) | > loss: 0.011549855582416058 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13664005939123613) | > loader_time: 0.0045 (0.004680420674406126)  --> TIME: 2025-02-25 17:46:09 -- STEP: 31826/36938 -- GLOBAL_STEP: 881400 | > loss_text_ce: 0.03806542977690697 (0.033503142650526584) | > loss_mel_ce: 2.8728456497192383 (nan) | > loss: 0.011551234871149063 (nan) | > current_lr: 5e-06 | > step_time: 0.1796 (0.1366393080469653) | > loader_time: 0.0048 (0.004680304601296879)  --> TIME: 2025-02-25 17:46:18 -- STEP: 31876/36938 -- GLOBAL_STEP: 881450 | > loss_text_ce: 0.03391439467668533 (0.033502583451011966) | > loss_mel_ce: 2.0967702865600586 (nan) | > loss: 0.008455098606646061 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13664491610569401) | > loader_time: 0.0046 (0.0046802811228296775)  --> TIME: 2025-02-25 17:46:27 -- STEP: 31926/36938 -- GLOBAL_STEP: 881500 | > loss_text_ce: 0.03554132580757141 (0.033502441515678855) | > loss_mel_ce: 0.8718380928039551 (nan) | > loss: 0.0036007121670991182 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1366467849806351) | > loader_time: 0.0045 (0.0046802598686435486)  --> TIME: 2025-02-25 17:46:37 -- STEP: 31976/36938 -- GLOBAL_STEP: 881550 | > loss_text_ce: 0.03340671584010124 (0.033502313137187836) | > loss_mel_ce: 2.7351253032684326 (nan) | > loss: 0.010986238718032837 (nan) | > current_lr: 5e-06 | > step_time: 0.181 (0.13665442327156166) | > loader_time: 0.0048 (0.004680291082908524)  --> TIME: 2025-02-25 17:46:47 -- STEP: 32026/36938 -- GLOBAL_STEP: 881600 | > loss_text_ce: 0.029122978448867798 (0.033501623765075796) | > loss_mel_ce: 3.1009721755981445 (nan) | > loss: 0.012421013787388802 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13665891263915575) | > loader_time: 0.0046 (0.004680252094551071)  --> TIME: 2025-02-25 17:46:56 -- STEP: 32076/36938 -- GLOBAL_STEP: 881650 | > loss_text_ce: 0.0368180014193058 (0.033502105405925235) | > loss_mel_ce: 3.070321798324585 (nan) | > loss: 0.012329920195043087 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13665643846888959) | > loader_time: 0.0046 (0.004680330222020742)  --> TIME: 2025-02-25 17:47:05 -- STEP: 32126/36938 -- GLOBAL_STEP: 881700 | > loss_text_ce: 0.028176307678222656 (0.0334998616736924) | > loss_mel_ce: 3.1388766765594482 (nan) | > loss: 0.01256767101585865 (nan) | > current_lr: 5e-06 | > step_time: 0.1621 (0.13665935584722863) | > loader_time: 0.0046 (0.0046804660671133915)  --> TIME: 2025-02-25 17:47:14 -- STEP: 32176/36938 -- GLOBAL_STEP: 881750 | > loss_text_ce: 0.031112393364310265 (0.03350087671500954) | > loss_mel_ce: 2.8027656078338623 (nan) | > loss: 0.011245548725128174 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1366554863532333) | > loader_time: 0.0047 (0.004680370207096059)  --> TIME: 2025-02-25 17:47:24 -- STEP: 32226/36938 -- GLOBAL_STEP: 881800 | > loss_text_ce: 0.04616521671414375 (0.033500346098423825) | > loss_mel_ce: 3.1496422290802 (nan) | > loss: 0.012681776657700539 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13665570068542734) | > loader_time: 0.0046 (0.004680304740946458)  --> TIME: 2025-02-25 17:47:33 -- STEP: 32276/36938 -- GLOBAL_STEP: 881850 | > loss_text_ce: 0.03214069828391075 (0.033499814142175253) | > loss_mel_ce: 2.9433352947235107 (nan) | > loss: 0.01180744543671608 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13665519484298708) | > loader_time: 0.0045 (0.0046803028865113816)  --> TIME: 2025-02-25 17:47:42 -- STEP: 32326/36938 -- GLOBAL_STEP: 881900 | > loss_text_ce: 0.0328458771109581 (0.03350008669030019) | > loss_mel_ce: 2.8378243446350098 (nan) | > loss: 0.011391549371182919 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13665479085664042) | > loader_time: 0.0045 (0.004680186563557999)  --> TIME: 2025-02-25 17:47:52 -- STEP: 32376/36938 -- GLOBAL_STEP: 881950 | > loss_text_ce: 0.02737247943878174 (0.03350100942324349) | > loss_mel_ce: 3.057371139526367 (nan) | > loss: 0.012241045944392681 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.13665380390567486) | > loader_time: 0.0049 (0.004680071630859634)  --> TIME: 2025-02-25 17:48:01 -- STEP: 32426/36938 -- GLOBAL_STEP: 882000 | > loss_text_ce: 0.0369068905711174 (0.03350298033605805) | > loss_mel_ce: 3.5921404361724854 (nan) | > loss: 0.014400982297956944 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.1366546156669176) | > loader_time: 0.0046 (0.0046799644567740795)  --> TIME: 2025-02-25 17:48:10 -- STEP: 32476/36938 -- GLOBAL_STEP: 882050 | > loss_text_ce: 0.041064146906137466 (0.03350437974702605) | > loss_mel_ce: 2.626253128051758 (nan) | > loss: 0.010584593750536442 (nan) | > current_lr: 5e-06 | > step_time: 0.1083 (0.136654926405) | > loader_time: 0.0045 (0.004679831014887578)  --> TIME: 2025-02-25 17:48:19 -- STEP: 32526/36938 -- GLOBAL_STEP: 882100 | > loss_text_ce: 0.02852884866297245 (0.03350517817104672) | > loss_mel_ce: 2.6229441165924072 (nan) | > loss: 0.010521719232201576 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1366458258446754) | > loader_time: 0.0045 (0.004679719900238984)  --> TIME: 2025-02-25 17:48:29 -- STEP: 32576/36938 -- GLOBAL_STEP: 882150 | > loss_text_ce: 0.04087860882282257 (0.03350472216748127) | > loss_mel_ce: 3.2125957012176514 (nan) | > loss: 0.012910612858831882 (nan) | > current_lr: 5e-06 | > step_time: 0.1883 (0.13664503223564511) | > loader_time: 0.0049 (0.004679705223072231)  --> TIME: 2025-02-25 17:48:38 -- STEP: 32626/36938 -- GLOBAL_STEP: 882200 | > loss_text_ce: 0.039471209049224854 (0.033503592461305215) | > loss_mel_ce: 1.1809335947036743 (nan) | > loss: 0.004842876922339201 (nan) | > current_lr: 5e-06 | > step_time: 0.1451 (0.13664382450980792) | > loader_time: 0.0046 (0.004679653862767432)  --> TIME: 2025-02-25 17:48:48 -- STEP: 32676/36938 -- GLOBAL_STEP: 882250 | > loss_text_ce: 0.04220878332853317 (0.03350229804457195) | > loss_mel_ce: 3.229290246963501 (nan) | > loss: 0.012982139363884926 (nan) | > current_lr: 5e-06 | > step_time: 0.1831 (0.1366515770246256) | > loader_time: 0.0048 (0.004679641498610479)  --> TIME: 2025-02-25 17:48:57 -- STEP: 32726/36938 -- GLOBAL_STEP: 882300 | > loss_text_ce: 0.032617662101984024 (0.03350064334029474) | > loss_mel_ce: 3.269333839416504 (nan) | > loss: 0.013102982193231583 (nan) | > current_lr: 5e-06 | > step_time: 0.1778 (0.13664841523695187) | > loader_time: 0.0049 (0.004679548283598915)  --> TIME: 2025-02-25 17:49:06 -- STEP: 32776/36938 -- GLOBAL_STEP: 882350 | > loss_text_ce: 0.02428436651825905 (0.03349972838297323) | > loss_mel_ce: 2.306687116622925 (nan) | > loss: 0.009249887429177761 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13665070237984098) | > loader_time: 0.0047 (0.0046795212134285846)  --> TIME: 2025-02-25 17:49:16 -- STEP: 32826/36938 -- GLOBAL_STEP: 882400 | > loss_text_ce: 0.04589381441473961 (0.03349826832631069) | > loss_mel_ce: 3.3470661640167236 (nan) | > loss: 0.013464127667248249 (nan) | > current_lr: 5e-06 | > step_time: 0.1505 (0.13666073264075002) | > loader_time: 0.0047 (0.004679575892040331)  --> TIME: 2025-02-25 17:49:25 -- STEP: 32876/36938 -- GLOBAL_STEP: 882450 | > loss_text_ce: 0.045947570353746414 (0.033498080697936895) | > loss_mel_ce: 3.361550807952881 (nan) | > loss: 0.01352181937545538 (nan) | > current_lr: 5e-06 | > step_time: 0.1835 (0.13665553487584284) | > loader_time: 0.0048 (0.004679481940252622)  --> TIME: 2025-02-25 17:49:35 -- STEP: 32926/36938 -- GLOBAL_STEP: 882500 | > loss_text_ce: 0.02792413718998432 (0.033499012119186135) | > loss_mel_ce: 2.4409308433532715 (nan) | > loss: 0.009797044098377228 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.13666824790373105) | > loader_time: 0.0045 (0.004679486194430531)  --> TIME: 2025-02-25 17:49:44 -- STEP: 32976/36938 -- GLOBAL_STEP: 882550 | > loss_text_ce: 0.03711291775107384 (0.03349795736855857) | > loss_mel_ce: 2.8988826274871826 (nan) | > loss: 0.011650776490569115 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13666790772241968) | > loader_time: 0.0047 (0.004679539310937919)  --> TIME: 2025-02-25 17:49:53 -- STEP: 33026/36938 -- GLOBAL_STEP: 882600 | > loss_text_ce: 0.04045907035470009 (0.033498080686421135) | > loss_mel_ce: 2.8846852779388428 (nan) | > loss: 0.011607716791331768 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13666202075143588) | > loader_time: 0.0047 (0.004679453414104268)  --> TIME: 2025-02-25 17:50:02 -- STEP: 33076/36938 -- GLOBAL_STEP: 882650 | > loss_text_ce: 0.026158936321735382 (0.03349968139907184) | > loss_mel_ce: 2.765629529953003 (nan) | > loss: 0.011078527197241783 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1366536696929703) | > loader_time: 0.0045 (0.004679382726782543)  --> TIME: 2025-02-25 17:50:12 -- STEP: 33126/36938 -- GLOBAL_STEP: 882700 | > loss_text_ce: 0.03336992487311363 (0.03349859370965679) | > loss_mel_ce: 2.7356772422790527 (nan) | > loss: 0.010988282971084118 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.13665600126149244) | > loader_time: 0.0048 (0.004679281009263398)  --> TIME: 2025-02-25 17:50:21 -- STEP: 33176/36938 -- GLOBAL_STEP: 882750 | > loss_text_ce: 0.032164618372917175 (0.033497949420285264) | > loss_mel_ce: 2.634218454360962 (nan) | > loss: 0.010580885224044323 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.13665277087058794) | > loader_time: 0.0046 (0.004679173633567597)  --> TIME: 2025-02-25 17:50:30 -- STEP: 33226/36938 -- GLOBAL_STEP: 882800 | > loss_text_ce: 0.04618115350604057 (0.03349924338339052) | > loss_mel_ce: 1.75933837890625 (nan) | > loss: 0.007164760492742062 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13664335423960072) | > loader_time: 0.0046 (0.004679116868085843)  --> TIME: 2025-02-25 17:50:39 -- STEP: 33276/36938 -- GLOBAL_STEP: 882850 | > loss_text_ce: 0.020063180476427078 (0.0334984028739286) | > loss_mel_ce: 2.6425621509552 (nan) | > loss: 0.010565973818302155 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.1366358150411549) | > loader_time: 0.0046 (0.004679134157446494)  --> TIME: 2025-02-25 17:50:48 -- STEP: 33326/36938 -- GLOBAL_STEP: 882900 | > loss_text_ce: 0.03892836347222328 (0.03349791464213547) | > loss_mel_ce: 2.6416659355163574 (nan) | > loss: 0.010637279599905014 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13664117266076742) | > loader_time: 0.0046 (0.004679159629332755)  --> TIME: 2025-02-25 17:50:58 -- STEP: 33376/36938 -- GLOBAL_STEP: 882950 | > loss_text_ce: 0.03301205486059189 (0.033496439589455494) | > loss_mel_ce: 2.4095699787139893 (nan) | > loss: 0.009692787192761898 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13664061088947074) | > loader_time: 0.0053 (0.004679315349342472)  --> TIME: 2025-02-25 17:51:07 -- STEP: 33426/36938 -- GLOBAL_STEP: 883000 | > loss_text_ce: 0.030162116512656212 (0.03349785885381233) | > loss_mel_ce: 2.236269474029541 (nan) | > loss: 0.008993776515126228 (nan) | > current_lr: 5e-06 | > step_time: 0.1614 (0.13663898382110895) | > loader_time: 0.0049 (0.004679380053500379)  --> TIME: 2025-02-25 17:51:17 -- STEP: 33476/36938 -- GLOBAL_STEP: 883050 | > loss_text_ce: 0.03545107692480087 (0.03349734270549126) | > loss_mel_ce: 2.383854389190674 (nan) | > loss: 0.00960041955113411 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.13664757149922216) | > loader_time: 0.0047 (0.004679325981826113)  --> TIME: 2025-02-25 17:51:26 -- STEP: 33526/36938 -- GLOBAL_STEP: 883100 | > loss_text_ce: 0.034903235733509064 (0.03349605466940343) | > loss_mel_ce: 2.970743417739868 (nan) | > loss: 0.011927169747650623 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13665099935628797) | > loader_time: 0.0046 (0.004679271410069508)  --> TIME: 2025-02-25 17:51:35 -- STEP: 33576/36938 -- GLOBAL_STEP: 883150 | > loss_text_ce: 0.028027256950736046 (0.03349361709810668) | > loss_mel_ce: 3.291412830352783 (nan) | > loss: 0.013172382488846779 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.1366449182434585) | > loader_time: 0.0048 (0.00467917788219018)  --> TIME: 2025-02-25 17:51:44 -- STEP: 33626/36938 -- GLOBAL_STEP: 883200 | > loss_text_ce: 0.038316793739795685 (0.03349340813587181) | > loss_mel_ce: 2.0318081378936768 (nan) | > loss: 0.008214781992137432 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13664291769732212) | > loader_time: 0.0046 (0.004679176926946076)  --> TIME: 2025-02-25 17:51:54 -- STEP: 33676/36938 -- GLOBAL_STEP: 883250 | > loss_text_ce: 0.03258233889937401 (0.03349090691607522) | > loss_mel_ce: 2.422240972518921 (nan) | > loss: 0.009741363115608692 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13663630801859009) | > loader_time: 0.0046 (0.004679300189230621)  --> TIME: 2025-02-25 17:52:03 -- STEP: 33726/36938 -- GLOBAL_STEP: 883300 | > loss_text_ce: 0.03156648948788643 (0.03349032637548907) | > loss_mel_ce: 1.0127214193344116 (nan) | > loss: 0.004143999889492989 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1366287635143677) | > loader_time: 0.0046 (0.0046793126638607135)  --> TIME: 2025-02-25 17:52:12 -- STEP: 33776/36938 -- GLOBAL_STEP: 883350 | > loss_text_ce: 0.03413301333785057 (0.03349117866782011) | > loss_mel_ce: 0.044864654541015625 (nan) | > loss: 0.0003134828293696046 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.1366281728716826) | > loader_time: 0.0046 (0.004679409602647231)  --> TIME: 2025-02-25 17:52:21 -- STEP: 33826/36938 -- GLOBAL_STEP: 883400 | > loss_text_ce: 0.04108481481671333 (0.03349275315143627) | > loss_mel_ce: 2.633283853530884 (nan) | > loss: 0.01061257440596819 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13663454451341245) | > loader_time: 0.0047 (0.0046793966102182585)  --> TIME: 2025-02-25 17:52:31 -- STEP: 33876/36938 -- GLOBAL_STEP: 883450 | > loss_text_ce: 0.025541136041283607 (0.03349101287928618) | > loss_mel_ce: 2.526949167251587 (nan) | > loss: 0.0101289302110672 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13663441518470865) | > loader_time: 0.0046 (0.004679345777738814)  --> TIME: 2025-02-25 17:52:40 -- STEP: 33926/36938 -- GLOBAL_STEP: 883500 | > loss_text_ce: 0.03370637446641922 (0.03349194618735646) | > loss_mel_ce: 3.0549707412719727 (nan) | > loss: 0.012256655842065811 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.13663498374824393) | > loader_time: 0.0048 (0.004679389012026423)  --> TIME: 2025-02-25 17:52:49 -- STEP: 33976/36938 -- GLOBAL_STEP: 883550 | > loss_text_ce: 0.033078741282224655 (0.03349280320396842) | > loss_mel_ce: 3.156290054321289 (nan) | > loss: 0.012656225822865963 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13663511883799775) | > loader_time: 0.0049 (0.0046793896084761175)  --> TIME: 2025-02-25 17:52:59 -- STEP: 34026/36938 -- GLOBAL_STEP: 883600 | > loss_text_ce: 0.02440931461751461 (0.033490314348584564) | > loss_mel_ce: 2.2281858921051025 (nan) | > loss: 0.008938870392739773 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13663236852999036) | > loader_time: 0.0067 (0.004679461134557014)  --> TIME: 2025-02-25 17:53:08 -- STEP: 34076/36938 -- GLOBAL_STEP: 883650 | > loss_text_ce: 0.027894753962755203 (0.03349156765654138) | > loss_mel_ce: 2.930081367492676 (nan) | > loss: 0.011738001368939877 (nan) | > current_lr: 5e-06 | > step_time: 0.1528 (0.1366244624626292) | > loader_time: 0.0047 (0.004679360360603294)  --> TIME: 2025-02-25 17:53:17 -- STEP: 34126/36938 -- GLOBAL_STEP: 883700 | > loss_text_ce: 0.0354151651263237 (0.03348979840817272) | > loss_mel_ce: 3.6888909339904785 (nan) | > loss: 0.014778993092477322 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13662028245613375) | > loader_time: 0.0045 (0.004679409419289274)  --> TIME: 2025-02-25 17:53:26 -- STEP: 34176/36938 -- GLOBAL_STEP: 883750 | > loss_text_ce: 0.02739543840289116 (0.033489599814931624) | > loss_mel_ce: 2.521402597427368 (nan) | > loss: 0.010114278644323349 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.13661674011763691) | > loader_time: 0.0048 (0.004679350768414751)  --> TIME: 2025-02-25 17:53:35 -- STEP: 34226/36938 -- GLOBAL_STEP: 883800 | > loss_text_ce: 0.035973191261291504 (0.033489916356864734) | > loss_mel_ce: 2.9036102294921875 (nan) | > loss: 0.011665013618767262 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13662075823641845) | > loader_time: 0.0047 (0.00467935595124799)  --> TIME: 2025-02-25 17:53:45 -- STEP: 34276/36938 -- GLOBAL_STEP: 883850 | > loss_text_ce: 0.034070685505867004 (0.03348926481103289) | > loss_mel_ce: 3.018393039703369 (nan) | > loss: 0.012112951837480068 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1366234662402306) | > loader_time: 0.0045 (0.004679285286326742)  --> TIME: 2025-02-25 17:53:54 -- STEP: 34326/36938 -- GLOBAL_STEP: 883900 | > loss_text_ce: 0.035247478634119034 (0.03348912122136115) | > loss_mel_ce: 2.155949831008911 (nan) | > loss: 0.008695228025317192 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13662038178375202) | > loader_time: 0.0045 (0.004679271219521672)  --> TIME: 2025-02-25 17:54:03 -- STEP: 34376/36938 -- GLOBAL_STEP: 883950 | > loss_text_ce: 0.0323074646294117 (0.03349009535857009) | > loss_mel_ce: 2.8610177040100098 (nan) | > loss: 0.011481449007987976 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1366130010469134) | > loader_time: 0.0046 (0.004679249661562994)  --> TIME: 2025-02-25 17:54:13 -- STEP: 34426/36938 -- GLOBAL_STEP: 884000 | > loss_text_ce: 0.03842337056994438 (0.033491029795964716) | > loss_mel_ce: 3.042457103729248 (nan) | > loss: 0.01222571637481451 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.13661796603565352) | > loader_time: 0.0047 (0.00467923146970721)  --> TIME: 2025-02-25 17:54:22 -- STEP: 34476/36938 -- GLOBAL_STEP: 884050 | > loss_text_ce: 0.03522934764623642 (0.03349149673253835) | > loss_mel_ce: 2.5549254417419434 (nan) | > loss: 0.01027839258313179 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.1366202096012955) | > loader_time: 0.0046 (0.004679305334353395)  --> TIME: 2025-02-25 17:54:32 -- STEP: 34526/36938 -- GLOBAL_STEP: 884100 | > loss_text_ce: 0.02637806534767151 (0.03349318465418413) | > loss_mel_ce: 2.849503993988037 (nan) | > loss: 0.011412231251597404 (nan) | > current_lr: 5e-06 | > step_time: 0.1753 (0.13662564379518466) | > loader_time: 0.0047 (0.0046792944896153895)  --> TIME: 2025-02-25 17:54:41 -- STEP: 34576/36938 -- GLOBAL_STEP: 884150 | > loss_text_ce: 0.030151639133691788 (0.03349149505275821) | > loss_mel_ce: 2.7928714752197266 (nan) | > loss: 0.011202473193407059 (nan) | > current_lr: 5e-06 | > step_time: 0.1769 (0.13662501111150743) | > loader_time: 0.0057 (0.004679372297123235)  --> TIME: 2025-02-25 17:54:50 -- STEP: 34626/36938 -- GLOBAL_STEP: 884200 | > loss_text_ce: 0.03130725026130676 (0.033490489208903414) | > loss_mel_ce: 2.359130859375 (nan) | > loss: 0.009485865943133831 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13661923493475728) | > loader_time: 0.0046 (0.004679350728200366)  --> TIME: 2025-02-25 17:54:59 -- STEP: 34676/36938 -- GLOBAL_STEP: 884250 | > loss_text_ce: 0.02426772564649582 (0.033490084865778105) | > loss_mel_ce: 2.7083096504211426 (nan) | > loss: 0.01084356103092432 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13661519601581767) | > loader_time: 0.0046 (0.00467940976234959)  --> TIME: 2025-02-25 17:55:09 -- STEP: 34726/36938 -- GLOBAL_STEP: 884300 | > loss_text_ce: 0.030736230313777924 (0.033490064129342044) | > loss_mel_ce: 3.2316508293151855 (nan) | > loss: 0.01294598076492548 (nan) | > current_lr: 5e-06 | > step_time: 0.148 (0.1366254352914942) | > loader_time: 0.0046 (0.0046794138038198695)  --> TIME: 2025-02-25 17:55:18 -- STEP: 34776/36938 -- GLOBAL_STEP: 884350 | > loss_text_ce: 0.03921210393309593 (0.03349006967112898) | > loss_mel_ce: 3.2269012928009033 (nan) | > loss: 0.01296076737344265 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.13662448683112915) | > loader_time: 0.0048 (0.004679387112664199)  --> TIME: 2025-02-25 17:55:28 -- STEP: 34826/36938 -- GLOBAL_STEP: 884400 | > loss_text_ce: 0.031906042248010635 (0.0334892684980434) | > loss_mel_ce: 2.701305627822876 (nan) | > loss: 0.01084607932716608 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13663027872272665) | > loader_time: 0.0046 (0.004679420879815057)  --> TIME: 2025-02-25 17:55:37 -- STEP: 34876/36938 -- GLOBAL_STEP: 884450 | > loss_text_ce: 0.04077671840786934 (0.03349358570424007) | > loss_mel_ce: 2.0925185680389404 (nan) | > loss: 0.008465457707643509 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13662936930427177) | > loader_time: 0.0045 (0.004679340139853437)  --> TIME: 2025-02-25 17:55:46 -- STEP: 34926/36938 -- GLOBAL_STEP: 884500 | > loss_text_ce: 0.03836642950773239 (0.033496902663152237) | > loss_mel_ce: 2.647000789642334 (nan) | > loss: 0.010656218975782394 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1366255210859415) | > loader_time: 0.0047 (0.00467926155610883)  --> TIME: 2025-02-25 17:55:55 -- STEP: 34976/36938 -- GLOBAL_STEP: 884550 | > loss_text_ce: 0.027236055582761765 (0.03349636653626724) | > loss_mel_ce: 3.2624833583831787 (nan) | > loss: 0.013054442591965199 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.13662259751710887) | > loader_time: 0.0047 (0.00467923553515656)  --> TIME: 2025-02-25 17:56:05 -- STEP: 35026/36938 -- GLOBAL_STEP: 884600 | > loss_text_ce: 0.03860950842499733 (0.033497587658712843) | > loss_mel_ce: 1.5078091621398926 (nan) | > loss: 0.0061365822330117226 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13662593017427282) | > loader_time: 0.0046 (0.004679177807065292)  --> TIME: 2025-02-25 17:56:14 -- STEP: 35076/36938 -- GLOBAL_STEP: 884650 | > loss_text_ce: 0.04329748451709747 (0.03349780268840334) | > loss_mel_ce: 0.159637451171875 (nan) | > loss: 0.0008052974008023739 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13662984876489012) | > loader_time: 0.0047 (0.004679137032637482)  --> TIME: 2025-02-25 17:56:23 -- STEP: 35126/36938 -- GLOBAL_STEP: 884700 | > loss_text_ce: 0.03669791296124458 (0.0334982896804421) | > loss_mel_ce: 2.965379238128662 (nan) | > loss: 0.011913005262613297 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13662920916928117) | > loader_time: 0.0045 (0.004679067547674381)  --> TIME: 2025-02-25 17:56:33 -- STEP: 35176/36938 -- GLOBAL_STEP: 884750 | > loss_text_ce: 0.03359569236636162 (0.03349888472574674) | > loss_mel_ce: 3.013606309890747 (nan) | > loss: 0.012092072516679764 (nan) | > current_lr: 5e-06 | > step_time: 0.1265 (0.13662412143143884) | > loader_time: 0.0047 (0.004678978699295229)  --> TIME: 2025-02-25 17:56:42 -- STEP: 35226/36938 -- GLOBAL_STEP: 884800 | > loss_text_ce: 0.033752281218767166 (0.033499419667340186) | > loss_mel_ce: 3.057532548904419 (nan) | > loss: 0.012267003767192364 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13662905949029513) | > loader_time: 0.0046 (0.004678913277647833)  --> TIME: 2025-02-25 17:56:52 -- STEP: 35276/36938 -- GLOBAL_STEP: 884850 | > loss_text_ce: 0.03492073714733124 (0.03350145511953497) | > loss_mel_ce: 2.8204143047332764 (nan) | > loss: 0.011330694891512394 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13662964702395858) | > loader_time: 0.0046 (0.004678820824323001)  --> TIME: 2025-02-25 17:57:01 -- STEP: 35326/36938 -- GLOBAL_STEP: 884900 | > loss_text_ce: 0.03677406162023544 (0.03350285890444722) | > loss_mel_ce: 2.9408841133117676 (nan) | > loss: 0.011816104874014854 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.1366317433692144) | > loader_time: 0.0049 (0.00467889160989248)  --> TIME: 2025-02-25 17:57:11 -- STEP: 35376/36938 -- GLOBAL_STEP: 884950 | > loss_text_ce: 0.0381220206618309 (0.033504223582360645) | > loss_mel_ce: 2.8455071449279785 (nan) | > loss: 0.011442973278462887 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.13663808249985016) | > loader_time: 0.0047 (0.004678914196239965)  --> TIME: 2025-02-25 17:57:20 -- STEP: 35426/36938 -- GLOBAL_STEP: 885000 | > loss_text_ce: 0.02630770578980446 (0.03350488870438671) | > loss_mel_ce: 1.7297141551971436 (nan) | > loss: 0.006968341302126646 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13663913228056337) | > loader_time: 0.0045 (0.004678937095713563)  --> TIME: 2025-02-25 17:57:29 -- STEP: 35476/36938 -- GLOBAL_STEP: 885050 | > loss_text_ce: 0.025546859949827194 (0.033505549235484634) | > loss_mel_ce: 1.8787841796875 (nan) | > loss: 0.007556869648396969 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13663441233764398) | > loader_time: 0.0046 (0.004679063252524173)  --> TIME: 2025-02-25 17:57:38 -- STEP: 35526/36938 -- GLOBAL_STEP: 885100 | > loss_text_ce: 0.03201546147465706 (0.03350515967080921) | > loss_mel_ce: 2.537871837615967 (nan) | > loss: 0.010197966359555721 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13662679359289018) | > loader_time: 0.0045 (0.004679009089335392)  --> TIME: 2025-02-25 17:57:47 -- STEP: 35576/36938 -- GLOBAL_STEP: 885150 | > loss_text_ce: 0.03538733348250389 (0.033504139083077984) | > loss_mel_ce: 2.521186351776123 (nan) | > loss: 0.010145134292542934 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13662086888621555) | > loader_time: 0.0053 (0.004679085700625028)  --> TIME: 2025-02-25 17:57:56 -- STEP: 35626/36938 -- GLOBAL_STEP: 885200 | > loss_text_ce: 0.031082870438694954 (0.033505183837622694) | > loss_mel_ce: 3.270566463470459 (nan) | > loss: 0.013101783581078053 (nan) | > current_lr: 5e-06 | > step_time: 0.1811 (0.1366140371971608) | > loader_time: 0.0048 (0.004679036141780317)  --> TIME: 2025-02-25 17:58:06 -- STEP: 35676/36938 -- GLOBAL_STEP: 885250 | > loss_text_ce: 0.033237673342227936 (0.03350379351334988) | > loss_mel_ce: 3.1951868534088135 (nan) | > loss: 0.012811209075152874 (nan) | > current_lr: 5e-06 | > step_time: 0.1751 (0.1366174768367027) | > loader_time: 0.0048 (0.004679184789173191)  --> TIME: 2025-02-25 17:58:15 -- STEP: 35726/36938 -- GLOBAL_STEP: 885300 | > loss_text_ce: 0.02980463020503521 (0.03350398452686704) | > loss_mel_ce: 2.4370028972625732 (nan) | > loss: 0.009788919240236282 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13661640751357054) | > loader_time: 0.0045 (0.004679228506316196)  --> TIME: 2025-02-25 17:58:24 -- STEP: 35776/36938 -- GLOBAL_STEP: 885350 | > loss_text_ce: 0.019390815868973732 (0.03350200153194188) | > loss_mel_ce: 2.2038638591766357 (nan) | > loss: 0.008822440169751644 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13661338677479762) | > loader_time: 0.0046 (0.004679199048252963)  --> TIME: 2025-02-25 17:58:33 -- STEP: 35826/36938 -- GLOBAL_STEP: 885400 | > loss_text_ce: 0.035094138234853745 (0.03350183996560385) | > loss_mel_ce: 0.03952789306640625 (nan) | > loss: 0.0002961191930808127 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13660960485472837) | > loader_time: 0.0046 (0.004679147085672028)  --> TIME: 2025-02-25 17:58:43 -- STEP: 35876/36938 -- GLOBAL_STEP: 885450 | > loss_text_ce: 0.02748151496052742 (0.033502334106847595) | > loss_mel_ce: 3.033381700515747 (nan) | > loss: 0.012146283872425556 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13661389926269532) | > loader_time: 0.0046 (0.004679098065740216)  --> TIME: 2025-02-25 17:58:52 -- STEP: 35926/36938 -- GLOBAL_STEP: 885500 | > loss_text_ce: 0.04069717228412628 (0.033503011723459986) | > loss_mel_ce: 2.5018310546875 (nan) | > loss: 0.010089398361742496 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13661648961288836) | > loader_time: 0.0046 (0.004679050841350296)  --> TIME: 2025-02-25 17:59:01 -- STEP: 35976/36938 -- GLOBAL_STEP: 885550 | > loss_text_ce: 0.042254701256752014 (0.03350282032468838) | > loss_mel_ce: 2.942455768585205 (nan) | > loss: 0.011844090186059475 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1366161138899504) | > loader_time: 0.0045 (0.004679068628088028)  --> TIME: 2025-02-25 17:59:11 -- STEP: 36026/36938 -- GLOBAL_STEP: 885600 | > loss_text_ce: 0.02947109565138817 (0.03350014066446061) | > loss_mel_ce: 2.524594783782959 (nan) | > loss: 0.010135183110833168 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13661591680947294) | > loader_time: 0.0046 (0.004679133690476276)  --> TIME: 2025-02-25 17:59:20 -- STEP: 36076/36938 -- GLOBAL_STEP: 885650 | > loss_text_ce: 0.03253050148487091 (0.033501085309288525) | > loss_mel_ce: 2.8577077388763428 (nan) | > loss: 0.011469200253486633 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13661313123694413) | > loader_time: 0.0045 (0.004679032791108904)  --> TIME: 2025-02-25 17:59:29 -- STEP: 36126/36938 -- GLOBAL_STEP: 885700 | > loss_text_ce: 0.034866850823163986 (0.03350069912595759) | > loss_mel_ce: 2.072568655014038 (nan) | > loss: 0.008362839929759502 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13660997504419134) | > loader_time: 0.0046 (0.004679026552485497)  --> TIME: 2025-02-25 17:59:38 -- STEP: 36176/36938 -- GLOBAL_STEP: 885750 | > loss_text_ce: 0.028733717277646065 (0.03349799155773032) | > loss_mel_ce: 2.377121686935425 (nan) | > loss: 0.009547045454382896 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13660060838095417) | > loader_time: 0.0045 (0.004679105902392979)  --> TIME: 2025-02-25 17:59:47 -- STEP: 36226/36938 -- GLOBAL_STEP: 885800 | > loss_text_ce: 0.04013129696249962 (0.033497661529110316) | > loss_mel_ce: 2.315807580947876 (nan) | > loss: 0.00934896431863308 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13659326042178752) | > loader_time: 0.0045 (0.004679026809302314)  --> TIME: 2025-02-25 17:59:56 -- STEP: 36276/36938 -- GLOBAL_STEP: 885850 | > loss_text_ce: 0.04635823145508766 (0.03349746963040961) | > loss_mel_ce: 2.935987710952759 (nan) | > loss: 0.011834707111120224 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13659069809291513) | > loader_time: 0.0046 (0.004678945292158284)  --> TIME: 2025-02-25 18:00:06 -- STEP: 36326/36938 -- GLOBAL_STEP: 885900 | > loss_text_ce: 0.03470410406589508 (0.03349873568166443) | > loss_mel_ce: 2.37178373336792 (nan) | > loss: 0.00954955630004406 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1365942985405176) | > loader_time: 0.0045 (0.004678881241218125)  --> TIME: 2025-02-25 18:00:15 -- STEP: 36376/36938 -- GLOBAL_STEP: 885950 | > loss_text_ce: 0.03603349253535271 (0.03349903684429675) | > loss_mel_ce: 2.070039749145508 (nan) | > loss: 0.00835743360221386 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13659482976748896) | > loader_time: 0.0045 (0.004678809566762046)  --> TIME: 2025-02-25 18:00:24 -- STEP: 36426/36938 -- GLOBAL_STEP: 886000 | > loss_text_ce: 0.033936407417058945 (0.03349859044501401) | > loss_mel_ce: 2.537506580352783 (nan) | > loss: 0.010204140096902847 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13658689899498913) | > loader_time: 0.0045 (0.004678707823671907)  --> TIME: 2025-02-25 18:00:33 -- STEP: 36476/36938 -- GLOBAL_STEP: 886050 | > loss_text_ce: 0.03445904701948166 (0.03349661061810163) | > loss_mel_ce: 3.03173828125 (nan) | > loss: 0.012167450971901417 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13657772695303022) | > loader_time: 0.0045 (0.004678629942533256)  --> TIME: 2025-02-25 18:00:42 -- STEP: 36526/36938 -- GLOBAL_STEP: 886100 | > loss_text_ce: 0.03948430344462395 (0.033497277272621605) | > loss_mel_ce: 2.9833645820617676 (nan) | > loss: 0.011995432898402214 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.13658190483736898) | > loader_time: 0.0046 (0.004678542581476847)  --> TIME: 2025-02-25 18:00:51 -- STEP: 36576/36938 -- GLOBAL_STEP: 886150 | > loss_text_ce: 0.030728042125701904 (0.03349601370805463) | > loss_mel_ce: 2.6750903129577637 (nan) | > loss: 0.010737375356256962 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13657630288407852) | > loader_time: 0.0045 (0.0046785965509718985)  --> TIME: 2025-02-25 18:01:01 -- STEP: 36626/36938 -- GLOBAL_STEP: 886200 | > loss_text_ce: 0.03100517764687538 (0.033496718167336424) | > loss_mel_ce: 3.0625405311584473 (nan) | > loss: 0.012275975197553635 (nan) | > current_lr: 5e-06 | > step_time: 0.1508 (0.13657451560611333) | > loader_time: 0.0046 (0.004678573410768643)  --> TIME: 2025-02-25 18:01:10 -- STEP: 36676/36938 -- GLOBAL_STEP: 886250 | > loss_text_ce: 0.030179917812347412 (0.03349702766604224) | > loss_mel_ce: 2.1111857891082764 (nan) | > loss: 0.008497484028339386 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13657167619233285) | > loader_time: 0.0045 (0.004678490364976664)  --> TIME: 2025-02-25 18:01:19 -- STEP: 36726/36938 -- GLOBAL_STEP: 886300 | > loss_text_ce: 0.031306758522987366 (0.033495931021684626) | > loss_mel_ce: 0.0098876953125 (nan) | > loss: 0.00016347006021533161 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13657214076393084) | > loader_time: 0.0044 (0.004678493821594303)  --> TIME: 2025-02-25 18:01:28 -- STEP: 36776/36938 -- GLOBAL_STEP: 886350 | > loss_text_ce: 0.03722526878118515 (0.03349439119392871) | > loss_mel_ce: 2.939279556274414 (nan) | > loss: 0.01181152742356062 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13656608693906097) | > loader_time: 0.0046 (0.00467843465605894)  --> TIME: 2025-02-25 18:01:37 -- STEP: 36826/36938 -- GLOBAL_STEP: 886400 | > loss_text_ce: 0.03029930591583252 (0.0334971934988758) | > loss_mel_ce: 2.631734609603882 (nan) | > loss: 0.010563627816736698 (nan) | > current_lr: 5e-06 | > step_time: 0.15 (0.13656342548154154) | > loader_time: 0.0047 (0.004678379451536431)  --> TIME: 2025-02-25 18:01:47 -- STEP: 36876/36938 -- GLOBAL_STEP: 886450 | > loss_text_ce: 0.034763436764478683 (0.033496377891835016) | > loss_mel_ce: 2.7619450092315674 (nan) | > loss: 0.011098049581050873 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13656023545900745) | > loader_time: 0.0046 (0.004678321099356281)  --> TIME: 2025-02-25 18:01:56 -- STEP: 36926/36938 -- GLOBAL_STEP: 886500 | > loss_text_ce: 0.039567068219184875 (0.033499017060290154) | > loss_mel_ce: 3.144446849822998 (nan) | > loss: 0.012634975835680962 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.1365690722868656) | > loader_time: 0.0045 (0.0046782459306450586)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.02089501961623088 (+0.00019753333365564452) | > avg_loss_text_ce: 0.03268051227683773 (-0.00026843359343489) | > avg_loss_mel_ce: 2.521518524625514 (-0.009626056888316636) | > avg_loss: 2.5541990399360683 (-0.00989448197997067) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_886512.pth  > EPOCH: 25/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-25 18:02:23)   --> TIME: 2025-02-25 18:02:31 -- STEP: 38/36938 -- GLOBAL_STEP: 886550 | > loss_text_ce: 0.03263938054442406 (0.03428467573892129) | > loss_mel_ce: 2.9566869735717773 (nan) | > loss: 0.011862406507134438 (nan) | > current_lr: 5e-06 | > step_time: 0.1762 (0.1344052678660343) | > loader_time: 0.0051 (0.0048070016660188356)  --> TIME: 2025-02-25 18:02:40 -- STEP: 88/36938 -- GLOBAL_STEP: 886600 | > loss_text_ce: 0.01808822713792324 (0.032983111746778544) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13572806119918826) | > loader_time: 0.0065 (0.004807496612722223)  --> TIME: 2025-02-25 18:02:49 -- STEP: 138/36938 -- GLOBAL_STEP: 886650 | > loss_text_ce: 0.03665562719106674 (0.03317241646025492) | > loss_mel_ce: 0.8953463435173035 (nan) | > loss: 0.003698420710861683 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.1337323206058447) | > loader_time: 0.0045 (0.0047412661538607835)  --> TIME: 2025-02-25 18:02:58 -- STEP: 188/36938 -- GLOBAL_STEP: 886700 | > loss_text_ce: 0.0273286122828722 (0.0334787730801296) | > loss_mel_ce: 2.0438177585601807 (nan) | > loss: 0.008218836039304733 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13357808488480596) | > loader_time: 0.0046 (0.004769628352307258)  --> TIME: 2025-02-25 18:03:07 -- STEP: 238/36938 -- GLOBAL_STEP: 886750 | > loss_text_ce: 0.03586670383810997 (0.033805450174112274) | > loss_mel_ce: 2.569580078125 (nan) | > loss: 0.010339075699448586 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1326392247897237) | > loader_time: 0.0046 (0.00475086484636579)  --> TIME: 2025-02-25 18:03:16 -- STEP: 288/36938 -- GLOBAL_STEP: 886800 | > loss_text_ce: 0.027030523866415024 (0.033795373106840984) | > loss_mel_ce: 3.093930244445801 (nan) | > loss: 0.012384765781462193 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13349218997690418) | > loader_time: 0.0048 (0.004756025142139858)  --> TIME: 2025-02-25 18:03:26 -- STEP: 338/36938 -- GLOBAL_STEP: 886850 | > loss_text_ce: 0.023120583966374397 (0.03369872594946588) | > loss_mel_ce: 3.0052080154418945 (nan) | > loss: 0.01201717834919691 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13493271762802758) | > loader_time: 0.0047 (0.00475854422213763)  --> TIME: 2025-02-25 18:03:35 -- STEP: 388/36938 -- GLOBAL_STEP: 886900 | > loss_text_ce: 0.03598112612962723 (0.03370487272316952) | > loss_mel_ce: 2.8582539558410645 (nan) | > loss: 0.011485060676932335 (nan) | > current_lr: 5e-06 | > step_time: 0.1705 (0.13435154968930277) | > loader_time: 0.0047 (0.004747951768108247)  --> TIME: 2025-02-25 18:03:44 -- STEP: 438/36938 -- GLOBAL_STEP: 886950 | > loss_text_ce: 0.02773689664900303 (0.033608411353710815) | > loss_mel_ce: 2.7552032470703125 (nan) | > loss: 0.011043413542211056 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.1344967356555538) | > loader_time: 0.0046 (0.00473708104869546)  --> TIME: 2025-02-25 18:03:53 -- STEP: 488/36938 -- GLOBAL_STEP: 887000 | > loss_text_ce: 0.02983073703944683 (0.033539543710801645) | > loss_mel_ce: 2.528860092163086 (nan) | > loss: 0.01015353575348854 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13443607873603955) | > loader_time: 0.0047 (0.004734296290600883)  --> TIME: 2025-02-25 18:04:02 -- STEP: 538/36938 -- GLOBAL_STEP: 887050 | > loss_text_ce: 0.03181568533182144 (0.03341062759398911) | > loss_mel_ce: 2.5015525817871094 (nan) | > loss: 0.010053049772977829 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.1344411093949386) | > loader_time: 0.0047 (0.004733429965476561)  --> TIME: 2025-02-25 18:04:11 -- STEP: 588/36938 -- GLOBAL_STEP: 887100 | > loss_text_ce: 0.0318760983645916 (0.03342171205163354) | > loss_mel_ce: 3.2719733715057373 (nan) | > loss: 0.013110514730215073 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13405566839944752) | > loader_time: 0.0045 (0.004732160341171989)  --> TIME: 2025-02-25 18:04:20 -- STEP: 638/36938 -- GLOBAL_STEP: 887150 | > loss_text_ce: 0.04057740047574043 (0.03337890253516151) | > loss_mel_ce: 2.7351555824279785 (nan) | > loss: 0.011014814488589764 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13401556538190215) | > loader_time: 0.0046 (0.004726459240091257)  --> TIME: 2025-02-25 18:04:30 -- STEP: 688/36938 -- GLOBAL_STEP: 887200 | > loss_text_ce: 0.04409312084317207 (0.033468501359430074) | > loss_mel_ce: 2.7253403663635254 (nan) | > loss: 0.010989815928041935 (nan) | > current_lr: 5e-06 | > step_time: 0.1719 (0.13403563166773605) | > loader_time: 0.0049 (0.004722493332485822)  --> TIME: 2025-02-25 18:04:39 -- STEP: 738/36938 -- GLOBAL_STEP: 887250 | > loss_text_ce: 0.03646765649318695 (0.03349075997298765) | > loss_mel_ce: 2.780393600463867 (nan) | > loss: 0.01117802131921053 (nan) | > current_lr: 5e-06 | > step_time: 0.1848 (0.13452000217386065) | > loader_time: 0.005 (0.004716484204217349)  --> TIME: 2025-02-25 18:04:49 -- STEP: 788/36938 -- GLOBAL_STEP: 887300 | > loss_text_ce: 0.03480633348226547 (0.03351635961358428) | > loss_mel_ce: 3.205698013305664 (nan) | > loss: 0.012859144248068333 (nan) | > current_lr: 5e-06 | > step_time: 0.1664 (0.1349206500247045) | > loader_time: 0.0048 (0.004710721182944207)  --> TIME: 2025-02-25 18:04:58 -- STEP: 838/36938 -- GLOBAL_STEP: 887350 | > loss_text_ce: 0.03244486823678017 (0.033525396348477315) | > loss_mel_ce: 2.7734978199005127 (nan) | > loss: 0.011134694330394268 (nan) | > current_lr: 5e-06 | > step_time: 0.1905 (0.13521350482767697) | > loader_time: 0.0048 (0.004704311241113483)  --> TIME: 2025-02-25 18:05:07 -- STEP: 888/36938 -- GLOBAL_STEP: 887400 | > loss_text_ce: 0.03255493938922882 (0.033567917345486024) | > loss_mel_ce: 2.7336294651031494 (nan) | > loss: 0.010976922698318958 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13523699088139587) | > loader_time: 0.0045 (0.004699394778088407)  --> TIME: 2025-02-25 18:05:17 -- STEP: 938/36938 -- GLOBAL_STEP: 887450 | > loss_text_ce: 0.034389350563287735 (0.03358918509142263) | > loss_mel_ce: 1.962890625 (nan) | > loss: 0.007925714366137981 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13512951351686353) | > loader_time: 0.0045 (0.004698041151327366)  --> TIME: 2025-02-25 18:05:26 -- STEP: 988/36938 -- GLOBAL_STEP: 887500 | > loss_text_ce: 0.03676778823137283 (0.03358271840850443) | > loss_mel_ce: 2.6131043434143066 (nan) | > loss: 0.01051536574959755 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.1353177763672495) | > loader_time: 0.0046 (0.004696866037391941)  --> TIME: 2025-02-25 18:05:35 -- STEP: 1038/36938 -- GLOBAL_STEP: 887550 | > loss_text_ce: 0.03197852894663811 (0.033549043402858164) | > loss_mel_ce: 1.34490966796875 (nan) | > loss: 0.005463842302560806 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13539820943953695) | > loader_time: 0.0055 (0.0047012693849831895)  --> TIME: 2025-02-25 18:05:45 -- STEP: 1088/36938 -- GLOBAL_STEP: 887600 | > loss_text_ce: 0.042365338653326035 (0.03355821722053568) | > loss_mel_ce: 2.6181211471557617 (nan) | > loss: 0.010557486675679684 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13562267295577965) | > loader_time: 0.0045 (0.004703406025381652)  --> TIME: 2025-02-25 18:05:55 -- STEP: 1138/36938 -- GLOBAL_STEP: 887650 | > loss_text_ce: 0.03581543639302254 (0.03357077670252517) | > loss_mel_ce: 2.134596109390259 (nan) | > loss: 0.008612744510173798 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13604075774576638) | > loader_time: 0.0061 (0.004705945930078706)  --> TIME: 2025-02-25 18:06:04 -- STEP: 1188/36938 -- GLOBAL_STEP: 887700 | > loss_text_ce: 0.026446564123034477 (0.03360519242341244) | > loss_mel_ce: 2.7736735343933105 (nan) | > loss: 0.011111588217318058 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13602680891049848) | > loader_time: 0.0054 (0.004705438910911385)  --> TIME: 2025-02-25 18:06:13 -- STEP: 1238/36938 -- GLOBAL_STEP: 887750 | > loss_text_ce: 0.026978837326169014 (0.03358215505485941) | > loss_mel_ce: 2.765282154083252 (nan) | > loss: 0.01108040101826191 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.1360271545142081) | > loader_time: 0.0049 (0.004702619859205506)  --> TIME: 2025-02-25 18:06:23 -- STEP: 1288/36938 -- GLOBAL_STEP: 887800 | > loss_text_ce: 0.030100826174020767 (0.03350781557375783) | > loss_mel_ce: 2.71760630607605 (nan) | > loss: 0.010903600603342056 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13620200531082874) | > loader_time: 0.0046 (0.004701329499298003)  --> TIME: 2025-02-25 18:06:32 -- STEP: 1338/36938 -- GLOBAL_STEP: 887850 | > loss_text_ce: 0.04076983779668808 (0.03345373162919003) | > loss_mel_ce: 1.9982409477233887 (nan) | > loss: 0.00809131283313036 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13617932012441097) | > loader_time: 0.0046 (0.004700573035954361)  --> TIME: 2025-02-25 18:06:41 -- STEP: 1388/36938 -- GLOBAL_STEP: 887900 | > loss_text_ce: 0.03425542637705803 (0.0333703975703148) | > loss_mel_ce: 1.8633301258087158 (nan) | > loss: 0.007530101574957371 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1361812153879437) | > loader_time: 0.0046 (0.004699520487606697)  --> TIME: 2025-02-25 18:06:51 -- STEP: 1438/36938 -- GLOBAL_STEP: 887950 | > loss_text_ce: 0.039056502282619476 (0.03332892146935582) | > loss_mel_ce: 3.5857770442962646 (nan) | > loss: 0.014384261332452297 (nan) | > current_lr: 5e-06 | > step_time: 0.1827 (0.13623016218814155) | > loader_time: 0.0048 (0.004700418473616427)  --> TIME: 2025-02-25 18:07:00 -- STEP: 1488/36938 -- GLOBAL_STEP: 888000 | > loss_text_ce: 0.04821240156888962 (0.033323689581706055) | > loss_mel_ce: 2.4116125106811523 (nan) | > loss: 0.009761210530996323 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13621573951295646) | > loader_time: 0.0046 (0.004697864254315694)  --> TIME: 2025-02-25 18:07:09 -- STEP: 1538/36938 -- GLOBAL_STEP: 888050 | > loss_text_ce: 0.03680923953652382 (0.033337342611578114) | > loss_mel_ce: 2.011535167694092 (nan) | > loss: 0.008128351531922817 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13624873722483205) | > loader_time: 0.0046 (0.004698698480355565)  --> TIME: 2025-02-25 18:07:19 -- STEP: 1588/36938 -- GLOBAL_STEP: 888100 | > loss_text_ce: 0.0313308909535408 (0.03337655783518294) | > loss_mel_ce: 3.0835297107696533 (nan) | > loss: 0.012360558845102787 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13618412846582081) | > loader_time: 0.0047 (0.0046985712399710885)  --> TIME: 2025-02-25 18:07:28 -- STEP: 1638/36938 -- GLOBAL_STEP: 888150 | > loss_text_ce: 0.028818821534514427 (0.03340148855789904) | > loss_mel_ce: 0.03271484375 (nan) | > loss: 0.00024418122484348714 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13630347359602565) | > loader_time: 0.0069 (0.004697710166484002)  --> TIME: 2025-02-25 18:07:38 -- STEP: 1688/36938 -- GLOBAL_STEP: 888200 | > loss_text_ce: 0.036600351333618164 (0.03339374100765145) | > loss_mel_ce: 1.8785560131072998 (nan) | > loss: 0.007599827367812395 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.1364521992997536) | > loader_time: 0.0045 (0.004697746857647641)  --> TIME: 2025-02-25 18:07:47 -- STEP: 1738/36938 -- GLOBAL_STEP: 888250 | > loss_text_ce: 0.025232573971152306 (0.03338925655174597) | > loss_mel_ce: 0.05504608154296875 (nan) | > loss: 0.00031856613350100815 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13646979010969376) | > loader_time: 0.0046 (0.0046978792469301155)  --> TIME: 2025-02-25 18:07:56 -- STEP: 1788/36938 -- GLOBAL_STEP: 888300 | > loss_text_ce: 0.022627869620919228 (0.03339937177249195) | > loss_mel_ce: 1.474712610244751 (nan) | > loss: 0.005941827315837145 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.1365033128117556) | > loader_time: 0.0045 (0.004695230949111694)  --> TIME: 2025-02-25 18:08:06 -- STEP: 1838/36938 -- GLOBAL_STEP: 888350 | > loss_text_ce: 0.02849695459008217 (0.03338190077492908) | > loss_mel_ce: 1.7215086221694946 (nan) | > loss: 0.006944466847926378 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1364406552226552) | > loader_time: 0.0045 (0.004696602790218178)  --> TIME: 2025-02-25 18:08:15 -- STEP: 1888/36938 -- GLOBAL_STEP: 888400 | > loss_text_ce: 0.04108564183115959 (0.03338407762437437) | > loss_mel_ce: 2.6721792221069336 (nan) | > loss: 0.010766925290226936 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13655709500535065) | > loader_time: 0.0046 (0.004696037557165496)  --> TIME: 2025-02-25 18:08:24 -- STEP: 1938/36938 -- GLOBAL_STEP: 888450 | > loss_text_ce: 0.02886275388300419 (0.03335989380478316) | > loss_mel_ce: 2.58052659034729 (nan) | > loss: 0.010354720056056976 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13650129798519112) | > loader_time: 0.0045 (0.004695232684644255)  --> TIME: 2025-02-25 18:08:33 -- STEP: 1988/36938 -- GLOBAL_STEP: 888500 | > loss_text_ce: 0.03693649172782898 (0.033356536019094975) | > loss_mel_ce: 2.974943161010742 (nan) | > loss: 0.011951903812587261 (nan) | > current_lr: 5e-06 | > step_time: 0.1439 (0.13641340871691948) | > loader_time: 0.0047 (0.004696662637069666)  --> TIME: 2025-02-25 18:08:43 -- STEP: 2038/36938 -- GLOBAL_STEP: 888550 | > loss_text_ce: 0.03410504013299942 (0.03334486858016268) | > loss_mel_ce: 2.2154159545898438 (nan) | > loss: 0.008926670998334885 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13640386152782202) | > loader_time: 0.0045 (0.004699247157140845)  --> TIME: 2025-02-25 18:08:52 -- STEP: 2088/36938 -- GLOBAL_STEP: 888600 | > loss_text_ce: 0.03381437808275223 (0.03335011602732637) | > loss_mel_ce: 3.040372610092163 (nan) | > loss: 0.012199155986309052 (nan) | > current_lr: 5e-06 | > step_time: 0.1527 (0.13648398408944587) | > loader_time: 0.0048 (0.004698766716595348)  --> TIME: 2025-02-25 18:09:02 -- STEP: 2138/36938 -- GLOBAL_STEP: 888650 | > loss_text_ce: 0.029472529888153076 (0.033363366817534476) | > loss_mel_ce: 2.77585768699646 (nan) | > loss: 0.011132263578474522 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13647207877031547) | > loader_time: 0.0046 (0.004696462509005394)  --> TIME: 2025-02-25 18:09:11 -- STEP: 2188/36938 -- GLOBAL_STEP: 888700 | > loss_text_ce: 0.030890528112649918 (0.03337096281254307) | > loss_mel_ce: 2.25207781791687 (nan) | > loss: 0.00905939843505621 (nan) | > current_lr: 5e-06 | > step_time: 0.1694 (0.13651697879735472) | > loader_time: 0.0048 (0.004694862601090168)  --> TIME: 2025-02-25 18:09:20 -- STEP: 2238/36938 -- GLOBAL_STEP: 888750 | > loss_text_ce: 0.039081502705812454 (0.03334833709215132) | > loss_mel_ce: 2.861623525619507 (nan) | > loss: 0.011510735377669334 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13652962025411433) | > loader_time: 0.0046 (0.004694063392891422)  --> TIME: 2025-02-25 18:09:30 -- STEP: 2288/36938 -- GLOBAL_STEP: 888800 | > loss_text_ce: 0.03434775769710541 (0.033362445384068545) | > loss_mel_ce: 2.762768030166626 (nan) | > loss: 0.011099666357040405 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.1364717198001755) | > loader_time: 0.0046 (0.004693148957265834)  --> TIME: 2025-02-25 18:09:39 -- STEP: 2338/36938 -- GLOBAL_STEP: 888850 | > loss_text_ce: 0.03427617624402046 (0.033364598625765754) | > loss_mel_ce: 1.9984616041183472 (nan) | > loss: 0.008066420443356037 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13646354656937382) | > loader_time: 0.0045 (0.004692899660741899)  --> TIME: 2025-02-25 18:09:48 -- STEP: 2388/36938 -- GLOBAL_STEP: 888900 | > loss_text_ce: 0.03956468775868416 (0.03333811966976902) | > loss_mel_ce: 0.05218505859375 (nan) | > loss: 0.0003640862996689975 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.1362951987951844) | > loader_time: 0.0048 (0.004693197245573872)  --> TIME: 2025-02-25 18:09:57 -- STEP: 2438/36938 -- GLOBAL_STEP: 888950 | > loss_text_ce: 0.02833864651620388 (0.03333958978531691) | > loss_mel_ce: 2.9116454124450684 (nan) | > loss: 0.011666604317724705 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13628617938763787) | > loader_time: 0.0066 (0.004694556877004789)  --> TIME: 2025-02-25 18:10:06 -- STEP: 2488/36938 -- GLOBAL_STEP: 889000 | > loss_text_ce: 0.02491057850420475 (0.03332047902835218) | > loss_mel_ce: 0.057098388671875 (nan) | > loss: 0.0003254324255976826 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13634010095305) | > loader_time: 0.0045 (0.0046948044054761105)  --> TIME: 2025-02-25 18:10:15 -- STEP: 2538/36938 -- GLOBAL_STEP: 889050 | > loss_text_ce: 0.045305922627449036 (0.03331472477938036) | > loss_mel_ce: 2.4404938220977783 (nan) | > loss: 0.009864285588264465 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13617608498362133) | > loader_time: 0.0047 (0.004693002095834951)  --> TIME: 2025-02-25 18:10:25 -- STEP: 2588/36938 -- GLOBAL_STEP: 889100 | > loss_text_ce: 0.036310598254203796 (0.03333229728624621) | > loss_mel_ce: 2.659058094024658 (nan) | > loss: 0.010695908218622208 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13618841771917345) | > loader_time: 0.0046 (0.004691987999439966)  --> TIME: 2025-02-25 18:10:34 -- STEP: 2638/36938 -- GLOBAL_STEP: 889150 | > loss_text_ce: 0.03087543696165085 (0.033339176077743116) | > loss_mel_ce: 2.663846015930176 (nan) | > loss: 0.010693339630961418 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13620092852536814) | > loader_time: 0.0046 (0.004692452255752241)  --> TIME: 2025-02-25 18:10:43 -- STEP: 2688/36938 -- GLOBAL_STEP: 889200 | > loss_text_ce: 0.032008543610572815 (0.033357291187885954) | > loss_mel_ce: 1.8599672317504883 (nan) | > loss: 0.007507840637117624 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13626443124598) | > loader_time: 0.0046 (0.004693157261326188)  --> TIME: 2025-02-25 18:10:53 -- STEP: 2738/36938 -- GLOBAL_STEP: 889250 | > loss_text_ce: 0.030148671939969063 (0.03336081887854549) | > loss_mel_ce: 2.2576076984405518 (nan) | > loss: 0.009078399278223515 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13625716762528806) | > loader_time: 0.0046 (0.004692677775815587)  --> TIME: 2025-02-25 18:11:02 -- STEP: 2788/36938 -- GLOBAL_STEP: 889300 | > loss_text_ce: 0.025575248524546623 (0.03336535793845375) | > loss_mel_ce: 2.9162421226501465 (nan) | > loss: 0.011673878878355026 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13638110687606156) | > loader_time: 0.0047 (0.0046927218286686575)  --> TIME: 2025-02-25 18:11:12 -- STEP: 2838/36938 -- GLOBAL_STEP: 889350 | > loss_text_ce: 0.034646451473236084 (0.03335003813346711) | > loss_mel_ce: 2.7834999561309814 (nan) | > loss: 0.011183121241629124 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.1363949003649734) | > loader_time: 0.0047 (0.004691827036781649)  --> TIME: 2025-02-25 18:11:21 -- STEP: 2888/36938 -- GLOBAL_STEP: 889400 | > loss_text_ce: 0.034124087542295456 (0.03333706490391055) | > loss_mel_ce: 2.8728575706481934 (nan) | > loss: 0.01153564266860485 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13639655998208877) | > loader_time: 0.0046 (0.004691018374672878)  --> TIME: 2025-02-25 18:11:30 -- STEP: 2938/36938 -- GLOBAL_STEP: 889450 | > loss_text_ce: 0.034807898104190826 (0.033335636609529984) | > loss_mel_ce: 2.6328022480010986 (nan) | > loss: 0.010585755109786987 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.1363818895419171) | > loader_time: 0.0045 (0.004690091897225676)  --> TIME: 2025-02-25 18:11:40 -- STEP: 2988/36938 -- GLOBAL_STEP: 889500 | > loss_text_ce: 0.03229101747274399 (0.03333562293926497) | > loss_mel_ce: 0.909332275390625 (nan) | > loss: 0.003736600512638688 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13638608123723148) | > loader_time: 0.0048 (0.004691009301258368)  --> TIME: 2025-02-25 18:11:49 -- STEP: 3038/36938 -- GLOBAL_STEP: 889550 | > loss_text_ce: 0.04044490307569504 (0.033379934605522514) | > loss_mel_ce: 3.4495596885681152 (nan) | > loss: 0.013849224895238876 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.13635386857176587) | > loader_time: 0.0047 (0.004690848344874415)  --> TIME: 2025-02-25 18:11:58 -- STEP: 3088/36938 -- GLOBAL_STEP: 889600 | > loss_text_ce: 0.0402284674346447 (0.03337746314219457) | > loss_mel_ce: 1.5462270975112915 (nan) | > loss: 0.006295458879321814 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.1363926366203193) | > loader_time: 0.0044 (0.0046895566381938555)  --> TIME: 2025-02-25 18:12:08 -- STEP: 3138/36938 -- GLOBAL_STEP: 889650 | > loss_text_ce: 0.030437257140874863 (0.03335398520355474) | > loss_mel_ce: 3.1682660579681396 (nan) | > loss: 0.012693267315626144 (nan) | > current_lr: 5e-06 | > step_time: 0.1679 (0.13634317539694094) | > loader_time: 0.0047 (0.0046879820492405746)  --> TIME: 2025-02-25 18:12:17 -- STEP: 3188/36938 -- GLOBAL_STEP: 889700 | > loss_text_ce: 0.031022554263472557 (0.03335299514061126) | > loss_mel_ce: 2.6244728565216064 (nan) | > loss: 0.010537681169807911 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13634908909779767) | > loader_time: 0.0045 (0.004688780535119977)  --> TIME: 2025-02-25 18:12:26 -- STEP: 3238/36938 -- GLOBAL_STEP: 889750 | > loss_text_ce: 0.04087599366903305 (0.033357352985919436) | > loss_mel_ce: 1.9702712297439575 (nan) | > loss: 0.007980743423104286 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13638320743779964) | > loader_time: 0.0045 (0.004688031515860118)  --> TIME: 2025-02-25 18:12:36 -- STEP: 3288/36938 -- GLOBAL_STEP: 889800 | > loss_text_ce: 0.028769472613930702 (0.03336026927698729) | > loss_mel_ce: 2.687103509902954 (nan) | > loss: 0.010777274146676064 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13635804422580458) | > loader_time: 0.0046 (0.004688502445708214)  --> TIME: 2025-02-25 18:12:45 -- STEP: 3338/36938 -- GLOBAL_STEP: 889850 | > loss_text_ce: 0.028969205915927887 (0.033369614640301096) | > loss_mel_ce: 2.9339942932128906 (nan) | > loss: 0.01175779290497303 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13631969665323973) | > loader_time: 0.0045 (0.0046872623385320125)  --> TIME: 2025-02-25 18:12:54 -- STEP: 3388/36938 -- GLOBAL_STEP: 889900 | > loss_text_ce: 0.0317089706659317 (0.03338397647652212) | > loss_mel_ce: 2.8020830154418945 (nan) | > loss: 0.01124520692974329 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.13628864590927514) | > loader_time: 0.0048 (0.00468703594230281)  --> TIME: 2025-02-25 18:13:03 -- STEP: 3438/36938 -- GLOBAL_STEP: 889950 | > loss_text_ce: 0.02819664403796196 (0.03336600852542937) | > loss_mel_ce: 2.124136447906494 (nan) | > loss: 0.008541004732251167 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1362678928386346) | > loader_time: 0.0047 (0.004687882218685435)  --> TIME: 2025-02-25 18:13:12 -- STEP: 3488/36938 -- GLOBAL_STEP: 890000 | > loss_text_ce: 0.039357442408800125 (0.033374743369235) | > loss_mel_ce: 2.7892985343933105 (nan) | > loss: 0.011224825866520405 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.13630353085218277) | > loader_time: 0.0048 (0.004688846955605589) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_890000.pth  --> TIME: 2025-02-25 18:13:28 -- STEP: 3538/36938 -- GLOBAL_STEP: 890050 | > loss_text_ce: 0.034376516938209534 (0.033374777828422755) | > loss_mel_ce: 2.700425863265991 (nan) | > loss: 0.010852391831576824 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13620651576935536) | > loader_time: 0.0045 (0.0046882409709803055)  --> TIME: 2025-02-25 18:13:37 -- STEP: 3588/36938 -- GLOBAL_STEP: 890100 | > loss_text_ce: 0.026199495419859886 (0.03336078083685655) | > loss_mel_ce: 1.34857177734375 (nan) | > loss: 0.005455441772937775 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13617458032525176) | > loader_time: 0.0046 (0.004687710284123571)  --> TIME: 2025-02-25 18:13:46 -- STEP: 3638/36938 -- GLOBAL_STEP: 890150 | > loss_text_ce: 0.03104228712618351 (0.033360470309310886) | > loss_mel_ce: 2.0256423950195312 (nan) | > loss: 0.008161447942256927 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13613977374579714) | > loader_time: 0.0048 (0.004687436994056625)  --> TIME: 2025-02-25 18:13:55 -- STEP: 3688/36938 -- GLOBAL_STEP: 890200 | > loss_text_ce: 0.025823427364230156 (0.03335623810655664) | > loss_mel_ce: 2.856959104537964 (nan) | > loss: 0.011439613997936249 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13615790281016577) | > loader_time: 0.0048 (0.004688478796187282)  --> TIME: 2025-02-25 18:14:05 -- STEP: 3738/36938 -- GLOBAL_STEP: 890250 | > loss_text_ce: 0.04146948084235191 (0.03335479301748804) | > loss_mel_ce: 1.456910252571106 (nan) | > loss: 0.0059459516778588295 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13619768874667001) | > loader_time: 0.0045 (0.004688932463215774)  --> TIME: 2025-02-25 18:14:14 -- STEP: 3788/36938 -- GLOBAL_STEP: 890300 | > loss_text_ce: 0.03138346970081329 (0.03336433303781424) | > loss_mel_ce: 2.2777862548828125 (nan) | > loss: 0.009163372218608856 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13615697225774182) | > loader_time: 0.0047 (0.004688957487768947)  --> TIME: 2025-02-25 18:14:23 -- STEP: 3838/36938 -- GLOBAL_STEP: 890350 | > loss_text_ce: 0.04075136408209801 (0.0333879162664679) | > loss_mel_ce: 3.0361015796661377 (nan) | > loss: 0.012209734879434109 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.1361425769024688) | > loader_time: 0.0045 (0.004689434967418696)  --> TIME: 2025-02-25 18:14:32 -- STEP: 3888/36938 -- GLOBAL_STEP: 890400 | > loss_text_ce: 0.040974415838718414 (0.033398020972302385) | > loss_mel_ce: 2.068636417388916 (nan) | > loss: 0.008371472358703613 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.13612932212068296) | > loader_time: 0.005 (0.004689830320852762)  --> TIME: 2025-02-25 18:14:42 -- STEP: 3938/36938 -- GLOBAL_STEP: 890450 | > loss_text_ce: 0.030657881870865822 (0.033403903259879276) | > loss_mel_ce: 2.9136476516723633 (nan) | > loss: 0.0116837527602911 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13610664274923884) | > loader_time: 0.0046 (0.0046890991602287725)  --> TIME: 2025-02-25 18:14:51 -- STEP: 3988/36938 -- GLOBAL_STEP: 890500 | > loss_text_ce: 0.03933731094002724 (0.03340427521004377) | > loss_mel_ce: 2.0697782039642334 (nan) | > loss: 0.008369507268071175 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13607407082526185) | > loader_time: 0.0045 (0.0046889663579590575)  --> TIME: 2025-02-25 18:15:00 -- STEP: 4038/36938 -- GLOBAL_STEP: 890550 | > loss_text_ce: 0.04021953046321869 (0.03341827165716442) | > loss_mel_ce: 2.202838182449341 (nan) | > loss: 0.008901023305952549 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13613518527854324) | > loader_time: 0.0045 (0.004688352272615614)  --> TIME: 2025-02-25 18:15:10 -- STEP: 4088/36938 -- GLOBAL_STEP: 890600 | > loss_text_ce: 0.030245831236243248 (0.033403241072158155) | > loss_mel_ce: 2.967233180999756 (nan) | > loss: 0.01189475879073143 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.1361793948596245) | > loader_time: 0.0047 (0.0046891449948812764)  --> TIME: 2025-02-25 18:15:19 -- STEP: 4138/36938 -- GLOBAL_STEP: 890650 | > loss_text_ce: 0.0414426364004612 (0.033411818129987815) | > loss_mel_ce: 2.8225066661834717 (nan) | > loss: 0.011364879086613655 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.1361195829077129) | > loader_time: 0.0046 (0.004689763685883722)  --> TIME: 2025-02-25 18:15:28 -- STEP: 4188/36938 -- GLOBAL_STEP: 890700 | > loss_text_ce: 0.029168691486120224 (0.03341886028047007) | > loss_mel_ce: 2.4653074741363525 (nan) | > loss: 0.00989871472120285 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13615877751293093) | > loader_time: 0.0046 (0.004690632323708655)  --> TIME: 2025-02-25 18:15:37 -- STEP: 4238/36938 -- GLOBAL_STEP: 890750 | > loss_text_ce: 0.032306425273418427 (0.03341675293401117) | > loss_mel_ce: 2.513568878173828 (nan) | > loss: 0.010102679952979088 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13610409351383154) | > loader_time: 0.0045 (0.004690541361457269)  --> TIME: 2025-02-25 18:15:47 -- STEP: 4288/36938 -- GLOBAL_STEP: 890800 | > loss_text_ce: 0.04244779795408249 (0.03343928369077507) | > loss_mel_ce: 2.5946621894836426 (nan) | > loss: 0.010464723221957684 (nan) | > current_lr: 5e-06 | > step_time: 0.1077 (0.13608880129767956) | > loader_time: 0.0044 (0.004689775740922376)  --> TIME: 2025-02-25 18:15:55 -- STEP: 4338/36938 -- GLOBAL_STEP: 890850 | > loss_text_ce: 0.03466837480664253 (0.03342958833355036) | > loss_mel_ce: 1.6486519575119019 (nan) | > loss: 0.006679842714220285 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13603910601815153) | > loader_time: 0.0047 (0.004689198092086452)  --> TIME: 2025-02-25 18:16:05 -- STEP: 4388/36938 -- GLOBAL_STEP: 890900 | > loss_text_ce: 0.02946965955197811 (0.033411710104313395) | > loss_mel_ce: 2.94118595123291 (nan) | > loss: 0.011788317002356052 (nan) | > current_lr: 5e-06 | > step_time: 0.1078 (0.13604461799020065) | > loader_time: 0.0045 (0.004690773941325175)  --> TIME: 2025-02-25 18:16:14 -- STEP: 4438/36938 -- GLOBAL_STEP: 890950 | > loss_text_ce: 0.030618561431765556 (0.033416243237512584) | > loss_mel_ce: 2.8962063789367676 (nan) | > loss: 0.01161438599228859 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.1360509233358699) | > loader_time: 0.0045 (0.004691551536415646)  --> TIME: 2025-02-25 18:16:24 -- STEP: 4488/36938 -- GLOBAL_STEP: 891000 | > loss_text_ce: 0.031414538621902466 (0.03341889823143626) | > loss_mel_ce: 2.8984577655792236 (nan) | > loss: 0.011626478284597397 (nan) | > current_lr: 5e-06 | > step_time: 0.1357 (0.13610002127560772) | > loader_time: 0.0046 (0.004691536095053109)  --> TIME: 2025-02-25 18:16:33 -- STEP: 4538/36938 -- GLOBAL_STEP: 891050 | > loss_text_ce: 0.0347956083714962 (0.03341418116362899) | > loss_mel_ce: 2.963829517364502 (nan) | > loss: 0.011899306438863277 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13614042191948922) | > loader_time: 0.0046 (0.004690645916864382)  --> TIME: 2025-02-25 18:16:43 -- STEP: 4588/36938 -- GLOBAL_STEP: 891100 | > loss_text_ce: 0.03001236356794834 (0.033415977658181106) | > loss_mel_ce: 2.856659173965454 (nan) | > loss: 0.011455046012997627 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.13613718711716744) | > loader_time: 0.0046 (0.004689724110683977)  --> TIME: 2025-02-25 18:16:52 -- STEP: 4638/36938 -- GLOBAL_STEP: 891150 | > loss_text_ce: 0.03623093664646149 (0.03342056708007182) | > loss_mel_ce: 2.7144775390625 (nan) | > loss: 0.010915511287748814 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.1361092001058365) | > loader_time: 0.0049 (0.004689421875819191)  --> TIME: 2025-02-25 18:17:01 -- STEP: 4688/36938 -- GLOBAL_STEP: 891200 | > loss_text_ce: 0.043330978602170944 (0.03340606126396328) | > loss_mel_ce: 2.5214264392852783 (nan) | > loss: 0.010177609510719776 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13613099836245415) | > loader_time: 0.0046 (0.004690082711978164)  --> TIME: 2025-02-25 18:17:10 -- STEP: 4738/36938 -- GLOBAL_STEP: 891250 | > loss_text_ce: 0.032906971871852875 (0.03342761659252504) | > loss_mel_ce: 2.116840124130249 (nan) | > loss: 0.008530743420124054 (nan) | > current_lr: 5e-06 | > step_time: 0.1634 (0.13614094242258673) | > loader_time: 0.0047 (0.004689424638760239)  --> TIME: 2025-02-25 18:17:20 -- STEP: 4788/36938 -- GLOBAL_STEP: 891300 | > loss_text_ce: 0.036464493721723557 (0.03342280470386451) | > loss_mel_ce: 0.02220662496984005 (nan) | > loss: 0.0002328219125047326 (nan) | > current_lr: 5e-06 | > step_time: 0.1062 (0.1361212979880798) | > loader_time: 0.0047 (0.004689559253335688)  --> TIME: 2025-02-25 18:17:29 -- STEP: 4838/36938 -- GLOBAL_STEP: 891350 | > loss_text_ce: 0.026899510994553566 (0.033410851127356894) | > loss_mel_ce: 2.1960110664367676 (nan) | > loss: 0.008821074850857258 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.1360989931251657) | > loader_time: 0.0045 (0.004688552609649059)  --> TIME: 2025-02-25 18:17:38 -- STEP: 4888/36938 -- GLOBAL_STEP: 891400 | > loss_text_ce: 0.035209763795137405 (0.033411972512607) | > loss_mel_ce: 3.1900079250335693 (nan) | > loss: 0.012798483483493328 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.13613231316721078) | > loader_time: 0.0046 (0.004688886934723668)  --> TIME: 2025-02-25 18:17:48 -- STEP: 4938/36938 -- GLOBAL_STEP: 891450 | > loss_text_ce: 0.036508675664663315 (0.033409530110393264) | > loss_mel_ce: 2.7343478202819824 (nan) | > loss: 0.010995462536811829 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.13612955497218018) | > loader_time: 0.0046 (0.0046884192296395375)  --> TIME: 2025-02-25 18:17:57 -- STEP: 4988/36938 -- GLOBAL_STEP: 891500 | > loss_text_ce: 0.034770842641592026 (0.033428225805385556) | > loss_mel_ce: 3.1467819213867188 (nan) | > loss: 0.012625209987163544 (nan) | > current_lr: 5e-06 | > step_time: 0.1835 (0.13613115670495626) | > loader_time: 0.0048 (0.004687688450098217)  --> TIME: 2025-02-25 18:18:06 -- STEP: 5038/36938 -- GLOBAL_STEP: 891550 | > loss_text_ce: 0.021687950938940048 (0.03343014052779575) | > loss_mel_ce: 0.1522216796875 (nan) | > loss: 0.0006901176529936492 (nan) | > current_lr: 5e-06 | > step_time: 0.1067 (0.1360987782147071) | > loader_time: 0.0055 (0.004688338232021461)  --> TIME: 2025-02-25 18:18:16 -- STEP: 5088/36938 -- GLOBAL_STEP: 891600 | > loss_text_ce: 0.03211401775479317 (0.033434354856649945) | > loss_mel_ce: 0.0692647323012352 (nan) | > loss: 0.0004022966604679823 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13613587467925425) | > loader_time: 0.0046 (0.004688014165035571)  --> TIME: 2025-02-25 18:18:25 -- STEP: 5138/36938 -- GLOBAL_STEP: 891650 | > loss_text_ce: 0.03001067414879799 (0.03342529530910681) | > loss_mel_ce: 2.2734005451202393 (nan) | > loss: 0.00914052128791809 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13609505307586314) | > loader_time: 0.0046 (0.004688908637103598)  --> TIME: 2025-02-25 18:18:34 -- STEP: 5188/36938 -- GLOBAL_STEP: 891700 | > loss_text_ce: 0.04254952445626259 (0.03342270356733664) | > loss_mel_ce: 2.453293561935425 (nan) | > loss: 0.00990414060652256 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13606776548700406) | > loader_time: 0.0046 (0.004689000437787233)  --> TIME: 2025-02-25 18:18:43 -- STEP: 5238/36938 -- GLOBAL_STEP: 891750 | > loss_text_ce: 0.03828001767396927 (0.03342845506234296) | > loss_mel_ce: 3.1006999015808105 (nan) | > loss: 0.012456269934773445 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13605208199367486) | > loader_time: 0.0048 (0.004689387029798566)  --> TIME: 2025-02-25 18:18:52 -- STEP: 5288/36938 -- GLOBAL_STEP: 891800 | > loss_text_ce: 0.03093673847615719 (0.03343121430420281) | > loss_mel_ce: 2.8127684593200684 (nan) | > loss: 0.011284545063972473 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13607347931335356) | > loader_time: 0.0046 (0.004688728189324093)  --> TIME: 2025-02-25 18:19:02 -- STEP: 5338/36938 -- GLOBAL_STEP: 891850 | > loss_text_ce: 0.032387688755989075 (0.03341809499352129) | > loss_mel_ce: 2.7474846839904785 (nan) | > loss: 0.011031240224838257 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.13607764476334572) | > loader_time: 0.0047 (0.004688276696088999)  --> TIME: 2025-02-25 18:19:11 -- STEP: 5388/36938 -- GLOBAL_STEP: 891900 | > loss_text_ce: 0.03324436396360397 (0.03341440603385244) | > loss_mel_ce: 3.1921002864837646 (nan) | > loss: 0.012798987329006195 (nan) | > current_lr: 5e-06 | > step_time: 0.2012 (0.13604949909223027) | > loader_time: 0.0049 (0.004687854955173356)  --> TIME: 2025-02-25 18:19:20 -- STEP: 5438/36938 -- GLOBAL_STEP: 891950 | > loss_text_ce: 0.030689574778079987 (0.03341715980431478) | > loss_mel_ce: 1.0932247638702393 (nan) | > loss: 0.0044599780812859535 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13607188198837467) | > loader_time: 0.0045 (0.004687814600286157)  --> TIME: 2025-02-25 18:19:30 -- STEP: 5488/36938 -- GLOBAL_STEP: 892000 | > loss_text_ce: 0.026631947606801987 (0.03341264544823972) | > loss_mel_ce: 2.7437376976013184 (nan) | > loss: 0.010993530973792076 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13612106772623062) | > loader_time: 0.0047 (0.004688248255510006)  --> TIME: 2025-02-25 18:19:39 -- STEP: 5538/36938 -- GLOBAL_STEP: 892050 | > loss_text_ce: 0.03800206258893013 (0.03341763560452009) | > loss_mel_ce: 2.5570359230041504 (nan) | > loss: 0.010297770611941814 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.1361658360046278) | > loader_time: 0.0046 (0.004689027187668385)  --> TIME: 2025-02-25 18:19:49 -- STEP: 5588/36938 -- GLOBAL_STEP: 892100 | > loss_text_ce: 0.037758540362119675 (0.033413183579885535) | > loss_mel_ce: 1.063477635383606 (nan) | > loss: 0.0043699853122234344 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13616810513976385) | > loader_time: 0.0046 (0.004689127697463348)  --> TIME: 2025-02-25 18:19:58 -- STEP: 5638/36938 -- GLOBAL_STEP: 892150 | > loss_text_ce: 0.026290757581591606 (0.03340227602138265) | > loss_mel_ce: 2.7116241455078125 (nan) | > loss: 0.0108647421002388 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.1362003624629542) | > loader_time: 0.0047 (0.004689636066392094)  --> TIME: 2025-02-25 18:20:08 -- STEP: 5688/36938 -- GLOBAL_STEP: 892200 | > loss_text_ce: 0.0362686924636364 (0.03340540469657519) | > loss_mel_ce: 2.268080472946167 (nan) | > loss: 0.009144243784248829 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13623522312161715) | > loader_time: 0.0048 (0.00468929734411117)  --> TIME: 2025-02-25 18:20:17 -- STEP: 5738/36938 -- GLOBAL_STEP: 892250 | > loss_text_ce: 0.028524087741971016 (0.0334032734243461) | > loss_mel_ce: 2.972759485244751 (nan) | > loss: 0.011909856460988522 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13622857606880587) | > loader_time: 0.0045 (0.0046886474506993596)  --> TIME: 2025-02-25 18:20:26 -- STEP: 5788/36938 -- GLOBAL_STEP: 892300 | > loss_text_ce: 0.0389765240252018 (0.033410260961478054) | > loss_mel_ce: 2.38455867767334 (nan) | > loss: 0.009617202915251255 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1362209989024435) | > loader_time: 0.0045 (0.004688332512531904)  --> TIME: 2025-02-25 18:20:35 -- STEP: 5838/36938 -- GLOBAL_STEP: 892350 | > loss_text_ce: 0.034438129514455795 (0.033403969033225456) | > loss_mel_ce: 1.0864601135253906 (nan) | > loss: 0.004448009189218283 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13621910788825925) | > loader_time: 0.0046 (0.004688124372437192)  --> TIME: 2025-02-25 18:20:45 -- STEP: 5888/36938 -- GLOBAL_STEP: 892400 | > loss_text_ce: 0.025327786803245544 (0.033411854898314275) | > loss_mel_ce: 2.443765878677368 (nan) | > loss: 0.009797991253435612 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13621383087466657) | > loader_time: 0.0048 (0.004688025330719719)  --> TIME: 2025-02-25 18:20:54 -- STEP: 5938/36938 -- GLOBAL_STEP: 892450 | > loss_text_ce: 0.03504028543829918 (0.03341296361367218) | > loss_mel_ce: 2.960707902908325 (nan) | > loss: 0.011887890286743641 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13625126215856762) | > loader_time: 0.0046 (0.004687708891293288)  --> TIME: 2025-02-25 18:21:04 -- STEP: 5988/36938 -- GLOBAL_STEP: 892500 | > loss_text_ce: 0.04610579088330269 (0.03341711876582953) | > loss_mel_ce: 2.2961742877960205 (nan) | > loss: 0.009294763207435608 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13625709482567636) | > loader_time: 0.0052 (0.004687829820330957)  --> TIME: 2025-02-25 18:21:13 -- STEP: 6038/36938 -- GLOBAL_STEP: 892550 | > loss_text_ce: 0.028187649324536324 (0.033415473124266105) | > loss_mel_ce: 2.9195313453674316 (nan) | > loss: 0.011697298847138882 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13625236586570477) | > loader_time: 0.0046 (0.00468768339829951)  --> TIME: 2025-02-25 18:21:22 -- STEP: 6088/36938 -- GLOBAL_STEP: 892600 | > loss_text_ce: 0.028421761468052864 (0.03341789651201974) | > loss_mel_ce: 2.343569755554199 (nan) | > loss: 0.009412665851414204 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13628384094983984) | > loader_time: 0.0045 (0.004688171691869456)  --> TIME: 2025-02-25 18:21:32 -- STEP: 6138/36938 -- GLOBAL_STEP: 892650 | > loss_text_ce: 0.04485969990491867 (0.03340625819899146) | > loss_mel_ce: 2.5718889236450195 (nan) | > loss: 0.010383923538029194 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13629835612923882) | > loader_time: 0.0049 (0.004687878199142589)  --> TIME: 2025-02-25 18:21:42 -- STEP: 6188/36938 -- GLOBAL_STEP: 892700 | > loss_text_ce: 0.03404038026928902 (0.03340371399931264) | > loss_mel_ce: 2.739449977874756 (nan) | > loss: 0.011005914770066738 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.13634093438107245) | > loader_time: 0.0048 (0.0046878863552268476)  --> TIME: 2025-02-25 18:21:51 -- STEP: 6238/36938 -- GLOBAL_STEP: 892750 | > loss_text_ce: 0.029742484912276268 (0.0334013175952652) | > loss_mel_ce: 2.18795108795166 (nan) | > loss: 0.008800371550023556 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.136311537182152) | > loader_time: 0.0045 (0.004687525439468768)  --> TIME: 2025-02-25 18:22:00 -- STEP: 6288/36938 -- GLOBAL_STEP: 892800 | > loss_text_ce: 0.033126939088106155 (0.033392790891289506) | > loss_mel_ce: 2.9656383991241455 (nan) | > loss: 0.011899862438440323 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.13633662617691958) | > loader_time: 0.0049 (0.004687405421230317)  --> TIME: 2025-02-25 18:22:09 -- STEP: 6338/36938 -- GLOBAL_STEP: 892850 | > loss_text_ce: 0.03598002344369888 (0.033386147539603166) | > loss_mel_ce: 3.427051544189453 (nan) | > loss: 0.013742189854383469 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.13631815803506223) | > loader_time: 0.0048 (0.004687613777348419)  --> TIME: 2025-02-25 18:22:19 -- STEP: 6388/36938 -- GLOBAL_STEP: 892900 | > loss_text_ce: 0.031242772936820984 (0.033377192896407795) | > loss_mel_ce: 3.0376806259155273 (nan) | > loss: 0.012178268283605576 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.1363199707785476) | > loader_time: 0.0047 (0.004687597061890538)  --> TIME: 2025-02-25 18:22:28 -- STEP: 6438/36938 -- GLOBAL_STEP: 892950 | > loss_text_ce: 0.03206221014261246 (0.03337810387844759) | > loss_mel_ce: 2.722355365753174 (nan) | > loss: 0.010930228978395462 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.1363102959045978) | > loader_time: 0.0046 (0.004687045960368554)  --> TIME: 2025-02-25 18:22:37 -- STEP: 6488/36938 -- GLOBAL_STEP: 893000 | > loss_text_ce: 0.040165893733501434 (0.03338680633070464) | > loss_mel_ce: 2.0394113063812256 (nan) | > loss: 0.00825229100883007 (nan) | > current_lr: 5e-06 | > step_time: 0.1083 (0.1363221111118282) | > loader_time: 0.0045 (0.004687195641485883)  --> TIME: 2025-02-25 18:22:47 -- STEP: 6538/36938 -- GLOBAL_STEP: 893050 | > loss_text_ce: 0.038796067237854004 (0.0333915374236928) | > loss_mel_ce: 2.3418447971343994 (nan) | > loss: 0.009446988813579082 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.1363340885714316) | > loader_time: 0.0046 (0.004687467384280029)  --> TIME: 2025-02-25 18:22:56 -- STEP: 6588/36938 -- GLOBAL_STEP: 893100 | > loss_text_ce: 0.03296510502696037 (0.0334074184081944) | > loss_mel_ce: 2.2246673107147217 (nan) | > loss: 0.008958859369158745 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13629223559216902) | > loader_time: 0.0049 (0.0046872943549280625)  --> TIME: 2025-02-25 18:23:05 -- STEP: 6638/36938 -- GLOBAL_STEP: 893150 | > loss_text_ce: 0.028015609830617905 (0.033405207227730556) | > loss_mel_ce: 3.0086333751678467 (nan) | > loss: 0.012050194665789604 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.13626441071284198) | > loader_time: 0.0049 (0.0046871154557584635)  --> TIME: 2025-02-25 18:23:14 -- STEP: 6688/36938 -- GLOBAL_STEP: 893200 | > loss_text_ce: 0.027820250019431114 (0.03340865306068303) | > loss_mel_ce: 1.9273186922073364 (nan) | > loss: 0.007758488412946463 (nan) | > current_lr: 5e-06 | > step_time: 0.108 (0.1362510029779099) | > loader_time: 0.0045 (0.0046870144502968265)  --> TIME: 2025-02-25 18:23:23 -- STEP: 6738/36938 -- GLOBAL_STEP: 893250 | > loss_text_ce: 0.023843824863433838 (0.0334037959750641) | > loss_mel_ce: 2.475627899169922 (nan) | > loss: 0.009918538853526115 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13625378794824403) | > loader_time: 0.0047 (0.004687664734570623)  --> TIME: 2025-02-25 18:23:33 -- STEP: 6788/36938 -- GLOBAL_STEP: 893300 | > loss_text_ce: 0.03303464502096176 (0.03339539438584354) | > loss_mel_ce: 2.776315927505493 (nan) | > loss: 0.011148217134177685 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13626116324538853) | > loader_time: 0.0046 (0.004687530929787969)  --> TIME: 2025-02-25 18:23:42 -- STEP: 6838/36938 -- GLOBAL_STEP: 893350 | > loss_text_ce: 0.032087862491607666 (0.03339367896401805) | > loss_mel_ce: 2.969820022583008 (nan) | > loss: 0.011912333779036999 (nan) | > current_lr: 5e-06 | > step_time: 0.1937 (0.1362494036760136) | > loader_time: 0.0048 (0.004687149052593435)  --> TIME: 2025-02-25 18:23:52 -- STEP: 6888/36938 -- GLOBAL_STEP: 893400 | > loss_text_ce: 0.037671804428100586 (0.03339658961423152) | > loss_mel_ce: 3.2110164165496826 (nan) | > loss: 0.012891620397567749 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.13626189915837622) | > loader_time: 0.0047 (0.004688087206130519)  --> TIME: 2025-02-25 18:24:01 -- STEP: 6938/36938 -- GLOBAL_STEP: 893450 | > loss_text_ce: 0.030758298933506012 (0.033395757710088285) | > loss_mel_ce: 2.74658203125 (nan) | > loss: 0.011021193116903305 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.1362773271416618) | > loader_time: 0.0048 (0.004688867989312196)  --> TIME: 2025-02-25 18:24:10 -- STEP: 6988/36938 -- GLOBAL_STEP: 893500 | > loss_text_ce: 0.027310311794281006 (0.03339026602034159) | > loss_mel_ce: 3.1442630290985107 (nan) | > loss: 0.012585609219968319 (nan) | > current_lr: 5e-06 | > step_time: 0.1518 (0.13626951569479182) | > loader_time: 0.0047 (0.004689243226032223)  --> TIME: 2025-02-25 18:24:20 -- STEP: 7038/36938 -- GLOBAL_STEP: 893550 | > loss_text_ce: 0.04510226473212242 (0.0333977828279211) | > loss_mel_ce: 2.6265101432800293 (nan) | > loss: 0.01060163788497448 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13631075046454813) | > loader_time: 0.0048 (0.00468974785562197)  --> TIME: 2025-02-25 18:24:29 -- STEP: 7088/36938 -- GLOBAL_STEP: 893600 | > loss_text_ce: 0.02001664601266384 (0.033407050350199634) | > loss_mel_ce: 1.2190872430801392 (nan) | > loss: 0.004917079117149115 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13631786392585485) | > loader_time: 0.0046 (0.004689678852498934)  --> TIME: 2025-02-25 18:24:38 -- STEP: 7138/36938 -- GLOBAL_STEP: 893650 | > loss_text_ce: 0.02742292359471321 (0.03340016386079493) | > loss_mel_ce: 3.0763742923736572 (nan) | > loss: 0.012316656298935413 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13630920653585082) | > loader_time: 0.0046 (0.004690132878614293)  --> TIME: 2025-02-25 18:24:48 -- STEP: 7188/36938 -- GLOBAL_STEP: 893700 | > loss_text_ce: 0.04871847853064537 (0.03341257609800134) | > loss_mel_ce: 2.7581305503845215 (nan) | > loss: 0.011138290166854858 (nan) | > current_lr: 5e-06 | > step_time: 0.15 (0.13630190201447867) | > loader_time: 0.0047 (0.004690261867620842)  --> TIME: 2025-02-25 18:24:57 -- STEP: 7238/36938 -- GLOBAL_STEP: 893750 | > loss_text_ce: 0.031226059421896935 (0.03342236762824699) | > loss_mel_ce: 2.490706443786621 (nan) | > loss: 0.01000766921788454 (nan) | > current_lr: 5e-06 | > step_time: 0.1463 (0.13628052597910287) | > loader_time: 0.0047 (0.004691105384331244)  --> TIME: 2025-02-25 18:25:06 -- STEP: 7288/36938 -- GLOBAL_STEP: 893800 | > loss_text_ce: 0.027864472940564156 (0.03340885571907484) | > loss_mel_ce: 0.039488013833761215 (nan) | > loss: 0.00026727179647423327 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13624425989736233) | > loader_time: 0.0046 (0.0046913905148972)  --> TIME: 2025-02-25 18:25:15 -- STEP: 7338/36938 -- GLOBAL_STEP: 893850 | > loss_text_ce: 0.04283280670642853 (0.03341294624699297) | > loss_mel_ce: 3.1211860179901123 (nan) | > loss: 0.012555630877614021 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13621327580727272) | > loader_time: 0.0047 (0.004691308381025262)  --> TIME: 2025-02-25 18:25:24 -- STEP: 7388/36938 -- GLOBAL_STEP: 893900 | > loss_text_ce: 0.020787950605154037 (0.03341397977825898) | > loss_mel_ce: 2.316619634628296 (nan) | > loss: 0.009275427088141441 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13620094886391543) | > loader_time: 0.0049 (0.004691298613370142)  --> TIME: 2025-02-25 18:25:33 -- STEP: 7438/36938 -- GLOBAL_STEP: 893950 | > loss_text_ce: 0.026680130511522293 (0.03340916608168011) | > loss_mel_ce: 4.443638324737549 (nan) | > loss: 0.01773935928940773 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13618496465054764) | > loader_time: 0.0046 (0.00469137850420614)  --> TIME: 2025-02-25 18:25:42 -- STEP: 7488/36938 -- GLOBAL_STEP: 894000 | > loss_text_ce: 0.033787801861763 (0.033407521549175534) | > loss_mel_ce: 3.2588720321655273 (nan) | > loss: 0.013066111132502556 (nan) | > current_lr: 5e-06 | > step_time: 0.1516 (0.1361738868439813) | > loader_time: 0.0049 (0.004691935375205471)  --> TIME: 2025-02-25 18:25:52 -- STEP: 7538/36938 -- GLOBAL_STEP: 894050 | > loss_text_ce: 0.026463445276021957 (0.033408146105092955) | > loss_mel_ce: 1.4803816080093384 (nan) | > loss: 0.0059795440174639225 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13617250504371028) | > loader_time: 0.0047 (0.0046922835724269646)  --> TIME: 2025-02-25 18:26:01 -- STEP: 7588/36938 -- GLOBAL_STEP: 894100 | > loss_text_ce: 0.03642048314213753 (0.03341016012148687) | > loss_mel_ce: 2.78546142578125 (nan) | > loss: 0.011197945103049278 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.1361582144097275) | > loader_time: 0.0045 (0.004691789856570358)  --> TIME: 2025-02-25 18:26:10 -- STEP: 7638/36938 -- GLOBAL_STEP: 894150 | > loss_text_ce: 0.025456450879573822 (0.03340738548407774) | > loss_mel_ce: 1.7845458984375 (nan) | > loss: 0.0071825492195785046 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13613302044669978) | > loader_time: 0.0045 (0.004691112569003368)  --> TIME: 2025-02-25 18:26:19 -- STEP: 7688/36938 -- GLOBAL_STEP: 894200 | > loss_text_ce: 0.032071713358163834 (0.03341328446948606) | > loss_mel_ce: 3.2939445972442627 (nan) | > loss: 0.01319847907871008 (nan) | > current_lr: 5e-06 | > step_time: 0.1531 (0.13609961673818974) | > loader_time: 0.0047 (0.0046906745086475405)  --> TIME: 2025-02-25 18:26:28 -- STEP: 7738/36938 -- GLOBAL_STEP: 894250 | > loss_text_ce: 0.045161500573158264 (0.0334215893214687) | > loss_mel_ce: 1.408538818359375 (nan) | > loss: 0.00576865253970027 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13610435983382296) | > loader_time: 0.0046 (0.004690376693625016)  --> TIME: 2025-02-25 18:26:38 -- STEP: 7788/36938 -- GLOBAL_STEP: 894300 | > loss_text_ce: 0.03467926010489464 (0.033419405582897195) | > loss_mel_ce: 2.7859692573547363 (nan) | > loss: 0.0111930500715971 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13613649683850976) | > loader_time: 0.0046 (0.004690427534018164)  --> TIME: 2025-02-25 18:26:47 -- STEP: 7838/36938 -- GLOBAL_STEP: 894350 | > loss_text_ce: 0.032607581466436386 (0.03341404970915451) | > loss_mel_ce: 2.391491651535034 (nan) | > loss: 0.00961944181472063 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13609997156658632) | > loader_time: 0.0046 (0.004689971109104569)  --> TIME: 2025-02-25 18:26:56 -- STEP: 7888/36938 -- GLOBAL_STEP: 894400 | > loss_text_ce: 0.0272984616458416 (0.033418075130892594) | > loss_mel_ce: 3.209501266479492 (nan) | > loss: 0.012844444252550602 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.13610715919285746) | > loader_time: 0.0048 (0.0046897160595860965)  --> TIME: 2025-02-25 18:27:05 -- STEP: 7938/36938 -- GLOBAL_STEP: 894450 | > loss_text_ce: 0.038141559809446335 (0.03342388802153808) | > loss_mel_ce: 2.7437896728515625 (nan) | > loss: 0.011039409786462784 (nan) | > current_lr: 5e-06 | > step_time: 0.1861 (0.13610311039871553) | > loader_time: 0.0051 (0.004689502427727546)  --> TIME: 2025-02-25 18:27:14 -- STEP: 7988/36938 -- GLOBAL_STEP: 894500 | > loss_text_ce: 0.019344614818692207 (0.033418577458568695) | > loss_mel_ce: 2.6481947898864746 (nan) | > loss: 0.010585473850369453 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13608574703089557) | > loader_time: 0.0046 (0.0046893053790241935)  --> TIME: 2025-02-25 18:27:24 -- STEP: 8038/36938 -- GLOBAL_STEP: 894550 | > loss_text_ce: 0.031061619520187378 (0.033429652197681875) | > loss_mel_ce: 2.9690897464752197 (nan) | > loss: 0.011905363760888577 (nan) | > current_lr: 5e-06 | > step_time: 0.168 (0.13607580159309082) | > loader_time: 0.0046 (0.004689592216465122)  --> TIME: 2025-02-25 18:27:33 -- STEP: 8088/36938 -- GLOBAL_STEP: 894600 | > loss_text_ce: 0.025849593803286552 (0.03342512551647673) | > loss_mel_ce: 2.7292802333831787 (nan) | > loss: 0.01093305554240942 (nan) | > current_lr: 5e-06 | > step_time: 0.1268 (0.1361105580594022) | > loader_time: 0.0045 (0.004690215477957569)  --> TIME: 2025-02-25 18:27:42 -- STEP: 8138/36938 -- GLOBAL_STEP: 894650 | > loss_text_ce: 0.03148245811462402 (0.03341746645924428) | > loss_mel_ce: 3.568211078643799 (nan) | > loss: 0.014284498989582062 (nan) | > current_lr: 5e-06 | > step_time: 0.1742 (0.13608909119080817) | > loader_time: 0.0051 (0.004689785970695834)  --> TIME: 2025-02-25 18:27:52 -- STEP: 8188/36938 -- GLOBAL_STEP: 894700 | > loss_text_ce: 0.042233627289533615 (0.033409750060272574) | > loss_mel_ce: 1.8087986707687378 (nan) | > loss: 0.007345366291701794 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1360909001502286) | > loader_time: 0.0046 (0.004690450112529655)  --> TIME: 2025-02-25 18:28:01 -- STEP: 8238/36938 -- GLOBAL_STEP: 894750 | > loss_text_ce: 0.038491684943437576 (0.03340997724097682) | > loss_mel_ce: 3.045565605163574 (nan) | > loss: 0.01223832368850708 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13608704784825842) | > loader_time: 0.0047 (0.00469059914053861)  --> TIME: 2025-02-25 18:28:10 -- STEP: 8288/36938 -- GLOBAL_STEP: 894800 | > loss_text_ce: 0.03397165238857269 (0.03341982437423112) | > loss_mel_ce: 2.477466106414795 (nan) | > loss: 0.00996602326631546 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.13611109574904345) | > loader_time: 0.0048 (0.004691342474410892)  --> TIME: 2025-02-25 18:28:20 -- STEP: 8338/36938 -- GLOBAL_STEP: 894850 | > loss_text_ce: 0.0362919420003891 (0.03341515059041171) | > loss_mel_ce: 2.239905834197998 (nan) | > loss: 0.009032530710101128 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13615591304712285) | > loader_time: 0.0046 (0.004691452824488964)  --> TIME: 2025-02-25 18:28:29 -- STEP: 8388/36938 -- GLOBAL_STEP: 894900 | > loss_text_ce: 0.027320072054862976 (0.033408656429333376) | > loss_mel_ce: 3.1579442024230957 (nan) | > loss: 0.012639938853681087 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1361597134320012) | > loader_time: 0.0047 (0.004691081724680772)  --> TIME: 2025-02-25 18:28:39 -- STEP: 8438/36938 -- GLOBAL_STEP: 894950 | > loss_text_ce: 0.026205599308013916 (0.03340772358708589) | > loss_mel_ce: 3.2764320373535156 (nan) | > loss: 0.013105705380439758 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.13616018751329342) | > loader_time: 0.0048 (0.004690720334836279)  --> TIME: 2025-02-25 18:28:48 -- STEP: 8488/36938 -- GLOBAL_STEP: 895000 | > loss_text_ce: 0.02409631386399269 (0.03340601267147832) | > loss_mel_ce: 2.086451530456543 (nan) | > loss: 0.008375190198421478 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13615998968439913) | > loader_time: 0.0046 (0.004691057925399578)  --> TIME: 2025-02-25 18:28:57 -- STEP: 8538/36938 -- GLOBAL_STEP: 895050 | > loss_text_ce: 0.032195404171943665 (0.033403696457518) | > loss_mel_ce: 2.5607643127441406 (nan) | > loss: 0.01028952281922102 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1361530768728896) | > loader_time: 0.0046 (0.0046907837314386774)  --> TIME: 2025-02-25 18:29:06 -- STEP: 8588/36938 -- GLOBAL_STEP: 895100 | > loss_text_ce: 0.02691337838768959 (0.03340548693014185) | > loss_mel_ce: 1.9133820533752441 (nan) | > loss: 0.007699585519731045 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1361336482087571) | > loader_time: 0.0046 (0.004690966941503018)  --> TIME: 2025-02-25 18:29:16 -- STEP: 8638/36938 -- GLOBAL_STEP: 895150 | > loss_text_ce: 0.035198986530303955 (0.033404842314607386) | > loss_mel_ce: 2.2340242862701416 (nan) | > loss: 0.00900485459715128 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13612838924745385) | > loader_time: 0.0046 (0.004690956782345199)  --> TIME: 2025-02-25 18:29:25 -- STEP: 8688/36938 -- GLOBAL_STEP: 895200 | > loss_text_ce: 0.03150663897395134 (0.033406965668561134) | > loss_mel_ce: 3.0596728324890137 (nan) | > loss: 0.012266585603356361 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13612321019172696) | > loader_time: 0.0046 (0.004691433483943975)  --> TIME: 2025-02-25 18:29:34 -- STEP: 8738/36938 -- GLOBAL_STEP: 895250 | > loss_text_ce: 0.03395366296172142 (0.033400700648325364) | > loss_mel_ce: 3.4795238971710205 (nan) | > loss: 0.013942372053861618 (nan) | > current_lr: 5e-06 | > step_time: 0.1798 (0.13614165125353786) | > loader_time: 0.0049 (0.0046920213472327514)  --> TIME: 2025-02-25 18:29:44 -- STEP: 8788/36938 -- GLOBAL_STEP: 895300 | > loss_text_ce: 0.03922310099005699 (0.03340189196909103) | > loss_mel_ce: 2.3541369438171387 (nan) | > loss: 0.009497461840510368 (nan) | > current_lr: 5e-06 | > step_time: 0.1523 (0.13613935587457182) | > loader_time: 0.0047 (0.004692086589623979)  --> TIME: 2025-02-25 18:29:53 -- STEP: 8838/36938 -- GLOBAL_STEP: 895350 | > loss_text_ce: 0.022899901494383812 (0.03339540933672552) | > loss_mel_ce: 3.1820597648620605 (nan) | > loss: 0.012718094512820244 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13614109841696584) | > loader_time: 0.0047 (0.004691959452537375)  --> TIME: 2025-02-25 18:30:02 -- STEP: 8888/36938 -- GLOBAL_STEP: 895400 | > loss_text_ce: 0.03037768229842186 (0.033397062400060136) | > loss_mel_ce: 3.1386446952819824 (nan) | > loss: 0.012575485743582249 (nan) | > current_lr: 5e-06 | > step_time: 0.1824 (0.13613833617133989) | > loader_time: 0.0049 (0.004692539586128151)  --> TIME: 2025-02-25 18:30:11 -- STEP: 8938/36938 -- GLOBAL_STEP: 895450 | > loss_text_ce: 0.03670932352542877 (0.033400435796835784) | > loss_mel_ce: 3.2693028450012207 (nan) | > loss: 0.013119096867740154 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13613467935108448) | > loader_time: 0.0046 (0.004692532173817422)  --> TIME: 2025-02-25 18:30:21 -- STEP: 8988/36938 -- GLOBAL_STEP: 895500 | > loss_text_ce: 0.037703607231378555 (0.033400191927642146) | > loss_mel_ce: 3.0367820262908936 (nan) | > loss: 0.012200339697301388 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.13614053290103548) | > loader_time: 0.0047 (0.004693010779662194)  --> TIME: 2025-02-25 18:30:30 -- STEP: 9038/36938 -- GLOBAL_STEP: 895550 | > loss_text_ce: 0.0339931845664978 (0.033402458485257226) | > loss_mel_ce: 2.921128034591675 (nan) | > loss: 0.011726672761142254 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13617155836282185) | > loader_time: 0.0045 (0.004692783844262585)  --> TIME: 2025-02-25 18:30:40 -- STEP: 9088/36938 -- GLOBAL_STEP: 895600 | > loss_text_ce: 0.03242863714694977 (0.033398304337002034) | > loss_mel_ce: 2.4202935695648193 (nan) | > loss: 0.009733025915920734 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13615671471810664) | > loader_time: 0.0046 (0.004692752229076041)  --> TIME: 2025-02-25 18:30:49 -- STEP: 9138/36938 -- GLOBAL_STEP: 895650 | > loss_text_ce: 0.033972110599279404 (0.03340187880472926) | > loss_mel_ce: 3.6115758419036865 (nan) | > loss: 0.014466460794210434 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.1361773088166737) | > loader_time: 0.0048 (0.004693374615003625)  --> TIME: 2025-02-25 18:30:59 -- STEP: 9188/36938 -- GLOBAL_STEP: 895700 | > loss_text_ce: 0.03679874166846275 (0.033400990017521615) | > loss_mel_ce: 2.8607239723205566 (nan) | > loss: 0.011498106643557549 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.13619129401889082) | > loader_time: 0.0046 (0.004693978005094952)  --> TIME: 2025-02-25 18:31:08 -- STEP: 9238/36938 -- GLOBAL_STEP: 895750 | > loss_text_ce: 0.03660416603088379 (0.03340714242566218) | > loss_mel_ce: 3.1866259574890137 (nan) | > loss: 0.01279059611260891 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13620974393498944) | > loader_time: 0.0047 (0.004694209235084606)  --> TIME: 2025-02-25 18:31:17 -- STEP: 9288/36938 -- GLOBAL_STEP: 895800 | > loss_text_ce: 0.028667399659752846 (0.033410894981999464) | > loss_mel_ce: 1.2715799808502197 (nan) | > loss: 0.0051597123965620995 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13621456611782756) | > loader_time: 0.0047 (0.004694512648175857)  --> TIME: 2025-02-25 18:31:27 -- STEP: 9338/36938 -- GLOBAL_STEP: 895850 | > loss_text_ce: 0.030141208320856094 (0.03341299199909508) | > loss_mel_ce: 2.640346050262451 (nan) | > loss: 0.010597172193229198 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13622278589128925) | > loader_time: 0.0046 (0.0046946764196566255)  --> TIME: 2025-02-25 18:31:36 -- STEP: 9388/36938 -- GLOBAL_STEP: 895900 | > loss_text_ce: 0.03210148960351944 (0.03341885581428077) | > loss_mel_ce: 2.884753465652466 (nan) | > loss: 0.011574821546673775 (nan) | > current_lr: 5e-06 | > step_time: 0.1618 (0.13623880021969273) | > loader_time: 0.0049 (0.004694878064575331)  --> TIME: 2025-02-25 18:31:46 -- STEP: 9438/36938 -- GLOBAL_STEP: 895950 | > loss_text_ce: 0.028707223013043404 (0.033424214156126335) | > loss_mel_ce: 2.394986152648926 (nan) | > loss: 0.00961783155798912 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13622203719997356) | > loader_time: 0.0045 (0.004694543316294067)  --> TIME: 2025-02-25 18:31:55 -- STEP: 9488/36938 -- GLOBAL_STEP: 896000 | > loss_text_ce: 0.039784472435712814 (0.03342839649320521) | > loss_mel_ce: 1.0389617681503296 (nan) | > loss: 0.00428073899820447 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13620047055147122) | > loader_time: 0.0045 (0.004693990287451083)  --> TIME: 2025-02-25 18:32:04 -- STEP: 9538/36938 -- GLOBAL_STEP: 896050 | > loss_text_ce: 0.041841063648462296 (0.03342253197514546) | > loss_mel_ce: 2.1939194202423096 (nan) | > loss: 0.00887206569314003 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13617490804727814) | > loader_time: 0.0046 (0.004693826431247225)  --> TIME: 2025-02-25 18:32:13 -- STEP: 9588/36938 -- GLOBAL_STEP: 896100 | > loss_text_ce: 0.02690870314836502 (0.03342367280020148) | > loss_mel_ce: 3.02974534034729 (nan) | > loss: 0.012129579670727253 (nan) | > current_lr: 5e-06 | > step_time: 0.1762 (0.13616282807124544) | > loader_time: 0.0049 (0.004693743557943926)  --> TIME: 2025-02-25 18:32:22 -- STEP: 9638/36938 -- GLOBAL_STEP: 896150 | > loss_text_ce: 0.03173832967877388 (0.033422234557545165) | > loss_mel_ce: 2.5383760929107666 (nan) | > loss: 0.010198866948485374 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13618221478878767) | > loader_time: 0.0047 (0.004693956537399096)  --> TIME: 2025-02-25 18:32:32 -- STEP: 9688/36938 -- GLOBAL_STEP: 896200 | > loss_text_ce: 0.032370809465646744 (0.03342222040613336) | > loss_mel_ce: 3.453376531600952 (nan) | > loss: 0.013832331635057926 (nan) | > current_lr: 5e-06 | > step_time: 0.1441 (0.13618613694641396) | > loader_time: 0.0047 (0.004693810108966038)  --> TIME: 2025-02-25 18:32:41 -- STEP: 9738/36938 -- GLOBAL_STEP: 896250 | > loss_text_ce: 0.03389178216457367 (0.03342095432913217) | > loss_mel_ce: 2.8170557022094727 (nan) | > loss: 0.011313283815979958 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.1361806573111749) | > loader_time: 0.0047 (0.0046935919301175555)  --> TIME: 2025-02-25 18:32:50 -- STEP: 9788/36938 -- GLOBAL_STEP: 896300 | > loss_text_ce: 0.035999640822410583 (0.03341937123803168) | > loss_mel_ce: 1.8395501375198364 (nan) | > loss: 0.0074426583014428616 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13618562669231749) | > loader_time: 0.0046 (0.004693551481525232)  --> TIME: 2025-02-25 18:33:00 -- STEP: 9838/36938 -- GLOBAL_STEP: 896350 | > loss_text_ce: 0.03628764674067497 (0.03341487839200404) | > loss_mel_ce: 2.555687427520752 (nan) | > loss: 0.010285615921020508 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13619228067570574) | > loader_time: 0.0049 (0.004693703792957464)  --> TIME: 2025-02-25 18:33:09 -- STEP: 9888/36938 -- GLOBAL_STEP: 896400 | > loss_text_ce: 0.03200122341513634 (0.033419507236745936) | > loss_mel_ce: 3.1791536808013916 (nan) | > loss: 0.012742678634822369 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13619167907434718) | > loader_time: 0.0047 (0.004693665864205302)  --> TIME: 2025-02-25 18:33:19 -- STEP: 9938/36938 -- GLOBAL_STEP: 896450 | > loss_text_ce: 0.042923226952552795 (0.033421691610373123) | > loss_mel_ce: 0.931526780128479 (nan) | > loss: 0.003866865299642086 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.1362124877715879) | > loader_time: 0.0045 (0.004693769573757518)  --> TIME: 2025-02-25 18:33:28 -- STEP: 9988/36938 -- GLOBAL_STEP: 896500 | > loss_text_ce: 0.031198998913168907 (0.033422824511924416) | > loss_mel_ce: 2.659923553466797 (nan) | > loss: 0.010679058730602264 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13619899599371654) | > loader_time: 0.0046 (0.0046935775636146775)  --> TIME: 2025-02-25 18:33:37 -- STEP: 10038/36938 -- GLOBAL_STEP: 896550 | > loss_text_ce: 0.033695295453071594 (0.03342133264581917) | > loss_mel_ce: 3.0598642826080322 (nan) | > loss: 0.012276030145585537 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.13620302667178835) | > loader_time: 0.0048 (0.004693409365281384)  --> TIME: 2025-02-25 18:33:47 -- STEP: 10088/36938 -- GLOBAL_STEP: 896600 | > loss_text_ce: 0.026989681646227837 (0.03341823786590916) | > loss_mel_ce: 0.2304229736328125 (nan) | > loss: 0.0010214787907898426 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.1362443163364674) | > loader_time: 0.0047 (0.004693776392728735)  --> TIME: 2025-02-25 18:33:56 -- STEP: 10138/36938 -- GLOBAL_STEP: 896650 | > loss_text_ce: 0.03787827491760254 (0.03341877316766873) | > loss_mel_ce: 3.1949472427368164 (nan) | > loss: 0.012828673236072063 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.13624645398904334) | > loader_time: 0.0049 (0.004693634389161276)  --> TIME: 2025-02-25 18:34:06 -- STEP: 10188/36938 -- GLOBAL_STEP: 896700 | > loss_text_ce: 0.02690255269408226 (0.03341098810772082) | > loss_mel_ce: 2.7396676540374756 (nan) | > loss: 0.010978453792631626 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13626942327549543) | > loader_time: 0.0049 (0.004693602411055228)  --> TIME: 2025-02-25 18:34:15 -- STEP: 10238/36938 -- GLOBAL_STEP: 896750 | > loss_text_ce: 0.03438276797533035 (0.033403846558509524) | > loss_mel_ce: 3.0414750576019287 (nan) | > loss: 0.012205786071717739 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.1362720518564892) | > loader_time: 0.0046 (0.004693462131870619)  --> TIME: 2025-02-25 18:34:24 -- STEP: 10288/36938 -- GLOBAL_STEP: 896800 | > loss_text_ce: 0.034121908247470856 (0.03340844732872544) | > loss_mel_ce: 3.091775894165039 (nan) | > loss: 0.012404357083141804 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.1362508106194609) | > loader_time: 0.0045 (0.004693443978185055)  --> TIME: 2025-02-25 18:34:34 -- STEP: 10338/36938 -- GLOBAL_STEP: 896850 | > loss_text_ce: 0.028504619374871254 (0.03341258463759146) | > loss_mel_ce: 1.877726674079895 (nan) | > loss: 0.007564410101622343 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13625373400302374) | > loader_time: 0.0048 (0.004693611467523448)  --> TIME: 2025-02-25 18:34:43 -- STEP: 10388/36938 -- GLOBAL_STEP: 896900 | > loss_text_ce: 0.02881992980837822 (0.03341764765185541) | > loss_mel_ce: 3.3273561000823975 (nan) | > loss: 0.013318159617483616 (nan) | > current_lr: 5e-06 | > step_time: 0.1758 (0.13624942272867094) | > loader_time: 0.0048 (0.004693616432642364)  --> TIME: 2025-02-25 18:34:52 -- STEP: 10438/36938 -- GLOBAL_STEP: 896950 | > loss_text_ce: 0.02859104983508587 (0.03341319760636732) | > loss_mel_ce: 2.383366107940674 (nan) | > loss: 0.009571259841322899 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13625658193432183) | > loader_time: 0.0045 (0.004693718471845057)  --> TIME: 2025-02-25 18:35:01 -- STEP: 10488/36938 -- GLOBAL_STEP: 897000 | > loss_text_ce: 0.03865230083465576 (0.033413775696690594) | > loss_mel_ce: 2.7330563068389893 (nan) | > loss: 0.010998844169080257 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13625532820604833) | > loader_time: 0.0045 (0.004693692918045665)  --> TIME: 2025-02-25 18:35:11 -- STEP: 10538/36938 -- GLOBAL_STEP: 897050 | > loss_text_ce: 0.03331112489104271 (0.033416256950795246) | > loss_mel_ce: 2.692070960998535 (nan) | > loss: 0.010815008543431759 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13626760686681416) | > loader_time: 0.0046 (0.004693278055486069)  --> TIME: 2025-02-25 18:35:20 -- STEP: 10588/36938 -- GLOBAL_STEP: 897100 | > loss_text_ce: 0.036584753543138504 (0.03341657048584575) | > loss_mel_ce: 0.0473836250603199 (nan) | > loss: 0.0003332078631501645 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13626123942831642) | > loader_time: 0.0046 (0.004693383602182691)  --> TIME: 2025-02-25 18:35:29 -- STEP: 10638/36938 -- GLOBAL_STEP: 897150 | > loss_text_ce: 0.027147945016622543 (0.03341382729058726) | > loss_mel_ce: 2.535867214202881 (nan) | > loss: 0.010170696303248405 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13627283849714367) | > loader_time: 0.0049 (0.004693242790421552)  --> TIME: 2025-02-25 18:35:39 -- STEP: 10688/36938 -- GLOBAL_STEP: 897200 | > loss_text_ce: 0.031856365501880646 (0.033415041681627254) | > loss_mel_ce: 3.2627062797546387 (nan) | > loss: 0.013073661364614964 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.1362809046746012) | > loader_time: 0.0046 (0.004693478122799707)  --> TIME: 2025-02-25 18:35:48 -- STEP: 10738/36938 -- GLOBAL_STEP: 897250 | > loss_text_ce: 0.03464827686548233 (0.03341626763655349) | > loss_mel_ce: 3.064246654510498 (nan) | > loss: 0.012297202832996845 (nan) | > current_lr: 5e-06 | > step_time: 0.1878 (0.13628357707688094) | > loader_time: 0.0049 (0.0046931991232554)  --> TIME: 2025-02-25 18:35:57 -- STEP: 10788/36938 -- GLOBAL_STEP: 897300 | > loss_text_ce: 0.035867076367139816 (0.033413800610865435) | > loss_mel_ce: 2.68782114982605 (nan) | > loss: 0.010808287188410759 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13627665163103087) | > loader_time: 0.0047 (0.004693114960861076)  --> TIME: 2025-02-25 18:36:07 -- STEP: 10838/36938 -- GLOBAL_STEP: 897350 | > loss_text_ce: 0.03639709949493408 (0.0334129141135618) | > loss_mel_ce: 3.25374436378479 (nan) | > loss: 0.013056118041276932 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.13629208285994657) | > loader_time: 0.0047 (0.004693603577220511)  --> TIME: 2025-02-25 18:36:16 -- STEP: 10888/36938 -- GLOBAL_STEP: 897400 | > loss_text_ce: 0.03204292431473732 (0.033408817359055404) | > loss_mel_ce: 2.520700454711914 (nan) | > loss: 0.01012993510812521 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13626979697635289) | > loader_time: 0.0047 (0.004693927679930788)  --> TIME: 2025-02-25 18:36:25 -- STEP: 10938/36938 -- GLOBAL_STEP: 897450 | > loss_text_ce: 0.023504503071308136 (0.033406775658802054) | > loss_mel_ce: 2.8050425052642822 (nan) | > loss: 0.011224393732845783 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13627146327489023) | > loader_time: 0.0048 (0.004693775099403672)  --> TIME: 2025-02-25 18:36:35 -- STEP: 10988/36938 -- GLOBAL_STEP: 897500 | > loss_text_ce: 0.03220787271857262 (0.0334038210823904) | > loss_mel_ce: 3.091174840927124 (nan) | > loss: 0.012394377030432224 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.13628360215733157) | > loader_time: 0.0048 (0.0046936428933131566)  --> TIME: 2025-02-25 18:36:44 -- STEP: 11038/36938 -- GLOBAL_STEP: 897550 | > loss_text_ce: 0.03444145247340202 (0.03339910495558506) | > loss_mel_ce: 3.033054828643799 (nan) | > loss: 0.012172604911029339 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13626798790290423) | > loader_time: 0.0046 (0.004693268584824933)  --> TIME: 2025-02-25 18:36:53 -- STEP: 11088/36938 -- GLOBAL_STEP: 897600 | > loss_text_ce: 0.03817830979824066 (0.03339996365769825) | > loss_mel_ce: 3.1551315784454346 (nan) | > loss: 0.012671864591538906 (nan) | > current_lr: 5e-06 | > step_time: 0.1488 (0.13626487161776016) | > loader_time: 0.0049 (0.004693286092938466)  --> TIME: 2025-02-25 18:37:02 -- STEP: 11138/36938 -- GLOBAL_STEP: 897650 | > loss_text_ce: 0.04006082937121391 (0.03339318530361455) | > loss_mel_ce: 0.21337890625 (nan) | > loss: 0.00100571324583143 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.1362488969349736) | > loader_time: 0.0046 (0.0046932873252393354)  --> TIME: 2025-02-25 18:37:11 -- STEP: 11188/36938 -- GLOBAL_STEP: 897700 | > loss_text_ce: 0.04331028461456299 (0.033399315375613595) | > loss_mel_ce: 3.07119083404541 (nan) | > loss: 0.012359132058918476 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13625571313908486) | > loader_time: 0.0046 (0.00469323919208979)  --> TIME: 2025-02-25 18:37:21 -- STEP: 11238/36938 -- GLOBAL_STEP: 897750 | > loss_text_ce: 0.03862890601158142 (0.03339908558071123) | > loss_mel_ce: 2.7815895080566406 (nan) | > loss: 0.011191342957317829 (nan) | > current_lr: 5e-06 | > step_time: 0.1612 (0.13626610515677298) | > loader_time: 0.0047 (0.004693317570323989)  --> TIME: 2025-02-25 18:37:30 -- STEP: 11288/36938 -- GLOBAL_STEP: 897800 | > loss_text_ce: 0.045523252338171005 (0.03340151681169375) | > loss_mel_ce: 2.9692952632904053 (nan) | > loss: 0.011963565833866596 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13625611129337617) | > loader_time: 0.0047 (0.004693306839947669)  --> TIME: 2025-02-25 18:37:40 -- STEP: 11338/36938 -- GLOBAL_STEP: 897850 | > loss_text_ce: 0.030745485797524452 (0.033404640925196766) | > loss_mel_ce: 3.1389379501342773 (nan) | > loss: 0.01257810927927494 (nan) | > current_lr: 5e-06 | > step_time: 0.1889 (0.1362804143091086) | > loader_time: 0.0048 (0.00469333729346103)  --> TIME: 2025-02-25 18:37:49 -- STEP: 11388/36938 -- GLOBAL_STEP: 897900 | > loss_text_ce: 0.03704214096069336 (0.033409615002787066) | > loss_mel_ce: 3.3617050647735596 (nan) | > loss: 0.013487093150615692 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.13627900977196658) | > loader_time: 0.0047 (0.004693482062001797)  --> TIME: 2025-02-25 18:37:58 -- STEP: 11438/36938 -- GLOBAL_STEP: 897950 | > loss_text_ce: 0.034476593136787415 (0.03340788100191299) | > loss_mel_ce: 2.7047529220581055 (nan) | > loss: 0.010869958437979221 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1362664543846556) | > loader_time: 0.0046 (0.004693409262222103)  --> TIME: 2025-02-25 18:38:07 -- STEP: 11488/36938 -- GLOBAL_STEP: 898000 | > loss_text_ce: 0.022611910477280617 (0.033396959166748744) | > loss_mel_ce: 2.8485352993011475 (nan) | > loss: 0.011393441818654537 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13627480674005776) | > loader_time: 0.0046 (0.004693086163081163)  --> TIME: 2025-02-25 18:38:17 -- STEP: 11538/36938 -- GLOBAL_STEP: 898050 | > loss_text_ce: 0.03421810269355774 (0.0334042194694917) | > loss_mel_ce: 3.501476764678955 (nan) | > loss: 0.014030535705387592 (nan) | > current_lr: 5e-06 | > step_time: 0.1527 (0.1362717724854824) | > loader_time: 0.0049 (0.004693151739487137)  --> TIME: 2025-02-25 18:38:26 -- STEP: 11588/36938 -- GLOBAL_STEP: 898100 | > loss_text_ce: 0.03198777884244919 (0.03339850817100405) | > loss_mel_ce: 2.007770299911499 (nan) | > loss: 0.008094278164207935 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.13626378433104364) | > loader_time: 0.0046 (0.004693068304183689)  --> TIME: 2025-02-25 18:38:35 -- STEP: 11638/36938 -- GLOBAL_STEP: 898150 | > loss_text_ce: 0.041583746671676636 (0.03339748712919785) | > loss_mel_ce: 0.48855191469192505 (nan) | > loss: 0.0021037128753960133 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13627493625286155) | > loader_time: 0.0045 (0.004693177664560596)  --> TIME: 2025-02-25 18:38:45 -- STEP: 11688/36938 -- GLOBAL_STEP: 898200 | > loss_text_ce: 0.03167783468961716 (0.03339426549211133) | > loss_mel_ce: 2.8248844146728516 (nan) | > loss: 0.011335564777255058 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13627809180044892) | > loader_time: 0.0047 (0.004692978478718259)  --> TIME: 2025-02-25 18:38:54 -- STEP: 11738/36938 -- GLOBAL_STEP: 898250 | > loss_text_ce: 0.03079063631594181 (0.033388277329314) | > loss_mel_ce: 0.06734674423933029 (nan) | > loss: 0.00038943407707847655 (nan) | > current_lr: 5e-06 | > step_time: 0.1761 (0.13628725476231582) | > loader_time: 0.0048 (0.004692806643467983)  --> TIME: 2025-02-25 18:39:04 -- STEP: 11788/36938 -- GLOBAL_STEP: 898300 | > loss_text_ce: 0.02793320082128048 (0.033392901233235225) | > loss_mel_ce: 1.8067270517349243 (nan) | > loss: 0.00728039862588048 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13631166726563088) | > loader_time: 0.0046 (0.004692974376160687)  --> TIME: 2025-02-25 18:39:13 -- STEP: 11838/36938 -- GLOBAL_STEP: 898350 | > loss_text_ce: 0.04266274720430374 (0.033392056531059845) | > loss_mel_ce: 2.620762825012207 (nan) | > loss: 0.010569150559604168 (nan) | > current_lr: 5e-06 | > step_time: 0.1537 (0.1363206266410281) | > loader_time: 0.0047 (0.0046934043259450565)  --> TIME: 2025-02-25 18:39:23 -- STEP: 11888/36938 -- GLOBAL_STEP: 898400 | > loss_text_ce: 0.02880897931754589 (0.03339042694277771) | > loss_mel_ce: 2.9123053550720215 (nan) | > loss: 0.011671089567244053 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13632714072515567) | > loader_time: 0.0048 (0.004693491482157073)  --> TIME: 2025-02-25 18:39:32 -- STEP: 11938/36938 -- GLOBAL_STEP: 898450 | > loss_text_ce: 0.031780749559402466 (0.033383824312389995) | > loss_mel_ce: 3.05224871635437 (nan) | > loss: 0.0122382128611207 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.1363359832508031) | > loader_time: 0.0046 (0.004693167236386905)  --> TIME: 2025-02-25 18:39:41 -- STEP: 11988/36938 -- GLOBAL_STEP: 898500 | > loss_text_ce: 0.03172072023153305 (0.03338040899243577) | > loss_mel_ce: 3.3177480697631836 (nan) | > loss: 0.01329154334962368 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.1363453584430128) | > loader_time: 0.0049 (0.00469299662618346)  --> TIME: 2025-02-25 18:39:51 -- STEP: 12038/36938 -- GLOBAL_STEP: 898550 | > loss_text_ce: 0.01227150671184063 (0.03337743066911659) | > loss_mel_ce: 0.05810546875 (nan) | > loss: 0.0002792737213894725 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.13633827326166573) | > loader_time: 0.0045 (0.004693138240916228)  --> TIME: 2025-02-25 18:40:00 -- STEP: 12088/36938 -- GLOBAL_STEP: 898600 | > loss_text_ce: 0.04081530123949051 (0.03337375620305457) | > loss_mel_ce: 0.046184539794921875 (nan) | > loss: 0.0003452374949119985 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13632026947862108) | > loader_time: 0.0045 (0.004693221801326082)  --> TIME: 2025-02-25 18:40:09 -- STEP: 12138/36938 -- GLOBAL_STEP: 898650 | > loss_text_ce: 0.036057889461517334 (0.033372091359298715) | > loss_mel_ce: 3.5157196521759033 (nan) | > loss: 0.014094356447458267 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.136342072962221) | > loader_time: 0.0046 (0.004693363325307876)  --> TIME: 2025-02-25 18:40:19 -- STEP: 12188/36938 -- GLOBAL_STEP: 898700 | > loss_text_ce: 0.035077594220638275 (0.03337537141265851) | > loss_mel_ce: 2.977104902267456 (nan) | > loss: 0.01195310615003109 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13634240858117352) | > loader_time: 0.0047 (0.0046936184177642564)  --> TIME: 2025-02-25 18:40:28 -- STEP: 12238/36938 -- GLOBAL_STEP: 898750 | > loss_text_ce: 0.030752817168831825 (0.03336855224427295) | > loss_mel_ce: 2.810131549835205 (nan) | > loss: 0.01127335149794817 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13634525736010558) | > loader_time: 0.0045 (0.004693469476769939)  --> TIME: 2025-02-25 18:40:37 -- STEP: 12288/36938 -- GLOBAL_STEP: 898800 | > loss_text_ce: 0.03806937485933304 (0.03336424931785573) | > loss_mel_ce: 3.1736690998077393 (nan) | > loss: 0.012744994834065437 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.136343337595462) | > loader_time: 0.0046 (0.004693574019862939)  --> TIME: 2025-02-25 18:40:47 -- STEP: 12338/36938 -- GLOBAL_STEP: 898850 | > loss_text_ce: 0.031520090997219086 (0.033362196836892795) | > loss_mel_ce: 2.47519588470459 (nan) | > loss: 0.009947285987436771 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13635041219408528) | > loader_time: 0.0049 (0.004693401885933127)  --> TIME: 2025-02-25 18:40:56 -- STEP: 12388/36938 -- GLOBAL_STEP: 898900 | > loss_text_ce: 0.024136262014508247 (0.033359690359515345) | > loss_mel_ce: 1.7843735218048096 (nan) | > loss: 0.007176626939326525 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13633021158982583) | > loader_time: 0.0046 (0.004693227696503425)  --> TIME: 2025-02-25 18:41:05 -- STEP: 12438/36938 -- GLOBAL_STEP: 898950 | > loss_text_ce: 0.031430114060640335 (0.033360979516265134) | > loss_mel_ce: 2.626922607421875 (nan) | > loss: 0.01054901909083128 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13631023095914893) | > loader_time: 0.0048 (0.0046935193042966575)  --> TIME: 2025-02-25 18:41:14 -- STEP: 12488/36938 -- GLOBAL_STEP: 899000 | > loss_text_ce: 0.04515133425593376 (0.03336216501504102) | > loss_mel_ce: 2.1341094970703125 (nan) | > loss: 0.008647860959172249 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.1363192216315984) | > loader_time: 0.0046 (0.00469355299486553)  --> TIME: 2025-02-25 18:41:24 -- STEP: 12538/36938 -- GLOBAL_STEP: 899050 | > loss_text_ce: 0.03518415987491608 (0.033360484780033235) | > loss_mel_ce: 3.026017665863037 (nan) | > loss: 0.012147626839578152 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1363391084788148) | > loader_time: 0.0047 (0.004693544449123915)  --> TIME: 2025-02-25 18:41:33 -- STEP: 12588/36938 -- GLOBAL_STEP: 899100 | > loss_text_ce: 0.042027540504932404 (0.03335912832698781) | > loss_mel_ce: 3.256138324737549 (nan) | > loss: 0.013087959960103035 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13634594806156503) | > loader_time: 0.0046 (0.004693369051748719)  --> TIME: 2025-02-25 18:41:43 -- STEP: 12638/36938 -- GLOBAL_STEP: 899150 | > loss_text_ce: 0.03626417741179466 (0.03335943741842973) | > loss_mel_ce: 3.382420301437378 (nan) | > loss: 0.013566209003329277 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.1363523505801426) | > loader_time: 0.0046 (0.004693544256213632)  --> TIME: 2025-02-25 18:41:52 -- STEP: 12688/36938 -- GLOBAL_STEP: 899200 | > loss_text_ce: 0.0316786915063858 (0.03336259980122566) | > loss_mel_ce: 2.0518553256988525 (nan) | > loss: 0.008267992176115513 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1363444701306899) | > loader_time: 0.0045 (0.00469365935833726)  --> TIME: 2025-02-25 18:42:01 -- STEP: 12738/36938 -- GLOBAL_STEP: 899250 | > loss_text_ce: 0.034230709075927734 (0.033366471221124974) | > loss_mel_ce: 2.934664249420166 (nan) | > loss: 0.011781330220401287 (nan) | > current_lr: 5e-06 | > step_time: 0.1438 (0.13636097615658632) | > loader_time: 0.0047 (0.0046940579633732065)  --> TIME: 2025-02-25 18:42:10 -- STEP: 12788/36938 -- GLOBAL_STEP: 899300 | > loss_text_ce: 0.02085730992257595 (0.03336180643962723) | > loss_mel_ce: 2.5203535556793213 (nan) | > loss: 0.01008417084813118 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.1363511500710569) | > loader_time: 0.0046 (0.004694230544555903)  --> TIME: 2025-02-25 18:42:20 -- STEP: 12838/36938 -- GLOBAL_STEP: 899350 | > loss_text_ce: 0.02870781160891056 (0.03335832457498756) | > loss_mel_ce: 2.7767081260681152 (nan) | > loss: 0.011132603511214256 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13633963509797398) | > loader_time: 0.0045 (0.004694077934797599)  --> TIME: 2025-02-25 18:42:29 -- STEP: 12888/36938 -- GLOBAL_STEP: 899400 | > loss_text_ce: 0.03626669943332672 (0.03335966901949927) | > loss_mel_ce: 2.6176843643188477 (nan) | > loss: 0.010531553067266941 (nan) | > current_lr: 5e-06 | > step_time: 0.1079 (0.1363340985841581) | > loader_time: 0.0066 (0.0046940524891841955)  --> TIME: 2025-02-25 18:42:38 -- STEP: 12938/36938 -- GLOBAL_STEP: 899450 | > loss_text_ce: 0.037065740674734116 (0.033359566687602336) | > loss_mel_ce: 3.1837337017059326 (nan) | > loss: 0.012780951336026192 (nan) | > current_lr: 5e-06 | > step_time: 0.1871 (0.1363537780725148) | > loader_time: 0.0049 (0.004694081546903139)  --> TIME: 2025-02-25 18:42:48 -- STEP: 12988/36938 -- GLOBAL_STEP: 899500 | > loss_text_ce: 0.028118614107370377 (0.033357220176408846) | > loss_mel_ce: 2.783693313598633 (nan) | > loss: 0.011157984845340252 (nan) | > current_lr: 5e-06 | > step_time: 0.1478 (0.1363632671092147) | > loader_time: 0.0047 (0.0046939160737911895)  --> TIME: 2025-02-25 18:42:57 -- STEP: 13038/36938 -- GLOBAL_STEP: 899550 | > loss_text_ce: 0.028953468427062035 (0.03335883187164091) | > loss_mel_ce: 3.240508794784546 (nan) | > loss: 0.012974057346582413 (nan) | > current_lr: 5e-06 | > step_time: 0.1591 (0.13636853848887484) | > loader_time: 0.0048 (0.0046938153786387745)  --> TIME: 2025-02-25 18:43:07 -- STEP: 13088/36938 -- GLOBAL_STEP: 899600 | > loss_text_ce: 0.03490222245454788 (0.0333572646465225) | > loss_mel_ce: 2.833225727081299 (nan) | > loss: 0.011381461285054684 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.1363745688991316) | > loader_time: 0.0046 (0.004693780431391832)  --> TIME: 2025-02-25 18:43:16 -- STEP: 13138/36938 -- GLOBAL_STEP: 899650 | > loss_text_ce: 0.034817975014448166 (0.03335420240634337) | > loss_mel_ce: 2.3942131996154785 (nan) | > loss: 0.0096390126273036 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.1363778465649612) | > loader_time: 0.0048 (0.004693717131931883)  --> TIME: 2025-02-25 18:43:26 -- STEP: 13188/36938 -- GLOBAL_STEP: 899700 | > loss_text_ce: 0.03737865760922432 (0.03335485445230423) | > loss_mel_ce: 2.132538318634033 (nan) | > loss: 0.008610781282186508 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1363898689073607) | > loader_time: 0.0046 (0.004694275325234798)  --> TIME: 2025-02-25 18:43:35 -- STEP: 13238/36938 -- GLOBAL_STEP: 899750 | > loss_text_ce: 0.04080996289849281 (0.03335304229211694) | > loss_mel_ce: 2.8097455501556396 (nan) | > loss: 0.01131172850728035 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13639048331272557) | > loader_time: 0.0046 (0.004694532062157619)  --> TIME: 2025-02-25 18:43:44 -- STEP: 13288/36938 -- GLOBAL_STEP: 899800 | > loss_text_ce: 0.0369359590113163 (0.03335218006379963) | > loss_mel_ce: 2.680452585220337 (nan) | > loss: 0.01078328862786293 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13638722175291462) | > loader_time: 0.0047 (0.004694476337909399)  --> TIME: 2025-02-25 18:43:54 -- STEP: 13338/36938 -- GLOBAL_STEP: 899850 | > loss_text_ce: 0.03974364325404167 (0.03335281821661287) | > loss_mel_ce: 2.6059210300445557 (nan) | > loss: 0.010498669929802418 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.13639503378688567) | > loader_time: 0.0048 (0.004694359558850206)  --> TIME: 2025-02-25 18:44:03 -- STEP: 13388/36938 -- GLOBAL_STEP: 899900 | > loss_text_ce: 0.03560332953929901 (0.033356036645287065) | > loss_mel_ce: 2.9036312103271484 (nan) | > loss: 0.011663629673421383 (nan) | > current_lr: 5e-06 | > step_time: 0.1898 (0.13639565295235606) | > loader_time: 0.0049 (0.004694479203487737)  --> TIME: 2025-02-25 18:44:12 -- STEP: 13438/36938 -- GLOBAL_STEP: 899950 | > loss_text_ce: 0.032533202320337296 (0.03335968802528773) | > loss_mel_ce: 2.3708391189575195 (nan) | > loss: 0.009537192061543465 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.1364106239174635) | > loader_time: 0.0045 (0.004694760546547284)  --> TIME: 2025-02-25 18:44:22 -- STEP: 13488/36938 -- GLOBAL_STEP: 900000 | > loss_text_ce: 0.03323877602815628 (0.03335177202649351) | > loss_mel_ce: 0.0130615234375 (nan) | > loss: 0.0001837313611758873 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13641295924093336) | > loader_time: 0.0046 (0.0046947432829957435) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_900000.pth  --> TIME: 2025-02-25 18:44:37 -- STEP: 13538/36938 -- GLOBAL_STEP: 900050 | > loss_text_ce: 0.024759752675890923 (0.033352512156037344) | > loss_mel_ce: 1.5599157810211182 (nan) | > loss: 0.0062883952632546425 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1364216684925269) | > loader_time: 0.0046 (0.004695263865568511)  --> TIME: 2025-02-25 18:44:47 -- STEP: 13588/36938 -- GLOBAL_STEP: 900100 | > loss_text_ce: 0.03909233957529068 (0.033352315119361466) | > loss_mel_ce: 2.9794890880584717 (nan) | > loss: 0.01197849866002798 (nan) | > current_lr: 5e-06 | > step_time: 0.1759 (0.136420700860297) | > loader_time: 0.0049 (0.004695525213589121)  --> TIME: 2025-02-25 18:44:56 -- STEP: 13638/36938 -- GLOBAL_STEP: 900150 | > loss_text_ce: 0.039135925471782684 (0.033354094600399654) | > loss_mel_ce: 3.3744914531707764 (nan) | > loss: 0.013546140864491463 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13640840133275944) | > loader_time: 0.0046 (0.004695370323515672)  --> TIME: 2025-02-25 18:45:05 -- STEP: 13688/36938 -- GLOBAL_STEP: 900200 | > loss_text_ce: 0.0321720652282238 (0.033355340058139574) | > loss_mel_ce: 2.638887405395508 (nan) | > loss: 0.010599442757666111 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13639886357643516) | > loader_time: 0.0046 (0.004695590757473653)  --> TIME: 2025-02-25 18:45:14 -- STEP: 13738/36938 -- GLOBAL_STEP: 900250 | > loss_text_ce: 0.04316069558262825 (0.033359034592983894) | > loss_mel_ce: 3.0777971744537354 (nan) | > loss: 0.012384753674268723 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13640426247899792) | > loader_time: 0.0046 (0.004695566413114108)  --> TIME: 2025-02-25 18:45:23 -- STEP: 13788/36938 -- GLOBAL_STEP: 900300 | > loss_text_ce: 0.03261152282357216 (0.03336175286686858) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13639287568881645) | > loader_time: 0.0048 (0.004695374463515437)  --> TIME: 2025-02-25 18:45:32 -- STEP: 13838/36938 -- GLOBAL_STEP: 900350 | > loss_text_ce: 0.026166852563619614 (0.03335899079113779) | > loss_mel_ce: 2.841587543487549 (nan) | > loss: 0.011379978619515896 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13638333808068026) | > loader_time: 0.0045 (0.004695378919095023)  --> TIME: 2025-02-25 18:45:42 -- STEP: 13888/36938 -- GLOBAL_STEP: 900400 | > loss_text_ce: 0.0364392064511776 (0.033362679812501336) | > loss_mel_ce: 2.696542739868164 (nan) | > loss: 0.010845166631042957 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.1363830749065642) | > loader_time: 0.0047 (0.004695463307579527)  --> TIME: 2025-02-25 18:45:51 -- STEP: 13938/36938 -- GLOBAL_STEP: 900450 | > loss_text_ce: 0.030932623893022537 (0.033361487400360205) | > loss_mel_ce: 2.5685391426086426 (nan) | > loss: 0.01031536515802145 (nan) | > current_lr: 5e-06 | > step_time: 0.1582 (0.13639622215972144) | > loader_time: 0.0047 (0.004695550802534343)  --> TIME: 2025-02-25 18:46:00 -- STEP: 13988/36938 -- GLOBAL_STEP: 900500 | > loss_text_ce: 0.029380010440945625 (0.0333606655684677) | > loss_mel_ce: 2.3570749759674072 (nan) | > loss: 0.009470060467720032 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13638684962318676) | > loader_time: 0.0045 (0.004695306616508506)  --> TIME: 2025-02-25 18:46:09 -- STEP: 14038/36938 -- GLOBAL_STEP: 900550 | > loss_text_ce: 0.03528673201799393 (0.03335667610797913) | > loss_mel_ce: 2.2506911754608154 (nan) | > loss: 0.009071340784430504 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13636592241548853) | > loader_time: 0.0046 (0.004695018636380377)  --> TIME: 2025-02-25 18:46:19 -- STEP: 14088/36938 -- GLOBAL_STEP: 900600 | > loss_text_ce: 0.03134692832827568 (0.03335214115729233) | > loss_mel_ce: 2.5836875438690186 (nan) | > loss: 0.010377122089266777 (nan) | > current_lr: 5e-06 | > step_time: 0.1765 (0.13637892965360576) | > loader_time: 0.0048 (0.004694924833150413)  --> TIME: 2025-02-25 18:46:28 -- STEP: 14138/36938 -- GLOBAL_STEP: 900650 | > loss_text_ce: 0.03261011838912964 (0.033349036805140314) | > loss_mel_ce: 2.7502238750457764 (nan) | > loss: 0.01104299258440733 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13637218107376436) | > loader_time: 0.0047 (0.004694749044552421)  --> TIME: 2025-02-25 18:46:38 -- STEP: 14188/36938 -- GLOBAL_STEP: 900700 | > loss_text_ce: 0.032980166375637054 (0.033345912357037016) | > loss_mel_ce: 3.0107481479644775 (nan) | > loss: 0.01207828801125288 (nan) | > current_lr: 5e-06 | > step_time: 0.2071 (0.13638369132002348) | > loader_time: 0.0057 (0.004694791740184089)  --> TIME: 2025-02-25 18:46:47 -- STEP: 14238/36938 -- GLOBAL_STEP: 900750 | > loss_text_ce: 0.040621817111968994 (0.033342436559156516) | > loss_mel_ce: 2.9114513397216797 (nan) | > loss: 0.011714576743543148 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13637985565602376) | > loader_time: 0.0047 (0.004694963727276672)  --> TIME: 2025-02-25 18:46:56 -- STEP: 14288/36938 -- GLOBAL_STEP: 900800 | > loss_text_ce: 0.03298472240567207 (0.03334090079673403) | > loss_mel_ce: 2.810750722885132 (nan) | > loss: 0.011284665204584599 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.13638291645624007) | > loader_time: 0.0046 (0.004695015401482434)  --> TIME: 2025-02-25 18:47:06 -- STEP: 14338/36938 -- GLOBAL_STEP: 900850 | > loss_text_ce: 0.02803819440305233 (0.033343010940385616) | > loss_mel_ce: 2.219097137451172 (nan) | > loss: 0.008917204104363918 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13638370293821925) | > loader_time: 0.0048 (0.00469501205760408)  --> TIME: 2025-02-25 18:47:15 -- STEP: 14388/36938 -- GLOBAL_STEP: 900900 | > loss_text_ce: 0.04363517090678215 (0.03334468907859424) | > loss_mel_ce: 3.147352457046509 (nan) | > loss: 0.01266265008598566 (nan) | > current_lr: 5e-06 | > step_time: 0.1522 (0.1363764361338309) | > loader_time: 0.0058 (0.004695545178769448)  --> TIME: 2025-02-25 18:47:24 -- STEP: 14438/36938 -- GLOBAL_STEP: 900950 | > loss_text_ce: 0.0318615660071373 (0.03334267780347032) | > loss_mel_ce: 2.2492153644561768 (nan) | > loss: 0.009051892906427383 (nan) | > current_lr: 5e-06 | > step_time: 0.1577 (0.13637647073608025) | > loader_time: 0.0048 (0.0046955643144071845)  --> TIME: 2025-02-25 18:47:33 -- STEP: 14488/36938 -- GLOBAL_STEP: 901000 | > loss_text_ce: 0.029503094032406807 (0.03334282409696208) | > loss_mel_ce: 2.85683536529541 (nan) | > loss: 0.011453724466264248 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1363748396910859) | > loader_time: 0.0047 (0.0046958508147908855)  --> TIME: 2025-02-25 18:47:43 -- STEP: 14538/36938 -- GLOBAL_STEP: 901050 | > loss_text_ce: 0.036634575575590134 (0.033342290642197285) | > loss_mel_ce: 0.0672149658203125 (nan) | > loss: 0.0004121013917028904 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13636474555911998) | > loader_time: 0.0045 (0.004695746704806866)  --> TIME: 2025-02-25 18:47:52 -- STEP: 14588/36938 -- GLOBAL_STEP: 901100 | > loss_text_ce: 0.0396634005010128 (0.03334438287607565) | > loss_mel_ce: 3.501230478286743 (nan) | > loss: 0.014051166363060474 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13635329238218474) | > loader_time: 0.0045 (0.004695650810145455)  --> TIME: 2025-02-25 18:48:01 -- STEP: 14638/36938 -- GLOBAL_STEP: 901150 | > loss_text_ce: 0.029852494597434998 (0.03334368440142007) | > loss_mel_ce: 2.683997869491577 (nan) | > loss: 0.010769248008728027 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.1363510286112528) | > loader_time: 0.0045 (0.004695567607162969)  --> TIME: 2025-02-25 18:48:10 -- STEP: 14688/36938 -- GLOBAL_STEP: 901200 | > loss_text_ce: 0.03853761777281761 (0.03334142609081157) | > loss_mel_ce: 2.1727311611175537 (nan) | > loss: 0.008774876594543457 (nan) | > current_lr: 5e-06 | > step_time: 0.1508 (0.13635449539991748) | > loader_time: 0.0048 (0.00469535100628866)  --> TIME: 2025-02-25 18:48:20 -- STEP: 14738/36938 -- GLOBAL_STEP: 901250 | > loss_text_ce: 0.032025136053562164 (0.033342407761091826) | > loss_mel_ce: 3.1368701457977295 (nan) | > loss: 0.012574981898069382 (nan) | > current_lr: 5e-06 | > step_time: 0.1628 (0.13635907066064107) | > loader_time: 0.0046 (0.004695041190333877)  --> TIME: 2025-02-25 18:48:29 -- STEP: 14788/36938 -- GLOBAL_STEP: 901300 | > loss_text_ce: 0.03791904076933861 (0.03334158504124067) | > loss_mel_ce: 2.9023633003234863 (nan) | > loss: 0.011667788028717041 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.13635839639175623) | > loader_time: 0.0046 (0.004694814533035517)  --> TIME: 2025-02-25 18:48:38 -- STEP: 14838/36938 -- GLOBAL_STEP: 901350 | > loss_text_ce: 0.03236062079668045 (0.033341703604362484) | > loss_mel_ce: 3.0887157917022705 (nan) | > loss: 0.012385223992168903 (nan) | > current_lr: 5e-06 | > step_time: 0.1444 (0.1363677444572411) | > loader_time: 0.0046 (0.004694692030285921)  --> TIME: 2025-02-25 18:48:48 -- STEP: 14888/36938 -- GLOBAL_STEP: 901400 | > loss_text_ce: 0.03212396055459976 (0.03334317521887999) | > loss_mel_ce: 3.4912567138671875 (nan) | > loss: 0.013981670141220093 (nan) | > current_lr: 5e-06 | > step_time: 0.1861 (0.13636458419715736) | > loader_time: 0.0048 (0.004694615878577119)  --> TIME: 2025-02-25 18:48:57 -- STEP: 14938/36938 -- GLOBAL_STEP: 901450 | > loss_text_ce: 0.039068929851055145 (0.03334364382844896) | > loss_mel_ce: 2.750049591064453 (nan) | > loss: 0.011067931540310383 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13636382993309176) | > loader_time: 0.0047 (0.00469457420068725)  --> TIME: 2025-02-25 18:49:06 -- STEP: 14988/36938 -- GLOBAL_STEP: 901500 | > loss_text_ce: 0.03298211470246315 (0.03334516922228465) | > loss_mel_ce: 2.223314046859741 (nan) | > loss: 0.008953556418418884 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13636665332148726) | > loader_time: 0.0046 (0.00469452327240107)  --> TIME: 2025-02-25 18:49:16 -- STEP: 15038/36938 -- GLOBAL_STEP: 901550 | > loss_text_ce: 0.028703831136226654 (0.03334228171869002) | > loss_mel_ce: 2.8540380001068115 (nan) | > loss: 0.01143945287913084 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13638848292255262) | > loader_time: 0.0045 (0.004694649998471673)  --> TIME: 2025-02-25 18:49:26 -- STEP: 15088/36938 -- GLOBAL_STEP: 901600 | > loss_text_ce: 0.03526713699102402 (0.033341361294623534) | > loss_mel_ce: 2.9251928329467773 (nan) | > loss: 0.011747857555747032 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13640834724536505) | > loader_time: 0.0045 (0.0046947912756483854)  --> TIME: 2025-02-25 18:49:35 -- STEP: 15138/36938 -- GLOBAL_STEP: 901650 | > loss_text_ce: 0.024057703092694283 (0.0333469944809448) | > loss_mel_ce: 2.772324562072754 (nan) | > loss: 0.011096755042672157 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.13640771025253076) | > loader_time: 0.0048 (0.004694865848920751)  --> TIME: 2025-02-25 18:49:44 -- STEP: 15188/36938 -- GLOBAL_STEP: 901700 | > loss_text_ce: 0.04801800101995468 (0.03334412803856992) | > loss_mel_ce: 3.44313907623291 (nan) | > loss: 0.013853798620402813 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13640226594454244) | > loader_time: 0.0045 (0.004694707493484399)  --> TIME: 2025-02-25 18:49:54 -- STEP: 15238/36938 -- GLOBAL_STEP: 901750 | > loss_text_ce: 0.0370686836540699 (0.03334081503086522) | > loss_mel_ce: 2.6717212200164795 (nan) | > loss: 0.010749166831374168 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1364224483716799) | > loader_time: 0.0044 (0.004694748322221128)  --> TIME: 2025-02-25 18:50:03 -- STEP: 15288/36938 -- GLOBAL_STEP: 901800 | > loss_text_ce: 0.03297768905758858 (0.03333771923633203) | > loss_mel_ce: 2.3215560913085938 (nan) | > loss: 0.009343388490378857 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13643187415468824) | > loader_time: 0.0046 (0.004694876466229085)  --> TIME: 2025-02-25 18:50:13 -- STEP: 15338/36938 -- GLOBAL_STEP: 901850 | > loss_text_ce: 0.03844337910413742 (0.03334103206044839) | > loss_mel_ce: 2.2438762187957764 (nan) | > loss: 0.009056824259459972 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13643320232594802) | > loader_time: 0.0046 (0.004694750635731796)  --> TIME: 2025-02-25 18:50:22 -- STEP: 15388/36938 -- GLOBAL_STEP: 901900 | > loss_text_ce: 0.026266949251294136 (0.03333870680369192) | > loss_mel_ce: 2.002016067504883 (nan) | > loss: 0.00804874300956726 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.13640878528689668) | > loader_time: 0.0067 (0.004694752501672901)  --> TIME: 2025-02-25 18:50:31 -- STEP: 15438/36938 -- GLOBAL_STEP: 901950 | > loss_text_ce: 0.04081767424941063 (0.03334215822961314) | > loss_mel_ce: 3.455810785293579 (nan) | > loss: 0.013875510543584824 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.1364135244748555) | > loader_time: 0.0045 (0.004694949995283778)  --> TIME: 2025-02-25 18:50:40 -- STEP: 15488/36938 -- GLOBAL_STEP: 902000 | > loss_text_ce: 0.02342556230723858 (0.033335084648149195) | > loss_mel_ce: 2.8229405879974365 (nan) | > loss: 0.011295104399323463 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.1364038015507962) | > loader_time: 0.0046 (0.004695043568153014)  --> TIME: 2025-02-25 18:50:49 -- STEP: 15538/36938 -- GLOBAL_STEP: 902050 | > loss_text_ce: 0.030344359576702118 (0.03333852016911865) | > loss_mel_ce: 2.808385133743286 (nan) | > loss: 0.01126480009406805 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13640244638678925) | > loader_time: 0.0049 (0.004695246372770369)  --> TIME: 2025-02-25 18:50:59 -- STEP: 15588/36938 -- GLOBAL_STEP: 902100 | > loss_text_ce: 0.024143937975168228 (0.033334881085618134) | > loss_mel_ce: 3.1986405849456787 (nan) | > loss: 0.01278882846236229 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.1364015964076615) | > loader_time: 0.0047 (0.0046952809619389526)  --> TIME: 2025-02-25 18:51:08 -- STEP: 15638/36938 -- GLOBAL_STEP: 902150 | > loss_text_ce: 0.03906365483999252 (0.03333138354275822) | > loss_mel_ce: 2.7303285598754883 (nan) | > loss: 0.010989652015268803 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13639150406488268) | > loader_time: 0.0046 (0.004695132529130282)  --> TIME: 2025-02-25 18:51:17 -- STEP: 15688/36938 -- GLOBAL_STEP: 902200 | > loss_text_ce: 0.036816127598285675 (0.03332806027451398) | > loss_mel_ce: 2.0586867332458496 (nan) | > loss: 0.008315487764775753 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13638114334976473) | > loader_time: 0.0045 (0.004695299235854071)  --> TIME: 2025-02-25 18:51:26 -- STEP: 15738/36938 -- GLOBAL_STEP: 902250 | > loss_text_ce: 0.03680266812443733 (0.033331226291257406) | > loss_mel_ce: 3.2769951820373535 (nan) | > loss: 0.013149992562830448 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.136378716977098) | > loader_time: 0.0047 (0.004695277972117741)  --> TIME: 2025-02-25 18:51:36 -- STEP: 15788/36938 -- GLOBAL_STEP: 902300 | > loss_text_ce: 0.034052636474370956 (0.03333009939803814) | > loss_mel_ce: 3.1722683906555176 (nan) | > loss: 0.012723497115075588 (nan) | > current_lr: 5e-06 | > step_time: 0.172 (0.13638066085707184) | > loader_time: 0.0047 (0.0046953494137439355)  --> TIME: 2025-02-25 18:51:45 -- STEP: 15838/36938 -- GLOBAL_STEP: 902350 | > loss_text_ce: 0.03348127380013466 (0.03333199104366633) | > loss_mel_ce: 2.8148298263549805 (nan) | > loss: 0.011302823200821877 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1363767290946314) | > loader_time: 0.0045 (0.004695104760855088)  --> TIME: 2025-02-25 18:51:54 -- STEP: 15888/36938 -- GLOBAL_STEP: 902400 | > loss_text_ce: 0.03508966043591499 (0.03333330758178355) | > loss_mel_ce: 2.2726352214813232 (nan) | > loss: 0.009157638996839523 (nan) | > current_lr: 5e-06 | > step_time: 0.1784 (0.13639173624438702) | > loader_time: 0.0048 (0.004695333547942696)  --> TIME: 2025-02-25 18:52:04 -- STEP: 15938/36938 -- GLOBAL_STEP: 902450 | > loss_text_ce: 0.031425297260284424 (0.033331165542892276) | > loss_mel_ce: 3.1690921783447266 (nan) | > loss: 0.012700466439127922 (nan) | > current_lr: 5e-06 | > step_time: 0.1728 (0.1363936691096293) | > loader_time: 0.0048 (0.004695257468599931)  --> TIME: 2025-02-25 18:52:13 -- STEP: 15988/36938 -- GLOBAL_STEP: 902500 | > loss_text_ce: 0.03656308725476265 (0.03332846230046245) | > loss_mel_ce: 2.8448102474212646 (nan) | > loss: 0.011434022337198257 (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.13639947098733665) | > loader_time: 0.0048 (0.004695072676916542)  --> TIME: 2025-02-25 18:52:23 -- STEP: 16038/36938 -- GLOBAL_STEP: 902550 | > loss_text_ce: 0.03478538990020752 (0.03332728407270112) | > loss_mel_ce: 3.0311765670776367 (nan) | > loss: 0.012166515924036503 (nan) | > current_lr: 5e-06 | > step_time: 0.1505 (0.1364049557543021) | > loader_time: 0.0048 (0.004694784991327005)  --> TIME: 2025-02-25 18:52:32 -- STEP: 16088/36938 -- GLOBAL_STEP: 902600 | > loss_text_ce: 0.036641545593738556 (0.033325096437453364) | > loss_mel_ce: 0.09169311821460724 (nan) | > loss: 0.0005092645878903568 (nan) | > current_lr: 5e-06 | > step_time: 0.1634 (0.13640034072793936) | > loader_time: 0.0048 (0.004694553230130085)  --> TIME: 2025-02-25 18:52:41 -- STEP: 16138/36938 -- GLOBAL_STEP: 902650 | > loss_text_ce: 0.032661207020282745 (0.0333215688115887) | > loss_mel_ce: 2.89835262298584 (nan) | > loss: 0.011631008237600327 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13640786827622567) | > loader_time: 0.0046 (0.004694625929188193)  --> TIME: 2025-02-25 18:52:51 -- STEP: 16188/36938 -- GLOBAL_STEP: 902700 | > loss_text_ce: 0.03813290223479271 (0.033323550703554534) | > loss_mel_ce: 2.1349897384643555 (nan) | > loss: 0.008623503148555756 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13641795369469445) | > loader_time: 0.0046 (0.004694618853575381)  --> TIME: 2025-02-25 18:53:00 -- STEP: 16238/36938 -- GLOBAL_STEP: 902750 | > loss_text_ce: 0.03065340779721737 (0.03332084253269093) | > loss_mel_ce: 1.99796724319458 (nan) | > loss: 0.008050082251429558 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.13643220869069733) | > loader_time: 0.0046 (0.004694467181715405)  --> TIME: 2025-02-25 18:53:10 -- STEP: 16288/36938 -- GLOBAL_STEP: 902800 | > loss_text_ce: 0.027012024074792862 (0.033322359535265474) | > loss_mel_ce: 1.3967678546905518 (nan) | > loss: 0.005649920552968979 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13643566716630706) | > loader_time: 0.0045 (0.00469438734597915)  --> TIME: 2025-02-25 18:53:19 -- STEP: 16338/36938 -- GLOBAL_STEP: 902850 | > loss_text_ce: 0.0355779230594635 (0.03332314174593803) | > loss_mel_ce: 2.584050416946411 (nan) | > loss: 0.0103953517973423 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1364340857027035) | > loader_time: 0.0045 (0.0046941804570657785)  --> TIME: 2025-02-25 18:53:28 -- STEP: 16388/36938 -- GLOBAL_STEP: 902900 | > loss_text_ce: 0.03437507152557373 (0.03332434545240645) | > loss_mel_ce: 2.7476699352264404 (nan) | > loss: 0.011039861477911472 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13642189905414762) | > loader_time: 0.0045 (0.004694028557699879)  --> TIME: 2025-02-25 18:53:37 -- STEP: 16438/36938 -- GLOBAL_STEP: 902950 | > loss_text_ce: 0.03208618238568306 (0.03332250750835433) | > loss_mel_ce: 2.844980001449585 (nan) | > loss: 0.01141692977398634 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13642982044769966) | > loader_time: 0.0046 (0.004694701923565749)  --> TIME: 2025-02-25 18:53:47 -- STEP: 16488/36938 -- GLOBAL_STEP: 903000 | > loss_text_ce: 0.03922741860151291 (0.033324163377658776) | > loss_mel_ce: 3.2812745571136475 (nan) | > loss: 0.01317659579217434 (nan) | > current_lr: 5e-06 | > step_time: 0.1923 (0.13642609359453398) | > loader_time: 0.0051 (0.004694848862277356)  --> TIME: 2025-02-25 18:53:56 -- STEP: 16538/36938 -- GLOBAL_STEP: 903050 | > loss_text_ce: 0.03376638889312744 (0.03332149108953142) | > loss_mel_ce: 1.52140212059021 (nan) | > loss: 0.006171303801238537 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13641949495160316) | > loader_time: 0.0046 (0.004694711485888208)  --> TIME: 2025-02-25 18:54:05 -- STEP: 16588/36938 -- GLOBAL_STEP: 903100 | > loss_text_ce: 0.027716707438230515 (0.03331955745966705) | > loss_mel_ce: 3.1021087169647217 (nan) | > loss: 0.012419942766427994 (nan) | > current_lr: 5e-06 | > step_time: 0.1483 (0.13642202600502581) | > loader_time: 0.0047 (0.004694787226213085)  --> TIME: 2025-02-25 18:54:15 -- STEP: 16638/36938 -- GLOBAL_STEP: 903150 | > loss_text_ce: 0.0392831489443779 (0.03332454987326609) | > loss_mel_ce: 2.69268536567688 (nan) | > loss: 0.010841145180165768 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13643808467469262) | > loader_time: 0.0047 (0.00469476531253781)  --> TIME: 2025-02-25 18:54:25 -- STEP: 16688/36938 -- GLOBAL_STEP: 903200 | > loss_text_ce: 0.0339718796312809 (0.033322368382386156) | > loss_mel_ce: 3.291410207748413 (nan) | > loss: 0.013195960782468319 (nan) | > current_lr: 5e-06 | > step_time: 0.1848 (0.1364616871965926) | > loader_time: 0.0048 (0.004694905014188945)  --> TIME: 2025-02-25 18:54:34 -- STEP: 16738/36938 -- GLOBAL_STEP: 903250 | > loss_text_ce: 0.03426404297351837 (0.03332594606988891) | > loss_mel_ce: 2.7253201007843018 (nan) | > loss: 0.010950731113553047 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.1364578834812422) | > loader_time: 0.0045 (0.0046945672292181655)  --> TIME: 2025-02-25 18:54:43 -- STEP: 16788/36938 -- GLOBAL_STEP: 903300 | > loss_text_ce: 0.03373822569847107 (0.03332659179785921) | > loss_mel_ce: 2.2645764350891113 (nan) | > loss: 0.009120296686887741 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.1364514800460958) | > loader_time: 0.0047 (0.004694419600777824)  --> TIME: 2025-02-25 18:54:52 -- STEP: 16838/36938 -- GLOBAL_STEP: 903350 | > loss_text_ce: 0.030331052839756012 (0.0333244370111639) | > loss_mel_ce: 1.231534481048584 (nan) | > loss: 0.0050074029713869095 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13644876253832863) | > loader_time: 0.0046 (0.0046943708473719285)  --> TIME: 2025-02-25 18:55:02 -- STEP: 16888/36938 -- GLOBAL_STEP: 903400 | > loss_text_ce: 0.036027949303388596 (0.033323195005102266) | > loss_mel_ce: 3.124218702316284 (nan) | > loss: 0.012540661729872227 (nan) | > current_lr: 5e-06 | > step_time: 0.1764 (0.13644962485002607) | > loader_time: 0.0048 (0.004694162853404496)  --> TIME: 2025-02-25 18:55:11 -- STEP: 16938/36938 -- GLOBAL_STEP: 903450 | > loss_text_ce: 0.023321308195590973 (0.033322306600381664) | > loss_mel_ce: 2.2386136054992676 (nan) | > loss: 0.00897593330591917 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13645508968079859) | > loader_time: 0.0048 (0.004694303524284431)  --> TIME: 2025-02-25 18:55:20 -- STEP: 16988/36938 -- GLOBAL_STEP: 903500 | > loss_text_ce: 0.04612475633621216 (0.03332310149047905) | > loss_mel_ce: 1.8861275911331177 (nan) | > loss: 0.007667668629437685 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13644138003114195) | > loader_time: 0.0045 (0.004694238673498236)  --> TIME: 2025-02-25 18:55:30 -- STEP: 17038/36938 -- GLOBAL_STEP: 903550 | > loss_text_ce: 0.03297986835241318 (0.033320864801805586) | > loss_mel_ce: 2.865165948867798 (nan) | > loss: 0.011500579304993153 (nan) | > current_lr: 5e-06 | > step_time: 0.1818 (0.13645598075104115) | > loader_time: 0.0048 (0.004694336428162572)  --> TIME: 2025-02-25 18:55:39 -- STEP: 17088/36938 -- GLOBAL_STEP: 903600 | > loss_text_ce: 0.03897526115179062 (0.03332041734646425) | > loss_mel_ce: 3.5146074295043945 (nan) | > loss: 0.014101519249379635 (nan) | > current_lr: 5e-06 | > step_time: 0.1897 (0.13647535247972392) | > loader_time: 0.0049 (0.004694279883238248)  --> TIME: 2025-02-25 18:55:48 -- STEP: 17138/36938 -- GLOBAL_STEP: 903650 | > loss_text_ce: 0.01900385320186615 (0.03331942283400064) | > loss_mel_ce: 1.8314157724380493 (nan) | > loss: 0.007342935539782047 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13646537772915518) | > loader_time: 0.0045 (0.004694329758816708)  --> TIME: 2025-02-25 18:55:58 -- STEP: 17188/36938 -- GLOBAL_STEP: 903700 | > loss_text_ce: 0.015941902995109558 (0.03331698370990814) | > loss_mel_ce: 0.8910156488418579 (nan) | > loss: 0.003599038114771247 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.1364636989923751) | > loader_time: 0.0045 (0.004694464180617645)  --> TIME: 2025-02-25 18:56:07 -- STEP: 17238/36938 -- GLOBAL_STEP: 903750 | > loss_text_ce: 0.03373122587800026 (0.03331987027442798) | > loss_mel_ce: 3.0621848106384277 (nan) | > loss: 0.012285381555557251 (nan) | > current_lr: 5e-06 | > step_time: 0.1695 (0.13648441443934015) | > loader_time: 0.0047 (0.004694403206780652)  --> TIME: 2025-02-25 18:56:17 -- STEP: 17288/36938 -- GLOBAL_STEP: 903800 | > loss_text_ce: 0.019985033199191093 (0.03332300343900259) | > loss_mel_ce: 2.5942959785461426 (nan) | > loss: 0.010374131612479687 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13648207733224013) | > loader_time: 0.0045 (0.004694415539958186)  --> TIME: 2025-02-25 18:56:26 -- STEP: 17338/36938 -- GLOBAL_STEP: 903850 | > loss_text_ce: 0.033761363476514816 (0.03332219154451828) | > loss_mel_ce: 2.648864269256592 (nan) | > loss: 0.010645340196788311 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13648883807423998) | > loader_time: 0.0048 (0.004694347646165174)  --> TIME: 2025-02-25 18:56:36 -- STEP: 17388/36938 -- GLOBAL_STEP: 903900 | > loss_text_ce: 0.04468166083097458 (0.03332137586012242) | > loss_mel_ce: 3.1705079078674316 (nan) | > loss: 0.012758689001202583 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13649977019992068) | > loader_time: 0.0048 (0.004694539512851895)  --> TIME: 2025-02-25 18:56:45 -- STEP: 17438/36938 -- GLOBAL_STEP: 903950 | > loss_text_ce: 0.031265731900930405 (0.03332200274741966) | > loss_mel_ce: 2.8053669929504395 (nan) | > loss: 0.011256479658186436 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13648604206405032) | > loader_time: 0.0045 (0.004694565294915224)  --> TIME: 2025-02-25 18:56:54 -- STEP: 17488/36938 -- GLOBAL_STEP: 904000 | > loss_text_ce: 0.030598439276218414 (0.03332002689116757) | > loss_mel_ce: 2.8536672592163086 (nan) | > loss: 0.011445499025285244 (nan) | > current_lr: 5e-06 | > step_time: 0.1756 (0.1364904733311661) | > loader_time: 0.0048 (0.004694438018824989)  --> TIME: 2025-02-25 18:57:03 -- STEP: 17538/36938 -- GLOBAL_STEP: 904050 | > loss_text_ce: 0.03766823187470436 (0.03331944979560523) | > loss_mel_ce: 2.9230637550354004 (nan) | > loss: 0.011748936958611012 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.13647992051657798) | > loader_time: 0.0049 (0.004694331153141387)  --> TIME: 2025-02-25 18:57:13 -- STEP: 17588/36938 -- GLOBAL_STEP: 904100 | > loss_text_ce: 0.0259036086499691 (0.03331678628961538) | > loss_mel_ce: 2.262561321258545 (nan) | > loss: 0.009081210941076279 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13649038532958507) | > loader_time: 0.0047 (0.004694201538501486)  --> TIME: 2025-02-25 18:57:22 -- STEP: 17638/36938 -- GLOBAL_STEP: 904150 | > loss_text_ce: 0.03329046815633774 (0.03331445208745912) | > loss_mel_ce: 2.531742811203003 (nan) | > loss: 0.01017870381474495 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.13648985236230957) | > loader_time: 0.0045 (0.00469396845366302)  --> TIME: 2025-02-25 18:57:32 -- STEP: 17688/36938 -- GLOBAL_STEP: 904200 | > loss_text_ce: 0.037512898445129395 (0.03331668360787685) | > loss_mel_ce: 2.888064384460449 (nan) | > loss: 0.011609434150159359 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.1365083626954603) | > loader_time: 0.0045 (0.004693875710169458)  --> TIME: 2025-02-25 18:57:41 -- STEP: 17738/36938 -- GLOBAL_STEP: 904250 | > loss_text_ce: 0.034525785595178604 (0.03331554711431228) | > loss_mel_ce: 3.1198112964630127 (nan) | > loss: 0.012517211958765984 (nan) | > current_lr: 5e-06 | > step_time: 0.1515 (0.13650416681363023) | > loader_time: 0.0046 (0.004693679791290372)  --> TIME: 2025-02-25 18:57:51 -- STEP: 17788/36938 -- GLOBAL_STEP: 904300 | > loss_text_ce: 0.03473496064543724 (0.033316050884834296) | > loss_mel_ce: 1.5368810892105103 (nan) | > loss: 0.006236571818590164 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13652901817552976) | > loader_time: 0.0047 (0.004693819588158715)  --> TIME: 2025-02-25 18:58:00 -- STEP: 17838/36938 -- GLOBAL_STEP: 904350 | > loss_text_ce: 0.034422580152750015 (0.03331548568479067) | > loss_mel_ce: 2.724674940109253 (nan) | > loss: 0.01094880048185587 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.1365328131936728) | > loader_time: 0.0048 (0.004693715157216394)  --> TIME: 2025-02-25 18:58:09 -- STEP: 17888/36938 -- GLOBAL_STEP: 904400 | > loss_text_ce: 0.027943016961216927 (0.03331599042735066) | > loss_mel_ce: 1.3283733129501343 (nan) | > loss: 0.005382208153605461 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13653809998404365) | > loader_time: 0.0045 (0.0046936896677946885)  --> TIME: 2025-02-25 18:58:19 -- STEP: 17938/36938 -- GLOBAL_STEP: 904450 | > loss_text_ce: 0.03444308042526245 (0.03331940572728364) | > loss_mel_ce: 3.236055612564087 (nan) | > loss: 0.012978170067071915 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.13652631664690812) | > loader_time: 0.0048 (0.00469346154904415)  --> TIME: 2025-02-25 18:58:28 -- STEP: 17988/36938 -- GLOBAL_STEP: 904500 | > loss_text_ce: 0.03805377706885338 (0.03331988059886814) | > loss_mel_ce: 2.9348790645599365 (nan) | > loss: 0.011797352693974972 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13652621064155346) | > loader_time: 0.0047 (0.004693291718519221)  --> TIME: 2025-02-25 18:58:37 -- STEP: 18038/36938 -- GLOBAL_STEP: 904550 | > loss_text_ce: 0.03378259018063545 (0.033321933084480324) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13651456504032522) | > loader_time: 0.0045 (0.004693446303169972)  --> TIME: 2025-02-25 18:58:46 -- STEP: 18088/36938 -- GLOBAL_STEP: 904600 | > loss_text_ce: 0.031790345907211304 (0.03332161731245254) | > loss_mel_ce: 3.060166597366333 (nan) | > loss: 0.012269671075046062 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.13652025983693153) | > loader_time: 0.0048 (0.00469369405352504)  --> TIME: 2025-02-25 18:58:55 -- STEP: 18138/36938 -- GLOBAL_STEP: 904650 | > loss_text_ce: 0.028084609657526016 (0.033320900100640774) | > loss_mel_ce: 2.409804582595825 (nan) | > loss: 0.009674163535237312 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13651199682056717) | > loader_time: 0.0046 (0.004693762287834342)  --> TIME: 2025-02-25 18:59:05 -- STEP: 18188/36938 -- GLOBAL_STEP: 904700 | > loss_text_ce: 0.03247992321848869 (0.033324271668984155) | > loss_mel_ce: 3.030057907104492 (nan) | > loss: 0.012152928858995438 (nan) | > current_lr: 5e-06 | > step_time: 0.1449 (0.13652364048088436) | > loader_time: 0.0046 (0.004693621130914554)  --> TIME: 2025-02-25 18:59:15 -- STEP: 18238/36938 -- GLOBAL_STEP: 904750 | > loss_text_ce: 0.0244780033826828 (0.03332508637594661) | > loss_mel_ce: 3.0065314769744873 (nan) | > loss: 0.012027815915644169 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.1365331151477858) | > loader_time: 0.0046 (0.004693499664058143)  --> TIME: 2025-02-25 18:59:24 -- STEP: 18288/36938 -- GLOBAL_STEP: 904800 | > loss_text_ce: 0.04538734257221222 (0.03332514311474189) | > loss_mel_ce: 2.074436664581299 (nan) | > loss: 0.00841199979186058 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13653571886757612) | > loader_time: 0.0048 (0.004693293130825389)  --> TIME: 2025-02-25 18:59:33 -- STEP: 18338/36938 -- GLOBAL_STEP: 904850 | > loss_text_ce: 0.035065002739429474 (0.033325634581269574) | > loss_mel_ce: 2.6948978900909424 (nan) | > loss: 0.010833186097443104 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13654492773527577) | > loader_time: 0.0047 (0.004693165497868757)  --> TIME: 2025-02-25 18:59:43 -- STEP: 18388/36938 -- GLOBAL_STEP: 904900 | > loss_text_ce: 0.02768031321465969 (0.03332725552425085) | > loss_mel_ce: 1.5261374711990356 (nan) | > loss: 0.0061659435741603374 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1365572687255469) | > loader_time: 0.0046 (0.004693138047044664)  --> TIME: 2025-02-25 18:59:53 -- STEP: 18438/36938 -- GLOBAL_STEP: 904950 | > loss_text_ce: 0.026088332757353783 (0.033323959778189825) | > loss_mel_ce: 3.6135032176971436 (nan) | > loss: 0.014442823827266693 (nan) | > current_lr: 5e-06 | > step_time: 0.1469 (0.13656614580360252) | > loader_time: 0.0046 (0.004693049621085465)  --> TIME: 2025-02-25 19:00:02 -- STEP: 18488/36938 -- GLOBAL_STEP: 905000 | > loss_text_ce: 0.02601560577750206 (0.03331857287755772) | > loss_mel_ce: 2.870971918106079 (nan) | > loss: 0.01149598229676485 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13656738421092632) | > loader_time: 0.0046 (0.004693273998142453)  --> TIME: 2025-02-25 19:00:12 -- STEP: 18538/36938 -- GLOBAL_STEP: 905050 | > loss_text_ce: 0.03517872095108032 (0.03331901318857758) | > loss_mel_ce: 2.982635021209717 (nan) | > loss: 0.011975451372563839 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.1365823497122988) | > loader_time: 0.0046 (0.004693167972801319)  --> TIME: 2025-02-25 19:00:21 -- STEP: 18588/36938 -- GLOBAL_STEP: 905100 | > loss_text_ce: 0.03097706474363804 (0.03331871204783058) | > loss_mel_ce: 2.254683017730713 (nan) | > loss: 0.009070079773664474 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.1365802452106076) | > loader_time: 0.0045 (0.00469298885539188)  --> TIME: 2025-02-25 19:00:30 -- STEP: 18638/36938 -- GLOBAL_STEP: 905150 | > loss_text_ce: 0.028872331604361534 (0.03331632925315814) | > loss_mel_ce: 3.0267646312713623 (nan) | > loss: 0.012125544250011444 (nan) | > current_lr: 5e-06 | > step_time: 0.1556 (0.13658507147345972) | > loader_time: 0.0046 (0.004692982713965597)  --> TIME: 2025-02-25 19:00:39 -- STEP: 18688/36938 -- GLOBAL_STEP: 905200 | > loss_text_ce: 0.022293269634246826 (0.03331615464796343) | > loss_mel_ce: 2.5625 (nan) | > loss: 0.0102571165189147 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13656801124396784) | > loader_time: 0.0045 (0.00469279928413562)  --> TIME: 2025-02-25 19:00:48 -- STEP: 18738/36938 -- GLOBAL_STEP: 905250 | > loss_text_ce: 0.03314543887972832 (0.03331591614106233) | > loss_mel_ce: 2.46820068359375 (nan) | > loss: 0.00992597732692957 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13656337683121164) | > loader_time: 0.0048 (0.0046928783836735054)  --> TIME: 2025-02-25 19:00:58 -- STEP: 18788/36938 -- GLOBAL_STEP: 905300 | > loss_text_ce: 0.034487929195165634 (0.03331539383385518) | > loss_mel_ce: 2.330535650253296 (nan) | > loss: 0.00938501488417387 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.13655646099342728) | > loader_time: 0.0048 (0.004692806166132516)  --> TIME: 2025-02-25 19:01:07 -- STEP: 18838/36938 -- GLOBAL_STEP: 905350 | > loss_text_ce: 0.02893981710076332 (0.033313456238194594) | > loss_mel_ce: 0.28619384765625 (nan) | > loss: 0.001250530476681888 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13656051084992457) | > loader_time: 0.0046 (0.004692912202995647)  --> TIME: 2025-02-25 19:01:16 -- STEP: 18888/36938 -- GLOBAL_STEP: 905400 | > loss_text_ce: 0.03899630159139633 (0.03331077602035771) | > loss_mel_ce: 2.390423536300659 (nan) | > loss: 0.009640554897487164 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13656270615078006) | > loader_time: 0.0045 (0.004692726862829431)  --> TIME: 2025-02-25 19:01:26 -- STEP: 18938/36938 -- GLOBAL_STEP: 905450 | > loss_text_ce: 0.04158078506588936 (0.0333140896031238) | > loss_mel_ce: 2.8818728923797607 (nan) | > loss: 0.011601006612181664 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13656531160541702) | > loader_time: 0.0046 (0.004692821596350864)  --> TIME: 2025-02-25 19:01:35 -- STEP: 18988/36938 -- GLOBAL_STEP: 905500 | > loss_text_ce: 0.03322708606719971 (0.0333146199112461) | > loss_mel_ce: 2.6918795108795166 (nan) | > loss: 0.010813916102051735 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.13655919326288363) | > loader_time: 0.0048 (0.004692659180667068)  --> TIME: 2025-02-25 19:01:45 -- STEP: 19038/36938 -- GLOBAL_STEP: 905550 | > loss_text_ce: 0.036435317248106 (0.03331138157490338) | > loss_mel_ce: 2.8105523586273193 (nan) | > loss: 0.011297570541501045 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13656831453898183) | > loader_time: 0.0046 (0.004692659744733947)  --> TIME: 2025-02-25 19:01:54 -- STEP: 19088/36938 -- GLOBAL_STEP: 905600 | > loss_text_ce: 0.02395695634186268 (0.03330692713887011) | > loss_mel_ce: 2.6136231422424316 (nan) | > loss: 0.010466588661074638 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13656603852841148) | > loader_time: 0.0045 (0.004692818348085966)  --> TIME: 2025-02-25 19:02:03 -- STEP: 19138/36938 -- GLOBAL_STEP: 905650 | > loss_text_ce: 0.03871138021349907 (0.033305523231701974) | > loss_mel_ce: 2.6064484119415283 (nan) | > loss: 0.010496665723621845 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1365675672192667) | > loader_time: 0.0046 (0.004693125754093345)  --> TIME: 2025-02-25 19:02:13 -- STEP: 19188/36938 -- GLOBAL_STEP: 905700 | > loss_text_ce: 0.032948996871709824 (0.033305413805900526) | > loss_mel_ce: 1.8258757591247559 (nan) | > loss: 0.007376288995146751 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.13657467810591173) | > loader_time: 0.0048 (0.0046932190588322)  --> TIME: 2025-02-25 19:02:22 -- STEP: 19238/36938 -- GLOBAL_STEP: 905750 | > loss_text_ce: 0.04608406871557236 (0.033306334833945316) | > loss_mel_ce: 0.0566304512321949 (nan) | > loss: 0.0004075973411090672 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13657781931302723) | > loader_time: 0.0045 (0.0046931200208900625)  --> TIME: 2025-02-25 19:02:31 -- STEP: 19288/36938 -- GLOBAL_STEP: 905800 | > loss_text_ce: 0.03677000105381012 (0.0333065317479384) | > loss_mel_ce: 3.3820948600769043 (nan) | > loss: 0.013566925190389156 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13657846547468458) | > loader_time: 0.0046 (0.0046931032024938625)  --> TIME: 2025-02-25 19:02:40 -- STEP: 19338/36938 -- GLOBAL_STEP: 905850 | > loss_text_ce: 0.030704591423273087 (0.03330312509192586) | > loss_mel_ce: 2.726255178451538 (nan) | > loss: 0.010940317064523697 (nan) | > current_lr: 5e-06 | > step_time: 0.1725 (0.13656492941701764) | > loader_time: 0.0048 (0.004693163823336724)  --> TIME: 2025-02-25 19:02:50 -- STEP: 19388/36938 -- GLOBAL_STEP: 905900 | > loss_text_ce: 0.03628842532634735 (0.033305294176353105) | > loss_mel_ce: 2.517091989517212 (nan) | > loss: 0.010132462717592716 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13656150195336358) | > loader_time: 0.0047 (0.004693055093866365)  --> TIME: 2025-02-25 19:02:59 -- STEP: 19438/36938 -- GLOBAL_STEP: 905950 | > loss_text_ce: 0.035990383476018906 (0.033302787606693515) | > loss_mel_ce: 2.9790117740631104 (nan) | > loss: 0.01196429505944252 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.1365468154805759) | > loader_time: 0.0045 (0.004692983352570046)  --> TIME: 2025-02-25 19:03:08 -- STEP: 19488/36938 -- GLOBAL_STEP: 906000 | > loss_text_ce: 0.03259002044796944 (0.03330245208782183) | > loss_mel_ce: 2.4205920696258545 (nan) | > loss: 0.009734849445521832 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13654649346968942) | > loader_time: 0.0045 (0.004693034442970485)  --> TIME: 2025-02-25 19:03:18 -- STEP: 19538/36938 -- GLOBAL_STEP: 906050 | > loss_text_ce: 0.033212363719940186 (0.03330287606932926) | > loss_mel_ce: 3.214411497116089 (nan) | > loss: 0.012887396849691868 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.1365549018895782) | > loader_time: 0.0046 (0.004693037571078693)  --> TIME: 2025-02-25 19:03:27 -- STEP: 19588/36938 -- GLOBAL_STEP: 906100 | > loss_text_ce: 0.03436806797981262 (0.03330316517439509) | > loss_mel_ce: 2.844231128692627 (nan) | > loss: 0.011423013173043728 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13654611761627852) | > loader_time: 0.0046 (0.004692877826336213)  --> TIME: 2025-02-25 19:03:36 -- STEP: 19638/36938 -- GLOBAL_STEP: 906150 | > loss_text_ce: 0.018253760412335396 (0.03329747987927048) | > loss_mel_ce: 2.8363304138183594 (nan) | > loss: 0.011327715590596199 (nan) | > current_lr: 5e-06 | > step_time: 0.1794 (0.13654367722686928) | > loader_time: 0.005 (0.004692789298816182)  --> TIME: 2025-02-25 19:03:45 -- STEP: 19688/36938 -- GLOBAL_STEP: 906200 | > loss_text_ce: 0.030039653182029724 (0.03329583555949108) | > loss_mel_ce: 2.371507406234741 (nan) | > loss: 0.00952994916588068 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1365352369502804) | > loader_time: 0.0045 (0.004692725537512062)  --> TIME: 2025-02-25 19:03:54 -- STEP: 19738/36938 -- GLOBAL_STEP: 906250 | > loss_text_ce: 0.03956601768732071 (0.03329735717579824) | > loss_mel_ce: 0.04095458984375 (nan) | > loss: 0.00031952623976394534 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13653764920172282) | > loader_time: 0.0048 (0.004692926161894845)  --> TIME: 2025-02-25 19:04:04 -- STEP: 19788/36938 -- GLOBAL_STEP: 906300 | > loss_text_ce: 0.03667286038398743 (0.0332972442819556) | > loss_mel_ce: 2.999854326248169 (nan) | > loss: 0.012049711309373379 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13654523418001172) | > loader_time: 0.0046 (0.004693082517774635)  --> TIME: 2025-02-25 19:04:13 -- STEP: 19838/36938 -- GLOBAL_STEP: 906350 | > loss_text_ce: 0.035597290843725204 (0.03329553000321178) | > loss_mel_ce: 2.900193691253662 (nan) | > loss: 0.011649965308606625 (nan) | > current_lr: 5e-06 | > step_time: 0.1079 (0.13654177162913744) | > loader_time: 0.0045 (0.004693097459631736)  --> TIME: 2025-02-25 19:04:23 -- STEP: 19888/36938 -- GLOBAL_STEP: 906400 | > loss_text_ce: 0.0326724536716938 (0.03329679284225865) | > loss_mel_ce: 3.0247740745544434 (nan) | > loss: 0.01213272474706173 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13655641951919587) | > loader_time: 0.0045 (0.004693095075537459)  --> TIME: 2025-02-25 19:04:32 -- STEP: 19938/36938 -- GLOBAL_STEP: 906450 | > loss_text_ce: 0.03773752972483635 (0.033295647519432287) | > loss_mel_ce: 0.00885009765625 (nan) | > loss: 0.00018487154738977551 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.1365694803703218) | > loader_time: 0.0045 (0.004693067495939353)  --> TIME: 2025-02-25 19:04:42 -- STEP: 19988/36938 -- GLOBAL_STEP: 906500 | > loss_text_ce: 0.04598305746912956 (0.03329708107327054) | > loss_mel_ce: 2.505619764328003 (nan) | > loss: 0.010125408880412579 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13657308407299248) | > loader_time: 0.0045 (0.004693027100420841)  --> TIME: 2025-02-25 19:04:51 -- STEP: 20038/36938 -- GLOBAL_STEP: 906550 | > loss_text_ce: 0.03192658722400665 (0.03329652653168997) | > loss_mel_ce: 2.957468271255493 (nan) | > loss: 0.011862678453326225 (nan) | > current_lr: 5e-06 | > step_time: 0.1664 (0.13656985194564722) | > loader_time: 0.0047 (0.0046928567864459375)  --> TIME: 2025-02-25 19:05:00 -- STEP: 20088/36938 -- GLOBAL_STEP: 906600 | > loss_text_ce: 0.03003942221403122 (0.03329488816218586) | > loss_mel_ce: 1.3093764781951904 (nan) | > loss: 0.005315142683684826 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13657075826602283) | > loader_time: 0.0047 (0.004692708731457202)  --> TIME: 2025-02-25 19:05:09 -- STEP: 20138/36938 -- GLOBAL_STEP: 906650 | > loss_text_ce: 0.03087512031197548 (0.03329510233903416) | > loss_mel_ce: 2.7255361080169678 (nan) | > loss: 0.01093814056366682 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13656053455243602) | > loader_time: 0.0045 (0.0046924601980230475)  --> TIME: 2025-02-25 19:05:19 -- STEP: 20188/36938 -- GLOBAL_STEP: 906700 | > loss_text_ce: 0.024281734600663185 (0.033290296399427904) | > loss_mel_ce: 2.991619110107422 (nan) | > loss: 0.011967861093580723 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13656693693432664) | > loader_time: 0.0046 (0.004692302851812022)  --> TIME: 2025-02-25 19:05:29 -- STEP: 20238/36938 -- GLOBAL_STEP: 906750 | > loss_text_ce: 0.03933391347527504 (0.03328936904268825) | > loss_mel_ce: 3.2174599170684814 (nan) | > loss: 0.012923785485327244 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13658063818619254) | > loader_time: 0.0047 (0.004692223647187618)  --> TIME: 2025-02-25 19:05:38 -- STEP: 20288/36938 -- GLOBAL_STEP: 906800 | > loss_text_ce: 0.02886422723531723 (0.03328972793102886) | > loss_mel_ce: 3.2115604877471924 (nan) | > loss: 0.012858828529715538 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13657441710588974) | > loader_time: 0.0046 (0.004692026810597146)  --> TIME: 2025-02-25 19:05:47 -- STEP: 20338/36938 -- GLOBAL_STEP: 906850 | > loss_text_ce: 0.027531461790204048 (0.033295628429036266) | > loss_mel_ce: 2.4016034603118896 (nan) | > loss: 0.00963942427188158 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13657924698474969) | > loader_time: 0.0045 (0.004691795597550932)  --> TIME: 2025-02-25 19:05:56 -- STEP: 20388/36938 -- GLOBAL_STEP: 906900 | > loss_text_ce: 0.028857264667749405 (0.03329457661575708) | > loss_mel_ce: 2.1906466484069824 (nan) | > loss: 0.00880755577236414 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13657256332312806) | > loader_time: 0.0045 (0.004691555333039258)  --> TIME: 2025-02-25 19:06:06 -- STEP: 20438/36938 -- GLOBAL_STEP: 906950 | > loss_text_ce: 0.03503688797354698 (0.03329601057076315) | > loss_mel_ce: 2.7305922508239746 (nan) | > loss: 0.010974719189107418 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.13657684262711714) | > loader_time: 0.0048 (0.004691410337668899)  --> TIME: 2025-02-25 19:06:15 -- STEP: 20488/36938 -- GLOBAL_STEP: 907000 | > loss_text_ce: 0.03894380107522011 (0.033298093511334725) | > loss_mel_ce: 3.0089242458343506 (nan) | > loss: 0.012094714678823948 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13659039137420137) | > loader_time: 0.0046 (0.00469126300111684)  --> TIME: 2025-02-25 19:06:25 -- STEP: 20538/36938 -- GLOBAL_STEP: 907050 | > loss_text_ce: 0.037422794848680496 (0.0332988524777829) | > loss_mel_ce: 2.6941323280334473 (nan) | > loss: 0.01083950512111187 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.1365994436504747) | > loader_time: 0.0049 (0.004691226072141427)  --> TIME: 2025-02-25 19:06:34 -- STEP: 20588/36938 -- GLOBAL_STEP: 907100 | > loss_text_ce: 0.037980955094099045 (0.03329786179466563) | > loss_mel_ce: 2.844618082046509 (nan) | > loss: 0.0114388857036829 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1366176730872628) | > loader_time: 0.0044 (0.004691066963361794)  --> TIME: 2025-02-25 19:06:44 -- STEP: 20638/36938 -- GLOBAL_STEP: 907150 | > loss_text_ce: 0.0285508893430233 (0.03329704044157871) | > loss_mel_ce: 2.827716112136841 (nan) | > loss: 0.011334393173456192 (nan) | > current_lr: 5e-06 | > step_time: 0.1833 (0.13662980195221078) | > loader_time: 0.0065 (0.004690924741140733)  --> TIME: 2025-02-25 19:06:54 -- STEP: 20688/36938 -- GLOBAL_STEP: 907200 | > loss_text_ce: 0.040804389864206314 (0.033296048652774426) | > loss_mel_ce: 2.8894364833831787 (nan) | > loss: 0.011627940461039543 (nan) | > current_lr: 5e-06 | > step_time: 0.1265 (0.1366371387026603) | > loader_time: 0.0046 (0.004690770978901773)  --> TIME: 2025-02-25 19:07:03 -- STEP: 20738/36938 -- GLOBAL_STEP: 907250 | > loss_text_ce: 0.03530682623386383 (0.03330080092751048) | > loss_mel_ce: 3.097874402999878 (nan) | > loss: 0.01243325974792242 (nan) | > current_lr: 5e-06 | > step_time: 0.1667 (0.1366368428940407) | > loader_time: 0.0046 (0.004690565855066875)  --> TIME: 2025-02-25 19:07:12 -- STEP: 20788/36938 -- GLOBAL_STEP: 907300 | > loss_text_ce: 0.025599947199225426 (0.03329968695091197) | > loss_mel_ce: 2.975304126739502 (nan) | > loss: 0.011908350512385368 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13664550748677834) | > loader_time: 0.0045 (0.004690385619470493)  --> TIME: 2025-02-25 19:07:22 -- STEP: 20838/36938 -- GLOBAL_STEP: 907350 | > loss_text_ce: 0.029927203431725502 (0.03329621400539857) | > loss_mel_ce: 3.080487012863159 (nan) | > loss: 0.012342914007604122 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.13664817229973011) | > loader_time: 0.0047 (0.004690201614992024)  --> TIME: 2025-02-25 19:07:31 -- STEP: 20888/36938 -- GLOBAL_STEP: 907400 | > loss_text_ce: 0.03296297788619995 (0.03329226101208443) | > loss_mel_ce: 2.5436882972717285 (nan) | > loss: 0.01022480707615614 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13664978129660044) | > loader_time: 0.0046 (0.004690102579400964)  --> TIME: 2025-02-25 19:07:41 -- STEP: 20938/36938 -- GLOBAL_STEP: 907450 | > loss_text_ce: 0.03674255684018135 (0.03329390190517375) | > loss_mel_ce: 1.8919724225997925 (nan) | > loss: 0.007653631269931793 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13665711778386896) | > loader_time: 0.0045 (0.004689975583754838)  --> TIME: 2025-02-25 19:07:50 -- STEP: 20988/36938 -- GLOBAL_STEP: 907500 | > loss_text_ce: 0.037973061203956604 (0.03329525768249411) | > loss_mel_ce: 1.0997720956802368 (nan) | > loss: 0.004514861851930618 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13665533373645256) | > loader_time: 0.0046 (0.004689693291891114)  --> TIME: 2025-02-25 19:07:59 -- STEP: 21038/36938 -- GLOBAL_STEP: 907550 | > loss_text_ce: 0.029661795124411583 (0.03329610880978722) | > loss_mel_ce: 2.2458202838897705 (nan) | > loss: 0.0090296920388937 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13666002444865488) | > loader_time: 0.0045 (0.004689718734312542)  --> TIME: 2025-02-25 19:08:09 -- STEP: 21088/36938 -- GLOBAL_STEP: 907600 | > loss_text_ce: 0.032659564167261124 (0.0332981679118744) | > loss_mel_ce: 2.264326810836792 (nan) | > loss: 0.009115025401115417 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.1366623896251854) | > loader_time: 0.0045 (0.004689814446547663)  --> TIME: 2025-02-25 19:08:18 -- STEP: 21138/36938 -- GLOBAL_STEP: 907650 | > loss_text_ce: 0.03181114420294762 (0.033296659940185616) | > loss_mel_ce: 2.907684087753296 (nan) | > loss: 0.01166466437280178 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.13665179347279033) | > loader_time: 0.0045 (0.004689833876290082)  --> TIME: 2025-02-25 19:08:27 -- STEP: 21188/36938 -- GLOBAL_STEP: 907700 | > loss_text_ce: 0.03682965785264969 (0.033296925772963615) | > loss_mel_ce: 2.6290488243103027 (nan) | > loss: 0.010578883811831474 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13663741387532272) | > loader_time: 0.0054 (0.004689713097842903)  --> TIME: 2025-02-25 19:08:36 -- STEP: 21238/36938 -- GLOBAL_STEP: 907750 | > loss_text_ce: 0.02049783617258072 (0.03329865901274634) | > loss_mel_ce: 2.6380388736724854 (nan) | > loss: 0.010549749247729778 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1366293864439222) | > loader_time: 0.0045 (0.004689648232449667)  --> TIME: 2025-02-25 19:08:45 -- STEP: 21288/36938 -- GLOBAL_STEP: 907800 | > loss_text_ce: 0.024344103410840034 (0.03329810910888489) | > loss_mel_ce: 2.6355457305908203 (nan) | > loss: 0.010555119253695011 (nan) | > current_lr: 5e-06 | > step_time: 0.2007 (0.1366362458711628) | > loader_time: 0.0049 (0.004689628896028707)  --> TIME: 2025-02-25 19:08:55 -- STEP: 21338/36938 -- GLOBAL_STEP: 907850 | > loss_text_ce: 0.03463508188724518 (0.03329820451875305) | > loss_mel_ce: 2.89054012298584 (nan) | > loss: 0.011607838794589043 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13663668837651355) | > loader_time: 0.0046 (0.004689730166807876)  --> TIME: 2025-02-25 19:09:04 -- STEP: 21388/36938 -- GLOBAL_STEP: 907900 | > loss_text_ce: 0.040364522486925125 (0.03329702585580105) | > loss_mel_ce: 2.7782695293426514 (nan) | > loss: 0.011185056529939175 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13662879194347316) | > loader_time: 0.0046 (0.004689749834001077)  --> TIME: 2025-02-25 19:09:13 -- STEP: 21438/36938 -- GLOBAL_STEP: 907950 | > loss_text_ce: 0.02535860240459442 (0.03329812441625412) | > loss_mel_ce: 2.3777716159820557 (nan) | > loss: 0.009536231867969036 (nan) | > current_lr: 5e-06 | > step_time: 0.1066 (0.13662142557421142) | > loader_time: 0.0046 (0.004689598708513525)  --> TIME: 2025-02-25 19:09:22 -- STEP: 21488/36938 -- GLOBAL_STEP: 908000 | > loss_text_ce: 0.021638454869389534 (0.033297332806527725) | > loss_mel_ce: 2.4933760166168213 (nan) | > loss: 0.0099802166223526 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13662056654846955) | > loader_time: 0.0046 (0.004689450117073548)  --> TIME: 2025-02-25 19:09:31 -- STEP: 21538/36938 -- GLOBAL_STEP: 908050 | > loss_text_ce: 0.04723804444074631 (0.033297780005026006) | > loss_mel_ce: 3.3419528007507324 (nan) | > loss: 0.013449170626699924 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.13661396321905794) | > loader_time: 0.0048 (0.0046894302473547145)  --> TIME: 2025-02-25 19:09:40 -- STEP: 21588/36938 -- GLOBAL_STEP: 908100 | > loss_text_ce: 0.03735402598977089 (0.03329365436146998) | > loss_mel_ce: 2.6688404083251953 (nan) | > loss: 0.010738867335021496 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13660322253695592) | > loader_time: 0.0045 (0.004689431508558849)  --> TIME: 2025-02-25 19:09:50 -- STEP: 21638/36938 -- GLOBAL_STEP: 908150 | > loss_text_ce: 0.03687277436256409 (0.033295835916032926) | > loss_mel_ce: 3.336545705795288 (nan) | > loss: 0.013386582024395466 (nan) | > current_lr: 5e-06 | > step_time: 0.1579 (0.1366139184592306) | > loader_time: 0.0047 (0.0046892996933557305)  --> TIME: 2025-02-25 19:09:59 -- STEP: 21688/36938 -- GLOBAL_STEP: 908200 | > loss_text_ce: 0.03323115408420563 (0.033295583885775706) | > loss_mel_ce: 3.15421986579895 (nan) | > loss: 0.01264861598610878 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13660083321441466) | > loader_time: 0.0046 (0.004689344419671431)  --> TIME: 2025-02-25 19:10:08 -- STEP: 21738/36938 -- GLOBAL_STEP: 908250 | > loss_text_ce: 0.0325864776968956 (0.03329483663181036) | > loss_mel_ce: 0.2933593690395355 (nan) | > loss: 0.0012934360420331359 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13659763002759678) | > loader_time: 0.0046 (0.004689403286150432)  --> TIME: 2025-02-25 19:10:17 -- STEP: 21788/36938 -- GLOBAL_STEP: 908300 | > loss_text_ce: 0.034993987530469894 (0.03329929309989122) | > loss_mel_ce: 3.322866201400757 (nan) | > loss: 0.013324842788279057 (nan) | > current_lr: 5e-06 | > step_time: 0.1664 (0.13659307204716195) | > loader_time: 0.0048 (0.0046894936489800506)  --> TIME: 2025-02-25 19:10:27 -- STEP: 21838/36938 -- GLOBAL_STEP: 908350 | > loss_text_ce: 0.029634131118655205 (0.0333001222657212) | > loss_mel_ce: 2.6414244174957275 (nan) | > loss: 0.010599439963698387 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13659513152795216) | > loader_time: 0.0045 (0.0046894669871221355)  --> TIME: 2025-02-25 19:10:36 -- STEP: 21888/36938 -- GLOBAL_STEP: 908400 | > loss_text_ce: 0.030697256326675415 (0.033301166773303054) | > loss_mel_ce: 2.786891222000122 (nan) | > loss: 0.011180907487869263 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13659880553203296) | > loader_time: 0.0047 (0.004689404533969014)  --> TIME: 2025-02-25 19:10:46 -- STEP: 21938/36938 -- GLOBAL_STEP: 908450 | > loss_text_ce: 0.02636917494237423 (0.03329926032510333) | > loss_mel_ce: 1.5417110919952393 (nan) | > loss: 0.006222541444003582 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1366030545099477) | > loader_time: 0.0046 (0.004689193193583632)  --> TIME: 2025-02-25 19:10:55 -- STEP: 21988/36938 -- GLOBAL_STEP: 908500 | > loss_text_ce: 0.043036311864852905 (0.03329846386141048) | > loss_mel_ce: 1.9278377294540405 (nan) | > loss: 0.007820929400622845 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13660140683352934) | > loader_time: 0.0045 (0.004689095506933519)  --> TIME: 2025-02-25 19:11:04 -- STEP: 22038/36938 -- GLOBAL_STEP: 908550 | > loss_text_ce: 0.03848784789443016 (0.03329642035973312) | > loss_mel_ce: 3.049320697784424 (nan) | > loss: 0.01225320901721716 (nan) | > current_lr: 5e-06 | > step_time: 0.1773 (0.13659481062561443) | > loader_time: 0.0073 (0.0046893504388817574)  --> TIME: 2025-02-25 19:11:14 -- STEP: 22088/36938 -- GLOBAL_STEP: 908600 | > loss_text_ce: 0.03191116079688072 (0.03329735495937219) | > loss_mel_ce: 1.6239992380142212 (nan) | > loss: 0.006571073085069656 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13660181064141075) | > loader_time: 0.0046 (0.004689611491842649)  --> TIME: 2025-02-25 19:11:23 -- STEP: 22138/36938 -- GLOBAL_STEP: 908650 | > loss_text_ce: 0.02487693727016449 (0.03329660572603484) | > loss_mel_ce: 2.5256896018981934 (nan) | > loss: 0.010121296159923077 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.13660041552936547) | > loader_time: 0.0046 (0.0046896079829611605)  --> TIME: 2025-02-25 19:11:32 -- STEP: 22188/36938 -- GLOBAL_STEP: 908700 | > loss_text_ce: 0.03439480438828468 (0.03329595687189025) | > loss_mel_ce: 3.308236598968506 (nan) | > loss: 0.013264411129057407 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13659835876119547) | > loader_time: 0.0046 (0.00468958891983265)  --> TIME: 2025-02-25 19:11:41 -- STEP: 22238/36938 -- GLOBAL_STEP: 908750 | > loss_text_ce: 0.031870074570178986 (0.03329342813078198) | > loss_mel_ce: 2.182626962661743 (nan) | > loss: 0.008787686936557293 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13659122812736096) | > loader_time: 0.0048 (0.004689468712630433)  --> TIME: 2025-02-25 19:11:51 -- STEP: 22288/36938 -- GLOBAL_STEP: 908800 | > loss_text_ce: 0.03937795013189316 (0.03329331680474783) | > loss_mel_ce: 2.6947808265686035 (nan) | > loss: 0.010849837213754654 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.13659019788562618) | > loader_time: 0.0047 (0.004689606247578082)  --> TIME: 2025-02-25 19:12:00 -- STEP: 22338/36938 -- GLOBAL_STEP: 908850 | > loss_text_ce: 0.03220594674348831 (0.033290641232202725) | > loss_mel_ce: 1.402288556098938 (nan) | > loss: 0.005692438688129187 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13659251238945563) | > loader_time: 0.0047 (0.004689556929661484)  --> TIME: 2025-02-25 19:12:10 -- STEP: 22388/36938 -- GLOBAL_STEP: 908900 | > loss_text_ce: 0.03300965577363968 (0.03328930372286582) | > loss_mel_ce: 3.165966033935547 (nan) | > loss: 0.012694349512457848 (nan) | > current_lr: 5e-06 | > step_time: 0.1452 (0.13660858871043463) | > loader_time: 0.0046 (0.00468974653814484)  --> TIME: 2025-02-25 19:12:19 -- STEP: 22438/36938 -- GLOBAL_STEP: 908950 | > loss_text_ce: 0.04118817672133446 (0.033293416656410016) | > loss_mel_ce: 1.9494770765304565 (nan) | > loss: 0.007899465970695019 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13661034972774524) | > loader_time: 0.0045 (0.0046896078931062185)  --> TIME: 2025-02-25 19:12:28 -- STEP: 22488/36938 -- GLOBAL_STEP: 909000 | > loss_text_ce: 0.034579191356897354 (0.03329246575268747) | > loss_mel_ce: 3.2950551509857178 (nan) | > loss: 0.013212835416197777 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1366084991125646) | > loader_time: 0.0044 (0.004689569566140423)  --> TIME: 2025-02-25 19:12:38 -- STEP: 22538/36938 -- GLOBAL_STEP: 909050 | > loss_text_ce: 0.03197113424539566 (0.033292701741797374) | > loss_mel_ce: 3.013861656188965 (nan) | > loss: 0.012086639180779457 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13661031997429093) | > loader_time: 0.0046 (0.004689556311054645)  --> TIME: 2025-02-25 19:12:47 -- STEP: 22588/36938 -- GLOBAL_STEP: 909100 | > loss_text_ce: 0.036383677273988724 (0.03329411019819829) | > loss_mel_ce: 2.5284316539764404 (nan) | > loss: 0.010177838616073132 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13660670967044578) | > loader_time: 0.0046 (0.00468936419516553)  --> TIME: 2025-02-25 19:12:56 -- STEP: 22638/36938 -- GLOBAL_STEP: 909150 | > loss_text_ce: 0.027366163209080696 (0.0332942640415703) | > loss_mel_ce: 2.8644583225250244 (nan) | > loss: 0.01147549506276846 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.13660405817275306) | > loader_time: 0.0046 (0.004689223617407437)  --> TIME: 2025-02-25 19:13:05 -- STEP: 22688/36938 -- GLOBAL_STEP: 909200 | > loss_text_ce: 0.028015511110424995 (0.03329070618811096) | > loss_mel_ce: 2.9625496864318848 (nan) | > loss: 0.011867322959005833 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13659107758649092) | > loader_time: 0.0046 (0.004689654106818053)  --> TIME: 2025-02-25 19:13:14 -- STEP: 22738/36938 -- GLOBAL_STEP: 909250 | > loss_text_ce: 0.02608732134103775 (0.03328675426738876) | > loss_mel_ce: 2.1721298694610596 (nan) | > loss: 0.008723084814846516 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13658345948319142) | > loader_time: 0.0046 (0.004689769135038194)  --> TIME: 2025-02-25 19:13:24 -- STEP: 22788/36938 -- GLOBAL_STEP: 909300 | > loss_text_ce: 0.030320201069116592 (0.03328858239785315) | > loss_mel_ce: 2.6247169971466064 (nan) | > loss: 0.010535862296819687 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13659284924422876) | > loader_time: 0.0047 (0.004689859636670673)  --> TIME: 2025-02-25 19:13:33 -- STEP: 22838/36938 -- GLOBAL_STEP: 909350 | > loss_text_ce: 0.032161422073841095 (0.0332894977963486) | > loss_mel_ce: 2.8982348442077637 (nan) | > loss: 0.011628557927906513 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.1365855971906024) | > loader_time: 0.0045 (0.004689922891488691)  --> TIME: 2025-02-25 19:13:42 -- STEP: 22888/36938 -- GLOBAL_STEP: 909400 | > loss_text_ce: 0.02905770018696785 (0.0332893341498457) | > loss_mel_ce: 2.6692452430725098 (nan) | > loss: 0.010707552544772625 (nan) | > current_lr: 5e-06 | > step_time: 0.1763 (0.136586876838034) | > loader_time: 0.0048 (0.004690082245713038)  --> TIME: 2025-02-25 19:13:52 -- STEP: 22938/36938 -- GLOBAL_STEP: 909450 | > loss_text_ce: 0.03736886382102966 (0.03329154513324819) | > loss_mel_ce: 2.991452693939209 (nan) | > loss: 0.012019133195281029 (nan) | > current_lr: 5e-06 | > step_time: 0.1864 (0.13659444316989) | > loader_time: 0.0048 (0.004689959881455149)  --> TIME: 2025-02-25 19:14:01 -- STEP: 22988/36938 -- GLOBAL_STEP: 909500 | > loss_text_ce: 0.0346049889922142 (0.03329002047904014) | > loss_mel_ce: 0.020904541015625 (nan) | > loss: 0.00022027592058293521 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.1365925153172566) | > loader_time: 0.0046 (0.0046898605969961414)  --> TIME: 2025-02-25 19:14:11 -- STEP: 23038/36938 -- GLOBAL_STEP: 909550 | > loss_text_ce: 0.032423991709947586 (0.033287586305526286) | > loss_mel_ce: 1.8124356269836426 (nan) | > loss: 0.007320872042328119 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.1365893899861462) | > loader_time: 0.0045 (0.004689696327929416)  --> TIME: 2025-02-25 19:14:20 -- STEP: 23088/36938 -- GLOBAL_STEP: 909600 | > loss_text_ce: 0.04640002176165581 (0.033288741402801796) | > loss_mel_ce: 2.983109951019287 (nan) | > loss: 0.01202186569571495 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.136579738915668) | > loader_time: 0.0046 (0.004689579126079786)  --> TIME: 2025-02-25 19:14:29 -- STEP: 23138/36938 -- GLOBAL_STEP: 909650 | > loss_text_ce: 0.031705085188150406 (0.03329217185606278) | > loss_mel_ce: 2.456453323364258 (nan) | > loss: 0.009873645380139351 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.1365718793019944) | > loader_time: 0.0045 (0.004689633645351053)  --> TIME: 2025-02-25 19:14:38 -- STEP: 23188/36938 -- GLOBAL_STEP: 909700 | > loss_text_ce: 0.02990793250501156 (0.03329303091435622) | > loss_mel_ce: 2.1821532249450684 (nan) | > loss: 0.008778020739555359 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.1365708690152728) | > loader_time: 0.0046 (0.0046896723831483375)  --> TIME: 2025-02-25 19:14:47 -- STEP: 23238/36938 -- GLOBAL_STEP: 909750 | > loss_text_ce: 0.028755657374858856 (0.03329280428595605) | > loss_mel_ce: 3.2327301502227783 (nan) | > loss: 0.012942404486238956 (nan) | > current_lr: 5e-06 | > step_time: 0.1854 (0.13656590240158975) | > loader_time: 0.005 (0.004689674039276486)  --> TIME: 2025-02-25 19:14:57 -- STEP: 23288/36938 -- GLOBAL_STEP: 909800 | > loss_text_ce: 0.03512108325958252 (0.03329224804646973) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.136567668739113) | > loader_time: 0.0046 (0.004689727256241549)  --> TIME: 2025-02-25 19:15:06 -- STEP: 23338/36938 -- GLOBAL_STEP: 909850 | > loss_text_ce: 0.029478149488568306 (0.03329380259946742) | > loss_mel_ce: 3.1260986328125 (nan) | > loss: 0.012522130273282528 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.136563878070011) | > loader_time: 0.0046 (0.004689757167470616)  --> TIME: 2025-02-25 19:15:15 -- STEP: 23388/36938 -- GLOBAL_STEP: 909900 | > loss_text_ce: 0.031270306557416916 (0.033297617539855225) | > loss_mel_ce: 2.545822858810425 (nan) | > loss: 0.01022656075656414 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13656859930425602) | > loader_time: 0.0046 (0.004689795962353739)  --> TIME: 2025-02-25 19:15:25 -- STEP: 23438/36938 -- GLOBAL_STEP: 909950 | > loss_text_ce: 0.02916446328163147 (0.033298488184463196) | > loss_mel_ce: 3.3593547344207764 (nan) | > loss: 0.01344650611281395 (nan) | > current_lr: 5e-06 | > step_time: 0.1982 (0.13656551299399047) | > loader_time: 0.0048 (0.004689798022264587)  --> TIME: 2025-02-25 19:15:34 -- STEP: 23488/36938 -- GLOBAL_STEP: 910000 | > loss_text_ce: 0.021683553233742714 (0.0332984998395568) | > loss_mel_ce: 1.4114445447921753 (nan) | > loss: 0.005687016528099775 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13656547856428272) | > loader_time: 0.0045 (0.004689620488021288) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_910000.pth  --> TIME: 2025-02-25 19:15:50 -- STEP: 23538/36938 -- GLOBAL_STEP: 910050 | > loss_text_ce: 0.04384271427989006 (0.03329760358193354) | > loss_mel_ce: 3.366828680038452 (nan) | > loss: 0.013534411787986755 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13657919690424378) | > loader_time: 0.0047 (0.004689836706682859)  --> TIME: 2025-02-25 19:15:59 -- STEP: 23588/36938 -- GLOBAL_STEP: 910100 | > loss_text_ce: 0.04071898013353348 (0.03330014102492272) | > loss_mel_ce: 2.555497646331787 (nan) | > loss: 0.010302447713911533 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13657806006370565) | > loader_time: 0.0046 (0.004689743099565199)  --> TIME: 2025-02-25 19:16:08 -- STEP: 23638/36938 -- GLOBAL_STEP: 910150 | > loss_text_ce: 0.033386267721652985 (0.03330270087765488) | > loss_mel_ce: 1.5427734851837158 (nan) | > loss: 0.00625460222363472 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13657040135672183) | > loader_time: 0.0046 (0.004689608938309311)  --> TIME: 2025-02-25 19:16:18 -- STEP: 23688/36938 -- GLOBAL_STEP: 910200 | > loss_text_ce: 0.03302460163831711 (0.033300877755872045) | > loss_mel_ce: 1.9487991333007812 (nan) | > loss: 0.00786438025534153 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13657761754437203) | > loader_time: 0.0048 (0.004689575630601535)  --> TIME: 2025-02-25 19:16:27 -- STEP: 23738/36938 -- GLOBAL_STEP: 910250 | > loss_text_ce: 0.04303272068500519 (0.03330519845173589) | > loss_mel_ce: 0.01373291015625 (nan) | > loss: 0.00022526044631376863 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13656766244501115) | > loader_time: 0.0044 (0.004689389737918486)  --> TIME: 2025-02-25 19:16:36 -- STEP: 23788/36938 -- GLOBAL_STEP: 910300 | > loss_text_ce: 0.03308570384979248 (0.03330600084217017) | > loss_mel_ce: 3.3788914680480957 (nan) | > loss: 0.013539593666791916 (nan) | > current_lr: 5e-06 | > step_time: 0.2044 (0.1365778176488239) | > loader_time: 0.0055 (0.004689329308342563)  --> TIME: 2025-02-25 19:16:45 -- STEP: 23838/36938 -- GLOBAL_STEP: 910350 | > loss_text_ce: 0.03374570608139038 (0.03330427892523311) | > loss_mel_ce: 2.79106068611145 (nan) | > loss: 0.011209550313651562 (nan) | > current_lr: 5e-06 | > step_time: 0.1817 (0.136573891060015) | > loader_time: 0.0048 (0.004689180837980556)  --> TIME: 2025-02-25 19:16:55 -- STEP: 23888/36938 -- GLOBAL_STEP: 910400 | > loss_text_ce: 0.03181537613272667 (0.033304848318809706) | > loss_mel_ce: 2.6894350051879883 (nan) | > loss: 0.010798612609505653 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.1365801217726861) | > loader_time: 0.0046 (0.004689086036480914)  --> TIME: 2025-02-25 19:17:04 -- STEP: 23938/36938 -- GLOBAL_STEP: 910450 | > loss_text_ce: 0.03496482968330383 (0.03330457681127508) | > loss_mel_ce: 3.2083771228790283 (nan) | > loss: 0.012870404869318008 (nan) | > current_lr: 5e-06 | > step_time: 0.1874 (0.1365859918560888) | > loader_time: 0.005 (0.004689015425061287)  --> TIME: 2025-02-25 19:17:14 -- STEP: 23988/36938 -- GLOBAL_STEP: 910500 | > loss_text_ce: 0.03083167038857937 (0.03330423353610746) | > loss_mel_ce: 3.4987688064575195 (nan) | > loss: 0.014006351120769978 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.1365860960215985) | > loader_time: 0.0055 (0.004689047023298664)  --> TIME: 2025-02-25 19:17:23 -- STEP: 24038/36938 -- GLOBAL_STEP: 910550 | > loss_text_ce: 0.033949900418519974 (0.03330717579414286) | > loss_mel_ce: 1.2233585119247437 (nan) | > loss: 0.004989319015294313 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13659189036824942) | > loader_time: 0.0046 (0.004688986248933931)  --> TIME: 2025-02-25 19:17:32 -- STEP: 24088/36938 -- GLOBAL_STEP: 910600 | > loss_text_ce: 0.03520137816667557 (0.033306245246208586) | > loss_mel_ce: 2.6863698959350586 (nan) | > loss: 0.010799886658787727 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13659176562997122) | > loader_time: 0.0046 (0.004688883760133007)  --> TIME: 2025-02-25 19:17:42 -- STEP: 24138/36938 -- GLOBAL_STEP: 910650 | > loss_text_ce: 0.026579026132822037 (0.03330822415164968) | > loss_mel_ce: 2.187854051589966 (nan) | > loss: 0.008787433616816998 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13659575146701816) | > loader_time: 0.0049 (0.004688887304157005)  --> TIME: 2025-02-25 19:17:51 -- STEP: 24188/36938 -- GLOBAL_STEP: 910700 | > loss_text_ce: 0.04158861190080643 (0.03330480039833718) | > loss_mel_ce: 1.0149807929992676 (nan) | > loss: 0.004192735999822617 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13659690657351092) | > loader_time: 0.0045 (0.004688913376248137)  --> TIME: 2025-02-25 19:18:01 -- STEP: 24238/36938 -- GLOBAL_STEP: 910750 | > loss_text_ce: 0.04131696745753288 (0.03330486132214237) | > loss_mel_ce: 2.1847586631774902 (nan) | > loss: 0.008833633735775948 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13660087397767526) | > loader_time: 0.0047 (0.0046888550020934465)  --> TIME: 2025-02-25 19:18:10 -- STEP: 24288/36938 -- GLOBAL_STEP: 910800 | > loss_text_ce: 0.026753680780529976 (0.03330543174658993) | > loss_mel_ce: 2.9128201007843018 (nan) | > loss: 0.011664976365864277 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.13660380867590763) | > loader_time: 0.0048 (0.004688810787810325)  --> TIME: 2025-02-25 19:18:19 -- STEP: 24338/36938 -- GLOBAL_STEP: 910850 | > loss_text_ce: 0.03865218162536621 (0.03330687463818648) | > loss_mel_ce: 0.0284423828125 (nan) | > loss: 0.00026624827296473086 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.1366057407592463) | > loader_time: 0.0045 (0.004688626954408375)  --> TIME: 2025-02-25 19:18:28 -- STEP: 24388/36938 -- GLOBAL_STEP: 910900 | > loss_text_ce: 0.03739415481686592 (0.033303279800706166) | > loss_mel_ce: 3.004089593887329 (nan) | > loss: 0.012069379910826683 (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.1365968427913033) | > loader_time: 0.0048 (0.004688373487152507)  --> TIME: 2025-02-25 19:18:38 -- STEP: 24438/36938 -- GLOBAL_STEP: 910950 | > loss_text_ce: 0.023958930745720863 (0.03329967992196264) | > loss_mel_ce: 2.823525905609131 (nan) | > loss: 0.011299544014036655 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13660085063702915) | > loader_time: 0.0045 (0.0046881581691314575)  --> TIME: 2025-02-25 19:18:47 -- STEP: 24488/36938 -- GLOBAL_STEP: 911000 | > loss_text_ce: 0.032480090856552124 (0.03329874292835688) | > loss_mel_ce: 2.6399850845336914 (nan) | > loss: 0.010605020448565483 (nan) | > current_lr: 5e-06 | > step_time: 0.1446 (0.13660836987968897) | > loader_time: 0.0047 (0.00468822067487244)  --> TIME: 2025-02-25 19:18:57 -- STEP: 24538/36938 -- GLOBAL_STEP: 911050 | > loss_text_ce: 0.04802602529525757 (0.03329912739422462) | > loss_mel_ce: 3.3274571895599365 (nan) | > loss: 0.013394775800406933 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1366016814834818) | > loader_time: 0.0046 (0.004688269701998203)  --> TIME: 2025-02-25 19:19:06 -- STEP: 24588/36938 -- GLOBAL_STEP: 911100 | > loss_text_ce: 0.026009928435087204 (0.033298753554824756) | > loss_mel_ce: 2.966806650161743 (nan) | > loss: 0.011876257136464119 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13661094806204682) | > loader_time: 0.0049 (0.004688425220783038)  --> TIME: 2025-02-25 19:19:15 -- STEP: 24638/36938 -- GLOBAL_STEP: 911150 | > loss_text_ce: 0.02108681946992874 (0.033295742370646655) | > loss_mel_ce: 2.361457347869873 (nan) | > loss: 0.00945454090833664 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1366066546184259) | > loader_time: 0.0045 (0.004688375355584041)  --> TIME: 2025-02-25 19:19:25 -- STEP: 24688/36938 -- GLOBAL_STEP: 911200 | > loss_text_ce: 0.03411112725734711 (0.03329632902354643) | > loss_mel_ce: 2.8206984996795654 (nan) | > loss: 0.01132860966026783 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.13660852514492156) | > loader_time: 0.0046 (0.004688210741931252)  --> TIME: 2025-02-25 19:19:34 -- STEP: 24738/36938 -- GLOBAL_STEP: 911250 | > loss_text_ce: 0.037046611309051514 (0.033296484217211514) | > loss_mel_ce: 3.3396120071411133 (nan) | > loss: 0.013399439863860607 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.13661077312841335) | > loader_time: 0.0046 (0.004688317999900311)  --> TIME: 2025-02-25 19:19:43 -- STEP: 24788/36938 -- GLOBAL_STEP: 911300 | > loss_text_ce: 0.040513597428798676 (0.03329572202974431) | > loss_mel_ce: 2.9365475177764893 (nan) | > loss: 0.011813734658062458 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.1366081920596532) | > loader_time: 0.0045 (0.004688273933177348)  --> TIME: 2025-02-25 19:19:53 -- STEP: 24838/36938 -- GLOBAL_STEP: 911350 | > loss_text_ce: 0.023443026468157768 (0.033292981793416894) | > loss_mel_ce: 2.8369877338409424 (nan) | > loss: 0.011350915767252445 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13660512363884666) | > loader_time: 0.0045 (0.004688281263837845)  --> TIME: 2025-02-25 19:20:02 -- STEP: 24888/36938 -- GLOBAL_STEP: 911400 | > loss_text_ce: 0.03186642378568649 (0.03329411101006916) | > loss_mel_ce: 2.452941417694092 (nan) | > loss: 0.009860348887741566 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13660680789803337) | > loader_time: 0.0047 (0.004688253628023574)  --> TIME: 2025-02-25 19:20:11 -- STEP: 24938/36938 -- GLOBAL_STEP: 911450 | > loss_text_ce: 0.03001059964299202 (0.033294460276865195) | > loss_mel_ce: 2.1108272075653076 (nan) | > loss: 0.008495388552546501 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13659621496343505) | > loader_time: 0.0045 (0.0046881098192364835)  --> TIME: 2025-02-25 19:20:21 -- STEP: 24988/36938 -- GLOBAL_STEP: 911500 | > loss_text_ce: 0.03944232314825058 (0.033294613629878296) | > loss_mel_ce: 3.0161497592926025 (nan) | > loss: 0.012125366367399693 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.13660208706056515) | > loader_time: 0.0046 (0.004688113283801894)  --> TIME: 2025-02-25 19:20:30 -- STEP: 25038/36938 -- GLOBAL_STEP: 911550 | > loss_text_ce: 0.03585948795080185 (0.03329328473878167) | > loss_mel_ce: 2.1423513889312744 (nan) | > loss: 0.008643695153295994 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13660924831169072) | > loader_time: 0.0047 (0.004688069266017543)  --> TIME: 2025-02-25 19:20:39 -- STEP: 25088/36938 -- GLOBAL_STEP: 911600 | > loss_text_ce: 0.036863699555397034 (0.03329254159600207) | > loss_mel_ce: 3.141254186630249 (nan) | > loss: 0.012611580081284046 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.1366110043410127) | > loader_time: 0.0045 (0.004687932044343691)  --> TIME: 2025-02-25 19:20:49 -- STEP: 25138/36938 -- GLOBAL_STEP: 911650 | > loss_text_ce: 0.03091934695839882 (0.03329343078065953) | > loss_mel_ce: 2.880204439163208 (nan) | > loss: 0.011552078649401665 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13661763943866262) | > loader_time: 0.0045 (0.004687809206267353)  --> TIME: 2025-02-25 19:20:58 -- STEP: 25188/36938 -- GLOBAL_STEP: 911700 | > loss_text_ce: 0.04660620912909508 (0.03329388129184571) | > loss_mel_ce: 2.881035566329956 (nan) | > loss: 0.011617626994848251 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.13661515838629942) | > loader_time: 0.0046 (0.004687634378799443)  --> TIME: 2025-02-25 19:21:08 -- STEP: 25238/36938 -- GLOBAL_STEP: 911750 | > loss_text_ce: 0.03886311128735542 (0.03329464072074197) | > loss_mel_ce: 1.8644415140151978 (nan) | > loss: 0.007552796509116888 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.1366192611237406) | > loader_time: 0.0046 (0.004687627131384037)  --> TIME: 2025-02-25 19:21:17 -- STEP: 25288/36938 -- GLOBAL_STEP: 911800 | > loss_text_ce: 0.028906553983688354 (0.033292559988599896) | > loss_mel_ce: 1.430963158607483 (nan) | > loss: 0.005793134216219187 (nan) | > current_lr: 5e-06 | > step_time: 0.1074 (0.13660370543650918) | > loader_time: 0.0045 (0.004687538811844468)  --> TIME: 2025-02-25 19:21:26 -- STEP: 25338/36938 -- GLOBAL_STEP: 911850 | > loss_text_ce: 0.02742169052362442 (0.0332944899367689) | > loss_mel_ce: 2.925147533416748 (nan) | > loss: 0.011716545559465885 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.13660593917896918) | > loader_time: 0.0047 (0.004687777172654404)  --> TIME: 2025-02-25 19:21:35 -- STEP: 25388/36938 -- GLOBAL_STEP: 911900 | > loss_text_ce: 0.03162068873643875 (0.033294754459977406) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1439 (0.13659827822265533) | > loader_time: 0.0061 (0.004687868470711396)  --> TIME: 2025-02-25 19:21:44 -- STEP: 25438/36938 -- GLOBAL_STEP: 911950 | > loss_text_ce: 0.029784807935357094 (0.03329718467102952) | > loss_mel_ce: 2.5235838890075684 (nan) | > loss: 0.010132416151463985 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13660150334767335) | > loader_time: 0.0046 (0.004688279060215615)  --> TIME: 2025-02-25 19:21:54 -- STEP: 25488/36938 -- GLOBAL_STEP: 912000 | > loss_text_ce: 0.03028208576142788 (0.033297319931451116) | > loss_mel_ce: 2.650175094604492 (nan) | > loss: 0.010636734776198864 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.1366023106239755) | > loader_time: 0.0048 (0.004688353768251122)  --> TIME: 2025-02-25 19:22:03 -- STEP: 25538/36938 -- GLOBAL_STEP: 912050 | > loss_text_ce: 0.03542526066303253 (0.03329420353123542) | > loss_mel_ce: 3.4069573879241943 (nan) | > loss: 0.013660249300301075 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13659886600958288) | > loader_time: 0.0046 (0.004688199437096843)  --> TIME: 2025-02-25 19:22:12 -- STEP: 25588/36938 -- GLOBAL_STEP: 912100 | > loss_text_ce: 0.03667705878615379 (0.03329271188748886) | > loss_mel_ce: 1.8882899284362793 (nan) | > loss: 0.007638758048415184 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13660294330919484) | > loader_time: 0.0056 (0.004688174105517969)  --> TIME: 2025-02-25 19:22:22 -- STEP: 25638/36938 -- GLOBAL_STEP: 912150 | > loss_text_ce: 0.034842751920223236 (0.033293878978692446) | > loss_mel_ce: 2.2184510231018066 (nan) | > loss: 0.008941642008721828 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.1366108289067366) | > loader_time: 0.0046 (0.004688185568262793)  --> TIME: 2025-02-25 19:22:31 -- STEP: 25688/36938 -- GLOBAL_STEP: 912200 | > loss_text_ce: 0.026131531223654747 (0.03329250883042459) | > loss_mel_ce: 2.8990626335144043 (nan) | > loss: 0.011607914231717587 (nan) | > current_lr: 5e-06 | > step_time: 0.1961 (0.1366134887263615) | > loader_time: 0.0049 (0.004688189403545236)  --> TIME: 2025-02-25 19:22:41 -- STEP: 25738/36938 -- GLOBAL_STEP: 912250 | > loss_text_ce: 0.04123565927147865 (0.033291579070384214) | > loss_mel_ce: 3.1717889308929443 (nan) | > loss: 0.01275009848177433 (nan) | > current_lr: 5e-06 | > step_time: 0.1692 (0.1366136638941631) | > loader_time: 0.0048 (0.004688060221596854)  --> TIME: 2025-02-25 19:22:50 -- STEP: 25788/36938 -- GLOBAL_STEP: 912300 | > loss_text_ce: 0.05111284181475639 (0.03329275625880524) | > loss_mel_ce: 0.6370417475700378 (nan) | > loss: 0.002730772364884615 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.1366149441242138) | > loader_time: 0.0046 (0.004688023614831664)  --> TIME: 2025-02-25 19:22:59 -- STEP: 25838/36938 -- GLOBAL_STEP: 912350 | > loss_text_ce: 0.02796812169253826 (0.03328981905297859) | > loss_mel_ce: 3.1962296962738037 (nan) | > loss: 0.012794436886906624 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.1366120919657888) | > loader_time: 0.0047 (0.00468791823922708)  --> TIME: 2025-02-25 19:23:09 -- STEP: 25888/36938 -- GLOBAL_STEP: 912400 | > loss_text_ce: 0.028630198910832405 (0.033289765625054216) | > loss_mel_ce: 2.084458827972412 (nan) | > loss: 0.008385274559259415 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13661910471942662) | > loader_time: 0.0047 (0.004687974752100804)  --> TIME: 2025-02-25 19:23:18 -- STEP: 25938/36938 -- GLOBAL_STEP: 912450 | > loss_text_ce: 0.03356291726231575 (0.03329079027373626) | > loss_mel_ce: 2.108346939086914 (nan) | > loss: 0.008499642834067345 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13662213129824027) | > loader_time: 0.0048 (0.004688041865922183)  --> TIME: 2025-02-25 19:23:28 -- STEP: 25988/36938 -- GLOBAL_STEP: 912500 | > loss_text_ce: 0.032855745404958725 (0.03328983964221763) | > loss_mel_ce: 3.393805503845215 (nan) | > loss: 0.013597862794995308 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13662182959993513) | > loader_time: 0.0046 (0.004687980420592762)  --> TIME: 2025-02-25 19:23:37 -- STEP: 26038/36938 -- GLOBAL_STEP: 912550 | > loss_text_ce: 0.02843943051993847 (0.033289925528597136) | > loss_mel_ce: 3.278114080429077 (nan) | > loss: 0.013121245428919792 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13662782503040527) | > loader_time: 0.0047 (0.004687891878907032)  --> TIME: 2025-02-25 19:23:46 -- STEP: 26088/36938 -- GLOBAL_STEP: 912600 | > loss_text_ce: 0.029690582305192947 (0.033289185744894235) | > loss_mel_ce: 0.3729654848575592 (nan) | > loss: 0.0015978416195139289 (nan) | > current_lr: 5e-06 | > step_time: 0.1491 (0.13662278051757926) | > loader_time: 0.0046 (0.004687723801641235)  --> TIME: 2025-02-25 19:23:56 -- STEP: 26138/36938 -- GLOBAL_STEP: 912650 | > loss_text_ce: 0.028255769982933998 (0.03329122399572653) | > loss_mel_ce: 2.0884881019592285 (nan) | > loss: 0.008399777114391327 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13662354504107024) | > loader_time: 0.0045 (0.004687556540722504)  --> TIME: 2025-02-25 19:24:05 -- STEP: 26188/36938 -- GLOBAL_STEP: 912700 | > loss_text_ce: 0.036334775388240814 (0.033290503370736815) | > loss_mel_ce: 2.8987879753112793 (nan) | > loss: 0.01164731290191412 (nan) | > current_lr: 5e-06 | > step_time: 0.1626 (0.1366256803743012) | > loader_time: 0.0047 (0.004687435165951239)  --> TIME: 2025-02-25 19:24:14 -- STEP: 26238/36938 -- GLOBAL_STEP: 912750 | > loss_text_ce: 0.03649779409170151 (0.03328880764755137) | > loss_mel_ce: 3.142460346221924 (nan) | > loss: 0.01261491421610117 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.13662286366855872) | > loader_time: 0.0047 (0.004687310937101765)  --> TIME: 2025-02-25 19:24:23 -- STEP: 26288/36938 -- GLOBAL_STEP: 912800 | > loss_text_ce: 0.02673938497900963 (0.033285108927575836) | > loss_mel_ce: 2.4430177211761475 (nan) | > loss: 0.00980062410235405 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13661589726071016) | > loader_time: 0.0045 (0.0046871542767352465)  --> TIME: 2025-02-25 19:24:32 -- STEP: 26338/36938 -- GLOBAL_STEP: 912850 | > loss_text_ce: 0.034881819039583206 (0.033282768048228106) | > loss_mel_ce: 3.0466041564941406 (nan) | > loss: 0.012228120118379593 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13661030230665863) | > loader_time: 0.0046 (0.004687003905051351)  --> TIME: 2025-02-25 19:24:41 -- STEP: 26388/36938 -- GLOBAL_STEP: 912900 | > loss_text_ce: 0.03309734910726547 (0.03328357888327069) | > loss_mel_ce: 2.4161217212677 (nan) | > loss: 0.009719123132526875 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13660865041800102) | > loader_time: 0.0048 (0.004686916834010285)  --> TIME: 2025-02-25 19:24:51 -- STEP: 26438/36938 -- GLOBAL_STEP: 912950 | > loss_text_ce: 0.03105849213898182 (0.03328387382349315) | > loss_mel_ce: 2.6094322204589844 (nan) | > loss: 0.01047813892364502 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13661010747453967) | > loader_time: 0.0046 (0.004686801063283087)  --> TIME: 2025-02-25 19:25:00 -- STEP: 26488/36938 -- GLOBAL_STEP: 913000 | > loss_text_ce: 0.028764357790350914 (0.03328520556026074) | > loss_mel_ce: 3.0984699726104736 (nan) | > loss: 0.012409660033881664 (nan) | > current_lr: 5e-06 | > step_time: 0.1579 (0.13660723579430298) | > loader_time: 0.0046 (0.004686700428263092)  --> TIME: 2025-02-25 19:25:09 -- STEP: 26538/36938 -- GLOBAL_STEP: 913050 | > loss_text_ce: 0.029223615303635597 (0.03328865326433416) | > loss_mel_ce: 2.720158576965332 (nan) | > loss: 0.010910247452557087 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13661054317355856) | > loader_time: 0.0045 (0.004686644850108566)  --> TIME: 2025-02-25 19:25:19 -- STEP: 26588/36938 -- GLOBAL_STEP: 913100 | > loss_text_ce: 0.0354655459523201 (0.033288720013214784) | > loss_mel_ce: 2.144287109375 (nan) | > loss: 0.008649812079966068 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.1366162012174677) | > loader_time: 0.0046 (0.004686490743702233)  --> TIME: 2025-02-25 19:25:28 -- STEP: 26638/36938 -- GLOBAL_STEP: 913150 | > loss_text_ce: 0.04215041920542717 (0.033289613165923465) | > loss_mel_ce: 3.0548110008239746 (nan) | > loss: 0.012289530597627163 (nan) | > current_lr: 5e-06 | > step_time: 0.1519 (0.13662289471335862) | > loader_time: 0.0047 (0.0046864637285661285)  --> TIME: 2025-02-25 19:25:38 -- STEP: 26688/36938 -- GLOBAL_STEP: 913200 | > loss_text_ce: 0.04004041478037834 (0.033289075672332774) | > loss_mel_ce: 2.8757858276367188 (nan) | > loss: 0.01157074049115181 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13662940502273876) | > loader_time: 0.0045 (0.004686439843129209)  --> TIME: 2025-02-25 19:25:47 -- STEP: 26738/36938 -- GLOBAL_STEP: 913250 | > loss_text_ce: 0.03938523307442665 (0.033287933586716) | > loss_mel_ce: 3.2810189723968506 (nan) | > loss: 0.0131762083619833 (nan) | > current_lr: 5e-06 | > step_time: 0.1529 (0.13662777262499354) | > loader_time: 0.0046 (0.0046863483681765525)  --> TIME: 2025-02-25 19:25:56 -- STEP: 26788/36938 -- GLOBAL_STEP: 913300 | > loss_text_ce: 0.03135386481881142 (0.03328931928219146) | > loss_mel_ce: 2.8611156940460205 (nan) | > loss: 0.011478055268526077 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.1366239974743319) | > loader_time: 0.0045 (0.0046862301513902055)  --> TIME: 2025-02-25 19:26:06 -- STEP: 26838/36938 -- GLOBAL_STEP: 913350 | > loss_text_ce: 0.03240841627120972 (0.03328946804914682) | > loss_mel_ce: 3.3287012577056885 (nan) | > loss: 0.013337737880647182 (nan) | > current_lr: 5e-06 | > step_time: 0.1769 (0.1366206415542247) | > loader_time: 0.005 (0.004686134708506036)  --> TIME: 2025-02-25 19:26:15 -- STEP: 26888/36938 -- GLOBAL_STEP: 913400 | > loss_text_ce: 0.032285723835229874 (0.033290104950578346) | > loss_mel_ce: 3.0438013076782227 (nan) | > loss: 0.012206695042550564 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.1366226951730488) | > loader_time: 0.005 (0.004686143019837101)  --> TIME: 2025-02-25 19:26:25 -- STEP: 26938/36938 -- GLOBAL_STEP: 913450 | > loss_text_ce: 0.03212818503379822 (0.03329260176235333) | > loss_mel_ce: 3.155933380126953 (nan) | > loss: 0.012651038356125355 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.1366316739200644) | > loader_time: 0.0046 (0.004686406605917579)  --> TIME: 2025-02-25 19:26:34 -- STEP: 26988/36938 -- GLOBAL_STEP: 913500 | > loss_text_ce: 0.03161124140024185 (0.03329348634966825) | > loss_mel_ce: 2.9007716178894043 (nan) | > loss: 0.011636440642178059 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13663609658274137) | > loader_time: 0.0046 (0.004686403596102625)  --> TIME: 2025-02-25 19:26:44 -- STEP: 27038/36938 -- GLOBAL_STEP: 913550 | > loss_text_ce: 0.03552577644586563 (0.03329412658712692) | > loss_mel_ce: 2.4966204166412354 (nan) | > loss: 0.010048199445009232 (nan) | > current_lr: 5e-06 | > step_time: 0.1824 (0.13664181806145376) | > loader_time: 0.0048 (0.004686347284351596)  --> TIME: 2025-02-25 19:26:53 -- STEP: 27088/36938 -- GLOBAL_STEP: 913600 | > loss_text_ce: 0.028492536395788193 (0.0332935248290221) | > loss_mel_ce: 2.1897411346435547 (nan) | > loss: 0.008802514523267746 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13664103427443014) | > loader_time: 0.0046 (0.0046864396199909775)  --> TIME: 2025-02-25 19:27:01 -- STEP: 27138/36938 -- GLOBAL_STEP: 913650 | > loss_text_ce: 0.03772381320595741 (0.03329276481222023) | > loss_mel_ce: 2.4404263496398926 (nan) | > loss: 0.009833930060267448 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.1366233336072436) | > loader_time: 0.0046 (0.00468655799798533)  --> TIME: 2025-02-25 19:27:11 -- STEP: 27188/36938 -- GLOBAL_STEP: 913700 | > loss_text_ce: 0.0370926596224308 (0.033292763095406006) | > loss_mel_ce: 1.3919696807861328 (nan) | > loss: 0.005670882761478424 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13662162636665212) | > loader_time: 0.0047 (0.004686570876097536)  --> TIME: 2025-02-25 19:27:20 -- STEP: 27238/36938 -- GLOBAL_STEP: 913750 | > loss_text_ce: 0.02739800326526165 (0.033292263435524314) | > loss_mel_ce: 2.980801820755005 (nan) | > loss: 0.011937301605939865 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13661965926283925) | > loader_time: 0.0047 (0.00468666298430225)  --> TIME: 2025-02-25 19:27:29 -- STEP: 27288/36938 -- GLOBAL_STEP: 913800 | > loss_text_ce: 0.028415324166417122 (0.03329364702335408) | > loss_mel_ce: 2.4300808906555176 (nan) | > loss: 0.009755938313901424 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13662035857187688) | > loader_time: 0.0049 (0.004686622160405792)  --> TIME: 2025-02-25 19:27:38 -- STEP: 27338/36938 -- GLOBAL_STEP: 913850 | > loss_text_ce: 0.024321453645825386 (0.03329146683869628) | > loss_mel_ce: 2.7060415744781494 (nan) | > loss: 0.010834774933755398 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13660982000652425) | > loader_time: 0.0046 (0.004686638338964039)  --> TIME: 2025-02-25 19:27:47 -- STEP: 27388/36938 -- GLOBAL_STEP: 913900 | > loss_text_ce: 0.028707686811685562 (0.03329106657155748) | > loss_mel_ce: 0.9362127184867859 (nan) | > loss: 0.003829049412161112 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13660582319844286) | > loader_time: 0.0045 (0.004686749327943611)  --> TIME: 2025-02-25 19:27:57 -- STEP: 27438/36938 -- GLOBAL_STEP: 913950 | > loss_text_ce: 0.029899153858423233 (0.033291488850578865) | > loss_mel_ce: 3.1900200843811035 (nan) | > loss: 0.012777457945048809 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13660821924057287) | > loader_time: 0.0045 (0.004686943225970434)  --> TIME: 2025-02-25 19:28:06 -- STEP: 27488/36938 -- GLOBAL_STEP: 914000 | > loss_text_ce: 0.038171231746673584 (0.0332936309744213) | > loss_mel_ce: 2.7802865505218506 (nan) | > loss: 0.011184357106685638 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13660396054404356) | > loader_time: 0.0045 (0.004686849757734089)  --> TIME: 2025-02-25 19:28:15 -- STEP: 27538/36938 -- GLOBAL_STEP: 914050 | > loss_text_ce: 0.020245734602212906 (0.03329112581272608) | > loss_mel_ce: 0.013946533203125 (nan) | > loss: 0.00013568361464422196 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13659778055312435) | > loader_time: 0.0045 (0.00468667098592592)  --> TIME: 2025-02-25 19:28:24 -- STEP: 27588/36938 -- GLOBAL_STEP: 914100 | > loss_text_ce: 0.03276157006621361 (0.033290474062883366) | > loss_mel_ce: 1.8469876050949097 (nan) | > loss: 0.007459322456270456 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.1365909835420095) | > loader_time: 0.0046 (0.004686515720509029)  --> TIME: 2025-02-25 19:28:34 -- STEP: 27638/36938 -- GLOBAL_STEP: 914150 | > loss_text_ce: 0.04242146760225296 (0.033289531661785485) | > loss_mel_ce: 2.479492664337158 (nan) | > loss: 0.01000759657472372 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13659559291526452) | > loader_time: 0.0045 (0.004686563902993707)  --> TIME: 2025-02-25 19:28:43 -- STEP: 27688/36938 -- GLOBAL_STEP: 914200 | > loss_text_ce: 0.024853384122252464 (0.03328904084522481) | > loss_mel_ce: 0.035552978515625 (nan) | > loss: 0.00023970780603121966 (nan) | > current_lr: 5e-06 | > step_time: 0.1069 (0.13660956191141618) | > loader_time: 0.0044 (0.0046866656176377825)  --> TIME: 2025-02-25 19:28:53 -- STEP: 27738/36938 -- GLOBAL_STEP: 914250 | > loss_text_ce: 0.0372452437877655 (0.033290905142102804) | > loss_mel_ce: 2.305936813354492 (nan) | > loss: 0.009298342280089855 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13660553340759105) | > loader_time: 0.0048 (0.004686526372374438)  --> TIME: 2025-02-25 19:29:02 -- STEP: 27788/36938 -- GLOBAL_STEP: 914300 | > loss_text_ce: 0.026945719495415688 (0.033290570404549906) | > loss_mel_ce: 2.979140520095825 (nan) | > loss: 0.011928915046155453 (nan) | > current_lr: 5e-06 | > step_time: 0.1824 (0.13659895047260037) | > loader_time: 0.0048 (0.004686419734238245)  --> TIME: 2025-02-25 19:29:11 -- STEP: 27838/36938 -- GLOBAL_STEP: 914350 | > loss_text_ce: 0.040712907910346985 (0.033289710614647895) | > loss_mel_ce: 2.8700523376464844 (nan) | > loss: 0.011550656519830227 (nan) | > current_lr: 5e-06 | > step_time: 0.186 (0.13660123525692416) | > loader_time: 0.0048 (0.004686456737577239)  --> TIME: 2025-02-25 19:29:21 -- STEP: 27888/36938 -- GLOBAL_STEP: 914400 | > loss_text_ce: 0.03365403413772583 (0.03329200754345533) | > loss_mel_ce: 2.9602487087249756 (nan) | > loss: 0.01188056729733944 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13660751720030015) | > loader_time: 0.0046 (0.004686530215673184)  --> TIME: 2025-02-25 19:29:30 -- STEP: 27938/36938 -- GLOBAL_STEP: 914450 | > loss_text_ce: 0.03686933591961861 (0.03329206964474277) | > loss_mel_ce: 2.7701163291931152 (nan) | > loss: 0.011138832196593285 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13660396410132047) | > loader_time: 0.0046 (0.004686397389596194)  --> TIME: 2025-02-25 19:29:39 -- STEP: 27988/36938 -- GLOBAL_STEP: 914500 | > loss_text_ce: 0.034220676869153976 (0.033291318553742136) | > loss_mel_ce: 2.5891335010528564 (nan) | > loss: 0.010410136543214321 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.1366037503896436) | > loader_time: 0.0046 (0.004686235248556956)  --> TIME: 2025-02-25 19:29:48 -- STEP: 28038/36938 -- GLOBAL_STEP: 914550 | > loss_text_ce: 0.039713308215141296 (0.0332918815750967) | > loss_mel_ce: 2.81000018119812 (nan) | > loss: 0.011308387853205204 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13659772673774856) | > loader_time: 0.0045 (0.004686240777793175)  --> TIME: 2025-02-25 19:29:57 -- STEP: 28088/36938 -- GLOBAL_STEP: 914600 | > loss_text_ce: 0.03024030663073063 (0.03329328428713821) | > loss_mel_ce: 3.0125560760498047 (nan) | > loss: 0.012074589729309082 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13659646011288323) | > loader_time: 0.0046 (0.0046863045423535445)  --> TIME: 2025-02-25 19:30:07 -- STEP: 28138/36938 -- GLOBAL_STEP: 914650 | > loss_text_ce: 0.029836120083928108 (0.033293807255582744) | > loss_mel_ce: 2.57820725440979 (nan) | > loss: 0.010349378921091557 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13660419640556895) | > loader_time: 0.0046 (0.0046863276377713285)  --> TIME: 2025-02-25 19:30:16 -- STEP: 28188/36938 -- GLOBAL_STEP: 914700 | > loss_text_ce: 0.030222469940781593 (0.03329483963298298) | > loss_mel_ce: 2.9210891723632812 (nan) | > loss: 0.01171155460178852 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.13660570972774622) | > loader_time: 0.0082 (0.004686321216863121)  --> TIME: 2025-02-25 19:30:26 -- STEP: 28238/36938 -- GLOBAL_STEP: 914750 | > loss_text_ce: 0.03753038868308067 (0.03329604935148751) | > loss_mel_ce: 2.844529151916504 (nan) | > loss: 0.011436744593083858 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.1366082783908483) | > loader_time: 0.0047 (0.004686213931109284)  --> TIME: 2025-02-25 19:30:35 -- STEP: 28288/36938 -- GLOBAL_STEP: 914800 | > loss_text_ce: 0.03377528488636017 (0.03329430445443998) | > loss_mel_ce: 2.2547521591186523 (nan) | > loss: 0.009081458672881126 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.13660298712652616) | > loader_time: 0.0048 (0.004686173464580364)  --> TIME: 2025-02-25 19:30:44 -- STEP: 28338/36938 -- GLOBAL_STEP: 914850 | > loss_text_ce: 0.03145423159003258 (0.03329574111361352) | > loss_mel_ce: 3.0962769985198975 (nan) | > loss: 0.0124116325750947 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13660199261559625) | > loader_time: 0.0046 (0.004686299010788401)  --> TIME: 2025-02-25 19:30:53 -- STEP: 28388/36938 -- GLOBAL_STEP: 914900 | > loss_text_ce: 0.035901810973882675 (0.033296838078943874) | > loss_mel_ce: 3.0025882720947266 (nan) | > loss: 0.012057500891387463 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13659763854373258) | > loader_time: 0.0046 (0.0046864664183379244)  --> TIME: 2025-02-25 19:31:02 -- STEP: 28438/36938 -- GLOBAL_STEP: 914950 | > loss_text_ce: 0.03615851700305939 (0.03329504699545751) | > loss_mel_ce: 2.425981044769287 (nan) | > loss: 0.009770396165549755 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1365897415374943) | > loader_time: 0.0045 (0.004686483938454731)  --> TIME: 2025-02-25 19:31:12 -- STEP: 28488/36938 -- GLOBAL_STEP: 915000 | > loss_text_ce: 0.035942189395427704 (0.03329391434222609) | > loss_mel_ce: 2.7896201610565186 (nan) | > loss: 0.011212549172341824 (nan) | > current_lr: 5e-06 | > step_time: 0.179 (0.13659279971134689) | > loader_time: 0.0048 (0.0046864800810044)  --> TIME: 2025-02-25 19:31:21 -- STEP: 28538/36938 -- GLOBAL_STEP: 915050 | > loss_text_ce: 0.02970379777252674 (0.033296792185188155) | > loss_mel_ce: 1.9920278787612915 (nan) | > loss: 0.00802274513989687 (nan) | > current_lr: 5e-06 | > step_time: 0.1068 (0.13658751930282442) | > loader_time: 0.0044 (0.0046863659335984005)  --> TIME: 2025-02-25 19:31:30 -- STEP: 28588/36938 -- GLOBAL_STEP: 915100 | > loss_text_ce: 0.031577520072460175 (0.03329850433059937) | > loss_mel_ce: 2.891080856323242 (nan) | > loss: 0.011597851291298866 (nan) | > current_lr: 5e-06 | > step_time: 0.1954 (0.13658656532887442) | > loader_time: 0.0049 (0.004686269390511321)  --> TIME: 2025-02-25 19:31:40 -- STEP: 28638/36938 -- GLOBAL_STEP: 915150 | > loss_text_ce: 0.03051936626434326 (0.033299954067656755) | > loss_mel_ce: 2.1849026679992676 (nan) | > loss: 0.008791358210146427 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.13658885192717818) | > loader_time: 0.0049 (0.004686420011623624)  --> TIME: 2025-02-25 19:31:49 -- STEP: 28688/36938 -- GLOBAL_STEP: 915200 | > loss_text_ce: 0.039125364273786545 (0.033299091862053806) | > loss_mel_ce: 0.898393452167511 (nan) | > loss: 0.0037203130777925253 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13659944444711866) | > loader_time: 0.0046 (0.004686505857354364)  --> TIME: 2025-02-25 19:31:59 -- STEP: 28738/36938 -- GLOBAL_STEP: 915250 | > loss_text_ce: 0.03466447815299034 (0.0332999053430046) | > loss_mel_ce: 2.9630699157714844 (nan) | > loss: 0.011895772069692612 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.13660691407339184) | > loader_time: 0.0046 (0.0046864667112340025)  --> TIME: 2025-02-25 19:32:09 -- STEP: 28788/36938 -- GLOBAL_STEP: 915300 | > loss_text_ce: 0.028932571411132812 (0.03329972527659537) | > loss_mel_ce: 2.190483808517456 (nan) | > loss: 0.008807208389043808 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13661124597113136) | > loader_time: 0.0045 (0.004686474766982374)  --> TIME: 2025-02-25 19:32:18 -- STEP: 28838/36938 -- GLOBAL_STEP: 915350 | > loss_text_ce: 0.018597006797790527 (0.03329934652930124) | > loss_mel_ce: 2.4311234951019287 (nan) | > loss: 0.009721113368868828 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13661504352346465) | > loader_time: 0.0045 (0.004686370174715648)  --> TIME: 2025-02-25 19:32:27 -- STEP: 28888/36938 -- GLOBAL_STEP: 915400 | > loss_text_ce: 0.02260412834584713 (0.03330102940359516) | > loss_mel_ce: 2.003188133239746 (nan) | > loss: 0.008038858883082867 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1366165532064839) | > loader_time: 0.0047 (0.004686288459252801)  --> TIME: 2025-02-25 19:32:37 -- STEP: 28938/36938 -- GLOBAL_STEP: 915450 | > loss_text_ce: 0.03343239054083824 (0.033301750866712825) | > loss_mel_ce: 2.8308656215667725 (nan) | > loss: 0.011366263031959534 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13662119006161397) | > loader_time: 0.0045 (0.00468624421676709)  --> TIME: 2025-02-25 19:32:46 -- STEP: 28988/36938 -- GLOBAL_STEP: 915500 | > loss_text_ce: 0.04096684977412224 (0.033301172564360035) | > loss_mel_ce: 2.966742515563965 (nan) | > loss: 0.011935355141758919 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.1366254586062788) | > loader_time: 0.0045 (0.004686356972575571)  --> TIME: 2025-02-25 19:32:56 -- STEP: 29038/36938 -- GLOBAL_STEP: 915550 | > loss_text_ce: 0.033599723130464554 (0.03329984572711499) | > loss_mel_ce: 0.9445146918296814 (nan) | > loss: 0.0038814067374914885 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.136622517551795) | > loader_time: 0.0046 (0.004686467385963683)  --> TIME: 2025-02-25 19:33:05 -- STEP: 29088/36938 -- GLOBAL_STEP: 915600 | > loss_text_ce: 0.030861789360642433 (0.033298585591112845) | > loss_mel_ce: 2.5102226734161377 (nan) | > loss: 0.010083669796586037 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13661910760717375) | > loader_time: 0.0048 (0.004686328206453129)  --> TIME: 2025-02-25 19:33:14 -- STEP: 29138/36938 -- GLOBAL_STEP: 915650 | > loss_text_ce: 0.03499654307961464 (0.03329968581236808) | > loss_mel_ce: 1.6741312742233276 (nan) | > loss: 0.00678225327283144 (nan) | > current_lr: 5e-06 | > step_time: 0.1505 (0.1366163351383763) | > loader_time: 0.0047 (0.004686282407493683)  --> TIME: 2025-02-25 19:33:24 -- STEP: 29188/36938 -- GLOBAL_STEP: 915700 | > loss_text_ce: 0.028826525434851646 (0.03329791439888845) | > loss_mel_ce: 3.1124913692474365 (nan) | > loss: 0.012465547770261765 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13662431132912636) | > loader_time: 0.0046 (0.004686312224908421)  --> TIME: 2025-02-25 19:33:33 -- STEP: 29238/36938 -- GLOBAL_STEP: 915750 | > loss_text_ce: 0.02691826969385147 (0.03329685970397278) | > loss_mel_ce: 2.586343765258789 (nan) | > loss: 0.010370087809860706 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13662229718404562) | > loader_time: 0.0045 (0.004686178884800441)  --> TIME: 2025-02-25 19:33:42 -- STEP: 29288/36938 -- GLOBAL_STEP: 915800 | > loss_text_ce: 0.03628205135464668 (0.033294169916050116) | > loss_mel_ce: 2.1727116107940674 (nan) | > loss: 0.008765848353505135 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13661996397904322) | > loader_time: 0.0047 (0.004686075297576818)  --> TIME: 2025-02-25 19:33:51 -- STEP: 29338/36938 -- GLOBAL_STEP: 915850 | > loss_text_ce: 0.02469494566321373 (0.033295502599014586) | > loss_mel_ce: 2.6730093955993652 (nan) | > loss: 0.010705176740884781 (nan) | > current_lr: 5e-06 | > step_time: 0.1831 (0.13661891457265524) | > loader_time: 0.0048 (0.004685930227630347)  --> TIME: 2025-02-25 19:34:01 -- STEP: 29388/36938 -- GLOBAL_STEP: 915900 | > loss_text_ce: 0.027576956897974014 (0.033295007448932934) | > loss_mel_ce: 2.8305819034576416 (nan) | > loss: 0.011341900564730167 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.1366213628819669) | > loader_time: 0.0047 (0.004685895287521656)  --> TIME: 2025-02-25 19:34:10 -- STEP: 29438/36938 -- GLOBAL_STEP: 915950 | > loss_text_ce: 0.031443480402231216 (0.0332940387228829) | > loss_mel_ce: 2.810107946395874 (nan) | > loss: 0.011275999248027802 (nan) | > current_lr: 5e-06 | > step_time: 0.1814 (0.13661596322158387) | > loader_time: 0.0048 (0.0046859035933177326)  --> TIME: 2025-02-25 19:34:19 -- STEP: 29488/36938 -- GLOBAL_STEP: 916000 | > loss_text_ce: 0.034552332013845444 (0.03329631890344862) | > loss_mel_ce: 2.729106903076172 (nan) | > loss: 0.01096690259873867 (nan) | > current_lr: 5e-06 | > step_time: 0.1531 (0.13661434449259466) | > loader_time: 0.0046 (0.004685947308705935)  --> TIME: 2025-02-25 19:34:28 -- STEP: 29538/36938 -- GLOBAL_STEP: 916050 | > loss_text_ce: 0.03480909392237663 (0.033298584232464325) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13661289728440876) | > loader_time: 0.0048 (0.004685955918047523)  --> TIME: 2025-02-25 19:34:38 -- STEP: 29588/36938 -- GLOBAL_STEP: 916100 | > loss_text_ce: 0.0373230054974556 (0.03329904238183874) | > loss_mel_ce: 3.438786029815674 (nan) | > loss: 0.013794084079563618 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13661434220287813) | > loader_time: 0.0046 (0.004685831993387187)  --> TIME: 2025-02-25 19:34:47 -- STEP: 29638/36938 -- GLOBAL_STEP: 916150 | > loss_text_ce: 0.03645862638950348 (0.03329982142615788) | > loss_mel_ce: 2.7295172214508057 (nan) | > loss: 0.0109760956838727 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13660330138987375) | > loader_time: 0.0046 (0.0046856486690342895)  --> TIME: 2025-02-25 19:34:56 -- STEP: 29688/36938 -- GLOBAL_STEP: 916200 | > loss_text_ce: 0.0321626141667366 (0.03330057973931912) | > loss_mel_ce: 2.7375776767730713 (nan) | > loss: 0.010991034097969532 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.1366015810090806) | > loader_time: 0.0047 (0.00468563048931442)  --> TIME: 2025-02-25 19:35:05 -- STEP: 29738/36938 -- GLOBAL_STEP: 916250 | > loss_text_ce: 0.026573551818728447 (0.0333018749337551) | > loss_mel_ce: 2.556206703186035 (nan) | > loss: 0.010249128565192223 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13659564285770834) | > loader_time: 0.0046 (0.004685524845822405)  --> TIME: 2025-02-25 19:35:14 -- STEP: 29788/36938 -- GLOBAL_STEP: 916300 | > loss_text_ce: 0.030669352039694786 (0.03330129861914321) | > loss_mel_ce: 2.465266466140747 (nan) | > loss: 0.009904508478939533 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.1365999340754917) | > loader_time: 0.0046 (0.004685495545500095)  --> TIME: 2025-02-25 19:35:24 -- STEP: 29838/36938 -- GLOBAL_STEP: 916350 | > loss_text_ce: 0.030011530965566635 (0.033301772072475935) | > loss_mel_ce: 3.1686553955078125 (nan) | > loss: 0.012693122960627079 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.13660383458606956) | > loader_time: 0.0049 (0.0046854205182349975)  --> TIME: 2025-02-25 19:35:33 -- STEP: 29888/36938 -- GLOBAL_STEP: 916400 | > loss_text_ce: 0.0293675996363163 (0.03330019508820386) | > loss_mel_ce: 1.1255338191986084 (nan) | > loss: 0.004582942463457584 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.1365962746918066) | > loader_time: 0.0045 (0.004685362980439785)  --> TIME: 2025-02-25 19:35:42 -- STEP: 29938/36938 -- GLOBAL_STEP: 916450 | > loss_text_ce: 0.033626433461904526 (0.03330036428110509) | > loss_mel_ce: 2.942458391189575 (nan) | > loss: 0.011809860356152058 (nan) | > current_lr: 5e-06 | > step_time: 0.1779 (0.13659851205747867) | > loader_time: 0.0048 (0.004685340101920072)  --> TIME: 2025-02-25 19:35:51 -- STEP: 29988/36938 -- GLOBAL_STEP: 916500 | > loss_text_ce: 0.030276553705334663 (0.03329984813901143) | > loss_mel_ce: 2.4516217708587646 (nan) | > loss: 0.009848803281784058 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13659462389571286) | > loader_time: 0.0048 (0.004685214182146557)  --> TIME: 2025-02-25 19:36:01 -- STEP: 30038/36938 -- GLOBAL_STEP: 916550 | > loss_text_ce: 0.029257362708449364 (0.03329949441210681) | > loss_mel_ce: 2.9821884632110596 (nan) | > loss: 0.011950182728469372 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.13659744685273478) | > loader_time: 0.0048 (0.00468518153925347)  --> TIME: 2025-02-25 19:36:10 -- STEP: 30088/36938 -- GLOBAL_STEP: 916600 | > loss_text_ce: 0.03142639249563217 (0.03330163917508826) | > loss_mel_ce: 2.2918660640716553 (nan) | > loss: 0.009219415485858917 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13659869272700578) | > loader_time: 0.0044 (0.00468512993960646)  --> TIME: 2025-02-25 19:36:19 -- STEP: 30138/36938 -- GLOBAL_STEP: 916650 | > loss_text_ce: 0.03455390781164169 (0.03329963407727857) | > loss_mel_ce: 3.0048086643218994 (nan) | > loss: 0.012060963548719883 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.13658833675389512) | > loader_time: 0.0046 (0.004685043837714196)  --> TIME: 2025-02-25 19:36:28 -- STEP: 30188/36938 -- GLOBAL_STEP: 916700 | > loss_text_ce: 0.02962500974535942 (0.033299357443942974) | > loss_mel_ce: 2.5990569591522217 (nan) | > loss: 0.010431277565658092 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13658276083801973) | > loader_time: 0.0052 (0.004685085696764655)  --> TIME: 2025-02-25 19:36:37 -- STEP: 30238/36938 -- GLOBAL_STEP: 916750 | > loss_text_ce: 0.022743679583072662 (0.03329943340639206) | > loss_mel_ce: 1.3653043508529663 (nan) | > loss: 0.005508127622306347 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13657805083707672) | > loader_time: 0.0046 (0.004685022668695319)  --> TIME: 2025-02-25 19:36:47 -- STEP: 30288/36938 -- GLOBAL_STEP: 916800 | > loss_text_ce: 0.03176376223564148 (0.033298250189712914) | > loss_mel_ce: 3.073254108428955 (nan) | > loss: 0.012321500107645988 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.1365854998650017) | > loader_time: 0.0046 (0.004684956668547964)  --> TIME: 2025-02-25 19:36:56 -- STEP: 30338/36938 -- GLOBAL_STEP: 916850 | > loss_text_ce: 0.036316171288490295 (0.033298100426463195) | > loss_mel_ce: 2.799835205078125 (nan) | > loss: 0.0112545695155859 (nan) | > current_lr: 5e-06 | > step_time: 0.1628 (0.13658408709526054) | > loader_time: 0.0046 (0.004684817139495224)  --> TIME: 2025-02-25 19:37:06 -- STEP: 30388/36938 -- GLOBAL_STEP: 916900 | > loss_text_ce: 0.0255740974098444 (0.033299469443972655) | > loss_mel_ce: 3.344942331314087 (nan) | > loss: 0.013375066220760345 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13658076936639632) | > loader_time: 0.0045 (0.0046846656967530695)  --> TIME: 2025-02-25 19:37:15 -- STEP: 30438/36938 -- GLOBAL_STEP: 916950 | > loss_text_ce: 0.040563762187957764 (0.03329937991611681) | > loss_mel_ce: 2.558398485183716 (nan) | > loss: 0.01031334325671196 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13658490153710587) | > loader_time: 0.0049 (0.0046845285140055815)  --> TIME: 2025-02-25 19:37:24 -- STEP: 30488/36938 -- GLOBAL_STEP: 917000 | > loss_text_ce: 0.02894924022257328 (0.033298819427494686) | > loss_mel_ce: 2.8034722805023193 (nan) | > loss: 0.01123976893723011 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13658696904923082) | > loader_time: 0.0045 (0.004684543443833463)  --> TIME: 2025-02-25 19:37:34 -- STEP: 30538/36938 -- GLOBAL_STEP: 917050 | > loss_text_ce: 0.03664928302168846 (0.03329968944949352) | > loss_mel_ce: 0.8728163242340088 (nan) | > loss: 0.0036089906934648752 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1365900506938283) | > loader_time: 0.0045 (0.004684416599299919)  --> TIME: 2025-02-25 19:37:43 -- STEP: 30588/36938 -- GLOBAL_STEP: 917100 | > loss_text_ce: 0.03935086727142334 (0.03329923393451912) | > loss_mel_ce: 3.14151668548584 (nan) | > loss: 0.012622491456568241 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.1365873549706916) | > loader_time: 0.0047 (0.004684407640566655)  --> TIME: 2025-02-25 19:37:52 -- STEP: 30638/36938 -- GLOBAL_STEP: 917150 | > loss_text_ce: 0.049561142921447754 (0.03329913944558103) | > loss_mel_ce: 3.205059289932251 (nan) | > loss: 0.012915161438286304 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.1365815560993113) | > loader_time: 0.0046 (0.004684345608117403)  --> TIME: 2025-02-25 19:38:02 -- STEP: 30688/36938 -- GLOBAL_STEP: 917200 | > loss_text_ce: 0.03440900892019272 (0.03330056131669903) | > loss_mel_ce: 2.745414972305298 (nan) | > loss: 0.011031048372387886 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.136593801595806) | > loader_time: 0.0047 (0.004684533997670955)  --> TIME: 2025-02-25 19:38:12 -- STEP: 30738/36938 -- GLOBAL_STEP: 917250 | > loss_text_ce: 0.030802959576249123 (0.03329946926313007) | > loss_mel_ce: 3.1825990676879883 (nan) | > loss: 0.012751596048474312 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13660240428527853) | > loader_time: 0.0047 (0.004684721595937413)  --> TIME: 2025-02-25 19:38:21 -- STEP: 30788/36938 -- GLOBAL_STEP: 917300 | > loss_text_ce: 0.033906903117895126 (0.03329828992400447) | > loss_mel_ce: 3.1677939891815186 (nan) | > loss: 0.012705163098871708 (nan) | > current_lr: 5e-06 | > step_time: 0.1642 (0.13661011085891803) | > loader_time: 0.0047 (0.004684890611335588)  --> TIME: 2025-02-25 19:38:31 -- STEP: 30838/36938 -- GLOBAL_STEP: 917350 | > loss_text_ce: 0.025699572637677193 (0.03329924442086269) | > loss_mel_ce: 2.9815127849578857 (nan) | > loss: 0.011933383531868458 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13661097883366302) | > loader_time: 0.0046 (0.004684892337187251)  --> TIME: 2025-02-25 19:38:41 -- STEP: 30888/36938 -- GLOBAL_STEP: 917400 | > loss_text_ce: 0.03989337757229805 (0.033299757883224465) | > loss_mel_ce: 3.2324697971343994 (nan) | > loss: 0.01298556849360466 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.1366276739484063) | > loader_time: 0.0048 (0.004685095171025618)  --> TIME: 2025-02-25 19:38:50 -- STEP: 30938/36938 -- GLOBAL_STEP: 917450 | > loss_text_ce: 0.02972143515944481 (0.033300149127376374) | > loss_mel_ce: 0.8686023950576782 (nan) | > loss: 0.0035647773183882236 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13663883787665917) | > loader_time: 0.0045 (0.0046851899075334)  --> TIME: 2025-02-25 19:39:00 -- STEP: 30988/36938 -- GLOBAL_STEP: 917500 | > loss_text_ce: 0.03279759734869003 (0.03329968369326424) | > loss_mel_ce: 2.819735288619995 (nan) | > loss: 0.01131957583129406 (nan) | > current_lr: 5e-06 | > step_time: 0.1629 (0.13664905784421472) | > loader_time: 0.0047 (0.00468544634108361)  --> TIME: 2025-02-25 19:39:09 -- STEP: 31038/36938 -- GLOBAL_STEP: 917550 | > loss_text_ce: 0.034137412905693054 (0.03330139251189649) | > loss_mel_ce: 2.668069362640381 (nan) | > loss: 0.010723043233156204 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13665134784325525) | > loader_time: 0.0047 (0.004685657472522602)  --> TIME: 2025-02-25 19:39:19 -- STEP: 31088/36938 -- GLOBAL_STEP: 917600 | > loss_text_ce: 0.03810379281640053 (0.03330215609356695) | > loss_mel_ce: 2.9666099548339844 (nan) | > loss: 0.011923467740416527 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.1366490170904114) | > loader_time: 0.0046 (0.004685565415142867)  --> TIME: 2025-02-25 19:39:28 -- STEP: 31138/36938 -- GLOBAL_STEP: 917650 | > loss_text_ce: 0.040313657373189926 (0.033301145767412416) | > loss_mel_ce: 1.2633401155471802 (nan) | > loss: 0.005173229146748781 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13664498018448096) | > loader_time: 0.0045 (0.004685483155540595)  --> TIME: 2025-02-25 19:39:37 -- STEP: 31188/36938 -- GLOBAL_STEP: 917700 | > loss_text_ce: 0.037719760090112686 (0.033302183751449416) | > loss_mel_ce: 0.5821609497070312 (nan) | > loss: 0.0024598443415015936 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.1366453827520625) | > loader_time: 0.0047 (0.004685487696126717)  --> TIME: 2025-02-25 19:39:46 -- STEP: 31238/36938 -- GLOBAL_STEP: 917750 | > loss_text_ce: 0.025553707033395767 (0.03330208427284715) | > loss_mel_ce: 2.1528849601745605 (nan) | > loss: 0.00864459853619337 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13664046428490337) | > loader_time: 0.0046 (0.004685396963124763)  --> TIME: 2025-02-25 19:39:56 -- STEP: 31288/36938 -- GLOBAL_STEP: 917800 | > loss_text_ce: 0.02683264948427677 (0.03330123321214772) | > loss_mel_ce: 2.760768175125122 (nan) | > loss: 0.011061908677220345 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.1366379539608735) | > loader_time: 0.0047 (0.004685347364004301)  --> TIME: 2025-02-25 19:40:05 -- STEP: 31338/36938 -- GLOBAL_STEP: 917850 | > loss_text_ce: 0.032724615186452866 (0.03330134306635185) | > loss_mel_ce: 3.059494733810425 (nan) | > loss: 0.012270712293684483 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13663651349937178) | > loader_time: 0.0047 (0.004685291037942042)  --> TIME: 2025-02-25 19:40:14 -- STEP: 31388/36938 -- GLOBAL_STEP: 917900 | > loss_text_ce: 0.032972075045108795 (0.033301624513105456) | > loss_mel_ce: 2.233205556869507 (nan) | > loss: 0.008992768824100494 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13663636423515357) | > loader_time: 0.0047 (0.004685227014432979)  --> TIME: 2025-02-25 19:40:23 -- STEP: 31438/36938 -- GLOBAL_STEP: 917950 | > loss_text_ce: 0.03550725057721138 (0.03329993672353726) | > loss_mel_ce: 3.2377283573150635 (nan) | > loss: 0.01298903115093708 (nan) | > current_lr: 5e-06 | > step_time: 0.1895 (0.1366276874578025) | > loader_time: 0.0048 (0.004685062322841668)  --> TIME: 2025-02-25 19:40:33 -- STEP: 31488/36938 -- GLOBAL_STEP: 918000 | > loss_text_ce: 0.042651157826185226 (0.033299742122436235) | > loss_mel_ce: 2.307115077972412 (nan) | > loss: 0.009324469603598118 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13663043860134) | > loader_time: 0.0045 (0.004685199622640285)  --> TIME: 2025-02-25 19:40:42 -- STEP: 31538/36938 -- GLOBAL_STEP: 918050 | > loss_text_ce: 0.026943480595946312 (0.03329825744367016) | > loss_mel_ce: 3.0920231342315674 (nan) | > loss: 0.012376852333545685 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.136625340733457) | > loader_time: 0.0045 (0.004685265765872621)  --> TIME: 2025-02-25 19:40:51 -- STEP: 31588/36938 -- GLOBAL_STEP: 918100 | > loss_text_ce: 0.02333889901638031 (0.03329640661426541) | > loss_mel_ce: 2.132890462875366 (nan) | > loss: 0.008556465618312359 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.13662275308534308) | > loader_time: 0.0048 (0.00468525684104776)  --> TIME: 2025-02-25 19:41:00 -- STEP: 31638/36938 -- GLOBAL_STEP: 918150 | > loss_text_ce: 0.03085077926516533 (0.033296844035662965) | > loss_mel_ce: 3.4642863273620605 (nan) | > loss: 0.013869592919945717 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13662344332975332) | > loader_time: 0.0046 (0.004685343408322352)  --> TIME: 2025-02-25 19:41:10 -- STEP: 31688/36938 -- GLOBAL_STEP: 918200 | > loss_text_ce: 0.030843649059534073 (0.03329615025599808) | > loss_mel_ce: 2.8446290493011475 (nan) | > loss: 0.011410607025027275 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.1366273780809264) | > loader_time: 0.0046 (0.004685408469854781)  --> TIME: 2025-02-25 19:41:19 -- STEP: 31738/36938 -- GLOBAL_STEP: 918250 | > loss_text_ce: 0.02777068503201008 (0.03329550314117614) | > loss_mel_ce: 2.666515350341797 (nan) | > loss: 0.010691612027585506 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13662593361259948) | > loader_time: 0.0046 (0.004685508648218949)  --> TIME: 2025-02-25 19:41:29 -- STEP: 31788/36938 -- GLOBAL_STEP: 918300 | > loss_text_ce: 0.033556099981069565 (0.033294454344227224) | > loss_mel_ce: 2.332705020904541 (nan) | > loss: 0.009389924816787243 (nan) | > current_lr: 5e-06 | > step_time: 0.1121 (0.13663096580622908) | > loader_time: 0.0048 (0.00468552708850802)  --> TIME: 2025-02-25 19:41:38 -- STEP: 31838/36938 -- GLOBAL_STEP: 918350 | > loss_text_ce: 0.031873106956481934 (0.033293666887301225) | > loss_mel_ce: 2.4330081939697266 (nan) | > loss: 0.009781275875866413 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1366275697356823) | > loader_time: 0.0045 (0.004685522316463544)  --> TIME: 2025-02-25 19:41:47 -- STEP: 31888/36938 -- GLOBAL_STEP: 918400 | > loss_text_ce: 0.027686186134815216 (0.03329289682323738) | > loss_mel_ce: 1.9950200319290161 (nan) | > loss: 0.008026612922549248 (nan) | > current_lr: 5e-06 | > step_time: 0.1496 (0.13662821248106355) | > loader_time: 0.0047 (0.0046856622419582244)  --> TIME: 2025-02-25 19:41:57 -- STEP: 31938/36938 -- GLOBAL_STEP: 918450 | > loss_text_ce: 0.02195132151246071 (0.033291263835503064) | > loss_mel_ce: 2.0990262031555176 (nan) | > loss: 0.00841657817363739 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13663491480739653) | > loader_time: 0.0045 (0.004685723249328052)  --> TIME: 2025-02-25 19:42:06 -- STEP: 31988/36938 -- GLOBAL_STEP: 918500 | > loss_text_ce: 0.033134929835796356 (0.03329060062489129) | > loss_mel_ce: 2.7365431785583496 (nan) | > loss: 0.010990786366164684 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13663426808182702) | > loader_time: 0.0047 (0.004685887503925084)  --> TIME: 2025-02-25 19:42:15 -- STEP: 32038/36938 -- GLOBAL_STEP: 918550 | > loss_text_ce: 0.03893359750509262 (0.03329016586649479) | > loss_mel_ce: 2.5587167739868164 (nan) | > loss: 0.010308136232197285 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.13663400788976582) | > loader_time: 0.0047 (0.004685840116129724)  --> TIME: 2025-02-25 19:42:24 -- STEP: 32088/36938 -- GLOBAL_STEP: 918600 | > loss_text_ce: 0.028462376445531845 (0.03328936448237382) | > loss_mel_ce: 2.410991907119751 (nan) | > loss: 0.009680374525487423 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13662571649663072) | > loader_time: 0.0047 (0.004685857154222777)  --> TIME: 2025-02-25 19:42:33 -- STEP: 32138/36938 -- GLOBAL_STEP: 918650 | > loss_text_ce: 0.03259546682238579 (0.03328928131690362) | > loss_mel_ce: 2.630215644836426 (nan) | > loss: 0.01056671142578125 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.1366223217696325) | > loader_time: 0.0047 (0.00468582340357752)  --> TIME: 2025-02-25 19:42:43 -- STEP: 32188/36938 -- GLOBAL_STEP: 918700 | > loss_text_ce: 0.0368223674595356 (0.033288966501922056) | > loss_mel_ce: 3.2062885761260986 (nan) | > loss: 0.012869488447904587 (nan) | > current_lr: 5e-06 | > step_time: 0.1488 (0.13662376913562677) | > loader_time: 0.0047 (0.004685786106104409)  --> TIME: 2025-02-25 19:42:52 -- STEP: 32238/36938 -- GLOBAL_STEP: 918750 | > loss_text_ce: 0.034189656376838684 (0.03329057442755812) | > loss_mel_ce: 0.0508219413459301 (nan) | > loss: 0.0003373476502019912 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13661466002634326) | > loader_time: 0.0048 (0.00468567835573923)  --> TIME: 2025-02-25 19:43:01 -- STEP: 32288/36938 -- GLOBAL_STEP: 918800 | > loss_text_ce: 0.03591948375105858 (0.03329242798494564) | > loss_mel_ce: 1.2147657871246338 (nan) | > loss: 0.004963036626577377 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13661277748282463) | > loader_time: 0.0046 (0.004685654534752475)  --> TIME: 2025-02-25 19:43:11 -- STEP: 32338/36938 -- GLOBAL_STEP: 918850 | > loss_text_ce: 0.020806750282645226 (0.033290243924775226) | > loss_mel_ce: 3.8046875 (nan) | > loss: 0.01518053375184536 (nan) | > current_lr: 5e-06 | > step_time: 0.1078 (0.1366175864562519) | > loader_time: 0.0045 (0.004685723137948431)  --> TIME: 2025-02-25 19:43:20 -- STEP: 32388/36938 -- GLOBAL_STEP: 918900 | > loss_text_ce: 0.02153669111430645 (0.03329190505545787) | > loss_mel_ce: 2.4342472553253174 (nan) | > loss: 0.009745174087584019 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13661234077294776) | > loader_time: 0.0046 (0.004685742363041093)  --> TIME: 2025-02-25 19:43:29 -- STEP: 32438/36938 -- GLOBAL_STEP: 918950 | > loss_text_ce: 0.036754198372364044 (0.03329208102374683) | > loss_mel_ce: 2.445089340209961 (nan) | > loss: 0.009848585352301598 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13661387085186896) | > loader_time: 0.0046 (0.0046858427755209)  --> TIME: 2025-02-25 19:43:38 -- STEP: 32488/36938 -- GLOBAL_STEP: 919000 | > loss_text_ce: 0.02974265068769455 (0.0332918802399061) | > loss_mel_ce: 2.7363247871398926 (nan) | > loss: 0.0109764588996768 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.1366168340565793) | > loader_time: 0.0046 (0.004685759515017899)  --> TIME: 2025-02-25 19:43:48 -- STEP: 32538/36938 -- GLOBAL_STEP: 919050 | > loss_text_ce: 0.03414032235741615 (0.03329113089690791) | > loss_mel_ce: 2.9446024894714355 (nan) | > loss: 0.011820408515632153 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.1366116746612529) | > loader_time: 0.0047 (0.00468566184096935)  --> TIME: 2025-02-25 19:43:57 -- STEP: 32588/36938 -- GLOBAL_STEP: 919100 | > loss_text_ce: 0.038819801062345505 (0.03329191219376658) | > loss_mel_ce: 3.2291324138641357 (nan) | > loss: 0.012968065217137337 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13660696579427525) | > loader_time: 0.0045 (0.0046857245146635675)  --> TIME: 2025-02-25 19:44:06 -- STEP: 32638/36938 -- GLOBAL_STEP: 919150 | > loss_text_ce: 0.03338007256388664 (0.03329097539362474) | > loss_mel_ce: 2.7417500019073486 (nan) | > loss: 0.011012420989573002 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.136605605284908) | > loader_time: 0.0046 (0.004685670278025464)  --> TIME: 2025-02-25 19:44:16 -- STEP: 32688/36938 -- GLOBAL_STEP: 919200 | > loss_text_ce: 0.031599145382642746 (0.03329187402149771) | > loss_mel_ce: 3.2967610359191895 (nan) | > loss: 0.01320777926594019 (nan) | > current_lr: 5e-06 | > step_time: 0.1715 (0.13661316993390518) | > loader_time: 0.005 (0.00468575440226848)  --> TIME: 2025-02-25 19:44:25 -- STEP: 32738/36938 -- GLOBAL_STEP: 919250 | > loss_text_ce: 0.02896992303431034 (0.033294034105473984) | > loss_mel_ce: 2.9193789958953857 (nan) | > loss: 0.01169979851692915 (nan) | > current_lr: 5e-06 | > step_time: 0.178 (0.13661640312626588) | > loader_time: 0.0049 (0.0046857234735306575)  --> TIME: 2025-02-25 19:44:34 -- STEP: 32788/36938 -- GLOBAL_STEP: 919300 | > loss_text_ce: 0.03248044103384018 (0.03329459729023717) | > loss_mel_ce: 3.215714931488037 (nan) | > loss: 0.012889664620161057 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.1366169370426345) | > loader_time: 0.0047 (0.004685766321428673)  --> TIME: 2025-02-25 19:44:44 -- STEP: 32838/36938 -- GLOBAL_STEP: 919350 | > loss_text_ce: 0.02180582471191883 (0.03329532402385764) | > loss_mel_ce: 0.6761633157730103 (nan) | > loss: 0.0027697188779711723 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.1366178048383777) | > loader_time: 0.0045 (0.004685732905790227)  --> TIME: 2025-02-25 19:44:53 -- STEP: 32888/36938 -- GLOBAL_STEP: 919400 | > loss_text_ce: 0.03550165891647339 (0.03329726912771514) | > loss_mel_ce: 3.2814371585845947 (nan) | > loss: 0.013162456452846527 (nan) | > current_lr: 5e-06 | > step_time: 0.1725 (0.13661360781258247) | > loader_time: 0.0048 (0.004685681236251319)  --> TIME: 2025-02-25 19:45:02 -- STEP: 32938/36938 -- GLOBAL_STEP: 919450 | > loss_text_ce: 0.027498576790094376 (0.033295165183487054) | > loss_mel_ce: 3.2571756839752197 (nan) | > loss: 0.01303442195057869 (nan) | > current_lr: 5e-06 | > step_time: 0.1976 (0.13661136856799402) | > loader_time: 0.0049 (0.004685815670963806)  --> TIME: 2025-02-25 19:45:12 -- STEP: 32988/36938 -- GLOBAL_STEP: 919500 | > loss_text_ce: 0.029948174953460693 (0.03329547487360336) | > loss_mel_ce: 1.2606689929962158 (nan) | > loss: 0.0051214974373579025 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13661361018917748) | > loader_time: 0.0045 (0.0046857403483089605)  --> TIME: 2025-02-25 19:45:21 -- STEP: 33038/36938 -- GLOBAL_STEP: 919550 | > loss_text_ce: 0.03368914872407913 (0.033295506502598736) | > loss_mel_ce: 1.403631329536438 (nan) | > loss: 0.005703653208911419 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13661169489097194) | > loader_time: 0.0046 (0.004685940613461739)  --> TIME: 2025-02-25 19:45:30 -- STEP: 33088/36938 -- GLOBAL_STEP: 919600 | > loss_text_ce: 0.03645622357726097 (0.0332961899067506) | > loss_mel_ce: 2.973677158355713 (nan) | > loss: 0.011944973841309547 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.1366032840219289) | > loader_time: 0.0046 (0.004686008828982167)  --> TIME: 2025-02-25 19:45:39 -- STEP: 33138/36938 -- GLOBAL_STEP: 919650 | > loss_text_ce: 0.03555076941847801 (0.03329823461154767) | > loss_mel_ce: 3.0142979621887207 (nan) | > loss: 0.012102575041353703 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.1366024972360901) | > loader_time: 0.0049 (0.004685902093328677)  --> TIME: 2025-02-25 19:45:48 -- STEP: 33188/36938 -- GLOBAL_STEP: 919700 | > loss_text_ce: 0.035522766411304474 (0.03329741037349139) | > loss_mel_ce: 3.325672149658203 (nan) | > loss: 0.013338075950741768 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13659757388648075) | > loader_time: 0.0046 (0.004685820657631946)  --> TIME: 2025-02-25 19:45:57 -- STEP: 33238/36938 -- GLOBAL_STEP: 919750 | > loss_text_ce: 0.029200097545981407 (0.033295630733021546) | > loss_mel_ce: 2.524946928024292 (nan) | > loss: 0.01013550441712141 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13659585823442572) | > loader_time: 0.0045 (0.0046857997350969955)  --> TIME: 2025-02-25 19:46:07 -- STEP: 33288/36938 -- GLOBAL_STEP: 919800 | > loss_text_ce: 0.03429190441966057 (0.033296985968079354) | > loss_mel_ce: 3.2103018760681152 (nan) | > loss: 0.012875373475253582 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.1365952766792494) | > loader_time: 0.0049 (0.00468581455082196)  --> TIME: 2025-02-25 19:46:16 -- STEP: 33338/36938 -- GLOBAL_STEP: 919850 | > loss_text_ce: 0.0320240817964077 (0.03329835441118821) | > loss_mel_ce: 2.0151543617248535 (nan) | > loss: 0.008123724721372128 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.13659705196890654) | > loader_time: 0.0048 (0.0046857398990478045)  --> TIME: 2025-02-25 19:46:26 -- STEP: 33388/36938 -- GLOBAL_STEP: 919900 | > loss_text_ce: 0.04116324335336685 (0.03330099720696861) | > loss_mel_ce: 3.335836172103882 (nan) | > loss: 0.013400792144238949 (nan) | > current_lr: 5e-06 | > step_time: 0.144 (0.13660464720110685) | > loader_time: 0.0046 (0.004685683380105021)  --> TIME: 2025-02-25 19:46:35 -- STEP: 33438/36938 -- GLOBAL_STEP: 919950 | > loss_text_ce: 0.04016697779297829 (0.03330146466263244) | > loss_mel_ce: 2.7050111293792725 (nan) | > loss: 0.010893564671278 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13660658799088152) | > loader_time: 0.0045 (0.004685797027669122)  --> TIME: 2025-02-25 19:46:45 -- STEP: 33488/36938 -- GLOBAL_STEP: 920000 | > loss_text_ce: 0.038824114948511124 (0.03330168035167794) | > loss_mel_ce: 2.8124587535858154 (nan) | > loss: 0.011314614675939083 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13660439025170978) | > loader_time: 0.0045 (0.004685830255472888) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_920000.pth  --> TIME: 2025-02-25 19:47:00 -- STEP: 33538/36938 -- GLOBAL_STEP: 920050 | > loss_text_ce: 0.028403265401721 (0.03330077176505514) | > loss_mel_ce: 1.9176379442214966 (nan) | > loss: 0.007722386159002781 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13660493978440802) | > loader_time: 0.0045 (0.004686174981886553)  --> TIME: 2025-02-25 19:47:09 -- STEP: 33588/36938 -- GLOBAL_STEP: 920100 | > loss_text_ce: 0.025615932419896126 (0.033299999567355555) | > loss_mel_ce: 2.83748722076416 (nan) | > loss: 0.011361520737409592 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13659747493427404) | > loader_time: 0.0047 (0.00468618371904898)  --> TIME: 2025-02-25 19:47:18 -- STEP: 33638/36938 -- GLOBAL_STEP: 920150 | > loss_text_ce: 0.03377845138311386 (0.03329938609047039) | > loss_mel_ce: 1.6782640218734741 (nan) | > loss: 0.006793819833546877 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13659063377280503) | > loader_time: 0.0046 (0.004686264321540249)  --> TIME: 2025-02-25 19:47:27 -- STEP: 33688/36938 -- GLOBAL_STEP: 920200 | > loss_text_ce: 0.02826526015996933 (0.03329896123586507) | > loss_mel_ce: 2.4089784622192383 (nan) | > loss: 0.00967160239815712 (nan) | > current_lr: 5e-06 | > step_time: 0.1084 (0.1365837532433322) | > loader_time: 0.0045 (0.004686198249278806)  --> TIME: 2025-02-25 19:47:36 -- STEP: 33738/36938 -- GLOBAL_STEP: 920250 | > loss_text_ce: 0.031186960637569427 (0.03329844605170454) | > loss_mel_ce: 2.9387013912200928 (nan) | > loss: 0.011785272508859634 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13657690017463137) | > loader_time: 0.0046 (0.004686178525910784)  --> TIME: 2025-02-25 19:47:46 -- STEP: 33788/36938 -- GLOBAL_STEP: 920300 | > loss_text_ce: 0.03321661800146103 (0.03329937598154022) | > loss_mel_ce: 3.0150554180145264 (nan) | > loss: 0.012096318416297436 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13658018820353757) | > loader_time: 0.0046 (0.004686261840694776)  --> TIME: 2025-02-25 19:47:55 -- STEP: 33838/36938 -- GLOBAL_STEP: 920350 | > loss_text_ce: 0.028321001678705215 (0.033299450792762524) | > loss_mel_ce: 3.2129690647125244 (nan) | > loss: 0.01286226324737072 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13658019137781244) | > loader_time: 0.0046 (0.00468641702387166)  --> TIME: 2025-02-25 19:48:04 -- STEP: 33888/36938 -- GLOBAL_STEP: 920400 | > loss_text_ce: 0.03283514082431793 (0.033300641942913205) | > loss_mel_ce: 2.2826218605041504 (nan) | > loss: 0.009188322350382805 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1365816267286408) | > loader_time: 0.0046 (0.004686408005197514)  --> TIME: 2025-02-25 19:48:14 -- STEP: 33938/36938 -- GLOBAL_STEP: 920450 | > loss_text_ce: 0.029943061992526054 (0.033299707555107666) | > loss_mel_ce: 3.228290319442749 (nan) | > loss: 0.012929498217999935 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1365834308981475) | > loader_time: 0.0046 (0.004686338807804662)  --> TIME: 2025-02-25 19:48:23 -- STEP: 33988/36938 -- GLOBAL_STEP: 920500 | > loss_text_ce: 0.03416932374238968 (0.033299535541723116) | > loss_mel_ce: 2.5311319828033447 (nan) | > loss: 0.010179768316447735 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13658459703263104) | > loader_time: 0.0046 (0.004686320608077764)  --> TIME: 2025-02-25 19:48:32 -- STEP: 34038/36938 -- GLOBAL_STEP: 920550 | > loss_text_ce: 0.04168053716421127 (0.0332989447356805) | > loss_mel_ce: 2.6658694744110107 (nan) | > loss: 0.010744246654212475 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1365844364682904) | > loader_time: 0.0047 (0.004686234231154608)  --> TIME: 2025-02-25 19:48:42 -- STEP: 34088/36938 -- GLOBAL_STEP: 920600 | > loss_text_ce: 0.034772373735904694 (0.03329845360621546) | > loss_mel_ce: 2.948143482208252 (nan) | > loss: 0.01183696836233139 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13658695722012298) | > loader_time: 0.0045 (0.004686282130648518)  --> TIME: 2025-02-25 19:48:51 -- STEP: 34138/36938 -- GLOBAL_STEP: 920650 | > loss_text_ce: 0.03276332467794418 (0.033298385946725045) | > loss_mel_ce: 0.00823974609375 (nan) | > loss: 0.00016271061031147838 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.1365816756804446) | > loader_time: 0.0046 (0.00468624675970197)  --> TIME: 2025-02-25 19:49:00 -- STEP: 34188/36938 -- GLOBAL_STEP: 920700 | > loss_text_ce: 0.032395098358392715 (0.033298798019122604) | > loss_mel_ce: 2.712620973587036 (nan) | > loss: 0.010892922058701515 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13658113427032106) | > loader_time: 0.0046 (0.004686254373806831)  --> TIME: 2025-02-25 19:49:10 -- STEP: 34238/36938 -- GLOBAL_STEP: 920750 | > loss_text_ce: 0.03415266051888466 (0.03329933316161926) | > loss_mel_ce: 0.04403368756175041 (nan) | > loss: 0.0003102633054368198 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13658646194238) | > loader_time: 0.0045 (0.004686342443607357)  --> TIME: 2025-02-25 19:49:19 -- STEP: 34288/36938 -- GLOBAL_STEP: 920800 | > loss_text_ce: 0.03653867542743683 (0.033297935478604115) | > loss_mel_ce: 2.6898252964019775 (nan) | > loss: 0.01081890519708395 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1365839374493723) | > loader_time: 0.0046 (0.004686383717370495)  --> TIME: 2025-02-25 19:49:29 -- STEP: 34338/36938 -- GLOBAL_STEP: 920850 | > loss_text_ce: 0.02471950650215149 (0.033297682471235455) | > loss_mel_ce: 1.4633190631866455 (nan) | > loss: 0.005904915276914835 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.1365888437301884) | > loader_time: 0.0045 (0.004686522951803216)  --> TIME: 2025-02-25 19:49:38 -- STEP: 34388/36938 -- GLOBAL_STEP: 920900 | > loss_text_ce: 0.030622389167547226 (0.03329578248712549) | > loss_mel_ce: 2.7243728637695312 (nan) | > loss: 0.010932521894574165 (nan) | > current_lr: 5e-06 | > step_time: 0.1794 (0.1365934072004517) | > loader_time: 0.0048 (0.00468656023089188)  --> TIME: 2025-02-25 19:49:47 -- STEP: 34438/36938 -- GLOBAL_STEP: 920950 | > loss_text_ce: 0.03577901050448418 (0.033297092190096744) | > loss_mel_ce: 3.060546875 (nan) | > loss: 0.012287008576095104 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13658887207096088) | > loader_time: 0.0045 (0.004686448575312334)  --> TIME: 2025-02-25 19:49:56 -- STEP: 34488/36938 -- GLOBAL_STEP: 921000 | > loss_text_ce: 0.03736259788274765 (0.03329723993831203) | > loss_mel_ce: 3.1115682125091553 (nan) | > loss: 0.012495758011937141 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.1365840095457185) | > loader_time: 0.0048 (0.00468652615052757)  --> TIME: 2025-02-25 19:50:06 -- STEP: 34538/36938 -- GLOBAL_STEP: 921050 | > loss_text_ce: 0.040310267359018326 (0.03329716695411542) | > loss_mel_ce: 2.906172752380371 (nan) | > loss: 0.011692394502460957 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.1365941625138708) | > loader_time: 0.0045 (0.004686745442266695)  --> TIME: 2025-02-25 19:50:16 -- STEP: 34588/36938 -- GLOBAL_STEP: 921100 | > loss_text_ce: 0.04371032491326332 (0.03329638013153821) | > loss_mel_ce: 1.6672643423080444 (nan) | > loss: 0.0067895823158323765 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13659446703652783) | > loader_time: 0.0053 (0.004686854458329482)  --> TIME: 2025-02-25 19:50:24 -- STEP: 34638/36938 -- GLOBAL_STEP: 921150 | > loss_text_ce: 0.024969298392534256 (0.03329881896119171) | > loss_mel_ce: 2.118352174758911 (nan) | > loss: 0.008505244739353657 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13658414834007743) | > loader_time: 0.0047 (0.004686813547495749)  --> TIME: 2025-02-25 19:50:34 -- STEP: 34688/36938 -- GLOBAL_STEP: 921200 | > loss_text_ce: 0.031478676944971085 (0.03329878906370674) | > loss_mel_ce: 3.1051902770996094 (nan) | > loss: 0.012447099201381207 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.13659277739779893) | > loader_time: 0.0047 (0.004686789002915603)  --> TIME: 2025-02-25 19:50:43 -- STEP: 34738/36938 -- GLOBAL_STEP: 921250 | > loss_text_ce: 0.025846710428595543 (0.033299420468982745) | > loss_mel_ce: 3.054474353790283 (nan) | > loss: 0.012223497033119202 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13659034195914763) | > loader_time: 0.0045 (0.004686723719593635)  --> TIME: 2025-02-25 19:50:52 -- STEP: 34788/36938 -- GLOBAL_STEP: 921300 | > loss_text_ce: 0.04381128028035164 (0.03329946770322793) | > loss_mel_ce: 2.664022445678711 (nan) | > loss: 0.010745372623205185 (nan) | > current_lr: 5e-06 | > step_time: 0.1495 (0.13658632124047287) | > loader_time: 0.0047 (0.004686761556226015)  --> TIME: 2025-02-25 19:51:02 -- STEP: 34838/36938 -- GLOBAL_STEP: 921350 | > loss_text_ce: 0.038653239607810974 (0.03330142917059652) | > loss_mel_ce: 3.0161526203155518 (nan) | > loss: 0.012122245505452156 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13659057347411302) | > loader_time: 0.0048 (0.004686779273460363)  --> TIME: 2025-02-25 19:51:11 -- STEP: 34888/36938 -- GLOBAL_STEP: 921400 | > loss_text_ce: 0.03470195084810257 (0.0333014401096066) | > loss_mel_ce: 2.5279245376586914 (nan) | > loss: 0.010169154033064842 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13659768022820565) | > loader_time: 0.0047 (0.00468686967233923)  --> TIME: 2025-02-25 19:51:21 -- STEP: 34938/36938 -- GLOBAL_STEP: 921450 | > loss_text_ce: 0.036737941205501556 (0.0333013267877574) | > loss_mel_ce: 2.7907567024230957 (nan) | > loss: 0.011220217682421207 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13660459446370593) | > loader_time: 0.0049 (0.0046868676878330795)  --> TIME: 2025-02-25 19:51:31 -- STEP: 34988/36938 -- GLOBAL_STEP: 921500 | > loss_text_ce: 0.03120911680161953 (0.03330128262829059) | > loss_mel_ce: 2.920424222946167 (nan) | > loss: 0.011712831445038319 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13660842338343657) | > loader_time: 0.0047 (0.004686875105912707)  --> TIME: 2025-02-25 19:51:40 -- STEP: 35038/36938 -- GLOBAL_STEP: 921550 | > loss_text_ce: 0.036501169204711914 (0.033301799915821516) | > loss_mel_ce: 0.0240478515625 (nan) | > loss: 0.0002402739046374336 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13660247982266108) | > loader_time: 0.0045 (0.004686931767919063)  --> TIME: 2025-02-25 19:51:49 -- STEP: 35088/36938 -- GLOBAL_STEP: 921600 | > loss_text_ce: 0.027909649536013603 (0.033301981183470046) | > loss_mel_ce: 2.6896121501922607 (nan) | > loss: 0.010783817619085312 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13659478925810545) | > loader_time: 0.0045 (0.004686852255424204)  --> TIME: 2025-02-25 19:51:58 -- STEP: 35138/36938 -- GLOBAL_STEP: 921650 | > loss_text_ce: 0.03420186787843704 (0.03330263693700494) | > loss_mel_ce: 2.5812854766845703 (nan) | > loss: 0.01037891861051321 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13658765885071128) | > loader_time: 0.0048 (0.004686703800822628)  --> TIME: 2025-02-25 19:52:07 -- STEP: 35188/36938 -- GLOBAL_STEP: 921700 | > loss_text_ce: 0.03571084514260292 (0.03330199970257449) | > loss_mel_ce: 2.996795654296875 (nan) | > loss: 0.012033756822347641 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13659004522369933) | > loader_time: 0.0046 (0.004686696361083409)  --> TIME: 2025-02-25 19:52:16 -- STEP: 35238/36938 -- GLOBAL_STEP: 921750 | > loss_text_ce: 0.041086792945861816 (0.03330347767711102) | > loss_mel_ce: 1.7389849424362183 (nan) | > loss: 0.007063777185976505 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13658923373142975) | > loader_time: 0.0044 (0.004686672406472933)  --> TIME: 2025-02-25 19:52:25 -- STEP: 35288/36938 -- GLOBAL_STEP: 921800 | > loss_text_ce: 0.028698060661554337 (0.03330406061448809) | > loss_mel_ce: 2.84085750579834 (nan) | > loss: 0.011387125588953495 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13657802198782257) | > loader_time: 0.0046 (0.0046867108607178826)  --> TIME: 2025-02-25 19:52:35 -- STEP: 35338/36938 -- GLOBAL_STEP: 921850 | > loss_text_ce: 0.030589474365115166 (0.033303779702964775) | > loss_mel_ce: 2.8217201232910156 (nan) | > loss: 0.0113186901435256 (nan) | > current_lr: 5e-06 | > step_time: 0.1521 (0.1365853417296847) | > loader_time: 0.0047 (0.004686839134302252)  --> TIME: 2025-02-25 19:52:44 -- STEP: 35388/36938 -- GLOBAL_STEP: 921900 | > loss_text_ce: 0.03587567061185837 (0.033303929614996855) | > loss_mel_ce: 2.625391960144043 (nan) | > loss: 0.010560586117208004 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.136589536117647) | > loader_time: 0.0056 (0.0046869601599170765)  --> TIME: 2025-02-25 19:52:54 -- STEP: 35438/36938 -- GLOBAL_STEP: 921950 | > loss_text_ce: 0.02916124276816845 (0.033303930912021905) | > loss_mel_ce: 3.210737705230713 (nan) | > loss: 0.0128567423671484 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13659544319807446) | > loader_time: 0.0047 (0.004687001207449552)  --> TIME: 2025-02-25 19:53:03 -- STEP: 35488/36938 -- GLOBAL_STEP: 922000 | > loss_text_ce: 0.03440368175506592 (0.03330175165514007) | > loss_mel_ce: 1.463227391242981 (nan) | > loss: 0.005942980758845806 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1365899694957523) | > loader_time: 0.0046 (0.004687092822075337)  --> TIME: 2025-02-25 19:53:13 -- STEP: 35538/36938 -- GLOBAL_STEP: 922050 | > loss_text_ce: 0.02936576120555401 (0.03330290538953155) | > loss_mel_ce: 2.371793270111084 (nan) | > loss: 0.00952840968966484 (nan) | > current_lr: 5e-06 | > step_time: 0.1599 (0.13659581450271674) | > loader_time: 0.0049 (0.004687207988563204)  --> TIME: 2025-02-25 19:53:22 -- STEP: 35588/36938 -- GLOBAL_STEP: 922100 | > loss_text_ce: 0.03299557790160179 (0.03330364989020894) | > loss_mel_ce: 3.038975238800049 (nan) | > loss: 0.012190360575914383 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13659022085396183) | > loader_time: 0.0045 (0.004687304588949023)  --> TIME: 2025-02-25 19:53:31 -- STEP: 35638/36938 -- GLOBAL_STEP: 922150 | > loss_text_ce: 0.035789258778095245 (0.033303674416649616) | > loss_mel_ce: 3.03090500831604 (nan) | > loss: 0.012169422581791878 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1365971446766495) | > loader_time: 0.0045 (0.004687405059391018)  --> TIME: 2025-02-25 19:53:41 -- STEP: 35688/36938 -- GLOBAL_STEP: 922200 | > loss_text_ce: 0.025985317304730415 (0.033302668608840645) | > loss_mel_ce: 3.1480607986450195 (nan) | > loss: 0.012595421634614468 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.1365983830695987) | > loader_time: 0.0045 (0.004687440559698911)  --> TIME: 2025-02-25 19:53:50 -- STEP: 35738/36938 -- GLOBAL_STEP: 922250 | > loss_text_ce: 0.042007043957710266 (0.033303146573947125) | > loss_mel_ce: 2.8109357357025146 (nan) | > loss: 0.011321201920509338 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13659563119413118) | > loader_time: 0.0046 (0.0046874669877862134)  --> TIME: 2025-02-25 19:53:59 -- STEP: 35788/36938 -- GLOBAL_STEP: 922300 | > loss_text_ce: 0.023281769827008247 (0.033302336915406364) | > loss_mel_ce: 2.694150447845459 (nan) | > loss: 0.010783461853861809 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13659235353747975) | > loader_time: 0.0046 (0.004687488312240006)  --> TIME: 2025-02-25 19:54:08 -- STEP: 35838/36938 -- GLOBAL_STEP: 922350 | > loss_text_ce: 0.03989886865019798 (0.033302689316113354) | > loss_mel_ce: 2.877981662750244 (nan) | > loss: 0.011578891426324844 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13658764804205825) | > loader_time: 0.0048 (0.0046874947550307675)  --> TIME: 2025-02-25 19:54:18 -- STEP: 35888/36938 -- GLOBAL_STEP: 922400 | > loss_text_ce: 0.042458079755306244 (0.0333034705354678) | > loss_mel_ce: 3.0005462169647217 (nan) | > loss: 0.012075414881110191 (nan) | > current_lr: 5e-06 | > step_time: 0.1889 (0.13658894661596324) | > loader_time: 0.0049 (0.004687509610352121)  --> TIME: 2025-02-25 19:54:27 -- STEP: 35938/36938 -- GLOBAL_STEP: 922450 | > loss_text_ce: 0.03457584232091904 (0.033303120141172274) | > loss_mel_ce: 3.010791301727295 (nan) | > loss: 0.01208479143679142 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13659200731226764) | > loader_time: 0.0046 (0.004687574571981761)  --> TIME: 2025-02-25 19:54:37 -- STEP: 35988/36938 -- GLOBAL_STEP: 922500 | > loss_text_ce: 0.04054141789674759 (0.03330380694586245) | > loss_mel_ce: 1.1915404796600342 (nan) | > loss: 0.0048892139457166195 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13660166550307948) | > loader_time: 0.0053 (0.004687690749438172)  --> TIME: 2025-02-25 19:54:46 -- STEP: 36038/36938 -- GLOBAL_STEP: 922550 | > loss_text_ce: 0.03450135514140129 (0.033302978015720686) | > loss_mel_ce: 1.2182555198669434 (nan) | > loss: 0.004971257410943508 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13660247901203204) | > loader_time: 0.0045 (0.004687786161836725)  --> TIME: 2025-02-25 19:54:56 -- STEP: 36088/36938 -- GLOBAL_STEP: 922600 | > loss_text_ce: 0.03376907855272293 (0.03330193742971575) | > loss_mel_ce: 2.692620038986206 (nan) | > loss: 0.010819004848599434 (nan) | > current_lr: 5e-06 | > step_time: 0.1872 (0.13660672160348122) | > loader_time: 0.0049 (0.004687786128721287)  --> TIME: 2025-02-25 19:55:05 -- STEP: 36138/36938 -- GLOBAL_STEP: 922650 | > loss_text_ce: 0.023895421996712685 (0.033301992127639284) | > loss_mel_ce: 2.3545093536376953 (nan) | > loss: 0.009438115172088146 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13660174127606428) | > loader_time: 0.0046 (0.0046878569522899784)  --> TIME: 2025-02-25 19:55:14 -- STEP: 36188/36938 -- GLOBAL_STEP: 922700 | > loss_text_ce: 0.03930400684475899 (0.033302472658883336) | > loss_mel_ce: 3.153719186782837 (nan) | > loss: 0.0126707274466753 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.13660606823271046) | > loader_time: 0.0047 (0.004687978942789305)  --> TIME: 2025-02-25 19:55:24 -- STEP: 36238/36938 -- GLOBAL_STEP: 922750 | > loss_text_ce: 0.032277677208185196 (0.03330326249755735) | > loss_mel_ce: 2.2088100910186768 (nan) | > loss: 0.008893205784261227 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13660470117196993) | > loader_time: 0.0045 (0.00468803892941447)  --> TIME: 2025-02-25 19:55:33 -- STEP: 36288/36938 -- GLOBAL_STEP: 922800 | > loss_text_ce: 0.05879129096865654 (0.03330479593534522) | > loss_mel_ce: 2.6531496047973633 (nan) | > loss: 0.01076167169958353 (nan) | > current_lr: 5e-06 | > step_time: 0.1077 (0.13660217188889942) | > loader_time: 0.0047 (0.004688027805962482)  --> TIME: 2025-02-25 19:55:42 -- STEP: 36338/36938 -- GLOBAL_STEP: 922850 | > loss_text_ce: 0.022772405296564102 (0.033305063638178894) | > loss_mel_ce: 2.247582197189331 (nan) | > loss: 0.009009343571960926 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.1366019456097475) | > loader_time: 0.0045 (0.00468810236219779)  --> TIME: 2025-02-25 19:55:51 -- STEP: 36388/36938 -- GLOBAL_STEP: 922900 | > loss_text_ce: 0.03047754429280758 (0.03330580652674521) | > loss_mel_ce: 1.8762705326080322 (nan) | > loss: 0.007566460873931646 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.13659754023651977) | > loader_time: 0.0047 (0.004688365650502006)  --> TIME: 2025-02-25 19:56:01 -- STEP: 36438/36938 -- GLOBAL_STEP: 922950 | > loss_text_ce: 0.03386311233043671 (0.03330557743318761) | > loss_mel_ce: 3.0395593643188477 (nan) | > loss: 0.012196121737360954 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.13659691715732442) | > loader_time: 0.0049 (0.004688605020844387)  --> TIME: 2025-02-25 19:56:10 -- STEP: 36488/36938 -- GLOBAL_STEP: 923000 | > loss_text_ce: 0.033624909818172455 (0.033304526328891776) | > loss_mel_ce: 2.6887152194976807 (nan) | > loss: 0.010802937671542168 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13660150369095428) | > loader_time: 0.0047 (0.004688838795333879)  --> TIME: 2025-02-25 19:56:19 -- STEP: 36538/36938 -- GLOBAL_STEP: 923050 | > loss_text_ce: 0.03838946297764778 (0.03330437722805781) | > loss_mel_ce: 2.7925872802734375 (nan) | > loss: 0.011234035715460777 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.1366039283613994) | > loader_time: 0.0047 (0.004689055218916645)  --> TIME: 2025-02-25 19:56:29 -- STEP: 36588/36938 -- GLOBAL_STEP: 923100 | > loss_text_ce: 0.03581567853689194 (0.033306594045102215) | > loss_mel_ce: 2.6951465606689453 (nan) | > loss: 0.010837152600288391 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13660081857095466) | > loader_time: 0.0046 (0.004689014784244834)  --> TIME: 2025-02-25 19:56:38 -- STEP: 36638/36938 -- GLOBAL_STEP: 923150 | > loss_text_ce: 0.03500877693295479 (0.033305648713236255) | > loss_mel_ce: 2.995983362197876 (nan) | > loss: 0.012027746997773647 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13659313510641907) | > loader_time: 0.0046 (0.004688948788193375)  --> TIME: 2025-02-25 19:56:47 -- STEP: 36688/36938 -- GLOBAL_STEP: 923200 | > loss_text_ce: 0.04189389944076538 (0.033304628225411304) | > loss_mel_ce: 2.597520589828491 (nan) | > loss: 0.010473867878317833 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.1365971131728711) | > loader_time: 0.0045 (0.004688896612470162)  --> TIME: 2025-02-25 19:56:56 -- STEP: 36738/36938 -- GLOBAL_STEP: 923250 | > loss_text_ce: 0.039207469671964645 (0.03330582433880442) | > loss_mel_ce: 0.7567535638809204 (nan) | > loss: 0.0031585756223648787 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.1365989854712311) | > loader_time: 0.0047 (0.004689056733533432)  --> TIME: 2025-02-25 19:57:06 -- STEP: 36788/36938 -- GLOBAL_STEP: 923300 | > loss_text_ce: 0.03314802795648575 (0.03330332307505287) | > loss_mel_ce: 2.063197612762451 (nan) | > loss: 0.008318832144141197 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13659965152699968) | > loader_time: 0.0045 (0.0046891347732287035)  --> TIME: 2025-02-25 19:57:15 -- STEP: 36838/36938 -- GLOBAL_STEP: 923350 | > loss_text_ce: 0.03585365042090416 (0.03330387522770507) | > loss_mel_ce: 3.2191805839538574 (nan) | > loss: 0.012916803359985352 (nan) | > current_lr: 5e-06 | > step_time: 0.1783 (0.13659619660478142) | > loader_time: 0.0049 (0.004689111746615064)  --> TIME: 2025-02-25 19:57:24 -- STEP: 36888/36938 -- GLOBAL_STEP: 923400 | > loss_text_ce: 0.034163378179073334 (0.03330211505867253) | > loss_mel_ce: 2.4953417778015137 (nan) | > loss: 0.010037720203399658 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.1365989613716572) | > loader_time: 0.0046 (0.004689237360994642)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.020607905812782816 (-0.0002871138034480636) | > avg_loss_text_ce: 0.03240393346525948 (-0.0002765788115782486) | > avg_loss_mel_ce: 2.509476142059459 (-0.012042382566054943) | > avg_loss: 2.5418800670321615 (-0.012318972903906822) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_923450.pth  > EPOCH: 26/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-25 19:57:59)   --> TIME: 2025-02-25 19:58:00 -- STEP: 0/36938 -- GLOBAL_STEP: 923450 | > loss_text_ce: 0.023465244099497795 (0.023465244099497795) | > loss_mel_ce: 2.9319615364074707 (2.9319615364074707) | > loss: 0.011727884411811829 (0.011727884411811829) | > current_lr: 5e-06 | > step_time: 0.1647 (0.16469645500183105) | > loader_time: 0.3832 (0.3831641674041748)  --> TIME: 2025-02-25 19:58:09 -- STEP: 50/36938 -- GLOBAL_STEP: 923500 | > loss_text_ce: 0.03530637174844742 (0.03251473907381294) | > loss_mel_ce: 2.5629498958587646 (nan) | > loss: 0.010310541838407516 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13427437305450446) | > loader_time: 0.0052 (0.004884705543518065)  --> TIME: 2025-02-25 19:58:18 -- STEP: 100/36938 -- GLOBAL_STEP: 923550 | > loss_text_ce: 0.03751204535365105 (0.03262873638421295) | > loss_mel_ce: 2.908446788787842 (nan) | > loss: 0.011690313927829266 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13379201650619504) | > loader_time: 0.0046 (0.0047890472412109395)  --> TIME: 2025-02-25 19:58:27 -- STEP: 150/36938 -- GLOBAL_STEP: 923600 | > loss_text_ce: 0.03836555406451225 (0.033082396338383335) | > loss_mel_ce: 2.071256637573242 (nan) | > loss: 0.008371517062187195 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.1343266026178996) | > loader_time: 0.0046 (0.004778024355570476)  --> TIME: 2025-02-25 19:58:36 -- STEP: 200/36938 -- GLOBAL_STEP: 923650 | > loss_text_ce: 0.0273626446723938 (0.033144976347684846) | > loss_mel_ce: 2.1638338565826416 (nan) | > loss: 0.008695224300026894 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.13304666876792912) | > loader_time: 0.0055 (0.004763574600219727)  --> TIME: 2025-02-25 19:58:45 -- STEP: 250/36938 -- GLOBAL_STEP: 923700 | > loss_text_ce: 0.03237829729914665 (0.0330652543231845) | > loss_mel_ce: 2.5043251514434814 (nan) | > loss: 0.010066283866763115 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.1332571649551393) | > loader_time: 0.0053 (0.0047565755844116205)  --> TIME: 2025-02-25 19:58:54 -- STEP: 300/36938 -- GLOBAL_STEP: 923750 | > loss_text_ce: 0.034002188593149185 (0.03306280226136248) | > loss_mel_ce: 3.02679443359375 (nan) | > loss: 0.012146019376814365 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13303677956263235) | > loader_time: 0.0044 (0.0047278889020284)  --> TIME: 2025-02-25 19:59:04 -- STEP: 350/36938 -- GLOBAL_STEP: 923800 | > loss_text_ce: 0.032411765307188034 (0.03303651453661067) | > loss_mel_ce: 2.76129150390625 (nan) | > loss: 0.011086124926805496 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13366380350930354) | > loader_time: 0.0048 (0.0047117158344813755)  --> TIME: 2025-02-25 19:59:13 -- STEP: 400/36938 -- GLOBAL_STEP: 923850 | > loss_text_ce: 0.020762480795383453 (0.03317419638857245) | > loss_mel_ce: 1.4651715755462646 (nan) | > loss: 0.005896564107388258 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.1341913437843323) | > loader_time: 0.0045 (0.004697670936584477)  --> TIME: 2025-02-25 19:59:22 -- STEP: 450/36938 -- GLOBAL_STEP: 923900 | > loss_text_ce: 0.030379170551896095 (0.033226445242762574) | > loss_mel_ce: 3.280107021331787 (nan) | > loss: 0.01313685066998005 (nan) | > current_lr: 5e-06 | > step_time: 0.1699 (0.13399361557430692) | > loader_time: 0.0055 (0.0046951617134942)  --> TIME: 2025-02-25 19:59:31 -- STEP: 500/36938 -- GLOBAL_STEP: 923950 | > loss_text_ce: 0.03164537250995636 (0.033186564885079844) | > loss_mel_ce: 1.0609537363052368 (nan) | > loss: 0.004335711244493723 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13410878801345819) | > loader_time: 0.0053 (0.004706660747528081)  --> TIME: 2025-02-25 19:59:41 -- STEP: 550/36938 -- GLOBAL_STEP: 924000 | > loss_text_ce: 0.03770267963409424 (0.03311235463754696) | > loss_mel_ce: 2.9927611351013184 (nan) | > loss: 0.012025650590658188 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13441486748782067) | > loader_time: 0.0047 (0.004709767861799764)  --> TIME: 2025-02-25 19:59:50 -- STEP: 600/36938 -- GLOBAL_STEP: 924050 | > loss_text_ce: 0.02780175395309925 (0.03308300929764906) | > loss_mel_ce: 3.2171692848205566 (nan) | > loss: 0.012876870110630989 (nan) | > current_lr: 5e-06 | > step_time: 0.1567 (0.1343636333942412) | > loader_time: 0.0046 (0.004700921773910528)  --> TIME: 2025-02-25 19:59:59 -- STEP: 650/36938 -- GLOBAL_STEP: 924100 | > loss_text_ce: 0.025914020836353302 (0.0331103881276571) | > loss_mel_ce: 2.162297248840332 (nan) | > loss: 0.008683377876877785 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13483150848975534) | > loader_time: 0.0046 (0.0046984694554255595)  --> TIME: 2025-02-25 20:00:08 -- STEP: 700/36938 -- GLOBAL_STEP: 924150 | > loss_text_ce: 0.034084923565387726 (0.03303605061556613) | > loss_mel_ce: 2.568242073059082 (nan) | > loss: 0.010326695628464222 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.1346771802221024) | > loader_time: 0.0048 (0.004693361691066201)  --> TIME: 2025-02-25 20:00:18 -- STEP: 750/36938 -- GLOBAL_STEP: 924200 | > loss_text_ce: 0.028757767751812935 (0.03299466851105294) | > loss_mel_ce: 2.6478707790374756 (nan) | > loss: 0.010621543042361736 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13462545967102035) | > loader_time: 0.0047 (0.004687488238016766)  --> TIME: 2025-02-25 20:00:27 -- STEP: 800/36938 -- GLOBAL_STEP: 924250 | > loss_text_ce: 0.04336860403418541 (0.033034174798522144) | > loss_mel_ce: 0.10980224609375 (nan) | > loss: 0.0006078208680264652 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13475364059209805) | > loader_time: 0.0045 (0.0046881809830665614)  --> TIME: 2025-02-25 20:00:36 -- STEP: 850/36938 -- GLOBAL_STEP: 924300 | > loss_text_ce: 0.03469160199165344 (0.032977113758816436) | > loss_mel_ce: 2.7099897861480713 (nan) | > loss: 0.010891593061387539 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.13458775239832235) | > loader_time: 0.0046 (0.004695426997016463)  --> TIME: 2025-02-25 20:00:45 -- STEP: 900/36938 -- GLOBAL_STEP: 924350 | > loss_text_ce: 0.03656793013215065 (0.033062230532781955) | > loss_mel_ce: 3.265943765640259 (nan) | > loss: 0.013105206191539764 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.1345976291762455) | > loader_time: 0.0046 (0.004693973064422612)  --> TIME: 2025-02-25 20:00:55 -- STEP: 950/36938 -- GLOBAL_STEP: 924400 | > loss_text_ce: 0.02966924197971821 (0.03309289882841866) | > loss_mel_ce: 2.643430471420288 (nan) | > loss: 0.010607539676129818 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.134729234544854) | > loader_time: 0.0046 (0.004694822712948452)  --> TIME: 2025-02-25 20:01:04 -- STEP: 1000/36938 -- GLOBAL_STEP: 924450 | > loss_text_ce: 0.04500715062022209 (0.03308752174489203) | > loss_mel_ce: 2.9145314693450928 (nan) | > loss: 0.011744202114641666 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1347239694595333) | > loader_time: 0.0045 (0.004703548431396489)  --> TIME: 2025-02-25 20:01:13 -- STEP: 1050/36938 -- GLOBAL_STEP: 924500 | > loss_text_ce: 0.026224154978990555 (0.03309690432002149) | > loss_mel_ce: 2.858642578125 (nan) | > loss: 0.011447884142398834 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13471236728486521) | > loader_time: 0.0046 (0.00471177941276914)  --> TIME: 2025-02-25 20:01:23 -- STEP: 1100/36938 -- GLOBAL_STEP: 924550 | > loss_text_ce: 0.03294208273291588 (0.033110101908784044) | > loss_mel_ce: 0.0228271484375 (nan) | > loss: 0.00022130648721940815 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.1349566966837099) | > loader_time: 0.0046 (0.004713687029751867)  --> TIME: 2025-02-25 20:01:32 -- STEP: 1150/36938 -- GLOBAL_STEP: 924600 | > loss_text_ce: 0.03840405493974686 (0.033117445802883026) | > loss_mel_ce: 3.2108919620513916 (nan) | > loss: 0.012894032523036003 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13496785143147308) | > loader_time: 0.0048 (0.004708456578462022)  --> TIME: 2025-02-25 20:01:41 -- STEP: 1200/36938 -- GLOBAL_STEP: 924650 | > loss_text_ce: 0.026335030794143677 (0.03307109642308214) | > loss_mel_ce: 2.5755162239074707 (nan) | > loss: 0.010324806906282902 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13520024160544042) | > loader_time: 0.0045 (0.004706470767656964)  --> TIME: 2025-02-25 20:01:51 -- STEP: 1250/36938 -- GLOBAL_STEP: 924700 | > loss_text_ce: 0.03111170418560505 (0.033113681082427585) | > loss_mel_ce: 2.28094482421875 (nan) | > loss: 0.009174828417599201 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13533945350646948) | > loader_time: 0.0045 (0.004703680610656742)  --> TIME: 2025-02-25 20:02:00 -- STEP: 1300/36938 -- GLOBAL_STEP: 924750 | > loss_text_ce: 0.02002245932817459 (0.033089568986056014) | > loss_mel_ce: 2.788895845413208 (nan) | > loss: 0.011146501637995243 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13537290059603152) | > loader_time: 0.0047 (0.00470085529180674)  --> TIME: 2025-02-25 20:02:09 -- STEP: 1350/36938 -- GLOBAL_STEP: 924800 | > loss_text_ce: 0.031149208545684814 (0.03308198798447858) | > loss_mel_ce: 2.7945775985717773 (nan) | > loss: 0.011213202029466629 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13550146915294484) | > loader_time: 0.0045 (0.004699349050168642)  --> TIME: 2025-02-25 20:02:19 -- STEP: 1400/36938 -- GLOBAL_STEP: 924850 | > loss_text_ce: 0.02882532589137554 (0.03310937863641561) | > loss_mel_ce: 2.6504969596862793 (nan) | > loss: 0.010632231831550598 (nan) | > current_lr: 5e-06 | > step_time: 0.1667 (0.1354228404590059) | > loader_time: 0.0046 (0.00469571879931859)  --> TIME: 2025-02-25 20:02:28 -- STEP: 1450/36938 -- GLOBAL_STEP: 924900 | > loss_text_ce: 0.03197476267814636 (0.033129316719313226) | > loss_mel_ce: 3.4257900714874268 (nan) | > loss: 0.013721290044486523 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.13539149087050856) | > loader_time: 0.0056 (0.004695239889210671)  --> TIME: 2025-02-25 20:02:37 -- STEP: 1500/36938 -- GLOBAL_STEP: 924950 | > loss_text_ce: 0.04025948420166969 (0.03306936791601294) | > loss_mel_ce: 2.2639989852905273 (nan) | > loss: 0.009143883362412453 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.1355471490224202) | > loader_time: 0.0046 (0.00469697443644206)  --> TIME: 2025-02-25 20:02:47 -- STEP: 1550/36938 -- GLOBAL_STEP: 925000 | > loss_text_ce: 0.03752367943525314 (0.03302418182153386) | > loss_mel_ce: 2.4579367637634277 (nan) | > loss: 0.009902621619403362 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13560834900025398) | > loader_time: 0.0046 (0.004699755791694889)  --> TIME: 2025-02-25 20:02:56 -- STEP: 1600/36938 -- GLOBAL_STEP: 925050 | > loss_text_ce: 0.040009208023548126 (0.033004042254178904) | > loss_mel_ce: 2.95190691947937 (nan) | > loss: 0.01187268365174532 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13550584644079205) | > loader_time: 0.0048 (0.0046969376504421255)  --> TIME: 2025-02-25 20:03:05 -- STEP: 1650/36938 -- GLOBAL_STEP: 925100 | > loss_text_ce: 0.03009539097547531 (0.032998519859869506) | > loss_mel_ce: 2.939694881439209 (nan) | > loss: 0.01178488228470087 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.1355864485827359) | > loader_time: 0.0047 (0.004694274989041418)  --> TIME: 2025-02-25 20:03:14 -- STEP: 1700/36938 -- GLOBAL_STEP: 925150 | > loss_text_ce: 0.03809002414345741 (0.03296562618959492) | > loss_mel_ce: 2.592930793762207 (nan) | > loss: 0.01044055912643671 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13563466450747302) | > loader_time: 0.0045 (0.0046919453845304625)  --> TIME: 2025-02-25 20:03:24 -- STEP: 1750/36938 -- GLOBAL_STEP: 925200 | > loss_text_ce: 0.03530265390872955 (0.03297324236695271) | > loss_mel_ce: 2.911125898361206 (nan) | > loss: 0.011692177504301071 (nan) | > current_lr: 5e-06 | > step_time: 0.165 (0.1356361511775424) | > loader_time: 0.0046 (0.004692601612636022)  --> TIME: 2025-02-25 20:03:33 -- STEP: 1800/36938 -- GLOBAL_STEP: 925250 | > loss_text_ce: 0.0352071151137352 (0.032956349873501545) | > loss_mel_ce: 3.0905420780181885 (nan) | > loss: 0.012403766624629498 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.1356827996836767) | > loader_time: 0.0047 (0.00469105985429552)  --> TIME: 2025-02-25 20:03:42 -- STEP: 1850/36938 -- GLOBAL_STEP: 925300 | > loss_text_ce: 0.03755425289273262 (0.032978256309556114) | > loss_mel_ce: 2.9721853733062744 (nan) | > loss: 0.011943412013351917 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13564082223015844) | > loader_time: 0.0046 (0.004688867362769875)  --> TIME: 2025-02-25 20:03:51 -- STEP: 1900/36938 -- GLOBAL_STEP: 925350 | > loss_text_ce: 0.033608946949243546 (0.032952149167264855) | > loss_mel_ce: 2.495853900909424 (nan) | > loss: 0.010037551634013653 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13553424132497674) | > loader_time: 0.0045 (0.004687299351943168)  --> TIME: 2025-02-25 20:04:00 -- STEP: 1950/36938 -- GLOBAL_STEP: 925400 | > loss_text_ce: 0.026075730100274086 (0.03293842721443923) | > loss_mel_ce: 2.326632261276245 (nan) | > loss: 0.009336143732070923 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13548318924047997) | > loader_time: 0.0048 (0.004687233215723283)  --> TIME: 2025-02-25 20:04:09 -- STEP: 2000/36938 -- GLOBAL_STEP: 925450 | > loss_text_ce: 0.03176911175251007 (0.03290998274460448) | > loss_mel_ce: 0.0592244453728199 (nan) | > loss: 0.000361085549229756 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.1355254521369933) | > loader_time: 0.0045 (0.004685778379440309)  --> TIME: 2025-02-25 20:04:19 -- STEP: 2050/36938 -- GLOBAL_STEP: 925500 | > loss_text_ce: 0.028278380632400513 (0.032929569797181625) | > loss_mel_ce: 2.4824867248535156 (nan) | > loss: 0.009963354095816612 (nan) | > current_lr: 5e-06 | > step_time: 0.1501 (0.13550877489694724) | > loader_time: 0.0048 (0.004686391760663289)  --> TIME: 2025-02-25 20:04:28 -- STEP: 2100/36938 -- GLOBAL_STEP: 925550 | > loss_text_ce: 0.023472830653190613 (0.032905105128884435) | > loss_mel_ce: 2.389374017715454 (nan) | > loss: 0.009574789553880692 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.13557700872421255) | > loader_time: 0.0055 (0.004685101736159551)  --> TIME: 2025-02-25 20:04:37 -- STEP: 2150/36938 -- GLOBAL_STEP: 925600 | > loss_text_ce: 0.032521508634090424 (0.032895345532790284) | > loss_mel_ce: 2.308138847351074 (nan) | > loss: 0.009288335219025612 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.135459259498951) | > loader_time: 0.0045 (0.004684257729108944)  --> TIME: 2025-02-25 20:04:46 -- STEP: 2200/36938 -- GLOBAL_STEP: 925650 | > loss_text_ce: 0.03940792754292488 (0.032872419632463394) | > loss_mel_ce: 1.67578125 (nan) | > loss: 0.006806307006627321 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1354641196944496) | > loader_time: 0.0045 (0.004684680808674206)  --> TIME: 2025-02-25 20:04:56 -- STEP: 2250/36938 -- GLOBAL_STEP: 925700 | > loss_text_ce: 0.027992457151412964 (0.03289377540101619) | > loss_mel_ce: 1.6605478525161743 (nan) | > loss: 0.006700557190924883 (nan) | > current_lr: 5e-06 | > step_time: 0.1648 (0.13549808661142976) | > loader_time: 0.0047 (0.0046831570731269015)  --> TIME: 2025-02-25 20:05:05 -- STEP: 2300/36938 -- GLOBAL_STEP: 925750 | > loss_text_ce: 0.03261611983180046 (0.03287738274863893) | > loss_mel_ce: 2.02972412109375 (nan) | > loss: 0.008183890022337437 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13548366847245577) | > loader_time: 0.0045 (0.0046815719811812715)  --> TIME: 2025-02-25 20:05:15 -- STEP: 2350/36938 -- GLOBAL_STEP: 925800 | > loss_text_ce: 0.0323844738304615 (0.03291538514672451) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13559995225135316) | > loader_time: 0.0046 (0.004680500131972294)  --> TIME: 2025-02-25 20:05:24 -- STEP: 2400/36938 -- GLOBAL_STEP: 925850 | > loss_text_ce: 0.03593676909804344 (0.032924723611989455) | > loss_mel_ce: 2.876861572265625 (nan) | > loss: 0.01155872456729412 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13557640373706817) | > loader_time: 0.0046 (0.0046793122092882835)  --> TIME: 2025-02-25 20:05:33 -- STEP: 2450/36938 -- GLOBAL_STEP: 925900 | > loss_text_ce: 0.04207538068294525 (0.03294266275225254) | > loss_mel_ce: 1.6289397478103638 (nan) | > loss: 0.006631013005971909 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13570506796544915) | > loader_time: 0.0045 (0.004679610193992153)  --> TIME: 2025-02-25 20:05:43 -- STEP: 2500/36938 -- GLOBAL_STEP: 925950 | > loss_text_ce: 0.013966171070933342 (0.032931043615192265) | > loss_mel_ce: 2.8072102069854736 (nan) | > loss: 0.01119514461606741 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13571848659515384) | > loader_time: 0.0047 (0.004679391384124764)  --> TIME: 2025-02-25 20:05:52 -- STEP: 2550/36938 -- GLOBAL_STEP: 926000 | > loss_text_ce: 0.03295641019940376 (0.032954311683481896) | > loss_mel_ce: 2.691427230834961 (nan) | > loss: 0.010811046697199345 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13574678486468744) | > loader_time: 0.0045 (0.004679601519715557)  --> TIME: 2025-02-25 20:06:02 -- STEP: 2600/36938 -- GLOBAL_STEP: 926050 | > loss_text_ce: 0.03424910455942154 (0.03296686109656903) | > loss_mel_ce: 1.3155028820037842 (nan) | > loss: 0.005356159061193466 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1358868972154765) | > loader_time: 0.0045 (0.0046794985807859036)  --> TIME: 2025-02-25 20:06:11 -- STEP: 2650/36938 -- GLOBAL_STEP: 926100 | > loss_text_ce: 0.03375415876507759 (0.03298939144906573) | > loss_mel_ce: 2.8419883251190186 (nan) | > loss: 0.011411677114665508 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.1358620826253352) | > loader_time: 0.0046 (0.004680493822637602)  --> TIME: 2025-02-25 20:06:20 -- STEP: 2700/36938 -- GLOBAL_STEP: 926150 | > loss_text_ce: 0.03590095415711403 (0.032982153582766134) | > loss_mel_ce: 2.360610246658325 (nan) | > loss: 0.009509965777397156 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.1359142695532906) | > loader_time: 0.0047 (0.004679980101408789)  --> TIME: 2025-02-25 20:06:30 -- STEP: 2750/36938 -- GLOBAL_STEP: 926200 | > loss_text_ce: 0.0317852720618248 (0.03298383389447225) | > loss_mel_ce: 2.832446336746216 (nan) | > loss: 0.011365999467670918 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.1359128116260878) | > loader_time: 0.0045 (0.004679311405528682)  --> TIME: 2025-02-25 20:06:39 -- STEP: 2800/36938 -- GLOBAL_STEP: 926250 | > loss_text_ce: 0.030885031446814537 (0.03298249275523373) | > loss_mel_ce: 0.7232005000114441 (nan) | > loss: 0.0029924032278358936 (nan) | > current_lr: 5e-06 | > step_time: 0.1766 (0.13591567218303696) | > loader_time: 0.0051 (0.004679105452128827)  --> TIME: 2025-02-25 20:06:48 -- STEP: 2850/36938 -- GLOBAL_STEP: 926300 | > loss_text_ce: 0.031163891777396202 (0.0329793985942869) | > loss_mel_ce: 3.1697869300842285 (nan) | > loss: 0.012702186591923237 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.1358710079862361) | > loader_time: 0.0046 (0.004677633068017801)  --> TIME: 2025-02-25 20:06:57 -- STEP: 2900/36938 -- GLOBAL_STEP: 926350 | > loss_text_ce: 0.03080342896282673 (0.032963838789868985) | > loss_mel_ce: 3.0022194385528564 (nan) | > loss: 0.012035805732011795 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13590283106113304) | > loader_time: 0.0046 (0.004676594734191899)  --> TIME: 2025-02-25 20:07:07 -- STEP: 2950/36938 -- GLOBAL_STEP: 926400 | > loss_text_ce: 0.02189013734459877 (0.03293406920539123) | > loss_mel_ce: 2.553342342376709 (nan) | > loss: 0.010219177231192589 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13595317929478004) | > loader_time: 0.0047 (0.004675720505795236)  --> TIME: 2025-02-25 20:07:16 -- STEP: 3000/36938 -- GLOBAL_STEP: 926450 | > loss_text_ce: 0.02941731922328472 (0.032940661029269305) | > loss_mel_ce: 3.2776999473571777 (nan) | > loss: 0.013123481534421444 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13597128844261155) | > loader_time: 0.0046 (0.004675057093302412)  --> TIME: 2025-02-25 20:07:26 -- STEP: 3050/36938 -- GLOBAL_STEP: 926500 | > loss_text_ce: 0.03207802399992943 (0.032947080743606) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13597482657823387) | > loader_time: 0.0046 (0.004673630526808443)  --> TIME: 2025-02-25 20:07:35 -- STEP: 3100/36938 -- GLOBAL_STEP: 926550 | > loss_text_ce: 0.05251024663448334 (0.03294959992590931) | > loss_mel_ce: 2.5531351566314697 (nan) | > loss: 0.01033986359834671 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13602524219020692) | > loader_time: 0.0046 (0.004672232135649654)  --> TIME: 2025-02-25 20:07:44 -- STEP: 3150/36938 -- GLOBAL_STEP: 926600 | > loss_text_ce: 0.037116821855306625 (0.032942312611710454) | > loss_mel_ce: 2.7450339794158936 (nan) | > loss: 0.011040281504392624 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13597984957316533) | > loader_time: 0.0045 (0.004670955491444426)  --> TIME: 2025-02-25 20:07:53 -- STEP: 3200/36938 -- GLOBAL_STEP: 926650 | > loss_text_ce: 0.027866527438163757 (0.03294389560702255) | > loss_mel_ce: 2.749638319015503 (nan) | > loss: 0.01102184597402811 (nan) | > current_lr: 5e-06 | > step_time: 0.1086 (0.1359071747213599) | > loader_time: 0.0044 (0.004669905081391343)  --> TIME: 2025-02-25 20:08:03 -- STEP: 3250/36938 -- GLOBAL_STEP: 926700 | > loss_text_ce: 0.02777707204222679 (0.03294368867461495) | > loss_mel_ce: 2.858090877532959 (nan) | > loss: 0.011451857164502144 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13602473787160993) | > loader_time: 0.0046 (0.004670064632709216)  --> TIME: 2025-02-25 20:08:12 -- STEP: 3300/36938 -- GLOBAL_STEP: 926750 | > loss_text_ce: 0.011322042904794216 (0.032951826275693194) | > loss_mel_ce: 1.8479822874069214 (nan) | > loss: 0.007378192152827978 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13602609222585485) | > loader_time: 0.0045 (0.0046692828698591815)  --> TIME: 2025-02-25 20:08:22 -- STEP: 3350/36938 -- GLOBAL_STEP: 926800 | > loss_text_ce: 0.035923104733228683 (0.03296564130223729) | > loss_mel_ce: 3.1909303665161133 (nan) | > loss: 0.012804974801838398 (nan) | > current_lr: 5e-06 | > step_time: 0.1766 (0.1360770061834532) | > loader_time: 0.0048 (0.0046692263902123315)  --> TIME: 2025-02-25 20:08:31 -- STEP: 3400/36938 -- GLOBAL_STEP: 926850 | > loss_text_ce: 0.033600784838199615 (0.03296460665461114) | > loss_mel_ce: 2.607536554336548 (nan) | > loss: 0.010480704717338085 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13607198462766723) | > loader_time: 0.0055 (0.004668787016588098)  --> TIME: 2025-02-25 20:08:40 -- STEP: 3450/36938 -- GLOBAL_STEP: 926900 | > loss_text_ce: 0.022320831194519997 (0.03295948004350063) | > loss_mel_ce: 2.1265625953674316 (nan) | > loss: 0.008527315221726894 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13605614268261415) | > loader_time: 0.0051 (0.004668889114822165)  --> TIME: 2025-02-25 20:08:50 -- STEP: 3500/36938 -- GLOBAL_STEP: 926950 | > loss_text_ce: 0.036555204540491104 (0.03296376145870557) | > loss_mel_ce: 0.7503505349159241 (nan) | > loss: 0.003122642170637846 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13604172093527608) | > loader_time: 0.0046 (0.004668486663273401)  --> TIME: 2025-02-25 20:08:59 -- STEP: 3550/36938 -- GLOBAL_STEP: 927000 | > loss_text_ce: 0.02955179661512375 (0.03295936916889224) | > loss_mel_ce: 2.9470863342285156 (nan) | > loss: 0.011812056414783001 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13603657594868773) | > loader_time: 0.0046 (0.004668549416770396)  --> TIME: 2025-02-25 20:09:08 -- STEP: 3600/36938 -- GLOBAL_STEP: 927050 | > loss_text_ce: 0.025234397500753403 (0.032958818761269625) | > loss_mel_ce: 3.0289926528930664 (nan) | > loss: 0.012119949795305729 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13611472917927572) | > loader_time: 0.0046 (0.0046682441896862435)  --> TIME: 2025-02-25 20:09:18 -- STEP: 3650/36938 -- GLOBAL_STEP: 927100 | > loss_text_ce: 0.022670282050967216 (0.0329489897245107) | > loss_mel_ce: 3.48250675201416 (nan) | > loss: 0.013909433037042618 (nan) | > current_lr: 5e-06 | > step_time: 0.1074 (0.1361829262563621) | > loader_time: 0.0044 (0.004668216835962581)  --> TIME: 2025-02-25 20:09:27 -- STEP: 3700/36938 -- GLOBAL_STEP: 927150 | > loss_text_ce: 0.020475296303629875 (0.03292206585432421) | > loss_mel_ce: 1.9634640216827393 (nan) | > loss: 0.007872775197029114 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13610847879100416) | > loader_time: 0.0053 (0.004669293584050357)  --> TIME: 2025-02-25 20:09:37 -- STEP: 3750/36938 -- GLOBAL_STEP: 927200 | > loss_text_ce: 0.04014689475297928 (0.032934167769799544) | > loss_mel_ce: 3.0155508518218994 (nan) | > loss: 0.01212578546255827 (nan) | > current_lr: 5e-06 | > step_time: 0.161 (0.13622543150583827) | > loader_time: 0.0047 (0.004669213867187498)  --> TIME: 2025-02-25 20:09:46 -- STEP: 3800/36938 -- GLOBAL_STEP: 927250 | > loss_text_ce: 0.04008530452847481 (0.032944133367977545) | > loss_mel_ce: 2.5815844535827637 (nan) | > loss: 0.010403452441096306 (nan) | > current_lr: 5e-06 | > step_time: 0.169 (0.1361710592320083) | > loader_time: 0.0048 (0.004669508808537531)  --> TIME: 2025-02-25 20:09:55 -- STEP: 3850/36938 -- GLOBAL_STEP: 927300 | > loss_text_ce: 0.03436940908432007 (0.032939080694182214) | > loss_mel_ce: 2.9290847778320312 (nan) | > loss: 0.011759739369153976 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13608309677668903) | > loader_time: 0.0048 (0.00466811266812411)  --> TIME: 2025-02-25 20:10:04 -- STEP: 3900/36938 -- GLOBAL_STEP: 927350 | > loss_text_ce: 0.031816139817237854 (0.03294777726587372) | > loss_mel_ce: 2.7774720191955566 (nan) | > loss: 0.011147970333695412 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13610374915294082) | > loader_time: 0.0045 (0.004667245546976723)  --> TIME: 2025-02-25 20:10:13 -- STEP: 3950/36938 -- GLOBAL_STEP: 927400 | > loss_text_ce: 0.03176846727728844 (0.03293902459283217) | > loss_mel_ce: 2.8231945037841797 (nan) | > loss: 0.011329218745231628 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13608732096756487) | > loader_time: 0.0045 (0.004666331689569013)  --> TIME: 2025-02-25 20:10:22 -- STEP: 4000/36938 -- GLOBAL_STEP: 927450 | > loss_text_ce: 0.03201815113425255 (0.032942695085657635) | > loss_mel_ce: 2.73563289642334 (nan) | > loss: 0.010982743464410305 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13602523487806245) | > loader_time: 0.0046 (0.004665768027305601)  --> TIME: 2025-02-25 20:10:32 -- STEP: 4050/36938 -- GLOBAL_STEP: 927500 | > loss_text_ce: 0.030846495181322098 (0.03293212529777732) | > loss_mel_ce: 0.6678548455238342 (nan) | > loss: 0.002772624371573329 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13603507542315751) | > loader_time: 0.0046 (0.004665668334489986)  --> TIME: 2025-02-25 20:10:41 -- STEP: 4100/36938 -- GLOBAL_STEP: 927550 | > loss_text_ce: 0.03651633486151695 (0.032934585572106684) | > loss_mel_ce: 2.986931085586548 (nan) | > loss: 0.011997808702290058 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13604996960337498) | > loader_time: 0.0046 (0.004665362427874307)  --> TIME: 2025-02-25 20:10:50 -- STEP: 4150/36938 -- GLOBAL_STEP: 927600 | > loss_text_ce: 0.024035636335611343 (0.03295325464378267) | > loss_mel_ce: 1.8558523654937744 (nan) | > loss: 0.007459873799234629 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1360622590421187) | > loader_time: 0.0046 (0.004664762048836212)  --> TIME: 2025-02-25 20:11:00 -- STEP: 4200/36938 -- GLOBAL_STEP: 927650 | > loss_text_ce: 0.03795527666807175 (0.032963114729417084) | > loss_mel_ce: 2.8882389068603516 (nan) | > loss: 0.011611882597208023 (nan) | > current_lr: 5e-06 | > step_time: 0.1624 (0.13606312706356888) | > loader_time: 0.0046 (0.004665013267880392)  --> TIME: 2025-02-25 20:11:09 -- STEP: 4250/36938 -- GLOBAL_STEP: 927700 | > loss_text_ce: 0.03885766118764877 (0.03296471033026177) | > loss_mel_ce: 2.327432632446289 (nan) | > loss: 0.00939004123210907 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13609243370504873) | > loader_time: 0.0046 (0.004664974549237419)  --> TIME: 2025-02-25 20:11:18 -- STEP: 4300/36938 -- GLOBAL_STEP: 927750 | > loss_text_ce: 0.030815428122878075 (0.032971886386444965) | > loss_mel_ce: 1.833014965057373 (nan) | > loss: 0.007396153174340725 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13611755831296982) | > loader_time: 0.0047 (0.004665433861488519)  --> TIME: 2025-02-25 20:11:28 -- STEP: 4350/36938 -- GLOBAL_STEP: 927800 | > loss_text_ce: 0.03246254473924637 (0.032983432580427734) | > loss_mel_ce: 2.5986649990081787 (nan) | > loss: 0.01044098287820816 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13621020459580624) | > loader_time: 0.0046 (0.004665186980675004)  --> TIME: 2025-02-25 20:11:37 -- STEP: 4400/36938 -- GLOBAL_STEP: 927850 | > loss_text_ce: 0.027413489297032356 (0.03299898687623091) | > loss_mel_ce: 2.4223484992980957 (nan) | > loss: 0.00972127914428711 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13615440108559282) | > loader_time: 0.0046 (0.0046652823686599695)  --> TIME: 2025-02-25 20:11:47 -- STEP: 4450/36938 -- GLOBAL_STEP: 927900 | > loss_text_ce: 0.031259048730134964 (0.03300269908449611) | > loss_mel_ce: 1.4565038681030273 (nan) | > loss: 0.005903821438550949 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13619310148646255) | > loader_time: 0.0045 (0.004665262565184169)  --> TIME: 2025-02-25 20:11:56 -- STEP: 4500/36938 -- GLOBAL_STEP: 927950 | > loss_text_ce: 0.039933942258358 (0.032997444789028725) | > loss_mel_ce: 2.2758665084838867 (nan) | > loss: 0.009189684875309467 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.1361754023763862) | > loader_time: 0.0045 (0.004665923277537019)  --> TIME: 2025-02-25 20:12:05 -- STEP: 4550/36938 -- GLOBAL_STEP: 928000 | > loss_text_ce: 0.03349549323320389 (0.032991627349548844) | > loss_mel_ce: 1.9811623096466064 (nan) | > loss: 0.007994674146175385 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13619166033608512) | > loader_time: 0.0045 (0.004666996788192571)  --> TIME: 2025-02-25 20:12:15 -- STEP: 4600/36938 -- GLOBAL_STEP: 928050 | > loss_text_ce: 0.03255347162485123 (0.03299037538185381) | > loss_mel_ce: 1.1760362386703491 (nan) | > loss: 0.004795990884304047 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.1361918141012601) | > loader_time: 0.0046 (0.0046671856486278935)  --> TIME: 2025-02-25 20:12:24 -- STEP: 4650/36938 -- GLOBAL_STEP: 928100 | > loss_text_ce: 0.03827179968357086 (0.03298085451907199) | > loss_mel_ce: 2.6729044914245605 (nan) | > loss: 0.010758637450635433 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.1361598286064717) | > loader_time: 0.0047 (0.004668385597967323)  --> TIME: 2025-02-25 20:12:33 -- STEP: 4700/36938 -- GLOBAL_STEP: 928150 | > loss_text_ce: 0.028443297371268272 (0.032982097912302256) | > loss_mel_ce: 2.1582465171813965 (nan) | > loss: 0.008677341043949127 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1362335137610735) | > loader_time: 0.0045 (0.004668695774484182)  --> TIME: 2025-02-25 20:12:43 -- STEP: 4750/36938 -- GLOBAL_STEP: 928200 | > loss_text_ce: 0.03594038262963295 (0.03298865869151124) | > loss_mel_ce: 3.2861993312835693 (nan) | > loss: 0.013183094561100006 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13623938987129552) | > loader_time: 0.005 (0.004668326729222343)  --> TIME: 2025-02-25 20:12:52 -- STEP: 4800/36938 -- GLOBAL_STEP: 928250 | > loss_text_ce: 0.03227444738149643 (0.03300189518466739) | > loss_mel_ce: 2.8395793437957764 (nan) | > loss: 0.011396246030926704 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13623029912511447) | > loader_time: 0.0046 (0.004668143689632411)  --> TIME: 2025-02-25 20:13:02 -- STEP: 4850/36938 -- GLOBAL_STEP: 928300 | > loss_text_ce: 0.027233678847551346 (0.032985811777336) | > loss_mel_ce: 2.674022912979126 (nan) | > loss: 0.010719272308051586 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13630430560750997) | > loader_time: 0.0045 (0.004667783668360758)  --> TIME: 2025-02-25 20:13:11 -- STEP: 4900/36938 -- GLOBAL_STEP: 928350 | > loss_text_ce: 0.02952231839299202 (0.032989902500899344) | > loss_mel_ce: 2.8847928047180176 (nan) | > loss: 0.01156474370509386 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13632550220100192) | > loader_time: 0.0045 (0.0046694971590626015)  --> TIME: 2025-02-25 20:13:21 -- STEP: 4950/36938 -- GLOBAL_STEP: 928400 | > loss_text_ce: 0.036134686321020126 (0.03300265898307152) | > loss_mel_ce: 1.897581696510315 (nan) | > loss: 0.007673478219658136 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.1363147514998305) | > loader_time: 0.0048 (0.004669729194255792)  --> TIME: 2025-02-25 20:13:30 -- STEP: 5000/36938 -- GLOBAL_STEP: 928450 | > loss_text_ce: 0.03796999901533127 (0.03300223322883235) | > loss_mel_ce: 2.6577377319335938 (nan) | > loss: 0.010697253979742527 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1363302379608149) | > loader_time: 0.0045 (0.004670057010650636)  --> TIME: 2025-02-25 20:13:39 -- STEP: 5050/36938 -- GLOBAL_STEP: 928500 | > loss_text_ce: 0.03565570339560509 (0.032988073013280154) | > loss_mel_ce: 2.221791982650757 (nan) | > loss: 0.008958126418292522 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13632644341723685) | > loader_time: 0.0047 (0.004670684125163767)  --> TIME: 2025-02-25 20:13:48 -- STEP: 5100/36938 -- GLOBAL_STEP: 928550 | > loss_text_ce: 0.030800633132457733 (0.032979677837049685) | > loss_mel_ce: 3.1631171703338623 (nan) | > loss: 0.012674277648329735 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.1363201430732121) | > loader_time: 0.0047 (0.004670681906681431)  --> TIME: 2025-02-25 20:13:58 -- STEP: 5150/36938 -- GLOBAL_STEP: 928600 | > loss_text_ce: 0.026776935905218124 (0.03298442306557601) | > loss_mel_ce: 0.0117034912109375 (nan) | > loss: 0.00015270011499524117 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13633627160081513) | > loader_time: 0.0044 (0.004670616538779245)  --> TIME: 2025-02-25 20:14:07 -- STEP: 5200/36938 -- GLOBAL_STEP: 928650 | > loss_text_ce: 0.0283455029129982 (0.03298453981414993) | > loss_mel_ce: 2.4772791862487793 (nan) | > loss: 0.009942956268787384 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.1363883062509382) | > loader_time: 0.0049 (0.004671650024560774)  --> TIME: 2025-02-25 20:14:17 -- STEP: 5250/36938 -- GLOBAL_STEP: 928700 | > loss_text_ce: 0.047478724271059036 (0.03299122153666029) | > loss_mel_ce: 0.6392733454704285 (nan) | > loss: 0.002725206781178713 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13639535840352307) | > loader_time: 0.0046 (0.004671547708057216)  --> TIME: 2025-02-25 20:14:26 -- STEP: 5300/36938 -- GLOBAL_STEP: 928750 | > loss_text_ce: 0.03291918337345123 (0.03299032296581207) | > loss_mel_ce: 2.802008628845215 (nan) | > loss: 0.011249714531004429 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13637617124701384) | > loader_time: 0.0046 (0.004670801702535369)  --> TIME: 2025-02-25 20:14:35 -- STEP: 5350/36938 -- GLOBAL_STEP: 928800 | > loss_text_ce: 0.030411802232265472 (0.0329721993119149) | > loss_mel_ce: 3.175180196762085 (nan) | > loss: 0.012720603495836258 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13636126794547604) | > loader_time: 0.0046 (0.004671629878962138)  --> TIME: 2025-02-25 20:14:45 -- STEP: 5400/36938 -- GLOBAL_STEP: 928850 | > loss_text_ce: 0.03819742798805237 (0.03297642074387361) | > loss_mel_ce: 2.846978187561035 (nan) | > loss: 0.011449110694229603 (nan) | > current_lr: 5e-06 | > step_time: 0.1701 (0.13640977930139542) | > loader_time: 0.0048 (0.0046717275513542935)  --> TIME: 2025-02-25 20:14:54 -- STEP: 5450/36938 -- GLOBAL_STEP: 928900 | > loss_text_ce: 0.029377125203609467 (0.032981303973687114) | > loss_mel_ce: 1.9608677625656128 (nan) | > loss: 0.007897797971963882 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13639393412738704) | > loader_time: 0.0046 (0.004671516593443138)  --> TIME: 2025-02-25 20:15:03 -- STEP: 5500/36938 -- GLOBAL_STEP: 928950 | > loss_text_ce: 0.03189423307776451 (0.032976430066606964) | > loss_mel_ce: 2.5225605964660645 (nan) | > loss: 0.010136726312339306 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13637665176391534) | > loader_time: 0.0046 (0.00467154997045343)  --> TIME: 2025-02-25 20:15:13 -- STEP: 5550/36938 -- GLOBAL_STEP: 929000 | > loss_text_ce: 0.024907175451517105 (0.03297541930533179) | > loss_mel_ce: 2.7994019985198975 (nan) | > loss: 0.011207576841115952 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13639767307418893) | > loader_time: 0.0046 (0.004672186439101755)  --> TIME: 2025-02-25 20:15:22 -- STEP: 5600/36938 -- GLOBAL_STEP: 929050 | > loss_text_ce: 0.029076704755425453 (0.03297619723681599) | > loss_mel_ce: 2.111842155456543 (nan) | > loss: 0.008495710790157318 (nan) | > current_lr: 5e-06 | > step_time: 0.1965 (0.13638636052608424) | > loader_time: 0.0057 (0.0046737681542124015)  --> TIME: 2025-02-25 20:15:31 -- STEP: 5650/36938 -- GLOBAL_STEP: 929100 | > loss_text_ce: 0.03955796733498573 (0.03298058339254492) | > loss_mel_ce: 3.065786361694336 (nan) | > loss: 0.012322795577347279 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13639350536650185) | > loader_time: 0.0051 (0.004673836505518547)  --> TIME: 2025-02-25 20:15:40 -- STEP: 5700/36938 -- GLOBAL_STEP: 929150 | > loss_text_ce: 0.021881725639104843 (0.03297054963251712) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13637418646561383) | > loader_time: 0.0045 (0.00467439580381962)  --> TIME: 2025-02-25 20:15:50 -- STEP: 5750/36938 -- GLOBAL_STEP: 929200 | > loss_text_ce: 0.031085914000868797 (0.03297422024834386) | > loss_mel_ce: 3.3085925579071045 (nan) | > loss: 0.013252693228423595 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.13637651804219048) | > loader_time: 0.0056 (0.004674360814301861)  --> TIME: 2025-02-25 20:15:59 -- STEP: 5800/36938 -- GLOBAL_STEP: 929250 | > loss_text_ce: 0.030880268663167953 (0.03296111605740284) | > loss_mel_ce: 3.2252256870269775 (nan) | > loss: 0.012921055778861046 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13639765891535463) | > loader_time: 0.0047 (0.004675272949810686)  --> TIME: 2025-02-25 20:16:09 -- STEP: 5850/36938 -- GLOBAL_STEP: 929300 | > loss_text_ce: 0.028881529346108437 (0.03296092149124921) | > loss_mel_ce: 2.503319025039673 (nan) | > loss: 0.010048415511846542 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13641882945329656) | > loader_time: 0.0053 (0.00467602933573927)  --> TIME: 2025-02-25 20:16:18 -- STEP: 5900/36938 -- GLOBAL_STEP: 929350 | > loss_text_ce: 0.040329720824956894 (0.03297067229241357) | > loss_mel_ce: 3.3370120525360107 (nan) | > loss: 0.013402150943875313 (nan) | > current_lr: 5e-06 | > step_time: 0.1822 (0.13643877829535492) | > loader_time: 0.0049 (0.004676324593818795)  --> TIME: 2025-02-25 20:16:27 -- STEP: 5950/36938 -- GLOBAL_STEP: 929400 | > loss_text_ce: 0.037571001797914505 (0.032969630922527075) | > loss_mel_ce: 0.030029296875 (nan) | > loss: 0.0002682551566977054 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.1364127450029385) | > loader_time: 0.0045 (0.004676154441192368)  --> TIME: 2025-02-25 20:16:37 -- STEP: 6000/36938 -- GLOBAL_STEP: 929450 | > loss_text_ce: 0.031806252896785736 (0.032973548988811344) | > loss_mel_ce: 2.608057737350464 (nan) | > loss: 0.010475651361048222 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13645142014821338) | > loader_time: 0.0046 (0.004676477710405982)  --> TIME: 2025-02-25 20:16:46 -- STEP: 6050/36938 -- GLOBAL_STEP: 929500 | > loss_text_ce: 0.032540883868932724 (0.032965837408257086) | > loss_mel_ce: 3.02952241897583 (nan) | > loss: 0.01215104479342699 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13644403481286382) | > loader_time: 0.0047 (0.004676091730102031)  --> TIME: 2025-02-25 20:16:55 -- STEP: 6100/36938 -- GLOBAL_STEP: 929550 | > loss_text_ce: 0.031820014119148254 (0.03297094055680433) | > loss_mel_ce: 2.9584429264068604 (nan) | > loss: 0.011866123415529728 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13645246458835245) | > loader_time: 0.0047 (0.0046759590555409845)  --> TIME: 2025-02-25 20:17:05 -- STEP: 6150/36938 -- GLOBAL_STEP: 929600 | > loss_text_ce: 0.029270242899656296 (0.03297005129205708) | > loss_mel_ce: 3.244629144668579 (nan) | > loss: 0.012991664931178093 (nan) | > current_lr: 5e-06 | > step_time: 0.1659 (0.13647316230990983) | > loader_time: 0.0046 (0.0046755758921305296)  --> TIME: 2025-02-25 20:17:14 -- STEP: 6200/36938 -- GLOBAL_STEP: 929650 | > loss_text_ce: 0.03075139783322811 (0.03296714088518979) | > loss_mel_ce: 0.65346759557724 (nan) | > loss: 0.002715155016630888 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13650606539941557) | > loader_time: 0.0046 (0.0046753886822731225)  --> TIME: 2025-02-25 20:17:24 -- STEP: 6250/36938 -- GLOBAL_STEP: 929700 | > loss_text_ce: 0.02830490842461586 (0.03296458026319708) | > loss_mel_ce: 1.3298081159591675 (nan) | > loss: 0.00538933789357543 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13652363132476764) | > loader_time: 0.0045 (0.0046749798965454075)  --> TIME: 2025-02-25 20:17:33 -- STEP: 6300/36938 -- GLOBAL_STEP: 929750 | > loss_text_ce: 0.034074269235134125 (0.03297566265281668) | > loss_mel_ce: 1.4015984535217285 (nan) | > loss: 0.005697114393115044 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.1365133859240815) | > loader_time: 0.0044 (0.004674305991520955)  --> TIME: 2025-02-25 20:17:43 -- STEP: 6350/36938 -- GLOBAL_STEP: 929800 | > loss_text_ce: 0.02847813069820404 (0.03297990502862912) | > loss_mel_ce: 2.17474365234375 (nan) | > loss: 0.008742944337427616 (nan) | > current_lr: 5e-06 | > step_time: 0.163 (0.1365246940222307) | > loader_time: 0.0047 (0.004673916261027179)  --> TIME: 2025-02-25 20:17:52 -- STEP: 6400/36938 -- GLOBAL_STEP: 929850 | > loss_text_ce: 0.034948281943798065 (0.03297867821674989) | > loss_mel_ce: 2.972024440765381 (nan) | > loss: 0.011932432651519775 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.13653040625154925) | > loader_time: 0.0047 (0.0046736712008714695)  --> TIME: 2025-02-25 20:18:01 -- STEP: 6450/36938 -- GLOBAL_STEP: 929900 | > loss_text_ce: 0.03916018456220627 (0.032968598802884067) | > loss_mel_ce: 3.2533152103424072 (nan) | > loss: 0.013065379112958908 (nan) | > current_lr: 5e-06 | > step_time: 0.1442 (0.1365216041535365) | > loader_time: 0.0045 (0.004673177068547686)  --> TIME: 2025-02-25 20:18:11 -- STEP: 6500/36938 -- GLOBAL_STEP: 929950 | > loss_text_ce: 0.03420736640691757 (0.03297334607661884) | > loss_mel_ce: 2.5863795280456543 (nan) | > loss: 0.01039915531873703 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.1365880464407109) | > loader_time: 0.0045 (0.004673136894519514)  --> TIME: 2025-02-25 20:18:20 -- STEP: 6550/36938 -- GLOBAL_STEP: 930000 | > loss_text_ce: 0.022876722738146782 (0.03297185038962862) | > loss_mel_ce: 2.722635507583618 (nan) | > loss: 0.010894890874624252 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13661251402993188) | > loader_time: 0.0047 (0.004672979616936837) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_930000.pth  --> TIME: 2025-02-25 20:18:36 -- STEP: 6600/36938 -- GLOBAL_STEP: 930050 | > loss_text_ce: 0.03597651794552803 (0.03297109746656376) | > loss_mel_ce: 1.0992906093597412 (nan) | > loss: 0.004505028482526541 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13661251133138427) | > loader_time: 0.0053 (0.004694922403855754)  --> TIME: 2025-02-25 20:18:45 -- STEP: 6650/36938 -- GLOBAL_STEP: 930100 | > loss_text_ce: 0.033262938261032104 (0.03297443187690514) | > loss_mel_ce: 2.767448902130127 (nan) | > loss: 0.011113937012851238 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13660208361489393) | > loader_time: 0.0046 (0.004695291339902946)  --> TIME: 2025-02-25 20:18:55 -- STEP: 6700/36938 -- GLOBAL_STEP: 930150 | > loss_text_ce: 0.03870489448308945 (0.032984680648503614) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.1366379283435306) | > loader_time: 0.0046 (0.004695265542215371)  --> TIME: 2025-02-25 20:19:04 -- STEP: 6750/36938 -- GLOBAL_STEP: 930200 | > loss_text_ce: 0.034538257867097855 (0.03298413460149779) | > loss_mel_ce: 2.507905960083008 (nan) | > loss: 0.010089064948260784 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.1366705691372905) | > loader_time: 0.0045 (0.00469633367326524)  --> TIME: 2025-02-25 20:19:13 -- STEP: 6800/36938 -- GLOBAL_STEP: 930250 | > loss_text_ce: 0.046331387013196945 (0.03299348577939161) | > loss_mel_ce: 2.9713597297668457 (nan) | > loss: 0.01197496522217989 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.13665280727779136) | > loader_time: 0.0047 (0.00469591628102695)  --> TIME: 2025-02-25 20:19:22 -- STEP: 6850/36938 -- GLOBAL_STEP: 930300 | > loss_text_ce: 0.027415277436375618 (0.03298499005826255) | > loss_mel_ce: 2.9415454864501953 (nan) | > loss: 0.01178159099072218 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13662439308027252) | > loader_time: 0.0051 (0.0046959355277736686)  --> TIME: 2025-02-25 20:19:32 -- STEP: 6900/36938 -- GLOBAL_STEP: 930350 | > loss_text_ce: 0.03448765352368355 (0.0329880735396904) | > loss_mel_ce: 1.9012647867202759 (nan) | > loss: 0.007681557442992926 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13665504348450783) | > loader_time: 0.0046 (0.004695614075315171)  --> TIME: 2025-02-25 20:19:41 -- STEP: 6950/36938 -- GLOBAL_STEP: 930400 | > loss_text_ce: 0.028333844617009163 (0.03299235659189116) | > loss_mel_ce: 2.568500280380249 (nan) | > loss: 0.010304898023605347 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.13665300979888692) | > loader_time: 0.0046 (0.004695727327744737)  --> TIME: 2025-02-25 20:19:51 -- STEP: 7000/36938 -- GLOBAL_STEP: 930450 | > loss_text_ce: 0.03842935711145401 (0.03300087288660629) | > loss_mel_ce: 2.1298513412475586 (nan) | > loss: 0.008604289032518864 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13669842263630452) | > loader_time: 0.0047 (0.004695429972239902)  --> TIME: 2025-02-25 20:20:00 -- STEP: 7050/36938 -- GLOBAL_STEP: 930500 | > loss_text_ce: 0.011970721185207367 (0.03299982240066851) | > loss_mel_ce: 2.3090648651123047 (nan) | > loss: 0.00921045895665884 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1367149078085067) | > loader_time: 0.0044 (0.004694991686665419)  --> TIME: 2025-02-25 20:20:10 -- STEP: 7100/36938 -- GLOBAL_STEP: 930550 | > loss_text_ce: 0.05190416797995567 (0.03299294559547354) | > loss_mel_ce: 3.0841331481933594 (nan) | > loss: 0.012444593012332916 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13673920671704792) | > loader_time: 0.0045 (0.004694530795997296)  --> TIME: 2025-02-25 20:20:19 -- STEP: 7150/36938 -- GLOBAL_STEP: 930600 | > loss_text_ce: 0.03458276018500328 (0.032992062288355634) | > loss_mel_ce: 2.971020221710205 (nan) | > loss: 0.01192699745297432 (nan) | > current_lr: 5e-06 | > step_time: 0.1817 (0.13673319546492793) | > loader_time: 0.0048 (0.004694047907849291)  --> TIME: 2025-02-25 20:20:29 -- STEP: 7200/36938 -- GLOBAL_STEP: 930650 | > loss_text_ce: 0.02620655484497547 (0.032992354231731) | > loss_mel_ce: 1.418426513671875 (nan) | > loss: 0.005732671357691288 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13673096706469864) | > loader_time: 0.0054 (0.0046940155161751635)  --> TIME: 2025-02-25 20:20:38 -- STEP: 7250/36938 -- GLOBAL_STEP: 930700 | > loss_text_ce: 0.03543496131896973 (0.032994880974806226) | > loss_mel_ce: 2.275541305541992 (nan) | > loss: 0.009170541539788246 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13676803947317215) | > loader_time: 0.0046 (0.004694072065682246)  --> TIME: 2025-02-25 20:20:48 -- STEP: 7300/36938 -- GLOBAL_STEP: 930750 | > loss_text_ce: 0.030366728082299232 (0.03300891867341207) | > loss_mel_ce: 2.9418280124664307 (nan) | > loss: 0.011794423684477806 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.13679756033910476) | > loader_time: 0.0048 (0.004693956995663577)  --> TIME: 2025-02-25 20:20:57 -- STEP: 7350/36938 -- GLOBAL_STEP: 930800 | > loss_text_ce: 0.039755724370479584 (0.033005465841820056) | > loss_mel_ce: 3.4466872215270996 (nan) | > loss: 0.013835092075169086 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.13679322216786519) | > loader_time: 0.0047 (0.004693619572386449)  --> TIME: 2025-02-25 20:21:07 -- STEP: 7400/36938 -- GLOBAL_STEP: 930850 | > loss_text_ce: 0.03994753211736679 (0.03300592548566251) | > loss_mel_ce: 2.5948667526245117 (nan) | > loss: 0.010455613024532795 (nan) | > current_lr: 5e-06 | > step_time: 0.1821 (0.1368238868584505) | > loader_time: 0.0049 (0.004693826050371736)  --> TIME: 2025-02-25 20:21:16 -- STEP: 7450/36938 -- GLOBAL_STEP: 930900 | > loss_text_ce: 0.03893590718507767 (0.03300600188090458) | > loss_mel_ce: 0.02532958984375 (nan) | > loss: 0.0002550218196120113 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13679942959907077) | > loader_time: 0.0048 (0.004694270735619051)  --> TIME: 2025-02-25 20:21:25 -- STEP: 7500/36938 -- GLOBAL_STEP: 930950 | > loss_text_ce: 0.033708687871694565 (0.03300618560810849) | > loss_mel_ce: 2.743941307067871 (nan) | > loss: 0.011022420600056648 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13680912055969244) | > loader_time: 0.0046 (0.004694621117909748)  --> TIME: 2025-02-25 20:21:35 -- STEP: 7550/36938 -- GLOBAL_STEP: 931000 | > loss_text_ce: 0.03262515366077423 (0.03300329079444404) | > loss_mel_ce: 2.7197635173797607 (nan) | > loss: 0.010922177694737911 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13682721472733864) | > loader_time: 0.0046 (0.004694521506101089)  --> TIME: 2025-02-25 20:21:44 -- STEP: 7600/36938 -- GLOBAL_STEP: 931050 | > loss_text_ce: 0.03217773512005806 (0.03300066399816882) | > loss_mel_ce: 1.2792192697525024 (nan) | > loss: 0.00520395627245307 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.1368281979937303) | > loader_time: 0.0046 (0.004695297636483844)  --> TIME: 2025-02-25 20:21:53 -- STEP: 7650/36938 -- GLOBAL_STEP: 931100 | > loss_text_ce: 0.031614065170288086 (0.03300503677292848) | > loss_mel_ce: 2.141380786895752 (nan) | > loss: 0.00862299557775259 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13680657704671229) | > loader_time: 0.0046 (0.004695195715411815)  --> TIME: 2025-02-25 20:22:02 -- STEP: 7700/36938 -- GLOBAL_STEP: 931150 | > loss_text_ce: 0.03623591363430023 (0.033005078295003515) | > loss_mel_ce: 2.964003562927246 (nan) | > loss: 0.011905712075531483 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13679308377303093) | > loader_time: 0.0046 (0.004694610385151652)  --> TIME: 2025-02-25 20:22:12 -- STEP: 7750/36938 -- GLOBAL_STEP: 931200 | > loss_text_ce: 0.03220033273100853 (0.03301449625061851) | > loss_mel_ce: 2.7291276454925537 (nan) | > loss: 0.010957651771605015 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.1367829292974165) | > loader_time: 0.0047 (0.0046946807984382865)  --> TIME: 2025-02-25 20:22:21 -- STEP: 7800/36938 -- GLOBAL_STEP: 931250 | > loss_text_ce: 0.037717655301094055 (0.033011841273221494) | > loss_mel_ce: 2.986185312271118 (nan) | > loss: 0.011999615468084812 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13682615677515672) | > loader_time: 0.0046 (0.004694618108945014)  --> TIME: 2025-02-25 20:22:31 -- STEP: 7850/36938 -- GLOBAL_STEP: 931300 | > loss_text_ce: 0.027049876749515533 (0.03300988637302393) | > loss_mel_ce: 2.3575363159179688 (nan) | > loss: 0.009462643414735794 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13685404862567882) | > loader_time: 0.0046 (0.004694511935969066)  --> TIME: 2025-02-25 20:22:40 -- STEP: 7900/36938 -- GLOBAL_STEP: 931350 | > loss_text_ce: 0.0364765040576458 (0.03301483150229674) | > loss_mel_ce: 2.936138868331909 (nan) | > loss: 0.011796093545854092 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13686703196054778) | > loader_time: 0.0048 (0.004694813987876794)  --> TIME: 2025-02-25 20:22:49 -- STEP: 7950/36938 -- GLOBAL_STEP: 931400 | > loss_text_ce: 0.03116774745285511 (0.03300137283842767) | > loss_mel_ce: 2.690227746963501 (nan) | > loss: 0.01079918909817934 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.1368171055511859) | > loader_time: 0.0045 (0.004693994342156176)  --> TIME: 2025-02-25 20:22:59 -- STEP: 8000/36938 -- GLOBAL_STEP: 931450 | > loss_text_ce: 0.034179192036390305 (0.03299967419716963) | > loss_mel_ce: 3.382000684738159 (nan) | > loss: 0.01355626992881298 (nan) | > current_lr: 5e-06 | > step_time: 0.1844 (0.13683597582578666) | > loader_time: 0.0048 (0.00469404861330986)  --> TIME: 2025-02-25 20:23:08 -- STEP: 8050/36938 -- GLOBAL_STEP: 931500 | > loss_text_ce: 0.03374229371547699 (0.03300557886302346) | > loss_mel_ce: 2.9426724910736084 (nan) | > loss: 0.011811169795691967 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.13687200830589916) | > loader_time: 0.0047 (0.004694004888119904)  --> TIME: 2025-02-25 20:23:17 -- STEP: 8100/36938 -- GLOBAL_STEP: 931550 | > loss_text_ce: 0.03269919008016586 (0.03300627935482272) | > loss_mel_ce: 2.8932180404663086 (nan) | > loss: 0.011610782705247402 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1368355563540518) | > loader_time: 0.0046 (0.004693726227607256)  --> TIME: 2025-02-25 20:23:26 -- STEP: 8150/36938 -- GLOBAL_STEP: 931600 | > loss_text_ce: 0.03101874701678753 (0.033010406564646565) | > loss_mel_ce: 3.362858295440674 (nan) | > loss: 0.013467766344547272 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13681963964473987) | > loader_time: 0.0046 (0.004693613783713498)  --> TIME: 2025-02-25 20:23:36 -- STEP: 8200/36938 -- GLOBAL_STEP: 931650 | > loss_text_ce: 0.03936119005084038 (0.03301890223366517) | > loss_mel_ce: 2.6328506469726562 (nan) | > loss: 0.010604016482830048 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13680223831316327) | > loader_time: 0.0046 (0.004693706500820996)  --> TIME: 2025-02-25 20:23:45 -- STEP: 8250/36938 -- GLOBAL_STEP: 931700 | > loss_text_ce: 0.02671143226325512 (0.033019477346516606) | > loss_mel_ce: 0.0389302559196949 (nan) | > loss: 0.0002604828914627433 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.1368016151659417) | > loader_time: 0.0047 (0.00469407202980735)  --> TIME: 2025-02-25 20:23:55 -- STEP: 8300/36938 -- GLOBAL_STEP: 931750 | > loss_text_ce: 0.031835246831178665 (0.033023299190092024) | > loss_mel_ce: 2.390150785446167 (nan) | > loss: 0.009611056186258793 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13686046626194423) | > loader_time: 0.0052 (0.00469451984727239)  --> TIME: 2025-02-25 20:24:04 -- STEP: 8350/36938 -- GLOBAL_STEP: 931800 | > loss_text_ce: 0.021094363182783127 (0.03303028830767628) | > loss_mel_ce: 2.68571138381958 (nan) | > loss: 0.01074129343032837 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13686358725953257) | > loader_time: 0.0047 (0.004694546710945174)  --> TIME: 2025-02-25 20:24:13 -- STEP: 8400/36938 -- GLOBAL_STEP: 931850 | > loss_text_ce: 0.03784569725394249 (0.033035514613590986) | > loss_mel_ce: 3.0320191383361816 (nan) | > loss: 0.012182003818452358 (nan) | > current_lr: 5e-06 | > step_time: 0.1692 (0.13683860509168547) | > loader_time: 0.0048 (0.004694288060778661)  --> TIME: 2025-02-25 20:24:23 -- STEP: 8450/36938 -- GLOBAL_STEP: 931900 | > loss_text_ce: 0.033867862075567245 (0.03304272083234913) | > loss_mel_ce: 1.9778951406478882 (nan) | > loss: 0.007983187213540077 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13682634838939428) | > loader_time: 0.0046 (0.004694671941226754)  --> TIME: 2025-02-25 20:24:32 -- STEP: 8500/36938 -- GLOBAL_STEP: 931950 | > loss_text_ce: 0.035257261246442795 (0.033047965706380536) | > loss_mel_ce: 3.217026948928833 (nan) | > loss: 0.012905891053378582 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13684983800439285) | > loader_time: 0.0046 (0.004694854876574345)  --> TIME: 2025-02-25 20:24:42 -- STEP: 8550/36938 -- GLOBAL_STEP: 932000 | > loss_text_ce: 0.025684136897325516 (0.033049596416823555) | > loss_mel_ce: 1.1351083517074585 (nan) | > loss: 0.0046063195914030075 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13685538311450815) | > loader_time: 0.0045 (0.004694554261993941)  --> TIME: 2025-02-25 20:24:51 -- STEP: 8600/36938 -- GLOBAL_STEP: 932050 | > loss_text_ce: 0.024017591029405594 (0.03304080541267266) | > loss_mel_ce: 2.7774300575256348 (nan) | > loss: 0.011116856709122658 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13683544153390942) | > loader_time: 0.0046 (0.004694585467493809)  --> TIME: 2025-02-25 20:25:00 -- STEP: 8650/36938 -- GLOBAL_STEP: 932100 | > loss_text_ce: 0.03250568360090256 (0.03304554437669339) | > loss_mel_ce: 2.7552273273468018 (nan) | > loss: 0.011062433943152428 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13682434795908854) | > loader_time: 0.0046 (0.004695451631711391)  --> TIME: 2025-02-25 20:25:10 -- STEP: 8700/36938 -- GLOBAL_STEP: 932150 | > loss_text_ce: 0.0340665727853775 (0.03304040164140759) | > loss_mel_ce: 1.3351662158966064 (nan) | > loss: 0.005433463491499424 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13683826005321817) | > loader_time: 0.0052 (0.004696174407827434)  --> TIME: 2025-02-25 20:25:19 -- STEP: 8750/36938 -- GLOBAL_STEP: 932200 | > loss_text_ce: 0.02687549963593483 (0.03303750416870597) | > loss_mel_ce: 2.563067674636841 (nan) | > loss: 0.010277552530169487 (nan) | > current_lr: 5e-06 | > step_time: 0.1767 (0.1368542255401613) | > loader_time: 0.0048 (0.004696010589599602)  --> TIME: 2025-02-25 20:25:28 -- STEP: 8800/36938 -- GLOBAL_STEP: 932250 | > loss_text_ce: 0.03509689122438431 (0.03303795583186302) | > loss_mel_ce: 3.0709352493286133 (nan) | > loss: 0.012325525283813477 (nan) | > current_lr: 5e-06 | > step_time: 0.1512 (0.13685039961879925) | > loader_time: 0.0048 (0.004695540205998847)  --> TIME: 2025-02-25 20:25:38 -- STEP: 8850/36938 -- GLOBAL_STEP: 932300 | > loss_text_ce: 0.03021916374564171 (0.03303936239500304) | > loss_mel_ce: 2.6957576274871826 (nan) | > loss: 0.010817368514835835 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.1368373791376752) | > loader_time: 0.0047 (0.004695518057225108)  --> TIME: 2025-02-25 20:25:47 -- STEP: 8900/36938 -- GLOBAL_STEP: 932350 | > loss_text_ce: 0.042179618030786514 (0.03303694488522512) | > loss_mel_ce: 2.859008550643921 (nan) | > loss: 0.011512652039527893 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13683367705077285) | > loader_time: 0.0055 (0.004695611964450788)  --> TIME: 2025-02-25 20:25:56 -- STEP: 8950/36938 -- GLOBAL_STEP: 932400 | > loss_text_ce: 0.029971536248922348 (0.03303563844017113) | > loss_mel_ce: 2.700894832611084 (nan) | > loss: 0.010836772620677948 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1368066271989707) | > loader_time: 0.0046 (0.004695357157531391)  --> TIME: 2025-02-25 20:26:05 -- STEP: 9000/36938 -- GLOBAL_STEP: 932450 | > loss_text_ce: 0.0378357395529747 (0.03303070653229921) | > loss_mel_ce: 0.140869140625 (nan) | > loss: 0.0007091464358381927 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13678277823660123) | > loader_time: 0.0045 (0.004694914526409567)  --> TIME: 2025-02-25 20:26:14 -- STEP: 9050/36938 -- GLOBAL_STEP: 932500 | > loss_text_ce: 0.0342685803771019 (0.03303740457194255) | > loss_mel_ce: 3.1604812145233154 (nan) | > loss: 0.012677579186856747 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13680076686058268) | > loader_time: 0.0047 (0.004695029179694238)  --> TIME: 2025-02-25 20:26:24 -- STEP: 9100/36938 -- GLOBAL_STEP: 932550 | > loss_text_ce: 0.030326740816235542 (0.03303974555825762) | > loss_mel_ce: 2.9003336429595947 (nan) | > loss: 0.011629605665802956 (nan) | > current_lr: 5e-06 | > step_time: 0.1799 (0.1367809136883244) | > loader_time: 0.0048 (0.004694931873908402)  --> TIME: 2025-02-25 20:26:33 -- STEP: 9150/36938 -- GLOBAL_STEP: 932600 | > loss_text_ce: 0.03319934383034706 (0.03304399198684506) | > loss_mel_ce: 2.7395808696746826 (nan) | > loss: 0.011003096587955952 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.1368016779748469) | > loader_time: 0.0045 (0.004694711367289217)  --> TIME: 2025-02-25 20:26:42 -- STEP: 9200/36938 -- GLOBAL_STEP: 932650 | > loss_text_ce: 0.03704890236258507 (0.033046350463131216) | > loss_mel_ce: 2.39111328125 (nan) | > loss: 0.009635564871132374 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13679985963779953) | > loader_time: 0.0047 (0.00469434885875038)  --> TIME: 2025-02-25 20:26:52 -- STEP: 9250/36938 -- GLOBAL_STEP: 932700 | > loss_text_ce: 0.030734118074178696 (0.03304507512800566) | > loss_mel_ce: 2.9860973358154297 (nan) | > loss: 0.011971553787589073 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13682265817796863) | > loader_time: 0.0049 (0.004694820017428009)  --> TIME: 2025-02-25 20:27:02 -- STEP: 9300/36938 -- GLOBAL_STEP: 932750 | > loss_text_ce: 0.03389038145542145 (0.033046739354368845) | > loss_mel_ce: 1.7429789304733276 (nan) | > loss: 0.00705106882378459 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13684117571000134) | > loader_time: 0.0054 (0.004695536039208849)  --> TIME: 2025-02-25 20:27:11 -- STEP: 9350/36938 -- GLOBAL_STEP: 932800 | > loss_text_ce: 0.03549947962164879 (0.03304966955181719) | > loss_mel_ce: 1.6992061138153076 (nan) | > loss: 0.006883752532303333 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13685785181382132) | > loader_time: 0.0046 (0.004696041708961524)  --> TIME: 2025-02-25 20:27:21 -- STEP: 9400/36938 -- GLOBAL_STEP: 932850 | > loss_text_ce: 0.031106024980545044 (0.033050562968437676) | > loss_mel_ce: 1.2012755870819092 (nan) | > loss: 0.0048904032446444035 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.1368660108079303) | > loader_time: 0.0046 (0.00469588979761651)  --> TIME: 2025-02-25 20:27:30 -- STEP: 9450/36938 -- GLOBAL_STEP: 932900 | > loss_text_ce: 0.030602147802710533 (0.03304975593669516) | > loss_mel_ce: 1.851806640625 (nan) | > loss: 0.007469876203685999 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13685107168066463) | > loader_time: 0.0046 (0.004695637112572075)  --> TIME: 2025-02-25 20:27:39 -- STEP: 9500/36938 -- GLOBAL_STEP: 932950 | > loss_text_ce: 0.03775351494550705 (0.033046513345092225) | > loss_mel_ce: 2.4993574619293213 (nan) | > loss: 0.010067901574075222 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13684901942704883) | > loader_time: 0.0045 (0.0046953167664377255)  --> TIME: 2025-02-25 20:27:48 -- STEP: 9550/36938 -- GLOBAL_STEP: 933000 | > loss_text_ce: 0.028998980298638344 (0.03303511491016069) | > loss_mel_ce: 2.919367790222168 (nan) | > loss: 0.01169986929744482 (nan) | > current_lr: 5e-06 | > step_time: 0.1456 (0.13684130389028837) | > loader_time: 0.0045 (0.004694876595941503)  --> TIME: 2025-02-25 20:27:58 -- STEP: 9600/36938 -- GLOBAL_STEP: 933050 | > loss_text_ce: 0.029581060633063316 (0.03303555574462097) | > loss_mel_ce: 3.316162109375 (nan) | > loss: 0.013276759535074234 (nan) | > current_lr: 5e-06 | > step_time: 0.1612 (0.1368381809443237) | > loader_time: 0.0046 (0.004695198461413373)  --> TIME: 2025-02-25 20:28:07 -- STEP: 9650/36938 -- GLOBAL_STEP: 933100 | > loss_text_ce: 0.0358472615480423 (0.0330341431952122) | > loss_mel_ce: 1.538771390914917 (nan) | > loss: 0.006248487159609795 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13685971035240863) | > loader_time: 0.0046 (0.0046953603270140235)  --> TIME: 2025-02-25 20:28:17 -- STEP: 9700/36938 -- GLOBAL_STEP: 933150 | > loss_text_ce: 0.02678489312529564 (0.03303573322549586) | > loss_mel_ce: 2.9513397216796875 (nan) | > loss: 0.011817955411970615 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13686698709566586) | > loader_time: 0.0049 (0.004695624985645721)  --> TIME: 2025-02-25 20:28:26 -- STEP: 9750/36938 -- GLOBAL_STEP: 933200 | > loss_text_ce: 0.03563066944479942 (0.03303550733816899) | > loss_mel_ce: 2.666109561920166 (nan) | > loss: 0.010721191763877869 (nan) | > current_lr: 5e-06 | > step_time: 0.1762 (0.1368445374415473) | > loader_time: 0.0048 (0.004695263984875794)  --> TIME: 2025-02-25 20:28:35 -- STEP: 9800/36938 -- GLOBAL_STEP: 933250 | > loss_text_ce: 0.03575950488448143 (0.033029826377157054) | > loss_mel_ce: 2.4661285877227783 (nan) | > loss: 0.009928127750754356 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13683296549076934) | > loader_time: 0.0045 (0.0046957083137667815)  --> TIME: 2025-02-25 20:28:44 -- STEP: 9850/36938 -- GLOBAL_STEP: 933300 | > loss_text_ce: 0.03461376577615738 (0.03302942468959641) | > loss_mel_ce: 2.436077117919922 (nan) | > loss: 0.009804329834878445 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13685012996499313) | > loader_time: 0.0046 (0.004695933579188307)  --> TIME: 2025-02-25 20:28:54 -- STEP: 9900/36938 -- GLOBAL_STEP: 933350 | > loss_text_ce: 0.030726416036486626 (0.03303587999364165) | > loss_mel_ce: 2.9277141094207764 (nan) | > loss: 0.01173984445631504 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13685161990348746) | > loader_time: 0.0045 (0.004696113918766825)  --> TIME: 2025-02-25 20:29:03 -- STEP: 9950/36938 -- GLOBAL_STEP: 933400 | > loss_text_ce: 0.02735907770693302 (0.03303776722962955) | > loss_mel_ce: 2.72324800491333 (nan) | > loss: 0.010915108025074005 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.13686331878355404) | > loader_time: 0.0049 (0.00469609651134241)  --> TIME: 2025-02-25 20:29:13 -- STEP: 10000/36938 -- GLOBAL_STEP: 933450 | > loss_text_ce: 0.03194562718272209 (0.03303632119800868) | > loss_mel_ce: 2.6707165241241455 (nan) | > loss: 0.010724850930273533 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.13685913391113277) | > loader_time: 0.0049 (0.004696035766601558)  --> TIME: 2025-02-25 20:29:22 -- STEP: 10050/36938 -- GLOBAL_STEP: 933500 | > loss_text_ce: 0.036215513944625854 (0.03303454174998359) | > loss_mel_ce: 2.6269419193267822 (nan) | > loss: 0.010568086057901382 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13685188589997554) | > loader_time: 0.0046 (0.00469567377175857)  --> TIME: 2025-02-25 20:29:31 -- STEP: 10100/36938 -- GLOBAL_STEP: 933550 | > loss_text_ce: 0.029310647398233414 (0.03303569093365011) | > loss_mel_ce: 2.6033101081848145 (nan) | > loss: 0.010446908883750439 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13684713762585485) | > loader_time: 0.0045 (0.004695477768926326)  --> TIME: 2025-02-25 20:29:41 -- STEP: 10150/36938 -- GLOBAL_STEP: 933600 | > loss_text_ce: 0.04165627062320709 (0.033033540220793266) | > loss_mel_ce: 2.6394131183624268 (nan) | > loss: 0.010639164596796036 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.1368748114496617) | > loader_time: 0.0047 (0.004695046124199917)  --> TIME: 2025-02-25 20:29:50 -- STEP: 10200/36938 -- GLOBAL_STEP: 933650 | > loss_text_ce: 0.030457261949777603 (0.03303223673546402) | > loss_mel_ce: 2.244391441345215 (nan) | > loss: 0.009027177467942238 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.13685698766334384) | > loader_time: 0.0047 (0.004694625583349473)  --> TIME: 2025-02-25 20:29:59 -- STEP: 10250/36938 -- GLOBAL_STEP: 933700 | > loss_text_ce: 0.03336075320839882 (0.03303737562912965) | > loss_mel_ce: 2.5077741146087646 (nan) | > loss: 0.010083869099617004 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13685068049081942) | > loader_time: 0.0046 (0.004694426536560046)  --> TIME: 2025-02-25 20:30:08 -- STEP: 10300/36938 -- GLOBAL_STEP: 933750 | > loss_text_ce: 0.028238406404852867 (0.0330397152808465) | > loss_mel_ce: 0.032958984375 (nan) | > loss: 0.0002428468142170459 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.1368182111943809) | > loader_time: 0.0046 (0.004694642465091432)  --> TIME: 2025-02-25 20:30:17 -- STEP: 10350/36938 -- GLOBAL_STEP: 933800 | > loss_text_ce: 0.030587034299969673 (0.03304361706914508) | > loss_mel_ce: 0.04547119140625 (nan) | > loss: 0.0003018183633685112 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13679439293589551) | > loader_time: 0.0045 (0.004695348601410343)  --> TIME: 2025-02-25 20:30:26 -- STEP: 10400/36938 -- GLOBAL_STEP: 933850 | > loss_text_ce: 0.024127818644046783 (0.033042620988920655) | > loss_mel_ce: 2.4255237579345703 (nan) | > loss: 0.009720839560031891 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13676164102095803) | > loader_time: 0.0045 (0.00469525694847106)  --> TIME: 2025-02-25 20:30:36 -- STEP: 10450/36938 -- GLOBAL_STEP: 933900 | > loss_text_ce: 0.024979209527373314 (0.03304157898400362) | > loss_mel_ce: 2.4100558757781982 (nan) | > loss: 0.009662837721407413 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.1367997807635073) | > loader_time: 0.0049 (0.004695358413258226)  --> TIME: 2025-02-25 20:30:45 -- STEP: 10500/36938 -- GLOBAL_STEP: 933950 | > loss_text_ce: 0.042167212814092636 (0.033040095934200576) | > loss_mel_ce: 0.1881306916475296 (nan) | > loss: 0.000913880649022758 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1367894691739765) | > loader_time: 0.0046 (0.004695697807130347)  --> TIME: 2025-02-25 20:30:55 -- STEP: 10550/36938 -- GLOBAL_STEP: 934000 | > loss_text_ce: 0.030353309586644173 (0.033039845863436376) | > loss_mel_ce: 1.8187934160232544 (nan) | > loss: 0.007337884046137333 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1368153405754488) | > loader_time: 0.0045 (0.00469555775701151)  --> TIME: 2025-02-25 20:31:04 -- STEP: 10600/36938 -- GLOBAL_STEP: 934050 | > loss_text_ce: 0.03456101939082146 (0.033037064444923624) | > loss_mel_ce: 0.9224700927734375 (nan) | > loss: 0.003797742770984769 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.13684302829346573) | > loader_time: 0.0048 (0.004695318105085831)  --> TIME: 2025-02-25 20:31:14 -- STEP: 10650/36938 -- GLOBAL_STEP: 934100 | > loss_text_ce: 0.031528815627098083 (0.03303621244752327) | > loss_mel_ce: 3.062289237976074 (nan) | > loss: 0.012277056463062763 (nan) | > current_lr: 5e-06 | > step_time: 0.1614 (0.1368762239715863) | > loader_time: 0.0047 (0.0046955209382822756)  --> TIME: 2025-02-25 20:31:24 -- STEP: 10700/36938 -- GLOBAL_STEP: 934150 | > loss_text_ce: 0.03615362197160721 (0.03303555403705488) | > loss_mel_ce: 3.201917886734009 (nan) | > loss: 0.012849490158259869 (nan) | > current_lr: 5e-06 | > step_time: 0.1852 (0.13689082945618686) | > loader_time: 0.0067 (0.0046956512415520435)  --> TIME: 2025-02-25 20:31:33 -- STEP: 10750/36938 -- GLOBAL_STEP: 934200 | > loss_text_ce: 0.026942798867821693 (0.03303649457855949) | > loss_mel_ce: 2.4635801315307617 (nan) | > loss: 0.009883027523756027 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13687857523629843) | > loader_time: 0.0045 (0.00469542738448741)  --> TIME: 2025-02-25 20:31:42 -- STEP: 10800/36938 -- GLOBAL_STEP: 934250 | > loss_text_ce: 0.04476281628012657 (0.03303769064653228) | > loss_mel_ce: 3.234375 (nan) | > loss: 0.013012452982366085 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13687564980100717) | > loader_time: 0.0046 (0.004695288892145499)  --> TIME: 2025-02-25 20:31:51 -- STEP: 10850/36938 -- GLOBAL_STEP: 934300 | > loss_text_ce: 0.03444567695260048 (0.03303973385325962) | > loss_mel_ce: 2.373401403427124 (nan) | > loss: 0.009554949589073658 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13685775934825858) | > loader_time: 0.0045 (0.004695725199264301)  --> TIME: 2025-02-25 20:32:01 -- STEP: 10900/36938 -- GLOBAL_STEP: 934350 | > loss_text_ce: 0.0345466323196888 (0.0330413868540571) | > loss_mel_ce: 2.8943979740142822 (nan) | > loss: 0.011622796766459942 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13686741588312562) | > loader_time: 0.0048 (0.004695587770654504)  --> TIME: 2025-02-25 20:32:10 -- STEP: 10950/36938 -- GLOBAL_STEP: 934400 | > loss_text_ce: 0.028339534997940063 (0.03303992881307771) | > loss_mel_ce: 2.4659032821655273 (nan) | > loss: 0.009897789917886257 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.13686990944762217) | > loader_time: 0.0046 (0.00469547267373837)  --> TIME: 2025-02-25 20:32:19 -- STEP: 11000/36938 -- GLOBAL_STEP: 934450 | > loss_text_ce: 0.030432647094130516 (0.03303929334667235) | > loss_mel_ce: 2.5115091800689697 (nan) | > loss: 0.010087071917951107 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.13686583445288933) | > loader_time: 0.0047 (0.00469541441310534)  --> TIME: 2025-02-25 20:32:29 -- STEP: 11050/36938 -- GLOBAL_STEP: 934500 | > loss_text_ce: 0.0320863351225853 (0.03304040508024474) | > loss_mel_ce: 2.836360216140747 (nan) | > loss: 0.011382725089788437 (nan) | > current_lr: 5e-06 | > step_time: 0.1891 (0.1368718159360585) | > loader_time: 0.0051 (0.004695583339190569)  --> TIME: 2025-02-25 20:32:38 -- STEP: 11100/36938 -- GLOBAL_STEP: 934550 | > loss_text_ce: 0.03145201504230499 (0.03304154914787001) | > loss_mel_ce: 2.8507838249206543 (nan) | > loss: 0.011437444016337395 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13687038552653702) | > loader_time: 0.0046 (0.004695238637494589)  --> TIME: 2025-02-25 20:32:47 -- STEP: 11150/36938 -- GLOBAL_STEP: 934600 | > loss_text_ce: 0.03401952609419823 (0.03304272127767896) | > loss_mel_ce: 2.558647394180298 (nan) | > loss: 0.010288361459970474 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13687111324258996) | > loader_time: 0.0045 (0.004695024019934114)  --> TIME: 2025-02-25 20:32:57 -- STEP: 11200/36938 -- GLOBAL_STEP: 934650 | > loss_text_ce: 0.040809474885463715 (0.033045863534706156) | > loss_mel_ce: 2.959554433822632 (nan) | > loss: 0.011906206607818604 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.1368711236545019) | > loader_time: 0.0046 (0.004695299948964777)  --> TIME: 2025-02-25 20:33:07 -- STEP: 11250/36938 -- GLOBAL_STEP: 934700 | > loss_text_ce: 0.02576335519552231 (0.03304679668173173) | > loss_mel_ce: 2.9161593914031982 (nan) | > loss: 0.011674297042191029 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.1369074567159016) | > loader_time: 0.0048 (0.004695093218485493)  --> TIME: 2025-02-25 20:33:16 -- STEP: 11300/36938 -- GLOBAL_STEP: 934750 | > loss_text_ce: 0.01738196425139904 (0.03304689746770359) | > loss_mel_ce: 2.012267827987671 (nan) | > loss: 0.008054166100919247 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13690307178328512) | > loader_time: 0.0044 (0.004694687497299301)  --> TIME: 2025-02-25 20:33:25 -- STEP: 11350/36938 -- GLOBAL_STEP: 934800 | > loss_text_ce: 0.03767858073115349 (0.0330519091547157) | > loss_mel_ce: 3.181540012359619 (nan) | > loss: 0.012774677015841007 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1368923561373471) | > loader_time: 0.0045 (0.004694872469628931)  --> TIME: 2025-02-25 20:33:35 -- STEP: 11400/36938 -- GLOBAL_STEP: 934850 | > loss_text_ce: 0.027494529262185097 (0.03304815955664289) | > loss_mel_ce: 2.9452433586120605 (nan) | > loss: 0.011796578764915466 (nan) | > current_lr: 5e-06 | > step_time: 0.1877 (0.13692730907808268) | > loader_time: 0.0048 (0.004694827334922628)  --> TIME: 2025-02-25 20:33:44 -- STEP: 11450/36938 -- GLOBAL_STEP: 934900 | > loss_text_ce: 0.04056136682629585 (0.03305003632746583) | > loss_mel_ce: 2.8161251544952393 (nan) | > loss: 0.011336058378219604 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13693618636985005) | > loader_time: 0.0046 (0.004694700678363086)  --> TIME: 2025-02-25 20:33:54 -- STEP: 11500/36938 -- GLOBAL_STEP: 934950 | > loss_text_ce: 0.0293022021651268 (0.03304715955848581) | > loss_mel_ce: 1.0174424648284912 (nan) | > loss: 0.004153748974204063 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.1369297933785811) | > loader_time: 0.0047 (0.004694452700407576)  --> TIME: 2025-02-25 20:34:03 -- STEP: 11550/36938 -- GLOBAL_STEP: 935000 | > loss_text_ce: 0.04341501742601395 (0.033050655883321435) | > loss_mel_ce: 2.5410239696502686 (nan) | > loss: 0.010255711153149605 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13693351446292096) | > loader_time: 0.0046 (0.004694714401707474)  --> TIME: 2025-02-25 20:34:12 -- STEP: 11600/36938 -- GLOBAL_STEP: 935050 | > loss_text_ce: 0.0379410982131958 (0.033051988562038934) | > loss_mel_ce: 2.8075156211853027 (nan) | > loss: 0.011291495524346828 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.1369299006256563) | > loader_time: 0.0052 (0.0046948068306363585)  --> TIME: 2025-02-25 20:34:21 -- STEP: 11650/36938 -- GLOBAL_STEP: 935100 | > loss_text_ce: 0.032081566751003265 (0.03305088636693185) | > loss_mel_ce: 2.9863953590393066 (nan) | > loss: 0.011978084221482277 (nan) | > current_lr: 5e-06 | > step_time: 0.1435 (0.13689391238494997) | > loader_time: 0.0054 (0.004695025615937688)  --> TIME: 2025-02-25 20:34:30 -- STEP: 11700/36938 -- GLOBAL_STEP: 935150 | > loss_text_ce: 0.035562291741371155 (0.03305340206529131) | > loss_mel_ce: 1.62112295627594 (nan) | > loss: 0.006574148312211037 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13688141963420764) | > loader_time: 0.0045 (0.004694959078079587)  --> TIME: 2025-02-25 20:34:40 -- STEP: 11750/36938 -- GLOBAL_STEP: 935200 | > loss_text_ce: 0.031309131532907486 (0.033055767556057804) | > loss_mel_ce: 3.1674246788024902 (nan) | > loss: 0.012693389318883419 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.1369016525187391) | > loader_time: 0.0046 (0.00469511567785382)  --> TIME: 2025-02-25 20:34:50 -- STEP: 11800/36938 -- GLOBAL_STEP: 935250 | > loss_text_ce: 0.036698807030916214 (0.033056014496596134) | > loss_mel_ce: 2.811748743057251 (nan) | > loss: 0.011303364299237728 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13692050083208904) | > loader_time: 0.0046 (0.004695009841757275)  --> TIME: 2025-02-25 20:34:58 -- STEP: 11850/36938 -- GLOBAL_STEP: 935300 | > loss_text_ce: 0.02589692734181881 (0.0330571633958519) | > loss_mel_ce: 2.300447702407837 (nan) | > loss: 0.009231527335941792 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13689167541793637) | > loader_time: 0.0052 (0.00469501833372475)  --> TIME: 2025-02-25 20:35:08 -- STEP: 11900/36938 -- GLOBAL_STEP: 935350 | > loss_text_ce: 0.029603030532598495 (0.03305462065324981) | > loss_mel_ce: 2.375889539718628 (nan) | > loss: 0.009545605629682541 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1368776357073744) | > loader_time: 0.0046 (0.004695085216970973)  --> TIME: 2025-02-25 20:35:17 -- STEP: 11950/36938 -- GLOBAL_STEP: 935400 | > loss_text_ce: 0.04823882505297661 (0.03305833343751549) | > loss_mel_ce: 3.0036065578460693 (nan) | > loss: 0.012110497802495956 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13687257968232228) | > loader_time: 0.0048 (0.0046951891288597395)  --> TIME: 2025-02-25 20:35:26 -- STEP: 12000/36938 -- GLOBAL_STEP: 935450 | > loss_text_ce: 0.0391397625207901 (0.03305711481999584) | > loss_mel_ce: 3.190859317779541 (nan) | > loss: 0.01281745731830597 (nan) | > current_lr: 5e-06 | > step_time: 0.1711 (0.13687269882361106) | > loader_time: 0.0055 (0.004695314367612171)  --> TIME: 2025-02-25 20:35:35 -- STEP: 12050/36938 -- GLOBAL_STEP: 935500 | > loss_text_ce: 0.029076363891363144 (0.03304645485658161) | > loss_mel_ce: 1.7981140613555908 (nan) | > loss: 0.007250756025314331 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13685371664055165) | > loader_time: 0.0046 (0.004695200188031306)  --> TIME: 2025-02-25 20:35:45 -- STEP: 12100/36938 -- GLOBAL_STEP: 935550 | > loss_text_ce: 0.04331817105412483 (0.0330497169179824) | > loss_mel_ce: 2.187762975692749 (nan) | > loss: 0.008853497914969921 (nan) | > current_lr: 5e-06 | > step_time: 0.1618 (0.1368652643645106) | > loader_time: 0.0048 (0.004695085631914345)  --> TIME: 2025-02-25 20:35:54 -- STEP: 12150/36938 -- GLOBAL_STEP: 935600 | > loss_text_ce: 0.032122839242219925 (0.03305877407000918) | > loss_mel_ce: 3.069089412689209 (nan) | > loss: 0.012306398712098598 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13684597556973688) | > loader_time: 0.0048 (0.004695428349844195)  --> TIME: 2025-02-25 20:36:03 -- STEP: 12200/36938 -- GLOBAL_STEP: 935650 | > loss_text_ce: 0.023604124784469604 (0.03305934378655881) | > loss_mel_ce: 2.2836406230926514 (nan) | > loss: 0.00915573351085186 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13684190247879666) | > loader_time: 0.0047 (0.00469563349348597)  --> TIME: 2025-02-25 20:36:12 -- STEP: 12250/36938 -- GLOBAL_STEP: 935700 | > loss_text_ce: 0.038363609462976456 (0.033059316140276254) | > loss_mel_ce: 2.6479690074920654 (nan) | > loss: 0.010660051368176937 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13682908344268818) | > loader_time: 0.0046 (0.004695532643065137)  --> TIME: 2025-02-25 20:36:21 -- STEP: 12300/36938 -- GLOBAL_STEP: 935750 | > loss_text_ce: 0.014127272181212902 (0.033063412638971536) | > loss_mel_ce: 3.5475852489471436 (nan) | > loss: 0.014133780263364315 (nan) | > current_lr: 5e-06 | > step_time: 0.1074 (0.13680981403443893) | > loader_time: 0.0044 (0.004695111212691619)  --> TIME: 2025-02-25 20:36:30 -- STEP: 12350/36938 -- GLOBAL_STEP: 935800 | > loss_text_ce: 0.019463732838630676 (0.0330613732184112) | > loss_mel_ce: 2.0223116874694824 (nan) | > loss: 0.00810228381305933 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.1367834077865974) | > loader_time: 0.0045 (0.004694956034301242)  --> TIME: 2025-02-25 20:36:39 -- STEP: 12400/36938 -- GLOBAL_STEP: 935850 | > loss_text_ce: 0.0363796092569828 (0.03306658083197429) | > loss_mel_ce: 3.492868423461914 (nan) | > loss: 0.01400495320558548 (nan) | > current_lr: 5e-06 | > step_time: 0.1634 (0.13677377710419367) | > loader_time: 0.0049 (0.004694664555211202)  --> TIME: 2025-02-25 20:36:49 -- STEP: 12450/36938 -- GLOBAL_STEP: 935900 | > loss_text_ce: 0.03653300181031227 (0.03306496442800526) | > loss_mel_ce: 2.9212400913238525 (nan) | > loss: 0.011737195774912834 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.1367868594280692) | > loader_time: 0.0047 (0.004694721210433757)  --> TIME: 2025-02-25 20:36:58 -- STEP: 12500/36938 -- GLOBAL_STEP: 935950 | > loss_text_ce: 0.028819553554058075 (0.03306741908028702) | > loss_mel_ce: 2.3561248779296875 (nan) | > loss: 0.009464065544307232 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13677430938720758) | > loader_time: 0.0045 (0.004694510574340796)  --> TIME: 2025-02-25 20:37:07 -- STEP: 12550/36938 -- GLOBAL_STEP: 936000 | > loss_text_ce: 0.03813082352280617 (0.03306448435804145) | > loss_mel_ce: 2.9505417346954346 (nan) | > loss: 0.011859811842441559 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13675970493559866) | > loader_time: 0.0046 (0.004694371869364557)  --> TIME: 2025-02-25 20:37:16 -- STEP: 12600/36938 -- GLOBAL_STEP: 936050 | > loss_text_ce: 0.02670757658779621 (0.03306063496699859) | > loss_mel_ce: 0.07419433444738388 (nan) | > loss: 0.00040040441672317684 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.1367431714610453) | > loader_time: 0.0046 (0.004694046595739913)  --> TIME: 2025-02-25 20:37:25 -- STEP: 12650/36938 -- GLOBAL_STEP: 936100 | > loss_text_ce: 0.03218952193856239 (0.0330604431041967) | > loss_mel_ce: 3.114119529724121 (nan) | > loss: 0.012485354207456112 (nan) | > current_lr: 5e-06 | > step_time: 0.1869 (0.13674040898032863) | > loader_time: 0.0056 (0.004693937452414263)  --> TIME: 2025-02-25 20:37:35 -- STEP: 12700/36938 -- GLOBAL_STEP: 936150 | > loss_text_ce: 0.04828425869345665 (0.03306616313212828) | > loss_mel_ce: 2.9726946353912354 (nan) | > loss: 0.011988012120127678 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13673976778045457) | > loader_time: 0.0047 (0.004693711592456463)  --> TIME: 2025-02-25 20:37:44 -- STEP: 12750/36938 -- GLOBAL_STEP: 936200 | > loss_text_ce: 0.03407910093665123 (0.033061367462662854) | > loss_mel_ce: 2.0770468711853027 (nan) | > loss: 0.008377484045922756 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13674942742142068) | > loader_time: 0.0045 (0.004693438660864711)  --> TIME: 2025-02-25 20:37:54 -- STEP: 12800/36938 -- GLOBAL_STEP: 936250 | > loss_text_ce: 0.042759913951158524 (0.03306184678702242) | > loss_mel_ce: 2.996044874191284 (nan) | > loss: 0.012058749794960022 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.13676832105964454) | > loader_time: 0.0044 (0.00469319362193344)  --> TIME: 2025-02-25 20:38:03 -- STEP: 12850/36938 -- GLOBAL_STEP: 936300 | > loss_text_ce: 0.028776509687304497 (0.033062316088679676) | > loss_mel_ce: 3.2486212253570557 (nan) | > loss: 0.013005547225475311 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.136768689842076) | > loader_time: 0.0046 (0.004692767269416521)  --> TIME: 2025-02-25 20:38:13 -- STEP: 12900/36938 -- GLOBAL_STEP: 936350 | > loss_text_ce: 0.026828162372112274 (0.03306377288257288) | > loss_mel_ce: 2.885364055633545 (nan) | > loss: 0.011556318961083889 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13676815045896457) | > loader_time: 0.0045 (0.00469243866528649)  --> TIME: 2025-02-25 20:38:22 -- STEP: 12950/36938 -- GLOBAL_STEP: 936400 | > loss_text_ce: 0.03735419735312462 (0.03306257142595435) | > loss_mel_ce: 2.747283935546875 (nan) | > loss: 0.011050151661038399 (nan) | > current_lr: 5e-06 | > step_time: 0.1558 (0.13675571299888029) | > loader_time: 0.0047 (0.004692702403860178)  --> TIME: 2025-02-25 20:38:31 -- STEP: 13000/36938 -- GLOBAL_STEP: 936450 | > loss_text_ce: 0.04006805270910263 (0.03306974920329541) | > loss_mel_ce: 2.654250144958496 (nan) | > loss: 0.010691739618778229 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.13675295600524345) | > loader_time: 0.0049 (0.004692728060942405)  --> TIME: 2025-02-25 20:38:40 -- STEP: 13050/36938 -- GLOBAL_STEP: 936500 | > loss_text_ce: 0.03319409862160683 (0.03307205038195114) | > loss_mel_ce: 2.805816411972046 (nan) | > loss: 0.011265914887189865 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.13675752394957597) | > loader_time: 0.0047 (0.004693047589269155)  --> TIME: 2025-02-25 20:38:49 -- STEP: 13100/36938 -- GLOBAL_STEP: 936550 | > loss_text_ce: 0.05149872601032257 (0.03307054758569521) | > loss_mel_ce: 3.351107597351074 (nan) | > loss: 0.013502406887710094 (nan) | > current_lr: 5e-06 | > step_time: 0.1266 (0.1367403004187666) | > loader_time: 0.0046 (0.004692933286419329)  --> TIME: 2025-02-25 20:38:59 -- STEP: 13150/36938 -- GLOBAL_STEP: 936600 | > loss_text_ce: 0.0287484023720026 (0.03307036682113465) | > loss_mel_ce: 2.9418632984161377 (nan) | > loss: 0.0117881428450346 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13673451539681467) | > loader_time: 0.0046 (0.004692753349420213)  --> TIME: 2025-02-25 20:39:08 -- STEP: 13200/36938 -- GLOBAL_STEP: 936650 | > loss_text_ce: 0.03527360409498215 (0.03307414529915642) | > loss_mel_ce: 2.2881948947906494 (nan) | > loss: 0.009220113046467304 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13672341968074025) | > loader_time: 0.0046 (0.004692636005806178)  --> TIME: 2025-02-25 20:39:17 -- STEP: 13250/36938 -- GLOBAL_STEP: 936700 | > loss_text_ce: 0.03974808752536774 (0.03307240329542217) | > loss_mel_ce: 3.2151200771331787 (nan) | > loss: 0.012916144914925098 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.13672109970956509) | > loader_time: 0.0047 (0.004692973748692902)  --> TIME: 2025-02-25 20:39:26 -- STEP: 13300/36938 -- GLOBAL_STEP: 936750 | > loss_text_ce: 0.03197440132498741 (0.033070504389478894) | > loss_mel_ce: 2.7017455101013184 (nan) | > loss: 0.010848094709217548 (nan) | > current_lr: 5e-06 | > step_time: 0.1856 (0.13670968853441423) | > loader_time: 0.0058 (0.004693617498067962)  --> TIME: 2025-02-25 20:39:35 -- STEP: 13350/36938 -- GLOBAL_STEP: 936800 | > loss_text_ce: 0.028442736715078354 (0.03306952824948626) | > loss_mel_ce: 2.4991166591644287 (nan) | > loss: 0.010029997676610947 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13669950697752442) | > loader_time: 0.0045 (0.004693354970953413)  --> TIME: 2025-02-25 20:39:45 -- STEP: 13400/36938 -- GLOBAL_STEP: 936850 | > loss_text_ce: 0.020270876586437225 (0.03306489935784171) | > loss_mel_ce: 2.697699546813965 (nan) | > loss: 0.010785597376525402 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13670831379605794) | > loader_time: 0.0045 (0.004693377178106712)  --> TIME: 2025-02-25 20:39:54 -- STEP: 13450/36938 -- GLOBAL_STEP: 936900 | > loss_text_ce: 0.02946477197110653 (0.03305755720581125) | > loss_mel_ce: 1.4623141288757324 (nan) | > loss: 0.005919757764786482 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13670962315952925) | > loader_time: 0.0045 (0.0046933841705322004)  --> TIME: 2025-02-25 20:40:03 -- STEP: 13500/36938 -- GLOBAL_STEP: 936950 | > loss_text_ce: 0.04420239478349686 (0.03305258329157449) | > loss_mel_ce: 2.552961826324463 (nan) | > loss: 0.010306207463145256 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1367045656310189) | > loader_time: 0.0045 (0.004693162935751428)  --> TIME: 2025-02-25 20:40:13 -- STEP: 13550/36938 -- GLOBAL_STEP: 937000 | > loss_text_ce: 0.027847277000546455 (0.03305522546117132) | > loss_mel_ce: 3.2266063690185547 (nan) | > loss: 0.012914499267935753 (nan) | > current_lr: 5e-06 | > step_time: 0.1717 (0.136706726190349) | > loader_time: 0.0047 (0.0046931293001913656)  --> TIME: 2025-02-25 20:40:22 -- STEP: 13600/36938 -- GLOBAL_STEP: 937050 | > loss_text_ce: 0.03217775374650955 (0.033054721462950944) | > loss_mel_ce: 2.7170584201812744 (nan) | > loss: 0.010909668169915676 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.1367119424833973) | > loader_time: 0.0045 (0.004693141386789404)  --> TIME: 2025-02-25 20:40:31 -- STEP: 13650/36938 -- GLOBAL_STEP: 937100 | > loss_text_ce: 0.03459211811423302 (0.033055804902653294) | > loss_mel_ce: 3.107454299926758 (nan) | > loss: 0.012468438595533371 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.13670637169163763) | > loader_time: 0.0046 (0.004692911927079952)  --> TIME: 2025-02-25 20:40:41 -- STEP: 13700/36938 -- GLOBAL_STEP: 937150 | > loss_text_ce: 0.03723432868719101 (0.03305613292571484) | > loss_mel_ce: 2.8924548625946045 (nan) | > loss: 0.011625750921666622 (nan) | > current_lr: 5e-06 | > step_time: 0.1778 (0.13670763706638897) | > loader_time: 0.0048 (0.004693264230324376)  --> TIME: 2025-02-25 20:40:50 -- STEP: 13750/36938 -- GLOBAL_STEP: 937200 | > loss_text_ce: 0.028522560372948647 (0.03305142585526785) | > loss_mel_ce: 2.389791488647461 (nan) | > loss: 0.009596484713256359 (nan) | > current_lr: 5e-06 | > step_time: 0.1945 (0.13669844471324608) | > loader_time: 0.0051 (0.004693395077098472)  --> TIME: 2025-02-25 20:40:59 -- STEP: 13800/36938 -- GLOBAL_STEP: 937250 | > loss_text_ce: 0.02857070602476597 (0.03304908006816431) | > loss_mel_ce: 3.0322465896606445 (nan) | > loss: 0.012146100401878357 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.13670854383620698) | > loader_time: 0.0048 (0.004693416771681384)  --> TIME: 2025-02-25 20:41:09 -- STEP: 13850/36938 -- GLOBAL_STEP: 937300 | > loss_text_ce: 0.03448525071144104 (0.03305261485399624) | > loss_mel_ce: 0.06116485595703125 (nan) | > loss: 0.00037956394953653216 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13674141603256426) | > loader_time: 0.0047 (0.004693600795759594)  --> TIME: 2025-02-25 20:41:18 -- STEP: 13900/36938 -- GLOBAL_STEP: 937350 | > loss_text_ce: 0.04356173798441887 (0.03305073474808541) | > loss_mel_ce: 2.468052864074707 (nan) | > loss: 0.00996672548353672 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1367316569355756) | > loader_time: 0.0051 (0.004693500686892475)  --> TIME: 2025-02-25 20:41:28 -- STEP: 13950/36938 -- GLOBAL_STEP: 937400 | > loss_text_ce: 0.030347174033522606 (0.03305502741233721) | > loss_mel_ce: 2.3611321449279785 (nan) | > loss: 0.009489997290074825 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13673659856174186) | > loader_time: 0.0066 (0.004693689585586575)  --> TIME: 2025-02-25 20:41:37 -- STEP: 14000/36938 -- GLOBAL_STEP: 937450 | > loss_text_ce: 0.04089249670505524 (0.03305501830045643) | > loss_mel_ce: 2.565480947494507 (nan) | > loss: 0.010342752560973167 (nan) | > current_lr: 5e-06 | > step_time: 0.1528 (0.13672755817004645) | > loader_time: 0.0046 (0.00469347919736587)  --> TIME: 2025-02-25 20:41:46 -- STEP: 14050/36938 -- GLOBAL_STEP: 937500 | > loss_text_ce: 0.0382552444934845 (0.03305666541912671) | > loss_mel_ce: 3.09924054145813 (nan) | > loss: 0.012450380250811577 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13670581500301177) | > loader_time: 0.0045 (0.004693272037438199)  --> TIME: 2025-02-25 20:41:55 -- STEP: 14100/36938 -- GLOBAL_STEP: 937550 | > loss_text_ce: 0.02766082063317299 (0.03305140240078252) | > loss_mel_ce: 2.637847900390625 (nan) | > loss: 0.01057741604745388 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13669873161518845) | > loader_time: 0.0046 (0.004693199708952099)  --> TIME: 2025-02-25 20:42:05 -- STEP: 14150/36938 -- GLOBAL_STEP: 937600 | > loss_text_ce: 0.0342574268579483 (0.03304918696057329) | > loss_mel_ce: 1.8649653196334839 (nan) | > loss: 0.007536598481237888 (nan) | > current_lr: 5e-06 | > step_time: 0.1588 (0.1367031071295589) | > loader_time: 0.0048 (0.004693098388375299)  --> TIME: 2025-02-25 20:42:14 -- STEP: 14200/36938 -- GLOBAL_STEP: 937650 | > loss_text_ce: 0.036006517708301544 (0.03305241434192128) | > loss_mel_ce: 3.0290539264678955 (nan) | > loss: 0.012162938714027405 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.13670777220121605) | > loader_time: 0.005 (0.004692916786166956)  --> TIME: 2025-02-25 20:42:24 -- STEP: 14250/36938 -- GLOBAL_STEP: 937700 | > loss_text_ce: 0.026231534779071808 (0.03305194179347705) | > loss_mel_ce: 2.3634183406829834 (nan) | > loss: 0.009482738561928272 (nan) | > current_lr: 5e-06 | > step_time: 0.1634 (0.13671796633067912) | > loader_time: 0.0048 (0.004692987308167548)  --> TIME: 2025-02-25 20:42:33 -- STEP: 14300/36938 -- GLOBAL_STEP: 937750 | > loss_text_ce: 0.03920670226216316 (0.033057169393355956) | > loss_mel_ce: 2.8943142890930176 (nan) | > loss: 0.011640957556664944 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13672285368392553) | > loader_time: 0.0047 (0.004693190267869616)  --> TIME: 2025-02-25 20:42:42 -- STEP: 14350/36938 -- GLOBAL_STEP: 937800 | > loss_text_ce: 0.03583833947777748 (0.03305978479285659) | > loss_mel_ce: 0.169677734375 (nan) | > loss: 0.000815540028270334 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13672342448284433) | > loader_time: 0.0046 (0.004692958090778816)  --> TIME: 2025-02-25 20:42:51 -- STEP: 14400/36938 -- GLOBAL_STEP: 937850 | > loss_text_ce: 0.02884451486170292 (0.033055589318197576) | > loss_mel_ce: 2.645982503890991 (nan) | > loss: 0.010614394210278988 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13670241463515523) | > loader_time: 0.0045 (0.004692606478929493)  --> TIME: 2025-02-25 20:43:00 -- STEP: 14450/36938 -- GLOBAL_STEP: 937900 | > loss_text_ce: 0.03777230530977249 (0.0330583091796886) | > loss_mel_ce: 2.899564027786255 (nan) | > loss: 0.011656098067760468 (nan) | > current_lr: 5e-06 | > step_time: 0.1612 (0.13669497417331178) | > loader_time: 0.0048 (0.004692526358634104)  --> TIME: 2025-02-25 20:43:10 -- STEP: 14500/36938 -- GLOBAL_STEP: 937950 | > loss_text_ce: 0.029193608090281487 (0.03305852688091043) | > loss_mel_ce: 2.1614582538604736 (nan) | > loss: 0.008693063631653786 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13669510908784563) | > loader_time: 0.0047 (0.004692439473908496)  --> TIME: 2025-02-25 20:43:19 -- STEP: 14550/36938 -- GLOBAL_STEP: 938000 | > loss_text_ce: 0.022772053256630898 (0.03305492370424584) | > loss_mel_ce: 2.057380437850952 (nan) | > loss: 0.008254573680460453 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13669141426938533) | > loader_time: 0.0046 (0.004692373963975389)  --> TIME: 2025-02-25 20:43:28 -- STEP: 14600/36938 -- GLOBAL_STEP: 938050 | > loss_text_ce: 0.025489933788776398 (0.033052477392380555) | > loss_mel_ce: 2.9113731384277344 (nan) | > loss: 0.011654219590127468 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13668163196681316) | > loader_time: 0.0045 (0.004692208048415484)  --> TIME: 2025-02-25 20:43:38 -- STEP: 14650/36938 -- GLOBAL_STEP: 938100 | > loss_text_ce: 0.02404235117137432 (0.033049372280772815) | > loss_mel_ce: 3.364556312561035 (nan) | > loss: 0.01344682089984417 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1366986446738652) | > loader_time: 0.0047 (0.004692083446645874)  --> TIME: 2025-02-25 20:43:47 -- STEP: 14700/36938 -- GLOBAL_STEP: 938150 | > loss_text_ce: 0.03329801186919212 (0.03305331869342268) | > loss_mel_ce: 2.8883073329925537 (nan) | > loss: 0.011593672446906567 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.1366750180478001) | > loader_time: 0.0047 (0.004692040316912566)  --> TIME: 2025-02-25 20:43:56 -- STEP: 14750/36938 -- GLOBAL_STEP: 938200 | > loss_text_ce: 0.029786216095089912 (0.03304711713124102) | > loss_mel_ce: 3.2613778114318848 (nan) | > loss: 0.013060174882411957 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13666836339336347) | > loader_time: 0.0046 (0.004692265526723027)  --> TIME: 2025-02-25 20:44:05 -- STEP: 14800/36938 -- GLOBAL_STEP: 938250 | > loss_text_ce: 0.019557258114218712 (0.033044695380093206) | > loss_mel_ce: 0.07844237983226776 (nan) | > loss: 0.00038888747803866863 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13665772189965134) | > loader_time: 0.0046 (0.004692345809292123)  --> TIME: 2025-02-25 20:44:14 -- STEP: 14850/36938 -- GLOBAL_STEP: 938300 | > loss_text_ce: 0.03920029476284981 (0.033046808049383515) | > loss_mel_ce: 3.0634686946868896 (nan) | > loss: 0.012312179431319237 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13664573534570568) | > loader_time: 0.0046 (0.004692194229023198)  --> TIME: 2025-02-25 20:44:23 -- STEP: 14900/36938 -- GLOBAL_STEP: 938350 | > loss_text_ce: 0.030296292155981064 (0.0330434794495217) | > loss_mel_ce: 3.3336808681488037 (nan) | > loss: 0.013349116779863834 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13664667795168484) | > loader_time: 0.0047 (0.004692055266975531)  --> TIME: 2025-02-25 20:44:33 -- STEP: 14950/36938 -- GLOBAL_STEP: 938400 | > loss_text_ce: 0.03658366575837135 (0.033042109315610385) | > loss_mel_ce: 3.2131292819976807 (nan) | > loss: 0.012895687483251095 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13666131877580234) | > loader_time: 0.0045 (0.004692054783620139)  --> TIME: 2025-02-25 20:44:42 -- STEP: 15000/36938 -- GLOBAL_STEP: 938450 | > loss_text_ce: 0.03522646427154541 (0.03304334155209336) | > loss_mel_ce: 2.9281253814697266 (nan) | > loss: 0.011759333312511444 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.13666874844233223) | > loader_time: 0.0047 (0.00469191644986468)  --> TIME: 2025-02-25 20:44:52 -- STEP: 15050/36938 -- GLOBAL_STEP: 938500 | > loss_text_ce: 0.03436039760708809 (0.03304667668633655) | > loss_mel_ce: 2.5696451663970947 (nan) | > loss: 0.010333356447517872 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.1366847522076581) | > loader_time: 0.0046 (0.004691822489234671)  --> TIME: 2025-02-25 20:45:01 -- STEP: 15100/36938 -- GLOBAL_STEP: 938550 | > loss_text_ce: 0.024619681760668755 (0.033044257755236825) | > loss_mel_ce: 2.1444613933563232 (nan) | > loss: 0.008607464842498302 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.136679209589169) | > loader_time: 0.0047 (0.0046915080215757)  --> TIME: 2025-02-25 20:45:11 -- STEP: 15150/36938 -- GLOBAL_STEP: 938600 | > loss_text_ce: 0.033907998353242874 (0.03304591413488946) | > loss_mel_ce: 2.9057412147521973 (nan) | > loss: 0.011665275320410728 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13669274279780147) | > loader_time: 0.0046 (0.004691327239813952)  --> TIME: 2025-02-25 20:45:20 -- STEP: 15200/36938 -- GLOBAL_STEP: 938650 | > loss_text_ce: 0.03302053362131119 (0.03304494046213959) | > loss_mel_ce: 2.9445557594299316 (nan) | > loss: 0.011815779842436314 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13669305812371432) | > loader_time: 0.0045 (0.004691114378602856)  --> TIME: 2025-02-25 20:45:29 -- STEP: 15250/36938 -- GLOBAL_STEP: 938700 | > loss_text_ce: 0.04152403771877289 (0.033042717946601435) | > loss_mel_ce: 2.770886182785034 (nan) | > loss: 0.011160357855260372 (nan) | > current_lr: 5e-06 | > step_time: 0.1617 (0.13668587561122733) | > loader_time: 0.0046 (0.004691309084657738)  --> TIME: 2025-02-25 20:45:39 -- STEP: 15300/36938 -- GLOBAL_STEP: 938750 | > loss_text_ce: 0.027199503034353256 (0.03304579528219049) | > loss_mel_ce: 3.0325562953948975 (nan) | > loss: 0.012141888961195946 (nan) | > current_lr: 5e-06 | > step_time: 0.1797 (0.13669270608939404) | > loader_time: 0.0047 (0.004691333926581065)  --> TIME: 2025-02-25 20:45:48 -- STEP: 15350/36938 -- GLOBAL_STEP: 938800 | > loss_text_ce: 0.04198954999446869 (0.03305116319743718) | > loss_mel_ce: 2.9695889949798584 (nan) | > loss: 0.01195070892572403 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.13669836128185187) | > loader_time: 0.0046 (0.004691452281094504)  --> TIME: 2025-02-25 20:45:57 -- STEP: 15400/36938 -- GLOBAL_STEP: 938850 | > loss_text_ce: 0.01977943442761898 (0.0330501355479965) | > loss_mel_ce: 2.239980459213257 (nan) | > loss: 0.008967301808297634 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.1366827714133575) | > loader_time: 0.0045 (0.004691258266374637)  --> TIME: 2025-02-25 20:46:07 -- STEP: 15450/36938 -- GLOBAL_STEP: 938900 | > loss_text_ce: 0.029019441455602646 (0.03304541547381835) | > loss_mel_ce: 2.769263982772827 (nan) | > loss: 0.011104299686849117 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13669150531484883) | > loader_time: 0.0047 (0.004691249282614674)  --> TIME: 2025-02-25 20:46:16 -- STEP: 15500/36938 -- GLOBAL_STEP: 938950 | > loss_text_ce: 0.032991714775562286 (0.03304819925369733) | > loss_mel_ce: 2.864774703979492 (nan) | > loss: 0.01149907335639 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13668604540055823) | > loader_time: 0.0048 (0.004691353997876542)  --> TIME: 2025-02-25 20:46:25 -- STEP: 15550/36938 -- GLOBAL_STEP: 939000 | > loss_text_ce: 0.029728706926107407 (0.03304694754110442) | > loss_mel_ce: 1.8016482591629028 (nan) | > loss: 0.007267368957400322 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.1366767923778279) | > loader_time: 0.0047 (0.00469112781083083)  --> TIME: 2025-02-25 20:46:34 -- STEP: 15600/36938 -- GLOBAL_STEP: 939050 | > loss_text_ce: 0.024743234738707542 (0.03304581782374605) | > loss_mel_ce: 1.6588953733444214 (nan) | > loss: 0.006681106053292751 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13666433372558717) | > loader_time: 0.0045 (0.004690827574485363)  --> TIME: 2025-02-25 20:46:43 -- STEP: 15650/36938 -- GLOBAL_STEP: 939100 | > loss_text_ce: 0.021145541220903397 (0.03304428528910987) | > loss_mel_ce: 2.5265183448791504 (nan) | > loss: 0.010109778493642807 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13666488877500615) | > loader_time: 0.0045 (0.004690719184022337)  --> TIME: 2025-02-25 20:46:53 -- STEP: 15700/36938 -- GLOBAL_STEP: 939150 | > loss_text_ce: 0.037461671978235245 (0.033042447804502786) | > loss_mel_ce: 2.926420211791992 (nan) | > loss: 0.0117614371702075 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.13665573337275538) | > loader_time: 0.0047 (0.0046905459567999136)  --> TIME: 2025-02-25 20:47:02 -- STEP: 15750/36938 -- GLOBAL_STEP: 939200 | > loss_text_ce: 0.025744739919900894 (0.033041805575054574) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13666838703458298) | > loader_time: 0.0045 (0.004690640812828403)  --> TIME: 2025-02-25 20:47:12 -- STEP: 15800/36938 -- GLOBAL_STEP: 939250 | > loss_text_ce: 0.026689909398555756 (0.033041423755827366) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13667704459987134) | > loader_time: 0.0045 (0.004690539233292181)  --> TIME: 2025-02-25 20:47:21 -- STEP: 15850/36938 -- GLOBAL_STEP: 939300 | > loss_text_ce: 0.028621673583984375 (0.03304562574728409) | > loss_mel_ce: 2.5747642517089844 (nan) | > loss: 0.010330896824598312 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13668285762474014) | > loader_time: 0.0053 (0.004690582007639768)  --> TIME: 2025-02-25 20:47:30 -- STEP: 15900/36938 -- GLOBAL_STEP: 939350 | > loss_text_ce: 0.04184050112962723 (0.033045951499246176) | > loss_mel_ce: 2.054950475692749 (nan) | > loss: 0.008320599794387817 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13668693029655626) | > loader_time: 0.0045 (0.00469045556566248)  --> TIME: 2025-02-25 20:47:40 -- STEP: 15950/36938 -- GLOBAL_STEP: 939400 | > loss_text_ce: 0.031125275418162346 (0.03304781421096419) | > loss_mel_ce: 3.659839153289795 (nan) | > loss: 0.014646684750914574 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.13669069859675131) | > loader_time: 0.0048 (0.004690291141641527)  --> TIME: 2025-02-25 20:47:49 -- STEP: 16000/36938 -- GLOBAL_STEP: 939450 | > loss_text_ce: 0.03587060794234276 (0.0330490258160278) | > loss_mel_ce: 1.19219970703125 (nan) | > loss: 0.004873294848948717 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13667227062582996) | > loader_time: 0.0047 (0.004689945936202979)  --> TIME: 2025-02-25 20:47:58 -- STEP: 16050/36938 -- GLOBAL_STEP: 939500 | > loss_text_ce: 0.029054908081889153 (0.03304954543048234) | > loss_mel_ce: 2.839263916015625 (nan) | > loss: 0.011382218450307846 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.1366729633978968) | > loader_time: 0.0046 (0.004689696570423138)  --> TIME: 2025-02-25 20:48:07 -- STEP: 16100/36938 -- GLOBAL_STEP: 939550 | > loss_text_ce: 0.03198305144906044 (0.03305101401340311) | > loss_mel_ce: 2.208969831466675 (nan) | > loss: 0.008892671205103397 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13667900729623667) | > loader_time: 0.0046 (0.004689458305050844)  --> TIME: 2025-02-25 20:48:17 -- STEP: 16150/36938 -- GLOBAL_STEP: 939600 | > loss_text_ce: 0.0330098457634449 (0.03304952925964938) | > loss_mel_ce: 1.807013988494873 (nan) | > loss: 0.007301682606339455 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.1366678137380644) | > loader_time: 0.0046 (0.004689457630594425)  --> TIME: 2025-02-25 20:48:26 -- STEP: 16200/36938 -- GLOBAL_STEP: 939650 | > loss_text_ce: 0.02520526759326458 (0.03305054773254598) | > loss_mel_ce: 1.9006574153900146 (nan) | > loss: 0.007642312441021204 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13666165091373328) | > loader_time: 0.0045 (0.004689733687742234)  --> TIME: 2025-02-25 20:48:35 -- STEP: 16250/36938 -- GLOBAL_STEP: 939700 | > loss_text_ce: 0.024056434631347656 (0.03305383367435269) | > loss_mel_ce: 2.40203857421875 (nan) | > loss: 0.00962736178189516 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.1366571909977842) | > loader_time: 0.0054 (0.004689729499816871)  --> TIME: 2025-02-25 20:48:44 -- STEP: 16300/36938 -- GLOBAL_STEP: 939750 | > loss_text_ce: 0.0332452617585659 (0.03305194275762202) | > loss_mel_ce: 1.4840832948684692 (nan) | > loss: 0.006021145265549421 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13664073824151188) | > loader_time: 0.0045 (0.0046896323104577324)  --> TIME: 2025-02-25 20:48:53 -- STEP: 16350/36938 -- GLOBAL_STEP: 939800 | > loss_text_ce: 0.025535114109516144 (0.033052352686449446) | > loss_mel_ce: 3.0303280353546143 (nan) | > loss: 0.012126442044973373 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.13664182905028016) | > loader_time: 0.0048 (0.004689511523698789)  --> TIME: 2025-02-25 20:49:03 -- STEP: 16400/36938 -- GLOBAL_STEP: 939850 | > loss_text_ce: 0.030954943969845772 (0.0330499809406832) | > loss_mel_ce: 2.9865825176239014 (nan) | > loss: 0.011974355205893517 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13664328998181904) | > loader_time: 0.0045 (0.004689508167708769)  --> TIME: 2025-02-25 20:49:12 -- STEP: 16450/36938 -- GLOBAL_STEP: 939900 | > loss_text_ce: 0.03137120231986046 (0.03304773669395473) | > loss_mel_ce: 2.8475637435913086 (nan) | > loss: 0.011424344964325428 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13665741956705044) | > loader_time: 0.0047 (0.004689372810549263)  --> TIME: 2025-02-25 20:49:22 -- STEP: 16500/36938 -- GLOBAL_STEP: 939950 | > loss_text_ce: 0.036293745040893555 (0.033047668681225786) | > loss_mel_ce: 2.932337760925293 (nan) | > loss: 0.011780284345149994 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13665140266129489) | > loader_time: 0.0045 (0.004689022772239892)  --> TIME: 2025-02-25 20:49:31 -- STEP: 16550/36938 -- GLOBAL_STEP: 940000 | > loss_text_ce: 0.03336803987622261 (0.033050559525718295) | > loss_mel_ce: 2.548957586288452 (nan) | > loss: 0.010247324593365192 (nan) | > current_lr: 5e-06 | > step_time: 0.1667 (0.13664555134730064) | > loader_time: 0.0046 (0.004688811258967166) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_940000.pth  --> TIME: 2025-02-25 20:49:46 -- STEP: 16600/36938 -- GLOBAL_STEP: 940050 | > loss_text_ce: 0.032907553017139435 (0.033050970367032334) | > loss_mel_ce: 3.1249005794525146 (nan) | > loss: 0.012530985288321972 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13664075594350544) | > loader_time: 0.0046 (0.00468896659023788)  --> TIME: 2025-02-25 20:49:56 -- STEP: 16650/36938 -- GLOBAL_STEP: 940100 | > loss_text_ce: 0.031128568574786186 (0.03305339108839298) | > loss_mel_ce: 2.7921128273010254 (nan) | > loss: 0.011203340254724026 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1366398568912314) | > loader_time: 0.0046 (0.004688811416740508)  --> TIME: 2025-02-25 20:50:05 -- STEP: 16700/36938 -- GLOBAL_STEP: 940150 | > loss_text_ce: 0.027182206511497498 (0.033051489658185086) | > loss_mel_ce: 3.060776710510254 (nan) | > loss: 0.012253805063664913 (nan) | > current_lr: 5e-06 | > step_time: 0.1624 (0.13663894764677523) | > loader_time: 0.0047 (0.004688798781640491)  --> TIME: 2025-02-25 20:50:14 -- STEP: 16750/36938 -- GLOBAL_STEP: 940200 | > loss_text_ce: 0.035055916756391525 (0.03305074756136562) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13663167559210965) | > loader_time: 0.0045 (0.004688839129547552)  --> TIME: 2025-02-25 20:50:23 -- STEP: 16800/36938 -- GLOBAL_STEP: 940250 | > loss_text_ce: 0.04875205084681511 (0.03304812374702147) | > loss_mel_ce: 2.7984461784362793 (nan) | > loss: 0.011298405937850475 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.1366366732830096) | > loader_time: 0.0045 (0.004688811656974588)  --> TIME: 2025-02-25 20:50:33 -- STEP: 16850/36938 -- GLOBAL_STEP: 940300 | > loss_text_ce: 0.029522662982344627 (0.03304767395510631) | > loss_mel_ce: 3.2976107597351074 (nan) | > loss: 0.013202911242842674 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.1366377851871074) | > loader_time: 0.0048 (0.004688727622216088)  --> TIME: 2025-02-25 20:50:42 -- STEP: 16900/36938 -- GLOBAL_STEP: 940350 | > loss_text_ce: 0.03277299925684929 (0.03304330134406093) | > loss_mel_ce: 0.1457148939371109 (nan) | > loss: 0.0007082853699102998 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13664023919923793) | > loader_time: 0.0047 (0.004688582942330604)  --> TIME: 2025-02-25 20:50:52 -- STEP: 16950/36938 -- GLOBAL_STEP: 940400 | > loss_text_ce: 0.037267472594976425 (0.033042029801008684) | > loss_mel_ce: 2.063793182373047 (nan) | > loss: 0.008337542414665222 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.1366530026939417) | > loader_time: 0.0046 (0.00468862072204768)  --> TIME: 2025-02-25 20:51:01 -- STEP: 17000/36938 -- GLOBAL_STEP: 940450 | > loss_text_ce: 0.0339500829577446 (0.03304362808299395) | > loss_mel_ce: 2.7683045864105225 (nan) | > loss: 0.011120058596134186 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13664747698166815) | > loader_time: 0.0045 (0.004688580092261801)  --> TIME: 2025-02-25 20:51:10 -- STEP: 17050/36938 -- GLOBAL_STEP: 940500 | > loss_text_ce: 0.03714810684323311 (0.033041106605481534) | > loss_mel_ce: 3.0485386848449707 (nan) | > loss: 0.01224478892982006 (nan) | > current_lr: 5e-06 | > step_time: 0.1642 (0.13665981784943634) | > loader_time: 0.0054 (0.004688797598360553)  --> TIME: 2025-02-25 20:51:20 -- STEP: 17100/36938 -- GLOBAL_STEP: 940550 | > loss_text_ce: 0.03245997428894043 (0.033043606523206336) | > loss_mel_ce: 3.068560838699341 (nan) | > loss: 0.012305638752877712 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.1366838677027075) | > loader_time: 0.0046 (0.004688817520587749)  --> TIME: 2025-02-25 20:51:29 -- STEP: 17150/36938 -- GLOBAL_STEP: 940600 | > loss_text_ce: 0.03307431563735008 (0.03303976528694069) | > loss_mel_ce: 2.810823440551758 (nan) | > loss: 0.011285309679806232 (nan) | > current_lr: 5e-06 | > step_time: 0.1799 (0.13667481448838065) | > loader_time: 0.0049 (0.004688928593004392)  --> TIME: 2025-02-25 20:51:38 -- STEP: 17200/36938 -- GLOBAL_STEP: 940650 | > loss_text_ce: 0.026615802198648453 (0.03303832938071582) | > loss_mel_ce: 0.057107485830783844 (nan) | > loss: 0.0003322353004477918 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1366572164518892) | > loader_time: 0.0046 (0.004688783385032811)  --> TIME: 2025-02-25 20:51:48 -- STEP: 17250/36938 -- GLOBAL_STEP: 940700 | > loss_text_ce: 0.03822740167379379 (0.0330416386038927) | > loss_mel_ce: 3.3943030834198 (nan) | > loss: 0.013621153309941292 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.1366483694712324) | > loader_time: 0.0045 (0.0046887378277985755)  --> TIME: 2025-02-25 20:51:57 -- STEP: 17300/36938 -- GLOBAL_STEP: 940750 | > loss_text_ce: 0.03208596259355545 (0.033043035682226) | > loss_mel_ce: 2.885576009750366 (nan) | > loss: 0.011578024365007877 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13665015533480374) | > loader_time: 0.0045 (0.0046886827490922175)  --> TIME: 2025-02-25 20:52:06 -- STEP: 17350/36938 -- GLOBAL_STEP: 940800 | > loss_text_ce: 0.025580765679478645 (0.03304578273769138) | > loss_mel_ce: 2.1640026569366455 (nan) | > loss: 0.008688824251294136 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.1366496857167668) | > loader_time: 0.0046 (0.004688755071128804)  --> TIME: 2025-02-25 20:52:15 -- STEP: 17400/36938 -- GLOBAL_STEP: 940850 | > loss_text_ce: 0.035670362412929535 (0.033045728931674304) | > loss_mel_ce: 2.539668083190918 (nan) | > loss: 0.01021959725767374 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1366361269731635) | > loader_time: 0.0045 (0.004688492437888818)  --> TIME: 2025-02-25 20:52:24 -- STEP: 17450/36938 -- GLOBAL_STEP: 940900 | > loss_text_ce: 0.0353114977478981 (0.033042847307713605) | > loss_mel_ce: 2.893599271774292 (nan) | > loss: 0.011622662656009197 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.1366283999609746) | > loader_time: 0.0046 (0.004688231856228611)  --> TIME: 2025-02-25 20:52:34 -- STEP: 17500/36938 -- GLOBAL_STEP: 940950 | > loss_text_ce: 0.03598939627408981 (0.03304591364509273) | > loss_mel_ce: 3.2441577911376953 (nan) | > loss: 0.013016457669436932 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13664587557656457) | > loader_time: 0.0046 (0.004688150691986066)  --> TIME: 2025-02-25 20:52:43 -- STEP: 17550/36938 -- GLOBAL_STEP: 941000 | > loss_text_ce: 0.026641666889190674 (0.03304763460876744) | > loss_mel_ce: 2.849731683731079 (nan) | > loss: 0.011414180509746075 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13664507694733485) | > loader_time: 0.0046 (0.004688029221320067)  --> TIME: 2025-02-25 20:52:52 -- STEP: 17600/36938 -- GLOBAL_STEP: 941050 | > loss_text_ce: 0.031498637050390244 (0.03304851897981588) | > loss_mel_ce: 2.113635540008545 (nan) | > loss: 0.008512438274919987 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13663110845468296) | > loader_time: 0.0048 (0.004688050828196767)  --> TIME: 2025-02-25 20:53:02 -- STEP: 17650/36938 -- GLOBAL_STEP: 941100 | > loss_text_ce: 0.035415422171354294 (0.03304846966718287) | > loss_mel_ce: 3.090104579925537 (nan) | > loss: 0.012402857653796673 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13662702725224082) | > loader_time: 0.0046 (0.004688149295515092)  --> TIME: 2025-02-25 20:53:11 -- STEP: 17700/36938 -- GLOBAL_STEP: 941150 | > loss_text_ce: 0.01964455284178257 (0.033050137495484994) | > loss_mel_ce: 1.1029047966003418 (nan) | > loss: 0.004454561043530703 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13662344998559073) | > loader_time: 0.0046 (0.004688007265834468)  --> TIME: 2025-02-25 20:53:21 -- STEP: 17750/36938 -- GLOBAL_STEP: 941200 | > loss_text_ce: 0.025886479765176773 (0.0330504531748066) | > loss_mel_ce: 0.01606241799890995 (nan) | > loss: 0.00016646389849483967 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.1366310549588274) | > loader_time: 0.0047 (0.004687880784692881)  --> TIME: 2025-02-25 20:53:30 -- STEP: 17800/36938 -- GLOBAL_STEP: 941250 | > loss_text_ce: 0.03036593087017536 (0.03305104139731821) | > loss_mel_ce: 2.837663412094116 (nan) | > loss: 0.01138106919825077 (nan) | > current_lr: 5e-06 | > step_time: 0.1613 (0.1366282945134671) | > loader_time: 0.0048 (0.004687703754124997)  --> TIME: 2025-02-25 20:53:39 -- STEP: 17850/36938 -- GLOBAL_STEP: 941300 | > loss_text_ce: 0.03340616077184677 (0.033046447677271495) | > loss_mel_ce: 3.0034921169281006 (nan) | > loss: 0.01205118466168642 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13662256355713107) | > loader_time: 0.0046 (0.004687511460119919)  --> TIME: 2025-02-25 20:53:49 -- STEP: 17900/36938 -- GLOBAL_STEP: 941350 | > loss_text_ce: 0.030903292819857597 (0.0330454088777156) | > loss_mel_ce: 2.843517303466797 (nan) | > loss: 0.011406431905925274 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.136628149594675) | > loader_time: 0.0045 (0.004687323250584075)  --> TIME: 2025-02-25 20:53:58 -- STEP: 17950/36938 -- GLOBAL_STEP: 941400 | > loss_text_ce: 0.0394158698618412 (0.03304603048109928) | > loss_mel_ce: 3.2990126609802246 (nan) | > loss: 0.013247733004391193 (nan) | > current_lr: 5e-06 | > step_time: 0.1624 (0.13663575062180672) | > loader_time: 0.0047 (0.004687172762198677)  --> TIME: 2025-02-25 20:54:07 -- STEP: 18000/36938 -- GLOBAL_STEP: 941450 | > loss_text_ce: 0.03719918057322502 (0.033047598844342135) | > loss_mel_ce: 2.9874565601348877 (nan) | > loss: 0.012002603150904179 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.1366384832329224) | > loader_time: 0.0046 (0.004687021745575773)  --> TIME: 2025-02-25 20:54:17 -- STEP: 18050/36938 -- GLOBAL_STEP: 941500 | > loss_text_ce: 0.04366510733962059 (0.033047044507364336) | > loss_mel_ce: 0.01654052734375 (nan) | > loss: 0.00023891126329544932 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13663864785614488) | > loader_time: 0.0045 (0.004686882859121703)  --> TIME: 2025-02-25 20:54:26 -- STEP: 18100/36938 -- GLOBAL_STEP: 941550 | > loss_text_ce: 0.02183661237359047 (0.03304257948862273) | > loss_mel_ce: 2.64453125 (nan) | > loss: 0.010580824688076973 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.136639662236799) | > loader_time: 0.0045 (0.0046867077653579)  --> TIME: 2025-02-25 20:54:35 -- STEP: 18150/36938 -- GLOBAL_STEP: 941600 | > loss_text_ce: 0.037420958280563354 (0.033040173019413116) | > loss_mel_ce: 1.6283293962478638 (nan) | > loss: 0.006610121112316847 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.1366335695440124) | > loader_time: 0.0045 (0.004686498103391344)  --> TIME: 2025-02-25 20:54:45 -- STEP: 18200/36938 -- GLOBAL_STEP: 941650 | > loss_text_ce: 0.03678717836737633 (0.033042005777389465) | > loss_mel_ce: 2.5277609825134277 (nan) | > loss: 0.010176779702305794 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13663141170700882) | > loader_time: 0.0049 (0.004686363332874148)  --> TIME: 2025-02-25 20:54:54 -- STEP: 18250/36938 -- GLOBAL_STEP: 941700 | > loss_text_ce: 0.01572342775762081 (0.033043551872190706) | > loss_mel_ce: 2.2433104515075684 (nan) | > loss: 0.008964421227574348 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1366328758474905) | > loader_time: 0.0047 (0.004686403718713186)  --> TIME: 2025-02-25 20:55:03 -- STEP: 18300/36938 -- GLOBAL_STEP: 941750 | > loss_text_ce: 0.03532085195183754 (0.03304549576007212) | > loss_mel_ce: 3.1782937049865723 (nan) | > loss: 0.012752438895404339 (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.13663662253833259) | > loader_time: 0.0048 (0.004686273890114847)  --> TIME: 2025-02-25 20:55:13 -- STEP: 18350/36938 -- GLOBAL_STEP: 941800 | > loss_text_ce: 0.027189061045646667 (0.03304542151439594) | > loss_mel_ce: 3.2706375122070312 (nan) | > loss: 0.013086614198982716 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13663548382491494) | > loader_time: 0.005 (0.004686260106453101)  --> TIME: 2025-02-25 20:55:22 -- STEP: 18400/36938 -- GLOBAL_STEP: 941850 | > loss_text_ce: 0.02739817649126053 (0.033043844421626736) | > loss_mel_ce: 2.4706461429595947 (nan) | > loss: 0.009912874549627304 (nan) | > current_lr: 5e-06 | > step_time: 0.1591 (0.13663311141988557) | > loader_time: 0.0046 (0.004686083210551191)  --> TIME: 2025-02-25 20:55:31 -- STEP: 18450/36938 -- GLOBAL_STEP: 941900 | > loss_text_ce: 0.04148739576339722 (0.03304184232013106) | > loss_mel_ce: 2.823798656463623 (nan) | > loss: 0.011370183899998665 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13663492300969204) | > loader_time: 0.0046 (0.004685938726595717)  --> TIME: 2025-02-25 20:55:41 -- STEP: 18500/36938 -- GLOBAL_STEP: 941950 | > loss_text_ce: 0.021890342235565186 (0.033041894352264636) | > loss_mel_ce: 1.6940758228302002 (nan) | > loss: 0.006809390150010586 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1366416233036974) | > loader_time: 0.0045 (0.004685839498365217)  --> TIME: 2025-02-25 20:55:50 -- STEP: 18550/36938 -- GLOBAL_STEP: 942000 | > loss_text_ce: 0.02614155039191246 (0.033038910750319485) | > loss_mel_ce: 0.943453311920166 (nan) | > loss: 0.003847598796710372 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1366453151754295) | > loader_time: 0.0045 (0.004685715408016699)  --> TIME: 2025-02-25 20:55:59 -- STEP: 18600/36938 -- GLOBAL_STEP: 942050 | > loss_text_ce: 0.03150917962193489 (0.033042508401126575) | > loss_mel_ce: 3.058889389038086 (nan) | > loss: 0.012263487093150616 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13664183032128158) | > loader_time: 0.0046 (0.004685600906290003)  --> TIME: 2025-02-25 20:56:09 -- STEP: 18650/36938 -- GLOBAL_STEP: 942100 | > loss_text_ce: 0.0307094044983387 (0.03304543011789318) | > loss_mel_ce: 2.488973617553711 (nan) | > loss: 0.009998743422329426 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13663997248733994) | > loader_time: 0.0046 (0.004685742298975039)  --> TIME: 2025-02-25 20:56:18 -- STEP: 18700/36938 -- GLOBAL_STEP: 942150 | > loss_text_ce: 0.028433989733457565 (0.03304101256474345) | > loss_mel_ce: 2.788403034210205 (nan) | > loss: 0.011177925392985344 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1366510277635913) | > loader_time: 0.0048 (0.004685825523845624)  --> TIME: 2025-02-25 20:56:28 -- STEP: 18750/36938 -- GLOBAL_STEP: 942200 | > loss_text_ce: 0.025080488994717598 (0.033040426022509436) | > loss_mel_ce: 2.0363473892211914 (nan) | > loss: 0.008180269971489906 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.1366565766143802) | > loader_time: 0.0046 (0.0046857974751790075)  --> TIME: 2025-02-25 20:56:37 -- STEP: 18800/36938 -- GLOBAL_STEP: 942250 | > loss_text_ce: 0.031070923432707787 (0.033040696010884954) | > loss_mel_ce: 2.685451030731201 (nan) | > loss: 0.01077985018491745 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13664904321761875) | > loader_time: 0.0046 (0.004685600184379712)  --> TIME: 2025-02-25 20:56:46 -- STEP: 18850/36938 -- GLOBAL_STEP: 942300 | > loss_text_ce: 0.02981315553188324 (0.03303888915007813) | > loss_mel_ce: 2.1610896587371826 (nan) | > loss: 0.00869405921548605 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13664813470461046) | > loader_time: 0.0068 (0.0046859866807568)  --> TIME: 2025-02-25 20:56:56 -- STEP: 18900/36938 -- GLOBAL_STEP: 942350 | > loss_text_ce: 0.03973388671875 (0.03304064563120734) | > loss_mel_ce: 0.0102996826171875 (nan) | > loss: 0.00019854592392221093 (nan) | > current_lr: 5e-06 | > step_time: 0.1072 (0.1366583491759329) | > loader_time: 0.0045 (0.004686247583419528)  --> TIME: 2025-02-25 20:57:05 -- STEP: 18950/36938 -- GLOBAL_STEP: 942400 | > loss_text_ce: 0.033478617668151855 (0.03304326927102586) | > loss_mel_ce: 2.4979758262634277 (nan) | > loss: 0.01004545483738184 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.1366591452294104) | > loader_time: 0.0046 (0.004686086762863568)  --> TIME: 2025-02-25 20:57:15 -- STEP: 19000/36938 -- GLOBAL_STEP: 942450 | > loss_text_ce: 0.03434576094150543 (0.033039641376927636) | > loss_mel_ce: 2.9656426906585693 (nan) | > loss: 0.011904717423021793 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13665762626497396) | > loader_time: 0.0047 (0.00468618421805529)  --> TIME: 2025-02-25 20:57:24 -- STEP: 19050/36938 -- GLOBAL_STEP: 942500 | > loss_text_ce: 0.03555427864193916 (0.03304287797201817) | > loss_mel_ce: 2.7323739528656006 (nan) | > loss: 0.010983842425048351 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.1366482161849817) | > loader_time: 0.0045 (0.004686047974533893)  --> TIME: 2025-02-25 20:57:33 -- STEP: 19100/36938 -- GLOBAL_STEP: 942550 | > loss_text_ce: 0.029425475746393204 (0.03304285269586523) | > loss_mel_ce: 3.0777945518493652 (nan) | > loss: 0.012330238707363605 (nan) | > current_lr: 5e-06 | > step_time: 0.1783 (0.13664052088223227) | > loader_time: 0.0048 (0.004685843128184344)  --> TIME: 2025-02-25 20:57:42 -- STEP: 19150/36938 -- GLOBAL_STEP: 942600 | > loss_text_ce: 0.025744372978806496 (0.033042441714535256) | > loss_mel_ce: 2.747844934463501 (nan) | > loss: 0.011006307788193226 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13662872179992533) | > loader_time: 0.0049 (0.004685739437531523)  --> TIME: 2025-02-25 20:57:51 -- STEP: 19200/36938 -- GLOBAL_STEP: 942650 | > loss_text_ce: 0.02739814482629299 (0.03304245810960582) | > loss_mel_ce: 2.908626079559326 (nan) | > loss: 0.011650890111923218 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.13662479265282568) | > loader_time: 0.0048 (0.004685568089286456)  --> TIME: 2025-02-25 20:58:01 -- STEP: 19250/36938 -- GLOBAL_STEP: 942700 | > loss_text_ce: 0.03028831258416176 (0.033045700441561805) | > loss_mel_ce: 2.2249808311462402 (nan) | > loss: 0.008949480950832367 (nan) | > current_lr: 5e-06 | > step_time: 0.1271 (0.13662774785772563) | > loader_time: 0.0045 (0.004685469020496685)  --> TIME: 2025-02-25 20:58:10 -- STEP: 19300/36938 -- GLOBAL_STEP: 942750 | > loss_text_ce: 0.04226452112197876 (0.033048599955089866) | > loss_mel_ce: 2.402794599533081 (nan) | > loss: 0.009702615439891815 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13663837357506253) | > loader_time: 0.0046 (0.004685548500693495)  --> TIME: 2025-02-25 20:58:19 -- STEP: 19350/36938 -- GLOBAL_STEP: 942800 | > loss_text_ce: 0.052006401121616364 (0.03304600804334219) | > loss_mel_ce: 1.14312744140625 (nan) | > loss: 0.00474259490147233 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.1366346162973449) | > loader_time: 0.0045 (0.0046857508824468845)  --> TIME: 2025-02-25 20:58:28 -- STEP: 19400/36938 -- GLOBAL_STEP: 942850 | > loss_text_ce: 0.032622821629047394 (0.03305118401412407) | > loss_mel_ce: 2.470827341079712 (nan) | > loss: 0.009934326633810997 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13662300956617926) | > loader_time: 0.0045 (0.00468575867180968)  --> TIME: 2025-02-25 20:58:38 -- STEP: 19450/36938 -- GLOBAL_STEP: 942900 | > loss_text_ce: 0.03006715700030327 (0.033050118197871015) | > loss_mel_ce: 2.8605964183807373 (nan) | > loss: 0.011470887809991837 (nan) | > current_lr: 5e-06 | > step_time: 0.1992 (0.13661589399401514) | > loader_time: 0.0049 (0.004685754702452793)  --> TIME: 2025-02-25 20:58:47 -- STEP: 19500/36938 -- GLOBAL_STEP: 942950 | > loss_text_ce: 0.02992442063987255 (0.03304874751857713) | > loss_mel_ce: 2.2005996704101562 (nan) | > loss: 0.00885128602385521 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13661398729911237) | > loader_time: 0.0046 (0.004685669165391153)  --> TIME: 2025-02-25 20:58:56 -- STEP: 19550/36938 -- GLOBAL_STEP: 943000 | > loss_text_ce: 0.02916763722896576 (0.03304822316910098) | > loss_mel_ce: 2.380995750427246 (nan) | > loss: 0.009564140811562538 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13661488052524276) | > loader_time: 0.0045 (0.004685611115087296)  --> TIME: 2025-02-25 20:59:06 -- STEP: 19600/36938 -- GLOBAL_STEP: 943050 | > loss_text_ce: 0.04032787308096886 (0.03305177344282471) | > loss_mel_ce: 2.7636353969573975 (nan) | > loss: 0.011126838624477386 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1366128522036028) | > loader_time: 0.0047 (0.004685526587525162)  --> TIME: 2025-02-25 20:59:15 -- STEP: 19650/36938 -- GLOBAL_STEP: 943100 | > loss_text_ce: 0.037981707602739334 (0.03305265017780365) | > loss_mel_ce: 3.133636474609375 (nan) | > loss: 0.01258578710258007 (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.1366102572797819) | > loader_time: 0.0047 (0.004685542298333907)  --> TIME: 2025-02-25 20:59:24 -- STEP: 19700/36938 -- GLOBAL_STEP: 943150 | > loss_text_ce: 0.0252334326505661 (0.03304742767553523) | > loss_mel_ce: 2.5589516162872314 (nan) | > loss: 0.01025470346212387 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13659577663779865) | > loader_time: 0.0047 (0.004685442798633833)  --> TIME: 2025-02-25 20:59:33 -- STEP: 19750/36938 -- GLOBAL_STEP: 943200 | > loss_text_ce: 0.035622015595436096 (0.03304905193570174) | > loss_mel_ce: 2.274045705795288 (nan) | > loss: 0.009165349416434765 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13659376327900954) | > loader_time: 0.0046 (0.004685415062723252)  --> TIME: 2025-02-25 20:59:43 -- STEP: 19800/36938 -- GLOBAL_STEP: 943250 | > loss_text_ce: 0.03198370337486267 (0.03304530614382571) | > loss_mel_ce: 2.2444465160369873 (nan) | > loss: 0.009033453650772572 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13659427524817125) | > loader_time: 0.0046 (0.0046853674421406455)  --> TIME: 2025-02-25 20:59:52 -- STEP: 19850/36938 -- GLOBAL_STEP: 943300 | > loss_text_ce: 0.023952752351760864 (0.03304254699862001) | > loss_mel_ce: 2.8689255714416504 (nan) | > loss: 0.011479676701128483 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13659638147810557) | > loader_time: 0.0046 (0.004685557290949178)  --> TIME: 2025-02-25 21:00:01 -- STEP: 19900/36938 -- GLOBAL_STEP: 943350 | > loss_text_ce: 0.04058798775076866 (0.03304571731331912) | > loss_mel_ce: 2.333533525466919 (nan) | > loss: 0.009421117603778839 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13657608937977564) | > loader_time: 0.0045 (0.004685427663314248)  --> TIME: 2025-02-25 21:00:10 -- STEP: 19950/36938 -- GLOBAL_STEP: 943400 | > loss_text_ce: 0.044163383543491364 (0.03305139884509046) | > loss_mel_ce: 2.287403106689453 (nan) | > loss: 0.009252249263226986 (nan) | > current_lr: 5e-06 | > step_time: 0.1121 (0.13657888478205035) | > loader_time: 0.0047 (0.004685501634028885)  --> TIME: 2025-02-25 21:00:19 -- STEP: 20000/36938 -- GLOBAL_STEP: 943450 | > loss_text_ce: 0.029859060421586037 (0.03304946896086888) | > loss_mel_ce: 2.9576363563537598 (nan) | > loss: 0.011855141259729862 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13657467938661597) | > loader_time: 0.0045 (0.00468536146879193)  --> TIME: 2025-02-25 21:00:29 -- STEP: 20050/36938 -- GLOBAL_STEP: 943500 | > loss_text_ce: 0.015590023249387741 (0.033049136295129414) | > loss_mel_ce: 2.9923148155212402 (nan) | > loss: 0.011936130933463573 (nan) | > current_lr: 5e-06 | > step_time: 0.1261 (0.13659322252297362) | > loader_time: 0.0048 (0.004685338500729194)  --> TIME: 2025-02-25 21:00:38 -- STEP: 20100/36938 -- GLOBAL_STEP: 943550 | > loss_text_ce: 0.03964885324239731 (0.033049773379849656) | > loss_mel_ce: 3.017606019973755 (nan) | > loss: 0.012131963856518269 (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.13658750024007957) | > loader_time: 0.0046 (0.004685138458043157)  --> TIME: 2025-02-25 21:00:47 -- STEP: 20150/36938 -- GLOBAL_STEP: 943600 | > loss_text_ce: 0.036362674087285995 (0.03304873789013538) | > loss_mel_ce: 3.1940765380859375 (nan) | > loss: 0.012819203548133373 (nan) | > current_lr: 5e-06 | > step_time: 0.1513 (0.13658569636475104) | > loader_time: 0.0046 (0.004684976407375258)  --> TIME: 2025-02-25 21:00:57 -- STEP: 20200/36938 -- GLOBAL_STEP: 943650 | > loss_text_ce: 0.026737358421087265 (0.03304694815163044) | > loss_mel_ce: 2.03808331489563 (nan) | > loss: 0.008193734101951122 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13659270836575235) | > loader_time: 0.0046 (0.004684860930584415)  --> TIME: 2025-02-25 21:01:07 -- STEP: 20250/36938 -- GLOBAL_STEP: 943700 | > loss_text_ce: 0.022823287174105644 (0.033047788075849516) | > loss_mel_ce: 2.527109146118164 (nan) | > loss: 0.01011878065764904 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13660176426687387) | > loader_time: 0.0046 (0.004684738006120815)  --> TIME: 2025-02-25 21:01:16 -- STEP: 20300/36938 -- GLOBAL_STEP: 943750 | > loss_text_ce: 0.03245587646961212 (0.033046948408519614) | > loss_mel_ce: 0.7396657466888428 (nan) | > loss: 0.003063974902033806 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13660563351485558) | > loader_time: 0.0045 (0.004684579572066851)  --> TIME: 2025-02-25 21:01:25 -- STEP: 20350/36938 -- GLOBAL_STEP: 943800 | > loss_text_ce: 0.034321289509534836 (0.03304852960647244) | > loss_mel_ce: 2.7844789028167725 (nan) | > loss: 0.011185715906322002 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13659207656869746) | > loader_time: 0.0045 (0.00468430405459764)  --> TIME: 2025-02-25 21:01:34 -- STEP: 20400/36938 -- GLOBAL_STEP: 943850 | > loss_text_ce: 0.03343576937913895 (0.03304514797835803) | > loss_mel_ce: 2.6469311714172363 (nan) | > loss: 0.010636377148330212 (nan) | > current_lr: 5e-06 | > step_time: 0.1625 (0.13659752533716338) | > loader_time: 0.0046 (0.004684125033079377)  --> TIME: 2025-02-25 21:01:44 -- STEP: 20450/36938 -- GLOBAL_STEP: 943900 | > loss_text_ce: 0.031183911487460136 (0.03303768366577491) | > loss_mel_ce: 2.668245792388916 (nan) | > loss: 0.010712023824453354 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13660178624038896) | > loader_time: 0.0046 (0.004683968723548913)  --> TIME: 2025-02-25 21:01:53 -- STEP: 20500/36938 -- GLOBAL_STEP: 943950 | > loss_text_ce: 0.03473683446645737 (0.03303774735215661) | > loss_mel_ce: 2.6518936157226562 (nan) | > loss: 0.01066123228520155 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13658803219911544) | > loader_time: 0.0047 (0.004683683872222867)  --> TIME: 2025-02-25 21:02:02 -- STEP: 20550/36938 -- GLOBAL_STEP: 944000 | > loss_text_ce: 0.03357601910829544 (0.033038685180704276) | > loss_mel_ce: 1.6942991018295288 (nan) | > loss: 0.006856647785753012 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13657950719197604) | > loader_time: 0.0046 (0.00468346883490715)  --> TIME: 2025-02-25 21:02:11 -- STEP: 20600/36938 -- GLOBAL_STEP: 944050 | > loss_text_ce: 0.032800741493701935 (0.03303661432996523) | > loss_mel_ce: 2.4048805236816406 (nan) | > loss: 0.009673338383436203 (nan) | > current_lr: 5e-06 | > step_time: 0.1517 (0.13657180933119034) | > loader_time: 0.0045 (0.0046833554980824045)  --> TIME: 2025-02-25 21:02:21 -- STEP: 20650/36938 -- GLOBAL_STEP: 944100 | > loss_text_ce: 0.04070727899670601 (0.03303567045765106) | > loss_mel_ce: 2.7982354164123535 (nan) | > loss: 0.011265646666288376 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.1365943854311189) | > loader_time: 0.0046 (0.004683318496039044)  --> TIME: 2025-02-25 21:02:31 -- STEP: 20700/36938 -- GLOBAL_STEP: 944150 | > loss_text_ce: 0.02496054209768772 (0.033031822012552656) | > loss_mel_ce: 2.989555835723877 (nan) | > loss: 0.011962367221713066 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13661155206569744) | > loader_time: 0.0044 (0.0046832461172831566)  --> TIME: 2025-02-25 21:02:40 -- STEP: 20750/36938 -- GLOBAL_STEP: 944200 | > loss_text_ce: 0.04418808966875076 (0.03303053637614155) | > loss_mel_ce: 0.5250754356384277 (nan) | > loss: 0.002258982276543975 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.1366114030056691) | > loader_time: 0.0045 (0.004683271293180513)  --> TIME: 2025-02-25 21:02:49 -- STEP: 20800/36938 -- GLOBAL_STEP: 944250 | > loss_text_ce: 0.036821652203798294 (0.03303103342618712) | > loss_mel_ce: 2.315749406814575 (nan) | > loss: 0.009335599839687347 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13660420244702942) | > loader_time: 0.0046 (0.004683208958460711)  --> TIME: 2025-02-25 21:02:58 -- STEP: 20850/36938 -- GLOBAL_STEP: 944300 | > loss_text_ce: 0.036377646028995514 (0.03303099981951334) | > loss_mel_ce: 2.324169874191284 (nan) | > loss: 0.009367252700030804 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13660741626501674) | > loader_time: 0.0052 (0.004683195532654665)  --> TIME: 2025-02-25 21:03:08 -- STEP: 20900/36938 -- GLOBAL_STEP: 944350 | > loss_text_ce: 0.03169579058885574 (0.0330300074188553) | > loss_mel_ce: 2.6436731815338135 (nan) | > loss: 0.010616544634103775 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13661486509314) | > loader_time: 0.0045 (0.004683306525198444)  --> TIME: 2025-02-25 21:03:17 -- STEP: 20950/36938 -- GLOBAL_STEP: 944400 | > loss_text_ce: 0.030988389626145363 (0.033027027689102456) | > loss_mel_ce: 2.2914693355560303 (nan) | > loss: 0.00921610277146101 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13661864650562436) | > loader_time: 0.0047 (0.004683305426258437)  --> TIME: 2025-02-25 21:03:27 -- STEP: 21000/36938 -- GLOBAL_STEP: 944450 | > loss_text_ce: 0.02786252461373806 (0.03302521885747413) | > loss_mel_ce: 2.776737689971924 (nan) | > loss: 0.011129366233944893 (nan) | > current_lr: 5e-06 | > step_time: 0.1591 (0.1366164483115787) | > loader_time: 0.0046 (0.004683248088473339)  --> TIME: 2025-02-25 21:03:36 -- STEP: 21050/36938 -- GLOBAL_STEP: 944500 | > loss_text_ce: 0.040511664003133774 (0.03303144473850147) | > loss_mel_ce: 2.529656410217285 (nan) | > loss: 0.010199080221354961 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.1366200651522204) | > loader_time: 0.0047 (0.004683461698953418)  --> TIME: 2025-02-25 21:03:46 -- STEP: 21100/36938 -- GLOBAL_STEP: 944550 | > loss_text_ce: 0.03202333301305771 (0.03302949072274467) | > loss_mel_ce: 2.9754819869995117 (nan) | > loss: 0.011934545822441578 (nan) | > current_lr: 5e-06 | > step_time: 0.1725 (0.1366318990833953) | > loader_time: 0.0048 (0.004683523935164297)  --> TIME: 2025-02-25 21:03:55 -- STEP: 21150/36938 -- GLOBAL_STEP: 944600 | > loss_text_ce: 0.03134705498814583 (0.033028077227448445) | > loss_mel_ce: 2.5094547271728516 (nan) | > loss: 0.010082547552883625 (nan) | > current_lr: 5e-06 | > step_time: 0.1687 (0.13662830955875133) | > loader_time: 0.0048 (0.004683534935574697)  --> TIME: 2025-02-25 21:04:04 -- STEP: 21200/36938 -- GLOBAL_STEP: 944650 | > loss_text_ce: 0.02650526538491249 (0.033027391655849093) | > loss_mel_ce: 3.023698329925537 (nan) | > loss: 0.012103983201086521 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13662733362530774) | > loader_time: 0.0054 (0.0046834316006246325)  --> TIME: 2025-02-25 21:04:14 -- STEP: 21250/36938 -- GLOBAL_STEP: 944700 | > loss_text_ce: 0.03211381286382675 (0.03302896795757081) | > loss_mel_ce: 2.9468939304351807 (nan) | > loss: 0.011821459978818893 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13662449907415058) | > loader_time: 0.0047 (0.0046838650647331405)  --> TIME: 2025-02-25 21:04:23 -- STEP: 21300/36938 -- GLOBAL_STEP: 944750 | > loss_text_ce: 0.034101452678442 (0.033030628291168235) | > loss_mel_ce: 3.2932140827178955 (nan) | > loss: 0.013203633949160576 (nan) | > current_lr: 5e-06 | > step_time: 0.1349 (0.13662282476962448) | > loader_time: 0.0046 (0.004684029856757907)  --> TIME: 2025-02-25 21:04:32 -- STEP: 21350/36938 -- GLOBAL_STEP: 944800 | > loss_text_ce: 0.034683432430028915 (0.03303312625048922) | > loss_mel_ce: 3.24933123588562 (nan) | > loss: 0.01303180493414402 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.1366220161786404) | > loader_time: 0.0047 (0.004683889784355043)  --> TIME: 2025-02-25 21:04:41 -- STEP: 21400/36938 -- GLOBAL_STEP: 944850 | > loss_text_ce: 0.03274921700358391 (0.03303296313701581) | > loss_mel_ce: 2.44417142868042 (nan) | > loss: 0.009829050861299038 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.13662509367844783) | > loader_time: 0.0048 (0.004683822036903563)  --> TIME: 2025-02-25 21:04:51 -- STEP: 21450/36938 -- GLOBAL_STEP: 944900 | > loss_text_ce: 0.026093212887644768 (0.033034918380961655) | > loss_mel_ce: 2.4950993061065674 (nan) | > loss: 0.010004732757806778 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13661591035224915) | > loader_time: 0.0047 (0.004683865867294601)  --> TIME: 2025-02-25 21:05:00 -- STEP: 21500/36938 -- GLOBAL_STEP: 944950 | > loss_text_ce: 0.03834270313382149 (0.0330377431195009) | > loss_mel_ce: 3.662627696990967 (nan) | > loss: 0.014686391688883305 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13662243003623448) | > loader_time: 0.0045 (0.0046839496590370035)  --> TIME: 2025-02-25 21:05:09 -- STEP: 21550/36938 -- GLOBAL_STEP: 945000 | > loss_text_ce: 0.02762478031218052 (0.03303643212545618) | > loss_mel_ce: 2.6511528491973877 (nan) | > loss: 0.01063007116317749 (nan) | > current_lr: 5e-06 | > step_time: 0.17 (0.13662087750268934) | > loader_time: 0.0047 (0.0046840348984968)  --> TIME: 2025-02-25 21:05:19 -- STEP: 21600/36938 -- GLOBAL_STEP: 945050 | > loss_text_ce: 0.04273022711277008 (0.033037811916394454) | > loss_mel_ce: 2.288538694381714 (nan) | > loss: 0.009251068346202374 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.13662685991437357) | > loader_time: 0.0046 (0.0046843280284492705)  --> TIME: 2025-02-25 21:05:28 -- STEP: 21650/36938 -- GLOBAL_STEP: 945100 | > loss_text_ce: 0.03353475034236908 (0.03303684392322344) | > loss_mel_ce: 2.9638519287109375 (nan) | > loss: 0.011894392780959606 (nan) | > current_lr: 5e-06 | > step_time: 0.1832 (0.13661977804285283) | > loader_time: 0.0049 (0.00468431901160891)  --> TIME: 2025-02-25 21:05:37 -- STEP: 21700/36938 -- GLOBAL_STEP: 945150 | > loss_text_ce: 0.02944221720099449 (0.033036512133132856) | > loss_mel_ce: 3.3841254711151123 (nan) | > loss: 0.01354590430855751 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13661540715925163) | > loader_time: 0.0047 (0.0046842685932387655)  --> TIME: 2025-02-25 21:05:47 -- STEP: 21750/36938 -- GLOBAL_STEP: 945200 | > loss_text_ce: 0.028485536575317383 (0.03303379095039546) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.1366172641227986) | > loader_time: 0.0044 (0.004684108207965688)  --> TIME: 2025-02-25 21:05:56 -- STEP: 21800/36938 -- GLOBAL_STEP: 945250 | > loss_text_ce: 0.03558725863695145 (0.033036701610298744) | > loss_mel_ce: 1.9494677782058716 (nan) | > loss: 0.00787720363587141 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.1366256560754339) | > loader_time: 0.0048 (0.0046840508924711425)  --> TIME: 2025-02-25 21:06:06 -- STEP: 21850/36938 -- GLOBAL_STEP: 945300 | > loss_text_ce: 0.038637127727270126 (0.033035335809235046) | > loss_mel_ce: 2.149574041366577 (nan) | > loss: 0.008683377876877785 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13663300636431192) | > loader_time: 0.0048 (0.004684283738823693)  --> TIME: 2025-02-25 21:06:15 -- STEP: 21900/36938 -- GLOBAL_STEP: 945350 | > loss_text_ce: 0.03146474063396454 (0.03303516409587361) | > loss_mel_ce: 2.3268330097198486 (nan) | > loss: 0.00935832504183054 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.1366342337817363) | > loader_time: 0.0054 (0.004684495207381547)  --> TIME: 2025-02-25 21:06:24 -- STEP: 21950/36938 -- GLOBAL_STEP: 945400 | > loss_text_ce: 0.015579348430037498 (0.03303217540847233) | > loss_mel_ce: 2.3713490962982178 (nan) | > loss: 0.009471938945353031 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13663741238839547) | > loader_time: 0.0045 (0.0046845921297007905)  --> TIME: 2025-02-25 21:06:34 -- STEP: 22000/36938 -- GLOBAL_STEP: 945450 | > loss_text_ce: 0.033264804631471634 (0.03303306939628543) | > loss_mel_ce: 3.2221574783325195 (nan) | > loss: 0.012918343767523766 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13663387104597965) | > loader_time: 0.0053 (0.004684634035283836)  --> TIME: 2025-02-25 21:06:43 -- STEP: 22050/36938 -- GLOBAL_STEP: 945500 | > loss_text_ce: 0.04487296938896179 (0.03303576740453126) | > loss_mel_ce: 2.391720771789551 (nan) | > loss: 0.00966902356594801 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13664494209549063) | > loader_time: 0.0047 (0.004684878585019589)  --> TIME: 2025-02-25 21:06:52 -- STEP: 22100/36938 -- GLOBAL_STEP: 945550 | > loss_text_ce: 0.032339904457330704 (0.03303515909439796) | > loss_mel_ce: 2.688173770904541 (nan) | > loss: 0.010795689187943935 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13664475093599893) | > loader_time: 0.0046 (0.004684798609617038)  --> TIME: 2025-02-25 21:07:02 -- STEP: 22150/36938 -- GLOBAL_STEP: 945600 | > loss_text_ce: 0.045280516147613525 (0.033037414626190795) | > loss_mel_ce: 2.639955759048462 (nan) | > loss: 0.010655700229108334 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13663612141835224) | > loader_time: 0.0045 (0.004684920461667547)  --> TIME: 2025-02-25 21:07:11 -- STEP: 22200/36938 -- GLOBAL_STEP: 945650 | > loss_text_ce: 0.03836745023727417 (0.03303784051274453) | > loss_mel_ce: 0.16563323140144348 (nan) | > loss: 0.0008095265366137028 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13662926238936338) | > loader_time: 0.0045 (0.00468483702556504)  --> TIME: 2025-02-25 21:07:20 -- STEP: 22250/36938 -- GLOBAL_STEP: 945700 | > loss_text_ce: 0.030978519469499588 (0.03303627134217032) | > loss_mel_ce: 0.24481201171875 (nan) | > loss: 0.0010944069363176823 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.1366286458004725) | > loader_time: 0.0047 (0.004684692564974983)  --> TIME: 2025-02-25 21:07:29 -- STEP: 22300/36938 -- GLOBAL_STEP: 945750 | > loss_text_ce: 0.02816329337656498 (0.0330349816585943) | > loss_mel_ce: 2.2876412868499756 (nan) | > loss: 0.009189700707793236 (nan) | > current_lr: 5e-06 | > step_time: 0.1609 (0.13662949646534922) | > loader_time: 0.0048 (0.004685036387678728)  --> TIME: 2025-02-25 21:07:39 -- STEP: 22350/36938 -- GLOBAL_STEP: 945800 | > loss_text_ce: 0.025617267936468124 (0.03303526322432337) | > loss_mel_ce: 2.9344661235809326 (nan) | > loss: 0.011746363714337349 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13664842571454805) | > loader_time: 0.0046 (0.004685010707351691)  --> TIME: 2025-02-25 21:07:48 -- STEP: 22400/36938 -- GLOBAL_STEP: 945850 | > loss_text_ce: 0.03459523245692253 (0.03303267086334252) | > loss_mel_ce: 2.54296875 (nan) | > loss: 0.010228428989648819 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13664532361286) | > loader_time: 0.0044 (0.004685095080307521)  --> TIME: 2025-02-25 21:07:58 -- STEP: 22450/36938 -- GLOBAL_STEP: 945900 | > loss_text_ce: 0.046813830733299255 (0.033030724091712274) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13664215206834388) | > loader_time: 0.0046 (0.004684914831063764)  --> TIME: 2025-02-25 21:08:07 -- STEP: 22500/36938 -- GLOBAL_STEP: 945950 | > loss_text_ce: 0.04051738232374191 (0.033031294293473734) | > loss_mel_ce: 3.008281707763672 (nan) | > loss: 0.012098409235477448 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13663197250366182) | > loader_time: 0.0052 (0.0046849430508083476)  --> TIME: 2025-02-25 21:08:16 -- STEP: 22550/36938 -- GLOBAL_STEP: 946000 | > loss_text_ce: 0.030153948813676834 (0.033030157719166515) | > loss_mel_ce: 2.8902359008789062 (nan) | > loss: 0.011588849127292633 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.13663680699342068) | > loader_time: 0.0048 (0.004684957844719364)  --> TIME: 2025-02-25 21:08:26 -- STEP: 22600/36938 -- GLOBAL_STEP: 946050 | > loss_text_ce: 0.0369345098733902 (0.03303000123909673) | > loss_mel_ce: 1.8965777158737183 (nan) | > loss: 0.007672667969018221 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.1366410619917165) | > loader_time: 0.0044 (0.00468510149854472)  --> TIME: 2025-02-25 21:08:35 -- STEP: 22650/36938 -- GLOBAL_STEP: 946100 | > loss_text_ce: 0.0315614677965641 (0.0330292287272569) | > loss_mel_ce: 3.138672351837158 (nan) | > loss: 0.012580293230712414 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.136642725904519) | > loader_time: 0.0048 (0.004685011004770011)  --> TIME: 2025-02-25 21:08:44 -- STEP: 22700/36938 -- GLOBAL_STEP: 946150 | > loss_text_ce: 0.0398525670170784 (0.03302728024366723) | > loss_mel_ce: 2.6984450817108154 (nan) | > loss: 0.010866261087357998 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.1366419233939717) | > loader_time: 0.0045 (0.004684842210509153)  --> TIME: 2025-02-25 21:08:54 -- STEP: 22750/36938 -- GLOBAL_STEP: 946200 | > loss_text_ce: 0.03123817779123783 (0.03302608136269798) | > loss_mel_ce: 2.010136127471924 (nan) | > loss: 0.008100692182779312 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13663828280731838) | > loader_time: 0.0048 (0.004684798701778846)  --> TIME: 2025-02-25 21:09:03 -- STEP: 22800/36938 -- GLOBAL_STEP: 946250 | > loss_text_ce: 0.037243232131004333 (0.033027801327264114) | > loss_mel_ce: 0.1579182893037796 (nan) | > loss: 0.0007744505419395864 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13664422979480306) | > loader_time: 0.0046 (0.004684746610490869)  --> TIME: 2025-02-25 21:09:12 -- STEP: 22850/36938 -- GLOBAL_STEP: 946300 | > loss_text_ce: 0.029985567554831505 (0.033027052345096425) | > loss_mel_ce: 2.663213014602661 (nan) | > loss: 0.010687297210097313 (nan) | > current_lr: 5e-06 | > step_time: 0.1739 (0.13664677319432728) | > loader_time: 0.0051 (0.004684978293120439)  --> TIME: 2025-02-25 21:09:22 -- STEP: 22900/36938 -- GLOBAL_STEP: 946350 | > loss_text_ce: 0.03672165423631668 (0.03302829317840024) | > loss_mel_ce: 0.0228271484375 (nan) | > loss: 0.00023630478244740516 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13666684032007023) | > loader_time: 0.0046 (0.0046851487243019015)  --> TIME: 2025-02-25 21:09:32 -- STEP: 22950/36938 -- GLOBAL_STEP: 946400 | > loss_text_ce: 0.021619820967316628 (0.03302684434003742) | > loss_mel_ce: 2.993062973022461 (nan) | > loss: 0.011963027529418468 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1366738725383811) | > loader_time: 0.0045 (0.0046852096337378375)  --> TIME: 2025-02-25 21:09:42 -- STEP: 23000/36938 -- GLOBAL_STEP: 946450 | > loss_text_ce: 0.03186170011758804 (0.033026884314178676) | > loss_mel_ce: 3.06732439994812 (nan) | > loss: 0.012298358604311943 (nan) | > current_lr: 5e-06 | > step_time: 0.1694 (0.13668634115094658) | > loader_time: 0.0049 (0.004685144030529492)  --> TIME: 2025-02-25 21:09:51 -- STEP: 23050/36938 -- GLOBAL_STEP: 946500 | > loss_text_ce: 0.038123439997434616 (0.03302702147378251) | > loss_mel_ce: 2.2936556339263916 (nan) | > loss: 0.009253092110157013 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1366843219641226) | > loader_time: 0.0046 (0.004685061448566797)  --> TIME: 2025-02-25 21:10:00 -- STEP: 23100/36938 -- GLOBAL_STEP: 946550 | > loss_text_ce: 0.030075356364250183 (0.0330244476167741) | > loss_mel_ce: 3.0728328227996826 (nan) | > loss: 0.01231312844902277 (nan) | > current_lr: 5e-06 | > step_time: 0.1793 (0.13668918452737594) | > loader_time: 0.0048 (0.004685032161283259)  --> TIME: 2025-02-25 21:10:10 -- STEP: 23150/36938 -- GLOBAL_STEP: 946600 | > loss_text_ce: 0.03605007007718086 (0.03302758018574688) | > loss_mel_ce: 1.4447212219238281 (nan) | > loss: 0.005876076873391867 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13669028251268886) | > loader_time: 0.0046 (0.004685001744049901)  --> TIME: 2025-02-25 21:10:19 -- STEP: 23200/36938 -- GLOBAL_STEP: 946650 | > loss_text_ce: 0.03559848293662071 (0.033024925053348295) | > loss_mel_ce: 1.857354760169983 (nan) | > loss: 0.007511719595640898 (nan) | > current_lr: 5e-06 | > step_time: 0.1659 (0.13670063896425771) | > loader_time: 0.0052 (0.004684972074525082)  --> TIME: 2025-02-25 21:10:29 -- STEP: 23250/36938 -- GLOBAL_STEP: 946700 | > loss_text_ce: 0.030433528125286102 (0.03302575155766162) | > loss_mel_ce: 2.7974936962127686 (nan) | > loss: 0.01122193317860365 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13670338828589262) | > loader_time: 0.0045 (0.004685186355344686)  --> TIME: 2025-02-25 21:10:38 -- STEP: 23300/36938 -- GLOBAL_STEP: 946750 | > loss_text_ce: 0.03575880825519562 (0.03302777162639113) | > loss_mel_ce: 3.135979652404785 (nan) | > loss: 0.012586263939738274 (nan) | > current_lr: 5e-06 | > step_time: 0.1896 (0.13671259467693805) | > loader_time: 0.0048 (0.00468534947464904)  --> TIME: 2025-02-25 21:10:48 -- STEP: 23350/36938 -- GLOBAL_STEP: 946800 | > loss_text_ce: 0.032950256019830704 (0.03302365309885851) | > loss_mel_ce: 0.23586425185203552 (nan) | > loss: 0.0010667243041098118 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13671607593397447) | > loader_time: 0.0045 (0.004685492423668134)  --> TIME: 2025-02-25 21:10:57 -- STEP: 23400/36938 -- GLOBAL_STEP: 946850 | > loss_text_ce: 0.03720226511359215 (0.03302386419204819) | > loss_mel_ce: 2.573312282562256 (nan) | > loss: 0.010359185747802258 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13672504936527918) | > loader_time: 0.0046 (0.004685471964697536)  --> TIME: 2025-02-25 21:11:07 -- STEP: 23450/36938 -- GLOBAL_STEP: 946900 | > loss_text_ce: 0.03458772972226143 (0.03302617603910945) | > loss_mel_ce: 2.893521547317505 (nan) | > loss: 0.01161948125809431 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.1367349017454354) | > loader_time: 0.0055 (0.004685473431910519)  --> TIME: 2025-02-25 21:11:16 -- STEP: 23500/36938 -- GLOBAL_STEP: 946950 | > loss_text_ce: 0.03751019388437271 (0.033025383086617316) | > loss_mel_ce: 2.7502028942108154 (nan) | > loss: 0.01106235384941101 (nan) | > current_lr: 5e-06 | > step_time: 0.1779 (0.13673366422856012) | > loader_time: 0.0047 (0.0046855184473889805)  --> TIME: 2025-02-25 21:11:26 -- STEP: 23550/36938 -- GLOBAL_STEP: 947000 | > loss_text_ce: 0.039676208049058914 (0.033022877087642434) | > loss_mel_ce: 2.40906023979187 (nan) | > loss: 0.009717208333313465 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.1367330478558864) | > loader_time: 0.0047 (0.004685439263685986)  --> TIME: 2025-02-25 21:11:35 -- STEP: 23600/36938 -- GLOBAL_STEP: 947050 | > loss_text_ce: 0.042088594287633896 (0.03302260129793403) | > loss_mel_ce: 1.916748046875 (nan) | > loss: 0.007773161865770817 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13672634689484506) | > loader_time: 0.0045 (0.00468537842823286)  --> TIME: 2025-02-25 21:11:44 -- STEP: 23650/36938 -- GLOBAL_STEP: 947100 | > loss_text_ce: 0.02818278782069683 (0.03302798827777327) | > loss_mel_ce: 2.7898409366607666 (nan) | > loss: 0.0111826341599226 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13671715092205236) | > loader_time: 0.0048 (0.004685357408342087)  --> TIME: 2025-02-25 21:11:54 -- STEP: 23700/36938 -- GLOBAL_STEP: 947150 | > loss_text_ce: 0.03741590306162834 (0.033028946270643414) | > loss_mel_ce: 2.288588523864746 (nan) | > loss: 0.009230176918208599 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13673149069653168) | > loader_time: 0.0047 (0.004685286811635449)  --> TIME: 2025-02-25 21:12:03 -- STEP: 23750/36938 -- GLOBAL_STEP: 947200 | > loss_text_ce: 0.026880986988544464 (0.033027513587415847) | > loss_mel_ce: 0.06520334631204605 (nan) | > loss: 0.0003654140455182642 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1367260707554053) | > loader_time: 0.0046 (0.004685167232312638)  --> TIME: 2025-02-25 21:12:12 -- STEP: 23800/36938 -- GLOBAL_STEP: 947250 | > loss_text_ce: 0.039446257054805756 (0.0330273713443247) | > loss_mel_ce: 1.7713165283203125 (nan) | > loss: 0.007185566704720259 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13672837837403443) | > loader_time: 0.0067 (0.0046851474597674305)  --> TIME: 2025-02-25 21:12:22 -- STEP: 23850/36938 -- GLOBAL_STEP: 947300 | > loss_text_ce: 0.031340423971414566 (0.033027152581007496) | > loss_mel_ce: 2.6092567443847656 (nan) | > loss: 0.01047856081277132 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13674039436836036) | > loader_time: 0.0046 (0.004685066730971094)  --> TIME: 2025-02-25 21:12:31 -- STEP: 23900/36938 -- GLOBAL_STEP: 947350 | > loss_text_ce: 0.03341500833630562 (0.03302644530796357) | > loss_mel_ce: 1.585679531097412 (nan) | > loss: 0.0064249783754348755 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13674121456664853) | > loader_time: 0.0045 (0.0046851652735945435)  --> TIME: 2025-02-25 21:12:41 -- STEP: 23950/36938 -- GLOBAL_STEP: 947400 | > loss_text_ce: 0.03889419883489609 (0.033024821842190265) | > loss_mel_ce: 2.772223949432373 (nan) | > loss: 0.011155230924487114 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1367299298304344) | > loader_time: 0.0045 (0.004685172686248319)  --> TIME: 2025-02-25 21:12:50 -- STEP: 24000/36938 -- GLOBAL_STEP: 947450 | > loss_text_ce: 0.038722578436136246 (0.03302507310851541) | > loss_mel_ce: 3.213482618331909 (nan) | > loss: 0.012905576266348362 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13671974150339639) | > loader_time: 0.0045 (0.004685226738452897)  --> TIME: 2025-02-25 21:12:59 -- STEP: 24050/36938 -- GLOBAL_STEP: 947500 | > loss_text_ce: 0.0309800636023283 (0.033027133752307196) | > loss_mel_ce: 2.2976226806640625 (nan) | > loss: 0.009240487590432167 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.1367241970863251) | > loader_time: 0.0046 (0.004685551203214191)  --> TIME: 2025-02-25 21:13:08 -- STEP: 24100/36938 -- GLOBAL_STEP: 947550 | > loss_text_ce: 0.03759843856096268 (0.03302990476985326) | > loss_mel_ce: 2.5314414501190186 (nan) | > loss: 0.010194603353738785 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13672859262133905) | > loader_time: 0.0048 (0.004685411878641201)  --> TIME: 2025-02-25 21:13:18 -- STEP: 24150/36938 -- GLOBAL_STEP: 947600 | > loss_text_ce: 0.028506632894277573 (0.03302870522777678) | > loss_mel_ce: 2.775092363357544 (nan) | > loss: 0.011125393211841583 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.1367347366024976) | > loader_time: 0.0045 (0.004685282450531817)  --> TIME: 2025-02-25 21:13:27 -- STEP: 24200/36938 -- GLOBAL_STEP: 947650 | > loss_text_ce: 0.03200500085949898 (0.03303231232470536) | > loss_mel_ce: 2.351280450820923 (nan) | > loss: 0.00945748295634985 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.13674123936448096) | > loader_time: 0.0049 (0.004685354400272197)  --> TIME: 2025-02-25 21:13:37 -- STEP: 24250/36938 -- GLOBAL_STEP: 947700 | > loss_text_ce: 0.026544494554400444 (0.03303177861871242) | > loss_mel_ce: 2.8926503658294678 (nan) | > loss: 0.011584107764065266 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1367414400749588) | > loader_time: 0.0052 (0.004685335247786989)  --> TIME: 2025-02-25 21:13:46 -- STEP: 24300/36938 -- GLOBAL_STEP: 947750 | > loss_text_ce: 0.03306807950139046 (0.03303156606226363) | > loss_mel_ce: 2.655470132827759 (nan) | > loss: 0.010668803006410599 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.13674939939514488) | > loader_time: 0.0048 (0.004685371039826178)  --> TIME: 2025-02-25 21:13:56 -- STEP: 24350/36938 -- GLOBAL_STEP: 947800 | > loss_text_ce: 0.029359154403209686 (0.03302955767061373) | > loss_mel_ce: 2.2487895488739014 (nan) | > loss: 0.009040272794663906 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13675887753096835) | > loader_time: 0.0045 (0.004685339859134103)  --> TIME: 2025-02-25 21:14:05 -- STEP: 24400/36938 -- GLOBAL_STEP: 947850 | > loss_text_ce: 0.03560848534107208 (0.03303074965033987) | > loss_mel_ce: 2.706745147705078 (nan) | > loss: 0.010882356204092503 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13675468170251906) | > loader_time: 0.0046 (0.004685158094421741)  --> TIME: 2025-02-25 21:14:15 -- STEP: 24450/36938 -- GLOBAL_STEP: 947900 | > loss_text_ce: 0.04270126670598984 (0.0330296922419676) | > loss_mel_ce: 2.865204095840454 (nan) | > loss: 0.011539307422935963 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13675964060980256) | > loader_time: 0.0046 (0.00468512130417461)  --> TIME: 2025-02-25 21:14:24 -- STEP: 24500/36938 -- GLOBAL_STEP: 947950 | > loss_text_ce: 0.03101384826004505 (0.03302915586213742) | > loss_mel_ce: 2.598245859146118 (nan) | > loss: 0.01043357141315937 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1367580310568509) | > loader_time: 0.0045 (0.0046851907652251355)  --> TIME: 2025-02-25 21:14:33 -- STEP: 24550/36938 -- GLOBAL_STEP: 948000 | > loss_text_ce: 0.023215526714920998 (0.033028069304931955) | > loss_mel_ce: 2.856287717819214 (nan) | > loss: 0.011426600627601147 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.13675447608691282) | > loader_time: 0.0047 (0.004685004956853339)  --> TIME: 2025-02-25 21:14:43 -- STEP: 24600/36938 -- GLOBAL_STEP: 948050 | > loss_text_ce: 0.028061186894774437 (0.033026277412701956) | > loss_mel_ce: 2.5596368312835693 (nan) | > loss: 0.010268643498420715 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.1367593696349996) | > loader_time: 0.0046 (0.0046848744396271475)  --> TIME: 2025-02-25 21:14:52 -- STEP: 24650/36938 -- GLOBAL_STEP: 948100 | > loss_text_ce: 0.03064620867371559 (0.03302591811599195) | > loss_mel_ce: 2.7725791931152344 (nan) | > loss: 0.01112391147762537 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13676274722294582) | > loader_time: 0.0046 (0.004684748717302218)  --> TIME: 2025-02-25 21:15:01 -- STEP: 24700/36938 -- GLOBAL_STEP: 948150 | > loss_text_ce: 0.028776178136467934 (0.03302802669500866) | > loss_mel_ce: 3.3966071605682373 (nan) | > loss: 0.013592791743576527 (nan) | > current_lr: 5e-06 | > step_time: 0.1612 (0.13675516985688524) | > loader_time: 0.0046 (0.004684584198693009)  --> TIME: 2025-02-25 21:15:10 -- STEP: 24750/36938 -- GLOBAL_STEP: 948200 | > loss_text_ce: 0.03275420889258385 (0.03302942434177168) | > loss_mel_ce: 2.3582849502563477 (nan) | > loss: 0.009488251060247421 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13674651402174728) | > loader_time: 0.0044 (0.004684564137699605)  --> TIME: 2025-02-25 21:15:20 -- STEP: 24800/36938 -- GLOBAL_STEP: 948250 | > loss_text_ce: 0.03735165670514107 (0.033029881652065204) | > loss_mel_ce: 3.1938114166259766 (nan) | > loss: 0.012822076678276062 (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.13674749935826844) | > loader_time: 0.0048 (0.004684488619527484)  --> TIME: 2025-02-25 21:15:29 -- STEP: 24850/36938 -- GLOBAL_STEP: 948300 | > loss_text_ce: 0.022488871589303017 (0.03303090122899391) | > loss_mel_ce: 2.346238136291504 (nan) | > loss: 0.009399711154401302 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13675170724780394) | > loader_time: 0.0046 (0.004684394111095993)  --> TIME: 2025-02-25 21:15:38 -- STEP: 24900/36938 -- GLOBAL_STEP: 948350 | > loss_text_ce: 0.03816166892647743 (0.03302736965282976) | > loss_mel_ce: 2.6179943084716797 (nan) | > loss: 0.010540302842855453 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.13675644833399955) | > loader_time: 0.0046 (0.0046842449448673325)  --> TIME: 2025-02-25 21:15:48 -- STEP: 24950/36938 -- GLOBAL_STEP: 948400 | > loss_text_ce: 0.026881713420152664 (0.03302643574010901) | > loss_mel_ce: 1.3055506944656372 (nan) | > loss: 0.005287430249154568 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.1367531043685259) | > loader_time: 0.0046 (0.004684064813510657)  --> TIME: 2025-02-25 21:15:57 -- STEP: 25000/36938 -- GLOBAL_STEP: 948450 | > loss_text_ce: 0.03469381853938103 (0.033029119416680006) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.1367599749469746) | > loader_time: 0.0047 (0.004683917608261074)  --> TIME: 2025-02-25 21:16:06 -- STEP: 25050/36938 -- GLOBAL_STEP: 948500 | > loss_text_ce: 0.03454935550689697 (0.03302801913209985) | > loss_mel_ce: 3.4401402473449707 (nan) | > loss: 0.013788451440632343 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13674929117252146) | > loader_time: 0.0044 (0.004683691841399562)  --> TIME: 2025-02-25 21:16:15 -- STEP: 25100/36938 -- GLOBAL_STEP: 948550 | > loss_text_ce: 0.04023050516843796 (0.03303141642285796) | > loss_mel_ce: 2.502751350402832 (nan) | > loss: 0.010091198608279228 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13674146080396937) | > loader_time: 0.0045 (0.004683487472306173)  --> TIME: 2025-02-25 21:16:24 -- STEP: 25150/36938 -- GLOBAL_STEP: 948600 | > loss_text_ce: 0.039847150444984436 (0.0330303844753799) | > loss_mel_ce: 2.841960906982422 (nan) | > loss: 0.011435747146606445 (nan) | > current_lr: 5e-06 | > step_time: 0.1797 (0.13673668239509976) | > loader_time: 0.0047 (0.00468331946530348)  --> TIME: 2025-02-25 21:16:34 -- STEP: 25200/36938 -- GLOBAL_STEP: 948650 | > loss_text_ce: 0.02109644189476967 (0.03303041031892719) | > loss_mel_ce: 1.8097506761550903 (nan) | > loss: 0.007265266962349415 (nan) | > current_lr: 5e-06 | > step_time: 0.1116 (0.13674446650913682) | > loader_time: 0.0044 (0.004683178994390663)  --> TIME: 2025-02-25 21:16:43 -- STEP: 25250/36938 -- GLOBAL_STEP: 948700 | > loss_text_ce: 0.025292469188570976 (0.03302802140813413) | > loss_mel_ce: 2.7561697959899902 (nan) | > loss: 0.011037549003958702 (nan) | > current_lr: 5e-06 | > step_time: 0.1706 (0.13674522325307925) | > loader_time: 0.0046 (0.004683006919256494)  --> TIME: 2025-02-25 21:16:52 -- STEP: 25300/36938 -- GLOBAL_STEP: 948750 | > loss_text_ce: 0.02897702157497406 (0.03302729642060988) | > loss_mel_ce: 2.9069879055023193 (nan) | > loss: 0.01165065448731184 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1367387685379954) | > loader_time: 0.0045 (0.004682833488750798)  --> TIME: 2025-02-25 21:17:02 -- STEP: 25350/36938 -- GLOBAL_STEP: 948800 | > loss_text_ce: 0.03167063370347023 (0.033026596182090064) | > loss_mel_ce: 2.7899839878082275 (nan) | > loss: 0.011197042651474476 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.1367403832083849) | > loader_time: 0.0047 (0.004682723427195011)  --> TIME: 2025-02-25 21:17:11 -- STEP: 25400/36938 -- GLOBAL_STEP: 948850 | > loss_text_ce: 0.038349542766809464 (0.03302765424852786) | > loss_mel_ce: 2.82855486869812 (nan) | > loss: 0.011376605369150639 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13672942187842335) | > loader_time: 0.0045 (0.004682593927608658)  --> TIME: 2025-02-25 21:17:19 -- STEP: 25450/36938 -- GLOBAL_STEP: 948900 | > loss_text_ce: 0.037489406764507294 (0.03302823716154285) | > loss_mel_ce: 3.3880550861358643 (nan) | > loss: 0.013593431562185287 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.1367133535804814) | > loader_time: 0.0048 (0.004682592558720211)  --> TIME: 2025-02-25 21:17:29 -- STEP: 25500/36938 -- GLOBAL_STEP: 948950 | > loss_text_ce: 0.03664654493331909 (0.03302867333085123) | > loss_mel_ce: 3.185518264770508 (nan) | > loss: 0.01278636883944273 (nan) | > current_lr: 5e-06 | > step_time: 0.1264 (0.13672383545894154) | > loader_time: 0.0046 (0.004682581882850753)  --> TIME: 2025-02-25 21:17:38 -- STEP: 25550/36938 -- GLOBAL_STEP: 949000 | > loss_text_ce: 0.03179264813661575 (0.03302866014269193) | > loss_mel_ce: 1.2902405261993408 (nan) | > loss: 0.005246163811534643 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.1367211376011012) | > loader_time: 0.0045 (0.0046826186338749)  --> TIME: 2025-02-25 21:17:47 -- STEP: 25600/36938 -- GLOBAL_STEP: 949050 | > loss_text_ce: 0.03597836568951607 (0.03303108452220813) | > loss_mel_ce: 2.6920888423919678 (nan) | > loss: 0.010825663805007935 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.13671538393944402) | > loader_time: 0.0047 (0.004682509768754203)  --> TIME: 2025-02-25 21:17:57 -- STEP: 25650/36938 -- GLOBAL_STEP: 949100 | > loss_text_ce: 0.024503124877810478 (0.03303267009510879) | > loss_mel_ce: 2.7372756004333496 (nan) | > loss: 0.010959440842270851 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.13671907562493713) | > loader_time: 0.0049 (0.004682458901730622)  --> TIME: 2025-02-25 21:18:06 -- STEP: 25700/36938 -- GLOBAL_STEP: 949150 | > loss_text_ce: 0.03816971555352211 (0.033031932258170076) | > loss_mel_ce: 2.8639659881591797 (nan) | > loss: 0.011516411788761616 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13672262732620746) | > loader_time: 0.0046 (0.00468237533643548)  --> TIME: 2025-02-25 21:18:16 -- STEP: 25750/36938 -- GLOBAL_STEP: 949200 | > loss_text_ce: 0.036547087132930756 (0.033032371575374184) | > loss_mel_ce: 2.7941482067108154 (nan) | > loss: 0.011232919059693813 (nan) | > current_lr: 5e-06 | > step_time: 0.1651 (0.13673882662208248) | > loader_time: 0.0048 (0.00468234058491231)  --> TIME: 2025-02-25 21:18:26 -- STEP: 25800/36938 -- GLOBAL_STEP: 949250 | > loss_text_ce: 0.03584975004196167 (0.033033079205770026) | > loss_mel_ce: 2.7022600173950195 (nan) | > loss: 0.010865516029298306 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13673803109531157) | > loader_time: 0.0045 (0.004682318101557616)  --> TIME: 2025-02-25 21:18:35 -- STEP: 25850/36938 -- GLOBAL_STEP: 949300 | > loss_text_ce: 0.031143341213464737 (0.03303321900518085) | > loss_mel_ce: 2.6569411754608154 (nan) | > loss: 0.010667002759873867 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1367438390314915) | > loader_time: 0.0047 (0.004682371076808877)  --> TIME: 2025-02-25 21:18:44 -- STEP: 25900/36938 -- GLOBAL_STEP: 949350 | > loss_text_ce: 0.039513975381851196 (0.03303417445083107) | > loss_mel_ce: 3.047962188720703 (nan) | > loss: 0.012251890264451504 (nan) | > current_lr: 5e-06 | > step_time: 0.1764 (0.1367425002646713) | > loader_time: 0.0049 (0.004682280731937566)  --> TIME: 2025-02-25 21:18:53 -- STEP: 25950/36938 -- GLOBAL_STEP: 949400 | > loss_text_ce: 0.02854476124048233 (0.033032765436346596) | > loss_mel_ce: 3.0088374614715576 (nan) | > loss: 0.012053104117512703 (nan) | > current_lr: 5e-06 | > step_time: 0.1832 (0.13673245095563433) | > loader_time: 0.007 (0.004682432608337956)  --> TIME: 2025-02-25 21:19:03 -- STEP: 26000/36938 -- GLOBAL_STEP: 949450 | > loss_text_ce: 0.028997568413615227 (0.03303327510747673) | > loss_mel_ce: 3.2270359992980957 (nan) | > loss: 0.012920768931508064 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13673496768107687) | > loader_time: 0.0046 (0.004682325940865704)  --> TIME: 2025-02-25 21:19:12 -- STEP: 26050/36938 -- GLOBAL_STEP: 949500 | > loss_text_ce: 0.03145167604088783 (0.03303183829978498) | > loss_mel_ce: 2.736982822418213 (nan) | > loss: 0.010985852219164371 (nan) | > current_lr: 5e-06 | > step_time: 0.152 (0.13673366451446387) | > loader_time: 0.0047 (0.004682695604789245)  --> TIME: 2025-02-25 21:19:22 -- STEP: 26100/36938 -- GLOBAL_STEP: 949550 | > loss_text_ce: 0.02167217619717121 (0.03302986256912025) | > loss_mel_ce: 2.136908769607544 (nan) | > loss: 0.00856579840183258 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.1367345124186218) | > loader_time: 0.0045 (0.004682903052289791)  --> TIME: 2025-02-25 21:19:31 -- STEP: 26150/36938 -- GLOBAL_STEP: 949600 | > loss_text_ce: 0.030287986621260643 (0.033029056507470546) | > loss_mel_ce: 2.585313320159912 (nan) | > loss: 0.010379371233284473 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.1367392153037884) | > loader_time: 0.0047 (0.004682926338443755)  --> TIME: 2025-02-25 21:19:40 -- STEP: 26200/36938 -- GLOBAL_STEP: 949650 | > loss_text_ce: 0.03502073884010315 (0.03303012790888847) | > loss_mel_ce: 3.100944995880127 (nan) | > loss: 0.012444309890270233 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13673786005900934) | > loader_time: 0.0067 (0.004682929888936362)  --> TIME: 2025-02-25 21:19:50 -- STEP: 26250/36938 -- GLOBAL_STEP: 949700 | > loss_text_ce: 0.03567584604024887 (0.03303373445703485) | > loss_mel_ce: 3.012782335281372 (nan) | > loss: 0.012097056955099106 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13673921698615582) | > loader_time: 0.0047 (0.00468309993743893)  --> TIME: 2025-02-25 21:19:59 -- STEP: 26300/36938 -- GLOBAL_STEP: 949750 | > loss_text_ce: 0.026132909581065178 (0.03303481052638473) | > loss_mel_ce: 2.8907008171081543 (nan) | > loss: 0.01157473772764206 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.1367359611471335) | > loader_time: 0.0046 (0.00468296681973414)  --> TIME: 2025-02-25 21:20:08 -- STEP: 26350/36938 -- GLOBAL_STEP: 949800 | > loss_text_ce: 0.03369307145476341 (0.0330384321473854) | > loss_mel_ce: 2.809610366821289 (nan) | > loss: 0.011282950639724731 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.13673852502507922) | > loader_time: 0.0048 (0.004682886143563106)  --> TIME: 2025-02-25 21:20:18 -- STEP: 26400/36938 -- GLOBAL_STEP: 949850 | > loss_text_ce: 0.035235073417425156 (0.03303979158350305) | > loss_mel_ce: 3.064310312271118 (nan) | > loss: 0.01229978445917368 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13674290204589937) | > loader_time: 0.0046 (0.0046826914404377445)  --> TIME: 2025-02-25 21:20:27 -- STEP: 26450/36938 -- GLOBAL_STEP: 949900 | > loss_text_ce: 0.034207526594400406 (0.033039597889394035) | > loss_mel_ce: 2.628136157989502 (nan) | > loss: 0.01056485716253519 (nan) | > current_lr: 5e-06 | > step_time: 0.1733 (0.13674388003935123) | > loader_time: 0.0049 (0.004682784594299628)  --> TIME: 2025-02-25 21:20:36 -- STEP: 26500/36938 -- GLOBAL_STEP: 949950 | > loss_text_ce: 0.029249371960759163 (0.03303739553125856) | > loss_mel_ce: 2.7976038455963135 (nan) | > loss: 0.011217672377824783 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13674118582707448) | > loader_time: 0.0047 (0.004682676684181614)  --> TIME: 2025-02-25 21:20:46 -- STEP: 26550/36938 -- GLOBAL_STEP: 950000 | > loss_text_ce: 0.03115737996995449 (0.03303686434801164) | > loss_mel_ce: 3.0691044330596924 (nan) | > loss: 0.01230262778699398 (nan) | > current_lr: 5e-06 | > step_time: 0.1767 (0.13674356522533085) | > loader_time: 0.0047 (0.0046826577860083) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_950000.pth  --> TIME: 2025-02-25 21:21:01 -- STEP: 26600/36938 -- GLOBAL_STEP: 950050 | > loss_text_ce: 0.031215613707900047 (0.033036941954589116) | > loss_mel_ce: 3.1173269748687744 (nan) | > loss: 0.012494217604398727 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13673361521914396) | > loader_time: 0.0046 (0.004682565120826063)  --> TIME: 2025-02-25 21:21:10 -- STEP: 26650/36938 -- GLOBAL_STEP: 950100 | > loss_text_ce: 0.042276304215192795 (0.03303561094905414) | > loss_mel_ce: 2.752558469772339 (nan) | > loss: 0.011090614832937717 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13672482556741994) | > loader_time: 0.0053 (0.004682546717588245)  --> TIME: 2025-02-25 21:21:19 -- STEP: 26700/36938 -- GLOBAL_STEP: 950150 | > loss_text_ce: 0.0319090336561203 (0.033037050874007025) | > loss_mel_ce: 3.0481255054473877 (nan) | > loss: 0.012222359888255596 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13673213881053192) | > loader_time: 0.0046 (0.004682600998253385)  --> TIME: 2025-02-25 21:21:29 -- STEP: 26750/36938 -- GLOBAL_STEP: 950200 | > loss_text_ce: 0.021937958896160126 (0.03303325341539167) | > loss_mel_ce: 3.1025614738464355 (nan) | > loss: 0.01239880733191967 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13672967507905953) | > loader_time: 0.0046 (0.004682586643183318)  --> TIME: 2025-02-25 21:21:38 -- STEP: 26800/36938 -- GLOBAL_STEP: 950250 | > loss_text_ce: 0.02677597850561142 (0.03303331078323786) | > loss_mel_ce: 2.958960771560669 (nan) | > loss: 0.011848162859678268 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13672862294004828) | > loader_time: 0.0048 (0.004682564352875305)  --> TIME: 2025-02-25 21:21:47 -- STEP: 26850/36938 -- GLOBAL_STEP: 950300 | > loss_text_ce: 0.03186825290322304 (0.033032317038094484) | > loss_mel_ce: 3.1029443740844727 (nan) | > loss: 0.012439733371138573 (nan) | > current_lr: 5e-06 | > step_time: 0.1123 (0.1367248177483974) | > loader_time: 0.0052 (0.0046825885683908375)  --> TIME: 2025-02-25 21:21:56 -- STEP: 26900/36938 -- GLOBAL_STEP: 950350 | > loss_text_ce: 0.03378243371844292 (0.03302982119678881) | > loss_mel_ce: 2.8876545429229736 (nan) | > loss: 0.011593004688620567 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13672068086255357) | > loader_time: 0.0046 (0.004682592299791052)  --> TIME: 2025-02-25 21:22:06 -- STEP: 26950/36938 -- GLOBAL_STEP: 950400 | > loss_text_ce: 0.03076282888650894 (0.03302970566969947) | > loss_mel_ce: 2.630647897720337 (nan) | > loss: 0.01056115422397852 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13672323017261828) | > loader_time: 0.0069 (0.004683046765583983)  --> TIME: 2025-02-25 21:22:15 -- STEP: 27000/36938 -- GLOBAL_STEP: 950450 | > loss_text_ce: 0.03587090224027634 (0.03302928908736651) | > loss_mel_ce: 3.0661110877990723 (nan) | > loss: 0.012309452518820763 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13671195492037952) | > loader_time: 0.0046 (0.004683175705097312)  --> TIME: 2025-02-25 21:22:24 -- STEP: 27050/36938 -- GLOBAL_STEP: 950500 | > loss_text_ce: 0.02030215971171856 (0.03302955562784175) | > loss_mel_ce: 2.6822314262390137 (nan) | > loss: 0.010724339634180069 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13670552087138557) | > loader_time: 0.0046 (0.0046830708702919185)  --> TIME: 2025-02-25 21:22:33 -- STEP: 27100/36938 -- GLOBAL_STEP: 950550 | > loss_text_ce: 0.04346035048365593 (0.03303112866523613) | > loss_mel_ce: 3.520596504211426 (nan) | > loss: 0.014143083244562149 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.13670090422858974) | > loader_time: 0.0046 (0.004682897078595015)  --> TIME: 2025-02-25 21:22:42 -- STEP: 27150/36938 -- GLOBAL_STEP: 950600 | > loss_text_ce: 0.03301795944571495 (0.03302830565122181) | > loss_mel_ce: 2.4038825035095215 (nan) | > loss: 0.009670239873230457 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.1367041741594198) | > loader_time: 0.0047 (0.0046829507715354934)  --> TIME: 2025-02-25 21:22:52 -- STEP: 27200/36938 -- GLOBAL_STEP: 950650 | > loss_text_ce: 0.045604459941387177 (0.03302753157192905) | > loss_mel_ce: 2.7806859016418457 (nan) | > loss: 0.011215439066290855 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13670491231714893) | > loader_time: 0.0045 (0.004683105261886794)  --> TIME: 2025-02-25 21:23:01 -- STEP: 27250/36938 -- GLOBAL_STEP: 950700 | > loss_text_ce: 0.03612593188881874 (0.03302655315421351) | > loss_mel_ce: 2.9247167110443115 (nan) | > loss: 0.01174937654286623 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13670433140675883) | > loader_time: 0.0049 (0.004683017809456616)  --> TIME: 2025-02-25 21:23:10 -- STEP: 27300/36938 -- GLOBAL_STEP: 950750 | > loss_text_ce: 0.03149070590734482 (0.033030009916471474) | > loss_mel_ce: 2.704591751098633 (nan) | > loss: 0.010857471264898777 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.1367049395295719) | > loader_time: 0.0047 (0.004682917411510737)  --> TIME: 2025-02-25 21:23:19 -- STEP: 27350/36938 -- GLOBAL_STEP: 950800 | > loss_text_ce: 0.02607128769159317 (0.03303258093828184) | > loss_mel_ce: 3.153620481491089 (nan) | > loss: 0.012617825530469418 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13669783913892986) | > loader_time: 0.0046 (0.004682863739334446)  --> TIME: 2025-02-25 21:23:29 -- STEP: 27400/36938 -- GLOBAL_STEP: 950850 | > loss_text_ce: 0.028975410386919975 (0.03303078854434418) | > loss_mel_ce: 2.7432355880737305 (nan) | > loss: 0.01100083813071251 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13669810268130536) | > loader_time: 0.0047 (0.004683092388793477)  --> TIME: 2025-02-25 21:23:39 -- STEP: 27450/36938 -- GLOBAL_STEP: 950900 | > loss_text_ce: 0.022599471732974052 (0.03303233854708405) | > loss_mel_ce: 2.731614828109741 (nan) | > loss: 0.010929422453045845 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13671091315090464) | > loader_time: 0.0046 (0.004683163300672276)  --> TIME: 2025-02-25 21:23:48 -- STEP: 27500/36938 -- GLOBAL_STEP: 950950 | > loss_text_ce: 0.03763637691736221 (0.033030673370764034) | > loss_mel_ce: 2.5783538818359375 (nan) | > loss: 0.010380913503468037 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.1367155648318195) | > loader_time: 0.0046 (0.0046831935535777474)  --> TIME: 2025-02-25 21:23:58 -- STEP: 27550/36938 -- GLOBAL_STEP: 951000 | > loss_text_ce: 0.035235341638326645 (0.03303033936388435) | > loss_mel_ce: 3.004396677017212 (nan) | > loss: 0.012062032707035542 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13672396115512375) | > loader_time: 0.0046 (0.00468334675700606)  --> TIME: 2025-02-25 21:24:07 -- STEP: 27600/36938 -- GLOBAL_STEP: 951050 | > loss_text_ce: 0.0398181788623333 (0.033033597938078335) | > loss_mel_ce: 1.3374671936035156 (nan) | > loss: 0.005465418566018343 (nan) | > current_lr: 5e-06 | > step_time: 0.1821 (0.13672487899013097) | > loader_time: 0.0048 (0.004683383219483947)  --> TIME: 2025-02-25 21:24:16 -- STEP: 27650/36938 -- GLOBAL_STEP: 951100 | > loss_text_ce: 0.02840753272175789 (0.03303517796548813) | > loss_mel_ce: 2.8236982822418213 (nan) | > loss: 0.01131788082420826 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13672628185416996) | > loader_time: 0.0046 (0.0046834312856304915)  --> TIME: 2025-02-25 21:24:26 -- STEP: 27700/36938 -- GLOBAL_STEP: 951150 | > loss_text_ce: 0.02837449684739113 (0.03303369804771266) | > loss_mel_ce: 7.46875 (nan) | > loss: 0.02975049614906311 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13672640205720624) | > loader_time: 0.0047 (0.0046835601889269225)  --> TIME: 2025-02-25 21:24:35 -- STEP: 27750/36938 -- GLOBAL_STEP: 951200 | > loss_text_ce: 0.031555503606796265 (0.033034027330425986) | > loss_mel_ce: 1.8126863241195679 (nan) | > loss: 0.007318420335650444 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13671675548037893) | > loader_time: 0.0046 (0.0046836218533215)  --> TIME: 2025-02-25 21:24:44 -- STEP: 27800/36938 -- GLOBAL_STEP: 951250 | > loss_text_ce: 0.030919469892978668 (0.033033113897687275) | > loss_mel_ce: 1.8871269226074219 (nan) | > loss: 0.00761129567399621 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.136718817151707) | > loader_time: 0.0045 (0.004683542345925178)  --> TIME: 2025-02-25 21:24:54 -- STEP: 27850/36938 -- GLOBAL_STEP: 951300 | > loss_text_ce: 0.032226089388132095 (0.033032657700562004) | > loss_mel_ce: 3.191150665283203 (nan) | > loss: 0.012791178189218044 (nan) | > current_lr: 5e-06 | > step_time: 0.1625 (0.13672357031858098) | > loader_time: 0.0047 (0.004683467395857783)  --> TIME: 2025-02-25 21:25:03 -- STEP: 27900/36938 -- GLOBAL_STEP: 951350 | > loss_text_ce: 0.03249329328536987 (0.033032796899019966) | > loss_mel_ce: 2.6067707538604736 (nan) | > loss: 0.010473270900547504 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.13673069940245636) | > loader_time: 0.0046 (0.004683536218485929)  --> TIME: 2025-02-25 21:25:12 -- STEP: 27950/36938 -- GLOBAL_STEP: 951400 | > loss_text_ce: 0.02985147386789322 (0.03302867718659009) | > loss_mel_ce: 3.1415863037109375 (nan) | > loss: 0.012585070915520191 (nan) | > current_lr: 5e-06 | > step_time: 0.1905 (0.1367295712520477) | > loader_time: 0.0047 (0.004683451260138474)  --> TIME: 2025-02-25 21:25:22 -- STEP: 28000/36938 -- GLOBAL_STEP: 951450 | > loss_text_ce: 0.029070386663079262 (0.03302652878242705) | > loss_mel_ce: 2.97409725189209 (nan) | > loss: 0.01191733218729496 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13673512314898556) | > loader_time: 0.0046 (0.0046833839075905714)  --> TIME: 2025-02-25 21:25:31 -- STEP: 28050/36938 -- GLOBAL_STEP: 951500 | > loss_text_ce: 0.029914038255810738 (0.033026531481709934) | > loss_mel_ce: 2.7272770404815674 (nan) | > loss: 0.010941235348582268 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13673374080827871) | > loader_time: 0.0046 (0.004683461733255347)  --> TIME: 2025-02-25 21:25:40 -- STEP: 28100/36938 -- GLOBAL_STEP: 951550 | > loss_text_ce: 0.03022884950041771 (0.03302705581679409) | > loss_mel_ce: 2.5555100440979004 (nan) | > loss: 0.010260868817567825 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13672972742772957) | > loader_time: 0.0045 (0.0046836227030092065)  --> TIME: 2025-02-25 21:25:50 -- STEP: 28150/36938 -- GLOBAL_STEP: 951600 | > loss_text_ce: 0.039296604692935944 (0.033026771447463715) | > loss_mel_ce: 2.8719377517700195 (nan) | > loss: 0.011552518233656883 (nan) | > current_lr: 5e-06 | > step_time: 0.1445 (0.1367250027665017) | > loader_time: 0.0049 (0.004683541057376602)  --> TIME: 2025-02-25 21:25:59 -- STEP: 28200/36938 -- GLOBAL_STEP: 951650 | > loss_text_ce: 0.028727859258651733 (0.03302780957937079) | > loss_mel_ce: 3.2210421562194824 (nan) | > loss: 0.012895912863314152 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13672598039850203) | > loader_time: 0.0046 (0.004683698973757133)  --> TIME: 2025-02-25 21:26:08 -- STEP: 28250/36938 -- GLOBAL_STEP: 951700 | > loss_text_ce: 0.03176447004079819 (0.03302871230598992) | > loss_mel_ce: 2.7350218296051025 (nan) | > loss: 0.010979311540722847 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.1367216505540148) | > loader_time: 0.0046 (0.004683674390337086)  --> TIME: 2025-02-25 21:26:17 -- STEP: 28300/36938 -- GLOBAL_STEP: 951750 | > loss_text_ce: 0.02306652069091797 (0.03302806313704106) | > loss_mel_ce: 2.399414539337158 (nan) | > loss: 0.009613020345568657 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13672262971898202) | > loader_time: 0.0054 (0.0046836886389095235)  --> TIME: 2025-02-25 21:26:27 -- STEP: 28350/36938 -- GLOBAL_STEP: 951800 | > loss_text_ce: 0.03633739799261093 (0.03303026396477292) | > loss_mel_ce: 3.0499792098999023 (nan) | > loss: 0.012247288599610329 (nan) | > current_lr: 5e-06 | > step_time: 0.1908 (0.13672855013051718) | > loader_time: 0.0049 (0.004683734273153616)  --> TIME: 2025-02-25 21:26:36 -- STEP: 28400/36938 -- GLOBAL_STEP: 951850 | > loss_text_ce: 0.030883027240633965 (0.03302960746178793) | > loss_mel_ce: 2.436981201171875 (nan) | > loss: 0.009793112985789776 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13672858521132344) | > loader_time: 0.0046 (0.004683855730043309)  --> TIME: 2025-02-25 21:26:45 -- STEP: 28450/36938 -- GLOBAL_STEP: 951900 | > loss_text_ce: 0.039166033267974854 (0.03302866917042928) | > loss_mel_ce: 2.8961522579193115 (nan) | > loss: 0.011648088693618774 (nan) | > current_lr: 5e-06 | > step_time: 0.1862 (0.13672103166999136) | > loader_time: 0.0073 (0.004683968589259764)  --> TIME: 2025-02-25 21:26:55 -- STEP: 28500/36938 -- GLOBAL_STEP: 951950 | > loss_text_ce: 0.03345261886715889 (0.03302810969012505) | > loss_mel_ce: 3.0782556533813477 (nan) | > loss: 0.012348049320280552 (nan) | > current_lr: 5e-06 | > step_time: 0.1812 (0.13672225206776592) | > loader_time: 0.0048 (0.00468384816353779)  --> TIME: 2025-02-25 21:27:04 -- STEP: 28550/36938 -- GLOBAL_STEP: 952000 | > loss_text_ce: 0.04752926900982857 (0.033027407758691035) | > loss_mel_ce: 2.542926073074341 (nan) | > loss: 0.010279584676027298 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13672149682420678) | > loader_time: 0.0047 (0.004683960565543612)  --> TIME: 2025-02-25 21:27:13 -- STEP: 28600/36938 -- GLOBAL_STEP: 952050 | > loss_text_ce: 0.031972818076610565 (0.0330271144044399) | > loss_mel_ce: 2.0387797355651855 (nan) | > loss: 0.008217272348701954 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.13672612427831446) | > loader_time: 0.0048 (0.004684069640152916)  --> TIME: 2025-02-25 21:27:23 -- STEP: 28650/36938 -- GLOBAL_STEP: 952100 | > loss_text_ce: 0.031090280041098595 (0.033028331543375394) | > loss_mel_ce: 2.629145383834839 (nan) | > loss: 0.01055649109184742 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13673214958184177) | > loader_time: 0.0046 (0.004684229870741258)  --> TIME: 2025-02-25 21:27:32 -- STEP: 28700/36938 -- GLOBAL_STEP: 952150 | > loss_text_ce: 0.018174830824136734 (0.0330270887200817) | > loss_mel_ce: 1.789752721786499 (nan) | > loss: 0.007174316328018904 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.1367321091828022) | > loader_time: 0.0045 (0.004684214085236637)  --> TIME: 2025-02-25 21:27:42 -- STEP: 28750/36938 -- GLOBAL_STEP: 952200 | > loss_text_ce: 0.02848743088543415 (0.033027025699890404) | > loss_mel_ce: 3.322779655456543 (nan) | > loss: 0.013298680074512959 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.1367318116644145) | > loader_time: 0.0045 (0.004684283447265603)  --> TIME: 2025-02-25 21:27:51 -- STEP: 28800/36938 -- GLOBAL_STEP: 952250 | > loss_text_ce: 0.028596317395567894 (0.03302782405466898) | > loss_mel_ce: 2.5752878189086914 (nan) | > loss: 0.010332874022424221 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.13673902684615624) | > loader_time: 0.0047 (0.0046844226204686475)  --> TIME: 2025-02-25 21:28:01 -- STEP: 28850/36938 -- GLOBAL_STEP: 952300 | > loss_text_ce: 0.02953878603875637 (0.03302803621788757) | > loss_mel_ce: 2.6653518676757812 (nan) | > loss: 0.010694011114537716 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.13674886098560782) | > loader_time: 0.0049 (0.004684701874929634)  --> TIME: 2025-02-25 21:28:10 -- STEP: 28900/36938 -- GLOBAL_STEP: 952350 | > loss_text_ce: 0.03112422674894333 (0.03302866268472763) | > loss_mel_ce: 2.3725180625915527 (nan) | > loss: 0.009538263082504272 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13674197199435287) | > loader_time: 0.0066 (0.004684737363901083)  --> TIME: 2025-02-25 21:28:19 -- STEP: 28950/36938 -- GLOBAL_STEP: 952400 | > loss_text_ce: 0.03363369032740593 (0.03302844076211458) | > loss_mel_ce: 3.198831796646118 (nan) | > loss: 0.012827244587242603 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1367395540922613) | > loader_time: 0.0045 (0.004684812433773144)  --> TIME: 2025-02-25 21:28:29 -- STEP: 29000/36938 -- GLOBAL_STEP: 952450 | > loss_text_ce: 0.03662976622581482 (0.03302862261442814) | > loss_mel_ce: 2.521801471710205 (nan) | > loss: 0.010152504779398441 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.1367447349449674) | > loader_time: 0.0046 (0.004684845759950813)  --> TIME: 2025-02-25 21:28:38 -- STEP: 29050/36938 -- GLOBAL_STEP: 952500 | > loss_text_ce: 0.029399219900369644 (0.033029013739372635) | > loss_mel_ce: 0.9289828538894653 (nan) | > loss: 0.003803103696554899 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13673247203564187) | > loader_time: 0.0047 (0.004684960846236137)  --> TIME: 2025-02-25 21:28:48 -- STEP: 29100/36938 -- GLOBAL_STEP: 952550 | > loss_text_ce: 0.032479122281074524 (0.0330295995346193) | > loss_mel_ce: 2.1564741134643555 (nan) | > loss: 0.008686322718858719 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13673975042461012) | > loader_time: 0.0046 (0.004684944857436746)  --> TIME: 2025-02-25 21:28:57 -- STEP: 29150/36938 -- GLOBAL_STEP: 952600 | > loss_text_ce: 0.025256549939513206 (0.03302752478819743) | > loss_mel_ce: 2.481616258621216 (nan) | > loss: 0.00994790904223919 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.13673068242048425) | > loader_time: 0.0046 (0.00468508305721379)  --> TIME: 2025-02-25 21:29:06 -- STEP: 29200/36938 -- GLOBAL_STEP: 952650 | > loss_text_ce: 0.05019859969615936 (0.03302993672531479) | > loss_mel_ce: 2.4699642658233643 (nan) | > loss: 0.01000064704567194 (nan) | > current_lr: 5e-06 | > step_time: 0.1084 (0.13673350799573514) | > loader_time: 0.0051 (0.004685217852461806)  --> TIME: 2025-02-25 21:29:15 -- STEP: 29250/36938 -- GLOBAL_STEP: 952700 | > loss_text_ce: 0.026966392993927002 (0.033031087433186626) | > loss_mel_ce: 1.5560461282730103 (nan) | > loss: 0.0062817963771522045 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1367319406607202) | > loader_time: 0.0046 (0.004685292553697875)  --> TIME: 2025-02-25 21:29:24 -- STEP: 29300/36938 -- GLOBAL_STEP: 952750 | > loss_text_ce: 0.0303810965269804 (0.03303088062765429) | > loss_mel_ce: 2.57038950920105 (nan) | > loss: 0.010320519097149372 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.13672894870054902) | > loader_time: 0.0045 (0.004685197332206418)  --> TIME: 2025-02-25 21:29:34 -- STEP: 29350/36938 -- GLOBAL_STEP: 952800 | > loss_text_ce: 0.02880602702498436 (0.03303119466313186) | > loss_mel_ce: 2.6569337844848633 (nan) | > loss: 0.010657697916030884 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13672673100304047) | > loader_time: 0.0045 (0.0046853014232758755)  --> TIME: 2025-02-25 21:29:43 -- STEP: 29400/36938 -- GLOBAL_STEP: 952850 | > loss_text_ce: 0.027736080810427666 (0.03303102058588122) | > loss_mel_ce: 2.9267988204956055 (nan) | > loss: 0.011724346317350864 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13672736113574124) | > loader_time: 0.0046 (0.004685345604306154)  --> TIME: 2025-02-25 21:29:52 -- STEP: 29450/36938 -- GLOBAL_STEP: 952900 | > loss_text_ce: 0.02599661611020565 (0.03302960315173706) | > loss_mel_ce: 1.9747132062911987 (nan) | > loss: 0.007939324714243412 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13672204754348416) | > loader_time: 0.0046 (0.004685337960416452)  --> TIME: 2025-02-25 21:30:01 -- STEP: 29500/36938 -- GLOBAL_STEP: 952950 | > loss_text_ce: 0.032202333211898804 (0.033028436576363236) | > loss_mel_ce: 3.162146806716919 (nan) | > loss: 0.012675989419221878 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13671943520691332) | > loader_time: 0.0048 (0.0046854347536119155)  --> TIME: 2025-02-25 21:30:10 -- STEP: 29550/36938 -- GLOBAL_STEP: 953000 | > loss_text_ce: 0.039410125464200974 (0.03302706222736929) | > loss_mel_ce: 3.4246864318847656 (nan) | > loss: 0.013746415264904499 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13670994281768684) | > loader_time: 0.0046 (0.00468529585049235)  --> TIME: 2025-02-25 21:30:20 -- STEP: 29600/36938 -- GLOBAL_STEP: 953050 | > loss_text_ce: 0.03842180222272873 (0.03302774995655055) | > loss_mel_ce: 3.031546115875244 (nan) | > loss: 0.012182413600385189 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.13671203726852146) | > loader_time: 0.0047 (0.004685448996118576)  --> TIME: 2025-02-25 21:30:29 -- STEP: 29650/36938 -- GLOBAL_STEP: 953100 | > loss_text_ce: 0.033802345395088196 (0.033027148594526656) | > loss_mel_ce: 0.9909537434577942 (nan) | > loss: 0.004066492430865765 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13670678133104036) | > loader_time: 0.0046 (0.00468553630738795)  --> TIME: 2025-02-25 21:30:38 -- STEP: 29700/36938 -- GLOBAL_STEP: 953150 | > loss_text_ce: 0.025246748700737953 (0.03302470069797208) | > loss_mel_ce: 1.4395999908447266 (nan) | > loss: 0.00581288430839777 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.13670466925559918) | > loader_time: 0.0046 (0.004685525918247703)  --> TIME: 2025-02-25 21:30:47 -- STEP: 29750/36938 -- GLOBAL_STEP: 953200 | > loss_text_ce: 0.04402947425842285 (0.03302688816845574) | > loss_mel_ce: 3.399151563644409 (nan) | > loss: 0.013663417659699917 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13669912106650106) | > loader_time: 0.0045 (0.004685519843542254)  --> TIME: 2025-02-25 21:30:57 -- STEP: 29800/36938 -- GLOBAL_STEP: 953250 | > loss_text_ce: 0.03263678401708603 (0.033027935014621715) | > loss_mel_ce: 3.0164616107940674 (nan) | > loss: 0.012099597603082657 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13670278281173326) | > loader_time: 0.0046 (0.004685627318068618)  --> TIME: 2025-02-25 21:31:06 -- STEP: 29850/36938 -- GLOBAL_STEP: 953300 | > loss_text_ce: 0.027888797223567963 (0.03302775127377952) | > loss_mel_ce: 2.5343141555786133 (nan) | > loss: 0.010167472995817661 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1367000070089459) | > loader_time: 0.0045 (0.004685503490007082)  --> TIME: 2025-02-25 21:31:15 -- STEP: 29900/36938 -- GLOBAL_STEP: 953350 | > loss_text_ce: 0.03553742915391922 (0.03302940562309786) | > loss_mel_ce: 2.4751603603363037 (nan) | > loss: 0.009963086806237698 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13669566716637385) | > loader_time: 0.0045 (0.004685390163026106)  --> TIME: 2025-02-25 21:31:24 -- STEP: 29950/36938 -- GLOBAL_STEP: 953400 | > loss_text_ce: 0.023019734770059586 (0.03302966563769577) | > loss_mel_ce: 2.163393974304199 (nan) | > loss: 0.00867624580860138 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13669166671612704) | > loader_time: 0.0049 (0.00468536238439491)  --> TIME: 2025-02-25 21:31:34 -- STEP: 30000/36938 -- GLOBAL_STEP: 953450 | > loss_text_ce: 0.04015246406197548 (0.03302791111309364) | > loss_mel_ce: 0.1692759245634079 (nan) | > loss: 0.0008310650591738522 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13669700761635978) | > loader_time: 0.0046 (0.0046853330691655265)  --> TIME: 2025-02-25 21:31:43 -- STEP: 30050/36938 -- GLOBAL_STEP: 953500 | > loss_text_ce: 0.02507399022579193 (0.03302561366817452) | > loss_mel_ce: 1.400736689567566 (nan) | > loss: 0.005657979287207127 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.1366914319436669) | > loader_time: 0.0046 (0.004685498307429593)  --> TIME: 2025-02-25 21:31:52 -- STEP: 30100/36938 -- GLOBAL_STEP: 953550 | > loss_text_ce: 0.0329478457570076 (0.033025588841808416) | > loss_mel_ce: 4.2198662757873535 (nan) | > loss: 0.01687624864280224 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13667902792014827) | > loader_time: 0.0047 (0.004685641380639547)  --> TIME: 2025-02-25 21:32:01 -- STEP: 30150/36938 -- GLOBAL_STEP: 953600 | > loss_text_ce: 0.03174648806452751 (0.03302724495377195) | > loss_mel_ce: 0.8998336791992188 (nan) | > loss: 0.0036967468913644552 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13668185521911702) | > loader_time: 0.0046 (0.004685605240500763)  --> TIME: 2025-02-25 21:32:10 -- STEP: 30200/36938 -- GLOBAL_STEP: 953650 | > loss_text_ce: 0.030096735805273056 (0.033024835429286636) | > loss_mel_ce: 3.0469610691070557 (nan) | > loss: 0.012210547924041748 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.1366811665399173) | > loader_time: 0.0047 (0.004685566883213452)  --> TIME: 2025-02-25 21:32:20 -- STEP: 30250/36938 -- GLOBAL_STEP: 953700 | > loss_text_ce: 0.032540880143642426 (0.03302554943660353) | > loss_mel_ce: 2.815331220626831 (nan) | > loss: 0.011301079764962196 (nan) | > current_lr: 5e-06 | > step_time: 0.1697 (0.13667709850279633) | > loader_time: 0.0047 (0.004685545795219968)  --> TIME: 2025-02-25 21:32:29 -- STEP: 30300/36938 -- GLOBAL_STEP: 953750 | > loss_text_ce: 0.03829250857234001 (0.03302660673790623) | > loss_mel_ce: 2.8237686157226562 (nan) | > loss: 0.011357385665178299 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.1366694183868925) | > loader_time: 0.0045 (0.0046854061025990695)  --> TIME: 2025-02-25 21:32:38 -- STEP: 30350/36938 -- GLOBAL_STEP: 953800 | > loss_text_ce: 0.02985176257789135 (0.03302601769647827) | > loss_mel_ce: 2.777071237564087 (nan) | > loss: 0.011138583533465862 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13667424633239678) | > loader_time: 0.0046 (0.004685536774222088)  --> TIME: 2025-02-25 21:32:47 -- STEP: 30400/36938 -- GLOBAL_STEP: 953850 | > loss_text_ce: 0.02693004161119461 (0.03302698653566587) | > loss_mel_ce: 2.429434061050415 (nan) | > loss: 0.009747477248311043 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.1366709901392448) | > loader_time: 0.0056 (0.004685550732047914)  --> TIME: 2025-02-25 21:32:57 -- STEP: 30450/36938 -- GLOBAL_STEP: 953900 | > loss_text_ce: 0.03529135510325432 (0.03302339801028012) | > loss_mel_ce: 2.8120553493499756 (nan) | > loss: 0.011298995465040207 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13666900809370902) | > loader_time: 0.0049 (0.00468550058029749)  --> TIME: 2025-02-25 21:33:06 -- STEP: 30500/36938 -- GLOBAL_STEP: 953950 | > loss_text_ce: 0.035320818424224854 (0.03302314139223156) | > loss_mel_ce: 3.0840344429016113 (nan) | > loss: 0.012378394603729248 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13666751353466977) | > loader_time: 0.005 (0.004685387627023145)  --> TIME: 2025-02-25 21:33:15 -- STEP: 30550/36938 -- GLOBAL_STEP: 954000 | > loss_text_ce: 0.04025851562619209 (0.033023091003819775) | > loss_mel_ce: 3.038860559463501 (nan) | > loss: 0.012218726798892021 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13666949426678976) | > loader_time: 0.0046 (0.004685455063165872)  --> TIME: 2025-02-25 21:33:24 -- STEP: 30600/36938 -- GLOBAL_STEP: 954050 | > loss_text_ce: 0.031789034605026245 (0.03302509515903851) | > loss_mel_ce: 3.049268960952759 (nan) | > loss: 0.012226421386003494 (nan) | > current_lr: 5e-06 | > step_time: 0.1576 (0.13666377508562255) | > loader_time: 0.0046 (0.004685368015875204)  --> TIME: 2025-02-25 21:33:33 -- STEP: 30650/36938 -- GLOBAL_STEP: 954100 | > loss_text_ce: 0.02996133454144001 (0.033024880477799785) | > loss_mel_ce: 2.3601467609405518 (nan) | > loss: 0.009484556503593922 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13665793665470422) | > loader_time: 0.0046 (0.0046854515838000545)  --> TIME: 2025-02-25 21:33:42 -- STEP: 30700/36938 -- GLOBAL_STEP: 954150 | > loss_text_ce: 0.024881167337298393 (0.03302624453579627) | > loss_mel_ce: 0.1143798828125 (nan) | > loss: 0.000552623241674155 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13664793061122713) | > loader_time: 0.0046 (0.00468544648602263)  --> TIME: 2025-02-25 21:33:52 -- STEP: 30750/36938 -- GLOBAL_STEP: 954200 | > loss_text_ce: 0.02439139597117901 (0.03302341005719446) | > loss_mel_ce: 1.1299136877059937 (nan) | > loss: 0.0045805759727954865 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.136652661044422) | > loader_time: 0.0044 (0.004685503223078018)  --> TIME: 2025-02-25 21:34:01 -- STEP: 30800/36938 -- GLOBAL_STEP: 954250 | > loss_text_ce: 0.03524276241660118 (0.033023563869003944) | > loss_mel_ce: 2.6453287601470947 (nan) | > loss: 0.010637189261615276 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.13665356077931026) | > loader_time: 0.0055 (0.004685493181278159)  --> TIME: 2025-02-25 21:34:10 -- STEP: 30850/36938 -- GLOBAL_STEP: 954300 | > loss_text_ce: 0.024916997179389 (0.03302282121876223) | > loss_mel_ce: 1.5234375 (nan) | > loss: 0.006144264247268438 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13664946567683667) | > loader_time: 0.0048 (0.004685599297719777)  --> TIME: 2025-02-25 21:34:20 -- STEP: 30900/36938 -- GLOBAL_STEP: 954350 | > loss_text_ce: 0.036363840103149414 (0.03302314012288764) | > loss_mel_ce: 1.998838186264038 (nan) | > loss: 0.0080761993303895 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13664529352126267) | > loader_time: 0.0046 (0.00468546166003327)  --> TIME: 2025-02-25 21:34:29 -- STEP: 30950/36938 -- GLOBAL_STEP: 954400 | > loss_text_ce: 0.031559914350509644 (0.033024386399243204) | > loss_mel_ce: 2.3320627212524414 (nan) | > loss: 0.00937945581972599 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13664356485899776) | > loader_time: 0.0045 (0.004685352692118734)  --> TIME: 2025-02-25 21:34:38 -- STEP: 31000/36938 -- GLOBAL_STEP: 954450 | > loss_text_ce: 0.027952246367931366 (0.0330234734397529) | > loss_mel_ce: 3.1386194229125977 (nan) | > loss: 0.012565760873258114 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13663941676385824) | > loader_time: 0.0052 (0.004685408807569914)  --> TIME: 2025-02-25 21:34:48 -- STEP: 31050/36938 -- GLOBAL_STEP: 954500 | > loss_text_ce: 0.023217177018523216 (0.03302255257635668) | > loss_mel_ce: 3.2416293621063232 (nan) | > loss: 0.01295574102550745 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1366483833325262) | > loader_time: 0.0046 (0.0046855922136905485)  --> TIME: 2025-02-25 21:34:57 -- STEP: 31100/36938 -- GLOBAL_STEP: 954550 | > loss_text_ce: 0.03607385605573654 (0.033022863518916555) | > loss_mel_ce: 2.906742811203003 (nan) | > loss: 0.011677844449877739 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1366509842719284) | > loader_time: 0.0046 (0.004685741314167358)  --> TIME: 2025-02-25 21:35:06 -- STEP: 31150/36938 -- GLOBAL_STEP: 954600 | > loss_text_ce: 0.03212166577577591 (0.03302391194914817) | > loss_mel_ce: 2.464221239089966 (nan) | > loss: 0.009906123392283916 (nan) | > current_lr: 5e-06 | > step_time: 0.1875 (0.13664510602935626) | > loader_time: 0.0048 (0.004685822306245689)  --> TIME: 2025-02-25 21:35:16 -- STEP: 31200/36938 -- GLOBAL_STEP: 954650 | > loss_text_ce: 0.03436101973056793 (0.03302647691896121) | > loss_mel_ce: 3.0374245643615723 (nan) | > loss: 0.012189626693725586 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13664704467241465) | > loader_time: 0.0046 (0.00468599742039654)  --> TIME: 2025-02-25 21:35:25 -- STEP: 31250/36938 -- GLOBAL_STEP: 954700 | > loss_text_ce: 0.033637069165706635 (0.033027377244069886) | > loss_mel_ce: 2.8354909420013428 (nan) | > loss: 0.011385428719222546 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13664488394164923) | > loader_time: 0.0046 (0.004686071540832499)  --> TIME: 2025-02-25 21:35:34 -- STEP: 31300/36938 -- GLOBAL_STEP: 954750 | > loss_text_ce: 0.021788649260997772 (0.033027315903410105) | > loss_mel_ce: 1.5984923839569092 (nan) | > loss: 0.006429686676710844 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13664053370015575) | > loader_time: 0.0049 (0.004686282770321372)  --> TIME: 2025-02-25 21:35:44 -- STEP: 31350/36938 -- GLOBAL_STEP: 954800 | > loss_text_ce: 0.03376232460141182 (0.03302589757687293) | > loss_mel_ce: 2.7961838245391846 (nan) | > loss: 0.011229945346713066 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13664554499362908) | > loader_time: 0.0045 (0.004686277220694052)  --> TIME: 2025-02-25 21:35:53 -- STEP: 31400/36938 -- GLOBAL_STEP: 954850 | > loss_text_ce: 0.030625663697719574 (0.033025865468246714) | > loss_mel_ce: 0.08994140475988388 (nan) | > loss: 0.000478440779261291 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13664167452010265) | > loader_time: 0.0045 (0.004686143474214376)  --> TIME: 2025-02-25 21:36:02 -- STEP: 31450/36938 -- GLOBAL_STEP: 954900 | > loss_text_ce: 0.041055165231227875 (0.03302596724941552) | > loss_mel_ce: 3.107907295227051 (nan) | > loss: 0.012495883740484715 (nan) | > current_lr: 5e-06 | > step_time: 0.169 (0.13665297377848548) | > loader_time: 0.0048 (0.004686070739370062)  --> TIME: 2025-02-25 21:36:12 -- STEP: 31500/36938 -- GLOBAL_STEP: 954950 | > loss_text_ce: 0.0389401949942112 (0.033025249096357706) | > loss_mel_ce: 2.5432372093200684 (nan) | > loss: 0.010246735997498035 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13665312089617174) | > loader_time: 0.0045 (0.004686074453686891)  --> TIME: 2025-02-25 21:36:21 -- STEP: 31550/36938 -- GLOBAL_STEP: 955000 | > loss_text_ce: 0.02626773528754711 (0.03302464157701538) | > loss_mel_ce: 3.197242259979248 (nan) | > loss: 0.012791707180440426 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13665842595077618) | > loader_time: 0.0046 (0.0046860288704631626)  --> TIME: 2025-02-25 21:36:30 -- STEP: 31600/36938 -- GLOBAL_STEP: 955050 | > loss_text_ce: 0.030443066731095314 (0.03302588343789738) | > loss_mel_ce: 2.595329523086548 (nan) | > loss: 0.010419732891023159 (nan) | > current_lr: 5e-06 | > step_time: 0.1855 (0.1366542053901685) | > loader_time: 0.0048 (0.0046860488230668846)  --> TIME: 2025-02-25 21:36:40 -- STEP: 31650/36938 -- GLOBAL_STEP: 955100 | > loss_text_ce: 0.032597608864307404 (0.03302223485561963) | > loss_mel_ce: 2.800962448120117 (nan) | > loss: 0.01124428678303957 (nan) | > current_lr: 5e-06 | > step_time: 0.1439 (0.13665718345280392) | > loader_time: 0.0052 (0.004686119944375065)  --> TIME: 2025-02-25 21:36:49 -- STEP: 31700/36938 -- GLOBAL_STEP: 955150 | > loss_text_ce: 0.03220946341753006 (0.033022410627557064) | > loss_mel_ce: 0.948724627494812 (nan) | > loss: 0.0038925958797335625 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13665613501605814) | > loader_time: 0.0046 (0.004686191164732712)  --> TIME: 2025-02-25 21:36:58 -- STEP: 31750/36938 -- GLOBAL_STEP: 955200 | > loss_text_ce: 0.04052329808473587 (0.03302309359690719) | > loss_mel_ce: 2.6899497509002686 (nan) | > loss: 0.010835210792720318 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13665762956874464) | > loader_time: 0.0047 (0.004686152908745693)  --> TIME: 2025-02-25 21:37:07 -- STEP: 31800/36938 -- GLOBAL_STEP: 955250 | > loss_text_ce: 0.037268850952386856 (0.0330263922803845) | > loss_mel_ce: 1.1047841310501099 (nan) | > loss: 0.004531956743448973 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13664566044537413) | > loader_time: 0.0052 (0.004686191914216507)  --> TIME: 2025-02-25 21:37:17 -- STEP: 31850/36938 -- GLOBAL_STEP: 955300 | > loss_text_ce: 0.03635789826512337 (0.03302577068942923) | > loss_mel_ce: 3.2985951900482178 (nan) | > loss: 0.013233941979706287 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.13664582764522132) | > loader_time: 0.0047 (0.00468649621661078)  --> TIME: 2025-02-25 21:37:26 -- STEP: 31900/36938 -- GLOBAL_STEP: 955350 | > loss_text_ce: 0.037313785403966904 (0.03302444621335197) | > loss_mel_ce: 1.947265625 (nan) | > loss: 0.007875315845012665 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13665261844856125) | > loader_time: 0.0053 (0.00468651990531752)  --> TIME: 2025-02-25 21:37:35 -- STEP: 31950/36938 -- GLOBAL_STEP: 955400 | > loss_text_ce: 0.03261668607592583 (0.033024211579420454) | > loss_mel_ce: 3.001180648803711 (nan) | > loss: 0.012038879096508026 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.1366475485561401) | > loader_time: 0.0047 (0.004686679869936833)  --> TIME: 2025-02-25 21:37:45 -- STEP: 32000/36938 -- GLOBAL_STEP: 955450 | > loss_text_ce: 0.03245249390602112 (0.03302438501080878) | > loss_mel_ce: 2.951908826828003 (nan) | > loss: 0.011842704378068447 (nan) | > current_lr: 5e-06 | > step_time: 0.1508 (0.13664505635201818) | > loader_time: 0.0053 (0.0046867824792861746)  --> TIME: 2025-02-25 21:37:54 -- STEP: 32050/36938 -- GLOBAL_STEP: 955500 | > loss_text_ce: 0.04113982990384102 (0.033022747055762254) | > loss_mel_ce: 1.9289402961730957 (nan) | > loss: 0.00781777873635292 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13664107265412787) | > loader_time: 0.0046 (0.00468692443300147)  --> TIME: 2025-02-25 21:38:03 -- STEP: 32100/36938 -- GLOBAL_STEP: 955550 | > loss_text_ce: 0.04039042070508003 (0.033022769352528236) | > loss_mel_ce: 0.013336181640625 (nan) | > loss: 0.0002132008085027337 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13664070832766362) | > loader_time: 0.0046 (0.00468702337080814)  --> TIME: 2025-02-25 21:38:13 -- STEP: 32150/36938 -- GLOBAL_STEP: 955600 | > loss_text_ce: 0.03694645315408707 (0.03302197471641927) | > loss_mel_ce: 2.365496873855591 (nan) | > loss: 0.009533505886793137 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13664871058604838) | > loader_time: 0.0046 (0.004687128237580418)  --> TIME: 2025-02-25 21:38:22 -- STEP: 32200/36938 -- GLOBAL_STEP: 955650 | > loss_text_ce: 0.03165309131145477 (0.033022345699447724) | > loss_mel_ce: 1.820043921470642 (nan) | > loss: 0.007348004262894392 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13665187397358466) | > loader_time: 0.0049 (0.004687290643312897)  --> TIME: 2025-02-25 21:38:32 -- STEP: 32250/36938 -- GLOBAL_STEP: 955700 | > loss_text_ce: 0.03529250621795654 (0.033022387866002714) | > loss_mel_ce: 3.1038105487823486 (nan) | > loss: 0.012456757947802544 (nan) | > current_lr: 5e-06 | > step_time: 0.1648 (0.13665180304623276) | > loader_time: 0.0048 (0.004687409046084362)  --> TIME: 2025-02-25 21:38:41 -- STEP: 32300/36938 -- GLOBAL_STEP: 955750 | > loss_text_ce: 0.03196628391742706 (0.03302183747837933) | > loss_mel_ce: 3.0704309940338135 (nan) | > loss: 0.012311100959777832 (nan) | > current_lr: 5e-06 | > step_time: 0.1809 (0.13665812231807772) | > loader_time: 0.0049 (0.004687399229398064)  --> TIME: 2025-02-25 21:38:50 -- STEP: 32350/36938 -- GLOBAL_STEP: 955800 | > loss_text_ce: 0.03592764958739281 (0.03301939746208024) | > loss_mel_ce: 2.9339675903320312 (nan) | > loss: 0.011785298585891724 (nan) | > current_lr: 5e-06 | > step_time: 0.1449 (0.1366473613824494) | > loader_time: 0.0047 (0.004687405362195522)  --> TIME: 2025-02-25 21:38:59 -- STEP: 32400/36938 -- GLOBAL_STEP: 955850 | > loss_text_ce: 0.03545856475830078 (0.03301788160575952) | > loss_mel_ce: 2.5903289318084717 (nan) | > loss: 0.010419792495667934 (nan) | > current_lr: 5e-06 | > step_time: 0.177 (0.1366460398317844) | > loader_time: 0.0049 (0.004687577824533703)  --> TIME: 2025-02-25 21:39:08 -- STEP: 32450/36938 -- GLOBAL_STEP: 955900 | > loss_text_ce: 0.0295557901263237 (0.03301747228621617) | > loss_mel_ce: 3.1393868923187256 (nan) | > loss: 0.012575170025229454 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13664124150121779) | > loader_time: 0.0054 (0.004687630502027778)  --> TIME: 2025-02-25 21:39:18 -- STEP: 32500/36938 -- GLOBAL_STEP: 955950 | > loss_text_ce: 0.03506851568818092 (0.03301638119591803) | > loss_mel_ce: 2.2139475345611572 (nan) | > loss: 0.008924667723476887 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13664509228926328) | > loader_time: 0.0045 (0.004687709052746093)  --> TIME: 2025-02-25 21:39:27 -- STEP: 32550/36938 -- GLOBAL_STEP: 956000 | > loss_text_ce: 0.03376017138361931 (0.03301384977388316) | > loss_mel_ce: 2.834869384765625 (nan) | > loss: 0.011383450590074062 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.1366366018641002) | > loader_time: 0.0047 (0.004687679337649465)  --> TIME: 2025-02-25 21:39:36 -- STEP: 32600/36938 -- GLOBAL_STEP: 956050 | > loss_text_ce: 0.031434204429388046 (0.03301456738684722) | > loss_mel_ce: 1.8539865016937256 (nan) | > loss: 0.007481828331947327 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1366315192752078) | > loader_time: 0.0046 (0.004687593875487139)  --> TIME: 2025-02-25 21:39:46 -- STEP: 32650/36938 -- GLOBAL_STEP: 956100 | > loss_text_ce: 0.033124979585409164 (0.0330134935697729) | > loss_mel_ce: 3.012284517288208 (nan) | > loss: 0.01208495907485485 (nan) | > current_lr: 5e-06 | > step_time: 0.1767 (0.1366368778777878) | > loader_time: 0.0051 (0.0046875986388412215)  --> TIME: 2025-02-25 21:39:55 -- STEP: 32700/36938 -- GLOBAL_STEP: 956150 | > loss_text_ce: 0.028799206018447876 (0.03301399132916079) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13664642096294793) | > loader_time: 0.0045 (0.004687487823882944)  --> TIME: 2025-02-25 21:40:05 -- STEP: 32750/36938 -- GLOBAL_STEP: 956200 | > loss_text_ce: 0.023086704313755035 (0.033012462297255143) | > loss_mel_ce: 2.3887460231781006 (nan) | > loss: 0.009570765309035778 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13664997891433256) | > loader_time: 0.0046 (0.0046874996214422366)  --> TIME: 2025-02-25 21:40:14 -- STEP: 32800/36938 -- GLOBAL_STEP: 956250 | > loss_text_ce: 0.03318043053150177 (0.03301445790136308) | > loss_mel_ce: 2.8660802841186523 (nan) | > loss: 0.01150500401854515 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13665041894447402) | > loader_time: 0.0046 (0.004687395350235246)  --> TIME: 2025-02-25 21:40:24 -- STEP: 32850/36938 -- GLOBAL_STEP: 956300 | > loss_text_ce: 0.025687072426080704 (0.03301541818621934) | > loss_mel_ce: 3.3736767768859863 (nan) | > loss: 0.013489539735019207 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.1366549132040817) | > loader_time: 0.0046 (0.004687327293500489)  --> TIME: 2025-02-25 21:40:33 -- STEP: 32900/36938 -- GLOBAL_STEP: 956350 | > loss_text_ce: 0.04331487417221069 (0.03301552499212739) | > loss_mel_ce: 3.271749973297119 (nan) | > loss: 0.013155019842088223 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1366541950898319) | > loader_time: 0.0045 (0.00468725326213429)  --> TIME: 2025-02-25 21:40:42 -- STEP: 32950/36938 -- GLOBAL_STEP: 956400 | > loss_text_ce: 0.020424753427505493 (0.033012694639700434) | > loss_mel_ce: 2.1462554931640625 (nan) | > loss: 0.00859793834388256 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1366609883416936) | > loader_time: 0.0046 (0.004687216162500667)  --> TIME: 2025-02-25 21:40:52 -- STEP: 33000/36938 -- GLOBAL_STEP: 956450 | > loss_text_ce: 0.034583624452352524 (0.03301202091355922) | > loss_mel_ce: 2.4652225971221924 (nan) | > loss: 0.009919865988194942 (nan) | > current_lr: 5e-06 | > step_time: 0.1831 (0.13666125189174191) | > loader_time: 0.0049 (0.0046871960885596815)  --> TIME: 2025-02-25 21:41:01 -- STEP: 33050/36938 -- GLOBAL_STEP: 956500 | > loss_text_ce: 0.029973115772008896 (0.033010989920772706) | > loss_mel_ce: 2.971383571624756 (nan) | > loss: 0.011910146102309227 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.13666294605035711) | > loader_time: 0.0047 (0.004687398241074813)  --> TIME: 2025-02-25 21:41:10 -- STEP: 33100/36938 -- GLOBAL_STEP: 956550 | > loss_text_ce: 0.03087649680674076 (0.03301025423934526) | > loss_mel_ce: 2.731243848800659 (nan) | > loss: 0.010960794985294342 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13665856323933562) | > loader_time: 0.0048 (0.004687445912836523)  --> TIME: 2025-02-25 21:41:20 -- STEP: 33150/36938 -- GLOBAL_STEP: 956600 | > loss_text_ce: 0.025593837723135948 (0.03300929765340592) | > loss_mel_ce: 2.880157470703125 (nan) | > loss: 0.011530759744346142 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13666142906718196) | > loader_time: 0.0046 (0.004687503358717219)  --> TIME: 2025-02-25 21:41:29 -- STEP: 33200/36938 -- GLOBAL_STEP: 956650 | > loss_text_ce: 0.034337230026721954 (0.033008811437617136) | > loss_mel_ce: 2.4751272201538086 (nan) | > loss: 0.009958192706108093 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13666114939982527) | > loader_time: 0.0047 (0.004687427354146181)  --> TIME: 2025-02-25 21:41:38 -- STEP: 33250/36938 -- GLOBAL_STEP: 956700 | > loss_text_ce: 0.03155137598514557 (0.03301141242449024) | > loss_mel_ce: 0.49609375 (nan) | > loss: 0.0020938299130648375 (nan) | > current_lr: 5e-06 | > step_time: 0.1514 (0.13666156601726454) | > loader_time: 0.0047 (0.004687384928079458)  --> TIME: 2025-02-25 21:41:48 -- STEP: 33300/36938 -- GLOBAL_STEP: 956750 | > loss_text_ce: 0.026907693594694138 (0.033010747105445055) | > loss_mel_ce: 2.016491651535034 (nan) | > loss: 0.008108727633953094 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13666832178562505) | > loader_time: 0.0047 (0.004687359333038314)  --> TIME: 2025-02-25 21:41:57 -- STEP: 33350/36938 -- GLOBAL_STEP: 956800 | > loss_text_ce: 0.04005751013755798 (0.0330129953489023) | > loss_mel_ce: 2.9698715209960938 (nan) | > loss: 0.011944163590669632 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1366707086777569) | > loader_time: 0.0045 (0.004687316986038215)  --> TIME: 2025-02-25 21:42:07 -- STEP: 33400/36938 -- GLOBAL_STEP: 956850 | > loss_text_ce: 0.02817508950829506 (0.03301506363431208) | > loss_mel_ce: 2.8405239582061768 (nan) | > loss: 0.011383727192878723 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13667047050184725) | > loader_time: 0.0046 (0.004687271260929662)  --> TIME: 2025-02-25 21:42:16 -- STEP: 33450/36938 -- GLOBAL_STEP: 956900 | > loss_text_ce: 0.03403448686003685 (0.03301569843908355) | > loss_mel_ce: 2.511908531188965 (nan) | > loss: 0.010102949105203152 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13667601540484128) | > loader_time: 0.0046 (0.0046872929072700655)  --> TIME: 2025-02-25 21:42:26 -- STEP: 33500/36938 -- GLOBAL_STEP: 956950 | > loss_text_ce: 0.021622955799102783 (0.03301560569169498) | > loss_mel_ce: 2.466426372528076 (nan) | > loss: 0.009873212315142155 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13667399181536669) | > loader_time: 0.0046 (0.004687173935904416)  --> TIME: 2025-02-25 21:42:35 -- STEP: 33550/36938 -- GLOBAL_STEP: 957000 | > loss_text_ce: 0.04053666442632675 (0.03301432241805023) | > loss_mel_ce: 3.035712242126465 (nan) | > loss: 0.012207337655127048 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13667438163842496) | > loader_time: 0.0048 (0.004687171550870103)  --> TIME: 2025-02-25 21:42:44 -- STEP: 33600/36938 -- GLOBAL_STEP: 957050 | > loss_text_ce: 0.035414714366197586 (0.033013217854936903) | > loss_mel_ce: 2.611492395401001 (nan) | > loss: 0.010503600351512432 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13666442842000903) | > loader_time: 0.0048 (0.004687060507990049)  --> TIME: 2025-02-25 21:42:53 -- STEP: 33650/36938 -- GLOBAL_STEP: 957100 | > loss_text_ce: 0.032618504017591476 (0.03301255250398044) | > loss_mel_ce: 2.742802143096924 (nan) | > loss: 0.01101357489824295 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13666266960693751) | > loader_time: 0.0054 (0.00468710263806323)  --> TIME: 2025-02-25 21:43:03 -- STEP: 33700/36938 -- GLOBAL_STEP: 957150 | > loss_text_ce: 0.03463103994727135 (0.033012787504660246) | > loss_mel_ce: 2.9292969703674316 (nan) | > loss: 0.011761619709432125 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1366662096764981) | > loader_time: 0.0044 (0.004687106665942349)  --> TIME: 2025-02-25 21:43:12 -- STEP: 33750/36938 -- GLOBAL_STEP: 957200 | > loss_text_ce: 0.029817134141921997 (0.033014948485529216) | > loss_mel_ce: 3.180349111557007 (nan) | > loss: 0.012738755904138088 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13667320877357658) | > loader_time: 0.0046 (0.004687086804707829)  --> TIME: 2025-02-25 21:43:21 -- STEP: 33800/36938 -- GLOBAL_STEP: 957250 | > loss_text_ce: 0.03532666340470314 (0.03301695766391686) | > loss_mel_ce: 3.06018328666687 (nan) | > loss: 0.012283770367503166 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.13666899176038821) | > loader_time: 0.0047 (0.004687021208938039)  --> TIME: 2025-02-25 21:43:31 -- STEP: 33850/36938 -- GLOBAL_STEP: 957300 | > loss_text_ce: 0.04009171947836876 (0.03301453800386329) | > loss_mel_ce: 2.4938833713531494 (nan) | > loss: 0.01005545724183321 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.1366719194986961) | > loader_time: 0.0046 (0.004687009364850747)  --> TIME: 2025-02-25 21:43:40 -- STEP: 33900/36938 -- GLOBAL_STEP: 957350 | > loss_text_ce: 0.03380191698670387 (0.03301472487473407) | > loss_mel_ce: 2.7910633087158203 (nan) | > loss: 0.01120978407561779 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.13666891921586227) | > loader_time: 0.0047 (0.0046869539581568705)  --> TIME: 2025-02-25 21:43:49 -- STEP: 33950/36938 -- GLOBAL_STEP: 957400 | > loss_text_ce: 0.03289255499839783 (0.033015625772624374) | > loss_mel_ce: 2.9512736797332764 (nan) | > loss: 0.011841929517686367 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13667060719322796) | > loader_time: 0.0047 (0.004686874999213444)  --> TIME: 2025-02-25 21:43:59 -- STEP: 34000/36938 -- GLOBAL_STEP: 957450 | > loss_text_ce: 0.022374674677848816 (0.03301457155973841) | > loss_mel_ce: 3.2408194541931152 (nan) | > loss: 0.012949183583259583 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13667002146384177) | > loader_time: 0.0048 (0.004686755460851314)  --> TIME: 2025-02-25 21:44:08 -- STEP: 34050/36938 -- GLOBAL_STEP: 957500 | > loss_text_ce: 0.022375838831067085 (0.03301269880805009) | > loss_mel_ce: 0.19848306477069855 (nan) | > loss: 0.0008764242520555854 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1366713817109084) | > loader_time: 0.0045 (0.0046866525742450815)  --> TIME: 2025-02-25 21:44:17 -- STEP: 34100/36938 -- GLOBAL_STEP: 957550 | > loss_text_ce: 0.030666593462228775 (0.033013137627264356) | > loss_mel_ce: 3.225452423095703 (nan) | > loss: 0.012921107932925224 (nan) | > current_lr: 5e-06 | > step_time: 0.1978 (0.13666379674788337) | > loader_time: 0.0048 (0.00468667432359934)  --> TIME: 2025-02-25 21:44:26 -- STEP: 34150/36938 -- GLOBAL_STEP: 957600 | > loss_text_ce: 0.040753986686468124 (0.03301208322364563) | > loss_mel_ce: 2.720802068710327 (nan) | > loss: 0.010958557017147541 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.1366567574343244) | > loader_time: 0.0047 (0.004686704896728585)  --> TIME: 2025-02-25 21:44:36 -- STEP: 34200/36938 -- GLOBAL_STEP: 957650 | > loss_text_ce: 0.035851672291755676 (0.03301442005422365) | > loss_mel_ce: 2.7083253860473633 (nan) | > loss: 0.010889592580497265 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13666020290196182) | > loader_time: 0.0046 (0.0046867218700765685)  --> TIME: 2025-02-25 21:44:45 -- STEP: 34250/36938 -- GLOBAL_STEP: 957700 | > loss_text_ce: 0.03422144800424576 (0.03301414637094895) | > loss_mel_ce: 2.579636335372925 (nan) | > loss: 0.010372452437877655 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.1366633841799982) | > loader_time: 0.0053 (0.004686737666164851)  --> TIME: 2025-02-25 21:44:54 -- STEP: 34300/36938 -- GLOBAL_STEP: 957750 | > loss_text_ce: 0.03050675243139267 (0.0330130859308686) | > loss_mel_ce: 2.310736894607544 (nan) | > loss: 0.009290650486946106 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.1366580631781586) | > loader_time: 0.0046 (0.004686732945567297)  --> TIME: 2025-02-25 21:45:04 -- STEP: 34350/36938 -- GLOBAL_STEP: 957800 | > loss_text_ce: 0.042601004242897034 (0.0330115718182629) | > loss_mel_ce: 3.1187169551849365 (nan) | > loss: 0.012544913217425346 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.136665116411455) | > loader_time: 0.0046 (0.004686671073780291)  --> TIME: 2025-02-25 21:45:13 -- STEP: 34400/36938 -- GLOBAL_STEP: 957850 | > loss_text_ce: 0.03471098467707634 (0.0330109678121243) | > loss_mel_ce: 2.1175618171691895 (nan) | > loss: 0.008540765382349491 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13666879552048194) | > loader_time: 0.0046 (0.004686612514562367)  --> TIME: 2025-02-25 21:45:23 -- STEP: 34450/36938 -- GLOBAL_STEP: 957900 | > loss_text_ce: 0.031752943992614746 (0.03301025148785238) | > loss_mel_ce: 2.9475173950195312 (nan) | > loss: 0.011822502128779888 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13667149806057144) | > loader_time: 0.0048 (0.004686559122082786)  --> TIME: 2025-02-25 21:45:32 -- STEP: 34500/36938 -- GLOBAL_STEP: 957950 | > loss_text_ce: 0.03153257071971893 (0.03301057633353994) | > loss_mel_ce: 2.715205669403076 (nan) | > loss: 0.010899755172431469 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13667449911089813) | > loader_time: 0.0046 (0.0046864748415739454)  --> TIME: 2025-02-25 21:45:42 -- STEP: 34550/36938 -- GLOBAL_STEP: 958000 | > loss_text_ce: 0.04661955311894417 (0.03301081839427274) | > loss_mel_ce: 2.1373021602630615 (nan) | > loss: 0.008666357025504112 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13667749571903645) | > loader_time: 0.0046 (0.004686407766880396)  --> TIME: 2025-02-25 21:45:51 -- STEP: 34600/36938 -- GLOBAL_STEP: 958050 | > loss_text_ce: 0.034029845148324966 (0.03301080553756057) | > loss_mel_ce: 2.8044700622558594 (nan) | > loss: 0.011263889260590076 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13668297390028145) | > loader_time: 0.0045 (0.00468635845735581)  --> TIME: 2025-02-25 21:46:00 -- STEP: 34650/36938 -- GLOBAL_STEP: 958100 | > loss_text_ce: 0.03735611215233803 (0.033010942630945325) | > loss_mel_ce: 2.669912338256836 (nan) | > loss: 0.010743129067122936 (nan) | > current_lr: 5e-06 | > step_time: 0.1558 (0.13667832268608854) | > loader_time: 0.0047 (0.004686282393220164)  --> TIME: 2025-02-25 21:46:09 -- STEP: 34700/36938 -- GLOBAL_STEP: 958150 | > loss_text_ce: 0.03901776298880577 (0.033010760693467124) | > loss_mel_ce: 3.668342113494873 (nan) | > loss: 0.014711746014654636 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13667354079419577) | > loader_time: 0.0045 (0.004686254932832286)  --> TIME: 2025-02-25 21:46:19 -- STEP: 34750/36938 -- GLOBAL_STEP: 958200 | > loss_text_ce: 0.03270639106631279 (0.033011052357809906) | > loss_mel_ce: 2.750558376312256 (nan) | > loss: 0.011044702492654324 (nan) | > current_lr: 5e-06 | > step_time: 0.1851 (0.13667613715576515) | > loader_time: 0.0047 (0.004686251139469266)  --> TIME: 2025-02-25 21:46:28 -- STEP: 34800/36938 -- GLOBAL_STEP: 958250 | > loss_text_ce: 0.033037468791007996 (0.03300915760313268) | > loss_mel_ce: 3.233227491378784 (nan) | > loss: 0.012961369007825851 (nan) | > current_lr: 5e-06 | > step_time: 0.1977 (0.13668563182326465) | > loader_time: 0.0048 (0.00468618144934203)  --> TIME: 2025-02-25 21:46:37 -- STEP: 34850/36938 -- GLOBAL_STEP: 958300 | > loss_text_ce: 0.040174320340156555 (0.03301011378278232) | > loss_mel_ce: 2.568708658218384 (nan) | > loss: 0.010352710261940956 (nan) | > current_lr: 5e-06 | > step_time: 0.1615 (0.1366790569050914) | > loader_time: 0.0053 (0.004686088739884959)  --> TIME: 2025-02-25 21:46:47 -- STEP: 34900/36938 -- GLOBAL_STEP: 958350 | > loss_text_ce: 0.04262809082865715 (0.033010534631998156) | > loss_mel_ce: 3.349125623703003 (nan) | > loss: 0.013459340669214725 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.13668410466530165) | > loader_time: 0.0048 (0.004686120424024678)  --> TIME: 2025-02-25 21:46:56 -- STEP: 34950/36938 -- GLOBAL_STEP: 958400 | > loss_text_ce: 0.034141574054956436 (0.033009841208451254) | > loss_mel_ce: 2.7183144092559814 (nan) | > loss: 0.010922444984316826 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13668639303106347) | > loader_time: 0.0046 (0.00468608007581106)  --> TIME: 2025-02-25 21:47:06 -- STEP: 35000/36938 -- GLOBAL_STEP: 958450 | > loss_text_ce: 0.029372652992606163 (0.03301045162399678) | > loss_mel_ce: 2.6932713985443115 (nan) | > loss: 0.010804143734276295 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13669276864187965) | > loader_time: 0.0046 (0.004686029073170234)  --> TIME: 2025-02-25 21:47:15 -- STEP: 35050/36938 -- GLOBAL_STEP: 958500 | > loss_text_ce: 0.02585926465690136 (0.03300941837777481) | > loss_mel_ce: 2.9338603019714355 (nan) | > loss: 0.011744920164346695 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13669064261943817) | > loader_time: 0.0046 (0.0046859360489457345)  --> TIME: 2025-02-25 21:47:24 -- STEP: 35100/36938 -- GLOBAL_STEP: 958550 | > loss_text_ce: 0.03543408587574959 (0.033007827149823324) | > loss_mel_ce: 2.678089141845703 (nan) | > loss: 0.010767949745059013 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.13669075455081342) | > loader_time: 0.0048 (0.0046859096665667465)  --> TIME: 2025-02-25 21:47:34 -- STEP: 35150/36938 -- GLOBAL_STEP: 958600 | > loss_text_ce: 0.020716268569231033 (0.033006050503702596) | > loss_mel_ce: 2.7702598571777344 (nan) | > loss: 0.01107530202716589 (nan) | > current_lr: 5e-06 | > step_time: 0.1444 (0.13668986110904224) | > loader_time: 0.0046 (0.004685837350902294)  --> TIME: 2025-02-25 21:47:43 -- STEP: 35200/36938 -- GLOBAL_STEP: 958650 | > loss_text_ce: 0.03347214311361313 (0.03300635854045536) | > loss_mel_ce: 2.5674850940704346 (nan) | > loss: 0.010321259498596191 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1366960279169399) | > loader_time: 0.0045 (0.004685820971022935)  --> TIME: 2025-02-25 21:47:52 -- STEP: 35250/36938 -- GLOBAL_STEP: 958700 | > loss_text_ce: 0.0410492978990078 (0.03300570825684191) | > loss_mel_ce: 2.5607454776763916 (nan) | > loss: 0.01032458245754242 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13669371522402848) | > loader_time: 0.0046 (0.004685711853893072)  --> TIME: 2025-02-25 21:48:02 -- STEP: 35300/36938 -- GLOBAL_STEP: 958750 | > loss_text_ce: 0.03524549677968025 (0.03300653240900662) | > loss_mel_ce: 3.114521026611328 (nan) | > loss: 0.012499073520302773 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13669758313775984) | > loader_time: 0.0045 (0.004685640625507887)  --> TIME: 2025-02-25 21:48:11 -- STEP: 35350/36938 -- GLOBAL_STEP: 958800 | > loss_text_ce: 0.0282901618629694 (0.033007215553594156) | > loss_mel_ce: 2.6661860942840576 (nan) | > loss: 0.01069236733019352 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13670246934958105) | > loader_time: 0.0048 (0.004685635742068779)  --> TIME: 2025-02-25 21:48:21 -- STEP: 35400/36938 -- GLOBAL_STEP: 958850 | > loss_text_ce: 0.026357602328062057 (0.033007162077323314) | > loss_mel_ce: 2.5092787742614746 (nan) | > loss: 0.010062050074338913 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.136706034901451) | > loader_time: 0.0046 (0.004685636994528887)  --> TIME: 2025-02-25 21:48:30 -- STEP: 35450/36938 -- GLOBAL_STEP: 958900 | > loss_text_ce: 0.028735263273119926 (0.03300600765770802) | > loss_mel_ce: 1.8880419731140137 (nan) | > loss: 0.007606259547173977 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13670536158283175) | > loader_time: 0.0046 (0.00468554939974849)  --> TIME: 2025-02-25 21:48:40 -- STEP: 35500/36938 -- GLOBAL_STEP: 958950 | > loss_text_ce: 0.03474944829940796 (0.03300687546614855) | > loss_mel_ce: 1.1076937913894653 (nan) | > loss: 0.004533505067229271 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.1367076233541453) | > loader_time: 0.0047 (0.0046854599563168646)  --> TIME: 2025-02-25 21:48:49 -- STEP: 35550/36938 -- GLOBAL_STEP: 959000 | > loss_text_ce: 0.04262938350439072 (0.03300585858551334) | > loss_mel_ce: 0.21384277939796448 (nan) | > loss: 0.0010177467484027147 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13670350149210908) | > loader_time: 0.0046 (0.004685363481148705)  --> TIME: 2025-02-25 21:48:58 -- STEP: 35600/36938 -- GLOBAL_STEP: 959050 | > loss_text_ce: 0.028347518295049667 (0.03300353205083913) | > loss_mel_ce: 3.0150413513183594 (nan) | > loss: 0.012076940387487411 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13669289056504627) | > loader_time: 0.0045 (0.004685268308339477)  --> TIME: 2025-02-25 21:49:07 -- STEP: 35650/36938 -- GLOBAL_STEP: 959100 | > loss_text_ce: 0.041506316512823105 (0.03300492847838575) | > loss_mel_ce: 3.3516695499420166 (nan) | > loss: 0.013464984484016895 (nan) | > current_lr: 5e-06 | > step_time: 0.1485 (0.1366927704202502) | > loader_time: 0.0053 (0.004685242881614381)  --> TIME: 2025-02-25 21:49:16 -- STEP: 35700/36938 -- GLOBAL_STEP: 959150 | > loss_text_ce: 0.030956557020545006 (0.03300370531377079) | > loss_mel_ce: 2.812992572784424 (nan) | > loss: 0.011285512708127499 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.13668998641460134) | > loader_time: 0.0047 (0.004685176767888844)  --> TIME: 2025-02-25 21:49:25 -- STEP: 35750/36938 -- GLOBAL_STEP: 959200 | > loss_text_ce: 0.03821055218577385 (0.03300342430196475) | > loss_mel_ce: 2.833756446838379 (nan) | > loss: 0.011396695859730244 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.1366880553185515) | > loader_time: 0.0046 (0.00468507947454918)  --> TIME: 2025-02-25 21:49:35 -- STEP: 35800/36938 -- GLOBAL_STEP: 959250 | > loss_text_ce: 0.03637223318219185 (0.033005129606866045) | > loss_mel_ce: 1.7790355682373047 (nan) | > loss: 0.007203999441117048 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13669155846094874) | > loader_time: 0.0047 (0.004684981846942568)  --> TIME: 2025-02-25 21:49:45 -- STEP: 35850/36938 -- GLOBAL_STEP: 959300 | > loss_text_ce: 0.029031824320554733 (0.03300300346174345) | > loss_mel_ce: 2.970949649810791 (nan) | > loss: 0.01190468855202198 (nan) | > current_lr: 5e-06 | > step_time: 0.1777 (0.13670174038060956) | > loader_time: 0.0056 (0.004684937003599867)  --> TIME: 2025-02-25 21:49:54 -- STEP: 35900/36938 -- GLOBAL_STEP: 959350 | > loss_text_ce: 0.0336831733584404 (0.03300306890210413) | > loss_mel_ce: 2.58449125289917 (nan) | > loss: 0.010389581322669983 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.1366946579021987) | > loader_time: 0.0045 (0.004684812670630648)  --> TIME: 2025-02-25 21:50:03 -- STEP: 35950/36938 -- GLOBAL_STEP: 959400 | > loss_text_ce: 0.03061768040060997 (0.03300283050533722) | > loss_mel_ce: 3.257380485534668 (nan) | > loss: 0.013047612272202969 (nan) | > current_lr: 5e-06 | > step_time: 0.1703 (0.1366982495768171) | > loader_time: 0.0048 (0.004684735922886351)  --> TIME: 2025-02-25 21:50:12 -- STEP: 36000/36938 -- GLOBAL_STEP: 959450 | > loss_text_ce: 0.024712586775422096 (0.03300429822239333) | > loss_mel_ce: 2.745387315750122 (nan) | > loss: 0.010992460884153843 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13669570861260005) | > loader_time: 0.0045 (0.004684596392843443)  --> TIME: 2025-02-25 21:50:22 -- STEP: 36050/36938 -- GLOBAL_STEP: 959500 | > loss_text_ce: 0.037450160831213 (0.033004837275757634) | > loss_mel_ce: 0.041015625 (nan) | > loss: 0.0003113721904810518 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13670117166601084) | > loader_time: 0.0046 (0.004684492301676359)  --> TIME: 2025-02-25 21:50:31 -- STEP: 36100/36938 -- GLOBAL_STEP: 959550 | > loss_text_ce: 0.032058630138635635 (0.033001253321041694) | > loss_mel_ce: 3.171875 (nan) | > loss: 0.0127140237018466 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13669831130643215) | > loader_time: 0.0047 (0.00468437825543727)  --> TIME: 2025-02-25 21:50:40 -- STEP: 36150/36938 -- GLOBAL_STEP: 959600 | > loss_text_ce: 0.03205457702279091 (0.0330008475628665) | > loss_mel_ce: 3.069019079208374 (nan) | > loss: 0.012305849231779575 (nan) | > current_lr: 5e-06 | > step_time: 0.1357 (0.13670242469175478) | > loader_time: 0.0046 (0.0046843195752978685)  --> TIME: 2025-02-25 21:50:50 -- STEP: 36200/36938 -- GLOBAL_STEP: 959650 | > loss_text_ce: 0.041103534400463104 (0.03300165155592473) | > loss_mel_ce: 3.636914014816284 (nan) | > loss: 0.014595309272408485 (nan) | > current_lr: 5e-06 | > step_time: 0.1451 (0.13670758543752098) | > loader_time: 0.0046 (0.004684272062712598)  --> TIME: 2025-02-25 21:50:59 -- STEP: 36250/36938 -- GLOBAL_STEP: 959700 | > loss_text_ce: 0.03241083398461342 (0.03300129788303406) | > loss_mel_ce: 0.49152833223342896 (nan) | > loss: 0.0020791238639503717 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13669763353281927) | > loader_time: 0.0047 (0.004684150840496185)  --> TIME: 2025-02-25 21:51:08 -- STEP: 36300/36938 -- GLOBAL_STEP: 959750 | > loss_text_ce: 0.028305942192673683 (0.03300117350921023) | > loss_mel_ce: 2.3938076496124268 (nan) | > loss: 0.009611562825739384 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1366952032556868) | > loader_time: 0.0045 (0.004684191012842258)  --> TIME: 2025-02-25 21:51:17 -- STEP: 36350/36938 -- GLOBAL_STEP: 959800 | > loss_text_ce: 0.03619512543082237 (0.0330003634154246) | > loss_mel_ce: 2.2064876556396484 (nan) | > loss: 0.008899535052478313 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.1366937722333352) | > loader_time: 0.0046 (0.0046842690839072115)  --> TIME: 2025-02-25 21:51:26 -- STEP: 36400/36938 -- GLOBAL_STEP: 959850 | > loss_text_ce: 0.03541472181677818 (0.032997705161169566) | > loss_mel_ce: 2.4414215087890625 (nan) | > loss: 0.009828715585172176 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.1366870777292558) | > loader_time: 0.0046 (0.004684265707875341)  --> TIME: 2025-02-25 21:51:36 -- STEP: 36450/36938 -- GLOBAL_STEP: 959900 | > loss_text_ce: 0.03008914180099964 (0.032998859596531725) | > loss_mel_ce: 2.9782979488372803 (nan) | > loss: 0.011938044801354408 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13668577805140963) | > loader_time: 0.0046 (0.00468422587516376)  --> TIME: 2025-02-25 21:51:45 -- STEP: 36500/36938 -- GLOBAL_STEP: 959950 | > loss_text_ce: 0.0312819741666317 (0.03299721843832184) | > loss_mel_ce: 1.8196851015090942 (nan) | > loss: 0.007345107849687338 (nan) | > current_lr: 5e-06 | > step_time: 0.1577 (0.13668558641329126) | > loader_time: 0.0047 (0.004684152805641903)  --> TIME: 2025-02-25 21:51:54 -- STEP: 36550/36938 -- GLOBAL_STEP: 960000 | > loss_text_ce: 0.028451453894376755 (0.0329982565930072) | > loss_mel_ce: 3.517578125 (nan) | > loss: 0.014071546494960785 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13668088899917624) | > loader_time: 0.0046 (0.004684022832993117) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_960000.pth  --> TIME: 2025-02-25 21:52:09 -- STEP: 36600/36938 -- GLOBAL_STEP: 960050 | > loss_text_ce: 0.03560750558972359 (0.03299836287250754) | > loss_mel_ce: 1.979175090789795 (nan) | > loss: 0.00799516960978508 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.13667973686437046) | > loader_time: 0.0047 (0.004684185656042045)  --> TIME: 2025-02-25 21:52:18 -- STEP: 36650/36938 -- GLOBAL_STEP: 960100 | > loss_text_ce: 0.030718863010406494 (0.032997980081827964) | > loss_mel_ce: 2.5377755165100098 (nan) | > loss: 0.010192438028752804 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13667475522784134) | > loader_time: 0.0052 (0.004684197931953211)  --> TIME: 2025-02-25 21:52:28 -- STEP: 36700/36938 -- GLOBAL_STEP: 960150 | > loss_text_ce: 0.03260786458849907 (0.03299939230232846) | > loss_mel_ce: 2.6916615962982178 (nan) | > loss: 0.010810593143105507 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.13668314380931507) | > loader_time: 0.0049 (0.004684313642881224)  --> TIME: 2025-02-25 21:52:37 -- STEP: 36750/36938 -- GLOBAL_STEP: 960200 | > loss_text_ce: 0.03207634389400482 (0.03299856720906563) | > loss_mel_ce: 0.8132193684577942 (nan) | > loss: 0.0033543482422828674 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13667654092619932) | > loader_time: 0.0046 (0.004684254594400629)  --> TIME: 2025-02-25 21:52:46 -- STEP: 36800/36938 -- GLOBAL_STEP: 960250 | > loss_text_ce: 0.038550280034542084 (0.03299799715339575) | > loss_mel_ce: 2.133992910385132 (nan) | > loss: 0.00862120371311903 (nan) | > current_lr: 5e-06 | > step_time: 0.1583 (0.1366719004576612) | > loader_time: 0.0047 (0.004684237766525019)  --> TIME: 2025-02-25 21:52:56 -- STEP: 36850/36938 -- GLOBAL_STEP: 960300 | > loss_text_ce: 0.039286691695451736 (0.03299837082127788) | > loss_mel_ce: 2.9593312740325928 (nan) | > loss: 0.011899277567863464 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1366778333992606) | > loader_time: 0.0046 (0.004684405669751939)  --> TIME: 2025-02-25 21:53:05 -- STEP: 36900/36938 -- GLOBAL_STEP: 960350 | > loss_text_ce: 0.032836172729730606 (0.03300050751403261) | > loss_mel_ce: 3.0074245929718018 (nan) | > loss: 0.012064527720212936 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.1366747728272819) | > loader_time: 0.0046 (0.004684398452117836)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.020979299403653298 (+0.0003713935908704821) | > avg_loss_text_ce: 0.03214991503017076 (-0.00025401843508871674) | > avg_loss_mel_ce: 2.500662212619686 (-0.008813929439773105) | > avg_loss: 2.5328121273824467 (-0.009067939649714862) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_960388.pth  > EPOCH: 27/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-25 21:53:37)   --> TIME: 2025-02-25 21:53:40 -- STEP: 12/36938 -- GLOBAL_STEP: 960400 | > loss_text_ce: 0.039072684943675995 (0.03023483557626605) | > loss_mel_ce: 2.709871768951416 (2.3736234505971274) | > loss: 0.010908510535955429 (0.009539120742071342) | > current_lr: 5e-06 | > step_time: 0.1806 (0.13596399625142416) | > loader_time: 0.0045 (0.0047254761060078945)  --> TIME: 2025-02-25 21:53:49 -- STEP: 62/36938 -- GLOBAL_STEP: 960450 | > loss_text_ce: 0.016291778534650803 (0.03273576074430058) | > loss_mel_ce: 1.4603819847106934 (2.4084693165796427) | > loss: 0.005859816912561655 (0.009687322335066124) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13187362686280285) | > loader_time: 0.0049 (0.00472821343329645)  --> TIME: 2025-02-25 21:53:58 -- STEP: 112/36938 -- GLOBAL_STEP: 960500 | > loss_text_ce: 0.04069295898079872 (0.03295480433319297) | > loss_mel_ce: 2.3496768474578857 (2.4315403171681944) | > loss: 0.009485594928264618 (0.00977974313328949) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13199903283800402) | > loader_time: 0.0046 (0.004731497594288417)  --> TIME: 2025-02-25 21:54:08 -- STEP: 162/36938 -- GLOBAL_STEP: 960550 | > loss_text_ce: 0.03477306663990021 (0.03281954483899439) | > loss_mel_ce: 2.718609094619751 (nan) | > loss: 0.010926120914518833 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13476068149378262) | > loader_time: 0.0047 (0.0047400939611741055)  --> TIME: 2025-02-25 21:54:16 -- STEP: 212/36938 -- GLOBAL_STEP: 960600 | > loss_text_ce: 0.04008388891816139 (0.03295213698391928) | > loss_mel_ce: 3.3827695846557617 (nan) | > loss: 0.013582753017544746 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13341132199989167) | > loader_time: 0.0047 (0.0047414820149259745)  --> TIME: 2025-02-25 21:54:25 -- STEP: 262/36938 -- GLOBAL_STEP: 960650 | > loss_text_ce: 0.033934518694877625 (0.03303179601278472) | > loss_mel_ce: 2.4361226558685303 (nan) | > loss: 0.009801815263926983 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13300298734475627) | > loader_time: 0.0047 (0.004731328432796566)  --> TIME: 2025-02-25 21:54:35 -- STEP: 312/36938 -- GLOBAL_STEP: 960700 | > loss_text_ce: 0.0335417315363884 (0.032975518464063054) | > loss_mel_ce: 3.3278777599334717 (nan) | > loss: 0.013338967226445675 (nan) | > current_lr: 5e-06 | > step_time: 0.1628 (0.1329692640365697) | > loader_time: 0.0047 (0.004725420322173681)  --> TIME: 2025-02-25 21:54:44 -- STEP: 362/36938 -- GLOBAL_STEP: 960750 | > loss_text_ce: 0.03296627476811409 (0.032908437058370744) | > loss_mel_ce: 3.012923002243042 (nan) | > loss: 0.01208686362951994 (nan) | > current_lr: 5e-06 | > step_time: 0.1812 (0.1338364432529847) | > loader_time: 0.0048 (0.00471877788311869)  --> TIME: 2025-02-25 21:54:54 -- STEP: 412/36938 -- GLOBAL_STEP: 960800 | > loss_text_ce: 0.029513290151953697 (0.032977734036852134) | > loss_mel_ce: 2.926666259765625 (nan) | > loss: 0.011730872094631195 (nan) | > current_lr: 5e-06 | > step_time: 0.1963 (0.1345519481353386) | > loader_time: 0.0049 (0.004721219678526947)  --> TIME: 2025-02-25 21:55:03 -- STEP: 462/36938 -- GLOBAL_STEP: 960850 | > loss_text_ce: 0.029837165027856827 (0.03278631588960171) | > loss_mel_ce: 2.7821044921875 (nan) | > loss: 0.01115849893540144 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13480844177724965) | > loader_time: 0.0046 (0.004714930728400428)  --> TIME: 2025-02-25 21:55:12 -- STEP: 512/36938 -- GLOBAL_STEP: 960900 | > loss_text_ce: 0.03972376510500908 (0.03288505249838635) | > loss_mel_ce: 3.0606865882873535 (nan) | > loss: 0.012303216382861137 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.1348813399672506) | > loader_time: 0.0049 (0.00471586221829057)  --> TIME: 2025-02-25 21:55:22 -- STEP: 562/36938 -- GLOBAL_STEP: 960950 | > loss_text_ce: 0.0267727542668581 (0.03285391339899646) | > loss_mel_ce: 2.569239616394043 (nan) | > loss: 0.010301636531949043 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13593407372987126) | > loader_time: 0.0046 (0.004716137974287693)  --> TIME: 2025-02-25 21:55:31 -- STEP: 612/36938 -- GLOBAL_STEP: 961000 | > loss_text_ce: 0.043681614100933075 (0.032851263879207906) | > loss_mel_ce: 2.8700854778289795 (nan) | > loss: 0.011562568135559559 (nan) | > current_lr: 5e-06 | > step_time: 0.1819 (0.13591103226530776) | > loader_time: 0.0049 (0.004713361948923349)  --> TIME: 2025-02-25 21:55:41 -- STEP: 662/36938 -- GLOBAL_STEP: 961050 | > loss_text_ce: 0.0313514806330204 (0.03288630931145786) | > loss_mel_ce: 2.885190486907959 (nan) | > loss: 0.011573580093681812 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13583894114479905) | > loader_time: 0.0047 (0.004709833338181414)  --> TIME: 2025-02-25 21:55:50 -- STEP: 712/36938 -- GLOBAL_STEP: 961100 | > loss_text_ce: 0.024643059819936752 (0.03284590612976607) | > loss_mel_ce: 0.0361328125 (nan) | > loss: 0.00024117411521729082 (nan) | > current_lr: 5e-06 | > step_time: 0.181 (0.1360913756188381) | > loader_time: 0.0049 (0.0047061667683419165)  --> TIME: 2025-02-25 21:56:00 -- STEP: 762/36938 -- GLOBAL_STEP: 961150 | > loss_text_ce: 0.03569953143596649 (0.03284978899454542) | > loss_mel_ce: 2.8302090167999268 (nan) | > loss: 0.011372653767466545 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13633721550618563) | > loader_time: 0.0046 (0.004709512229979504)  --> TIME: 2025-02-25 21:56:09 -- STEP: 812/36938 -- GLOBAL_STEP: 961200 | > loss_text_ce: 0.028868993744254112 (0.03283776303069672) | > loss_mel_ce: 2.491614580154419 (nan) | > loss: 0.010001919232308865 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13600105252759195) | > loader_time: 0.0046 (0.004708566982757867)  --> TIME: 2025-02-25 21:56:18 -- STEP: 862/36938 -- GLOBAL_STEP: 961250 | > loss_text_ce: 0.030290309339761734 (0.03282744430217312) | > loss_mel_ce: 2.7556979656219482 (nan) | > loss: 0.011055509559810162 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13603062834374577) | > loader_time: 0.0046 (0.004706869545779481)  --> TIME: 2025-02-25 21:56:28 -- STEP: 912/36938 -- GLOBAL_STEP: 961300 | > loss_text_ce: 0.03400425985455513 (0.03276690805694415) | > loss_mel_ce: 2.847405433654785 (nan) | > loss: 0.011434165760874748 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13629016332459018) | > loader_time: 0.0067 (0.004706453074488723)  --> TIME: 2025-02-25 21:56:37 -- STEP: 962/36938 -- GLOBAL_STEP: 961350 | > loss_text_ce: 0.02192547731101513 (0.03279431245489174) | > loss_mel_ce: 0.004913330078125 (nan) | > loss: 0.00010650321200955659 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13626159253586337) | > loader_time: 0.0048 (0.00470477107161048)  --> TIME: 2025-02-25 21:56:46 -- STEP: 1012/36938 -- GLOBAL_STEP: 961400 | > loss_text_ce: 0.03728620335459709 (0.03282466197750815) | > loss_mel_ce: 2.0693023204803467 (nan) | > loss: 0.008359478786587715 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13599059327317775) | > loader_time: 0.0047 (0.004701412713574797)  --> TIME: 2025-02-25 21:56:56 -- STEP: 1062/36938 -- GLOBAL_STEP: 961450 | > loss_text_ce: 0.03675317019224167 (0.032789581573678776) | > loss_mel_ce: 2.9954917430877686 (nan) | > loss: 0.012032718397676945 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13620248711939795) | > loader_time: 0.0046 (0.004707801813459666)  --> TIME: 2025-02-25 21:57:05 -- STEP: 1112/36938 -- GLOBAL_STEP: 961500 | > loss_text_ce: 0.03130673989653587 (0.032803757792077406) | > loss_mel_ce: 2.8054206371307373 (nan) | > loss: 0.011256854981184006 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.136298399820602) | > loader_time: 0.0048 (0.004706729444668449)  --> TIME: 2025-02-25 21:57:14 -- STEP: 1162/36938 -- GLOBAL_STEP: 961550 | > loss_text_ce: 0.029523147270083427 (0.03276060171515106) | > loss_mel_ce: 2.921013355255127 (nan) | > loss: 0.011708478443324566 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13631241723715518) | > loader_time: 0.0046 (0.004703068692179434)  --> TIME: 2025-02-25 21:57:24 -- STEP: 1212/36938 -- GLOBAL_STEP: 961600 | > loss_text_ce: 0.036611415445804596 (0.03283430784541138) | > loss_mel_ce: 0.8073577880859375 (nan) | > loss: 0.0033490844070911407 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.13649227477536321) | > loader_time: 0.0048 (0.004701816996332053)  --> TIME: 2025-02-25 21:57:33 -- STEP: 1262/36938 -- GLOBAL_STEP: 961650 | > loss_text_ce: 0.03511793166399002 (0.032829071780321295) | > loss_mel_ce: 0.0894775390625 (nan) | > loss: 0.0004944265238009393 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13651362444065998) | > loader_time: 0.0046 (0.0047024019924472174)  --> TIME: 2025-02-25 21:57:42 -- STEP: 1312/36938 -- GLOBAL_STEP: 961700 | > loss_text_ce: 0.0313798151910305 (0.03282336260493069) | > loss_mel_ce: 3.1609253883361816 (nan) | > loss: 0.012667877599596977 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.1364461169010254) | > loader_time: 0.0047 (0.004703183363123642)  --> TIME: 2025-02-25 21:57:52 -- STEP: 1362/36938 -- GLOBAL_STEP: 961750 | > loss_text_ce: 0.03478487953543663 (0.032819713245688886) | > loss_mel_ce: 3.0685222148895264 (nan) | > loss: 0.012314710766077042 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13650305155615428) | > loader_time: 0.0047 (0.0047034135473982405)  --> TIME: 2025-02-25 21:58:02 -- STEP: 1412/36938 -- GLOBAL_STEP: 961800 | > loss_text_ce: 0.03459671512246132 (0.03284681268648451) | > loss_mel_ce: 2.985185146331787 (nan) | > loss: 0.011983262374997139 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13675859747797484) | > loader_time: 0.0047 (0.004703959898637988)  --> TIME: 2025-02-25 21:58:11 -- STEP: 1462/36938 -- GLOBAL_STEP: 961850 | > loss_text_ce: 0.03297213464975357 (0.03290123675686148) | > loss_mel_ce: 2.993230104446411 (nan) | > loss: 0.01200873963534832 (nan) | > current_lr: 5e-06 | > step_time: 0.186 (0.1367296675161525) | > loader_time: 0.0048 (0.004701872367702339)  --> TIME: 2025-02-25 21:58:20 -- STEP: 1512/36938 -- GLOBAL_STEP: 961900 | > loss_text_ce: 0.03895851597189903 (0.032887201476595816) | > loss_mel_ce: 3.0566623210906982 (nan) | > loss: 0.01228421088308096 (nan) | > current_lr: 5e-06 | > step_time: 0.1618 (0.13673735216811841) | > loader_time: 0.0048 (0.0047021183387312366)  --> TIME: 2025-02-25 21:58:30 -- STEP: 1562/36938 -- GLOBAL_STEP: 961950 | > loss_text_ce: 0.029278848320245743 (0.03289343645407437) | > loss_mel_ce: 2.8940558433532715 (nan) | > loss: 0.011600534431636333 (nan) | > current_lr: 5e-06 | > step_time: 0.1476 (0.1368538220621712) | > loader_time: 0.0047 (0.004701704893466306)  --> TIME: 2025-02-25 21:58:39 -- STEP: 1612/36938 -- GLOBAL_STEP: 962000 | > loss_text_ce: 0.02844947576522827 (0.03288851099992381) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13678338776155377) | > loader_time: 0.0045 (0.004700815115613922)  --> TIME: 2025-02-25 21:58:48 -- STEP: 1662/36938 -- GLOBAL_STEP: 962050 | > loss_text_ce: 0.032892126590013504 (0.03291815570045696) | > loss_mel_ce: 2.239423990249634 (nan) | > loss: 0.009017128497362137 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1366794278618804) | > loader_time: 0.0046 (0.004697992196295385)  --> TIME: 2025-02-25 21:58:57 -- STEP: 1712/36938 -- GLOBAL_STEP: 962100 | > loss_text_ce: 0.04661836475133896 (0.032896512590031006) | > loss_mel_ce: 2.4037158489227295 (nan) | > loss: 0.009723549708724022 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13668127065507044) | > loader_time: 0.0049 (0.004696108331190098)  --> TIME: 2025-02-25 21:59:07 -- STEP: 1762/36938 -- GLOBAL_STEP: 962150 | > loss_text_ce: 0.029042117297649384 (0.03286931213154271) | > loss_mel_ce: 2.502387285232544 (nan) | > loss: 0.010045355185866356 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13660695062998862) | > loader_time: 0.0046 (0.004697699010981069)  --> TIME: 2025-02-25 21:59:16 -- STEP: 1812/36938 -- GLOBAL_STEP: 962200 | > loss_text_ce: 0.03579401597380638 (0.03287033954990933) | > loss_mel_ce: 2.9005329608917236 (nan) | > loss: 0.011652091518044472 (nan) | > current_lr: 5e-06 | > step_time: 0.17 (0.13656713315193222) | > loader_time: 0.0048 (0.004696839166266508)  --> TIME: 2025-02-25 21:59:25 -- STEP: 1862/36938 -- GLOBAL_STEP: 962250 | > loss_text_ce: 0.04012990742921829 (0.03285801035595134) | > loss_mel_ce: 1.4342198371887207 (nan) | > loss: 0.005850594490766525 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13653328288126465) | > loader_time: 0.0047 (0.004696005653233846)  --> TIME: 2025-02-25 21:59:35 -- STEP: 1912/36938 -- GLOBAL_STEP: 962300 | > loss_text_ce: 0.041619833558797836 (0.03286891995108914) | > loss_mel_ce: 2.4320616722106934 (nan) | > loss: 0.009816196747124195 (nan) | > current_lr: 5e-06 | > step_time: 0.1818 (0.13660766415017422) | > loader_time: 0.0049 (0.00469721772680722)  --> TIME: 2025-02-25 21:59:44 -- STEP: 1962/36938 -- GLOBAL_STEP: 962350 | > loss_text_ce: 0.03383462876081467 (0.03282737241066783) | > loss_mel_ce: 1.5192718505859375 (nan) | > loss: 0.006163121201097965 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13661112782908996) | > loader_time: 0.0046 (0.00469640137344812)  --> TIME: 2025-02-25 21:59:54 -- STEP: 2012/36938 -- GLOBAL_STEP: 962400 | > loss_text_ce: 0.02631288394331932 (0.0328275760895874) | > loss_mel_ce: 2.753603935241699 (nan) | > loss: 0.01103141624480486 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.13671076439957972) | > loader_time: 0.0046 (0.004695511006455777)  --> TIME: 2025-02-25 22:00:03 -- STEP: 2062/36938 -- GLOBAL_STEP: 962450 | > loss_text_ce: 0.03150554001331329 (0.0328488394615324) | > loss_mel_ce: 0.089202880859375 (nan) | > loss: 0.00047900169738568366 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13668730036941812) | > loader_time: 0.0047 (0.0046944010038033656)  --> TIME: 2025-02-25 22:00:12 -- STEP: 2112/36938 -- GLOBAL_STEP: 962500 | > loss_text_ce: 0.02948138117790222 (0.032836470174846547) | > loss_mel_ce: 2.8093433380126953 (nan) | > loss: 0.011265178211033344 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.13656997996749304) | > loader_time: 0.0047 (0.0046937842712257895)  --> TIME: 2025-02-25 22:00:21 -- STEP: 2162/36938 -- GLOBAL_STEP: 962550 | > loss_text_ce: 0.027824634686112404 (0.03283081312295973) | > loss_mel_ce: 2.248137950897217 (nan) | > loss: 0.009031598456203938 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13659288064954905) | > loader_time: 0.0045 (0.004692547195603961)  --> TIME: 2025-02-25 22:00:31 -- STEP: 2212/36938 -- GLOBAL_STEP: 962600 | > loss_text_ce: 0.033138878643512726 (0.032826773842938126) | > loss_mel_ce: 1.9359928369522095 (nan) | > loss: 0.0078140152618289 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13656389476162187) | > loader_time: 0.0046 (0.00469364988437398)  --> TIME: 2025-02-25 22:00:40 -- STEP: 2262/36938 -- GLOBAL_STEP: 962650 | > loss_text_ce: 0.036305021494627 (0.03284470201449369) | > loss_mel_ce: 3.112438440322876 (nan) | > loss: 0.012495013885200024 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13661681699289047) | > loader_time: 0.0047 (0.0046932378983097186)  --> TIME: 2025-02-25 22:00:49 -- STEP: 2312/36938 -- GLOBAL_STEP: 962700 | > loss_text_ce: 0.03695692867040634 (0.032862467838332014) | > loss_mel_ce: 3.076725721359253 (nan) | > loss: 0.012355884537100792 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.13657744244308212) | > loader_time: 0.005 (0.004693387804559369)  --> TIME: 2025-02-25 22:00:59 -- STEP: 2362/36938 -- GLOBAL_STEP: 962750 | > loss_text_ce: 0.02007230557501316 (0.0328589721092328) | > loss_mel_ce: 2.0894198417663574 (nan) | > loss: 0.008371001109480858 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13653606928373982) | > loader_time: 0.0046 (0.004693048571652025)  --> TIME: 2025-02-25 22:01:08 -- STEP: 2412/36938 -- GLOBAL_STEP: 962800 | > loss_text_ce: 0.038740068674087524 (0.03282381542479528) | > loss_mel_ce: 2.955510139465332 (nan) | > loss: 0.011881946586072445 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.13657549671075053) | > loader_time: 0.0048 (0.004696009566337123)  --> TIME: 2025-02-25 22:01:17 -- STEP: 2462/36938 -- GLOBAL_STEP: 962850 | > loss_text_ce: 0.030453357845544815 (0.0328148438710232) | > loss_mel_ce: 2.691629648208618 (nan) | > loss: 0.010801917873322964 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13650661004838877) | > loader_time: 0.0045 (0.004694950570912991)  --> TIME: 2025-02-25 22:01:26 -- STEP: 2512/36938 -- GLOBAL_STEP: 962900 | > loss_text_ce: 0.04077848419547081 (0.03281275125718703) | > loss_mel_ce: 2.8080971240997314 (nan) | > loss: 0.011305062100291252 (nan) | > current_lr: 5e-06 | > step_time: 0.1271 (0.13643002823279937) | > loader_time: 0.0046 (0.004694404116102089)  --> TIME: 2025-02-25 22:01:36 -- STEP: 2562/36938 -- GLOBAL_STEP: 962950 | > loss_text_ce: 0.03614168241620064 (0.032816999926319756) | > loss_mel_ce: 3.025398015975952 (nan) | > loss: 0.012148967944085598 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13650833276172455) | > loader_time: 0.0045 (0.0046940251870791925)  --> TIME: 2025-02-25 22:01:45 -- STEP: 2612/36938 -- GLOBAL_STEP: 963000 | > loss_text_ce: 0.03680362552404404 (0.0328314649964473) | > loss_mel_ce: 2.903618335723877 (nan) | > loss: 0.011668342165648937 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1364195230909002) | > loader_time: 0.0046 (0.004693525947436444)  --> TIME: 2025-02-25 22:01:54 -- STEP: 2662/36938 -- GLOBAL_STEP: 963050 | > loss_text_ce: 0.029622742906212807 (0.03280105900157951) | > loss_mel_ce: 2.4298086166381836 (nan) | > loss: 0.009759648703038692 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.1363808772870244) | > loader_time: 0.0048 (0.004692272079519367)  --> TIME: 2025-02-25 22:02:03 -- STEP: 2712/36938 -- GLOBAL_STEP: 963100 | > loss_text_ce: 0.04873301833868027 (0.032809966928902506) | > loss_mel_ce: 2.528156280517578 (nan) | > loss: 0.0102257514372468 (nan) | > current_lr: 5e-06 | > step_time: 0.1077 (0.13643374036898664) | > loader_time: 0.0045 (0.004693364499241207)  --> TIME: 2025-02-25 22:02:13 -- STEP: 2762/36938 -- GLOBAL_STEP: 963150 | > loss_text_ce: 0.03234345465898514 (0.032806201654187614) | > loss_mel_ce: 2.844661235809326 (nan) | > loss: 0.01141668576747179 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1364468081458422) | > loader_time: 0.0047 (0.004693749588007177)  --> TIME: 2025-02-25 22:02:22 -- STEP: 2812/36938 -- GLOBAL_STEP: 963200 | > loss_text_ce: 0.039380501955747604 (0.03282355239706254) | > loss_mel_ce: 2.8796846866607666 (nan) | > loss: 0.011583592742681503 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.13642319710461562) | > loader_time: 0.0046 (0.004694091731080972)  --> TIME: 2025-02-25 22:02:31 -- STEP: 2862/36938 -- GLOBAL_STEP: 963250 | > loss_text_ce: 0.030381185933947563 (0.03286442092628294) | > loss_mel_ce: 3.571890115737915 (nan) | > loss: 0.014294728636741638 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13645907034830665) | > loader_time: 0.0045 (0.004693319795849778)  --> TIME: 2025-02-25 22:02:41 -- STEP: 2912/36938 -- GLOBAL_STEP: 963300 | > loss_text_ce: 0.025706760585308075 (0.032856763702891464) | > loss_mel_ce: 2.1498565673828125 (nan) | > loss: 0.008633187972009182 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.1364656008861879) | > loader_time: 0.0044 (0.00469210138032725)  --> TIME: 2025-02-25 22:02:50 -- STEP: 2962/36938 -- GLOBAL_STEP: 963350 | > loss_text_ce: 0.013410847634077072 (0.03285473194268861) | > loss_mel_ce: 2.149714469909668 (nan) | > loss: 0.008583830669522285 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13644117692449) | > loader_time: 0.0044 (0.004690922811818561)  --> TIME: 2025-02-25 22:02:59 -- STEP: 3012/36938 -- GLOBAL_STEP: 963400 | > loss_text_ce: 0.027517762035131454 (0.03285588357865981) | > loss_mel_ce: 3.0277175903320312 (nan) | > loss: 0.012123950757086277 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13648058495831863) | > loader_time: 0.0045 (0.004689946909191443)  --> TIME: 2025-02-25 22:03:09 -- STEP: 3062/36938 -- GLOBAL_STEP: 963450 | > loss_text_ce: 0.030440902337431908 (0.03284509072766428) | > loss_mel_ce: 3.0111494064331055 (nan) | > loss: 0.012069802731275558 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.13657554039995654) | > loader_time: 0.0047 (0.004691290746398571)  --> TIME: 2025-02-25 22:03:18 -- STEP: 3112/36938 -- GLOBAL_STEP: 963500 | > loss_text_ce: 0.03829856961965561 (0.032855173607313735) | > loss_mel_ce: 1.9185904264450073 (nan) | > loss: 0.00776543328538537 (nan) | > current_lr: 5e-06 | > step_time: 0.1075 (0.13659446458583654) | > loader_time: 0.0044 (0.004692662574331684)  --> TIME: 2025-02-25 22:03:28 -- STEP: 3162/36938 -- GLOBAL_STEP: 963550 | > loss_text_ce: 0.024875182658433914 (0.032839508919524835) | > loss_mel_ce: 2.34671688079834 (nan) | > loss: 0.009411079809069633 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.1366123952570029) | > loader_time: 0.0044 (0.004693382906808196)  --> TIME: 2025-02-25 22:03:37 -- STEP: 3212/36938 -- GLOBAL_STEP: 963600 | > loss_text_ce: 0.025591179728507996 (0.0328362210381748) | > loss_mel_ce: 1.8685302734375 (nan) | > loss: 0.007516355253756046 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13658879821653847) | > loader_time: 0.0047 (0.004693612215082495)  --> TIME: 2025-02-25 22:03:47 -- STEP: 3262/36938 -- GLOBAL_STEP: 963650 | > loss_text_ce: 0.0241463053971529 (0.032831815404590195) | > loss_mel_ce: 1.9095038175582886 (nan) | > loss: 0.0076732151210308075 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13667420512105297) | > loader_time: 0.0048 (0.004693956991999079)  --> TIME: 2025-02-25 22:03:56 -- STEP: 3312/36938 -- GLOBAL_STEP: 963700 | > loss_text_ce: 0.037322998046875 (0.03282781697698345) | > loss_mel_ce: 2.471402168273926 (nan) | > loss: 0.009955259039998055 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13671627496751648) | > loader_time: 0.0045 (0.004695026698895695)  --> TIME: 2025-02-25 22:04:05 -- STEP: 3362/36938 -- GLOBAL_STEP: 963750 | > loss_text_ce: 0.02175932377576828 (0.03280568941967794) | > loss_mel_ce: 2.422386884689331 (nan) | > loss: 0.009698993526399136 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1367197121842005) | > loader_time: 0.0046 (0.004695793264753259)  --> TIME: 2025-02-25 22:04:15 -- STEP: 3412/36938 -- GLOBAL_STEP: 963800 | > loss_text_ce: 0.029493479058146477 (0.03278527428900468) | > loss_mel_ce: 3.1778671741485596 (nan) | > loss: 0.012727622874081135 (nan) | > current_lr: 5e-06 | > step_time: 0.1449 (0.13675580054067388) | > loader_time: 0.0046 (0.004697175615697507)  --> TIME: 2025-02-25 22:04:24 -- STEP: 3462/36938 -- GLOBAL_STEP: 963850 | > loss_text_ce: 0.034885600209236145 (0.032788261289938066) | > loss_mel_ce: 2.2999439239501953 (nan) | > loss: 0.009265197440981865 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13672038821524002) | > loader_time: 0.0047 (0.0046965728806044466)  --> TIME: 2025-02-25 22:04:33 -- STEP: 3512/36938 -- GLOBAL_STEP: 963900 | > loss_text_ce: 0.042303357273340225 (0.03278390193492777) | > loss_mel_ce: 1.3405537605285645 (nan) | > loss: 0.005487528629601002 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.1367239791195597) | > loader_time: 0.0045 (0.004697057136370546)  --> TIME: 2025-02-25 22:04:43 -- STEP: 3562/36938 -- GLOBAL_STEP: 963950 | > loss_text_ce: 0.03981835022568703 (0.03277104214468635) | > loss_mel_ce: 2.269432544708252 (nan) | > loss: 0.009163694456219673 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13671157127146943) | > loader_time: 0.0046 (0.004696643667097913)  --> TIME: 2025-02-25 22:04:52 -- STEP: 3612/36938 -- GLOBAL_STEP: 964000 | > loss_text_ce: 0.030542707070708275 (0.032778770695590015) | > loss_mel_ce: 3.340216875076294 (nan) | > loss: 0.013376030139625072 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13670116313140182) | > loader_time: 0.0049 (0.004696036296033386)  --> TIME: 2025-02-25 22:05:01 -- STEP: 3662/36938 -- GLOBAL_STEP: 964050 | > loss_text_ce: 0.0337873175740242 (0.03275655926872047) | > loss_mel_ce: 1.9801199436187744 (nan) | > loss: 0.007991695776581764 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13668992798032878) | > loader_time: 0.0047 (0.00469588745102604)  --> TIME: 2025-02-25 22:05:11 -- STEP: 3712/36938 -- GLOBAL_STEP: 964100 | > loss_text_ce: 0.03213538974523544 (0.03275438515315885) | > loss_mel_ce: 2.44321346282959 (nan) | > loss: 0.009822813794016838 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13666463277206406) | > loader_time: 0.0045 (0.004695225442791809)  --> TIME: 2025-02-25 22:05:20 -- STEP: 3762/36938 -- GLOBAL_STEP: 964150 | > loss_text_ce: 0.03613097220659256 (0.03275522036757174) | > loss_mel_ce: 3.1389434337615967 (nan) | > loss: 0.012599501758813858 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.13676538320376175) | > loader_time: 0.0047 (0.004694858616428895)  --> TIME: 2025-02-25 22:05:29 -- STEP: 3812/36938 -- GLOBAL_STEP: 964200 | > loss_text_ce: 0.039804331958293915 (0.032751947813613624) | > loss_mel_ce: 2.779568910598755 (nan) | > loss: 0.011187989264726639 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.13670067378131265) | > loader_time: 0.0048 (0.004694001622114954)  --> TIME: 2025-02-25 22:05:38 -- STEP: 3862/36938 -- GLOBAL_STEP: 964250 | > loss_text_ce: 0.030632711946964264 (0.03274794184862758) | > loss_mel_ce: 0.0770263671875 (nan) | > loss: 0.0004272185906302184 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13663900182507754) | > loader_time: 0.0046 (0.004692935992991105)  --> TIME: 2025-02-25 22:05:48 -- STEP: 3912/36938 -- GLOBAL_STEP: 964300 | > loss_text_ce: 0.028733717277646065 (0.032748644245676084) | > loss_mel_ce: 3.2213218212127686 (nan) | > loss: 0.012897046282887459 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.1366079348484195) | > loader_time: 0.0047 (0.00469506975331921)  --> TIME: 2025-02-25 22:05:57 -- STEP: 3962/36938 -- GLOBAL_STEP: 964350 | > loss_text_ce: 0.03202423080801964 (0.03273598445111162) | > loss_mel_ce: 2.729722023010254 (nan) | > loss: 0.010959310457110405 (nan) | > current_lr: 5e-06 | > step_time: 0.1642 (0.1366340009808728) | > loader_time: 0.0048 (0.004694550101170934)  --> TIME: 2025-02-25 22:06:06 -- STEP: 4012/36938 -- GLOBAL_STEP: 964400 | > loss_text_ce: 0.028332462534308434 (0.032736352352603734) | > loss_mel_ce: 3.503535747528076 (nan) | > loss: 0.014015350490808487 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13664334596451425) | > loader_time: 0.0046 (0.004694774702325064)  --> TIME: 2025-02-25 22:06:16 -- STEP: 4062/36938 -- GLOBAL_STEP: 964450 | > loss_text_ce: 0.027376383543014526 (0.0327573991013789) | > loss_mel_ce: 2.029524803161621 (nan) | > loss: 0.008162306621670723 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.13666444661757043) | > loader_time: 0.0046 (0.004695317711165715)  --> TIME: 2025-02-25 22:06:25 -- STEP: 4112/36938 -- GLOBAL_STEP: 964500 | > loss_text_ce: 0.037786852568387985 (0.03276093481792543) | > loss_mel_ce: 2.2940378189086914 (nan) | > loss: 0.009253273718059063 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13669128773973216) | > loader_time: 0.0048 (0.004695106921029)  --> TIME: 2025-02-25 22:06:35 -- STEP: 4162/36938 -- GLOBAL_STEP: 964550 | > loss_text_ce: 0.03149542212486267 (0.03275096065493221) | > loss_mel_ce: 3.0339627265930176 (nan) | > loss: 0.012164516374468803 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13675116226685793) | > loader_time: 0.0046 (0.004695770947879351)  --> TIME: 2025-02-25 22:06:44 -- STEP: 4212/36938 -- GLOBAL_STEP: 964600 | > loss_text_ce: 0.03559623286128044 (0.03275526451642987) | > loss_mel_ce: 3.132617712020874 (nan) | > loss: 0.012572278268635273 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13674054073359163) | > loader_time: 0.0046 (0.004695107001751123)  --> TIME: 2025-02-25 22:06:53 -- STEP: 4262/36938 -- GLOBAL_STEP: 964650 | > loss_text_ce: 0.033259619027376175 (0.03276111455092313) | > loss_mel_ce: 3.2999672889709473 (nan) | > loss: 0.01322709210216999 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13669995990844536) | > loader_time: 0.0046 (0.004694540363027142)  --> TIME: 2025-02-25 22:07:03 -- STEP: 4312/36938 -- GLOBAL_STEP: 964700 | > loss_text_ce: 0.03261316195130348 (0.03274413661971683) | > loss_mel_ce: 0.09287261962890625 (nan) | > loss: 0.0004979594377800822 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.1367266979863344) | > loader_time: 0.0045 (0.004694486482687475)  --> TIME: 2025-02-25 22:07:12 -- STEP: 4362/36938 -- GLOBAL_STEP: 964750 | > loss_text_ce: 0.031495410948991776 (0.032730482116637094) | > loss_mel_ce: 2.975543737411499 (nan) | > loss: 0.011932695284485817 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.1367385084614853) | > loader_time: 0.0046 (0.004693825274418932)  --> TIME: 2025-02-25 22:07:22 -- STEP: 4412/36938 -- GLOBAL_STEP: 964800 | > loss_text_ce: 0.03389642387628555 (0.03271997428215459) | > loss_mel_ce: 3.0903091430664062 (nan) | > loss: 0.012397642247378826 (nan) | > current_lr: 5e-06 | > step_time: 0.1441 (0.13681668605138742) | > loader_time: 0.0047 (0.004693464917263333)  --> TIME: 2025-02-25 22:07:31 -- STEP: 4462/36938 -- GLOBAL_STEP: 964850 | > loss_text_ce: 0.036216750741004944 (0.03272262825899859) | > loss_mel_ce: 3.0179293155670166 (nan) | > loss: 0.012119627557694912 (nan) | > current_lr: 5e-06 | > step_time: 0.1343 (0.13678356143510334) | > loader_time: 0.0048 (0.004692849584995818)  --> TIME: 2025-02-25 22:07:40 -- STEP: 4512/36938 -- GLOBAL_STEP: 964900 | > loss_text_ce: 0.03440408036112785 (0.03272594161086439) | > loss_mel_ce: 1.922607421875 (nan) | > loss: 0.007765918970108032 (nan) | > current_lr: 5e-06 | > step_time: 0.1767 (0.13679945077879274) | > loader_time: 0.0048 (0.004692724304841762)  --> TIME: 2025-02-25 22:07:50 -- STEP: 4562/36938 -- GLOBAL_STEP: 964950 | > loss_text_ce: 0.03912739083170891 (0.03271162797480901) | > loss_mel_ce: 3.3365509510040283 (nan) | > loss: 0.013395549729466438 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13681674797981322) | > loader_time: 0.0049 (0.004693103955012563)  --> TIME: 2025-02-25 22:07:59 -- STEP: 4612/36938 -- GLOBAL_STEP: 965000 | > loss_text_ce: 0.04203757271170616 (0.032713014783308646) | > loss_mel_ce: 3.2410566806793213 (nan) | > loss: 0.013028152287006378 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13683533528940905) | > loader_time: 0.0045 (0.004693306743426616)  --> TIME: 2025-02-25 22:08:09 -- STEP: 4662/36938 -- GLOBAL_STEP: 965050 | > loss_text_ce: 0.03498854488134384 (0.03270431291108217) | > loss_mel_ce: 2.601137161254883 (nan) | > loss: 0.010460817255079746 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13685140835783777) | > loader_time: 0.0046 (0.004693306960143117)  --> TIME: 2025-02-25 22:08:18 -- STEP: 4712/36938 -- GLOBAL_STEP: 965100 | > loss_text_ce: 0.025345273315906525 (0.03270606705756839) | > loss_mel_ce: 2.7703781127929688 (nan) | > loss: 0.011094140820205212 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.1368765692354673) | > loader_time: 0.0047 (0.004693178450514779)  --> TIME: 2025-02-25 22:08:27 -- STEP: 4762/36938 -- GLOBAL_STEP: 965150 | > loss_text_ce: 0.034763410687446594 (0.03270704380237742) | > loss_mel_ce: 2.903047561645508 (nan) | > loss: 0.011657980270683765 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1368208044169086) | > loader_time: 0.0046 (0.004693745415233746)  --> TIME: 2025-02-25 22:08:37 -- STEP: 4812/36938 -- GLOBAL_STEP: 965200 | > loss_text_ce: 0.03407111391425133 (0.03270365160631354) | > loss_mel_ce: 2.982558250427246 (nan) | > loss: 0.011970752850174904 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.1368250208206215) | > loader_time: 0.0049 (0.00469416399747256)  --> TIME: 2025-02-25 22:08:46 -- STEP: 4862/36938 -- GLOBAL_STEP: 965250 | > loss_text_ce: 0.03305725008249283 (0.03270683807854557) | > loss_mel_ce: 1.6080589294433594 (nan) | > loss: 0.006512366235256195 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.1367887400737378) | > loader_time: 0.0046 (0.004693706944762595)  --> TIME: 2025-02-25 22:08:55 -- STEP: 4912/36938 -- GLOBAL_STEP: 965300 | > loss_text_ce: 0.035125732421875 (0.03270912870305138) | > loss_mel_ce: 2.8356034755706787 (nan) | > loss: 0.011391783133149147 (nan) | > current_lr: 5e-06 | > step_time: 0.1798 (0.13675106309524238) | > loader_time: 0.0048 (0.004693159014471956)  --> TIME: 2025-02-25 22:09:04 -- STEP: 4962/36938 -- GLOBAL_STEP: 965350 | > loss_text_ce: 0.03362935408949852 (0.032720656960403806) | > loss_mel_ce: 2.655078649520874 (nan) | > loss: 0.01066947728395462 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13674357902230153) | > loader_time: 0.0045 (0.004692968703334908)  --> TIME: 2025-02-25 22:09:13 -- STEP: 5012/36938 -- GLOBAL_STEP: 965400 | > loss_text_ce: 0.03664172440767288 (0.032711855009501425) | > loss_mel_ce: 1.518945336341858 (nan) | > loss: 0.0061729648150503635 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.1367029383100711) | > loader_time: 0.0047 (0.004692247269730126)  --> TIME: 2025-02-25 22:09:22 -- STEP: 5062/36938 -- GLOBAL_STEP: 965450 | > loss_text_ce: 0.022819962352514267 (0.032693278992711204) | > loss_mel_ce: 2.5643484592437744 (nan) | > loss: 0.010266542434692383 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1366711602009417) | > loader_time: 0.0045 (0.004692539731677166)  --> TIME: 2025-02-25 22:09:32 -- STEP: 5112/36938 -- GLOBAL_STEP: 965500 | > loss_text_ce: 0.024412179365754128 (0.032685796323222105) | > loss_mel_ce: 3.2257111072540283 (nan) | > loss: 0.012897315435111523 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13666043548330314) | > loader_time: 0.0045 (0.004693061999498574)  --> TIME: 2025-02-25 22:09:41 -- STEP: 5162/36938 -- GLOBAL_STEP: 965550 | > loss_text_ce: 0.027854539453983307 (0.03269617370405929) | > loss_mel_ce: 2.4586598873138428 (nan) | > loss: 0.009867120534181595 (nan) | > current_lr: 5e-06 | > step_time: 0.1446 (0.13670545218296684) | > loader_time: 0.0046 (0.0046923481554358645)  --> TIME: 2025-02-25 22:09:50 -- STEP: 5212/36938 -- GLOBAL_STEP: 965600 | > loss_text_ce: 0.03397909924387932 (0.032699551454946046) | > loss_mel_ce: 2.7201766967773438 (nan) | > loss: 0.010929190553724766 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.1366609737494322) | > loader_time: 0.0046 (0.0046922002768205375)  --> TIME: 2025-02-25 22:09:59 -- STEP: 5262/36938 -- GLOBAL_STEP: 965650 | > loss_text_ce: 0.03286987915635109 (0.03270062395179989) | > loss_mel_ce: 2.463717460632324 (nan) | > loss: 0.009907092899084091 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.13661444549640367) | > loader_time: 0.0048 (0.004692045920070057)  --> TIME: 2025-02-25 22:10:09 -- STEP: 5312/36938 -- GLOBAL_STEP: 965700 | > loss_text_ce: 0.02185979299247265 (0.03269035941679222) | > loss_mel_ce: 2.8602707386016846 (nan) | > loss: 0.011437026783823967 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.13664915076041872) | > loader_time: 0.0047 (0.004692700657858898)  --> TIME: 2025-02-25 22:10:18 -- STEP: 5362/36938 -- GLOBAL_STEP: 965750 | > loss_text_ce: 0.02848191000521183 (0.032690367621641604) | > loss_mel_ce: 1.5949350595474243 (nan) | > loss: 0.006442131474614143 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13666595073388416) | > loader_time: 0.0045 (0.004692462965962777)  --> TIME: 2025-02-25 22:10:27 -- STEP: 5412/36938 -- GLOBAL_STEP: 965800 | > loss_text_ce: 0.038570836186409 (0.03269604624163682) | > loss_mel_ce: 2.960428237915039 (nan) | > loss: 0.011900790967047215 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.1366396515526253) | > loader_time: 0.0045 (0.004693234971251563)  --> TIME: 2025-02-25 22:10:37 -- STEP: 5462/36938 -- GLOBAL_STEP: 965850 | > loss_text_ce: 0.03801940754055977 (0.03269648660265562) | > loss_mel_ce: 3.29166316986084 (nan) | > loss: 0.013213027268648148 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.13663681559141955) | > loader_time: 0.0049 (0.004694020254888712)  --> TIME: 2025-02-25 22:10:46 -- STEP: 5512/36938 -- GLOBAL_STEP: 965900 | > loss_text_ce: 0.041127968579530716 (0.03271725729627579) | > loss_mel_ce: 2.584620714187622 (nan) | > loss: 0.010419637896120548 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.1366572968759113) | > loader_time: 0.0046 (0.0046943876843667)  --> TIME: 2025-02-25 22:10:56 -- STEP: 5562/36938 -- GLOBAL_STEP: 965950 | > loss_text_ce: 0.03362568840384483 (0.03271935256506135) | > loss_mel_ce: 0.8228352665901184 (nan) | > loss: 0.0033986547496169806 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13664213571990877) | > loader_time: 0.0046 (0.0046942162453721065)  --> TIME: 2025-02-25 22:11:05 -- STEP: 5612/36938 -- GLOBAL_STEP: 966000 | > loss_text_ce: 0.03659236803650856 (0.032717526351823144) | > loss_mel_ce: 2.704026222229004 (nan) | > loss: 0.010875471867620945 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.136591963188528) | > loader_time: 0.0046 (0.004694284750407541)  --> TIME: 2025-02-25 22:11:14 -- STEP: 5662/36938 -- GLOBAL_STEP: 966050 | > loss_text_ce: 0.03187543898820877 (0.032707459052004076) | > loss_mel_ce: 2.295121431350708 (nan) | > loss: 0.009234114550054073 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.1365758765505974) | > loader_time: 0.0046 (0.004693921738317688)  --> TIME: 2025-02-25 22:11:23 -- STEP: 5712/36938 -- GLOBAL_STEP: 966100 | > loss_text_ce: 0.026761353015899658 (0.03270708824033742) | > loss_mel_ce: 3.0235812664031982 (nan) | > loss: 0.012104534544050694 (nan) | > current_lr: 5e-06 | > step_time: 0.1897 (0.13659598183732122) | > loader_time: 0.0049 (0.004693688089106259)  --> TIME: 2025-02-25 22:11:33 -- STEP: 5762/36938 -- GLOBAL_STEP: 966150 | > loss_text_ce: 0.03689083829522133 (0.03271246177937301) | > loss_mel_ce: 2.6575686931610107 (nan) | > loss: 0.010692300274968147 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1366001174334895) | > loader_time: 0.0045 (0.004693045007069465)  --> TIME: 2025-02-25 22:11:42 -- STEP: 5812/36938 -- GLOBAL_STEP: 966200 | > loss_text_ce: 0.03140203282237053 (0.03271841252987317) | > loss_mel_ce: 2.2439608573913574 (nan) | > loss: 0.009029218927025795 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13663169755988225) | > loader_time: 0.0046 (0.004693324985950464)  --> TIME: 2025-02-25 22:11:51 -- STEP: 5862/36938 -- GLOBAL_STEP: 966250 | > loss_text_ce: 0.03601538762450218 (0.032728199030268426) | > loss_mel_ce: 2.748811960220337 (nan) | > loss: 0.011050902307033539 (nan) | > current_lr: 5e-06 | > step_time: 0.1762 (0.13659771063839488) | > loader_time: 0.0048 (0.004692331713972864)  --> TIME: 2025-02-25 22:12:00 -- STEP: 5912/36938 -- GLOBAL_STEP: 966300 | > loss_text_ce: 0.028093546628952026 (0.03272685137402799) | > loss_mel_ce: 2.9262125492095947 (nan) | > loss: 0.01172343734651804 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13659497954364783) | > loader_time: 0.0048 (0.004693169877719485)  --> TIME: 2025-02-25 22:12:10 -- STEP: 5962/36938 -- GLOBAL_STEP: 966350 | > loss_text_ce: 0.03080444224178791 (0.03273022313908737) | > loss_mel_ce: 1.3033695220947266 (nan) | > loss: 0.005294341128319502 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13658805498937746) | > loader_time: 0.0046 (0.004692746824324024)  --> TIME: 2025-02-25 22:12:19 -- STEP: 6012/36938 -- GLOBAL_STEP: 966400 | > loss_text_ce: 0.029698606580495834 (0.032738743732229056) | > loss_mel_ce: 3.0633785724639893 (nan) | > loss: 0.012274116277694702 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13658524075112882) | > loader_time: 0.0045 (0.004692916345056968)  --> TIME: 2025-02-25 22:12:28 -- STEP: 6062/36938 -- GLOBAL_STEP: 966450 | > loss_text_ce: 0.03471861407160759 (0.03274384616523556) | > loss_mel_ce: 1.688584566116333 (nan) | > loss: 0.006838505156338215 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.1365861165488139) | > loader_time: 0.0045 (0.004692401558760557)  --> TIME: 2025-02-25 22:12:38 -- STEP: 6112/36938 -- GLOBAL_STEP: 966500 | > loss_text_ce: 0.03501948341727257 (0.03273948814893005) | > loss_mel_ce: 2.5295865535736084 (nan) | > loss: 0.010177007876336575 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13659261007115533) | > loader_time: 0.0046 (0.0046921790972429935)  --> TIME: 2025-02-25 22:12:47 -- STEP: 6162/36938 -- GLOBAL_STEP: 966550 | > loss_text_ce: 0.0381031408905983 (0.03273657746872973) | > loss_mel_ce: 0.0220184326171875 (nan) | > loss: 0.00023857768974266946 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13664425289967566) | > loader_time: 0.005 (0.004692101625605305)  --> TIME: 2025-02-25 22:12:57 -- STEP: 6212/36938 -- GLOBAL_STEP: 966600 | > loss_text_ce: 0.04055432602763176 (0.032753825528308356) | > loss_mel_ce: 2.5206398963928223 (nan) | > loss: 0.01016346924006939 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13663344347162307) | > loader_time: 0.0045 (0.004691542653368108)  --> TIME: 2025-02-25 22:13:06 -- STEP: 6262/36938 -- GLOBAL_STEP: 966650 | > loss_text_ce: 0.03372989967465401 (0.032742732232457224) | > loss_mel_ce: 3.199422836303711 (nan) | > loss: 0.012829971499741077 (nan) | > current_lr: 5e-06 | > step_time: 0.199 (0.1366638665913553) | > loader_time: 0.0049 (0.0046921635694451855)  --> TIME: 2025-02-25 22:13:16 -- STEP: 6312/36938 -- GLOBAL_STEP: 966700 | > loss_text_ce: 0.027533654123544693 (0.0327453149859739) | > loss_mel_ce: 2.7374346256256104 (nan) | > loss: 0.010972097516059875 (nan) | > current_lr: 5e-06 | > step_time: 0.165 (0.13673423342076318) | > loader_time: 0.0047 (0.004691946869114041)  --> TIME: 2025-02-25 22:13:26 -- STEP: 6362/36938 -- GLOBAL_STEP: 966750 | > loss_text_ce: 0.034114185720682144 (0.03274257198920014) | > loss_mel_ce: 1.9117904901504517 (nan) | > loss: 0.007721844594925642 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13676425913001616) | > loader_time: 0.0067 (0.004691965060577337)  --> TIME: 2025-02-25 22:13:35 -- STEP: 6412/36938 -- GLOBAL_STEP: 966800 | > loss_text_ce: 0.026400476694107056 (0.032734094370337675) | > loss_mel_ce: 2.530158281326294 (nan) | > loss: 0.010145075619220734 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1367290897512174) | > loader_time: 0.0045 (0.004691575775277362)  --> TIME: 2025-02-25 22:13:44 -- STEP: 6462/36938 -- GLOBAL_STEP: 966850 | > loss_text_ce: 0.029848797246813774 (0.03273683857343936) | > loss_mel_ce: 1.7285292148590088 (nan) | > loss: 0.006977690849453211 (nan) | > current_lr: 5e-06 | > step_time: 0.1853 (0.13675360943798148) | > loader_time: 0.0048 (0.004691186278864748)  --> TIME: 2025-02-25 22:13:54 -- STEP: 6512/36938 -- GLOBAL_STEP: 966900 | > loss_text_ce: 0.03958876058459282 (0.03275638618360603) | > loss_mel_ce: 3.2174289226531982 (nan) | > loss: 0.012924673967063427 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.1367566616295016) | > loader_time: 0.0046 (0.004691112356338211)  --> TIME: 2025-02-25 22:14:03 -- STEP: 6562/36938 -- GLOBAL_STEP: 966950 | > loss_text_ce: 0.024939512833952904 (0.032758983715157085) | > loss_mel_ce: 3.2050459384918213 (nan) | > loss: 0.012817403301596642 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13678421021525186) | > loader_time: 0.0046 (0.004691164219608315)  --> TIME: 2025-02-25 22:14:12 -- STEP: 6612/36938 -- GLOBAL_STEP: 967000 | > loss_text_ce: 0.033309195190668106 (0.032750003331262406) | > loss_mel_ce: 2.7670247554779053 (nan) | > loss: 0.011112436652183533 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.1367471707783106) | > loader_time: 0.0048 (0.004691766163686936)  --> TIME: 2025-02-25 22:14:22 -- STEP: 6662/36938 -- GLOBAL_STEP: 967050 | > loss_text_ce: 0.030964698642492294 (0.03274224519143018) | > loss_mel_ce: 3.2942326068878174 (nan) | > loss: 0.013195227831602097 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13678158181309893) | > loader_time: 0.0046 (0.00469206747297265)  --> TIME: 2025-02-25 22:14:31 -- STEP: 6712/36938 -- GLOBAL_STEP: 967100 | > loss_text_ce: 0.031039608642458916 (0.032743584308456815) | > loss_mel_ce: 3.1718146800994873 (nan) | > loss: 0.012709740549325943 (nan) | > current_lr: 5e-06 | > step_time: 0.1867 (0.13678341507201272) | > loader_time: 0.0048 (0.004692513162672015)  --> TIME: 2025-02-25 22:14:40 -- STEP: 6762/36938 -- GLOBAL_STEP: 967150 | > loss_text_ce: 0.0304435845464468 (0.032743080415429844) | > loss_mel_ce: 3.0076417922973633 (nan) | > loss: 0.012055895291268826 (nan) | > current_lr: 5e-06 | > step_time: 0.1945 (0.1368085956545281) | > loader_time: 0.0049 (0.004692326033478544)  --> TIME: 2025-02-25 22:14:50 -- STEP: 6812/36938 -- GLOBAL_STEP: 967200 | > loss_text_ce: 0.031582172960042953 (0.032743732470371455) | > loss_mel_ce: 2.313321113586426 (nan) | > loss: 0.00930517166852951 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13682933516594792) | > loader_time: 0.0048 (0.004691964132395192)  --> TIME: 2025-02-25 22:14:59 -- STEP: 6862/36938 -- GLOBAL_STEP: 967250 | > loss_text_ce: 0.035899464040994644 (0.032744162120967264) | > loss_mel_ce: 2.8656115531921387 (nan) | > loss: 0.01151393260806799 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13681273597977467) | > loader_time: 0.0045 (0.004691705068601035)  --> TIME: 2025-02-25 22:15:09 -- STEP: 6912/36938 -- GLOBAL_STEP: 967300 | > loss_text_ce: 0.032654114067554474 (0.03274411612505051) | > loss_mel_ce: 2.2667644023895264 (nan) | > loss: 0.009124676696956158 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.13682683429646286) | > loader_time: 0.0047 (0.004691921691927637)  --> TIME: 2025-02-25 22:15:18 -- STEP: 6962/36938 -- GLOBAL_STEP: 967350 | > loss_text_ce: 0.027608823031187057 (0.032744116561326156) | > loss_mel_ce: 2.7848260402679443 (nan) | > loss: 0.011160456575453281 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13682233362628007) | > loader_time: 0.0047 (0.004691558882820978)  --> TIME: 2025-02-25 22:15:27 -- STEP: 7012/36938 -- GLOBAL_STEP: 967400 | > loss_text_ce: 0.02516714483499527 (0.03273998698962371) | > loss_mel_ce: 2.652808666229248 (nan) | > loss: 0.01062688883394003 (nan) | > current_lr: 5e-06 | > step_time: 0.1343 (0.13684217585064798) | > loader_time: 0.0046 (0.00469167771097325)  --> TIME: 2025-02-25 22:15:37 -- STEP: 7062/36938 -- GLOBAL_STEP: 967450 | > loss_text_ce: 0.030888855457305908 (0.03273951648770109) | > loss_mel_ce: 1.4209442138671875 (nan) | > loss: 0.005761242471635342 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1368632195050573) | > loader_time: 0.0045 (0.004691634256486606)  --> TIME: 2025-02-25 22:15:46 -- STEP: 7112/36938 -- GLOBAL_STEP: 967500 | > loss_text_ce: 0.034230079501867294 (0.032742315738307826) | > loss_mel_ce: 2.613896608352661 (nan) | > loss: 0.010508439503610134 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1368649068913927) | > loader_time: 0.0046 (0.0046912258065591565)  --> TIME: 2025-02-25 22:15:56 -- STEP: 7162/36938 -- GLOBAL_STEP: 967550 | > loss_text_ce: 0.03249365836381912 (0.03273684670797549) | > loss_mel_ce: 2.8309123516082764 (nan) | > loss: 0.01136272307485342 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13687021903437777) | > loader_time: 0.0046 (0.004691178190811349)  --> TIME: 2025-02-25 22:16:05 -- STEP: 7212/36938 -- GLOBAL_STEP: 967600 | > loss_text_ce: 0.04300693795084953 (0.032746143233345015) | > loss_mel_ce: 2.461906671524048 (nan) | > loss: 0.009940133430063725 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13688615921531477) | > loader_time: 0.0044 (0.004690837840272258)  --> TIME: 2025-02-25 22:16:14 -- STEP: 7262/36938 -- GLOBAL_STEP: 967650 | > loss_text_ce: 0.034551799297332764 (0.032743054495714435) | > loss_mel_ce: 2.626392126083374 (nan) | > loss: 0.01055930182337761 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.1368643455156051) | > loader_time: 0.0048 (0.004690578803974082)  --> TIME: 2025-02-25 22:16:24 -- STEP: 7312/36938 -- GLOBAL_STEP: 967700 | > loss_text_ce: 0.037104688584804535 (0.03273479815183258) | > loss_mel_ce: 3.045189142227173 (nan) | > loss: 0.012231324799358845 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13687570673501226) | > loader_time: 0.0047 (0.004690454714407799)  --> TIME: 2025-02-25 22:16:33 -- STEP: 7362/36938 -- GLOBAL_STEP: 967750 | > loss_text_ce: 0.03691219165921211 (0.032741664427858065) | > loss_mel_ce: 2.0194091796875 (nan) | > loss: 0.008160006254911423 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13687203546413704) | > loader_time: 0.0046 (0.004690024879308401)  --> TIME: 2025-02-25 22:16:43 -- STEP: 7412/36938 -- GLOBAL_STEP: 967800 | > loss_text_ce: 0.0321824811398983 (0.032732625118232106) | > loss_mel_ce: 2.6137373447418213 (nan) | > loss: 0.010499682277441025 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13687838211100775) | > loader_time: 0.0046 (0.0046896435927521185)  --> TIME: 2025-02-25 22:16:52 -- STEP: 7462/36938 -- GLOBAL_STEP: 967850 | > loss_text_ce: 0.0370192751288414 (0.03273888478538353) | > loss_mel_ce: 2.700869560241699 (nan) | > loss: 0.010864638723433018 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13686131702546842) | > loader_time: 0.0047 (0.004689186515900117)  --> TIME: 2025-02-25 22:17:01 -- STEP: 7512/36938 -- GLOBAL_STEP: 967900 | > loss_text_ce: 0.025754358619451523 (0.03273788272404169) | > loss_mel_ce: 2.4854910373687744 (nan) | > loss: 0.00996526051312685 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13685861426674423) | > loader_time: 0.0045 (0.004688901642260072)  --> TIME: 2025-02-25 22:17:10 -- STEP: 7562/36938 -- GLOBAL_STEP: 967950 | > loss_text_ce: 0.04441415145993233 (0.03273099395984675) | > loss_mel_ce: 2.7656612396240234 (nan) | > loss: 0.011151093058288097 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.13685370629756324) | > loader_time: 0.0049 (0.0046889005404746956)  --> TIME: 2025-02-25 22:17:19 -- STEP: 7612/36938 -- GLOBAL_STEP: 968000 | > loss_text_ce: 0.03808268904685974 (0.032728700688881) | > loss_mel_ce: 3.1590840816497803 (nan) | > loss: 0.012687169946730137 (nan) | > current_lr: 5e-06 | > step_time: 0.144 (0.1368218821346669) | > loader_time: 0.0048 (0.004688702034311292)  --> TIME: 2025-02-25 22:17:29 -- STEP: 7662/36938 -- GLOBAL_STEP: 968050 | > loss_text_ce: 0.03501065820455551 (0.03272550828031423) | > loss_mel_ce: 2.918470621109009 (nan) | > loss: 0.011720163747668266 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13684204794102844) | > loader_time: 0.0046 (0.004688330992367826)  --> TIME: 2025-02-25 22:17:38 -- STEP: 7712/36938 -- GLOBAL_STEP: 968100 | > loss_text_ce: 0.03500433638691902 (0.032729217532396056) | > loss_mel_ce: 3.164792537689209 (nan) | > loss: 0.012697607278823853 (nan) | > current_lr: 5e-06 | > step_time: 0.1794 (0.13685602784404152) | > loader_time: 0.005 (0.004688481139444209)  --> TIME: 2025-02-25 22:17:47 -- STEP: 7762/36938 -- GLOBAL_STEP: 968150 | > loss_text_ce: 0.02908366359770298 (0.03272319753423683) | > loss_mel_ce: 2.8213133811950684 (nan) | > loss: 0.011311099864542484 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.1368247400206166) | > loader_time: 0.0046 (0.00468824054739641)  --> TIME: 2025-02-25 22:17:57 -- STEP: 7812/36938 -- GLOBAL_STEP: 968200 | > loss_text_ce: 0.03523649275302887 (0.03273458136159761) | > loss_mel_ce: 3.3470981121063232 (nan) | > loss: 0.013421963900327682 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13683632261864204) | > loader_time: 0.0045 (0.004687869542693694)  --> TIME: 2025-02-25 22:18:06 -- STEP: 7862/36938 -- GLOBAL_STEP: 968250 | > loss_text_ce: 0.03809027001261711 (0.0327356454902793) | > loss_mel_ce: 2.6543350219726562 (nan) | > loss: 0.01068422757089138 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13684310515216244) | > loader_time: 0.0046 (0.004688075983139182)  --> TIME: 2025-02-25 22:18:15 -- STEP: 7912/36938 -- GLOBAL_STEP: 968300 | > loss_text_ce: 0.04026384651660919 (0.03273411899832366) | > loss_mel_ce: 3.195543050765991 (nan) | > loss: 0.012840504758059978 (nan) | > current_lr: 5e-06 | > step_time: 0.1662 (0.13680852258048748) | > loader_time: 0.0049 (0.004688139631967566)  --> TIME: 2025-02-25 22:18:25 -- STEP: 7962/36938 -- GLOBAL_STEP: 968350 | > loss_text_ce: 0.031398020684719086 (0.032733492786006145) | > loss_mel_ce: 3.053514003753662 (nan) | > loss: 0.01224171556532383 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.13681775249149683) | > loader_time: 0.0046 (0.0046878865962350635)  --> TIME: 2025-02-25 22:18:34 -- STEP: 8012/36938 -- GLOBAL_STEP: 968400 | > loss_text_ce: 0.02695852518081665 (0.03272932612570108) | > loss_mel_ce: 2.7500643730163574 (nan) | > loss: 0.011019932106137276 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13681571366366807) | > loader_time: 0.0047 (0.004687722777939407)  --> TIME: 2025-02-25 22:18:43 -- STEP: 8062/36938 -- GLOBAL_STEP: 968450 | > loss_text_ce: 0.03169324994087219 (0.03273096049162408) | > loss_mel_ce: 2.6985135078430176 (nan) | > loss: 0.010834154672920704 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1368032839954183) | > loader_time: 0.0046 (0.004687437375938581)  --> TIME: 2025-02-25 22:18:53 -- STEP: 8112/36938 -- GLOBAL_STEP: 968500 | > loss_text_ce: 0.024846497923135757 (0.03273894318633865) | > loss_mel_ce: 1.1024612188339233 (nan) | > loss: 0.004473443608731031 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.1368111187125572) | > loader_time: 0.0045 (0.00468755664792521)  --> TIME: 2025-02-25 22:19:02 -- STEP: 8162/36938 -- GLOBAL_STEP: 968550 | > loss_text_ce: 0.028231391683220863 (0.03273954317160807) | > loss_mel_ce: 3.018439769744873 (nan) | > loss: 0.012089965865015984 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1368146466729808) | > loader_time: 0.0048 (0.004687249149302873)  --> TIME: 2025-02-25 22:19:12 -- STEP: 8212/36938 -- GLOBAL_STEP: 968600 | > loss_text_ce: 0.0422385148704052 (0.032732447700260056) | > loss_mel_ce: 3.0977742671966553 (nan) | > loss: 0.012460368685424328 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13684194078111003) | > loader_time: 0.0046 (0.00468758798144935)  --> TIME: 2025-02-25 22:19:21 -- STEP: 8262/36938 -- GLOBAL_STEP: 968650 | > loss_text_ce: 0.040331825613975525 (0.032735057984064064) | > loss_mel_ce: 3.2530345916748047 (nan) | > loss: 0.013068915344774723 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13681828182103212) | > loader_time: 0.0047 (0.004687293595651303)  --> TIME: 2025-02-25 22:19:30 -- STEP: 8312/36938 -- GLOBAL_STEP: 968700 | > loss_text_ce: 0.03853177651762962 (0.03274388231661544) | > loss_mel_ce: 3.191624641418457 (nan) | > loss: 0.012818082235753536 (nan) | > current_lr: 5e-06 | > step_time: 0.1502 (0.13680289394587045) | > loader_time: 0.0047 (0.004687014569217356)  --> TIME: 2025-02-25 22:19:40 -- STEP: 8362/36938 -- GLOBAL_STEP: 968750 | > loss_text_ce: 0.032009225338697433 (0.03275118070759029) | > loss_mel_ce: 2.828679084777832 (nan) | > loss: 0.011351938359439373 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.13683603129812325) | > loader_time: 0.0048 (0.004687008832636355)  --> TIME: 2025-02-25 22:19:49 -- STEP: 8412/36938 -- GLOBAL_STEP: 968800 | > loss_text_ce: 0.030812852084636688 (0.032752038446209296) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13683002481446987) | > loader_time: 0.0046 (0.004686817151276877)  --> TIME: 2025-02-25 22:19:58 -- STEP: 8462/36938 -- GLOBAL_STEP: 968850 | > loss_text_ce: 0.03368295729160309 (0.03274603756736751) | > loss_mel_ce: 2.9826719760894775 (nan) | > loss: 0.011969663202762604 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13683257664216397) | > loader_time: 0.0046 (0.004686513062037347)  --> TIME: 2025-02-25 22:20:08 -- STEP: 8512/36938 -- GLOBAL_STEP: 968900 | > loss_text_ce: 0.028643175959587097 (0.0327463026565089) | > loss_mel_ce: 3.0258235931396484 (nan) | > loss: 0.012120900675654411 (nan) | > current_lr: 5e-06 | > step_time: 0.1679 (0.13682857847639515) | > loader_time: 0.0047 (0.004686329317720306)  --> TIME: 2025-02-25 22:20:17 -- STEP: 8562/36938 -- GLOBAL_STEP: 968950 | > loss_text_ce: 0.03764616698026657 (0.0327458627260063) | > loss_mel_ce: 3.1875 (nan) | > loss: 0.01279819943010807 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.1368078392706474) | > loader_time: 0.0046 (0.004686498330091092)  --> TIME: 2025-02-25 22:20:26 -- STEP: 8612/36938 -- GLOBAL_STEP: 969000 | > loss_text_ce: 0.034648288041353226 (0.032752891921788985) | > loss_mel_ce: 2.6897380352020264 (nan) | > loss: 0.010811056941747665 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13680596679407572) | > loader_time: 0.0047 (0.004686116950765516)  --> TIME: 2025-02-25 22:20:36 -- STEP: 8662/36938 -- GLOBAL_STEP: 969050 | > loss_text_ce: 0.045786723494529724 (0.03276346849621234) | > loss_mel_ce: 2.5906221866607666 (nan) | > loss: 0.010461940430104733 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.1368270317805171) | > loader_time: 0.0046 (0.004685791252777438)  --> TIME: 2025-02-25 22:20:45 -- STEP: 8712/36938 -- GLOBAL_STEP: 969100 | > loss_text_ce: 0.03743397444486618 (0.03277009851093966) | > loss_mel_ce: 1.3667566776275635 (nan) | > loss: 0.005572185385972261 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13685449648604056) | > loader_time: 0.0046 (0.00468538809632685)  --> TIME: 2025-02-25 22:20:55 -- STEP: 8762/36938 -- GLOBAL_STEP: 969150 | > loss_text_ce: 0.03031482920050621 (0.03276694614267576) | > loss_mel_ce: 0.1525442898273468 (nan) | > loss: 0.0007256314856931567 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13686862040317077) | > loader_time: 0.0048 (0.004685049213739426)  --> TIME: 2025-02-25 22:21:04 -- STEP: 8812/36938 -- GLOBAL_STEP: 969200 | > loss_text_ce: 0.0408073291182518 (0.03276985297461473) | > loss_mel_ce: 3.2114508152008057 (nan) | > loss: 0.012905786745250225 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.1368800993946215) | > loader_time: 0.0047 (0.004685187253203108)  --> TIME: 2025-02-25 22:21:14 -- STEP: 8862/36938 -- GLOBAL_STEP: 969250 | > loss_text_ce: 0.03404143825173378 (0.03277270393490214) | > loss_mel_ce: 2.839812755584717 (nan) | > loss: 0.011404183693230152 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.13690009342577597) | > loader_time: 0.0046 (0.00468498776443815)  --> TIME: 2025-02-25 22:21:23 -- STEP: 8912/36938 -- GLOBAL_STEP: 969300 | > loss_text_ce: 0.029735326766967773 (0.032774485514333596) | > loss_mel_ce: 2.600062370300293 (nan) | > loss: 0.010435706004500389 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.13689593911920067) | > loader_time: 0.0047 (0.00468464912696201)  --> TIME: 2025-02-25 22:21:32 -- STEP: 8962/36938 -- GLOBAL_STEP: 969350 | > loss_text_ce: 0.02465006522834301 (0.03277385329399093) | > loss_mel_ce: 2.8101563453674316 (nan) | > loss: 0.011249232105910778 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.13687052880512593) | > loader_time: 0.0045 (0.004684233074873531)  --> TIME: 2025-02-25 22:21:41 -- STEP: 9012/36938 -- GLOBAL_STEP: 969400 | > loss_text_ce: 0.034778892993927 (0.03278331365601651) | > loss_mel_ce: 2.521998405456543 (nan) | > loss: 0.010145941749215126 (nan) | > current_lr: 5e-06 | > step_time: 0.1799 (0.13687035032658426) | > loader_time: 0.005 (0.004684117413921668)  --> TIME: 2025-02-25 22:21:51 -- STEP: 9062/36938 -- GLOBAL_STEP: 969450 | > loss_text_ce: 0.035876527428627014 (0.03278587718213247) | > loss_mel_ce: 2.175938844680786 (nan) | > loss: 0.008777045644819736 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13686991827137016) | > loader_time: 0.0046 (0.004683613935018898)  --> TIME: 2025-02-25 22:22:00 -- STEP: 9112/36938 -- GLOBAL_STEP: 969500 | > loss_text_ce: 0.028696460649371147 (0.03278418130664876) | > loss_mel_ce: 2.4638707637786865 (nan) | > loss: 0.009891140274703503 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13688908686127169) | > loader_time: 0.0047 (0.00468344406876468)  --> TIME: 2025-02-25 22:22:10 -- STEP: 9162/36938 -- GLOBAL_STEP: 969550 | > loss_text_ce: 0.038942087441682816 (0.03278307036367187) | > loss_mel_ce: 3.1622369289398193 (nan) | > loss: 0.012703091837465763 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13691440460802182) | > loader_time: 0.0045 (0.004683264190259134)  --> TIME: 2025-02-25 22:22:19 -- STEP: 9212/36938 -- GLOBAL_STEP: 969600 | > loss_text_ce: 0.04591004177927971 (0.032783743415146664) | > loss_mel_ce: 1.5580416917800903 (nan) | > loss: 0.006364888045936823 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.1369147593178963) | > loader_time: 0.0045 (0.004683013253247179)  --> TIME: 2025-02-25 22:22:28 -- STEP: 9262/36938 -- GLOBAL_STEP: 969650 | > loss_text_ce: 0.02628597617149353 (0.03278681816218669) | > loss_mel_ce: 2.2265625 (nan) | > loss: 0.008939875289797783 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13691463590929664) | > loader_time: 0.0045 (0.004683581008120355)  --> TIME: 2025-02-25 22:22:37 -- STEP: 9312/36938 -- GLOBAL_STEP: 969700 | > loss_text_ce: 0.03226519748568535 (0.032787782063365994) | > loss_mel_ce: 1.7539812326431274 (nan) | > loss: 0.007088279817253351 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.13689283348133538) | > loader_time: 0.0045 (0.00468310493579026)  --> TIME: 2025-02-25 22:22:47 -- STEP: 9362/36938 -- GLOBAL_STEP: 969750 | > loss_text_ce: 0.02845393680036068 (0.03279351687720932) | > loss_mel_ce: 2.9064786434173584 (nan) | > loss: 0.011646558530628681 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13689814485359708) | > loader_time: 0.0046 (0.004682749490711433)  --> TIME: 2025-02-25 22:22:56 -- STEP: 9412/36938 -- GLOBAL_STEP: 969800 | > loss_text_ce: 0.03400614485144615 (0.03279175448407129) | > loss_mel_ce: 3.3143224716186523 (nan) | > loss: 0.013287018984556198 (nan) | > current_lr: 5e-06 | > step_time: 0.1813 (0.1368782382491701) | > loader_time: 0.0048 (0.004682607666968895)  --> TIME: 2025-02-25 22:23:05 -- STEP: 9462/36938 -- GLOBAL_STEP: 969850 | > loss_text_ce: 0.031386636197566986 (0.0327901808712123) | > loss_mel_ce: 2.96893310546875 (nan) | > loss: 0.011906031519174576 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.1368670139522392) | > loader_time: 0.0048 (0.004682634376214644)  --> TIME: 2025-02-25 22:23:15 -- STEP: 9512/36938 -- GLOBAL_STEP: 969900 | > loss_text_ce: 0.043703701347112656 (0.032792250878299685) | > loss_mel_ce: 2.9153850078582764 (nan) | > loss: 0.011742416769266129 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13687229788132996) | > loader_time: 0.0049 (0.004682665040655435)  --> TIME: 2025-02-25 22:23:24 -- STEP: 9562/36938 -- GLOBAL_STEP: 969950 | > loss_text_ce: 0.028519323095679283 (0.032788509405316336) | > loss_mel_ce: 2.316758871078491 (nan) | > loss: 0.009306659922003746 (nan) | > current_lr: 5e-06 | > step_time: 0.1261 (0.13685627281354193) | > loader_time: 0.0046 (0.004682915301982191)  --> TIME: 2025-02-25 22:23:33 -- STEP: 9612/36938 -- GLOBAL_STEP: 970000 | > loss_text_ce: 0.020569762215018272 (0.03278995699367791) | > loss_mel_ce: 2.9192745685577393 (nan) | > loss: 0.011666050180792809 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.1368534086001606) | > loader_time: 0.0046 (0.004682704428062805) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_970000.pth  --> TIME: 2025-02-25 22:23:49 -- STEP: 9662/36938 -- GLOBAL_STEP: 970050 | > loss_text_ce: 0.0237968098372221 (0.03279807511996602) | > loss_mel_ce: 2.5368924140930176 (nan) | > loss: 0.010161465965211391 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13685455928799406) | > loader_time: 0.0045 (0.004682868762283779)  --> TIME: 2025-02-25 22:23:58 -- STEP: 9712/36938 -- GLOBAL_STEP: 970100 | > loss_text_ce: 0.025409618392586708 (0.032797715606615034) | > loss_mel_ce: 2.771183729171753 (nan) | > loss: 0.011097593232989311 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13687129821965968) | > loader_time: 0.0047 (0.004683331538857111)  --> TIME: 2025-02-25 22:24:08 -- STEP: 9762/36938 -- GLOBAL_STEP: 970150 | > loss_text_ce: 0.03405599668622017 (0.03279923776197456) | > loss_mel_ce: 3.0253987312316895 (nan) | > loss: 0.012140694074332714 (nan) | > current_lr: 5e-06 | > step_time: 0.1757 (0.1368944122517272) | > loader_time: 0.0048 (0.004683265356811385)  --> TIME: 2025-02-25 22:24:17 -- STEP: 9812/36938 -- GLOBAL_STEP: 970200 | > loss_text_ce: 0.02998950704932213 (0.03280273029404913) | > loss_mel_ce: 2.830329656600952 (nan) | > loss: 0.011350473389029503 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13689213714160395) | > loader_time: 0.0045 (0.004683164689086963)  --> TIME: 2025-02-25 22:24:27 -- STEP: 9862/36938 -- GLOBAL_STEP: 970250 | > loss_text_ce: 0.05140211805701256 (0.0328051281062259) | > loss_mel_ce: 0.5127067565917969 (nan) | > loss: 0.0022385274060070515 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13690493378091034) | > loader_time: 0.0045 (0.0046830221064746835)  --> TIME: 2025-02-25 22:24:36 -- STEP: 9912/36938 -- GLOBAL_STEP: 970300 | > loss_text_ce: 0.03287595137953758 (0.032807740645465745) | > loss_mel_ce: 3.1358792781829834 (nan) | > loss: 0.012574426829814911 (nan) | > current_lr: 5e-06 | > step_time: 0.1528 (0.13688749899972058) | > loader_time: 0.0051 (0.004682900854615648)  --> TIME: 2025-02-25 22:24:45 -- STEP: 9962/36938 -- GLOBAL_STEP: 970350 | > loss_text_ce: 0.02626495249569416 (0.032804703487924976) | > loss_mel_ce: 2.720646381378174 (nan) | > loss: 0.010900442488491535 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.136874021715105) | > loader_time: 0.0048 (0.0046829091715204765)  --> TIME: 2025-02-25 22:24:54 -- STEP: 10012/36938 -- GLOBAL_STEP: 970400 | > loss_text_ce: 0.0514298677444458 (0.03280359270046274) | > loss_mel_ce: 1.3763021230697632 (nan) | > loss: 0.005665603559464216 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.13687907165877197) | > loader_time: 0.0047 (0.004682818651675605)  --> TIME: 2025-02-25 22:25:04 -- STEP: 10062/36938 -- GLOBAL_STEP: 970450 | > loss_text_ce: 0.02812112122774124 (0.032804477148781985) | > loss_mel_ce: 2.527111291885376 (nan) | > loss: 0.010139811784029007 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13688124823205433) | > loader_time: 0.0046 (0.004683059220568981)  --> TIME: 2025-02-25 22:25:13 -- STEP: 10112/36938 -- GLOBAL_STEP: 970500 | > loss_text_ce: 0.034304291009902954 (0.03280778542719849) | > loss_mel_ce: 2.277574062347412 (nan) | > loss: 0.009174121543765068 (nan) | > current_lr: 5e-06 | > step_time: 0.1911 (0.13688505931368442) | > loader_time: 0.0049 (0.004684034970742235)  --> TIME: 2025-02-25 22:25:22 -- STEP: 10162/36938 -- GLOBAL_STEP: 970550 | > loss_text_ce: 0.038357239216566086 (0.032806984773406045) | > loss_mel_ce: 2.1194441318511963 (nan) | > loss: 0.008562704548239708 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13687790946401204) | > loader_time: 0.0045 (0.004684230305739819)  --> TIME: 2025-02-25 22:25:32 -- STEP: 10212/36938 -- GLOBAL_STEP: 970600 | > loss_text_ce: 0.037407442927360535 (0.03280979463165926) | > loss_mel_ce: 0.177734375 (nan) | > loss: 0.0008537374087609351 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13688476896734364) | > loader_time: 0.0046 (0.004684572681183442)  --> TIME: 2025-02-25 22:25:41 -- STEP: 10262/36938 -- GLOBAL_STEP: 970650 | > loss_text_ce: 0.032625045627355576 (0.03281083557298436) | > loss_mel_ce: 2.626410722732544 (nan) | > loss: 0.01055172923952341 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13689570660666484) | > loader_time: 0.0046 (0.004685061876129339)  --> TIME: 2025-02-25 22:25:50 -- STEP: 10312/36938 -- GLOBAL_STEP: 970700 | > loss_text_ce: 0.024094291031360626 (0.0328068024768039) | > loss_mel_ce: 2.370526075363159 (nan) | > loss: 0.009502462111413479 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13685904079116254) | > loader_time: 0.0045 (0.004684611842833895)  --> TIME: 2025-02-25 22:25:59 -- STEP: 10362/36938 -- GLOBAL_STEP: 970750 | > loss_text_ce: 0.035687800496816635 (0.03280782681150846) | > loss_mel_ce: 2.684903860092163 (nan) | > loss: 0.010795998387038708 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13686623547533308) | > loader_time: 0.0045 (0.004684693241321854)  --> TIME: 2025-02-25 22:26:09 -- STEP: 10412/36938 -- GLOBAL_STEP: 970800 | > loss_text_ce: 0.044268760830163956 (0.03281221029247053) | > loss_mel_ce: 2.184877395629883 (nan) | > loss: 0.008845819160342216 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13688411762565184) | > loader_time: 0.0049 (0.004684521036334913)  --> TIME: 2025-02-25 22:26:18 -- STEP: 10462/36938 -- GLOBAL_STEP: 970850 | > loss_text_ce: 0.036688029766082764 (0.03281397359781109) | > loss_mel_ce: 2.8466663360595703 (nan) | > loss: 0.011441883631050587 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13687377257685396) | > loader_time: 0.0045 (0.004684452458153572)  --> TIME: 2025-02-25 22:26:28 -- STEP: 10512/36938 -- GLOBAL_STEP: 970900 | > loss_text_ce: 0.03929310292005539 (0.03281062862191191) | > loss_mel_ce: 2.8335139751434326 (nan) | > loss: 0.011400028131902218 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.13687984588665236) | > loader_time: 0.0046 (0.0046846544089382556)  --> TIME: 2025-02-25 22:26:37 -- STEP: 10562/36938 -- GLOBAL_STEP: 970950 | > loss_text_ce: 0.034040968865156174 (0.032819693738707786) | > loss_mel_ce: 3.0664279460906982 (nan) | > loss: 0.012303448282182217 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1368544833313861) | > loader_time: 0.0046 (0.0046843174528249385)  --> TIME: 2025-02-25 22:26:46 -- STEP: 10612/36938 -- GLOBAL_STEP: 971000 | > loss_text_ce: 0.026619011536240578 (0.03282361953498916) | > loss_mel_ce: 2.125490427017212 (nan) | > loss: 0.008540117181837559 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13685128464592836) | > loader_time: 0.0044 (0.00468409153546111)  --> TIME: 2025-02-25 22:26:55 -- STEP: 10662/36938 -- GLOBAL_STEP: 971050 | > loss_text_ce: 0.02684916742146015 (0.03282327951634193) | > loss_mel_ce: 2.1444647312164307 (nan) | > loss: 0.008616325445473194 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1368663799156737) | > loader_time: 0.0045 (0.004683914293694545)  --> TIME: 2025-02-25 22:27:04 -- STEP: 10712/36938 -- GLOBAL_STEP: 971100 | > loss_text_ce: 0.0350518561899662 (0.03282232941300451) | > loss_mel_ce: 2.5331051349639893 (nan) | > loss: 0.010191099718213081 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13684474593939666) | > loader_time: 0.0047 (0.004683576585464811)  --> TIME: 2025-02-25 22:27:14 -- STEP: 10762/36938 -- GLOBAL_STEP: 971150 | > loss_text_ce: 0.04659121856093407 (0.03281986167639847) | > loss_mel_ce: 3.054473400115967 (nan) | > loss: 0.012305812910199165 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13685506592229404) | > loader_time: 0.0045 (0.004683280983637568)  --> TIME: 2025-02-25 22:27:23 -- STEP: 10812/36938 -- GLOBAL_STEP: 971200 | > loss_text_ce: 0.022258009761571884 (0.03282007238744256) | > loss_mel_ce: 1.31533682346344 (nan) | > loss: 0.005307916551828384 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.13683199842815094) | > loader_time: 0.0046 (0.00468301190917155)  --> TIME: 2025-02-25 22:27:32 -- STEP: 10862/36938 -- GLOBAL_STEP: 971250 | > loss_text_ce: 0.035866837948560715 (0.03282172914261383) | > loss_mel_ce: 3.1326606273651123 (nan) | > loss: 0.012573521584272385 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.13681740564133227) | > loader_time: 0.005 (0.004682671802080341)  --> TIME: 2025-02-25 22:27:41 -- STEP: 10912/36938 -- GLOBAL_STEP: 971300 | > loss_text_ce: 0.03933243826031685 (0.03282650330612944) | > loss_mel_ce: 3.010301113128662 (nan) | > loss: 0.01210172101855278 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.1368151559027417) | > loader_time: 0.0047 (0.004682680597403171)  --> TIME: 2025-02-25 22:27:51 -- STEP: 10962/36938 -- GLOBAL_STEP: 971350 | > loss_text_ce: 0.03341494873166084 (0.032833867959791624) | > loss_mel_ce: 3.244053602218628 (nan) | > loss: 0.01300582755357027 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13683253738745832) | > loader_time: 0.0046 (0.00468252762632851)  --> TIME: 2025-02-25 22:28:00 -- STEP: 11012/36938 -- GLOBAL_STEP: 971400 | > loss_text_ce: 0.026063688099384308 (0.0328363412800875) | > loss_mel_ce: 0.44639283418655396 (nan) | > loss: 0.0018748275469988585 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13681883888594595) | > loader_time: 0.0046 (0.004682661726654019)  --> TIME: 2025-02-25 22:28:09 -- STEP: 11062/36938 -- GLOBAL_STEP: 971450 | > loss_text_ce: 0.033368177711963654 (0.0328358608835816) | > loss_mel_ce: 2.2538859844207764 (nan) | > loss: 0.009076405316591263 (nan) | > current_lr: 5e-06 | > step_time: 0.151 (0.136819350398314) | > loader_time: 0.0047 (0.004682632019368492)  --> TIME: 2025-02-25 22:28:19 -- STEP: 11112/36938 -- GLOBAL_STEP: 971500 | > loss_text_ce: 0.03187742084264755 (0.032832768341996305) | > loss_mel_ce: 1.2659204006195068 (nan) | > loss: 0.00514999171718955 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13682182898102688) | > loader_time: 0.0047 (0.004682654610119581)  --> TIME: 2025-02-25 22:28:28 -- STEP: 11162/36938 -- GLOBAL_STEP: 971550 | > loss_text_ce: 0.028104862198233604 (0.03283523624132767) | > loss_mel_ce: 2.523737668991089 (nan) | > loss: 0.010126359760761261 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.1368047279432422) | > loader_time: 0.0048 (0.0046825694516661525)  --> TIME: 2025-02-25 22:28:37 -- STEP: 11212/36938 -- GLOBAL_STEP: 971600 | > loss_text_ce: 0.03776080906391144 (0.03283520758142361) | > loss_mel_ce: 2.280742883682251 (nan) | > loss: 0.009200411848723888 (nan) | > current_lr: 5e-06 | > step_time: 0.1533 (0.13678937816551912) | > loader_time: 0.0047 (0.004682475037120547)  --> TIME: 2025-02-25 22:28:46 -- STEP: 11262/36938 -- GLOBAL_STEP: 971650 | > loss_text_ce: 0.025851445272564888 (0.03283378182530135) | > loss_mel_ce: 2.1362483501434326 (nan) | > loss: 0.00857976172119379 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.1367950985643805) | > loader_time: 0.0046 (0.004682351907348355)  --> TIME: 2025-02-25 22:28:56 -- STEP: 11312/36938 -- GLOBAL_STEP: 971700 | > loss_text_ce: 0.030677825212478638 (0.032830652511198356) | > loss_mel_ce: 3.1835989952087402 (nan) | > loss: 0.012755067087709904 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13680348400938208) | > loader_time: 0.0046 (0.0046822886276447345)  --> TIME: 2025-02-25 22:29:05 -- STEP: 11362/36938 -- GLOBAL_STEP: 971750 | > loss_text_ce: 0.021147556602954865 (0.03283438402833132) | > loss_mel_ce: 1.0521240234375 (nan) | > loss: 0.004259014502167702 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13681872834036332) | > loader_time: 0.0046 (0.004682191869067246)  --> TIME: 2025-02-25 22:29:15 -- STEP: 11412/36938 -- GLOBAL_STEP: 971800 | > loss_text_ce: 0.03341304510831833 (0.03282803285772621) | > loss_mel_ce: 2.408384084701538 (nan) | > loss: 0.009689670987427235 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13681189762346876) | > loader_time: 0.0047 (0.00468196350359055)  --> TIME: 2025-02-25 22:29:24 -- STEP: 11462/36938 -- GLOBAL_STEP: 971850 | > loss_text_ce: 0.028861479833722115 (0.03282535916744871) | > loss_mel_ce: 3.224160671234131 (nan) | > loss: 0.012908819131553173 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13681890202783448) | > loader_time: 0.0046 (0.004681826261840874)  --> TIME: 2025-02-25 22:29:34 -- STEP: 11512/36938 -- GLOBAL_STEP: 971900 | > loss_text_ce: 0.029829896986484528 (0.032823413005244155) | > loss_mel_ce: 1.5561141967773438 (nan) | > loss: 0.006293429061770439 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13684285092966922) | > loader_time: 0.0047 (0.004681749361268829)  --> TIME: 2025-02-25 22:29:43 -- STEP: 11562/36938 -- GLOBAL_STEP: 971950 | > loss_text_ce: 0.031830064952373505 (0.03282855104893762) | > loss_mel_ce: 1.4419773817062378 (nan) | > loss: 0.005848442669957876 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13682497314795652) | > loader_time: 0.0046 (0.004681406250422308)  --> TIME: 2025-02-25 22:29:52 -- STEP: 11612/36938 -- GLOBAL_STEP: 972000 | > loss_text_ce: 0.035218991339206696 (0.03282790893072496) | > loss_mel_ce: 3.3881545066833496 (nan) | > loss: 0.01358481589704752 (nan) | > current_lr: 5e-06 | > step_time: 0.188 (0.13683914542814207) | > loader_time: 0.0048 (0.004681366889426665)  --> TIME: 2025-02-25 22:30:02 -- STEP: 11662/36938 -- GLOBAL_STEP: 972050 | > loss_text_ce: 0.024834344163537025 (0.03282896015651557) | > loss_mel_ce: 2.1190764904022217 (nan) | > loss: 0.008507583290338516 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.136844175734052) | > loader_time: 0.0045 (0.004681289206236421)  --> TIME: 2025-02-25 22:30:11 -- STEP: 11712/36938 -- GLOBAL_STEP: 972100 | > loss_text_ce: 0.033778537064790726 (0.03282574720925932) | > loss_mel_ce: 2.583660364151001 (nan) | > loss: 0.010386662557721138 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13685662692280476) | > loader_time: 0.0045 (0.004681135481986831)  --> TIME: 2025-02-25 22:30:20 -- STEP: 11762/36938 -- GLOBAL_STEP: 972150 | > loss_text_ce: 0.02771296165883541 (0.03282552499672782) | > loss_mel_ce: 2.237060546875 (nan) | > loss: 0.008987197652459145 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1368354947322533) | > loader_time: 0.0048 (0.0046808737067417245)  --> TIME: 2025-02-25 22:30:29 -- STEP: 11812/36938 -- GLOBAL_STEP: 972200 | > loss_text_ce: 0.03499094024300575 (0.03282709763059869) | > loss_mel_ce: 1.9475047588348389 (nan) | > loss: 0.007867046631872654 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13682706419427784) | > loader_time: 0.0045 (0.004680763673992256)  --> TIME: 2025-02-25 22:30:38 -- STEP: 11862/36938 -- GLOBAL_STEP: 972250 | > loss_text_ce: 0.03799402713775635 (0.032831141487366156) | > loss_mel_ce: 1.721881628036499 (nan) | > loss: 0.0069836340844631195 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.1368084948373639) | > loader_time: 0.0047 (0.004680503120190809)  --> TIME: 2025-02-25 22:30:48 -- STEP: 11912/36938 -- GLOBAL_STEP: 972300 | > loss_text_ce: 0.03326992690563202 (0.03282664763490055) | > loss_mel_ce: 2.572166919708252 (nan) | > loss: 0.010339035652577877 (nan) | > current_lr: 5e-06 | > step_time: 0.1912 (0.13681912754429087) | > loader_time: 0.0048 (0.004680510392678985)  --> TIME: 2025-02-25 22:30:57 -- STEP: 11962/36938 -- GLOBAL_STEP: 972350 | > loss_text_ce: 0.025187959894537926 (0.03283045425457668) | > loss_mel_ce: 2.3066391944885254 (nan) | > loss: 0.009253283031284809 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13681426049873366) | > loader_time: 0.0046 (0.004680350878788385)  --> TIME: 2025-02-25 22:31:07 -- STEP: 12012/36938 -- GLOBAL_STEP: 972400 | > loss_text_ce: 0.027929186820983887 (0.03282821521239265) | > loss_mel_ce: 0.10174846649169922 (nan) | > loss: 0.0005145939067006111 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13681803516177765) | > loader_time: 0.0047 (0.004680334866701902)  --> TIME: 2025-02-25 22:31:16 -- STEP: 12062/36938 -- GLOBAL_STEP: 972450 | > loss_text_ce: 0.030149877071380615 (0.032828928091100065) | > loss_mel_ce: 2.785997152328491 (nan) | > loss: 0.011175187304615974 (nan) | > current_lr: 5e-06 | > step_time: 0.1352 (0.136814252493335) | > loader_time: 0.0046 (0.004680100156979822)  --> TIME: 2025-02-25 22:31:25 -- STEP: 12112/36938 -- GLOBAL_STEP: 972500 | > loss_text_ce: 0.031080404296517372 (0.03282833084263526) | > loss_mel_ce: 2.944518804550171 (nan) | > loss: 0.011807934381067753 (nan) | > current_lr: 5e-06 | > step_time: 0.1492 (0.13682013828814318) | > loader_time: 0.0047 (0.004680058423090173)  --> TIME: 2025-02-25 22:31:35 -- STEP: 12162/36938 -- GLOBAL_STEP: 972550 | > loss_text_ce: 0.03286442905664444 (0.03282851491906345) | > loss_mel_ce: 3.016467571258545 (nan) | > loss: 0.0121005242690444 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.1368330455183146) | > loader_time: 0.0047 (0.004680249432634512)  --> TIME: 2025-02-25 22:31:44 -- STEP: 12212/36938 -- GLOBAL_STEP: 972600 | > loss_text_ce: 0.04468866065144539 (0.032828136805269165) | > loss_mel_ce: 2.9710121154785156 (nan) | > loss: 0.01196706760674715 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.1368437146422052) | > loader_time: 0.0048 (0.0046801181687630864)  --> TIME: 2025-02-25 22:31:54 -- STEP: 12262/36938 -- GLOBAL_STEP: 972650 | > loss_text_ce: 0.021870533004403114 (0.03282972874508438) | > loss_mel_ce: 2.432713747024536 (nan) | > loss: 0.009740415029227734 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13684439287744157) | > loader_time: 0.0045 (0.004680679043166462)  --> TIME: 2025-02-25 22:32:03 -- STEP: 12312/36938 -- GLOBAL_STEP: 972700 | > loss_text_ce: 0.03020143136382103 (0.032830696315017366) | > loss_mel_ce: 2.4289228916168213 (nan) | > loss: 0.009758430533111095 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.1368565041186988) | > loader_time: 0.0046 (0.00468060823521729)  --> TIME: 2025-02-25 22:32:13 -- STEP: 12362/36938 -- GLOBAL_STEP: 972750 | > loss_text_ce: 0.03731582686305046 (0.032833890151518635) | > loss_mel_ce: 2.002173900604248 (nan) | > loss: 0.00809321366250515 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13688267162559437) | > loader_time: 0.0045 (0.004680728684773898)  --> TIME: 2025-02-25 22:32:23 -- STEP: 12412/36938 -- GLOBAL_STEP: 972800 | > loss_text_ce: 0.03059871681034565 (0.032834648982151285) | > loss_mel_ce: 2.0027811527252197 (nan) | > loss: 0.00806896761059761 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13690885339596245) | > loader_time: 0.0045 (0.004680682469520729)  --> TIME: 2025-02-25 22:32:32 -- STEP: 12462/36938 -- GLOBAL_STEP: 972850 | > loss_text_ce: 0.040959808975458145 (0.03283820288485079) | > loss_mel_ce: 2.3608553409576416 (nan) | > loss: 0.009531012736260891 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13689560332319026) | > loader_time: 0.0045 (0.004680567253766638)  --> TIME: 2025-02-25 22:32:41 -- STEP: 12512/36938 -- GLOBAL_STEP: 972900 | > loss_text_ce: 0.02368372678756714 (0.03283508962941467) | > loss_mel_ce: 2.8920516967773438 (nan) | > loss: 0.011570379137992859 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.1368965406704446) | > loader_time: 0.0045 (0.004680228164738708)  --> TIME: 2025-02-25 22:32:50 -- STEP: 12562/36938 -- GLOBAL_STEP: 972950 | > loss_text_ce: 0.03365197405219078 (0.03283702139249646) | > loss_mel_ce: 3.3249199390411377 (nan) | > loss: 0.013327667489647865 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.1368964952451571) | > loader_time: 0.0046 (0.004680001456690831)  --> TIME: 2025-02-25 22:33:00 -- STEP: 12612/36938 -- GLOBAL_STEP: 973000 | > loss_text_ce: 0.032195065170526505 (0.03283753881941029) | > loss_mel_ce: 2.4537816047668457 (nan) | > loss: 0.009864987805485725 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.1368894112363461) | > loader_time: 0.0046 (0.004679692309506921)  --> TIME: 2025-02-25 22:33:09 -- STEP: 12662/36938 -- GLOBAL_STEP: 973050 | > loss_text_ce: 0.0304778553545475 (0.03283505935999797) | > loss_mel_ce: 2.62528920173645 (nan) | > loss: 0.010538758710026741 (nan) | > current_lr: 5e-06 | > step_time: 0.1265 (0.13689892888163377) | > loader_time: 0.0048 (0.004679577796121402)  --> TIME: 2025-02-25 22:33:19 -- STEP: 12712/36938 -- GLOBAL_STEP: 973100 | > loss_text_ce: 0.014657788909971714 (0.032831987536417286) | > loss_mel_ce: 1.634016513824463 (nan) | > loss: 0.006542358547449112 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13690982390455433) | > loader_time: 0.0044 (0.004679435731630945)  --> TIME: 2025-02-25 22:33:28 -- STEP: 12762/36938 -- GLOBAL_STEP: 973150 | > loss_text_ce: 0.01766127347946167 (0.03283964829575431) | > loss_mel_ce: 2.5357706546783447 (nan) | > loss: 0.010132667608559132 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13691564062848233) | > loader_time: 0.0046 (0.004679297545247277)  --> TIME: 2025-02-25 22:33:37 -- STEP: 12812/36938 -- GLOBAL_STEP: 973200 | > loss_text_ce: 0.03429800271987915 (0.03284607596817249) | > loss_mel_ce: 1.6900060176849365 (nan) | > loss: 0.006842476315796375 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.1369102545317516) | > loader_time: 0.0046 (0.004679116073564341)  --> TIME: 2025-02-25 22:33:47 -- STEP: 12862/36938 -- GLOBAL_STEP: 973250 | > loss_text_ce: 0.033463262021541595 (0.032846372340180306) | > loss_mel_ce: 2.7838196754455566 (nan) | > loss: 0.011179694905877113 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13690605355984867) | > loader_time: 0.0046 (0.004678899124832408)  --> TIME: 2025-02-25 22:33:56 -- STEP: 12912/36938 -- GLOBAL_STEP: 973300 | > loss_text_ce: 0.03550349920988083 (0.03285097974894074) | > loss_mel_ce: 1.3759218454360962 (nan) | > loss: 0.0056008948013186455 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.1369076384303007) | > loader_time: 0.0046 (0.00467861967237466)  --> TIME: 2025-02-25 22:34:05 -- STEP: 12962/36938 -- GLOBAL_STEP: 973350 | > loss_text_ce: 0.04908079653978348 (0.032846083595562375) | > loss_mel_ce: 0.39617976546287537 (nan) | > loss: 0.001766907051205635 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.1369045405195857) | > loader_time: 0.0045 (0.004678717054641652)  --> TIME: 2025-02-25 22:34:14 -- STEP: 13012/36938 -- GLOBAL_STEP: 973400 | > loss_text_ce: 0.031087078154087067 (0.03284120435589995) | > loss_mel_ce: 3.100193738937378 (nan) | > loss: 0.012425718829035759 (nan) | > current_lr: 5e-06 | > step_time: 0.1825 (0.136902833567596) | > loader_time: 0.0049 (0.004678611329545111)  --> TIME: 2025-02-25 22:34:24 -- STEP: 13062/36938 -- GLOBAL_STEP: 973450 | > loss_text_ce: 0.025409676134586334 (0.03284319014900379) | > loss_mel_ce: 1.7033895254135132 (nan) | > loss: 0.00686031486839056 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.136906665821862) | > loader_time: 0.0045 (0.004678792016025425)  --> TIME: 2025-02-25 22:34:33 -- STEP: 13112/36938 -- GLOBAL_STEP: 973500 | > loss_text_ce: 0.03310456871986389 (0.03284395448421706) | > loss_mel_ce: 2.591695547103882 (nan) | > loss: 0.010415874421596527 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13691500176592183) | > loader_time: 0.005 (0.0046788268768620175)  --> TIME: 2025-02-25 22:34:43 -- STEP: 13162/36938 -- GLOBAL_STEP: 973550 | > loss_text_ce: 0.03147008642554283 (0.0328453845190796) | > loss_mel_ce: 3.257657527923584 (nan) | > loss: 0.013052094727754593 (nan) | > current_lr: 5e-06 | > step_time: 0.1797 (0.1369251814534872) | > loader_time: 0.0048 (0.004678615156485857)  --> TIME: 2025-02-25 22:34:52 -- STEP: 13212/36938 -- GLOBAL_STEP: 973600 | > loss_text_ce: 0.03280477598309517 (0.03284807583166867) | > loss_mel_ce: 2.225113868713379 (nan) | > loss: 0.008959994651377201 (nan) | > current_lr: 5e-06 | > step_time: 0.1689 (0.13692843405145907) | > loader_time: 0.0048 (0.004678539027238021)  --> TIME: 2025-02-25 22:35:02 -- STEP: 13262/36938 -- GLOBAL_STEP: 973650 | > loss_text_ce: 0.03218366205692291 (0.03284863493048674) | > loss_mel_ce: 2.8118488788604736 (nan) | > loss: 0.011285844258964062 (nan) | > current_lr: 5e-06 | > step_time: 0.145 (0.136931740883483) | > loader_time: 0.0047 (0.004678561845312584)  --> TIME: 2025-02-25 22:35:11 -- STEP: 13312/36938 -- GLOBAL_STEP: 973700 | > loss_text_ce: 0.0334295928478241 (0.03284472778355284) | > loss_mel_ce: 2.777557373046875 (nan) | > loss: 0.011154711246490479 (nan) | > current_lr: 5e-06 | > step_time: 0.1617 (0.13691099084770542) | > loader_time: 0.0046 (0.004678333661733917)  --> TIME: 2025-02-25 22:35:20 -- STEP: 13362/36938 -- GLOBAL_STEP: 973750 | > loss_text_ce: 0.034805309027433395 (0.03284456046206694) | > loss_mel_ce: 3.1696078777313232 (nan) | > loss: 0.012715926393866539 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1369015584715013) | > loader_time: 0.0046 (0.004678093143395192)  --> TIME: 2025-02-25 22:35:29 -- STEP: 13412/36938 -- GLOBAL_STEP: 973800 | > loss_text_ce: 0.02846716344356537 (0.03284500319665684) | > loss_mel_ce: 2.617210865020752 (nan) | > loss: 0.01049872301518917 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13689535413611892) | > loader_time: 0.0046 (0.004678000203538505)  --> TIME: 2025-02-25 22:35:39 -- STEP: 13462/36938 -- GLOBAL_STEP: 973850 | > loss_text_ce: 0.033929936587810516 (0.03284028020639388) | > loss_mel_ce: 2.6521012783050537 (nan) | > loss: 0.010658854618668556 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13689234966580802) | > loader_time: 0.0046 (0.0046779144892383)  --> TIME: 2025-02-25 22:35:47 -- STEP: 13512/36938 -- GLOBAL_STEP: 973900 | > loss_text_ce: 0.033477213233709335 (0.032845045956289516) | > loss_mel_ce: 2.4076061248779297 (nan) | > loss: 0.009686839766800404 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13686566752678914) | > loader_time: 0.0046 (0.004677651459963247)  --> TIME: 2025-02-25 22:35:57 -- STEP: 13562/36938 -- GLOBAL_STEP: 973950 | > loss_text_ce: 0.026191916316747665 (0.03284186777565234) | > loss_mel_ce: 2.9901723861694336 (nan) | > loss: 0.011969700455665588 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13686449549539642) | > loader_time: 0.0047 (0.004677424387249025)  --> TIME: 2025-02-25 22:36:06 -- STEP: 13612/36938 -- GLOBAL_STEP: 974000 | > loss_text_ce: 0.03219541162252426 (0.032841792672652526) | > loss_mel_ce: 2.997863531112671 (nan) | > loss: 0.012024044059216976 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.13686343289248448) | > loader_time: 0.0046 (0.004677445615980864)  --> TIME: 2025-02-25 22:36:15 -- STEP: 13662/36938 -- GLOBAL_STEP: 974050 | > loss_text_ce: 0.030925558879971504 (0.03283898685688666) | > loss_mel_ce: 3.014143228530884 (nan) | > loss: 0.012083606794476509 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1368590407133496) | > loader_time: 0.0046 (0.0046776146582289515)  --> TIME: 2025-02-25 22:36:25 -- STEP: 13712/36938 -- GLOBAL_STEP: 974100 | > loss_text_ce: 0.03559180349111557 (0.03284065406880908) | > loss_mel_ce: 3.5370404720306396 (nan) | > loss: 0.014177112840116024 (nan) | > current_lr: 5e-06 | > step_time: 0.1833 (0.1368547575059275) | > loader_time: 0.0048 (0.004677769409594768)  --> TIME: 2025-02-25 22:36:34 -- STEP: 13762/36938 -- GLOBAL_STEP: 974150 | > loss_text_ce: 0.028478389605879784 (0.0328445188386336) | > loss_mel_ce: 1.9794955253601074 (nan) | > loss: 0.00796815101057291 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1368662949790626) | > loader_time: 0.0046 (0.004677758177358251)  --> TIME: 2025-02-25 22:36:44 -- STEP: 13812/36938 -- GLOBAL_STEP: 974200 | > loss_text_ce: 0.02841557376086712 (0.03284563532472957) | > loss_mel_ce: 2.9326906204223633 (nan) | > loss: 0.011750422418117523 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13688691436398676) | > loader_time: 0.0046 (0.004677699677605168)  --> TIME: 2025-02-25 22:36:53 -- STEP: 13862/36938 -- GLOBAL_STEP: 974250 | > loss_text_ce: 0.041254620999097824 (0.032852423843735434) | > loss_mel_ce: 2.9036567211151123 (nan) | > loss: 0.011686156503856182 (nan) | > current_lr: 5e-06 | > step_time: 0.1357 (0.13688410919408595) | > loader_time: 0.0046 (0.00467784993663321)  --> TIME: 2025-02-25 22:37:02 -- STEP: 13912/36938 -- GLOBAL_STEP: 974300 | > loss_text_ce: 0.04312026873230934 (0.03284806612375088) | > loss_mel_ce: 2.832796096801758 (nan) | > loss: 0.011412367224693298 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.1368719355822853) | > loader_time: 0.0046 (0.004677849641265786)  --> TIME: 2025-02-25 22:37:12 -- STEP: 13962/36938 -- GLOBAL_STEP: 974350 | > loss_text_ce: 0.02796895243227482 (0.03285013001610669) | > loss_mel_ce: 1.931640625 (nan) | > loss: 0.007776229176670313 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13687615834476535) | > loader_time: 0.0045 (0.004677938093276411)  --> TIME: 2025-02-25 22:37:21 -- STEP: 14012/36938 -- GLOBAL_STEP: 974400 | > loss_text_ce: 0.026717863976955414 (0.032848445290384366) | > loss_mel_ce: 2.043992519378662 (nan) | > loss: 0.008217105641961098 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13687571063914322) | > loader_time: 0.0048 (0.004677811299873158)  --> TIME: 2025-02-25 22:37:30 -- STEP: 14062/36938 -- GLOBAL_STEP: 974450 | > loss_text_ce: 0.02879771962761879 (0.032846100527177034) | > loss_mel_ce: 2.7982375621795654 (nan) | > loss: 0.01121839415282011 (nan) | > current_lr: 5e-06 | > step_time: 0.1434 (0.13687822712279962) | > loader_time: 0.0047 (0.004677818215193445)  --> TIME: 2025-02-25 22:37:40 -- STEP: 14112/36938 -- GLOBAL_STEP: 974500 | > loss_text_ce: 0.034721970558166504 (0.032846024518097705) | > loss_mel_ce: 2.8258399963378906 (nan) | > loss: 0.011351436376571655 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13686902947774623) | > loader_time: 0.0046 (0.004677923358216582)  --> TIME: 2025-02-25 22:37:49 -- STEP: 14162/36938 -- GLOBAL_STEP: 974550 | > loss_text_ce: 0.023454934358596802 (0.03284535807479943) | > loss_mel_ce: 2.2139790058135986 (nan) | > loss: 0.008878706954419613 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.1368581004574421) | > loader_time: 0.0046 (0.004678155284634322)  --> TIME: 2025-02-25 22:37:58 -- STEP: 14212/36938 -- GLOBAL_STEP: 974600 | > loss_text_ce: 0.02628565952181816 (0.03284143274834779) | > loss_mel_ce: 2.686147451400757 (nan) | > loss: 0.010763623751699924 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13685704552553635) | > loader_time: 0.0046 (0.004678149693722867)  --> TIME: 2025-02-25 22:38:07 -- STEP: 14262/36938 -- GLOBAL_STEP: 974650 | > loss_text_ce: 0.026394983753561974 (0.03283956327656235) | > loss_mel_ce: 2.576871633529663 (nan) | > loss: 0.010330423712730408 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.1368573038762932) | > loader_time: 0.0046 (0.004678045745451614)  --> TIME: 2025-02-25 22:38:17 -- STEP: 14312/36938 -- GLOBAL_STEP: 974700 | > loss_text_ce: 0.034906622022390366 (0.032839495557685255) | > loss_mel_ce: 2.78857684135437 (nan) | > loss: 0.011204300448298454 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13685972654599668) | > loader_time: 0.0045 (0.004678358623207186)  --> TIME: 2025-02-25 22:38:26 -- STEP: 14362/36938 -- GLOBAL_STEP: 974750 | > loss_text_ce: 0.036327093839645386 (0.03284048400148799) | > loss_mel_ce: 3.1277294158935547 (nan) | > loss: 0.012555780820548534 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13685895211018395) | > loader_time: 0.0046 (0.0046783392848684345)  --> TIME: 2025-02-25 22:38:36 -- STEP: 14412/36938 -- GLOBAL_STEP: 974800 | > loss_text_ce: 0.015289518050849438 (0.03284240590344864) | > loss_mel_ce: 2.39117431640625 (nan) | > loss: 0.009549460373818874 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13686109417979736) | > loader_time: 0.0045 (0.00467825134429805)  --> TIME: 2025-02-25 22:38:45 -- STEP: 14462/36938 -- GLOBAL_STEP: 974850 | > loss_text_ce: 0.0338548980653286 (0.03284053323676324) | > loss_mel_ce: 3.141615629196167 (nan) | > loss: 0.012601074762642384 (nan) | > current_lr: 5e-06 | > step_time: 0.1708 (0.1368507976074343) | > loader_time: 0.0049 (0.004678128913401441)  --> TIME: 2025-02-25 22:38:54 -- STEP: 14512/36938 -- GLOBAL_STEP: 974900 | > loss_text_ce: 0.042518433183431625 (0.03284279392598884) | > loss_mel_ce: 2.91886568069458 (nan) | > loss: 0.011751525104045868 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13685906805763515) | > loader_time: 0.0046 (0.004677975864499778)  --> TIME: 2025-02-25 22:39:04 -- STEP: 14562/36938 -- GLOBAL_STEP: 974950 | > loss_text_ce: 0.03568551316857338 (0.032843005784493486) | > loss_mel_ce: 3.0937747955322266 (nan) | > loss: 0.012418494559824467 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.1368686291660986) | > loader_time: 0.0046 (0.004677991555330113)  --> TIME: 2025-02-25 22:39:13 -- STEP: 14612/36938 -- GLOBAL_STEP: 975000 | > loss_text_ce: 0.03345717117190361 (0.03284112762013757) | > loss_mel_ce: 2.8818087577819824 (nan) | > loss: 0.011568516492843628 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13687801002118233) | > loader_time: 0.0046 (0.004677745077403125)  --> TIME: 2025-02-25 22:39:22 -- STEP: 14662/36938 -- GLOBAL_STEP: 975050 | > loss_text_ce: 0.03791447728872299 (0.03284168150911145) | > loss_mel_ce: 2.431396484375 (nan) | > loss: 0.009798853658139706 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13686760166887935) | > loader_time: 0.0046 (0.004677726942428449)  --> TIME: 2025-02-25 22:39:32 -- STEP: 14712/36938 -- GLOBAL_STEP: 975100 | > loss_text_ce: 0.034739959985017776 (0.03284325179504157) | > loss_mel_ce: 2.528109550476074 (nan) | > loss: 0.010170038789510727 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1368657367718234) | > loader_time: 0.0046 (0.004677756105576478)  --> TIME: 2025-02-25 22:39:41 -- STEP: 14762/36938 -- GLOBAL_STEP: 975150 | > loss_text_ce: 0.03323575109243393 (0.03284655508590701) | > loss_mel_ce: 2.8549716472625732 (nan) | > loss: 0.011461141519248486 (nan) | > current_lr: 5e-06 | > step_time: 0.1785 (0.13686369008414626) | > loader_time: 0.0048 (0.004677766513863229)  --> TIME: 2025-02-25 22:39:51 -- STEP: 14812/36938 -- GLOBAL_STEP: 975200 | > loss_text_ce: 0.035845525562763214 (0.03284785640369391) | > loss_mel_ce: 0.9703640341758728 (nan) | > loss: 0.003992895595729351 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13688203786599729) | > loader_time: 0.0044 (0.00467777097672667)  --> TIME: 2025-02-25 22:40:00 -- STEP: 14862/36938 -- GLOBAL_STEP: 975250 | > loss_text_ce: 0.032387204468250275 (0.032846976764040955) | > loss_mel_ce: 2.5888445377349854 (nan) | > loss: 0.010401713661849499 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13689407121925082) | > loader_time: 0.0045 (0.004677874213223698)  --> TIME: 2025-02-25 22:40:10 -- STEP: 14912/36938 -- GLOBAL_STEP: 975300 | > loss_text_ce: 0.031195018440485 (0.03284627787744863) | > loss_mel_ce: 2.148021697998047 (nan) | > loss: 0.008647685870528221 (nan) | > current_lr: 5e-06 | > step_time: 0.1489 (0.1368993173469297) | > loader_time: 0.0048 (0.004677672914425194)  --> TIME: 2025-02-25 22:40:19 -- STEP: 14962/36938 -- GLOBAL_STEP: 975350 | > loss_text_ce: 0.02244666777551174 (0.03284748900425784) | > loss_mel_ce: 2.6831958293914795 (nan) | > loss: 0.010736676864326 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.13690178449452595) | > loader_time: 0.0047 (0.004677772665387124)  --> TIME: 2025-02-25 22:40:28 -- STEP: 15012/36938 -- GLOBAL_STEP: 975400 | > loss_text_ce: 0.030664395540952682 (0.03285119409221034) | > loss_mel_ce: 1.883055329322815 (nan) | > loss: 0.007594126742333174 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.1368979849626064) | > loader_time: 0.0045 (0.004677510868222947)  --> TIME: 2025-02-25 22:40:37 -- STEP: 15062/36938 -- GLOBAL_STEP: 975450 | > loss_text_ce: 0.030102161690592766 (0.032846526370044156) | > loss_mel_ce: 3.174453020095825 (nan) | > loss: 0.012716489844024181 (nan) | > current_lr: 5e-06 | > step_time: 0.1632 (0.13689235237834851) | > loader_time: 0.0046 (0.004677504819475663)  --> TIME: 2025-02-25 22:40:47 -- STEP: 15112/36938 -- GLOBAL_STEP: 975500 | > loss_text_ce: 0.04889234900474548 (0.032855230586700986) | > loss_mel_ce: 2.87111496925354 (nan) | > loss: 0.01158733107149601 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.1368807388588896) | > loader_time: 0.0051 (0.0046779509257109135)  --> TIME: 2025-02-25 22:40:56 -- STEP: 15162/36938 -- GLOBAL_STEP: 975550 | > loss_text_ce: 0.027060601860284805 (0.0328509592595013) | > loss_mel_ce: 2.548088550567627 (nan) | > loss: 0.010218845680356026 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13688149944079558) | > loader_time: 0.0047 (0.004677892250178786)  --> TIME: 2025-02-25 22:41:05 -- STEP: 15212/36938 -- GLOBAL_STEP: 975600 | > loss_text_ce: 0.04188761115074158 (0.03284939941689343) | > loss_mel_ce: 2.9170167446136475 (nan) | > loss: 0.011741683818399906 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13688163930668326) | > loader_time: 0.0046 (0.004678493137896269)  --> TIME: 2025-02-25 22:41:15 -- STEP: 15262/36938 -- GLOBAL_STEP: 975650 | > loss_text_ce: 0.028577221557497978 (0.03284624783543618) | > loss_mel_ce: 3.1572265625 (nan) | > loss: 0.012642079964280128 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13687931316549767) | > loader_time: 0.0045 (0.004678579024133797)  --> TIME: 2025-02-25 22:41:24 -- STEP: 15312/36938 -- GLOBAL_STEP: 975700 | > loss_text_ce: 0.03182663023471832 (0.032843684738447516) | > loss_mel_ce: 2.044116497039795 (nan) | > loss: 0.008237870410084724 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.1368706974365377) | > loader_time: 0.0046 (0.004678474558184517)  --> TIME: 2025-02-25 22:41:33 -- STEP: 15362/36938 -- GLOBAL_STEP: 975750 | > loss_text_ce: 0.029298173263669014 (0.03283918874047035) | > loss_mel_ce: 1.542189121246338 (nan) | > loss: 0.006236060988157988 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13685917449236576) | > loader_time: 0.0047 (0.004678481569701135)  --> TIME: 2025-02-25 22:41:42 -- STEP: 15412/36938 -- GLOBAL_STEP: 975800 | > loss_text_ce: 0.029732786118984222 (0.03283864517301974) | > loss_mel_ce: 2.510674476623535 (nan) | > loss: 0.010080981068313122 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13685084982658474) | > loader_time: 0.0045 (0.004678729924479928)  --> TIME: 2025-02-25 22:41:51 -- STEP: 15462/36938 -- GLOBAL_STEP: 975850 | > loss_text_ce: 0.028285851702094078 (0.03284160949999594) | > loss_mel_ce: 2.4871108531951904 (nan) | > loss: 0.009981732815504074 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.1368456367933808) | > loader_time: 0.0045 (0.004679102975359067)  --> TIME: 2025-02-25 22:42:01 -- STEP: 15512/36938 -- GLOBAL_STEP: 975900 | > loss_text_ce: 0.03344593942165375 (0.03283680139636405) | > loss_mel_ce: 2.9405970573425293 (nan) | > loss: 0.011801757849752903 (nan) | > current_lr: 5e-06 | > step_time: 0.1857 (0.13685667190495268) | > loader_time: 0.0049 (0.004679117330636731)  --> TIME: 2025-02-25 22:42:10 -- STEP: 15562/36938 -- GLOBAL_STEP: 975950 | > loss_text_ce: 0.0361005999147892 (0.03283461338107085) | > loss_mel_ce: 3.0171964168548584 (nan) | > loss: 0.012116258963942528 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13684882075571253) | > loader_time: 0.0047 (0.004679156719389653)  --> TIME: 2025-02-25 22:42:19 -- STEP: 15612/36938 -- GLOBAL_STEP: 976000 | > loss_text_ce: 0.028583884239196777 (0.03283506956525578) | > loss_mel_ce: 2.7252321243286133 (nan) | > loss: 0.010927841998636723 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13684122691605363) | > loader_time: 0.0047 (0.0046790559994939265)  --> TIME: 2025-02-25 22:42:29 -- STEP: 15662/36938 -- GLOBAL_STEP: 976050 | > loss_text_ce: 0.02761301025748253 (0.03282948452877062) | > loss_mel_ce: 2.864811658859253 (nan) | > loss: 0.011477876454591751 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.1368475250624838) | > loader_time: 0.0046 (0.0046792956333844855)  --> TIME: 2025-02-25 22:42:38 -- STEP: 15712/36938 -- GLOBAL_STEP: 976100 | > loss_text_ce: 0.0331081859767437 (0.03282506044590366) | > loss_mel_ce: 1.0089064836502075 (nan) | > loss: 0.004134979099035263 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.136847831490934) | > loader_time: 0.0046 (0.004679657078809023)  --> TIME: 2025-02-25 22:42:47 -- STEP: 15762/36938 -- GLOBAL_STEP: 976150 | > loss_text_ce: 0.036975424736738205 (0.03282520625810734) | > loss_mel_ce: 3.330460548400879 (nan) | > loss: 0.013362841680645943 (nan) | > current_lr: 5e-06 | > step_time: 0.1609 (0.1368593543242302) | > loader_time: 0.0048 (0.004679931312810802)  --> TIME: 2025-02-25 22:42:56 -- STEP: 15812/36938 -- GLOBAL_STEP: 976200 | > loss_text_ce: 0.03284623473882675 (0.032828916336487386) | > loss_mel_ce: 0.892016589641571 (nan) | > loss: 0.0036700908094644547 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1368390306804026) | > loader_time: 0.0048 (0.004680031497298152)  --> TIME: 2025-02-25 22:43:06 -- STEP: 15862/36938 -- GLOBAL_STEP: 976250 | > loss_text_ce: 0.030847560614347458 (0.03282890259179643) | > loss_mel_ce: 3.0441997051239014 (nan) | > loss: 0.012202569283545017 (nan) | > current_lr: 5e-06 | > step_time: 0.1961 (0.1368393859080484) | > loader_time: 0.0049 (0.004680176368052508)  --> TIME: 2025-02-25 22:43:15 -- STEP: 15912/36938 -- GLOBAL_STEP: 976300 | > loss_text_ce: 0.038624025881290436 (0.03282929521172038) | > loss_mel_ce: 2.498941659927368 (nan) | > loss: 0.010069705545902252 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13683309150197695) | > loader_time: 0.0046 (0.00468011255118043)  --> TIME: 2025-02-25 22:43:24 -- STEP: 15962/36938 -- GLOBAL_STEP: 976350 | > loss_text_ce: 0.03705810382962227 (0.03283069208740469) | > loss_mel_ce: 3.1357421875 (nan) | > loss: 0.012590478174388409 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13681843767427895) | > loader_time: 0.0045 (0.00468012168135295)  --> TIME: 2025-02-25 22:43:33 -- STEP: 16012/36938 -- GLOBAL_STEP: 976400 | > loss_text_ce: 0.03412388265132904 (0.032829938942667614) | > loss_mel_ce: 2.732670783996582 (nan) | > loss: 0.010979345068335533 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1368173357846593) | > loader_time: 0.0046 (0.004680091623601219)  --> TIME: 2025-02-25 22:43:42 -- STEP: 16062/36938 -- GLOBAL_STEP: 976450 | > loss_text_ce: 0.027295134961605072 (0.03282792210454152) | > loss_mel_ce: 1.186008334159851 (nan) | > loss: 0.0048146964982151985 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13680511724160754) | > loader_time: 0.0046 (0.004679945215551308)  --> TIME: 2025-02-25 22:43:51 -- STEP: 16112/36938 -- GLOBAL_STEP: 976500 | > loss_text_ce: 0.0345037579536438 (0.03282704535150837) | > loss_mel_ce: 2.576098680496216 (nan) | > loss: 0.010359534062445164 (nan) | > current_lr: 5e-06 | > step_time: 0.1533 (0.1367943319894358) | > loader_time: 0.0047 (0.004679860874581364)  --> TIME: 2025-02-25 22:44:01 -- STEP: 16162/36938 -- GLOBAL_STEP: 976550 | > loss_text_ce: 0.029462549835443497 (0.03282832672070163) | > loss_mel_ce: 2.893284797668457 (nan) | > loss: 0.011598204262554646 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.13678972562665598) | > loader_time: 0.0045 (0.004679619314469647)  --> TIME: 2025-02-25 22:44:10 -- STEP: 16212/36938 -- GLOBAL_STEP: 976600 | > loss_text_ce: 0.03547289967536926 (0.032829946865893744) | > loss_mel_ce: 3.2535386085510254 (nan) | > loss: 0.013051633723080158 (nan) | > current_lr: 5e-06 | > step_time: 0.15 (0.13678425778408573) | > loader_time: 0.0046 (0.00467934411100891)  --> TIME: 2025-02-25 22:44:19 -- STEP: 16262/36938 -- GLOBAL_STEP: 976650 | > loss_text_ce: 0.02860236167907715 (0.03282793607176107) | > loss_mel_ce: 3.17071270942688 (nan) | > loss: 0.012695695273578167 (nan) | > current_lr: 5e-06 | > step_time: 0.1615 (0.1367751730278612) | > loader_time: 0.0048 (0.004679246782127631)  --> TIME: 2025-02-25 22:44:28 -- STEP: 16312/36938 -- GLOBAL_STEP: 976700 | > loss_text_ce: 0.02674121782183647 (0.032824382320792884) | > loss_mel_ce: 3.128077507019043 (nan) | > loss: 0.012519123032689095 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13675492845545167) | > loader_time: 0.0046 (0.004679036365530551)  --> TIME: 2025-02-25 22:44:37 -- STEP: 16362/36938 -- GLOBAL_STEP: 976750 | > loss_text_ce: 0.046283330768346786 (0.03282309079579425) | > loss_mel_ce: 3.185863494873047 (nan) | > loss: 0.012825979851186275 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13675320776498934) | > loader_time: 0.0047 (0.0046789549539759815)  --> TIME: 2025-02-25 22:44:47 -- STEP: 16412/36938 -- GLOBAL_STEP: 976800 | > loss_text_ce: 0.035400960594415665 (0.032821755662543334) | > loss_mel_ce: 0.03259905055165291 (nan) | > loss: 0.0002698413154575974 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13676413003915533) | > loader_time: 0.0046 (0.0046789425627475805)  --> TIME: 2025-02-25 22:44:56 -- STEP: 16462/36938 -- GLOBAL_STEP: 976850 | > loss_text_ce: 0.038398291915655136 (0.03282622447052501) | > loss_mel_ce: 3.3610153198242188 (nan) | > loss: 0.013489737175405025 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1367699025019371) | > loader_time: 0.0045 (0.004678875399800298)  --> TIME: 2025-02-25 22:45:06 -- STEP: 16512/36938 -- GLOBAL_STEP: 976900 | > loss_text_ce: 0.032007262110710144 (0.032826791202709156) | > loss_mel_ce: 2.70623779296875 (nan) | > loss: 0.010866052471101284 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.13677720852600575) | > loader_time: 0.005 (0.004679068172162822)  --> TIME: 2025-02-25 22:45:15 -- STEP: 16562/36938 -- GLOBAL_STEP: 976950 | > loss_text_ce: 0.02779446355998516 (0.03282504838579475) | > loss_mel_ce: 2.457548141479492 (nan) | > loss: 0.00986247044056654 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13677659875139442) | > loader_time: 0.0046 (0.004679069946877787)  --> TIME: 2025-02-25 22:45:24 -- STEP: 16612/36938 -- GLOBAL_STEP: 977000 | > loss_text_ce: 0.034788213670253754 (0.032826234469252716) | > loss_mel_ce: 2.6858596801757812 (nan) | > loss: 0.010796221904456615 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13676999397746348) | > loader_time: 0.0045 (0.004679326993311689)  --> TIME: 2025-02-25 22:45:33 -- STEP: 16662/36938 -- GLOBAL_STEP: 977050 | > loss_text_ce: 0.028908779844641685 (0.03282513786740047) | > loss_mel_ce: 2.6208910942077637 (nan) | > loss: 0.010515078902244568 (nan) | > current_lr: 5e-06 | > step_time: 0.1263 (0.13675927025489745) | > loader_time: 0.0046 (0.004679093368341627)  --> TIME: 2025-02-25 22:45:43 -- STEP: 16712/36938 -- GLOBAL_STEP: 977100 | > loss_text_ce: 0.035782381892204285 (0.03282522236272552) | > loss_mel_ce: 3.2370238304138184 (nan) | > loss: 0.012987326830625534 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.1367591994124894) | > loader_time: 0.0048 (0.004678897877068203)  --> TIME: 2025-02-25 22:45:52 -- STEP: 16762/36938 -- GLOBAL_STEP: 977150 | > loss_text_ce: 0.021241314709186554 (0.03282716783376347) | > loss_mel_ce: 2.5372588634490967 (nan) | > loss: 0.010152779519557953 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13677193899169518) | > loader_time: 0.0045 (0.004678980346341053)  --> TIME: 2025-02-25 22:46:02 -- STEP: 16812/36938 -- GLOBAL_STEP: 977200 | > loss_text_ce: 0.03525366261601448 (0.032824444212507076) | > loss_mel_ce: 0.7498751282691956 (nan) | > loss: 0.003115590661764145 (nan) | > current_lr: 5e-06 | > step_time: 0.1353 (0.13676805597471453) | > loader_time: 0.0046 (0.004678903251723286)  --> TIME: 2025-02-25 22:46:11 -- STEP: 16862/36938 -- GLOBAL_STEP: 977250 | > loss_text_ce: 0.02918587625026703 (0.03282552527299344) | > loss_mel_ce: 0.0967254638671875 (nan) | > loss: 0.0004996482166461647 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13676445851220287) | > loader_time: 0.0047 (0.004678923737859976)  --> TIME: 2025-02-25 22:46:20 -- STEP: 16912/36938 -- GLOBAL_STEP: 977300 | > loss_text_ce: 0.03242622688412666 (0.032825129548512556) | > loss_mel_ce: 2.727997303009033 (nan) | > loss: 0.010954061523079872 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.1367569708581685) | > loader_time: 0.0047 (0.004678768009765039)  --> TIME: 2025-02-25 22:46:29 -- STEP: 16962/36938 -- GLOBAL_STEP: 977350 | > loss_text_ce: 0.044740136712789536 (0.03282572028519595) | > loss_mel_ce: 2.300081968307495 (nan) | > loss: 0.009304850362241268 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13675523880953103) | > loader_time: 0.0047 (0.004678609727927645)  --> TIME: 2025-02-25 22:46:38 -- STEP: 17012/36938 -- GLOBAL_STEP: 977400 | > loss_text_ce: 0.02519441954791546 (0.03282110638983952) | > loss_mel_ce: 2.078904151916504 (nan) | > loss: 0.00834959838539362 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13673842072402848) | > loader_time: 0.0048 (0.004678546793736099)  --> TIME: 2025-02-25 22:46:48 -- STEP: 17062/36938 -- GLOBAL_STEP: 977450 | > loss_text_ce: 0.03392083942890167 (0.03282270763421365) | > loss_mel_ce: 2.567291498184204 (nan) | > loss: 0.010322271846234798 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13674108932619486) | > loader_time: 0.0045 (0.004678536168492959)  --> TIME: 2025-02-25 22:46:57 -- STEP: 17112/36938 -- GLOBAL_STEP: 977500 | > loss_text_ce: 0.029387811198830605 (0.032822343650243196) | > loss_mel_ce: 2.5708940029144287 (nan) | > loss: 0.010318579152226448 (nan) | > current_lr: 5e-06 | > step_time: 0.1817 (0.1367313364723571) | > loader_time: 0.0049 (0.004678505347008017)  --> TIME: 2025-02-25 22:47:06 -- STEP: 17162/36938 -- GLOBAL_STEP: 977550 | > loss_text_ce: 0.027311235666275024 (0.032820581667869476) | > loss_mel_ce: 3.051608085632324 (nan) | > loss: 0.012217935174703598 (nan) | > current_lr: 5e-06 | > step_time: 0.1999 (0.13673901298137567) | > loader_time: 0.0051 (0.004678850448460814)  --> TIME: 2025-02-25 22:47:15 -- STEP: 17212/36938 -- GLOBAL_STEP: 977600 | > loss_text_ce: 0.033160094171762466 (0.032818630515084296) | > loss_mel_ce: 2.8942313194274902 (nan) | > loss: 0.011616633273661137 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13672218956061957) | > loader_time: 0.0045 (0.004678675235218159)  --> TIME: 2025-02-25 22:47:25 -- STEP: 17262/36938 -- GLOBAL_STEP: 977650 | > loss_text_ce: 0.025630570948123932 (0.03281838837016163) | > loss_mel_ce: 2.8125 (nan) | > loss: 0.011262423358857632 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13674599315542577) | > loader_time: 0.0046 (0.004678740657200457)  --> TIME: 2025-02-25 22:47:35 -- STEP: 17312/36938 -- GLOBAL_STEP: 977700 | > loss_text_ce: 0.03789188712835312 (0.03281809594383213) | > loss_mel_ce: 3.0303215980529785 (nan) | > loss: 0.012175451032817364 (nan) | > current_lr: 5e-06 | > step_time: 0.1442 (0.13676303075609314) | > loader_time: 0.0046 (0.004679025362577994)  --> TIME: 2025-02-25 22:47:45 -- STEP: 17362/36938 -- GLOBAL_STEP: 977750 | > loss_text_ce: 0.032492492347955704 (0.03281967097412959) | > loss_mel_ce: 3.2254464626312256 (nan) | > loss: 0.012928329408168793 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13677992286282284) | > loader_time: 0.0045 (0.004679123853541326)  --> TIME: 2025-02-25 22:47:54 -- STEP: 17412/36938 -- GLOBAL_STEP: 977800 | > loss_text_ce: 0.030794015154242516 (0.032820381257959615) | > loss_mel_ce: 2.2231569290161133 (nan) | > loss: 0.008944249711930752 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.1367910235825832) | > loader_time: 0.0047 (0.004679242742491577)  --> TIME: 2025-02-25 22:48:03 -- STEP: 17462/36938 -- GLOBAL_STEP: 977850 | > loss_text_ce: 0.03823743760585785 (0.03281806495819137) | > loss_mel_ce: 3.767890214920044 (nan) | > loss: 0.015103681944310665 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.13678282400325031) | > loader_time: 0.0046 (0.004679203279211475)  --> TIME: 2025-02-25 22:48:12 -- STEP: 17512/36938 -- GLOBAL_STEP: 977900 | > loss_text_ce: 0.03277470916509628 (0.03281456962716684) | > loss_mel_ce: 3.0528147220611572 (nan) | > loss: 0.01224440336227417 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13677000708098644) | > loader_time: 0.0045 (0.00467938059483996)  --> TIME: 2025-02-25 22:48:22 -- STEP: 17562/36938 -- GLOBAL_STEP: 977950 | > loss_text_ce: 0.03525124117732048 (0.03281459324091582) | > loss_mel_ce: 2.1666979789733887 (nan) | > loss: 0.008737893775105476 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.13676511775712882) | > loader_time: 0.0046 (0.0046794060870110855)  --> TIME: 2025-02-25 22:48:31 -- STEP: 17612/36938 -- GLOBAL_STEP: 978000 | > loss_text_ce: 0.03553782403469086 (0.03281739734213915) | > loss_mel_ce: 2.905341386795044 (nan) | > loss: 0.011670155450701714 (nan) | > current_lr: 5e-06 | > step_time: 0.178 (0.13675806894915382) | > loader_time: 0.0052 (0.004679334345065339)  --> TIME: 2025-02-25 22:48:40 -- STEP: 17662/36938 -- GLOBAL_STEP: 978050 | > loss_text_ce: 0.02567598596215248 (0.03281633834146942) | > loss_mel_ce: 2.716458320617676 (nan) | > loss: 0.010881486348807812 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13675724771605727) | > loader_time: 0.0045 (0.004679314224356351)  --> TIME: 2025-02-25 22:48:49 -- STEP: 17712/36938 -- GLOBAL_STEP: 978100 | > loss_text_ce: 0.030044743791222572 (0.03281269178781741) | > loss_mel_ce: 2.738758087158203 (nan) | > loss: 0.010987313464283943 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13673500979172876) | > loader_time: 0.0045 (0.004679284067765603)  --> TIME: 2025-02-25 22:48:58 -- STEP: 17762/36938 -- GLOBAL_STEP: 978150 | > loss_text_ce: 0.035419903695583344 (0.03281288588648811) | > loss_mel_ce: 2.774881362915039 (nan) | > loss: 0.011151989921927452 (nan) | > current_lr: 5e-06 | > step_time: 0.1505 (0.13674152795426356) | > loader_time: 0.0047 (0.004679368471404003)  --> TIME: 2025-02-25 22:49:08 -- STEP: 17812/36938 -- GLOBAL_STEP: 978200 | > loss_text_ce: 0.03246605023741722 (0.03281037385611879) | > loss_mel_ce: 2.448895215988159 (nan) | > loss: 0.009846671484410763 (nan) | > current_lr: 5e-06 | > step_time: 0.1642 (0.1367564127735954) | > loader_time: 0.0047 (0.004679403759082364)  --> TIME: 2025-02-25 22:49:17 -- STEP: 17862/36938 -- GLOBAL_STEP: 978250 | > loss_text_ce: 0.03399346396327019 (0.03281044234479886) | > loss_mel_ce: 2.970949649810791 (nan) | > loss: 0.0119243785738945 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13675180467642584) | > loader_time: 0.0046 (0.004679620873286052)  --> TIME: 2025-02-25 22:49:27 -- STEP: 17912/36938 -- GLOBAL_STEP: 978300 | > loss_text_ce: 0.03863804414868355 (0.0328123417942191) | > loss_mel_ce: 3.2543952465057373 (nan) | > loss: 0.013067593798041344 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.1367542308028551) | > loader_time: 0.0045 (0.004679761823664408)  --> TIME: 2025-02-25 22:49:36 -- STEP: 17962/36938 -- GLOBAL_STEP: 978350 | > loss_text_ce: 0.03024504892528057 (0.03281317079324048) | > loss_mel_ce: 3.209655284881592 (nan) | > loss: 0.012856747955083847 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13675771679701487) | > loader_time: 0.0046 (0.004679929107085816)  --> TIME: 2025-02-25 22:49:45 -- STEP: 18012/36938 -- GLOBAL_STEP: 978400 | > loss_text_ce: 0.0242945346981287 (0.0328119214346095) | > loss_mel_ce: 2.4609763622283936 (nan) | > loss: 0.009862187318503857 (nan) | > current_lr: 5e-06 | > step_time: 0.1523 (0.136755120958087) | > loader_time: 0.0046 (0.004679868744395465)  --> TIME: 2025-02-25 22:49:55 -- STEP: 18062/36938 -- GLOBAL_STEP: 978450 | > loss_text_ce: 0.03131385147571564 (0.03280787615263517) | > loss_mel_ce: 2.736370801925659 (nan) | > loss: 0.010982876643538475 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13675015906719531) | > loader_time: 0.0048 (0.0046800505400839305)  --> TIME: 2025-02-25 22:50:04 -- STEP: 18112/36938 -- GLOBAL_STEP: 978500 | > loss_text_ce: 0.04122741520404816 (0.0328119298657211) | > loss_mel_ce: 0.031967781484127045 (nan) | > loss: 0.000290457159280777 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1367499753174644) | > loader_time: 0.0045 (0.004680152218986326)  --> TIME: 2025-02-25 22:50:13 -- STEP: 18162/36938 -- GLOBAL_STEP: 978550 | > loss_text_ce: 0.02807743102312088 (0.03281304027497345) | > loss_mel_ce: 3.0667366981506348 (nan) | > loss: 0.012281008996069431 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.13673574013957399) | > loader_time: 0.0047 (0.004679956489716749)  --> TIME: 2025-02-25 22:50:23 -- STEP: 18212/36938 -- GLOBAL_STEP: 978600 | > loss_text_ce: 0.023396184667944908 (0.03281350405373363) | > loss_mel_ce: 3.357977867126465 (nan) | > loss: 0.013418151997029781 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13674597467350905) | > loader_time: 0.0045 (0.00467997028778969)  --> TIME: 2025-02-25 22:50:32 -- STEP: 18262/36938 -- GLOBAL_STEP: 978650 | > loss_text_ce: 0.03283693641424179 (0.03281040107031731) | > loss_mel_ce: 3.1114585399627686 (nan) | > loss: 0.012477363459765911 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13674066194946477) | > loader_time: 0.0046 (0.004679965654347489)  --> TIME: 2025-02-25 22:50:41 -- STEP: 18312/36938 -- GLOBAL_STEP: 978700 | > loss_text_ce: 0.033819280564785004 (0.032810738275786774) | > loss_mel_ce: 3.0802879333496094 (nan) | > loss: 0.012357568368315697 (nan) | > current_lr: 5e-06 | > step_time: 0.1815 (0.13674237747388243) | > loader_time: 0.0049 (0.004679888617060198)  --> TIME: 2025-02-25 22:50:51 -- STEP: 18362/36938 -- GLOBAL_STEP: 978750 | > loss_text_ce: 0.03362410143017769 (0.03281086096304771) | > loss_mel_ce: 2.5538651943206787 (nan) | > loss: 0.010267815552651882 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.1367416802918313) | > loader_time: 0.0046 (0.004679865377970676)  --> TIME: 2025-02-25 22:51:00 -- STEP: 18412/36938 -- GLOBAL_STEP: 978800 | > loss_text_ce: 0.02465640753507614 (0.03280892661764998) | > loss_mel_ce: 0.301338255405426 (nan) | > loss: 0.001293629757128656 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13673757325818253) | > loader_time: 0.0045 (0.004679851808574924)  --> TIME: 2025-02-25 22:51:09 -- STEP: 18462/36938 -- GLOBAL_STEP: 978850 | > loss_text_ce: 0.03155262768268585 (0.032804118224596696) | > loss_mel_ce: 3.0455517768859863 (nan) | > loss: 0.012210732325911522 (nan) | > current_lr: 5e-06 | > step_time: 0.1605 (0.1367356319647549) | > loader_time: 0.0046 (0.004679816384677617)  --> TIME: 2025-02-25 22:51:18 -- STEP: 18512/36938 -- GLOBAL_STEP: 978900 | > loss_text_ce: 0.034820932894945145 (0.03280248357287554) | > loss_mel_ce: 2.9839508533477783 (nan) | > loss: 0.011979253962635994 (nan) | > current_lr: 5e-06 | > step_time: 0.1691 (0.1367416084714257) | > loader_time: 0.0051 (0.004679911759462267)  --> TIME: 2025-02-25 22:51:28 -- STEP: 18562/36938 -- GLOBAL_STEP: 978950 | > loss_text_ce: 0.040068045258522034 (0.03280623116618901) | > loss_mel_ce: 0.995269775390625 (nan) | > loss: 0.00410848343744874 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1367356967674278) | > loader_time: 0.0046 (0.004680002420296165)  --> TIME: 2025-02-25 22:51:37 -- STEP: 18612/36938 -- GLOBAL_STEP: 979000 | > loss_text_ce: 0.028018280863761902 (0.03280810268149287) | > loss_mel_ce: 2.844503402709961 (nan) | > loss: 0.011398895643651485 (nan) | > current_lr: 5e-06 | > step_time: 0.1761 (0.13674322112052945) | > loader_time: 0.0048 (0.0046801716569621525)  --> TIME: 2025-02-25 22:51:47 -- STEP: 18662/36938 -- GLOBAL_STEP: 979050 | > loss_text_ce: 0.03180208429694176 (0.032807711230836825) | > loss_mel_ce: 3.1961402893066406 (nan) | > loss: 0.012809296138584614 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.13674729840827826) | > loader_time: 0.0046 (0.0046800610439377156)  --> TIME: 2025-02-25 22:51:56 -- STEP: 18712/36938 -- GLOBAL_STEP: 979100 | > loss_text_ce: 0.03323592245578766 (0.03280420096402197) | > loss_mel_ce: 3.4689018726348877 (nan) | > loss: 0.013897373341023922 (nan) | > current_lr: 5e-06 | > step_time: 0.148 (0.13674577710056726) | > loader_time: 0.0047 (0.0046801277081545585)  --> TIME: 2025-02-25 22:52:05 -- STEP: 18762/36938 -- GLOBAL_STEP: 979150 | > loss_text_ce: 0.028965329751372337 (0.032802058200357684) | > loss_mel_ce: 2.62894606590271 (nan) | > loss: 0.010547267273068428 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13674159704933675) | > loader_time: 0.0047 (0.0046801525142805235)  --> TIME: 2025-02-25 22:52:15 -- STEP: 18812/36938 -- GLOBAL_STEP: 979200 | > loss_text_ce: 0.03789162635803223 (0.032797824923589215) | > loss_mel_ce: 2.8983702659606934 (nan) | > loss: 0.011651833541691303 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13674805026446815) | > loader_time: 0.0046 (0.00468034363289781)  --> TIME: 2025-02-25 22:52:23 -- STEP: 18862/36938 -- GLOBAL_STEP: 979250 | > loss_text_ce: 0.0286768339574337 (0.032798485147759265) | > loss_mel_ce: 1.8618640899658203 (nan) | > loss: 0.0075021469965577126 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13672694935170224) | > loader_time: 0.0045 (0.0046804067805644965)  --> TIME: 2025-02-25 22:52:33 -- STEP: 18912/36938 -- GLOBAL_STEP: 979300 | > loss_text_ce: 0.02335524745285511 (0.0327988413753274) | > loss_mel_ce: 1.7021881341934204 (nan) | > loss: 0.00684739463031292 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13673114595074384) | > loader_time: 0.0044 (0.0046804299209323745)  --> TIME: 2025-02-25 22:52:42 -- STEP: 18962/36938 -- GLOBAL_STEP: 979350 | > loss_text_ce: 0.03864889219403267 (0.03279970935052264) | > loss_mel_ce: 1.210102915763855 (nan) | > loss: 0.004955364391207695 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13673890968526398) | > loader_time: 0.0047 (0.004680333025859599)  --> TIME: 2025-02-25 22:52:52 -- STEP: 19012/36938 -- GLOBAL_STEP: 979400 | > loss_text_ce: 0.026780175045132637 (0.03280061632717128) | > loss_mel_ce: 0.1019999161362648 (nan) | > loss: 0.000511032179929316 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13674936233358453) | > loader_time: 0.0046 (0.0046803139647057165)  --> TIME: 2025-02-25 22:53:01 -- STEP: 19062/36938 -- GLOBAL_STEP: 979450 | > loss_text_ce: 0.020354697480797768 (0.03280144887687612) | > loss_mel_ce: 2.10831618309021 (nan) | > loss: 0.008447106927633286 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1367444436355781) | > loader_time: 0.0045 (0.004680211403202063)  --> TIME: 2025-02-25 22:53:10 -- STEP: 19112/36938 -- GLOBAL_STEP: 979500 | > loss_text_ce: 0.029276937246322632 (0.03279899993067203) | > loss_mel_ce: 2.9469521045684814 (nan) | > loss: 0.011810433119535446 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13674703537295163) | > loader_time: 0.0045 (0.004680185774072247)  --> TIME: 2025-02-25 22:53:20 -- STEP: 19162/36938 -- GLOBAL_STEP: 979550 | > loss_text_ce: 0.029127683490514755 (0.032799358206678064) | > loss_mel_ce: 2.828063726425171 (nan) | > loss: 0.011338061653077602 (nan) | > current_lr: 5e-06 | > step_time: 0.1599 (0.13675808823192112) | > loader_time: 0.0046 (0.004680216629686079)  --> TIME: 2025-02-25 22:53:29 -- STEP: 19212/36938 -- GLOBAL_STEP: 979600 | > loss_text_ce: 0.026309356093406677 (0.032799286547613646) | > loss_mel_ce: 0.00712509173899889 (nan) | > loss: 0.0001326763886027038 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.1367484112841234) | > loader_time: 0.0046 (0.004680119354030438)  --> TIME: 2025-02-25 22:53:39 -- STEP: 19262/36938 -- GLOBAL_STEP: 979650 | > loss_text_ce: 0.035260554403066635 (0.03279814686077527) | > loss_mel_ce: 2.997591257095337 (nan) | > loss: 0.0120351267978549 (nan) | > current_lr: 5e-06 | > step_time: 0.1521 (0.13676079234754174) | > loader_time: 0.0047 (0.004680069012008675)  --> TIME: 2025-02-25 22:53:48 -- STEP: 19312/36938 -- GLOBAL_STEP: 979700 | > loss_text_ce: 0.03376288712024689 (0.032800200153244564) | > loss_mel_ce: 3.4638819694519043 (nan) | > loss: 0.013879544101655483 (nan) | > current_lr: 5e-06 | > step_time: 0.1563 (0.13678058843624616) | > loader_time: 0.0047 (0.004680061436627863)  --> TIME: 2025-02-25 22:53:58 -- STEP: 19362/36938 -- GLOBAL_STEP: 979750 | > loss_text_ce: 0.03327900543808937 (0.0328005287362778) | > loss_mel_ce: 2.8713982105255127 (nan) | > loss: 0.011526497080922127 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.136788390648414) | > loader_time: 0.0049 (0.00468002344850002)  --> TIME: 2025-02-25 22:54:07 -- STEP: 19412/36938 -- GLOBAL_STEP: 979800 | > loss_text_ce: 0.04028295353055 (0.03280284648623756) | > loss_mel_ce: 1.9380083084106445 (nan) | > loss: 0.007850362919270992 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13678520542062741) | > loader_time: 0.0048 (0.004679983531276697)  --> TIME: 2025-02-25 22:54:16 -- STEP: 19462/36938 -- GLOBAL_STEP: 979850 | > loss_text_ce: 0.02988317422568798 (0.032799720396071985) | > loss_mel_ce: 2.61867618560791 (nan) | > loss: 0.010510156862437725 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.1367677081686033) | > loader_time: 0.0046 (0.004679971407207339)  --> TIME: 2025-02-25 22:54:25 -- STEP: 19512/36938 -- GLOBAL_STEP: 979900 | > loss_text_ce: 0.03137804567813873 (0.032794600478940596) | > loss_mel_ce: 1.6952766180038452 (nan) | > loss: 0.006851804442703724 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13676623662215198) | > loader_time: 0.0045 (0.004679839622783787)  --> TIME: 2025-02-25 22:54:35 -- STEP: 19562/36938 -- GLOBAL_STEP: 979950 | > loss_text_ce: 0.028423283249139786 (0.032795651255667066) | > loss_mel_ce: 2.921431303024292 (nan) | > loss: 0.011705772951245308 (nan) | > current_lr: 5e-06 | > step_time: 0.1353 (0.13677915740922456) | > loader_time: 0.0046 (0.004679772851657516)  --> TIME: 2025-02-25 22:54:44 -- STEP: 19612/36938 -- GLOBAL_STEP: 980000 | > loss_text_ce: 0.026873674243688583 (0.03279769194108966) | > loss_mel_ce: 3.032792568206787 (nan) | > loss: 0.012141533195972443 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13677876010953632) | > loader_time: 0.0047 (0.004679668929512208) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_980000.pth  --> TIME: 2025-02-25 22:55:00 -- STEP: 19662/36938 -- GLOBAL_STEP: 980050 | > loss_text_ce: 0.027858981862664223 (0.03279607374000935) | > loss_mel_ce: 2.730619430541992 (nan) | > loss: 0.010946343652904034 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.1367801434025385) | > loader_time: 0.0046 (0.004679922508982442)  --> TIME: 2025-02-25 22:55:09 -- STEP: 19712/36938 -- GLOBAL_STEP: 980100 | > loss_text_ce: 0.02345612272620201 (0.0327938030132391) | > loss_mel_ce: 1.219916820526123 (nan) | > loss: 0.00493401987478137 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13677347486524338) | > loader_time: 0.0045 (0.004679873670366689)  --> TIME: 2025-02-25 22:55:18 -- STEP: 19762/36938 -- GLOBAL_STEP: 980150 | > loss_text_ce: 0.026571616530418396 (0.03279238361179121) | > loss_mel_ce: 2.670457124710083 (nan) | > loss: 0.010702495463192463 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13677204983544097) | > loader_time: 0.0045 (0.004679863323338699)  --> TIME: 2025-02-25 22:55:28 -- STEP: 19812/36938 -- GLOBAL_STEP: 980200 | > loss_text_ce: 0.02815917879343033 (0.032792106026912654) | > loss_mel_ce: 1.6241551637649536 (nan) | > loss: 0.006556803360581398 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.1367656014289575) | > loader_time: 0.0046 (0.004679802846552372)  --> TIME: 2025-02-25 22:55:37 -- STEP: 19862/36938 -- GLOBAL_STEP: 980250 | > loss_text_ce: 0.03602055460214615 (0.03279096987531723) | > loss_mel_ce: 3.261965274810791 (nan) | > loss: 0.013087245635688305 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.1367627913324568) | > loader_time: 0.0051 (0.004679783967167656)  --> TIME: 2025-02-25 22:55:46 -- STEP: 19912/36938 -- GLOBAL_STEP: 980300 | > loss_text_ce: 0.03578776866197586 (0.032790032180805284) | > loss_mel_ce: 3.1949303150177 (nan) | > loss: 0.012820310890674591 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13676116435630406) | > loader_time: 0.0045 (0.004679687653453832)  --> TIME: 2025-02-25 22:55:56 -- STEP: 19962/36938 -- GLOBAL_STEP: 980350 | > loss_text_ce: 0.031924035400152206 (0.03278833568878475) | > loss_mel_ce: 3.4665305614471436 (nan) | > loss: 0.013882757164537907 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13677277475186514) | > loader_time: 0.0046 (0.0046797222226885125)  --> TIME: 2025-02-25 22:56:05 -- STEP: 20012/36938 -- GLOBAL_STEP: 980400 | > loss_text_ce: 0.03448096662759781 (0.032793781771387004) | > loss_mel_ce: 2.4784364700317383 (nan) | > loss: 0.00997189525514841 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13677549472980874) | > loader_time: 0.0046 (0.004679701077135101)  --> TIME: 2025-02-25 22:56:15 -- STEP: 20062/36938 -- GLOBAL_STEP: 980450 | > loss_text_ce: 0.031753163784742355 (0.03279504083702706) | > loss_mel_ce: 3.082751750946045 (nan) | > loss: 0.01235914696007967 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.13678277811634834) | > loader_time: 0.0047 (0.004679619844075573)  --> TIME: 2025-02-25 22:56:24 -- STEP: 20112/36938 -- GLOBAL_STEP: 980500 | > loss_text_ce: 0.039294157177209854 (0.03279458287043476) | > loss_mel_ce: 1.660309910774231 (nan) | > loss: 0.006744460668414831 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13679155297192214) | > loader_time: 0.0045 (0.004679590522911034)  --> TIME: 2025-02-25 22:56:34 -- STEP: 20162/36938 -- GLOBAL_STEP: 980550 | > loss_text_ce: 0.019081085920333862 (0.03279149855467932) | > loss_mel_ce: 2.4824509620666504 (nan) | > loss: 0.009926714934408665 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13679616595057295) | > loader_time: 0.0046 (0.004679459922903304)  --> TIME: 2025-02-25 22:56:43 -- STEP: 20212/36938 -- GLOBAL_STEP: 980600 | > loss_text_ce: 0.03196733072400093 (0.03279424623797097) | > loss_mel_ce: 2.7836735248565674 (nan) | > loss: 0.011173179373145103 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13678718396467565) | > loader_time: 0.0047 (0.004679319116825314)  --> TIME: 2025-02-25 22:56:52 -- STEP: 20262/36938 -- GLOBAL_STEP: 980650 | > loss_text_ce: 0.03472217544913292 (0.032793299918418) | > loss_mel_ce: 2.7531354427337646 (nan) | > loss: 0.011062927544116974 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.1367737710940718) | > loader_time: 0.0046 (0.004679175028501154)  --> TIME: 2025-02-25 22:57:01 -- STEP: 20312/36938 -- GLOBAL_STEP: 980700 | > loss_text_ce: 0.024266812950372696 (0.03279375608649346) | > loss_mel_ce: 1.8578119277954102 (nan) | > loss: 0.0074685667641460896 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.136774569362485) | > loader_time: 0.0047 (0.0046791019238745886)  --> TIME: 2025-02-25 22:57:11 -- STEP: 20362/36938 -- GLOBAL_STEP: 980750 | > loss_text_ce: 0.02803667262196541 (0.03279487950037252) | > loss_mel_ce: 2.5421836376190186 (nan) | > loss: 0.010199286974966526 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.13677916916938268) | > loader_time: 0.0045 (0.004679238792721443)  --> TIME: 2025-02-25 22:57:20 -- STEP: 20412/36938 -- GLOBAL_STEP: 980800 | > loss_text_ce: 0.035206034779548645 (0.03279563658401726) | > loss_mel_ce: 3.260341167449951 (nan) | > loss: 0.01307756919413805 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.13677443598103142) | > loader_time: 0.0048 (0.004679169931436046)  --> TIME: 2025-02-25 22:57:29 -- STEP: 20462/36938 -- GLOBAL_STEP: 980850 | > loss_text_ce: 0.03022068925201893 (0.03279529156490319) | > loss_mel_ce: 2.4005727767944336 (nan) | > loss: 0.009646006859838963 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13677649943782755) | > loader_time: 0.0045 (0.004678993056846064)  --> TIME: 2025-02-25 22:57:38 -- STEP: 20512/36938 -- GLOBAL_STEP: 980900 | > loss_text_ce: 0.04165281355381012 (0.032802049452942066) | > loss_mel_ce: 2.651611566543579 (nan) | > loss: 0.010687557980418205 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.1367721663389869) | > loader_time: 0.0045 (0.004678986664402326)  --> TIME: 2025-02-25 22:57:48 -- STEP: 20562/36938 -- GLOBAL_STEP: 980950 | > loss_text_ce: 0.04529232531785965 (0.032805880152427185) | > loss_mel_ce: 3.59375 (nan) | > loss: 0.014440645463764668 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13678218551721538) | > loader_time: 0.0048 (0.004679288408216163)  --> TIME: 2025-02-25 22:57:58 -- STEP: 20612/36938 -- GLOBAL_STEP: 981000 | > loss_text_ce: 0.036256883293390274 (0.032804902350221356) | > loss_mel_ce: 2.4595255851745605 (nan) | > loss: 0.00990389846265316 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13679405073506146) | > loader_time: 0.0047 (0.004679400100075071)  --> TIME: 2025-02-25 22:58:07 -- STEP: 20662/36938 -- GLOBAL_STEP: 981050 | > loss_text_ce: 0.030390679836273193 (0.032801192739321985) | > loss_mel_ce: 2.9948959350585938 (nan) | > loss: 0.012005106545984745 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.1367886098630553) | > loader_time: 0.0046 (0.0046792610168641755)  --> TIME: 2025-02-25 22:58:16 -- STEP: 20712/36938 -- GLOBAL_STEP: 981100 | > loss_text_ce: 0.03149135783314705 (0.03280020067564511) | > loss_mel_ce: 2.9355695247650146 (nan) | > loss: 0.011774051934480667 (nan) | > current_lr: 5e-06 | > step_time: 0.1701 (0.13679315965347907) | > loader_time: 0.0046 (0.004679334755337998)  --> TIME: 2025-02-25 22:58:26 -- STEP: 20762/36938 -- GLOBAL_STEP: 981150 | > loss_text_ce: 0.03148268535733223 (0.03280097661533093) | > loss_mel_ce: 2.5853183269500732 (nan) | > loss: 0.010384131222963333 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13679826083916927) | > loader_time: 0.0047 (0.004679322254220239)  --> TIME: 2025-02-25 22:58:35 -- STEP: 20812/36938 -- GLOBAL_STEP: 981200 | > loss_text_ce: 0.03261673450469971 (0.03280261687973956) | > loss_mel_ce: 3.0942227840423584 (nan) | > loss: 0.012408094480633736 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.13680897964414943) | > loader_time: 0.0049 (0.004679212541780003)  --> TIME: 2025-02-25 22:58:44 -- STEP: 20862/36938 -- GLOBAL_STEP: 981250 | > loss_text_ce: 0.029639700427651405 (0.03280324335742602) | > loss_mel_ce: 2.138887643814087 (nan) | > loss: 0.008605267852544785 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13678293210549824) | > loader_time: 0.0044 (0.004678914604336964)  --> TIME: 2025-02-25 22:58:53 -- STEP: 20912/36938 -- GLOBAL_STEP: 981300 | > loss_text_ce: 0.0380026139318943 (0.03280261642298344) | > loss_mel_ce: 1.6751047372817993 (nan) | > loss: 0.006798045244067907 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13678010397497428) | > loader_time: 0.0045 (0.0046787582788197545)  --> TIME: 2025-02-25 22:59:03 -- STEP: 20962/36938 -- GLOBAL_STEP: 981350 | > loss_text_ce: 0.03419867530465126 (0.032802151726407186) | > loss_mel_ce: 3.088273525238037 (nan) | > loss: 0.012390763498842716 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.13677999317253714) | > loader_time: 0.0048 (0.004678593292887832)  --> TIME: 2025-02-25 22:59:12 -- STEP: 21012/36938 -- GLOBAL_STEP: 981400 | > loss_text_ce: 0.03546091541647911 (0.03280247461498235) | > loss_mel_ce: 2.6520071029663086 (nan) | > loss: 0.010664556175470352 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13677732061255107) | > loader_time: 0.0046 (0.004678424803071005)  --> TIME: 2025-02-25 22:59:21 -- STEP: 21062/36938 -- GLOBAL_STEP: 981450 | > loss_text_ce: 0.01591961272060871 (0.0328019947570067) | > loss_mel_ce: 2.078606605529785 (nan) | > loss: 0.008311612531542778 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13677110470277515) | > loader_time: 0.0045 (0.004678358663555895)  --> TIME: 2025-02-25 22:59:31 -- STEP: 21112/36938 -- GLOBAL_STEP: 981500 | > loss_text_ce: 0.033303823322057724 (0.03279947628289694) | > loss_mel_ce: 2.7490923404693604 (nan) | > loss: 0.011041254736483097 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13679273855103552) | > loader_time: 0.0046 (0.004678349697757023)  --> TIME: 2025-02-25 22:59:40 -- STEP: 21162/36938 -- GLOBAL_STEP: 981550 | > loss_text_ce: 0.03208761289715767 (0.03279998169476253) | > loss_mel_ce: 3.1665737628936768 (nan) | > loss: 0.012693101540207863 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13679955378787473) | > loader_time: 0.0045 (0.004678391698244771)  --> TIME: 2025-02-25 22:59:50 -- STEP: 21212/36938 -- GLOBAL_STEP: 981600 | > loss_text_ce: 0.02697344683110714 (0.03279657887141041) | > loss_mel_ce: 2.465453863143921 (nan) | > loss: 0.009890585206449032 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.1368084715466092) | > loader_time: 0.005 (0.004678538401756749)  --> TIME: 2025-02-25 22:59:59 -- STEP: 21262/36938 -- GLOBAL_STEP: 981650 | > loss_text_ce: 0.03197512403130531 (0.03279930050482645) | > loss_mel_ce: 2.73197340965271 (nan) | > loss: 0.010968049988150597 (nan) | > current_lr: 5e-06 | > step_time: 0.1687 (0.13680874147505856) | > loader_time: 0.0048 (0.004678795707942342)  --> TIME: 2025-02-25 23:00:09 -- STEP: 21312/36938 -- GLOBAL_STEP: 981700 | > loss_text_ce: 0.03177347406744957 (0.032798788669503726) | > loss_mel_ce: 2.6614460945129395 (nan) | > loss: 0.010687380097806454 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13681149838475587) | > loader_time: 0.0046 (0.004678952868457324)  --> TIME: 2025-02-25 23:00:18 -- STEP: 21362/36938 -- GLOBAL_STEP: 981750 | > loss_text_ce: 0.030517611652612686 (0.032798480013256866) | > loss_mel_ce: 2.4405980110168457 (nan) | > loss: 0.009806014597415924 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13681079027481186) | > loader_time: 0.0045 (0.004679068969035136)  --> TIME: 2025-02-25 23:00:27 -- STEP: 21412/36938 -- GLOBAL_STEP: 981800 | > loss_text_ce: 0.03344394266605377 (0.03279687607171652) | > loss_mel_ce: 3.4127631187438965 (nan) | > loss: 0.013675425201654434 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13681180095663814) | > loader_time: 0.0047 (0.004678969057657877)  --> TIME: 2025-02-25 23:00:36 -- STEP: 21462/36938 -- GLOBAL_STEP: 981850 | > loss_text_ce: 0.032992884516716 (0.03279779323409432) | > loss_mel_ce: 2.2405824661254883 (nan) | > loss: 0.009022125042974949 (nan) | > current_lr: 5e-06 | > step_time: 0.1867 (0.1368023960513278) | > loader_time: 0.0072 (0.004679324553381894)  --> TIME: 2025-02-25 23:00:46 -- STEP: 21512/36938 -- GLOBAL_STEP: 981900 | > loss_text_ce: 0.040747955441474915 (0.03279873779093426) | > loss_mel_ce: 2.9755654335021973 (nan) | > loss: 0.011969498358666897 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13680358319178013) | > loader_time: 0.0047 (0.004679504126676097)  --> TIME: 2025-02-25 23:00:55 -- STEP: 21562/36938 -- GLOBAL_STEP: 981950 | > loss_text_ce: 0.029956774786114693 (0.03279971126943834) | > loss_mel_ce: 3.06227445602417 (nan) | > loss: 0.012270759791135788 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13680649790026655) | > loader_time: 0.0047 (0.004679604006127637)  --> TIME: 2025-02-25 23:01:04 -- STEP: 21612/36938 -- GLOBAL_STEP: 982000 | > loss_text_ce: 0.024166323244571686 (0.03280009412423547) | > loss_mel_ce: 2.8139185905456543 (nan) | > loss: 0.011262242682278156 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.1368086076479994) | > loader_time: 0.0045 (0.004679515618605468)  --> TIME: 2025-02-25 23:01:13 -- STEP: 21662/36938 -- GLOBAL_STEP: 982050 | > loss_text_ce: 0.030565056949853897 (0.03280161533271006) | > loss_mel_ce: 2.1066033840179443 (nan) | > loss: 0.008480827324092388 (nan) | > current_lr: 5e-06 | > step_time: 0.19 (0.13679986343017594) | > loader_time: 0.005 (0.004679337882696243)  --> TIME: 2025-02-25 23:01:23 -- STEP: 21712/36938 -- GLOBAL_STEP: 982100 | > loss_text_ce: 0.0365837961435318 (0.03280068677335329) | > loss_mel_ce: 1.3751057386398315 (nan) | > loss: 0.005601943004876375 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13679611502804212) | > loader_time: 0.0046 (0.004679233264027055)  --> TIME: 2025-02-25 23:01:32 -- STEP: 21762/36938 -- GLOBAL_STEP: 982150 | > loss_text_ce: 0.027108749374747276 (0.03279890270045902) | > loss_mel_ce: 1.7070673704147339 (nan) | > loss: 0.006881651934236288 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.13678966506572107) | > loader_time: 0.0047 (0.004679188210345649)  --> TIME: 2025-02-25 23:01:41 -- STEP: 21812/36938 -- GLOBAL_STEP: 982200 | > loss_text_ce: 0.030899249017238617 (0.032799879877795396) | > loss_mel_ce: 3.089900255203247 (nan) | > loss: 0.012384125962853432 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13678503438745776) | > loader_time: 0.0046 (0.0046790080094105835)  --> TIME: 2025-02-25 23:01:51 -- STEP: 21862/36938 -- GLOBAL_STEP: 982250 | > loss_text_ce: 0.03856489434838295 (0.032797857715162276) | > loss_mel_ce: 0.2016245573759079 (nan) | > loss: 0.000953132810536772 (nan) | > current_lr: 5e-06 | > step_time: 0.1569 (0.13679724959526354) | > loader_time: 0.0046 (0.004678910064976459)  --> TIME: 2025-02-25 23:02:00 -- STEP: 21912/36938 -- GLOBAL_STEP: 982300 | > loss_text_ce: 0.027025431394577026 (0.03279402071059932) | > loss_mel_ce: 2.9819703102111816 (nan) | > loss: 0.011940459720790386 (nan) | > current_lr: 5e-06 | > step_time: 0.1858 (0.13680192399477717) | > loader_time: 0.0048 (0.00467923709013551)  --> TIME: 2025-02-25 23:02:09 -- STEP: 21962/36938 -- GLOBAL_STEP: 982350 | > loss_text_ce: 0.036965806037187576 (0.03279611935469102) | > loss_mel_ce: 0.858154296875 (nan) | > loss: 0.0035520640667527914 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1367989658451778) | > loader_time: 0.0047 (0.004679200482296833)  --> TIME: 2025-02-25 23:02:19 -- STEP: 22012/36938 -- GLOBAL_STEP: 982400 | > loss_text_ce: 0.030804023146629333 (0.03279787605200529) | > loss_mel_ce: 0.0264129638671875 (nan) | > loss: 0.00022705155424773693 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13679351034152412) | > loader_time: 0.0047 (0.004679269515967657)  --> TIME: 2025-02-25 23:02:28 -- STEP: 22062/36938 -- GLOBAL_STEP: 982450 | > loss_text_ce: 0.029772618785500526 (0.03279925491898913) | > loss_mel_ce: 2.704293966293335 (nan) | > loss: 0.01084947120398283 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.136781628961178) | > loader_time: 0.0047 (0.004679187050259969)  --> TIME: 2025-02-25 23:02:37 -- STEP: 22112/36938 -- GLOBAL_STEP: 982500 | > loss_text_ce: 0.03272440657019615 (0.032799136174428915) | > loss_mel_ce: 2.727095365524292 (nan) | > loss: 0.010951666161417961 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13678263596834686) | > loader_time: 0.0048 (0.004679293734292405)  --> TIME: 2025-02-25 23:02:46 -- STEP: 22162/36938 -- GLOBAL_STEP: 982550 | > loss_text_ce: 0.03191499784588814 (0.032799427006983656) | > loss_mel_ce: 2.7585809230804443 (nan) | > loss: 0.011073396541178226 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13678229050928595) | > loader_time: 0.0045 (0.004679283998081622)  --> TIME: 2025-02-25 23:02:56 -- STEP: 22212/36938 -- GLOBAL_STEP: 982600 | > loss_text_ce: 0.02470049262046814 (0.03279767412276034) | > loss_mel_ce: 2.7143993377685547 (nan) | > loss: 0.010869444347918034 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13678244316060673) | > loader_time: 0.0047 (0.004679259342823562)  --> TIME: 2025-02-25 23:03:05 -- STEP: 22262/36938 -- GLOBAL_STEP: 982650 | > loss_text_ce: 0.04635230079293251 (0.03280007249987185) | > loss_mel_ce: 3.3104631900787354 (nan) | > loss: 0.013320697471499443 (nan) | > current_lr: 5e-06 | > step_time: 0.1515 (0.13678994792221752) | > loader_time: 0.0046 (0.004679195879390554)  --> TIME: 2025-02-25 23:03:15 -- STEP: 22312/36938 -- GLOBAL_STEP: 982700 | > loss_text_ce: 0.02267240546643734 (0.03279948217110409) | > loss_mel_ce: 2.9816653728485107 (nan) | > loss: 0.011921975761651993 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.1367935459645442) | > loader_time: 0.0046 (0.0046790531135537305)  --> TIME: 2025-02-25 23:03:24 -- STEP: 22362/36938 -- GLOBAL_STEP: 982750 | > loss_text_ce: 0.03449941426515579 (0.03279735402269641) | > loss_mel_ce: 3.302386522293091 (nan) | > loss: 0.013241611421108246 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1367876322748593) | > loader_time: 0.0045 (0.004678827024681102)  --> TIME: 2025-02-25 23:03:33 -- STEP: 22412/36938 -- GLOBAL_STEP: 982800 | > loss_text_ce: 0.03629770129919052 (0.032794423777217335) | > loss_mel_ce: 3.343082904815674 (nan) | > loss: 0.013410241343080997 (nan) | > current_lr: 5e-06 | > step_time: 0.1875 (0.1367861583427356) | > loader_time: 0.0048 (0.004678674357375281)  --> TIME: 2025-02-25 23:03:42 -- STEP: 22462/36938 -- GLOBAL_STEP: 982850 | > loss_text_ce: 0.028851622715592384 (0.032796386233089274) | > loss_mel_ce: 3.5390625 (nan) | > loss: 0.014158389531075954 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.13678297944723308) | > loader_time: 0.0047 (0.00467850347627103)  --> TIME: 2025-02-25 23:03:51 -- STEP: 22512/36938 -- GLOBAL_STEP: 982900 | > loss_text_ce: 0.04065743461251259 (0.03279593082128463) | > loss_mel_ce: 2.8771002292633057 (nan) | > loss: 0.01157840434461832 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13677016516992677) | > loader_time: 0.0047 (0.004678361186679587)  --> TIME: 2025-02-25 23:04:01 -- STEP: 22562/36938 -- GLOBAL_STEP: 982950 | > loss_text_ce: 0.03917151689529419 (0.03279490594083165) | > loss_mel_ce: 3.181015968322754 (nan) | > loss: 0.012778522446751595 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13676758119373453) | > loader_time: 0.0048 (0.004678212764700434)  --> TIME: 2025-02-25 23:04:10 -- STEP: 22612/36938 -- GLOBAL_STEP: 983000 | > loss_text_ce: 0.027471857145428658 (0.03279562210745693) | > loss_mel_ce: 2.8312015533447266 (nan) | > loss: 0.011343942023813725 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13677166084178025) | > loader_time: 0.0045 (0.004678047907451565)  --> TIME: 2025-02-25 23:04:20 -- STEP: 22662/36938 -- GLOBAL_STEP: 983050 | > loss_text_ce: 0.04413345828652382 (0.032797536695146276) | > loss_mel_ce: 2.9256532192230225 (nan) | > loss: 0.011784868314862251 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13678289578121242) | > loader_time: 0.0046 (0.004678049688042134)  --> TIME: 2025-02-25 23:04:29 -- STEP: 22712/36938 -- GLOBAL_STEP: 983100 | > loss_text_ce: 0.03326723724603653 (0.03279869158173415) | > loss_mel_ce: 2.5042576789855957 (nan) | > loss: 0.010069544427096844 (nan) | > current_lr: 5e-06 | > step_time: 0.1487 (0.13678652716800077) | > loader_time: 0.0047 (0.0046779323249855915)  --> TIME: 2025-02-25 23:04:38 -- STEP: 22762/36938 -- GLOBAL_STEP: 983150 | > loss_text_ce: 0.033563289791345596 (0.032797731793871805) | > loss_mel_ce: 2.7999634742736816 (nan) | > loss: 0.011244154535233974 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.136778299779804) | > loader_time: 0.0045 (0.004677720568886274)  --> TIME: 2025-02-25 23:04:48 -- STEP: 22812/36938 -- GLOBAL_STEP: 983200 | > loss_text_ce: 0.032956112176179886 (0.032796467426148786) | > loss_mel_ce: 2.225125312805176 (nan) | > loss: 0.008960640989243984 (nan) | > current_lr: 5e-06 | > step_time: 0.156 (0.1367834188134232) | > loader_time: 0.0047 (0.004677545537284733)  --> TIME: 2025-02-25 23:04:57 -- STEP: 22862/36938 -- GLOBAL_STEP: 983250 | > loss_text_ce: 0.035875674337148666 (0.032794780295352544) | > loss_mel_ce: 2.3428919315338135 (nan) | > loss: 0.009439554065465927 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13677981462746613) | > loader_time: 0.0045 (0.004677344657504833)  --> TIME: 2025-02-25 23:05:07 -- STEP: 22912/36938 -- GLOBAL_STEP: 983300 | > loss_text_ce: 0.0364694707095623 (0.03279625599310098) | > loss_mel_ce: 2.2436890602111816 (nan) | > loss: 0.009048248641192913 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13678144171376308) | > loader_time: 0.0045 (0.004677241668105124)  --> TIME: 2025-02-25 23:05:16 -- STEP: 22962/36938 -- GLOBAL_STEP: 983350 | > loss_text_ce: 0.03645847365260124 (0.032795293571421066) | > loss_mel_ce: 2.929858922958374 (nan) | > loss: 0.011771101504564285 (nan) | > current_lr: 5e-06 | > step_time: 0.1897 (0.13677846863132717) | > loader_time: 0.0049 (0.004677081427895202)  --> TIME: 2025-02-25 23:05:25 -- STEP: 23012/36938 -- GLOBAL_STEP: 983400 | > loss_text_ce: 0.035501860082149506 (0.032795651418121415) | > loss_mel_ce: 3.137063980102539 (nan) | > loss: 0.012589547783136368 (nan) | > current_lr: 5e-06 | > step_time: 0.1909 (0.1367750769487879) | > loader_time: 0.005 (0.004676949971658378)  --> TIME: 2025-02-25 23:05:34 -- STEP: 23062/36938 -- GLOBAL_STEP: 983450 | > loss_text_ce: 0.044688981026411057 (0.03279462061038486) | > loss_mel_ce: 2.843022108078003 (nan) | > loss: 0.011459171772003174 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13677883917352424) | > loader_time: 0.0046 (0.004676816986787965)  --> TIME: 2025-02-25 23:05:44 -- STEP: 23112/36938 -- GLOBAL_STEP: 983500 | > loss_text_ce: 0.03123912774026394 (0.032796196022564755) | > loss_mel_ce: 1.0185004472732544 (nan) | > loss: 0.0041656335815787315 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13678098752367568) | > loader_time: 0.0045 (0.004676697461734633)  --> TIME: 2025-02-25 23:05:53 -- STEP: 23162/36938 -- GLOBAL_STEP: 983550 | > loss_text_ce: 0.0294248815625906 (0.032792819238992924) | > loss_mel_ce: 2.6468095779418945 (nan) | > loss: 0.010619978420436382 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1367804160051005) | > loader_time: 0.0045 (0.0046765804496641875)  --> TIME: 2025-02-25 23:06:03 -- STEP: 23212/36938 -- GLOBAL_STEP: 983600 | > loss_text_ce: 0.026207031682133675 (0.03279502822270615) | > loss_mel_ce: 2.4398303031921387 (nan) | > loss: 0.009785862639546394 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.13678052595683654) | > loader_time: 0.0046 (0.004676424396997394)  --> TIME: 2025-02-25 23:06:11 -- STEP: 23262/36938 -- GLOBAL_STEP: 983650 | > loss_text_ce: 0.038655124604701996 (0.032794411503191324) | > loss_mel_ce: 2.5663652420043945 (nan) | > loss: 0.010337382555007935 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.1367674267370955) | > loader_time: 0.0045 (0.004676229299247315)  --> TIME: 2025-02-25 23:06:21 -- STEP: 23312/36938 -- GLOBAL_STEP: 983700 | > loss_text_ce: 0.029246265068650246 (0.03279802773058169) | > loss_mel_ce: 2.663540840148926 (nan) | > loss: 0.010685663670301437 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.1367622488008937) | > loader_time: 0.0047 (0.004676058980480966)  --> TIME: 2025-02-25 23:06:30 -- STEP: 23362/36938 -- GLOBAL_STEP: 983750 | > loss_text_ce: 0.044123467057943344 (0.03280125667099709) | > loss_mel_ce: 2.143501043319702 (nan) | > loss: 0.008681049570441246 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1367720502786948) | > loader_time: 0.0046 (0.004675921292840158)  --> TIME: 2025-02-25 23:06:40 -- STEP: 23412/36938 -- GLOBAL_STEP: 983800 | > loss_text_ce: 0.03368629142642021 (0.032802546744645514) | > loss_mel_ce: 2.559877634048462 (nan) | > loss: 0.010291920974850655 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13677855552658463) | > loader_time: 0.0067 (0.0046759137796903456)  --> TIME: 2025-02-25 23:06:49 -- STEP: 23462/36938 -- GLOBAL_STEP: 983850 | > loss_text_ce: 0.03639082983136177 (0.03280290554523463) | > loss_mel_ce: 3.343200206756592 (nan) | > loss: 0.013411075808107853 (nan) | > current_lr: 5e-06 | > step_time: 0.1788 (0.13677872212969144) | > loader_time: 0.005 (0.004675786428749885)  --> TIME: 2025-02-25 23:06:58 -- STEP: 23512/36938 -- GLOBAL_STEP: 983900 | > loss_text_ce: 0.03451323136687279 (0.03280498515306727) | > loss_mel_ce: 2.2752902507781982 (nan) | > loss: 0.009165887720882893 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.136774544510643) | > loader_time: 0.0046 (0.004675580940915024)  --> TIME: 2025-02-25 23:07:08 -- STEP: 23562/36938 -- GLOBAL_STEP: 983950 | > loss_text_ce: 0.02472693659365177 (0.03280381468345156) | > loss_mel_ce: 3.0604546070098877 (nan) | > loss: 0.012242784723639488 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13677557752671066) | > loader_time: 0.0045 (0.004675449625602462)  --> TIME: 2025-02-25 23:07:18 -- STEP: 23612/36938 -- GLOBAL_STEP: 984000 | > loss_text_ce: 0.03656935319304466 (0.032803496612471186) | > loss_mel_ce: 2.7106661796569824 (nan) | > loss: 0.01090172864496708 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.1367945433188038) | > loader_time: 0.0045 (0.004675387164889278)  --> TIME: 2025-02-25 23:07:27 -- STEP: 23662/36938 -- GLOBAL_STEP: 984050 | > loss_text_ce: 0.02940710075199604 (0.032803711290695925) | > loss_mel_ce: 1.84988272190094 (nan) | > loss: 0.007457499857991934 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13679309969832745) | > loader_time: 0.0046 (0.004675211401423418)  --> TIME: 2025-02-25 23:07:36 -- STEP: 23712/36938 -- GLOBAL_STEP: 984100 | > loss_text_ce: 0.037644002586603165 (0.032802646597883414) | > loss_mel_ce: 1.5464197397232056 (nan) | > loss: 0.006285967770963907 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.1367975822605918) | > loader_time: 0.0045 (0.004675105274447544)  --> TIME: 2025-02-25 23:07:46 -- STEP: 23762/36938 -- GLOBAL_STEP: 984150 | > loss_text_ce: 0.03131033480167389 (0.03280362187285156) | > loss_mel_ce: 3.1450893878936768 (nan) | > loss: 0.012604761868715286 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13680009741181642) | > loader_time: 0.0045 (0.004674915221491782)  --> TIME: 2025-02-25 23:07:56 -- STEP: 23812/36938 -- GLOBAL_STEP: 984200 | > loss_text_ce: 0.03367546200752258 (0.032804076427253644) | > loss_mel_ce: 2.6578314304351807 (nan) | > loss: 0.01068058330565691 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13681066696050803) | > loader_time: 0.0046 (0.0046748054361976125)  --> TIME: 2025-02-25 23:08:05 -- STEP: 23862/36938 -- GLOBAL_STEP: 984250 | > loss_text_ce: 0.04115712642669678 (0.03280146821014757) | > loss_mel_ce: 1.6525920629501343 (nan) | > loss: 0.006721227429807186 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13681340136658826) | > loader_time: 0.0046 (0.004674676667413984)  --> TIME: 2025-02-25 23:08:14 -- STEP: 23912/36938 -- GLOBAL_STEP: 984300 | > loss_text_ce: 0.03575390577316284 (0.032799756242440234) | > loss_mel_ce: 2.944800615310669 (nan) | > loss: 0.011827598325908184 (nan) | > current_lr: 5e-06 | > step_time: 0.1634 (0.1368163932362872) | > loader_time: 0.0048 (0.0046745501520961494)  --> TIME: 2025-02-25 23:08:24 -- STEP: 23962/36938 -- GLOBAL_STEP: 984350 | > loss_text_ce: 0.029561907052993774 (0.03279777308594061) | > loss_mel_ce: 1.8000589609146118 (nan) | > loss: 0.007260400801897049 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1368152547207007) | > loader_time: 0.0045 (0.004674379171488149)  --> TIME: 2025-02-25 23:08:33 -- STEP: 24012/36938 -- GLOBAL_STEP: 984400 | > loss_text_ce: 0.0381869301199913 (0.03279711777977634) | > loss_mel_ce: 4.5 (nan) | > loss: 0.018008679151535034 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13681506627046566) | > loader_time: 0.0045 (0.0046743375449186195)  --> TIME: 2025-02-25 23:08:42 -- STEP: 24062/36938 -- GLOBAL_STEP: 984450 | > loss_text_ce: 0.04296250641345978 (0.032795391873413224) | > loss_mel_ce: 2.2124485969543457 (nan) | > loss: 0.00895004440099001 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13681111302263763) | > loader_time: 0.0045 (0.00467421893377513)  --> TIME: 2025-02-25 23:08:52 -- STEP: 24112/36938 -- GLOBAL_STEP: 984500 | > loss_text_ce: 0.044434040784835815 (0.03279825726891079) | > loss_mel_ce: 3.190122127532959 (nan) | > loss: 0.012835540808737278 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13681744787577677) | > loader_time: 0.0046 (0.004674113866661116)  --> TIME: 2025-02-25 23:09:01 -- STEP: 24162/36938 -- GLOBAL_STEP: 984550 | > loss_text_ce: 0.04246261343359947 (0.03280139880893315) | > loss_mel_ce: 0.06352560967206955 (nan) | > loss: 0.00042058818507939577 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13681033155734917) | > loader_time: 0.0046 (0.00467394774998767)  --> TIME: 2025-02-25 23:09:10 -- STEP: 24212/36938 -- GLOBAL_STEP: 984600 | > loss_text_ce: 0.023351140320301056 (0.03279900219632706) | > loss_mel_ce: 3.0521867275238037 (nan) | > loss: 0.012204516679048538 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13681133181797236) | > loader_time: 0.0046 (0.00467380254540033)  --> TIME: 2025-02-25 23:09:20 -- STEP: 24262/36938 -- GLOBAL_STEP: 984650 | > loss_text_ce: 0.02472686767578125 (0.03279930798650237) | > loss_mel_ce: 1.357405185699463 (nan) | > loss: 0.005484651308506727 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13681284926924597) | > loader_time: 0.0046 (0.004673639002994876)  --> TIME: 2025-02-25 23:09:29 -- STEP: 24312/36938 -- GLOBAL_STEP: 984700 | > loss_text_ce: 0.0316702276468277 (0.03280067567204217) | > loss_mel_ce: 2.939962863922119 (nan) | > loss: 0.011792195960879326 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.13681475306742014) | > loader_time: 0.0046 (0.004673528559469168)  --> TIME: 2025-02-25 23:09:38 -- STEP: 24362/36938 -- GLOBAL_STEP: 984750 | > loss_text_ce: 0.040140774101018906 (0.032801312592081376) | > loss_mel_ce: 3.361069679260254 (nan) | > loss: 0.013496868312358856 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13681120575379715) | > loader_time: 0.005 (0.004673313207887051)  --> TIME: 2025-02-25 23:09:48 -- STEP: 24412/36938 -- GLOBAL_STEP: 984800 | > loss_text_ce: 0.03794024512171745 (0.03280160759393907) | > loss_mel_ce: 2.109407424926758 (nan) | > loss: 0.008521221578121185 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.1368148093162642) | > loader_time: 0.0046 (0.00467315010022046)  --> TIME: 2025-02-25 23:09:57 -- STEP: 24462/36938 -- GLOBAL_STEP: 984850 | > loss_text_ce: 0.03262549638748169 (0.03280104120139288) | > loss_mel_ce: 3.2560524940490723 (nan) | > loss: 0.01305031031370163 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13682034755219127) | > loader_time: 0.0045 (0.004672981021975597)  --> TIME: 2025-02-25 23:10:06 -- STEP: 24512/36938 -- GLOBAL_STEP: 984900 | > loss_text_ce: 0.03829481452703476 (0.03280134812497715) | > loss_mel_ce: 0.4773925840854645 (nan) | > loss: 0.002046378795057535 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13681377116444962) | > loader_time: 0.0046 (0.0046728685031360425)  --> TIME: 2025-02-25 23:10:16 -- STEP: 24562/36938 -- GLOBAL_STEP: 984950 | > loss_text_ce: 0.033314406871795654 (0.03280100917329718) | > loss_mel_ce: 2.272519588470459 (nan) | > loss: 0.009150135330855846 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13681030816048104) | > loader_time: 0.0046 (0.004672748201319136)  --> TIME: 2025-02-25 23:10:25 -- STEP: 24612/36938 -- GLOBAL_STEP: 985000 | > loss_text_ce: 0.02725106105208397 (0.03280243239649003) | > loss_mel_ce: 3.0272247791290283 (nan) | > loss: 0.012120936065912247 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.13680902157840513) | > loader_time: 0.0047 (0.004672643064231995)  --> TIME: 2025-02-25 23:10:34 -- STEP: 24662/36938 -- GLOBAL_STEP: 985050 | > loss_text_ce: 0.03490007668733597 (0.032802403358890946) | > loss_mel_ce: 3.1515724658966064 (nan) | > loss: 0.012644733302295208 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.13680470215995) | > loader_time: 0.0047 (0.004672557291985708)  --> TIME: 2025-02-25 23:10:44 -- STEP: 24712/36938 -- GLOBAL_STEP: 985100 | > loss_text_ce: 0.035249631851911545 (0.032803100802341265) | > loss_mel_ce: 2.7057278156280518 (nan) | > loss: 0.010876895859837532 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13680194696974204) | > loader_time: 0.0048 (0.004672408074999202)  --> TIME: 2025-02-25 23:10:53 -- STEP: 24762/36938 -- GLOBAL_STEP: 985150 | > loss_text_ce: 0.028613947331905365 (0.03280229093464041) | > loss_mel_ce: 3.0565145015716553 (nan) | > loss: 0.012242574244737625 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.13680128988238852) | > loader_time: 0.0048 (0.004672268723143614)  --> TIME: 2025-02-25 23:11:02 -- STEP: 24812/36938 -- GLOBAL_STEP: 985200 | > loss_text_ce: 0.03633631020784378 (0.03280193305825226) | > loss_mel_ce: 3.110327959060669 (nan) | > loss: 0.01248676422983408 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13680146993715334) | > loader_time: 0.0048 (0.004672154186042144)  --> TIME: 2025-02-25 23:11:11 -- STEP: 24862/36938 -- GLOBAL_STEP: 985250 | > loss_text_ce: 0.036827314645051956 (0.032801283355021464) | > loss_mel_ce: 2.8743629455566406 (nan) | > loss: 0.011552343145012856 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13679278760911248) | > loader_time: 0.0045 (0.004671974554592004)  --> TIME: 2025-02-25 23:11:21 -- STEP: 24912/36938 -- GLOBAL_STEP: 985300 | > loss_text_ce: 0.03150492534041405 (0.032800173093986376) | > loss_mel_ce: 3.36033034324646 (nan) | > loss: 0.013459664769470692 (nan) | > current_lr: 5e-06 | > step_time: 0.1632 (0.13679689097151962) | > loader_time: 0.0046 (0.004671946866587329)  --> TIME: 2025-02-25 23:11:30 -- STEP: 24962/36938 -- GLOBAL_STEP: 985350 | > loss_text_ce: 0.03244214132428169 (0.03280084986238463) | > loss_mel_ce: 2.622288227081299 (nan) | > loss: 0.010534645058214664 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.1367856752279348) | > loader_time: 0.0045 (0.004671863491036041)  --> TIME: 2025-02-25 23:11:39 -- STEP: 25012/36938 -- GLOBAL_STEP: 985400 | > loss_text_ce: 0.0374484583735466 (0.032799638490382026) | > loss_mel_ce: 2.6617236137390137 (nan) | > loss: 0.010711000300943851 (nan) | > current_lr: 5e-06 | > step_time: 0.1818 (0.13678784768295182) | > loader_time: 0.0048 (0.0046718409018346495)  --> TIME: 2025-02-25 23:11:49 -- STEP: 25062/36938 -- GLOBAL_STEP: 985450 | > loss_text_ce: 0.03602808713912964 (0.03280010453922022) | > loss_mel_ce: 2.67297625541687 (nan) | > loss: 0.010750018060207367 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13679308581548347) | > loader_time: 0.0048 (0.004671720150947645)  --> TIME: 2025-02-25 23:11:58 -- STEP: 25112/36938 -- GLOBAL_STEP: 985500 | > loss_text_ce: 0.030299220234155655 (0.032800069405240495) | > loss_mel_ce: 2.587768077850342 (nan) | > loss: 0.010389156639575958 (nan) | > current_lr: 5e-06 | > step_time: 0.1769 (0.13678706866725068) | > loader_time: 0.0048 (0.004671558391216183)  --> TIME: 2025-02-25 23:12:07 -- STEP: 25162/36938 -- GLOBAL_STEP: 985550 | > loss_text_ce: 0.0393093042075634 (0.0328017000462652) | > loss_mel_ce: 3.025054454803467 (nan) | > loss: 0.012160174548625946 (nan) | > current_lr: 5e-06 | > step_time: 0.1514 (0.13679606140763903) | > loader_time: 0.0048 (0.004671473825653462)  --> TIME: 2025-02-25 23:12:17 -- STEP: 25212/36938 -- GLOBAL_STEP: 985600 | > loss_text_ce: 0.03265400603413582 (0.032802855350075766) | > loss_mel_ce: 2.1864593029022217 (nan) | > loss: 0.00880600605159998 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13680005786571497) | > loader_time: 0.0045 (0.004671352724413936)  --> TIME: 2025-02-25 23:12:26 -- STEP: 25262/36938 -- GLOBAL_STEP: 985650 | > loss_text_ce: 0.031075147911906242 (0.032803841755289415) | > loss_mel_ce: 2.6425395011901855 (nan) | > loss: 0.010609582997858524 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13680905318394362) | > loader_time: 0.0048 (0.004671227440265133)  --> TIME: 2025-02-25 23:12:35 -- STEP: 25312/36938 -- GLOBAL_STEP: 985700 | > loss_text_ce: 0.03185010328888893 (0.03280592003848101) | > loss_mel_ce: 3.0168259143829346 (nan) | > loss: 0.012097921222448349 (nan) | > current_lr: 5e-06 | > step_time: 0.1439 (0.1368024967488401) | > loader_time: 0.0046 (0.004671060801607317)  --> TIME: 2025-02-25 23:12:45 -- STEP: 25362/36938 -- GLOBAL_STEP: 985750 | > loss_text_ce: 0.02999972738325596 (0.03280542357814811) | > loss_mel_ce: 2.618989944458008 (nan) | > loss: 0.010511863976716995 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.13679930692064757) | > loader_time: 0.0047 (0.00467090342155946)  --> TIME: 2025-02-25 23:12:54 -- STEP: 25412/36938 -- GLOBAL_STEP: 985800 | > loss_text_ce: 0.02660471200942993 (0.032803116178551216) | > loss_mel_ce: 2.968017578125 (nan) | > loss: 0.011883421801030636 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13680564529704894) | > loader_time: 0.0045 (0.004670864425267685)  --> TIME: 2025-02-25 23:13:04 -- STEP: 25462/36938 -- GLOBAL_STEP: 985850 | > loss_text_ce: 0.032858192920684814 (0.03280271506806765) | > loss_mel_ce: 2.0912201404571533 (nan) | > loss: 0.00842888280749321 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.1368067975355514) | > loader_time: 0.0049 (0.004670722403498968)  --> TIME: 2025-02-25 23:13:13 -- STEP: 25512/36938 -- GLOBAL_STEP: 985900 | > loss_text_ce: 0.03569265455007553 (0.032801430445015656) | > loss_mel_ce: 2.828709363937378 (nan) | > loss: 0.011366675607860088 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1368059546792734) | > loader_time: 0.0047 (0.004670850112543622)  --> TIME: 2025-02-25 23:13:22 -- STEP: 25562/36938 -- GLOBAL_STEP: 985950 | > loss_text_ce: 0.03489244729280472 (0.032803236936325184) | > loss_mel_ce: 3.557500123977661 (nan) | > loss: 0.014255527406930923 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.1368007505362563) | > loader_time: 0.0046 (0.00467086862430225)  --> TIME: 2025-02-25 23:13:31 -- STEP: 25612/36938 -- GLOBAL_STEP: 986000 | > loss_text_ce: 0.034012068063020706 (0.032803871256776106) | > loss_mel_ce: 2.9401819705963135 (nan) | > loss: 0.011802357621490955 (nan) | > current_lr: 5e-06 | > step_time: 0.1434 (0.13679984655489108) | > loader_time: 0.0047 (0.0046709009712233565)  --> TIME: 2025-02-25 23:13:41 -- STEP: 25662/36938 -- GLOBAL_STEP: 986050 | > loss_text_ce: 0.029869506135582924 (0.03280169649749914) | > loss_mel_ce: 3.3399875164031982 (nan) | > loss: 0.013372449204325676 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13680510534499643) | > loader_time: 0.0047 (0.004670960246685118)  --> TIME: 2025-02-25 23:13:50 -- STEP: 25712/36938 -- GLOBAL_STEP: 986100 | > loss_text_ce: 0.03146449476480484 (0.03280396854159958) | > loss_mel_ce: 3.1481499671936035 (nan) | > loss: 0.012617519125342369 (nan) | > current_lr: 5e-06 | > step_time: 0.1614 (0.13679636550305843) | > loader_time: 0.0049 (0.004670965769829489)  --> TIME: 2025-02-25 23:13:59 -- STEP: 25762/36938 -- GLOBAL_STEP: 986150 | > loss_text_ce: 0.03367939963936806 (0.032802191740343524) | > loss_mel_ce: 2.0295932292938232 (nan) | > loss: 0.008187590166926384 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13678268239268948) | > loader_time: 0.0047 (0.004670909061702872)  --> TIME: 2025-02-25 23:14:08 -- STEP: 25812/36938 -- GLOBAL_STEP: 986200 | > loss_text_ce: 0.03178565949201584 (0.032803165076580924) | > loss_mel_ce: 2.563812255859375 (nan) | > loss: 0.010299992747604847 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1367872874348487) | > loader_time: 0.0047 (0.004671115090491226)  --> TIME: 2025-02-25 23:14:17 -- STEP: 25862/36938 -- GLOBAL_STEP: 986250 | > loss_text_ce: 0.04447419196367264 (0.03280319428320128) | > loss_mel_ce: 3.282043695449829 (nan) | > loss: 0.013200468383729458 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.1367834498439616) | > loader_time: 0.0046 (0.004671405237702491)  --> TIME: 2025-02-25 23:14:27 -- STEP: 25912/36938 -- GLOBAL_STEP: 986300 | > loss_text_ce: 0.036759257316589355 (0.0328040857768152) | > loss_mel_ce: 2.6160337924957275 (nan) | > loss: 0.010526956990361214 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1367846937616077) | > loader_time: 0.0045 (0.004671273361425714)  --> TIME: 2025-02-25 23:14:36 -- STEP: 25962/36938 -- GLOBAL_STEP: 986350 | > loss_text_ce: 0.03091881051659584 (0.03280518618064696) | > loss_mel_ce: 2.6791114807128906 (nan) | > loss: 0.010754088871181011 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.1367868764146483) | > loader_time: 0.0046 (0.004671249080352643)  --> TIME: 2025-02-25 23:14:46 -- STEP: 26012/36938 -- GLOBAL_STEP: 986400 | > loss_text_ce: 0.027699854224920273 (0.03280407537424193) | > loss_mel_ce: 3.257714033126831 (nan) | > loss: 0.013037357479333878 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.1367913132334949) | > loader_time: 0.0047 (0.0046714001594278425)  --> TIME: 2025-02-25 23:14:55 -- STEP: 26062/36938 -- GLOBAL_STEP: 986450 | > loss_text_ce: 0.035481929779052734 (0.03280397967735082) | > loss_mel_ce: 3.2659313678741455 (nan) | > loss: 0.013100847601890564 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13678622791932102) | > loader_time: 0.0046 (0.004671683471371735)  --> TIME: 2025-02-25 23:15:04 -- STEP: 26112/36938 -- GLOBAL_STEP: 986500 | > loss_text_ce: 0.03839004412293434 (0.03280297922545135) | > loss_mel_ce: 3.042771339416504 (nan) | > loss: 0.0122268320992589 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13678380695409198) | > loader_time: 0.0047 (0.00467173306855794)  --> TIME: 2025-02-25 23:15:14 -- STEP: 26162/36938 -- GLOBAL_STEP: 986550 | > loss_text_ce: 0.034231036901474 (0.032803072639885515) | > loss_mel_ce: 2.1845757961273193 (nan) | > loss: 0.008804788812994957 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.1367858920054717) | > loader_time: 0.005 (0.0046717781656406936)  --> TIME: 2025-02-25 23:15:23 -- STEP: 26212/36938 -- GLOBAL_STEP: 986600 | > loss_text_ce: 0.03124706633388996 (0.03280040370571854) | > loss_mel_ce: 3.097954511642456 (nan) | > loss: 0.012417467311024666 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13678465372694334) | > loader_time: 0.0047 (0.004671793747690469)  --> TIME: 2025-02-25 23:15:32 -- STEP: 26262/36938 -- GLOBAL_STEP: 986650 | > loss_text_ce: 0.0408734455704689 (0.032800104412675044) | > loss_mel_ce: 0.08574140816926956 (nan) | > loss: 0.0005024399142712355 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1367933786920374) | > loader_time: 0.0045 (0.004671977730355156)  --> TIME: 2025-02-25 23:15:41 -- STEP: 26312/36938 -- GLOBAL_STEP: 986700 | > loss_text_ce: 0.03312058001756668 (0.03279920740798616) | > loss_mel_ce: 3.3709840774536133 (nan) | > loss: 0.01350835245102644 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13678792037475412) | > loader_time: 0.0045 (0.0046720399288248384)  --> TIME: 2025-02-25 23:15:51 -- STEP: 26362/36938 -- GLOBAL_STEP: 986750 | > loss_text_ce: 0.03435833752155304 (0.03279846652383574) | > loss_mel_ce: 0.834381103515625 (nan) | > loss: 0.003447378985583782 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13679751733232032) | > loader_time: 0.0045 (0.004672028679974708)  --> TIME: 2025-02-25 23:16:01 -- STEP: 26412/36938 -- GLOBAL_STEP: 986800 | > loss_text_ce: 0.034744635224342346 (0.03280072513356737) | > loss_mel_ce: 2.3770201206207275 (nan) | > loss: 0.009570496156811714 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.13680268868558126) | > loader_time: 0.0046 (0.004671985328910617)  --> TIME: 2025-02-25 23:16:10 -- STEP: 26462/36938 -- GLOBAL_STEP: 986850 | > loss_text_ce: 0.0268607996404171 (0.03279985931301649) | > loss_mel_ce: 2.793760061264038 (nan) | > loss: 0.011192940175533295 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.1368016501635666) | > loader_time: 0.0048 (0.0046719721985209435)  --> TIME: 2025-02-25 23:16:19 -- STEP: 26512/36938 -- GLOBAL_STEP: 986900 | > loss_text_ce: 0.02824362739920616 (0.03279813412315833) | > loss_mel_ce: 2.304133415222168 (nan) | > loss: 0.009255464188754559 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13679343282133388) | > loader_time: 0.0046 (0.004671838820232442)  --> TIME: 2025-02-25 23:16:28 -- STEP: 26562/36938 -- GLOBAL_STEP: 986950 | > loss_text_ce: 0.034347761422395706 (0.03279728036497689) | > loss_mel_ce: 2.9619970321655273 (nan) | > loss: 0.011890257708728313 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13679935669882964) | > loader_time: 0.0046 (0.004671937038305899)  --> TIME: 2025-02-25 23:16:38 -- STEP: 26612/36938 -- GLOBAL_STEP: 987000 | > loss_text_ce: 0.0355849526822567 (0.03279900147533973) | > loss_mel_ce: 2.2040131092071533 (nan) | > loss: 0.008887294679880142 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13679396624173207) | > loader_time: 0.0047 (0.004671842554144899)  --> TIME: 2025-02-25 23:16:47 -- STEP: 26662/36938 -- GLOBAL_STEP: 987050 | > loss_text_ce: 0.03111334890127182 (0.03280079471165069) | > loss_mel_ce: 2.3681156635284424 (nan) | > loss: 0.009520750492811203 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13678991224070022) | > loader_time: 0.0044 (0.0046717262564949765)  --> TIME: 2025-02-25 23:16:56 -- STEP: 26712/36938 -- GLOBAL_STEP: 987100 | > loss_text_ce: 0.03474869579076767 (0.032799837258661084) | > loss_mel_ce: 3.2867767810821533 (nan) | > loss: 0.013180658221244812 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13678941237744535) | > loader_time: 0.0047 (0.004671625888929599)  --> TIME: 2025-02-25 23:17:06 -- STEP: 26762/36938 -- GLOBAL_STEP: 987150 | > loss_text_ce: 0.026871362701058388 (0.03279901700149741) | > loss_mel_ce: 2.5207366943359375 (nan) | > loss: 0.010109556838870049 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13679039595898929) | > loader_time: 0.0046 (0.004671559447121841)  --> TIME: 2025-02-25 23:17:15 -- STEP: 26812/36938 -- GLOBAL_STEP: 987200 | > loss_text_ce: 0.031178291887044907 (0.03279869821431554) | > loss_mel_ce: 2.7923295497894287 (nan) | > loss: 0.01120439637452364 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.1367976278951428) | > loader_time: 0.0046 (0.004671524180310306)  --> TIME: 2025-02-25 23:17:24 -- STEP: 26862/36938 -- GLOBAL_STEP: 987250 | > loss_text_ce: 0.03915177658200264 (0.03279815785430543) | > loss_mel_ce: 3.0720138549804688 (nan) | > loss: 0.012345896102488041 (nan) | > current_lr: 5e-06 | > step_time: 0.1779 (0.13679722588093166) | > loader_time: 0.0048 (0.004671447639729499)  --> TIME: 2025-02-25 23:17:33 -- STEP: 26912/36938 -- GLOBAL_STEP: 987300 | > loss_text_ce: 0.021418333053588867 (0.032800405560576035) | > loss_mel_ce: 2.8588507175445557 (nan) | > loss: 0.011429639533162117 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1367882389606267) | > loader_time: 0.0045 (0.004671414253190095)  --> TIME: 2025-02-25 23:17:43 -- STEP: 26962/36938 -- GLOBAL_STEP: 987350 | > loss_text_ce: 0.0351090244948864 (0.03280003316195182) | > loss_mel_ce: 3.0335607528686523 (nan) | > loss: 0.012177261523902416 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.13680000095956169) | > loader_time: 0.0046 (0.004671350208820413)  --> TIME: 2025-02-25 23:17:53 -- STEP: 27012/36938 -- GLOBAL_STEP: 987400 | > loss_text_ce: 0.02795681729912758 (0.03280125603587455) | > loss_mel_ce: 3.040950059890747 (nan) | > loss: 0.012178202159702778 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.1368086425424538) | > loader_time: 0.0045 (0.004671380561527545)  --> TIME: 2025-02-25 23:18:02 -- STEP: 27062/36938 -- GLOBAL_STEP: 987450 | > loss_text_ce: 0.024144483730196953 (0.03280159510101782) | > loss_mel_ce: 2.374859094619751 (nan) | > loss: 0.009519855491816998 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13680936376409564) | > loader_time: 0.0048 (0.0046713954813332)  --> TIME: 2025-02-25 23:18:12 -- STEP: 27112/36938 -- GLOBAL_STEP: 987500 | > loss_text_ce: 0.039314985275268555 (0.03279974420905182) | > loss_mel_ce: 3.1737136840820312 (nan) | > loss: 0.012750114314258099 (nan) | > current_lr: 5e-06 | > step_time: 0.1902 (0.13681049847645207) | > loader_time: 0.0049 (0.004671368575370515)  --> TIME: 2025-02-25 23:18:21 -- STEP: 27162/36938 -- GLOBAL_STEP: 987550 | > loss_text_ce: 0.0252238716930151 (0.032800781302444605) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13680559415615876) | > loader_time: 0.0045 (0.004671301891592919)  --> TIME: 2025-02-25 23:18:30 -- STEP: 27212/36938 -- GLOBAL_STEP: 987600 | > loss_text_ce: 0.031554851680994034 (0.03279960692154342) | > loss_mel_ce: 0.056304931640625 (nan) | > loss: 0.000348649948136881 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13680463527276998) | > loader_time: 0.0047 (0.004671249830528321)  --> TIME: 2025-02-25 23:18:39 -- STEP: 27262/36938 -- GLOBAL_STEP: 987650 | > loss_text_ce: 0.03569748252630234 (0.03279894483621779) | > loss_mel_ce: 1.5934549570083618 (nan) | > loss: 0.0064648911356925964 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13679933498999253) | > loader_time: 0.0046 (0.004671145312793638)  --> TIME: 2025-02-25 23:18:49 -- STEP: 27312/36938 -- GLOBAL_STEP: 987700 | > loss_text_ce: 0.04748731851577759 (0.03279846674704297) | > loss_mel_ce: 3.2596092224121094 (nan) | > loss: 0.013123399578034878 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13680261170124014) | > loader_time: 0.0047 (0.004671134617709962)  --> TIME: 2025-02-25 23:18:58 -- STEP: 27362/36938 -- GLOBAL_STEP: 987750 | > loss_text_ce: 0.03307686373591423 (0.03280007038643964) | > loss_mel_ce: 2.9242324829101562 (nan) | > loss: 0.01173535455018282 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13679226085584115) | > loader_time: 0.0046 (0.004671055952905446)  --> TIME: 2025-02-25 23:19:07 -- STEP: 27412/36938 -- GLOBAL_STEP: 987800 | > loss_text_ce: 0.03252016380429268 (0.0327979322222812) | > loss_mel_ce: 2.146693468093872 (nan) | > loss: 0.008647672832012177 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13679741963355133) | > loader_time: 0.0045 (0.004670981741222917)  --> TIME: 2025-02-25 23:19:17 -- STEP: 27462/36938 -- GLOBAL_STEP: 987850 | > loss_text_ce: 0.030104894191026688 (0.0327968581476327) | > loss_mel_ce: 2.555370569229126 (nan) | > loss: 0.010259823873639107 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13680724595503252) | > loader_time: 0.0046 (0.0046708618298386875)  --> TIME: 2025-02-25 23:19:26 -- STEP: 27512/36938 -- GLOBAL_STEP: 987900 | > loss_text_ce: 0.03296346217393875 (0.03279718684883458) | > loss_mel_ce: 2.733527421951294 (nan) | > loss: 0.010978139005601406 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13680851993328022) | > loader_time: 0.0046 (0.004670746210668987)  --> TIME: 2025-02-25 23:19:36 -- STEP: 27562/36938 -- GLOBAL_STEP: 987950 | > loss_text_ce: 0.031312376260757446 (0.03279590111137455) | > loss_mel_ce: 2.8272316455841064 (nan) | > loss: 0.011343429796397686 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13681345930224906) | > loader_time: 0.0047 (0.004670641633511191)  --> TIME: 2025-02-25 23:19:45 -- STEP: 27612/36938 -- GLOBAL_STEP: 988000 | > loss_text_ce: 0.03461330756545067 (0.03279621314902827) | > loss_mel_ce: 2.0579726696014404 (nan) | > loss: 0.008303913287818432 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1368176525837282) | > loader_time: 0.0046 (0.004670606106066396)  --> TIME: 2025-02-25 23:19:54 -- STEP: 27662/36938 -- GLOBAL_STEP: 988050 | > loss_text_ce: 0.03357082977890968 (0.03279593302144792) | > loss_mel_ce: 2.5178000926971436 (nan) | > loss: 0.010124487802386284 (nan) | > current_lr: 5e-06 | > step_time: 0.1511 (0.1368129017571823) | > loader_time: 0.0046 (0.004670545048314121)  --> TIME: 2025-02-25 23:20:03 -- STEP: 27712/36938 -- GLOBAL_STEP: 988100 | > loss_text_ce: 0.02415359951555729 (0.03279526502276589) | > loss_mel_ce: 2.630836248397827 (nan) | > loss: 0.01053567510098219 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13680731358036877) | > loader_time: 0.0047 (0.004670641111859562)  --> TIME: 2025-02-25 23:20:13 -- STEP: 27762/36938 -- GLOBAL_STEP: 988150 | > loss_text_ce: 0.02469366230070591 (0.032792980004941796) | > loss_mel_ce: 2.5458385944366455 (nan) | > loss: 0.010200525633990765 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13680697263777317) | > loader_time: 0.0046 (0.004670696989894485)  --> TIME: 2025-02-25 23:20:22 -- STEP: 27812/36938 -- GLOBAL_STEP: 988200 | > loss_text_ce: 0.03704957291483879 (0.03279385584014502) | > loss_mel_ce: 1.562041163444519 (nan) | > loss: 0.006345598492771387 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1368063387983277) | > loader_time: 0.0045 (0.004670720940165682)  --> TIME: 2025-02-25 23:20:31 -- STEP: 27862/36938 -- GLOBAL_STEP: 988250 | > loss_text_ce: 0.03910857066512108 (0.03279435766225614) | > loss_mel_ce: 3.2646641731262207 (nan) | > loss: 0.013110210187733173 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13680054868305758) | > loader_time: 0.0045 (0.00467069955440668)  --> TIME: 2025-02-25 23:20:41 -- STEP: 27912/36938 -- GLOBAL_STEP: 988300 | > loss_text_ce: 0.03249116986989975 (0.032794298908699185) | > loss_mel_ce: 2.110748291015625 (nan) | > loss: 0.008504918776452541 (nan) | > current_lr: 5e-06 | > step_time: 0.1583 (0.13680345274446246) | > loader_time: 0.0046 (0.004670640140319164)  --> TIME: 2025-02-25 23:20:50 -- STEP: 27962/36938 -- GLOBAL_STEP: 988350 | > loss_text_ce: 0.02877216972410679 (0.03279400616491222) | > loss_mel_ce: 2.8056387901306152 (nan) | > loss: 0.011247662827372551 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.13680758100067392) | > loader_time: 0.0048 (0.004670573946965998)  --> TIME: 2025-02-25 23:21:00 -- STEP: 28012/36938 -- GLOBAL_STEP: 988400 | > loss_text_ce: 0.03249064087867737 (0.03279345322342625) | > loss_mel_ce: 3.155552387237549 (nan) | > loss: 0.012650965712964535 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13680832117570568) | > loader_time: 0.0045 (0.004670554495668077)  --> TIME: 2025-02-25 23:21:09 -- STEP: 28062/36938 -- GLOBAL_STEP: 988450 | > loss_text_ce: 0.030720235779881477 (0.03279332507235244) | > loss_mel_ce: 3.0746076107025146 (nan) | > loss: 0.012322730384767056 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.13681294742870148) | > loader_time: 0.0047 (0.0046705440431249285)  --> TIME: 2025-02-25 23:21:18 -- STEP: 28112/36938 -- GLOBAL_STEP: 988500 | > loss_text_ce: 0.014035285450518131 (0.0327935509583315) | > loss_mel_ce: 2.171562433242798 (nan) | > loss: 0.008673006668686867 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13681600052454332) | > loader_time: 0.0044 (0.004670460979302104)  --> TIME: 2025-02-25 23:21:28 -- STEP: 28162/36938 -- GLOBAL_STEP: 988550 | > loss_text_ce: 0.03210240602493286 (0.03279336814776367) | > loss_mel_ce: 2.7284281253814697 (nan) | > loss: 0.010954486206173897 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13681959418459602) | > loader_time: 0.0046 (0.004670377219911086)  --> TIME: 2025-02-25 23:21:37 -- STEP: 28212/36938 -- GLOBAL_STEP: 988600 | > loss_text_ce: 0.03850097581744194 (0.0327941323671778) | > loss_mel_ce: 3.1188693046569824 (nan) | > loss: 0.012529248371720314 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.1368204093682569) | > loader_time: 0.0045 (0.0046703193976357655)  --> TIME: 2025-02-25 23:21:47 -- STEP: 28262/36938 -- GLOBAL_STEP: 988650 | > loss_text_ce: 0.03210682049393654 (0.03279573146467245) | > loss_mel_ce: 2.6229519844055176 (nan) | > loss: 0.010535948909819126 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.1368173643937176) | > loader_time: 0.0045 (0.00467021249677041)  --> TIME: 2025-02-25 23:21:56 -- STEP: 28312/36938 -- GLOBAL_STEP: 988700 | > loss_text_ce: 0.033844780176877975 (0.03279480730175148) | > loss_mel_ce: 2.729534864425659 (nan) | > loss: 0.010965792462229729 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.1368145387483963) | > loader_time: 0.0046 (0.004670136727394792)  --> TIME: 2025-02-25 23:22:05 -- STEP: 28362/36938 -- GLOBAL_STEP: 988750 | > loss_text_ce: 0.03523819521069527 (0.03279457577806924) | > loss_mel_ce: 2.594784736633301 (nan) | > loss: 0.01043660007417202 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13681841216044838) | > loader_time: 0.0048 (0.004670217707853019)  --> TIME: 2025-02-25 23:22:15 -- STEP: 28412/36938 -- GLOBAL_STEP: 988800 | > loss_text_ce: 0.025981634855270386 (0.03279147699695448) | > loss_mel_ce: 3.1573359966278076 (nan) | > loss: 0.012632213532924652 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13682078610301882) | > loader_time: 0.0048 (0.004670251990042668)  --> TIME: 2025-02-25 23:22:24 -- STEP: 28462/36938 -- GLOBAL_STEP: 988850 | > loss_text_ce: 0.030471565201878548 (0.032793574842181004) | > loss_mel_ce: 2.1280672550201416 (nan) | > loss: 0.00856563076376915 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13681401317482195) | > loader_time: 0.0047 (0.004670249059608917)  --> TIME: 2025-02-25 23:22:33 -- STEP: 28512/36938 -- GLOBAL_STEP: 988900 | > loss_text_ce: 0.03911209851503372 (0.032794032061550314) | > loss_mel_ce: 0.045654296875 (nan) | > loss: 0.00033637459273450077 (nan) | > current_lr: 5e-06 | > step_time: 0.177 (0.13681965708465227) | > loader_time: 0.0048 (0.004670306814439382)  --> TIME: 2025-02-25 23:22:43 -- STEP: 28562/36938 -- GLOBAL_STEP: 988950 | > loss_text_ce: 0.04001481831073761 (0.03279325396217612) | > loss_mel_ce: 3.381640911102295 (nan) | > loss: 0.013577999547123909 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.1368238196871514) | > loader_time: 0.0047 (0.004670353181538629)  --> TIME: 2025-02-25 23:22:52 -- STEP: 28612/36938 -- GLOBAL_STEP: 989000 | > loss_text_ce: 0.034237269312143326 (0.03279137898490557) | > loss_mel_ce: 2.5981099605560303 (nan) | > loss: 0.010445822961628437 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1368127207097439) | > loader_time: 0.0045 (0.004670377971241741)  --> TIME: 2025-02-25 23:23:01 -- STEP: 28662/36938 -- GLOBAL_STEP: 989050 | > loss_text_ce: 0.031735409051179886 (0.032791816594669294) | > loss_mel_ce: 2.7367477416992188 (nan) | > loss: 0.010986045002937317 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.136808983413369) | > loader_time: 0.0046 (0.004670357081955168)  --> TIME: 2025-02-25 23:23:10 -- STEP: 28712/36938 -- GLOBAL_STEP: 989100 | > loss_text_ce: 0.0399659238755703 (0.03279206908376627) | > loss_mel_ce: 2.9065825939178467 (nan) | > loss: 0.0116926534101367 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.1367978467835682) | > loader_time: 0.0046 (0.00467034639605377)  --> TIME: 2025-02-25 23:23:19 -- STEP: 28762/36938 -- GLOBAL_STEP: 989150 | > loss_text_ce: 0.035352304577827454 (0.03279197480571403) | > loss_mel_ce: 0.9741237163543701 (nan) | > loss: 0.004005857743322849 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13679303242429197) | > loader_time: 0.0045 (0.004670304098527838)  --> TIME: 2025-02-25 23:23:28 -- STEP: 28812/36938 -- GLOBAL_STEP: 989200 | > loss_text_ce: 0.036093030124902725 (0.032792774903023535) | > loss_mel_ce: 2.5250730514526367 (nan) | > loss: 0.010163357481360435 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13678874318904125) | > loader_time: 0.0045 (0.004670261674732971)  --> TIME: 2025-02-25 23:23:37 -- STEP: 28862/36938 -- GLOBAL_STEP: 989250 | > loss_text_ce: 0.026777643710374832 (0.03279085596274762) | > loss_mel_ce: 3.3902587890625 (nan) | > loss: 0.013559669256210327 (nan) | > current_lr: 5e-06 | > step_time: 0.1796 (0.13678837406537225) | > loader_time: 0.005 (0.004670259916363577)  --> TIME: 2025-02-25 23:23:47 -- STEP: 28912/36938 -- GLOBAL_STEP: 989300 | > loss_text_ce: 0.02465067058801651 (0.03279114820590251) | > loss_mel_ce: 1.644372582435608 (nan) | > loss: 0.00662310840561986 (nan) | > current_lr: 5e-06 | > step_time: 0.1567 (0.13679311614572498) | > loader_time: 0.0047 (0.004670215002660548)  --> TIME: 2025-02-25 23:23:57 -- STEP: 28962/36938 -- GLOBAL_STEP: 989350 | > loss_text_ce: 0.03320517763495445 (0.03279140549981693) | > loss_mel_ce: 2.625476121902466 (nan) | > loss: 0.010550323873758316 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.1368029716635424) | > loader_time: 0.0045 (0.004670170680337774)  --> TIME: 2025-02-25 23:24:06 -- STEP: 29012/36938 -- GLOBAL_STEP: 989400 | > loss_text_ce: 0.03275433927774429 (0.03279129907783016) | > loss_mel_ce: 2.9749655723571777 (nan) | > loss: 0.011935397051274776 (nan) | > current_lr: 5e-06 | > step_time: 0.1779 (0.1368005206202658) | > loader_time: 0.0049 (0.004670283325323657)  --> TIME: 2025-02-25 23:24:15 -- STEP: 29062/36938 -- GLOBAL_STEP: 989450 | > loss_text_ce: 0.03934403881430626 (0.03278980046661586) | > loss_mel_ce: 3.425917148590088 (nan) | > loss: 0.013751037418842316 (nan) | > current_lr: 5e-06 | > step_time: 0.1934 (0.1367976005337776) | > loader_time: 0.0049 (0.0046703417576333265)  --> TIME: 2025-02-25 23:24:24 -- STEP: 29112/36938 -- GLOBAL_STEP: 989500 | > loss_text_ce: 0.039072658866643906 (0.03278895231282089) | > loss_mel_ce: 2.6762616634368896 (nan) | > loss: 0.010775136761367321 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13679705388633373) | > loader_time: 0.0046 (0.004670524046820066)  --> TIME: 2025-02-25 23:24:34 -- STEP: 29162/36938 -- GLOBAL_STEP: 989550 | > loss_text_ce: 0.03056958317756653 (0.03278912600511433) | > loss_mel_ce: 2.448817729949951 (nan) | > loss: 0.009838839061558247 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13679264344090797) | > loader_time: 0.0048 (0.004670738111053453)  --> TIME: 2025-02-25 23:24:43 -- STEP: 29212/36938 -- GLOBAL_STEP: 989600 | > loss_text_ce: 0.028642650693655014 (0.03279023226189241) | > loss_mel_ce: 3.0588977336883545 (nan) | > loss: 0.012252145446836948 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13679581137041974) | > loader_time: 0.0046 (0.004670782749022468)  --> TIME: 2025-02-25 23:24:53 -- STEP: 29262/36938 -- GLOBAL_STEP: 989650 | > loss_text_ce: 0.03195430710911751 (0.032789012868663835) | > loss_mel_ce: 2.7750604152679443 (nan) | > loss: 0.011138947680592537 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.1368043725142363) | > loader_time: 0.0046 (0.004670940732183541)  --> TIME: 2025-02-25 23:25:02 -- STEP: 29312/36938 -- GLOBAL_STEP: 989700 | > loss_text_ce: 0.02888995036482811 (0.03279028937123872) | > loss_mel_ce: 2.410402536392212 (nan) | > loss: 0.009679732844233513 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13680180964999863) | > loader_time: 0.0048 (0.00467095184078905)  --> TIME: 2025-02-25 23:25:11 -- STEP: 29362/36938 -- GLOBAL_STEP: 989750 | > loss_text_ce: 0.03673803433775902 (0.03279051636559165) | > loss_mel_ce: 2.7260825634002686 (nan) | > loss: 0.010963574051856995 (nan) | > current_lr: 5e-06 | > step_time: 0.1614 (0.1367978616888208) | > loader_time: 0.0047 (0.0046709106351900485)  --> TIME: 2025-02-25 23:25:21 -- STEP: 29412/36938 -- GLOBAL_STEP: 989800 | > loss_text_ce: 0.042613714933395386 (0.032791919502078055) | > loss_mel_ce: 1.758599877357483 (nan) | > loss: 0.007147673517465591 (nan) | > current_lr: 5e-06 | > step_time: 0.1123 (0.13679990057684183) | > loader_time: 0.0046 (0.004670985260902105)  --> TIME: 2025-02-25 23:25:30 -- STEP: 29462/36938 -- GLOBAL_STEP: 989850 | > loss_text_ce: 0.03860495984554291 (0.032792701027225674) | > loss_mel_ce: 2.8086657524108887 (nan) | > loss: 0.011298693716526031 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.13679832058634211) | > loader_time: 0.0047 (0.004670971952058607)  --> TIME: 2025-02-25 23:25:39 -- STEP: 29512/36938 -- GLOBAL_STEP: 989900 | > loss_text_ce: 0.03856102004647255 (0.03279235644472914) | > loss_mel_ce: 2.584059476852417 (nan) | > loss: 0.010407225228846073 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13679398799466336) | > loader_time: 0.0047 (0.004670946570262265)  --> TIME: 2025-02-25 23:25:48 -- STEP: 29562/36938 -- GLOBAL_STEP: 989950 | > loss_text_ce: 0.025833506137132645 (0.032793010412147494) | > loss_mel_ce: 2.346555471420288 (nan) | > loss: 0.009414242580533028 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13678748536986055) | > loader_time: 0.0045 (0.004670844858114209)  --> TIME: 2025-02-25 23:25:58 -- STEP: 29612/36938 -- GLOBAL_STEP: 990000 | > loss_text_ce: 0.036537859588861465 (0.03279109374781315) | > loss_mel_ce: 2.684119462966919 (nan) | > loss: 0.010796260088682175 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13678726372357727) | > loader_time: 0.0046 (0.004670847497662077) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_990000.pth  --> TIME: 2025-02-25 23:26:13 -- STEP: 29662/36938 -- GLOBAL_STEP: 990050 | > loss_text_ce: 0.036232221871614456 (0.03279157584974188) | > loss_mel_ce: 3.0687289237976074 (nan) | > loss: 0.012321274727582932 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13678797555222433) | > loader_time: 0.0046 (0.004670943897819107)  --> TIME: 2025-02-25 23:26:22 -- STEP: 29712/36938 -- GLOBAL_STEP: 990100 | > loss_text_ce: 0.03454916551709175 (0.032789592592199525) | > loss_mel_ce: 3.1698856353759766 (nan) | > loss: 0.012716012075543404 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13678957681599646) | > loader_time: 0.0046 (0.004671048647816886)  --> TIME: 2025-02-25 23:26:31 -- STEP: 29762/36938 -- GLOBAL_STEP: 990150 | > loss_text_ce: 0.027728676795959473 (0.03278945980635634) | > loss_mel_ce: 2.6052260398864746 (nan) | > loss: 0.010448233224451542 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13677726256720268) | > loader_time: 0.0048 (0.004670954040602722)  --> TIME: 2025-02-25 23:26:40 -- STEP: 29812/36938 -- GLOBAL_STEP: 990200 | > loss_text_ce: 0.039793502539396286 (0.03279095907997009) | > loss_mel_ce: 3.3428618907928467 (nan) | > loss: 0.013423236086964607 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1367740416933089) | > loader_time: 0.0045 (0.004671004335784003)  --> TIME: 2025-02-25 23:26:49 -- STEP: 29862/36938 -- GLOBAL_STEP: 990250 | > loss_text_ce: 0.030220432206988335 (0.032788956353961125) | > loss_mel_ce: 2.483553886413574 (nan) | > loss: 0.009975295513868332 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13676530358684788) | > loader_time: 0.0045 (0.004670933153455104)  --> TIME: 2025-02-25 23:26:59 -- STEP: 29912/36938 -- GLOBAL_STEP: 990300 | > loss_text_ce: 0.03043294884264469 (0.03278538682044725) | > loss_mel_ce: 3.010370969772339 (nan) | > loss: 0.012066682800650597 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.1367646884506952) | > loader_time: 0.0047 (0.004670973352073647)  --> TIME: 2025-02-25 23:27:08 -- STEP: 29962/36938 -- GLOBAL_STEP: 990350 | > loss_text_ce: 0.025029117241501808 (0.03278242325083019) | > loss_mel_ce: 2.8545870780944824 (nan) | > loss: 0.011427049525082111 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.1367578503048084) | > loader_time: 0.0048 (0.004670909127297238)  --> TIME: 2025-02-25 23:27:17 -- STEP: 30012/36938 -- GLOBAL_STEP: 990400 | > loss_text_ce: 0.02914705127477646 (0.03278429993902721) | > loss_mel_ce: 2.0803616046905518 (nan) | > loss: 0.008371067233383656 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1367607276852467) | > loader_time: 0.0045 (0.004670845887096325)  --> TIME: 2025-02-25 23:27:26 -- STEP: 30062/36938 -- GLOBAL_STEP: 990450 | > loss_text_ce: 0.04256892576813698 (0.032783052732642926) | > loss_mel_ce: 2.8744795322418213 (nan) | > loss: 0.011575589887797832 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.13675351754672818) | > loader_time: 0.0048 (0.004670789638176938)  --> TIME: 2025-02-25 23:27:36 -- STEP: 30112/36938 -- GLOBAL_STEP: 990500 | > loss_text_ce: 0.025666292756795883 (0.03278446949597924) | > loss_mel_ce: 3.163024663925171 (nan) | > loss: 0.012653536163270473 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13675548171864294) | > loader_time: 0.0046 (0.004670723425531247)  --> TIME: 2025-02-25 23:27:45 -- STEP: 30162/36938 -- GLOBAL_STEP: 990550 | > loss_text_ce: 0.03084738180041313 (0.032784429383470266) | > loss_mel_ce: 2.759990930557251 (nan) | > loss: 0.01107475534081459 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.13675019760978185) | > loader_time: 0.0046 (0.004670699152893682)  --> TIME: 2025-02-25 23:27:54 -- STEP: 30212/36938 -- GLOBAL_STEP: 990600 | > loss_text_ce: 0.03721431642770767 (0.032781779602488084) | > loss_mel_ce: 2.7332913875579834 (nan) | > loss: 0.010994071140885353 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13674621405734952) | > loader_time: 0.0048 (0.004670619656830371)  --> TIME: 2025-02-25 23:28:04 -- STEP: 30262/36938 -- GLOBAL_STEP: 990650 | > loss_text_ce: 0.03179916366934776 (0.03278132128286084) | > loss_mel_ce: 3.1232757568359375 (nan) | > loss: 0.012520139105618 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.13675592500451036) | > loader_time: 0.0048 (0.004670570369564414)  --> TIME: 2025-02-25 23:28:13 -- STEP: 30312/36938 -- GLOBAL_STEP: 990700 | > loss_text_ce: 0.033685848116874695 (0.03278224086382206) | > loss_mel_ce: 3.019399881362915 (nan) | > loss: 0.012115420773625374 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.1367470635198067) | > loader_time: 0.0045 (0.00467043202670278)  --> TIME: 2025-02-25 23:28:22 -- STEP: 30362/36938 -- GLOBAL_STEP: 990750 | > loss_text_ce: 0.038194067776203156 (0.03278244928913353) | > loss_mel_ce: 2.8571598529815674 (nan) | > loss: 0.011489501222968102 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13675118590460006) | > loader_time: 0.0046 (0.004670485866909242)  --> TIME: 2025-02-25 23:28:31 -- STEP: 30412/36938 -- GLOBAL_STEP: 990800 | > loss_text_ce: 0.03355279564857483 (0.03278114126045771) | > loss_mel_ce: 2.9979941844940186 (nan) | > loss: 0.012029949575662613 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.1367461462405353) | > loader_time: 0.0046 (0.0046704715213602655)  --> TIME: 2025-02-25 23:28:41 -- STEP: 30462/36938 -- GLOBAL_STEP: 990850 | > loss_text_ce: 0.04481625556945801 (0.032784777095101386) | > loss_mel_ce: 2.126810073852539 (nan) | > loss: 0.008617565035820007 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13674404975519927) | > loader_time: 0.0045 (0.004670452824268825)  --> TIME: 2025-02-25 23:28:50 -- STEP: 30512/36938 -- GLOBAL_STEP: 990900 | > loss_text_ce: 0.025221075862646103 (0.03278370608244667) | > loss_mel_ce: 1.9872685670852661 (nan) | > loss: 0.007986070588231087 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13674781807099318) | > loader_time: 0.0045 (0.004670510366441832)  --> TIME: 2025-02-25 23:28:59 -- STEP: 30562/36938 -- GLOBAL_STEP: 990950 | > loss_text_ce: 0.03704569861292839 (0.03278149631676832) | > loss_mel_ce: 3.5341341495513916 (nan) | > loss: 0.014171349816024303 (nan) | > current_lr: 5e-06 | > step_time: 0.1708 (0.1367509958982944) | > loader_time: 0.0048 (0.004670457841214015)  --> TIME: 2025-02-25 23:29:09 -- STEP: 30612/36938 -- GLOBAL_STEP: 991000 | > loss_text_ce: 0.03680173307657242 (0.03278191337183912) | > loss_mel_ce: 1.234686255455017 (nan) | > loss: 0.005045587196946144 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13674702560856708) | > loader_time: 0.0045 (0.004670641258092215)  --> TIME: 2025-02-25 23:29:18 -- STEP: 30662/36938 -- GLOBAL_STEP: 991050 | > loss_text_ce: 0.04027145355939865 (0.03278123767477677) | > loss_mel_ce: 3.1532700061798096 (nan) | > loss: 0.012672784738242626 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13675268240991031) | > loader_time: 0.0048 (0.004670651526174619)  --> TIME: 2025-02-25 23:29:27 -- STEP: 30712/36938 -- GLOBAL_STEP: 991100 | > loss_text_ce: 0.03572920337319374 (0.03278112741928725) | > loss_mel_ce: 3.0813846588134766 (nan) | > loss: 0.01236950047314167 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.1367429790629753) | > loader_time: 0.0046 (0.004670602909193973)  --> TIME: 2025-02-25 23:29:37 -- STEP: 30762/36938 -- GLOBAL_STEP: 991150 | > loss_text_ce: 0.03379714861512184 (0.03278146894869335) | > loss_mel_ce: 2.8708245754241943 (nan) | > loss: 0.011526278220117092 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13674560054971252) | > loader_time: 0.0045 (0.004670560046062627)  --> TIME: 2025-02-25 23:29:45 -- STEP: 30812/36938 -- GLOBAL_STEP: 991200 | > loss_text_ce: 0.024444574490189552 (0.03277843213667011) | > loss_mel_ce: 2.4111897945404053 (nan) | > loss: 0.009665216319262981 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.13673462416465745) | > loader_time: 0.0046 (0.004670426054123434)  --> TIME: 2025-02-25 23:29:55 -- STEP: 30862/36938 -- GLOBAL_STEP: 991250 | > loss_text_ce: 0.024719998240470886 (0.032778158847616465) | > loss_mel_ce: 2.859569787979126 (nan) | > loss: 0.01144559495151043 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13673056882350867) | > loader_time: 0.0045 (0.004670334583848841)  --> TIME: 2025-02-25 23:30:04 -- STEP: 30912/36938 -- GLOBAL_STEP: 991300 | > loss_text_ce: 0.035505447536706924 (0.03277798504028151) | > loss_mel_ce: 2.8360328674316406 (nan) | > loss: 0.011394994333386421 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.136742918750208) | > loader_time: 0.0046 (0.004670336734557509)  --> TIME: 2025-02-25 23:30:14 -- STEP: 30962/36938 -- GLOBAL_STEP: 991350 | > loss_text_ce: 0.02711332216858864 (0.032774341032341066) | > loss_mel_ce: 2.8126420974731445 (nan) | > loss: 0.01126887183636427 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13673891251759357) | > loader_time: 0.0047 (0.0046702622366262755)  --> TIME: 2025-02-25 23:30:23 -- STEP: 31012/36938 -- GLOBAL_STEP: 991400 | > loss_text_ce: 0.027301978319883347 (0.0327733344075967) | > loss_mel_ce: 3.430288553237915 (nan) | > loss: 0.013720598071813583 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13674060514661387) | > loader_time: 0.0046 (0.00467018325851916)  --> TIME: 2025-02-25 23:30:32 -- STEP: 31062/36938 -- GLOBAL_STEP: 991450 | > loss_text_ce: 0.03436356782913208 (0.0327742533926727) | > loss_mel_ce: 3.007103443145752 (nan) | > loss: 0.012069313786923885 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13674458434397624) | > loader_time: 0.0048 (0.004670155577217199)  --> TIME: 2025-02-25 23:30:42 -- STEP: 31112/36938 -- GLOBAL_STEP: 991500 | > loss_text_ce: 0.0353713296353817 (0.03277527653682501) | > loss_mel_ce: 2.6995849609375 (nan) | > loss: 0.010853001847863197 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13674986472808098) | > loader_time: 0.0048 (0.0046701104973851755)  --> TIME: 2025-02-25 23:30:51 -- STEP: 31162/36938 -- GLOBAL_STEP: 991550 | > loss_text_ce: 0.03162609785795212 (0.03277551840990681) | > loss_mel_ce: 2.8264856338500977 (nan) | > loss: 0.01134171336889267 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13675131436513394) | > loader_time: 0.0047 (0.00467009515605027)  --> TIME: 2025-02-25 23:31:01 -- STEP: 31212/36938 -- GLOBAL_STEP: 991600 | > loss_text_ce: 0.029663406312465668 (0.032773526695419976) | > loss_mel_ce: 2.8035523891448975 (nan) | > loss: 0.011242920532822609 (nan) | > current_lr: 5e-06 | > step_time: 0.1577 (0.13675411992939213) | > loader_time: 0.0046 (0.004670034665779087)  --> TIME: 2025-02-25 23:31:10 -- STEP: 31262/36938 -- GLOBAL_STEP: 991650 | > loss_text_ce: 0.0357539989054203 (0.03277001988458408) | > loss_mel_ce: 2.6735289096832275 (nan) | > loss: 0.010751123540103436 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.1367542720727222) | > loader_time: 0.0046 (0.0046699613201192)  --> TIME: 2025-02-25 23:31:19 -- STEP: 31312/36938 -- GLOBAL_STEP: 991700 | > loss_text_ce: 0.02835289016366005 (0.032770355364685526) | > loss_mel_ce: 3.958423137664795 (nan) | > loss: 0.015820540487766266 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.1367481823365323) | > loader_time: 0.0046 (0.004669906840862896)  --> TIME: 2025-02-25 23:31:29 -- STEP: 31362/36938 -- GLOBAL_STEP: 991750 | > loss_text_ce: 0.03251003846526146 (0.03276986930175241) | > loss_mel_ce: 3.0812137126922607 (nan) | > loss: 0.01235604751855135 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13675438498715028) | > loader_time: 0.0046 (0.004670218898054832)  --> TIME: 2025-02-25 23:31:38 -- STEP: 31412/36938 -- GLOBAL_STEP: 991800 | > loss_text_ce: 0.033214252442121506 (0.032769776834993436) | > loss_mel_ce: 3.576808452606201 (nan) | > loss: 0.014325488358736038 (nan) | > current_lr: 5e-06 | > step_time: 0.1506 (0.13675697215208127) | > loader_time: 0.0047 (0.004670276325188924)  --> TIME: 2025-02-25 23:31:47 -- STEP: 31462/36938 -- GLOBAL_STEP: 991850 | > loss_text_ce: 0.023220598697662354 (0.032770086351133834) | > loss_mel_ce: 2.319082498550415 (nan) | > loss: 0.009294853545725346 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.13675412718107652) | > loader_time: 0.0047 (0.004670354204648916)  --> TIME: 2025-02-25 23:31:57 -- STEP: 31512/36938 -- GLOBAL_STEP: 991900 | > loss_text_ce: 0.02737935073673725 (0.03276802446364195) | > loss_mel_ce: 2.6637346744537354 (nan) | > loss: 0.010679024271667004 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13676005442241376) | > loader_time: 0.0045 (0.004670448565307575)  --> TIME: 2025-02-25 23:32:06 -- STEP: 31562/36938 -- GLOBAL_STEP: 991950 | > loss_text_ce: 0.03564890846610069 (0.032766374808122944) | > loss_mel_ce: 2.5402963161468506 (nan) | > loss: 0.010222004726529121 (nan) | > current_lr: 5e-06 | > step_time: 0.1823 (0.1367530919898056) | > loader_time: 0.005 (0.004670368575844348)  --> TIME: 2025-02-25 23:32:15 -- STEP: 31612/36938 -- GLOBAL_STEP: 992000 | > loss_text_ce: 0.029281822964549065 (0.032766571076589104) | > loss_mel_ce: 2.5663416385650635 (nan) | > loss: 0.01030009426176548 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13674326625848582) | > loader_time: 0.0046 (0.004670322001711966)  --> TIME: 2025-02-25 23:32:24 -- STEP: 31662/36938 -- GLOBAL_STEP: 992050 | > loss_text_ce: 0.030972477048635483 (0.03276786157653935) | > loss_mel_ce: 2.5572009086608887 (nan) | > loss: 0.010270530357956886 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13674329801726787) | > loader_time: 0.0047 (0.004670330484294005)  --> TIME: 2025-02-25 23:32:34 -- STEP: 31712/36938 -- GLOBAL_STEP: 992100 | > loss_text_ce: 0.026896707713603973 (0.0327677042504274) | > loss_mel_ce: 2.9534080028533936 (nan) | > loss: 0.011826606467366219 (nan) | > current_lr: 5e-06 | > step_time: 0.1819 (0.13675397390915323) | > loader_time: 0.0048 (0.004670367750041311)  --> TIME: 2025-02-25 23:32:43 -- STEP: 31762/36938 -- GLOBAL_STEP: 992150 | > loss_text_ce: 0.033016033470630646 (0.03277050188976757) | > loss_mel_ce: 2.631218194961548 (nan) | > loss: 0.010572358034551144 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.13675502586497087) | > loader_time: 0.0047 (0.004670390733865769)  --> TIME: 2025-02-25 23:32:53 -- STEP: 31812/36938 -- GLOBAL_STEP: 992200 | > loss_text_ce: 0.03169924020767212 (0.03276980564347907) | > loss_mel_ce: 0.8645574450492859 (nan) | > loss: 0.003556574461981654 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13676188296527292) | > loader_time: 0.0046 (0.0046705238537025255)  --> TIME: 2025-02-25 23:33:02 -- STEP: 31862/36938 -- GLOBAL_STEP: 992250 | > loss_text_ce: 0.0382087416946888 (0.03276783137915887) | > loss_mel_ce: 1.7801061868667603 (nan) | > loss: 0.00721553573384881 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.13676119073697177) | > loader_time: 0.0047 (0.004670572433533949)  --> TIME: 2025-02-25 23:33:12 -- STEP: 31912/36938 -- GLOBAL_STEP: 992300 | > loss_text_ce: 0.032177504152059555 (0.03276813995389865) | > loss_mel_ce: 2.7261810302734375 (nan) | > loss: 0.010945867747068405 (nan) | > current_lr: 5e-06 | > step_time: 0.1495 (0.13676233406980393) | > loader_time: 0.0049 (0.004670619949657368)  --> TIME: 2025-02-25 23:33:21 -- STEP: 31962/36938 -- GLOBAL_STEP: 992350 | > loss_text_ce: 0.029353434219956398 (0.03276732177800844) | > loss_mel_ce: 2.020622491836548 (nan) | > loss: 0.00813482515513897 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13675808554469898) | > loader_time: 0.0046 (0.004670765811544699)  --> TIME: 2025-02-25 23:33:30 -- STEP: 32012/36938 -- GLOBAL_STEP: 992400 | > loss_text_ce: 0.030591359362006187 (0.03276683230949905) | > loss_mel_ce: 2.4895565509796143 (nan) | > loss: 0.010000587441027164 (nan) | > current_lr: 5e-06 | > step_time: 0.1522 (0.13676566161112194) | > loader_time: 0.0049 (0.004670850242746792)  --> TIME: 2025-02-25 23:33:40 -- STEP: 32062/36938 -- GLOBAL_STEP: 992450 | > loss_text_ce: 0.0359339639544487 (0.03276625281549211) | > loss_mel_ce: 1.9982774257659912 (nan) | > loss: 0.008072268217802048 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.13676107853351968) | > loader_time: 0.0047 (0.004670895318650072)  --> TIME: 2025-02-25 23:33:49 -- STEP: 32112/36938 -- GLOBAL_STEP: 992500 | > loss_text_ce: 0.04034245386719704 (0.032766001334580616) | > loss_mel_ce: 2.297659397125244 (nan) | > loss: 0.009277786128222942 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.1367602229608393) | > loader_time: 0.0044 (0.004670908321001438)  --> TIME: 2025-02-25 23:33:58 -- STEP: 32162/36938 -- GLOBAL_STEP: 992550 | > loss_text_ce: 0.02084130235016346 (0.032767139205636384) | > loss_mel_ce: 2.8914380073547363 (nan) | > loss: 0.011556664481759071 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.1367603651616335) | > loader_time: 0.0046 (0.004670931750154575)  --> TIME: 2025-02-25 23:34:07 -- STEP: 32212/36938 -- GLOBAL_STEP: 992600 | > loss_text_ce: 0.038189396262168884 (0.03276574250277598) | > loss_mel_ce: 1.954788088798523 (nan) | > loss: 0.00790864136070013 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13675878087319562) | > loader_time: 0.0046 (0.00467095392972776)  --> TIME: 2025-02-25 23:34:17 -- STEP: 32262/36938 -- GLOBAL_STEP: 992650 | > loss_text_ce: 0.03906215727329254 (0.0327677697987248) | > loss_mel_ce: 3.0800862312316895 (nan) | > loss: 0.012377574108541012 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13675584365796947) | > loader_time: 0.0046 (0.004671028051896025)  --> TIME: 2025-02-25 23:34:26 -- STEP: 32312/36938 -- GLOBAL_STEP: 992700 | > loss_text_ce: 0.03034960851073265 (0.03276877788722845) | > loss_mel_ce: 0.9304989576339722 (nan) | > loss: 0.0038128914311528206 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13675036720192046) | > loader_time: 0.0046 (0.0046709578841238565)  --> TIME: 2025-02-25 23:34:35 -- STEP: 32362/36938 -- GLOBAL_STEP: 992750 | > loss_text_ce: 0.0327143631875515 (0.032768150069191246) | > loss_mel_ce: 3.429445266723633 (nan) | > loss: 0.013738729059696198 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.13675064022730904) | > loader_time: 0.0048 (0.004670979390074518)  --> TIME: 2025-02-25 23:34:44 -- STEP: 32412/36938 -- GLOBAL_STEP: 992800 | > loss_text_ce: 0.03051804192364216 (0.03276755670720007) | > loss_mel_ce: 2.7233197689056396 (nan) | > loss: 0.010927928611636162 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.1367524811460364) | > loader_time: 0.0046 (0.0046711160373676085)  --> TIME: 2025-02-25 23:34:54 -- STEP: 32462/36938 -- GLOBAL_STEP: 992850 | > loss_text_ce: 0.02731645293533802 (0.032769835386630815) | > loss_mel_ce: 1.7482188940048218 (nan) | > loss: 0.007045775651931763 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.1367603346744936) | > loader_time: 0.0048 (0.004671218199721411)  --> TIME: 2025-02-25 23:35:03 -- STEP: 32512/36938 -- GLOBAL_STEP: 992900 | > loss_text_ce: 0.025474736467003822 (0.03276946579688701) | > loss_mel_ce: 1.8704664707183838 (nan) | > loss: 0.007523576728999615 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1367574659520856) | > loader_time: 0.0045 (0.0046711883133673405)  --> TIME: 2025-02-25 23:35:13 -- STEP: 32562/36938 -- GLOBAL_STEP: 992950 | > loss_text_ce: 0.032106656581163406 (0.03276657377255939) | > loss_mel_ce: 2.3946194648742676 (nan) | > loss: 0.009629866108298302 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1367635654001731) | > loader_time: 0.0048 (0.0046711986652573775)  --> TIME: 2025-02-25 23:35:22 -- STEP: 32612/36938 -- GLOBAL_STEP: 993000 | > loss_text_ce: 0.03223596513271332 (0.03276641464503888) | > loss_mel_ce: 2.756382942199707 (nan) | > loss: 0.011065947823226452 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13675487434792621) | > loader_time: 0.0046 (0.004671166809617307)  --> TIME: 2025-02-25 23:35:31 -- STEP: 32662/36938 -- GLOBAL_STEP: 993050 | > loss_text_ce: 0.03857564181089401 (0.0327660842282207) | > loss_mel_ce: 1.9576267004013062 (nan) | > loss: 0.007921438664197922 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13675309447297554) | > loader_time: 0.0045 (0.004671122474348073)  --> TIME: 2025-02-25 23:35:40 -- STEP: 32712/36938 -- GLOBAL_STEP: 993100 | > loss_text_ce: 0.03403186798095703 (0.032766209675189895) | > loss_mel_ce: 3.139314651489258 (nan) | > loss: 0.012592645362019539 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13675509395037505) | > loader_time: 0.0046 (0.0046711444242509715)  --> TIME: 2025-02-25 23:35:50 -- STEP: 32762/36938 -- GLOBAL_STEP: 993150 | > loss_text_ce: 0.02585209533572197 (0.03276395294428141) | > loss_mel_ce: 3.1425819396972656 (nan) | > loss: 0.012573151849210262 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13675400708069932) | > loader_time: 0.0047 (0.004671158607782862)  --> TIME: 2025-02-25 23:36:00 -- STEP: 32812/36938 -- GLOBAL_STEP: 993200 | > loss_text_ce: 0.036292459815740585 (0.03276211430138602) | > loss_mel_ce: 2.9223735332489014 (nan) | > loss: 0.011740739457309246 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13676820704664866) | > loader_time: 0.0046 (0.004671223299046729)  --> TIME: 2025-02-25 23:36:09 -- STEP: 32862/36938 -- GLOBAL_STEP: 993250 | > loss_text_ce: 0.023486247286200523 (0.0327618338873515) | > loss_mel_ce: 1.7135401964187622 (nan) | > loss: 0.006892962381243706 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.136773914793422) | > loader_time: 0.0045 (0.004671207834498817)  --> TIME: 2025-02-25 23:36:19 -- STEP: 32912/36938 -- GLOBAL_STEP: 993300 | > loss_text_ce: 0.03274992108345032 (0.03275966168750059) | > loss_mel_ce: 1.9687519073486328 (nan) | > loss: 0.00794246792793274 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13677635654901485) | > loader_time: 0.0048 (0.004671174712301644)  --> TIME: 2025-02-25 23:36:28 -- STEP: 32962/36938 -- GLOBAL_STEP: 993350 | > loss_text_ce: 0.0376221239566803 (0.0327607934399862) | > loss_mel_ce: 2.776824951171875 (nan) | > loss: 0.011168441735208035 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13678525283450998) | > loader_time: 0.0045 (0.004671205812327741)  --> TIME: 2025-02-25 23:36:38 -- STEP: 33012/36938 -- GLOBAL_STEP: 993400 | > loss_text_ce: 0.043319448828697205 (0.03276191559732723) | > loss_mel_ce: 3.7713541984558105 (nan) | > loss: 0.015137595124542713 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.1367952985632417) | > loader_time: 0.0046 (0.004671350813021774)  --> TIME: 2025-02-25 23:36:47 -- STEP: 33062/36938 -- GLOBAL_STEP: 993450 | > loss_text_ce: 0.02749207243323326 (0.03276091463630399) | > loss_mel_ce: 3.278095006942749 (nan) | > loss: 0.013117409311234951 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13679062455530416) | > loader_time: 0.0045 (0.004671354149886563)  --> TIME: 2025-02-25 23:36:57 -- STEP: 33112/36938 -- GLOBAL_STEP: 993500 | > loss_text_ce: 0.02556457370519638 (0.03276073787025906) | > loss_mel_ce: 2.8210463523864746 (nan) | > loss: 0.011296075768768787 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13679725767679843) | > loader_time: 0.0046 (0.0046713604000231754)  --> TIME: 2025-02-25 23:37:06 -- STEP: 33162/36938 -- GLOBAL_STEP: 993550 | > loss_text_ce: 0.041427090764045715 (0.03276150213442966) | > loss_mel_ce: 2.3069989681243896 (nan) | > loss: 0.009319151751697063 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.1367922069712813) | > loader_time: 0.0045 (0.004671242993283891)  --> TIME: 2025-02-25 23:37:15 -- STEP: 33212/36938 -- GLOBAL_STEP: 993600 | > loss_text_ce: 0.027163241058588028 (0.0327633764860294) | > loss_mel_ce: 2.8051164150238037 (nan) | > loss: 0.011239205487072468 (nan) | > current_lr: 5e-06 | > step_time: 0.1918 (0.1367860926384978) | > loader_time: 0.0049 (0.00467115086445562)  --> TIME: 2025-02-25 23:37:24 -- STEP: 33262/36938 -- GLOBAL_STEP: 993650 | > loss_text_ce: 0.0423293299973011 (0.032764903758749256) | > loss_mel_ce: 3.0279226303100586 (nan) | > loss: 0.012183540500700474 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.1367892839454034) | > loader_time: 0.0046 (0.00467102993244977)  --> TIME: 2025-02-25 23:37:34 -- STEP: 33312/36938 -- GLOBAL_STEP: 993700 | > loss_text_ce: 0.034910377115011215 (0.032765052888111344) | > loss_mel_ce: 2.683382034301758 (nan) | > loss: 0.010786875151097775 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.13678625892350577) | > loader_time: 0.0046 (0.004670914774268561)  --> TIME: 2025-02-25 23:37:43 -- STEP: 33362/36938 -- GLOBAL_STEP: 993750 | > loss_text_ce: 0.02647937275469303 (0.03276574257240908) | > loss_mel_ce: 2.1150054931640625 (nan) | > loss: 0.00849795714020729 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.1367888111096572) | > loader_time: 0.0048 (0.004670800704491834)  --> TIME: 2025-02-25 23:37:52 -- STEP: 33412/36938 -- GLOBAL_STEP: 993800 | > loss_text_ce: 0.03628167882561684 (0.032765668135738155) | > loss_mel_ce: 3.083465576171875 (nan) | > loss: 0.012379949912428856 (nan) | > current_lr: 5e-06 | > step_time: 0.1858 (0.13678952010109385) | > loader_time: 0.0048 (0.00467070011297387)  --> TIME: 2025-02-25 23:38:02 -- STEP: 33462/36938 -- GLOBAL_STEP: 993850 | > loss_text_ce: 0.029299693182110786 (0.03276303807164643) | > loss_mel_ce: 2.4242537021636963 (nan) | > loss: 0.009736323729157448 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13678299198660437) | > loader_time: 0.0045 (0.004670636045847991)  --> TIME: 2025-02-25 23:38:11 -- STEP: 33512/36938 -- GLOBAL_STEP: 993900 | > loss_text_ce: 0.02986173890531063 (0.032762831248476315) | > loss_mel_ce: 2.846543788909912 (nan) | > loss: 0.011414308100938797 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.1367876238500249) | > loader_time: 0.0047 (0.004670612579827444)  --> TIME: 2025-02-25 23:38:20 -- STEP: 33562/36938 -- GLOBAL_STEP: 993950 | > loss_text_ce: 0.03126010671257973 (0.03276240006346185) | > loss_mel_ce: 1.5780242681503296 (nan) | > loss: 0.006386049557477236 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13678525936389943) | > loader_time: 0.0045 (0.004670538164053216)  --> TIME: 2025-02-25 23:38:30 -- STEP: 33612/36938 -- GLOBAL_STEP: 994000 | > loss_text_ce: 0.04509182646870613 (0.03276177817571818) | > loss_mel_ce: 2.6859052181243896 (nan) | > loss: 0.010837290436029434 (nan) | > current_lr: 5e-06 | > step_time: 0.1739 (0.13678567242540324) | > loader_time: 0.0049 (0.004670539158207461)  --> TIME: 2025-02-25 23:38:39 -- STEP: 33662/36938 -- GLOBAL_STEP: 994050 | > loss_text_ce: 0.03268047422170639 (0.03276311351603018) | > loss_mel_ce: 2.900317430496216 (nan) | > loss: 0.011638881638646126 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.1367968512505031) | > loader_time: 0.0046 (0.004670731304985318)  --> TIME: 2025-02-25 23:38:49 -- STEP: 33712/36938 -- GLOBAL_STEP: 994100 | > loss_text_ce: 0.03844922035932541 (0.032765221266281896) | > loss_mel_ce: 1.2702515125274658 (nan) | > loss: 0.0051932567730546 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.1367944802471876) | > loader_time: 0.0044 (0.004670765588619617)  --> TIME: 2025-02-25 23:38:58 -- STEP: 33762/36938 -- GLOBAL_STEP: 994150 | > loss_text_ce: 0.04195641353726387 (0.032766431881983714) | > loss_mel_ce: 2.6254007816314697 (nan) | > loss: 0.010584751144051552 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1367913626655664) | > loader_time: 0.0045 (0.004670783217982463)  --> TIME: 2025-02-25 23:39:07 -- STEP: 33812/36938 -- GLOBAL_STEP: 994200 | > loss_text_ce: 0.023617301136255264 (0.032765249050400215) | > loss_mel_ce: 2.133152723312378 (nan) | > loss: 0.008558611385524273 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13679209832791114) | > loader_time: 0.0048 (0.004670753452773075)  --> TIME: 2025-02-25 23:39:16 -- STEP: 33862/36938 -- GLOBAL_STEP: 994250 | > loss_text_ce: 0.025032324716448784 (0.03276378428588245) | > loss_mel_ce: 2.4226937294006348 (nan) | > loss: 0.009713198989629745 (nan) | > current_lr: 5e-06 | > step_time: 0.171 (0.13679091338673338) | > loader_time: 0.0049 (0.004670711841158553)  --> TIME: 2025-02-25 23:39:26 -- STEP: 33912/36938 -- GLOBAL_STEP: 994300 | > loss_text_ce: 0.037111151963472366 (0.0327631021469396) | > loss_mel_ce: 3.0388355255126953 (nan) | > loss: 0.012206138111650944 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13680017352329393) | > loader_time: 0.0046 (0.004670669993692825)  --> TIME: 2025-02-25 23:39:36 -- STEP: 33962/36938 -- GLOBAL_STEP: 994350 | > loss_text_ce: 0.03099571168422699 (0.03276279360266761) | > loss_mel_ce: 3.3775153160095215 (nan) | > loss: 0.013525838032364845 (nan) | > current_lr: 5e-06 | > step_time: 0.1789 (0.13680340792207613) | > loader_time: 0.0048 (0.004670620912320838)  --> TIME: 2025-02-25 23:39:45 -- STEP: 34012/36938 -- GLOBAL_STEP: 994400 | > loss_text_ce: 0.028410473838448524 (0.03276346798201546) | > loss_mel_ce: 0.01104736328125 (nan) | > loss: 0.00015657872427254915 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.1367981279932409) | > loader_time: 0.0045 (0.0046705589089185585)  --> TIME: 2025-02-25 23:39:54 -- STEP: 34062/36938 -- GLOBAL_STEP: 994450 | > loss_text_ce: 0.04170825332403183 (0.032763087981691315) | > loss_mel_ce: 2.334193468093872 (nan) | > loss: 0.009428181685507298 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13679713946977645) | > loader_time: 0.0047 (0.004670499530389289)  --> TIME: 2025-02-25 23:40:03 -- STEP: 34112/36938 -- GLOBAL_STEP: 994500 | > loss_text_ce: 0.04315296560525894 (0.03276446616179044) | > loss_mel_ce: 2.838726043701172 (nan) | > loss: 0.01143602840602398 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.1367996693510187) | > loader_time: 0.0046 (0.0046704465184381734)  --> TIME: 2025-02-25 23:40:12 -- STEP: 34162/36938 -- GLOBAL_STEP: 994550 | > loss_text_ce: 0.027713732793927193 (0.032764039800152626) | > loss_mel_ce: 2.7617833614349365 (nan) | > loss: 0.011069433763623238 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.1367943427384213) | > loader_time: 0.0046 (0.004670359701568997)  --> TIME: 2025-02-25 23:40:22 -- STEP: 34212/36938 -- GLOBAL_STEP: 994600 | > loss_text_ce: 0.024380013346672058 (0.03276384944862469) | > loss_mel_ce: 3.2865307331085205 (nan) | > loss: 0.013138535432517529 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13679402422015982) | > loader_time: 0.0048 (0.004670291738343998)  --> TIME: 2025-02-25 23:40:31 -- STEP: 34262/36938 -- GLOBAL_STEP: 994650 | > loss_text_ce: 0.04081602022051811 (0.03276235514589172) | > loss_mel_ce: 1.939161777496338 (nan) | > loss: 0.007857055403292179 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13679202450086744) | > loader_time: 0.0049 (0.004670239630530985)  --> TIME: 2025-02-25 23:40:41 -- STEP: 34312/36938 -- GLOBAL_STEP: 994700 | > loss_text_ce: 0.0198521725833416 (0.032761837402179646) | > loss_mel_ce: 1.1845899820327759 (nan) | > loss: 0.004779532551765442 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13679732149173968) | > loader_time: 0.0045 (0.004670254276415278)  --> TIME: 2025-02-25 23:40:50 -- STEP: 34362/36938 -- GLOBAL_STEP: 994750 | > loss_text_ce: 0.030457250773906708 (0.03276308740569805) | > loss_mel_ce: 2.464204788208008 (nan) | > loss: 0.00989945326000452 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13679487465438564) | > loader_time: 0.0046 (0.004670206454557747)  --> TIME: 2025-02-25 23:40:59 -- STEP: 34412/36938 -- GLOBAL_STEP: 994800 | > loss_text_ce: 0.026718564331531525 (0.03276357962863579) | > loss_mel_ce: 1.0404607057571411 (nan) | > loss: 0.004234838765114546 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.1367934067055606) | > loader_time: 0.0045 (0.004670209681227037)  --> TIME: 2025-02-25 23:41:09 -- STEP: 34462/36938 -- GLOBAL_STEP: 994850 | > loss_text_ce: 0.039518870413303375 (0.03276388291405216) | > loss_mel_ce: 2.622223377227783 (nan) | > loss: 0.010562469251453876 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.1367986451783227) | > loader_time: 0.0046 (0.004670179718349226)  --> TIME: 2025-02-25 23:41:18 -- STEP: 34512/36938 -- GLOBAL_STEP: 994900 | > loss_text_ce: 0.029112571850419044 (0.03276358828397943) | > loss_mel_ce: 2.7279701232910156 (nan) | > loss: 0.010940805077552795 (nan) | > current_lr: 5e-06 | > step_time: 0.1609 (0.13680215312559438) | > loader_time: 0.0046 (0.004670159354992482)  --> TIME: 2025-02-25 23:41:28 -- STEP: 34562/36938 -- GLOBAL_STEP: 994950 | > loss_text_ce: 0.03429313749074936 (0.03276452919441134) | > loss_mel_ce: 2.87347412109375 (nan) | > loss: 0.011538759805262089 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13680120499816836) | > loader_time: 0.0045 (0.004670088286063493)  --> TIME: 2025-02-25 23:41:37 -- STEP: 34612/36938 -- GLOBAL_STEP: 995000 | > loss_text_ce: 0.03807883709669113 (0.03276253134090481) | > loss_mel_ce: 1.3914031982421875 (nan) | > loss: 0.0056725479662418365 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.1368037986642429) | > loader_time: 0.0045 (0.004670021954980279)  --> TIME: 2025-02-25 23:41:46 -- STEP: 34662/36938 -- GLOBAL_STEP: 995050 | > loss_text_ce: 0.03021974302828312 (0.03276273426685521) | > loss_mel_ce: 2.4028613567352295 (nan) | > loss: 0.00965508446097374 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.13680884062360255) | > loader_time: 0.0047 (0.004669980384853524)  --> TIME: 2025-02-25 23:41:56 -- STEP: 34712/36938 -- GLOBAL_STEP: 995100 | > loss_text_ce: 0.032230328768491745 (0.03276279007042715) | > loss_mel_ce: 2.8811097145080566 (nan) | > loss: 0.011560874059796333 (nan) | > current_lr: 5e-06 | > step_time: 0.1441 (0.13682024560372752) | > loader_time: 0.0048 (0.004669946682126223)  --> TIME: 2025-02-25 23:42:05 -- STEP: 34762/36938 -- GLOBAL_STEP: 995150 | > loss_text_ce: 0.03698215261101723 (0.032763306526263136) | > loss_mel_ce: 2.9362127780914307 (nan) | > loss: 0.011798392981290817 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.1368116450442836) | > loader_time: 0.0047 (0.004669845210525805)  --> TIME: 2025-02-25 23:42:15 -- STEP: 34812/36938 -- GLOBAL_STEP: 995200 | > loss_text_ce: 0.036512065678834915 (0.03276400746936361) | > loss_mel_ce: 2.619943857192993 (nan) | > loss: 0.010541493073105812 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.1368167714238206) | > loader_time: 0.0046 (0.004669892449605258)  --> TIME: 2025-02-25 23:42:24 -- STEP: 34862/36938 -- GLOBAL_STEP: 995250 | > loss_text_ce: 0.03206774219870567 (0.03276145909085544) | > loss_mel_ce: 2.7623276710510254 (nan) | > loss: 0.011088871397078037 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.1368093961186993) | > loader_time: 0.0046 (0.0046698655628595945)  --> TIME: 2025-02-25 23:42:33 -- STEP: 34912/36938 -- GLOBAL_STEP: 995300 | > loss_text_ce: 0.034310489892959595 (0.03275981565997057) | > loss_mel_ce: 3.073904275894165 (nan) | > loss: 0.0123341865837574 (nan) | > current_lr: 5e-06 | > step_time: 0.1905 (0.1368002116366032) | > loader_time: 0.0048 (0.004669775057833317)  --> TIME: 2025-02-25 23:42:42 -- STEP: 34962/36938 -- GLOBAL_STEP: 995350 | > loss_text_ce: 0.02660689502954483 (0.032760751763264304) | > loss_mel_ce: 1.7139533758163452 (nan) | > loss: 0.006906985770910978 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1368039413731715) | > loader_time: 0.0047 (0.004669758569907156)  --> TIME: 2025-02-25 23:42:52 -- STEP: 35012/36938 -- GLOBAL_STEP: 995400 | > loss_text_ce: 0.03181871399283409 (0.03276012451235801) | > loss_mel_ce: 2.5886497497558594 (nan) | > loss: 0.01039868500083685 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13681207767666836) | > loader_time: 0.0045 (0.004669843939771716)  --> TIME: 2025-02-25 23:43:01 -- STEP: 35062/36938 -- GLOBAL_STEP: 995450 | > loss_text_ce: 0.03474156931042671 (0.032759006917607456) | > loss_mel_ce: 0.01309769507497549 (nan) | > loss: 0.000189838363439776 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13681506218447292) | > loader_time: 0.0045 (0.004669887491480465)  --> TIME: 2025-02-25 23:43:11 -- STEP: 35112/36938 -- GLOBAL_STEP: 995500 | > loss_text_ce: 0.027673596516251564 (0.03275992181922914) | > loss_mel_ce: 3.1646997928619385 (nan) | > loss: 0.01266814861446619 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.1368174509799544) | > loader_time: 0.0049 (0.0046700033233822566)  --> TIME: 2025-02-25 23:43:20 -- STEP: 35162/36938 -- GLOBAL_STEP: 995550 | > loss_text_ce: 0.041210222989320755 (0.03276063751574582) | > loss_mel_ce: 2.4635977745056152 (nan) | > loss: 0.009939714334905148 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13682188384803484) | > loader_time: 0.0051 (0.004670096761869382)  --> TIME: 2025-02-25 23:43:30 -- STEP: 35212/36938 -- GLOBAL_STEP: 995600 | > loss_text_ce: 0.04161142557859421 (0.032760902756635187) | > loss_mel_ce: 3.5061256885528564 (nan) | > loss: 0.014078322798013687 (nan) | > current_lr: 5e-06 | > step_time: 0.1818 (0.1368301457658947) | > loader_time: 0.0049 (0.004670154888576029)  --> TIME: 2025-02-25 23:43:39 -- STEP: 35262/36938 -- GLOBAL_STEP: 995650 | > loss_text_ce: 0.0424044132232666 (0.03276225648076978) | > loss_mel_ce: 2.1580145359039307 (nan) | > loss: 0.008731821551918983 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.1368223402897882) | > loader_time: 0.0046 (0.004670055013581661)  --> TIME: 2025-02-25 23:43:49 -- STEP: 35312/36938 -- GLOBAL_STEP: 995700 | > loss_text_ce: 0.030227910727262497 (0.032762841821291064) | > loss_mel_ce: 1.7929328680038452 (nan) | > loss: 0.007234765216708183 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.1368297388921482) | > loader_time: 0.0045 (0.004670019867088564)  --> TIME: 2025-02-25 23:43:58 -- STEP: 35362/36938 -- GLOBAL_STEP: 995750 | > loss_text_ce: 0.03309086337685585 (0.032763458642093964) | > loss_mel_ce: 2.5523383617401123 (nan) | > loss: 0.010259640403091908 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.1368323946611367) | > loader_time: 0.0047 (0.004670015247649842)  --> TIME: 2025-02-25 23:44:07 -- STEP: 35412/36938 -- GLOBAL_STEP: 995800 | > loss_text_ce: 0.03288697823882103 (0.03276406090524497) | > loss_mel_ce: 3.251650810241699 (nan) | > loss: 0.013033880852162838 (nan) | > current_lr: 5e-06 | > step_time: 0.1488 (0.13683104565571247) | > loader_time: 0.0047 (0.0046700291292661854)  --> TIME: 2025-02-25 23:44:17 -- STEP: 35462/36938 -- GLOBAL_STEP: 995850 | > loss_text_ce: 0.038182035088539124 (0.032765653569904386) | > loss_mel_ce: 2.4775850772857666 (nan) | > loss: 0.009983203373849392 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.13682578287178157) | > loader_time: 0.0047 (0.004670168063838401)  --> TIME: 2025-02-25 23:44:26 -- STEP: 35512/36938 -- GLOBAL_STEP: 995900 | > loss_text_ce: 0.03023306280374527 (0.03276461176702743) | > loss_mel_ce: 2.620762825012207 (nan) | > loss: 0.010519825853407383 (nan) | > current_lr: 5e-06 | > step_time: 0.1262 (0.13682821473578238) | > loader_time: 0.0046 (0.00467018931801993)  --> TIME: 2025-02-25 23:44:35 -- STEP: 35562/36938 -- GLOBAL_STEP: 995950 | > loss_text_ce: 0.028930097818374634 (0.032764480761350114) | > loss_mel_ce: 2.5619122982025146 (nan) | > loss: 0.010281121358275414 (nan) | > current_lr: 5e-06 | > step_time: 0.148 (0.13682661682835215) | > loader_time: 0.0047 (0.004670278856144356)  --> TIME: 2025-02-25 23:44:44 -- STEP: 35612/36938 -- GLOBAL_STEP: 996000 | > loss_text_ce: 0.03191390261054039 (0.03276531504507202) | > loss_mel_ce: 2.121803045272827 (nan) | > loss: 0.008546496741473675 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.1368222044623756) | > loader_time: 0.0049 (0.0046703266478864995)  --> TIME: 2025-02-25 23:44:54 -- STEP: 35662/36938 -- GLOBAL_STEP: 996050 | > loss_text_ce: 0.02613050490617752 (0.032762977483515636) | > loss_mel_ce: 2.772639274597168 (nan) | > loss: 0.01110622938722372 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.13682028450505382) | > loader_time: 0.0047 (0.00467041945953524)  --> TIME: 2025-02-25 23:45:03 -- STEP: 35712/36938 -- GLOBAL_STEP: 996100 | > loss_text_ce: 0.030399207025766373 (0.03276157346429452) | > loss_mel_ce: 2.1339218616485596 (nan) | > loss: 0.008588576689362526 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13681638049209033) | > loader_time: 0.0045 (0.004670499934143924)  --> TIME: 2025-02-25 23:45:12 -- STEP: 35762/36938 -- GLOBAL_STEP: 996150 | > loss_text_ce: 0.03389072045683861 (0.03276135715478434) | > loss_mel_ce: 1.0896352529525757 (nan) | > loss: 0.004458436742424965 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13681347915093287) | > loader_time: 0.0045 (0.004670508602149382)  --> TIME: 2025-02-25 23:45:21 -- STEP: 35812/36938 -- GLOBAL_STEP: 996200 | > loss_text_ce: 0.032265204936265945 (0.03276028400016059) | > loss_mel_ce: 2.4282827377319336 (nan) | > loss: 0.009764079935848713 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13681127160112105) | > loader_time: 0.0046 (0.0046704980057603864)  --> TIME: 2025-02-25 23:45:31 -- STEP: 35862/36938 -- GLOBAL_STEP: 996250 | > loss_text_ce: 0.033769406378269196 (0.032760069843418706) | > loss_mel_ce: 2.402566909790039 (nan) | > loss: 0.009668001905083656 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.13681039289758004) | > loader_time: 0.0047 (0.004670429001037463)  --> TIME: 2025-02-25 23:45:40 -- STEP: 35912/36938 -- GLOBAL_STEP: 996300 | > loss_text_ce: 0.029174374416470528 (0.03276069929986046) | > loss_mel_ce: 2.758415699005127 (nan) | > loss: 0.011061865836381912 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13681318838789383) | > loader_time: 0.0046 (0.00467042337815092)  --> TIME: 2025-02-25 23:45:49 -- STEP: 35962/36938 -- GLOBAL_STEP: 996350 | > loss_text_ce: 0.023167073726654053 (0.03276167583316891) | > loss_mel_ce: 2.888920307159424 (nan) | > loss: 0.01155590359121561 (nan) | > current_lr: 5e-06 | > step_time: 0.1468 (0.13681136851090595) | > loader_time: 0.0046 (0.004670396290554195)  --> TIME: 2025-02-25 23:45:58 -- STEP: 36012/36938 -- GLOBAL_STEP: 996400 | > loss_text_ce: 0.04063962399959564 (0.03276311567591227) | > loss_mel_ce: 2.5048511028289795 (nan) | > loss: 0.010101154446601868 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13679983479360117) | > loader_time: 0.0046 (0.0046703194652969815)  --> TIME: 2025-02-25 23:46:08 -- STEP: 36062/36938 -- GLOBAL_STEP: 996450 | > loss_text_ce: 0.029822632670402527 (0.03276434134292489) | > loss_mel_ce: 2.6640195846557617 (nan) | > loss: 0.010689850896596909 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13680047006048235) | > loader_time: 0.0048 (0.004670338572231933)  --> TIME: 2025-02-25 23:46:17 -- STEP: 36112/36938 -- GLOBAL_STEP: 996500 | > loss_text_ce: 0.03369978815317154 (0.03276486015882441) | > loss_mel_ce: 3.1276185512542725 (nan) | > loss: 0.012544915080070496 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.13680045937852967) | > loader_time: 0.0048 (0.0046703396484711526)  --> TIME: 2025-02-25 23:46:26 -- STEP: 36162/36938 -- GLOBAL_STEP: 996550 | > loss_text_ce: 0.020002014935016632 (0.03276193938392934) | > loss_mel_ce: 0.9889593124389648 (nan) | > loss: 0.00400381488725543 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13680265224861685) | > loader_time: 0.0045 (0.004670327561967495)  --> TIME: 2025-02-25 23:46:35 -- STEP: 36212/36938 -- GLOBAL_STEP: 996600 | > loss_text_ce: 0.017774121835827827 (0.032763890512912106) | > loss_mel_ce: 1.6582881212234497 (nan) | > loss: 0.0066510410979390144 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13679524640227142) | > loader_time: 0.0046 (0.0046703021697256635)  --> TIME: 2025-02-25 23:46:45 -- STEP: 36262/36938 -- GLOBAL_STEP: 996650 | > loss_text_ce: 0.027100838720798492 (0.03276214476983101) | > loss_mel_ce: 3.0029778480529785 (nan) | > loss: 0.012024122290313244 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.13679563739468212) | > loader_time: 0.0046 (0.004670413144930988)  --> TIME: 2025-02-25 23:46:54 -- STEP: 36312/36938 -- GLOBAL_STEP: 996700 | > loss_text_ce: 0.030126439407467842 (0.032763231264766525) | > loss_mel_ce: 0.0619913749396801 (nan) | > loss: 0.00036554690450429916 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.1367922289029041) | > loader_time: 0.0047 (0.004670597338734251)  --> TIME: 2025-02-25 23:47:03 -- STEP: 36362/36938 -- GLOBAL_STEP: 996750 | > loss_text_ce: 0.03373895585536957 (0.032763988656769424) | > loss_mel_ce: 2.710768938064575 (nan) | > loss: 0.010890904814004898 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13678897135289836) | > loader_time: 0.0044 (0.00467061330288475)  --> TIME: 2025-02-25 23:47:13 -- STEP: 36412/36938 -- GLOBAL_STEP: 996800 | > loss_text_ce: 0.02888641320168972 (0.03276506679669811) | > loss_mel_ce: 2.9104485511779785 (nan) | > loss: 0.011664028279483318 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.13679206166361405) | > loader_time: 0.0048 (0.004670556071144314)  --> TIME: 2025-02-25 23:47:22 -- STEP: 36462/36938 -- GLOBAL_STEP: 996850 | > loss_text_ce: 0.03484255447983742 (0.0327627430949907) | > loss_mel_ce: 2.953784942626953 (nan) | > loss: 0.011859633028507233 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13679340470151954) | > loader_time: 0.0045 (0.00467046618455072)  --> TIME: 2025-02-25 23:47:31 -- STEP: 36512/36938 -- GLOBAL_STEP: 996900 | > loss_text_ce: 0.03013492003083229 (0.0327630641244609) | > loss_mel_ce: 2.41628098487854 (nan) | > loss: 0.009708000347018242 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.1367923237259323) | > loader_time: 0.0046 (0.00467041284368293)  --> TIME: 2025-02-25 23:47:41 -- STEP: 36562/36938 -- GLOBAL_STEP: 996950 | > loss_text_ce: 0.031845077872276306 (0.032763060710911195) | > loss_mel_ce: 3.3276193141937256 (nan) | > loss: 0.013331208378076553 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.136793691013454) | > loader_time: 0.0046 (0.004670413792064881)  --> TIME: 2025-02-25 23:47:50 -- STEP: 36612/36938 -- GLOBAL_STEP: 997000 | > loss_text_ce: 0.032153286039829254 (0.032762371999485645) | > loss_mel_ce: 2.8622665405273438 (nan) | > loss: 0.011485793627798557 (nan) | > current_lr: 5e-06 | > step_time: 0.162 (0.1367940692566657) | > loader_time: 0.0048 (0.004670348797002874)  --> TIME: 2025-02-25 23:47:59 -- STEP: 36662/36938 -- GLOBAL_STEP: 997050 | > loss_text_ce: 0.04526155814528465 (0.03276215272884319) | > loss_mel_ce: 1.9628174304962158 (nan) | > loss: 0.007968568243086338 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13678775734496137) | > loader_time: 0.0045 (0.004670335406153124)  --> TIME: 2025-02-25 23:48:09 -- STEP: 36712/36938 -- GLOBAL_STEP: 997100 | > loss_text_ce: 0.027574118226766586 (0.03276340387920197) | > loss_mel_ce: 2.3736348152160645 (nan) | > loss: 0.00952860713005066 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.13679102815218686) | > loader_time: 0.0047 (0.004670411043111024)  --> TIME: 2025-02-25 23:48:18 -- STEP: 36762/36938 -- GLOBAL_STEP: 997150 | > loss_text_ce: 0.03617338836193085 (0.0327640469008671) | > loss_mel_ce: 2.0310516357421875 (nan) | > loss: 0.008203274570405483 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1367967548363121) | > loader_time: 0.0044 (0.004670407299792838)  --> TIME: 2025-02-25 23:48:28 -- STEP: 36812/36938 -- GLOBAL_STEP: 997200 | > loss_text_ce: 0.04015972465276718 (0.03276389350041021) | > loss_mel_ce: 2.9006898403167725 (nan) | > loss: 0.011670038104057312 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.1367928776730171) | > loader_time: 0.0048 (0.004670317563157795)  --> TIME: 2025-02-25 23:48:37 -- STEP: 36862/36938 -- GLOBAL_STEP: 997250 | > loss_text_ce: 0.031295400112867355 (0.03276262051995801) | > loss_mel_ce: 1.477331519126892 (nan) | > loss: 0.0059866150841116905 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13678875224187337) | > loader_time: 0.0045 (0.004670371404402277)  --> TIME: 2025-02-25 23:48:46 -- STEP: 36912/36938 -- GLOBAL_STEP: 997300 | > loss_text_ce: 0.030377671122550964 (0.03276367299046765) | > loss_mel_ce: 2.283440113067627 (nan) | > loss: 0.009181817062199116 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13679039092047612) | > loader_time: 0.0045 (0.004670375351736359)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.021167687850423368 (+0.00018838844677007) | > avg_loss_text_ce: 0.03191479372948702 (-0.00023512130068373965) | > avg_loss_mel_ce: 2.49713953060679 (-0.0035226820128961833) | > avg_loss: 2.529054323635479 (-0.0037578037469674364) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_997326.pth  > EPOCH: 28/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-25 23:49:16)   --> TIME: 2025-02-25 23:49:21 -- STEP: 24/36938 -- GLOBAL_STEP: 997350 | > loss_text_ce: 0.03909916430711746 (0.03247912429894011) | > loss_mel_ce: 2.930197238922119 (2.3186307259214414) | > loss: 0.01178292278200388 (0.009329801624213966) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13216692209243774) | > loader_time: 0.0046 (0.004620283842086792)  --> TIME: 2025-02-25 23:49:31 -- STEP: 74/36938 -- GLOBAL_STEP: 997400 | > loss_text_ce: 0.043026432394981384 (0.03254981643545467) | > loss_mel_ce: 2.337181806564331 (2.473905504001556) | > loss: 0.009445271454751492 (0.009946251925784227) | > current_lr: 5e-06 | > step_time: 0.1196 (0.1340588395659988) | > loader_time: 0.0047 (0.0047031318819200675)  --> TIME: 2025-02-25 23:49:40 -- STEP: 124/36938 -- GLOBAL_STEP: 997450 | > loss_text_ce: 0.03325766697525978 (0.0330888110723707) | > loss_mel_ce: 2.2768373489379883 (nan) | > loss: 0.009167044423520565 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.1333068955329157) | > loader_time: 0.0048 (0.0047063231468200675)  --> TIME: 2025-02-25 23:49:49 -- STEP: 174/36938 -- GLOBAL_STEP: 997500 | > loss_text_ce: 0.03519773110747337 (0.03301465824974337) | > loss_mel_ce: 2.4819769859313965 (nan) | > loss: 0.009988789446651936 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13309678538092262) | > loader_time: 0.0047 (0.004712820053100584)  --> TIME: 2025-02-25 23:49:58 -- STEP: 224/36938 -- GLOBAL_STEP: 997550 | > loss_text_ce: 0.030553894117474556 (0.03319464209822141) | > loss_mel_ce: 3.238131046295166 (nan) | > loss: 0.012970972806215286 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13378503599337174) | > loader_time: 0.0048 (0.0047348524842943445)  --> TIME: 2025-02-25 23:50:07 -- STEP: 274/36938 -- GLOBAL_STEP: 997600 | > loss_text_ce: 0.0390661358833313 (0.033219470968809874) | > loss_mel_ce: 2.8262171745300293 (nan) | > loss: 0.01137017272412777 (nan) | > current_lr: 5e-06 | > step_time: 0.1619 (0.13475738389648664) | > loader_time: 0.0047 (0.0047427702994242175)  --> TIME: 2025-02-25 23:50:17 -- STEP: 324/36938 -- GLOBAL_STEP: 997650 | > loss_text_ce: 0.03285343945026398 (0.033022936789985795) | > loss_mel_ce: 3.2907042503356934 (nan) | > loss: 0.013188721612095833 (nan) | > current_lr: 5e-06 | > step_time: 0.1454 (0.13633514260068358) | > loader_time: 0.0048 (0.004735341778507937)  --> TIME: 2025-02-25 23:50:27 -- STEP: 374/36938 -- GLOBAL_STEP: 997700 | > loss_text_ce: 0.03392559289932251 (0.03304644524652692) | > loss_mel_ce: 3.0775527954101562 (nan) | > loss: 0.012347136624157429 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13668923390740384) | > loader_time: 0.0047 (0.004735506154636645)  --> TIME: 2025-02-25 23:50:36 -- STEP: 424/36938 -- GLOBAL_STEP: 997750 | > loss_text_ce: 0.029507257044315338 (0.03293532242049586) | > loss_mel_ce: 2.16025972366333 (nan) | > loss: 0.00868955161422491 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13642986540524465) | > loader_time: 0.0045 (0.004728855389469075)  --> TIME: 2025-02-25 23:50:45 -- STEP: 474/36938 -- GLOBAL_STEP: 997800 | > loss_text_ce: 0.02396036498248577 (0.0329774070711893) | > loss_mel_ce: 2.813912868499756 (nan) | > loss: 0.011261402629315853 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13631951356235933) | > loader_time: 0.0046 (0.004720748225344887)  --> TIME: 2025-02-25 23:50:55 -- STEP: 524/36938 -- GLOBAL_STEP: 997850 | > loss_text_ce: 0.03771310672163963 (0.033003663082557345) | > loss_mel_ce: 2.7809858322143555 (nan) | > loss: 0.011185313574969769 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1372310619317848) | > loader_time: 0.0045 (0.004719900265904784)  --> TIME: 2025-02-25 23:51:05 -- STEP: 574/36938 -- GLOBAL_STEP: 997900 | > loss_text_ce: 0.0263972207903862 (0.03296691317189136) | > loss_mel_ce: 1.307915210723877 (nan) | > loss: 0.005294891074299812 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.1375006852665015) | > loader_time: 0.0046 (0.004716439529578445)  --> TIME: 2025-02-25 23:51:14 -- STEP: 624/36938 -- GLOBAL_STEP: 997950 | > loss_text_ce: 0.030250953510403633 (0.03298743718411204) | > loss_mel_ce: 2.5297157764434814 (nan) | > loss: 0.010158599354326725 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.13738308044580322) | > loader_time: 0.0047 (0.0047113574468172536)  --> TIME: 2025-02-25 23:51:23 -- STEP: 674/36938 -- GLOBAL_STEP: 998000 | > loss_text_ce: 0.028798190876841545 (0.0329869693789068) | > loss_mel_ce: 2.7259881496429443 (nan) | > loss: 0.010931692086160183 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.137441062078278) | > loader_time: 0.0046 (0.004706272739331165)  --> TIME: 2025-02-25 23:51:33 -- STEP: 724/36938 -- GLOBAL_STEP: 998050 | > loss_text_ce: 0.029809851199388504 (0.032955536576379056) | > loss_mel_ce: 3.059044122695923 (nan) | > loss: 0.012257358059287071 (nan) | > current_lr: 5e-06 | > step_time: 0.1343 (0.13759272335642622) | > loader_time: 0.0047 (0.004703937973106768)  --> TIME: 2025-02-25 23:51:42 -- STEP: 774/36938 -- GLOBAL_STEP: 998100 | > loss_text_ce: 0.02499959245324135 (0.032862263537933784) | > loss_mel_ce: 2.3443992137908936 (nan) | > loss: 0.009402376599609852 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.1373851496427868) | > loader_time: 0.0047 (0.004708216787924757)  --> TIME: 2025-02-25 23:51:51 -- STEP: 824/36938 -- GLOBAL_STEP: 998150 | > loss_text_ce: 0.033465676009655 (0.03288315979252996) | > loss_mel_ce: 2.953301429748535 (nan) | > loss: 0.011852250434458256 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.1371626364953312) | > loader_time: 0.0048 (0.004705102408973918)  --> TIME: 2025-02-25 23:52:01 -- STEP: 874/36938 -- GLOBAL_STEP: 998200 | > loss_text_ce: 0.03645767271518707 (0.03288447780713646) | > loss_mel_ce: 2.524138927459717 (nan) | > loss: 0.010161099024116993 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13726935533964418) | > loader_time: 0.0046 (0.004702344093497342)  --> TIME: 2025-02-25 23:52:10 -- STEP: 924/36938 -- GLOBAL_STEP: 998250 | > loss_text_ce: 0.03424079343676567 (0.03286178991726695) | > loss_mel_ce: 2.9709291458129883 (nan) | > loss: 0.01192527823150158 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.13715035574776796) | > loader_time: 0.0048 (0.00469810080218625)  --> TIME: 2025-02-25 23:52:19 -- STEP: 974/36938 -- GLOBAL_STEP: 998300 | > loss_text_ce: 0.037763528525829315 (0.03284187987733644) | > loss_mel_ce: 2.6112630367279053 (nan) | > loss: 0.010512011125683784 (nan) | > current_lr: 5e-06 | > step_time: 0.1579 (0.1370854438942317) | > loader_time: 0.0046 (0.004693682188860447)  --> TIME: 2025-02-25 23:52:28 -- STEP: 1024/36938 -- GLOBAL_STEP: 998350 | > loss_text_ce: 0.02874286100268364 (0.032812863660183246) | > loss_mel_ce: 2.3770086765289307 (nan) | > loss: 0.009546633809804916 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13689390290528544) | > loader_time: 0.0046 (0.004690265050157906)  --> TIME: 2025-02-25 23:52:38 -- STEP: 1074/36938 -- GLOBAL_STEP: 998400 | > loss_text_ce: 0.03811164200305939 (0.0327877708974345) | > loss_mel_ce: 2.4475290775299072 (nan) | > loss: 0.009863654151558876 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13690073641096864) | > loader_time: 0.0046 (0.004689022593436074)  --> TIME: 2025-02-25 23:52:47 -- STEP: 1124/36938 -- GLOBAL_STEP: 998450 | > loss_text_ce: 0.031019696965813637 (0.032763841925719225) | > loss_mel_ce: 1.4161784648895264 (nan) | > loss: 0.005742850247770548 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13704317871786104) | > loader_time: 0.0046 (0.004688181062610127)  --> TIME: 2025-02-25 23:52:57 -- STEP: 1174/36938 -- GLOBAL_STEP: 998500 | > loss_text_ce: 0.03315212205052376 (0.03272660685468348) | > loss_mel_ce: 1.7145304679870605 (nan) | > loss: 0.006935248617082834 (nan) | > current_lr: 5e-06 | > step_time: 0.1614 (0.1370356984601517) | > loader_time: 0.0048 (0.00468697478864628)  --> TIME: 2025-02-25 23:53:06 -- STEP: 1224/36938 -- GLOBAL_STEP: 998550 | > loss_text_ce: 0.028466667979955673 (0.03274679590043997) | > loss_mel_ce: 2.835066556930542 (nan) | > loss: 0.011363227851688862 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.1368979153290294) | > loader_time: 0.0049 (0.004684817752027824)  --> TIME: 2025-02-25 23:53:15 -- STEP: 1274/36938 -- GLOBAL_STEP: 998600 | > loss_text_ce: 0.026953866705298424 (0.03277649775109425) | > loss_mel_ce: 2.90199613571167 (nan) | > loss: 0.011622818186879158 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13704596172135153) | > loader_time: 0.0046 (0.0046864912685831445)  --> TIME: 2025-02-25 23:53:25 -- STEP: 1324/36938 -- GLOBAL_STEP: 998650 | > loss_text_ce: 0.023623013868927956 (0.032754790091329984) | > loss_mel_ce: 0.8610253930091858 (nan) | > loss: 0.003510509617626667 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13715244077843863) | > loader_time: 0.0047 (0.004687693724099241)  --> TIME: 2025-02-25 23:53:35 -- STEP: 1374/36938 -- GLOBAL_STEP: 998700 | > loss_text_ce: 0.026958217844367027 (0.03272546582693964) | > loss_mel_ce: 3.3105618953704834 (nan) | > loss: 0.013244127854704857 (nan) | > current_lr: 5e-06 | > step_time: 0.1875 (0.13731473224478888) | > loader_time: 0.0049 (0.004687676957253977)  --> TIME: 2025-02-25 23:53:44 -- STEP: 1424/36938 -- GLOBAL_STEP: 998750 | > loss_text_ce: 0.03944168984889984 (0.032662483334431454) | > loss_mel_ce: 1.268540382385254 (nan) | > loss: 0.0051904055289924145 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13725847608587746) | > loader_time: 0.0046 (0.00468669496895222)  --> TIME: 2025-02-25 23:53:53 -- STEP: 1474/36938 -- GLOBAL_STEP: 998800 | > loss_text_ce: 0.03205772116780281 (0.03266264337937636) | > loss_mel_ce: 3.2229421138763428 (nan) | > loss: 0.012916666455566883 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.1373228208342598) | > loader_time: 0.0046 (0.004685757118131927)  --> TIME: 2025-02-25 23:54:03 -- STEP: 1524/36938 -- GLOBAL_STEP: 998850 | > loss_text_ce: 0.034424133598804474 (0.03264043872914428) | > loss_mel_ce: 1.6771972179412842 (nan) | > loss: 0.006792149040848017 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.1373229402256767) | > loader_time: 0.0046 (0.0046844138248073085)  --> TIME: 2025-02-25 23:54:12 -- STEP: 1574/36938 -- GLOBAL_STEP: 998900 | > loss_text_ce: 0.041694946587085724 (0.03264567553535526) | > loss_mel_ce: 2.6707866191864014 (nan) | > loss: 0.01076381653547287 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13731134377107565) | > loader_time: 0.0046 (0.004683507140134151)  --> TIME: 2025-02-25 23:54:21 -- STEP: 1624/36938 -- GLOBAL_STEP: 998950 | > loss_text_ce: 0.033328913152217865 (0.032644805940570086) | > loss_mel_ce: 2.0242416858673096 (nan) | > loss: 0.008164963684976101 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13715826657605312) | > loader_time: 0.0047 (0.004681907263882637)  --> TIME: 2025-02-25 23:54:31 -- STEP: 1674/36938 -- GLOBAL_STEP: 999000 | > loss_text_ce: 0.03978940472006798 (0.03263155326943766) | > loss_mel_ce: 2.0293920040130615 (nan) | > loss: 0.008211038075387478 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.1372342771954008) | > loader_time: 0.0049 (0.004681572834365821)  --> TIME: 2025-02-25 23:54:40 -- STEP: 1724/36938 -- GLOBAL_STEP: 999050 | > loss_text_ce: 0.03257875144481659 (0.032681495500642165) | > loss_mel_ce: 3.247941255569458 (nan) | > loss: 0.013017937541007996 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13726991539488276) | > loader_time: 0.0048 (0.004680160718285007)  --> TIME: 2025-02-25 23:54:50 -- STEP: 1774/36938 -- GLOBAL_STEP: 999100 | > loss_text_ce: 0.03487517684698105 (0.03264623069442748) | > loss_mel_ce: 3.08030366897583 (nan) | > loss: 0.012361821718513966 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.1373427638609758) | > loader_time: 0.0046 (0.004681050172653105)  --> TIME: 2025-02-25 23:54:59 -- STEP: 1824/36938 -- GLOBAL_STEP: 999150 | > loss_text_ce: 0.03827961906790733 (0.03262974272314638) | > loss_mel_ce: 0.0509033203125 (nan) | > loss: 0.0003539005992934108 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13732708755292414) | > loader_time: 0.0046 (0.004680656550223361)  --> TIME: 2025-02-25 23:55:08 -- STEP: 1874/36938 -- GLOBAL_STEP: 999200 | > loss_text_ce: 0.03442586958408356 (0.03258960325272894) | > loss_mel_ce: 2.376692056655884 (nan) | > loss: 0.009567929431796074 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13721609980726718) | > loader_time: 0.0047 (0.0046812940941548965)  --> TIME: 2025-02-25 23:55:17 -- STEP: 1924/36938 -- GLOBAL_STEP: 999250 | > loss_text_ce: 0.02579859457910061 (0.032598857161358706) | > loss_mel_ce: 2.2200984954833984 (nan) | > loss: 0.008912290446460247 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13698332683955822) | > loader_time: 0.0046 (0.004679051357594442)  --> TIME: 2025-02-25 23:55:26 -- STEP: 1974/36938 -- GLOBAL_STEP: 999300 | > loss_text_ce: 0.029288524761795998 (0.032597079772592914) | > loss_mel_ce: 3.060044050216675 (nan) | > loss: 0.012259257026016712 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13700258103789065) | > loader_time: 0.0046 (0.004677885329952833)  --> TIME: 2025-02-25 23:55:36 -- STEP: 2024/36938 -- GLOBAL_STEP: 999350 | > loss_text_ce: 0.032243650406599045 (0.032617999084768846) | > loss_mel_ce: 2.480185031890869 (nan) | > loss: 0.009969956241548061 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13712035679063322) | > loader_time: 0.0045 (0.004676930282426911)  --> TIME: 2025-02-25 23:55:45 -- STEP: 2074/36938 -- GLOBAL_STEP: 999400 | > loss_text_ce: 0.04091879352927208 (0.032682337756417004) | > loss_mel_ce: 2.8945083618164062 (nan) | > loss: 0.011648520827293396 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1371075932292149) | > loader_time: 0.0046 (0.004675887934612095)  --> TIME: 2025-02-25 23:55:55 -- STEP: 2124/36938 -- GLOBAL_STEP: 999450 | > loss_text_ce: 0.041141342371702194 (0.032682312708155234) | > loss_mel_ce: 2.4616189002990723 (nan) | > loss: 0.009931588545441628 (nan) | > current_lr: 5e-06 | > step_time: 0.1569 (0.13707484038311674) | > loader_time: 0.0048 (0.004675658633704458)  --> TIME: 2025-02-25 23:56:04 -- STEP: 2174/36938 -- GLOBAL_STEP: 999500 | > loss_text_ce: 0.03444494679570198 (0.03267139079659273) | > loss_mel_ce: 2.362041473388672 (nan) | > loss: 0.009509867988526821 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13708431104398597) | > loader_time: 0.0045 (0.004674328283750358)  --> TIME: 2025-02-25 23:56:14 -- STEP: 2224/36938 -- GLOBAL_STEP: 999550 | > loss_text_ce: 0.05068368837237358 (0.03266474715954969) | > loss_mel_ce: 2.2710535526275635 (nan) | > loss: 0.00921324361115694 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13721703346684694) | > loader_time: 0.0045 (0.004673694963935463)  --> TIME: 2025-02-25 23:56:23 -- STEP: 2274/36938 -- GLOBAL_STEP: 999600 | > loss_text_ce: 0.02414759434759617 (0.03267131548861455) | > loss_mel_ce: 3.2400288581848145 (nan) | > loss: 0.012953081168234348 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13716549535749675) | > loader_time: 0.0046 (0.004672436621908354)  --> TIME: 2025-02-25 23:56:32 -- STEP: 2324/36938 -- GLOBAL_STEP: 999650 | > loss_text_ce: 0.02835313230752945 (0.03266273629883713) | > loss_mel_ce: 2.6350677013397217 (nan) | > loss: 0.010569131001830101 (nan) | > current_lr: 5e-06 | > step_time: 0.1462 (0.13709802040750124) | > loader_time: 0.0047 (0.004671071256910047)  --> TIME: 2025-02-25 23:56:41 -- STEP: 2374/36938 -- GLOBAL_STEP: 999700 | > loss_text_ce: 0.033367011696100235 (0.03266505197768615) | > loss_mel_ce: 2.7578155994415283 (nan) | > loss: 0.011076121591031551 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.13699836769120094) | > loader_time: 0.0046 (0.00466948270596871)  --> TIME: 2025-02-25 23:56:50 -- STEP: 2424/36938 -- GLOBAL_STEP: 999750 | > loss_text_ce: 0.045436400920152664 (0.03266480599765189) | > loss_mel_ce: 1.349518060684204 (nan) | > loss: 0.005535533651709557 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1369689945930697) | > loader_time: 0.0046 (0.004670364726888065)  --> TIME: 2025-02-25 23:57:00 -- STEP: 2474/36938 -- GLOBAL_STEP: 999800 | > loss_text_ce: 0.037952687591314316 (0.03266072514065661) | > loss_mel_ce: 2.772106409072876 (nan) | > loss: 0.011151028797030449 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13700963714571882) | > loader_time: 0.0046 (0.004670766157398509)  --> TIME: 2025-02-25 23:57:09 -- STEP: 2524/36938 -- GLOBAL_STEP: 999850 | > loss_text_ce: 0.025096828117966652 (0.03266554979774256) | > loss_mel_ce: 3.190134048461914 (nan) | > loss: 0.01275885384529829 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.1369706411180333) | > loader_time: 0.0047 (0.004671402854133511)  --> TIME: 2025-02-25 23:57:18 -- STEP: 2574/36938 -- GLOBAL_STEP: 999900 | > loss_text_ce: 0.03478506579995155 (0.03264335740796172) | > loss_mel_ce: 2.516814947128296 (nan) | > loss: 0.010125397704541683 (nan) | > current_lr: 5e-06 | > step_time: 0.1885 (0.1369242131848994) | > loader_time: 0.0051 (0.004671541960922979)  --> TIME: 2025-02-25 23:57:28 -- STEP: 2624/36938 -- GLOBAL_STEP: 999950 | > loss_text_ce: 0.0313664935529232 (0.032631206990960816) | > loss_mel_ce: 2.154120922088623 (nan) | > loss: 0.008672569878399372 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13702165362675034) | > loader_time: 0.0045 (0.004671690213244125)  --> TIME: 2025-02-25 23:57:38 -- STEP: 2674/36938 -- GLOBAL_STEP: 1000000 | > loss_text_ce: 0.03650858253240585 (0.032621551887045235) | > loss_mel_ce: 2.52274489402771 (nan) | > loss: 0.010155768133699894 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13713542498486575) | > loader_time: 0.0046 (0.004672266336724922) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1000000.pth  --> TIME: 2025-02-25 23:57:54 -- STEP: 2724/36938 -- GLOBAL_STEP: 1000050 | > loss_text_ce: 0.03125593066215515 (0.032623460788140345) | > loss_mel_ce: 2.7446465492248535 (nan) | > loss: 0.011015486903488636 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1370715889286542) | > loader_time: 0.0045 (0.004673036534705694)  --> TIME: 2025-02-25 23:58:03 -- STEP: 2774/36938 -- GLOBAL_STEP: 1000100 | > loss_text_ce: 0.031538788229227066 (0.03262546847734899) | > loss_mel_ce: 2.969597578048706 (nan) | > loss: 0.011909271590411663 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13703613884602) | > loader_time: 0.0049 (0.004672332040337056)  --> TIME: 2025-02-25 23:58:12 -- STEP: 2824/36938 -- GLOBAL_STEP: 1000150 | > loss_text_ce: 0.030474599450826645 (0.03263436089053479) | > loss_mel_ce: 3.108250379562378 (nan) | > loss: 0.012455258518457413 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1369768503536247) | > loader_time: 0.0045 (0.0046715755807103816)  --> TIME: 2025-02-25 23:58:21 -- STEP: 2874/36938 -- GLOBAL_STEP: 1000200 | > loss_text_ce: 0.024579955264925957 (0.032631461887063735) | > loss_mel_ce: 2.7075839042663574 (nan) | > loss: 0.010841920971870422 (nan) | > current_lr: 5e-06 | > step_time: 0.1818 (0.13705776637016948) | > loader_time: 0.0049 (0.004673214695398883)  --> TIME: 2025-02-25 23:58:31 -- STEP: 2924/36938 -- GLOBAL_STEP: 1000250 | > loss_text_ce: 0.030147546902298927 (0.03262959042864715) | > loss_mel_ce: 3.5228428840637207 (nan) | > loss: 0.01409916952252388 (nan) | > current_lr: 5e-06 | > step_time: 0.182 (0.13706873045411236) | > loader_time: 0.0049 (0.004673365284414841)  --> TIME: 2025-02-25 23:58:40 -- STEP: 2974/36938 -- GLOBAL_STEP: 1000300 | > loss_text_ce: 0.03131178021430969 (0.032618953348478856) | > loss_mel_ce: 2.4798786640167236 (nan) | > loss: 0.009965041652321815 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.13711390546607607) | > loader_time: 0.0047 (0.004674425763256522)  --> TIME: 2025-02-25 23:58:50 -- STEP: 3024/36938 -- GLOBAL_STEP: 1000350 | > loss_text_ce: 0.035308558493852615 (0.03263332896185898) | > loss_mel_ce: 3.5223875045776367 (nan) | > loss: 0.01411784254014492 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.13713947009472643) | > loader_time: 0.0049 (0.004674183470862242)  --> TIME: 2025-02-25 23:58:59 -- STEP: 3074/36938 -- GLOBAL_STEP: 1000400 | > loss_text_ce: 0.027031950652599335 (0.03264304885923112) | > loss_mel_ce: 2.9066922664642334 (nan) | > loss: 0.011641763150691986 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13707387098338444) | > loader_time: 0.0046 (0.004674334066908463)  --> TIME: 2025-02-25 23:59:08 -- STEP: 3124/36938 -- GLOBAL_STEP: 1000450 | > loss_text_ce: 0.03185724839568138 (0.032639142668122796) | > loss_mel_ce: 3.1629014015197754 (nan) | > loss: 0.012677614577114582 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13700838430895554) | > loader_time: 0.0046 (0.004674160144698446)  --> TIME: 2025-02-25 23:59:17 -- STEP: 3174/36938 -- GLOBAL_STEP: 1000500 | > loss_text_ce: 0.0371512696146965 (0.032650772815778016) | > loss_mel_ce: 0.055511474609375 (nan) | > loss: 0.0003677093191072345 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.1370308704923068) | > loader_time: 0.0047 (0.004675362270886439)  --> TIME: 2025-02-25 23:59:27 -- STEP: 3224/36938 -- GLOBAL_STEP: 1000550 | > loss_text_ce: 0.033464353531599045 (0.032639839608746904) | > loss_mel_ce: 2.6508777141571045 (nan) | > loss: 0.010652151890099049 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.1370222163999054) | > loader_time: 0.0047 (0.004675463470572568)  --> TIME: 2025-02-25 23:59:36 -- STEP: 3274/36938 -- GLOBAL_STEP: 1000600 | > loss_text_ce: 0.034779924899339676 (0.03265257936145115) | > loss_mel_ce: 2.0543856620788574 (nan) | > loss: 0.008290340192615986 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.13704147401493966) | > loader_time: 0.0052 (0.004675404065690824)  --> TIME: 2025-02-25 23:59:46 -- STEP: 3324/36938 -- GLOBAL_STEP: 1000650 | > loss_text_ce: 0.03466489538550377 (0.03268191931969865) | > loss_mel_ce: 2.1261978149414062 (nan) | > loss: 0.008574852719902992 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.1370540929041474) | > loader_time: 0.0046 (0.004675670220963785)  --> TIME: 2025-02-25 23:59:55 -- STEP: 3374/36938 -- GLOBAL_STEP: 1000700 | > loss_text_ce: 0.03022252768278122 (0.03267702949992433) | > loss_mel_ce: 2.784212112426758 (nan) | > loss: 0.011168391443789005 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13709560455340378) | > loader_time: 0.0046 (0.004676015191906356)  --> TIME: 2025-02-26 00:00:04 -- STEP: 3424/36938 -- GLOBAL_STEP: 1000750 | > loss_text_ce: 0.030174152925610542 (0.0326667868058419) | > loss_mel_ce: 3.0529708862304688 (nan) | > loss: 0.01223470363765955 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13704228011247166) | > loader_time: 0.0045 (0.0046750781970603394)  --> TIME: 2025-02-26 00:00:13 -- STEP: 3474/36938 -- GLOBAL_STEP: 1000800 | > loss_text_ce: 0.03176340088248253 (0.03265709615359161) | > loss_mel_ce: 3.040351390838623 (nan) | > loss: 0.012190932407975197 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13697244802561745) | > loader_time: 0.0047 (0.004674900792835079)  --> TIME: 2025-02-26 00:00:23 -- STEP: 3524/36938 -- GLOBAL_STEP: 1000850 | > loss_text_ce: 0.04023207724094391 (0.03263811949348746) | > loss_mel_ce: 1.7842351198196411 (nan) | > loss: 0.00723994942381978 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.1369556450410817) | > loader_time: 0.0045 (0.004675295918537182)  --> TIME: 2025-02-26 00:00:32 -- STEP: 3574/36938 -- GLOBAL_STEP: 1000900 | > loss_text_ce: 0.03235843405127525 (0.03262830357873527) | > loss_mel_ce: 2.62567138671875 (nan) | > loss: 0.010547737590968609 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13698590155346962) | > loader_time: 0.0046 (0.00467639244263173)  --> TIME: 2025-02-26 00:00:41 -- STEP: 3624/36938 -- GLOBAL_STEP: 1000950 | > loss_text_ce: 0.03161577135324478 (0.032618242413089256) | > loss_mel_ce: 2.401754379272461 (nan) | > loss: 0.009656230919063091 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13690885908030004) | > loader_time: 0.0046 (0.004676112341828163)  --> TIME: 2025-02-26 00:00:51 -- STEP: 3674/36938 -- GLOBAL_STEP: 1001000 | > loss_text_ce: 0.03634493425488472 (0.032609211754461444) | > loss_mel_ce: 2.9754836559295654 (nan) | > loss: 0.01195170171558857 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13691835618888692) | > loader_time: 0.0046 (0.00467695810902553)  --> TIME: 2025-02-26 00:01:00 -- STEP: 3724/36938 -- GLOBAL_STEP: 1001050 | > loss_text_ce: 0.030534662306308746 (0.03260955318286375) | > loss_mel_ce: 3.0350029468536377 (nan) | > loss: 0.012164833024144173 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13691529788469023) | > loader_time: 0.0047 (0.004677209895100164)  --> TIME: 2025-02-26 00:01:09 -- STEP: 3774/36938 -- GLOBAL_STEP: 1001100 | > loss_text_ce: 0.026597516611218452 (0.03259571663498343) | > loss_mel_ce: 2.8307945728302 (nan) | > loss: 0.011338857933878899 (nan) | > current_lr: 5e-06 | > step_time: 0.1714 (0.1368506374217622) | > loader_time: 0.0048 (0.0046772288467496126)  --> TIME: 2025-02-26 00:01:18 -- STEP: 3824/36938 -- GLOBAL_STEP: 1001150 | > loss_text_ce: 0.03013436682522297 (0.03259622237146309) | > loss_mel_ce: 3.0401320457458496 (nan) | > loss: 0.0121835982427001 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13681407035893486) | > loader_time: 0.0048 (0.004677960314012467)  --> TIME: 2025-02-26 00:01:27 -- STEP: 3874/36938 -- GLOBAL_STEP: 1001200 | > loss_text_ce: 0.038948871195316315 (0.03260192318608394) | > loss_mel_ce: 2.8045897483825684 (nan) | > loss: 0.011283883824944496 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13680310592739997) | > loader_time: 0.0046 (0.004678093470059904)  --> TIME: 2025-02-26 00:01:37 -- STEP: 3924/36938 -- GLOBAL_STEP: 1001250 | > loss_text_ce: 0.031076619401574135 (0.032607031578841796) | > loss_mel_ce: 3.201660633087158 (nan) | > loss: 0.012828323058784008 (nan) | > current_lr: 5e-06 | > step_time: 0.1846 (0.1367682053894565) | > loader_time: 0.005 (0.00467832275608385)  --> TIME: 2025-02-26 00:01:46 -- STEP: 3974/36938 -- GLOBAL_STEP: 1001300 | > loss_text_ce: 0.03306462988257408 (0.03262230831751007) | > loss_mel_ce: 3.3771533966064453 (nan) | > loss: 0.013532611541450024 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13671741529991724) | > loader_time: 0.0046 (0.004678119830768429)  --> TIME: 2025-02-26 00:01:55 -- STEP: 4024/36938 -- GLOBAL_STEP: 1001350 | > loss_text_ce: 0.03003072738647461 (0.03262161301443392) | > loss_mel_ce: 1.140700101852417 (nan) | > loss: 0.0046457573771476746 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13677651525491757) | > loader_time: 0.0045 (0.0046786952089836725)  --> TIME: 2025-02-26 00:02:05 -- STEP: 4074/36938 -- GLOBAL_STEP: 1001400 | > loss_text_ce: 0.031900133937597275 (0.032620250500709415) | > loss_mel_ce: 2.9043378829956055 (nan) | > loss: 0.011651738546788692 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13679229235848092) | > loader_time: 0.0048 (0.004679150363658064)  --> TIME: 2025-02-26 00:02:14 -- STEP: 4124/36938 -- GLOBAL_STEP: 1001450 | > loss_text_ce: 0.024595137685537338 (0.032629470596321264) | > loss_mel_ce: 2.5525288581848145 (nan) | > loss: 0.010226682759821415 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13678332096621562) | > loader_time: 0.0046 (0.004679724906510683)  --> TIME: 2025-02-26 00:02:23 -- STEP: 4174/36938 -- GLOBAL_STEP: 1001500 | > loss_text_ce: 0.0359945148229599 (0.0326209607376585) | > loss_mel_ce: 2.38336181640625 (nan) | > loss: 0.009600620716810226 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13678863194780494) | > loader_time: 0.0046 (0.004680273917855742)  --> TIME: 2025-02-26 00:02:33 -- STEP: 4224/36938 -- GLOBAL_STEP: 1001550 | > loss_text_ce: 0.026071084663271904 (0.032610921691450565) | > loss_mel_ce: 3.18229079246521 (nan) | > loss: 0.012731595896184444 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.13685461686867667) | > loader_time: 0.0046 (0.004680883365147037)  --> TIME: 2025-02-26 00:02:42 -- STEP: 4274/36938 -- GLOBAL_STEP: 1001600 | > loss_text_ce: 0.0337759405374527 (0.032598569324814544) | > loss_mel_ce: 2.635868549346924 (nan) | > loss: 0.010593828745186329 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13682367849640056) | > loader_time: 0.0047 (0.004681574221688224)  --> TIME: 2025-02-26 00:02:51 -- STEP: 4324/36938 -- GLOBAL_STEP: 1001650 | > loss_text_ce: 0.02532918192446232 (0.032576231008729875) | > loss_mel_ce: 2.4285590648651123 (nan) | > loss: 0.00973765179514885 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1368351370289192) | > loader_time: 0.0046 (0.00468202904349229)  --> TIME: 2025-02-26 00:03:01 -- STEP: 4374/36938 -- GLOBAL_STEP: 1001700 | > loss_text_ce: 0.027883674949407578 (0.03257596515528742) | > loss_mel_ce: 3.1229681968688965 (nan) | > loss: 0.012503381818532944 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.1368861403375879) | > loader_time: 0.0048 (0.004682563405414122)  --> TIME: 2025-02-26 00:03:10 -- STEP: 4424/36938 -- GLOBAL_STEP: 1001750 | > loss_text_ce: 0.027317022904753685 (0.03257356627838382) | > loss_mel_ce: 2.7000436782836914 (nan) | > loss: 0.010822860524058342 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13687943059540147) | > loader_time: 0.0045 (0.004682305600716453)  --> TIME: 2025-02-26 00:03:20 -- STEP: 4474/36938 -- GLOBAL_STEP: 1001800 | > loss_text_ce: 0.042907584458589554 (0.032576601561591895) | > loss_mel_ce: 2.3030312061309814 (nan) | > loss: 0.009309280663728714 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13687834491354578) | > loader_time: 0.0047 (0.004682387312342134)  --> TIME: 2025-02-26 00:03:29 -- STEP: 4524/36938 -- GLOBAL_STEP: 1001850 | > loss_text_ce: 0.04566429555416107 (0.03257907444226643) | > loss_mel_ce: 2.1605193614959717 (nan) | > loss: 0.008754697628319263 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13687596249432812) | > loader_time: 0.0045 (0.004682214728084736)  --> TIME: 2025-02-26 00:03:38 -- STEP: 4574/36938 -- GLOBAL_STEP: 1001900 | > loss_text_ce: 0.024797314777970314 (0.032569730091778167) | > loss_mel_ce: 2.7117063999176025 (nan) | > loss: 0.010859142057597637 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1368602930743672) | > loader_time: 0.0047 (0.004682232888442332)  --> TIME: 2025-02-26 00:03:47 -- STEP: 4624/36938 -- GLOBAL_STEP: 1001950 | > loss_text_ce: 0.031031662598252296 (0.03257594423656193) | > loss_mel_ce: 2.731111764907837 (nan) | > loss: 0.010960887186229229 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.1368290027120121) | > loader_time: 0.0044 (0.004682431635559637)  --> TIME: 2025-02-26 00:03:57 -- STEP: 4674/36938 -- GLOBAL_STEP: 1002000 | > loss_text_ce: 0.04139953479170799 (0.03256458162709082) | > loss_mel_ce: 1.0227690935134888 (nan) | > loss: 0.004222891293466091 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.1368143079522637) | > loader_time: 0.0046 (0.004682085378383234)  --> TIME: 2025-02-26 00:04:06 -- STEP: 4724/36938 -- GLOBAL_STEP: 1002050 | > loss_text_ce: 0.037768565118312836 (0.032584275021813186) | > loss_mel_ce: 2.408015489578247 (nan) | > loss: 0.009705493226647377 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13681154792133163) | > loader_time: 0.0045 (0.0046817870790124665)  --> TIME: 2025-02-26 00:04:15 -- STEP: 4774/36938 -- GLOBAL_STEP: 1002100 | > loss_text_ce: 0.026413798332214355 (0.03258234913085188) | > loss_mel_ce: 2.6080739498138428 (nan) | > loss: 0.01045431662350893 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.1368253100203028) | > loader_time: 0.0048 (0.004681928316618944)  --> TIME: 2025-02-26 00:04:25 -- STEP: 4824/36938 -- GLOBAL_STEP: 1002150 | > loss_text_ce: 0.021291984245181084 (0.03257730117111716) | > loss_mel_ce: 0.3173358738422394 (nan) | > loss: 0.0013437613379210234 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.1368353699965658) | > loader_time: 0.0047 (0.004682672359852452)  --> TIME: 2025-02-26 00:04:34 -- STEP: 4874/36938 -- GLOBAL_STEP: 1002200 | > loss_text_ce: 0.036317482590675354 (0.03258696926649162) | > loss_mel_ce: 2.865809917449951 (nan) | > loss: 0.011516379192471504 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.1368655331270783) | > loader_time: 0.0046 (0.004683002028545494)  --> TIME: 2025-02-26 00:04:43 -- STEP: 4924/36938 -- GLOBAL_STEP: 1002250 | > loss_text_ce: 0.03545285761356354 (0.0325921244106379) | > loss_mel_ce: 2.748507022857666 (nan) | > loss: 0.01104746013879776 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.13682250352916814) | > loader_time: 0.0048 (0.004683077286937969)  --> TIME: 2025-02-26 00:04:53 -- STEP: 4974/36938 -- GLOBAL_STEP: 1002300 | > loss_text_ce: 0.030982257798314095 (0.03259168608019757) | > loss_mel_ce: 2.873690605163574 (nan) | > loss: 0.011526480317115784 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1368098534586729) | > loader_time: 0.0045 (0.004683656629233363)  --> TIME: 2025-02-26 00:05:02 -- STEP: 5024/36938 -- GLOBAL_STEP: 1002350 | > loss_text_ce: 0.02670140191912651 (0.032589270661556875) | > loss_mel_ce: 3.406616687774658 (nan) | > loss: 0.013624278828501701 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.13679365870679247) | > loader_time: 0.0047 (0.00468369226926451)  --> TIME: 2025-02-26 00:05:11 -- STEP: 5074/36938 -- GLOBAL_STEP: 1002400 | > loss_text_ce: 0.03274688497185707 (0.03259647380766945) | > loss_mel_ce: 3.2412109375 (nan) | > loss: 0.012991896830499172 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13676637441659925) | > loader_time: 0.0047 (0.004683209865827183)  --> TIME: 2025-02-26 00:05:20 -- STEP: 5124/36938 -- GLOBAL_STEP: 1002450 | > loss_text_ce: 0.04323486238718033 (0.03258543235800904) | > loss_mel_ce: 2.3844237327575684 (nan) | > loss: 0.009633566252887249 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.1366947998672978) | > loader_time: 0.0045 (0.0046828235154222604)  --> TIME: 2025-02-26 00:05:29 -- STEP: 5174/36938 -- GLOBAL_STEP: 1002500 | > loss_text_ce: 0.027551375329494476 (0.03257653952148177) | > loss_mel_ce: 1.8325647115707397 (nan) | > loss: 0.007381413597613573 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13668597776732186) | > loader_time: 0.0045 (0.004684054671819484)  --> TIME: 2025-02-26 00:05:38 -- STEP: 5224/36938 -- GLOBAL_STEP: 1002550 | > loss_text_ce: 0.02970799431204796 (0.03256546546028324) | > loss_mel_ce: 1.6085621118545532 (nan) | > loss: 0.0065010725520551205 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13665761480287583) | > loader_time: 0.0049 (0.004684421725879217)  --> TIME: 2025-02-26 00:05:47 -- STEP: 5274/36938 -- GLOBAL_STEP: 1002600 | > loss_text_ce: 0.036224015057086945 (0.03257935572430423) | > loss_mel_ce: 2.6363556385040283 (nan) | > loss: 0.01060547586530447 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1366193462240907) | > loader_time: 0.0048 (0.004684295760983986)  --> TIME: 2025-02-26 00:05:56 -- STEP: 5324/36938 -- GLOBAL_STEP: 1002650 | > loss_text_ce: 0.025950180366635323 (0.03258504498323843) | > loss_mel_ce: 3.0612361431121826 (nan) | > loss: 0.012250740081071854 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13656098111781004) | > loader_time: 0.0047 (0.004683737151162411)  --> TIME: 2025-02-26 00:06:06 -- STEP: 5374/36938 -- GLOBAL_STEP: 1002700 | > loss_text_ce: 0.031041132286190987 (0.03259195019565537) | > loss_mel_ce: 3.35219407081604 (nan) | > loss: 0.013425537385046482 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.13661231447162323) | > loader_time: 0.0047 (0.004684249175539852)  --> TIME: 2025-02-26 00:06:15 -- STEP: 5424/36938 -- GLOBAL_STEP: 1002750 | > loss_text_ce: 0.031236281618475914 (0.03259315240216804) | > loss_mel_ce: 2.3535232543945312 (nan) | > loss: 0.009463331662118435 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13660613181492545) | > loader_time: 0.0048 (0.004684558748143957)  --> TIME: 2025-02-26 00:06:25 -- STEP: 5474/36938 -- GLOBAL_STEP: 1002800 | > loss_text_ce: 0.033702701330184937 (0.03258426060864702) | > loss_mel_ce: 2.5420727729797363 (nan) | > loss: 0.010221331380307674 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13660821093230668) | > loader_time: 0.0046 (0.004685367159831248)  --> TIME: 2025-02-26 00:06:34 -- STEP: 5524/36938 -- GLOBAL_STEP: 1002850 | > loss_text_ce: 0.0284290574491024 (0.03257742941505654) | > loss_mel_ce: 1.1655547618865967 (nan) | > loss: 0.004738031420856714 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13660122591200977) | > loader_time: 0.0045 (0.004685032117375421)  --> TIME: 2025-02-26 00:06:43 -- STEP: 5574/36938 -- GLOBAL_STEP: 1002900 | > loss_text_ce: 0.02629917301237583 (0.03257362916881829) | > loss_mel_ce: 1.0627034902572632 (nan) | > loss: 0.004321439191699028 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.1365632909919182) | > loader_time: 0.0046 (0.004684607615640266)  --> TIME: 2025-02-26 00:06:52 -- STEP: 5624/36938 -- GLOBAL_STEP: 1002950 | > loss_text_ce: 0.04073413833975792 (0.03258603378738891) | > loss_mel_ce: 2.7299532890319824 (nan) | > loss: 0.010994791984558105 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13655320331348283) | > loader_time: 0.0046 (0.004684395123681161)  --> TIME: 2025-02-26 00:07:01 -- STEP: 5674/36938 -- GLOBAL_STEP: 1003000 | > loss_text_ce: 0.02879500947892666 (0.03258694250679586) | > loss_mel_ce: 2.7714931964874268 (nan) | > loss: 0.011112255044281483 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.1365576873247497) | > loader_time: 0.0047 (0.004685421875761052)  --> TIME: 2025-02-26 00:07:11 -- STEP: 5724/36938 -- GLOBAL_STEP: 1003050 | > loss_text_ce: 0.028841158375144005 (0.03257888665921978) | > loss_mel_ce: 2.9810309410095215 (nan) | > loss: 0.011943938210606575 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13654163431904665) | > loader_time: 0.0048 (0.004684976353502039)  --> TIME: 2025-02-26 00:07:20 -- STEP: 5774/36938 -- GLOBAL_STEP: 1003100 | > loss_text_ce: 0.03395131230354309 (0.032576928740889265) | > loss_mel_ce: 1.9961118698120117 (nan) | > loss: 0.008055806159973145 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13652485384014507) | > loader_time: 0.0045 (0.004684505100930732)  --> TIME: 2025-02-26 00:07:29 -- STEP: 5824/36938 -- GLOBAL_STEP: 1003150 | > loss_text_ce: 0.033470459282398224 (0.03258492718601043) | > loss_mel_ce: 2.894717216491699 (nan) | > loss: 0.011619792319834232 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.1365199450526262) | > loader_time: 0.0045 (0.004684991602386747)  --> TIME: 2025-02-26 00:07:38 -- STEP: 5874/36938 -- GLOBAL_STEP: 1003200 | > loss_text_ce: 0.02863362804055214 (0.032584602860045775) | > loss_mel_ce: 2.2372078895568848 (nan) | > loss: 0.008991435170173645 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.1365024795084125) | > loader_time: 0.0047 (0.004685099164685472)  --> TIME: 2025-02-26 00:07:48 -- STEP: 5924/36938 -- GLOBAL_STEP: 1003250 | > loss_text_ce: 0.041802238672971725 (0.032588975238824446) | > loss_mel_ce: 0.03857421875 (nan) | > loss: 0.0003189542330801487 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13653498779350948) | > loader_time: 0.0046 (0.004684859156045141)  --> TIME: 2025-02-26 00:07:57 -- STEP: 5974/36938 -- GLOBAL_STEP: 1003300 | > loss_text_ce: 0.024788975715637207 (0.032579592443599814) | > loss_mel_ce: 2.6064465045928955 (nan) | > loss: 0.010441411286592484 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.1364967834602279) | > loader_time: 0.0045 (0.004685336145861403)  --> TIME: 2025-02-26 00:08:06 -- STEP: 6024/36938 -- GLOBAL_STEP: 1003350 | > loss_text_ce: 0.025560103356838226 (0.03257040869136027) | > loss_mel_ce: 0.44317626953125 (nan) | > loss: 0.0018600651528686285 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.13647476725210994) | > loader_time: 0.0047 (0.004685446323151612)  --> TIME: 2025-02-26 00:08:16 -- STEP: 6074/36938 -- GLOBAL_STEP: 1003400 | > loss_text_ce: 0.03489512577652931 (0.03257201881518062) | > loss_mel_ce: 3.2322239875793457 (nan) | > loss: 0.012964759021997452 (nan) | > current_lr: 5e-06 | > step_time: 0.1862 (0.1364897596314033) | > loader_time: 0.0049 (0.004685384449222429)  --> TIME: 2025-02-26 00:08:25 -- STEP: 6124/36938 -- GLOBAL_STEP: 1003450 | > loss_text_ce: 0.035476479679346085 (0.03257458490714343) | > loss_mel_ce: 2.3966362476348877 (nan) | > loss: 0.009651241824030876 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13649437067011919) | > loader_time: 0.0047 (0.004685558344631085)  --> TIME: 2025-02-26 00:08:34 -- STEP: 6174/36938 -- GLOBAL_STEP: 1003500 | > loss_text_ce: 0.03359935060143471 (0.0325775829155588) | > loss_mel_ce: 1.1512044668197632 (nan) | > loss: 0.00470160273835063 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13649056252824626) | > loader_time: 0.0048 (0.004685193502984084)  --> TIME: 2025-02-26 00:08:44 -- STEP: 6224/36938 -- GLOBAL_STEP: 1003550 | > loss_text_ce: 0.0409940630197525 (0.03257659889124978) | > loss_mel_ce: 2.645259380340576 (nan) | > loss: 0.010659736581146717 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.1365183983333313) | > loader_time: 0.0048 (0.004685153308441836)  --> TIME: 2025-02-26 00:08:53 -- STEP: 6274/36938 -- GLOBAL_STEP: 1003600 | > loss_text_ce: 0.038264200091362 (0.03257451848945453) | > loss_mel_ce: 3.599644184112549 (nan) | > loss: 0.014436145313084126 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13653485559147938) | > loader_time: 0.0046 (0.004685895474123511)  --> TIME: 2025-02-26 00:09:02 -- STEP: 6324/36938 -- GLOBAL_STEP: 1003650 | > loss_text_ce: 0.052607662975788116 (0.03258474735359928) | > loss_mel_ce: 1.965506672859192 (nan) | > loss: 0.008008390665054321 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13649690746884044) | > loader_time: 0.0045 (0.004685742013594686)  --> TIME: 2025-02-26 00:09:12 -- STEP: 6374/36938 -- GLOBAL_STEP: 1003700 | > loss_text_ce: 0.02516869083046913 (0.032578473220891115) | > loss_mel_ce: 0.042938232421875 (nan) | > loss: 0.0002702655619941652 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13649224112839553) | > loader_time: 0.0045 (0.004685614862373194)  --> TIME: 2025-02-26 00:09:21 -- STEP: 6424/36938 -- GLOBAL_STEP: 1003750 | > loss_text_ce: 0.039440229535102844 (0.03257668021458037) | > loss_mel_ce: 2.559623956680298 (nan) | > loss: 0.010313747450709343 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1364816431684482) | > loader_time: 0.0046 (0.0046856204792393285)  --> TIME: 2025-02-26 00:09:30 -- STEP: 6474/36938 -- GLOBAL_STEP: 1003800 | > loss_text_ce: 0.033238306641578674 (0.03258232138679812) | > loss_mel_ce: 3.3984646797180176 (nan) | > loss: 0.013617870397865772 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.1364930471521047) | > loader_time: 0.0047 (0.004685818099445284)  --> TIME: 2025-02-26 00:09:39 -- STEP: 6524/36938 -- GLOBAL_STEP: 1003850 | > loss_text_ce: 0.04291292279958725 (0.03258606552827021) | > loss_mel_ce: 1.8031994104385376 (nan) | > loss: 0.007325842976570129 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.1364619757849916) | > loader_time: 0.0047 (0.004685636242060197)  --> TIME: 2025-02-26 00:09:48 -- STEP: 6574/36938 -- GLOBAL_STEP: 1003900 | > loss_text_ce: 0.024533119052648544 (0.03257938717589107) | > loss_mel_ce: 1.198837161064148 (nan) | > loss: 0.0048546441830694675 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13645425207679454) | > loader_time: 0.0046 (0.0046855494139729165)  --> TIME: 2025-02-26 00:09:57 -- STEP: 6624/36938 -- GLOBAL_STEP: 1003950 | > loss_text_ce: 0.039985645562410355 (0.03258076486369516) | > loss_mel_ce: 1.3398358821868896 (nan) | > loss: 0.005475482437759638 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13643433902286675) | > loader_time: 0.0048 (0.004685415197973679)  --> TIME: 2025-02-26 00:10:07 -- STEP: 6674/36938 -- GLOBAL_STEP: 1004000 | > loss_text_ce: 0.035986773669719696 (0.03258827326909725) | > loss_mel_ce: 2.883699417114258 (nan) | > loss: 0.011586056090891361 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13646151521515257) | > loader_time: 0.0048 (0.004685922836450046)  --> TIME: 2025-02-26 00:10:16 -- STEP: 6724/36938 -- GLOBAL_STEP: 1004050 | > loss_text_ce: 0.030016161501407623 (0.0325844469855571) | > loss_mel_ce: 2.2627434730529785 (nan) | > loss: 0.009098253212869167 (nan) | > current_lr: 5e-06 | > step_time: 0.1565 (0.13646994574590487) | > loader_time: 0.0047 (0.004685768444292758)  --> TIME: 2025-02-26 00:10:26 -- STEP: 6774/36938 -- GLOBAL_STEP: 1004100 | > loss_text_ce: 0.02404581569135189 (0.032581463688090846) | > loss_mel_ce: 1.795732021331787 (nan) | > loss: 0.007221341133117676 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13645593715974072) | > loader_time: 0.0046 (0.004687699555013263)  --> TIME: 2025-02-26 00:10:35 -- STEP: 6824/36938 -- GLOBAL_STEP: 1004150 | > loss_text_ce: 0.031326644122600555 (0.03257899444890375) | > loss_mel_ce: 2.8475167751312256 (nan) | > loss: 0.011423981748521328 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.13645248329932927) | > loader_time: 0.0047 (0.004687610223173124)  --> TIME: 2025-02-26 00:10:44 -- STEP: 6874/36938 -- GLOBAL_STEP: 1004200 | > loss_text_ce: 0.04207950457930565 (0.032575093387635035) | > loss_mel_ce: 3.1176321506500244 (nan) | > loss: 0.012538539245724678 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.13644115398330406) | > loader_time: 0.0047 (0.004687562355094786)  --> TIME: 2025-02-26 00:10:53 -- STEP: 6924/36938 -- GLOBAL_STEP: 1004250 | > loss_text_ce: 0.038022372871637344 (0.032575142128793325) | > loss_mel_ce: 1.7873575687408447 (nan) | > loss: 0.007243571802973747 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13644694713827246) | > loader_time: 0.0046 (0.00468755453601179)  --> TIME: 2025-02-26 00:11:03 -- STEP: 6974/36938 -- GLOBAL_STEP: 1004300 | > loss_text_ce: 0.0315425731241703 (0.032579223517593445) | > loss_mel_ce: 3.029871940612793 (nan) | > loss: 0.012148470617830753 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13644680742346393) | > loader_time: 0.0046 (0.004687404030881101)  --> TIME: 2025-02-26 00:11:12 -- STEP: 7024/36938 -- GLOBAL_STEP: 1004350 | > loss_text_ce: 0.03133510425686836 (0.032584992902865755) | > loss_mel_ce: 2.027700424194336 (nan) | > loss: 0.008170776069164276 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.1364114800794792) | > loader_time: 0.0046 (0.004687353663129531)  --> TIME: 2025-02-26 00:11:21 -- STEP: 7074/36938 -- GLOBAL_STEP: 1004400 | > loss_text_ce: 0.04017740488052368 (0.03259248607129741) | > loss_mel_ce: 2.767113447189331 (nan) | > loss: 0.01114004384726286 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13640690328443256) | > loader_time: 0.0048 (0.004687806088472114)  --> TIME: 2025-02-26 00:11:30 -- STEP: 7124/36938 -- GLOBAL_STEP: 1004450 | > loss_text_ce: 0.03337514027953148 (0.03259695780067144) | > loss_mel_ce: 2.4755454063415527 (nan) | > loss: 0.009956033900380135 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13641591519870358) | > loader_time: 0.0046 (0.0046876604673502425)  --> TIME: 2025-02-26 00:11:40 -- STEP: 7174/36938 -- GLOBAL_STEP: 1004500 | > loss_text_ce: 0.040298931300640106 (0.03259458292599177) | > loss_mel_ce: 3.24422287940979 (nan) | > loss: 0.013033817522227764 (nan) | > current_lr: 5e-06 | > step_time: 0.1632 (0.13645475526224254) | > loader_time: 0.0048 (0.004687682878416104)  --> TIME: 2025-02-26 00:11:49 -- STEP: 7224/36938 -- GLOBAL_STEP: 1004550 | > loss_text_ce: 0.03138042613863945 (0.03260132522466113) | > loss_mel_ce: 2.859814167022705 (nan) | > loss: 0.011472995392978191 (nan) | > current_lr: 5e-06 | > step_time: 0.1813 (0.13643601916293124) | > loader_time: 0.0048 (0.004687364678314233)  --> TIME: 2025-02-26 00:11:58 -- STEP: 7274/36938 -- GLOBAL_STEP: 1004600 | > loss_text_ce: 0.040839049965143204 (0.032609770693277224) | > loss_mel_ce: 2.5111329555511475 (nan) | > loss: 0.010126872919499874 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13640860023818452) | > loader_time: 0.0046 (0.004686988380081817)  --> TIME: 2025-02-26 00:12:07 -- STEP: 7324/36938 -- GLOBAL_STEP: 1004650 | > loss_text_ce: 0.03922830522060394 (0.03260797549421623) | > loss_mel_ce: 3.0742061138153076 (nan) | > loss: 0.012354899197816849 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13640178399031327) | > loader_time: 0.0046 (0.004686940862206675)  --> TIME: 2025-02-26 00:12:16 -- STEP: 7374/36938 -- GLOBAL_STEP: 1004700 | > loss_text_ce: 0.03561112657189369 (0.032615357748497255) | > loss_mel_ce: 1.5495837926864624 (nan) | > loss: 0.0062904562801122665 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1363787612909867) | > loader_time: 0.0045 (0.004686814418872398)  --> TIME: 2025-02-26 00:12:26 -- STEP: 7424/36938 -- GLOBAL_STEP: 1004750 | > loss_text_ce: 0.023518672212958336 (0.03262076889273132) | > loss_mel_ce: 2.9768316745758057 (nan) | > loss: 0.011906152591109276 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13635282526756168) | > loader_time: 0.0044 (0.004686444708756338)  --> TIME: 2025-02-26 00:12:35 -- STEP: 7474/36938 -- GLOBAL_STEP: 1004800 | > loss_text_ce: 0.03204033896327019 (0.03262368715718362) | > loss_mel_ce: 2.2922022342681885 (nan) | > loss: 0.00922318547964096 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13636264381224958) | > loader_time: 0.0047 (0.00468621341372545)  --> TIME: 2025-02-26 00:12:44 -- STEP: 7524/36938 -- GLOBAL_STEP: 1004850 | > loss_text_ce: 0.028439795598387718 (0.032612874019759326) | > loss_mel_ce: 2.4403839111328125 (nan) | > loss: 0.009796920232474804 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13632514792386055) | > loader_time: 0.0048 (0.004685879989484097)  --> TIME: 2025-02-26 00:12:53 -- STEP: 7574/36938 -- GLOBAL_STEP: 1004900 | > loss_text_ce: 0.028913579881191254 (0.032611952056424094) | > loss_mel_ce: 2.7143495082855225 (nan) | > loss: 0.010885965079069138 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13629641906233628) | > loader_time: 0.0046 (0.004685501514653408)  --> TIME: 2025-02-26 00:13:02 -- STEP: 7624/36938 -- GLOBAL_STEP: 1004950 | > loss_text_ce: 0.028278352692723274 (0.03260788943724564) | > loss_mel_ce: 2.1338937282562256 (nan) | > loss: 0.008580048568546772 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13627248855477753) | > loader_time: 0.0047 (0.004684956414001567)  --> TIME: 2025-02-26 00:13:12 -- STEP: 7674/36938 -- GLOBAL_STEP: 1005000 | > loss_text_ce: 0.02506513148546219 (0.03260186277507105) | > loss_mel_ce: 2.1499061584472656 (nan) | > loss: 0.008630839176476002 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13630230615683958) | > loader_time: 0.0045 (0.004685245331979718)  --> TIME: 2025-02-26 00:13:21 -- STEP: 7724/36938 -- GLOBAL_STEP: 1005050 | > loss_text_ce: 0.03496164828538895 (0.032598037480201576) | > loss_mel_ce: 2.776364803314209 (nan) | > loss: 0.011156057938933372 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.1363074492136474) | > loader_time: 0.0046 (0.004685454236479519)  --> TIME: 2025-02-26 00:13:30 -- STEP: 7774/36938 -- GLOBAL_STEP: 1005100 | > loss_text_ce: 0.024410346522927284 (0.032592969645506675) | > loss_mel_ce: 1.9965344667434692 (nan) | > loss: 0.008019622415304184 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.13632295893494206) | > loader_time: 0.0047 (0.004685394218656813)  --> TIME: 2025-02-26 00:13:40 -- STEP: 7824/36938 -- GLOBAL_STEP: 1005150 | > loss_text_ce: 0.024825632572174072 (0.03259315178670073) | > loss_mel_ce: 2.5225234031677246 (nan) | > loss: 0.010108528658747673 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.1363334342502135) | > loader_time: 0.0046 (0.004685120897059067)  --> TIME: 2025-02-26 00:13:49 -- STEP: 7874/36938 -- GLOBAL_STEP: 1005200 | > loss_text_ce: 0.02327539399266243 (0.03259237335993888) | > loss_mel_ce: 2.5840139389038086 (nan) | > loss: 0.010346386581659317 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13634225451343715) | > loader_time: 0.0045 (0.0046851522946133725)  --> TIME: 2025-02-26 00:13:59 -- STEP: 7924/36938 -- GLOBAL_STEP: 1005250 | > loss_text_ce: 0.0395328663289547 (0.0325871330660291) | > loss_mel_ce: 2.5840673446655273 (nan) | > loss: 0.010411112569272518 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13636365332788808) | > loader_time: 0.0046 (0.004685051449617318)  --> TIME: 2025-02-26 00:14:09 -- STEP: 7974/36938 -- GLOBAL_STEP: 1005300 | > loss_text_ce: 0.02864980697631836 (0.03259115073031474) | > loss_mel_ce: 3.003939628601074 (nan) | > loss: 0.01203408557921648 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13640431951752946) | > loader_time: 0.005 (0.004685126332851595)  --> TIME: 2025-02-26 00:14:18 -- STEP: 8024/36938 -- GLOBAL_STEP: 1005350 | > loss_text_ce: 0.031164392828941345 (0.032606578506654625) | > loss_mel_ce: 3.0450189113616943 (nan) | > loss: 0.01220707781612873 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.1364021923166449) | > loader_time: 0.0046 (0.004684969678833151)  --> TIME: 2025-02-26 00:14:27 -- STEP: 8074/36938 -- GLOBAL_STEP: 1005400 | > loss_text_ce: 0.03593165799975395 (0.032610092080131964) | > loss_mel_ce: 0.021484375 (nan) | > loss: 0.00022784141765441746 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13641032789909294) | > loader_time: 0.0046 (0.004685138870590779)  --> TIME: 2025-02-26 00:14:37 -- STEP: 8124/36938 -- GLOBAL_STEP: 1005450 | > loss_text_ce: 0.03214668855071068 (0.032603624192557544) | > loss_mel_ce: 2.8003573417663574 (nan) | > loss: 0.0112400958314538 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.1364332507068686) | > loader_time: 0.0046 (0.004685056291737039)  --> TIME: 2025-02-26 00:14:46 -- STEP: 8174/36938 -- GLOBAL_STEP: 1005500 | > loss_text_ce: 0.05959997698664665 (0.032603984591210396) | > loss_mel_ce: 2.5437424182891846 (nan) | > loss: 0.010330723598599434 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13643111805184313) | > loader_time: 0.0046 (0.0046848539387885245)  --> TIME: 2025-02-26 00:14:55 -- STEP: 8224/36938 -- GLOBAL_STEP: 1005550 | > loss_text_ce: 0.03359290212392807 (0.032614446568673294) | > loss_mel_ce: 2.657865047454834 (nan) | > loss: 0.01068038959056139 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13644172448360514) | > loader_time: 0.0046 (0.004684464158019209)  --> TIME: 2025-02-26 00:15:05 -- STEP: 8274/36938 -- GLOBAL_STEP: 1005600 | > loss_text_ce: 0.022928714752197266 (0.03260833775203202) | > loss_mel_ce: 3.1304688453674316 (nan) | > loss: 0.012513482943177223 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13645849551790326) | > loader_time: 0.0044 (0.004684309121459938)  --> TIME: 2025-02-26 00:15:14 -- STEP: 8324/36938 -- GLOBAL_STEP: 1005650 | > loss_text_ce: 0.025735020637512207 (0.032613977660205205) | > loss_mel_ce: 0.011789957992732525 (nan) | > loss: 0.00014890865713823587 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13646364332333843) | > loader_time: 0.0045 (0.004684158754371674)  --> TIME: 2025-02-26 00:15:24 -- STEP: 8374/36938 -- GLOBAL_STEP: 1005700 | > loss_text_ce: 0.03567025065422058 (0.0326119544982584) | > loss_mel_ce: 2.41926908493042 (nan) | > loss: 0.009741823188960552 (nan) | > current_lr: 5e-06 | > step_time: 0.1714 (0.13648870767429858) | > loader_time: 0.0048 (0.004683990907862679)  --> TIME: 2025-02-26 00:15:33 -- STEP: 8424/36938 -- GLOBAL_STEP: 1005750 | > loss_text_ce: 0.03793524578213692 (0.032610554981411535) | > loss_mel_ce: 2.928783893585205 (nan) | > loss: 0.011772695928812027 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.1364767005753768) | > loader_time: 0.0049 (0.004683493803708988)  --> TIME: 2025-02-26 00:15:42 -- STEP: 8474/36938 -- GLOBAL_STEP: 1005800 | > loss_text_ce: 0.02802589349448681 (0.03260860016419142) | > loss_mel_ce: 1.7148336172103882 (nan) | > loss: 0.006916109472513199 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13646736081996472) | > loader_time: 0.0045 (0.0046830425741876845)  --> TIME: 2025-02-26 00:15:52 -- STEP: 8524/36938 -- GLOBAL_STEP: 1005850 | > loss_text_ce: 0.039365824311971664 (0.032611970093860784) | > loss_mel_ce: 3.2136943340301514 (nan) | > loss: 0.012908969074487686 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.1364999938492261) | > loader_time: 0.0045 (0.004682920925609392)  --> TIME: 2025-02-26 00:16:01 -- STEP: 8574/36938 -- GLOBAL_STEP: 1005900 | > loss_text_ce: 0.021277865394949913 (0.03262082023236741) | > loss_mel_ce: 1.1384832859039307 (nan) | > loss: 0.004602227360010147 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13648375770777102) | > loader_time: 0.0045 (0.004682975408217282)  --> TIME: 2025-02-26 00:16:10 -- STEP: 8624/36938 -- GLOBAL_STEP: 1005950 | > loss_text_ce: 0.031220300123095512 (0.03261206505395959) | > loss_mel_ce: 2.4525604248046875 (nan) | > loss: 0.009856272488832474 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1364859332898998) | > loader_time: 0.0045 (0.004682852850331895)  --> TIME: 2025-02-26 00:16:19 -- STEP: 8674/36938 -- GLOBAL_STEP: 1006000 | > loss_text_ce: 0.035317663103342056 (0.03260893359714168) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13643966637532928) | > loader_time: 0.0046 (0.004682543064994369)  --> TIME: 2025-02-26 00:16:28 -- STEP: 8724/36938 -- GLOBAL_STEP: 1006050 | > loss_text_ce: 0.03134000301361084 (0.0326132386400168) | > loss_mel_ce: 2.0378947257995605 (nan) | > loss: 0.008211249485611916 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13644386709747636) | > loader_time: 0.0045 (0.004682413813500493)  --> TIME: 2025-02-26 00:16:38 -- STEP: 8774/36938 -- GLOBAL_STEP: 1006100 | > loss_text_ce: 0.02787158265709877 (0.032619642710696575) | > loss_mel_ce: 3.1939187049865723 (nan) | > loss: 0.012784883379936218 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.13645577729631492) | > loader_time: 0.0047 (0.004682295355569649)  --> TIME: 2025-02-26 00:16:47 -- STEP: 8824/36938 -- GLOBAL_STEP: 1006150 | > loss_text_ce: 0.02819298952817917 (0.03262297027146549) | > loss_mel_ce: 2.46604585647583 (nan) | > loss: 0.009897774085402489 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13645728476892713) | > loader_time: 0.0046 (0.004682191317446754)  --> TIME: 2025-02-26 00:16:57 -- STEP: 8874/36938 -- GLOBAL_STEP: 1006200 | > loss_text_ce: 0.031767867505550385 (0.03261295198256151) | > loss_mel_ce: 2.9796910285949707 (nan) | > loss: 0.011950233951210976 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.13647723611690274) | > loader_time: 0.0049 (0.00468211797865119)  --> TIME: 2025-02-26 00:17:06 -- STEP: 8924/36938 -- GLOBAL_STEP: 1006250 | > loss_text_ce: 0.027444008737802505 (0.03260578161449071) | > loss_mel_ce: 2.7551496028900146 (nan) | > loss: 0.011042038910090923 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13645223826370378) | > loader_time: 0.0046 (0.004681792749732347)  --> TIME: 2025-02-26 00:17:15 -- STEP: 8974/36938 -- GLOBAL_STEP: 1006300 | > loss_text_ce: 0.02989966981112957 (0.03260585646991266) | > loss_mel_ce: 0.2666015625 (nan) | > loss: 0.0011765921954065561 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13643717242424364) | > loader_time: 0.0046 (0.004681392956183861)  --> TIME: 2025-02-26 00:17:24 -- STEP: 9024/36938 -- GLOBAL_STEP: 1006350 | > loss_text_ce: 0.04569355770945549 (0.0326070675556424) | > loss_mel_ce: 3.0195252895355225 (nan) | > loss: 0.012163568288087845 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.13645128567590825) | > loader_time: 0.0047 (0.00468102718393008)  --> TIME: 2025-02-26 00:17:34 -- STEP: 9074/36938 -- GLOBAL_STEP: 1006400 | > loss_text_ce: 0.02385903336107731 (0.032608143416768494) | > loss_mel_ce: 1.552459716796875 (nan) | > loss: 0.006255233660340309 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.13643301573781355) | > loader_time: 0.0046 (0.004680577526803005)  --> TIME: 2025-02-26 00:17:43 -- STEP: 9124/36938 -- GLOBAL_STEP: 1006450 | > loss_text_ce: 0.03484830632805824 (0.03261535895503699) | > loss_mel_ce: 1.86063551902771 (nan) | > loss: 0.007521761581301689 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13640524953251013) | > loader_time: 0.0047 (0.004680081633610038)  --> TIME: 2025-02-26 00:17:52 -- STEP: 9174/36938 -- GLOBAL_STEP: 1006500 | > loss_text_ce: 0.037242088466882706 (0.032618896303065366) | > loss_mel_ce: 2.528893232345581 (nan) | > loss: 0.010183077305555344 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13641045606549582) | > loader_time: 0.0046 (0.004679780472122767)  --> TIME: 2025-02-26 00:18:02 -- STEP: 9224/36938 -- GLOBAL_STEP: 1006550 | > loss_text_ce: 0.03408270329236984 (0.03261729968412466) | > loss_mel_ce: 2.949769973754883 (nan) | > loss: 0.01184068527072668 (nan) | > current_lr: 5e-06 | > step_time: 0.1475 (0.13642705187317855) | > loader_time: 0.0046 (0.004679451351670317)  --> TIME: 2025-02-26 00:18:11 -- STEP: 9274/36938 -- GLOBAL_STEP: 1006600 | > loss_text_ce: 0.017298055812716484 (0.03261391490993906) | > loss_mel_ce: 0.9727662205696106 (nan) | > loss: 0.003928826656192541 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13642290766084444) | > loader_time: 0.0045 (0.004679413918459598)  --> TIME: 2025-02-26 00:18:20 -- STEP: 9324/36938 -- GLOBAL_STEP: 1006650 | > loss_text_ce: 0.025621119886636734 (0.032612561598237964) | > loss_mel_ce: 1.9929929971694946 (nan) | > loss: 0.008010373450815678 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13641633093996391) | > loader_time: 0.0048 (0.004679186131400848)  --> TIME: 2025-02-26 00:18:29 -- STEP: 9374/36938 -- GLOBAL_STEP: 1006700 | > loss_text_ce: 0.0355311818420887 (0.032607619018586616) | > loss_mel_ce: 2.6524791717529297 (nan) | > loss: 0.010666708461940289 (nan) | > current_lr: 5e-06 | > step_time: 0.1566 (0.13640546351067212) | > loader_time: 0.0046 (0.004678966954798744)  --> TIME: 2025-02-26 00:18:38 -- STEP: 9424/36938 -- GLOBAL_STEP: 1006750 | > loss_text_ce: 0.030608270317316055 (0.032609693929774426) | > loss_mel_ce: 2.009235382080078 (nan) | > loss: 0.00809461809694767 (nan) | > current_lr: 5e-06 | > step_time: 0.1267 (0.13640797540469612) | > loader_time: 0.0046 (0.004678855373418188)  --> TIME: 2025-02-26 00:18:48 -- STEP: 9474/36938 -- GLOBAL_STEP: 1006800 | > loss_text_ce: 0.03517263010144234 (0.032615274551674726) | > loss_mel_ce: 2.2499871253967285 (nan) | > loss: 0.009068095125257969 (nan) | > current_lr: 5e-06 | > step_time: 0.1486 (0.1364235125288627) | > loader_time: 0.0047 (0.004678742327417328)  --> TIME: 2025-02-26 00:18:57 -- STEP: 9524/36938 -- GLOBAL_STEP: 1006850 | > loss_text_ce: 0.03197696805000305 (0.032617209128548916) | > loss_mel_ce: 0.0382893867790699 (nan) | > loss: 0.00027883474831469357 (nan) | > current_lr: 5e-06 | > step_time: 0.1074 (0.1364052016311516) | > loader_time: 0.0045 (0.004678596397850103)  --> TIME: 2025-02-26 00:19:07 -- STEP: 9574/36938 -- GLOBAL_STEP: 1006900 | > loss_text_ce: 0.03520674630999565 (0.03261568746952675) | > loss_mel_ce: 2.8760697841644287 (nan) | > loss: 0.011552685871720314 (nan) | > current_lr: 5e-06 | > step_time: 0.1833 (0.13642408269069828) | > loader_time: 0.0057 (0.004678640904495308)  --> TIME: 2025-02-26 00:19:16 -- STEP: 9624/36938 -- GLOBAL_STEP: 1006950 | > loss_text_ce: 0.03129496052861214 (0.03261605527016616) | > loss_mel_ce: 2.6210498809814453 (nan) | > loss: 0.010525179095566273 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.1364179422780073) | > loader_time: 0.0048 (0.0046786626031373)  --> TIME: 2025-02-26 00:19:25 -- STEP: 9674/36938 -- GLOBAL_STEP: 1007000 | > loss_text_ce: 0.029985295608639717 (0.032610488613246814) | > loss_mel_ce: 2.507499933242798 (nan) | > loss: 0.010069386102259159 (nan) | > current_lr: 5e-06 | > step_time: 0.1775 (0.13641380494799793) | > loader_time: 0.0048 (0.004679597851065337)  --> TIME: 2025-02-26 00:19:34 -- STEP: 9724/36938 -- GLOBAL_STEP: 1007050 | > loss_text_ce: 0.03691079840064049 (0.032605622975513504) | > loss_mel_ce: 2.395331859588623 (nan) | > loss: 0.009651756845414639 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13639495675331448) | > loader_time: 0.0046 (0.004679504979642415)  --> TIME: 2025-02-26 00:19:43 -- STEP: 9774/36938 -- GLOBAL_STEP: 1007100 | > loss_text_ce: 0.03790031000971794 (0.03260589097494057) | > loss_mel_ce: 3.4547345638275146 (nan) | > loss: 0.013859662227332592 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.1363815995650185) | > loader_time: 0.005 (0.0046796053495537825)  --> TIME: 2025-02-26 00:19:53 -- STEP: 9824/36938 -- GLOBAL_STEP: 1007150 | > loss_text_ce: 0.03577065467834473 (0.032607089855529126) | > loss_mel_ce: 2.3593173027038574 (nan) | > loss: 0.009504318237304688 (nan) | > current_lr: 5e-06 | > step_time: 0.1265 (0.13637206528687792) | > loader_time: 0.0066 (0.004679841623438296)  --> TIME: 2025-02-26 00:20:02 -- STEP: 9874/36938 -- GLOBAL_STEP: 1007200 | > loss_text_ce: 0.027519216760993004 (0.03260534130439003) | > loss_mel_ce: 2.900424003601074 (nan) | > loss: 0.011618822813034058 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13637721357448043) | > loader_time: 0.0046 (0.004679899213766549)  --> TIME: 2025-02-26 00:20:11 -- STEP: 9924/36938 -- GLOBAL_STEP: 1007250 | > loss_text_ce: 0.02622634917497635 (0.0326068775917161) | > loss_mel_ce: 2.55131459236145 (nan) | > loss: 0.010228337720036507 (nan) | > current_lr: 5e-06 | > step_time: 0.1116 (0.1363761184773882) | > loader_time: 0.0045 (0.00467973784158807)  --> TIME: 2025-02-26 00:20:21 -- STEP: 9974/36938 -- GLOBAL_STEP: 1007300 | > loss_text_ce: 0.03392554819583893 (0.03260348302037473) | > loss_mel_ce: 3.0412392616271973 (nan) | > loss: 0.0122030358761549 (nan) | > current_lr: 5e-06 | > step_time: 0.1879 (0.13636783196258317) | > loader_time: 0.0048 (0.004679584350188172)  --> TIME: 2025-02-26 00:20:30 -- STEP: 10024/36938 -- GLOBAL_STEP: 1007350 | > loss_text_ce: 0.03421661630272865 (0.03260639272031867) | > loss_mel_ce: 2.7522270679473877 (nan) | > loss: 0.01105731725692749 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13640825905898898) | > loader_time: 0.0047 (0.004679422186359058)  --> TIME: 2025-02-26 00:20:40 -- STEP: 10074/36938 -- GLOBAL_STEP: 1007400 | > loss_text_ce: 0.03949389234185219 (0.032605941568889296) | > loss_mel_ce: 2.8908042907714844 (nan) | > loss: 0.01162816770374775 (nan) | > current_lr: 5e-06 | > step_time: 0.1503 (0.13642008450856394) | > loader_time: 0.0048 (0.004679433121339459)  --> TIME: 2025-02-26 00:20:49 -- STEP: 10124/36938 -- GLOBAL_STEP: 1007450 | > loss_text_ce: 0.03639564663171768 (0.0325989761474975) | > loss_mel_ce: 2.9560434818267822 (nan) | > loss: 0.011874758638441563 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13642471138548454) | > loader_time: 0.0046 (0.00467984429558761)  --> TIME: 2025-02-26 00:20:59 -- STEP: 10174/36938 -- GLOBAL_STEP: 1007500 | > loss_text_ce: 0.03484378382563591 (0.032592591916285725) | > loss_mel_ce: 3.1358580589294434 (nan) | > loss: 0.012582150287926197 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.13644345188178467) | > loader_time: 0.0047 (0.004679990934908485)  --> TIME: 2025-02-26 00:21:08 -- STEP: 10224/36938 -- GLOBAL_STEP: 1007550 | > loss_text_ce: 0.033307068049907684 (0.03259163360151121) | > loss_mel_ce: 3.40999174118042 (nan) | > loss: 0.013663885183632374 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13642731360240076) | > loader_time: 0.0049 (0.004680015368267574)  --> TIME: 2025-02-26 00:21:17 -- STEP: 10274/36938 -- GLOBAL_STEP: 1007600 | > loss_text_ce: 0.023951934650540352 (0.032589819574815916) | > loss_mel_ce: 1.828747272491455 (nan) | > loss: 0.007351981010288 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.1364131829653691) | > loader_time: 0.0046 (0.004679831614725442)  --> TIME: 2025-02-26 00:21:26 -- STEP: 10324/36938 -- GLOBAL_STEP: 1007650 | > loss_text_ce: 0.03279327601194382 (0.03259018692819426) | > loss_mel_ce: 3.0011885166168213 (nan) | > loss: 0.012039611116051674 (nan) | > current_lr: 5e-06 | > step_time: 0.1622 (0.136402724002601) | > loader_time: 0.0046 (0.004680049391690714)  --> TIME: 2025-02-26 00:21:35 -- STEP: 10374/36938 -- GLOBAL_STEP: 1007700 | > loss_text_ce: 0.034609537571668625 (0.03258577850140038) | > loss_mel_ce: 2.4470479488372803 (nan) | > loss: 0.009847847744822502 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13637473885668838) | > loader_time: 0.0046 (0.004679795402723939)  --> TIME: 2025-02-26 00:21:44 -- STEP: 10424/36938 -- GLOBAL_STEP: 1007750 | > loss_text_ce: 0.02372066117823124 (0.032587232074167034) | > loss_mel_ce: 2.9875762462615967 (nan) | > loss: 0.011949592269957066 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.13637090214197298) | > loader_time: 0.0047 (0.004679860011120519)  --> TIME: 2025-02-26 00:21:54 -- STEP: 10474/36938 -- GLOBAL_STEP: 1007800 | > loss_text_ce: 0.03859404847025871 (0.032588385276447286) | > loss_mel_ce: 2.225982666015625 (nan) | > loss: 0.008986416272819042 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13638218850927494) | > loader_time: 0.0046 (0.004679997822745318)  --> TIME: 2025-02-26 00:22:03 -- STEP: 10524/36938 -- GLOBAL_STEP: 1007850 | > loss_text_ce: 0.028146253898739815 (0.03258670740180329) | > loss_mel_ce: 2.8423054218292236 (nan) | > loss: 0.011390682309865952 (nan) | > current_lr: 5e-06 | > step_time: 0.1868 (0.1363635698184013) | > loader_time: 0.005 (0.004679835146255512)  --> TIME: 2025-02-26 00:22:12 -- STEP: 10574/36938 -- GLOBAL_STEP: 1007900 | > loss_text_ce: 0.042055852711200714 (0.0325911222574469) | > loss_mel_ce: 1.3261111974716187 (nan) | > loss: 0.005429234821349382 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13636763147264397) | > loader_time: 0.0047 (0.0046796483941219675)  --> TIME: 2025-02-26 00:22:21 -- STEP: 10624/36938 -- GLOBAL_STEP: 1007950 | > loss_text_ce: 0.032681919634342194 (0.03259533115251315) | > loss_mel_ce: 2.703692674636841 (nan) | > loss: 0.010858629830181599 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.1363409429519301) | > loader_time: 0.0046 (0.004679309024688699)  --> TIME: 2025-02-26 00:22:31 -- STEP: 10674/36938 -- GLOBAL_STEP: 1008000 | > loss_text_ce: 0.026145419105887413 (0.03259462229490442) | > loss_mel_ce: 2.6561641693115234 (nan) | > loss: 0.010644086636602879 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13636131594416578) | > loader_time: 0.0048 (0.004679377838000382)  --> TIME: 2025-02-26 00:22:40 -- STEP: 10724/36938 -- GLOBAL_STEP: 1008050 | > loss_text_ce: 0.032072000205516815 (0.03259819054672836) | > loss_mel_ce: 2.9444501399993896 (nan) | > loss: 0.011811596341431141 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.1363543324931181) | > loader_time: 0.0047 (0.004679265261496763)  --> TIME: 2025-02-26 00:22:49 -- STEP: 10774/36938 -- GLOBAL_STEP: 1008100 | > loss_text_ce: 0.026949895545840263 (0.03259549082006338) | > loss_mel_ce: 3.344764471054077 (nan) | > loss: 0.013379819691181183 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.1363592675162849) | > loader_time: 0.0049 (0.004679320782218314)  --> TIME: 2025-02-26 00:22:58 -- STEP: 10824/36938 -- GLOBAL_STEP: 1008150 | > loss_text_ce: 0.03879356011748314 (0.03259844065746057) | > loss_mel_ce: 3.3944520950317383 (nan) | > loss: 0.013623991049826145 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13634787371047682) | > loader_time: 0.0045 (0.004679415218052988)  --> TIME: 2025-02-26 00:23:08 -- STEP: 10874/36938 -- GLOBAL_STEP: 1008200 | > loss_text_ce: 0.03421308845281601 (0.03259552268418263) | > loss_mel_ce: 3.3396947383880615 (nan) | > loss: 0.013388523831963539 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13633738617391086) | > loader_time: 0.0047 (0.004679295427772462)  --> TIME: 2025-02-26 00:23:16 -- STEP: 10924/36938 -- GLOBAL_STEP: 1008250 | > loss_text_ce: 0.031352173537015915 (0.032595532832047205) | > loss_mel_ce: 2.213169574737549 (nan) | > loss: 0.008906832896173 (nan) | > current_lr: 5e-06 | > step_time: 0.1075 (0.13630867923235904) | > loader_time: 0.0045 (0.004678854244788092)  --> TIME: 2025-02-26 00:23:26 -- STEP: 10974/36938 -- GLOBAL_STEP: 1008300 | > loss_text_ce: 0.03192991390824318 (0.03259403883517381) | > loss_mel_ce: 2.4845163822174072 (nan) | > loss: 0.009985898621380329 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1362941182548293) | > loader_time: 0.0048 (0.004678861417469687)  --> TIME: 2025-02-26 00:23:35 -- STEP: 11024/36938 -- GLOBAL_STEP: 1008350 | > loss_text_ce: 0.02131813019514084 (0.032590517894187786) | > loss_mel_ce: 0.008153175935149193 (nan) | > loss: 0.00011694963177433237 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13632701616328857) | > loader_time: 0.0045 (0.004678868135796907)  --> TIME: 2025-02-26 00:23:44 -- STEP: 11074/36938 -- GLOBAL_STEP: 1008400 | > loss_text_ce: 0.02818600833415985 (0.03258976556464329) | > loss_mel_ce: 3.287088632583618 (nan) | > loss: 0.013155852444469929 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13630635367633534) | > loader_time: 0.0047 (0.004678855223067501)  --> TIME: 2025-02-26 00:23:54 -- STEP: 11124/36938 -- GLOBAL_STEP: 1008450 | > loss_text_ce: 0.03240910544991493 (0.032596083341185744) | > loss_mel_ce: 2.7148995399475098 (nan) | > loss: 0.01090201921761036 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1362944183397629) | > loader_time: 0.0048 (0.004679483567498122)  --> TIME: 2025-02-26 00:24:03 -- STEP: 11174/36938 -- GLOBAL_STEP: 1008500 | > loss_text_ce: 0.021102607250213623 (0.03259259940236564) | > loss_mel_ce: 1.9103314876556396 (nan) | > loss: 0.007664421573281288 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13631726218097878) | > loader_time: 0.0045 (0.004679589444801067)  --> TIME: 2025-02-26 00:24:13 -- STEP: 11224/36938 -- GLOBAL_STEP: 1008550 | > loss_text_ce: 0.03463887795805931 (0.03259639676606528) | > loss_mel_ce: 2.9504799842834473 (nan) | > loss: 0.011845710687339306 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13634162453014614) | > loader_time: 0.0047 (0.004679805048856229)  --> TIME: 2025-02-26 00:24:22 -- STEP: 11274/36938 -- GLOBAL_STEP: 1008600 | > loss_text_ce: 0.03641904518008232 (0.032598305129126676) | > loss_mel_ce: 2.3706588745117188 (nan) | > loss: 0.009551897644996643 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13633874637640403) | > loader_time: 0.0047 (0.0046799148421180015)  --> TIME: 2025-02-26 00:24:31 -- STEP: 11324/36938 -- GLOBAL_STEP: 1008650 | > loss_text_ce: 0.03401465713977814 (0.03259644025712356) | > loss_mel_ce: 3.0056865215301514 (nan) | > loss: 0.012062307447195053 (nan) | > current_lr: 5e-06 | > step_time: 0.1733 (0.1363376150363675) | > loader_time: 0.0048 (0.004680064616376787)  --> TIME: 2025-02-26 00:24:41 -- STEP: 11374/36938 -- GLOBAL_STEP: 1008700 | > loss_text_ce: 0.021617917343974113 (0.032590274476641534) | > loss_mel_ce: 2.2655975818634033 (nan) | > loss: 0.009076252579689026 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13634741834540334) | > loader_time: 0.0045 (0.004680025927605845)  --> TIME: 2025-02-26 00:24:50 -- STEP: 11424/36938 -- GLOBAL_STEP: 1008750 | > loss_text_ce: 0.026249339804053307 (0.032592419554487204) | > loss_mel_ce: 2.21325945854187 (nan) | > loss: 0.008886940777301788 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13636941882241777) | > loader_time: 0.0046 (0.004679932397286754)  --> TIME: 2025-02-26 00:25:00 -- STEP: 11474/36938 -- GLOBAL_STEP: 1008800 | > loss_text_ce: 0.028836838901042938 (0.032594471218748484) | > loss_mel_ce: 2.2307536602020264 (nan) | > loss: 0.008966629393398762 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1363593159973662) | > loader_time: 0.0046 (0.0046800304543832705)  --> TIME: 2025-02-26 00:25:09 -- STEP: 11524/36938 -- GLOBAL_STEP: 1008850 | > loss_text_ce: 0.027976976707577705 (0.03259965133811556) | > loss_mel_ce: 1.6817773580551147 (nan) | > loss: 0.006784739904105663 (nan) | > current_lr: 5e-06 | > step_time: 0.1123 (0.1363494632683551) | > loader_time: 0.0045 (0.004679843788550813)  --> TIME: 2025-02-26 00:25:18 -- STEP: 11574/36938 -- GLOBAL_STEP: 1008900 | > loss_text_ce: 0.038141872733831406 (0.03259709451208545) | > loss_mel_ce: 1.2780685424804688 (nan) | > loss: 0.005223057232797146 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1363453523923607) | > loader_time: 0.0046 (0.004679793827023259)  --> TIME: 2025-02-26 00:25:28 -- STEP: 11624/36938 -- GLOBAL_STEP: 1008950 | > loss_text_ce: 0.03817659616470337 (0.03259501942428294) | > loss_mel_ce: 2.5904626846313477 (nan) | > loss: 0.010431108996272087 (nan) | > current_lr: 5e-06 | > step_time: 0.1816 (0.1363807379050832) | > loader_time: 0.0048 (0.004680005296377657)  --> TIME: 2025-02-26 00:25:37 -- STEP: 11674/36938 -- GLOBAL_STEP: 1009000 | > loss_text_ce: 0.025983771309256554 (0.03259038757925346) | > loss_mel_ce: 2.9378437995910645 (nan) | > loss: 0.01176122110337019 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13637908376690883) | > loader_time: 0.0048 (0.004679900990868944)  --> TIME: 2025-02-26 00:25:46 -- STEP: 11724/36938 -- GLOBAL_STEP: 1009050 | > loss_text_ce: 0.03521725907921791 (0.032586162262292814) | > loss_mel_ce: 2.640244960784912 (nan) | > loss: 0.010616914369165897 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13636489613639513) | > loader_time: 0.0047 (0.00468007672650542)  --> TIME: 2025-02-26 00:25:56 -- STEP: 11774/36938 -- GLOBAL_STEP: 1009100 | > loss_text_ce: 0.03215992450714111 (0.03259123508418688) | > loss_mel_ce: 3.564053773880005 (nan) | > loss: 0.014270690269768238 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13638875422502922) | > loader_time: 0.0046 (0.004680273790847095)  --> TIME: 2025-02-26 00:26:05 -- STEP: 11824/36938 -- GLOBAL_STEP: 1009150 | > loss_text_ce: 0.028251998126506805 (0.032590984592733595) | > loss_mel_ce: 2.5425000190734863 (nan) | > loss: 0.010201397351920605 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1363779426790552) | > loader_time: 0.0045 (0.004680570109126037)  --> TIME: 2025-02-26 00:26:14 -- STEP: 11874/36938 -- GLOBAL_STEP: 1009200 | > loss_text_ce: 0.041930850595235825 (0.032588336335177465) | > loss_mel_ce: 2.1481823921203613 (nan) | > loss: 0.008690926246345043 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13638135984085803) | > loader_time: 0.0045 (0.004680257123067186)  --> TIME: 2025-02-26 00:26:24 -- STEP: 11924/36938 -- GLOBAL_STEP: 1009250 | > loss_text_ce: 0.0324581004679203 (0.03258720732361837) | > loss_mel_ce: 3.181748628616333 (nan) | > loss: 0.012754789553582668 (nan) | > current_lr: 5e-06 | > step_time: 0.177 (0.13640037000039437) | > loader_time: 0.0048 (0.004680285114683737)  --> TIME: 2025-02-26 00:26:33 -- STEP: 11974/36938 -- GLOBAL_STEP: 1009300 | > loss_text_ce: 0.04389479011297226 (0.03258589437256019) | > loss_mel_ce: 2.7448105812072754 (nan) | > loss: 0.011066291481256485 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13641427537724815) | > loader_time: 0.0046 (0.0046802891182823715)  --> TIME: 2025-02-26 00:26:43 -- STEP: 12024/36938 -- GLOBAL_STEP: 1009350 | > loss_text_ce: 0.03578965738415718 (0.032586533667337714) | > loss_mel_ce: 2.64400053024292 (nan) | > loss: 0.010634088888764381 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.1364080188517095) | > loader_time: 0.0048 (0.004680124704471354)  --> TIME: 2025-02-26 00:26:52 -- STEP: 12074/36938 -- GLOBAL_STEP: 1009400 | > loss_text_ce: 0.04633353278040886 (0.032593022276802934) | > loss_mel_ce: 2.687544107437134 (nan) | > loss: 0.010848721489310265 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.13643428154569984) | > loader_time: 0.0049 (0.00468033006180589)  --> TIME: 2025-02-26 00:27:01 -- STEP: 12124/36938 -- GLOBAL_STEP: 1009450 | > loss_text_ce: 0.03165321797132492 (0.03259212097966726) | > loss_mel_ce: 3.457996129989624 (nan) | > loss: 0.013847815804183483 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13641539234723432) | > loader_time: 0.0048 (0.004680500806105554)  --> TIME: 2025-02-26 00:27:11 -- STEP: 12174/36938 -- GLOBAL_STEP: 1009500 | > loss_text_ce: 0.03119940496981144 (0.03259779585604057) | > loss_mel_ce: 2.803499698638916 (nan) | > loss: 0.011248806491494179 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.1364170304656097) | > loader_time: 0.0046 (0.00468076699195637)  --> TIME: 2025-02-26 00:27:20 -- STEP: 12224/36938 -- GLOBAL_STEP: 1009550 | > loss_text_ce: 0.03585948050022125 (0.032597326246457385) | > loss_mel_ce: 2.5567710399627686 (nan) | > loss: 0.010288217104971409 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13642890006303876) | > loader_time: 0.0045 (0.004680740973712255)  --> TIME: 2025-02-26 00:27:30 -- STEP: 12274/36938 -- GLOBAL_STEP: 1009600 | > loss_text_ce: 0.03900314122438431 (0.03259491927846522) | > loss_mel_ce: 3.0635552406311035 (nan) | > loss: 0.012311739847064018 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1364348721228388) | > loader_time: 0.0046 (0.004680627193049994)  --> TIME: 2025-02-26 00:27:39 -- STEP: 12324/36938 -- GLOBAL_STEP: 1009650 | > loss_text_ce: 0.03985443338751793 (0.03259002150216877) | > loss_mel_ce: 3.0028913021087646 (nan) | > loss: 0.012074388563632965 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.13644433313668222) | > loader_time: 0.0046 (0.0046806839409295415)  --> TIME: 2025-02-26 00:27:48 -- STEP: 12374/36938 -- GLOBAL_STEP: 1009700 | > loss_text_ce: 0.02986432984471321 (0.032582666710921304) | > loss_mel_ce: 2.7296090126037598 (nan) | > loss: 0.010950291529297829 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13643744068151994) | > loader_time: 0.0049 (0.004680509557241528)  --> TIME: 2025-02-26 00:27:57 -- STEP: 12424/36938 -- GLOBAL_STEP: 1009750 | > loss_text_ce: 0.033410899341106415 (0.032588002323973804) | > loss_mel_ce: 2.5945589542388916 (nan) | > loss: 0.01042845193296671 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13642428597556572) | > loader_time: 0.0045 (0.004680315775020761)  --> TIME: 2025-02-26 00:28:07 -- STEP: 12474/36938 -- GLOBAL_STEP: 1009800 | > loss_text_ce: 0.03653973340988159 (0.0325890968774572) | > loss_mel_ce: 1.1728302240371704 (nan) | > loss: 0.004799087066203356 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1364275739612316) | > loader_time: 0.0047 (0.0046805264725409)  --> TIME: 2025-02-26 00:28:16 -- STEP: 12524/36938 -- GLOBAL_STEP: 1009850 | > loss_text_ce: 0.0294821634888649 (0.03258758216709073) | > loss_mel_ce: 2.9208250045776367 (nan) | > loss: 0.011707568541169167 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.13642052974384244) | > loader_time: 0.0047 (0.004680362934884981)  --> TIME: 2025-02-26 00:28:25 -- STEP: 12574/36938 -- GLOBAL_STEP: 1009900 | > loss_text_ce: 0.03119317628443241 (0.0325857279641509) | > loss_mel_ce: 3.227565050125122 (nan) | > loss: 0.01293158158659935 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.13643794814789909) | > loader_time: 0.0046 (0.004680423207933004)  --> TIME: 2025-02-26 00:28:35 -- STEP: 12624/36938 -- GLOBAL_STEP: 1009950 | > loss_text_ce: 0.030750814825296402 (0.03258564520901137) | > loss_mel_ce: 2.5586774349212646 (nan) | > loss: 0.01027550920844078 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.1364358290368923) | > loader_time: 0.0046 (0.004680270572276768)  --> TIME: 2025-02-26 00:28:44 -- STEP: 12674/36938 -- GLOBAL_STEP: 1010000 | > loss_text_ce: 0.0213889479637146 (0.03258079799509131) | > loss_mel_ce: 1.8905210494995117 (nan) | > loss: 0.007586945313960314 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13641716910213775) | > loader_time: 0.0045 (0.00468008902352447) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1010000.pth  --> TIME: 2025-02-26 00:28:59 -- STEP: 12724/36938 -- GLOBAL_STEP: 1010050 | > loss_text_ce: 0.030260900035500526 (0.03257974627255342) | > loss_mel_ce: 3.0977847576141357 (nan) | > loss: 0.01241287961602211 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13641190191590977) | > loader_time: 0.0047 (0.0046800605645909744)  --> TIME: 2025-02-26 00:29:08 -- STEP: 12774/36938 -- GLOBAL_STEP: 1010100 | > loss_text_ce: 0.03499094024300575 (0.03258180334077484) | > loss_mel_ce: 2.293759346008301 (nan) | > loss: 0.0092410733923316 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13639300433574503) | > loader_time: 0.0046 (0.004679952612949007)  --> TIME: 2025-02-26 00:29:17 -- STEP: 12824/36938 -- GLOBAL_STEP: 1010150 | > loss_text_ce: 0.021078135818243027 (0.03257972090688103) | > loss_mel_ce: 1.9694050550460815 (nan) | > loss: 0.007898743264377117 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13636824776809597) | > loader_time: 0.0046 (0.004680098962278118)  --> TIME: 2025-02-26 00:29:26 -- STEP: 12874/36938 -- GLOBAL_STEP: 1010200 | > loss_text_ce: 0.03390295058488846 (0.03258806413075187) | > loss_mel_ce: 2.256500720977783 (nan) | > loss: 0.009088903665542603 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13638347641955487) | > loader_time: 0.0046 (0.004680115835478126)  --> TIME: 2025-02-26 00:29:36 -- STEP: 12924/36938 -- GLOBAL_STEP: 1010250 | > loss_text_ce: 0.035317983478307724 (0.03258402709347946) | > loss_mel_ce: 2.708097219467163 (nan) | > loss: 0.010886568576097488 (nan) | > current_lr: 5e-06 | > step_time: 0.1879 (0.1363888038123449) | > loader_time: 0.0052 (0.004680192846884412)  --> TIME: 2025-02-26 00:29:45 -- STEP: 12974/36938 -- GLOBAL_STEP: 1010300 | > loss_text_ce: 0.04266330227255821 (0.0325844142553262) | > loss_mel_ce: 2.9798734188079834 (nan) | > loss: 0.011994194239377975 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13640901218012755) | > loader_time: 0.0047 (0.0046803353103665925)  --> TIME: 2025-02-26 00:29:55 -- STEP: 13024/36938 -- GLOBAL_STEP: 1010350 | > loss_text_ce: 0.0334298349916935 (0.032582697376331106) | > loss_mel_ce: 0.0238037109375 (nan) | > loss: 0.000227117256144993 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.1364288793588455) | > loader_time: 0.0045 (0.004680277711050515)  --> TIME: 2025-02-26 00:30:04 -- STEP: 13074/36938 -- GLOBAL_STEP: 1010400 | > loss_text_ce: 0.022808309644460678 (0.032578128801466114) | > loss_mel_ce: 1.2095626592636108 (nan) | > loss: 0.004890361335128546 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13643313502798632) | > loader_time: 0.0046 (0.004680474872743752)  --> TIME: 2025-02-26 00:30:14 -- STEP: 13124/36938 -- GLOBAL_STEP: 1010450 | > loss_text_ce: 0.03290247172117233 (0.03258269807261108) | > loss_mel_ce: 2.6445369720458984 (nan) | > loss: 0.010624760761857033 (nan) | > current_lr: 5e-06 | > step_time: 0.1614 (0.13644390640662907) | > loader_time: 0.0048 (0.004680442613859219)  --> TIME: 2025-02-26 00:30:23 -- STEP: 13174/36938 -- GLOBAL_STEP: 1010500 | > loss_text_ce: 0.03717292100191116 (0.03258290829228047) | > loss_mel_ce: 2.148954153060913 (nan) | > loss: 0.008675108663737774 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13644159073639367) | > loader_time: 0.0047 (0.004680394981561495)  --> TIME: 2025-02-26 00:30:32 -- STEP: 13224/36938 -- GLOBAL_STEP: 1010550 | > loss_text_ce: 0.028185192495584488 (0.03258137781060616) | > loss_mel_ce: 2.5277035236358643 (nan) | > loss: 0.010142415761947632 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.13643042235683217) | > loader_time: 0.0047 (0.004680320575463731)  --> TIME: 2025-02-26 00:30:42 -- STEP: 13274/36938 -- GLOBAL_STEP: 1010600 | > loss_text_ce: 0.031402233988046646 (0.032585343547994694) | > loss_mel_ce: 2.7637317180633545 (nan) | > loss: 0.011091802269220352 (nan) | > current_lr: 5e-06 | > step_time: 0.1774 (0.13643651344841987) | > loader_time: 0.0048 (0.004680425858163548)  --> TIME: 2025-02-26 00:30:51 -- STEP: 13324/36938 -- GLOBAL_STEP: 1010650 | > loss_text_ce: 0.02731509320437908 (0.032584141330996214) | > loss_mel_ce: 2.4044106006622314 (nan) | > loss: 0.009649706073105335 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13642642713459247) | > loader_time: 0.0046 (0.004680520204837822)  --> TIME: 2025-02-26 00:31:00 -- STEP: 13374/36938 -- GLOBAL_STEP: 1010700 | > loss_text_ce: 0.04006997123360634 (0.03258763482605895) | > loss_mel_ce: 3.1434593200683594 (nan) | > loss: 0.012633053585886955 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.13643102223730255) | > loader_time: 0.0049 (0.004680533695777158)  --> TIME: 2025-02-26 00:31:09 -- STEP: 13424/36938 -- GLOBAL_STEP: 1010750 | > loss_text_ce: 0.022039908915758133 (0.032589695576306116) | > loss_mel_ce: 2.3061258792877197 (nan) | > loss: 0.009238753467798233 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13641478552863856) | > loader_time: 0.0045 (0.00468041947615446)  --> TIME: 2025-02-26 00:31:19 -- STEP: 13474/36938 -- GLOBAL_STEP: 1010800 | > loss_text_ce: 0.03446377441287041 (0.03259237396239285) | > loss_mel_ce: 2.2561144828796387 (nan) | > loss: 0.009089597500860691 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.136421758112541) | > loader_time: 0.0046 (0.00468070089100412)  --> TIME: 2025-02-26 00:31:28 -- STEP: 13524/36938 -- GLOBAL_STEP: 1010850 | > loss_text_ce: 0.029337989166378975 (0.03259339322709566) | > loss_mel_ce: 2.7619831562042236 (nan) | > loss: 0.011076671071350574 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1364216089283925) | > loader_time: 0.0046 (0.0046805720715718865)  --> TIME: 2025-02-26 00:31:38 -- STEP: 13574/36938 -- GLOBAL_STEP: 1010900 | > loss_text_ce: 0.029919398948550224 (0.032596512137704876) | > loss_mel_ce: 3.1117916107177734 (nan) | > loss: 0.012467107735574245 (nan) | > current_lr: 5e-06 | > step_time: 0.1632 (0.13644452488122136) | > loader_time: 0.0046 (0.00468058185301984)  --> TIME: 2025-02-26 00:31:47 -- STEP: 13624/36938 -- GLOBAL_STEP: 1010950 | > loss_text_ce: 0.0351673848927021 (0.0325946615982494) | > loss_mel_ce: 0.0659993514418602 (nan) | > loss: 0.00040145532693713903 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13646065065319052) | > loader_time: 0.0048 (0.004680634909913748)  --> TIME: 2025-02-26 00:31:57 -- STEP: 13674/36938 -- GLOBAL_STEP: 1011000 | > loss_text_ce: 0.035736553370952606 (0.03259695090425168) | > loss_mel_ce: 1.733445405960083 (nan) | > loss: 0.007020563818514347 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13647020614616304) | > loader_time: 0.0045 (0.004680617800298737)  --> TIME: 2025-02-26 00:32:06 -- STEP: 13724/36938 -- GLOBAL_STEP: 1011050 | > loss_text_ce: 0.03698798269033432 (0.03259492448055552) | > loss_mel_ce: 3.0377843379974365 (nan) | > loss: 0.012201478704810143 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13648279539804367) | > loader_time: 0.0047 (0.004680915655043281)  --> TIME: 2025-02-26 00:32:15 -- STEP: 13774/36938 -- GLOBAL_STEP: 1011100 | > loss_text_ce: 0.03759573772549629 (0.032598910373136177) | > loss_mel_ce: 2.6526877880096436 (nan) | > loss: 0.010675729252398014 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13645919866203315) | > loader_time: 0.0046 (0.0046808202778702915)  --> TIME: 2025-02-26 00:32:25 -- STEP: 13824/36938 -- GLOBAL_STEP: 1011150 | > loss_text_ce: 0.022936074063181877 (0.03259802450156737) | > loss_mel_ce: 2.604736328125 (nan) | > loss: 0.010427271947264671 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13646842635892129) | > loader_time: 0.0047 (0.004681058814404198)  --> TIME: 2025-02-26 00:32:34 -- STEP: 13874/36938 -- GLOBAL_STEP: 1011200 | > loss_text_ce: 0.03744012862443924 (0.03259878417198323) | > loss_mel_ce: 3.1017379760742188 (nan) | > loss: 0.01245705597102642 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13646345558196818) | > loader_time: 0.0047 (0.004681090843235888)  --> TIME: 2025-02-26 00:32:43 -- STEP: 13924/36938 -- GLOBAL_STEP: 1011250 | > loss_text_ce: 0.03711707890033722 (0.03259655813118308) | > loss_mel_ce: 2.1558125019073486 (nan) | > loss: 0.008702102117240429 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13644821991080833) | > loader_time: 0.0046 (0.004681201013331107)  --> TIME: 2025-02-26 00:32:52 -- STEP: 13974/36938 -- GLOBAL_STEP: 1011300 | > loss_text_ce: 0.02708052285015583 (0.0325899930713247) | > loss_mel_ce: 2.914527654647827 (nan) | > loss: 0.011673049069941044 (nan) | > current_lr: 5e-06 | > step_time: 0.1621 (0.13644426478427893) | > loader_time: 0.0047 (0.004681336652811163)  --> TIME: 2025-02-26 00:33:02 -- STEP: 14024/36938 -- GLOBAL_STEP: 1011350 | > loss_text_ce: 0.028881791979074478 (0.03258574846388205) | > loss_mel_ce: 1.181431770324707 (nan) | > loss: 0.004802831914275885 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13644005413471663) | > loader_time: 0.0047 (0.004681276122434314)  --> TIME: 2025-02-26 00:33:11 -- STEP: 14074/36938 -- GLOBAL_STEP: 1011400 | > loss_text_ce: 0.03695446625351906 (0.03259094682568064) | > loss_mel_ce: 3.0664563179016113 (nan) | > loss: 0.012315122410655022 (nan) | > current_lr: 5e-06 | > step_time: 0.1874 (0.1364481851291271) | > loader_time: 0.0049 (0.004681621642042948)  --> TIME: 2025-02-26 00:33:21 -- STEP: 14124/36938 -- GLOBAL_STEP: 1011450 | > loss_text_ce: 0.03244669735431671 (0.03259094035531478) | > loss_mel_ce: 2.6224348545074463 (nan) | > loss: 0.010535244829952717 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13646617243965908) | > loader_time: 0.0046 (0.0046817597536688346)  --> TIME: 2025-02-26 00:33:30 -- STEP: 14174/36938 -- GLOBAL_STEP: 1011500 | > loss_text_ce: 0.029939908534288406 (0.03259180465623017) | > loss_mel_ce: 2.5397369861602783 (nan) | > loss: 0.010197130963206291 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.13646513998247808) | > loader_time: 0.0046 (0.004681776722816121)  --> TIME: 2025-02-26 00:33:39 -- STEP: 14224/36938 -- GLOBAL_STEP: 1011550 | > loss_text_ce: 0.033331748098134995 (0.03259059380370929) | > loss_mel_ce: 3.241238594055176 (nan) | > loss: 0.012994326651096344 (nan) | > current_lr: 5e-06 | > step_time: 0.1688 (0.13645107869825782) | > loader_time: 0.0047 (0.004681582140439819)  --> TIME: 2025-02-26 00:33:49 -- STEP: 14274/36938 -- GLOBAL_STEP: 1011600 | > loss_text_ce: 0.03772381320595741 (0.032591578216772776) | > loss_mel_ce: 3.055767774581909 (nan) | > loss: 0.012275760993361473 (nan) | > current_lr: 5e-06 | > step_time: 0.1878 (0.1364543749468667) | > loader_time: 0.005 (0.004681539415260893)  --> TIME: 2025-02-26 00:33:58 -- STEP: 14324/36938 -- GLOBAL_STEP: 1011650 | > loss_text_ce: 0.040420420467853546 (0.03258698193414413) | > loss_mel_ce: 2.81941819190979 (nan) | > loss: 0.011348566971719265 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13644023137663944) | > loader_time: 0.0047 (0.004681376147889395)  --> TIME: 2025-02-26 00:34:07 -- STEP: 14374/36938 -- GLOBAL_STEP: 1011700 | > loss_text_ce: 0.0351686105132103 (0.0325868832436888) | > loss_mel_ce: 1.924098014831543 (nan) | > loss: 0.007774868048727512 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13645589600456073) | > loader_time: 0.0046 (0.004681308975766438)  --> TIME: 2025-02-26 00:34:16 -- STEP: 14424/36938 -- GLOBAL_STEP: 1011750 | > loss_text_ce: 0.03631225973367691 (0.03258885785283132) | > loss_mel_ce: 4.792782783508301 (nan) | > loss: 0.01916307583451271 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13644869685900446) | > loader_time: 0.0046 (0.0046811868961956354)  --> TIME: 2025-02-26 00:34:26 -- STEP: 14474/36938 -- GLOBAL_STEP: 1011800 | > loss_text_ce: 0.031051278114318848 (0.03258572511792698) | > loss_mel_ce: 2.957064151763916 (nan) | > loss: 0.011857600882649422 (nan) | > current_lr: 5e-06 | > step_time: 0.1628 (0.13644935183059378) | > loader_time: 0.0047 (0.004681121945562369)  --> TIME: 2025-02-26 00:34:35 -- STEP: 14524/36938 -- GLOBAL_STEP: 1011850 | > loss_text_ce: 0.03258918225765228 (0.03258468581526967) | > loss_mel_ce: 3.005723714828491 (nan) | > loss: 0.012056797742843628 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13643485673759045) | > loader_time: 0.0047 (0.004681176175364654)  --> TIME: 2025-02-26 00:34:44 -- STEP: 14574/36938 -- GLOBAL_STEP: 1011900 | > loss_text_ce: 0.03213756904006004 (0.03258562348781763) | > loss_mel_ce: 3.367046356201172 (nan) | > loss: 0.013488826341927052 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.1364299643754612) | > loader_time: 0.0046 (0.004681183327629986)  --> TIME: 2025-02-26 00:34:54 -- STEP: 14624/36938 -- GLOBAL_STEP: 1011950 | > loss_text_ce: 0.03743607923388481 (0.032588752892153405) | > loss_mel_ce: 3.3885064125061035 (nan) | > loss: 0.01359501015394926 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13644521944306498) | > loader_time: 0.0047 (0.004681311270601128)  --> TIME: 2025-02-26 00:35:03 -- STEP: 14674/36938 -- GLOBAL_STEP: 1012000 | > loss_text_ce: 0.03766836225986481 (0.03258996073866694) | > loss_mel_ce: 2.496284008026123 (nan) | > loss: 0.010055367834866047 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13643700319040494) | > loader_time: 0.0046 (0.004681352765095151)  --> TIME: 2025-02-26 00:35:12 -- STEP: 14724/36938 -- GLOBAL_STEP: 1012050 | > loss_text_ce: 0.03688251972198486 (0.03258848128478801) | > loss_mel_ce: 2.6011624336242676 (nan) | > loss: 0.010468432679772377 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.1364201785137863) | > loader_time: 0.0048 (0.004681469062462163)  --> TIME: 2025-02-26 00:35:21 -- STEP: 14774/36938 -- GLOBAL_STEP: 1012100 | > loss_text_ce: 0.03470759466290474 (0.032587555298828785) | > loss_mel_ce: 2.89043927192688 (nan) | > loss: 0.011607726104557514 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.13642751568626194) | > loader_time: 0.0047 (0.004681561818477257)  --> TIME: 2025-02-26 00:35:31 -- STEP: 14824/36938 -- GLOBAL_STEP: 1012150 | > loss_text_ce: 0.026373354718089104 (0.032588018012520766) | > loss_mel_ce: 3.027616024017334 (nan) | > loss: 0.012119006365537643 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.13645318073320734) | > loader_time: 0.0067 (0.004681691358492182)  --> TIME: 2025-02-26 00:35:40 -- STEP: 14874/36938 -- GLOBAL_STEP: 1012200 | > loss_text_ce: 0.02435137704014778 (0.032591511196679565) | > loss_mel_ce: 2.083531618118286 (nan) | > loss: 0.00836461503058672 (nan) | > current_lr: 5e-06 | > step_time: 0.108 (0.1364469939988853) | > loader_time: 0.0045 (0.004681606181802435)  --> TIME: 2025-02-26 00:35:50 -- STEP: 14924/36938 -- GLOBAL_STEP: 1012250 | > loss_text_ce: 0.04231633245944977 (0.03259146736498717) | > loss_mel_ce: 2.408865451812744 (nan) | > loss: 0.009726912714540958 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.136447008861931) | > loader_time: 0.0046 (0.00468159873758428)  --> TIME: 2025-02-26 00:35:59 -- STEP: 14974/36938 -- GLOBAL_STEP: 1012300 | > loss_text_ce: 0.02604253962635994 (0.0325947352518351) | > loss_mel_ce: 2.557384490966797 (nan) | > loss: 0.010251695290207863 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.13645402387160888) | > loader_time: 0.0047 (0.004681577443025684)  --> TIME: 2025-02-26 00:36:08 -- STEP: 15024/36938 -- GLOBAL_STEP: 1012350 | > loss_text_ce: 0.03097371570765972 (0.03259261323532599) | > loss_mel_ce: 2.293226480484009 (nan) | > loss: 0.009223016910254955 (nan) | > current_lr: 5e-06 | > step_time: 0.1558 (0.1364473129550252) | > loader_time: 0.0046 (0.004681630922963454)  --> TIME: 2025-02-26 00:36:17 -- STEP: 15074/36938 -- GLOBAL_STEP: 1012400 | > loss_text_ce: 0.03344208002090454 (0.03259570128318395) | > loss_mel_ce: 2.5383589267730713 (nan) | > loss: 0.010205560363829136 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.1364358575935325) | > loader_time: 0.0046 (0.004681482213204945)  --> TIME: 2025-02-26 00:36:27 -- STEP: 15124/36938 -- GLOBAL_STEP: 1012450 | > loss_text_ce: 0.036528971046209335 (0.03259649916691932) | > loss_mel_ce: 2.622851610183716 (nan) | > loss: 0.010553098283708096 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13642790636919358) | > loader_time: 0.0046 (0.0046813073564351436)  --> TIME: 2025-02-26 00:36:36 -- STEP: 15174/36938 -- GLOBAL_STEP: 1012500 | > loss_text_ce: 0.029786620289087296 (0.03259515463623393) | > loss_mel_ce: 2.888474941253662 (nan) | > loss: 0.01158040389418602 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.13642976075682048) | > loader_time: 0.0049 (0.004681329930175895)  --> TIME: 2025-02-26 00:36:45 -- STEP: 15224/36938 -- GLOBAL_STEP: 1012550 | > loss_text_ce: 0.02872209995985031 (0.0325920330014897) | > loss_mel_ce: 3.1841928958892822 (nan) | > loss: 0.012749663554131985 (nan) | > current_lr: 5e-06 | > step_time: 0.1791 (0.13643865748511866) | > loader_time: 0.0048 (0.004681252581285174)  --> TIME: 2025-02-26 00:36:55 -- STEP: 15274/36938 -- GLOBAL_STEP: 1012600 | > loss_text_ce: 0.02784266695380211 (0.03258956787903871) | > loss_mel_ce: 2.061297655105591 (nan) | > loss: 0.008290240541100502 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13644435097601032) | > loader_time: 0.0046 (0.004681182825488861)  --> TIME: 2025-02-26 00:37:04 -- STEP: 15324/36938 -- GLOBAL_STEP: 1012650 | > loss_text_ce: 0.04679819568991661 (0.03258898343784686) | > loss_mel_ce: 3.444138765335083 (nan) | > loss: 0.013852925039827824 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.13645660072853077) | > loader_time: 0.0048 (0.004681392131104425)  --> TIME: 2025-02-26 00:37:14 -- STEP: 15374/36938 -- GLOBAL_STEP: 1012700 | > loss_text_ce: 0.0344056710600853 (0.03259089707966494) | > loss_mel_ce: 2.9440360069274902 (nan) | > loss: 0.011819213628768921 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13646640093068993) | > loader_time: 0.0047 (0.0046812248149339436)  --> TIME: 2025-02-26 00:37:23 -- STEP: 15424/36938 -- GLOBAL_STEP: 1012750 | > loss_text_ce: 0.03976909816265106 (0.03258788455051869) | > loss_mel_ce: 2.7320990562438965 (nan) | > loss: 0.010999477468430996 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13648022148619218) | > loader_time: 0.0046 (0.0046810968102755975)  --> TIME: 2025-02-26 00:37:33 -- STEP: 15474/36938 -- GLOBAL_STEP: 1012800 | > loss_text_ce: 0.019668906927108765 (0.032588045136261484) | > loss_mel_ce: 1.5724046230316162 (nan) | > loss: 0.006317752413451672 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13647362772032207) | > loader_time: 0.0044 (0.004681074096972566)  --> TIME: 2025-02-26 00:37:42 -- STEP: 15524/36938 -- GLOBAL_STEP: 1012850 | > loss_text_ce: 0.03551046550273895 (0.03258667453507749) | > loss_mel_ce: 3.1268796920776367 (nan) | > loss: 0.012549168430268764 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1364719107021426) | > loader_time: 0.0046 (0.004680936636306744)  --> TIME: 2025-02-26 00:37:51 -- STEP: 15574/36938 -- GLOBAL_STEP: 1012900 | > loss_text_ce: 0.039292529225349426 (0.032585171825850974) | > loss_mel_ce: 3.2678415775299072 (nan) | > loss: 0.013123548589646816 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13647467697847185) | > loader_time: 0.0046 (0.004680870493860329)  --> TIME: 2025-02-26 00:38:00 -- STEP: 15624/36938 -- GLOBAL_STEP: 1012950 | > loss_text_ce: 0.03462819755077362 (0.032585674043124076) | > loss_mel_ce: 1.5469902753829956 (nan) | > loss: 0.006276263855397701 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1364680130666074) | > loader_time: 0.0047 (0.00468076717469001)  --> TIME: 2025-02-26 00:38:10 -- STEP: 15674/36938 -- GLOBAL_STEP: 1013000 | > loss_text_ce: 0.04272989183664322 (0.03258928898957244) | > loss_mel_ce: 3.030200719833374 (nan) | > loss: 0.01219416968524456 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13649823891655813) | > loader_time: 0.0047 (0.004680945250518601)  --> TIME: 2025-02-26 00:38:20 -- STEP: 15724/36938 -- GLOBAL_STEP: 1013050 | > loss_text_ce: 0.03713894635438919 (0.032587763588852944) | > loss_mel_ce: 2.6768412590026855 (nan) | > loss: 0.01076976303011179 (nan) | > current_lr: 5e-06 | > step_time: 0.1717 (0.13650906031382395) | > loader_time: 0.0048 (0.004680984379892836)  --> TIME: 2025-02-26 00:38:29 -- STEP: 15774/36938 -- GLOBAL_STEP: 1013100 | > loss_text_ce: 0.03984854370355606 (0.032586284903676155) | > loss_mel_ce: 2.644462823867798 (nan) | > loss: 0.010652029886841774 (nan) | > current_lr: 5e-06 | > step_time: 0.1519 (0.13651736002995132) | > loader_time: 0.0046 (0.004680951647965866)  --> TIME: 2025-02-26 00:38:39 -- STEP: 15824/36938 -- GLOBAL_STEP: 1013150 | > loss_text_ce: 0.037413109093904495 (0.032585799243113917) | > loss_mel_ce: 2.0192747116088867 (nan) | > loss: 0.008161460049450397 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.1365330301534073) | > loader_time: 0.0046 (0.00468106509098752)  --> TIME: 2025-02-26 00:38:48 -- STEP: 15874/36938 -- GLOBAL_STEP: 1013200 | > loss_text_ce: 0.03607599064707756 (0.032588781121636365) | > loss_mel_ce: 2.9348995685577393 (nan) | > loss: 0.011789586395025253 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.13654560332065938) | > loader_time: 0.0047 (0.004681048577087458)  --> TIME: 2025-02-26 00:38:58 -- STEP: 15924/36938 -- GLOBAL_STEP: 1013250 | > loss_text_ce: 0.02749435044825077 (0.032591214178224234) | > loss_mel_ce: 2.6577250957489014 (nan) | > loss: 0.010655634105205536 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.136538572465318) | > loader_time: 0.0047 (0.004681026971510872)  --> TIME: 2025-02-26 00:39:07 -- STEP: 15974/36938 -- GLOBAL_STEP: 1013300 | > loss_text_ce: 0.027157936245203018 (0.032588828622798716) | > loss_mel_ce: 2.6294384002685547 (nan) | > loss: 0.010542050004005432 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13653444581147184) | > loader_time: 0.0047 (0.004681001277296487)  --> TIME: 2025-02-26 00:39:17 -- STEP: 16024/36938 -- GLOBAL_STEP: 1013350 | > loss_text_ce: 0.034315600991249084 (0.03258823954044144) | > loss_mel_ce: 0.7363985180854797 (nan) | > loss: 0.003058389527723193 (nan) | > current_lr: 5e-06 | > step_time: 0.1116 (0.13654889021286573) | > loader_time: 0.0045 (0.004681009815980726)  --> TIME: 2025-02-26 00:39:26 -- STEP: 16074/36938 -- GLOBAL_STEP: 1013400 | > loss_text_ce: 0.025287045165896416 (0.03258724803172454) | > loss_mel_ce: 2.7427501678466797 (nan) | > loss: 0.010984275490045547 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13655386756305252) | > loader_time: 0.0047 (0.004680938769353933)  --> TIME: 2025-02-26 00:39:36 -- STEP: 16124/36938 -- GLOBAL_STEP: 1013450 | > loss_text_ce: 0.041388873010873795 (0.03258502655827805) | > loss_mel_ce: 2.503375291824341 (nan) | > loss: 0.010098271071910858 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.1365698369405209) | > loader_time: 0.0046 (0.0046809278715044325)  --> TIME: 2025-02-26 00:39:45 -- STEP: 16174/36938 -- GLOBAL_STEP: 1013500 | > loss_text_ce: 0.0377378948032856 (0.03258594780152925) | > loss_mel_ce: 1.8021984100341797 (nan) | > loss: 0.007301334757357836 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13656788578696352) | > loader_time: 0.0045 (0.004680755156970702)  --> TIME: 2025-02-26 00:39:54 -- STEP: 16224/36938 -- GLOBAL_STEP: 1013550 | > loss_text_ce: 0.03552693873643875 (0.03259033729557397) | > loss_mel_ce: 0.11505126953125 (nan) | > loss: 0.0005975326057523489 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1365749608629805) | > loader_time: 0.0045 (0.004680714369760706)  --> TIME: 2025-02-26 00:40:04 -- STEP: 16274/36938 -- GLOBAL_STEP: 1013600 | > loss_text_ce: 0.03122505359351635 (0.03259062154017815) | > loss_mel_ce: 1.4873372316360474 (nan) | > loss: 0.006026041228324175 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13658484976605623) | > loader_time: 0.0045 (0.0046805539206760055)  --> TIME: 2025-02-26 00:40:13 -- STEP: 16324/36938 -- GLOBAL_STEP: 1013650 | > loss_text_ce: 0.034639354795217514 (0.03258569812262024) | > loss_mel_ce: 3.1499838829040527 (nan) | > loss: 0.012637394480407238 (nan) | > current_lr: 5e-06 | > step_time: 0.1785 (0.13660112879960548) | > loader_time: 0.0048 (0.004680557319329852)  --> TIME: 2025-02-26 00:40:23 -- STEP: 16374/36938 -- GLOBAL_STEP: 1013700 | > loss_text_ce: 0.029991954565048218 (0.032588014378575086) | > loss_mel_ce: 3.250218152999878 (nan) | > loss: 0.013016707263886929 (nan) | > current_lr: 5e-06 | > step_time: 0.1516 (0.13661103563400895) | > loader_time: 0.0047 (0.004680441487937741)  --> TIME: 2025-02-26 00:40:32 -- STEP: 16424/36938 -- GLOBAL_STEP: 1013750 | > loss_text_ce: 0.027195099741220474 (0.03258815056590412) | > loss_mel_ce: 2.529041051864624 (nan) | > loss: 0.010143795050680637 (nan) | > current_lr: 5e-06 | > step_time: 0.1577 (0.13661787324688152) | > loader_time: 0.0046 (0.004680491094756472)  --> TIME: 2025-02-26 00:40:42 -- STEP: 16474/36938 -- GLOBAL_STEP: 1013800 | > loss_text_ce: 0.022298695519566536 (0.032590570540200446) | > loss_mel_ce: 1.4762369394302368 (nan) | > loss: 0.0059465705417096615 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13661107314187412) | > loader_time: 0.0046 (0.00468053433651107)  --> TIME: 2025-02-26 00:40:51 -- STEP: 16524/36938 -- GLOBAL_STEP: 1013850 | > loss_text_ce: 0.021702684462070465 (0.03259396192188704) | > loss_mel_ce: 2.710845708847046 (nan) | > loss: 0.010843447409570217 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.1365983655279023) | > loader_time: 0.0048 (0.004680503182306162)  --> TIME: 2025-02-26 00:41:00 -- STEP: 16574/36938 -- GLOBAL_STEP: 1013900 | > loss_text_ce: 0.02512926049530506 (0.0325884312274179) | > loss_mel_ce: 2.3060667514801025 (nan) | > loss: 0.009250778704881668 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.1366070517156429) | > loader_time: 0.0046 (0.0046803378305058955)  --> TIME: 2025-02-26 00:41:09 -- STEP: 16624/36938 -- GLOBAL_STEP: 1013950 | > loss_text_ce: 0.025380229577422142 (0.03258660044021385) | > loss_mel_ce: 2.025087356567383 (nan) | > loss: 0.008136776275932789 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13659178127854843) | > loader_time: 0.0045 (0.004680087866980495)  --> TIME: 2025-02-26 00:41:19 -- STEP: 16674/36938 -- GLOBAL_STEP: 1014000 | > loss_text_ce: 0.030712973326444626 (0.03258975282594303) | > loss_mel_ce: 2.773017406463623 (nan) | > loss: 0.011125915683805943 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13661577572307393) | > loader_time: 0.0046 (0.004680165086874114)  --> TIME: 2025-02-26 00:41:29 -- STEP: 16724/36938 -- GLOBAL_STEP: 1014050 | > loss_text_ce: 0.03193289786577225 (0.03258684371379562) | > loss_mel_ce: 3.1102938652038574 (nan) | > loss: 0.012469153851270676 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13663228344786293) | > loader_time: 0.0045 (0.004680139857084387)  --> TIME: 2025-02-26 00:41:38 -- STEP: 16774/36938 -- GLOBAL_STEP: 1014100 | > loss_text_ce: 0.034124426543712616 (0.032585811110896916) | > loss_mel_ce: 0.7526680827140808 (nan) | > loss: 0.003122192807495594 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13662710111076254) | > loader_time: 0.0046 (0.0046800190776753615)  --> TIME: 2025-02-26 00:41:47 -- STEP: 16824/36938 -- GLOBAL_STEP: 1014150 | > loss_text_ce: 0.03329528868198395 (0.03258432643186924) | > loss_mel_ce: 2.766693592071533 (nan) | > loss: 0.011111067607998848 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13662840831977133) | > loader_time: 0.0046 (0.004679901099352407)  --> TIME: 2025-02-26 00:41:57 -- STEP: 16874/36938 -- GLOBAL_STEP: 1014200 | > loss_text_ce: 0.032887235283851624 (0.032589422031213726) | > loss_mel_ce: 3.0555591583251953 (nan) | > loss: 0.012255740351974964 (nan) | > current_lr: 5e-06 | > step_time: 0.1874 (0.13664193617954376) | > loader_time: 0.0048 (0.004679753413849771)  --> TIME: 2025-02-26 00:42:06 -- STEP: 16924/36938 -- GLOBAL_STEP: 1014250 | > loss_text_ce: 0.029011031612753868 (0.03259370462536209) | > loss_mel_ce: 2.35681414604187 (nan) | > loss: 0.009467560797929764 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13663697768204516) | > loader_time: 0.0048 (0.00467967164344581)  --> TIME: 2025-02-26 00:42:16 -- STEP: 16974/36938 -- GLOBAL_STEP: 1014300 | > loss_text_ce: 0.035410914570093155 (0.03259414093869831) | > loss_mel_ce: 2.8444738388061523 (nan) | > loss: 0.011428114958107471 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1366453116839339) | > loader_time: 0.0047 (0.004679774653719167)  --> TIME: 2025-02-26 00:42:25 -- STEP: 17024/36938 -- GLOBAL_STEP: 1014350 | > loss_text_ce: 0.03373154252767563 (0.032591591822147735) | > loss_mel_ce: 2.6060791015625 (nan) | > loss: 0.01047543901950121 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.13665513621461256) | > loader_time: 0.0045 (0.004679703370745026)  --> TIME: 2025-02-26 00:42:34 -- STEP: 17074/36938 -- GLOBAL_STEP: 1014400 | > loss_text_ce: 0.03815298154950142 (0.032586677507752265) | > loss_mel_ce: 3.3971798419952393 (nan) | > loss: 0.0136322733014822 (nan) | > current_lr: 5e-06 | > step_time: 0.1455 (0.13665004594552077) | > loader_time: 0.0054 (0.004679753278504637)  --> TIME: 2025-02-26 00:42:44 -- STEP: 17124/36938 -- GLOBAL_STEP: 1014450 | > loss_text_ce: 0.026237912476062775 (0.03258331837008587) | > loss_mel_ce: 3.3462023735046387 (nan) | > loss: 0.013382700271904469 (nan) | > current_lr: 5e-06 | > step_time: 0.1265 (0.13664750035358925) | > loader_time: 0.0046 (0.004679682098296004)  --> TIME: 2025-02-26 00:42:53 -- STEP: 17174/36938 -- GLOBAL_STEP: 1014500 | > loss_text_ce: 0.029762839898467064 (0.03258526227907139) | > loss_mel_ce: 2.3195459842681885 (nan) | > loss: 0.009322654455900192 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.13664010018926007) | > loader_time: 0.0049 (0.004679699847566887)  --> TIME: 2025-02-26 00:43:02 -- STEP: 17224/36938 -- GLOBAL_STEP: 1014550 | > loss_text_ce: 0.035500507801771164 (0.03258397978766421) | > loss_mel_ce: 3.375469446182251 (nan) | > loss: 0.013535595498979092 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13664964069835228) | > loader_time: 0.0045 (0.004679680299714621)  --> TIME: 2025-02-26 00:43:12 -- STEP: 17274/36938 -- GLOBAL_STEP: 1014600 | > loss_text_ce: 0.03761614114046097 (0.03258350404872049) | > loss_mel_ce: 1.7823486328125 (nan) | > loss: 0.007222082931548357 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13665830637689144) | > loader_time: 0.0046 (0.004679686688874438)  --> TIME: 2025-02-26 00:43:21 -- STEP: 17324/36938 -- GLOBAL_STEP: 1014650 | > loss_text_ce: 0.030229803174734116 (0.03258323538960284) | > loss_mel_ce: 2.9926135540008545 (nan) | > loss: 0.011995410546660423 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13665605684098645) | > loader_time: 0.0046 (0.004679631482258437)  --> TIME: 2025-02-26 00:43:30 -- STEP: 17374/36938 -- GLOBAL_STEP: 1014700 | > loss_text_ce: 0.04161573201417923 (0.0325837410660296) | > loss_mel_ce: 2.538546085357666 (nan) | > loss: 0.010238737799227238 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.13665489835103634) | > loader_time: 0.0047 (0.004679811540126975)  --> TIME: 2025-02-26 00:43:40 -- STEP: 17424/36938 -- GLOBAL_STEP: 1014750 | > loss_text_ce: 0.03260842338204384 (0.03258268124626194) | > loss_mel_ce: 3.2379448413848877 (nan) | > loss: 0.0129783870652318 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13665927868572614) | > loader_time: 0.0051 (0.004680031778836499)  --> TIME: 2025-02-26 00:43:49 -- STEP: 17474/36938 -- GLOBAL_STEP: 1014800 | > loss_text_ce: 0.034609824419021606 (0.03258090908322026) | > loss_mel_ce: 3.6608994007110596 (nan) | > loss: 0.014664719812572002 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13665936716882562) | > loader_time: 0.0046 (0.004680048236451797)  --> TIME: 2025-02-26 00:43:58 -- STEP: 17524/36938 -- GLOBAL_STEP: 1014850 | > loss_text_ce: 0.03344549611210823 (0.0325807183957559) | > loss_mel_ce: 2.8258399963378906 (nan) | > loss: 0.011346371844410896 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1366512633518293) | > loader_time: 0.0045 (0.004679899010530926)  --> TIME: 2025-02-26 00:44:07 -- STEP: 17574/36938 -- GLOBAL_STEP: 1014900 | > loss_text_ce: 0.05392306670546532 (0.03258246084518553) | > loss_mel_ce: 0.0372975654900074 (nan) | > loss: 0.0003619866620283574 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.1366513477490206) | > loader_time: 0.0045 (0.004679701889118843)  --> TIME: 2025-02-26 00:44:17 -- STEP: 17624/36938 -- GLOBAL_STEP: 1014950 | > loss_text_ce: 0.03349275141954422 (0.032584373580052686) | > loss_mel_ce: 2.9870893955230713 (nan) | > loss: 0.011986438184976578 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.13665016111167547) | > loader_time: 0.0048 (0.00467975482960155)  --> TIME: 2025-02-26 00:44:26 -- STEP: 17674/36938 -- GLOBAL_STEP: 1015000 | > loss_text_ce: 0.030135108157992363 (0.03258675286145671) | > loss_mel_ce: 2.667961597442627 (nan) | > loss: 0.010706733912229538 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.13666530861174264) | > loader_time: 0.0048 (0.004679759743662332)  --> TIME: 2025-02-26 00:44:36 -- STEP: 17724/36938 -- GLOBAL_STEP: 1015050 | > loss_text_ce: 0.03504586219787598 (0.032585234962836) | > loss_mel_ce: 0.677490234375 (nan) | > loss: 0.0028275244403630495 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13666098852562358) | > loader_time: 0.0045 (0.004679865571828931)  --> TIME: 2025-02-26 00:44:45 -- STEP: 17774/36938 -- GLOBAL_STEP: 1015100 | > loss_text_ce: 0.035008467733860016 (0.03258301647930393) | > loss_mel_ce: 2.7679808139801025 (nan) | > loss: 0.011122973635792732 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.13666079142800167) | > loader_time: 0.0049 (0.004679884673962387)  --> TIME: 2025-02-26 00:44:54 -- STEP: 17824/36938 -- GLOBAL_STEP: 1015150 | > loss_text_ce: 0.03555380925536156 (0.03258246377095234) | > loss_mel_ce: 2.0090959072113037 (nan) | > loss: 0.008113689720630646 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13665432561877952) | > loader_time: 0.0046 (0.004679777169570043)  --> TIME: 2025-02-26 00:45:04 -- STEP: 17874/36938 -- GLOBAL_STEP: 1015200 | > loss_text_ce: 0.04129733517765999 (0.03258358220349603) | > loss_mel_ce: 2.758246421813965 (nan) | > loss: 0.011109301820397377 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13666107711395165) | > loader_time: 0.0049 (0.004679804041946692)  --> TIME: 2025-02-26 00:45:13 -- STEP: 17924/36938 -- GLOBAL_STEP: 1015250 | > loss_text_ce: 0.03192069008946419 (0.032583154634082366) | > loss_mel_ce: 2.318838357925415 (nan) | > loss: 0.009328409098088741 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.1366515405022189) | > loader_time: 0.0047 (0.004680049576362427)  --> TIME: 2025-02-26 00:45:22 -- STEP: 17974/36938 -- GLOBAL_STEP: 1015300 | > loss_text_ce: 0.02511901780962944 (0.03258172194645997) | > loss_mel_ce: 1.750835657119751 (nan) | > loss: 0.0070474399253726006 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.1366446468553423) | > loader_time: 0.0045 (0.004680260503545446)  --> TIME: 2025-02-26 00:45:31 -- STEP: 18024/36938 -- GLOBAL_STEP: 1015350 | > loss_text_ce: 0.032354071736335754 (0.032581478794143866) | > loss_mel_ce: 2.0803661346435547 (nan) | > loss: 0.008383811451494694 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1366316621905704) | > loader_time: 0.0046 (0.004680279118190388)  --> TIME: 2025-02-26 00:45:40 -- STEP: 18074/36938 -- GLOBAL_STEP: 1015400 | > loss_text_ce: 0.034243520349264145 (0.032582428962502664) | > loss_mel_ce: 3.2045414447784424 (nan) | > loss: 0.012852322310209274 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13662336253085822) | > loader_time: 0.0046 (0.004680238836460897)  --> TIME: 2025-02-26 00:45:49 -- STEP: 18124/36938 -- GLOBAL_STEP: 1015450 | > loss_text_ce: 0.03906796872615814 (0.032581113025532155) | > loss_mel_ce: 2.4968953132629395 (nan) | > loss: 0.010063347406685352 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.1366127912399931) | > loader_time: 0.0047 (0.004680044773108552)  --> TIME: 2025-02-26 00:45:59 -- STEP: 18174/36938 -- GLOBAL_STEP: 1015500 | > loss_text_ce: 0.034073349088430405 (0.03258162671658103) | > loss_mel_ce: 1.066361904144287 (nan) | > loss: 0.004366806708276272 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1366126127233611) | > loader_time: 0.0046 (0.00467993306079898)  --> TIME: 2025-02-26 00:46:09 -- STEP: 18224/36938 -- GLOBAL_STEP: 1015550 | > loss_text_ce: 0.02996249683201313 (0.0325837052862436) | > loss_mel_ce: 2.7917399406433105 (nan) | > loss: 0.011197232641279697 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.1366439729510339) | > loader_time: 0.0046 (0.004680502600477297)  --> TIME: 2025-02-26 00:46:18 -- STEP: 18274/36938 -- GLOBAL_STEP: 1015600 | > loss_text_ce: 0.02970285341143608 (0.03258474671101566) | > loss_mel_ce: 2.7915053367614746 (nan) | > loss: 0.011195271275937557 (nan) | > current_lr: 5e-06 | > step_time: 0.1606 (0.13666210155512235) | > loader_time: 0.0048 (0.00468053018758449)  --> TIME: 2025-02-26 00:46:28 -- STEP: 18324/36938 -- GLOBAL_STEP: 1015650 | > loss_text_ce: 0.02488907240331173 (0.032584682077493134) | > loss_mel_ce: 0.4918212890625 (nan) | > loss: 0.0020504379644989967 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1366574933398279) | > loader_time: 0.0048 (0.004680428148955412)  --> TIME: 2025-02-26 00:46:36 -- STEP: 18374/36938 -- GLOBAL_STEP: 1015700 | > loss_text_ce: 0.022573014721274376 (0.03257863097555015) | > loss_mel_ce: 1.687732458114624 (nan) | > loss: 0.0067869266495108604 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13663745946355887) | > loader_time: 0.0046 (0.004680297898171827)  --> TIME: 2025-02-26 00:46:46 -- STEP: 18424/36938 -- GLOBAL_STEP: 1015750 | > loss_text_ce: 0.03849395737051964 (0.03258088236091131) | > loss_mel_ce: 2.585399866104126 (nan) | > loss: 0.010412277653813362 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.1366442458116133) | > loader_time: 0.0045 (0.004680260828252584)  --> TIME: 2025-02-26 00:46:55 -- STEP: 18474/36938 -- GLOBAL_STEP: 1015800 | > loss_text_ce: 0.03968963399529457 (0.032578866008029575) | > loss_mel_ce: 2.3777503967285156 (nan) | > loss: 0.009593016467988491 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13662807914593156) | > loader_time: 0.0045 (0.004680190236586274)  --> TIME: 2025-02-26 00:47:04 -- STEP: 18524/36938 -- GLOBAL_STEP: 1015850 | > loss_text_ce: 0.03374657407402992 (0.03257843515265774) | > loss_mel_ce: 2.514756202697754 (nan) | > loss: 0.010113106109201908 (nan) | > current_lr: 5e-06 | > step_time: 0.1721 (0.1366375795298284) | > loader_time: 0.0048 (0.0046801182369618655)  --> TIME: 2025-02-26 00:47:14 -- STEP: 18574/36938 -- GLOBAL_STEP: 1015900 | > loss_text_ce: 0.028194032609462738 (0.03257897053602049) | > loss_mel_ce: 2.150381326675415 (nan) | > loss: 0.008645140565931797 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.1366317715418415) | > loader_time: 0.0047 (0.004680059294250343)  --> TIME: 2025-02-26 00:47:23 -- STEP: 18624/36938 -- GLOBAL_STEP: 1015950 | > loss_text_ce: 0.02766943909227848 (0.032580061984651396) | > loss_mel_ce: 2.684523582458496 (nan) | > loss: 0.010762671008706093 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13662211773629132) | > loader_time: 0.0046 (0.004680121119079735)  --> TIME: 2025-02-26 00:47:32 -- STEP: 18674/36938 -- GLOBAL_STEP: 1016000 | > loss_text_ce: 0.027145465835928917 (0.03258302540674852) | > loss_mel_ce: 2.6790549755096436 (nan) | > loss: 0.010738890618085861 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13661139395937483) | > loader_time: 0.0046 (0.0046801061871297855)  --> TIME: 2025-02-26 00:47:41 -- STEP: 18724/36938 -- GLOBAL_STEP: 1016050 | > loss_text_ce: 0.03924758732318878 (0.032580930661301556) | > loss_mel_ce: 2.7916622161865234 (nan) | > loss: 0.011233769357204437 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.13660437827486271) | > loader_time: 0.0048 (0.004680026586023683)  --> TIME: 2025-02-26 00:47:50 -- STEP: 18774/36938 -- GLOBAL_STEP: 1016100 | > loss_text_ce: 0.029757341369986534 (0.032584010369357895) | > loss_mel_ce: 2.618197441101074 (nan) | > loss: 0.010507757775485516 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.13660449120416077) | > loader_time: 0.0049 (0.004679975804777679)  --> TIME: 2025-02-26 00:48:00 -- STEP: 18824/36938 -- GLOBAL_STEP: 1016150 | > loss_text_ce: 0.02814621478319168 (0.03257980120412443) | > loss_mel_ce: 2.3908183574676514 (nan) | > loss: 0.009599066339433193 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.136600856128978) | > loader_time: 0.0046 (0.004679972004292638)  --> TIME: 2025-02-26 00:48:09 -- STEP: 18874/36938 -- GLOBAL_STEP: 1016200 | > loss_text_ce: 0.0313350185751915 (0.032579276441378716) | > loss_mel_ce: 3.0907301902770996 (nan) | > loss: 0.012389148585498333 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13660794848340807) | > loader_time: 0.0046 (0.004680038584836214)  --> TIME: 2025-02-26 00:48:19 -- STEP: 18924/36938 -- GLOBAL_STEP: 1016250 | > loss_text_ce: 0.032356228679418564 (0.0325768490295261) | > loss_mel_ce: 1.677001953125 (nan) | > loss: 0.006783167831599712 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.1366231087977132) | > loader_time: 0.0045 (0.004680039073316795)  --> TIME: 2025-02-26 00:48:29 -- STEP: 18974/36938 -- GLOBAL_STEP: 1016300 | > loss_text_ce: 0.03359629213809967 (0.032577570557978795) | > loss_mel_ce: 2.794379949569702 (nan) | > loss: 0.011222128756344318 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13663844670462363) | > loader_time: 0.0049 (0.004680226823483699)  --> TIME: 2025-02-26 00:48:38 -- STEP: 19024/36938 -- GLOBAL_STEP: 1016350 | > loss_text_ce: 0.030677465721964836 (0.032578110638733795) | > loss_mel_ce: 2.6401102542877197 (nan) | > loss: 0.010598365217447281 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13663857449645125) | > loader_time: 0.0045 (0.004680191472962308)  --> TIME: 2025-02-26 00:48:47 -- STEP: 19074/36938 -- GLOBAL_STEP: 1016400 | > loss_text_ce: 0.05085974186658859 (0.03257764848140558) | > loss_mel_ce: 2.548527717590332 (nan) | > loss: 0.010315029881894588 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13663771985723966) | > loader_time: 0.0046 (0.004680239505533851)  --> TIME: 2025-02-26 00:48:57 -- STEP: 19124/36938 -- GLOBAL_STEP: 1016450 | > loss_text_ce: 0.032729849219322205 (0.03257816382960436) | > loss_mel_ce: 2.504491090774536 (nan) | > loss: 0.010068337433040142 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13663611080925342) | > loader_time: 0.0048 (0.004680321882818836)  --> TIME: 2025-02-26 00:49:06 -- STEP: 19174/36938 -- GLOBAL_STEP: 1016500 | > loss_text_ce: 0.031565356999635696 (0.03258057586518262) | > loss_mel_ce: 3.0105137825012207 (nan) | > loss: 0.012071743607521057 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13663878511236116) | > loader_time: 0.0046 (0.0046803019424045585)  --> TIME: 2025-02-26 00:49:15 -- STEP: 19224/36938 -- GLOBAL_STEP: 1016550 | > loss_text_ce: 0.0436088964343071 (0.03258111969682372) | > loss_mel_ce: 2.8719489574432373 (nan) | > loss: 0.01156967505812645 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.1366303313711119) | > loader_time: 0.0048 (0.004680286334843842)  --> TIME: 2025-02-26 00:49:24 -- STEP: 19274/36938 -- GLOBAL_STEP: 1016600 | > loss_text_ce: 0.03966787829995155 (0.032579584452072964) | > loss_mel_ce: 2.869997262954712 (nan) | > loss: 0.01154629047960043 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13663420252153347) | > loader_time: 0.0047 (0.004680279949659698)  --> TIME: 2025-02-26 00:49:34 -- STEP: 19324/36938 -- GLOBAL_STEP: 1016650 | > loss_text_ce: 0.0343853235244751 (0.03258240660084791) | > loss_mel_ce: 2.854037284851074 (nan) | > loss: 0.011461994610726833 (nan) | > current_lr: 5e-06 | > step_time: 0.1707 (0.13663005992952237) | > loader_time: 0.0046 (0.004680255966585307)  --> TIME: 2025-02-26 00:49:43 -- STEP: 19374/36938 -- GLOBAL_STEP: 1016700 | > loss_text_ce: 0.021627962589263916 (0.03257992947860621) | > loss_mel_ce: 2.775787353515625 (nan) | > loss: 0.011100854724645615 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13662885451718076) | > loader_time: 0.0046 (0.004680162873124903)  --> TIME: 2025-02-26 00:49:52 -- STEP: 19424/36938 -- GLOBAL_STEP: 1016750 | > loss_text_ce: 0.026034988462924957 (0.03257468375322707) | > loss_mel_ce: 2.119166135787964 (nan) | > loss: 0.008512703701853752 (nan) | > current_lr: 5e-06 | > step_time: 0.1438 (0.13662891350134274) | > loader_time: 0.0047 (0.004680112753020498)  --> TIME: 2025-02-26 00:50:02 -- STEP: 19474/36938 -- GLOBAL_STEP: 1016800 | > loss_text_ce: 0.03297585994005203 (0.03257384590028501) | > loss_mel_ce: 2.9375360012054443 (nan) | > loss: 0.011787746101617813 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13664180658014746) | > loader_time: 0.0049 (0.00468007026052188)  --> TIME: 2025-02-26 00:50:11 -- STEP: 19524/36938 -- GLOBAL_STEP: 1016850 | > loss_text_ce: 0.03697871044278145 (0.03257401330215088) | > loss_mel_ce: 2.2907521724700928 (nan) | > loss: 0.00923702772706747 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13664095815980232) | > loader_time: 0.0046 (0.004680260664809579)  --> TIME: 2025-02-26 00:50:21 -- STEP: 19574/36938 -- GLOBAL_STEP: 1016900 | > loss_text_ce: 0.035628773272037506 (0.0325767761772465) | > loss_mel_ce: 3.0660197734832764 (nan) | > loss: 0.012308130040764809 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13663695189126354) | > loader_time: 0.0047 (0.004680283164373905)  --> TIME: 2025-02-26 00:50:30 -- STEP: 19624/36938 -- GLOBAL_STEP: 1016950 | > loss_text_ce: 0.04257027432322502 (0.03258061625639193) | > loss_mel_ce: 2.7433605194091797 (nan) | > loss: 0.011055282317101955 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13662828284528813) | > loader_time: 0.0046 (0.004680257061283027)  --> TIME: 2025-02-26 00:50:39 -- STEP: 19674/36938 -- GLOBAL_STEP: 1017000 | > loss_text_ce: 0.033516451716423035 (0.032580337404178206) | > loss_mel_ce: 2.2437005043029785 (nan) | > loss: 0.009036575444042683 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13663544156292806) | > loader_time: 0.0048 (0.004680402627669442)  --> TIME: 2025-02-26 00:50:49 -- STEP: 19724/36938 -- GLOBAL_STEP: 1017050 | > loss_text_ce: 0.028684066608548164 (0.03257959296355907) | > loss_mel_ce: 2.6720523834228516 (nan) | > loss: 0.010717209428548813 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.13663831824005063) | > loader_time: 0.0049 (0.004680354414835269)  --> TIME: 2025-02-26 00:50:58 -- STEP: 19774/36938 -- GLOBAL_STEP: 1017100 | > loss_text_ce: 0.03262424096465111 (0.03257878068125253) | > loss_mel_ce: 2.6250665187835693 (nan) | > loss: 0.010546392761170864 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13663736350184072) | > loader_time: 0.0046 (0.004680415129196172)  --> TIME: 2025-02-26 00:51:07 -- STEP: 19824/36938 -- GLOBAL_STEP: 1017150 | > loss_text_ce: 0.039139524102211 (0.032577875052938535) | > loss_mel_ce: 3.247316837310791 (nan) | > loss: 0.01304149441421032 (nan) | > current_lr: 5e-06 | > step_time: 0.1874 (0.13664119288361426) | > loader_time: 0.0049 (0.004680466938153516)  --> TIME: 2025-02-26 00:51:17 -- STEP: 19874/36938 -- GLOBAL_STEP: 1017200 | > loss_text_ce: 0.026910029351711273 (0.03257590420140213) | > loss_mel_ce: 2.5688881874084473 (nan) | > loss: 0.010300787165760994 (nan) | > current_lr: 5e-06 | > step_time: 0.182 (0.1366504021330096) | > loader_time: 0.0049 (0.004680505818112407)  --> TIME: 2025-02-26 00:51:26 -- STEP: 19924/36938 -- GLOBAL_STEP: 1017250 | > loss_text_ce: 0.038528162986040115 (0.0325734242441179) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13664197687211505) | > loader_time: 0.0046 (0.004680336706371742)  --> TIME: 2025-02-26 00:51:35 -- STEP: 19974/36938 -- GLOBAL_STEP: 1017300 | > loss_text_ce: 0.03736179322004318 (0.03257023576051909) | > loss_mel_ce: 0.024749755859375 (nan) | > loss: 0.00024647440295666456 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13664272657037116) | > loader_time: 0.0047 (0.004680508391282067)  --> TIME: 2025-02-26 00:51:44 -- STEP: 20024/36938 -- GLOBAL_STEP: 1017350 | > loss_text_ce: 0.02286657691001892 (0.032568704143538324) | > loss_mel_ce: 0.8895173668861389 (nan) | > loss: 0.0036205712240189314 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13663437313620153) | > loader_time: 0.0045 (0.00468056839178242)  --> TIME: 2025-02-26 00:51:53 -- STEP: 20074/36938 -- GLOBAL_STEP: 1017400 | > loss_text_ce: 0.03316640853881836 (0.03256909316461336) | > loss_mel_ce: 2.339653491973877 (nan) | > loss: 0.009415952488780022 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.13663225181313968) | > loader_time: 0.0047 (0.004680640528588553)  --> TIME: 2025-02-26 00:52:03 -- STEP: 20124/36938 -- GLOBAL_STEP: 1017450 | > loss_text_ce: 0.03206988796591759 (0.03257041737947993) | > loss_mel_ce: 2.600661277770996 (nan) | > loss: 0.010447346605360508 (nan) | > current_lr: 5e-06 | > step_time: 0.1816 (0.13663212832672883) | > loader_time: 0.0051 (0.004680684548300099)  --> TIME: 2025-02-26 00:52:12 -- STEP: 20174/36938 -- GLOBAL_STEP: 1017500 | > loss_text_ce: 0.030825430527329445 (0.03257189201893882) | > loss_mel_ce: 2.7050018310546875 (nan) | > loss: 0.010856457985937595 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.13662463239272946) | > loader_time: 0.0048 (0.00468065019664084)  --> TIME: 2025-02-26 00:52:21 -- STEP: 20224/36938 -- GLOBAL_STEP: 1017550 | > loss_text_ce: 0.03282234072685242 (0.03257390238742367) | > loss_mel_ce: 2.9835004806518555 (nan) | > loss: 0.011969535611569881 (nan) | > current_lr: 5e-06 | > step_time: 0.1565 (0.13662478630725758) | > loader_time: 0.0046 (0.004680741542973869)  --> TIME: 2025-02-26 00:52:31 -- STEP: 20274/36938 -- GLOBAL_STEP: 1017600 | > loss_text_ce: 0.030092772096395493 (0.03257346654856924) | > loss_mel_ce: 2.9747872352600098 (nan) | > loss: 0.011924127116799355 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.1366288446399301) | > loader_time: 0.0046 (0.0046807942310945665)  --> TIME: 2025-02-26 00:52:40 -- STEP: 20324/36938 -- GLOBAL_STEP: 1017650 | > loss_text_ce: 0.0345279723405838 (0.03257539730044875) | > loss_mel_ce: 2.6302692890167236 (nan) | > loss: 0.01057459320873022 (nan) | > current_lr: 5e-06 | > step_time: 0.1818 (0.1366251025864332) | > loader_time: 0.0049 (0.004680811068458474)  --> TIME: 2025-02-26 00:52:50 -- STEP: 20374/36938 -- GLOBAL_STEP: 1017700 | > loss_text_ce: 0.03638487681746483 (0.032577825454239734) | > loss_mel_ce: 2.2361769676208496 (nan) | > loss: 0.009018102660775185 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1366395516637573) | > loader_time: 0.0046 (0.0046810708406852366)  --> TIME: 2025-02-26 00:52:59 -- STEP: 20424/36938 -- GLOBAL_STEP: 1017750 | > loss_text_ce: 0.032701361924409866 (0.032581474387748686) | > loss_mel_ce: 2.647913932800293 (nan) | > loss: 0.010637362487614155 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.1366309808181218) | > loader_time: 0.0046 (0.004681124133406284)  --> TIME: 2025-02-26 00:53:08 -- STEP: 20474/36938 -- GLOBAL_STEP: 1017800 | > loss_text_ce: 0.035205185413360596 (0.03257890160049876) | > loss_mel_ce: 3.367812395095825 (nan) | > loss: 0.01350403856486082 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.1366392293339242) | > loader_time: 0.0051 (0.004681217841621362)  --> TIME: 2025-02-26 00:53:18 -- STEP: 20524/36938 -- GLOBAL_STEP: 1017850 | > loss_text_ce: 0.03360666334629059 (0.03258100892785594) | > loss_mel_ce: 1.9436614513397217 (nan) | > loss: 0.007846302352845669 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13664675939155196) | > loader_time: 0.0046 (0.004681333873360708)  --> TIME: 2025-02-26 00:53:27 -- STEP: 20574/36938 -- GLOBAL_STEP: 1017900 | > loss_text_ce: 0.03346963971853256 (0.032584335425244124) | > loss_mel_ce: 3.0527217388153076 (nan) | > loss: 0.012246792204678059 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.13663947988995795) | > loader_time: 0.0051 (0.004681237112204575)  --> TIME: 2025-02-26 00:53:36 -- STEP: 20624/36938 -- GLOBAL_STEP: 1017950 | > loss_text_ce: 0.02999253384768963 (0.0325830535769634) | > loss_mel_ce: 2.98571515083313 (nan) | > loss: 0.011967094615101814 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13664284746160504) | > loader_time: 0.0045 (0.004681178726273256)  --> TIME: 2025-02-26 00:53:46 -- STEP: 20674/36938 -- GLOBAL_STEP: 1018000 | > loss_text_ce: 0.037999190390110016 (0.03258159001191807) | > loss_mel_ce: 3.040076732635498 (nan) | > loss: 0.01221458800137043 (nan) | > current_lr: 5e-06 | > step_time: 0.1605 (0.13664118385850132) | > loader_time: 0.0047 (0.004681142603300473)  --> TIME: 2025-02-26 00:53:55 -- STEP: 20724/36938 -- GLOBAL_STEP: 1018050 | > loss_text_ce: 0.039431314915418625 (0.03258185536594178) | > loss_mel_ce: 2.9828944206237793 (nan) | > loss: 0.011993356980383396 (nan) | > current_lr: 5e-06 | > step_time: 0.19 (0.13664309750926834) | > loader_time: 0.0049 (0.004681078089040559)  --> TIME: 2025-02-26 00:54:04 -- STEP: 20774/36938 -- GLOBAL_STEP: 1018100 | > loss_text_ce: 0.0300537571310997 (0.032581060768438704) | > loss_mel_ce: 0.0322265625 (nan) | > loss: 0.0002471441403031349 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13663351261502227) | > loader_time: 0.0046 (0.004681151629619013)  --> TIME: 2025-02-26 00:54:13 -- STEP: 20824/36938 -- GLOBAL_STEP: 1018150 | > loss_text_ce: 0.03572256863117218 (0.032580660316061584) | > loss_mel_ce: 2.491489887237549 (nan) | > loss: 0.010028621181845665 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.1366327526910104) | > loader_time: 0.0046 (0.004681090529130811)  --> TIME: 2025-02-26 00:54:23 -- STEP: 20874/36938 -- GLOBAL_STEP: 1018200 | > loss_text_ce: 0.04354133829474449 (0.03258372185383436) | > loss_mel_ce: 2.4756133556365967 (nan) | > loss: 0.009996647015213966 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13663695336439077) | > loader_time: 0.0045 (0.004681101278908338)  --> TIME: 2025-02-26 00:54:32 -- STEP: 20924/36938 -- GLOBAL_STEP: 1018250 | > loss_text_ce: 0.03511064499616623 (0.032582276192887324) | > loss_mel_ce: 2.5208840370178223 (nan) | > loss: 0.010142836719751358 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13663802060769267) | > loader_time: 0.0048 (0.0046811473116640774)  --> TIME: 2025-02-26 00:54:42 -- STEP: 20974/36938 -- GLOBAL_STEP: 1018300 | > loss_text_ce: 0.036231983453035355 (0.03258414410868687) | > loss_mel_ce: 2.76619815826416 (nan) | > loss: 0.011120755225419998 (nan) | > current_lr: 5e-06 | > step_time: 0.1724 (0.13665241502631162) | > loader_time: 0.0047 (0.004681290406636099)  --> TIME: 2025-02-26 00:54:51 -- STEP: 21024/36938 -- GLOBAL_STEP: 1018350 | > loss_text_ce: 0.03471516817808151 (0.03258102500180943) | > loss_mel_ce: 2.645009994506836 (nan) | > loss: 0.010633830912411213 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13666774077366572) | > loader_time: 0.0048 (0.004681300003325413)  --> TIME: 2025-02-26 00:55:00 -- STEP: 21074/36938 -- GLOBAL_STEP: 1018400 | > loss_text_ce: 0.02884526737034321 (0.03258044469978382) | > loss_mel_ce: 2.527646064758301 (nan) | > loss: 0.010144807398319244 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1366570312315917) | > loader_time: 0.0046 (0.004681294801803848)  --> TIME: 2025-02-26 00:55:09 -- STEP: 21124/36938 -- GLOBAL_STEP: 1018450 | > loss_text_ce: 0.03523802384734154 (0.032580970641240686) | > loss_mel_ce: 2.745342969894409 (nan) | > loss: 0.011034051887691021 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13664581826003697) | > loader_time: 0.0046 (0.004681177356888316)  --> TIME: 2025-02-26 00:55:19 -- STEP: 21174/36938 -- GLOBAL_STEP: 1018500 | > loss_text_ce: 0.01577422395348549 (0.03258002856547502) | > loss_mel_ce: 1.3563307523727417 (nan) | > loss: 0.0054448614828288555 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13664061556775023) | > loader_time: 0.0045 (0.004681139027440713)  --> TIME: 2025-02-26 00:55:28 -- STEP: 21224/36938 -- GLOBAL_STEP: 1018550 | > loss_text_ce: 0.0282474122941494 (0.03257925051045587) | > loss_mel_ce: 2.7194509506225586 (nan) | > loss: 0.010903565213084221 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.13662785379292694) | > loader_time: 0.0047 (0.004681043700366702)  --> TIME: 2025-02-26 00:55:37 -- STEP: 21274/36938 -- GLOBAL_STEP: 1018600 | > loss_text_ce: 0.029783446341753006 (0.032576845664250106) | > loss_mel_ce: 2.7849388122558594 (nan) | > loss: 0.011169533245265484 (nan) | > current_lr: 5e-06 | > step_time: 0.1752 (0.13664133408424517) | > loader_time: 0.0049 (0.00468107616698554)  --> TIME: 2025-02-26 00:55:47 -- STEP: 21324/36938 -- GLOBAL_STEP: 1018650 | > loss_text_ce: 0.025764908641576767 (0.03257317990381701) | > loss_mel_ce: 0.10091552883386612 (nan) | > loss: 0.0005027001607231796 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.1366374364849159) | > loader_time: 0.0046 (0.004681128137130939)  --> TIME: 2025-02-26 00:55:56 -- STEP: 21374/36938 -- GLOBAL_STEP: 1018700 | > loss_text_ce: 0.031562600284814835 (0.03257664283386343) | > loss_mel_ce: 2.643472671508789 (nan) | > loss: 0.010615219362080097 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13663816586997218) | > loader_time: 0.0045 (0.004681071497118643)  --> TIME: 2025-02-26 00:56:05 -- STEP: 21424/36938 -- GLOBAL_STEP: 1018750 | > loss_text_ce: 0.0390750989317894 (0.0325754262815432) | > loss_mel_ce: 2.4505488872528076 (nan) | > loss: 0.00987946055829525 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13663654432223846) | > loader_time: 0.0046 (0.004681172779997094)  --> TIME: 2025-02-26 00:56:15 -- STEP: 21474/36938 -- GLOBAL_STEP: 1018800 | > loss_text_ce: 0.04569471627473831 (0.03257670152533059) | > loss_mel_ce: 2.797327995300293 (nan) | > loss: 0.011281837709248066 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.1366375861603586) | > loader_time: 0.0048 (0.004681199891746921)  --> TIME: 2025-02-26 00:56:24 -- STEP: 21524/36938 -- GLOBAL_STEP: 1018850 | > loss_text_ce: 0.024594368413090706 (0.03257561675590479) | > loss_mel_ce: 3.049955129623413 (nan) | > loss: 0.012200593948364258 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13664254010598828) | > loader_time: 0.0046 (0.0046812402140893085)  --> TIME: 2025-02-26 00:56:33 -- STEP: 21574/36938 -- GLOBAL_STEP: 1018900 | > loss_text_ce: 0.023255761712789536 (0.03258011675171008) | > loss_mel_ce: 2.900660753250122 (nan) | > loss: 0.01160284411162138 (nan) | > current_lr: 5e-06 | > step_time: 0.1969 (0.13664741616458273) | > loader_time: 0.0049 (0.004681292782128703)  --> TIME: 2025-02-26 00:56:43 -- STEP: 21624/36938 -- GLOBAL_STEP: 1018950 | > loss_text_ce: 0.03096677176654339 (0.032577391676646125) | > loss_mel_ce: 3.37229061126709 (nan) | > loss: 0.013504990376532078 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.1366401906812638) | > loader_time: 0.0047 (0.004681221300083534)  --> TIME: 2025-02-26 00:56:52 -- STEP: 21674/36938 -- GLOBAL_STEP: 1019000 | > loss_text_ce: 0.026240097358822823 (0.03257771221618517) | > loss_mel_ce: 2.0460991859436035 (nan) | > loss: 0.00822356902062893 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13663743596527197) | > loader_time: 0.0045 (0.004681354014759365)  --> TIME: 2025-02-26 00:57:01 -- STEP: 21724/36938 -- GLOBAL_STEP: 1019050 | > loss_text_ce: 0.02875538542866707 (0.0325807944347263) | > loss_mel_ce: 2.8475699424743652 (nan) | > loss: 0.011413990519940853 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13664807439882146) | > loader_time: 0.0047 (0.004681276004192859)  --> TIME: 2025-02-26 00:57:11 -- STEP: 21774/36938 -- GLOBAL_STEP: 1019100 | > loss_text_ce: 0.02958519384264946 (0.0325811216363033) | > loss_mel_ce: 2.1860835552215576 (nan) | > loss: 0.008792336098849773 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13664944413674793) | > loader_time: 0.0046 (0.004681549322356612)  --> TIME: 2025-02-26 00:57:20 -- STEP: 21824/36938 -- GLOBAL_STEP: 1019150 | > loss_text_ce: 0.032159920781850815 (0.03257805522076539) | > loss_mel_ce: 1.6206134557724 (nan) | > loss: 0.006558625027537346 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13664190922358607) | > loader_time: 0.0046 (0.004681516952179081)  --> TIME: 2025-02-26 00:57:29 -- STEP: 21874/36938 -- GLOBAL_STEP: 1019200 | > loss_text_ce: 0.030003610998392105 (0.03257498945315366) | > loss_mel_ce: 2.5172083377838135 (nan) | > loss: 0.010107983835041523 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.136627898973119) | > loader_time: 0.0046 (0.004681535685772013)  --> TIME: 2025-02-26 00:57:38 -- STEP: 21924/36938 -- GLOBAL_STEP: 1019250 | > loss_text_ce: 0.036586664617061615 (0.03257617010813821) | > loss_mel_ce: 3.0279369354248047 (nan) | > loss: 0.01216080877929926 (nan) | > current_lr: 5e-06 | > step_time: 0.1267 (0.13661628811491297) | > loader_time: 0.0048 (0.004681609088411949)  --> TIME: 2025-02-26 00:57:47 -- STEP: 21974/36938 -- GLOBAL_STEP: 1019300 | > loss_text_ce: 0.030228683724999428 (0.032575589483672514) | > loss_mel_ce: 2.7122855186462402 (nan) | > loss: 0.01088299322873354 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.1366222554020497) | > loader_time: 0.0047 (0.004681692996189615)  --> TIME: 2025-02-26 00:57:57 -- STEP: 22024/36938 -- GLOBAL_STEP: 1019350 | > loss_text_ce: 0.03232580050826073 (0.032574277677535415) | > loss_mel_ce: 3.0908203125 (nan) | > loss: 0.012393437325954437 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13661738199578075) | > loader_time: 0.0047 (0.004681691814239747)  --> TIME: 2025-02-26 00:58:06 -- STEP: 22074/36938 -- GLOBAL_STEP: 1019400 | > loss_text_ce: 0.035028815269470215 (0.032575695174523904) | > loss_mel_ce: 1.516522765159607 (nan) | > loss: 0.006156951189041138 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13661106014917118) | > loader_time: 0.0046 (0.004681801303439483)  --> TIME: 2025-02-26 00:58:15 -- STEP: 22124/36938 -- GLOBAL_STEP: 1019450 | > loss_text_ce: 0.0446961335837841 (0.03257455087649601) | > loss_mel_ce: 1.5768961906433105 (nan) | > loss: 0.006434890441596508 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13661159286445645) | > loader_time: 0.0046 (0.004682117453795708)  --> TIME: 2025-02-26 00:58:24 -- STEP: 22174/36938 -- GLOBAL_STEP: 1019500 | > loss_text_ce: 0.03358197584748268 (0.032574101059549) | > loss_mel_ce: 3.0150272846221924 (nan) | > loss: 0.012097656726837158 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13660739609405387) | > loader_time: 0.0045 (0.004681973146813851)  --> TIME: 2025-02-26 00:58:34 -- STEP: 22224/36938 -- GLOBAL_STEP: 1019550 | > loss_text_ce: 0.025940826162695885 (0.03257350089932644) | > loss_mel_ce: 2.6839587688446045 (nan) | > loss: 0.010753571055829525 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.13660793817455344) | > loader_time: 0.0047 (0.0046818539382229175)  --> TIME: 2025-02-26 00:58:43 -- STEP: 22274/36938 -- GLOBAL_STEP: 1019600 | > loss_text_ce: 0.03674490004777908 (0.032570513572725814) | > loss_mel_ce: 2.056887149810791 (nan) | > loss: 0.008308064192533493 (nan) | > current_lr: 5e-06 | > step_time: 0.1267 (0.1365998224905118) | > loader_time: 0.0046 (0.004681708248190191)  --> TIME: 2025-02-26 00:58:52 -- STEP: 22324/36938 -- GLOBAL_STEP: 1019650 | > loss_text_ce: 0.029587239027023315 (0.03256969385647896) | > loss_mel_ce: 2.441446304321289 (nan) | > loss: 0.009805689565837383 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1365998779930673) | > loader_time: 0.0045 (0.004681741063514973)  --> TIME: 2025-02-26 00:59:01 -- STEP: 22374/36938 -- GLOBAL_STEP: 1019700 | > loss_text_ce: 0.027416076511144638 (0.03256761422963554) | > loss_mel_ce: 2.9201409816741943 (nan) | > loss: 0.011696655303239822 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.13660740237501665) | > loader_time: 0.0049 (0.004681746399390938)  --> TIME: 2025-02-26 00:59:11 -- STEP: 22424/36938 -- GLOBAL_STEP: 1019750 | > loss_text_ce: 0.026345273479819298 (0.032567208309164736) | > loss_mel_ce: 1.1701231002807617 (nan) | > loss: 0.004747890401631594 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13661739004751378) | > loader_time: 0.0046 (0.004681833048526914)  --> TIME: 2025-02-26 00:59:21 -- STEP: 22474/36938 -- GLOBAL_STEP: 1019800 | > loss_text_ce: 0.018264085054397583 (0.03256473165446194) | > loss_mel_ce: 2.2431206703186035 (nan) | > loss: 0.008973749354481697 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13661892821252714) | > loader_time: 0.0047 (0.004681768531507411)  --> TIME: 2025-02-26 00:59:30 -- STEP: 22524/36938 -- GLOBAL_STEP: 1019850 | > loss_text_ce: 0.03843284025788307 (0.03256373947992869) | > loss_mel_ce: 2.5269558429718018 (nan) | > loss: 0.010180114768445492 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13661342037865254) | > loader_time: 0.0045 (0.004681633952413013)  --> TIME: 2025-02-26 00:59:39 -- STEP: 22574/36938 -- GLOBAL_STEP: 1019900 | > loss_text_ce: 0.030942121520638466 (0.03256231964440961) | > loss_mel_ce: 2.6758317947387695 (nan) | > loss: 0.010741167701780796 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13662570068049892) | > loader_time: 0.0048 (0.00468173769099771)  --> TIME: 2025-02-26 00:59:48 -- STEP: 22624/36938 -- GLOBAL_STEP: 1019950 | > loss_text_ce: 0.02994769997894764 (0.03256235546103925) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13661647010515654) | > loader_time: 0.0046 (0.004681633714233504)  --> TIME: 2025-02-26 00:59:57 -- STEP: 22674/36938 -- GLOBAL_STEP: 1020000 | > loss_text_ce: 0.042770061641931534 (0.032561571402303396) | > loss_mel_ce: 2.9744858741760254 (nan) | > loss: 0.011973238550126553 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13661212776875975) | > loader_time: 0.0047 (0.004681576788294049) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1020000.pth  --> TIME: 2025-02-26 01:00:13 -- STEP: 22724/36938 -- GLOBAL_STEP: 1020050 | > loss_text_ce: 0.042082779109478 (0.032562612249482835) | > loss_mel_ce: 2.870252847671509 (nan) | > loss: 0.011556887999176979 (nan) | > current_lr: 5e-06 | > step_time: 0.1491 (0.13661245837259536) | > loader_time: 0.0046 (0.004681583137878205)  --> TIME: 2025-02-26 01:00:22 -- STEP: 22774/36938 -- GLOBAL_STEP: 1020100 | > loss_text_ce: 0.03692817687988281 (0.03256277912341732) | > loss_mel_ce: 3.0887794494628906 (nan) | > loss: 0.012403602711856365 (nan) | > current_lr: 5e-06 | > step_time: 0.1537 (0.13660518518184805) | > loader_time: 0.0047 (0.004681541857523066)  --> TIME: 2025-02-26 01:00:31 -- STEP: 22824/36938 -- GLOBAL_STEP: 1020150 | > loss_text_ce: 0.031216297298669815 (0.03256374459903502) | > loss_mel_ce: 2.319075584411621 (nan) | > loss: 0.009326555766165257 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13660473233141815) | > loader_time: 0.0048 (0.004681531260233402)  --> TIME: 2025-02-26 01:00:41 -- STEP: 22874/36938 -- GLOBAL_STEP: 1020200 | > loss_text_ce: 0.03151136636734009 (0.03256068817549339) | > loss_mel_ce: 3.0223488807678223 (nan) | > loss: 0.01211849320679903 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13660776009559283) | > loader_time: 0.0046 (0.004681471376618395)  --> TIME: 2025-02-26 01:00:50 -- STEP: 22924/36938 -- GLOBAL_STEP: 1020250 | > loss_text_ce: 0.02707580476999283 (0.032562585979263724) | > loss_mel_ce: 2.677262306213379 (nan) | > loss: 0.010731500573456287 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13660414748074587) | > loader_time: 0.0045 (0.004681482560096742)  --> TIME: 2025-02-26 01:00:59 -- STEP: 22974/36938 -- GLOBAL_STEP: 1020300 | > loss_text_ce: 0.03764510899782181 (0.032559271782744766) | > loss_mel_ce: 2.5792675018310547 (nan) | > loss: 0.01038457453250885 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13659798967171072) | > loader_time: 0.0046 (0.004681409012402756)  --> TIME: 2025-02-26 01:01:08 -- STEP: 23024/36938 -- GLOBAL_STEP: 1020350 | > loss_text_ce: 0.0373101569712162 (0.03255805757321924) | > loss_mel_ce: 2.249055862426758 (nan) | > loss: 0.009072881191968918 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13660112128976892) | > loader_time: 0.0046 (0.004681411532580357)  --> TIME: 2025-02-26 01:01:18 -- STEP: 23074/36938 -- GLOBAL_STEP: 1020400 | > loss_text_ce: 0.032218288630247116 (0.03255844782085969) | > loss_mel_ce: 1.99468994140625 (nan) | > loss: 0.008043286390602589 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13660797086447787) | > loader_time: 0.0046 (0.004681370788816297)  --> TIME: 2025-02-26 01:01:27 -- STEP: 23124/36938 -- GLOBAL_STEP: 1020450 | > loss_text_ce: 0.038394760340452194 (0.03255912915382512) | > loss_mel_ce: 2.941183090209961 (nan) | > loss: 0.011823722161352634 (nan) | > current_lr: 5e-06 | > step_time: 0.1737 (0.13660968145972036) | > loader_time: 0.0048 (0.004681312889402048)  --> TIME: 2025-02-26 01:01:36 -- STEP: 23174/36938 -- GLOBAL_STEP: 1020500 | > loss_text_ce: 0.03600631654262543 (0.03256084669911519) | > loss_mel_ce: 2.694506883621216 (nan) | > loss: 0.010835370048880577 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13660922866947142) | > loader_time: 0.0045 (0.004681324685135389)  --> TIME: 2025-02-26 01:01:46 -- STEP: 23224/36938 -- GLOBAL_STEP: 1020550 | > loss_text_ce: 0.028490036725997925 (0.0325605121750448) | > loss_mel_ce: 0.52734375 (nan) | > loss: 0.0022056899033486843 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13661067513733058) | > loader_time: 0.0046 (0.004681220505921755)  --> TIME: 2025-02-26 01:01:55 -- STEP: 23274/36938 -- GLOBAL_STEP: 1020600 | > loss_text_ce: 0.042380440980196 (0.032560797509142245) | > loss_mel_ce: 2.905935287475586 (nan) | > loss: 0.011699666269123554 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13660664448595194) | > loader_time: 0.0048 (0.004681136893521332)  --> TIME: 2025-02-26 01:02:04 -- STEP: 23324/36938 -- GLOBAL_STEP: 1020650 | > loss_text_ce: 0.03545524552464485 (0.03256095860976292) | > loss_mel_ce: 2.8437254428863525 (nan) | > loss: 0.011425320990383625 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13660160345435138) | > loader_time: 0.0046 (0.004681063779854857)  --> TIME: 2025-02-26 01:02:13 -- STEP: 23374/36938 -- GLOBAL_STEP: 1020700 | > loss_text_ce: 0.03006182797253132 (0.03255719247206268) | > loss_mel_ce: 2.6235435009002686 (nan) | > loss: 0.010530180297791958 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13659658009312228) | > loader_time: 0.0047 (0.004680965203178931)  --> TIME: 2025-02-26 01:02:23 -- STEP: 23424/36938 -- GLOBAL_STEP: 1020750 | > loss_text_ce: 0.028619471937417984 (0.03255393758914422) | > loss_mel_ce: 2.695272445678711 (nan) | > loss: 0.010809095576405525 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13659558752602652) | > loader_time: 0.0046 (0.004680906173065752)  --> TIME: 2025-02-26 01:02:32 -- STEP: 23474/36938 -- GLOBAL_STEP: 1020800 | > loss_text_ce: 0.03384600579738617 (0.032554508339028856) | > loss_mel_ce: 2.8798694610595703 (nan) | > loss: 0.011562363244593143 (nan) | > current_lr: 5e-06 | > step_time: 0.1959 (0.1366108937371972) | > loader_time: 0.0049 (0.004680926769100951)  --> TIME: 2025-02-26 01:02:42 -- STEP: 23524/36938 -- GLOBAL_STEP: 1020850 | > loss_text_ce: 0.03230198100209236 (0.03255832957605364) | > loss_mel_ce: 2.5103259086608887 (nan) | > loss: 0.010089793242514133 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13661406031475545) | > loader_time: 0.0047 (0.004680927848556497)  --> TIME: 2025-02-26 01:02:51 -- STEP: 23574/36938 -- GLOBAL_STEP: 1020900 | > loss_text_ce: 0.03261756896972656 (0.032557475591128655) | > loss_mel_ce: 3.0712456703186035 (nan) | > loss: 0.01231691800057888 (nan) | > current_lr: 5e-06 | > step_time: 0.1699 (0.13662328660017933) | > loader_time: 0.0049 (0.004680961610666754)  --> TIME: 2025-02-26 01:03:01 -- STEP: 23624/36938 -- GLOBAL_STEP: 1020950 | > loss_text_ce: 0.0383157841861248 (0.03255441113059888) | > loss_mel_ce: 3.162000894546509 (nan) | > loss: 0.012699670158326626 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.1366274793532361) | > loader_time: 0.0047 (0.004681095243751777)  --> TIME: 2025-02-26 01:03:10 -- STEP: 23674/36938 -- GLOBAL_STEP: 1021000 | > loss_text_ce: 0.04434636980295181 (0.03255369067460359) | > loss_mel_ce: 2.950178623199463 (nan) | > loss: 0.011883036233484745 (nan) | > current_lr: 5e-06 | > step_time: 0.1846 (0.1366276135565097) | > loader_time: 0.0048 (0.00468110382592167)  --> TIME: 2025-02-26 01:03:20 -- STEP: 23724/36938 -- GLOBAL_STEP: 1021050 | > loss_text_ce: 0.03183789551258087 (0.03255256001735157) | > loss_mel_ce: 2.610257387161255 (nan) | > loss: 0.010484505444765091 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13663165951394293) | > loader_time: 0.0048 (0.004681059942742376)  --> TIME: 2025-02-26 01:03:29 -- STEP: 23774/36938 -- GLOBAL_STEP: 1021100 | > loss_text_ce: 0.04132188856601715 (0.032552867741998094) | > loss_mel_ce: 2.1511154174804688 (nan) | > loss: 0.00870014913380146 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13662172964386107) | > loader_time: 0.0045 (0.004680954548551861)  --> TIME: 2025-02-26 01:03:38 -- STEP: 23824/36938 -- GLOBAL_STEP: 1021150 | > loss_text_ce: 0.04010283201932907 (0.03255432747929745) | > loss_mel_ce: 1.8410487174987793 (nan) | > loss: 0.007464887574315071 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1366179160538587) | > loader_time: 0.0045 (0.0046807893015859935)  --> TIME: 2025-02-26 01:03:47 -- STEP: 23874/36938 -- GLOBAL_STEP: 1021200 | > loss_text_ce: 0.03692925348877907 (0.03255675784133909) | > loss_mel_ce: 2.2916765213012695 (nan) | > loss: 0.009240499697625637 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13662133392021863) | > loader_time: 0.0046 (0.004680729615405742)  --> TIME: 2025-02-26 01:03:57 -- STEP: 23924/36938 -- GLOBAL_STEP: 1021250 | > loss_text_ce: 0.03170419856905937 (0.03255634879338515) | > loss_mel_ce: 2.8149266242980957 (nan) | > loss: 0.01129615493118763 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13662423558945624) | > loader_time: 0.0046 (0.004680762839464849)  --> TIME: 2025-02-26 01:04:06 -- STEP: 23974/36938 -- GLOBAL_STEP: 1021300 | > loss_text_ce: 0.04089805483818054 (0.03255443839785798) | > loss_mel_ce: 3.2379097938537598 (nan) | > loss: 0.013011143542826176 (nan) | > current_lr: 5e-06 | > step_time: 0.1705 (0.1366236876474774) | > loader_time: 0.0048 (0.004680759655958931)  --> TIME: 2025-02-26 01:04:15 -- STEP: 24024/36938 -- GLOBAL_STEP: 1021350 | > loss_text_ce: 0.03804437816143036 (0.032554056264341935) | > loss_mel_ce: 2.7383217811584473 (nan) | > loss: 0.011017327196896076 (nan) | > current_lr: 5e-06 | > step_time: 0.169 (0.13662395917333311) | > loader_time: 0.0048 (0.004680750551042771)  --> TIME: 2025-02-26 01:04:25 -- STEP: 24074/36938 -- GLOBAL_STEP: 1021400 | > loss_text_ce: 0.03192269802093506 (0.0325545494526218) | > loss_mel_ce: 0.4474690854549408 (nan) | > loss: 0.0019023484783247113 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13661960152343056) | > loader_time: 0.0045 (0.004680860722947412)  --> TIME: 2025-02-26 01:04:34 -- STEP: 24124/36938 -- GLOBAL_STEP: 1021450 | > loss_text_ce: 0.040140580385923386 (0.03255545449805435) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13661828947944987) | > loader_time: 0.0046 (0.004680829950676444)  --> TIME: 2025-02-26 01:04:43 -- STEP: 24174/36938 -- GLOBAL_STEP: 1021500 | > loss_text_ce: 0.024506062269210815 (0.03255732008130808) | > loss_mel_ce: 2.2232344150543213 (nan) | > loss: 0.008919605985283852 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.13661423505284703) | > loader_time: 0.0048 (0.0046807710197533)  --> TIME: 2025-02-26 01:04:52 -- STEP: 24224/36938 -- GLOBAL_STEP: 1021550 | > loss_text_ce: 0.03149067237973213 (0.032558523227875585) | > loss_mel_ce: 2.936279296875 (nan) | > loss: 0.01177686545997858 (nan) | > current_lr: 5e-06 | > step_time: 0.1783 (0.136610962273345) | > loader_time: 0.0048 (0.004680717617391472)  --> TIME: 2025-02-26 01:05:02 -- STEP: 24274/36938 -- GLOBAL_STEP: 1021600 | > loss_text_ce: 0.03627879172563553 (0.0325604360528005) | > loss_mel_ce: 2.6662817001342773 (nan) | > loss: 0.01072444673627615 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13661297670824796) | > loader_time: 0.0047 (0.0046807787333307555)  --> TIME: 2025-02-26 01:05:11 -- STEP: 24324/36938 -- GLOBAL_STEP: 1021650 | > loss_text_ce: 0.027151281014084816 (0.03256148025118575) | > loss_mel_ce: 1.7471646070480347 (nan) | > loss: 0.007040936034172773 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13661550703144493) | > loader_time: 0.0045 (0.0046807836494327)  --> TIME: 2025-02-26 01:05:21 -- STEP: 24374/36938 -- GLOBAL_STEP: 1021700 | > loss_text_ce: 0.04388608783483505 (0.03256452941268718) | > loss_mel_ce: 2.234344244003296 (nan) | > loss: 0.009040597826242447 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1366222750282752) | > loader_time: 0.0045 (0.004680726920804255)  --> TIME: 2025-02-26 01:05:30 -- STEP: 24424/36938 -- GLOBAL_STEP: 1021750 | > loss_text_ce: 0.031015433371067047 (0.03256579244028427) | > loss_mel_ce: 2.5873494148254395 (nan) | > loss: 0.01039033755660057 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.1366368490693777) | > loader_time: 0.0047 (0.004680725499680154)  --> TIME: 2025-02-26 01:05:39 -- STEP: 24474/36938 -- GLOBAL_STEP: 1021800 | > loss_text_ce: 0.030767979100346565 (0.03256673631905597) | > loss_mel_ce: 1.7939941883087158 (nan) | > loss: 0.007241120561957359 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1366282780205326) | > loader_time: 0.0046 (0.004680600004218815)  --> TIME: 2025-02-26 01:05:49 -- STEP: 24524/36938 -- GLOBAL_STEP: 1021850 | > loss_text_ce: 0.03477859869599342 (0.032565706789995104) | > loss_mel_ce: 3.11579966545105 (nan) | > loss: 0.012502294965088367 (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.13663499896045403) | > loader_time: 0.0049 (0.004680527858721736)  --> TIME: 2025-02-26 01:05:58 -- STEP: 24574/36938 -- GLOBAL_STEP: 1021900 | > loss_text_ce: 0.029704617336392403 (0.03256417183090875) | > loss_mel_ce: 0.012201573699712753 (nan) | > loss: 0.0001662944268900901 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1366397724146188) | > loader_time: 0.0045 (0.00468044669282561)  --> TIME: 2025-02-26 01:06:08 -- STEP: 24624/36938 -- GLOBAL_STEP: 1021950 | > loss_text_ce: 0.030010515823960304 (0.03256478648592062) | > loss_mel_ce: 1.7319650650024414 (nan) | > loss: 0.006991966627538204 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.1366416230904283) | > loader_time: 0.0047 (0.004680347101807722)  --> TIME: 2025-02-26 01:06:17 -- STEP: 24674/36938 -- GLOBAL_STEP: 1022000 | > loss_text_ce: 0.028937799856066704 (0.03256359010246502) | > loss_mel_ce: 2.2887001037597656 (nan) | > loss: 0.009196976199746132 (nan) | > current_lr: 5e-06 | > step_time: 0.1622 (0.13664553440244356) | > loader_time: 0.0048 (0.004680284439594686)  --> TIME: 2025-02-26 01:06:27 -- STEP: 24724/36938 -- GLOBAL_STEP: 1022050 | > loss_text_ce: 0.03278183192014694 (0.03256198422752898) | > loss_mel_ce: 2.7935526371002197 (nan) | > loss: 0.011215613223612309 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13665097457192754) | > loader_time: 0.0046 (0.004680279841837362)  --> TIME: 2025-02-26 01:06:36 -- STEP: 24774/36938 -- GLOBAL_STEP: 1022100 | > loss_text_ce: 0.029388904571533203 (0.03256126604305861) | > loss_mel_ce: 3.0376899242401123 (nan) | > loss: 0.012170948088169098 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13665592727220846) | > loader_time: 0.0045 (0.004680255072028562)  --> TIME: 2025-02-26 01:06:46 -- STEP: 24824/36938 -- GLOBAL_STEP: 1022150 | > loss_text_ce: 0.02715660072863102 (0.032561514375299354) | > loss_mel_ce: 2.0491111278533936 (nan) | > loss: 0.008239158429205418 (nan) | > current_lr: 5e-06 | > step_time: 0.1776 (0.1366671058599935) | > loader_time: 0.0049 (0.004680324447873461)  --> TIME: 2025-02-26 01:06:55 -- STEP: 24874/36938 -- GLOBAL_STEP: 1022200 | > loss_text_ce: 0.03504091128706932 (0.03256053857681586) | > loss_mel_ce: 2.0289299488067627 (nan) | > loss: 0.008190360851585865 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1366602375814662) | > loader_time: 0.0046 (0.004680259363668776)  --> TIME: 2025-02-26 01:07:04 -- STEP: 24924/36938 -- GLOBAL_STEP: 1022250 | > loss_text_ce: 0.03502384200692177 (0.03256181980285329) | > loss_mel_ce: 0.167266845703125 (nan) | > loss: 0.0008027408621273935 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13667151693011487) | > loader_time: 0.0047 (0.004680421661936575)  --> TIME: 2025-02-26 01:07:14 -- STEP: 24974/36938 -- GLOBAL_STEP: 1022300 | > loss_text_ce: 0.032732509076595306 (0.03256233718676255) | > loss_mel_ce: 2.696516752243042 (nan) | > loss: 0.010830354876816273 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13666919156607188) | > loader_time: 0.0046 (0.004680437408551805)  --> TIME: 2025-02-26 01:07:23 -- STEP: 25024/36938 -- GLOBAL_STEP: 1022350 | > loss_text_ce: 0.03835848718881607 (0.03256337657855016) | > loss_mel_ce: 2.4900474548339844 (nan) | > loss: 0.010033356957137585 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.1366708570200475) | > loader_time: 0.0045 (0.00468041042766309)  --> TIME: 2025-02-26 01:07:32 -- STEP: 25074/36938 -- GLOBAL_STEP: 1022400 | > loss_text_ce: 0.03129083290696144 (0.03256292097282214) | > loss_mel_ce: 3.1937637329101562 (nan) | > loss: 0.01279783621430397 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.1366718594818934) | > loader_time: 0.0049 (0.004680361998388397)  --> TIME: 2025-02-26 01:07:42 -- STEP: 25124/36938 -- GLOBAL_STEP: 1022450 | > loss_text_ce: 0.028061281889677048 (0.03256227648151233) | > loss_mel_ce: 2.9900293350219727 (nan) | > loss: 0.01197655126452446 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.13666414128919918) | > loader_time: 0.0047 (0.004680245948661594)  --> TIME: 2025-02-26 01:07:51 -- STEP: 25174/36938 -- GLOBAL_STEP: 1022500 | > loss_text_ce: 0.029981138184666634 (0.032558168754917634) | > loss_mel_ce: 3.136246919631958 (nan) | > loss: 0.012564397417008877 (nan) | > current_lr: 5e-06 | > step_time: 0.1715 (0.13667557681251893) | > loader_time: 0.005 (0.004680344249064419)  --> TIME: 2025-02-26 01:08:01 -- STEP: 25224/36938 -- GLOBAL_STEP: 1022550 | > loss_text_ce: 0.03150385618209839 (0.032559311942402885) | > loss_mel_ce: 0.7314351201057434 (nan) | > loss: 0.003027535742148757 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13667546209365058) | > loader_time: 0.0045 (0.0046804374242789895)  --> TIME: 2025-02-26 01:08:10 -- STEP: 25274/36938 -- GLOBAL_STEP: 1022600 | > loss_text_ce: 0.03571365401148796 (0.03255847279277038) | > loss_mel_ce: 2.884979724884033 (nan) | > loss: 0.011590054258704185 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13667494340194708) | > loader_time: 0.0049 (0.004680455688473003)  --> TIME: 2025-02-26 01:08:19 -- STEP: 25324/36938 -- GLOBAL_STEP: 1022650 | > loss_text_ce: 0.028431188315153122 (0.032559882242682935) | > loss_mel_ce: 2.944125175476074 (nan) | > loss: 0.011795858852565289 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.1366816082943721) | > loader_time: 0.0046 (0.004680506954485409)  --> TIME: 2025-02-26 01:08:29 -- STEP: 25374/36938 -- GLOBAL_STEP: 1022700 | > loss_text_ce: 0.026595434173941612 (0.03256086776271759) | > loss_mel_ce: 0.03271484375 (nan) | > loss: 0.00023535826767329127 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.1366817634081449) | > loader_time: 0.0045 (0.004680488965956198)  --> TIME: 2025-02-26 01:08:38 -- STEP: 25424/36938 -- GLOBAL_STEP: 1022750 | > loss_text_ce: 0.03577668219804764 (0.032561866948423515) | > loss_mel_ce: 3.2255969047546387 (nan) | > loss: 0.01294195931404829 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13667939754309713) | > loader_time: 0.0045 (0.004680422096600602)  --> TIME: 2025-02-26 01:08:47 -- STEP: 25474/36938 -- GLOBAL_STEP: 1022800 | > loss_text_ce: 0.029462620615959167 (0.03255928640066726) | > loss_mel_ce: 3.0415117740631104 (nan) | > loss: 0.01218640711158514 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.1366797014655604) | > loader_time: 0.0049 (0.004680379711590546)  --> TIME: 2025-02-26 01:08:57 -- STEP: 25524/36938 -- GLOBAL_STEP: 1022850 | > loss_text_ce: 0.035082992166280746 (0.0325606463996533) | > loss_mel_ce: 2.8973329067230225 (nan) | > loss: 0.01163657195866108 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13668019577097285) | > loader_time: 0.0069 (0.004680504546309753)  --> TIME: 2025-02-26 01:09:06 -- STEP: 25574/36938 -- GLOBAL_STEP: 1022900 | > loss_text_ce: 0.02657892182469368 (0.03255895565839966) | > loss_mel_ce: 2.875013589859009 (nan) | > loss: 0.011514256708323956 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13668018094236867) | > loader_time: 0.0046 (0.004680448703865868)  --> TIME: 2025-02-26 01:09:15 -- STEP: 25624/36938 -- GLOBAL_STEP: 1022950 | > loss_text_ce: 0.03929954394698143 (0.03255876796674089) | > loss_mel_ce: 3.1997127532958984 (nan) | > loss: 0.012853223830461502 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.1366834649342214) | > loader_time: 0.0047 (0.0046804001600788325)  --> TIME: 2025-02-26 01:09:25 -- STEP: 25674/36938 -- GLOBAL_STEP: 1023000 | > loss_text_ce: 0.028468765318393707 (0.03255992225146615) | > loss_mel_ce: 3.2634036540985107 (nan) | > loss: 0.013062986545264721 (nan) | > current_lr: 5e-06 | > step_time: 0.1825 (0.13669039189578783) | > loader_time: 0.0049 (0.004680432429740519)  --> TIME: 2025-02-26 01:09:34 -- STEP: 25724/36938 -- GLOBAL_STEP: 1023050 | > loss_text_ce: 0.030487630516290665 (0.03255992167069725) | > loss_mel_ce: 3.094403028488159 (nan) | > loss: 0.012400359846651554 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13669434334845912) | > loader_time: 0.0044 (0.004680414432278935)  --> TIME: 2025-02-26 01:09:44 -- STEP: 25774/36938 -- GLOBAL_STEP: 1023100 | > loss_text_ce: 0.030520640313625336 (0.0325600942774212) | > loss_mel_ce: 2.8149526119232178 (nan) | > loss: 0.011291561648249626 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.1367017693133674) | > loader_time: 0.0046 (0.004680332704965799)  --> TIME: 2025-02-26 01:09:53 -- STEP: 25824/36938 -- GLOBAL_STEP: 1023150 | > loss_text_ce: 0.03422032669186592 (0.0325594796769767) | > loss_mel_ce: 3.100806951522827 (nan) | > loss: 0.012440584599971771 (nan) | > current_lr: 5e-06 | > step_time: 0.1747 (0.13671265275689654) | > loader_time: 0.0048 (0.004680279757749111)  --> TIME: 2025-02-26 01:10:03 -- STEP: 25874/36938 -- GLOBAL_STEP: 1023200 | > loss_text_ce: 0.04047742858529091 (0.0325610725691228) | > loss_mel_ce: 2.5983455181121826 (nan) | > loss: 0.010471520945429802 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13672379998820983) | > loader_time: 0.0045 (0.004680352416668491)  --> TIME: 2025-02-26 01:10:12 -- STEP: 25924/36938 -- GLOBAL_STEP: 1023250 | > loss_text_ce: 0.03949085250496864 (0.032561218351549064) | > loss_mel_ce: 2.8788845539093018 (nan) | > loss: 0.011580855585634708 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13672106320466998) | > loader_time: 0.0047 (0.004680374157574125)  --> TIME: 2025-02-26 01:10:22 -- STEP: 25974/36938 -- GLOBAL_STEP: 1023300 | > loss_text_ce: 0.02400541678071022 (0.03256254292613059) | > loss_mel_ce: 2.9225966930389404 (nan) | > loss: 0.011692865751683712 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13672706043967203) | > loader_time: 0.0049 (0.004680411777274643)  --> TIME: 2025-02-26 01:10:31 -- STEP: 26024/36938 -- GLOBAL_STEP: 1023350 | > loss_text_ce: 0.030732041224837303 (0.03255984546596494) | > loss_mel_ce: 2.979943037033081 (nan) | > loss: 0.011947124265134335 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.1367155999504091) | > loader_time: 0.0045 (0.0046803562633082425)  --> TIME: 2025-02-26 01:10:40 -- STEP: 26074/36938 -- GLOBAL_STEP: 1023400 | > loss_text_ce: 0.02861732244491577 (0.032558810984574506) | > loss_mel_ce: 2.612064838409424 (nan) | > loss: 0.010478897951543331 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1367127127540236) | > loader_time: 0.0045 (0.004680659102179317)  --> TIME: 2025-02-26 01:10:49 -- STEP: 26124/36938 -- GLOBAL_STEP: 1023450 | > loss_text_ce: 0.028792956843972206 (0.03255790487627546) | > loss_mel_ce: 3.0735785961151123 (nan) | > loss: 0.012310998514294624 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13670277647540632) | > loader_time: 0.0047 (0.004680574679517145)  --> TIME: 2025-02-26 01:10:58 -- STEP: 26174/36938 -- GLOBAL_STEP: 1023500 | > loss_text_ce: 0.03722981736063957 (0.03255785179037727) | > loss_mel_ce: 3.347487211227417 (nan) | > loss: 0.013431417755782604 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13669840764114025) | > loader_time: 0.0046 (0.004680644594121023)  --> TIME: 2025-02-26 01:11:08 -- STEP: 26224/36938 -- GLOBAL_STEP: 1023550 | > loss_text_ce: 0.026269609108567238 (0.03255789122980197) | > loss_mel_ce: 2.55000638961792 (nan) | > loss: 0.01022331789135933 (nan) | > current_lr: 5e-06 | > step_time: 0.1529 (0.1367115801776455) | > loader_time: 0.0046 (0.004680646509568606)  --> TIME: 2025-02-26 01:11:17 -- STEP: 26274/36938 -- GLOBAL_STEP: 1023600 | > loss_text_ce: 0.0338357649743557 (0.032557070309689934) | > loss_mel_ce: 0.6333084106445312 (nan) | > loss: 0.0026473975740373135 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13671142355150928) | > loader_time: 0.0046 (0.00468060129480282)  --> TIME: 2025-02-26 01:11:27 -- STEP: 26324/36938 -- GLOBAL_STEP: 1023650 | > loss_text_ce: 0.02928541786968708 (0.032555723338496954) | > loss_mel_ce: 1.4443511962890625 (nan) | > loss: 0.005847764667123556 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.1367071886643002) | > loader_time: 0.0046 (0.004680692515571114)  --> TIME: 2025-02-26 01:11:36 -- STEP: 26374/36938 -- GLOBAL_STEP: 1023700 | > loss_text_ce: 0.04890096187591553 (0.032557523585988665) | > loss_mel_ce: 3.1204426288604736 (nan) | > loss: 0.012576760724186897 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13670667741751963) | > loader_time: 0.0045 (0.004680714931229213)  --> TIME: 2025-02-26 01:11:45 -- STEP: 26424/36938 -- GLOBAL_STEP: 1023750 | > loss_text_ce: 0.03426259383559227 (0.03255706969424431) | > loss_mel_ce: 2.8248484134674072 (nan) | > loss: 0.011345679871737957 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.1367034271760961) | > loader_time: 0.0048 (0.004680606386281206)  --> TIME: 2025-02-26 01:11:55 -- STEP: 26474/36938 -- GLOBAL_STEP: 1023800 | > loss_text_ce: 0.030500153079628944 (0.032558810171644595) | > loss_mel_ce: 2.162670373916626 (nan) | > loss: 0.008703058585524559 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13670695361174662) | > loader_time: 0.0046 (0.004680699737282624)  --> TIME: 2025-02-26 01:12:04 -- STEP: 26524/36938 -- GLOBAL_STEP: 1023850 | > loss_text_ce: 0.035024046897888184 (0.032559409158863496) | > loss_mel_ce: 2.52001953125 (nan) | > loss: 0.010139063000679016 (nan) | > current_lr: 5e-06 | > step_time: 0.1783 (0.13671313701505108) | > loader_time: 0.0048 (0.004680916116421449)  --> TIME: 2025-02-26 01:12:13 -- STEP: 26574/36938 -- GLOBAL_STEP: 1023900 | > loss_text_ce: 0.03657371923327446 (0.03255745111925029) | > loss_mel_ce: 3.5791923999786377 (nan) | > loss: 0.014348278753459454 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.13670522761845552) | > loader_time: 0.0048 (0.004680812677357805)  --> TIME: 2025-02-26 01:12:22 -- STEP: 26624/36938 -- GLOBAL_STEP: 1023950 | > loss_text_ce: 0.037791892886161804 (0.032557022188764315) | > loss_mel_ce: 2.5440852642059326 (nan) | > loss: 0.010245544835925102 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13670379337138291) | > loader_time: 0.0046 (0.0046807270264253836)  --> TIME: 2025-02-26 01:12:32 -- STEP: 26674/36938 -- GLOBAL_STEP: 1024000 | > loss_text_ce: 0.023664439097046852 (0.03255769831898707) | > loss_mel_ce: 2.2825613021850586 (nan) | > loss: 0.00915168970823288 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1367001223708949) | > loader_time: 0.0046 (0.004680646451738642)  --> TIME: 2025-02-26 01:12:41 -- STEP: 26724/36938 -- GLOBAL_STEP: 1024050 | > loss_text_ce: 0.029608380049467087 (0.03255643303778038) | > loss_mel_ce: 2.9159443378448486 (nan) | > loss: 0.01168870273977518 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13669701375356094) | > loader_time: 0.0046 (0.004680669775196235)  --> TIME: 2025-02-26 01:12:50 -- STEP: 26774/36938 -- GLOBAL_STEP: 1024100 | > loss_text_ce: 0.030048241838812828 (0.03255608625659362) | > loss_mel_ce: 2.2537753582000732 (nan) | > loss: 0.009062792174518108 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.13669480690279717) | > loader_time: 0.0045 (0.0046806386830231085)  --> TIME: 2025-02-26 01:12:59 -- STEP: 26824/36938 -- GLOBAL_STEP: 1024150 | > loss_text_ce: 0.03153017535805702 (0.03255514088508971) | > loss_mel_ce: 2.869211435317993 (nan) | > loss: 0.0115108797326684 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.1366897721522395) | > loader_time: 0.0046 (0.004680589983579272)  --> TIME: 2025-02-26 01:13:09 -- STEP: 26874/36938 -- GLOBAL_STEP: 1024200 | > loss_text_ce: 0.02697787620127201 (0.032555192707028405) | > loss_mel_ce: 5.21875 (nan) | > loss: 0.0208163820207119 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.1366983686270989) | > loader_time: 0.0047 (0.00468057436168384)  --> TIME: 2025-02-26 01:13:18 -- STEP: 26924/36938 -- GLOBAL_STEP: 1024250 | > loss_text_ce: 0.028263548389077187 (0.03255538464270784) | > loss_mel_ce: 2.0160248279571533 (nan) | > loss: 0.008112256415188313 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13669217570613337) | > loader_time: 0.0045 (0.004680431964156297)  --> TIME: 2025-02-26 01:13:28 -- STEP: 26974/36938 -- GLOBAL_STEP: 1024300 | > loss_text_ce: 0.026985764503479004 (0.03255431941109857) | > loss_mel_ce: 3.347546100616455 (nan) | > loss: 0.013391000218689442 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13670049654489083) | > loader_time: 0.0046 (0.004680479820004346)  --> TIME: 2025-02-26 01:13:37 -- STEP: 27024/36938 -- GLOBAL_STEP: 1024350 | > loss_text_ce: 0.029542773962020874 (0.03255441226134085) | > loss_mel_ce: 2.596964120864868 (nan) | > loss: 0.01042264699935913 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13669879645409166) | > loader_time: 0.0046 (0.004680486024308388)  --> TIME: 2025-02-26 01:13:46 -- STEP: 27074/36938 -- GLOBAL_STEP: 1024400 | > loss_text_ce: 0.029394134879112244 (0.0325513637136282) | > loss_mel_ce: 3.256781816482544 (nan) | > loss: 0.013040381483733654 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13670533216095634) | > loader_time: 0.0047 (0.00468047694458244)  --> TIME: 2025-02-26 01:13:56 -- STEP: 27124/36938 -- GLOBAL_STEP: 1024450 | > loss_text_ce: 0.03468340262770653 (0.032550460883282895) | > loss_mel_ce: 2.627106189727783 (nan) | > loss: 0.010562658309936523 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.13670470902322449) | > loader_time: 0.0049 (0.004680399275198828)  --> TIME: 2025-02-26 01:14:05 -- STEP: 27174/36938 -- GLOBAL_STEP: 1024500 | > loss_text_ce: 0.038771167397499084 (0.03254960757256149) | > loss_mel_ce: 2.569483518600464 (nan) | > loss: 0.01035021711140871 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13670123533284184) | > loader_time: 0.0046 (0.004680696128263563)  --> TIME: 2025-02-26 01:14:14 -- STEP: 27224/36938 -- GLOBAL_STEP: 1024550 | > loss_text_ce: 0.028231408447027206 (0.032551788703577896) | > loss_mel_ce: 2.7028815746307373 (nan) | > loss: 0.010837750509381294 (nan) | > current_lr: 5e-06 | > step_time: 0.19 (0.13669579415471314) | > loader_time: 0.0049 (0.004680677648505722)  --> TIME: 2025-02-26 01:14:24 -- STEP: 27274/36938 -- GLOBAL_STEP: 1024600 | > loss_text_ce: 0.029573185369372368 (0.03255164225998869) | > loss_mel_ce: 3.0506486892700195 (nan) | > loss: 0.01222310308367014 (nan) | > current_lr: 5e-06 | > step_time: 0.1818 (0.13670008778878082) | > loader_time: 0.0049 (0.004680687655466971)  --> TIME: 2025-02-26 01:14:33 -- STEP: 27324/36938 -- GLOBAL_STEP: 1024650 | > loss_text_ce: 0.03399943187832832 (0.032551366786103424) | > loss_mel_ce: 0.02045694924890995 (nan) | > loss: 0.00021609677060041577 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13669003720095774) | > loader_time: 0.0048 (0.004680610570389725)  --> TIME: 2025-02-26 01:14:42 -- STEP: 27374/36938 -- GLOBAL_STEP: 1024700 | > loss_text_ce: 0.032255642116069794 (0.03255159982110694) | > loss_mel_ce: 2.843341827392578 (nan) | > loss: 0.011411101557314396 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.1366827019365172) | > loader_time: 0.0046 (0.0046805394020702894)  --> TIME: 2025-02-26 01:14:51 -- STEP: 27424/36938 -- GLOBAL_STEP: 1024750 | > loss_text_ce: 0.03204774856567383 (0.03255089010541102) | > loss_mel_ce: 3.2937979698181152 (nan) | > loss: 0.013197801075875759 (nan) | > current_lr: 5e-06 | > step_time: 0.177 (0.13669071638632863) | > loader_time: 0.0048 (0.004680507997857312)  --> TIME: 2025-02-26 01:15:01 -- STEP: 27474/36938 -- GLOBAL_STEP: 1024800 | > loss_text_ce: 0.014713896438479424 (0.03255229424325235) | > loss_mel_ce: 0.13884277641773224 (nan) | > loss: 0.0006093519041314721 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13669390443927013) | > loader_time: 0.0046 (0.004680485889243008)  --> TIME: 2025-02-26 01:15:10 -- STEP: 27524/36938 -- GLOBAL_STEP: 1024850 | > loss_text_ce: 0.03163778781890869 (0.03255199621536552) | > loss_mel_ce: 2.4439175128936768 (nan) | > loss: 0.0098236333578825 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13668356059924994) | > loader_time: 0.0045 (0.004680471102558422)  --> TIME: 2025-02-26 01:15:19 -- STEP: 27574/36938 -- GLOBAL_STEP: 1024900 | > loss_text_ce: 0.0292414128780365 (0.03255261931773864) | > loss_mel_ce: 2.156977415084839 (nan) | > loss: 0.008675471879541874 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.1366906510726136) | > loader_time: 0.0046 (0.004680445881295321)  --> TIME: 2025-02-26 01:15:29 -- STEP: 27624/36938 -- GLOBAL_STEP: 1024950 | > loss_text_ce: 0.027610039338469505 (0.03255184397705957) | > loss_mel_ce: 2.6251776218414307 (nan) | > loss: 0.010526935569941998 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13669479165151815) | > loader_time: 0.0045 (0.0046806057104331915)  --> TIME: 2025-02-26 01:15:38 -- STEP: 27674/36938 -- GLOBAL_STEP: 1025000 | > loss_text_ce: 0.04218202456831932 (0.032552436579413396) | > loss_mel_ce: 2.219501256942749 (nan) | > loss: 0.008974933996796608 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13670048232435947) | > loader_time: 0.0046 (0.0046806536443129015)  --> TIME: 2025-02-26 01:15:47 -- STEP: 27724/36938 -- GLOBAL_STEP: 1025050 | > loss_text_ce: 0.03228276968002319 (0.03255347322812743) | > loss_mel_ce: 3.120192527770996 (nan) | > loss: 0.012509823776781559 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13669654439321582) | > loader_time: 0.0046 (0.004680644930948457)  --> TIME: 2025-02-26 01:15:57 -- STEP: 27774/36938 -- GLOBAL_STEP: 1025100 | > loss_text_ce: 0.03127773851156235 (0.03255220321561345) | > loss_mel_ce: 2.4884610176086426 (nan) | > loss: 0.009998963214457035 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13670124074179568) | > loader_time: 0.0047 (0.004680714159485084)  --> TIME: 2025-02-26 01:16:06 -- STEP: 27824/36938 -- GLOBAL_STEP: 1025150 | > loss_text_ce: 0.027620334178209305 (0.032551141401812686) | > loss_mel_ce: 2.5743486881256104 (nan) | > loss: 0.010325274430215359 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13669680820723418) | > loader_time: 0.0045 (0.004680631188499109)  --> TIME: 2025-02-26 01:16:15 -- STEP: 27874/36938 -- GLOBAL_STEP: 1025200 | > loss_text_ce: 0.03313513845205307 (0.032551958171673744) | > loss_mel_ce: 2.419340133666992 (nan) | > loss: 0.009732045233249664 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13669767029582944) | > loader_time: 0.0047 (0.004680673104580717)  --> TIME: 2025-02-26 01:16:25 -- STEP: 27924/36938 -- GLOBAL_STEP: 1025250 | > loss_text_ce: 0.04248430207371712 (0.03255224895699965) | > loss_mel_ce: 2.060786008834839 (nan) | > loss: 0.008346310816705227 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.1366978098321246) | > loader_time: 0.0046 (0.004680587362208361)  --> TIME: 2025-02-26 01:16:34 -- STEP: 27974/36938 -- GLOBAL_STEP: 1025300 | > loss_text_ce: 0.04308879375457764 (0.032549377379998384) | > loss_mel_ce: 2.1483497619628906 (nan) | > loss: 0.008696185424923897 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13668826995656516) | > loader_time: 0.0046 (0.004680492687559518)  --> TIME: 2025-02-26 01:16:43 -- STEP: 28024/36938 -- GLOBAL_STEP: 1025350 | > loss_text_ce: 0.024263843894004822 (0.03255074614708923) | > loss_mel_ce: 0.8369830846786499 (nan) | > loss: 0.0034176467452198267 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13669312360999308) | > loader_time: 0.0045 (0.004680396393983742)  --> TIME: 2025-02-26 01:16:53 -- STEP: 28074/36938 -- GLOBAL_STEP: 1025400 | > loss_text_ce: 0.02716512605547905 (0.03255052120654557) | > loss_mel_ce: 0.03558349609375 (nan) | > loss: 0.00024900250718928874 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13669486353945468) | > loader_time: 0.0046 (0.004680475482423049)  --> TIME: 2025-02-26 01:17:02 -- STEP: 28124/36938 -- GLOBAL_STEP: 1025450 | > loss_text_ce: 0.029262356460094452 (0.032553447275752885) | > loss_mel_ce: 2.1002604961395264 (nan) | > loss: 0.00845048762857914 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13670375127233522) | > loader_time: 0.0045 (0.004680529891673246)  --> TIME: 2025-02-26 01:17:12 -- STEP: 28174/36938 -- GLOBAL_STEP: 1025500 | > loss_text_ce: 0.02801978960633278 (0.032557359433616394) | > loss_mel_ce: 1.9871171712875366 (nan) | > loss: 0.00799657590687275 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.1367074075834791) | > loader_time: 0.0045 (0.004680515875782236)  --> TIME: 2025-02-26 01:17:21 -- STEP: 28224/36938 -- GLOBAL_STEP: 1025550 | > loss_text_ce: 0.02908279374241829 (0.03255638877371242) | > loss_mel_ce: 3.4983208179473877 (nan) | > loss: 0.013997633941471577 (nan) | > current_lr: 5e-06 | > step_time: 0.1477 (0.13670938402887367) | > loader_time: 0.0049 (0.004680458895541181)  --> TIME: 2025-02-26 01:17:31 -- STEP: 28274/36938 -- GLOBAL_STEP: 1025600 | > loss_text_ce: 0.033007219433784485 (0.03255521667064653) | > loss_mel_ce: 2.843549966812134 (nan) | > loss: 0.011414909735321999 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13671546786140254) | > loader_time: 0.0047 (0.00468054314081972)  --> TIME: 2025-02-26 01:17:40 -- STEP: 28324/36938 -- GLOBAL_STEP: 1025650 | > loss_text_ce: 0.0394258052110672 (0.032553726255295484) | > loss_mel_ce: 2.7003865242004395 (nan) | > loss: 0.010872271843254566 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.13671573462450576) | > loader_time: 0.0046 (0.004680623418613852)  --> TIME: 2025-02-26 01:17:49 -- STEP: 28374/36938 -- GLOBAL_STEP: 1025700 | > loss_text_ce: 0.032940950244665146 (0.0325505465178234) | > loss_mel_ce: 0.0391845703125 (nan) | > loss: 0.0002862124238163233 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13670793230525033) | > loader_time: 0.0046 (0.0046806438718562495)  --> TIME: 2025-02-26 01:17:58 -- STEP: 28424/36938 -- GLOBAL_STEP: 1025750 | > loss_text_ce: 0.029535992071032524 (0.03255003076342897) | > loss_mel_ce: 0.0562947578728199 (nan) | > loss: 0.00034059822792187333 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13670107956305722) | > loader_time: 0.0046 (0.00468057101288891)  --> TIME: 2025-02-26 01:18:07 -- STEP: 28474/36938 -- GLOBAL_STEP: 1025800 | > loss_text_ce: 0.027401408180594444 (0.03254816655614888) | > loss_mel_ce: 2.7372734546661377 (nan) | > loss: 0.010970932431519032 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.13668958456017058) | > loader_time: 0.0046 (0.004680463367946408)  --> TIME: 2025-02-26 01:18:17 -- STEP: 28524/36938 -- GLOBAL_STEP: 1025850 | > loss_text_ce: 0.03335602208971977 (0.03254706920810898) | > loss_mel_ce: 1.0609252452850342 (nan) | > loss: 0.004342386499047279 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13669127689774013) | > loader_time: 0.0046 (0.004680466173808928)  --> TIME: 2025-02-26 01:18:25 -- STEP: 28574/36938 -- GLOBAL_STEP: 1025900 | > loss_text_ce: 0.030774034559726715 (0.03254605010488223) | > loss_mel_ce: 2.9917984008789062 (nan) | > loss: 0.011994335800409317 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13668001627599985) | > loader_time: 0.0047 (0.004680356302179326)  --> TIME: 2025-02-26 01:18:35 -- STEP: 28624/36938 -- GLOBAL_STEP: 1025950 | > loss_text_ce: 0.02990840934216976 (0.032544636089563476) | > loss_mel_ce: 2.0975239276885986 (nan) | > loss: 0.00844219233840704 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13668660218986217) | > loader_time: 0.0046 (0.0046804753793825956)  --> TIME: 2025-02-26 01:18:44 -- STEP: 28674/36938 -- GLOBAL_STEP: 1026000 | > loss_text_ce: 0.03310380503535271 (0.03254574635801103) | > loss_mel_ce: 3.0055136680603027 (nan) | > loss: 0.012058006599545479 (nan) | > current_lr: 5e-06 | > step_time: 0.187 (0.13668713696358673) | > loader_time: 0.0051 (0.004680568889036094)  --> TIME: 2025-02-26 01:18:54 -- STEP: 28724/36938 -- GLOBAL_STEP: 1026050 | > loss_text_ce: 0.03950174152851105 (0.032545746269207435) | > loss_mel_ce: 3.7247121334075928 (nan) | > loss: 0.014937357045710087 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13668534678845753) | > loader_time: 0.0046 (0.004680593255137)  --> TIME: 2025-02-26 01:19:03 -- STEP: 28774/36938 -- GLOBAL_STEP: 1026100 | > loss_text_ce: 0.027713414281606674 (0.03254475377944382) | > loss_mel_ce: 3.072237253189087 (nan) | > loss: 0.012301391921937466 (nan) | > current_lr: 5e-06 | > step_time: 0.1574 (0.13668080471451383) | > loader_time: 0.0046 (0.0046805390359098845)  --> TIME: 2025-02-26 01:19:12 -- STEP: 28824/36938 -- GLOBAL_STEP: 1026150 | > loss_text_ce: 0.03325679898262024 (0.032543134273130866) | > loss_mel_ce: 3.3637783527374268 (nan) | > loss: 0.013480299152433872 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.1366817023384877) | > loader_time: 0.0049 (0.004680492780025368)  --> TIME: 2025-02-26 01:19:22 -- STEP: 28874/36938 -- GLOBAL_STEP: 1026200 | > loss_text_ce: 0.04185249283909798 (0.032542438109033636) | > loss_mel_ce: 2.5948760509490967 (nan) | > loss: 0.010463208891451359 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13669603425990565) | > loader_time: 0.0046 (0.004680496045920109)  --> TIME: 2025-02-26 01:19:31 -- STEP: 28924/36938 -- GLOBAL_STEP: 1026250 | > loss_text_ce: 0.0323752723634243 (0.03253963306613695) | > loss_mel_ce: 0.7580934166908264 (nan) | > loss: 0.0031367808114737272 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13669821072244606) | > loader_time: 0.0047 (0.004680428988309485)  --> TIME: 2025-02-26 01:19:40 -- STEP: 28974/36938 -- GLOBAL_STEP: 1026300 | > loss_text_ce: 0.04031529650092125 (0.0325390468015644) | > loss_mel_ce: 2.8082830905914307 (nan) | > loss: 0.011303963139653206 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13669058667162692) | > loader_time: 0.0046 (0.0046803561304966995)  --> TIME: 2025-02-26 01:19:49 -- STEP: 29024/36938 -- GLOBAL_STEP: 1026350 | > loss_text_ce: 0.021738281473517418 (0.03253935382434733) | > loss_mel_ce: 2.0308125019073486 (nan) | > loss: 0.00814504362642765 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13668343378795667) | > loader_time: 0.0045 (0.004680266979642396)  --> TIME: 2025-02-26 01:19:59 -- STEP: 29074/36938 -- GLOBAL_STEP: 1026400 | > loss_text_ce: 0.02989194542169571 (0.032540444284164895) | > loss_mel_ce: 3.573349952697754 (nan) | > loss: 0.014298579655587673 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13668764624871488) | > loader_time: 0.0046 (0.004680235177440553)  --> TIME: 2025-02-26 01:20:08 -- STEP: 29124/36938 -- GLOBAL_STEP: 1026450 | > loss_text_ce: 0.014373037032783031 (0.03254197407087263) | > loss_mel_ce: 1.0075156688690186 (nan) | > loss: 0.004055114462971687 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.1366875098557478) | > loader_time: 0.0047 (0.004680223000637446)  --> TIME: 2025-02-26 01:20:18 -- STEP: 29174/36938 -- GLOBAL_STEP: 1026500 | > loss_text_ce: 0.032181043177843094 (0.03254165308343894) | > loss_mel_ce: 2.8725533485412598 (nan) | > loss: 0.011526724323630333 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.1366913967677823) | > loader_time: 0.0046 (0.004680269469111333)  --> TIME: 2025-02-26 01:20:27 -- STEP: 29224/36938 -- GLOBAL_STEP: 1026550 | > loss_text_ce: 0.02268332615494728 (0.032543446081603665) | > loss_mel_ce: 2.2821152210235596 (nan) | > loss: 0.009146026335656643 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13669824591245575) | > loader_time: 0.0046 (0.004680259331200188)  --> TIME: 2025-02-26 01:20:37 -- STEP: 29274/36938 -- GLOBAL_STEP: 1026600 | > loss_text_ce: 0.037274010479450226 (0.03254493738343834) | > loss_mel_ce: 3.012218952178955 (nan) | > loss: 0.01210116408765316 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13670859374642955) | > loader_time: 0.0047 (0.0046803470337791545)  --> TIME: 2025-02-26 01:20:46 -- STEP: 29324/36938 -- GLOBAL_STEP: 1026650 | > loss_text_ce: 0.045431990176439285 (0.03254562036469414) | > loss_mel_ce: 2.696707248687744 (nan) | > loss: 0.010881505906581879 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13669999525339216) | > loader_time: 0.0046 (0.004680292698391017)  --> TIME: 2025-02-26 01:20:56 -- STEP: 29374/36938 -- GLOBAL_STEP: 1026700 | > loss_text_ce: 0.023012172430753708 (0.0325446623457906) | > loss_mel_ce: 2.181150197982788 (nan) | > loss: 0.0087466761469841 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13670541445197779) | > loader_time: 0.0046 (0.004680322368661121)  --> TIME: 2025-02-26 01:21:05 -- STEP: 29424/36938 -- GLOBAL_STEP: 1026750 | > loss_text_ce: 0.02425033040344715 (0.03254194984355191) | > loss_mel_ce: 2.4644997119903564 (nan) | > loss: 0.009875992313027382 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13670350147726226) | > loader_time: 0.0046 (0.004680344515873671)  --> TIME: 2025-02-26 01:21:14 -- STEP: 29474/36938 -- GLOBAL_STEP: 1026800 | > loss_text_ce: 0.025576360523700714 (0.032544576475888175) | > loss_mel_ce: 2.598093032836914 (nan) | > loss: 0.01041138730943203 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13670319907254477) | > loader_time: 0.0046 (0.004680330494313025)  --> TIME: 2025-02-26 01:21:24 -- STEP: 29524/36938 -- GLOBAL_STEP: 1026850 | > loss_text_ce: 0.02994755282998085 (0.032546290344138494) | > loss_mel_ce: 2.6123738288879395 (nan) | > loss: 0.01048540323972702 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13670446240890632) | > loader_time: 0.0057 (0.00468027705568694)  --> TIME: 2025-02-26 01:21:33 -- STEP: 29574/36938 -- GLOBAL_STEP: 1026900 | > loss_text_ce: 0.035682763904333115 (0.032545246626985665) | > loss_mel_ce: 0.559648871421814 (nan) | > loss: 0.00236242706887424 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.1367040840004328) | > loader_time: 0.0048 (0.004680237277023216)  --> TIME: 2025-02-26 01:21:43 -- STEP: 29624/36938 -- GLOBAL_STEP: 1026950 | > loss_text_ce: 0.03183375298976898 (0.03254576803251308) | > loss_mel_ce: 2.2537753582000732 (nan) | > loss: 0.009069877676665783 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13671601394881994) | > loader_time: 0.0046 (0.004680286806208628)  --> TIME: 2025-02-26 01:21:52 -- STEP: 29674/36938 -- GLOBAL_STEP: 1027000 | > loss_text_ce: 0.03315622732043266 (0.03254585693045929) | > loss_mel_ce: 3.514960765838623 (nan) | > loss: 0.014079829677939415 (nan) | > current_lr: 5e-06 | > step_time: 0.1523 (0.13671662599850642) | > loader_time: 0.0047 (0.004680187841820295)  --> TIME: 2025-02-26 01:22:01 -- STEP: 29724/36938 -- GLOBAL_STEP: 1027050 | > loss_text_ce: 0.024889953434467316 (0.032545870952083764) | > loss_mel_ce: 3.166261672973633 (nan) | > loss: 0.012663301080465317 (nan) | > current_lr: 5e-06 | > step_time: 0.1761 (0.1367102432918329) | > loader_time: 0.0049 (0.004680093148726578)  --> TIME: 2025-02-26 01:22:11 -- STEP: 29774/36938 -- GLOBAL_STEP: 1027100 | > loss_text_ce: 0.03218529000878334 (0.032546463442715126) | > loss_mel_ce: 1.9009671211242676 (nan) | > loss: 0.007671240251511335 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13672185952310914) | > loader_time: 0.0047 (0.0046800843510001015)  --> TIME: 2025-02-26 01:22:20 -- STEP: 29824/36938 -- GLOBAL_STEP: 1027150 | > loss_text_ce: 0.03322886303067207 (0.03254723569767719) | > loss_mel_ce: 2.9935975074768066 (nan) | > loss: 0.012011216953396797 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1367193386518264) | > loader_time: 0.0046 (0.004680020398912074)  --> TIME: 2025-02-26 01:22:29 -- STEP: 29874/36938 -- GLOBAL_STEP: 1027200 | > loss_text_ce: 0.037798453122377396 (0.03254723141104294) | > loss_mel_ce: 2.1959524154663086 (nan) | > loss: 0.008864090777933598 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.13671546028011045) | > loader_time: 0.0048 (0.004680028959012468)  --> TIME: 2025-02-26 01:22:39 -- STEP: 29924/36938 -- GLOBAL_STEP: 1027250 | > loss_text_ce: 0.031754788011312485 (0.03254845063563103) | > loss_mel_ce: 2.967902183532715 (nan) | > loss: 0.011903401464223862 (nan) | > current_lr: 5e-06 | > step_time: 0.1691 (0.13671741687491384) | > loader_time: 0.0049 (0.0046799930479577935)  --> TIME: 2025-02-26 01:22:48 -- STEP: 29974/36938 -- GLOBAL_STEP: 1027300 | > loss_text_ce: 0.03663204237818718 (0.0325499193148109) | > loss_mel_ce: 3.422945499420166 (nan) | > loss: 0.013728483580052853 (nan) | > current_lr: 5e-06 | > step_time: 0.1632 (0.13671792799121832) | > loader_time: 0.0048 (0.0046799294886695104)  --> TIME: 2025-02-26 01:22:57 -- STEP: 30024/36938 -- GLOBAL_STEP: 1027350 | > loss_text_ce: 0.0243518128991127 (0.032549087420946345) | > loss_mel_ce: 2.6480252742767334 (nan) | > loss: 0.010604672133922577 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1367150329126855) | > loader_time: 0.0047 (0.004679839816882529)  --> TIME: 2025-02-26 01:23:06 -- STEP: 30074/36938 -- GLOBAL_STEP: 1027400 | > loss_text_ce: 0.03195054456591606 (0.03254749285205328) | > loss_mel_ce: 1.414566993713379 (nan) | > loss: 0.005740149412304163 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.1367139500205548) | > loader_time: 0.0045 (0.004679746106797216)  --> TIME: 2025-02-26 01:23:16 -- STEP: 30124/36938 -- GLOBAL_STEP: 1027450 | > loss_text_ce: 0.03488029167056084 (0.032547161218018504) | > loss_mel_ce: 2.7409560680389404 (nan) | > loss: 0.011015224270522594 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13671000176985534) | > loader_time: 0.0046 (0.00467964789573303)  --> TIME: 2025-02-26 01:23:25 -- STEP: 30174/36938 -- GLOBAL_STEP: 1027500 | > loss_text_ce: 0.030282553285360336 (0.03254633260665644) | > loss_mel_ce: 3.0551071166992188 (nan) | > loss: 0.012243609875440598 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13670954515755884) | > loader_time: 0.0046 (0.004679541579296089)  --> TIME: 2025-02-26 01:23:34 -- STEP: 30224/36938 -- GLOBAL_STEP: 1027550 | > loss_text_ce: 0.029799019917845726 (0.032544844882539746) | > loss_mel_ce: 2.5119948387145996 (nan) | > loss: 0.010086484253406525 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13670239232721115) | > loader_time: 0.0047 (0.004679450531558751)  --> TIME: 2025-02-26 01:23:43 -- STEP: 30274/36938 -- GLOBAL_STEP: 1027600 | > loss_text_ce: 0.02676054649055004 (0.03254522653537765) | > loss_mel_ce: 3.1276121139526367 (nan) | > loss: 0.01251735258847475 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13670013600929234) | > loader_time: 0.0046 (0.004679347656516198)  --> TIME: 2025-02-26 01:23:52 -- STEP: 30324/36938 -- GLOBAL_STEP: 1027650 | > loss_text_ce: 0.04416481778025627 (0.03254561384839692) | > loss_mel_ce: 2.148386001586914 (nan) | > loss: 0.008700598962605 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.13669346057712956) | > loader_time: 0.0049 (0.004679307249192821)  --> TIME: 2025-02-26 01:24:01 -- STEP: 30374/36938 -- GLOBAL_STEP: 1027700 | > loss_text_ce: 0.027065107598900795 (0.03254854530039014) | > loss_mel_ce: 2.818246364593506 (nan) | > loss: 0.011290919035673141 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13669002703699773) | > loader_time: 0.0046 (0.004679323844020557)  --> TIME: 2025-02-26 01:24:11 -- STEP: 30424/36938 -- GLOBAL_STEP: 1027750 | > loss_text_ce: 0.0299809742718935 (0.0325484608463556) | > loss_mel_ce: 2.387324333190918 (nan) | > loss: 0.009592481888830662 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13668843295083302) | > loader_time: 0.0046 (0.00467931058981725)  --> TIME: 2025-02-26 01:24:20 -- STEP: 30474/36938 -- GLOBAL_STEP: 1027800 | > loss_text_ce: 0.036461241543293 (0.03254868173539019) | > loss_mel_ce: 2.9681918621063232 (nan) | > loss: 0.011923227459192276 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13668464202146055) | > loader_time: 0.0046 (0.004679240008120989)  --> TIME: 2025-02-26 01:24:30 -- STEP: 30524/36938 -- GLOBAL_STEP: 1027850 | > loss_text_ce: 0.037506863474845886 (0.032546968373500844) | > loss_mel_ce: 3.458888053894043 (nan) | > loss: 0.013874583877623081 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.13669204404166316) | > loader_time: 0.0047 (0.00467934406626952)  --> TIME: 2025-02-26 01:24:39 -- STEP: 30574/36938 -- GLOBAL_STEP: 1027900 | > loss_text_ce: 0.04950882866978645 (0.03254634372717822) | > loss_mel_ce: 3.3031630516052246 (nan) | > loss: 0.013304254040122032 (nan) | > current_lr: 5e-06 | > step_time: 0.1632 (0.13669215402305568) | > loader_time: 0.0049 (0.004679300548470942)  --> TIME: 2025-02-26 01:24:48 -- STEP: 30624/36938 -- GLOBAL_STEP: 1027950 | > loss_text_ce: 0.03337501361966133 (0.032545500652955243) | > loss_mel_ce: 2.9009814262390137 (nan) | > loss: 0.011644272133708 (nan) | > current_lr: 5e-06 | > step_time: 0.1894 (0.1366929505935158) | > loader_time: 0.0071 (0.004679320171833609)  --> TIME: 2025-02-26 01:24:58 -- STEP: 30674/36938 -- GLOBAL_STEP: 1028000 | > loss_text_ce: 0.036655109375715256 (0.03254869151852576) | > loss_mel_ce: 2.4947924613952637 (nan) | > loss: 0.010045427829027176 (nan) | > current_lr: 5e-06 | > step_time: 0.1071 (0.1366993988198407) | > loader_time: 0.0044 (0.004679440503761894)  --> TIME: 2025-02-26 01:25:07 -- STEP: 30724/36938 -- GLOBAL_STEP: 1028050 | > loss_text_ce: 0.02204000949859619 (0.03254673334442764) | > loss_mel_ce: 2.639613628387451 (nan) | > loss: 0.010562118142843246 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.1367000992635531) | > loader_time: 0.0046 (0.004679441622746682)  --> TIME: 2025-02-26 01:25:17 -- STEP: 30774/36938 -- GLOBAL_STEP: 1028100 | > loss_text_ce: 0.03135070577263832 (0.03254567398302612) | > loss_mel_ce: 2.4591267108917236 (nan) | > loss: 0.009882846847176552 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13670565086924288) | > loader_time: 0.0046 (0.0046794599838143516)  --> TIME: 2025-02-26 01:25:26 -- STEP: 30824/36938 -- GLOBAL_STEP: 1028150 | > loss_text_ce: 0.031550243496894836 (0.03254578807453459) | > loss_mel_ce: 2.7348878383636475 (nan) | > loss: 0.010977929458022118 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13669652017469125) | > loader_time: 0.0046 (0.004679379867263696)  --> TIME: 2025-02-26 01:25:35 -- STEP: 30874/36938 -- GLOBAL_STEP: 1028200 | > loss_text_ce: 0.030983172357082367 (0.032547727170228705) | > loss_mel_ce: 2.7210745811462402 (nan) | > loss: 0.010920864529907703 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.13669448689011435) | > loader_time: 0.0047 (0.0046793343822071985)  --> TIME: 2025-02-26 01:25:44 -- STEP: 30924/36938 -- GLOBAL_STEP: 1028250 | > loss_text_ce: 0.029100939631462097 (0.03254672399387859) | > loss_mel_ce: 0.0218505859375 (nan) | > loss: 0.0002021886029979214 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13669354317182397) | > loader_time: 0.0048 (0.0046792860836650995)  --> TIME: 2025-02-26 01:25:54 -- STEP: 30974/36938 -- GLOBAL_STEP: 1028300 | > loss_text_ce: 0.028537414968013763 (0.03254469223182764) | > loss_mel_ce: 0.77630615234375 (nan) | > loss: 0.003193823853507638 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13669542232692494) | > loader_time: 0.0045 (0.004679234307893224)  --> TIME: 2025-02-26 01:26:03 -- STEP: 31024/36938 -- GLOBAL_STEP: 1028350 | > loss_text_ce: 0.042667727917432785 (0.032546333620774234) | > loss_mel_ce: 0.03369140625 (nan) | > loss: 0.0003030124644283205 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1367022879056674) | > loader_time: 0.0045 (0.004679274472949378)  --> TIME: 2025-02-26 01:26:12 -- STEP: 31074/36938 -- GLOBAL_STEP: 1028400 | > loss_text_ce: 0.021182825788855553 (0.032545465343489254) | > loss_mel_ce: 1.9070757627487183 (nan) | > loss: 0.007651819847524166 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13669772169462166) | > loader_time: 0.0045 (0.0046791381999211285)  --> TIME: 2025-02-26 01:26:22 -- STEP: 31124/36938 -- GLOBAL_STEP: 1028450 | > loss_text_ce: 0.03365036100149155 (0.03254246591177283) | > loss_mel_ce: 3.096986770629883 (nan) | > loss: 0.012423164211213589 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13669829111359075) | > loader_time: 0.0046 (0.004679028440327722)  --> TIME: 2025-02-26 01:26:31 -- STEP: 31174/36938 -- GLOBAL_STEP: 1028500 | > loss_text_ce: 0.03092750534415245 (0.032541270198888014) | > loss_mel_ce: 3.394411325454712 (nan) | > loss: 0.01359261479228735 (nan) | > current_lr: 5e-06 | > step_time: 0.184 (0.13669526199184917) | > loader_time: 0.0049 (0.004678929143469958)  --> TIME: 2025-02-26 01:26:40 -- STEP: 31224/36938 -- GLOBAL_STEP: 1028550 | > loss_text_ce: 0.034448325634002686 (0.03254133734416356) | > loss_mel_ce: 3.2236592769622803 (nan) | > loss: 0.012928999029099941 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13669221344862778) | > loader_time: 0.0045 (0.004678851361461713)  --> TIME: 2025-02-26 01:26:49 -- STEP: 31274/36938 -- GLOBAL_STEP: 1028600 | > loss_text_ce: 0.03346648067235947 (0.03254203228347245) | > loss_mel_ce: 2.8970601558685303 (nan) | > loss: 0.011629074811935425 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13668842911682152) | > loader_time: 0.0046 (0.004678797087584364)  --> TIME: 2025-02-26 01:26:59 -- STEP: 31324/36938 -- GLOBAL_STEP: 1028650 | > loss_text_ce: 0.03722989559173584 (0.03254251438939289) | > loss_mel_ce: 3.155776023864746 (nan) | > loss: 0.012670659460127354 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13668290748402553) | > loader_time: 0.0046 (0.004678726675805494)  --> TIME: 2025-02-26 01:27:08 -- STEP: 31374/36938 -- GLOBAL_STEP: 1028700 | > loss_text_ce: 0.0290269386023283 (0.032542810518212104) | > loss_mel_ce: 2.7002639770507812 (nan) | > loss: 0.010830520652234554 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13667787193621272) | > loader_time: 0.005 (0.004678664080095292)  --> TIME: 2025-02-26 01:27:17 -- STEP: 31424/36938 -- GLOBAL_STEP: 1028750 | > loss_text_ce: 0.033383190631866455 (0.03254213241887293) | > loss_mel_ce: 1.572184681892395 (nan) | > loss: 0.006371301598846912 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.1366717284222848) | > loader_time: 0.0046 (0.004678565250091371)  --> TIME: 2025-02-26 01:27:26 -- STEP: 31474/36938 -- GLOBAL_STEP: 1028800 | > loss_text_ce: 0.043182194232940674 (0.03254248009966332) | > loss_mel_ce: 2.6044082641601562 (nan) | > loss: 0.010506311431527138 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13666997087223745) | > loader_time: 0.0045 (0.004678483573531618)  --> TIME: 2025-02-26 01:27:35 -- STEP: 31524/36938 -- GLOBAL_STEP: 1028850 | > loss_text_ce: 0.029354577884078026 (0.032542203101363125) | > loss_mel_ce: 1.5755577087402344 (nan) | > loss: 0.006368699949234724 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.1366647550439852) | > loader_time: 0.0047 (0.004678419437706201)  --> TIME: 2025-02-26 01:27:45 -- STEP: 31574/36938 -- GLOBAL_STEP: 1028900 | > loss_text_ce: 0.033784959465265274 (0.03254375267854563) | > loss_mel_ce: 2.24603009223938 (nan) | > loss: 0.009046885184943676 (nan) | > current_lr: 5e-06 | > step_time: 0.1523 (0.13666591735202405) | > loader_time: 0.0046 (0.004678381873466923)  --> TIME: 2025-02-26 01:27:54 -- STEP: 31624/36938 -- GLOBAL_STEP: 1028950 | > loss_text_ce: 0.024819759652018547 (0.03254392376129976) | > loss_mel_ce: 2.6151442527770996 (nan) | > loss: 0.010476049035787582 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13666587063791533) | > loader_time: 0.0046 (0.0046783267185111525)  --> TIME: 2025-02-26 01:28:03 -- STEP: 31674/36938 -- GLOBAL_STEP: 1029000 | > loss_text_ce: 0.03225540742278099 (0.032542031705334734) | > loss_mel_ce: 2.786566734313965 (nan) | > loss: 0.011185802519321442 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13666902818696064) | > loader_time: 0.0047 (0.0046783551774167475)  --> TIME: 2025-02-26 01:28:13 -- STEP: 31724/36938 -- GLOBAL_STEP: 1029050 | > loss_text_ce: 0.033957064151763916 (0.03254228008023486) | > loss_mel_ce: 2.651491403579712 (nan) | > loss: 0.010656542144715786 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13667140032872016) | > loader_time: 0.0046 (0.004678460812451494)  --> TIME: 2025-02-26 01:28:22 -- STEP: 31774/36938 -- GLOBAL_STEP: 1029100 | > loss_text_ce: 0.033772703260183334 (0.03254017964257069) | > loss_mel_ce: 3.21878981590271 (nan) | > loss: 0.012906994670629501 (nan) | > current_lr: 5e-06 | > step_time: 0.168 (0.13667473810588143) | > loader_time: 0.0047 (0.004678402266973975)  --> TIME: 2025-02-26 01:28:31 -- STEP: 31824/36938 -- GLOBAL_STEP: 1029150 | > loss_text_ce: 0.03657236322760582 (0.032539943518288386) | > loss_mel_ce: 1.2188092470169067 (nan) | > loss: 0.004981673322618008 (nan) | > current_lr: 5e-06 | > step_time: 0.184 (0.13667055757696628) | > loader_time: 0.0051 (0.004678289043118281)  --> TIME: 2025-02-26 01:28:40 -- STEP: 31874/36938 -- GLOBAL_STEP: 1029200 | > loss_text_ce: 0.04152344912290573 (0.03253931757422644) | > loss_mel_ce: 2.9041550159454346 (nan) | > loss: 0.0116892009973526 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13666403489586484) | > loader_time: 0.0046 (0.0046782031549656815)  --> TIME: 2025-02-26 01:28:50 -- STEP: 31924/36938 -- GLOBAL_STEP: 1029250 | > loss_text_ce: 0.026825984939932823 (0.0325399088190653) | > loss_mel_ce: 2.6696155071258545 (nan) | > loss: 0.010700165294110775 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13666729741251235) | > loader_time: 0.0046 (0.004678104862117132)  --> TIME: 2025-02-26 01:28:59 -- STEP: 31974/36938 -- GLOBAL_STEP: 1029300 | > loss_text_ce: 0.03247132524847984 (0.032539648561376436) | > loss_mel_ce: 2.7485158443450928 (nan) | > loss: 0.01103566400706768 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1366649106719998) | > loader_time: 0.0044 (0.004678004087900479)  --> TIME: 2025-02-26 01:29:08 -- STEP: 32024/36938 -- GLOBAL_STEP: 1029350 | > loss_text_ce: 0.04275840148329735 (0.03254114574192739) | > loss_mel_ce: 3.0779483318328857 (nan) | > loss: 0.012383758090436459 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13666637818455363) | > loader_time: 0.0047 (0.0046779267897524394)  --> TIME: 2025-02-26 01:29:18 -- STEP: 32074/36938 -- GLOBAL_STEP: 1029400 | > loss_text_ce: 0.026452815160155296 (0.0325415881310333) | > loss_mel_ce: 2.090082883834839 (nan) | > loss: 0.008398951962590218 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1366705170234941) | > loader_time: 0.0045 (0.004677852832327798)  --> TIME: 2025-02-26 01:29:27 -- STEP: 32124/36938 -- GLOBAL_STEP: 1029450 | > loss_text_ce: 0.03498915955424309 (0.03254157198034523) | > loss_mel_ce: 2.8154211044311523 (nan) | > loss: 0.011311152018606663 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1366633533347871) | > loader_time: 0.0047 (0.0046777253711436555)  --> TIME: 2025-02-26 01:29:36 -- STEP: 32174/36938 -- GLOBAL_STEP: 1029500 | > loss_text_ce: 0.04074440896511078 (0.03254117356467267) | > loss_mel_ce: 3.1439452171325684 (nan) | > loss: 0.01263765711337328 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.13666334012611864) | > loader_time: 0.0049 (0.004677690793915632)  --> TIME: 2025-02-26 01:29:45 -- STEP: 32224/36938 -- GLOBAL_STEP: 1029550 | > loss_text_ce: 0.035692229866981506 (0.03254155205461434) | > loss_mel_ce: 2.9438207149505615 (nan) | > loss: 0.01182346511632204 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13666534025773266) | > loader_time: 0.0046 (0.004677642784206323)  --> TIME: 2025-02-26 01:29:55 -- STEP: 32274/36938 -- GLOBAL_STEP: 1029600 | > loss_text_ce: 0.028704732656478882 (0.03253953722716222) | > loss_mel_ce: 2.2701575756073 (nan) | > loss: 0.009122469462454319 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.13666221107360688) | > loader_time: 0.0048 (0.004677607031081945)  --> TIME: 2025-02-26 01:30:04 -- STEP: 32324/36938 -- GLOBAL_STEP: 1029650 | > loss_text_ce: 0.04398753121495247 (0.03253843767297399) | > loss_mel_ce: 2.4558329582214355 (nan) | > loss: 0.009919922798871994 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.1366674692570875) | > loader_time: 0.0046 (0.004677549290370927)  --> TIME: 2025-02-26 01:30:14 -- STEP: 32374/36938 -- GLOBAL_STEP: 1029700 | > loss_text_ce: 0.031330350786447525 (0.032538554327214995) | > loss_mel_ce: 2.2924602031707764 (nan) | > loss: 0.009221391752362251 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13667512524744363) | > loader_time: 0.0045 (0.004677589955821618)  --> TIME: 2025-02-26 01:30:23 -- STEP: 32424/36938 -- GLOBAL_STEP: 1029750 | > loss_text_ce: 0.032127395272254944 (0.032538549004404275) | > loss_mel_ce: 3.074132204055786 (nan) | > loss: 0.012326427735388279 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.13666956691779586) | > loader_time: 0.0046 (0.004677450461297112)  --> TIME: 2025-02-26 01:30:32 -- STEP: 32474/36938 -- GLOBAL_STEP: 1029800 | > loss_text_ce: 0.027514895424246788 (0.03253705945492745) | > loss_mel_ce: 2.587472438812256 (nan) | > loss: 0.01037693489342928 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.13667252237458907) | > loader_time: 0.0047 (0.004677505418864814)  --> TIME: 2025-02-26 01:30:42 -- STEP: 32524/36938 -- GLOBAL_STEP: 1029850 | > loss_text_ce: 0.03297051414847374 (0.03253559053099723) | > loss_mel_ce: 3.152423620223999 (nan) | > loss: 0.012640452943742275 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.1366704857198386) | > loader_time: 0.0046 (0.004677555662520457)  --> TIME: 2025-02-26 01:30:51 -- STEP: 32574/36938 -- GLOBAL_STEP: 1029900 | > loss_text_ce: 0.030322125181555748 (0.03253451136480699) | > loss_mel_ce: 1.5321907997131348 (nan) | > loss: 0.006200448144227266 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13666025775568164) | > loader_time: 0.0045 (0.004677539958826277)  --> TIME: 2025-02-26 01:31:00 -- STEP: 32624/36938 -- GLOBAL_STEP: 1029950 | > loss_text_ce: 0.01638755574822426 (0.03253416936368872) | > loss_mel_ce: 1.3078707456588745 (nan) | > loss: 0.0052549936808645725 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13665634370306176) | > loader_time: 0.0047 (0.004677517214436527)  --> TIME: 2025-02-26 01:31:09 -- STEP: 32674/36938 -- GLOBAL_STEP: 1030000 | > loss_text_ce: 0.038314785808324814 (0.03253404697746602) | > loss_mel_ce: 3.0966796875 (nan) | > loss: 0.0124404551461339 (nan) | > current_lr: 5e-06 | > step_time: 0.1861 (0.13665623692438195) | > loader_time: 0.0048 (0.004677580548097545) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1030000.pth  --> TIME: 2025-02-26 01:31:25 -- STEP: 32724/36938 -- GLOBAL_STEP: 1030050 | > loss_text_ce: 0.03287763148546219 (0.03253480649398753) | > loss_mel_ce: 2.39516282081604 (nan) | > loss: 0.009635081514716148 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13664915633513422) | > loader_time: 0.0049 (0.004677688939955261)  --> TIME: 2025-02-26 01:31:34 -- STEP: 32774/36938 -- GLOBAL_STEP: 1030100 | > loss_text_ce: 0.026686090975999832 (0.03253535094931866) | > loss_mel_ce: 3.2212698459625244 (nan) | > loss: 0.012888715602457523 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.13664965681372965) | > loader_time: 0.0046 (0.00467774481292385)  --> TIME: 2025-02-26 01:31:43 -- STEP: 32824/36938 -- GLOBAL_STEP: 1030150 | > loss_text_ce: 0.026549331843852997 (0.0325355751869726) | > loss_mel_ce: 2.490274429321289 (nan) | > loss: 0.00998739618808031 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13664854777867458) | > loader_time: 0.0047 (0.0046777556414957)  --> TIME: 2025-02-26 01:31:52 -- STEP: 32874/36938 -- GLOBAL_STEP: 1030200 | > loss_text_ce: 0.022341160103678703 (0.032536902697402705) | > loss_mel_ce: 2.722198486328125 (nan) | > loss: 0.010891031473875046 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13664564621044548) | > loader_time: 0.0045 (0.004677760975997852)  --> TIME: 2025-02-26 01:32:01 -- STEP: 32924/36938 -- GLOBAL_STEP: 1030250 | > loss_text_ce: 0.025785576552152634 (0.032536654908387856) | > loss_mel_ce: 3.381788730621338 (nan) | > loss: 0.013522121123969555 (nan) | > current_lr: 5e-06 | > step_time: 0.1888 (0.13664004186019177) | > loader_time: 0.005 (0.004677810040095016)  --> TIME: 2025-02-26 01:32:11 -- STEP: 32974/36938 -- GLOBAL_STEP: 1030300 | > loss_text_ce: 0.031427398324012756 (0.03253440881486792) | > loss_mel_ce: 2.4230010509490967 (nan) | > loss: 0.009739795699715614 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13664941308482037) | > loader_time: 0.0046 (0.004677776383060892)  --> TIME: 2025-02-26 01:32:20 -- STEP: 33024/36938 -- GLOBAL_STEP: 1030350 | > loss_text_ce: 0.03194844722747803 (0.03253439151554033) | > loss_mel_ce: 3.2753684520721436 (nan) | > loss: 0.013124274089932442 (nan) | > current_lr: 5e-06 | > step_time: 0.1955 (0.1366478688726129) | > loader_time: 0.0049 (0.004677667203106689)  --> TIME: 2025-02-26 01:32:30 -- STEP: 33074/36938 -- GLOBAL_STEP: 1030400 | > loss_text_ce: 0.03652890771627426 (0.032534582610508116) | > loss_mel_ce: 2.935500383377075 (nan) | > loss: 0.011793767102062702 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13665297641686094) | > loader_time: 0.0046 (0.004677635709182837)  --> TIME: 2025-02-26 01:32:39 -- STEP: 33124/36938 -- GLOBAL_STEP: 1030450 | > loss_text_ce: 0.023460593074560165 (0.03253437471802646) | > loss_mel_ce: 2.823521137237549 (nan) | > loss: 0.01129754725843668 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.1366495818507914) | > loader_time: 0.0045 (0.004677598084276054)  --> TIME: 2025-02-26 01:32:48 -- STEP: 33174/36938 -- GLOBAL_STEP: 1030500 | > loss_text_ce: 0.04074545204639435 (0.03253427409743967) | > loss_mel_ce: 3.381873607635498 (nan) | > loss: 0.013581822626292706 (nan) | > current_lr: 5e-06 | > step_time: 0.1799 (0.13665385169233168) | > loader_time: 0.005 (0.004677515561163742)  --> TIME: 2025-02-26 01:32:58 -- STEP: 33224/36938 -- GLOBAL_STEP: 1030550 | > loss_text_ce: 0.03616028651595116 (0.032532717092586066) | > loss_mel_ce: 2.6307106018066406 (nan) | > loss: 0.010582821443676949 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13665307874912638) | > loader_time: 0.0045 (0.004677425880705465)  --> TIME: 2025-02-26 01:33:07 -- STEP: 33274/36938 -- GLOBAL_STEP: 1030600 | > loss_text_ce: 0.021001068875193596 (0.03253120595026462) | > loss_mel_ce: 2.2574737071990967 (nan) | > loss: 0.009041567333042622 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.1366470546083839) | > loader_time: 0.0046 (0.004677317431535313)  --> TIME: 2025-02-26 01:33:16 -- STEP: 33324/36938 -- GLOBAL_STEP: 1030650 | > loss_text_ce: 0.03844192251563072 (0.03253095973258006) | > loss_mel_ce: 2.897399425506592 (nan) | > loss: 0.011650164611637592 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13665062596912128) | > loader_time: 0.0045 (0.004677360176024222)  --> TIME: 2025-02-26 01:33:26 -- STEP: 33374/36938 -- GLOBAL_STEP: 1030700 | > loss_text_ce: 0.028586244210600853 (0.032533666929638015) | > loss_mel_ce: 2.166041612625122 (nan) | > loss: 0.008708841167390347 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13665157445447068) | > loader_time: 0.0046 (0.004677265902137699)  --> TIME: 2025-02-26 01:33:35 -- STEP: 33424/36938 -- GLOBAL_STEP: 1030750 | > loss_text_ce: 0.031369712203741074 (0.03253206014871688) | > loss_mel_ce: 2.6350350379943848 (nan) | > loss: 0.010580971837043762 (nan) | > current_lr: 5e-06 | > step_time: 0.1268 (0.13665483972568615) | > loader_time: 0.0045 (0.004677178544139976)  --> TIME: 2025-02-26 01:33:44 -- STEP: 33474/36938 -- GLOBAL_STEP: 1030800 | > loss_text_ce: 0.02761600725352764 (0.03253115810612585) | > loss_mel_ce: 3.0405726432800293 (nan) | > loss: 0.012175352312624454 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13665070641729554) | > loader_time: 0.0048 (0.004677048014107373)  --> TIME: 2025-02-26 01:33:54 -- STEP: 33524/36938 -- GLOBAL_STEP: 1030850 | > loss_text_ce: 0.02185596153140068 (0.03252908847656003) | > loss_mel_ce: 1.7280240058898926 (nan) | > loss: 0.006943968590348959 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13665196639856256) | > loader_time: 0.0044 (0.004676937843589546)  --> TIME: 2025-02-26 01:34:03 -- STEP: 33574/36938 -- GLOBAL_STEP: 1030900 | > loss_text_ce: 0.02825670689344406 (0.0325295446947265) | > loss_mel_ce: 2.443203926086426 (nan) | > loss: 0.009807383641600609 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13665054895726306) | > loader_time: 0.0045 (0.004676926410836686)  --> TIME: 2025-02-26 01:34:13 -- STEP: 33624/36938 -- GLOBAL_STEP: 1030950 | > loss_text_ce: 0.024809328839182854 (0.032530841552916065) | > loss_mel_ce: 2.1116020679473877 (nan) | > loss: 0.008477823808789253 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13665652537442324) | > loader_time: 0.0048 (0.004676883784527228)  --> TIME: 2025-02-26 01:34:22 -- STEP: 33674/36938 -- GLOBAL_STEP: 1031000 | > loss_text_ce: 0.029288163408637047 (0.03252986299941358) | > loss_mel_ce: 2.65293550491333 (nan) | > loss: 0.010643744841217995 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13665557980303772) | > loader_time: 0.0046 (0.004676829793642044)  --> TIME: 2025-02-26 01:34:31 -- STEP: 33724/36938 -- GLOBAL_STEP: 1031050 | > loss_text_ce: 0.034271251410245895 (0.032529170241297665) | > loss_mel_ce: 2.8984951972961426 (nan) | > loss: 0.01163796242326498 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.13666097856390852) | > loader_time: 0.0049 (0.004676843146216881)  --> TIME: 2025-02-26 01:34:40 -- STEP: 33774/36938 -- GLOBAL_STEP: 1031100 | > loss_text_ce: 0.031032146885991096 (0.03252863476055968) | > loss_mel_ce: 2.340909004211426 (nan) | > loss: 0.009412465617060661 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.13665341201455133) | > loader_time: 0.0047 (0.004676763312660115)  --> TIME: 2025-02-26 01:34:50 -- STEP: 33824/36938 -- GLOBAL_STEP: 1031150 | > loss_text_ce: 0.028900133445858955 (0.03252661770381262) | > loss_mel_ce: 2.7965171337127686 (nan) | > loss: 0.011211973614990711 (nan) | > current_lr: 5e-06 | > step_time: 0.1829 (0.13665369182627723) | > loader_time: 0.005 (0.00467673216152559)  --> TIME: 2025-02-26 01:34:59 -- STEP: 33874/36938 -- GLOBAL_STEP: 1031200 | > loss_text_ce: 0.03735336661338806 (0.03252449802795449) | > loss_mel_ce: 2.7610971927642822 (nan) | > loss: 0.011104962788522243 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.13666031140606003) | > loader_time: 0.0052 (0.004676709168353073)  --> TIME: 2025-02-26 01:35:08 -- STEP: 33924/36938 -- GLOBAL_STEP: 1031250 | > loss_text_ce: 0.025882910937070847 (0.03252240983992105) | > loss_mel_ce: 2.8978464603424072 (nan) | > loss: 0.011602101847529411 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13666176155706772) | > loader_time: 0.0046 (0.004676660949112023)  --> TIME: 2025-02-26 01:35:18 -- STEP: 33974/36938 -- GLOBAL_STEP: 1031300 | > loss_text_ce: 0.03469729423522949 (0.032522736755989455) | > loss_mel_ce: 1.88226318359375 (nan) | > loss: 0.007606986444443464 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13665756660232814) | > loader_time: 0.0045 (0.004676620584228498)  --> TIME: 2025-02-26 01:35:27 -- STEP: 34024/36938 -- GLOBAL_STEP: 1031350 | > loss_text_ce: 0.031056616455316544 (0.03252331398853069) | > loss_mel_ce: 2.8547630310058594 (nan) | > loss: 0.011451666243374348 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.13665632548652867) | > loader_time: 0.0046 (0.004676585285181005)  --> TIME: 2025-02-26 01:35:36 -- STEP: 34074/36938 -- GLOBAL_STEP: 1031400 | > loss_text_ce: 0.033718228340148926 (0.03252246453875406) | > loss_mel_ce: 2.9627878665924072 (nan) | > loss: 0.011890898458659649 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.136660036125772) | > loader_time: 0.0049 (0.004676590406916442)  --> TIME: 2025-02-26 01:35:45 -- STEP: 34124/36938 -- GLOBAL_STEP: 1031450 | > loss_text_ce: 0.037396445870399475 (0.032522893861928893) | > loss_mel_ce: 2.6987497806549072 (nan) | > loss: 0.010857723653316498 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.1366563384941798) | > loader_time: 0.0045 (0.0046765302287053116)  --> TIME: 2025-02-26 01:35:55 -- STEP: 34174/36938 -- GLOBAL_STEP: 1031500 | > loss_text_ce: 0.03305375576019287 (0.03252197091741885) | > loss_mel_ce: 1.908552646636963 (nan) | > loss: 0.007704787887632847 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13665498471590762) | > loader_time: 0.0045 (0.004676457864039536)  --> TIME: 2025-02-26 01:36:04 -- STEP: 34224/36938 -- GLOBAL_STEP: 1031550 | > loss_text_ce: 0.03515898808836937 (0.03252295797673654) | > loss_mel_ce: 2.44686222076416 (nan) | > loss: 0.009849290363490582 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13665220509237522) | > loader_time: 0.0045 (0.004676371736189941)  --> TIME: 2025-02-26 01:36:13 -- STEP: 34274/36938 -- GLOBAL_STEP: 1031600 | > loss_text_ce: 0.03866008296608925 (0.0325227674117093) | > loss_mel_ce: 2.960179090499878 (nan) | > loss: 0.011900155805051327 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13665076990962866) | > loader_time: 0.0046 (0.004676302812018415)  --> TIME: 2025-02-26 01:36:23 -- STEP: 34324/36938 -- GLOBAL_STEP: 1031650 | > loss_text_ce: 0.0233682319521904 (0.03252363086543717) | > loss_mel_ce: 2.7261574268341064 (nan) | > loss: 0.010910816490650177 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13665557971104586) | > loader_time: 0.0046 (0.004676261247975704)  --> TIME: 2025-02-26 01:36:32 -- STEP: 34374/36938 -- GLOBAL_STEP: 1031700 | > loss_text_ce: 0.034009840339422226 (0.03252428582979844) | > loss_mel_ce: 2.445570468902588 (nan) | > loss: 0.009839605540037155 (nan) | > current_lr: 5e-06 | > step_time: 0.1857 (0.13665583000942186) | > loader_time: 0.0049 (0.0046762110585351945)  --> TIME: 2025-02-26 01:36:41 -- STEP: 34424/36938 -- GLOBAL_STEP: 1031750 | > loss_text_ce: 0.040128834545612335 (0.03252428916032205) | > loss_mel_ce: 2.1787941455841064 (nan) | > loss: 0.008805250748991966 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.13665295573215053) | > loader_time: 0.0048 (0.004676136961100808)  --> TIME: 2025-02-26 01:36:51 -- STEP: 34474/36938 -- GLOBAL_STEP: 1031800 | > loss_text_ce: 0.037746451795101166 (0.03252397844447509) | > loss_mel_ce: 3.0820722579956055 (nan) | > loss: 0.012380233965814114 (nan) | > current_lr: 5e-06 | > step_time: 0.1741 (0.1366598126976383) | > loader_time: 0.0048 (0.004676099393970484)  --> TIME: 2025-02-26 01:37:00 -- STEP: 34524/36938 -- GLOBAL_STEP: 1031850 | > loss_text_ce: 0.034624677151441574 (0.03252270723040895) | > loss_mel_ce: 2.659437417984009 (nan) | > loss: 0.010690722614526749 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13665285469303118) | > loader_time: 0.0048 (0.004675984942005686)  --> TIME: 2025-02-26 01:37:09 -- STEP: 34574/36938 -- GLOBAL_STEP: 1031900 | > loss_text_ce: 0.0364287905395031 (0.03252342472076149) | > loss_mel_ce: 3.113011598587036 (nan) | > loss: 0.012497779913246632 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.1366566433600101) | > loader_time: 0.0047 (0.004675994554070546)  --> TIME: 2025-02-26 01:37:19 -- STEP: 34624/36938 -- GLOBAL_STEP: 1031950 | > loss_text_ce: 0.025142215192317963 (0.03252209318837153) | > loss_mel_ce: 2.302292823791504 (nan) | > loss: 0.00923585332930088 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.13665641132955625) | > loader_time: 0.0048 (0.0046759928178986155)  --> TIME: 2025-02-26 01:37:28 -- STEP: 34674/36938 -- GLOBAL_STEP: 1032000 | > loss_text_ce: 0.027349313721060753 (0.0325223280959172) | > loss_mel_ce: 2.875037908554077 (nan) | > loss: 0.011517410166561604 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.1366615030202664) | > loader_time: 0.0047 (0.004675922697993328)  --> TIME: 2025-02-26 01:37:37 -- STEP: 34724/36938 -- GLOBAL_STEP: 1032050 | > loss_text_ce: 0.03202835097908974 (0.032521228188064716) | > loss_mel_ce: 2.563725471496582 (nan) | > loss: 0.010300611145794392 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.1366578829465628) | > loader_time: 0.0047 (0.004675817268535549)  --> TIME: 2025-02-26 01:37:47 -- STEP: 34774/36938 -- GLOBAL_STEP: 1032100 | > loss_text_ce: 0.03355329856276512 (0.03252393246874252) | > loss_mel_ce: 2.030646800994873 (nan) | > loss: 0.008191270753741264 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13666005480685067) | > loader_time: 0.0045 (0.0046757501394950746)  --> TIME: 2025-02-26 01:37:56 -- STEP: 34824/36938 -- GLOBAL_STEP: 1032150 | > loss_text_ce: 0.034215155988931656 (0.032524093837210215) | > loss_mel_ce: 2.519923448562622 (nan) | > loss: 0.010135471820831299 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.1366550096240286) | > loader_time: 0.0046 (0.004675709883591136)  --> TIME: 2025-02-26 01:38:05 -- STEP: 34874/36938 -- GLOBAL_STEP: 1032200 | > loss_text_ce: 0.03405262157320976 (0.03252556655099722) | > loss_mel_ce: 2.9832100868225098 (nan) | > loss: 0.011973265558481216 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13665629229004414) | > loader_time: 0.0046 (0.0046756739134279635)  --> TIME: 2025-02-26 01:38:15 -- STEP: 34924/36938 -- GLOBAL_STEP: 1032250 | > loss_text_ce: 0.04160312935709953 (0.03252780241148789) | > loss_mel_ce: 3.0911896228790283 (nan) | > loss: 0.012431717477738857 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13666022155664723) | > loader_time: 0.0047 (0.004675702204369556)  --> TIME: 2025-02-26 01:38:24 -- STEP: 34974/36938 -- GLOBAL_STEP: 1032300 | > loss_text_ce: 0.029895784333348274 (0.03252640617274257) | > loss_mel_ce: 2.7217445373535156 (nan) | > loss: 0.010919208638370037 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.13666394556912334) | > loader_time: 0.0054 (0.004675758998199241)  --> TIME: 2025-02-26 01:38:33 -- STEP: 35024/36938 -- GLOBAL_STEP: 1032350 | > loss_text_ce: 0.03207087144255638 (0.03252717431346091) | > loss_mel_ce: 3.345885992050171 (nan) | > loss: 0.01340459194034338 (nan) | > current_lr: 5e-06 | > step_time: 0.1833 (0.13666021889913788) | > loader_time: 0.0049 (0.004675758435008465)  --> TIME: 2025-02-26 01:38:43 -- STEP: 35074/36938 -- GLOBAL_STEP: 1032400 | > loss_text_ce: 0.03005051612854004 (0.0325286214360127) | > loss_mel_ce: 1.916792631149292 (nan) | > loss: 0.007725568488240242 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13665647815868143) | > loader_time: 0.0047 (0.004675751164204684)  --> TIME: 2025-02-26 01:38:52 -- STEP: 35124/36938 -- GLOBAL_STEP: 1032450 | > loss_text_ce: 0.03014935739338398 (0.03253036069710449) | > loss_mel_ce: 2.811231851577759 (nan) | > loss: 0.01127532310783863 (nan) | > current_lr: 5e-06 | > step_time: 0.1722 (0.13665501869236638) | > loader_time: 0.0048 (0.004675795977377164)  --> TIME: 2025-02-26 01:39:01 -- STEP: 35174/36938 -- GLOBAL_STEP: 1032500 | > loss_text_ce: 0.03602953627705574 (0.03253054301573075) | > loss_mel_ce: 0.6336669921875 (nan) | > loss: 0.0026575259398669004 (nan) | > current_lr: 5e-06 | > step_time: 0.1473 (0.13665599919791632) | > loader_time: 0.0048 (0.0046757959062829965)  --> TIME: 2025-02-26 01:39:10 -- STEP: 35224/36938 -- GLOBAL_STEP: 1032550 | > loss_text_ce: 0.03569789230823517 (0.03253135200460965) | > loss_mel_ce: 3.288587808609009 (nan) | > loss: 0.01319161057472229 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13665666736285048) | > loader_time: 0.0046 (0.00467584108374977)  --> TIME: 2025-02-26 01:39:20 -- STEP: 35274/36938 -- GLOBAL_STEP: 1032600 | > loss_text_ce: 0.0339348204433918 (0.032530858350789095) | > loss_mel_ce: 3.433384895324707 (nan) | > loss: 0.013759206049144268 (nan) | > current_lr: 5e-06 | > step_time: 0.1667 (0.13665875459451385) | > loader_time: 0.0049 (0.004675861057081061)  --> TIME: 2025-02-26 01:39:29 -- STEP: 35324/36938 -- GLOBAL_STEP: 1032650 | > loss_text_ce: 0.03318127617239952 (0.032528794165760704) | > loss_mel_ce: 3.2707161903381348 (nan) | > loss: 0.013110704720020294 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.136656859246106) | > loader_time: 0.0049 (0.004676024096561049)  --> TIME: 2025-02-26 01:39:38 -- STEP: 35374/36938 -- GLOBAL_STEP: 1032700 | > loss_text_ce: 0.03615535423159599 (0.032527576651265704) | > loss_mel_ce: 2.7770893573760986 (nan) | > loss: 0.01116367056965828 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.1366523264315397) | > loader_time: 0.0046 (0.0046760486883812334)  --> TIME: 2025-02-26 01:39:47 -- STEP: 35424/36938 -- GLOBAL_STEP: 1032750 | > loss_text_ce: 0.03617497533559799 (0.03252682804163977) | > loss_mel_ce: 2.9006612300872803 (nan) | > loss: 0.011654112488031387 (nan) | > current_lr: 5e-06 | > step_time: 0.193 (0.13665153902586025) | > loader_time: 0.0051 (0.004676132997146317)  --> TIME: 2025-02-26 01:39:57 -- STEP: 35474/36938 -- GLOBAL_STEP: 1032800 | > loss_text_ce: 0.02111944369971752 (0.03252457519203465) | > loss_mel_ce: 2.263594388961792 (nan) | > loss: 0.009066324681043625 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13664699200948877) | > loader_time: 0.0046 (0.0046761875767688844)  --> TIME: 2025-02-26 01:40:06 -- STEP: 35524/36938 -- GLOBAL_STEP: 1032850 | > loss_text_ce: 0.025391079485416412 (0.03252451387988014) | > loss_mel_ce: 2.352504014968872 (nan) | > loss: 0.009436092339456081 (nan) | > current_lr: 5e-06 | > step_time: 0.1506 (0.1366515650148932) | > loader_time: 0.0047 (0.004676216499129802)  --> TIME: 2025-02-26 01:40:15 -- STEP: 35574/36938 -- GLOBAL_STEP: 1032900 | > loss_text_ce: 0.026732074096798897 (0.032524807104991456) | > loss_mel_ce: 3.012019157409668 (nan) | > loss: 0.012058536522090435 (nan) | > current_lr: 5e-06 | > step_time: 0.1906 (0.13664944649627347) | > loader_time: 0.0049 (0.004676262028282926)  --> TIME: 2025-02-26 01:40:25 -- STEP: 35624/36938 -- GLOBAL_STEP: 1032950 | > loss_text_ce: 0.03255583345890045 (0.03252518860390138) | > loss_mel_ce: 2.7488064765930176 (nan) | > loss: 0.011037152260541916 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13664475268067972) | > loader_time: 0.0047 (0.004676368145254214)  --> TIME: 2025-02-26 01:40:34 -- STEP: 35674/36938 -- GLOBAL_STEP: 1033000 | > loss_text_ce: 0.03853411599993706 (0.03252461363575067) | > loss_mel_ce: 3.3549082279205322 (nan) | > loss: 0.013466042466461658 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13664530709953115) | > loader_time: 0.0047 (0.004676469620644138)  --> TIME: 2025-02-26 01:40:43 -- STEP: 35724/36938 -- GLOBAL_STEP: 1033050 | > loss_text_ce: 0.0424286350607872 (0.03252417511586579) | > loss_mel_ce: 0.3076171875 (nan) | > loss: 0.0013890708796679974 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.13664097151127874) | > loader_time: 0.0048 (0.004676469101669363)  --> TIME: 2025-02-26 01:40:53 -- STEP: 35774/36938 -- GLOBAL_STEP: 1033100 | > loss_text_ce: 0.038490179926157 (0.03252451138627334) | > loss_mel_ce: 1.8539612293243408 (nan) | > loss: 0.00750972842797637 (nan) | > current_lr: 5e-06 | > step_time: 0.1872 (0.13664378896343135) | > loader_time: 0.0049 (0.004676579749290355)  --> TIME: 2025-02-26 01:41:02 -- STEP: 35824/36938 -- GLOBAL_STEP: 1033150 | > loss_text_ce: 0.029311394318938255 (0.03252069091945528) | > loss_mel_ce: 2.627124071121216 (nan) | > loss: 0.010541411116719246 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13664225887884898) | > loader_time: 0.0047 (0.004676557368548614)  --> TIME: 2025-02-26 01:41:12 -- STEP: 35874/36938 -- GLOBAL_STEP: 1033200 | > loss_text_ce: 0.031175851821899414 (0.03252136212609901) | > loss_mel_ce: 2.9176013469696045 (nan) | > loss: 0.011701497249305248 (nan) | > current_lr: 5e-06 | > step_time: 0.1629 (0.13665519349782537) | > loader_time: 0.0049 (0.004676623741054692)  --> TIME: 2025-02-26 01:41:21 -- STEP: 35924/36938 -- GLOBAL_STEP: 1033250 | > loss_text_ce: 0.03628343716263771 (0.03252218990780502) | > loss_mel_ce: 3.12638783454895 (nan) | > loss: 0.012550284154713154 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.1366588970943464) | > loader_time: 0.0047 (0.00467665859670957)  --> TIME: 2025-02-26 01:41:30 -- STEP: 35974/36938 -- GLOBAL_STEP: 1033300 | > loss_text_ce: 0.047170817852020264 (0.03252131899083216) | > loss_mel_ce: 2.140259027481079 (nan) | > loss: 0.00868027750402689 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.1366512822078917) | > loader_time: 0.0046 (0.0046766025981478535)  --> TIME: 2025-02-26 01:41:40 -- STEP: 36024/36938 -- GLOBAL_STEP: 1033350 | > loss_text_ce: 0.03132079169154167 (0.032521214649286305) | > loss_mel_ce: 1.9545356035232544 (nan) | > loss: 0.007880383171141148 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.1366550004299602) | > loader_time: 0.0046 (0.0046766019651103136)  --> TIME: 2025-02-26 01:41:49 -- STEP: 36074/36938 -- GLOBAL_STEP: 1033400 | > loss_text_ce: 0.03338921070098877 (0.032520829571984364) | > loss_mel_ce: 2.783428907394409 (nan) | > loss: 0.011177850887179375 (nan) | > current_lr: 5e-06 | > step_time: 0.1537 (0.13665126978957778) | > loader_time: 0.0046 (0.004676649871445971)  --> TIME: 2025-02-26 01:41:58 -- STEP: 36124/36938 -- GLOBAL_STEP: 1033450 | > loss_text_ce: 0.03527849540114403 (0.03251936515161452) | > loss_mel_ce: 0.043597135692834854 (nan) | > loss: 0.00031299854163080454 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1366517556134731) | > loader_time: 0.0045 (0.004676585478081005)  --> TIME: 2025-02-26 01:42:08 -- STEP: 36174/36938 -- GLOBAL_STEP: 1033500 | > loss_text_ce: 0.032748013734817505 (0.03251950443553046) | > loss_mel_ce: 3.0081734657287598 (nan) | > loss: 0.012067149393260479 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.13665644396344626) | > loader_time: 0.0048 (0.004676562027333619)  --> TIME: 2025-02-26 01:42:17 -- STEP: 36224/36938 -- GLOBAL_STEP: 1033550 | > loss_text_ce: 0.01290688756853342 (0.03252049210796955) | > loss_mel_ce: 2.2433931827545166 (nan) | > loss: 0.008953572250902653 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13665036090350163) | > loader_time: 0.0044 (0.004676463490503482)  --> TIME: 2025-02-26 01:42:26 -- STEP: 36274/36938 -- GLOBAL_STEP: 1033600 | > loss_text_ce: 0.03831744194030762 (0.0325214910900568) | > loss_mel_ce: 3.113224983215332 (nan) | > loss: 0.012506121769547462 (nan) | > current_lr: 5e-06 | > step_time: 0.1521 (0.13665042274768682) | > loader_time: 0.0046 (0.004676462974701944)  --> TIME: 2025-02-26 01:42:36 -- STEP: 36324/36938 -- GLOBAL_STEP: 1033650 | > loss_text_ce: 0.039194606244564056 (0.03252093645679611) | > loss_mel_ce: 2.5230438709259033 (nan) | > loss: 0.010167613625526428 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13665000853618817) | > loader_time: 0.0048 (0.0046764215096124755)  --> TIME: 2025-02-26 01:42:45 -- STEP: 36374/36938 -- GLOBAL_STEP: 1033700 | > loss_text_ce: 0.034100912511348724 (0.03251986972069071) | > loss_mel_ce: 2.753038167953491 (nan) | > loss: 0.011060076765716076 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.13664766040833848) | > loader_time: 0.0045 (0.004676358875594174)  --> TIME: 2025-02-26 01:42:54 -- STEP: 36424/36938 -- GLOBAL_STEP: 1033750 | > loss_text_ce: 0.034012001007795334 (0.032519257423566456) | > loss_mel_ce: 2.735874652862549 (nan) | > loss: 0.010991614311933517 (nan) | > current_lr: 5e-06 | > step_time: 0.1517 (0.13664094926680959) | > loader_time: 0.0047 (0.004676427882816894)  --> TIME: 2025-02-26 01:43:04 -- STEP: 36474/36938 -- GLOBAL_STEP: 1033800 | > loss_text_ce: 0.03462100028991699 (0.03251920978723499) | > loss_mel_ce: 2.4240305423736572 (nan) | > loss: 0.009756553918123245 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13664757593581273) | > loader_time: 0.0047 (0.004676451486682869)  --> TIME: 2025-02-26 01:43:13 -- STEP: 36524/36938 -- GLOBAL_STEP: 1033850 | > loss_text_ce: 0.027413303032517433 (0.03251846202390305) | > loss_mel_ce: 3.2810909748077393 (nan) | > loss: 0.013128986582159996 (nan) | > current_lr: 5e-06 | > step_time: 0.1835 (0.1366461880522216) | > loader_time: 0.0049 (0.004676451695838838)  --> TIME: 2025-02-26 01:43:22 -- STEP: 36574/36938 -- GLOBAL_STEP: 1033900 | > loss_text_ce: 0.03441642224788666 (0.03251759638950647) | > loss_mel_ce: 2.809248685836792 (nan) | > loss: 0.011284385807812214 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.1366525888051861) | > loader_time: 0.0046 (0.00467648178008182)  --> TIME: 2025-02-26 01:43:31 -- STEP: 36624/36938 -- GLOBAL_STEP: 1033950 | > loss_text_ce: 0.032697949558496475 (0.032518523298954535) | > loss_mel_ce: 3.2295093536376953 (nan) | > loss: 0.012945267371833324 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.13664368712886404) | > loader_time: 0.0049 (0.004676442184839943)  --> TIME: 2025-02-26 01:43:41 -- STEP: 36674/36938 -- GLOBAL_STEP: 1034000 | > loss_text_ce: 0.032873064279556274 (0.03251775487138618) | > loss_mel_ce: 1.9735822677612305 (nan) | > loss: 0.007962125353515148 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13664702986360247) | > loader_time: 0.0046 (0.004676466934188355)  --> TIME: 2025-02-26 01:43:50 -- STEP: 36724/36938 -- GLOBAL_STEP: 1034050 | > loss_text_ce: 0.03059142455458641 (0.032517459326463334) | > loss_mel_ce: 3.330148696899414 (nan) | > loss: 0.013336271047592163 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.1366464131188251) | > loader_time: 0.0046 (0.004676510034440105)  --> TIME: 2025-02-26 01:44:00 -- STEP: 36774/36938 -- GLOBAL_STEP: 1034100 | > loss_text_ce: 0.03363674506545067 (0.032517729474831975) | > loss_mel_ce: 3.1743741035461426 (nan) | > loss: 0.01273020263761282 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13665066266154496) | > loader_time: 0.005 (0.0046765196866051535)  --> TIME: 2025-02-26 01:44:09 -- STEP: 36824/36938 -- GLOBAL_STEP: 1034150 | > loss_text_ce: 0.02735726349055767 (0.03251514388246803) | > loss_mel_ce: 3.366309404373169 (nan) | > loss: 0.013466931879520416 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.1366511291132109) | > loader_time: 0.0046 (0.004676425965899899)  --> TIME: 2025-02-26 01:44:18 -- STEP: 36874/36938 -- GLOBAL_STEP: 1034200 | > loss_text_ce: 0.031034817919135094 (0.0325135315972294) | > loss_mel_ce: 2.813267230987549 (nan) | > loss: 0.01128691341727972 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.13665233591654566) | > loader_time: 0.0046 (0.004676354806244934)  --> TIME: 2025-02-26 01:44:27 -- STEP: 36924/36938 -- GLOBAL_STEP: 1034250 | > loss_text_ce: 0.03179952874779701 (0.0325138795083069) | > loss_mel_ce: 2.7373294830322266 (nan) | > loss: 0.010988608002662659 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13664708618171884) | > loader_time: 0.0045 (0.004676291000131561)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.02081391953005651 (-0.0003537683203668589) | > avg_loss_text_ce: 0.031670478375462593 (-0.0002443153540244297) | > avg_loss_mel_ce: 2.4852544886641024 (-0.011885041942687469) | > avg_loss: 2.5169249590080587 (-0.012129364627420536) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_1034264.pth  > EPOCH: 29/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-26 01:44:57)   --> TIME: 2025-02-26 01:45:03 -- STEP: 36/36938 -- GLOBAL_STEP: 1034300 | > loss_text_ce: 0.02802610769867897 (0.0335806186000506) | > loss_mel_ce: 2.3104031085968018 (nan) | > loss: 0.009279481135308743 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1264091001616584) | > loader_time: 0.0051 (0.0048459702067905)  --> TIME: 2025-02-26 01:45:13 -- STEP: 86/36938 -- GLOBAL_STEP: 1034350 | > loss_text_ce: 0.03952302411198616 (0.03319159182611593) | > loss_mel_ce: 2.8194632530212402 (nan) | > loss: 0.011345184408128262 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13298710834148317) | > loader_time: 0.0048 (0.004843445711357651)  --> TIME: 2025-02-26 01:45:22 -- STEP: 136/36938 -- GLOBAL_STEP: 1034400 | > loss_text_ce: 0.033237848430871964 (0.03285041921224226) | > loss_mel_ce: 1.7605671882629395 (nan) | > loss: 0.007118274457752705 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13388349028194652) | > loader_time: 0.0048 (0.004831899614895093)  --> TIME: 2025-02-26 01:45:31 -- STEP: 186/36938 -- GLOBAL_STEP: 1034450 | > loss_text_ce: 0.02669907733798027 (0.03225246477391452) | > loss_mel_ce: 3.2344655990600586 (nan) | > loss: 0.012941130436956882 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.13414298718975443) | > loader_time: 0.0046 (0.004820953133285692)  --> TIME: 2025-02-26 01:45:41 -- STEP: 236/36938 -- GLOBAL_STEP: 1034500 | > loss_text_ce: 0.031373847275972366 (0.03247346352564832) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13545572252596846) | > loader_time: 0.0046 (0.004810529240107136)  --> TIME: 2025-02-26 01:45:50 -- STEP: 286/36938 -- GLOBAL_STEP: 1034550 | > loss_text_ce: 0.023230385035276413 (0.032324010193660505) | > loss_mel_ce: 2.875948667526245 (nan) | > loss: 0.01150467898696661 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13521102591828038) | > loader_time: 0.0049 (0.004791599053602956)  --> TIME: 2025-02-26 01:45:59 -- STEP: 336/36938 -- GLOBAL_STEP: 1034600 | > loss_text_ce: 0.0267915241420269 (0.032353931677616375) | > loss_mel_ce: 2.3941407203674316 (nan) | > loss: 0.00960687454789877 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13585881817908513) | > loader_time: 0.0046 (0.0047732350372132835)  --> TIME: 2025-02-26 01:46:09 -- STEP: 386/36938 -- GLOBAL_STEP: 1034650 | > loss_text_ce: 0.03596390038728714 (0.03232170616812189) | > loss_mel_ce: 2.7328062057495117 (nan) | > loss: 0.010987184010446072 (nan) | > current_lr: 5e-06 | > step_time: 0.1261 (0.13612172270068226) | > loader_time: 0.0046 (0.004755337621263899)  --> TIME: 2025-02-26 01:46:18 -- STEP: 436/36938 -- GLOBAL_STEP: 1034700 | > loss_text_ce: 0.03139315918087959 (0.032349173688765484) | > loss_mel_ce: 3.063857316970825 (nan) | > loss: 0.012282740324735641 (nan) | > current_lr: 5e-06 | > step_time: 0.1435 (0.13600167604761382) | > loader_time: 0.0046 (0.004741876497181191)  --> TIME: 2025-02-26 01:46:27 -- STEP: 486/36938 -- GLOBAL_STEP: 1034750 | > loss_text_ce: 0.029102616012096405 (0.03228522174121663) | > loss_mel_ce: 2.428558349609375 (nan) | > loss: 0.009752622805535793 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13613822185453558) | > loader_time: 0.0045 (0.004727472984251178)  --> TIME: 2025-02-26 01:46:37 -- STEP: 536/36938 -- GLOBAL_STEP: 1034800 | > loss_text_ce: 0.034460414201021194 (0.03240520711432194) | > loss_mel_ce: 2.9784908294677734 (nan) | > loss: 0.011956156231462955 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.1363410157943839) | > loader_time: 0.0048 (0.004716598275882101)  --> TIME: 2025-02-26 01:46:46 -- STEP: 586/36938 -- GLOBAL_STEP: 1034850 | > loss_text_ce: 0.029235992580652237 (0.03247710498869925) | > loss_mel_ce: 2.9664323329925537 (nan) | > loss: 0.011887573637068272 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.1364828893349032) | > loader_time: 0.0046 (0.004709538746205618)  --> TIME: 2025-02-26 01:46:55 -- STEP: 636/36938 -- GLOBAL_STEP: 1034900 | > loss_text_ce: 0.028158631175756454 (0.03259426866417879) | > loss_mel_ce: 2.4027655124664307 (nan) | > loss: 0.009646524675190449 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13647403657061502) | > loader_time: 0.0046 (0.004713227538942542)  --> TIME: 2025-02-26 01:47:04 -- STEP: 686/36938 -- GLOBAL_STEP: 1034950 | > loss_text_ce: 0.04967446252703667 (0.03262273119029429) | > loss_mel_ce: 1.4509766101837158 (nan) | > loss: 0.005954965017735958 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13597468940579158) | > loader_time: 0.0045 (0.004705554194770125)  --> TIME: 2025-02-26 01:47:14 -- STEP: 736/36938 -- GLOBAL_STEP: 1035000 | > loss_text_ce: 0.034414198249578476 (0.03248374927763666) | > loss_mel_ce: 2.507800579071045 (nan) | > loss: 0.010088155046105385 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13599235661651776) | > loader_time: 0.0049 (0.004705138828443442)  --> TIME: 2025-02-26 01:47:23 -- STEP: 786/36938 -- GLOBAL_STEP: 1035050 | > loss_text_ce: 0.046846985816955566 (0.03255402005417396) | > loss_mel_ce: 2.997124433517456 (nan) | > loss: 0.012079252861440182 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.13620728603149493) | > loader_time: 0.0048 (0.004699676394765916)  --> TIME: 2025-02-26 01:47:33 -- STEP: 836/36938 -- GLOBAL_STEP: 1035100 | > loss_text_ce: 0.020402271300554276 (0.03251391564580537) | > loss_mel_ce: 1.1936696767807007 (nan) | > loss: 0.004817746113985777 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.136652812147825) | > loader_time: 0.0046 (0.004694239944932562)  --> TIME: 2025-02-26 01:47:42 -- STEP: 886/36938 -- GLOBAL_STEP: 1035150 | > loss_text_ce: 0.03327663987874985 (0.03255730406980358) | > loss_mel_ce: 0.6153564453125 (nan) | > loss: 0.0025739409029483795 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1365574576784742) | > loader_time: 0.0045 (0.004688132428154181)  --> TIME: 2025-02-26 01:47:51 -- STEP: 936/36938 -- GLOBAL_STEP: 1035200 | > loss_text_ce: 0.029227707535028458 (0.03254198408327434) | > loss_mel_ce: 2.4472179412841797 (nan) | > loss: 0.009827165864408016 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13647969079832764) | > loader_time: 0.0045 (0.004684555224883251)  --> TIME: 2025-02-26 01:48:01 -- STEP: 986/36938 -- GLOBAL_STEP: 1035250 | > loss_text_ce: 0.028653264045715332 (0.03255359144998502) | > loss_mel_ce: 2.7630960941314697 (nan) | > loss: 0.011078371666371822 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13664860319172412) | > loader_time: 0.0046 (0.00468244562284458)  --> TIME: 2025-02-26 01:48:10 -- STEP: 1036/36938 -- GLOBAL_STEP: 1035300 | > loss_text_ce: 0.015362906269729137 (0.03253758235198085) | > loss_mel_ce: 1.81513249874115 (nan) | > loss: 0.007263870909810066 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1365189071327564) | > loader_time: 0.0045 (0.004679964546071056)  --> TIME: 2025-02-26 01:48:19 -- STEP: 1086/36938 -- GLOBAL_STEP: 1035350 | > loss_text_ce: 0.027320824563503265 (0.03255680647831331) | > loss_mel_ce: 2.4599978923797607 (nan) | > loss: 0.009870313107967377 (nan) | > current_lr: 5e-06 | > step_time: 0.1436 (0.13624924520102663) | > loader_time: 0.0046 (0.0046785016086220085)  --> TIME: 2025-02-26 01:48:28 -- STEP: 1136/36938 -- GLOBAL_STEP: 1035400 | > loss_text_ce: 0.020947370678186417 (0.03253473456568607) | > loss_mel_ce: 1.8357551097869873 (nan) | > loss: 0.007367867510765791 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13627569419397453) | > loader_time: 0.0044 (0.004677177105151429)  --> TIME: 2025-02-26 01:48:37 -- STEP: 1186/36938 -- GLOBAL_STEP: 1035450 | > loss_text_ce: 0.03768828511238098 (0.03250286526016863) | > loss_mel_ce: 3.0507006645202637 (nan) | > loss: 0.012255512177944183 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.13641628467212458) | > loader_time: 0.0047 (0.004675981166189876)  --> TIME: 2025-02-26 01:48:47 -- STEP: 1236/36938 -- GLOBAL_STEP: 1035500 | > loss_text_ce: 0.036222271621227264 (0.03251328663659384) | > loss_mel_ce: 1.5621060132980347 (nan) | > loss: 0.006342572625726461 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.1365517649835756) | > loader_time: 0.0045 (0.0046753734832442675)  --> TIME: 2025-02-26 01:48:56 -- STEP: 1286/36938 -- GLOBAL_STEP: 1035550 | > loss_text_ce: 0.035116832703351974 (0.03251072094945208) | > loss_mel_ce: 2.572138786315918 (nan) | > loss: 0.010346253402531147 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.13656164198233248) | > loader_time: 0.0047 (0.0046753178859016515)  --> TIME: 2025-02-26 01:49:06 -- STEP: 1336/36938 -- GLOBAL_STEP: 1035600 | > loss_text_ce: 0.03703835606575012 (0.03252154002161315) | > loss_mel_ce: 2.193966865539551 (nan) | > loss: 0.008853196166455746 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13658850278683088) | > loader_time: 0.0046 (0.004673725652123632)  --> TIME: 2025-02-26 01:49:15 -- STEP: 1386/36938 -- GLOBAL_STEP: 1035650 | > loss_text_ce: 0.024990364909172058 (0.03249293196715238) | > loss_mel_ce: 2.4967548847198486 (nan) | > loss: 0.010006926022469997 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13664107560079364) | > loader_time: 0.0045 (0.004671487457308416)  --> TIME: 2025-02-26 01:49:25 -- STEP: 1436/36938 -- GLOBAL_STEP: 1035700 | > loss_text_ce: 0.029715191572904587 (0.03247855528462733) | > loss_mel_ce: 2.22196364402771 (nan) | > loss: 0.008935234509408474 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13675772678885312) | > loader_time: 0.0046 (0.004669010639190669)  --> TIME: 2025-02-26 01:49:34 -- STEP: 1486/36938 -- GLOBAL_STEP: 1035750 | > loss_text_ce: 0.031716182827949524 (0.03250826989873384) | > loss_mel_ce: 2.867335557937622 (nan) | > loss: 0.011504174210131168 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13697007013682908) | > loader_time: 0.0046 (0.004669069441619499)  --> TIME: 2025-02-26 01:49:44 -- STEP: 1536/36938 -- GLOBAL_STEP: 1035800 | > loss_text_ce: 0.023815829306840897 (0.03248637018017088) | > loss_mel_ce: 2.9492478370666504 (nan) | > loss: 0.011797872371971607 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13695146826406343) | > loader_time: 0.0045 (0.00466862445076306)  --> TIME: 2025-02-26 01:49:53 -- STEP: 1586/36938 -- GLOBAL_STEP: 1035850 | > loss_text_ce: 0.029398785904049873 (0.032480207378159875) | > loss_mel_ce: 2.788663387298584 (nan) | > loss: 0.011182786896824837 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13694023169004024) | > loader_time: 0.0047 (0.004670584487433989)  --> TIME: 2025-02-26 01:50:02 -- STEP: 1636/36938 -- GLOBAL_STEP: 1035900 | > loss_text_ce: 0.03158615157008171 (0.032468592737229914) | > loss_mel_ce: 2.8772594928741455 (nan) | > loss: 0.011543039232492447 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13691551746540975) | > loader_time: 0.005 (0.00467024207989569)  --> TIME: 2025-02-26 01:50:11 -- STEP: 1686/36938 -- GLOBAL_STEP: 1035950 | > loss_text_ce: 0.025834545493125916 (0.03247758935891709) | > loss_mel_ce: 0.125030517578125 (nan) | > loss: 0.0005986709147691727 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13676377590016553) | > loader_time: 0.0045 (0.004669354196686473)  --> TIME: 2025-02-26 01:50:20 -- STEP: 1736/36938 -- GLOBAL_STEP: 1036000 | > loss_text_ce: 0.025399383157491684 (0.03250542477852661) | > loss_mel_ce: 0.019203949719667435 (nan) | > loss: 0.00017699736054055393 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13679312519763495) | > loader_time: 0.0045 (0.004669128200425525)  --> TIME: 2025-02-26 01:50:30 -- STEP: 1786/36938 -- GLOBAL_STEP: 1036050 | > loss_text_ce: 0.019051121547818184 (0.0325017161995866) | > loss_mel_ce: 2.1787285804748535 (nan) | > loss: 0.008721347898244858 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13682797187386556) | > loader_time: 0.0046 (0.004669330288287902)  --> TIME: 2025-02-26 01:50:39 -- STEP: 1836/36938 -- GLOBAL_STEP: 1036100 | > loss_text_ce: 0.029889559373259544 (0.032486801496291066) | > loss_mel_ce: 2.4709768295288086 (nan) | > loss: 0.009924073703587055 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13667398976864126) | > loader_time: 0.0046 (0.004669143613387292)  --> TIME: 2025-02-26 01:50:48 -- STEP: 1886/36938 -- GLOBAL_STEP: 1036150 | > loss_text_ce: 0.027138201519846916 (0.03246675728408437) | > loss_mel_ce: 2.3248653411865234 (nan) | > loss: 0.009333347901701927 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13665939097945537) | > loader_time: 0.0045 (0.004668043880927849)  --> TIME: 2025-02-26 01:50:57 -- STEP: 1936/36938 -- GLOBAL_STEP: 1036200 | > loss_text_ce: 0.03644918277859688 (0.032487113675302805) | > loss_mel_ce: 2.4472496509552 (nan) | > loss: 0.009855948388576508 (nan) | > current_lr: 5e-06 | > step_time: 0.1533 (0.1365583474971046) | > loader_time: 0.0047 (0.004666817336043046)  --> TIME: 2025-02-26 01:51:07 -- STEP: 1986/36938 -- GLOBAL_STEP: 1036250 | > loss_text_ce: 0.033419668674468994 (0.03248376576258077) | > loss_mel_ce: 2.214409828186035 (nan) | > loss: 0.008919958956539631 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13660333103763853) | > loader_time: 0.0045 (0.004666175847087137)  --> TIME: 2025-02-26 01:51:16 -- STEP: 2036/36938 -- GLOBAL_STEP: 1036300 | > loss_text_ce: 0.024940315634012222 (0.0325125711617813) | > loss_mel_ce: 2.1416015625 (nan) | > loss: 0.008597388863563538 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.13656640685143312) | > loader_time: 0.0045 (0.004664582564460735)  --> TIME: 2025-02-26 01:51:25 -- STEP: 2086/36938 -- GLOBAL_STEP: 1036350 | > loss_text_ce: 0.03326088935136795 (0.032500475569613094) | > loss_mel_ce: 3.1333060264587402 (nan) | > loss: 0.012565742246806622 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13655754463784647) | > loader_time: 0.0048 (0.00466452086074697)  --> TIME: 2025-02-26 01:51:34 -- STEP: 2136/36938 -- GLOBAL_STEP: 1036400 | > loss_text_ce: 0.032053809612989426 (0.03246073495755365) | > loss_mel_ce: 2.671207904815674 (nan) | > loss: 0.0107272295281291 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.1365914553515473) | > loader_time: 0.0046 (0.004663344291265533)  --> TIME: 2025-02-26 01:51:44 -- STEP: 2186/36938 -- GLOBAL_STEP: 1036450 | > loss_text_ce: 0.03751581162214279 (0.03243847264143949) | > loss_mel_ce: 3.2220096588134766 (nan) | > loss: 0.012934626080095768 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13665366358202952) | > loader_time: 0.0045 (0.00466226080853461)  --> TIME: 2025-02-26 01:51:53 -- STEP: 2236/36938 -- GLOBAL_STEP: 1036500 | > loss_text_ce: 0.03180199861526489 (0.032500033986993054) | > loss_mel_ce: 2.1698389053344727 (nan) | > loss: 0.008736670948565006 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.13661255990030077) | > loader_time: 0.0046 (0.00466063314772249)  --> TIME: 2025-02-26 01:52:02 -- STEP: 2286/36938 -- GLOBAL_STEP: 1036550 | > loss_text_ce: 0.02906888723373413 (0.0325003343647345) | > loss_mel_ce: 0.0702260360121727 (nan) | > loss: 0.00039402750553563237 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13657630699721726) | > loader_time: 0.0045 (0.0046595630161822684)  --> TIME: 2025-02-26 01:52:12 -- STEP: 2336/36938 -- GLOBAL_STEP: 1036600 | > loss_text_ce: 0.027410291135311127 (0.03252171718257113) | > loss_mel_ce: 2.513282299041748 (nan) | > loss: 0.010082113556563854 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13658726623613518) | > loader_time: 0.0047 (0.0046602814164880164)  --> TIME: 2025-02-26 01:52:21 -- STEP: 2386/36938 -- GLOBAL_STEP: 1036650 | > loss_text_ce: 0.03288201987743378 (0.03249606034232108) | > loss_mel_ce: 3.162578821182251 (nan) | > loss: 0.012680401094257832 (nan) | > current_lr: 5e-06 | > step_time: 0.169 (0.13662563405512548) | > loader_time: 0.0048 (0.004660460495289293)  --> TIME: 2025-02-26 01:52:30 -- STEP: 2436/36938 -- GLOBAL_STEP: 1036700 | > loss_text_ce: 0.0320771150290966 (0.03249158513577802) | > loss_mel_ce: 1.75 (nan) | > loss: 0.007071734871715307 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13661103295575197) | > loader_time: 0.0046 (0.0046604614343940705)  --> TIME: 2025-02-26 01:52:40 -- STEP: 2486/36938 -- GLOBAL_STEP: 1036750 | > loss_text_ce: 0.0405106395483017 (0.03246199893701468) | > loss_mel_ce: 3.0763607025146484 (nan) | > loss: 0.012368537485599518 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.13662329021232872) | > loader_time: 0.0048 (0.004661388197735442)  --> TIME: 2025-02-26 01:52:49 -- STEP: 2536/36938 -- GLOBAL_STEP: 1036800 | > loss_text_ce: 0.034926049411296844 (0.03244596981070355) | > loss_mel_ce: 3.1088600158691406 (nan) | > loss: 0.012475341558456421 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.13657686941631184) | > loader_time: 0.0049 (0.0046606830994037375)  --> TIME: 2025-02-26 01:52:58 -- STEP: 2586/36938 -- GLOBAL_STEP: 1036850 | > loss_text_ce: 0.030104344710707664 (0.03245461450987674) | > loss_mel_ce: 2.9315571784973145 (nan) | > loss: 0.011752625927329063 (nan) | > current_lr: 5e-06 | > step_time: 0.1773 (0.1366298170119221) | > loader_time: 0.0048 (0.00465938054768502)  --> TIME: 2025-02-26 01:53:07 -- STEP: 2636/36938 -- GLOBAL_STEP: 1036900 | > loss_text_ce: 0.02439851686358452 (0.03247340317718037) | > loss_mel_ce: 2.3004262447357178 (nan) | > loss: 0.009225496090948582 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13652572438279087) | > loader_time: 0.0046 (0.00465868103666985)  --> TIME: 2025-02-26 01:53:17 -- STEP: 2686/36938 -- GLOBAL_STEP: 1036950 | > loss_text_ce: 0.03612508997321129 (0.032477367748745704) | > loss_mel_ce: 1.811798095703125 (nan) | > loss: 0.007333029061555862 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13647105306808544) | > loader_time: 0.0046 (0.0046586126688249185)  --> TIME: 2025-02-26 01:53:26 -- STEP: 2736/36938 -- GLOBAL_STEP: 1037000 | > loss_text_ce: 0.03342166543006897 (0.032496146664053455) | > loss_mel_ce: 2.641254186630249 (nan) | > loss: 0.010613794438540936 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.1365406923649601) | > loader_time: 0.0047 (0.004658730145086316)  --> TIME: 2025-02-26 01:53:35 -- STEP: 2786/36938 -- GLOBAL_STEP: 1037050 | > loss_text_ce: 0.03282318264245987 (0.032498092312708185) | > loss_mel_ce: 2.4369070529937744 (nan) | > loss: 0.009800517000257969 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13654222919713288) | > loader_time: 0.0045 (0.004657718917917871)  --> TIME: 2025-02-26 01:53:45 -- STEP: 2836/36938 -- GLOBAL_STEP: 1037100 | > loss_text_ce: 0.04058685153722763 (0.032481623323219286) | > loss_mel_ce: 2.866731643676758 (nan) | > loss: 0.011536979116499424 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.1366011688672266) | > loader_time: 0.0046 (0.004657556543229154)  --> TIME: 2025-02-26 01:53:54 -- STEP: 2886/36938 -- GLOBAL_STEP: 1037150 | > loss_text_ce: 0.028908507898449898 (0.032497663667576615) | > loss_mel_ce: 2.6400678157806396 (nan) | > loss: 0.010591176338493824 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.1366106108997894) | > loader_time: 0.0046 (0.004656825805817385)  --> TIME: 2025-02-26 01:54:04 -- STEP: 2936/36938 -- GLOBAL_STEP: 1037200 | > loss_text_ce: 0.03508303686976433 (0.032515086581593085) | > loss_mel_ce: 3.23336124420166 (nan) | > loss: 0.012970018200576305 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.13667594525729573) | > loader_time: 0.0047 (0.004657054873188442)  --> TIME: 2025-02-26 01:54:13 -- STEP: 2986/36938 -- GLOBAL_STEP: 1037250 | > loss_text_ce: 0.04310078173875809 (0.032530011250827794) | > loss_mel_ce: 2.7884926795959473 (nan) | > loss: 0.011236483231186867 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13674527394715663) | > loader_time: 0.0049 (0.004656934929788309)  --> TIME: 2025-02-26 01:54:23 -- STEP: 3036/36938 -- GLOBAL_STEP: 1037300 | > loss_text_ce: 0.03020460344851017 (0.03252258851641995) | > loss_mel_ce: 2.5536656379699707 (nan) | > loss: 0.010253453627228737 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13684152378866296) | > loader_time: 0.0048 (0.00465728666471398)  --> TIME: 2025-02-26 01:54:32 -- STEP: 3086/36938 -- GLOBAL_STEP: 1037350 | > loss_text_ce: 0.03405965864658356 (0.03250956222021445) | > loss_mel_ce: 2.9064459800720215 (nan) | > loss: 0.011668674647808075 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13681560843622817) | > loader_time: 0.0045 (0.004656754912143388)  --> TIME: 2025-02-26 01:54:41 -- STEP: 3136/36938 -- GLOBAL_STEP: 1037400 | > loss_text_ce: 0.03448820114135742 (0.03248261925360498) | > loss_mel_ce: 2.2088623046875 (nan) | > loss: 0.008902184665203094 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.1367312233362876) | > loader_time: 0.0047 (0.0046579668260350445)  --> TIME: 2025-02-26 01:54:50 -- STEP: 3186/36938 -- GLOBAL_STEP: 1037450 | > loss_text_ce: 0.039230819791555405 (0.03246807477279701) | > loss_mel_ce: 2.809288501739502 (nan) | > loss: 0.011303648352622986 (nan) | > current_lr: 5e-06 | > step_time: 0.1516 (0.13674364042072543) | > loader_time: 0.0046 (0.004658300698514405)  --> TIME: 2025-02-26 01:55:00 -- STEP: 3236/36938 -- GLOBAL_STEP: 1037500 | > loss_text_ce: 0.0356481559574604 (0.03245854313768531) | > loss_mel_ce: 2.6890928745269775 (nan) | > loss: 0.010812465101480484 (nan) | > current_lr: 5e-06 | > step_time: 0.1494 (0.13673696492894155) | > loader_time: 0.0048 (0.004660365902743906)  --> TIME: 2025-02-26 01:55:09 -- STEP: 3286/36938 -- GLOBAL_STEP: 1037550 | > loss_text_ce: 0.033605173230171204 (0.03246491470153677) | > loss_mel_ce: 2.948580503463745 (nan) | > loss: 0.011834070086479187 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.1366567873621014) | > loader_time: 0.0049 (0.0046631730480739654)  --> TIME: 2025-02-26 01:55:18 -- STEP: 3336/36938 -- GLOBAL_STEP: 1037600 | > loss_text_ce: 0.03182746469974518 (0.03246849743907331) | > loss_mel_ce: 2.7340168952941895 (nan) | > loss: 0.01097557321190834 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13661839257327185) | > loader_time: 0.0045 (0.004664515419829661)  --> TIME: 2025-02-26 01:55:27 -- STEP: 3386/36938 -- GLOBAL_STEP: 1037650 | > loss_text_ce: 0.028056854382157326 (0.03246955889949945) | > loss_mel_ce: 3.414585590362549 (nan) | > loss: 0.013661280274391174 (nan) | > current_lr: 5e-06 | > step_time: 0.188 (0.13663479180569763) | > loader_time: 0.0049 (0.0046645807079218286)  --> TIME: 2025-02-26 01:55:37 -- STEP: 3436/36938 -- GLOBAL_STEP: 1037700 | > loss_text_ce: 0.031162714585661888 (0.03245685842973479) | > loss_mel_ce: 2.789304256439209 (nan) | > loss: 0.011192330159246922 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.1366184545340443) | > loader_time: 0.0048 (0.00466561900306497)  --> TIME: 2025-02-26 01:55:46 -- STEP: 3486/36938 -- GLOBAL_STEP: 1037750 | > loss_text_ce: 0.04200558736920357 (0.032471190443678244) | > loss_mel_ce: 2.8885269165039062 (nan) | > loss: 0.01162909809499979 (nan) | > current_lr: 5e-06 | > step_time: 0.1848 (0.13662953419173773) | > loader_time: 0.0048 (0.0046655487480754465)  --> TIME: 2025-02-26 01:55:56 -- STEP: 3536/36938 -- GLOBAL_STEP: 1037800 | > loss_text_ce: 0.02411893755197525 (0.03246044210101406) | > loss_mel_ce: 2.2467572689056396 (nan) | > loss: 0.009011413902044296 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13669127111251525) | > loader_time: 0.0046 (0.004667242727668035)  --> TIME: 2025-02-26 01:56:05 -- STEP: 3586/36938 -- GLOBAL_STEP: 1037850 | > loss_text_ce: 0.03308704495429993 (0.03245729646955587) | > loss_mel_ce: 2.4996097087860107 (nan) | > loss: 0.01005038432776928 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.1366687826251291) | > loader_time: 0.0046 (0.004668122420281718)  --> TIME: 2025-02-26 01:56:14 -- STEP: 3636/36938 -- GLOBAL_STEP: 1037900 | > loss_text_ce: 0.03183348849415779 (0.03246515649253031) | > loss_mel_ce: 2.744548797607422 (nan) | > loss: 0.01101739052683115 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.1366620560159252) | > loader_time: 0.0046 (0.0046684799676943375)  --> TIME: 2025-02-26 01:56:24 -- STEP: 3686/36938 -- GLOBAL_STEP: 1037950 | > loss_text_ce: 0.032220181077718735 (0.03247210379524191) | > loss_mel_ce: 1.6689453125 (nan) | > loss: 0.006750656757503748 (nan) | > current_lr: 5e-06 | > step_time: 0.1496 (0.13668150521463102) | > loader_time: 0.0048 (0.00466848486995024)  --> TIME: 2025-02-26 01:56:33 -- STEP: 3736/36938 -- GLOBAL_STEP: 1038000 | > loss_text_ce: 0.03246208280324936 (0.032467376434282186) | > loss_mel_ce: 2.627509117126465 (nan) | > loss: 0.010555442422628403 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13676878465319578) | > loader_time: 0.0044 (0.004668650011944819)  --> TIME: 2025-02-26 01:56:43 -- STEP: 3786/36938 -- GLOBAL_STEP: 1038050 | > loss_text_ce: 0.02135545387864113 (0.032465956757820325) | > loss_mel_ce: 3.1025869846343994 (nan) | > loss: 0.012396597303450108 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.13677133533983568) | > loader_time: 0.0047 (0.00466820057648801)  --> TIME: 2025-02-26 01:56:52 -- STEP: 3836/36938 -- GLOBAL_STEP: 1038100 | > loss_text_ce: 0.0279046893119812 (0.03246486623848219) | > loss_mel_ce: 3.437434673309326 (nan) | > loss: 0.013751347549259663 (nan) | > current_lr: 5e-06 | > step_time: 0.188 (0.13678538078312064) | > loader_time: 0.0049 (0.004668729832324044)  --> TIME: 2025-02-26 01:57:01 -- STEP: 3886/36938 -- GLOBAL_STEP: 1038150 | > loss_text_ce: 0.024244682863354683 (0.03245502513005564) | > loss_mel_ce: 2.5078680515289307 (nan) | > loss: 0.010048067197203636 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13676401768771243) | > loader_time: 0.0049 (0.004668884711724483)  --> TIME: 2025-02-26 01:57:11 -- STEP: 3936/36938 -- GLOBAL_STEP: 1038200 | > loss_text_ce: 0.038496606051921844 (0.03246466860570766) | > loss_mel_ce: 3.1600191593170166 (nan) | > loss: 0.012692523188889027 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13673967195720188) | > loader_time: 0.0046 (0.00466833100086305)  --> TIME: 2025-02-26 01:57:20 -- STEP: 3986/36938 -- GLOBAL_STEP: 1038250 | > loss_text_ce: 0.03469134122133255 (0.03245221401040947) | > loss_mel_ce: 2.878434896469116 (nan) | > loss: 0.011560025624930859 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13674232598470773) | > loader_time: 0.0046 (0.0046677235915800796)  --> TIME: 2025-02-26 01:57:29 -- STEP: 4036/36938 -- GLOBAL_STEP: 1038300 | > loss_text_ce: 0.02334612049162388 (0.0324233342827427) | > loss_mel_ce: 2.7621350288391113 (nan) | > loss: 0.011053497903048992 (nan) | > current_lr: 5e-06 | > step_time: 0.1353 (0.13677217029367358) | > loader_time: 0.0047 (0.004668124485299417)  --> TIME: 2025-02-26 01:57:39 -- STEP: 4086/36938 -- GLOBAL_STEP: 1038350 | > loss_text_ce: 0.030602842569351196 (0.032417030627800617) | > loss_mel_ce: 2.6065189838409424 (nan) | > loss: 0.010464769788086414 (nan) | > current_lr: 5e-06 | > step_time: 0.1878 (0.1367295022344564) | > loader_time: 0.0048 (0.0046678522434878775)  --> TIME: 2025-02-26 01:57:48 -- STEP: 4136/36938 -- GLOBAL_STEP: 1038400 | > loss_text_ce: 0.028688674792647362 (0.03241111372432697) | > loss_mel_ce: 2.676745891571045 (nan) | > loss: 0.010735851712524891 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13672261277296538) | > loader_time: 0.0094 (0.004670853771600776)  --> TIME: 2025-02-26 01:57:57 -- STEP: 4186/36938 -- GLOBAL_STEP: 1038450 | > loss_text_ce: 0.03891965001821518 (0.03240000952246114) | > loss_mel_ce: 2.9446327686309814 (nan) | > loss: 0.011839494109153748 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.13672248792033168) | > loader_time: 0.0048 (0.0046728443313430925)  --> TIME: 2025-02-26 01:58:06 -- STEP: 4236/36938 -- GLOBAL_STEP: 1038500 | > loss_text_ce: 0.03626037389039993 (0.03240962757428237) | > loss_mel_ce: 2.528838634490967 (nan) | > loss: 0.010178964585065842 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13670826802510358) | > loader_time: 0.0044 (0.004672339887862164)  --> TIME: 2025-02-26 01:58:15 -- STEP: 4286/36938 -- GLOBAL_STEP: 1038550 | > loss_text_ce: 0.04714181646704674 (0.03242925873132301) | > loss_mel_ce: 2.470322608947754 (nan) | > loss: 0.009989938698709011 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1366728817365238) | > loader_time: 0.0045 (0.004672291230999636)  --> TIME: 2025-02-26 01:58:25 -- STEP: 4336/36938 -- GLOBAL_STEP: 1038600 | > loss_text_ce: 0.038381241261959076 (0.03243439178076165) | > loss_mel_ce: 0.0543212890625 (nan) | > loss: 0.00036786720738746226 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.1366651376565003) | > loader_time: 0.0049 (0.004671996644942082)  --> TIME: 2025-02-26 01:58:34 -- STEP: 4386/36938 -- GLOBAL_STEP: 1038650 | > loss_text_ce: 0.02961120568215847 (0.03243488265622742) | > loss_mel_ce: 0.014702987857162952 (nan) | > loss: 0.00017584998568054289 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13664249106188417) | > loader_time: 0.0046 (0.004671579292252136)  --> TIME: 2025-02-26 01:58:43 -- STEP: 4436/36938 -- GLOBAL_STEP: 1038700 | > loss_text_ce: 0.039806462824344635 (0.032436415743094624) | > loss_mel_ce: 2.8834705352783203 (nan) | > loss: 0.011600306257605553 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13660522491895363) | > loader_time: 0.0047 (0.0046709697455087126)  --> TIME: 2025-02-26 01:58:52 -- STEP: 4486/36938 -- GLOBAL_STEP: 1038750 | > loss_text_ce: 0.034186359494924545 (0.03243999636216774) | > loss_mel_ce: 2.85766863822937 (nan) | > loss: 0.01147561613470316 (nan) | > current_lr: 5e-06 | > step_time: 0.1858 (0.1365765716367356) | > loader_time: 0.0049 (0.004670468441779941)  --> TIME: 2025-02-26 01:59:01 -- STEP: 4536/36938 -- GLOBAL_STEP: 1038800 | > loss_text_ce: 0.03539813309907913 (0.03242783103593206) | > loss_mel_ce: 3.5295212268829346 (nan) | > loss: 0.014146506786346436 (nan) | > current_lr: 5e-06 | > step_time: 0.1268 (0.13655354757998564) | > loader_time: 0.0046 (0.004670014930149861)  --> TIME: 2025-02-26 01:59:11 -- STEP: 4586/36938 -- GLOBAL_STEP: 1038850 | > loss_text_ce: 0.0319785512983799 (0.032431616168428684) | > loss_mel_ce: 2.717797040939331 (nan) | > loss: 0.010911808349192142 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13654632127498154) | > loader_time: 0.0045 (0.004669925452214472)  --> TIME: 2025-02-26 01:59:20 -- STEP: 4636/36938 -- GLOBAL_STEP: 1038900 | > loss_text_ce: 0.041168369352817535 (0.03242890633051264) | > loss_mel_ce: 2.746229410171509 (nan) | > loss: 0.011061103083193302 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13654050902110731) | > loader_time: 0.0047 (0.004669471893771334)  --> TIME: 2025-02-26 01:59:30 -- STEP: 4686/36938 -- GLOBAL_STEP: 1038950 | > loss_text_ce: 0.036146655678749084 (0.03243119317586684) | > loss_mel_ce: 3.0481345653533936 (nan) | > loss: 0.012239212170243263 (nan) | > current_lr: 5e-06 | > step_time: 0.1986 (0.13662335132870762) | > loader_time: 0.0048 (0.004669648228269629)  --> TIME: 2025-02-26 01:59:39 -- STEP: 4736/36938 -- GLOBAL_STEP: 1039000 | > loss_text_ce: 0.026716982945799828 (0.03242676205126063) | > loss_mel_ce: 2.4851198196411133 (nan) | > loss: 0.009967606514692307 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1365809376678756) | > loader_time: 0.0047 (0.004669273070789666)  --> TIME: 2025-02-26 01:59:48 -- STEP: 4786/36938 -- GLOBAL_STEP: 1039050 | > loss_text_ce: 0.024400612339377403 (0.03241994905348207) | > loss_mel_ce: 1.0136346817016602 (nan) | > loss: 0.00411918805912137 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.13658494232989118) | > loader_time: 0.0049 (0.00466907960921214)  --> TIME: 2025-02-26 01:59:58 -- STEP: 4836/36938 -- GLOBAL_STEP: 1039100 | > loss_text_ce: 0.04010023921728134 (0.03241951582045128) | > loss_mel_ce: 2.1905689239501953 (nan) | > loss: 0.008851862512528896 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13658289203375581) | > loader_time: 0.0045 (0.004668884527791713)  --> TIME: 2025-02-26 02:00:07 -- STEP: 4886/36938 -- GLOBAL_STEP: 1039150 | > loss_text_ce: 0.0320596769452095 (0.03243221639363142) | > loss_mel_ce: 3.525369167327881 (nan) | > loss: 0.014116781763732433 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13661293528678206) | > loader_time: 0.0046 (0.004668681142372126)  --> TIME: 2025-02-26 02:00:16 -- STEP: 4936/36938 -- GLOBAL_STEP: 1039200 | > loss_text_ce: 0.043057650327682495 (0.03242979806752029) | > loss_mel_ce: 2.2554874420166016 (nan) | > loss: 0.009121211245656013 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13660227180106893) | > loader_time: 0.0046 (0.0046684223210599)  --> TIME: 2025-02-26 02:00:26 -- STEP: 4986/36938 -- GLOBAL_STEP: 1039250 | > loss_text_ce: 0.03243353217840195 (0.03244028600325799) | > loss_mel_ce: 2.7899296283721924 (nan) | > loss: 0.01119985431432724 (nan) | > current_lr: 5e-06 | > step_time: 0.163 (0.13658667619669237) | > loader_time: 0.0047 (0.004668702908992946)  --> TIME: 2025-02-26 02:00:35 -- STEP: 5036/36938 -- GLOBAL_STEP: 1039300 | > loss_text_ce: 0.030612925067543983 (0.032448051135426426) | > loss_mel_ce: 2.565610647201538 (nan) | > loss: 0.010302474722266197 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13655876568899536) | > loader_time: 0.0046 (0.004668272090772476)  --> TIME: 2025-02-26 02:00:44 -- STEP: 5086/36938 -- GLOBAL_STEP: 1039350 | > loss_text_ce: 0.026139792054891586 (0.03243498954240047) | > loss_mel_ce: 2.519242286682129 (nan) | > loss: 0.01010072324424982 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13653372222871887) | > loader_time: 0.0046 (0.004667911902685299)  --> TIME: 2025-02-26 02:00:53 -- STEP: 5136/36938 -- GLOBAL_STEP: 1039400 | > loss_text_ce: 0.035792410373687744 (0.03245210961849985) | > loss_mel_ce: 2.6111111640930176 (nan) | > loss: 0.010503586381673813 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.1365431593121769) | > loader_time: 0.0045 (0.004667319612711006)  --> TIME: 2025-02-26 02:01:03 -- STEP: 5186/36938 -- GLOBAL_STEP: 1039450 | > loss_text_ce: 0.03894763067364693 (0.03246278212000966) | > loss_mel_ce: 2.6937320232391357 (nan) | > loss: 0.010843967087566853 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13653269795161427) | > loader_time: 0.0045 (0.0046667581904683755)  --> TIME: 2025-02-26 02:01:12 -- STEP: 5236/36938 -- GLOBAL_STEP: 1039500 | > loss_text_ce: 0.03789150342345238 (0.03247155932153918) | > loss_mel_ce: 2.2883498668670654 (nan) | > loss: 0.009231116622686386 (nan) | > current_lr: 5e-06 | > step_time: 0.1492 (0.13654492710090202) | > loader_time: 0.0048 (0.004666236313906572)  --> TIME: 2025-02-26 02:01:21 -- STEP: 5286/36938 -- GLOBAL_STEP: 1039550 | > loss_text_ce: 0.024232031777501106 (0.032464392236749365) | > loss_mel_ce: 0.09615478664636612 (nan) | > loss: 0.0004777254944201559 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.1365470100973664) | > loader_time: 0.0046 (0.004666059746960544)  --> TIME: 2025-02-26 02:01:30 -- STEP: 5336/36938 -- GLOBAL_STEP: 1039600 | > loss_text_ce: 0.026416625827550888 (0.03247394240122699) | > loss_mel_ce: 2.7752020359039307 (nan) | > loss: 0.011117534711956978 (nan) | > current_lr: 5e-06 | > step_time: 0.1667 (0.13651045535815332) | > loader_time: 0.0047 (0.004665531720714766)  --> TIME: 2025-02-26 02:01:40 -- STEP: 5386/36938 -- GLOBAL_STEP: 1039650 | > loss_text_ce: 0.02764073759317398 (0.0324686641135346) | > loss_mel_ce: 3.4012203216552734 (nan) | > loss: 0.013606593012809753 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.1365566335112198) | > loader_time: 0.0047 (0.00466671106853407)  --> TIME: 2025-02-26 02:01:50 -- STEP: 5436/36938 -- GLOBAL_STEP: 1039700 | > loss_text_ce: 0.035991229116916656 (0.0324593096359197) | > loss_mel_ce: 2.943502187728882 (nan) | > loss: 0.011823386885225773 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13660490951439835) | > loader_time: 0.0046 (0.0046670029407217695)  --> TIME: 2025-02-26 02:01:59 -- STEP: 5486/36938 -- GLOBAL_STEP: 1039750 | > loss_text_ce: 0.029236216098070145 (0.032462205669414594) | > loss_mel_ce: 0.051706213504076004 (nan) | > loss: 0.0003212001465726644 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13655946722353984) | > loader_time: 0.0046 (0.004666805397712353)  --> TIME: 2025-02-26 02:02:08 -- STEP: 5536/36938 -- GLOBAL_STEP: 1039800 | > loss_text_ce: 0.03490572050213814 (0.03244538632214126) | > loss_mel_ce: 1.5525634288787842 (nan) | > loss: 0.0062994807958602905 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13655254033776368) | > loader_time: 0.0046 (0.004667153721944444)  --> TIME: 2025-02-26 02:02:17 -- STEP: 5586/36938 -- GLOBAL_STEP: 1039850 | > loss_text_ce: 0.03167225047945976 (0.03243774569697082) | > loss_mel_ce: 2.613313913345337 (nan) | > loss: 0.010495977476239204 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.136548904932147) | > loader_time: 0.0045 (0.004667164858549324)  --> TIME: 2025-02-26 02:02:27 -- STEP: 5636/36938 -- GLOBAL_STEP: 1039900 | > loss_text_ce: 0.028823241591453552 (0.032433499388940135) | > loss_mel_ce: 1.6951215267181396 (nan) | > loss: 0.006841050926595926 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13655900566014426) | > loader_time: 0.0046 (0.0046669385634558296)  --> TIME: 2025-02-26 02:02:36 -- STEP: 5686/36938 -- GLOBAL_STEP: 1039950 | > loss_text_ce: 0.030454350635409355 (0.032433208359049705) | > loss_mel_ce: 3.0822525024414062 (nan) | > loss: 0.01235201209783554 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13654795235577827) | > loader_time: 0.0047 (0.004666417784729428)  --> TIME: 2025-02-26 02:02:45 -- STEP: 5736/36938 -- GLOBAL_STEP: 1040000 | > loss_text_ce: 0.02630523592233658 (0.03242898264294561) | > loss_mel_ce: 0.9080062508583069 (nan) | > loss: 0.00370758562348783 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13656595806530158) | > loader_time: 0.0045 (0.004666397137621952) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1040000.pth  --> TIME: 2025-02-26 02:03:00 -- STEP: 5786/36938 -- GLOBAL_STEP: 1040050 | > loss_text_ce: 0.03447301685810089 (0.03242510744952241) | > loss_mel_ce: 2.8381781578063965 (nan) | > loss: 0.011399409733712673 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.1365292687503433) | > loader_time: 0.0046 (0.004666528279831609)  --> TIME: 2025-02-26 02:03:09 -- STEP: 5836/36938 -- GLOBAL_STEP: 1040100 | > loss_text_ce: 0.0316065289080143 (0.03242591432933434) | > loss_mel_ce: 3.023603677749634 (nan) | > loss: 0.012123850174248219 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.1364895127831133) | > loader_time: 0.0046 (0.004666650883541926)  --> TIME: 2025-02-26 02:03:19 -- STEP: 5886/36938 -- GLOBAL_STEP: 1040150 | > loss_text_ce: 0.032567016780376434 (0.03242600223127373) | > loss_mel_ce: 2.4594860076904297 (nan) | > loss: 0.00988909974694252 (nan) | > current_lr: 5e-06 | > step_time: 0.1486 (0.1365019002416043) | > loader_time: 0.0047 (0.004666878664286975)  --> TIME: 2025-02-26 02:03:28 -- STEP: 5936/36938 -- GLOBAL_STEP: 1040200 | > loss_text_ce: 0.037317872047424316 (0.03242657850954606) | > loss_mel_ce: 2.8089187145233154 (nan) | > loss: 0.011294590309262276 (nan) | > current_lr: 5e-06 | > step_time: 0.1566 (0.13652809811730932) | > loader_time: 0.0047 (0.004667175466802239)  --> TIME: 2025-02-26 02:03:38 -- STEP: 5986/36938 -- GLOBAL_STEP: 1040250 | > loss_text_ce: 0.030799534171819687 (0.03241971779099397) | > loss_mel_ce: 2.9267325401306152 (nan) | > loss: 0.011736239306628704 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13658605576198804) | > loader_time: 0.0047 (0.004667250519805714)  --> TIME: 2025-02-26 02:03:47 -- STEP: 6036/36938 -- GLOBAL_STEP: 1040300 | > loss_text_ce: 0.02594493329524994 (0.03241797934012891) | > loss_mel_ce: 2.560750722885132 (nan) | > loss: 0.010264665819704533 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13656196268762963) | > loader_time: 0.0044 (0.004667211795500999)  --> TIME: 2025-02-26 02:03:57 -- STEP: 6086/36938 -- GLOBAL_STEP: 1040350 | > loss_text_ce: 0.03356114402413368 (0.03242103050417304) | > loss_mel_ce: 2.7465460300445557 (nan) | > loss: 0.011032172478735447 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13660110296718878) | > loader_time: 0.0048 (0.004669603571232079)  --> TIME: 2025-02-26 02:04:06 -- STEP: 6136/36938 -- GLOBAL_STEP: 1040400 | > loss_text_ce: 0.03046809695661068 (0.0324049376188371) | > loss_mel_ce: 2.306201219558716 (nan) | > loss: 0.009272497147321701 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13665691810760855) | > loader_time: 0.0048 (0.004670344555393502)  --> TIME: 2025-02-26 02:04:16 -- STEP: 6186/36938 -- GLOBAL_STEP: 1040450 | > loss_text_ce: 0.03364858403801918 (0.03238092073336645) | > loss_mel_ce: 2.7902705669403076 (nan) | > loss: 0.01120602898299694 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13664249983798962) | > loader_time: 0.0049 (0.004671560457675463)  --> TIME: 2025-02-26 02:04:25 -- STEP: 6236/36938 -- GLOBAL_STEP: 1040500 | > loss_text_ce: 0.03910836577415466 (0.03237204884571421) | > loss_mel_ce: 3.11598539352417 (nan) | > loss: 0.01252021361142397 (nan) | > current_lr: 5e-06 | > step_time: 0.1266 (0.13662603717342744) | > loader_time: 0.0047 (0.004674159654674815)  --> TIME: 2025-02-26 02:04:34 -- STEP: 6286/36938 -- GLOBAL_STEP: 1040550 | > loss_text_ce: 0.0321829579770565 (0.03237603760418946) | > loss_mel_ce: 2.1061925888061523 (nan) | > loss: 0.00848561804741621 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13662440106249027) | > loader_time: 0.005 (0.0046760331966146495)  --> TIME: 2025-02-26 02:04:44 -- STEP: 6336/36938 -- GLOBAL_STEP: 1040600 | > loss_text_ce: 0.031686931848526 (0.032369171720788334) | > loss_mel_ce: 2.762932777404785 (nan) | > loss: 0.01108976174145937 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13665634024925938) | > loader_time: 0.0046 (0.004676731585553201)  --> TIME: 2025-02-26 02:04:53 -- STEP: 6386/36938 -- GLOBAL_STEP: 1040650 | > loss_text_ce: 0.014963258989155293 (0.03237063977870282) | > loss_mel_ce: 2.5042078495025635 (nan) | > loss: 0.009996711276471615 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13665083747951007) | > loader_time: 0.0049 (0.004677156949543564)  --> TIME: 2025-02-26 02:05:02 -- STEP: 6436/36938 -- GLOBAL_STEP: 1040700 | > loss_text_ce: 0.0299751628190279 (0.03238010711460317) | > loss_mel_ce: 2.942627429962158 (nan) | > loss: 0.011796042323112488 (nan) | > current_lr: 5e-06 | > step_time: 0.1353 (0.1366500877459797) | > loader_time: 0.0047 (0.0046779487798345905)  --> TIME: 2025-02-26 02:05:11 -- STEP: 6486/36938 -- GLOBAL_STEP: 1040750 | > loss_text_ce: 0.02836916409432888 (0.0323698015690006) | > loss_mel_ce: 2.433868408203125 (nan) | > loss: 0.00977078452706337 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13666509751217862) | > loader_time: 0.0048 (0.004679483136680813)  --> TIME: 2025-02-26 02:05:21 -- STEP: 6536/36938 -- GLOBAL_STEP: 1040800 | > loss_text_ce: 0.03175271302461624 (0.03236614970172293) | > loss_mel_ce: 2.375325918197632 (nan) | > loss: 0.009551900438964367 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.13671367091093709) | > loader_time: 0.0047 (0.0046819363842687586)  --> TIME: 2025-02-26 02:05:31 -- STEP: 6586/36938 -- GLOBAL_STEP: 1040850 | > loss_text_ce: 0.03147716075181961 (0.03236390481740591) | > loss_mel_ce: 2.3959107398986816 (nan) | > loss: 0.009632492437958717 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13672955590759864) | > loader_time: 0.0048 (0.004681931380403113)  --> TIME: 2025-02-26 02:05:40 -- STEP: 6636/36938 -- GLOBAL_STEP: 1040900 | > loss_text_ce: 0.0319208949804306 (0.03235800439200508) | > loss_mel_ce: 3.2082831859588623 (nan) | > loss: 0.0128579530864954 (nan) | > current_lr: 5e-06 | > step_time: 0.1984 (0.13672078906088606) | > loader_time: 0.0049 (0.004681994320039951)  --> TIME: 2025-02-26 02:05:49 -- STEP: 6686/36938 -- GLOBAL_STEP: 1040950 | > loss_text_ce: 0.02506241574883461 (0.03234794498007542) | > loss_mel_ce: 2.5887246131896973 (nan) | > loss: 0.010372171178460121 (nan) | > current_lr: 5e-06 | > step_time: 0.1496 (0.13673222303889743) | > loader_time: 0.0047 (0.004682525987936083)  --> TIME: 2025-02-26 02:05:59 -- STEP: 6736/36938 -- GLOBAL_STEP: 1041000 | > loss_text_ce: 0.033148229122161865 (0.03235455019515105) | > loss_mel_ce: 0.94000244140625 (nan) | > loss: 0.003861709265038371 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13675181567668923) | > loader_time: 0.0049 (0.004683735924208822)  --> TIME: 2025-02-26 02:06:08 -- STEP: 6786/36938 -- GLOBAL_STEP: 1041050 | > loss_text_ce: 0.03195199370384216 (0.03235510840390016) | > loss_mel_ce: 2.1730384826660156 (nan) | > loss: 0.008749962784349918 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13675997926884623) | > loader_time: 0.0051 (0.0046842075750394285)  --> TIME: 2025-02-26 02:06:18 -- STEP: 6836/36938 -- GLOBAL_STEP: 1041100 | > loss_text_ce: 0.03163653984665871 (0.03236050825473947) | > loss_mel_ce: 0.0707600936293602 (nan) | > loss: 0.000406335893785581 (nan) | > current_lr: 5e-06 | > step_time: 0.156 (0.1367825659971755) | > loader_time: 0.0048 (0.004684385882266583)  --> TIME: 2025-02-26 02:06:27 -- STEP: 6886/36938 -- GLOBAL_STEP: 1041150 | > loss_text_ce: 0.03477974608540535 (0.032364860260488446) | > loss_mel_ce: 2.978724718093872 (nan) | > loss: 0.011958352290093899 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13682575171124675) | > loader_time: 0.0048 (0.004684700475610213)  --> TIME: 2025-02-26 02:06:37 -- STEP: 6936/36938 -- GLOBAL_STEP: 1041200 | > loss_text_ce: 0.027729684486985207 (0.03236590211256588) | > loss_mel_ce: 2.278231382369995 (nan) | > loss: 0.009150640107691288 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13681454318892433) | > loader_time: 0.0047 (0.0046861166222560285)  --> TIME: 2025-02-26 02:06:46 -- STEP: 6986/36938 -- GLOBAL_STEP: 1041250 | > loss_text_ce: 0.037491440773010254 (0.032366016711437094) | > loss_mel_ce: 2.8454177379608154 (nan) | > loss: 0.011440116912126541 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.13684670276024785) | > loader_time: 0.0047 (0.004686303340770991)  --> TIME: 2025-02-26 02:06:56 -- STEP: 7036/36938 -- GLOBAL_STEP: 1041300 | > loss_text_ce: 0.024359699338674545 (0.032359282509701714) | > loss_mel_ce: 3.0334272384643555 (nan) | > loss: 0.012134076096117496 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.136868977695788) | > loader_time: 0.0047 (0.004686415026037451)  --> TIME: 2025-02-26 02:07:05 -- STEP: 7086/36938 -- GLOBAL_STEP: 1041350 | > loss_text_ce: 0.02132837474346161 (0.032350302621960855) | > loss_mel_ce: 2.3666956424713135 (nan) | > loss: 0.009476286359131336 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1368408682586578) | > loader_time: 0.0046 (0.004686242135069721)  --> TIME: 2025-02-26 02:07:14 -- STEP: 7136/36938 -- GLOBAL_STEP: 1041400 | > loss_text_ce: 0.032128944993019104 (0.03234422301707556) | > loss_mel_ce: 3.0769426822662354 (nan) | > loss: 0.012337586842477322 (nan) | > current_lr: 5e-06 | > step_time: 0.1679 (0.13687673351422544) | > loader_time: 0.0046 (0.004687140006655541)  --> TIME: 2025-02-26 02:07:24 -- STEP: 7186/36938 -- GLOBAL_STEP: 1041450 | > loss_text_ce: 0.028804022818803787 (0.03234539982040314) | > loss_mel_ce: 3.5303194522857666 (nan) | > loss: 0.014123506844043732 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.1368933667587165) | > loader_time: 0.005 (0.0046870966592008935)  --> TIME: 2025-02-26 02:07:33 -- STEP: 7236/36938 -- GLOBAL_STEP: 1041500 | > loss_text_ce: 0.039406511932611465 (0.03235893344855161) | > loss_mel_ce: 3.110889434814453 (nan) | > loss: 0.01250117551535368 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.1368629842782428) | > loader_time: 0.0048 (0.004687034207328773)  --> TIME: 2025-02-26 02:07:42 -- STEP: 7286/36938 -- GLOBAL_STEP: 1041550 | > loss_text_ce: 0.03584524244070053 (0.03236311270182761) | > loss_mel_ce: 3.045595407485962 (nan) | > loss: 0.012227940373122692 (nan) | > current_lr: 5e-06 | > step_time: 0.1861 (0.13686657033711377) | > loader_time: 0.0048 (0.0046872125364351295)  --> TIME: 2025-02-26 02:07:52 -- STEP: 7336/36938 -- GLOBAL_STEP: 1041600 | > loss_text_ce: 0.02986789122223854 (0.03236052414428773) | > loss_mel_ce: 2.932086229324341 (nan) | > loss: 0.011753787286579609 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13689153951022975) | > loader_time: 0.0047 (0.00468756923108741)  --> TIME: 2025-02-26 02:08:01 -- STEP: 7386/36938 -- GLOBAL_STEP: 1041650 | > loss_text_ce: 0.03737425059080124 (0.032361499451740404) | > loss_mel_ce: 1.9720909595489502 (nan) | > loss: 0.007974068634212017 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13688672392664644) | > loader_time: 0.0046 (0.004687373824730307)  --> TIME: 2025-02-26 02:08:11 -- STEP: 7436/36938 -- GLOBAL_STEP: 1041700 | > loss_text_ce: 0.03284206986427307 (0.03236267903021622) | > loss_mel_ce: 2.6604433059692383 (nan) | > loss: 0.01068764179944992 (nan) | > current_lr: 5e-06 | > step_time: 0.1578 (0.13691519107018318) | > loader_time: 0.0046 (0.004687387017295724)  --> TIME: 2025-02-26 02:08:20 -- STEP: 7486/36938 -- GLOBAL_STEP: 1041750 | > loss_text_ce: 0.03245396167039871 (0.032370051839459846) | > loss_mel_ce: 2.7655670642852783 (nan) | > loss: 0.011103258468210697 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13692946132096326) | > loader_time: 0.0048 (0.00468714686915291)  --> TIME: 2025-02-26 02:08:29 -- STEP: 7536/36938 -- GLOBAL_STEP: 1041800 | > loss_text_ce: 0.0311758890748024 (0.032367136791489864) | > loss_mel_ce: 3.4017722606658936 (nan) | > loss: 0.013622811064124107 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13689630774429148) | > loader_time: 0.0047 (0.00468685644067777)  --> TIME: 2025-02-26 02:08:38 -- STEP: 7586/36938 -- GLOBAL_STEP: 1041850 | > loss_text_ce: 0.029631564393639565 (0.032365992626373206) | > loss_mel_ce: 2.623518943786621 (nan) | > loss: 0.010528375394642353 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13689204208963382) | > loader_time: 0.0046 (0.004686693041424565)  --> TIME: 2025-02-26 02:08:48 -- STEP: 7636/36938 -- GLOBAL_STEP: 1041900 | > loss_text_ce: 0.03689955919981003 (0.03236855926480481) | > loss_mel_ce: 2.9892983436584473 (nan) | > loss: 0.012008722871541977 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13694046881887043) | > loader_time: 0.0047 (0.004686792930924754)  --> TIME: 2025-02-26 02:08:57 -- STEP: 7686/36938 -- GLOBAL_STEP: 1041950 | > loss_text_ce: 0.022743184119462967 (0.03235957069121812) | > loss_mel_ce: 1.7219650745391846 (nan) | > loss: 0.006923445966094732 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1369377942312332) | > loader_time: 0.0045 (0.004686709558346486)  --> TIME: 2025-02-26 02:09:07 -- STEP: 7736/36938 -- GLOBAL_STEP: 1042000 | > loss_text_ce: 0.03247741609811783 (0.032359043411085205) | > loss_mel_ce: 3.1104631423950195 (nan) | > loss: 0.012471986934542656 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.1369519010571438) | > loader_time: 0.0058 (0.0046866227022248)  --> TIME: 2025-02-26 02:09:16 -- STEP: 7786/36938 -- GLOBAL_STEP: 1042050 | > loss_text_ce: 0.03147497773170471 (0.03236636796400659) | > loss_mel_ce: 0.0378570556640625 (nan) | > loss: 0.0002751271240413189 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.13696626455717095) | > loader_time: 0.0046 (0.004686683944586528)  --> TIME: 2025-02-26 02:09:26 -- STEP: 7836/36938 -- GLOBAL_STEP: 1042100 | > loss_text_ce: 0.026301030069589615 (0.03236178401405984) | > loss_mel_ce: 1.1018880605697632 (nan) | > loss: 0.00447694119066 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13695955605090673) | > loader_time: 0.0047 (0.0046865423459554025)  --> TIME: 2025-02-26 02:09:35 -- STEP: 7886/36938 -- GLOBAL_STEP: 1042150 | > loss_text_ce: 0.031008346006274223 (0.03237119615087426) | > loss_mel_ce: 3.4251391887664795 (nan) | > loss: 0.013714871369302273 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.13699040977053317) | > loader_time: 0.0048 (0.004686385763498728)  --> TIME: 2025-02-26 02:09:45 -- STEP: 7936/36938 -- GLOBAL_STEP: 1042200 | > loss_text_ce: 0.02482200600206852 (0.03237287914887407) | > loss_mel_ce: 2.6960983276367188 (nan) | > loss: 0.010797303169965744 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13700865704806586) | > loader_time: 0.0047 (0.004686439980662644)  --> TIME: 2025-02-26 02:09:54 -- STEP: 7986/36938 -- GLOBAL_STEP: 1042250 | > loss_text_ce: 0.03284363076090813 (0.0323735781540153) | > loss_mel_ce: 3.1596972942352295 (nan) | > loss: 0.012668813578784466 (nan) | > current_lr: 5e-06 | > step_time: 0.1824 (0.13701467832884875) | > loader_time: 0.0049 (0.0046861007223743135)  --> TIME: 2025-02-26 02:10:03 -- STEP: 8036/36938 -- GLOBAL_STEP: 1042300 | > loss_text_ce: 0.03716022148728371 (0.03237600474571561) | > loss_mel_ce: 3.1872925758361816 (nan) | > loss: 0.012795448303222656 (nan) | > current_lr: 5e-06 | > step_time: 0.1783 (0.13699639570422295) | > loader_time: 0.0048 (0.004685827663966593)  --> TIME: 2025-02-26 02:10:12 -- STEP: 8086/36938 -- GLOBAL_STEP: 1042350 | > loss_text_ce: 0.04709990695118904 (0.032383062780242555) | > loss_mel_ce: 2.9276435375213623 (nan) | > loss: 0.01180453784763813 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.1369763882243336) | > loader_time: 0.0046 (0.004685742678749605)  --> TIME: 2025-02-26 02:10:22 -- STEP: 8136/36938 -- GLOBAL_STEP: 1042400 | > loss_text_ce: 0.03201696649193764 (0.03237728299579938) | > loss_mel_ce: 2.2340617179870605 (nan) | > loss: 0.008992376737296581 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13697254766170727) | > loader_time: 0.0054 (0.004685511191685999)  --> TIME: 2025-02-26 02:10:31 -- STEP: 8186/36938 -- GLOBAL_STEP: 1042450 | > loss_text_ce: 0.03791651874780655 (0.03238814898190679) | > loss_mel_ce: 3.3086185455322266 (nan) | > loss: 0.013279901817440987 (nan) | > current_lr: 5e-06 | > step_time: 0.174 (0.13697458621257394) | > loader_time: 0.0048 (0.004685261154594212)  --> TIME: 2025-02-26 02:10:40 -- STEP: 8236/36938 -- GLOBAL_STEP: 1042500 | > loss_text_ce: 0.038609687238931656 (0.03238646086789818) | > loss_mel_ce: 2.3432211875915527 (nan) | > loss: 0.009451710619032383 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13697485572913143) | > loader_time: 0.0046 (0.004684985031370638)  --> TIME: 2025-02-26 02:10:50 -- STEP: 8286/36938 -- GLOBAL_STEP: 1042550 | > loss_text_ce: 0.03367181122303009 (0.03239141927174232) | > loss_mel_ce: 2.818051815032959 (nan) | > loss: 0.011316364631056786 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13698995078844411) | > loader_time: 0.0048 (0.004684907124579434)  --> TIME: 2025-02-26 02:11:00 -- STEP: 8336/36938 -- GLOBAL_STEP: 1042600 | > loss_text_ce: 0.03175422549247742 (0.03238621092302688) | > loss_mel_ce: 0.019378662109375 (nan) | > loss: 0.00020290829706937075 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13701207644079103) | > loader_time: 0.0046 (0.004684823145106754)  --> TIME: 2025-02-26 02:11:09 -- STEP: 8386/36938 -- GLOBAL_STEP: 1042650 | > loss_text_ce: 0.024409718811511993 (0.032384281042579524) | > loss_mel_ce: 2.9148447513580322 (nan) | > loss: 0.011663708835840225 (nan) | > current_lr: 5e-06 | > step_time: 0.1618 (0.1369950603875857) | > loader_time: 0.0049 (0.0046844820461095825)  --> TIME: 2025-02-26 02:11:18 -- STEP: 8436/36938 -- GLOBAL_STEP: 1042700 | > loss_text_ce: 0.03321808576583862 (0.03238355515835192) | > loss_mel_ce: 3.20589017868042 (nan) | > loss: 0.01285360474139452 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13703263365527135) | > loader_time: 0.0045 (0.004684247214288838)  --> TIME: 2025-02-26 02:11:28 -- STEP: 8486/36938 -- GLOBAL_STEP: 1042750 | > loss_text_ce: 0.025950143113732338 (0.0323773410436642) | > loss_mel_ce: 2.3710076808929443 (nan) | > loss: 0.009511738084256649 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.1370252724275986) | > loader_time: 0.0047 (0.004683793419912801)  --> TIME: 2025-02-26 02:11:37 -- STEP: 8536/36938 -- GLOBAL_STEP: 1042800 | > loss_text_ce: 0.023724855855107307 (0.032383264941355454) | > loss_mel_ce: 2.2683496475219727 (nan) | > loss: 0.00909553375095129 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13700946137630277) | > loader_time: 0.0045 (0.004683225732488732)  --> TIME: 2025-02-26 02:11:46 -- STEP: 8586/36938 -- GLOBAL_STEP: 1042850 | > loss_text_ce: 0.03478972241282463 (0.03238200168913914) | > loss_mel_ce: 2.4341752529144287 (nan) | > loss: 0.009797480888664722 (nan) | > current_lr: 5e-06 | > step_time: 0.1812 (0.1370087320533022) | > loader_time: 0.0048 (0.004683066686216615)  --> TIME: 2025-02-26 02:11:55 -- STEP: 8636/36938 -- GLOBAL_STEP: 1042900 | > loss_text_ce: 0.027926156297326088 (0.03238213293833709) | > loss_mel_ce: 2.618217706680298 (nan) | > loss: 0.010500571690499783 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13699640861646398) | > loader_time: 0.0047 (0.004682996472919249)  --> TIME: 2025-02-26 02:12:05 -- STEP: 8686/36938 -- GLOBAL_STEP: 1042950 | > loss_text_ce: 0.03400562331080437 (0.03237966133942936) | > loss_mel_ce: 2.4822330474853516 (nan) | > loss: 0.009985074400901794 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.13700267111901016) | > loader_time: 0.0047 (0.004682729657608165)  --> TIME: 2025-02-26 02:12:14 -- STEP: 8736/36938 -- GLOBAL_STEP: 1043000 | > loss_text_ce: 0.029010135680437088 (0.03236886595459849) | > loss_mel_ce: 2.314453125 (nan) | > loss: 0.009299458004534245 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.13701077064860878) | > loader_time: 0.0046 (0.0046823310491802975)  --> TIME: 2025-02-26 02:12:24 -- STEP: 8786/36938 -- GLOBAL_STEP: 1043050 | > loss_text_ce: 0.032167404890060425 (0.03236352791299513) | > loss_mel_ce: 2.8523123264312744 (nan) | > loss: 0.011446349322795868 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13701517140162503) | > loader_time: 0.0045 (0.0046819280498043596)  --> TIME: 2025-02-26 02:12:33 -- STEP: 8836/36938 -- GLOBAL_STEP: 1043100 | > loss_text_ce: 0.027905208989977837 (0.03236226190626544) | > loss_mel_ce: 2.4879720211029053 (nan) | > loss: 0.009983640164136887 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13700890794713452) | > loader_time: 0.0048 (0.004681527155888685)  --> TIME: 2025-02-26 02:12:42 -- STEP: 8886/36938 -- GLOBAL_STEP: 1043150 | > loss_text_ce: 0.04363380745053291 (0.032360126516561095) | > loss_mel_ce: 3.519343376159668 (nan) | > loss: 0.014138799160718918 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.13702320668870685) | > loader_time: 0.0048 (0.004681644583510416)  --> TIME: 2025-02-26 02:12:52 -- STEP: 8936/36938 -- GLOBAL_STEP: 1043200 | > loss_text_ce: 0.028439151123166084 (0.03236266014840704) | > loss_mel_ce: 1.9821550846099854 (nan) | > loss: 0.007978548295795918 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1370521173107912) | > loader_time: 0.0046 (0.004681714326009867)  --> TIME: 2025-02-26 02:13:02 -- STEP: 8986/36938 -- GLOBAL_STEP: 1043250 | > loss_text_ce: 0.03334895893931389 (0.03236158837865064) | > loss_mel_ce: 2.019800901412964 (nan) | > loss: 0.008147421292960644 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13708571185254848) | > loader_time: 0.0046 (0.00468177517352433)  --> TIME: 2025-02-26 02:13:11 -- STEP: 9036/36938 -- GLOBAL_STEP: 1043300 | > loss_text_ce: 0.036626946181058884 (0.03236596188815124) | > loss_mel_ce: 3.0715103149414062 (nan) | > loss: 0.012333879247307777 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1370807926757191) | > loader_time: 0.0045 (0.004681614369826814)  --> TIME: 2025-02-26 02:13:21 -- STEP: 9086/36938 -- GLOBAL_STEP: 1043350 | > loss_text_ce: 0.04362773150205612 (0.03236270886617318) | > loss_mel_ce: 3.174534559249878 (nan) | > loss: 0.012770486064255238 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13710645424382897) | > loader_time: 0.0047 (0.004681585723546127)  --> TIME: 2025-02-26 02:13:30 -- STEP: 9136/36938 -- GLOBAL_STEP: 1043400 | > loss_text_ce: 0.037331026047468185 (0.03235858332772701) | > loss_mel_ce: 2.5036113262176514 (nan) | > loss: 0.01008310541510582 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.1370935397526011) | > loader_time: 0.0045 (0.004681297285961825)  --> TIME: 2025-02-26 02:13:39 -- STEP: 9186/36938 -- GLOBAL_STEP: 1043450 | > loss_text_ce: 0.04301564022898674 (0.03235418854320649) | > loss_mel_ce: 1.8465533256530762 (nan) | > loss: 0.0074982899241149426 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.13708394878756205) | > loader_time: 0.0049 (0.004681039214679268)  --> TIME: 2025-02-26 02:13:49 -- STEP: 9236/36938 -- GLOBAL_STEP: 1043500 | > loss_text_ce: 0.025933582335710526 (0.03236023888141955) | > loss_mel_ce: 2.595728635787964 (nan) | > loss: 0.010403421707451344 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13708465968751954) | > loader_time: 0.0046 (0.00468085022806967)  --> TIME: 2025-02-26 02:13:58 -- STEP: 9286/36938 -- GLOBAL_STEP: 1043550 | > loss_text_ce: 0.03664411976933479 (0.032360369034794334) | > loss_mel_ce: 3.1439502239227295 (nan) | > loss: 0.012621407397091389 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.13710045765731535) | > loader_time: 0.0049 (0.00468068314913196)  --> TIME: 2025-02-26 02:14:07 -- STEP: 9336/36938 -- GLOBAL_STEP: 1043600 | > loss_text_ce: 0.03844963386654854 (0.03235436986859306) | > loss_mel_ce: 3.162688970565796 (nan) | > loss: 0.01270293164998293 (nan) | > current_lr: 5e-06 | > step_time: 0.1688 (0.13709334200431653) | > loader_time: 0.0048 (0.0046802436631259116)  --> TIME: 2025-02-26 02:14:17 -- STEP: 9386/36938 -- GLOBAL_STEP: 1043650 | > loss_text_ce: 0.025768214836716652 (0.0323533239060386) | > loss_mel_ce: 3.1221749782562256 (nan) | > loss: 0.01249183900654316 (nan) | > current_lr: 5e-06 | > step_time: 0.1948 (0.13708114608782504) | > loader_time: 0.0048 (0.00467990053352862)  --> TIME: 2025-02-26 02:14:26 -- STEP: 9436/36938 -- GLOBAL_STEP: 1043700 | > loss_text_ce: 0.03106389008462429 (0.032350558721752215) | > loss_mel_ce: 2.8483927249908447 (nan) | > loss: 0.011426415294408798 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.13707267434579798) | > loader_time: 0.0047 (0.004679545425368148)  --> TIME: 2025-02-26 02:14:35 -- STEP: 9486/36938 -- GLOBAL_STEP: 1043750 | > loss_text_ce: 0.021558215841650963 (0.0323435425682372) | > loss_mel_ce: 1.7959562540054321 (nan) | > loss: 0.007212358992546797 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1370855201881642) | > loader_time: 0.0044 (0.004679418881935287)  --> TIME: 2025-02-26 02:14:45 -- STEP: 9536/36938 -- GLOBAL_STEP: 1043800 | > loss_text_ce: 0.025190167129039764 (0.0323384609675987) | > loss_mel_ce: 1.0447183847427368 (nan) | > loss: 0.004245669115334749 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13710433207202252) | > loader_time: 0.0047 (0.004680074051322559)  --> TIME: 2025-02-26 02:14:54 -- STEP: 9586/36938 -- GLOBAL_STEP: 1043850 | > loss_text_ce: 0.04277338087558746 (0.03233179734950247) | > loss_mel_ce: 1.316192626953125 (nan) | > loss: 0.0053927223198115826 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13707456993156322) | > loader_time: 0.0046 (0.004680473098619587)  --> TIME: 2025-02-26 02:15:03 -- STEP: 9636/36938 -- GLOBAL_STEP: 1043900 | > loss_text_ce: 0.023433959111571312 (0.032335155387979334) | > loss_mel_ce: 2.646904230117798 (nan) | > loss: 0.010596580803394318 (nan) | > current_lr: 5e-06 | > step_time: 0.1799 (0.13706901363932542) | > loader_time: 0.0049 (0.004680836284838564)  --> TIME: 2025-02-26 02:15:13 -- STEP: 9686/36938 -- GLOBAL_STEP: 1043950 | > loss_text_ce: 0.030756814405322075 (0.03233354068248732) | > loss_mel_ce: 2.8374624252319336 (nan) | > loss: 0.011381822638213634 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13706992690582362) | > loader_time: 0.0046 (0.004680997326480888)  --> TIME: 2025-02-26 02:15:22 -- STEP: 9736/36938 -- GLOBAL_STEP: 1044000 | > loss_text_ce: 0.03466784954071045 (0.032341221606683415) | > loss_mel_ce: 2.343653917312622 (nan) | > loss: 0.009437784552574158 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.1370810474297404) | > loader_time: 0.0047 (0.004681297644495283)  --> TIME: 2025-02-26 02:15:31 -- STEP: 9786/36938 -- GLOBAL_STEP: 1044050 | > loss_text_ce: 0.035292256623506546 (0.032335311277841325) | > loss_mel_ce: 3.1124508380889893 (nan) | > loss: 0.012491044588387012 (nan) | > current_lr: 5e-06 | > step_time: 0.189 (0.13705405839596088) | > loader_time: 0.0048 (0.004681193704330289)  --> TIME: 2025-02-26 02:15:41 -- STEP: 9836/36938 -- GLOBAL_STEP: 1044100 | > loss_text_ce: 0.02259998209774494 (0.03233844366338566) | > loss_mel_ce: 2.8593080043792725 (nan) | > loss: 0.011436142958700657 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.1370704647921699) | > loader_time: 0.0046 (0.004681129603911441)  --> TIME: 2025-02-26 02:15:50 -- STEP: 9886/36938 -- GLOBAL_STEP: 1044150 | > loss_text_ce: 0.03626861423254013 (0.03233760362867841) | > loss_mel_ce: 2.5686397552490234 (nan) | > loss: 0.010336939245462418 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13703707977752536) | > loader_time: 0.0045 (0.004680719979494762)  --> TIME: 2025-02-26 02:15:59 -- STEP: 9936/36938 -- GLOBAL_STEP: 1044200 | > loss_text_ce: 0.021466096863150597 (0.03233399113552838) | > loss_mel_ce: 2.733259439468384 (nan) | > loss: 0.010931450873613358 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.1370437277447964) | > loader_time: 0.0047 (0.00468057751559598)  --> TIME: 2025-02-26 02:16:09 -- STEP: 9986/36938 -- GLOBAL_STEP: 1044250 | > loss_text_ce: 0.03586851432919502 (0.03233329360150428) | > loss_mel_ce: 2.9438629150390625 (nan) | > loss: 0.011824331246316433 (nan) | > current_lr: 5e-06 | > step_time: 0.1517 (0.13705897173660897) | > loader_time: 0.0048 (0.004680788853639797)  --> TIME: 2025-02-26 02:16:18 -- STEP: 10036/36938 -- GLOBAL_STEP: 1044300 | > loss_text_ce: 0.032172512263059616 (0.03233505013931867) | > loss_mel_ce: 2.429363250732422 (nan) | > loss: 0.009767998941242695 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.1370654381713622) | > loader_time: 0.0047 (0.004682033078135518)  --> TIME: 2025-02-26 02:16:27 -- STEP: 10086/36938 -- GLOBAL_STEP: 1044350 | > loss_text_ce: 0.03817432001233101 (0.03233790072345983) | > loss_mel_ce: 2.9451582431793213 (nan) | > loss: 0.011838622391223907 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13706952289718946) | > loader_time: 0.0046 (0.004682177237851655)  --> TIME: 2025-02-26 02:16:37 -- STEP: 10136/36938 -- GLOBAL_STEP: 1044400 | > loss_text_ce: 0.024131858721375465 (0.032336712355178654) | > loss_mel_ce: 2.799370527267456 (nan) | > loss: 0.011204374954104424 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13704991180687867) | > loader_time: 0.0045 (0.004682213373793904)  --> TIME: 2025-02-26 02:16:46 -- STEP: 10186/36938 -- GLOBAL_STEP: 1044450 | > loss_text_ce: 0.03176358342170715 (0.03233372299931403) | > loss_mel_ce: 3.040055274963379 (nan) | > loss: 0.012189758010208607 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13704981090932133) | > loader_time: 0.0049 (0.0046821807143778696)  --> TIME: 2025-02-26 02:16:55 -- STEP: 10236/36938 -- GLOBAL_STEP: 1044500 | > loss_text_ce: 0.04178202897310257 (0.03233950330114096) | > loss_mel_ce: 3.202939987182617 (nan) | > loss: 0.012875881977379322 (nan) | > current_lr: 5e-06 | > step_time: 0.1797 (0.13703054021978422) | > loader_time: 0.0048 (0.0046819477819194655)  --> TIME: 2025-02-26 02:17:04 -- STEP: 10286/36938 -- GLOBAL_STEP: 1044550 | > loss_text_ce: 0.04269278421998024 (0.0323444897486048) | > loss_mel_ce: 2.967634439468384 (nan) | > loss: 0.011945744045078754 (nan) | > current_lr: 5e-06 | > step_time: 0.1436 (0.1370187885631922) | > loader_time: 0.0047 (0.004682114053370546)  --> TIME: 2025-02-26 02:17:14 -- STEP: 10336/36938 -- GLOBAL_STEP: 1044600 | > loss_text_ce: 0.03592884540557861 (0.032349667691188896) | > loss_mel_ce: 3.3688504695892334 (nan) | > loss: 0.01351103000342846 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13703318902121245) | > loader_time: 0.0046 (0.00468233783639991)  --> TIME: 2025-02-26 02:17:23 -- STEP: 10386/36938 -- GLOBAL_STEP: 1044650 | > loss_text_ce: 0.0351334884762764 (0.03235096720131772) | > loss_mel_ce: 2.546457529067993 (nan) | > loss: 0.010244409553706646 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13704963621550537) | > loader_time: 0.0046 (0.004682793200371841)  --> TIME: 2025-02-26 02:17:33 -- STEP: 10436/36938 -- GLOBAL_STEP: 1044700 | > loss_text_ce: 0.030427448451519012 (0.03235027372452267) | > loss_mel_ce: 2.8110737800598145 (nan) | > loss: 0.01127579901367426 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.13705108125989318) | > loader_time: 0.0047 (0.004682934275071898)  --> TIME: 2025-02-26 02:17:42 -- STEP: 10486/36938 -- GLOBAL_STEP: 1044750 | > loss_text_ce: 0.04475744068622589 (0.0323446692208594) | > loss_mel_ce: 1.271716594696045 (nan) | > loss: 0.005224103573709726 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13706344826494782) | > loader_time: 0.0046 (0.004682750368127385)  --> TIME: 2025-02-26 02:17:51 -- STEP: 10536/36938 -- GLOBAL_STEP: 1044800 | > loss_text_ce: 0.04171066731214523 (0.03234614605835777) | > loss_mel_ce: 2.3130970001220703 (nan) | > loss: 0.009344475343823433 (nan) | > current_lr: 5e-06 | > step_time: 0.17 (0.13704521826206076) | > loader_time: 0.0049 (0.0046824685035552135)  --> TIME: 2025-02-26 02:18:00 -- STEP: 10586/36938 -- GLOBAL_STEP: 1044850 | > loss_text_ce: 0.036411404609680176 (0.03234475068452404) | > loss_mel_ce: 2.787766695022583 (nan) | > loss: 0.011207057163119316 (nan) | > current_lr: 5e-06 | > step_time: 0.1737 (0.13704476008765695) | > loader_time: 0.0048 (0.0046825292496471375)  --> TIME: 2025-02-26 02:18:10 -- STEP: 10636/36938 -- GLOBAL_STEP: 1044900[0m | > loss_text_ce: 0.030517231673002243 (0.032342647515088904) | > loss_mel_ce: 2.9243948459625244 (nan) | > loss: 0.011725842952728271 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13704922114964432) | > loader_time: 0.0048 (0.004682546789012227)  --> TIME: 2025-02-26 02:18:19 -- STEP: 10686/36938 -- GLOBAL_STEP: 1044950 | > loss_text_ce: 0.03153563290834427 (0.03234938230911454) | > loss_mel_ce: 2.7336835861206055 (nan) | > loss: 0.01097309309989214 (nan) | > current_lr: 5e-06 | > step_time: 0.1811 (0.13705044329980795) | > loader_time: 0.0048 (0.004682310551667118)  --> TIME: 2025-02-26 02:18:28 -- STEP: 10736/36938 -- GLOBAL_STEP: 1045000 | > loss_text_ce: 0.03114829771220684 (0.032345230042696474) | > loss_mel_ce: 3.445458173751831 (nan) | > loss: 0.013796057552099228 (nan) | > current_lr: 5e-06 | > step_time: 0.1747 (0.13704229895741687) | > loader_time: 0.0049 (0.0046820455576375136)  --> TIME: 2025-02-26 02:18:38 -- STEP: 10786/36938 -- GLOBAL_STEP: 1045050 | > loss_text_ce: 0.029465708881616592 (0.03234911532523443) | > loss_mel_ce: 3.1340041160583496 (nan) | > loss: 0.012553452514111996 (nan) | > current_lr: 5e-06 | > step_time: 0.1998 (0.13705497967960353) | > loader_time: 0.0049 (0.004682039520458901)  --> TIME: 2025-02-26 02:18:47 -- STEP: 10836/36938 -- GLOBAL_STEP: 1045100 | > loss_text_ce: 0.037314336746931076 (0.032352858552971986) | > loss_mel_ce: 0.1075439453125 (nan) | > loss: 0.0005748345283791423 (nan) | > current_lr: 5e-06 | > step_time: 0.1062 (0.13705140697687662) | > loader_time: 0.0046 (0.004681976090589986)  --> TIME: 2025-02-26 02:18:57 -- STEP: 10886/36938 -- GLOBAL_STEP: 1045150 | > loss_text_ce: 0.029987027868628502 (0.03235096419377266) | > loss_mel_ce: 2.9658267498016357 (nan) | > loss: 0.011888151057064533 (nan) | > current_lr: 5e-06 | > step_time: 0.1784 (0.13703578468545646) | > loader_time: 0.0048 (0.004681899138896885)  --> TIME: 2025-02-26 02:19:06 -- STEP: 10936/36938 -- GLOBAL_STEP: 1045200 | > loss_text_ce: 0.03526356443762779 (0.032351407245869226) | > loss_mel_ce: 2.615156650543213 (nan) | > loss: 0.010517541319131851 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.137038726743853) | > loader_time: 0.0047 (0.0046817487447734455)  --> TIME: 2025-02-26 02:19:15 -- STEP: 10986/36938 -- GLOBAL_STEP: 1045250 | > loss_text_ce: 0.03669640049338341 (0.032357445203806276) | > loss_mel_ce: 2.774089813232422 (nan) | > loss: 0.011153914034366608 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.1370539338823531) | > loader_time: 0.0047 (0.004681799074524026)  --> TIME: 2025-02-26 02:19:25 -- STEP: 11036/36938 -- GLOBAL_STEP: 1045300 | > loss_text_ce: 0.028821812942624092 (0.03235782122692823) | > loss_mel_ce: 2.491929054260254 (nan) | > loss: 0.010002980008721352 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.1370849561846832) | > loader_time: 0.0067 (0.0046820301601704405)  --> TIME: 2025-02-26 02:19:34 -- STEP: 11086/36938 -- GLOBAL_STEP: 1045350 | > loss_text_ce: 0.03417950123548508 (0.03235704296021719) | > loss_mel_ce: 2.7129149436950684 (nan) | > loss: 0.010901168920099735 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13705192071617742) | > loader_time: 0.0048 (0.004681768753716555)  --> TIME: 2025-02-26 02:19:43 -- STEP: 11136/36938 -- GLOBAL_STEP: 1045400 | > loss_text_ce: 0.031778182834386826 (0.032360421770554156) | > loss_mel_ce: 2.5795235633850098 (nan) | > loss: 0.010362308472394943 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13703223264337305) | > loader_time: 0.0045 (0.004681486486532217)  --> TIME: 2025-02-26 02:19:53 -- STEP: 11186/36938 -- GLOBAL_STEP: 1045450 | > loss_text_ce: 0.030517395585775375 (0.03235725065125119) | > loss_mel_ce: 2.3731420040130615 (nan) | > loss: 0.00953833106905222 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13705299694764495) | > loader_time: 0.0046 (0.004681351294399008)  --> TIME: 2025-02-26 02:20:02 -- STEP: 11236/36938 -- GLOBAL_STEP: 1045500 | > loss_text_ce: 0.023076627403497696 (0.03235824475299953) | > loss_mel_ce: 2.2364728450775146 (nan) | > loss: 0.008966466411948204 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13705847935135085) | > loader_time: 0.0046 (0.004681565809097099)  --> TIME: 2025-02-26 02:20:12 -- STEP: 11286/36938 -- GLOBAL_STEP: 1045550 | > loss_text_ce: 0.03303360193967819 (0.03235734769617009) | > loss_mel_ce: 2.3460745811462402 (nan) | > loss: 0.009440906345844269 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.1370644719385627) | > loader_time: 0.0049 (0.00468194256558943)  --> TIME: 2025-02-26 02:20:21 -- STEP: 11336/36938 -- GLOBAL_STEP: 1045600 | > loss_text_ce: 0.05502331629395485 (0.03235856941518316) | > loss_mel_ce: 0.9525431394577026 (nan) | > loss: 0.003998279571533203 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13705799221235387) | > loader_time: 0.0045 (0.004682038166023266)  --> TIME: 2025-02-26 02:20:30 -- STEP: 11386/36938 -- GLOBAL_STEP: 1045650 | > loss_text_ce: 0.0336005836725235 (0.0323603578959982) | > loss_mel_ce: 3.245236396789551 (nan) | > loss: 0.013011258095502853 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13705512567708145) | > loader_time: 0.0046 (0.004681961242833927)  --> TIME: 2025-02-26 02:20:40 -- STEP: 11436/36938 -- GLOBAL_STEP: 1045700 | > loss_text_ce: 0.03380229324102402 (0.03236510927610419) | > loss_mel_ce: 2.6773736476898193 (nan) | > loss: 0.010758635587990284 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.1370749715409446) | > loader_time: 0.0046 (0.004682229548411262)  --> TIME: 2025-02-26 02:20:49 -- STEP: 11486/36938 -- GLOBAL_STEP: 1045750 | > loss_text_ce: 0.033089883625507355 (0.03236361548890646) | > loss_mel_ce: 2.8744523525238037 (nan) | > loss: 0.011537866666913033 (nan) | > current_lr: 5e-06 | > step_time: 0.1504 (0.1370833791051621) | > loader_time: 0.0046 (0.004682298281996929)  --> TIME: 2025-02-26 02:20:59 -- STEP: 11536/36938 -- GLOBAL_STEP: 1045800 | > loss_text_ce: 0.031011110171675682 (0.03236549274245784) | > loss_mel_ce: 1.973785400390625 (nan) | > loss: 0.007955541834235191 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13709679622094662) | > loader_time: 0.0047 (0.0046820525446811204)  --> TIME: 2025-02-26 02:21:08 -- STEP: 11586/36938 -- GLOBAL_STEP: 1045850 | > loss_text_ce: 0.021726403385400772 (0.032361642153454426) | > loss_mel_ce: 2.7189197540283203 (nan) | > loss: 0.010875580832362175 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13708125604530472) | > loader_time: 0.0045 (0.0046817109968796955)  --> TIME: 2025-02-26 02:21:17 -- STEP: 11636/36938 -- GLOBAL_STEP: 1045900 | > loss_text_ce: 0.03571755439043045 (0.03236540720971617) | > loss_mel_ce: 2.7874016761779785 (nan) | > loss: 0.011202854104340076 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13707467537641005) | > loader_time: 0.005 (0.004681706817832843)  --> TIME: 2025-02-26 02:21:27 -- STEP: 11686/36938 -- GLOBAL_STEP: 1045950 | > loss_text_ce: 0.033543359488248825 (0.03236642039841468) | > loss_mel_ce: 2.5231213569641113 (nan) | > loss: 0.010145495645701885 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13708788883843223) | > loader_time: 0.0046 (0.004681567796468537)  --> TIME: 2025-02-26 02:21:36 -- STEP: 11736/36938 -- GLOBAL_STEP: 1046000 | > loss_text_ce: 0.03361871466040611 (0.03236274223562972) | > loss_mel_ce: 2.641256093978882 (nan) | > loss: 0.010614583268761635 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13707980257601893) | > loader_time: 0.0048 (0.004681255940116833)  --> TIME: 2025-02-26 02:21:45 -- STEP: 11786/36938 -- GLOBAL_STEP: 1046050 | > loss_text_ce: 0.032652847468853 (0.032361005996707354) | > loss_mel_ce: 2.8300585746765137 (nan) | > loss: 0.011359966360032558 (nan) | > current_lr: 5e-06 | > step_time: 0.1962 (0.1370742488105659) | > loader_time: 0.0049 (0.00468102736240856)  --> TIME: 2025-02-26 02:21:54 -- STEP: 11836/36938 -- GLOBAL_STEP: 1046100 | > loss_text_ce: 0.03261009231209755 (0.03236059187532207) | > loss_mel_ce: 2.962386131286621 (nan) | > loss: 0.011884906329214573 (nan) | > current_lr: 5e-06 | > step_time: 0.1717 (0.13706920798786162) | > loader_time: 0.0048 (0.0046809905684852444)  --> TIME: 2025-02-26 02:22:03 -- STEP: 11886/36938 -- GLOBAL_STEP: 1046150 | > loss_text_ce: 0.022905446588993073 (0.03236059146856021) | > loss_mel_ce: 1.597947120666504 (nan) | > loss: 0.00643195491284132 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.137039006147525) | > loader_time: 0.0046 (0.0046807862323040164)  --> TIME: 2025-02-26 02:22:13 -- STEP: 11936/36938 -- GLOBAL_STEP: 1046200 | > loss_text_ce: 0.039841923862695694 (0.032354034109062554) | > loss_mel_ce: 2.5412139892578125 (nan) | > loss: 0.010242286138236523 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13702933422362276) | > loader_time: 0.0046 (0.0046806409555529786)  --> TIME: 2025-02-26 02:22:22 -- STEP: 11986/36938 -- GLOBAL_STEP: 1046250 | > loss_text_ce: 0.02508467622101307 (0.032355896288325074) | > loss_mel_ce: 2.5028440952301025 (nan) | > loss: 0.010031464509665966 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.1370289325913033) | > loader_time: 0.0047 (0.004680751998337407)  --> TIME: 2025-02-26 02:22:31 -- STEP: 12036/36938 -- GLOBAL_STEP: 1046300 | > loss_text_ce: 0.03015265241265297 (0.032352120788559556) | > loss_mel_ce: 2.095184326171875 (nan) | > loss: 0.008433876559138298 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13700796916818492) | > loader_time: 0.0045 (0.004680801800771902)  --> TIME: 2025-02-26 02:22:40 -- STEP: 12086/36938 -- GLOBAL_STEP: 1046350 | > loss_text_ce: 0.03479897603392601 (0.032353401473974336) | > loss_mel_ce: 2.5863003730773926 (nan) | > loss: 0.010401188395917416 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13699984546530788) | > loader_time: 0.0045 (0.004681189249983376)  --> TIME: 2025-02-26 02:22:50 -- STEP: 12136/36938 -- GLOBAL_STEP: 1046400 | > loss_text_ce: 0.027538111433386803 (0.032350436823925396) | > loss_mel_ce: 1.1977274417877197 (nan) | > loss: 0.004862165078520775 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.13702432146257923) | > loader_time: 0.0047 (0.004681496456727122)  --> TIME: 2025-02-26 02:22:59 -- STEP: 12186/36938 -- GLOBAL_STEP: 1046450 | > loss_text_ce: 0.02938806265592575 (0.032351499117223924) | > loss_mel_ce: 2.9497807025909424 (nan) | > loss: 0.01182209886610508 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13703292169027875) | > loader_time: 0.0047 (0.004681444375479379)  --> TIME: 2025-02-26 02:23:09 -- STEP: 12236/36938 -- GLOBAL_STEP: 1046500 | > loss_text_ce: 0.02739214338362217 (0.03235558515804932) | > loss_mel_ce: 2.691344976425171 (nan) | > loss: 0.01078864000737667 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13704179684762827) | > loader_time: 0.0046 (0.0046813365640730655)  --> TIME: 2025-02-26 02:23:18 -- STEP: 12286/36938 -- GLOBAL_STEP: 1046550 | > loss_text_ce: 0.03358352929353714 (0.03236096120986344) | > loss_mel_ce: 2.971001625061035 (nan) | > loss: 0.011922958306968212 (nan) | > current_lr: 5e-06 | > step_time: 0.1922 (0.13704339019564965) | > loader_time: 0.0048 (0.0046811856568373875)  --> TIME: 2025-02-26 02:23:27 -- STEP: 12336/36938 -- GLOBAL_STEP: 1046600 | > loss_text_ce: 0.0317181795835495 (0.03236714887252194) | > loss_mel_ce: 3.0444281101226807 (nan) | > loss: 0.012206930667161942 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13704211555721013) | > loader_time: 0.0047 (0.0046810232557056766)  --> TIME: 2025-02-26 02:23:37 -- STEP: 12386/36938 -- GLOBAL_STEP: 1046650 | > loss_text_ce: 0.029723603278398514 (0.0323615688623088) | > loss_mel_ce: 2.8560192584991455 (nan) | > loss: 0.011451361700892448 (nan) | > current_lr: 5e-06 | > step_time: 0.1893 (0.13705831324363207) | > loader_time: 0.0048 (0.004680939046396838)  --> TIME: 2025-02-26 02:23:47 -- STEP: 12436/36938 -- GLOBAL_STEP: 1046700 | > loss_text_ce: 0.03522561118006706 (0.03236377764066086) | > loss_mel_ce: 3.283482074737549 (nan) | > loss: 0.013169475831091404 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.1370709944131797) | > loader_time: 0.0047 (0.004681001736367003)  --> TIME: 2025-02-26 02:23:56 -- STEP: 12486/36938 -- GLOBAL_STEP: 1046750 | > loss_text_ce: 0.037637773901224136 (0.03236683589232839) | > loss_mel_ce: 2.320648193359375 (nan) | > loss: 0.009358277544379234 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13707603366447985) | > loader_time: 0.0046 (0.004680892719627477)  --> TIME: 2025-02-26 02:24:06 -- STEP: 12536/36938 -- GLOBAL_STEP: 1046800 | > loss_text_ce: 0.03183087334036827 (0.0323687872915378) | > loss_mel_ce: 3.1913018226623535 (nan) | > loss: 0.012790209613740444 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.13711129221173776) | > loader_time: 0.0049 (0.004681844656529821)  --> TIME: 2025-02-26 02:24:15 -- STEP: 12586/36938 -- GLOBAL_STEP: 1046850 | > loss_text_ce: 0.035464175045490265 (0.03236645805131059) | > loss_mel_ce: 0.2794189453125 (nan) | > loss: 0.001249536289833486 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13711179895884498) | > loader_time: 0.0046 (0.004681569166409276)  --> TIME: 2025-02-26 02:24:25 -- STEP: 12636/36938 -- GLOBAL_STEP: 1046900 | > loss_text_ce: 0.03195364773273468 (0.03236605519310647) | > loss_mel_ce: 3.0218505859375 (nan) | > loss: 0.012118271552026272 (nan) | > current_lr: 5e-06 | > step_time: 0.168 (0.13710870628456626) | > loader_time: 0.0049 (0.004681349649728198)  --> TIME: 2025-02-26 02:24:34 -- STEP: 12686/36938 -- GLOBAL_STEP: 1046950 | > loss_text_ce: 0.0396517850458622 (0.03236429138642382) | > loss_mel_ce: 2.3670151233673096 (nan) | > loss: 0.009550265967845917 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13712581452296535) | > loader_time: 0.0045 (0.004681275899376799)  --> TIME: 2025-02-26 02:24:44 -- STEP: 12736/36938 -- GLOBAL_STEP: 1047000 | > loss_text_ce: 0.033854901790618896 (0.03236135057457122) | > loss_mel_ce: 2.9442410469055176 (nan) | > loss: 0.011817841790616512 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13713615546798658) | > loader_time: 0.0046 (0.004681247787259933)  --> TIME: 2025-02-26 02:24:53 -- STEP: 12786/36938 -- GLOBAL_STEP: 1047050 | > loss_text_ce: 0.03839647024869919 (0.03236434926459109) | > loss_mel_ce: 3.1016242504119873 (nan) | > loss: 0.012460400350391865 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13712268826287696) | > loader_time: 0.0046 (0.00468113561126337)  --> TIME: 2025-02-26 02:25:02 -- STEP: 12836/36938 -- GLOBAL_STEP: 1047100 | > loss_text_ce: 0.02422964945435524 (0.032363021740255855) | > loss_mel_ce: 2.4268972873687744 (nan) | > loss: 0.009726694785058498 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13712732460908272) | > loader_time: 0.0044 (0.004681021615923216)  --> TIME: 2025-02-26 02:25:12 -- STEP: 12886/36938 -- GLOBAL_STEP: 1047150 | > loss_text_ce: 0.036249320954084396 (0.032369631456855175) | > loss_mel_ce: 0.014712679199874401 (nan) | > loss: 0.00020223017781972885 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13712096040374394) | > loader_time: 0.0048 (0.004680951615214705)  --> TIME: 2025-02-26 02:25:21 -- STEP: 12936/36938 -- GLOBAL_STEP: 1047200 | > loss_text_ce: 0.03708706051111221 (0.03237487674031613) | > loss_mel_ce: 1.0987142324447632 (nan) | > loss: 0.004507148172706366 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.1371054455831461) | > loader_time: 0.0046 (0.004680753896910248)  --> TIME: 2025-02-26 02:25:30 -- STEP: 12986/36938 -- GLOBAL_STEP: 1047250 | > loss_text_ce: 0.03388999029994011 (0.032378080374640704) | > loss_mel_ce: 3.0292246341705322 (nan) | > loss: 0.012155217118561268 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13710240926321102) | > loader_time: 0.0048 (0.004680673862741044)  --> TIME: 2025-02-26 02:25:39 -- STEP: 13036/36938 -- GLOBAL_STEP: 1047300 | > loss_text_ce: 0.0277479849755764 (0.03237903249923276) | > loss_mel_ce: 2.047267436981201 (nan) | > loss: 0.008234187960624695 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13711179791509404) | > loader_time: 0.0045 (0.004680557242197726)  --> TIME: 2025-02-26 02:25:48 -- STEP: 13086/36938 -- GLOBAL_STEP: 1047350 | > loss_text_ce: 0.03338030353188515 (0.03237797752463725) | > loss_mel_ce: 0.8756866455078125 (nan) | > loss: 0.003607408842071891 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13710022318824225) | > loader_time: 0.0045 (0.004680341980462043)  --> TIME: 2025-02-26 02:25:58 -- STEP: 13136/36938 -- GLOBAL_STEP: 1047400 | > loss_text_ce: 0.04204818233847618 (0.03237850814835149) | > loss_mel_ce: 2.695821762084961 (nan) | > loss: 0.010864564217627048 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13710379821955157) | > loader_time: 0.0047 (0.004680345722005662)  --> TIME: 2025-02-26 02:26:07 -- STEP: 13186/36938 -- GLOBAL_STEP: 1047450 | > loss_text_ce: 0.029682321473956108 (0.03237837801393476) | > loss_mel_ce: 3.258120059967041 (nan) | > loss: 0.01304683554917574 (nan) | > current_lr: 5e-06 | > step_time: 0.1815 (0.13710914474688746) | > loader_time: 0.0048 (0.004680264923689345)  --> TIME: 2025-02-26 02:26:17 -- STEP: 13236/36938 -- GLOBAL_STEP: 1047500 | > loss_text_ce: 0.027660522609949112 (0.03237766804981092) | > loss_mel_ce: 0.009307861328125 (nan) | > loss: 0.0001466999383410439 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1370965450975111) | > loader_time: 0.0046 (0.0046801947149563245)  --> TIME: 2025-02-26 02:26:26 -- STEP: 13286/36938 -- GLOBAL_STEP: 1047550 | > loss_text_ce: 0.03340475261211395 (0.03237729083785102) | > loss_mel_ce: 0.0756683349609375 (nan) | > loss: 0.00043282975093461573 (nan) | > current_lr: 5e-06 | > step_time: 0.1613 (0.13709866765499126) | > loader_time: 0.0048 (0.004680194894933352)  --> TIME: 2025-02-26 02:26:35 -- STEP: 13336/36938 -- GLOBAL_STEP: 1047600 | > loss_text_ce: 0.022454530000686646 (0.03238023861458606) | > loss_mel_ce: 0.059146881103515625 (nan) | > loss: 0.00032381515484303236 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.1370906206792041) | > loader_time: 0.0047 (0.004680005193257232)  --> TIME: 2025-02-26 02:26:44 -- STEP: 13386/36938 -- GLOBAL_STEP: 1047650 | > loss_text_ce: 0.04058963060379028 (0.0323787439015779) | > loss_mel_ce: 2.2368903160095215 (nan) | > loss: 0.009037619456648827 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13709053331723176) | > loader_time: 0.0045 (0.0046797236857634384)  --> TIME: 2025-02-26 02:26:54 -- STEP: 13436/36938 -- GLOBAL_STEP: 1047700 | > loss_text_ce: 0.037045739591121674 (0.032372002916293705) | > loss_mel_ce: 2.1076343059539795 (nan) | > loss: 0.008510635234415531 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.13708729696401667) | > loader_time: 0.0047 (0.004679470393727954)  --> TIME: 2025-02-26 02:27:03 -- STEP: 13486/36938 -- GLOBAL_STEP: 1047750 | > loss_text_ce: 0.033697132021188736 (0.03237066262095174) | > loss_mel_ce: 2.2799839973449707 (nan) | > loss: 0.00918127503246069 (nan) | > current_lr: 5e-06 | > step_time: 0.1785 (0.13708240012112827) | > loader_time: 0.0049 (0.004679274332730189)  --> TIME: 2025-02-26 02:27:12 -- STEP: 13536/36938 -- GLOBAL_STEP: 1047800 | > loss_text_ce: 0.040321044623851776 (0.03236824836324362) | > loss_mel_ce: 1.2965223789215088 (nan) | > loss: 0.005304934456944466 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13708369418027647) | > loader_time: 0.0046 (0.004679080935516536)  --> TIME: 2025-02-26 02:27:21 -- STEP: 13586/36938 -- GLOBAL_STEP: 1047850 | > loss_text_ce: 0.03609209880232811 (0.03237134155890145) | > loss_mel_ce: 3.042466163635254 (nan) | > loss: 0.012216501869261265 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.1370648145500179) | > loader_time: 0.0068 (0.004679024263385047)  --> TIME: 2025-02-26 02:27:30 -- STEP: 13636/36938 -- GLOBAL_STEP: 1047900 | > loss_text_ce: 0.026797913014888763 (0.03236897604514871) | > loss_mel_ce: 2.7405834197998047 (nan) | > loss: 0.010981673374772072 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13704398145910743) | > loader_time: 0.0046 (0.004678985001789787)  --> TIME: 2025-02-26 02:27:40 -- STEP: 13686/36938 -- GLOBAL_STEP: 1047950 | > loss_text_ce: 0.0262104794383049 (0.032370315059931466) | > loss_mel_ce: 2.064391851425171 (nan) | > loss: 0.008296041749417782 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13703332113867864) | > loader_time: 0.0046 (0.004678956113606539)  --> TIME: 2025-02-26 02:27:49 -- STEP: 13736/36938 -- GLOBAL_STEP: 1048000 | > loss_text_ce: 0.030688302591443062 (0.03236836754707727) | > loss_mel_ce: 0.0550130195915699 (nan) | > loss: 0.00034008463262580335 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13704477244501087) | > loader_time: 0.0046 (0.004679186908603062)  --> TIME: 2025-02-26 02:27:58 -- STEP: 13786/36938 -- GLOBAL_STEP: 1048050 | > loss_text_ce: 0.02892421744763851 (0.03236932441620335) | > loss_mel_ce: 2.9174697399139404 (nan) | > loss: 0.011692040599882603 (nan) | > current_lr: 5e-06 | > step_time: 0.1878 (0.13701982173382946) | > loader_time: 0.0072 (0.004679195095384898)  --> TIME: 2025-02-26 02:28:07 -- STEP: 13836/36938 -- GLOBAL_STEP: 1048100 | > loss_text_ce: 0.032858364284038544 (0.032367849860897295) | > loss_mel_ce: 2.921654462814331 (nan) | > loss: 0.011724257841706276 (nan) | > current_lr: 5e-06 | > step_time: 0.1261 (0.13702758308365504) | > loader_time: 0.0045 (0.004678961892250681)  --> TIME: 2025-02-26 02:28:16 -- STEP: 13886/36938 -- GLOBAL_STEP: 1048150 | > loss_text_ce: 0.026812171563506126 (0.032371685778217696) | > loss_mel_ce: 0.936889111995697 (nan) | > loss: 0.0038242116570472717 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13700467575597472) | > loader_time: 0.0045 (0.0046787778599434275)  --> TIME: 2025-02-26 02:28:26 -- STEP: 13936/36938 -- GLOBAL_STEP: 1048200 | > loss_text_ce: 0.04053905978798866 (0.032371056837780535) | > loss_mel_ce: 2.8923165798187256 (nan) | > loss: 0.011638316325843334 (nan) | > current_lr: 5e-06 | > step_time: 0.1762 (0.13701515440346945) | > loader_time: 0.0049 (0.004678778341799188)  --> TIME: 2025-02-26 02:28:35 -- STEP: 13986/36938 -- GLOBAL_STEP: 1048250 | > loss_text_ce: 0.04122307151556015 (0.032364946953051164) | > loss_mel_ce: 3.4079976081848145 (nan) | > loss: 0.013687384314835072 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13702001768377098) | > loader_time: 0.0045 (0.0046789039277423524)  --> TIME: 2025-02-26 02:28:44 -- STEP: 14036/36938 -- GLOBAL_STEP: 1048300 | > loss_text_ce: 0.035261377692222595 (0.032367950381648164) | > loss_mel_ce: 2.4435276985168457 (nan) | > loss: 0.009836465120315552 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13700618941616158) | > loader_time: 0.0046 (0.004678803126877888)  --> TIME: 2025-02-26 02:28:54 -- STEP: 14086/36938 -- GLOBAL_STEP: 1048350 | > loss_text_ce: 0.03584272786974907 (0.032362910672011164) | > loss_mel_ce: 2.4931488037109375 (nan) | > loss: 0.01003568060696125 (nan) | > current_lr: 5e-06 | > step_time: 0.1514 (0.13699547919160107) | > loader_time: 0.0046 (0.004678805223432739)  --> TIME: 2025-02-26 02:29:03 -- STEP: 14136/36938 -- GLOBAL_STEP: 1048400 | > loss_text_ce: 0.03467446193099022 (0.0323629745308061) | > loss_mel_ce: 3.3185369968414307 (nan) | > loss: 0.013306395150721073 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.1369997092776205) | > loader_time: 0.0047 (0.004678688129598836)  --> TIME: 2025-02-26 02:29:12 -- STEP: 14186/36938 -- GLOBAL_STEP: 1048450 | > loss_text_ce: 0.026293890550732613 (0.03236090246759754) | > loss_mel_ce: 2.808784246444702 (nan) | > loss: 0.011250311508774757 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.1369868513224549) | > loader_time: 0.0046 (0.004678614230648715)  --> TIME: 2025-02-26 02:29:21 -- STEP: 14236/36938 -- GLOBAL_STEP: 1048500 | > loss_text_ce: 0.03165537118911743 (0.0323639926974263) | > loss_mel_ce: 3.12296986579895 (nan) | > loss: 0.012518354691565037 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.1369748317180445) | > loader_time: 0.0048 (0.004678552540610031)  --> TIME: 2025-02-26 02:29:31 -- STEP: 14286/36938 -- GLOBAL_STEP: 1048550 | > loss_text_ce: 0.04022640734910965 (0.03236191954146129) | > loss_mel_ce: 2.6108572483062744 (nan) | > loss: 0.01052017416805029 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.1369657672688226) | > loader_time: 0.0046 (0.004678495671591237)  --> TIME: 2025-02-26 02:29:40 -- STEP: 14336/36938 -- GLOBAL_STEP: 1048600 | > loss_text_ce: 0.03129763901233673 (0.03235889720593039) | > loss_mel_ce: 2.7306745052337646 (nan) | > loss: 0.01096020732074976 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13694724537033104) | > loader_time: 0.0046 (0.004678474107225027)  --> TIME: 2025-02-26 02:29:49 -- STEP: 14386/36938 -- GLOBAL_STEP: 1048650 | > loss_text_ce: 0.04081703722476959 (0.03236079511241625) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13694118604894737) | > loader_time: 0.0046 (0.004678268086573817)  --> TIME: 2025-02-26 02:29:58 -- STEP: 14436/36938 -- GLOBAL_STEP: 1048700 | > loss_text_ce: 0.026225140318274498 (0.03235789369960788) | > loss_mel_ce: 2.9502155780792236 (nan) | > loss: 0.01181127317249775 (nan) | > current_lr: 5e-06 | > step_time: 0.1527 (0.13694266218926418) | > loader_time: 0.0047 (0.0046782911104586485)  --> TIME: 2025-02-26 02:30:07 -- STEP: 14486/36938 -- GLOBAL_STEP: 1048750 | > loss_text_ce: 0.04291744530200958 (0.032357738670460746) | > loss_mel_ce: 2.9476876258850098 (nan) | > loss: 0.011867481283843517 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13692677322218147) | > loader_time: 0.0046 (0.0046781421975011115)  --> TIME: 2025-02-26 02:30:17 -- STEP: 14536/36938 -- GLOBAL_STEP: 1048800 | > loss_text_ce: 0.04513925313949585 (0.032355834254505646) | > loss_mel_ce: 2.3148715496063232 (nan) | > loss: 0.009365122765302658 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13692295482411945) | > loader_time: 0.0045 (0.004678059605103505)  --> TIME: 2025-02-26 02:30:26 -- STEP: 14586/36938 -- GLOBAL_STEP: 1048850 | > loss_text_ce: 0.03235947713255882 (0.03235903902829321) | > loss_mel_ce: 2.76300048828125 (nan) | > loss: 0.011092698201537132 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13693077520433022) | > loader_time: 0.0046 (0.004678006788740494)  --> TIME: 2025-02-26 02:30:36 -- STEP: 14636/36938 -- GLOBAL_STEP: 1048900 | > loss_text_ce: 0.02618790976703167 (0.032361219253202296) | > loss_mel_ce: 2.7724826335906982 (nan) | > loss: 0.011105836369097233 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.1369349788563866) | > loader_time: 0.0047 (0.004677891209986276)  --> TIME: 2025-02-26 02:30:45 -- STEP: 14686/36938 -- GLOBAL_STEP: 1048950 | > loss_text_ce: 0.025411494076251984 (0.03235996483770496) | > loss_mel_ce: 2.836240530014038 (nan) | > loss: 0.011355763301253319 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.1369379541618406) | > loader_time: 0.0047 (0.0046777575051412746)  --> TIME: 2025-02-26 02:30:54 -- STEP: 14736/36938 -- GLOBAL_STEP: 1049000 | > loss_text_ce: 0.03237197920680046 (0.03235917360918511) | > loss_mel_ce: 2.2393972873687744 (nan) | > loss: 0.009014957584440708 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13693226288183027) | > loader_time: 0.0044 (0.004677580279195995)  --> TIME: 2025-02-26 02:31:04 -- STEP: 14786/36938 -- GLOBAL_STEP: 1049050 | > loss_text_ce: 0.03087509796023369 (0.03236248427785337) | > loss_mel_ce: 2.7171759605407715 (nan) | > loss: 0.010904965922236443 (nan) | > current_lr: 5e-06 | > step_time: 0.1577 (0.13693841437630458) | > loader_time: 0.0046 (0.004677387304885226)  --> TIME: 2025-02-26 02:31:13 -- STEP: 14836/36938 -- GLOBAL_STEP: 1049100 | > loss_text_ce: 0.03041314333677292 (0.03236241426978529) | > loss_mel_ce: 2.275026559829712 (nan) | > loss: 0.009148570708930492 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.1369221664304268) | > loader_time: 0.0048 (0.00467708812116066)  --> TIME: 2025-02-26 02:31:22 -- STEP: 14886/36938 -- GLOBAL_STEP: 1049150 | > loss_text_ce: 0.029054036363959312 (0.0323583201064046) | > loss_mel_ce: 2.5249502658843994 (nan) | > loss: 0.010134938172996044 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13692701434666335) | > loader_time: 0.0045 (0.004676882384301641)  --> TIME: 2025-02-26 02:31:31 -- STEP: 14936/36938 -- GLOBAL_STEP: 1049200 | > loss_text_ce: 0.037018951028585434 (0.03236313652068621) | > loss_mel_ce: 2.048464298248291 (nan) | > loss: 0.008275727741420269 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13691947845832958) | > loader_time: 0.0044 (0.004676633600762137)  --> TIME: 2025-02-26 02:31:41 -- STEP: 14986/36938 -- GLOBAL_STEP: 1049250 | > loss_text_ce: 0.031021343544125557 (0.032360975879166205) | > loss_mel_ce: 2.279176712036133 (nan) | > loss: 0.009167453274130821 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13691786341652604) | > loader_time: 0.0046 (0.004676412043768113)  --> TIME: 2025-02-26 02:31:50 -- STEP: 15036/36938 -- GLOBAL_STEP: 1049300 | > loss_text_ce: 0.03354162722826004 (0.03235872121953697) | > loss_mel_ce: 3.2745919227600098 (nan) | > loss: 0.013127515092492104 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.13692237868465112) | > loader_time: 0.0048 (0.004676241654606378)  --> TIME: 2025-02-26 02:31:59 -- STEP: 15086/36938 -- GLOBAL_STEP: 1049350 | > loss_text_ce: 0.029309634119272232 (0.03236009712293595) | > loss_mel_ce: 4.153124809265137 (nan) | > loss: 0.01659696362912655 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.1369197884937289) | > loader_time: 0.0045 (0.00467605437837955)  --> TIME: 2025-02-26 02:32:09 -- STEP: 15136/36938 -- GLOBAL_STEP: 1049400 | > loss_text_ce: 0.025890840217471123 (0.03235664635476559) | > loss_mel_ce: 1.9750021696090698 (nan) | > loss: 0.007940052077174187 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13691526222934747) | > loader_time: 0.0047 (0.004675890297390694)  --> TIME: 2025-02-26 02:32:18 -- STEP: 15186/36938 -- GLOBAL_STEP: 1049450 | > loss_text_ce: 0.033260855823755264 (0.03235926349448279) | > loss_mel_ce: 2.6389617919921875 (nan) | > loss: 0.010604058392345905 (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.13690788056968872) | > loader_time: 0.0046 (0.004675634903005346)  --> TIME: 2025-02-26 02:32:27 -- STEP: 15236/36938 -- GLOBAL_STEP: 1049500 | > loss_text_ce: 0.027961919084191322 (0.032361630170700734) | > loss_mel_ce: 2.710275173187256 (nan) | > loss: 0.010866020806133747 (nan) | > current_lr: 5e-06 | > step_time: 0.1729 (0.13691769387347144) | > loader_time: 0.0048 (0.0046755757723683355)  --> TIME: 2025-02-26 02:32:36 -- STEP: 15286/36938 -- GLOBAL_STEP: 1049550 | > loss_text_ce: 0.029337452724575996 (0.03235750733559749) | > loss_mel_ce: 1.2076280117034912 (nan) | > loss: 0.004908593371510506 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13691628241311346) | > loader_time: 0.0045 (0.004675396665078752)  --> TIME: 2025-02-26 02:32:46 -- STEP: 15336/36938 -- GLOBAL_STEP: 1049600 | > loss_text_ce: 0.03244916722178459 (0.03235490684689983) | > loss_mel_ce: 2.97637677192688 (nan) | > loss: 0.01193978637456894 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13692097916973775) | > loader_time: 0.0047 (0.0046752148390935624)  --> TIME: 2025-02-26 02:32:55 -- STEP: 15386/36938 -- GLOBAL_STEP: 1049650 | > loss_text_ce: 0.030607813969254494 (0.03235542691116249) | > loss_mel_ce: 1.150193691253662 (nan) | > loss: 0.004685720428824425 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13690376114383562) | > loader_time: 0.0046 (0.004674947805713339)  --> TIME: 2025-02-26 02:33:04 -- STEP: 15436/36938 -- GLOBAL_STEP: 1049700 | > loss_text_ce: 0.030884303152561188 (0.03235405059297236) | > loss_mel_ce: 2.5899970531463623 (nan) | > loss: 0.010400323197245598 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.13691133024409016) | > loader_time: 0.0049 (0.004674860590262313)  --> TIME: 2025-02-26 02:33:14 -- STEP: 15486/36938 -- GLOBAL_STEP: 1049750 | > loss_text_ce: 0.04006873443722725 (0.032351146472215535) | > loss_mel_ce: 2.9968602657318115 (nan) | > loss: 0.01205130573362112 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13693138511701625) | > loader_time: 0.0046 (0.004675036450927608)  --> TIME: 2025-02-26 02:33:23 -- STEP: 15536/36938 -- GLOBAL_STEP: 1049800 | > loss_text_ce: 0.02750788815319538 (0.03235440196176409) | > loss_mel_ce: 2.7714571952819824 (nan) | > loss: 0.0111070042476058 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.1369187304429739) | > loader_time: 0.0046 (0.004674974403101433)  --> TIME: 2025-02-26 02:33:33 -- STEP: 15586/36938 -- GLOBAL_STEP: 1049850 | > loss_text_ce: 0.028283093124628067 (0.03235423069629649) | > loss_mel_ce: 0.07252945005893707 (nan) | > loss: 0.00040004978654906154 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13691985932972744) | > loader_time: 0.0048 (0.004674921549132925)  --> TIME: 2025-02-26 02:33:42 -- STEP: 15636/36938 -- GLOBAL_STEP: 1049900 | > loss_text_ce: 0.03811155632138252 (0.032353230722329096) | > loss_mel_ce: 2.990689516067505 (nan) | > loss: 0.012019052170217037 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13691809797079674) | > loader_time: 0.0046 (0.004674875406878596)  --> TIME: 2025-02-26 02:33:51 -- STEP: 15686/36938 -- GLOBAL_STEP: 1049950 | > loss_text_ce: 0.030201494693756104 (0.032352692350528144) | > loss_mel_ce: 2.677314281463623 (nan) | > loss: 0.010744110681116581 (nan) | > current_lr: 5e-06 | > step_time: 0.179 (0.13692524755972374) | > loader_time: 0.0049 (0.004674813812155665)  --> TIME: 2025-02-26 02:34:00 -- STEP: 15736/36938 -- GLOBAL_STEP: 1050000 | > loss_text_ce: 0.032484665513038635 (0.032350238378469356) | > loss_mel_ce: 1.9381428956985474 (nan) | > loss: 0.007819950580596924 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13690345061267822) | > loader_time: 0.0045 (0.004674704549393802) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1050000.pth  --> TIME: 2025-02-26 02:34:15 -- STEP: 15786/36938 -- GLOBAL_STEP: 1050050 | > loss_text_ce: 0.03518577665090561 (0.03234990703175171) | > loss_mel_ce: 2.8495845794677734 (nan) | > loss: 0.011447502300143242 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.13690195401690902) | > loader_time: 0.0047 (0.004674704087244215)  --> TIME: 2025-02-26 02:34:25 -- STEP: 15836/36938 -- GLOBAL_STEP: 1050100 | > loss_text_ce: 0.03454749286174774 (0.03234854477611085) | > loss_mel_ce: 2.837592840194702 (nan) | > loss: 0.01139738317579031 (nan) | > current_lr: 5e-06 | > step_time: 0.1875 (0.13690755101700447) | > loader_time: 0.0049 (0.004674622840308035)  --> TIME: 2025-02-26 02:34:34 -- STEP: 15886/36938 -- GLOBAL_STEP: 1050150 | > loss_text_ce: 0.044424209743738174 (0.03235393459735724) | > loss_mel_ce: 3.058485746383667 (nan) | > loss: 0.012313135899603367 (nan) | > current_lr: 5e-06 | > step_time: 0.1759 (0.1368980029961127) | > loader_time: 0.0049 (0.004674631703129557)  --> TIME: 2025-02-26 02:34:43 -- STEP: 15936/36938 -- GLOBAL_STEP: 1050200 | > loss_text_ce: 0.036677129566669464 (0.03235159604237457) | > loss_mel_ce: 2.8836116790771484 (nan) | > loss: 0.011588448658585548 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13688498216281036) | > loader_time: 0.0045 (0.004674380443182336)  --> TIME: 2025-02-26 02:34:53 -- STEP: 15986/36938 -- GLOBAL_STEP: 1050250 | > loss_text_ce: 0.02769484929740429 (0.032351504902726064) | > loss_mel_ce: 2.7540299892425537 (nan) | > loss: 0.011038592085242271 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13689847440217354) | > loader_time: 0.0045 (0.0046742652811694475)  --> TIME: 2025-02-26 02:35:01 -- STEP: 16036/36938 -- GLOBAL_STEP: 1050300 | > loss_text_ce: 0.03403688222169876 (0.032352667336216785) | > loss_mel_ce: 2.9933784008026123 (nan) | > loss: 0.012013553641736507 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.1368834735985027) | > loader_time: 0.0048 (0.004674084601660386)  --> TIME: 2025-02-26 02:35:11 -- STEP: 16086/36938 -- GLOBAL_STEP: 1050350 | > loss_text_ce: 0.03809409588575363 (0.03235169979014748) | > loss_mel_ce: 2.7239990234375 (nan) | > loss: 0.010960686951875687 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13688022731978844) | > loader_time: 0.0046 (0.0046739625527641455)  --> TIME: 2025-02-26 02:35:20 -- STEP: 16136/36938 -- GLOBAL_STEP: 1050400 | > loss_text_ce: 0.03760124742984772 (0.03235329391501053) | > loss_mel_ce: 3.335069417953491 (nan) | > loss: 0.013383614830672741 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.1368849183523726) | > loader_time: 0.0045 (0.0046739599523889465)  --> TIME: 2025-02-26 02:35:30 -- STEP: 16186/36938 -- GLOBAL_STEP: 1050450 | > loss_text_ce: 0.024847805500030518 (0.03235445769451242) | > loss_mel_ce: 2.453933000564575 (nan) | > loss: 0.00983643252402544 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.1368895592949356) | > loader_time: 0.0046 (0.004674082042171356)  --> TIME: 2025-02-26 02:35:39 -- STEP: 16236/36938 -- GLOBAL_STEP: 1050500 | > loss_text_ce: 0.05015182867646217 (0.03236071060879248) | > loss_mel_ce: 3.5671868324279785 (nan) | > loss: 0.014354519546031952 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.1369006175893849) | > loader_time: 0.0046 (0.004674046049566802)  --> TIME: 2025-02-26 02:35:49 -- STEP: 16286/36938 -- GLOBAL_STEP: 1050550 | > loss_text_ce: 0.03159375116229057 (0.032361652266568626) | > loss_mel_ce: 2.8250584602355957 (nan) | > loss: 0.01133592240512371 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13690270770610674) | > loader_time: 0.0046 (0.004674047315851207)  --> TIME: 2025-02-26 02:35:58 -- STEP: 16336/36938 -- GLOBAL_STEP: 1050600 | > loss_text_ce: 0.03143996000289917 (0.0323633811813205) | > loss_mel_ce: 2.326676607131958 (nan) | > loss: 0.009357606060802937 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.1368972910431299) | > loader_time: 0.0046 (0.00467394012136861)  --> TIME: 2025-02-26 02:36:07 -- STEP: 16386/36938 -- GLOBAL_STEP: 1050650 | > loss_text_ce: 0.03700042515993118 (0.032363430678562136) | > loss_mel_ce: 0.7641357183456421 (nan) | > loss: 0.0031791117507964373 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13689644371911835) | > loader_time: 0.0045 (0.004674007906285773)  --> TIME: 2025-02-26 02:36:17 -- STEP: 16436/36938 -- GLOBAL_STEP: 1050700 | > loss_text_ce: 0.027721839025616646 (0.03236229761719417) | > loss_mel_ce: 2.1466782093048096 (nan) | > loss: 0.008628572337329388 (nan) | > current_lr: 5e-06 | > step_time: 0.1068 (0.13690887841612093) | > loader_time: 0.0045 (0.004674111761063896)  --> TIME: 2025-02-26 02:36:26 -- STEP: 16486/36938 -- GLOBAL_STEP: 1050750 | > loss_text_ce: 0.02426975406706333 (0.03236555854508476) | > loss_mel_ce: 0.9456571936607361 (nan) | > loss: 0.003848916618153453 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13691029522035214) | > loader_time: 0.0047 (0.004674110845875627)  --> TIME: 2025-02-26 02:36:36 -- STEP: 16536/36938 -- GLOBAL_STEP: 1050800 | > loss_text_ce: 0.0354078970849514 (0.03236521310228174) | > loss_mel_ce: 3.3220338821411133 (nan) | > loss: 0.013323182240128517 (nan) | > current_lr: 5e-06 | > step_time: 0.1355 (0.13692794207449344) | > loader_time: 0.0046 (0.004673995888631126)  --> TIME: 2025-02-26 02:36:45 -- STEP: 16586/36938 -- GLOBAL_STEP: 1050850 | > loss_text_ce: 0.03244313970208168 (0.03236558697301301) | > loss_mel_ce: 1.566619873046875 (nan) | > loss: 0.0063454885967075825 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13690699581414012) | > loader_time: 0.0046 (0.004673731193372284)  --> TIME: 2025-02-26 02:36:54 -- STEP: 16636/36938 -- GLOBAL_STEP: 1050900 | > loss_text_ce: 0.032762475311756134 (0.032362467780484044) | > loss_mel_ce: 3.062933921813965 (nan) | > loss: 0.012284510768949986 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13690175301296661) | > loader_time: 0.0046 (0.004673528453426081)  --> TIME: 2025-02-26 02:37:03 -- STEP: 16686/36938 -- GLOBAL_STEP: 1050950 | > loss_text_ce: 0.0202188603579998 (0.03235888585362762) | > loss_mel_ce: 2.3534276485443115 (nan) | > loss: 0.009419232606887817 (nan) | > current_lr: 5e-06 | > step_time: 0.1697 (0.13688942235278923) | > loader_time: 0.0047 (0.00467326894561464)  --> TIME: 2025-02-26 02:37:13 -- STEP: 16736/36938 -- GLOBAL_STEP: 1051000 | > loss_text_ce: 0.03173833340406418 (0.03236090554315986) | > loss_mel_ce: 3.014366388320923 (nan) | > loss: 0.012087717652320862 (nan) | > current_lr: 5e-06 | > step_time: 0.1506 (0.1368968513995472) | > loader_time: 0.0046 (0.00467317106903853)  --> TIME: 2025-02-26 02:37:22 -- STEP: 16786/36938 -- GLOBAL_STEP: 1051050 | > loss_text_ce: 0.029385201632976532 (0.03235830932689831) | > loss_mel_ce: 2.717653751373291 (nan) | > loss: 0.010900948196649551 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1368997355517599) | > loader_time: 0.0046 (0.004672953983917291)  --> TIME: 2025-02-26 02:37:31 -- STEP: 16836/36938 -- GLOBAL_STEP: 1051100 | > loss_text_ce: 0.03873845562338829 (0.032360872233916725) | > loss_mel_ce: 3.290959119796753 (nan) | > loss: 0.013213086873292923 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13689687972002001) | > loader_time: 0.0047 (0.004673023763703292)  --> TIME: 2025-02-26 02:37:41 -- STEP: 16886/36938 -- GLOBAL_STEP: 1051150 | > loss_text_ce: 0.03345071151852608 (0.032356828726648565) | > loss_mel_ce: 1.7745732069015503 (nan) | > loss: 0.007174698635935783 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.1369058593788184) | > loader_time: 0.0045 (0.004672881750138214)  --> TIME: 2025-02-26 02:37:50 -- STEP: 16936/36938 -- GLOBAL_STEP: 1051200 | > loss_text_ce: 0.030838655307888985 (0.03236083717218063) | > loss_mel_ce: 3.275569200515747 (nan) | > loss: 0.013120667077600956 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13690946747742058) | > loader_time: 0.0048 (0.004672739040643245)  --> TIME: 2025-02-26 02:38:00 -- STEP: 16986/36938 -- GLOBAL_STEP: 1051250 | > loss_text_ce: 0.02042091265320778 (0.03235854905722194) | > loss_mel_ce: 1.7832846641540527 (nan) | > loss: 0.007157562300562859 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13691633797275865) | > loader_time: 0.0045 (0.004672596216847446)  --> TIME: 2025-02-26 02:38:09 -- STEP: 17036/36938 -- GLOBAL_STEP: 1051300 | > loss_text_ce: 0.029056867584586143 (0.032356334353752784) | > loss_mel_ce: 0.8844528198242188 (nan) | > loss: 0.003625038545578718 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13692581881713306) | > loader_time: 0.0046 (0.004672589017125078)  --> TIME: 2025-02-26 02:38:19 -- STEP: 17086/36938 -- GLOBAL_STEP: 1051350 | > loss_text_ce: 0.03030233643949032 (0.032354923861879034) | > loss_mel_ce: 1.2819273471832275 (nan) | > loss: 0.005207260604947805 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13692238756997244) | > loader_time: 0.0046 (0.004672568700889314)  --> TIME: 2025-02-26 02:38:28 -- STEP: 17136/36938 -- GLOBAL_STEP: 1051400 | > loss_text_ce: 0.04145172983407974 (0.03236113326491646) | > loss_mel_ce: 1.6987398862838745 (nan) | > loss: 0.006905522663146257 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13693243450199144) | > loader_time: 0.0045 (0.004672511521619023)  --> TIME: 2025-02-26 02:38:38 -- STEP: 17186/36938 -- GLOBAL_STEP: 1051450 | > loss_text_ce: 0.026322148740291595 (0.03236544557863235) | > loss_mel_ce: 2.6060104370117188 (nan) | > loss: 0.010445764288306236 (nan) | > current_lr: 5e-06 | > step_time: 0.1455 (0.13692996336280996) | > loader_time: 0.0046 (0.004672340723572882)  --> TIME: 2025-02-26 02:38:47 -- STEP: 17236/36938 -- GLOBAL_STEP: 1051500 | > loss_text_ce: 0.033594079315662384 (0.0323657794251116) | > loss_mel_ce: 2.600935459136963 (nan) | > loss: 0.010454483330249786 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13693981370950326) | > loader_time: 0.0045 (0.00467228694425766)  --> TIME: 2025-02-26 02:38:56 -- STEP: 17286/36938 -- GLOBAL_STEP: 1051550 | > loss_text_ce: 0.03222210705280304 (0.03236626726617764) | > loss_mel_ce: 2.4648730754852295 (nan) | > loss: 0.009909108281135559 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13692602040411073) | > loader_time: 0.0046 (0.004672190912148061)  --> TIME: 2025-02-26 02:39:06 -- STEP: 17336/36938 -- GLOBAL_STEP: 1051600 | > loss_text_ce: 0.03133498504757881 (0.0323686524607301) | > loss_mel_ce: 0.1261444091796875 (nan) | > loss: 0.0006249182624742389 (nan) | > current_lr: 5e-06 | > step_time: 0.1261 (0.13693852330002382) | > loader_time: 0.0046 (0.004672099229758051)  --> TIME: 2025-02-26 02:39:15 -- STEP: 17386/36938 -- GLOBAL_STEP: 1051650 | > loss_text_ce: 0.0425603985786438 (0.03236653838319742) | > loss_mel_ce: 3.0928120613098145 (nan) | > loss: 0.01244195457547903 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13694080036062042) | > loader_time: 0.0049 (0.004671966578405579)  --> TIME: 2025-02-26 02:39:25 -- STEP: 17436/36938 -- GLOBAL_STEP: 1051700 | > loss_text_ce: 0.02696184813976288 (0.03236364095208367) | > loss_mel_ce: 2.1431217193603516 (nan) | > loss: 0.00861144345253706 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1369414398308666) | > loader_time: 0.0045 (0.0046717562678757815)  --> TIME: 2025-02-26 02:39:33 -- STEP: 17486/36938 -- GLOBAL_STEP: 1051750 | > loss_text_ce: 0.0337105467915535 (0.03235951761862012) | > loss_mel_ce: 2.8218579292297363 (nan) | > loss: 0.011331621557474136 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13692291259056694) | > loader_time: 0.0047 (0.004671494802348225)  --> TIME: 2025-02-26 02:39:43 -- STEP: 17536/36938 -- GLOBAL_STEP: 1051800 | > loss_text_ce: 0.027729002758860588 (0.032362040781490825) | > loss_mel_ce: 0.01988675445318222 (nan) | > loss: 0.00018895143875852227 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.1369242440147783) | > loader_time: 0.0046 (0.004671351657840457)  --> TIME: 2025-02-26 02:39:52 -- STEP: 17586/36938 -- GLOBAL_STEP: 1051850 | > loss_text_ce: 0.03231428563594818 (0.032361394585400834) | > loss_mel_ce: 2.8392958641052246 (nan) | > loss: 0.011395279318094254 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13691615666543888) | > loader_time: 0.0046 (0.004671143303277563)  --> TIME: 2025-02-26 02:40:01 -- STEP: 17636/36938 -- GLOBAL_STEP: 1051900 | > loss_text_ce: 0.03959042578935623 (0.03236224744692841) | > loss_mel_ce: 3.0103251934051514 (nan) | > loss: 0.012102840468287468 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.1369154300108936) | > loader_time: 0.0047 (0.0046709560434086625)  --> TIME: 2025-02-26 02:40:11 -- STEP: 17686/36938 -- GLOBAL_STEP: 1051950 | > loss_text_ce: 0.026740171015262604 (0.03235765768285209) | > loss_mel_ce: 3.412435293197632 (nan) | > loss: 0.01364752184599638 (nan) | > current_lr: 5e-06 | > step_time: 0.1958 (0.13691699936232837) | > loader_time: 0.0048 (0.004670760554182412)  --> TIME: 2025-02-26 02:40:20 -- STEP: 17736/36938 -- GLOBAL_STEP: 1052000 | > loss_text_ce: 0.04258044809103012 (0.03235848392775802) | > loss_mel_ce: 2.7223236560821533 (nan) | > loss: 0.01097184233367443 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13692118389465693) | > loader_time: 0.0045 (0.0046705780907907515)  --> TIME: 2025-02-26 02:40:29 -- STEP: 17786/36938 -- GLOBAL_STEP: 1052050 | > loss_text_ce: 0.034747883677482605 (0.032359285283564465) | > loss_mel_ce: 3.0356650352478027 (nan) | > loss: 0.012184178456664085 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.13692988619337454) | > loader_time: 0.0048 (0.004670410433462203)  --> TIME: 2025-02-26 02:40:39 -- STEP: 17836/36938 -- GLOBAL_STEP: 1052100 | > loss_text_ce: 0.030177844688296318 (0.032364124111391813) | > loss_mel_ce: 2.9168906211853027 (nan) | > loss: 0.011694717220962048 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13694318177909198) | > loader_time: 0.0047 (0.004670250800779301)  --> TIME: 2025-02-26 02:40:48 -- STEP: 17886/36938 -- GLOBAL_STEP: 1052150 | > loss_text_ce: 0.029271580278873444 (0.032367539178339746) | > loss_mel_ce: 3.333066940307617 (nan) | > loss: 0.013342614285647869 (nan) | > current_lr: 5e-06 | > step_time: 0.1885 (0.13694842117925612) | > loader_time: 0.0048 (0.004670044899413372)  --> TIME: 2025-02-26 02:40:58 -- STEP: 17936/36938 -- GLOBAL_STEP: 1052200 | > loss_text_ce: 0.0266279187053442 (0.03236417092210939) | > loss_mel_ce: 2.7555789947509766 (nan) | > loss: 0.011040505021810532 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13694214786300793) | > loader_time: 0.0045 (0.004669786430064531)  --> TIME: 2025-02-26 02:41:07 -- STEP: 17986/36938 -- GLOBAL_STEP: 1052250 | > loss_text_ce: 0.03871738910675049 (0.03236382689219717) | > loss_mel_ce: 2.4623985290527344 (nan) | > loss: 0.009925063699483871 (nan) | > current_lr: 5e-06 | > step_time: 0.1599 (0.1369259747039972) | > loader_time: 0.0049 (0.00466956620909382)  --> TIME: 2025-02-26 02:41:16 -- STEP: 18036/36938 -- GLOBAL_STEP: 1052300 | > loss_text_ce: 0.029318932443857193 (0.03236636163375203) | > loss_mel_ce: 2.943578004837036 (nan) | > loss: 0.01179721113294363 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13693181474078328) | > loader_time: 0.0045 (0.004669465453430664)  --> TIME: 2025-02-26 02:41:25 -- STEP: 18086/36938 -- GLOBAL_STEP: 1052350 | > loss_text_ce: 0.03786524757742882 (0.03236806927230096) | > loss_mel_ce: 3.0163815021514893 (nan) | > loss: 0.012120027095079422 (nan) | > current_lr: 5e-06 | > step_time: 0.151 (0.1369310374098078) | > loader_time: 0.0046 (0.004669261126342652)  --> TIME: 2025-02-26 02:41:34 -- STEP: 18136/36938 -- GLOBAL_STEP: 1052400 | > loss_text_ce: 0.023591965436935425 (0.0323719131151982) | > loss_mel_ce: 3.3684895038604736 (nan) | > loss: 0.013460641726851463 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13692284657085518) | > loader_time: 0.0045 (0.004669027531993608)  --> TIME: 2025-02-26 02:41:44 -- STEP: 18186/36938 -- GLOBAL_STEP: 1052450 | > loss_text_ce: 0.03660563752055168 (0.03237385054245018) | > loss_mel_ce: 3.0446410179138184 (nan) | > loss: 0.012227170169353485 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.13693272506366186) | > loader_time: 0.0048 (0.0046693628854016294)  --> TIME: 2025-02-26 02:41:54 -- STEP: 18236/36938 -- GLOBAL_STEP: 1052500 | > loss_text_ce: 0.029112670570611954 (0.03237292762657) | > loss_mel_ce: 2.946712017059326 (nan) | > loss: 0.011808828450739384 (nan) | > current_lr: 5e-06 | > step_time: 0.1832 (0.13694373092831374) | > loader_time: 0.0048 (0.0046694214000438244)  --> TIME: 2025-02-26 02:42:03 -- STEP: 18286/36938 -- GLOBAL_STEP: 1052550 | > loss_text_ce: 0.031791623681783676 (0.03237010581120621) | > loss_mel_ce: 2.568464756011963 (nan) | > loss: 0.01031847856938839 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13694998722009766) | > loader_time: 0.0048 (0.004669364753234147)  --> TIME: 2025-02-26 02:42:12 -- STEP: 18336/36938 -- GLOBAL_STEP: 1052600 | > loss_text_ce: 0.03049384616315365 (0.03236957495440087) | > loss_mel_ce: 3.147157907485962 (nan) | > loss: 0.012609729543328285 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.13693425212751187) | > loader_time: 0.0047 (0.0046693096116159055)  --> TIME: 2025-02-26 02:42:21 -- STEP: 18386/36938 -- GLOBAL_STEP: 1052650 | > loss_text_ce: 0.032056424766778946 (0.03237001218174632) | > loss_mel_ce: 3.161909818649292 (nan) | > loss: 0.012674469500780106 (nan) | > current_lr: 5e-06 | > step_time: 0.1453 (0.13692879086802337) | > loader_time: 0.0047 (0.004669297821669834)  --> TIME: 2025-02-26 02:42:31 -- STEP: 18436/36938 -- GLOBAL_STEP: 1052700 | > loss_text_ce: 0.03132476285099983 (0.03236870085217391) | > loss_mel_ce: 2.520914316177368 (nan) | > loss: 0.010127933695912361 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.1369362102332883) | > loader_time: 0.0047 (0.004669371707032301)  --> TIME: 2025-02-26 02:42:40 -- STEP: 18486/36938 -- GLOBAL_STEP: 1052750 | > loss_text_ce: 0.036207616329193115 (0.032369281240418206) | > loss_mel_ce: 2.4140625 (nan) | > loss: 0.009723294526338577 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13693157201152334) | > loader_time: 0.0046 (0.004669362353669796)  --> TIME: 2025-02-26 02:42:49 -- STEP: 18536/36938 -- GLOBAL_STEP: 1052800 | > loss_text_ce: 0.030688991770148277 (0.03236854082399702) | > loss_mel_ce: 2.8522772789001465 (nan) | > loss: 0.011440343223512173 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.1369338562944412) | > loader_time: 0.0047 (0.004669257302607331)  --> TIME: 2025-02-26 02:42:59 -- STEP: 18586/36938 -- GLOBAL_STEP: 1052850 | > loss_text_ce: 0.022550396621227264 (0.03237062821717526) | > loss_mel_ce: 0.8135834336280823 (nan) | > loss: 0.003317991504445672 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13693233753936354) | > loader_time: 0.0045 (0.004669252694467661)  --> TIME: 2025-02-26 02:43:08 -- STEP: 18636/36938 -- GLOBAL_STEP: 1052900 | > loss_text_ce: 0.028915144503116608 (0.03237186939944118) | > loss_mel_ce: 2.6840715408325195 (nan) | > loss: 0.010765820741653442 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13694192683090442) | > loader_time: 0.0046 (0.004669356561060179)  --> TIME: 2025-02-26 02:43:18 -- STEP: 18686/36938 -- GLOBAL_STEP: 1052950 | > loss_text_ce: 0.025469191372394562 (0.032368902035823495) | > loss_mel_ce: 1.3949145078659058 (nan) | > loss: 0.005636443383991718 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13694708724738072) | > loader_time: 0.0045 (0.004669374091638214)  --> TIME: 2025-02-26 02:43:27 -- STEP: 18736/36938 -- GLOBAL_STEP: 1053000 | > loss_text_ce: 0.032473232597112656 (0.03236706054994729) | > loss_mel_ce: 2.65666127204895 (nan) | > loss: 0.010671169497072697 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.13695963724722446) | > loader_time: 0.0047 (0.004669781338445746)  --> TIME: 2025-02-26 02:43:37 -- STEP: 18786/36938 -- GLOBAL_STEP: 1053050 | > loss_text_ce: 0.027073707431554794 (0.03236747676221875) | > loss_mel_ce: 3.1277596950531006 (nan) | > loss: 0.01251918077468872 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13696086589462478) | > loader_time: 0.0045 (0.004669771056884406)  --> TIME: 2025-02-26 02:43:46 -- STEP: 18836/36938 -- GLOBAL_STEP: 1053100 | > loss_text_ce: 0.026624571532011032 (0.03236620593544098) | > loss_mel_ce: 2.490285873413086 (nan) | > loss: 0.009987740777432919 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.13696134666344564) | > loader_time: 0.0047 (0.004669787436185745)  --> TIME: 2025-02-26 02:43:55 -- STEP: 18886/36938 -- GLOBAL_STEP: 1053150 | > loss_text_ce: 0.026839351281523705 (0.032369959578271094) | > loss_mel_ce: 2.5809075832366943 (nan) | > loss: 0.010348202660679817 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.1369433648301233) | > loader_time: 0.0049 (0.004669701524120644)  --> TIME: 2025-02-26 02:44:04 -- STEP: 18936/36938 -- GLOBAL_STEP: 1053200 | > loss_text_ce: 0.03103114850819111 (0.03236722367990561) | > loss_mel_ce: 2.1900551319122314 (nan) | > loss: 0.008813834749162197 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13693495534823225) | > loader_time: 0.0047 (0.004669663872857792)  --> TIME: 2025-02-26 02:44:14 -- STEP: 18986/36938 -- GLOBAL_STEP: 1053250 | > loss_text_ce: 0.03378135710954666 (0.03236461001038537) | > loss_mel_ce: 3.918750047683716 (nan) | > loss: 0.015684649348258972 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.1369445049686135) | > loader_time: 0.0046 (0.0046697041891176775)  --> TIME: 2025-02-26 02:44:23 -- STEP: 19036/36938 -- GLOBAL_STEP: 1053300 | > loss_text_ce: 0.02757982350885868 (0.032366207663083645) | > loss_mel_ce: 2.6968259811401367 (nan) | > loss: 0.010811134241521358 (nan) | > current_lr: 5e-06 | > step_time: 0.1537 (0.13694947311722497) | > loader_time: 0.0046 (0.004669725231123342)  --> TIME: 2025-02-26 02:44:32 -- STEP: 19086/36938 -- GLOBAL_STEP: 1053350 | > loss_text_ce: 0.01772395521402359 (0.032365467798071715) | > loss_mel_ce: 2.092480421066284 (nan) | > loss: 0.008373827673494816 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13694418539706704) | > loader_time: 0.0045 (0.004669714296287559)  --> TIME: 2025-02-26 02:44:41 -- STEP: 19136/36938 -- GLOBAL_STEP: 1053400 | > loss_text_ce: 0.03173040226101875 (0.032365554683936336) | > loss_mel_ce: 2.8206467628479004 (nan) | > loss: 0.011318957433104515 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13692994127953415) | > loader_time: 0.0046 (0.004669679409783801)  --> TIME: 2025-02-26 02:44:51 -- STEP: 19186/36938 -- GLOBAL_STEP: 1053450 | > loss_text_ce: 0.028217332437634468 (0.03236530407263048) | > loss_mel_ce: 2.4113383293151855 (nan) | > loss: 0.009680776856839657 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13692360708192444) | > loader_time: 0.0047 (0.004669651291261764)  --> TIME: 2025-02-26 02:45:00 -- STEP: 19236/36938 -- GLOBAL_STEP: 1053500 | > loss_text_ce: 0.03284183144569397 (0.0323661380231223) | > loss_mel_ce: 1.9712088108062744 (nan) | > loss: 0.007952583022415638 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13693624273807148) | > loader_time: 0.0047 (0.004669698528104157)  --> TIME: 2025-02-26 02:45:10 -- STEP: 19286/36938 -- GLOBAL_STEP: 1053550 | > loss_text_ce: 0.037796471267938614 (0.03236553913027153) | > loss_mel_ce: 3.269716262817383 (nan) | > loss: 0.013125051744282246 (nan) | > current_lr: 5e-06 | > step_time: 0.1619 (0.13693694823339447) | > loader_time: 0.0047 (0.004669847669386811)  --> TIME: 2025-02-26 02:45:19 -- STEP: 19336/36938 -- GLOBAL_STEP: 1053600 | > loss_text_ce: 0.031140677630901337 (0.03236451867849738) | > loss_mel_ce: 0.009124755859375 (nan) | > loss: 0.00015978347801137716 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1369519117798853) | > loader_time: 0.0045 (0.00466990493080759)  --> TIME: 2025-02-26 02:45:28 -- STEP: 19386/36938 -- GLOBAL_STEP: 1053650 | > loss_text_ce: 0.023484889417886734 (0.032369161346131874) | > loss_mel_ce: 2.089200258255005 (nan) | > loss: 0.008383671753108501 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13694106648239693) | > loader_time: 0.0046 (0.0046697702494418085)  --> TIME: 2025-02-26 02:45:38 -- STEP: 19436/36938 -- GLOBAL_STEP: 1053700 | > loss_text_ce: 0.028177065774798393 (0.03236553718870298) | > loss_mel_ce: 2.6858723163604736 (nan) | > loss: 0.010770037770271301 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13695051320959886) | > loader_time: 0.0046 (0.0046697653451312085)  --> TIME: 2025-02-26 02:45:47 -- STEP: 19486/36938 -- GLOBAL_STEP: 1053750 | > loss_text_ce: 0.022736165672540665 (0.032364455086849035) | > loss_mel_ce: 2.921461820602417 (nan) | > loss: 0.011683325283229351 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.13695153240108812) | > loader_time: 0.0047 (0.0046698650295308635)  --> TIME: 2025-02-26 02:45:56 -- STEP: 19536/36938 -- GLOBAL_STEP: 1053800 | > loss_text_ce: 0.0231489147990942 (0.03236440615923949) | > loss_mel_ce: 3.0391626358032227 (nan) | > loss: 0.012152031064033508 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1369440669558469) | > loader_time: 0.0046 (0.004669949607610878)  --> TIME: 2025-02-26 02:46:06 -- STEP: 19586/36938 -- GLOBAL_STEP: 1053850 | > loss_text_ce: 0.031100144609808922 (0.03236768461341036) | > loss_mel_ce: 3.5080983638763428 (nan) | > loss: 0.014044438488781452 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13694218925566531) | > loader_time: 0.0046 (0.0046699163096437565)  --> TIME: 2025-02-26 02:46:15 -- STEP: 19636/36938 -- GLOBAL_STEP: 1053900 | > loss_text_ce: 0.02467459812760353 (0.032361953160197125) | > loss_mel_ce: 2.792271852493286 (nan) | > loss: 0.011178359389305115 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13694538482946636) | > loader_time: 0.0049 (0.004669947120560024)  --> TIME: 2025-02-26 02:46:24 -- STEP: 19686/36938 -- GLOBAL_STEP: 1053950 | > loss_text_ce: 0.02743174508213997 (0.03235803873966164) | > loss_mel_ce: 2.5521183013916016 (nan) | > loss: 0.010236309841275215 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.1369288280510904) | > loader_time: 0.0046 (0.0046698431725014)  --> TIME: 2025-02-26 02:46:34 -- STEP: 19736/36938 -- GLOBAL_STEP: 1054000 | > loss_text_ce: 0.032353032380342484 (0.032356974004001844) | > loss_mel_ce: 2.5636401176452637 (nan) | > loss: 0.010301560163497925 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13693814082913128) | > loader_time: 0.0047 (0.0046702371611979585)  --> TIME: 2025-02-26 02:46:43 -- STEP: 19786/36938 -- GLOBAL_STEP: 1054050 | > loss_text_ce: 0.03360726311802864 (0.03235902053333292) | > loss_mel_ce: 3.2143402099609375 (nan) | > loss: 0.012888681143522263 (nan) | > current_lr: 5e-06 | > step_time: 0.1532 (0.13693153455719495) | > loader_time: 0.0046 (0.004670187121489315)  --> TIME: 2025-02-26 02:46:52 -- STEP: 19836/36938 -- GLOBAL_STEP: 1054100 | > loss_text_ce: 0.028995858505368233 (0.03235880042294634) | > loss_mel_ce: 3.0756754875183105 (nan) | > loss: 0.012320124544203281 (nan) | > current_lr: 5e-06 | > step_time: 0.1762 (0.136932378287373) | > loader_time: 0.0049 (0.004670189258239259)  --> TIME: 2025-02-26 02:47:02 -- STEP: 19886/36938 -- GLOBAL_STEP: 1054150 | > loss_text_ce: 0.018938584253191948 (0.03235637006782317) | > loss_mel_ce: 2.6873605251312256 (nan) | > loss: 0.010739282704889774 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1369305457026271) | > loader_time: 0.0046 (0.004670126929940616)  --> TIME: 2025-02-26 02:47:11 -- STEP: 19936/36938 -- GLOBAL_STEP: 1054200 | > loss_text_ce: 0.03197302669286728 (0.0323548416067423) | > loss_mel_ce: 0.644268810749054 (nan) | > loss: 0.002683499362319708 (nan) | > current_lr: 5e-06 | > step_time: 0.1651 (0.13694060725394733) | > loader_time: 0.0048 (0.004670189098600004)  --> TIME: 2025-02-26 02:47:21 -- STEP: 19986/36938 -- GLOBAL_STEP: 1054250 | > loss_text_ce: 0.027050437405705452 (0.03235538479762352) | > loss_mel_ce: 2.5031514167785645 (nan) | > loss: 0.010040484368801117 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13694724958603696) | > loader_time: 0.0046 (0.004670224580561483)  --> TIME: 2025-02-26 02:47:30 -- STEP: 20036/36938 -- GLOBAL_STEP: 1054300 | > loss_text_ce: 0.025652548298239708 (0.03235667590790027) | > loss_mel_ce: 2.5265870094299316 (nan) | > loss: 0.01012793555855751 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1369450546043993) | > loader_time: 0.0046 (0.004670235907919594)  --> TIME: 2025-02-26 02:47:40 -- STEP: 20086/36938 -- GLOBAL_STEP: 1054350 | > loss_text_ce: 0.03164663538336754 (0.03235678231410578) | > loss_mel_ce: 2.894231081008911 (nan) | > loss: 0.011610627174377441 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13695329696572175) | > loader_time: 0.0048 (0.0046702647700580715)  --> TIME: 2025-02-26 02:47:49 -- STEP: 20136/36938 -- GLOBAL_STEP: 1054400 | > loss_text_ce: 0.03345937281847 (0.03235300290538903) | > loss_mel_ce: 0.0657145157456398 (nan) | > loss: 0.0003935472050216049 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1369515840714547) | > loader_time: 0.0045 (0.004670148798999765)  --> TIME: 2025-02-26 02:47:58 -- STEP: 20186/36938 -- GLOBAL_STEP: 1054450 | > loss_text_ce: 0.039981864392757416 (0.03235555601039678) | > loss_mel_ce: 2.515421152114868 (nan) | > loss: 0.010140488855540752 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13693669667202) | > loader_time: 0.0047 (0.004670162521244367)  --> TIME: 2025-02-26 02:48:07 -- STEP: 20236/36938 -- GLOBAL_STEP: 1054500 | > loss_text_ce: 0.025849999859929085 (0.03235746685142682) | > loss_mel_ce: 2.2252554893493652 (nan) | > loss: 0.008932959288358688 (nan) | > current_lr: 5e-06 | > step_time: 0.1778 (0.1369250739815451) | > loader_time: 0.0048 (0.004670171309752187)  --> TIME: 2025-02-26 02:48:16 -- STEP: 20286/36938 -- GLOBAL_STEP: 1054550 | > loss_text_ce: 0.036029692739248276 (0.03235744769867415) | > loss_mel_ce: 2.2552733421325684 (nan) | > loss: 0.009092472493648529 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1369237391939071) | > loader_time: 0.0046 (0.004670050009507734)  --> TIME: 2025-02-26 02:48:26 -- STEP: 20336/36938 -- GLOBAL_STEP: 1054600 | > loss_text_ce: 0.03715579956769943 (0.03235716505200009) | > loss_mel_ce: 2.526430606842041 (nan) | > loss: 0.01017296314239502 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.136918288202009) | > loader_time: 0.0046 (0.004670006824608948)  --> TIME: 2025-02-26 02:48:35 -- STEP: 20386/36938 -- GLOBAL_STEP: 1054650 | > loss_text_ce: 0.02289513312280178 (0.0323540890235485) | > loss_mel_ce: 0.009777069091796875 (nan) | > loss: 0.00012965161295142025 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13690563936082978) | > loader_time: 0.0045 (0.004669922953392041)  --> TIME: 2025-02-26 02:48:44 -- STEP: 20436/36938 -- GLOBAL_STEP: 1054700 | > loss_text_ce: 0.028656035661697388 (0.03235444112408875) | > loss_mel_ce: 0.01324462890625 (nan) | > loss: 0.00016627248260192573 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13689544278363516) | > loader_time: 0.0045 (0.004669879520679074)  --> TIME: 2025-02-26 02:48:53 -- STEP: 20486/36938 -- GLOBAL_STEP: 1054750 | > loss_text_ce: 0.029747258871793747 (0.03235342141568522) | > loss_mel_ce: 2.930108070373535 (nan) | > loss: 0.011745458468794823 (nan) | > current_lr: 5e-06 | > step_time: 0.1507 (0.1368972936389303) | > loader_time: 0.0047 (0.004670024348762381)  --> TIME: 2025-02-26 02:49:02 -- STEP: 20536/36938 -- GLOBAL_STEP: 1054800 | > loss_text_ce: 0.03137034922838211 (0.03235440141128476) | > loss_mel_ce: 2.6537277698516846 (nan) | > loss: 0.010655151680111885 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13688718344991674) | > loader_time: 0.0048 (0.004670028039626112)  --> TIME: 2025-02-26 02:49:11 -- STEP: 20586/36938 -- GLOBAL_STEP: 1054850 | > loss_text_ce: 0.03943731263279915 (0.03235357745278137) | > loss_mel_ce: 2.624173879623413 (nan) | > loss: 0.010569886304438114 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1368728235652062) | > loader_time: 0.0046 (0.004669901025917179)  --> TIME: 2025-02-26 02:49:20 -- STEP: 20636/36938 -- GLOBAL_STEP: 1054900 | > loss_text_ce: 0.028257928788661957 (0.03235750467817781) | > loss_mel_ce: 2.2060546875 (nan) | > loss: 0.008866320364177227 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13685814202998808) | > loader_time: 0.0045 (0.004669901023287242)  --> TIME: 2025-02-26 02:49:29 -- STEP: 20686/36938 -- GLOBAL_STEP: 1054950 | > loss_text_ce: 0.035463884472846985 (0.032358157062363) | > loss_mel_ce: 2.8371784687042236 (nan) | > loss: 0.011399374343454838 (nan) | > current_lr: 5e-06 | > step_time: 0.1837 (0.1368551624359513) | > loader_time: 0.0048 (0.004669912534745267)  --> TIME: 2025-02-26 02:49:38 -- STEP: 20736/36938 -- GLOBAL_STEP: 1055000 | > loss_text_ce: 0.017931317910552025 (0.03235717078783688) | > loss_mel_ce: 0.0066372789442539215 (nan) | > loss: 9.749443415785208e-05 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.1368460505855866) | > loader_time: 0.0045 (0.004669833307464909)  --> TIME: 2025-02-26 02:49:47 -- STEP: 20786/36938 -- GLOBAL_STEP: 1055050 | > loss_text_ce: 0.027742203325033188 (0.03235904795827309) | > loss_mel_ce: 1.5439108610153198 (nan) | > loss: 0.006236718501895666 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.1368380850024175) | > loader_time: 0.0045 (0.004669880426550548)  --> TIME: 2025-02-26 02:49:57 -- STEP: 20836/36938 -- GLOBAL_STEP: 1055100 | > loss_text_ce: 0.03004641644656658 (0.032358036142766666) | > loss_mel_ce: 2.5118627548217773 (nan) | > loss: 0.01008694153279066 (nan) | > current_lr: 5e-06 | > step_time: 0.1816 (0.13682972622778314) | > loader_time: 0.0048 (0.004669897900499144)  --> TIME: 2025-02-26 02:50:06 -- STEP: 20886/36938 -- GLOBAL_STEP: 1055150 | > loss_text_ce: 0.03687247261404991 (0.03236261546356017) | > loss_mel_ce: 0.8568115234375 (nan) | > loss: 0.0035463650710880756 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.1368240906248657) | > loader_time: 0.0045 (0.0046699334295912465)  --> TIME: 2025-02-26 02:50:15 -- STEP: 20936/36938 -- GLOBAL_STEP: 1055200 | > loss_text_ce: 0.032365862280130386 (0.0323625455990706) | > loss_mel_ce: 1.4482945203781128 (nan) | > loss: 0.005875636823475361 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1368239960189453) | > loader_time: 0.0045 (0.004669932586617428)  --> TIME: 2025-02-26 02:50:24 -- STEP: 20986/36938 -- GLOBAL_STEP: 1055250 | > loss_text_ce: 0.03501659259200096 (0.03236094297367058) | > loss_mel_ce: 2.780388832092285 (nan) | > loss: 0.01117224432528019 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13682641465432463) | > loader_time: 0.0047 (0.004669922295441937)  --> TIME: 2025-02-26 02:50:34 -- STEP: 21036/36938 -- GLOBAL_STEP: 1055300 | > loss_text_ce: 0.025499608367681503 (0.03236080358223894) | > loss_mel_ce: 1.177198052406311 (nan) | > loss: 0.004772610031068325 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13682663508707446) | > loader_time: 0.0045 (0.0046697440970689605)  --> TIME: 2025-02-26 02:50:43 -- STEP: 21086/36938 -- GLOBAL_STEP: 1055350 | > loss_text_ce: 0.03459945321083069 (0.03236255183030476) | > loss_mel_ce: 2.674628257751465 (nan) | > loss: 0.010750904679298401 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.13683412063283565) | > loader_time: 0.0046 (0.00466962916953031)  --> TIME: 2025-02-26 02:50:52 -- STEP: 21136/36938 -- GLOBAL_STEP: 1055400 | > loss_text_ce: 0.029683558270335197 (0.03236178148352456) | > loss_mel_ce: 1.5982177257537842 (nan) | > loss: 0.006459926255047321 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.1368210577292489) | > loader_time: 0.0045 (0.004669436196501185)  --> TIME: 2025-02-26 02:51:02 -- STEP: 21186/36938 -- GLOBAL_STEP: 1055450 | > loss_text_ce: 0.0319080576300621 (0.032359136535857216) | > loss_mel_ce: 2.579859972000122 (nan) | > loss: 0.010364159010350704 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1368335263199959) | > loader_time: 0.0045 (0.004669361137909411)  --> TIME: 2025-02-26 02:51:11 -- STEP: 21236/36938 -- GLOBAL_STEP: 1055500 | > loss_text_ce: 0.030956873670220375 (0.032358557801047405) | > loss_mel_ce: 2.6438968181610107 (nan) | > loss: 0.01061449944972992 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13683709595966465) | > loader_time: 0.0045 (0.004669262058745212)  --> TIME: 2025-02-26 02:51:20 -- STEP: 21286/36938 -- GLOBAL_STEP: 1055550 | > loss_text_ce: 0.02856101654469967 (0.03235768937080242) | > loss_mel_ce: 2.798470973968506 (nan) | > loss: 0.01121838204562664 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.13682925854591327) | > loader_time: 0.0048 (0.0046691795852884695)  --> TIME: 2025-02-26 02:51:29 -- STEP: 21336/36938 -- GLOBAL_STEP: 1055600 | > loss_text_ce: 0.0354301892220974 (0.03235793780457938) | > loss_mel_ce: 3.0693554878234863 (nan) | > loss: 0.012320579029619694 (nan) | > current_lr: 5e-06 | > step_time: 0.1955 (0.1368234767912038) | > loader_time: 0.0049 (0.004669015321295302)  --> TIME: 2025-02-26 02:51:38 -- STEP: 21386/36938 -- GLOBAL_STEP: 1055650 | > loss_text_ce: 0.03438867628574371 (0.03235757984147824) | > loss_mel_ce: 2.989443302154541 (nan) | > loss: 0.011999334208667278 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13681086499017947) | > loader_time: 0.0047 (0.004668814924360146)  --> TIME: 2025-02-26 02:51:48 -- STEP: 21436/36938 -- GLOBAL_STEP: 1055700 | > loss_text_ce: 0.02925366535782814 (0.03235527938939141) | > loss_mel_ce: 3.19732666015625 (nan) | > loss: 0.012803890742361546 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13680554926673555) | > loader_time: 0.0048 (0.004668705097261827)  --> TIME: 2025-02-26 02:51:57 -- STEP: 21486/36938 -- GLOBAL_STEP: 1055750 | > loss_text_ce: 0.028668109327554703 (0.032353276314643395) | > loss_mel_ce: 3.0655529499053955 (nan) | > loss: 0.012278655543923378 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13679937017837335) | > loader_time: 0.0047 (0.004668603371299965)  --> TIME: 2025-02-26 02:52:06 -- STEP: 21536/36938 -- GLOBAL_STEP: 1055800 | > loss_text_ce: 0.029801487922668457 (0.032352977039906215) | > loss_mel_ce: 3.398627281188965 (nan) | > loss: 0.013604876585304737 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13680164323342064) | > loader_time: 0.0047 (0.00466865551330004)  --> TIME: 2025-02-26 02:52:16 -- STEP: 21586/36938 -- GLOBAL_STEP: 1055850 | > loss_text_ce: 0.042194999754428864 (0.03235397585101181) | > loss_mel_ce: 2.7126944065093994 (nan) | > loss: 0.010932101868093014 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13682297908993707) | > loader_time: 0.0046 (0.004668693673695722)  --> TIME: 2025-02-26 02:52:25 -- STEP: 21636/36938 -- GLOBAL_STEP: 1055900 | > loss_text_ce: 0.035047128796577454 (0.032354045375636825) | > loss_mel_ce: 2.774848461151123 (nan) | > loss: 0.011150379665195942 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.13681738210002212) | > loader_time: 0.0046 (0.00466862598041174)  --> TIME: 2025-02-26 02:52:35 -- STEP: 21686/36938 -- GLOBAL_STEP: 1055950 | > loss_text_ce: 0.024791192263364792 (0.03235443584532658) | > loss_mel_ce: 0.6683344841003418 (nan) | > loss: 0.002750498941168189 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13681576628642433) | > loader_time: 0.0046 (0.004668742882790918)  --> TIME: 2025-02-26 02:52:44 -- STEP: 21736/36938 -- GLOBAL_STEP: 1056000 | > loss_text_ce: 0.027722742408514023 (0.03235371481184361) | > loss_mel_ce: 2.713070869445801 (nan) | > loss: 0.010876165702939034 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.13681377498463945) | > loader_time: 0.0046 (0.004668620565510587)  --> TIME: 2025-02-26 02:52:53 -- STEP: 21786/36938 -- GLOBAL_STEP: 1056050 | > loss_text_ce: 0.047584570944309235 (0.03235520782924005) | > loss_mel_ce: 2.877727508544922 (nan) | > loss: 0.011608381755650043 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13681418932054487) | > loader_time: 0.0045 (0.00466861799708149)  --> TIME: 2025-02-26 02:53:02 -- STEP: 21836/36938 -- GLOBAL_STEP: 1056100 | > loss_text_ce: 0.02845233865082264 (0.032354355399764875) | > loss_mel_ce: 2.6223456859588623 (nan) | > loss: 0.010519040748476982 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13680341536799248) | > loader_time: 0.0045 (0.0046684756932360615)  --> TIME: 2025-02-26 02:53:12 -- STEP: 21886/36938 -- GLOBAL_STEP: 1056150 | > loss_text_ce: 0.04183647036552429 (0.03235615735928387) | > loss_mel_ce: 2.637481689453125 (nan) | > loss: 0.01063221599906683 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13680397753936535) | > loader_time: 0.0048 (0.004668454065909369)  --> TIME: 2025-02-26 02:53:21 -- STEP: 21936/36938 -- GLOBAL_STEP: 1056200 | > loss_text_ce: 0.03998803719878197 (0.03235767425733694) | > loss_mel_ce: 2.175732135772705 (nan) | > loss: 0.008792540989816189 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13679371758980371) | > loader_time: 0.0046 (0.00466829103592273)  --> TIME: 2025-02-26 02:53:30 -- STEP: 21986/36938 -- GLOBAL_STEP: 1056250 | > loss_text_ce: 0.0309748612344265 (0.03235609632440031) | > loss_mel_ce: 1.5579882860183716 (nan) | > loss: 0.006305409595370293 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1367992587142901) | > loader_time: 0.0046 (0.004668261403437993)  --> TIME: 2025-02-26 02:53:40 -- STEP: 22036/36938 -- GLOBAL_STEP: 1056300 | > loss_text_ce: 0.032596200704574585 (0.03235500636236682) | > loss_mel_ce: 2.005126953125 (nan) | > loss: 0.008086202666163445 (nan) | > current_lr: 5e-06 | > step_time: 0.1877 (0.1368106844464594) | > loader_time: 0.0049 (0.0046683134736632)  --> TIME: 2025-02-26 02:53:49 -- STEP: 22086/36938 -- GLOBAL_STEP: 1056350 | > loss_text_ce: 0.046330321580171585 (0.032357757614518465) | > loss_mel_ce: 3.046034097671509 (nan) | > loss: 0.012271287851035595 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13680710647240776) | > loader_time: 0.0046 (0.004668335859341349)  --> TIME: 2025-02-26 02:53:58 -- STEP: 22136/36938 -- GLOBAL_STEP: 1056400 | > loss_text_ce: 0.026558227837085724 (0.03235785334421849) | > loss_mel_ce: 2.68411922454834 (nan) | > loss: 0.010756656527519226 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1367956615868666) | > loader_time: 0.0045 (0.004668168279294873)  --> TIME: 2025-02-26 02:54:08 -- STEP: 22186/36938 -- GLOBAL_STEP: 1056450 | > loss_text_ce: 0.03290358930826187 (0.032357326800245896) | > loss_mel_ce: 2.6940736770629883 (nan) | > loss: 0.010821339674293995 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1368076828810721) | > loader_time: 0.0045 (0.004668201712870784)  --> TIME: 2025-02-26 02:54:17 -- STEP: 22236/36938 -- GLOBAL_STEP: 1056500 | > loss_text_ce: 0.03058669902384281 (0.0323593853629407) | > loss_mel_ce: 3.136295795440674 (nan) | > loss: 0.01256699487566948 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.1368113438159715) | > loader_time: 0.0047 (0.0046681303368088034)  --> TIME: 2025-02-26 02:54:26 -- STEP: 22286/36938 -- GLOBAL_STEP: 1056550 | > loss_text_ce: 0.02722295932471752 (0.032361030559473675) | > loss_mel_ce: 2.712911367416382 (nan) | > loss: 0.010873549617826939 (nan) | > current_lr: 5e-06 | > step_time: 0.1529 (0.13680967743923414) | > loader_time: 0.0047 (0.004668295378104279)  --> TIME: 2025-02-26 02:54:36 -- STEP: 22336/36938 -- GLOBAL_STEP: 1056600 | > loss_text_ce: 0.02795012667775154 (0.03236128083402878) | > loss_mel_ce: 3.0962841510772705 (nan) | > loss: 0.01239775586873293 (nan) | > current_lr: 5e-06 | > step_time: 0.1964 (0.13681511369595323) | > loader_time: 0.0054 (0.004668285605900586)  --> TIME: 2025-02-26 02:54:45 -- STEP: 22386/36938 -- GLOBAL_STEP: 1056650 | > loss_text_ce: 0.034035809338092804 (0.032358835149309484) | > loss_mel_ce: 2.4383959770202637 (nan) | > loss: 0.009811238385736942 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13681234249872387) | > loader_time: 0.0045 (0.004668219505084419)  --> TIME: 2025-02-26 02:54:55 -- STEP: 22436/36938 -- GLOBAL_STEP: 1056700 | > loss_text_ce: 0.027418695390224457 (0.03235700568517174) | > loss_mel_ce: 1.5144551992416382 (nan) | > loss: 0.006118547637015581 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13681331987282477) | > loader_time: 0.0044 (0.0046680631814399794)  --> TIME: 2025-02-26 02:55:04 -- STEP: 22486/36938 -- GLOBAL_STEP: 1056750 | > loss_text_ce: 0.02569277212023735 (0.03235845193452423) | > loss_mel_ce: 2.885937452316284 (nan) | > loss: 0.011554088443517685 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13681500221840545) | > loader_time: 0.0045 (0.0046683867970342355)  --> TIME: 2025-02-26 02:55:13 -- STEP: 22536/36938 -- GLOBAL_STEP: 1056800 | > loss_text_ce: 0.038660239428281784 (0.03236135124914249) | > loss_mel_ce: 2.7231199741363525 (nan) | > loss: 0.010959446430206299 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13681729031574075) | > loader_time: 0.0048 (0.0046682795883282845)  --> TIME: 2025-02-26 02:55:23 -- STEP: 22586/36938 -- GLOBAL_STEP: 1056850 | > loss_text_ce: 0.03356894850730896 (0.03236174492257323) | > loss_mel_ce: 1.642680048942566 (nan) | > loss: 0.006651782430708408 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13682244035759256) | > loader_time: 0.0046 (0.004668163596649551)  --> TIME: 2025-02-26 02:55:32 -- STEP: 22636/36938 -- GLOBAL_STEP: 1056900 | > loss_text_ce: 0.026643017306923866 (0.032362442468820536) | > loss_mel_ce: 2.442539691925049 (nan) | > loss: 0.009798345156013966 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13681488899528324) | > loader_time: 0.0046 (0.004668131620771442)  --> TIME: 2025-02-26 02:55:41 -- STEP: 22686/36938 -- GLOBAL_STEP: 1056950 | > loss_text_ce: 0.0370379239320755 (0.03236378506886806) | > loss_mel_ce: 3.159271240234375 (nan) | > loss: 0.012683766894042492 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13680458549132032) | > loader_time: 0.0046 (0.004667962983646026)  --> TIME: 2025-02-26 02:55:50 -- STEP: 22736/36938 -- GLOBAL_STEP: 1057000 | > loss_text_ce: 0.03029853291809559 (0.03236250326864525) | > loss_mel_ce: 2.7877912521362305 (nan) | > loss: 0.011182896792888641 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13679167193103148) | > loader_time: 0.0046 (0.004667930048087535)  --> TIME: 2025-02-26 02:55:59 -- STEP: 22786/36938 -- GLOBAL_STEP: 1057050 | > loss_text_ce: 0.023891789838671684 (0.03236116662715998) | > loss_mel_ce: 3.004887342453003 (nan) | > loss: 0.012018965557217598 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.13679053267990737) | > loader_time: 0.0048 (0.004667810944649503)  --> TIME: 2025-02-26 02:56:08 -- STEP: 22836/36938 -- GLOBAL_STEP: 1057100 | > loss_text_ce: 0.026805082336068153 (0.03235915365644721) | > loss_mel_ce: 2.1772384643554688 (nan) | > loss: 0.00874620582908392 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13678381920779695) | > loader_time: 0.0045 (0.004667566127045416)  --> TIME: 2025-02-26 02:56:18 -- STEP: 22886/36938 -- GLOBAL_STEP: 1057150 | > loss_text_ce: 0.031159695237874985 (0.03235992163427536) | > loss_mel_ce: 3.2925655841827393 (nan) | > loss: 0.013189386576414108 (nan) | > current_lr: 5e-06 | > step_time: 0.1782 (0.1367806731472178) | > loader_time: 0.0048 (0.0046673668625783205)  --> TIME: 2025-02-26 02:56:27 -- STEP: 22936/36938 -- GLOBAL_STEP: 1057200 | > loss_text_ce: 0.04468461498618126 (0.032360323198187727) | > loss_mel_ce: 3.19757080078125 (nan) | > loss: 0.012866093777120113 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13678075717765475) | > loader_time: 0.0045 (0.004667239620330471)  --> TIME: 2025-02-26 02:56:36 -- STEP: 22986/36938 -- GLOBAL_STEP: 1057250 | > loss_text_ce: 0.040440987795591354 (0.032363612723326) | > loss_mel_ce: 1.3550446033477783 (nan) | > loss: 0.005537641700357199 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13678168165541757) | > loader_time: 0.0046 (0.004667252335879885)  --> TIME: 2025-02-26 02:56:45 -- STEP: 23036/36938 -- GLOBAL_STEP: 1057300 | > loss_text_ce: 0.03239244967699051 (0.03236582674400469) | > loss_mel_ce: 2.515780210494995 (nan) | > loss: 0.010111796669661999 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13677421705365653) | > loader_time: 0.0046 (0.004667205836628421)  --> TIME: 2025-02-26 02:56:55 -- STEP: 23086/36938 -- GLOBAL_STEP: 1057350 | > loss_text_ce: 0.042086776345968246 (0.0323661943544375) | > loss_mel_ce: 3.1524722576141357 (nan) | > loss: 0.012676822952926159 (nan) | > current_lr: 5e-06 | > step_time: 0.1566 (0.1367879248177657) | > loader_time: 0.0046 (0.00466720486579543)  --> TIME: 2025-02-26 02:57:04 -- STEP: 23136/36938 -- GLOBAL_STEP: 1057400 | > loss_text_ce: 0.02743677608668804 (0.032366531768464005) | > loss_mel_ce: 2.4096148014068604 (nan) | > loss: 0.00967083964496851 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.1367795414923809) | > loader_time: 0.0045 (0.004667185885860043)  --> TIME: 2025-02-26 02:57:13 -- STEP: 23186/36938 -- GLOBAL_STEP: 1057450 | > loss_text_ce: 0.03694574534893036 (0.03236332527525451) | > loss_mel_ce: 2.518944263458252 (nan) | > loss: 0.01014242134988308 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13677842777653426) | > loader_time: 0.0045 (0.004667036487905429)  --> TIME: 2025-02-26 02:57:23 -- STEP: 23236/36938 -- GLOBAL_STEP: 1057500 | > loss_text_ce: 0.0353829488158226 (0.03236635654355405) | > loss_mel_ce: 1.9140625 (nan) | > loss: 0.007735895458608866 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13677597474797532) | > loader_time: 0.0047 (0.004666902067165068)  --> TIME: 2025-02-26 02:57:32 -- STEP: 23286/36938 -- GLOBAL_STEP: 1057550 | > loss_text_ce: 0.03889145329594612 (0.032366464358537586) | > loss_mel_ce: 2.4757773876190186 (nan) | > loss: 0.009978845715522766 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13677964416922744) | > loader_time: 0.0045 (0.004666839710351245)  --> TIME: 2025-02-26 02:57:42 -- STEP: 23336/36938 -- GLOBAL_STEP: 1057600 | > loss_text_ce: 0.03014312870800495 (0.03236562798681219) | > loss_mel_ce: 2.532888174057007 (nan) | > loss: 0.01017075963318348 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13678640121052052) | > loader_time: 0.0045 (0.004666712815768913)  --> TIME: 2025-02-26 02:57:51 -- STEP: 23386/36938 -- GLOBAL_STEP: 1057650 | > loss_text_ce: 0.03472207859158516 (0.03236246465456908) | > loss_mel_ce: 2.7925567626953125 (nan) | > loss: 0.011219360865652561 (nan) | > current_lr: 5e-06 | > step_time: 0.1691 (0.13678946089705935) | > loader_time: 0.0046 (0.004666719670721038)  --> TIME: 2025-02-26 02:58:01 -- STEP: 23436/36938 -- GLOBAL_STEP: 1057700 | > loss_text_ce: 0.028555598109960556 (0.032363674375539275) | > loss_mel_ce: 1.763580322265625 (nan) | > loss: 0.007111650891602039 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13679564266730684) | > loader_time: 0.0045 (0.004666630939770761)  --> TIME: 2025-02-26 02:58:10 -- STEP: 23486/36938 -- GLOBAL_STEP: 1057750 | > loss_text_ce: 0.03401997312903404 (0.032364781386737845) | > loss_mel_ce: 3.0835607051849365 (nan) | > loss: 0.01237135287374258 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.1367973590601681) | > loader_time: 0.0047 (0.00466648842826933)  --> TIME: 2025-02-26 02:58:19 -- STEP: 23536/36938 -- GLOBAL_STEP: 1057800 | > loss_text_ce: 0.015781821683049202 (0.03236148734778013) | > loss_mel_ce: 0.6842516660690308 (nan) | > loss: 0.00277791079133749 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.1368055682373575) | > loader_time: 0.0044 (0.004666595607692083)  --> TIME: 2025-02-26 02:58:29 -- STEP: 23586/36938 -- GLOBAL_STEP: 1057850 | > loss_text_ce: 0.03349599987268448 (0.03236020236899731) | > loss_mel_ce: 3.0523252487182617 (nan) | > loss: 0.012245322577655315 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.1368080669309441) | > loader_time: 0.0048 (0.004666481543302959)  --> TIME: 2025-02-26 02:58:38 -- STEP: 23636/36938 -- GLOBAL_STEP: 1057900 | > loss_text_ce: 0.030365100130438805 (0.03235853797085338) | > loss_mel_ce: 2.923215627670288 (nan) | > loss: 0.011720558628439903 (nan) | > current_lr: 5e-06 | > step_time: 0.1773 (0.1368141950014027) | > loader_time: 0.0048 (0.004666446439101017)  --> TIME: 2025-02-26 02:58:48 -- STEP: 23686/36938 -- GLOBAL_STEP: 1057950 | > loss_text_ce: 0.04390477016568184 (0.03235666367763946) | > loss_mel_ce: 2.9958701133728027 (nan) | > loss: 0.012062599882483482 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.13680998675879483) | > loader_time: 0.0048 (0.004666438308467156)  --> TIME: 2025-02-26 02:58:57 -- STEP: 23736/36938 -- GLOBAL_STEP: 1058000 | > loss_text_ce: 0.024395134299993515 (0.03235733804755496) | > loss_mel_ce: 1.8401620388031006 (nan) | > loss: 0.0073990365490317345 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13680926186290404) | > loader_time: 0.0046 (0.004666431608286863)  --> TIME: 2025-02-26 02:59:06 -- STEP: 23786/36938 -- GLOBAL_STEP: 1058050 | > loss_text_ce: 0.032188575714826584 (0.03235818955020356) | > loss_mel_ce: 2.3653981685638428 (nan) | > loss: 0.009514234028756618 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13681495296918125) | > loader_time: 0.0045 (0.004666311089549287)  --> TIME: 2025-02-26 02:59:15 -- STEP: 23836/36938 -- GLOBAL_STEP: 1058100 | > loss_text_ce: 0.027798082679510117 (0.032355997168903895) | > loss_mel_ce: 2.6737937927246094 (nan) | > loss: 0.010720604099333286 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13680872111577458) | > loader_time: 0.0046 (0.004666291691149056)  --> TIME: 2025-02-26 02:59:25 -- STEP: 23886/36938 -- GLOBAL_STEP: 1058150 | > loss_text_ce: 0.03442220017313957 (0.032354728806051704) | > loss_mel_ce: 2.1676299571990967 (nan) | > loss: 0.008738302625715733 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13680756386526452) | > loader_time: 0.0046 (0.004666218194272425)  --> TIME: 2025-02-26 02:59:34 -- STEP: 23936/36938 -- GLOBAL_STEP: 1058200 | > loss_text_ce: 0.03583234176039696 (0.03235485863115166) | > loss_mel_ce: 2.7990617752075195 (nan) | > loss: 0.011249580420553684 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13681027458552467) | > loader_time: 0.0046 (0.004666240238410268)  --> TIME: 2025-02-26 02:59:44 -- STEP: 23986/36938 -- GLOBAL_STEP: 1058250 | > loss_text_ce: 0.03152744099497795 (0.032355488549843624) | > loss_mel_ce: 3.2466859817504883 (nan) | > loss: 0.013008784502744675 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13681002932732875) | > loader_time: 0.0046 (0.004666175077310012)  --> TIME: 2025-02-26 02:59:52 -- STEP: 24036/36938 -- GLOBAL_STEP: 1058300 | > loss_text_ce: 0.027005046606063843 (0.03235606094107784) | > loss_mel_ce: 2.3787028789520264 (nan) | > loss: 0.009546460583806038 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13679678142715632) | > loader_time: 0.0045 (0.004666032420555316)  --> TIME: 2025-02-26 03:00:02 -- STEP: 24086/36938 -- GLOBAL_STEP: 1058350 | > loss_text_ce: 0.03054404817521572 (0.03235440125625024) | > loss_mel_ce: 2.3850979804992676 (nan) | > loss: 0.009585881605744362 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13679099838212921) | > loader_time: 0.0046 (0.004665923071077853)  --> TIME: 2025-02-26 03:00:10 -- STEP: 24136/36938 -- GLOBAL_STEP: 1058400 | > loss_text_ce: 0.03214158117771149 (0.03235295279208344) | > loss_mel_ce: 1.6913539171218872 (nan) | > loss: 0.006839268375188112 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13677635575550015) | > loader_time: 0.0047 (0.004665864977885607)  --> TIME: 2025-02-26 03:00:20 -- STEP: 24186/36938 -- GLOBAL_STEP: 1058450 | > loss_text_ce: 0.03130623325705528 (0.03235123431474701) | > loss_mel_ce: 2.1225569248199463 (nan) | > loss: 0.00854707695543766 (nan) | > current_lr: 5e-06 | > step_time: 0.1504 (0.13677418286181633) | > loader_time: 0.0046 (0.004665759314992319)  --> TIME: 2025-02-26 03:00:29 -- STEP: 24236/36938 -- GLOBAL_STEP: 1058500 | > loss_text_ce: 0.03317127749323845 (0.03235116235539555) | > loss_mel_ce: 2.2096445560455322 (nan) | > loss: 0.00890006311237812 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13677199967211787) | > loader_time: 0.0045 (0.004665678297849672)  --> TIME: 2025-02-26 03:00:38 -- STEP: 24286/36938 -- GLOBAL_STEP: 1058550 | > loss_text_ce: 0.037011999636888504 (0.03235340806434925) | > loss_mel_ce: 2.3015122413635254 (nan) | > loss: 0.009279859252274036 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.1367732967957737) | > loader_time: 0.0047 (0.004665715066328157)  --> TIME: 2025-02-26 03:00:48 -- STEP: 24336/36938 -- GLOBAL_STEP: 1058600 | > loss_text_ce: 0.029172709211707115 (0.03235183798713449) | > loss_mel_ce: 2.65511155128479 (nan) | > loss: 0.010651921853423119 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.13677519747961792) | > loader_time: 0.0048 (0.004665610921139951)  --> TIME: 2025-02-26 03:00:57 -- STEP: 24386/36938 -- GLOBAL_STEP: 1058650 | > loss_text_ce: 0.023568838834762573 (0.032351760728129474) | > loss_mel_ce: 2.061298370361328 (nan) | > loss: 0.008273283019661903 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13678881271061757) | > loader_time: 0.0046 (0.004665547161060073)  --> TIME: 2025-02-26 03:01:07 -- STEP: 24436/36938 -- GLOBAL_STEP: 1058700 | > loss_text_ce: 0.037656646221876144 (0.032351442880087386) | > loss_mel_ce: 3.055739164352417 (nan) | > loss: 0.012275380082428455 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.1367908425740134) | > loader_time: 0.0045 (0.00466548292038582)  --> TIME: 2025-02-26 03:01:16 -- STEP: 24486/36938 -- GLOBAL_STEP: 1058750 | > loss_text_ce: 0.009654284454882145 (0.032347099550369095) | > loss_mel_ce: 1.5454668998718262 (nan) | > loss: 0.006171116139739752 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13678348680770672) | > loader_time: 0.0045 (0.004665336849973226)  --> TIME: 2025-02-26 03:01:25 -- STEP: 24536/36938 -- GLOBAL_STEP: 1058800 | > loss_text_ce: 0.03830472752451897 (0.03234618258629343) | > loss_mel_ce: 3.1068406105041504 (nan) | > loss: 0.012480736710131168 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13677577613657163) | > loader_time: 0.0046 (0.004665247248172924)  --> TIME: 2025-02-26 03:01:34 -- STEP: 24586/36938 -- GLOBAL_STEP: 1058850 | > loss_text_ce: 0.009255561977624893 (0.03234268348819125) | > loss_mel_ce: 2.20393967628479 (nan) | > loss: 0.008782521821558475 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13677687025493104) | > loader_time: 0.0047 (0.0046652360355399335)  --> TIME: 2025-02-26 03:01:43 -- STEP: 24636/36938 -- GLOBAL_STEP: 1058900 | > loss_text_ce: 0.02058660052716732 (0.03234190452493618) | > loss_mel_ce: 1.4376084804534912 (nan) | > loss: 0.005786488763988018 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13676481538052052) | > loader_time: 0.0044 (0.004665068206626087)  --> TIME: 2025-02-26 03:01:53 -- STEP: 24686/36938 -- GLOBAL_STEP: 1058950 | > loss_text_ce: 0.027504760771989822 (0.03234254493744654) | > loss_mel_ce: 2.8932559490203857 (nan) | > loss: 0.011590320616960526 (nan) | > current_lr: 5e-06 | > step_time: 0.1457 (0.13676233266708324) | > loader_time: 0.0046 (0.004664942046323639)  --> TIME: 2025-02-26 03:02:02 -- STEP: 24736/36938 -- GLOBAL_STEP: 1059000 | > loss_text_ce: 0.037452030926942825 (0.03234083002975271) | > loss_mel_ce: 2.082697629928589 (nan) | > loss: 0.008413292467594147 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13675838531528794) | > loader_time: 0.0046 (0.0046649262752446715)  --> TIME: 2025-02-26 03:02:11 -- STEP: 24786/36938 -- GLOBAL_STEP: 1059050 | > loss_text_ce: 0.03046983666718006 (0.03233907032241496) | > loss_mel_ce: 3.224386692047119 (nan) | > loss: 0.012916098348796368 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13675921808612437) | > loader_time: 0.0046 (0.0046647467163432035)  --> TIME: 2025-02-26 03:02:21 -- STEP: 24836/36938 -- GLOBAL_STEP: 1059100 | > loss_text_ce: 0.029514962807297707 (0.03233909489567671) | > loss_mel_ce: 2.9246137142181396 (nan) | > loss: 0.011722734197974205 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.13676065690353578) | > loader_time: 0.0068 (0.004664659730515297)  --> TIME: 2025-02-26 03:02:30 -- STEP: 24886/36938 -- GLOBAL_STEP: 1059150 | > loss_text_ce: 0.03934391215443611 (0.03233967849598624) | > loss_mel_ce: 0.01104736328125 (nan) | > loss: 0.00019996539049316198 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.13676031439823536) | > loader_time: 0.0045 (0.004664511195066847)  --> TIME: 2025-02-26 03:02:39 -- STEP: 24936/36938 -- GLOBAL_STEP: 1059200 | > loss_text_ce: 0.032249677926301956 (0.0323398334858707) | > loss_mel_ce: 3.4053354263305664 (nan) | > loss: 0.013641211204230785 (nan) | > current_lr: 5e-06 | > step_time: 0.1966 (0.13675679996452042) | > loader_time: 0.0048 (0.004664387014916522)  --> TIME: 2025-02-26 03:02:48 -- STEP: 24986/36938 -- GLOBAL_STEP: 1059250 | > loss_text_ce: 0.030202345922589302 (0.0323407604352716) | > loss_mel_ce: 2.2493274211883545 (nan) | > loss: 0.009045753628015518 (nan) | > current_lr: 5e-06 | > step_time: 0.1622 (0.13675139615049775) | > loader_time: 0.0046 (0.004664224371424986)  --> TIME: 2025-02-26 03:02:57 -- STEP: 25036/36938 -- GLOBAL_STEP: 1059300 | > loss_text_ce: 0.023196088150143623 (0.03234186461412219) | > loss_mel_ce: 1.799983263015747 (nan) | > loss: 0.007234839256852865 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.1367382420520669) | > loader_time: 0.0049 (0.0046640579779320766)  --> TIME: 2025-02-26 03:03:06 -- STEP: 25086/36938 -- GLOBAL_STEP: 1059350 | > loss_text_ce: 0.03991078585386276 (0.032337465964492766) | > loss_mel_ce: 0.0194091796875 (nan) | > loss: 0.00023539669928140938 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.13673040200237913) | > loader_time: 0.0046 (0.004663882430048551)  --> TIME: 2025-02-26 03:03:16 -- STEP: 25136/36938 -- GLOBAL_STEP: 1059400 | > loss_text_ce: 0.026504985988140106 (0.03233755873852256) | > loss_mel_ce: 2.569702386856079 (nan) | > loss: 0.010302410461008549 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13674375953604626) | > loader_time: 0.0046 (0.004663867651847126)  --> TIME: 2025-02-26 03:03:26 -- STEP: 25186/36938 -- GLOBAL_STEP: 1059450 | > loss_text_ce: 0.03785181790590286 (0.03234013223612437) | > loss_mel_ce: 3.0555062294006348 (nan) | > loss: 0.012275231070816517 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.1367466634898658) | > loader_time: 0.0048 (0.0046638879671492975)  --> TIME: 2025-02-26 03:03:35 -- STEP: 25236/36938 -- GLOBAL_STEP: 1059500 | > loss_text_ce: 0.04050378128886223 (0.032339564284766906) | > loss_mel_ce: 3.1733901500701904 (nan) | > loss: 0.012753548100590706 (nan) | > current_lr: 5e-06 | > step_time: 0.1903 (0.13675357448372716) | > loader_time: 0.0048 (0.004663874096265032)  --> TIME: 2025-02-26 03:03:45 -- STEP: 25286/36938 -- GLOBAL_STEP: 1059550 | > loss_text_ce: 0.037101224064826965 (0.032341974769790165) | > loss_mel_ce: 2.6845901012420654 (nan) | > loss: 0.01080036349594593 (nan) | > current_lr: 5e-06 | > step_time: 0.1704 (0.13676414585543598) | > loader_time: 0.0047 (0.004664184171584163)  --> TIME: 2025-02-26 03:03:54 -- STEP: 25336/36938 -- GLOBAL_STEP: 1059600 | > loss_text_ce: 0.033004455268383026 (0.03234242211056802) | > loss_mel_ce: 2.851635217666626 (nan) | > loss: 0.011446983553469181 (nan) | > current_lr: 5e-06 | > step_time: 0.1588 (0.13676365824162892) | > loader_time: 0.0048 (0.004664176122826815)  --> TIME: 2025-02-26 03:04:03 -- STEP: 25386/36938 -- GLOBAL_STEP: 1059650 | > loss_text_ce: 0.03025462105870247 (0.03234236677925504) | > loss_mel_ce: 2.511338710784912 (nan) | > loss: 0.010085687972605228 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13676299180468438) | > loader_time: 0.0046 (0.004664159502945906)  --> TIME: 2025-02-26 03:04:12 -- STEP: 25436/36938 -- GLOBAL_STEP: 1059700 | > loss_text_ce: 0.04125017672777176 (0.03234508903320369) | > loss_mel_ce: 0.0380859375 (nan) | > loss: 0.000314825854729861 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13674972854633266) | > loader_time: 0.0046 (0.004664093570001362)  --> TIME: 2025-02-26 03:04:21 -- STEP: 25486/36938 -- GLOBAL_STEP: 1059750 | > loss_text_ce: 0.023544136434793472 (0.032344883664743626) | > loss_mel_ce: 1.8551431894302368 (nan) | > loss: 0.007455109152942896 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.1367442764386861) | > loader_time: 0.0047 (0.004663977781644679)  --> TIME: 2025-02-26 03:04:30 -- STEP: 25536/36938 -- GLOBAL_STEP: 1059800 | > loss_text_ce: 0.0350693017244339 (0.03234539415485928) | > loss_mel_ce: 2.902404308319092 (nan) | > loss: 0.011656641960144043 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13673598694622166) | > loader_time: 0.0046 (0.004663880979805665)  --> TIME: 2025-02-26 03:04:40 -- STEP: 25586/36938 -- GLOBAL_STEP: 1059850 | > loss_text_ce: 0.0278544370085001 (0.032343677958933045) | > loss_mel_ce: 3.0564072132110596 (nan) | > loss: 0.012239133939146996 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.13673173679586584) | > loader_time: 0.0047 (0.004663913885292664)  --> TIME: 2025-02-26 03:04:49 -- STEP: 25636/36938 -- GLOBAL_STEP: 1059900 | > loss_text_ce: 0.03105095587670803 (0.03234393521502161) | > loss_mel_ce: 2.2878572940826416 (nan) | > loss: 0.009202017448842525 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.1367260733080805) | > loader_time: 0.0047 (0.004663876716014402)  --> TIME: 2025-02-26 03:04:58 -- STEP: 25686/36938 -- GLOBAL_STEP: 1059950 | > loss_text_ce: 0.025922276079654694 (0.03234075213177559) | > loss_mel_ce: 2.982405662536621 (nan) | > loss: 0.011937810108065605 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13672718835250242) | > loader_time: 0.0045 (0.004663773278418548)  --> TIME: 2025-02-26 03:05:08 -- STEP: 25736/36938 -- GLOBAL_STEP: 1060000 | > loss_text_ce: 0.0290262121707201 (0.0323401920336998) | > loss_mel_ce: 1.4437255859375 (nan) | > loss: 0.005844253581017256 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.1367258095374425) | > loader_time: 0.0045 (0.00466367816346976) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1060000.pth  --> TIME: 2025-02-26 03:05:23 -- STEP: 25786/36938 -- GLOBAL_STEP: 1060050 | > loss_text_ce: 0.03742590919137001 (0.032341084439800216) | > loss_mel_ce: 2.339449167251587 (nan) | > loss: 0.009432044811546803 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13671265292517246) | > loader_time: 0.0045 (0.004663541939614932)  --> TIME: 2025-02-26 03:05:32 -- STEP: 25836/36938 -- GLOBAL_STEP: 1060100 | > loss_text_ce: 0.03663479909300804 (0.03234303752879246) | > loss_mel_ce: 1.9025484323501587 (nan) | > loss: 0.007695171982049942 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13669731309122746) | > loader_time: 0.0046 (0.004663392594584499)  --> TIME: 2025-02-26 03:05:41 -- STEP: 25886/36938 -- GLOBAL_STEP: 1060150 | > loss_text_ce: 0.029869621619582176 (0.03234228111189678) | > loss_mel_ce: 2.8946373462677 (nan) | > loss: 0.011605186387896538 (nan) | > current_lr: 5e-06 | > step_time: 0.1762 (0.13668430414283173) | > loader_time: 0.0049 (0.004663306668563111)  --> TIME: 2025-02-26 03:05:50 -- STEP: 25936/36938 -- GLOBAL_STEP: 1060200 | > loss_text_ce: 0.027928940951824188 (0.03234237819797799) | > loss_mel_ce: 1.9953196048736572 (nan) | > loss: 0.00802876427769661 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13667699932721217) | > loader_time: 0.0048 (0.004663171433946531)  --> TIME: 2025-02-26 03:05:59 -- STEP: 25986/36938 -- GLOBAL_STEP: 1060250 | > loss_text_ce: 0.03000452183187008 (0.03234468841003689) | > loss_mel_ce: 0.1304931640625 (nan) | > loss: 0.0006368955946527421 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.1366739017550731) | > loader_time: 0.0046 (0.004663026058526161)  --> TIME: 2025-02-26 03:06:08 -- STEP: 26036/36938 -- GLOBAL_STEP: 1060300 | > loss_text_ce: 0.02791016548871994 (0.032344881209960055) | > loss_mel_ce: 1.5025994777679443 (nan) | > loss: 0.00607345113530755 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.1366657091095482) | > loader_time: 0.0045 (0.0046630419606454995)  --> TIME: 2025-02-26 03:06:17 -- STEP: 26086/36938 -- GLOBAL_STEP: 1060350 | > loss_text_ce: 0.03267841786146164 (0.03234583926043678) | > loss_mel_ce: 2.3260812759399414 (nan) | > loss: 0.009360157884657383 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.13666438620432209) | > loader_time: 0.0047 (0.00466298942760413)  --> TIME: 2025-02-26 03:06:27 -- STEP: 26136/36938 -- GLOBAL_STEP: 1060400 | > loss_text_ce: 0.03379496559500694 (0.03234493157390278) | > loss_mel_ce: 2.367313861846924 (nan) | > loss: 0.009528209455311298 (nan) | > current_lr: 5e-06 | > step_time: 0.1576 (0.13666579764955603) | > loader_time: 0.0046 (0.004662958158788477)  --> TIME: 2025-02-26 03:06:36 -- STEP: 26186/36938 -- GLOBAL_STEP: 1060450 | > loss_text_ce: 0.03480635955929756 (0.03234276021316457) | > loss_mel_ce: 2.7728657722473145 (nan) | > loss: 0.011141556315124035 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13666026674015927) | > loader_time: 0.0046 (0.00466284725123948)  --> TIME: 2025-02-26 03:06:45 -- STEP: 26236/36938 -- GLOBAL_STEP: 1060500 | > loss_text_ce: 0.03594076633453369 (0.032344507320278784) | > loss_mel_ce: 2.9198551177978516 (nan) | > loss: 0.011729349382221699 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13666437360676695) | > loader_time: 0.0046 (0.004662819862147648)  --> TIME: 2025-02-26 03:06:54 -- STEP: 26286/36938 -- GLOBAL_STEP: 1060550 | > loss_text_ce: 0.027778446674346924 (0.03234290947025764) | > loss_mel_ce: 2.1845860481262207 (nan) | > loss: 0.008779224008321762 (nan) | > current_lr: 5e-06 | > step_time: 0.1628 (0.13665965608762196) | > loader_time: 0.0047 (0.004662689358678544)  --> TIME: 2025-02-26 03:07:04 -- STEP: 26336/36938 -- GLOBAL_STEP: 1060600 | > loss_text_ce: 0.03304369002580643 (0.0323425206130806) | > loss_mel_ce: 2.8191657066345215 (nan) | > loss: 0.011318291537463665 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.1366585849321491) | > loader_time: 0.0045 (0.004662627275050943)  --> TIME: 2025-02-26 03:07:13 -- STEP: 26386/36938 -- GLOBAL_STEP: 1060650 | > loss_text_ce: 0.03261185809969902 (0.03234111754989845) | > loss_mel_ce: 3.0104455947875977 (nan) | > loss: 0.012075625360012054 (nan) | > current_lr: 5e-06 | > step_time: 0.1799 (0.13666393093096785) | > loader_time: 0.005 (0.004662632598839386)  --> TIME: 2025-02-26 03:07:22 -- STEP: 26436/36938 -- GLOBAL_STEP: 1060700 | > loss_text_ce: 0.028279008343815804 (0.03234309411425129) | > loss_mel_ce: 2.1381702423095703 (nan) | > loss: 0.008597021922469139 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.1366567573004086) | > loader_time: 0.0045 (0.004662557690095626)  --> TIME: 2025-02-26 03:07:32 -- STEP: 26486/36938 -- GLOBAL_STEP: 1060750 | > loss_text_ce: 0.025638706982135773 (0.03234001946851307) | > loss_mel_ce: 0.26544851064682007 (nan) | > loss: 0.0011551080970093608 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.13664911955961925) | > loader_time: 0.0046 (0.004662590364238572)  --> TIME: 2025-02-26 03:07:41 -- STEP: 26536/36938 -- GLOBAL_STEP: 1060800 | > loss_text_ce: 0.02891523204743862 (0.032338279261972455) | > loss_mel_ce: 2.8893697261810303 (nan) | > loss: 0.011580496095120907 (nan) | > current_lr: 5e-06 | > step_time: 0.1578 (0.1366559246469063) | > loader_time: 0.0046 (0.004662684029328183)  --> TIME: 2025-02-26 03:07:50 -- STEP: 26586/36938 -- GLOBAL_STEP: 1060850 | > loss_text_ce: 0.022481480613350868 (0.032335936555872635) | > loss_mel_ce: 0.0061706542037427425 (nan) | > loss: 0.00011369895946700126 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13665570010542094) | > loader_time: 0.0045 (0.004662652725220314)  --> TIME: 2025-02-26 03:08:00 -- STEP: 26636/36938 -- GLOBAL_STEP: 1060900 | > loss_text_ce: 0.037024665623903275 (0.03233754394217143) | > loss_mel_ce: 3.0024962425231934 (nan) | > loss: 0.012061592191457748 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1366576629906637) | > loader_time: 0.0045 (0.004662641893190075)  --> TIME: 2025-02-26 03:08:09 -- STEP: 26686/36938 -- GLOBAL_STEP: 1060950 | > loss_text_ce: 0.03834265097975731 (0.0323407464477099) | > loss_mel_ce: 3.040226459503174 (nan) | > loss: 0.012216544710099697 (nan) | > current_lr: 5e-06 | > step_time: 0.178 (0.13664732126427448) | > loader_time: 0.0048 (0.004662587511694031)  --> TIME: 2025-02-26 03:08:18 -- STEP: 26736/36938 -- GLOBAL_STEP: 1061000 | > loss_text_ce: 0.03244752436876297 (0.03234282911509988) | > loss_mel_ce: 2.265251636505127 (nan) | > loss: 0.0091178547590971 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.136654110020111) | > loader_time: 0.0046 (0.0046626632884902855)  --> TIME: 2025-02-26 03:08:27 -- STEP: 26786/36938 -- GLOBAL_STEP: 1061050 | > loss_text_ce: 0.030318373814225197 (0.03234321892709403) | > loss_mel_ce: 3.3887054920196533 (nan) | > loss: 0.013567555695772171 (nan) | > current_lr: 5e-06 | > step_time: 0.1837 (0.13664655930910757) | > loader_time: 0.0069 (0.004662667682277155)  --> TIME: 2025-02-26 03:08:37 -- STEP: 26836/36938 -- GLOBAL_STEP: 1061100 | > loss_text_ce: 0.029463917016983032 (0.03234278987958834) | > loss_mel_ce: 2.8060686588287354 (nan) | > loss: 0.011252114549279213 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.1366489946637864) | > loader_time: 0.0047 (0.004662652354354862)  --> TIME: 2025-02-26 03:08:46 -- STEP: 26886/36938 -- GLOBAL_STEP: 1061150 | > loss_text_ce: 0.03211527690291405 (0.03233940181154604) | > loss_mel_ce: 2.938586950302124 (nan) | > loss: 0.011788501404225826 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.13665218041303798) | > loader_time: 0.0048 (0.004662589747343708)  --> TIME: 2025-02-26 03:08:56 -- STEP: 26936/36938 -- GLOBAL_STEP: 1061200 | > loss_text_ce: 0.03005368821322918 (0.0323402902458306) | > loss_mel_ce: 1.73469877243042 (nan) | > loss: 0.007002986501902342 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13665557110348625) | > loader_time: 0.0046 (0.004662549686885084)  --> TIME: 2025-02-26 03:09:05 -- STEP: 26986/36938 -- GLOBAL_STEP: 1061250 | > loss_text_ce: 0.039276547729969025 (0.032341289814414045) | > loss_mel_ce: 0.8102213740348816 (nan) | > loss: 0.0033710235729813576 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1366514587017149) | > loader_time: 0.0045 (0.004662662459596619)  --> TIME: 2025-02-26 03:09:14 -- STEP: 27036/36938 -- GLOBAL_STEP: 1061300 | > loss_text_ce: 0.022672118619084358 (0.03233903928237601) | > loss_mel_ce: 2.2678627967834473 (nan) | > loss: 0.009089425206184387 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13663475119974192) | > loader_time: 0.0045 (0.004662599519858591)  --> TIME: 2025-02-26 03:09:23 -- STEP: 27086/36938 -- GLOBAL_STEP: 1061350 | > loss_text_ce: 0.03040538728237152 (0.032337552016845274) | > loss_mel_ce: 2.454385280609131 (nan) | > loss: 0.009860280901193619 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13662902856255196) | > loader_time: 0.0045 (0.004662469730308204)  --> TIME: 2025-02-26 03:09:33 -- STEP: 27136/36938 -- GLOBAL_STEP: 1061400 | > loss_text_ce: 0.035241544246673584 (0.0323381013988595) | > loss_mel_ce: 3.573850154876709 (nan) | > loss: 0.014321793802082539 (nan) | > current_lr: 5e-06 | > step_time: 0.1871 (0.1366406710256109) | > loader_time: 0.0048 (0.004662396386265769)  --> TIME: 2025-02-26 03:09:42 -- STEP: 27186/36938 -- GLOBAL_STEP: 1061450 | > loss_text_ce: 0.03889215737581253 (0.03233652892526633) | > loss_mel_ce: 2.84089732170105 (nan) | > loss: 0.011427736841142178 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13664533163567616) | > loader_time: 0.0046 (0.004662311297242493)  --> TIME: 2025-02-26 03:09:51 -- STEP: 27236/36938 -- GLOBAL_STEP: 1061500 | > loss_text_ce: 0.03580763563513756 (0.03233794371670585) | > loss_mel_ce: 3.0630483627319336 (nan) | > loss: 0.012297048233449459 (nan) | > current_lr: 5e-06 | > step_time: 0.1917 (0.13664347234495766) | > loader_time: 0.0048 (0.004662261798459202)  --> TIME: 2025-02-26 03:10:00 -- STEP: 27286/36938 -- GLOBAL_STEP: 1061550 | > loss_text_ce: 0.025956766679883003 (0.03233778718373949) | > loss_mel_ce: 3.06221604347229 (nan) | > loss: 0.012254655361175537 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13664359421811548) | > loader_time: 0.0048 (0.004662138446024133)  --> TIME: 2025-02-26 03:10:10 -- STEP: 27336/36938 -- GLOBAL_STEP: 1061600 | > loss_text_ce: 0.032807543873786926 (0.03233662446226477) | > loss_mel_ce: 3.3135464191436768 (nan) | > loss: 0.013279183767735958 (nan) | > current_lr: 5e-06 | > step_time: 0.176 (0.1366436765718649) | > loader_time: 0.0048 (0.004662105134958979)  --> TIME: 2025-02-26 03:10:19 -- STEP: 27386/36938 -- GLOBAL_STEP: 1061650 | > loss_text_ce: 0.03323835879564285 (0.03233690376427234) | > loss_mel_ce: 3.2124295234680176 (nan) | > loss: 0.012879635207355022 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13663861599726349) | > loader_time: 0.0045 (0.004661950568481447)  --> TIME: 2025-02-26 03:10:28 -- STEP: 27436/36938 -- GLOBAL_STEP: 1061700 | > loss_text_ce: 0.03200480714440346 (0.03233584636264761) | > loss_mel_ce: 3.1457207202911377 (nan) | > loss: 0.012610022909939289 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.13663521099236647) | > loader_time: 0.0046 (0.004661851668604652)  --> TIME: 2025-02-26 03:10:37 -- STEP: 27486/36938 -- GLOBAL_STEP: 1061750 | > loss_text_ce: 0.028449123725295067 (0.03233427908511108) | > loss_mel_ce: 3.2073264122009277 (nan) | > loss: 0.01284037996083498 (nan) | > current_lr: 5e-06 | > step_time: 0.1566 (0.13662992414234326) | > loader_time: 0.0046 (0.004661684402999326)  --> TIME: 2025-02-26 03:10:46 -- STEP: 27536/36938 -- GLOBAL_STEP: 1061800 | > loss_text_ce: 0.030071619898080826 (0.03233686364392148) | > loss_mel_ce: 3.379129409790039 (nan) | > loss: 0.01352857705205679 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13661990278789646) | > loader_time: 0.0045 (0.00466167418538382)  --> TIME: 2025-02-26 03:10:55 -- STEP: 27586/36938 -- GLOBAL_STEP: 1061850 | > loss_text_ce: 0.04576511308550835 (0.032338463224724924) | > loss_mel_ce: 3.6722168922424316 (nan) | > loss: 0.014753897674381733 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1366130936476808) | > loader_time: 0.0046 (0.0046616516370489225)  --> TIME: 2025-02-26 03:11:04 -- STEP: 27636/36938 -- GLOBAL_STEP: 1061900 | > loss_text_ce: 0.03915490210056305 (0.032339374965382583) | > loss_mel_ce: 2.6380903720855713 (nan) | > loss: 0.010623990558087826 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13660406817022072) | > loader_time: 0.0045 (0.0046615442796249795)  --> TIME: 2025-02-26 03:11:13 -- STEP: 27686/36938 -- GLOBAL_STEP: 1061950 | > loss_text_ce: 0.03224175050854683 (0.0323424596391907) | > loss_mel_ce: 2.7897911071777344 (nan) | > loss: 0.011198543943464756 (nan) | > current_lr: 5e-06 | > step_time: 0.1462 (0.13660446401191217) | > loader_time: 0.0046 (0.004661519084975346)  --> TIME: 2025-02-26 03:11:23 -- STEP: 27736/36938 -- GLOBAL_STEP: 1062000 | > loss_text_ce: 0.02877841331064701 (0.03234009818798968) | > loss_mel_ce: 2.7847940921783447 (nan) | > loss: 0.011164970695972443 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1366081782693243) | > loader_time: 0.0047 (0.004661520224742606)  --> TIME: 2025-02-26 03:11:32 -- STEP: 27786/36938 -- GLOBAL_STEP: 1062050 | > loss_text_ce: 0.03370939940214157 (0.0323395564431236) | > loss_mel_ce: 2.9713141918182373 (nan) | > loss: 0.01192469708621502 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13660381931401708) | > loader_time: 0.0046 (0.0046614230532870385)  --> TIME: 2025-02-26 03:11:41 -- STEP: 27836/36938 -- GLOBAL_STEP: 1062100 | > loss_text_ce: 0.030624747276306152 (0.032341764168576245) | > loss_mel_ce: 2.3994996547698975 (nan) | > loss: 0.00964335072785616 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1365948320976854) | > loader_time: 0.0047 (0.004661322428005362)  --> TIME: 2025-02-26 03:11:50 -- STEP: 27886/36938 -- GLOBAL_STEP: 1062150 | > loss_text_ce: 0.0253861416131258 (0.03234215505739572) | > loss_mel_ce: 2.0796101093292236 (nan) | > loss: 0.008353159762918949 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13658456952673065) | > loader_time: 0.0045 (0.0046612095355748236)  --> TIME: 2025-02-26 03:11:59 -- STEP: 27936/36938 -- GLOBAL_STEP: 1062200 | > loss_text_ce: 0.027479132637381554 (0.03234200041730395) | > loss_mel_ce: 2.2506377696990967 (nan) | > loss: 0.009040147066116333 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.1365838660629223) | > loader_time: 0.0045 (0.0046612238095388465)  --> TIME: 2025-02-26 03:12:09 -- STEP: 27986/36938 -- GLOBAL_STEP: 1062250 | > loss_text_ce: 0.03851056471467018 (0.03234292188075809) | > loss_mel_ce: 3.1921162605285645 (nan) | > loss: 0.012819948606193066 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13658669457837538) | > loader_time: 0.0045 (0.004661154782789963)  --> TIME: 2025-02-26 03:12:18 -- STEP: 28036/36938 -- GLOBAL_STEP: 1062300 | > loss_text_ce: 0.03953736275434494 (0.032342843302369535) | > loss_mel_ce: 2.476797103881836 (nan) | > loss: 0.009985455311834812 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13657859708055797) | > loader_time: 0.0048 (0.00466120536461046)  --> TIME: 2025-02-26 03:12:27 -- STEP: 28086/36938 -- GLOBAL_STEP: 1062350 | > loss_text_ce: 0.03943978622555733 (0.0323445695397645) | > loss_mel_ce: 3.3071417808532715 (nan) | > loss: 0.01328008621931076 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13657487123803722) | > loader_time: 0.0046 (0.004661240392979814)  --> TIME: 2025-02-26 03:12:36 -- STEP: 28136/36938 -- GLOBAL_STEP: 1062400 | > loss_text_ce: 0.040901362895965576 (0.03234469618365672) | > loss_mel_ce: 0.00034021586179733276 (nan) | > loss: 0.00016365706687793136 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1365738423212318) | > loader_time: 0.0045 (0.004661224165998613)  --> TIME: 2025-02-26 03:12:46 -- STEP: 28186/36938 -- GLOBAL_STEP: 1062450 | > loss_text_ce: 0.027140812948346138 (0.03234405651466727) | > loss_mel_ce: 1.486236572265625 (nan) | > loss: 0.00600546645000577 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13657655689212886) | > loader_time: 0.0048 (0.004661296179150288)  --> TIME: 2025-02-26 03:12:55 -- STEP: 28236/36938 -- GLOBAL_STEP: 1062500 | > loss_text_ce: 0.030665859580039978 (0.032339892121017226) | > loss_mel_ce: 2.5928804874420166 (nan) | > loss: 0.010410899296402931 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13657488543785817) | > loader_time: 0.0046 (0.004661420162042598)  --> TIME: 2025-02-26 03:13:04 -- STEP: 28286/36938 -- GLOBAL_STEP: 1062550 | > loss_text_ce: 0.032654669135808945 (0.03233784532456897) | > loss_mel_ce: 1.7707979679107666 (nan) | > loss: 0.007156558334827423 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13657876557325593) | > loader_time: 0.0046 (0.004661476714080212)  --> TIME: 2025-02-26 03:13:14 -- STEP: 28336/36938 -- GLOBAL_STEP: 1062600 | > loss_text_ce: 0.026853298768401146 (0.032335244179363384) | > loss_mel_ce: 1.91539466381073 (nan) | > loss: 0.007707333657890558 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13657826605084292) | > loader_time: 0.0046 (0.004661516246992213)  --> TIME: 2025-02-26 03:13:23 -- STEP: 28386/36938 -- GLOBAL_STEP: 1062650 | > loss_text_ce: 0.025995727628469467 (0.03233548791555516) | > loss_mel_ce: 2.931640625 (nan) | > loss: 0.011736652813851833 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13657094821158722) | > loader_time: 0.0045 (0.004661512326162805)  --> TIME: 2025-02-26 03:13:32 -- STEP: 28436/36938 -- GLOBAL_STEP: 1062700 | > loss_text_ce: 0.02961471490561962 (0.032335634302239236) | > loss_mel_ce: 2.269972085952759 (nan) | > loss: 0.009125344455242157 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13656984596820076) | > loader_time: 0.0046 (0.004661573381393103)  --> TIME: 2025-02-26 03:13:41 -- STEP: 28486/36938 -- GLOBAL_STEP: 1062750 | > loss_text_ce: 0.030335955321788788 (0.03233405344798864) | > loss_mel_ce: 2.3240580558776855 (nan) | > loss: 0.009342833422124386 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.1365613514290845) | > loader_time: 0.0046 (0.004661510334351027)  --> TIME: 2025-02-26 03:13:51 -- STEP: 28536/36938 -- GLOBAL_STEP: 1062800 | > loss_text_ce: 0.03562144935131073 (0.03233617133791408) | > loss_mel_ce: 2.913100481033325 (nan) | > loss: 0.011701278388500214 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13656450465075237) | > loader_time: 0.0045 (0.0046615067369168645)  --> TIME: 2025-02-26 03:14:00 -- STEP: 28586/36938 -- GLOBAL_STEP: 1062850 | > loss_text_ce: 0.02788885310292244 (0.03233609573815831) | > loss_mel_ce: 1.9797818660736084 (nan) | > loss: 0.007966947741806507 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13656392363031541) | > loader_time: 0.0045 (0.004661384693413454)  --> TIME: 2025-02-26 03:14:09 -- STEP: 28636/36938 -- GLOBAL_STEP: 1062900 | > loss_text_ce: 0.03817697614431381 (0.03233605695548269) | > loss_mel_ce: 3.1527326107025146 (nan) | > loss: 0.012662340886890888 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.1365614747814616) | > loader_time: 0.0047 (0.00466128625522065)  --> TIME: 2025-02-26 03:14:18 -- STEP: 28686/36938 -- GLOBAL_STEP: 1062950 | > loss_text_ce: 0.03899114578962326 (0.03233598490687951) | > loss_mel_ce: 2.433823347091675 (nan) | > loss: 0.009812756441533566 (nan) | > current_lr: 5e-06 | > step_time: 0.1556 (0.13656563072501302) | > loader_time: 0.0047 (0.004661241094994435)  --> TIME: 2025-02-26 03:14:28 -- STEP: 28736/36938 -- GLOBAL_STEP: 1063000 | > loss_text_ce: 0.03459028899669647 (0.032336761891687674) | > loss_mel_ce: 1.7176920175552368 (nan) | > loss: 0.006953501608222723 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13657533154223597) | > loader_time: 0.0047 (0.004661289680508577)  --> TIME: 2025-02-26 03:14:37 -- STEP: 28786/36938 -- GLOBAL_STEP: 1063050 | > loss_text_ce: 0.03829433023929596 (0.03233731175407374) | > loss_mel_ce: 2.939584732055664 (nan) | > loss: 0.011816981248557568 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.1365741616631219) | > loader_time: 0.0047 (0.00466128189254154)  --> TIME: 2025-02-26 03:14:47 -- STEP: 28836/36938 -- GLOBAL_STEP: 1063100 | > loss_text_ce: 0.02804596535861492 (0.032336708822495286) | > loss_mel_ce: 3.1612589359283447 (nan) | > loss: 0.012655972503125668 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.13658372627281748) | > loader_time: 0.0047 (0.004661397061511347)  --> TIME: 2025-02-26 03:14:56 -- STEP: 28886/36938 -- GLOBAL_STEP: 1063150 | > loss_text_ce: 0.030187483876943588 (0.03233814785884198) | > loss_mel_ce: 3.0835673809051514 (nan) | > loss: 0.012356170453131199 (nan) | > current_lr: 5e-06 | > step_time: 0.1492 (0.13658459903146963) | > loader_time: 0.0047 (0.004661507193157244)  --> TIME: 2025-02-26 03:15:06 -- STEP: 28936/36938 -- GLOBAL_STEP: 1063200 | > loss_text_ce: 0.034028682857751846 (0.03233911333518934) | > loss_mel_ce: 3.075683832168579 (nan) | > loss: 0.012340130284428596 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13658365651755178) | > loader_time: 0.0046 (0.004661537350492984)  --> TIME: 2025-02-26 03:15:15 -- STEP: 28986/36938 -- GLOBAL_STEP: 1063250 | > loss_text_ce: 0.03200775384902954 (0.03233760835251438) | > loss_mel_ce: 2.4779677391052246 (nan) | > loss: 0.009960220195353031 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.1365779709521528) | > loader_time: 0.0045 (0.004661500754172474)  --> TIME: 2025-02-26 03:15:24 -- STEP: 29036/36938 -- GLOBAL_STEP: 1063300 | > loss_text_ce: 0.04101647809147835 (0.032339206723919586) | > loss_mel_ce: 2.9640867710113525 (nan) | > loss: 0.011925013735890388 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.1365779689856361) | > loader_time: 0.0046 (0.00466148018065129)  --> TIME: 2025-02-26 03:15:34 -- STEP: 29086/36938 -- GLOBAL_STEP: 1063350 | > loss_text_ce: 0.032091643661260605 (0.032337582576895056) | > loss_mel_ce: 3.332775354385376 (nan) | > loss: 0.013352647423744202 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1365852041765197) | > loader_time: 0.0048 (0.004661481686867333)  --> TIME: 2025-02-26 03:15:43 -- STEP: 29136/36938 -- GLOBAL_STEP: 1063400 | > loss_text_ce: 0.02873528003692627 (0.032337939683452124) | > loss_mel_ce: 2.9575648307800293 (nan) | > loss: 0.011850397102534771 (nan) | > current_lr: 5e-06 | > step_time: 0.1769 (0.13659197672227943) | > loader_time: 0.0048 (0.004661453892934599)  --> TIME: 2025-02-26 03:15:53 -- STEP: 29186/36938 -- GLOBAL_STEP: 1063450 | > loss_text_ce: 0.046083997935056686 (0.03233673934251532) | > loss_mel_ce: 3.333808422088623 (nan) | > loss: 0.01341227162629366 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.13659319460682157) | > loader_time: 0.0046 (0.004661387007844094)  --> TIME: 2025-02-26 03:16:02 -- STEP: 29236/36938 -- GLOBAL_STEP: 1063500 | > loss_text_ce: 0.032085590064525604 (0.032335020605747215) | > loss_mel_ce: 2.9141228199005127 (nan) | > loss: 0.011691303923726082 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13659520830333854) | > loader_time: 0.0046 (0.0046613534443811395)  --> TIME: 2025-02-26 03:16:11 -- STEP: 29286/36938 -- GLOBAL_STEP: 1063550 | > loss_text_ce: 0.030209217220544815 (0.03233578302463743) | > loss_mel_ce: 1.1277767419815063 (nan) | > loss: 0.004595182370394468 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1366013989604545) | > loader_time: 0.0046 (0.004661339973643016)  --> TIME: 2025-02-26 03:16:21 -- STEP: 29336/36938 -- GLOBAL_STEP: 1063600 | > loss_text_ce: 0.03970271348953247 (0.03233633866940435) | > loss_mel_ce: 2.291862726211548 (nan) | > loss: 0.009252243675291538 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13659876703663515) | > loader_time: 0.0047 (0.004661387965826766)  --> TIME: 2025-02-26 03:16:30 -- STEP: 29386/36938 -- GLOBAL_STEP: 1063650 | > loss_text_ce: 0.029918620362877846 (0.03233911159825083) | > loss_mel_ce: 2.492323875427246 (nan) | > loss: 0.010008899495005608 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1366024944039577) | > loader_time: 0.0045 (0.004661380429268865)  --> TIME: 2025-02-26 03:16:40 -- STEP: 29436/36938 -- GLOBAL_STEP: 1063700 | > loss_text_ce: 0.035555560141801834 (0.03233829007835143) | > loss_mel_ce: 0.0054931640625 (nan) | > loss: 0.00016289176710415632 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13660361038989077) | > loader_time: 0.0047 (0.004661438557129391)  --> TIME: 2025-02-26 03:16:49 -- STEP: 29486/36938 -- GLOBAL_STEP: 1063750 | > loss_text_ce: 0.033883579075336456 (0.032336968768485644) | > loss_mel_ce: 2.4533371925354004 (nan) | > loss: 0.009869923815131187 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.1366063020757034) | > loader_time: 0.0047 (0.004661439296826177)  --> TIME: 2025-02-26 03:16:58 -- STEP: 29536/36938 -- GLOBAL_STEP: 1063800 | > loss_text_ce: 0.026581358164548874 (0.032336425925372246) | > loss_mel_ce: 2.918735980987549 (nan) | > loss: 0.011687767691910267 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13660574711673723) | > loader_time: 0.0047 (0.0046613544774598)  --> TIME: 2025-02-26 03:17:07 -- STEP: 29586/36938 -- GLOBAL_STEP: 1063850 | > loss_text_ce: 0.03020699881017208 (0.032332764129240424) | > loss_mel_ce: 3.4381377696990967 (nan) | > loss: 0.013763273134827614 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13659350303755027) | > loader_time: 0.0045 (0.004661196612493032)  --> TIME: 2025-02-26 03:17:16 -- STEP: 29636/36938 -- GLOBAL_STEP: 1063900 | > loss_text_ce: 0.03752650320529938 (0.032331777915244976) | > loss_mel_ce: 3.025068759918213 (nan) | > loss: 0.012153157033026218 (nan) | > current_lr: 5e-06 | > step_time: 0.1502 (0.13658981590172065) | > loader_time: 0.0046 (0.004661120042930916)  --> TIME: 2025-02-26 03:17:26 -- STEP: 29686/36938 -- GLOBAL_STEP: 1063950 | > loss_text_ce: 0.03966570645570755 (0.03233069779415283) | > loss_mel_ce: 2.391230344772339 (nan) | > loss: 0.00964641384780407 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.1365888577494479) | > loader_time: 0.0046 (0.004661040422385227)  --> TIME: 2025-02-26 03:17:35 -- STEP: 29736/36938 -- GLOBAL_STEP: 1064000 | > loss_text_ce: 0.02963380329310894 (0.032330633706185996) | > loss_mel_ce: 2.5571882724761963 (nan) | > loss: 0.01026516780257225 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13658003985705597) | > loader_time: 0.0048 (0.004660987031373224)  --> TIME: 2025-02-26 03:17:44 -- STEP: 29786/36938 -- GLOBAL_STEP: 1064050 | > loss_text_ce: 0.02000480517745018 (0.03232993135894548) | > loss_mel_ce: 2.1528167724609375 (nan) | > loss: 0.008622308261692524 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13658580765301415) | > loader_time: 0.0045 (0.004660925334962957)  --> TIME: 2025-02-26 03:17:54 -- STEP: 29836/36938 -- GLOBAL_STEP: 1064100 | > loss_text_ce: 0.035886090248823166 (0.03232959507903549) | > loss_mel_ce: 3.2799785137176514 (nan) | > loss: 0.013158193789422512 (nan) | > current_lr: 5e-06 | > step_time: 0.184 (0.1365977289098459) | > loader_time: 0.0048 (0.0046609332389207685)  --> TIME: 2025-02-26 03:18:03 -- STEP: 29886/36938 -- GLOBAL_STEP: 1064150 | > loss_text_ce: 0.027223197743296623 (0.032328828920549744) | > loss_mel_ce: 2.9330971240997314 (nan) | > loss: 0.011747303418815136 (nan) | > current_lr: 5e-06 | > step_time: 0.1434 (0.13659616392762924) | > loader_time: 0.0047 (0.004660981275674251)  --> TIME: 2025-02-26 03:18:12 -- STEP: 29936/36938 -- GLOBAL_STEP: 1064200 | > loss_text_ce: 0.02661105990409851 (0.032328606379046686) | > loss_mel_ce: 2.8090248107910156 (nan) | > loss: 0.011252524331212044 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13659025064482086) | > loader_time: 0.0047 (0.004660892875228963)  --> TIME: 2025-02-26 03:18:22 -- STEP: 29986/36938 -- GLOBAL_STEP: 1064250 | > loss_text_ce: 0.04314664751291275 (0.032330119847893565) | > loss_mel_ce: 2.377713203430176 (nan) | > loss: 0.009606586769223213 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13659650196682963) | > loader_time: 0.0047 (0.00466093127789368)  --> TIME: 2025-02-26 03:18:31 -- STEP: 30036/36938 -- GLOBAL_STEP: 1064300 | > loss_text_ce: 0.041646409779787064 (0.03233090118037898) | > loss_mel_ce: 2.5624561309814453 (nan) | > loss: 0.010333741083741188 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13659545902660583) | > loader_time: 0.0045 (0.004660883007298496)  --> TIME: 2025-02-26 03:18:41 -- STEP: 30086/36938 -- GLOBAL_STEP: 1064350 | > loss_text_ce: 0.03541908785700798 (0.03233146627611515) | > loss_mel_ce: 2.9296958446502686 (nan) | > loss: 0.011766329407691956 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.1365965746790398) | > loader_time: 0.0046 (0.004660866294287236)  --> TIME: 2025-02-26 03:18:50 -- STEP: 30136/36938 -- GLOBAL_STEP: 1064400 | > loss_text_ce: 0.038205988705158234 (0.03232981005570978) | > loss_mel_ce: 1.5784672498703003 (nan) | > loss: 0.006415370386093855 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13659334943026996) | > loader_time: 0.0045 (0.004660822310686942)  --> TIME: 2025-02-26 03:18:59 -- STEP: 30186/36938 -- GLOBAL_STEP: 1064450 | > loss_text_ce: 0.0325692854821682 (0.032329050923623454) | > loss_mel_ce: 2.759341239929199 (nan) | > loss: 0.011079010553658009 (nan) | > current_lr: 5e-06 | > step_time: 0.1785 (0.136586216092704) | > loader_time: 0.0048 (0.0046607253882106755)  --> TIME: 2025-02-26 03:19:08 -- STEP: 30236/36938 -- GLOBAL_STEP: 1064500 | > loss_text_ce: 0.031288281083106995 (0.032329870200837114) | > loss_mel_ce: 2.1702635288238525 (nan) | > loss: 0.008736317977309227 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.1365895378698466) | > loader_time: 0.0045 (0.004660615152681239)  --> TIME: 2025-02-26 03:19:18 -- STEP: 30286/36938 -- GLOBAL_STEP: 1064550 | > loss_text_ce: 0.03494644537568092 (0.032330711082599724) | > loss_mel_ce: 2.113386631011963 (nan) | > loss: 0.008525131270289421 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13658847787897932) | > loader_time: 0.0045 (0.004660568022864951)  --> TIME: 2025-02-26 03:19:27 -- STEP: 30336/36938 -- GLOBAL_STEP: 1064600 | > loss_text_ce: 0.028063228353857994 (0.03232961137562488) | > loss_mel_ce: 0.018890380859375 (nan) | > loss: 0.00018632385763339698 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13659630662404224) | > loader_time: 0.0047 (0.0046605578061774885)  --> TIME: 2025-02-26 03:19:36 -- STEP: 30386/36938 -- GLOBAL_STEP: 1064650 | > loss_text_ce: 0.03403870388865471 (0.03232804044732272) | > loss_mel_ce: 0.10514068603515625 (nan) | > loss: 0.0005522991996258497 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1365958814175947) | > loader_time: 0.0046 (0.004660531883375338)  --> TIME: 2025-02-26 03:19:46 -- STEP: 30436/36938 -- GLOBAL_STEP: 1064700 | > loss_text_ce: 0.030748235061764717 (0.03232850231221253) | > loss_mel_ce: 1.5119181871414185 (nan) | > loss: 0.006121692713350058 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13659491902786608) | > loader_time: 0.0047 (0.004660501228179117)  --> TIME: 2025-02-26 03:19:55 -- STEP: 30486/36938 -- GLOBAL_STEP: 1064750 | > loss_text_ce: 0.03770718351006508 (0.03232889135668396) | > loss_mel_ce: 2.0893077850341797 (nan) | > loss: 0.008440535515546799 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13658689218486397) | > loader_time: 0.0047 (0.004660417782871669)  --> TIME: 2025-02-26 03:20:04 -- STEP: 30536/36938 -- GLOBAL_STEP: 1064800 | > loss_text_ce: 0.03722583130002022 (0.03233019682277218) | > loss_mel_ce: 3.0600178241729736 (nan) | > loss: 0.01229065004736185 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13659105082753947) | > loader_time: 0.0046 (0.004660364678619988)  --> TIME: 2025-02-26 03:20:14 -- STEP: 30586/36938 -- GLOBAL_STEP: 1064850 | > loss_text_ce: 0.024087829515337944 (0.03233005081092453) | > loss_mel_ce: 1.937950849533081 (nan) | > loss: 0.007785867899656296 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13658829705208536) | > loader_time: 0.0045 (0.004660327462757787)  --> TIME: 2025-02-26 03:20:23 -- STEP: 30636/36938 -- GLOBAL_STEP: 1064900 | > loss_text_ce: 0.03276524692773819 (0.03233020628864419) | > loss_mel_ce: 2.560621976852417 (nan) | > loss: 0.01029121968895197 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13659414214368373) | > loader_time: 0.0045 (0.004660248733061795)  --> TIME: 2025-02-26 03:20:32 -- STEP: 30686/36938 -- GLOBAL_STEP: 1064950 | > loss_text_ce: 0.023437300696969032 (0.03233006827888996) | > loss_mel_ce: 0.0322265625 (nan) | > loss: 0.00022088835248723626 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13659364167259438) | > loader_time: 0.0046 (0.004660147510482078)  --> TIME: 2025-02-26 03:20:42 -- STEP: 30736/36938 -- GLOBAL_STEP: 1065000 | > loss_text_ce: 0.03298358991742134 (0.03232912989782016) | > loss_mel_ce: 2.9305760860443115 (nan) | > loss: 0.011760157532989979 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13658997634467643) | > loader_time: 0.0046 (0.004660028388325224)  --> TIME: 2025-02-26 03:20:51 -- STEP: 30786/36938 -- GLOBAL_STEP: 1065050 | > loss_text_ce: 0.032809287309646606 (0.03232946403602966) | > loss_mel_ce: 2.305445671081543 (nan) | > loss: 0.009278790093958378 (nan) | > current_lr: 5e-06 | > step_time: 0.1621 (0.13658292574824768) | > loader_time: 0.0048 (0.004659896673518369)  --> TIME: 2025-02-26 03:21:00 -- STEP: 30836/36938 -- GLOBAL_STEP: 1065100 | > loss_text_ce: 0.03443155810236931 (0.03232751394030228) | > loss_mel_ce: 3.0184311866760254 (nan) | > loss: 0.012114535085856915 (nan) | > current_lr: 5e-06 | > step_time: 0.1659 (0.13658759274330134) | > loader_time: 0.0046 (0.004659815403038564)  --> TIME: 2025-02-26 03:21:10 -- STEP: 30886/36938 -- GLOBAL_STEP: 1065150 | > loss_text_ce: 0.02572856843471527 (0.03232846173837964) | > loss_mel_ce: 3.468308210372925 (nan) | > loss: 0.013865225948393345 (nan) | > current_lr: 5e-06 | > step_time: 0.1914 (0.13659728400132962) | > loader_time: 0.005 (0.004659834252671018)  --> TIME: 2025-02-26 03:21:19 -- STEP: 30936/36938 -- GLOBAL_STEP: 1065200 | > loss_text_ce: 0.03380121290683746 (0.032329431369762435) | > loss_mel_ce: 3.526644706726074 (nan) | > loss: 0.01412875484675169 (nan) | > current_lr: 5e-06 | > step_time: 0.1565 (0.13660094349274135) | > loader_time: 0.0046 (0.00465979767548567)  --> TIME: 2025-02-26 03:21:28 -- STEP: 30986/36938 -- GLOBAL_STEP: 1065250 | > loss_text_ce: 0.01879100315272808 (0.032327875110672775) | > loss_mel_ce: 1.6899971961975098 (nan) | > loss: 0.006780905649065971 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1365918593556583) | > loader_time: 0.0047 (0.004659681348505817)  --> TIME: 2025-02-26 03:21:37 -- STEP: 31036/36938 -- GLOBAL_STEP: 1065300 | > loss_text_ce: 0.031493861228227615 (0.032327351400784916) | > loss_mel_ce: 2.4300713539123535 (nan) | > loss: 0.009768116287887096 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.13658364559821026) | > loader_time: 0.0047 (0.004659796563109297)  --> TIME: 2025-02-26 03:21:47 -- STEP: 31086/36938 -- GLOBAL_STEP: 1065350 | > loss_text_ce: 0.03196410462260246 (0.032328236284082254) | > loss_mel_ce: 2.195814371109009 (nan) | > loss: 0.008840391412377357 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13658335366427515) | > loader_time: 0.0045 (0.004659687468791302)  --> TIME: 2025-02-26 03:21:56 -- STEP: 31136/36938 -- GLOBAL_STEP: 1065400 | > loss_text_ce: 0.02794475480914116 (0.0323260170065853) | > loss_mel_ce: 0.04856109619140625 (nan) | > loss: 0.0003035946865566075 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13658005666879997) | > loader_time: 0.0045 (0.004659566166834628)  --> TIME: 2025-02-26 03:22:05 -- STEP: 31186/36938 -- GLOBAL_STEP: 1065450 | > loss_text_ce: 0.03500703349709511 (0.03232625187609359) | > loss_mel_ce: 1.4994964599609375 (nan) | > loss: 0.006089299917221069 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.13658007163092808) | > loader_time: 0.0045 (0.004659524976438864)  --> TIME: 2025-02-26 03:22:15 -- STEP: 31236/36938 -- GLOBAL_STEP: 1065500 | > loss_text_ce: 0.03749167174100876 (0.03232696719090692) | > loss_mel_ce: 3.0646588802337646 (nan) | > loss: 0.012310121208429337 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13658076644569864) | > loader_time: 0.0045 (0.004659464454235661)  --> TIME: 2025-02-26 03:22:24 -- STEP: 31286/36938 -- GLOBAL_STEP: 1065550 | > loss_text_ce: 0.038105256855487823 (0.03232720030857995) | > loss_mel_ce: 2.533632755279541 (nan) | > loss: 0.010205309838056564 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13658540552019124) | > loader_time: 0.0045 (0.004659513016456709)  --> TIME: 2025-02-26 03:22:33 -- STEP: 31336/36938 -- GLOBAL_STEP: 1065600 | > loss_text_ce: 0.023401722311973572 (0.03232727720084928) | > loss_mel_ce: 2.3619048595428467 (nan) | > loss: 0.009465502575039864 (nan) | > current_lr: 5e-06 | > step_time: 0.1355 (0.13658386811268036) | > loader_time: 0.0048 (0.004659435602667268)  --> TIME: 2025-02-26 03:22:42 -- STEP: 31386/36938 -- GLOBAL_STEP: 1065650 | > loss_text_ce: 0.028431730344891548 (0.032324924768353094) | > loss_mel_ce: 0.007781982421875 (nan) | > loss: 0.0001437052123947069 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13657330832380996) | > loader_time: 0.0045 (0.004659353262424203)  --> TIME: 2025-02-26 03:22:51 -- STEP: 31436/36938 -- GLOBAL_STEP: 1065700 | > loss_text_ce: 0.025336837396025658 (0.03232461645193289) | > loss_mel_ce: 1.580450415611267 (nan) | > loss: 0.0063721719197928905 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13657066032986498) | > loader_time: 0.0045 (0.004659306443303372)  --> TIME: 2025-02-26 03:23:01 -- STEP: 31486/36938 -- GLOBAL_STEP: 1065750 | > loss_text_ce: 0.04300442710518837 (0.032325109151524735) | > loss_mel_ce: 0.38015982508659363 (nan) | > loss: 0.0016792232636362314 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13656856753521468) | > loader_time: 0.0046 (0.004659174759687988)  --> TIME: 2025-02-26 03:23:10 -- STEP: 31536/36938 -- GLOBAL_STEP: 1065800 | > loss_text_ce: 0.030888838693499565 (0.032325152644162955) | > loss_mel_ce: 2.1257574558258057 (nan) | > loss: 0.00855812057852745 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13656626390751764) | > loader_time: 0.0046 (0.004659092619841267)  --> TIME: 2025-02-26 03:23:19 -- STEP: 31586/36938 -- GLOBAL_STEP: 1065850 | > loss_text_ce: 0.032947979867458344 (0.03232398577870841) | > loss_mel_ce: 2.1699776649475098 (nan) | > loss: 0.008741769008338451 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13656097951325338) | > loader_time: 0.0046 (0.004659039929074203)  --> TIME: 2025-02-26 03:23:28 -- STEP: 31636/36938 -- GLOBAL_STEP: 1065900 | > loss_text_ce: 0.038311827927827835 (0.03232523466460454) | > loss_mel_ce: 3.5231778621673584 (nan) | > loss: 0.014132896438241005 (nan) | > current_lr: 5e-06 | > step_time: 0.1613 (0.13656768658351565) | > loader_time: 0.005 (0.004659073763391683)  --> TIME: 2025-02-26 03:23:38 -- STEP: 31686/36938 -- GLOBAL_STEP: 1065950 | > loss_text_ce: 0.033509399741888046 (0.03232498803825659) | > loss_mel_ce: 2.296542167663574 (nan) | > loss: 0.009246237576007843 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13656593469149086) | > loader_time: 0.0045 (0.0046591078746742514)  --> TIME: 2025-02-26 03:23:47 -- STEP: 31736/36938 -- GLOBAL_STEP: 1066000 | > loss_text_ce: 0.04055747017264366 (0.032324603409697) | > loss_mel_ce: 2.7117130756378174 (nan) | > loss: 0.01092170923948288 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.1365585865273072) | > loader_time: 0.005 (0.004659110408361782)  --> TIME: 2025-02-26 03:23:56 -- STEP: 31786/36938 -- GLOBAL_STEP: 1066050 | > loss_text_ce: 0.024772552773356438 (0.03232401077206507) | > loss_mel_ce: 2.209787607192993 (nan) | > loss: 0.008867302909493446 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.1365568896623511) | > loader_time: 0.0047 (0.004659003813286413)  --> TIME: 2025-02-26 03:24:05 -- STEP: 31836/36938 -- GLOBAL_STEP: 1066100 | > loss_text_ce: 0.03127697855234146 (0.03232465443484351) | > loss_mel_ce: 2.910611629486084 (nan) | > loss: 0.01167416200041771 (nan) | > current_lr: 5e-06 | > step_time: 0.1352 (0.13654875771785957) | > loader_time: 0.0046 (0.00465892222916486)  --> TIME: 2025-02-26 03:24:14 -- STEP: 31886/36938 -- GLOBAL_STEP: 1066150 | > loss_text_ce: 0.0405261255800724 (0.032322993949704545) | > loss_mel_ce: 2.589170217514038 (nan) | > loss: 0.010435303673148155 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1365369962552784) | > loader_time: 0.0045 (0.004658790631567326)  --> TIME: 2025-02-26 03:24:23 -- STEP: 31936/36938 -- GLOBAL_STEP: 1066200 | > loss_text_ce: 0.027011116966605186 (0.032323792221463456) | > loss_mel_ce: 2.7343385219573975 (nan) | > loss: 0.01095773745328188 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13653674303649366) | > loader_time: 0.0045 (0.004658820014260343)  --> TIME: 2025-02-26 03:24:32 -- STEP: 31986/36938 -- GLOBAL_STEP: 1066250 | > loss_text_ce: 0.032118745148181915 (0.03232329173283116) | > loss_mel_ce: 3.1504898071289062 (nan) | > loss: 0.012629400007426739 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13653995130878183) | > loader_time: 0.0046 (0.00465870332905733)  --> TIME: 2025-02-26 03:24:42 -- STEP: 32036/36938 -- GLOBAL_STEP: 1066300 | > loss_text_ce: 0.019580531865358353 (0.03232128128120929) | > loss_mel_ce: 1.4986343383789062 (nan) | > loss: 0.006024662405252457 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.136535594437749) | > loader_time: 0.0045 (0.004658657649525223)  --> TIME: 2025-02-26 03:24:51 -- STEP: 32086/36938 -- GLOBAL_STEP: 1066350 | > loss_text_ce: 0.0361795574426651 (0.03232274142996995) | > loss_mel_ce: 1.9413878917694092 (nan) | > loss: 0.007847490720450878 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13653764185715117) | > loader_time: 0.0045 (0.004658661124668082)  --> TIME: 2025-02-26 03:25:00 -- STEP: 32136/36938 -- GLOBAL_STEP: 1066400 | > loss_text_ce: 0.04278791323304176 (0.03232258925543791) | > loss_mel_ce: 2.9706063270568848 (nan) | > loss: 0.01195791456848383 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.13653898877174192) | > loader_time: 0.0047 (0.004658629533987549)  --> TIME: 2025-02-26 03:25:10 -- STEP: 32186/36938 -- GLOBAL_STEP: 1066450 | > loss_text_ce: 0.031258631497621536 (0.03232203292468733) | > loss_mel_ce: 2.567584753036499 (nan) | > loss: 0.01031287107616663 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13654007417896735) | > loader_time: 0.0045 (0.004658487484156296)  --> TIME: 2025-02-26 03:25:20 -- STEP: 32236/36938 -- GLOBAL_STEP: 1066500 | > loss_text_ce: 0.03233901411294937 (0.03232238727878792) | > loss_mel_ce: 2.717473030090332 (nan) | > loss: 0.010911953635513783 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.1365519237956174) | > loader_time: 0.0049 (0.004658421358912506)  --> TIME: 2025-02-26 03:25:29 -- STEP: 32286/36938 -- GLOBAL_STEP: 1066550 | > loss_text_ce: 0.02803710848093033 (0.032321613349911925) | > loss_mel_ce: 2.621556282043457 (nan) | > loss: 0.01051426026970148 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1365439559509063) | > loader_time: 0.0045 (0.004658340322242493)  --> TIME: 2025-02-26 03:25:38 -- STEP: 32336/36938 -- GLOBAL_STEP: 1066600 | > loss_text_ce: 0.03447316214442253 (0.03232174664126056) | > loss_mel_ce: 2.421687602996826 (nan) | > loss: 0.009746670722961426 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13654125406269213) | > loader_time: 0.0046 (0.004658389407063057)  --> TIME: 2025-02-26 03:25:47 -- STEP: 32386/36938 -- GLOBAL_STEP: 1066650 | > loss_text_ce: 0.025927845388650894 (0.032321478029331475) | > loss_mel_ce: 2.7750635147094727 (nan) | > loss: 0.01111504528671503 (nan) | > current_lr: 5e-06 | > step_time: 0.1866 (0.13653968057513072) | > loader_time: 0.0048 (0.0046583464726799315)  --> TIME: 2025-02-26 03:25:56 -- STEP: 32436/36938 -- GLOBAL_STEP: 1066700 | > loss_text_ce: 0.0271085686981678 (0.03232272527955438) | > loss_mel_ce: 2.7702078819274902 (nan) | > loss: 0.011100463569164276 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.1365335458288494) | > loader_time: 0.0045 (0.00465840688541501)  --> TIME: 2025-02-26 03:26:06 -- STEP: 32486/36938 -- GLOBAL_STEP: 1066750 | > loss_text_ce: 0.03288356214761734 (0.032323176663077886) | > loss_mel_ce: 2.7112395763397217 (nan) | > loss: 0.010889378376305103 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13653462977345837) | > loader_time: 0.0046 (0.004658302503112713)  --> TIME: 2025-02-26 03:26:15 -- STEP: 32536/36938 -- GLOBAL_STEP: 1066800 | > loss_text_ce: 0.052188046276569366 (0.03232153969776376) | > loss_mel_ce: 0.01708984375 (nan) | > loss: 0.0002749122795648873 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13652764819516816) | > loader_time: 0.0046 (0.004658214130535278)  --> TIME: 2025-02-26 03:26:24 -- STEP: 32586/36938 -- GLOBAL_STEP: 1066850 | > loss_text_ce: 0.02785232663154602 (0.03231940267635623) | > loss_mel_ce: 2.7596170902252197 (nan) | > loss: 0.01106138713657856 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.1365263995446476) | > loader_time: 0.0047 (0.004658101657576414)  --> TIME: 2025-02-26 03:26:33 -- STEP: 32636/36938 -- GLOBAL_STEP: 1066900 | > loss_text_ce: 0.03488729149103165 (0.03231983205764062) | > loss_mel_ce: 2.664602041244507 (nan) | > loss: 0.010712260380387306 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13652037150399188) | > loader_time: 0.0045 (0.004658098357582029)  --> TIME: 2025-02-26 03:26:42 -- STEP: 32686/36938 -- GLOBAL_STEP: 1066950 | > loss_text_ce: 0.035021405667066574 (0.032318437374496856) | > loss_mel_ce: 2.8609275817871094 (nan) | > loss: 0.011491861194372177 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13651707703704302) | > loader_time: 0.0047 (0.004657968950776373)  --> TIME: 2025-02-26 03:26:51 -- STEP: 32736/36938 -- GLOBAL_STEP: 1067000 | > loss_text_ce: 0.028153108432888985 (0.03231761709743753) | > loss_mel_ce: 2.3300609588623047 (nan) | > loss: 0.009357993490993977 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13651134247071528) | > loader_time: 0.0046 (0.004657867265353578)  --> TIME: 2025-02-26 03:27:00 -- STEP: 32786/36938 -- GLOBAL_STEP: 1067050 | > loss_text_ce: 0.041118133813142776 (0.03231775745926317) | > loss_mel_ce: 2.523329973220825 (nan) | > loss: 0.010176382027566433 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13650975893904263) | > loader_time: 0.0045 (0.004657775001849343)  --> TIME: 2025-02-26 03:27:10 -- STEP: 32836/36938 -- GLOBAL_STEP: 1067100 | > loss_text_ce: 0.028537923470139503 (0.03231764167257289) | > loss_mel_ce: 2.727607488632202 (nan) | > loss: 0.010937086306512356 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13651357646516654) | > loader_time: 0.0045 (0.004657753725873239)  --> TIME: 2025-02-26 03:27:19 -- STEP: 32886/36938 -- GLOBAL_STEP: 1067150 | > loss_text_ce: 0.030458087101578712 (0.03231891526551527) | > loss_mel_ce: 2.357113838195801 (nan) | > loss: 0.009474491700530052 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13651415366206784) | > loader_time: 0.0046 (0.0046576882977559094)  --> TIME: 2025-02-26 03:27:28 -- STEP: 32936/36938 -- GLOBAL_STEP: 1067200 | > loss_text_ce: 0.03153407573699951 (0.03231743755832065) | > loss_mel_ce: 2.8210484981536865 (nan) | > loss: 0.011319772340357304 (nan) | > current_lr: 5e-06 | > step_time: 0.1689 (0.1365094535525129) | > loader_time: 0.0046 (0.004657495237537638)  --> TIME: 2025-02-26 03:27:38 -- STEP: 32986/36938 -- GLOBAL_STEP: 1067250 | > loss_text_ce: 0.04200207069516182 (0.03231779853804303) | > loss_mel_ce: 2.74609375 (nan) | > loss: 0.011063872836530209 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13650813428988479) | > loader_time: 0.0046 (0.0046573551285298016)  --> TIME: 2025-02-26 03:27:47 -- STEP: 33036/36938 -- GLOBAL_STEP: 1067300 | > loss_text_ce: 0.03270603343844414 (0.032318402718720815) | > loss_mel_ce: 3.332071304321289 (nan) | > loss: 0.013352291658520699 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13650593446291107) | > loader_time: 0.0045 (0.004657304767604491)  --> TIME: 2025-02-26 03:27:57 -- STEP: 33086/36938 -- GLOBAL_STEP: 1067350 | > loss_text_ce: 0.029991716146469116 (0.03231893805287318) | > loss_mel_ce: 3.184215784072876 (nan) | > loss: 0.012754792347550392 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.13651202080695726) | > loader_time: 0.0046 (0.004657252649294459)  --> TIME: 2025-02-26 03:28:06 -- STEP: 33136/36938 -- GLOBAL_STEP: 1067400 | > loss_text_ce: 0.0364876314997673 (0.032319179290751114) | > loss_mel_ce: 3.3461859226226807 (nan) | > loss: 0.013423308730125427 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13650695160067466) | > loader_time: 0.0047 (0.0046571697059430645)  --> TIME: 2025-02-26 03:28:15 -- STEP: 33186/36938 -- GLOBAL_STEP: 1067450 | > loss_text_ce: 0.030002713203430176 (0.032319764322635655) | > loss_mel_ce: 2.884089946746826 (nan) | > loss: 0.01156385987997055 (nan) | > current_lr: 5e-06 | > step_time: 0.1688 (0.13650873974967126) | > loader_time: 0.0048 (0.00465712011783628)  --> TIME: 2025-02-26 03:28:24 -- STEP: 33236/36938 -- GLOBAL_STEP: 1067500 | > loss_text_ce: 0.033746980130672455 (0.03231950666755594) | > loss_mel_ce: 2.6353282928466797 (nan) | > loss: 0.010591569356620312 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.1365097977148397) | > loader_time: 0.0046 (0.004657053971838651)  --> TIME: 2025-02-26 03:28:34 -- STEP: 33286/36938 -- GLOBAL_STEP: 1067550 | > loss_text_ce: 0.03093155100941658 (0.032318289092638425) | > loss_mel_ce: 1.89276123046875 (nan) | > loss: 0.007633701898157597 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.1365079279738997) | > loader_time: 0.0046 (0.004656960268988464)  --> TIME: 2025-02-26 03:28:43 -- STEP: 33336/36938 -- GLOBAL_STEP: 1067600 | > loss_text_ce: 0.03261100500822067 (0.03231917220563788) | > loss_mel_ce: 2.654125928878784 (nan) | > loss: 0.010661655105650425 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13650510289384846) | > loader_time: 0.0045 (0.004656891392837598)  --> TIME: 2025-02-26 03:28:52 -- STEP: 33386/36938 -- GLOBAL_STEP: 1067650 | > loss_text_ce: 0.029243268072605133 (0.032319058133443074) | > loss_mel_ce: 1.2565785646438599 (nan) | > loss: 0.0051024677231907845 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.1365041507552316) | > loader_time: 0.0045 (0.004656813482180462)  --> TIME: 2025-02-26 03:29:02 -- STEP: 33436/36938 -- GLOBAL_STEP: 1067700 | > loss_text_ce: 0.04113377630710602 (0.032319638930458326) | > loss_mel_ce: 3.116563558578491 (nan) | > loss: 0.012530545704066753 (nan) | > current_lr: 5e-06 | > step_time: 0.144 (0.13650717766950535) | > loader_time: 0.0046 (0.004656755328107108)  --> TIME: 2025-02-26 03:29:11 -- STEP: 33486/36938 -- GLOBAL_STEP: 1067750 | > loss_text_ce: 0.028419798240065575 (0.03231807303292724) | > loss_mel_ce: 2.3183224201202393 (nan) | > loss: 0.009312469512224197 (nan) | > current_lr: 5e-06 | > step_time: 0.1438 (0.13650977185057028) | > loader_time: 0.0046 (0.004656686297541264)  --> TIME: 2025-02-26 03:29:21 -- STEP: 33536/36938 -- GLOBAL_STEP: 1067800 | > loss_text_ce: 0.035331498831510544 (0.03231806955928713) | > loss_mel_ce: 2.6768369674682617 (nan) | > loss: 0.010762574151158333 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13651519982528315) | > loader_time: 0.0046 (0.004656692526040194)  --> TIME: 2025-02-26 03:29:30 -- STEP: 33586/36938 -- GLOBAL_STEP: 1067850 | > loss_text_ce: 0.023231584578752518 (0.03231760748686553) | > loss_mel_ce: 2.302294969558716 (nan) | > loss: 0.009228279814124107 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.136512296746262) | > loader_time: 0.0048 (0.004656619506858207)  --> TIME: 2025-02-26 03:29:39 -- STEP: 33636/36938 -- GLOBAL_STEP: 1067900 | > loss_text_ce: 0.02973744086921215 (0.03231815754004061) | > loss_mel_ce: 2.3489699363708496 (nan) | > loss: 0.009439315646886826 (nan) | > current_lr: 5e-06 | > step_time: 0.1712 (0.13651327048999465) | > loader_time: 0.0048 (0.004656592203896265)  --> TIME: 2025-02-26 03:29:48 -- STEP: 33686/36938 -- GLOBAL_STEP: 1067950 | > loss_text_ce: 0.03141787275671959 (0.03231665746736605) | > loss_mel_ce: 0.08030395209789276 (nan) | > loss: 0.00044334062840789557 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13651211163511306) | > loader_time: 0.0045 (0.004656638901229292)  --> TIME: 2025-02-26 03:29:58 -- STEP: 33736/36938 -- GLOBAL_STEP: 1068000 | > loss_text_ce: 0.030964480713009834 (0.03231652350293194) | > loss_mel_ce: 2.8947503566741943 (nan) | > loss: 0.011609979905188084 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13651313145267707) | > loader_time: 0.0046 (0.0046565198406439395)  --> TIME: 2025-02-26 03:30:07 -- STEP: 33786/36938 -- GLOBAL_STEP: 1068050 | > loss_text_ce: 0.022453827783465385 (0.03231528147165859) | > loss_mel_ce: 0.24458007514476776 (nan) | > loss: 0.0010596583597362041 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13650994647186626) | > loader_time: 0.0045 (0.004656402402665663)  --> TIME: 2025-02-26 03:30:16 -- STEP: 33836/36938 -- GLOBAL_STEP: 1068100 | > loss_text_ce: 0.0318317711353302 (0.032314941754394616) | > loss_mel_ce: 2.756208658218384 (nan) | > loss: 0.0110636530444026 (nan) | > current_lr: 5e-06 | > step_time: 0.1434 (0.13650780792199307) | > loader_time: 0.0046 (0.004656257958033934)  --> TIME: 2025-02-26 03:30:26 -- STEP: 33886/36938 -- GLOBAL_STEP: 1068150 | > loss_text_ce: 0.03012055531144142 (0.03231495045117939) | > loss_mel_ce: 2.78117299079895 (nan) | > loss: 0.011155927553772926 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1365139499603084) | > loader_time: 0.0045 (0.004656248072106543)  --> TIME: 2025-02-26 03:30:35 -- STEP: 33936/36938 -- GLOBAL_STEP: 1068200 | > loss_text_ce: 0.03236079961061478 (0.03231508146078931) | > loss_mel_ce: 3.0339815616607666 (nan) | > loss: 0.012168025597929955 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13651643459496954) | > loader_time: 0.0047 (0.0046561565575089425)  --> TIME: 2025-02-26 03:30:44 -- STEP: 33986/36938 -- GLOBAL_STEP: 1068250 | > loss_text_ce: 0.03521941602230072 (0.03231403962288082) | > loss_mel_ce: 2.7650680541992188 (nan) | > loss: 0.011112252250313759 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13651463243656065) | > loader_time: 0.0045 (0.0046560449400426945)  --> TIME: 2025-02-26 03:30:54 -- STEP: 34036/36938 -- GLOBAL_STEP: 1068300 | > loss_text_ce: 0.03682873770594597 (0.03231455598235605) | > loss_mel_ce: 2.77046799659729 (nan) | > loss: 0.011140067130327225 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13651405034775432) | > loader_time: 0.0047 (0.0046559442769403886)  --> TIME: 2025-02-26 03:31:03 -- STEP: 34086/36938 -- GLOBAL_STEP: 1068350 | > loss_text_ce: 0.03697405755519867 (0.03231480256873434) | > loss_mel_ce: 0.9802029728889465 (nan) | > loss: 0.00403641676530242 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13651758032646008) | > loader_time: 0.0046 (0.0046559219830949634)  --> TIME: 2025-02-26 03:31:12 -- STEP: 34136/36938 -- GLOBAL_STEP: 1068400 | > loss_text_ce: 0.03356560692191124 (0.03231504227978768) | > loss_mel_ce: 2.450681209564209 (nan) | > loss: 0.00985812209546566 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.1365144960154481) | > loader_time: 0.0045 (0.004655882181874322)  --> TIME: 2025-02-26 03:31:21 -- STEP: 34186/36938 -- GLOBAL_STEP: 1068450 | > loss_text_ce: 0.029733628034591675 (0.03231404744066458) | > loss_mel_ce: 0.01946258544921875 (nan) | > loss: 0.00019522308139130473 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13650885692134196) | > loader_time: 0.0045 (0.004655755375892428)  --> TIME: 2025-02-26 03:31:31 -- STEP: 34236/36938 -- GLOBAL_STEP: 1068500 | > loss_text_ce: 0.03573021665215492 (0.032315138903008266) | > loss_mel_ce: 2.302370071411133 (nan) | > loss: 0.009278176352381706 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13650497826533955) | > loader_time: 0.0046 (0.004655670842523827)  --> TIME: 2025-02-26 03:31:40 -- STEP: 34286/36938 -- GLOBAL_STEP: 1068550 | > loss_text_ce: 0.02400769665837288 (0.032313647268201635) | > loss_mel_ce: 3.024780750274658 (nan) | > loss: 0.012098368257284164 (nan) | > current_lr: 5e-06 | > step_time: 0.199 (0.13649896776245915) | > loader_time: 0.0049 (0.004655579212477786)  --> TIME: 2025-02-26 03:31:49 -- STEP: 34336/36938 -- GLOBAL_STEP: 1068600 | > loss_text_ce: 0.031397659331560135 (0.032313196438153294) | > loss_mel_ce: 2.9559082984924316 (nan) | > loss: 0.011854388751089573 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.13649775153104274) | > loader_time: 0.0048 (0.004655536850923049)  --> TIME: 2025-02-26 03:31:59 -- STEP: 34386/36938 -- GLOBAL_STEP: 1068650 | > loss_text_ce: 0.031016455963253975 (0.032314366421274866) | > loss_mel_ce: 2.838430643081665 (nan) | > loss: 0.011386695317924023 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.1365060333749969) | > loader_time: 0.0045 (0.004655459140293245)  --> TIME: 2025-02-26 03:32:08 -- STEP: 34436/36938 -- GLOBAL_STEP: 1068700 | > loss_text_ce: 0.03916744142770767 (0.032314639209564185) | > loss_mel_ce: 2.688765525817871 (nan) | > loss: 0.010825131088495255 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13650204819909734) | > loader_time: 0.0046 (0.004655292099507276)  --> TIME: 2025-02-26 03:32:17 -- STEP: 34486/36938 -- GLOBAL_STEP: 1068750 | > loss_text_ce: 0.03128266707062721 (0.0323133821288914) | > loss_mel_ce: 2.6909019947052 (nan) | > loss: 0.010802321135997772 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13650248452087035) | > loader_time: 0.0048 (0.004655533362892311)  --> TIME: 2025-02-26 03:32:26 -- STEP: 34536/36938 -- GLOBAL_STEP: 1068800 | > loss_text_ce: 0.029351331293582916 (0.0323163855616404) | > loss_mel_ce: 0.9935329556465149 (nan) | > loss: 0.004059064667671919 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13650275005050683) | > loader_time: 0.0045 (0.004655430341501451)  --> TIME: 2025-02-26 03:32:36 -- STEP: 34586/36938 -- GLOBAL_STEP: 1068850 | > loss_text_ce: 0.029279986396431923 (0.03231689448058315) | > loss_mel_ce: 2.8057093620300293 (nan) | > loss: 0.011249958537518978 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1365048112377359) | > loader_time: 0.0045 (0.004655347112797825)  --> TIME: 2025-02-26 03:32:45 -- STEP: 34636/36938 -- GLOBAL_STEP: 1068900 | > loss_text_ce: 0.02558785304427147 (0.032317133565326235) | > loss_mel_ce: 2.4480490684509277 (nan) | > loss: 0.009816019795835018 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1365035127679326) | > loader_time: 0.0045 (0.004655212022812445)  --> TIME: 2025-02-26 03:32:55 -- STEP: 34686/36938 -- GLOBAL_STEP: 1068950 | > loss_text_ce: 0.041886162012815475 (0.032319611513088924) | > loss_mel_ce: 2.7328600883483887 (nan) | > loss: 0.011010898277163506 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.13650817994956652) | > loader_time: 0.0047 (0.0046552395604785995)  --> TIME: 2025-02-26 03:33:04 -- STEP: 34736/36938 -- GLOBAL_STEP: 1069000 | > loss_text_ce: 0.03493112698197365 (0.03231926193254731) | > loss_mel_ce: 2.2098939418792725 (nan) | > loss: 0.008908037096261978 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.1365110173183139) | > loader_time: 0.0046 (0.00465521696167354)  --> TIME: 2025-02-26 03:33:13 -- STEP: 34786/36938 -- GLOBAL_STEP: 1069050 | > loss_text_ce: 0.02095096930861473 (0.0323178426611748) | > loss_mel_ce: 3.03133225440979 (nan) | > loss: 0.012112235650420189 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1365095252433657) | > loader_time: 0.0046 (0.004655334027374304)  --> TIME: 2025-02-26 03:33:23 -- STEP: 34836/36938 -- GLOBAL_STEP: 1069100 | > loss_text_ce: 0.03141013905405998 (0.03231943580488509) | > loss_mel_ce: 1.02534818649292 (nan) | > loss: 0.004193485714495182 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13650812936001802) | > loader_time: 0.0045 (0.004655296827980553)  --> TIME: 2025-02-26 03:33:32 -- STEP: 34886/36938 -- GLOBAL_STEP: 1069150 | > loss_text_ce: 0.035420313477516174 (0.03231979245119994) | > loss_mel_ce: 2.3069863319396973 (nan) | > loss: 0.009295265190303326 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13650814898345154) | > loader_time: 0.0045 (0.004655249996455407)  --> TIME: 2025-02-26 03:33:41 -- STEP: 34936/36938 -- GLOBAL_STEP: 1069200 | > loss_text_ce: 0.032746873795986176 (0.032319077883640794) | > loss_mel_ce: 2.9933807849884033 (nan) | > loss: 0.012008443474769592 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.1365113841919559) | > loader_time: 0.005 (0.0046552278328357)  --> TIME: 2025-02-26 03:33:51 -- STEP: 34986/36938 -- GLOBAL_STEP: 1069250 | > loss_text_ce: 0.034849751740694046 (0.0323204051249531) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13651080018543452) | > loader_time: 0.0045 (0.00465521790359386)  --> TIME: 2025-02-26 03:34:00 -- STEP: 35036/36938 -- GLOBAL_STEP: 1069300 | > loss_text_ce: 0.023135852068662643 (0.03231858993909519) | > loss_mel_ce: 2.5171072483062744 (nan) | > loss: 0.010080330073833466 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13651110111598874) | > loader_time: 0.0045 (0.004655256590106647)  --> TIME: 2025-02-26 03:34:09 -- STEP: 35086/36938 -- GLOBAL_STEP: 1069350 | > loss_text_ce: 0.0336528904736042 (0.03231864666564066) | > loss_mel_ce: 2.854444980621338 (nan) | > loss: 0.011460705660283566 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13651649708528685) | > loader_time: 0.0048 (0.004655293936414961)  --> TIME: 2025-02-26 03:34:19 -- STEP: 35136/36938 -- GLOBAL_STEP: 1069400 | > loss_text_ce: 0.025112198665738106 (0.032318728024232454) | > loss_mel_ce: 1.9011255502700806 (nan) | > loss: 0.0076438006944954395 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13652396467404246) | > loader_time: 0.0053 (0.004655323657996031)  --> TIME: 2025-02-26 03:34:28 -- STEP: 35186/36938 -- GLOBAL_STEP: 1069450 | > loss_text_ce: 0.03316164016723633 (0.0323179412862212) | > loss_mel_ce: 2.2894034385681152 (nan) | > loss: 0.00921652838587761 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13652312449989543) | > loader_time: 0.0047 (0.004655249196186298)  --> TIME: 2025-02-26 03:34:38 -- STEP: 35236/36938 -- GLOBAL_STEP: 1069500 | > loss_text_ce: 0.03066529706120491 (0.032319368200166884) | > loss_mel_ce: 3.056640625 (nan) | > loss: 0.012251215055584908 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.1365256625632464) | > loader_time: 0.0045 (0.004655192842655673)  --> TIME: 2025-02-26 03:34:47 -- STEP: 35286/36938 -- GLOBAL_STEP: 1069550 | > loss_text_ce: 0.03993288800120354 (0.03231836369462837) | > loss_mel_ce: 3.0332798957824707 (nan) | > loss: 0.012195290066301823 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.13652681341546016) | > loader_time: 0.0048 (0.004655116885344772)  --> TIME: 2025-02-26 03:34:57 -- STEP: 35336/36938 -- GLOBAL_STEP: 1069600 | > loss_text_ce: 0.03283688798546791 (0.03231645629436092) | > loss_mel_ce: 2.5781569480895996 (nan) | > loss: 0.010361087508499622 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.1365405943751594) | > loader_time: 0.0046 (0.0046551211106445394)  --> TIME: 2025-02-26 03:35:06 -- STEP: 35386/36938 -- GLOBAL_STEP: 1069650 | > loss_text_ce: 0.032662373036146164 (0.03231743437113066) | > loss_mel_ce: 2.0045366287231445 (nan) | > loss: 0.008084123954176903 (nan) | > current_lr: 5e-06 | > step_time: 0.1571 (0.13653885420751605) | > loader_time: 0.0048 (0.004655002469644038)  --> TIME: 2025-02-26 03:35:15 -- STEP: 35436/36938 -- GLOBAL_STEP: 1069700 | > loss_text_ce: 0.036269962787628174 (0.03231927453984577) | > loss_mel_ce: 3.0474231243133545 (nan) | > loss: 0.012236877344548702 (nan) | > current_lr: 5e-06 | > step_time: 0.1788 (0.13653487356646604) | > loader_time: 0.0048 (0.004654918968146013)  --> TIME: 2025-02-26 03:35:25 -- STEP: 35486/36938 -- GLOBAL_STEP: 1069750 | > loss_text_ce: 0.03678159788250923 (0.03231831382316707) | > loss_mel_ce: 2.7196269035339355 (nan) | > loss: 0.010938129387795925 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13654141077253454) | > loader_time: 0.0045 (0.00465492403897589)  --> TIME: 2025-02-26 03:35:34 -- STEP: 35536/36938 -- GLOBAL_STEP: 1069800 | > loss_text_ce: 0.03647459298372269 (0.0323184271120573) | > loss_mel_ce: 3.0407402515411377 (nan) | > loss: 0.012211170978844166 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13654340223947392) | > loader_time: 0.0045 (0.004654965815056789)  --> TIME: 2025-02-26 03:35:44 -- STEP: 35586/36938 -- GLOBAL_STEP: 1069850 | > loss_text_ce: 0.03482475131750107 (0.03231950205506737) | > loss_mel_ce: 3.216073989868164 (nan) | > loss: 0.012900392524898052 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.13654171924047742) | > loader_time: 0.0046 (0.004654938311853696)  --> TIME: 2025-02-26 03:35:53 -- STEP: 35636/36938 -- GLOBAL_STEP: 1069900 | > loss_text_ce: 0.026979705318808556 (0.032318469359604605) | > loss_mel_ce: 3.4229423999786377 (nan) | > loss: 0.013690168038010597 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13655261100777133) | > loader_time: 0.0046 (0.0046549866343738285)  --> TIME: 2025-02-26 03:36:03 -- STEP: 35686/36938 -- GLOBAL_STEP: 1069950 | > loss_text_ce: 0.024463560432195663 (0.03231709187644228) | > loss_mel_ce: 2.4377551078796387 (nan) | > loss: 0.0097707100212574 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13655264837381376) | > loader_time: 0.0045 (0.00465494300434157)  --> TIME: 2025-02-26 03:36:12 -- STEP: 35736/36938 -- GLOBAL_STEP: 1070000 | > loss_text_ce: 0.038777001202106476 (0.032317049275499936) | > loss_mel_ce: 3.006258964538574 (nan) | > loss: 0.012083477340638638 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13654991143737888) | > loader_time: 0.0045 (0.004654885078933606) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1070000.pth  --> TIME: 2025-02-26 03:36:27 -- STEP: 35786/36938 -- GLOBAL_STEP: 1070050 | > loss_text_ce: 0.03571093827486038 (0.0323171626043416) | > loss_mel_ce: 0.0476226806640625 (nan) | > loss: 0.0003306889848317951 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13654378045033805) | > loader_time: 0.0046 (0.004654883185793715)  --> TIME: 2025-02-26 03:36:36 -- STEP: 35836/36938 -- GLOBAL_STEP: 1070100 | > loss_text_ce: 0.0295194610953331 (0.03231803644324023) | > loss_mel_ce: 3.1085259914398193 (nan) | > loss: 0.012452562339603901 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1365397819944215) | > loader_time: 0.0047 (0.004654784775536074)  --> TIME: 2025-02-26 03:36:45 -- STEP: 35886/36938 -- GLOBAL_STEP: 1070150 | > loss_text_ce: 0.03428884595632553 (0.032316711542179785) | > loss_mel_ce: 2.3934948444366455 (nan) | > loss: 0.009634062647819519 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13653945435862877) | > loader_time: 0.0045 (0.0046546970170882294)  --> TIME: 2025-02-26 03:36:54 -- STEP: 35936/36938 -- GLOBAL_STEP: 1070200 | > loss_text_ce: 0.03371630609035492 (0.032316995602980986) | > loss_mel_ce: 2.367574691772461 (nan) | > loss: 0.009528933092951775 (nan) | > current_lr: 5e-06 | > step_time: 0.1619 (0.1365374088791594) | > loader_time: 0.0046 (0.00465460524347779)  --> TIME: 2025-02-26 03:37:04 -- STEP: 35986/36938 -- GLOBAL_STEP: 1070250 | > loss_text_ce: 0.030592093244194984 (0.03231541963105678) | > loss_mel_ce: 0.8643455505371094 (nan) | > loss: 0.003551339963451028 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.1365368730302521) | > loader_time: 0.0044 (0.004654504952976399)  --> TIME: 2025-02-26 03:37:13 -- STEP: 36036/36938 -- GLOBAL_STEP: 1070300 | > loss_text_ce: 0.035343389958143234 (0.03231663911353778) | > loss_mel_ce: 3.3997416496276855 (nan) | > loss: 0.013631290756165981 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13653721071292785) | > loader_time: 0.0045 (0.004654435156611454)  --> TIME: 2025-02-26 03:37:22 -- STEP: 36086/36938 -- GLOBAL_STEP: 1070350 | > loss_text_ce: 0.01848568022251129 (0.03231650041312787) | > loss_mel_ce: 2.2502455711364746 (nan) | > loss: 0.009002902545034885 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13653046940909036) | > loader_time: 0.0046 (0.004654341788446018)  --> TIME: 2025-02-26 03:37:31 -- STEP: 36136/36938 -- GLOBAL_STEP: 1070400 | > loss_text_ce: 0.03565472736954689 (0.03231790021211832) | > loss_mel_ce: 3.0325803756713867 (nan) | > loss: 0.012175536714494228 (nan) | > current_lr: 5e-06 | > step_time: 0.1761 (0.13653006981452281) | > loader_time: 0.0047 (0.0046543391742668435)  --> TIME: 2025-02-26 03:37:41 -- STEP: 36186/36938 -- GLOBAL_STEP: 1070450 | > loss_text_ce: 0.032899513840675354 (0.03231632865424928) | > loss_mel_ce: 3.2261273860931396 (nan) | > loss: 0.012932647950947285 (nan) | > current_lr: 5e-06 | > step_time: 0.1444 (0.1365446474647971) | > loader_time: 0.0046 (0.004654370393680855)  --> TIME: 2025-02-26 03:37:51 -- STEP: 36236/36938 -- GLOBAL_STEP: 1070500 | > loss_text_ce: 0.03150079771876335 (0.03231515381133872) | > loss_mel_ce: 1.75290846824646 (nan) | > loss: 0.007080989424139261 (nan) | > current_lr: 5e-06 | > step_time: 0.174 (0.13654688974131798) | > loader_time: 0.0049 (0.004654484304953853)  --> TIME: 2025-02-26 03:38:00 -- STEP: 36286/36938 -- GLOBAL_STEP: 1070550 | > loss_text_ce: 0.041302405297756195 (0.03231519000069187) | > loss_mel_ce: 3.0447921752929688 (nan) | > loss: 0.012246407568454742 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.1365558890978915) | > loader_time: 0.0048 (0.004654504738627767)  --> TIME: 2025-02-26 03:38:10 -- STEP: 36336/36938 -- GLOBAL_STEP: 1070600 | > loss_text_ce: 0.03044249676167965 (0.032315572904456476) | > loss_mel_ce: 3.1019670963287354 (nan) | > loss: 0.012430197559297085 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.1365506066028283) | > loader_time: 0.0046 (0.004654415742461785)  --> TIME: 2025-02-26 03:38:19 -- STEP: 36386/36938 -- GLOBAL_STEP: 1070650 | > loss_text_ce: 0.032342322170734406 (0.0323149857941161) | > loss_mel_ce: 1.1712449789047241 (nan) | > loss: 0.004776140209287405 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13654634426118994) | > loader_time: 0.0044 (0.004654296810156661)  --> TIME: 2025-02-26 03:38:28 -- STEP: 36436/36938 -- GLOBAL_STEP: 1070700 | > loss_text_ce: 0.03638501092791557 (0.03231475963472716) | > loss_mel_ce: 3.058589220046997 (nan) | > loss: 0.012281645089387894 (nan) | > current_lr: 5e-06 | > step_time: 0.1617 (0.13655381345869458) | > loader_time: 0.0047 (0.004654295182225484)  --> TIME: 2025-02-26 03:38:38 -- STEP: 36486/36938 -- GLOBAL_STEP: 1070750 | > loss_text_ce: 0.02237628400325775 (0.03231310326204405) | > loss_mel_ce: 2.120821475982666 (nan) | > loss: 0.008504753932356834 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13655939597097724) | > loader_time: 0.0045 (0.004654296113754311)  --> TIME: 2025-02-26 03:38:47 -- STEP: 36536/36938 -- GLOBAL_STEP: 1070800 | > loss_text_ce: 0.031396299600601196 (0.0323128557415613) | > loss_mel_ce: 0.28015899658203125 (nan) | > loss: 0.001236330601386726 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.1365613315881736) | > loader_time: 0.0045 (0.004654246404235221)  --> TIME: 2025-02-26 03:38:57 -- STEP: 36586/36938 -- GLOBAL_STEP: 1070850 | > loss_text_ce: 0.02364976517856121 (0.03231198965086723) | > loss_mel_ce: 2.210982322692871 (nan) | > loss: 0.008867587894201279 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1365585986960433) | > loader_time: 0.0045 (0.004654209805259583)  --> TIME: 2025-02-26 03:39:06 -- STEP: 36636/36938 -- GLOBAL_STEP: 1070900 | > loss_text_ce: 0.03282089903950691 (0.032309960134675346) | > loss_mel_ce: 0.0024279754143208265 (nan) | > loss: 0.000139876501634717 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13655676148919035) | > loader_time: 0.0046 (0.004654206873248912)  --> TIME: 2025-02-26 03:39:15 -- STEP: 36686/36938 -- GLOBAL_STEP: 1070950 | > loss_text_ce: 0.036382969468832016 (0.03230975107323347) | > loss_mel_ce: 2.1807467937469482 (nan) | > loss: 0.00879813451319933 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13655091160640914) | > loader_time: 0.0045 (0.004654203819252458)  --> TIME: 2025-02-26 03:39:24 -- STEP: 36736/36938 -- GLOBAL_STEP: 1071000 | > loss_text_ce: 0.02164563350379467 (0.03231329489738386) | > loss_mel_ce: 1.25701904296875 (nan) | > loss: 0.005074066575616598 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13654463946533893) | > loader_time: 0.0047 (0.004654236585680633)  --> TIME: 2025-02-26 03:39:33 -- STEP: 36786/36938 -- GLOBAL_STEP: 1071050 | > loss_text_ce: 0.029187168926000595 (0.032312859447796605) | > loss_mel_ce: 3.0682551860809326 (nan) | > loss: 0.01229143887758255 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.13655027167246206) | > loader_time: 0.0047 (0.004654172835274612)  --> TIME: 2025-02-26 03:39:43 -- STEP: 36836/36938 -- GLOBAL_STEP: 1071100 | > loss_text_ce: 0.038045018911361694 (0.032314286007796976) | > loss_mel_ce: 2.3499233722686768 (nan) | > loss: 0.009476065635681152 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13655057356921865) | > loader_time: 0.0047 (0.0046542799813259015)  --> TIME: 2025-02-26 03:39:52 -- STEP: 36886/36938 -- GLOBAL_STEP: 1071150 | > loss_text_ce: 0.025362249463796616 (0.03231506558281431) | > loss_mel_ce: 3.160212278366089 (nan) | > loss: 0.012641169130802155 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13654865841347832) | > loader_time: 0.0045 (0.0046543135906981425)  --> TIME: 2025-02-26 03:40:01 -- STEP: 36936/36938 -- GLOBAL_STEP: 1071200 | > loss_text_ce: 0.027990682050585747 (0.03231406872411001) | > loss_mel_ce: 2.4054107666015625 (nan) | > loss: 0.009656354784965515 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13654497403305496) | > loader_time: 0.0048 (0.0046542532288566165)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.021623711774844935 (+0.0008097922447884258) | > avg_loss_text_ce: 0.03144679751505353 (-0.00022368086040906038) | > avg_loss_mel_ce: 2.4808858824838524 (-0.004368606180249923) | > avg_loss: 2.512332681086984 (-0.00459227792107475) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_1071202.pth  > EPOCH: 30/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-26 03:40:26)   --> TIME: 2025-02-26 03:40:36 -- STEP: 48/36938 -- GLOBAL_STEP: 1071250 | > loss_text_ce: 0.03205419331789017 (0.03218767271998029) | > loss_mel_ce: 2.9620015621185303 (2.6258681677281865) | > loss: 0.011881174519658089 (0.010547841268513972) | > current_lr: 5e-06 | > step_time: 0.1552 (0.13709983726342523) | > loader_time: 0.0047 (0.004726648330688476)  --> TIME: 2025-02-26 03:40:45 -- STEP: 98/36938 -- GLOBAL_STEP: 1071300 | > loss_text_ce: 0.029237467795610428 (0.03201472033195349) | > loss_mel_ce: 3.210092067718506 (2.614514585660429) | > loss: 0.012854482978582382 (0.010502101040958447) | > current_lr: 5e-06 | > step_time: 0.1632 (0.13740241527557376) | > loader_time: 0.0049 (0.004730915536685862)  --> TIME: 2025-02-26 03:40:54 -- STEP: 148/36938 -- GLOBAL_STEP: 1071350 | > loss_text_ce: 0.02571949176490307 (0.032308837826791635) | > loss_mel_ce: 1.2196451425552368 (nan) | > loss: 0.004941923543810844 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13800653251441744) | > loader_time: 0.0045 (0.004723043055147735)  --> TIME: 2025-02-26 03:41:04 -- STEP: 198/36938 -- GLOBAL_STEP: 1071400 | > loss_text_ce: 0.038279369473457336 (0.03239341265512537) | > loss_mel_ce: 2.51120662689209 (nan) | > loss: 0.010117008350789547 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1383350497544414) | > loader_time: 0.0046 (0.0047241955092458985)  --> TIME: 2025-02-26 03:41:13 -- STEP: 248/36938 -- GLOBAL_STEP: 1071450 | > loss_text_ce: 0.032945018261671066 (0.032548511926565396) | > loss_mel_ce: 3.0741281509399414 (nan) | > loss: 0.012329655699431896 (nan) | > current_lr: 5e-06 | > step_time: 0.1761 (0.13800797058689987) | > loader_time: 0.0049 (0.004718317139533254)  --> TIME: 2025-02-26 03:41:22 -- STEP: 298/36938 -- GLOBAL_STEP: 1071500 | > loss_text_ce: 0.024973710998892784 (0.032313130239662285) | > loss_mel_ce: 3.2060110569000244 (nan) | > loss: 0.012821368873119354 (nan) | > current_lr: 5e-06 | > step_time: 0.1508 (0.13780980622208366) | > loader_time: 0.0047 (0.004709094962817707)  --> TIME: 2025-02-26 03:41:32 -- STEP: 348/36938 -- GLOBAL_STEP: 1071550 | > loss_text_ce: 0.020680179819464684 (0.03215718771405944) | > loss_mel_ce: 1.927258849143982 (nan) | > loss: 0.007729917299002409 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13802862509913838) | > loader_time: 0.0046 (0.0047024913217829515)  --> TIME: 2025-02-26 03:41:41 -- STEP: 398/36938 -- GLOBAL_STEP: 1071600 | > loss_text_ce: 0.030433744192123413 (0.03229827170807811) | > loss_mel_ce: 1.8478515148162842 (nan) | > loss: 0.007453513331711292 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13783531752064013) | > loader_time: 0.0046 (0.004692442453087268)  --> TIME: 2025-02-26 03:41:50 -- STEP: 448/36938 -- GLOBAL_STEP: 1071650 | > loss_text_ce: 0.036522991955280304 (0.03214576683032124) | > loss_mel_ce: 2.9455559253692627 (nan) | > loss: 0.011833647266030312 (nan) | > current_lr: 5e-06 | > step_time: 0.1443 (0.1372019770954337) | > loader_time: 0.0046 (0.004684622266462865)  --> TIME: 2025-02-26 03:42:00 -- STEP: 498/36938 -- GLOBAL_STEP: 1071700 | > loss_text_ce: 0.030031396076083183 (0.032200638621850984) | > loss_mel_ce: 2.3029403686523438 (nan) | > loss: 0.009257825091481209 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1376452939098619) | > loader_time: 0.0046 (0.00468033528232191)  --> TIME: 2025-02-26 03:42:09 -- STEP: 548/36938 -- GLOBAL_STEP: 1071750 | > loss_text_ce: 0.04060520604252815 (0.03212168140941888) | > loss_mel_ce: 2.745396137237549 (nan) | > loss: 0.01105556171387434 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13739941459502622) | > loader_time: 0.0047 (0.004672268011274125)  --> TIME: 2025-02-26 03:42:18 -- STEP: 598/36938 -- GLOBAL_STEP: 1071800 | > loss_text_ce: 0.030857397243380547 (0.0320928177615646) | > loss_mel_ce: 0.7723338007926941 (nan) | > loss: 0.0031872668769210577 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13722173784887512) | > loader_time: 0.0046 (0.004664340944194473)  --> TIME: 2025-02-26 03:42:28 -- STEP: 648/36938 -- GLOBAL_STEP: 1071850 | > loss_text_ce: 0.03476227819919586 (0.03212443843087248) | > loss_mel_ce: 0.02703857421875 (nan) | > loss: 0.0002452415064908564 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13710823875886427) | > loader_time: 0.0045 (0.004658385927294505)  --> TIME: 2025-02-26 03:42:37 -- STEP: 698/36938 -- GLOBAL_STEP: 1071900 | > loss_text_ce: 0.035565875470638275 (0.03204963310490915) | > loss_mel_ce: 2.926337718963623 (nan) | > loss: 0.011753586120903492 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.13696196598446475) | > loader_time: 0.0046 (0.004653274159035235)  --> TIME: 2025-02-26 03:42:46 -- STEP: 748/36938 -- GLOBAL_STEP: 1071950 | > loss_text_ce: 0.025086116045713425 (0.032021451344275274) | > loss_mel_ce: 2.947326183319092 (nan) | > loss: 0.01179528795182705 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13691912328495714) | > loader_time: 0.0046 (0.004649285964149845)  --> TIME: 2025-02-26 03:42:55 -- STEP: 798/36938 -- GLOBAL_STEP: 1072000 | > loss_text_ce: 0.028346044942736626 (0.03206552109123209) | > loss_mel_ce: 0.7820078730583191 (nan) | > loss: 0.003215690376237035 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13679574575639333) | > loader_time: 0.0046 (0.004644848947835752)  --> TIME: 2025-02-26 03:43:05 -- STEP: 848/36938 -- GLOBAL_STEP: 1072050 | > loss_text_ce: 0.02620743215084076 (0.0320321564806432) | > loss_mel_ce: 3.0405032634735107 (nan) | > loss: 0.0121694877743721 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.13706464306363542) | > loader_time: 0.0048 (0.004641857911955642)  --> TIME: 2025-02-26 03:43:14 -- STEP: 898/36938 -- GLOBAL_STEP: 1072100 | > loss_text_ce: 0.029895557090640068 (0.03208061814295721) | > loss_mel_ce: 2.899894952774048 (nan) | > loss: 0.011626153253018856 (nan) | > current_lr: 5e-06 | > step_time: 0.1449 (0.1373344941234804) | > loader_time: 0.0046 (0.004639946538249746)  --> TIME: 2025-02-26 03:43:24 -- STEP: 948/36938 -- GLOBAL_STEP: 1072150 | > loss_text_ce: 0.03150634095072746 (0.032127810511463) | > loss_mel_ce: 3.035290002822876 (nan) | > loss: 0.012169827707111835 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13734979066164715) | > loader_time: 0.0045 (0.004640379032505218)  --> TIME: 2025-02-26 03:43:33 -- STEP: 998/36938 -- GLOBAL_STEP: 1072200 | > loss_text_ce: 0.03361448273062706 (0.032091910537436784) | > loss_mel_ce: 3.2098968029022217 (nan) | > loss: 0.01287107728421688 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.13720971310067062) | > loader_time: 0.0049 (0.004637969758563152)  --> TIME: 2025-02-26 03:43:42 -- STEP: 1048/36938 -- GLOBAL_STEP: 1072250 | > loss_text_ce: 0.05167150869965553 (0.03204402271107206) | > loss_mel_ce: 2.546630859375 (nan) | > loss: 0.010310724377632141 (nan) | > current_lr: 5e-06 | > step_time: 0.1494 (0.13733250016474569) | > loader_time: 0.0046 (0.0046361357201146775)  --> TIME: 2025-02-26 03:43:52 -- STEP: 1098/36938 -- GLOBAL_STEP: 1072300 | > loss_text_ce: 0.03185294196009636 (0.032029829928668374) | > loss_mel_ce: 2.912454605102539 (nan) | > loss: 0.011683761142194271 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.1372210013193296) | > loader_time: 0.0045 (0.00463236898238108)  --> TIME: 2025-02-26 03:44:01 -- STEP: 1148/36938 -- GLOBAL_STEP: 1072350 | > loss_text_ce: 0.030481182038784027 (0.032084352103182015) | > loss_mel_ce: 3.1277763843536377 (nan) | > loss: 0.01253276877105236 (nan) | > current_lr: 5e-06 | > step_time: 0.1567 (0.13714638861214265) | > loader_time: 0.0046 (0.004630437296026682)  --> TIME: 2025-02-26 03:44:10 -- STEP: 1198/36938 -- GLOBAL_STEP: 1072400 | > loss_text_ce: 0.033959947526454926 (0.0321163078459082) | > loss_mel_ce: 2.4122893810272217 (nan) | > loss: 0.009707339107990265 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13720939374328284) | > loader_time: 0.0045 (0.004628798002392707)  --> TIME: 2025-02-26 03:44:20 -- STEP: 1248/36938 -- GLOBAL_STEP: 1072450 | > loss_text_ce: 0.024620475247502327 (0.032152232214008446) | > loss_mel_ce: 2.936060905456543 (nan) | > loss: 0.01174873672425747 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.1373364751537643) | > loader_time: 0.0045 (0.004630481012356592)  --> TIME: 2025-02-26 03:44:29 -- STEP: 1298/36938 -- GLOBAL_STEP: 1072500 | > loss_text_ce: 0.026756230741739273 (0.03216354398470141) | > loss_mel_ce: 2.718440055847168 (nan) | > loss: 0.01089363731443882 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13734027178518962) | > loader_time: 0.0045 (0.004632611671471274)  --> TIME: 2025-02-26 03:44:39 -- STEP: 1348/36938 -- GLOBAL_STEP: 1072550 | > loss_text_ce: 0.030903080478310585 (0.032168140254212774) | > loss_mel_ce: 3.0728747844696045 (nan) | > loss: 0.012316579930484295 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.1373609987140057) | > loader_time: 0.0047 (0.004632343701156038)  --> TIME: 2025-02-26 03:44:48 -- STEP: 1398/36938 -- GLOBAL_STEP: 1072600 | > loss_text_ce: 0.03018064610660076 (0.03215838289447925) | > loss_mel_ce: 1.1164207458496094 (nan) | > loss: 0.004550005774945021 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13740027019053244) | > loader_time: 0.0044 (0.004631242868044188)  --> TIME: 2025-02-26 03:44:57 -- STEP: 1448/36938 -- GLOBAL_STEP: 1072650 | > loss_text_ce: 0.0361260250210762 (0.032213390640816644) | > loss_mel_ce: 2.7831313610076904 (nan) | > loss: 0.011187530122697353 (nan) | > current_lr: 5e-06 | > step_time: 0.1788 (0.1374217113737246) | > loader_time: 0.005 (0.004631710645243618)  --> TIME: 2025-02-26 03:45:06 -- STEP: 1498/36938 -- GLOBAL_STEP: 1072700 | > loss_text_ce: 0.031068073585629463 (0.032230662116693794) | > loss_mel_ce: 3.107541561126709 (nan) | > loss: 0.012454801239073277 (nan) | > current_lr: 5e-06 | > step_time: 0.1261 (0.13728509487233612) | > loader_time: 0.0046 (0.004631319415266914)  --> TIME: 2025-02-26 03:45:16 -- STEP: 1548/36938 -- GLOBAL_STEP: 1072750 | > loss_text_ce: 0.03378809988498688 (0.03222368730927597) | > loss_mel_ce: 2.9951364994049072 (nan) | > loss: 0.01201954297721386 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13716141875708163) | > loader_time: 0.0046 (0.004631044482691968)  --> TIME: 2025-02-26 03:45:25 -- STEP: 1598/36938 -- GLOBAL_STEP: 1072800 | > loss_text_ce: 0.026490049436688423 (0.03220917724564814) | > loss_mel_ce: 1.303466796875 (nan) | > loss: 0.005277607124298811 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.1371214373389236) | > loader_time: 0.0045 (0.00462911633287414)  --> TIME: 2025-02-26 03:45:34 -- STEP: 1648/36938 -- GLOBAL_STEP: 1072850 | > loss_text_ce: 0.030428249388933182 (0.03216692172635106) | > loss_mel_ce: 2.7908010482788086 (nan) | > loss: 0.011195355094969273 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13706259414987687) | > loader_time: 0.0046 (0.004628470366440937)  --> TIME: 2025-02-26 03:45:43 -- STEP: 1698/36938 -- GLOBAL_STEP: 1072900 | > loss_text_ce: 0.032090846449136734 (0.032176275545066385) | > loss_mel_ce: 1.6851400136947632 (nan) | > loss: 0.006814408581703901 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13693871823862422) | > loader_time: 0.0046 (0.0046273646843307085)  --> TIME: 2025-02-26 03:45:52 -- STEP: 1748/36938 -- GLOBAL_STEP: 1072950 | > loss_text_ce: 0.033829666674137115 (0.03220900951978454) | > loss_mel_ce: 3.116586446762085 (nan) | > loss: 0.012501652352511883 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13685839768132602) | > loader_time: 0.0046 (0.004626002546306069)  --> TIME: 2025-02-26 03:46:02 -- STEP: 1798/36938 -- GLOBAL_STEP: 1073000 | > loss_text_ce: 0.033619463443756104 (0.03219415067443342) | > loss_mel_ce: 3.351776123046875 (nan) | > loss: 0.013434110209345818 (nan) | > current_lr: 5e-06 | > step_time: 0.1567 (0.13684304988424054) | > loader_time: 0.0046 (0.0046241681223053634)  --> TIME: 2025-02-26 03:46:11 -- STEP: 1848/36938 -- GLOBAL_STEP: 1073050 | > loss_text_ce: 0.037234291434288025 (0.03222054419424218) | > loss_mel_ce: 2.8610503673553467 (nan) | > loss: 0.011501130647957325 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13687552427832722) | > loader_time: 0.0045 (0.004622535530106854)  --> TIME: 2025-02-26 03:46:21 -- STEP: 1898/36938 -- GLOBAL_STEP: 1073100 | > loss_text_ce: 0.02676384150981903 (0.032213701245358266) | > loss_mel_ce: 2.974066734313965 (nan) | > loss: 0.011908059008419514 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13704995686940097) | > loader_time: 0.0047 (0.004621615274186136)  --> TIME: 2025-02-26 03:46:30 -- STEP: 1948/36938 -- GLOBAL_STEP: 1073150 | > loss_text_ce: 0.03469081223011017 (0.03222276595778747) | > loss_mel_ce: 3.355113983154297 (nan) | > loss: 0.013451606966555119 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13694051636317942) | > loader_time: 0.0046 (0.004619643061557597)  --> TIME: 2025-02-26 03:46:39 -- STEP: 1998/36938 -- GLOBAL_STEP: 1073200 | > loss_text_ce: 0.029892200604081154 (0.03219662371135362) | > loss_mel_ce: 3.0480644702911377 (nan) | > loss: 0.012214114889502525 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13693639382466444) | > loader_time: 0.0046 (0.0046191263246584055)  --> TIME: 2025-02-26 03:46:48 -- STEP: 2048/36938 -- GLOBAL_STEP: 1073250 | > loss_text_ce: 0.03191043436527252 (0.03221038909214259) | > loss_mel_ce: 1.2624969482421875 (nan) | > loss: 0.005136537365615368 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13694734929595165) | > loader_time: 0.0045 (0.004618029575794942)  --> TIME: 2025-02-26 03:46:57 -- STEP: 2098/36938 -- GLOBAL_STEP: 1073300 | > loss_text_ce: 0.03179577365517616 (0.03221457926121438) | > loss_mel_ce: 2.4160053730010986 (nan) | > loss: 0.009713497012853622 (nan) | > current_lr: 5e-06 | > step_time: 0.1629 (0.1368234945093599) | > loader_time: 0.0047 (0.004617657515977661)  --> TIME: 2025-02-26 03:47:07 -- STEP: 2148/36938 -- GLOBAL_STEP: 1073350 | > loss_text_ce: 0.04035239666700363 (0.03219405921033864) | > loss_mel_ce: 3.5589826107025146 (nan) | > loss: 0.01428307592868805 (nan) | > current_lr: 5e-06 | > step_time: 0.1829 (0.13705815172284255) | > loader_time: 0.0048 (0.004619444992733187)  --> TIME: 2025-02-26 03:47:17 -- STEP: 2198/36938 -- GLOBAL_STEP: 1073400 | > loss_text_ce: 0.030606819316744804 (0.032209223065121924) | > loss_mel_ce: 2.341296672821045 (nan) | > loss: 0.009412315674126148 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13714457556157922) | > loader_time: 0.0045 (0.004619962633252693)  --> TIME: 2025-02-26 03:47:26 -- STEP: 2248/36938 -- GLOBAL_STEP: 1073450 | > loss_text_ce: 0.03702598065137863 (0.032270816648610964) | > loss_mel_ce: 2.1931910514831543 (nan) | > loss: 0.008850067853927612 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.1371439863990637) | > loader_time: 0.0045 (0.004619849322105227)  --> TIME: 2025-02-26 03:47:36 -- STEP: 2298/36938 -- GLOBAL_STEP: 1073500 | > loss_text_ce: 0.031082719564437866 (0.03224654049913297) | > loss_mel_ce: 2.920597553253174 (nan) | > loss: 0.011713017709553242 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.1371278513816258) | > loader_time: 0.0044 (0.004620537226671139)  --> TIME: 2025-02-26 03:47:45 -- STEP: 2348/36938 -- GLOBAL_STEP: 1073550 | > loss_text_ce: 0.03911225125193596 (0.0322716995729435) | > loss_mel_ce: 0.31979236006736755 (nan) | > loss: 0.001424224697984755 (nan) | > current_lr: 5e-06 | > step_time: 0.1082 (0.1370998349872117) | > loader_time: 0.0046 (0.004620717676575395)  --> TIME: 2025-02-26 03:47:54 -- STEP: 2398/36938 -- GLOBAL_STEP: 1073600 | > loss_text_ce: 0.029652422294020653 (0.03225177227503141) | > loss_mel_ce: 3.060720443725586 (nan) | > loss: 0.012263384647667408 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13703933877682495) | > loader_time: 0.0046 (0.004619976895565397)  --> TIME: 2025-02-26 03:48:04 -- STEP: 2448/36938 -- GLOBAL_STEP: 1073650 | > loss_text_ce: 0.032388828694820404 (0.032232076077253334) | > loss_mel_ce: 0.11279296875 (nan) | > loss: 0.0005761182983405888 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13716839224684502) | > loader_time: 0.0045 (0.004621013998985299)  --> TIME: 2025-02-26 03:48:13 -- STEP: 2498/36938 -- GLOBAL_STEP: 1073700 | > loss_text_ce: 0.029579350724816322 (0.03222868245712416) | > loss_mel_ce: 3.1115496158599854 (nan) | > loss: 0.012464798055589199 (nan) | > current_lr: 5e-06 | > step_time: 0.1583 (0.13710232941602338) | > loader_time: 0.0046 (0.00462014537701329)  --> TIME: 2025-02-26 03:48:22 -- STEP: 2548/36938 -- GLOBAL_STEP: 1073750 | > loss_text_ce: 0.030134186148643494 (0.032231523931681466) | > loss_mel_ce: 2.627244234085083 (nan) | > loss: 0.010545153170824051 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13712329383547722) | > loader_time: 0.0046 (0.004620535126660657)  --> TIME: 2025-02-26 03:48:32 -- STEP: 2598/36938 -- GLOBAL_STEP: 1073800 | > loss_text_ce: 0.030037229880690575 (0.03222414386172632) | > loss_mel_ce: 2.73982310295105 (nan) | > loss: 0.01099151000380516 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13705273671183282) | > loader_time: 0.0046 (0.004620581612943053)  --> TIME: 2025-02-26 03:48:41 -- STEP: 2648/36938 -- GLOBAL_STEP: 1073850 | > loss_text_ce: 0.03002089262008667 (0.03223707436462946) | > loss_mel_ce: 3.3313283920288086 (nan) | > loss: 0.013338688760995865 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.1371238605313435) | > loader_time: 0.0045 (0.004620833339287803)  --> TIME: 2025-02-26 03:48:50 -- STEP: 2698/36938 -- GLOBAL_STEP: 1073900 | > loss_text_ce: 0.03662832826375961 (0.0322273445306684) | > loss_mel_ce: 2.856809139251709 (nan) | > loss: 0.011481895111501217 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13713118860331033) | > loader_time: 0.0046 (0.0046207212005570115)  --> TIME: 2025-02-26 03:49:00 -- STEP: 2748/36938 -- GLOBAL_STEP: 1073950 | > loss_text_ce: 0.022500082850456238 (0.0322194169817717) | > loss_mel_ce: 3.0552866458892822 (nan) | > loss: 0.012213439680635929 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13711893220135254) | > loader_time: 0.0048 (0.004623950221826601)  --> TIME: 2025-02-26 03:49:09 -- STEP: 2798/36938 -- GLOBAL_STEP: 1074000 | > loss_text_ce: 0.04123016446828842 (0.032214241839939935) | > loss_mel_ce: 2.252384662628174 (nan) | > loss: 0.009101646952331066 (nan) | > current_lr: 5e-06 | > step_time: 0.1448 (0.1371846910542129) | > loader_time: 0.0049 (0.00462507937447015)  --> TIME: 2025-02-26 03:49:19 -- STEP: 2848/36938 -- GLOBAL_STEP: 1074050 | > loss_text_ce: 0.02393937110900879 (0.03222630280553978) | > loss_mel_ce: 2.3217196464538574 (nan) | > loss: 0.009308171458542347 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13725311443042257) | > loader_time: 0.0045 (0.004625301096546524)  --> TIME: 2025-02-26 03:49:28 -- STEP: 2898/36938 -- GLOBAL_STEP: 1074100 | > loss_text_ce: 0.03620157763361931 (0.032237768198284375) | > loss_mel_ce: 2.702335834503174 (nan) | > loss: 0.01086721196770668 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1372660846032304) | > loader_time: 0.0045 (0.004625961729540679)  --> TIME: 2025-02-26 03:49:38 -- STEP: 2948/36938 -- GLOBAL_STEP: 1074150 | > loss_text_ce: 0.034469809383153915 (0.032252932693986144) | > loss_mel_ce: 3.136397123336792 (nan) | > loss: 0.012582805939018726 (nan) | > current_lr: 5e-06 | > step_time: 0.1578 (0.1372515820421716) | > loader_time: 0.0046 (0.00462710372143233)  --> TIME: 2025-02-26 03:49:47 -- STEP: 2998/36938 -- GLOBAL_STEP: 1074200 | > loss_text_ce: 0.035065796226263046 (0.03222600129427381) | > loss_mel_ce: 2.8975863456726074 (nan) | > loss: 0.011637509800493717 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13723859602487298) | > loader_time: 0.0045 (0.004627432880439802)  --> TIME: 2025-02-26 03:49:56 -- STEP: 3048/36938 -- GLOBAL_STEP: 1074250 | > loss_text_ce: 0.03913178667426109 (0.032211752854387814) | > loss_mel_ce: 2.4553709030151367 (nan) | > loss: 0.009898820891976357 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13714302532629105) | > loader_time: 0.0045 (0.00462741864322053)  --> TIME: 2025-02-26 03:50:05 -- STEP: 3098/36938 -- GLOBAL_STEP: 1074300 | > loss_text_ce: 0.036933667957782745 (0.0322123058107209) | > loss_mel_ce: 2.8540279865264893 (nan) | > loss: 0.011472070589661598 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13712968104419754) | > loader_time: 0.0046 (0.0046277050051556955)  --> TIME: 2025-02-26 03:50:15 -- STEP: 3148/36938 -- GLOBAL_STEP: 1074350 | > loss_text_ce: 0.032390251755714417 (0.03222864204479242) | > loss_mel_ce: 2.1952874660491943 (nan) | > loss: 0.008839991874992847 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.1371640761440865) | > loader_time: 0.0044 (0.004627113193841586)  --> TIME: 2025-02-26 03:50:24 -- STEP: 3198/36938 -- GLOBAL_STEP: 1074400 | > loss_text_ce: 0.030835041776299477 (0.03223264909694766) | > loss_mel_ce: 2.535616636276245 (nan) | > loss: 0.010184332728385925 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1371755799030497) | > loader_time: 0.0045 (0.004627205641139982)  --> TIME: 2025-02-26 03:50:33 -- STEP: 3248/36938 -- GLOBAL_STEP: 1074450 | > loss_text_ce: 0.03168701007962227 (0.03224246588901478) | > loss_mel_ce: 2.203763484954834 (nan) | > loss: 0.008870835416018963 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13714714106080572) | > loader_time: 0.0048 (0.004627216405469229)  --> TIME: 2025-02-26 03:50:43 -- STEP: 3298/36938 -- GLOBAL_STEP: 1074500 | > loss_text_ce: 0.023130392655730247 (0.032226360870074515) | > loss_mel_ce: 2.8347439765930176 (nan) | > loss: 0.011340771801769733 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13715260067876717) | > loader_time: 0.0046 (0.0046274633101075465)  --> TIME: 2025-02-26 03:50:52 -- STEP: 3348/36938 -- GLOBAL_STEP: 1074550 | > loss_text_ce: 0.036129534244537354 (0.0322183341916156) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13716788438628366) | > loader_time: 0.0046 (0.004627258831739731)  --> TIME: 2025-02-26 03:51:01 -- STEP: 3398/36938 -- GLOBAL_STEP: 1074600 | > loss_text_ce: 0.03784506767988205 (0.03222458468385041) | > loss_mel_ce: 0.9630314707756042 (nan) | > loss: 0.003971732687205076 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13712448432489024) | > loader_time: 0.0045 (0.004628683203034313)  --> TIME: 2025-02-26 03:51:11 -- STEP: 3448/36938 -- GLOBAL_STEP: 1074650 | > loss_text_ce: 0.036821167916059494 (0.032206125697337044) | > loss_mel_ce: 3.0075581073760986 (nan) | > loss: 0.012080870568752289 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13717547736698957) | > loader_time: 0.0046 (0.004628157394391494)  --> TIME: 2025-02-26 03:51:20 -- STEP: 3498/36938 -- GLOBAL_STEP: 1074700 | > loss_text_ce: 0.032586708664894104 (0.03219819082913137) | > loss_mel_ce: 3.24277925491333 (nan) | > loss: 0.012997484765946865 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13717627668462523) | > loader_time: 0.0046 (0.004627388160251918)  --> TIME: 2025-02-26 03:51:30 -- STEP: 3548/36938 -- GLOBAL_STEP: 1074750 | > loss_text_ce: 0.025811493396759033 (0.032193755315198774) | > loss_mel_ce: 2.5163869857788086 (nan) | > loss: 0.010088089853525162 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13711105372752516) | > loader_time: 0.0046 (0.004626452385236891)  --> TIME: 2025-02-26 03:51:39 -- STEP: 3598/36938 -- GLOBAL_STEP: 1074800 | > loss_text_ce: 0.03352155536413193 (0.032182487352684364) | > loss_mel_ce: 1.190498948097229 (nan) | > loss: 0.00485722441226244 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13707718251479079) | > loader_time: 0.0047 (0.00462673391614643)  --> TIME: 2025-02-26 03:51:48 -- STEP: 3648/36938 -- GLOBAL_STEP: 1074850 | > loss_text_ce: 0.02844390831887722 (0.03219240248099111) | > loss_mel_ce: 2.9887545108795166 (nan) | > loss: 0.011973010376095772 (nan) | > current_lr: 5e-06 | > step_time: 0.1349 (0.13712974977597844) | > loader_time: 0.0047 (0.004627484436097912)  --> TIME: 2025-02-26 03:51:58 -- STEP: 3698/36938 -- GLOBAL_STEP: 1074900 | > loss_text_ce: 0.03151765838265419 (0.032191873305936015) | > loss_mel_ce: 1.38909912109375 (nan) | > loss: 0.0056373681873083115 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13716017348886225) | > loader_time: 0.0047 (0.004628054962859676)  --> TIME: 2025-02-26 03:52:07 -- STEP: 3748/36938 -- GLOBAL_STEP: 1074950 | > loss_text_ce: 0.035217031836509705 (0.032200026660839494) | > loss_mel_ce: 2.2547295093536377 (nan) | > loss: 0.009087090380489826 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13719027801918743) | > loader_time: 0.0045 (0.004628383426299868)  --> TIME: 2025-02-26 03:52:17 -- STEP: 3798/36938 -- GLOBAL_STEP: 1075000 | > loss_text_ce: 0.0247805155813694 (0.03218354441038178) | > loss_mel_ce: 2.532119035720825 (nan) | > loss: 0.0101464269682765 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13722324547107237) | > loader_time: 0.0045 (0.004627607570565847)  --> TIME: 2025-02-26 03:52:26 -- STEP: 3848/36938 -- GLOBAL_STEP: 1075050 | > loss_text_ce: 0.042470645159482956 (0.0321875592885372) | > loss_mel_ce: 3.142411708831787 (nan) | > loss: 0.012638422660529613 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13722197466729324) | > loader_time: 0.0045 (0.004626736261740557)  --> TIME: 2025-02-26 03:52:35 -- STEP: 3898/36938 -- GLOBAL_STEP: 1075100 | > loss_text_ce: 0.025530749931931496 (0.03219740487407677) | > loss_mel_ce: 1.210418939590454 (nan) | > loss: 0.004904562141746283 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13717409119720994) | > loader_time: 0.0044 (0.004625587478058361)  --> TIME: 2025-02-26 03:52:44 -- STEP: 3948/36938 -- GLOBAL_STEP: 1075150 | > loss_text_ce: 0.034460898488759995 (0.032181645037036376) | > loss_mel_ce: 2.860135078430176 (nan) | > loss: 0.011486492119729519 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.1371657293980606) | > loader_time: 0.0048 (0.004624417970728863)  --> TIME: 2025-02-26 03:52:54 -- STEP: 3998/36938 -- GLOBAL_STEP: 1075200 | > loss_text_ce: 0.03756953775882721 (0.03217904599431921) | > loss_mel_ce: 3.123056173324585 (nan) | > loss: 0.01254216581583023 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13717636476462325) | > loader_time: 0.0046 (0.004623662596526548)  --> TIME: 2025-02-26 03:53:03 -- STEP: 4048/36938 -- GLOBAL_STEP: 1075250 | > loss_text_ce: 0.03299567103385925 (0.03217265145598239) | > loss_mel_ce: 2.224088430404663 (nan) | > loss: 0.008956683799624443 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13719517593327235) | > loader_time: 0.0044 (0.004623075129957552)  --> TIME: 2025-02-26 03:53:13 -- STEP: 4098/36938 -- GLOBAL_STEP: 1075300 | > loss_text_ce: 0.027415746822953224 (0.032175157482568946) | > loss_mel_ce: 2.0993704795837402 (nan) | > loss: 0.008439628407359123 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13723480195287022) | > loader_time: 0.0045 (0.004622476458142258)  --> TIME: 2025-02-26 03:53:22 -- STEP: 4148/36938 -- GLOBAL_STEP: 1075350 | > loss_text_ce: 0.02807570807635784 (0.03218033685970403) | > loss_mel_ce: 7.21875 (nan) | > loss: 0.02875724621117115 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13717166121003702) | > loader_time: 0.0047 (0.0046214969431767824)  --> TIME: 2025-02-26 03:53:32 -- STEP: 4198/36938 -- GLOBAL_STEP: 1075400 | > loss_text_ce: 0.030041135847568512 (0.03218811909233762) | > loss_mel_ce: 0.5784072875976562 (nan) | > loss: 0.002414477989077568 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13723045522227287) | > loader_time: 0.0044 (0.004621117384221351)  --> TIME: 2025-02-26 03:53:41 -- STEP: 4248/36938 -- GLOBAL_STEP: 1075450 | > loss_text_ce: 0.02716488391160965 (0.032188128369769094) | > loss_mel_ce: 2.7113919258117676 (nan) | > loss: 0.010867290198802948 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13718131341727657) | > loader_time: 0.0047 (0.004620519173796356)  --> TIME: 2025-02-26 03:53:50 -- STEP: 4298/36938 -- GLOBAL_STEP: 1075500 | > loss_text_ce: 0.030216073617339134 (0.032185072724812684) | > loss_mel_ce: 2.4170806407928467 (nan) | > loss: 0.0097114946693182 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13719207378474418) | > loader_time: 0.0044 (0.004619927337525788)  --> TIME: 2025-02-26 03:54:00 -- STEP: 4348/36938 -- GLOBAL_STEP: 1075550 | > loss_text_ce: 0.028703812509775162 (0.03219211868716444) | > loss_mel_ce: 2.805881977081299 (nan) | > loss: 0.011248357594013214 (nan) | > current_lr: 5e-06 | > step_time: 0.1728 (0.13720981501173063) | > loader_time: 0.0047 (0.004619227436121834)  --> TIME: 2025-02-26 03:54:09 -- STEP: 4398/36938 -- GLOBAL_STEP: 1075600 | > loss_text_ce: 0.026733417063951492 (0.03220350596153355) | > loss_mel_ce: 1.8869391679763794 (nan) | > loss: 0.007593939080834389 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13718995857152017) | > loader_time: 0.0044 (0.00461846131961853)  --> TIME: 2025-02-26 03:54:18 -- STEP: 4448/36938 -- GLOBAL_STEP: 1075650 | > loss_text_ce: 0.020098524168133736 (0.03219530672499094) | > loss_mel_ce: 2.2866647243499756 (nan) | > loss: 0.009153822436928749 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13718699830041506) | > loader_time: 0.0045 (0.004617707924448339)  --> TIME: 2025-02-26 03:54:28 -- STEP: 4498/36938 -- GLOBAL_STEP: 1075700 | > loss_text_ce: 0.027861615642905235 (0.03218759483649075) | > loss_mel_ce: 2.1826536655426025 (nan) | > loss: 0.008771886117756367 (nan) | > current_lr: 5e-06 | > step_time: 0.1445 (0.1372102972346975) | > loader_time: 0.0047 (0.004617256554246967)  --> TIME: 2025-02-26 03:54:37 -- STEP: 4548/36938 -- GLOBAL_STEP: 1075750 | > loss_text_ce: 0.028831468895077705 (0.032173288007460894) | > loss_mel_ce: 2.2973709106445312 (nan) | > loss: 0.009230962023139 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.13724479611337229) | > loader_time: 0.0046 (0.004617254568383488)  --> TIME: 2025-02-26 03:54:46 -- STEP: 4598/36938 -- GLOBAL_STEP: 1075800 | > loss_text_ce: 0.029362499713897705 (0.03217334730636525) | > loss_mel_ce: 2.761564254760742 (nan) | > loss: 0.01107510644942522 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.13716578758400694) | > loader_time: 0.0048 (0.004616771121811182)  --> TIME: 2025-02-26 03:54:55 -- STEP: 4648/36938 -- GLOBAL_STEP: 1075850 | > loss_text_ce: 0.03453906998038292 (0.03216633919515066) | > loss_mel_ce: 2.0693023204803467 (nan) | > loss: 0.008348576724529266 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.137128984558808) | > loader_time: 0.0045 (0.0046163506536598745)  --> TIME: 2025-02-26 03:55:05 -- STEP: 4698/36938 -- GLOBAL_STEP: 1075900 | > loss_text_ce: 0.030841507017612457 (0.032171241805257086) | > loss_mel_ce: 2.797210931777954 (nan) | > loss: 0.011222431436181068 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13719850836941005) | > loader_time: 0.0046 (0.004616313408973023)  --> TIME: 2025-02-26 03:55:15 -- STEP: 4748/36938 -- GLOBAL_STEP: 1075950 | > loss_text_ce: 0.0368276946246624 (0.032191633940623327) | > loss_mel_ce: 0.9974368810653687 (nan) | > loss: 0.004104224499315023 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13726217794940568) | > loader_time: 0.0045 (0.004616490260030214)  --> TIME: 2025-02-26 03:55:24 -- STEP: 4798/36938 -- GLOBAL_STEP: 1076000 | > loss_text_ce: 0.02912093512713909 (0.032205638809385005) | > loss_mel_ce: 0.40693360567092896 (nan) | > loss: 0.001730375224724412 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13723013593833513) | > loader_time: 0.0045 (0.0046164186460964955)  --> TIME: 2025-02-26 03:55:33 -- STEP: 4848/36938 -- GLOBAL_STEP: 1076050 | > loss_text_ce: 0.03137733042240143 (0.03220801115233491) | > loss_mel_ce: 2.4617743492126465 (nan) | > loss: 0.009893459267914295 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13722055973392902) | > loader_time: 0.0045 (0.004615954636740616)  --> TIME: 2025-02-26 03:55:42 -- STEP: 4898/36938 -- GLOBAL_STEP: 1076100 | > loss_text_ce: 0.044224679470062256 (0.032210917578475405) | > loss_mel_ce: 1.6605485677719116 (nan) | > loss: 0.006764973513782024 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13712956064231258) | > loader_time: 0.0045 (0.004615162138356739)  --> TIME: 2025-02-26 03:55:51 -- STEP: 4948/36938 -- GLOBAL_STEP: 1076150 | > loss_text_ce: 0.0375836156308651 (0.032209089062181874) | > loss_mel_ce: 2.3085408210754395 (nan) | > loss: 0.00931001827120781 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13706952598466926) | > loader_time: 0.0045 (0.0046144235297953294)  --> TIME: 2025-02-26 03:56:00 -- STEP: 4998/36938 -- GLOBAL_STEP: 1076200 | > loss_text_ce: 0.03353650122880936 (0.03219664736813739) | > loss_mel_ce: 2.909757137298584 (nan) | > loss: 0.011679736897349358 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.137009560036249) | > loader_time: 0.0046 (0.004613808891018574)  --> TIME: 2025-02-26 03:56:09 -- STEP: 5048/36938 -- GLOBAL_STEP: 1076250 | > loss_text_ce: 0.034606628119945526 (0.03219390906214195) | > loss_mel_ce: 2.955092191696167 (nan) | > loss: 0.011863885447382927 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13699012588585616) | > loader_time: 0.0048 (0.004613365452565416)  --> TIME: 2025-02-26 03:56:19 -- STEP: 5098/36938 -- GLOBAL_STEP: 1076300 | > loss_text_ce: 0.02534335106611252 (0.03218347698734341) | > loss_mel_ce: 2.9245541095733643 (nan) | > loss: 0.011705943383276463 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.13704490310867978) | > loader_time: 0.0048 (0.004614015053935787)  --> TIME: 2025-02-26 03:56:28 -- STEP: 5148/36938 -- GLOBAL_STEP: 1076350 | > loss_text_ce: 0.034605514258146286 (0.03219156522356641) | > loss_mel_ce: 3.216818332672119 (nan) | > loss: 0.012902475893497467 (nan) | > current_lr: 5e-06 | > step_time: 0.1572 (0.13697351493294507) | > loader_time: 0.0047 (0.004614073867101314)  --> TIME: 2025-02-26 03:56:37 -- STEP: 5198/36938 -- GLOBAL_STEP: 1076400 | > loss_text_ce: 0.029426829889416695 (0.032195860740921074) | > loss_mel_ce: 2.693157196044922 (nan) | > loss: 0.010803905315697193 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13699353998007324) | > loader_time: 0.0046 (0.004614549675736353)  --> TIME: 2025-02-26 03:56:46 -- STEP: 5248/36938 -- GLOBAL_STEP: 1076450 | > loss_text_ce: 0.03179455175995827 (0.032193110477592386) | > loss_mel_ce: 2.669680595397949 (nan) | > loss: 0.010720140300691128 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13695326297566643) | > loader_time: 0.0047 (0.004615147257360021)  --> TIME: 2025-02-26 03:56:56 -- STEP: 5298/36938 -- GLOBAL_STEP: 1076500 | > loss_text_ce: 0.03415990248322487 (0.032204171822843325) | > loss_mel_ce: 2.7083983421325684 (nan) | > loss: 0.010883168317377567 (nan) | > current_lr: 5e-06 | > step_time: 0.1872 (0.1369315590125831) | > loader_time: 0.0048 (0.004616104687956792)  --> TIME: 2025-02-26 03:57:05 -- STEP: 5348/36938 -- GLOBAL_STEP: 1076550 | > loss_text_ce: 0.028990909457206726 (0.03220221649635255) | > loss_mel_ce: 2.836332321166992 (nan) | > loss: 0.011370331048965454 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13688992580194093) | > loader_time: 0.0046 (0.0046162934738393216)  --> TIME: 2025-02-26 03:57:14 -- STEP: 5398/36938 -- GLOBAL_STEP: 1076600 | > loss_text_ce: 0.028093451634049416 (0.03219669711579045) | > loss_mel_ce: 2.483213424682617 (nan) | > loss: 0.009965503588318825 (nan) | > current_lr: 5e-06 | > step_time: 0.1823 (0.13688794888139136) | > loader_time: 0.0048 (0.0046167154584208425)  --> TIME: 2025-02-26 03:57:24 -- STEP: 5448/36938 -- GLOBAL_STEP: 1076650 | > loss_text_ce: 0.01934933476150036 (0.032198304503611795) | > loss_mel_ce: 2.0024285316467285 (nan) | > loss: 0.008022928610444069 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13693255277982336) | > loader_time: 0.0045 (0.004616624764583892)  --> TIME: 2025-02-26 03:57:33 -- STEP: 5498/36938 -- GLOBAL_STEP: 1076700 | > loss_text_ce: 0.030978068709373474 (0.03220283019597985) | > loss_mel_ce: 2.906524181365967 (nan) | > loss: 0.011656755581498146 (nan) | > current_lr: 5e-06 | > step_time: 0.1903 (0.13698218067414777) | > loader_time: 0.0051 (0.0046173716597489485)  --> TIME: 2025-02-26 03:57:43 -- STEP: 5548/36938 -- GLOBAL_STEP: 1076750 | > loss_text_ce: 0.030429434031248093 (0.03220110399938622) | > loss_mel_ce: 2.692603349685669 (nan) | > loss: 0.010805686004459858 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.136976947695038) | > loader_time: 0.0047 (0.00461778459397683)  --> TIME: 2025-02-26 03:57:52 -- STEP: 5598/36938 -- GLOBAL_STEP: 1076800 | > loss_text_ce: 0.028939751908183098 (0.03220212358413066) | > loss_mel_ce: 2.515505313873291 (nan) | > loss: 0.010097004473209381 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13700454726564956) | > loader_time: 0.0046 (0.004618025243431391)  --> TIME: 2025-02-26 03:58:01 -- STEP: 5648/36938 -- GLOBAL_STEP: 1076850 | > loss_text_ce: 0.04323357343673706 (0.03220598904776099) | > loss_mel_ce: 3.063673257827759 (nan) | > loss: 0.012328996323049068 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13699306252151247) | > loader_time: 0.0045 (0.004618433776388096)  --> TIME: 2025-02-26 03:58:11 -- STEP: 5698/36938 -- GLOBAL_STEP: 1076900 | > loss_text_ce: 0.04378705471754074 (0.03220720575889005) | > loss_mel_ce: 1.8800591230392456 (nan) | > loss: 0.007634310517460108 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13697089392413284) | > loader_time: 0.007 (0.00461911293195817)  --> TIME: 2025-02-26 03:58:20 -- STEP: 5748/36938 -- GLOBAL_STEP: 1076950 | > loss_text_ce: 0.03537340462207794 (0.032214185723642215) | > loss_mel_ce: 2.8808677196502686 (nan) | > loss: 0.011572386138141155 (nan) | > current_lr: 5e-06 | > step_time: 0.1261 (0.13693393886130745) | > loader_time: 0.0046 (0.004619894794233167)  --> TIME: 2025-02-26 03:58:29 -- STEP: 5798/36938 -- GLOBAL_STEP: 1077000 | > loss_text_ce: 0.031956788152456284 (0.03220430402924958) | > loss_mel_ce: 2.9407756328582764 (nan) | > loss: 0.01179655734449625 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.1369295619034933) | > loader_time: 0.0045 (0.004620570731845643)  --> TIME: 2025-02-26 03:58:38 -- STEP: 5848/36938 -- GLOBAL_STEP: 1077050 | > loss_text_ce: 0.029958130791783333 (0.03219495849794093) | > loss_mel_ce: 3.129981279373169 (nan) | > loss: 0.01253944355994463 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.13687894437283665) | > loader_time: 0.0048 (0.004620132139704305)  --> TIME: 2025-02-26 03:58:47 -- STEP: 5898/36938 -- GLOBAL_STEP: 1077100 | > loss_text_ce: 0.013048354536294937 (0.03219633080932472) | > loss_mel_ce: 2.3864753246307373 (nan) | > loss: 0.00952192023396492 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.136892122792243) | > loader_time: 0.0044 (0.004620702721863213)  --> TIME: 2025-02-26 03:58:56 -- STEP: 5948/36938 -- GLOBAL_STEP: 1077150 | > loss_text_ce: 0.022095367312431335 (0.03219488001583255) | > loss_mel_ce: 2.3776872158050537 (nan) | > loss: 0.009522947482764721 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13686485200739512) | > loader_time: 0.0051 (0.004620616638027114)  --> TIME: 2025-02-26 03:59:06 -- STEP: 5998/36938 -- GLOBAL_STEP: 1077200 | > loss_text_ce: 0.03134239464998245 (0.032198746147695786) | > loss_mel_ce: 2.594249725341797 (nan) | > loss: 0.010419017635285854 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13690875390641966) | > loader_time: 0.0046 (0.004620763650533562)  --> TIME: 2025-02-26 03:59:16 -- STEP: 6048/36938 -- GLOBAL_STEP: 1077250 | > loss_text_ce: 0.027320126071572304 (0.03220323476941641) | > loss_mel_ce: 2.6540849208831787 (nan) | > loss: 0.01064049731940031 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.1369427340055893) | > loader_time: 0.0051 (0.004620762374350637)  --> TIME: 2025-02-26 03:59:25 -- STEP: 6098/36938 -- GLOBAL_STEP: 1077300 | > loss_text_ce: 0.032683633267879486 (0.032211324114813235) | > loss_mel_ce: 2.7006795406341553 (nan) | > loss: 0.010846680030226707 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13697402545294554) | > loader_time: 0.0044 (0.004620680030192188)  --> TIME: 2025-02-26 03:59:35 -- STEP: 6148/36938 -- GLOBAL_STEP: 1077350 | > loss_text_ce: 0.037994273006916046 (0.03220987774518976) | > loss_mel_ce: 2.664745569229126 (nan) | > loss: 0.01072515919804573 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13697978045619233) | > loader_time: 0.0045 (0.0046204834614159964)  --> TIME: 2025-02-26 03:59:44 -- STEP: 6198/36938 -- GLOBAL_STEP: 1077400 | > loss_text_ce: 0.032649531960487366 (0.03220571617849208) | > loss_mel_ce: 2.8913440704345703 (nan) | > loss: 0.011603149585425854 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13699645713745778) | > loader_time: 0.0045 (0.004620755438113763)  --> TIME: 2025-02-26 03:59:53 -- STEP: 6248/36938 -- GLOBAL_STEP: 1077450 | > loss_text_ce: 0.0418359600007534 (0.03220833556308628) | > loss_mel_ce: 2.9635612964630127 (nan) | > loss: 0.011926180683076382 (nan) | > current_lr: 5e-06 | > step_time: 0.1817 (0.1369613955863455) | > loader_time: 0.0048 (0.004620327496192829)  --> TIME: 2025-02-26 04:00:02 -- STEP: 6298/36938 -- GLOBAL_STEP: 1077500 | > loss_text_ce: 0.034195154905319214 (0.03222489820866632) | > loss_mel_ce: 2.9400079250335693 (nan) | > loss: 0.011802393943071365 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.13694748983719296) | > loader_time: 0.0049 (0.004620183381098501)  --> TIME: 2025-02-26 04:00:12 -- STEP: 6348/36938 -- GLOBAL_STEP: 1077550 | > loss_text_ce: 0.03772168979048729 (0.032225982186272546) | > loss_mel_ce: 1.4813368320465088 (nan) | > loss: 0.006028010044246912 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13696949900360872) | > loader_time: 0.0045 (0.004620127093514034)  --> TIME: 2025-02-26 04:00:22 -- STEP: 6398/36938 -- GLOBAL_STEP: 1077600 | > loss_text_ce: 0.03864515200257301 (0.032230891430562535) | > loss_mel_ce: 2.508126974105835 (nan) | > loss: 0.010106239467859268 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13702413401554459) | > loader_time: 0.0047 (0.004620312116562709)  --> TIME: 2025-02-26 04:00:31 -- STEP: 6448/36938 -- GLOBAL_STEP: 1077650 | > loss_text_ce: 0.025813646614551544 (0.032231076987675314) | > loss_mel_ce: 2.8300607204437256 (nan) | > loss: 0.01133283507078886 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13705170679624798) | > loader_time: 0.0045 (0.004620232815777996)  --> TIME: 2025-02-26 04:00:40 -- STEP: 6498/36938 -- GLOBAL_STEP: 1077700 | > loss_text_ce: 0.033520907163619995 (0.03222918801844392) | > loss_mel_ce: 3.054746627807617 (nan) | > loss: 0.012255030684173107 (nan) | > current_lr: 5e-06 | > step_time: 0.1779 (0.13703268188518763) | > loader_time: 0.0048 (0.004619953191328072)  --> TIME: 2025-02-26 04:00:49 -- STEP: 6548/36938 -- GLOBAL_STEP: 1077750 | > loss_text_ce: 0.03203707933425903 (0.032231807787962544) | > loss_mel_ce: 2.675828456878662 (nan) | > loss: 0.010745498351752758 (nan) | > current_lr: 5e-06 | > step_time: 0.1576 (0.13698904901724646) | > loader_time: 0.0046 (0.004619615465384407)  --> TIME: 2025-02-26 04:00:58 -- STEP: 6598/36938 -- GLOBAL_STEP: 1077800 | > loss_text_ce: 0.030918292701244354 (0.03222524100986882) | > loss_mel_ce: 2.300912618637085 (nan) | > loss: 0.009253297932446003 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13692351762870386) | > loader_time: 0.0046 (0.004619411317895284)  --> TIME: 2025-02-26 04:01:08 -- STEP: 6648/36938 -- GLOBAL_STEP: 1077850 | > loss_text_ce: 0.03476212918758392 (0.0322331437695645) | > loss_mel_ce: 3.2629454135894775 (nan) | > loss: 0.013086142018437386 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.1369157068781035) | > loader_time: 0.0049 (0.0046192836173318185)  --> TIME: 2025-02-26 04:01:17 -- STEP: 6698/36938 -- GLOBAL_STEP: 1077900 | > loss_text_ce: 0.022785313427448273 (0.03223407002128105) | > loss_mel_ce: 1.9372780323028564 (nan) | > loss: 0.007778029423207045 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1369054667378005) | > loader_time: 0.0044 (0.004619073782368693)  --> TIME: 2025-02-26 04:01:27 -- STEP: 6748/36938 -- GLOBAL_STEP: 1077950 | > loss_text_ce: 0.03316328302025795 (0.03223787018050473) | > loss_mel_ce: 3.184659004211426 (nan) | > loss: 0.01276913657784462 (nan) | > current_lr: 5e-06 | > step_time: 0.152 (0.13695245101456024) | > loader_time: 0.0046 (0.004619139676696121)  --> TIME: 2025-02-26 04:01:36 -- STEP: 6798/36938 -- GLOBAL_STEP: 1078000 | > loss_text_ce: 0.018681839108467102 (0.032231881360354465) | > loss_mel_ce: 1.799046516418457 (nan) | > loss: 0.007213208358734846 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.1369603697991988) | > loader_time: 0.0044 (0.004618885973756136)  --> TIME: 2025-02-26 04:01:45 -- STEP: 6848/36938 -- GLOBAL_STEP: 1078050 | > loss_text_ce: 0.02649707719683647 (0.032236333768629985) | > loss_mel_ce: 2.2686495780944824 (nan) | > loss: 0.009107725694775581 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13695387949592508) | > loader_time: 0.0047 (0.004618454237129101)  --> TIME: 2025-02-26 04:01:55 -- STEP: 6898/36938 -- GLOBAL_STEP: 1078100 | > loss_text_ce: 0.03758879378437996 (0.032227395295181) | > loss_mel_ce: 2.965724229812622 (nan) | > loss: 0.011917909607291222 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1369726100359206) | > loader_time: 0.0044 (0.004618265553466421)  --> TIME: 2025-02-26 04:02:04 -- STEP: 6948/36938 -- GLOBAL_STEP: 1078150 | > loss_text_ce: 0.02773701399564743 (0.03222055048142957) | > loss_mel_ce: 0.28233641386032104 (nan) | > loss: 0.0012304502306506038 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.1369623849978024) | > loader_time: 0.0045 (0.0046182626544159725)  --> TIME: 2025-02-26 04:02:13 -- STEP: 6998/36938 -- GLOBAL_STEP: 1078200 | > loss_text_ce: 0.03149965777993202 (0.032220473818169115) | > loss_mel_ce: 1.510137915611267 (nan) | > loss: 0.006117609795182943 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.1369517831401702) | > loader_time: 0.0047 (0.004618066384063936)  --> TIME: 2025-02-26 04:02:22 -- STEP: 7048/36938 -- GLOBAL_STEP: 1078250 | > loss_text_ce: 0.027925819158554077 (0.03221955532025741) | > loss_mel_ce: 2.8558733463287354 (nan) | > loss: 0.011443647556006908 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13692956543807594) | > loader_time: 0.0045 (0.004617758560397176)  --> TIME: 2025-02-26 04:02:32 -- STEP: 7098/36938 -- GLOBAL_STEP: 1078300 | > loss_text_ce: 0.03980632126331329 (0.03221653712830479) | > loss_mel_ce: 2.108668088912964 (nan) | > loss: 0.008525692857801914 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13695224226948566) | > loader_time: 0.0044 (0.004617863723881731)  --> TIME: 2025-02-26 04:02:42 -- STEP: 7148/36938 -- GLOBAL_STEP: 1078350 | > loss_text_ce: 0.04353521764278412 (0.032222702411442865) | > loss_mel_ce: 1.6486274003982544 (nan) | > loss: 0.0067149316892027855 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13699834451504572) | > loader_time: 0.0046 (0.004617954241075407)  --> TIME: 2025-02-26 04:02:51 -- STEP: 7198/36938 -- GLOBAL_STEP: 1078400 | > loss_text_ce: 0.038007523864507675 (0.03221818688306185) | > loss_mel_ce: 0.8468704223632812 (nan) | > loss: 0.003511420451104641 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13699880822826332) | > loader_time: 0.0045 (0.004618369529365877)  --> TIME: 2025-02-26 04:03:00 -- STEP: 7248/36938 -- GLOBAL_STEP: 1078450 | > loss_text_ce: 0.03138094022870064 (0.032212928011391626) | > loss_mel_ce: 2.7532107830047607 (nan) | > loss: 0.011049967259168625 (nan) | > current_lr: 5e-06 | > step_time: 0.165 (0.1369840185965917) | > loader_time: 0.0048 (0.0046185792216139)  --> TIME: 2025-02-26 04:03:10 -- STEP: 7298/36938 -- GLOBAL_STEP: 1078500 | > loss_text_ce: 0.029149876907467842 (0.03220322691008793) | > loss_mel_ce: 2.365469217300415 (nan) | > loss: 0.009502456523478031 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13700393148564885) | > loader_time: 0.0047 (0.004618523283571908)  --> TIME: 2025-02-26 04:03:19 -- STEP: 7348/36938 -- GLOBAL_STEP: 1078550 | > loss_text_ce: 0.029096517711877823 (0.032187999291933674) | > loss_mel_ce: 0.3919921815395355 (nan) | > loss: 0.0016709869960322976 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13697615530287052) | > loader_time: 0.0046 (0.004618425796261688)  --> TIME: 2025-02-26 04:03:28 -- STEP: 7398/36938 -- GLOBAL_STEP: 1078600 | > loss_text_ce: 0.02679464779794216 (0.03219535571774043) | > loss_mel_ce: 2.144055128097534 (nan) | > loss: 0.00861448422074318 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13693865375539374) | > loader_time: 0.0046 (0.0046182900836384735)  --> TIME: 2025-02-26 04:03:37 -- STEP: 7448/36938 -- GLOBAL_STEP: 1078650 | > loss_text_ce: 0.039780598133802414 (0.03219067938745863) | > loss_mel_ce: 2.799987316131592 (nan) | > loss: 0.011268921196460724 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.1369171997594265) | > loader_time: 0.0049 (0.004618641001069341)  --> TIME: 2025-02-26 04:03:47 -- STEP: 7498/36938 -- GLOBAL_STEP: 1078700 | > loss_text_ce: 0.02873351238667965 (0.03217901666525727) | > loss_mel_ce: 2.9976913928985596 (nan) | > loss: 0.01200962346047163 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.13693491692032936) | > loader_time: 0.0046 (0.00461861619188743)  --> TIME: 2025-02-26 04:03:56 -- STEP: 7548/36938 -- GLOBAL_STEP: 1078750 | > loss_text_ce: 0.03455546125769615 (0.03218315208464511) | > loss_mel_ce: 1.4063271284103394 (nan) | > loss: 0.005717788357287645 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.1369512971899272) | > loader_time: 0.0045 (0.004618507942701069)  --> TIME: 2025-02-26 04:04:06 -- STEP: 7598/36938 -- GLOBAL_STEP: 1078800 | > loss_text_ce: 0.03762109577655792 (0.03218173067850076) | > loss_mel_ce: 2.863191843032837 (nan) | > loss: 0.011511162854731083 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13698219277099927) | > loader_time: 0.0046 (0.004618434191817025)  --> TIME: 2025-02-26 04:04:15 -- STEP: 7648/36938 -- GLOBAL_STEP: 1078850 | > loss_text_ce: 0.045606959611177444 (0.03217753450772014) | > loss_mel_ce: 1.8161439895629883 (nan) | > loss: 0.0073879011906683445 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1369988164517664) | > loader_time: 0.0044 (0.0046183719420532935)  --> TIME: 2025-02-26 04:04:24 -- STEP: 7698/36938 -- GLOBAL_STEP: 1078900 | > loss_text_ce: 0.04025224223732948 (0.03217775835472953) | > loss_mel_ce: 2.737215995788574 (nan) | > loss: 0.011021699756383896 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13698117864320794) | > loader_time: 0.0045 (0.004618376717997341)  --> TIME: 2025-02-26 04:04:33 -- STEP: 7748/36938 -- GLOBAL_STEP: 1078950 | > loss_text_ce: 0.028119312599301338 (0.0321749509218456) | > loss_mel_ce: 0.011688232421875 (nan) | > loss: 0.00015796645311638713 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13695738776541214) | > loader_time: 0.0047 (0.004618109934220452)  --> TIME: 2025-02-26 04:04:43 -- STEP: 7798/36938 -- GLOBAL_STEP: 1079000 | > loss_text_ce: 0.032876156270504 (0.0321801800945324) | > loss_mel_ce: 3.2413523197174072 (nan) | > loss: 0.012992971576750278 (nan) | > current_lr: 5e-06 | > step_time: 0.1648 (0.13697753367774945) | > loader_time: 0.0046 (0.00461798281081368)  --> TIME: 2025-02-26 04:04:52 -- STEP: 7848/36938 -- GLOBAL_STEP: 1079050 | > loss_text_ce: 0.038416191935539246 (0.03218070679191398) | > loss_mel_ce: 2.6182892322540283 (nan) | > loss: 0.010542482137680054 (nan) | > current_lr: 5e-06 | > step_time: 0.1625 (0.1369638816965216) | > loader_time: 0.0048 (0.00461797068364517)  --> TIME: 2025-02-26 04:05:02 -- STEP: 7898/36938 -- GLOBAL_STEP: 1079100 | > loss_text_ce: 0.030306179076433182 (0.03219012703669209) | > loss_mel_ce: 2.7357773780822754 (nan) | > loss: 0.010976522229611874 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13699624975532365) | > loader_time: 0.0046 (0.004618049996144937)  --> TIME: 2025-02-26 04:05:11 -- STEP: 7948/36938 -- GLOBAL_STEP: 1079150 | > loss_text_ce: 0.03799896314740181 (0.03218300753401037) | > loss_mel_ce: 3.738570213317871 (nan) | > loss: 0.014986386522650719 (nan) | > current_lr: 5e-06 | > step_time: 0.1855 (0.13699054568030997) | > loader_time: 0.0049 (0.004618050377754105)  --> TIME: 2025-02-26 04:05:21 -- STEP: 7998/36938 -- GLOBAL_STEP: 1079200 | > loss_text_ce: 0.03096678853034973 (0.03219258742803817) | > loss_mel_ce: 3.2557365894317627 (nan) | > loss: 0.013042474165558815 (nan) | > current_lr: 5e-06 | > step_time: 0.178 (0.1370603804529885) | > loader_time: 0.0048 (0.004618566612745423)  --> TIME: 2025-02-26 04:05:30 -- STEP: 8048/36938 -- GLOBAL_STEP: 1079250 | > loss_text_ce: 0.03397301211953163 (0.03219157555980699) | > loss_mel_ce: 3.363802194595337 (nan) | > loss: 0.013483235612511635 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13703994245581258) | > loader_time: 0.0044 (0.004618520202267017)  --> TIME: 2025-02-26 04:05:40 -- STEP: 8098/36938 -- GLOBAL_STEP: 1079300 | > loss_text_ce: 0.030272308737039566 (0.032191996028953763) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13705259507071565) | > loader_time: 0.0045 (0.004619312775108555)  --> TIME: 2025-02-26 04:05:49 -- STEP: 8148/36938 -- GLOBAL_STEP: 1079350 | > loss_text_ce: 0.035131365060806274 (0.03219115117070571) | > loss_mel_ce: 2.608731269836426 (nan) | > loss: 0.01049151923507452 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13703872306591458) | > loader_time: 0.0046 (0.004619274557251304)  --> TIME: 2025-02-26 04:05:58 -- STEP: 8198/36938 -- GLOBAL_STEP: 1079400 | > loss_text_ce: 0.02905321680009365 (0.03218982136195113) | > loss_mel_ce: 2.754772901535034 (nan) | > loss: 0.01104692928493023 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13702891613978124) | > loader_time: 0.0048 (0.004619237125487013)  --> TIME: 2025-02-26 04:06:08 -- STEP: 8248/36938 -- GLOBAL_STEP: 1079450 | > loss_text_ce: 0.03896721452474594 (0.03219340237977208) | > loss_mel_ce: 2.125032663345337 (nan) | > loss: 0.008587301708757877 (nan) | > current_lr: 5e-06 | > step_time: 0.1574 (0.13703070273107923) | > loader_time: 0.0047 (0.004619336006828712)  --> TIME: 2025-02-26 04:06:17 -- STEP: 8298/36938 -- GLOBAL_STEP: 1079500 | > loss_text_ce: 0.035717811435461044 (0.032191950681595416) | > loss_mel_ce: 2.715885639190674 (nan) | > loss: 0.010919061489403248 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13702780179271706) | > loader_time: 0.0046 (0.004619842093948813)  --> TIME: 2025-02-26 04:06:26 -- STEP: 8348/36938 -- GLOBAL_STEP: 1079550 | > loss_text_ce: 0.03101903758943081 (0.03218936837866473) | > loss_mel_ce: 0.5326130986213684 (nan) | > loss: 0.0022366356570273638 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.1370175874113289) | > loader_time: 0.0046 (0.00462000231224211)  --> TIME: 2025-02-26 04:06:36 -- STEP: 8398/36938 -- GLOBAL_STEP: 1079600 | > loss_text_ce: 0.033242132514715195 (0.032195261398906654) | > loss_mel_ce: 2.2513694763183594 (nan) | > loss: 0.009065919555723667 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.1370254768079501) | > loader_time: 0.0049 (0.00462071570250841)  --> TIME: 2025-02-26 04:06:45 -- STEP: 8448/36938 -- GLOBAL_STEP: 1079650 | > loss_text_ce: 0.03214571997523308 (0.032191664346597045) | > loss_mel_ce: 2.808262825012207 (nan) | > loss: 0.01127146277576685 (nan) | > current_lr: 5e-06 | > step_time: 0.1818 (0.13701175427007803) | > loader_time: 0.0048 (0.004620707001198433)  --> TIME: 2025-02-26 04:06:54 -- STEP: 8498/36938 -- GLOBAL_STEP: 1079700 | > loss_text_ce: 0.032541632652282715 (0.03218607488455456) | > loss_mel_ce: 2.325247287750244 (nan) | > loss: 0.009356305934488773 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13701379607778072) | > loader_time: 0.0045 (0.004620815142656007)  --> TIME: 2025-02-26 04:07:03 -- STEP: 8548/36938 -- GLOBAL_STEP: 1079750 | > loss_text_ce: 0.024262189865112305 (0.03218618775286473) | > loss_mel_ce: 2.9077610969543457 (nan) | > loss: 0.011635013855993748 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13700784636146235) | > loader_time: 0.0046 (0.004620991553108895)  --> TIME: 2025-02-26 04:07:13 -- STEP: 8598/36938 -- GLOBAL_STEP: 1079800 | > loss_text_ce: 0.0380309633910656 (0.0321812882438211) | > loss_mel_ce: 2.9961843490600586 (nan) | > loss: 0.012040536850690842 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13701458087103846) | > loader_time: 0.0045 (0.004621150771471823)  --> TIME: 2025-02-26 04:07:22 -- STEP: 8648/36938 -- GLOBAL_STEP: 1079850 | > loss_text_ce: 0.0429028756916523 (0.03218327378964007) | > loss_mel_ce: 2.4972798824310303 (nan) | > loss: 0.01008009072393179 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13701576471438967) | > loader_time: 0.0047 (0.004621863144619626)  --> TIME: 2025-02-26 04:07:32 -- STEP: 8698/36938 -- GLOBAL_STEP: 1079900 | > loss_text_ce: 0.03460186347365379 (0.03217265095228109) | > loss_mel_ce: 3.0186452865600586 (nan) | > loss: 0.01211606152355671 (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.13706046736193747) | > loader_time: 0.0045 (0.004622156331116492)  --> TIME: 2025-02-26 04:07:41 -- STEP: 8748/36938 -- GLOBAL_STEP: 1079950 | > loss_text_ce: 0.02450837567448616 (0.032171501884846805) | > loss_mel_ce: 3.2079033851623535 (nan) | > loss: 0.012827032245695591 (nan) | > current_lr: 5e-06 | > step_time: 0.1693 (0.1370396217255485) | > loader_time: 0.0048 (0.004622490372241344)  --> TIME: 2025-02-26 04:07:51 -- STEP: 8798/36938 -- GLOBAL_STEP: 1080000 | > loss_text_ce: 0.03644691780209541 (0.03216863899652848) | > loss_mel_ce: 2.643095016479492 (nan) | > loss: 0.010633103549480438 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13705091192982954) | > loader_time: 0.0045 (0.004622649756256201) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1080000.pth  --> TIME: 2025-02-26 04:08:06 -- STEP: 8848/36938 -- GLOBAL_STEP: 1080050 | > loss_text_ce: 0.0332784466445446 (0.03216861072343748) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13704890104357578) | > loader_time: 0.0044 (0.0046231052004622785)  --> TIME: 2025-02-26 04:08:15 -- STEP: 8898/36938 -- GLOBAL_STEP: 1080100 | > loss_text_ce: 0.033797185868024826 (0.0321603622985638) | > loss_mel_ce: 1.1554884910583496 (nan) | > loss: 0.004719387739896774 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13705917456681074) | > loader_time: 0.0045 (0.0046227538845570754)  --> TIME: 2025-02-26 04:08:25 -- STEP: 8948/36938 -- GLOBAL_STEP: 1080150 | > loss_text_ce: 0.03200604021549225 (0.03215536307589317) | > loss_mel_ce: 2.2170169353485107 (nan) | > loss: 0.00892469473183155 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13706348948996627) | > loader_time: 0.0046 (0.004622617975430244)  --> TIME: 2025-02-26 04:08:34 -- STEP: 8998/36938 -- GLOBAL_STEP: 1080200 | > loss_text_ce: 0.03239184617996216 (0.03214672584304845) | > loss_mel_ce: 1.3773531913757324 (nan) | > loss: 0.005594226531684399 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13706422011092814) | > loader_time: 0.0045 (0.004622577110590688)  --> TIME: 2025-02-26 04:08:43 -- STEP: 9048/36938 -- GLOBAL_STEP: 1080250 | > loss_text_ce: 0.026254579424858093 (0.03214801994632443) | > loss_mel_ce: 1.8347293138504028 (nan) | > loss: 0.007384857162833214 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13704985701647773) | > loader_time: 0.0045 (0.004622891347667589)  --> TIME: 2025-02-26 04:08:52 -- STEP: 9098/36938 -- GLOBAL_STEP: 1080300 | > loss_text_ce: 0.026982618495821953 (0.03214586709963218) | > loss_mel_ce: 2.5317494869232178 (nan) | > loss: 0.010153698734939098 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13703805843692016) | > loader_time: 0.0046 (0.004623009807907853)  --> TIME: 2025-02-26 04:09:02 -- STEP: 9148/36938 -- GLOBAL_STEP: 1080350 | > loss_text_ce: 0.03211621567606926 (0.032143079801566485) | > loss_mel_ce: 2.715383291244507 (nan) | > loss: 0.010902776382863522 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13703424966017286) | > loader_time: 0.0045 (0.004622828011748365)  --> TIME: 2025-02-26 04:09:11 -- STEP: 9198/36938 -- GLOBAL_STEP: 1080400 | > loss_text_ce: 0.02775651589035988 (0.0321341897127087) | > loss_mel_ce: 2.555835008621216 (nan) | > loss: 0.010252347216010094 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.1370337218869166) | > loader_time: 0.0045 (0.004622729090561844)  --> TIME: 2025-02-26 04:09:20 -- STEP: 9248/36938 -- GLOBAL_STEP: 1080450 | > loss_text_ce: 0.03115743398666382 (0.032133241308728966) | > loss_mel_ce: 3.0400173664093018 (nan) | > loss: 0.012187202461063862 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.13702726426009024) | > loader_time: 0.0046 (0.004622407077093031)  --> TIME: 2025-02-26 04:09:29 -- STEP: 9298/36938 -- GLOBAL_STEP: 1080500 | > loss_text_ce: 0.0332428477704525 (0.03213557609609154) | > loss_mel_ce: 2.274592399597168 (nan) | > loss: 0.009158077649772167 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13699635458802792) | > loader_time: 0.0046 (0.004622056653971935)  --> TIME: 2025-02-26 04:09:38 -- STEP: 9348/36938 -- GLOBAL_STEP: 1080550 | > loss_text_ce: 0.03232121840119362 (0.032133671891854454) | > loss_mel_ce: 1.1826785802841187 (nan) | > loss: 0.004821428097784519 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.13697344386072569) | > loader_time: 0.0047 (0.004621785244066395)  --> TIME: 2025-02-26 04:09:48 -- STEP: 9398/36938 -- GLOBAL_STEP: 1080600 | > loss_text_ce: 0.045384667813777924 (0.032136900268398355) | > loss_mel_ce: 2.889147996902466 (nan) | > loss: 0.011644971556961536 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13696050798672227) | > loader_time: 0.0045 (0.004621724824647653)  --> TIME: 2025-02-26 04:09:57 -- STEP: 9448/36938 -- GLOBAL_STEP: 1080650 | > loss_text_ce: 0.03486059978604317 (0.0321340863908358) | > loss_mel_ce: 2.655036687850952 (nan) | > loss: 0.010674196295440197 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13694761217801235) | > loader_time: 0.0046 (0.004621474319953829)  --> TIME: 2025-02-26 04:10:06 -- STEP: 9498/36938 -- GLOBAL_STEP: 1080700 | > loss_text_ce: 0.030047325417399406 (0.032127391788571186) | > loss_mel_ce: 2.86529541015625 (nan) | > loss: 0.01148945651948452 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13693981154086945) | > loader_time: 0.0045 (0.004621204111947093)  --> TIME: 2025-02-26 04:10:15 -- STEP: 9548/36938 -- GLOBAL_STEP: 1080750 | > loss_text_ce: 0.03996496647596359 (0.03212443085972208) | > loss_mel_ce: 1.294235348701477 (nan) | > loss: 0.005294445902109146 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.1369338522508742) | > loader_time: 0.0049 (0.0046209802825330085)  --> TIME: 2025-02-26 04:10:25 -- STEP: 9598/36938 -- GLOBAL_STEP: 1080800 | > loss_text_ce: 0.03647036850452423 (0.032120120783720146) | > loss_mel_ce: 2.067840576171875 (nan) | > loss: 0.008350441232323647 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13693028058725482) | > loader_time: 0.0044 (0.004620710967306949)  --> TIME: 2025-02-26 04:10:34 -- STEP: 9648/36938 -- GLOBAL_STEP: 1080850 | > loss_text_ce: 0.0201729703694582 (0.03211539129737789) | > loss_mel_ce: 2.3504903316497803 (nan) | > loss: 0.00940739456564188 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13692657871922395) | > loader_time: 0.0045 (0.004620689756636999)  --> TIME: 2025-02-26 04:10:43 -- STEP: 9698/36938 -- GLOBAL_STEP: 1080900 | > loss_text_ce: 0.0328819639980793 (0.0321101071181459) | > loss_mel_ce: 3.1934046745300293 (nan) | > loss: 0.012802725657820702 (nan) | > current_lr: 5e-06 | > step_time: 0.1858 (0.13691122373567902) | > loader_time: 0.0048 (0.0046204934540078225)  --> TIME: 2025-02-26 04:10:52 -- STEP: 9748/36938 -- GLOBAL_STEP: 1080950 | > loss_text_ce: 0.01571328565478325 (0.03211085844521666) | > loss_mel_ce: 1.8302615880966187 (nan) | > loss: 0.007325297687202692 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13688604800878162) | > loader_time: 0.0044 (0.004620188956481796)  --> TIME: 2025-02-26 04:11:01 -- STEP: 9798/36938 -- GLOBAL_STEP: 1081000 | > loss_text_ce: 0.02685859426856041 (0.03210228245663922) | > loss_mel_ce: 2.9823899269104004 (nan) | > loss: 0.011941462755203247 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.1368946775316481) | > loader_time: 0.0044 (0.004619894526057853)  --> TIME: 2025-02-26 04:11:11 -- STEP: 9848/36938 -- GLOBAL_STEP: 1081050 | > loss_text_ce: 0.03527471050620079 (0.032108525385922015) | > loss_mel_ce: 2.454460382461548 (nan) | > loss: 0.009879902005195618 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13692427277080765) | > loader_time: 0.0049 (0.004619805746016531)  --> TIME: 2025-02-26 04:11:20 -- STEP: 9898/36938 -- GLOBAL_STEP: 1081100 | > loss_text_ce: 0.03351200371980667 (0.032113945462689586) | > loss_mel_ce: 2.240636110305786 (nan) | > loss: 0.009024398401379585 (nan) | > current_lr: 5e-06 | > step_time: 0.1532 (0.1368979255366404) | > loader_time: 0.0046 (0.00461968724764727)  --> TIME: 2025-02-26 04:11:30 -- STEP: 9948/36938 -- GLOBAL_STEP: 1081150 | > loss_text_ce: 0.036161698400974274 (0.0321146090433322) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.1369141733785395) | > loader_time: 0.0045 (0.004619736076120313)  --> TIME: 2025-02-26 04:11:39 -- STEP: 9998/36938 -- GLOBAL_STEP: 1081200 | > loss_text_ce: 0.022054314613342285 (0.032106441354733416) | > loss_mel_ce: 2.6815176010131836 (nan) | > loss: 0.010728459805250168 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.136923430967054) | > loader_time: 0.0047 (0.004619783343112728)  --> TIME: 2025-02-26 04:11:49 -- STEP: 10048/36938 -- GLOBAL_STEP: 1081250 | > loss_text_ce: 0.04346452280879021 (0.032108859579667595) | > loss_mel_ce: 2.7181432247161865 (nan) | > loss: 0.010958760976791382 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13693935530865234) | > loader_time: 0.0046 (0.0046197460240619305)  --> TIME: 2025-02-26 04:11:58 -- STEP: 10098/36938 -- GLOBAL_STEP: 1081300 | > loss_text_ce: 0.03282281383872032 (0.03211289581673571) | > loss_mel_ce: 0.08624267578125 (nan) | > loss: 0.00047248214832507074 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13690585104586628) | > loader_time: 0.0047 (0.004619779693511349)  --> TIME: 2025-02-26 04:12:07 -- STEP: 10148/36938 -- GLOBAL_STEP: 1081350 | > loss_text_ce: 0.03402535989880562 (0.032113503596819186) | > loss_mel_ce: 3.3407490253448486 (nan) | > loss: 0.013391963206231594 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.1368980985670798) | > loader_time: 0.0048 (0.004620023491723743)  --> TIME: 2025-02-26 04:12:16 -- STEP: 10198/36938 -- GLOBAL_STEP: 1081400 | > loss_text_ce: 0.03157523646950722 (0.032118823687405464) | > loss_mel_ce: 3.2509846687316895 (nan) | > loss: 0.013026031665503979 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.1368995665755776) | > loader_time: 0.0048 (0.004620305578669849)  --> TIME: 2025-02-26 04:12:25 -- STEP: 10248/36938 -- GLOBAL_STEP: 1081450 | > loss_text_ce: 0.027472957968711853 (0.03211666090769442) | > loss_mel_ce: 2.749101400375366 (nan) | > loss: 0.01101815328001976 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.13688563051212554) | > loader_time: 0.0046 (0.004620291054202288)  --> TIME: 2025-02-26 04:12:35 -- STEP: 10298/36938 -- GLOBAL_STEP: 1081500 | > loss_text_ce: 0.0346842035651207 (0.032123045010970006) | > loss_mel_ce: 1.2928166389465332 (nan) | > loss: 0.00526786083355546 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1368872184479296) | > loader_time: 0.0044 (0.00462033489788128)  --> TIME: 2025-02-26 04:12:44 -- STEP: 10348/36938 -- GLOBAL_STEP: 1081550 | > loss_text_ce: 0.02619888074696064 (0.03212692659883921) | > loss_mel_ce: 2.9260194301605225 (nan) | > loss: 0.01171515230089426 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.1368980151278932) | > loader_time: 0.0045 (0.004620251159756449)  --> TIME: 2025-02-26 04:12:54 -- STEP: 10398/36938 -- GLOBAL_STEP: 1081600 | > loss_text_ce: 0.0435384064912796 (0.03212967358983128) | > loss_mel_ce: 2.785419225692749 (nan) | > loss: 0.011226022616028786 (nan) | > current_lr: 5e-06 | > step_time: 0.1773 (0.13690063226083085) | > loader_time: 0.0049 (0.004620106937014444)  --> TIME: 2025-02-26 04:13:03 -- STEP: 10448/36938 -- GLOBAL_STEP: 1081650 | > loss_text_ce: 0.03733168914914131 (0.03213505810454683) | > loss_mel_ce: 2.6388938426971436 (nan) | > loss: 0.010619943030178547 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13690936843797558) | > loader_time: 0.0048 (0.004620100715258969)  --> TIME: 2025-02-26 04:13:12 -- STEP: 10498/36938 -- GLOBAL_STEP: 1081700 | > loss_text_ce: 0.023340599611401558 (0.03213339380846822) | > loss_mel_ce: 2.2344000339508057 (nan) | > loss: 0.008959288708865643 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13690356484684008) | > loader_time: 0.0045 (0.004620217327663727)  --> TIME: 2025-02-26 04:13:21 -- STEP: 10548/36938 -- GLOBAL_STEP: 1081750 | > loss_text_ce: 0.030562255531549454 (0.03213484643400126) | > loss_mel_ce: 2.5358104705810547 (nan) | > loss: 0.01018401887267828 (nan) | > current_lr: 5e-06 | > step_time: 0.1876 (0.1368686146567974) | > loader_time: 0.0048 (0.004619944127256073)  --> TIME: 2025-02-26 04:13:31 -- STEP: 10598/36938 -- GLOBAL_STEP: 1081800 | > loss_text_ce: 0.03154142200946808 (0.03213602342210651) | > loss_mel_ce: 1.9780120849609375 (nan) | > loss: 0.007974418811500072 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1368701425357274) | > loader_time: 0.0045 (0.004619842273915904)  --> TIME: 2025-02-26 04:13:40 -- STEP: 10648/36938 -- GLOBAL_STEP: 1081850 | > loss_text_ce: 0.023780452087521553 (0.03213862516475755) | > loss_mel_ce: 1.4591162204742432 (nan) | > loss: 0.0058845109306275845 (nan) | > current_lr: 5e-06 | > step_time: 0.1084 (0.13687409831114375) | > loader_time: 0.0045 (0.004619991147903298)  --> TIME: 2025-02-26 04:13:50 -- STEP: 10698/36938 -- GLOBAL_STEP: 1081900 | > loss_text_ce: 0.03180784732103348 (0.03214355629298926) | > loss_mel_ce: 3.115494728088379 (nan) | > loss: 0.01248929649591446 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13688838553219118) | > loader_time: 0.0045 (0.0046199821360558445)  --> TIME: 2025-02-26 04:13:59 -- STEP: 10748/36938 -- GLOBAL_STEP: 1081950 | > loss_text_ce: 0.035090234130620956 (0.03214594372296864) | > loss_mel_ce: 3.183612108230591 (nan) | > loss: 0.012772629037499428 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.13689148727739214) | > loader_time: 0.0046 (0.004619847055611949)  --> TIME: 2025-02-26 04:14:09 -- STEP: 10798/36938 -- GLOBAL_STEP: 1082000 | > loss_text_ce: 0.036958903074264526 (0.03214456827593131) | > loss_mel_ce: 3.0391483306884766 (nan) | > loss: 0.01220677513629198 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.13694982973375305) | > loader_time: 0.0048 (0.0046201429669118395)  --> TIME: 2025-02-26 04:14:18 -- STEP: 10848/36938 -- GLOBAL_STEP: 1082050 | > loss_text_ce: 0.02868819795548916 (0.03214906517603036) | > loss_mel_ce: 2.618105411529541 (nan) | > loss: 0.010503149591386318 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.1369507127059589) | > loader_time: 0.0045 (0.004620324127969504)  --> TIME: 2025-02-26 04:14:28 -- STEP: 10898/36938 -- GLOBAL_STEP: 1082100 | > loss_text_ce: 0.03686492145061493 (0.03214339754831494) | > loss_mel_ce: 3.0920002460479736 (nan) | > loss: 0.012416132725775242 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.13695553644662925) | > loader_time: 0.0048 (0.004620087586492346)  --> TIME: 2025-02-26 04:14:37 -- STEP: 10948/36938 -- GLOBAL_STEP: 1082150 | > loss_text_ce: 0.020628515630960464 (0.032148813986881804) | > loss_mel_ce: 2.3918421268463135 (nan) | > loss: 0.009573296643793583 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13694000997637326) | > loader_time: 0.0046 (0.004620143454310942)  --> TIME: 2025-02-26 04:14:46 -- STEP: 10998/36938 -- GLOBAL_STEP: 1082200 | > loss_text_ce: 0.028354043141007423 (0.03214713706865085) | > loss_mel_ce: 2.973003625869751 (nan) | > loss: 0.011910149827599525 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13691807495591604) | > loader_time: 0.0045 (0.004620232350567442)  --> TIME: 2025-02-26 04:14:55 -- STEP: 11048/36938 -- GLOBAL_STEP: 1082250 | > loss_text_ce: 0.03385060653090477 (0.032146911374525786) | > loss_mel_ce: 2.4696199893951416 (nan) | > loss: 0.009934407658874989 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13691963502499171) | > loader_time: 0.0046 (0.004620068643854475)  --> TIME: 2025-02-26 04:15:05 -- STEP: 11098/36938 -- GLOBAL_STEP: 1082300 | > loss_text_ce: 0.03340150788426399 (0.032151246952101975) | > loss_mel_ce: 3.1571755409240723 (nan) | > loss: 0.012661020271480083 (nan) | > current_lr: 5e-06 | > step_time: 0.1523 (0.13691496739496548) | > loader_time: 0.0046 (0.004619842156523496)  --> TIME: 2025-02-26 04:15:14 -- STEP: 11148/36938 -- GLOBAL_STEP: 1082350 | > loss_text_ce: 0.009897972457110882 (0.03214719560046431) | > loss_mel_ce: 0.544049084186554 (nan) | > loss: 0.002198202535510063 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1369048415823217) | > loader_time: 0.0046 (0.0046195327529866305)  --> TIME: 2025-02-26 04:15:23 -- STEP: 11198/36938 -- GLOBAL_STEP: 1082400 | > loss_text_ce: 0.027072949334979057 (0.032147075931166724) | > loss_mel_ce: 0.6850296258926392 (nan) | > loss: 0.002825804054737091 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1368963593222188) | > loader_time: 0.0045 (0.004619236353529982)  --> TIME: 2025-02-26 04:15:32 -- STEP: 11248/36938 -- GLOBAL_STEP: 1082450 | > loss_text_ce: 0.028811829164624214 (0.032144011210108195) | > loss_mel_ce: 3.1605312824249268 (nan) | > loss: 0.01265612430870533 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.1368775926689327) | > loader_time: 0.0047 (0.004618914334228004)  --> TIME: 2025-02-26 04:15:41 -- STEP: 11298/36938 -- GLOBAL_STEP: 1082500 | > loss_text_ce: 0.039579953998327255 (0.032148276353483884) | > loss_mel_ce: 0.002197265625 (nan) | > loss: 0.00016578262147959322 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13689233579768914) | > loader_time: 0.0045 (0.0046187074949602565)  --> TIME: 2025-02-26 04:15:51 -- STEP: 11348/36938 -- GLOBAL_STEP: 1082550 | > loss_text_ce: 0.026587530970573425 (0.03214494262609886) | > loss_mel_ce: 3.055694103240967 (nan) | > loss: 0.01223127730190754 (nan) | > current_lr: 5e-06 | > step_time: 0.188 (0.13687916345313872) | > loader_time: 0.0049 (0.004618363688014794)  --> TIME: 2025-02-26 04:16:00 -- STEP: 11398/36938 -- GLOBAL_STEP: 1082600 | > loss_text_ce: 0.025505270808935165 (0.03214550328037791) | > loss_mel_ce: 2.618053674697876 (nan) | > loss: 0.010490314103662968 (nan) | > current_lr: 5e-06 | > step_time: 0.1912 (0.13688390918898294) | > loader_time: 0.0048 (0.004618104768857816)  --> TIME: 2025-02-26 04:16:09 -- STEP: 11448/36938 -- GLOBAL_STEP: 1082650 | > loss_text_ce: 0.040415938943624496 (0.03214812784531842) | > loss_mel_ce: 1.9896528720855713 (nan) | > loss: 0.00805582944303751 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.13688841247625164) | > loader_time: 0.0045 (0.004618258783652355)  --> TIME: 2025-02-26 04:16:19 -- STEP: 11498/36938 -- GLOBAL_STEP: 1082700 | > loss_text_ce: 0.029633302241563797 (0.0321475564915109) | > loss_mel_ce: 3.742431640625 (nan) | > loss: 0.014968512579798698 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13688146657457614) | > loader_time: 0.0045 (0.004618341517585288)  --> TIME: 2025-02-26 04:16:28 -- STEP: 11548/36938 -- GLOBAL_STEP: 1082750 | > loss_text_ce: 0.037566177546978 (0.032155713742449674) | > loss_mel_ce: 2.769315481185913 (nan) | > loss: 0.01113841962069273 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13688487687819406) | > loader_time: 0.0047 (0.004618398264530504)  --> TIME: 2025-02-26 04:16:37 -- STEP: 11598/36938 -- GLOBAL_STEP: 1082800 | > loss_text_ce: 0.0311029851436615 (0.032159410464135674) | > loss_mel_ce: 1.616118311882019 (nan) | > loss: 0.0065365927293896675 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13687555250124045) | > loader_time: 0.0047 (0.004618812211732824)  --> TIME: 2025-02-26 04:16:46 -- STEP: 11648/36938 -- GLOBAL_STEP: 1082850 | > loss_text_ce: 0.023530149832367897 (0.032158084191505835) | > loss_mel_ce: 2.0343992710113525 (nan) | > loss: 0.008166387677192688 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13685366740593488) | > loader_time: 0.0048 (0.00461876558160389)  --> TIME: 2025-02-26 04:16:56 -- STEP: 11698/36938 -- GLOBAL_STEP: 1082900 | > loss_text_ce: 0.03651604428887367 (0.03215955027668392) | > loss_mel_ce: 2.3995866775512695 (nan) | > loss: 0.00966707430779934 (nan) | > current_lr: 5e-06 | > step_time: 0.1517 (0.13685225920710556) | > loader_time: 0.0046 (0.004618698051799605)  --> TIME: 2025-02-26 04:17:05 -- STEP: 11748/36938 -- GLOBAL_STEP: 1082950 | > loss_text_ce: 0.02015424706041813 (0.032154368267181654) | > loss_mel_ce: 2.9718825817108154 (nan) | > loss: 0.011873162351548672 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.1368307628759846) | > loader_time: 0.0069 (0.004618736242087146)  --> TIME: 2025-02-26 04:17:14 -- STEP: 11798/36938 -- GLOBAL_STEP: 1083000 | > loss_text_ce: 0.02733827568590641 (0.03215768499407853) | > loss_mel_ce: 2.7315828800201416 (nan) | > loss: 0.010948100127279758 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.136826312568232) | > loader_time: 0.0047 (0.004619181772514328)  --> TIME: 2025-02-26 04:17:23 -- STEP: 11848/36938 -- GLOBAL_STEP: 1083050 | > loss_text_ce: 0.029227009043097496 (0.032152526746635844) | > loss_mel_ce: 2.359306812286377 (nan) | > loss: 0.009478309191763401 (nan) | > current_lr: 5e-06 | > step_time: 0.1519 (0.13682999059930143) | > loader_time: 0.0046 (0.004619201359758498)  --> TIME: 2025-02-26 04:17:33 -- STEP: 11898/36938 -- GLOBAL_STEP: 1083100 | > loss_text_ce: 0.029077580198645592 (0.03214500552393346) | > loss_mel_ce: 1.1574993133544922 (nan) | > loss: 0.0047086384147405624 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13683420166205384) | > loader_time: 0.0046 (0.004619241181612614)  --> TIME: 2025-02-26 04:17:42 -- STEP: 11948/36938 -- GLOBAL_STEP: 1083150 | > loss_text_ce: 0.02790159359574318 (0.032141284222920494) | > loss_mel_ce: 3.3960461616516113 (nan) | > loss: 0.013587094843387604 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.13684482944821805) | > loader_time: 0.0047 (0.004619404109856493)  --> TIME: 2025-02-26 04:17:52 -- STEP: 11998/36938 -- GLOBAL_STEP: 1083200 | > loss_text_ce: 0.030648671090602875 (0.03213701650006905) | > loss_mel_ce: 3.0479753017425537 (nan) | > loss: 0.012216762639582157 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.13685902092213953) | > loader_time: 0.0048 (0.004619642364360778)  --> TIME: 2025-02-26 04:18:01 -- STEP: 12048/36938 -- GLOBAL_STEP: 1083250 | > loss_text_ce: 0.03342411294579506 (0.0321375973884275) | > loss_mel_ce: 2.236095428466797 (nan) | > loss: 0.009006030857563019 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.13687111556055026) | > loader_time: 0.0048 (0.004619791470517512)  --> TIME: 2025-02-26 04:18:10 -- STEP: 12098/36938 -- GLOBAL_STEP: 1083300 | > loss_text_ce: 0.03624137490987778 (0.03213957061493463) | > loss_mel_ce: 1.9666577577590942 (nan) | > loss: 0.007948013022542 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13685045921816294) | > loader_time: 0.0063 (0.0046198283528311995)  --> TIME: 2025-02-26 04:18:20 -- STEP: 12148/36938 -- GLOBAL_STEP: 1083350 | > loss_text_ce: 0.03142218664288521 (0.032140554323657915) | > loss_mel_ce: 1.8411511182785034 (nan) | > loss: 0.007430846802890301 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.13688531970365422) | > loader_time: 0.0046 (0.004619937842717324)  --> TIME: 2025-02-26 04:18:29 -- STEP: 12198/36938 -- GLOBAL_STEP: 1083400 | > loss_text_ce: 0.02466646209359169 (0.03213753788664016) | > loss_mel_ce: 2.87363862991333 (nan) | > loss: 0.011501211673021317 (nan) | > current_lr: 5e-06 | > step_time: 0.1746 (0.13688587038062328) | > loader_time: 0.0048 (0.004619940614520496)  --> TIME: 2025-02-26 04:18:39 -- STEP: 12248/36938 -- GLOBAL_STEP: 1083450 | > loss_text_ce: 0.027785396203398705 (0.032134503227731905) | > loss_mel_ce: 1.5824135541915894 (nan) | > loss: 0.006389678921550512 (nan) | > current_lr: 5e-06 | > step_time: 0.1617 (0.1368821153720165) | > loader_time: 0.0049 (0.004619935071211398)  --> TIME: 2025-02-26 04:18:48 -- STEP: 12298/36938 -- GLOBAL_STEP: 1083500 | > loss_text_ce: 0.03310861065983772 (0.03213509190860901) | > loss_mel_ce: 0.18581542372703552 (nan) | > loss: 0.0008687461959198117 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13688191578124131) | > loader_time: 0.0045 (0.004619979823502236)  --> TIME: 2025-02-26 04:18:57 -- STEP: 12348/36938 -- GLOBAL_STEP: 1083550 | > loss_text_ce: 0.03386178985238075 (0.032140369328980956) | > loss_mel_ce: 3.485858917236328 (nan) | > loss: 0.013967147096991539 (nan) | > current_lr: 5e-06 | > step_time: 0.1651 (0.13688929873760877) | > loader_time: 0.0047 (0.00462004381126673)  --> TIME: 2025-02-26 04:19:07 -- STEP: 12398/36938 -- GLOBAL_STEP: 1083600 | > loss_text_ce: 0.03327873721718788 (0.032142336499850085) | > loss_mel_ce: 2.7666945457458496 (nan) | > loss: 0.011111005209386349 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13688409022697204) | > loader_time: 0.0046 (0.004620245645691992)  --> TIME: 2025-02-26 04:19:16 -- STEP: 12448/36938 -- GLOBAL_STEP: 1083650 | > loss_text_ce: 0.029412856325507164 (0.032141483837593486) | > loss_mel_ce: 2.1606478691101074 (nan) | > loss: 0.00869071763008833 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13690174453362688) | > loader_time: 0.0046 (0.004620480966445416)  --> TIME: 2025-02-26 04:19:26 -- STEP: 12498/36938 -- GLOBAL_STEP: 1083700 | > loss_text_ce: 0.03884539380669594 (0.0321427257701747) | > loss_mel_ce: 3.015774965286255 (nan) | > loss: 0.01212150976061821 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.136915007330433) | > loader_time: 0.0046 (0.004620849542530809)  --> TIME: 2025-02-26 04:19:35 -- STEP: 12548/36938 -- GLOBAL_STEP: 1083750 | > loss_text_ce: 0.020254869014024734 (0.03214559782219793) | > loss_mel_ce: 0.5107964277267456 (nan) | > loss: 0.0021073464304208755 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13692885595482163) | > loader_time: 0.0046 (0.004621243986081731)  --> TIME: 2025-02-26 04:19:45 -- STEP: 12598/36938 -- GLOBAL_STEP: 1083800 | > loss_text_ce: 0.02938539907336235 (0.03214844193970121) | > loss_mel_ce: 1.4611107110977173 (nan) | > loss: 0.005914667621254921 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13692048289015432) | > loader_time: 0.005 (0.00462142057429422)  --> TIME: 2025-02-26 04:19:54 -- STEP: 12648/36938 -- GLOBAL_STEP: 1083850 | > loss_text_ce: 0.031510576605796814 (0.03215256427838226) | > loss_mel_ce: 2.9471628665924072 (nan) | > loss: 0.011820133775472641 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13692311287529477) | > loader_time: 0.0046 (0.00462177755861325)  --> TIME: 2025-02-26 04:20:04 -- STEP: 12698/36938 -- GLOBAL_STEP: 1083900 | > loss_text_ce: 0.04337048903107643 (0.032155065874133575) | > loss_mel_ce: 1.6578407287597656 (nan) | > loss: 0.006750838365405798 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13694178158701034) | > loader_time: 0.0047 (0.004622079590433771)  --> TIME: 2025-02-26 04:20:13 -- STEP: 12748/36938 -- GLOBAL_STEP: 1083950 | > loss_text_ce: 0.03400397300720215 (0.03214862923496372) | > loss_mel_ce: 2.4454052448272705 (nan) | > loss: 0.009838925674557686 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13694160520226212) | > loader_time: 0.0046 (0.004622315215884909)  --> TIME: 2025-02-26 04:20:23 -- STEP: 12798/36938 -- GLOBAL_STEP: 1084000 | > loss_text_ce: 0.025718877092003822 (0.03214970429109588) | > loss_mel_ce: 2.6326215267181396 (nan) | > loss: 0.0105489706620574 (nan) | > current_lr: 5e-06 | > step_time: 0.1735 (0.13695792554598102) | > loader_time: 0.0046 (0.004622380199572465)  --> TIME: 2025-02-26 04:20:32 -- STEP: 12848/36938 -- GLOBAL_STEP: 1084050 | > loss_text_ce: 0.02952929399907589 (0.03215050323420329) | > loss_mel_ce: 2.404531240463257 (nan) | > loss: 0.00965897087007761 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.1369502655001979) | > loader_time: 0.0046 (0.004622269853262941)  --> TIME: 2025-02-26 04:20:41 -- STEP: 12898/36938 -- GLOBAL_STEP: 1084100 | > loss_text_ce: 0.0271097794175148 (0.032148416970888896) | > loss_mel_ce: 2.3555688858032227 (nan) | > loss: 0.009455074556171894 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.1369397039468165) | > loader_time: 0.0051 (0.004622472609858457)  --> TIME: 2025-02-26 04:20:50 -- STEP: 12948/36938 -- GLOBAL_STEP: 1084150 | > loss_text_ce: 0.038679927587509155 (0.032153199869329956) | > loss_mel_ce: 3.4796242713928223 (nan) | > loss: 0.013961524702608585 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.1369389632730008) | > loader_time: 0.0046 (0.00462248550361419)  --> TIME: 2025-02-26 04:21:00 -- STEP: 12998/36938 -- GLOBAL_STEP: 1084200 | > loss_text_ce: 0.04976564645767212 (0.03215562519483354) | > loss_mel_ce: 2.5151636600494385 (nan) | > loss: 0.010178291238844395 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13693175187091072) | > loader_time: 0.0045 (0.004622603585342573)  --> TIME: 2025-02-26 04:21:09 -- STEP: 13048/36938 -- GLOBAL_STEP: 1084250 | > loss_text_ce: 0.025719966739416122 (0.03215602845476551) | > loss_mel_ce: 2.8538923263549805 (nan) | > loss: 0.011427033692598343 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13693095781114606) | > loader_time: 0.0047 (0.004622722625878602)  --> TIME: 2025-02-26 04:21:18 -- STEP: 13098/36938 -- GLOBAL_STEP: 1084300 | > loss_text_ce: 0.029685502871870995 (0.03214800842549998) | > loss_mel_ce: 2.7008471488952637 (nan) | > loss: 0.01083544734865427 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13693982134303662) | > loader_time: 0.0045 (0.004622757189108914)  --> TIME: 2025-02-26 04:21:27 -- STEP: 13148/36938 -- GLOBAL_STEP: 1084350 | > loss_text_ce: 0.03401691094040871 (0.03214553008700318) | > loss_mel_ce: 2.9096333980560303 (nan) | > loss: 0.011681152507662773 (nan) | > current_lr: 5e-06 | > step_time: 0.1688 (0.13692073726639523) | > loader_time: 0.005 (0.0046227492747932)  --> TIME: 2025-02-26 04:21:37 -- STEP: 13198/36938 -- GLOBAL_STEP: 1084400 | > loss_text_ce: 0.03616425022482872 (0.03214263030659345) | > loss_mel_ce: 3.2692878246307373 (nan) | > loss: 0.013116873800754547 (nan) | > current_lr: 5e-06 | > step_time: 0.1978 (0.13693186548157563) | > loader_time: 0.0047 (0.004622716996900629)  --> TIME: 2025-02-26 04:21:46 -- STEP: 13248/36938 -- GLOBAL_STEP: 1084450 | > loss_text_ce: 0.03052496910095215 (0.03213892375904385) | > loss_mel_ce: 3.1103765964508057 (nan) | > loss: 0.012463895604014397 (nan) | > current_lr: 5e-06 | > step_time: 0.1736 (0.1369320019575704) | > loader_time: 0.0047 (0.004622569694611178)  --> TIME: 2025-02-26 04:21:55 -- STEP: 13298/36938 -- GLOBAL_STEP: 1084500 | > loss_text_ce: 0.035086989402770996 (0.03213605179841752) | > loss_mel_ce: 1.7305718660354614 (nan) | > loss: 0.007006583269685507 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.1369196939045257) | > loader_time: 0.0045 (0.0046223773510227625)  --> TIME: 2025-02-26 04:22:04 -- STEP: 13348/36938 -- GLOBAL_STEP: 1084550 | > loss_text_ce: 0.035498589277267456 (0.032140072546985574) | > loss_mel_ce: 1.1643747091293335 (nan) | > loss: 0.004761402495205402 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.1369074167767999) | > loader_time: 0.0045 (0.004622332807568898)  --> TIME: 2025-02-26 04:22:14 -- STEP: 13398/36938 -- GLOBAL_STEP: 1084600 | > loss_text_ce: 0.03138682618737221 (0.032135973835122116) | > loss_mel_ce: 2.5605204105377197 (nan) | > loss: 0.010285346768796444 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.1369098578661047) | > loader_time: 0.0071 (0.0046225505402629425)  --> TIME: 2025-02-26 04:22:24 -- STEP: 13448/36938 -- GLOBAL_STEP: 1084650 | > loss_text_ce: 0.03223506733775139 (0.032133949236793906) | > loss_mel_ce: 3.3822226524353027 (nan) | > loss: 0.013549436815083027 (nan) | > current_lr: 5e-06 | > step_time: 0.1901 (0.1369265236363811) | > loader_time: 0.0047 (0.004622739262697167)  --> TIME: 2025-02-26 04:22:33 -- STEP: 13498/36938 -- GLOBAL_STEP: 1084700 | > loss_text_ce: 0.03567764163017273 (0.032134191941658403) | > loss_mel_ce: 3.4115889072418213 (nan) | > loss: 0.013679630123078823 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.1369237277857156) | > loader_time: 0.0046 (0.004622817816672842)  --> TIME: 2025-02-26 04:22:42 -- STEP: 13548/36938 -- GLOBAL_STEP: 1084750 | > loss_text_ce: 0.03405746445059776 (0.032132707019620836) | > loss_mel_ce: 2.3678646087646484 (nan) | > loss: 0.009531437419354916 (nan) | > current_lr: 5e-06 | > step_time: 0.1831 (0.1369102476161944) | > loader_time: 0.0048 (0.004622585888911317)  --> TIME: 2025-02-26 04:22:51 -- STEP: 13598/36938 -- GLOBAL_STEP: 1084800 | > loss_text_ce: 0.044466860592365265 (0.032130735401904406) | > loss_mel_ce: 2.5721936225891113 (nan) | > loss: 0.010383574292063713 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.136915624132926) | > loader_time: 0.0048 (0.004622587282388662)  --> TIME: 2025-02-26 04:23:00 -- STEP: 13648/36938 -- GLOBAL_STEP: 1084850 | > loss_text_ce: 0.027612347155809402 (0.03213324788298625) | > loss_mel_ce: 2.5869140625 (nan) | > loss: 0.010375105775892735 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13689854454602846) | > loader_time: 0.0045 (0.004622395789078225)  --> TIME: 2025-02-26 04:23:10 -- STEP: 13698/36938 -- GLOBAL_STEP: 1084900 | > loss_text_ce: 0.03592083230614662 (0.03213272598576266) | > loss_mel_ce: 3.1206202507019043 (nan) | > loss: 0.012525957077741623 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.1368810374086865) | > loader_time: 0.0046 (0.004622205850381471)  --> TIME: 2025-02-26 04:23:19 -- STEP: 13748/36938 -- GLOBAL_STEP: 1084950 | > loss_text_ce: 0.03387153148651123 (0.03212594273650507) | > loss_mel_ce: 2.5472240447998047 (nan) | > loss: 0.010242443531751633 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.1368643199249443) | > loader_time: 0.0046 (0.0046221200622789445)  --> TIME: 2025-02-26 04:23:28 -- STEP: 13798/36938 -- GLOBAL_STEP: 1085000 | > loss_text_ce: 0.027317136526107788 (0.032124038411965296) | > loss_mel_ce: 1.4540561437606812 (nan) | > loss: 0.005878465715795755 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13687707011397948) | > loader_time: 0.0046 (0.0046222615300753715)  --> TIME: 2025-02-26 04:23:37 -- STEP: 13848/36938 -- GLOBAL_STEP: 1085050 | > loss_text_ce: 0.03276931121945381 (0.03212820428819272) | > loss_mel_ce: 3.083380937576294 (nan) | > loss: 0.012365675531327724 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.1368633563431748) | > loader_time: 0.0045 (0.004622168085741763)  --> TIME: 2025-02-26 04:23:46 -- STEP: 13898/36938 -- GLOBAL_STEP: 1085100 | > loss_text_ce: 0.042207907885313034 (0.032130051553605564) | > loss_mel_ce: 2.846181869506836 (nan) | > loss: 0.011461865156888962 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13683733986099564) | > loader_time: 0.0044 (0.004622061727098088)  --> TIME: 2025-02-26 04:23:55 -- STEP: 13948/36938 -- GLOBAL_STEP: 1085150 | > loss_text_ce: 0.03828483074903488 (0.03212907888880062) | > loss_mel_ce: 2.8488361835479736 (nan) | > loss: 0.011456829495728016 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13683603354843182) | > loader_time: 0.0045 (0.004622068725139595)  --> TIME: 2025-02-26 04:24:05 -- STEP: 13998/36938 -- GLOBAL_STEP: 1085200 | > loss_text_ce: 0.034857127815485 (0.03213270182526608) | > loss_mel_ce: 2.7589614391326904 (nan) | > loss: 0.011086582206189632 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.136821755785178) | > loader_time: 0.0046 (0.0046219535854343415)  --> TIME: 2025-02-26 04:24:14 -- STEP: 14048/36938 -- GLOBAL_STEP: 1085250 | > loss_text_ce: 0.03537960723042488 (0.03213339018989963) | > loss_mel_ce: 2.7419137954711914 (nan) | > loss: 0.011021005921065807 (nan) | > current_lr: 5e-06 | > step_time: 0.1702 (0.1368374598969898) | > loader_time: 0.0049 (0.004621946984787465)  --> TIME: 2025-02-26 04:24:23 -- STEP: 14098/36938 -- GLOBAL_STEP: 1085300 | > loss_text_ce: 0.02636386826634407 (0.03212972440385427) | > loss_mel_ce: 2.5043256282806396 (nan) | > loss: 0.010042418725788593 (nan) | > current_lr: 5e-06 | > step_time: 0.1556 (0.1368348773345315) | > loader_time: 0.0046 (0.004622003088134081)  --> TIME: 2025-02-26 04:24:33 -- STEP: 14148/36938 -- GLOBAL_STEP: 1085350 | > loss_text_ce: 0.04573189467191696 (0.032128854759459965) | > loss_mel_ce: 1.3116399049758911 (nan) | > loss: 0.005386396311223507 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.1368430758857239) | > loader_time: 0.0046 (0.004621927351273485)  --> TIME: 2025-02-26 04:24:42 -- STEP: 14198/36938 -- GLOBAL_STEP: 1085400 | > loss_text_ce: 0.027919108048081398 (0.032129550209497544) | > loss_mel_ce: 2.5472350120544434 (nan) | > loss: 0.010218866169452667 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13683867258057641) | > loader_time: 0.0044 (0.004621847513142162)  --> TIME: 2025-02-26 04:24:51 -- STEP: 14248/36938 -- GLOBAL_STEP: 1085450 | > loss_text_ce: 0.02090885303914547 (0.0321283787173873) | > loss_mel_ce: 2.785937547683716 (nan) | > loss: 0.011138279922306538 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13682765988470907) | > loader_time: 0.0045 (0.004621965924653359)  --> TIME: 2025-02-26 04:25:01 -- STEP: 14298/36938 -- GLOBAL_STEP: 1085500 | > loss_text_ce: 0.036983832716941833 (0.032130318058083335) | > loss_mel_ce: 3.0813417434692383 (nan) | > loss: 0.012374307960271835 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13684409310691295) | > loader_time: 0.0049 (0.004622130381308964)  --> TIME: 2025-02-26 04:25:11 -- STEP: 14348/36938 -- GLOBAL_STEP: 1085550 | > loss_text_ce: 0.023382369428873062 (0.03213180613189638) | > loss_mel_ce: 0.02625528909265995 (nan) | > loss: 0.00019697485549841076 (nan) | > current_lr: 5e-06 | > step_time: 0.179 (0.13685914265069232) | > loader_time: 0.0049 (0.004622332475495797)  --> TIME: 2025-02-26 04:25:20 -- STEP: 14398/36938 -- GLOBAL_STEP: 1085600 | > loss_text_ce: 0.03102973662316799 (0.03213404508736779) | > loss_mel_ce: 2.821023941040039 (nan) | > loss: 0.011317674070596695 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13685607501145347) | > loader_time: 0.0047 (0.004622556381978033)  --> TIME: 2025-02-26 04:25:29 -- STEP: 14448/36938 -- GLOBAL_STEP: 1085650 | > loss_text_ce: 0.032284803688526154 (0.032134583685159944) | > loss_mel_ce: 3.330693483352661 (nan) | > loss: 0.013345152139663696 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.1368410771471321) | > loader_time: 0.0047 (0.004622664149945492)  --> TIME: 2025-02-26 04:25:39 -- STEP: 14498/36938 -- GLOBAL_STEP: 1085700 | > loss_text_ce: 0.033317964524030685 (0.03213276049300239) | > loss_mel_ce: 1.5404174327850342 (nan) | > loss: 0.006244982127100229 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13686316785261132) | > loader_time: 0.0048 (0.004622843828278914)  --> TIME: 2025-02-26 04:25:48 -- STEP: 14548/36938 -- GLOBAL_STEP: 1085750 | > loss_text_ce: 0.023920323699712753 (0.03212927188506576) | > loss_mel_ce: 3.1661784648895264 (nan) | > loss: 0.012659123167395592 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13685326401532466) | > loader_time: 0.0045 (0.004622749781457808)  --> TIME: 2025-02-26 04:25:57 -- STEP: 14598/36938 -- GLOBAL_STEP: 1085800 | > loss_text_ce: 0.033374786376953125 (0.032132293884686) | > loss_mel_ce: 2.4146947860717773 (nan) | > loss: 0.009714562445878983 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13684774658617568) | > loader_time: 0.0045 (0.00462275141640091)  --> TIME: 2025-02-26 04:26:06 -- STEP: 14648/36938 -- GLOBAL_STEP: 1085850 | > loss_text_ce: 0.032382939010858536 (0.032131720439961285) | > loss_mel_ce: 2.8515448570251465 (nan) | > loss: 0.011444158852100372 (nan) | > current_lr: 5e-06 | > step_time: 0.1906 (0.13683256868258764) | > loader_time: 0.0052 (0.004622734403558144)  --> TIME: 2025-02-26 04:26:16 -- STEP: 14698/36938 -- GLOBAL_STEP: 1085900 | > loss_text_ce: 0.029141554608941078 (0.032135535005070534) | > loss_mel_ce: 2.840425729751587 (nan) | > loss: 0.0113871730864048 (nan) | > current_lr: 5e-06 | > step_time: 0.1817 (0.13684264988068046) | > loader_time: 0.0047 (0.004622818191191958)  --> TIME: 2025-02-26 04:26:25 -- STEP: 14748/36938 -- GLOBAL_STEP: 1085950 | > loss_text_ce: 0.03111371025443077 (0.03213631418721262) | > loss_mel_ce: 2.7998859882354736 (nan) | > loss: 0.011234126053750515 (nan) | > current_lr: 5e-06 | > step_time: 0.1261 (0.13681519143071175) | > loader_time: 0.0045 (0.004622802942595186)  --> TIME: 2025-02-26 04:26:34 -- STEP: 14798/36938 -- GLOBAL_STEP: 1086000 | > loss_text_ce: 0.025272689759731293 (0.032137809626270324) | > loss_mel_ce: 2.515444755554199 (nan) | > loss: 0.010082212276756763 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.1368166491669212) | > loader_time: 0.0045 (0.0046231509672820465)  --> TIME: 2025-02-26 04:26:44 -- STEP: 14848/36938 -- GLOBAL_STEP: 1086050 | > loss_text_ce: 0.024915052577853203 (0.03213579791980849) | > loss_mel_ce: 2.829353094100952 (nan) | > loss: 0.011326461099088192 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13682584080780885) | > loader_time: 0.0046 (0.004623394009882006)  --> TIME: 2025-02-26 04:26:53 -- STEP: 14898/36938 -- GLOBAL_STEP: 1086100 | > loss_text_ce: 0.02661689929664135 (0.03214073677907688) | > loss_mel_ce: 2.6439528465270996 (nan) | > loss: 0.010597499087452888 (nan) | > current_lr: 5e-06 | > step_time: 0.161 (0.13681134330712644) | > loader_time: 0.0046 (0.004623519460504239)  --> TIME: 2025-02-26 04:27:02 -- STEP: 14948/36938 -- GLOBAL_STEP: 1086150 | > loss_text_ce: 0.03266249969601631 (0.03214130901033955) | > loss_mel_ce: 2.8436286449432373 (nan) | > loss: 0.011413853615522385 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13682500205002984) | > loader_time: 0.0048 (0.004623507875984035)  --> TIME: 2025-02-26 04:27:12 -- STEP: 14998/36938 -- GLOBAL_STEP: 1086200 | > loss_text_ce: 0.02861366607248783 (0.03214185338768863) | > loss_mel_ce: 2.6881301403045654 (nan) | > loss: 0.010780729353427887 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.1368222397348853) | > loader_time: 0.0046 (0.004623499738802861)  --> TIME: 2025-02-26 04:27:21 -- STEP: 15048/36938 -- GLOBAL_STEP: 1086250 | > loss_text_ce: 0.02835216373205185 (0.032142062851211115) | > loss_mel_ce: 1.0460149049758911 (nan) | > loss: 0.0042633614502847195 (nan) | > current_lr: 5e-06 | > step_time: 0.1118 (0.13681901277600939) | > loader_time: 0.0045 (0.004623446579881959)  --> TIME: 2025-02-26 04:27:31 -- STEP: 15098/36938 -- GLOBAL_STEP: 1086300 | > loss_text_ce: 0.031531162559986115 (0.032142654484600686) | > loss_mel_ce: 2.597501039505005 (nan) | > loss: 0.010432668030261993 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13683959249945454) | > loader_time: 0.0045 (0.004623380050325665)  --> TIME: 2025-02-26 04:27:40 -- STEP: 15148/36938 -- GLOBAL_STEP: 1086350 | > loss_text_ce: 0.026489414274692535 (0.032145651533229444) | > loss_mel_ce: 1.4197804927825928 (nan) | > loss: 0.005739166401326656 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.1368277812495156) | > loader_time: 0.0046 (0.004623502060074525)  --> TIME: 2025-02-26 04:27:49 -- STEP: 15198/36938 -- GLOBAL_STEP: 1086400 | > loss_text_ce: 0.032298408448696136 (0.0321482545538095) | > loss_mel_ce: 3.1393344402313232 (nan) | > loss: 0.012585844844579697 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13683775762677722) | > loader_time: 0.0045 (0.0046234801813620835)  --> TIME: 2025-02-26 04:27:59 -- STEP: 15248/36938 -- GLOBAL_STEP: 1086450 | > loss_text_ce: 0.03434992954134941 (0.03214671287302272) | > loss_mel_ce: 2.9328064918518066 (nan) | > loss: 0.011774430982768536 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13685100812539236) | > loader_time: 0.0045 (0.004623518738766657)  --> TIME: 2025-02-26 04:28:08 -- STEP: 15298/36938 -- GLOBAL_STEP: 1086500 | > loss_text_ce: 0.03795802965760231 (0.03214537177059766) | > loss_mel_ce: 2.823662281036377 (nan) | > loss: 0.011355637572705746 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13684154206716195) | > loader_time: 0.0045 (0.004623570525111646)  --> TIME: 2025-02-26 04:28:17 -- STEP: 15348/36938 -- GLOBAL_STEP: 1086550 | > loss_text_ce: 0.021493837237358093 (0.032146516778310145) | > loss_mel_ce: 2.4331817626953125 (nan) | > loss: 0.009740777313709259 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13684755372665572) | > loader_time: 0.0047 (0.004623568893752745)  --> TIME: 2025-02-26 04:28:27 -- STEP: 15398/36938 -- GLOBAL_STEP: 1086600 | > loss_text_ce: 0.0352681428194046 (0.03214536168138198) | > loss_mel_ce: 2.092695713043213 (nan) | > loss: 0.008444300852715969 (nan) | > current_lr: 5e-06 | > step_time: 0.1121 (0.13683659995681433) | > loader_time: 0.0046 (0.00462385245245267)  --> TIME: 2025-02-26 04:28:36 -- STEP: 15448/36938 -- GLOBAL_STEP: 1086650 | > loss_text_ce: 0.048542365431785583 (0.03214200972921824) | > loss_mel_ce: 0.8716871738433838 (nan) | > loss: 0.0036517048720270395 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.1368453363443768) | > loader_time: 0.0046 (0.004623878517649694)  --> TIME: 2025-02-26 04:28:45 -- STEP: 15498/36938 -- GLOBAL_STEP: 1086700 | > loss_text_ce: 0.03368895873427391 (0.032138178304581703) | > loss_mel_ce: 3.177872896194458 (nan) | > loss: 0.012744294479489326 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13684131310268882) | > loader_time: 0.0046 (0.004623782420899887)  --> TIME: 2025-02-26 04:28:55 -- STEP: 15548/36938 -- GLOBAL_STEP: 1086750 | > loss_text_ce: 0.03659084066748619 (0.032133485409803465) | > loss_mel_ce: 3.1042981147766113 (nan) | > loss: 0.012463845312595367 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.1368539103481499) | > loader_time: 0.0048 (0.004623749552393806)  --> TIME: 2025-02-26 04:29:04 -- STEP: 15598/36938 -- GLOBAL_STEP: 1086800 | > loss_text_ce: 0.03794518858194351 (0.0321333434934049) | > loss_mel_ce: 3.1556332111358643 (nan) | > loss: 0.01267293095588684 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1368478406375916) | > loader_time: 0.0045 (0.00462372074648117)  --> TIME: 2025-02-26 04:29:14 -- STEP: 15648/36938 -- GLOBAL_STEP: 1086850 | > loss_text_ce: 0.03708255663514137 (0.032134797754040184) | > loss_mel_ce: 2.304755210876465 (nan) | > loss: 0.009293007664382458 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.1368484785189903) | > loader_time: 0.0048 (0.004623672987785808)  --> TIME: 2025-02-26 04:29:23 -- STEP: 15698/36938 -- GLOBAL_STEP: 1086900 | > loss_text_ce: 0.026947174221277237 (0.0321350371705242) | > loss_mel_ce: 3.0385868549346924 (nan) | > loss: 0.012164819054305553 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13685543864589553) | > loader_time: 0.0045 (0.004623621857869487)  --> TIME: 2025-02-26 04:29:32 -- STEP: 15748/36938 -- GLOBAL_STEP: 1086950 | > loss_text_ce: 0.04470481723546982 (0.032140441342373316) | > loss_mel_ce: 2.97110652923584 (nan) | > loss: 0.011967506259679794 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13685133197085347) | > loader_time: 0.0046 (0.0046235925962938886)  --> TIME: 2025-02-26 04:29:41 -- STEP: 15798/36938 -- GLOBAL_STEP: 1087000 | > loss_text_ce: 0.02734721079468727 (0.032145721526375966) | > loss_mel_ce: 1.9240047931671143 (nan) | > loss: 0.007743460591882467 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13683952032908542) | > loader_time: 0.0045 (0.004623478312238227)  --> TIME: 2025-02-26 04:29:51 -- STEP: 15848/36938 -- GLOBAL_STEP: 1087050 | > loss_text_ce: 0.038713183254003525 (0.03214553770456851) | > loss_mel_ce: 2.8542850017547607 (nan) | > loss: 0.011480152606964111 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13683512320968624) | > loader_time: 0.0048 (0.004623518259100454)  --> TIME: 2025-02-26 04:30:00 -- STEP: 15898/36938 -- GLOBAL_STEP: 1087100 | > loss_text_ce: 0.027985960245132446 (0.03214299093067464) | > loss_mel_ce: 1.0153610706329346 (nan) | > loss: 0.004140266217291355 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1368231764678342) | > loader_time: 0.0045 (0.004623399483871124)  --> TIME: 2025-02-26 04:30:09 -- STEP: 15948/36938 -- GLOBAL_STEP: 1087150 | > loss_text_ce: 0.03183064982295036 (0.03214401681638819) | > loss_mel_ce: 2.602574586868286 (nan) | > loss: 0.010453988797962666 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13682942855977054) | > loader_time: 0.0046 (0.0046233767729999126)  --> TIME: 2025-02-26 04:30:18 -- STEP: 15998/36938 -- GLOBAL_STEP: 1087200 | > loss_text_ce: 0.03456879034638405 (0.032143371375709315) | > loss_mel_ce: 1.660233974456787 (nan) | > loss: 0.006725408136844635 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.13681742579866352) | > loader_time: 0.0048 (0.0046232866635842985)  --> TIME: 2025-02-26 04:30:27 -- STEP: 16048/36938 -- GLOBAL_STEP: 1087250 | > loss_text_ce: 0.03276802599430084 (0.03214297459094227) | > loss_mel_ce: 2.01147723197937 (nan) | > loss: 0.008112085051834583 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13680974341342134) | > loader_time: 0.0045 (0.00462330696530499)  --> TIME: 2025-02-26 04:30:37 -- STEP: 16098/36938 -- GLOBAL_STEP: 1087300 | > loss_text_ce: 0.034071940928697586 (0.03214009266939315) | > loss_mel_ce: 2.8806958198547363 (nan) | > loss: 0.01156653929501772 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13680874344606206) | > loader_time: 0.0045 (0.004623351948411439)  --> TIME: 2025-02-26 04:30:46 -- STEP: 16148/36938 -- GLOBAL_STEP: 1087350 | > loss_text_ce: 0.028551502153277397 (0.03213919094311042) | > loss_mel_ce: 2.810678005218506 (nan) | > loss: 0.011266784742474556 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.13679790189722738) | > loader_time: 0.0046 (0.004623620838465772)  --> TIME: 2025-02-26 04:30:56 -- STEP: 16198/36938 -- GLOBAL_STEP: 1087400 | > loss_text_ce: 0.04795776307582855 (0.03214014049350097) | > loss_mel_ce: 3.4953792095184326 (nan) | > loss: 0.014060861431062222 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.1368112386134043) | > loader_time: 0.0046 (0.0046235859165928775)  --> TIME: 2025-02-26 04:31:05 -- STEP: 16248/36938 -- GLOBAL_STEP: 1087450 | > loss_text_ce: 0.03378717973828316 (0.032139337447091965) | > loss_mel_ce: 2.7753031253814697 (nan) | > loss: 0.011147184297442436 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13681196328573553) | > loader_time: 0.0045 (0.0046234724264553415)  --> TIME: 2025-02-26 04:31:14 -- STEP: 16298/36938 -- GLOBAL_STEP: 1087500 | > loss_text_ce: 0.03288211300969124 (0.032138881999231005) | > loss_mel_ce: 2.5933406352996826 (nan) | > loss: 0.010421520099043846 (nan) | > current_lr: 5e-06 | > step_time: 0.1763 (0.13682222348778578) | > loader_time: 0.0049 (0.00462369961831625)  --> TIME: 2025-02-26 04:31:24 -- STEP: 16348/36938 -- GLOBAL_STEP: 1087550 | > loss_text_ce: 0.0382036417722702 (0.03213944239137384) | > loss_mel_ce: 3.086089849472046 (nan) | > loss: 0.012397991493344307 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13683134438444952) | > loader_time: 0.0046 (0.004623539791382523)  --> TIME: 2025-02-26 04:31:33 -- STEP: 16398/36938 -- GLOBAL_STEP: 1087600 | > loss_text_ce: 0.027263278141617775 (0.0321414923712364) | > loss_mel_ce: 0.02353820763528347 (nan) | > loss: 0.00020159321138635278 (nan) | > current_lr: 5e-06 | > step_time: 0.1116 (0.13681170361436634) | > loader_time: 0.0045 (0.004623592459642243)  --> TIME: 2025-02-26 04:31:43 -- STEP: 16448/36938 -- GLOBAL_STEP: 1087650 | > loss_text_ce: 0.03482108190655708 (0.03214129662508768) | > loss_mel_ce: 2.597078800201416 (nan) | > loss: 0.01044404786080122 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.1368350742189106) | > loader_time: 0.0047 (0.004623718052406719)  --> TIME: 2025-02-26 04:31:52 -- STEP: 16498/36938 -- GLOBAL_STEP: 1087700 | > loss_text_ce: 0.029711173847317696 (0.03214448822367761) | > loss_mel_ce: 3.3494396209716797 (nan) | > loss: 0.013409329578280449 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.1368392519061242) | > loader_time: 0.0046 (0.004623803792974995)  --> TIME: 2025-02-26 04:32:02 -- STEP: 16548/36938 -- GLOBAL_STEP: 1087750 | > loss_text_ce: 0.0419967919588089 (0.032146374546761065) | > loss_mel_ce: 2.7786214351654053 (nan) | > loss: 0.011192929930984974 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13685118044117162) | > loader_time: 0.0046 (0.004623835029768027)  --> TIME: 2025-02-26 04:32:11 -- STEP: 16598/36938 -- GLOBAL_STEP: 1087800 | > loss_text_ce: 0.03188236430287361 (0.032143970821485686) | > loss_mel_ce: 2.518559694290161 (nan) | > loss: 0.01012080255895853 (nan) | > current_lr: 5e-06 | > step_time: 0.1629 (0.13685425803925372) | > loader_time: 0.0047 (0.004623927255900083)  --> TIME: 2025-02-26 04:32:20 -- STEP: 16648/36938 -- GLOBAL_STEP: 1087850 | > loss_text_ce: 0.03689918294548988 (0.032148705713396036) | > loss_mel_ce: 3.2062642574310303 (nan) | > loss: 0.012869696132838726 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13685289424866923) | > loader_time: 0.0046 (0.00462397759720318)  --> TIME: 2025-02-26 04:32:30 -- STEP: 16698/36938 -- GLOBAL_STEP: 1087900 | > loss_text_ce: 0.03226841241121292 (0.03214706118191914) | > loss_mel_ce: 2.8385982513427734 (nan) | > loss: 0.011392328888177872 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13684586409766294) | > loader_time: 0.0046 (0.0046238649503986835)  --> TIME: 2025-02-26 04:32:39 -- STEP: 16748/36938 -- GLOBAL_STEP: 1087950 | > loss_text_ce: 0.02602691389620304 (0.032149139941757145) | > loss_mel_ce: 2.290766716003418 (nan) | > loss: 0.00919362623244524 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13683123301856132) | > loader_time: 0.0047 (0.00462390723363978)  --> TIME: 2025-02-26 04:32:48 -- STEP: 16798/36938 -- GLOBAL_STEP: 1088000 | > loss_text_ce: 0.040921833366155624 (0.0321528405579367) | > loss_mel_ce: 2.7627108097076416 (nan) | > loss: 0.011125527322292328 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.1368275446641416) | > loader_time: 0.0048 (0.004623744498833204)  --> TIME: 2025-02-26 04:32:57 -- STEP: 16848/36938 -- GLOBAL_STEP: 1088050 | > loss_text_ce: 0.033675309270620346 (0.03215093810806876) | > loss_mel_ce: 2.329525947570801 (nan) | > loss: 0.009377783164381981 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.1368265260092327) | > loader_time: 0.0045 (0.004623843606380996)  --> TIME: 2025-02-26 04:33:07 -- STEP: 16898/36938 -- GLOBAL_STEP: 1088100 | > loss_text_ce: 0.033045072108507156 (0.032145229750353936) | > loss_mel_ce: 3.0478498935699463 (nan) | > loss: 0.012225774116814137 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1368250555647296) | > loader_time: 0.0046 (0.004623886677961772)  --> TIME: 2025-02-26 04:33:16 -- STEP: 16948/36938 -- GLOBAL_STEP: 1088150 | > loss_text_ce: 0.028876012191176414 (0.032145796910676275) | > loss_mel_ce: 2.0710887908935547 (nan) | > loss: 0.008333194069564342 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1368194133120662) | > loader_time: 0.0045 (0.004623853600416261)  --> TIME: 2025-02-26 04:33:26 -- STEP: 16998/36938 -- GLOBAL_STEP: 1088200 | > loss_text_ce: 0.04503551498055458 (0.032145293044209475) | > loss_mel_ce: 3.100735664367676 (nan) | > loss: 0.012483219616115093 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13684024729719163) | > loader_time: 0.0046 (0.004623913290865607)  --> TIME: 2025-02-26 04:33:35 -- STEP: 17048/36938 -- GLOBAL_STEP: 1088250 | > loss_text_ce: 0.030982820317149162 (0.03214524352594359) | > loss_mel_ce: 1.9824920892715454 (nan) | > loss: 0.007989980280399323 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13683583995424375) | > loader_time: 0.0046 (0.004623949527740479)  --> TIME: 2025-02-26 04:33:44 -- STEP: 17098/36938 -- GLOBAL_STEP: 1088300 | > loss_text_ce: 0.031149525195360184 (0.03214360035422949) | > loss_mel_ce: 2.2050514221191406 (nan) | > loss: 0.008873813785612583 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13682266803830256) | > loader_time: 0.0046 (0.00462397368613259)  --> TIME: 2025-02-26 04:33:53 -- STEP: 17148/36938 -- GLOBAL_STEP: 1088350 | > loss_text_ce: 0.02674120105803013 (0.032143553337010335) | > loss_mel_ce: 2.774374008178711 (nan) | > loss: 0.011115537025034428 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13682175083040277) | > loader_time: 0.0047 (0.00462395286459953)  --> TIME: 2025-02-26 04:34:02 -- STEP: 17198/36938 -- GLOBAL_STEP: 1088400 | > loss_text_ce: 0.030738631263375282 (0.03214718406110828) | > loss_mel_ce: 3.0260555744171143 (nan) | > loss: 0.012130136601626873 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1368106659768224) | > loader_time: 0.0047 (0.004623848957465907)  --> TIME: 2025-02-26 04:34:12 -- STEP: 17248/36938 -- GLOBAL_STEP: 1088450 | > loss_text_ce: 0.026888258755207062 (0.032148283267508355) | > loss_mel_ce: 0.11083602905273438 (nan) | > loss: 0.0005465249414555728 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.13680654346721735) | > loader_time: 0.0046 (0.004623718711794641)  --> TIME: 2025-02-26 04:34:21 -- STEP: 17298/36938 -- GLOBAL_STEP: 1088500 | > loss_text_ce: 0.021317966282367706 (0.032145073828228496) | > loss_mel_ce: 2.9025418758392334 (nan) | > loss: 0.011602618731558323 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.1368106708097684) | > loader_time: 0.0046 (0.004623631656976203)  --> TIME: 2025-02-26 04:34:30 -- STEP: 17348/36938 -- GLOBAL_STEP: 1088550 | > loss_text_ce: 0.02351384051144123 (0.03214478986877825) | > loss_mel_ce: 2.5687694549560547 (nan) | > loss: 0.010286838747560978 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13680954032541284) | > loader_time: 0.0045 (0.004623762838901509)  --> TIME: 2025-02-26 04:34:40 -- STEP: 17398/36938 -- GLOBAL_STEP: 1088600 | > loss_text_ce: 0.029435379430651665 (0.03214382741015484) | > loss_mel_ce: 2.729349374771118 (nan) | > loss: 0.010947559028863907 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13681283359185545) | > loader_time: 0.0045 (0.004624245193912077)  --> TIME: 2025-02-26 04:34:49 -- STEP: 17448/36938 -- GLOBAL_STEP: 1088650 | > loss_text_ce: 0.030255984514951706 (0.03214525270468118) | > loss_mel_ce: 3.2431414127349854 (nan) | > loss: 0.01298967283219099 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.1368139001110834) | > loader_time: 0.0046 (0.004624467645419516)  --> TIME: 2025-02-26 04:34:58 -- STEP: 17498/36938 -- GLOBAL_STEP: 1088700 | > loss_text_ce: 0.019308093935251236 (0.03214496794023751) | > loss_mel_ce: 1.9855068922042847 (nan) | > loss: 0.00795561634004116 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13680686678446577) | > loader_time: 0.0046 (0.004624507238911501)  --> TIME: 2025-02-26 04:35:08 -- STEP: 17548/36938 -- GLOBAL_STEP: 1088750 | > loss_text_ce: 0.02824828401207924 (0.03214384448692878) | > loss_mel_ce: 2.8917033672332764 (nan) | > loss: 0.011587110348045826 (nan) | > current_lr: 5e-06 | > step_time: 0.1474 (0.13681349921177585) | > loader_time: 0.0046 (0.004624484705984772)  --> TIME: 2025-02-26 04:35:17 -- STEP: 17598/36938 -- GLOBAL_STEP: 1088800 | > loss_text_ce: 0.03431731089949608 (0.03214250756128084) | > loss_mel_ce: 2.62246036529541 (nan) | > loss: 0.010542768985033035 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13681350004701456) | > loader_time: 0.0046 (0.004624622113895172)  --> TIME: 2025-02-26 04:35:26 -- STEP: 17648/36938 -- GLOBAL_STEP: 1088850 | > loss_text_ce: 0.024249771609902382 (0.0321429599236829) | > loss_mel_ce: 2.7345433235168457 (nan) | > loss: 0.010947592556476593 (nan) | > current_lr: 5e-06 | > step_time: 0.1763 (0.1368156366475797) | > loader_time: 0.005 (0.00462463392740148)  --> TIME: 2025-02-26 04:35:36 -- STEP: 17698/36938 -- GLOBAL_STEP: 1088900 | > loss_text_ce: 0.028074661269783974 (0.032145278568799604) | > loss_mel_ce: 2.8514699935913086 (nan) | > loss: 0.011426765471696854 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13680845549556803) | > loader_time: 0.0045 (0.004624615740487776)  --> TIME: 2025-02-26 04:35:45 -- STEP: 17748/36938 -- GLOBAL_STEP: 1088950 | > loss_text_ce: 0.034197065979242325 (0.03214638425029472) | > loss_mel_ce: 2.5276122093200684 (nan) | > loss: 0.010165910236537457 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13682341515171584) | > loader_time: 0.0045 (0.004624629399516928)  --> TIME: 2025-02-26 04:35:55 -- STEP: 17798/36938 -- GLOBAL_STEP: 1089000 | > loss_text_ce: 0.035891804844141006 (0.03214294554566026) | > loss_mel_ce: 3.0521392822265625 (nan) | > loss: 0.012254091911017895 (nan) | > current_lr: 5e-06 | > step_time: 0.157 (0.13682246809662654) | > loader_time: 0.0046 (0.004624571086503324)  --> TIME: 2025-02-26 04:36:04 -- STEP: 17848/36938 -- GLOBAL_STEP: 1089050 | > loss_text_ce: 0.025001034140586853 (0.0321362686756447) | > loss_mel_ce: 2.4378747940063477 (nan) | > loss: 0.009773317724466324 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13681485169621008) | > loader_time: 0.0046 (0.004624630639714443)  --> TIME: 2025-02-26 04:36:13 -- STEP: 17898/36938 -- GLOBAL_STEP: 1089100 | > loss_text_ce: 0.03360188752412796 (0.03213668921897829) | > loss_mel_ce: 3.061288833618164 (nan) | > loss: 0.01228131353855133 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13680427061644226) | > loader_time: 0.0046 (0.004624619352336655)  --> TIME: 2025-02-26 04:36:22 -- STEP: 17948/36938 -- GLOBAL_STEP: 1089150 | > loss_text_ce: 0.033121541142463684 (0.032132826770987846) | > loss_mel_ce: 3.075986623764038 (nan) | > loss: 0.012337732128798962 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.1368056588723393) | > loader_time: 0.0046 (0.004624599294086493)  --> TIME: 2025-02-26 04:36:31 -- STEP: 17998/36938 -- GLOBAL_STEP: 1089200 | > loss_text_ce: 0.03427032753825188 (0.03213359469838446) | > loss_mel_ce: 2.492856025695801 (nan) | > loss: 0.010028279386460781 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13680064455643373) | > loader_time: 0.0045 (0.004624541513998524)  --> TIME: 2025-02-26 04:36:41 -- STEP: 18048/36938 -- GLOBAL_STEP: 1089250 | > loss_text_ce: 0.02267780900001526 (0.0321291891684801) | > loss_mel_ce: 1.6045985221862793 (nan) | > loss: 0.00645744614303112 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13680440476758643) | > loader_time: 0.0045 (0.004624500738602162)  --> TIME: 2025-02-26 04:36:50 -- STEP: 18098/36938 -- GLOBAL_STEP: 1089300 | > loss_text_ce: 0.02411825768649578 (0.032127167716772764) | > loss_mel_ce: 2.3113560676574707 (nan) | > loss: 0.009267755784094334 (nan) | > current_lr: 5e-06 | > step_time: 0.1522 (0.1368048043261045) | > loader_time: 0.0048 (0.004624535436985416)  --> TIME: 2025-02-26 04:37:00 -- STEP: 18148/36938 -- GLOBAL_STEP: 1089350 | > loss_text_ce: 0.02262480929493904 (0.03212588292540182) | > loss_mel_ce: 2.659404993057251 (nan) | > loss: 0.010642975568771362 (nan) | > current_lr: 5e-06 | > step_time: 0.1083 (0.1368320188334206) | > loader_time: 0.0065 (0.0046247963157024915)  --> TIME: 2025-02-26 04:37:09 -- STEP: 18198/36938 -- GLOBAL_STEP: 1089400 | > loss_text_ce: 0.031509943306446075 (0.03213037017397933) | > loss_mel_ce: 2.618983745574951 (nan) | > loss: 0.010517832823097706 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13682070573957017) | > loader_time: 0.0047 (0.004624726327124085)  --> TIME: 2025-02-26 04:37:19 -- STEP: 18248/36938 -- GLOBAL_STEP: 1089450 | > loss_text_ce: 0.03560987859964371 (0.032126428535639046) | > loss_mel_ce: 3.1161317825317383 (nan) | > loss: 0.012506912462413311 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.136831157657463) | > loader_time: 0.0046 (0.0046248386456014265)  --> TIME: 2025-02-26 04:37:28 -- STEP: 18298/36938 -- GLOBAL_STEP: 1089500 | > loss_text_ce: 0.029289331287145615 (0.03212867735855847) | > loss_mel_ce: 2.6948063373565674 (nan) | > loss: 0.010809903964400291 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13681125055832766) | > loader_time: 0.0045 (0.004624762278669511)  --> TIME: 2025-02-26 04:37:37 -- STEP: 18348/36938 -- GLOBAL_STEP: 1089550 | > loss_text_ce: 0.034137994050979614 (0.032128917180238664) | > loss_mel_ce: 2.7922756671905518 (nan) | > loss: 0.011215928010642529 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13679773509437365) | > loader_time: 0.0045 (0.0046249740467316875)  --> TIME: 2025-02-26 04:37:46 -- STEP: 18398/36938 -- GLOBAL_STEP: 1089600 | > loss_text_ce: 0.03571286052465439 (0.03212788414907819) | > loss_mel_ce: 2.415682792663574 (nan) | > loss: 0.009727761149406433 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.136792443746949) | > loader_time: 0.0044 (0.004624772530584859)  --> TIME: 2025-02-26 04:37:55 -- STEP: 18448/36938 -- GLOBAL_STEP: 1089650 | > loss_text_ce: 0.034358058124780655 (0.03213089375380255) | > loss_mel_ce: 3.0260705947875977 (nan) | > loss: 0.012144559063017368 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.1367899196776526) | > loader_time: 0.0045 (0.004624557670881104)  --> TIME: 2025-02-26 04:38:04 -- STEP: 18498/36938 -- GLOBAL_STEP: 1089700 | > loss_text_ce: 0.023335019126534462 (0.03212924014786193) | > loss_mel_ce: 2.651076555252075 (nan) | > loss: 0.010612744837999344 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13679454368905106) | > loader_time: 0.0045 (0.004624580793064187)  --> TIME: 2025-02-26 04:38:14 -- STEP: 18548/36938 -- GLOBAL_STEP: 1089750 | > loss_text_ce: 0.03679383918642998 (0.03213153743243937) | > loss_mel_ce: 2.758737802505493 (nan) | > loss: 0.011093380860984325 (nan) | > current_lr: 5e-06 | > step_time: 0.1727 (0.13680038998192223) | > loader_time: 0.0049 (0.004624465904231726)  --> TIME: 2025-02-26 04:38:23 -- STEP: 18598/36938 -- GLOBAL_STEP: 1089800 | > loss_text_ce: 0.03426087647676468 (0.03213071514548768) | > loss_mel_ce: 3.199395179748535 (nan) | > loss: 0.012831970117986202 (nan) | > current_lr: 5e-06 | > step_time: 0.17 (0.13681257701789218) | > loader_time: 0.0047 (0.004624600153557013)  --> TIME: 2025-02-26 04:38:33 -- STEP: 18648/36938 -- GLOBAL_STEP: 1089850 | > loss_text_ce: 0.03346826508641243 (0.03213121445253413) | > loss_mel_ce: 2.7140631675720215 (nan) | > loss: 0.01090290304273367 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13680433693846472) | > loader_time: 0.0045 (0.004624618897362362)  --> TIME: 2025-02-26 04:38:42 -- STEP: 18698/36938 -- GLOBAL_STEP: 1089900 | > loss_text_ce: 0.01834164559841156 (0.03212919230752673) | > loss_mel_ce: 1.1777169704437256 (nan) | > loss: 0.004746264778077602 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13681117061982573) | > loader_time: 0.0045 (0.0046245411432086474)  --> TIME: 2025-02-26 04:38:51 -- STEP: 18748/36938 -- GLOBAL_STEP: 1089950 | > loss_text_ce: 0.0336952768266201 (0.03213053165730498) | > loss_mel_ce: 2.7557260990142822 (nan) | > loss: 0.011069132946431637 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1368109515691475) | > loader_time: 0.0045 (0.004624619358303651)  --> TIME: 2025-02-26 04:39:01 -- STEP: 18798/36938 -- GLOBAL_STEP: 1090000 | > loss_text_ce: 0.029171153903007507 (0.032127451108902555) | > loss_mel_ce: 2.7377078533172607 (nan) | > loss: 0.01097967941313982 (nan) | > current_lr: 5e-06 | > step_time: 0.1824 (0.13682093260351347) | > loader_time: 0.0048 (0.004624803138040505) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1090000.pth  --> TIME: 2025-02-26 04:39:16 -- STEP: 18848/36938 -- GLOBAL_STEP: 1090050 | > loss_text_ce: 0.039167359471321106 (0.03212717213491608) | > loss_mel_ce: 2.8514609336853027 (nan) | > loss: 0.011470748111605644 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13681421945020122) | > loader_time: 0.0046 (0.00462485680353459)  --> TIME: 2025-02-26 04:39:25 -- STEP: 18898/36938 -- GLOBAL_STEP: 1090100 | > loss_text_ce: 0.027104459702968597 (0.032130695123973194) | > loss_mel_ce: 2.949354648590088 (nan) | > loss: 0.01181134581565857 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.1368093175072553) | > loader_time: 0.0047 (0.0046251901483672)  --> TIME: 2025-02-26 04:39:34 -- STEP: 18948/36938 -- GLOBAL_STEP: 1090150 | > loss_text_ce: 0.03475533053278923 (0.032129138792999105) | > loss_mel_ce: 2.56352162361145 (nan) | > loss: 0.010310622863471508 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.13681266985791124) | > loader_time: 0.0046 (0.004625171831898829)  --> TIME: 2025-02-26 04:39:44 -- STEP: 18998/36938 -- GLOBAL_STEP: 1090200 | > loss_text_ce: 0.035357214510440826 (0.032125057907997835) | > loss_mel_ce: 2.9031176567077637 (nan) | > loss: 0.011660614982247353 (nan) | > current_lr: 5e-06 | > step_time: 0.1517 (0.1368154358745859) | > loader_time: 0.0048 (0.004625372114100196)  --> TIME: 2025-02-26 04:39:53 -- STEP: 19048/36938 -- GLOBAL_STEP: 1090250 | > loss_text_ce: 0.028422459959983826 (0.03212390596515229) | > loss_mel_ce: 2.8081891536712646 (nan) | > loss: 0.01125639583915472 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.13681565268307017) | > loader_time: 0.0048 (0.004625356082203304)  --> TIME: 2025-02-26 04:40:02 -- STEP: 19098/36938 -- GLOBAL_STEP: 1090300 | > loss_text_ce: 0.034098826348781586 (0.03212743625981102) | > loss_mel_ce: 3.0948445796966553 (nan) | > loss: 0.012416442856192589 (nan) | > current_lr: 5e-06 | > step_time: 0.1343 (0.13681844415833405) | > loader_time: 0.0046 (0.004625401867410998)  --> TIME: 2025-02-26 04:40:12 -- STEP: 19148/36938 -- GLOBAL_STEP: 1090350 | > loss_text_ce: 0.03834371268749237 (0.03212896113163096) | > loss_mel_ce: 3.0203540325164795 (nan) | > loss: 0.012137689627707005 (nan) | > current_lr: 5e-06 | > step_time: 0.162 (0.1368189444864666) | > loader_time: 0.0046 (0.004625357813543279)  --> TIME: 2025-02-26 04:40:21 -- STEP: 19198/36938 -- GLOBAL_STEP: 1090400 | > loss_text_ce: 0.0293342974036932 (0.032125331015803345) | > loss_mel_ce: 2.3208515644073486 (nan) | > loss: 0.00932613480836153 (nan) | > current_lr: 5e-06 | > step_time: 0.1078 (0.13681191482299504) | > loader_time: 0.0045 (0.004625389049147321)  --> TIME: 2025-02-26 04:40:30 -- STEP: 19248/36938 -- GLOBAL_STEP: 1090450 | > loss_text_ce: 0.029281113296747208 (0.03212791395873734) | > loss_mel_ce: 2.949016571044922 (nan) | > loss: 0.011818642728030682 (nan) | > current_lr: 5e-06 | > step_time: 0.197 (0.13681758947651845) | > loader_time: 0.0049 (0.004625350459852713)  --> TIME: 2025-02-26 04:40:40 -- STEP: 19298/36938 -- GLOBAL_STEP: 1090500 | > loss_text_ce: 0.02820775844156742 (0.032129596081895216) | > loss_mel_ce: 2.4356765747070312 (nan) | > loss: 0.009777319617569447 (nan) | > current_lr: 5e-06 | > step_time: 0.1531 (0.1368331262788344) | > loader_time: 0.0067 (0.004625498797562665)  --> TIME: 2025-02-26 04:40:49 -- STEP: 19348/36938 -- GLOBAL_STEP: 1090550 | > loss_text_ce: 0.03570624813437462 (0.032133039268912235) | > loss_mel_ce: 1.8038222789764404 (nan) | > loss: 0.007299717050045729 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13683318073946166) | > loader_time: 0.0046 (0.004625522501340004)  --> TIME: 2025-02-26 04:40:59 -- STEP: 19398/36938 -- GLOBAL_STEP: 1090600 | > loss_text_ce: 0.025833498686552048 (0.032133511331073145) | > loss_mel_ce: 3.0253093242645264 (nan) | > loss: 0.012107710354030132 (nan) | > current_lr: 5e-06 | > step_time: 0.1261 (0.13682585425543525) | > loader_time: 0.0045 (0.004625380315759984)  --> TIME: 2025-02-26 04:41:08 -- STEP: 19448/36938 -- GLOBAL_STEP: 1090650 | > loss_text_ce: 0.029509246349334717 (0.032131575706235774) | > loss_mel_ce: 2.486851930618286 (nan) | > loss: 0.009985561482608318 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13682677043607214) | > loader_time: 0.0047 (0.004625506064667216)  --> TIME: 2025-02-26 04:41:17 -- STEP: 19498/36938 -- GLOBAL_STEP: 1090700 | > loss_text_ce: 0.03172427788376808 (0.032130183839950775) | > loss_mel_ce: 2.871715784072876 (nan) | > loss: 0.011521588079631329 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.1368178525867553) | > loader_time: 0.0048 (0.004625500844238785)  --> TIME: 2025-02-26 04:41:26 -- STEP: 19548/36938 -- GLOBAL_STEP: 1090750 | > loss_text_ce: 0.03523297980427742 (0.03212870626028279) | > loss_mel_ce: 2.8036632537841797 (nan) | > loss: 0.011265462264418602 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.1368076358808435) | > loader_time: 0.0045 (0.004625537509149332)  --> TIME: 2025-02-26 04:41:35 -- STEP: 19598/36938 -- GLOBAL_STEP: 1090800 | > loss_text_ce: 0.033366695046424866 (0.03212960148580879) | > loss_mel_ce: 3.035930871963501 (nan) | > loss: 0.012179752811789513 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.136809654607616) | > loader_time: 0.0047 (0.004625587064838609)  --> TIME: 2025-02-26 04:41:45 -- STEP: 19648/36938 -- GLOBAL_STEP: 1090850 | > loss_text_ce: 0.03084697388112545 (0.03212817597558629) | > loss_mel_ce: 3.4291188716888428 (nan) | > loss: 0.013730023987591267 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13681670607155613) | > loader_time: 0.0046 (0.004625592016727213)  --> TIME: 2025-02-26 04:41:54 -- STEP: 19698/36938 -- GLOBAL_STEP: 1090900 | > loss_text_ce: 0.03548097237944603 (0.03213093103996953) | > loss_mel_ce: 3.0450432300567627 (nan) | > loss: 0.012224303558468819 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.13680736003569363) | > loader_time: 0.0047 (0.00462550988160047)  --> TIME: 2025-02-26 04:42:03 -- STEP: 19748/36938 -- GLOBAL_STEP: 1090950 | > loss_text_ce: 0.030834777280688286 (0.03212614354637107) | > loss_mel_ce: 1.105811595916748 (nan) | > loss: 0.004510501865297556 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13679981708574895) | > loader_time: 0.0045 (0.004625661039439341)  --> TIME: 2025-02-26 04:42:12 -- STEP: 19798/36938 -- GLOBAL_STEP: 1091000 | > loss_text_ce: 0.03146268054842949 (0.0321256654210129) | > loss_mel_ce: 2.4061782360076904 (nan) | > loss: 0.009673179127275944 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13679351110821564) | > loader_time: 0.0046 (0.004625605518064762)  --> TIME: 2025-02-26 04:42:22 -- STEP: 19848/36938 -- GLOBAL_STEP: 1091050 | > loss_text_ce: 0.0258751530200243 (0.03212314812594677) | > loss_mel_ce: 2.660484552383423 (nan) | > loss: 0.010660157538950443 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13679882321853581) | > loader_time: 0.0045 (0.004625721883024243)  --> TIME: 2025-02-26 04:42:31 -- STEP: 19898/36938 -- GLOBAL_STEP: 1091100 | > loss_text_ce: 0.034548189491033554 (0.032123367483769115) | > loss_mel_ce: 2.7031102180480957 (nan) | > loss: 0.01086372509598732 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13678705427488236) | > loader_time: 0.0072 (0.004625800207327666)  --> TIME: 2025-02-26 04:42:40 -- STEP: 19948/36938 -- GLOBAL_STEP: 1091150 | > loss_text_ce: 0.028242267668247223 (0.032122919818803455) | > loss_mel_ce: 2.0977272987365723 (nan) | > loss: 0.008436388336122036 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13678627730563284) | > loader_time: 0.0045 (0.004625783180315607)  --> TIME: 2025-02-26 04:42:49 -- STEP: 19998/36938 -- GLOBAL_STEP: 1091200 | > loss_text_ce: 0.02310555800795555 (0.03212252363200432) | > loss_mel_ce: 0.9465212821960449 (nan) | > loss: 0.0038477256894111633 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13677544140293552) | > loader_time: 0.0045 (0.00462571945604366)  --> TIME: 2025-02-26 04:42:58 -- STEP: 20048/36938 -- GLOBAL_STEP: 1091250 | > loss_text_ce: 0.027938101440668106 (0.032124171498047846) | > loss_mel_ce: 2.4421186447143555 (nan) | > loss: 0.009801813401281834 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.13677023371409533) | > loader_time: 0.0046 (0.004625761202118062)  --> TIME: 2025-02-26 04:43:08 -- STEP: 20098/36938 -- GLOBAL_STEP: 1091300 | > loss_text_ce: 0.0373074896633625 (0.03212795383827475) | > loss_mel_ce: 2.5826847553253174 (nan) | > loss: 0.010396795347332954 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13677787646712072) | > loader_time: 0.0045 (0.004625860358105452)  --> TIME: 2025-02-26 04:43:17 -- STEP: 20148/36938 -- GLOBAL_STEP: 1091350 | > loss_text_ce: 0.023009218275547028 (0.032127345774595494) | > loss_mel_ce: 2.771289825439453 (nan) | > loss: 0.011088489554822445 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13677741466165902) | > loader_time: 0.0045 (0.004625689954967851)  --> TIME: 2025-02-26 04:43:27 -- STEP: 20198/36938 -- GLOBAL_STEP: 1091400 | > loss_text_ce: 0.027134010568261147 (0.032127864264502703) | > loss_mel_ce: 2.5067546367645264 (nan) | > loss: 0.010055113583803177 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.1367812408339993) | > loader_time: 0.0045 (0.004625840663484967)  --> TIME: 2025-02-26 04:43:36 -- STEP: 20248/36938 -- GLOBAL_STEP: 1091450 | > loss_text_ce: 0.022181298583745956 (0.03212791237071599) | > loss_mel_ce: 2.6184909343719482 (nan) | > loss: 0.010478858835995197 (nan) | > current_lr: 5e-06 | > step_time: 0.1502 (0.1367592528868648) | > loader_time: 0.0048 (0.004625908344548385)  --> TIME: 2025-02-26 04:43:45 -- STEP: 20298/36938 -- GLOBAL_STEP: 1091500 | > loss_text_ce: 0.038102615624666214 (0.03212767214044369) | > loss_mel_ce: 2.7044079303741455 (nan) | > loss: 0.01088297925889492 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13675638288920242) | > loader_time: 0.0045 (0.004625786629929688)  --> TIME: 2025-02-26 04:43:54 -- STEP: 20348/36938 -- GLOBAL_STEP: 1091550 | > loss_text_ce: 0.028481971472501755 (0.03212291667715146) | > loss_mel_ce: 2.6607375144958496 (nan) | > loss: 0.010671506635844707 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13675962713956613) | > loader_time: 0.0046 (0.00462582576759958)  --> TIME: 2025-02-26 04:44:03 -- STEP: 20398/36938 -- GLOBAL_STEP: 1091600 | > loss_text_ce: 0.03977539390325546 (0.032124953304878995) | > loss_mel_ce: 2.774439573287964 (nan) | > loss: 0.01116752065718174 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13675397618726998) | > loader_time: 0.0046 (0.004626198356812441)  --> TIME: 2025-02-26 04:44:13 -- STEP: 20448/36938 -- GLOBAL_STEP: 1091650 | > loss_text_ce: 0.02409520372748375 (0.03212576781197119) | > loss_mel_ce: 2.1829211711883545 (nan) | > loss: 0.008758001960814 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13674868720238223) | > loader_time: 0.0045 (0.004626283261790898)  --> TIME: 2025-02-26 04:44:22 -- STEP: 20498/36938 -- GLOBAL_STEP: 1091700 | > loss_text_ce: 0.028092466294765472 (0.03212279969117898) | > loss_mel_ce: 2.874077320098877 (nan) | > loss: 0.011516546830534935 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13675025632083931) | > loader_time: 0.0045 (0.004626339465213971)  --> TIME: 2025-02-26 04:44:31 -- STEP: 20548/36938 -- GLOBAL_STEP: 1091750 | > loss_text_ce: 0.02645888179540634 (0.032121077633740204) | > loss_mel_ce: 2.792943000793457 (nan) | > loss: 0.011188103817403316 (nan) | > current_lr: 5e-06 | > step_time: 0.1785 (0.13675498345122417) | > loader_time: 0.0047 (0.0046263146265856795)  --> TIME: 2025-02-26 04:44:41 -- STEP: 20598/36938 -- GLOBAL_STEP: 1091800 | > loss_text_ce: 0.033582091331481934 (0.032123376514562756) | > loss_mel_ce: 2.95762300491333 (nan) | > loss: 0.011869862675666809 (nan) | > current_lr: 5e-06 | > step_time: 0.1785 (0.1367691090088678) | > loader_time: 0.0048 (0.004626670882868324)  --> TIME: 2025-02-26 04:44:51 -- STEP: 20648/36938 -- GLOBAL_STEP: 1091850 | > loss_text_ce: 0.03678203001618385 (0.032122139127104675) | > loss_mel_ce: 2.049952268600464 (nan) | > loss: 0.00828069169074297 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13678408035101358) | > loader_time: 0.0046 (0.004626738036708844)  --> TIME: 2025-02-26 04:45:00 -- STEP: 20698/36938 -- GLOBAL_STEP: 1091900 | > loss_text_ce: 0.029921820387244225 (0.0321247357533949) | > loss_mel_ce: 2.650151491165161 (nan) | > loss: 0.010635212063789368 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.136791041259294) | > loader_time: 0.0047 (0.004626720006229319)  --> TIME: 2025-02-26 04:45:10 -- STEP: 20748/36938 -- GLOBAL_STEP: 1091950 | > loss_text_ce: 0.027587661519646645 (0.032122781677480235) | > loss_mel_ce: 2.423828125 (nan) | > loss: 0.00972784124314785 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.1367930667304296) | > loader_time: 0.0044 (0.004626751417182469)  --> TIME: 2025-02-26 04:45:19 -- STEP: 20798/36938 -- GLOBAL_STEP: 1092000 | > loss_text_ce: 0.03456933796405792 (0.03212501239281405) | > loss_mel_ce: 3.460627555847168 (nan) | > loss: 0.01386982947587967 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.1367869908012489) | > loader_time: 0.0047 (0.004626824725349427)  --> TIME: 2025-02-26 04:45:28 -- STEP: 20848/36938 -- GLOBAL_STEP: 1092050 | > loss_text_ce: 0.03525932878255844 (0.032121965131521754) | > loss_mel_ce: 2.8998005390167236 (nan) | > loss: 0.011647063307464123 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.13678425429238808) | > loader_time: 0.0046 (0.004626774241260078)  --> TIME: 2025-02-26 04:45:38 -- STEP: 20898/36938 -- GLOBAL_STEP: 1092100 | > loss_text_ce: 0.018053090199828148 (0.032122832477363195) | > loss_mel_ce: 1.0261725187301636 (nan) | > loss: 0.004143752623349428 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.136794903618558) | > loader_time: 0.0046 (0.004626905853091863)  --> TIME: 2025-02-26 04:45:47 -- STEP: 20948/36938 -- GLOBAL_STEP: 1092150 | > loss_text_ce: 0.033903978765010834 (0.032121836483140175) | > loss_mel_ce: 3.0294721126556396 (nan) | > loss: 0.012156255543231964 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13678897883979954) | > loader_time: 0.0046 (0.0046269240806365445)  --> TIME: 2025-02-26 04:45:56 -- STEP: 20998/36938 -- GLOBAL_STEP: 1092200 | > loss_text_ce: 0.03900471329689026 (0.03212260885378088) | > loss_mel_ce: 1.8954639434814453 (nan) | > loss: 0.007676463108509779 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13678521710993624) | > loader_time: 0.0048 (0.004626848831870738)  --> TIME: 2025-02-26 04:46:06 -- STEP: 21048/36938 -- GLOBAL_STEP: 1092250 | > loss_text_ce: 0.024400362744927406 (0.03212061554421979) | > loss_mel_ce: 2.9564225673675537 (nan) | > loss: 0.011828663758933544 (nan) | > current_lr: 5e-06 | > step_time: 0.1591 (0.1367847557441223) | > loader_time: 0.0046 (0.004627213533365971)  --> TIME: 2025-02-26 04:46:15 -- STEP: 21098/36938 -- GLOBAL_STEP: 1092300 | > loss_text_ce: 0.033133309334516525 (0.03212091113941349) | > loss_mel_ce: 1.7013307809829712 (nan) | > loss: 0.006882794201374054 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.1367828185122798) | > loader_time: 0.0046 (0.004627312582379242)  --> TIME: 2025-02-26 04:46:24 -- STEP: 21148/36938 -- GLOBAL_STEP: 1092350 | > loss_text_ce: 0.034514449536800385 (0.032118651717652755) | > loss_mel_ce: 2.5077621936798096 (nan) | > loss: 0.010088399983942509 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13678426078075231) | > loader_time: 0.0045 (0.0046273211867366975)  --> TIME: 2025-02-26 04:46:33 -- STEP: 21198/36938 -- GLOBAL_STEP: 1092400 | > loss_text_ce: 0.0352732352912426 (0.0321173513605076) | > loss_mel_ce: 3.0943655967712402 (nan) | > loss: 0.012419202364981174 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13678000320611314) | > loader_time: 0.0045 (0.004627450275718079)  --> TIME: 2025-02-26 04:46:43 -- STEP: 21248/36938 -- GLOBAL_STEP: 1092450 | > loss_text_ce: 0.03907069191336632 (0.032116130746402016) | > loss_mel_ce: 2.8876421451568604 (nan) | > loss: 0.011613939888775349 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13678523787039262) | > loader_time: 0.0048 (0.004627546127212346)  --> TIME: 2025-02-26 04:46:52 -- STEP: 21298/36938 -- GLOBAL_STEP: 1092500 | > loss_text_ce: 0.032031986862421036 (0.032115971424458274) | > loss_mel_ce: 2.5585384368896484 (nan) | > loss: 0.010280041955411434 (nan) | > current_lr: 5e-06 | > step_time: 0.187 (0.1367768669634408) | > loader_time: 0.0048 (0.0046276780672169305)  --> TIME: 2025-02-26 04:47:01 -- STEP: 21348/36938 -- GLOBAL_STEP: 1092550 | > loss_text_ce: 0.04511898756027222 (0.0321138920048253) | > loss_mel_ce: 0.003161787986755371 (nan) | > loss: 0.00019159038492944092 (nan) | > current_lr: 5e-06 | > step_time: 0.1121 (0.13676580428764468) | > loader_time: 0.0045 (0.004627728553798426)  --> TIME: 2025-02-26 04:47:10 -- STEP: 21398/36938 -- GLOBAL_STEP: 1092600 | > loss_text_ce: 0.03464296832680702 (0.03211860998789318) | > loss_mel_ce: 2.6107072830200195 (nan) | > loss: 0.010497421957552433 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1367588590624091) | > loader_time: 0.0046 (0.004627752687365275)  --> TIME: 2025-02-26 04:47:20 -- STEP: 21448/36938 -- GLOBAL_STEP: 1092650 | > loss_text_ce: 0.03892875462770462 (0.03211900202851146) | > loss_mel_ce: 3.004016399383545 (nan) | > loss: 0.012075180187821388 (nan) | > current_lr: 5e-06 | > step_time: 0.1702 (0.1367664199540618) | > loader_time: 0.0048 (0.004627784978805399)  --> TIME: 2025-02-26 04:47:29 -- STEP: 21498/36938 -- GLOBAL_STEP: 1092700 | > loss_text_ce: 0.04704375937581062 (0.03211882483626205) | > loss_mel_ce: 1.1726998090744019 (nan) | > loss: 0.004840252920985222 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13676593303458637) | > loader_time: 0.0046 (0.0046277836274297825)  --> TIME: 2025-02-26 04:47:38 -- STEP: 21548/36938 -- GLOBAL_STEP: 1092750 | > loss_text_ce: 0.033326201140880585 (0.03211831247580273) | > loss_mel_ce: 0.06982421875 (nan) | > loss: 0.0004093270981684327 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13677864017614058) | > loader_time: 0.0045 (0.0046278979841578955)  --> TIME: 2025-02-26 04:47:48 -- STEP: 21598/36938 -- GLOBAL_STEP: 1092800 | > loss_text_ce: 0.028579240664839745 (0.032119722508483176) | > loss_mel_ce: 2.4735355377197266 (nan) | > loss: 0.009929027408361435 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13678715622903087) | > loader_time: 0.0045 (0.004627974996611814)  --> TIME: 2025-02-26 04:47:58 -- STEP: 21648/36938 -- GLOBAL_STEP: 1092850 | > loss_text_ce: 0.02648080326616764 (0.032118660446785664) | > loss_mel_ce: 2.783125162124634 (nan) | > loss: 0.011149231344461441 (nan) | > current_lr: 5e-06 | > step_time: 0.1662 (0.13680978677753883) | > loader_time: 0.0047 (0.004628089858881215)  --> TIME: 2025-02-26 04:48:08 -- STEP: 21698/36938 -- GLOBAL_STEP: 1092900 | > loss_text_ce: 0.03005785308778286 (0.03211871815236042) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13682608735628415) | > loader_time: 0.0047 (0.004628138538219605)  --> TIME: 2025-02-26 04:48:17 -- STEP: 21748/36938 -- GLOBAL_STEP: 1092950 | > loss_text_ce: 0.033862970769405365 (0.032117835878527574) | > loss_mel_ce: 2.7580137252807617 (nan) | > loss: 0.011078876443207264 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.1368178647626123) | > loader_time: 0.0047 (0.0046279781417678315)  --> TIME: 2025-02-26 04:48:26 -- STEP: 21798/36938 -- GLOBAL_STEP: 1093000 | > loss_text_ce: 0.032433610409498215 (0.03211487618520479) | > loss_mel_ce: 2.501953125 (nan) | > loss: 0.010057090781629086 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.136820341843139) | > loader_time: 0.0045 (0.004627999028696668)  --> TIME: 2025-02-26 04:48:35 -- STEP: 21848/36938 -- GLOBAL_STEP: 1093050 | > loss_text_ce: 0.034491896629333496 (0.03211271900977595) | > loss_mel_ce: 1.111480712890625 (nan) | > loss: 0.004547510761767626 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1368073164456413) | > loader_time: 0.0047 (0.004627937364386183)  --> TIME: 2025-02-26 04:48:45 -- STEP: 21898/36938 -- GLOBAL_STEP: 1093100 | > loss_text_ce: 0.026692738756537437 (0.032110879796539475) | > loss_mel_ce: 2.70818829536438 (nan) | > loss: 0.010852702893316746 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13681506830782184) | > loader_time: 0.0046 (0.004627946163706717)  --> TIME: 2025-02-26 04:48:54 -- STEP: 21948/36938 -- GLOBAL_STEP: 1093150 | > loss_text_ce: 0.030230814591050148 (0.032109960359354) | > loss_mel_ce: 2.29461669921875 (nan) | > loss: 0.009225585497915745 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13681218958474614) | > loader_time: 0.0046 (0.004627921432664058)  --> TIME: 2025-02-26 04:49:03 -- STEP: 21998/36938 -- GLOBAL_STEP: 1093200 | > loss_text_ce: 0.03152510151267052 (0.03211068134543721) | > loss_mel_ce: 2.5893194675445557 (nan) | > loss: 0.010400177910923958 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13680444961136584) | > loader_time: 0.0046 (0.004627833540672198)  --> TIME: 2025-02-26 04:49:13 -- STEP: 22048/36938 -- GLOBAL_STEP: 1093250 | > loss_text_ce: 0.03874482586979866 (0.03210699853110696) | > loss_mel_ce: 2.367171287536621 (nan) | > loss: 0.009547286666929722 (nan) | > current_lr: 5e-06 | > step_time: 0.1083 (0.1368090508746311) | > loader_time: 0.0045 (0.004627755282146665)  --> TIME: 2025-02-26 04:49:22 -- STEP: 22098/36938 -- GLOBAL_STEP: 1093300 | > loss_text_ce: 0.02928335592150688 (0.03210488044807607) | > loss_mel_ce: 2.985954523086548 (nan) | > loss: 0.011965230107307434 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.1368180173200146) | > loader_time: 0.0046 (0.004627719649643651)  --> TIME: 2025-02-26 04:49:32 -- STEP: 22148/36938 -- GLOBAL_STEP: 1093350 | > loss_text_ce: 0.021616851910948753 (0.03210474460236714) | > loss_mel_ce: 2.0201752185821533 (nan) | > loss: 0.008102349936962128 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13682282262703666) | > loader_time: 0.0047 (0.004627794646303951)  --> TIME: 2025-02-26 04:49:41 -- STEP: 22198/36938 -- GLOBAL_STEP: 1093400 | > loss_text_ce: 0.028067877516150475 (0.03210845073508503) | > loss_mel_ce: 2.4367897510528564 (nan) | > loss: 0.009781181812286377 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13682190530288832) | > loader_time: 0.0046 (0.004627813357621217)  --> TIME: 2025-02-26 04:49:50 -- STEP: 22248/36938 -- GLOBAL_STEP: 1093450 | > loss_text_ce: 0.038273267447948456 (0.03210793120766621) | > loss_mel_ce: 2.639385461807251 (nan) | > loss: 0.010625630617141724 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.13682259127888893) | > loader_time: 0.0046 (0.004627826862393511)  --> TIME: 2025-02-26 04:49:59 -- STEP: 22298/36938 -- GLOBAL_STEP: 1093500 | > loss_text_ce: 0.041813354939222336 (0.03210850727127669) | > loss_mel_ce: 1.167944312095642 (nan) | > loss: 0.004800626076757908 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1368214610296797) | > loader_time: 0.0045 (0.004627784150247038)  --> TIME: 2025-02-26 04:50:09 -- STEP: 22348/36938 -- GLOBAL_STEP: 1093550 | > loss_text_ce: 0.02603953145444393 (0.032108474489406805) | > loss_mel_ce: 2.3770177364349365 (nan) | > loss: 0.00953594222664833 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13682874022090105) | > loader_time: 0.0046 (0.004627911278956994)  --> TIME: 2025-02-26 04:50:18 -- STEP: 22398/36938 -- GLOBAL_STEP: 1093600 | > loss_text_ce: 0.035387370735406876 (0.0321082714414509) | > loss_mel_ce: 3.305854558944702 (nan) | > loss: 0.013258896768093109 (nan) | > current_lr: 5e-06 | > step_time: 0.1506 (0.13681902140525384) | > loader_time: 0.0046 (0.004627979592623829)  --> TIME: 2025-02-26 04:50:27 -- STEP: 22448/36938 -- GLOBAL_STEP: 1093650 | > loss_text_ce: 0.02470596693456173 (0.032108344179322554) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13681061187771051) | > loader_time: 0.0046 (0.004627983695850664)  --> TIME: 2025-02-26 04:50:37 -- STEP: 22498/36938 -- GLOBAL_STEP: 1093700 | > loss_text_ce: 0.02723895199596882 (0.03210822219832645) | > loss_mel_ce: 2.101372480392456 (nan) | > loss: 0.008446870371699333 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1368099150948874) | > loader_time: 0.0046 (0.0046280188098018725)  --> TIME: 2025-02-26 04:50:46 -- STEP: 22548/36938 -- GLOBAL_STEP: 1093750 | > loss_text_ce: 0.030009722337126732 (0.032109675239421334) | > loss_mel_ce: 2.840745449066162 (nan) | > loss: 0.01139188650995493 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13681737767988192) | > loader_time: 0.0045 (0.004627999259961596)  --> TIME: 2025-02-26 04:50:55 -- STEP: 22598/36938 -- GLOBAL_STEP: 1093800 | > loss_text_ce: 0.03181937336921692 (0.03210845072633799) | > loss_mel_ce: 3.0939247608184814 (nan) | > loss: 0.01240374706685543 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.1368125999740867) | > loader_time: 0.0047 (0.004627978150765874)  --> TIME: 2025-02-26 04:51:05 -- STEP: 22648/36938 -- GLOBAL_STEP: 1093850 | > loss_text_ce: 0.028728531673550606 (0.03210853847542778) | > loss_mel_ce: 2.7400524616241455 (nan) | > loss: 0.010987226851284504 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13680760143966278) | > loader_time: 0.0046 (0.004627963777398613)  --> TIME: 2025-02-26 04:51:14 -- STEP: 22698/36938 -- GLOBAL_STEP: 1093900 | > loss_text_ce: 0.03554602339863777 (0.032107322692586496) | > loss_mel_ce: 2.954045534133911 (nan) | > loss: 0.011863459832966328 (nan) | > current_lr: 5e-06 | > step_time: 0.1783 (0.13680983839691366) | > loader_time: 0.0052 (0.004628032616596731)  --> TIME: 2025-02-26 04:51:23 -- STEP: 22748/36938 -- GLOBAL_STEP: 1093950 | > loss_text_ce: 0.028794625774025917 (0.03210498745213714) | > loss_mel_ce: 2.9031355381011963 (nan) | > loss: 0.011634644120931625 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13680833795063566) | > loader_time: 0.0046 (0.0046282138643353714)  --> TIME: 2025-02-26 04:51:33 -- STEP: 22798/36938 -- GLOBAL_STEP: 1094000 | > loss_text_ce: 0.02680428884923458 (0.032105213219251076) | > loss_mel_ce: 2.755187749862671 (nan) | > loss: 0.01103965099900961 (nan) | > current_lr: 5e-06 | > step_time: 0.1123 (0.13680933350920257) | > loader_time: 0.0047 (0.004628220193479576)  --> TIME: 2025-02-26 04:51:42 -- STEP: 22848/36938 -- GLOBAL_STEP: 1094050 | > loss_text_ce: 0.0343988873064518 (0.03210245715041716) | > loss_mel_ce: 0.13922119140625 (nan) | > loss: 0.0006889685755595565 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13681361205479384) | > loader_time: 0.0047 (0.004628229750638581)  --> TIME: 2025-02-26 04:51:51 -- STEP: 22898/36938 -- GLOBAL_STEP: 1094100 | > loss_text_ce: 0.02874078042805195 (0.03210241422836443) | > loss_mel_ce: 2.6714282035827637 (nan) | > loss: 0.010714957490563393 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1368149406619029) | > loader_time: 0.0045 (0.004628262287436101)  --> TIME: 2025-02-26 04:52:01 -- STEP: 22948/36938 -- GLOBAL_STEP: 1094150 | > loss_text_ce: 0.02991410717368126 (0.03210413692113556) | > loss_mel_ce: 3.2766287326812744 (nan) | > loss: 0.01312120258808136 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13681600560123003) | > loader_time: 0.0047 (0.00462839301920361)  --> TIME: 2025-02-26 04:52:10 -- STEP: 22998/36938 -- GLOBAL_STEP: 1094200 | > loss_text_ce: 0.029938600957393646 (0.03210956452150205) | > loss_mel_ce: 2.904052495956421 (nan) | > loss: 0.011642822995781898 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.13680748019345781) | > loader_time: 0.0046 (0.004628533311010036)  --> TIME: 2025-02-26 04:52:19 -- STEP: 23048/36938 -- GLOBAL_STEP: 1094250 | > loss_text_ce: 0.031564053148031235 (0.03210702643389358) | > loss_mel_ce: 3.147792100906372 (nan) | > loss: 0.012616492807865143 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.1368098100081622) | > loader_time: 0.0047 (0.004628596600456723)  --> TIME: 2025-02-26 04:52:29 -- STEP: 23098/36938 -- GLOBAL_STEP: 1094300 | > loss_text_ce: 0.032374635338783264 (0.03210735047607527) | > loss_mel_ce: 2.175178050994873 (nan) | > loss: 0.008760130032896996 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13681064867768986) | > loader_time: 0.0047 (0.004628835709975056)  --> TIME: 2025-02-26 04:52:38 -- STEP: 23148/36938 -- GLOBAL_STEP: 1094350 | > loss_text_ce: 0.03233122453093529 (0.03210683967858623) | > loss_mel_ce: 2.9379119873046875 (nan) | > loss: 0.011786679737269878 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.13681367261640098) | > loader_time: 0.0046 (0.0046289740334690905)  --> TIME: 2025-02-26 04:52:48 -- STEP: 23198/36938 -- GLOBAL_STEP: 1094400 | > loss_text_ce: 0.03661083057522774 (0.032104901643642665) | > loss_mel_ce: 2.704313039779663 (nan) | > loss: 0.010876682586967945 (nan) | > current_lr: 5e-06 | > step_time: 0.1893 (0.13681207730735404) | > loader_time: 0.0048 (0.00462915577943165)  --> TIME: 2025-02-26 04:52:57 -- STEP: 23248/36938 -- GLOBAL_STEP: 1094450 | > loss_text_ce: 0.03544918820261955 (0.03210484501690226) | > loss_mel_ce: 2.6357812881469727 (nan) | > loss: 0.010600121691823006 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.136814894066215) | > loader_time: 0.0046 (0.00462919714651022)  --> TIME: 2025-02-26 04:53:06 -- STEP: 23298/36938 -- GLOBAL_STEP: 1094500 | > loss_text_ce: 0.027581432834267616 (0.03210554502720384) | > loss_mel_ce: 2.218595027923584 (nan) | > loss: 0.008913399651646614 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.13680994529070853) | > loader_time: 0.0067 (0.0046292329225450886)  --> TIME: 2025-02-26 04:53:16 -- STEP: 23348/36938 -- GLOBAL_STEP: 1094550 | > loss_text_ce: 0.029294878244400024 (0.03210756242595476) | > loss_mel_ce: 3.334534168243408 (nan) | > loss: 0.013348529115319252 (nan) | > current_lr: 5e-06 | > step_time: 0.1769 (0.13681075688153524) | > loader_time: 0.0047 (0.004629134876547924)  --> TIME: 2025-02-26 04:53:25 -- STEP: 23398/36938 -- GLOBAL_STEP: 1094600 | > loss_text_ce: 0.030693501234054565 (0.032107596147722114) | > loss_mel_ce: 2.6981327533721924 (nan) | > loss: 0.010828676633536816 (nan) | > current_lr: 5e-06 | > step_time: 0.1881 (0.13680627446101717) | > loader_time: 0.0048 (0.004629053206655325)  --> TIME: 2025-02-26 04:53:34 -- STEP: 23448/36938 -- GLOBAL_STEP: 1094650 | > loss_text_ce: 0.03810669854283333 (0.032109085133187144) | > loss_mel_ce: 2.9204795360565186 (nan) | > loss: 0.011740421876311302 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13680742394798134) | > loader_time: 0.0045 (0.004629071876889651)  --> TIME: 2025-02-26 04:53:43 -- STEP: 23498/36938 -- GLOBAL_STEP: 1094700 | > loss_text_ce: 0.031073110178112984 (0.03211037147071553) | > loss_mel_ce: 2.1976306438446045 (nan) | > loss: 0.008844062685966492 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.136807565740225) | > loader_time: 0.0048 (0.004629129571644376)  --> TIME: 2025-02-26 04:53:52 -- STEP: 23548/36938 -- GLOBAL_STEP: 1094750 | > loss_text_ce: 0.04034712538123131 (0.03210862749584461) | > loss_mel_ce: 2.5681803226470947 (nan) | > loss: 0.010351300239562988 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13679236527997507) | > loader_time: 0.0045 (0.004629016135166998)  --> TIME: 2025-02-26 04:54:02 -- STEP: 23598/36938 -- GLOBAL_STEP: 1094800 | > loss_text_ce: 0.035739004611968994 (0.032106917543802946) | > loss_mel_ce: 2.717223882675171 (nan) | > loss: 0.010924456641077995 (nan) | > current_lr: 5e-06 | > step_time: 0.1533 (0.1368024574921231) | > loader_time: 0.005 (0.004629200621675318)  --> TIME: 2025-02-26 04:54:11 -- STEP: 23648/36938 -- GLOBAL_STEP: 1094850 | > loss_text_ce: 0.03450804948806763 (0.0321070668324989) | > loss_mel_ce: 3.146453619003296 (nan) | > loss: 0.012622863985598087 (nan) | > current_lr: 5e-06 | > step_time: 0.1897 (0.13679736886924657) | > loader_time: 0.005 (0.004629338172759674)  --> TIME: 2025-02-26 04:54:21 -- STEP: 23698/36938 -- GLOBAL_STEP: 1094900 | > loss_text_ce: 0.02751404047012329 (0.03210606486030777) | > loss_mel_ce: 2.7160181999206543 (nan) | > loss: 0.010887033306062222 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13680403055908538) | > loader_time: 0.0046 (0.004629330017745962)  --> TIME: 2025-02-26 04:54:30 -- STEP: 23748/36938 -- GLOBAL_STEP: 1094950 | > loss_text_ce: 0.036321837455034256 (0.03210549645124457) | > loss_mel_ce: 2.8647937774658203 (nan) | > loss: 0.011512364260852337 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13680714709160055) | > loader_time: 0.0045 (0.004629263236142378)  --> TIME: 2025-02-26 04:54:39 -- STEP: 23798/36938 -- GLOBAL_STEP: 1095000 | > loss_text_ce: 0.027682630345225334 (0.032104376978498814) | > loss_mel_ce: 3.3000171184539795 (nan) | > loss: 0.013205158524215221 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.13679290217545437) | > loader_time: 0.0048 (0.004629235005557049)  --> TIME: 2025-02-26 04:54:48 -- STEP: 23848/36938 -- GLOBAL_STEP: 1095050 | > loss_text_ce: 0.023917606100440025 (0.03210339137456569) | > loss_mel_ce: 1.1014404296875 (nan) | > loss: 0.004465706646442413 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.1367870901043649) | > loader_time: 0.0046 (0.00462914586906983)  --> TIME: 2025-02-26 04:54:57 -- STEP: 23898/36938 -- GLOBAL_STEP: 1095100 | > loss_text_ce: 0.02910573221743107 (0.032104540254196796) | > loss_mel_ce: 3.444000244140625 (nan) | > loss: 0.013782166875898838 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13677376031336935) | > loader_time: 0.0045 (0.004629051758161756)  --> TIME: 2025-02-26 04:55:06 -- STEP: 23948/36938 -- GLOBAL_STEP: 1095150 | > loss_text_ce: 0.032025087624788284 (0.032105265925340506) | > loss_mel_ce: 2.7623257637023926 (nan) | > loss: 0.01108869444578886 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13676731364842964) | > loader_time: 0.0048 (0.004629060464250201)  --> TIME: 2025-02-26 04:55:16 -- STEP: 23998/36938 -- GLOBAL_STEP: 1095200 | > loss_text_ce: 0.03514379262924194 (0.03210895096992464) | > loss_mel_ce: 3.5199737548828125 (nan) | > loss: 0.01410761009901762 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.136764261213698) | > loader_time: 0.0047 (0.004629100280082645)  --> TIME: 2025-02-26 04:55:25 -- STEP: 24048/36938 -- GLOBAL_STEP: 1095250 | > loss_text_ce: 0.02658325806260109 (0.03211080215997439) | > loss_mel_ce: 2.4131107330322266 (nan) | > loss: 0.009681325405836105 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.1367627219485991) | > loader_time: 0.0045 (0.0046292671503738335)  --> TIME: 2025-02-26 04:55:34 -- STEP: 24098/36938 -- GLOBAL_STEP: 1095300 | > loss_text_ce: 0.031823303550481796 (0.032108552671374746) | > loss_mel_ce: 1.362595796585083 (nan) | > loss: 0.005533409304916859 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13675876612168253) | > loader_time: 0.0046 (0.004629228211842252)  --> TIME: 2025-02-26 04:55:44 -- STEP: 24148/36938 -- GLOBAL_STEP: 1095350 | > loss_text_ce: 0.03548416495323181 (0.03210850861306119) | > loss_mel_ce: 2.111466407775879 (nan) | > loss: 0.008519644849002361 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13676469195100413) | > loader_time: 0.0045 (0.004629229421111668)  --> TIME: 2025-02-26 04:55:53 -- STEP: 24198/36938 -- GLOBAL_STEP: 1095400 | > loss_text_ce: 0.02935939095914364 (0.032104099094399714) | > loss_mel_ce: 3.4134109020233154 (nan) | > loss: 0.01366178784519434 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.13675916921068784) | > loader_time: 0.0047 (0.00462930382199598)  --> TIME: 2025-02-26 04:56:02 -- STEP: 24248/36938 -- GLOBAL_STEP: 1095450 | > loss_text_ce: 0.04058179631829262 (0.032105634813293725) | > loss_mel_ce: 3.4400291442871094 (nan) | > loss: 0.013811948709189892 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13674727870819445) | > loader_time: 0.0046 (0.004629222591967121)  --> TIME: 2025-02-26 04:56:11 -- STEP: 24298/36938 -- GLOBAL_STEP: 1095500 | > loss_text_ce: 0.030403880402445793 (0.03210444666745142) | > loss_mel_ce: 3.168917179107666 (nan) | > loss: 0.012695719487965107 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13673904377969312) | > loader_time: 0.0045 (0.004629074452747398)  --> TIME: 2025-02-26 04:56:20 -- STEP: 24348/36938 -- GLOBAL_STEP: 1095550 | > loss_text_ce: 0.0272662490606308 (0.03210575871704458) | > loss_mel_ce: 1.979958415031433 (nan) | > loss: 0.007965177297592163 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.1367459724018765) | > loader_time: 0.0046 (0.004629177864655019)  --> TIME: 2025-02-26 04:56:30 -- STEP: 24398/36938 -- GLOBAL_STEP: 1095600 | > loss_text_ce: 0.042298272252082825 (0.03210499940834141) | > loss_mel_ce: 3.3305294513702393 (nan) | > loss: 0.013384237885475159 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.13674226924331218) | > loader_time: 0.0048 (0.004629361325575191)  --> TIME: 2025-02-26 04:56:39 -- STEP: 24448/36938 -- GLOBAL_STEP: 1095650 | > loss_text_ce: 0.02578563243150711 (0.03210410641424636) | > loss_mel_ce: 2.448216199874878 (nan) | > loss: 0.009817468002438545 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.1367449308721186) | > loader_time: 0.0046 (0.004629357147435238)  --> TIME: 2025-02-26 04:56:48 -- STEP: 24498/36938 -- GLOBAL_STEP: 1095700 | > loss_text_ce: 0.020985359326004982 (0.032103179843973206) | > loss_mel_ce: 1.0278308391571045 (nan) | > loss: 0.004161969292908907 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13674333225183014) | > loader_time: 0.0045 (0.004629461305347182)  --> TIME: 2025-02-26 04:56:58 -- STEP: 24548/36938 -- GLOBAL_STEP: 1095750 | > loss_text_ce: 0.03090418316423893 (0.032103393214417936) | > loss_mel_ce: 2.1749141216278076 (nan) | > loss: 0.008753248490393162 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13675262630034396) | > loader_time: 0.0045 (0.004629633452695333)  --> TIME: 2025-02-26 04:57:07 -- STEP: 24598/36938 -- GLOBAL_STEP: 1095800 | > loss_text_ce: 0.030415775254368782 (0.032101603340503174) | > loss_mel_ce: 2.714892625808716 (nan) | > loss: 0.010894081555306911 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1367467625159174) | > loader_time: 0.0045 (0.004629618899203261)  --> TIME: 2025-02-26 04:57:16 -- STEP: 24648/36938 -- GLOBAL_STEP: 1095850 | > loss_text_ce: 0.027601806446909904 (0.03210048948591299) | > loss_mel_ce: 2.247779607772827 (nan) | > loss: 0.009029291570186615 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.1367464385726012) | > loader_time: 0.0045 (0.004629661475091493)  --> TIME: 2025-02-26 04:57:26 -- STEP: 24698/36938 -- GLOBAL_STEP: 1095900 | > loss_text_ce: 0.03640005365014076 (0.03210206912474198) | > loss_mel_ce: 2.656282424926758 (nan) | > loss: 0.010685248300433159 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.13673900501845276) | > loader_time: 0.0047 (0.004629608667546472)  --> TIME: 2025-02-26 04:57:35 -- STEP: 24748/36938 -- GLOBAL_STEP: 1095950 | > loss_text_ce: 0.025280224159359932 (0.03210417331853873) | > loss_mel_ce: 2.9358866214752197 (nan) | > loss: 0.011750662699341774 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13673495332085314) | > loader_time: 0.0045 (0.00462965769125759)  --> TIME: 2025-02-26 04:57:44 -- STEP: 24798/36938 -- GLOBAL_STEP: 1096000 | > loss_text_ce: 0.026510674506425858 (0.03210413306635292) | > loss_mel_ce: 0.6346996426582336 (nan) | > loss: 0.002623850479722023 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1367351416859267) | > loader_time: 0.0046 (0.004629674856965964)  --> TIME: 2025-02-26 04:57:53 -- STEP: 24848/36938 -- GLOBAL_STEP: 1096050 | > loss_text_ce: 0.03321749344468117 (0.03210416976372206) | > loss_mel_ce: 0.0463663749396801 (nan) | > loss: 0.00031580901122651994 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13673929346237224) | > loader_time: 0.0046 (0.004629645244735917)  --> TIME: 2025-02-26 04:58:03 -- STEP: 24898/36938 -- GLOBAL_STEP: 1096100 | > loss_text_ce: 0.02799932472407818 (0.03210440699864079) | > loss_mel_ce: 2.6351568698883057 (nan) | > loss: 0.01056808140128851 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.1367434472365117) | > loader_time: 0.0047 (0.004629595460293401)  --> TIME: 2025-02-26 04:58:12 -- STEP: 24948/36938 -- GLOBAL_STEP: 1096150 | > loss_text_ce: 0.02617739699780941 (0.032103519575432615) | > loss_mel_ce: 1.597825527191162 (nan) | > loss: 0.006444456521421671 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.1367502065639153) | > loader_time: 0.0045 (0.004629575930976564)  --> TIME: 2025-02-26 04:58:22 -- STEP: 24998/36938 -- GLOBAL_STEP: 1096200 | > loss_text_ce: 0.025054985657334328 (0.03210463927585143) | > loss_mel_ce: 2.767568826675415 (nan) | > loss: 0.011081840842962265 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.1367590335457125) | > loader_time: 0.0046 (0.004629577586284389)  --> TIME: 2025-02-26 04:58:31 -- STEP: 25048/36938 -- GLOBAL_STEP: 1096250 | > loss_text_ce: 0.03098522312939167 (0.03210159707159313) | > loss_mel_ce: 2.852532386779785 (nan) | > loss: 0.011442530900239944 (nan) | > current_lr: 5e-06 | > step_time: 0.1771 (0.1367532476111751) | > loader_time: 0.0048 (0.004629625589921398)  --> TIME: 2025-02-26 04:58:41 -- STEP: 25098/36938 -- GLOBAL_STEP: 1096300 | > loss_text_ce: 0.03899594768881798 (0.03210060416225571) | > loss_mel_ce: 2.8146092891693115 (nan) | > loss: 0.011323831044137478 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.1367535115295283) | > loader_time: 0.0046 (0.0046297091394318195)  --> TIME: 2025-02-26 04:58:50 -- STEP: 25148/36938 -- GLOBAL_STEP: 1096350 | > loss_text_ce: 0.030324773862957954 (0.03209883023634987) | > loss_mel_ce: 1.7691694498062134 (nan) | > loss: 0.0071408506482839584 (nan) | > current_lr: 5e-06 | > step_time: 0.1789 (0.136755388236604) | > loader_time: 0.0049 (0.004629742080994251)  --> TIME: 2025-02-26 04:59:00 -- STEP: 25198/36938 -- GLOBAL_STEP: 1096400 | > loss_text_ce: 0.04094754904508591 (0.03210079415259804) | > loss_mel_ce: 2.8244974613189697 (nan) | > loss: 0.011370813474059105 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13676171616359734) | > loader_time: 0.0046 (0.004629816769777687)  --> TIME: 2025-02-26 04:59:09 -- STEP: 25248/36938 -- GLOBAL_STEP: 1096450 | > loss_text_ce: 0.04241592437028885 (0.03210174798858038) | > loss_mel_ce: 1.7338343858718872 (nan) | > loss: 0.007048612926155329 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.1367667634866394) | > loader_time: 0.0045 (0.004629889717951137)  --> TIME: 2025-02-26 04:59:19 -- STEP: 25298/36938 -- GLOBAL_STEP: 1096500 | > loss_text_ce: 0.040550537407398224 (0.03210347781928383) | > loss_mel_ce: 2.2335712909698486 (nan) | > loss: 0.009024293161928654 (nan) | > current_lr: 5e-06 | > step_time: 0.107 (0.13677232622091937) | > loader_time: 0.0046 (0.004629982706209088)  --> TIME: 2025-02-26 04:59:28 -- STEP: 25348/36938 -- GLOBAL_STEP: 1096550 | > loss_text_ce: 0.03676914423704147 (0.03210224175025633) | > loss_mel_ce: 2.7765684127807617 (nan) | > loss: 0.011164038442075253 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13678382025055535) | > loader_time: 0.0045 (0.004629977629428109)  --> TIME: 2025-02-26 04:59:37 -- STEP: 25398/36938 -- GLOBAL_STEP: 1096600 | > loss_text_ce: 0.03212151676416397 (0.03210180912416771) | > loss_mel_ce: 3.0542078018188477 (nan) | > loss: 0.012247338891029358 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13677962556280776) | > loader_time: 0.0051 (0.004630037964549643)  --> TIME: 2025-02-26 04:59:46 -- STEP: 25448/36938 -- GLOBAL_STEP: 1096650 | > loss_text_ce: 0.02613334357738495 (0.03210000920780492) | > loss_mel_ce: 1.8819385766983032 (nan) | > loss: 0.007571713998913765 (nan) | > current_lr: 5e-06 | > step_time: 0.1075 (0.13676898260605325) | > loader_time: 0.0045 (0.004630069065978372)  --> TIME: 2025-02-26 04:59:56 -- STEP: 25498/36938 -- GLOBAL_STEP: 1096700 | > loss_text_ce: 0.027529068291187286 (0.03209769757189418) | > loss_mel_ce: 3.028198480606079 (nan) | > loss: 0.012125903740525246 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.13677219810070004) | > loader_time: 0.0045 (0.004630228885716723)  --> TIME: 2025-02-26 05:00:05 -- STEP: 25548/36938 -- GLOBAL_STEP: 1096750 | > loss_text_ce: 0.04036026820540428 (0.03209714444871236) | > loss_mel_ce: 1.9070333242416382 (nan) | > loss: 0.007727752439677715 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1367652085030125) | > loader_time: 0.0045 (0.004630081284413823)  --> TIME: 2025-02-26 05:00:14 -- STEP: 25598/36938 -- GLOBAL_STEP: 1096800 | > loss_text_ce: 0.027715830132365227 (0.03209369742294876) | > loss_mel_ce: 1.4576306343078613 (nan) | > loss: 0.005894232075661421 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1367610110110709) | > loader_time: 0.0045 (0.004629927870351535)  --> TIME: 2025-02-26 05:00:23 -- STEP: 25648/36938 -- GLOBAL_STEP: 1096850 | > loss_text_ce: 0.03364318609237671 (0.03209337542483207) | > loss_mel_ce: 3.3621327877044678 (nan) | > loss: 0.01347530260682106 (nan) | > current_lr: 5e-06 | > step_time: 0.1503 (0.13675874860228787) | > loader_time: 0.0046 (0.004629824071174369)  --> TIME: 2025-02-26 05:00:33 -- STEP: 25698/36938 -- GLOBAL_STEP: 1096900 | > loss_text_ce: 0.03527231141924858 (0.03209427232437366) | > loss_mel_ce: 1.070526123046875 (nan) | > loss: 0.004388089291751385 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13676363421336366) | > loader_time: 0.0046 (0.004629897991500728)  --> TIME: 2025-02-26 05:00:43 -- STEP: 25748/36938 -- GLOBAL_STEP: 1096950 | > loss_text_ce: 0.03415194898843765 (0.032094028211688205) | > loss_mel_ce: 3.4453911781311035 (nan) | > loss: 0.013807712122797966 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.13677441564538984) | > loader_time: 0.005 (0.004629851517255774)  --> TIME: 2025-02-26 05:00:52 -- STEP: 25798/36938 -- GLOBAL_STEP: 1097000 | > loss_text_ce: 0.03217706456780434 (0.03209461434298453) | > loss_mel_ce: 2.2459750175476074 (nan) | > loss: 0.00904028583317995 (nan) | > current_lr: 5e-06 | > step_time: 0.1979 (0.1367744834804388) | > loader_time: 0.0047 (0.004629844001526303)  --> TIME: 2025-02-26 05:01:01 -- STEP: 25848/36938 -- GLOBAL_STEP: 1097050 | > loss_text_ce: 0.0291947890073061 (0.03209525744149825) | > loss_mel_ce: 3.0300424098968506 (nan) | > loss: 0.012139830738306046 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.13677427631087233) | > loader_time: 0.0048 (0.004629898812888145)  --> TIME: 2025-02-26 05:01:11 -- STEP: 25898/36938 -- GLOBAL_STEP: 1097100 | > loss_text_ce: 0.026300989091396332 (0.03209272974351777) | > loss_mel_ce: 2.6051738262176514 (nan) | > loss: 0.010442360304296017 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13678078817505665) | > loader_time: 0.0045 (0.004629922047555445)  --> TIME: 2025-02-26 05:01:20 -- STEP: 25948/36938 -- GLOBAL_STEP: 1097150 | > loss_text_ce: 0.03497655689716339 (0.03209037208302705) | > loss_mel_ce: 2.44392728805542 (nan) | > loss: 0.009836920537054539 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13677800962328548) | > loader_time: 0.0046 (0.004629897514476726)  --> TIME: 2025-02-26 05:01:29 -- STEP: 25998/36938 -- GLOBAL_STEP: 1097200 | > loss_text_ce: 0.02649618126451969 (0.0320923622393886) | > loss_mel_ce: 3.338657855987549 (nan) | > loss: 0.013353786431252956 (nan) | > current_lr: 5e-06 | > step_time: 0.1905 (0.13676879314085516) | > loader_time: 0.0048 (0.004629819996036517)  --> TIME: 2025-02-26 05:01:38 -- STEP: 26048/36938 -- GLOBAL_STEP: 1097250 | > loss_text_ce: 0.03560544550418854 (0.03208929020046208) | > loss_mel_ce: 3.0537819862365723 (nan) | > loss: 0.012259474955499172 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.13676552174278864) | > loader_time: 0.0046 (0.004629751800099498)  --> TIME: 2025-02-26 05:01:48 -- STEP: 26098/36938 -- GLOBAL_STEP: 1097300 | > loss_text_ce: 0.032505374401807785 (0.03208830612967478) | > loss_mel_ce: 2.6362273693084717 (nan) | > loss: 0.010590209625661373 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13677157071442755) | > loader_time: 0.0046 (0.004629751879352726)  --> TIME: 2025-02-26 05:01:57 -- STEP: 26148/36938 -- GLOBAL_STEP: 1097350 | > loss_text_ce: 0.027259845286607742 (0.03208704876813332) | > loss_mel_ce: 2.8821959495544434 (nan) | > loss: 0.011545459739863873 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13676504835689365) | > loader_time: 0.0045 (0.004629730275597446)  --> TIME: 2025-02-26 05:02:07 -- STEP: 26198/36938 -- GLOBAL_STEP: 1097400 | > loss_text_ce: 0.03366254270076752 (0.03208904004871853) | > loss_mel_ce: 2.8784239292144775 (nan) | > loss: 0.011555899865925312 (nan) | > current_lr: 5e-06 | > step_time: 0.1563 (0.13676791021283205) | > loader_time: 0.0047 (0.004629741871536221)  --> TIME: 2025-02-26 05:02:16 -- STEP: 26248/36938 -- GLOBAL_STEP: 1097450 | > loss_text_ce: 0.032288409769535065 (0.03209091978589305) | > loss_mel_ce: 3.318915367126465 (nan) | > loss: 0.013298427686095238 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13677465575056086) | > loader_time: 0.0045 (0.004629705935780185)  --> TIME: 2025-02-26 05:02:26 -- STEP: 26298/36938 -- GLOBAL_STEP: 1097500 | > loss_text_ce: 0.03188707306981087 (0.03209267156515227) | > loss_mel_ce: 2.2768161296844482 (nan) | > loss: 0.00916152074933052 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13678083301760105) | > loader_time: 0.0045 (0.004629635504420057)  --> TIME: 2025-02-26 05:02:35 -- STEP: 26348/36938 -- GLOBAL_STEP: 1097550 | > loss_text_ce: 0.034951791167259216 (0.032091284441434095) | > loss_mel_ce: 2.714867115020752 (nan) | > loss: 0.010911979712545872 (nan) | > current_lr: 5e-06 | > step_time: 0.163 (0.13678061348617074) | > loader_time: 0.0047 (0.004629545577726661)  --> TIME: 2025-02-26 05:02:44 -- STEP: 26398/36938 -- GLOBAL_STEP: 1097600 | > loss_text_ce: 0.0337904691696167 (0.03209312068389767) | > loss_mel_ce: 2.8073465824127197 (nan) | > loss: 0.011274353601038456 (nan) | > current_lr: 5e-06 | > step_time: 0.1831 (0.13678675922934622) | > loader_time: 0.0048 (0.004629642812218892)  --> TIME: 2025-02-26 05:02:54 -- STEP: 26448/36938 -- GLOBAL_STEP: 1097650 | > loss_text_ce: 0.036803796887397766 (0.032095602749583785) | > loss_mel_ce: 0.7878092527389526 (nan) | > loss: 0.0032722740434110165 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13678955208764915) | > loader_time: 0.0046 (0.004629672691451348)  --> TIME: 2025-02-26 05:03:03 -- STEP: 26498/36938 -- GLOBAL_STEP: 1097700 | > loss_text_ce: 0.03150459751486778 (0.032095732888296424) | > loss_mel_ce: 2.9424033164978027 (nan) | > loss: 0.011801222339272499 (nan) | > current_lr: 5e-06 | > step_time: 0.1907 (0.13678254921199406) | > loader_time: 0.0048 (0.004629561141514561)  --> TIME: 2025-02-26 05:03:12 -- STEP: 26548/36938 -- GLOBAL_STEP: 1097750 | > loss_text_ce: 0.040485866367816925 (0.032093887815468895) | > loss_mel_ce: 2.7551307678222656 (nan) | > loss: 0.011093717068433762 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.1367834616770717) | > loader_time: 0.0045 (0.004629516091460956)  --> TIME: 2025-02-26 05:03:22 -- STEP: 26598/36938 -- GLOBAL_STEP: 1097800 | > loss_text_ce: 0.03573726490139961 (0.03209411127437321) | > loss_mel_ce: 3.0045859813690186 (nan) | > loss: 0.012064775452017784 (nan) | > current_lr: 5e-06 | > step_time: 0.1912 (0.13678617660815987) | > loader_time: 0.0048 (0.004629481321923274)  --> TIME: 2025-02-26 05:03:31 -- STEP: 26648/36938 -- GLOBAL_STEP: 1097850 | > loss_text_ce: 0.02465570904314518 (0.032094115309278345) | > loss_mel_ce: 1.8844341039657593 (nan) | > loss: 0.00757575361058116 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13677708987572787) | > loader_time: 0.0048 (0.0046293808153462035)  --> TIME: 2025-02-26 05:03:40 -- STEP: 26698/36938 -- GLOBAL_STEP: 1097900 | > loss_text_ce: 0.026669444516301155 (0.03209236693447206) | > loss_mel_ce: 1.8243359327316284 (nan) | > loss: 0.0073452601209282875 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13676950558981046) | > loader_time: 0.0045 (0.004629584321083384)  --> TIME: 2025-02-26 05:03:49 -- STEP: 26748/36938 -- GLOBAL_STEP: 1097950 | > loss_text_ce: 0.033038705587387085 (0.03209167551288962) | > loss_mel_ce: 2.2197611331939697 (nan) | > loss: 0.008939682506024837 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13676466401853102) | > loader_time: 0.0045 (0.004629541151153126)  --> TIME: 2025-02-26 05:03:58 -- STEP: 26798/36938 -- GLOBAL_STEP: 1098000 | > loss_text_ce: 0.04596381261944771 (0.03209360975769224) | > loss_mel_ce: 2.515061140060425 (nan) | > loss: 0.010162797756493092 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13675862248686474) | > loader_time: 0.0044 (0.004629512608642926)  --> TIME: 2025-02-26 05:04:08 -- STEP: 26848/36938 -- GLOBAL_STEP: 1098050 | > loss_text_ce: 0.03138548508286476 (0.03209480554639268) | > loss_mel_ce: 2.760725975036621 (nan) | > loss: 0.01107980776578188 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.13676252499382668) | > loader_time: 0.0048 (0.004629494571287939)  --> TIME: 2025-02-26 05:04:17 -- STEP: 26898/36938 -- GLOBAL_STEP: 1098100 | > loss_text_ce: 0.029085354879498482 (0.032094253152118585) | > loss_mel_ce: 2.8774614334106445 (nan) | > loss: 0.011533916927874088 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13675600511450423) | > loader_time: 0.0045 (0.004629405539880857)  --> TIME: 2025-02-26 05:04:26 -- STEP: 26948/36938 -- GLOBAL_STEP: 1098150 | > loss_text_ce: 0.031813882291316986 (0.03209450183681235) | > loss_mel_ce: 3.0400197505950928 (nan) | > loss: 0.012189816683530807 (nan) | > current_lr: 5e-06 | > step_time: 0.1573 (0.13675450869938707) | > loader_time: 0.0047 (0.004629323102784847)  --> TIME: 2025-02-26 05:04:36 -- STEP: 26998/36938 -- GLOBAL_STEP: 1098200 | > loss_text_ce: 0.04280979558825493 (0.032094230718968095) | > loss_mel_ce: 2.772824287414551 (nan) | > loss: 0.011173151433467865 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.136762285843648) | > loader_time: 0.0045 (0.004629381860359574)  --> TIME: 2025-02-26 05:04:45 -- STEP: 27048/36938 -- GLOBAL_STEP: 1098250 | > loss_text_ce: 0.035853613168001175 (0.032095406262032736) | > loss_mel_ce: 2.9232730865478516 (nan) | > loss: 0.011742566712200642 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.13676331024556437) | > loader_time: 0.0046 (0.004629319234558605)  --> TIME: 2025-02-26 05:04:54 -- STEP: 27098/36938 -- GLOBAL_STEP: 1098300 | > loss_text_ce: 0.03011535108089447 (0.03209236134280832) | > loss_mel_ce: 2.493079423904419 (nan) | > loss: 0.010012677870690823 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13676324897397088) | > loader_time: 0.0047 (0.004629255608092607)  --> TIME: 2025-02-26 05:05:03 -- STEP: 27148/36938 -- GLOBAL_STEP: 1098350 | > loss_text_ce: 0.041255392134189606 (0.03209209395188326) | > loss_mel_ce: 2.5195472240448 (nan) | > loss: 0.01016191579401493 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13676239848927388) | > loader_time: 0.0045 (0.004629139926905326)  --> TIME: 2025-02-26 05:05:13 -- STEP: 27198/36938 -- GLOBAL_STEP: 1098400 | > loss_text_ce: 0.037564657628536224 (0.032092734538725486) | > loss_mel_ce: 2.158442258834839 (nan) | > loss: 0.008714313618838787 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.1367713711929554) | > loader_time: 0.0046 (0.004629207898679993)  --> TIME: 2025-02-26 05:05:22 -- STEP: 27248/36938 -- GLOBAL_STEP: 1098450 | > loss_text_ce: 0.034283142536878586 (0.03209175185067448) | > loss_mel_ce: 3.0969808101654053 (nan) | > loss: 0.012425651773810387 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.1367753744912871) | > loader_time: 0.0047 (0.004629204335173488)  --> TIME: 2025-02-26 05:05:31 -- STEP: 27298/36938 -- GLOBAL_STEP: 1098500 | > loss_text_ce: 0.030396023765206337 (0.03209433196000204) | > loss_mel_ce: 0.004608154296875 (nan) | > loss: 0.0001389054668834433 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13676742921688603) | > loader_time: 0.0046 (0.004629281075658061)  --> TIME: 2025-02-26 05:05:41 -- STEP: 27348/36938 -- GLOBAL_STEP: 1098550 | > loss_text_ce: 0.04603090509772301 (0.032097107387468383) | > loss_mel_ce: 2.840167760848999 (nan) | > loss: 0.011453170329332352 (nan) | > current_lr: 5e-06 | > step_time: 0.1783 (0.13676592888531605) | > loader_time: 0.0048 (0.004629296091402467)  --> TIME: 2025-02-26 05:05:50 -- STEP: 27398/36938 -- GLOBAL_STEP: 1098600 | > loss_text_ce: 0.05476748198270798 (0.0320960016823537) | > loss_mel_ce: 0.045562744140625 (nan) | > loss: 0.00039813583134673536 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13676753780495424) | > loader_time: 0.0046 (0.004629370313254956)  --> TIME: 2025-02-26 05:05:59 -- STEP: 27448/36938 -- GLOBAL_STEP: 1098650 | > loss_text_ce: 0.034840233623981476 (0.032093773757260136) | > loss_mel_ce: 3.562389850616455 (nan) | > loss: 0.014274723827838898 (nan) | > current_lr: 5e-06 | > step_time: 0.19 (0.13676774636436614) | > loader_time: 0.005 (0.004629378353878814)  --> TIME: 2025-02-26 05:06:09 -- STEP: 27498/36938 -- GLOBAL_STEP: 1098700 | > loss_text_ce: 0.029589083045721054 (0.03209283200254983) | > loss_mel_ce: 2.391537666320801 (nan) | > loss: 0.009607646614313126 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.1367696831292082) | > loader_time: 0.0046 (0.004629292031341609)  --> TIME: 2025-02-26 05:06:18 -- STEP: 27548/36938 -- GLOBAL_STEP: 1098750 | > loss_text_ce: 0.033553361892700195 (0.0320923288462797) | > loss_mel_ce: 2.455888271331787 (nan) | > loss: 0.009878736920654774 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13676544486031125) | > loader_time: 0.0048 (0.004629216978957404)  --> TIME: 2025-02-26 05:06:28 -- STEP: 27598/36938 -- GLOBAL_STEP: 1098800 | > loss_text_ce: 0.03698328137397766 (0.03209390305148024) | > loss_mel_ce: 2.6292011737823486 (nan) | > loss: 0.010580097325146198 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.1367686797038441) | > loader_time: 0.0044 (0.004629135131835923)  --> TIME: 2025-02-26 05:06:37 -- STEP: 27648/36938 -- GLOBAL_STEP: 1098850 | > loss_text_ce: 0.038548823446035385 (0.03209310749820435) | > loss_mel_ce: 2.9337079524993896 (nan) | > loss: 0.011794669553637505 (nan) | > current_lr: 5e-06 | > step_time: 0.1556 (0.13676928233838292) | > loader_time: 0.0046 (0.004629102820116595)  --> TIME: 2025-02-26 05:06:46 -- STEP: 27698/36938 -- GLOBAL_STEP: 1098900 | > loss_text_ce: 0.03243878111243248 (0.032091980658508525) | > loss_mel_ce: 2.7512638568878174 (nan) | > loss: 0.011046439409255981 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13676015059692762) | > loader_time: 0.0045 (0.004629248212378689)  --> TIME: 2025-02-26 05:06:55 -- STEP: 27748/36938 -- GLOBAL_STEP: 1098950 | > loss_text_ce: 0.028678562492132187 (0.03209213831293347) | > loss_mel_ce: 3.183945417404175 (nan) | > loss: 0.012748508714139462 (nan) | > current_lr: 5e-06 | > step_time: 0.1708 (0.13676012244574523) | > loader_time: 0.0049 (0.004629390640459486)  --> TIME: 2025-02-26 05:07:04 -- STEP: 27798/36938 -- GLOBAL_STEP: 1099000 | > loss_text_ce: 0.032584138214588165 (0.03209163395969902) | > loss_mel_ce: 2.5302999019622803 (nan) | > loss: 0.010170175693929195 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.1367537305203958) | > loader_time: 0.0047 (0.004629442791122593)  --> TIME: 2025-02-26 05:07:13 -- STEP: 27848/36938 -- GLOBAL_STEP: 1099050 | > loss_text_ce: 0.024582620710134506 (0.03209096601462842) | > loss_mel_ce: 2.6826021671295166 (nan) | > loss: 0.010742797516286373 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.1367446759229291) | > loader_time: 0.0045 (0.0046293260088188645)  --> TIME: 2025-02-26 05:07:23 -- STEP: 27898/36938 -- GLOBAL_STEP: 1099100 | > loss_text_ce: 0.027820467948913574 (0.03208885676872406) | > loss_mel_ce: 0.0364605188369751 (nan) | > loss: 0.00025508328690193594 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13675084355245223) | > loader_time: 0.0045 (0.00462925154655685)  --> TIME: 2025-02-26 05:07:32 -- STEP: 27948/36938 -- GLOBAL_STEP: 1099150 | > loss_text_ce: 0.0347723588347435 (0.03208935975176564) | > loss_mel_ce: 2.838977813720703 (nan) | > loss: 0.011403771117329597 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.1367441490772415) | > loader_time: 0.0045 (0.004629220166772104)  --> TIME: 2025-02-26 05:07:41 -- STEP: 27998/36938 -- GLOBAL_STEP: 1099200 | > loss_text_ce: 0.023739656433463097 (0.03209069665425984) | > loss_mel_ce: 2.80859375 (nan) | > loss: 0.011239418759942055 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13674117767927568) | > loader_time: 0.0047 (0.004629241073886873)  --> TIME: 2025-02-26 05:07:51 -- STEP: 28048/36938 -- GLOBAL_STEP: 1099250 | > loss_text_ce: 0.03511783853173256 (0.03208854858975625) | > loss_mel_ce: 2.3811495304107666 (nan) | > loss: 0.009588363580405712 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13674237994350785) | > loader_time: 0.0045 (0.004629204333406529)  --> TIME: 2025-02-26 05:08:00 -- STEP: 28098/36938 -- GLOBAL_STEP: 1099300 | > loss_text_ce: 0.03714137151837349 (0.032088900567090406) | > loss_mel_ce: 2.501988172531128 (nan) | > loss: 0.010075910948216915 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.1367437766959172) | > loader_time: 0.0046 (0.0046291907526554085)  --> TIME: 2025-02-26 05:08:10 -- STEP: 28148/36938 -- GLOBAL_STEP: 1099350 | > loss_text_ce: 0.029325343668460846 (0.032091823417535345) | > loss_mel_ce: 0.2979736328125 (nan) | > loss: 0.0012988054659217596 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13675081188644622) | > loader_time: 0.0045 (0.0046292428894443936)  --> TIME: 2025-02-26 05:08:19 -- STEP: 28198/36938 -- GLOBAL_STEP: 1099400 | > loss_text_ce: 0.03478424623608589 (0.03208996930352312) | > loss_mel_ce: 2.7887015342712402 (nan) | > loss: 0.0112043097615242 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13674128199310306) | > loader_time: 0.0046 (0.004629204049838553)  --> TIME: 2025-02-26 05:08:28 -- STEP: 28248/36938 -- GLOBAL_STEP: 1099450 | > loss_text_ce: 0.03279608115553856 (0.032089505996304495) | > loss_mel_ce: 2.5213510990142822 (nan) | > loss: 0.010135505348443985 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.1367299262675854) | > loader_time: 0.0046 (0.0046291594902336565)  --> TIME: 2025-02-26 05:08:37 -- STEP: 28298/36938 -- GLOBAL_STEP: 1099500 | > loss_text_ce: 0.0298626646399498 (0.03209209945779539) | > loss_mel_ce: 3.1280829906463623 (nan) | > loss: 0.012531531043350697 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.13673124487589386) | > loader_time: 0.005 (0.004629198161357096)  --> TIME: 2025-02-26 05:08:47 -- STEP: 28348/36938 -- GLOBAL_STEP: 1099550 | > loss_text_ce: 0.03393641114234924 (0.03209276219800479) | > loss_mel_ce: 2.8253676891326904 (nan) | > loss: 0.011346445418894291 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.1367330499591598) | > loader_time: 0.0045 (0.004629121784499775)  --> TIME: 2025-02-26 05:08:56 -- STEP: 28398/36938 -- GLOBAL_STEP: 1099600 | > loss_text_ce: 0.026162447407841682 (0.03209311430021687) | > loss_mel_ce: 1.1394253969192505 (nan) | > loss: 0.004625349305570126 (nan) | > current_lr: 5e-06 | > step_time: 0.1116 (0.13672803715002682) | > loader_time: 0.0044 (0.00462911084333147)  --> TIME: 2025-02-26 05:09:05 -- STEP: 28448/36938 -- GLOBAL_STEP: 1099650 | > loss_text_ce: 0.029174242168664932 (0.032092935206691695) | > loss_mel_ce: 2.2904791831970215 (nan) | > loss: 0.00920497439801693 (nan) | > current_lr: 5e-06 | > step_time: 0.173 (0.1367254488178536) | > loader_time: 0.007 (0.00462914998271198)  --> TIME: 2025-02-26 05:09:14 -- STEP: 28498/36938 -- GLOBAL_STEP: 1099700 | > loss_text_ce: 0.03536796569824219 (0.0320921093665355) | > loss_mel_ce: 0.33197021484375 (nan) | > loss: 0.001457691309042275 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13672658785392358) | > loader_time: 0.0045 (0.004629074611097579)  --> TIME: 2025-02-26 05:09:24 -- STEP: 28548/36938 -- GLOBAL_STEP: 1099750 | > loss_text_ce: 0.035275235772132874 (0.032090465453416366) | > loss_mel_ce: 2.9932520389556885 (nan) | > loss: 0.012017966248095036 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.1367276043505742) | > loader_time: 0.0046 (0.0046291646543831074)  --> TIME: 2025-02-26 05:09:33 -- STEP: 28598/36938 -- GLOBAL_STEP: 1099800 | > loss_text_ce: 0.034303151071071625 (0.032090746897436) | > loss_mel_ce: 2.97938871383667 (nan) | > loss: 0.011959095485508442 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.1367281567542676) | > loader_time: 0.0046 (0.004629263578407296)  --> TIME: 2025-02-26 05:09:43 -- STEP: 28648/36938 -- GLOBAL_STEP: 1099850 | > loss_text_ce: 0.03257716819643974 (0.032090653569949314) | > loss_mel_ce: 2.6308274269104004 (nan) | > loss: 0.010569066740572453 (nan) | > current_lr: 5e-06 | > step_time: 0.1123 (0.13673891380525055) | > loader_time: 0.0046 (0.004629349165940127)  --> TIME: 2025-02-26 05:09:52 -- STEP: 28698/36938 -- GLOBAL_STEP: 1099900 | > loss_text_ce: 0.03967035934329033 (0.032088606023327665) | > loss_mel_ce: 2.669930934906006 (nan) | > loss: 0.010752387344837189 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13674491145019543) | > loader_time: 0.0045 (0.004629315594877777)  --> TIME: 2025-02-26 05:10:02 -- STEP: 28748/36938 -- GLOBAL_STEP: 1099950 | > loss_text_ce: 0.03290466219186783 (0.03208845077261726) | > loss_mel_ce: 3.406919240951538 (nan) | > loss: 0.013650095090270042 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.13675020896992107) | > loader_time: 0.0046 (0.0046292752404860985)  --> TIME: 2025-02-26 05:10:11 -- STEP: 28798/36938 -- GLOBAL_STEP: 1100000 | > loss_text_ce: 0.02206246554851532 (0.03209109595313214) | > loss_mel_ce: 0.01747894287109375 (nan) | > loss: 0.00015691036242060363 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13675007291763824) | > loader_time: 0.0045 (0.004629209568305283) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1100000.pth  --> TIME: 2025-02-26 05:10:27 -- STEP: 28848/36938 -- GLOBAL_STEP: 1100050 | > loss_text_ce: 0.03247446566820145 (0.032091279902217204) | > loss_mel_ce: 2.6623194217681885 (nan) | > loss: 0.010693627409636974 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.13674546298026044) | > loader_time: 0.0046 (0.00462917908323649)  --> TIME: 2025-02-26 05:10:36 -- STEP: 28898/36938 -- GLOBAL_STEP: 1100100 | > loss_text_ce: 0.03808525204658508 (0.03209168493456489) | > loss_mel_ce: 2.8193564414978027 (nan) | > loss: 0.011339054442942142 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.13674551781935523) | > loader_time: 0.005 (0.004629123441091696)  --> TIME: 2025-02-26 05:10:45 -- STEP: 28948/36938 -- GLOBAL_STEP: 1100150 | > loss_text_ce: 0.03829599916934967 (0.032091552518246046) | > loss_mel_ce: 2.6725640296936035 (nan) | > loss: 0.010757382027804852 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13674598491635018) | > loader_time: 0.0066 (0.004629168916311928)  --> TIME: 2025-02-26 05:10:54 -- STEP: 28998/36938 -- GLOBAL_STEP: 1100200 | > loss_text_ce: 0.02177368849515915 (0.03209163853391189) | > loss_mel_ce: 1.5581104755401611 (nan) | > loss: 0.00626938184723258 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1367465122805542) | > loader_time: 0.0045 (0.004629228006388287)  --> TIME: 2025-02-26 05:11:04 -- STEP: 29048/36938 -- GLOBAL_STEP: 1100250 | > loss_text_ce: 0.034693919122219086 (0.032091225107396315) | > loss_mel_ce: 3.130556583404541 (nan) | > loss: 0.012560518458485603 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.1367418015177214) | > loader_time: 0.0046 (0.004629188728411296)  --> TIME: 2025-02-26 05:11:13 -- STEP: 29098/36938 -- GLOBAL_STEP: 1100300 | > loss_text_ce: 0.03124845400452614 (0.032091425156590844) | > loss_mel_ce: 2.35302734375 (nan) | > loss: 0.009461413137614727 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13674468554633748) | > loader_time: 0.0045 (0.0046291589835262)  --> TIME: 2025-02-26 05:11:22 -- STEP: 29148/36938 -- GLOBAL_STEP: 1100350 | > loss_text_ce: 0.03968432545661926 (0.0320924627460772) | > loss_mel_ce: 1.7460702657699585 (nan) | > loss: 0.0070863282307982445 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13673847241380133) | > loader_time: 0.0045 (0.0046291013419652504)  --> TIME: 2025-02-26 05:11:31 -- STEP: 29198/36938 -- GLOBAL_STEP: 1100400 | > loss_text_ce: 0.033060699701309204 (0.03209080501702011) | > loss_mel_ce: 3.3330078125 (nan) | > loss: 0.013357415795326233 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.136736597522611) | > loader_time: 0.0046 (0.004629042697480299)  --> TIME: 2025-02-26 05:11:41 -- STEP: 29248/36938 -- GLOBAL_STEP: 1100450 | > loss_text_ce: 0.03025306947529316 (0.032090192303323536) | > loss_mel_ce: 3.2739577293395996 (nan) | > loss: 0.01311194896697998 (nan) | > current_lr: 5e-06 | > step_time: 0.15 (0.13673675684686365) | > loader_time: 0.0047 (0.00462915128019496)  --> TIME: 2025-02-26 05:11:50 -- STEP: 29298/36938 -- GLOBAL_STEP: 1100500 | > loss_text_ce: 0.036839596927165985 (0.03208990783508006) | > loss_mel_ce: 1.3070272207260132 (nan) | > loss: 0.0053328052163124084 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13672843200125867) | > loader_time: 0.0044 (0.004629235526743777)  --> TIME: 2025-02-26 05:11:59 -- STEP: 29348/36938 -- GLOBAL_STEP: 1100550 | > loss_text_ce: 0.02714339829981327 (0.03209051696896015) | > loss_mel_ce: 0.110809326171875 (nan) | > loss: 0.0005474315257743001 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13672030207667962) | > loader_time: 0.0047 (0.004629134059488488)  --> TIME: 2025-02-26 05:12:08 -- STEP: 29398/36938 -- GLOBAL_STEP: 1100600 | > loss_text_ce: 0.026357922703027725 (0.03208914768245137) | > loss_mel_ce: 1.5803298950195312 (nan) | > loss: 0.00637574540451169 (nan) | > current_lr: 5e-06 | > step_time: 0.1079 (0.1367167038888029) | > loader_time: 0.0044 (0.004629047908493576)  --> TIME: 2025-02-26 05:12:17 -- STEP: 29448/36938 -- GLOBAL_STEP: 1100650 | > loss_text_ce: 0.026784280315041542 (0.03208794006617966) | > loss_mel_ce: 3.378995180130005 (nan) | > loss: 0.013514998368918896 (nan) | > current_lr: 5e-06 | > step_time: 0.1628 (0.1367148795438124) | > loader_time: 0.0046 (0.004629039547424589)  --> TIME: 2025-02-26 05:12:27 -- STEP: 29498/36938 -- GLOBAL_STEP: 1100700 | > loss_text_ce: 0.03326134383678436 (0.03208873690522031) | > loss_mel_ce: 2.9010753631591797 (nan) | > loss: 0.011644193902611732 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.13672034502336608) | > loader_time: 0.0047 (0.0046290789582267)  --> TIME: 2025-02-26 05:12:36 -- STEP: 29548/36938 -- GLOBAL_STEP: 1100750 | > loss_text_ce: 0.024161159992218018 (0.032086694442360765) | > loss_mel_ce: 2.94773268699646 (nan) | > loss: 0.011793229728937149 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13671006677243414) | > loader_time: 0.0045 (0.004629031842400727)  --> TIME: 2025-02-26 05:12:45 -- STEP: 29598/36938 -- GLOBAL_STEP: 1100800 | > loss_text_ce: 0.03214705362915993 (0.03208752066906779) | > loss_mel_ce: 2.3123300075531006 (nan) | > loss: 0.009303481318056583 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1367030600549666) | > loader_time: 0.0045 (0.004628975565864117)  --> TIME: 2025-02-26 05:12:54 -- STEP: 29648/36938 -- GLOBAL_STEP: 1100850 | > loss_text_ce: 0.03286609426140785 (0.03208694608568059) | > loss_mel_ce: 2.6615350246429443 (nan) | > loss: 0.010692068375647068 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13670497365533188) | > loader_time: 0.0045 (0.004629002452797314)  --> TIME: 2025-02-26 05:13:04 -- STEP: 29698/36938 -- GLOBAL_STEP: 1100900 | > loss_text_ce: 0.03273436054587364 (0.032088056174245726) | > loss_mel_ce: 3.1946821212768555 (nan) | > loss: 0.0128072090446949 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.136705638069972) | > loader_time: 0.0045 (0.004629045963704797)  --> TIME: 2025-02-26 05:13:13 -- STEP: 29748/36938 -- GLOBAL_STEP: 1100950 | > loss_text_ce: 0.03581613674759865 (0.03208990760770073) | > loss_mel_ce: 2.4635913372039795 (nan) | > loss: 0.009918284602463245 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13670840713948063) | > loader_time: 0.0045 (0.004629095651873548)  --> TIME: 2025-02-26 05:13:22 -- STEP: 29798/36938 -- GLOBAL_STEP: 1101000 | > loss_text_ce: 0.027268843725323677 (0.032088061611018175) | > loss_mel_ce: 2.7865242958068848 (nan) | > loss: 0.011165847070515156 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13670812285068928) | > loader_time: 0.0049 (0.004629143237011457)  --> TIME: 2025-02-26 05:13:32 -- STEP: 29848/36938 -- GLOBAL_STEP: 1101050 | > loss_text_ce: 0.029776109382510185 (0.032088174679839696) | > loss_mel_ce: 2.606793165206909 (nan) | > loss: 0.01046257745474577 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13670808733452236) | > loader_time: 0.0045 (0.004629113429099408)  --> TIME: 2025-02-26 05:13:41 -- STEP: 29898/36938 -- GLOBAL_STEP: 1101100 | > loss_text_ce: 0.02587495744228363 (0.03208638702050296) | > loss_mel_ce: 2.8491759300231934 (nan) | > loss: 0.011408932507038116 (nan) | > current_lr: 5e-06 | > step_time: 0.1816 (0.13670569797432752) | > loader_time: 0.0048 (0.004629050260308147)  --> TIME: 2025-02-26 05:13:50 -- STEP: 29948/36938 -- GLOBAL_STEP: 1101150 | > loss_text_ce: 0.0300894808024168 (0.03208518906699712) | > loss_mel_ce: 2.3992953300476074 (nan) | > loss: 0.009640416130423546 (nan) | > current_lr: 5e-06 | > step_time: 0.186 (0.1367119146202225) | > loader_time: 0.0048 (0.004629043643044981)  --> TIME: 2025-02-26 05:14:00 -- STEP: 29998/36938 -- GLOBAL_STEP: 1101200 | > loss_text_ce: 0.02613934874534607 (0.032085317700048896) | > loss_mel_ce: 2.4843008518218994 (nan) | > loss: 0.00996206421405077 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13671031421212806) | > loader_time: 0.0045 (0.004628991554543052)  --> TIME: 2025-02-26 05:14:09 -- STEP: 30048/36938 -- GLOBAL_STEP: 1101250 | > loss_text_ce: 0.03359917178750038 (0.03208572418870681) | > loss_mel_ce: 3.0912466049194336 (nan) | > loss: 0.012400181964039803 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13671422771181227) | > loader_time: 0.0045 (0.00462894978773351)  --> TIME: 2025-02-26 05:14:18 -- STEP: 30098/36938 -- GLOBAL_STEP: 1101300 | > loss_text_ce: 0.023066159337759018 (0.03208570016298752) | > loss_mel_ce: 2.8292484283447266 (nan) | > loss: 0.011318708769977093 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13671185296352953) | > loader_time: 0.0055 (0.004629068877167982)  --> TIME: 2025-02-26 05:14:28 -- STEP: 30148/36938 -- GLOBAL_STEP: 1101350 | > loss_text_ce: 0.03514169901609421 (0.03208457040151923) | > loss_mel_ce: 3.1991777420043945 (nan) | > loss: 0.012834602035582066 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13672212147026336) | > loader_time: 0.0046 (0.004629081553294279)  --> TIME: 2025-02-26 05:14:37 -- STEP: 30198/36938 -- GLOBAL_STEP: 1101400 | > loss_text_ce: 0.026630887761712074 (0.03208290477130259) | > loss_mel_ce: 2.1724154949188232 (nan) | > loss: 0.008726375177502632 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1367203990105208) | > loader_time: 0.0045 (0.0046290966270538065)  --> TIME: 2025-02-26 05:14:47 -- STEP: 30248/36938 -- GLOBAL_STEP: 1101450 | > loss_text_ce: 0.0453018881380558 (0.03208426453500876) | > loss_mel_ce: 2.6571223735809326 (nan) | > loss: 0.010723906569182873 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13672705783285374) | > loader_time: 0.0048 (0.0046290895416130455)  --> TIME: 2025-02-26 05:14:56 -- STEP: 30298/36938 -- GLOBAL_STEP: 1101500 | > loss_text_ce: 0.030576573684811592 (0.03208392944195646) | > loss_mel_ce: 2.578002691268921 (nan) | > loss: 0.010351505130529404 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.1367344752153764) | > loader_time: 0.0045 (0.004629130913987931)  --> TIME: 2025-02-26 05:15:06 -- STEP: 30348/36938 -- GLOBAL_STEP: 1101550 | > loss_text_ce: 0.02733999863266945 (0.03208269737363204) | > loss_mel_ce: 3.2934906482696533 (nan) | > loss: 0.013177899643778801 (nan) | > current_lr: 5e-06 | > step_time: 0.1952 (0.1367364560126385) | > loader_time: 0.0049 (0.004629153232416066)  --> TIME: 2025-02-26 05:15:16 -- STEP: 30398/36938 -- GLOBAL_STEP: 1101600 | > loss_text_ce: 0.027230659499764442 (0.03208232812945086) | > loss_mel_ce: 2.174351692199707 (nan) | > loss: 0.008736439049243927 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13674914893354884) | > loader_time: 0.0046 (0.0046292883101650825)  --> TIME: 2025-02-26 05:15:25 -- STEP: 30448/36938 -- GLOBAL_STEP: 1101650 | > loss_text_ce: 0.036040764302015305 (0.03208191493072544) | > loss_mel_ce: 0.42422252893447876 (nan) | > loss: 0.0018264417303726077 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13674706019386831) | > loader_time: 0.0045 (0.00462931676469722)  --> TIME: 2025-02-26 05:15:34 -- STEP: 30498/36938 -- GLOBAL_STEP: 1101700 | > loss_text_ce: 0.031353116035461426 (0.03208283785628844) | > loss_mel_ce: 3.1367979049682617 (nan) | > loss: 0.012572027742862701 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.13674852146665578) | > loader_time: 0.0047 (0.004629567135544352)  --> TIME: 2025-02-26 05:15:44 -- STEP: 30548/36938 -- GLOBAL_STEP: 1101750 | > loss_text_ce: 0.03215527907013893 (0.03208054540205326) | > loss_mel_ce: 1.855220079421997 (nan) | > loss: 0.007489585317671299 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13675034022759003) | > loader_time: 0.0046 (0.004629634181226549)  --> TIME: 2025-02-26 05:15:53 -- STEP: 30598/36938 -- GLOBAL_STEP: 1101800 | > loss_text_ce: 0.036218106746673584 (0.032080920117072156) | > loss_mel_ce: 0.8613077402114868 (nan) | > loss: 0.003561610821634531 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13674303925702105) | > loader_time: 0.0046 (0.0046296465107798996)  --> TIME: 2025-02-26 05:16:02 -- STEP: 30648/36938 -- GLOBAL_STEP: 1101850 | > loss_text_ce: 0.022154971957206726 (0.032083574379478216) | > loss_mel_ce: 2.5074756145477295 (nan) | > loss: 0.010038217529654503 (nan) | > current_lr: 5e-06 | > step_time: 0.1507 (0.13674837946736265) | > loader_time: 0.0046 (0.004629772781734504)  --> TIME: 2025-02-26 05:16:11 -- STEP: 30698/36938 -- GLOBAL_STEP: 1101900 | > loss_text_ce: 0.03979640081524849 (0.03208334321987162) | > loss_mel_ce: 2.392958641052246 (nan) | > loss: 0.009653789922595024 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13674081365979437) | > loader_time: 0.0047 (0.004629731954942021)  --> TIME: 2025-02-26 05:16:21 -- STEP: 30748/36938 -- GLOBAL_STEP: 1101950 | > loss_text_ce: 0.028312722221016884 (0.03208212159808828) | > loss_mel_ce: 2.707085132598877 (nan) | > loss: 0.010854753665626049 (nan) | > current_lr: 5e-06 | > step_time: 0.1711 (0.1367431822749879) | > loader_time: 0.0047 (0.004629720865524789)  --> TIME: 2025-02-26 05:16:30 -- STEP: 30798/36938 -- GLOBAL_STEP: 1102000 | > loss_text_ce: 0.034467875957489014 (0.03208200318688133) | > loss_mel_ce: 2.729384660720825 (nan) | > loss: 0.010967670008540154 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13674316203271458) | > loader_time: 0.0046 (0.004629742248436546)  --> TIME: 2025-02-26 05:16:40 -- STEP: 30848/36938 -- GLOBAL_STEP: 1102050 | > loss_text_ce: 0.029833512380719185 (0.032080827173701976) | > loss_mel_ce: 2.79142689704895 (nan) | > loss: 0.011195478960871696 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13674875865864372) | > loader_time: 0.0045 (0.004629848123029538)  --> TIME: 2025-02-26 05:16:49 -- STEP: 30898/36938 -- GLOBAL_STEP: 1102100 | > loss_text_ce: 0.03692615404725075 (0.032080820303585486) | > loss_mel_ce: 3.412668228149414 (nan) | > loss: 0.013688867911696434 (nan) | > current_lr: 5e-06 | > step_time: 0.1824 (0.13675237475139765) | > loader_time: 0.0049 (0.00463003177489729)  --> TIME: 2025-02-26 05:16:59 -- STEP: 30948/36938 -- GLOBAL_STEP: 1102150 | > loss_text_ce: 0.027981434017419815 (0.03207997244760946) | > loss_mel_ce: 2.8060355186462402 (nan) | > loss: 0.011246100068092346 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.13676367180283103) | > loader_time: 0.0047 (0.004630230788006174)  --> TIME: 2025-02-26 05:17:08 -- STEP: 30998/36938 -- GLOBAL_STEP: 1102200 | > loss_text_ce: 0.029646877199411392 (0.03208094080002127) | > loss_mel_ce: 3.3704833984375 (nan) | > loss: 0.013492581434547901 (nan) | > current_lr: 5e-06 | > step_time: 0.1771 (0.13675767679015796) | > loader_time: 0.0048 (0.004630404800374658)  --> TIME: 2025-02-26 05:17:17 -- STEP: 31048/36938 -- GLOBAL_STEP: 1102250 | > loss_text_ce: 0.029745468869805336 (0.032078345094984195) | > loss_mel_ce: 0.0460052490234375 (nan) | > loss: 0.00030059809796512127 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.1367588270512238) | > loader_time: 0.0045 (0.004630444383719491)  --> TIME: 2025-02-26 05:17:27 -- STEP: 31098/36938 -- GLOBAL_STEP: 1102300 | > loss_text_ce: 0.04697577282786369 (0.03207782058210451) | > loss_mel_ce: 3.5273139476776123 (nan) | > loss: 0.014183690771460533 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13675791769213616) | > loader_time: 0.0047 (0.004630493407801742)  --> TIME: 2025-02-26 05:17:36 -- STEP: 31148/36938 -- GLOBAL_STEP: 1102350 | > loss_text_ce: 0.030671361833810806 (0.032078471191261324) | > loss_mel_ce: 1.0077003240585327 (nan) | > loss: 0.004120522644370794 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1367614859256087) | > loader_time: 0.0047 (0.0046307395278251525)  --> TIME: 2025-02-26 05:17:46 -- STEP: 31198/36938 -- GLOBAL_STEP: 1102400 | > loss_text_ce: 0.0394384041428566 (0.03208024120231117) | > loss_mel_ce: 3.6137473583221436 (nan) | > loss: 0.01449676975607872 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13676382382028143) | > loader_time: 0.0047 (0.00463083906275803)  --> TIME: 2025-02-26 05:17:55 -- STEP: 31248/36938 -- GLOBAL_STEP: 1102450 | > loss_text_ce: 0.02522055059671402 (0.03208164354261471) | > loss_mel_ce: 0.16147613525390625 (nan) | > loss: 0.0007408598903566599 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13676136481078358) | > loader_time: 0.0047 (0.004630990444667739)  --> TIME: 2025-02-26 05:18:04 -- STEP: 31298/36938 -- GLOBAL_STEP: 1102500 | > loss_text_ce: 0.03480316326022148 (0.03208247444450345) | > loss_mel_ce: 3.118946075439453 (nan) | > loss: 0.012514878995716572 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.1367621878190004) | > loader_time: 0.0046 (0.004631000674565414)  --> TIME: 2025-02-26 05:18:14 -- STEP: 31348/36938 -- GLOBAL_STEP: 1102550 | > loss_text_ce: 0.03163794428110123 (0.03208129064576735) | > loss_mel_ce: 2.9011590480804443 (nan) | > loss: 0.01163808349519968 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.136763900385321) | > loader_time: 0.0047 (0.004631020051720815)  --> TIME: 2025-02-26 05:18:23 -- STEP: 31398/36938 -- GLOBAL_STEP: 1102600 | > loss_text_ce: 0.03191910311579704 (0.03207951199777769) | > loss_mel_ce: 3.168640613555908 (nan) | > loss: 0.012700634077191353 (nan) | > current_lr: 5e-06 | > step_time: 0.1565 (0.13676940677985988) | > loader_time: 0.0047 (0.00463109971975943)  --> TIME: 2025-02-26 05:18:32 -- STEP: 31448/36938 -- GLOBAL_STEP: 1102650 | > loss_text_ce: 0.034050580114126205 (0.032080759808501016) | > loss_mel_ce: 2.3003790378570557 (nan) | > loss: 0.009263609535992146 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13677142087026217) | > loader_time: 0.0046 (0.004631238102276052)  --> TIME: 2025-02-26 05:18:42 -- STEP: 31498/36938 -- GLOBAL_STEP: 1102700 | > loss_text_ce: 0.033988285809755325 (0.032080097330126144) | > loss_mel_ce: 1.9643938541412354 (nan) | > loss: 0.007930087856948376 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.1367750387659866) | > loader_time: 0.0046 (0.004631387263193925)  --> TIME: 2025-02-26 05:18:52 -- STEP: 31548/36938 -- GLOBAL_STEP: 1102750 | > loss_text_ce: 0.025953909382224083 (0.032078586521185004) | > loss_mel_ce: 2.1070396900177 (nan) | > loss: 0.008464260958135128 (nan) | > current_lr: 5e-06 | > step_time: 0.1435 (0.1367794307855903) | > loader_time: 0.0046 (0.004631409963092906)  --> TIME: 2025-02-26 05:19:01 -- STEP: 31598/36938 -- GLOBAL_STEP: 1102800 | > loss_text_ce: 0.035050202161073685 (0.03207671008168209) | > loss_mel_ce: 2.5372631549835205 (nan) | > loss: 0.010207593441009521 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.1367741725483885) | > loader_time: 0.005 (0.0046315342800942565)  --> TIME: 2025-02-26 05:19:10 -- STEP: 31648/36938 -- GLOBAL_STEP: 1102850 | > loss_text_ce: 0.02914223074913025 (0.03207624416909688) | > loss_mel_ce: 3.398413896560669 (nan) | > loss: 0.013601413927972317 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13678336893071844) | > loader_time: 0.0047 (0.0046316753428794575)  --> TIME: 2025-02-26 05:19:20 -- STEP: 31698/36938 -- GLOBAL_STEP: 1102900 | > loss_text_ce: 0.03533463180065155 (0.03207572133815229) | > loss_mel_ce: 2.893918514251709 (nan) | > loss: 0.011624020524322987 (nan) | > current_lr: 5e-06 | > step_time: 0.1757 (0.13679006225461565) | > loader_time: 0.0048 (0.004631728770657798)  --> TIME: 2025-02-26 05:19:29 -- STEP: 31748/36938 -- GLOBAL_STEP: 1102950 | > loss_text_ce: 0.02791154570877552 (0.03207512542609372) | > loss_mel_ce: 2.39572811126709 (nan) | > loss: 0.009617618285119534 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13678184560553772) | > loader_time: 0.0045 (0.0046316764359812254)  --> TIME: 2025-02-26 05:19:38 -- STEP: 31798/36938 -- GLOBAL_STEP: 1103000 | > loss_text_ce: 0.027008580043911934 (0.032075145856730033) | > loss_mel_ce: 2.810208320617676 (nan) | > loss: 0.011258797720074654 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13678489090384327) | > loader_time: 0.0046 (0.004631712748739072)  --> TIME: 2025-02-26 05:19:48 -- STEP: 31848/36938 -- GLOBAL_STEP: 1103050 | > loss_text_ce: 0.0411803275346756 (0.032076227168008044) | > loss_mel_ce: 2.2411298751831055 (nan) | > loss: 0.009056787006556988 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13678637488317805) | > loader_time: 0.0045 (0.00463171764792762)  --> TIME: 2025-02-26 05:19:57 -- STEP: 31898/36938 -- GLOBAL_STEP: 1103100 | > loss_text_ce: 0.024982614442706108 (0.03207514042267025) | > loss_mel_ce: 2.439626932144165 (nan) | > loss: 0.009780197404325008 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13678364326486203) | > loader_time: 0.0045 (0.0046317837546107615)  --> TIME: 2025-02-26 05:20:06 -- STEP: 31948/36938 -- GLOBAL_STEP: 1103150 | > loss_text_ce: 0.035390812903642654 (0.032074133190451645) | > loss_mel_ce: 1.77272367477417 (nan) | > loss: 0.007175058126449585 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13678385601095108) | > loader_time: 0.0045 (0.004631875818626757)  --> TIME: 2025-02-26 05:20:16 -- STEP: 31998/36938 -- GLOBAL_STEP: 1103200 | > loss_text_ce: 0.022398466244339943 (0.03207362305405076) | > loss_mel_ce: 1.3414506912231445 (nan) | > loss: 0.005412100348621607 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13678584031607544) | > loader_time: 0.0045 (0.004631925965929114)  --> TIME: 2025-02-26 05:20:25 -- STEP: 32048/36938 -- GLOBAL_STEP: 1103250 | > loss_text_ce: 0.03798266872763634 (0.03207374122001907) | > loss_mel_ce: 0.08740234375 (nan) | > loss: 0.0004975596093572676 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13678927425735962) | > loader_time: 0.0048 (0.004632004516694867)  --> TIME: 2025-02-26 05:20:35 -- STEP: 32098/36938 -- GLOBAL_STEP: 1103300 | > loss_text_ce: 0.029763996601104736 (0.03207555100566298) | > loss_mel_ce: 2.040609836578369 (nan) | > loss: 0.008215769194066525 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.1367968893235364) | > loader_time: 0.0047 (0.004632140098758111)  --> TIME: 2025-02-26 05:20:44 -- STEP: 32148/36938 -- GLOBAL_STEP: 1103350 | > loss_text_ce: 0.03157905861735344 (0.032074440332686645) | > loss_mel_ce: 2.44599986076355 (nan) | > loss: 0.009831663221120834 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.1367871194273236) | > loader_time: 0.0045 (0.004632250874348342)  --> TIME: 2025-02-26 05:20:53 -- STEP: 32198/36938 -- GLOBAL_STEP: 1103400 | > loss_text_ce: 0.02743684872984886 (0.03207304375188308) | > loss_mel_ce: 2.506474256515503 (nan) | > loss: 0.01005520299077034 (nan) | > current_lr: 5e-06 | > step_time: 0.1606 (0.1367829544395094) | > loader_time: 0.0047 (0.0046323578626785485)  --> TIME: 2025-02-26 05:21:02 -- STEP: 32248/36938 -- GLOBAL_STEP: 1103450 | > loss_text_ce: 0.035031989216804504 (0.032070443211090906) | > loss_mel_ce: 2.943058729171753 (nan) | > loss: 0.01181782130151987 (nan) | > current_lr: 5e-06 | > step_time: 0.1477 (0.13677917288894073) | > loader_time: 0.0046 (0.004632318095278411)  --> TIME: 2025-02-26 05:21:11 -- STEP: 32298/36938 -- GLOBAL_STEP: 1103500 | > loss_text_ce: 0.029072174802422523 (0.032069319955975915) | > loss_mel_ce: 2.16682505607605 (nan) | > loss: 0.008713878691196442 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13677517806304856) | > loader_time: 0.0046 (0.004632333999329996)  --> TIME: 2025-02-26 05:21:21 -- STEP: 32348/36938 -- GLOBAL_STEP: 1103550 | > loss_text_ce: 0.03460582345724106 (0.032072261598609916) | > loss_mel_ce: 2.65649151802063 (nan) | > loss: 0.010678958147764206 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1367815260437851) | > loader_time: 0.0045 (0.004632486369268714)  --> TIME: 2025-02-26 05:21:30 -- STEP: 32398/36938 -- GLOBAL_STEP: 1103600 | > loss_text_ce: 0.02705913409590721 (0.032072948774001846) | > loss_mel_ce: 3.150456428527832 (nan) | > loss: 0.012609189376235008 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13678802226132764) | > loader_time: 0.0046 (0.004632504937589738)  --> TIME: 2025-02-26 05:21:40 -- STEP: 32448/36938 -- GLOBAL_STEP: 1103650 | > loss_text_ce: 0.03284335508942604 (0.032071048192845335) | > loss_mel_ce: 2.221468210220337 (nan) | > loss: 0.008945681154727936 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.1367907166716151) | > loader_time: 0.0046 (0.0046325221922270485)  --> TIME: 2025-02-26 05:21:49 -- STEP: 32498/36938 -- GLOBAL_STEP: 1103700 | > loss_text_ce: 0.03530820831656456 (0.03207030094443208) | > loss_mel_ce: 2.8495171070098877 (nan) | > loss: 0.011447720229625702 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13679171155816588) | > loader_time: 0.0047 (0.004632568878792653)  --> TIME: 2025-02-26 05:21:58 -- STEP: 32548/36938 -- GLOBAL_STEP: 1103750 | > loss_text_ce: 0.033138204365968704 (0.03207204349150033) | > loss_mel_ce: 2.6082355976104736 (nan) | > loss: 0.010481642559170723 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.13679083579293075) | > loader_time: 0.0046 (0.00463263512653932)  --> TIME: 2025-02-26 05:22:08 -- STEP: 32598/36938 -- GLOBAL_STEP: 1103800 | > loss_text_ce: 0.03520403802394867 (0.032072715572933876) | > loss_mel_ce: 2.00714111328125 (nan) | > loss: 0.008104544132947922 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13678745879257537) | > loader_time: 0.0045 (0.004632638395838257)  --> TIME: 2025-02-26 05:22:17 -- STEP: 32648/36938 -- GLOBAL_STEP: 1103850 | > loss_text_ce: 0.03217253461480141 (0.032071245123387514) | > loss_mel_ce: 2.852541208267212 (nan) | > loss: 0.011447276920080185 (nan) | > current_lr: 5e-06 | > step_time: 0.1908 (0.13678707165976364) | > loader_time: 0.0049 (0.004632633293530879)  --> TIME: 2025-02-26 05:22:26 -- STEP: 32698/36938 -- GLOBAL_STEP: 1103900 | > loss_text_ce: 0.02307167649269104 (0.032070127360508294) | > loss_mel_ce: 2.5164921283721924 (nan) | > loss: 0.010077634826302528 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13678097191903477) | > loader_time: 0.0045 (0.004632631385936671)  --> TIME: 2025-02-26 05:22:35 -- STEP: 32748/36938 -- GLOBAL_STEP: 1103950 | > loss_text_ce: 0.03225642442703247 (0.03206987590141106) | > loss_mel_ce: 2.5701711177825928 (nan) | > loss: 0.010327094234526157 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13678171406943182) | > loader_time: 0.0045 (0.004632715735080769)  --> TIME: 2025-02-26 05:22:45 -- STEP: 32798/36938 -- GLOBAL_STEP: 1104000 | > loss_text_ce: 0.027868716046214104 (0.03207253597135976) | > loss_mel_ce: 3.288076162338257 (nan) | > loss: 0.01315851230174303 (nan) | > current_lr: 5e-06 | > step_time: 0.1533 (0.1367837334996403) | > loader_time: 0.0046 (0.004632806100455105)  --> TIME: 2025-02-26 05:22:54 -- STEP: 32848/36938 -- GLOBAL_STEP: 1104050 | > loss_text_ce: 0.029633203521370888 (0.03207390566527833) | > loss_mel_ce: 2.677027702331543 (nan) | > loss: 0.010740718804299831 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13677487289011395) | > loader_time: 0.0045 (0.0046327480573626146)  --> TIME: 2025-02-26 05:23:03 -- STEP: 32898/36938 -- GLOBAL_STEP: 1104100 | > loss_text_ce: 0.02880534902215004 (0.032073023655393475) | > loss_mel_ce: 1.616602897644043 (nan) | > loss: 0.006529398262500763 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13677234478334316) | > loader_time: 0.0045 (0.004632754567633472)  --> TIME: 2025-02-26 05:23:12 -- STEP: 32948/36938 -- GLOBAL_STEP: 1104150 | > loss_text_ce: 0.030001183971762657 (0.032073025472980604) | > loss_mel_ce: 3.0588974952697754 (nan) | > loss: 0.012257535010576248 (nan) | > current_lr: 5e-06 | > step_time: 0.1629 (0.13677641506607305) | > loader_time: 0.0048 (0.004632818130117097)  --> TIME: 2025-02-26 05:23:22 -- STEP: 32998/36938 -- GLOBAL_STEP: 1104200 | > loss_text_ce: 0.030319323763251305 (0.032069854545131164) | > loss_mel_ce: 2.500194549560547 (nan) | > loss: 0.010041722096502781 (nan) | > current_lr: 5e-06 | > step_time: 0.1868 (0.13678029099553546) | > loader_time: 0.0049 (0.004632859990426096)  --> TIME: 2025-02-26 05:23:31 -- STEP: 33048/36938 -- GLOBAL_STEP: 1104250 | > loss_text_ce: 0.03030405007302761 (0.03207005091543908) | > loss_mel_ce: 2.404082775115967 (nan) | > loss: 0.009660265408456326 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1367773031876763) | > loader_time: 0.0045 (0.004632874901258332)  --> TIME: 2025-02-26 05:23:40 -- STEP: 33098/36938 -- GLOBAL_STEP: 1104300 | > loss_text_ce: 0.031767524778842926 (0.03207052157042845) | > loss_mel_ce: 2.948019027709961 (nan) | > loss: 0.011824551038444042 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13677878544491867) | > loader_time: 0.0046 (0.004633049596378136)  --> TIME: 2025-02-26 05:23:50 -- STEP: 33148/36938 -- GLOBAL_STEP: 1104350 | > loss_text_ce: 0.03227142617106438 (0.03207219809990403) | > loss_mel_ce: 2.5228490829467773 (nan) | > loss: 0.010139367543160915 (nan) | > current_lr: 5e-06 | > step_time: 0.151 (0.13678252101448055) | > loader_time: 0.0047 (0.004633082488455746)  --> TIME: 2025-02-26 05:23:59 -- STEP: 33198/36938 -- GLOBAL_STEP: 1104400 | > loss_text_ce: 0.02383916825056076 (0.03207140209205352) | > loss_mel_ce: 2.8289241790771484 (nan) | > loss: 0.011320490390062332 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13678374363863494) | > loader_time: 0.0045 (0.004633085642516494)  --> TIME: 2025-02-26 05:24:09 -- STEP: 33248/36938 -- GLOBAL_STEP: 1104450 | > loss_text_ce: 0.02817675843834877 (0.03207095196518004) | > loss_mel_ce: 2.180910348892212 (nan) | > loss: 0.008766219019889832 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13679283275067347) | > loader_time: 0.0046 (0.004633092530308362)  --> TIME: 2025-02-26 05:24:18 -- STEP: 33298/36938 -- GLOBAL_STEP: 1104500 | > loss_text_ce: 0.03363754227757454 (0.03207085506922969) | > loss_mel_ce: 1.9425002336502075 (nan) | > loss: 0.00784181710332632 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13679455149903416) | > loader_time: 0.0045 (0.004633119402866006)  --> TIME: 2025-02-26 05:24:27 -- STEP: 33348/36938 -- GLOBAL_STEP: 1104550 | > loss_text_ce: 0.02966504730284214 (0.03207206366969875) | > loss_mel_ce: 0.01807403564453125 (nan) | > loss: 0.00018944080511573702 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.1367887858865764) | > loader_time: 0.0046 (0.004633301344223257)  --> TIME: 2025-02-26 05:24:37 -- STEP: 33398/36938 -- GLOBAL_STEP: 1104600 | > loss_text_ce: 0.03916456922888756 (0.032071462979515485) | > loss_mel_ce: 2.89675235748291 (nan) | > loss: 0.011650464497506618 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13678770406109608) | > loader_time: 0.0048 (0.004633263511082036)  --> TIME: 2025-02-26 05:24:46 -- STEP: 33448/36938 -- GLOBAL_STEP: 1104650 | > loss_text_ce: 0.03080473095178604 (0.03206993679063154) | > loss_mel_ce: 2.586526393890381 (nan) | > loss: 0.01038623508065939 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13678135909095618) | > loader_time: 0.0045 (0.004633299772946395)  --> TIME: 2025-02-26 05:24:55 -- STEP: 33498/36938 -- GLOBAL_STEP: 1104700 | > loss_text_ce: 0.034166183322668076 (0.03207081385708379) | > loss_mel_ce: 1.9277135133743286 (nan) | > loss: 0.007785237394273281 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13677781505352463) | > loader_time: 0.0046 (0.004633450936584988)  --> TIME: 2025-02-26 05:25:04 -- STEP: 33548/36938 -- GLOBAL_STEP: 1104750 | > loss_text_ce: 0.03268982470035553 (0.032070989951049754) | > loss_mel_ce: 3.1475188732147217 (nan) | > loss: 0.01261987630277872 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13677712776414508) | > loader_time: 0.0046 (0.00463344952169526)  --> TIME: 2025-02-26 05:25:14 -- STEP: 33598/36938 -- GLOBAL_STEP: 1104800 | > loss_text_ce: 0.04077475517988205 (0.03207030728258326) | > loss_mel_ce: 2.495568037033081 (nan) | > loss: 0.010064853355288506 (nan) | > current_lr: 5e-06 | > step_time: 0.1706 (0.13678851122089522) | > loader_time: 0.0048 (0.004633475261129379)  --> TIME: 2025-02-26 05:25:23 -- STEP: 33648/36938 -- GLOBAL_STEP: 1104850 | > loss_text_ce: 0.04798533767461777 (0.032070412448412654) | > loss_mel_ce: 2.181342601776123 (nan) | > loss: 0.008846540004014969 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13678525683713258) | > loader_time: 0.0045 (0.004633453953169768)  --> TIME: 2025-02-26 05:25:33 -- STEP: 33698/36938 -- GLOBAL_STEP: 1104900 | > loss_text_ce: 0.02930544503033161 (0.03207012060788112) | > loss_mel_ce: 3.0462448596954346 (nan) | > loss: 0.012204565107822418 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13678429210792403) | > loader_time: 0.0047 (0.004633464030154236)  --> TIME: 2025-02-26 05:25:42 -- STEP: 33748/36938 -- GLOBAL_STEP: 1104950 | > loss_text_ce: 0.027746152132749557 (0.03206909967457126) | > loss_mel_ce: 2.5881173610687256 (nan) | > loss: 0.010380411520600319 (nan) | > current_lr: 5e-06 | > step_time: 0.1447 (0.13678461037725498) | > loader_time: 0.0046 (0.004633497228212109)  --> TIME: 2025-02-26 05:25:51 -- STEP: 33798/36938 -- GLOBAL_STEP: 1105000 | > loss_text_ce: 0.023474447429180145 (0.03206768857871354) | > loss_mel_ce: 2.5379951000213623 (nan) | > loss: 0.010164562612771988 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13678187465193742) | > loader_time: 0.0047 (0.004633448068767862)  --> TIME: 2025-02-26 05:26:00 -- STEP: 33848/36938 -- GLOBAL_STEP: 1105050 | > loss_text_ce: 0.03253777697682381 (0.03206648300989022) | > loss_mel_ce: 1.9820435047149658 (nan) | > loss: 0.00799437053501606 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1367762929664614) | > loader_time: 0.0048 (0.0046335027815742735)  --> TIME: 2025-02-26 05:26:10 -- STEP: 33898/36938 -- GLOBAL_STEP: 1105100 | > loss_text_ce: 0.02789730578660965 (0.03206753157586604) | > loss_mel_ce: 2.446223497390747 (nan) | > loss: 0.009817940182983875 (nan) | > current_lr: 5e-06 | > step_time: 0.179 (0.13677541084560818) | > loader_time: 0.0048 (0.004633515005906739)  --> TIME: 2025-02-26 05:26:19 -- STEP: 33948/36938 -- GLOBAL_STEP: 1105150 | > loss_text_ce: 0.02366296760737896 (0.03206683632056709) | > loss_mel_ce: 2.0300190448760986 (nan) | > loss: 0.008149532601237297 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.1367771745051155) | > loader_time: 0.0045 (0.0046335213159354495)  --> TIME: 2025-02-26 05:26:29 -- STEP: 33998/36938 -- GLOBAL_STEP: 1105200 | > loss_text_ce: 0.04352831840515137 (0.03206525041118931) | > loss_mel_ce: 0.057861328125 (nan) | > loss: 0.00040233987965621054 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13678168613733035) | > loader_time: 0.0045 (0.004633513617020454)  --> TIME: 2025-02-26 05:26:38 -- STEP: 34048/36938 -- GLOBAL_STEP: 1105250 | > loss_text_ce: 0.028013231232762337 (0.03206272870462481) | > loss_mel_ce: 2.7754766941070557 (nan) | > loss: 0.011124961078166962 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13678332098892818) | > loader_time: 0.0045 (0.0046335368984399775)  --> TIME: 2025-02-26 05:26:47 -- STEP: 34098/36938 -- GLOBAL_STEP: 1105300 | > loss_text_ce: 0.03214612230658531 (0.032061190596192765) | > loss_mel_ce: 0.7219441533088684 (nan) | > loss: 0.002992421854287386 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13677885177317475) | > loader_time: 0.0046 (0.004633516746222057)  --> TIME: 2025-02-26 05:26:56 -- STEP: 34148/36938 -- GLOBAL_STEP: 1105350 | > loss_text_ce: 0.03458867222070694 (0.03206234343969678) | > loss_mel_ce: 2.9540090560913086 (nan) | > loss: 0.011859514750540257 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1367753227463348) | > loader_time: 0.0045 (0.0046335803243389645)  --> TIME: 2025-02-26 05:27:05 -- STEP: 34198/36938 -- GLOBAL_STEP: 1105400 | > loss_text_ce: 0.03310973942279816 (0.032062842164124694) | > loss_mel_ce: 2.4894843101501465 (nan) | > loss: 0.010010293684899807 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1367652644134494) | > loader_time: 0.0045 (0.0046336335378457435)  --> TIME: 2025-02-26 05:27:15 -- STEP: 34248/36938 -- GLOBAL_STEP: 1105450 | > loss_text_ce: 0.03585263341665268 (0.03206113277284231) | > loss_mel_ce: 2.848200559616089 (nan) | > loss: 0.011444656178355217 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13677068758835184) | > loader_time: 0.0045 (0.0046336365355923925)  --> TIME: 2025-02-26 05:27:24 -- STEP: 34298/36938 -- GLOBAL_STEP: 1105500 | > loss_text_ce: 0.03346199914813042 (0.032058989439181035) | > loss_mel_ce: 1.932666540145874 (nan) | > loss: 0.00780209805816412 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13677270524084367) | > loader_time: 0.0048 (0.0046335669521465365)  --> TIME: 2025-02-26 05:27:34 -- STEP: 34348/36938 -- GLOBAL_STEP: 1105550 | > loss_text_ce: 0.039337772876024246 (0.03205985216958772) | > loss_mel_ce: 2.132105588912964 (nan) | > loss: 0.008616839535534382 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13677427636684475) | > loader_time: 0.0046 (0.004633529896756679)  --> TIME: 2025-02-26 05:27:43 -- STEP: 34398/36938 -- GLOBAL_STEP: 1105600 | > loss_text_ce: 0.03082120046019554 (0.03205891007950522) | > loss_mel_ce: 2.7211737632751465 (nan) | > loss: 0.010920614935457706 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13677722619393512) | > loader_time: 0.0046 (0.004633484853479114)  --> TIME: 2025-02-26 05:27:53 -- STEP: 34448/36938 -- GLOBAL_STEP: 1105650 | > loss_text_ce: 0.03382417932152748 (0.0320577804757047) | > loss_mel_ce: 2.0819790363311768 (nan) | > loss: 0.008396045304834843 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1367784722068402) | > loader_time: 0.0045 (0.0046334525581520096)  --> TIME: 2025-02-26 05:28:02 -- STEP: 34498/36938 -- GLOBAL_STEP: 1105700 | > loss_text_ce: 0.03406041860580444 (0.032057707431915106) | > loss_mel_ce: 2.8215365409851074 (nan) | > loss: 0.01133173517882824 (nan) | > current_lr: 5e-06 | > step_time: 0.1799 (0.13678347367785396) | > loader_time: 0.0049 (0.0046334332387007295)  --> TIME: 2025-02-26 05:28:12 -- STEP: 34548/36938 -- GLOBAL_STEP: 1105750 | > loss_text_ce: 0.032068025320768356 (0.032059738324564484) | > loss_mel_ce: 3.404674530029297 (nan) | > loss: 0.013637867756187916 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.13679011334999783) | > loader_time: 0.0048 (0.0046334022226275015)  --> TIME: 2025-02-26 05:28:21 -- STEP: 34598/36938 -- GLOBAL_STEP: 1105800 | > loss_text_ce: 0.03802315518260002 (0.032059704529886913) | > loss_mel_ce: 2.043546676635742 (nan) | > loss: 0.00826019886881113 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.1367873856081253) | > loader_time: 0.0045 (0.004633344338184016)  --> TIME: 2025-02-26 05:28:31 -- STEP: 34648/36938 -- GLOBAL_STEP: 1105850 | > loss_text_ce: 0.02388039045035839 (0.032059730269877934) | > loss_mel_ce: 3.0219106674194336 (nan) | > loss: 0.012086473405361176 (nan) | > current_lr: 5e-06 | > step_time: 0.1782 (0.13679486497311197) | > loader_time: 0.0047 (0.004633323435033605)  --> TIME: 2025-02-26 05:28:40 -- STEP: 34698/36938 -- GLOBAL_STEP: 1105900 | > loss_text_ce: 0.03133251145482063 (0.03205842685258883) | > loss_mel_ce: 2.550349712371826 (nan) | > loss: 0.010244770906865597 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13679559161817165) | > loader_time: 0.0047 (0.004633460520900924)  --> TIME: 2025-02-26 05:28:49 -- STEP: 34748/36938 -- GLOBAL_STEP: 1105950 | > loss_text_ce: 0.030359184369444847 (0.03205671253666334) | > loss_mel_ce: 2.79248309135437 (nan) | > loss: 0.011201756075024605 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.13678840083028665) | > loader_time: 0.0048 (0.004633548153521225)  --> TIME: 2025-02-26 05:28:58 -- STEP: 34798/36938 -- GLOBAL_STEP: 1106000 | > loss_text_ce: 0.027764255180954933 (0.03205658946003449) | > loss_mel_ce: 3.0162620544433594 (nan) | > loss: 0.012079470790922642 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.1367933791467153) | > loader_time: 0.0047 (0.004633601167173343)  --> TIME: 2025-02-26 05:29:08 -- STEP: 34848/36938 -- GLOBAL_STEP: 1106050 | > loss_text_ce: 0.02980875037610531 (0.032056825690777856) | > loss_mel_ce: 2.999238967895508 (nan) | > loss: 0.012020030990242958 (nan) | > current_lr: 5e-06 | > step_time: 0.1489 (0.13679485479361259) | > loader_time: 0.0047 (0.00463370214617503)  --> TIME: 2025-02-26 05:29:17 -- STEP: 34898/36938 -- GLOBAL_STEP: 1106100 | > loss_text_ce: 0.028748836368322372 (0.03205496995020827) | > loss_mel_ce: 2.922650098800659 (nan) | > loss: 0.011711901053786278 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.1367899826573295) | > loader_time: 0.0048 (0.004633827205100247)  --> TIME: 2025-02-26 05:29:26 -- STEP: 34948/36938 -- GLOBAL_STEP: 1106150 | > loss_text_ce: 0.01830713264644146 (0.032055116130780534) | > loss_mel_ce: 0.9962751865386963 (nan) | > loss: 0.004026120528578758 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13678967208437753) | > loader_time: 0.0046 (0.004633933500170053)  --> TIME: 2025-02-26 05:29:35 -- STEP: 34998/36938 -- GLOBAL_STEP: 1106200 | > loss_text_ce: 0.04192421957850456 (0.03205268592525534) | > loss_mel_ce: 3.016813278198242 (nan) | > loss: 0.01213784795254469 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13678829702242976) | > loader_time: 0.0049 (0.0046339889302704)  --> TIME: 2025-02-26 05:29:45 -- STEP: 35048/36938 -- GLOBAL_STEP: 1106250 | > loss_text_ce: 0.0320516899228096 (0.03205303875614112) | > loss_mel_ce: 3.079824686050415 (nan) | > loss: 0.012348717078566551 (nan) | > current_lr: 5e-06 | > step_time: 0.196 (0.13679068783352188) | > loader_time: 0.0049 (0.004634089725416439)  --> TIME: 2025-02-26 05:29:54 -- STEP: 35098/36938 -- GLOBAL_STEP: 1106300 | > loss_text_ce: 0.03414851799607277 (0.03205537975239998) | > loss_mel_ce: 2.3893544673919678 (nan) | > loss: 0.009617075324058533 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.1367876045153258) | > loader_time: 0.0049 (0.004634150868297947)  --> TIME: 2025-02-26 05:30:03 -- STEP: 35148/36938 -- GLOBAL_STEP: 1106350 | > loss_text_ce: 0.02889118529856205 (0.032056115441372116) | > loss_mel_ce: 1.9836037158966064 (nan) | > loss: 0.007986091077327728 (nan) | > current_lr: 5e-06 | > step_time: 0.1439 (0.1367773529380834) | > loader_time: 0.0046 (0.004634243766100462)  --> TIME: 2025-02-26 05:30:12 -- STEP: 35198/36938 -- GLOBAL_STEP: 1106400 | > loss_text_ce: 0.036775071173906326 (0.032056219548693296) | > loss_mel_ce: 1.641021728515625 (nan) | > loss: 0.0066579245030879974 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13678053693545447) | > loader_time: 0.0048 (0.004634357675976868)  --> TIME: 2025-02-26 05:30:22 -- STEP: 35248/36938 -- GLOBAL_STEP: 1106450 | > loss_text_ce: 0.024563444778323174 (0.03205542390261609) | > loss_mel_ce: 2.3394522666931152 (nan) | > loss: 0.009381015785038471 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1367790771244988) | > loader_time: 0.0046 (0.004634395769331798)  --> TIME: 2025-02-26 05:30:31 -- STEP: 35298/36938 -- GLOBAL_STEP: 1106500 | > loss_text_ce: 0.02686968445777893 (0.032053577870287134) | > loss_mel_ce: 2.8707995414733887 (nan) | > loss: 0.011498688720166683 (nan) | > current_lr: 5e-06 | > step_time: 0.1711 (0.1367836944023084) | > loader_time: 0.0049 (0.004634514004388888)  --> TIME: 2025-02-26 05:30:40 -- STEP: 35348/36938 -- GLOBAL_STEP: 1106550 | > loss_text_ce: 0.02549058571457863 (0.0320531388603912) | > loss_mel_ce: 2.1824331283569336 (nan) | > loss: 0.008761602453887463 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13677940605937833) | > loader_time: 0.0045 (0.004634492487959732)  --> TIME: 2025-02-26 05:30:49 -- STEP: 35398/36938 -- GLOBAL_STEP: 1106600 | > loss_text_ce: 0.04056372120976448 (0.03205395069670411) | > loss_mel_ce: 2.2999417781829834 (nan) | > loss: 0.00928772147744894 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13677207921425474) | > loader_time: 0.0047 (0.004634508204814406)  --> TIME: 2025-02-26 05:30:59 -- STEP: 35448/36938 -- GLOBAL_STEP: 1106650 | > loss_text_ce: 0.018371127545833588 (0.03205261496883545) | > loss_mel_ce: 2.09828519821167 (nan) | > loss: 0.00839943066239357 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1367736162302113) | > loader_time: 0.0045 (0.004634488694314285)  --> TIME: 2025-02-26 05:31:08 -- STEP: 35498/36938 -- GLOBAL_STEP: 1106700 | > loss_text_ce: 0.03288678452372551 (0.0320519665227283) | > loss_mel_ce: 2.579263687133789 (nan) | > loss: 0.010365677066147327 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13677992730202804) | > loader_time: 0.0046 (0.004634609336053488)  --> TIME: 2025-02-26 05:31:17 -- STEP: 35548/36938 -- GLOBAL_STEP: 1106750 | > loss_text_ce: 0.03944756090641022 (0.03205095074281324) | > loss_mel_ce: 2.590834856033325 (nan) | > loss: 0.01043762918561697 (nan) | > current_lr: 5e-06 | > step_time: 0.1265 (0.13677126130304468) | > loader_time: 0.0046 (0.004634613621633266)  --> TIME: 2025-02-26 05:31:26 -- STEP: 35598/36938 -- GLOBAL_STEP: 1106800 | > loss_text_ce: 0.03377662971615791 (0.03205046559641541) | > loss_mel_ce: 3.206559896469116 (nan) | > loss: 0.012858478352427483 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13676744111770178) | > loader_time: 0.0048 (0.0046347580543293675)  --> TIME: 2025-02-26 05:31:35 -- STEP: 35648/36938 -- GLOBAL_STEP: 1106850 | > loss_text_ce: 0.038934413343667984 (0.032051573023616976) | > loss_mel_ce: 2.9238784313201904 (nan) | > loss: 0.011757194995880127 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13676355855415764) | > loader_time: 0.0047 (0.004634798168333449)  --> TIME: 2025-02-26 05:31:45 -- STEP: 35698/36938 -- GLOBAL_STEP: 1106900 | > loss_text_ce: 0.03582945838570595 (0.03205094780854634) | > loss_mel_ce: 3.0864906311035156 (nan) | > loss: 0.012390160001814365 (nan) | > current_lr: 5e-06 | > step_time: 0.1888 (0.13676298681818408) | > loader_time: 0.0049 (0.0046349154232503)  --> TIME: 2025-02-26 05:31:54 -- STEP: 35748/36938 -- GLOBAL_STEP: 1106950 | > loss_text_ce: 0.030942443758249283 (0.032051880482876605) | > loss_mel_ce: 2.5790557861328125 (nan) | > loss: 0.010357136838138103 (nan) | > current_lr: 5e-06 | > step_time: 0.1784 (0.13675678500636196) | > loader_time: 0.0049 (0.004634933489286943)  --> TIME: 2025-02-26 05:32:03 -- STEP: 35798/36938 -- GLOBAL_STEP: 1107000 | > loss_text_ce: 0.02038886584341526 (0.03205033996868238) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13675510077458178) | > loader_time: 0.0046 (0.00463506392083543)  --> TIME: 2025-02-26 05:32:13 -- STEP: 35848/36938 -- GLOBAL_STEP: 1107050 | > loss_text_ce: 0.027733897790312767 (0.03205014533045079) | > loss_mel_ce: 3.0703680515289307 (nan) | > loss: 0.01229405589401722 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13675850093085443) | > loader_time: 0.0046 (0.004635180640342991)  --> TIME: 2025-02-26 05:32:22 -- STEP: 35898/36938 -- GLOBAL_STEP: 1107100 | > loss_text_ce: 0.024874228984117508 (0.032049704829652276) | > loss_mel_ce: 0.0857696533203125 (nan) | > loss: 0.0004390630347188562 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.1367552989345724) | > loader_time: 0.0046 (0.004635254196671937)  --> TIME: 2025-02-26 05:32:31 -- STEP: 35948/36938 -- GLOBAL_STEP: 1107150 | > loss_text_ce: 0.037925563752651215 (0.03204911010111555) | > loss_mel_ce: 3.144481658935547 (nan) | > loss: 0.012628600932657719 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13675651548170684) | > loader_time: 0.0046 (0.004635262571559567)  --> TIME: 2025-02-26 05:32:41 -- STEP: 35998/36938 -- GLOBAL_STEP: 1107200 | > loss_text_ce: 0.03742869943380356 (0.03204929831652653) | > loss_mel_ce: 2.127910614013672 (nan) | > loss: 0.008592616766691208 (nan) | > current_lr: 5e-06 | > step_time: 0.1706 (0.13676208685196733) | > loader_time: 0.005 (0.004635367998050763)  --> TIME: 2025-02-26 05:32:50 -- STEP: 36048/36938 -- GLOBAL_STEP: 1107250 | > loss_text_ce: 0.030079111456871033 (0.03205038338298261) | > loss_mel_ce: 2.9168412685394287 (nan) | > loss: 0.011694129556417465 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.136754586856153) | > loader_time: 0.0046 (0.0046354074690852795)  --> TIME: 2025-02-26 05:32:59 -- STEP: 36098/36938 -- GLOBAL_STEP: 1107300 | > loss_text_ce: 0.023210730403661728 (0.03204974734067875) | > loss_mel_ce: 3.264608860015869 (nan) | > loss: 0.01304690446704626 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.13674988187352144) | > loader_time: 0.0049 (0.004635393801168188)  --> TIME: 2025-02-26 05:33:08 -- STEP: 36148/36938 -- GLOBAL_STEP: 1107350 | > loss_text_ce: 0.042580120265483856 (0.03204847245763224) | > loss_mel_ce: 0.01267089881002903 (nan) | > loss: 0.0002192500833189115 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13675103291640014) | > loader_time: 0.0045 (0.004635449576802593)  --> TIME: 2025-02-26 05:33:18 -- STEP: 36198/36938 -- GLOBAL_STEP: 1107400 | > loss_text_ce: 0.028049608692526817 (0.03204778652974004) | > loss_mel_ce: 2.7727644443511963 (nan) | > loss: 0.011114342138171196 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.13675109012540995) | > loader_time: 0.0047 (0.004635520650310449)  --> TIME: 2025-02-26 05:33:27 -- STEP: 36248/36938 -- GLOBAL_STEP: 1107450 | > loss_text_ce: 0.02983796037733555 (0.03204891265098529) | > loss_mel_ce: 3.264915704727173 (nan) | > loss: 0.0130744194611907 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.13674904751056807) | > loader_time: 0.0048 (0.00463556173199768)  --> TIME: 2025-02-26 05:33:36 -- STEP: 36298/36938 -- GLOBAL_STEP: 1107500 | > loss_text_ce: 0.04075802117586136 (0.03205022618527704) | > loss_mel_ce: 2.942105293273926 (nan) | > loss: 0.011836760677397251 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.1367455372778052) | > loader_time: 0.0047 (0.004635514007848715)  --> TIME: 2025-02-26 05:33:45 -- STEP: 36348/36938 -- GLOBAL_STEP: 1107550 | > loss_text_ce: 0.023872438818216324 (0.03204803437585815) | > loss_mel_ce: 2.8896944522857666 (nan) | > loss: 0.011561773717403412 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13673949433826635) | > loader_time: 0.0047 (0.0046355698228577324)  --> TIME: 2025-02-26 05:33:54 -- STEP: 36398/36938 -- GLOBAL_STEP: 1107600 | > loss_text_ce: 0.036584436893463135 (0.03204933724685101) | > loss_mel_ce: 2.4731626510620117 (nan) | > loss: 0.00995931401848793 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13674259382300671) | > loader_time: 0.0046 (0.004635601883714289)  --> TIME: 2025-02-26 05:34:04 -- STEP: 36448/36938 -- GLOBAL_STEP: 1107650 | > loss_text_ce: 0.02657628431916237 (0.03204894470410401) | > loss_mel_ce: 2.774433135986328 (nan) | > loss: 0.011115116998553276 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13674257975644066) | > loader_time: 0.0046 (0.004635739649618968)  --> TIME: 2025-02-26 05:34:14 -- STEP: 36498/36938 -- GLOBAL_STEP: 1107700 | > loss_text_ce: 0.029373012483119965 (0.03204906407468213) | > loss_mel_ce: 2.7195427417755127 (nan) | > loss: 0.010908395983278751 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.13675376192969105) | > loader_time: 0.0046 (0.0046358846221481185)  --> TIME: 2025-02-26 05:34:23 -- STEP: 36548/36938 -- GLOBAL_STEP: 1107750 | > loss_text_ce: 0.03792351484298706 (0.03204818490518166) | > loss_mel_ce: 2.631117105484009 (nan) | > loss: 0.010591431520879269 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13675715277619563) | > loader_time: 0.0046 (0.00463595153649523)  --> TIME: 2025-02-26 05:34:32 -- STEP: 36598/36938 -- GLOBAL_STEP: 1107800 | > loss_text_ce: 0.02654043212532997 (0.0320489904617553) | > loss_mel_ce: 2.4446463584899902 (nan) | > loss: 0.009806297719478607 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13674943255846625) | > loader_time: 0.0045 (0.004635944230431769)  --> TIME: 2025-02-26 05:34:41 -- STEP: 36648/36938 -- GLOBAL_STEP: 1107850 | > loss_text_ce: 0.030220946297049522 (0.03204907327191044) | > loss_mel_ce: 2.557966470718384 (nan) | > loss: 0.010270585305988789 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13675036036247437) | > loader_time: 0.0045 (0.004635885686391382)  --> TIME: 2025-02-26 05:34:51 -- STEP: 36698/36938 -- GLOBAL_STEP: 1107900 | > loss_text_ce: 0.033639032393693924 (0.03205081325855144) | > loss_mel_ce: 2.8366920948028564 (nan) | > loss: 0.0113902036100626 (nan) | > current_lr: 5e-06 | > step_time: 0.1505 (0.13674729576850977) | > loader_time: 0.0047 (0.004635844433870891)  --> TIME: 2025-02-26 05:35:00 -- STEP: 36748/36938 -- GLOBAL_STEP: 1107950 | > loss_text_ce: 0.03828927502036095 (0.03205288320816706) | > loss_mel_ce: 2.5773892402648926 (nan) | > loss: 0.010379677638411522 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13674878139185498) | > loader_time: 0.0046 (0.004635942076993091)  --> TIME: 2025-02-26 05:35:09 -- STEP: 36798/36938 -- GLOBAL_STEP: 1108000 | > loss_text_ce: 0.03186458349227905 (0.03205274709303627) | > loss_mel_ce: 2.8256967067718506 (nan) | > loss: 0.011339529417455196 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.13674730905741966) | > loader_time: 0.0047 (0.004636014257476932)  --> TIME: 2025-02-26 05:35:19 -- STEP: 36848/36938 -- GLOBAL_STEP: 1108050 | > loss_text_ce: 0.032442763447761536 (0.03205329456041675) | > loss_mel_ce: 3.091583490371704 (nan) | > loss: 0.01239693071693182 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13674620325752754) | > loader_time: 0.0046 (0.004636035230022047)  --> TIME: 2025-02-26 05:35:28 -- STEP: 36898/36938 -- GLOBAL_STEP: 1108100 | > loss_text_ce: 0.04284931719303131 (0.03205281634315237) | > loss_mel_ce: 2.556234836578369 (nan) | > loss: 0.010313826613128185 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13673601342937935) | > loader_time: 0.0045 (0.004636090217524967)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.02101975266296085 (-0.0006039591118840865) | > avg_loss_text_ce: 0.031255127889758864 (-0.0001916696252946687) | > avg_loss_mel_ce: 2.478114526755739 (-0.0027713557281132672) | > avg_loss: 2.509369651279827 (-0.002963029807157014) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_1108140.pth  > EPOCH: 31/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-26 05:36:01)   --> TIME: 2025-02-26 05:36:04 -- STEP: 10/36938 -- GLOBAL_STEP: 1108150 | > loss_text_ce: 0.03477040305733681 (0.03279378116130829) | > loss_mel_ce: 2.840360403060913 (2.316349194943905) | > loss: 0.01140925008803606 (0.00932199644157663) | > current_lr: 5e-06 | > step_time: 0.1639 (0.13622114658355713) | > loader_time: 0.0048 (0.0046924114227294925)  --> TIME: 2025-02-26 05:36:13 -- STEP: 60/36938 -- GLOBAL_STEP: 1108200 | > loss_text_ce: 0.032147184014320374 (0.030859163589775564) | > loss_mel_ce: 2.5826008319854736 (2.5952048674846684) | > loss: 0.010375984944403172 (0.010420889663024957) | > current_lr: 5e-06 | > step_time: 0.1384 (0.1386652747790018) | > loader_time: 0.0048 (0.004764676094055176)  --> TIME: 2025-02-26 05:36:22 -- STEP: 110/36938 -- GLOBAL_STEP: 1108250 | > loss_text_ce: 0.028370538726449013 (0.030964842049235646) | > loss_mel_ce: 2.636791467666626 (nan) | > loss: 0.010576040484011173 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13554226051677346) | > loader_time: 0.0047 (0.004724459214643997)  --> TIME: 2025-02-26 05:36:31 -- STEP: 160/36938 -- GLOBAL_STEP: 1108300 | > loss_text_ce: 0.02962934598326683 (0.03131059091538192) | > loss_mel_ce: 2.9808449745178223 (nan) | > loss: 0.011946327053010464 (nan) | > current_lr: 5e-06 | > step_time: 0.1492 (0.13572767078876494) | > loader_time: 0.0046 (0.004719278216361999)  --> TIME: 2025-02-26 05:36:41 -- STEP: 210/36938 -- GLOBAL_STEP: 1108350 | > loss_text_ce: 0.0345623679459095 (0.031201835468943635) | > loss_mel_ce: 2.6560773849487305 (nan) | > loss: 0.010677142068743706 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13624871571858713) | > loader_time: 0.0046 (0.004714605921790713)  --> TIME: 2025-02-26 05:36:50 -- STEP: 260/36938 -- GLOBAL_STEP: 1108400 | > loss_text_ce: 0.030156034976243973 (0.031458410101297955) | > loss_mel_ce: 3.0803534984588623 (nan) | > loss: 0.01234329305589199 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13502875566482542) | > loader_time: 0.0047 (0.0047093382248511685)  --> TIME: 2025-02-26 05:36:59 -- STEP: 310/36938 -- GLOBAL_STEP: 1108450 | > loss_text_ce: 0.03268842771649361 (0.031459614337091484) | > loss_mel_ce: 3.244213104248047 (nan) | > loss: 0.013003578409552574 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.13612782955169672) | > loader_time: 0.0049 (0.004726822914615754)  --> TIME: 2025-02-26 05:37:09 -- STEP: 360/36938 -- GLOBAL_STEP: 1108500 | > loss_text_ce: 0.035224996507167816 (0.031474767234693786) | > loss_mel_ce: 2.8253085613250732 (nan) | > loss: 0.011351323686540127 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13669601612620888) | > loader_time: 0.0049 (0.004747609959708318)  --> TIME: 2025-02-26 05:37:19 -- STEP: 410/36938 -- GLOBAL_STEP: 1108550 | > loss_text_ce: 0.020602544769644737 (0.031623019548360165) | > loss_mel_ce: 2.042463779449463 (nan) | > loss: 0.008186771534383297 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13719776723443008) | > loader_time: 0.0045 (0.00474150529721888)  --> TIME: 2025-02-26 05:37:28 -- STEP: 460/36938 -- GLOBAL_STEP: 1108600 | > loss_text_ce: 0.027326438575983047 (0.031687546126625464) | > loss_mel_ce: 2.4759678840637207 (nan) | > loss: 0.009933708235621452 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.13695962066235753) | > loader_time: 0.0047 (0.004730644951695979)  --> TIME: 2025-02-26 05:37:37 -- STEP: 510/36938 -- GLOBAL_STEP: 1108650 | > loss_text_ce: 0.0302836075425148 (0.031819047466577856) | > loss_mel_ce: 2.265637159347534 (nan) | > loss: 0.009110797196626663 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13725575980018173) | > loader_time: 0.0045 (0.004722403545005647)  --> TIME: 2025-02-26 05:37:47 -- STEP: 560/36938 -- GLOBAL_STEP: 1108700 | > loss_text_ce: 0.03260678052902222 (0.03187914524792829) | > loss_mel_ce: 2.3449058532714844 (nan) | > loss: 0.009434574283659458 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.1374231568404607) | > loader_time: 0.005 (0.0047200322151184065)  --> TIME: 2025-02-26 05:37:56 -- STEP: 610/36938 -- GLOBAL_STEP: 1108750 | > loss_text_ce: 0.035308849066495895 (0.03192916297979777) | > loss_mel_ce: 2.1154749393463135 (nan) | > loss: 0.00853485707193613 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1369369553737954) | > loader_time: 0.0046 (0.004729674683242546)  --> TIME: 2025-02-26 05:38:06 -- STEP: 660/36938 -- GLOBAL_STEP: 1108800 | > loss_text_ce: 0.034583646804094315 (0.03190146100312248) | > loss_mel_ce: 2.7957491874694824 (nan) | > loss: 0.011231480166316032 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1374577475316598) | > loader_time: 0.0045 (0.004729713815631287)  --> TIME: 2025-02-26 05:38:15 -- STEP: 710/36938 -- GLOBAL_STEP: 1108850 | > loss_text_ce: 0.03340119868516922 (0.031838365472380975) | > loss_mel_ce: 3.032280206680298 (nan) | > loss: 0.012165402993559837 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13753466673300313) | > loader_time: 0.0045 (0.004720691895820725)  --> TIME: 2025-02-26 05:38:24 -- STEP: 760/36938 -- GLOBAL_STEP: 1108900 | > loss_text_ce: 0.026836536824703217 (0.03179603345128463) | > loss_mel_ce: 3.2688958644866943 (nan) | > loss: 0.013078304007649422 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.1374741899339777) | > loader_time: 0.0046 (0.00471623288957696)  --> TIME: 2025-02-26 05:38:34 -- STEP: 810/36938 -- GLOBAL_STEP: 1108950 | > loss_text_ce: 0.031251635402441025 (0.031698055313555185) | > loss_mel_ce: 2.589690685272217 (nan) | > loss: 0.010400565341114998 (nan) | > current_lr: 5e-06 | > step_time: 0.1861 (0.1372520676365606) | > loader_time: 0.0049 (0.004713557384632252)  --> TIME: 2025-02-26 05:38:43 -- STEP: 860/36938 -- GLOBAL_STEP: 1109000 | > loss_text_ce: 0.02953307330608368 (0.0316979508868657) | > loss_mel_ce: 2.9208478927612305 (nan) | > loss: 0.01170786190778017 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.1374575620473819) | > loader_time: 0.0046 (0.0047089352164157595)  --> TIME: 2025-02-26 05:38:53 -- STEP: 910/36938 -- GLOBAL_STEP: 1109050 | > loss_text_ce: 0.030747178941965103 (0.03169597909263855) | > loss_mel_ce: 2.7180464267730713 (nan) | > loss: 0.010907911695539951 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13764661013425072) | > loader_time: 0.0047 (0.004701817952669583)  --> TIME: 2025-02-26 05:39:02 -- STEP: 960/36938 -- GLOBAL_STEP: 1109100 | > loss_text_ce: 0.03527306392788887 (0.03175971492698104) | > loss_mel_ce: 2.6588454246520996 (nan) | > loss: 0.010690947063267231 (nan) | > current_lr: 5e-06 | > step_time: 0.1827 (0.13759594708681128) | > loader_time: 0.0049 (0.004695094128449757)  --> TIME: 2025-02-26 05:39:11 -- STEP: 1010/36938 -- GLOBAL_STEP: 1109150 | > loss_text_ce: 0.022106753662228584 (0.031781030582762) | > loss_mel_ce: 2.477524518966675 (nan) | > loss: 0.009919172152876854 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.1376686674533506) | > loader_time: 0.0045 (0.004689886782429005)  --> TIME: 2025-02-26 05:39:21 -- STEP: 1060/36938 -- GLOBAL_STEP: 1109200 | > loss_text_ce: 0.027097880840301514 (0.03181650914286947) | > loss_mel_ce: 1.81634521484375 (nan) | > loss: 0.0073152510449290276 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13777277941973723) | > loader_time: 0.0044 (0.004686654963583316)  --> TIME: 2025-02-26 05:39:30 -- STEP: 1110/36938 -- GLOBAL_STEP: 1109250 | > loss_text_ce: 0.027180656790733337 (0.03182310390009266) | > loss_mel_ce: 2.8106460571289062 (nan) | > loss: 0.01126121822744608 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.13776830080393218) | > loader_time: 0.0048 (0.004682163504866866)  --> TIME: 2025-02-26 05:39:40 -- STEP: 1160/36938 -- GLOBAL_STEP: 1109300 | > loss_text_ce: 0.04166046157479286 (0.03182671391886883) | > loss_mel_ce: 0.13010406494140625 (nan) | > loss: 0.0006816053064540029 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.1377406194292268) | > loader_time: 0.0045 (0.004679240440500195)  --> TIME: 2025-02-26 05:39:49 -- STEP: 1210/36938 -- GLOBAL_STEP: 1109350 | > loss_text_ce: 0.0328981839120388 (0.031817642179094545) | > loss_mel_ce: 2.6651673316955566 (nan) | > loss: 0.010706610046327114 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13776968136306653) | > loader_time: 0.0046 (0.004676776286984281)  --> TIME: 2025-02-26 05:39:59 -- STEP: 1260/36938 -- GLOBAL_STEP: 1109400 | > loss_text_ce: 0.028813883662223816 (0.0317633878794454) | > loss_mel_ce: 2.569782018661499 (nan) | > loss: 0.01031188853085041 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13782397073412722) | > loader_time: 0.0048 (0.004674867978171697)  --> TIME: 2025-02-26 05:40:08 -- STEP: 1310/36938 -- GLOBAL_STEP: 1109450 | > loss_text_ce: 0.032763950526714325 (0.03178162388361362) | > loss_mel_ce: 3.6718406677246094 (nan) | > loss: 0.014700813218951225 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.1378399752478565) | > loader_time: 0.0046 (0.0046730278102496215)  --> TIME: 2025-02-26 05:40:17 -- STEP: 1360/36938 -- GLOBAL_STEP: 1109500 | > loss_text_ce: 0.02733694761991501 (0.03174347062458227) | > loss_mel_ce: 2.4648797512054443 (nan) | > loss: 0.009889748878777027 (nan) | > current_lr: 5e-06 | > step_time: 0.1268 (0.13782726684037383) | > loader_time: 0.0045 (0.004670261810807622)  --> TIME: 2025-02-26 05:40:27 -- STEP: 1410/36938 -- GLOBAL_STEP: 1109550 | > loss_text_ce: 0.028720788657665253 (0.031745246162387046) | > loss_mel_ce: 2.0243701934814453 (nan) | > loss: 0.008147187530994415 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13789598163983516) | > loader_time: 0.0045 (0.0046689026744653125)  --> TIME: 2025-02-26 05:40:36 -- STEP: 1460/36938 -- GLOBAL_STEP: 1109600 | > loss_text_ce: 0.03363500535488129 (0.03172083756159226) | > loss_mel_ce: 2.3974609375 (nan) | > loss: 0.009647206403315067 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.13784008581344417) | > loader_time: 0.0046 (0.004666835641207756)  --> TIME: 2025-02-26 05:40:45 -- STEP: 1510/36938 -- GLOBAL_STEP: 1109650 | > loss_text_ce: 0.02948017790913582 (0.031781915069514555) | > loss_mel_ce: 0.8401947021484375 (nan) | > loss: 0.003451091004535556 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13765736526211395) | > loader_time: 0.0046 (0.004663853140066785)  --> TIME: 2025-02-26 05:40:54 -- STEP: 1560/36938 -- GLOBAL_STEP: 1109700 | > loss_text_ce: 0.03663153946399689 (0.03179054448309428) | > loss_mel_ce: 3.087775707244873 (nan) | > loss: 0.01239844225347042 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13757625283339112) | > loader_time: 0.0046 (0.004661342425224101)  --> TIME: 2025-02-26 05:41:04 -- STEP: 1610/36938 -- GLOBAL_STEP: 1109750 | > loss_text_ce: 0.03278695046901703 (0.03185262207797966) | > loss_mel_ce: 3.556640625 (nan) | > loss: 0.014243761077523232 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13768549157966034) | > loader_time: 0.0044 (0.004659608876483032)  --> TIME: 2025-02-26 05:41:14 -- STEP: 1660/36938 -- GLOBAL_STEP: 1109800 | > loss_text_ce: 0.03154474496841431 (0.031829692016591346) | > loss_mel_ce: 2.5618093013763428 (nan) | > loss: 0.010291087441146374 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13775220721601028) | > loader_time: 0.0045 (0.00465749372918921)  --> TIME: 2025-02-26 05:41:23 -- STEP: 1710/36938 -- GLOBAL_STEP: 1109850 | > loss_text_ce: 0.03509247303009033 (0.03184218037252628) | > loss_mel_ce: 4.589285850524902 (nan) | > loss: 0.01835070736706257 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13766714062607102) | > loader_time: 0.0047 (0.004655009124711236)  --> TIME: 2025-02-26 05:41:32 -- STEP: 1760/36938 -- GLOBAL_STEP: 1109900 | > loss_text_ce: 0.03523730859160423 (0.03182681468156678) | > loss_mel_ce: 2.830150604248047 (nan) | > loss: 0.011370587162673473 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.13776505819775858) | > loader_time: 0.0044 (0.004652141847393723)  --> TIME: 2025-02-26 05:41:42 -- STEP: 1810/36938 -- GLOBAL_STEP: 1109950 | > loss_text_ce: 0.028868453577160835 (0.03183131225005785) | > loss_mel_ce: 2.8001697063446045 (nan) | > loss: 0.011226342059671879 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13783593190967716) | > loader_time: 0.0044 (0.004649435617647113)  --> TIME: 2025-02-26 05:41:51 -- STEP: 1860/36938 -- GLOBAL_STEP: 1110000 | > loss_text_ce: 0.03339315205812454 (0.031802985554320734) | > loss_mel_ce: 3.1300036907196045 (nan) | > loss: 0.01255316287279129 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.13785977517404863) | > loader_time: 0.0046 (0.004647164447333217) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1110000.pth  --> TIME: 2025-02-26 05:42:07 -- STEP: 1910/36938 -- GLOBAL_STEP: 1110050 | > loss_text_ce: 0.035515040159225464 (0.031819757463225275) | > loss_mel_ce: 2.2585740089416504 (nan) | > loss: 0.00910352822393179 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.1378092345142864) | > loader_time: 0.0046 (0.004647501850627474)  --> TIME: 2025-02-26 05:42:16 -- STEP: 1960/36938 -- GLOBAL_STEP: 1110100 | > loss_text_ce: 0.023859135806560516 (0.03183302794469106) | > loss_mel_ce: 0.002452254295349121 (nan) | > loss: 0.00010441028280183673 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1377649527423236) | > loader_time: 0.0045 (0.0046477660840871324)  --> TIME: 2025-02-26 05:42:25 -- STEP: 2010/36938 -- GLOBAL_STEP: 1110150 | > loss_text_ce: 0.034900736063718796 (0.031820676473909994) | > loss_mel_ce: 3.1011176109313965 (nan) | > loss: 0.012444517575204372 (nan) | > current_lr: 5e-06 | > step_time: 0.1782 (0.137684762062718) | > loader_time: 0.0048 (0.004646901942011131)  --> TIME: 2025-02-26 05:42:34 -- STEP: 2060/36938 -- GLOBAL_STEP: 1110200 | > loss_text_ce: 0.03024410828948021 (0.03181725491145721) | > loss_mel_ce: 2.8983981609344482 (nan) | > loss: 0.011621597222983837 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13759037661320955) | > loader_time: 0.0047 (0.004646889908799842)  --> TIME: 2025-02-26 05:42:44 -- STEP: 2110/36938 -- GLOBAL_STEP: 1110250 | > loss_text_ce: 0.033709049224853516 (0.03181442382670413) | > loss_mel_ce: 3.0030438899993896 (nan) | > loss: 0.012050607241690159 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13756248758867456) | > loader_time: 0.0045 (0.004646017427128068)  --> TIME: 2025-02-26 05:42:53 -- STEP: 2160/36938 -- GLOBAL_STEP: 1110300 | > loss_text_ce: 0.032557301223278046 (0.03182856819885615) | > loss_mel_ce: 2.9570443630218506 (nan) | > loss: 0.011863498948514462 (nan) | > current_lr: 5e-06 | > step_time: 0.1648 (0.13767492495201256) | > loader_time: 0.0048 (0.004645815933192212)  --> TIME: 2025-02-26 05:43:03 -- STEP: 2210/36938 -- GLOBAL_STEP: 1110350 | > loss_text_ce: 0.02957214042544365 (0.03183293986593576) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.1376953826231115) | > loader_time: 0.0046 (0.004645737048187939)  --> TIME: 2025-02-26 05:43:12 -- STEP: 2260/36938 -- GLOBAL_STEP: 1110400 | > loss_text_ce: 0.025016631931066513 (0.03184514092116628) | > loss_mel_ce: 2.7069904804229736 (nan) | > loss: 0.010841297917068005 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13763427702726524) | > loader_time: 0.0046 (0.004644904516439515)  --> TIME: 2025-02-26 05:43:21 -- STEP: 2310/36938 -- GLOBAL_STEP: 1110450 | > loss_text_ce: 0.035962723195552826 (0.031853918205479025) | > loss_mel_ce: 1.9720362424850464 (nan) | > loss: 0.007968250662088394 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13760855301118002) | > loader_time: 0.0046 (0.004645002971995956)  --> TIME: 2025-02-26 05:43:31 -- STEP: 2360/36938 -- GLOBAL_STEP: 1110500 | > loss_text_ce: 0.028232788667082787 (0.031836865276445504) | > loss_mel_ce: 1.4178804159164429 (nan) | > loss: 0.0057385447435081005 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13760565701177568) | > loader_time: 0.0046 (0.004644734374547404)  --> TIME: 2025-02-26 05:43:40 -- STEP: 2410/36938 -- GLOBAL_STEP: 1110550 | > loss_text_ce: 0.029436206445097923 (0.031834205854452235) | > loss_mel_ce: 3.259777784347534 (nan) | > loss: 0.013052436523139477 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.1376118614465864) | > loader_time: 0.0046 (0.0046445294534516995)  --> TIME: 2025-02-26 05:43:50 -- STEP: 2460/36938 -- GLOBAL_STEP: 1110600 | > loss_text_ce: 0.022273344919085503 (0.03185188219678113) | > loss_mel_ce: 1.8023681640625 (nan) | > loss: 0.007240640930831432 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13763256324985168) | > loader_time: 0.0045 (0.004644519236029645)  --> TIME: 2025-02-26 05:43:59 -- STEP: 2510/36938 -- GLOBAL_STEP: 1110650 | > loss_text_ce: 0.029107026755809784 (0.031859527272636086) | > loss_mel_ce: 0.16864013671875 (nan) | > loss: 0.000784711039159447 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.13765217194044263) | > loader_time: 0.0046 (0.004644588740223433)  --> TIME: 2025-02-26 05:44:08 -- STEP: 2560/36938 -- GLOBAL_STEP: 1110700 | > loss_text_ce: 0.03354283794760704 (0.03184172663022763) | > loss_mel_ce: 2.5988214015960693 (nan) | > loss: 0.010445890948176384 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.13760598339140429) | > loader_time: 0.0048 (0.004644520394504071)  --> TIME: 2025-02-26 05:44:18 -- STEP: 2610/36938 -- GLOBAL_STEP: 1110750 | > loss_text_ce: 0.03718914836645126 (0.03187814467797106) | > loss_mel_ce: 3.1979103088378906 (nan) | > loss: 0.012837697751820087 (nan) | > current_lr: 5e-06 | > step_time: 0.1851 (0.1376419947065158) | > loader_time: 0.0049 (0.004644203368731387)  --> TIME: 2025-02-26 05:44:27 -- STEP: 2660/36938 -- GLOBAL_STEP: 1110800 | > loss_text_ce: 0.026815464720129967 (0.03189304198761633) | > loss_mel_ce: 2.7519712448120117 (nan) | > loss: 0.011026931926608086 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1377107734966996) | > loader_time: 0.0045 (0.004644930631594551)  --> TIME: 2025-02-26 05:44:37 -- STEP: 2710/36938 -- GLOBAL_STEP: 1110850 | > loss_text_ce: 0.034387968480587006 (0.031878840429124145) | > loss_mel_ce: 3.182929039001465 (nan) | > loss: 0.012767132371664047 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.13775617838785661) | > loader_time: 0.0049 (0.004645133194448323)  --> TIME: 2025-02-26 05:44:46 -- STEP: 2760/36938 -- GLOBAL_STEP: 1110900 | > loss_text_ce: 0.035099152475595474 (0.03188249898831481) | > loss_mel_ce: 2.8454017639160156 (nan) | > loss: 0.011430560611188412 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13776564053867177) | > loader_time: 0.0046 (0.004645150727119996)  --> TIME: 2025-02-26 05:44:56 -- STEP: 2810/36938 -- GLOBAL_STEP: 1110950 | > loss_text_ce: 0.032927174121141434 (0.031876199478714354) | > loss_mel_ce: 2.80517578125 (nan) | > loss: 0.011262314394116402 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13783105073026067) | > loader_time: 0.0048 (0.004645184180914721)  --> TIME: 2025-02-26 05:45:06 -- STEP: 2860/36938 -- GLOBAL_STEP: 1111000 | > loss_text_ce: 0.029077952727675438 (0.03188382842239327) | > loss_mel_ce: 2.2303428649902344 (nan) | > loss: 0.008965956047177315 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13790155640848867) | > loader_time: 0.0046 (0.004646761017245846)  --> TIME: 2025-02-26 05:45:15 -- STEP: 2910/36938 -- GLOBAL_STEP: 1111050 | > loss_text_ce: 0.03820345178246498 (0.031882790180684864) | > loss_mel_ce: 1.8437069654464722 (nan) | > loss: 0.00746789900586009 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13783808917933715) | > loader_time: 0.0046 (0.004646907557326901)  --> TIME: 2025-02-26 05:45:25 -- STEP: 2960/36938 -- GLOBAL_STEP: 1111100 | > loss_text_ce: 0.03146378695964813 (0.03189824990254851) | > loss_mel_ce: 2.507641077041626 (nan) | > loss: 0.010075814090669155 (nan) | > current_lr: 5e-06 | > step_time: 0.181 (0.13795092524708927) | > loader_time: 0.0048 (0.004649212795334889)  --> TIME: 2025-02-26 05:45:34 -- STEP: 3010/36938 -- GLOBAL_STEP: 1111150 | > loss_text_ce: 0.036642950028181076 (0.03187696354469884) | > loss_mel_ce: 3.198136329650879 (nan) | > loss: 0.012836426496505737 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.13795933913550903) | > loader_time: 0.0047 (0.004648673890833043)  --> TIME: 2025-02-26 05:45:44 -- STEP: 3060/36938 -- GLOBAL_STEP: 1111200 | > loss_text_ce: 0.03999489173293114 (0.031888698902760235) | > loss_mel_ce: 2.687582492828369 (nan) | > loss: 0.010823721066117287 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.13799565726635513) | > loader_time: 0.0048 (0.004649042537788932)  --> TIME: 2025-02-26 05:45:53 -- STEP: 3110/36938 -- GLOBAL_STEP: 1111250 | > loss_text_ce: 0.033592358231544495 (0.0318835289319348) | > loss_mel_ce: 2.240760087966919 (nan) | > loss: 0.0090252086520195 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1379706852689051) | > loader_time: 0.0045 (0.004648299539204188)  --> TIME: 2025-02-26 05:46:03 -- STEP: 3160/36938 -- GLOBAL_STEP: 1111300 | > loss_text_ce: 0.028894374147057533 (0.031900221619554595) | > loss_mel_ce: 3.0741965770721436 (nan) | > loss: 0.012313853949308395 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.1379882436764392) | > loader_time: 0.0046 (0.004647599669951419)  --> TIME: 2025-02-26 05:46:12 -- STEP: 3210/36938 -- GLOBAL_STEP: 1111350 | > loss_text_ce: 0.029601458460092545 (0.031908886205706734) | > loss_mel_ce: 2.6996078491210938 (nan) | > loss: 0.010830196551978588 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13792477263097078) | > loader_time: 0.0045 (0.004647586130278867)  --> TIME: 2025-02-26 05:46:21 -- STEP: 3260/36938 -- GLOBAL_STEP: 1111400 | > loss_text_ce: 0.028267255052924156 (0.03189859526557561) | > loss_mel_ce: 2.6802427768707275 (nan) | > loss: 0.010748055763542652 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.13787484439604117) | > loader_time: 0.0046 (0.004647458842926951)  --> TIME: 2025-02-26 05:46:30 -- STEP: 3310/36938 -- GLOBAL_STEP: 1111450 | > loss_text_ce: 0.03126518055796623 (0.03190397726022988) | > loss_mel_ce: 2.5927698612213135 (nan) | > loss: 0.01041283831000328 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.1377955465518455) | > loader_time: 0.0046 (0.0046480129852755825)  --> TIME: 2025-02-26 05:46:40 -- STEP: 3360/36938 -- GLOBAL_STEP: 1111500 | > loss_text_ce: 0.03277156502008438 (0.03191696773809823) | > loss_mel_ce: 3.397317409515381 (nan) | > loss: 0.013611464761197567 (nan) | > current_lr: 5e-06 | > step_time: 0.1958 (0.13783154750154122) | > loader_time: 0.0048 (0.004647989571094501)  --> TIME: 2025-02-26 05:46:49 -- STEP: 3410/36938 -- GLOBAL_STEP: 1111550 | > loss_text_ce: 0.035615283995866776 (0.03192526888501041) | > loss_mel_ce: 0.812695324420929 (nan) | > loss: 0.003366312012076378 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13780702055374527) | > loader_time: 0.0047 (0.004647616557012192)  --> TIME: 2025-02-26 05:46:58 -- STEP: 3460/36938 -- GLOBAL_STEP: 1111600 | > loss_text_ce: 0.03623080253601074 (0.031922912531709986) | > loss_mel_ce: 2.2651638984680176 (nan) | > loss: 0.009132519364356995 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13779409669038173) | > loader_time: 0.0045 (0.00464732715849241)  --> TIME: 2025-02-26 05:47:08 -- STEP: 3510/36938 -- GLOBAL_STEP: 1111650 | > loss_text_ce: 0.04631347581744194 (0.03192196675042948) | > loss_mel_ce: 1.448873519897461 (nan) | > loss: 0.005933282431215048 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13775264374550916) | > loader_time: 0.0047 (0.004647093552809481)  --> TIME: 2025-02-26 05:47:17 -- STEP: 3560/36938 -- GLOBAL_STEP: 1111700 | > loss_text_ce: 0.03198551759123802 (0.031922059553005644) | > loss_mel_ce: 3.027979850769043 (nan) | > loss: 0.012142720632255077 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13765327722838738) | > loader_time: 0.0045 (0.004645800858401166)  --> TIME: 2025-02-26 05:47:26 -- STEP: 3610/36938 -- GLOBAL_STEP: 1111750 | > loss_text_ce: 0.02443443238735199 (0.03191381663605159) | > loss_mel_ce: 3.2378714084625244 (nan) | > loss: 0.012945658527314663 (nan) | > current_lr: 5e-06 | > step_time: 0.1794 (0.13768867369834062) | > loader_time: 0.0048 (0.004645437282868686)  --> TIME: 2025-02-26 05:47:35 -- STEP: 3660/36938 -- GLOBAL_STEP: 1111800 | > loss_text_ce: 0.02752033621072769 (0.03190688149357225) | > loss_mel_ce: 2.8810653686523438 (nan) | > loss: 0.011542007327079773 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13769247082413216) | > loader_time: 0.0044 (0.004644638918788042)  --> TIME: 2025-02-26 05:47:45 -- STEP: 3710/36938 -- GLOBAL_STEP: 1111850 | > loss_text_ce: 0.026932047680020332 (0.03191813955866703) | > loss_mel_ce: 2.701298475265503 (nan) | > loss: 0.010826312005519867 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13766276309432365) | > loader_time: 0.0045 (0.0046438601460418154)  --> TIME: 2025-02-26 05:47:54 -- STEP: 3760/36938 -- GLOBAL_STEP: 1111900 | > loss_text_ce: 0.03326649218797684 (0.031941625114500584) | > loss_mel_ce: 2.9079744815826416 (nan) | > loss: 0.011671592481434345 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13757521364283054) | > loader_time: 0.0045 (0.00464265536754688)  --> TIME: 2025-02-26 05:48:03 -- STEP: 3810/36938 -- GLOBAL_STEP: 1111950 | > loss_text_ce: 0.024015966802835464 (0.03192805495000688) | > loss_mel_ce: 2.3214333057403564 (nan) | > loss: 0.00930733885616064 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13759933775804173) | > loader_time: 0.0045 (0.004642143337119583)  --> TIME: 2025-02-26 05:48:12 -- STEP: 3860/36938 -- GLOBAL_STEP: 1112000 | > loss_text_ce: 0.020455416291952133 (0.03192820839965088) | > loss_mel_ce: 2.3317065238952637 (nan) | > loss: 0.009333976544439793 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13757742556883262) | > loader_time: 0.0045 (0.004641412947462)  --> TIME: 2025-02-26 05:48:22 -- STEP: 3910/36938 -- GLOBAL_STEP: 1112050 | > loss_text_ce: 0.03692926838994026 (0.03192448358282522) | > loss_mel_ce: 0.00604248046875 (nan) | > loss: 0.00017052282055374235 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13756806856531012) | > loader_time: 0.0045 (0.004640604650882796)  --> TIME: 2025-02-26 05:48:31 -- STEP: 3960/36938 -- GLOBAL_STEP: 1112100 | > loss_text_ce: 0.023598017171025276 (0.03192070373780834) | > loss_mel_ce: 2.504445791244507 (nan) | > loss: 0.010031919926404953 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13752018875545927) | > loader_time: 0.0048 (0.004640598248953759)  --> TIME: 2025-02-26 05:48:40 -- STEP: 4010/36938 -- GLOBAL_STEP: 1112150 | > loss_text_ce: 0.03187848627567291 (0.03191850087408001) | > loss_mel_ce: 3.1761937141418457 (nan) | > loss: 0.012730445712804794 (nan) | > current_lr: 5e-06 | > step_time: 0.1784 (0.1374732005031329) | > loader_time: 0.0048 (0.004640342410365827)  --> TIME: 2025-02-26 05:48:50 -- STEP: 4060/36938 -- GLOBAL_STEP: 1112200 | > loss_text_ce: 0.03184390813112259 (0.03191042007451755) | > loss_mel_ce: 2.902665138244629 (nan) | > loss: 0.0116448774933815 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13751152818426124) | > loader_time: 0.0046 (0.0046407749500180576)  --> TIME: 2025-02-26 05:48:59 -- STEP: 4110/36938 -- GLOBAL_STEP: 1112250 | > loss_text_ce: 0.03230714052915573 (0.03191249293599212) | > loss_mel_ce: 0.1099853515625 (nan) | > loss: 0.0005646527861244977 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13749184544649148) | > loader_time: 0.0046 (0.004640261274184611)  --> TIME: 2025-02-26 05:49:08 -- STEP: 4160/36938 -- GLOBAL_STEP: 1112300 | > loss_text_ce: 0.03830298036336899 (0.031908175565275476) | > loss_mel_ce: 2.83876895904541 (nan) | > loss: 0.01141695212572813 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.1375288243477161) | > loader_time: 0.0045 (0.004640008107973966)  --> TIME: 2025-02-26 05:49:18 -- STEP: 4210/36938 -- GLOBAL_STEP: 1112350 | > loss_text_ce: 0.03843627870082855 (0.03191193863152217) | > loss_mel_ce: 0.0043487548828125 (nan) | > loss: 0.00016978189523797482 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13749989574142418) | > loader_time: 0.0045 (0.004640006509359533)  --> TIME: 2025-02-26 05:49:27 -- STEP: 4260/36938 -- GLOBAL_STEP: 1112400 | > loss_text_ce: 0.02577412873506546 (0.03190359992342407) | > loss_mel_ce: 1.156698226928711 (nan) | > loss: 0.0046923509798944 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13757747333374384) | > loader_time: 0.0045 (0.004640272973288939)  --> TIME: 2025-02-26 05:49:37 -- STEP: 4310/36938 -- GLOBAL_STEP: 1112450 | > loss_text_ce: 0.03175736963748932 (0.03192720396461484) | > loss_mel_ce: 3.0611441135406494 (nan) | > loss: 0.01227341964840889 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13754650783096276) | > loader_time: 0.0047 (0.0046411390370946345)  --> TIME: 2025-02-26 05:49:46 -- STEP: 4360/36938 -- GLOBAL_STEP: 1112500 | > loss_text_ce: 0.032810330390930176 (0.03192684588069518) | > loss_mel_ce: 2.900524616241455 (nan) | > loss: 0.011640218086540699 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13758314175343295) | > loader_time: 0.0047 (0.004641872042909664)  --> TIME: 2025-02-26 05:49:56 -- STEP: 4410/36938 -- GLOBAL_STEP: 1112550 | > loss_text_ce: 0.03215101733803749 (0.03191063617147046) | > loss_mel_ce: 3.0195651054382324 (nan) | > loss: 0.012109985575079918 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13757874414223395) | > loader_time: 0.0048 (0.004641932749153524)  --> TIME: 2025-02-26 05:50:05 -- STEP: 4460/36938 -- GLOBAL_STEP: 1112600 | > loss_text_ce: 0.02845204435288906 (0.03191273453678816) | > loss_mel_ce: 1.9708226919174194 (nan) | > loss: 0.007933630608022213 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13760158032045236) | > loader_time: 0.0045 (0.004641668037448753)  --> TIME: 2025-02-26 05:50:14 -- STEP: 4510/36938 -- GLOBAL_STEP: 1112650 | > loss_text_ce: 0.03913233429193497 (0.03190393341238506) | > loss_mel_ce: 2.7956912517547607 (nan) | > loss: 0.0112493010237813 (nan) | > current_lr: 5e-06 | > step_time: 0.163 (0.13760214412291677) | > loader_time: 0.0046 (0.004641599718588675)  --> TIME: 2025-02-26 05:50:24 -- STEP: 4560/36938 -- GLOBAL_STEP: 1112700 | > loss_text_ce: 0.03006819449365139 (0.03190963034428195) | > loss_mel_ce: 3.008777379989624 (nan) | > loss: 0.012058911845088005 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.1375776165410092) | > loader_time: 0.0047 (0.004641451072274584)  --> TIME: 2025-02-26 05:50:33 -- STEP: 4610/36938 -- GLOBAL_STEP: 1112750 | > loss_text_ce: 0.04399758204817772 (0.03191786634770104) | > loss_mel_ce: 2.270216226577759 (nan) | > loss: 0.009183388203382492 (nan) | > current_lr: 5e-06 | > step_time: 0.1264 (0.13753727211610878) | > loader_time: 0.0046 (0.004641160323666374)  --> TIME: 2025-02-26 05:50:42 -- STEP: 4660/36938 -- GLOBAL_STEP: 1112800 | > loss_text_ce: 0.017711445689201355 (0.03190199244695477) | > loss_mel_ce: 1.602891206741333 (nan) | > loss: 0.0064309630542993546 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13753079677344401) | > loader_time: 0.0044 (0.004641185144498098)  --> TIME: 2025-02-26 05:50:51 -- STEP: 4710/36938 -- GLOBAL_STEP: 1112850 | > loss_text_ce: 0.033358167856931686 (0.031892124234032315) | > loss_mel_ce: 2.2919921875 (nan) | > loss: 0.009227581322193146 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.1374851448520734) | > loader_time: 0.0046 (0.004640611114015989)  --> TIME: 2025-02-26 05:51:01 -- STEP: 4760/36938 -- GLOBAL_STEP: 1112900 | > loss_text_ce: 0.038688693195581436 (0.031899515761281624) | > loss_mel_ce: 3.176163673400879 (nan) | > loss: 0.012757351621985435 (nan) | > current_lr: 5e-06 | > step_time: 0.1651 (0.1375462493475746) | > loader_time: 0.0046 (0.004640524427429953)  --> TIME: 2025-02-26 05:51:11 -- STEP: 4810/36938 -- GLOBAL_STEP: 1112950 | > loss_text_ce: 0.03274838253855705 (0.03190196490228919) | > loss_mel_ce: 3.2737998962402344 (nan) | > loss: 0.013121224008500576 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.13759124977920748) | > loader_time: 0.0047 (0.00464031086642131)  --> TIME: 2025-02-26 05:51:20 -- STEP: 4860/36938 -- GLOBAL_STEP: 1113000 | > loss_text_ce: 0.03780181333422661 (0.03191566884885599) | > loss_mel_ce: 2.907228946685791 (nan) | > loss: 0.011686630547046661 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.1375505253611278) | > loader_time: 0.0048 (0.004639886926721626)  --> TIME: 2025-02-26 05:51:29 -- STEP: 4910/36938 -- GLOBAL_STEP: 1113050 | > loss_text_ce: 0.029442355036735535 (0.03192693945741218) | > loss_mel_ce: 2.298933982849121 (nan) | > loss: 0.009239588864147663 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13753278381712813) | > loader_time: 0.0045 (0.004639284479642337)  --> TIME: 2025-02-26 05:51:38 -- STEP: 4960/36938 -- GLOBAL_STEP: 1113100 | > loss_text_ce: 0.029006656259298325 (0.031924923283663885) | > loss_mel_ce: 3.2087562084198 (nan) | > loss: 0.012848266400396824 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13755647121898593) | > loader_time: 0.0046 (0.004638984943589844)  --> TIME: 2025-02-26 05:51:48 -- STEP: 5010/36938 -- GLOBAL_STEP: 1113150 | > loss_text_ce: 0.029740605503320694 (0.0319349479119846) | > loss_mel_ce: 3.1360838413238525 (nan) | > loss: 0.012562796473503113 (nan) | > current_lr: 5e-06 | > step_time: 0.1861 (0.13755169371644901) | > loader_time: 0.0048 (0.004638432314295957)  --> TIME: 2025-02-26 05:51:57 -- STEP: 5060/36938 -- GLOBAL_STEP: 1113200 | > loss_text_ce: 0.032805655151605606 (0.031934214535393884) | > loss_mel_ce: 1.581085205078125 (nan) | > loss: 0.006404329091310501 (nan) | > current_lr: 5e-06 | > step_time: 0.1452 (0.13758561573480912) | > loader_time: 0.0046 (0.0046380207472639)  --> TIME: 2025-02-26 05:52:07 -- STEP: 5110/36938 -- GLOBAL_STEP: 1113250 | > loss_text_ce: 0.030464736744761467 (0.03194631395748514) | > loss_mel_ce: 2.7738025188446045 (nan) | > loss: 0.011128044687211514 (nan) | > current_lr: 5e-06 | > step_time: 0.1783 (0.13758348135565604) | > loader_time: 0.0048 (0.004638031504158858)  --> TIME: 2025-02-26 05:52:16 -- STEP: 5160/36938 -- GLOBAL_STEP: 1113300 | > loss_text_ce: 0.02970574051141739 (0.031948962132266015) | > loss_mel_ce: 2.63629150390625 (nan) | > loss: 0.01057935506105423 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13760322586510543) | > loader_time: 0.0046 (0.004637722220531716)  --> TIME: 2025-02-26 05:52:25 -- STEP: 5210/36938 -- GLOBAL_STEP: 1113350 | > loss_text_ce: 0.03732869029045105 (0.03193498657884046) | > loss_mel_ce: 2.814924478530884 (nan) | > loss: 0.011318465694785118 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.1376046839236299) | > loader_time: 0.0045 (0.004637214318346829)  --> TIME: 2025-02-26 05:52:35 -- STEP: 5260/36938 -- GLOBAL_STEP: 1113400 | > loss_text_ce: 0.03765803575515747 (0.03193433498999128) | > loss_mel_ce: 1.9024180173873901 (nan) | > loss: 0.007698715198785067 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.1376377063559037) | > loader_time: 0.0044 (0.004636936314658963)  --> TIME: 2025-02-26 05:52:44 -- STEP: 5310/36938 -- GLOBAL_STEP: 1113450 | > loss_text_ce: 0.02990959770977497 (0.031917663442984724) | > loss_mel_ce: 2.6718413829803467 (nan) | > loss: 0.010721234604716301 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13763476792028412) | > loader_time: 0.0048 (0.00463628490541389)  --> TIME: 2025-02-26 05:52:54 -- STEP: 5360/36938 -- GLOBAL_STEP: 1113500 | > loss_text_ce: 0.031948283314704895 (0.03191213469219103) | > loss_mel_ce: 3.017671585083008 (nan) | > loss: 0.012101667001843452 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.13762969499203706) | > loader_time: 0.0046 (0.0046355043329409625)  --> TIME: 2025-02-26 05:53:03 -- STEP: 5410/36938 -- GLOBAL_STEP: 1113550 | > loss_text_ce: 0.022335747256875038 (0.031913466728335414) | > loss_mel_ce: 0.6987356543540955 (nan) | > loss: 0.0028613947797566652 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13759430151109123) | > loader_time: 0.0044 (0.00463459575462693)  --> TIME: 2025-02-26 05:53:12 -- STEP: 5460/36938 -- GLOBAL_STEP: 1113600 | > loss_text_ce: 0.04066259413957596 (0.031918855490599585) | > loss_mel_ce: 2.8220841884613037 (nan) | > loss: 0.011360106989741325 (nan) | > current_lr: 5e-06 | > step_time: 0.1728 (0.13761176207126727) | > loader_time: 0.0047 (0.004634007997128547)  --> TIME: 2025-02-26 05:53:22 -- STEP: 5510/36938 -- GLOBAL_STEP: 1113650 | > loss_text_ce: 0.035097479820251465 (0.03192428440217284) | > loss_mel_ce: 3.042454242706299 (nan) | > loss: 0.012212508358061314 (nan) | > current_lr: 5e-06 | > step_time: 0.1828 (0.1376654518494373) | > loader_time: 0.0049 (0.004633875204733621)  --> TIME: 2025-02-26 05:53:32 -- STEP: 5560/36938 -- GLOBAL_STEP: 1113700 | > loss_text_ce: 0.04828193411231041 (0.03191910061613391) | > loss_mel_ce: 0.8168321847915649 (nan) | > loss: 0.0034329926129430532 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13768235324955674) | > loader_time: 0.0045 (0.004633346307191918)  --> TIME: 2025-02-26 05:53:41 -- STEP: 5610/36938 -- GLOBAL_STEP: 1113750 | > loss_text_ce: 0.024861551821231842 (0.0319226575469176) | > loss_mel_ce: 2.7951512336730957 (nan) | > loss: 0.01119052805006504 (nan) | > current_lr: 5e-06 | > step_time: 0.1486 (0.13763116455758098) | > loader_time: 0.0045 (0.0046328047380090375)  --> TIME: 2025-02-26 05:53:50 -- STEP: 5660/36938 -- GLOBAL_STEP: 1113800 | > loss_text_ce: 0.03302045911550522 (0.03192697412222484) | > loss_mel_ce: 2.750307321548462 (nan) | > loss: 0.01104495208710432 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.1376264771387347) | > loader_time: 0.0045 (0.0046323469586591415)  --> TIME: 2025-02-26 05:53:59 -- STEP: 5710/36938 -- GLOBAL_STEP: 1113850 | > loss_text_ce: 0.028100065886974335 (0.03192399766821336) | > loss_mel_ce: 2.822784900665283 (nan) | > loss: 0.01131303608417511 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13756592812346044) | > loader_time: 0.0046 (0.00463189018169343)  --> TIME: 2025-02-26 05:54:08 -- STEP: 5760/36938 -- GLOBAL_STEP: 1113900 | > loss_text_ce: 0.03454370051622391 (0.031924264394102495) | > loss_mel_ce: 0.0760091170668602 (nan) | > loss: 0.0004387016815599054 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13746344641678826) | > loader_time: 0.0047 (0.004631468446718321)  --> TIME: 2025-02-26 05:54:17 -- STEP: 5810/36938 -- GLOBAL_STEP: 1113950 | > loss_text_ce: 0.033161114901304245 (0.0319219050401574) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13750034318323134) | > loader_time: 0.0045 (0.004631203710519912)  --> TIME: 2025-02-26 05:54:27 -- STEP: 5860/36938 -- GLOBAL_STEP: 1114000 | > loss_text_ce: 0.031839530915021896 (0.031927255433173485) | > loss_mel_ce: 2.3117165565490723 (nan) | > loss: 0.009299826808273792 (nan) | > current_lr: 5e-06 | > step_time: 0.1522 (0.13754607919133105) | > loader_time: 0.0046 (0.004631158882440551)  --> TIME: 2025-02-26 05:54:36 -- STEP: 5910/36938 -- GLOBAL_STEP: 1114050 | > loss_text_ce: 0.027461158111691475 (0.03192701350736807) | > loss_mel_ce: 2.6760923862457275 (nan) | > loss: 0.010728387162089348 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13751968171592444) | > loader_time: 0.0045 (0.004630646528124598)  --> TIME: 2025-02-26 05:54:46 -- STEP: 5960/36938 -- GLOBAL_STEP: 1114100 | > loss_text_ce: 0.03494872897863388 (0.03192791579034948) | > loss_mel_ce: 2.715833902359009 (nan) | > loss: 0.010915804654359818 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.13752741273617544) | > loader_time: 0.0047 (0.004630550962166486)  --> TIME: 2025-02-26 05:54:55 -- STEP: 6010/36938 -- GLOBAL_STEP: 1114150 | > loss_text_ce: 0.02892431430518627 (0.03192857600168973) | > loss_mel_ce: 2.725515365600586 (nan) | > loss: 0.010930316522717476 (nan) | > current_lr: 5e-06 | > step_time: 0.1452 (0.13750794544791187) | > loader_time: 0.0048 (0.004630927595243266)  --> TIME: 2025-02-26 05:55:04 -- STEP: 6060/36938 -- GLOBAL_STEP: 1114200 | > loss_text_ce: 0.03232865408062935 (0.031913258692114366) | > loss_mel_ce: 1.4450089931488037 (nan) | > loss: 0.005862451158463955 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13749460206173403) | > loader_time: 0.0046 (0.00463099353777692)  --> TIME: 2025-02-26 05:55:14 -- STEP: 6110/36938 -- GLOBAL_STEP: 1114250 | > loss_text_ce: 0.033496104180812836 (0.03192354221424348) | > loss_mel_ce: 3.1585397720336914 (nan) | > loss: 0.012666810303926468 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13751972646447885) | > loader_time: 0.0046 (0.004631097109930397)  --> TIME: 2025-02-26 05:55:23 -- STEP: 6160/36938 -- GLOBAL_STEP: 1114300 | > loss_text_ce: 0.03186941519379616 (0.03192766131727359) | > loss_mel_ce: 0.4189300537109375 (nan) | > loss: 0.0017888868460431695 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.13749257713169208) | > loader_time: 0.0045 (0.004630835025341464)  --> TIME: 2025-02-26 05:55:33 -- STEP: 6210/36938 -- GLOBAL_STEP: 1114350 | > loss_text_ce: 0.030715228989720345 (0.03193085337261668) | > loss_mel_ce: 2.8999950885772705 (nan) | > loss: 0.011629803106188774 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.13749329967007304) | > loader_time: 0.0049 (0.004631369509366765)  --> TIME: 2025-02-26 05:55:42 -- STEP: 6260/36938 -- GLOBAL_STEP: 1114400 | > loss_text_ce: 0.028392668813467026 (0.031931365732871225) | > loss_mel_ce: 2.324801445007324 (nan) | > loss: 0.009338071569800377 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.13750059372319917) | > loader_time: 0.0047 (0.00463093043135376)  --> TIME: 2025-02-26 05:55:51 -- STEP: 6310/36938 -- GLOBAL_STEP: 1114450 | > loss_text_ce: 0.03090728633105755 (0.03192799042426222) | > loss_mel_ce: 2.786588430404663 (nan) | > loss: 0.01118053961545229 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13750482977097778) | > loader_time: 0.0045 (0.004630616047491909)  --> TIME: 2025-02-26 05:56:01 -- STEP: 6360/36938 -- GLOBAL_STEP: 1114500 | > loss_text_ce: 0.025351975113153458 (0.03192598482055007) | > loss_mel_ce: 2.878917932510376 (nan) | > loss: 0.011524881236255169 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13747771573516496) | > loader_time: 0.0045 (0.004630239129816194)  --> TIME: 2025-02-26 05:56:10 -- STEP: 6410/36938 -- GLOBAL_STEP: 1114550 | > loss_text_ce: 0.03022833541035652 (0.03191817857531386) | > loss_mel_ce: 2.3061630725860596 (nan) | > loss: 0.009271395392715931 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13748090709799285) | > loader_time: 0.0046 (0.004629993587499832)  --> TIME: 2025-02-26 05:56:19 -- STEP: 6460/36938 -- GLOBAL_STEP: 1114600 | > loss_text_ce: 0.03417612612247467 (0.03191318345204243) | > loss_mel_ce: 2.7264962196350098 (nan) | > loss: 0.01095504965633154 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.1374663507237148) | > loader_time: 0.0046 (0.004629668473459233)  --> TIME: 2025-02-26 05:56:28 -- STEP: 6510/36938 -- GLOBAL_STEP: 1114650 | > loss_text_ce: 0.03377167880535126 (0.031925250442066185) | > loss_mel_ce: 2.060760498046875 (nan) | > loss: 0.008311636745929718 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13746372375253843) | > loader_time: 0.0045 (0.004629334583077365)  --> TIME: 2025-02-26 05:56:38 -- STEP: 6560/36938 -- GLOBAL_STEP: 1114700 | > loss_text_ce: 0.03560042381286621 (0.031919320333158356) | > loss_mel_ce: 3.1423051357269287 (nan) | > loss: 0.01261073723435402 (nan) | > current_lr: 5e-06 | > step_time: 0.1695 (0.13747857790894577) | > loader_time: 0.0046 (0.004629170204081171)  --> TIME: 2025-02-26 05:56:47 -- STEP: 6610/36938 -- GLOBAL_STEP: 1114750 | > loss_text_ce: 0.03853717818856239 (0.03192251345280787) | > loss_mel_ce: 2.9266273975372314 (nan) | > loss: 0.011766526848077774 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.1374925235396249) | > loader_time: 0.0047 (0.00462901635537889)  --> TIME: 2025-02-26 05:56:57 -- STEP: 6660/36938 -- GLOBAL_STEP: 1114800 | > loss_text_ce: 0.039277222007513046 (0.0319238681764293) | > loss_mel_ce: 2.896655559539795 (nan) | > loss: 0.011650527827441692 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.13748564405126204) | > loader_time: 0.0051 (0.004628841690830978)  --> TIME: 2025-02-26 05:57:06 -- STEP: 6710/36938 -- GLOBAL_STEP: 1114850 | > loss_text_ce: 0.029758300632238388 (0.03193036164663681) | > loss_mel_ce: 2.737581968307495 (nan) | > loss: 0.01098150946199894 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13749993447930495) | > loader_time: 0.0045 (0.0046285079595051095)  --> TIME: 2025-02-26 05:57:16 -- STEP: 6760/36938 -- GLOBAL_STEP: 1114900 | > loss_text_ce: 0.0344279482960701 (0.03192432448860328) | > loss_mel_ce: 3.2859864234924316 (nan) | > loss: 0.013176248408854008 (nan) | > current_lr: 5e-06 | > step_time: 0.1951 (0.13752871328556973) | > loader_time: 0.0049 (0.004628409083778316)  --> TIME: 2025-02-26 05:57:25 -- STEP: 6810/36938 -- GLOBAL_STEP: 1114950 | > loss_text_ce: 0.03242753818631172 (0.031916601009387736) | > loss_mel_ce: 2.618361234664917 (nan) | > loss: 0.010519003495573997 (nan) | > current_lr: 5e-06 | > step_time: 0.1869 (0.13756390692729026) | > loader_time: 0.0048 (0.0046282508124688765)  --> TIME: 2025-02-26 05:57:35 -- STEP: 6860/36938 -- GLOBAL_STEP: 1115000 | > loss_text_ce: 0.030960284173488617 (0.031908318265028775) | > loss_mel_ce: 1.4408305883407593 (nan) | > loss: 0.005840440280735493 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13756985122191256) | > loader_time: 0.0045 (0.0046280151553473505)  --> TIME: 2025-02-26 05:57:44 -- STEP: 6910/36938 -- GLOBAL_STEP: 1115050 | > loss_text_ce: 0.03127502277493477 (0.03191446426988988) | > loss_mel_ce: 2.2894694805145264 (nan) | > loss: 0.009209304116666317 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.137542679720435) | > loader_time: 0.0045 (0.00462765493544757)  --> TIME: 2025-02-26 05:57:53 -- STEP: 6960/36938 -- GLOBAL_STEP: 1115100 | > loss_text_ce: 0.034826744347810745 (0.03190858633198552) | > loss_mel_ce: 2.4969730377197266 (nan) | > loss: 0.010046824812889099 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1375423102543265) | > loader_time: 0.0046 (0.004627407385014917)  --> TIME: 2025-02-26 05:58:03 -- STEP: 7010/36938 -- GLOBAL_STEP: 1115150 | > loss_text_ce: 0.03262939304113388 (0.03191544758833486) | > loss_mel_ce: 3.1633434295654297 (nan) | > loss: 0.012682433240115643 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13753919489203376) | > loader_time: 0.0045 (0.004627055346370572)  --> TIME: 2025-02-26 05:58:12 -- STEP: 7060/36938 -- GLOBAL_STEP: 1115200 | > loss_text_ce: 0.034237898886203766 (0.03191562303956631) | > loss_mel_ce: 0.09527587890625 (nan) | > loss: 0.0005139435525052249 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13748412831309084) | > loader_time: 0.0045 (0.004626446979241714)  --> TIME: 2025-02-26 05:58:21 -- STEP: 7110/36938 -- GLOBAL_STEP: 1115250 | > loss_text_ce: 0.03038983978331089 (0.031923404330473915) | > loss_mel_ce: 2.854071617126465 (nan) | > loss: 0.011446275748312473 (nan) | > current_lr: 5e-06 | > step_time: 0.1773 (0.1375141570336701) | > loader_time: 0.005 (0.0046265542758667805)  --> TIME: 2025-02-26 05:58:31 -- STEP: 7160/36938 -- GLOBAL_STEP: 1115300 | > loss_text_ce: 0.03523532673716545 (0.03192916669565431) | > loss_mel_ce: 2.935575246810913 (nan) | > loss: 0.011788931675255299 (nan) | > current_lr: 5e-06 | > step_time: 0.1691 (0.13752722134137216) | > loader_time: 0.0048 (0.004626485455635518)  --> TIME: 2025-02-26 05:58:40 -- STEP: 7210/36938 -- GLOBAL_STEP: 1115350 | > loss_text_ce: 0.039482951164245605 (0.03192937596133999) | > loss_mel_ce: 3.084609270095825 (nan) | > loss: 0.012397191487252712 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13748139700181866) | > loader_time: 0.0045 (0.004626320172282077)  --> TIME: 2025-02-26 05:58:49 -- STEP: 7260/36938 -- GLOBAL_STEP: 1115400 | > loss_text_ce: 0.027788260951638222 (0.03192170723482488) | > loss_mel_ce: 3.0068137645721436 (nan) | > loss: 0.012042071670293808 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.13749694705994583) | > loader_time: 0.0046 (0.004626152338075222)  --> TIME: 2025-02-26 05:58:59 -- STEP: 7310/36938 -- GLOBAL_STEP: 1115450 | > loss_text_ce: 0.03128957748413086 (0.03192773895559837) | > loss_mel_ce: 3.0908515453338623 (nan) | > loss: 0.012389449402689934 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13748763269667014) | > loader_time: 0.0046 (0.004625903695780972)  --> TIME: 2025-02-26 05:59:08 -- STEP: 7360/36938 -- GLOBAL_STEP: 1115500 | > loss_text_ce: 0.03593696653842926 (0.03193421917132319) | > loss_mel_ce: 3.457162380218506 (nan) | > loss: 0.01386150624603033 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.13747535778776387) | > loader_time: 0.0048 (0.004625889302595781)  --> TIME: 2025-02-26 05:59:17 -- STEP: 7410/36938 -- GLOBAL_STEP: 1115550 | > loss_text_ce: 0.026400107890367508 (0.03193848606339766) | > loss_mel_ce: 2.271843194961548 (nan) | > loss: 0.009120013564825058 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13749484620113525) | > loader_time: 0.0045 (0.004626203419869385)  --> TIME: 2025-02-26 05:59:27 -- STEP: 7460/36938 -- GLOBAL_STEP: 1115600 | > loss_text_ce: 0.027634527534246445 (0.03193938158185322) | > loss_mel_ce: 2.5589559078216553 (nan) | > loss: 0.010264248587191105 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13749188766402703) | > loader_time: 0.0044 (0.004625937574350781)  --> TIME: 2025-02-26 05:59:36 -- STEP: 7510/36938 -- GLOBAL_STEP: 1115650 | > loss_text_ce: 0.034745682030916214 (0.03193150856476323) | > loss_mel_ce: 1.959716796875 (nan) | > loss: 0.007914533838629723 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13746762488399375) | > loader_time: 0.0045 (0.004625448056765763)  --> TIME: 2025-02-26 05:59:45 -- STEP: 7560/36938 -- GLOBAL_STEP: 1115700 | > loss_text_ce: 0.03875373676419258 (0.031928722465918116) | > loss_mel_ce: 2.073107957839966 (nan) | > loss: 0.008380403742194176 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13746703304941607) | > loader_time: 0.0047 (0.004625532677564651)  --> TIME: 2025-02-26 05:59:55 -- STEP: 7610/36938 -- GLOBAL_STEP: 1115750 | > loss_text_ce: 0.029413534328341484 (0.03192801853964929) | > loss_mel_ce: 2.9639406204223633 (nan) | > loss: 0.011878389865159988 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13746096705638386) | > loader_time: 0.0045 (0.004625696609587279)  --> TIME: 2025-02-26 06:00:04 -- STEP: 7660/36938 -- GLOBAL_STEP: 1115800 | > loss_text_ce: 0.030698806047439575 (0.03193252678904609) | > loss_mel_ce: 2.111846446990967 (nan) | > loss: 0.008502163924276829 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.137463812572837) | > loader_time: 0.0046 (0.004625815666686752)  --> TIME: 2025-02-26 06:00:13 -- STEP: 7710/36938 -- GLOBAL_STEP: 1115850 | > loss_text_ce: 0.034780990332365036 (0.03193001746152893) | > loss_mel_ce: 2.75423002243042 (nan) | > loss: 0.011067504994571209 (nan) | > current_lr: 5e-06 | > step_time: 0.1569 (0.13745636052503887) | > loader_time: 0.0046 (0.004625308034330315)  --> TIME: 2025-02-26 06:00:23 -- STEP: 7760/36938 -- GLOBAL_STEP: 1115900 | > loss_text_ce: 0.0372912622988224 (0.03192904808104843) | > loss_mel_ce: 3.203645706176758 (nan) | > loss: 0.012860861606895924 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.1374582023657466) | > loader_time: 0.0046 (0.004625067268450215)  --> TIME: 2025-02-26 06:00:32 -- STEP: 7810/36938 -- GLOBAL_STEP: 1115950 | > loss_text_ce: 0.03088511899113655 (0.03192837734536155) | > loss_mel_ce: 2.8068270683288574 (nan) | > loss: 0.011260763742029667 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.13747251720648182) | > loader_time: 0.0045 (0.0046247630052163365)  --> TIME: 2025-02-26 06:00:41 -- STEP: 7860/36938 -- GLOBAL_STEP: 1116000 | > loss_text_ce: 0.0348740890622139 (0.03193319153204141) | > loss_mel_ce: 1.3843200206756592 (nan) | > loss: 0.00563172297552228 (nan) | > current_lr: 5e-06 | > step_time: 0.1436 (0.13744064878871343) | > loader_time: 0.0045 (0.00462435826697117)  --> TIME: 2025-02-26 06:00:50 -- STEP: 7910/36938 -- GLOBAL_STEP: 1116050 | > loss_text_ce: 0.03492980822920799 (0.031935153616293004) | > loss_mel_ce: 2.810457468032837 (nan) | > loss: 0.011291219852864742 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.1374422630076153) | > loader_time: 0.0047 (0.0046240505768285044)  --> TIME: 2025-02-26 06:01:00 -- STEP: 7960/36938 -- GLOBAL_STEP: 1116100 | > loss_text_ce: 0.027795258909463882 (0.031936478117204535) | > loss_mel_ce: 2.4725279808044434 (nan) | > loss: 0.009921918623149395 (nan) | > current_lr: 5e-06 | > step_time: 0.1856 (0.13744271870833466) | > loader_time: 0.0048 (0.0046237537309751844)  --> TIME: 2025-02-26 06:01:09 -- STEP: 8010/36938 -- GLOBAL_STEP: 1116150 | > loss_text_ce: 0.030986163765192032 (0.03193187989543326) | > loss_mel_ce: 3.0947678089141846 (nan) | > loss: 0.012403786182403564 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13742252527253526) | > loader_time: 0.0048 (0.0046234056446584825)  --> TIME: 2025-02-26 06:01:18 -- STEP: 8060/36938 -- GLOBAL_STEP: 1116200 | > loss_text_ce: 0.04449159651994705 (0.03193607658492127) | > loss_mel_ce: 3.3520233631134033 (nan) | > loss: 0.01347823441028595 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13743127912800523) | > loader_time: 0.0046 (0.0046231084071081075)  --> TIME: 2025-02-26 06:01:28 -- STEP: 8110/36938 -- GLOBAL_STEP: 1116250 | > loss_text_ce: 0.029624737799167633 (0.03193076252167894) | > loss_mel_ce: 3.345120906829834 (nan) | > loss: 0.013391848653554916 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.13744146953234612) | > loader_time: 0.0046 (0.004622828681136746)  --> TIME: 2025-02-26 06:01:37 -- STEP: 8160/36938 -- GLOBAL_STEP: 1116300 | > loss_text_ce: 0.03608175367116928 (0.03192520399233215) | > loss_mel_ce: 3.0735573768615723 (nan) | > loss: 0.012339838780462742 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.137440934017592) | > loader_time: 0.0045 (0.00462236831001205)  --> TIME: 2025-02-26 06:01:46 -- STEP: 8210/36938 -- GLOBAL_STEP: 1116350 | > loss_text_ce: 0.03328799456357956 (0.0319299972772678) | > loss_mel_ce: 3.180182456970215 (nan) | > loss: 0.0127518679946661 (nan) | > current_lr: 5e-06 | > step_time: 0.1261 (0.1374285573936111) | > loader_time: 0.0045 (0.0046219849557446605)  --> TIME: 2025-02-26 06:01:56 -- STEP: 8260/36938 -- GLOBAL_STEP: 1116400 | > loss_text_ce: 0.047889456152915955 (0.03193750446470611) | > loss_mel_ce: 2.6163721084594727 (nan) | > loss: 0.010572466999292374 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13742935726775307) | > loader_time: 0.0046 (0.0046216186830552765)  --> TIME: 2025-02-26 06:02:05 -- STEP: 8310/36938 -- GLOBAL_STEP: 1116450 | > loss_text_ce: 0.025033393874764442 (0.03194038406424609) | > loss_mel_ce: 3.2879464626312256 (nan) | > loss: 0.013146746903657913 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13744367792861625) | > loader_time: 0.0045 (0.004622026384952678)  --> TIME: 2025-02-26 06:02:14 -- STEP: 8360/36938 -- GLOBAL_STEP: 1116500 | > loss_text_ce: 0.035536278039216995 (0.03194072847911829) | > loss_mel_ce: 2.538790225982666 (nan) | > loss: 0.010215582326054573 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13742529638646248) | > loader_time: 0.0046 (0.004621798056734788)  --> TIME: 2025-02-26 06:02:23 -- STEP: 8410/36938 -- GLOBAL_STEP: 1116550 | > loss_text_ce: 0.022976113483309746 (0.031924988297849496) | > loss_mel_ce: 1.9019291400909424 (nan) | > loss: 0.0076385135762393475 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13740388859466401) | > loader_time: 0.0045 (0.00462151446892445)  --> TIME: 2025-02-26 06:02:33 -- STEP: 8460/36938 -- GLOBAL_STEP: 1116600 | > loss_text_ce: 0.024608159437775612 (0.031925573354877425) | > loss_mel_ce: 2.4216628074645996 (nan) | > loss: 0.00970742478966713 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.1373835967223969) | > loader_time: 0.005 (0.004621753568626727)  --> TIME: 2025-02-26 06:02:42 -- STEP: 8510/36938 -- GLOBAL_STEP: 1116650 | > loss_text_ce: 0.031785257160663605 (0.031924768658650196) | > loss_mel_ce: 3.2229840755462646 (nan) | > loss: 0.01291575189679861 (nan) | > current_lr: 5e-06 | > step_time: 0.1817 (0.13738529858661933) | > loader_time: 0.0049 (0.004621872498481991)  --> TIME: 2025-02-26 06:02:51 -- STEP: 8560/36938 -- GLOBAL_STEP: 1116700 | > loss_text_ce: 0.034357596188783646 (0.031918927294606714) | > loss_mel_ce: 2.9506044387817383 (nan) | > loss: 0.011845087632536888 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.1373689117275658) | > loader_time: 0.0045 (0.004621674774963146)  --> TIME: 2025-02-26 06:03:00 -- STEP: 8610/36938 -- GLOBAL_STEP: 1116750 | > loss_text_ce: 0.020191319286823273 (0.03192271385633919) | > loss_mel_ce: 0.0332539863884449 (nan) | > loss: 0.00021208456018939614 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13735278009952903) | > loader_time: 0.0046 (0.004621699435093384)  --> TIME: 2025-02-26 06:03:10 -- STEP: 8660/36938 -- GLOBAL_STEP: 1116800 | > loss_text_ce: 0.023456314578652382 (0.031917919726035554) | > loss_mel_ce: 0.1279296875 (nan) | > loss: 0.0006007381598465145 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1373608932208787) | > loader_time: 0.0046 (0.004621672575248015)  --> TIME: 2025-02-26 06:03:19 -- STEP: 8710/36938 -- GLOBAL_STEP: 1116850 | > loss_text_ce: 0.031724002212285995 (0.031920360995030865) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13734884631619992) | > loader_time: 0.0045 (0.004621824687165028)  --> TIME: 2025-02-26 06:03:28 -- STEP: 8760/36938 -- GLOBAL_STEP: 1116900 | > loss_text_ce: 0.027899106964468956 (0.03191585440505451) | > loss_mel_ce: 2.731205701828003 (nan) | > loss: 0.010948829352855682 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13734900431545813) | > loader_time: 0.0045 (0.00462163615988813)  --> TIME: 2025-02-26 06:03:38 -- STEP: 8810/36938 -- GLOBAL_STEP: 1116950 | > loss_text_ce: 0.022000648081302643 (0.03190229001236168) | > loss_mel_ce: 2.647209644317627 (nan) | > loss: 0.010592104867100716 (nan) | > current_lr: 5e-06 | > step_time: 0.156 (0.1373425834159972) | > loader_time: 0.0046 (0.004621259335356583)  --> TIME: 2025-02-26 06:03:47 -- STEP: 8860/36938 -- GLOBAL_STEP: 1117000 | > loss_text_ce: 0.02787008509039879 (0.03190193805217087) | > loss_mel_ce: 3.190476417541504 (nan) | > loss: 0.01277121715247631 (nan) | > current_lr: 5e-06 | > step_time: 0.1724 (0.13733706008768962) | > loader_time: 0.0047 (0.004621232686139651)  --> TIME: 2025-02-26 06:03:56 -- STEP: 8910/36938 -- GLOBAL_STEP: 1117050 | > loss_text_ce: 0.03284360095858574 (0.03190092677292297) | > loss_mel_ce: 2.6213724613189697 (nan) | > loss: 0.010532603599131107 (nan) | > current_lr: 5e-06 | > step_time: 0.1445 (0.1373062748432685) | > loader_time: 0.0048 (0.004621347299997509)  --> TIME: 2025-02-26 06:04:05 -- STEP: 8960/36938 -- GLOBAL_STEP: 1117100 | > loss_text_ce: 0.02493792399764061 (0.03189429768566938) | > loss_mel_ce: 2.5937693119049072 (nan) | > loss: 0.01039169542491436 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13729491316314155) | > loader_time: 0.0046 (0.004621286317706097)  --> TIME: 2025-02-26 06:04:15 -- STEP: 9010/36938 -- GLOBAL_STEP: 1117150 | > loss_text_ce: 0.03796212747693062 (0.0318829729734452) | > loss_mel_ce: 3.0629537105560303 (nan) | > loss: 0.012305222451686859 (nan) | > current_lr: 5e-06 | > step_time: 0.1491 (0.13730736914009056) | > loader_time: 0.0046 (0.0046213859458609915)  --> TIME: 2025-02-26 06:04:24 -- STEP: 9060/36938 -- GLOBAL_STEP: 1117200 | > loss_text_ce: 0.030144499614834785 (0.03188369275074821) | > loss_mel_ce: 2.7516374588012695 (nan) | > loss: 0.011038817465305328 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.1373123761570733) | > loader_time: 0.0046 (0.0046214178961107395)  --> TIME: 2025-02-26 06:04:34 -- STEP: 9110/36938 -- GLOBAL_STEP: 1117250 | > loss_text_ce: 0.04443088546395302 (0.031888281078916975) | > loss_mel_ce: 2.5228092670440674 (nan) | > loss: 0.010187461972236633 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.1373409959016322) | > loader_time: 0.0045 (0.004621327355193239)  --> TIME: 2025-02-26 06:04:44 -- STEP: 9160/36938 -- GLOBAL_STEP: 1117300 | > loss_text_ce: 0.028245069086551666 (0.031892919821733624) | > loss_mel_ce: 2.280703067779541 (nan) | > loss: 0.00916249305009842 (nan) | > current_lr: 5e-06 | > step_time: 0.1438 (0.1373630869336513) | > loader_time: 0.0046 (0.004621157557683206)  --> TIME: 2025-02-26 06:04:53 -- STEP: 9210/36938 -- GLOBAL_STEP: 1117350 | > loss_text_ce: 0.03090519644320011 (0.03189294649142427) | > loss_mel_ce: 2.3813283443450928 (nan) | > loss: 0.009572356007993221 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13735185707560213) | > loader_time: 0.0045 (0.004621056754477751)  --> TIME: 2025-02-26 06:05:02 -- STEP: 9260/36938 -- GLOBAL_STEP: 1117400 | > loss_text_ce: 0.024543480947613716 (0.031892955009413) | > loss_mel_ce: 2.01717209815979 (nan) | > loss: 0.008102046325802803 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13734097122888483) | > loader_time: 0.0045 (0.004620824158835345)  --> TIME: 2025-02-26 06:05:11 -- STEP: 9310/36938 -- GLOBAL_STEP: 1117450 | > loss_text_ce: 0.03429669141769409 (0.03189022939414107) | > loss_mel_ce: 2.323637008666992 (nan) | > loss: 0.009356880560517311 (nan) | > current_lr: 5e-06 | > step_time: 0.1634 (0.13733210246109578) | > loader_time: 0.0046 (0.0046205712696407155)  --> TIME: 2025-02-26 06:05:21 -- STEP: 9360/36938 -- GLOBAL_STEP: 1117500 | > loss_text_ce: 0.02763683721423149 (0.031895107374442844) | > loss_mel_ce: 2.830312490463257 (nan) | > loss: 0.011341068893671036 (nan) | > current_lr: 5e-06 | > step_time: 0.1482 (0.1373206035703665) | > loader_time: 0.0048 (0.0046206830403743475)  --> TIME: 2025-02-26 06:05:30 -- STEP: 9410/36938 -- GLOBAL_STEP: 1117550 | > loss_text_ce: 0.033751118928194046 (0.031894750556117644) | > loss_mel_ce: 2.7064692974090576 (nan) | > loss: 0.010873890481889248 (nan) | > current_lr: 5e-06 | > step_time: 0.1798 (0.13735070540218272) | > loader_time: 0.0047 (0.004620879109430248)  --> TIME: 2025-02-26 06:05:39 -- STEP: 9460/36938 -- GLOBAL_STEP: 1117600 | > loss_text_ce: 0.033945005387067795 (0.031893357020718) | > loss_mel_ce: 2.2493736743927 (nan) | > loss: 0.00906078889966011 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.1373396616657481) | > loader_time: 0.0045 (0.0046206234121423315)  --> TIME: 2025-02-26 06:05:49 -- STEP: 9510/36938 -- GLOBAL_STEP: 1117650 | > loss_text_ce: 0.01479199156165123 (0.03189333592023449) | > loss_mel_ce: 2.47300386428833 (nan) | > loss: 0.00987220648676157 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13733026973832563) | > loader_time: 0.0045 (0.0046203269316696235)  --> TIME: 2025-02-26 06:05:58 -- STEP: 9560/36938 -- GLOBAL_STEP: 1117700 | > loss_text_ce: 0.02693728543817997 (0.031890699130319174) | > loss_mel_ce: 2.0643906593322754 (nan) | > loss: 0.008298920467495918 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13733171885482334) | > loader_time: 0.0045 (0.0046204101590431755)  --> TIME: 2025-02-26 06:06:07 -- STEP: 9610/36938 -- GLOBAL_STEP: 1117750 | > loss_text_ce: 0.03163354471325874 (0.03188689928075421) | > loss_mel_ce: 2.6008763313293457 (nan) | > loss: 0.010446468368172646 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.1373179144467315) | > loader_time: 0.0045 (0.004620077781200891)  --> TIME: 2025-02-26 06:06:16 -- STEP: 9660/36938 -- GLOBAL_STEP: 1117800 | > loss_text_ce: 0.04168223589658737 (0.03188319026529162) | > loss_mel_ce: 2.8497848510742188 (nan) | > loss: 0.01147407665848732 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13728966382226004) | > loader_time: 0.0044 (0.004619686623290937)  --> TIME: 2025-02-26 06:06:25 -- STEP: 9710/36938 -- GLOBAL_STEP: 1117850 | > loss_text_ce: 0.0429312027990818 (0.031885371551331865) | > loss_mel_ce: 2.9387683868408203 (nan) | > loss: 0.011832142248749733 (nan) | > current_lr: 5e-06 | > step_time: 0.1687 (0.1372743600674433) | > loader_time: 0.0047 (0.004619409912769366)  --> TIME: 2025-02-26 06:06:34 -- STEP: 9760/36938 -- GLOBAL_STEP: 1117900 | > loss_text_ce: 0.03619075194001198 (0.0318908872025575) | > loss_mel_ce: 2.946007251739502 (nan) | > loss: 0.011834119446575642 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13725143399883408) | > loader_time: 0.0046 (0.004619015191422115)  --> TIME: 2025-02-26 06:06:43 -- STEP: 9810/36938 -- GLOBAL_STEP: 1117950 | > loss_text_ce: 0.028468281030654907 (0.03189046372480065) | > loss_mel_ce: 2.974926710128784 (nan) | > loss: 0.011918235570192337 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13723441599341263) | > loader_time: 0.0047 (0.004618767374760012)  --> TIME: 2025-02-26 06:06:53 -- STEP: 9860/36938 -- GLOBAL_STEP: 1118000 | > loss_text_ce: 0.026843726634979248 (0.03189344978617385) | > loss_mel_ce: 2.8802106380462646 (nan) | > loss: 0.011535931378602982 (nan) | > current_lr: 5e-06 | > step_time: 0.1445 (0.13724501309481804) | > loader_time: 0.0045 (0.004618949411365117)  --> TIME: 2025-02-26 06:07:02 -- STEP: 9910/36938 -- GLOBAL_STEP: 1118050 | > loss_text_ce: 0.03359639272093773 (0.031894876927280144) | > loss_mel_ce: 2.9968206882476807 (nan) | > loss: 0.01202546525746584 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.13725048324053668) | > loader_time: 0.0046 (0.004619049328246813)  --> TIME: 2025-02-26 06:07:12 -- STEP: 9960/36938 -- GLOBAL_STEP: 1118100 | > loss_text_ce: 0.032942403107881546 (0.03189463103040447) | > loss_mel_ce: 1.52978515625 (nan) | > loss: 0.0062013003043830395 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.13724889266921714) | > loader_time: 0.0045 (0.00461908827823806)  --> TIME: 2025-02-26 06:07:21 -- STEP: 10010/36938 -- GLOBAL_STEP: 1118150 | > loss_text_ce: 0.028953997418284416 (0.031894649483066154) | > loss_mel_ce: 2.6657297611236572 (nan) | > loss: 0.01069319061934948 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13726604518356755) | > loader_time: 0.0046 (0.004619249382933641)  --> TIME: 2025-02-26 06:07:31 -- STEP: 10060/36938 -- GLOBAL_STEP: 1118200 | > loss_text_ce: 0.034149039536714554 (0.031898114557246414) | > loss_mel_ce: 1.3027887344360352 (nan) | > loss: 0.005305308848619461 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13725694247315828) | > loader_time: 0.0046 (0.00461897240955359)  --> TIME: 2025-02-26 06:07:40 -- STEP: 10110/36938 -- GLOBAL_STEP: 1118250 | > loss_text_ce: 0.02548382245004177 (0.03189632466143956) | > loss_mel_ce: 2.9713845252990723 (nan) | > loss: 0.01189233548939228 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13724308832226942) | > loader_time: 0.0046 (0.0046186711031180674)  --> TIME: 2025-02-26 06:07:49 -- STEP: 10160/36938 -- GLOBAL_STEP: 1118300 | > loss_text_ce: 0.016474081203341484 (0.03189808865119949) | > loss_mel_ce: 0.9260637760162354 (nan) | > loss: 0.003740229643881321 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13724653458031966) | > loader_time: 0.0045 (0.0046183783707656086)  --> TIME: 2025-02-26 06:07:59 -- STEP: 10210/36938 -- GLOBAL_STEP: 1118350 | > loss_text_ce: 0.0504475012421608 (0.03189399219186461) | > loss_mel_ce: 2.276440382003784 (nan) | > loss: 0.009233682416379452 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13724760268974384) | > loader_time: 0.0044 (0.0046181370532477495)  --> TIME: 2025-02-26 06:08:08 -- STEP: 10260/36938 -- GLOBAL_STEP: 1118400 | > loss_text_ce: 0.029748452827334404 (0.031891801651861236) | > loss_mel_ce: 2.132110595703125 (nan) | > loss: 0.008578806184232235 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13722908313278917) | > loader_time: 0.0047 (0.004617888560304385)  --> TIME: 2025-02-26 06:08:17 -- STEP: 10310/36938 -- GLOBAL_STEP: 1118450 | > loss_text_ce: 0.03039591573178768 (0.03189417182013501) | > loss_mel_ce: 2.6101672649383545 (nan) | > loss: 0.010478425770998001 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.13724115886928615) | > loader_time: 0.0045 (0.004617612045095667)  --> TIME: 2025-02-26 06:08:26 -- STEP: 10360/36938 -- GLOBAL_STEP: 1118500 | > loss_text_ce: 0.029049472883343697 (0.03189832031606368) | > loss_mel_ce: 2.4690468311309814 (nan) | > loss: 0.009913081303238869 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13723366467648826) | > loader_time: 0.0046 (0.004617339280581365)  --> TIME: 2025-02-26 06:08:36 -- STEP: 10410/36938 -- GLOBAL_STEP: 1118550 | > loss_text_ce: 0.030424531549215317 (0.03189204882653235) | > loss_mel_ce: 3.0039236545562744 (nan) | > loss: 0.012041064910590649 (nan) | > current_lr: 5e-06 | > step_time: 0.1828 (0.13723582186226985) | > loader_time: 0.0048 (0.004617774475090314)  --> TIME: 2025-02-26 06:08:46 -- STEP: 10460/36938 -- GLOBAL_STEP: 1118600 | > loss_text_ce: 0.03314341604709625 (0.03189306133658213) | > loss_mel_ce: 2.8331403732299805 (nan) | > loss: 0.011374142952263355 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13727435785315323) | > loader_time: 0.0045 (0.004617956126853104)  --> TIME: 2025-02-26 06:08:55 -- STEP: 10510/36938 -- GLOBAL_STEP: 1118650 | > loss_text_ce: 0.02963843010365963 (0.03189540940366114) | > loss_mel_ce: 3.1762197017669678 (nan) | > loss: 0.012721660546958447 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.1372662702818111) | > loader_time: 0.0047 (0.0046179399390542655)  --> TIME: 2025-02-26 06:09:04 -- STEP: 10560/36938 -- GLOBAL_STEP: 1118700 | > loss_text_ce: 0.03410239517688751 (0.031900040730816245) | > loss_mel_ce: 2.395298957824707 (nan) | > loss: 0.009640482254326344 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13725850157665342) | > loader_time: 0.0045 (0.004617743081215641)  --> TIME: 2025-02-26 06:09:14 -- STEP: 10610/36938 -- GLOBAL_STEP: 1118750 | > loss_text_ce: 0.018447481095790863 (0.031897231340969966) | > loss_mel_ce: 2.000938892364502 (nan) | > loss: 0.008013438433408737 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13725448569298543) | > loader_time: 0.0045 (0.004617701017216156)  --> TIME: 2025-02-26 06:09:23 -- STEP: 10660/36938 -- GLOBAL_STEP: 1118800 | > loss_text_ce: 0.03413882479071617 (0.03190381414768941) | > loss_mel_ce: 3.164238691329956 (nan) | > loss: 0.012691975571215153 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13727048548256343) | > loader_time: 0.0046 (0.004617809645454146)  --> TIME: 2025-02-26 06:09:32 -- STEP: 10710/36938 -- GLOBAL_STEP: 1118850 | > loss_text_ce: 0.033212147653102875 (0.031902334714059524) | > loss_mel_ce: 1.9241020679473877 (nan) | > loss: 0.007767120376229286 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13724111357545407) | > loader_time: 0.0047 (0.004617534520801914)  --> TIME: 2025-02-26 06:09:41 -- STEP: 10760/36938 -- GLOBAL_STEP: 1118900 | > loss_text_ce: 0.028976622968912125 (0.03190322754676111) | > loss_mel_ce: 3.1388161182403564 (nan) | > loss: 0.012570607475936413 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13722238611554446) | > loader_time: 0.0046 (0.004617357143238999)  --> TIME: 2025-02-26 06:09:51 -- STEP: 10810/36938 -- GLOBAL_STEP: 1118950 | > loss_text_ce: 0.03478226065635681 (0.031901029848030844) | > loss_mel_ce: 1.26116943359375 (nan) | > loss: 0.005142665933817625 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13723615150998505) | > loader_time: 0.0048 (0.004617626903015175)  --> TIME: 2025-02-26 06:10:00 -- STEP: 10860/36938 -- GLOBAL_STEP: 1119000 | > loss_text_ce: 0.030104033648967743 (0.03189567958620495) | > loss_mel_ce: 3.3758604526519775 (nan) | > loss: 0.013515732251107693 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13723303277189425) | > loader_time: 0.0044 (0.004617480524776156)  --> TIME: 2025-02-26 06:10:10 -- STEP: 10910/36938 -- GLOBAL_STEP: 1119050 | > loss_text_ce: 0.029436316341161728 (0.031900881812160685) | > loss_mel_ce: 2.8922228813171387 (nan) | > loss: 0.011593886651098728 (nan) | > current_lr: 5e-06 | > step_time: 0.161 (0.1372365420504939) | > loader_time: 0.0046 (0.004617507232645021)  --> TIME: 2025-02-26 06:10:19 -- STEP: 10960/36938 -- GLOBAL_STEP: 1119100 | > loss_text_ce: 0.040216315537691116 (0.031898644501275275) | > loss_mel_ce: 1.363568902015686 (nan) | > loss: 0.005570576526224613 (nan) | > current_lr: 5e-06 | > step_time: 0.1079 (0.13722155661913446) | > loader_time: 0.0045 (0.004617306764108405)  --> TIME: 2025-02-26 06:10:28 -- STEP: 11010/36938 -- GLOBAL_STEP: 1119150 | > loss_text_ce: 0.031845588237047195 (0.03190056590141751) | > loss_mel_ce: 3.223842144012451 (nan) | > loss: 0.012919396162033081 (nan) | > current_lr: 5e-06 | > step_time: 0.1762 (0.13722657035633443) | > loader_time: 0.0047 (0.004617137675064447)  --> TIME: 2025-02-26 06:10:38 -- STEP: 11060/36938 -- GLOBAL_STEP: 1119200 | > loss_text_ce: 0.033627357333898544 (0.03190142621924488) | > loss_mel_ce: 2.2951509952545166 (nan) | > loss: 0.00924118421971798 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13723977082029878) | > loader_time: 0.0045 (0.004617177728718471)  --> TIME: 2025-02-26 06:10:47 -- STEP: 11110/36938 -- GLOBAL_STEP: 1119250 | > loss_text_ce: 0.038129158318042755 (0.031901395364482485) | > loss_mel_ce: 2.6351187229156494 (nan) | > loss: 0.010608126409351826 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.1372290043654885) | > loader_time: 0.0046 (0.004617216177184103)  --> TIME: 2025-02-26 06:10:56 -- STEP: 11160/36938 -- GLOBAL_STEP: 1119300 | > loss_text_ce: 0.03107755444943905 (0.03190478003036395) | > loss_mel_ce: 2.6434595584869385 (nan) | > loss: 0.010613243095576763 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13721947672119103) | > loader_time: 0.0047 (0.004617045301690305)  --> TIME: 2025-02-26 06:11:06 -- STEP: 11210/36938 -- GLOBAL_STEP: 1119350 | > loss_text_ce: 0.024006463587284088 (0.031903128865956615) | > loss_mel_ce: 2.8606512546539307 (nan) | > loss: 0.011447054333984852 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13723202968260342) | > loader_time: 0.0045 (0.004616908788893695)  --> TIME: 2025-02-26 06:11:15 -- STEP: 11260/36938 -- GLOBAL_STEP: 1119400 | > loss_text_ce: 0.03893951326608658 (0.031903186040147906) | > loss_mel_ce: 3.077927589416504 (nan) | > loss: 0.012368520721793175 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13722071224165047) | > loader_time: 0.0044 (0.00461663132662458)  --> TIME: 2025-02-26 06:11:24 -- STEP: 11310/36938 -- GLOBAL_STEP: 1119450 | > loss_text_ce: 0.030980706214904785 (0.031904927478604114) | > loss_mel_ce: 2.7204415798187256 (nan) | > loss: 0.01091834343969822 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13720198057058147) | > loader_time: 0.0046 (0.004616549202660121)  --> TIME: 2025-02-26 06:11:33 -- STEP: 11360/36938 -- GLOBAL_STEP: 1119500 | > loss_text_ce: 0.031998589634895325 (0.03190294745801218) | > loss_mel_ce: 0.376953125 (nan) | > loss: 0.001622824347577989 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13718031753956253) | > loader_time: 0.0045 (0.00461651920013023)  --> TIME: 2025-02-26 06:11:43 -- STEP: 11410/36938 -- GLOBAL_STEP: 1119550 | > loss_text_ce: 0.03829353302717209 (0.0319012086503827) | > loss_mel_ce: 0.9746500849723816 (nan) | > loss: 0.004019617568701506 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13718744749358228) | > loader_time: 0.0046 (0.004616662956558729)  --> TIME: 2025-02-26 06:11:52 -- STEP: 11460/36938 -- GLOBAL_STEP: 1119600 | > loss_text_ce: 0.03621985763311386 (0.03189788247112427) | > loss_mel_ce: 3.3612923622131348 (nan) | > loss: 0.013482191599905491 (nan) | > current_lr: 5e-06 | > step_time: 0.1871 (0.13717195646508862) | > loader_time: 0.0048 (0.0046169025735705105)  --> TIME: 2025-02-26 06:12:01 -- STEP: 11510/36938 -- GLOBAL_STEP: 1119650 | > loss_text_ce: 0.033793315291404724 (0.03189485026029884) | > loss_mel_ce: 2.555856704711914 (nan) | > loss: 0.010276389308273792 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13718093580830754) | > loader_time: 0.0046 (0.0046172388525862395)  --> TIME: 2025-02-26 06:12:10 -- STEP: 11560/36938 -- GLOBAL_STEP: 1119700 | > loss_text_ce: 0.036826472729444504 (0.03189064692351376) | > loss_mel_ce: 3.0752334594726562 (nan) | > loss: 0.012349444441497326 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.137149801984376) | > loader_time: 0.0046 (0.004617339331385979)  --> TIME: 2025-02-26 06:12:19 -- STEP: 11610/36938 -- GLOBAL_STEP: 1119750 | > loss_text_ce: 0.031142115592956543 (0.0318881442873039) | > loss_mel_ce: 2.8599371910095215 (nan) | > loss: 0.011472538113594055 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13714611598071322) | > loader_time: 0.0047 (0.004617389638705239)  --> TIME: 2025-02-26 06:12:29 -- STEP: 11660/36938 -- GLOBAL_STEP: 1119800 | > loss_text_ce: 0.030954668298363686 (0.03188850126288324) | > loss_mel_ce: 2.9020743370056152 (nan) | > loss: 0.011639004573225975 (nan) | > current_lr: 5e-06 | > step_time: 0.1452 (0.13716643952587065) | > loader_time: 0.0046 (0.0046176467506186016)  --> TIME: 2025-02-26 06:12:38 -- STEP: 11710/36938 -- GLOBAL_STEP: 1119850 | > loss_text_ce: 0.02811848558485508 (0.0318950175055573) | > loss_mel_ce: 2.3479883670806885 (nan) | > loss: 0.009428995661437511 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.1371601782325576) | > loader_time: 0.0045 (0.004617772582857939)  --> TIME: 2025-02-26 06:12:48 -- STEP: 11760/36938 -- GLOBAL_STEP: 1119900 | > loss_text_ce: 0.037557560950517654 (0.03189798522262262) | > loss_mel_ce: 3.087792158126831 (nan) | > loss: 0.012402182444930077 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13715834226332382) | > loader_time: 0.0066 (0.004618088079958523)  --> TIME: 2025-02-26 06:12:57 -- STEP: 11810/36938 -- GLOBAL_STEP: 1119950 | > loss_text_ce: 0.032932035624980927 (0.03190178680426854) | > loss_mel_ce: 2.8746676445007324 (nan) | > loss: 0.011538094840943813 (nan) | > current_lr: 5e-06 | > step_time: 0.1558 (0.13716377406479482) | > loader_time: 0.0046 (0.004618301581367386)  --> TIME: 2025-02-26 06:13:07 -- STEP: 11860/36938 -- GLOBAL_STEP: 1120000 | > loss_text_ce: 0.027155710384249687 (0.03189929276783307) | > loss_mel_ce: 2.340202569961548 (nan) | > loss: 0.00939427874982357 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.13716053735546324) | > loader_time: 0.0049 (0.004618448250209277) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1120000.pth  --> TIME: 2025-02-26 06:13:22 -- STEP: 11910/36938 -- GLOBAL_STEP: 1120050 | > loss_text_ce: 0.03331254422664642 (0.031903220149809564) | > loss_mel_ce: 3.4006972312927246 (nan) | > loss: 0.013627023436129093 (nan) | > current_lr: 5e-06 | > step_time: 0.1796 (0.13716762978724167) | > loader_time: 0.0048 (0.004618516256587011)  --> TIME: 2025-02-26 06:13:31 -- STEP: 11960/36938 -- GLOBAL_STEP: 1120100 | > loss_text_ce: 0.03290560841560364 (0.03189962405608612) | > loss_mel_ce: 2.288600206375122 (nan) | > loss: 0.00921232532709837 (nan) | > current_lr: 5e-06 | > step_time: 0.1794 (0.1371598016856889) | > loader_time: 0.0048 (0.004618689009178422)  --> TIME: 2025-02-26 06:13:40 -- STEP: 12010/36938 -- GLOBAL_STEP: 1120150 | > loss_text_ce: 0.02299313060939312 (0.03189216985504483) | > loss_mel_ce: 2.7229385375976562 (nan) | > loss: 0.01089655514806509 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.1371466016292954) | > loader_time: 0.0046 (0.00461871913827327)  --> TIME: 2025-02-26 06:13:49 -- STEP: 12060/36938 -- GLOBAL_STEP: 1120200 | > loss_text_ce: 0.0323929525911808 (0.03189746236573788) | > loss_mel_ce: 2.7975075244903564 (nan) | > loss: 0.01122976467013359 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13715299300688666) | > loader_time: 0.0049 (0.0046188156798506385)  --> TIME: 2025-02-26 06:13:59 -- STEP: 12110/36938 -- GLOBAL_STEP: 1120250 | > loss_text_ce: 0.03290564566850662 (0.03189407049652877) | > loss_mel_ce: 3.0624494552612305 (nan) | > loss: 0.01228315569460392 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13713501864085847) | > loader_time: 0.0045 (0.004618877895994654)  --> TIME: 2025-02-26 06:14:08 -- STEP: 12160/36938 -- GLOBAL_STEP: 1120300 | > loss_text_ce: 0.03089938499033451 (0.03189492632962725) | > loss_mel_ce: 2.9698829650878906 (nan) | > loss: 0.011907867155969143 (nan) | > current_lr: 5e-06 | > step_time: 0.188 (0.13714239056172975) | > loader_time: 0.0048 (0.004619039908835735)  --> TIME: 2025-02-26 06:14:17 -- STEP: 12210/36938 -- GLOBAL_STEP: 1120350 | > loss_text_ce: 0.02864912524819374 (0.03189177349801665) | > loss_mel_ce: 2.262500047683716 (nan) | > loss: 0.009091862477362156 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.1371438275976892) | > loader_time: 0.0047 (0.0046194474292914215)  --> TIME: 2025-02-26 06:14:27 -- STEP: 12260/36938 -- GLOBAL_STEP: 1120400 | > loss_text_ce: 0.03435131534934044 (0.03188691688184936) | > loss_mel_ce: 2.641226053237915 (nan) | > loss: 0.010617371648550034 (nan) | > current_lr: 5e-06 | > step_time: 0.1079 (0.13714049286398816) | > loader_time: 0.0061 (0.004620031042550147)  --> TIME: 2025-02-26 06:14:36 -- STEP: 12310/36938 -- GLOBAL_STEP: 1120450 | > loss_text_ce: 0.038261182606220245 (0.03188307783895503) | > loss_mel_ce: 2.646110773086548 (nan) | > loss: 0.010652270168066025 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13710503049442832) | > loader_time: 0.0048 (0.004620203809172599)  --> TIME: 2025-02-26 06:14:45 -- STEP: 12360/36938 -- GLOBAL_STEP: 1120500 | > loss_text_ce: 0.027805261313915253 (0.031888009775068164) | > loss_mel_ce: 2.663741111755371 (nan) | > loss: 0.01068074069917202 (nan) | > current_lr: 5e-06 | > step_time: 0.1874 (0.1371043397577823) | > loader_time: 0.0048 (0.004620267021617437)  --> TIME: 2025-02-26 06:14:54 -- STEP: 12410/36938 -- GLOBAL_STEP: 1120550 | > loss_text_ce: 0.04389496520161629 (0.031891509948191635) | > loss_mel_ce: 2.349170684814453 (nan) | > loss: 0.009496293030679226 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13711081434890673) | > loader_time: 0.0046 (0.004620299139491572)  --> TIME: 2025-02-26 06:15:04 -- STEP: 12460/36938 -- GLOBAL_STEP: 1120600 | > loss_text_ce: 0.02956099435687065 (0.031897297763320856) | > loss_mel_ce: 3.185711622238159 (nan) | > loss: 0.012759019620716572 (nan) | > current_lr: 5e-06 | > step_time: 0.2015 (0.1371192360766233) | > loader_time: 0.0049 (0.004620600933056558)  --> TIME: 2025-02-26 06:15:13 -- STEP: 12510/36938 -- GLOBAL_STEP: 1120650 | > loss_text_ce: 0.02823534980416298 (0.03190047104703735) | > loss_mel_ce: 3.319373607635498 (nan) | > loss: 0.013284163549542427 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.1371229054544744) | > loader_time: 0.005 (0.004620702775548095)  --> TIME: 2025-02-26 06:15:23 -- STEP: 12560/36938 -- GLOBAL_STEP: 1120700 | > loss_text_ce: 0.03357327729463577 (0.031896275282795844) | > loss_mel_ce: 2.3237314224243164 (nan) | > loss: 0.009354384616017342 (nan) | > current_lr: 5e-06 | > step_time: 0.1499 (0.13715171216020033) | > loader_time: 0.0047 (0.004620940024685689)  --> TIME: 2025-02-26 06:15:32 -- STEP: 12610/36938 -- GLOBAL_STEP: 1120750 | > loss_text_ce: 0.03415128216147423 (0.03189701932149322) | > loss_mel_ce: 2.6234288215637207 (nan) | > loss: 0.010545953176915646 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13714861758259994) | > loader_time: 0.0046 (0.004621102051731125)  --> TIME: 2025-02-26 06:15:42 -- STEP: 12660/36938 -- GLOBAL_STEP: 1120800 | > loss_text_ce: 0.03669322654604912 (0.03190141882509052) | > loss_mel_ce: 2.6477577686309814 (nan) | > loss: 0.01065258402377367 (nan) | > current_lr: 5e-06 | > step_time: 0.1456 (0.1371465638727176) | > loader_time: 0.0047 (0.004621304211458301)  --> TIME: 2025-02-26 06:15:51 -- STEP: 12710/36938 -- GLOBAL_STEP: 1120850 | > loss_text_ce: 0.03267434984445572 (0.03190374412829595) | > loss_mel_ce: 2.1007778644561768 (nan) | > loss: 0.008466080762445927 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.1371611447900783) | > loader_time: 0.0045 (0.0046215367448507886)  --> TIME: 2025-02-26 06:16:01 -- STEP: 12760/36938 -- GLOBAL_STEP: 1120900 | > loss_text_ce: 0.03275435045361519 (0.03190680370727389) | > loss_mel_ce: 3.152008056640625 (nan) | > loss: 0.012637946754693985 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.13718634120722856) | > loader_time: 0.0046 (0.004621632252367288)  --> TIME: 2025-02-26 06:16:11 -- STEP: 12810/36938 -- GLOBAL_STEP: 1120950 | > loss_text_ce: 0.03300222009420395 (0.0319062222221358) | > loss_mel_ce: 2.2141623497009277 (nan) | > loss: 0.00891731958836317 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13720345430128394) | > loader_time: 0.0045 (0.004621751247021831)  --> TIME: 2025-02-26 06:16:20 -- STEP: 12860/36938 -- GLOBAL_STEP: 1121000 | > loss_text_ce: 0.028279799968004227 (0.03190349705769304) | > loss_mel_ce: 2.953166961669922 (nan) | > loss: 0.011831138283014297 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13718787691270523) | > loader_time: 0.0047 (0.004621858897142401)  --> TIME: 2025-02-26 06:16:29 -- STEP: 12910/36938 -- GLOBAL_STEP: 1121050 | > loss_text_ce: 0.0434282012283802 (0.031904472630271775) | > loss_mel_ce: 2.535708427429199 (nan) | > loss: 0.010234669782221317 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.1371922496120468) | > loader_time: 0.0047 (0.00462200817075649)  --> TIME: 2025-02-26 06:16:39 -- STEP: 12960/36938 -- GLOBAL_STEP: 1121100 | > loss_text_ce: 0.030657216906547546 (0.031902792472909614) | > loss_mel_ce: 2.9626777172088623 (nan) | > loss: 0.011878314428031445 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13720600857042917) | > loader_time: 0.0047 (0.004621969936070605)  --> TIME: 2025-02-26 06:16:48 -- STEP: 13010/36938 -- GLOBAL_STEP: 1121150 | > loss_text_ce: 0.03597766533493996 (0.03190674848845941) | > loss_mel_ce: 1.2937970161437988 (nan) | > loss: 0.005276883952319622 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13720279753345538) | > loader_time: 0.0044 (0.0046218227002365825)  --> TIME: 2025-02-26 06:16:57 -- STEP: 13060/36938 -- GLOBAL_STEP: 1121200 | > loss_text_ce: 0.029801109805703163 (0.03190321096063653) | > loss_mel_ce: 2.705051898956299 (nan) | > loss: 0.010852592065930367 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13718378399634185) | > loader_time: 0.0046 (0.004621684259148857)  --> TIME: 2025-02-26 06:17:07 -- STEP: 13110/36938 -- GLOBAL_STEP: 1121250 | > loss_text_ce: 0.025843225419521332 (0.03190198555932539) | > loss_mel_ce: 2.7526185512542725 (nan) | > loss: 0.011025642044842243 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.137183729516559) | > loader_time: 0.0046 (0.0046215542757629185)  --> TIME: 2025-02-26 06:17:16 -- STEP: 13160/36938 -- GLOBAL_STEP: 1121300 | > loss_text_ce: 0.0328015461564064 (0.031906304725992055) | > loss_mel_ce: 2.1155080795288086 (nan) | > loss: 0.008525039069354534 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13717895039671407) | > loader_time: 0.0046 (0.004621409989417849)  --> TIME: 2025-02-26 06:17:25 -- STEP: 13210/36938 -- GLOBAL_STEP: 1121350 | > loss_text_ce: 0.03567694500088692 (0.03190734075758817) | > loss_mel_ce: 2.7634685039520264 (nan) | > loss: 0.01110772043466568 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13718859268624475) | > loader_time: 0.0045 (0.004621339096976065)  --> TIME: 2025-02-26 06:17:34 -- STEP: 13260/36938 -- GLOBAL_STEP: 1121400 | > loss_text_ce: 0.026362426578998566 (0.031907594271234456) | > loss_mel_ce: 1.1301207542419434 (nan) | > loss: 0.004589219111949205 (nan) | > current_lr: 5e-06 | > step_time: 0.1118 (0.13716860533120184) | > loader_time: 0.0046 (0.0046212888411267085)  --> TIME: 2025-02-26 06:17:44 -- STEP: 13310/36938 -- GLOBAL_STEP: 1121450 | > loss_text_ce: 0.036391738802194595 (0.031901508327335515) | > loss_mel_ce: 2.804608106613159 (nan) | > loss: 0.01127380970865488 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13716155110580558) | > loader_time: 0.0045 (0.004621215532992116)  --> TIME: 2025-02-26 06:17:53 -- STEP: 13360/36938 -- GLOBAL_STEP: 1121500 | > loss_text_ce: 0.035196103155612946 (0.031899829426763456) | > loss_mel_ce: 3.0082316398620605 (nan) | > loss: 0.012077094987034798 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.1371794968308085) | > loader_time: 0.0047 (0.004621176305645215)  --> TIME: 2025-02-26 06:18:03 -- STEP: 13410/36938 -- GLOBAL_STEP: 1121550 | > loss_text_ce: 0.033807042986154556 (0.031900962378404514) | > loss_mel_ce: 2.3473103046417236 (nan) | > loss: 0.009448878467082977 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13719327728931416) | > loader_time: 0.0047 (0.004621098114428972)  --> TIME: 2025-02-26 06:18:12 -- STEP: 13460/36938 -- GLOBAL_STEP: 1121600 | > loss_text_ce: 0.03253822401165962 (0.03190223346609621) | > loss_mel_ce: 2.9587655067443848 (nan) | > loss: 0.011870253831148148 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13718284768831457) | > loader_time: 0.0046 (0.004620971385649707)  --> TIME: 2025-02-26 06:18:22 -- STEP: 13510/36938 -- GLOBAL_STEP: 1121650 | > loss_text_ce: 0.032626137137413025 (0.0319004058440819) | > loss_mel_ce: 1.65399169921875 (nan) | > loss: 0.00669292826205492 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13720102641954818) | > loader_time: 0.0046 (0.004620887313746748)  --> TIME: 2025-02-26 06:18:31 -- STEP: 13560/36938 -- GLOBAL_STEP: 1121700 | > loss_text_ce: 0.025605082511901855 (0.031905398630072634) | > loss_mel_ce: 2.751173257827759 (nan) | > loss: 0.011018961668014526 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13719399116032213) | > loader_time: 0.0046 (0.004620835686151936)  --> TIME: 2025-02-26 06:18:40 -- STEP: 13610/36938 -- GLOBAL_STEP: 1121750 | > loss_text_ce: 0.03402487188577652 (0.03190150288721252) | > loss_mel_ce: 2.4490907192230225 (nan) | > loss: 0.009853634051978588 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13717565044246094) | > loader_time: 0.0045 (0.004620616055866249)  --> TIME: 2025-02-26 06:18:49 -- STEP: 13660/36938 -- GLOBAL_STEP: 1121800 | > loss_text_ce: 0.035644691437482834 (0.031904685646322636) | > loss_mel_ce: 3.0717785358428955 (nan) | > loss: 0.012331045232713223 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.1371584140015204) | > loader_time: 0.0046 (0.0046203649061490396)  --> TIME: 2025-02-26 06:18:58 -- STEP: 13710/36938 -- GLOBAL_STEP: 1121850 | > loss_text_ce: 0.025094814598560333 (0.03190637566991549) | > loss_mel_ce: 2.608060121536255 (nan) | > loss: 0.010449027642607689 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13713945926844304) | > loader_time: 0.0047 (0.004620534968497865)  --> TIME: 2025-02-26 06:19:07 -- STEP: 13760/36938 -- GLOBAL_STEP: 1121900 | > loss_text_ce: 0.03405129536986351 (0.03189945282789501) | > loss_mel_ce: 2.2014708518981934 (nan) | > loss: 0.00887111946940422 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.1371389816666756) | > loader_time: 0.0046 (0.0046206305539885015)  --> TIME: 2025-02-26 06:19:17 -- STEP: 13810/36938 -- GLOBAL_STEP: 1121950 | > loss_text_ce: 0.04004279896616936 (0.031902278377286576) | > loss_mel_ce: 2.6559760570526123 (nan) | > loss: 0.010698488913476467 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.1371504573352433) | > loader_time: 0.0047 (0.004620839408305463)  --> TIME: 2025-02-26 06:19:26 -- STEP: 13860/36938 -- GLOBAL_STEP: 1122000 | > loss_text_ce: 0.028461825102567673 (0.03190545063340833) | > loss_mel_ce: 2.0731728076934814 (nan) | > loss: 0.008339820429682732 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13714863662251817) | > loader_time: 0.0048 (0.004621231848371305)  --> TIME: 2025-02-26 06:19:35 -- STEP: 13910/36938 -- GLOBAL_STEP: 1122050 | > loss_text_ce: 0.040976107120513916 (0.03190869682767737) | > loss_mel_ce: 2.0150740146636963 (nan) | > loss: 0.008158929646015167 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13713286995973448) | > loader_time: 0.0045 (0.004621153388582127)  --> TIME: 2025-02-26 06:19:45 -- STEP: 13960/36938 -- GLOBAL_STEP: 1122100 | > loss_text_ce: 0.03832339867949486 (0.03191021842352313) | > loss_mel_ce: 1.6432383060455322 (nan) | > loss: 0.006672864314168692 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13713714760490664) | > loader_time: 0.0045 (0.004621308973662134)  --> TIME: 2025-02-26 06:19:54 -- STEP: 14010/36938 -- GLOBAL_STEP: 1122150 | > loss_text_ce: 0.03693756088614464 (0.03190625830600652) | > loss_mel_ce: 3.041774272918701 (nan) | > loss: 0.012217110954225063 (nan) | > current_lr: 5e-06 | > step_time: 0.1927 (0.1371399102765776) | > loader_time: 0.0048 (0.004621439146876404)  --> TIME: 2025-02-26 06:20:03 -- STEP: 14060/36938 -- GLOBAL_STEP: 1122200 | > loss_text_ce: 0.028973564505577087 (0.031903791401447675) | > loss_mel_ce: 2.531341075897217 (nan) | > loss: 0.010159979574382305 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.1371350429983938) | > loader_time: 0.0046 (0.004621476994808827)  --> TIME: 2025-02-26 06:20:13 -- STEP: 14110/36938 -- GLOBAL_STEP: 1122250 | > loss_text_ce: 0.037457313388586044 (0.031902902558759966) | > loss_mel_ce: 3.2408323287963867 (nan) | > loss: 0.01300908625125885 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1371422052890336) | > loader_time: 0.0045 (0.004621388319111463)  --> TIME: 2025-02-26 06:20:22 -- STEP: 14160/36938 -- GLOBAL_STEP: 1122300 | > loss_text_ce: 0.025976870208978653 (0.03190518359747035) | > loss_mel_ce: 2.8898558616638184 (nan) | > loss: 0.011570765636861324 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13716228668299077) | > loader_time: 0.0046 (0.004621428369128756)  --> TIME: 2025-02-26 06:20:32 -- STEP: 14210/36938 -- GLOBAL_STEP: 1122350 | > loss_text_ce: 0.03015611320734024 (0.03190492591030888) | > loss_mel_ce: 2.982476234436035 (nan) | > loss: 0.011954891495406628 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.13717559899015375) | > loader_time: 0.0048 (0.004621503455466054)  --> TIME: 2025-02-26 06:20:42 -- STEP: 14260/36938 -- GLOBAL_STEP: 1122400 | > loss_text_ce: 0.031687717884778976 (0.031907370961813154) | > loss_mel_ce: 2.9559736251831055 (nan) | > loss: 0.011855799704790115 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13719335008637262) | > loader_time: 0.0046 (0.004621513026961088)  --> TIME: 2025-02-26 06:20:51 -- STEP: 14310/36938 -- GLOBAL_STEP: 1122450 | > loss_text_ce: 0.03119819611310959 (0.03190567014702456) | > loss_mel_ce: 1.8735315799713135 (nan) | > loss: 0.007558451499789953 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13716541989377795) | > loader_time: 0.0048 (0.004621407687455754)  --> TIME: 2025-02-26 06:21:00 -- STEP: 14360/36938 -- GLOBAL_STEP: 1122500 | > loss_text_ce: 0.04332960024476051 (0.03190433597004076) | > loss_mel_ce: 3.1285500526428223 (nan) | > loss: 0.012586825527250767 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13715358339976486) | > loader_time: 0.0045 (0.004621302766056098)  --> TIME: 2025-02-26 06:21:09 -- STEP: 14410/36938 -- GLOBAL_STEP: 1122550 | > loss_text_ce: 0.0360635481774807 (0.03190122680715292) | > loss_mel_ce: 2.4366331100463867 (nan) | > loss: 0.009812287986278534 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13715433769967322) | > loader_time: 0.0045 (0.004621333880692518)  --> TIME: 2025-02-26 06:21:18 -- STEP: 14460/36938 -- GLOBAL_STEP: 1122600 | > loss_text_ce: 0.038536980748176575 (0.03189902213378898) | > loss_mel_ce: 2.970703125 (nan) | > loss: 0.011941430158913136 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13715324561790498) | > loader_time: 0.0046 (0.00462129658518324)  --> TIME: 2025-02-26 06:21:28 -- STEP: 14510/36938 -- GLOBAL_STEP: 1122650 | > loss_text_ce: 0.03023446910083294 (0.03190083384615263) | > loss_mel_ce: 3.5293772220611572 (nan) | > loss: 0.014125443994998932 (nan) | > current_lr: 5e-06 | > step_time: 0.169 (0.13715656851999206) | > loader_time: 0.0048 (0.004621423367054014)  --> TIME: 2025-02-26 06:21:37 -- STEP: 14560/36938 -- GLOBAL_STEP: 1122700 | > loss_text_ce: 0.025999831035733223 (0.031905367052000866) | > loss_mel_ce: 3.117306709289551 (nan) | > loss: 0.012473438866436481 (nan) | > current_lr: 5e-06 | > step_time: 0.1756 (0.13716050846563604) | > loader_time: 0.0048 (0.004621574855767748)  --> TIME: 2025-02-26 06:21:47 -- STEP: 14610/36938 -- GLOBAL_STEP: 1122750 | > loss_text_ce: 0.031084096059203148 (0.031903665785822426) | > loss_mel_ce: 2.319373846054077 (nan) | > loss: 0.00932721421122551 (nan) | > current_lr: 5e-06 | > step_time: 0.1877 (0.13715999537669665) | > loader_time: 0.0048 (0.0046216361902903895)  --> TIME: 2025-02-26 06:21:56 -- STEP: 14660/36938 -- GLOBAL_STEP: 1122800 | > loss_text_ce: 0.029793454334139824 (0.03190511240163944) | > loss_mel_ce: 2.344420909881592 (nan) | > loss: 0.009421486407518387 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.1371572011652292) | > loader_time: 0.0047 (0.004621582190629389)  --> TIME: 2025-02-26 06:22:05 -- STEP: 14710/36938 -- GLOBAL_STEP: 1122850 | > loss_text_ce: 0.03161931410431862 (0.03190701537455963) | > loss_mel_ce: 3.192526340484619 (nan) | > loss: 0.012794229201972485 (nan) | > current_lr: 5e-06 | > step_time: 0.1689 (0.13715378518496776) | > loader_time: 0.0046 (0.004621577473581429)  --> TIME: 2025-02-26 06:22:15 -- STEP: 14760/36938 -- GLOBAL_STEP: 1122900 | > loss_text_ce: 0.03155343607068062 (0.031902093157805664) | > loss_mel_ce: 2.017144203186035 (nan) | > loss: 0.008129753172397614 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13715946307673352) | > loader_time: 0.0068 (0.004621571447791108)  --> TIME: 2025-02-26 06:22:24 -- STEP: 14810/36938 -- GLOBAL_STEP: 1122950 | > loss_text_ce: 0.02801147662103176 (0.03190535787728171) | > loss_mel_ce: 2.777961254119873 (nan) | > loss: 0.011134813539683819 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13715366570229942) | > loader_time: 0.0045 (0.004621538900186033)  --> TIME: 2025-02-26 06:22:33 -- STEP: 14860/36938 -- GLOBAL_STEP: 1123000 | > loss_text_ce: 0.0334312804043293 (0.03190599359179592) | > loss_mel_ce: 2.5814669132232666 (nan) | > loss: 0.010376580990850925 (nan) | > current_lr: 5e-06 | > step_time: 0.1623 (0.13714958576944272) | > loader_time: 0.0046 (0.004621388854441365)  --> TIME: 2025-02-26 06:22:42 -- STEP: 14910/36938 -- GLOBAL_STEP: 1123050 | > loss_text_ce: 0.027613230049610138 (0.03190412799047583) | > loss_mel_ce: 2.0273234844207764 (nan) | > loss: 0.008154510520398617 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13712978223919625) | > loader_time: 0.0045 (0.004621550063012336)  --> TIME: 2025-02-26 06:22:51 -- STEP: 14960/36938 -- GLOBAL_STEP: 1123100 | > loss_text_ce: 0.023821474984288216 (0.03190277535904107) | > loss_mel_ce: 3.2669832706451416 (nan) | > loss: 0.013058749958872795 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.1371233968333127) | > loader_time: 0.0046 (0.004621331274190673)  --> TIME: 2025-02-26 06:23:01 -- STEP: 15010/36938 -- GLOBAL_STEP: 1123150 | > loss_text_ce: 0.026485659182071686 (0.03190201782682857) | > loss_mel_ce: 3.134971857070923 (nan) | > loss: 0.012545467354357243 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.1371306534849112) | > loader_time: 0.0045 (0.00462117128416666)  --> TIME: 2025-02-26 06:23:10 -- STEP: 15060/36938 -- GLOBAL_STEP: 1123200 | > loss_text_ce: 0.028295109048485756 (0.03190334768233634) | > loss_mel_ce: 2.8417632579803467 (nan) | > loss: 0.011389120481908321 (nan) | > current_lr: 5e-06 | > step_time: 0.1688 (0.13712208763378392) | > loader_time: 0.0047 (0.0046209883404917775)  --> TIME: 2025-02-26 06:23:19 -- STEP: 15110/36938 -- GLOBAL_STEP: 1123250 | > loss_text_ce: 0.034935325384140015 (0.03190300591859503) | > loss_mel_ce: 2.7449750900268555 (nan) | > loss: 0.011031391099095345 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.1371253462203838) | > loader_time: 0.0046 (0.004621025428797203)  --> TIME: 2025-02-26 06:23:29 -- STEP: 15160/36938 -- GLOBAL_STEP: 1123300 | > loss_text_ce: 0.023601680994033813 (0.03190032949389772) | > loss_mel_ce: 2.659053325653076 (nan) | > loss: 0.010645457543432713 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13711572493915508) | > loader_time: 0.0046 (0.0046209176800810735)  --> TIME: 2025-02-26 06:23:38 -- STEP: 15210/36938 -- GLOBAL_STEP: 1123350 | > loss_text_ce: 0.029924822971224785 (0.03189918340608112) | > loss_mel_ce: 2.741952657699585 (nan) | > loss: 0.01099951472133398 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13711705112206474) | > loader_time: 0.0047 (0.004620821016701628)  --> TIME: 2025-02-26 06:23:47 -- STEP: 15260/36938 -- GLOBAL_STEP: 1123400 | > loss_text_ce: 0.04031839221715927 (0.03190162650621965) | > loss_mel_ce: 7.53125 (nan) | > loss: 0.03004590794444084 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.1371249242035152) | > loader_time: 0.0044 (0.004620664960278738)  --> TIME: 2025-02-26 06:23:57 -- STEP: 15310/36938 -- GLOBAL_STEP: 1123450 | > loss_text_ce: 0.03434053808450699 (0.03189960080737285) | > loss_mel_ce: 2.198394536972046 (nan) | > loss: 0.008860060945153236 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.1371197990153367) | > loader_time: 0.0045 (0.004620631483938849)  --> TIME: 2025-02-26 06:24:06 -- STEP: 15360/36938 -- GLOBAL_STEP: 1123500 | > loss_text_ce: 0.0388069823384285 (0.03189582749782844) | > loss_mel_ce: 3.2604012489318848 (nan) | > loss: 0.013092096894979477 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13711692201904951) | > loader_time: 0.0046 (0.004620596580207333)  --> TIME: 2025-02-26 06:24:15 -- STEP: 15410/36938 -- GLOBAL_STEP: 1123550 | > loss_text_ce: 0.034055937081575394 (0.031899487114608584) | > loss_mel_ce: 0.04375267028808594 (nan) | > loss: 0.0003087643126491457 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13711471732447164) | > loader_time: 0.0045 (0.004620551351291015)  --> TIME: 2025-02-26 06:24:25 -- STEP: 15460/36938 -- GLOBAL_STEP: 1123600 | > loss_text_ce: 0.0364757776260376 (0.0319014381793812) | > loss_mel_ce: 2.8641915321350098 (nan) | > loss: 0.011510584503412247 (nan) | > current_lr: 5e-06 | > step_time: 0.1855 (0.13711696139899449) | > loader_time: 0.0048 (0.004620405434019955)  --> TIME: 2025-02-26 06:24:34 -- STEP: 15510/36938 -- GLOBAL_STEP: 1123650 | > loss_text_ce: 0.03135925158858299 (0.03189991059149215) | > loss_mel_ce: 2.3926637172698975 (nan) | > loss: 0.00961913913488388 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13711191355067634) | > loader_time: 0.0045 (0.004620253909103015)  --> TIME: 2025-02-26 06:24:43 -- STEP: 15560/36938 -- GLOBAL_STEP: 1123700 | > loss_text_ce: 0.032325129956007004 (0.03189954738052437) | > loss_mel_ce: 0.1081949844956398 (nan) | > loss: 0.0005576194962486625 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.1371143233500287) | > loader_time: 0.0044 (0.004620098332206495)  --> TIME: 2025-02-26 06:24:53 -- STEP: 15610/36938 -- GLOBAL_STEP: 1123750 | > loss_text_ce: 0.0386771522462368 (0.03189782066232512) | > loss_mel_ce: 1.7633428573608398 (nan) | > loss: 0.007150873076170683 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.1371202717523862) | > loader_time: 0.0045 (0.004619937444023396)  --> TIME: 2025-02-26 06:25:02 -- STEP: 15660/36938 -- GLOBAL_STEP: 1123800 | > loss_text_ce: 0.03858800604939461 (0.031899230238252266) | > loss_mel_ce: 2.8794071674346924 (nan) | > loss: 0.011579346843063831 (nan) | > current_lr: 5e-06 | > step_time: 0.1881 (0.13712369575049838) | > loader_time: 0.005 (0.004619837187868106)  --> TIME: 2025-02-26 06:25:12 -- STEP: 15710/36938 -- GLOBAL_STEP: 1123850 | > loss_text_ce: 0.033782683312892914 (0.03189771582089695) | > loss_mel_ce: 2.0849976539611816 (nan) | > loss: 0.00840785913169384 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.1371234249567849) | > loader_time: 0.0045 (0.0046197283275562825)  --> TIME: 2025-02-26 06:25:21 -- STEP: 15760/36938 -- GLOBAL_STEP: 1123900 | > loss_text_ce: 0.03578200563788414 (0.031902629758117716) | > loss_mel_ce: 3.261005401611328 (nan) | > loss: 0.013082491233944893 (nan) | > current_lr: 5e-06 | > step_time: 0.1893 (0.1371180647974692) | > loader_time: 0.0048 (0.004619565560732987)  --> TIME: 2025-02-26 06:25:30 -- STEP: 15810/36938 -- GLOBAL_STEP: 1123950 | > loss_text_ce: 0.027353353798389435 (0.031904271776050826) | > loss_mel_ce: 2.1840643882751465 (nan) | > loss: 0.008775467053055763 (nan) | > current_lr: 5e-06 | > step_time: 0.1769 (0.1371052376308598) | > loader_time: 0.0047 (0.004619531824496772)  --> TIME: 2025-02-26 06:25:39 -- STEP: 15860/36938 -- GLOBAL_STEP: 1124000 | > loss_text_ce: 0.023838235065340996 (0.031901880885881695) | > loss_mel_ce: 0.3052825927734375 (nan) | > loss: 0.001306035090237856 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13708926828632137) | > loader_time: 0.0045 (0.004619313684417787)  --> TIME: 2025-02-26 06:25:48 -- STEP: 15910/36938 -- GLOBAL_STEP: 1124050 | > loss_text_ce: 0.028347155079245567 (0.03190099628347157) | > loss_mel_ce: 2.2083017826080322 (nan) | > loss: 0.008875591680407524 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.1370949856370294) | > loader_time: 0.0046 (0.0046192121985271705)  --> TIME: 2025-02-26 06:25:58 -- STEP: 15960/36938 -- GLOBAL_STEP: 1124100 | > loss_text_ce: 0.039968762546777725 (0.03190470481219036) | > loss_mel_ce: 3.1428112983703613 (nan) | > loss: 0.012630079872906208 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.1370982651573076) | > loader_time: 0.0046 (0.004619120700018732)  --> TIME: 2025-02-26 06:26:07 -- STEP: 16010/36938 -- GLOBAL_STEP: 1124150 | > loss_text_ce: 0.028764422982931137 (0.03190318199385319) | > loss_mel_ce: 2.6757876873016357 (nan) | > loss: 0.010732350870966911 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.1370819134685414) | > loader_time: 0.0045 (0.004618998083153925)  --> TIME: 2025-02-26 06:26:17 -- STEP: 16060/36938 -- GLOBAL_STEP: 1124200 | > loss_text_ce: 0.027369221672415733 (0.03190662040204665) | > loss_mel_ce: 2.6943955421447754 (nan) | > loss: 0.010800654999911785 (nan) | > current_lr: 5e-06 | > step_time: 0.1974 (0.1370927812895769) | > loader_time: 0.0048 (0.004618972413715049)  --> TIME: 2025-02-26 06:26:26 -- STEP: 16110/36938 -- GLOBAL_STEP: 1124250 | > loss_text_ce: 0.036536701023578644 (0.03190827553006118) | > loss_mel_ce: 2.731673240661621 (nan) | > loss: 0.010984960943460464 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13708984098546922) | > loader_time: 0.0047 (0.0046188780863339685)  --> TIME: 2025-02-26 06:26:35 -- STEP: 16160/36938 -- GLOBAL_STEP: 1124300 | > loss_text_ce: 0.018956439569592476 (0.03190927209901966) | > loss_mel_ce: 1.7786908149719238 (nan) | > loss: 0.007133521139621735 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1370723920588446) | > loader_time: 0.0046 (0.004618755100977289)  --> TIME: 2025-02-26 06:26:45 -- STEP: 16210/36938 -- GLOBAL_STEP: 1124350 | > loss_text_ce: 0.03669964522123337 (0.03190999643292254) | > loss_mel_ce: 2.6595654487609863 (nan) | > loss: 0.010699464939534664 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13708248815001123) | > loader_time: 0.0045 (0.004618693001104079)  --> TIME: 2025-02-26 06:26:54 -- STEP: 16260/36938 -- GLOBAL_STEP: 1124400 | > loss_text_ce: 0.030883779749274254 (0.03190807826487104) | > loss_mel_ce: 3.1021244525909424 (nan) | > loss: 0.012432573363184929 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13707972272295793) | > loader_time: 0.0046 (0.004618715624087837)  --> TIME: 2025-02-26 06:27:03 -- STEP: 16310/36938 -- GLOBAL_STEP: 1124450 | > loss_text_ce: 0.030727187171578407 (0.03191032159735462) | > loss_mel_ce: 2.846179723739624 (nan) | > loss: 0.011416297405958176 (nan) | > current_lr: 5e-06 | > step_time: 0.165 (0.13707476261566928) | > loader_time: 0.0046 (0.004618757023977971)  --> TIME: 2025-02-26 06:27:12 -- STEP: 16360/36938 -- GLOBAL_STEP: 1124500 | > loss_text_ce: 0.02174031175673008 (0.031914082300008854) | > loss_mel_ce: 2.992743730545044 (nan) | > loss: 0.01196223869919777 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13706712045121602) | > loader_time: 0.0046 (0.004618711838803825)  --> TIME: 2025-02-26 06:27:22 -- STEP: 16410/36938 -- GLOBAL_STEP: 1124550 | > loss_text_ce: 0.03050278313457966 (0.031910777967230075) | > loss_mel_ce: 2.633908748626709 (nan) | > loss: 0.010573062114417553 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13705674176678143) | > loader_time: 0.0046 (0.00461868012118527)  --> TIME: 2025-02-26 06:27:31 -- STEP: 16460/36938 -- GLOBAL_STEP: 1124600 | > loss_text_ce: 0.027760948985815048 (0.03190796073732081) | > loss_mel_ce: 2.8884437084198 (nan) | > loss: 0.01157224178314209 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13706137661800638) | > loader_time: 0.0045 (0.004618614339886416)  --> TIME: 2025-02-26 06:27:40 -- STEP: 16510/36938 -- GLOBAL_STEP: 1124650 | > loss_text_ce: 0.03668654337525368 (0.03190677087372664) | > loss_mel_ce: 2.575141191482544 (nan) | > loss: 0.010364396497607231 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13704846381852295) | > loader_time: 0.0045 (0.00461855367918146)  --> TIME: 2025-02-26 06:27:50 -- STEP: 16560/36938 -- GLOBAL_STEP: 1124700 | > loss_text_ce: 0.034159738570451736 (0.03190587513151017) | > loss_mel_ce: 2.622731924057007 (nan) | > loss: 0.010543221607804298 (nan) | > current_lr: 5e-06 | > step_time: 0.1779 (0.1370705932378769) | > loader_time: 0.005 (0.004618738541280576)  --> TIME: 2025-02-26 06:27:59 -- STEP: 16610/36938 -- GLOBAL_STEP: 1124750 | > loss_text_ce: 0.029640259221196175 (0.031909958525487184) | > loss_mel_ce: 2.0924923419952393 (nan) | > loss: 0.008421161212027073 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13706911343684192) | > loader_time: 0.0044 (0.004618602714791189)  --> TIME: 2025-02-26 06:28:09 -- STEP: 16660/36938 -- GLOBAL_STEP: 1124800 | > loss_text_ce: 0.032221194356679916 (0.031908273426223155) | > loss_mel_ce: 1.5485687255859375 (nan) | > loss: 0.006272976286709309 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13706631709118278) | > loader_time: 0.0045 (0.004618534020015158)  --> TIME: 2025-02-26 06:28:18 -- STEP: 16710/36938 -- GLOBAL_STEP: 1124850 | > loss_text_ce: 0.045228634029626846 (0.03190934971919049) | > loss_mel_ce: 2.3390064239501953 (nan) | > loss: 0.009461251087486744 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13705394957181155) | > loader_time: 0.0046 (0.0046183859638508455)  --> TIME: 2025-02-26 06:28:27 -- STEP: 16760/36938 -- GLOBAL_STEP: 1124900 | > loss_text_ce: 0.0335092656314373 (0.031907486188923244) | > loss_mel_ce: 3.3558411598205566 (nan) | > loss: 0.013449803926050663 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13704278291267544) | > loader_time: 0.0045 (0.004618226585638537)  --> TIME: 2025-02-26 06:28:36 -- STEP: 16810/36938 -- GLOBAL_STEP: 1124950 | > loss_text_ce: 0.030297569930553436 (0.0319093252040205) | > loss_mel_ce: 2.7573623657226562 (nan) | > loss: 0.011062143370509148 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.1370430815014224) | > loader_time: 0.0045 (0.004618079970097116)  --> TIME: 2025-02-26 06:28:46 -- STEP: 16860/36938 -- GLOBAL_STEP: 1125000 | > loss_text_ce: 0.037123799324035645 (0.03190552480839187) | > loss_mel_ce: 2.5794270038604736 (nan) | > loss: 0.010383139364421368 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.1370420844812269) | > loader_time: 0.0046 (0.004617967993056348)  --> TIME: 2025-02-26 06:28:55 -- STEP: 16910/36938 -- GLOBAL_STEP: 1125050 | > loss_text_ce: 0.039618950337171555 (0.031907720876266) | > loss_mel_ce: 2.9951677322387695 (nan) | > loss: 0.01204280462116003 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.13704013480440821) | > loader_time: 0.0049 (0.004617871073559834)  --> TIME: 2025-02-26 06:29:04 -- STEP: 16960/36938 -- GLOBAL_STEP: 1125100 | > loss_text_ce: 0.024024980142712593 (0.03191329998893317) | > loss_mel_ce: 2.429368019104004 (nan) | > loss: 0.00973568670451641 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13703244934385686) | > loader_time: 0.0044 (0.004617668238450882)  --> TIME: 2025-02-26 06:29:13 -- STEP: 17010/36938 -- GLOBAL_STEP: 1125150 | > loss_text_ce: 0.03635237365961075 (0.03191579827477048) | > loss_mel_ce: 2.7904787063598633 (nan) | > loss: 0.01121758483350277 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13703359361959172) | > loader_time: 0.0048 (0.004617544793157548)  --> TIME: 2025-02-26 06:29:23 -- STEP: 17060/36938 -- GLOBAL_STEP: 1125200 | > loss_text_ce: 0.03423984348773956 (0.031919289105609885) | > loss_mel_ce: 3.050865411758423 (nan) | > loss: 0.012242482043802738 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13704082884794114) | > loader_time: 0.0046 (0.004617413127268591)  --> TIME: 2025-02-26 06:29:32 -- STEP: 17110/36938 -- GLOBAL_STEP: 1125250 | > loss_text_ce: 0.04957210272550583 (0.03191911709793703) | > loss_mel_ce: 2.8769211769104004 (nan) | > loss: 0.011613069102168083 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13704345980142035) | > loader_time: 0.0046 (0.004617272198095704)  --> TIME: 2025-02-26 06:29:42 -- STEP: 17160/36938 -- GLOBAL_STEP: 1125300 | > loss_text_ce: 0.02754044719040394 (0.031921586654168395) | > loss_mel_ce: 3.2501003742218018 (nan) | > loss: 0.013006512075662613 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13704097921198072) | > loader_time: 0.0046 (0.004617109832230135)  --> TIME: 2025-02-26 06:29:51 -- STEP: 17210/36938 -- GLOBAL_STEP: 1125350 | > loss_text_ce: 0.034291528165340424 (0.03192086126438866) | > loss_mel_ce: 2.720914125442505 (nan) | > loss: 0.01093335635960102 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13702631996926162) | > loader_time: 0.0045 (0.00461687829728599)  --> TIME: 2025-02-26 06:30:00 -- STEP: 17260/36938 -- GLOBAL_STEP: 1125400 | > loss_text_ce: 0.033698707818984985 (0.03192148351143959) | > loss_mel_ce: 3.1003682613372803 (nan) | > loss: 0.01243677455931902 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.13702111564671607) | > loader_time: 0.0046 (0.004616710719752824)  --> TIME: 2025-02-26 06:30:09 -- STEP: 17310/36938 -- GLOBAL_STEP: 1125450 | > loss_text_ce: 0.02256845496594906 (0.03191857740975976) | > loss_mel_ce: 2.8491828441619873 (nan) | > loss: 0.011395839042961597 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.1370213483538263) | > loader_time: 0.0045 (0.00461692905095461)  --> TIME: 2025-02-26 06:30:18 -- STEP: 17360/36938 -- GLOBAL_STEP: 1125500 | > loss_text_ce: 0.030685804784297943 (0.03192106990909731) | > loss_mel_ce: 1.5581064224243164 (nan) | > loss: 0.006304731126874685 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1370207599231176) | > loader_time: 0.0045 (0.004616768980905192)  --> TIME: 2025-02-26 06:30:28 -- STEP: 17410/36938 -- GLOBAL_STEP: 1125550 | > loss_text_ce: 0.034572865813970566 (0.03192081461066234) | > loss_mel_ce: 3.1234097480773926 (nan) | > loss: 0.012531677260994911 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13700580094341305) | > loader_time: 0.0045 (0.004616522515114512)  --> TIME: 2025-02-26 06:30:37 -- STEP: 17460/36938 -- GLOBAL_STEP: 1125600 | > loss_text_ce: 0.03543214872479439 (0.03192157595826963) | > loss_mel_ce: 2.7252237796783447 (nan) | > loss: 0.010954984463751316 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.1370119564175469) | > loader_time: 0.0049 (0.004616571114508405)  --> TIME: 2025-02-26 06:30:47 -- STEP: 17510/36938 -- GLOBAL_STEP: 1125650 | > loss_text_ce: 0.035669922828674316 (0.03192561484590318) | > loss_mel_ce: 3.163635730743408 (nan) | > loss: 0.012695658020675182 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13702572059250093) | > loader_time: 0.0045 (0.004616568198822196)  --> TIME: 2025-02-26 06:30:56 -- STEP: 17560/36938 -- GLOBAL_STEP: 1125700 | > loss_text_ce: 0.028397701680660248 (0.03192431972763042) | > loss_mel_ce: 2.452789783477783 (nan) | > loss: 0.009845983237028122 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13702590170916773) | > loader_time: 0.0045 (0.0046164689275832674)  --> TIME: 2025-02-26 06:31:05 -- STEP: 17610/36938 -- GLOBAL_STEP: 1125750 | > loss_text_ce: 0.029489269480109215 (0.031923539847112145) | > loss_mel_ce: 2.5540525913238525 (nan) | > loss: 0.01025215070694685 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13702223615034406) | > loader_time: 0.0046 (0.004616357114489674)  --> TIME: 2025-02-26 06:31:14 -- STEP: 17660/36938 -- GLOBAL_STEP: 1125800 | > loss_text_ce: 0.034062452614307404 (0.031923863776966724) | > loss_mel_ce: 2.5082156658172607 (nan) | > loss: 0.010088405571877956 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13701032371780206) | > loader_time: 0.0045 (0.0046161874504348464)  --> TIME: 2025-02-26 06:31:24 -- STEP: 17710/36938 -- GLOBAL_STEP: 1125850 | > loss_text_ce: 0.043846819549798965 (0.03192333019949074) | > loss_mel_ce: 2.5842461585998535 (nan) | > loss: 0.010428940877318382 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.137035241092012) | > loader_time: 0.0046 (0.004616159964657845)  --> TIME: 2025-02-26 06:31:34 -- STEP: 17760/36938 -- GLOBAL_STEP: 1125900 | > loss_text_ce: 0.03548913076519966 (0.03192493663476724) | > loss_mel_ce: 3.245480537414551 (nan) | > loss: 0.013019721023738384 (nan) | > current_lr: 5e-06 | > step_time: 0.1981 (0.13703399272383876) | > loader_time: 0.0049 (0.004616493389413153)  --> TIME: 2025-02-26 06:31:43 -- STEP: 17810/36938 -- GLOBAL_STEP: 1125950 | > loss_text_ce: 0.0321800597012043 (0.03192504336879664) | > loss_mel_ce: 2.6872849464416504 (nan) | > loss: 0.010791528038680553 (nan) | > current_lr: 5e-06 | > step_time: 0.1815 (0.1370456569587027) | > loader_time: 0.0048 (0.004616544837030638)  --> TIME: 2025-02-26 06:31:52 -- STEP: 17860/36938 -- GLOBAL_STEP: 1126000 | > loss_text_ce: 0.024523265659809113 (0.03192680301567086) | > loss_mel_ce: 2.7786178588867188 (nan) | > loss: 0.011123576201498508 (nan) | > current_lr: 5e-06 | > step_time: 0.1261 (0.13703363933595283) | > loader_time: 0.0045 (0.004616550115027849)  --> TIME: 2025-02-26 06:32:02 -- STEP: 17910/36938 -- GLOBAL_STEP: 1126050 | > loss_text_ce: 0.03782166913151741 (0.0319292846259554) | > loss_mel_ce: 2.7854056358337402 (nan) | > loss: 0.011203284375369549 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1370399857951836) | > loader_time: 0.0048 (0.004616574652696831)  --> TIME: 2025-02-26 06:32:11 -- STEP: 17960/36938 -- GLOBAL_STEP: 1126100 | > loss_text_ce: 0.026031384244561195 (0.03193105448240646) | > loss_mel_ce: 2.2857229709625244 (nan) | > loss: 0.009173629805445671 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.13704549602517058) | > loader_time: 0.0046 (0.004616750959298647)  --> TIME: 2025-02-26 06:32:21 -- STEP: 18010/36938 -- GLOBAL_STEP: 1126150 | > loss_text_ce: 0.04031872749328613 (0.03193234555373604) | > loss_mel_ce: 2.6446690559387207 (nan) | > loss: 0.010654713958501816 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13704132866952123) | > loader_time: 0.0046 (0.00461692939792189)  --> TIME: 2025-02-26 06:32:30 -- STEP: 18060/36938 -- GLOBAL_STEP: 1126200 | > loss_text_ce: 0.03515069931745529 (0.03193385913217238) | > loss_mel_ce: 3.098576545715332 (nan) | > loss: 0.01243542693555355 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13703943931911225) | > loader_time: 0.0045 (0.004616984906518696)  --> TIME: 2025-02-26 06:32:39 -- STEP: 18110/36938 -- GLOBAL_STEP: 1126250 | > loss_text_ce: 0.03259598836302757 (0.03193378517972952) | > loss_mel_ce: 3.4057672023773193 (nan) | > loss: 0.013644298538565636 (nan) | > current_lr: 5e-06 | > step_time: 0.1911 (0.1370439534290005) | > loader_time: 0.0048 (0.004616829244423002)  --> TIME: 2025-02-26 06:32:49 -- STEP: 18160/36938 -- GLOBAL_STEP: 1126300 | > loss_text_ce: 0.026985876262187958 (0.03193673721016141) | > loss_mel_ce: 0.9528070688247681 (nan) | > loss: 0.0038880675565451384 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13704982326419354) | > loader_time: 0.0045 (0.00461684419457606)  --> TIME: 2025-02-26 06:32:59 -- STEP: 18210/36938 -- GLOBAL_STEP: 1126350 | > loss_text_ce: 0.029190046712756157 (0.03193466504048672) | > loss_mel_ce: 3.0719387531280518 (nan) | > loss: 0.012306067161262035 (nan) | > current_lr: 5e-06 | > step_time: 0.1659 (0.13707270304100141) | > loader_time: 0.0047 (0.004616861262208859)  --> TIME: 2025-02-26 06:33:08 -- STEP: 18260/36938 -- GLOBAL_STEP: 1126400 | > loss_text_ce: 0.03312711417675018 (0.031934681957483405) | > loss_mel_ce: 2.642995595932007 (nan) | > loss: 0.010619535110890865 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13706384653068474) | > loader_time: 0.0044 (0.004616777515620271)  --> TIME: 2025-02-26 06:33:17 -- STEP: 18310/36938 -- GLOBAL_STEP: 1126450 | > loss_text_ce: 0.025158438831567764 (0.03193279270648306) | > loss_mel_ce: 2.6188852787017822 (nan) | > loss: 0.010492237284779549 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13707206224496757) | > loader_time: 0.0045 (0.004616620526295538)  --> TIME: 2025-02-26 06:33:27 -- STEP: 18360/36938 -- GLOBAL_STEP: 1126500 | > loss_text_ce: 0.027442051097750664 (0.03193365744039126) | > loss_mel_ce: 1.6939107179641724 (nan) | > loss: 0.006830765400081873 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13707175631408874) | > loader_time: 0.0047 (0.00461661432563348)  --> TIME: 2025-02-26 06:33:36 -- STEP: 18410/36938 -- GLOBAL_STEP: 1126550 | > loss_text_ce: 0.0313897468149662 (0.03193135176143319) | > loss_mel_ce: 2.5511813163757324 (nan) | > loss: 0.01024829875677824 (nan) | > current_lr: 5e-06 | > step_time: 0.1818 (0.13706597497060483) | > loader_time: 0.0048 (0.004616538433197189)  --> TIME: 2025-02-26 06:33:46 -- STEP: 18460/36938 -- GLOBAL_STEP: 1126600 | > loss_text_ce: 0.032913707196712494 (0.0319296973437082) | > loss_mel_ce: 3.0846734046936035 (nan) | > loss: 0.012371378019452095 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.13707331610139262) | > loader_time: 0.0046 (0.0046165264569795915)  --> TIME: 2025-02-26 06:33:55 -- STEP: 18510/36938 -- GLOBAL_STEP: 1126650 | > loss_text_ce: 0.04623328894376755 (0.031932995440252146) | > loss_mel_ce: 0.04232597351074219 (nan) | > loss: 0.00035142566775903106 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13707555538251698) | > loader_time: 0.0045 (0.004616445454309454)  --> TIME: 2025-02-26 06:34:05 -- STEP: 18560/36938 -- GLOBAL_STEP: 1126700 | > loss_text_ce: 0.03046857938170433 (0.03193130577289179) | > loss_mel_ce: 3.2274186611175537 (nan) | > loss: 0.012928124517202377 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.1370819915086033) | > loader_time: 0.0045 (0.00461655763973449)  --> TIME: 2025-02-26 06:34:14 -- STEP: 18610/36938 -- GLOBAL_STEP: 1126750 | > loss_text_ce: 0.04351508989930153 (0.031932312222362744) | > loss_mel_ce: 0.052978515625 (nan) | > loss: 0.00038291115197353065 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13708354717041482) | > loader_time: 0.0045 (0.004616426089193015)  --> TIME: 2025-02-26 06:34:23 -- STEP: 18660/36938 -- GLOBAL_STEP: 1126800 | > loss_text_ce: 0.03101181983947754 (0.03192834326231769) | > loss_mel_ce: 2.908674955368042 (nan) | > loss: 0.011665424332022667 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.1370890215789903) | > loader_time: 0.0046 (0.004616323059139263)  --> TIME: 2025-02-26 06:34:32 -- STEP: 18710/36938 -- GLOBAL_STEP: 1126850 | > loss_text_ce: 0.029530730098485947 (0.03192816653404871) | > loss_mel_ce: 2.0770583152770996 (nan) | > loss: 0.008359480649232864 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13708322589758073) | > loader_time: 0.0044 (0.00461618839133428)  --> TIME: 2025-02-26 06:34:42 -- STEP: 18760/36938 -- GLOBAL_STEP: 1126900 | > loss_text_ce: 0.03800981491804123 (0.03192813898980816) | > loss_mel_ce: 2.256376028060913 (nan) | > loss: 0.009104706346988678 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13708854390105743) | > loader_time: 0.0045 (0.00461610546752587)  --> TIME: 2025-02-26 06:34:51 -- STEP: 18810/36938 -- GLOBAL_STEP: 1126950 | > loss_text_ce: 0.029683200642466545 (0.03192933213544021) | > loss_mel_ce: 1.9177497625350952 (nan) | > loss: 0.007727909367531538 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.1370917524827538) | > loader_time: 0.0048 (0.004616015506259679)  --> TIME: 2025-02-26 06:35:01 -- STEP: 18860/36938 -- GLOBAL_STEP: 1127000 | > loss_text_ce: 0.03669225424528122 (0.03193133243360609) | > loss_mel_ce: 2.7615349292755127 (nan) | > loss: 0.01110407616943121 (nan) | > current_lr: 5e-06 | > step_time: 0.1648 (0.13710582964240842) | > loader_time: 0.0046 (0.004615888792788354)  --> TIME: 2025-02-26 06:35:11 -- STEP: 18910/36938 -- GLOBAL_STEP: 1127050 | > loss_text_ce: 0.03985793888568878 (0.031932214162416324) | > loss_mel_ce: 3.1558804512023926 (nan) | > loss: 0.012681501917541027 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13711298750166823) | > loader_time: 0.0046 (0.00461583604010748)  --> TIME: 2025-02-26 06:35:20 -- STEP: 18960/36938 -- GLOBAL_STEP: 1127100 | > loss_text_ce: 0.03296002000570297 (0.03193225192902253) | > loss_mel_ce: 2.5391249656677246 (nan) | > loss: 0.010206686332821846 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1371029365414812) | > loader_time: 0.0044 (0.004615681131177805)  --> TIME: 2025-02-26 06:35:29 -- STEP: 19010/36938 -- GLOBAL_STEP: 1127150 | > loss_text_ce: 0.05582204833626747 (0.031934886393791294) | > loss_mel_ce: 1.9043055772781372 (nan) | > loss: 0.007778284605592489 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13708844598753117) | > loader_time: 0.0045 (0.0046154980656976435)  --> TIME: 2025-02-26 06:35:38 -- STEP: 19060/36938 -- GLOBAL_STEP: 1127200 | > loss_text_ce: 0.032161254435777664 (0.03193504051564325) | > loss_mel_ce: 1.7119276523590088 (nan) | > loss: 0.006920988205820322 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13708094237108448) | > loader_time: 0.0044 (0.004615327618930164)  --> TIME: 2025-02-26 06:35:47 -- STEP: 19110/36938 -- GLOBAL_STEP: 1127250 | > loss_text_ce: 0.0357028990983963 (0.031934745532661776) | > loss_mel_ce: 2.1753756999969482 (nan) | > loss: 0.008774122223258018 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13707219354149464) | > loader_time: 0.0045 (0.004615149106235784)  --> TIME: 2025-02-26 06:35:56 -- STEP: 19160/36938 -- GLOBAL_STEP: 1127300 | > loss_text_ce: 0.03477383777499199 (0.031933786478262956) | > loss_mel_ce: 2.855456590652466 (nan) | > loss: 0.011469168588519096 (nan) | > current_lr: 5e-06 | > step_time: 0.168 (0.1370671201847294) | > loader_time: 0.0047 (0.004615055133504997)  --> TIME: 2025-02-26 06:36:06 -- STEP: 19210/36938 -- GLOBAL_STEP: 1127350 | > loss_text_ce: 0.031837061047554016 (0.03193660853348205) | > loss_mel_ce: 2.241872787475586 (nan) | > loss: 0.009022658690810204 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13708217697550673) | > loader_time: 0.0045 (0.004615171560081935)  --> TIME: 2025-02-26 06:36:15 -- STEP: 19260/36938 -- GLOBAL_STEP: 1127400 | > loss_text_ce: 0.03963591903448105 (0.03193885524740761) | > loss_mel_ce: 2.663111448287964 (nan) | > loss: 0.010725188069045544 (nan) | > current_lr: 5e-06 | > step_time: 0.1854 (0.13708632602622353) | > loader_time: 0.0049 (0.004615183089504844)  --> TIME: 2025-02-26 06:36:25 -- STEP: 19310/36938 -- GLOBAL_STEP: 1127450 | > loss_text_ce: 0.03821496292948723 (0.03194157683402914) | > loss_mel_ce: 3.3466849327087402 (nan) | > loss: 0.01343214325606823 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13708347014847502) | > loader_time: 0.0046 (0.004615112810416387)  --> TIME: 2025-02-26 06:36:34 -- STEP: 19360/36938 -- GLOBAL_STEP: 1127500 | > loss_text_ce: 0.03468859940767288 (0.03194150611948725) | > loss_mel_ce: 2.575117349624634 (nan) | > loss: 0.010356374084949493 (nan) | > current_lr: 5e-06 | > step_time: 0.1617 (0.13708976866785197) | > loader_time: 0.0046 (0.0046151008364582755)  --> TIME: 2025-02-26 06:36:43 -- STEP: 19410/36938 -- GLOBAL_STEP: 1127550 | > loss_text_ce: 0.03620515763759613 (0.0319408751670549) | > loss_mel_ce: 3.039862632751465 (nan) | > loss: 0.012206618674099445 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.1370828098147759) | > loader_time: 0.0046 (0.004615152391927984)  --> TIME: 2025-02-26 06:36:53 -- STEP: 19460/36938 -- GLOBAL_STEP: 1127600 | > loss_text_ce: 0.029540305957198143 (0.031940977968503315) | > loss_mel_ce: 2.377973794937134 (nan) | > loss: 0.009553628042340279 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.13708802254570032) | > loader_time: 0.0046 (0.004615196809249195)  --> TIME: 2025-02-26 06:37:02 -- STEP: 19510/36938 -- GLOBAL_STEP: 1127650 | > loss_text_ce: 0.04345857352018356 (0.03194173088980819) | > loss_mel_ce: 2.440390110015869 (nan) | > loss: 0.009856542572379112 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13708755533735592) | > loader_time: 0.0045 (0.004615137101686156)  --> TIME: 2025-02-26 06:37:11 -- STEP: 19560/36938 -- GLOBAL_STEP: 1127700 | > loss_text_ce: 0.029293494299054146 (0.03194201588506879) | > loss_mel_ce: 0.050048828125 (nan) | > loss: 0.0003148505056742579 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.1370930965570832) | > loader_time: 0.0045 (0.004615112779574395)  --> TIME: 2025-02-26 06:37:21 -- STEP: 19610/36938 -- GLOBAL_STEP: 1127750 | > loss_text_ce: 0.033223267644643784 (0.031940482108588256) | > loss_mel_ce: 2.5764267444610596 (nan) | > loss: 0.0103557538241148 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13709823607425833) | > loader_time: 0.0045 (0.004615055049699759)  --> TIME: 2025-02-26 06:37:31 -- STEP: 19660/36938 -- GLOBAL_STEP: 1127800 | > loss_text_ce: 0.040076494216918945 (0.03194183162948837) | > loss_mel_ce: 2.252617359161377 (nan) | > loss: 0.009097992442548275 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13710793743055963) | > loader_time: 0.0046 (0.004615163099729387)  --> TIME: 2025-02-26 06:37:40 -- STEP: 19710/36938 -- GLOBAL_STEP: 1127850 | > loss_text_ce: 0.026363031938672066 (0.03194181839328305) | > loss_mel_ce: 2.1304779052734375 (nan) | > loss: 0.008558893576264381 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13710104772005555) | > loader_time: 0.0049 (0.004615237711650467)  --> TIME: 2025-02-26 06:37:49 -- STEP: 19760/36938 -- GLOBAL_STEP: 1127900 | > loss_text_ce: 0.04021385684609413 (0.031945390647337604) | > loss_mel_ce: 2.7215592861175537 (nan) | > loss: 0.010959417559206486 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13708788604147557) | > loader_time: 0.0046 (0.004615135424532865)  --> TIME: 2025-02-26 06:37:58 -- STEP: 19810/36938 -- GLOBAL_STEP: 1127950 | > loss_text_ce: 0.032641492784023285 (0.031946521933420464) | > loss_mel_ce: 2.8778700828552246 (nan) | > loss: 0.011549649760127068 (nan) | > current_lr: 5e-06 | > step_time: 0.1563 (0.13708770124675212) | > loader_time: 0.0047 (0.004615411693432417)  --> TIME: 2025-02-26 06:38:07 -- STEP: 19860/36938 -- GLOBAL_STEP: 1128000 | > loss_text_ce: 0.028395215049386024 (0.031946538876337006) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13708206357552613) | > loader_time: 0.0045 (0.0046154105771344135)  --> TIME: 2025-02-26 06:38:17 -- STEP: 19910/36938 -- GLOBAL_STEP: 1128050 | > loss_text_ce: 0.045692767947912216 (0.0319463456117856) | > loss_mel_ce: 2.7044525146484375 (nan) | > loss: 0.010913275182247162 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.137087116799482) | > loader_time: 0.0045 (0.004615556301632333)  --> TIME: 2025-02-26 06:38:26 -- STEP: 19960/36938 -- GLOBAL_STEP: 1128100 | > loss_text_ce: 0.029631389304995537 (0.03194249094051422) | > loss_mel_ce: 2.830095052719116 (nan) | > loss: 0.011348121799528599 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13708799831375149) | > loader_time: 0.0046 (0.004615586530230558)  --> TIME: 2025-02-26 06:38:36 -- STEP: 20010/36938 -- GLOBAL_STEP: 1128150 | > loss_text_ce: 0.04086718335747719 (0.03193903405671719) | > loss_mel_ce: 2.4527859687805176 (nan) | > loss: 0.009895449504256248 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.1370983510181827) | > loader_time: 0.0046 (0.004615663242959659)  --> TIME: 2025-02-26 06:38:45 -- STEP: 20060/36938 -- GLOBAL_STEP: 1128200 | > loss_text_ce: 0.03319337218999863 (0.0319406825504039) | > loss_mel_ce: 3.0623209476470947 (nan) | > loss: 0.012283788062632084 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.1370949236250832) | > loader_time: 0.0047 (0.004615711535913038)  --> TIME: 2025-02-26 06:38:54 -- STEP: 20110/36938 -- GLOBAL_STEP: 1128250 | > loss_text_ce: 0.03265795856714249 (0.0319401091980285) | > loss_mel_ce: 2.251804828643799 (nan) | > loss: 0.009065329097211361 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.137086047040237) | > loader_time: 0.0046 (0.004615753554159695)  --> TIME: 2025-02-26 06:39:03 -- STEP: 20160/36938 -- GLOBAL_STEP: 1128300 | > loss_text_ce: 0.02364351414144039 (0.03193846947069117) | > loss_mel_ce: 1.1340365409851074 (nan) | > loss: 0.004593968857079744 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.1370666028369991) | > loader_time: 0.0047 (0.004615807722485248)  --> TIME: 2025-02-26 06:39:12 -- STEP: 20210/36938 -- GLOBAL_STEP: 1128350 | > loss_text_ce: 0.02929689548909664 (0.031942965213821436) | > loss_mel_ce: 2.8322501182556152 (nan) | > loss: 0.011355346068739891 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13705282573473476) | > loader_time: 0.0045 (0.004615639932435193)  --> TIME: 2025-02-26 06:39:21 -- STEP: 20260/36938 -- GLOBAL_STEP: 1128400 | > loss_text_ce: 0.033881548792123795 (0.03194302565635163) | > loss_mel_ce: 3.0152437686920166 (nan) | > loss: 0.012099704705178738 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13705228054041013) | > loader_time: 0.0048 (0.004615599381841293)  --> TIME: 2025-02-26 06:39:31 -- STEP: 20310/36938 -- GLOBAL_STEP: 1128450 | > loss_text_ce: 0.031002914533019066 (0.03194068880276679) | > loss_mel_ce: 3.3701276779174805 (nan) | > loss: 0.013496550731360912 (nan) | > current_lr: 5e-06 | > step_time: 0.1883 (0.13707276631434667) | > loader_time: 0.0049 (0.004615621306167214)  --> TIME: 2025-02-26 06:39:40 -- STEP: 20360/36938 -- GLOBAL_STEP: 1128500 | > loss_text_ce: 0.02759668231010437 (0.03194214747564546) | > loss_mel_ce: 0.23604202270507812 (nan) | > loss: 0.0010461853817105293 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1370676690796741) | > loader_time: 0.0045 (0.004615657397478171)  --> TIME: 2025-02-26 06:39:50 -- STEP: 20410/36938 -- GLOBAL_STEP: 1128550 | > loss_text_ce: 0.02829526551067829 (0.03193684413354711) | > loss_mel_ce: 3.1231138706207275 (nan) | > loss: 0.012505592778325081 (nan) | > current_lr: 5e-06 | > step_time: 0.1664 (0.1370681005386328) | > loader_time: 0.0046 (0.004615740703636961)  --> TIME: 2025-02-26 06:39:59 -- STEP: 20460/36938 -- GLOBAL_STEP: 1128600 | > loss_text_ce: 0.036126744002103806 (0.031940598456546) | > loss_mel_ce: 2.341127872467041 (nan) | > loss: 0.009433550760149956 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.1370770369690074) | > loader_time: 0.0047 (0.00461605379425302)  --> TIME: 2025-02-26 06:40:08 -- STEP: 20510/36938 -- GLOBAL_STEP: 1128650 | > loss_text_ce: 0.03501306474208832 (0.031941132643725625) | > loss_mel_ce: 2.4683218002319336 (nan) | > loss: 0.009933868423104286 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13707143595484927) | > loader_time: 0.0045 (0.00461624349866594)  --> TIME: 2025-02-26 06:40:18 -- STEP: 20560/36938 -- GLOBAL_STEP: 1128700 | > loss_text_ce: 0.03095901384949684 (0.031940224679533134) | > loss_mel_ce: 3.1169044971466064 (nan) | > loss: 0.012491523288190365 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.13707056595193798) | > loader_time: 0.0047 (0.004616263195698361)  --> TIME: 2025-02-26 06:40:27 -- STEP: 20610/36938 -- GLOBAL_STEP: 1128750 | > loss_text_ce: 0.035090796649456024 (0.0319396613406589) | > loss_mel_ce: 2.6608126163482666 (nan) | > loss: 0.010698029771447182 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1370829774244853) | > loader_time: 0.0047 (0.00461619986543836)  --> TIME: 2025-02-26 06:40:37 -- STEP: 20660/36938 -- GLOBAL_STEP: 1128800 | > loss_text_ce: 0.02530953660607338 (0.03193876072435338) | > loss_mel_ce: 2.3941850662231445 (nan) | > loss: 0.009601169265806675 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1370882737786595) | > loader_time: 0.0046 (0.004616236744407073)  --> TIME: 2025-02-26 06:40:47 -- STEP: 20710/36938 -- GLOBAL_STEP: 1128850 | > loss_text_ce: 0.030608871951699257 (0.03193868949028121) | > loss_mel_ce: 3.004882574081421 (nan) | > loss: 0.0120456013828516 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.1371041071121392) | > loader_time: 0.0046 (0.004616379323413215)  --> TIME: 2025-02-26 06:40:56 -- STEP: 20760/36938 -- GLOBAL_STEP: 1128900 | > loss_text_ce: 0.02933119237422943 (0.03193645193569038) | > loss_mel_ce: 2.7445068359375 (nan) | > loss: 0.011007294058799744 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13708957989551257) | > loader_time: 0.0045 (0.0046162880569524225)  --> TIME: 2025-02-26 06:41:05 -- STEP: 20810/36938 -- GLOBAL_STEP: 1128950 | > loss_text_ce: 0.03090020827949047 (0.031935270251871976) | > loss_mel_ce: 2.6218454837799072 (nan) | > loss: 0.010526769794523716 (nan) | > current_lr: 5e-06 | > step_time: 0.1758 (0.1370838782353111) | > loader_time: 0.0048 (0.004616327116204141)  --> TIME: 2025-02-26 06:41:14 -- STEP: 20860/36938 -- GLOBAL_STEP: 1129000 | > loss_text_ce: 0.038004182279109955 (0.03193644674049306) | > loss_mel_ce: 2.5709927082061768 (nan) | > loss: 0.01035316288471222 (nan) | > current_lr: 5e-06 | > step_time: 0.1503 (0.13708365904176276) | > loader_time: 0.0046 (0.004616456120949129)  --> TIME: 2025-02-26 06:41:23 -- STEP: 20910/36938 -- GLOBAL_STEP: 1129050 | > loss_text_ce: 0.03895173966884613 (0.03193756669723241) | > loss_mel_ce: 2.745440721511841 (nan) | > loss: 0.011049176566302776 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13707455701659305) | > loader_time: 0.0045 (0.00461640152732817)  --> TIME: 2025-02-26 06:41:33 -- STEP: 20960/36938 -- GLOBAL_STEP: 1129100 | > loss_text_ce: 0.03279395401477814 (0.03193360073167581) | > loss_mel_ce: 1.6463308334350586 (nan) | > loss: 0.006663194391876459 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13707273528548655) | > loader_time: 0.0045 (0.004616313740497331)  --> TIME: 2025-02-26 06:41:42 -- STEP: 21010/36938 -- GLOBAL_STEP: 1129150 | > loss_text_ce: 0.02485079877078533 (0.03193362548939372) | > loss_mel_ce: 2.0968334674835205 (nan) | > loss: 0.008419382385909557 (nan) | > current_lr: 5e-06 | > step_time: 0.1083 (0.13707379285974935) | > loader_time: 0.0044 (0.004616332769053488)  --> TIME: 2025-02-26 06:41:51 -- STEP: 21060/36938 -- GLOBAL_STEP: 1129200 | > loss_text_ce: 0.020064424723386765 (0.031931288990591146) | > loss_mel_ce: 2.5720126628875732 (nan) | > loss: 0.010286020115017891 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13706854312395045) | > loader_time: 0.0048 (0.004616362269674163)  --> TIME: 2025-02-26 06:42:00 -- STEP: 21110/36938 -- GLOBAL_STEP: 1129250 | > loss_text_ce: 0.03945760801434517 (0.03193416149805042) | > loss_mel_ce: 0.2604126036167145 (nan) | > loss: 0.0011899612145498395 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.1370636972529472) | > loader_time: 0.0047 (0.004616425478983587)  --> TIME: 2025-02-26 06:42:10 -- STEP: 21160/36938 -- GLOBAL_STEP: 1129300 | > loss_text_ce: 0.04297559708356857 (0.03193509747993409) | > loss_mel_ce: 2.957960367202759 (nan) | > loss: 0.011908477172255516 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13706903048634358) | > loader_time: 0.0046 (0.0046164863389020665)  --> TIME: 2025-02-26 06:42:19 -- STEP: 21210/36938 -- GLOBAL_STEP: 1129350 | > loss_text_ce: 0.03823705390095711 (0.03193658916762691) | > loss_mel_ce: 2.881256580352783 (nan) | > loss: 0.011585293337702751 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.1370612935714152) | > loader_time: 0.0047 (0.0046165400774396915)  --> TIME: 2025-02-26 06:42:29 -- STEP: 21260/36938 -- GLOBAL_STEP: 1129400 | > loss_text_ce: 0.014121141284704208 (0.03193484375858736) | > loss_mel_ce: 1.8570481538772583 (nan) | > loss: 0.007425275631248951 (nan) | > current_lr: 5e-06 | > step_time: 0.1073 (0.13706800669627994) | > loader_time: 0.0044 (0.004616734214470511)  --> TIME: 2025-02-26 06:42:38 -- STEP: 21310/36938 -- GLOBAL_STEP: 1129450 | > loss_text_ce: 0.040516357868909836 (0.031933767225240335) | > loss_mel_ce: 0.9352561235427856 (nan) | > loss: 0.0038721133023500443 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.1370630364863322) | > loader_time: 0.0044 (0.004616603030111244)  --> TIME: 2025-02-26 06:42:47 -- STEP: 21360/36938 -- GLOBAL_STEP: 1129500 | > loss_text_ce: 0.02391878515481949 (0.03193016885540006) | > loss_mel_ce: 3.016571283340454 (nan) | > loss: 0.012065437622368336 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.13705250572845787) | > loader_time: 0.0047 (0.00461649499582441)  --> TIME: 2025-02-26 06:42:56 -- STEP: 21410/36938 -- GLOBAL_STEP: 1129550 | > loss_text_ce: 0.021251995116472244 (0.03193124245555579) | > loss_mel_ce: 2.597460985183716 (nan) | > loss: 0.010391718707978725 (nan) | > current_lr: 5e-06 | > step_time: 0.1078 (0.13705440805872648) | > loader_time: 0.0045 (0.004616544136428656)  --> TIME: 2025-02-26 06:43:06 -- STEP: 21460/36938 -- GLOBAL_STEP: 1129600 | > loss_text_ce: 0.03198421373963356 (0.031931853097041676) | > loss_mel_ce: 2.5044844150543213 (nan) | > loss: 0.0100653525441885 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13705434988062778) | > loader_time: 0.0046 (0.004616604624567804)  --> TIME: 2025-02-26 06:43:15 -- STEP: 21510/36938 -- GLOBAL_STEP: 1129650 | > loss_text_ce: 0.03380267694592476 (0.031931958965366755) | > loss_mel_ce: 2.894498109817505 (nan) | > loss: 0.011620242148637772 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.13704582958762387) | > loader_time: 0.0048 (0.004616525826039618)  --> TIME: 2025-02-26 06:43:24 -- STEP: 21560/36938 -- GLOBAL_STEP: 1129700 | > loss_text_ce: 0.024019721895456314 (0.0319340350646752) | > loss_mel_ce: 2.098442792892456 (nan) | > loss: 0.008422470651566982 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.137037697351488) | > loader_time: 0.0046 (0.004616584472620863)  --> TIME: 2025-02-26 06:43:33 -- STEP: 21610/36938 -- GLOBAL_STEP: 1129750 | > loss_text_ce: 0.038238998502492905 (0.03193345448555599) | > loss_mel_ce: 2.8649845123291016 (nan) | > loss: 0.011520728468894958 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13703509077657366) | > loader_time: 0.0047 (0.004616541081366303)  --> TIME: 2025-02-26 06:43:43 -- STEP: 21660/36938 -- GLOBAL_STEP: 1129800 | > loss_text_ce: 0.030878335237503052 (0.03193347768609567) | > loss_mel_ce: 0.11493682861328125 (nan) | > loss: 0.0005786316469311714 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13703175646666613) | > loader_time: 0.0045 (0.004616581634141075)  --> TIME: 2025-02-26 06:43:52 -- STEP: 21710/36938 -- GLOBAL_STEP: 1129850 | > loss_text_ce: 0.033992912620306015 (0.031934188262275225) | > loss_mel_ce: 3.154742956161499 (nan) | > loss: 0.0126537149772048 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.13704639112603473) | > loader_time: 0.0048 (0.004616640076446184)  --> TIME: 2025-02-26 06:44:02 -- STEP: 21760/36938 -- GLOBAL_STEP: 1129900 | > loss_text_ce: 0.03291282430291176 (0.03193361214563264) | > loss_mel_ce: 1.5091896057128906 (nan) | > loss: 0.006119454745203257 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.13704141951440044) | > loader_time: 0.0046 (0.004616730693070333)  --> TIME: 2025-02-26 06:44:11 -- STEP: 21810/36938 -- GLOBAL_STEP: 1129950 | > loss_text_ce: 0.03514466807246208 (0.03193047935764871) | > loss_mel_ce: 2.5517704486846924 (nan) | > loss: 0.010265536606311798 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13703700918934883) | > loader_time: 0.0046 (0.004616795128389882)  --> TIME: 2025-02-26 06:44:20 -- STEP: 21860/36938 -- GLOBAL_STEP: 1130000 | > loss_text_ce: 0.035326991230249405 (0.03192902380342415) | > loss_mel_ce: 3.010857343673706 (nan) | > loss: 0.012088033370673656 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.1370371630076195) | > loader_time: 0.0046 (0.004616812032401189) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1130000.pth  --> TIME: 2025-02-26 06:44:35 -- STEP: 21910/36938 -- GLOBAL_STEP: 1130050 | > loss_text_ce: 0.03131315857172012 (0.03193054929777787) | > loss_mel_ce: 2.725844383239746 (nan) | > loss: 0.010941102169454098 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.1370335567601466) | > loader_time: 0.0048 (0.004616958406188083)  --> TIME: 2025-02-26 06:44:44 -- STEP: 21960/36938 -- GLOBAL_STEP: 1130100 | > loss_text_ce: 0.027201835066080093 (0.03193070161269188) | > loss_mel_ce: 1.873579740524292 (nan) | > loss: 0.007542784791439772 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.1370266692777369) | > loader_time: 0.0047 (0.0046169433654549025)  --> TIME: 2025-02-26 06:44:53 -- STEP: 22010/36938 -- GLOBAL_STEP: 1130150 | > loss_text_ce: 0.023934239521622658 (0.031926346000543115) | > loss_mel_ce: 2.2350594997406006 (nan) | > loss: 0.008964261040091515 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13700824229081426) | > loader_time: 0.0047 (0.004617004879817606)  --> TIME: 2025-02-26 06:45:03 -- STEP: 22060/36938 -- GLOBAL_STEP: 1130200 | > loss_text_ce: 0.031599707901477814 (0.031924351585264174) | > loss_mel_ce: 3.223658561706543 (nan) | > loss: 0.01291769277304411 (nan) | > current_lr: 5e-06 | > step_time: 0.1449 (0.13701222243356606) | > loader_time: 0.0047 (0.004617107735907948)  --> TIME: 2025-02-26 06:45:11 -- STEP: 22110/36938 -- GLOBAL_STEP: 1130250 | > loss_text_ce: 0.03952738270163536 (0.03192267382230207) | > loss_mel_ce: 2.4302587509155273 (nan) | > loss: 0.009800739586353302 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13699843031634396) | > loader_time: 0.0046 (0.004617120269627022)  --> TIME: 2025-02-26 06:45:21 -- STEP: 22160/36938 -- GLOBAL_STEP: 1130300 | > loss_text_ce: 0.03355515003204346 (0.031922735938456265) | > loss_mel_ce: 2.910583019256592 (nan) | > loss: 0.011683087795972824 (nan) | > current_lr: 5e-06 | > step_time: 0.1766 (0.13699581149683146) | > loader_time: 0.0048 (0.004617340115003218)  --> TIME: 2025-02-26 06:45:30 -- STEP: 22210/36938 -- GLOBAL_STEP: 1130350 | > loss_text_ce: 0.033179596066474915 (0.031921570550504624) | > loss_mel_ce: 3.2248096466064453 (nan) | > loss: 0.012928529642522335 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1369840557754501) | > loader_time: 0.0046 (0.00461726944385377)  --> TIME: 2025-02-26 06:45:39 -- STEP: 22260/36938 -- GLOBAL_STEP: 1130400 | > loss_text_ce: 0.03308650851249695 (0.03191922368395587) | > loss_mel_ce: 2.777463912963867 (nan) | > loss: 0.011152978986501694 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13698268105934505) | > loader_time: 0.0046 (0.004617235720532509)  --> TIME: 2025-02-26 06:45:49 -- STEP: 22310/36938 -- GLOBAL_STEP: 1130450 | > loss_text_ce: 0.036718301475048065 (0.03192017751435738) | > loss_mel_ce: 2.601834297180176 (nan) | > loss: 0.010470448061823845 (nan) | > current_lr: 5e-06 | > step_time: 0.1485 (0.13698986203934507) | > loader_time: 0.0047 (0.004617258627176405)  --> TIME: 2025-02-26 06:45:57 -- STEP: 22360/36938 -- GLOBAL_STEP: 1130500 | > loss_text_ce: 0.0412634015083313 (0.031919290020450644) | > loss_mel_ce: 3.032317876815796 (nan) | > loss: 0.012196751311421394 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13696946951699016) | > loader_time: 0.0046 (0.004617580872742154)  --> TIME: 2025-02-26 06:46:07 -- STEP: 22410/36938 -- GLOBAL_STEP: 1130550 | > loss_text_ce: 0.0315336138010025 (0.03191989170236135) | > loss_mel_ce: 3.146397113800049 (nan) | > loss: 0.012610837817192078 (nan) | > current_lr: 5e-06 | > step_time: 0.1694 (0.13696259053250337) | > loader_time: 0.005 (0.004617801738178995)  --> TIME: 2025-02-26 06:46:16 -- STEP: 22460/36938 -- GLOBAL_STEP: 1130600 | > loss_text_ce: 0.03526955470442772 (0.031920692047771204) | > loss_mel_ce: 0.5882440209388733 (nan) | > loss: 0.0024742602836340666 (nan) | > current_lr: 5e-06 | > step_time: 0.1794 (0.13696134938582338) | > loader_time: 0.0048 (0.004617855905531151)  --> TIME: 2025-02-26 06:46:26 -- STEP: 22510/36938 -- GLOBAL_STEP: 1130650 | > loss_text_ce: 0.030501626431941986 (0.0319200896000163) | > loss_mel_ce: 2.5213072299957275 (nan) | > loss: 0.010126226581633091 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13696635221704212) | > loader_time: 0.0046 (0.00461790980047145)  --> TIME: 2025-02-26 06:46:35 -- STEP: 22560/36938 -- GLOBAL_STEP: 1130700 | > loss_text_ce: 0.04313471540808678 (0.031922369827111996) | > loss_mel_ce: 2.545072317123413 (nan) | > loss: 0.010270663537085056 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13696456593824652) | > loader_time: 0.0046 (0.0046179382196555)  --> TIME: 2025-02-26 06:46:44 -- STEP: 22610/36938 -- GLOBAL_STEP: 1130750 | > loss_text_ce: 0.03335041552782059 (0.03192276773918299) | > loss_mel_ce: 2.4787065982818604 (nan) | > loss: 0.00996848102658987 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13695304221887636) | > loader_time: 0.0047 (0.00461800048858495)  --> TIME: 2025-02-26 06:46:53 -- STEP: 22660/36938 -- GLOBAL_STEP: 1130800 | > loss_text_ce: 0.03016558103263378 (0.031922186092191794) | > loss_mel_ce: 2.7412490844726562 (nan) | > loss: 0.010997678153216839 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.1369613686180286) | > loader_time: 0.0046 (0.00461816224816198)  --> TIME: 2025-02-26 06:47:03 -- STEP: 22710/36938 -- GLOBAL_STEP: 1130850 | > loss_text_ce: 0.024155745282769203 (0.03192106042937356) | > loss_mel_ce: 2.9145758152008057 (nan) | > loss: 0.011661633849143982 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13694762941080824) | > loader_time: 0.0046 (0.004618188307076405)  --> TIME: 2025-02-26 06:47:12 -- STEP: 22760/36938 -- GLOBAL_STEP: 1130900 | > loss_text_ce: 0.01870541460812092 (0.031920203354957856) | > loss_mel_ce: 2.8470458984375 (nan) | > loss: 0.011372029781341553 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13694195327314668) | > loader_time: 0.0045 (0.004618342427671074)  --> TIME: 2025-02-26 06:47:21 -- STEP: 22810/36938 -- GLOBAL_STEP: 1130950 | > loss_text_ce: 0.023261917755007744 (0.03191837143403666) | > loss_mel_ce: 1.0942599773406982 (nan) | > loss: 0.004434610716998577 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.13693374437702616) | > loader_time: 0.0047 (0.0046183651447923415)  --> TIME: 2025-02-26 06:47:30 -- STEP: 22860/36938 -- GLOBAL_STEP: 1131000 | > loss_text_ce: 0.036886587738990784 (0.03191610552702196) | > loss_mel_ce: 3.0701043605804443 (nan) | > loss: 0.012329330667853355 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.1369207202710298) | > loader_time: 0.0049 (0.004618520071291664)  --> TIME: 2025-02-26 06:47:39 -- STEP: 22910/36938 -- GLOBAL_STEP: 1131050 | > loss_text_ce: 0.03932436928153038 (0.0319166106479639) | > loss_mel_ce: 3.245008707046509 (nan) | > loss: 0.013033067807555199 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.1369122586568754) | > loader_time: 0.0047 (0.004618605460104918)  --> TIME: 2025-02-26 06:47:48 -- STEP: 22960/36938 -- GLOBAL_STEP: 1131100 | > loss_text_ce: 0.033328086137771606 (0.031917857994951976) | > loss_mel_ce: 2.9450085163116455 (nan) | > loss: 0.011818796396255493 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.13690603085718936) | > loader_time: 0.0048 (0.004618830464858223)  --> TIME: 2025-02-26 06:47:58 -- STEP: 23010/36938 -- GLOBAL_STEP: 1131150 | > loss_text_ce: 0.034974779933691025 (0.03191685460261804) | > loss_mel_ce: 2.9856693744659424 (nan) | > loss: 0.011986684054136276 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1369040542040532) | > loader_time: 0.0046 (0.0046188941575920385)  --> TIME: 2025-02-26 06:48:07 -- STEP: 23060/36938 -- GLOBAL_STEP: 1131200 | > loss_text_ce: 0.03461097553372383 (0.03191554566436603) | > loss_mel_ce: 1.6864100694656372 (nan) | > loss: 0.006829448975622654 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.13690034297891862) | > loader_time: 0.0045 (0.004618994556916667)  --> TIME: 2025-02-26 06:48:16 -- STEP: 23110/36938 -- GLOBAL_STEP: 1131250 | > loss_text_ce: 0.03156180679798126 (0.03191740725008924) | > loss_mel_ce: 0.048583984375 (nan) | > loss: 0.0003180388594046235 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13689711552995085) | > loader_time: 0.0045 (0.004619046507950533)  --> TIME: 2025-02-26 06:48:26 -- STEP: 23160/36938 -- GLOBAL_STEP: 1131300 | > loss_text_ce: 0.030697714537382126 (0.03191478893873406) | > loss_mel_ce: 2.8228964805603027 (nan) | > loss: 0.011323787271976471 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.1369018576305769) | > loader_time: 0.0047 (0.004619248419845667)  --> TIME: 2025-02-26 06:48:35 -- STEP: 23210/36938 -- GLOBAL_STEP: 1131350 | > loss_text_ce: 0.03494010493159294 (0.031914493102357416) | > loss_mel_ce: 2.9921982288360596 (nan) | > loss: 0.012012453749775887 (nan) | > current_lr: 5e-06 | > step_time: 0.1716 (0.13690990839781123) | > loader_time: 0.0046 (0.004619259004292961)  --> TIME: 2025-02-26 06:48:44 -- STEP: 23260/36938 -- GLOBAL_STEP: 1131400 | > loss_text_ce: 0.02956799603998661 (0.031912608537262604) | > loss_mel_ce: 2.8655169010162354 (nan) | > loss: 0.011488432995975018 (nan) | > current_lr: 5e-06 | > step_time: 0.1874 (0.13690125078142273) | > loader_time: 0.0049 (0.00461941709420033)  --> TIME: 2025-02-26 06:48:54 -- STEP: 23310/36938 -- GLOBAL_STEP: 1131450 | > loss_text_ce: 0.03275448456406593 (0.03191388146987772) | > loss_mel_ce: 1.266783356666565 (nan) | > loss: 0.0051568965427577496 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13689854936710202) | > loader_time: 0.0047 (0.004619655911884627)  --> TIME: 2025-02-26 06:49:03 -- STEP: 23360/36938 -- GLOBAL_STEP: 1131500 | > loss_text_ce: 0.030012745410203934 (0.031915145370334415) | > loss_mel_ce: 2.9232869148254395 (nan) | > loss: 0.011719443835318089 (nan) | > current_lr: 5e-06 | > step_time: 0.1514 (0.13690658673031691) | > loader_time: 0.0047 (0.00461972590586912)  --> TIME: 2025-02-26 06:49:12 -- STEP: 23410/36938 -- GLOBAL_STEP: 1131550 | > loss_text_ce: 0.03464823588728905 (0.03191636855046084) | > loss_mel_ce: 3.1292765140533447 (nan) | > loss: 0.0125552574172616 (nan) | > current_lr: 5e-06 | > step_time: 0.192 (0.1369012058519593) | > loader_time: 0.0049 (0.004619743833171171)  --> TIME: 2025-02-26 06:49:22 -- STEP: 23460/36938 -- GLOBAL_STEP: 1131600 | > loss_text_ce: 0.027738047763705254 (0.031916919224224856) | > loss_mel_ce: 2.4813835620880127 (nan) | > loss: 0.009956832975149155 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13689834628466932) | > loader_time: 0.0046 (0.004619775109075645)  --> TIME: 2025-02-26 06:49:31 -- STEP: 23510/36938 -- GLOBAL_STEP: 1131650 | > loss_text_ce: 0.031244246289134026 (0.03191508116063525) | > loss_mel_ce: 1.465112328529358 (nan) | > loss: 0.005937923211604357 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13690201854868042) | > loader_time: 0.0046 (0.0046198192022040676)  --> TIME: 2025-02-26 06:49:40 -- STEP: 23560/36938 -- GLOBAL_STEP: 1131700 | > loss_text_ce: 0.03557196632027626 (0.03191446735917382) | > loss_mel_ce: 2.806781053543091 (nan) | > loss: 0.011279179714620113 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13689890239760888) | > loader_time: 0.0046 (0.004619796601944769)  --> TIME: 2025-02-26 06:49:50 -- STEP: 23610/36938 -- GLOBAL_STEP: 1131750 | > loss_text_ce: 0.041789717972278595 (0.03191349878145919) | > loss_mel_ce: 3.023940324783325 (nan) | > loss: 0.012165596708655357 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.1368927675705245) | > loader_time: 0.0046 (0.0046197603133612646)  --> TIME: 2025-02-26 06:49:59 -- STEP: 23660/36938 -- GLOBAL_STEP: 1131800 | > loss_text_ce: 0.03326728940010071 (0.03191478483379698) | > loss_mel_ce: 2.758687734603882 (nan) | > loss: 0.011079187504947186 (nan) | > current_lr: 5e-06 | > step_time: 0.1893 (0.13690078022715796) | > loader_time: 0.0048 (0.0046198949608275)  --> TIME: 2025-02-26 06:50:09 -- STEP: 23710/36938 -- GLOBAL_STEP: 1131850 | > loss_text_ce: 0.030155634507536888 (0.03191295571901) | > loss_mel_ce: 3.4661457538604736 (nan) | > loss: 0.013874213211238384 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13689963203198488) | > loader_time: 0.0045 (0.0046199169504848835)  --> TIME: 2025-02-26 06:50:18 -- STEP: 23760/36938 -- GLOBAL_STEP: 1131900 | > loss_text_ce: 0.025951305404305458 (0.0319124026301238) | > loss_mel_ce: 2.612226724624634 (nan) | > loss: 0.010468961670994759 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.13689787867494704) | > loader_time: 0.0048 (0.004619936419255827)  --> TIME: 2025-02-26 06:50:27 -- STEP: 23810/36938 -- GLOBAL_STEP: 1131950 | > loss_text_ce: 0.03095412254333496 (0.03191619744056571) | > loss_mel_ce: 3.2370331287384033 (nan) | > loss: 0.012968203984200954 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13688200473384868) | > loader_time: 0.0047 (0.004620010449274528)  --> TIME: 2025-02-26 06:50:36 -- STEP: 23860/36938 -- GLOBAL_STEP: 1132000 | > loss_text_ce: 0.03972655534744263 (0.03191674537570318) | > loss_mel_ce: 3.11551570892334 (nan) | > loss: 0.0125208031386137 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.1368705961850144) | > loader_time: 0.0046 (0.004620102215292895)  --> TIME: 2025-02-26 06:50:45 -- STEP: 23910/36938 -- GLOBAL_STEP: 1132050 | > loss_text_ce: 0.025853101164102554 (0.03191651619432525) | > loss_mel_ce: 2.6163852214813232 (nan) | > loss: 0.010485073551535606 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1368777914458068) | > loader_time: 0.0046 (0.0046202031900971705)  --> TIME: 2025-02-26 06:50:55 -- STEP: 23960/36938 -- GLOBAL_STEP: 1132100 | > loss_text_ce: 0.030596114695072174 (0.031918362461035486) | > loss_mel_ce: 2.976447343826294 (nan) | > loss: 0.01193271204829216 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.136875864808667) | > loader_time: 0.0046 (0.00462033798380171)  --> TIME: 2025-02-26 06:51:04 -- STEP: 24010/36938 -- GLOBAL_STEP: 1132150 | > loss_text_ce: 0.03132399544119835 (0.03191787704638472) | > loss_mel_ce: 2.5215516090393066 (nan) | > loss: 0.010130459442734718 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1368689177384829) | > loader_time: 0.0046 (0.00462033394126783)  --> TIME: 2025-02-26 06:51:13 -- STEP: 24060/36938 -- GLOBAL_STEP: 1132200 | > loss_text_ce: 0.04060828685760498 (0.031916678024081546) | > loss_mel_ce: 2.8939476013183594 (nan) | > loss: 0.011645063757896423 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13688100675494116) | > loader_time: 0.0047 (0.00462062355883403)  --> TIME: 2025-02-26 06:51:22 -- STEP: 24110/36938 -- GLOBAL_STEP: 1132250 | > loss_text_ce: 0.04041736200451851 (0.031915415400289554) | > loss_mel_ce: 2.6953537464141846 (nan) | > loss: 0.010856235399842262 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13687325158785726) | > loader_time: 0.0046 (0.004620708750768721)  --> TIME: 2025-02-26 06:51:32 -- STEP: 24160/36938 -- GLOBAL_STEP: 1132300 | > loss_text_ce: 0.021552249789237976 (0.03191491298563351) | > loss_mel_ce: 1.3259729146957397 (nan) | > loss: 0.005347322206944227 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13686591411465876) | > loader_time: 0.0045 (0.00462076592919055)  --> TIME: 2025-02-26 06:51:41 -- STEP: 24210/36938 -- GLOBAL_STEP: 1132350 | > loss_text_ce: 0.032867807894945145 (0.031915976587938724) | > loss_mel_ce: 2.6753058433532715 (nan) | > loss: 0.010746722109615803 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13687004045433698) | > loader_time: 0.0047 (0.004620986061616182)  --> TIME: 2025-02-26 06:51:50 -- STEP: 24260/36938 -- GLOBAL_STEP: 1132400 | > loss_text_ce: 0.029973695054650307 (0.03191407890733201) | > loss_mel_ce: 2.4126999378204346 (nan) | > loss: 0.009693150408565998 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13686762285547055) | > loader_time: 0.0046 (0.004621110056808752)  --> TIME: 2025-02-26 06:52:00 -- STEP: 24310/36938 -- GLOBAL_STEP: 1132450 | > loss_text_ce: 0.030253520235419273 (0.031912721963105455) | > loss_mel_ce: 2.5321097373962402 (nan) | > loss: 0.010168108157813549 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13687872388440392) | > loader_time: 0.0045 (0.004621159299711607)  --> TIME: 2025-02-26 06:52:09 -- STEP: 24360/36938 -- GLOBAL_STEP: 1132500 | > loss_text_ce: 0.025675257667899132 (0.03191415537672523) | > loss_mel_ce: 2.2612946033477783 (nan) | > loss: 0.009075277484953403 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.1368743364251113) | > loader_time: 0.0046 (0.004621258422071724)  --> TIME: 2025-02-26 06:52:18 -- STEP: 24410/36938 -- GLOBAL_STEP: 1132550 | > loss_text_ce: 0.03136417269706726 (0.03191248116440975) | > loss_mel_ce: 2.925567388534546 (nan) | > loss: 0.011733856052160263 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13686516738143076) | > loader_time: 0.0046 (0.004621322376715373)  --> TIME: 2025-02-26 06:52:28 -- STEP: 24460/36938 -- GLOBAL_STEP: 1132600 | > loss_text_ce: 0.02841888926923275 (0.031913484874785666) | > loss_mel_ce: 2.757413148880005 (nan) | > loss: 0.011054889298975468 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.1368812405955863) | > loader_time: 0.0046 (0.004621369236332432)  --> TIME: 2025-02-26 06:52:38 -- STEP: 24510/36938 -- GLOBAL_STEP: 1132650 | > loss_text_ce: 0.035799816250801086 (0.03191598694279872) | > loss_mel_ce: 2.62791109085083 (nan) | > loss: 0.010570282116532326 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13688580812798995) | > loader_time: 0.0046 (0.004621401586030674)  --> TIME: 2025-02-26 06:52:48 -- STEP: 24560/36938 -- GLOBAL_STEP: 1132700 | > loss_text_ce: 0.037121571600437164 (0.03191525002396229) | > loss_mel_ce: 2.8696041107177734 (nan) | > loss: 0.011534626595675945 (nan) | > current_lr: 5e-06 | > step_time: 0.1754 (0.13690753684952542) | > loader_time: 0.0048 (0.004621536257989253)  --> TIME: 2025-02-26 06:52:57 -- STEP: 24610/36938 -- GLOBAL_STEP: 1132750 | > loss_text_ce: 0.039238911122083664 (0.031916033458994304) | > loss_mel_ce: 1.5604044198989868 (nan) | > loss: 0.006347791291773319 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.13690894382458235) | > loader_time: 0.0047 (0.004621546571693603)  --> TIME: 2025-02-26 06:53:07 -- STEP: 24660/36938 -- GLOBAL_STEP: 1132800 | > loss_text_ce: 0.03260685130953789 (0.03191447600111282) | > loss_mel_ce: 2.5761971473693848 (nan) | > loss: 0.010352397337555885 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.1369182566269957) | > loader_time: 0.0047 (0.0046216167189295785)  --> TIME: 2025-02-26 06:53:16 -- STEP: 24710/36938 -- GLOBAL_STEP: 1132850 | > loss_text_ce: 0.03650107607245445 (0.03191559617405765) | > loss_mel_ce: 2.2692484855651855 (nan) | > loss: 0.00914980098605156 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.1369244077339777) | > loader_time: 0.0046 (0.004621698353658184)  --> TIME: 2025-02-26 06:53:25 -- STEP: 24760/36938 -- GLOBAL_STEP: 1132900 | > loss_text_ce: 0.033322904258966446 (0.03191359047458256) | > loss_mel_ce: 1.9665583372116089 (nan) | > loss: 0.00793603714555502 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.1369219183729233) | > loader_time: 0.0046 (0.004621758580400416)  --> TIME: 2025-02-26 06:53:35 -- STEP: 24810/36938 -- GLOBAL_STEP: 1132950 | > loss_text_ce: 0.030679285526275635 (0.031914547749850454) | > loss_mel_ce: 2.097994565963745 (nan) | > loss: 0.008447119034826756 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1369241085229313) | > loader_time: 0.0048 (0.0046218396770718794)  --> TIME: 2025-02-26 06:53:45 -- STEP: 24860/36938 -- GLOBAL_STEP: 1133000 | > loss_text_ce: 0.03431796655058861 (0.03191427800025927) | > loss_mel_ce: 2.417069435119629 (nan) | > loss: 0.009727728553116322 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.13693504571339266) | > loader_time: 0.0048 (0.004621959854687603)  --> TIME: 2025-02-26 06:53:54 -- STEP: 24910/36938 -- GLOBAL_STEP: 1133050 | > loss_text_ce: 0.033651113510131836 (0.03191252052955973) | > loss_mel_ce: 3.5030760765075684 (nan) | > loss: 0.014034632593393326 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.13693813880444178) | > loader_time: 0.0047 (0.004622023797619064)  --> TIME: 2025-02-26 06:54:04 -- STEP: 24960/36938 -- GLOBAL_STEP: 1133100 | > loss_text_ce: 0.03588774427771568 (0.031916114985823675) | > loss_mel_ce: 1.8195312023162842 (nan) | > loss: 0.007362774107605219 (nan) | > current_lr: 5e-06 | > step_time: 0.1123 (0.1369437786726618) | > loader_time: 0.0044 (0.004622107018262913)  --> TIME: 2025-02-26 06:54:13 -- STEP: 25010/36938 -- GLOBAL_STEP: 1133150 | > loss_text_ce: 0.030880838632583618 (0.031913765441136464) | > loss_mel_ce: 2.109375 (nan) | > loss: 0.00849307980388403 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13694958051935496) | > loader_time: 0.0047 (0.004622242022685552)  --> TIME: 2025-02-26 06:54:22 -- STEP: 25060/36938 -- GLOBAL_STEP: 1133200 | > loss_text_ce: 0.03888898342847824 (0.03191725050388347) | > loss_mel_ce: 2.6158506870269775 (nan) | > loss: 0.010534681379795074 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13694490697606526) | > loader_time: 0.0047 (0.00462238859959059)  --> TIME: 2025-02-26 06:54:32 -- STEP: 25110/36938 -- GLOBAL_STEP: 1133250 | > loss_text_ce: 0.02943442575633526 (0.03191856864983719) | > loss_mel_ce: 0.09326171875 (nan) | > loss: 0.000486889504827559 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13694907424460373) | > loader_time: 0.0048 (0.004622404084458185)  --> TIME: 2025-02-26 06:54:41 -- STEP: 25160/36938 -- GLOBAL_STEP: 1133300 | > loss_text_ce: 0.025530308485031128 (0.031919472034286404) | > loss_mel_ce: 3.1231822967529297 (nan) | > loss: 0.012494891881942749 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13695283420324708) | > loader_time: 0.0045 (0.00462239370437039)  --> TIME: 2025-02-26 06:54:51 -- STEP: 25210/36938 -- GLOBAL_STEP: 1133350 | > loss_text_ce: 0.03439394012093544 (0.031920707516094556) | > loss_mel_ce: 2.109698534011841 (nan) | > loss: 0.008508304134011269 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1369536091154977) | > loader_time: 0.0045 (0.004622419558551178)  --> TIME: 2025-02-26 06:55:00 -- STEP: 25260/36938 -- GLOBAL_STEP: 1133400 | > loss_text_ce: 0.026154067367315292 (0.03192107416541933) | > loss_mel_ce: 2.7252416610717773 (nan) | > loss: 0.010918237268924713 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13695552900303756) | > loader_time: 0.0045 (0.004622449302899859)  --> TIME: 2025-02-26 06:55:09 -- STEP: 25310/36938 -- GLOBAL_STEP: 1133450 | > loss_text_ce: 0.022103458642959595 (0.03192111624228914) | > loss_mel_ce: 2.882293939590454 (nan) | > loss: 0.01152538787573576 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13695106183243846) | > loader_time: 0.0045 (0.004622424086861242)  --> TIME: 2025-02-26 06:55:19 -- STEP: 25360/36938 -- GLOBAL_STEP: 1133500 | > loss_text_ce: 0.028756938874721527 (0.03192149375797458) | > loss_mel_ce: 2.8282878398895264 (nan) | > loss: 0.011337479576468468 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1369557558456053) | > loader_time: 0.0045 (0.004622417537947928)  --> TIME: 2025-02-26 06:55:28 -- STEP: 25410/36938 -- GLOBAL_STEP: 1133550 | > loss_text_ce: 0.025537023320794106 (0.031920019354831165) | > loss_mel_ce: 2.5497171878814697 (nan) | > loss: 0.010219262912869453 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.1369649056607651) | > loader_time: 0.0045 (0.004622494888230594)  --> TIME: 2025-02-26 06:55:38 -- STEP: 25460/36938 -- GLOBAL_STEP: 1133600 | > loss_text_ce: 0.039477746933698654 (0.03192015431056109) | > loss_mel_ce: 3.0401365756988525 (nan) | > loss: 0.012220692820847034 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13696607499261218) | > loader_time: 0.0049 (0.004622544965122214)  --> TIME: 2025-02-26 06:55:47 -- STEP: 25510/36938 -- GLOBAL_STEP: 1133650 | > loss_text_ce: 0.03292981535196304 (0.031920913331610845) | > loss_mel_ce: 3.2378668785095215 (nan) | > loss: 0.012979352846741676 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13696521476220738) | > loader_time: 0.0045 (0.004622652912924958)  --> TIME: 2025-02-26 06:55:56 -- STEP: 25560/36938 -- GLOBAL_STEP: 1133700 | > loss_text_ce: 0.032819606363773346 (0.03192014013043966) | > loss_mel_ce: 3.091548442840576 (nan) | > loss: 0.012398285791277885 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13695754581400682) | > loader_time: 0.0048 (0.004622768926695099)  --> TIME: 2025-02-26 06:56:05 -- STEP: 25610/36938 -- GLOBAL_STEP: 1133750 | > loss_text_ce: 0.035048533231019974 (0.031918889844164015) | > loss_mel_ce: 3.3710200786590576 (nan) | > loss: 0.013516145758330822 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.13695923317600536) | > loader_time: 0.0046 (0.004622783915971412)  --> TIME: 2025-02-26 06:56:15 -- STEP: 25660/36938 -- GLOBAL_STEP: 1133800 | > loss_text_ce: 0.03010771982371807 (0.031916610158933566) | > loss_mel_ce: 3.022507905960083 (nan) | > loss: 0.012113554403185844 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13695573895031463) | > loader_time: 0.0046 (0.004622974854749524)  --> TIME: 2025-02-26 06:56:24 -- STEP: 25710/36938 -- GLOBAL_STEP: 1133850 | > loss_text_ce: 0.026053287088871002 (0.03191754509035966) | > loss_mel_ce: 2.761361837387085 (nan) | > loss: 0.011061171069741249 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13697029238589228) | > loader_time: 0.0045 (0.004623100906900275)  --> TIME: 2025-02-26 06:56:34 -- STEP: 25760/36938 -- GLOBAL_STEP: 1133900 | > loss_text_ce: 0.03200080245733261 (0.03191633998913125) | > loss_mel_ce: 1.0381622314453125 (nan) | > loss: 0.004246678669005632 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13696805025850037) | > loader_time: 0.0047 (0.004623183126775366)  --> TIME: 2025-02-26 06:56:43 -- STEP: 25810/36938 -- GLOBAL_STEP: 1133950 | > loss_text_ce: 0.02360793948173523 (0.03191856165427008) | > loss_mel_ce: 0.0383562371134758 (nan) | > loss: 0.00024588959058746696 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13695798503110154) | > loader_time: 0.0045 (0.0046232383595746955)  --> TIME: 2025-02-26 06:56:52 -- STEP: 25860/36938 -- GLOBAL_STEP: 1134000 | > loss_text_ce: 0.030135992914438248 (0.03192071358000541) | > loss_mel_ce: 2.9208052158355713 (nan) | > loss: 0.011710084974765778 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13696557236381604) | > loader_time: 0.0047 (0.004623297546044476)  --> TIME: 2025-02-26 06:57:02 -- STEP: 25910/36938 -- GLOBAL_STEP: 1134050 | > loss_text_ce: 0.032415568828582764 (0.03192119102954543) | > loss_mel_ce: 2.2984862327575684 (nan) | > loss: 0.0092496108263731 (nan) | > current_lr: 5e-06 | > step_time: 0.163 (0.13696727879670376) | > loader_time: 0.0045 (0.004623303832993076)  --> TIME: 2025-02-26 06:57:11 -- STEP: 25960/36938 -- GLOBAL_STEP: 1134100 | > loss_text_ce: 0.03317033872008324 (0.03191910224345351) | > loss_mel_ce: 2.960906505584717 (nan) | > loss: 0.01188125740736723 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13697155523924706) | > loader_time: 0.0045 (0.004623374769977859)  --> TIME: 2025-02-26 06:57:21 -- STEP: 26010/36938 -- GLOBAL_STEP: 1134150 | > loss_text_ce: 0.03384755179286003 (0.03192024259910035) | > loss_mel_ce: 2.716261625289917 (nan) | > loss: 0.010913132689893246 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.13697199644560284) | > loader_time: 0.005 (0.004623432674576612)  --> TIME: 2025-02-26 06:57:30 -- STEP: 26060/36938 -- GLOBAL_STEP: 1134200 | > loss_text_ce: 0.03358102962374687 (0.03192083201466204) | > loss_mel_ce: 2.560441017150879 (nan) | > loss: 0.010293738916516304 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13696760493613006) | > loader_time: 0.0046 (0.004623474739920782)  --> TIME: 2025-02-26 06:57:39 -- STEP: 26110/36938 -- GLOBAL_STEP: 1134250 | > loss_text_ce: 0.03335902839899063 (0.03191773714101767) | > loss_mel_ce: 2.023322105407715 (nan) | > loss: 0.008161433972418308 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13697057032393412) | > loader_time: 0.0046 (0.004623575623306618)  --> TIME: 2025-02-26 06:57:49 -- STEP: 26160/36938 -- GLOBAL_STEP: 1134300 | > loss_text_ce: 0.04011603817343712 (0.031918950452944995) | > loss_mel_ce: 2.860182523727417 (nan) | > loss: 0.0115091223269701 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13697378608976324) | > loader_time: 0.0045 (0.00462363749651368)  --> TIME: 2025-02-26 06:57:58 -- STEP: 26210/36938 -- GLOBAL_STEP: 1134350 | > loss_text_ce: 0.028929557651281357 (0.03191855376545816) | > loss_mel_ce: 3.0025792121887207 (nan) | > loss: 0.012029796838760376 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13696152105990034) | > loader_time: 0.0046 (0.004623586173641538)  --> TIME: 2025-02-26 06:58:07 -- STEP: 26260/36938 -- GLOBAL_STEP: 1134400 | > loss_text_ce: 0.033041857182979584 (0.03192031732763922) | > loss_mel_ce: 2.2905080318450928 (nan) | > loss: 0.00922043714672327 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13695967684968788) | > loader_time: 0.0044 (0.004623568884216897)  --> TIME: 2025-02-26 06:58:16 -- STEP: 26310/36938 -- GLOBAL_STEP: 1134450 | > loss_text_ce: 0.028790006414055824 (0.031919618775832416) | > loss_mel_ce: 3.4548611640930176 (nan) | > loss: 0.01382401306182146 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13695874589335014) | > loader_time: 0.0046 (0.004623536282462871)  --> TIME: 2025-02-26 06:58:25 -- STEP: 26360/36938 -- GLOBAL_STEP: 1134500 | > loss_text_ce: 0.020459100604057312 (0.03191627685488693) | > loss_mel_ce: 2.774416208267212 (nan) | > loss: 0.011090775951743126 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.1369490980378414) | > loader_time: 0.0045 (0.004623477846067468)  --> TIME: 2025-02-26 06:58:35 -- STEP: 26410/36938 -- GLOBAL_STEP: 1134550 | > loss_text_ce: 0.03605659678578377 (0.03191523793044504) | > loss_mel_ce: 2.9185805320739746 (nan) | > loss: 0.011724750511348248 (nan) | > current_lr: 5e-06 | > step_time: 0.1888 (0.1369530571179607) | > loader_time: 0.0048 (0.004623542568621319)  --> TIME: 2025-02-26 06:58:44 -- STEP: 26460/36938 -- GLOBAL_STEP: 1134600 | > loss_text_ce: 0.02824798785150051 (0.03191245187694128) | > loss_mel_ce: 1.8200249671936035 (nan) | > loss: 0.0073344167321920395 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13694762672696814) | > loader_time: 0.0046 (0.004623553713252829)  --> TIME: 2025-02-26 06:58:53 -- STEP: 26510/36938 -- GLOBAL_STEP: 1134650 | > loss_text_ce: 0.0333121083676815 (0.03190867315344755) | > loss_mel_ce: 2.523012399673462 (nan) | > loss: 0.010144145227968693 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13694642847414457) | > loader_time: 0.0045 (0.0046236252703787065)  --> TIME: 2025-02-26 06:59:03 -- STEP: 26560/36938 -- GLOBAL_STEP: 1134700 | > loss_text_ce: 0.0222721379250288 (0.031908798108932185) | > loss_mel_ce: 1.9632747173309326 (nan) | > loss: 0.00787915475666523 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13694602462003055) | > loader_time: 0.0047 (0.004623698685542632)  --> TIME: 2025-02-26 06:59:12 -- STEP: 26610/36938 -- GLOBAL_STEP: 1134750 | > loss_text_ce: 0.026539500802755356 (0.03191002180932142) | > loss_mel_ce: 1.9364593029022217 (nan) | > loss: 0.007789677940309048 (nan) | > current_lr: 5e-06 | > step_time: 0.1078 (0.13694934268859693) | > loader_time: 0.0045 (0.004623883212253306)  --> TIME: 2025-02-26 06:59:21 -- STEP: 26660/36938 -- GLOBAL_STEP: 1134800 | > loss_text_ce: 0.04414847865700722 (0.03191002949936751) | > loss_mel_ce: 2.779803514480591 (nan) | > loss: 0.011206159368157387 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.136942492216401) | > loader_time: 0.0046 (0.00462401804848889)  --> TIME: 2025-02-26 06:59:30 -- STEP: 26710/36938 -- GLOBAL_STEP: 1134850 | > loss_text_ce: 0.03690159320831299 (0.03190922808349807) | > loss_mel_ce: 2.5281946659088135 (nan) | > loss: 0.010178954340517521 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13693249583556769) | > loader_time: 0.0046 (0.004623955477047203)  --> TIME: 2025-02-26 06:59:39 -- STEP: 26760/36938 -- GLOBAL_STEP: 1134900 | > loss_text_ce: 0.03570853918790817 (0.03190861708221483) | > loss_mel_ce: 2.3817808628082275 (nan) | > loss: 0.00959321204572916 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13692081182170554) | > loader_time: 0.0046 (0.004623882537109061)  --> TIME: 2025-02-26 06:59:48 -- STEP: 26810/36938 -- GLOBAL_STEP: 1134950 | > loss_text_ce: 0.025859786197543144 (0.03190995610851613) | > loss_mel_ce: 3.1936023235321045 (nan) | > loss: 0.012775644659996033 (nan) | > current_lr: 5e-06 | > step_time: 0.1713 (0.1369145865050864) | > loader_time: 0.0048 (0.004624051364754954)  --> TIME: 2025-02-26 06:59:58 -- STEP: 26860/36938 -- GLOBAL_STEP: 1135000 | > loss_text_ce: 0.03362679108977318 (0.03190820631467098) | > loss_mel_ce: 2.6191701889038086 (nan) | > loss: 0.010526972822844982 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.13691506219815638) | > loader_time: 0.0047 (0.004624187316099192)  --> TIME: 2025-02-26 07:00:07 -- STEP: 26910/36938 -- GLOBAL_STEP: 1135050 | > loss_text_ce: 0.03750133514404297 (0.031908766459189164) | > loss_mel_ce: 1.8240320682525635 (nan) | > loss: 0.007387037854641676 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13691566064870764) | > loader_time: 0.0048 (0.0046245438332275755)  --> TIME: 2025-02-26 07:00:16 -- STEP: 26960/36938 -- GLOBAL_STEP: 1135100 | > loss_text_ce: 0.02869991771876812 (0.0319072085272767) | > loss_mel_ce: 1.4786758422851562 (nan) | > loss: 0.005981650203466415 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13691351499507984) | > loader_time: 0.0047 (0.004624696902775959)  --> TIME: 2025-02-26 07:00:26 -- STEP: 27010/36938 -- GLOBAL_STEP: 1135150 | > loss_text_ce: 0.020670251920819283 (0.03191026565510198) | > loss_mel_ce: 1.6060372591018677 (nan) | > loss: 0.00645518908277154 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13691876029932973) | > loader_time: 0.0047 (0.004624885993725885)  --> TIME: 2025-02-26 07:00:35 -- STEP: 27060/36938 -- GLOBAL_STEP: 1135200 | > loss_text_ce: 0.05520065128803253 (0.03191134574367402) | > loss_mel_ce: 1.667562484741211 (nan) | > loss: 0.006836362183094025 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13691819285958773) | > loader_time: 0.0046 (0.004624873562733148)  --> TIME: 2025-02-26 07:00:44 -- STEP: 27110/36938 -- GLOBAL_STEP: 1135250 | > loss_text_ce: 0.035520873963832855 (0.03191192750968461) | > loss_mel_ce: 2.673203706741333 (nan) | > loss: 0.01074890699237585 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13691695913183252) | > loader_time: 0.0046 (0.004624869470796818)  --> TIME: 2025-02-26 07:00:54 -- STEP: 27160/36938 -- GLOBAL_STEP: 1135300 | > loss_text_ce: 0.03614326938986778 (0.031910457401618665) | > loss_mel_ce: 1.6595357656478882 (nan) | > loss: 0.006728885695338249 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13691029062327298) | > loader_time: 0.0045 (0.004624905940885909)  --> TIME: 2025-02-26 07:01:03 -- STEP: 27210/36938 -- GLOBAL_STEP: 1135350 | > loss_text_ce: 0.03222472965717316 (0.03191156801221718) | > loss_mel_ce: 2.623725652694702 (nan) | > loss: 0.010539486072957516 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13691367551887001) | > loader_time: 0.0046 (0.004624933269434827)  --> TIME: 2025-02-26 07:01:12 -- STEP: 27260/36938 -- GLOBAL_STEP: 1135400 | > loss_text_ce: 0.031339965760707855 (0.031913521585942246) | > loss_mel_ce: 2.825613260269165 (nan) | > loss: 0.011337116360664368 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.1369064450963664) | > loader_time: 0.0046 (0.004625078377615234)  --> TIME: 2025-02-26 07:01:21 -- STEP: 27310/36938 -- GLOBAL_STEP: 1135450 | > loss_text_ce: 0.025171564891934395 (0.031913940357136364) | > loss_mel_ce: 2.174103260040283 (nan) | > loss: 0.008727281354367733 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13690085936622778) | > loader_time: 0.0045 (0.004625139442480159)  --> TIME: 2025-02-26 07:01:30 -- STEP: 27360/36938 -- GLOBAL_STEP: 1135500 | > loss_text_ce: 0.025381993502378464 (0.03191290054519992) | > loss_mel_ce: 0.0061492919921875 (nan) | > loss: 0.00012512416287790984 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.13690020000202588) | > loader_time: 0.0048 (0.004625225258849515)  --> TIME: 2025-02-26 07:01:40 -- STEP: 27410/36938 -- GLOBAL_STEP: 1135550 | > loss_text_ce: 0.02935684658586979 (0.03191233034966326) | > loss_mel_ce: 2.520108938217163 (nan) | > loss: 0.010116929188370705 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13689637405947994) | > loader_time: 0.0047 (0.004625242559286153)  --> TIME: 2025-02-26 07:01:49 -- STEP: 27460/36938 -- GLOBAL_STEP: 1135600 | > loss_text_ce: 0.022342341020703316 (0.03191147478411306) | > loss_mel_ce: 2.4265799522399902 (nan) | > loss: 0.009717946872115135 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13689544361521505) | > loader_time: 0.0046 (0.004625279931195526)  --> TIME: 2025-02-26 07:01:59 -- STEP: 27510/36938 -- GLOBAL_STEP: 1135650 | > loss_text_ce: 0.03930189460515976 (0.03191331152437556) | > loss_mel_ce: 3.372840404510498 (nan) | > loss: 0.013540247455239296 (nan) | > current_lr: 5e-06 | > step_time: 0.1689 (0.1369022330298329) | > loader_time: 0.0048 (0.004625410706032317)  --> TIME: 2025-02-26 07:02:08 -- STEP: 27560/36938 -- GLOBAL_STEP: 1135700 | > loss_text_ce: 0.04286102205514908 (0.031912564089405515) | > loss_mel_ce: 2.3995234966278076 (nan) | > loss: 0.009692002087831497 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13690285572811864) | > loader_time: 0.0045 (0.004625439522747371)  --> TIME: 2025-02-26 07:02:17 -- STEP: 27610/36938 -- GLOBAL_STEP: 1135750 | > loss_text_ce: 0.033381298184394836 (0.031913188810580646) | > loss_mel_ce: 2.3674156665802 (nan) | > loss: 0.009526972658932209 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1369082114179249) | > loader_time: 0.0045 (0.004625513673649015)  --> TIME: 2025-02-26 07:02:27 -- STEP: 27660/36938 -- GLOBAL_STEP: 1135800 | > loss_text_ce: 0.03876937925815582 (0.03191359300033028) | > loss_mel_ce: 2.5161983966827393 (nan) | > loss: 0.010138762183487415 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13690871604524957) | > loader_time: 0.0045 (0.004625543958452925)  --> TIME: 2025-02-26 07:02:36 -- STEP: 27710/36938 -- GLOBAL_STEP: 1135850 | > loss_text_ce: 0.03479161486029625 (0.03191413250733304) | > loss_mel_ce: 2.6907074451446533 (nan) | > loss: 0.010815473273396492 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13691040798457038) | > loader_time: 0.0046 (0.0046255962894264785)  --> TIME: 2025-02-26 07:02:46 -- STEP: 27760/36938 -- GLOBAL_STEP: 1135900 | > loss_text_ce: 0.028998028486967087 (0.031915054462047565) | > loss_mel_ce: 2.5112998485565186 (nan) | > loss: 0.010080548003315926 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13691416406837548) | > loader_time: 0.0045 (0.004625731904156261)  --> TIME: 2025-02-26 07:02:55 -- STEP: 27810/36938 -- GLOBAL_STEP: 1135950 | > loss_text_ce: 0.02324390597641468 (0.03191446910094212) | > loss_mel_ce: 2.605003833770752 (nan) | > loss: 0.010429555550217628 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13691309407990973) | > loader_time: 0.0049 (0.0046258711034726)  --> TIME: 2025-02-26 07:03:04 -- STEP: 27860/36938 -- GLOBAL_STEP: 1136000 | > loss_text_ce: 0.0425226055085659 (0.03191572215531101) | > loss_mel_ce: 3.5231781005859375 (nan) | > loss: 0.01414960715919733 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13691314762646678) | > loader_time: 0.0047 (0.004625905552786705)  --> TIME: 2025-02-26 07:03:14 -- STEP: 27910/36938 -- GLOBAL_STEP: 1136050 | > loss_text_ce: 0.027549797669053078 (0.03191563676203829) | > loss_mel_ce: 2.8387012481689453 (nan) | > loss: 0.011374012567102909 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.13691601594346026) | > loader_time: 0.0048 (0.00462591045862113)  --> TIME: 2025-02-26 07:03:23 -- STEP: 27960/36938 -- GLOBAL_STEP: 1136100 | > loss_text_ce: 0.03492409735918045 (0.03191479209531364) | > loss_mel_ce: 1.7033830881118774 (nan) | > loss: 0.00689804507419467 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13691351565509666) | > loader_time: 0.0046 (0.004625917598072202)  --> TIME: 2025-02-26 07:03:32 -- STEP: 28010/36938 -- GLOBAL_STEP: 1136150 | > loss_text_ce: 0.031536396592855453 (0.03191200614920479) | > loss_mel_ce: 1.9640547037124634 (nan) | > loss: 0.007919012568891048 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13691575940869893) | > loader_time: 0.0045 (0.004626026574052764)  --> TIME: 2025-02-26 07:03:42 -- STEP: 28060/36938 -- GLOBAL_STEP: 1136200 | > loss_text_ce: 0.026497304439544678 (0.03191317371398084) | > loss_mel_ce: 2.6792166233062744 (nan) | > loss: 0.010736960917711258 (nan) | > current_lr: 5e-06 | > step_time: 0.1759 (0.13691649979892423) | > loader_time: 0.0047 (0.00462603857070313)  --> TIME: 2025-02-26 07:03:51 -- STEP: 28110/36938 -- GLOBAL_STEP: 1136250 | > loss_text_ce: 0.03647080808877945 (0.031912091354551524) | > loss_mel_ce: 2.3124027252197266 (nan) | > loss: 0.009320927783846855 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13692524484025692) | > loader_time: 0.0046 (0.0046260688619569465)  --> TIME: 2025-02-26 07:04:00 -- STEP: 28160/36938 -- GLOBAL_STEP: 1136300 | > loss_text_ce: 0.0373569056391716 (0.03191239732368551) | > loss_mel_ce: 2.8080286979675293 (nan) | > loss: 0.01129121333360672 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.1369195964610706) | > loader_time: 0.0046 (0.004626092179255044)  --> TIME: 2025-02-26 07:04:10 -- STEP: 28210/36938 -- GLOBAL_STEP: 1136350 | > loss_text_ce: 0.039518311619758606 (0.031910574813668796) | > loss_mel_ce: 3.117480516433716 (nan) | > loss: 0.012527774088084698 (nan) | > current_lr: 5e-06 | > step_time: 0.1877 (0.13691725547497036) | > loader_time: 0.0048 (0.004626115261768872)  --> TIME: 2025-02-26 07:04:20 -- STEP: 28260/36938 -- GLOBAL_STEP: 1136400 | > loss_text_ce: 0.030318936333060265 (0.031910994878501395) | > loss_mel_ce: 2.781891345977783 (nan) | > loss: 0.011159565299749374 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.1369328534189719) | > loader_time: 0.0049 (0.004626226290185141)  --> TIME: 2025-02-26 07:04:29 -- STEP: 28310/36938 -- GLOBAL_STEP: 1136450 | > loss_text_ce: 0.036934010684490204 (0.03191283962935075) | > loss_mel_ce: 3.3130176067352295 (nan) | > loss: 0.013293459080159664 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13692907336237345) | > loader_time: 0.0046 (0.00462619335312002)  --> TIME: 2025-02-26 07:04:38 -- STEP: 28360/36938 -- GLOBAL_STEP: 1136500 | > loss_text_ce: 0.032537057995796204 (0.03191267382438919) | > loss_mel_ce: 2.8752307891845703 (nan) | > loss: 0.011538761667907238 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13692863658385765) | > loader_time: 0.0047 (0.004626339177991162)  --> TIME: 2025-02-26 07:04:48 -- STEP: 28410/36938 -- GLOBAL_STEP: 1136550 | > loss_text_ce: 0.03365321829915047 (0.031913454254872096) | > loss_mel_ce: 0.174530029296875 (nan) | > loss: 0.0008261240436695516 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13693165781631342) | > loader_time: 0.0046 (0.004626413559838099)  --> TIME: 2025-02-26 07:04:57 -- STEP: 28460/36938 -- GLOBAL_STEP: 1136600 | > loss_text_ce: 0.028081195428967476 (0.03191380966651081) | > loss_mel_ce: 2.663792610168457 (nan) | > loss: 0.010682039894163609 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13693271350961084) | > loader_time: 0.0046 (0.004626472475490174)  --> TIME: 2025-02-26 07:05:07 -- STEP: 28510/36938 -- GLOBAL_STEP: 1136650 | > loss_text_ce: 0.0317702516913414 (0.031911547342200905) | > loss_mel_ce: 2.8523013591766357 (nan) | > loss: 0.011444728821516037 (nan) | > current_lr: 5e-06 | > step_time: 0.1832 (0.13693665702984362) | > loader_time: 0.0048 (0.004626617277768226)  --> TIME: 2025-02-26 07:05:16 -- STEP: 28560/36938 -- GLOBAL_STEP: 1136700 | > loss_text_ce: 0.03210529312491417 (0.03190965238985981) | > loss_mel_ce: 2.7587227821350098 (nan) | > loss: 0.011074715293943882 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13693842721753394) | > loader_time: 0.0047 (0.004626640702496048)  --> TIME: 2025-02-26 07:05:25 -- STEP: 28610/36938 -- GLOBAL_STEP: 1136750 | > loss_text_ce: 0.04802854731678963 (0.03191031427999929) | > loss_mel_ce: 2.6476173400878906 (nan) | > loss: 0.010697008110582829 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1369324977228746) | > loader_time: 0.0045 (0.004626598278153647)  --> TIME: 2025-02-26 07:05:35 -- STEP: 28660/36938 -- GLOBAL_STEP: 1136800 | > loss_text_ce: 0.02279227413237095 (0.03190836681154421) | > loss_mel_ce: 2.69352650642395 (nan) | > loss: 0.010779043659567833 (nan) | > current_lr: 5e-06 | > step_time: 0.145 (0.13694200676301044) | > loader_time: 0.0046 (0.0046266505789773335)  --> TIME: 2025-02-26 07:05:44 -- STEP: 28710/36938 -- GLOBAL_STEP: 1136850 | > loss_text_ce: 0.029849691316485405 (0.03190787578808044) | > loss_mel_ce: 2.257772445678711 (nan) | > loss: 0.009077866561710835 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13693493243004487) | > loader_time: 0.0048 (0.004626652946126698)  --> TIME: 2025-02-26 07:05:53 -- STEP: 28760/36938 -- GLOBAL_STEP: 1136900 | > loss_text_ce: 0.03827423229813576 (0.031907382386869916) | > loss_mel_ce: 2.8046152591705322 (nan) | > loss: 0.011281308718025684 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13693158953179835) | > loader_time: 0.0045 (0.004626643317796914)  --> TIME: 2025-02-26 07:06:02 -- STEP: 28810/36938 -- GLOBAL_STEP: 1136950 | > loss_text_ce: 0.034139301627874374 (0.03191001007474983) | > loss_mel_ce: 2.7803409099578857 (nan) | > loss: 0.011168573051691055 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.13692613103165407) | > loader_time: 0.0047 (0.004626689806285719)  --> TIME: 2025-02-26 07:06:11 -- STEP: 28860/36938 -- GLOBAL_STEP: 1137000 | > loss_text_ce: 0.03484081104397774 (0.031909935286866226) | > loss_mel_ce: 2.096005439758301 (nan) | > loss: 0.008455739356577396 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.1369219722304995) | > loader_time: 0.0047 (0.004626741486957143)  --> TIME: 2025-02-26 07:06:20 -- STEP: 28910/36938 -- GLOBAL_STEP: 1137050 | > loss_text_ce: 0.0304859708994627 (0.03191065939477686) | > loss_mel_ce: 3.183795213699341 (nan) | > loss: 0.012755084782838821 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.1369105829164992) | > loader_time: 0.0048 (0.004626782531764664)  --> TIME: 2025-02-26 07:06:29 -- STEP: 28960/36938 -- GLOBAL_STEP: 1137100 | > loss_text_ce: 0.029817914590239525 (0.03191068768424098) | > loss_mel_ce: 2.635382652282715 (nan) | > loss: 0.01057619322091341 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13689430058825675) | > loader_time: 0.0045 (0.004626765270918106)  --> TIME: 2025-02-26 07:06:39 -- STEP: 29010/36938 -- GLOBAL_STEP: 1137150 | > loss_text_ce: 0.02983073703944683 (0.031909734406147326) | > loss_mel_ce: 2.5838067531585693 (nan) | > loss: 0.010371577925980091 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.13689938647629457) | > loader_time: 0.0048 (0.004626897284590915)  --> TIME: 2025-02-26 07:06:48 -- STEP: 29060/36938 -- GLOBAL_STEP: 1137200 | > loss_text_ce: 0.029786787927150726 (0.0319114468236058) | > loss_mel_ce: 2.827537775039673 (nan) | > loss: 0.011338590644299984 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13690821948907927) | > loader_time: 0.0045 (0.004626886399958919)  --> TIME: 2025-02-26 07:06:58 -- STEP: 29110/36938 -- GLOBAL_STEP: 1137250 | > loss_text_ce: 0.036479730159044266 (0.03191323236906874) | > loss_mel_ce: 2.6251091957092285 (nan) | > loss: 0.010561861097812653 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.1369107509879705) | > loader_time: 0.0046 (0.0046268755936696255)  --> TIME: 2025-02-26 07:07:07 -- STEP: 29160/36938 -- GLOBAL_STEP: 1137300 | > loss_text_ce: 0.037178099155426025 (0.031913065632731094) | > loss_mel_ce: 2.4971835613250732 (nan) | > loss: 0.010056991130113602 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.1369058337885323) | > loader_time: 0.0047 (0.004626796201423354)  --> TIME: 2025-02-26 07:07:16 -- STEP: 29210/36938 -- GLOBAL_STEP: 1137350 | > loss_text_ce: 0.031162841245532036 (0.03191209975955252) | > loss_mel_ce: 0.196258544921875 (nan) | > loss: 0.0009024658938869834 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13690920322572256) | > loader_time: 0.0046 (0.004626795462817609)  --> TIME: 2025-02-26 07:07:26 -- STEP: 29260/36938 -- GLOBAL_STEP: 1137400 | > loss_text_ce: 0.0317065604031086 (0.03191363863648253) | > loss_mel_ce: 3.005384683609009 (nan) | > loss: 0.012051950208842754 (nan) | > current_lr: 5e-06 | > step_time: 0.1512 (0.13691587412267636) | > loader_time: 0.0046 (0.004626781477638075)  --> TIME: 2025-02-26 07:07:35 -- STEP: 29310/36938 -- GLOBAL_STEP: 1137450 | > loss_text_ce: 0.02964148297905922 (0.03191216624570999) | > loss_mel_ce: 2.2963645458221436 (nan) | > loss: 0.009230182506144047 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.13691631359925427) | > loader_time: 0.0046 (0.004626758193774494)  --> TIME: 2025-02-26 07:07:44 -- STEP: 29360/36938 -- GLOBAL_STEP: 1137500 | > loss_text_ce: 0.026366673409938812 (0.031910665365277265) | > loss_mel_ce: 3.0889670848846436 (nan) | > loss: 0.012362436391413212 (nan) | > current_lr: 5e-06 | > step_time: 0.1606 (0.13691192007032266) | > loader_time: 0.0046 (0.004626724806078765)  --> TIME: 2025-02-26 07:07:54 -- STEP: 29410/36938 -- GLOBAL_STEP: 1137550 | > loss_text_ce: 0.03277682140469551 (0.031912907047259846) | > loss_mel_ce: 2.949694871902466 (nan) | > loss: 0.011835206300020218 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.13691502012228743) | > loader_time: 0.0047 (0.0046267416071056715)  --> TIME: 2025-02-26 07:08:03 -- STEP: 29460/36938 -- GLOBAL_STEP: 1137600 | > loss_text_ce: 0.029541483148932457 (0.03191258524062743) | > loss_mel_ce: 1.7092459201812744 (nan) | > loss: 0.006899950560182333 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13692136329610655) | > loader_time: 0.0046 (0.004626849129826369)  --> TIME: 2025-02-26 07:08:13 -- STEP: 29510/36938 -- GLOBAL_STEP: 1137650 | > loss_text_ce: 0.02703385055065155 (0.03191342685262431) | > loss_mel_ce: 2.162104368209839 (nan) | > loss: 0.008687056601047516 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13691757198755064) | > loader_time: 0.0045 (0.0046269480150378465)  --> TIME: 2025-02-26 07:08:22 -- STEP: 29560/36938 -- GLOBAL_STEP: 1137700 | > loss_text_ce: 0.034036267548799515 (0.03191243939404773) | > loss_mel_ce: 1.5971909761428833 (nan) | > loss: 0.006473124492913485 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13691692096615904) | > loader_time: 0.0045 (0.004627042532935025)  --> TIME: 2025-02-26 07:08:31 -- STEP: 29610/36938 -- GLOBAL_STEP: 1137750 | > loss_text_ce: 0.03368818387389183 (0.03191162696132115) | > loss_mel_ce: 3.311289072036743 (nan) | > loss: 0.013273719698190689 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13691944282870477) | > loader_time: 0.0047 (0.004627034197945476)  --> TIME: 2025-02-26 07:08:41 -- STEP: 29660/36938 -- GLOBAL_STEP: 1137800 | > loss_text_ce: 0.025387194007635117 (0.031913183334761334) | > loss_mel_ce: 2.1865906715393066 (nan) | > loss: 0.008777691051363945 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13692118218840493) | > loader_time: 0.0045 (0.004627046204063423)  --> TIME: 2025-02-26 07:08:50 -- STEP: 29710/36938 -- GLOBAL_STEP: 1137850 | > loss_text_ce: 0.03433405980467796 (0.03191494018065747) | > loss_mel_ce: 3.2036259174346924 (nan) | > loss: 0.012849047780036926 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13691453112772586) | > loader_time: 0.0047 (0.004627080462830342)  --> TIME: 2025-02-26 07:08:59 -- STEP: 29760/36938 -- GLOBAL_STEP: 1137900 | > loss_text_ce: 0.032199252396821976 (0.03191424640297995) | > loss_mel_ce: 0.08066558837890625 (nan) | > loss: 0.00044787637307308614 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13691543024233482) | > loader_time: 0.0047 (0.004627249558125765)  --> TIME: 2025-02-26 07:09:08 -- STEP: 29810/36938 -- GLOBAL_STEP: 1137950 | > loss_text_ce: 0.030227918177843094 (0.03191691540086029) | > loss_mel_ce: 1.5257468223571777 (nan) | > loss: 0.006174503359943628 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13690685756411938) | > loader_time: 0.0046 (0.004627363660198769)  --> TIME: 2025-02-26 07:09:17 -- STEP: 29860/36938 -- GLOBAL_STEP: 1138000 | > loss_text_ce: 0.042950354516506195 (0.03191623239611422) | > loss_mel_ce: 3.32853364944458 (nan) | > loss: 0.013378906063735485 (nan) | > current_lr: 5e-06 | > step_time: 0.1954 (0.13689969720572054) | > loader_time: 0.0049 (0.004627424011434866)  --> TIME: 2025-02-26 07:09:26 -- STEP: 29910/36938 -- GLOBAL_STEP: 1138050 | > loss_text_ce: 0.038696739822626114 (0.03191545475454915) | > loss_mel_ce: 1.4885550737380981 (nan) | > loss: 0.006060523446649313 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1368883257046136) | > loader_time: 0.0046 (0.004627513749987646)  --> TIME: 2025-02-26 07:09:35 -- STEP: 29960/36938 -- GLOBAL_STEP: 1138100 | > loss_text_ce: 0.03227994590997696 (0.03191477206942232) | > loss_mel_ce: 2.9586644172668457 (nan) | > loss: 0.011868827976286411 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13688311036501968) | > loader_time: 0.0047 (0.0046276501963071655)  --> TIME: 2025-02-26 07:09:45 -- STEP: 30010/36938 -- GLOBAL_STEP: 1138150 | > loss_text_ce: 0.034771911799907684 (0.031914450556737424) | > loss_mel_ce: 3.097754716873169 (nan) | > loss: 0.012430662289261818 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.13688304510087856) | > loader_time: 0.0047 (0.00462772768364473)  --> TIME: 2025-02-26 07:09:54 -- STEP: 30060/36938 -- GLOBAL_STEP: 1138200 | > loss_text_ce: 0.034729965031147 (0.03191174094135179) | > loss_mel_ce: 0.7547607421875 (nan) | > loss: 0.003132899757474661 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.1368786170017842) | > loader_time: 0.0048 (0.0046277551831837346)  --> TIME: 2025-02-26 07:10:03 -- STEP: 30110/36938 -- GLOBAL_STEP: 1138250 | > loss_text_ce: 0.025294721126556396 (0.031913327758068426) | > loss_mel_ce: 2.3743326663970947 (nan) | > loss: 0.0095223318785429 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.13687526381558057) | > loader_time: 0.0047 (0.004627773437892738)  --> TIME: 2025-02-26 07:10:13 -- STEP: 30160/36938 -- GLOBAL_STEP: 1138300 | > loss_text_ce: 0.034230154007673264 (0.031911328578255474) | > loss_mel_ce: 2.689638137817383 (nan) | > loss: 0.010809002444148064 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.13688323555479512) | > loader_time: 0.0046 (0.0046279068018460386)  --> TIME: 2025-02-26 07:10:22 -- STEP: 30210/36938 -- GLOBAL_STEP: 1138350 | > loss_text_ce: 0.025317486375570297 (0.031911820755640787) | > loss_mel_ce: 2.8448400497436523 (nan) | > loss: 0.011389514431357384 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.1368816782330397) | > loader_time: 0.0047 (0.0046280054413139725)  --> TIME: 2025-02-26 07:10:32 -- STEP: 30260/36938 -- GLOBAL_STEP: 1138400 | > loss_text_ce: 0.027502451092004776 (0.03191341058232921) | > loss_mel_ce: 1.9253774881362915 (nan) | > loss: 0.007749523967504501 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.13688953904544485) | > loader_time: 0.0048 (0.004628068425366986)  --> TIME: 2025-02-26 07:10:41 -- STEP: 30310/36938 -- GLOBAL_STEP: 1138450 | > loss_text_ce: 0.045473769307136536 (0.03191367820251267) | > loss_mel_ce: 2.1755855083465576 (nan) | > loss: 0.008813727647066116 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.1368870248341462) | > loader_time: 0.0045 (0.004628065717137706)  --> TIME: 2025-02-26 07:10:50 -- STEP: 30360/36938 -- GLOBAL_STEP: 1138500 | > loss_text_ce: 0.02957824245095253 (0.031914676091331945) | > loss_mel_ce: 2.6994073390960693 (nan) | > loss: 0.010829308070242405 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.13688401644251758) | > loader_time: 0.0047 (0.0046281854233094595)  --> TIME: 2025-02-26 07:10:59 -- STEP: 30410/36938 -- GLOBAL_STEP: 1138550 | > loss_text_ce: 0.027654310688376427 (0.03191486871132154) | > loss_mel_ce: 2.972479820251465 (nan) | > loss: 0.011905294843018055 (nan) | > current_lr: 5e-06 | > step_time: 0.1692 (0.13687892528553885) | > loader_time: 0.0049 (0.004628256981563341)  --> TIME: 2025-02-26 07:11:08 -- STEP: 30460/36938 -- GLOBAL_STEP: 1138600 | > loss_text_ce: 0.030337531119585037 (0.03191369783872585) | > loss_mel_ce: 2.3237133026123047 (nan) | > loss: 0.009341471828520298 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1368745497630277) | > loader_time: 0.0045 (0.004628179031066487)  --> TIME: 2025-02-26 07:11:18 -- STEP: 30510/36938 -- GLOBAL_STEP: 1138650 | > loss_text_ce: 0.02753620781004429 (0.03191127137863122) | > loss_mel_ce: 1.6336512565612793 (nan) | > loss: 0.006592013873159885 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13687758686424395) | > loader_time: 0.0046 (0.004628160991499902)  --> TIME: 2025-02-26 07:11:27 -- STEP: 30560/36938 -- GLOBAL_STEP: 1138700 | > loss_text_ce: 0.03455619886517525 (0.03191276636216628) | > loss_mel_ce: 2.554481267929077 (nan) | > loss: 0.010273958556354046 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13687625002330392) | > loader_time: 0.0046 (0.004628243012578068)  --> TIME: 2025-02-26 07:11:37 -- STEP: 30610/36938 -- GLOBAL_STEP: 1138750 | > loss_text_ce: 0.02697100304067135 (0.031911876298668616) | > loss_mel_ce: 1.8956137895584106 (nan) | > loss: 0.007629305124282837 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13687957601195064) | > loader_time: 0.0045 (0.004628349269616444)  --> TIME: 2025-02-26 07:11:46 -- STEP: 30660/36938 -- GLOBAL_STEP: 1138800 | > loss_text_ce: 0.024454154074192047 (0.03191199147489372) | > loss_mel_ce: 1.9728630781173706 (nan) | > loss: 0.007925862446427345 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13687788041992466) | > loader_time: 0.0048 (0.004628287649434599)  --> TIME: 2025-02-26 07:11:55 -- STEP: 30710/36938 -- GLOBAL_STEP: 1138850 | > loss_text_ce: 0.033220916986465454 (0.031914314935096146) | > loss_mel_ce: 2.5075912475585938 (nan) | > loss: 0.010082588531076908 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13687605976631298) | > loader_time: 0.0045 (0.004628235367601363)  --> TIME: 2025-02-26 07:12:05 -- STEP: 30760/36938 -- GLOBAL_STEP: 1138900 | > loss_text_ce: 0.035084206610918045 (0.03191416634365401) | > loss_mel_ce: 1.4437332153320312 (nan) | > loss: 0.005868323612958193 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13688122004619227) | > loader_time: 0.0046 (0.004628290792125402)  --> TIME: 2025-02-26 07:12:14 -- STEP: 30810/36938 -- GLOBAL_STEP: 1138950 | > loss_text_ce: 0.03020157478749752 (0.031913524221644674) | > loss_mel_ce: 3.498387098312378 (nan) | > loss: 0.014002337120473385 (nan) | > current_lr: 5e-06 | > step_time: 0.1579 (0.13689060961027685) | > loader_time: 0.0046 (0.004628332703580171)  --> TIME: 2025-02-26 07:12:24 -- STEP: 30860/36938 -- GLOBAL_STEP: 1139000 | > loss_text_ce: 0.03099174238741398 (0.03191283028530553) | > loss_mel_ce: 2.8200693130493164 (nan) | > loss: 0.011313735507428646 (nan) | > current_lr: 5e-06 | > step_time: 0.1664 (0.13688690485462804) | > loader_time: 0.0046 (0.00462830131577734)  --> TIME: 2025-02-26 07:12:33 -- STEP: 30910/36938 -- GLOBAL_STEP: 1139050 | > loss_text_ce: 0.024215949699282646 (0.03191197672072443) | > loss_mel_ce: 2.1780872344970703 (nan) | > loss: 0.008739299140870571 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13688970304005857) | > loader_time: 0.0046 (0.00462835438233754)  --> TIME: 2025-02-26 07:12:42 -- STEP: 30960/36938 -- GLOBAL_STEP: 1139100 | > loss_text_ce: 0.04531518742442131 (0.03191427050251039) | > loss_mel_ce: 2.651520013809204 (nan) | > loss: 0.01070172805339098 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1368879890488088) | > loader_time: 0.0048 (0.004628389480810133)  --> TIME: 2025-02-26 07:12:51 -- STEP: 31010/36938 -- GLOBAL_STEP: 1139150 | > loss_text_ce: 0.03625078499317169 (0.03191372240637972) | > loss_mel_ce: 3.5162041187286377 (nan) | > loss: 0.014097044244408607 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.1368765516305719) | > loader_time: 0.0047 (0.0046283284605260515)  --> TIME: 2025-02-26 07:13:01 -- STEP: 31060/36938 -- GLOBAL_STEP: 1139200 | > loss_text_ce: 0.0395689494907856 (0.0319146202044192) | > loss_mel_ce: 0.9236408472061157 (nan) | > loss: 0.003822261467576027 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13688769090275302) | > loader_time: 0.0045 (0.00462841966884487)  --> TIME: 2025-02-26 07:13:10 -- STEP: 31110/36938 -- GLOBAL_STEP: 1139250 | > loss_text_ce: 0.03761538490653038 (0.031914144298777355) | > loss_mel_ce: 3.283479690551758 (nan) | > loss: 0.013178949244320393 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13688706489515048) | > loader_time: 0.0046 (0.0046284624940902165)  --> TIME: 2025-02-26 07:13:20 -- STEP: 31160/36938 -- GLOBAL_STEP: 1139300 | > loss_text_ce: 0.035171013325452805 (0.0319147607510067) | > loss_mel_ce: 2.911811351776123 (nan) | > loss: 0.011694375425577164 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13688899224315473) | > loader_time: 0.0047 (0.004628406593215014)  --> TIME: 2025-02-26 07:13:29 -- STEP: 31210/36938 -- GLOBAL_STEP: 1139350 | > loss_text_ce: 0.037244949489831924 (0.031915892955621354) | > loss_mel_ce: 3.089799404144287 (nan) | > loss: 0.012408907525241375 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13688639303462125) | > loader_time: 0.0045 (0.004628335883395094)  --> TIME: 2025-02-26 07:13:38 -- STEP: 31260/36938 -- GLOBAL_STEP: 1139400 | > loss_text_ce: 0.026651982218027115 (0.031917568627251695) | > loss_mel_ce: 3.973544120788574 (nan) | > loss: 0.015873795375227928 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13688931311847036) | > loader_time: 0.0045 (0.00462830467858683)  --> TIME: 2025-02-26 07:13:48 -- STEP: 31310/36938 -- GLOBAL_STEP: 1139450 | > loss_text_ce: 0.03396190330386162 (0.03191806455926596) | > loss_mel_ce: 2.1616907119750977 (nan) | > loss: 0.008712908253073692 (nan) | > current_lr: 5e-06 | > step_time: 0.1799 (0.1368933768822175) | > loader_time: 0.0048 (0.004628249784888105)  --> TIME: 2025-02-26 07:13:58 -- STEP: 31360/36938 -- GLOBAL_STEP: 1139500 | > loss_text_ce: 0.030715571716427803 (0.031919342323535535) | > loss_mel_ce: 2.1062235832214355 (nan) | > loss: 0.008479917421936989 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.13690594072092438) | > loader_time: 0.0048 (0.0046282025624294594)  --> TIME: 2025-02-26 07:14:07 -- STEP: 31410/36938 -- GLOBAL_STEP: 1139550 | > loss_text_ce: 0.03296404704451561 (0.03192091130211826) | > loss_mel_ce: 2.7279212474823 (nan) | > loss: 0.010955894365906715 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.1369055904806199) | > loader_time: 0.0048 (0.0046281089299957)  --> TIME: 2025-02-26 07:14:16 -- STEP: 31460/36938 -- GLOBAL_STEP: 1139600 | > loss_text_ce: 0.01923295296728611 (0.031919010037565865) | > loss_mel_ce: 3.056979179382324 (nan) | > loss: 0.012207191437482834 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13690488107327092) | > loader_time: 0.0044 (0.00462799981681692)  --> TIME: 2025-02-26 07:14:26 -- STEP: 31510/36938 -- GLOBAL_STEP: 1139650 | > loss_text_ce: 0.02858276292681694 (0.031919438354062246) | > loss_mel_ce: 2.959243059158325 (nan) | > loss: 0.01185645256191492 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13690629870276305) | > loader_time: 0.0045 (0.004627918765795333)  --> TIME: 2025-02-26 07:14:35 -- STEP: 31560/36938 -- GLOBAL_STEP: 1139700 | > loss_text_ce: 0.03083270974457264 (0.0319189802708969) | > loss_mel_ce: 2.4015703201293945 (nan) | > loss: 0.009652393870055676 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13691755855038024) | > loader_time: 0.0046 (0.004627880994206894)  --> TIME: 2025-02-26 07:14:45 -- STEP: 31610/36938 -- GLOBAL_STEP: 1139750 | > loss_text_ce: 0.026450105011463165 (0.03191929312472376) | > loss_mel_ce: 0.2328948974609375 (nan) | > loss: 0.0010291468352079391 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13691218034333233) | > loader_time: 0.0045 (0.004627846178092816)  --> TIME: 2025-02-26 07:14:53 -- STEP: 31660/36938 -- GLOBAL_STEP: 1139800 | > loss_text_ce: 0.03297744691371918 (0.031920641091689364) | > loss_mel_ce: 2.578094482421875 (nan) | > loss: 0.010361396707594395 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13689734788913815) | > loader_time: 0.0045 (0.004627754759050508)  --> TIME: 2025-02-26 07:15:03 -- STEP: 31710/36938 -- GLOBAL_STEP: 1139850 | > loss_text_ce: 0.039281513541936874 (0.031922041573886395) | > loss_mel_ce: 2.8840153217315674 (nan) | > loss: 0.011600385420024395 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13689502174265447) | > loader_time: 0.0047 (0.004627722146495281)  --> TIME: 2025-02-26 07:15:12 -- STEP: 31760/36938 -- GLOBAL_STEP: 1139900 | > loss_text_ce: 0.03116958774626255 (0.03192019512837591) | > loss_mel_ce: 2.216796875 (nan) | > loss: 0.008920502848923206 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13688802525438534) | > loader_time: 0.0045 (0.004627768436367018)  --> TIME: 2025-02-26 07:15:21 -- STEP: 31810/36938 -- GLOBAL_STEP: 1139950 | > loss_text_ce: 0.04496663808822632 (0.031919682441681745) | > loss_mel_ce: 2.737413167953491 (nan) | > loss: 0.011041190475225449 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13688644587862345) | > loader_time: 0.0044 (0.004627713824322815)  --> TIME: 2025-02-26 07:15:30 -- STEP: 31860/36938 -- GLOBAL_STEP: 1140000 | > loss_text_ce: 0.03426661342382431 (0.03191831640777012) | > loss_mel_ce: 2.7127909660339355 (nan) | > loss: 0.01090102270245552 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13688703989428724) | > loader_time: 0.0047 (0.00462763552686663) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1140000.pth  --> TIME: 2025-02-26 07:15:45 -- STEP: 31910/36938 -- GLOBAL_STEP: 1140050 | > loss_text_ce: 0.025109080597758293 (0.03191772938671852) | > loss_mel_ce: 3.3328866958618164 (nan) | > loss: 0.013325381092727184 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.1368778405687111) | > loader_time: 0.0048 (0.004627625974358074)  --> TIME: 2025-02-26 07:15:55 -- STEP: 31960/36938 -- GLOBAL_STEP: 1140100 | > loss_text_ce: 0.023425979539752007 (0.03191718400420478) | > loss_mel_ce: 2.1518146991729736 (nan) | > loss: 0.00863190833479166 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.13687231254219448) | > loader_time: 0.0045 (0.004627590304769764)  --> TIME: 2025-02-26 07:16:04 -- STEP: 32010/36938 -- GLOBAL_STEP: 1140150 | > loss_text_ce: 0.02770164981484413 (0.03191616855889361) | > loss_mel_ce: 3.979567289352417 (nan) | > loss: 0.01590186171233654 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.13687236048362544) | > loader_time: 0.0046 (0.004627578536334235)  --> TIME: 2025-02-26 07:16:13 -- STEP: 32060/36938 -- GLOBAL_STEP: 1140200 | > loss_text_ce: 0.03640175610780716 (0.03191643068357783) | > loss_mel_ce: 3.3245327472686768 (nan) | > loss: 0.013337042182683945 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13686818863106492) | > loader_time: 0.0045 (0.00462753739270729)  --> TIME: 2025-02-26 07:16:22 -- STEP: 32110/36938 -- GLOBAL_STEP: 1140250 | > loss_text_ce: 0.03349757194519043 (0.0319153194023082) | > loss_mel_ce: 2.770430088043213 (nan) | > loss: 0.01112669799476862 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13685859226059183) | > loader_time: 0.0044 (0.004627439649778518)  --> TIME: 2025-02-26 07:16:31 -- STEP: 32160/36938 -- GLOBAL_STEP: 1140300 | > loss_text_ce: 0.040526047348976135 (0.03191607455255031) | > loss_mel_ce: 2.3734519481658936 (nan) | > loss: 0.009579278528690338 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.1368522832049648) | > loader_time: 0.0046 (0.0046273905320547044)  --> TIME: 2025-02-26 07:16:41 -- STEP: 32210/36938 -- GLOBAL_STEP: 1140350 | > loss_text_ce: 0.04302641376852989 (0.031916707405923055) | > loss_mel_ce: 2.921170711517334 (nan) | > loss: 0.011762687936425209 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13685203412052008) | > loader_time: 0.0045 (0.004627375971342007)  --> TIME: 2025-02-26 07:16:50 -- STEP: 32260/36938 -- GLOBAL_STEP: 1140400 | > loss_text_ce: 0.04120668023824692 (0.03191766881381667) | > loss_mel_ce: 3.238778829574585 (nan) | > loss: 0.013015815988183022 (nan) | > current_lr: 5e-06 | > step_time: 0.1889 (0.13684803993895453) | > loader_time: 0.0048 (0.004627435841474626)  --> TIME: 2025-02-26 07:16:59 -- STEP: 32310/36938 -- GLOBAL_STEP: 1140450 | > loss_text_ce: 0.0390547439455986 (0.03191695233330807) | > loss_mel_ce: 2.5544652938842773 (nan) | > loss: 0.010291745886206627 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13684625150396482) | > loader_time: 0.0048 (0.0046274787831402315)  --> TIME: 2025-02-26 07:17:08 -- STEP: 32360/36938 -- GLOBAL_STEP: 1140500 | > loss_text_ce: 0.03867431730031967 (0.03191761830864825) | > loss_mel_ce: 1.6026020050048828 (nan) | > loss: 0.00651300186291337 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13684586536309515) | > loader_time: 0.0045 (0.004627443428239945)  --> TIME: 2025-02-26 07:17:17 -- STEP: 32410/36938 -- GLOBAL_STEP: 1140550 | > loss_text_ce: 0.03295925632119179 (0.03191493012112222) | > loss_mel_ce: 3.4367988109588623 (nan) | > loss: 0.013768881559371948 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13684137158098023) | > loader_time: 0.0045 (0.0046273973097737875)  --> TIME: 2025-02-26 07:17:27 -- STEP: 32460/36938 -- GLOBAL_STEP: 1140600 | > loss_text_ce: 0.03502322733402252 (0.03191548613234641) | > loss_mel_ce: 3.054335117340088 (nan) | > loss: 0.01225935947149992 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.1368377332032193) | > loader_time: 0.0045 (0.004627353037424603)  --> TIME: 2025-02-26 07:17:36 -- STEP: 32510/36938 -- GLOBAL_STEP: 1140650 | > loss_text_ce: 0.030098404735326767 (0.031916039745701424) | > loss_mel_ce: 2.9905173778533936 (nan) | > loss: 0.011986571364104748 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13683239339278644) | > loader_time: 0.0046 (0.0046272913370452124)  --> TIME: 2025-02-26 07:17:45 -- STEP: 32560/36938 -- GLOBAL_STEP: 1140700 | > loss_text_ce: 0.029039708897471428 (0.03191620680819348) | > loss_mel_ce: 2.526257038116455 (nan) | > loss: 0.010140066966414452 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13683451802197374) | > loader_time: 0.0045 (0.004627259460069617)  --> TIME: 2025-02-26 07:17:54 -- STEP: 32610/36938 -- GLOBAL_STEP: 1140750 | > loss_text_ce: 0.013867856934666634 (0.03191472769819726) | > loss_mel_ce: 1.0163726806640625 (nan) | > loss: 0.004088256508111954 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13682638211909665) | > loader_time: 0.0044 (0.004627180201844247)  --> TIME: 2025-02-26 07:18:03 -- STEP: 32660/36938 -- GLOBAL_STEP: 1140800 | > loss_text_ce: 0.03729122877120972 (0.03191361903399478) | > loss_mel_ce: 2.592580795288086 (nan) | > loss: 0.010436001233756542 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13681914553785243) | > loader_time: 0.0046 (0.004627235784022551)  --> TIME: 2025-02-26 07:18:12 -- STEP: 32710/36938 -- GLOBAL_STEP: 1140850 | > loss_text_ce: 0.03811540827155113 (0.031913756698914574) | > loss_mel_ce: 3.025681734085083 (nan) | > loss: 0.012157926335930824 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13681632605797175) | > loader_time: 0.0046 (0.004627249751824256)  --> TIME: 2025-02-26 07:18:22 -- STEP: 32760/36938 -- GLOBAL_STEP: 1140900 | > loss_text_ce: 0.03590840846300125 (0.03191467393496719) | > loss_mel_ce: 3.0926718711853027 (nan) | > loss: 0.012415002100169659 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.1368174193033503) | > loader_time: 0.0047 (0.00462718747183225)  --> TIME: 2025-02-26 07:18:32 -- STEP: 32810/36938 -- GLOBAL_STEP: 1140950 | > loss_text_ce: 0.033799368888139725 (0.0319165336966167) | > loss_mel_ce: 3.2834396362304688 (nan) | > loss: 0.01316364761441946 (nan) | > current_lr: 5e-06 | > step_time: 0.1859 (0.13682620353867206) | > loader_time: 0.0049 (0.004627271935330111)  --> TIME: 2025-02-26 07:18:41 -- STEP: 32860/36938 -- GLOBAL_STEP: 1141000 | > loss_text_ce: 0.035692647099494934 (0.031916530168549166) | > loss_mel_ce: 1.3870110511779785 (nan) | > loss: 0.00564564997330308 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13681895058109964) | > loader_time: 0.0045 (0.004627263604034677)  --> TIME: 2025-02-26 07:18:50 -- STEP: 32910/36938 -- GLOBAL_STEP: 1141050 | > loss_text_ce: 0.03257916495203972 (0.03191630805012587) | > loss_mel_ce: 1.9853531122207642 (nan) | > loss: 0.00800766795873642 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13683127381090804) | > loader_time: 0.0045 (0.00462736188731487)  --> TIME: 2025-02-26 07:19:00 -- STEP: 32960/36938 -- GLOBAL_STEP: 1141100 | > loss_text_ce: 0.03193454444408417 (0.03191680723718768) | > loss_mel_ce: 2.8242146968841553 (nan) | > loss: 0.011333925649523735 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13683391865885264) | > loader_time: 0.0049 (0.004627706016440986)  --> TIME: 2025-02-26 07:19:09 -- STEP: 33010/36938 -- GLOBAL_STEP: 1141150 | > loss_text_ce: 0.04048937186598778 (0.031915197157467955) | > loss_mel_ce: 2.3867595195770264 (nan) | > loss: 0.009631941094994545 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13683454802598816) | > loader_time: 0.0047 (0.004627931287742962)  --> TIME: 2025-02-26 07:19:18 -- STEP: 33060/36938 -- GLOBAL_STEP: 1141200 | > loss_text_ce: 0.028836088255047798 (0.03191617616391449) | > loss_mel_ce: 2.7260072231292725 (nan) | > loss: 0.010931918397545815 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.13682570108412428) | > loader_time: 0.0047 (0.004627969015459538)  --> TIME: 2025-02-26 07:19:28 -- STEP: 33110/36938 -- GLOBAL_STEP: 1141250 | > loss_text_ce: 0.0284026600420475 (0.031915191066013066) | > loss_mel_ce: 2.1671791076660156 (nan) | > loss: 0.008712626062333584 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1368300717102357) | > loader_time: 0.0045 (0.004628014326887663)  --> TIME: 2025-02-26 07:19:37 -- STEP: 33160/36938 -- GLOBAL_STEP: 1141300 | > loss_text_ce: 0.034940317273139954 (0.031915888122386486) | > loss_mel_ce: 3.2111265659332275 (nan) | > loss: 0.012881218455731869 (nan) | > current_lr: 5e-06 | > step_time: 0.1756 (0.13682758282550883) | > loader_time: 0.0048 (0.00462809562395507)  --> TIME: 2025-02-26 07:19:46 -- STEP: 33210/36938 -- GLOBAL_STEP: 1141350 | > loss_text_ce: 0.023861702531576157 (0.031917498736914254) | > loss_mel_ce: 3.1804604530334473 (nan) | > loss: 0.012715565040707588 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13682630754888242) | > loader_time: 0.0048 (0.004628229837609997)  --> TIME: 2025-02-26 07:19:55 -- STEP: 33260/36938 -- GLOBAL_STEP: 1141400 | > loss_text_ce: 0.03509039804339409 (0.0319152125793607) | > loss_mel_ce: 2.502061605453491 (nan) | > loss: 0.01006806455552578 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.1368242240210135) | > loader_time: 0.0046 (0.004628278065071857)  --> TIME: 2025-02-26 07:20:05 -- STEP: 33310/36938 -- GLOBAL_STEP: 1141450 | > loss_text_ce: 0.0358322411775589 (0.03191389479425302) | > loss_mel_ce: 2.6584177017211914 (nan) | > loss: 0.010691468603909016 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13682386301831298) | > loader_time: 0.0045 (0.004628293573659514)  --> TIME: 2025-02-26 07:20:14 -- STEP: 33360/36938 -- GLOBAL_STEP: 1141500 | > loss_text_ce: 0.029449164867401123 (0.03191209175460378) | > loss_mel_ce: 2.0023748874664307 (nan) | > loss: 0.008062794804573059 (nan) | > current_lr: 5e-06 | > step_time: 0.1081 (0.1368224500609233) | > loader_time: 0.0045 (0.004628486527527545)  --> TIME: 2025-02-26 07:20:24 -- STEP: 33410/36938 -- GLOBAL_STEP: 1141550 | > loss_text_ce: 0.03414950892329216 (0.031911117900709496) | > loss_mel_ce: 3.1648213863372803 (nan) | > loss: 0.012694329023361206 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.13682985445654444) | > loader_time: 0.0048 (0.004628629500335717)  --> TIME: 2025-02-26 07:20:33 -- STEP: 33460/36938 -- GLOBAL_STEP: 1141600 | > loss_text_ce: 0.03603034093976021 (0.03191074873253976) | > loss_mel_ce: 2.594998598098755 (nan) | > loss: 0.010440591722726822 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13682976490141482) | > loader_time: 0.0046 (0.004628779905254227)  --> TIME: 2025-02-26 07:20:42 -- STEP: 33510/36938 -- GLOBAL_STEP: 1141650 | > loss_text_ce: 0.03462127968668938 (0.03190857940702266) | > loss_mel_ce: 2.9090254306793213 (nan) | > loss: 0.011681138537824154 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13683268791239594) | > loader_time: 0.0045 (0.004628727330054426)  --> TIME: 2025-02-26 07:20:52 -- STEP: 33560/36938 -- GLOBAL_STEP: 1141700 | > loss_text_ce: 0.03510839119553566 (0.031907836058736705) | > loss_mel_ce: 3.097001552581787 (nan) | > loss: 0.0124290082603693 (nan) | > current_lr: 5e-06 | > step_time: 0.1811 (0.13682866931245663) | > loader_time: 0.0047 (0.004628661015604901)  --> TIME: 2025-02-26 07:21:01 -- STEP: 33610/36938 -- GLOBAL_STEP: 1141750 | > loss_text_ce: 0.03781226649880409 (0.031907616874795715) | > loss_mel_ce: 2.220140218734741 (nan) | > loss: 0.00896012969315052 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13682261003478824) | > loader_time: 0.0046 (0.004628900451626105)  --> TIME: 2025-02-26 07:21:10 -- STEP: 33660/36938 -- GLOBAL_STEP: 1141800 | > loss_text_ce: 0.04237276315689087 (0.03190770927806687) | > loss_mel_ce: 3.0727291107177734 (nan) | > loss: 0.012361516244709492 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.1368236037994935) | > loader_time: 0.0049 (0.004628999751731885)  --> TIME: 2025-02-26 07:21:20 -- STEP: 33710/36938 -- GLOBAL_STEP: 1141850 | > loss_text_ce: 0.028871051967144012 (0.03190954691437332) | > loss_mel_ce: 2.895570755004883 (nan) | > loss: 0.01160492841154337 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.13682615679896667) | > loader_time: 0.0046 (0.004629030499250377)  --> TIME: 2025-02-26 07:21:29 -- STEP: 33760/36938 -- GLOBAL_STEP: 1141900 | > loss_text_ce: 0.04637143015861511 (0.03190988806430579) | > loss_mel_ce: 2.659402370452881 (nan) | > loss: 0.010737198404967785 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13682024350380886) | > loader_time: 0.0046 (0.004629080463642194)  --> TIME: 2025-02-26 07:21:38 -- STEP: 33810/36938 -- GLOBAL_STEP: 1141950 | > loss_text_ce: 0.028439482674002647 (0.0319089160071888) | > loss_mel_ce: 2.989229917526245 (nan) | > loss: 0.011974879540503025 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13682354069856045) | > loader_time: 0.0045 (0.004629112911873125)  --> TIME: 2025-02-26 07:21:48 -- STEP: 33860/36938 -- GLOBAL_STEP: 1142000 | > loss_text_ce: 0.032673392444849014 (0.03190870155523702) | > loss_mel_ce: 2.866943597793579 (nan) | > loss: 0.011506416834890842 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.1368250410231473) | > loader_time: 0.0045 (0.0046291299283117236)  --> TIME: 2025-02-26 07:21:57 -- STEP: 33910/36938 -- GLOBAL_STEP: 1142050 | > loss_text_ce: 0.03005927987396717 (0.031907471804103936) | > loss_mel_ce: 2.8070831298828125 (nan) | > loss: 0.0112585024908185 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13682858294984945) | > loader_time: 0.0047 (0.0046291016294906315)  --> TIME: 2025-02-26 07:22:06 -- STEP: 33960/36938 -- GLOBAL_STEP: 1142100 | > loss_text_ce: 0.03345555439591408 (0.03190725078193098) | > loss_mel_ce: 2.769757032394409 (nan) | > loss: 0.011123860254883766 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13682887592781717) | > loader_time: 0.0047 (0.00462912175062829)  --> TIME: 2025-02-26 07:22:16 -- STEP: 34010/36938 -- GLOBAL_STEP: 1142150 | > loss_text_ce: 0.04880670830607414 (0.03190942421709984) | > loss_mel_ce: 3.518049955368042 (nan) | > loss: 0.014154193922877312 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.1368316327708837) | > loader_time: 0.0046 (0.004629113196765842)  --> TIME: 2025-02-26 07:22:25 -- STEP: 34060/36938 -- GLOBAL_STEP: 1142200 | > loss_text_ce: 0.027872078120708466 (0.03191074453867419) | > loss_mel_ce: 0.03912353515625 (nan) | > loss: 0.00026585563318803906 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.13683277997141666) | > loader_time: 0.0046 (0.00462921321637058)  --> TIME: 2025-02-26 07:22:35 -- STEP: 34110/36938 -- GLOBAL_STEP: 1142250 | > loss_text_ce: 0.02629178762435913 (0.031908991560067095) | > loss_mel_ce: 1.2153778076171875 (nan) | > loss: 0.004927260801196098 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13683320889546935) | > loader_time: 0.0047 (0.004629245058807314)  --> TIME: 2025-02-26 07:22:44 -- STEP: 34160/36938 -- GLOBAL_STEP: 1142300 | > loss_text_ce: 0.02807159349322319 (0.0319099844769256) | > loss_mel_ce: 2.742112398147583 (nan) | > loss: 0.010992794297635555 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13683380964070235) | > loader_time: 0.0046 (0.004629227546953209)  --> TIME: 2025-02-26 07:22:53 -- STEP: 34210/36938 -- GLOBAL_STEP: 1142350 | > loss_text_ce: 0.02465624362230301 (0.031909777701077634) | > loss_mel_ce: 1.8260091543197632 (nan) | > loss: 0.007343911100178957 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13682834535479887) | > loader_time: 0.0045 (0.004629188823058781)  --> TIME: 2025-02-26 07:23:02 -- STEP: 34260/36938 -- GLOBAL_STEP: 1142400 | > loss_text_ce: 0.03102346509695053 (0.031909633141638596) | > loss_mel_ce: 2.5267934799194336 (nan) | > loss: 0.010150068439543247 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.13682159740049274) | > loader_time: 0.0046 (0.004629160177617208)  --> TIME: 2025-02-26 07:23:11 -- STEP: 34310/36938 -- GLOBAL_STEP: 1142450 | > loss_text_ce: 0.030638782307505608 (0.03191017946999304) | > loss_mel_ce: 3.0907585620880127 (nan) | > loss: 0.012386498041450977 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13682069189865953) | > loader_time: 0.0046 (0.004629149481423988)  --> TIME: 2025-02-26 07:23:21 -- STEP: 34360/36938 -- GLOBAL_STEP: 1142500 | > loss_text_ce: 0.029701227322220802 (0.03191131727568874) | > loss_mel_ce: 2.929011344909668 (nan) | > loss: 0.01174092385917902 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.13682290496731705) | > loader_time: 0.0046 (0.004629139149424921)  --> TIME: 2025-02-26 07:23:30 -- STEP: 34410/36938 -- GLOBAL_STEP: 1142550 | > loss_text_ce: 0.038429129868745804 (0.031911571024608995) | > loss_mel_ce: 2.3895223140716553 (nan) | > loss: 0.009634728543460369 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1368333611367772) | > loader_time: 0.0045 (0.0046291235053792235)  --> TIME: 2025-02-26 07:23:40 -- STEP: 34460/36938 -- GLOBAL_STEP: 1142600 | > loss_text_ce: 0.029220597818493843 (0.031910710255585895) | > loss_mel_ce: 2.123809814453125 (nan) | > loss: 0.00854377169162035 (nan) | > current_lr: 5e-06 | > step_time: 0.1588 (0.13683427251222346) | > loader_time: 0.0046 (0.004629104910931894)  --> TIME: 2025-02-26 07:23:49 -- STEP: 34510/36938 -- GLOBAL_STEP: 1142650 | > loss_text_ce: 0.045585792511701584 (0.031910959450580596) | > loss_mel_ce: 3.0379812717437744 (nan) | > loss: 0.012236378155648708 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13683976541218956) | > loader_time: 0.0045 (0.004629120982857387)  --> TIME: 2025-02-26 07:23:59 -- STEP: 34560/36938 -- GLOBAL_STEP: 1142700 | > loss_text_ce: 0.03032005950808525 (0.03191032930295095) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13684037202210245) | > loader_time: 0.0047 (0.004629110600109458)  --> TIME: 2025-02-26 07:24:08 -- STEP: 34610/36938 -- GLOBAL_STEP: 1142750 | > loss_text_ce: 0.032645564526319504 (0.031910831540606824) | > loss_mel_ce: 2.9975576400756836 (nan) | > loss: 0.01202461589127779 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13684311588869222) | > loader_time: 0.0045 (0.004629123345237017)  --> TIME: 2025-02-26 07:24:18 -- STEP: 34660/36938 -- GLOBAL_STEP: 1142800 | > loss_text_ce: 0.0384395457804203 (0.031911145235276044) | > loss_mel_ce: 2.7544734477996826 (nan) | > loss: 0.011082988232374191 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13685043562510948) | > loader_time: 0.0045 (0.004629179066615653)  --> TIME: 2025-02-26 07:24:27 -- STEP: 34710/36938 -- GLOBAL_STEP: 1142850 | > loss_text_ce: 0.034667883068323135 (0.0319102820810721) | > loss_mel_ce: 2.714083194732666 (nan) | > loss: 0.010907743126153946 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13685188307194684) | > loader_time: 0.0045 (0.0046291910375926525)  --> TIME: 2025-02-26 07:24:36 -- STEP: 34760/36938 -- GLOBAL_STEP: 1142900 | > loss_text_ce: 0.03083774819970131 (0.031907922780019214) | > loss_mel_ce: 0.01844024658203125 (nan) | > loss: 0.00019554760365281254 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13684399374197942) | > loader_time: 0.0045 (0.0046291222712250175)  --> TIME: 2025-02-26 07:24:46 -- STEP: 34810/36938 -- GLOBAL_STEP: 1142950 | > loss_text_ce: 0.041758280247449875 (0.03190827734976557) | > loss_mel_ce: 2.8944461345672607 (nan) | > loss: 0.011651605367660522 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13684297627808112) | > loader_time: 0.0046 (0.0046290674623282865)  --> TIME: 2025-02-26 07:24:55 -- STEP: 34860/36938 -- GLOBAL_STEP: 1143000 | > loss_text_ce: 0.03298839554190636 (0.03190745386803712) | > loss_mel_ce: 2.2069544792175293 (nan) | > loss: 0.008888662792742252 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13684465889949785) | > loader_time: 0.0045 (0.004629019602098761)  --> TIME: 2025-02-26 07:25:04 -- STEP: 34910/36938 -- GLOBAL_STEP: 1143050 | > loss_text_ce: 0.02940165065228939 (0.03190544098532963) | > loss_mel_ce: 1.9510595798492432 (nan) | > loss: 0.007858973927795887 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.1368444183841872) | > loader_time: 0.0046 (0.004629018394833004)  --> TIME: 2025-02-26 07:25:14 -- STEP: 34960/36938 -- GLOBAL_STEP: 1143100 | > loss_text_ce: 0.040061239153146744 (0.031904325625636665) | > loss_mel_ce: 2.012666940689087 (nan) | > loss: 0.008145746774971485 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13684851231236658) | > loader_time: 0.0045 (0.004629059009748417)  --> TIME: 2025-02-26 07:25:23 -- STEP: 35010/36938 -- GLOBAL_STEP: 1143150 | > loss_text_ce: 0.02774578519165516 (0.03190459126906667) | > loss_mel_ce: 3.2278122901916504 (nan) | > loss: 0.012918882071971893 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13684044171659826) | > loader_time: 0.0048 (0.004629008016120505)  --> TIME: 2025-02-26 07:25:32 -- STEP: 35060/36938 -- GLOBAL_STEP: 1143200 | > loss_text_ce: 0.03742780536413193 (0.03190454360333017) | > loss_mel_ce: 2.527614116668701 (nan) | > loss: 0.010178739205002785 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13684181674711285) | > loader_time: 0.0046 (0.004628972903837568)  --> TIME: 2025-02-26 07:25:42 -- STEP: 35110/36938 -- GLOBAL_STEP: 1143250 | > loss_text_ce: 0.028857644647359848 (0.03190584298681108) | > loss_mel_ce: 2.846691608428955 (nan) | > loss: 0.0114109106361866 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13684481473024654) | > loader_time: 0.0046 (0.004628981507697091)  --> TIME: 2025-02-26 07:25:51 -- STEP: 35160/36938 -- GLOBAL_STEP: 1143300 | > loss_text_ce: 0.03792266920208931 (0.031905558644486826) | > loss_mel_ce: 0.2467041015625 (nan) | > loss: 0.0011294714640825987 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13684019727782779) | > loader_time: 0.0044 (0.00462893777197403)  --> TIME: 2025-02-26 07:26:00 -- STEP: 35210/36938 -- GLOBAL_STEP: 1143350 | > loss_text_ce: 0.02331283502280712 (0.031906477113415085) | > loss_mel_ce: 2.7217698097229004 (nan) | > loss: 0.010893185622990131 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.1368402537078005) | > loader_time: 0.0045 (0.004628934131634234)  --> TIME: 2025-02-26 07:26:09 -- STEP: 35260/36938 -- GLOBAL_STEP: 1143400 | > loss_text_ce: 0.04155385494232178 (0.03190578618678542) | > loss_mel_ce: 2.6240928173065186 (nan) | > loss: 0.01057796273380518 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13683662687048406) | > loader_time: 0.0045 (0.004628942260266161)  --> TIME: 2025-02-26 07:26:19 -- STEP: 35310/36938 -- GLOBAL_STEP: 1143450 | > loss_text_ce: 0.03327369689941406 (0.03190722895186929) | > loss_mel_ce: 1.552931785583496 (nan) | > loss: 0.006294466555118561 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1368498261067381) | > loader_time: 0.0046 (0.004628963107193065)  --> TIME: 2025-02-26 07:26:28 -- STEP: 35360/36938 -- GLOBAL_STEP: 1143500 | > loss_text_ce: 0.03174041584134102 (0.03190944060023228) | > loss_mel_ce: 3.2148549556732178 (nan) | > loss: 0.012883315794169903 (nan) | > current_lr: 5e-06 | > step_time: 0.1712 (0.13684945699855683) | > loader_time: 0.0046 (0.0046289409582431496)  --> TIME: 2025-02-26 07:26:38 -- STEP: 35410/36938 -- GLOBAL_STEP: 1143550 | > loss_text_ce: 0.035960603505373 (0.03190713343085606) | > loss_mel_ce: 2.5328776836395264 (nan) | > loss: 0.01019380334764719 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1368490480831261) | > loader_time: 0.0045 (0.004628912643738568)  --> TIME: 2025-02-26 07:26:47 -- STEP: 35460/36938 -- GLOBAL_STEP: 1143600 | > loss_text_ce: 0.03177570551633835 (0.03190655087533719) | > loss_mel_ce: 3.136079788208008 (nan) | > loss: 0.012570856139063835 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13685314665389273) | > loader_time: 0.0048 (0.004628906630555318)  --> TIME: 2025-02-26 07:26:57 -- STEP: 35510/36938 -- GLOBAL_STEP: 1143650 | > loss_text_ce: 0.03489632159471512 (0.03190591981418403) | > loss_mel_ce: 2.811513662338257 (nan) | > loss: 0.011295278556644917 (nan) | > current_lr: 5e-06 | > step_time: 0.1687 (0.1368562960617974) | > loader_time: 0.0046 (0.004628863095632912)  --> TIME: 2025-02-26 07:27:06 -- STEP: 35560/36938 -- GLOBAL_STEP: 1143700 | > loss_text_ce: 0.038896482437849045 (0.03190584318194852) | > loss_mel_ce: 2.263214349746704 (nan) | > loss: 0.009135360829532146 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13685884324182074) | > loader_time: 0.0046 (0.00462881767843652)  --> TIME: 2025-02-26 07:27:15 -- STEP: 35610/36938 -- GLOBAL_STEP: 1143750 | > loss_text_ce: 0.031270429491996765 (0.03190652893056416) | > loss_mel_ce: 0.3779500424861908 (nan) | > loss: 0.0016238908283412457 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1368537280573593) | > loader_time: 0.0044 (0.004628742768904429)  --> TIME: 2025-02-26 07:27:24 -- STEP: 35660/36938 -- GLOBAL_STEP: 1143800 | > loss_text_ce: 0.03874563053250313 (0.031906973913445845) | > loss_mel_ce: 1.1259765625 (nan) | > loss: 0.004621913656592369 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1368568526648826) | > loader_time: 0.0045 (0.004628691851123293)  --> TIME: 2025-02-26 07:27:34 -- STEP: 35710/36938 -- GLOBAL_STEP: 1143850 | > loss_text_ce: 0.039161600172519684 (0.03190804129496163) | > loss_mel_ce: 2.232083559036255 (nan) | > loss: 0.009012878872454166 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13685810679449273) | > loader_time: 0.0047 (0.00462862930522183)  --> TIME: 2025-02-26 07:27:43 -- STEP: 35760/36938 -- GLOBAL_STEP: 1143900 | > loss_text_ce: 0.04425950348377228 (0.03190710516555369) | > loss_mel_ce: 2.8318591117858887 (nan) | > loss: 0.011413170024752617 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13685499531710632) | > loader_time: 0.0046 (0.004628568921046506)  --> TIME: 2025-02-26 07:27:52 -- STEP: 35810/36938 -- GLOBAL_STEP: 1143950 | > loss_text_ce: 0.022860169410705566 (0.03190771877061915) | > loss_mel_ce: 2.943809986114502 (nan) | > loss: 0.011772501282393932 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13685897330737046) | > loader_time: 0.0043 (0.004628512400616755)  --> TIME: 2025-02-26 07:28:01 -- STEP: 35860/36938 -- GLOBAL_STEP: 1144000 | > loss_text_ce: 0.03470856696367264 (0.03190707348588909) | > loss_mel_ce: 2.77606201171875 (nan) | > loss: 0.011153851635754108 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13684990197270458) | > loader_time: 0.0045 (0.004628425188325207)  --> TIME: 2025-02-26 07:28:11 -- STEP: 35910/36938 -- GLOBAL_STEP: 1144050 | > loss_text_ce: 0.03631296753883362 (0.031907792913485666) | > loss_mel_ce: 3.1020426750183105 (nan) | > loss: 0.012453793548047543 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13685321601377895) | > loader_time: 0.0047 (0.004628387721791247)  --> TIME: 2025-02-26 07:28:20 -- STEP: 35960/36938 -- GLOBAL_STEP: 1144100 | > loss_text_ce: 0.03511793166399002 (0.03190899733873568) | > loss_mel_ce: 3.6184375286102295 (nan) | > loss: 0.01449823658913374 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13685431266918263) | > loader_time: 0.0045 (0.004628329461355504)  --> TIME: 2025-02-26 07:28:29 -- STEP: 36010/36938 -- GLOBAL_STEP: 1144150 | > loss_text_ce: 0.027821039780974388 (0.03191052736625011) | > loss_mel_ce: 1.8717989921569824 (nan) | > loss: 0.007538175210356712 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13684869123743235) | > loader_time: 0.0044 (0.004628230591212278)  --> TIME: 2025-02-26 07:28:39 -- STEP: 36060/36938 -- GLOBAL_STEP: 1144200 | > loss_text_ce: 0.04112904146313667 (0.03191166416758964) | > loss_mel_ce: 2.5024526119232178 (nan) | > loss: 0.010093579068779945 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.1368486642374698) | > loader_time: 0.0045 (0.0046281753284562336)  --> TIME: 2025-02-26 07:28:48 -- STEP: 36110/36938 -- GLOBAL_STEP: 1144250 | > loss_text_ce: 0.03041802905499935 (0.03190982391136008) | > loss_mel_ce: 1.5388875007629395 (nan) | > loss: 0.006227403413504362 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13684207242930255) | > loader_time: 0.0046 (0.004628083812203386)  --> TIME: 2025-02-26 07:28:57 -- STEP: 36160/36938 -- GLOBAL_STEP: 1144300 | > loss_text_ce: 0.03548353537917137 (0.031909250618789076) | > loss_mel_ce: 2.9353790283203125 (nan) | > loss: 0.011789138428866863 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.13684379527658438) | > loader_time: 0.0046 (0.004628020043657955)  --> TIME: 2025-02-26 07:29:06 -- STEP: 36210/36938 -- GLOBAL_STEP: 1144350 | > loss_text_ce: 0.036668017506599426 (0.03190662793561033) | > loss_mel_ce: 2.700565814971924 (nan) | > loss: 0.010862039402127266 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.1368346361717571) | > loader_time: 0.0045 (0.004627905600441664)  --> TIME: 2025-02-26 07:29:16 -- STEP: 36260/36938 -- GLOBAL_STEP: 1144400 | > loss_text_ce: 0.02606811933219433 (0.03190512765343818) | > loss_mel_ce: 2.2583861351013184 (nan) | > loss: 0.009065295569598675 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13683880810834873) | > loader_time: 0.0046 (0.0046278675879422295)  --> TIME: 2025-02-26 07:29:25 -- STEP: 36310/36938 -- GLOBAL_STEP: 1144450 | > loss_text_ce: 0.03788582608103752 (0.03190366758253714) | > loss_mel_ce: 3.188537359237671 (nan) | > loss: 0.012803267687559128 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13684478912495285) | > loader_time: 0.0046 (0.004627858131459133)  --> TIME: 2025-02-26 07:29:35 -- STEP: 36360/36938 -- GLOBAL_STEP: 1144500 | > loss_text_ce: 0.0258964691311121 (0.031901468236585497) | > loss_mel_ce: 2.1179683208465576 (nan) | > loss: 0.008507400751113892 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.1368451594716245) | > loader_time: 0.0048 (0.004627763766004456)  --> TIME: 2025-02-26 07:29:44 -- STEP: 36410/36938 -- GLOBAL_STEP: 1144550 | > loss_text_ce: 0.032053232192993164 (0.03190184127696971) | > loss_mel_ce: 2.304530620574951 (nan) | > loss: 0.009272159077227116 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.13684231850577663) | > loader_time: 0.0046 (0.004627664198557205)  --> TIME: 2025-02-26 07:29:53 -- STEP: 36460/36938 -- GLOBAL_STEP: 1144600 | > loss_text_ce: 0.028471779078245163 (0.03190100826236193) | > loss_mel_ce: 2.231072425842285 (nan) | > loss: 0.008966445922851562 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1368481362251803) | > loader_time: 0.0045 (0.004627593565431868)  --> TIME: 2025-02-26 07:30:03 -- STEP: 36510/36938 -- GLOBAL_STEP: 1144650 | > loss_text_ce: 0.03636689484119415 (0.031901828894379994) | > loss_mel_ce: 2.7593588829040527 (nan) | > loss: 0.011094151064753532 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13685010937722283) | > loader_time: 0.0046 (0.004627666731985673)  --> TIME: 2025-02-26 07:30:12 -- STEP: 36560/36938 -- GLOBAL_STEP: 1144700 | > loss_text_ce: 0.039657801389694214 (0.03190253303570711) | > loss_mel_ce: 3.0176172256469727 (nan) | > loss: 0.01213204488158226 (nan) | > current_lr: 5e-06 | > step_time: 0.1679 (0.13685281469081056) | > loader_time: 0.0046 (0.004627700701062498)  --> TIME: 2025-02-26 07:30:22 -- STEP: 36610/36938 -- GLOBAL_STEP: 1144750 | > loss_text_ce: 0.02219843491911888 (0.03190412067616664) | > loss_mel_ce: 2.4694957733154297 (nan) | > loss: 0.009887675754725933 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13685086502204358) | > loader_time: 0.0046 (0.00462770023752323)  --> TIME: 2025-02-26 07:30:31 -- STEP: 36660/36938 -- GLOBAL_STEP: 1144800 | > loss_text_ce: 0.029259122908115387 (0.0319046010101185) | > loss_mel_ce: 3.0469167232513428 (nan) | > loss: 0.012207048013806343 (nan) | > current_lr: 5e-06 | > step_time: 0.1848 (0.13684909917979127) | > loader_time: 0.0048 (0.004627779235101071)  --> TIME: 2025-02-26 07:30:40 -- STEP: 36710/36938 -- GLOBAL_STEP: 1144850 | > loss_text_ce: 0.029123984277248383 (0.03190194527765482) | > loss_mel_ce: 2.549412488937378 (nan) | > loss: 0.010232288390398026 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13684024537366224) | > loader_time: 0.0046 (0.004627745172952144)  --> TIME: 2025-02-26 07:30:49 -- STEP: 36760/36938 -- GLOBAL_STEP: 1144900 | > loss_text_ce: 0.027751466259360313 (0.03190124364183678) | > loss_mel_ce: 2.3122146129608154 (nan) | > loss: 0.0092855803668499 (nan) | > current_lr: 5e-06 | > step_time: 0.1355 (0.13684327711996697) | > loader_time: 0.0048 (0.004627791964839177)  --> TIME: 2025-02-26 07:30:59 -- STEP: 36810/36938 -- GLOBAL_STEP: 1144950 | > loss_text_ce: 0.03312983736395836 (0.031899503948704895) | > loss_mel_ce: 3.279651165008545 (nan) | > loss: 0.01314595714211464 (nan) | > current_lr: 5e-06 | > step_time: 0.1703 (0.1368459485465171) | > loader_time: 0.0049 (0.004627829464645852)  --> TIME: 2025-02-26 07:31:08 -- STEP: 36860/36938 -- GLOBAL_STEP: 1145000 | > loss_text_ce: 0.025604264810681343 (0.0318972637356379) | > loss_mel_ce: 1.5991482734680176 (nan) | > loss: 0.006447431165724993 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.136850485184628) | > loader_time: 0.0045 (0.004627803888129494)  --> TIME: 2025-02-26 07:31:18 -- STEP: 36910/36938 -- GLOBAL_STEP: 1145050 | > loss_text_ce: 0.04481915384531021 (0.031898012335367584) | > loss_mel_ce: 3.2267301082611084 (nan) | > loss: 0.012982338666915894 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1368517792344378) | > loader_time: 0.0046 (0.004627758782913128)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.021000033558005168 (-1.971910495568094e-05) | > avg_loss_text_ce: 0.03104875784477975 (-0.00020637004497911504) | > avg_loss_mel_ce: 2.4722151328431505 (-0.005899393912588646) | > avg_loss: 2.5032638945791987 (-0.006105756700628184) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_1145078.pth  > EPOCH: 32/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-26 07:31:47)   --> TIME: 2025-02-26 07:31:52 -- STEP: 22/36938 -- GLOBAL_STEP: 1145100 | > loss_text_ce: 0.028975866734981537 (0.030160481110215187) | > loss_mel_ce: 1.800623893737793 (nan) | > loss: 0.0072603169828653336 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13508221236142245) | > loader_time: 0.0047 (0.004840850830078125)  --> TIME: 2025-02-26 07:32:01 -- STEP: 72/36938 -- GLOBAL_STEP: 1145150 | > loss_text_ce: 0.03648434951901436 (0.03173739201803173) | > loss_mel_ce: 2.499434471130371 (nan) | > loss: 0.010063170455396175 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.1387987666659885) | > loader_time: 0.0047 (0.004906684160232543)  --> TIME: 2025-02-26 07:32:11 -- STEP: 122/36938 -- GLOBAL_STEP: 1145200 | > loss_text_ce: 0.02630850300192833 (0.031774049090435276) | > loss_mel_ce: 0.11678314208984375 (nan) | > loss: 0.0005678240559063852 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13885174813817752) | > loader_time: 0.0047 (0.0048271104937694116)  --> TIME: 2025-02-26 07:32:20 -- STEP: 172/36938 -- GLOBAL_STEP: 1145250 | > loss_text_ce: 0.02969713881611824 (0.031674932926719974) | > loss_mel_ce: 2.811108112335205 (nan) | > loss: 0.011273037642240524 (nan) | > current_lr: 5e-06 | > step_time: 0.1955 (0.1374490538308786) | > loader_time: 0.0049 (0.004786374957062478)  --> TIME: 2025-02-26 07:32:29 -- STEP: 222/36938 -- GLOBAL_STEP: 1145300 | > loss_text_ce: 0.03541604429483414 (0.03201951548826321) | > loss_mel_ce: 3.196701765060425 (nan) | > loss: 0.012825865298509598 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13708484280216798) | > loader_time: 0.0048 (0.004782141865910712)  --> TIME: 2025-02-26 07:32:38 -- STEP: 272/36938 -- GLOBAL_STEP: 1145350 | > loss_text_ce: 0.0262959785759449 (0.031934309079695314) | > loss_mel_ce: 2.675520420074463 (nan) | > loss: 0.010721493512392044 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1365097787450342) | > loader_time: 0.0046 (0.004785577163976783)  --> TIME: 2025-02-26 07:32:48 -- STEP: 322/36938 -- GLOBAL_STEP: 1145400 | > loss_text_ce: 0.031911589205265045 (0.032141761270045165) | > loss_mel_ce: 1.8418289422988892 (nan) | > loss: 0.007435478735715151 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.1369967194077391) | > loader_time: 0.005 (0.004786554330624412)  --> TIME: 2025-02-26 07:32:57 -- STEP: 372/36938 -- GLOBAL_STEP: 1145450 | > loss_text_ce: 0.027143895626068115 (0.03208831881964081) | > loss_mel_ce: 2.7631099224090576 (nan) | > loss: 0.011072436347603798 (nan) | > current_lr: 5e-06 | > step_time: 0.1997 (0.13640912822497783) | > loader_time: 0.0051 (0.004790789337568388)  --> TIME: 2025-02-26 07:33:06 -- STEP: 422/36938 -- GLOBAL_STEP: 1145500 | > loss_text_ce: 0.029063617810606956 (0.03201964694865368) | > loss_mel_ce: 2.4500536918640137 (nan) | > loss: 0.009837768040597439 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13652581400215913) | > loader_time: 0.0047 (0.004789778406586128)  --> TIME: 2025-02-26 07:33:16 -- STEP: 472/36938 -- GLOBAL_STEP: 1145550 | > loss_text_ce: 0.031979046761989594 (0.03218512207474877) | > loss_mel_ce: 2.908457040786743 (nan) | > loss: 0.011668398045003414 (nan) | > current_lr: 5e-06 | > step_time: 0.1453 (0.1368007765988172) | > loader_time: 0.0048 (0.0047847992282802785)  --> TIME: 2025-02-26 07:33:26 -- STEP: 522/36938 -- GLOBAL_STEP: 1145600 | > loss_text_ce: 0.03201954439282417 (0.032190251024975185) | > loss_mel_ce: 2.622738838195801 (nan) | > loss: 0.010534756816923618 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13748656378851998) | > loader_time: 0.0046 (0.004779747162742177)  --> TIME: 2025-02-26 07:33:35 -- STEP: 572/36938 -- GLOBAL_STEP: 1145650 | > loss_text_ce: 0.028034498915076256 (0.032127569312347) | > loss_mel_ce: 2.7274539470672607 (nan) | > loss: 0.010934478603303432 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13745966616210406) | > loader_time: 0.0046 (0.004768645846760358)  --> TIME: 2025-02-26 07:33:44 -- STEP: 622/36938 -- GLOBAL_STEP: 1145700 | > loss_text_ce: 0.04413510859012604 (0.032097389475349646) | > loss_mel_ce: 3.06925368309021 (nan) | > loss: 0.012354718521237373 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13752256290705658) | > loader_time: 0.0045 (0.004759981701228397)  --> TIME: 2025-02-26 07:33:53 -- STEP: 672/36938 -- GLOBAL_STEP: 1145750 | > loss_text_ce: 0.028854185715317726 (0.03217350854815561) | > loss_mel_ce: 3.1400156021118164 (nan) | > loss: 0.012574880383908749 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.1372458963167101) | > loader_time: 0.0047 (0.0047560131975582665)  --> TIME: 2025-02-26 07:34:02 -- STEP: 722/36938 -- GLOBAL_STEP: 1145800 | > loss_text_ce: 0.03393897786736488 (0.03213318597803982) | > loss_mel_ce: 1.5781792402267456 (nan) | > loss: 0.00639729481190443 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13701365991312403) | > loader_time: 0.0047 (0.004754903904288758)  --> TIME: 2025-02-26 07:34:12 -- STEP: 772/36938 -- GLOBAL_STEP: 1145850 | > loss_text_ce: 0.0406964048743248 (0.03216075497664938) | > loss_mel_ce: 2.3689932823181152 (nan) | > loss: 0.009562261402606964 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.1374671891563299) | > loader_time: 0.0049 (0.004751048248666551)  --> TIME: 2025-02-26 07:34:21 -- STEP: 822/36938 -- GLOBAL_STEP: 1145900 | > loss_text_ce: 0.03183433786034584 (0.03217071923370638) | > loss_mel_ce: 2.8720197677612305 (nan) | > loss: 0.01152323093265295 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.137317476771464) | > loader_time: 0.0047 (0.004746004612776479)  --> TIME: 2025-02-26 07:34:31 -- STEP: 872/36938 -- GLOBAL_STEP: 1145950 | > loss_text_ce: 0.02758815698325634 (0.032086063494666625) | > loss_mel_ce: 2.7051608562469482 (nan) | > loss: 0.010844242759048939 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13713873936495682) | > loader_time: 0.0047 (0.004740907784995677)  --> TIME: 2025-02-26 07:34:40 -- STEP: 922/36938 -- GLOBAL_STEP: 1146000 | > loss_text_ce: 0.03915579617023468 (0.0321481022882779) | > loss_mel_ce: 1.8579349517822266 (nan) | > loss: 0.007528138346970081 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13696430587975456) | > loader_time: 0.0046 (0.004735586441519973)  --> TIME: 2025-02-26 07:34:49 -- STEP: 972/36938 -- GLOBAL_STEP: 1146050 | > loss_text_ce: 0.031003078445792198 (0.03207110862033025) | > loss_mel_ce: 2.9533276557922363 (nan) | > loss: 0.011842582374811172 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.13694809420118637) | > loader_time: 0.0047 (0.004734947112362075)  --> TIME: 2025-02-26 07:34:58 -- STEP: 1022/36938 -- GLOBAL_STEP: 1146100 | > loss_text_ce: 0.034764762967824936 (0.03202985577037429) | > loss_mel_ce: 3.1619091033935547 (nan) | > loss: 0.012685214169323444 (nan) | > current_lr: 5e-06 | > step_time: 0.1892 (0.1369791971027504) | > loader_time: 0.0048 (0.0047353247140484965)  --> TIME: 2025-02-26 07:35:08 -- STEP: 1072/36938 -- GLOBAL_STEP: 1146150 | > loss_text_ce: 0.03161539137363434 (0.03202605473030531) | > loss_mel_ce: 2.909308910369873 (nan) | > loss: 0.011670335195958614 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13693319817087568) | > loader_time: 0.0047 (0.004734986086389907)  --> TIME: 2025-02-26 07:35:17 -- STEP: 1122/36938 -- GLOBAL_STEP: 1146200 | > loss_text_ce: 0.03271669149398804 (0.03200811139057247) | > loss_mel_ce: 2.600041389465332 (nan) | > loss: 0.010447453707456589 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.1368962564570384) | > loader_time: 0.0046 (0.0047303038697404185)  --> TIME: 2025-02-26 07:35:26 -- STEP: 1172/36938 -- GLOBAL_STEP: 1146250 | > loss_text_ce: 0.03838258236646652 (0.03201800294271313) | > loss_mel_ce: 2.7406628131866455 (nan) | > loss: 0.011027958244085312 (nan) | > current_lr: 5e-06 | > step_time: 0.1447 (0.13695301191798687) | > loader_time: 0.0046 (0.004725875911452252)  --> TIME: 2025-02-26 07:35:35 -- STEP: 1222/36938 -- GLOBAL_STEP: 1146300 | > loss_text_ce: 0.04187139496207237 (0.03197772291210016) | > loss_mel_ce: 2.3520851135253906 (nan) | > loss: 0.009499827399849892 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13683698454387097) | > loader_time: 0.0047 (0.0047210747598626195)  --> TIME: 2025-02-26 07:35:45 -- STEP: 1272/36938 -- GLOBAL_STEP: 1146350 | > loss_text_ce: 0.03383004665374756 (0.03195966331510806) | > loss_mel_ce: 1.8687330484390259 (nan) | > loss: 0.0075498539954423904 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13711548038998528) | > loader_time: 0.0046 (0.004721573126391044)  --> TIME: 2025-02-26 07:35:55 -- STEP: 1322/36938 -- GLOBAL_STEP: 1146400 | > loss_text_ce: 0.027014341205358505 (0.03199476568358034) | > loss_mel_ce: 0.0091552734375 (nan) | > loss: 0.0001435302256140858 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13724203567822335) | > loader_time: 0.0046 (0.0047205470152954475)  --> TIME: 2025-02-26 07:36:04 -- STEP: 1372/36938 -- GLOBAL_STEP: 1146450 | > loss_text_ce: 0.030401641502976418 (0.03204088054228115) | > loss_mel_ce: 1.1919759511947632 (nan) | > loss: 0.004850704688578844 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13732875940055989) | > loader_time: 0.0046 (0.004721007611244139)  --> TIME: 2025-02-26 07:36:14 -- STEP: 1422/36938 -- GLOBAL_STEP: 1146500 | > loss_text_ce: 0.02624530717730522 (0.031990630146110886) | > loss_mel_ce: 3.084371566772461 (nan) | > loss: 0.01234371867030859 (nan) | > current_lr: 5e-06 | > step_time: 0.1574 (0.13736646939933586) | > loader_time: 0.0046 (0.004718762409837942)  --> TIME: 2025-02-26 07:36:23 -- STEP: 1472/36938 -- GLOBAL_STEP: 1146550 | > loss_text_ce: 0.0342877171933651 (0.0320248276491286) | > loss_mel_ce: 2.8801205158233643 (nan) | > loss: 0.011565112508833408 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13715674307035397) | > loader_time: 0.0045 (0.004716613046500999)  --> TIME: 2025-02-26 07:36:32 -- STEP: 1522/36938 -- GLOBAL_STEP: 1146600 | > loss_text_ce: 0.030644940212368965 (0.03203713962791124) | > loss_mel_ce: 2.6324594020843506 (nan) | > loss: 0.010567874647676945 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13725780955125436) | > loader_time: 0.0045 (0.004713679268544988)  --> TIME: 2025-02-26 07:36:42 -- STEP: 1572/36938 -- GLOBAL_STEP: 1146650 | > loss_text_ce: 0.028066350147128105 (0.03203841718032244) | > loss_mel_ce: 2.9005126953125 (nan) | > loss: 0.011621346697211266 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13728128756578972) | > loader_time: 0.0046 (0.004709880922283534)  --> TIME: 2025-02-26 07:36:51 -- STEP: 1622/36938 -- GLOBAL_STEP: 1146700 | > loss_text_ce: 0.03918636962771416 (0.03205996431736455) | > loss_mel_ce: 3.481513738632202 (nan) | > loss: 0.01397103350609541 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.13722861430500952) | > loader_time: 0.0046 (0.004706666296125191)  --> TIME: 2025-02-26 07:37:01 -- STEP: 1672/36938 -- GLOBAL_STEP: 1146750 | > loss_text_ce: 0.03349098190665245 (0.03202019980105994) | > loss_mel_ce: 0.003387451171875 (nan) | > loss: 0.00014634299441240728 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13735976629850413) | > loader_time: 0.0048 (0.004709262311743777)  --> TIME: 2025-02-26 07:37:10 -- STEP: 1722/36938 -- GLOBAL_STEP: 1146800 | > loss_text_ce: 0.031048323959112167 (0.031980923596557645) | > loss_mel_ce: 2.135397434234619 (nan) | > loss: 0.008597007021307945 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13717868269943626) | > loader_time: 0.0046 (0.004709130241757346)  --> TIME: 2025-02-26 07:37:19 -- STEP: 1772/36938 -- GLOBAL_STEP: 1146850 | > loss_text_ce: 0.022798778489232063 (0.03196694373214123) | > loss_mel_ce: 2.6823642253875732 (nan) | > loss: 0.01073477417230606 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13712739554269593) | > loader_time: 0.0045 (0.004706364722068905)  --> TIME: 2025-02-26 07:37:29 -- STEP: 1822/36938 -- GLOBAL_STEP: 1146900 | > loss_text_ce: 0.026851331815123558 (0.03198037713226324) | > loss_mel_ce: 2.767437219619751 (nan) | > loss: 0.011088447645306587 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13728148837775228) | > loader_time: 0.0046 (0.0047060972249337776)  --> TIME: 2025-02-26 07:37:38 -- STEP: 1872/36938 -- GLOBAL_STEP: 1146950 | > loss_text_ce: 0.0372239351272583 (0.03198547496731027) | > loss_mel_ce: 2.471102237701416 (nan) | > loss: 0.009953675791621208 (nan) | > current_lr: 5e-06 | > step_time: 0.1353 (0.13733484105676674) | > loader_time: 0.0046 (0.00470605021358555)  --> TIME: 2025-02-26 07:37:47 -- STEP: 1922/36938 -- GLOBAL_STEP: 1147000 | > loss_text_ce: 0.03523926064372063 (0.03197062488463129) | > loss_mel_ce: 2.881575345993042 (nan) | > loss: 0.01157466135919094 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1372388695578918) | > loader_time: 0.0046 (0.004705775788869866)  --> TIME: 2025-02-26 07:37:56 -- STEP: 1972/36938 -- GLOBAL_STEP: 1147050 | > loss_text_ce: 0.02360115386545658 (0.031954197508696776) | > loss_mel_ce: 2.186763286590576 (nan) | > loss: 0.008771287277340889 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13716838712866367) | > loader_time: 0.0045 (0.00470628929428348)  --> TIME: 2025-02-26 07:38:06 -- STEP: 2022/36938 -- GLOBAL_STEP: 1147100 | > loss_text_ce: 0.03054906241595745 (0.031939900845923117) | > loss_mel_ce: 2.198009729385376 (nan) | > loss: 0.008843488059937954 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1373206878156502) | > loader_time: 0.0045 (0.004707279238337695)  --> TIME: 2025-02-26 07:38:15 -- STEP: 2072/36938 -- GLOBAL_STEP: 1147150 | > loss_text_ce: 0.03342556953430176 (0.03191991002814905) | > loss_mel_ce: 2.6045262813568115 (nan) | > loss: 0.01046806387603283 (nan) | > current_lr: 5e-06 | > step_time: 0.1848 (0.1372640209998864) | > loader_time: 0.0048 (0.004704847759261553)  --> TIME: 2025-02-26 07:38:25 -- STEP: 2122/36938 -- GLOBAL_STEP: 1147200 | > loss_text_ce: 0.03312024474143982 (0.03194696084323238) | > loss_mel_ce: 3.1630098819732666 (nan) | > loss: 0.01268305629491806 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.1372952708424992) | > loader_time: 0.0048 (0.004705323687597449)  --> TIME: 2025-02-26 07:38:35 -- STEP: 2172/36938 -- GLOBAL_STEP: 1147250 | > loss_text_ce: 0.03207347169518471 (0.03197234173813026) | > loss_mel_ce: 2.231015920639038 (nan) | > loss: 0.008980514481663704 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.13744488803062654) | > loader_time: 0.0048 (0.00470492227301413)  --> TIME: 2025-02-26 07:38:44 -- STEP: 2222/36938 -- GLOBAL_STEP: 1147300 | > loss_text_ce: 0.03226756677031517 (0.03196536287167444) | > loss_mel_ce: 2.8327701091766357 (nan) | > loss: 0.011369197629392147 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.1375046044614914) | > loader_time: 0.0046 (0.004703234381551252)  --> TIME: 2025-02-26 07:38:53 -- STEP: 2272/36938 -- GLOBAL_STEP: 1147350 | > loss_text_ce: 0.0327017605304718 (0.03195989461866561) | > loss_mel_ce: 2.901318311691284 (nan) | > loss: 0.011642937548458576 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.1374667668846291) | > loader_time: 0.0047 (0.0047017036819122175)  --> TIME: 2025-02-26 07:39:03 -- STEP: 2322/36938 -- GLOBAL_STEP: 1147400 | > loss_text_ce: 0.03466528281569481 (0.03196852169666231) | > loss_mel_ce: 3.0953638553619385 (nan) | > loss: 0.01242075115442276 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13746244295417584) | > loader_time: 0.0047 (0.0047006072102693695)  --> TIME: 2025-02-26 07:39:12 -- STEP: 2372/36938 -- GLOBAL_STEP: 1147450 | > loss_text_ce: 0.03397316485643387 (0.03198396851242692) | > loss_mel_ce: 2.4014363288879395 (nan) | > loss: 0.0096643241122365 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.1374433260743975) | > loader_time: 0.0046 (0.00469960249696574)  --> TIME: 2025-02-26 07:39:21 -- STEP: 2422/36938 -- GLOBAL_STEP: 1147500 | > loss_text_ce: 0.03371518850326538 (0.03197448951569815) | > loss_mel_ce: 2.395967960357666 (nan) | > loss: 0.009641600772738457 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13737702192500229) | > loader_time: 0.0046 (0.004699444002990976)  --> TIME: 2025-02-26 07:39:31 -- STEP: 2472/36938 -- GLOBAL_STEP: 1147550 | > loss_text_ce: 0.04034540057182312 (0.031992731143835114) | > loss_mel_ce: 2.8673765659332275 (nan) | > loss: 0.011538580060005188 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13736990923634604) | > loader_time: 0.0048 (0.0046981780467295715)  --> TIME: 2025-02-26 07:39:40 -- STEP: 2522/36938 -- GLOBAL_STEP: 1147600 | > loss_text_ce: 0.034696064889431 (0.03197273149278432) | > loss_mel_ce: 2.8849172592163086 (nan) | > loss: 0.011585768312215805 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.137349196410576) | > loader_time: 0.0048 (0.0046969724968253)  --> TIME: 2025-02-26 07:39:49 -- STEP: 2572/36938 -- GLOBAL_STEP: 1147650 | > loss_text_ce: 0.037882354110479355 (0.032005457795974024) | > loss_mel_ce: 2.5288920402526855 (nan) | > loss: 0.010185613296926022 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.13732144528562493) | > loader_time: 0.005 (0.004696921927747237)  --> TIME: 2025-02-26 07:39:59 -- STEP: 2622/36938 -- GLOBAL_STEP: 1147700 | > loss_text_ce: 0.0282436516135931 (0.031988078028228034) | > loss_mel_ce: 2.992220878601074 (nan) | > loss: 0.01198597066104412 (nan) | > current_lr: 5e-06 | > step_time: 0.1761 (0.13737429443158586) | > loader_time: 0.0048 (0.004697072696904022)  --> TIME: 2025-02-26 07:40:08 -- STEP: 2672/36938 -- GLOBAL_STEP: 1147750 | > loss_text_ce: 0.037282466888427734 (0.03196287921366357) | > loss_mel_ce: 2.4325547218322754 (nan) | > loss: 0.009800941683351994 (nan) | > current_lr: 5e-06 | > step_time: 0.1828 (0.1374058659205178) | > loader_time: 0.0048 (0.00469610184252619)  --> TIME: 2025-02-26 07:40:17 -- STEP: 2722/36938 -- GLOBAL_STEP: 1147800 | > loss_text_ce: 0.023929791525006294 (0.03194300185657732) | > loss_mel_ce: 2.60105299949646 (nan) | > loss: 0.010416598990559578 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.1373657920971939) | > loader_time: 0.0047 (0.004694588389316092)  --> TIME: 2025-02-26 07:40:27 -- STEP: 2772/36938 -- GLOBAL_STEP: 1147850 | > loss_text_ce: 0.03007480315864086 (0.03196574658495465) | > loss_mel_ce: 2.572152853012085 (nan) | > loss: 0.010326300747692585 (nan) | > current_lr: 5e-06 | > step_time: 0.1531 (0.13741466011663872) | > loader_time: 0.0047 (0.0046936153161405425)  --> TIME: 2025-02-26 07:40:36 -- STEP: 2822/36938 -- GLOBAL_STEP: 1147900 | > loss_text_ce: 0.0321943573653698 (0.031961046978052604) | > loss_mel_ce: 3.6227922439575195 (nan) | > loss: 0.014503915794193745 (nan) | > current_lr: 5e-06 | > step_time: 0.1854 (0.137415972861054) | > loader_time: 0.0048 (0.0046922197416266465)  --> TIME: 2025-02-26 07:40:46 -- STEP: 2872/36938 -- GLOBAL_STEP: 1147950 | > loss_text_ce: 0.031396958976984024 (0.031946951534124496) | > loss_mel_ce: 3.2142202854156494 (nan) | > loss: 0.012879434041678905 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1375386489134976) | > loader_time: 0.0046 (0.004691425057174779)  --> TIME: 2025-02-26 07:40:55 -- STEP: 2922/36938 -- GLOBAL_STEP: 1148000 | > loss_text_ce: 0.029200034216046333 (0.031944861609111516) | > loss_mel_ce: 3.15083909034729 (nan) | > loss: 0.012619203887879848 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13750651370654923) | > loader_time: 0.0046 (0.004689750893323259)  --> TIME: 2025-02-26 07:41:05 -- STEP: 2972/36938 -- GLOBAL_STEP: 1148050 | > loss_text_ce: 0.030676348134875298 (0.03193233891084434) | > loss_mel_ce: 2.696053981781006 (nan) | > loss: 0.010820358991622925 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13750744908847518) | > loader_time: 0.0046 (0.004689775437397407)  --> TIME: 2025-02-26 07:41:14 -- STEP: 3022/36938 -- GLOBAL_STEP: 1148100 | > loss_text_ce: 0.02755293995141983 (0.03190873227526604) | > loss_mel_ce: 2.028273582458496 (nan) | > loss: 0.008158042095601559 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.1374460842970741) | > loader_time: 0.0048 (0.004689104899438626)  --> TIME: 2025-02-26 07:41:23 -- STEP: 3072/36938 -- GLOBAL_STEP: 1148150 | > loss_text_ce: 0.03175757825374603 (0.031901481256985324) | > loss_mel_ce: 1.24651038646698 (nan) | > loss: 0.005072492174804211 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13732575027582536) | > loader_time: 0.0046 (0.004688442917540672)  --> TIME: 2025-02-26 07:41:32 -- STEP: 3122/36938 -- GLOBAL_STEP: 1148200 | > loss_text_ce: 0.023124560713768005 (0.0319042958451638) | > loss_mel_ce: 2.898505926132202 (nan) | > loss: 0.01159377209842205 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.13730796616020885) | > loader_time: 0.0048 (0.004687429467200623)  --> TIME: 2025-02-26 07:41:42 -- STEP: 3172/36938 -- GLOBAL_STEP: 1148250 | > loss_text_ce: 0.04226289689540863 (0.03190232764854383) | > loss_mel_ce: 2.8939292430877686 (nan) | > loss: 0.011651556007564068 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13742587567878975) | > loader_time: 0.0045 (0.004686689767681184)  --> TIME: 2025-02-26 07:41:51 -- STEP: 3222/36938 -- GLOBAL_STEP: 1148300 | > loss_text_ce: 0.03368455171585083 (0.03191697029633927) | > loss_mel_ce: 1.0000766515731812 (nan) | > loss: 0.0041022272780537605 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13743553251897503) | > loader_time: 0.0045 (0.004686525562104048)  --> TIME: 2025-02-26 07:42:01 -- STEP: 3272/36938 -- GLOBAL_STEP: 1148350 | > loss_text_ce: 0.03500944375991821 (0.03190115236331127) | > loss_mel_ce: 3.298546552658081 (nan) | > loss: 0.013228396885097027 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.1374644170763438) | > loader_time: 0.0046 (0.0046855531141052585)  --> TIME: 2025-02-26 07:42:10 -- STEP: 3322/36938 -- GLOBAL_STEP: 1148400 | > loss_text_ce: 0.029251741245388985 (0.031918830536575175) | > loss_mel_ce: 3.1507537364959717 (nan) | > loss: 0.012619070708751678 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.1373937961783915) | > loader_time: 0.0046 (0.004684400156561972)  --> TIME: 2025-02-26 07:42:19 -- STEP: 3372/36938 -- GLOBAL_STEP: 1148450 | > loss_text_ce: 0.033529672771692276 (0.03191316211435953) | > loss_mel_ce: 2.5691094398498535 (nan) | > loss: 0.01032793428748846 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13737255115667388) | > loader_time: 0.0047 (0.0046837265522595975)  --> TIME: 2025-02-26 07:42:28 -- STEP: 3422/36938 -- GLOBAL_STEP: 1148500 | > loss_text_ce: 0.03284242004156113 (0.03191144275601504) | > loss_mel_ce: 2.8368897438049316 (nan) | > loss: 0.01138782687485218 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13733250707440142) | > loader_time: 0.0048 (0.004682614188804604)  --> TIME: 2025-02-26 07:42:38 -- STEP: 3472/36938 -- GLOBAL_STEP: 1148550 | > loss_text_ce: 0.025052590295672417 (0.03191262071878207) | > loss_mel_ce: 3.2091877460479736 (nan) | > loss: 0.012834287248551846 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13739226167927118) | > loader_time: 0.0047 (0.004681847474542081)  --> TIME: 2025-02-26 07:42:47 -- STEP: 3522/36938 -- GLOBAL_STEP: 1148600 | > loss_text_ce: 0.03712966665625572 (0.031924559901617224) | > loss_mel_ce: 2.934802770614624 (nan) | > loss: 0.01179338339716196 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13738602701064723) | > loader_time: 0.0045 (0.004680821463711082)  --> TIME: 2025-02-26 07:42:56 -- STEP: 3572/36938 -- GLOBAL_STEP: 1148650 | > loss_text_ce: 0.027801355347037315 (0.0319392298885872) | > loss_mel_ce: 3.1984212398529053 (nan) | > loss: 0.012802471406757832 (nan) | > current_lr: 5e-06 | > step_time: 0.1634 (0.13741208564527602) | > loader_time: 0.0047 (0.004680241609340553)  --> TIME: 2025-02-26 07:43:06 -- STEP: 3622/36938 -- GLOBAL_STEP: 1148700 | > loss_text_ce: 0.03300242871046066 (0.03193627761508245) | > loss_mel_ce: 3.173901319503784 (nan) | > loss: 0.012725808657705784 (nan) | > current_lr: 5e-06 | > step_time: 0.1466 (0.13742624252030744) | > loader_time: 0.0047 (0.004679448363663807)  --> TIME: 2025-02-26 07:43:15 -- STEP: 3672/36938 -- GLOBAL_STEP: 1148750 | > loss_text_ce: 0.02035515196621418 (0.031942767445414105) | > loss_mel_ce: 1.2954051494598389 (nan) | > loss: 0.005221270956099033 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.1373885346004384) | > loader_time: 0.0045 (0.004678241827389254)  --> TIME: 2025-02-26 07:43:24 -- STEP: 3722/36938 -- GLOBAL_STEP: 1148800 | > loss_text_ce: 0.03303854539990425 (0.031944147597153735) | > loss_mel_ce: 2.8655786514282227 (nan) | > loss: 0.011502450332045555 (nan) | > current_lr: 5e-06 | > step_time: 0.1912 (0.13742160323100452) | > loader_time: 0.005 (0.004677426488344417)  --> TIME: 2025-02-26 07:43:34 -- STEP: 3772/36938 -- GLOBAL_STEP: 1148850 | > loss_text_ce: 0.031794995069503784 (0.031940896492421926) | > loss_mel_ce: 3.180246591567993 (nan) | > loss: 0.012746197171509266 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.1374812173565269) | > loader_time: 0.0046 (0.004676627139397921)  --> TIME: 2025-02-26 07:43:43 -- STEP: 3822/36938 -- GLOBAL_STEP: 1148900 | > loss_text_ce: 0.023838026449084282 (0.031919342093608444) | > loss_mel_ce: 3.451286792755127 (nan) | > loss: 0.013790179044008255 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13747044841011785) | > loader_time: 0.0045 (0.004675939905275418)  --> TIME: 2025-02-26 07:43:53 -- STEP: 3872/36938 -- GLOBAL_STEP: 1148950 | > loss_text_ce: 0.02960609644651413 (0.031922512743304716) | > loss_mel_ce: 2.559103488922119 (nan) | > loss: 0.010272657498717308 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13745138597143566) | > loader_time: 0.0045 (0.004675079968349993)  --> TIME: 2025-02-26 07:44:02 -- STEP: 3922/36938 -- GLOBAL_STEP: 1149000 | > loss_text_ce: 0.03914734348654747 (0.031914721414768026) | > loss_mel_ce: 2.709756851196289 (nan) | > loss: 0.010908350348472595 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13741766901906186) | > loader_time: 0.0046 (0.00467384572785342)  --> TIME: 2025-02-26 07:44:11 -- STEP: 3972/36938 -- GLOBAL_STEP: 1149050 | > loss_text_ce: 0.02231908217072487 (0.03190868828058341) | > loss_mel_ce: 2.640378713607788 (nan) | > loss: 0.01056626159697771 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13735328820296747) | > loader_time: 0.0046 (0.004672579954998251)  --> TIME: 2025-02-26 07:44:20 -- STEP: 4022/36938 -- GLOBAL_STEP: 1149100 | > loss_text_ce: 0.019042998552322388 (0.03190865523002647) | > loss_mel_ce: 1.896195650100708 (nan) | > loss: 0.007600153796374798 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13735296950299944) | > loader_time: 0.0045 (0.004671603633893894)  --> TIME: 2025-02-26 07:44:30 -- STEP: 4072/36938 -- GLOBAL_STEP: 1149150 | > loss_text_ce: 0.03917935863137245 (0.031910131835472186) | > loss_mel_ce: 2.7073395252227783 (nan) | > loss: 0.010898885317146778 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13733649095758224) | > loader_time: 0.0045 (0.004670592211552828)  --> TIME: 2025-02-26 07:44:39 -- STEP: 4122/36938 -- GLOBAL_STEP: 1149200 | > loss_text_ce: 0.027353515848517418 (0.03190929900216789) | > loss_mel_ce: 3.0567779541015625 (nan) | > loss: 0.012238617986440659 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.13736979016281325) | > loader_time: 0.0046 (0.004670695446464172)  --> TIME: 2025-02-26 07:44:49 -- STEP: 4172/36938 -- GLOBAL_STEP: 1149250 | > loss_text_ce: 0.035555265843868256 (0.03191701822832943) | > loss_mel_ce: 3.106879234313965 (nan) | > loss: 0.012469979003071785 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13735460554993376) | > loader_time: 0.0044 (0.004669754411435882)  --> TIME: 2025-02-26 07:44:58 -- STEP: 4222/36938 -- GLOBAL_STEP: 1149300 | > loss_text_ce: 0.03240408003330231 (0.031925323522949425) | > loss_mel_ce: 3.0086822509765625 (nan) | > loss: 0.012067804113030434 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13733814565675717) | > loader_time: 0.0046 (0.0046688997593043475)  --> TIME: 2025-02-26 07:45:07 -- STEP: 4272/36938 -- GLOBAL_STEP: 1149350 | > loss_text_ce: 0.02925366722047329 (0.03193203214388566) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13734937439697048) | > loader_time: 0.0047 (0.004668555735202319)  --> TIME: 2025-02-26 07:45:17 -- STEP: 4322/36938 -- GLOBAL_STEP: 1149400 | > loss_text_ce: 0.03320394083857536 (0.03192253802825006) | > loss_mel_ce: 2.2725608348846436 (nan) | > loss: 0.009149860590696335 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13737713308700628) | > loader_time: 0.0046 (0.004668116569519045)  --> TIME: 2025-02-26 07:45:26 -- STEP: 4372/36938 -- GLOBAL_STEP: 1149450 | > loss_text_ce: 0.03348810225725174 (0.03192756839406717) | > loss_mel_ce: 2.7591538429260254 (nan) | > loss: 0.01108191255480051 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13737298754083918) | > loader_time: 0.0048 (0.004667493747334251)  --> TIME: 2025-02-26 07:45:35 -- STEP: 4422/36938 -- GLOBAL_STEP: 1149500 | > loss_text_ce: 0.028499331325292587 (0.03192882209701191) | > loss_mel_ce: 2.8310346603393555 (nan) | > loss: 0.011347358115017414 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13731187104855316) | > loader_time: 0.0046 (0.004667309062721398)  --> TIME: 2025-02-26 07:45:45 -- STEP: 4472/36938 -- GLOBAL_STEP: 1149550 | > loss_text_ce: 0.026085760444402695 (0.031920217374116586) | > loss_mel_ce: 3.7150657176971436 (nan) | > loss: 0.014845840632915497 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.13736741018209964) | > loader_time: 0.0047 (0.004667222606474689)  --> TIME: 2025-02-26 07:45:55 -- STEP: 4522/36938 -- GLOBAL_STEP: 1149600 | > loss_text_ce: 0.030621113255620003 (0.03192489142605886) | > loss_mel_ce: 3.292398452758789 (nan) | > loss: 0.01318658608943224 (nan) | > current_lr: 5e-06 | > step_time: 0.1788 (0.13743460051837272) | > loader_time: 0.0048 (0.004667355971017919)  --> TIME: 2025-02-26 07:46:04 -- STEP: 4572/36938 -- GLOBAL_STEP: 1149650 | > loss_text_ce: 0.03888645023107529 (0.031920918174273935) | > loss_mel_ce: 2.5312983989715576 (nan) | > loss: 0.010199147276580334 (nan) | > current_lr: 5e-06 | > step_time: 0.1503 (0.1374439039672555) | > loader_time: 0.0047 (0.004666948881674949)  --> TIME: 2025-02-26 07:46:13 -- STEP: 4622/36938 -- GLOBAL_STEP: 1149700 | > loss_text_ce: 0.026968665421009064 (0.03191846478585378) | > loss_mel_ce: 2.8624002933502197 (nan) | > loss: 0.011465750634670258 (nan) | > current_lr: 5e-06 | > step_time: 0.1435 (0.1374374280394746) | > loader_time: 0.0047 (0.004666375474257171)  --> TIME: 2025-02-26 07:46:23 -- STEP: 4672/36938 -- GLOBAL_STEP: 1149750 | > loss_text_ce: 0.031622517853975296 (0.031930236014288876) | > loss_mel_ce: 3.0045511722564697 (nan) | > loss: 0.012048308737576008 (nan) | > current_lr: 5e-06 | > step_time: 0.1824 (0.1374031027702433) | > loader_time: 0.0049 (0.00466620034142717)  --> TIME: 2025-02-26 07:46:32 -- STEP: 4722/36938 -- GLOBAL_STEP: 1149800 | > loss_text_ce: 0.029729733243584633 (0.03192797312509063) | > loss_mel_ce: 1.3733673095703125 (nan) | > loss: 0.005567845888435841 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1374392760400797) | > loader_time: 0.0045 (0.004666320048667079)  --> TIME: 2025-02-26 07:46:42 -- STEP: 4772/36938 -- GLOBAL_STEP: 1149850 | > loss_text_ce: 0.03584233298897743 (0.031925223814596934) | > loss_mel_ce: 3.26763653755188 (nan) | > loss: 0.013109044171869755 (nan) | > current_lr: 5e-06 | > step_time: 0.1799 (0.13751380791308346) | > loader_time: 0.0047 (0.004666375894058889)  --> TIME: 2025-02-26 07:46:52 -- STEP: 4822/36938 -- GLOBAL_STEP: 1149900 | > loss_text_ce: 0.032619722187519073 (0.03191504083660361) | > loss_mel_ce: 3.1003475189208984 (nan) | > loss: 0.012432410381734371 (nan) | > current_lr: 5e-06 | > step_time: 0.1567 (0.13758574334084822) | > loader_time: 0.0048 (0.0046660865505223316)  --> TIME: 2025-02-26 07:47:01 -- STEP: 4872/36938 -- GLOBAL_STEP: 1149950 | > loss_text_ce: 0.03078390285372734 (0.031923032673504145) | > loss_mel_ce: 3.067190647125244 (nan) | > loss: 0.012293550185859203 (nan) | > current_lr: 5e-06 | > step_time: 0.1825 (0.1375804548388823) | > loader_time: 0.0047 (0.0046650830943791985)  --> TIME: 2025-02-26 07:47:10 -- STEP: 4922/36938 -- GLOBAL_STEP: 1150000 | > loss_text_ce: 0.02583826519548893 (0.031910880917228504) | > loss_mel_ce: 2.553446054458618 (nan) | > loss: 0.010235256515443325 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.13758641677578953) | > loader_time: 0.0047 (0.004665898724524584) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1150000.pth  --> TIME: 2025-02-26 07:47:25 -- STEP: 4972/36938 -- GLOBAL_STEP: 1150050 | > loss_text_ce: 0.027027329429984093 (0.03191908509062195) | > loss_mel_ce: 2.9448745250701904 (nan) | > loss: 0.01179326232522726 (nan) | > current_lr: 5e-06 | > step_time: 0.1503 (0.13751364358470758) | > loader_time: 0.0047 (0.004665475455581038)  --> TIME: 2025-02-26 07:47:34 -- STEP: 5022/36938 -- GLOBAL_STEP: 1150100 | > loss_text_ce: 0.02753824181854725 (0.031921112175536996) | > loss_mel_ce: 2.79891037940979 (nan) | > loss: 0.011216066777706146 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13742569560123472) | > loader_time: 0.0046 (0.004665155089836773)  --> TIME: 2025-02-26 07:47:43 -- STEP: 5072/36938 -- GLOBAL_STEP: 1150150 | > loss_text_ce: 0.0283748097717762 (0.03192594347702361) | > loss_mel_ce: 2.5300214290618896 (nan) | > loss: 0.010152366943657398 (nan) | > current_lr: 5e-06 | > step_time: 0.1936 (0.13740268142817486) | > loader_time: 0.0049 (0.004665186117500162)  --> TIME: 2025-02-26 07:47:52 -- STEP: 5122/36938 -- GLOBAL_STEP: 1150200 | > loss_text_ce: 0.02564946375787258 (0.03191063248169469) | > loss_mel_ce: 2.2250049114227295 (nan) | > loss: 0.008931169286370277 (nan) | > current_lr: 5e-06 | > step_time: 0.1563 (0.1373598693358341) | > loader_time: 0.0047 (0.004664735419821896)  --> TIME: 2025-02-26 07:48:01 -- STEP: 5172/36938 -- GLOBAL_STEP: 1150250 | > loss_text_ce: 0.03696874901652336 (0.03191135281949749) | > loss_mel_ce: 2.7631454467773438 (nan) | > loss: 0.011111564934253693 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.1373861263357013) | > loader_time: 0.0048 (0.004664581410200763)  --> TIME: 2025-02-26 07:48:11 -- STEP: 5222/36938 -- GLOBAL_STEP: 1150300 | > loss_text_ce: 0.02975831925868988 (0.03191269863231395) | > loss_mel_ce: 3.0120608806610107 (nan) | > loss: 0.012070711702108383 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13735510900257059) | > loader_time: 0.0047 (0.004664229552255464)  --> TIME: 2025-02-26 07:48:20 -- STEP: 5272/36938 -- GLOBAL_STEP: 1150350 | > loss_text_ce: 0.028638619929552078 (0.031915475270952696) | > loss_mel_ce: 0.1405029296875 (nan) | > loss: 0.0006711966707371175 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13737078429353766) | > loader_time: 0.0047 (0.004663989513162775)  --> TIME: 2025-02-26 07:48:29 -- STEP: 5322/36938 -- GLOBAL_STEP: 1150400 | > loss_text_ce: 0.03013896942138672 (0.03192118352873364) | > loss_mel_ce: 3.086104154586792 (nan) | > loss: 0.012366045266389847 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.1373408110416815) | > loader_time: 0.0045 (0.004664456427702101)  --> TIME: 2025-02-26 07:48:39 -- STEP: 5372/36938 -- GLOBAL_STEP: 1150450 | > loss_text_ce: 0.024303937330842018 (0.031908830663165855) | > loss_mel_ce: 2.654690980911255 (nan) | > loss: 0.01063093263655901 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13735930252146175) | > loader_time: 0.0045 (0.004663926669225172)  --> TIME: 2025-02-26 07:48:48 -- STEP: 5422/36938 -- GLOBAL_STEP: 1150500 | > loss_text_ce: 0.028878603130578995 (0.03191568329277601) | > loss_mel_ce: 2.2200927734375 (nan) | > loss: 0.00892449077218771 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13740665333950808) | > loader_time: 0.0046 (0.004664172197522734)  --> TIME: 2025-02-26 07:48:58 -- STEP: 5472/36938 -- GLOBAL_STEP: 1150550 | > loss_text_ce: 0.03519993647933006 (0.03191225607511845) | > loss_mel_ce: 2.5793519020080566 (nan) | > loss: 0.010375206358730793 (nan) | > current_lr: 5e-06 | > step_time: 0.1516 (0.13739215024904874) | > loader_time: 0.0047 (0.004663765735444964)  --> TIME: 2025-02-26 07:49:07 -- STEP: 5522/36938 -- GLOBAL_STEP: 1150600 | > loss_text_ce: 0.025760170072317123 (0.03191146418575056) | > loss_mel_ce: 2.0342044830322266 (nan) | > loss: 0.008174463175237179 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13735052534135844) | > loader_time: 0.0046 (0.004663328509622618)  --> TIME: 2025-02-26 07:49:16 -- STEP: 5572/36938 -- GLOBAL_STEP: 1150650 | > loss_text_ce: 0.032358214259147644 (0.03191386133082069) | > loss_mel_ce: 2.1572232246398926 (nan) | > loss: 0.008688815869390965 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13735307709227665) | > loader_time: 0.0047 (0.00466345581149165)  --> TIME: 2025-02-26 07:49:26 -- STEP: 5622/36938 -- GLOBAL_STEP: 1150700 | > loss_text_ce: 0.03274154290556908 (0.03192680612622602) | > loss_mel_ce: 3.01491641998291 (nan) | > loss: 0.012093881145119667 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13736471804127226) | > loader_time: 0.0047 (0.00466388805359549)  --> TIME: 2025-02-26 07:49:35 -- STEP: 5672/36938 -- GLOBAL_STEP: 1150750 | > loss_text_ce: 0.04173250496387482 (0.03191413698134739) | > loss_mel_ce: 3.3770430088043213 (nan) | > loss: 0.013566570356488228 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.13737249365982765) | > loader_time: 0.0046 (0.0046637915757882726)  --> TIME: 2025-02-26 07:49:44 -- STEP: 5722/36938 -- GLOBAL_STEP: 1150800 | > loss_text_ce: 0.028956955298781395 (0.03191070367570112) | > loss_mel_ce: 1.9433557987213135 (nan) | > loss: 0.007826638408005238 (nan) | > current_lr: 5e-06 | > step_time: 0.1267 (0.13736814741736625) | > loader_time: 0.0049 (0.004663521699328722)  --> TIME: 2025-02-26 07:49:53 -- STEP: 5772/36938 -- GLOBAL_STEP: 1150850 | > loss_text_ce: 0.033212438225746155 (0.03189488428248572) | > loss_mel_ce: 3.0814208984375 (nan) | > loss: 0.012359656393527985 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.13732489189668717) | > loader_time: 0.0047 (0.004663150961499864)  --> TIME: 2025-02-26 07:50:03 -- STEP: 5822/36938 -- GLOBAL_STEP: 1150900 | > loss_text_ce: 0.027788283303380013 (0.03189740187307111) | > loss_mel_ce: 3.0346922874450684 (nan) | > loss: 0.012152701616287231 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.1373506947425336) | > loader_time: 0.0047 (0.004663062890097678)  --> TIME: 2025-02-26 07:50:12 -- STEP: 5872/36938 -- GLOBAL_STEP: 1150950 | > loss_text_ce: 0.03251223266124725 (0.03190284257590383) | > loss_mel_ce: 3.21062970161438 (nan) | > loss: 0.012869611382484436 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13732819823543105) | > loader_time: 0.0045 (0.004663039942853132)  --> TIME: 2025-02-26 07:50:21 -- STEP: 5922/36938 -- GLOBAL_STEP: 1151000 | > loss_text_ce: 0.031091131269931793 (0.03191779876896564) | > loss_mel_ce: 2.8262593746185303 (nan) | > loss: 0.011338693089783192 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.1372588528688191) | > loader_time: 0.0045 (0.004662985134994368)  --> TIME: 2025-02-26 07:50:31 -- STEP: 5972/36938 -- GLOBAL_STEP: 1151050 | > loss_text_ce: 0.040676090866327286 (0.03191744711088519) | > loss_mel_ce: 3.0100643634796143 (nan) | > loss: 0.012106114067137241 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.1372912996819731) | > loader_time: 0.0046 (0.004663065105816378)  --> TIME: 2025-02-26 07:50:40 -- STEP: 6022/36938 -- GLOBAL_STEP: 1151100 | > loss_text_ce: 0.03785736858844757 (0.031917675460924944) | > loss_mel_ce: 3.180142402648926 (nan) | > loss: 0.012769840657711029 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.1372659491447465) | > loader_time: 0.0046 (0.004662636742960842)  --> TIME: 2025-02-26 07:50:49 -- STEP: 6072/36938 -- GLOBAL_STEP: 1151150 | > loss_text_ce: 0.028192760422825813 (0.03191848372600679) | > loss_mel_ce: 2.8080086708068848 (nan) | > loss: 0.011254767887294292 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.137279594054492) | > loader_time: 0.0046 (0.0046625785598327614)  --> TIME: 2025-02-26 07:50:59 -- STEP: 6122/36938 -- GLOBAL_STEP: 1151200 | > loss_text_ce: 0.0377863384783268 (0.03192094374647059) | > loss_mel_ce: 2.9344398975372314 (nan) | > loss: 0.011794548481702805 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.1372921736323725) | > loader_time: 0.0047 (0.004662530401180626)  --> TIME: 2025-02-26 07:51:08 -- STEP: 6172/36938 -- GLOBAL_STEP: 1151250 | > loss_text_ce: 0.02176274172961712 (0.031913696885729585) | > loss_mel_ce: 2.7056829929351807 (nan) | > loss: 0.010823198594152927 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13726485548124437) | > loader_time: 0.0049 (0.004662733841313216)  --> TIME: 2025-02-26 07:51:17 -- STEP: 6222/36938 -- GLOBAL_STEP: 1151300 | > loss_text_ce: 0.036037739366292953 (0.0319131151478163) | > loss_mel_ce: 2.9447975158691406 (nan) | > loss: 0.011828712187707424 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.1372178500199076) | > loader_time: 0.0046 (0.004662578989554349)  --> TIME: 2025-02-26 07:51:26 -- STEP: 6272/36938 -- GLOBAL_STEP: 1151350 | > loss_text_ce: 0.030122481286525726 (0.03190773815850321) | > loss_mel_ce: 2.8829848766326904 (nan) | > loss: 0.011559950187802315 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.1372217125430394) | > loader_time: 0.0047 (0.00466283464006014)  --> TIME: 2025-02-26 07:51:36 -- STEP: 6322/36938 -- GLOBAL_STEP: 1151400 | > loss_text_ce: 0.023526784032583237 (0.031914436395672936) | > loss_mel_ce: 2.1705241203308105 (nan) | > loss: 0.008706551045179367 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13723361692033292) | > loader_time: 0.0045 (0.004662726582397717)  --> TIME: 2025-02-26 07:51:45 -- STEP: 6372/36938 -- GLOBAL_STEP: 1151450 | > loss_text_ce: 0.03339700400829315 (0.031913028680208025) | > loss_mel_ce: 2.3757100105285645 (nan) | > loss: 0.009559948928654194 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.1372139629745955) | > loader_time: 0.0047 (0.00466289504548698)  --> TIME: 2025-02-26 07:51:55 -- STEP: 6422/36938 -- GLOBAL_STEP: 1151500 | > loss_text_ce: 0.03148209676146507 (0.03191798972139949) | > loss_mel_ce: 1.1830637454986572 (nan) | > loss: 0.0048196264542639256 (nan) | > current_lr: 5e-06 | > step_time: 0.1521 (0.1372293715386397) | > loader_time: 0.0047 (0.0046627347768559485)  --> TIME: 2025-02-26 07:52:04 -- STEP: 6472/36938 -- GLOBAL_STEP: 1151550 | > loss_text_ce: 0.027171777561306953 (0.03191432640418697) | > loss_mel_ce: 2.460630178451538 (nan) | > loss: 0.00987223070114851 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13721234748331004) | > loader_time: 0.0045 (0.004662336945091679)  --> TIME: 2025-02-26 07:52:13 -- STEP: 6522/36938 -- GLOBAL_STEP: 1151600 | > loss_text_ce: 0.026269862428307533 (0.031910108407655076) | > loss_mel_ce: 1.1394740343093872 (nan) | > loss: 0.0046259681694209576 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1371747012593117) | > loader_time: 0.0046 (0.004661807872078515)  --> TIME: 2025-02-26 07:52:23 -- STEP: 6572/36938 -- GLOBAL_STEP: 1151650 | > loss_text_ce: 0.02642468363046646 (0.031901781217518944) | > loss_mel_ce: 3.236126661300659 (nan) | > loss: 0.01294663269072771 (nan) | > current_lr: 5e-06 | > step_time: 0.201 (0.1371943128957875) | > loader_time: 0.0049 (0.004661706911668628)  --> TIME: 2025-02-26 07:52:32 -- STEP: 6622/36938 -- GLOBAL_STEP: 1151700 | > loss_text_ce: 0.02266143076121807 (0.03190371497429876) | > loss_mel_ce: 2.668170690536499 (nan) | > loss: 0.01067790575325489 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.1372209495016176) | > loader_time: 0.0048 (0.004661510517070802)  --> TIME: 2025-02-26 07:52:42 -- STEP: 6672/36938 -- GLOBAL_STEP: 1151750 | > loss_text_ce: 0.02813982591032982 (0.03190260560001093) | > loss_mel_ce: 2.666247606277466 (nan) | > loss: 0.01069201435893774 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13727391659498722) | > loader_time: 0.0045 (0.0046614458163579155)  --> TIME: 2025-02-26 07:52:51 -- STEP: 6722/36938 -- GLOBAL_STEP: 1151800 | > loss_text_ce: 0.03276780992746353 (0.03190106490936739) | > loss_mel_ce: 0.5773681402206421 (nan) | > loss: 0.0024211746640503407 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.1372731679325046) | > loader_time: 0.0045 (0.004661032359751148)  --> TIME: 2025-02-26 07:53:01 -- STEP: 6772/36938 -- GLOBAL_STEP: 1151850 | > loss_text_ce: 0.03554399684071541 (0.03190276470135447) | > loss_mel_ce: 0.91485595703125 (nan) | > loss: 0.003771428484469652 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.1372946309606767) | > loader_time: 0.0045 (0.0046611001557734745)  --> TIME: 2025-02-26 07:53:10 -- STEP: 6822/36938 -- GLOBAL_STEP: 1151900 | > loss_text_ce: 0.04177209734916687 (0.03190075034044678) | > loss_mel_ce: 2.818944215774536 (nan) | > loss: 0.011352049186825752 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13730284927248435) | > loader_time: 0.0047 (0.004661430958882334)  --> TIME: 2025-02-26 07:53:20 -- STEP: 6872/36938 -- GLOBAL_STEP: 1151950 | > loss_text_ce: 0.031074775382876396 (0.03190531336124547) | > loss_mel_ce: 2.5339431762695312 (nan) | > loss: 0.0101786432787776 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13733507083930507) | > loader_time: 0.0046 (0.004661861273684507)  --> TIME: 2025-02-26 07:53:29 -- STEP: 6922/36938 -- GLOBAL_STEP: 1152000 | > loss_text_ce: 0.039431821554899216 (0.03191166110035416) | > loss_mel_ce: 2.4352853298187256 (nan) | > loss: 0.009820306673645973 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13736527636367307) | > loader_time: 0.0046 (0.004661987779182483)  --> TIME: 2025-02-26 07:53:39 -- STEP: 6972/36938 -- GLOBAL_STEP: 1152050 | > loss_text_ce: 0.028314504772424698 (0.03191675351076679) | > loss_mel_ce: 2.273345470428467 (nan) | > loss: 0.00913357175886631 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13736412826790806) | > loader_time: 0.0048 (0.004662471363343377)  --> TIME: 2025-02-26 07:53:48 -- STEP: 7022/36938 -- GLOBAL_STEP: 1152100 | > loss_text_ce: 0.04722428321838379 (0.031922209071808885) | > loss_mel_ce: 2.859968662261963 (nan) | > loss: 0.011536480858922005 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13735350846768854) | > loader_time: 0.0045 (0.004662518588201631)  --> TIME: 2025-02-26 07:53:58 -- STEP: 7072/36938 -- GLOBAL_STEP: 1152150 | > loss_text_ce: 0.04448657110333443 (0.03191505676389336) | > loss_mel_ce: 0.03354644775390625 (nan) | > loss: 0.00030965483165346086 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13738116399465033) | > loader_time: 0.0046 (0.0046623526520319005)  --> TIME: 2025-02-26 07:54:07 -- STEP: 7122/36938 -- GLOBAL_STEP: 1152200 | > loss_text_ce: 0.030679134652018547 (0.031922692599335376) | > loss_mel_ce: 2.5469000339508057 (nan) | > loss: 0.010228489525616169 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13736872507661424) | > loader_time: 0.0045 (0.004662237385624202)  --> TIME: 2025-02-26 07:54:17 -- STEP: 7172/36938 -- GLOBAL_STEP: 1152250 | > loss_text_ce: 0.03001839853823185 (0.03191843487782552) | > loss_mel_ce: 2.88035249710083 (nan) | > loss: 0.011549090966582298 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.13740416614954734) | > loader_time: 0.0047 (0.0046623212893582185)  --> TIME: 2025-02-26 07:54:26 -- STEP: 7222/36938 -- GLOBAL_STEP: 1152300 | > loss_text_ce: 0.025083519518375397 (0.031920680114305766) | > loss_mel_ce: 2.4728281497955322 (nan) | > loss: 0.009912348352372646 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1374149379291808) | > loader_time: 0.0046 (0.004662448631631)  --> TIME: 2025-02-26 07:54:35 -- STEP: 7272/36938 -- GLOBAL_STEP: 1152350 | > loss_text_ce: 0.024573219940066338 (0.03192320227076397) | > loss_mel_ce: 2.4384407997131348 (nan) | > loss: 0.009773866273462772 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.1374092223358358) | > loader_time: 0.0047 (0.0046627568070775624)  --> TIME: 2025-02-26 07:54:45 -- STEP: 7322/36938 -- GLOBAL_STEP: 1152400 | > loss_text_ce: 0.03089442104101181 (0.031918925206856026) | > loss_mel_ce: 2.891385793685913 (nan) | > loss: 0.01159635093063116 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13739677994214083) | > loader_time: 0.0046 (0.004663087116242487)  --> TIME: 2025-02-26 07:54:54 -- STEP: 7372/36938 -- GLOBAL_STEP: 1152450 | > loss_text_ce: 0.029805244877934456 (0.03192190606032353) | > loss_mel_ce: 1.908193588256836 (nan) | > loss: 0.007690472062677145 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13737283607442508) | > loader_time: 0.0045 (0.004662769453977015)  --> TIME: 2025-02-26 07:55:03 -- STEP: 7422/36938 -- GLOBAL_STEP: 1152500 | > loss_text_ce: 0.02647436037659645 (0.03191869853368595) | > loss_mel_ce: 3.240969657897949 (nan) | > loss: 0.012966048903763294 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.13736696035770055) | > loader_time: 0.0048 (0.004662723850927648)  --> TIME: 2025-02-26 07:55:13 -- STEP: 7472/36938 -- GLOBAL_STEP: 1152550 | > loss_text_ce: 0.03210685774683952 (0.03191976908634765) | > loss_mel_ce: 2.506486654281616 (nan) | > loss: 0.010073784738779068 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1373635902522204) | > loader_time: 0.0046 (0.004662636739283598)  --> TIME: 2025-02-26 07:55:22 -- STEP: 7522/36938 -- GLOBAL_STEP: 1152600 | > loss_text_ce: 0.034156229346990585 (0.031916243331350944) | > loss_mel_ce: 1.796904444694519 (nan) | > loss: 0.007266114000231028 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13738639219680096) | > loader_time: 0.0049 (0.004662681595818272)  --> TIME: 2025-02-26 07:55:32 -- STEP: 7572/36938 -- GLOBAL_STEP: 1152650 | > loss_text_ce: 0.027871884405612946 (0.031907820943383965) | > loss_mel_ce: 2.6195807456970215 (nan) | > loss: 0.010505764745175838 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.1373826307527485) | > loader_time: 0.0046 (0.004662695443636506)  --> TIME: 2025-02-26 07:55:41 -- STEP: 7622/36938 -- GLOBAL_STEP: 1152700 | > loss_text_ce: 0.025624509900808334 (0.03191075060671542) | > loss_mel_ce: 2.845534086227417 (nan) | > loss: 0.011393487453460693 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.1373872936819829) | > loader_time: 0.0046 (0.004662439317035089)  --> TIME: 2025-02-26 07:55:50 -- STEP: 7672/36938 -- GLOBAL_STEP: 1152750 | > loss_text_ce: 0.032945990562438965 (0.031919636028844116) | > loss_mel_ce: 3.631124258041382 (nan) | > loss: 0.014539961703121662 (nan) | > current_lr: 5e-06 | > step_time: 0.1853 (0.13734991983651335) | > loader_time: 0.0048 (0.004661812523731464)  --> TIME: 2025-02-26 07:55:59 -- STEP: 7722/36938 -- GLOBAL_STEP: 1152800 | > loss_text_ce: 0.03239614516496658 (0.03192131310989387) | > loss_mel_ce: 3.3180437088012695 (nan) | > loss: 0.013295397162437439 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13733510741357538) | > loader_time: 0.0046 (0.004661450636754276)  --> TIME: 2025-02-26 07:56:09 -- STEP: 7772/36938 -- GLOBAL_STEP: 1152850 | > loss_text_ce: 0.03128174692392349 (0.03192305250696819) | > loss_mel_ce: 2.381153106689453 (nan) | > loss: 0.009573154151439667 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.1373756092906424) | > loader_time: 0.0045 (0.004661213811745783)  --> TIME: 2025-02-26 07:56:19 -- STEP: 7822/36938 -- GLOBAL_STEP: 1152900 | > loss_text_ce: 0.03168341517448425 (0.03192654491407696) | > loss_mel_ce: 2.531470537185669 (nan) | > loss: 0.010171246714890003 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13740262473766435) | > loader_time: 0.0046 (0.0046614587108852875)  --> TIME: 2025-02-26 07:56:28 -- STEP: 7872/36938 -- GLOBAL_STEP: 1152950 | > loss_text_ce: 0.038792066276073456 (0.031926368110376954) | > loss_mel_ce: 2.363922119140625 (nan) | > loss: 0.009534580633044243 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13742788650882887) | > loader_time: 0.0046 (0.004661280570960617)  --> TIME: 2025-02-26 07:56:38 -- STEP: 7922/36938 -- GLOBAL_STEP: 1153000 | > loss_text_ce: 0.035745859146118164 (0.03192533697000465) | > loss_mel_ce: 3.0772054195404053 (nan) | > loss: 0.012352981604635715 (nan) | > current_lr: 5e-06 | > step_time: 0.1954 (0.13742512366712073) | > loader_time: 0.0048 (0.004661078616784868)  --> TIME: 2025-02-26 07:56:47 -- STEP: 7972/36938 -- GLOBAL_STEP: 1153050 | > loss_text_ce: 0.03312160074710846 (0.031921926926965445) | > loss_mel_ce: 1.8089529275894165 (nan) | > loss: 0.007309820037335157 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13739376614216964) | > loader_time: 0.0048 (0.004660884668883272)  --> TIME: 2025-02-26 07:56:56 -- STEP: 8022/36938 -- GLOBAL_STEP: 1153100 | > loss_text_ce: 0.02755029872059822 (0.0319203424472497) | > loss_mel_ce: 2.898893117904663 (nan) | > loss: 0.01161287073045969 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.1373669406892768) | > loader_time: 0.0045 (0.004660895090096007)  --> TIME: 2025-02-26 07:57:05 -- STEP: 8072/36938 -- GLOBAL_STEP: 1153150 | > loss_text_ce: 0.025518307462334633 (0.031912078528514604) | > loss_mel_ce: 2.5882415771484375 (nan) | > loss: 0.01037206407636404 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13737566392532746) | > loader_time: 0.0046 (0.004660664393951437)  --> TIME: 2025-02-26 07:57:15 -- STEP: 8122/36938 -- GLOBAL_STEP: 1153200 | > loss_text_ce: 0.03752202168107033 (0.03191114693001281) | > loss_mel_ce: 2.885550022125244 (nan) | > loss: 0.011599493212997913 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13738209512602984) | > loader_time: 0.0044 (0.0046606515317502055)  --> TIME: 2025-02-26 07:57:24 -- STEP: 8172/36938 -- GLOBAL_STEP: 1153250 | > loss_text_ce: 0.03312603756785393 (0.03191142883201919) | > loss_mel_ce: 2.8566842079162598 (nan) | > loss: 0.011467501521110535 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13738732485577423) | > loader_time: 0.0046 (0.004660558916456948)  --> TIME: 2025-02-26 07:57:33 -- STEP: 8222/36938 -- GLOBAL_STEP: 1153300 | > loss_text_ce: 0.025749873369932175 (0.03191557939646204) | > loss_mel_ce: 2.898566961288452 (nan) | > loss: 0.011604432947933674 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13740292876574595) | > loader_time: 0.0046 (0.004660276043185978)  --> TIME: 2025-02-26 07:57:43 -- STEP: 8272/36938 -- GLOBAL_STEP: 1153350 | > loss_text_ce: 0.031887028366327286 (0.03191803115383119) | > loss_mel_ce: 3.2257027626037598 (nan) | > loss: 0.01292694453150034 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.13738883922247605) | > loader_time: 0.0046 (0.00465989599951922)  --> TIME: 2025-02-26 07:57:52 -- STEP: 8322/36938 -- GLOBAL_STEP: 1153400 | > loss_text_ce: 0.03180685266852379 (0.03191474902742842) | > loss_mel_ce: 3.0085010528564453 (nan) | > loss: 0.012064714916050434 (nan) | > current_lr: 5e-06 | > step_time: 0.1436 (0.13740801066339228) | > loader_time: 0.0046 (0.004659782519451663)  --> TIME: 2025-02-26 07:58:02 -- STEP: 8372/36938 -- GLOBAL_STEP: 1153450 | > loss_text_ce: 0.03068169206380844 (0.0319103257249146) | > loss_mel_ce: 2.8184139728546143 (nan) | > loss: 0.011305935680866241 (nan) | > current_lr: 5e-06 | > step_time: 0.1456 (0.13739774720382844) | > loader_time: 0.0046 (0.004659457008518548)  --> TIME: 2025-02-26 07:58:11 -- STEP: 8422/36938 -- GLOBAL_STEP: 1153500 | > loss_text_ce: 0.02548273652791977 (0.03190884190783834) | > loss_mel_ce: 3.062635660171509 (nan) | > loss: 0.012254438363015652 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13740106267052596) | > loader_time: 0.0047 (0.0046596898230881175)  --> TIME: 2025-02-26 07:58:20 -- STEP: 8472/36938 -- GLOBAL_STEP: 1153550 | > loss_text_ce: 0.03276153653860092 (0.03191188657462843) | > loss_mel_ce: 2.43965220451355 (nan) | > loss: 0.009811166673898697 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13740976164653873) | > loader_time: 0.0044 (0.004659263422401351)  --> TIME: 2025-02-26 07:58:29 -- STEP: 8522/36938 -- GLOBAL_STEP: 1153600 | > loss_text_ce: 0.023333637043833733 (0.03190347593018912) | > loss_mel_ce: 2.3902571201324463 (nan) | > loss: 0.009577740915119648 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13737845423635045) | > loader_time: 0.0046 (0.004658940671559526)  --> TIME: 2025-02-26 07:58:39 -- STEP: 8572/36938 -- GLOBAL_STEP: 1153650 | > loss_text_ce: 0.03220905363559723 (0.03190440146862679) | > loss_mel_ce: 3.3272945880889893 (nan) | > loss: 0.013331364840269089 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13736316273827634) | > loader_time: 0.0046 (0.004658422345487622)  --> TIME: 2025-02-26 07:58:48 -- STEP: 8622/36938 -- GLOBAL_STEP: 1153700 | > loss_text_ce: 0.035629890859127045 (0.03190438791040558) | > loss_mel_ce: 2.974858045578003 (nan) | > loss: 0.011946382001042366 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13737930869370346) | > loader_time: 0.0045 (0.004658082664718029)  --> TIME: 2025-02-26 07:58:57 -- STEP: 8672/36938 -- GLOBAL_STEP: 1153750 | > loss_text_ce: 0.03742591664195061 (0.03190868519799877) | > loss_mel_ce: 2.3534634113311768 (nan) | > loss: 0.009487656876444817 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13737646960464325) | > loader_time: 0.0045 (0.00465774475648393)  --> TIME: 2025-02-26 07:59:06 -- STEP: 8722/36938 -- GLOBAL_STEP: 1153800 | > loss_text_ce: 0.033014390617609024 (0.03190524463333163) | > loss_mel_ce: 3.2122371196746826 (nan) | > loss: 0.01287798210978508 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.13735699013979202) | > loader_time: 0.0046 (0.004657377482717983)  --> TIME: 2025-02-26 07:59:16 -- STEP: 8772/36938 -- GLOBAL_STEP: 1153850 | > loss_text_ce: 0.032471757382154465 (0.03189849360632371) | > loss_mel_ce: 2.8623526096343994 (nan) | > loss: 0.01148739829659462 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.1373413457644348) | > loader_time: 0.0046 (0.004657145129237267)  --> TIME: 2025-02-26 07:59:25 -- STEP: 8822/36938 -- GLOBAL_STEP: 1153900 | > loss_text_ce: 0.03291863948106766 (0.03189936509059049) | > loss_mel_ce: 2.861816167831421 (nan) | > loss: 0.011487043462693691 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1373363085491427) | > loader_time: 0.0045 (0.004656700286917108)  --> TIME: 2025-02-26 07:59:34 -- STEP: 8872/36938 -- GLOBAL_STEP: 1153950 | > loss_text_ce: 0.02954595908522606 (0.031891601526105394) | > loss_mel_ce: 1.1933212280273438 (nan) | > loss: 0.004852647427469492 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13732424622940717) | > loader_time: 0.0047 (0.004656227135894733)  --> TIME: 2025-02-26 07:59:44 -- STEP: 8922/36938 -- GLOBAL_STEP: 1154000 | > loss_text_ce: 0.03938622400164604 (0.03189760074416099) | > loss_mel_ce: 2.4814395904541016 (nan) | > loss: 0.010003278031945229 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.1373560711611173) | > loader_time: 0.0046 (0.004656012377325269)  --> TIME: 2025-02-26 07:59:53 -- STEP: 8972/36938 -- GLOBAL_STEP: 1154050 | > loss_text_ce: 0.022078987210989 (0.03189399602606635) | > loss_mel_ce: 2.085216999053955 (nan) | > loss: 0.008362285792827606 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13735816136460927) | > loader_time: 0.0046 (0.004655830890963327)  --> TIME: 2025-02-26 08:00:02 -- STEP: 9022/36938 -- GLOBAL_STEP: 1154100 | > loss_text_ce: 0.025967909023165703 (0.03189216002762465) | > loss_mel_ce: 3.3541479110717773 (nan) | > loss: 0.013413158245384693 (nan) | > current_lr: 5e-06 | > step_time: 0.1619 (0.13734997632206997) | > loader_time: 0.0046 (0.004655439688239208)  --> TIME: 2025-02-26 08:00:12 -- STEP: 9072/36938 -- GLOBAL_STEP: 1154150 | > loss_text_ce: 0.017257267609238625 (0.031883230428453695) | > loss_mel_ce: 2.5516602993011475 (nan) | > loss: 0.01019411813467741 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13736904080057052) | > loader_time: 0.0045 (0.004655313791421333)  --> TIME: 2025-02-26 08:00:21 -- STEP: 9122/36938 -- GLOBAL_STEP: 1154200 | > loss_text_ce: 0.021826572716236115 (0.03188398974571715) | > loss_mel_ce: 2.100571632385254 (nan) | > loss: 0.008422215469181538 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13737533605299976) | > loader_time: 0.0044 (0.004654807731839163)  --> TIME: 2025-02-26 08:00:31 -- STEP: 9172/36938 -- GLOBAL_STEP: 1154250 | > loss_text_ce: 0.03442828357219696 (0.03188610221628903) | > loss_mel_ce: 2.4364326000213623 (nan) | > loss: 0.009805004112422466 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.1373776613340282) | > loader_time: 0.0045 (0.004654331052350081)  --> TIME: 2025-02-26 08:00:40 -- STEP: 9222/36938 -- GLOBAL_STEP: 1154300 | > loss_text_ce: 0.027005469426512718 (0.031899658667084815) | > loss_mel_ce: 3.000267267227173 (nan) | > loss: 0.012012987397611141 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.13738061181636976) | > loader_time: 0.0047 (0.004654177071292736)  --> TIME: 2025-02-26 08:00:49 -- STEP: 9272/36938 -- GLOBAL_STEP: 1154350 | > loss_text_ce: 0.03327929973602295 (0.03189944865363243) | > loss_mel_ce: 3.0620436668395996 (nan) | > loss: 0.012283029034733772 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13738175547770934) | > loader_time: 0.0046 (0.004654062010277113)  --> TIME: 2025-02-26 08:00:59 -- STEP: 9322/36938 -- GLOBAL_STEP: 1154400 | > loss_text_ce: 0.02597467042505741 (0.031894156068317275) | > loss_mel_ce: 3.0885725021362305 (nan) | > loss: 0.012359315529465675 (nan) | > current_lr: 5e-06 | > step_time: 0.1679 (0.137420746395706) | > loader_time: 0.0047 (0.004653849000200321)  --> TIME: 2025-02-26 08:01:09 -- STEP: 9372/36938 -- GLOBAL_STEP: 1154450 | > loss_text_ce: 0.025706490501761436 (0.03188893468028969) | > loss_mel_ce: 2.7202084064483643 (nan) | > loss: 0.010896488092839718 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.13744355118076262) | > loader_time: 0.0045 (0.00465349404668379)  --> TIME: 2025-02-26 08:01:18 -- STEP: 9422/36938 -- GLOBAL_STEP: 1154500 | > loss_text_ce: 0.03193320333957672 (0.03188193960178964) | > loss_mel_ce: 2.991692543029785 (nan) | > loss: 0.011998515576124191 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13744946309042902) | > loader_time: 0.0046 (0.004653244230099918)  --> TIME: 2025-02-26 08:01:27 -- STEP: 9472/36938 -- GLOBAL_STEP: 1154550 | > loss_text_ce: 0.03337496146559715 (0.03187750267101652) | > loss_mel_ce: 3.1664838790893555 (nan) | > loss: 0.012697853147983551 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.1374430782287507) | > loader_time: 0.0046 (0.0046528786722872546)  --> TIME: 2025-02-26 08:01:37 -- STEP: 9522/36938 -- GLOBAL_STEP: 1154600 | > loss_text_ce: 0.029660237953066826 (0.03187180635267168) | > loss_mel_ce: 3.1193716526031494 (nan) | > loss: 0.0124961594119668 (nan) | > current_lr: 5e-06 | > step_time: 0.1501 (0.1374427984552559) | > loader_time: 0.0045 (0.004652476741395746)  --> TIME: 2025-02-26 08:01:46 -- STEP: 9572/36938 -- GLOBAL_STEP: 1154650 | > loss_text_ce: 0.030271967872977257 (0.0318692725191084) | > loss_mel_ce: 2.6945111751556396 (nan) | > loss: 0.010812632739543915 (nan) | > current_lr: 5e-06 | > step_time: 0.1744 (0.13742348968957813) | > loader_time: 0.0048 (0.004651987821648307)  --> TIME: 2025-02-26 08:01:55 -- STEP: 9622/36938 -- GLOBAL_STEP: 1154700 | > loss_text_ce: 0.024477092549204826 (0.03187022110355518) | > loss_mel_ce: 1.1002049446105957 (nan) | > loss: 0.004463024437427521 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.13741218005513206) | > loader_time: 0.0045 (0.0046516376135379315)  --> TIME: 2025-02-26 08:02:05 -- STEP: 9672/36938 -- GLOBAL_STEP: 1154750 | > loss_text_ce: 0.031876128166913986 (0.031869420282946204) | > loss_mel_ce: 2.969202995300293 (nan) | > loss: 0.011909044347703457 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.1374174201163112) | > loader_time: 0.0046 (0.004651444894011281)  --> TIME: 2025-02-26 08:02:14 -- STEP: 9722/36938 -- GLOBAL_STEP: 1154800 | > loss_text_ce: 0.034067291766405106 (0.03187007779254346) | > loss_mel_ce: 2.6788787841796875 (nan) | > loss: 0.010765659622848034 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13741140386690914) | > loader_time: 0.0046 (0.0046513791291268635)  --> TIME: 2025-02-26 08:02:23 -- STEP: 9772/36938 -- GLOBAL_STEP: 1154850 | > loss_text_ce: 0.038324687629938126 (0.031874795990599086) | > loss_mel_ce: 2.348878860473633 (nan) | > loss: 0.00947303045541048 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13739441056452587) | > loader_time: 0.0047 (0.004651771331487283)  --> TIME: 2025-02-26 08:02:32 -- STEP: 9822/36938 -- GLOBAL_STEP: 1154900 | > loss_text_ce: 0.03398745134472847 (0.031866910409980964) | > loss_mel_ce: 2.7331247329711914 (nan) | > loss: 0.010980605147778988 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.13739797372823834) | > loader_time: 0.0047 (0.004651918354939334)  --> TIME: 2025-02-26 08:02:42 -- STEP: 9872/36938 -- GLOBAL_STEP: 1154950 | > loss_text_ce: 0.031140677630901337 (0.0318688308943524) | > loss_mel_ce: 3.017850875854492 (nan) | > loss: 0.012099172919988632 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13741234592139434) | > loader_time: 0.0064 (0.0046523654866643385)  --> TIME: 2025-02-26 08:02:51 -- STEP: 9922/36938 -- GLOBAL_STEP: 1155000 | > loss_text_ce: 0.02538120187819004 (0.03187226616405578) | > loss_mel_ce: 2.6443698406219482 (nan) | > loss: 0.01059425063431263 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13740497437054006) | > loader_time: 0.0046 (0.004652292034938843)  --> TIME: 2025-02-26 08:03:00 -- STEP: 9972/36938 -- GLOBAL_STEP: 1155050 | > loss_text_ce: 0.029526149854063988 (0.0318728777685684) | > loss_mel_ce: 3.1027743816375732 (nan) | > loss: 0.012429765425622463 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.13736704354108303) | > loader_time: 0.0047 (0.004651981857566031)  --> TIME: 2025-02-26 08:03:10 -- STEP: 10022/36938 -- GLOBAL_STEP: 1155100 | > loss_text_ce: 0.028194377198815346 (0.031868189974178465) | > loss_mel_ce: 3.0176920890808105 (nan) | > loss: 0.01208685152232647 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1373863783788596) | > loader_time: 0.0046 (0.004651941431898507)  --> TIME: 2025-02-26 08:03:19 -- STEP: 10072/36938 -- GLOBAL_STEP: 1155150 | > loss_text_ce: 0.033751316368579865 (0.03186840004479242) | > loss_mel_ce: 2.308354139328003 (nan) | > loss: 0.00929406937211752 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13736878095210966) | > loader_time: 0.0046 (0.004651668575474703)  --> TIME: 2025-02-26 08:03:29 -- STEP: 10122/36938 -- GLOBAL_STEP: 1155200 | > loss_text_ce: 0.04173167794942856 (0.03187133046262716) | > loss_mel_ce: 2.3705453872680664 (nan) | > loss: 0.009572528302669525 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.13739504217748172) | > loader_time: 0.0047 (0.004651849883697016)  --> TIME: 2025-02-26 08:03:38 -- STEP: 10172/36938 -- GLOBAL_STEP: 1155250 | > loss_text_ce: 0.02577763795852661 (0.03186212732180843) | > loss_mel_ce: 2.373591899871826 (nan) | > loss: 0.009521308355033398 (nan) | > current_lr: 5e-06 | > step_time: 0.107 (0.13738668457148237) | > loader_time: 0.0045 (0.0046518437514565586)  --> TIME: 2025-02-26 08:03:48 -- STEP: 10222/36938 -- GLOBAL_STEP: 1155300 | > loss_text_ce: 0.03870180621743202 (0.031860604869120436) | > loss_mel_ce: 2.6300551891326904 (nan) | > loss: 0.010590305551886559 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13740732492100943) | > loader_time: 0.0047 (0.004651713525215661)  --> TIME: 2025-02-26 08:03:57 -- STEP: 10272/36938 -- GLOBAL_STEP: 1155350 | > loss_text_ce: 0.03335602208971977 (0.03185746175885438) | > loss_mel_ce: 3.1520519256591797 (nan) | > loss: 0.012640507891774178 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.13739252647506847) | > loader_time: 0.0048 (0.0046512883539511635)  --> TIME: 2025-02-26 08:04:06 -- STEP: 10322/36938 -- GLOBAL_STEP: 1155400 | > loss_text_ce: 0.035703882575035095 (0.03186202095415472) | > loss_mel_ce: 2.494619369506836 (nan) | > loss: 0.010040965862572193 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13740706081146456) | > loader_time: 0.0046 (0.004651087934807769)  --> TIME: 2025-02-26 08:04:15 -- STEP: 10372/36938 -- GLOBAL_STEP: 1155450 | > loss_text_ce: 0.037887707352638245 (0.03186099372809189) | > loss_mel_ce: 3.5826590061187744 (nan) | > loss: 0.014367249794304371 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13738921738663915) | > loader_time: 0.0045 (0.004650765710294438)  --> TIME: 2025-02-26 08:04:25 -- STEP: 10422/36938 -- GLOBAL_STEP: 1155500 | > loss_text_ce: 0.038485001772642136 (0.03186303082009266) | > loss_mel_ce: 2.451815128326416 (nan) | > loss: 0.009882143698632717 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13738193436013002) | > loader_time: 0.0045 (0.00465053961616036)  --> TIME: 2025-02-26 08:04:34 -- STEP: 10472/36938 -- GLOBAL_STEP: 1155550 | > loss_text_ce: 0.04284335672855377 (0.031857587636510296) | > loss_mel_ce: 2.46583890914917 (nan) | > loss: 0.0099550886079669 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1373729971959628) | > loader_time: 0.0045 (0.004650282190445262)  --> TIME: 2025-02-26 08:04:43 -- STEP: 10522/36938 -- GLOBAL_STEP: 1155600 | > loss_text_ce: 0.026493074372410774 (0.03185968913918227) | > loss_mel_ce: 2.969522714614868 (nan) | > loss: 0.011888951994478703 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.13735725401109936) | > loader_time: 0.0046 (0.004649952005873243)  --> TIME: 2025-02-26 08:04:53 -- STEP: 10572/36938 -- GLOBAL_STEP: 1155650 | > loss_text_ce: 0.036498844623565674 (0.03185395000560956) | > loss_mel_ce: 2.009655237197876 (nan) | > loss: 0.008119659498333931 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13736393169340994) | > loader_time: 0.0045 (0.004649777598242114)  --> TIME: 2025-02-26 08:05:02 -- STEP: 10622/36938 -- GLOBAL_STEP: 1155700 | > loss_text_ce: 0.031209958717226982 (0.03185830148092302) | > loss_mel_ce: 1.782266616821289 (nan) | > loss: 0.007196336053311825 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13737841649954186) | > loader_time: 0.0049 (0.004649739753861041)  --> TIME: 2025-02-26 08:05:12 -- STEP: 10672/36938 -- GLOBAL_STEP: 1155750 | > loss_text_ce: 0.028347892686724663 (0.03185767543937092) | > loss_mel_ce: 2.116229295730591 (nan) | > loss: 0.00851022731512785 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13736595637884083) | > loader_time: 0.0048 (0.004649539915160607)  --> TIME: 2025-02-26 08:05:21 -- STEP: 10722/36938 -- GLOBAL_STEP: 1155800 | > loss_text_ce: 0.028529351577162743 (0.03185450608952037) | > loss_mel_ce: 2.0910067558288574 (nan) | > loss: 0.008410857990384102 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.1373832844860671) | > loader_time: 0.0045 (0.004649530927362418)  --> TIME: 2025-02-26 08:05:30 -- STEP: 10772/36938 -- GLOBAL_STEP: 1155850 | > loss_text_ce: 0.02916353940963745 (0.03185491313648777) | > loss_mel_ce: 3.056565046310425 (nan) | > loss: 0.012244955636560917 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.13735996200566467) | > loader_time: 0.0047 (0.004649593955825075)  --> TIME: 2025-02-26 08:05:40 -- STEP: 10822/36938 -- GLOBAL_STEP: 1155900 | > loss_text_ce: 0.03993431106209755 (0.03185467850595897) | > loss_mel_ce: 3.0496602058410645 (nan) | > loss: 0.012260296382009983 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13735569347363236) | > loader_time: 0.0046 (0.004649724433347616)  --> TIME: 2025-02-26 08:05:49 -- STEP: 10872/36938 -- GLOBAL_STEP: 1155950 | > loss_text_ce: 0.030292129144072533 (0.03185756435423177) | > loss_mel_ce: 2.1569602489471436 (nan) | > loss: 0.00867957342416048 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13733297209251935) | > loader_time: 0.0045 (0.004649477837452377)  --> TIME: 2025-02-26 08:05:58 -- STEP: 10922/36938 -- GLOBAL_STEP: 1156000 | > loss_text_ce: 0.033045995980501175 (0.031856491802933214) | > loss_mel_ce: 0.6916258931159973 (nan) | > loss: 0.0028756821993738413 (nan) | > current_lr: 5e-06 | > step_time: 0.1613 (0.13732360699874846) | > loader_time: 0.0047 (0.004649666612607999)  --> TIME: 2025-02-26 08:06:07 -- STEP: 10972/36938 -- GLOBAL_STEP: 1156050 | > loss_text_ce: 0.0331408716738224 (0.031857031630097475) | > loss_mel_ce: 2.994265079498291 (nan) | > loss: 0.012013516388833523 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.13733627919597902) | > loader_time: 0.0048 (0.004649642389097909)  --> TIME: 2025-02-26 08:06:17 -- STEP: 11022/36938 -- GLOBAL_STEP: 1156100 | > loss_text_ce: 0.030468720942735672 (0.03186104617130082) | > loss_mel_ce: 2.623415470123291 (nan) | > loss: 0.010531286709010601 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.1373447395456911) | > loader_time: 0.0047 (0.004649612285376581)  --> TIME: 2025-02-26 08:06:26 -- STEP: 11072/36938 -- GLOBAL_STEP: 1156150 | > loss_text_ce: 0.03057221509516239 (0.03186573201842266) | > loss_mel_ce: 2.6657657623291016 (nan) | > loss: 0.01069975458085537 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13731832610647746) | > loader_time: 0.0044 (0.004649244787203775)  --> TIME: 2025-02-26 08:06:35 -- STEP: 11122/36938 -- GLOBAL_STEP: 1156200 | > loss_text_ce: 0.023435430601239204 (0.03186830760004401) | > loss_mel_ce: 2.9733238220214844 (nan) | > loss: 0.011891902424395084 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13731451258344624) | > loader_time: 0.0047 (0.004649042568710969)  --> TIME: 2025-02-26 08:06:45 -- STEP: 11172/36938 -- GLOBAL_STEP: 1156250 | > loss_text_ce: 0.021399570629000664 (0.031863127614601874) | > loss_mel_ce: 1.41927969455719 (nan) | > loss: 0.005716981831938028 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13730596636001458) | > loader_time: 0.0046 (0.004649014977456155)  --> TIME: 2025-02-26 08:06:54 -- STEP: 11222/36938 -- GLOBAL_STEP: 1156300 | > loss_text_ce: 0.0281999334692955 (0.031857506696113515) | > loss_mel_ce: 2.0485711097717285 (nan) | > loss: 0.008241155184805393 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13730916399472173) | > loader_time: 0.0045 (0.004648818474278213)  --> TIME: 2025-02-26 08:07:03 -- STEP: 11272/36938 -- GLOBAL_STEP: 1156350 | > loss_text_ce: 0.03253968060016632 (0.03185721257933847) | > loss_mel_ce: 3.0630364418029785 (nan) | > loss: 0.012284033000469208 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13731163232524451) | > loader_time: 0.0045 (0.00464855723553637)  --> TIME: 2025-02-26 08:07:13 -- STEP: 11322/36938 -- GLOBAL_STEP: 1156400 | > loss_text_ce: 0.027384623885154724 (0.03186094594660026) | > loss_mel_ce: 2.714982748031616 (nan) | > loss: 0.01088241022080183 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13731308213772828) | > loader_time: 0.0045 (0.004648805085771634)  --> TIME: 2025-02-26 08:07:22 -- STEP: 11372/36938 -- GLOBAL_STEP: 1156450 | > loss_text_ce: 0.033304452896118164 (0.0318583325600671) | > loss_mel_ce: 2.8471786975860596 (nan) | > loss: 0.011430488899350166 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13731792331203083) | > loader_time: 0.0048 (0.0046487232349810255)  --> TIME: 2025-02-26 08:07:32 -- STEP: 11422/36938 -- GLOBAL_STEP: 1156500 | > loss_text_ce: 0.02554444968700409 (0.03186407641746908) | > loss_mel_ce: 3.053356170654297 (nan) | > loss: 0.012217859737575054 (nan) | > current_lr: 5e-06 | > step_time: 0.178 (0.1373177615674857) | > loader_time: 0.0048 (0.004648840066284907)  --> TIME: 2025-02-26 08:07:41 -- STEP: 11472/36938 -- GLOBAL_STEP: 1156550 | > loss_text_ce: 0.03343954309821129 (0.03186942525933757) | > loss_mel_ce: 2.6786296367645264 (nan) | > loss: 0.01076218020170927 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13730313245472014) | > loader_time: 0.0045 (0.004648805433569918)  --> TIME: 2025-02-26 08:07:50 -- STEP: 11522/36938 -- GLOBAL_STEP: 1156600 | > loss_text_ce: 0.03577026352286339 (0.03186777311245789) | > loss_mel_ce: 1.8948835134506226 (nan) | > loss: 0.007661324925720692 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13728427597268342) | > loader_time: 0.0045 (0.004648502161469309)  --> TIME: 2025-02-26 08:07:59 -- STEP: 11572/36938 -- GLOBAL_STEP: 1156650 | > loss_text_ce: 0.02845741994678974 (0.031864276282747406) | > loss_mel_ce: 2.7408335208892822 (nan) | > loss: 0.010989250615239143 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.1372965980556814) | > loader_time: 0.0049 (0.004648331288767232)  --> TIME: 2025-02-26 08:08:08 -- STEP: 11622/36938 -- GLOBAL_STEP: 1156700 | > loss_text_ce: 0.027498416602611542 (0.03186121132893752) | > loss_mel_ce: 3.0044026374816895 (nan) | > loss: 0.012031354010105133 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.13728208989905016) | > loader_time: 0.0046 (0.004648228537665032)  --> TIME: 2025-02-26 08:08:18 -- STEP: 11672/36938 -- GLOBAL_STEP: 1156750 | > loss_text_ce: 0.02949935756623745 (0.031855573363438014) | > loss_mel_ce: 0.0557454414665699 (nan) | > loss: 0.0003382730355951935 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13729030306491224) | > loader_time: 0.0046 (0.004648057155367277)  --> TIME: 2025-02-26 08:08:27 -- STEP: 11722/36938 -- GLOBAL_STEP: 1156800 | > loss_text_ce: 0.026295572519302368 (0.03185501266447246) | > loss_mel_ce: 2.6003658771514893 (nan) | > loss: 0.010423260740935802 (nan) | > current_lr: 5e-06 | > step_time: 0.1706 (0.13730674606808768) | > loader_time: 0.0047 (0.004648020275215546)  --> TIME: 2025-02-26 08:08:37 -- STEP: 11772/36938 -- GLOBAL_STEP: 1156850 | > loss_text_ce: 0.03355270251631737 (0.031854420494414025) | > loss_mel_ce: 2.492663621902466 (nan) | > loss: 0.010024668648838997 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.137288265393536) | > loader_time: 0.0045 (0.004648073753288234)  --> TIME: 2025-02-26 08:08:46 -- STEP: 11822/36938 -- GLOBAL_STEP: 1156900 | > loss_text_ce: 0.035642560571432114 (0.031858667818177935) | > loss_mel_ce: 3.201704978942871 (nan) | > loss: 0.012846617959439754 (nan) | > current_lr: 5e-06 | > step_time: 0.182 (0.13729419754233546) | > loader_time: 0.0048 (0.004647836308454299)  --> TIME: 2025-02-26 08:08:56 -- STEP: 11872/36938 -- GLOBAL_STEP: 1156950 | > loss_text_ce: 0.03385774418711662 (0.03185592744977331) | > loss_mel_ce: 2.3156802654266357 (nan) | > loss: 0.009323564358055592 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13732050039938534) | > loader_time: 0.0045 (0.0046478471746341865)  --> TIME: 2025-02-26 08:09:05 -- STEP: 11922/36938 -- GLOBAL_STEP: 1157000 | > loss_text_ce: 0.03196161612868309 (0.03185848463906315) | > loss_mel_ce: 2.4937174320220947 (nan) | > loss: 0.010022536851465702 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.1373166786586453) | > loader_time: 0.0046 (0.004647708083135017)  --> TIME: 2025-02-26 08:09:15 -- STEP: 11972/36938 -- GLOBAL_STEP: 1157050 | > loss_text_ce: 0.030239079147577286 (0.031858827890373795) | > loss_mel_ce: 2.9359302520751953 (nan) | > loss: 0.011770513840019703 (nan) | > current_lr: 5e-06 | > step_time: 0.182 (0.13733019617302464) | > loader_time: 0.0048 (0.004647682253190442)  --> TIME: 2025-02-26 08:09:24 -- STEP: 12022/36938 -- GLOBAL_STEP: 1157100 | > loss_text_ce: 0.03838898241519928 (0.03185739314708504) | > loss_mel_ce: 2.9965527057647705 (nan) | > loss: 0.012043420225381851 (nan) | > current_lr: 5e-06 | > step_time: 0.1662 (0.13732182670206783) | > loader_time: 0.0049 (0.004647435175759728)  --> TIME: 2025-02-26 08:09:33 -- STEP: 12072/36938 -- GLOBAL_STEP: 1157150 | > loss_text_ce: 0.03056684508919716 (0.03185691392103801) | > loss_mel_ce: 1.8942304849624634 (nan) | > loss: 0.007638085167855024 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13731911567128047) | > loader_time: 0.0045 (0.004647420130477513)  --> TIME: 2025-02-26 08:09:43 -- STEP: 12122/36938 -- GLOBAL_STEP: 1157200 | > loss_text_ce: 0.03634686395525932 (0.03185410139162737) | > loss_mel_ce: 2.583652973175049 (nan) | > loss: 0.010396825149655342 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13733541491479692) | > loader_time: 0.0046 (0.004647387822573973)  --> TIME: 2025-02-26 08:09:52 -- STEP: 12172/36938 -- GLOBAL_STEP: 1157250 | > loss_text_ce: 0.03491990640759468 (0.03185689040062423) | > loss_mel_ce: 0.11279678344726562 (nan) | > loss: 0.0005861773388460279 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13733970041616622) | > loader_time: 0.0046 (0.004647406609560602)  --> TIME: 2025-02-26 08:10:01 -- STEP: 12222/36938 -- GLOBAL_STEP: 1157300 | > loss_text_ce: 0.03236701339483261 (0.03186499388633044) | > loss_mel_ce: 2.741262197494507 (nan) | > loss: 0.01100646611303091 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13732689495192774) | > loader_time: 0.0045 (0.00464715426652097)  --> TIME: 2025-02-26 08:10:11 -- STEP: 12272/36938 -- GLOBAL_STEP: 1157350 | > loss_text_ce: 0.016474634408950806 (0.03186449800814385) | > loss_mel_ce: 1.5243617296218872 (nan) | > loss: 0.006114430259913206 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13732820617173438) | > loader_time: 0.0044 (0.004646952996198013)  --> TIME: 2025-02-26 08:10:20 -- STEP: 12322/36938 -- GLOBAL_STEP: 1157400 | > loss_text_ce: 0.031257644295692444 (0.031863242372126045) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.13732173924244082) | > loader_time: 0.0048 (0.004647024303573114)  --> TIME: 2025-02-26 08:10:30 -- STEP: 12372/36938 -- GLOBAL_STEP: 1157450 | > loss_text_ce: 0.03176872804760933 (0.03186790713481236) | > loss_mel_ce: 2.4952197074890137 (nan) | > loss: 0.010027732700109482 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13732743423192417) | > loader_time: 0.0045 (0.0046469548586460935)  --> TIME: 2025-02-26 08:10:39 -- STEP: 12422/36938 -- GLOBAL_STEP: 1157500 | > loss_text_ce: 0.029978353530168533 (0.03186393988637669) | > loss_mel_ce: 3.0323801040649414 (nan) | > loss: 0.012152216397225857 (nan) | > current_lr: 5e-06 | > step_time: 0.145 (0.13731160737373763) | > loader_time: 0.0047 (0.004646676555191951)  --> TIME: 2025-02-26 08:10:48 -- STEP: 12472/36938 -- GLOBAL_STEP: 1157550 | > loss_text_ce: 0.03523822873830795 (0.031864918185731804) | > loss_mel_ce: 2.8570938110351562 (nan) | > loss: 0.011477508582174778 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.13730459495413508) | > loader_time: 0.0049 (0.004646505221409096)  --> TIME: 2025-02-26 08:10:57 -- STEP: 12522/36938 -- GLOBAL_STEP: 1157600 | > loss_text_ce: 0.036962609738111496 (0.03186251901944256) | > loss_mel_ce: 2.843153238296509 (nan) | > loss: 0.011429031379520893 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13730803116349188) | > loader_time: 0.0046 (0.004646364901130952)  --> TIME: 2025-02-26 08:11:07 -- STEP: 12572/36938 -- GLOBAL_STEP: 1157650 | > loss_text_ce: 0.031925804913043976 (0.031865637328145506) | > loss_mel_ce: 3.224271774291992 (nan) | > loss: 0.01292141992598772 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13730988667870686) | > loader_time: 0.0046 (0.004646258372392708)  --> TIME: 2025-02-26 08:11:16 -- STEP: 12622/36938 -- GLOBAL_STEP: 1157700 | > loss_text_ce: 0.03505171835422516 (0.031869569286067004) | > loss_mel_ce: 3.2407541275024414 (nan) | > loss: 0.012999230995774269 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.13731552528477423) | > loader_time: 0.0045 (0.004646221500741883)  --> TIME: 2025-02-26 08:11:25 -- STEP: 12672/36938 -- GLOBAL_STEP: 1157750 | > loss_text_ce: 0.032561082392930984 (0.03187244809404617) | > loss_mel_ce: 2.9428458213806152 (nan) | > loss: 0.011807170696556568 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.13729762673528478) | > loader_time: 0.0047 (0.004646012615976923)  --> TIME: 2025-02-26 08:11:35 -- STEP: 12722/36938 -- GLOBAL_STEP: 1157800 | > loss_text_ce: 0.0244488213211298 (0.03187239754951178) | > loss_mel_ce: 0.0289306640625 (nan) | > loss: 0.00021182335331104696 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13730583133196733) | > loader_time: 0.0045 (0.004645960339583844)  --> TIME: 2025-02-26 08:11:44 -- STEP: 12772/36938 -- GLOBAL_STEP: 1157850 | > loss_text_ce: 0.04055938497185707 (0.031875978851010546) | > loss_mel_ce: 2.165607452392578 (nan) | > loss: 0.008754630573093891 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.1373093843534749) | > loader_time: 0.0044 (0.004645855494737258)  --> TIME: 2025-02-26 08:11:54 -- STEP: 12822/36938 -- GLOBAL_STEP: 1157900 | > loss_text_ce: 0.029936011880636215 (0.03187549830601883) | > loss_mel_ce: 2.558936834335327 (nan) | > loss: 0.01027330569922924 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13732661205750477) | > loader_time: 0.0046 (0.004645830680122436)  --> TIME: 2025-02-26 08:12:03 -- STEP: 12872/36938 -- GLOBAL_STEP: 1157950 | > loss_text_ce: 0.0359937846660614 (0.031867469956498015) | > loss_mel_ce: 2.8437530994415283 (nan) | > loss: 0.011427567340433598 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13732000811267406) | > loader_time: 0.0045 (0.004645786424687818)  --> TIME: 2025-02-26 08:12:12 -- STEP: 12922/36938 -- GLOBAL_STEP: 1158000 | > loss_text_ce: 0.03700718656182289 (0.03186507464832354) | > loss_mel_ce: 3.2773876190185547 (nan) | > loss: 0.013152360916137695 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13731656495566452) | > loader_time: 0.0047 (0.004645734006010555)  --> TIME: 2025-02-26 08:12:22 -- STEP: 12972/36938 -- GLOBAL_STEP: 1158050 | > loss_text_ce: 0.0276651494204998 (0.0318593535268741) | > loss_mel_ce: 2.55365252494812 (nan) | > loss: 0.010243324562907219 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.13732634241699296) | > loader_time: 0.0047 (0.0046457554725743465)  --> TIME: 2025-02-26 08:12:31 -- STEP: 13022/36938 -- GLOBAL_STEP: 1158100 | > loss_text_ce: 0.03286060690879822 (0.031858943996590706) | > loss_mel_ce: 1.1770037412643433 (nan) | > loss: 0.004801049362868071 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13733524030079888) | > loader_time: 0.0045 (0.0046457579527339225)  --> TIME: 2025-02-26 08:12:40 -- STEP: 13072/36938 -- GLOBAL_STEP: 1158150 | > loss_text_ce: 0.03422509506344795 (0.03186307256282417) | > loss_mel_ce: 2.5221786499023438 (nan) | > loss: 0.010144459083676338 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13732694368467463) | > loader_time: 0.0045 (0.0046457883376386545)  --> TIME: 2025-02-26 08:12:50 -- STEP: 13122/36938 -- GLOBAL_STEP: 1158200 | > loss_text_ce: 0.034499023109674454 (0.03185792823440351) | > loss_mel_ce: 3.150848388671875 (nan) | > loss: 0.012640267610549927 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13733259687843757) | > loader_time: 0.0044 (0.00464556348376514)  --> TIME: 2025-02-26 08:12:59 -- STEP: 13172/36938 -- GLOBAL_STEP: 1158250 | > loss_text_ce: 0.02855660207569599 (0.03185988839928107) | > loss_mel_ce: 3.2788968086242676 (nan) | > loss: 0.013124816119670868 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.13733339736862) | > loader_time: 0.0046 (0.0046456033177843595)  --> TIME: 2025-02-26 08:13:09 -- STEP: 13222/36938 -- GLOBAL_STEP: 1158300 | > loss_text_ce: 0.0296211875975132 (0.031864692530627216) | > loss_mel_ce: 2.474895715713501 (nan) | > loss: 0.009938559494912624 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13734888904232803) | > loader_time: 0.0045 (0.004645562067350674)  --> TIME: 2025-02-26 08:13:19 -- STEP: 13272/36938 -- GLOBAL_STEP: 1158350 | > loss_text_ce: 0.03508085012435913 (0.03186042259163489) | > loss_mel_ce: 3.0428693294525146 (nan) | > loss: 0.012214088812470436 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.1373654269647568) | > loader_time: 0.0048 (0.004645623684360593)  --> TIME: 2025-02-26 08:13:28 -- STEP: 13322/36938 -- GLOBAL_STEP: 1158400 | > loss_text_ce: 0.035760547965765 (0.03186242775463038) | > loss_mel_ce: 2.277895212173462 (nan) | > loss: 0.009181174449622631 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13735533820099166) | > loader_time: 0.0045 (0.004645608760379614)  --> TIME: 2025-02-26 08:13:37 -- STEP: 13372/36938 -- GLOBAL_STEP: 1158450 | > loss_text_ce: 0.035584282130002975 (0.031859658355887235) | > loss_mel_ce: 4.03515625 (nan) | > loss: 0.016153734177350998 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.1373552531315971) | > loader_time: 0.0048 (0.004645498470029126)  --> TIME: 2025-02-26 08:13:47 -- STEP: 13422/36938 -- GLOBAL_STEP: 1158500 | > loss_text_ce: 0.03410176932811737 (0.03185934600424588) | > loss_mel_ce: 3.029207229614258 (nan) | > loss: 0.01215598825365305 (nan) | > current_lr: 5e-06 | > step_time: 0.1699 (0.13736899930058424) | > loader_time: 0.0048 (0.004645575906494641)  --> TIME: 2025-02-26 08:13:56 -- STEP: 13472/36938 -- GLOBAL_STEP: 1158550 | > loss_text_ce: 0.030141912400722504 (0.03185840909176543) | > loss_mel_ce: 0.2159423828125 (nan) | > loss: 0.0009765250724740326 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.1373511450520037) | > loader_time: 0.0046 (0.004645546336355249)  --> TIME: 2025-02-26 08:14:05 -- STEP: 13522/36938 -- GLOBAL_STEP: 1158600 | > loss_text_ce: 0.025725383311510086 (0.03186285374449181) | > loss_mel_ce: 1.167439341545105 (nan) | > loss: 0.004734780639410019 (nan) | > current_lr: 5e-06 | > step_time: 0.1118 (0.1373479216255005) | > loader_time: 0.0045 (0.0046455307201492295)  --> TIME: 2025-02-26 08:14:14 -- STEP: 13572/36938 -- GLOBAL_STEP: 1158650 | > loss_text_ce: 0.033745307475328445 (0.03186085012331296) | > loss_mel_ce: 2.9354758262634277 (nan) | > loss: 0.011782623827457428 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.1373449008161959) | > loader_time: 0.0047 (0.004645433497576474)  --> TIME: 2025-02-26 08:14:24 -- STEP: 13622/36938 -- GLOBAL_STEP: 1158700 | > loss_text_ce: 0.033170051872730255 (0.03186570342285281) | > loss_mel_ce: 0.0694986954331398 (nan) | > loss: 0.00040741567499935627 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1373422912801524) | > loader_time: 0.0045 (0.004645285111414581)  --> TIME: 2025-02-26 08:14:33 -- STEP: 13672/36938 -- GLOBAL_STEP: 1158750 | > loss_text_ce: 0.03220846503973007 (0.031868044357878045) | > loss_mel_ce: 2.191326856613159 (nan) | > loss: 0.00882355310022831 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.1373192843391993) | > loader_time: 0.0045 (0.004644962972615336)  --> TIME: 2025-02-26 08:14:42 -- STEP: 13722/36938 -- GLOBAL_STEP: 1158800 | > loss_text_ce: 0.029391080141067505 (0.03186840658766882) | > loss_mel_ce: 3.789663553237915 (nan) | > loss: 0.015154979191720486 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13732200486547746) | > loader_time: 0.0046 (0.004644965294967375)  --> TIME: 2025-02-26 08:14:52 -- STEP: 13772/36938 -- GLOBAL_STEP: 1158850 | > loss_text_ce: 0.03090301901102066 (0.031868383430462076) | > loss_mel_ce: 2.970566511154175 (nan) | > loss: 0.011910594068467617 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.1373372268545186) | > loader_time: 0.0046 (0.004644799433300201)  --> TIME: 2025-02-26 08:15:01 -- STEP: 13822/36938 -- GLOBAL_STEP: 1158900 | > loss_text_ce: 0.029943721368908882 (0.03187119882105589) | > loss_mel_ce: 2.1730055809020996 (nan) | > loss: 0.008741863071918488 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13733133688760957) | > loader_time: 0.0047 (0.004644520995827713)  --> TIME: 2025-02-26 08:15:11 -- STEP: 13872/36938 -- GLOBAL_STEP: 1158950 | > loss_text_ce: 0.03500116989016533 (0.03187612241268039) | > loss_mel_ce: 2.933852195739746 (nan) | > loss: 0.01178116537630558 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13735012653377215) | > loader_time: 0.0046 (0.004644504003871285)  --> TIME: 2025-02-26 08:15:20 -- STEP: 13922/36938 -- GLOBAL_STEP: 1159000 | > loss_text_ce: 0.0241585373878479 (0.03186963871027745) | > loss_mel_ce: 0.0539703369140625 (nan) | > loss: 0.0003100352478213608 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13734511927684254) | > loader_time: 0.0044 (0.00464426679971524)  --> TIME: 2025-02-26 08:15:29 -- STEP: 13972/36938 -- GLOBAL_STEP: 1159050 | > loss_text_ce: 0.036270108073949814 (0.0318668377245761) | > loss_mel_ce: 2.8409247398376465 (nan) | > loss: 0.011417441070079803 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.1373477981386093) | > loader_time: 0.0047 (0.00464399441790862)  --> TIME: 2025-02-26 08:15:39 -- STEP: 14022/36938 -- GLOBAL_STEP: 1159100 | > loss_text_ce: 0.03375023603439331 (0.03186223379039666) | > loss_mel_ce: 2.955707550048828 (nan) | > loss: 0.011862928979098797 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13734359643272612) | > loader_time: 0.0048 (0.00464403607198847)  --> TIME: 2025-02-26 08:15:48 -- STEP: 14072/36938 -- GLOBAL_STEP: 1159150 | > loss_text_ce: 0.03550286591053009 (0.03185993332083313) | > loss_mel_ce: 1.9805355072021484 (nan) | > loss: 0.008000153116881847 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.1373546257352475) | > loader_time: 0.0045 (0.0046439322305715245)  --> TIME: 2025-02-26 08:15:57 -- STEP: 14122/36938 -- GLOBAL_STEP: 1159200 | > loss_text_ce: 0.02865646220743656 (0.03185688504191329) | > loss_mel_ce: 1.8227962255477905 (nan) | > loss: 0.007347034756094217 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13734395553633152) | > loader_time: 0.005 (0.00464365406695482)  --> TIME: 2025-02-26 08:16:07 -- STEP: 14172/36938 -- GLOBAL_STEP: 1159250 | > loss_text_ce: 0.026417605578899384 (0.03185739008630402) | > loss_mel_ce: 0.00823974609375 (nan) | > loss: 0.00013752917584497482 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13733868411527375) | > loader_time: 0.0045 (0.004643554223852255)  --> TIME: 2025-02-26 08:16:16 -- STEP: 14222/36938 -- GLOBAL_STEP: 1159300 | > loss_text_ce: 0.03407104313373566 (0.031858255780238966) | > loss_mel_ce: 2.1018030643463135 (nan) | > loss: 0.008475691080093384 (nan) | > current_lr: 5e-06 | > step_time: 0.1558 (0.13731624519853466) | > loader_time: 0.0046 (0.004643361136943824)  --> TIME: 2025-02-26 08:16:25 -- STEP: 14272/36938 -- GLOBAL_STEP: 1159350 | > loss_text_ce: 0.028567906469106674 (0.03185675989898049) | > loss_mel_ce: 0.17755126953125 (nan) | > loss: 0.0008179332944564521 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13731402591405406) | > loader_time: 0.0047 (0.004643283984853561)  --> TIME: 2025-02-26 08:16:34 -- STEP: 14322/36938 -- GLOBAL_STEP: 1159400 | > loss_text_ce: 0.03578541427850723 (0.031852676885676126) | > loss_mel_ce: 3.012734889984131 (nan) | > loss: 0.012097303755581379 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.13729469955275436) | > loader_time: 0.0048 (0.00464313552293378)  --> TIME: 2025-02-26 08:16:43 -- STEP: 14372/36938 -- GLOBAL_STEP: 1159450 | > loss_text_ce: 0.0351291187107563 (0.031856730612406396) | > loss_mel_ce: 2.1533846855163574 (nan) | > loss: 0.008684579282999039 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.1372825766452464) | > loader_time: 0.0046 (0.004643089055285108)  --> TIME: 2025-02-26 08:16:52 -- STEP: 14422/36938 -- GLOBAL_STEP: 1159500 | > loss_text_ce: 0.02613653987646103 (0.03185734027140883) | > loss_mel_ce: 2.670273780822754 (nan) | > loss: 0.010700042359530926 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13727546839224009) | > loader_time: 0.0046 (0.004643157606001896)  --> TIME: 2025-02-26 08:17:01 -- STEP: 14472/36938 -- GLOBAL_STEP: 1159550 | > loss_text_ce: 0.035521578043699265 (0.031853822411556355) | > loss_mel_ce: 2.569817304611206 (nan) | > loss: 0.010338646359741688 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.137256974538694) | > loader_time: 0.0047 (0.004643279620466356)  --> TIME: 2025-02-26 08:17:11 -- STEP: 14522/36938 -- GLOBAL_STEP: 1159600 | > loss_text_ce: 0.03962896019220352 (0.03185470207895625) | > loss_mel_ce: 2.9177136421203613 (nan) | > loss: 0.01173548772931099 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.13724591042151196) | > loader_time: 0.0048 (0.0046432715377734204)  --> TIME: 2025-02-26 08:17:20 -- STEP: 14572/36938 -- GLOBAL_STEP: 1159650 | > loss_text_ce: 0.033789850771427155 (0.031852179591912354) | > loss_mel_ce: 2.3139538764953613 (nan) | > loss: 0.009316444396972656 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13723858782723428) | > loader_time: 0.0047 (0.0046435030417032765)  --> TIME: 2025-02-26 08:17:29 -- STEP: 14622/36938 -- GLOBAL_STEP: 1159700 | > loss_text_ce: 0.028923295438289642 (0.03184859683889254) | > loss_mel_ce: 2.7909793853759766 (nan) | > loss: 0.011190090328454971 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13724712273145406) | > loader_time: 0.0045 (0.004643459258759807)  --> TIME: 2025-02-26 08:17:39 -- STEP: 14672/36938 -- GLOBAL_STEP: 1159750 | > loss_text_ce: 0.029749324545264244 (0.031848767497591995) | > loss_mel_ce: 2.521242380142212 (nan) | > loss: 0.01012298371642828 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.13724353374870618) | > loader_time: 0.0048 (0.004643306412379654)  --> TIME: 2025-02-26 08:17:48 -- STEP: 14722/36938 -- GLOBAL_STEP: 1159800 | > loss_text_ce: 0.04151947423815727 (0.03185006454721093) | > loss_mel_ce: 2.3414647579193115 (nan) | > loss: 0.009456287138164043 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1372280867425918) | > loader_time: 0.0047 (0.00464314702509086)  --> TIME: 2025-02-26 08:17:57 -- STEP: 14772/36938 -- GLOBAL_STEP: 1159850 | > loss_text_ce: 0.0223713256418705 (0.03185168639780696) | > loss_mel_ce: 2.912839412689209 (nan) | > loss: 0.0116476621478796 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13722739752604923) | > loader_time: 0.0047 (0.0046430639771307455)  --> TIME: 2025-02-26 08:18:06 -- STEP: 14822/36938 -- GLOBAL_STEP: 1159900 | > loss_text_ce: 0.02679540030658245 (0.031848099563386925) | > loss_mel_ce: 2.5959784984588623 (nan) | > loss: 0.010407833382487297 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13721801449357826) | > loader_time: 0.0046 (0.004642931335031268)  --> TIME: 2025-02-26 08:18:15 -- STEP: 14872/36938 -- GLOBAL_STEP: 1159950 | > loss_text_ce: 0.025209635496139526 (0.031844615209043615) | > loss_mel_ce: 2.9289236068725586 (nan) | > loss: 0.011722751893103123 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13721660459278823) | > loader_time: 0.0046 (0.004642858131397419)  --> TIME: 2025-02-26 08:18:25 -- STEP: 14922/36938 -- GLOBAL_STEP: 1160000 | > loss_text_ce: 0.03091474063694477 (0.03184025513565956) | > loss_mel_ce: 3.113701105117798 (nan) | > loss: 0.012478635646402836 (nan) | > current_lr: 5e-06 | > step_time: 0.144 (0.13722383453513284) | > loader_time: 0.0046 (0.004642806285157196) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1160000.pth  --> TIME: 2025-02-26 08:18:40 -- STEP: 14972/36938 -- GLOBAL_STEP: 1160050 | > loss_text_ce: 0.03835522010922432 (0.03183972255205882) | > loss_mel_ce: 2.3631067276000977 (nan) | > loss: 0.009529611095786095 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.1372089840943498) | > loader_time: 0.0048 (0.004642859471535397)  --> TIME: 2025-02-26 08:18:49 -- STEP: 15022/36938 -- GLOBAL_STEP: 1160100 | > loss_text_ce: 0.0285661518573761 (0.031837930641768326) | > loss_mel_ce: 2.5734703540802 (nan) | > loss: 0.010325541719794273 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.13720046320064358) | > loader_time: 0.0047 (0.004642816726924304)  --> TIME: 2025-02-26 08:18:58 -- STEP: 15072/36938 -- GLOBAL_STEP: 1160150 | > loss_text_ce: 0.03719126433134079 (0.03184022672155293) | > loss_mel_ce: 3.3249878883361816 (nan) | > loss: 0.01334198098629713 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.1371934534764337) | > loader_time: 0.0048 (0.004643004332251876)  --> TIME: 2025-02-26 08:19:07 -- STEP: 15122/36938 -- GLOBAL_STEP: 1160200 | > loss_text_ce: 0.029886338859796524 (0.031836221596237696) | > loss_mel_ce: 2.7793800830841064 (nan) | > loss: 0.011147882789373398 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1371793992587801) | > loader_time: 0.0045 (0.004642861921499382)  --> TIME: 2025-02-26 08:19:16 -- STEP: 15172/36938 -- GLOBAL_STEP: 1160250 | > loss_text_ce: 0.03651252016425133 (0.03183614488880401) | > loss_mel_ce: 3.2323341369628906 (nan) | > loss: 0.012971614487469196 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13717110135641347) | > loader_time: 0.0049 (0.00464284088440003)  --> TIME: 2025-02-26 08:19:25 -- STEP: 15222/36938 -- GLOBAL_STEP: 1160300 | > loss_text_ce: 0.03376173973083496 (0.03183326194152895) | > loss_mel_ce: 0.629608154296875 (nan) | > loss: 0.002632420277222991 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.1371487277422761) | > loader_time: 0.0045 (0.004642719540203199)  --> TIME: 2025-02-26 08:19:34 -- STEP: 15272/36938 -- GLOBAL_STEP: 1160350 | > loss_text_ce: 0.03535766899585724 (0.03183398480963012) | > loss_mel_ce: 3.0530402660369873 (nan) | > loss: 0.012255548499524593 (nan) | > current_lr: 5e-06 | > step_time: 0.1626 (0.13712900866070826) | > loader_time: 0.0046 (0.004642587750291513)  --> TIME: 2025-02-26 08:19:43 -- STEP: 15322/36938 -- GLOBAL_STEP: 1160400 | > loss_text_ce: 0.02444659173488617 (0.03183597690048846) | > loss_mel_ce: 2.823089361190796 (nan) | > loss: 0.011299746111035347 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13711246000445984) | > loader_time: 0.0046 (0.004642540489533396)  --> TIME: 2025-02-26 08:19:52 -- STEP: 15372/36938 -- GLOBAL_STEP: 1160450 | > loss_text_ce: 0.024308715015649796 (0.03183632578128621) | > loss_mel_ce: 2.0330371856689453 (nan) | > loss: 0.00816407147794962 (nan) | > current_lr: 5e-06 | > step_time: 0.1266 (0.137098268562389) | > loader_time: 0.0046 (0.004642456870757768)  --> TIME: 2025-02-26 08:20:02 -- STEP: 15422/36938 -- GLOBAL_STEP: 1160500 | > loss_text_ce: 0.030270271003246307 (0.0318374913314465) | > loss_mel_ce: 1.103540062904358 (nan) | > loss: 0.004499247297644615 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.1371057768335538) | > loader_time: 0.0046 (0.004642287745826407)  --> TIME: 2025-02-26 08:20:11 -- STEP: 15472/36938 -- GLOBAL_STEP: 1160550 | > loss_text_ce: 0.029638225212693214 (0.03184308132091489) | > loss_mel_ce: 2.693110704421997 (nan) | > loss: 0.010804560035467148 (nan) | > current_lr: 5e-06 | > step_time: 0.1558 (0.13710986063916744) | > loader_time: 0.0046 (0.00464218759364156)  --> TIME: 2025-02-26 08:20:20 -- STEP: 15522/36938 -- GLOBAL_STEP: 1160600 | > loss_text_ce: 0.030590837821364403 (0.03184642670750523) | > loss_mel_ce: 2.530473232269287 (nan) | > loss: 0.010162953287363052 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.1371018760153009) | > loader_time: 0.0046 (0.004642223270417747)  --> TIME: 2025-02-26 08:20:30 -- STEP: 15572/36938 -- GLOBAL_STEP: 1160650 | > loss_text_ce: 0.03731084614992142 (0.031843884723957716) | > loss_mel_ce: 2.5115795135498047 (nan) | > loss: 0.010114644654095173 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13711766095201824) | > loader_time: 0.0045 (0.004642476589674219)  --> TIME: 2025-02-26 08:20:40 -- STEP: 15622/36938 -- GLOBAL_STEP: 1160700 | > loss_text_ce: 0.03780354931950569 (0.031845073021374674) | > loss_mel_ce: 2.5150952339172363 (nan) | > loss: 0.010130551643669605 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1371272122901158) | > loader_time: 0.0047 (0.004642585346113029)  --> TIME: 2025-02-26 08:20:49 -- STEP: 15672/36938 -- GLOBAL_STEP: 1160750 | > loss_text_ce: 0.03047093376517296 (0.03184187739752117) | > loss_mel_ce: 2.8572020530700684 (nan) | > loss: 0.011459019966423512 (nan) | > current_lr: 5e-06 | > step_time: 0.1468 (0.1371344273463991) | > loader_time: 0.0047 (0.00464268110125817)  --> TIME: 2025-02-26 08:20:58 -- STEP: 15722/36938 -- GLOBAL_STEP: 1160800 | > loss_text_ce: 0.03424423187971115 (0.03184039193364178) | > loss_mel_ce: 2.2123444080352783 (nan) | > loss: 0.008915035054087639 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13712682684267613) | > loader_time: 0.0046 (0.004642868842685558)  --> TIME: 2025-02-26 08:21:08 -- STEP: 15772/36938 -- GLOBAL_STEP: 1160850 | > loss_text_ce: 0.02261192910373211 (0.03184020996107695) | > loss_mel_ce: 2.3093624114990234 (nan) | > loss: 0.009253866970539093 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13712107657539668) | > loader_time: 0.0045 (0.004642837609308745)  --> TIME: 2025-02-26 08:21:17 -- STEP: 15822/36938 -- GLOBAL_STEP: 1160900 | > loss_text_ce: 0.03348585590720177 (0.031835476423266194) | > loss_mel_ce: 1.5897798538208008 (nan) | > loss: 0.006441531237214804 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.1371195617087974) | > loader_time: 0.0046 (0.004642634321865441)  --> TIME: 2025-02-26 08:21:27 -- STEP: 15872/36938 -- GLOBAL_STEP: 1160950 | > loss_text_ce: 0.027475709095597267 (0.03183449890210836) | > loss_mel_ce: 2.719467878341675 (nan) | > loss: 0.010900570079684258 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13714337716960584) | > loader_time: 0.0046 (0.004642500091464301)  --> TIME: 2025-02-26 08:21:36 -- STEP: 15922/36938 -- GLOBAL_STEP: 1161000 | > loss_text_ce: 0.023729776963591576 (0.03183140401401957) | > loss_mel_ce: 2.4394547939300537 (nan) | > loss: 0.009774542413651943 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13712843969949057) | > loader_time: 0.0045 (0.004642376422343023)  --> TIME: 2025-02-26 08:21:45 -- STEP: 15972/36938 -- GLOBAL_STEP: 1161050 | > loss_text_ce: 0.03722088038921356 (0.03183531995521975) | > loss_mel_ce: 2.81504487991333 (nan) | > loss: 0.011318515986204147 (nan) | > current_lr: 5e-06 | > step_time: 0.1896 (0.13714084912146504) | > loader_time: 0.0048 (0.004642233629439262)  --> TIME: 2025-02-26 08:21:55 -- STEP: 16022/36938 -- GLOBAL_STEP: 1161100 | > loss_text_ce: 0.0271809920668602 (0.031833421336918376) | > loss_mel_ce: 2.6998486518859863 (nan) | > loss: 0.01082154642790556 (nan) | > current_lr: 5e-06 | > step_time: 0.1765 (0.13714592087181) | > loader_time: 0.0048 (0.004642155060025891)  --> TIME: 2025-02-26 08:22:04 -- STEP: 16072/36938 -- GLOBAL_STEP: 1161150 | > loss_text_ce: 0.02845306135714054 (0.03183169172484083) | > loss_mel_ce: 2.0292587280273438 (nan) | > loss: 0.008165523409843445 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.1371545592696584) | > loader_time: 0.0048 (0.00464209166553379)  --> TIME: 2025-02-26 08:22:14 -- STEP: 16122/36938 -- GLOBAL_STEP: 1161200 | > loss_text_ce: 0.02356189489364624 (0.03183105985016282) | > loss_mel_ce: 1.6789429187774658 (nan) | > loss: 0.006755972281098366 (nan) | > current_lr: 5e-06 | > step_time: 0.1123 (0.13716175251246027) | > loader_time: 0.0046 (0.004642083440491273)  --> TIME: 2025-02-26 08:22:23 -- STEP: 16172/36938 -- GLOBAL_STEP: 1161250 | > loss_text_ce: 0.03126218169927597 (0.031825681053233) | > loss_mel_ce: 2.8790454864501953 (nan) | > loss: 0.011548840440809727 (nan) | > current_lr: 5e-06 | > step_time: 0.1784 (0.13717131695180043) | > loader_time: 0.0047 (0.0046420801461347526)  --> TIME: 2025-02-26 08:22:33 -- STEP: 16222/36938 -- GLOBAL_STEP: 1161300 | > loss_text_ce: 0.03940238803625107 (0.031829188542121245) | > loss_mel_ce: 2.5827460289001465 (nan) | > loss: 0.010405351407825947 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13717001943314266) | > loader_time: 0.0045 (0.004641945552273324)  --> TIME: 2025-02-26 08:22:42 -- STEP: 16272/36938 -- GLOBAL_STEP: 1161350 | > loss_text_ce: 0.03487534075975418 (0.03183397314842758) | > loss_mel_ce: 2.932173252105713 (nan) | > loss: 0.011774003505706787 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13717950452686847) | > loader_time: 0.0046 (0.004641792942992366)  --> TIME: 2025-02-26 08:22:52 -- STEP: 16322/36938 -- GLOBAL_STEP: 1161400 | > loss_text_ce: 0.027321195229887962 (0.031833892334957994) | > loss_mel_ce: 3.1640419960021973 (nan) | > loss: 0.012664140202105045 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13718101798925658) | > loader_time: 0.0046 (0.004641659849050258)  --> TIME: 2025-02-26 08:23:01 -- STEP: 16372/36938 -- GLOBAL_STEP: 1161450 | > loss_text_ce: 0.027130000293254852 (0.03183064442102928) | > loss_mel_ce: 2.592369318008423 (nan) | > loss: 0.010394838638603687 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13717255828551225) | > loader_time: 0.0047 (0.004641856551315987)  --> TIME: 2025-02-26 08:23:10 -- STEP: 16422/36938 -- GLOBAL_STEP: 1161500 | > loss_text_ce: 0.030721472576260567 (0.03183201760007826) | > loss_mel_ce: 2.6234424114227295 (nan) | > loss: 0.010532396845519543 (nan) | > current_lr: 5e-06 | > step_time: 0.168 (0.13717700068632088) | > loader_time: 0.0046 (0.004641939350667567)  --> TIME: 2025-02-26 08:23:20 -- STEP: 16472/36938 -- GLOBAL_STEP: 1161550 | > loss_text_ce: 0.029119759798049927 (0.03183110692479987) | > loss_mel_ce: 0.9634171724319458 (nan) | > loss: 0.0039386386051774025 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13717470290939884) | > loader_time: 0.0045 (0.004641993060860036)  --> TIME: 2025-02-26 08:23:29 -- STEP: 16522/36938 -- GLOBAL_STEP: 1161600 | > loss_text_ce: 0.03153827413916588 (0.03182879431531047) | > loss_mel_ce: 2.586930990219116 (nan) | > loss: 0.010390751995146275 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.1371900085068946) | > loader_time: 0.0048 (0.004641886557466034)  --> TIME: 2025-02-26 08:23:39 -- STEP: 16572/36938 -- GLOBAL_STEP: 1161650 | > loss_text_ce: 0.031460996717214584 (0.03183011548690765) | > loss_mel_ce: 3.0030994415283203 (nan) | > loss: 0.012041906826198101 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13719110627868364) | > loader_time: 0.0046 (0.004641650452753107)  --> TIME: 2025-02-26 08:23:48 -- STEP: 16622/36938 -- GLOBAL_STEP: 1161700 | > loss_text_ce: 0.031172513961791992 (0.03182585637255819) | > loss_mel_ce: 2.3898324966430664 (nan) | > loss: 0.0096071632578969 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13717664609157876) | > loader_time: 0.0047 (0.004641602263383507)  --> TIME: 2025-02-26 08:23:57 -- STEP: 16672/36938 -- GLOBAL_STEP: 1161750 | > loss_text_ce: 0.02870924398303032 (0.031827318484223835) | > loss_mel_ce: 2.977278232574463 (nan) | > loss: 0.011928522028028965 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13716087062734061) | > loader_time: 0.0046 (0.0046417122267029464)  --> TIME: 2025-02-26 08:24:06 -- STEP: 16722/36938 -- GLOBAL_STEP: 1161800 | > loss_text_ce: 0.029039889574050903 (0.031830236692778784) | > loss_mel_ce: 2.9092977046966553 (nan) | > loss: 0.011660070158541203 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.1371628734258751) | > loader_time: 0.0047 (0.004641854710277236)  --> TIME: 2025-02-26 08:24:16 -- STEP: 16772/36938 -- GLOBAL_STEP: 1161850 | > loss_text_ce: 0.03465535491704941 (0.0318280906375275) | > loss_mel_ce: 2.4085097312927246 (nan) | > loss: 0.009695099666714668 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.13716417047819898) | > loader_time: 0.0047 (0.004641732352461715)  --> TIME: 2025-02-26 08:24:25 -- STEP: 16822/36938 -- GLOBAL_STEP: 1161900 | > loss_text_ce: 0.026472710072994232 (0.03182498092151153) | > loss_mel_ce: 2.9868221282958984 (nan) | > loss: 0.011957520619034767 (nan) | > current_lr: 5e-06 | > step_time: 0.1571 (0.13718330463293274) | > loader_time: 0.0046 (0.004641685541398103)  --> TIME: 2025-02-26 08:24:34 -- STEP: 16872/36938 -- GLOBAL_STEP: 1161950 | > loss_text_ce: 0.039948150515556335 (0.03182023788656392) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1266 (0.13717141025379478) | > loader_time: 0.0045 (0.0046416095022117545)  --> TIME: 2025-02-26 08:24:44 -- STEP: 16922/36938 -- GLOBAL_STEP: 1162000 | > loss_text_ce: 0.03148426115512848 (0.03181978404631368) | > loss_mel_ce: 2.3138842582702637 (nan) | > loss: 0.009307018481194973 (nan) | > current_lr: 5e-06 | > step_time: 0.177 (0.13717236999022048) | > loader_time: 0.0048 (0.004641519146824469)  --> TIME: 2025-02-26 08:24:53 -- STEP: 16972/36938 -- GLOBAL_STEP: 1162050 | > loss_text_ce: 0.03952030465006828 (0.03181869372548368) | > loss_mel_ce: 1.0308736562728882 (nan) | > loss: 0.004247595090419054 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13715978137114476) | > loader_time: 0.0048 (0.004641427343082192)  --> TIME: 2025-02-26 08:25:02 -- STEP: 17022/36938 -- GLOBAL_STEP: 1162100 | > loss_text_ce: 0.023202145472168922 (0.03181728501468798) | > loss_mel_ce: 2.116715431213379 (nan) | > loss: 0.008491736836731434 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13714766958992106) | > loader_time: 0.0045 (0.004641246899302381)  --> TIME: 2025-02-26 08:25:11 -- STEP: 17072/36938 -- GLOBAL_STEP: 1162150 | > loss_text_ce: 0.028687192127108574 (0.03181460287035506) | > loss_mel_ce: 2.9417645931243896 (nan) | > loss: 0.011787507683038712 (nan) | > current_lr: 5e-06 | > step_time: 0.1443 (0.1371585078041686) | > loader_time: 0.0046 (0.004641104367367006)  --> TIME: 2025-02-26 08:25:21 -- STEP: 17122/36938 -- GLOBAL_STEP: 1162200 | > loss_text_ce: 0.03200359269976616 (0.031811672804985876) | > loss_mel_ce: 2.900318145751953 (nan) | > loss: 0.011636198498308659 (nan) | > current_lr: 5e-06 | > step_time: 0.1852 (0.13716367422845085) | > loader_time: 0.0047 (0.004640983805563425)  --> TIME: 2025-02-26 08:25:30 -- STEP: 17172/36938 -- GLOBAL_STEP: 1162250 | > loss_text_ce: 0.02799159660935402 (0.03181116898490347) | > loss_mel_ce: 2.596550226211548 (nan) | > loss: 0.010414849035441875 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.13716848363638612) | > loader_time: 0.0046 (0.004640936157294105)  --> TIME: 2025-02-26 08:25:40 -- STEP: 17222/36938 -- GLOBAL_STEP: 1162300 | > loss_text_ce: 0.029277844354510307 (0.03181138651126681) | > loss_mel_ce: 2.9677324295043945 (nan) | > loss: 0.011892898008227348 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.13717463762415974) | > loader_time: 0.0046 (0.00464088124080398)  --> TIME: 2025-02-26 08:25:49 -- STEP: 17272/36938 -- GLOBAL_STEP: 1162350 | > loss_text_ce: 0.027657970786094666 (0.031811604874336825) | > loss_mel_ce: 3.2765793800354004 (nan) | > loss: 0.013112054206430912 (nan) | > current_lr: 5e-06 | > step_time: 0.1702 (0.13717558579678962) | > loader_time: 0.0048 (0.0046410402968947015)  --> TIME: 2025-02-26 08:25:58 -- STEP: 17322/36938 -- GLOBAL_STEP: 1162400 | > loss_text_ce: 0.03653525561094284 (0.031815695591477955) | > loss_mel_ce: 3.3940269947052 (nan) | > loss: 0.013613343238830566 (nan) | > current_lr: 5e-06 | > step_time: 0.1798 (0.13717330427040933) | > loader_time: 0.0047 (0.004641190272750664)  --> TIME: 2025-02-26 08:26:08 -- STEP: 17372/36938 -- GLOBAL_STEP: 1162450 | > loss_text_ce: 0.022233275696635246 (0.03181309472153421) | > loss_mel_ce: 2.12184739112854 (nan) | > loss: 0.008508256636559963 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13716501678268686) | > loader_time: 0.0067 (0.004641128607452848)  --> TIME: 2025-02-26 08:26:17 -- STEP: 17422/36938 -- GLOBAL_STEP: 1162500 | > loss_text_ce: 0.02380787767469883 (0.03181052563909591) | > loss_mel_ce: 2.664461851119995 (nan) | > loss: 0.010667737573385239 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13714819144140647) | > loader_time: 0.0047 (0.004640960006376685)  --> TIME: 2025-02-26 08:26:26 -- STEP: 17472/36938 -- GLOBAL_STEP: 1162550 | > loss_text_ce: 0.03824969381093979 (0.03181205314046575) | > loss_mel_ce: 2.894773483276367 (nan) | > loss: 0.01163898129016161 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.13714262439217723) | > loader_time: 0.0047 (0.004640852002215486)  --> TIME: 2025-02-26 08:26:35 -- STEP: 17522/36938 -- GLOBAL_STEP: 1162600 | > loss_text_ce: 0.02977108024060726 (0.03180900129167834) | > loss_mel_ce: 3.143392562866211 (nan) | > loss: 0.012591919861733913 (nan) | > current_lr: 5e-06 | > step_time: 0.1689 (0.1371440860645747) | > loader_time: 0.0046 (0.004640720353498278)  --> TIME: 2025-02-26 08:26:44 -- STEP: 17572/36938 -- GLOBAL_STEP: 1162650 | > loss_text_ce: 0.02199562080204487 (0.0318068471823492) | > loss_mel_ce: 0.67535400390625 (nan) | > loss: 0.002767260419204831 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13712967588612218) | > loader_time: 0.0045 (0.004640688066897198)  --> TIME: 2025-02-26 08:26:54 -- STEP: 17622/36938 -- GLOBAL_STEP: 1162700 | > loss_text_ce: 0.041175976395606995 (0.03180942691250803) | > loss_mel_ce: 2.5333261489868164 (nan) | > loss: 0.010216278955340385 (nan) | > current_lr: 5e-06 | > step_time: 0.177 (0.13712519577068766) | > loader_time: 0.0049 (0.00464066221418805)  --> TIME: 2025-02-26 08:27:03 -- STEP: 17672/36938 -- GLOBAL_STEP: 1162750 | > loss_text_ce: 0.030816642567515373 (0.03180729071107978) | > loss_mel_ce: 2.778817892074585 (nan) | > loss: 0.01114934403449297 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.13712480360698537) | > loader_time: 0.0047 (0.004640719007182963)  --> TIME: 2025-02-26 08:27:13 -- STEP: 17722/36938 -- GLOBAL_STEP: 1162800 | > loss_text_ce: 0.041143760085105896 (0.03180614823540444) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13713773910488844) | > loader_time: 0.0045 (0.004640637785650364)  --> TIME: 2025-02-26 08:27:22 -- STEP: 17772/36938 -- GLOBAL_STEP: 1162850 | > loss_text_ce: 0.026068758219480515 (0.03180600145446861) | > loss_mel_ce: 2.0976996421813965 (nan) | > loss: 0.008427652530372143 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13714258483796096) | > loader_time: 0.0044 (0.004640529465895378)  --> TIME: 2025-02-26 08:27:32 -- STEP: 17822/36938 -- GLOBAL_STEP: 1162900 | > loss_text_ce: 0.024514401331543922 (0.03180450991378452) | > loss_mel_ce: 2.334686279296875 (nan) | > loss: 0.009361907839775085 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1371505553708196) | > loader_time: 0.0044 (0.004640439840669313)  --> TIME: 2025-02-26 08:27:41 -- STEP: 17872/36938 -- GLOBAL_STEP: 1162950 | > loss_text_ce: 0.03380642458796501 (0.031805898550849594) | > loss_mel_ce: 2.9695188999176025 (nan) | > loss: 0.011917958036065102 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.13715183864295072) | > loader_time: 0.0048 (0.004640337189821236)  --> TIME: 2025-02-26 08:27:50 -- STEP: 17922/36938 -- GLOBAL_STEP: 1163000 | > loss_text_ce: 0.03644769638776779 (0.0318111373374967) | > loss_mel_ce: 2.481203079223633 (nan) | > loss: 0.009990678168833256 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.1371502753626936) | > loader_time: 0.0048 (0.004640379823114903)  --> TIME: 2025-02-26 08:27:59 -- STEP: 17972/36938 -- GLOBAL_STEP: 1163050 | > loss_text_ce: 0.026635145768523216 (0.03181353344426777) | > loss_mel_ce: 3.090970277786255 (nan) | > loss: 0.012371450662612915 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.13713764390832903) | > loader_time: 0.0046 (0.004640383296412993)  --> TIME: 2025-02-26 08:28:08 -- STEP: 18022/36938 -- GLOBAL_STEP: 1163100 | > loss_text_ce: 0.03806377202272415 (0.031811454461420634) | > loss_mel_ce: 2.7846388816833496 (nan) | > loss: 0.011201201938092709 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13711773849144795) | > loader_time: 0.0045 (0.0046401535442065775)  --> TIME: 2025-02-26 08:28:18 -- STEP: 18072/36938 -- GLOBAL_STEP: 1163150 | > loss_text_ce: 0.03154027834534645 (0.03181140053990108) | > loss_mel_ce: 2.591252565383911 (nan) | > loss: 0.010407908819615841 (nan) | > current_lr: 5e-06 | > step_time: 0.163 (0.13712503481037877) | > loader_time: 0.0046 (0.00463997720088934)  --> TIME: 2025-02-26 08:28:27 -- STEP: 18122/36938 -- GLOBAL_STEP: 1163200 | > loss_text_ce: 0.03630702570080757 (0.03181152274893195) | > loss_mel_ce: 2.809420585632324 (nan) | > loss: 0.01129257120192051 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13711951237063735) | > loader_time: 0.0046 (0.004639790147861998)  --> TIME: 2025-02-26 08:28:36 -- STEP: 18172/36938 -- GLOBAL_STEP: 1163250 | > loss_text_ce: 0.041517630219459534 (0.0318138179123885) | > loss_mel_ce: 1.8012666702270508 (nan) | > loss: 0.007312636356800795 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.13711722717656696) | > loader_time: 0.0047 (0.004639988477708175)  --> TIME: 2025-02-26 08:28:46 -- STEP: 18222/36938 -- GLOBAL_STEP: 1163300 | > loss_text_ce: 0.03450730815529823 (0.03181097259774249) | > loss_mel_ce: 3.433773994445801 (nan) | > loss: 0.013763021677732468 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13712239866002338) | > loader_time: 0.0045 (0.004640026708005649)  --> TIME: 2025-02-26 08:28:55 -- STEP: 18272/36938 -- GLOBAL_STEP: 1163350 | > loss_text_ce: 0.03430061787366867 (0.031809117918733906) | > loss_mel_ce: 2.4571824073791504 (nan) | > loss: 0.00988683756440878 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13712194255323185) | > loader_time: 0.0045 (0.004639960629718736)  --> TIME: 2025-02-26 08:29:05 -- STEP: 18322/36938 -- GLOBAL_STEP: 1163400 | > loss_text_ce: 0.02163737639784813 (0.03181015997543585) | > loss_mel_ce: 2.3108909130096436 (nan) | > loss: 0.00925606582313776 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13714099828234744) | > loader_time: 0.0048 (0.004640212656043397)  --> TIME: 2025-02-26 08:29:14 -- STEP: 18372/36938 -- GLOBAL_STEP: 1163450 | > loss_text_ce: 0.032690998166799545 (0.03181227704898612) | > loss_mel_ce: 2.8286077976226807 (nan) | > loss: 0.011354360729455948 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.13714536072614508) | > loader_time: 0.0046 (0.00464022367212888)  --> TIME: 2025-02-26 08:29:25 -- STEP: 18422/36938 -- GLOBAL_STEP: 1163500 | > loss_text_ce: 0.03562691807746887 (0.031812215027553284) | > loss_mel_ce: 2.4547910690307617 (nan) | > loss: 0.009882611222565174 (nan) | > current_lr: 5e-06 | > step_time: 0.1898 (0.13717376278583288) | > loader_time: 0.0049 (0.004640314092647499)  --> TIME: 2025-02-26 08:29:34 -- STEP: 18472/36938 -- GLOBAL_STEP: 1163550 | > loss_text_ce: 0.020124999806284904 (0.03181068809729627) | > loss_mel_ce: 0.5598070621490479 (nan) | > loss: 0.0023013176396489143 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13717992452121258) | > loader_time: 0.0046 (0.00464036647712787)  --> TIME: 2025-02-26 08:29:43 -- STEP: 18522/36938 -- GLOBAL_STEP: 1163600 | > loss_text_ce: 0.03330054506659508 (0.031811919975746374) | > loss_mel_ce: 2.959561586380005 (nan) | > loss: 0.011876437813043594 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13718083044864995) | > loader_time: 0.0047 (0.004640562966065311)  --> TIME: 2025-02-26 08:29:53 -- STEP: 18572/36938 -- GLOBAL_STEP: 1163650 | > loss_text_ce: 0.025162378326058388 (0.03181177313904319) | > loss_mel_ce: 2.682241439819336 (nan) | > loss: 0.01074366644024849 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13717053572268345) | > loader_time: 0.0045 (0.0046405442413676905)  --> TIME: 2025-02-26 08:30:02 -- STEP: 18622/36938 -- GLOBAL_STEP: 1163700 | > loss_text_ce: 0.029375232756137848 (0.031810627771827055) | > loss_mel_ce: 3.5442514419555664 (nan) | > loss: 0.014181059785187244 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13717474161317456) | > loader_time: 0.0046 (0.004640549302882134)  --> TIME: 2025-02-26 08:30:12 -- STEP: 18672/36938 -- GLOBAL_STEP: 1163750 | > loss_text_ce: 0.03656746447086334 (0.03181088619277728) | > loss_mel_ce: 1.609519124031067 (nan) | > loss: 0.006532089784741402 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13718755480664932) | > loader_time: 0.0046 (0.004640501231951103)  --> TIME: 2025-02-26 08:30:21 -- STEP: 18722/36938 -- GLOBAL_STEP: 1163800 | > loss_text_ce: 0.028415996581315994 (0.03180729167026729) | > loss_mel_ce: 2.8112010955810547 (nan) | > loss: 0.011268322356045246 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13716710599463833) | > loader_time: 0.0046 (0.004640321779449774)  --> TIME: 2025-02-26 08:30:30 -- STEP: 18772/36938 -- GLOBAL_STEP: 1163850 | > loss_text_ce: 0.036217499524354935 (0.03180766388025383) | > loss_mel_ce: 0.016788804903626442 (nan) | > loss: 0.00021034249220974743 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13716254929299254) | > loader_time: 0.0047 (0.004640245714498548)  --> TIME: 2025-02-26 08:30:39 -- STEP: 18822/36938 -- GLOBAL_STEP: 1163900 | > loss_text_ce: 0.03488347679376602 (0.03180587871490835) | > loss_mel_ce: 1.147405743598938 (nan) | > loss: 0.004691624082624912 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13715323286144632) | > loader_time: 0.0045 (0.004640102285138492)  --> TIME: 2025-02-26 08:30:48 -- STEP: 18872/36938 -- GLOBAL_STEP: 1163950 | > loss_text_ce: 0.02996242232620716 (0.031806765306301346) | > loss_mel_ce: 1.7099324464797974 (nan) | > loss: 0.006904345005750656 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.1371507287379184) | > loader_time: 0.0046 (0.0046400079008582)  --> TIME: 2025-02-26 08:30:58 -- STEP: 18922/36938 -- GLOBAL_STEP: 1164000 | > loss_text_ce: 0.03159112110733986 (0.031807601557596384) | > loss_mel_ce: 2.46486759185791 (nan) | > loss: 0.009906582534313202 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13714189750424188) | > loader_time: 0.0046 (0.004639852010426356)  --> TIME: 2025-02-26 08:31:07 -- STEP: 18972/36938 -- GLOBAL_STEP: 1164050 | > loss_text_ce: 0.04112697020173073 (0.03180765630329805) | > loss_mel_ce: 3.115684986114502 (nan) | > loss: 0.01252703182399273 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.1371443463784339) | > loader_time: 0.0045 (0.0046398276496531846)  --> TIME: 2025-02-26 08:31:16 -- STEP: 19022/36938 -- GLOBAL_STEP: 1164100 | > loss_text_ce: 0.02970050275325775 (0.03180586400033217) | > loss_mel_ce: 1.9879721403121948 (nan) | > loss: 0.008006637915968895 (nan) | > current_lr: 5e-06 | > step_time: 0.1807 (0.13713982298025462) | > loader_time: 0.0048 (0.004639635476112476)  --> TIME: 2025-02-26 08:31:26 -- STEP: 19072/36938 -- GLOBAL_STEP: 1164150 | > loss_text_ce: 0.03257787600159645 (0.031804591426248995) | > loss_mel_ce: 2.403223752975464 (nan) | > loss: 0.009665880352258682 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.13714234787195703) | > loader_time: 0.0046 (0.004639528516755015)  --> TIME: 2025-02-26 08:31:35 -- STEP: 19122/36938 -- GLOBAL_STEP: 1164200 | > loss_text_ce: 0.02425559051334858 (0.03180156223696674) | > loss_mel_ce: 3.1482927799224854 (nan) | > loss: 0.012589477933943272 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13713668809377438) | > loader_time: 0.0045 (0.00463941694241091)  --> TIME: 2025-02-26 08:31:44 -- STEP: 19172/36938 -- GLOBAL_STEP: 1164250 | > loss_text_ce: 0.031686294823884964 (0.03179879280413664) | > loss_mel_ce: 2.913498640060425 (nan) | > loss: 0.011687242425978184 (nan) | > current_lr: 5e-06 | > step_time: 0.168 (0.13713909453995496) | > loader_time: 0.0047 (0.00463932281293577)  --> TIME: 2025-02-26 08:31:54 -- STEP: 19222/36938 -- GLOBAL_STEP: 1164300 | > loss_text_ce: 0.035532329231500626 (0.03180245556397202) | > loss_mel_ce: 2.2932684421539307 (nan) | > loss: 0.009241272695362568 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13714944498929174) | > loader_time: 0.0047 (0.004639219014754522)  --> TIME: 2025-02-26 08:32:03 -- STEP: 19272/36938 -- GLOBAL_STEP: 1164350 | > loss_text_ce: 0.02447345480322838 (0.03180356463141974) | > loss_mel_ce: 2.584195375442505 (nan) | > loss: 0.010351860895752907 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13715342122670166) | > loader_time: 0.0048 (0.0046391286088896845)  --> TIME: 2025-02-26 08:32:12 -- STEP: 19322/36938 -- GLOBAL_STEP: 1164400 | > loss_text_ce: 0.04558052867650986 (0.03180522393278512) | > loss_mel_ce: 0.9544015526771545 (nan) | > loss: 0.003968182951211929 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.1371404432927139) | > loader_time: 0.0045 (0.004638931183893487)  --> TIME: 2025-02-26 08:32:22 -- STEP: 19372/36938 -- GLOBAL_STEP: 1164450 | > loss_text_ce: 0.03218873590230942 (0.03180687944683639) | > loss_mel_ce: 2.595573902130127 (nan) | > loss: 0.010427629575133324 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13714549726330236) | > loader_time: 0.0045 (0.004638792216568587)  --> TIME: 2025-02-26 08:32:31 -- STEP: 19422/36938 -- GLOBAL_STEP: 1164500 | > loss_text_ce: 0.02897891402244568 (0.03180511749272156) | > loss_mel_ce: 0.41605472564697266 (nan) | > loss: 0.0017660065786913037 (nan) | > current_lr: 5e-06 | > step_time: 0.1822 (0.13715025507302625) | > loader_time: 0.0048 (0.004638623889302055)  --> TIME: 2025-02-26 08:32:41 -- STEP: 19472/36938 -- GLOBAL_STEP: 1164550 | > loss_text_ce: 0.026743421331048012 (0.03180448079625984) | > loss_mel_ce: 1.1949794292449951 (nan) | > loss: 0.004848106764256954 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13715249907314123) | > loader_time: 0.0045 (0.004638466674868173)  --> TIME: 2025-02-26 08:32:50 -- STEP: 19522/36938 -- GLOBAL_STEP: 1164600 | > loss_text_ce: 0.03075122833251953 (0.03180589803578773) | > loss_mel_ce: 2.027401924133301 (nan) | > loss: 0.008167275227606297 (nan) | > current_lr: 5e-06 | > step_time: 0.1799 (0.13714203464729627) | > loader_time: 0.0048 (0.0046383011916318675)  --> TIME: 2025-02-26 08:32:59 -- STEP: 19572/36938 -- GLOBAL_STEP: 1164650 | > loss_text_ce: 0.02693936973810196 (0.031807395021455805) | > loss_mel_ce: 2.883241891860962 (nan) | > loss: 0.011548339389264584 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1371459320428154) | > loader_time: 0.0045 (0.004638144216141829)  --> TIME: 2025-02-26 08:33:09 -- STEP: 19622/36938 -- GLOBAL_STEP: 1164700 | > loss_text_ce: 0.02822176180779934 (0.031810174143771026) | > loss_mel_ce: 3.0310733318328857 (nan) | > loss: 0.012140060774981976 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.13716362723394337) | > loader_time: 0.0046 (0.004638090178381117)  --> TIME: 2025-02-26 08:33:19 -- STEP: 19672/36938 -- GLOBAL_STEP: 1164750 | > loss_text_ce: 0.0195956788957119 (0.03180767899347328) | > loss_mel_ce: 0.47354575991630554 (nan) | > loss: 0.0019569105934351683 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13716730795385562) | > loader_time: 0.0046 (0.0046379795618394755)  --> TIME: 2025-02-26 08:33:28 -- STEP: 19722/36938 -- GLOBAL_STEP: 1164800 | > loss_text_ce: 0.037796929478645325 (0.031809962343361545) | > loss_mel_ce: 2.6724131107330322 (nan) | > loss: 0.010754802264273167 (nan) | > current_lr: 5e-06 | > step_time: 0.1823 (0.13716546727077852) | > loader_time: 0.0049 (0.004637839827766269)  --> TIME: 2025-02-26 08:33:37 -- STEP: 19772/36938 -- GLOBAL_STEP: 1164850 | > loss_text_ce: 0.02706066146492958 (0.031807608697885695) | > loss_mel_ce: 2.2734925746917725 (nan) | > loss: 0.009129180572926998 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13715379107415632) | > loader_time: 0.0045 (0.004637768628890049)  --> TIME: 2025-02-26 08:33:46 -- STEP: 19822/36938 -- GLOBAL_STEP: 1164900 | > loss_text_ce: 0.034639205783605576 (0.031804806352909304) | > loss_mel_ce: 2.4511144161224365 (nan) | > loss: 0.009864102117717266 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13715018646085086) | > loader_time: 0.0046 (0.004637783260016385)  --> TIME: 2025-02-26 08:33:56 -- STEP: 19872/36938 -- GLOBAL_STEP: 1164950 | > loss_text_ce: 0.03858046978712082 (0.03180385060761557) | > loss_mel_ce: 2.4902284145355225 (nan) | > loss: 0.0100349560379982 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13714710768198227) | > loader_time: 0.0045 (0.004637771110603784)  --> TIME: 2025-02-26 08:34:05 -- STEP: 19922/36938 -- GLOBAL_STEP: 1165000 | > loss_text_ce: 0.03482162579894066 (0.0318025135842438) | > loss_mel_ce: 2.4059150218963623 (nan) | > loss: 0.009685463272035122 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13715011470828373) | > loader_time: 0.0045 (0.004637611497538772)  --> TIME: 2025-02-26 08:34:14 -- STEP: 19972/36938 -- GLOBAL_STEP: 1165050 | > loss_text_ce: 0.05094347521662712 (0.03180226042439391) | > loss_mel_ce: 2.940317153930664 (nan) | > loss: 0.011870082467794418 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13714124338435388) | > loader_time: 0.0045 (0.004637429142628607)  --> TIME: 2025-02-26 08:34:24 -- STEP: 20022/36938 -- GLOBAL_STEP: 1165100 | > loss_text_ce: 0.026348821818828583 (0.031803960722474316) | > loss_mel_ce: 2.201472043991089 (nan) | > loss: 0.008840559981763363 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.1371637918579747) | > loader_time: 0.0046 (0.004637324861231755)  --> TIME: 2025-02-26 08:34:34 -- STEP: 20072/36938 -- GLOBAL_STEP: 1165150 | > loss_text_ce: 0.03367522358894348 (0.03180387748102644) | > loss_mel_ce: 3.029954671859741 (nan) | > loss: 0.012157262302935123 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13716775649284754) | > loader_time: 0.0045 (0.004637296038730645)  --> TIME: 2025-02-26 08:34:43 -- STEP: 20122/36938 -- GLOBAL_STEP: 1165200 | > loss_text_ce: 0.02904280088841915 (0.03180360590691762) | > loss_mel_ce: 2.3598082065582275 (nan) | > loss: 0.009479567408561707 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13716803591384807) | > loader_time: 0.0048 (0.004637198862410047)  --> TIME: 2025-02-26 08:34:52 -- STEP: 20172/36938 -- GLOBAL_STEP: 1165250 | > loss_text_ce: 0.029851634055376053 (0.03180327181535934) | > loss_mel_ce: 2.2157225608825684 (nan) | > loss: 0.008911008946597576 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1371565944685626) | > loader_time: 0.0044 (0.004637203388168738)  --> TIME: 2025-02-26 08:35:01 -- STEP: 20222/36938 -- GLOBAL_STEP: 1165300 | > loss_text_ce: 0.023854097351431847 (0.03180144133982859) | > loss_mel_ce: 3.1494407653808594 (nan) | > loss: 0.012592440471053123 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13714435047264364) | > loader_time: 0.0045 (0.0046370362990240535)  --> TIME: 2025-02-26 08:35:11 -- STEP: 20272/36938 -- GLOBAL_STEP: 1165350 | > loss_text_ce: 0.03701509162783623 (0.03180231865352676) | > loss_mel_ce: 3.0164318084716797 (nan) | > loss: 0.012116854079067707 (nan) | > current_lr: 5e-06 | > step_time: 0.1343 (0.1371511279501686) | > loader_time: 0.0046 (0.004636953936944176)  --> TIME: 2025-02-26 08:35:20 -- STEP: 20322/36938 -- GLOBAL_STEP: 1165400 | > loss_text_ce: 0.0349600724875927 (0.03180412751001179) | > loss_mel_ce: 2.8592939376831055 (nan) | > loss: 0.011485135182738304 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.1371632589386763) | > loader_time: 0.0046 (0.004636904900262436)  --> TIME: 2025-02-26 08:35:30 -- STEP: 20372/36938 -- GLOBAL_STEP: 1165450 | > loss_text_ce: 0.022747015580534935 (0.031806927282996444) | > loss_mel_ce: 3.737499952316284 (nan) | > loss: 0.014921615831553936 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13716240145007128) | > loader_time: 0.0044 (0.004636794088772996)  --> TIME: 2025-02-26 08:35:39 -- STEP: 20422/36938 -- GLOBAL_STEP: 1165500 | > loss_text_ce: 0.034666892141103745 (0.031807870005638235) | > loss_mel_ce: 2.6830971240997314 (nan) | > loss: 0.01078477781265974 (nan) | > current_lr: 5e-06 | > step_time: 0.1869 (0.1371541141167805) | > loader_time: 0.0048 (0.0046366627122242056)  --> TIME: 2025-02-26 08:35:48 -- STEP: 20472/36938 -- GLOBAL_STEP: 1165550 | > loss_text_ce: 0.03733239695429802 (0.03180335685420634) | > loss_mel_ce: 2.943964719772339 (nan) | > loss: 0.011830544099211693 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13715913888298467) | > loader_time: 0.0046 (0.004636580704617867)  --> TIME: 2025-02-26 08:35:57 -- STEP: 20522/36938 -- GLOBAL_STEP: 1165600 | > loss_text_ce: 0.030649418011307716 (0.03180527851921681) | > loss_mel_ce: 1.4455227851867676 (nan) | > loss: 0.005857826676219702 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1371554300550619) | > loader_time: 0.0047 (0.004636691880846543)  --> TIME: 2025-02-26 08:36:06 -- STEP: 20572/36938 -- GLOBAL_STEP: 1165650 | > loss_text_ce: 0.03724788501858711 (0.03180452507234995) | > loss_mel_ce: 0.002422233344987035 (nan) | > loss: 0.00015742110554128885 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13713847558229028) | > loader_time: 0.0047 (0.004636578260403604)  --> TIME: 2025-02-26 08:36:16 -- STEP: 20622/36938 -- GLOBAL_STEP: 1165700 | > loss_text_ce: 0.030427230522036552 (0.03180269813873221) | > loss_mel_ce: 3.0827574729919434 (nan) | > loss: 0.012353908270597458 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13713043625816518) | > loader_time: 0.0045 (0.0046364638844929515)  --> TIME: 2025-02-26 08:36:25 -- STEP: 20672/36938 -- GLOBAL_STEP: 1165750 | > loss_text_ce: 0.01755346916615963 (0.031802274812283066) | > loss_mel_ce: 0.003149350406602025 (nan) | > loss: 8.215405250666663e-05 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13712616038765324) | > loader_time: 0.0045 (0.004636424037140614)  --> TIME: 2025-02-26 08:36:34 -- STEP: 20722/36938 -- GLOBAL_STEP: 1165800 | > loss_text_ce: 0.04105278104543686 (0.03180325331868248) | > loss_mel_ce: 2.6474318504333496 (nan) | > loss: 0.010668590664863586 (nan) | > current_lr: 5e-06 | > step_time: 0.1714 (0.13713641802255122) | > loader_time: 0.0047 (0.0046364240187980335)  --> TIME: 2025-02-26 08:36:44 -- STEP: 20772/36938 -- GLOBAL_STEP: 1165850 | > loss_text_ce: 0.039212606847286224 (0.03179846215922706) | > loss_mel_ce: 3.2340188026428223 (nan) | > loss: 0.012989014387130737 (nan) | > current_lr: 5e-06 | > step_time: 0.1563 (0.1371334749633538) | > loader_time: 0.0046 (0.004636326909134133)  --> TIME: 2025-02-26 08:36:53 -- STEP: 20822/36938 -- GLOBAL_STEP: 1165900 | > loss_text_ce: 0.03218662366271019 (0.03179884278712416) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.1371423826473224) | > loader_time: 0.0045 (0.004636278838111049)  --> TIME: 2025-02-26 08:37:03 -- STEP: 20872/36938 -- GLOBAL_STEP: 1165950 | > loss_text_ce: 0.039948225021362305 (0.03179827315464957) | > loss_mel_ce: 2.680276393890381 (nan) | > loss: 0.010794542729854584 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.1371488089120998) | > loader_time: 0.0045 (0.004636208608535229)  --> TIME: 2025-02-26 08:37:12 -- STEP: 20922/36938 -- GLOBAL_STEP: 1166000 | > loss_text_ce: 0.03480532020330429 (0.03180032500549578) | > loss_mel_ce: 3.061957836151123 (nan) | > loss: 0.012288743630051613 (nan) | > current_lr: 5e-06 | > step_time: 0.1908 (0.13715403958130934) | > loader_time: 0.0049 (0.004636153791001168)  --> TIME: 2025-02-26 08:37:22 -- STEP: 20972/36938 -- GLOBAL_STEP: 1166050 | > loss_text_ce: 0.03158734366297722 (0.03179708850568629) | > loss_mel_ce: 0.01715150475502014 (nan) | > loss: 0.00019340813742019236 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13716617298790185) | > loader_time: 0.0045 (0.00463606409505424)  --> TIME: 2025-02-26 08:37:31 -- STEP: 21022/36938 -- GLOBAL_STEP: 1166100 | > loss_text_ce: 0.0259717907756567 (0.031798825220513215) | > loss_mel_ce: 2.7326712608337402 (nan) | > loss: 0.010946997441351414 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13717304151843754) | > loader_time: 0.0048 (0.004636027517854972)  --> TIME: 2025-02-26 08:37:41 -- STEP: 21072/36938 -- GLOBAL_STEP: 1166150 | > loss_text_ce: 0.02939142845571041 (0.03179643054692743) | > loss_mel_ce: 2.0021188259124756 (nan) | > loss: 0.008061549626290798 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13717825405282744) | > loader_time: 0.0044 (0.004635926768828661)  --> TIME: 2025-02-26 08:37:50 -- STEP: 21122/36938 -- GLOBAL_STEP: 1166200 | > loss_text_ce: 0.02591879852116108 (0.03179140971485369) | > loss_mel_ce: 2.6281793117523193 (nan) | > loss: 0.01053213607519865 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1371734417610221) | > loader_time: 0.0046 (0.004635807341579352)  --> TIME: 2025-02-26 08:37:59 -- STEP: 21172/36938 -- GLOBAL_STEP: 1166250 | > loss_text_ce: 0.039594825357198715 (0.03179419511633048) | > loss_mel_ce: 2.9385552406311035 (nan) | > loss: 0.011818056926131248 (nan) | > current_lr: 5e-06 | > step_time: 0.1578 (0.1371638636811573) | > loader_time: 0.0047 (0.004635697611108461)  --> TIME: 2025-02-26 08:38:08 -- STEP: 21222/36938 -- GLOBAL_STEP: 1166300 | > loss_text_ce: 0.03116264007985592 (0.031790860162806085) | > loss_mel_ce: 2.394646406173706 (nan) | > loss: 0.009626227430999279 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.13715286710329525) | > loader_time: 0.0048 (0.004635656804573824)  --> TIME: 2025-02-26 08:38:17 -- STEP: 21272/36938 -- GLOBAL_STEP: 1166350 | > loss_text_ce: 0.0333574004471302 (0.031787820823768256) | > loss_mel_ce: 2.9316248893737793 (nan) | > loss: 0.011765803210437298 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.13714494806969732) | > loader_time: 0.0049 (0.004635631959659828)  --> TIME: 2025-02-26 08:38:27 -- STEP: 21322/36938 -- GLOBAL_STEP: 1166400 | > loss_text_ce: 0.035971175879240036 (0.03178615484144901) | > loss_mel_ce: 0.9591756463050842 (nan) | > loss: 0.003948995377868414 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.1371458026036889) | > loader_time: 0.0046 (0.004635637601067572)  --> TIME: 2025-02-26 08:38:36 -- STEP: 21372/36938 -- GLOBAL_STEP: 1166450 | > loss_text_ce: 0.030646465718746185 (0.03178909107528187) | > loss_mel_ce: 2.8821041584014893 (nan) | > loss: 0.011558535508811474 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13714394625398388) | > loader_time: 0.0045 (0.004635604059743081)  --> TIME: 2025-02-26 08:38:45 -- STEP: 21422/36938 -- GLOBAL_STEP: 1166500 | > loss_text_ce: 0.027555543929338455 (0.03178528107833354) | > loss_mel_ce: 2.798861503601074 (nan) | > loss: 0.011215941049158573 (nan) | > current_lr: 5e-06 | > step_time: 0.1457 (0.13714517067709942) | > loader_time: 0.0046 (0.0046355234743021894)  --> TIME: 2025-02-26 08:38:55 -- STEP: 21472/36938 -- GLOBAL_STEP: 1166550 | > loss_text_ce: 0.031732991337776184 (0.03178463795290203) | > loss_mel_ce: 2.6642627716064453 (nan) | > loss: 0.01069839671254158 (nan) | > current_lr: 5e-06 | > step_time: 0.1522 (0.1371564897535104) | > loader_time: 0.0048 (0.004635510130628595)  --> TIME: 2025-02-26 08:39:04 -- STEP: 21522/36938 -- GLOBAL_STEP: 1166600 | > loss_text_ce: 0.028934866189956665 (0.03178606931432264) | > loss_mel_ce: 0.04656982421875 (nan) | > loss: 0.00029962180997245014 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13715939316405315) | > loader_time: 0.0046 (0.004635441282205193)  --> TIME: 2025-02-26 08:39:14 -- STEP: 21572/36938 -- GLOBAL_STEP: 1166650 | > loss_text_ce: 0.027726490050554276 (0.031783453282910205) | > loss_mel_ce: 2.8542075157165527 (nan) | > loss: 0.01143624633550644 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.13716024819097725) | > loader_time: 0.0046 (0.004635457457184002)  --> TIME: 2025-02-26 08:39:23 -- STEP: 21622/36938 -- GLOBAL_STEP: 1166700 | > loss_text_ce: 0.025851022452116013 (0.031782472146291506) | > loss_mel_ce: 2.4907126426696777 (nan) | > loss: 0.009986364282667637 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13714804481527199) | > loader_time: 0.0045 (0.004635400009490511)  --> TIME: 2025-02-26 08:39:32 -- STEP: 21672/36938 -- GLOBAL_STEP: 1166750 | > loss_text_ce: 0.028079861775040627 (0.03177916823714292) | > loss_mel_ce: 2.727390766143799 (nan) | > loss: 0.010934407822787762 (nan) | > current_lr: 5e-06 | > step_time: 0.1465 (0.13715187723938527) | > loader_time: 0.0046 (0.00463542975856619)  --> TIME: 2025-02-26 08:39:42 -- STEP: 21722/36938 -- GLOBAL_STEP: 1166800 | > loss_text_ce: 0.03329525515437126 (0.03177605396590792) | > loss_mel_ce: 2.991715431213379 (nan) | > loss: 0.012004011310636997 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13715606589265106) | > loader_time: 0.0046 (0.0046353759099168395)  --> TIME: 2025-02-26 08:39:51 -- STEP: 21772/36938 -- GLOBAL_STEP: 1166850 | > loss_text_ce: 0.03995034098625183 (0.03177647429074236) | > loss_mel_ce: 1.3149359226226807 (nan) | > loss: 0.005376533139497042 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13714626301699404) | > loader_time: 0.0045 (0.004635289226541118)  --> TIME: 2025-02-26 08:40:00 -- STEP: 21822/36938 -- GLOBAL_STEP: 1166900 | > loss_text_ce: 0.021538330242037773 (0.031777461270947306) | > loss_mel_ce: 1.0037803649902344 (nan) | > loss: 0.004068725276738405 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13714650557661973) | > loader_time: 0.0046 (0.004635214368578801)  --> TIME: 2025-02-26 08:40:09 -- STEP: 21872/36938 -- GLOBAL_STEP: 1166950 | > loss_text_ce: 0.02989078126847744 (0.0317790401511795) | > loss_mel_ce: 3.019498348236084 (nan) | > loss: 0.012100751511752605 (nan) | > current_lr: 5e-06 | > step_time: 0.1567 (0.13713685635153017) | > loader_time: 0.0047 (0.00463513331249326)  --> TIME: 2025-02-26 08:40:18 -- STEP: 21922/36938 -- GLOBAL_STEP: 1167000 | > loss_text_ce: 0.04093354195356369 (0.031776198910918395) | > loss_mel_ce: 3.6426784992218018 (nan) | > loss: 0.014617509208619595 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1371311057213177) | > loader_time: 0.0047 (0.004635138425086632)  --> TIME: 2025-02-26 08:40:28 -- STEP: 21972/36938 -- GLOBAL_STEP: 1167050 | > loss_text_ce: 0.0288859773427248 (0.031774324636998835) | > loss_mel_ce: 2.745595693588257 (nan) | > loss: 0.011009848676621914 (nan) | > current_lr: 5e-06 | > step_time: 0.178 (0.13712766892962513) | > loader_time: 0.0048 (0.00463505266968421)  --> TIME: 2025-02-26 08:40:37 -- STEP: 22022/36938 -- GLOBAL_STEP: 1167100 | > loss_text_ce: 0.02861175872385502 (0.03177355986026624) | > loss_mel_ce: 1.5504950284957886 (nan) | > loss: 0.006266297306865454 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13712289919916865) | > loader_time: 0.0046 (0.004635096473250429)  --> TIME: 2025-02-26 08:40:46 -- STEP: 22072/36938 -- GLOBAL_STEP: 1167150 | > loss_text_ce: 0.039020393043756485 (0.03177314018552192) | > loss_mel_ce: 3.113403081893921 (nan) | > loss: 0.012509617023169994 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.1371177182157819) | > loader_time: 0.0046 (0.004634968998554034)  --> TIME: 2025-02-26 08:40:55 -- STEP: 22122/36938 -- GLOBAL_STEP: 1167200 | > loss_text_ce: 0.03886125981807709 (0.03177028920862874) | > loss_mel_ce: 2.5523059368133545 (nan) | > loss: 0.010282410308718681 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.1371115356099054) | > loader_time: 0.0047 (0.00463492953748217)  --> TIME: 2025-02-26 08:41:05 -- STEP: 22172/36938 -- GLOBAL_STEP: 1167250 | > loss_text_ce: 0.0362866185605526 (0.031769996663994604) | > loss_mel_ce: 2.986703872680664 (nan) | > loss: 0.011995994485914707 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13710913172315303) | > loader_time: 0.0046 (0.004634912244554878)  --> TIME: 2025-02-26 08:41:14 -- STEP: 22222/36938 -- GLOBAL_STEP: 1167300 | > loss_text_ce: 0.031283531337976456 (0.03176804726987656) | > loss_mel_ce: 2.965451717376709 (nan) | > loss: 0.011891807429492474 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.137104001292851) | > loader_time: 0.0045 (0.004634891220671646)  --> TIME: 2025-02-26 08:41:23 -- STEP: 22272/36938 -- GLOBAL_STEP: 1167350 | > loss_text_ce: 0.030822433531284332 (0.03176845498196572) | > loss_mel_ce: 2.2546536922454834 (nan) | > loss: 0.00906935054808855 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.137104618844801) | > loader_time: 0.0046 (0.004634896753591394)  --> TIME: 2025-02-26 08:41:33 -- STEP: 22322/36938 -- GLOBAL_STEP: 1167400 | > loss_text_ce: 0.029298538342118263 (0.031764514688582914) | > loss_mel_ce: 3.0586283206939697 (nan) | > loss: 0.012253678403794765 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.1371128903036849) | > loader_time: 0.0048 (0.004634978939754011)  --> TIME: 2025-02-26 08:41:42 -- STEP: 22372/36938 -- GLOBAL_STEP: 1167450 | > loss_text_ce: 0.028505971655249596 (0.03176432988203784) | > loss_mel_ce: 2.4369888305664062 (nan) | > loss: 0.009783710353076458 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.137111720865322) | > loader_time: 0.0047 (0.004634915897847526)  --> TIME: 2025-02-26 08:41:51 -- STEP: 22422/36938 -- GLOBAL_STEP: 1167500 | > loss_text_ce: 0.033354490995407104 (0.03176621721403539) | > loss_mel_ce: 2.6220710277557373 (nan) | > loss: 0.010537403635680676 (nan) | > current_lr: 5e-06 | > step_time: 0.1897 (0.1371045255839647) | > loader_time: 0.0049 (0.00463490375133963)  --> TIME: 2025-02-26 08:42:01 -- STEP: 22472/36938 -- GLOBAL_STEP: 1167550 | > loss_text_ce: 0.030963456258177757 (0.03176539594307729) | > loss_mel_ce: 3.039954900741577 (nan) | > loss: 0.012186184525489807 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13711578306374303) | > loader_time: 0.0046 (0.004635048977799949)  --> TIME: 2025-02-26 08:42:11 -- STEP: 22522/36938 -- GLOBAL_STEP: 1167600 | > loss_text_ce: 0.0295565165579319 (0.03176696245374072) | > loss_mel_ce: 2.3889710903167725 (nan) | > loss: 0.00959733221679926 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13712530294475392) | > loader_time: 0.0046 (0.00463511344436094)  --> TIME: 2025-02-26 08:42:20 -- STEP: 22572/36938 -- GLOBAL_STEP: 1167650 | > loss_text_ce: 0.032215237617492676 (0.03176511323798901) | > loss_mel_ce: 2.9312584400177 (nan) | > loss: 0.011759817600250244 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13712404313164342) | > loader_time: 0.0045 (0.004635050061026665)  --> TIME: 2025-02-26 08:42:29 -- STEP: 22622/36938 -- GLOBAL_STEP: 1167700 | > loss_text_ce: 0.028550662100315094 (0.03176438547401622) | > loss_mel_ce: 2.8430426120758057 (nan) | > loss: 0.011395212262868881 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13712187428213446) | > loader_time: 0.0045 (0.004635093551701868)  --> TIME: 2025-02-26 08:42:39 -- STEP: 22672/36938 -- GLOBAL_STEP: 1167750 | > loss_text_ce: 0.030415508896112442 (0.031761886431658705) | > loss_mel_ce: 2.3393990993499756 (nan) | > loss: 0.00940402690321207 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13712407607550992) | > loader_time: 0.0047 (0.004635033667634415)  --> TIME: 2025-02-26 08:42:48 -- STEP: 22722/36938 -- GLOBAL_STEP: 1167800 | > loss_text_ce: 0.030613379552960396 (0.03176236782404506) | > loss_mel_ce: 1.7531132698059082 (nan) | > loss: 0.0070782811380922794 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1371166496018308) | > loader_time: 0.0046 (0.00463511239536927)  --> TIME: 2025-02-26 08:42:57 -- STEP: 22772/36938 -- GLOBAL_STEP: 1167850 | > loss_text_ce: 0.024082127958536148 (0.03176329321366718) | > loss_mel_ce: 1.9592459201812744 (nan) | > loss: 0.007870350033044815 (nan) | > current_lr: 5e-06 | > step_time: 0.1079 (0.1371052668464424) | > loader_time: 0.0045 (0.004635063569155878)  --> TIME: 2025-02-26 08:43:07 -- STEP: 22822/36938 -- GLOBAL_STEP: 1167900 | > loss_text_ce: 0.0361412838101387 (0.031762439266705085) | > loss_mel_ce: 2.9686994552612305 (nan) | > loss: 0.011923972517251968 (nan) | > current_lr: 5e-06 | > step_time: 0.1755 (0.1371281352702135) | > loader_time: 0.0049 (0.0046351525840161115)  --> TIME: 2025-02-26 08:43:16 -- STEP: 22872/36938 -- GLOBAL_STEP: 1167950 | > loss_text_ce: 0.03718217462301254 (0.03176319406897374) | > loss_mel_ce: 2.649712085723877 (nan) | > loss: 0.010662279091775417 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13713222204200304) | > loader_time: 0.0046 (0.004635057621545936)  --> TIME: 2025-02-26 08:43:26 -- STEP: 22922/36938 -- GLOBAL_STEP: 1168000 | > loss_text_ce: 0.033092498779296875 (0.03176117026959842) | > loss_mel_ce: 2.490380048751831 (nan) | > loss: 0.010013780556619167 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13713673818861277) | > loader_time: 0.0046 (0.004635091810240611)  --> TIME: 2025-02-26 08:43:35 -- STEP: 22972/36938 -- GLOBAL_STEP: 1168050 | > loss_text_ce: 0.030574187636375427 (0.03176382045481646) | > loss_mel_ce: 2.8139519691467285 (nan) | > loss: 0.011287802830338478 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13713284643522858) | > loader_time: 0.0045 (0.004635164635160276)  --> TIME: 2025-02-26 08:43:45 -- STEP: 23022/36938 -- GLOBAL_STEP: 1168100 | > loss_text_ce: 0.02917574532330036 (0.031764626913352155) | > loss_mel_ce: 3.134821891784668 (nan) | > loss: 0.012555547058582306 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.1371327605797193) | > loader_time: 0.0045 (0.004635093877197692)  --> TIME: 2025-02-26 08:43:54 -- STEP: 23072/36938 -- GLOBAL_STEP: 1168150 | > loss_text_ce: 0.03668469563126564 (0.03176406727341346) | > loss_mel_ce: 0.9011974930763245 (nan) | > loss: 0.0037217549979686737 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1371277428119747) | > loader_time: 0.0046 (0.004634971127182725)  --> TIME: 2025-02-26 08:44:03 -- STEP: 23122/36938 -- GLOBAL_STEP: 1168200 | > loss_text_ce: 0.02794089913368225 (0.031765425552626084) | > loss_mel_ce: 3.0051989555358887 (nan) | > loss: 0.012036270461976528 (nan) | > current_lr: 5e-06 | > step_time: 0.1659 (0.1371181509649649) | > loader_time: 0.0046 (0.004634849722642595)  --> TIME: 2025-02-26 08:44:12 -- STEP: 23172/36938 -- GLOBAL_STEP: 1168250 | > loss_text_ce: 0.03349987417459488 (0.03176446608400378) | > loss_mel_ce: 3.2628068923950195 (nan) | > loss: 0.01308058388531208 (nan) | > current_lr: 5e-06 | > step_time: 0.1664 (0.13711173953570382) | > loader_time: 0.0048 (0.004634738349140838)  --> TIME: 2025-02-26 08:44:21 -- STEP: 23222/36938 -- GLOBAL_STEP: 1168300 | > loss_text_ce: 0.03239937871694565 (0.03176678890607366) | > loss_mel_ce: 2.9750022888183594 (nan) | > loss: 0.011934134177863598 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13709398529940373) | > loader_time: 0.0046 (0.004634689518816279)  --> TIME: 2025-02-26 08:44:30 -- STEP: 23272/36938 -- GLOBAL_STEP: 1168350 | > loss_text_ce: 0.03037894330918789 (0.03176870207974933) | > loss_mel_ce: 2.7404608726501465 (nan) | > loss: 0.010995397344231606 (nan) | > current_lr: 5e-06 | > step_time: 0.1699 (0.13708762746162056) | > loader_time: 0.0049 (0.0046348241072989745)  --> TIME: 2025-02-26 08:44:39 -- STEP: 23322/36938 -- GLOBAL_STEP: 1168400 | > loss_text_ce: 0.03565094247460365 (0.031768475893120623) | > loss_mel_ce: 3.0888760089874268 (nan) | > loss: 0.012398917227983475 (nan) | > current_lr: 5e-06 | > step_time: 0.1791 (0.13707715585213098) | > loader_time: 0.0048 (0.004634737293733154)  --> TIME: 2025-02-26 08:44:48 -- STEP: 23372/36938 -- GLOBAL_STEP: 1168450 | > loss_text_ce: 0.042678967118263245 (0.03176854655243693) | > loss_mel_ce: 2.335286855697632 (nan) | > loss: 0.009436373598873615 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.137075733132666) | > loader_time: 0.0045 (0.004634646985417849)  --> TIME: 2025-02-26 08:44:58 -- STEP: 23422/36938 -- GLOBAL_STEP: 1168500 | > loss_text_ce: 0.03900611400604248 (0.03176998003482133) | > loss_mel_ce: 2.701530933380127 (nan) | > loss: 0.010875147767364979 (nan) | > current_lr: 5e-06 | > step_time: 0.1436 (0.13708473819497755) | > loader_time: 0.0047 (0.004634700559671977)  --> TIME: 2025-02-26 08:45:07 -- STEP: 23472/36938 -- GLOBAL_STEP: 1168550 | > loss_text_ce: 0.037286557257175446 (0.03176914421861685) | > loss_mel_ce: 1.5052083730697632 (nan) | > loss: 0.006121011916548014 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13707991641195041) | > loader_time: 0.0045 (0.0046347826617708565)  --> TIME: 2025-02-26 08:45:17 -- STEP: 23522/36938 -- GLOBAL_STEP: 1168600 | > loss_text_ce: 0.031339455395936966 (0.031767691303096805) | > loss_mel_ce: 2.5373775959014893 (nan) | > loss: 0.010193321853876114 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13708436149549752) | > loader_time: 0.0046 (0.004634697394107119)  --> TIME: 2025-02-26 08:45:26 -- STEP: 23572/36938 -- GLOBAL_STEP: 1168650 | > loss_text_ce: 0.046569693833589554 (0.03176690185114354) | > loss_mel_ce: 2.983978748321533 (nan) | > loss: 0.01202598586678505 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13707894347791358) | > loader_time: 0.0045 (0.004634604811284075)  --> TIME: 2025-02-26 08:45:35 -- STEP: 23622/36938 -- GLOBAL_STEP: 1168700 | > loss_text_ce: 0.02566838450729847 (0.03176646072103856) | > loss_mel_ce: 2.7642822265625 (nan) | > loss: 0.011071233078837395 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13707159674951822) | > loader_time: 0.0047 (0.004634596867279169)  --> TIME: 2025-02-26 08:45:44 -- STEP: 23672/36938 -- GLOBAL_STEP: 1168750 | > loss_text_ce: 0.02058589830994606 (0.03176408616817945) | > loss_mel_ce: 1.944123387336731 (nan) | > loss: 0.007796465884894133 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.1370651364104893) | > loader_time: 0.0046 (0.004634539162077294)  --> TIME: 2025-02-26 08:45:54 -- STEP: 23722/36938 -- GLOBAL_STEP: 1168800 | > loss_text_ce: 0.020318234339356422 (0.03176370040382403) | > loss_mel_ce: 2.7158586978912354 (nan) | > loss: 0.010857845656573772 (nan) | > current_lr: 5e-06 | > step_time: 0.17 (0.13707370982585332) | > loader_time: 0.0049 (0.004634480453866248)  --> TIME: 2025-02-26 08:46:03 -- STEP: 23772/36938 -- GLOBAL_STEP: 1168850 | > loss_text_ce: 0.0438055582344532 (0.03176467952949435) | > loss_mel_ce: 0.018771830946207047 (nan) | > loss: 0.0002483229909557849 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13706774317855103) | > loader_time: 0.0045 (0.004634347323835398)  --> TIME: 2025-02-26 08:46:12 -- STEP: 23822/36938 -- GLOBAL_STEP: 1168900 | > loss_text_ce: 0.03045864962041378 (0.03176211513513714) | > loss_mel_ce: 2.045510768890381 (nan) | > loss: 0.008237974718213081 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13706180263892664) | > loader_time: 0.0045 (0.004634375126233167)  --> TIME: 2025-02-26 08:46:21 -- STEP: 23872/36938 -- GLOBAL_STEP: 1168950 | > loss_text_ce: 0.02275993488729 (0.03176168676217813) | > loss_mel_ce: 2.278353691101074 (nan) | > loss: 0.009131403639912605 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.1370576689114675) | > loader_time: 0.0045 (0.004634224287865975)  --> TIME: 2025-02-26 08:46:30 -- STEP: 23922/36938 -- GLOBAL_STEP: 1169000 | > loss_text_ce: 0.0320182628929615 (0.031761053814696676) | > loss_mel_ce: 1.2787827253341675 (nan) | > loss: 0.005201591644436121 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13703946673185352) | > loader_time: 0.0045 (0.004634141832298608)  --> TIME: 2025-02-26 08:46:39 -- STEP: 23972/36938 -- GLOBAL_STEP: 1169050 | > loss_text_ce: 0.034029774367809296 (0.03175898694646821) | > loss_mel_ce: 3.091914415359497 (nan) | > loss: 0.012404540553689003 (nan) | > current_lr: 5e-06 | > step_time: 0.1887 (0.13704010024206764) | > loader_time: 0.0049 (0.004634088155484558)  --> TIME: 2025-02-26 08:46:48 -- STEP: 24022/36938 -- GLOBAL_STEP: 1169100 | > loss_text_ce: 0.0412767194211483 (0.03175873599242313) | > loss_mel_ce: 1.9948601722717285 (nan) | > loss: 0.008079908788204193 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13702866813938167) | > loader_time: 0.0044 (0.004634072724830605)  --> TIME: 2025-02-26 08:46:58 -- STEP: 24072/36938 -- GLOBAL_STEP: 1169150 | > loss_text_ce: 0.03755272924900055 (0.031759430170076465) | > loss_mel_ce: 2.677565097808838 (nan) | > loss: 0.010774278081953526 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.1370438877709798) | > loader_time: 0.0048 (0.004634085318383783)  --> TIME: 2025-02-26 08:47:07 -- STEP: 24122/36938 -- GLOBAL_STEP: 1169200 | > loss_text_ce: 0.02758828178048134 (0.03175912453372872) | > loss_mel_ce: 2.167295217514038 (nan) | > loss: 0.008709856308996677 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13703766816805518) | > loader_time: 0.0044 (0.004633977414956611)  --> TIME: 2025-02-26 08:47:16 -- STEP: 24172/36938 -- GLOBAL_STEP: 1169250 | > loss_text_ce: 0.034674305468797684 (0.03176078341375637) | > loss_mel_ce: 2.2644059658050537 (nan) | > loss: 0.009123335592448711 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13703344219590596) | > loader_time: 0.0046 (0.0046338639511053084)  --> TIME: 2025-02-26 08:47:26 -- STEP: 24222/36938 -- GLOBAL_STEP: 1169300 | > loss_text_ce: 0.03327065706253052 (0.031758843917164895) | > loss_mel_ce: 3.1898155212402344 (nan) | > loss: 0.01279002521187067 (nan) | > current_lr: 5e-06 | > step_time: 0.169 (0.13703643916393593) | > loader_time: 0.0047 (0.004633885342973828)  --> TIME: 2025-02-26 08:47:35 -- STEP: 24272/36938 -- GLOBAL_STEP: 1169350 | > loss_text_ce: 0.03814276307821274 (0.03175806063866627) | > loss_mel_ce: 1.276503324508667 (nan) | > loss: 0.005216849967837334 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13703171340116305) | > loader_time: 0.0045 (0.004633875704943251)  --> TIME: 2025-02-26 08:47:45 -- STEP: 24322/36938 -- GLOBAL_STEP: 1169400 | > loss_text_ce: 0.039627108722925186 (0.03176082350584329) | > loss_mel_ce: 0.1409759521484375 (nan) | > loss: 0.0007166788564063609 (nan) | > current_lr: 5e-06 | > step_time: 0.1582 (0.1370343900944346) | > loader_time: 0.0047 (0.004634038151785828)  --> TIME: 2025-02-26 08:47:54 -- STEP: 24372/36938 -- GLOBAL_STEP: 1169450 | > loss_text_ce: 0.022139038890600204 (0.031762141716057936) | > loss_mel_ce: 1.383147954940796 (nan) | > loss: 0.005576536059379578 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1370308417395848) | > loader_time: 0.0045 (0.0046339236846722484)  --> TIME: 2025-02-26 08:48:03 -- STEP: 24422/36938 -- GLOBAL_STEP: 1169500 | > loss_text_ce: 0.028271283954381943 (0.03175909330160204) | > loss_mel_ce: 2.48956561088562 (nan) | > loss: 0.009991416707634926 (nan) | > current_lr: 5e-06 | > step_time: 0.1537 (0.13703332198416882) | > loader_time: 0.0046 (0.00463391123527921)  --> TIME: 2025-02-26 08:48:13 -- STEP: 24472/36938 -- GLOBAL_STEP: 1169550 | > loss_text_ce: 0.033725857734680176 (0.03175901437629474) | > loss_mel_ce: 3.1974611282348633 (nan) | > loss: 0.012822170741856098 (nan) | > current_lr: 5e-06 | > step_time: 0.1975 (0.1370376710234687) | > loader_time: 0.0048 (0.004633960808831617)  --> TIME: 2025-02-26 08:48:22 -- STEP: 24522/36938 -- GLOBAL_STEP: 1169600 | > loss_text_ce: 0.03657393157482147 (0.031760060406681306) | > loss_mel_ce: 0.00286865234375 (nan) | > loss: 0.00015651820285711437 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13703093534775576) | > loader_time: 0.0045 (0.004633854083238656)  --> TIME: 2025-02-26 08:48:31 -- STEP: 24572/36938 -- GLOBAL_STEP: 1169650 | > loss_text_ce: 0.029761094599962234 (0.031760944017751626) | > loss_mel_ce: 2.7567734718322754 (nan) | > loss: 0.011057677678763866 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13702754431587447) | > loader_time: 0.0046 (0.004633771001215639)  --> TIME: 2025-02-26 08:48:40 -- STEP: 24622/36938 -- GLOBAL_STEP: 1169700 | > loss_text_ce: 0.031794432550668716 (0.03175970220360533) | > loss_mel_ce: 2.894876718521118 (nan) | > loss: 0.011613775976002216 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13702886338528678) | > loader_time: 0.0048 (0.004633777467505978)  --> TIME: 2025-02-26 08:48:50 -- STEP: 24672/36938 -- GLOBAL_STEP: 1169750 | > loss_text_ce: 0.032243762165308 (0.031759803382385286) | > loss_mel_ce: 3.107361078262329 (nan) | > loss: 0.012458750046789646 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1370259574608423) | > loader_time: 0.0046 (0.004633752269194459)  --> TIME: 2025-02-26 08:48:59 -- STEP: 24722/36938 -- GLOBAL_STEP: 1169800 | > loss_text_ce: 0.03345451131463051 (0.03176311906793336) | > loss_mel_ce: 3.0417709350585938 (nan) | > loss: 0.01220327615737915 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.13703778071481018) | > loader_time: 0.0046 (0.004633800139194591)  --> TIME: 2025-02-26 08:49:09 -- STEP: 24772/36938 -- GLOBAL_STEP: 1169850 | > loss_text_ce: 0.029562771320343018 (0.03176193326397676) | > loss_mel_ce: 2.8679656982421875 (nan) | > loss: 0.01149812899529934 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.13703683316563223) | > loader_time: 0.0047 (0.004633728058065235)  --> TIME: 2025-02-26 08:49:18 -- STEP: 24822/36938 -- GLOBAL_STEP: 1169900 | > loss_text_ce: 0.03354531526565552 (0.031762130080860776) | > loss_mel_ce: 3.121187686920166 (nan) | > loss: 0.012518782168626785 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.1370405248992054) | > loader_time: 0.0046 (0.004633659504257369)  --> TIME: 2025-02-26 08:49:28 -- STEP: 24872/36938 -- GLOBAL_STEP: 1169950 | > loss_text_ce: 0.03781939297914505 (0.03176050471670973) | > loss_mel_ce: 2.383148670196533 (nan) | > loss: 0.009607017040252686 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13703662386159865) | > loader_time: 0.0046 (0.0046335708657887905)  --> TIME: 2025-02-26 08:49:37 -- STEP: 24922/36938 -- GLOBAL_STEP: 1170000 | > loss_text_ce: 0.027587389573454857 (0.03175771709816475) | > loss_mel_ce: 2.1480374336242676 (nan) | > loss: 0.008633431978523731 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13704931743645712) | > loader_time: 0.0046 (0.0046335229539974715) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1170000.pth  --> TIME: 2025-02-26 08:49:52 -- STEP: 24972/36938 -- GLOBAL_STEP: 1170050 | > loss_text_ce: 0.03243402764201164 (0.0317568754659165) | > loss_mel_ce: 1.7974356412887573 (nan) | > loss: 0.007261388003826141 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13704287834929757) | > loader_time: 0.0046 (0.0046336640346208964)  --> TIME: 2025-02-26 08:50:01 -- STEP: 25022/36938 -- GLOBAL_STEP: 1170100 | > loss_text_ce: 0.012631532736122608 (0.03175923529951798) | > loss_mel_ce: 2.5485568046569824 (nan) | > loss: 0.010163445957005024 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13703621384737527) | > loader_time: 0.0045 (0.004633658329232918)  --> TIME: 2025-02-26 08:50:10 -- STEP: 25072/36938 -- GLOBAL_STEP: 1170150 | > loss_text_ce: 0.028060181066393852 (0.031760069775392655) | > loss_mel_ce: 2.4218058586120605 (nan) | > loss: 0.009721691720187664 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13702822855573288) | > loader_time: 0.0046 (0.004633540645403415)  --> TIME: 2025-02-26 08:50:19 -- STEP: 25122/36938 -- GLOBAL_STEP: 1170200 | > loss_text_ce: 0.03496646508574486 (0.03175886498376259) | > loss_mel_ce: 2.9069721698760986 (nan) | > loss: 0.011674360372126102 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13701980385029827) | > loader_time: 0.0045 (0.004633422604355803)  --> TIME: 2025-02-26 08:50:28 -- STEP: 25172/36938 -- GLOBAL_STEP: 1170250 | > loss_text_ce: 0.027393506839871407 (0.03175705419333158) | > loss_mel_ce: 2.8154497146606445 (nan) | > loss: 0.011281125247478485 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.13701624657568617) | > loader_time: 0.0046 (0.004633333409095883)  --> TIME: 2025-02-26 08:50:37 -- STEP: 25222/36938 -- GLOBAL_STEP: 1170300 | > loss_text_ce: 0.024873411282896996 (0.03175646389139163) | > loss_mel_ce: 0.070953369140625 (nan) | > loss: 0.0003802650317084044 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13701492175075827) | > loader_time: 0.0046 (0.004633346582118875)  --> TIME: 2025-02-26 08:50:47 -- STEP: 25272/36938 -- GLOBAL_STEP: 1170350 | > loss_text_ce: 0.025457849726080894 (0.03175624842166186) | > loss_mel_ce: 2.5323598384857178 (nan) | > loss: 0.010150071233510971 (nan) | > current_lr: 5e-06 | > step_time: 0.1517 (0.13701482843130044) | > loader_time: 0.0046 (0.004633456015745373)  --> TIME: 2025-02-26 08:50:56 -- STEP: 25322/36938 -- GLOBAL_STEP: 1170400 | > loss_text_ce: 0.024732742458581924 (0.03175402457484036) | > loss_mel_ce: 2.8593966960906982 (nan) | > loss: 0.011444958858191967 (nan) | > current_lr: 5e-06 | > step_time: 0.1896 (0.1370152540563189) | > loader_time: 0.0049 (0.004633483987652636)  --> TIME: 2025-02-26 08:51:05 -- STEP: 25372/36938 -- GLOBAL_STEP: 1170450 | > loss_text_ce: 0.03446439281105995 (0.031753445285983915) | > loss_mel_ce: 2.535975217819214 (nan) | > loss: 0.010200157761573792 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13701051213322948) | > loader_time: 0.0045 (0.0046334468790283434)  --> TIME: 2025-02-26 08:51:15 -- STEP: 25422/36938 -- GLOBAL_STEP: 1170500 | > loss_text_ce: 0.03387462720274925 (0.0317523688944408) | > loss_mel_ce: 1.4212062358856201 (nan) | > loss: 0.005774130579084158 (nan) | > current_lr: 5e-06 | > step_time: 0.1714 (0.13701083235095055) | > loader_time: 0.0048 (0.004633451134597175)  --> TIME: 2025-02-26 08:51:24 -- STEP: 25472/36938 -- GLOBAL_STEP: 1170550 | > loss_text_ce: 0.029635250568389893 (0.03175272656045699) | > loss_mel_ce: 2.5081369876861572 (nan) | > loss: 0.010070525109767914 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.1369994838052808) | > loader_time: 0.0046 (0.004633381335654474)  --> TIME: 2025-02-26 08:51:33 -- STEP: 25522/36938 -- GLOBAL_STEP: 1170600 | > loss_text_ce: 0.032161254435777664 (0.03174885747249702) | > loss_mel_ce: 2.5809590816497803 (nan) | > loss: 0.010369525291025639 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13700040430589558) | > loader_time: 0.0048 (0.004633370560077436)  --> TIME: 2025-02-26 08:51:43 -- STEP: 25572/36938 -- GLOBAL_STEP: 1170650 | > loss_text_ce: 0.03626561909914017 (0.03174832968506615) | > loss_mel_ce: 2.974965810775757 (nan) | > loss: 0.011949331499636173 (nan) | > current_lr: 5e-06 | > step_time: 0.1721 (0.13700633279859187) | > loader_time: 0.0048 (0.0046333796389129685)  --> TIME: 2025-02-26 08:51:52 -- STEP: 25622/36938 -- GLOBAL_STEP: 1170700 | > loss_text_ce: 0.03954325616359711 (0.03174604265905198) | > loss_mel_ce: 2.964357852935791 (nan) | > loss: 0.011920242570340633 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13700524179320006) | > loader_time: 0.0046 (0.0046335114182861166)  --> TIME: 2025-02-26 08:52:01 -- STEP: 25672/36938 -- GLOBAL_STEP: 1170750 | > loss_text_ce: 0.025708405300974846 (0.03174337956814377) | > loss_mel_ce: 2.599783420562744 (nan) | > loss: 0.010418619029223919 (nan) | > current_lr: 5e-06 | > step_time: 0.1807 (0.13700054395083655) | > loader_time: 0.0049 (0.00463355763302524)  --> TIME: 2025-02-26 08:52:10 -- STEP: 25722/36938 -- GLOBAL_STEP: 1170800 | > loss_text_ce: 0.034263089299201965 (0.031743409477789065) | > loss_mel_ce: 2.694300889968872 (nan) | > loss: 0.010827635414898396 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13699483655491013) | > loader_time: 0.0046 (0.00463353756120888)  --> TIME: 2025-02-26 08:52:20 -- STEP: 25772/36938 -- GLOBAL_STEP: 1170850 | > loss_text_ce: 0.032605621963739395 (0.03174500341181565) | > loss_mel_ce: 3.1125876903533936 (nan) | > loss: 0.01248092669993639 (nan) | > current_lr: 5e-06 | > step_time: 0.1879 (0.1369970501181538) | > loader_time: 0.0048 (0.004633530176483358)  --> TIME: 2025-02-26 08:52:29 -- STEP: 25822/36938 -- GLOBAL_STEP: 1170900 | > loss_text_ce: 0.0208431389182806 (0.03174442935502154) | > loss_mel_ce: 0.9206215143203735 (nan) | > loss: 0.0037359711714088917 (nan) | > current_lr: 5e-06 | > step_time: 0.1118 (0.13699205228632538) | > loader_time: 0.0046 (0.004633577693008248)  --> TIME: 2025-02-26 08:52:38 -- STEP: 25872/36938 -- GLOBAL_STEP: 1170950 | > loss_text_ce: 0.03542717546224594 (0.031745815228604216) | > loss_mel_ce: 2.8313241004943848 (nan) | > loss: 0.011375997215509415 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13698599026316685) | > loader_time: 0.0045 (0.004633636849119735)  --> TIME: 2025-02-26 08:52:47 -- STEP: 25922/36938 -- GLOBAL_STEP: 1171000 | > loss_text_ce: 0.028785739094018936 (0.03174484813273038) | > loss_mel_ce: 1.7172037363052368 (nan) | > loss: 0.006928530056029558 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13697412517848692) | > loader_time: 0.0045 (0.00463365369828691)  --> TIME: 2025-02-26 08:52:56 -- STEP: 25972/36938 -- GLOBAL_STEP: 1171050 | > loss_text_ce: 0.028382914140820503 (0.03174530646654608) | > loss_mel_ce: 2.421394109725952 (nan) | > loss: 0.009721338748931885 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1369754970670388) | > loader_time: 0.0047 (0.004633699729522318)  --> TIME: 2025-02-26 08:53:06 -- STEP: 26022/36938 -- GLOBAL_STEP: 1171100 | > loss_text_ce: 0.052350886166095734 (0.03174452719845352) | > loss_mel_ce: 2.533902883529663 (nan) | > loss: 0.010262913070619106 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13697824916285464) | > loader_time: 0.0045 (0.004633822243930726)  --> TIME: 2025-02-26 08:53:15 -- STEP: 26072/36938 -- GLOBAL_STEP: 1171150 | > loss_text_ce: 0.029899010434746742 (0.03174231276803439) | > loss_mel_ce: 2.0833208560943604 (nan) | > loss: 0.00838579423725605 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13698596820285552) | > loader_time: 0.0046 (0.004634015653059542)  --> TIME: 2025-02-26 08:53:25 -- STEP: 26122/36938 -- GLOBAL_STEP: 1171200 | > loss_text_ce: 0.0388057567179203 (0.0317444929498838) | > loss_mel_ce: 2.008312940597534 (nan) | > loss: 0.0081234872341156 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.136983596496401) | > loader_time: 0.0046 (0.004634222642229615)  --> TIME: 2025-02-26 08:53:34 -- STEP: 26172/36938 -- GLOBAL_STEP: 1171250 | > loss_text_ce: 0.03954456374049187 (0.03174258724301848) | > loss_mel_ce: 3.495192289352417 (nan) | > loss: 0.014026734046638012 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1369919505265807) | > loader_time: 0.0045 (0.004634268127521896)  --> TIME: 2025-02-26 08:53:44 -- STEP: 26222/36938 -- GLOBAL_STEP: 1171300 | > loss_text_ce: 0.02471751719713211 (0.03174338222013014) | > loss_mel_ce: 1.7096099853515625 (nan) | > loss: 0.006882252637296915 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13699575791087265) | > loader_time: 0.0046 (0.004634280325155182)  --> TIME: 2025-02-26 08:53:53 -- STEP: 26272/36938 -- GLOBAL_STEP: 1171350 | > loss_text_ce: 0.032701339572668076 (0.031744742060760237) | > loss_mel_ce: 2.3746962547302246 (nan) | > loss: 0.009553165175020695 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.136995133398221) | > loader_time: 0.0047 (0.004634260886261466)  --> TIME: 2025-02-26 08:54:02 -- STEP: 26322/36938 -- GLOBAL_STEP: 1171400 | > loss_text_ce: 0.04041389375925064 (0.03174451182834677) | > loss_mel_ce: 2.423572301864624 (nan) | > loss: 0.00977772381156683 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.1369961963291543) | > loader_time: 0.0047 (0.004634285777474392)  --> TIME: 2025-02-26 08:54:12 -- STEP: 26372/36938 -- GLOBAL_STEP: 1171450 | > loss_text_ce: 0.03251677379012108 (0.03174716732015554) | > loss_mel_ce: 2.740131139755249 (nan) | > loss: 0.011002571322023869 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.13699989169316185) | > loader_time: 0.0047 (0.004634318276889505)  --> TIME: 2025-02-26 08:54:21 -- STEP: 26422/36938 -- GLOBAL_STEP: 1171500 | > loss_text_ce: 0.028357701376080513 (0.03174500692896357) | > loss_mel_ce: 2.4314815998077393 (nan) | > loss: 0.009761267341673374 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13699804291881426) | > loader_time: 0.0048 (0.0046343914575124445)  --> TIME: 2025-02-26 08:54:30 -- STEP: 26472/36938 -- GLOBAL_STEP: 1171550 | > loss_text_ce: 0.03727744147181511 (0.03174529788845686) | > loss_mel_ce: 1.1714578866958618 (nan) | > loss: 0.004796569235622883 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.136993832676819) | > loader_time: 0.0046 (0.004634505917421487)  --> TIME: 2025-02-26 08:54:40 -- STEP: 26522/36938 -- GLOBAL_STEP: 1171600 | > loss_text_ce: 0.023505529388785362 (0.03174684674556846) | > loss_mel_ce: 2.6559152603149414 (nan) | > loss: 0.010632622055709362 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13699809995750026) | > loader_time: 0.0046 (0.004634572184739407)  --> TIME: 2025-02-26 08:54:49 -- STEP: 26572/36938 -- GLOBAL_STEP: 1171650 | > loss_text_ce: 0.03387590870261192 (0.031744888344232905) | > loss_mel_ce: 2.9355685710906982 (nan) | > loss: 0.011783510446548462 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.1370052429956067) | > loader_time: 0.0046 (0.004634593788550693)  --> TIME: 2025-02-26 08:54:58 -- STEP: 26622/36938 -- GLOBAL_STEP: 1171700 | > loss_text_ce: 0.04213341325521469 (0.03174508323127693) | > loss_mel_ce: 2.3124890327453613 (nan) | > loss: 0.009343740530312061 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13700610182070522) | > loader_time: 0.0045 (0.00463460497633683)  --> TIME: 2025-02-26 08:55:08 -- STEP: 26672/36938 -- GLOBAL_STEP: 1171750 | > loss_text_ce: 0.029506027698516846 (0.03174419949349693) | > loss_mel_ce: 2.225067615509033 (nan) | > loss: 0.008946721442043781 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1370012577897996) | > loader_time: 0.0046 (0.004634598503587651)  --> TIME: 2025-02-26 08:55:17 -- STEP: 26722/36938 -- GLOBAL_STEP: 1171800 | > loss_text_ce: 0.03137876093387604 (0.03174217876237782) | > loss_mel_ce: 3.1375162601470947 (nan) | > loss: 0.012574980966746807 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.1370027124698836) | > loader_time: 0.0047 (0.004634596257409302)  --> TIME: 2025-02-26 08:55:26 -- STEP: 26772/36938 -- GLOBAL_STEP: 1171850 | > loss_text_ce: 0.02897346392273903 (0.03174085236213253) | > loss_mel_ce: 2.849604606628418 (nan) | > loss: 0.011422930285334587 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13699170046281509) | > loader_time: 0.0045 (0.004634560285511552)  --> TIME: 2025-02-26 08:55:35 -- STEP: 26822/36938 -- GLOBAL_STEP: 1171900 | > loss_text_ce: 0.02904418110847473 (0.031741913522382) | > loss_mel_ce: 0.0072883605025708675 (nan) | > loss: 0.0001441767526557669 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13698772287734948) | > loader_time: 0.0045 (0.004634628208075948)  --> TIME: 2025-02-26 08:55:45 -- STEP: 26872/36938 -- GLOBAL_STEP: 1171950 | > loss_text_ce: 0.023090621456503868 (0.03174003123308683) | > loss_mel_ce: 2.7499866485595703 (nan) | > loss: 0.011004275642335415 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.1369881872514694) | > loader_time: 0.0047 (0.004634739450132207)  --> TIME: 2025-02-26 08:55:54 -- STEP: 26922/36938 -- GLOBAL_STEP: 1172000 | > loss_text_ce: 0.03107895329594612 (0.03174046894288322) | > loss_mel_ce: 2.490466356277466 (nan) | > loss: 0.010006133466959 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.1369821197006603) | > loader_time: 0.0045 (0.004634734567778684)  --> TIME: 2025-02-26 08:56:03 -- STEP: 26972/36938 -- GLOBAL_STEP: 1172050 | > loss_text_ce: 0.03449028730392456 (0.03174260862117957) | > loss_mel_ce: 2.8164236545562744 (nan) | > loss: 0.011313151568174362 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.13698049711930005) | > loader_time: 0.0049 (0.004634906502058562)  --> TIME: 2025-02-26 08:56:13 -- STEP: 27022/36938 -- GLOBAL_STEP: 1172100 | > loss_text_ce: 0.030460869893431664 (0.03174161108426607) | > loss_mel_ce: 3.3109540939331055 (nan) | > loss: 0.013259584084153175 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1369816946706288) | > loader_time: 0.0046 (0.004635050536596105)  --> TIME: 2025-02-26 08:56:22 -- STEP: 27072/36938 -- GLOBAL_STEP: 1172150 | > loss_text_ce: 0.03849916532635689 (0.031740855355348305) | > loss_mel_ce: 2.9281985759735107 (nan) | > loss: 0.011772610247135162 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13697745329557262) | > loader_time: 0.0045 (0.00463508622584334)  --> TIME: 2025-02-26 08:56:32 -- STEP: 27122/36938 -- GLOBAL_STEP: 1172200 | > loss_text_ce: 0.019616706296801567 (0.031743104030160035) | > loss_mel_ce: 2.860884428024292 (nan) | > loss: 0.011430560611188412 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.13698771134228754) | > loader_time: 0.0045 (0.004635271504955427)  --> TIME: 2025-02-26 08:56:41 -- STEP: 27172/36938 -- GLOBAL_STEP: 1172250 | > loss_text_ce: 0.02486976608633995 (0.031742378182358026) | > loss_mel_ce: 2.054844379425049 (nan) | > loss: 0.008252833969891071 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13698626127893027) | > loader_time: 0.0046 (0.0046353095254823866)  --> TIME: 2025-02-26 08:56:50 -- STEP: 27222/36938 -- GLOBAL_STEP: 1172300 | > loss_text_ce: 0.03194161504507065 (0.03174033693688197) | > loss_mel_ce: 2.758037805557251 (nan) | > loss: 0.011071347631514072 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13698854009196515) | > loader_time: 0.0045 (0.004635402268363575)  --> TIME: 2025-02-26 08:57:00 -- STEP: 27272/36938 -- GLOBAL_STEP: 1172350 | > loss_text_ce: 0.03532428666949272 (0.03174079671894454) | > loss_mel_ce: 3.407451868057251 (nan) | > loss: 0.013661811128258705 (nan) | > current_lr: 5e-06 | > step_time: 0.1813 (0.13699673117518446) | > loader_time: 0.0048 (0.004635413709222572)  --> TIME: 2025-02-26 08:57:09 -- STEP: 27322/36938 -- GLOBAL_STEP: 1172400 | > loss_text_ce: 0.0331839956343174 (0.03174284069118068) | > loss_mel_ce: 2.9977405071258545 (nan) | > loss: 0.012027478776872158 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13698975440044098) | > loader_time: 0.0049 (0.004635409732556417)  --> TIME: 2025-02-26 08:57:18 -- STEP: 27372/36938 -- GLOBAL_STEP: 1172450 | > loss_text_ce: 0.04205108433961868 (0.031742943409026586) | > loss_mel_ce: 2.637225389480591 (nan) | > loss: 0.010632050223648548 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1369844573066367) | > loader_time: 0.0045 (0.004635347454899449)  --> TIME: 2025-02-26 08:57:28 -- STEP: 27422/36938 -- GLOBAL_STEP: 1172500 | > loss_text_ce: 0.02604815736413002 (0.031741515459631275) | > loss_mel_ce: 3.1241068840026855 (nan) | > loss: 0.012500615790486336 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.1369866693178974) | > loader_time: 0.0046 (0.004635348021614605)  --> TIME: 2025-02-26 08:57:37 -- STEP: 27472/36938 -- GLOBAL_STEP: 1172550 | > loss_text_ce: 0.03912103921175003 (0.03174295874433779) | > loss_mel_ce: 1.9999995231628418 (nan) | > loss: 0.008091749623417854 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13698386889109446) | > loader_time: 0.0045 (0.004635280285661987)  --> TIME: 2025-02-26 08:57:46 -- STEP: 27522/36938 -- GLOBAL_STEP: 1172600 | > loss_text_ce: 0.024554090574383736 (0.031744203365658405) | > loss_mel_ce: 1.7274168729782104 (nan) | > loss: 0.006952266208827496 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.1369891685937416) | > loader_time: 0.0048 (0.004635261645558613)  --> TIME: 2025-02-26 08:57:56 -- STEP: 27572/36938 -- GLOBAL_STEP: 1172650 | > loss_text_ce: 0.0300205796957016 (0.03174492805557435) | > loss_mel_ce: 3.0625526905059814 (nan) | > loss: 0.012272116728127003 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.1369965286658184) | > loader_time: 0.0047 (0.004635351153497835)  --> TIME: 2025-02-26 08:58:05 -- STEP: 27622/36938 -- GLOBAL_STEP: 1172700 | > loss_text_ce: 0.032268691807985306 (0.031747869148944584) | > loss_mel_ce: 3.467017650604248 (nan) | > loss: 0.013886057771742344 (nan) | > current_lr: 5e-06 | > step_time: 0.1679 (0.13699674021109304) | > loader_time: 0.0046 (0.004635444126608386)  --> TIME: 2025-02-26 08:58:15 -- STEP: 27672/36938 -- GLOBAL_STEP: 1172750 | > loss_text_ce: 0.041004374623298645 (0.03174585495230316) | > loss_mel_ce: 2.591886281967163 (nan) | > loss: 0.010447979904711246 (nan) | > current_lr: 5e-06 | > step_time: 0.1765 (0.13699837114708244) | > loader_time: 0.0048 (0.00463544124811157)  --> TIME: 2025-02-26 08:58:24 -- STEP: 27722/36938 -- GLOBAL_STEP: 1172800 | > loss_text_ce: 0.02543732151389122 (0.03174550615872735) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.1370055356864057) | > loader_time: 0.0046 (0.004635473950997514)  --> TIME: 2025-02-26 08:58:34 -- STEP: 27772/36938 -- GLOBAL_STEP: 1172850 | > loss_text_ce: 0.035070065408945084 (0.03174607993410634) | > loss_mel_ce: 0.0177001953125 (nan) | > loss: 0.00020940581453032792 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.1370119503728083) | > loader_time: 0.0049 (0.004635495796475598)  --> TIME: 2025-02-26 08:58:43 -- STEP: 27822/36938 -- GLOBAL_STEP: 1172900 | > loss_text_ce: 0.028686320409178734 (0.03174876663307666) | > loss_mel_ce: 2.7281346321105957 (nan) | > loss: 0.010939765721559525 (nan) | > current_lr: 5e-06 | > step_time: 0.1492 (0.13702455928466015) | > loader_time: 0.0046 (0.0046355140071230394)  --> TIME: 2025-02-26 08:58:53 -- STEP: 27872/36938 -- GLOBAL_STEP: 1172950 | > loss_text_ce: 0.025213858112692833 (0.03174654755269039) | > loss_mel_ce: 2.6308250427246094 (nan) | > loss: 0.01053983811289072 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13702372439316843) | > loader_time: 0.0046 (0.004635486011869204)  --> TIME: 2025-02-26 08:59:02 -- STEP: 27922/36938 -- GLOBAL_STEP: 1173000 | > loss_text_ce: 0.03514867648482323 (0.03174868739202692) | > loss_mel_ce: 2.002049207687378 (nan) | > loss: 0.00808411929756403 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13702554211269155) | > loader_time: 0.0046 (0.004635449501293366)  --> TIME: 2025-02-26 08:59:12 -- STEP: 27972/36938 -- GLOBAL_STEP: 1173050 | > loss_text_ce: 0.03684374317526817 (0.03174914370827542) | > loss_mel_ce: 2.8385205268859863 (nan) | > loss: 0.011410176753997803 (nan) | > current_lr: 5e-06 | > step_time: 0.1615 (0.1370293640040301) | > loader_time: 0.0047 (0.004635412669498913)  --> TIME: 2025-02-26 08:59:21 -- STEP: 28022/36938 -- GLOBAL_STEP: 1173100 | > loss_text_ce: 0.027561062946915627 (0.031748325845072596) | > loss_mel_ce: 1.028093934059143 (nan) | > loss: 0.004189107567071915 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13702377138177976) | > loader_time: 0.0045 (0.0046354135586544965)  --> TIME: 2025-02-26 08:59:30 -- STEP: 28072/36938 -- GLOBAL_STEP: 1173150 | > loss_text_ce: 0.027114497497677803 (0.03174596315258587) | > loss_mel_ce: 3.767463207244873 (nan) | > loss: 0.015057848766446114 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13701605309309453) | > loader_time: 0.0044 (0.004635469395063259)  --> TIME: 2025-02-26 08:59:39 -- STEP: 28122/36938 -- GLOBAL_STEP: 1173200 | > loss_text_ce: 0.031620509922504425 (0.03174623778739934) | > loss_mel_ce: 2.270862579345703 (nan) | > loss: 0.009136837907135487 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13701480329057902) | > loader_time: 0.0045 (0.004635441906039655)  --> TIME: 2025-02-26 08:59:48 -- STEP: 28172/36938 -- GLOBAL_STEP: 1173250 | > loss_text_ce: 0.032668910920619965 (0.031744473133790245) | > loss_mel_ce: 1.968161702156067 (nan) | > loss: 0.007939804345369339 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.1370114727858434) | > loader_time: 0.0045 (0.004635385249667473)  --> TIME: 2025-02-26 08:59:58 -- STEP: 28222/36938 -- GLOBAL_STEP: 1173300 | > loss_text_ce: 0.02847391553223133 (0.031745505148094946) | > loss_mel_ce: 2.628704309463501 (nan) | > loss: 0.010544358752667904 (nan) | > current_lr: 5e-06 | > step_time: 0.1711 (0.13701092515117788) | > loader_time: 0.005 (0.004635395761097493)  --> TIME: 2025-02-26 09:00:07 -- STEP: 28272/36938 -- GLOBAL_STEP: 1173350 | > loss_text_ce: 0.03080625645816326 (0.03174717789297148) | > loss_mel_ce: 3.1927566528320312 (nan) | > loss: 0.012791917659342289 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13700604731407814) | > loader_time: 0.0046 (0.004635469381864572)  --> TIME: 2025-02-26 09:00:16 -- STEP: 28322/36938 -- GLOBAL_STEP: 1173400 | > loss_text_ce: 0.029406005516648293 (0.03174434325343251) | > loss_mel_ce: 2.9215762615203857 (nan) | > loss: 0.011710247956216335 (nan) | > current_lr: 5e-06 | > step_time: 0.1523 (0.13699307186473392) | > loader_time: 0.0047 (0.004635438324073218)  --> TIME: 2025-02-26 09:00:25 -- STEP: 28372/36938 -- GLOBAL_STEP: 1173450 | > loss_text_ce: 0.027791913598775864 (0.03174335951296504) | > loss_mel_ce: 2.093334674835205 (nan) | > loss: 0.008417169563472271 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13699392171230756) | > loader_time: 0.0045 (0.004635437190675556)  --> TIME: 2025-02-26 09:00:34 -- STEP: 28422/36938 -- GLOBAL_STEP: 1173500 | > loss_text_ce: 0.03142009675502777 (0.03174198310152246) | > loss_mel_ce: 0.7646484375 (nan) | > loss: 0.003159002400934696 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13699504596142992) | > loader_time: 0.0048 (0.004635417900114993)  --> TIME: 2025-02-26 09:00:44 -- STEP: 28472/36938 -- GLOBAL_STEP: 1173550 | > loss_text_ce: 0.029099252074956894 (0.03174077331854716) | > loss_mel_ce: 3.084070920944214 (nan) | > loss: 0.012353850528597832 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.13698937150828447) | > loader_time: 0.0048 (0.004635360718912289)  --> TIME: 2025-02-26 09:00:53 -- STEP: 28522/36938 -- GLOBAL_STEP: 1173600 | > loss_text_ce: 0.03824993222951889 (0.031740039982772854) | > loss_mel_ce: 1.8304697275161743 (nan) | > loss: 0.007415554951876402 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13698455503466545) | > loader_time: 0.0048 (0.004635457370301465)  --> TIME: 2025-02-26 09:01:02 -- STEP: 28572/36938 -- GLOBAL_STEP: 1173650 | > loss_text_ce: 0.03662364184856415 (0.03173988912503369) | > loss_mel_ce: 2.523455858230591 (nan) | > loss: 0.01015904638916254 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.1369919845841935) | > loader_time: 0.0047 (0.004635518987055654)  --> TIME: 2025-02-26 09:01:12 -- STEP: 28622/36938 -- GLOBAL_STEP: 1173700 | > loss_text_ce: 0.026316294446587563 (0.03174140883964875) | > loss_mel_ce: 2.8835318088531494 (nan) | > loss: 0.011547017842531204 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.13699646150530198) | > loader_time: 0.0047 (0.004635626086398395)  --> TIME: 2025-02-26 09:01:21 -- STEP: 28672/36938 -- GLOBAL_STEP: 1173750 | > loss_text_ce: 0.03295435011386871 (0.0317418357246557) | > loss_mel_ce: 3.0214173793792725 (nan) | > loss: 0.012120523490011692 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13699739517843604) | > loader_time: 0.0047 (0.00463560710029146)  --> TIME: 2025-02-26 09:01:31 -- STEP: 28722/36938 -- GLOBAL_STEP: 1173800 | > loss_text_ce: 0.033878665417432785 (0.03174386570315368) | > loss_mel_ce: 3.116868257522583 (nan) | > loss: 0.012502964586019516 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.13700105379272082) | > loader_time: 0.0047 (0.004635626970414273)  --> TIME: 2025-02-26 09:01:40 -- STEP: 28772/36938 -- GLOBAL_STEP: 1173850 | > loss_text_ce: 0.03217237815260887 (0.03174335802365376) | > loss_mel_ce: 3.2559261322021484 (nan) | > loss: 0.013048010878264904 (nan) | > current_lr: 5e-06 | > step_time: 0.1846 (0.13699993886852152) | > loader_time: 0.005 (0.004635588393230606)  --> TIME: 2025-02-26 09:01:49 -- STEP: 28822/36938 -- GLOBAL_STEP: 1173900 | > loss_text_ce: 0.031433384865522385 (0.03174282568257796) | > loss_mel_ce: 2.387230396270752 (nan) | > loss: 0.009597872383892536 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1369954021937741) | > loader_time: 0.0046 (0.004635616771948692)  --> TIME: 2025-02-26 09:01:58 -- STEP: 28872/36938 -- GLOBAL_STEP: 1173950 | > loss_text_ce: 0.03791207820177078 (0.03174147979038583) | > loss_mel_ce: 1.4663581848144531 (nan) | > loss: 0.005969326943159103 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13699280245378756) | > loader_time: 0.0046 (0.004635634903565589)  --> TIME: 2025-02-26 09:02:08 -- STEP: 28922/36938 -- GLOBAL_STEP: 1174000 | > loss_text_ce: 0.042427945882081985 (0.0317433071336267) | > loss_mel_ce: 2.7706282138824463 (nan) | > loss: 0.011162921786308289 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13698606402553637) | > loader_time: 0.0046 (0.00463562948675245)  --> TIME: 2025-02-26 09:02:17 -- STEP: 28972/36938 -- GLOBAL_STEP: 1174050 | > loss_text_ce: 0.0328088253736496 (0.03174396298610912) | > loss_mel_ce: 2.567215919494629 (nan) | > loss: 0.010317559354007244 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13698420523115884) | > loader_time: 0.0046 (0.0046356223604881045)  --> TIME: 2025-02-26 09:02:26 -- STEP: 29022/36938 -- GLOBAL_STEP: 1174100 | > loss_text_ce: 0.03710021823644638 (0.03174421205203678) | > loss_mel_ce: 1.508139967918396 (nan) | > loss: 0.006131905596703291 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13698092310230292) | > loader_time: 0.0045 (0.00463560321544454)  --> TIME: 2025-02-26 09:02:36 -- STEP: 29072/36938 -- GLOBAL_STEP: 1174150 | > loss_text_ce: 0.03262582793831825 (0.03174116280199386) | > loss_mel_ce: 2.7441585063934326 (nan) | > loss: 0.011018986813724041 (nan) | > current_lr: 5e-06 | > step_time: 0.1563 (0.1369835161365645) | > loader_time: 0.0047 (0.004635560017204195)  --> TIME: 2025-02-26 09:02:45 -- STEP: 29122/36938 -- GLOBAL_STEP: 1174200 | > loss_text_ce: 0.02619013749063015 (0.03174059976047249) | > loss_mel_ce: 2.910889148712158 (nan) | > loss: 0.011655076406896114 (nan) | > current_lr: 5e-06 | > step_time: 0.1709 (0.13698029144805288) | > loader_time: 0.005 (0.004635569739985241)  --> TIME: 2025-02-26 09:02:54 -- STEP: 29172/36938 -- GLOBAL_STEP: 1174250 | > loss_text_ce: 0.02857021801173687 (0.031740393739797514) | > loss_mel_ce: 2.2205591201782227 (nan) | > loss: 0.008925116620957851 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13697430218065368) | > loader_time: 0.0045 (0.00463554609235417)  --> TIME: 2025-02-26 09:03:04 -- STEP: 29222/36938 -- GLOBAL_STEP: 1174300 | > loss_text_ce: 0.023159055039286613 (0.031741008830474614) | > loss_mel_ce: 2.6829426288604736 (nan) | > loss: 0.010738499462604523 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13697904613944706) | > loader_time: 0.0045 (0.0046355582941452434)  --> TIME: 2025-02-26 09:03:13 -- STEP: 29272/36938 -- GLOBAL_STEP: 1174350 | > loss_text_ce: 0.033404186367988586 (0.031740683044221216) | > loss_mel_ce: 0.0164794921875 (nan) | > loss: 0.00019795111438725144 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.1369752131237555) | > loader_time: 0.0046 (0.004635545652920722)  --> TIME: 2025-02-26 09:03:22 -- STEP: 29322/36938 -- GLOBAL_STEP: 1174400 | > loss_text_ce: 0.029409391805529594 (0.03174128203317616) | > loss_mel_ce: 0.0247802734375 (nan) | > loss: 0.0002150383807020262 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1369734654020995) | > loader_time: 0.0045 (0.004635509938239481)  --> TIME: 2025-02-26 09:03:32 -- STEP: 29372/36938 -- GLOBAL_STEP: 1174450 | > loss_text_ce: 0.02771715819835663 (0.03174071661734446) | > loss_mel_ce: 2.9717941284179688 (nan) | > loss: 0.011902823112905025 (nan) | > current_lr: 5e-06 | > step_time: 0.1771 (0.13697881845854487) | > loader_time: 0.0048 (0.004635468784866338)  --> TIME: 2025-02-26 09:03:41 -- STEP: 29422/36938 -- GLOBAL_STEP: 1174500 | > loss_text_ce: 0.050318941473960876 (0.031741876033302245) | > loss_mel_ce: 2.371242046356201 (nan) | > loss: 0.009609369561076164 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.1369785046190107) | > loader_time: 0.0044 (0.004635443702673395)  --> TIME: 2025-02-26 09:03:51 -- STEP: 29472/36938 -- GLOBAL_STEP: 1174550 | > loss_text_ce: 0.03279596567153931 (0.03174319349129635) | > loss_mel_ce: 2.7411458492279053 (nan) | > loss: 0.011007705703377724 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.13698076581366003) | > loader_time: 0.0047 (0.004635416715528253)  --> TIME: 2025-02-26 09:04:00 -- STEP: 29522/36938 -- GLOBAL_STEP: 1174600 | > loss_text_ce: 0.030145691707730293 (0.03174349688535276) | > loss_mel_ce: 3.073895215988159 (nan) | > loss: 0.012317623011767864 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.13698162791706509) | > loader_time: 0.0046 (0.004635391031191234)  --> TIME: 2025-02-26 09:04:10 -- STEP: 29572/36938 -- GLOBAL_STEP: 1174650 | > loss_text_ce: 0.03498886153101921 (0.03174355590862036) | > loss_mel_ce: 3.1803133487701416 (nan) | > loss: 0.012759136967360973 (nan) | > current_lr: 5e-06 | > step_time: 0.1853 (0.1369909471351626) | > loader_time: 0.0049 (0.004635402342954878)  --> TIME: 2025-02-26 09:04:19 -- STEP: 29622/36938 -- GLOBAL_STEP: 1174700 | > loss_text_ce: 0.04019017517566681 (0.03174477880278629) | > loss_mel_ce: 2.4992406368255615 (nan) | > loss: 0.010077106766402721 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13698551234834527) | > loader_time: 0.0046 (0.0046353350209800415)  --> TIME: 2025-02-26 09:04:28 -- STEP: 29672/36938 -- GLOBAL_STEP: 1174750 | > loss_text_ce: 0.032855600118637085 (0.03174472987094663) | > loss_mel_ce: 1.8697911500930786 (nan) | > loss: 0.007550186011940241 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13699127832137853) | > loader_time: 0.0045 (0.004635305241067716)  --> TIME: 2025-02-26 09:04:38 -- STEP: 29722/36938 -- GLOBAL_STEP: 1174800 | > loss_text_ce: 0.03331410139799118 (0.03174435046125656) | > loss_mel_ce: 3.314023494720459 (nan) | > loss: 0.013283086009323597 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13699333731607227) | > loader_time: 0.0046 (0.004635276235166287)  --> TIME: 2025-02-26 09:04:47 -- STEP: 29772/36938 -- GLOBAL_STEP: 1174850 | > loss_text_ce: 0.030384322628378868 (0.03174248160818799) | > loss_mel_ce: 3.02382493019104 (nan) | > loss: 0.012119879014790058 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13699940596154736) | > loader_time: 0.0046 (0.004635238237443781)  --> TIME: 2025-02-26 09:04:57 -- STEP: 29822/36938 -- GLOBAL_STEP: 1174900 | > loss_text_ce: 0.0313275121152401 (0.031742688535215884) | > loss_mel_ce: 2.6960113048553467 (nan) | > loss: 0.010822773911058903 (nan) | > current_lr: 5e-06 | > step_time: 0.1736 (0.1369935899071511) | > loader_time: 0.0047 (0.00463511234525465)  --> TIME: 2025-02-26 09:05:06 -- STEP: 29872/36938 -- GLOBAL_STEP: 1174950 | > loss_text_ce: 0.02198704704642296 (0.03174048992129909) | > loss_mel_ce: 2.4326171875 (nan) | > loss: 0.009740493260324001 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13698352209277875) | > loader_time: 0.0045 (0.004634996962917652)  --> TIME: 2025-02-26 09:05:15 -- STEP: 29922/36938 -- GLOBAL_STEP: 1175000 | > loss_text_ce: 0.0340951643884182 (0.03173941855020874) | > loss_mel_ce: 3.248462438583374 (nan) | > loss: 0.013026022352278233 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13699263989642507) | > loader_time: 0.0046 (0.004634979582193055)  --> TIME: 2025-02-26 09:05:25 -- STEP: 29972/36938 -- GLOBAL_STEP: 1175050 | > loss_text_ce: 0.026235874742269516 (0.031738471460641855) | > loss_mel_ce: 1.782012939453125 (nan) | > loss: 0.007175590842962265 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.13699971126106847) | > loader_time: 0.0047 (0.004634993091915711)  --> TIME: 2025-02-26 09:05:34 -- STEP: 30022/36938 -- GLOBAL_STEP: 1175100 | > loss_text_ce: 0.023396383970975876 (0.03173695413149347) | > loss_mel_ce: 2.4342312812805176 (nan) | > loss: 0.009752491489052773 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.1370026763819192) | > loader_time: 0.0048 (0.004634954833895446)  --> TIME: 2025-02-26 09:05:44 -- STEP: 30072/36938 -- GLOBAL_STEP: 1175150 | > loss_text_ce: 0.03020435757935047 (0.031736165136610664) | > loss_mel_ce: 2.2269296646118164 (nan) | > loss: 0.008956881240010262 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13700347077215425) | > loader_time: 0.0046 (0.004634900156821824)  --> TIME: 2025-02-26 09:05:53 -- STEP: 30122/36938 -- GLOBAL_STEP: 1175200 | > loss_text_ce: 0.03311353549361229 (0.03173475845953561) | > loss_mel_ce: 2.9157021045684814 (nan) | > loss: 0.011701649986207485 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13700274312712254) | > loader_time: 0.0046 (0.004634850046231177)  --> TIME: 2025-02-26 09:06:02 -- STEP: 30172/36938 -- GLOBAL_STEP: 1175250 | > loss_text_ce: 0.033377669751644135 (0.031733223699764626) | > loss_mel_ce: 2.6968040466308594 (nan) | > loss: 0.01083405502140522 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.1369931149526977) | > loader_time: 0.0046 (0.004634815368351451)  --> TIME: 2025-02-26 09:06:11 -- STEP: 30222/36938 -- GLOBAL_STEP: 1175300 | > loss_text_ce: 0.04178675636649132 (0.031733312324717905) | > loss_mel_ce: 2.7906227111816406 (nan) | > loss: 0.01123972050845623 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1369923687211556) | > loader_time: 0.0045 (0.004634822592761496)  --> TIME: 2025-02-26 09:06:20 -- STEP: 30272/36938 -- GLOBAL_STEP: 1175350 | > loss_text_ce: 0.020970523357391357 (0.03173277960685601) | > loss_mel_ce: 1.9225327968597412 (nan) | > loss: 0.0077123153023421764 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13699038690605816) | > loader_time: 0.0044 (0.004634748908039395)  --> TIME: 2025-02-26 09:06:30 -- STEP: 30322/36938 -- GLOBAL_STEP: 1175400 | > loss_text_ce: 0.03341911360621452 (0.03173100607047433) | > loss_mel_ce: 2.5107944011688232 (nan) | > loss: 0.01009608618915081 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13699313152371845) | > loader_time: 0.0046 (0.004634821110653645)  --> TIME: 2025-02-26 09:06:39 -- STEP: 30372/36938 -- GLOBAL_STEP: 1175450 | > loss_text_ce: 0.03206784278154373 (0.03172929841532228) | > loss_mel_ce: 2.470827579498291 (nan) | > loss: 0.009932124987244606 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13699786646256523) | > loader_time: 0.0045 (0.004634745174695847)  --> TIME: 2025-02-26 09:06:48 -- STEP: 30422/36938 -- GLOBAL_STEP: 1175500 | > loss_text_ce: 0.03442142903804779 (0.031728486967428804) | > loss_mel_ce: 0.10906982421875 (nan) | > loss: 0.0005694097490049899 (nan) | > current_lr: 5e-06 | > step_time: 0.1815 (0.13699515538607163) | > loader_time: 0.0048 (0.004634655812703444)  --> TIME: 2025-02-26 09:06:58 -- STEP: 30472/36938 -- GLOBAL_STEP: 1175550 | > loss_text_ce: 0.02456529252231121 (0.03172771676181178) | > loss_mel_ce: 2.1772494316101074 (nan) | > loss: 0.008737360127270222 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.13699902578832335) | > loader_time: 0.005 (0.004634602414431878)  --> TIME: 2025-02-26 09:07:07 -- STEP: 30522/36938 -- GLOBAL_STEP: 1175600 | > loss_text_ce: 0.026252135634422302 (0.03172794834235097) | > loss_mel_ce: 2.558781623840332 (nan) | > loss: 0.010258070193231106 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13700188266270957) | > loader_time: 0.0046 (0.004634570102142995)  --> TIME: 2025-02-26 09:07:17 -- STEP: 30572/36938 -- GLOBAL_STEP: 1175650 | > loss_text_ce: 0.03535410389304161 (0.031726919596518535) | > loss_mel_ce: 1.7707858085632324 (nan) | > loss: 0.007167222443968058 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13700262518838693) | > loader_time: 0.0046 (0.004634478212915835)  --> TIME: 2025-02-26 09:07:26 -- STEP: 30622/36938 -- GLOBAL_STEP: 1175700 | > loss_text_ce: 0.024227015674114227 (0.031724815091870084) | > loss_mel_ce: 2.0325958728790283 (nan) | > loss: 0.008161995559930801 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13700345552736073) | > loader_time: 0.0044 (0.004634418304425436)  --> TIME: 2025-02-26 09:07:35 -- STEP: 30672/36938 -- GLOBAL_STEP: 1175750 | > loss_text_ce: 0.02616363950073719 (0.03172616492396981) | > loss_mel_ce: 2.569195508956909 (nan) | > loss: 0.010299044661223888 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.1370015694455174) | > loader_time: 0.0046 (0.004634323992888397)  --> TIME: 2025-02-26 09:07:45 -- STEP: 30722/36938 -- GLOBAL_STEP: 1175800 | > loss_text_ce: 0.037838928401470184 (0.031724338828739033) | > loss_mel_ce: 3.583786725997925 (nan) | > loss: 0.014371531084179878 (nan) | > current_lr: 5e-06 | > step_time: 0.1825 (0.13700426680277605) | > loader_time: 0.0048 (0.004634261659001738)  --> TIME: 2025-02-26 09:07:54 -- STEP: 30772/36938 -- GLOBAL_STEP: 1175850 | > loss_text_ce: 0.0362359844148159 (0.03172340421421335) | > loss_mel_ce: 2.567802906036377 (nan) | > loss: 0.010333488695323467 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13700558079183128) | > loader_time: 0.0045 (0.004634205261132798)  --> TIME: 2025-02-26 09:08:04 -- STEP: 30822/36938 -- GLOBAL_STEP: 1175900 | > loss_text_ce: 0.033693235367536545 (0.03172442600799187) | > loss_mel_ce: 2.900259017944336 (nan) | > loss: 0.011642668396234512 (nan) | > current_lr: 5e-06 | > step_time: 0.1864 (0.13700866473346876) | > loader_time: 0.0048 (0.004634147174291395)  --> TIME: 2025-02-26 09:08:13 -- STEP: 30872/36938 -- GLOBAL_STEP: 1175950 | > loss_text_ce: 0.02626834809780121 (0.03172473024106791) | > loss_mel_ce: 2.6192145347595215 (nan) | > loss: 0.01049794815480709 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.1370038091441329) | > loader_time: 0.0046 (0.0046340304586856)  --> TIME: 2025-02-26 09:08:22 -- STEP: 30922/36938 -- GLOBAL_STEP: 1176000 | > loss_text_ce: 0.02945098839700222 (0.031724653261701805) | > loss_mel_ce: 0.10498046875 (nan) | > loss: 0.0005334581946954131 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13699471608963726) | > loader_time: 0.0048 (0.004633913318657981)  --> TIME: 2025-02-26 09:08:31 -- STEP: 30972/36938 -- GLOBAL_STEP: 1176050 | > loss_text_ce: 0.022412601858377457 (0.0317255426891655) | > loss_mel_ce: 2.694149971008301 (nan) | > loss: 0.010780010372400284 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1370018715392275) | > loader_time: 0.0045 (0.004633859525459104)  --> TIME: 2025-02-26 09:08:41 -- STEP: 31022/36938 -- GLOBAL_STEP: 1176100 | > loss_text_ce: 0.03408200666308403 (0.031726969109878086) | > loss_mel_ce: 2.498746871948242 (nan) | > loss: 0.010050908662378788 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13700207938585976) | > loader_time: 0.0046 (0.004633753452346538)  --> TIME: 2025-02-26 09:08:50 -- STEP: 31072/36938 -- GLOBAL_STEP: 1176150 | > loss_text_ce: 0.025313589721918106 (0.03172701995675458) | > loss_mel_ce: 2.945753574371338 (nan) | > loss: 0.011789949610829353 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13700951063424244) | > loader_time: 0.0045 (0.004633762249310662)  --> TIME: 2025-02-26 09:08:59 -- STEP: 31122/36938 -- GLOBAL_STEP: 1176200 | > loss_text_ce: 0.03623904660344124 (0.03172704604869158) | > loss_mel_ce: 2.4761929512023926 (nan) | > loss: 0.009969969280064106 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13700801517531178) | > loader_time: 0.0046 (0.004633653700930118)  --> TIME: 2025-02-26 09:09:09 -- STEP: 31172/36938 -- GLOBAL_STEP: 1176250 | > loss_text_ce: 0.03073425218462944 (0.03172741349241327) | > loss_mel_ce: 2.226691246032715 (nan) | > loss: 0.008958037942647934 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.13701784421830696) | > loader_time: 0.0046 (0.004633616272206653)  --> TIME: 2025-02-26 09:09:19 -- STEP: 31222/36938 -- GLOBAL_STEP: 1176300 | > loss_text_ce: 0.028977714478969574 (0.03172524381263349) | > loss_mel_ce: 2.964712142944336 (nan) | > loss: 0.011879721656441689 (nan) | > current_lr: 5e-06 | > step_time: 0.1857 (0.13702255330517343) | > loader_time: 0.0048 (0.004633538185859076)  --> TIME: 2025-02-26 09:09:28 -- STEP: 31272/36938 -- GLOBAL_STEP: 1176350 | > loss_text_ce: 0.03312746062874794 (0.031728176056924615) | > loss_mel_ce: 2.2193918228149414 (nan) | > loss: 0.008938569575548172 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13702894492995912) | > loader_time: 0.0046 (0.004633467896998965)  --> TIME: 2025-02-26 09:09:37 -- STEP: 31322/36938 -- GLOBAL_STEP: 1176400 | > loss_text_ce: 0.031125405803322792 (0.03172961666988098) | > loss_mel_ce: 3.0700387954711914 (nan) | > loss: 0.012306207790970802 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13702260776540712) | > loader_time: 0.0046 (0.004633392664095396)  --> TIME: 2025-02-26 09:09:47 -- STEP: 31372/36938 -- GLOBAL_STEP: 1176450 | > loss_text_ce: 0.03450971096754074 (0.03172938685116585) | > loss_mel_ce: 3.115095615386963 (nan) | > loss: 0.012498434633016586 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13701761541782423) | > loader_time: 0.0049 (0.004633341024956455)  --> TIME: 2025-02-26 09:09:56 -- STEP: 31422/36938 -- GLOBAL_STEP: 1176500 | > loss_text_ce: 0.03806757926940918 (0.03173085649527267) | > loss_mel_ce: 2.413839340209961 (nan) | > loss: 0.009729789569973946 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13701034498946157) | > loader_time: 0.0046 (0.0046332382501769005)  --> TIME: 2025-02-26 09:10:05 -- STEP: 31472/36938 -- GLOBAL_STEP: 1176550 | > loss_text_ce: 0.031914737075567245 (0.03173009708650876) | > loss_mel_ce: 3.1044538021087646 (nan) | > loss: 0.012445907108485699 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.1370122891182697) | > loader_time: 0.0046 (0.004633145332033417)  --> TIME: 2025-02-26 09:10:14 -- STEP: 31522/36938 -- GLOBAL_STEP: 1176600 | > loss_text_ce: 0.026778317987918854 (0.0317306397638751) | > loss_mel_ce: 3.0075860023498535 (nan) | > loss: 0.012041128240525723 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.1370110437732519) | > loader_time: 0.0046 (0.0046330744312103305)  --> TIME: 2025-02-26 09:10:24 -- STEP: 31572/36938 -- GLOBAL_STEP: 1176650 | > loss_text_ce: 0.02463425323367119 (0.031730861871128176) | > loss_mel_ce: 2.4524471759796143 (nan) | > loss: 0.00982968881726265 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.137015065123762) | > loader_time: 0.0045 (0.004633012031490911)  --> TIME: 2025-02-26 09:10:33 -- STEP: 31622/36938 -- GLOBAL_STEP: 1176700 | > loss_text_ce: 0.03466855734586716 (0.031730756395617815) | > loss_mel_ce: 2.870396137237549 (nan) | > loss: 0.011528034694492817 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13702504893165324) | > loader_time: 0.0046 (0.0046330683598232916)  --> TIME: 2025-02-26 09:10:42 -- STEP: 31672/36938 -- GLOBAL_STEP: 1176750 | > loss_text_ce: 0.029637407511472702 (0.03173043496457913) | > loss_mel_ce: 2.3857176303863525 (nan) | > loss: 0.009584742598235607 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.1370164838424984) | > loader_time: 0.0046 (0.004633017051819162)  --> TIME: 2025-02-26 09:10:52 -- STEP: 31722/36938 -- GLOBAL_STEP: 1176800 | > loss_text_ce: 0.030301827937364578 (0.03172923134069471) | > loss_mel_ce: 1.3982346057891846 (nan) | > loss: 0.005668795667588711 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1370163029180138) | > loader_time: 0.0045 (0.00463290344863844)  --> TIME: 2025-02-26 09:11:01 -- STEP: 31772/36938 -- GLOBAL_STEP: 1176850 | > loss_text_ce: 0.03580719977617264 (0.031730043813450175) | > loss_mel_ce: 2.7171225547790527 (nan) | > loss: 0.0109243243932724 (nan) | > current_lr: 5e-06 | > step_time: 0.1563 (0.13701942426108077) | > loader_time: 0.0047 (0.004632891830322844)  --> TIME: 2025-02-26 09:11:10 -- STEP: 31822/36938 -- GLOBAL_STEP: 1176900 | > loss_text_ce: 0.017487678676843643 (0.031729967944697623) | > loss_mel_ce: 1.0893208980560303 (nan) | > loss: 0.00439209770411253 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1370148832198173) | > loader_time: 0.0047 (0.004632885058546102)  --> TIME: 2025-02-26 09:11:20 -- STEP: 31872/36938 -- GLOBAL_STEP: 1176950 | > loss_text_ce: 0.029275257140398026 (0.031731015968933116) | > loss_mel_ce: 2.9622278213500977 (nan) | > loss: 0.011871044524013996 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.13702109753038258) | > loader_time: 0.0049 (0.004632875293373597)  --> TIME: 2025-02-26 09:11:29 -- STEP: 31922/36938 -- GLOBAL_STEP: 1177000 | > loss_text_ce: 0.0213821642100811 (0.031730045608176674) | > loss_mel_ce: 2.6673688888549805 (nan) | > loss: 0.010669647715985775 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13701793569556803) | > loader_time: 0.0046 (0.0046329000720508285)  --> TIME: 2025-02-26 09:11:39 -- STEP: 31972/36938 -- GLOBAL_STEP: 1177050 | > loss_text_ce: 0.031314849853515625 (0.031730012745925795) | > loss_mel_ce: 2.2761778831481934 (nan) | > loss: 0.009156717918813229 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.1370266464913008) | > loader_time: 0.0045 (0.004632944102044503)  --> TIME: 2025-02-26 09:11:48 -- STEP: 32022/36938 -- GLOBAL_STEP: 1177100 | > loss_text_ce: 0.035647887736558914 (0.0317299864820572) | > loss_mel_ce: 2.7964236736297607 (nan) | > loss: 0.011238379403948784 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13702431339199336) | > loader_time: 0.0045 (0.004632964511552329)  --> TIME: 2025-02-26 09:11:57 -- STEP: 32072/36938 -- GLOBAL_STEP: 1177150 | > loss_text_ce: 0.03148454800248146 (0.03173119617280158) | > loss_mel_ce: 1.6351138353347778 (nan) | > loss: 0.00661348644644022 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13702652696868542) | > loader_time: 0.0046 (0.004632911455545037)  --> TIME: 2025-02-26 09:12:06 -- STEP: 32122/36938 -- GLOBAL_STEP: 1177200 | > loss_text_ce: 0.036964334547519684 (0.03173281556807963) | > loss_mel_ce: 2.3314414024353027 (nan) | > loss: 0.009398436173796654 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13701847858249622) | > loader_time: 0.0045 (0.004632803573012254)  --> TIME: 2025-02-26 09:12:15 -- STEP: 32172/36938 -- GLOBAL_STEP: 1177250 | > loss_text_ce: 0.03914203494787216 (0.03173302924780402) | > loss_mel_ce: 2.607234001159668 (nan) | > loss: 0.010501492768526077 (nan) | > current_lr: 5e-06 | > step_time: 0.1079 (0.13700621642832808) | > loader_time: 0.0044 (0.0046327206146712764)  --> TIME: 2025-02-26 09:12:24 -- STEP: 32222/36938 -- GLOBAL_STEP: 1177300 | > loss_text_ce: 0.03131154179573059 (0.031732011415435614) | > loss_mel_ce: 1.76043701171875 (nan) | > loss: 0.0071101137436926365 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13700251739552374) | > loader_time: 0.0044 (0.004632607044125037)  --> TIME: 2025-02-26 09:12:34 -- STEP: 32272/36938 -- GLOBAL_STEP: 1177350 | > loss_text_ce: 0.040683627128601074 (0.031730614599564395) | > loss_mel_ce: 2.840519428253174 (nan) | > loss: 0.011433346197009087 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1370033265253547) | > loader_time: 0.0047 (0.004632488742699776)  --> TIME: 2025-02-26 09:12:43 -- STEP: 32322/36938 -- GLOBAL_STEP: 1177400 | > loss_text_ce: 0.03721155971288681 (0.031729581504690144) | > loss_mel_ce: 2.5194361209869385 (nan) | > loss: 0.010145428590476513 (nan) | > current_lr: 5e-06 | > step_time: 0.1882 (0.13700704780827863) | > loader_time: 0.0047 (0.004632364094799531)  --> TIME: 2025-02-26 09:12:53 -- STEP: 32372/36938 -- GLOBAL_STEP: 1177450 | > loss_text_ce: 0.03702652454376221 (0.0317304348588911) | > loss_mel_ce: 3.1664512157440186 (nan) | > loss: 0.012712214142084122 (nan) | > current_lr: 5e-06 | > step_time: 0.1852 (0.13701922150694001) | > loader_time: 0.0048 (0.0046323669217476415)  --> TIME: 2025-02-26 09:13:02 -- STEP: 32422/36938 -- GLOBAL_STEP: 1177500 | > loss_text_ce: 0.030265552923083305 (0.03172834662139848) | > loss_mel_ce: 3.4840266704559326 (nan) | > loss: 0.013945604674518108 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.13701097431809967) | > loader_time: 0.0047 (0.004632343465546103)  --> TIME: 2025-02-26 09:13:12 -- STEP: 32472/36938 -- GLOBAL_STEP: 1177550 | > loss_text_ce: 0.03359488770365715 (0.031729967855165915) | > loss_mel_ce: 2.7503364086151123 (nan) | > loss: 0.011047346517443657 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13701089013236636) | > loader_time: 0.0046 (0.004632247987713373)  --> TIME: 2025-02-26 09:13:21 -- STEP: 32522/36938 -- GLOBAL_STEP: 1177600 | > loss_text_ce: 0.024858908727765083 (0.0317304633580075) | > loss_mel_ce: 1.0898902416229248 (nan) | > loss: 0.00442360807210207 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13701161669083714) | > loader_time: 0.0045 (0.004632248436273474)  --> TIME: 2025-02-26 09:13:30 -- STEP: 32572/36938 -- GLOBAL_STEP: 1177650 | > loss_text_ce: 0.037314917892217636 (0.031731903598843106) | > loss_mel_ce: 2.74239182472229 (nan) | > loss: 0.011030582711100578 (nan) | > current_lr: 5e-06 | > step_time: 0.1443 (0.1370156460424378) | > loader_time: 0.0045 (0.004632205535954739)  --> TIME: 2025-02-26 09:13:39 -- STEP: 32622/36938 -- GLOBAL_STEP: 1177700 | > loss_text_ce: 0.028563885018229485 (0.03173208012533988) | > loss_mel_ce: 2.931919574737549 (nan) | > loss: 0.011747951619327068 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13700690031797094) | > loader_time: 0.0044 (0.0046320680194742855)  --> TIME: 2025-02-26 09:13:49 -- STEP: 32672/36938 -- GLOBAL_STEP: 1177750 | > loss_text_ce: 0.025642454624176025 (0.03173168986145468) | > loss_mel_ce: 1.191291332244873 (nan) | > loss: 0.004829102195799351 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13701664868753163) | > loader_time: 0.0045 (0.004631970468160585)  --> TIME: 2025-02-26 09:13:59 -- STEP: 32722/36938 -- GLOBAL_STEP: 1177800 | > loss_text_ce: 0.03498436510562897 (0.03173125894424519) | > loss_mel_ce: 2.654721260070801 (nan) | > loss: 0.010673435404896736 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13701664407622954) | > loader_time: 0.0046 (0.004631852558630994)  --> TIME: 2025-02-26 09:14:08 -- STEP: 32772/36938 -- GLOBAL_STEP: 1177850 | > loss_text_ce: 0.0311236884444952 (0.03172966590740918) | > loss_mel_ce: 2.8307149410247803 (nan) | > loss: 0.011356502771377563 (nan) | > current_lr: 5e-06 | > step_time: 0.177 (0.13702619360957863) | > loader_time: 0.0048 (0.004631921970901029)  --> TIME: 2025-02-26 09:14:18 -- STEP: 32822/36938 -- GLOBAL_STEP: 1177900 | > loss_text_ce: 0.020644590258598328 (0.03172699361625283) | > loss_mel_ce: 3.0579237937927246 (nan) | > loss: 0.012216541916131973 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.1370351467161042) | > loader_time: 0.0046 (0.004631890463775301)  --> TIME: 2025-02-26 09:14:27 -- STEP: 32872/36938 -- GLOBAL_STEP: 1177950 | > loss_text_ce: 0.02466297335922718 (0.031726536520125005) | > loss_mel_ce: 3.2428386211395264 (nan) | > loss: 0.012966277077794075 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13702948482461894) | > loader_time: 0.0045 (0.0046318935982413495)  --> TIME: 2025-02-26 09:14:36 -- STEP: 32922/36938 -- GLOBAL_STEP: 1178000 | > loss_text_ce: 0.02784557454288006 (0.03172713854784599) | > loss_mel_ce: 1.2330824136734009 (nan) | > loss: 0.005003682803362608 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13702751994748552) | > loader_time: 0.0045 (0.004631803888979431)  --> TIME: 2025-02-26 09:14:46 -- STEP: 32972/36938 -- GLOBAL_STEP: 1178050 | > loss_text_ce: 0.03379008546471596 (0.03172709038151195) | > loss_mel_ce: 2.2760183811187744 (nan) | > loss: 0.00916590727865696 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13703289480879216) | > loader_time: 0.0045 (0.004631765668548095)  --> TIME: 2025-02-26 09:14:55 -- STEP: 33022/36938 -- GLOBAL_STEP: 1178100 | > loss_text_ce: 0.02932317927479744 (0.03172588039581704) | > loss_mel_ce: 2.0870769023895264 (nan) | > loss: 0.00839841365814209 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.1370227593698663) | > loader_time: 0.0045 (0.004631626779989373)  --> TIME: 2025-02-26 09:15:04 -- STEP: 33072/36938 -- GLOBAL_STEP: 1178150 | > loss_text_ce: 0.03560289368033409 (0.03172742949163845) | > loss_mel_ce: 3.0096075534820557 (nan) | > loss: 0.012084168381989002 (nan) | > current_lr: 5e-06 | > step_time: 0.1747 (0.13702812849264176) | > loader_time: 0.0047 (0.004631536352674815)  --> TIME: 2025-02-26 09:15:14 -- STEP: 33122/36938 -- GLOBAL_STEP: 1178200 | > loss_text_ce: 0.029602818191051483 (0.03172762980633454) | > loss_mel_ce: 3.0278117656707764 (nan) | > loss: 0.012132598087191582 (nan) | > current_lr: 5e-06 | > step_time: 0.181 (0.13703739495533937) | > loader_time: 0.0048 (0.004631604270711641)  --> TIME: 2025-02-26 09:15:23 -- STEP: 33172/36938 -- GLOBAL_STEP: 1178250 | > loss_text_ce: 0.028633343055844307 (0.031727793751704175) | > loss_mel_ce: 2.4622673988342285 (nan) | > loss: 0.009884526953101158 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13703109118959414) | > loader_time: 0.0047 (0.004631507889749817)  --> TIME: 2025-02-26 09:15:33 -- STEP: 33222/36938 -- GLOBAL_STEP: 1178300 | > loss_text_ce: 0.038089554756879807 (0.031728335731074524) | > loss_mel_ce: 2.782616138458252 (nan) | > loss: 0.011193277314305305 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13703547050230788) | > loader_time: 0.0046 (0.004631436787566774)  --> TIME: 2025-02-26 09:15:42 -- STEP: 33272/36938 -- GLOBAL_STEP: 1178350 | > loss_text_ce: 0.04044848307967186 (0.031728159835388244) | > loss_mel_ce: 3.307368040084839 (nan) | > loss: 0.013284986838698387 (nan) | > current_lr: 5e-06 | > step_time: 0.1851 (0.1370420409570262) | > loader_time: 0.0049 (0.004631472926611063)  --> TIME: 2025-02-26 09:15:52 -- STEP: 33322/36938 -- GLOBAL_STEP: 1178400 | > loss_text_ce: 0.03276120498776436 (0.031727867909035076) | > loss_mel_ce: 2.6207447052001953 (nan) | > loss: 0.010529785417020321 (nan) | > current_lr: 5e-06 | > step_time: 0.1864 (0.13704469985130768) | > loader_time: 0.0048 (0.004631463000700127)  --> TIME: 2025-02-26 09:16:01 -- STEP: 33372/36938 -- GLOBAL_STEP: 1178450 | > loss_text_ce: 0.03946109861135483 (0.03172738903804793) | > loss_mel_ce: 2.641248941421509 (nan) | > loss: 0.0106377387419343 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13704265101299146) | > loader_time: 0.0044 (0.004631353620572396)  --> TIME: 2025-02-26 09:16:11 -- STEP: 33422/36938 -- GLOBAL_STEP: 1178500 | > loss_text_ce: 0.04065338522195816 (0.03172760022387598) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13705141511099522) | > loader_time: 0.0045 (0.0046313457789765615)  --> TIME: 2025-02-26 09:16:20 -- STEP: 33472/36938 -- GLOBAL_STEP: 1178550 | > loss_text_ce: 0.024677535519003868 (0.031727489720338536) | > loss_mel_ce: 2.3843822479248047 (nan) | > loss: 0.009559761732816696 (nan) | > current_lr: 5e-06 | > step_time: 0.1599 (0.1370457061789001) | > loader_time: 0.0046 (0.004631286162890179)  --> TIME: 2025-02-26 09:16:29 -- STEP: 33522/36938 -- GLOBAL_STEP: 1178600 | > loss_text_ce: 0.045136984437704086 (0.031729401398302694) | > loss_mel_ce: 2.6112332344055176 (nan) | > loss: 0.010541152209043503 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.1370513637896175) | > loader_time: 0.0065 (0.004631273594713973)  --> TIME: 2025-02-26 09:16:39 -- STEP: 33572/36938 -- GLOBAL_STEP: 1178650 | > loss_text_ce: 0.027653856202960014 (0.03173150142793917) | > loss_mel_ce: 2.5449912548065186 (nan) | > loss: 0.010208910331130028 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13705044019695206) | > loader_time: 0.0046 (0.004631252073210484)  --> TIME: 2025-02-26 09:16:48 -- STEP: 33622/36938 -- GLOBAL_STEP: 1178700 | > loss_text_ce: 0.02374517172574997 (0.03173282900051202) | > loss_mel_ce: 2.975494146347046 (nan) | > loss: 0.011901743710041046 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13705373156411763) | > loader_time: 0.0047 (0.004631246641711727)  --> TIME: 2025-02-26 09:16:58 -- STEP: 33672/36938 -- GLOBAL_STEP: 1178750 | > loss_text_ce: 0.037160441279411316 (0.031733757950565496) | > loss_mel_ce: 3.226773977279663 (nan) | > loss: 0.01295212097465992 (nan) | > current_lr: 5e-06 | > step_time: 0.1817 (0.1370604420096267) | > loader_time: 0.0048 (0.004631229366307942)  --> TIME: 2025-02-26 09:17:07 -- STEP: 33722/36938 -- GLOBAL_STEP: 1178800 | > loss_text_ce: 0.023340100422501564 (0.031733235963285165) | > loss_mel_ce: 2.250056743621826 (nan) | > loss: 0.009021416306495667 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13706249563191003) | > loader_time: 0.0046 (0.004631174500815349)  --> TIME: 2025-02-26 09:17:16 -- STEP: 33772/36938 -- GLOBAL_STEP: 1178850 | > loss_text_ce: 0.031741250306367874 (0.03173198602124789) | > loss_mel_ce: 2.7788941860198975 (nan) | > loss: 0.01115331519395113 (nan) | > current_lr: 5e-06 | > step_time: 0.1579 (0.13705512321801516) | > loader_time: 0.0046 (0.004631149794244109)  --> TIME: 2025-02-26 09:17:26 -- STEP: 33822/36938 -- GLOBAL_STEP: 1178900 | > loss_text_ce: 0.03544072061777115 (0.031730749475681004) | > loss_mel_ce: 2.469989538192749 (nan) | > loss: 0.009942184202373028 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13705899299378838) | > loader_time: 0.0065 (0.004631209659672301)  --> TIME: 2025-02-26 09:17:35 -- STEP: 33872/36938 -- GLOBAL_STEP: 1178950 | > loss_text_ce: 0.04334663599729538 (0.031728931063653876) | > loss_mel_ce: 2.5259575843811035 (nan) | > loss: 0.010195652022957802 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13705915109160688) | > loader_time: 0.0045 (0.004631261880182707)  --> TIME: 2025-02-26 09:17:44 -- STEP: 33922/36938 -- GLOBAL_STEP: 1179000 | > loss_text_ce: 0.04107750207185745 (0.03172892649699144) | > loss_mel_ce: 1.686707615852356 (nan) | > loss: 0.006856290623545647 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13705645824306625) | > loader_time: 0.0045 (0.004631283183289312)  --> TIME: 2025-02-26 09:17:54 -- STEP: 33972/36938 -- GLOBAL_STEP: 1179050 | > loss_text_ce: 0.040766701102256775 (0.03173028424263878) | > loss_mel_ce: 1.1328006982803345 (nan) | > loss: 0.0046570138074457645 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13705981320238872) | > loader_time: 0.0044 (0.004631181635172242)  --> TIME: 2025-02-26 09:18:03 -- STEP: 34022/36938 -- GLOBAL_STEP: 1179100 | > loss_text_ce: 0.03799881041049957 (0.031731230544611584) | > loss_mel_ce: 3.087817668914795 (nan) | > loss: 0.012404034845530987 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.13705916764503354) | > loader_time: 0.0046 (0.0046310939525886765)  --> TIME: 2025-02-26 09:18:13 -- STEP: 34072/36938 -- GLOBAL_STEP: 1179150 | > loss_text_ce: 0.03971289470791817 (0.03173204566271477) | > loss_mel_ce: 2.789771556854248 (nan) | > loss: 0.011228113435208797 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13705885284353939) | > loader_time: 0.0048 (0.004631046860899343)  --> TIME: 2025-02-26 09:18:22 -- STEP: 34122/36938 -- GLOBAL_STEP: 1179200 | > loss_text_ce: 0.037800293415784836 (0.03173244726429744) | > loss_mel_ce: 2.6628217697143555 (nan) | > loss: 0.0107167549431324 (nan) | > current_lr: 5e-06 | > step_time: 0.152 (0.13706867264161285) | > loader_time: 0.0046 (0.004631006209709609)  --> TIME: 2025-02-26 09:18:31 -- STEP: 34172/36938 -- GLOBAL_STEP: 1179250 | > loss_text_ce: 0.01834951899945736 (0.03173144680248802) | > loss_mel_ce: 1.758013129234314 (nan) | > loss: 0.007049058564007282 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.1370650905783051) | > loader_time: 0.0045 (0.0046309311761379614)  --> TIME: 2025-02-26 09:18:41 -- STEP: 34222/36938 -- GLOBAL_STEP: 1179300 | > loss_text_ce: 0.03702426329255104 (0.03173195877620307) | > loss_mel_ce: 2.791304349899292 (nan) | > loss: 0.011223526671528816 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.1370686287910874) | > loader_time: 0.0048 (0.004630915510139623)  --> TIME: 2025-02-26 09:18:50 -- STEP: 34272/36938 -- GLOBAL_STEP: 1179350 | > loss_text_ce: 0.02215057611465454 (0.03172816208247531) | > loss_mel_ce: 0.7708272933959961 (nan) | > loss: 0.003146737813949585 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13706149002945803) | > loader_time: 0.0045 (0.004630862532599208)  --> TIME: 2025-02-26 09:18:59 -- STEP: 34322/36938 -- GLOBAL_STEP: 1179400 | > loss_text_ce: 0.039195466786623 (0.031729028559108544) | > loss_mel_ce: 3.1976351737976074 (nan) | > loss: 0.012844567187130451 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13705862021364223) | > loader_time: 0.0047 (0.004630825123748878)  --> TIME: 2025-02-26 09:19:09 -- STEP: 34372/36938 -- GLOBAL_STEP: 1179450 | > loss_text_ce: 0.034452881664037704 (0.031730097190095445) | > loss_mel_ce: 3.6416306495666504 (nan) | > loss: 0.01458763424307108 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.1370576811477713) | > loader_time: 0.0046 (0.004630787761306027)  --> TIME: 2025-02-26 09:19:18 -- STEP: 34422/36938 -- GLOBAL_STEP: 1179500 | > loss_text_ce: 0.03411221131682396 (0.031730707964204116) | > loss_mel_ce: 2.6090457439422607 (nan) | > loss: 0.01048872247338295 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.137058522234289) | > loader_time: 0.0047 (0.004630845869296079)  --> TIME: 2025-02-26 09:19:27 -- STEP: 34472/36938 -- GLOBAL_STEP: 1179550 | > loss_text_ce: 0.034472379833459854 (0.03172928551434019) | > loss_mel_ce: 2.6384973526000977 (nan) | > loss: 0.010607023723423481 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13705764279672028) | > loader_time: 0.0045 (0.004630929945405513)  --> TIME: 2025-02-26 09:19:36 -- STEP: 34522/36938 -- GLOBAL_STEP: 1179600 | > loss_text_ce: 0.03425022214651108 (0.03172777109863523) | > loss_mel_ce: 3.178447723388672 (nan) | > loss: 0.012748802080750465 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.13705266592476442) | > loader_time: 0.0048 (0.004630906309356895)  --> TIME: 2025-02-26 09:19:46 -- STEP: 34572/36938 -- GLOBAL_STEP: 1179650 | > loss_text_ce: 0.034708619117736816 (0.03172899522134048) | > loss_mel_ce: 2.968432664871216 (nan) | > loss: 0.011917228810489178 (nan) | > current_lr: 5e-06 | > step_time: 0.1513 (0.1370514284939614) | > loader_time: 0.0046 (0.004630818116531118)  --> TIME: 2025-02-26 09:19:55 -- STEP: 34622/36938 -- GLOBAL_STEP: 1179700 | > loss_text_ce: 0.03096729703247547 (0.03172795522647497) | > loss_mel_ce: 0.01416015625 (nan) | > loss: 0.00017907719302456826 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13704470677486208) | > loader_time: 0.0045 (0.004630729200576821)  --> TIME: 2025-02-26 09:20:04 -- STEP: 34672/36938 -- GLOBAL_STEP: 1179750 | > loss_text_ce: 0.02879280410706997 (0.0317273816906291) | > loss_mel_ce: 2.38149094581604 (nan) | > loss: 0.009564618580043316 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.1370439116284763) | > loader_time: 0.0046 (0.004630637329792087)  --> TIME: 2025-02-26 09:20:13 -- STEP: 34722/36938 -- GLOBAL_STEP: 1179800 | > loss_text_ce: 0.03893258795142174 (0.03172765422872485) | > loss_mel_ce: 3.380872964859009 (nan) | > loss: 0.013570657931268215 (nan) | > current_lr: 5e-06 | > step_time: 0.1499 (0.1370485278155337) | > loader_time: 0.0046 (0.004630567854323257)  --> TIME: 2025-02-26 09:20:23 -- STEP: 34772/36938 -- GLOBAL_STEP: 1179850 | > loss_text_ce: 0.030424784868955612 (0.031726950857417315) | > loss_mel_ce: 2.7331910133361816 (nan) | > loss: 0.010966730304062366 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.1370427675684915) | > loader_time: 0.0045 (0.004630479914922946)  --> TIME: 2025-02-26 09:20:32 -- STEP: 34822/36938 -- GLOBAL_STEP: 1179900 | > loss_text_ce: 0.031797513365745544 (0.031728773645796154) | > loss_mel_ce: 2.5545103549957275 (nan) | > loss: 0.010263126343488693 (nan) | > current_lr: 5e-06 | > step_time: 0.1877 (0.13704946473002036) | > loader_time: 0.0048 (0.004630498894050689)  --> TIME: 2025-02-26 09:20:42 -- STEP: 34872/36938 -- GLOBAL_STEP: 1179950 | > loss_text_ce: 0.04298866540193558 (0.031731135991218545) | > loss_mel_ce: 2.3822200298309326 (nan) | > loss: 0.009623845107853413 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13705303580642234) | > loader_time: 0.0046 (0.004630483305763311)  --> TIME: 2025-02-26 09:20:51 -- STEP: 34922/36938 -- GLOBAL_STEP: 1180000 | > loss_text_ce: 0.022382646799087524 (0.03172993211489122) | > loss_mel_ce: 0.1814313679933548 (nan) | > loss: 0.0008087858441285789 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13705210898464698) | > loader_time: 0.0045 (0.004630443368619784) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1180000.pth  --> TIME: 2025-02-26 09:21:06 -- STEP: 34972/36938 -- GLOBAL_STEP: 1180050 | > loss_text_ce: 0.03402665629982948 (0.0317315774474481) | > loss_mel_ce: 2.1772053241729736 (nan) | > loss: 0.008774730376899242 (nan) | > current_lr: 5e-06 | > step_time: 0.1501 (0.13704761758789236) | > loader_time: 0.0049 (0.004630474576303261)  --> TIME: 2025-02-26 09:21:15 -- STEP: 35022/36938 -- GLOBAL_STEP: 1180100 | > loss_text_ce: 0.03203391656279564 (0.03172978409075341) | > loss_mel_ce: 2.9332926273345947 (nan) | > loss: 0.011767169460654259 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13703876440028467) | > loader_time: 0.0046 (0.004630463909336321)  --> TIME: 2025-02-26 09:21:24 -- STEP: 35072/36938 -- GLOBAL_STEP: 1180150 | > loss_text_ce: 0.034787874668836594 (0.0317297476823332) | > loss_mel_ce: 2.0308117866516113 (nan) | > loss: 0.008196824230253696 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1370320759346556) | > loader_time: 0.0046 (0.004630474162961032)  --> TIME: 2025-02-26 09:21:33 -- STEP: 35122/36938 -- GLOBAL_STEP: 1180200 | > loss_text_ce: 0.03533807024359703 (0.03173309118315575) | > loss_mel_ce: 2.6790966987609863 (nan) | > loss: 0.010771567933261395 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.13703132017850367) | > loader_time: 0.0046 (0.004630448246496408)  --> TIME: 2025-02-26 09:21:43 -- STEP: 35172/36938 -- GLOBAL_STEP: 1180250 | > loss_text_ce: 0.02445090003311634 (0.03173338753299622) | > loss_mel_ce: 2.666515588760376 (nan) | > loss: 0.010678439401090145 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13703498532716987) | > loader_time: 0.0045 (0.004630428470605957)  --> TIME: 2025-02-26 09:21:52 -- STEP: 35222/36938 -- GLOBAL_STEP: 1180300 | > loss_text_ce: 0.03133248910307884 (0.031734200953428326) | > loss_mel_ce: 2.2799973487854004 (nan) | > loss: 0.009171944111585617 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.1370310788462994) | > loader_time: 0.0046 (0.0046306371580538625)  --> TIME: 2025-02-26 09:22:01 -- STEP: 35272/36938 -- GLOBAL_STEP: 1180350 | > loss_text_ce: 0.03311583772301674 (0.03173325634026519) | > loss_mel_ce: 1.6212047338485718 (nan) | > loss: 0.006564764771610498 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13702757159418197) | > loader_time: 0.0046 (0.0046306568888424176)  --> TIME: 2025-02-26 09:22:10 -- STEP: 35322/36938 -- GLOBAL_STEP: 1180400 | > loss_text_ce: 0.03382469341158867 (0.03173364478021397) | > loss_mel_ce: 2.9826531410217285 (nan) | > loss: 0.011970151215791702 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13703003231946215) | > loader_time: 0.0046 (0.0046306857840809)  --> TIME: 2025-02-26 09:22:19 -- STEP: 35372/36938 -- GLOBAL_STEP: 1180450 | > loss_text_ce: 0.024104207754135132 (0.0317319255482773) | > loss_mel_ce: 2.402863025665283 (nan) | > loss: 0.009630822576582432 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13702447250877997) | > loader_time: 0.0045 (0.00463074600077805)  --> TIME: 2025-02-26 09:22:28 -- STEP: 35422/36938 -- GLOBAL_STEP: 1180500 | > loss_text_ce: 0.02072550542652607 (0.03173143177439254) | > loss_mel_ce: 0.011141827329993248 (nan) | > loss: 0.00012645767128560692 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13701823607686975) | > loader_time: 0.0047 (0.004630675887167699)  --> TIME: 2025-02-26 09:22:38 -- STEP: 35472/36938 -- GLOBAL_STEP: 1180550 | > loss_text_ce: 0.038381028920412064 (0.031731651616993636) | > loss_mel_ce: 0.16940918564796448 (nan) | > loss: 0.0008245643693953753 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13701200797770724) | > loader_time: 0.0046 (0.004630592118580732)  --> TIME: 2025-02-26 09:22:47 -- STEP: 35522/36938 -- GLOBAL_STEP: 1180600 | > loss_text_ce: 0.03463635966181755 (0.03173131580737146) | > loss_mel_ce: 3.173874855041504 (nan) | > loss: 0.012732187286019325 (nan) | > current_lr: 5e-06 | > step_time: 0.1503 (0.13700816587153083) | > loader_time: 0.0047 (0.004630606578925517)  --> TIME: 2025-02-26 09:22:56 -- STEP: 35572/36938 -- GLOBAL_STEP: 1180650 | > loss_text_ce: 0.03703613579273224 (0.03173133656546445) | > loss_mel_ce: 3.0123379230499268 (nan) | > loss: 0.01210069190710783 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13700674290760495) | > loader_time: 0.0046 (0.004630624651439729)  --> TIME: 2025-02-26 09:23:05 -- STEP: 35622/36938 -- GLOBAL_STEP: 1180700 | > loss_text_ce: 0.029820093885064125 (0.031729347541444824) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.13700782624911229) | > loader_time: 0.0048 (0.00463065244502166)  --> TIME: 2025-02-26 09:23:15 -- STEP: 35672/36938 -- GLOBAL_STEP: 1180750 | > loss_text_ce: 0.03407042846083641 (0.03172979469347466) | > loss_mel_ce: 2.5745561122894287 (nan) | > loss: 0.010351693257689476 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13700687529715083) | > loader_time: 0.0046 (0.004630602296357437)  --> TIME: 2025-02-26 09:23:24 -- STEP: 35722/36938 -- GLOBAL_STEP: 1180800 | > loss_text_ce: 0.03999391198158264 (0.031729081884447966) | > loss_mel_ce: 0.0060577392578125 (nan) | > loss: 0.00018274465401191264 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13700946751183324) | > loader_time: 0.0046 (0.00463073557044298)  --> TIME: 2025-02-26 09:23:33 -- STEP: 35772/36938 -- GLOBAL_STEP: 1180850 | > loss_text_ce: 0.03497292101383209 (0.0317274415590871) | > loss_mel_ce: 2.167877197265625 (nan) | > loss: 0.008741469122469425 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.137001546226624) | > loader_time: 0.0046 (0.004630773416451153)  --> TIME: 2025-02-26 09:23:42 -- STEP: 35822/36938 -- GLOBAL_STEP: 1180900 | > loss_text_ce: 0.03499400243163109 (0.03172714474431743) | > loss_mel_ce: 3.198190689086914 (nan) | > loss: 0.012830098159611225 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13699677805027058) | > loader_time: 0.0046 (0.004630793392887881)  --> TIME: 2025-02-26 09:23:52 -- STEP: 35872/36938 -- GLOBAL_STEP: 1180950 | > loss_text_ce: 0.0353781133890152 (0.031727972670785726) | > loss_mel_ce: 2.880223512649536 (nan) | > loss: 0.011569849215447903 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13699734167766592) | > loader_time: 0.0049 (0.004630915548097465)  --> TIME: 2025-02-26 09:24:01 -- STEP: 35922/36938 -- GLOBAL_STEP: 1181000 | > loss_text_ce: 0.04300681874155998 (0.03172962544981628) | > loss_mel_ce: 2.2510294914245605 (nan) | > loss: 0.009103319607675076 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1369921632721271) | > loader_time: 0.0045 (0.004630988368039752)  --> TIME: 2025-02-26 09:24:10 -- STEP: 35972/36938 -- GLOBAL_STEP: 1181050 | > loss_text_ce: 0.028021370992064476 (0.03172692814423989) | > loss_mel_ce: 2.915809392929077 (nan) | > loss: 0.011681868694722652 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13699012379697723) | > loader_time: 0.0045 (0.004631012562158255)  --> TIME: 2025-02-26 09:24:19 -- STEP: 36022/36938 -- GLOBAL_STEP: 1181100 | > loss_text_ce: 0.027834724634885788 (0.03172466121952143) | > loss_mel_ce: 2.1662790775299072 (nan) | > loss: 0.008706800639629364 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.1369800608199544) | > loader_time: 0.0045 (0.004630978669644248)  --> TIME: 2025-02-26 09:24:28 -- STEP: 36072/36938 -- GLOBAL_STEP: 1181150 | > loss_text_ce: 0.030318932607769966 (0.03172414662222658) | > loss_mel_ce: 2.859421491622925 (nan) | > loss: 0.011467224918305874 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13698000480526423) | > loader_time: 0.0046 (0.004631020054369893)  --> TIME: 2025-02-26 09:24:38 -- STEP: 36122/36938 -- GLOBAL_STEP: 1181200 | > loss_text_ce: 0.027441255748271942 (0.03172426192407237) | > loss_mel_ce: 0.016143765300512314 (nan) | > loss: 0.00017295643920078874 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13698601646152972) | > loader_time: 0.0046 (0.004631074545068113)  --> TIME: 2025-02-26 09:24:47 -- STEP: 36172/36938 -- GLOBAL_STEP: 1181250 | > loss_text_ce: 0.02921987511217594 (0.031723155004525196) | > loss_mel_ce: 1.971527099609375 (nan) | > loss: 0.007939472794532776 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13697973094418356) | > loader_time: 0.0046 (0.004631202713678384)  --> TIME: 2025-02-26 09:24:57 -- STEP: 36222/36938 -- GLOBAL_STEP: 1181300 | > loss_text_ce: 0.025725768879055977 (0.03172296585461549) | > loss_mel_ce: 2.855111837387085 (nan) | > loss: 0.01143189612776041 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.136984095996789) | > loader_time: 0.0046 (0.004631417254833992)  --> TIME: 2025-02-26 09:25:06 -- STEP: 36272/36938 -- GLOBAL_STEP: 1181350 | > loss_text_ce: 0.03721487894654274 (0.03172344260830154) | > loss_mel_ce: 2.456369161605835 (nan) | > loss: 0.009895175695419312 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13698393682261945) | > loader_time: 0.0047 (0.00463142256853496)  --> TIME: 2025-02-26 09:25:15 -- STEP: 36322/36938 -- GLOBAL_STEP: 1181400 | > loss_text_ce: 0.02979704551398754 (0.03172105913843842) | > loss_mel_ce: 2.8989415168762207 (nan) | > loss: 0.01162197906523943 (nan) | > current_lr: 5e-06 | > step_time: 0.1526 (0.13697588131308422) | > loader_time: 0.0049 (0.0046314384159974594)  --> TIME: 2025-02-26 09:25:24 -- STEP: 36372/36938 -- GLOBAL_STEP: 1181450 | > loss_text_ce: 0.03361929953098297 (0.03172107090177779) | > loss_mel_ce: 2.5508134365081787 (nan) | > loss: 0.01025568600744009 (nan) | > current_lr: 5e-06 | > step_time: 0.1776 (0.13697719623137125) | > loader_time: 0.0048 (0.004631482426068549)  --> TIME: 2025-02-26 09:25:34 -- STEP: 36422/36938 -- GLOBAL_STEP: 1181500 | > loss_text_ce: 0.029516730457544327 (0.03172176305501649) | > loss_mel_ce: 2.7655744552612305 (nan) | > loss: 0.011091631837189198 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13697753135340135) | > loader_time: 0.0045 (0.004631496400064917)  --> TIME: 2025-02-26 09:25:43 -- STEP: 36472/36938 -- GLOBAL_STEP: 1181550 | > loss_text_ce: 0.02381427399814129 (0.031720382494469464) | > loss_mel_ce: 1.6198115348815918 (nan) | > loss: 0.006522325333207846 (nan) | > current_lr: 5e-06 | > step_time: 0.1077 (0.13697401331715175) | > loader_time: 0.0045 (0.004631524305381305)  --> TIME: 2025-02-26 09:25:52 -- STEP: 36522/36938 -- GLOBAL_STEP: 1181600 | > loss_text_ce: 0.035798080265522 (0.0317203880663938) | > loss_mel_ce: 0.00597381591796875 (nan) | > loss: 0.00016576150665059686 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13697051426829185) | > loader_time: 0.0046 (0.004631605370792459)  --> TIME: 2025-02-26 09:26:01 -- STEP: 36572/36938 -- GLOBAL_STEP: 1181650 | > loss_text_ce: 0.0262998566031456 (0.03172034964210491) | > loss_mel_ce: 2.8133909702301025 (nan) | > loss: 0.01126861572265625 (nan) | > current_lr: 5e-06 | > step_time: 0.1865 (0.13697226687957778) | > loader_time: 0.0048 (0.004631677413683026)  --> TIME: 2025-02-26 09:26:11 -- STEP: 36622/36938 -- GLOBAL_STEP: 1181700 | > loss_text_ce: 0.025971485301852226 (0.03172103124871837) | > loss_mel_ce: 2.6097471714019775 (nan) | > loss: 0.010459201410412788 (nan) | > current_lr: 5e-06 | > step_time: 0.1118 (0.13697422464139603) | > loader_time: 0.0045 (0.004631745145371829)  --> TIME: 2025-02-26 09:26:20 -- STEP: 36672/36938 -- GLOBAL_STEP: 1181750 | > loss_text_ce: 0.03440440446138382 (0.031719526813702145) | > loss_mel_ce: 2.5086965560913086 (nan) | > loss: 0.010091671720147133 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13696936767606854) | > loader_time: 0.0048 (0.0046318529684090315)  --> TIME: 2025-02-26 09:26:29 -- STEP: 36722/36938 -- GLOBAL_STEP: 1181800 | > loss_text_ce: 0.027336589992046356 (0.03171890613539624) | > loss_mel_ce: 1.91162109375 (nan) | > loss: 0.0076942769810557365 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.1369611312475518) | > loader_time: 0.0045 (0.004631845742410529)  --> TIME: 2025-02-26 09:26:38 -- STEP: 36772/36938 -- GLOBAL_STEP: 1181850 | > loss_text_ce: 0.02616097405552864 (0.03172015175866149) | > loss_mel_ce: 2.776378631591797 (nan) | > loss: 0.011121189221739769 (nan) | > current_lr: 5e-06 | > step_time: 0.1563 (0.1369579076754002) | > loader_time: 0.0048 (0.004631886372793957)  --> TIME: 2025-02-26 09:26:48 -- STEP: 36822/36938 -- GLOBAL_STEP: 1181900 | > loss_text_ce: 0.03295104578137398 (0.031719949510050485) | > loss_mel_ce: 2.7838799953460693 (nan) | > loss: 0.011177902109920979 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13695943931178464) | > loader_time: 0.0046 (0.004631865446082333)  --> TIME: 2025-02-26 09:26:57 -- STEP: 36872/36938 -- GLOBAL_STEP: 1181950 | > loss_text_ce: 0.029062194749712944 (0.03171890650256634) | > loss_mel_ce: 2.4578566551208496 (nan) | > loss: 0.009868726134300232 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13695545471601725) | > loader_time: 0.0047 (0.004631942757610463)  --> TIME: 2025-02-26 09:27:06 -- STEP: 36922/36938 -- GLOBAL_STEP: 1182000 | > loss_text_ce: 0.029277343302965164 (0.031717677576643254) | > loss_mel_ce: 0.8158348202705383 (nan) | > loss: 0.0033536197151988745 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1369524958668239) | > loader_time: 0.0045 (0.0046320080427816115)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.02052762248728535 (-0.00047241107071981786) | > avg_loss_text_ce: 0.030886289691275888 (-0.00016246815350386135) | > avg_loss_mel_ce: 2.470547062040556 (-0.0016680708025944568) | > avg_loss: 2.5014333565636457 (-0.001830538015552996) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_1182016.pth  > EPOCH: 33/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-26 09:27:34)   --> TIME: 2025-02-26 09:27:41 -- STEP: 34/36938 -- GLOBAL_STEP: 1182050 | > loss_text_ce: 0.025090239942073822 (0.0315269167699358) | > loss_mel_ce: 1.8607102632522583 (2.37932827559245) | > loss: 0.007483335677534342 (0.009566886283316896) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13102290209601908) | > loader_time: 0.0047 (0.004932010875028722)  --> TIME: 2025-02-26 09:27:50 -- STEP: 84/36938 -- GLOBAL_STEP: 1182100 | > loss_text_ce: 0.03324155509471893 (0.03111016040756589) | > loss_mel_ce: 1.3455148935317993 (nan) | > loss: 0.005471256095916033 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13195298399244038) | > loader_time: 0.0048 (0.004877121675582157)  --> TIME: 2025-02-26 09:27:59 -- STEP: 134/36938 -- GLOBAL_STEP: 1182150 | > loss_text_ce: 0.03226196765899658 (0.031410312154955836) | > loss_mel_ce: 2.8675971031188965 (nan) | > loss: 0.011507377028465271 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.1330191484138147) | > loader_time: 0.0055 (0.004845645890307069)  --> TIME: 2025-02-26 09:28:08 -- STEP: 184/36938 -- GLOBAL_STEP: 1182200 | > loss_text_ce: 0.033072713762521744 (0.03167484604510601) | > loss_mel_ce: 3.4272303581237793 (nan) | > loss: 0.013731362298130989 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.1341083917928778) | > loader_time: 0.0046 (0.004819910163464755)  --> TIME: 2025-02-26 09:28:18 -- STEP: 234/36938 -- GLOBAL_STEP: 1182250 | > loss_text_ce: 0.031192408874630928 (0.03167033651604866) | > loss_mel_ce: 3.0186562538146973 (nan) | > loss: 0.012102574110031128 (nan) | > current_lr: 5e-06 | > step_time: 0.1793 (0.13531435045421633) | > loader_time: 0.0048 (0.0048057533736921764)  --> TIME: 2025-02-26 09:28:27 -- STEP: 284/36938 -- GLOBAL_STEP: 1182300 | > loss_text_ce: 0.02295028232038021 (0.03165715056615818) | > loss_mel_ce: 2.1136906147003174 (nan) | > loss: 0.008478733710944653 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13598917487641454) | > loader_time: 0.0052 (0.004799388663869507)  --> TIME: 2025-02-26 09:28:36 -- STEP: 334/36938 -- GLOBAL_STEP: 1182350 | > loss_text_ce: 0.029867222532629967 (0.03157907816397397) | > loss_mel_ce: 2.9200596809387207 (nan) | > loss: 0.01170605979859829 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13576374082508194) | > loader_time: 0.0047 (0.004785282883101589)  --> TIME: 2025-02-26 09:28:46 -- STEP: 384/36938 -- GLOBAL_STEP: 1182400 | > loss_text_ce: 0.026053693145513535 (0.03154197571954379) | > loss_mel_ce: 2.6738433837890625 (nan) | > loss: 0.010713878087699413 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1369298187394937) | > loader_time: 0.0045 (0.004783319930235544)  --> TIME: 2025-02-26 09:28:55 -- STEP: 434/36938 -- GLOBAL_STEP: 1182450 | > loss_text_ce: 0.03534829616546631 (0.031508675596166054) | > loss_mel_ce: 3.3720004558563232 (nan) | > loss: 0.013521225191652775 (nan) | > current_lr: 5e-06 | > step_time: 0.1569 (0.13682869392605984) | > loader_time: 0.0046 (0.004767902985146516)  --> TIME: 2025-02-26 09:29:05 -- STEP: 484/36938 -- GLOBAL_STEP: 1182500 | > loss_text_ce: 0.028832243755459785 (0.031483408948791425) | > loss_mel_ce: 3.068578004837036 (nan) | > loss: 0.012291311286389828 (nan) | > current_lr: 5e-06 | > step_time: 0.1434 (0.13714633943620805) | > loader_time: 0.0047 (0.0047562688835396235)  --> TIME: 2025-02-26 09:29:14 -- STEP: 534/36938 -- GLOBAL_STEP: 1182550 | > loss_text_ce: 0.031186189502477646 (0.03138128557845894) | > loss_mel_ce: 2.305870294570923 (nan) | > loss: 0.009274033829569817 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13721039679166544) | > loader_time: 0.0046 (0.004749872711267362)  --> TIME: 2025-02-26 09:29:23 -- STEP: 584/36938 -- GLOBAL_STEP: 1182600 | > loss_text_ce: 0.034844838082790375 (0.031368691097842925) | > loss_mel_ce: 2.0414466857910156 (nan) | > loss: 0.008239252492785454 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.1369445536234608) | > loader_time: 0.0046 (0.004747520570885643)  --> TIME: 2025-02-26 09:29:33 -- STEP: 634/36938 -- GLOBAL_STEP: 1182650 | > loss_text_ce: 0.025875365361571312 (0.03131324957545013) | > loss_mel_ce: 2.6926519870758057 (nan) | > loss: 0.010787807404994965 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13694832828894785) | > loader_time: 0.0045 (0.004746227610374474)  --> TIME: 2025-02-26 09:29:42 -- STEP: 684/36938 -- GLOBAL_STEP: 1182700 | > loss_text_ce: 0.02916383370757103 (0.031267315251426) | > loss_mel_ce: 2.689093589782715 (nan) | > loss: 0.010786736384034157 (nan) | > current_lr: 5e-06 | > step_time: 0.1917 (0.13644278572316765) | > loader_time: 0.0048 (0.004736411989780894)  --> TIME: 2025-02-26 09:29:51 -- STEP: 734/36938 -- GLOBAL_STEP: 1182750 | > loss_text_ce: 0.023617684841156006 (0.031344453970411974) | > loss_mel_ce: 2.217256546020508 (nan) | > loss: 0.008892359212040901 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1363592456407055) | > loader_time: 0.0045 (0.004727581866105832)  --> TIME: 2025-02-26 09:30:00 -- STEP: 784/36938 -- GLOBAL_STEP: 1182800 | > loss_text_ce: 0.04279516637325287 (0.03138116064685761) | > loss_mel_ce: 1.6184122562408447 (nan) | > loss: 0.006592093501240015 (nan) | > current_lr: 5e-06 | > step_time: 0.1073 (0.13618292553084252) | > loader_time: 0.0045 (0.004718116655641676)  --> TIME: 2025-02-26 09:30:09 -- STEP: 834/36938 -- GLOBAL_STEP: 1182850 | > loss_text_ce: 0.031191548332571983 (0.03145743511746724) | > loss_mel_ce: 2.603712320327759 (nan) | > loss: 0.010455968789756298 (nan) | > current_lr: 5e-06 | > step_time: 0.1881 (0.13614938985243705) | > loader_time: 0.0049 (0.004710904414133491)  --> TIME: 2025-02-26 09:30:18 -- STEP: 884/36938 -- GLOBAL_STEP: 1182900 | > loss_text_ce: 0.03381245583295822 (0.03142522553896444) | > loss_mel_ce: 3.061037063598633 (nan) | > loss: 0.012281149625778198 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13592684268951433) | > loader_time: 0.0046 (0.0047077237211201545)  --> TIME: 2025-02-26 09:30:27 -- STEP: 934/36938 -- GLOBAL_STEP: 1182950 | > loss_text_ce: 0.027736373245716095 (0.03147281087013623) | > loss_mel_ce: 2.9493844509124756 (nan) | > loss: 0.011813972145318985 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.13579804458250597) | > loader_time: 0.0047 (0.004704447834088411)  --> TIME: 2025-02-26 09:30:37 -- STEP: 984/36938 -- GLOBAL_STEP: 1183000 | > loss_text_ce: 0.023222899064421654 (0.03149109529009533) | > loss_mel_ce: 2.2313125133514404 (nan) | > loss: 0.008946569636464119 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13587735750810898) | > loader_time: 0.0046 (0.004702927862725609)  --> TIME: 2025-02-26 09:30:46 -- STEP: 1034/36938 -- GLOBAL_STEP: 1183050 | > loss_text_ce: 0.026538969948887825 (0.03151443675142399) | > loss_mel_ce: 2.3901984691619873 (nan) | > loss: 0.009590229019522667 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13590318206661686) | > loader_time: 0.0047 (0.004703160180805732)  --> TIME: 2025-02-26 09:30:55 -- STEP: 1084/36938 -- GLOBAL_STEP: 1183100 | > loss_text_ce: 0.02873740904033184 (0.03150625197591465) | > loss_mel_ce: 2.2303590774536133 (nan) | > loss: 0.008964668959379196 (nan) | > current_lr: 5e-06 | > step_time: 0.1693 (0.13565438699898255) | > loader_time: 0.0048 (0.004699657763942143)  --> TIME: 2025-02-26 09:31:05 -- STEP: 1134/36938 -- GLOBAL_STEP: 1183150 | > loss_text_ce: 0.02862825058400631 (0.03155840523590255) | > loss_mel_ce: 3.0015952587127686 (nan) | > loss: 0.012024697847664356 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.1359655035025651) | > loader_time: 0.0049 (0.0046988349322499)  --> TIME: 2025-02-26 09:31:14 -- STEP: 1184/36938 -- GLOBAL_STEP: 1183200 | > loss_text_ce: 0.027575206011533737 (0.03152813473197233) | > loss_mel_ce: 2.322096347808838 (nan) | > loss: 0.009324094280600548 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13626040035002962) | > loader_time: 0.0047 (0.004697860696831262)  --> TIME: 2025-02-26 09:31:24 -- STEP: 1234/36938 -- GLOBAL_STEP: 1183250 | > loss_text_ce: 0.032194919884204865 (0.031550840022588844) | > loss_mel_ce: 2.674241065979004 (nan) | > loss: 0.010739825665950775 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.1363466375264396) | > loader_time: 0.0057 (0.00469967920930884)  --> TIME: 2025-02-26 09:31:33 -- STEP: 1284/36938 -- GLOBAL_STEP: 1183300 | > loss_text_ce: 0.031318310648202896 (0.03155104599594262) | > loss_mel_ce: 3.37068247795105 (nan) | > loss: 0.013500004075467587 (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.13638797188845017) | > loader_time: 0.0046 (0.004702662938851801)  --> TIME: 2025-02-26 09:31:42 -- STEP: 1334/36938 -- GLOBAL_STEP: 1183350 | > loss_text_ce: 0.029631754383444786 (0.03159479451096289) | > loss_mel_ce: 2.716181516647339 (nan) | > loss: 0.010896085761487484 (nan) | > current_lr: 5e-06 | > step_time: 0.1528 (0.13612749247715394) | > loader_time: 0.0047 (0.004701214215566012)  --> TIME: 2025-02-26 09:31:51 -- STEP: 1384/36938 -- GLOBAL_STEP: 1183400 | > loss_text_ce: 0.03249417617917061 (0.0315769922770183) | > loss_mel_ce: 1.9589948654174805 (nan) | > loss: 0.007902734912931919 (nan) | > current_lr: 5e-06 | > step_time: 0.161 (0.13621152377542042) | > loader_time: 0.0048 (0.004699299790266615)  --> TIME: 2025-02-26 09:32:01 -- STEP: 1434/36938 -- GLOBAL_STEP: 1183450 | > loss_text_ce: 0.02709101140499115 (0.031608860442905874) | > loss_mel_ce: 1.6291447877883911 (nan) | > loss: 0.006572364829480648 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13624344261121554) | > loader_time: 0.0053 (0.0047004621896783606)  --> TIME: 2025-02-26 09:32:10 -- STEP: 1484/36938 -- GLOBAL_STEP: 1183500 | > loss_text_ce: 0.023767951875925064 (0.03162768063125403) | > loss_mel_ce: 2.6664111614227295 (nan) | > loss: 0.01067531481385231 (nan) | > current_lr: 5e-06 | > step_time: 0.1662 (0.13646953173403478) | > loader_time: 0.0048 (0.0046989459875780215)  --> TIME: 2025-02-26 09:32:20 -- STEP: 1534/36938 -- GLOBAL_STEP: 1183550 | > loss_text_ce: 0.03583170473575592 (0.031686541565476574) | > loss_mel_ce: 3.160954713821411 (nan) | > loss: 0.01268566120415926 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.13663733471792092) | > loader_time: 0.0048 (0.004700652312735055)  --> TIME: 2025-02-26 09:32:29 -- STEP: 1584/36938 -- GLOBAL_STEP: 1183600 | > loss_text_ce: 0.03816498816013336 (0.03170715053060158) | > loss_mel_ce: 2.5299015045166016 (nan) | > loss: 0.010190741159021854 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.13655683307936708) | > loader_time: 0.0047 (0.004702520791930371)  --> TIME: 2025-02-26 09:32:38 -- STEP: 1634/36938 -- GLOBAL_STEP: 1183650 | > loss_text_ce: 0.029658939689397812 (0.03168383257627725) | > loss_mel_ce: 2.12060546875 (nan) | > loss: 0.008532796055078506 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13645363948336442) | > loader_time: 0.0044 (0.004701383402764861)  --> TIME: 2025-02-26 09:32:48 -- STEP: 1684/36938 -- GLOBAL_STEP: 1183700 | > loss_text_ce: 0.032581131905317307 (0.03166264726452958) | > loss_mel_ce: 2.137913942337036 (nan) | > loss: 0.00861307606101036 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13653339852629828) | > loader_time: 0.0053 (0.004705618084751223)  --> TIME: 2025-02-26 09:32:57 -- STEP: 1734/36938 -- GLOBAL_STEP: 1183750 | > loss_text_ce: 0.031833723187446594 (0.0316863322459729) | > loss_mel_ce: 2.820613384246826 (nan) | > loss: 0.01131923496723175 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.13664377505666536) | > loader_time: 0.005 (0.004705776942779992)  --> TIME: 2025-02-26 09:33:07 -- STEP: 1784/36938 -- GLOBAL_STEP: 1183800 | > loss_text_ce: 0.032668184489011765 (0.031671370872956336) | > loss_mel_ce: 0.5468215942382812 (nan) | > loss: 0.002299562795087695 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.1366665494281615) | > loader_time: 0.0047 (0.004708710272750507)  --> TIME: 2025-02-26 09:33:16 -- STEP: 1834/36938 -- GLOBAL_STEP: 1183850 | > loss_text_ce: 0.03493723273277283 (0.03166400605910178) | > loss_mel_ce: 0.3865898549556732 (nan) | > loss: 0.0016727265901863575 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13662693323719868) | > loader_time: 0.0045 (0.004709813951015987)  --> TIME: 2025-02-26 09:33:25 -- STEP: 1884/36938 -- GLOBAL_STEP: 1183900 | > loss_text_ce: 0.035140275955200195 (0.03166196716112944) | > loss_mel_ce: 1.9845753908157349 (nan) | > loss: 0.008014746010303497 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.13661571786661827) | > loader_time: 0.0047 (0.004708199490913661)  --> TIME: 2025-02-26 09:33:35 -- STEP: 1934/36938 -- GLOBAL_STEP: 1183950 | > loss_text_ce: 0.035108402371406555 (0.03172488569021006) | > loss_mel_ce: 0.0296630859375 (nan) | > loss: 0.00025702972197905183 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.13658627124952075) | > loader_time: 0.0047 (0.0047078512323925825)  --> TIME: 2025-02-26 09:33:44 -- STEP: 1984/36938 -- GLOBAL_STEP: 1184000 | > loss_text_ce: 0.02709970436990261 (0.03170160749587314) | > loss_mel_ce: 2.378232479095459 (nan) | > loss: 0.009544969536364079 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13658384401952078) | > loader_time: 0.0046 (0.004707906034684945)  --> TIME: 2025-02-26 09:33:53 -- STEP: 2034/36938 -- GLOBAL_STEP: 1184050 | > loss_text_ce: 0.02788653038442135 (0.031698413697348944) | > loss_mel_ce: 2.935945510864258 (nan) | > loss: 0.011761239729821682 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.1365152961971018) | > loader_time: 0.0046 (0.004706231893691336)  --> TIME: 2025-02-26 09:34:02 -- STEP: 2084/36938 -- GLOBAL_STEP: 1184100 | > loss_text_ce: 0.0340246707201004 (0.03166179073395328) | > loss_mel_ce: 3.2252182960510254 (nan) | > loss: 0.012933504767715931 (nan) | > current_lr: 5e-06 | > step_time: 0.1691 (0.13647863617785386) | > loader_time: 0.0048 (0.0047048199886095)  --> TIME: 2025-02-26 09:34:11 -- STEP: 2134/36938 -- GLOBAL_STEP: 1184150 | > loss_text_ce: 0.041235554963350296 (0.031645419751665016) | > loss_mel_ce: 1.2450588941574097 (nan) | > loss: 0.00510434340685606 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13631669695174026) | > loader_time: 0.0046 (0.0047024110822668636)  --> TIME: 2025-02-26 09:34:21 -- STEP: 2184/36938 -- GLOBAL_STEP: 1184200 | > loss_text_ce: 0.037205662578344345 (0.03161450660695424) | > loss_mel_ce: 3.2472164630889893 (nan) | > loss: 0.013033421710133553 (nan) | > current_lr: 5e-06 | > step_time: 0.1831 (0.1363993780516881) | > loader_time: 0.0048 (0.00470122127305893)  --> TIME: 2025-02-26 09:34:30 -- STEP: 2234/36938 -- GLOBAL_STEP: 1184250 | > loss_text_ce: 0.03382519260048866 (0.03161011960010588) | > loss_mel_ce: 2.640307903289795 (nan) | > loss: 0.010611639358103275 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13645840737044887) | > loader_time: 0.0045 (0.004699903080905234)  --> TIME: 2025-02-26 09:34:40 -- STEP: 2284/36938 -- GLOBAL_STEP: 1184300 | > loss_text_ce: 0.032609861344099045 (0.03162755847104465) | > loss_mel_ce: 3.103160858154297 (nan) | > loss: 0.012443535961210728 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1364747225298776) | > loader_time: 0.0046 (0.004699489072408852)  --> TIME: 2025-02-26 09:34:49 -- STEP: 2334/36938 -- GLOBAL_STEP: 1184350 | > loss_text_ce: 0.02398342825472355 (0.03162566446735122) | > loss_mel_ce: 2.309596538543701 (nan) | > loss: 0.009260239079594612 (nan) | > current_lr: 5e-06 | > step_time: 0.1629 (0.13638168210746313) | > loader_time: 0.0048 (0.0046987115774996336)  --> TIME: 2025-02-26 09:34:58 -- STEP: 2384/36938 -- GLOBAL_STEP: 1184400 | > loss_text_ce: 0.0373210571706295 (0.031633809519831704) | > loss_mel_ce: 2.9232654571533203 (nan) | > loss: 0.01174835953861475 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13634577113510005) | > loader_time: 0.0048 (0.004697697374644696)  --> TIME: 2025-02-26 09:35:07 -- STEP: 2434/36938 -- GLOBAL_STEP: 1184450 | > loss_text_ce: 0.029930664226412773 (0.03162856742926406) | > loss_mel_ce: 2.88918399810791 (nan) | > loss: 0.011583788320422173 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1362855670485424) | > loader_time: 0.0047 (0.004699568329233566)  --> TIME: 2025-02-26 09:35:16 -- STEP: 2484/36938 -- GLOBAL_STEP: 1184500 | > loss_text_ce: 0.026304077357053757 (0.031619404367622735) | > loss_mel_ce: 1.3983176946640015 (nan) | > loss: 0.005653261672705412 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1362625094619544) | > loader_time: 0.0046 (0.004700897396474649)  --> TIME: 2025-02-26 09:35:26 -- STEP: 2534/36938 -- GLOBAL_STEP: 1184550 | > loss_text_ce: 0.034359224140644073 (0.03160630833672184) | > loss_mel_ce: 2.419283628463745 (nan) | > loss: 0.009736678563058376 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13629101116824752) | > loader_time: 0.0065 (0.004703031907235911)  --> TIME: 2025-02-26 09:35:35 -- STEP: 2584/36938 -- GLOBAL_STEP: 1184600 | > loss_text_ce: 0.026876088231801987 (0.03161728617464177) | > loss_mel_ce: 2.1013400554656982 (nan) | > loss: 0.00844530202448368 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13626546516507046) | > loader_time: 0.0045 (0.004702308137350409)  --> TIME: 2025-02-26 09:35:44 -- STEP: 2634/36938 -- GLOBAL_STEP: 1184650 | > loss_text_ce: 0.030269600450992584 (0.03160690487134039) | > loss_mel_ce: 3.1123650074005127 (nan) | > loss: 0.012470773421227932 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13633240301136565) | > loader_time: 0.0046 (0.004701586410505084)  --> TIME: 2025-02-26 09:35:53 -- STEP: 2684/36938 -- GLOBAL_STEP: 1184700 | > loss_text_ce: 0.03235454112291336 (0.03161264785243435) | > loss_mel_ce: 3.5599241256713867 (nan) | > loss: 0.01425507478415966 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.1362210449980554) | > loader_time: 0.0046 (0.004698872477393571)  --> TIME: 2025-02-26 09:36:03 -- STEP: 2734/36938 -- GLOBAL_STEP: 1184750 | > loss_text_ce: 0.03114808164536953 (0.03162483244435359) | > loss_mel_ce: 2.7940986156463623 (nan) | > loss: 0.01121129747480154 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.1362666516035426) | > loader_time: 0.0045 (0.004697072130969257)  --> TIME: 2025-02-26 09:36:12 -- STEP: 2784/36938 -- GLOBAL_STEP: 1184800 | > loss_text_ce: 0.015014453791081905 (0.03161024754122978) | > loss_mel_ce: 1.898066520690918 (nan) | > loss: 0.007591591216623783 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13626116214469935) | > loader_time: 0.0045 (0.00469529877106349)  --> TIME: 2025-02-26 09:36:21 -- STEP: 2834/36938 -- GLOBAL_STEP: 1184850 | > loss_text_ce: 0.028571616858243942 (0.03162765220838201) | > loss_mel_ce: 2.725611686706543 (nan) | > loss: 0.010929299518465996 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13625415985064346) | > loader_time: 0.0047 (0.004693910279741744)  --> TIME: 2025-02-26 09:36:30 -- STEP: 2884/36938 -- GLOBAL_STEP: 1184900 | > loss_text_ce: 0.028145987540483475 (0.0316305054447275) | > loss_mel_ce: 3.151850700378418 (nan) | > loss: 0.012619035318493843 (nan) | > current_lr: 5e-06 | > step_time: 0.1456 (0.13624227352843366) | > loader_time: 0.0047 (0.004693420187940881)  --> TIME: 2025-02-26 09:36:40 -- STEP: 2934/36938 -- GLOBAL_STEP: 1184950 | > loss_text_ce: 0.023819275200366974 (0.03163122939455047) | > loss_mel_ce: 2.8411409854888916 (nan) | > loss: 0.011368890292942524 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13623867249440072) | > loader_time: 0.0044 (0.004691937624717461)  --> TIME: 2025-02-26 09:36:49 -- STEP: 2984/36938 -- GLOBAL_STEP: 1185000 | > loss_text_ce: 0.027978619560599327 (0.03163598554930473) | > loss_mel_ce: 2.7785091400146484 (nan) | > loss: 0.011136856861412525 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13620402434236553) | > loader_time: 0.0046 (0.004690860135945177)  --> TIME: 2025-02-26 09:36:58 -- STEP: 3034/36938 -- GLOBAL_STEP: 1185050 | > loss_text_ce: 0.024818817153573036 (0.03162720410310445) | > loss_mel_ce: 2.698437452316284 (nan) | > loss: 0.010806573554873466 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13608875727292025) | > loader_time: 0.0053 (0.004691631918135803)  --> TIME: 2025-02-26 09:37:07 -- STEP: 3084/36938 -- GLOBAL_STEP: 1185100 | > loss_text_ce: 0.03362719714641571 (0.031622166461800605) | > loss_mel_ce: 2.8676674365997314 (nan) | > loss: 0.011513074859976768 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13600412664401082) | > loader_time: 0.0046 (0.00469228822433531)  --> TIME: 2025-02-26 09:37:16 -- STEP: 3134/36938 -- GLOBAL_STEP: 1185150 | > loss_text_ce: 0.0452234260737896 (0.0316351273818621) | > loss_mel_ce: 0.067535400390625 (nan) | > loss: 0.0004474557063076645 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13604863867610595) | > loader_time: 0.0045 (0.004691713542487868)  --> TIME: 2025-02-26 09:37:25 -- STEP: 3184/36938 -- GLOBAL_STEP: 1185200 | > loss_text_ce: 0.03082343004643917 (0.03160569926603333) | > loss_mel_ce: 2.5080606937408447 (nan) | > loss: 0.010074937716126442 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13602427696462846) | > loader_time: 0.0046 (0.004691636442539083)  --> TIME: 2025-02-26 09:37:35 -- STEP: 3234/36938 -- GLOBAL_STEP: 1185250 | > loss_text_ce: 0.0339834950864315 (0.0315956687764942) | > loss_mel_ce: 1.4442976713180542 (nan) | > loss: 0.0058661955408751965 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1360582900622336) | > loader_time: 0.0045 (0.004691912424586481)  --> TIME: 2025-02-26 09:37:44 -- STEP: 3284/36938 -- GLOBAL_STEP: 1185300 | > loss_text_ce: 0.02994147501885891 (0.03159840834490397) | > loss_mel_ce: 2.5493288040161133 (nan) | > loss: 0.010235200636088848 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13606620481040363) | > loader_time: 0.005 (0.00469108098317006)  --> TIME: 2025-02-26 09:37:53 -- STEP: 3334/36938 -- GLOBAL_STEP: 1185350 | > loss_text_ce: 0.04029001668095589 (0.03160040145996914) | > loss_mel_ce: 2.45253324508667 (nan) | > loss: 0.009892156347632408 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13607140880326113) | > loader_time: 0.0045 (0.004690175079341136)  --> TIME: 2025-02-26 09:38:03 -- STEP: 3384/36938 -- GLOBAL_STEP: 1185400 | > loss_text_ce: 0.04283265396952629 (0.03160858434629381) | > loss_mel_ce: 2.776914358139038 (nan) | > loss: 0.011189472861588001 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.1361604006718798) | > loader_time: 0.0047 (0.004690929859806188)  --> TIME: 2025-02-26 09:38:12 -- STEP: 3434/36938 -- GLOBAL_STEP: 1185450 | > loss_text_ce: 0.03108280524611473 (0.03162123611273907) | > loss_mel_ce: 2.3051371574401855 (nan) | > loss: 0.009270714595913887 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13618265020340478) | > loader_time: 0.0046 (0.004690320254344206)  --> TIME: 2025-02-26 09:38:21 -- STEP: 3484/36938 -- GLOBAL_STEP: 1185500 | > loss_text_ce: 0.03818530589342117 (0.031632108884905356) | > loss_mel_ce: 2.0659003257751465 (nan) | > loss: 0.008349547162652016 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13612924842965857) | > loader_time: 0.0045 (0.004689762499247582)  --> TIME: 2025-02-26 09:38:31 -- STEP: 3534/36938 -- GLOBAL_STEP: 1185550 | > loss_text_ce: 0.03413985297083855 (0.03164722681273232) | > loss_mel_ce: 1.6593164205551147 (nan) | > loss: 0.006720065139234066 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13614815942432892) | > loader_time: 0.0044 (0.004689604465612002)  --> TIME: 2025-02-26 09:38:40 -- STEP: 3584/36938 -- GLOBAL_STEP: 1185600 | > loss_text_ce: 0.03426985815167427 (0.031630374462630687) | > loss_mel_ce: 3.0152103900909424 (nan) | > loss: 0.012101112864911556 (nan) | > current_lr: 5e-06 | > step_time: 0.1613 (0.13614916881280276) | > loader_time: 0.0047 (0.004689089953899393)  --> TIME: 2025-02-26 09:38:49 -- STEP: 3634/36938 -- GLOBAL_STEP: 1185650 | > loss_text_ce: 0.04177552089095116 (0.03163606187981703) | > loss_mel_ce: 1.5780662298202515 (nan) | > loss: 0.006427943706512451 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13617795466326973) | > loader_time: 0.0046 (0.0046890118519683915)  --> TIME: 2025-02-26 09:38:59 -- STEP: 3684/36938 -- GLOBAL_STEP: 1185700 | > loss_text_ce: 0.03210730850696564 (0.03161010708069346) | > loss_mel_ce: 2.6836423873901367 (nan) | > loss: 0.01077678520232439 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13626028168602686) | > loader_time: 0.0047 (0.004688441753387461)  --> TIME: 2025-02-26 09:39:08 -- STEP: 3734/36938 -- GLOBAL_STEP: 1185750 | > loss_text_ce: 0.03142267093062401 (0.031619950770298236) | > loss_mel_ce: 2.848822832107544 (nan) | > loss: 0.011429546400904655 (nan) | > current_lr: 5e-06 | > step_time: 0.1565 (0.13625748762806011) | > loader_time: 0.0048 (0.004687734957870223)  --> TIME: 2025-02-26 09:39:18 -- STEP: 3784/36938 -- GLOBAL_STEP: 1185800 | > loss_text_ce: 0.03175068646669388 (0.03162654512463415) | > loss_mel_ce: 2.885852336883545 (nan) | > loss: 0.011577790603041649 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13624238860783555) | > loader_time: 0.0045 (0.004686273618682)  --> TIME: 2025-02-26 09:39:27 -- STEP: 3834/36938 -- GLOBAL_STEP: 1185850 | > loss_text_ce: 0.030871739611029625 (0.03163920236332664) | > loss_mel_ce: 2.714890718460083 (nan) | > loss: 0.010895882733166218 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13626302009954624) | > loader_time: 0.0045 (0.004684999390822506)  --> TIME: 2025-02-26 09:39:36 -- STEP: 3884/36938 -- GLOBAL_STEP: 1185900 | > loss_text_ce: 0.02616235241293907 (0.031640863520823265) | > loss_mel_ce: 0.4814707338809967 (nan) | > loss: 0.002014417201280594 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.1362349084673408) | > loader_time: 0.0046 (0.004683750542504176)  --> TIME: 2025-02-26 09:39:45 -- STEP: 3934/36938 -- GLOBAL_STEP: 1185950 | > loss_text_ce: 0.03533872216939926 (0.031625022988124474) | > loss_mel_ce: 2.4744386672973633 (nan) | > loss: 0.009959434159100056 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1361947723857993) | > loader_time: 0.0046 (0.004682559741591834)  --> TIME: 2025-02-26 09:39:54 -- STEP: 3984/36938 -- GLOBAL_STEP: 1186000 | > loss_text_ce: 0.037869445979595184 (0.031609723367821876) | > loss_mel_ce: 2.7060546875 (nan) | > loss: 0.01088858861476183 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13616090677828183) | > loader_time: 0.0046 (0.004682076503952832)  --> TIME: 2025-02-26 09:40:04 -- STEP: 4034/36938 -- GLOBAL_STEP: 1186050 | > loss_text_ce: 0.02353832684457302 (0.031600256397374626) | > loss_mel_ce: 0.04764765128493309 (nan) | > loss: 0.00028248404851183295 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.1361239357755865) | > loader_time: 0.0047 (0.004681506958348251)  --> TIME: 2025-02-26 09:40:13 -- STEP: 4084/36938 -- GLOBAL_STEP: 1186100 | > loss_text_ce: 0.026044806465506554 (0.03160127498351781) | > loss_mel_ce: 2.1222150325775146 (nan) | > loss: 0.008524841628968716 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13614621060835616) | > loader_time: 0.0046 (0.004681602047426119)  --> TIME: 2025-02-26 09:40:22 -- STEP: 4134/36938 -- GLOBAL_STEP: 1186150 | > loss_text_ce: 0.032169755548238754 (0.03158277601078447) | > loss_mel_ce: 3.438741445541382 (nan) | > loss: 0.013773458078503609 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.13619124451061915) | > loader_time: 0.0047 (0.0046810273335184255)  --> TIME: 2025-02-26 09:40:32 -- STEP: 4184/36938 -- GLOBAL_STEP: 1186200 | > loss_text_ce: 0.031012287363409996 (0.03158935349293052) | > loss_mel_ce: 1.8046468496322632 (nan) | > loss: 0.007284362334758043 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13620854888545847) | > loader_time: 0.0049 (0.004680572572675318)  --> TIME: 2025-02-26 09:40:41 -- STEP: 4234/36938 -- GLOBAL_STEP: 1186250 | > loss_text_ce: 0.0339137464761734 (0.031595534114250735) | > loss_mel_ce: 2.7805914878845215 (nan) | > loss: 0.01116867270320654 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13620742419219878) | > loader_time: 0.0046 (0.004680141954417501)  --> TIME: 2025-02-26 09:40:51 -- STEP: 4284/36938 -- GLOBAL_STEP: 1186300 | > loss_text_ce: 0.030620358884334564 (0.031594404331812834) | > loss_mel_ce: 3.2473669052124023 (nan) | > loss: 0.013007886707782745 (nan) | > current_lr: 5e-06 | > step_time: 0.175 (0.13626653456888274) | > loader_time: 0.0049 (0.00468003571756971)  --> TIME: 2025-02-26 09:41:00 -- STEP: 4334/36938 -- GLOBAL_STEP: 1186350 | > loss_text_ce: 0.03717733919620514 (0.031595374430247465) | > loss_mel_ce: 2.4409358501434326 (nan) | > loss: 0.00983378291130066 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13624415350334393) | > loader_time: 0.0046 (0.0046796691984309635)  --> TIME: 2025-02-26 09:41:09 -- STEP: 4384/36938 -- GLOBAL_STEP: 1186400 | > loss_text_ce: 0.03742159903049469 (0.03159995445149725) | > loss_mel_ce: 3.2217540740966797 (nan) | > loss: 0.012933238409459591 (nan) | > current_lr: 5e-06 | > step_time: 0.1851 (0.13624868090570397) | > loader_time: 0.005 (0.004679184216652483)  --> TIME: 2025-02-26 09:41:19 -- STEP: 4434/36938 -- GLOBAL_STEP: 1186450 | > loss_text_ce: 0.027430973947048187 (0.03160500135310553) | > loss_mel_ce: 2.391143321990967 (nan) | > loss: 0.009597517549991608 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1362799805029049) | > loader_time: 0.0045 (0.00467855219481817)  --> TIME: 2025-02-26 09:41:28 -- STEP: 4484/36938 -- GLOBAL_STEP: 1186500 | > loss_text_ce: 0.03299244865775108 (0.031619082553724295) | > loss_mel_ce: 2.6912906169891357 (nan) | > loss: 0.010810647159814835 (nan) | > current_lr: 5e-06 | > step_time: 0.1442 (0.13624649936869987) | > loader_time: 0.0047 (0.004677719031988144)  --> TIME: 2025-02-26 09:41:37 -- STEP: 4534/36938 -- GLOBAL_STEP: 1186550 | > loss_text_ce: 0.03668676316738129 (0.031632512043931554) | > loss_mel_ce: 2.9656243324279785 (nan) | > loss: 0.011913932859897614 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.1362124764556446) | > loader_time: 0.0046 (0.004677184152876599)  --> TIME: 2025-02-26 09:41:46 -- STEP: 4584/36938 -- GLOBAL_STEP: 1186600 | > loss_text_ce: 0.03405251353979111 (0.03162871190122487) | > loss_mel_ce: 3.137965440750122 (nan) | > loss: 0.012587374076247215 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.1361706683877041) | > loader_time: 0.0046 (0.004677191922801961)  --> TIME: 2025-02-26 09:41:55 -- STEP: 4634/36938 -- GLOBAL_STEP: 1186650 | > loss_text_ce: 0.03230401501059532 (0.031628732477066816) | > loss_mel_ce: 3.1633200645446777 (nan) | > loss: 0.01268104836344719 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.1361400307794041) | > loader_time: 0.0048 (0.004676674633742316)  --> TIME: 2025-02-26 09:42:05 -- STEP: 4684/36938 -- GLOBAL_STEP: 1186700 | > loss_text_ce: 0.028895841911435127 (0.03162050275823998) | > loss_mel_ce: 2.0809478759765625 (nan) | > loss: 0.008372396230697632 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13618571416217962) | > loader_time: 0.0046 (0.004676888276123789)  --> TIME: 2025-02-26 09:42:15 -- STEP: 4734/36938 -- GLOBAL_STEP: 1186750 | > loss_text_ce: 0.03016175888478756 (0.031625867591101475) | > loss_mel_ce: 2.8479275703430176 (nan) | > loss: 0.011420990340411663 (nan) | > current_lr: 5e-06 | > step_time: 0.177 (0.13625753721608627) | > loader_time: 0.0048 (0.004677107881079471)  --> TIME: 2025-02-26 09:42:24 -- STEP: 4784/36938 -- GLOBAL_STEP: 1186800 | > loss_text_ce: 0.029799574986100197 (0.03162074494284638) | > loss_mel_ce: 1.78656005859375 (nan) | > loss: 0.007207776885479689 (nan) | > current_lr: 5e-06 | > step_time: 0.1621 (0.13626109397530917) | > loader_time: 0.0046 (0.004676968357634796)  --> TIME: 2025-02-26 09:42:33 -- STEP: 4834/36938 -- GLOBAL_STEP: 1186850 | > loss_text_ce: 0.028705738484859467 (0.03163332288432066) | > loss_mel_ce: 1.35550057888031 (nan) | > loss: 0.005492882337421179 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.1362219679981997) | > loader_time: 0.0046 (0.004676634928979916)  --> TIME: 2025-02-26 09:42:42 -- STEP: 4884/36938 -- GLOBAL_STEP: 1186900 | > loss_text_ce: 0.02798311598598957 (0.03161970748971121) | > loss_mel_ce: 1.6219865083694458 (nan) | > loss: 0.006547498516738415 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1362285326974102) | > loader_time: 0.0045 (0.004676135322483339)  --> TIME: 2025-02-26 09:42:51 -- STEP: 4934/36938 -- GLOBAL_STEP: 1186950 | > loss_text_ce: 0.03312286362051964 (0.0316101039945174) | > loss_mel_ce: 2.3666608333587646 (nan) | > loss: 0.00952295120805502 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13620737341187641) | > loader_time: 0.0045 (0.004675372728317644)  --> TIME: 2025-02-26 09:43:01 -- STEP: 4984/36938 -- GLOBAL_STEP: 1187000 | > loss_text_ce: 0.040611155331134796 (0.031615806878043655) | > loss_mel_ce: 3.0919506549835205 (nan) | > loss: 0.01243080198764801 (nan) | > current_lr: 5e-06 | > step_time: 0.1797 (0.13622327333085962) | > loader_time: 0.0047 (0.004674641651670984)  --> TIME: 2025-02-26 09:43:10 -- STEP: 5034/36938 -- GLOBAL_STEP: 1187050 | > loss_text_ce: 0.029064932838082314 (0.0316137858726792) | > loss_mel_ce: 3.243913412094116 (nan) | > loss: 0.012988009490072727 (nan) | > current_lr: 5e-06 | > step_time: 0.1943 (0.136232399069036) | > loader_time: 0.0049 (0.004674169910205104)  --> TIME: 2025-02-26 09:43:20 -- STEP: 5084/36938 -- GLOBAL_STEP: 1187100 | > loss_text_ce: 0.025692781433463097 (0.03161877201046478) | > loss_mel_ce: 2.294959545135498 (nan) | > loss: 0.009208938106894493 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13625661117837307) | > loader_time: 0.0047 (0.004673801755267384)  --> TIME: 2025-02-26 09:43:29 -- STEP: 5134/36938 -- GLOBAL_STEP: 1187150 | > loss_text_ce: 0.03266960009932518 (0.031616349834085296) | > loss_mel_ce: 3.2151951789855957 (nan) | > loss: 0.012888352386653423 (nan) | > current_lr: 5e-06 | > step_time: 0.189 (0.13622491439585996) | > loader_time: 0.0049 (0.004673009258696775)  --> TIME: 2025-02-26 09:43:38 -- STEP: 5184/36938 -- GLOBAL_STEP: 1187200 | > loss_text_ce: 0.031007399782538414 (0.03161153772466439) | > loss_mel_ce: 3.236128807067871 (nan) | > loss: 0.012964826077222824 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13623924801746976) | > loader_time: 0.0044 (0.0046724183139977785)  --> TIME: 2025-02-26 09:43:47 -- STEP: 5234/36938 -- GLOBAL_STEP: 1187250 | > loss_text_ce: 0.025957807898521423 (0.031605702871154266) | > loss_mel_ce: 2.308588981628418 (nan) | > loss: 0.009264075197279453 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13623014355507498) | > loader_time: 0.0046 (0.004671693804795217)  --> TIME: 2025-02-26 09:43:57 -- STEP: 5284/36938 -- GLOBAL_STEP: 1187300 | > loss_text_ce: 0.02750568650662899 (0.03160775765314146) | > loss_mel_ce: 1.3856210708618164 (nan) | > loss: 0.005607645958662033 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13625709014260137) | > loader_time: 0.0045 (0.004671020863364821)  --> TIME: 2025-02-26 09:44:06 -- STEP: 5334/36938 -- GLOBAL_STEP: 1187350 | > loss_text_ce: 0.026135480031371117 (0.031608873982447626) | > loss_mel_ce: 2.287086009979248 (nan) | > loss: 0.009179450571537018 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13622885569231646) | > loader_time: 0.0045 (0.004670222644999012)  --> TIME: 2025-02-26 09:44:15 -- STEP: 5384/36938 -- GLOBAL_STEP: 1187400 | > loss_text_ce: 0.029384110122919083 (0.03160650124060712) | > loss_mel_ce: 3.25195574760437 (nan) | > loss: 0.013021190650761127 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.13627140859438067) | > loader_time: 0.0044 (0.004669487830611263)  --> TIME: 2025-02-26 09:44:25 -- STEP: 5434/36938 -- GLOBAL_STEP: 1187450 | > loss_text_ce: 0.03806062415242195 (0.03161777240348617) | > loss_mel_ce: 2.323668956756592 (nan) | > loss: 0.009371943771839142 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1363328645279357) | > loader_time: 0.0045 (0.004668965688908336)  --> TIME: 2025-02-26 09:44:35 -- STEP: 5484/36938 -- GLOBAL_STEP: 1187500 | > loss_text_ce: 0.03894021362066269 (0.03161329361267075) | > loss_mel_ce: 2.936497449874878 (nan) | > loss: 0.011807292699813843 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.13637524299774614) | > loader_time: 0.0048 (0.004668516672761846)  --> TIME: 2025-02-26 09:44:44 -- STEP: 5534/36938 -- GLOBAL_STEP: 1187550 | > loss_text_ce: 0.0210415069013834 (0.03161938291616583) | > loss_mel_ce: 2.2153093814849854 (nan) | > loss: 0.008874407969415188 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1363958250886057) | > loader_time: 0.0047 (0.004668339521479096)  --> TIME: 2025-02-26 09:44:54 -- STEP: 5584/36938 -- GLOBAL_STEP: 1187600 | > loss_text_ce: 0.03693036735057831 (0.03162693055273226) | > loss_mel_ce: 2.898513078689575 (nan) | > loss: 0.011648586019873619 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1363845088847387) | > loader_time: 0.0045 (0.004667812184481372)  --> TIME: 2025-02-26 09:45:03 -- STEP: 5634/36938 -- GLOBAL_STEP: 1187650 | > loss_text_ce: 0.034353237599134445 (0.031643528783516076) | > loss_mel_ce: 0.07415771484375 (nan) | > loss: 0.0004305990587454289 (nan) | > current_lr: 5e-06 | > step_time: 0.1722 (0.13643031180391588) | > loader_time: 0.0048 (0.004667709218215819)  --> TIME: 2025-02-26 09:45:13 -- STEP: 5684/36938 -- GLOBAL_STEP: 1187700 | > loss_text_ce: 0.017845740541815758 (0.031640359810714114) | > loss_mel_ce: 1.3569031953811646 (nan) | > loss: 0.005455353297293186 (nan) | > current_lr: 5e-06 | > step_time: 0.1082 (0.136425468386436) | > loader_time: 0.0044 (0.0046672796297039885)  --> TIME: 2025-02-26 09:45:22 -- STEP: 5734/36938 -- GLOBAL_STEP: 1187750 | > loss_text_ce: 0.029252974316477776 (0.031643148775095276) | > loss_mel_ce: 2.9703009128570557 (nan) | > loss: 0.011902992613613605 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.13639744283598318) | > loader_time: 0.0047 (0.004666682066383544)  --> TIME: 2025-02-26 09:45:31 -- STEP: 5784/36938 -- GLOBAL_STEP: 1187800 | > loss_text_ce: 0.020391253754496574 (0.031647428772946885) | > loss_mel_ce: 2.704317569732666 (nan) | > loss: 0.010812337510287762 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.13643067329091507) | > loader_time: 0.0048 (0.004667442575365027)  --> TIME: 2025-02-26 09:45:41 -- STEP: 5834/36938 -- GLOBAL_STEP: 1187850 | > loss_text_ce: 0.0324423722922802 (0.031651888933782506) | > loss_mel_ce: 2.591047525405884 (nan) | > loss: 0.010410674847662449 (nan) | > current_lr: 5e-06 | > step_time: 0.1556 (0.1364670104264479) | > loader_time: 0.0047 (0.0046680599686050915)  --> TIME: 2025-02-26 09:45:50 -- STEP: 5884/36938 -- GLOBAL_STEP: 1187900 | > loss_text_ce: 0.024050716310739517 (0.03164830752556619) | > loss_mel_ce: 2.3202781677246094 (nan) | > loss: 0.009302892722189426 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.1364247220296752) | > loader_time: 0.0046 (0.004668334363513386)  --> TIME: 2025-02-26 09:45:59 -- STEP: 5934/36938 -- GLOBAL_STEP: 1187950 | > loss_text_ce: 0.025551456958055496 (0.03165481090426381) | > loss_mel_ce: 0.18450927734375 (nan) | > loss: 0.0008335743914358318 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13640934544297492) | > loader_time: 0.0045 (0.0046682612519092365)  --> TIME: 2025-02-26 09:46:08 -- STEP: 5984/36938 -- GLOBAL_STEP: 1188000 | > loss_text_ce: 0.03928692638874054 (0.03164437682060333) | > loss_mel_ce: 2.9194912910461426 (nan) | > loss: 0.011741183698177338 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.13642667692134747) | > loader_time: 0.0048 (0.0046679872800322215)  --> TIME: 2025-02-26 09:46:18 -- STEP: 6034/36938 -- GLOBAL_STEP: 1188050 | > loss_text_ce: 0.03052152507007122 (0.031646766110867044) | > loss_mel_ce: 2.0626161098480225 (nan) | > loss: 0.008306102827191353 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13646536281182367) | > loader_time: 0.0046 (0.004667836149125304)  --> TIME: 2025-02-26 09:46:27 -- STEP: 6084/36938 -- GLOBAL_STEP: 1188100 | > loss_text_ce: 0.039084576070308685 (0.031651856626930826) | > loss_mel_ce: 2.220202922821045 (nan) | > loss: 0.008965427055954933 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13647092489879603) | > loader_time: 0.0046 (0.0046677739897658475)  --> TIME: 2025-02-26 09:46:37 -- STEP: 6134/36938 -- GLOBAL_STEP: 1188150 | > loss_text_ce: 0.02862425334751606 (0.03165004404433588) | > loss_mel_ce: 2.4165520668029785 (nan) | > loss: 0.009703081101179123 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13647673353139855) | > loader_time: 0.0048 (0.004668855923645509)  --> TIME: 2025-02-26 09:46:46 -- STEP: 6184/36938 -- GLOBAL_STEP: 1188200 | > loss_text_ce: 0.028026733547449112 (0.031653761179975634) | > loss_mel_ce: 3.2591841220855713 (nan) | > loss: 0.013044488616287708 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.13648762165497708) | > loader_time: 0.0047 (0.004670124235930459)  --> TIME: 2025-02-26 09:46:55 -- STEP: 6234/36938 -- GLOBAL_STEP: 1188250 | > loss_text_ce: 0.03466540575027466 (0.03166492785099167) | > loss_mel_ce: 2.8817594051361084 (nan) | > loss: 0.011573114432394505 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.1365077554089619) | > loader_time: 0.0047 (0.0046716253842380515)  --> TIME: 2025-02-26 09:47:05 -- STEP: 6284/36938 -- GLOBAL_STEP: 1188300 | > loss_text_ce: 0.03282896801829338 (0.031662105672466946) | > loss_mel_ce: 3.147035837173462 (nan) | > loss: 0.01261851191520691 (nan) | > current_lr: 5e-06 | > step_time: 0.171 (0.1365202815194254) | > loader_time: 0.0048 (0.00467210530020643)  --> TIME: 2025-02-26 09:47:14 -- STEP: 6334/36938 -- GLOBAL_STEP: 1188350 | > loss_text_ce: 0.033697690814733505 (0.03166076300161701) | > loss_mel_ce: 2.5029985904693604 (nan) | > loss: 0.010066254995763302 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13649689277941623) | > loader_time: 0.0046 (0.00467202533183506)  --> TIME: 2025-02-26 09:47:23 -- STEP: 6384/36938 -- GLOBAL_STEP: 1188400 | > loss_text_ce: 0.02991299331188202 (0.03166852149100542) | > loss_mel_ce: 2.8204851150512695 (nan) | > loss: 0.011311104521155357 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13648107834627152) | > loader_time: 0.0046 (0.0046732370790682366)  --> TIME: 2025-02-26 09:47:32 -- STEP: 6434/36938 -- GLOBAL_STEP: 1188450 | > loss_text_ce: 0.03288976103067398 (0.03167365593902301) | > loss_mel_ce: 2.1444783210754395 (nan) | > loss: 0.00864035077393055 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13645756896845765) | > loader_time: 0.0049 (0.004673965198859541)  --> TIME: 2025-02-26 09:47:42 -- STEP: 6484/36938 -- GLOBAL_STEP: 1188500 | > loss_text_ce: 0.03417612239718437 (0.031670295217699033) | > loss_mel_ce: 3.377718687057495 (nan) | > loss: 0.013539265841245651 (nan) | > current_lr: 5e-06 | > step_time: 0.1835 (0.1364639494096989) | > loader_time: 0.0059 (0.004674059273945116)  --> TIME: 2025-02-26 09:47:51 -- STEP: 6534/36938 -- GLOBAL_STEP: 1188550 | > loss_text_ce: 0.037782635539770126 (0.03167544705020726) | > loss_mel_ce: 3.112149715423584 (nan) | > loss: 0.012499731965363026 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.13648272082460425) | > loader_time: 0.0048 (0.004674780759236333)  --> TIME: 2025-02-26 09:48:00 -- STEP: 6584/36938 -- GLOBAL_STEP: 1188600 | > loss_text_ce: 0.03713925927877426 (0.031677975178722256) | > loss_mel_ce: 3.0019371509552 (nan) | > loss: 0.012059827335178852 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13648822092488463) | > loader_time: 0.0047 (0.004674850989083327)  --> TIME: 2025-02-26 09:48:10 -- STEP: 6634/36938 -- GLOBAL_STEP: 1188650 | > loss_text_ce: 0.0295246709138155 (0.03167633886819936) | > loss_mel_ce: 2.6342427730560303 (nan) | > loss: 0.010570505633950233 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13648054601199486) | > loader_time: 0.0046 (0.004674685515299381)  --> TIME: 2025-02-26 09:48:19 -- STEP: 6684/36938 -- GLOBAL_STEP: 1188700 | > loss_text_ce: 0.03654506057500839 (0.031675752777080055) | > loss_mel_ce: 2.133592367172241 (nan) | > loss: 0.008611656725406647 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13652008447298936) | > loader_time: 0.0045 (0.00467497738159609)  --> TIME: 2025-02-26 09:48:29 -- STEP: 6734/36938 -- GLOBAL_STEP: 1188750 | > loss_text_ce: 0.03613456338644028 (0.03167444686226946) | > loss_mel_ce: 3.3347525596618652 (nan) | > loss: 0.013376536779105663 (nan) | > current_lr: 5e-06 | > step_time: 0.1626 (0.13654597215564632) | > loader_time: 0.0048 (0.004675102085184906)  --> TIME: 2025-02-26 09:48:38 -- STEP: 6784/36938 -- GLOBAL_STEP: 1188800 | > loss_text_ce: 0.021203646436333656 (0.03167326406730588) | > loss_mel_ce: 2.614291191101074 (nan) | > loss: 0.01045831385999918 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.1365340392825735) | > loader_time: 0.0045 (0.004675267861699163)  --> TIME: 2025-02-26 09:48:47 -- STEP: 6834/36938 -- GLOBAL_STEP: 1188850 | > loss_text_ce: 0.03193976730108261 (0.031673068517286414) | > loss_mel_ce: 2.9948112964630127 (nan) | > loss: 0.012010917067527771 (nan) | > current_lr: 5e-06 | > step_time: 0.184 (0.13653373983403802) | > loader_time: 0.0048 (0.004674890496977114)  --> TIME: 2025-02-26 09:48:57 -- STEP: 6884/36938 -- GLOBAL_STEP: 1188900 | > loss_text_ce: 0.03961968794465065 (0.03167665615014374) | > loss_mel_ce: 2.981808662414551 (nan) | > loss: 0.011989795602858067 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13653036264539223) | > loader_time: 0.0047 (0.004674907758580891)  --> TIME: 2025-02-26 09:49:06 -- STEP: 6934/36938 -- GLOBAL_STEP: 1188950 | > loss_text_ce: 0.03734980523586273 (0.03167635177560029) | > loss_mel_ce: 2.552354574203491 (nan) | > loss: 0.010276604443788528 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13653525265692812) | > loader_time: 0.0045 (0.004674526157659731)  --> TIME: 2025-02-26 09:49:15 -- STEP: 6984/36938 -- GLOBAL_STEP: 1189000 | > loss_text_ce: 0.03616917133331299 (0.03167384302506022) | > loss_mel_ce: 2.266648292541504 (nan) | > loss: 0.009138165041804314 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13651809256784128) | > loader_time: 0.0045 (0.00467467338768478)  --> TIME: 2025-02-26 09:49:24 -- STEP: 7034/36938 -- GLOBAL_STEP: 1189050 | > loss_text_ce: 0.035087358206510544 (0.031672800656218456) | > loss_mel_ce: 3.16748046875 (nan) | > loss: 0.012708603404462337 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.1365211981414628) | > loader_time: 0.0044 (0.004674694637768243)  --> TIME: 2025-02-26 09:49:34 -- STEP: 7084/36938 -- GLOBAL_STEP: 1189100 | > loss_text_ce: 0.03940035402774811 (0.031665836480082754) | > loss_mel_ce: 2.851729393005371 (nan) | > loss: 0.011472737416625023 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.1365040657817812) | > loader_time: 0.0045 (0.0046749681082908145)  --> TIME: 2025-02-26 09:49:43 -- STEP: 7134/36938 -- GLOBAL_STEP: 1189150 | > loss_text_ce: 0.034696824848651886 (0.031659140616847464) | > loss_mel_ce: 2.8481686115264893 (nan) | > loss: 0.011439942754805088 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.13649335163529253) | > loader_time: 0.0046 (0.004675139457129423)  --> TIME: 2025-02-26 09:49:52 -- STEP: 7184/36938 -- GLOBAL_STEP: 1189200 | > loss_text_ce: 0.03597576916217804 (0.031657054483110576) | > loss_mel_ce: 0.8919084668159485 (nan) | > loss: 0.0036820804234594107 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1365183345591835) | > loader_time: 0.0045 (0.004675327702726189)  --> TIME: 2025-02-26 09:50:02 -- STEP: 7234/36938 -- GLOBAL_STEP: 1189250 | > loss_text_ce: 0.03969564288854599 (0.03166426345345503) | > loss_mel_ce: 0.00158749520778656 (nan) | > loss: 0.00016382198373321444 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13653023384010912) | > loader_time: 0.0045 (0.004675783635108624)  --> TIME: 2025-02-26 09:50:11 -- STEP: 7284/36938 -- GLOBAL_STEP: 1189300 | > loss_text_ce: 0.03255515173077583 (0.03166675624498489) | > loss_mel_ce: 3.4735968112945557 (nan) | > loss: 0.01391330175101757 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13652633023353677) | > loader_time: 0.0046 (0.004675724590171641)  --> TIME: 2025-02-26 09:50:20 -- STEP: 7334/36938 -- GLOBAL_STEP: 1189350 | > loss_text_ce: 0.03407711908221245 (0.03165995420312558) | > loss_mel_ce: 2.153913974761963 (nan) | > loss: 0.00868250522762537 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13651289028337293) | > loader_time: 0.0048 (0.00467643995911802)  --> TIME: 2025-02-26 09:50:29 -- STEP: 7384/36938 -- GLOBAL_STEP: 1189400 | > loss_text_ce: 0.04319524019956589 (0.031664681617364264) | > loss_mel_ce: 2.9820852279663086 (nan) | > loss: 0.012005082331597805 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.13650311134101767) | > loader_time: 0.0047 (0.0046762266758332985)  --> TIME: 2025-02-26 09:50:39 -- STEP: 7434/36938 -- GLOBAL_STEP: 1189450 | > loss_text_ce: 0.05803442373871803 (0.03167189156598006) | > loss_mel_ce: 2.3935189247131348 (nan) | > loss: 0.009728386998176575 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13650226772617258) | > loader_time: 0.0046 (0.004676357949969914)  --> TIME: 2025-02-26 09:50:48 -- STEP: 7484/36938 -- GLOBAL_STEP: 1189500 | > loss_text_ce: 0.03754078596830368 (0.031673034296168426) | > loss_mel_ce: 3.374796152114868 (nan) | > loss: 0.013541019521653652 (nan) | > current_lr: 5e-06 | > step_time: 0.1879 (0.13648604260352387) | > loader_time: 0.0049 (0.004676229554914912)  --> TIME: 2025-02-26 09:50:57 -- STEP: 7534/36938 -- GLOBAL_STEP: 1189550 | > loss_text_ce: 0.03332241252064705 (0.0316647190270826) | > loss_mel_ce: 0.0142822265625 (nan) | > loss: 0.00018890730279963464 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13647660007836307) | > loader_time: 0.0046 (0.004676211566990878)  --> TIME: 2025-02-26 09:51:06 -- STEP: 7584/36938 -- GLOBAL_STEP: 1189600 | > loss_text_ce: 0.02374783530831337 (0.03166851483826728) | > loss_mel_ce: 2.1484375 (nan) | > loss: 0.008619784377515316 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13647073743086832) | > loader_time: 0.0045 (0.0046764640491220085)  --> TIME: 2025-02-26 09:51:16 -- STEP: 7634/36938 -- GLOBAL_STEP: 1189650 | > loss_text_ce: 0.03724317625164986 (0.03167851654901015) | > loss_mel_ce: 3.056508779525757 (nan) | > loss: 0.012276793830096722 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.13647034922787935) | > loader_time: 0.0049 (0.0046765940458574125)  --> TIME: 2025-02-26 09:51:25 -- STEP: 7684/36938 -- GLOBAL_STEP: 1189700 | > loss_text_ce: 0.03234421834349632 (0.03168587545771682) | > loss_mel_ce: 2.953584909439087 (nan) | > loss: 0.011848925612866879 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13644912682244834) | > loader_time: 0.0046 (0.00467716747742653)  --> TIME: 2025-02-26 09:51:34 -- STEP: 7734/36938 -- GLOBAL_STEP: 1189750 | > loss_text_ce: 0.032738495618104935 (0.031684456588924374) | > loss_mel_ce: 2.743467092514038 (nan) | > loss: 0.011016689240932465 (nan) | > current_lr: 5e-06 | > step_time: 0.1086 (0.1364782218758007) | > loader_time: 0.0045 (0.004677394609720114)  --> TIME: 2025-02-26 09:51:44 -- STEP: 7784/36938 -- GLOBAL_STEP: 1189800 | > loss_text_ce: 0.025470037013292313 (0.031682217410738775) | > loss_mel_ce: 2.6805121898651123 (nan) | > loss: 0.010738025419414043 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13647187745460876) | > loader_time: 0.0048 (0.00467841895364172)  --> TIME: 2025-02-26 09:51:53 -- STEP: 7834/36938 -- GLOBAL_STEP: 1189850 | > loss_text_ce: 0.03265373408794403 (0.03167534713070569) | > loss_mel_ce: 2.802388906478882 (nan) | > loss: 0.011250169947743416 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.13647902733152978) | > loader_time: 0.0047 (0.004678373285663174)  --> TIME: 2025-02-26 09:52:03 -- STEP: 7884/36938 -- GLOBAL_STEP: 1189900 | > loss_text_ce: 0.03465982899069786 (0.031675117322230016) | > loss_mel_ce: 2.7870073318481445 (nan) | > loss: 0.011197092942893505 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.1365092887242152) | > loader_time: 0.0047 (0.004678822815690417)  --> TIME: 2025-02-26 09:52:12 -- STEP: 7934/36938 -- GLOBAL_STEP: 1189950 | > loss_text_ce: 0.027948400005698204 (0.031672356874201425) | > loss_mel_ce: 1.1964857578277588 (nan) | > loss: 0.0048588658683001995 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13647334167570865) | > loader_time: 0.0045 (0.004678561851543782)  --> TIME: 2025-02-26 09:52:21 -- STEP: 7984/36938 -- GLOBAL_STEP: 1190000 | > loss_text_ce: 0.033408377319574356 (0.03167712331197982) | > loss_mel_ce: 2.672970771789551 (nan) | > loss: 0.010739600285887718 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.13648067811925813) | > loader_time: 0.0046 (0.004678754684681412) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1190000.pth  --> TIME: 2025-02-26 09:52:36 -- STEP: 8034/36938 -- GLOBAL_STEP: 1190050 | > loss_text_ce: 0.03287679702043533 (0.03167587974036188) | > loss_mel_ce: 3.1781234741210938 (nan) | > loss: 0.012742064893245697 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.13649848153230892) | > loader_time: 0.0047 (0.004679538226109825)  --> TIME: 2025-02-26 09:52:45 -- STEP: 8084/36938 -- GLOBAL_STEP: 1190100 | > loss_text_ce: 0.03572448343038559 (0.0316806460575434) | > loss_mel_ce: 1.6713378429412842 (nan) | > loss: 0.0067740571685135365 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13646756632383486) | > loader_time: 0.0046 (0.0046793981273241594)  --> TIME: 2025-02-26 09:52:55 -- STEP: 8134/36938 -- GLOBAL_STEP: 1190150 | > loss_text_ce: 0.03205893933773041 (0.03168523172088347) | > loss_mel_ce: 2.3221523761749268 (nan) | > loss: 0.009342108853161335 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1364602479824482) | > loader_time: 0.0045 (0.0046791460814803185)  --> TIME: 2025-02-26 09:53:04 -- STEP: 8184/36938 -- GLOBAL_STEP: 1190200 | > loss_text_ce: 0.02073487639427185 (0.031679559323776446) | > loss_mel_ce: 0.9821866154670715 (nan) | > loss: 0.003979847300797701 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13641915555573633) | > loader_time: 0.0046 (0.004678700268559914)  --> TIME: 2025-02-26 09:53:13 -- STEP: 8234/36938 -- GLOBAL_STEP: 1190250 | > loss_text_ce: 0.035639625042676926 (0.031679784966859366) | > loss_mel_ce: 2.6285181045532227 (nan) | > loss: 0.010572054423391819 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.1364156988774917) | > loader_time: 0.0047 (0.004678762303644927)  --> TIME: 2025-02-26 09:53:22 -- STEP: 8284/36938 -- GLOBAL_STEP: 1190300 | > loss_text_ce: 0.036019325256347656 (0.031684448654511095) | > loss_mel_ce: 3.4159393310546875 (nan) | > loss: 0.013698249123990536 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.1363976852782161) | > loader_time: 0.0047 (0.0046783280453205405)  --> TIME: 2025-02-26 09:53:31 -- STEP: 8334/36938 -- GLOBAL_STEP: 1190350 | > loss_text_ce: 0.02620299533009529 (0.03168202400082804) | > loss_mel_ce: 2.3554868698120117 (nan) | > loss: 0.009451150894165039 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13640849782546882) | > loader_time: 0.0046 (0.0046787545181505175)  --> TIME: 2025-02-26 09:53:41 -- STEP: 8384/36938 -- GLOBAL_STEP: 1190400 | > loss_text_ce: 0.032584428787231445 (0.031687491565950736) | > loss_mel_ce: 3.1229865550994873 (nan) | > loss: 0.012522107921540737 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13642019579428755) | > loader_time: 0.0046 (0.00467894718278456)  --> TIME: 2025-02-26 09:53:50 -- STEP: 8434/36938 -- GLOBAL_STEP: 1190450 | > loss_text_ce: 0.024778136983513832 (0.03168867457269796) | > loss_mel_ce: 2.7129557132720947 (nan) | > loss: 0.010864024050533772 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13639761919881435) | > loader_time: 0.0046 (0.0046788136315114035)  --> TIME: 2025-02-26 09:53:59 -- STEP: 8484/36938 -- GLOBAL_STEP: 1190500 | > loss_text_ce: 0.03549639508128166 (0.03168473570141718) | > loss_mel_ce: 3.0461134910583496 (nan) | > loss: 0.01222861185669899 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.1363798746912057) | > loader_time: 0.0046 (0.004679072611402762)  --> TIME: 2025-02-26 09:54:09 -- STEP: 8534/36938 -- GLOBAL_STEP: 1190550 | > loss_text_ce: 0.02549627423286438 (0.031686686926403675) | > loss_mel_ce: 2.7961907386779785 (nan) | > loss: 0.011197171173989773 (nan) | > current_lr: 5e-06 | > step_time: 0.1266 (0.13640666703639234) | > loader_time: 0.0045 (0.004679000905673686)  --> TIME: 2025-02-26 09:54:18 -- STEP: 8584/36938 -- GLOBAL_STEP: 1190600 | > loss_text_ce: 0.029605884104967117 (0.03168272673810297) | > loss_mel_ce: 2.8208279609680176 (nan) | > loss: 0.011311246082186699 (nan) | > current_lr: 5e-06 | > step_time: 0.1774 (0.13639284982819047) | > loader_time: 0.0048 (0.004679591796827012)  --> TIME: 2025-02-26 09:54:27 -- STEP: 8634/36938 -- GLOBAL_STEP: 1190650 | > loss_text_ce: 0.03279542922973633 (0.03168981412275912) | > loss_mel_ce: 3.5662357807159424 (nan) | > loss: 0.014281870797276497 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13641951347573766) | > loader_time: 0.0045 (0.004679582724837432)  --> TIME: 2025-02-26 09:54:37 -- STEP: 8684/36938 -- GLOBAL_STEP: 1190700 | > loss_text_ce: 0.03707413747906685 (0.031692876633773966) | > loss_mel_ce: 2.5436980724334717 (nan) | > loss: 0.010241160169243813 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13640947989306124) | > loader_time: 0.0047 (0.0046797324742440674)  --> TIME: 2025-02-26 09:54:46 -- STEP: 8734/36938 -- GLOBAL_STEP: 1190750 | > loss_text_ce: 0.03275267034769058 (0.031689113046195906) | > loss_mel_ce: 2.9928829669952393 (nan) | > loss: 0.012006491422653198 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13640639226169665) | > loader_time: 0.0046 (0.00467923521749865)  --> TIME: 2025-02-26 09:54:55 -- STEP: 8784/36938 -- GLOBAL_STEP: 1190800 | > loss_text_ce: 0.03353302553296089 (0.03169467298024311) | > loss_mel_ce: 2.5382089614868164 (nan) | > loss: 0.010205326601862907 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13641836525421128) | > loader_time: 0.0046 (0.004678956417872389)  --> TIME: 2025-02-26 09:55:04 -- STEP: 8834/36938 -- GLOBAL_STEP: 1190850 | > loss_text_ce: 0.03863734006881714 (0.03169315700438852) | > loss_mel_ce: 2.206866979598999 (nan) | > loss: 0.008910732343792915 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13638076856175751) | > loader_time: 0.0046 (0.0046783643041985986)  --> TIME: 2025-02-26 09:55:14 -- STEP: 8884/36938 -- GLOBAL_STEP: 1190900 | > loss_text_ce: 0.021951116621494293 (0.03169369144972791) | > loss_mel_ce: 0.13611602783203125 (nan) | > loss: 0.0006272505852393806 (nan) | > current_lr: 5e-06 | > step_time: 0.1528 (0.13639679608823824) | > loader_time: 0.0046 (0.0046781483452258904)  --> TIME: 2025-02-26 09:55:23 -- STEP: 8934/36938 -- GLOBAL_STEP: 1190950 | > loss_text_ce: 0.021417375653982162 (0.03168858713721626) | > loss_mel_ce: 1.7516378164291382 (nan) | > loss: 0.007035933900624514 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13637582178160707) | > loader_time: 0.0045 (0.004677859520469171)  --> TIME: 2025-02-26 09:55:32 -- STEP: 8984/36938 -- GLOBAL_STEP: 1191000 | > loss_text_ce: 0.02870159037411213 (0.03168278739019273) | > loss_mel_ce: 0.43202462792396545 (nan) | > loss: 0.0018282787641510367 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13637162319061935) | > loader_time: 0.0045 (0.004677429437000413)  --> TIME: 2025-02-26 09:55:41 -- STEP: 9034/36938 -- GLOBAL_STEP: 1191050 | > loss_text_ce: 0.030073940753936768 (0.03168125956426278) | > loss_mel_ce: 0.123046875 (nan) | > loss: 0.0006076223216950893 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13640438913903166) | > loader_time: 0.0058 (0.004677449624992877)  --> TIME: 2025-02-26 09:55:51 -- STEP: 9084/36938 -- GLOBAL_STEP: 1191100 | > loss_text_ce: 0.04340660572052002 (0.031675317951833586) | > loss_mel_ce: 2.927255392074585 (nan) | > loss: 0.011788342148065567 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13642526520658124) | > loader_time: 0.0046 (0.004677358622700042)  --> TIME: 2025-02-26 09:56:00 -- STEP: 9134/36938 -- GLOBAL_STEP: 1191150 | > loss_text_ce: 0.02983825094997883 (0.0316746585712021) | > loss_mel_ce: 2.871593475341797 (nan) | > loss: 0.011513618752360344 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13642280321557676) | > loader_time: 0.0047 (0.004676936543011316)  --> TIME: 2025-02-26 09:56:10 -- STEP: 9184/36938 -- GLOBAL_STEP: 1191200 | > loss_text_ce: 0.030719926580786705 (0.03166920127195404) | > loss_mel_ce: 2.978692054748535 (nan) | > loss: 0.011942111887037754 (nan) | > current_lr: 5e-06 | > step_time: 0.1793 (0.13643049472092783) | > loader_time: 0.0049 (0.00467678689063634)  --> TIME: 2025-02-26 09:56:19 -- STEP: 9234/36938 -- GLOBAL_STEP: 1191250 | > loss_text_ce: 0.027708163484930992 (0.0316612064690753) | > loss_mel_ce: 2.8115992546081543 (nan) | > loss: 0.011267093010246754 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.13641815023399553) | > loader_time: 0.0048 (0.004676453448075197)  --> TIME: 2025-02-26 09:56:28 -- STEP: 9284/36938 -- GLOBAL_STEP: 1191300 | > loss_text_ce: 0.034117553383111954 (0.031654764276169575) | > loss_mel_ce: 0.9858016967773438 (nan) | > loss: 0.004047298803925514 (nan) | > current_lr: 5e-06 | > step_time: 0.1566 (0.13642323363808323) | > loader_time: 0.0047 (0.004676393731202305)  --> TIME: 2025-02-26 09:56:37 -- STEP: 9334/36938 -- GLOBAL_STEP: 1191350 | > loss_text_ce: 0.031240100041031837 (0.031654306490453955) | > loss_mel_ce: 3.10290265083313 (nan) | > loss: 0.012437074445188046 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.1364086156282741) | > loader_time: 0.0049 (0.004676596776612037)  --> TIME: 2025-02-26 09:56:46 -- STEP: 9384/36938 -- GLOBAL_STEP: 1191400 | > loss_text_ce: 0.0298270545899868 (0.03165708253543602) | > loss_mel_ce: 1.131298542022705 (nan) | > loss: 0.004607641138136387 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13638548831195815) | > loader_time: 0.0045 (0.004676453673931036)  --> TIME: 2025-02-26 09:56:55 -- STEP: 9434/36938 -- GLOBAL_STEP: 1191450 | > loss_text_ce: 0.02011284790933132 (0.031656466539967294) | > loss_mel_ce: 1.5741435289382935 (nan) | > loss: 0.00632641464471817 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13636550879058967) | > loader_time: 0.0045 (0.004676239935806404)  --> TIME: 2025-02-26 09:57:05 -- STEP: 9484/36938 -- GLOBAL_STEP: 1191500 | > loss_text_ce: 0.034015554934740067 (0.03166017626152567) | > loss_mel_ce: 1.6123270988464355 (nan) | > loss: 0.006533105857670307 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13636047492697018) | > loader_time: 0.0046 (0.004676194972300125)  --> TIME: 2025-02-26 09:57:14 -- STEP: 9534/36938 -- GLOBAL_STEP: 1191550 | > loss_text_ce: 0.03523428738117218 (0.031658436835573275) | > loss_mel_ce: 2.0139129161834717 (nan) | > loss: 0.008131536655128002 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13636598970646524) | > loader_time: 0.0045 (0.004676393950438141)  --> TIME: 2025-02-26 09:57:23 -- STEP: 9584/36938 -- GLOBAL_STEP: 1191600 | > loss_text_ce: 0.030077874660491943 (0.031656997625246416) | > loss_mel_ce: 2.9449455738067627 (nan) | > loss: 0.011805649846792221 (nan) | > current_lr: 5e-06 | > step_time: 0.1875 (0.13635949760625737) | > loader_time: 0.005 (0.004676173445776435)  --> TIME: 2025-02-26 09:57:32 -- STEP: 9634/36938 -- GLOBAL_STEP: 1191650 | > loss_text_ce: 0.02651122957468033 (0.03165251172894174) | > loss_mel_ce: 2.605689764022827 (nan) | > loss: 0.010445242747664452 (nan) | > current_lr: 5e-06 | > step_time: 0.1515 (0.13634647579640624) | > loader_time: 0.0053 (0.004676198103082911)  --> TIME: 2025-02-26 09:57:42 -- STEP: 9684/36938 -- GLOBAL_STEP: 1191700 | > loss_text_ce: 0.022821377962827682 (0.03164926946000834) | > loss_mel_ce: 2.6933937072753906 (nan) | > loss: 0.010778632014989853 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13634667109478085) | > loader_time: 0.0046 (0.004676800062714511)  --> TIME: 2025-02-26 09:57:51 -- STEP: 9734/36938 -- GLOBAL_STEP: 1191750 | > loss_text_ce: 0.029442034661769867 (0.03165051905782351) | > loss_mel_ce: 2.9206697940826416 (nan) | > loss: 0.011706793680787086 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.1363497279080768) | > loader_time: 0.0046 (0.004676839324123499)  --> TIME: 2025-02-26 09:58:00 -- STEP: 9784/36938 -- GLOBAL_STEP: 1191800 | > loss_text_ce: 0.04674506559967995 (0.03165629442449185) | > loss_mel_ce: 2.599910259246826 (nan) | > loss: 0.01050260104238987 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13634752137974895) | > loader_time: 0.0045 (0.004677224017905103)  --> TIME: 2025-02-26 09:58:10 -- STEP: 9834/36938 -- GLOBAL_STEP: 1191850 | > loss_text_ce: 0.02827940694987774 (0.031656969156633616) | > loss_mel_ce: 2.37389874458313 (nan) | > loss: 0.009532453492283821 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13635458242173218) | > loader_time: 0.0044 (0.0046770210694752805)  --> TIME: 2025-02-26 09:58:19 -- STEP: 9884/36938 -- GLOBAL_STEP: 1191900 | > loss_text_ce: 0.008883140981197357 (0.03165354074375825) | > loss_mel_ce: 2.3256375789642334 (nan) | > loss: 0.009263971820473671 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13635467373374469) | > loader_time: 0.0045 (0.004676978701958531)  --> TIME: 2025-02-26 09:58:29 -- STEP: 9934/36938 -- GLOBAL_STEP: 1191950 | > loss_text_ce: 0.03454456850886345 (0.03164988146826332) | > loss_mel_ce: 2.533745050430298 (nan) | > loss: 0.010191625915467739 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.13636942338914737) | > loader_time: 0.0048 (0.0046768040634963285)  --> TIME: 2025-02-26 09:58:38 -- STEP: 9984/36938 -- GLOBAL_STEP: 1192000 | > loss_text_ce: 0.026936935260891914 (0.03164402346831189) | > loss_mel_ce: 1.1337395906448364 (nan) | > loss: 0.0046058595180511475 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13636864843563457) | > loader_time: 0.0045 (0.004676724760196155)  --> TIME: 2025-02-26 09:58:47 -- STEP: 10034/36938 -- GLOBAL_STEP: 1192050 | > loss_text_ce: 0.038438521325588226 (0.031648057654205905) | > loss_mel_ce: 2.6641910076141357 (nan) | > loss: 0.010724721476435661 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13636103179175915) | > loader_time: 0.0048 (0.00467626098524843)  --> TIME: 2025-02-26 09:58:56 -- STEP: 10084/36938 -- GLOBAL_STEP: 1192100 | > loss_text_ce: 0.026891836896538734 (0.03164698006673898) | > loss_mel_ce: 2.6452383995056152 (nan) | > loss: 0.010603692382574081 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.13636555320826055) | > loader_time: 0.0048 (0.004676146859074819)  --> TIME: 2025-02-26 09:59:06 -- STEP: 10134/36938 -- GLOBAL_STEP: 1192150 | > loss_text_ce: 0.034898288547992706 (0.031645840821662316) | > loss_mel_ce: 3.0300936698913574 (nan) | > loss: 0.012162666767835617 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.136405318946041) | > loader_time: 0.0045 (0.004676552032469204)  --> TIME: 2025-02-26 09:59:16 -- STEP: 10184/36938 -- GLOBAL_STEP: 1192200 | > loss_text_ce: 0.024327022954821587 (0.0316511740310312) | > loss_mel_ce: 2.6530349254608154 (nan) | > loss: 0.010624452494084835 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1364257911107533) | > loader_time: 0.0045 (0.004677379894518806)  --> TIME: 2025-02-26 09:59:25 -- STEP: 10234/36938 -- GLOBAL_STEP: 1192250 | > loss_text_ce: 0.02960253320634365 (0.03164116702413267) | > loss_mel_ce: 2.7633755207061768 (nan) | > loss: 0.011083247140049934 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13641024553754869) | > loader_time: 0.0045 (0.004677437957851651)  --> TIME: 2025-02-26 09:59:34 -- STEP: 10284/36938 -- GLOBAL_STEP: 1192300 | > loss_text_ce: 0.037877462804317474 (0.03164392900886829) | > loss_mel_ce: 2.690807580947876 (nan) | > loss: 0.010828115977346897 (nan) | > current_lr: 5e-06 | > step_time: 0.1704 (0.13643087004465387) | > loader_time: 0.0056 (0.004677336997718774)  --> TIME: 2025-02-26 09:59:44 -- STEP: 10334/36938 -- GLOBAL_STEP: 1192350 | > loss_text_ce: 0.027893342077732086 (0.03164035637845135) | > loss_mel_ce: 3.1130359172821045 (nan) | > loss: 0.012464005500078201 (nan) | > current_lr: 5e-06 | > step_time: 0.1812 (0.13642942716241607) | > loader_time: 0.0048 (0.00467719793181275)  --> TIME: 2025-02-26 09:59:53 -- STEP: 10384/36938 -- GLOBAL_STEP: 1192400 | > loss_text_ce: 0.02843477390706539 (0.03163855505220375) | > loss_mel_ce: 2.0623092651367188 (nan) | > loss: 0.008296603336930275 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13642570417724578) | > loader_time: 0.0046 (0.004676797150279787)  --> TIME: 2025-02-26 10:00:02 -- STEP: 10434/36938 -- GLOBAL_STEP: 1192450 | > loss_text_ce: 0.026295997202396393 (0.031637402729966554) | > loss_mel_ce: 2.5588576793670654 (nan) | > loss: 0.010258546099066734 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.1364376339986456) | > loader_time: 0.0046 (0.004677032085358017)  --> TIME: 2025-02-26 10:00:12 -- STEP: 10484/36938 -- GLOBAL_STEP: 1192500 | > loss_text_ce: 0.02887961082160473 (0.031634547188291756) | > loss_mel_ce: 2.8700921535491943 (nan) | > loss: 0.011503856629133224 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.1364454160160536) | > loader_time: 0.0054 (0.004677333753557441)  --> TIME: 2025-02-26 10:00:21 -- STEP: 10534/36938 -- GLOBAL_STEP: 1192550 | > loss_text_ce: 0.026087019592523575 (0.031636514039368895) | > loss_mel_ce: 2.2172555923461914 (nan) | > loss: 0.008902153931558132 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.1364228592576221) | > loader_time: 0.0046 (0.004677013855675768)  --> TIME: 2025-02-26 10:00:30 -- STEP: 10584/36938 -- GLOBAL_STEP: 1192600 | > loss_text_ce: 0.026362581178545952 (0.03163833222279814) | > loss_mel_ce: 2.276524066925049 (nan) | > loss: 0.009138439781963825 (nan) | > current_lr: 5e-06 | > step_time: 0.1465 (0.13642742770329144) | > loader_time: 0.0046 (0.004677062492341778)  --> TIME: 2025-02-26 10:00:40 -- STEP: 10634/36938 -- GLOBAL_STEP: 1192650 | > loss_text_ce: 0.030116846784949303 (0.03163731294844035) | > loss_mel_ce: 2.858558416366577 (nan) | > loss: 0.011462997645139694 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13643213822381192) | > loader_time: 0.0046 (0.004677145535367004)  --> TIME: 2025-02-26 10:00:49 -- STEP: 10684/36938 -- GLOBAL_STEP: 1192700 | > loss_text_ce: 0.03280125930905342 (0.031638112580347905) | > loss_mel_ce: 2.6583566665649414 (nan) | > loss: 0.010679198428988457 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1364249537518664) | > loader_time: 0.0044 (0.004676727220298921)  --> TIME: 2025-02-26 10:00:58 -- STEP: 10734/36938 -- GLOBAL_STEP: 1192750 | > loss_text_ce: 0.028987877070903778 (0.03163464436514395) | > loss_mel_ce: 3.119835138320923 (nan) | > loss: 0.012495330534875393 (nan) | > current_lr: 5e-06 | > step_time: 0.1453 (0.13640514021149094) | > loader_time: 0.0046 (0.004676427902490359)  --> TIME: 2025-02-26 10:01:07 -- STEP: 10784/36938 -- GLOBAL_STEP: 1192800 | > loss_text_ce: 0.03758798539638519 (0.03164417911343382) | > loss_mel_ce: 1.8392912149429321 (nan) | > loss: 0.007447933778166771 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13639015969696042) | > loader_time: 0.0045 (0.0046762528467249155)  --> TIME: 2025-02-26 10:01:16 -- STEP: 10834/36938 -- GLOBAL_STEP: 1192850 | > loss_text_ce: 0.024428751319646835 (0.03163803728965805) | > loss_mel_ce: 1.086073637008667 (nan) | > loss: 0.004406755790114403 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13638295600483472) | > loader_time: 0.0046 (0.004676652368373357)  --> TIME: 2025-02-26 10:01:26 -- STEP: 10884/36938 -- GLOBAL_STEP: 1192900 | > loss_text_ce: 0.029962169006466866 (0.031638203517957285) | > loss_mel_ce: 3.128369092941284 (nan) | > loss: 0.01253306120634079 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13639626849972653) | > loader_time: 0.0053 (0.004677441158701248)  --> TIME: 2025-02-26 10:01:35 -- STEP: 10934/36938 -- GLOBAL_STEP: 1192950 | > loss_text_ce: 0.019048022106289864 (0.03163563041050534) | > loss_mel_ce: 2.300483465194702 (nan) | > loss: 0.00920449011027813 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1364011993699531) | > loader_time: 0.0047 (0.004677857126333999)  --> TIME: 2025-02-26 10:01:44 -- STEP: 10984/36938 -- GLOBAL_STEP: 1193000 | > loss_text_ce: 0.033045195043087006 (0.0316326877292014) | > loss_mel_ce: 2.2935495376586914 (nan) | > loss: 0.009232520125806332 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13638909324971166) | > loader_time: 0.0045 (0.004677996658011241)  --> TIME: 2025-02-26 10:01:53 -- STEP: 11034/36938 -- GLOBAL_STEP: 1193050 | > loss_text_ce: 0.024748915806412697 (0.031630198841363255) | > loss_mel_ce: 2.749741315841675 (nan) | > loss: 0.0110098822042346 (nan) | > current_lr: 5e-06 | > step_time: 0.1899 (0.13637596261832005) | > loader_time: 0.0056 (0.0046782586936266165)  --> TIME: 2025-02-26 10:02:03 -- STEP: 11084/36938 -- GLOBAL_STEP: 1193100 | > loss_text_ce: 0.0348236970603466 (0.03162664934921881) | > loss_mel_ce: 0.08037414401769638 (nan) | > loss: 0.00045713430154137313 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1363815894113135) | > loader_time: 0.0046 (0.004678539251515426)  --> TIME: 2025-02-26 10:02:12 -- STEP: 11134/36938 -- GLOBAL_STEP: 1193150 | > loss_text_ce: 0.034274909645318985 (0.03162512331904902) | > loss_mel_ce: 3.046905040740967 (nan) | > loss: 0.01222690474241972 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.13638123895056967) | > loader_time: 0.0046 (0.004678666195493425)  --> TIME: 2025-02-26 10:02:21 -- STEP: 11184/36938 -- GLOBAL_STEP: 1193200 | > loss_text_ce: 0.03337952122092247 (0.03161962149164797) | > loss_mel_ce: 3.262253761291504 (nan) | > loss: 0.013077910989522934 (nan) | > current_lr: 5e-06 | > step_time: 0.1811 (0.13639910011758877) | > loader_time: 0.005 (0.004678829864882601)  --> TIME: 2025-02-26 10:02:31 -- STEP: 11234/36938 -- GLOBAL_STEP: 1193250 | > loss_text_ce: 0.029802003875374794 (0.03162365279289607) | > loss_mel_ce: 2.7903106212615967 (nan) | > loss: 0.011190923862159252 (nan) | > current_lr: 5e-06 | > step_time: 0.1744 (0.13638511078837842) | > loader_time: 0.0048 (0.004678857587549807)  --> TIME: 2025-02-26 10:02:40 -- STEP: 11284/36938 -- GLOBAL_STEP: 1193300 | > loss_text_ce: 0.031207988038659096 (0.031623143465458) | > loss_mel_ce: 2.97216796875 (nan) | > loss: 0.011918159201741219 (nan) | > current_lr: 5e-06 | > step_time: 0.163 (0.1363711696869522) | > loader_time: 0.0047 (0.004678577617819702)  --> TIME: 2025-02-26 10:02:49 -- STEP: 11334/36938 -- GLOBAL_STEP: 1193350 | > loss_text_ce: 0.03139820322394371 (0.031625127214484905) | > loss_mel_ce: 2.5650835037231445 (nan) | > loss: 0.010303500108420849 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.136395026772074) | > loader_time: 0.0046 (0.0046783589761430055)  --> TIME: 2025-02-26 10:02:59 -- STEP: 11384/36938 -- GLOBAL_STEP: 1193400 | > loss_text_ce: 0.026995059102773666 (0.031628928186606974) | > loss_mel_ce: 2.507347583770752 (nan) | > loss: 0.010056915692985058 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13640552528794267) | > loader_time: 0.0047 (0.004678176455487687)  --> TIME: 2025-02-26 10:03:08 -- STEP: 11434/36938 -- GLOBAL_STEP: 1193450 | > loss_text_ce: 0.02333032712340355 (0.03163079894111167) | > loss_mel_ce: 3.0097928047180176 (nan) | > loss: 0.012036203406751156 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13641885775295454) | > loader_time: 0.0046 (0.004678385729386877)  --> TIME: 2025-02-26 10:03:18 -- STEP: 11484/36938 -- GLOBAL_STEP: 1193500 | > loss_text_ce: 0.03550409525632858 (0.03163251734630011) | > loss_mel_ce: 2.378551483154297 (nan) | > loss: 0.009579585865139961 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.1364158720598126) | > loader_time: 0.0046 (0.0046781821559838226)  --> TIME: 2025-02-26 10:03:27 -- STEP: 11534/36938 -- GLOBAL_STEP: 1193550 | > loss_text_ce: 0.026375843212008476 (0.03162988201382873) | > loss_mel_ce: 2.3626251220703125 (nan) | > loss: 0.009480162523686886 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.1364071643061057) | > loader_time: 0.0046 (0.0046779606274928425)  --> TIME: 2025-02-26 10:03:36 -- STEP: 11584/36938 -- GLOBAL_STEP: 1193600 | > loss_text_ce: 0.02887190692126751 (0.03163319966234522) | > loss_mel_ce: 3.0389132499694824 (nan) | > loss: 0.01217375136911869 (nan) | > current_lr: 5e-06 | > step_time: 0.1483 (0.13643028393634857) | > loader_time: 0.0046 (0.004677972473327627)  --> TIME: 2025-02-26 10:03:46 -- STEP: 11634/36938 -- GLOBAL_STEP: 1193650 | > loss_text_ce: 0.025978131219744682 (0.03163314518516038) | > loss_mel_ce: 2.263026714324951 (nan) | > loss: 0.009083352982997894 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1364354755289479) | > loader_time: 0.0053 (0.004678177817162521)  --> TIME: 2025-02-26 10:03:55 -- STEP: 11684/36938 -- GLOBAL_STEP: 1193700 | > loss_text_ce: 0.024399656802415848 (0.031628063364507436) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13645595191725798) | > loader_time: 0.0046 (0.004678533241300703)  --> TIME: 2025-02-26 10:04:04 -- STEP: 11734/36938 -- GLOBAL_STEP: 1193750 | > loss_text_ce: 0.03086180053651333 (0.03162480225506884) | > loss_mel_ce: 2.3991851806640625 (nan) | > loss: 0.00964304432272911 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13644029955167514) | > loader_time: 0.0046 (0.004678518682316983)  --> TIME: 2025-02-26 10:04:14 -- STEP: 11784/36938 -- GLOBAL_STEP: 1193800 | > loss_text_ce: 0.023419180884957314 (0.03162809451634193) | > loss_mel_ce: 2.984604835510254 (nan) | > loss: 0.011936604045331478 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1364510180346438) | > loader_time: 0.0045 (0.004678206972256857)  --> TIME: 2025-02-26 10:04:23 -- STEP: 11834/36938 -- GLOBAL_STEP: 1193850 | > loss_text_ce: 0.030548393726348877 (0.031619883473812886) | > loss_mel_ce: 3.2353668212890625 (nan) | > loss: 0.012959981337189674 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13644794217432596) | > loader_time: 0.0048 (0.0046778278252513125)  --> TIME: 2025-02-26 10:04:32 -- STEP: 11884/36938 -- GLOBAL_STEP: 1193900 | > loss_text_ce: 0.04296397045254707 (0.031622712615202386) | > loss_mel_ce: 1.997632622718811 (nan) | > loss: 0.008097605779767036 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13644589983787098) | > loader_time: 0.0046 (0.004677869321841736)  --> TIME: 2025-02-26 10:04:42 -- STEP: 11934/36938 -- GLOBAL_STEP: 1193950 | > loss_text_ce: 0.036642201244831085 (0.031628120117072604) | > loss_mel_ce: 2.045994758605957 (nan) | > loss: 0.008264433592557907 (nan) | > current_lr: 5e-06 | > step_time: 0.1517 (0.13646353727532262) | > loader_time: 0.0046 (0.004677566687665594)  --> TIME: 2025-02-26 10:04:51 -- STEP: 11984/36938 -- GLOBAL_STEP: 1194000 | > loss_text_ce: 0.01845010370016098 (0.03163074346693521) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1445 (0.13647782199771225) | > loader_time: 0.0047 (0.004677310605870392)  --> TIME: 2025-02-26 10:05:01 -- STEP: 12034/36938 -- GLOBAL_STEP: 1194050 | > loss_text_ce: 0.032216738909482956 (0.03162906026280103) | > loss_mel_ce: 2.666759967803955 (nan) | > loss: 0.010710225440561771 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13650097846192483) | > loader_time: 0.0045 (0.004677125281108386)  --> TIME: 2025-02-26 10:05:10 -- STEP: 12084/36938 -- GLOBAL_STEP: 1194100 | > loss_text_ce: 0.03182714059948921 (0.03162743211757794) | > loss_mel_ce: 3.1106255054473877 (nan) | > loss: 0.012470051646232605 (nan) | > current_lr: 5e-06 | > step_time: 0.1855 (0.1365103221263221) | > loader_time: 0.0049 (0.004676844338950875)  --> TIME: 2025-02-26 10:05:20 -- STEP: 12134/36938 -- GLOBAL_STEP: 1194150 | > loss_text_ce: 0.031579047441482544 (0.03162636421567137) | > loss_mel_ce: 1.8049440383911133 (nan) | > loss: 0.007287790533155203 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13650034317688226) | > loader_time: 0.0045 (0.004676495231866091)  --> TIME: 2025-02-26 10:05:29 -- STEP: 12184/36938 -- GLOBAL_STEP: 1194200 | > loss_text_ce: 0.025168780237436295 (0.031624865507140304) | > loss_mel_ce: 1.7382309436798096 (nan) | > loss: 0.006997618358582258 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13649310178453114) | > loader_time: 0.0045 (0.004676109071006447)  --> TIME: 2025-02-26 10:05:38 -- STEP: 12234/36938 -- GLOBAL_STEP: 1194250 | > loss_text_ce: 0.03296078369021416 (0.031623811700599386) | > loss_mel_ce: 0.017333984375 (nan) | > loss: 0.00019958242774009705 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.1365206113375351) | > loader_time: 0.0046 (0.004675853110147062)  --> TIME: 2025-02-26 10:05:48 -- STEP: 12284/36938 -- GLOBAL_STEP: 1194300 | > loss_text_ce: 0.02831461653113365 (0.031616417390804966) | > loss_mel_ce: 1.4109783172607422 (nan) | > loss: 0.005711480043828487 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.1365196536007328) | > loader_time: 0.0046 (0.0046754622257559365)  --> TIME: 2025-02-26 10:05:57 -- STEP: 12334/36938 -- GLOBAL_STEP: 1194350 | > loss_text_ce: 0.03712807595729828 (0.031621003990942656) | > loss_mel_ce: 2.6981537342071533 (nan) | > loss: 0.010854292660951614 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13652999176475789) | > loader_time: 0.0046 (0.004675153184404864)  --> TIME: 2025-02-26 10:06:07 -- STEP: 12384/36938 -- GLOBAL_STEP: 1194400 | > loss_text_ce: 0.031716737896203995 (0.03162385774627456) | > loss_mel_ce: 3.2268474102020264 (nan) | > loss: 0.012930811382830143 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13654465012695213) | > loader_time: 0.0046 (0.00467555232596336)  --> TIME: 2025-02-26 10:06:16 -- STEP: 12434/36938 -- GLOBAL_STEP: 1194450 | > loss_text_ce: 0.03462168946862221 (0.03162134018115023) | > loss_mel_ce: 2.7690227031707764 (nan) | > loss: 0.011125573888421059 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.1365573861044061) | > loader_time: 0.0046 (0.0046755708412512835)  --> TIME: 2025-02-26 10:06:25 -- STEP: 12484/36938 -- GLOBAL_STEP: 1194500 | > loss_text_ce: 0.03930727764964104 (0.03162659667496486) | > loss_mel_ce: 2.169710874557495 (nan) | > loss: 0.00876594614237547 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13655454191555094) | > loader_time: 0.0045 (0.004675933543935263)  --> TIME: 2025-02-26 10:06:35 -- STEP: 12534/36938 -- GLOBAL_STEP: 1194550 | > loss_text_ce: 0.023687992244958878 (0.03162346400288749) | > loss_mel_ce: 1.8157589435577393 (nan) | > loss: 0.0072993929497897625 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13654070134860707) | > loader_time: 0.0045 (0.004675765385254162)  --> TIME: 2025-02-26 10:06:44 -- STEP: 12584/36938 -- GLOBAL_STEP: 1194600 | > loss_text_ce: 0.037942156195640564 (0.03161781225689501) | > loss_mel_ce: 3.040802240371704 (nan) | > loss: 0.012217240408062935 (nan) | > current_lr: 5e-06 | > step_time: 0.1612 (0.13655272253137227) | > loader_time: 0.0048 (0.00467599679236949)  --> TIME: 2025-02-26 10:06:53 -- STEP: 12634/36938 -- GLOBAL_STEP: 1194650 | > loss_text_ce: 0.03444943204522133 (0.03161545120790335) | > loss_mel_ce: 2.3514556884765625 (nan) | > loss: 0.009467877447605133 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.1365480233506922) | > loader_time: 0.0045 (0.00467585996489218)  --> TIME: 2025-02-26 10:07:03 -- STEP: 12684/36938 -- GLOBAL_STEP: 1194700 | > loss_text_ce: 0.026042483747005463 (0.0316148766711355) | > loss_mel_ce: 2.996359348297119 (nan) | > loss: 0.011993658728897572 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13654830586605826) | > loader_time: 0.0047 (0.004675905661641596)  --> TIME: 2025-02-26 10:07:12 -- STEP: 12734/36938 -- GLOBAL_STEP: 1194750 | > loss_text_ce: 0.024735307320952415 (0.031610941152785225) | > loss_mel_ce: 2.9780707359313965 (nan) | > loss: 0.011915897019207478 (nan) | > current_lr: 5e-06 | > step_time: 0.1619 (0.1365756967742878) | > loader_time: 0.0048 (0.004675908217498269)  --> TIME: 2025-02-26 10:07:21 -- STEP: 12784/36938 -- GLOBAL_STEP: 1194800 | > loss_text_ce: 0.032783739268779755 (0.03161148744672018) | > loss_mel_ce: 2.560838460922241 (nan) | > loss: 0.01029215194284916 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13656130649419912) | > loader_time: 0.0046 (0.004675918175968272)  --> TIME: 2025-02-26 10:07:31 -- STEP: 12834/36938 -- GLOBAL_STEP: 1194850 | > loss_text_ce: 0.023044854402542114 (0.03160637850181339) | > loss_mel_ce: 2.8369545936584473 (nan) | > loss: 0.011349204927682877 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13657054418861458) | > loader_time: 0.0046 (0.004676255831303063)  --> TIME: 2025-02-26 10:07:40 -- STEP: 12884/36938 -- GLOBAL_STEP: 1194900 | > loss_text_ce: 0.027276646345853806 (0.031607806819928724) | > loss_mel_ce: 2.50559139251709 (nan) | > loss: 0.010051064193248749 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13656025951007206) | > loader_time: 0.0045 (0.004676037584539899)  --> TIME: 2025-02-26 10:07:49 -- STEP: 12934/36938 -- GLOBAL_STEP: 1194950 | > loss_text_ce: 0.03169867396354675 (0.03160980870584061) | > loss_mel_ce: 3.0271332263946533 (nan) | > loss: 0.012138222344219685 (nan) | > current_lr: 5e-06 | > step_time: 0.1778 (0.13656237865685225) | > loader_time: 0.0074 (0.004676130891896966)  --> TIME: 2025-02-26 10:07:58 -- STEP: 12984/36938 -- GLOBAL_STEP: 1195000 | > loss_text_ce: 0.032300885766744614 (0.03160887555025868) | > loss_mel_ce: 2.3292665481567383 (nan) | > loss: 0.009371300227940083 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13655496376279205) | > loader_time: 0.0045 (0.004676117125074587)  --> TIME: 2025-02-26 10:08:08 -- STEP: 13034/36938 -- GLOBAL_STEP: 1195050 | > loss_text_ce: 0.028112640604376793 (0.03160745269040601) | > loss_mel_ce: 3.289980173110962 (nan) | > loss: 0.013167035765945911 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.1365517047864823) | > loader_time: 0.0046 (0.004676192527865824)  --> TIME: 2025-02-26 10:08:17 -- STEP: 13084/36938 -- GLOBAL_STEP: 1195100 | > loss_text_ce: 0.03198562562465668 (0.031610117673000544) | > loss_mel_ce: 1.5916675329208374 (nan) | > loss: 0.0064430683851242065 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13656962637652134) | > loader_time: 0.0046 (0.004676111573219595)  --> TIME: 2025-02-26 10:08:26 -- STEP: 13134/36938 -- GLOBAL_STEP: 1195150 | > loss_text_ce: 0.029528245329856873 (0.031614116639368715) | > loss_mel_ce: 3.2030961513519287 (nan) | > loss: 0.012827875092625618 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.13655738540485118) | > loader_time: 0.0046 (0.004675780799230234)  --> TIME: 2025-02-26 10:08:36 -- STEP: 13184/36938 -- GLOBAL_STEP: 1195200 | > loss_text_ce: 0.04156957566738129 (0.031615267342009805) | > loss_mel_ce: 2.009143352508545 (nan) | > loss: 0.008137749508023262 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13658030369111163) | > loader_time: 0.0047 (0.004676373566961987)  --> TIME: 2025-02-26 10:08:45 -- STEP: 13234/36938 -- GLOBAL_STEP: 1195250 | > loss_text_ce: 0.03245720639824867 (0.03161399758937302) | > loss_mel_ce: 3.523066997528076 (nan) | > loss: 0.014109223149716854 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.1365775346611867) | > loader_time: 0.0047 (0.004676056895458539)  --> TIME: 2025-02-26 10:08:55 -- STEP: 13284/36938 -- GLOBAL_STEP: 1195300 | > loss_text_ce: 0.026156282052397728 (0.03161168629283962) | > loss_mel_ce: 2.3480334281921387 (nan) | > loss: 0.009421387687325478 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.13658892196619773) | > loader_time: 0.0047 (0.004676096933285039)  --> TIME: 2025-02-26 10:09:04 -- STEP: 13334/36938 -- GLOBAL_STEP: 1195350 | > loss_text_ce: 0.030767017975449562 (0.03161428875713282) | > loss_mel_ce: 3.1349668502807617 (nan) | > loss: 0.012562436051666737 (nan) | > current_lr: 5e-06 | > step_time: 0.1526 (0.13659196871232215) | > loader_time: 0.0048 (0.004675885968360661)  --> TIME: 2025-02-26 10:09:13 -- STEP: 13384/36938 -- GLOBAL_STEP: 1195400 | > loss_text_ce: 0.03196585178375244 (0.031615109165637416) | > loss_mel_ce: 2.7770652770996094 (nan) | > loss: 0.011146948672831059 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.1365777160938968) | > loader_time: 0.0047 (0.004675637728000786)  --> TIME: 2025-02-26 10:09:22 -- STEP: 13434/36938 -- GLOBAL_STEP: 1195450 | > loss_text_ce: 0.030370136722922325 (0.03161303177243017) | > loss_mel_ce: 3.119407892227173 (nan) | > loss: 0.012499120086431503 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13657635877188273) | > loader_time: 0.0046 (0.004675509994612472)  --> TIME: 2025-02-26 10:09:32 -- STEP: 13484/36938 -- GLOBAL_STEP: 1195500 | > loss_text_ce: 0.029630722478032112 (0.03161508348339199) | > loss_mel_ce: 2.2217190265655518 (nan) | > loss: 0.008933927863836288 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.1365904764194099) | > loader_time: 0.0047 (0.004675444492916091)  --> TIME: 2025-02-26 10:09:41 -- STEP: 13534/36938 -- GLOBAL_STEP: 1195550 | > loss_text_ce: 0.041933365166187286 (0.0316161807331691) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13659407686620068) | > loader_time: 0.0045 (0.004675295427981103)  --> TIME: 2025-02-26 10:09:51 -- STEP: 13584/36938 -- GLOBAL_STEP: 1195600 | > loss_text_ce: 0.032428011298179626 (0.03162215188040351) | > loss_mel_ce: 2.5660855770111084 (nan) | > loss: 0.010311562567949295 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.1365977786349757) | > loader_time: 0.0048 (0.004675133261298694)  --> TIME: 2025-02-26 10:10:01 -- STEP: 13634/36938 -- GLOBAL_STEP: 1195650 | > loss_text_ce: 0.04353150352835655 (0.03162408295816117) | > loss_mel_ce: 3.1483054161071777 (nan) | > loss: 0.01266601961106062 (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.1366342435891084) | > loader_time: 0.005 (0.00467496450230299)  --> TIME: 2025-02-26 10:10:10 -- STEP: 13684/36938 -- GLOBAL_STEP: 1195700 | > loss_text_ce: 0.03847567364573479 (0.03162370303266523) | > loss_mel_ce: 2.768292188644409 (nan) | > loss: 0.011137968860566616 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13661439550860427) | > loader_time: 0.0045 (0.004675076949390822)  --> TIME: 2025-02-26 10:10:19 -- STEP: 13734/36938 -- GLOBAL_STEP: 1195750 | > loss_text_ce: 0.02315974421799183 (0.031621083716147524) | > loss_mel_ce: 2.804906129837036 (nan) | > loss: 0.011222484521567822 (nan) | > current_lr: 5e-06 | > step_time: 0.1262 (0.13660478041615515) | > loader_time: 0.0045 (0.004675123409289582)  --> TIME: 2025-02-26 10:10:28 -- STEP: 13784/36938 -- GLOBAL_STEP: 1195800 | > loss_text_ce: 0.03534316271543503 (0.03162196292821666) | > loss_mel_ce: 2.547136068344116 (nan) | > loss: 0.010247933678328991 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13660448628736765) | > loader_time: 0.0046 (0.004675292598599286)  --> TIME: 2025-02-26 10:10:38 -- STEP: 13834/36938 -- GLOBAL_STEP: 1195850 | > loss_text_ce: 0.030500847846269608 (0.031623323766095626) | > loss_mel_ce: 3.1268112659454346 (nan) | > loss: 0.01252901740372181 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13661504965461438) | > loader_time: 0.0046 (0.004675477868097108)  --> TIME: 2025-02-26 10:10:47 -- STEP: 13884/36938 -- GLOBAL_STEP: 1195900 | > loss_text_ce: 0.02806594967842102 (0.03161873712630503) | > loss_mel_ce: 1.060089111328125 (nan) | > loss: 0.004318075720220804 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13662913214126718) | > loader_time: 0.0046 (0.004675309430003966)  --> TIME: 2025-02-26 10:10:56 -- STEP: 13934/36938 -- GLOBAL_STEP: 1195950 | > loss_text_ce: 0.03232818841934204 (0.0316200704123729) | > loss_mel_ce: 3.0364773273468018 (nan) | > loss: 0.012177799828350544 (nan) | > current_lr: 5e-06 | > step_time: 0.1703 (0.1366147226730868) | > loader_time: 0.0047 (0.004675440403623606)  --> TIME: 2025-02-26 10:11:06 -- STEP: 13984/36938 -- GLOBAL_STEP: 1196000 | > loss_text_ce: 0.031639814376831055 (0.03161822672480751) | > loss_mel_ce: 1.41225004196167 (nan) | > loss: 0.005729721859097481 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13662770484282272) | > loader_time: 0.0044 (0.004675638706378598)  --> TIME: 2025-02-26 10:11:15 -- STEP: 14034/36938 -- GLOBAL_STEP: 1196050 | > loss_text_ce: 0.017729077488183975 (0.0316188133735006) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.1366164841780362) | > loader_time: 0.0045 (0.004675501888253955)  --> TIME: 2025-02-26 10:11:24 -- STEP: 14084/36938 -- GLOBAL_STEP: 1196100 | > loss_text_ce: 0.0303462203592062 (0.031624356513468566) | > loss_mel_ce: 2.6659140586853027 (nan) | > loss: 0.010699446313083172 (nan) | > current_lr: 5e-06 | > step_time: 0.1889 (0.1366200280338494) | > loader_time: 0.005 (0.0046754768882682225)  --> TIME: 2025-02-26 10:11:34 -- STEP: 14134/36938 -- GLOBAL_STEP: 1196150 | > loss_text_ce: 0.02485370822250843 (0.0316227669997644) | > loss_mel_ce: 2.316357374191284 (nan) | > loss: 0.009290521033108234 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13661722885899535) | > loader_time: 0.0045 (0.0046757151454042)  --> TIME: 2025-02-26 10:11:43 -- STEP: 14184/36938 -- GLOBAL_STEP: 1196200 | > loss_text_ce: 0.03644963726401329 (0.03161988727118336) | > loss_mel_ce: 2.9996583461761475 (nan) | > loss: 0.012048047967255116 (nan) | > current_lr: 5e-06 | > step_time: 0.1753 (0.13661232449480043) | > loader_time: 0.0048 (0.004675689149696287)  --> TIME: 2025-02-26 10:11:53 -- STEP: 14234/36938 -- GLOBAL_STEP: 1196250 | > loss_text_ce: 0.030034121125936508 (0.03162155867998792) | > loss_mel_ce: 3.059082269668579 (nan) | > loss: 0.012258399277925491 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13663688710319888) | > loader_time: 0.0045 (0.004675749615536199)  --> TIME: 2025-02-26 10:12:02 -- STEP: 14284/36938 -- GLOBAL_STEP: 1196300 | > loss_text_ce: 0.0328943207859993 (0.03161833153650846) | > loss_mel_ce: 2.384953498840332 (nan) | > loss: 0.009594635106623173 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13664244846014553) | > loader_time: 0.0052 (0.004676021086805786)  --> TIME: 2025-02-26 10:12:11 -- STEP: 14334/36938 -- GLOBAL_STEP: 1196350 | > loss_text_ce: 0.02936330810189247 (0.03161468805720053) | > loss_mel_ce: 2.6098387241363525 (nan) | > loss: 0.010473025031387806 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13663543052415314) | > loader_time: 0.0045 (0.004675853064457086)  --> TIME: 2025-02-26 10:12:20 -- STEP: 14384/36938 -- GLOBAL_STEP: 1196400 | > loss_text_ce: 0.03589225560426712 (0.031612958178158436) | > loss_mel_ce: 2.8630502223968506 (nan) | > loss: 0.011503740213811398 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.136633708957041) | > loader_time: 0.0049 (0.004675984216876777)  --> TIME: 2025-02-26 10:12:30 -- STEP: 14434/36938 -- GLOBAL_STEP: 1196450 | > loss_text_ce: 0.030551282688975334 (0.031612931213884955) | > loss_mel_ce: 3.822812557220459 (nan) | > loss: 0.015291126444935799 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13663266609295605) | > loader_time: 0.0046 (0.004675811985877978)  --> TIME: 2025-02-26 10:12:39 -- STEP: 14484/36938 -- GLOBAL_STEP: 1196500 | > loss_text_ce: 0.032099608331918716 (0.031605521369508546) | > loss_mel_ce: 0.0893820971250534 (nan) | > loss: 0.0004820702597498894 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13663761219376638) | > loader_time: 0.0045 (0.004675709453942014)  --> TIME: 2025-02-26 10:12:49 -- STEP: 14534/36938 -- GLOBAL_STEP: 1196550 | > loss_text_ce: 0.03321150317788124 (0.031603085058476796) | > loss_mel_ce: 2.168104410171509 (nan) | > loss: 0.008735381066799164 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.13664423712382365) | > loader_time: 0.0047 (0.004675417978563131)  --> TIME: 2025-02-26 10:12:58 -- STEP: 14584/36938 -- GLOBAL_STEP: 1196600 | > loss_text_ce: 0.036597393453121185 (0.03160426288511615) | > loss_mel_ce: 3.149611473083496 (nan) | > loss: 0.012643687427043915 (nan) | > current_lr: 5e-06 | > step_time: 0.1798 (0.1366516553760568) | > loader_time: 0.0069 (0.00467530831294079)  --> TIME: 2025-02-26 10:13:07 -- STEP: 14634/36938 -- GLOBAL_STEP: 1196650 | > loss_text_ce: 0.028766831383109093 (0.031608996567011924) | > loss_mel_ce: 2.61795973777771 (nan) | > loss: 0.010502884164452553 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13665974791851326) | > loader_time: 0.0045 (0.0046755367897891925)  --> TIME: 2025-02-26 10:13:17 -- STEP: 14684/36938 -- GLOBAL_STEP: 1196700 | > loss_text_ce: 0.034913018345832825 (0.031608356247640725) | > loss_mel_ce: 2.5855395793914795 (nan) | > loss: 0.01039862260222435 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.1366643881122359) | > loader_time: 0.0046 (0.004675464372289382)  --> TIME: 2025-02-26 10:13:26 -- STEP: 14734/36938 -- GLOBAL_STEP: 1196750 | > loss_text_ce: 0.023904554545879364 (0.031609647892778765) | > loss_mel_ce: 2.806673526763916 (nan) | > loss: 0.011232453398406506 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13666876459180072) | > loader_time: 0.0045 (0.0046752110028567976)  --> TIME: 2025-02-26 10:13:36 -- STEP: 14784/36938 -- GLOBAL_STEP: 1196800 | > loss_text_ce: 0.03811154514551163 (0.03160825672494446) | > loss_mel_ce: 2.5212290287017822 (nan) | > loss: 0.010156113654375076 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13667208480435133) | > loader_time: 0.0045 (0.004675146740623377)  --> TIME: 2025-02-26 10:13:45 -- STEP: 14834/36938 -- GLOBAL_STEP: 1196850 | > loss_text_ce: 0.03627322241663933 (0.0316081514243765) | > loss_mel_ce: 2.8670475482940674 (nan) | > loss: 0.011521114967763424 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13666314114067596) | > loader_time: 0.0045 (0.004674759550173047)  --> TIME: 2025-02-26 10:13:54 -- STEP: 14884/36938 -- GLOBAL_STEP: 1196900 | > loss_text_ce: 0.035465408116579056 (0.03161445920833937) | > loss_mel_ce: 2.540024995803833 (nan) | > loss: 0.010220200754702091 (nan) | > current_lr: 5e-06 | > step_time: 0.1754 (0.136641623960392) | > loader_time: 0.0048 (0.004675010364884117)  --> TIME: 2025-02-26 10:14:03 -- STEP: 14934/36938 -- GLOBAL_STEP: 1196950 | > loss_text_ce: 0.0332987904548645 (0.03161567466956403) | > loss_mel_ce: 2.5228047370910645 (nan) | > loss: 0.010143267922103405 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1366269372626143) | > loader_time: 0.0044 (0.00467484644239362)  --> TIME: 2025-02-26 10:14:12 -- STEP: 14984/36938 -- GLOBAL_STEP: 1197000 | > loss_text_ce: 0.03633395954966545 (0.03162232380933696) | > loss_mel_ce: 1.1778610944747925 (nan) | > loss: 0.004818234592676163 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13663355924201132) | > loader_time: 0.0046 (0.004674735342317011)  --> TIME: 2025-02-26 10:14:22 -- STEP: 15034/36938 -- GLOBAL_STEP: 1197050 | > loss_text_ce: 0.024630941450595856 (0.0316234158314355) | > loss_mel_ce: 1.783213496208191 (nan) | > loss: 0.0071739861741662025 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1366393082919277) | > loader_time: 0.0045 (0.004674965085319761)  --> TIME: 2025-02-26 10:14:31 -- STEP: 15084/36938 -- GLOBAL_STEP: 1197100 | > loss_text_ce: 0.033246222883462906 (0.031626787728543014) | > loss_mel_ce: 2.892709970474243 (nan) | > loss: 0.011610938236117363 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.13663937584913288) | > loader_time: 0.0045 (0.004674794376133554)  --> TIME: 2025-02-26 10:14:40 -- STEP: 15134/36938 -- GLOBAL_STEP: 1197150 | > loss_text_ce: 0.03467189148068428 (0.03162578767313495) | > loss_mel_ce: 3.091475486755371 (nan) | > loss: 0.01240534707903862 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.1366439688169496) | > loader_time: 0.0044 (0.004674771116576085)  --> TIME: 2025-02-26 10:14:50 -- STEP: 15184/36938 -- GLOBAL_STEP: 1197200 | > loss_text_ce: 0.026817474514245987 (0.03162710923427736) | > loss_mel_ce: 2.355088472366333 (nan) | > loss: 0.00945200864225626 (nan) | > current_lr: 5e-06 | > step_time: 0.1642 (0.1366439753450519) | > loader_time: 0.0046 (0.0046747128064062215)  --> TIME: 2025-02-26 10:14:59 -- STEP: 15234/36938 -- GLOBAL_STEP: 1197250 | > loss_text_ce: 0.03654921054840088 (0.03163021501625434) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13664882699852102) | > loader_time: 0.0053 (0.004674773086662047)  --> TIME: 2025-02-26 10:15:09 -- STEP: 15284/36938 -- GLOBAL_STEP: 1197300 | > loss_text_ce: 0.04249603673815727 (0.03163256268582479) | > loss_mel_ce: 2.6938061714172363 (nan) | > loss: 0.010858342051506042 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13665490515785653) | > loader_time: 0.0046 (0.004674816764920406)  --> TIME: 2025-02-26 10:15:18 -- STEP: 15334/36938 -- GLOBAL_STEP: 1197350 | > loss_text_ce: 0.028393855318427086 (0.03162899635227108) | > loss_mel_ce: 3.1170215606689453 (nan) | > loss: 0.012481807731091976 (nan) | > current_lr: 5e-06 | > step_time: 0.1651 (0.13664247365543408) | > loader_time: 0.0046 (0.004674585387684066)  --> TIME: 2025-02-26 10:15:27 -- STEP: 15384/36938 -- GLOBAL_STEP: 1197400 | > loss_text_ce: 0.02702629752457142 (0.031630638447247394) | > loss_mel_ce: 2.778066873550415 (nan) | > loss: 0.01113132294267416 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.1366262760592584) | > loader_time: 0.0048 (0.004674325045730949)  --> TIME: 2025-02-26 10:15:36 -- STEP: 15434/36938 -- GLOBAL_STEP: 1197450 | > loss_text_ce: 0.023465780541300774 (0.031628589971864636) | > loss_mel_ce: 2.122584104537964 (nan) | > loss: 0.008516071364283562 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.1366202313618549) | > loader_time: 0.0047 (0.004673951800134937)  --> TIME: 2025-02-26 10:15:45 -- STEP: 15484/36938 -- GLOBAL_STEP: 1197500 | > loss_text_ce: 0.025570664554834366 (0.03162692143407149) | > loss_mel_ce: 2.7436652183532715 (nan) | > loss: 0.010989031754434109 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13662104520709145) | > loader_time: 0.0045 (0.004673593021492363)  --> TIME: 2025-02-26 10:15:55 -- STEP: 15534/36938 -- GLOBAL_STEP: 1197550 | > loss_text_ce: 0.031061016023159027 (0.031626386360210375) | > loss_mel_ce: 1.7831990718841553 (nan) | > loss: 0.007199445273727179 (nan) | > current_lr: 5e-06 | > step_time: 0.1775 (0.13662468734757033) | > loader_time: 0.0048 (0.004673583298495682)  --> TIME: 2025-02-26 10:16:04 -- STEP: 15584/36938 -- GLOBAL_STEP: 1197600 | > loss_text_ce: 0.0332743264734745 (0.03162680773665225) | > loss_mel_ce: 2.687767267227173 (nan) | > loss: 0.01079778466373682 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13662302853574113) | > loader_time: 0.0044 (0.004673189099197279)  --> TIME: 2025-02-26 10:16:13 -- STEP: 15634/36938 -- GLOBAL_STEP: 1197650 | > loss_text_ce: 0.022093573585152626 (0.031630727872731475) | > loss_mel_ce: 2.5414726734161377 (nan) | > loss: 0.010172882117331028 (nan) | > current_lr: 5e-06 | > step_time: 0.1577 (0.13663090664635674) | > loader_time: 0.0046 (0.004673223567463431)  --> TIME: 2025-02-26 10:16:22 -- STEP: 15684/36938 -- GLOBAL_STEP: 1197700 | > loss_text_ce: 0.03169544041156769 (0.03162943678589776) | > loss_mel_ce: 0.0030670166015625 (nan) | > loss: 0.00013794626283925027 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13661759912405336) | > loader_time: 0.0046 (0.004673064043861793)  --> TIME: 2025-02-26 10:16:32 -- STEP: 15734/36938 -- GLOBAL_STEP: 1197750 | > loss_text_ce: 0.032290007919073105 (0.031630631853809016) | > loss_mel_ce: 1.3588848114013672 (nan) | > loss: 0.005520535167306662 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13662706981306033) | > loader_time: 0.0045 (0.004672994316040956)  --> TIME: 2025-02-26 10:16:41 -- STEP: 15784/36938 -- GLOBAL_STEP: 1197800 | > loss_text_ce: 0.03275759890675545 (0.03163276752215115) | > loss_mel_ce: 2.6142520904541016 (nan) | > loss: 0.010504007339477539 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.13663163377860552) | > loader_time: 0.0046 (0.004673073784808377)  --> TIME: 2025-02-26 10:16:50 -- STEP: 15834/36938 -- GLOBAL_STEP: 1197850 | > loss_text_ce: 0.034366510808467865 (0.03163276222978326) | > loss_mel_ce: 2.714860200881958 (nan) | > loss: 0.010909630917012691 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13662831419207297) | > loader_time: 0.0046 (0.004673228294572048)  --> TIME: 2025-02-26 10:16:59 -- STEP: 15884/36938 -- GLOBAL_STEP: 1197900 | > loss_text_ce: 0.03934096172451973 (0.03163436759943191) | > loss_mel_ce: 2.282355308532715 (nan) | > loss: 0.009213080629706383 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13660951472445582) | > loader_time: 0.0047 (0.0046730924391140205)  --> TIME: 2025-02-26 10:17:09 -- STEP: 15934/36938 -- GLOBAL_STEP: 1197950 | > loss_text_ce: 0.027972452342510223 (0.03163886718664504) | > loss_mel_ce: 2.0825889110565186 (nan) | > loss: 0.008375244215130806 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13660572768305196) | > loader_time: 0.0044 (0.0046728734047182235)  --> TIME: 2025-02-26 10:17:18 -- STEP: 15984/36938 -- GLOBAL_STEP: 1198000 | > loss_text_ce: 0.01966923102736473 (0.03163696400192178) | > loss_mel_ce: 1.4804054498672485 (nan) | > loss: 0.005952677223831415 (nan) | > current_lr: 5e-06 | > step_time: 0.1077 (0.13659850088087153) | > loader_time: 0.0043 (0.004672608561105326)  --> TIME: 2025-02-26 10:17:27 -- STEP: 16034/36938 -- GLOBAL_STEP: 1198050 | > loss_text_ce: 0.03473486378788948 (0.03163712865029324) | > loss_mel_ce: 2.793562650680542 (nan) | > loss: 0.011223403736948967 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13660145455708225) | > loader_time: 0.0046 (0.0046725068229741045)  --> TIME: 2025-02-26 10:17:36 -- STEP: 16084/36938 -- GLOBAL_STEP: 1198100 | > loss_text_ce: 0.035342130810022354 (0.03163945393241233) | > loss_mel_ce: 2.5145926475524902 (nan) | > loss: 0.010118789970874786 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13658710105358188) | > loader_time: 0.0045 (0.004672457406367282)  --> TIME: 2025-02-26 10:17:46 -- STEP: 16134/36938 -- GLOBAL_STEP: 1198150 | > loss_text_ce: 0.029350697994232178 (0.03163680352454867) | > loss_mel_ce: 2.4401652812957764 (nan) | > loss: 0.00979966763406992 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13659257795400875) | > loader_time: 0.0046 (0.004672369195047456)  --> TIME: 2025-02-26 10:17:55 -- STEP: 16184/36938 -- GLOBAL_STEP: 1198200 | > loss_text_ce: 0.030109018087387085 (0.031636720661808775) | > loss_mel_ce: 2.869236707687378 (nan) | > loss: 0.011505340225994587 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13660305476224116) | > loader_time: 0.0046 (0.004672214352017995)  --> TIME: 2025-02-26 10:18:04 -- STEP: 16234/36938 -- GLOBAL_STEP: 1198250 | > loss_text_ce: 0.03191055357456207 (0.031636645608164406) | > loss_mel_ce: 2.8998863697052 (nan) | > loss: 0.011634116061031818 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13659070358203623) | > loader_time: 0.0046 (0.004671943001196695)  --> TIME: 2025-02-26 10:18:14 -- STEP: 16284/36938 -- GLOBAL_STEP: 1198300 | > loss_text_ce: 0.036326758563518524 (0.031639930352800244) | > loss_mel_ce: 2.3045334815979004 (nan) | > loss: 0.00928912777453661 (nan) | > current_lr: 5e-06 | > step_time: 0.1532 (0.1365955582826981) | > loader_time: 0.0046 (0.004672014531735508)  --> TIME: 2025-02-26 10:18:23 -- STEP: 16334/36938 -- GLOBAL_STEP: 1198350 | > loss_text_ce: 0.03503518924117088 (0.031638432169034204) | > loss_mel_ce: 2.269097328186035 (nan) | > loss: 0.009143383242189884 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13658731954690756) | > loader_time: 0.0048 (0.004671778762969662)  --> TIME: 2025-02-26 10:18:32 -- STEP: 16384/36938 -- GLOBAL_STEP: 1198400 | > loss_text_ce: 0.0320141576230526 (0.03163914128049338) | > loss_mel_ce: 2.4280014038085938 (nan) | > loss: 0.00976196676492691 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1365942897828073) | > loader_time: 0.0045 (0.004671503120334828)  --> TIME: 2025-02-26 10:18:42 -- STEP: 16434/36938 -- GLOBAL_STEP: 1198450 | > loss_text_ce: 0.027047455310821533 (0.031644203448508414) | > loss_mel_ce: 0.01590983010828495 (nan) | > loss: 0.00017046542780008167 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.1366109410617082) | > loader_time: 0.0045 (0.004671452804770634)  --> TIME: 2025-02-26 10:18:51 -- STEP: 16484/36938 -- GLOBAL_STEP: 1198500 | > loss_text_ce: 0.039570048451423645 (0.031643317142278755) | > loss_mel_ce: 3.226055145263672 (nan) | > loss: 0.012958831153810024 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13661132637547957) | > loader_time: 0.0052 (0.004671315246055332)  --> TIME: 2025-02-26 10:19:00 -- STEP: 16534/36938 -- GLOBAL_STEP: 1198550 | > loss_text_ce: 0.030907556414604187 (0.03164080628972937) | > loss_mel_ce: 1.3017311096191406 (nan) | > loss: 0.005288248881697655 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13660711682346158) | > loader_time: 0.0045 (0.004671242111061172)  --> TIME: 2025-02-26 10:19:10 -- STEP: 16584/36938 -- GLOBAL_STEP: 1198600 | > loss_text_ce: 0.02755257487297058 (0.031643004530043074) | > loss_mel_ce: 2.921630859375 (nan) | > loss: 0.011703109368681908 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.1366035291323969) | > loader_time: 0.0046 (0.0046711380045826945)  --> TIME: 2025-02-26 10:19:19 -- STEP: 16634/36938 -- GLOBAL_STEP: 1198650 | > loss_text_ce: 0.029142653569579124 (0.031645604585139285) | > loss_mel_ce: 2.791078567504883 (nan) | > loss: 0.011191354133188725 (nan) | > current_lr: 5e-06 | > step_time: 0.1076 (0.1365923620379726) | > loader_time: 0.0044 (0.004670997085630924)  --> TIME: 2025-02-26 10:19:28 -- STEP: 16684/36938 -- GLOBAL_STEP: 1198700 | > loss_text_ce: 0.03236709535121918 (0.031644023972340486) | > loss_mel_ce: 2.8856797218322754 (nan) | > loss: 0.011579550802707672 (nan) | > current_lr: 5e-06 | > step_time: 0.1526 (0.13658206458858943) | > loader_time: 0.0046 (0.004670878218049722)  --> TIME: 2025-02-26 10:19:37 -- STEP: 16734/36938 -- GLOBAL_STEP: 1198750 | > loss_text_ce: 0.023780876770615578 (0.03164216990561395) | > loss_mel_ce: 2.95434308052063 (nan) | > loss: 0.011817952618002892 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13657089302413702) | > loader_time: 0.0045 (0.004670527640445646)  --> TIME: 2025-02-26 10:19:46 -- STEP: 16784/36938 -- GLOBAL_STEP: 1198800 | > loss_text_ce: 0.018528668209910393 (0.03164279562906347) | > loss_mel_ce: 2.8937859535217285 (nan) | > loss: 0.011556805111467838 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13656379367705296) | > loader_time: 0.0046 (0.004670163966338909)  --> TIME: 2025-02-26 10:19:55 -- STEP: 16834/36938 -- GLOBAL_STEP: 1198850 | > loss_text_ce: 0.031781818717718124 (0.0316495532111296) | > loss_mel_ce: 3.2466917037963867 (nan) | > loss: 0.013009816408157349 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.13656346010768192) | > loader_time: 0.0046 (0.004670138921054459)  --> TIME: 2025-02-26 10:20:05 -- STEP: 16884/36938 -- GLOBAL_STEP: 1198900 | > loss_text_ce: 0.0352487675845623 (0.031649172201808386) | > loss_mel_ce: 0.03466796875 (nan) | > loss: 0.00027744739782065153 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13655364800053166) | > loader_time: 0.0046 (0.004669856570451026)  --> TIME: 2025-02-26 10:20:15 -- STEP: 16934/36938 -- GLOBAL_STEP: 1198950 | > loss_text_ce: 0.024367718026041985 (0.03165083513714521) | > loss_mel_ce: 0.0253304373472929 (nan) | > loss: 0.00019721491844393313 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.1365823465409024) | > loader_time: 0.0044 (0.004669781163153583)  --> TIME: 2025-02-26 10:20:24 -- STEP: 16984/36938 -- GLOBAL_STEP: 1199000 | > loss_text_ce: 0.021336913108825684 (0.03164963213353133) | > loss_mel_ce: 0.6931619644165039 (nan) | > loss: 0.0028353130910545588 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.1365996927682723) | > loader_time: 0.0045 (0.004669533534483482)  --> TIME: 2025-02-26 10:20:34 -- STEP: 17034/36938 -- GLOBAL_STEP: 1199050 | > loss_text_ce: 0.036283496767282486 (0.031650607123795284) | > loss_mel_ce: 4.47265625 (nan) | > loss: 0.017892619594931602 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13660261378232952) | > loader_time: 0.0045 (0.004669301048248358)  --> TIME: 2025-02-26 10:20:43 -- STEP: 17084/36938 -- GLOBAL_STEP: 1199100 | > loss_text_ce: 0.027503997087478638 (0.031649071981675235) | > loss_mel_ce: 1.6374540328979492 (nan) | > loss: 0.006606976501643658 (nan) | > current_lr: 5e-06 | > step_time: 0.1511 (0.136619298415117) | > loader_time: 0.0047 (0.004669138794061433)  --> TIME: 2025-02-26 10:20:52 -- STEP: 17134/36938 -- GLOBAL_STEP: 1199150 | > loss_text_ce: 0.03897710144519806 (0.031654547437507265) | > loss_mel_ce: 2.983670234680176 (nan) | > loss: 0.01199463289231062 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13661488182029685) | > loader_time: 0.0045 (0.004668856899977937)  --> TIME: 2025-02-26 10:21:02 -- STEP: 17184/36938 -- GLOBAL_STEP: 1199200 | > loss_text_ce: 0.039077166467905045 (0.031657193139741004) | > loss_mel_ce: 1.496739387512207 (nan) | > loss: 0.006094510667026043 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.1366104226649595) | > loader_time: 0.0044 (0.004668610167014988)  --> TIME: 2025-02-26 10:21:11 -- STEP: 17234/36938 -- GLOBAL_STEP: 1199250 | > loss_text_ce: 0.031106851994991302 (0.03165857807081517) | > loss_mel_ce: 3.0816123485565186 (nan) | > loss: 0.012352061457931995 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13662387744531365) | > loader_time: 0.0048 (0.004668404777379006)  --> TIME: 2025-02-26 10:21:20 -- STEP: 17284/36938 -- GLOBAL_STEP: 1199300 | > loss_text_ce: 0.025514647364616394 (0.03165452713239498) | > loss_mel_ce: 1.7379556894302368 (nan) | > loss: 0.00699789822101593 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13661251568016364) | > loader_time: 0.0044 (0.004668069779665992)  --> TIME: 2025-02-26 10:21:29 -- STEP: 17334/36938 -- GLOBAL_STEP: 1199350 | > loss_text_ce: 0.037443943321704865 (0.031655284318975525) | > loss_mel_ce: 1.5432723760604858 (nan) | > loss: 0.006272684317082167 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1366034491576314) | > loader_time: 0.0044 (0.004667774882987264)  --> TIME: 2025-02-26 10:21:38 -- STEP: 17384/36938 -- GLOBAL_STEP: 1199400 | > loss_text_ce: 0.02715144492685795 (0.03165203358744894) | > loss_mel_ce: 2.8862357139587402 (nan) | > loss: 0.011561060324311256 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.13659003800749062) | > loader_time: 0.0046 (0.004667498867357056)  --> TIME: 2025-02-26 10:21:48 -- STEP: 17434/36938 -- GLOBAL_STEP: 1199450 | > loss_text_ce: 0.03187780827283859 (0.031650045190360895) | > loss_mel_ce: 2.9312045574188232 (nan) | > loss: 0.011758264154195786 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.13657601935143998) | > loader_time: 0.0048 (0.004667182259703506)  --> TIME: 2025-02-26 10:21:57 -- STEP: 17484/36938 -- GLOBAL_STEP: 1199500 | > loss_text_ce: 0.030611714348196983 (0.03164706122938977) | > loss_mel_ce: 1.482843041419983 (nan) | > loss: 0.0060057733207941055 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13658515334102175) | > loader_time: 0.0045 (0.00466733021366327)  --> TIME: 2025-02-26 10:22:07 -- STEP: 17534/36938 -- GLOBAL_STEP: 1199550 | > loss_text_ce: 0.03303906321525574 (0.03164494204420654) | > loss_mel_ce: 2.868597984313965 (nan) | > loss: 0.011514433659613132 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13659350191077071) | > loader_time: 0.0045 (0.004667431391453528)  --> TIME: 2025-02-26 10:22:16 -- STEP: 17584/36938 -- GLOBAL_STEP: 1199600 | > loss_text_ce: 0.026690663769841194 (0.031645637725774944) | > loss_mel_ce: 3.203474760055542 (nan) | > loss: 0.012818117626011372 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1365905997018808) | > loader_time: 0.0045 (0.004667276477575095)  --> TIME: 2025-02-26 10:22:25 -- STEP: 17634/36938 -- GLOBAL_STEP: 1199650 | > loss_text_ce: 0.03631561994552612 (0.031646713191212324) | > loss_mel_ce: 3.2677884101867676 (nan) | > loss: 0.01311152521520853 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13658601154447783) | > loader_time: 0.0052 (0.0046674363315950575)  --> TIME: 2025-02-26 10:22:35 -- STEP: 17684/36938 -- GLOBAL_STEP: 1199700 | > loss_text_ce: 0.03561020642518997 (0.03164561738322671) | > loss_mel_ce: 3.018561840057373 (nan) | > loss: 0.012119730934500694 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13659430847155138) | > loader_time: 0.0048 (0.004667676997060729)  --> TIME: 2025-02-26 10:22:44 -- STEP: 17734/36938 -- GLOBAL_STEP: 1199750 | > loss_text_ce: 0.02680540271103382 (0.031645921885978305) | > loss_mel_ce: 3.0695748329162598 (nan) | > loss: 0.01228722371160984 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.1366048041763481) | > loader_time: 0.0046 (0.00466747493395425)  --> TIME: 2025-02-26 10:22:53 -- STEP: 17784/36938 -- GLOBAL_STEP: 1199800 | > loss_text_ce: 0.028006207197904587 (0.03164530029925528) | > loss_mel_ce: 2.5675230026245117 (nan) | > loss: 0.010299719870090485 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13659675556745826) | > loader_time: 0.0044 (0.0046672541522465)  --> TIME: 2025-02-26 10:23:02 -- STEP: 17834/36938 -- GLOBAL_STEP: 1199850 | > loss_text_ce: 0.021806523203849792 (0.031644432677953194) | > loss_mel_ce: 2.6546263694763184 (nan) | > loss: 0.010620766319334507 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13658601040929563) | > loader_time: 0.0047 (0.0046669561338622614)  --> TIME: 2025-02-26 10:23:12 -- STEP: 17884/36938 -- GLOBAL_STEP: 1199900 | > loss_text_ce: 0.03241594135761261 (0.03164160913906317) | > loss_mel_ce: 2.7841670513153076 (nan) | > loss: 0.011176916770637035 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.13658902680306065) | > loader_time: 0.0047 (0.004666758287515568)  --> TIME: 2025-02-26 10:23:21 -- STEP: 17934/36938 -- GLOBAL_STEP: 1199950 | > loss_text_ce: 0.030327707529067993 (0.03164105440765296) | > loss_mel_ce: 2.7970333099365234 (nan) | > loss: 0.011219687759876251 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13657797179228073) | > loader_time: 0.0046 (0.0046665587658678565)  --> TIME: 2025-02-26 10:23:30 -- STEP: 17984/36938 -- GLOBAL_STEP: 1200000 | > loss_text_ce: 0.025966467335820198 (0.03163682414696057) | > loss_mel_ce: 2.4320132732391357 (nan) | > loss: 0.009753888472914696 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13657168264490968) | > loader_time: 0.0045 (0.004666323352644885) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1200000.pth  --> TIME: 2025-02-26 10:23:45 -- STEP: 18034/36938 -- GLOBAL_STEP: 1200050 | > loss_text_ce: 0.03335585817694664 (0.031638162275137036) | > loss_mel_ce: 2.769474744796753 (nan) | > loss: 0.011122344993054867 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.1365728007725167) | > loader_time: 0.0045 (0.004666384524776608)  --> TIME: 2025-02-26 10:23:54 -- STEP: 18084/36938 -- GLOBAL_STEP: 1200100 | > loss_text_ce: 0.03521593660116196 (0.03163937967244049) | > loss_mel_ce: 3.0889217853546143 (nan) | > loss: 0.012397372163832188 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13655564305635287) | > loader_time: 0.0046 (0.0046662616798292976)  --> TIME: 2025-02-26 10:24:03 -- STEP: 18134/36938 -- GLOBAL_STEP: 1200150 | > loss_text_ce: 0.035690102726221085 (0.03163661076938428) | > loss_mel_ce: 3.135836362838745 (nan) | > loss: 0.012585422955453396 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13654289322387544) | > loader_time: 0.0047 (0.004666171868554534)  --> TIME: 2025-02-26 10:24:12 -- STEP: 18184/36938 -- GLOBAL_STEP: 1200200 | > loss_text_ce: 0.027674708515405655 (0.031634769596504145) | > loss_mel_ce: 2.9241366386413574 (nan) | > loss: 0.011713537387549877 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13653480412673788) | > loader_time: 0.0046 (0.0046662603555353045)  --> TIME: 2025-02-26 10:24:21 -- STEP: 18234/36938 -- GLOBAL_STEP: 1200250 | > loss_text_ce: 0.033870283514261246 (0.03163360031583164) | > loss_mel_ce: 3.161524534225464 (nan) | > loss: 0.01268013846129179 (nan) | > current_lr: 5e-06 | > step_time: 0.182 (0.13652681221731408) | > loader_time: 0.0048 (0.004666294446963827)  --> TIME: 2025-02-26 10:24:31 -- STEP: 18284/36938 -- GLOBAL_STEP: 1200300 | > loss_text_ce: 0.03707824647426605 (0.03163546157168125) | > loss_mel_ce: 3.1466736793518066 (nan) | > loss: 0.01263393647968769 (nan) | > current_lr: 5e-06 | > step_time: 0.1788 (0.13654357649987459) | > loader_time: 0.0048 (0.004666190782695975)  --> TIME: 2025-02-26 10:24:40 -- STEP: 18334/36938 -- GLOBAL_STEP: 1200350 | > loss_text_ce: 0.02733178250491619 (0.03163165889187249) | > loss_mel_ce: 2.863978862762451 (nan) | > loss: 0.011473455466330051 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.1365514461926816) | > loader_time: 0.0046 (0.0046661626919413295)  --> TIME: 2025-02-26 10:24:49 -- STEP: 18384/36938 -- GLOBAL_STEP: 1200400 | > loss_text_ce: 0.01743633858859539 (0.03163099960314385) | > loss_mel_ce: 1.3060014247894287 (nan) | > loss: 0.005251737777143717 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13653259312618982) | > loader_time: 0.0045 (0.0046660143515667475)  --> TIME: 2025-02-26 10:24:59 -- STEP: 18434/36938 -- GLOBAL_STEP: 1200450 | > loss_text_ce: 0.031405866146087646 (0.03163032382931796) | > loss_mel_ce: 2.9485392570495605 (nan) | > loss: 0.011825179681181908 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.1365437685984193) | > loader_time: 0.0046 (0.004665951091451409)  --> TIME: 2025-02-26 10:25:08 -- STEP: 18484/36938 -- GLOBAL_STEP: 1200500 | > loss_text_ce: 0.03618519380688667 (0.031629668177312935) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1365411818298376) | > loader_time: 0.0045 (0.004665810704308664)  --> TIME: 2025-02-26 10:25:18 -- STEP: 18534/36938 -- GLOBAL_STEP: 1200550 | > loss_text_ce: 0.02699650637805462 (0.03163242939136242) | > loss_mel_ce: 2.1187686920166016 (nan) | > loss: 0.008514942601323128 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13654110899143482) | > loader_time: 0.0045 (0.004665645334061222)  --> TIME: 2025-02-26 10:25:27 -- STEP: 18584/36938 -- GLOBAL_STEP: 1200600 | > loss_text_ce: 0.033924952149391174 (0.03163621750195383) | > loss_mel_ce: 3.098905563354492 (nan) | > loss: 0.012431868351995945 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.1365483760371894) | > loader_time: 0.0047 (0.004665635381851339)  --> TIME: 2025-02-26 10:25:36 -- STEP: 18634/36938 -- GLOBAL_STEP: 1200650 | > loss_text_ce: 0.033605534583330154 (0.03163111076538423) | > loss_mel_ce: 1.7572753429412842 (nan) | > loss: 0.007106670644134283 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13654252089975122) | > loader_time: 0.0046 (0.0046655481128051995)  --> TIME: 2025-02-26 10:25:46 -- STEP: 18684/36938 -- GLOBAL_STEP: 1200700 | > loss_text_ce: 0.039555542171001434 (0.03162655367978955) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13653788966505828) | > loader_time: 0.0045 (0.004665538474033754)  --> TIME: 2025-02-26 10:25:55 -- STEP: 18734/36938 -- GLOBAL_STEP: 1200750 | > loss_text_ce: 0.03927183896303177 (0.031625692035905095) | > loss_mel_ce: 2.988053798675537 (nan) | > loss: 0.012013197876513004 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.1365469309924015) | > loader_time: 0.0047 (0.004665643959884354)  --> TIME: 2025-02-26 10:26:04 -- STEP: 18784/36938 -- GLOBAL_STEP: 1200800 | > loss_text_ce: 0.03495636209845543 (0.03162563728368718) | > loss_mel_ce: 3.28482723236084 (nan) | > loss: 0.013173745945096016 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.13652643695905284) | > loader_time: 0.0047 (0.0046655395444247846)  --> TIME: 2025-02-26 10:26:13 -- STEP: 18834/36938 -- GLOBAL_STEP: 1200850 | > loss_text_ce: 0.034174151718616486 (0.03162786601994652) | > loss_mel_ce: 2.7899014949798584 (nan) | > loss: 0.01120665017515421 (nan) | > current_lr: 5e-06 | > step_time: 0.1697 (0.1365303666624311) | > loader_time: 0.0048 (0.004665496509596978)  --> TIME: 2025-02-26 10:26:23 -- STEP: 18884/36938 -- GLOBAL_STEP: 1200900 | > loss_text_ce: 0.042569976300001144 (0.031629288488802955) | > loss_mel_ce: 2.674591064453125 (nan) | > loss: 0.010782385244965553 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13654055001497897) | > loader_time: 0.0046 (0.00466549377576752)  --> TIME: 2025-02-26 10:26:32 -- STEP: 18934/36938 -- GLOBAL_STEP: 1200950 | > loss_text_ce: 0.027985040098428726 (0.03162883897904702) | > loss_mel_ce: 1.8320002555847168 (nan) | > loss: 0.007380894850939512 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.1365247191956704) | > loader_time: 0.0045 (0.00466538703314441)  --> TIME: 2025-02-26 10:26:41 -- STEP: 18984/36938 -- GLOBAL_STEP: 1201000 | > loss_text_ce: 0.035937707871198654 (0.03162729837189935) | > loss_mel_ce: 3.099532127380371 (nan) | > loss: 0.01244234200567007 (nan) | > current_lr: 5e-06 | > step_time: 0.193 (0.13653273226789894) | > loader_time: 0.0048 (0.004665365562728337)  --> TIME: 2025-02-26 10:26:51 -- STEP: 19034/36938 -- GLOBAL_STEP: 1201050 | > loss_text_ce: 0.03187232464551926 (0.03162629675633683) | > loss_mel_ce: 2.372565746307373 (nan) | > loss: 0.009541421197354794 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.1365417626596274) | > loader_time: 0.0049 (0.004665294765261191)  --> TIME: 2025-02-26 10:27:00 -- STEP: 19084/36938 -- GLOBAL_STEP: 1201100 | > loss_text_ce: 0.02853991463780403 (0.03162671133619743) | > loss_mel_ce: 2.8324382305145264 (nan) | > loss: 0.011353088542819023 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13655039819874595) | > loader_time: 0.0046 (0.004665302907963327)  --> TIME: 2025-02-26 10:27:10 -- STEP: 19134/36938 -- GLOBAL_STEP: 1201150 | > loss_text_ce: 0.02507249265909195 (0.03162500189459191) | > loss_mel_ce: 1.3620330095291138 (nan) | > loss: 0.005504386965185404 (nan) | > current_lr: 5e-06 | > step_time: 0.1436 (0.13654942285482585) | > loader_time: 0.0047 (0.004665161195912317)  --> TIME: 2025-02-26 10:27:19 -- STEP: 19184/36938 -- GLOBAL_STEP: 1201200 | > loss_text_ce: 0.025733543559908867 (0.03162568746010558) | > loss_mel_ce: 0.7379111647605896 (nan) | > loss: 0.0030303362291306257 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13655518601843572) | > loader_time: 0.0045 (0.004665009024940612)  --> TIME: 2025-02-26 10:27:29 -- STEP: 19234/36938 -- GLOBAL_STEP: 1201250 | > loss_text_ce: 0.04364536702632904 (0.031625723548737515) | > loss_mel_ce: 2.511535406112671 (nan) | > loss: 0.010139606893062592 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.13655873869391968) | > loader_time: 0.0046 (0.004664864413168311)  --> TIME: 2025-02-26 10:27:38 -- STEP: 19284/36938 -- GLOBAL_STEP: 1201300 | > loss_text_ce: 0.022919470444321632 (0.03162573642834208) | > loss_mel_ce: 1.8376028537750244 (nan) | > loss: 0.007383025251328945 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13655108156552465) | > loader_time: 0.0045 (0.004664702957978623)  --> TIME: 2025-02-26 10:27:47 -- STEP: 19334/36938 -- GLOBAL_STEP: 1201350 | > loss_text_ce: 0.028066031634807587 (0.03162566182321418) | > loss_mel_ce: 2.985959053039551 (nan) | > loss: 0.011960417032241821 (nan) | > current_lr: 5e-06 | > step_time: 0.1809 (0.13654994831418732) | > loader_time: 0.0048 (0.004664526701442124)  --> TIME: 2025-02-26 10:27:56 -- STEP: 19384/36938 -- GLOBAL_STEP: 1201400 | > loss_text_ce: 0.022217197343707085 (0.03162513818361022) | > loss_mel_ce: 2.9589405059814453 (nan) | > loss: 0.011829991824924946 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13654998391313283) | > loader_time: 0.0046 (0.004664484413015525)  --> TIME: 2025-02-26 10:28:05 -- STEP: 19434/36938 -- GLOBAL_STEP: 1201450 | > loss_text_ce: 0.02685641497373581 (0.03162482726420557) | > loss_mel_ce: 3.093726396560669 (nan) | > loss: 0.012383265420794487 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13655038143463086) | > loader_time: 0.0047 (0.00466445979971915)  --> TIME: 2025-02-26 10:28:15 -- STEP: 19484/36938 -- GLOBAL_STEP: 1201500 | > loss_text_ce: 0.034172173589468 (0.03162369832833015) | > loss_mel_ce: 3.2200520038604736 (nan) | > loss: 0.012913588434457779 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13654485658756538) | > loader_time: 0.0046 (0.004664438726769376)  --> TIME: 2025-02-26 10:28:24 -- STEP: 19534/36938 -- GLOBAL_STEP: 1201550 | > loss_text_ce: 0.022513372823596 (0.03162453559532331) | > loss_mel_ce: 2.5315897464752197 (nan) | > loss: 0.010135330259799957 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13654635524242104) | > loader_time: 0.0046 (0.004664348838296863)  --> TIME: 2025-02-26 10:28:34 -- STEP: 19584/36938 -- GLOBAL_STEP: 1201600 | > loss_text_ce: 0.033549271523952484 (0.031627086944544545) | > loss_mel_ce: 3.332341194152832 (nan) | > loss: 0.0133567089214921 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.13656849589418044) | > loader_time: 0.0046 (0.004664385228667397)  --> TIME: 2025-02-26 10:28:43 -- STEP: 19634/36938 -- GLOBAL_STEP: 1201650 | > loss_text_ce: 0.03492652252316475 (0.03162785561290833) | > loss_mel_ce: 3.296800136566162 (nan) | > loss: 0.013221138156950474 (nan) | > current_lr: 5e-06 | > step_time: 0.1516 (0.13657353969143496) | > loader_time: 0.0047 (0.004664483545899244)  --> TIME: 2025-02-26 10:28:52 -- STEP: 19684/36938 -- GLOBAL_STEP: 1201700 | > loss_text_ce: 0.04283800348639488 (0.03162362697470683) | > loss_mel_ce: 2.908210277557373 (nan) | > loss: 0.011710509657859802 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.13657074636183442) | > loader_time: 0.0048 (0.0046647504756325295)  --> TIME: 2025-02-26 10:29:02 -- STEP: 19734/36938 -- GLOBAL_STEP: 1201750 | > loss_text_ce: 0.028446735814213753 (0.031621180350780105) | > loss_mel_ce: 2.984769582748413 (nan) | > loss: 0.011957207694649696 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13656916222795165) | > loader_time: 0.0046 (0.004664587856148534)  --> TIME: 2025-02-26 10:29:11 -- STEP: 19784/36938 -- GLOBAL_STEP: 1201800 | > loss_text_ce: 0.034472208470106125 (0.0316208882790321) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13656691422936845) | > loader_time: 0.0047 (0.004664542556435689)  --> TIME: 2025-02-26 10:29:20 -- STEP: 19834/36938 -- GLOBAL_STEP: 1201850 | > loss_text_ce: 0.027773333713412285 (0.03162001325161263) | > loss_mel_ce: 3.068795680999756 (nan) | > loss: 0.012287973426282406 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13655825479624245) | > loader_time: 0.0046 (0.004664413087776971)  --> TIME: 2025-02-26 10:29:29 -- STEP: 19884/36938 -- GLOBAL_STEP: 1201900 | > loss_text_ce: 0.031354885548353195 (0.03162096328798259) | > loss_mel_ce: 1.330654501914978 (nan) | > loss: 0.005404799710959196 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13655956443515407) | > loader_time: 0.0045 (0.004664331404790164)  --> TIME: 2025-02-26 10:29:39 -- STEP: 19934/36938 -- GLOBAL_STEP: 1201950 | > loss_text_ce: 0.036289017647504807 (0.03162373394555021) | > loss_mel_ce: 3.0658669471740723 (nan) | > loss: 0.012310143560171127 (nan) | > current_lr: 5e-06 | > step_time: 0.1563 (0.13656266893153618) | > loader_time: 0.0048 (0.004664211260276147)  --> TIME: 2025-02-26 10:29:48 -- STEP: 19984/36938 -- GLOBAL_STEP: 1202000 | > loss_text_ce: 0.03566914051771164 (0.03162339029393192) | > loss_mel_ce: 2.8966102600097656 (nan) | > loss: 0.011636029928922653 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.13655305621001748) | > loader_time: 0.0047 (0.004664119610412314)  --> TIME: 2025-02-26 10:29:57 -- STEP: 20034/36938 -- GLOBAL_STEP: 1202050 | > loss_text_ce: 0.04028899222612381 (0.031623068493636254) | > loss_mel_ce: 2.6321661472320557 (nan) | > loss: 0.01060498133301735 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.1365598438147175) | > loader_time: 0.0046 (0.004664095811671563)  --> TIME: 2025-02-26 10:30:07 -- STEP: 20084/36938 -- GLOBAL_STEP: 1202100 | > loss_text_ce: 0.030362648889422417 (0.03162409288780337) | > loss_mel_ce: 2.8867347240448 (nan) | > loss: 0.011575783602893353 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.1365587741249331) | > loader_time: 0.0046 (0.004664010152567965)  --> TIME: 2025-02-26 10:30:16 -- STEP: 20134/36938 -- GLOBAL_STEP: 1202150 | > loss_text_ce: 0.02970190718770027 (0.031623020918405916) | > loss_mel_ce: 2.81705641746521 (nan) | > loss: 0.011296660639345646 (nan) | > current_lr: 5e-06 | > step_time: 0.1898 (0.1365670898032429) | > loader_time: 0.0051 (0.004663951953260061)  --> TIME: 2025-02-26 10:30:25 -- STEP: 20184/36938 -- GLOBAL_STEP: 1202200 | > loss_text_ce: 0.031735774129629135 (0.031620341485129766) | > loss_mel_ce: 2.728180408477783 (nan) | > loss: 0.010952048934996128 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.1365535176324794) | > loader_time: 0.0046 (0.004663814652122027)  --> TIME: 2025-02-26 10:30:35 -- STEP: 20234/36938 -- GLOBAL_STEP: 1202250 | > loss_text_ce: 0.035341449081897736 (0.0316212286984792) | > loss_mel_ce: 3.4002699851989746 (nan) | > loss: 0.013633379712700844 (nan) | > current_lr: 5e-06 | > step_time: 0.1583 (0.13656246470075956) | > loader_time: 0.0047 (0.004663836841728424)  --> TIME: 2025-02-26 10:30:44 -- STEP: 20284/36938 -- GLOBAL_STEP: 1202300 | > loss_text_ce: 0.03219372406601906 (0.03162308999370696) | > loss_mel_ce: 3.0555853843688965 (nan) | > loss: 0.012253092601895332 (nan) | > current_lr: 5e-06 | > step_time: 0.1778 (0.13656727439246286) | > loader_time: 0.0048 (0.004663972806657856)  --> TIME: 2025-02-26 10:30:54 -- STEP: 20334/36938 -- GLOBAL_STEP: 1202350 | > loss_text_ce: 0.02646428532898426 (0.03162007098858126) | > loss_mel_ce: 2.3049747943878174 (nan) | > loss: 0.009251742623746395 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13658736909844085) | > loader_time: 0.0046 (0.004664234910096187)  --> TIME: 2025-02-26 10:31:03 -- STEP: 20384/36938 -- GLOBAL_STEP: 1202400 | > loss_text_ce: 0.029951099306344986 (0.03161724514485675) | > loss_mel_ce: 2.7570438385009766 (nan) | > loss: 0.011059504002332687 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.1365856868763179) | > loader_time: 0.0048 (0.004664304655986834)  --> TIME: 2025-02-26 10:31:13 -- STEP: 20434/36938 -- GLOBAL_STEP: 1202450 | > loss_text_ce: 0.02925807237625122 (0.031614670793898736) | > loss_mel_ce: 1.808131217956543 (nan) | > loss: 0.007291227579116821 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13659843339534616) | > loader_time: 0.0045 (0.00466445676148889)  --> TIME: 2025-02-26 10:31:23 -- STEP: 20484/36938 -- GLOBAL_STEP: 1202500 | > loss_text_ce: 0.032781932502985 (0.03161485652031753) | > loss_mel_ce: 2.906588554382324 (nan) | > loss: 0.011664168909192085 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.1366143401096032) | > loader_time: 0.0049 (0.004664518420665066)  --> TIME: 2025-02-26 10:31:32 -- STEP: 20534/36938 -- GLOBAL_STEP: 1202550 | > loss_text_ce: 0.02635841816663742 (0.031610473109915715) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13661458448854275) | > loader_time: 0.0045 (0.0046644310320992305)  --> TIME: 2025-02-26 10:31:41 -- STEP: 20584/36938 -- GLOBAL_STEP: 1202600 | > loss_text_ce: 0.02202809974551201 (0.03161424395264158) | > loss_mel_ce: 3.24932861328125 (nan) | > loss: 0.012981574982404709 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13661122864153571) | > loader_time: 0.0045 (0.004664422703122891)  --> TIME: 2025-02-26 10:31:51 -- STEP: 20634/36938 -- GLOBAL_STEP: 1202650 | > loss_text_ce: 0.02578025497496128 (0.03161630026214045) | > loss_mel_ce: 2.647092342376709 (nan) | > loss: 0.010606637224555016 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13662262391336968) | > loader_time: 0.0046 (0.004664481616336802)  --> TIME: 2025-02-26 10:32:00 -- STEP: 20684/36938 -- GLOBAL_STEP: 1202700 | > loss_text_ce: 0.026932116597890854 (0.03161654216820021) | > loss_mel_ce: 2.6427524089813232 (nan) | > loss: 0.010593986138701439 (nan) | > current_lr: 5e-06 | > step_time: 0.169 (0.13661557226203216) | > loader_time: 0.0049 (0.004664426210931274)  --> TIME: 2025-02-26 10:32:09 -- STEP: 20734/36938 -- GLOBAL_STEP: 1202750 | > loss_text_ce: 0.02571476809680462 (0.031614381627985556) | > loss_mel_ce: 2.0944671630859375 (nan) | > loss: 0.008413420990109444 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13660985288319233) | > loader_time: 0.0046 (0.004664451944640602)  --> TIME: 2025-02-26 10:32:18 -- STEP: 20784/36938 -- GLOBAL_STEP: 1202800 | > loss_text_ce: 0.03539104759693146 (0.03161642651376203) | > loss_mel_ce: 4.368923664093018 (nan) | > loss: 0.01747743971645832 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13660208978407065) | > loader_time: 0.0045 (0.004664348411137916)  --> TIME: 2025-02-26 10:32:28 -- STEP: 20834/36938 -- GLOBAL_STEP: 1202850 | > loss_text_ce: 0.036944855004549026 (0.03161413823141326) | > loss_mel_ce: 2.597154378890991 (nan) | > loss: 0.010452775284647942 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13661036451798805) | > loader_time: 0.0047 (0.00466426934918527)  --> TIME: 2025-02-26 10:32:37 -- STEP: 20884/36938 -- GLOBAL_STEP: 1202900 | > loss_text_ce: 0.03609450161457062 (0.03161230079117615) | > loss_mel_ce: 0.03564453125 (nan) | > loss: 0.00028467871015891433 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13661687625502553) | > loader_time: 0.0045 (0.004664226090672875)  --> TIME: 2025-02-26 10:32:47 -- STEP: 20934/36938 -- GLOBAL_STEP: 1202950 | > loss_text_ce: 0.03298405930399895 (0.03161059818395637) | > loss_mel_ce: 2.460965394973755 (nan) | > loss: 0.009896624833345413 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.1366093479323963) | > loader_time: 0.0046 (0.004664159497616117)  --> TIME: 2025-02-26 10:32:56 -- STEP: 20984/36938 -- GLOBAL_STEP: 1203000 | > loss_text_ce: 0.028899969533085823 (0.031607025166603644) | > loss_mel_ce: 3.0216681957244873 (nan) | > loss: 0.012105429545044899 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13661540365837266) | > loader_time: 0.0046 (0.004664099527777856)  --> TIME: 2025-02-26 10:33:05 -- STEP: 21034/36938 -- GLOBAL_STEP: 1203050 | > loss_text_ce: 0.03040681593120098 (0.03160668459969852) | > loss_mel_ce: 2.6964306831359863 (nan) | > loss: 0.01082078367471695 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13661594714821046) | > loader_time: 0.0046 (0.00466397580078508)  --> TIME: 2025-02-26 10:33:15 -- STEP: 21084/36938 -- GLOBAL_STEP: 1203100 | > loss_text_ce: 0.02827838808298111 (0.03160709953479789) | > loss_mel_ce: 2.7305688858032227 (nan) | > loss: 0.010947806760668755 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1366193824193445) | > loader_time: 0.0045 (0.004663893132073176)  --> TIME: 2025-02-26 10:33:24 -- STEP: 21134/36938 -- GLOBAL_STEP: 1203150 | > loss_text_ce: 0.03443962335586548 (0.031603990408586004) | > loss_mel_ce: 2.9897005558013916 (nan) | > loss: 0.012000557035207748 (nan) | > current_lr: 5e-06 | > step_time: 0.1889 (0.1366173173571944) | > loader_time: 0.0048 (0.00466380659020183)  --> TIME: 2025-02-26 10:33:33 -- STEP: 21184/36938 -- GLOBAL_STEP: 1203200 | > loss_text_ce: 0.03189517557621002 (0.03160462771506165) | > loss_mel_ce: 1.825508713722229 (nan) | > loss: 0.007370650768280029 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.136616469014808) | > loader_time: 0.0045 (0.004663659794272978)  --> TIME: 2025-02-26 10:33:43 -- STEP: 21234/36938 -- GLOBAL_STEP: 1203250 | > loss_text_ce: 0.036381796002388 (0.03160478296090845) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13661892146638496) | > loader_time: 0.0045 (0.00466357130131682)  --> TIME: 2025-02-26 10:33:52 -- STEP: 21284/36938 -- GLOBAL_STEP: 1203300 | > loss_text_ce: 0.035767633467912674 (0.031607701409005526) | > loss_mel_ce: 2.829521656036377 (nan) | > loss: 0.011370196007192135 (nan) | > current_lr: 5e-06 | > step_time: 0.182 (0.1366253454889987) | > loader_time: 0.005 (0.004663504193855669)  --> TIME: 2025-02-26 10:34:02 -- STEP: 21334/36938 -- GLOBAL_STEP: 1203350 | > loss_text_ce: 0.028935832902789116 (0.031606065349646044) | > loss_mel_ce: 2.899890184402466 (nan) | > loss: 0.011622326448559761 (nan) | > current_lr: 5e-06 | > step_time: 0.1819 (0.13662472357508176) | > loader_time: 0.0048 (0.004663665079853695)  --> TIME: 2025-02-26 10:34:11 -- STEP: 21384/36938 -- GLOBAL_STEP: 1203400 | > loss_text_ce: 0.03195169195532799 (0.03160961109437932) | > loss_mel_ce: 2.266361713409424 (nan) | > loss: 0.009120292030274868 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13662730174656956) | > loader_time: 0.0045 (0.004663858773152656)  --> TIME: 2025-02-26 10:34:20 -- STEP: 21434/36938 -- GLOBAL_STEP: 1203450 | > loss_text_ce: 0.0281437486410141 (0.03160781781352973) | > loss_mel_ce: 2.8721981048583984 (nan) | > loss: 0.01150929369032383 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13662279149406414) | > loader_time: 0.0045 (0.004663977981603689)  --> TIME: 2025-02-26 10:34:29 -- STEP: 21484/36938 -- GLOBAL_STEP: 1203500 | > loss_text_ce: 0.03311704471707344 (0.031608118824338484) | > loss_mel_ce: 2.733799934387207 (nan) | > loss: 0.010979830287396908 (nan) | > current_lr: 5e-06 | > step_time: 0.1558 (0.13661809520023688) | > loader_time: 0.0046 (0.004664011206668619)  --> TIME: 2025-02-26 10:34:39 -- STEP: 21534/36938 -- GLOBAL_STEP: 1203550 | > loss_text_ce: 0.03393247723579407 (0.03160984700846185) | > loss_mel_ce: 2.629209280014038 (nan) | > loss: 0.010568023659288883 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13662324690648353) | > loader_time: 0.0047 (0.004663936704538363)  --> TIME: 2025-02-26 10:34:48 -- STEP: 21584/36938 -- GLOBAL_STEP: 1203600 | > loss_text_ce: 0.029073113575577736 (0.031608429743430144) | > loss_mel_ce: 2.6724159717559814 (nan) | > loss: 0.01072019524872303 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13661708263576855) | > loader_time: 0.0046 (0.004664205550440506)  --> TIME: 2025-02-26 10:34:57 -- STEP: 21634/36938 -- GLOBAL_STEP: 1203650 | > loss_text_ce: 0.03340717405080795 (0.03160865808731842) | > loss_mel_ce: 2.987248659133911 (nan) | > loss: 0.011986730620265007 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.1366141543344733) | > loader_time: 0.0047 (0.004664360016675314)  --> TIME: 2025-02-26 10:35:06 -- STEP: 21684/36938 -- GLOBAL_STEP: 1203700 | > loss_text_ce: 0.03194848820567131 (0.03161012276234375) | > loss_mel_ce: 2.922769069671631 (nan) | > loss: 0.011725070886313915 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.136610132459125) | > loader_time: 0.0046 (0.004664360487335747)  --> TIME: 2025-02-26 10:35:16 -- STEP: 21734/36938 -- GLOBAL_STEP: 1203750 | > loss_text_ce: 0.03180095925927162 (0.03160791877536188) | > loss_mel_ce: 3.0211124420166016 (nan) | > loss: 0.012114736251533031 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.1366233460387812) | > loader_time: 0.0048 (0.004664358948349263)  --> TIME: 2025-02-26 10:35:26 -- STEP: 21784/36938 -- GLOBAL_STEP: 1203800 | > loss_text_ce: 0.024468906223773956 (0.031606920621331774) | > loss_mel_ce: 2.201051950454712 (nan) | > loss: 0.008831432089209557 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.1366323323819824) | > loader_time: 0.0045 (0.0046643495909921675)  --> TIME: 2025-02-26 10:35:35 -- STEP: 21834/36938 -- GLOBAL_STEP: 1203850 | > loss_text_ce: 0.03319573402404785 (0.031607099081234324) | > loss_mel_ce: 3.056544303894043 (nan) | > loss: 0.012260873802006245 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.1366342488931223) | > loader_time: 0.0053 (0.004664553951253997)  --> TIME: 2025-02-26 10:35:44 -- STEP: 21884/36938 -- GLOBAL_STEP: 1203900 | > loss_text_ce: 0.037105388939380646 (0.03160824322306341) | > loss_mel_ce: 3.1355340480804443 (nan) | > loss: 0.012589839287102222 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13663307493074314) | > loader_time: 0.0046 (0.004664469944060211)  --> TIME: 2025-02-26 10:35:53 -- STEP: 21934/36938 -- GLOBAL_STEP: 1203950 | > loss_text_ce: 0.03313093259930611 (0.031609158891971766) | > loss_mel_ce: 2.609131097793579 (nan) | > loss: 0.010485167615115643 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13662402702897555) | > loader_time: 0.0046 (0.004664276436880252)  --> TIME: 2025-02-26 10:36:03 -- STEP: 21984/36938 -- GLOBAL_STEP: 1204000 | > loss_text_ce: 0.03232121467590332 (0.0316099293102625) | > loss_mel_ce: 2.20000958442688 (nan) | > loss: 0.008858456276357174 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.1366255855326057) | > loader_time: 0.0045 (0.004664188475935047)  --> TIME: 2025-02-26 10:36:12 -- STEP: 22034/36938 -- GLOBAL_STEP: 1204050 | > loss_text_ce: 0.038139741867780685 (0.031609862021141105) | > loss_mel_ce: 2.684584140777588 (nan) | > loss: 0.010804460383951664 (nan) | > current_lr: 5e-06 | > step_time: 0.1556 (0.13663445833082807) | > loader_time: 0.0047 (0.004664247683261672)  --> TIME: 2025-02-26 10:36:21 -- STEP: 22084/36938 -- GLOBAL_STEP: 1204100 | > loss_text_ce: 0.035412661731243134 (0.03160914788758983) | > loss_mel_ce: 2.295849084854126 (nan) | > loss: 0.009251038543879986 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1366262796607022) | > loader_time: 0.0046 (0.004664132969035251)  --> TIME: 2025-02-26 10:36:30 -- STEP: 22134/36938 -- GLOBAL_STEP: 1204150 | > loss_text_ce: 0.02295820228755474 (0.03161119374866435) | > loss_mel_ce: 2.9752585887908936 (nan) | > loss: 0.011897686868906021 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.1366285430907866) | > loader_time: 0.0045 (0.004664055590406828)  --> TIME: 2025-02-26 10:36:39 -- STEP: 22184/36938 -- GLOBAL_STEP: 1204200 | > loss_text_ce: 0.03893916681408882 (0.03161133832298745) | > loss_mel_ce: 2.6087265014648438 (nan) | > loss: 0.01050661038607359 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13662015008573963) | > loader_time: 0.0048 (0.004663878868428378)  --> TIME: 2025-02-26 10:36:49 -- STEP: 22234/36938 -- GLOBAL_STEP: 1204250 | > loss_text_ce: 0.03597283363342285 (0.03161357107316848) | > loss_mel_ce: 0.1354573518037796 (nan) | > loss: 0.0006802785792388022 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13663240698073018) | > loader_time: 0.0045 (0.004663747571039478)  --> TIME: 2025-02-26 10:36:58 -- STEP: 22284/36938 -- GLOBAL_STEP: 1204300 | > loss_text_ce: 0.028911100700497627 (0.031613594225279144) | > loss_mel_ce: 2.5056490898132324 (nan) | > loss: 0.010057779029011726 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13662512604969082) | > loader_time: 0.0045 (0.004663589751350241)  --> TIME: 2025-02-26 10:37:07 -- STEP: 22334/36938 -- GLOBAL_STEP: 1204350 | > loss_text_ce: 0.02786833420395851 (0.031616520060242674) | > loss_mel_ce: 2.7950422763824463 (nan) | > loss: 0.011202027089893818 (nan) | > current_lr: 5e-06 | > step_time: 0.1807 (0.1366260022370665) | > loader_time: 0.0048 (0.00466353407347652)  --> TIME: 2025-02-26 10:37:17 -- STEP: 22384/36938 -- GLOBAL_STEP: 1204400 | > loss_text_ce: 0.039321888238191605 (0.03161858559283725) | > loss_mel_ce: 4.05859375 (nan) | > loss: 0.016261570155620575 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.1366188404515785) | > loader_time: 0.0046 (0.004663692596897728)  --> TIME: 2025-02-26 10:37:26 -- STEP: 22434/36938 -- GLOBAL_STEP: 1204450 | > loss_text_ce: 0.036859769374132156 (0.031617260772221586) | > loss_mel_ce: 2.82450532913208 (nan) | > loss: 0.011354624293744564 (nan) | > current_lr: 5e-06 | > step_time: 0.1436 (0.13660986030060376) | > loader_time: 0.0046 (0.004663662794833625)  --> TIME: 2025-02-26 10:37:35 -- STEP: 22484/36938 -- GLOBAL_STEP: 1204500 | > loss_text_ce: 0.03646426275372505 (0.03161422204077316) | > loss_mel_ce: 2.732429265975952 (nan) | > loss: 0.010987672954797745 (nan) | > current_lr: 5e-06 | > step_time: 0.156 (0.13661573509414182) | > loader_time: 0.0047 (0.0046635914200927175)  --> TIME: 2025-02-26 10:37:45 -- STEP: 22534/36938 -- GLOBAL_STEP: 1204550 | > loss_text_ce: 0.04895685985684395 (0.031613164176038565) | > loss_mel_ce: 1.7547500133514404 (nan) | > loss: 0.007157567422837019 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1366219680967612) | > loader_time: 0.0045 (0.004663478580121728)  --> TIME: 2025-02-26 10:37:54 -- STEP: 22584/36938 -- GLOBAL_STEP: 1204600 | > loss_text_ce: 0.03081362694501877 (0.03161416975961955) | > loss_mel_ce: 2.580432176589966 (nan) | > loss: 0.01036208774894476 (nan) | > current_lr: 5e-06 | > step_time: 0.189 (0.1366202896827091) | > loader_time: 0.0048 (0.004663404814959672)  --> TIME: 2025-02-26 10:38:03 -- STEP: 22634/36938 -- GLOBAL_STEP: 1204650 | > loss_text_ce: 0.02548159845173359 (0.031613166819964386) | > loss_mel_ce: 2.329930305480957 (nan) | > loss: 0.009346873499453068 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13662444877363328) | > loader_time: 0.0048 (0.004663378713914024)  --> TIME: 2025-02-26 10:38:13 -- STEP: 22684/36938 -- GLOBAL_STEP: 1204700 | > loss_text_ce: 0.031158218160271645 (0.03161553197653341) | > loss_mel_ce: 3.211496591567993 (nan) | > loss: 0.012867678888142109 (nan) | > current_lr: 5e-06 | > step_time: 0.19 (0.1366354290443933) | > loader_time: 0.0047 (0.004663313261269892)  --> TIME: 2025-02-26 10:38:22 -- STEP: 22734/36938 -- GLOBAL_STEP: 1204750 | > loss_text_ce: 0.03242945298552513 (0.031615679611252855) | > loss_mel_ce: 2.066394329071045 (nan) | > loss: 0.008328665979206562 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13663137610338125) | > loader_time: 0.0045 (0.004663122594854686)  --> TIME: 2025-02-26 10:38:31 -- STEP: 22784/36938 -- GLOBAL_STEP: 1204800 | > loss_text_ce: 0.03052128665149212 (0.03161785778165014) | > loss_mel_ce: 2.4929234981536865 (nan) | > loss: 0.010013670660555363 (nan) | > current_lr: 5e-06 | > step_time: 0.1511 (0.13663106066373668) | > loader_time: 0.0047 (0.004663038077983964)  --> TIME: 2025-02-26 10:38:41 -- STEP: 22834/36938 -- GLOBAL_STEP: 1204850 | > loss_text_ce: 0.041599057614803314 (0.031618358404557186) | > loss_mel_ce: 2.498490810394287 (nan) | > loss: 0.010079721920192242 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1366395480455075) | > loader_time: 0.0044 (0.004662984566272137)  --> TIME: 2025-02-26 10:38:50 -- STEP: 22884/36938 -- GLOBAL_STEP: 1204900 | > loss_text_ce: 0.03179418668150902 (0.03162009313776577) | > loss_mel_ce: 2.8365414142608643 (nan) | > loss: 0.011382284574210644 (nan) | > current_lr: 5e-06 | > step_time: 0.1697 (0.13663357086811806) | > loader_time: 0.0048 (0.004662846054069817)  --> TIME: 2025-02-26 10:38:59 -- STEP: 22934/36938 -- GLOBAL_STEP: 1204950 | > loss_text_ce: 0.03370429947972298 (0.03162427992020529) | > loss_mel_ce: 2.5179390907287598 (nan) | > loss: 0.010125569067895412 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.13663298823605427) | > loader_time: 0.0048 (0.004662716815972116)  --> TIME: 2025-02-26 10:39:09 -- STEP: 22984/36938 -- GLOBAL_STEP: 1205000 | > loss_text_ce: 0.03340097516775131 (0.031623189870335126) | > loss_mel_ce: 2.4142589569091797 (nan) | > loss: 0.009712937287986279 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.13663463410167764) | > loader_time: 0.005 (0.004662605805800357)  --> TIME: 2025-02-26 10:39:18 -- STEP: 23034/36938 -- GLOBAL_STEP: 1205050 | > loss_text_ce: 0.029465382918715477 (0.031622769233783744) | > loss_mel_ce: 2.8406100273132324 (nan) | > loss: 0.011389189399778843 (nan) | > current_lr: 5e-06 | > step_time: 0.1708 (0.13664522539092291) | > loader_time: 0.0047 (0.004662568964374004)  --> TIME: 2025-02-26 10:39:28 -- STEP: 23084/36938 -- GLOBAL_STEP: 1205100 | > loss_text_ce: 0.031328219920396805 (0.03162273403145864) | > loss_mel_ce: 2.1219890117645264 (nan) | > loss: 0.00854490976780653 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.1366448297985631) | > loader_time: 0.0048 (0.004662543974185492)  --> TIME: 2025-02-26 10:39:37 -- STEP: 23134/36938 -- GLOBAL_STEP: 1205150 | > loss_text_ce: 0.03168266639113426 (0.031620488168690765) | > loss_mel_ce: 2.3682310581207275 (nan) | > loss: 0.009523468092083931 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13664785261707452) | > loader_time: 0.0045 (0.004662448403292595)  --> TIME: 2025-02-26 10:39:47 -- STEP: 23184/36938 -- GLOBAL_STEP: 1205200 | > loss_text_ce: 0.0350600890815258 (0.03162128328589719) | > loss_mel_ce: 1.706440806388855 (nan) | > loss: 0.00691071804612875 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1366488244448806) | > loader_time: 0.0045 (0.004662356041808889)  --> TIME: 2025-02-26 10:39:56 -- STEP: 23234/36938 -- GLOBAL_STEP: 1205250 | > loss_text_ce: 0.021900512278079987 (0.03161862852377585) | > loss_mel_ce: 2.556973457336426 (nan) | > loss: 0.010233626700937748 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13666337915775068) | > loader_time: 0.0044 (0.004662362671538769)  --> TIME: 2025-02-26 10:40:05 -- STEP: 23284/36938 -- GLOBAL_STEP: 1205300 | > loss_text_ce: 0.03971753269433975 (0.03161757305182321) | > loss_mel_ce: 3.337756872177124 (nan) | > loss: 0.013402676209807396 (nan) | > current_lr: 5e-06 | > step_time: 0.1615 (0.1366608113260377) | > loader_time: 0.0048 (0.004662283761991586)  --> TIME: 2025-02-26 10:40:15 -- STEP: 23334/36938 -- GLOBAL_STEP: 1205350 | > loss_text_ce: 0.027900822460651398 (0.03161836669160397) | > loss_mel_ce: 3.0771875381469727 (nan) | > loss: 0.012321780435740948 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.136671941405225) | > loader_time: 0.0049 (0.0046622560234131985)  --> TIME: 2025-02-26 10:40:24 -- STEP: 23384/36938 -- GLOBAL_STEP: 1205400 | > loss_text_ce: 0.028462940827012062 (0.031616893624607345) | > loss_mel_ce: 3.061424732208252 (nan) | > loss: 0.012261459603905678 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13665998334799856) | > loader_time: 0.0046 (0.004662158450075427)  --> TIME: 2025-02-26 10:40:33 -- STEP: 23434/36938 -- GLOBAL_STEP: 1205450 | > loss_text_ce: 0.03326871991157532 (0.031616308924585466) | > loss_mel_ce: 3.1194496154785156 (nan) | > loss: 0.012510787695646286 (nan) | > current_lr: 5e-06 | > step_time: 0.2005 (0.1366700797985673) | > loader_time: 0.0048 (0.004662115296945548)  --> TIME: 2025-02-26 10:40:43 -- STEP: 23484/36938 -- GLOBAL_STEP: 1205500 | > loss_text_ce: 0.0280039981007576 (0.03161564077612108) | > loss_mel_ce: 4.096093654632568 (nan) | > loss: 0.016365468502044678 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13667415693374418) | > loader_time: 0.0044 (0.004662043616629978)  --> TIME: 2025-02-26 10:40:52 -- STEP: 23534/36938 -- GLOBAL_STEP: 1205550 | > loss_text_ce: 0.02765262871980667 (0.031615076025867626) | > loss_mel_ce: 2.15427565574646 (nan) | > loss: 0.008658446371555328 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.13667187993038407) | > loader_time: 0.0048 (0.004661967742814893)  --> TIME: 2025-02-26 10:41:01 -- STEP: 23584/36938 -- GLOBAL_STEP: 1205600 | > loss_text_ce: 0.02464929223060608 (0.031611249313818306) | > loss_mel_ce: 0.06256103515625 (nan) | > loss: 0.0003460727457422763 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13667159663005377) | > loader_time: 0.0045 (0.004661849509100747)  --> TIME: 2025-02-26 10:41:10 -- STEP: 23634/36938 -- GLOBAL_STEP: 1205650 | > loss_text_ce: 0.03946360945701599 (0.03161013425222869) | > loss_mel_ce: 2.281193733215332 (nan) | > loss: 0.00920895766466856 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.136662258062406) | > loader_time: 0.0046 (0.004661916475908569)  --> TIME: 2025-02-26 10:41:20 -- STEP: 23684/36938 -- GLOBAL_STEP: 1205700 | > loss_text_ce: 0.03066127561032772 (0.031609901813065965) | > loss_mel_ce: 2.624282121658325 (nan) | > loss: 0.010535490699112415 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13667146073586073) | > loader_time: 0.0046 (0.004661880530450947)  --> TIME: 2025-02-26 10:41:30 -- STEP: 23734/36938 -- GLOBAL_STEP: 1205750 | > loss_text_ce: 0.034864820539951324 (0.0316107216268332) | > loss_mel_ce: 2.9440577030181885 (nan) | > loss: 0.011821121908724308 (nan) | > current_lr: 5e-06 | > step_time: 0.1761 (0.13667584428934343) | > loader_time: 0.0057 (0.004662042189701194)  --> TIME: 2025-02-26 10:41:39 -- STEP: 23784/36938 -- GLOBAL_STEP: 1205800 | > loss_text_ce: 0.030146654695272446 (0.0316112204142559) | > loss_mel_ce: 2.5979886054992676 (nan) | > loss: 0.010429108515381813 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13668192513575303) | > loader_time: 0.0047 (0.004662039933122856)  --> TIME: 2025-02-26 10:41:48 -- STEP: 23834/36938 -- GLOBAL_STEP: 1205850 | > loss_text_ce: 0.029213720932602882 (0.03161085409870658) | > loss_mel_ce: 2.6400253772735596 (nan) | > loss: 0.010592219419777393 (nan) | > current_lr: 5e-06 | > step_time: 0.1462 (0.13667851516921817) | > loader_time: 0.0046 (0.004661957669640251)  --> TIME: 2025-02-26 10:41:58 -- STEP: 23884/36938 -- GLOBAL_STEP: 1205900 | > loss_text_ce: 0.03520147129893303 (0.03161317116708108) | > loss_mel_ce: 3.189284324645996 (nan) | > loss: 0.012795579619705677 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.1366775121107473) | > loader_time: 0.0046 (0.0046618791046583495)  --> TIME: 2025-02-26 10:42:07 -- STEP: 23934/36938 -- GLOBAL_STEP: 1205950 | > loss_text_ce: 0.028622543439269066 (0.0316139981624772) | > loss_mel_ce: 2.0778331756591797 (nan) | > loss: 0.008358951658010483 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13667661435245312) | > loader_time: 0.0046 (0.0046618263096083095)  --> TIME: 2025-02-26 10:42:16 -- STEP: 23984/36938 -- GLOBAL_STEP: 1206000 | > loss_text_ce: 0.028773335739970207 (0.03161486319858129) | > loss_mel_ce: 2.451777219772339 (nan) | > loss: 0.009843454696238041 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.1366698336927316) | > loader_time: 0.0048 (0.00466174302775516)  --> TIME: 2025-02-26 10:42:26 -- STEP: 24034/36938 -- GLOBAL_STEP: 1206050 | > loss_text_ce: 0.03401271253824234 (0.03161624319462894) | > loss_mel_ce: 2.817657947540283 (nan) | > loss: 0.011316154152154922 (nan) | > current_lr: 5e-06 | > step_time: 0.1781 (0.13667962542585727) | > loader_time: 0.0048 (0.004661703859697313)  --> TIME: 2025-02-26 10:42:35 -- STEP: 24084/36938 -- GLOBAL_STEP: 1206100 | > loss_text_ce: 0.02673383429646492 (0.03161592142856527) | > loss_mel_ce: 1.79112708568573 (nan) | > loss: 0.007213734555989504 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13667489163326393) | > loader_time: 0.0045 (0.00466161037754567)  --> TIME: 2025-02-26 10:42:44 -- STEP: 24134/36938 -- GLOBAL_STEP: 1206150 | > loss_text_ce: 0.03788955509662628 (0.03161762770610105) | > loss_mel_ce: 2.028519868850708 (nan) | > loss: 0.00820003729313612 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13667631775617461) | > loader_time: 0.0047 (0.004661528376801143)  --> TIME: 2025-02-26 10:42:53 -- STEP: 24184/36938 -- GLOBAL_STEP: 1206200 | > loss_text_ce: 0.033182185143232346 (0.031619627759139596) | > loss_mel_ce: 1.8409881591796875 (nan) | > loss: 0.007437183987349272 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1366735193613468) | > loader_time: 0.0045 (0.004661410642782278)  --> TIME: 2025-02-26 10:43:03 -- STEP: 24234/36938 -- GLOBAL_STEP: 1206250 | > loss_text_ce: 0.03252532333135605 (0.03162087094337488) | > loss_mel_ce: 3.3018486499786377 (nan) | > loss: 0.013231643475592136 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13667769443676958) | > loader_time: 0.0045 (0.004661304787203482)  --> TIME: 2025-02-26 10:43:12 -- STEP: 24284/36938 -- GLOBAL_STEP: 1206300 | > loss_text_ce: 0.02655007317662239 (0.03162140252157694) | > loss_mel_ce: 2.017683982849121 (nan) | > loss: 0.008112040348351002 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13667773150902907) | > loader_time: 0.0046 (0.004661190325219092)  --> TIME: 2025-02-26 10:43:21 -- STEP: 24334/36938 -- GLOBAL_STEP: 1206350 | > loss_text_ce: 0.03526269271969795 (0.03162306525634736) | > loss_mel_ce: 2.750195264816284 (nan) | > loss: 0.01105340477079153 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13667514699050373) | > loader_time: 0.0046 (0.004661057629698538)  --> TIME: 2025-02-26 10:43:31 -- STEP: 24384/36938 -- GLOBAL_STEP: 1206400 | > loss_text_ce: 0.024455739185214043 (0.031621887203550615) | > loss_mel_ce: 2.683319091796875 (nan) | > loss: 0.010745138861238956 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13667052661574755) | > loader_time: 0.0045 (0.004660866949267285)  --> TIME: 2025-02-26 10:43:40 -- STEP: 24434/36938 -- GLOBAL_STEP: 1206450 | > loss_text_ce: 0.032420139759778976 (0.031624161380980446) | > loss_mel_ce: 2.7555928230285645 (nan) | > loss: 0.01106354407966137 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.1366718298110439) | > loader_time: 0.005 (0.0046607612187708125)  --> TIME: 2025-02-26 10:43:49 -- STEP: 24484/36938 -- GLOBAL_STEP: 1206500 | > loss_text_ce: 0.03132041171193123 (0.03162701071347361) | > loss_mel_ce: 3.1125175952911377 (nan) | > loss: 0.012475548312067986 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.1366753155860691) | > loader_time: 0.0047 (0.004660717199643726)  --> TIME: 2025-02-26 10:43:59 -- STEP: 24534/36938 -- GLOBAL_STEP: 1206550 | > loss_text_ce: 0.02847868576645851 (0.03162623314534886) | > loss_mel_ce: 2.747203826904297 (nan) | > loss: 0.011014613322913647 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13667036375887143) | > loader_time: 0.0045 (0.004660634206580769)  --> TIME: 2025-02-26 10:44:08 -- STEP: 24584/36938 -- GLOBAL_STEP: 1206600 | > loss_text_ce: 0.03000398352742195 (0.03162625474453222) | > loss_mel_ce: 3.2915287017822266 (nan) | > loss: 0.01318068616092205 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13667152170788943) | > loader_time: 0.0046 (0.004660713373935858)  --> TIME: 2025-02-26 10:44:17 -- STEP: 24634/36938 -- GLOBAL_STEP: 1206650 | > loss_text_ce: 0.026314489543437958 (0.031628318908439576) | > loss_mel_ce: 0.10528411716222763 (nan) | > loss: 0.0005222167237661779 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13666463865130132) | > loader_time: 0.0046 (0.00466065812519332)  --> TIME: 2025-02-26 10:44:26 -- STEP: 24684/36938 -- GLOBAL_STEP: 1206700 | > loss_text_ce: 0.027727100998163223 (0.031627238347355764) | > loss_mel_ce: 3.1771838665008545 (nan) | > loss: 0.012717901729047298 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.136665496104078) | > loader_time: 0.0045 (0.00466082885612683)  --> TIME: 2025-02-26 10:44:35 -- STEP: 24734/36938 -- GLOBAL_STEP: 1206750 | > loss_text_ce: 0.0332610085606575 (0.03162523466788948) | > loss_mel_ce: 2.469313859939575 (nan) | > loss: 0.009930853731930256 (nan) | > current_lr: 5e-06 | > step_time: 0.1773 (0.13665946534160603) | > loader_time: 0.0048 (0.0046608546638658545)  --> TIME: 2025-02-26 10:44:45 -- STEP: 24784/36938 -- GLOBAL_STEP: 1206800 | > loss_text_ce: 0.0356319360435009 (0.031624310103948165) | > loss_mel_ce: 2.85542893409729 (nan) | > loss: 0.01147246453911066 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13665307212676314) | > loader_time: 0.0046 (0.004660792990297265)  --> TIME: 2025-02-26 10:44:54 -- STEP: 24834/36938 -- GLOBAL_STEP: 1206850 | > loss_text_ce: 0.021713800728321075 (0.03162375601874313) | > loss_mel_ce: 1.692169189453125 (nan) | > loss: 0.006801123730838299 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13664876594821526) | > loader_time: 0.0045 (0.004660726304003198)  --> TIME: 2025-02-26 10:45:03 -- STEP: 24884/36938 -- GLOBAL_STEP: 1206900 | > loss_text_ce: 0.030059214681386948 (0.031623323609329616) | > loss_mel_ce: 3.1976375579833984 (nan) | > loss: 0.012808321043848991 (nan) | > current_lr: 5e-06 | > step_time: 0.1264 (0.13664156497435073) | > loader_time: 0.0048 (0.004660614566622101)  --> TIME: 2025-02-26 10:45:12 -- STEP: 24934/36938 -- GLOBAL_STEP: 1206950 | > loss_text_ce: 0.03276275843381882 (0.03162320214951216) | > loss_mel_ce: 3.249401092529297 (nan) | > loss: 0.013024460524320602 (nan) | > current_lr: 5e-06 | > step_time: 0.1956 (0.13664984170461345) | > loader_time: 0.0049 (0.004660678672897638)  --> TIME: 2025-02-26 10:45:22 -- STEP: 24984/36938 -- GLOBAL_STEP: 1207000 | > loss_text_ce: 0.03420603275299072 (0.031620826433454444) | > loss_mel_ce: 0.7448921203613281 (nan) | > loss: 0.003091659629717469 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1366535810908182) | > loader_time: 0.0045 (0.0046606806276549345)  --> TIME: 2025-02-26 10:45:32 -- STEP: 25034/36938 -- GLOBAL_STEP: 1207050 | > loss_text_ce: 0.02425512671470642 (0.03162232057953049) | > loss_mel_ce: 0.07972411811351776 (nan) | > loss: 0.0004126160638406873 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13666201653053525) | > loader_time: 0.0046 (0.004660712955496505)  --> TIME: 2025-02-26 10:45:41 -- STEP: 25084/36938 -- GLOBAL_STEP: 1207100 | > loss_text_ce: 0.03394629806280136 (0.031620147063446256) | > loss_mel_ce: 3.199714183807373 (nan) | > loss: 0.012831986881792545 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.13666280310911738) | > loader_time: 0.0047 (0.004660668868878654)  --> TIME: 2025-02-26 10:45:50 -- STEP: 25134/36938 -- GLOBAL_STEP: 1207150 | > loss_text_ce: 0.031723055988550186 (0.03161996412691985) | > loss_mel_ce: 3.0570061206817627 (nan) | > loss: 0.012256862595677376 (nan) | > current_lr: 5e-06 | > step_time: 0.1815 (0.1366690254689519) | > loader_time: 0.0047 (0.0046608016515030555)  --> TIME: 2025-02-26 10:46:00 -- STEP: 25184/36938 -- GLOBAL_STEP: 1207200 | > loss_text_ce: 0.039294954389333725 (0.03162041894139734) | > loss_mel_ce: 2.9031054973602295 (nan) | > loss: 0.011676193214952946 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13667184538580754) | > loader_time: 0.0045 (0.00466092262213537)  --> TIME: 2025-02-26 10:46:09 -- STEP: 25234/36938 -- GLOBAL_STEP: 1207250 | > loss_text_ce: 0.029881872236728668 (0.03161835353390077) | > loss_mel_ce: 3.0229127407073975 (nan) | > loss: 0.012114265002310276 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13667106538659102) | > loader_time: 0.0046 (0.004660936706538745)  --> TIME: 2025-02-26 10:46:18 -- STEP: 25284/36938 -- GLOBAL_STEP: 1207300 | > loss_text_ce: 0.035288017243146896 (0.03161961069834698) | > loss_mel_ce: 3.041234254837036 (nan) | > loss: 0.012208422645926476 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13667176291641278) | > loader_time: 0.0046 (0.004660855241448728)  --> TIME: 2025-02-26 10:46:28 -- STEP: 25334/36938 -- GLOBAL_STEP: 1207350 | > loss_text_ce: 0.032425522804260254 (0.03161944642342372) | > loss_mel_ce: 2.6439754962921143 (nan) | > loss: 0.01062063965946436 (nan) | > current_lr: 5e-06 | > step_time: 0.1835 (0.13667367019828033) | > loader_time: 0.005 (0.004660859107594757)  --> TIME: 2025-02-26 10:46:37 -- STEP: 25384/36938 -- GLOBAL_STEP: 1207400 | > loss_text_ce: 0.03582782298326492 (0.03161888037427191) | > loss_mel_ce: 1.9916489124298096 (nan) | > loss: 0.008045542985200882 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13666874795842218) | > loader_time: 0.0045 (0.004660900396914501)  --> TIME: 2025-02-26 10:46:46 -- STEP: 25434/36938 -- GLOBAL_STEP: 1207450 | > loss_text_ce: 0.020311716943979263 (0.03161572403512641) | > loss_mel_ce: 2.627553701400757 (nan) | > loss: 0.010507403872907162 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1366639366087664) | > loader_time: 0.0052 (0.004660850764727027)  --> TIME: 2025-02-26 10:46:55 -- STEP: 25484/36938 -- GLOBAL_STEP: 1207500 | > loss_text_ce: 0.024449657648801804 (0.031615076418842585) | > loss_mel_ce: 2.869014024734497 (nan) | > loss: 0.011481999419629574 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13666379687513358) | > loader_time: 0.0046 (0.004660837421273537)  --> TIME: 2025-02-26 10:47:05 -- STEP: 25534/36938 -- GLOBAL_STEP: 1207550 | > loss_text_ce: 0.03246065229177475 (0.0316150295399511) | > loss_mel_ce: 2.745851993560791 (nan) | > loss: 0.011025050655007362 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13666024090045456) | > loader_time: 0.0045 (0.004660809937383965)  --> TIME: 2025-02-26 10:47:14 -- STEP: 25584/36938 -- GLOBAL_STEP: 1207600 | > loss_text_ce: 0.02924780361354351 (0.031613700119122616) | > loss_mel_ce: 1.4440029859542847 (nan) | > loss: 0.005846233572810888 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.1366536007813177) | > loader_time: 0.0046 (0.004660696052177914)  --> TIME: 2025-02-26 10:47:23 -- STEP: 25634/36938 -- GLOBAL_STEP: 1207650 | > loss_text_ce: 0.029813570901751518 (0.031613977849968264) | > loss_mel_ce: 3.11067795753479 (nan) | > loss: 0.012462268583476543 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13666308881377676) | > loader_time: 0.0047 (0.004660792141313384)  --> TIME: 2025-02-26 10:47:32 -- STEP: 25684/36938 -- GLOBAL_STEP: 1207700 | > loss_text_ce: 0.03326554596424103 (0.03161061696816345) | > loss_mel_ce: 3.1152398586273193 (nan) | > loss: 0.012494069524109364 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1366617439079467) | > loader_time: 0.0053 (0.004660886046188752)  --> TIME: 2025-02-26 10:47:42 -- STEP: 25734/36938 -- GLOBAL_STEP: 1207750 | > loss_text_ce: 0.03913811966776848 (0.03160969873599975) | > loss_mel_ce: 2.9258859157562256 (nan) | > loss: 0.01176596898585558 (nan) | > current_lr: 5e-06 | > step_time: 0.1565 (0.13666131540529597) | > loader_time: 0.0046 (0.004660972767316483)  --> TIME: 2025-02-26 10:47:51 -- STEP: 25784/36938 -- GLOBAL_STEP: 1207800 | > loss_text_ce: 0.020701104775071144 (0.031611021688355355) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13665970735508537) | > loader_time: 0.005 (0.004661177279525318)  --> TIME: 2025-02-26 10:48:00 -- STEP: 25834/36938 -- GLOBAL_STEP: 1207850 | > loss_text_ce: 0.02687278389930725 (0.031611226868001226) | > loss_mel_ce: 2.2697300910949707 (nan) | > loss: 0.00911350455135107 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13665817370307354) | > loader_time: 0.0054 (0.004661289800419626)  --> TIME: 2025-02-26 10:48:10 -- STEP: 25884/36938 -- GLOBAL_STEP: 1207900 | > loss_text_ce: 0.03459806740283966 (0.031611353109279705) | > loss_mel_ce: 4.515625 (nan) | > loss: 0.018056441098451614 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.1366607092149369) | > loader_time: 0.0046 (0.004661273558381874)  --> TIME: 2025-02-26 10:48:19 -- STEP: 25934/36938 -- GLOBAL_STEP: 1207950 | > loss_text_ce: 0.008571954444050789 (0.031611969596655166) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.1366665501946468) | > loader_time: 0.0045 (0.0046612887280645795)  --> TIME: 2025-02-26 10:48:29 -- STEP: 25984/36938 -- GLOBAL_STEP: 1208000 | > loss_text_ce: 0.029517868533730507 (0.03161479359104652) | > loss_mel_ce: 2.4925732612609863 (nan) | > loss: 0.01000829879194498 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13666374018883823) | > loader_time: 0.0045 (0.004661269596057581)  --> TIME: 2025-02-26 10:48:38 -- STEP: 26034/36938 -- GLOBAL_STEP: 1208050 | > loss_text_ce: 0.025699853897094727 (0.031614263399958716) | > loss_mel_ce: 3.024655818939209 (nan) | > loss: 0.012104586698114872 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13666652695341655) | > loader_time: 0.0046 (0.004661217660428456)  --> TIME: 2025-02-26 10:48:48 -- STEP: 26084/36938 -- GLOBAL_STEP: 1208100 | > loss_text_ce: 0.03278250992298126 (0.03161790209081219) | > loss_mel_ce: 2.987844705581665 (nan) | > loss: 0.011986616998910904 (nan) | > current_lr: 5e-06 | > step_time: 0.1578 (0.13667410478656317) | > loader_time: 0.0046 (0.004661325561252592)  --> TIME: 2025-02-26 10:48:57 -- STEP: 26134/36938 -- GLOBAL_STEP: 1208150 | > loss_text_ce: 0.03643536567687988 (0.031619223686542026) | > loss_mel_ce: 2.280468702316284 (nan) | > loss: 0.00919406395405531 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.136681027179598) | > loader_time: 0.0046 (0.004661779829909692)  --> TIME: 2025-02-26 10:49:07 -- STEP: 26184/36938 -- GLOBAL_STEP: 1208200 | > loss_text_ce: 0.024169858545064926 (0.03161960713535991) | > loss_mel_ce: 2.432111978530884 (nan) | > loss: 0.009747151285409927 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13669082552440498) | > loader_time: 0.0044 (0.004662137648180079)  --> TIME: 2025-02-26 10:49:16 -- STEP: 26234/36938 -- GLOBAL_STEP: 1208250 | > loss_text_ce: 0.02984878234565258 (0.031616744351652806) | > loss_mel_ce: 3.2875053882598877 (nan) | > loss: 0.013164104893803596 (nan) | > current_lr: 5e-06 | > step_time: 0.1694 (0.13669707790202348) | > loader_time: 0.005 (0.004662165602160339)  --> TIME: 2025-02-26 10:49:26 -- STEP: 26284/36938 -- GLOBAL_STEP: 1208300 | > loss_text_ce: 0.02543206699192524 (0.031615306562884596) | > loss_mel_ce: 2.270998954772949 (nan) | > loss: 0.009112821891903877 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.1366975100286079) | > loader_time: 0.0045 (0.004662217324300017)  --> TIME: 2025-02-26 10:49:35 -- STEP: 26334/36938 -- GLOBAL_STEP: 1208350 | > loss_text_ce: 0.028751756995916367 (0.031615346971166816) | > loss_mel_ce: 2.3430988788604736 (nan) | > loss: 0.009412107057869434 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.13669524890302584) | > loader_time: 0.0047 (0.004662258827651021)  --> TIME: 2025-02-26 10:49:44 -- STEP: 26384/36938 -- GLOBAL_STEP: 1208400 | > loss_text_ce: 0.03552012890577316 (0.03161461565921652) | > loss_mel_ce: 3.1003265380859375 (nan) | > loss: 0.012443836778402328 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.1366944521882452) | > loader_time: 0.0046 (0.004662253174946646)  --> TIME: 2025-02-26 10:49:53 -- STEP: 26434/36938 -- GLOBAL_STEP: 1208450 | > loss_text_ce: 0.03272407129406929 (0.03161776594797858) | > loss_mel_ce: 2.6116349697113037 (nan) | > loss: 0.010493488982319832 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.136696021630994) | > loader_time: 0.0046 (0.004662242276302205)  --> TIME: 2025-02-26 10:50:03 -- STEP: 26484/36938 -- GLOBAL_STEP: 1208500 | > loss_text_ce: 0.022278565913438797 (0.03161521179202018) | > loss_mel_ce: 2.507765293121338 (nan) | > loss: 0.010039856657385826 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.1367024939561826) | > loader_time: 0.0044 (0.004662149119207916)  --> TIME: 2025-02-26 10:50:13 -- STEP: 26534/36938 -- GLOBAL_STEP: 1208550 | > loss_text_ce: 0.029071716591715813 (0.0316176831885929) | > loss_mel_ce: 2.210726022720337 (nan) | > loss: 0.00888808723539114 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13670932099177108) | > loader_time: 0.0045 (0.004662263067232136)  --> TIME: 2025-02-26 10:50:22 -- STEP: 26584/36938 -- GLOBAL_STEP: 1208600 | > loss_text_ce: 0.028666403144598007 (0.03161585586024116) | > loss_mel_ce: 0.740814208984375 (nan) | > loss: 0.003053494496271014 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1367110302029984) | > loader_time: 0.0046 (0.004662446729253677)  --> TIME: 2025-02-26 10:50:31 -- STEP: 26634/36938 -- GLOBAL_STEP: 1208650 | > loss_text_ce: 0.024410046637058258 (0.03161769474386896) | > loss_mel_ce: 2.441772699356079 (nan) | > loss: 0.009786440059542656 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.1367095749129529) | > loader_time: 0.0053 (0.004662453291284958)  --> TIME: 2025-02-26 10:50:40 -- STEP: 26684/36938 -- GLOBAL_STEP: 1208700 | > loss_text_ce: 0.022343197837471962 (0.0316154679302593) | > loss_mel_ce: 2.4512765407562256 (nan) | > loss: 0.009815951809287071 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.1367012973650878) | > loader_time: 0.0049 (0.0046624691656836515)  --> TIME: 2025-02-26 10:50:50 -- STEP: 26734/36938 -- GLOBAL_STEP: 1208750 | > loss_text_ce: 0.029484977945685387 (0.031615149788248656) | > loss_mel_ce: 2.744150161743164 (nan) | > loss: 0.011006489396095276 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13670326482298104) | > loader_time: 0.0048 (0.004662450422762309)  --> TIME: 2025-02-26 10:50:59 -- STEP: 26784/36938 -- GLOBAL_STEP: 1208800 | > loss_text_ce: 0.03847120329737663 (0.03161432366162629) | > loss_mel_ce: 2.68168306350708 (nan) | > loss: 0.010794263333082199 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13670566325546593) | > loader_time: 0.0046 (0.004662412024028577)  --> TIME: 2025-02-26 10:51:09 -- STEP: 26834/36938 -- GLOBAL_STEP: 1208850 | > loss_text_ce: 0.032516539096832275 (0.031615163956646104) | > loss_mel_ce: 3.25722074508667 (nan) | > loss: 0.01305451337248087 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13670794785427) | > loader_time: 0.0046 (0.0046623507563879904)  --> TIME: 2025-02-26 10:51:18 -- STEP: 26884/36938 -- GLOBAL_STEP: 1208900 | > loss_text_ce: 0.03653591126203537 (0.03161595570767221) | > loss_mel_ce: 3.287794589996338 (nan) | > loss: 0.013191788457334042 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13671287306716196) | > loader_time: 0.0047 (0.004662509254358535)  --> TIME: 2025-02-26 10:51:27 -- STEP: 26934/36938 -- GLOBAL_STEP: 1208950 | > loss_text_ce: 0.021031316369771957 (0.03161582596747864) | > loss_mel_ce: 1.891993761062622 (nan) | > loss: 0.007591369561851025 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.1367167691535777) | > loader_time: 0.0045 (0.0046626216825118765)  --> TIME: 2025-02-26 10:51:36 -- STEP: 26984/36938 -- GLOBAL_STEP: 1209000 | > loss_text_ce: 0.030515452846884727 (0.03161286433304211) | > loss_mel_ce: 2.6045725345611572 (nan) | > loss: 0.010456698946654797 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13670577021829827) | > loader_time: 0.0045 (0.004662663115615366)  --> TIME: 2025-02-26 10:51:45 -- STEP: 27034/36938 -- GLOBAL_STEP: 1209050 | > loss_text_ce: 0.034317534416913986 (0.031611145132427465) | > loss_mel_ce: 0.18024902045726776 (nan) | > loss: 0.0008514546207152307 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13669437565570364) | > loader_time: 0.0046 (0.004662667698714491)  --> TIME: 2025-02-26 10:51:55 -- STEP: 27084/36938 -- GLOBAL_STEP: 1209100 | > loss_text_ce: 0.029719194397330284 (0.03161366964499431) | > loss_mel_ce: 2.71401047706604 (nan) | > loss: 0.010887816548347473 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13669723855077653) | > loader_time: 0.0046 (0.004662802028895463)  --> TIME: 2025-02-26 10:52:04 -- STEP: 27134/36938 -- GLOBAL_STEP: 1209150 | > loss_text_ce: 0.027827097102999687 (0.031612838069442664) | > loss_mel_ce: 2.7578213214874268 (nan) | > loss: 0.011054161004722118 (nan) | > current_lr: 5e-06 | > step_time: 0.1605 (0.136696993597327) | > loader_time: 0.0049 (0.004662732425326154)  --> TIME: 2025-02-26 10:52:13 -- STEP: 27184/36938 -- GLOBAL_STEP: 1209200 | > loss_text_ce: 0.04131012782454491 (0.031616546315396316) | > loss_mel_ce: 2.5298819541931152 (nan) | > loss: 0.010203143581748009 (nan) | > current_lr: 5e-06 | > step_time: 0.1355 (0.13669653320186223) | > loader_time: 0.0046 (0.004662825175072617)  --> TIME: 2025-02-26 10:52:23 -- STEP: 27234/36938 -- GLOBAL_STEP: 1209250 | > loss_text_ce: 0.03566787764430046 (0.03161637908388581) | > loss_mel_ce: 2.0517938137054443 (nan) | > loss: 0.008283578790724277 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13669389772145538) | > loader_time: 0.0045 (0.0046628028747487)  --> TIME: 2025-02-26 10:52:32 -- STEP: 27284/36938 -- GLOBAL_STEP: 1209300 | > loss_text_ce: 0.03306733816862106 (0.03161651231970128) | > loss_mel_ce: 3.1509580612182617 (nan) | > loss: 0.012635021470487118 (nan) | > current_lr: 5e-06 | > step_time: 0.168 (0.13669748361151604) | > loader_time: 0.0048 (0.004662756485737003)  --> TIME: 2025-02-26 10:52:42 -- STEP: 27334/36938 -- GLOBAL_STEP: 1209350 | > loss_text_ce: 0.025865932926535606 (0.03161766428217029) | > loss_mel_ce: 2.2941863536834717 (nan) | > loss: 0.009206557646393776 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.1367071332479349) | > loader_time: 0.0047 (0.004662875294589427)  --> TIME: 2025-02-26 10:52:51 -- STEP: 27384/36938 -- GLOBAL_STEP: 1209400 | > loss_text_ce: 0.03366241231560707 (0.03161537072392699) | > loss_mel_ce: 2.796097993850708 (nan) | > loss: 0.011229208670556545 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13670599451024676) | > loader_time: 0.0046 (0.004662845389312945)  --> TIME: 2025-02-26 10:53:00 -- STEP: 27434/36938 -- GLOBAL_STEP: 1209450 | > loss_text_ce: 0.033327143639326096 (0.03161794669049184) | > loss_mel_ce: 2.6645870208740234 (nan) | > loss: 0.010706009343266487 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.13670732378481681) | > loader_time: 0.0048 (0.004662888264042518)  --> TIME: 2025-02-26 10:53:10 -- STEP: 27484/36938 -- GLOBAL_STEP: 1209500 | > loss_text_ce: 0.04090014472603798 (0.031617537803707296) | > loss_mel_ce: 1.815208077430725 (nan) | > loss: 0.007365508936345577 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13670411805297386) | > loader_time: 0.0048 (0.004662844798499339)  --> TIME: 2025-02-26 10:53:19 -- STEP: 27534/36938 -- GLOBAL_STEP: 1209550 | > loss_text_ce: 0.03205900266766548 (0.03161785667573106) | > loss_mel_ce: 2.165926933288574 (nan) | > loss: 0.008722166530787945 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1367008081749824) | > loader_time: 0.0045 (0.004662935186715716)  --> TIME: 2025-02-26 10:53:28 -- STEP: 27584/36938 -- GLOBAL_STEP: 1209600 | > loss_text_ce: 0.038427576422691345 (0.03161839358086907) | > loss_mel_ce: 3.1220507621765137 (nan) | > loss: 0.012541581876575947 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.1367024014554832) | > loader_time: 0.0048 (0.004663068559150417)  --> TIME: 2025-02-26 10:53:37 -- STEP: 27634/36938 -- GLOBAL_STEP: 1209650 | > loss_text_ce: 0.03662273660302162 (0.03161857142789907) | > loss_mel_ce: 2.860328197479248 (nan) | > loss: 0.011495837941765785 (nan) | > current_lr: 5e-06 | > step_time: 0.152 (0.13669625056448237) | > loader_time: 0.0047 (0.0046631327895030475)  --> TIME: 2025-02-26 10:53:47 -- STEP: 27684/36938 -- GLOBAL_STEP: 1209700 | > loss_text_ce: 0.035833392292261124 (0.031618926480539586) | > loss_mel_ce: 2.9216437339782715 (nan) | > loss: 0.01173602044582367 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.13670118953809277) | > loader_time: 0.0049 (0.004663114516036813)  --> TIME: 2025-02-26 10:53:56 -- STEP: 27734/36938 -- GLOBAL_STEP: 1209750 | > loss_text_ce: 0.0311939287930727 (0.03161619626822939) | > loss_mel_ce: 0.1316048800945282 (nan) | > loss: 0.0006460270378738642 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13669925623365775) | > loader_time: 0.0046 (0.0046630316361083324)  --> TIME: 2025-02-26 10:54:05 -- STEP: 27784/36938 -- GLOBAL_STEP: 1209800 | > loss_text_ce: 0.028877947479486465 (0.03161672812818777) | > loss_mel_ce: 2.766444206237793 (nan) | > loss: 0.011092549189925194 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13669733756768615) | > loader_time: 0.0046 (0.004662964912439823)  --> TIME: 2025-02-26 10:54:15 -- STEP: 27834/36938 -- GLOBAL_STEP: 1209850 | > loss_text_ce: 0.03784453123807907 (0.03161703879247558) | > loss_mel_ce: 2.976191759109497 (nan) | > loss: 0.011960461735725403 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13670175489427172) | > loader_time: 0.0047 (0.004663181979911316)  --> TIME: 2025-02-26 10:54:24 -- STEP: 27884/36938 -- GLOBAL_STEP: 1209900 | > loss_text_ce: 0.030160831287503242 (0.03161784974621243) | > loss_mel_ce: 0.01873779296875 (nan) | > loss: 0.0001940421643666923 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13670977780371246) | > loader_time: 0.0046 (0.004663346684521137)  --> TIME: 2025-02-26 10:54:34 -- STEP: 27934/36938 -- GLOBAL_STEP: 1209950 | > loss_text_ce: 0.025921888649463654 (0.031617112360249236) | > loss_mel_ce: 2.498600721359253 (nan) | > loss: 0.010017947293817997 (nan) | > current_lr: 5e-06 | > step_time: 0.1867 (0.13670964382539255) | > loader_time: 0.0049 (0.004663403547850584)  --> TIME: 2025-02-26 10:54:43 -- STEP: 27984/36938 -- GLOBAL_STEP: 1210000 | > loss_text_ce: 0.025024794042110443 (0.03161681811320939) | > loss_mel_ce: 2.4089226722717285 (nan) | > loss: 0.00965852290391922 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13670646658688024) | > loader_time: 0.0045 (0.004663403840866556) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1210000.pth  --> TIME: 2025-02-26 10:54:58 -- STEP: 28034/36938 -- GLOBAL_STEP: 1210050 | > loss_text_ce: 0.030406486243009567 (0.03161611761310797) | > loss_mel_ce: 3.124929189682007 (nan) | > loss: 0.012521173804998398 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.1367052963935437) | > loader_time: 0.0056 (0.004663633238855091)  --> TIME: 2025-02-26 10:55:08 -- STEP: 28084/36938 -- GLOBAL_STEP: 1210100 | > loss_text_ce: 0.029227372258901596 (0.031616609771998726) | > loss_mel_ce: 2.8237531185150146 (nan) | > loss: 0.011321350932121277 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.1367099271967036) | > loader_time: 0.0053 (0.00466384487501188)  --> TIME: 2025-02-26 10:55:17 -- STEP: 28134/36938 -- GLOBAL_STEP: 1210150 | > loss_text_ce: 0.03237750753760338 (0.03161791140115173) | > loss_mel_ce: 3.0698883533477783 (nan) | > loss: 0.012310579419136047 (nan) | > current_lr: 5e-06 | > step_time: 0.1588 (0.1367108309337593) | > loader_time: 0.0047 (0.004664001319423171)  --> TIME: 2025-02-26 10:55:26 -- STEP: 28184/36938 -- GLOBAL_STEP: 1210200 | > loss_text_ce: 0.037440225481987 (0.03161613652502798) | > loss_mel_ce: 2.847668409347534 (nan) | > loss: 0.011448844335973263 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13670889601394795) | > loader_time: 0.0046 (0.004663961078509541)  --> TIME: 2025-02-26 10:55:35 -- STEP: 28234/36938 -- GLOBAL_STEP: 1210250 | > loss_text_ce: 0.02828861214220524 (0.03161696615960618) | > loss_mel_ce: 2.0519628524780273 (nan) | > loss: 0.00825496669858694 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13670419298944697) | > loader_time: 0.0046 (0.004663899176737376)  --> TIME: 2025-02-26 10:55:45 -- STEP: 28284/36938 -- GLOBAL_STEP: 1210300 | > loss_text_ce: 0.03402012214064598 (0.031618328202747674) | > loss_mel_ce: 2.637540578842163 (nan) | > loss: 0.01060143206268549 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13670265620642577) | > loader_time: 0.0047 (0.004663871127330782)  --> TIME: 2025-02-26 10:55:54 -- STEP: 28334/36938 -- GLOBAL_STEP: 1210350 | > loss_text_ce: 0.03493107855319977 (0.03162058515615491) | > loss_mel_ce: 3.1951904296875 (nan) | > loss: 0.012817943468689919 (nan) | > current_lr: 5e-06 | > step_time: 0.1842 (0.13670802856876665) | > loader_time: 0.0048 (0.00466390174236448)  --> TIME: 2025-02-26 10:56:04 -- STEP: 28384/36938 -- GLOBAL_STEP: 1210400 | > loss_text_ce: 0.028325194492936134 (0.03162045487326553) | > loss_mel_ce: 2.803906202316284 (nan) | > loss: 0.011239013634622097 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13670800131532476) | > loader_time: 0.0047 (0.004664054348342457)  --> TIME: 2025-02-26 10:56:13 -- STEP: 28434/36938 -- GLOBAL_STEP: 1210450 | > loss_text_ce: 0.03387134149670601 (0.03161808285776605) | > loss_mel_ce: 2.9576926231384277 (nan) | > loss: 0.011871286667883396 (nan) | > current_lr: 5e-06 | > step_time: 0.1763 (0.13669981953631022) | > loader_time: 0.0048 (0.004664030240760496)  --> TIME: 2025-02-26 10:56:22 -- STEP: 28484/36938 -- GLOBAL_STEP: 1210500 | > loss_text_ce: 0.02661074511706829 (0.03161723577365333) | > loss_mel_ce: 1.2510018348693848 (nan) | > loss: 0.005069891456514597 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13670518262351836) | > loader_time: 0.0054 (0.004664270810206417)  --> TIME: 2025-02-26 10:56:31 -- STEP: 28534/36938 -- GLOBAL_STEP: 1210550 | > loss_text_ce: 0.032419174909591675 (0.03161882329877444) | > loss_mel_ce: 2.9423255920410156 (nan) | > loss: 0.011804543435573578 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13670036228251042) | > loader_time: 0.0045 (0.0046641943858652035)  --> TIME: 2025-02-26 10:56:41 -- STEP: 28584/36938 -- GLOBAL_STEP: 1210600 | > loss_text_ce: 0.03655583783984184 (0.03161856495095186) | > loss_mel_ce: 0.006927490234375 (nan) | > loss: 0.0001725529000395909 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13670749998613113) | > loader_time: 0.0046 (0.004664192171681346)  --> TIME: 2025-02-26 10:56:50 -- STEP: 28634/36938 -- GLOBAL_STEP: 1210650 | > loss_text_ce: 0.03197912871837616 (0.03161709960214104) | > loss_mel_ce: 2.8234262466430664 (nan) | > loss: 0.011330974288284779 (nan) | > current_lr: 5e-06 | > step_time: 0.1621 (0.13671866234782473) | > loader_time: 0.0048 (0.0046642760770184595)  --> TIME: 2025-02-26 10:57:00 -- STEP: 28684/36938 -- GLOBAL_STEP: 1210700 | > loss_text_ce: 0.03379922732710838 (0.03161797480555568) | > loss_mel_ce: 3.031686544418335 (nan) | > loss: 0.012164626270532608 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.13672663348935618) | > loader_time: 0.0052 (0.00466435494374372)  --> TIME: 2025-02-26 10:57:09 -- STEP: 28734/36938 -- GLOBAL_STEP: 1210750 | > loss_text_ce: 0.0350523442029953 (0.03161828696135994) | > loss_mel_ce: 2.278993844985962 (nan) | > loss: 0.009182723239064217 (nan) | > current_lr: 5e-06 | > step_time: 0.1664 (0.13671710549760321) | > loader_time: 0.0047 (0.004664632201551562)  --> TIME: 2025-02-26 10:57:18 -- STEP: 28784/36938 -- GLOBAL_STEP: 1210800 | > loss_text_ce: 0.03416172415018082 (0.0316177694818048) | > loss_mel_ce: 2.8879010677337646 (nan) | > loss: 0.011595488525927067 (nan) | > current_lr: 5e-06 | > step_time: 0.1495 (0.13671195239507858) | > loader_time: 0.0046 (0.004664612974373093)  --> TIME: 2025-02-26 10:57:27 -- STEP: 28834/36938 -- GLOBAL_STEP: 1210850 | > loss_text_ce: 0.02654728665947914 (0.03161749980937502) | > loss_mel_ce: 0.7843889594078064 (nan) | > loss: 0.0032180012203752995 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1367054191650742) | > loader_time: 0.0046 (0.004664924279193314)  --> TIME: 2025-02-26 10:57:37 -- STEP: 28884/36938 -- GLOBAL_STEP: 1210900 | > loss_text_ce: 0.038712646812200546 (0.03161641800798516) | > loss_mel_ce: 2.347383975982666 (nan) | > loss: 0.009468638338148594 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13670406434331614) | > loader_time: 0.0046 (0.004665146935862735)  --> TIME: 2025-02-26 10:57:46 -- STEP: 28934/36938 -- GLOBAL_STEP: 1210950 | > loss_text_ce: 0.02827254682779312 (0.03161432919394541) | > loss_mel_ce: 3.0740134716033936 (nan) | > loss: 0.012310659512877464 (nan) | > current_lr: 5e-06 | > step_time: 0.1609 (0.13670040115486445) | > loader_time: 0.005 (0.004665112712960579)  --> TIME: 2025-02-26 10:57:55 -- STEP: 28984/36938 -- GLOBAL_STEP: 1211000 | > loss_text_ce: 0.025958353653550148 (0.03161385491012698) | > loss_mel_ce: 2.3296279907226562 (nan) | > loss: 0.009347565472126007 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13670560558823436) | > loader_time: 0.0046 (0.004665104577201138)  --> TIME: 2025-02-26 10:58:05 -- STEP: 29034/36938 -- GLOBAL_STEP: 1211050 | > loss_text_ce: 0.03250009939074516 (0.03161278169645729) | > loss_mel_ce: 1.9749592542648315 (nan) | > loss: 0.007966109551489353 (nan) | > current_lr: 5e-06 | > step_time: 0.1349 (0.1367088620997891) | > loader_time: 0.0053 (0.004665160290817351)  --> TIME: 2025-02-26 10:58:14 -- STEP: 29084/36938 -- GLOBAL_STEP: 1211100 | > loss_text_ce: 0.03810879960656166 (0.031612174680486486) | > loss_mel_ce: 3.1329762935638428 (nan) | > loss: 0.012583672069013119 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13671641441769403) | > loader_time: 0.0046 (0.004665223633369158)  --> TIME: 2025-02-26 10:58:24 -- STEP: 29134/36938 -- GLOBAL_STEP: 1211150 | > loss_text_ce: 0.023130981251597404 (0.03161287319149837) | > loss_mel_ce: 2.888036012649536 (nan) | > loss: 0.011552250012755394 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.1367104509073698) | > loader_time: 0.0046 (0.004665242379291266)  --> TIME: 2025-02-26 10:58:33 -- STEP: 29184/36938 -- GLOBAL_STEP: 1211200 | > loss_text_ce: 0.03936576098203659 (0.03161088095378539) | > loss_mel_ce: 1.8640576601028442 (nan) | > loss: 0.007553268224000931 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.1367135051588874) | > loader_time: 0.0046 (0.004665298599815168)  --> TIME: 2025-02-26 10:58:43 -- STEP: 29234/36938 -- GLOBAL_STEP: 1211250 | > loss_text_ce: 0.029096895828843117 (0.031610010158109) | > loss_mel_ce: 3.0222275257110596 (nan) | > loss: 0.01210843026638031 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13671634279116593) | > loader_time: 0.0047 (0.004665366445380759)  --> TIME: 2025-02-26 10:58:52 -- STEP: 29284/36938 -- GLOBAL_STEP: 1211300 | > loss_text_ce: 0.03069496713578701 (0.03160588127723947) | > loss_mel_ce: 2.8803088665008545 (nan) | > loss: 0.011551602743566036 (nan) | > current_lr: 5e-06 | > step_time: 0.181 (0.13672202084660806) | > loader_time: 0.0049 (0.004665381700643155)  --> TIME: 2025-02-26 10:59:01 -- STEP: 29334/36938 -- GLOBAL_STEP: 1211350 | > loss_text_ce: 0.03153020516037941 (0.031605137612112866) | > loss_mel_ce: 2.7659311294555664 (nan) | > loss: 0.01110103726387024 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13672194707171786) | > loader_time: 0.0046 (0.004665387906512585)  --> TIME: 2025-02-26 10:59:10 -- STEP: 29384/36938 -- GLOBAL_STEP: 1211400 | > loss_text_ce: 0.03841756656765938 (0.03160306287750601) | > loss_mel_ce: 3.140471935272217 (nan) | > loss: 0.012614641338586807 (nan) | > current_lr: 5e-06 | > step_time: 0.144 (0.1367152121243608) | > loader_time: 0.0049 (0.004665450953411669)  --> TIME: 2025-02-26 10:59:20 -- STEP: 29434/36938 -- GLOBAL_STEP: 1211450 | > loss_text_ce: 0.03369738534092903 (0.031602851113822474) | > loss_mel_ce: 1.82277512550354 (nan) | > loss: 0.007366954814642668 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13671287450565575) | > loader_time: 0.0047 (0.004665480761972882)  --> TIME: 2025-02-26 10:59:29 -- STEP: 29484/36938 -- GLOBAL_STEP: 1211500 | > loss_text_ce: 0.02438177727162838 (0.03160396716185037) | > loss_mel_ce: 2.826371192932129 (nan) | > loss: 0.011312512680888176 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13671637867154016) | > loader_time: 0.0045 (0.004665478948756946)  --> TIME: 2025-02-26 10:59:38 -- STEP: 29534/36938 -- GLOBAL_STEP: 1211550 | > loss_text_ce: 0.029351670295000076 (0.03160240580774338) | > loss_mel_ce: 2.733201503753662 (nan) | > loss: 0.010962513275444508 (nan) | > current_lr: 5e-06 | > step_time: 0.1632 (0.13671559397704652) | > loader_time: 0.0047 (0.00466567187897926)  --> TIME: 2025-02-26 10:59:48 -- STEP: 29584/36938 -- GLOBAL_STEP: 1211600 | > loss_text_ce: 0.03578229621052742 (0.031601500364290394) | > loss_mel_ce: 2.9334843158721924 (nan) | > loss: 0.011782804504036903 (nan) | > current_lr: 5e-06 | > step_time: 0.1634 (0.1367110139682787) | > loader_time: 0.0047 (0.004665777264510453)  --> TIME: 2025-02-26 10:59:57 -- STEP: 29634/36938 -- GLOBAL_STEP: 1211650 | > loss_text_ce: 0.02814675122499466 (0.03160342878876769) | > loss_mel_ce: 2.2142627239227295 (nan) | > loss: 0.008898450993001461 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13671721249221458) | > loader_time: 0.0046 (0.0046659350725065775)  --> TIME: 2025-02-26 11:00:06 -- STEP: 29684/36938 -- GLOBAL_STEP: 1211700 | > loss_text_ce: 0.03226986899971962 (0.031602932279016595) | > loss_mel_ce: 3.2156219482421875 (nan) | > loss: 0.012888460420072079 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13671066996833134) | > loader_time: 0.0047 (0.004665987998580479)  --> TIME: 2025-02-26 11:00:15 -- STEP: 29734/36938 -- GLOBAL_STEP: 1211750 | > loss_text_ce: 0.03218383714556694 (0.03160300850522124) | > loss_mel_ce: 2.728078603744507 (nan) | > loss: 0.01095342356711626 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13670157016690818) | > loader_time: 0.0045 (0.004666038413306633)  --> TIME: 2025-02-26 11:00:25 -- STEP: 29784/36938 -- GLOBAL_STEP: 1211800 | > loss_text_ce: 0.022536417469382286 (0.03160392157020194) | > loss_mel_ce: 2.3313755989074707 (nan) | > loss: 0.0093409214168787 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13670513568595674) | > loader_time: 0.0045 (0.004666264935139212)  --> TIME: 2025-02-26 11:00:34 -- STEP: 29834/36938 -- GLOBAL_STEP: 1211850 | > loss_text_ce: 0.041873060166835785 (0.031606149770992714) | > loss_mel_ce: 1.8222695589065552 (nan) | > loss: 0.0073973918333649635 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13670260294709508) | > loader_time: 0.0047 (0.004666432271800005)  --> TIME: 2025-02-26 11:00:43 -- STEP: 29884/36938 -- GLOBAL_STEP: 1211900 | > loss_text_ce: 0.030113747343420982 (0.031604835099939416) | > loss_mel_ce: 1.1511374711990356 (nan) | > loss: 0.004687504842877388 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13670053716293737) | > loader_time: 0.0045 (0.004666582717264924)  --> TIME: 2025-02-26 11:00:52 -- STEP: 29934/36938 -- GLOBAL_STEP: 1211950 | > loss_text_ce: 0.03761037811636925 (0.03160734463276457) | > loss_mel_ce: 1.5817935466766357 (nan) | > loss: 0.006426206324249506 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13669548281099184) | > loader_time: 0.0045 (0.00466647354898755)  --> TIME: 2025-02-26 11:01:02 -- STEP: 29984/36938 -- GLOBAL_STEP: 1212000 | > loss_text_ce: 0.03840324655175209 (0.03160644189245341) | > loss_mel_ce: 0.01202392578125 (nan) | > loss: 0.00020010783919133246 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.1366969467465757) | > loader_time: 0.0046 (0.004666577487897729)  --> TIME: 2025-02-26 11:01:11 -- STEP: 30034/36938 -- GLOBAL_STEP: 1212050 | > loss_text_ce: 0.024546468630433083 (0.03160661539569123) | > loss_mel_ce: 1.9530367851257324 (nan) | > loss: 0.007847553119063377 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.1367005988396845) | > loader_time: 0.0052 (0.004666717453978396)  --> TIME: 2025-02-26 11:01:21 -- STEP: 30084/36938 -- GLOBAL_STEP: 1212100 | > loss_text_ce: 0.03824716806411743 (0.03160592915325651) | > loss_mel_ce: 3.0179131031036377 (nan) | > loss: 0.012127620168030262 (nan) | > current_lr: 5e-06 | > step_time: 0.1714 (0.13670935788983293) | > loader_time: 0.0048 (0.004666672109876006)  --> TIME: 2025-02-26 11:01:30 -- STEP: 30134/36938 -- GLOBAL_STEP: 1212150 | > loss_text_ce: 0.026142070069909096 (0.031604285145464366) | > loss_mel_ce: 1.9205780029296875 (nan) | > loss: 0.0077250804752111435 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13671276051450515) | > loader_time: 0.0046 (0.004666721456090599)  --> TIME: 2025-02-26 11:01:40 -- STEP: 30184/36938 -- GLOBAL_STEP: 1212200 | > loss_text_ce: 0.036539915949106216 (0.03160531636870994) | > loss_mel_ce: 1.5555801391601562 (nan) | > loss: 0.006317937280982733 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13671815102308313) | > loader_time: 0.0046 (0.0046667152679538484)  --> TIME: 2025-02-26 11:01:49 -- STEP: 30234/36938 -- GLOBAL_STEP: 1212250 | > loss_text_ce: 0.037148382514715195 (0.031603855310498925) | > loss_mel_ce: 2.8205504417419434 (nan) | > loss: 0.011340076103806496 (nan) | > current_lr: 5e-06 | > step_time: 0.1878 (0.13672504262128718) | > loader_time: 0.0056 (0.0046667320794384895)  --> TIME: 2025-02-26 11:01:59 -- STEP: 30284/36938 -- GLOBAL_STEP: 1212300 | > loss_text_ce: 0.024155883118510246 (0.03160523706127292) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13672870026760928) | > loader_time: 0.0048 (0.004666800236642141)  --> TIME: 2025-02-26 11:02:08 -- STEP: 30334/36938 -- GLOBAL_STEP: 1212350 | > loss_text_ce: 0.03771691769361496 (0.03160727234451849) | > loss_mel_ce: 3.1897659301757812 (nan) | > loss: 0.012807472608983517 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.13672747889921388) | > loader_time: 0.0046 (0.00466676636928072)  --> TIME: 2025-02-26 11:02:17 -- STEP: 30384/36938 -- GLOBAL_STEP: 1212400 | > loss_text_ce: 0.03472478687763214 (0.0316065942444279) | > loss_mel_ce: 2.9117281436920166 (nan) | > loss: 0.011692274361848831 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.13672799521712936) | > loader_time: 0.0046 (0.004666699499692217)  --> TIME: 2025-02-26 11:02:26 -- STEP: 30434/36938 -- GLOBAL_STEP: 1212450 | > loss_text_ce: 0.0371677540242672 (0.03160705702539886) | > loss_mel_ce: 2.6016197204589844 (nan) | > loss: 0.01047137938439846 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13672517933450418) | > loader_time: 0.0046 (0.004666655881650715)  --> TIME: 2025-02-26 11:02:36 -- STEP: 30484/36938 -- GLOBAL_STEP: 1212500 | > loss_text_ce: 0.02869008295238018 (0.03160430709083222) | > loss_mel_ce: 3.226524829864502 (nan) | > loss: 0.012917520478367805 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13672880457855754) | > loader_time: 0.0052 (0.004666708410763871)  --> TIME: 2025-02-26 11:02:45 -- STEP: 30534/36938 -- GLOBAL_STEP: 1212550 | > loss_text_ce: 0.0388486310839653 (0.03160363358389712) | > loss_mel_ce: 3.211547613143921 (nan) | > loss: 0.0128983985632658 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.13672532336123336) | > loader_time: 0.0047 (0.004666787753320994)  --> TIME: 2025-02-26 11:02:55 -- STEP: 30584/36938 -- GLOBAL_STEP: 1212600 | > loss_text_ce: 0.03249964118003845 (0.031603863038877966) | > loss_mel_ce: 3.3363358974456787 (nan) | > loss: 0.013368395157158375 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13672930771631667) | > loader_time: 0.0046 (0.004666846201677403)  --> TIME: 2025-02-26 11:03:04 -- STEP: 30634/36938 -- GLOBAL_STEP: 1212650 | > loss_text_ce: 0.03371401131153107 (0.031604608151866026) | > loss_mel_ce: 0.05377197265625 (nan) | > loss: 0.00034716661320999265 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13672254251625962) | > loader_time: 0.0046 (0.004666839939749279)  --> TIME: 2025-02-26 11:03:13 -- STEP: 30684/36938 -- GLOBAL_STEP: 1212700 | > loss_text_ce: 0.040780775249004364 (0.0316034610956836) | > loss_mel_ce: 2.498265266418457 (nan) | > loss: 0.010075580328702927 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13672929015406426) | > loader_time: 0.0046 (0.00466688034230177)  --> TIME: 2025-02-26 11:03:23 -- STEP: 30734/36938 -- GLOBAL_STEP: 1212750 | > loss_text_ce: 0.023014845326542854 (0.03160435564146853) | > loss_mel_ce: 3.337207078933716 (nan) | > loss: 0.013334214687347412 (nan) | > current_lr: 5e-06 | > step_time: 0.1074 (0.13673384979162695) | > loader_time: 0.0048 (0.004666919473044982)  --> TIME: 2025-02-26 11:03:32 -- STEP: 30784/36938 -- GLOBAL_STEP: 1212800 | > loss_text_ce: 0.03352736309170723 (0.03160318296186742) | > loss_mel_ce: 1.9914875030517578 (nan) | > loss: 0.008035773411393166 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13673487446598145) | > loader_time: 0.0046 (0.0046669191249071275)  --> TIME: 2025-02-26 11:03:41 -- STEP: 30834/36938 -- GLOBAL_STEP: 1212850 | > loss_text_ce: 0.02453722432255745 (0.031604060706062496) | > loss_mel_ce: 2.398839235305786 (nan) | > loss: 0.00961657427251339 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13672570516652285) | > loader_time: 0.0045 (0.004666871394196822)  --> TIME: 2025-02-26 11:03:51 -- STEP: 30884/36938 -- GLOBAL_STEP: 1212900 | > loss_text_ce: 0.032022152096033096 (0.03160448134121994) | > loss_mel_ce: 3.131282329559326 (nan) | > loss: 0.012552796863019466 (nan) | > current_lr: 5e-06 | > step_time: 0.1757 (0.13673435466738237) | > loader_time: 0.0049 (0.004666893404392586)  --> TIME: 2025-02-26 11:04:00 -- STEP: 30934/36938 -- GLOBAL_STEP: 1212950 | > loss_text_ce: 0.028771013021469116 (0.031604769707636864) | > loss_mel_ce: 2.9846689701080322 (nan) | > loss: 0.011958095245063305 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.1367373833900175) | > loader_time: 0.0047 (0.0046668890306079845)  --> TIME: 2025-02-26 11:04:10 -- STEP: 30984/36938 -- GLOBAL_STEP: 1213000 | > loss_text_ce: 0.025118134915828705 (0.03160408370741413) | > loss_mel_ce: 1.0180362462997437 (nan) | > loss: 0.004139501601457596 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.1367478354430283) | > loader_time: 0.0045 (0.004666926146415425)  --> TIME: 2025-02-26 11:04:19 -- STEP: 31034/36938 -- GLOBAL_STEP: 1213050 | > loss_text_ce: 0.034317802637815475 (0.03160290676907334) | > loss_mel_ce: 3.0201756954193115 (nan) | > loss: 0.012121005915105343 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.1367513260645009) | > loader_time: 0.0046 (0.004666865060199919)  --> TIME: 2025-02-26 11:04:29 -- STEP: 31084/36938 -- GLOBAL_STEP: 1213100 | > loss_text_ce: 0.029236048460006714 (0.03160314228360936) | > loss_mel_ce: 2.4087655544281006 (nan) | > loss: 0.009674610570073128 (nan) | > current_lr: 5e-06 | > step_time: 0.1781 (0.13675513186673874) | > loader_time: 0.0048 (0.004666821957554947)  --> TIME: 2025-02-26 11:04:38 -- STEP: 31134/36938 -- GLOBAL_STEP: 1213150 | > loss_text_ce: 0.04704229161143303 (0.03160309453303161) | > loss_mel_ce: 2.907475233078003 (nan) | > loss: 0.011724276468157768 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.13676013544957144) | > loader_time: 0.0047 (0.0046667837871478815)  --> TIME: 2025-02-26 11:04:47 -- STEP: 31184/36938 -- GLOBAL_STEP: 1213200 | > loss_text_ce: 0.03384878858923912 (0.031600130186574475) | > loss_mel_ce: 1.336458683013916 (nan) | > loss: 0.0054377284832298756 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.1367544087157491) | > loader_time: 0.0047 (0.004666695400896663)  --> TIME: 2025-02-26 11:04:57 -- STEP: 31234/36938 -- GLOBAL_STEP: 1213250 | > loss_text_ce: 0.035340871661901474 (0.03160112408937281) | > loss_mel_ce: 2.295577049255371 (nan) | > loss: 0.009249674156308174 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.13675043235143025) | > loader_time: 0.0048 (0.004666642181820049)  --> TIME: 2025-02-26 11:05:06 -- STEP: 31284/36938 -- GLOBAL_STEP: 1213300 | > loss_text_ce: 0.02769617922604084 (0.03160118658274275) | > loss_mel_ce: 3.170623302459717 (nan) | > loss: 0.012691744603216648 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.13674808770185765) | > loader_time: 0.0047 (0.0046665667420594525)  --> TIME: 2025-02-26 11:05:15 -- STEP: 31334/36938 -- GLOBAL_STEP: 1213350 | > loss_text_ce: 0.03313354775309563 (0.03159988257515212) | > loss_mel_ce: 2.6263058185577393 (nan) | > loss: 0.01055333111435175 (nan) | > current_lr: 5e-06 | > step_time: 0.1528 (0.13674271099871962) | > loader_time: 0.0047 (0.004666462933438149)  --> TIME: 2025-02-26 11:05:24 -- STEP: 31384/36938 -- GLOBAL_STEP: 1213400 | > loss_text_ce: 0.03840326517820358 (0.0316017406931988) | > loss_mel_ce: 0.05615234375 (nan) | > loss: 0.0003752206976059824 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13674554778792417) | > loader_time: 0.0045 (0.004666400083376085)  --> TIME: 2025-02-26 11:05:34 -- STEP: 31434/36938 -- GLOBAL_STEP: 1213450 | > loss_text_ce: 0.026991939172148705 (0.03160256915571704) | > loss_mel_ce: 1.97812819480896 (nan) | > loss: 0.007956826128065586 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13675554919547805) | > loader_time: 0.0045 (0.004666383146458466)  --> TIME: 2025-02-26 11:05:43 -- STEP: 31484/36938 -- GLOBAL_STEP: 1213500 | > loss_text_ce: 0.03311813250184059 (0.031602523319867244) | > loss_mel_ce: 2.617462158203125 (nan) | > loss: 0.010518176481127739 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13675280323690156) | > loader_time: 0.0046 (0.0046663307625643795)  --> TIME: 2025-02-26 11:05:53 -- STEP: 31534/36938 -- GLOBAL_STEP: 1213550 | > loss_text_ce: 0.03245016932487488 (0.031601099227305515) | > loss_mel_ce: 2.09344482421875 (nan) | > loss: 0.008436092175543308 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13676069585297335) | > loader_time: 0.0046 (0.00466630081100063)  --> TIME: 2025-02-26 11:06:03 -- STEP: 31584/36938 -- GLOBAL_STEP: 1213600 | > loss_text_ce: 0.02753039449453354 (0.03160082754702201) | > loss_mel_ce: 2.7507622241973877 (nan) | > loss: 0.01102497149258852 (nan) | > current_lr: 5e-06 | > step_time: 0.182 (0.13677290169542966) | > loader_time: 0.0048 (0.004666309399868104)  --> TIME: 2025-02-26 11:06:12 -- STEP: 31634/36938 -- GLOBAL_STEP: 1213650 | > loss_text_ce: 0.034572359174489975 (0.03160105968467875) | > loss_mel_ce: 2.708899974822998 (nan) | > loss: 0.010886795818805695 (nan) | > current_lr: 5e-06 | > step_time: 0.1659 (0.1367773990636534) | > loader_time: 0.0048 (0.004666261156854772)  --> TIME: 2025-02-26 11:06:22 -- STEP: 31684/36938 -- GLOBAL_STEP: 1213700 | > loss_text_ce: 0.038838986307382584 (0.031599170416816376) | > loss_mel_ce: 3.1724796295166016 (nan) | > loss: 0.01274332869797945 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.13677859298058112) | > loader_time: 0.0048 (0.00466619550301382)  --> TIME: 2025-02-26 11:06:31 -- STEP: 31734/36938 -- GLOBAL_STEP: 1213750 | > loss_text_ce: 0.03605715557932854 (0.031599575458083895) | > loss_mel_ce: 3.287647247314453 (nan) | > loss: 0.013189304620027542 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13677712072682735) | > loader_time: 0.0045 (0.0046661208075176475)  --> TIME: 2025-02-26 11:06:40 -- STEP: 31784/36938 -- GLOBAL_STEP: 1213800 | > loss_text_ce: 0.042484745383262634 (0.03159939302107797) | > loss_mel_ce: 1.9911319017410278 (nan) | > loss: 0.008069907315075397 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13677683529642626) | > loader_time: 0.0046 (0.004666045311863713)  --> TIME: 2025-02-26 11:06:50 -- STEP: 31834/36938 -- GLOBAL_STEP: 1213850 | > loss_text_ce: 0.024268100038170815 (0.03159984426342443) | > loss_mel_ce: 2.6646134853363037 (nan) | > loss: 0.010670165531337261 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13678326595965745) | > loader_time: 0.0046 (0.004666124223322181)  --> TIME: 2025-02-26 11:07:00 -- STEP: 31884/36938 -- GLOBAL_STEP: 1213900 | > loss_text_ce: 0.026345232501626015 (0.03160009798155967) | > loss_mel_ce: 2.675734281539917 (nan) | > loss: 0.010722538456320763 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.1367917690473318) | > loader_time: 0.0052 (0.004666255619935396)  --> TIME: 2025-02-26 11:07:09 -- STEP: 31934/36938 -- GLOBAL_STEP: 1213950 | > loss_text_ce: 0.03290340304374695 (0.031602282642899014) | > loss_mel_ce: 2.3711068630218506 (nan) | > loss: 0.009539724327623844 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1367927071908876) | > loader_time: 0.0045 (0.004666370381512796)  --> TIME: 2025-02-26 11:07:18 -- STEP: 31984/36938 -- GLOBAL_STEP: 1214000 | > loss_text_ce: 0.02533414587378502 (0.03160052881100617) | > loss_mel_ce: 2.1441829204559326 (nan) | > loss: 0.008609195239841938 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.1367902537624264) | > loader_time: 0.0046 (0.004666451866057835)  --> TIME: 2025-02-26 11:07:28 -- STEP: 32034/36938 -- GLOBAL_STEP: 1214050 | > loss_text_ce: 0.03342326730489731 (0.03160008865941448) | > loss_mel_ce: 0.03558349609375 (nan) | > loss: 0.00027383636916056275 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13679298526481493) | > loader_time: 0.0046 (0.004666649030747475)  --> TIME: 2025-02-26 11:07:37 -- STEP: 32084/36938 -- GLOBAL_STEP: 1214100 | > loss_text_ce: 0.027484910562634468 (0.031601113762033416) | > loss_mel_ce: 3.197406530380249 (nan) | > loss: 0.012797188945114613 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.13679625050650954) | > loader_time: 0.0057 (0.004666749935556783)  --> TIME: 2025-02-26 11:07:47 -- STEP: 32134/36938 -- GLOBAL_STEP: 1214150 | > loss_text_ce: 0.034653447568416595 (0.0316006042497654) | > loss_mel_ce: 2.976231813430786 (nan) | > loss: 0.011947957798838615 (nan) | > current_lr: 5e-06 | > step_time: 0.1742 (0.13680460717613263) | > loader_time: 0.0048 (0.004666820269586029)  --> TIME: 2025-02-26 11:07:56 -- STEP: 32184/36938 -- GLOBAL_STEP: 1214200 | > loss_text_ce: 0.031534042209386826 (0.03160003633518774) | > loss_mel_ce: 2.98878812789917 (nan) | > loss: 0.01198540534824133 (nan) | > current_lr: 5e-06 | > step_time: 0.1537 (0.1368058442536087) | > loader_time: 0.0047 (0.00466684884109991)  --> TIME: 2025-02-26 11:08:05 -- STEP: 32234/36938 -- GLOBAL_STEP: 1214250 | > loss_text_ce: 0.03298458084464073 (0.03159887751129774) | > loss_mel_ce: 2.00229811668396 (nan) | > loss: 0.008076518774032593 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.1368052579780119) | > loader_time: 0.0046 (0.004666979735828248)  --> TIME: 2025-02-26 11:08:15 -- STEP: 32284/36938 -- GLOBAL_STEP: 1214300 | > loss_text_ce: 0.02541450969874859 (0.031597702592344554) | > loss_mel_ce: 2.697242498397827 (nan) | > loss: 0.010804194957017899 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13681239461296224) | > loader_time: 0.0046 (0.0046670983721223075)  --> TIME: 2025-02-26 11:08:24 -- STEP: 32334/36938 -- GLOBAL_STEP: 1214350 | > loss_text_ce: 0.027979569509625435 (0.03159697696593591) | > loss_mel_ce: 2.9464354515075684 (nan) | > loss: 0.011803234927356243 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.1368093443483168) | > loader_time: 0.0048 (0.004667180975315094)  --> TIME: 2025-02-26 11:08:34 -- STEP: 32384/36938 -- GLOBAL_STEP: 1214400 | > loss_text_ce: 0.02352876402437687 (0.03159659058742227) | > loss_mel_ce: 1.8880847692489624 (nan) | > loss: 0.007585768587887287 (nan) | > current_lr: 5e-06 | > step_time: 0.1632 (0.13680596860206684) | > loader_time: 0.0047 (0.0046672290964088975)  --> TIME: 2025-02-26 11:08:43 -- STEP: 32434/36938 -- GLOBAL_STEP: 1214450 | > loss_text_ce: 0.032726094126701355 (0.031595975705698325) | > loss_mel_ce: 3.349778413772583 (nan) | > loss: 0.01342263724654913 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.1368154744489751) | > loader_time: 0.0046 (0.004667382135324792)  --> TIME: 2025-02-26 11:08:53 -- STEP: 32484/36938 -- GLOBAL_STEP: 1214500 | > loss_text_ce: 0.02665221132338047 (0.03159298595858874) | > loss_mel_ce: 2.7760567665100098 (nan) | > loss: 0.011121861636638641 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13681602712159655) | > loader_time: 0.0052 (0.004667380498762342)  --> TIME: 2025-02-26 11:09:02 -- STEP: 32534/36938 -- GLOBAL_STEP: 1214550 | > loss_text_ce: 0.032763026654720306 (0.03159220772254337) | > loss_mel_ce: 2.7338993549346924 (nan) | > loss: 0.01097881980240345 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13682673533957873) | > loader_time: 0.0047 (0.004667549381876378)  --> TIME: 2025-02-26 11:09:12 -- STEP: 32584/36938 -- GLOBAL_STEP: 1214600 | > loss_text_ce: 0.038125548511743546 (0.03159207791087888) | > loss_mel_ce: 0.9642862677574158 (nan) | > loss: 0.003977824933826923 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.1368317783771559) | > loader_time: 0.0044 (0.004667644151855746)  --> TIME: 2025-02-26 11:09:21 -- STEP: 32634/36938 -- GLOBAL_STEP: 1214650 | > loss_text_ce: 0.03360016644001007 (0.03159179525005407) | > loss_mel_ce: 2.6887662410736084 (nan) | > loss: 0.010803041979670525 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13683575034148884) | > loader_time: 0.0046 (0.00466766389275311)  --> TIME: 2025-02-26 11:09:31 -- STEP: 32684/36938 -- GLOBAL_STEP: 1214700 | > loss_text_ce: 0.03738217428326607 (0.031592896741232626) | > loss_mel_ce: 2.619950532913208 (nan) | > loss: 0.010544971562922001 (nan) | > current_lr: 5e-06 | > step_time: 0.1556 (0.13683580117977912) | > loader_time: 0.0047 (0.004667623319381929)  --> TIME: 2025-02-26 11:09:40 -- STEP: 32734/36938 -- GLOBAL_STEP: 1214750 | > loss_text_ce: 0.03450235351920128 (0.031593709349210075) | > loss_mel_ce: 2.8530545234680176 (nan) | > loss: 0.011458558961749077 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13683239023388472) | > loader_time: 0.0046 (0.004667611909333654)  --> TIME: 2025-02-26 11:09:49 -- STEP: 32784/36938 -- GLOBAL_STEP: 1214800 | > loss_text_ce: 0.028149046003818512 (0.03159374932238582) | > loss_mel_ce: 1.1438192129135132 (nan) | > loss: 0.004650667775422335 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13683225271933405) | > loader_time: 0.0046 (0.004667849817643702)  --> TIME: 2025-02-26 11:09:58 -- STEP: 32834/36938 -- GLOBAL_STEP: 1214850 | > loss_text_ce: 0.032278623431921005 (0.03159142368295088) | > loss_mel_ce: 2.190488576889038 (nan) | > loss: 0.008820504881441593 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13682969573383982) | > loader_time: 0.0045 (0.004667875783745675)  --> TIME: 2025-02-26 11:10:08 -- STEP: 32884/36938 -- GLOBAL_STEP: 1214900 | > loss_text_ce: 0.03157170116901398 (0.03159307915209097) | > loss_mel_ce: 3.3423264026641846 (nan) | > loss: 0.013388484716415405 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.13683822150358632) | > loader_time: 0.0047 (0.00466796662624468)  --> TIME: 2025-02-26 11:10:18 -- STEP: 32934/36938 -- GLOBAL_STEP: 1214950 | > loss_text_ce: 0.02904844470322132 (0.03159281732942345) | > loss_mel_ce: 2.9109280109405518 (nan) | > loss: 0.011666573584079742 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13684635460756842) | > loader_time: 0.0048 (0.00466812457616715)  --> TIME: 2025-02-26 11:10:27 -- STEP: 32984/36938 -- GLOBAL_STEP: 1215000 | > loss_text_ce: 0.039274368435144424 (0.03159327862494616) | > loss_mel_ce: 2.400240421295166 (nan) | > loss: 0.009680614806711674 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.13685330384974087) | > loader_time: 0.005 (0.004668310505078027)  --> TIME: 2025-02-26 11:10:37 -- STEP: 33034/36938 -- GLOBAL_STEP: 1215050 | > loss_text_ce: 0.04306037724018097 (0.03159270809243911) | > loss_mel_ce: 2.5169320106506348 (nan) | > loss: 0.010158699937164783 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13685505853666738) | > loader_time: 0.0045 (0.004668386946609634)  --> TIME: 2025-02-26 11:10:46 -- STEP: 33084/36938 -- GLOBAL_STEP: 1215100 | > loss_text_ce: 0.0400150865316391 (0.0315924335838825) | > loss_mel_ce: 0.0015106201171875 (nan) | > loss: 0.00016478456382174045 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13685398384444108) | > loader_time: 0.0046 (0.004668517306441632)  --> TIME: 2025-02-26 11:10:56 -- STEP: 33134/36938 -- GLOBAL_STEP: 1215150 | > loss_text_ce: 0.028168434277176857 (0.03159238052561839) | > loss_mel_ce: 1.7681478261947632 (nan) | > loss: 0.00712823960930109 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13685821964477865) | > loader_time: 0.0045 (0.0046686662907778065)  --> TIME: 2025-02-26 11:11:05 -- STEP: 33184/36938 -- GLOBAL_STEP: 1215200 | > loss_text_ce: 0.03164282441139221 (0.03159140645828717) | > loss_mel_ce: 3.0101444721221924 (nan) | > loss: 0.01207058597356081 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.13686506948797583) | > loader_time: 0.0047 (0.0046689572924588285)  --> TIME: 2025-02-26 11:11:14 -- STEP: 33234/36938 -- GLOBAL_STEP: 1215250 | > loss_text_ce: 0.03318261727690697 (0.031591788801887025) | > loss_mel_ce: 2.844388008117676 (nan) | > loss: 0.011418932117521763 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13686107576930073) | > loader_time: 0.0047 (0.004669094692608329)  --> TIME: 2025-02-26 11:11:24 -- STEP: 33284/36938 -- GLOBAL_STEP: 1215300 | > loss_text_ce: 0.03571193292737007 (0.0315908723686494) | > loss_mel_ce: 3.037715196609497 (nan) | > loss: 0.012196140363812447 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.1368584700923548) | > loader_time: 0.0047 (0.004669092614380542)  --> TIME: 2025-02-26 11:11:33 -- STEP: 33334/36938 -- GLOBAL_STEP: 1215350 | > loss_text_ce: 0.03192954510450363 (0.0315911837775624) | > loss_mel_ce: 2.608750104904175 (nan) | > loss: 0.01047888770699501 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13687103383434002) | > loader_time: 0.0045 (0.004669211532628031)  --> TIME: 2025-02-26 11:11:43 -- STEP: 33384/36938 -- GLOBAL_STEP: 1215400 | > loss_text_ce: 0.03170302137732506 (0.03159141881118171) | > loss_mel_ce: 2.8658580780029297 (nan) | > loss: 0.01149825844913721 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13686788334224922) | > loader_time: 0.0046 (0.004669333329853221)  --> TIME: 2025-02-26 11:11:52 -- STEP: 33434/36938 -- GLOBAL_STEP: 1215450 | > loss_text_ce: 0.033905819058418274 (0.03159239285083662) | > loss_mel_ce: 2.950533390045166 (nan) | > loss: 0.01184301357716322 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13686274923105674) | > loader_time: 0.0047 (0.004669452972900452)  --> TIME: 2025-02-26 11:12:01 -- STEP: 33484/36938 -- GLOBAL_STEP: 1215500 | > loss_text_ce: 0.05312758684158325 (0.03159449916244113) | > loss_mel_ce: 1.5513153076171875 (nan) | > loss: 0.006366836838424206 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1368707975041356) | > loader_time: 0.0046 (0.004669462633366493)  --> TIME: 2025-02-26 11:12:11 -- STEP: 33534/36938 -- GLOBAL_STEP: 1215550 | > loss_text_ce: 0.03428421914577484 (0.031595780980355634) | > loss_mel_ce: 2.5828964710235596 (nan) | > loss: 0.010385638102889061 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13686524875377232) | > loader_time: 0.0046 (0.004669521955113704)  --> TIME: 2025-02-26 11:12:20 -- STEP: 33584/36938 -- GLOBAL_STEP: 1215600 | > loss_text_ce: 0.030207645148038864 (0.03159540148052087) | > loss_mel_ce: 2.801994800567627 (nan) | > loss: 0.01123889908194542 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13686464739703455) | > loader_time: 0.0045 (0.004669534692939216)  --> TIME: 2025-02-26 11:12:29 -- STEP: 33634/36938 -- GLOBAL_STEP: 1215650 | > loss_text_ce: 0.030633801594376564 (0.03159368318481097) | > loss_mel_ce: 2.5162417888641357 (nan) | > loss: 0.010106649249792099 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13686861003445477) | > loader_time: 0.0047 (0.004669665631041735)  --> TIME: 2025-02-26 11:12:38 -- STEP: 33684/36938 -- GLOBAL_STEP: 1215700 | > loss_text_ce: 0.026793397963047028 (0.03159391209793633) | > loss_mel_ce: 2.6860756874084473 (nan) | > loss: 0.01076535414904356 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13686437033344248) | > loader_time: 0.0046 (0.004669636357016053)  --> TIME: 2025-02-26 11:12:48 -- STEP: 33734/36938 -- GLOBAL_STEP: 1215750 | > loss_text_ce: 0.04105852544307709 (0.03159435332071173) | > loss_mel_ce: 2.6077675819396973 (nan) | > loss: 0.010511215776205063 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13686346239502867) | > loader_time: 0.0046 (0.0046697035931011575)  --> TIME: 2025-02-26 11:12:57 -- STEP: 33784/36938 -- GLOBAL_STEP: 1215800 | > loss_text_ce: 0.0388837568461895 (0.0315941582699376) | > loss_mel_ce: 0.12060546875 (nan) | > loss: 0.0006328938179649413 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13686695096477564) | > loader_time: 0.0047 (0.00466973833667423)  --> TIME: 2025-02-26 11:13:07 -- STEP: 33834/36938 -- GLOBAL_STEP: 1215850 | > loss_text_ce: 0.028423402458429337 (0.031593645720040024) | > loss_mel_ce: 2.9664244651794434 (nan) | > loss: 0.011884316802024841 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13686874499520868) | > loader_time: 0.0046 (0.004669803870360605)  --> TIME: 2025-02-26 11:13:16 -- STEP: 33884/36938 -- GLOBAL_STEP: 1215900 | > loss_text_ce: 0.03376513347029686 (0.03159479884275573) | > loss_mel_ce: 1.9113773107528687 (nan) | > loss: 0.007718819659203291 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13686978270967884) | > loader_time: 0.0048 (0.004669799361118063)  --> TIME: 2025-02-26 11:13:26 -- STEP: 33934/36938 -- GLOBAL_STEP: 1215950 | > loss_text_ce: 0.0307503379881382 (0.03159338243123932) | > loss_mel_ce: 3.2839221954345703 (nan) | > loss: 0.013153462670743465 (nan) | > current_lr: 5e-06 | > step_time: 0.1701 (0.13687430946689402) | > loader_time: 0.005 (0.004669954937377139)  --> TIME: 2025-02-26 11:13:35 -- STEP: 33984/36938 -- GLOBAL_STEP: 1216000 | > loss_text_ce: 0.03363494947552681 (0.03159341564687913) | > loss_mel_ce: 2.7009479999542236 (nan) | > loss: 0.010851520113646984 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.13687964418653975) | > loader_time: 0.0046 (0.004669998304150209)  --> TIME: 2025-02-26 11:13:45 -- STEP: 34034/36938 -- GLOBAL_STEP: 1216050 | > loss_text_ce: 0.03964865952730179 (0.03159425993476353) | > loss_mel_ce: 2.38995623588562 (nan) | > loss: 0.00964129064232111 (nan) | > current_lr: 5e-06 | > step_time: 0.1791 (0.13689144409213574) | > loader_time: 0.0048 (0.004670013515272687)  --> TIME: 2025-02-26 11:13:54 -- STEP: 34084/36938 -- GLOBAL_STEP: 1216100 | > loss_text_ce: 0.02712278626859188 (0.03159524498494199) | > loss_mel_ce: 2.9076125621795654 (nan) | > loss: 0.01164577528834343 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.13688365905417257) | > loader_time: 0.0046 (0.004669975778360134)  --> TIME: 2025-02-26 11:14:03 -- STEP: 34134/36938 -- GLOBAL_STEP: 1216150 | > loss_text_ce: 0.03386862203478813 (0.03159612677823035) | > loss_mel_ce: 3.487346887588501 (nan) | > loss: 0.01397307775914669 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13687831368409775) | > loader_time: 0.0046 (0.004670005953290278)  --> TIME: 2025-02-26 11:14:12 -- STEP: 34184/36938 -- GLOBAL_STEP: 1216200 | > loss_text_ce: 0.026388119906187057 (0.03159673911062013) | > loss_mel_ce: 2.768061637878418 (nan) | > loss: 0.011089087463915348 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.1368789878392038) | > loader_time: 0.0047 (0.00467010024780696)  --> TIME: 2025-02-26 11:14:22 -- STEP: 34234/36938 -- GLOBAL_STEP: 1216250 | > loss_text_ce: 0.03451727703213692 (0.03159407595736573) | > loss_mel_ce: 2.3854658603668213 (nan) | > loss: 0.009603108279407024 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.1368736149503257) | > loader_time: 0.0054 (0.004670173721962013)  --> TIME: 2025-02-26 11:14:31 -- STEP: 34284/36938 -- GLOBAL_STEP: 1216300 | > loss_text_ce: 0.030092310160398483 (0.03159254586116597) | > loss_mel_ce: 2.5979580879211426 (nan) | > loss: 0.010428771376609802 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13686756283938456) | > loader_time: 0.0047 (0.004670088328164692)  --> TIME: 2025-02-26 11:14:40 -- STEP: 34334/36938 -- GLOBAL_STEP: 1216350 | > loss_text_ce: 0.03700500354170799 (0.03159357358159767) | > loss_mel_ce: 2.9851889610290527 (nan) | > loss: 0.011992833577096462 (nan) | > current_lr: 5e-06 | > step_time: 0.1688 (0.13687802822816128) | > loader_time: 0.005 (0.0046700909703392005)  --> TIME: 2025-02-26 11:14:50 -- STEP: 34384/36938 -- GLOBAL_STEP: 1216400 | > loss_text_ce: 0.02741692028939724 (0.0315940973770832) | > loss_mel_ce: 2.829359531402588 (nan) | > loss: 0.011336415074765682 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13688073155767935) | > loader_time: 0.0046 (0.004670061708439682)  --> TIME: 2025-02-26 11:15:00 -- STEP: 34434/36938 -- GLOBAL_STEP: 1216450 | > loss_text_ce: 0.03127412125468254 (0.031593918434102385) | > loss_mel_ce: 3.021071434020996 (nan) | > loss: 0.012112482450902462 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13689078915345598) | > loader_time: 0.0046 (0.004670062429046247)  --> TIME: 2025-02-26 11:15:09 -- STEP: 34484/36938 -- GLOBAL_STEP: 1216500 | > loss_text_ce: 0.03159308433532715 (0.03159274654342199) | > loss_mel_ce: 3.0605967044830322 (nan) | > loss: 0.012270594947040081 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13689203586452273) | > loader_time: 0.0045 (0.004670026289616101)  --> TIME: 2025-02-26 11:15:18 -- STEP: 34534/36938 -- GLOBAL_STEP: 1216550 | > loss_text_ce: 0.024714751169085503 (0.03159335259049173) | > loss_mel_ce: 2.523620128631592 (nan) | > loss: 0.010112440213561058 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13688366400137178) | > loader_time: 0.0045 (0.004669940001494978)  --> TIME: 2025-02-26 11:15:27 -- STEP: 34584/36938 -- GLOBAL_STEP: 1216600 | > loss_text_ce: 0.02845558151602745 (0.031594153759555973) | > loss_mel_ce: 1.003543496131897 (nan) | > loss: 0.0040952349081635475 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.1368805030044131) | > loader_time: 0.0045 (0.0046698571547517855)  --> TIME: 2025-02-26 11:15:36 -- STEP: 34634/36938 -- GLOBAL_STEP: 1216650 | > loss_text_ce: 0.02760463021695614 (0.03159408121195625) | > loss_mel_ce: 3.0332462787628174 (nan) | > loss: 0.012146234512329102 (nan) | > current_lr: 5e-06 | > step_time: 0.1973 (0.13687850990202907) | > loader_time: 0.0049 (0.004669830479274676)  --> TIME: 2025-02-26 11:15:45 -- STEP: 34684/36938 -- GLOBAL_STEP: 1216700 | > loss_text_ce: 0.03337086737155914 (0.03159255488801598) | > loss_mel_ce: 1.947434425354004 (nan) | > loss: 0.007860339246690273 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13687120156502092) | > loader_time: 0.0045 (0.004669840237409968)  --> TIME: 2025-02-26 11:15:55 -- STEP: 34734/36938 -- GLOBAL_STEP: 1216750 | > loss_text_ce: 0.03222830593585968 (0.03159142774258932) | > loss_mel_ce: 3.3541646003723145 (nan) | > loss: 0.013438067398965359 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.13687117604138027) | > loader_time: 0.0047 (0.004669990970343394)  --> TIME: 2025-02-26 11:16:04 -- STEP: 34784/36938 -- GLOBAL_STEP: 1216800 | > loss_text_ce: 0.026391051709651947 (0.03158983093926129) | > loss_mel_ce: 2.3082032203674316 (nan) | > loss: 0.009264263324439526 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13687065907213405) | > loader_time: 0.0046 (0.0046700680527318765)  --> TIME: 2025-02-26 11:16:13 -- STEP: 34834/36938 -- GLOBAL_STEP: 1216850 | > loss_text_ce: 0.025398852303624153 (0.03158913206472105) | > loss_mel_ce: 1.5747191905975342 (nan) | > loss: 0.006349675357341766 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1368660052238668) | > loader_time: 0.0052 (0.004670152483765635)  --> TIME: 2025-02-26 11:16:22 -- STEP: 34884/36938 -- GLOBAL_STEP: 1216900 | > loss_text_ce: 0.016495196148753166 (0.0315894679116424) | > loss_mel_ce: 1.2159361839294434 (nan) | > loss: 0.004890601150691509 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1368620019346706) | > loader_time: 0.0046 (0.004670278555263483)  --> TIME: 2025-02-26 11:16:32 -- STEP: 34934/36938 -- GLOBAL_STEP: 1216950 | > loss_text_ce: 0.031519677489995956 (0.03158893489732372) | > loss_mel_ce: 2.6090831756591797 (nan) | > loss: 0.010478583164513111 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13686085657746666) | > loader_time: 0.0045 (0.004670374031883702)  --> TIME: 2025-02-26 11:16:41 -- STEP: 34984/36938 -- GLOBAL_STEP: 1217000 | > loss_text_ce: 0.04022718220949173 (0.031590495181061454) | > loss_mel_ce: 3.3152127265930176 (nan) | > loss: 0.013315238058567047 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1368593459894804) | > loader_time: 0.0045 (0.004670529910200345)  --> TIME: 2025-02-26 11:16:51 -- STEP: 35034/36938 -- GLOBAL_STEP: 1217050 | > loss_text_ce: 0.030104171484708786 (0.031591152998013586) | > loss_mel_ce: 2.793504476547241 (nan) | > loss: 0.011204796843230724 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13686742450219577) | > loader_time: 0.0046 (0.004670649418147898)  --> TIME: 2025-02-26 11:17:00 -- STEP: 35084/36938 -- GLOBAL_STEP: 1217100 | > loss_text_ce: 0.02961222268640995 (0.03159102828151035) | > loss_mel_ce: 2.935811996459961 (nan) | > loss: 0.011767557822167873 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.13686850878004767) | > loader_time: 0.0047 (0.004670713975619554)  --> TIME: 2025-02-26 11:17:09 -- STEP: 35134/36938 -- GLOBAL_STEP: 1217150 | > loss_text_ce: 0.03385549783706665 (0.03159185124842638) | > loss_mel_ce: 2.581310987472534 (nan) | > loss: 0.010377645492553711 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13686668864011492) | > loader_time: 0.0046 (0.0046707116974610205)  --> TIME: 2025-02-26 11:17:19 -- STEP: 35184/36938 -- GLOBAL_STEP: 1217200 | > loss_text_ce: 0.0265819001942873 (0.03159344964415846) | > loss_mel_ce: 2.4914610385894775 (nan) | > loss: 0.009992234408855438 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.1368662927193012) | > loader_time: 0.0047 (0.004670923422444365)  --> TIME: 2025-02-26 11:17:28 -- STEP: 35234/36938 -- GLOBAL_STEP: 1217250 | > loss_text_ce: 0.02738036960363388 (0.03159263050510612) | > loss_mel_ce: 1.2628116607666016 (nan) | > loss: 0.005119809880852699 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13687341216745025) | > loader_time: 0.0048 (0.004670963619193435)  --> TIME: 2025-02-26 11:17:38 -- STEP: 35284/36938 -- GLOBAL_STEP: 1217300 | > loss_text_ce: 0.036975014954805374 (0.03159384025113141) | > loss_mel_ce: 2.6425628662109375 (nan) | > loss: 0.010633086785674095 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13687431677143788) | > loader_time: 0.0046 (0.004671051360057785)  --> TIME: 2025-02-26 11:17:47 -- STEP: 35334/36938 -- GLOBAL_STEP: 1217350 | > loss_text_ce: 0.03771968558430672 (0.03159331127617381) | > loss_mel_ce: 1.8464261293411255 (nan) | > loss: 0.007476769853383303 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.1368789981147961) | > loader_time: 0.0045 (0.004670997929639205)  --> TIME: 2025-02-26 11:17:57 -- STEP: 35384/36938 -- GLOBAL_STEP: 1217400 | > loss_text_ce: 0.03059299662709236 (0.03159233826870743) | > loss_mel_ce: 2.7987277507781982 (nan) | > loss: 0.011227463372051716 (nan) | > current_lr: 5e-06 | > step_time: 0.1451 (0.13688160541395578) | > loader_time: 0.0047 (0.004671252490761014)  --> TIME: 2025-02-26 11:18:06 -- STEP: 35434/36938 -- GLOBAL_STEP: 1217450 | > loss_text_ce: 0.026395883411169052 (0.03159044506743452) | > loss_mel_ce: 2.94626784324646 (nan) | > loss: 0.011796285398304462 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.13687981754203296) | > loader_time: 0.0048 (0.004671348549546313)  --> TIME: 2025-02-26 11:18:15 -- STEP: 35484/36938 -- GLOBAL_STEP: 1217500 | > loss_text_ce: 0.02877427078783512 (0.0315907986004291) | > loss_mel_ce: 2.681169271469116 (nan) | > loss: 0.010753745213150978 (nan) | > current_lr: 5e-06 | > step_time: 0.1894 (0.13688554519623874) | > loader_time: 0.0049 (0.00467134830903584)  --> TIME: 2025-02-26 11:18:24 -- STEP: 35534/36938 -- GLOBAL_STEP: 1217550 | > loss_text_ce: 0.030711663886904716 (0.03159131581376409) | > loss_mel_ce: 2.4502367973327637 (nan) | > loss: 0.009845034219324589 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.13687901336662206) | > loader_time: 0.0047 (0.0046712702983419015)  --> TIME: 2025-02-26 11:18:34 -- STEP: 35584/36938 -- GLOBAL_STEP: 1217600 | > loss_text_ce: 0.02703717164695263 (0.03158980231383655) | > loss_mel_ce: 2.777195692062378 (nan) | > loss: 0.01112790871411562 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.1368802598134864) | > loader_time: 0.0047 (0.004671304500092296)  --> TIME: 2025-02-26 11:18:43 -- STEP: 35634/36938 -- GLOBAL_STEP: 1217650 | > loss_text_ce: 0.026466993615031242 (0.031588066414549094) | > loss_mel_ce: 1.55914306640625 (nan) | > loss: 0.006292103789746761 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13687807113131267) | > loader_time: 0.0053 (0.004671394915306984)  --> TIME: 2025-02-26 11:18:52 -- STEP: 35684/36938 -- GLOBAL_STEP: 1217700 | > loss_text_ce: 0.02932562679052353 (0.03158966766716213) | > loss_mel_ce: 0.005523681640625 (nan) | > loss: 0.0001382909103995189 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1368743619728004) | > loader_time: 0.0048 (0.004671420588406244)  --> TIME: 2025-02-26 11:19:02 -- STEP: 35734/36938 -- GLOBAL_STEP: 1217750 | > loss_text_ce: 0.03515998274087906 (0.03158894391414168) | > loss_mel_ce: 2.5906214714050293 (nan) | > loss: 0.010419768281280994 (nan) | > current_lr: 5e-06 | > step_time: 0.1519 (0.13687717123944895) | > loader_time: 0.0047 (0.004671422257062865)  --> TIME: 2025-02-26 11:19:11 -- STEP: 35784/36938 -- GLOBAL_STEP: 1217800 | > loss_text_ce: 0.03207043185830116 (0.03158937904481906) | > loss_mel_ce: 2.203857421875 (nan) | > loss: 0.00887272972613573 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13687934901872642) | > loader_time: 0.0046 (0.004671401647601454)  --> TIME: 2025-02-26 11:19:21 -- STEP: 35834/36938 -- GLOBAL_STEP: 1217850 | > loss_text_ce: 0.03324936702847481 (0.03159024521635231) | > loss_mel_ce: 0.1942342072725296 (nan) | > loss: 0.0009027126361615956 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13688816500340834) | > loader_time: 0.0046 (0.004671441242552091)  --> TIME: 2025-02-26 11:19:31 -- STEP: 35884/36938 -- GLOBAL_STEP: 1217900 | > loss_text_ce: 0.02837223745882511 (0.031589430324153685) | > loss_mel_ce: 3.2019128799438477 (nan) | > loss: 0.012818592600524426 (nan) | > current_lr: 5e-06 | > step_time: 0.1357 (0.1368974491301063) | > loader_time: 0.0046 (0.00467152966130554)  --> TIME: 2025-02-26 11:19:39 -- STEP: 35934/36938 -- GLOBAL_STEP: 1217950 | > loss_text_ce: 0.028898583725094795 (0.03158795948691519) | > loss_mel_ce: 1.4334920644760132 (nan) | > loss: 0.005803138017654419 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13688890720479335) | > loader_time: 0.0046 (0.004671472045292084)  --> TIME: 2025-02-26 11:19:49 -- STEP: 35984/36938 -- GLOBAL_STEP: 1218000 | > loss_text_ce: 0.023828910663723946 (0.0315884733986576) | > loss_mel_ce: 2.5458984375 (nan) | > loss: 0.010197331197559834 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13688988408521857) | > loader_time: 0.0045 (0.004671438832768576)  --> TIME: 2025-02-26 11:19:58 -- STEP: 36034/36938 -- GLOBAL_STEP: 1218050 | > loss_text_ce: 0.02782449498772621 (0.031589816656723224) | > loss_mel_ce: 3.182316780090332 (nan) | > loss: 0.012738656252622604 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.1368872840006068) | > loader_time: 0.0047 (0.0046713781878191165)  --> TIME: 2025-02-26 11:20:07 -- STEP: 36084/36938 -- GLOBAL_STEP: 1218100 | > loss_text_ce: 0.03237586095929146 (0.03158792920722967) | > loss_mel_ce: 3.0872092247009277 (nan) | > loss: 0.012379306368529797 (nan) | > current_lr: 5e-06 | > step_time: 0.1812 (0.13688443926376845) | > loader_time: 0.0048 (0.004671323723598008)  --> TIME: 2025-02-26 11:20:16 -- STEP: 36134/36938 -- GLOBAL_STEP: 1218150 | > loss_text_ce: 0.03458545356988907 (0.03158838020710248) | > loss_mel_ce: 2.6771786212921143 (nan) | > loss: 0.01076096948236227 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.1368846857746367) | > loader_time: 0.0049 (0.004671282751622127)  --> TIME: 2025-02-26 11:20:26 -- STEP: 36184/36938 -- GLOBAL_STEP: 1218200 | > loss_text_ce: 0.016471637412905693 (0.03158990625190279) | > loss_mel_ce: 1.3862539529800415 (nan) | > loss: 0.005566371604800224 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1368907607289615) | > loader_time: 0.0043 (0.00467125030052024)  --> TIME: 2025-02-26 11:20:35 -- STEP: 36234/36938 -- GLOBAL_STEP: 1218250 | > loss_text_ce: 0.02989957667887211 (0.03159026283761892) | > loss_mel_ce: 2.7497575283050537 (nan) | > loss: 0.011030386202037334 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.1368801217735319) | > loader_time: 0.0046 (0.0046712097995571114)  --> TIME: 2025-02-26 11:20:44 -- STEP: 36284/36938 -- GLOBAL_STEP: 1218300 | > loss_text_ce: 0.031804587692022324 (0.03158862384346181) | > loss_mel_ce: 2.7299015522003174 (nan) | > loss: 0.01095915213227272 (nan) | > current_lr: 5e-06 | > step_time: 0.1818 (0.13688100441565895) | > loader_time: 0.0048 (0.004671160519786759)  --> TIME: 2025-02-26 11:20:54 -- STEP: 36334/36938 -- GLOBAL_STEP: 1218350 | > loss_text_ce: 0.03427483141422272 (0.03158895529543049) | > loss_mel_ce: 3.0254125595092773 (nan) | > loss: 0.012141617015004158 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13688136209459706) | > loader_time: 0.0047 (0.00467112375131209)  --> TIME: 2025-02-26 11:21:03 -- STEP: 36384/36938 -- GLOBAL_STEP: 1218400 | > loss_text_ce: 0.047011930495500565 (0.031587527761099625) | > loss_mel_ce: 2.438122272491455 (nan) | > loss: 0.009861644357442856 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13687401445109104) | > loader_time: 0.0047 (0.004671016614638426)  --> TIME: 2025-02-26 11:21:12 -- STEP: 36434/36938 -- GLOBAL_STEP: 1218450 | > loss_text_ce: 0.03286057338118553 (0.031586388008008955) | > loss_mel_ce: 2.42537260055542 (nan) | > loss: 0.00975489430129528 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1368790416625396) | > loader_time: 0.0045 (0.004671004631650601)  --> TIME: 2025-02-26 11:21:21 -- STEP: 36484/36938 -- GLOBAL_STEP: 1218500 | > loss_text_ce: 0.019956648349761963 (0.03158433313690867) | > loss_mel_ce: 1.5677374601364136 (nan) | > loss: 0.006300373934209347 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.1368780153883278) | > loader_time: 0.0045 (0.004670943238601639)  --> TIME: 2025-02-26 11:21:31 -- STEP: 36534/36938 -- GLOBAL_STEP: 1218550 | > loss_text_ce: 0.02858024090528488 (0.03158487288181429) | > loss_mel_ce: 2.780386209487915 (nan) | > loss: 0.01114669255912304 (nan) | > current_lr: 5e-06 | > step_time: 0.1887 (0.13688033417527237) | > loader_time: 0.0049 (0.004670898504638851)  --> TIME: 2025-02-26 11:21:40 -- STEP: 36584/36938 -- GLOBAL_STEP: 1218600 | > loss_text_ce: 0.032265063375234604 (0.03158553388707107) | > loss_mel_ce: 2.5356249809265137 (nan) | > loss: 0.010190039873123169 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13688271878690067) | > loader_time: 0.0047 (0.004670845948708765)  --> TIME: 2025-02-26 11:21:49 -- STEP: 36634/36938 -- GLOBAL_STEP: 1218650 | > loss_text_ce: 0.024956481531262398 (0.03158613495513582) | > loss_mel_ce: 2.151822805404663 (nan) | > loss: 0.008638013154268265 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13688272296676257) | > loader_time: 0.0045 (0.004670776270190442)  --> TIME: 2025-02-26 11:21:58 -- STEP: 36684/36938 -- GLOBAL_STEP: 1218700 | > loss_text_ce: 0.02928566001355648 (0.03158469170192766) | > loss_mel_ce: 2.7113471031188965 (nan) | > loss: 0.010875527746975422 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13687659437556873) | > loader_time: 0.0047 (0.004670685542057451)  --> TIME: 2025-02-26 11:22:08 -- STEP: 36734/36938 -- GLOBAL_STEP: 1218750 | > loss_text_ce: 0.02488165721297264 (0.031585013001524574) | > loss_mel_ce: 2.2997000217437744 (nan) | > loss: 0.00922453124076128 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13688200844932827) | > loader_time: 0.0045 (0.004670630063671571)  --> TIME: 2025-02-26 11:22:17 -- STEP: 36784/36938 -- GLOBAL_STEP: 1218800 | > loss_text_ce: 0.032398853451013565 (0.03158467098218159) | > loss_mel_ce: 2.650341272354126 (nan) | > loss: 0.010645794682204723 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.1368854985697578) | > loader_time: 0.0046 (0.004670619225699069)  --> TIME: 2025-02-26 11:22:27 -- STEP: 36834/36938 -- GLOBAL_STEP: 1218850 | > loss_text_ce: 0.037224818021059036 (0.03158547912078339) | > loss_mel_ce: 3.2972190380096436 (nan) | > loss: 0.013231921009719372 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13688488379294825) | > loader_time: 0.0046 (0.004670535184055857)  --> TIME: 2025-02-26 11:22:36 -- STEP: 36884/36938 -- GLOBAL_STEP: 1218900 | > loss_text_ce: 0.037460483610630035 (0.031585186917892485) | > loss_mel_ce: 2.6639552116394043 (nan) | > loss: 0.01071990467607975 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13688432455399036) | > loader_time: 0.0046 (0.0046704585647107414)  --> TIME: 2025-02-26 11:22:45 -- STEP: 36934/36938 -- GLOBAL_STEP: 1218950 | > loss_text_ce: 0.030388589948415756 (0.03158435547497057) | > loss_mel_ce: 3.0016322135925293 (nan) | > loss: 0.012031828984618187 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.13688828129602384) | > loader_time: 0.0045 (0.0046703555635983374)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.02061292086497392 (+8.529837768856893e-05) | > avg_loss_text_ce: 0.03072887129786581 (-0.00015741839341007827) | > avg_loss_mel_ce: 2.4668190871724978 (-0.0037279748680583147) | > avg_loss: 2.497547955796272 (-0.00388540076737387) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_1218954.pth  > EPOCH: 34/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-26 11:23:11)   --> TIME: 2025-02-26 11:23:20 -- STEP: 46/36938 -- GLOBAL_STEP: 1219000 | > loss_text_ce: 0.020017286762595177 (0.031042874142851524) | > loss_mel_ce: 1.588442087173462 (nan) | > loss: 0.006382775492966175 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13625724937604827) | > loader_time: 0.0045 (0.004868471104165782)  --> TIME: 2025-02-26 11:23:29 -- STEP: 96/36938 -- GLOBAL_STEP: 1219050 | > loss_text_ce: 0.03460023179650307 (0.031083104161856088) | > loss_mel_ce: 3.019265651702881 (nan) | > loss: 0.012118516489863396 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13300550480683646) | > loader_time: 0.0048 (0.004842850069204965)  --> TIME: 2025-02-26 11:23:38 -- STEP: 146/36938 -- GLOBAL_STEP: 1219100 | > loss_text_ce: 0.01962476596236229 (0.03141185789279743) | > loss_mel_ce: 0.03813032805919647 (nan) | > loss: 0.00022918688773643225 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.1331609340563213) | > loader_time: 0.0046 (0.004843199089781878)  --> TIME: 2025-02-26 11:23:47 -- STEP: 196/36938 -- GLOBAL_STEP: 1219150 | > loss_text_ce: 0.028227416798472404 (0.031780904082923514) | > loss_mel_ce: 2.366187334060669 (nan) | > loss: 0.009501646272838116 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.1325182853912821) | > loader_time: 0.0046 (0.004851074851289088)  --> TIME: 2025-02-26 11:23:56 -- STEP: 246/36938 -- GLOBAL_STEP: 1219200 | > loss_text_ce: 0.027877042070031166 (0.03213078633132505) | > loss_mel_ce: 3.036085605621338 (nan) | > loss: 0.012158582918345928 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.13326584129798702) | > loader_time: 0.0049 (0.004840591089512276)  --> TIME: 2025-02-26 11:24:06 -- STEP: 296/36938 -- GLOBAL_STEP: 1219250 | > loss_text_ce: 0.03976159542798996 (0.03204245139165103) | > loss_mel_ce: 3.149923086166382 (nan) | > loss: 0.012657479383051395 (nan) | > current_lr: 5e-06 | > step_time: 0.1704 (0.13452221332369624) | > loader_time: 0.0048 (0.00483270674138456)  --> TIME: 2025-02-26 11:24:15 -- STEP: 346/36938 -- GLOBAL_STEP: 1219300 | > loss_text_ce: 0.018716448917984962 (0.03192443124993001) | > loss_mel_ce: 2.372283458709717 (nan) | > loss: 0.009488094598054886 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.1346477511301205) | > loader_time: 0.0047 (0.0048176980432058335)  --> TIME: 2025-02-26 11:24:24 -- STEP: 396/36938 -- GLOBAL_STEP: 1219350 | > loss_text_ce: 0.02254636026918888 (0.03184187335592479) | > loss_mel_ce: 2.400235176086426 (nan) | > loss: 0.009614212438464165 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13473016685909692) | > loader_time: 0.0046 (0.004802722521502564)  --> TIME: 2025-02-26 11:24:34 -- STEP: 446/36938 -- GLOBAL_STEP: 1219400 | > loss_text_ce: 0.026651794090867043 (0.03178545700661803) | > loss_mel_ce: 0.203125 (nan) | > loss: 0.0009118127636611462 (nan) | > current_lr: 5e-06 | > step_time: 0.1599 (0.1353205105648983) | > loader_time: 0.0047 (0.004791461298818546)  --> TIME: 2025-02-26 11:24:43 -- STEP: 496/36938 -- GLOBAL_STEP: 1219450 | > loss_text_ce: 0.03226031735539436 (0.031707258669719546) | > loss_mel_ce: 3.1584630012512207 (nan) | > loss: 0.012661601416766644 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13534347809130162) | > loader_time: 0.0046 (0.0047851060667345605)  --> TIME: 2025-02-26 11:24:52 -- STEP: 546/36938 -- GLOBAL_STEP: 1219500 | > loss_text_ce: 0.031474556773900986 (0.031660869410568504) | > loss_mel_ce: 2.8562638759613037 (nan) | > loss: 0.011459280736744404 (nan) | > current_lr: 5e-06 | > step_time: 0.1889 (0.13536774893820083) | > loader_time: 0.0049 (0.004786488337394518)  --> TIME: 2025-02-26 11:25:02 -- STEP: 596/36938 -- GLOBAL_STEP: 1219550 | > loss_text_ce: 0.030262622982263565 (0.031674989742920656) | > loss_mel_ce: 0.5923827886581421 (nan) | > loss: 0.002470815321430564 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13568668757509073) | > loader_time: 0.0046 (0.004780986965102637)  --> TIME: 2025-02-26 11:25:11 -- STEP: 646/36938 -- GLOBAL_STEP: 1219600 | > loss_text_ce: 0.03241553157567978 (0.03175581665565442) | > loss_mel_ce: 1.615309476852417 (nan) | > loss: 0.00653859181329608 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13537451726364286) | > loader_time: 0.0047 (0.004778732087220938)  --> TIME: 2025-02-26 11:25:20 -- STEP: 696/36938 -- GLOBAL_STEP: 1219650 | > loss_text_ce: 0.03947950899600983 (0.03182122712665843) | > loss_mel_ce: 2.11586332321167 (nan) | > loss: 0.008552948012948036 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13520948434698182) | > loader_time: 0.0048 (0.004776234256810157)  --> TIME: 2025-02-26 11:25:29 -- STEP: 746/36938 -- GLOBAL_STEP: 1219700 | > loss_text_ce: 0.030277129262685776 (0.03177086622941107) | > loss_mel_ce: 2.543410062789917 (nan) | > loss: 0.010213044472038746 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.1354007187222029) | > loader_time: 0.0046 (0.004770317921370025)  --> TIME: 2025-02-26 11:25:39 -- STEP: 796/36938 -- GLOBAL_STEP: 1219750 | > loss_text_ce: 0.027819953858852386 (0.03171850131220888) | > loss_mel_ce: 4.768750190734863 (nan) | > loss: 0.0190340094268322 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1352907068765344) | > loader_time: 0.0045 (0.004762221820390398)  --> TIME: 2025-02-26 11:25:48 -- STEP: 846/36938 -- GLOBAL_STEP: 1219800 | > loss_text_ce: 0.036112911999225616 (0.03171588174985544) | > loss_mel_ce: 2.599268913269043 (nan) | > loss: 0.01045786589384079 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1356620303846139) | > loader_time: 0.0045 (0.004758040110270179)  --> TIME: 2025-02-26 11:25:57 -- STEP: 896/36938 -- GLOBAL_STEP: 1219850 | > loss_text_ce: 0.03483450412750244 (0.03167368481912746) | > loss_mel_ce: 2.6250386238098145 (nan) | > loss: 0.010555052198469639 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13549686196659308) | > loader_time: 0.0045 (0.0047556318874869994)  --> TIME: 2025-02-26 11:26:07 -- STEP: 946/36938 -- GLOBAL_STEP: 1219900 | > loss_text_ce: 0.030861541628837585 (0.03168585677444146) | > loss_mel_ce: 2.926189661026001 (nan) | > loss: 0.011734331026673317 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.13582195136814273) | > loader_time: 0.0047 (0.004753726985439435)  --> TIME: 2025-02-26 11:26:16 -- STEP: 996/36938 -- GLOBAL_STEP: 1219950 | > loss_text_ce: 0.04005100578069687 (0.03168090965774612) | > loss_mel_ce: 2.4875032901763916 (nan) | > loss: 0.01002997811883688 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13594845619546395) | > loader_time: 0.0045 (0.004748673563501439)  --> TIME: 2025-02-26 11:26:26 -- STEP: 1046/36938 -- GLOBAL_STEP: 1220000 | > loss_text_ce: 0.029496563598513603 (0.03167191598240292) | > loss_mel_ce: 2.813843011856079 (nan) | > loss: 0.011283094994723797 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13589479795605489) | > loader_time: 0.0047 (0.00474502411216896) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1220000.pth  --> TIME: 2025-02-26 11:26:41 -- STEP: 1096/36938 -- GLOBAL_STEP: 1220050 | > loss_text_ce: 0.02317032590508461 (0.031695896720540864) | > loss_mel_ce: 2.4153759479522705 (nan) | > loss: 0.009676771238446236 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13582951678846913) | > loader_time: 0.0046 (0.004742612151333882)  --> TIME: 2025-02-26 11:26:50 -- STEP: 1146/36938 -- GLOBAL_STEP: 1220100 | > loss_text_ce: 0.030913302674889565 (0.031726737599332276) | > loss_mel_ce: 2.9879794120788574 (nan) | > loss: 0.011979733593761921 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13580779671460996) | > loader_time: 0.0047 (0.004744400320668909)  --> TIME: 2025-02-26 11:26:59 -- STEP: 1196/36938 -- GLOBAL_STEP: 1220150 | > loss_text_ce: 0.029076602309942245 (0.03170708274970258) | > loss_mel_ce: 2.496433734893799 (nan) | > loss: 0.010021867230534554 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.13583881001807407) | > loader_time: 0.0049 (0.004744032952299084)  --> TIME: 2025-02-26 11:27:09 -- STEP: 1246/36938 -- GLOBAL_STEP: 1220200 | > loss_text_ce: 0.025431761518120766 (0.03173893643094651) | > loss_mel_ce: 2.8644354343414307 (nan) | > loss: 0.011467727832496166 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1359314300466696) | > loader_time: 0.0046 (0.004741984998050892)  --> TIME: 2025-02-26 11:27:18 -- STEP: 1296/36938 -- GLOBAL_STEP: 1220250 | > loss_text_ce: 0.03166862204670906 (0.03169441631460493) | > loss_mel_ce: 0.01839192770421505 (nan) | > loss: 0.00019865298236254603 (nan) | > current_lr: 5e-06 | > step_time: 0.1357 (0.1357502352308346) | > loader_time: 0.0047 (0.004738847414652507)  --> TIME: 2025-02-26 11:27:27 -- STEP: 1346/36938 -- GLOBAL_STEP: 1220300 | > loss_text_ce: 0.0335875041782856 (0.031644353350378095) | > loss_mel_ce: 2.6330373287200928 (nan) | > loss: 0.010581845417618752 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13572773536636804) | > loader_time: 0.0045 (0.004736803261698801)  --> TIME: 2025-02-26 11:27:36 -- STEP: 1396/36938 -- GLOBAL_STEP: 1220350 | > loss_text_ce: 0.032984375953674316 (0.031641663224930006) | > loss_mel_ce: 2.5459163188934326 (nan) | > loss: 0.010233733803033829 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.13575428826761127) | > loader_time: 0.0046 (0.004735625917385512)  --> TIME: 2025-02-26 11:27:46 -- STEP: 1446/36938 -- GLOBAL_STEP: 1220400 | > loss_text_ce: 0.03811431676149368 (0.03166846603021167) | > loss_mel_ce: 2.9965972900390625 (nan) | > loss: 0.012042506597936153 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13596263185741184) | > loader_time: 0.0046 (0.004735559678506397)  --> TIME: 2025-02-26 11:27:55 -- STEP: 1496/36938 -- GLOBAL_STEP: 1220450 | > loss_text_ce: 0.03611777350306511 (0.03167728399658663) | > loss_mel_ce: 1.6310781240463257 (nan) | > loss: 0.006615857128053904 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13608931012969608) | > loader_time: 0.0052 (0.004736371378210145)  --> TIME: 2025-02-26 11:28:05 -- STEP: 1546/36938 -- GLOBAL_STEP: 1220500 | > loss_text_ce: 0.027815185487270355 (0.03170639633370942) | > loss_mel_ce: 2.053973913192749 (nan) | > loss: 0.008261067792773247 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13612490653374834) | > loader_time: 0.0046 (0.004734340028651787)  --> TIME: 2025-02-26 11:28:14 -- STEP: 1596/36938 -- GLOBAL_STEP: 1220550 | > loss_text_ce: 0.034373197704553604 (0.03167649880074953) | > loss_mel_ce: 2.3730735778808594 (nan) | > loss: 0.009553360752761364 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13623090196671653) | > loader_time: 0.0046 (0.004732977029374975)  --> TIME: 2025-02-26 11:28:24 -- STEP: 1646/36938 -- GLOBAL_STEP: 1220600 | > loss_text_ce: 0.03607810661196709 (0.03170443746518174) | > loss_mel_ce: 2.5812294483184814 (nan) | > loss: 0.010386141948401928 (nan) | > current_lr: 5e-06 | > step_time: 0.1816 (0.13636070135007675) | > loader_time: 0.0049 (0.004732624438669392)  --> TIME: 2025-02-26 11:28:33 -- STEP: 1696/36938 -- GLOBAL_STEP: 1220650 | > loss_text_ce: 0.03362703323364258 (0.03168049742103079) | > loss_mel_ce: 0.6143013834953308 (nan) | > loss: 0.0025711446069180965 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13634299922664242) | > loader_time: 0.0046 (0.00473094366069111)  --> TIME: 2025-02-26 11:28:42 -- STEP: 1746/36938 -- GLOBAL_STEP: 1220700 | > loss_text_ce: 0.03407701477408409 (0.031689742028516904) | > loss_mel_ce: 2.702550172805786 (nan) | > loss: 0.010859631933271885 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.136335377840652) | > loader_time: 0.0045 (0.0047280867621388055)  --> TIME: 2025-02-26 11:28:52 -- STEP: 1796/36938 -- GLOBAL_STEP: 1220750 | > loss_text_ce: 0.02473483793437481 (0.031689067373263796) | > loss_mel_ce: 2.3206655979156494 (nan) | > loss: 0.00930714514106512 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.136512455802187) | > loader_time: 0.0048 (0.0047274315543058025)  --> TIME: 2025-02-26 11:29:02 -- STEP: 1846/36938 -- GLOBAL_STEP: 1220800 | > loss_text_ce: 0.025499723851680756 (0.03167359196335752) | > loss_mel_ce: 0.006293105892837048 (nan) | > loss: 0.00012616203457582742 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13659019821514168) | > loader_time: 0.0044 (0.004724396657065487)  --> TIME: 2025-02-26 11:29:11 -- STEP: 1896/36938 -- GLOBAL_STEP: 1220850 | > loss_text_ce: 0.028676146641373634 (0.03170101171288686) | > loss_mel_ce: 1.238134741783142 (nan) | > loss: 0.005027027800679207 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13659817168984245) | > loader_time: 0.0046 (0.004722115108232467)  --> TIME: 2025-02-26 11:29:21 -- STEP: 1946/36938 -- GLOBAL_STEP: 1220900 | > loss_text_ce: 0.043001558631658554 (0.031684594024030205) | > loss_mel_ce: 3.37263822555542 (nan) | > loss: 0.01355412695556879 (nan) | > current_lr: 5e-06 | > step_time: 0.1902 (0.1367624606037435) | > loader_time: 0.0048 (0.004718960985495409)  --> TIME: 2025-02-26 11:29:31 -- STEP: 1996/36938 -- GLOBAL_STEP: 1220950 | > loss_text_ce: 0.0371818020939827 (0.03166674305955492) | > loss_mel_ce: 1.6642768383026123 (nan) | > loss: 0.00675182044506073 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13698325546566625) | > loader_time: 0.0045 (0.00471608779712288)  --> TIME: 2025-02-26 11:29:40 -- STEP: 2046/36938 -- GLOBAL_STEP: 1221000 | > loss_text_ce: 0.030859947204589844 (0.03165033994969036) | > loss_mel_ce: 0.2495269775390625 (nan) | > loss: 0.001112646539695561 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13691851924940998) | > loader_time: 0.0044 (0.004712326552278261)  --> TIME: 2025-02-26 11:29:49 -- STEP: 2096/36938 -- GLOBAL_STEP: 1221050 | > loss_text_ce: 0.028029778972268105 (0.031636197432761894) | > loss_mel_ce: 2.557929515838623 (nan) | > loss: 0.01026174332946539 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13689461885062798) | > loader_time: 0.0045 (0.004709033338168204)  --> TIME: 2025-02-26 11:29:59 -- STEP: 2146/36938 -- GLOBAL_STEP: 1221100 | > loss_text_ce: 0.02994990348815918 (0.03164376088314778) | > loss_mel_ce: 2.637181520462036 (nan) | > loss: 0.010583855211734772 (nan) | > current_lr: 5e-06 | > step_time: 0.174 (0.13694262126840845) | > loader_time: 0.0047 (0.004706379862889335)  --> TIME: 2025-02-26 11:30:08 -- STEP: 2196/36938 -- GLOBAL_STEP: 1221150 | > loss_text_ce: 0.034978125244379044 (0.031622566184358075) | > loss_mel_ce: 2.1829891204833984 (nan) | > loss: 0.008801457472145557 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13704455148544042) | > loader_time: 0.0045 (0.004703685234153212)  --> TIME: 2025-02-26 11:30:17 -- STEP: 2246/36938 -- GLOBAL_STEP: 1221200 | > loss_text_ce: 0.037273719906806946 (0.03162512138240429) | > loss_mel_ce: 2.389721393585205 (nan) | > loss: 0.009630933403968811 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.1370702493647222) | > loader_time: 0.0045 (0.00470145854797092)  --> TIME: 2025-02-26 11:30:27 -- STEP: 2296/36938 -- GLOBAL_STEP: 1221250 | > loss_text_ce: 0.03446827456355095 (0.031624468063727935) | > loss_mel_ce: 2.479917049407959 (nan) | > loss: 0.009977719746530056 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1370664631448142) | > loader_time: 0.0045 (0.004698757095204001)  --> TIME: 2025-02-26 11:30:37 -- STEP: 2346/36938 -- GLOBAL_STEP: 1221300 | > loss_text_ce: 0.022189440205693245 (0.03161302364517863) | > loss_mel_ce: 3.1188762187957764 (nan) | > loss: 0.012464546598494053 (nan) | > current_lr: 5e-06 | > step_time: 0.1516 (0.1372056329545056) | > loader_time: 0.0046 (0.004697411261556099)  --> TIME: 2025-02-26 11:30:46 -- STEP: 2396/36938 -- GLOBAL_STEP: 1221350 | > loss_text_ce: 0.025147873908281326 (0.03160133283625733) | > loss_mel_ce: 2.4784672260284424 (nan) | > loss: 0.009934981353580952 (nan) | > current_lr: 5e-06 | > step_time: 0.1807 (0.1372295603131214) | > loader_time: 0.0047 (0.004695136379917003)  --> TIME: 2025-02-26 11:30:55 -- STEP: 2446/36938 -- GLOBAL_STEP: 1221400 | > loss_text_ce: 0.03310754522681236 (0.03160230848231549) | > loss_mel_ce: 2.917004108428955 (nan) | > loss: 0.011706792749464512 (nan) | > current_lr: 5e-06 | > step_time: 0.178 (0.13723155610013582) | > loader_time: 0.0047 (0.004692653408814728)  --> TIME: 2025-02-26 11:31:05 -- STEP: 2496/36938 -- GLOBAL_STEP: 1221450 | > loss_text_ce: 0.027228590101003647 (0.03162363027401556) | > loss_mel_ce: 0.00333404541015625 (nan) | > loss: 0.00012128030357416719 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13736382184120335) | > loader_time: 0.0046 (0.00469267568909205)  --> TIME: 2025-02-26 11:31:15 -- STEP: 2546/36938 -- GLOBAL_STEP: 1221500 | > loss_text_ce: 0.0367826446890831 (0.03163953722971847) | > loss_mel_ce: 1.5262945890426636 (nan) | > loss: 0.006202687509357929 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13746925628943796) | > loader_time: 0.0044 (0.004691373150084587)  --> TIME: 2025-02-26 11:31:24 -- STEP: 2596/36938 -- GLOBAL_STEP: 1221550 | > loss_text_ce: 0.023990845307707787 (0.031630904390799526) | > loss_mel_ce: 2.365849733352661 (nan) | > loss: 0.00948349479585886 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13748564356464452) | > loader_time: 0.0046 (0.004689522810819888)  --> TIME: 2025-02-26 11:31:33 -- STEP: 2646/36938 -- GLOBAL_STEP: 1221600 | > loss_text_ce: 0.029784079641103745 (0.03163913204753526) | > loss_mel_ce: 1.981851577758789 (nan) | > loss: 0.007982681505382061 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.1374675959956889) | > loader_time: 0.0047 (0.004687990911603423)  --> TIME: 2025-02-26 11:31:43 -- STEP: 2696/36938 -- GLOBAL_STEP: 1221650 | > loss_text_ce: 0.02996981516480446 (0.03163287723547494) | > loss_mel_ce: 2.2023837566375732 (nan) | > loss: 0.008858546614646912 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13745385868259286) | > loader_time: 0.0046 (0.0046861372468025835)  --> TIME: 2025-02-26 11:31:52 -- STEP: 2746/36938 -- GLOBAL_STEP: 1221700 | > loss_text_ce: 0.0287463441491127 (0.031657009118271226) | > loss_mel_ce: 4.620404243469238 (nan) | > loss: 0.01844901032745838 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13742918674735727) | > loader_time: 0.0045 (0.004684621803272579)  --> TIME: 2025-02-26 11:32:01 -- STEP: 2796/36938 -- GLOBAL_STEP: 1221750 | > loss_text_ce: 0.028085047379136086 (0.03164843124975701) | > loss_mel_ce: 2.3893349170684814 (nan) | > loss: 0.009592936374247074 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1374089041663514) | > loader_time: 0.0044 (0.004683303474868315)  --> TIME: 2025-02-26 11:32:10 -- STEP: 2846/36938 -- GLOBAL_STEP: 1221800 | > loss_text_ce: 0.025522079318761826 (0.03162411441902148) | > loss_mel_ce: 1.319236397743225 (nan) | > loss: 0.00533634377643466 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1373810856927055) | > loader_time: 0.0045 (0.004681506126763027)  --> TIME: 2025-02-26 11:32:20 -- STEP: 2896/36938 -- GLOBAL_STEP: 1221850 | > loss_text_ce: 0.023120643571019173 (0.031609522543773) | > loss_mel_ce: 2.6982879638671875 (nan) | > loss: 0.010799241252243519 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13736191087335534) | > loader_time: 0.0045 (0.004680115955969245)  --> TIME: 2025-02-26 11:32:29 -- STEP: 2946/36938 -- GLOBAL_STEP: 1221900 | > loss_text_ce: 0.028690272942185402 (0.03161870698221361) | > loss_mel_ce: 3.5030853748321533 (nan) | > loss: 0.014014983549714088 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13738251363672718) | > loader_time: 0.005 (0.0046789799105611185)  --> TIME: 2025-02-26 11:32:38 -- STEP: 2996/36938 -- GLOBAL_STEP: 1221950 | > loss_text_ce: 0.03792518749833107 (0.03162583029823996) | > loss_mel_ce: 2.5493905544281006 (nan) | > loss: 0.010267126373946667 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13734133611215624) | > loader_time: 0.0045 (0.004677354254932683)  --> TIME: 2025-02-26 11:32:48 -- STEP: 3046/36938 -- GLOBAL_STEP: 1222000 | > loss_text_ce: 0.03078169748187065 (0.031589200036833846) | > loss_mel_ce: 2.8169355392456055 (nan) | > loss: 0.011300466023385525 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13738276366724128) | > loader_time: 0.0045 (0.00467614546780201)  --> TIME: 2025-02-26 11:32:57 -- STEP: 3096/36938 -- GLOBAL_STEP: 1222050 | > loss_text_ce: 0.03496269881725311 (0.03158506822804813) | > loss_mel_ce: 2.585871696472168 (nan) | > loss: 0.010400136932730675 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.13746056322595557) | > loader_time: 0.0049 (0.004675436158512917)  --> TIME: 2025-02-26 11:33:07 -- STEP: 3146/36938 -- GLOBAL_STEP: 1222100 | > loss_text_ce: 0.03687404468655586 (0.031569834288508104) | > loss_mel_ce: 2.146967649459839 (nan) | > loss: 0.008666039444506168 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13747363520972078) | > loader_time: 0.0047 (0.004675241768625386)  --> TIME: 2025-02-26 11:33:16 -- STEP: 3196/36938 -- GLOBAL_STEP: 1222150 | > loss_text_ce: 0.030894290655851364 (0.0315696809968873) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1079 (0.13736851328753116) | > loader_time: 0.0057 (0.0046746890446420645)  --> TIME: 2025-02-26 11:33:25 -- STEP: 3246/36938 -- GLOBAL_STEP: 1222200 | > loss_text_ce: 0.03861017897725105 (0.03156650932052406) | > loss_mel_ce: 0.00421142578125 (nan) | > loss: 0.0001699270069366321 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13737548646262898) | > loader_time: 0.0045 (0.004674563713449797)  --> TIME: 2025-02-26 11:33:35 -- STEP: 3296/36938 -- GLOBAL_STEP: 1222250 | > loss_text_ce: 0.04144304245710373 (0.03157670826151589) | > loss_mel_ce: 2.5269534587860107 (nan) | > loss: 0.010192050598561764 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13736131186913528) | > loader_time: 0.0045 (0.004673749859471929)  --> TIME: 2025-02-26 11:33:44 -- STEP: 3346/36938 -- GLOBAL_STEP: 1222300 | > loss_text_ce: 0.026952002197504044 (0.03160819254572865) | > loss_mel_ce: 2.550192356109619 (nan) | > loss: 0.010226763784885406 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13731583662540428) | > loader_time: 0.0046 (0.0046722249790928155)  --> TIME: 2025-02-26 11:33:53 -- STEP: 3396/36938 -- GLOBAL_STEP: 1222350 | > loss_text_ce: 0.044303927570581436 (0.03158749873531615) | > loss_mel_ce: 3.122457981109619 (nan) | > loss: 0.012566516175866127 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.1372185906195669) | > loader_time: 0.0045 (0.0046704144724968345)  --> TIME: 2025-02-26 11:34:02 -- STEP: 3446/36938 -- GLOBAL_STEP: 1222400 | > loss_text_ce: 0.03384726122021675 (0.03159029605769773) | > loss_mel_ce: 2.3887627124786377 (nan) | > loss: 0.009613532572984695 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.137300785032041) | > loader_time: 0.0045 (0.004669713821787513)  --> TIME: 2025-02-26 11:34:12 -- STEP: 3496/36938 -- GLOBAL_STEP: 1222450 | > loss_text_ce: 0.032738275825977325 (0.03159273090640195) | > loss_mel_ce: 3.222191572189331 (nan) | > loss: 0.012916388921439648 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.1373067750537968) | > loader_time: 0.0048 (0.004668523572402362)  --> TIME: 2025-02-26 11:34:21 -- STEP: 3546/36938 -- GLOBAL_STEP: 1222500 | > loss_text_ce: 0.028315469622612 (0.03159043996437783) | > loss_mel_ce: 2.5675864219665527 (nan) | > loss: 0.010301198810338974 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13724039842096744) | > loader_time: 0.0045 (0.004667186360539386)  --> TIME: 2025-02-26 11:34:31 -- STEP: 3596/36938 -- GLOBAL_STEP: 1222550 | > loss_text_ce: 0.033146679401397705 (0.03158713783499045) | > loss_mel_ce: 2.3940110206604004 (nan) | > loss: 0.00963157881051302 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13732394910893) | > loader_time: 0.0046 (0.0046689053531218175)  --> TIME: 2025-02-26 11:34:40 -- STEP: 3646/36938 -- GLOBAL_STEP: 1222600 | > loss_text_ce: 0.026614394038915634 (0.031567892418061634) | > loss_mel_ce: 2.3747169971466064 (nan) | > loss: 0.009529093280434608 (nan) | > current_lr: 5e-06 | > step_time: 0.1526 (0.13741466957203188) | > loader_time: 0.0046 (0.0046692933607284555)  --> TIME: 2025-02-26 11:34:50 -- STEP: 3696/36938 -- GLOBAL_STEP: 1222650 | > loss_text_ce: 0.03442817181348801 (0.03157110078823733) | > loss_mel_ce: 1.71146559715271 (nan) | > loss: 0.006928150076419115 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13747168961283435) | > loader_time: 0.0046 (0.004668975547278601)  --> TIME: 2025-02-26 11:34:59 -- STEP: 3746/36938 -- GLOBAL_STEP: 1222700 | > loss_text_ce: 0.023053867742419243 (0.031561925707686275) | > loss_mel_ce: 2.8440802097320557 (nan) | > loss: 0.011377517133951187 (nan) | > current_lr: 5e-06 | > step_time: 0.1784 (0.13735649640459566) | > loader_time: 0.0048 (0.004668574058239435)  --> TIME: 2025-02-26 11:35:08 -- STEP: 3796/36938 -- GLOBAL_STEP: 1222750 | > loss_text_ce: 0.02530997060239315 (0.031562636331856235) | > loss_mel_ce: 2.2419958114624023 (nan) | > loss: 0.008997245691716671 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13736848377704095) | > loader_time: 0.0045 (0.0046687436430421616)  --> TIME: 2025-02-26 11:35:18 -- STEP: 3846/36938 -- GLOBAL_STEP: 1222800 | > loss_text_ce: 0.052286047488451004 (0.03156453872184751) | > loss_mel_ce: 3.0044705867767334 (nan) | > loss: 0.012129987590014935 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13736605669022575) | > loader_time: 0.0045 (0.004668060839331158)  --> TIME: 2025-02-26 11:35:27 -- STEP: 3896/36938 -- GLOBAL_STEP: 1222850 | > loss_text_ce: 0.02725093625485897 (0.03155388415994756) | > loss_mel_ce: 3.5765624046325684 (nan) | > loss: 0.014300847426056862 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13736493399011013) | > loader_time: 0.0045 (0.004668479888590945)  --> TIME: 2025-02-26 11:35:36 -- STEP: 3946/36938 -- GLOBAL_STEP: 1222900 | > loss_text_ce: 0.04328456148505211 (0.03154548625940866) | > loss_mel_ce: 1.9235907793045044 (nan) | > loss: 0.007805061060935259 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13739339530739855) | > loader_time: 0.0045 (0.004668410937313617)  --> TIME: 2025-02-26 11:35:46 -- STEP: 3996/36938 -- GLOBAL_STEP: 1222950 | > loss_text_ce: 0.044010963290929794 (0.03156158488520694) | > loss_mel_ce: 2.6258506774902344 (nan) | > loss: 0.010594689287245274 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13734514308763315) | > loader_time: 0.0045 (0.004667391827156596)  --> TIME: 2025-02-26 11:35:55 -- STEP: 4046/36938 -- GLOBAL_STEP: 1223000 | > loss_text_ce: 0.026939140632748604 (0.03156981572073301) | > loss_mel_ce: 2.6911277770996094 (nan) | > loss: 0.010785980150103569 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13738113090771564) | > loader_time: 0.0044 (0.004667434725381925)  --> TIME: 2025-02-26 11:36:04 -- STEP: 4096/36938 -- GLOBAL_STEP: 1223050 | > loss_text_ce: 0.03905529901385307 (0.03157174427974492) | > loss_mel_ce: 2.9071247577667236 (nan) | > loss: 0.011691191233694553 (nan) | > current_lr: 5e-06 | > step_time: 0.1619 (0.13733958464581497) | > loader_time: 0.0046 (0.004667057015467437)  --> TIME: 2025-02-26 11:36:13 -- STEP: 4146/36938 -- GLOBAL_STEP: 1223100 | > loss_text_ce: 0.032473135739564896 (0.03154755109027958) | > loss_mel_ce: 2.686617851257324 (nan) | > loss: 0.010790044441819191 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.1372806925274469) | > loader_time: 0.0045 (0.0046672545017623366)  --> TIME: 2025-02-26 11:36:23 -- STEP: 4196/36938 -- GLOBAL_STEP: 1223150 | > loss_text_ce: 0.03690788149833679 (0.03153828400683482) | > loss_mel_ce: 2.8199853897094727 (nan) | > loss: 0.01133687887340784 (nan) | > current_lr: 5e-06 | > step_time: 0.1786 (0.1373067959020883) | > loader_time: 0.0048 (0.004667683711383774)  --> TIME: 2025-02-26 11:36:32 -- STEP: 4246/36938 -- GLOBAL_STEP: 1223200 | > loss_text_ce: 0.030301367864012718 (0.031523143817344654) | > loss_mel_ce: 2.4815473556518555 (nan) | > loss: 0.009967654012143612 (nan) | > current_lr: 5e-06 | > step_time: 0.157 (0.13731857284980717) | > loader_time: 0.0046 (0.0046681614344377224)  --> TIME: 2025-02-26 11:36:42 -- STEP: 4296/36938 -- GLOBAL_STEP: 1223250 | > loss_text_ce: 0.02028615027666092 (0.03151861096479104) | > loss_mel_ce: 2.1518406867980957 (nan) | > loss: 0.008619551546871662 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13735213515034814) | > loader_time: 0.0044 (0.00466762902128408)  --> TIME: 2025-02-26 11:36:51 -- STEP: 4346/36938 -- GLOBAL_STEP: 1223300 | > loss_text_ce: 0.034641727805137634 (0.0315242633892131) | > loss_mel_ce: 2.6028456687927246 (nan) | > loss: 0.010466220788657665 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.13736631307413202) | > loader_time: 0.0048 (0.00466685326923383)  --> TIME: 2025-02-26 11:37:01 -- STEP: 4396/36938 -- GLOBAL_STEP: 1223350 | > loss_text_ce: 0.03203752636909485 (0.031500605822246695) | > loss_mel_ce: 1.7027984857559204 (nan) | > loss: 0.0068842703476548195 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1374048090610208) | > loader_time: 0.0045 (0.004666658072606125)  --> TIME: 2025-02-26 11:37:10 -- STEP: 4446/36938 -- GLOBAL_STEP: 1223400 | > loss_text_ce: 0.028337130323052406 (0.03149340545580127) | > loss_mel_ce: 0.4596761167049408 (nan) | > loss: 0.001936560613103211 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.13735932650675484) | > loader_time: 0.0046 (0.004665791586182218)  --> TIME: 2025-02-26 11:37:19 -- STEP: 4496/36938 -- GLOBAL_STEP: 1223450 | > loss_text_ce: 0.02866532653570175 (0.03148389598594977) | > loss_mel_ce: 3.0328829288482666 (nan) | > loss: 0.012149001471698284 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.13739101116767544) | > loader_time: 0.0047 (0.004665357362333145)  --> TIME: 2025-02-26 11:37:29 -- STEP: 4546/36938 -- GLOBAL_STEP: 1223500 | > loss_text_ce: 0.03202765807509422 (0.0314932568736285) | > loss_mel_ce: 2.219195604324341 (nan) | > loss: 0.008933426812291145 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.13738028797555785) | > loader_time: 0.0046 (0.004664720179905625)  --> TIME: 2025-02-26 11:37:38 -- STEP: 4596/36938 -- GLOBAL_STEP: 1223550 | > loss_text_ce: 0.02482778951525688 (0.03147153350831794) | > loss_mel_ce: 2.7034103870391846 (nan) | > loss: 0.01082634273916483 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13741785951860055) | > loader_time: 0.0046 (0.004664125392704663)  --> TIME: 2025-02-26 11:37:48 -- STEP: 4646/36938 -- GLOBAL_STEP: 1223600 | > loss_text_ce: 0.028248798102140427 (0.03147711590120059) | > loss_mel_ce: 2.705096960067749 (nan) | > loss: 0.010846611112356186 (nan) | > current_lr: 5e-06 | > step_time: 0.1267 (0.13743638915334128) | > loader_time: 0.0045 (0.004663922229410507)  --> TIME: 2025-02-26 11:37:57 -- STEP: 4696/36938 -- GLOBAL_STEP: 1223650 | > loss_text_ce: 0.03003230318427086 (0.031472247586549) | > loss_mel_ce: 2.717343807220459 (nan) | > loss: 0.010902287438511848 (nan) | > current_lr: 5e-06 | > step_time: 0.1517 (0.1374554789959876) | > loader_time: 0.0046 (0.004663610224829424)  --> TIME: 2025-02-26 11:38:07 -- STEP: 4746/36938 -- GLOBAL_STEP: 1223700 | > loss_text_ce: 0.03772478178143501 (0.031467766709801434) | > loss_mel_ce: 3.242922067642212 (nan) | > loss: 0.013018440455198288 (nan) | > current_lr: 5e-06 | > step_time: 0.1567 (0.13750915968413085) | > loader_time: 0.0046 (0.004663838548495512)  --> TIME: 2025-02-26 11:38:17 -- STEP: 4796/36938 -- GLOBAL_STEP: 1223750 | > loss_text_ce: 0.02463904395699501 (0.03148226842076022) | > loss_mel_ce: 2.38946795463562 (nan) | > loss: 0.009579790756106377 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13756355481708515) | > loader_time: 0.0046 (0.004663470240808514)  --> TIME: 2025-02-26 11:38:27 -- STEP: 4846/36938 -- GLOBAL_STEP: 1223800 | > loss_text_ce: 0.039069585502147675 (0.03148134356288489) | > loss_mel_ce: 3.0761778354644775 (nan) | > loss: 0.012362093664705753 (nan) | > current_lr: 5e-06 | > step_time: 0.1938 (0.137628910662564) | > loader_time: 0.0048 (0.004663208964635564)  --> TIME: 2025-02-26 11:38:36 -- STEP: 4896/36938 -- GLOBAL_STEP: 1223850 | > loss_text_ce: 0.0351455993950367 (0.031468742871367764) | > loss_mel_ce: 2.795009136199951 (nan) | > loss: 0.0112307732924819 (nan) | > current_lr: 5e-06 | > step_time: 0.1625 (0.13765042319017287) | > loader_time: 0.0048 (0.004663095998218645)  --> TIME: 2025-02-26 11:38:45 -- STEP: 4946/36938 -- GLOBAL_STEP: 1223900 | > loss_text_ce: 0.02741275541484356 (0.03146529543653022) | > loss_mel_ce: 2.7551846504211426 (nan) | > loss: 0.011042053811252117 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13760289681184354) | > loader_time: 0.0045 (0.004663062105282961)  --> TIME: 2025-02-26 11:38:54 -- STEP: 4996/36938 -- GLOBAL_STEP: 1223950 | > loss_text_ce: 0.02728211134672165 (0.0314772313047241) | > loss_mel_ce: 2.61755108833313 (nan) | > loss: 0.010495370253920555 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.1375669383639999) | > loader_time: 0.0046 (0.004662807986104465)  --> TIME: 2025-02-26 11:39:04 -- STEP: 5046/36938 -- GLOBAL_STEP: 1224000 | > loss_text_ce: 0.04202903062105179 (0.03146977961123171) | > loss_mel_ce: 3.3125193119049072 (nan) | > loss: 0.013311700895428658 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1375599932774541) | > loader_time: 0.0046 (0.004663100991039305)  --> TIME: 2025-02-26 11:39:13 -- STEP: 5096/36938 -- GLOBAL_STEP: 1224050 | > loss_text_ce: 0.03532186150550842 (0.03147912553231562) | > loss_mel_ce: 2.9499597549438477 (nan) | > loss: 0.011846357025206089 (nan) | > current_lr: 5e-06 | > step_time: 0.1573 (0.13758508334728747) | > loader_time: 0.0047 (0.004663677520617218)  --> TIME: 2025-02-26 11:39:23 -- STEP: 5146/36938 -- GLOBAL_STEP: 1224100 | > loss_text_ce: 0.032958634197711945 (0.031489191986757814) | > loss_mel_ce: 3.133897066116333 (nan) | > loss: 0.012566888704895973 (nan) | > current_lr: 5e-06 | > step_time: 0.1784 (0.13758469220964611) | > loader_time: 0.0048 (0.004663530973159435)  --> TIME: 2025-02-26 11:39:32 -- STEP: 5196/36938 -- GLOBAL_STEP: 1224150 | > loss_text_ce: 0.02740338258445263 (0.03148209666879867) | > loss_mel_ce: 2.7409863471984863 (nan) | > loss: 0.01098567433655262 (nan) | > current_lr: 5e-06 | > step_time: 0.1444 (0.13753911042965947) | > loader_time: 0.0046 (0.004662574255255394)  --> TIME: 2025-02-26 11:39:41 -- STEP: 5246/36938 -- GLOBAL_STEP: 1224200 | > loss_text_ce: 0.033278875052928925 (0.03148580816660659) | > loss_mel_ce: 2.594882011413574 (nan) | > loss: 0.01042921096086502 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.1375201896906716) | > loader_time: 0.0045 (0.004661560558518336)  --> TIME: 2025-02-26 11:39:51 -- STEP: 5296/36938 -- GLOBAL_STEP: 1224250 | > loss_text_ce: 0.036704204976558685 (0.0314803538406815) | > loss_mel_ce: 2.490478515625 (nan) | > loss: 0.010028503835201263 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13755247528099196) | > loader_time: 0.0045 (0.004662396729532684)  --> TIME: 2025-02-26 11:40:00 -- STEP: 5346/36938 -- GLOBAL_STEP: 1224300 | > loss_text_ce: 0.022213714197278023 (0.03147480595983019) | > loss_mel_ce: 2.965744972229004 (nan) | > loss: 0.011856979690492153 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13756512355768732) | > loader_time: 0.0046 (0.004662731681207212)  --> TIME: 2025-02-26 11:40:09 -- STEP: 5396/36938 -- GLOBAL_STEP: 1224350 | > loss_text_ce: 0.021354619413614273 (0.03147010031512462) | > loss_mel_ce: 1.4296566247940063 (nan) | > loss: 0.005757981911301613 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.137551698400145) | > loader_time: 0.0045 (0.004661829141089436)  --> TIME: 2025-02-26 11:40:18 -- STEP: 5446/36938 -- GLOBAL_STEP: 1224400 | > loss_text_ce: 0.03294551745057106 (0.03147394504665552) | > loss_mel_ce: 2.734764337539673 (nan) | > loss: 0.010982976295053959 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13751888266390713) | > loader_time: 0.0045 (0.004661437259454132)  --> TIME: 2025-02-26 11:40:28 -- STEP: 5496/36938 -- GLOBAL_STEP: 1224450 | > loss_text_ce: 0.02391866408288479 (0.03147649303166074) | > loss_mel_ce: 2.8259811401367188 (nan) | > loss: 0.011309126392006874 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13752368669495968) | > loader_time: 0.0046 (0.004660635579274393)  --> TIME: 2025-02-26 11:40:37 -- STEP: 5546/36938 -- GLOBAL_STEP: 1224500 | > loss_text_ce: 0.017918594181537628 (0.03146312116409153) | > loss_mel_ce: 2.725583553314209 (nan) | > loss: 0.01088691409677267 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13753959451648545) | > loader_time: 0.0045 (0.00465998415735454)  --> TIME: 2025-02-26 11:40:47 -- STEP: 5596/36938 -- GLOBAL_STEP: 1224550 | > loss_text_ce: 0.035017360001802444 (0.0314593436838851) | > loss_mel_ce: 2.5619568824768066 (nan) | > loss: 0.010305453091859818 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13757252267124429) | > loader_time: 0.0045 (0.004659241271751797)  --> TIME: 2025-02-26 11:40:56 -- STEP: 5646/36938 -- GLOBAL_STEP: 1224600 | > loss_text_ce: 0.0368817001581192 (0.03145486755963641) | > loss_mel_ce: 3.2843892574310303 (nan) | > loss: 0.013179647736251354 (nan) | > current_lr: 5e-06 | > step_time: 0.1846 (0.13759574550624548) | > loader_time: 0.0048 (0.0046586969242508125)  --> TIME: 2025-02-26 11:41:06 -- STEP: 5696/36938 -- GLOBAL_STEP: 1224650 | > loss_text_ce: 0.03986120969057083 (0.031451605856837025) | > loss_mel_ce: 2.5697431564331055 (nan) | > loss: 0.010355573147535324 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13758136997564396) | > loader_time: 0.0046 (0.004658235969503276)  --> TIME: 2025-02-26 11:41:15 -- STEP: 5746/36938 -- GLOBAL_STEP: 1224700 | > loss_text_ce: 0.024384720250964165 (0.031455467365619695) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13761074697353032) | > loader_time: 0.0045 (0.004657895067947238)  --> TIME: 2025-02-26 11:41:25 -- STEP: 5796/36938 -- GLOBAL_STEP: 1224750 | > loss_text_ce: 0.02712276391685009 (0.0314455999275494) | > loss_mel_ce: 0.026824951171875 (nan) | > loss: 0.00021407825988717377 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13759593715990168) | > loader_time: 0.0046 (0.004658265061177742)  --> TIME: 2025-02-26 11:41:34 -- STEP: 5846/36938 -- GLOBAL_STEP: 1224800 | > loss_text_ce: 0.028490619733929634 (0.031447185691875386) | > loss_mel_ce: 0.775195300579071 (nan) | > loss: 0.0031892298720777035 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13759650879839316) | > loader_time: 0.0046 (0.004658155069341409)  --> TIME: 2025-02-26 11:41:43 -- STEP: 5896/36938 -- GLOBAL_STEP: 1224850 | > loss_text_ce: 0.03638147935271263 (0.0314365352555764) | > loss_mel_ce: 2.895186185836792 (nan) | > loss: 0.011633206158876419 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.13760633497742758) | > loader_time: 0.0047 (0.004658787108179351)  --> TIME: 2025-02-26 11:41:53 -- STEP: 5946/36938 -- GLOBAL_STEP: 1224900 | > loss_text_ce: 0.029471371322870255 (0.031440709258645394) | > loss_mel_ce: 0.09912109375 (nan) | > loss: 0.000510287587530911 (nan) | > current_lr: 5e-06 | > step_time: 0.1446 (0.1375767472132738) | > loader_time: 0.0047 (0.004658842904777032)  --> TIME: 2025-02-26 11:42:02 -- STEP: 5996/36938 -- GLOBAL_STEP: 1224950 | > loss_text_ce: 0.029971012845635414 (0.03143671164668093) | > loss_mel_ce: 3.005153179168701 (nan) | > loss: 0.012044144794344902 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13758343792184283) | > loader_time: 0.0046 (0.004658384669852946)  --> TIME: 2025-02-26 11:42:11 -- STEP: 6046/36938 -- GLOBAL_STEP: 1225000 | > loss_text_ce: 0.031929563730955124 (0.031425674787432474) | > loss_mel_ce: 2.8815956115722656 (nan) | > loss: 0.01156160794198513 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13752769229035827) | > loader_time: 0.0046 (0.00465831964015173)  --> TIME: 2025-02-26 11:42:20 -- STEP: 6096/36938 -- GLOBAL_STEP: 1225050 | > loss_text_ce: 0.0380549319088459 (0.03142572156220218) | > loss_mel_ce: 3.199087619781494 (nan) | > loss: 0.012845804914832115 (nan) | > current_lr: 5e-06 | > step_time: 0.1923 (0.13753054085678892) | > loader_time: 0.0049 (0.004658313717429103)  --> TIME: 2025-02-26 11:42:30 -- STEP: 6146/36938 -- GLOBAL_STEP: 1225100 | > loss_text_ce: 0.04107579216361046 (0.03142406474971655) | > loss_mel_ce: 2.6853299140930176 (nan) | > loss: 0.010819070972502232 (nan) | > current_lr: 5e-06 | > step_time: 0.1623 (0.13753919918762866) | > loader_time: 0.0048 (0.004658288960343586)  --> TIME: 2025-02-26 11:42:39 -- STEP: 6196/36938 -- GLOBAL_STEP: 1225150 | > loss_text_ce: 0.035350002348423004 (0.031430984521525075) | > loss_mel_ce: 2.4109370708465576 (nan) | > loss: 0.009707489050924778 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1375011855975664) | > loader_time: 0.0046 (0.004658327593658735)  --> TIME: 2025-02-26 11:42:48 -- STEP: 6246/36938 -- GLOBAL_STEP: 1225200 | > loss_text_ce: 0.02861267514526844 (0.03143033417468993) | > loss_mel_ce: 2.717268943786621 (nan) | > loss: 0.010896355845034122 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.1375027518985437) | > loader_time: 0.0046 (0.004658829620011998)  --> TIME: 2025-02-26 11:42:57 -- STEP: 6296/36938 -- GLOBAL_STEP: 1225250 | > loss_text_ce: 0.03257840499281883 (0.03143474625040715) | > loss_mel_ce: 2.8400495052337646 (nan) | > loss: 0.011399318464100361 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.1374805883576632) | > loader_time: 0.0046 (0.004658994218748759)  --> TIME: 2025-02-26 11:43:07 -- STEP: 6346/36938 -- GLOBAL_STEP: 1225300 | > loss_text_ce: 0.03273232281208038 (0.031430791975860406) | > loss_mel_ce: 3.3782403469085693 (nan) | > loss: 0.013535606674849987 (nan) | > current_lr: 5e-06 | > step_time: 0.169 (0.13749720217713435) | > loader_time: 0.0047 (0.004658724545605975)  --> TIME: 2025-02-26 11:43:16 -- STEP: 6396/36938 -- GLOBAL_STEP: 1225350 | > loss_text_ce: 0.033843304961919785 (0.03142516405742171) | > loss_mel_ce: 2.8967370986938477 (nan) | > loss: 0.011629288084805012 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.13747229033369307) | > loader_time: 0.0046 (0.004658072758198098)  --> TIME: 2025-02-26 11:43:25 -- STEP: 6446/36938 -- GLOBAL_STEP: 1225400 | > loss_text_ce: 0.03146931901574135 (0.031419645424058326) | > loss_mel_ce: 2.2950353622436523 (nan) | > loss: 0.00923216249793768 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13746609859425074) | > loader_time: 0.0046 (0.004658397739065993)  --> TIME: 2025-02-26 11:43:35 -- STEP: 6496/36938 -- GLOBAL_STEP: 1225450 | > loss_text_ce: 0.03346312418580055 (0.03142353021146907) | > loss_mel_ce: 2.7874529361724854 (nan) | > loss: 0.011194112710654736 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13744035899051843) | > loader_time: 0.0046 (0.00465833612263498)  --> TIME: 2025-02-26 11:43:44 -- STEP: 6546/36938 -- GLOBAL_STEP: 1225500 | > loss_text_ce: 0.02958240732550621 (0.0314132628660492) | > loss_mel_ce: 2.820021867752075 (nan) | > loss: 0.011307954788208008 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.13744102426162333) | > loader_time: 0.0046 (0.004658494853769446)  --> TIME: 2025-02-26 11:43:53 -- STEP: 6596/36938 -- GLOBAL_STEP: 1225550 | > loss_text_ce: 0.03363073617219925 (0.031415348648465756) | > loss_mel_ce: 2.7764523029327393 (nan) | > loss: 0.011151124723255634 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13743302860427684) | > loader_time: 0.0045 (0.0046583605651207925)  --> TIME: 2025-02-26 11:44:03 -- STEP: 6646/36938 -- GLOBAL_STEP: 1225600 | > loss_text_ce: 0.033499207347631454 (0.03141343256872306) | > loss_mel_ce: 3.2688417434692383 (nan) | > loss: 0.013104528188705444 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1374375051738909) | > loader_time: 0.0046 (0.004657947511297033)  --> TIME: 2025-02-26 11:44:12 -- STEP: 6696/36938 -- GLOBAL_STEP: 1225650 | > loss_text_ce: 0.027534611523151398 (0.031427894751073916) | > loss_mel_ce: 2.244346857070923 (nan) | > loss: 0.009015403687953949 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13745448498885368) | > loader_time: 0.0047 (0.004657782605329601)  --> TIME: 2025-02-26 11:44:21 -- STEP: 6746/36938 -- GLOBAL_STEP: 1225700 | > loss_text_ce: 0.04182446002960205 (0.031417282423837754) | > loss_mel_ce: 1.5968014001846313 (nan) | > loss: 0.00650248397141695 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1373928226917104) | > loader_time: 0.0046 (0.00465709831606485)  --> TIME: 2025-02-26 11:44:31 -- STEP: 6796/36938 -- GLOBAL_STEP: 1225750 | > loss_text_ce: 0.023705920204520226 (0.03141575308846608) | > loss_mel_ce: 2.7506213188171387 (nan) | > loss: 0.011009235866367817 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13742619009141407) | > loader_time: 0.0046 (0.004657475544186904)  --> TIME: 2025-02-26 11:44:40 -- STEP: 6846/36938 -- GLOBAL_STEP: 1225800 | > loss_text_ce: 0.03971652314066887 (0.031417032055395665) | > loss_mel_ce: 2.1019179821014404 (nan) | > loss: 0.00849855039268732 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.1374483758373075) | > loader_time: 0.0045 (0.004657887799538566)  --> TIME: 2025-02-26 11:44:50 -- STEP: 6896/36938 -- GLOBAL_STEP: 1225850 | > loss_text_ce: 0.03022434189915657 (0.03142596660867944) | > loss_mel_ce: 3.104153871536255 (nan) | > loss: 0.01243800949305296 (nan) | > current_lr: 5e-06 | > step_time: 0.1809 (0.13741312682490323) | > loader_time: 0.0056 (0.0046580813807843915)  --> TIME: 2025-02-26 11:44:59 -- STEP: 6946/36938 -- GLOBAL_STEP: 1225900 | > loss_text_ce: 0.03407258912920952 (0.031435141405061186) | > loss_mel_ce: 2.6563446521759033 (nan) | > loss: 0.010676260106265545 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.13737212743190721) | > loader_time: 0.0046 (0.004657671323194668)  --> TIME: 2025-02-26 11:45:08 -- STEP: 6996/36938 -- GLOBAL_STEP: 1225950 | > loss_text_ce: 0.02795376628637314 (0.031431468536202954) | > loss_mel_ce: 1.9764416217803955 (nan) | > loss: 0.007953951135277748 (nan) | > current_lr: 5e-06 | > step_time: 0.1264 (0.13734041267698174) | > loader_time: 0.0045 (0.004657810725369698)  --> TIME: 2025-02-26 11:45:17 -- STEP: 7046/36938 -- GLOBAL_STEP: 1226000 | > loss_text_ce: 0.023640796542167664 (0.031429082524795926) | > loss_mel_ce: 2.1326241493225098 (nan) | > loss: 0.008556608110666275 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.1372982931516064) | > loader_time: 0.0044 (0.004657412434788294)  --> TIME: 2025-02-26 11:45:26 -- STEP: 7096/36938 -- GLOBAL_STEP: 1226050 | > loss_text_ce: 0.009907037951052189 (0.031414357833763805) | > loss_mel_ce: 1.7091140747070312 (nan) | > loss: 0.006821512710303068 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13728136260990623) | > loader_time: 0.0044 (0.004656916036949907)  --> TIME: 2025-02-26 11:45:35 -- STEP: 7146/36938 -- GLOBAL_STEP: 1226100 | > loss_text_ce: 0.026103345677256584 (0.03141801687738926) | > loss_mel_ce: 2.7627177238464355 (nan) | > loss: 0.01106675062328577 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13726630238902157) | > loader_time: 0.0045 (0.004657269525087568)  --> TIME: 2025-02-26 11:45:44 -- STEP: 7196/36938 -- GLOBAL_STEP: 1226150 | > loss_text_ce: 0.030981438234448433 (0.03141822086402806) | > loss_mel_ce: 0.1071641743183136 (nan) | > loss: 0.0005481968983076513 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13726386396403834) | > loader_time: 0.0045 (0.004657301059890434)  --> TIME: 2025-02-26 11:45:54 -- STEP: 7246/36938 -- GLOBAL_STEP: 1226200 | > loss_text_ce: 0.027764691039919853 (0.03141912023233273) | > loss_mel_ce: 2.110400438308716 (nan) | > loss: 0.008484783582389355 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1372707951243265) | > loader_time: 0.0045 (0.004657345090556024)  --> TIME: 2025-02-26 11:46:03 -- STEP: 7296/36938 -- GLOBAL_STEP: 1226250 | > loss_text_ce: 0.024943146854639053 (0.03141282711626364) | > loss_mel_ce: 1.9062080383300781 (nan) | > loss: 0.007663298863917589 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13723895829497715) | > loader_time: 0.0046 (0.004657442632474415)  --> TIME: 2025-02-26 11:46:12 -- STEP: 7346/36938 -- GLOBAL_STEP: 1226300 | > loss_text_ce: 0.0380764864385128 (0.031405319481288836) | > loss_mel_ce: 2.5077083110809326 (nan) | > loss: 0.010102320462465286 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.13721777819626454) | > loader_time: 0.0047 (0.004658036098277719)  --> TIME: 2025-02-26 11:46:21 -- STEP: 7396/36938 -- GLOBAL_STEP: 1226350 | > loss_text_ce: 0.03379581496119499 (0.0314008259751394) | > loss_mel_ce: 1.7927449941635132 (nan) | > loss: 0.007248178124427795 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13721799321790962) | > loader_time: 0.0046 (0.0046586192189325614)  --> TIME: 2025-02-26 11:46:31 -- STEP: 7446/36938 -- GLOBAL_STEP: 1226400 | > loss_text_ce: 0.026139013469219208 (0.03140139254072706) | > loss_mel_ce: 2.962089776992798 (nan) | > loss: 0.011858051642775536 (nan) | > current_lr: 5e-06 | > step_time: 0.1343 (0.13720038191400594) | > loader_time: 0.0046 (0.004658926887727397)  --> TIME: 2025-02-26 11:46:40 -- STEP: 7496/36938 -- GLOBAL_STEP: 1226450 | > loss_text_ce: 0.029915349557995796 (0.03140687864132641) | > loss_mel_ce: 0.108642578125 (nan) | > loss: 0.0005498330574482679 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13721641179337174) | > loader_time: 0.0046 (0.004658741942083077)  --> TIME: 2025-02-26 11:46:50 -- STEP: 7546/36938 -- GLOBAL_STEP: 1226500 | > loss_text_ce: 0.030278168618679047 (0.031401409799410245) | > loss_mel_ce: 2.183532953262329 (nan) | > loss: 0.008784965611994267 (nan) | > current_lr: 5e-06 | > step_time: 0.165 (0.13723689322442534) | > loader_time: 0.0047 (0.004658633475274599)  --> TIME: 2025-02-26 11:46:59 -- STEP: 7596/36938 -- GLOBAL_STEP: 1226550 | > loss_text_ce: 0.03444253280758858 (0.03140295347568619) | > loss_mel_ce: 3.101325035095215 (nan) | > loss: 0.012443521991372108 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.13721996382577203) | > loader_time: 0.0046 (0.004658416015590867)  --> TIME: 2025-02-26 11:47:08 -- STEP: 7646/36938 -- GLOBAL_STEP: 1226600 | > loss_text_ce: 0.029515616595745087 (0.0313975757549207) | > loss_mel_ce: 3.3699209690093994 (nan) | > loss: 0.013489828445017338 (nan) | > current_lr: 5e-06 | > step_time: 0.1694 (0.13720708544454474) | > loader_time: 0.0048 (0.00465798190897934)  --> TIME: 2025-02-26 11:47:17 -- STEP: 7696/36938 -- GLOBAL_STEP: 1226650 | > loss_text_ce: 0.0386480987071991 (0.03139964828321901) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13719570989916063) | > loader_time: 0.0046 (0.004657850443945126)  --> TIME: 2025-02-26 11:47:27 -- STEP: 7746/36938 -- GLOBAL_STEP: 1226700 | > loss_text_ce: 0.029081081971526146 (0.03139384638343996) | > loss_mel_ce: 3.434474229812622 (nan) | > loss: 0.013744267635047436 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13720909846773927) | > loader_time: 0.0045 (0.004657544155253834)  --> TIME: 2025-02-26 11:47:36 -- STEP: 7796/36938 -- GLOBAL_STEP: 1226750 | > loss_text_ce: 0.03882298618555069 (0.03139073605956907) | > loss_mel_ce: 3.3180174827575684 (nan) | > loss: 0.013320796191692352 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.1372170291110879) | > loader_time: 0.0047 (0.00465722203927509)  --> TIME: 2025-02-26 11:47:46 -- STEP: 7846/36938 -- GLOBAL_STEP: 1226800 | > loss_text_ce: 0.038253799080848694 (0.031382525771435646) | > loss_mel_ce: 3.0524001121520996 (nan) | > loss: 0.012264500372111797 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.13722681266873138) | > loader_time: 0.0048 (0.004656877257585117)  --> TIME: 2025-02-26 11:47:55 -- STEP: 7896/36938 -- GLOBAL_STEP: 1226850 | > loss_text_ce: 0.03281630575656891 (0.031379167338349154) | > loss_mel_ce: 0.10107421875 (nan) | > loss: 0.0005313116125762463 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13725175058588052) | > loader_time: 0.0045 (0.004656716532016372)  --> TIME: 2025-02-26 11:48:05 -- STEP: 7946/36938 -- GLOBAL_STEP: 1226900 | > loss_text_ce: 0.03467370197176933 (0.03138638826569072) | > loss_mel_ce: 2.7836410999298096 (nan) | > loss: 0.011183789931237698 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.1372898755552423) | > loader_time: 0.0046 (0.0046565661105106414)  --> TIME: 2025-02-26 11:48:15 -- STEP: 7996/36938 -- GLOBAL_STEP: 1226950 | > loss_text_ce: 0.026240846142172813 (0.03137782898665375) | > loss_mel_ce: 2.7368483543395996 (nan) | > loss: 0.01096464041620493 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1373100858619654) | > loader_time: 0.0044 (0.004656373082905681)  --> TIME: 2025-02-26 11:48:24 -- STEP: 8046/36938 -- GLOBAL_STEP: 1227000 | > loss_text_ce: 0.035888299345970154 (0.03137661921987352) | > loss_mel_ce: 2.748631715774536 (nan) | > loss: 0.011049683205783367 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13731925984742227) | > loader_time: 0.0046 (0.0046560508885432365)  --> TIME: 2025-02-26 11:48:34 -- STEP: 8096/36938 -- GLOBAL_STEP: 1227050 | > loss_text_ce: 0.027668144553899765 (0.031376640205506214) | > loss_mel_ce: 2.9872751235961914 (nan) | > loss: 0.011964061297476292 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.1373318292406706) | > loader_time: 0.0046 (0.004655710498806061)  --> TIME: 2025-02-26 11:48:43 -- STEP: 8146/36938 -- GLOBAL_STEP: 1227100 | > loss_text_ce: 0.030568664893507957 (0.031366690999368445) | > loss_mel_ce: 1.2725974321365356 (nan) | > loss: 0.0051712943241000175 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13734348163305013) | > loader_time: 0.0045 (0.004655293946531945)  --> TIME: 2025-02-26 11:48:52 -- STEP: 8196/36938 -- GLOBAL_STEP: 1227150 | > loss_text_ce: 0.03442494198679924 (0.03136290540101016) | > loss_mel_ce: 3.1393141746520996 (nan) | > loss: 0.01259420346468687 (nan) | > current_lr: 5e-06 | > step_time: 0.1888 (0.13735726750612592) | > loader_time: 0.0048 (0.00465529476391974)  --> TIME: 2025-02-26 11:49:02 -- STEP: 8246/36938 -- GLOBAL_STEP: 1227200 | > loss_text_ce: 0.02788459323346615 (0.03136110205165798) | > loss_mel_ce: 2.5916025638580322 (nan) | > loss: 0.010394790209829807 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13733508946564238) | > loader_time: 0.0046 (0.004655165924425717)  --> TIME: 2025-02-26 11:49:11 -- STEP: 8296/36938 -- GLOBAL_STEP: 1227250 | > loss_text_ce: 0.023678012192249298 (0.0313544142404176) | > loss_mel_ce: 1.6636316776275635 (nan) | > loss: 0.006695673801004887 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13729860709051486) | > loader_time: 0.0046 (0.00465503346494678)  --> TIME: 2025-02-26 11:49:20 -- STEP: 8346/36938 -- GLOBAL_STEP: 1227300 | > loss_text_ce: 0.029485227540135384 (0.0313560950470271) | > loss_mel_ce: 2.5245914459228516 (nan) | > loss: 0.010135225020349026 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.1372679609090281) | > loader_time: 0.0046 (0.00465501785963925)  --> TIME: 2025-02-26 11:49:29 -- STEP: 8396/36938 -- GLOBAL_STEP: 1227350 | > loss_text_ce: 0.023877641186118126 (0.03135165675115523) | > loss_mel_ce: 2.861328601837158 (nan) | > loss: 0.011449231766164303 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13727918728696192) | > loader_time: 0.0045 (0.004655086153629239)  --> TIME: 2025-02-26 11:49:38 -- STEP: 8446/36938 -- GLOBAL_STEP: 1227400 | > loss_text_ce: 0.027483072131872177 (0.03136070537970366) | > loss_mel_ce: 1.12109375 (nan) | > loss: 0.004557844717055559 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13726577054768277) | > loader_time: 0.0045 (0.004655396404803843)  --> TIME: 2025-02-26 11:49:48 -- STEP: 8496/36938 -- GLOBAL_STEP: 1227450 | > loss_text_ce: 0.03529733419418335 (0.03135989189642111) | > loss_mel_ce: 2.6152617931365967 (nan) | > loss: 0.010518092662096024 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1372750899605381) | > loader_time: 0.0045 (0.004655250665607213)  --> TIME: 2025-02-26 11:49:57 -- STEP: 8546/36938 -- GLOBAL_STEP: 1227500 | > loss_text_ce: 0.030521800741553307 (0.031364253148800106) | > loss_mel_ce: 2.5503342151641846 (nan) | > loss: 0.01024149265140295 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13725204479490694) | > loader_time: 0.0044 (0.00465487613628879)  --> TIME: 2025-02-26 11:50:06 -- STEP: 8596/36938 -- GLOBAL_STEP: 1227550 | > loss_text_ce: 0.02345903031527996 (0.031363124244969065) | > loss_mel_ce: 3.0134518146514893 (nan) | > loss: 0.012051234021782875 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.13722573091064053) | > loader_time: 0.0046 (0.004654709823523206)  --> TIME: 2025-02-26 11:50:15 -- STEP: 8646/36938 -- GLOBAL_STEP: 1227600 | > loss_text_ce: 0.04115918651223183 (0.031366977620272705) | > loss_mel_ce: 0.050384521484375 (nan) | > loss: 0.0003632686857599765 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1372082438591367) | > loader_time: 0.0045 (0.004654351715795603)  --> TIME: 2025-02-26 11:50:24 -- STEP: 8696/36938 -- GLOBAL_STEP: 1227650 | > loss_text_ce: 0.04159308224916458 (0.03137362743672173) | > loss_mel_ce: 3.0905981063842773 (nan) | > loss: 0.012429330497980118 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13720313389035882) | > loader_time: 0.0046 (0.004654081540033163)  --> TIME: 2025-02-26 11:50:34 -- STEP: 8746/36938 -- GLOBAL_STEP: 1227700 | > loss_text_ce: 0.0254169013351202 (0.03136439368630537) | > loss_mel_ce: 2.128964424133301 (nan) | > loss: 0.008549132384359837 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13721651791599365) | > loader_time: 0.0045 (0.004654038642108684)  --> TIME: 2025-02-26 11:50:44 -- STEP: 8796/36938 -- GLOBAL_STEP: 1227750 | > loss_text_ce: 0.031175246462225914 (0.03136489084714675) | > loss_mel_ce: 3.1670517921447754 (nan) | > loss: 0.01269137766212225 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13724155225987952) | > loader_time: 0.0046 (0.004654156261815341)  --> TIME: 2025-02-26 11:50:53 -- STEP: 8846/36938 -- GLOBAL_STEP: 1227800 | > loss_text_ce: 0.029270734637975693 (0.03135682420315327) | > loss_mel_ce: 2.549398899078369 (nan) | > loss: 0.010232816450297832 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13722966303794099) | > loader_time: 0.0045 (0.004653733913482881)  --> TIME: 2025-02-26 11:51:02 -- STEP: 8896/36938 -- GLOBAL_STEP: 1227850 | > loss_text_ce: 0.02909763902425766 (0.03135784502327945) | > loss_mel_ce: 2.84993052482605 (nan) | > loss: 0.011424715630710125 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13723100480630232) | > loader_time: 0.0045 (0.004653786664172058)  --> TIME: 2025-02-26 11:51:11 -- STEP: 8946/36938 -- GLOBAL_STEP: 1227900 | > loss_text_ce: 0.023105982691049576 (0.03135484407316307) | > loss_mel_ce: 2.1113853454589844 (nan) | > loss: 0.008470204658806324 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13720951436377732) | > loader_time: 0.0045 (0.004653601206151174)  --> TIME: 2025-02-26 11:51:21 -- STEP: 8996/36938 -- GLOBAL_STEP: 1227950 | > loss_text_ce: 0.03719382360577583 (0.03135958525798173) | > loss_mel_ce: 2.287802219390869 (nan) | > loss: 0.009226175025105476 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13723742304615777) | > loader_time: 0.0046 (0.004653603752860514)  --> TIME: 2025-02-26 11:51:30 -- STEP: 9046/36938 -- GLOBAL_STEP: 1228000 | > loss_text_ce: 0.042644862085580826 (0.031351064145688605) | > loss_mel_ce: 2.4944660663604736 (nan) | > loss: 0.010067901574075222 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1372495954236492) | > loader_time: 0.0045 (0.004653385959577185)  --> TIME: 2025-02-26 11:51:40 -- STEP: 9096/36938 -- GLOBAL_STEP: 1228050 | > loss_text_ce: 0.027455631643533707 (0.03134815656050073) | > loss_mel_ce: 3.4524266719818115 (nan) | > loss: 0.01380905695259571 (nan) | > current_lr: 5e-06 | > step_time: 0.178 (0.13725271954699977) | > loader_time: 0.0048 (0.004653034549904389)  --> TIME: 2025-02-26 11:51:49 -- STEP: 9146/36938 -- GLOBAL_STEP: 1228100 | > loss_text_ce: 0.03933430463075638 (0.031342774106907736) | > loss_mel_ce: 0.023773193359375 (nan) | > loss: 0.00025042658671736717 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13722400388440265) | > loader_time: 0.0048 (0.004652736563925527)  --> TIME: 2025-02-26 11:51:58 -- STEP: 9196/36938 -- GLOBAL_STEP: 1228150 | > loss_text_ce: 0.03113558515906334 (0.031335621637346114) | > loss_mel_ce: 2.981149911880493 (nan) | > loss: 0.011953514069318771 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.1372063584356734) | > loader_time: 0.0045 (0.004652406143696233)  --> TIME: 2025-02-26 11:52:08 -- STEP: 9246/36938 -- GLOBAL_STEP: 1228200 | > loss_text_ce: 0.025830652564764023 (0.031326035768915385) | > loss_mel_ce: 2.364971160888672 (nan) | > loss: 0.009487309493124485 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13722172612213546) | > loader_time: 0.0045 (0.00465245564572358)  --> TIME: 2025-02-26 11:52:17 -- STEP: 9296/36938 -- GLOBAL_STEP: 1228250 | > loss_text_ce: 0.03341168537735939 (0.03132797100820876) | > loss_mel_ce: 3.0584959983825684 (nan) | > loss: 0.012269475497305393 (nan) | > current_lr: 5e-06 | > step_time: 0.1355 (0.13719352528375925) | > loader_time: 0.0048 (0.004652467118688392)  --> TIME: 2025-02-26 11:52:26 -- STEP: 9346/36938 -- GLOBAL_STEP: 1228300 | > loss_text_ce: 0.033841319382190704 (0.03132165905738703) | > loss_mel_ce: 2.6543002128601074 (nan) | > loss: 0.010667229071259499 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13716487213092304) | > loader_time: 0.0046 (0.004652255330941892)  --> TIME: 2025-02-26 11:52:35 -- STEP: 9396/36938 -- GLOBAL_STEP: 1228350 | > loss_text_ce: 0.03568052500486374 (0.031320011855185174) | > loss_mel_ce: 2.2879180908203125 (nan) | > loss: 0.009220629930496216 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13715306816328407) | > loader_time: 0.0046 (0.004652278354290768)  --> TIME: 2025-02-26 11:52:45 -- STEP: 9446/36938 -- GLOBAL_STEP: 1228400 | > loss_text_ce: 0.029743356630206108 (0.031315596871040206) | > loss_mel_ce: 3.0559773445129395 (nan) | > loss: 0.01224492397159338 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.1371777938529389) | > loader_time: 0.0047 (0.004652408808438238)  --> TIME: 2025-02-26 11:52:54 -- STEP: 9496/36938 -- GLOBAL_STEP: 1228450 | > loss_text_ce: 0.03701191395521164 (0.031319594488337224) | > loss_mel_ce: 2.847843647003174 (nan) | > loss: 0.011447840370237827 (nan) | > current_lr: 5e-06 | > step_time: 0.1809 (0.13721309096349193) | > loader_time: 0.0048 (0.004652222215175242)  --> TIME: 2025-02-26 11:53:03 -- STEP: 9546/36938 -- GLOBAL_STEP: 1228500 | > loss_text_ce: 0.020161045715212822 (0.0313186470928245) | > loss_mel_ce: 0.07578277587890625 (nan) | > loss: 0.000380729470634833 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13718437169448824) | > loader_time: 0.0045 (0.004652330267486813)  --> TIME: 2025-02-26 11:53:13 -- STEP: 9596/36938 -- GLOBAL_STEP: 1228550 | > loss_text_ce: 0.02589435689151287 (0.03131546106753567) | > loss_mel_ce: 3.4805474281311035 (nan) | > loss: 0.013914452865719795 (nan) | > current_lr: 5e-06 | > step_time: 0.1512 (0.13719249102114217) | > loader_time: 0.0046 (0.004652184737429332)  --> TIME: 2025-02-26 11:53:22 -- STEP: 9646/36938 -- GLOBAL_STEP: 1228600 | > loss_text_ce: 0.029193881899118423 (0.031314885111692926) | > loss_mel_ce: 2.6928231716156006 (nan) | > loss: 0.010801655240356922 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13715893973113824) | > loader_time: 0.0045 (0.004651908777621748)  --> TIME: 2025-02-26 11:53:31 -- STEP: 9696/36938 -- GLOBAL_STEP: 1228650 | > loss_text_ce: 0.024014003574848175 (0.03131329223110956) | > loss_mel_ce: 1.2848052978515625 (nan) | > loss: 0.005193727556616068 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13718650389229092) | > loader_time: 0.0051 (0.004652106402927517)  --> TIME: 2025-02-26 11:53:41 -- STEP: 9746/36938 -- GLOBAL_STEP: 1228700 | > loss_text_ce: 0.026219993829727173 (0.03131578660138352) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13720036208592898) | > loader_time: 0.0045 (0.004652093818440684)  --> TIME: 2025-02-26 11:53:50 -- STEP: 9796/36938 -- GLOBAL_STEP: 1228750 | > loss_text_ce: 0.027548793703317642 (0.03131986459300365) | > loss_mel_ce: 1.0886422395706177 (nan) | > loss: 0.00442932965233922 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13718989397760004) | > loader_time: 0.0045 (0.004651857060089382)  --> TIME: 2025-02-26 11:54:00 -- STEP: 9846/36938 -- GLOBAL_STEP: 1228800 | > loss_text_ce: 0.026511672884225845 (0.031321360259338654) | > loss_mel_ce: 2.8848154544830322 (nan) | > loss: 0.011552886106073856 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.1371812711710272) | > loader_time: 0.0045 (0.004651585972553703)  --> TIME: 2025-02-26 11:54:09 -- STEP: 9896/36938 -- GLOBAL_STEP: 1228850 | > loss_text_ce: 0.026263991370797157 (0.03132037678488785) | > loss_mel_ce: 2.6556363105773926 (nan) | > loss: 0.01064246241003275 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13715650062938792) | > loader_time: 0.0046 (0.004651588832744224)  --> TIME: 2025-02-26 11:54:18 -- STEP: 9946/36938 -- GLOBAL_STEP: 1228900 | > loss_text_ce: 0.026764290407299995 (0.03132907908429317) | > loss_mel_ce: 2.977543354034424 (nan) | > loss: 0.011921856552362442 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13718168193273275) | > loader_time: 0.0045 (0.004651414851847022)  --> TIME: 2025-02-26 11:54:28 -- STEP: 9996/36938 -- GLOBAL_STEP: 1228950 | > loss_text_ce: 0.0326797254383564 (0.03133033077344723) | > loss_mel_ce: 2.954153299331665 (nan) | > loss: 0.011852513067424297 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13719283892851727) | > loader_time: 0.0046 (0.004651185582761256)  --> TIME: 2025-02-26 11:54:37 -- STEP: 10046/36938 -- GLOBAL_STEP: 1229000 | > loss_text_ce: 0.032043781131505966 (0.031329854771609156) | > loss_mel_ce: 2.1543545722961426 (nan) | > loss: 0.008676184341311455 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13718502397630256) | > loader_time: 0.0045 (0.004651509835001927)  --> TIME: 2025-02-26 11:54:46 -- STEP: 10096/36938 -- GLOBAL_STEP: 1229050 | > loss_text_ce: 0.02855874039232731 (0.031332881284896946) | > loss_mel_ce: 2.4040184020996094 (nan) | > loss: 0.009653083980083466 (nan) | > current_lr: 5e-06 | > step_time: 0.1659 (0.13717548459437687) | > loader_time: 0.0046 (0.004651388965204678)  --> TIME: 2025-02-26 11:54:56 -- STEP: 10146/36938 -- GLOBAL_STEP: 1229100 | > loss_text_ce: 0.040377601981163025 (0.03133411063389889) | > loss_mel_ce: 2.9470062255859375 (nan) | > loss: 0.01185469888150692 (nan) | > current_lr: 5e-06 | > step_time: 0.1567 (0.13719390718372487) | > loader_time: 0.0046 (0.004651535833326066)  --> TIME: 2025-02-26 11:55:05 -- STEP: 10196/36938 -- GLOBAL_STEP: 1229150 | > loss_text_ce: 0.04085583984851837 (0.03133503979739875) | > loss_mel_ce: 2.5312695503234863 (nan) | > loss: 0.010206847451627254 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1371816455405944) | > loader_time: 0.0045 (0.004651306680063499)  --> TIME: 2025-02-26 11:55:14 -- STEP: 10246/36938 -- GLOBAL_STEP: 1229200 | > loss_text_ce: 0.021308379247784615 (0.031338715211550384) | > loss_mel_ce: 2.029191732406616 (nan) | > loss: 0.00813690572977066 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13717119958562224) | > loader_time: 0.0045 (0.004651022218015082)  --> TIME: 2025-02-26 11:55:24 -- STEP: 10296/36938 -- GLOBAL_STEP: 1229250 | > loss_text_ce: 0.02910505048930645 (0.03133724023886152) | > loss_mel_ce: 2.7986361980438232 (nan) | > loss: 0.011221195571124554 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.1371775485557833) | > loader_time: 0.0047 (0.004651119080086324)  --> TIME: 2025-02-26 11:55:33 -- STEP: 10346/36938 -- GLOBAL_STEP: 1229300 | > loss_text_ce: 0.03235830366611481 (0.031342125016524156) | > loss_mel_ce: 2.5096936225891113 (nan) | > loss: 0.010087508708238602 (nan) | > current_lr: 5e-06 | > step_time: 0.1613 (0.13716712095694597) | > loader_time: 0.0046 (0.004651005277775625)  --> TIME: 2025-02-26 11:55:43 -- STEP: 10396/36938 -- GLOBAL_STEP: 1229350 | > loss_text_ce: 0.02958647906780243 (0.03134440470264001) | > loss_mel_ce: 2.4771921634674072 (nan) | > loss: 0.009947535581886768 (nan) | > current_lr: 5e-06 | > step_time: 0.1875 (0.13718997778549427) | > loader_time: 0.0048 (0.004650833103096276)  --> TIME: 2025-02-26 11:55:52 -- STEP: 10446/36938 -- GLOBAL_STEP: 1229400 | > loss_text_ce: 0.036285463720560074 (0.03134682279657177) | > loss_mel_ce: 3.0557262897491455 (nan) | > loss: 0.012269888073205948 (nan) | > current_lr: 5e-06 | > step_time: 0.146 (0.13718387983823555) | > loader_time: 0.0047 (0.004650686153752954)  --> TIME: 2025-02-26 11:56:01 -- STEP: 10496/36938 -- GLOBAL_STEP: 1229450 | > loss_text_ce: 0.023364361375570297 (0.03134279688798955) | > loss_mel_ce: 2.989795446395874 (nan) | > loss: 0.011956984177231789 (nan) | > current_lr: 5e-06 | > step_time: 0.15 (0.13718325213143026) | > loader_time: 0.0046 (0.004650472845064441)  --> TIME: 2025-02-26 11:56:10 -- STEP: 10546/36938 -- GLOBAL_STEP: 1229500 | > loss_text_ce: 0.016321539878845215 (0.03133635888119545) | > loss_mel_ce: 0.00726318359375 (nan) | > loss: 9.35901771299541e-05 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13716672546561073) | > loader_time: 0.0045 (0.0046504240390496275)  --> TIME: 2025-02-26 11:56:20 -- STEP: 10596/36938 -- GLOBAL_STEP: 1229550 | > loss_text_ce: 0.022625215351581573 (0.031335674986916986) | > loss_mel_ce: 3.4182260036468506 (nan) | > loss: 0.013654172420501709 (nan) | > current_lr: 5e-06 | > step_time: 0.107 (0.13718600266832098) | > loader_time: 0.0044 (0.004650488490201528)  --> TIME: 2025-02-26 11:56:29 -- STEP: 10646/36938 -- GLOBAL_STEP: 1229600 | > loss_text_ce: 0.021432535722851753 (0.031329987479838284) | > loss_mel_ce: 2.789597749710083 (nan) | > loss: 0.011154883541166782 (nan) | > current_lr: 5e-06 | > step_time: 0.1436 (0.1371785804514551) | > loader_time: 0.0047 (0.004650499886550931)  --> TIME: 2025-02-26 11:56:39 -- STEP: 10696/36938 -- GLOBAL_STEP: 1229650 | > loss_text_ce: 0.03414451703429222 (0.03133050705943265) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.13718911287672222) | > loader_time: 0.0045 (0.0046503502616440226)  --> TIME: 2025-02-26 11:56:48 -- STEP: 10746/36938 -- GLOBAL_STEP: 1229700 | > loss_text_ce: 0.03867218270897865 (0.03132873302280208) | > loss_mel_ce: 0.08692678064107895 (nan) | > loss: 0.0004984086262993515 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13721896154175428) | > loader_time: 0.0045 (0.0046501984792383265)  --> TIME: 2025-02-26 11:56:58 -- STEP: 10796/36938 -- GLOBAL_STEP: 1229750 | > loss_text_ce: 0.03770814463496208 (0.03132905748252858) | > loss_mel_ce: 2.36310076713562 (nan) | > loss: 0.00952701922506094 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.1372099826847371) | > loader_time: 0.0046 (0.004650086550944043)  --> TIME: 2025-02-26 11:57:07 -- STEP: 10846/36938 -- GLOBAL_STEP: 1229800 | > loss_text_ce: 0.030849603936076164 (0.031324926606720864) | > loss_mel_ce: 1.3682149648666382 (nan) | > loss: 0.0055518439039587975 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1372058556876937) | > loader_time: 0.0046 (0.00464990718017839)  --> TIME: 2025-02-26 11:57:16 -- STEP: 10896/36938 -- GLOBAL_STEP: 1229850 | > loss_text_ce: 0.029167596250772476 (0.03132605489482577) | > loss_mel_ce: 2.975311040878296 (nan) | > loss: 0.011922535486519337 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.13720952510483633) | > loader_time: 0.0047 (0.004649611909308993)  --> TIME: 2025-02-26 11:57:26 -- STEP: 10946/36938 -- GLOBAL_STEP: 1229900 | > loss_text_ce: 0.025650909170508385 (0.03133014712510617) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13720758832962532) | > loader_time: 0.0045 (0.004649406853398283)  --> TIME: 2025-02-26 11:57:36 -- STEP: 10996/36938 -- GLOBAL_STEP: 1229950 | > loss_text_ce: 0.027525486424565315 (0.03132268956678785) | > loss_mel_ce: 3.159827947616577 (nan) | > loss: 0.01264822855591774 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.13724590557711727) | > loader_time: 0.0048 (0.004649685638173813)  --> TIME: 2025-02-26 11:57:45 -- STEP: 11046/36938 -- GLOBAL_STEP: 1230000 | > loss_text_ce: 0.027837011963129044 (0.03132070470073537) | > loss_mel_ce: 2.968485116958618 (nan) | > loss: 0.01189016830176115 (nan) | > current_lr: 5e-06 | > step_time: 0.1953 (0.1372608079570111) | > loader_time: 0.0049 (0.004649669390624377) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1230000.pth  --> TIME: 2025-02-26 11:58:00 -- STEP: 11096/36938 -- GLOBAL_STEP: 1230050 | > loss_text_ce: 0.031233923509716988 (0.03132125040291002) | > loss_mel_ce: 2.902388334274292 (nan) | > loss: 0.011641358956694603 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13725542869598797) | > loader_time: 0.0045 (0.004649629331614889)  --> TIME: 2025-02-26 11:58:09 -- STEP: 11146/36938 -- GLOBAL_STEP: 1230100 | > loss_text_ce: 0.03336286172270775 (0.031322179343712) | > loss_mel_ce: 3.4434685707092285 (nan) | > loss: 0.013796950690448284 (nan) | > current_lr: 5e-06 | > step_time: 0.1622 (0.1372296412286734) | > loader_time: 0.0046 (0.004649366293735546)  --> TIME: 2025-02-26 11:58:19 -- STEP: 11196/36938 -- GLOBAL_STEP: 1230150 | > loss_text_ce: 0.027693185955286026 (0.03132167218608059) | > loss_mel_ce: 3.159069538116455 (nan) | > loss: 0.012645885348320007 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1372219487231471) | > loader_time: 0.0046 (0.004649423411506648)  --> TIME: 2025-02-26 11:58:28 -- STEP: 11246/36938 -- GLOBAL_STEP: 1230200 | > loss_text_ce: 0.032978855073451996 (0.0313263007328042) | > loss_mel_ce: 2.3734514713287354 (nan) | > loss: 0.009549327194690704 (nan) | > current_lr: 5e-06 | > step_time: 0.1077 (0.13721045350850344) | > loader_time: 0.0045 (0.0046494964304162134)  --> TIME: 2025-02-26 11:58:37 -- STEP: 11296/36938 -- GLOBAL_STEP: 1230250 | > loss_text_ce: 0.02811698615550995 (0.031317595908007555) | > loss_mel_ce: 2.700073719024658 (nan) | > loss: 0.010826153680682182 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13721087634056878) | > loader_time: 0.0048 (0.004649454025979096)  --> TIME: 2025-02-26 11:58:46 -- STEP: 11346/36938 -- GLOBAL_STEP: 1230300 | > loss_text_ce: 0.03833815082907677 (0.0313184208370597) | > loss_mel_ce: 1.0912679433822632 (nan) | > loss: 0.004482564050704241 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13720315384860587) | > loader_time: 0.0045 (0.004649410062043867)  --> TIME: 2025-02-26 11:58:56 -- STEP: 11396/36938 -- GLOBAL_STEP: 1230350 | > loss_text_ce: 0.028105804696679115 (0.031321398263095136) | > loss_mel_ce: 2.5149054527282715 (nan) | > loss: 0.010091315023601055 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13719665062724595) | > loader_time: 0.0045 (0.004649319013573501)  --> TIME: 2025-02-26 11:59:05 -- STEP: 11446/36938 -- GLOBAL_STEP: 1230400 | > loss_text_ce: 0.027362486347556114 (0.03132256048637301) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.1372002231976837) | > loader_time: 0.0045 (0.00464937654842183)  --> TIME: 2025-02-26 11:59:14 -- STEP: 11496/36938 -- GLOBAL_STEP: 1230450 | > loss_text_ce: 0.02705787867307663 (0.03132041368078425) | > loss_mel_ce: 3.166372299194336 (nan) | > loss: 0.012672342360019684 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13720239142037635) | > loader_time: 0.0046 (0.004649368357807723)  --> TIME: 2025-02-26 11:59:24 -- STEP: 11546/36938 -- GLOBAL_STEP: 1230500 | > loss_text_ce: 0.03828965872526169 (0.03132546981574012) | > loss_mel_ce: 2.2550911903381348 (nan) | > loss: 0.0091007174924016 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13720561160013742) | > loader_time: 0.0046 (0.0046490571867670175)  --> TIME: 2025-02-26 11:59:33 -- STEP: 11596/36938 -- GLOBAL_STEP: 1230550 | > loss_text_ce: 0.02683262526988983 (0.03132620731440319) | > loss_mel_ce: 2.5164542198181152 (nan) | > loss: 0.010092408396303654 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13720465783786667) | > loader_time: 0.0045 (0.004648957860268337)  --> TIME: 2025-02-26 11:59:43 -- STEP: 11646/36938 -- GLOBAL_STEP: 1230600 | > loss_text_ce: 0.03287298604846001 (0.031329072240411385) | > loss_mel_ce: 2.6446025371551514 (nan) | > loss: 0.010624903254210949 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.1372078108701553) | > loader_time: 0.0047 (0.004648697881913163)  --> TIME: 2025-02-26 11:59:52 -- STEP: 11696/36938 -- GLOBAL_STEP: 1230650 | > loss_text_ce: 0.032553620636463165 (0.03132591796132766) | > loss_mel_ce: 3.035255193710327 (nan) | > loss: 0.012173845432698727 (nan) | > current_lr: 5e-06 | > step_time: 0.1816 (0.13722390509255092) | > loader_time: 0.0047 (0.004648604854513301)  --> TIME: 2025-02-26 12:00:02 -- STEP: 11746/36938 -- GLOBAL_STEP: 1230700 | > loss_text_ce: 0.042028654366731644 (0.03132984821819303) | > loss_mel_ce: 3.1558291912078857 (nan) | > loss: 0.01268991269171238 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13723650399615142) | > loader_time: 0.0044 (0.004648631463033806)  --> TIME: 2025-02-26 12:00:11 -- STEP: 11796/36938 -- GLOBAL_STEP: 1230750 | > loss_text_ce: 0.027300439774990082 (0.031330824595386345) | > loss_mel_ce: 2.739363193511963 (nan) | > loss: 0.010978824459016323 (nan) | > current_lr: 5e-06 | > step_time: 0.1642 (0.13725054860317154) | > loader_time: 0.0046 (0.004648519374831899)  --> TIME: 2025-02-26 12:00:20 -- STEP: 11846/36938 -- GLOBAL_STEP: 1230800 | > loss_text_ce: 0.026262667030096054 (0.03132617782457678) | > loss_mel_ce: 3.1848881244659424 (nan) | > loss: 0.0127426628023386 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13723061309359097) | > loader_time: 0.0047 (0.004648286145461615)  --> TIME: 2025-02-26 12:00:30 -- STEP: 11896/36938 -- GLOBAL_STEP: 1230850 | > loss_text_ce: 0.02977229841053486 (0.031324771208711896) | > loss_mel_ce: 2.341888427734375 (nan) | > loss: 0.009411352686583996 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.1372404835251408) | > loader_time: 0.0048 (0.00464854119364336)  --> TIME: 2025-02-26 12:00:39 -- STEP: 11946/36938 -- GLOBAL_STEP: 1230900 | > loss_text_ce: 0.028519848361611366 (0.031319972879298524) | > loss_mel_ce: 2.2618484497070312 (nan) | > loss: 0.00908876396715641 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13724787924211065) | > loader_time: 0.0045 (0.004648605942546525)  --> TIME: 2025-02-26 12:00:48 -- STEP: 11996/36938 -- GLOBAL_STEP: 1230950 | > loss_text_ce: 0.02675006166100502 (0.03131751463749773) | > loss_mel_ce: 1.2310400009155273 (nan) | > loss: 0.004991231020539999 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1372366955614996) | > loader_time: 0.0045 (0.004648493663117515)  --> TIME: 2025-02-26 12:00:58 -- STEP: 12046/36938 -- GLOBAL_STEP: 1231000 | > loss_text_ce: 0.029889285564422607 (0.03131261007815789) | > loss_mel_ce: 1.9684747457504272 (nan) | > loss: 0.00793001614511013 (nan) | > current_lr: 5e-06 | > step_time: 0.1263 (0.1372268838817527) | > loader_time: 0.0045 (0.004648710373883715)  --> TIME: 2025-02-26 12:01:07 -- STEP: 12096/36938 -- GLOBAL_STEP: 1231050 | > loss_text_ce: 0.027526723220944405 (0.031312117587375385) | > loss_mel_ce: 2.7703945636749268 (nan) | > loss: 0.011102862656116486 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13721700367473405) | > loader_time: 0.0046 (0.0046493296348859535)  --> TIME: 2025-02-26 12:01:16 -- STEP: 12146/36938 -- GLOBAL_STEP: 1231100 | > loss_text_ce: 0.017756039276719093 (0.03131201441157311) | > loss_mel_ce: 2.665069103240967 (nan) | > loss: 0.010646131820976734 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.13720840901730177) | > loader_time: 0.0045 (0.004649873681227824)  --> TIME: 2025-02-26 12:01:26 -- STEP: 12196/36938 -- GLOBAL_STEP: 1231150 | > loss_text_ce: 0.03601135313510895 (0.031313512382586665) | > loss_mel_ce: 2.6121749877929688 (nan) | > loss: 0.010508676990866661 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13721999867152426) | > loader_time: 0.0047 (0.0046499842657500325)  --> TIME: 2025-02-26 12:01:35 -- STEP: 12246/36938 -- GLOBAL_STEP: 1231200 | > loss_text_ce: 0.028838729485869408 (0.031313768186076195) | > loss_mel_ce: 0.0058034262619912624 (nan) | > loss: 0.0001374688872601837 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13721588959242972) | > loader_time: 0.0045 (0.004650131639421104)  --> TIME: 2025-02-26 12:01:44 -- STEP: 12296/36938 -- GLOBAL_STEP: 1231250 | > loss_text_ce: 0.03421233594417572 (0.031312990092088384) | > loss_mel_ce: 2.8646769523620605 (nan) | > loss: 0.011503529734909534 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13722837283460812) | > loader_time: 0.0046 (0.004650025202534625)  --> TIME: 2025-02-26 12:01:54 -- STEP: 12346/36938 -- GLOBAL_STEP: 1231300 | > loss_text_ce: 0.02933732233941555 (0.03130888327773355) | > loss_mel_ce: 2.726346969604492 (nan) | > loss: 0.010935256257653236 (nan) | > current_lr: 5e-06 | > step_time: 0.1511 (0.1372284772966544) | > loader_time: 0.0047 (0.004650221947775519)  --> TIME: 2025-02-26 12:02:03 -- STEP: 12396/36938 -- GLOBAL_STEP: 1231350 | > loss_text_ce: 0.0392782986164093 (0.03130582122134592) | > loss_mel_ce: 2.7114760875701904 (nan) | > loss: 0.010915692895650864 (nan) | > current_lr: 5e-06 | > step_time: 0.1442 (0.1372418597306923) | > loader_time: 0.0046 (0.004650391429114566)  --> TIME: 2025-02-26 12:02:13 -- STEP: 12446/36938 -- GLOBAL_STEP: 1231400 | > loss_text_ce: 0.04311758279800415 (0.03131127088247933) | > loss_mel_ce: 2.3867335319519043 (nan) | > loss: 0.009642266668379307 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13723646473811768) | > loader_time: 0.0045 (0.004650538304448628)  --> TIME: 2025-02-26 12:02:22 -- STEP: 12496/36938 -- GLOBAL_STEP: 1231450 | > loss_text_ce: 0.03530416265130043 (0.03131049960977191) | > loss_mel_ce: 3.18849778175354 (nan) | > loss: 0.012792864814400673 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.13724842680935334) | > loader_time: 0.0046 (0.004650290525982249)  --> TIME: 2025-02-26 12:02:32 -- STEP: 12546/36938 -- GLOBAL_STEP: 1231500 | > loss_text_ce: 0.027081310749053955 (0.03131846058171) | > loss_mel_ce: 3.294893980026245 (nan) | > loss: 0.0131824417039752 (nan) | > current_lr: 5e-06 | > step_time: 0.1483 (0.13724089658908337) | > loader_time: 0.0047 (0.004650143027856726)  --> TIME: 2025-02-26 12:02:41 -- STEP: 12596/36938 -- GLOBAL_STEP: 1231550 | > loss_text_ce: 0.029259145259857178 (0.03132463917800167) | > loss_mel_ce: 2.16158390045166 (nan) | > loss: 0.008693822659552097 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13725624407386317) | > loader_time: 0.0044 (0.004649979059716662)  --> TIME: 2025-02-26 12:02:51 -- STEP: 12646/36938 -- GLOBAL_STEP: 1231600 | > loss_text_ce: 0.03713871166110039 (0.03132731620705161) | > loss_mel_ce: 2.2173678874969482 (nan) | > loss: 0.008946455083787441 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13725615726059462) | > loader_time: 0.0044 (0.004649906695388703)  --> TIME: 2025-02-26 12:03:00 -- STEP: 12696/36938 -- GLOBAL_STEP: 1231650 | > loss_text_ce: 0.03607996553182602 (0.03132907358221496) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13723866056554607) | > loader_time: 0.0045 (0.004649616068988476)  --> TIME: 2025-02-26 12:03:09 -- STEP: 12746/36938 -- GLOBAL_STEP: 1231700 | > loss_text_ce: 0.020986123010516167 (0.031326218101561554) | > loss_mel_ce: 2.317971706390381 (nan) | > loss: 0.009281579405069351 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13724792643355044) | > loader_time: 0.0045 (0.0046499303721678435)  --> TIME: 2025-02-26 12:03:18 -- STEP: 12796/36938 -- GLOBAL_STEP: 1231750 | > loss_text_ce: 0.02978869527578354 (0.031327151960733234) | > loss_mel_ce: 2.597529172897339 (nan) | > loss: 0.010425865650177002 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13725167341476732) | > loader_time: 0.0045 (0.004649732309641352)  --> TIME: 2025-02-26 12:03:27 -- STEP: 12846/36938 -- GLOBAL_STEP: 1231800 | > loss_text_ce: 0.03371857479214668 (0.03132851157769818) | > loss_mel_ce: 2.9291632175445557 (nan) | > loss: 0.01175746787339449 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13722746563017144) | > loader_time: 0.0046 (0.004649605258090592)  --> TIME: 2025-02-26 12:03:36 -- STEP: 12896/36938 -- GLOBAL_STEP: 1231850 | > loss_text_ce: 0.04037934169173241 (0.031332311240904825) | > loss_mel_ce: 2.598233461380005 (nan) | > loss: 0.010470686480402946 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13720029873528486) | > loader_time: 0.0046 (0.004649486919638554)  --> TIME: 2025-02-26 12:03:46 -- STEP: 12946/36938 -- GLOBAL_STEP: 1231900 | > loss_text_ce: 0.028550218790769577 (0.03133034631507768) | > loss_mel_ce: 2.6597862243652344 (nan) | > loss: 0.01066800206899643 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13721255474880265) | > loader_time: 0.0045 (0.004649894509712564)  --> TIME: 2025-02-26 12:03:55 -- STEP: 12996/36938 -- GLOBAL_STEP: 1231950 | > loss_text_ce: 0.02870337665081024 (0.03133344649974745) | > loss_mel_ce: 2.8475847244262695 (nan) | > loss: 0.01141384243965149 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13722399050509174) | > loader_time: 0.0047 (0.00464991977964047)  --> TIME: 2025-02-26 12:04:05 -- STEP: 13046/36938 -- GLOBAL_STEP: 1232000 | > loss_text_ce: 0.032375384122133255 (0.03133702914878875) | > loss_mel_ce: 2.975306272506714 (nan) | > loss: 0.011935245245695114 (nan) | > current_lr: 5e-06 | > step_time: 0.1576 (0.13721765743412348) | > loader_time: 0.0046 (0.004650557313469076)  --> TIME: 2025-02-26 12:04:14 -- STEP: 13096/36938 -- GLOBAL_STEP: 1232050 | > loss_text_ce: 0.028773125261068344 (0.031336381759141316) | > loss_mel_ce: 2.4773130416870117 (nan) | > loss: 0.009944786317646503 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.13720251192898014) | > loader_time: 0.0047 (0.004650612338403426)  --> TIME: 2025-02-26 12:04:23 -- STEP: 13146/36938 -- GLOBAL_STEP: 1232100 | > loss_text_ce: 0.04548764228820801 (0.031338603533540954) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13717580331291365) | > loader_time: 0.0045 (0.004650460899320541)  --> TIME: 2025-02-26 12:04:32 -- STEP: 13196/36938 -- GLOBAL_STEP: 1232150 | > loss_text_ce: 0.03139634430408478 (0.03133946327124474) | > loss_mel_ce: 3.172515630722046 (nan) | > loss: 0.01271393708884716 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13717078255609724) | > loader_time: 0.0044 (0.004650303091775952)  --> TIME: 2025-02-26 12:04:41 -- STEP: 13246/36938 -- GLOBAL_STEP: 1232200 | > loss_text_ce: 0.03510623052716255 (0.03134522420251119) | > loss_mel_ce: 2.693505048751831 (nan) | > loss: 0.010827822610735893 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13718116598404292) | > loader_time: 0.0047 (0.004650364158989795)  --> TIME: 2025-02-26 12:04:51 -- STEP: 13296/36938 -- GLOBAL_STEP: 1232250 | > loss_text_ce: 0.02628829888999462 (0.0313430449265677) | > loss_mel_ce: 2.626126289367676 (nan) | > loss: 0.010525454767048359 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1371793388316203) | > loader_time: 0.0045 (0.004650349812817499)  --> TIME: 2025-02-26 12:05:00 -- STEP: 13346/36938 -- GLOBAL_STEP: 1232300 | > loss_text_ce: 0.030804067850112915 (0.03133752913244001) | > loss_mel_ce: 3.1642932891845703 (nan) | > loss: 0.012678958475589752 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13716165304505507) | > loader_time: 0.0046 (0.004650201072904394)  --> TIME: 2025-02-26 12:05:09 -- STEP: 13396/36938 -- GLOBAL_STEP: 1232350 | > loss_text_ce: 0.031497202813625336 (0.031344165556729656) | > loss_mel_ce: 3.1041433811187744 (nan) | > loss: 0.012443019077181816 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.13716238705070372) | > loader_time: 0.005 (0.004650533893422327)  --> TIME: 2025-02-26 12:05:19 -- STEP: 13446/36938 -- GLOBAL_STEP: 1232400 | > loss_text_ce: 0.03143363818526268 (0.03134335237686122) | > loss_mel_ce: 1.86181640625 (nan) | > loss: 0.007512897253036499 (nan) | > current_lr: 5e-06 | > step_time: 0.1629 (0.1371716441771963) | > loader_time: 0.0046 (0.0046505259205042145)  --> TIME: 2025-02-26 12:05:28 -- STEP: 13496/36938 -- GLOBAL_STEP: 1232450 | > loss_text_ce: 0.02990299090743065 (0.03134004979483541) | > loss_mel_ce: 2.906656503677368 (nan) | > loss: 0.011653014458715916 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13715009641279877) | > loader_time: 0.0045 (0.004650385936619476)  --> TIME: 2025-02-26 12:05:37 -- STEP: 13546/36938 -- GLOBAL_STEP: 1232500 | > loss_text_ce: 0.02709365263581276 (0.03133686991349621) | > loss_mel_ce: 2.3774876594543457 (nan) | > loss: 0.009541990235447884 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.13714595714348304) | > loader_time: 0.0046 (0.004650579093038711)  --> TIME: 2025-02-26 12:05:46 -- STEP: 13596/36938 -- GLOBAL_STEP: 1232550 | > loss_text_ce: 0.026612522080540657 (0.031336366323695075) | > loss_mel_ce: 2.356084108352661 (nan) | > loss: 0.00945514626801014 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1371394832122323) | > loader_time: 0.0045 (0.004650569428553051)  --> TIME: 2025-02-26 12:05:56 -- STEP: 13646/36938 -- GLOBAL_STEP: 1232600 | > loss_text_ce: 0.027617670595645905 (0.031335592492729755) | > loss_mel_ce: 2.974076986312866 (nan) | > loss: 0.011911487206816673 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13715167650462923) | > loader_time: 0.0047 (0.0046507293975686925)  --> TIME: 2025-02-26 12:06:05 -- STEP: 13696/36938 -- GLOBAL_STEP: 1232650 | > loss_text_ce: 0.02902134880423546 (0.03133492346026966) | > loss_mel_ce: 2.6300129890441895 (nan) | > loss: 0.010551723651587963 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13714956681931711) | > loader_time: 0.0046 (0.004650885970375246)  --> TIME: 2025-02-26 12:06:14 -- STEP: 13746/36938 -- GLOBAL_STEP: 1232700 | > loss_text_ce: 0.024703796952962875 (0.03133537590081739) | > loss_mel_ce: 1.830215573310852 (nan) | > loss: 0.0073607913218438625 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.1371491623465959) | > loader_time: 0.0045 (0.004650935706276393)  --> TIME: 2025-02-26 12:06:24 -- STEP: 13796/36938 -- GLOBAL_STEP: 1232750 | > loss_text_ce: 0.02698085457086563 (0.0313370381235547) | > loss_mel_ce: 3.045717477798462 (nan) | > loss: 0.012193248607218266 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13715209787083454) | > loader_time: 0.0044 (0.004651037047793463)  --> TIME: 2025-02-26 12:06:33 -- STEP: 13846/36938 -- GLOBAL_STEP: 1232800 | > loss_text_ce: 0.04112572222948074 (0.031338727475127504) | > loss_mel_ce: 2.11979341506958 (nan) | > loss: 0.0085750762373209 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13715214721043847) | > loader_time: 0.0046 (0.004651013781668631)  --> TIME: 2025-02-26 12:06:42 -- STEP: 13896/36938 -- GLOBAL_STEP: 1232850 | > loss_text_ce: 0.02908024936914444 (0.03133596343089658) | > loss_mel_ce: 2.416412353515625 (nan) | > loss: 0.00970433559268713 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13713703324689114) | > loader_time: 0.0047 (0.004651123154897694)  --> TIME: 2025-02-26 12:06:52 -- STEP: 13946/36938 -- GLOBAL_STEP: 1232900 | > loss_text_ce: 0.026088599115610123 (0.03133145169634532) | > loss_mel_ce: 2.1953132152557373 (nan) | > loss: 0.008815087378025055 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13714015181259878) | > loader_time: 0.0045 (0.0046509413026453806)  --> TIME: 2025-02-26 12:07:01 -- STEP: 13996/36938 -- GLOBAL_STEP: 1232950 | > loss_text_ce: 0.026699775829911232 (0.03133546738769839) | > loss_mel_ce: 0.3033599853515625 (nan) | > loss: 0.0013097610790282488 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1371423218481269) | > loader_time: 0.0045 (0.004650915016546081)  --> TIME: 2025-02-26 12:07:11 -- STEP: 14046/36938 -- GLOBAL_STEP: 1233000 | > loss_text_ce: 0.03296947106719017 (0.03133418101470742) | > loss_mel_ce: 2.7504262924194336 (nan) | > loss: 0.011045222170650959 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.13715263406350797) | > loader_time: 0.0047 (0.004650806830714573)  --> TIME: 2025-02-26 12:07:20 -- STEP: 14096/36938 -- GLOBAL_STEP: 1233050 | > loss_text_ce: 0.026954693719744682 (0.03133042198591316) | > loss_mel_ce: 2.534820318222046 (nan) | > loss: 0.010165774263441563 (nan) | > current_lr: 5e-06 | > step_time: 0.1447 (0.13713181999836568) | > loader_time: 0.0046 (0.004650555610250802)  --> TIME: 2025-02-26 12:07:29 -- STEP: 14146/36938 -- GLOBAL_STEP: 1233100 | > loss_text_ce: 0.03470557555556297 (0.031335368352786545) | > loss_mel_ce: 2.97243595123291 (nan) | > loss: 0.011933102272450924 (nan) | > current_lr: 5e-06 | > step_time: 0.1261 (0.1371265820359369) | > loader_time: 0.0045 (0.00465050649393503)  --> TIME: 2025-02-26 12:07:38 -- STEP: 14196/36938 -- GLOBAL_STEP: 1233150 | > loss_text_ce: 0.02795020304620266 (0.03133545007204264) | > loss_mel_ce: 2.2554290294647217 (nan) | > loss: 0.009061029180884361 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13714041072370384) | > loader_time: 0.0045 (0.004650605198496594)  --> TIME: 2025-02-26 12:07:48 -- STEP: 14246/36938 -- GLOBAL_STEP: 1233200 | > loss_text_ce: 0.027838164940476418 (0.03133954418400558) | > loss_mel_ce: 1.4935111999511719 (nan) | > loss: 0.006037100683897734 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13712765315970443) | > loader_time: 0.0046 (0.004650639162193467)  --> TIME: 2025-02-26 12:07:57 -- STEP: 14296/36938 -- GLOBAL_STEP: 1233250 | > loss_text_ce: 0.029125327244400978 (0.03133689931005714) | > loss_mel_ce: 2.500652551651001 (nan) | > loss: 0.010038801468908787 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.13713382840756938) | > loader_time: 0.0046 (0.0046505599997199686)  --> TIME: 2025-02-26 12:08:06 -- STEP: 14346/36938 -- GLOBAL_STEP: 1233300 | > loss_text_ce: 0.031124619767069817 (0.031335631637265084) | > loss_mel_ce: 2.93318247795105 (nan) | > loss: 0.011763123795390129 (nan) | > current_lr: 5e-06 | > step_time: 0.168 (0.13713467197572438) | > loader_time: 0.0047 (0.004650393141274934)  --> TIME: 2025-02-26 12:08:16 -- STEP: 14396/36938 -- GLOBAL_STEP: 1233350 | > loss_text_ce: 0.029240159317851067 (0.0313411712706392) | > loss_mel_ce: 3.1016128063201904 (nan) | > loss: 0.01242402009665966 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.1371352937063201) | > loader_time: 0.0045 (0.0046503069865170285)  --> TIME: 2025-02-26 12:08:25 -- STEP: 14446/36938 -- GLOBAL_STEP: 1233400 | > loss_text_ce: 0.0332496203482151 (0.03134294582564914) | > loss_mel_ce: 1.95846426486969 (nan) | > loss: 0.0079036271199584 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13714965286305053) | > loader_time: 0.0045 (0.004650329992280314)  --> TIME: 2025-02-26 12:08:35 -- STEP: 14496/36938 -- GLOBAL_STEP: 1233450 | > loss_text_ce: 0.033787984400987625 (0.031341442142839064) | > loss_mel_ce: 2.2067270278930664 (nan) | > loss: 0.008890933357179165 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13713921300622797) | > loader_time: 0.0046 (0.004650208712545186)  --> TIME: 2025-02-26 12:08:44 -- STEP: 14546/36938 -- GLOBAL_STEP: 1233500 | > loss_text_ce: 0.029497530311346054 (0.031341712339541306) | > loss_mel_ce: 2.9120028018951416 (nan) | > loss: 0.011672621592879295 (nan) | > current_lr: 5e-06 | > step_time: 0.1855 (0.13713690417678) | > loader_time: 0.0047 (0.004649913509331655)  --> TIME: 2025-02-26 12:08:53 -- STEP: 14596/36938 -- GLOBAL_STEP: 1233550 | > loss_text_ce: 0.036283742636442184 (0.03134068339420199) | > loss_mel_ce: 0.006195068359375 (nan) | > loss: 0.00016856672300491482 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.13714734112670832) | > loader_time: 0.0046 (0.004649784261803326)  --> TIME: 2025-02-26 12:09:02 -- STEP: 14646/36938 -- GLOBAL_STEP: 1233600 | > loss_text_ce: 0.029326269403100014 (0.03134237648715762) | > loss_mel_ce: 3.2083466053009033 (nan) | > loss: 0.012847908772528172 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13712751064472453) | > loader_time: 0.0046 (0.004649663580321847)  --> TIME: 2025-02-26 12:09:12 -- STEP: 14696/36938 -- GLOBAL_STEP: 1233650 | > loss_text_ce: 0.034864090383052826 (0.03134379334293811) | > loss_mel_ce: 2.885601758956909 (nan) | > loss: 0.011589150875806808 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13713184435470127) | > loader_time: 0.0047 (0.004649612036618318)  --> TIME: 2025-02-26 12:09:21 -- STEP: 14746/36938 -- GLOBAL_STEP: 1233700 | > loss_text_ce: 0.027660373598337173 (0.03134378324669937) | > loss_mel_ce: 2.412602424621582 (nan) | > loss: 0.009683582931756973 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13712656952286134) | > loader_time: 0.0045 (0.004649534132334134)  --> TIME: 2025-02-26 12:09:30 -- STEP: 14796/36938 -- GLOBAL_STEP: 1233750 | > loss_text_ce: 0.03913756459951401 (0.031340453646904604) | > loss_mel_ce: 2.7393670082092285 (nan) | > loss: 0.011025812476873398 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13711824459654476) | > loader_time: 0.0045 (0.004649954136721858)  --> TIME: 2025-02-26 12:09:40 -- STEP: 14846/36938 -- GLOBAL_STEP: 1233800 | > loss_text_ce: 0.03555678203701973 (0.031341228828526455) | > loss_mel_ce: 2.880113363265991 (nan) | > loss: 0.011570120230317116 (nan) | > current_lr: 5e-06 | > step_time: 0.1853 (0.13712486250131073) | > loader_time: 0.0049 (0.0046500776489516804)  --> TIME: 2025-02-26 12:09:49 -- STEP: 14896/36938 -- GLOBAL_STEP: 1233850 | > loss_text_ce: 0.03896206244826317 (0.0313427316575218) | > loss_mel_ce: 2.5626394748687744 (nan) | > loss: 0.010323816910386086 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13711989607770017) | > loader_time: 0.0045 (0.0046501437844318405)  --> TIME: 2025-02-26 12:09:58 -- STEP: 14946/36938 -- GLOBAL_STEP: 1233900 | > loss_text_ce: 0.042743366211652756 (0.031338695301186714) | > loss_mel_ce: 2.596940279006958 (nan) | > loss: 0.010474936105310917 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13712153111884984) | > loader_time: 0.0045 (0.004649913647591818)  --> TIME: 2025-02-26 12:10:07 -- STEP: 14996/36938 -- GLOBAL_STEP: 1233950 | > loss_text_ce: 0.027393179014325142 (0.0313373718297206) | > loss_mel_ce: 2.0224075317382812 (nan) | > loss: 0.008134130388498306 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13710749230406477) | > loader_time: 0.0045 (0.004650022672570268)  --> TIME: 2025-02-26 12:10:17 -- STEP: 15046/36938 -- GLOBAL_STEP: 1234000 | > loss_text_ce: 0.03178749233484268 (0.03133743717213258) | > loss_mel_ce: 2.9043078422546387 (nan) | > loss: 0.011651172302663326 (nan) | > current_lr: 5e-06 | > step_time: 0.2047 (0.13710611591975513) | > loader_time: 0.0048 (0.004650168607134489)  --> TIME: 2025-02-26 12:10:26 -- STEP: 15096/36938 -- GLOBAL_STEP: 1234050 | > loss_text_ce: 0.02524927817285061 (0.03133470421804076) | > loss_mel_ce: 2.5183610916137695 (nan) | > loss: 0.010093692690134048 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.1371203795209664) | > loader_time: 0.0046 (0.004650210743551854)  --> TIME: 2025-02-26 12:10:36 -- STEP: 15146/36938 -- GLOBAL_STEP: 1234100 | > loss_text_ce: 0.04056508466601372 (0.03132893585842382) | > loss_mel_ce: 3.0936617851257324 (nan) | > loss: 0.012437408789992332 (nan) | > current_lr: 5e-06 | > step_time: 0.1871 (0.13714334424194058) | > loader_time: 0.0049 (0.004650296173841212)  --> TIME: 2025-02-26 12:10:45 -- STEP: 15196/36938 -- GLOBAL_STEP: 1234150 | > loss_text_ce: 0.034379009157419205 (0.03132852328800047) | > loss_mel_ce: 2.802880048751831 (nan) | > loss: 0.011258965358138084 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13713346655100855) | > loader_time: 0.0047 (0.004650455316326183)  --> TIME: 2025-02-26 12:10:54 -- STEP: 15246/36938 -- GLOBAL_STEP: 1234200 | > loss_text_ce: 0.032181307673454285 (0.03133266844076782) | > loss_mel_ce: 1.431180477142334 (nan) | > loss: 0.0058069913648068905 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13711413041697448) | > loader_time: 0.0045 (0.004650497004911683)  --> TIME: 2025-02-26 12:11:04 -- STEP: 15296/36938 -- GLOBAL_STEP: 1234250 | > loss_text_ce: 0.03161763772368431 (0.031336405574552306) | > loss_mel_ce: 2.2551522254943848 (nan) | > loss: 0.009074483998119831 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.1371065081256955) | > loader_time: 0.0045 (0.0046504805152885065)  --> TIME: 2025-02-26 12:11:13 -- STEP: 15346/36938 -- GLOBAL_STEP: 1234300 | > loss_text_ce: 0.04413062334060669 (0.0313364310758976) | > loss_mel_ce: 2.3961594104766846 (nan) | > loss: 0.009683690965175629 (nan) | > current_lr: 5e-06 | > step_time: 0.1271 (0.13710749216128212) | > loader_time: 0.0047 (0.00465041876740333)  --> TIME: 2025-02-26 12:11:22 -- STEP: 15396/36938 -- GLOBAL_STEP: 1234350 | > loss_text_ce: 0.023442458361387253 (0.03133718313260484) | > loss_mel_ce: 3.025374174118042 (nan) | > loss: 0.012098480015993118 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13711360300569073) | > loader_time: 0.0046 (0.004650389088937664)  --> TIME: 2025-02-26 12:11:32 -- STEP: 15446/36938 -- GLOBAL_STEP: 1234400 | > loss_text_ce: 0.03263317421078682 (0.03133649376331852) | > loss_mel_ce: 2.402885675430298 (nan) | > loss: 0.009664757177233696 (nan) | > current_lr: 5e-06 | > step_time: 0.1576 (0.13710282806842045) | > loader_time: 0.0046 (0.004650301410329783)  --> TIME: 2025-02-26 12:11:41 -- STEP: 15496/36938 -- GLOBAL_STEP: 1234450 | > loss_text_ce: 0.034567125141620636 (0.03133807810213133) | > loss_mel_ce: 2.780224084854126 (nan) | > loss: 0.01116980705410242 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.13711289656057665) | > loader_time: 0.0047 (0.004650174986780703)  --> TIME: 2025-02-26 12:11:50 -- STEP: 15546/36938 -- GLOBAL_STEP: 1234500 | > loss_text_ce: 0.029709283262491226 (0.03134281472684612) | > loss_mel_ce: 2.8762130737304688 (nan) | > loss: 0.011531438678503036 (nan) | > current_lr: 5e-06 | > step_time: 0.1958 (0.1371073037612154) | > loader_time: 0.0048 (0.004650228704155356)  --> TIME: 2025-02-26 12:12:00 -- STEP: 15596/36938 -- GLOBAL_STEP: 1234550 | > loss_text_ce: 0.03946034237742424 (0.03134224602819345) | > loss_mel_ce: 3.3413760662078857 (nan) | > loss: 0.013416018337011337 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.1371151813790568) | > loader_time: 0.0046 (0.004650154490445202)  --> TIME: 2025-02-26 12:12:10 -- STEP: 15646/36938 -- GLOBAL_STEP: 1234600 | > loss_text_ce: 0.03772442415356636 (0.03134458469257704) | > loss_mel_ce: 2.3798933029174805 (nan) | > loss: 0.00959372241050005 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13713139180528797) | > loader_time: 0.0046 (0.004650388290613)  --> TIME: 2025-02-26 12:12:19 -- STEP: 15696/36938 -- GLOBAL_STEP: 1234650 | > loss_text_ce: 0.027751155197620392 (0.03133918497643575) | > loss_mel_ce: 3.0688812732696533 (nan) | > loss: 0.012288224883377552 (nan) | > current_lr: 5e-06 | > step_time: 0.1448 (0.1371269606881674) | > loader_time: 0.0047 (0.004650264051833039)  --> TIME: 2025-02-26 12:12:29 -- STEP: 15746/36938 -- GLOBAL_STEP: 1234700 | > loss_text_ce: 0.031185800209641457 (0.03133776470182255) | > loss_mel_ce: 2.6476094722747803 (nan) | > loss: 0.010630141012370586 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.1371431213071475) | > loader_time: 0.0046 (0.004650293425563993)  --> TIME: 2025-02-26 12:12:38 -- STEP: 15796/36938 -- GLOBAL_STEP: 1234750 | > loss_text_ce: 0.030160704627633095 (0.03133866124805095) | > loss_mel_ce: 2.7323858737945557 (nan) | > loss: 0.010962487198412418 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13715736530617445) | > loader_time: 0.0046 (0.0046505903738847275)  --> TIME: 2025-02-26 12:12:47 -- STEP: 15846/36938 -- GLOBAL_STEP: 1234800 | > loss_text_ce: 0.032133057713508606 (0.03133627797909428) | > loss_mel_ce: 2.7357020378112793 (nan) | > loss: 0.010983473621308804 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13714805823541235) | > loader_time: 0.0045 (0.004650384402103559)  --> TIME: 2025-02-26 12:12:57 -- STEP: 15896/36938 -- GLOBAL_STEP: 1234850 | > loss_text_ce: 0.026783877983689308 (0.031336092277688986) | > loss_mel_ce: 2.791337251663208 (nan) | > loss: 0.01118302159011364 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13715810029123943) | > loader_time: 0.0046 (0.00465020915342445)  --> TIME: 2025-02-26 12:13:06 -- STEP: 15946/36938 -- GLOBAL_STEP: 1234900 | > loss_text_ce: 0.03529597073793411 (0.0313346606294259) | > loss_mel_ce: 3.3911402225494385 (nan) | > loss: 0.013596969656646252 (nan) | > current_lr: 5e-06 | > step_time: 0.1632 (0.13715396593673543) | > loader_time: 0.0046 (0.004650175758336935)  --> TIME: 2025-02-26 12:13:15 -- STEP: 15996/36938 -- GLOBAL_STEP: 1234950 | > loss_text_ce: 0.019387969747185707 (0.031334118844632206) | > loss_mel_ce: 0.8583126664161682 (nan) | > loss: 0.0034829392097890377 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13715529872584767) | > loader_time: 0.0045 (0.004649963474893723)  --> TIME: 2025-02-26 12:13:25 -- STEP: 16046/36938 -- GLOBAL_STEP: 1235000 | > loss_text_ce: 0.027582792565226555 (0.031335261563960944) | > loss_mel_ce: 3.01669979095459 (nan) | > loss: 0.012080487795174122 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.1371549591375651) | > loader_time: 0.0046 (0.0046501183144308945)  --> TIME: 2025-02-26 12:13:34 -- STEP: 16096/36938 -- GLOBAL_STEP: 1235050 | > loss_text_ce: 0.029706614091992378 (0.031337242988301824) | > loss_mel_ce: 3.306819438934326 (nan) | > loss: 0.01324018370360136 (nan) | > current_lr: 5e-06 | > step_time: 0.1771 (0.1371615679553207) | > loader_time: 0.0048 (0.004650002697234598)  --> TIME: 2025-02-26 12:13:43 -- STEP: 16146/36938 -- GLOBAL_STEP: 1235100 | > loss_text_ce: 0.02506200782954693 (0.03133683143126855) | > loss_mel_ce: 0.016815185546875 (nan) | > loss: 0.00016617935034446418 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13714313902820097) | > loader_time: 0.0045 (0.004649746702991126)  --> TIME: 2025-02-26 12:13:52 -- STEP: 16196/36938 -- GLOBAL_STEP: 1235150 | > loss_text_ce: 0.041202593594789505 (0.031336745769892066) | > loss_mel_ce: 2.522089958190918 (nan) | > loss: 0.010171796195209026 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13713762867035612) | > loader_time: 0.0046 (0.004649967404170281)  --> TIME: 2025-02-26 12:14:02 -- STEP: 16246/36938 -- GLOBAL_STEP: 1235200 | > loss_text_ce: 0.0308645311743021 (0.031339079119890395) | > loss_mel_ce: 1.5085405111312866 (nan) | > loss: 0.006108750123530626 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.1371366068249369) | > loader_time: 0.0044 (0.004649885018059995)  --> TIME: 2025-02-26 12:14:11 -- STEP: 16296/36938 -- GLOBAL_STEP: 1235250 | > loss_text_ce: 0.03515951707959175 (0.031337300908091795) | > loss_mel_ce: 2.404865026473999 (nan) | > loss: 0.009682637639343739 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13710877979626912) | > loader_time: 0.0047 (0.004649700168072151)  --> TIME: 2025-02-26 12:14:20 -- STEP: 16346/36938 -- GLOBAL_STEP: 1235300 | > loss_text_ce: 0.04109993204474449 (0.03134249743435654) | > loss_mel_ce: 3.2378592491149902 (nan) | > loss: 0.013011744245886803 (nan) | > current_lr: 5e-06 | > step_time: 0.1824 (0.13712038936075926) | > loader_time: 0.0049 (0.004649687991882643)  --> TIME: 2025-02-26 12:14:30 -- STEP: 16396/36938 -- GLOBAL_STEP: 1235350 | > loss_text_ce: 0.030275823548436165 (0.031344973754386087) | > loss_mel_ce: 3.327894687652588 (nan) | > loss: 0.013326073996722698 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13713016749068185) | > loader_time: 0.0045 (0.004649720415309393)  --> TIME: 2025-02-26 12:14:39 -- STEP: 16446/36938 -- GLOBAL_STEP: 1235400 | > loss_text_ce: 0.03741438314318657 (0.03134321230601315) | > loss_mel_ce: 2.6377506256103516 (nan) | > loss: 0.01061573438346386 (nan) | > current_lr: 5e-06 | > step_time: 0.1693 (0.1371371604137059) | > loader_time: 0.0048 (0.00464970464183367)  --> TIME: 2025-02-26 12:14:49 -- STEP: 16496/36938 -- GLOBAL_STEP: 1235450 | > loss_text_ce: 0.02528848871588707 (0.031344188944433214) | > loss_mel_ce: 2.526123046875 (nan) | > loss: 0.010124649852514267 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13714841105022232) | > loader_time: 0.0045 (0.004649573657029351)  --> TIME: 2025-02-26 12:14:58 -- STEP: 16546/36938 -- GLOBAL_STEP: 1235500 | > loss_text_ce: 0.034288451075553894 (0.03134150137755829) | > loss_mel_ce: 2.8775904178619385 (nan) | > loss: 0.011555075645446777 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13713179291137623) | > loader_time: 0.0046 (0.0046493327849637224)  --> TIME: 2025-02-26 12:15:07 -- STEP: 16596/36938 -- GLOBAL_STEP: 1235550 | > loss_text_ce: 0.02954108454287052 (0.03133735248398586) | > loss_mel_ce: 1.9882158041000366 (nan) | > loss: 0.008006972260773182 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1371139053495055) | > loader_time: 0.0045 (0.004649190090742131)  --> TIME: 2025-02-26 12:15:16 -- STEP: 16646/36938 -- GLOBAL_STEP: 1235600 | > loss_text_ce: 0.03301578387618065 (0.03133244052827899) | > loss_mel_ce: 2.945600748062134 (nan) | > loss: 0.01181990746408701 (nan) | > current_lr: 5e-06 | > step_time: 0.1697 (0.13710158383718832) | > loader_time: 0.0048 (0.004649080551832208)  --> TIME: 2025-02-26 12:15:25 -- STEP: 16696/36938 -- GLOBAL_STEP: 1235650 | > loss_text_ce: 0.020120419561862946 (0.03133094627368916) | > loss_mel_ce: 0.0009643009980209172 (nan) | > loss: 8.366953261429444e-05 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13709786895592954) | > loader_time: 0.0045 (0.00464902953148802)  --> TIME: 2025-02-26 12:15:34 -- STEP: 16746/36938 -- GLOBAL_STEP: 1235700 | > loss_text_ce: 0.03208751231431961 (0.03133171162296987) | > loss_mel_ce: 3.167219638824463 (nan) | > loss: 0.012695664539933205 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13709408716105898) | > loader_time: 0.0047 (0.004649084157931196)  --> TIME: 2025-02-26 12:15:44 -- STEP: 16796/36938 -- GLOBAL_STEP: 1235750 | > loss_text_ce: 0.032404378056526184 (0.031331470880833295) | > loss_mel_ce: 2.3809566497802734 (nan) | > loss: 0.009576830081641674 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13708131462087195) | > loader_time: 0.0046 (0.0046489913283600404)  --> TIME: 2025-02-26 12:15:53 -- STEP: 16846/36938 -- GLOBAL_STEP: 1235800 | > loss_text_ce: 0.030652975663542747 (0.03133389893970111) | > loss_mel_ce: 2.8921542167663574 (nan) | > loss: 0.011598441749811172 (nan) | > current_lr: 5e-06 | > step_time: 0.1814 (0.1370900740154168) | > loader_time: 0.0048 (0.004649003016528362)  --> TIME: 2025-02-26 12:16:02 -- STEP: 16896/36938 -- GLOBAL_STEP: 1235850 | > loss_text_ce: 0.033523961901664734 (0.03133566015491294) | > loss_mel_ce: 2.99454402923584 (nan) | > loss: 0.012016143649816513 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13708987466595862) | > loader_time: 0.0045 (0.004648896399885415)  --> TIME: 2025-02-26 12:16:12 -- STEP: 16946/36938 -- GLOBAL_STEP: 1235900 | > loss_text_ce: 0.036738380789756775 (0.031335299557839466) | > loss_mel_ce: 2.402559280395508 (nan) | > loss: 0.009679753333330154 (nan) | > current_lr: 5e-06 | > step_time: 0.1519 (0.13709775562707358) | > loader_time: 0.0046 (0.004648899787237408)  --> TIME: 2025-02-26 12:16:21 -- STEP: 16996/36938 -- GLOBAL_STEP: 1235950 | > loss_text_ce: 0.04295540228486061 (0.031334664347888855) | > loss_mel_ce: 1.9939051866531372 (nan) | > loss: 0.008082780987024307 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.1370911243439396) | > loader_time: 0.0045 (0.004648717719545138)  --> TIME: 2025-02-26 12:16:31 -- STEP: 17046/36938 -- GLOBAL_STEP: 1236000 | > loss_text_ce: 0.033833473920822144 (0.031333714551702876) | > loss_mel_ce: 3.150446653366089 (nan) | > loss: 0.01263603288680315 (nan) | > current_lr: 5e-06 | > step_time: 0.1436 (0.13710555791938803) | > loader_time: 0.0047 (0.0046491554529471376)  --> TIME: 2025-02-26 12:16:40 -- STEP: 17096/36938 -- GLOBAL_STEP: 1236050 | > loss_text_ce: 0.03357510641217232 (0.03133777378320498) | > loss_mel_ce: 3.1224358081817627 (nan) | > loss: 0.012523853220045567 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13710318185706336) | > loader_time: 0.0046 (0.004649067446624335)  --> TIME: 2025-02-26 12:16:49 -- STEP: 17146/36938 -- GLOBAL_STEP: 1236100 | > loss_text_ce: 0.025287603959441185 (0.031339127338999656) | > loss_mel_ce: 1.6092720031738281 (nan) | > loss: 0.00648634834215045 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13708640047345041) | > loader_time: 0.0047 (0.004649300176455181)  --> TIME: 2025-02-26 12:16:58 -- STEP: 17196/36938 -- GLOBAL_STEP: 1236150 | > loss_text_ce: 0.028072809800505638 (0.031340638630219714) | > loss_mel_ce: 2.516530990600586 (nan) | > loss: 0.010097634978592396 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13707674729544028) | > loader_time: 0.0048 (0.004649280198481673)  --> TIME: 2025-02-26 12:17:07 -- STEP: 17246/36938 -- GLOBAL_STEP: 1236200 | > loss_text_ce: 0.02413812093436718 (0.031340012540091604) | > loss_mel_ce: 2.4482920169830322 (nan) | > loss: 0.00981123186647892 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13707493626987088) | > loader_time: 0.0046 (0.0046494417423910556)  --> TIME: 2025-02-26 12:17:17 -- STEP: 17296/36938 -- GLOBAL_STEP: 1236250 | > loss_text_ce: 0.029062217101454735 (0.031334743224040194) | > loss_mel_ce: 2.720123291015625 (nan) | > loss: 0.01090946700423956 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13706200875197191) | > loader_time: 0.0054 (0.00464953694433994)  --> TIME: 2025-02-26 12:17:26 -- STEP: 17346/36938 -- GLOBAL_STEP: 1236300 | > loss_text_ce: 0.03140140697360039 (0.03133839140306314) | > loss_mel_ce: 2.6179258823394775 (nan) | > loss: 0.010513204149901867 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.1370622710374545) | > loader_time: 0.0046 (0.004649464019947806)  --> TIME: 2025-02-26 12:17:35 -- STEP: 17396/36938 -- GLOBAL_STEP: 1236350 | > loss_text_ce: 0.024705182760953903 (0.03133976854030407) | > loss_mel_ce: 2.8390352725982666 (nan) | > loss: 0.011364050209522247 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1370573315052476) | > loader_time: 0.0045 (0.004649330046182941)  --> TIME: 2025-02-26 12:17:45 -- STEP: 17446/36938 -- GLOBAL_STEP: 1236400 | > loss_text_ce: 0.031941719353199005 (0.03134600804923528) | > loss_mel_ce: 1.5297088623046875 (nan) | > loss: 0.006197026465088129 (nan) | > current_lr: 5e-06 | > step_time: 0.1496 (0.1370784372388535) | > loader_time: 0.0047 (0.004649324194630338)  --> TIME: 2025-02-26 12:17:54 -- STEP: 17496/36938 -- GLOBAL_STEP: 1236450 | > loss_text_ce: 0.022999603301286697 (0.03134999383865028) | > loss_mel_ce: 2.0927581787109375 (nan) | > loss: 0.008395864628255367 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13706734385401068) | > loader_time: 0.0045 (0.004649498716684908)  --> TIME: 2025-02-26 12:18:04 -- STEP: 17546/36938 -- GLOBAL_STEP: 1236500 | > loss_text_ce: 0.03274732828140259 (0.031350256257053594) | > loss_mel_ce: 2.5128390789031982 (nan) | > loss: 0.010101533494889736 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13707528986137885) | > loader_time: 0.0046 (0.004649508275260648)  --> TIME: 2025-02-26 12:18:13 -- STEP: 17596/36938 -- GLOBAL_STEP: 1236550 | > loss_text_ce: 0.03620631992816925 (0.031348255223127494) | > loss_mel_ce: 0.0639190673828125 (nan) | > loss: 0.00039732299046590924 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.1370615596580466) | > loader_time: 0.0046 (0.004649554133062932)  --> TIME: 2025-02-26 12:18:22 -- STEP: 17646/36938 -- GLOBAL_STEP: 1236600 | > loss_text_ce: 0.023452462628483772 (0.03134790785378676) | > loss_mel_ce: 2.7873528003692627 (nan) | > loss: 0.011153990402817726 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.1370492609371667) | > loader_time: 0.0046 (0.004649555022438459)  --> TIME: 2025-02-26 12:18:31 -- STEP: 17696/36938 -- GLOBAL_STEP: 1236650 | > loss_text_ce: 0.03204625844955444 (0.03134904965690131) | > loss_mel_ce: 2.7781941890716553 (nan) | > loss: 0.011151748709380627 (nan) | > current_lr: 5e-06 | > step_time: 0.1798 (0.13704546400856188) | > loader_time: 0.0055 (0.0046497921427378335)  --> TIME: 2025-02-26 12:18:40 -- STEP: 17746/36938 -- GLOBAL_STEP: 1236700 | > loss_text_ce: 0.029670558869838715 (0.03135012723984134) | > loss_mel_ce: 0.45391845703125 (nan) | > loss: 0.0019190041348338127 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1370295929852329) | > loader_time: 0.0045 (0.004649989032347679)  --> TIME: 2025-02-26 12:18:49 -- STEP: 17796/36938 -- GLOBAL_STEP: 1236750 | > loss_text_ce: 0.03023654595017433 (0.031349882230608216) | > loss_mel_ce: 2.7597668170928955 (nan) | > loss: 0.011071442626416683 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13701796558471907) | > loader_time: 0.0046 (0.00464991481954206)  --> TIME: 2025-02-26 12:18:58 -- STEP: 17846/36938 -- GLOBAL_STEP: 1236800 | > loss_text_ce: 0.030634567141532898 (0.031351484165846565) | > loss_mel_ce: 1.5413066148757935 (nan) | > loss: 0.006237862166017294 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13701359779193784) | > loader_time: 0.0046 (0.0046499799241883875)  --> TIME: 2025-02-26 12:19:08 -- STEP: 17896/36938 -- GLOBAL_STEP: 1236850 | > loss_text_ce: 0.032708168029785156 (0.031355390806566394) | > loss_mel_ce: 2.8862013816833496 (nan) | > loss: 0.011582975275814533 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.1370127096765537) | > loader_time: 0.0056 (0.004650138468416235)  --> TIME: 2025-02-26 12:19:17 -- STEP: 17946/36938 -- GLOBAL_STEP: 1236900 | > loss_text_ce: 0.03960469737648964 (0.03135544697488496) | > loss_mel_ce: 2.55916690826416 (nan) | > loss: 0.010312586091458797 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1370071793670684) | > loader_time: 0.0046 (0.004650714271751172)  --> TIME: 2025-02-26 12:19:26 -- STEP: 17996/36938 -- GLOBAL_STEP: 1236950 | > loss_text_ce: 0.028322886675596237 (0.031354720990858245) | > loss_mel_ce: 2.880703926086426 (nan) | > loss: 0.01154375821352005 (nan) | > current_lr: 5e-06 | > step_time: 0.1736 (0.13700749682542876) | > loader_time: 0.0047 (0.004650805202635903)  --> TIME: 2025-02-26 12:19:36 -- STEP: 18046/36938 -- GLOBAL_STEP: 1237000 | > loss_text_ce: 0.04201596602797508 (0.031356026879988286) | > loss_mel_ce: 2.7073516845703125 (nan) | > loss: 0.01091018971055746 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13701910953041924) | > loader_time: 0.0046 (0.0046509587552031354)  --> TIME: 2025-02-26 12:19:45 -- STEP: 18096/36938 -- GLOBAL_STEP: 1237050 | > loss_text_ce: 0.027041640132665634 (0.03135299496795294) | > loss_mel_ce: 2.8686838150024414 (nan) | > loss: 0.011490974575281143 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.1370253007592199) | > loader_time: 0.0047 (0.004651084937530736)  --> TIME: 2025-02-26 12:19:55 -- STEP: 18146/36938 -- GLOBAL_STEP: 1237100 | > loss_text_ce: 0.02316662296652794 (0.03135290243370879) | > loss_mel_ce: 2.5335874557495117 (nan) | > loss: 0.010145850479602814 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13702536286217795) | > loader_time: 0.0046 (0.004651119122231367)  --> TIME: 2025-02-26 12:20:04 -- STEP: 18196/36938 -- GLOBAL_STEP: 1237150 | > loss_text_ce: 0.023562707006931305 (0.031355109707296734) | > loss_mel_ce: 2.9146029949188232 (nan) | > loss: 0.011659388430416584 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.1370141471996762) | > loader_time: 0.0046 (0.0046512526479441305)  --> TIME: 2025-02-26 12:20:14 -- STEP: 18246/36938 -- GLOBAL_STEP: 1237200 | > loss_text_ce: 0.02680264413356781 (0.03136002789255096) | > loss_mel_ce: 2.1792194843292236 (nan) | > loss: 0.008754055947065353 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13702049059890398) | > loader_time: 0.0046 (0.004651204857350064)  --> TIME: 2025-02-26 12:20:23 -- STEP: 18296/36938 -- GLOBAL_STEP: 1237250 | > loss_text_ce: 0.03425897657871246 (0.03135491922979512) | > loss_mel_ce: 2.987244129180908 (nan) | > loss: 0.011990092694759369 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13700739341528656) | > loader_time: 0.0046 (0.0046510416892192935)  --> TIME: 2025-02-26 12:20:32 -- STEP: 18346/36938 -- GLOBAL_STEP: 1237300 | > loss_text_ce: 0.03786113113164902 (0.03135264431545034) | > loss_mel_ce: 2.4000766277313232 (nan) | > loss: 0.009674356319010258 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13701238118093476) | > loader_time: 0.0048 (0.004651167589485156)  --> TIME: 2025-02-26 12:20:42 -- STEP: 18396/36938 -- GLOBAL_STEP: 1237350 | > loss_text_ce: 0.02747078612446785 (0.031347544361707946) | > loss_mel_ce: 2.9959146976470947 (nan) | > loss: 0.011997561901807785 (nan) | > current_lr: 5e-06 | > step_time: 0.1121 (0.13702506220944755) | > loader_time: 0.0046 (0.004651131695263992)  --> TIME: 2025-02-26 12:20:51 -- STEP: 18446/36938 -- GLOBAL_STEP: 1237400 | > loss_text_ce: 0.03163198381662369 (0.03134572198633677) | > loss_mel_ce: 2.308584213256836 (nan) | > loss: 0.009286572225391865 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.13703389598454) | > loader_time: 0.0048 (0.004651000090512914)  --> TIME: 2025-02-26 12:21:01 -- STEP: 18496/36938 -- GLOBAL_STEP: 1237450 | > loss_text_ce: 0.029681554064154625 (0.03134526483681162) | > loss_mel_ce: 1.0139644145965576 (nan) | > loss: 0.004141452722251415 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13703984247720438) | > loader_time: 0.0045 (0.004650813524182683)  --> TIME: 2025-02-26 12:21:10 -- STEP: 18546/36938 -- GLOBAL_STEP: 1237500 | > loss_text_ce: 0.034735772758722305 (0.03134384358483158) | > loss_mel_ce: 3.081061363220215 (nan) | > loss: 0.012364274822175503 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.13705573111023775) | > loader_time: 0.0046 (0.004650826825944388)  --> TIME: 2025-02-26 12:21:20 -- STEP: 18596/36938 -- GLOBAL_STEP: 1237550 | > loss_text_ce: 0.019970960915088654 (0.031343791294690074) | > loss_mel_ce: 1.8180971145629883 (nan) | > loss: 0.0072939214296638966 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1370581095258154) | > loader_time: 0.0045 (0.004650881403773067)  --> TIME: 2025-02-26 12:21:30 -- STEP: 18646/36938 -- GLOBAL_STEP: 1237600 | > loss_text_ce: 0.030023040249943733 (0.03134292406066476) | > loss_mel_ce: 1.8235787153244019 (nan) | > loss: 0.0073555633425712585 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13707303986789326) | > loader_time: 0.0044 (0.004650985403121252)  --> TIME: 2025-02-26 12:21:39 -- STEP: 18696/36938 -- GLOBAL_STEP: 1237650 | > loss_text_ce: 0.03412336856126785 (0.03134049074200288) | > loss_mel_ce: 3.0314290523529053 (nan) | > loss: 0.012164891697466373 (nan) | > current_lr: 5e-06 | > step_time: 0.1693 (0.13708224863285864) | > loader_time: 0.0055 (0.00465116842108298)  --> TIME: 2025-02-26 12:21:48 -- STEP: 18746/36938 -- GLOBAL_STEP: 1237700 | > loss_text_ce: 0.030538074672222137 (0.031340329797443195) | > loss_mel_ce: 2.0711162090301514 (nan) | > loss: 0.008339898660779 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13708320450505584) | > loader_time: 0.0046 (0.004651179426090053)  --> TIME: 2025-02-26 12:21:58 -- STEP: 18796/36938 -- GLOBAL_STEP: 1237750 | > loss_text_ce: 0.03222699090838432 (0.03133957954030479) | > loss_mel_ce: 2.6588969230651855 (nan) | > loss: 0.010679064318537712 (nan) | > current_lr: 5e-06 | > step_time: 0.1467 (0.13708327645620325) | > loader_time: 0.0045 (0.004651122307315674)  --> TIME: 2025-02-26 12:22:07 -- STEP: 18846/36938 -- GLOBAL_STEP: 1237800 | > loss_text_ce: 0.021959427744150162 (0.03134298331256342) | > loss_mel_ce: 2.3375704288482666 (nan) | > loss: 0.009363214485347271 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13709193308060244) | > loader_time: 0.0046 (0.00465096363934768)  --> TIME: 2025-02-26 12:22:16 -- STEP: 18896/36938 -- GLOBAL_STEP: 1237850 | > loss_text_ce: 0.026823120191693306 (0.03134156881432773) | > loss_mel_ce: 2.461148977279663 (nan) | > loss: 0.009872904978692532 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1370798259418366) | > loader_time: 0.0045 (0.004650727734537459)  --> TIME: 2025-02-26 12:22:26 -- STEP: 18946/36938 -- GLOBAL_STEP: 1237900 | > loss_text_ce: 0.029964208602905273 (0.03134106083172494) | > loss_mel_ce: 2.834451198577881 (nan) | > loss: 0.011366728693246841 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13708914355643814) | > loader_time: 0.0046 (0.004650664986419423)  --> TIME: 2025-02-26 12:22:35 -- STEP: 18996/36938 -- GLOBAL_STEP: 1237950 | > loss_text_ce: 0.02132541313767433 (0.031337520091153884) | > loss_mel_ce: 2.353107213973999 (nan) | > loss: 0.009422351606190205 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13709240708759857) | > loader_time: 0.0045 (0.0046506350254806525)  --> TIME: 2025-02-26 12:22:44 -- STEP: 19046/36938 -- GLOBAL_STEP: 1238000 | > loss_text_ce: 0.03997081518173218 (0.03134258167017637) | > loss_mel_ce: 2.928407907485962 (nan) | > loss: 0.011779281310737133 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13708592687949564) | > loader_time: 0.0046 (0.00465053782472586)  --> TIME: 2025-02-26 12:22:54 -- STEP: 19096/36938 -- GLOBAL_STEP: 1238050 | > loss_text_ce: 0.03328276425600052 (0.031344270020345795) | > loss_mel_ce: 2.4498867988586426 (nan) | > loss: 0.00985384825617075 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13707840974529728) | > loader_time: 0.0046 (0.004650506206172041)  --> TIME: 2025-02-26 12:23:03 -- STEP: 19146/36938 -- GLOBAL_STEP: 1238100 | > loss_text_ce: 0.03081018291413784 (0.03134395811964797) | > loss_mel_ce: 2.4750900268554688 (nan) | > loss: 0.009944048710167408 (nan) | > current_lr: 5e-06 | > step_time: 0.1066 (0.13708802343199428) | > loader_time: 0.0044 (0.004650405827303799)  --> TIME: 2025-02-26 12:23:13 -- STEP: 19196/36938 -- GLOBAL_STEP: 1238150 | > loss_text_ce: 0.036433905363082886 (0.03134368960075145) | > loss_mel_ce: 2.6905345916748047 (nan) | > loss: 0.01082130428403616 (nan) | > current_lr: 5e-06 | > step_time: 0.1505 (0.13709216326270265) | > loader_time: 0.0046 (0.004650281279950016)  --> TIME: 2025-02-26 12:23:22 -- STEP: 19246/36938 -- GLOBAL_STEP: 1238200 | > loss_text_ce: 0.03292063996195793 (0.031345473942986195) | > loss_mel_ce: 2.4303979873657227 (nan) | > loss: 0.009775074198842049 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.13709011580088595) | > loader_time: 0.0047 (0.004650138376607198)  --> TIME: 2025-02-26 12:23:32 -- STEP: 19296/36938 -- GLOBAL_STEP: 1238250 | > loss_text_ce: 0.03616446256637573 (0.031346720221408456) | > loss_mel_ce: 2.459428548812866 (nan) | > loss: 0.00990314781665802 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.1370983915171818) | > loader_time: 0.0046 (0.00465006571553437)  --> TIME: 2025-02-26 12:23:41 -- STEP: 19346/36938 -- GLOBAL_STEP: 1238300 | > loss_text_ce: 0.027801234275102615 (0.03134817731416825) | > loss_mel_ce: 0.709991455078125 (nan) | > loss: 0.002927748952060938 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13708316785626484) | > loader_time: 0.0046 (0.004650398431070262)  --> TIME: 2025-02-26 12:23:50 -- STEP: 19396/36938 -- GLOBAL_STEP: 1238350 | > loss_text_ce: 0.026965387165546417 (0.03134671542657152) | > loss_mel_ce: 0.0769246444106102 (nan) | > loss: 0.00041226204484701157 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13708553608728996) | > loader_time: 0.0046 (0.004650418771667854)  --> TIME: 2025-02-26 12:23:59 -- STEP: 19446/36938 -- GLOBAL_STEP: 1238400 | > loss_text_ce: 0.041071124374866486 (0.031348726507377876) | > loss_mel_ce: 3.283599853515625 (nan) | > loss: 0.013193139806389809 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.1370840263651148) | > loader_time: 0.0047 (0.004650594079051866)  --> TIME: 2025-02-26 12:24:09 -- STEP: 19496/36938 -- GLOBAL_STEP: 1238450 | > loss_text_ce: 0.03244198113679886 (0.031347370092941675) | > loss_mel_ce: 2.9199461936950684 (nan) | > loss: 0.011715827509760857 (nan) | > current_lr: 5e-06 | > step_time: 0.19 (0.13708969481078476) | > loader_time: 0.0048 (0.004650684253182323)  --> TIME: 2025-02-26 12:24:18 -- STEP: 19546/36938 -- GLOBAL_STEP: 1238500 | > loss_text_ce: 0.034390177577733994 (0.03134934152924862) | > loss_mel_ce: 3.0825130939483643 (nan) | > loss: 0.01236866507679224 (nan) | > current_lr: 5e-06 | > step_time: 0.1781 (0.13709482288672897) | > loader_time: 0.0048 (0.004650545281250469)  --> TIME: 2025-02-26 12:24:28 -- STEP: 19596/36938 -- GLOBAL_STEP: 1238550 | > loss_text_ce: 0.03214418515563011 (0.0313508897001971) | > loss_mel_ce: 2.968345880508423 (nan) | > loss: 0.011906706728041172 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.13709706980541192) | > loader_time: 0.0048 (0.004650505434909527)  --> TIME: 2025-02-26 12:24:37 -- STEP: 19646/36938 -- GLOBAL_STEP: 1238600 | > loss_text_ce: 0.03433987870812416 (0.03135172373750883) | > loss_mel_ce: 2.3906383514404297 (nan) | > loss: 0.00962293054908514 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13710871527251564) | > loader_time: 0.0045 (0.004650582512854844)  --> TIME: 2025-02-26 12:24:47 -- STEP: 19696/36938 -- GLOBAL_STEP: 1238650 | > loss_text_ce: 0.03613957017660141 (0.03135102823144774) | > loss_mel_ce: 3.1439590454101562 (nan) | > loss: 0.012619439512491226 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13711076990605203) | > loader_time: 0.0045 (0.004650550206728233)  --> TIME: 2025-02-26 12:24:56 -- STEP: 19746/36938 -- GLOBAL_STEP: 1238700 | > loss_text_ce: 0.02841501496732235 (0.03135025799570647) | > loss_mel_ce: 0.18710938096046448 (nan) | > loss: 0.0008552555809728801 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13710445068436294) | > loader_time: 0.0046 (0.004650571009893729)  --> TIME: 2025-02-26 12:25:05 -- STEP: 19796/36938 -- GLOBAL_STEP: 1238750 | > loss_text_ce: 0.03345879539847374 (0.03134703936209995) | > loss_mel_ce: 2.847738742828369 (nan) | > loss: 0.011433323845267296 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13710751950704714) | > loader_time: 0.0046 (0.004650584951413469)  --> TIME: 2025-02-26 12:25:15 -- STEP: 19846/36938 -- GLOBAL_STEP: 1238800 | > loss_text_ce: 0.035158246755599976 (0.03134603234954768) | > loss_mel_ce: 3.0509226322174072 (nan) | > loss: 0.01224635262042284 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.13710991513112838) | > loader_time: 0.0048 (0.004650530274039512)  --> TIME: 2025-02-26 12:25:24 -- STEP: 19896/36938 -- GLOBAL_STEP: 1238850 | > loss_text_ce: 0.03477498143911362 (0.03134624339477435) | > loss_mel_ce: 2.3323287963867188 (nan) | > loss: 0.009393269196152687 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13710575141866502) | > loader_time: 0.0045 (0.004650622138931029)  --> TIME: 2025-02-26 12:25:33 -- STEP: 19946/36938 -- GLOBAL_STEP: 1238900 | > loss_text_ce: 0.027375178411602974 (0.03134567739924232) | > loss_mel_ce: 2.209733009338379 (nan) | > loss: 0.008877414278686047 (nan) | > current_lr: 5e-06 | > step_time: 0.1753 (0.13709653253410936) | > loader_time: 0.0048 (0.004650573057264375)  --> TIME: 2025-02-26 12:25:43 -- STEP: 19996/36938 -- GLOBAL_STEP: 1238950 | > loss_text_ce: 0.02833438850939274 (0.031345567891931665) | > loss_mel_ce: 2.608452796936035 (nan) | > loss: 0.010463441722095013 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.1370951096566592) | > loader_time: 0.0049 (0.0046507409034335185)  --> TIME: 2025-02-26 12:25:52 -- STEP: 20046/36938 -- GLOBAL_STEP: 1239000 | > loss_text_ce: 0.030337121337652206 (0.031347782492312834) | > loss_mel_ce: 2.919981002807617 (nan) | > loss: 0.011707612313330173 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13709745588600458) | > loader_time: 0.0045 (0.004650701725303339)  --> TIME: 2025-02-26 12:26:02 -- STEP: 20096/36938 -- GLOBAL_STEP: 1239050 | > loss_text_ce: 0.02242489717900753 (0.03134483849746734) | > loss_mel_ce: 3.1912107467651367 (nan) | > loss: 0.012752523645758629 (nan) | > current_lr: 5e-06 | > step_time: 0.1679 (0.1371049234394438) | > loader_time: 0.0047 (0.004650600337583542)  --> TIME: 2025-02-26 12:26:11 -- STEP: 20146/36938 -- GLOBAL_STEP: 1239100 | > loss_text_ce: 0.01560015045106411 (0.03134245382761314) | > loss_mel_ce: 0.7847145199775696 (nan) | > loss: 0.003175852121785283 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13710338428171295) | > loader_time: 0.0045 (0.004650554625739238)  --> TIME: 2025-02-26 12:26:20 -- STEP: 20196/36938 -- GLOBAL_STEP: 1239150 | > loss_text_ce: 0.0365716814994812 (0.03133943994504833) | > loss_mel_ce: 2.3355021476745605 (nan) | > loss: 0.009412991814315319 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1371049398383336) | > loader_time: 0.0046 (0.004650559076391171)  --> TIME: 2025-02-26 12:26:29 -- STEP: 20246/36938 -- GLOBAL_STEP: 1239200 | > loss_text_ce: 0.025512946769595146 (0.03134124384822049) | > loss_mel_ce: 0.150146484375 (nan) | > loss: 0.0006970613030716777 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.1370957450740465) | > loader_time: 0.0045 (0.004650474077484926)  --> TIME: 2025-02-26 12:26:39 -- STEP: 20296/36938 -- GLOBAL_STEP: 1239250 | > loss_text_ce: 0.03213075175881386 (0.031340876025049424) | > loss_mel_ce: 1.931235909461975 (nan) | > loss: 0.0077911377884447575 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.1371094103657364) | > loader_time: 0.0047 (0.004650391130218074)  --> TIME: 2025-02-26 12:26:48 -- STEP: 20346/36938 -- GLOBAL_STEP: 1239300 | > loss_text_ce: 0.03556325286626816 (0.03134158209868386) | > loss_mel_ce: 2.969154119491577 (nan) | > loss: 0.01192348264157772 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.13710066739711135) | > loader_time: 0.0048 (0.004650240320377983)  --> TIME: 2025-02-26 12:26:58 -- STEP: 20396/36938 -- GLOBAL_STEP: 1239350 | > loss_text_ce: 0.03242655470967293 (0.03133985111931001) | > loss_mel_ce: 2.78947114944458 (nan) | > loss: 0.011198007501661777 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.1371048395950253) | > loader_time: 0.0045 (0.0046500510901978254)  --> TIME: 2025-02-26 12:27:07 -- STEP: 20446/36938 -- GLOBAL_STEP: 1239400 | > loss_text_ce: 0.036736298352479935 (0.03134439844762202) | > loss_mel_ce: 1.9436914920806885 (nan) | > loss: 0.007858840748667717 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.1371113549679986) | > loader_time: 0.0045 (0.004649947910034567)  --> TIME: 2025-02-26 12:27:16 -- STEP: 20496/36938 -- GLOBAL_STEP: 1239450 | > loss_text_ce: 0.013170945458114147 (0.03134245795210886) | > loss_mel_ce: 0.2267645001411438 (nan) | > loss: 0.0009521248284727335 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13711163996021655) | > loader_time: 0.0045 (0.004649824981201816)  --> TIME: 2025-02-26 12:27:26 -- STEP: 20546/36938 -- GLOBAL_STEP: 1239500 | > loss_text_ce: 0.02573210746049881 (0.03134099832189038) | > loss_mel_ce: 2.8365445137023926 (nan) | > loss: 0.011358240619301796 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.13712039311092566) | > loader_time: 0.0046 (0.004649768620192752)  --> TIME: 2025-02-26 12:27:35 -- STEP: 20596/36938 -- GLOBAL_STEP: 1239550 | > loss_text_ce: 0.028563711792230606 (0.03134151361715391) | > loss_mel_ce: 1.6414062976837158 (nan) | > loss: 0.006626865826547146 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1371133688885768) | > loader_time: 0.0045 (0.004649668590470084)  --> TIME: 2025-02-26 12:27:44 -- STEP: 20646/36938 -- GLOBAL_STEP: 1239600 | > loss_text_ce: 0.020008794963359833 (0.031339135460198765) | > loss_mel_ce: 2.4244675636291504 (nan) | > loss: 0.00970030389726162 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13710844746558012) | > loader_time: 0.0044 (0.00464983831989687)  --> TIME: 2025-02-26 12:27:54 -- STEP: 20696/36938 -- GLOBAL_STEP: 1239650 | > loss_text_ce: 0.027917405590415 (0.031337266575891164) | > loss_mel_ce: 2.8792343139648438 (nan) | > loss: 0.011536316946148872 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.1371023133771507) | > loader_time: 0.0046 (0.004649952958346232)  --> TIME: 2025-02-26 12:28:03 -- STEP: 20746/36938 -- GLOBAL_STEP: 1239700 | > loss_text_ce: 0.023874713107943535 (0.03133720329773329) | > loss_mel_ce: 2.7699501514434814 (nan) | > loss: 0.011086607351899147 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13710593113062167) | > loader_time: 0.0045 (0.004649847036247867)  --> TIME: 2025-02-26 12:28:12 -- STEP: 20796/36938 -- GLOBAL_STEP: 1239750 | > loss_text_ce: 0.034411683678627014 (0.03133430704192322) | > loss_mel_ce: 3.0643763542175293 (nan) | > loss: 0.01229677814990282 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.1371018005852795) | > loader_time: 0.0046 (0.004649738780258475)  --> TIME: 2025-02-26 12:28:22 -- STEP: 20846/36938 -- GLOBAL_STEP: 1239800 | > loss_text_ce: 0.029685020446777344 (0.031336423342517405) | > loss_mel_ce: 2.137880802154541 (nan) | > loss: 0.008601452223956585 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13709994414872223) | > loader_time: 0.0045 (0.004649601867443006)  --> TIME: 2025-02-26 12:28:31 -- STEP: 20896/36938 -- GLOBAL_STEP: 1239850 | > loss_text_ce: 0.030026208609342575 (0.031338634623304204) | > loss_mel_ce: 2.7819576263427734 (nan) | > loss: 0.01115866657346487 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.1370987306601424) | > loader_time: 0.0045 (0.004649446886403625)  --> TIME: 2025-02-26 12:28:40 -- STEP: 20946/36938 -- GLOBAL_STEP: 1239900 | > loss_text_ce: 0.039604220539331436 (0.03134192440967034) | > loss_mel_ce: 3.113110065460205 (nan) | > loss: 0.012510771863162518 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.13709484405120584) | > loader_time: 0.0046 (0.004649257484621057)  --> TIME: 2025-02-26 12:28:50 -- STEP: 20996/36938 -- GLOBAL_STEP: 1239950 | > loss_text_ce: 0.02841920219361782 (0.03134114202914831) | > loss_mel_ce: 0.13232421875 (nan) | > loss: 0.0006378707475960255 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13710559598830455) | > loader_time: 0.0044 (0.004649207611996038)  --> TIME: 2025-02-26 12:28:59 -- STEP: 21046/36938 -- GLOBAL_STEP: 1240000 | > loss_text_ce: 0.02927139215171337 (0.03134041289825208) | > loss_mel_ce: 2.7001748085021973 (nan) | > loss: 0.010831136256456375 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.137111871797391) | > loader_time: 0.0045 (0.0046490991930085565) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1240000.pth  --> TIME: 2025-02-26 12:29:14 -- STEP: 21096/36938 -- GLOBAL_STEP: 1240050 | > loss_text_ce: 0.034738678485155106 (0.031340001187916716) | > loss_mel_ce: 2.858196258544922 (nan) | > loss: 0.011479901149868965 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13710286186009982) | > loader_time: 0.0046 (0.004648984586103052)  --> TIME: 2025-02-26 12:29:23 -- STEP: 21146/36938 -- GLOBAL_STEP: 1240100 | > loss_text_ce: 0.03178435564041138 (0.03133902403164075) | > loss_mel_ce: 3.23081636428833 (nan) | > loss: 0.012946829199790955 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13709119854976431) | > loader_time: 0.0046 (0.004648838985341723)  --> TIME: 2025-02-26 12:29:32 -- STEP: 21196/36938 -- GLOBAL_STEP: 1240150 | > loss_text_ce: 0.036881010979413986 (0.03133714680707928) | > loss_mel_ce: 2.9649224281311035 (nan) | > loss: 0.011911919340491295 (nan) | > current_lr: 5e-06 | > step_time: 0.1825 (0.13708647359517628) | > loader_time: 0.0049 (0.004648699121984533)  --> TIME: 2025-02-26 12:29:41 -- STEP: 21246/36938 -- GLOBAL_STEP: 1240200 | > loss_text_ce: 0.02486325055360794 (0.03133792157354099) | > loss_mel_ce: 3.0950937271118164 (nan) | > loss: 0.012380782514810562 (nan) | > current_lr: 5e-06 | > step_time: 0.1355 (0.1370791361671514) | > loader_time: 0.0045 (0.00464856284582422)  --> TIME: 2025-02-26 12:29:51 -- STEP: 21296/36938 -- GLOBAL_STEP: 1240250 | > loss_text_ce: 0.02605600841343403 (0.03133441304002048) | > loss_mel_ce: 0.0546875 (nan) | > loss: 0.0003204107633791864 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13707577256355707) | > loader_time: 0.0045 (0.004648461064658241)  --> TIME: 2025-02-26 12:30:00 -- STEP: 21346/36938 -- GLOBAL_STEP: 1240300 | > loss_text_ce: 0.0465325266122818 (0.031331486290785596) | > loss_mel_ce: 3.387605667114258 (nan) | > loss: 0.013627533800899982 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.1370766030740218) | > loader_time: 0.0046 (0.004648409117176211)  --> TIME: 2025-02-26 12:30:09 -- STEP: 21396/36938 -- GLOBAL_STEP: 1240350 | > loss_text_ce: 0.026352006942033768 (0.03133531877755186) | > loss_mel_ce: 0.006439208984375 (nan) | > loss: 0.00013012388080824167 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13707572956088795) | > loader_time: 0.0045 (0.0046483151688489184)  --> TIME: 2025-02-26 12:30:19 -- STEP: 21446/36938 -- GLOBAL_STEP: 1240400 | > loss_text_ce: 0.02368384785950184 (0.03133658815758434) | > loss_mel_ce: 1.9468259811401367 (nan) | > loss: 0.007819483987987041 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13708945572693115) | > loader_time: 0.0045 (0.004648311785394031)  --> TIME: 2025-02-26 12:30:28 -- STEP: 21496/36938 -- GLOBAL_STEP: 1240450 | > loss_text_ce: 0.02853279560804367 (0.03133726630415873) | > loss_mel_ce: 2.74898624420166 (nan) | > loss: 0.011021900922060013 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13709535801060216) | > loader_time: 0.0046 (0.00464828600216231)  --> TIME: 2025-02-26 12:30:38 -- STEP: 21546/36938 -- GLOBAL_STEP: 1240500 | > loss_text_ce: 0.03610488399863243 (0.031334657027996274) | > loss_mel_ce: 3.3634588718414307 (nan) | > loss: 0.01349033322185278 (nan) | > current_lr: 5e-06 | > step_time: 0.1261 (0.13709776631462153) | > loader_time: 0.0046 (0.004648214527173681)  --> TIME: 2025-02-26 12:30:47 -- STEP: 21596/36938 -- GLOBAL_STEP: 1240550 | > loss_text_ce: 0.033379171043634415 (0.03133644572677206) | > loss_mel_ce: 3.6608145236968994 (nan) | > loss: 0.014659498818218708 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.1370969171280288) | > loader_time: 0.0045 (0.004648115562496197)  --> TIME: 2025-02-26 12:30:56 -- STEP: 21646/36938 -- GLOBAL_STEP: 1240600 | > loss_text_ce: 0.025939442217350006 (0.03133615208189082) | > loss_mel_ce: 3.346229314804077 (nan) | > loss: 0.013381622731685638 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13708836215266934) | > loader_time: 0.0046 (0.004648071851856352)  --> TIME: 2025-02-26 12:31:06 -- STEP: 21696/36938 -- GLOBAL_STEP: 1240650 | > loss_text_ce: 0.031060637906193733 (0.03133715756374364) | > loss_mel_ce: 2.687957525253296 (nan) | > loss: 0.01078975573182106 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.13708566928112068) | > loader_time: 0.0046 (0.004648007562378512)  --> TIME: 2025-02-26 12:31:15 -- STEP: 21746/36938 -- GLOBAL_STEP: 1240700 | > loss_text_ce: 0.02647992968559265 (0.03133732183335073) | > loss_mel_ce: 2.2379088401794434 (nan) | > loss: 0.008985670283436775 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.13708706306882754) | > loader_time: 0.0047 (0.004647919755186793)  --> TIME: 2025-02-26 12:31:24 -- STEP: 21796/36938 -- GLOBAL_STEP: 1240750 | > loss_text_ce: 0.028839116916060448 (0.03133736739196112) | > loss_mel_ce: 0.1864081472158432 (nan) | > loss: 0.000854155863635242 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13707648729879862) | > loader_time: 0.0046 (0.004647757989985277)  --> TIME: 2025-02-26 12:31:34 -- STEP: 21846/36938 -- GLOBAL_STEP: 1240800 | > loss_text_ce: 0.030071906745433807 (0.03133849206679057) | > loss_mel_ce: 2.9011213779449463 (nan) | > loss: 0.011631720699369907 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.137082103155891) | > loader_time: 0.0048 (0.004647670533863769)  --> TIME: 2025-02-26 12:31:43 -- STEP: 21896/36938 -- GLOBAL_STEP: 1240850 | > loss_text_ce: 0.029208680614829063 (0.03134045590642999) | > loss_mel_ce: 3.0534918308258057 (nan) | > loss: 0.012232938781380653 (nan) | > current_lr: 5e-06 | > step_time: 0.1935 (0.13708578719246065) | > loader_time: 0.0048 (0.0046475779565965086)  --> TIME: 2025-02-26 12:31:53 -- STEP: 21946/36938 -- GLOBAL_STEP: 1240900 | > loss_text_ce: 0.030297067016363144 (0.03134044605550618) | > loss_mel_ce: 2.303405523300171 (nan) | > loss: 0.009260725229978561 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.1370951801275017) | > loader_time: 0.0048 (0.004647514155881612)  --> TIME: 2025-02-26 12:32:02 -- STEP: 21996/36938 -- GLOBAL_STEP: 1240950 | > loss_text_ce: 0.03854161500930786 (0.03133877344905997) | > loss_mel_ce: 0.11767578125 (nan) | > loss: 0.0006199103663675487 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13707880698283967) | > loader_time: 0.0045 (0.004647302863857568)  --> TIME: 2025-02-26 12:32:11 -- STEP: 22046/36938 -- GLOBAL_STEP: 1241000 | > loss_text_ce: 0.026514602825045586 (0.031336449704259826) | > loss_mel_ce: 2.7502377033233643 (nan) | > loss: 0.011018858291208744 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13707890790008223) | > loader_time: 0.0045 (0.004647237910859256)  --> TIME: 2025-02-26 12:32:20 -- STEP: 22096/36938 -- GLOBAL_STEP: 1241050 | > loss_text_ce: 0.02725677192211151 (0.031336361368585225) | > loss_mel_ce: 2.025681257247925 (nan) | > loss: 0.008146579377353191 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.1370661697100246) | > loader_time: 0.0045 (0.004647022945392303)  --> TIME: 2025-02-26 12:32:29 -- STEP: 22146/36938 -- GLOBAL_STEP: 1241100 | > loss_text_ce: 0.023048222064971924 (0.03134007559467934) | > loss_mel_ce: 2.9113361835479736 (nan) | > loss: 0.01164438296109438 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.1370616255656582) | > loader_time: 0.0045 (0.004646875160039512)  --> TIME: 2025-02-26 12:32:39 -- STEP: 22196/36938 -- GLOBAL_STEP: 1241150 | > loss_text_ce: 0.020725192502141 (0.031343875248598614) | > loss_mel_ce: 2.2188398838043213 (nan) | > loss: 0.008887163363397121 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1370722800251555) | > loader_time: 0.0045 (0.004646732841961907)  --> TIME: 2025-02-26 12:32:48 -- STEP: 22246/36938 -- GLOBAL_STEP: 1241200 | > loss_text_ce: 0.0266730859875679 (0.03134278366933741) | > loss_mel_ce: 0.013753890991210938 (nan) | > loss: 0.00016042451898101717 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13707926459185107) | > loader_time: 0.0045 (0.004646640270613482)  --> TIME: 2025-02-26 12:32:58 -- STEP: 22296/36938 -- GLOBAL_STEP: 1241250 | > loss_text_ce: 0.03107498027384281 (0.03134065248844664) | > loss_mel_ce: 3.107651710510254 (nan) | > loss: 0.012455265037715435 (nan) | > current_lr: 5e-06 | > step_time: 0.1868 (0.13708868496598497) | > loader_time: 0.0049 (0.00464656157736543)  --> TIME: 2025-02-26 12:33:07 -- STEP: 22346/36938 -- GLOBAL_STEP: 1241300 | > loss_text_ce: 0.0355742983520031 (0.03134239288246598) | > loss_mel_ce: 2.982353925704956 (nan) | > loss: 0.011975905857980251 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13708919519854437) | > loader_time: 0.0046 (0.004646497522612062)  --> TIME: 2025-02-26 12:33:17 -- STEP: 22396/36938 -- GLOBAL_STEP: 1241350 | > loss_text_ce: 0.02685040980577469 (0.031341759305452244) | > loss_mel_ce: 1.5849746465682983 (nan) | > loss: 0.006396131590008736 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13709273154575016) | > loader_time: 0.0046 (0.004646409971625868)  --> TIME: 2025-02-26 12:33:26 -- STEP: 22446/36938 -- GLOBAL_STEP: 1241400 | > loss_text_ce: 0.025900933891534805 (0.031340661411889285) | > loss_mel_ce: 2.9672152996063232 (nan) | > loss: 0.011877445504069328 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13709620695259422) | > loader_time: 0.0044 (0.004646304795993345)  --> TIME: 2025-02-26 12:33:36 -- STEP: 22496/36938 -- GLOBAL_STEP: 1241450 | > loss_text_ce: 0.029514675959944725 (0.031340028637008265) | > loss_mel_ce: 3.3388171195983887 (nan) | > loss: 0.013366397470235825 (nan) | > current_lr: 5e-06 | > step_time: 0.1897 (0.1371060702658308) | > loader_time: 0.0049 (0.004646269930460369)  --> TIME: 2025-02-26 12:33:45 -- STEP: 22546/36938 -- GLOBAL_STEP: 1241500 | > loss_text_ce: 0.02626102603971958 (0.03133991308056999) | > loss_mel_ce: 3.078097105026245 (nan) | > loss: 0.012318882159888744 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13710403151576836) | > loader_time: 0.0047 (0.004646273944346806)  --> TIME: 2025-02-26 12:33:55 -- STEP: 22596/36938 -- GLOBAL_STEP: 1241550 | > loss_text_ce: 0.02826884016394615 (0.03134016642496762) | > loss_mel_ce: 2.1642565727233887 (nan) | > loss: 0.008700498379766941 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.13711729193603175) | > loader_time: 0.0046 (0.004646236631886724)  --> TIME: 2025-02-26 12:34:04 -- STEP: 22646/36938 -- GLOBAL_STEP: 1241600 | > loss_text_ce: 0.033561695367097855 (0.031340524594987235) | > loss_mel_ce: 2.85366153717041 (nan) | > loss: 0.011457235552370548 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.1371175348995578) | > loader_time: 0.0046 (0.004646189693089429)  --> TIME: 2025-02-26 12:34:13 -- STEP: 22696/36938 -- GLOBAL_STEP: 1241650 | > loss_text_ce: 0.03208380937576294 (0.03134267922493874) | > loss_mel_ce: 2.9267327785491943 (nan) | > loss: 0.011741336435079575 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13711362164290286) | > loader_time: 0.0047 (0.004646059804539207)  --> TIME: 2025-02-26 12:34:22 -- STEP: 22746/36938 -- GLOBAL_STEP: 1241700 | > loss_text_ce: 0.030829111114144325 (0.03134063507244695) | > loss_mel_ce: 2.126380443572998 (nan) | > loss: 0.008560355752706528 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13710815137371585) | > loader_time: 0.0046 (0.004645977078542366)  --> TIME: 2025-02-26 12:34:32 -- STEP: 22796/36938 -- GLOBAL_STEP: 1241750 | > loss_text_ce: 0.03900492191314697 (0.031343255641095275) | > loss_mel_ce: 2.358067274093628 (nan) | > loss: 0.009512192569673061 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13712073700702862) | > loader_time: 0.0046 (0.004646161906153177)  --> TIME: 2025-02-26 12:34:41 -- STEP: 22846/36938 -- GLOBAL_STEP: 1241800 | > loss_text_ce: 0.03309788554906845 (0.03134535211709956) | > loss_mel_ce: 2.542799949645996 (nan) | > loss: 0.010221817530691624 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13711753731004336) | > loader_time: 0.0046 (0.0046464680143468275)  --> TIME: 2025-02-26 12:34:51 -- STEP: 22896/36938 -- GLOBAL_STEP: 1241850 | > loss_text_ce: 0.0350046306848526 (0.03134833141143404) | > loss_mel_ce: 2.2235331535339355 (nan) | > loss: 0.008962452411651611 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.13711814072105433) | > loader_time: 0.0047 (0.004646472513134201)  --> TIME: 2025-02-26 12:35:00 -- STEP: 22946/36938 -- GLOBAL_STEP: 1241900 | > loss_text_ce: 0.026434950530529022 (0.03134675732085467) | > loss_mel_ce: 2.9632747173309326 (nan) | > loss: 0.011863927356898785 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.1371159927167336) | > loader_time: 0.0045 (0.004646597490017234)  --> TIME: 2025-02-26 12:35:09 -- STEP: 22996/36938 -- GLOBAL_STEP: 1241950 | > loss_text_ce: 0.03172660991549492 (0.031347210931764666) | > loss_mel_ce: 2.86519193649292 (nan) | > loss: 0.011495709419250488 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13712131534375008) | > loader_time: 0.0045 (0.0046465249968976685)  --> TIME: 2025-02-26 12:35:19 -- STEP: 23046/36938 -- GLOBAL_STEP: 1242000 | > loss_text_ce: 0.043491482734680176 (0.0313461791165439) | > loss_mel_ce: 0.013758341781795025 (nan) | > loss: 0.00022718186664860696 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13712194387380763) | > loader_time: 0.0045 (0.004646461746360143)  --> TIME: 2025-02-26 12:35:28 -- STEP: 23096/36938 -- GLOBAL_STEP: 1242050 | > loss_text_ce: 0.030075211077928543 (0.031347206110294484) | > loss_mel_ce: 0.03138885647058487 (nan) | > loss: 0.00024390504404436797 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.1371203491462318) | > loader_time: 0.0046 (0.004646513787858563)  --> TIME: 2025-02-26 12:35:37 -- STEP: 23146/36938 -- GLOBAL_STEP: 1242100 | > loss_text_ce: 0.04092942550778389 (0.03134881089902425) | > loss_mel_ce: 3.0448174476623535 (nan) | > loss: 0.012245027348399162 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13710983978129054) | > loader_time: 0.0046 (0.004646387012046246)  --> TIME: 2025-02-26 12:35:46 -- STEP: 23196/36938 -- GLOBAL_STEP: 1242150 | > loss_text_ce: 0.026071451604366302 (0.031350170273700996) | > loss_mel_ce: 3.3494482040405273 (nan) | > loss: 0.013394920155405998 (nan) | > current_lr: 5e-06 | > step_time: 0.1878 (0.13710180421927196) | > loader_time: 0.0048 (0.004646347070237755)  --> TIME: 2025-02-26 12:35:55 -- STEP: 23246/36938 -- GLOBAL_STEP: 1242200 | > loss_text_ce: 0.024245496839284897 (0.03135127767066407) | > loss_mel_ce: 1.9871994256973267 (nan) | > loss: 0.007981924340128899 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.1370851532128631) | > loader_time: 0.0045 (0.004646377627886278)  --> TIME: 2025-02-26 12:36:05 -- STEP: 23296/36938 -- GLOBAL_STEP: 1242250 | > loss_text_ce: 0.03346104547381401 (0.031349354902976155) | > loss_mel_ce: 1.4910043478012085 (nan) | > loss: 0.006049466319382191 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.1370861577791167) | > loader_time: 0.0046 (0.004646299202199803)  --> TIME: 2025-02-26 12:36:14 -- STEP: 23346/36938 -- GLOBAL_STEP: 1242300 | > loss_text_ce: 0.026044711470603943 (0.03135036025417709) | > loss_mel_ce: 2.410231590270996 (nan) | > loss: 0.009667763486504555 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13709108871164813) | > loader_time: 0.0047 (0.004646338667322573)  --> TIME: 2025-02-26 12:36:23 -- STEP: 23396/36938 -- GLOBAL_STEP: 1242350 | > loss_text_ce: 0.03366563096642494 (0.03135250212540192) | > loss_mel_ce: 2.5999350547790527 (nan) | > loss: 0.010450797155499458 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13708880259901363) | > loader_time: 0.0047 (0.004646243886430747)  --> TIME: 2025-02-26 12:36:32 -- STEP: 23446/36938 -- GLOBAL_STEP: 1242400 | > loss_text_ce: 0.03623144328594208 (0.031353788617729) | > loss_mel_ce: 3.3753294944763184 (nan) | > loss: 0.013537941500544548 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13707541664905742) | > loader_time: 0.0046 (0.004646334775852915)  --> TIME: 2025-02-26 12:36:42 -- STEP: 23496/36938 -- GLOBAL_STEP: 1242450 | > loss_text_ce: 0.03354639187455177 (0.03135270257594197) | > loss_mel_ce: 2.1358466148376465 (nan) | > loss: 0.008608703501522541 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.1370889606689322) | > loader_time: 0.0046 (0.004646373030519975)  --> TIME: 2025-02-26 12:36:51 -- STEP: 23546/36938 -- GLOBAL_STEP: 1242500 | > loss_text_ce: 0.029476486146450043 (0.03135028742288191) | > loss_mel_ce: 2.9366040229797363 (nan) | > loss: 0.011770160868763924 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13708011826389016) | > loader_time: 0.0046 (0.004646228881252038)  --> TIME: 2025-02-26 12:37:00 -- STEP: 23596/36938 -- GLOBAL_STEP: 1242550 | > loss_text_ce: 0.034186724573373795 (0.03134939005813691) | > loss_mel_ce: 2.989190101623535 (nan) | > loss: 0.011997528374195099 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13707709777231866) | > loader_time: 0.0048 (0.004646084079865286)  --> TIME: 2025-02-26 12:37:10 -- STEP: 23646/36938 -- GLOBAL_STEP: 1242600 | > loss_text_ce: 0.02547907829284668 (0.03134957367633787) | > loss_mel_ce: 2.74788761138916 (nan) | > loss: 0.011005423963069916 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.1370752700782912) | > loader_time: 0.0045 (0.0046460150986730765)  --> TIME: 2025-02-26 12:37:19 -- STEP: 23696/36938 -- GLOBAL_STEP: 1242650 | > loss_text_ce: 0.024688875302672386 (0.03134593582868304) | > loss_mel_ce: 2.8874878883361816 (nan) | > loss: 0.011556258425116539 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13708627809550888) | > loader_time: 0.0046 (0.004646026921143474)  --> TIME: 2025-02-26 12:37:29 -- STEP: 23746/36938 -- GLOBAL_STEP: 1242700 | > loss_text_ce: 0.04755504056811333 (0.031345936479676026) | > loss_mel_ce: 2.878486156463623 (nan) | > loss: 0.011611275374889374 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13710396636017286) | > loader_time: 0.0045 (0.004646203275539317)  --> TIME: 2025-02-26 12:37:39 -- STEP: 23796/36938 -- GLOBAL_STEP: 1242750 | > loss_text_ce: 0.025535160675644875 (0.03134494869030712) | > loss_mel_ce: 2.606398820877075 (nan) | > loss: 0.01044418290257454 (nan) | > current_lr: 5e-06 | > step_time: 0.1868 (0.1371167160559344) | > loader_time: 0.0048 (0.004646204393197338)  --> TIME: 2025-02-26 12:37:49 -- STEP: 23846/36938 -- GLOBAL_STEP: 1242800 | > loss_text_ce: 0.04038604721426964 (0.031345991591739286) | > loss_mel_ce: 2.34450364112854 (nan) | > loss: 0.009463848546147346 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13713094327890285) | > loader_time: 0.0046 (0.004646217164143017)  --> TIME: 2025-02-26 12:37:58 -- STEP: 23896/36938 -- GLOBAL_STEP: 1242850 | > loss_text_ce: 0.03294835239648819 (0.031345894372928305) | > loss_mel_ce: 1.7526613473892212 (nan) | > loss: 0.007085753139108419 (nan) | > current_lr: 5e-06 | > step_time: 0.1563 (0.1371291753616784) | > loader_time: 0.0047 (0.004646104127218531)  --> TIME: 2025-02-26 12:38:07 -- STEP: 23946/36938 -- GLOBAL_STEP: 1242900 | > loss_text_ce: 0.03240661695599556 (0.031347946832012495) | > loss_mel_ce: 2.6144421100616455 (nan) | > loss: 0.010503368452191353 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13712662409135065) | > loader_time: 0.0046 (0.004646021282524518)  --> TIME: 2025-02-26 12:38:16 -- STEP: 23996/36938 -- GLOBAL_STEP: 1242950 | > loss_text_ce: 0.027873782441020012 (0.031345556178330465) | > loss_mel_ce: 2.7427961826324463 (nan) | > loss: 0.010994723066687584 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13711323970992423) | > loader_time: 0.0047 (0.004645888349139138)  --> TIME: 2025-02-26 12:38:25 -- STEP: 24046/36938 -- GLOBAL_STEP: 1243000 | > loss_text_ce: 0.03730364888906479 (0.0313474685602625) | > loss_mel_ce: 2.2506051063537598 (nan) | > loss: 0.00907900370657444 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.137106439004987) | > loader_time: 0.0044 (0.004645773448910766)  --> TIME: 2025-02-26 12:38:35 -- STEP: 24096/36938 -- GLOBAL_STEP: 1243050 | > loss_text_ce: 0.02599884755909443 (0.03134740452891798) | > loss_mel_ce: 2.151201009750366 (nan) | > loss: 0.008639682084321976 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13711605408039687) | > loader_time: 0.0046 (0.00464571621590559)  --> TIME: 2025-02-26 12:38:44 -- STEP: 24146/36938 -- GLOBAL_STEP: 1243100 | > loss_text_ce: 0.021639810875058174 (0.031348593640245656) | > loss_mel_ce: 2.64658522605896 (nan) | > loss: 0.01058819517493248 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13711751571776273) | > loader_time: 0.0047 (0.004645716676970329)  --> TIME: 2025-02-26 12:38:54 -- STEP: 24196/36938 -- GLOBAL_STEP: 1243150 | > loss_text_ce: 0.027300843968987465 (0.03134664799977652) | > loss_mel_ce: 3.002044677734375 (nan) | > loss: 0.012021212838590145 (nan) | > current_lr: 5e-06 | > step_time: 0.1578 (0.13712681071938368) | > loader_time: 0.0068 (0.004645791235473006)  --> TIME: 2025-02-26 12:39:04 -- STEP: 24246/36938 -- GLOBAL_STEP: 1243200 | > loss_text_ce: 0.026638276875019073 (0.031348297590156914) | > loss_mel_ce: 2.3956973552703857 (nan) | > loss: 0.00961244385689497 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13713452404583554) | > loader_time: 0.0045 (0.004645807528902871)  --> TIME: 2025-02-26 12:39:13 -- STEP: 24296/36938 -- GLOBAL_STEP: 1243250 | > loss_text_ce: 0.03459474816918373 (0.031348395062184065) | > loss_mel_ce: 3.186218023300171 (nan) | > loss: 0.01278100349009037 (nan) | > current_lr: 5e-06 | > step_time: 0.1679 (0.13714796006385105) | > loader_time: 0.0048 (0.004645798231450816)  --> TIME: 2025-02-26 12:39:22 -- STEP: 24346/36938 -- GLOBAL_STEP: 1243300 | > loss_text_ce: 0.030106794089078903 (0.03135023858134011) | > loss_mel_ce: 2.9967095851898193 (nan) | > loss: 0.012011176906526089 (nan) | > current_lr: 5e-06 | > step_time: 0.1708 (0.1371367042716677) | > loader_time: 0.0049 (0.004645684951726437)  --> TIME: 2025-02-26 12:39:32 -- STEP: 24396/36938 -- GLOBAL_STEP: 1243350 | > loss_text_ce: 0.027514420449733734 (0.03134878821440653) | > loss_mel_ce: 3.7443530559539795 (nan) | > loss: 0.014967729337513447 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.13714018647329437) | > loader_time: 0.0046 (0.004645815724758922)  --> TIME: 2025-02-26 12:39:41 -- STEP: 24446/36938 -- GLOBAL_STEP: 1243400 | > loss_text_ce: 0.03405514359474182 (0.03134715996226919) | > loss_mel_ce: 2.38853120803833 (nan) | > loss: 0.00961343850940466 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13714631609505853) | > loader_time: 0.0045 (0.004645907517044371)  --> TIME: 2025-02-26 12:39:51 -- STEP: 24496/36938 -- GLOBAL_STEP: 1243450 | > loss_text_ce: 0.04082556068897247 (0.03134776836722777) | > loss_mel_ce: 2.192578077316284 (nan) | > loss: 0.008862713351845741 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13713624416458103) | > loader_time: 0.0045 (0.00464581157513337)  --> TIME: 2025-02-26 12:40:00 -- STEP: 24546/36938 -- GLOBAL_STEP: 1243500 | > loss_text_ce: 0.03294391930103302 (0.031349015368192915) | > loss_mel_ce: 2.0640597343444824 (nan) | > loss: 0.008321443572640419 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.1371365209707055) | > loader_time: 0.0045 (0.004645768407018022)  --> TIME: 2025-02-26 12:40:09 -- STEP: 24596/36938 -- GLOBAL_STEP: 1243550 | > loss_text_ce: 0.030932968482375145 (0.03134929088476625) | > loss_mel_ce: 2.5864014625549316 (nan) | > loss: 0.010386248119175434 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13713827575584925) | > loader_time: 0.0045 (0.004645694085380311)  --> TIME: 2025-02-26 12:40:18 -- STEP: 24646/36938 -- GLOBAL_STEP: 1243600 | > loss_text_ce: 0.03865495324134827 (0.031349014130175365) | > loss_mel_ce: 2.7298085689544678 (nan) | > loss: 0.010985967703163624 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13713437603693013) | > loader_time: 0.0046 (0.004645592059870866)  --> TIME: 2025-02-26 12:40:28 -- STEP: 24696/36938 -- GLOBAL_STEP: 1243650 | > loss_text_ce: 0.037859056144952774 (0.031347955338041224) | > loss_mel_ce: 2.7805635929107666 (nan) | > loss: 0.011184217408299446 (nan) | > current_lr: 5e-06 | > step_time: 0.1262 (0.13712617358151152) | > loader_time: 0.0047 (0.0046455152779286525)  --> TIME: 2025-02-26 12:40:37 -- STEP: 24746/36938 -- GLOBAL_STEP: 1243700 | > loss_text_ce: 0.036333974450826645 (0.031347734164579755) | > loss_mel_ce: 3.135366916656494 (nan) | > loss: 0.01258611585944891 (nan) | > current_lr: 5e-06 | > step_time: 0.1725 (0.13712817554301215) | > loader_time: 0.0047 (0.004645449837919169)  --> TIME: 2025-02-26 12:40:47 -- STEP: 24796/36938 -- GLOBAL_STEP: 1243750 | > loss_text_ce: 0.03692634776234627 (0.031349929325503484) | > loss_mel_ce: 2.9333584308624268 (nan) | > loss: 0.011786844581365585 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13713773286502226) | > loader_time: 0.0045 (0.004645391180931042)  --> TIME: 2025-02-26 12:40:56 -- STEP: 24846/36938 -- GLOBAL_STEP: 1243800 | > loss_text_ce: 0.03581603989005089 (0.03135132004181819) | > loss_mel_ce: 1.5082637071609497 (nan) | > loss: 0.006127301137894392 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1371403027586455) | > loader_time: 0.0045 (0.004645327683818568)  --> TIME: 2025-02-26 12:41:06 -- STEP: 24896/36938 -- GLOBAL_STEP: 1243850 | > loss_text_ce: 0.035730183124542236 (0.03134908756912078) | > loss_mel_ce: 1.9441412687301636 (nan) | > loss: 0.007856633514165878 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.13714640400824935) | > loader_time: 0.0044 (0.004645278126690862)  --> TIME: 2025-02-26 12:41:15 -- STEP: 24946/36938 -- GLOBAL_STEP: 1243900 | > loss_text_ce: 0.03296318277716637 (0.0313523995639012) | > loss_mel_ce: 0.9219089150428772 (nan) | > loss: 0.0037891750689595938 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13714036378215913) | > loader_time: 0.0046 (0.004645221533278718)  --> TIME: 2025-02-26 12:41:24 -- STEP: 24996/36938 -- GLOBAL_STEP: 1243950 | > loss_text_ce: 0.034457262605428696 (0.03135277211436509) | > loss_mel_ce: 3.1797358989715576 (nan) | > loss: 0.01275473553687334 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13713443210590812) | > loader_time: 0.0046 (0.004645152442225219)  --> TIME: 2025-02-26 12:41:34 -- STEP: 25046/36938 -- GLOBAL_STEP: 1244000 | > loss_text_ce: 0.03196236118674278 (0.031350864086636614) | > loss_mel_ce: 1.8289320468902588 (nan) | > loss: 0.007384502328932285 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.1371472040742366) | > loader_time: 0.0046 (0.004645282055331532)  --> TIME: 2025-02-26 12:41:43 -- STEP: 25096/36938 -- GLOBAL_STEP: 1244050 | > loss_text_ce: 0.032094087451696396 (0.03135305058906455) | > loss_mel_ce: 0.783355712890625 (nan) | > loss: 0.003235911950469017 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13713923810126483) | > loader_time: 0.0046 (0.004645291743173608)  --> TIME: 2025-02-26 12:41:52 -- STEP: 25146/36938 -- GLOBAL_STEP: 1244100 | > loss_text_ce: 0.03898460790514946 (0.03135186716827724) | > loss_mel_ce: 2.954676628112793 (nan) | > loss: 0.011879608035087585 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13712761688088285) | > loader_time: 0.0046 (0.004645227598971408)  --> TIME: 2025-02-26 12:42:02 -- STEP: 25196/36938 -- GLOBAL_STEP: 1244150 | > loss_text_ce: 0.027277780696749687 (0.03135266683224002) | > loss_mel_ce: 2.6030750274658203 (nan) | > loss: 0.010437908582389355 (nan) | > current_lr: 5e-06 | > step_time: 0.1569 (0.13714195318800382) | > loader_time: 0.0047 (0.004645303424000062)  --> TIME: 2025-02-26 12:42:11 -- STEP: 25246/36938 -- GLOBAL_STEP: 1244200 | > loss_text_ce: 0.03336828574538231 (0.031351894607288155) | > loss_mel_ce: 2.4166719913482666 (nan) | > loss: 0.009722382761538029 (nan) | > current_lr: 5e-06 | > step_time: 0.1439 (0.13713964042445984) | > loader_time: 0.0046 (0.004645391584509826)  --> TIME: 2025-02-26 12:42:20 -- STEP: 25296/36938 -- GLOBAL_STEP: 1244250 | > loss_text_ce: 0.041061438620090485 (0.03135151405947836) | > loss_mel_ce: 2.0568203926086426 (nan) | > loss: 0.00832492858171463 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13713592064674715) | > loader_time: 0.0047 (0.004645373806555299)  --> TIME: 2025-02-26 12:42:29 -- STEP: 25346/36938 -- GLOBAL_STEP: 1244300 | > loss_text_ce: 0.034881897270679474 (0.03135212218965579) | > loss_mel_ce: 2.4448130130767822 (nan) | > loss: 0.009840059094130993 (nan) | > current_lr: 5e-06 | > step_time: 0.1265 (0.1371186328067839) | > loader_time: 0.0046 (0.004645408662579133)  --> TIME: 2025-02-26 12:42:38 -- STEP: 25396/36938 -- GLOBAL_STEP: 1244350 | > loss_text_ce: 0.04877597093582153 (0.0313538102143413) | > loss_mel_ce: 2.713322639465332 (nan) | > loss: 0.010960709303617477 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.1371200134536755) | > loader_time: 0.0046 (0.0046453935872064495)  --> TIME: 2025-02-26 12:42:48 -- STEP: 25446/36938 -- GLOBAL_STEP: 1244400 | > loss_text_ce: 0.029017070308327675 (0.031350952919447714) | > loss_mel_ce: 3.321256637573242 (nan) | > loss: 0.013294736854732037 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13712425446684606) | > loader_time: 0.0047 (0.004645414700216152)  --> TIME: 2025-02-26 12:42:57 -- STEP: 25496/36938 -- GLOBAL_STEP: 1244450 | > loss_text_ce: 0.04449573531746864 (0.031352161340273604) | > loss_mel_ce: 2.5035510063171387 (nan) | > loss: 0.010111297480762005 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13711930810002515) | > loader_time: 0.0053 (0.004645551994073555)  --> TIME: 2025-02-26 12:43:06 -- STEP: 25546/36938 -- GLOBAL_STEP: 1244500 | > loss_text_ce: 0.026690758764743805 (0.03135129113576326) | > loss_mel_ce: 2.5011963844299316 (nan) | > loss: 0.010031298734247684 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.13711693720316218) | > loader_time: 0.0046 (0.004645721527681373)  --> TIME: 2025-02-26 12:43:16 -- STEP: 25596/36938 -- GLOBAL_STEP: 1244550 | > loss_text_ce: 0.03481398522853851 (0.03135107996093669) | > loss_mel_ce: 2.7466235160827637 (nan) | > loss: 0.011037450283765793 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13712185735832183) | > loader_time: 0.0046 (0.004645855375743865)  --> TIME: 2025-02-26 12:43:25 -- STEP: 25646/36938 -- GLOBAL_STEP: 1244600 | > loss_text_ce: 0.03033686988055706 (0.03134896830726952) | > loss_mel_ce: 2.8761184215545654 (nan) | > loss: 0.011533553712069988 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13712731704021736) | > loader_time: 0.0045 (0.004646094905356747)  --> TIME: 2025-02-26 12:43:35 -- STEP: 25696/36938 -- GLOBAL_STEP: 1244650 | > loss_text_ce: 0.038701511919498444 (0.03134816703950922) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13713175119602505) | > loader_time: 0.0046 (0.004646210081105198)  --> TIME: 2025-02-26 12:43:44 -- STEP: 25746/36938 -- GLOBAL_STEP: 1244700 | > loss_text_ce: 0.03534888103604317 (0.03134613116109737) | > loss_mel_ce: 2.8053057193756104 (nan) | > loss: 0.011272439733147621 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13713394013872068) | > loader_time: 0.0046 (0.004646268219119178)  --> TIME: 2025-02-26 12:43:53 -- STEP: 25796/36938 -- GLOBAL_STEP: 1244750 | > loss_text_ce: 0.02882310375571251 (0.03134584080953623) | > loss_mel_ce: 2.06545090675354 (nan) | > loss: 0.008310611359775066 (nan) | > current_lr: 5e-06 | > step_time: 0.1461 (0.13713106990396579) | > loader_time: 0.0047 (0.00464631510549855)  --> TIME: 2025-02-26 12:44:03 -- STEP: 25846/36938 -- GLOBAL_STEP: 1244800 | > loss_text_ce: 0.03169744461774826 (0.03134725193661994) | > loss_mel_ce: 2.5504167079925537 (nan) | > loss: 0.010246485471725464 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.1371299201265295) | > loader_time: 0.0046 (0.004646343536571761)  --> TIME: 2025-02-26 12:44:12 -- STEP: 25896/36938 -- GLOBAL_STEP: 1244850 | > loss_text_ce: 0.030508315190672874 (0.031347084598222906) | > loss_mel_ce: 2.9099483489990234 (nan) | > loss: 0.011668479070067406 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13712860837666216) | > loader_time: 0.0052 (0.004646616177975767)  --> TIME: 2025-02-26 12:44:21 -- STEP: 25946/36938 -- GLOBAL_STEP: 1244900 | > loss_text_ce: 0.03541671484708786 (0.031346556984792266) | > loss_mel_ce: 2.0605051517486572 (nan) | > loss: 0.008317150175571442 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13713516302651993) | > loader_time: 0.0046 (0.00464676392801575)  --> TIME: 2025-02-26 12:44:31 -- STEP: 25996/36938 -- GLOBAL_STEP: 1244950 | > loss_text_ce: 0.03262389451265335 (0.031347070541064455) | > loss_mel_ce: 2.76880145072937 (nan) | > loss: 0.01111676823347807 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.1371343966116627) | > loader_time: 0.0048 (0.004646724940336811)  --> TIME: 2025-02-26 12:44:40 -- STEP: 26046/36938 -- GLOBAL_STEP: 1245000 | > loss_text_ce: 0.03493586555123329 (0.03134796542788743) | > loss_mel_ce: 3.036046028137207 (nan) | > loss: 0.012186436913907528 (nan) | > current_lr: 5e-06 | > step_time: 0.1617 (0.13713869047651897) | > loader_time: 0.0047 (0.004646854220122804)  --> TIME: 2025-02-26 12:44:50 -- STEP: 26096/36938 -- GLOBAL_STEP: 1245050 | > loss_text_ce: 0.021392516791820526 (0.0313472177384253) | > loss_mel_ce: 2.368171215057373 (nan) | > loss: 0.009482396766543388 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.1371360867484664) | > loader_time: 0.0045 (0.004646817757847987)  --> TIME: 2025-02-26 12:44:59 -- STEP: 26146/36938 -- GLOBAL_STEP: 1245100 | > loss_text_ce: 0.02988172508776188 (0.03134631249681563) | > loss_mel_ce: 1.8769021034240723 (nan) | > loss: 0.007566602900624275 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13713902915745066) | > loader_time: 0.0045 (0.004647004023475848)  --> TIME: 2025-02-26 12:45:08 -- STEP: 26196/36938 -- GLOBAL_STEP: 1245150 | > loss_text_ce: 0.027792999520897865 (0.03134545286809703) | > loss_mel_ce: 2.603297710418701 (nan) | > loss: 0.010440836660563946 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.13714089566905138) | > loader_time: 0.0048 (0.004647007514946614)  --> TIME: 2025-02-26 12:45:18 -- STEP: 26246/36938 -- GLOBAL_STEP: 1245200 | > loss_text_ce: 0.020167479291558266 (0.03134510265486917) | > loss_mel_ce: 0.9806424379348755 (nan) | > loss: 0.003971468191593885 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13715139482203562) | > loader_time: 0.0045 (0.0046470243193377685)  --> TIME: 2025-02-26 12:45:28 -- STEP: 26296/36938 -- GLOBAL_STEP: 1245250 | > loss_text_ce: 0.02732369676232338 (0.031345878653376416) | > loss_mel_ce: 2.5870473384857178 (nan) | > loss: 0.010374489240348339 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.1371515458537557) | > loader_time: 0.0047 (0.004646967030021235)  --> TIME: 2025-02-26 12:45:37 -- STEP: 26346/36938 -- GLOBAL_STEP: 1245300 | > loss_text_ce: 0.03894597664475441 (0.0313457232612183) | > loss_mel_ce: 3.3487329483032227 (nan) | > loss: 0.013443171046674252 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.13714537112504546) | > loader_time: 0.0048 (0.004646875361850366)  --> TIME: 2025-02-26 12:45:46 -- STEP: 26396/36938 -- GLOBAL_STEP: 1245350 | > loss_text_ce: 0.03192615509033203 (0.03134847534456681) | > loss_mel_ce: 2.7953529357910156 (nan) | > loss: 0.011219361796975136 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13713398786688744) | > loader_time: 0.0045 (0.0046468156452413)  --> TIME: 2025-02-26 12:45:55 -- STEP: 26446/36938 -- GLOBAL_STEP: 1245400 | > loss_text_ce: 0.028254825621843338 (0.0313472609037045) | > loss_mel_ce: 0.008931444026529789 (nan) | > loss: 0.00014756456948816776 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13713532610955786) | > loader_time: 0.0046 (0.004646916707885657)  --> TIME: 2025-02-26 12:46:04 -- STEP: 26496/36938 -- GLOBAL_STEP: 1245450 | > loss_text_ce: 0.02955227717757225 (0.03134902948987801) | > loss_mel_ce: 2.9279937744140625 (nan) | > loss: 0.011736294254660606 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13713478585840136) | > loader_time: 0.0054 (0.004646987550788441)  --> TIME: 2025-02-26 12:46:14 -- STEP: 26546/36938 -- GLOBAL_STEP: 1245500 | > loss_text_ce: 0.02312605269253254 (0.03134941749511127) | > loss_mel_ce: 2.3729071617126465 (nan) | > loss: 0.009508069604635239 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13713008994785986) | > loader_time: 0.0044 (0.004647046621729718)  --> TIME: 2025-02-26 12:46:23 -- STEP: 26596/36938 -- GLOBAL_STEP: 1245550 | > loss_text_ce: 0.038926079869270325 (0.031350032443803064) | > loss_mel_ce: 3.3830699920654297 (nan) | > loss: 0.013579350896179676 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13712923447697442) | > loader_time: 0.0046 (0.004647052607189907)  --> TIME: 2025-02-26 12:46:33 -- STEP: 26646/36938 -- GLOBAL_STEP: 1245600 | > loss_text_ce: 0.031166553497314453 (0.031349578105220995) | > loss_mel_ce: 3.016018867492676 (nan) | > loss: 0.012092006392776966 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.1371412545561137) | > loader_time: 0.0046 (0.004647060028651089)  --> TIME: 2025-02-26 12:46:42 -- STEP: 26696/36938 -- GLOBAL_STEP: 1245650 | > loss_text_ce: 0.03672216460108757 (0.03134693468159741) | > loss_mel_ce: 2.740447759628296 (nan) | > loss: 0.01102051604539156 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.1371443059975278) | > loader_time: 0.0046 (0.004647079863018126)  --> TIME: 2025-02-26 12:46:52 -- STEP: 26746/36938 -- GLOBAL_STEP: 1245700 | > loss_text_ce: 0.025551313534379005 (0.03134774422780528) | > loss_mel_ce: 1.8910675048828125 (nan) | > loss: 0.0076056309044361115 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13714759953134498) | > loader_time: 0.0053 (0.004647086198476278)  --> TIME: 2025-02-26 12:47:01 -- STEP: 26796/36938 -- GLOBAL_STEP: 1245750 | > loss_text_ce: 0.029269689694046974 (0.031345729287736754) | > loss_mel_ce: 2.5917770862579346 (nan) | > loss: 0.010400979779660702 (nan) | > current_lr: 5e-06 | > step_time: 0.1846 (0.1371547084528041) | > loader_time: 0.0048 (0.0046471509849478725)  --> TIME: 2025-02-26 12:47:11 -- STEP: 26846/36938 -- GLOBAL_STEP: 1245800 | > loss_text_ce: 0.02882586605846882 (0.03134342314310711) | > loss_mel_ce: 1.8947428464889526 (nan) | > loss: 0.007633209694176912 (nan) | > current_lr: 5e-06 | > step_time: 0.1624 (0.13717011884792918) | > loader_time: 0.0047 (0.0046472897217346935)  --> TIME: 2025-02-26 12:47:20 -- STEP: 26896/36938 -- GLOBAL_STEP: 1245850 | > loss_text_ce: 0.024507569149136543 (0.03134690798788391) | > loss_mel_ce: 2.2275390625 (nan) | > loss: 0.008936692960560322 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1371568979995724) | > loader_time: 0.0045 (0.004647157390439586)  --> TIME: 2025-02-26 12:47:30 -- STEP: 26946/36938 -- GLOBAL_STEP: 1245900 | > loss_text_ce: 0.033859994262456894 (0.03134624418880814) | > loss_mel_ce: 2.847604274749756 (nan) | > loss: 0.011434382759034634 (nan) | > current_lr: 5e-06 | > step_time: 0.1812 (0.13716549513414852) | > loader_time: 0.007 (0.004647281647153208)  --> TIME: 2025-02-26 12:47:39 -- STEP: 26996/36938 -- GLOBAL_STEP: 1245950 | > loss_text_ce: 0.03218132257461548 (0.0313475577538717) | > loss_mel_ce: 2.883159875869751 (nan) | > loss: 0.011568814516067505 (nan) | > current_lr: 5e-06 | > step_time: 0.1817 (0.13716309956222614) | > loader_time: 0.0049 (0.004647231787323922)  --> TIME: 2025-02-26 12:47:48 -- STEP: 27046/36938 -- GLOBAL_STEP: 1246000 | > loss_text_ce: 0.03157597780227661 (0.031347059266682284) | > loss_mel_ce: 3.185765027999878 (nan) | > loss: 0.012767226435244083 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.1371634121628048) | > loader_time: 0.0047 (0.004647279009614145)  --> TIME: 2025-02-26 12:47:58 -- STEP: 27096/36938 -- GLOBAL_STEP: 1246050 | > loss_text_ce: 0.032113004475831985 (0.031346247041769755) | > loss_mel_ce: 2.742335557937622 (nan) | > loss: 0.011009717360138893 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.13716773183288744) | > loader_time: 0.0046 (0.004647272823481264)  --> TIME: 2025-02-26 12:48:07 -- STEP: 27146/36938 -- GLOBAL_STEP: 1246100 | > loss_text_ce: 0.028116798028349876 (0.03134474315796392) | > loss_mel_ce: 1.3550114631652832 (nan) | > loss: 0.00548860477283597 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.1371620623582164) | > loader_time: 0.0046 (0.0046473030034683855)  --> TIME: 2025-02-26 12:48:16 -- STEP: 27196/36938 -- GLOBAL_STEP: 1246150 | > loss_text_ce: 0.033705003559589386 (0.03134494154054292) | > loss_mel_ce: 3.1237435340881348 (nan) | > loss: 0.012529558502137661 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.13715703277171246) | > loader_time: 0.0048 (0.004647394772364205)  --> TIME: 2025-02-26 12:48:25 -- STEP: 27246/36938 -- GLOBAL_STEP: 1246200 | > loss_text_ce: 0.029413968324661255 (0.03134346334603802) | > loss_mel_ce: 2.8513472080230713 (nan) | > loss: 0.011431592516601086 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.13714792692021563) | > loader_time: 0.0047 (0.0046474920585888035)  --> TIME: 2025-02-26 12:48:34 -- STEP: 27296/36938 -- GLOBAL_STEP: 1246250 | > loss_text_ce: 0.03781897574663162 (0.03134569714502928) | > loss_mel_ce: 1.715032935142517 (nan) | > loss: 0.006955761928111315 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13714729587504843) | > loader_time: 0.0046 (0.004647522448515113)  --> TIME: 2025-02-26 12:48:44 -- STEP: 27346/36938 -- GLOBAL_STEP: 1246300 | > loss_text_ce: 0.030847294256091118 (0.03134477745825326) | > loss_mel_ce: 2.9774982929229736 (nan) | > loss: 0.011937879957258701 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.13714348215789277) | > loader_time: 0.0047 (0.004647483144238528)  --> TIME: 2025-02-26 12:48:53 -- STEP: 27396/36938 -- GLOBAL_STEP: 1246350 | > loss_text_ce: 0.039592474699020386 (0.03134802863070708) | > loss_mel_ce: 3.3253424167633057 (nan) | > loss: 0.01335291750729084 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.13714455909007658) | > loader_time: 0.0048 (0.004647624590115241)  --> TIME: 2025-02-26 12:49:02 -- STEP: 27446/36938 -- GLOBAL_STEP: 1246400 | > loss_text_ce: 0.03717345744371414 (0.03134909850962073) | > loss_mel_ce: 1.8748823404312134 (nan) | > loss: 0.00758752366527915 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13714459042659213) | > loader_time: 0.0047 (0.004647684333571639)  --> TIME: 2025-02-26 12:49:12 -- STEP: 27496/36938 -- GLOBAL_STEP: 1246450 | > loss_text_ce: 0.03569699823856354 (0.03135166578156403) | > loss_mel_ce: 1.60553777217865 (nan) | > loss: 0.006512836553156376 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13713835747324443) | > loader_time: 0.0047 (0.004647721211023005)  --> TIME: 2025-02-26 12:49:21 -- STEP: 27546/36938 -- GLOBAL_STEP: 1246500 | > loss_text_ce: 0.03126704320311546 (0.03135182504079805) | > loss_mel_ce: 3.2571473121643066 (nan) | > loss: 0.013049263507127762 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13712959643378897) | > loader_time: 0.0047 (0.004647824660907543)  --> TIME: 2025-02-26 12:49:30 -- STEP: 27596/36938 -- GLOBAL_STEP: 1246550 | > loss_text_ce: 0.04757465422153473 (0.03135402415453383) | > loss_mel_ce: 3.134493589401245 (nan) | > loss: 0.012627256102859974 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13713278415704455) | > loader_time: 0.0047 (0.0046480718618338275)  --> TIME: 2025-02-26 12:49:39 -- STEP: 27646/36938 -- GLOBAL_STEP: 1246600 | > loss_text_ce: 0.031898513436317444 (0.031351959221369075) | > loss_mel_ce: 2.5781471729278564 (nan) | > loss: 0.010357324965298176 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13713240016345762) | > loader_time: 0.0045 (0.004648236309767077)  --> TIME: 2025-02-26 12:49:48 -- STEP: 27696/36938 -- GLOBAL_STEP: 1246650 | > loss_text_ce: 0.034712959080934525 (0.03135559196410902) | > loss_mel_ce: 3.0447511672973633 (nan) | > loss: 0.01222009677439928 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13712343910159075) | > loader_time: 0.0047 (0.004648373124917603)  --> TIME: 2025-02-26 12:49:58 -- STEP: 27746/36938 -- GLOBAL_STEP: 1246700 | > loss_text_ce: 0.05071738734841347 (0.031357597457633175) | > loss_mel_ce: 2.5317909717559814 (nan) | > loss: 0.010248049162328243 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13712935359873346) | > loader_time: 0.0045 (0.0046485791439742795)  --> TIME: 2025-02-26 12:50:07 -- STEP: 27796/36938 -- GLOBAL_STEP: 1246750 | > loss_text_ce: 0.03565062955021858 (0.03135973407852693) | > loss_mel_ce: 2.4232261180877686 (nan) | > loss: 0.009757447987794876 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1371193924453168) | > loader_time: 0.0046 (0.004648650373584279)  --> TIME: 2025-02-26 12:50:16 -- STEP: 27846/36938 -- GLOBAL_STEP: 1246800 | > loss_text_ce: 0.04088333994150162 (0.03136078183182561) | > loss_mel_ce: 2.51434588432312 (nan) | > loss: 0.010139798745512962 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1371191431716925) | > loader_time: 0.0045 (0.004648647919336797)  --> TIME: 2025-02-26 12:50:26 -- STEP: 27896/36938 -- GLOBAL_STEP: 1246850 | > loss_text_ce: 0.03236027806997299 (0.03136274597967394) | > loss_mel_ce: 2.886573553085327 (nan) | > loss: 0.011583071202039719 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.13712165280984184) | > loader_time: 0.0046 (0.0046485613744170476)  --> TIME: 2025-02-26 12:50:35 -- STEP: 27946/36938 -- GLOBAL_STEP: 1246900 | > loss_text_ce: 0.03836699202656746 (0.03136213564243706) | > loss_mel_ce: 2.411330223083496 (nan) | > loss: 0.009721021167933941 (nan) | > current_lr: 5e-06 | > step_time: 0.1871 (0.13710677754279166) | > loader_time: 0.0049 (0.004648571467251131)  --> TIME: 2025-02-26 12:50:44 -- STEP: 27996/36938 -- GLOBAL_STEP: 1246950 | > loss_text_ce: 0.040600232779979706 (0.031361228511679454) | > loss_mel_ce: 2.4683210849761963 (nan) | > loss: 0.009956037625670433 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13711179036006704) | > loader_time: 0.0048 (0.004648664096778297)  --> TIME: 2025-02-26 12:50:54 -- STEP: 28046/36938 -- GLOBAL_STEP: 1247000 | > loss_text_ce: 0.030003434047102928 (0.03136034053062961) | > loss_mel_ce: 2.332076072692871 (nan) | > loss: 0.009373332373797894 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.1371145818404015) | > loader_time: 0.0046 (0.004648732406253301)  --> TIME: 2025-02-26 12:51:03 -- STEP: 28096/36938 -- GLOBAL_STEP: 1247050 | > loss_text_ce: 0.025333672761917114 (0.031360769780037365) | > loss_mel_ce: 2.7134389877319336 (nan) | > loss: 0.01086814608424902 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.1371121824483524) | > loader_time: 0.0047 (0.004648780420524364)  --> TIME: 2025-02-26 12:51:12 -- STEP: 28146/36938 -- GLOBAL_STEP: 1247100 | > loss_text_ce: 0.03589116781949997 (0.031360133760332524) | > loss_mel_ce: 1.0849806070327759 (nan) | > loss: 0.004447903949767351 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13711255723154503) | > loader_time: 0.0045 (0.00464872949490843)  --> TIME: 2025-02-26 12:51:22 -- STEP: 28196/36938 -- GLOBAL_STEP: 1247150 | > loss_text_ce: 0.030408181250095367 (0.03135887109067756) | > loss_mel_ce: 2.9494996070861816 (nan) | > loss: 0.011825031600892544 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13712061589652663) | > loader_time: 0.0045 (0.004648708505721119)  --> TIME: 2025-02-26 12:51:31 -- STEP: 28246/36938 -- GLOBAL_STEP: 1247200 | > loss_text_ce: 0.03210277855396271 (0.031358400085440656) | > loss_mel_ce: 2.61098313331604 (nan) | > loss: 0.010488436557352543 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1371197258709487) | > loader_time: 0.0046 (0.004648675621800874)  --> TIME: 2025-02-26 12:51:41 -- STEP: 28296/36938 -- GLOBAL_STEP: 1247250 | > loss_text_ce: 0.03885693475604057 (0.03135973793883474) | > loss_mel_ce: 2.7596819400787354 (nan) | > loss: 0.011105313897132874 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13712843236425687) | > loader_time: 0.0046 (0.004648738631023854)  --> TIME: 2025-02-26 12:51:50 -- STEP: 28346/36938 -- GLOBAL_STEP: 1247300 | > loss_text_ce: 0.03551165759563446 (0.031358781512187406) | > loss_mel_ce: 2.616067886352539 (nan) | > loss: 0.010522142052650452 (nan) | > current_lr: 5e-06 | > step_time: 0.1842 (0.1371261898147214) | > loader_time: 0.0049 (0.004648695060699277)  --> TIME: 2025-02-26 12:51:59 -- STEP: 28396/36938 -- GLOBAL_STEP: 1247350 | > loss_text_ce: 0.02694990485906601 (0.03136008840246412) | > loss_mel_ce: 2.1873738765716553 (nan) | > loss: 0.008786999620497227 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1371238005885031) | > loader_time: 0.0045 (0.004648689191632561)  --> TIME: 2025-02-26 12:52:09 -- STEP: 28446/36938 -- GLOBAL_STEP: 1247400 | > loss_text_ce: 0.03585498407483101 (0.03136399479698882) | > loss_mel_ce: 1.7028528451919556 (nan) | > loss: 0.006899634376168251 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.1371229597597372) | > loader_time: 0.0046 (0.0046487337659701375)  --> TIME: 2025-02-26 12:52:18 -- STEP: 28496/36938 -- GLOBAL_STEP: 1247450 | > loss_text_ce: 0.033708155155181885 (0.03136266544647227) | > loss_mel_ce: 2.2407567501068115 (nan) | > loss: 0.009025654755532742 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1371245026119896) | > loader_time: 0.0054 (0.004648727372681921)  --> TIME: 2025-02-26 12:52:27 -- STEP: 28546/36938 -- GLOBAL_STEP: 1247500 | > loss_text_ce: 0.02124338038265705 (0.031363001249886426) | > loss_mel_ce: 2.772751569747925 (nan) | > loss: 0.011087281629443169 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.1371215660403502) | > loader_time: 0.0046 (0.004648786949842432)  --> TIME: 2025-02-26 12:52:37 -- STEP: 28596/36938 -- GLOBAL_STEP: 1247550 | > loss_text_ce: 0.033531785011291504 (0.03136372322255792) | > loss_mel_ce: 2.533815383911133 (nan) | > loss: 0.010187885724008083 (nan) | > current_lr: 5e-06 | > step_time: 0.1515 (0.13712788514447916) | > loader_time: 0.0047 (0.0046488950428953835)  --> TIME: 2025-02-26 12:52:46 -- STEP: 28646/36938 -- GLOBAL_STEP: 1247600 | > loss_text_ce: 0.03405259922146797 (0.03136463814203985) | > loss_mel_ce: 1.2437012195587158 (nan) | > loss: 0.0050704521127045155 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1371261042389363) | > loader_time: 0.0046 (0.004648964140224244)  --> TIME: 2025-02-26 12:52:56 -- STEP: 28696/36938 -- GLOBAL_STEP: 1247650 | > loss_text_ce: 0.026878800243139267 (0.031362367985092725) | > loss_mel_ce: 2.9725611209869385 (nan) | > loss: 0.011902539990842342 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.13712928493374482) | > loader_time: 0.0046 (0.004649240242128802)  --> TIME: 2025-02-26 12:53:05 -- STEP: 28746/36938 -- GLOBAL_STEP: 1247700 | > loss_text_ce: 0.03348410874605179 (0.03136196672235869) | > loss_mel_ce: 2.443633556365967 (nan) | > loss: 0.009829833172261715 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13713602905091582) | > loader_time: 0.0045 (0.004649397269060305)  --> TIME: 2025-02-26 12:53:14 -- STEP: 28796/36938 -- GLOBAL_STEP: 1247750 | > loss_text_ce: 0.0384504459798336 (0.03136190560146426) | > loss_mel_ce: 1.210673213005066 (nan) | > loss: 0.0049568405374884605 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13712315107156886) | > loader_time: 0.0045 (0.00464941966300175)  --> TIME: 2025-02-26 12:53:24 -- STEP: 28846/36938 -- GLOBAL_STEP: 1247800 | > loss_text_ce: 0.03531772270798683 (0.03136347690755715) | > loss_mel_ce: 2.2457988262176514 (nan) | > loss: 0.009052050299942493 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13712209942915562) | > loader_time: 0.0047 (0.004649527350783221)  --> TIME: 2025-02-26 12:53:33 -- STEP: 28896/36938 -- GLOBAL_STEP: 1247850 | > loss_text_ce: 0.044403620064258575 (0.03136219672393903) | > loss_mel_ce: 3.129218816757202 (nan) | > loss: 0.012593740597367287 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13711856340253023) | > loader_time: 0.0045 (0.004649618857128656)  --> TIME: 2025-02-26 12:53:42 -- STEP: 28946/36938 -- GLOBAL_STEP: 1247900 | > loss_text_ce: 0.020735276862978935 (0.03136282272638254) | > loss_mel_ce: 0.1256103515625 (nan) | > loss: 0.000580736668780446 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.137119286821236) | > loader_time: 0.0047 (0.0046496178707996876)  --> TIME: 2025-02-26 12:53:51 -- STEP: 28996/36938 -- GLOBAL_STEP: 1247950 | > loss_text_ce: 0.029032932594418526 (0.03136135528929753) | > loss_mel_ce: 2.1982953548431396 (nan) | > loss: 0.008838605135679245 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13710871623752624) | > loader_time: 0.0045 (0.004649557858800059)  --> TIME: 2025-02-26 12:54:00 -- STEP: 29046/36938 -- GLOBAL_STEP: 1248000 | > loss_text_ce: 0.03763289749622345 (0.03136069438445127) | > loss_mel_ce: 2.0261807441711426 (nan) | > loss: 0.008189736865460873 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13710480621019744) | > loader_time: 0.0045 (0.004649588090365608)  --> TIME: 2025-02-26 12:54:10 -- STEP: 29096/36938 -- GLOBAL_STEP: 1248050 | > loss_text_ce: 0.03012196160852909 (0.03135987734755309) | > loss_mel_ce: 1.0701980590820312 (nan) | > loss: 0.004366349428892136 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13710710503814383) | > loader_time: 0.0045 (0.004649634712962335)  --> TIME: 2025-02-26 12:54:19 -- STEP: 29146/36938 -- GLOBAL_STEP: 1248100 | > loss_text_ce: 0.028506189584732056 (0.03135953857245826) | > loss_mel_ce: 1.4931946992874146 (nan) | > loss: 0.006038495805114508 (nan) | > current_lr: 5e-06 | > step_time: 0.1118 (0.1371034359151553) | > loader_time: 0.0045 (0.004649686582674313)  --> TIME: 2025-02-26 12:54:29 -- STEP: 29196/36938 -- GLOBAL_STEP: 1248150 | > loss_text_ce: 0.027795402333140373 (0.03136103922610757) | > loss_mel_ce: 0.6927986145019531 (nan) | > loss: 0.0028595002368092537 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13710373145192373) | > loader_time: 0.0046 (0.004649690470151285)  --> TIME: 2025-02-26 12:54:38 -- STEP: 29246/36938 -- GLOBAL_STEP: 1248200 | > loss_text_ce: 0.024013394489884377 (0.03136050251149138) | > loss_mel_ce: 2.6125435829162598 (nan) | > loss: 0.010462528094649315 (nan) | > current_lr: 5e-06 | > step_time: 0.1527 (0.13711409014275452) | > loader_time: 0.0046 (0.004649632933983858)  --> TIME: 2025-02-26 12:54:48 -- STEP: 29296/36938 -- GLOBAL_STEP: 1248250 | > loss_text_ce: 0.03088315576314926 (0.03136120659744837) | > loss_mel_ce: 2.3657970428466797 (nan) | > loss: 0.009510636329650879 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13711883509373865) | > loader_time: 0.0046 (0.004649583740614575)  --> TIME: 2025-02-26 12:54:57 -- STEP: 29346/36938 -- GLOBAL_STEP: 1248300 | > loss_text_ce: 0.023749446496367455 (0.03136197182074784) | > loss_mel_ce: 0.1465555876493454 (nan) | > loss: 0.0006758136441931129 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1371123717865112) | > loader_time: 0.0045 (0.004649499983076795)  --> TIME: 2025-02-26 12:55:06 -- STEP: 29396/36938 -- GLOBAL_STEP: 1248350 | > loss_text_ce: 0.02934388257563114 (0.031363617878450795) | > loss_mel_ce: 1.4169921875 (nan) | > loss: 0.005739429034292698 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13710234046452713) | > loader_time: 0.0045 (0.0046494469981336745)  --> TIME: 2025-02-26 12:55:15 -- STEP: 29446/36938 -- GLOBAL_STEP: 1248400 | > loss_text_ce: 0.026766914874315262 (0.03136260981916575) | > loss_mel_ce: 3.3066842555999756 (nan) | > loss: 0.013227982446551323 (nan) | > current_lr: 5e-06 | > step_time: 0.189 (0.1371013361143119) | > loader_time: 0.0048 (0.00464938527044859)  --> TIME: 2025-02-26 12:55:25 -- STEP: 29496/36938 -- GLOBAL_STEP: 1248450 | > loss_text_ce: 0.03155228868126869 (0.031361916839308245) | > loss_mel_ce: 3.1277852058410645 (nan) | > loss: 0.012537054717540741 (nan) | > current_lr: 5e-06 | > step_time: 0.1888 (0.13710399794487968) | > loader_time: 0.005 (0.004649353166372737)  --> TIME: 2025-02-26 12:55:34 -- STEP: 29546/36938 -- GLOBAL_STEP: 1248500 | > loss_text_ce: 0.03140890598297119 (0.0313618825431934) | > loss_mel_ce: 2.666635513305664 (nan) | > loss: 0.010706525295972824 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.13710331430224026) | > loader_time: 0.0046 (0.004649271084170812)  --> TIME: 2025-02-26 12:55:44 -- STEP: 29596/36938 -- GLOBAL_STEP: 1248550 | > loss_text_ce: 0.03194331005215645 (0.03136275267160963) | > loss_mel_ce: 2.3103597164154053 (nan) | > loss: 0.009294853545725346 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13711597908508202) | > loader_time: 0.0045 (0.004649222565754164)  --> TIME: 2025-02-26 12:55:53 -- STEP: 29646/36938 -- GLOBAL_STEP: 1248600 | > loss_text_ce: 0.03589678183197975 (0.03136132110613887) | > loss_mel_ce: 2.0109522342681885 (nan) | > loss: 0.008122417144477367 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13710982481220327) | > loader_time: 0.0047 (0.00464910171071035)  --> TIME: 2025-02-26 12:56:02 -- STEP: 29696/36938 -- GLOBAL_STEP: 1248650 | > loss_text_ce: 0.026660237461328506 (0.03136216788477129) | > loss_mel_ce: 2.7691798210144043 (nan) | > loss: 0.011094603687524796 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.13711635302752193) | > loader_time: 0.0046 (0.004649061741756963)  --> TIME: 2025-02-26 12:56:12 -- STEP: 29746/36938 -- GLOBAL_STEP: 1248700 | > loss_text_ce: 0.032159462571144104 (0.031360393784069795) | > loss_mel_ce: 2.0716094970703125 (nan) | > loss: 0.00834829080849886 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.1371137478689345) | > loader_time: 0.0047 (0.004648954724204278)  --> TIME: 2025-02-26 12:56:21 -- STEP: 29796/36938 -- GLOBAL_STEP: 1248750 | > loss_text_ce: 0.029605623334646225 (0.03135938291527136) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1371119649542429) | > loader_time: 0.0045 (0.004648889002503586)  --> TIME: 2025-02-26 12:56:30 -- STEP: 29846/36938 -- GLOBAL_STEP: 1248800 | > loss_text_ce: 0.02717636339366436 (0.03135937824048049) | > loss_mel_ce: 2.8233838081359863 (nan) | > loss: 0.011311747133731842 (nan) | > current_lr: 5e-06 | > step_time: 0.185 (0.13711521608423782) | > loader_time: 0.0048 (0.004648838311299969)  --> TIME: 2025-02-26 12:56:40 -- STEP: 29896/36938 -- GLOBAL_STEP: 1248850 | > loss_text_ce: 0.027831241488456726 (0.0313575356062568) | > loss_mel_ce: 2.628847122192383 (nan) | > loss: 0.010542375035583973 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13711286861532146) | > loader_time: 0.0045 (0.0046487385125475195)  --> TIME: 2025-02-26 12:56:49 -- STEP: 29946/36938 -- GLOBAL_STEP: 1248900 | > loss_text_ce: 0.025329455733299255 (0.031358415309131686) | > loss_mel_ce: 0.06787109375 (nan) | > loss: 0.0003698434738907963 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13710975280738208) | > loader_time: 0.0045 (0.004648625018688507)  --> TIME: 2025-02-26 12:56:58 -- STEP: 29996/36938 -- GLOBAL_STEP: 1248950 | > loss_text_ce: 0.03221878781914711 (0.03135869395538811) | > loss_mel_ce: 2.314359664916992 (nan) | > loss: 0.009311819449067116 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1371054553416487) | > loader_time: 0.0044 (0.004648599867535216)  --> TIME: 2025-02-26 12:57:07 -- STEP: 30046/36938 -- GLOBAL_STEP: 1249000 | > loss_text_ce: 0.032382842153310776 (0.0313584802245662) | > loss_mel_ce: 2.80291748046875 (nan) | > loss: 0.011251191608607769 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.1371015044460989) | > loader_time: 0.0045 (0.004648522721906145)  --> TIME: 2025-02-26 12:57:17 -- STEP: 30096/36938 -- GLOBAL_STEP: 1249050 | > loss_text_ce: 0.031897783279418945 (0.03135862022684189) | > loss_mel_ce: 2.25602126121521 (nan) | > loss: 0.009079044684767723 (nan) | > current_lr: 5e-06 | > step_time: 0.1526 (0.1371087065518117) | > loader_time: 0.0046 (0.00464844652181732)  --> TIME: 2025-02-26 12:57:27 -- STEP: 30146/36938 -- GLOBAL_STEP: 1249100 | > loss_text_ce: 0.03817509114742279 (0.031357773892008264) | > loss_mel_ce: 2.8510360717773438 (nan) | > loss: 0.011465124785900116 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.1371187904348792) | > loader_time: 0.0046 (0.004648414412957608)  --> TIME: 2025-02-26 12:57:36 -- STEP: 30196/36938 -- GLOBAL_STEP: 1249150 | > loss_text_ce: 0.027707507833838463 (0.031358720402803075) | > loss_mel_ce: 2.3731791973114014 (nan) | > loss: 0.009527329355478287 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13711526765304072) | > loader_time: 0.0047 (0.004648389982409704)  --> TIME: 2025-02-26 12:57:46 -- STEP: 30246/36938 -- GLOBAL_STEP: 1249200 | > loss_text_ce: 0.03306453302502632 (0.0313592337362476) | > loss_mel_ce: 4.87890625 (nan) | > loss: 0.019491948187351227 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.13712074741960517) | > loader_time: 0.0046 (0.004648427787316108)  --> TIME: 2025-02-26 12:57:55 -- STEP: 30296/36938 -- GLOBAL_STEP: 1249250 | > loss_text_ce: 0.03305155783891678 (0.03135970392972262) | > loss_mel_ce: 2.1019489765167236 (nan) | > loss: 0.008472224697470665 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13711543027473802) | > loader_time: 0.0052 (0.004648408648644754)  --> TIME: 2025-02-26 12:58:04 -- STEP: 30346/36938 -- GLOBAL_STEP: 1249300 | > loss_text_ce: 0.038949813693761826 (0.03136123503775744) | > loss_mel_ce: 3.0854413509368896 (nan) | > loss: 0.012398377992212772 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13712151328850888) | > loader_time: 0.0046 (0.004648484607350318)  --> TIME: 2025-02-26 12:58:14 -- STEP: 30396/36938 -- GLOBAL_STEP: 1249350 | > loss_text_ce: 0.035222359001636505 (0.031359947394259895) | > loss_mel_ce: 2.5272512435913086 (nan) | > loss: 0.010168546810746193 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13713156324355902) | > loader_time: 0.0045 (0.00464881046898568)  --> TIME: 2025-02-26 12:58:23 -- STEP: 30446/36938 -- GLOBAL_STEP: 1249400 | > loss_text_ce: 0.028554920107126236 (0.031361819872231786) | > loss_mel_ce: 2.2086498737335205 (nan) | > loss: 0.008877797052264214 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.1371252522129953) | > loader_time: 0.0048 (0.004648794304374652)  --> TIME: 2025-02-26 12:58:32 -- STEP: 30496/36938 -- GLOBAL_STEP: 1249450 | > loss_text_ce: 0.026452017948031425 (0.03136072186650237) | > loss_mel_ce: 2.6677987575531006 (nan) | > loss: 0.010691472329199314 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.1371200373718764) | > loader_time: 0.0054 (0.0046489315824768665)  --> TIME: 2025-02-26 12:58:42 -- STEP: 30546/36938 -- GLOBAL_STEP: 1249500 | > loss_text_ce: 0.026953119784593582 (0.03136084096466055) | > loss_mel_ce: 2.612225294113159 (nan) | > loss: 0.01047293096780777 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.1371200093037869) | > loader_time: 0.0045 (0.004648977534862893)  --> TIME: 2025-02-26 12:58:51 -- STEP: 30596/36938 -- GLOBAL_STEP: 1249550 | > loss_text_ce: 0.033531781286001205 (0.03136033960735272) | > loss_mel_ce: 2.659954309463501 (nan) | > loss: 0.010688437148928642 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13711208334965236) | > loader_time: 0.0046 (0.004648878295712397)  --> TIME: 2025-02-26 12:59:00 -- STEP: 30646/36938 -- GLOBAL_STEP: 1249600 | > loss_text_ce: 0.03905286639928818 (0.03136164601337623) | > loss_mel_ce: 2.6633169651031494 (nan) | > loss: 0.010723690502345562 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.1371078083940402) | > loader_time: 0.0046 (0.004648959886418566)  --> TIME: 2025-02-26 12:59:09 -- STEP: 30696/36938 -- GLOBAL_STEP: 1249650 | > loss_text_ce: 0.021991580724716187 (0.03136169031140249) | > loss_mel_ce: 2.7717156410217285 (nan) | > loss: 0.01108613982796669 (nan) | > current_lr: 5e-06 | > step_time: 0.1779 (0.13710874831704373) | > loader_time: 0.0048 (0.004649092823632018)  --> TIME: 2025-02-26 12:59:19 -- STEP: 30746/36938 -- GLOBAL_STEP: 1249700 | > loss_text_ce: 0.034105680882930756 (0.03136213655861755) | > loss_mel_ce: 3.078278064727783 (nan) | > loss: 0.01235073059797287 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13711156394868845) | > loader_time: 0.0045 (0.004649241690380386)  --> TIME: 2025-02-26 12:59:28 -- STEP: 30796/36938 -- GLOBAL_STEP: 1249750 | > loss_text_ce: 0.023702513426542282 (0.03136085851067104) | > loss_mel_ce: 2.0053393840789795 (nan) | > loss: 0.008051754906773567 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.1371082910656504) | > loader_time: 0.0046 (0.004649295948096426)  --> TIME: 2025-02-26 12:59:37 -- STEP: 30846/36938 -- GLOBAL_STEP: 1249800 | > loss_text_ce: 0.038856517523527145 (0.03135811034360244) | > loss_mel_ce: 1.5222296714782715 (nan) | > loss: 0.006194786634296179 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1371060424468424) | > loader_time: 0.0045 (0.0046492949044094406)  --> TIME: 2025-02-26 12:59:47 -- STEP: 30896/36938 -- GLOBAL_STEP: 1249850 | > loss_text_ce: 0.030023016035556793 (0.031357741239634446) | > loss_mel_ce: 3.1036455631256104 (nan) | > loss: 0.012435194104909897 (nan) | > current_lr: 5e-06 | > step_time: 0.1925 (0.13710815702183443) | > loader_time: 0.0049 (0.004649294365693207)  --> TIME: 2025-02-26 12:59:56 -- STEP: 30946/36938 -- GLOBAL_STEP: 1249900 | > loss_text_ce: 0.03926404193043709 (0.03135940544976246) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.1371084026549796) | > loader_time: 0.0046 (0.004649378792205498)  --> TIME: 2025-02-26 13:00:05 -- STEP: 30996/36938 -- GLOBAL_STEP: 1249950 | > loss_text_ce: 0.031243856996297836 (0.0313610941492273) | > loss_mel_ce: 2.5056703090667725 (nan) | > loss: 0.01006712019443512 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13710843276817478) | > loader_time: 0.0046 (0.004649472199712093)  --> TIME: 2025-02-26 13:00:15 -- STEP: 31046/36938 -- GLOBAL_STEP: 1250000 | > loss_text_ce: 0.03173502907156944 (0.03136215632053214) | > loss_mel_ce: 2.6835317611694336 (nan) | > loss: 0.010774868540465832 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.13711281626185895) | > loader_time: 0.0049 (0.004649656147469334) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1250000.pth  --> TIME: 2025-02-26 13:00:30 -- STEP: 31096/36938 -- GLOBAL_STEP: 1250050 | > loss_text_ce: 0.03456389531493187 (0.03136292115713843) | > loss_mel_ce: 2.8931634426116943 (nan) | > loss: 0.011617965996265411 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1371074774086177) | > loader_time: 0.0046 (0.004649769640349146)  --> TIME: 2025-02-26 13:00:39 -- STEP: 31146/36938 -- GLOBAL_STEP: 1250100 | > loss_text_ce: 0.025590483099222183 (0.03136233717929699) | > loss_mel_ce: 1.3800323009490967 (nan) | > loss: 0.005577868781983852 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.1371017982372918) | > loader_time: 0.0045 (0.004649758292998985)  --> TIME: 2025-02-26 13:00:48 -- STEP: 31196/36938 -- GLOBAL_STEP: 1250150 | > loss_text_ce: 0.03404540568590164 (0.03136235865290424) | > loss_mel_ce: 3.08077335357666 (nan) | > loss: 0.012360393069684505 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13710503753812936) | > loader_time: 0.0047 (0.004649777896344024)  --> TIME: 2025-02-26 13:00:58 -- STEP: 31246/36938 -- GLOBAL_STEP: 1250200 | > loss_text_ce: 0.038928158581256866 (0.031362227943475506) | > loss_mel_ce: 2.8168907165527344 (nan) | > loss: 0.011332615278661251 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13710597291963555) | > loader_time: 0.0046 (0.004649990866860725)  --> TIME: 2025-02-26 13:01:07 -- STEP: 31296/36938 -- GLOBAL_STEP: 1250250 | > loss_text_ce: 0.031310927122831345 (0.03136118388254636) | > loss_mel_ce: 2.730825185775757 (nan) | > loss: 0.010960858315229416 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.137098143619993) | > loader_time: 0.0046 (0.004649952594914545)  --> TIME: 2025-02-26 13:01:16 -- STEP: 31346/36938 -- GLOBAL_STEP: 1250300 | > loss_text_ce: 0.0337599515914917 (0.031359718623248774) | > loss_mel_ce: 2.1048061847686768 (nan) | > loss: 0.008486373350024223 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1370945347296297) | > loader_time: 0.0044 (0.004650022481096744)  --> TIME: 2025-02-26 13:01:26 -- STEP: 31396/36938 -- GLOBAL_STEP: 1250350 | > loss_text_ce: 0.037773482501506805 (0.03136067611470233) | > loss_mel_ce: 2.6731350421905518 (nan) | > loss: 0.010757573880255222 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.1371002298416576) | > loader_time: 0.0046 (0.004650043786627588)  --> TIME: 2025-02-26 13:01:35 -- STEP: 31446/36938 -- GLOBAL_STEP: 1250400 | > loss_text_ce: 0.03445690870285034 (0.031358823188713456) | > loss_mel_ce: 2.805414915084839 (nan) | > loss: 0.011269333772361279 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.1371017929055427) | > loader_time: 0.0046 (0.004650032589288927)  --> TIME: 2025-02-26 13:01:44 -- STEP: 31496/36938 -- GLOBAL_STEP: 1250450 | > loss_text_ce: 0.03218669071793556 (0.03135799311046381) | > loss_mel_ce: 2.9835853576660156 (nan) | > loss: 0.011967350728809834 (nan) | > current_lr: 5e-06 | > step_time: 0.1929 (0.13708801453628838) | > loader_time: 0.0057 (0.004650005977529828)  --> TIME: 2025-02-26 13:01:53 -- STEP: 31546/36938 -- GLOBAL_STEP: 1250500 | > loss_text_ce: 0.03127046301960945 (0.031358501479915486) | > loss_mel_ce: 2.508148193359375 (nan) | > loss: 0.010077059268951416 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13709303812453652) | > loader_time: 0.0045 (0.004650120048012628)  --> TIME: 2025-02-26 13:02:03 -- STEP: 31596/36938 -- GLOBAL_STEP: 1250550 | > loss_text_ce: 0.02466314658522606 (0.031357983957412225) | > loss_mel_ce: 3.0397608280181885 (nan) | > loss: 0.012160413898527622 (nan) | > current_lr: 5e-06 | > step_time: 0.1702 (0.13709294036123137) | > loader_time: 0.0047 (0.004650128492902735)  --> TIME: 2025-02-26 13:02:12 -- STEP: 31646/36938 -- GLOBAL_STEP: 1250600 | > loss_text_ce: 0.03141206502914429 (0.03135864320134602) | > loss_mel_ce: 2.2141201496124268 (nan) | > loss: 0.008910843171179295 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13709956496070544) | > loader_time: 0.0046 (0.004650161479235596)  --> TIME: 2025-02-26 13:02:22 -- STEP: 31696/36938 -- GLOBAL_STEP: 1250650 | > loss_text_ce: 0.02349640056490898 (0.03135876466838002) | > loss_mel_ce: 1.6960618495941162 (nan) | > loss: 0.006823644042015076 (nan) | > current_lr: 5e-06 | > step_time: 0.1262 (0.13710623042322231) | > loader_time: 0.0067 (0.004650344945215788)  --> TIME: 2025-02-26 13:02:32 -- STEP: 31746/36938 -- GLOBAL_STEP: 1250700 | > loss_text_ce: 0.039441149681806564 (0.03135943022633587) | > loss_mel_ce: 2.979287624359131 (nan) | > loss: 0.011979082599282265 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.13711211461016842) | > loader_time: 0.0046 (0.004650345876327571)  --> TIME: 2025-02-26 13:02:41 -- STEP: 31796/36938 -- GLOBAL_STEP: 1250750 | > loss_text_ce: 0.02844993956387043 (0.03135927345106998) | > loss_mel_ce: 2.118570327758789 (nan) | > loss: 0.008519922383129597 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.1371039987990108) | > loader_time: 0.0045 (0.0046502640523585215)  --> TIME: 2025-02-26 13:02:50 -- STEP: 31846/36938 -- GLOBAL_STEP: 1250800 | > loss_text_ce: 0.048219021409749985 (0.031359775339661) | > loss_mel_ce: 1.9212517738342285 (nan) | > loss: 0.007815360091626644 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13710229426616707) | > loader_time: 0.0045 (0.004650310865712569)  --> TIME: 2025-02-26 13:02:59 -- STEP: 31896/36938 -- GLOBAL_STEP: 1250850 | > loss_text_ce: 0.023470954969525337 (0.03135841307364903) | > loss_mel_ce: 2.7752859592437744 (nan) | > loss: 0.011106178164482117 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13709897972719925) | > loader_time: 0.0045 (0.004650272415908363)  --> TIME: 2025-02-26 13:03:09 -- STEP: 31946/36938 -- GLOBAL_STEP: 1250900 | > loss_text_ce: 0.04443700239062309 (0.031357976079740106) | > loss_mel_ce: 2.8862974643707275 (nan) | > loss: 0.01162989903241396 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13709886521259523) | > loader_time: 0.0045 (0.004650485729787579)  --> TIME: 2025-02-26 13:03:18 -- STEP: 31996/36938 -- GLOBAL_STEP: 1250950 | > loss_text_ce: 0.034435223788022995 (0.0313584120698737) | > loss_mel_ce: 2.3871254920959473 (nan) | > loss: 0.00960936862975359 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13709791079359085) | > loader_time: 0.0046 (0.004650543549460503)  --> TIME: 2025-02-26 13:03:27 -- STEP: 32046/36938 -- GLOBAL_STEP: 1251000 | > loss_text_ce: 0.03984810784459114 (0.0313564691472329) | > loss_mel_ce: 1.8104101419448853 (nan) | > loss: 0.007342294789850712 (nan) | > current_lr: 5e-06 | > step_time: 0.1886 (0.13709940334045737) | > loader_time: 0.0049 (0.004650713129227475)  --> TIME: 2025-02-26 13:03:37 -- STEP: 32096/36938 -- GLOBAL_STEP: 1251050 | > loss_text_ce: 0.03161129727959633 (0.03135897098385652) | > loss_mel_ce: 2.9976258277893066 (nan) | > loss: 0.012020782567560673 (nan) | > current_lr: 5e-06 | > step_time: 0.1526 (0.1370988333911618) | > loader_time: 0.0054 (0.004650869025131029)  --> TIME: 2025-02-26 13:03:46 -- STEP: 32146/36938 -- GLOBAL_STEP: 1251100 | > loss_text_ce: 0.04421219229698181 (0.031358418416430334) | > loss_mel_ce: 1.822052001953125 (nan) | > loss: 0.0074058109894394875 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.13709594911476636) | > loader_time: 0.0046 (0.004650902015333812)  --> TIME: 2025-02-26 13:03:55 -- STEP: 32196/36938 -- GLOBAL_STEP: 1251150 | > loss_text_ce: 0.03271136060357094 (0.0313595798479567) | > loss_mel_ce: 3.21820068359375 (nan) | > loss: 0.01290044467896223 (nan) | > current_lr: 5e-06 | > step_time: 0.1615 (0.13710143350195203) | > loader_time: 0.0048 (0.00465108844297599)  --> TIME: 2025-02-26 13:04:05 -- STEP: 32246/36938 -- GLOBAL_STEP: 1251200 | > loss_text_ce: 0.02775459550321102 (0.03135653197594539) | > loss_mel_ce: 2.2681121826171875 (nan) | > loss: 0.009110582992434502 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13710705007948037) | > loader_time: 0.0045 (0.004651066483985766)  --> TIME: 2025-02-26 13:04:14 -- STEP: 32296/36938 -- GLOBAL_STEP: 1251250 | > loss_text_ce: 0.02976362220942974 (0.031355424016927584) | > loss_mel_ce: 2.3512461185455322 (nan) | > loss: 0.009448451921343803 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13711191938600764) | > loader_time: 0.0045 (0.004651085896925475)  --> TIME: 2025-02-26 13:04:24 -- STEP: 32346/36938 -- GLOBAL_STEP: 1251300 | > loss_text_ce: 0.035747937858104706 (0.031355116958532665) | > loss_mel_ce: 2.997011423110962 (nan) | > loss: 0.012034760788083076 (nan) | > current_lr: 5e-06 | > step_time: 0.1813 (0.13710562037985208) | > loader_time: 0.0048 (0.004651151450542851)  --> TIME: 2025-02-26 13:04:33 -- STEP: 32396/36938 -- GLOBAL_STEP: 1251350 | > loss_text_ce: 0.031083034351468086 (0.03135451580016758) | > loss_mel_ce: 2.63730525970459 (nan) | > loss: 0.010588843375444412 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13711355201019557) | > loader_time: 0.0046 (0.004651287357400014)  --> TIME: 2025-02-26 13:04:43 -- STEP: 32446/36938 -- GLOBAL_STEP: 1251400 | > loss_text_ce: 0.030879292637109756 (0.031354258174537325) | > loss_mel_ce: 2.891000747680664 (nan) | > loss: 0.011594763025641441 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.1371136079959972) | > loader_time: 0.0046 (0.004651336078247568)  --> TIME: 2025-02-26 13:04:52 -- STEP: 32496/36938 -- GLOBAL_STEP: 1251450 | > loss_text_ce: 0.03186910226941109 (0.03135210364158056) | > loss_mel_ce: 2.2029595375061035 (nan) | > loss: 0.008868368342518806 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13711004587947698) | > loader_time: 0.0045 (0.004651404047822301)  --> TIME: 2025-02-26 13:05:01 -- STEP: 32546/36938 -- GLOBAL_STEP: 1251500 | > loss_text_ce: 0.033028751611709595 (0.03135144865318136) | > loss_mel_ce: 0.7906703352928162 (nan) | > loss: 0.003268647473305464 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13711174755715233) | > loader_time: 0.0045 (0.0046513929925568066)  --> TIME: 2025-02-26 13:05:11 -- STEP: 32596/36938 -- GLOBAL_STEP: 1251550 | > loss_text_ce: 0.02985716238617897 (0.03134849382148022) | > loss_mel_ce: 3.0734803676605225 (nan) | > loss: 0.012314831838011742 (nan) | > current_lr: 5e-06 | > step_time: 0.2012 (0.13711195125771697) | > loader_time: 0.0049 (0.004651365543175405)  --> TIME: 2025-02-26 13:05:20 -- STEP: 32646/36938 -- GLOBAL_STEP: 1251600 | > loss_text_ce: 0.03381124511361122 (0.031349098034936684) | > loss_mel_ce: 2.5642967224121094 (nan) | > loss: 0.01030995324254036 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13711476548734725) | > loader_time: 0.0046 (0.004651471891217973)  --> TIME: 2025-02-26 13:05:30 -- STEP: 32696/36938 -- GLOBAL_STEP: 1251650 | > loss_text_ce: 0.02806921675801277 (0.031349808847316224) | > loss_mel_ce: 2.732323408126831 (nan) | > loss: 0.010953939519822598 (nan) | > current_lr: 5e-06 | > step_time: 0.2013 (0.13712486712221406) | > loader_time: 0.0049 (0.004651584075720508)  --> TIME: 2025-02-26 13:05:39 -- STEP: 32746/36938 -- GLOBAL_STEP: 1251700 | > loss_text_ce: 0.016046680510044098 (0.03134913067029084) | > loss_mel_ce: 2.1082284450531006 (nan) | > loss: 0.008429664187133312 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13712617490823467) | > loader_time: 0.0046 (0.004651536729218837)  --> TIME: 2025-02-26 13:05:48 -- STEP: 32796/36938 -- GLOBAL_STEP: 1251750 | > loss_text_ce: 0.013392009772360325 (0.0313476770646421) | > loss_mel_ce: 0.0873684361577034 (nan) | > loss: 0.00039984306204132736 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13712148933036042) | > loader_time: 0.0045 (0.004651453549112525)  --> TIME: 2025-02-26 13:05:58 -- STEP: 32846/36938 -- GLOBAL_STEP: 1251800 | > loss_text_ce: 0.023399056866765022 (0.03134671754630978) | > loss_mel_ce: 1.9319496154785156 (nan) | > loss: 0.007759321015328169 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13712382588789762) | > loader_time: 0.0045 (0.004651466175502771)  --> TIME: 2025-02-26 13:06:07 -- STEP: 32896/36938 -- GLOBAL_STEP: 1251850 | > loss_text_ce: 0.03233128786087036 (0.03134546369322517) | > loss_mel_ce: 2.5695033073425293 (nan) | > loss: 0.010324740782380104 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1371233374141295) | > loader_time: 0.0047 (0.004651516973102586)  --> TIME: 2025-02-26 13:06:17 -- STEP: 32946/36938 -- GLOBAL_STEP: 1251900 | > loss_text_ce: 0.03052908182144165 (0.03134403215595569) | > loss_mel_ce: 3.0138964653015137 (nan) | > loss: 0.012081054039299488 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13713069622947724) | > loader_time: 0.0046 (0.004651699106397851)  --> TIME: 2025-02-26 13:06:26 -- STEP: 32996/36938 -- GLOBAL_STEP: 1251950 | > loss_text_ce: 0.03394046798348427 (0.031342621402289876) | > loss_mel_ce: 2.4302704334259033 (nan) | > loss: 0.009778616018593311 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13712194290633914) | > loader_time: 0.0047 (0.004651753790292712)  --> TIME: 2025-02-26 13:06:35 -- STEP: 33046/36938 -- GLOBAL_STEP: 1252000 | > loss_text_ce: 0.04092812538146973 (0.03134144462411644) | > loss_mel_ce: 2.7058517932891846 (nan) | > loss: 0.01089992094784975 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13712798047713554) | > loader_time: 0.0045 (0.0046518840779982695)  --> TIME: 2025-02-26 13:06:45 -- STEP: 33096/36938 -- GLOBAL_STEP: 1252050 | > loss_text_ce: 0.02179827354848385 (0.03134074509005098) | > loss_mel_ce: 2.037417411804199 (nan) | > loss: 0.00817149132490158 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13712218429750012) | > loader_time: 0.0045 (0.004651950779876341)  --> TIME: 2025-02-26 13:06:54 -- STEP: 33146/36938 -- GLOBAL_STEP: 1252100 | > loss_text_ce: 0.034840311855077744 (0.03134074003515692) | > loss_mel_ce: 2.585261583328247 (nan) | > loss: 0.0103972302749753 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13711567533718894) | > loader_time: 0.0045 (0.004651858265266269)  --> TIME: 2025-02-26 13:07:03 -- STEP: 33196/36938 -- GLOBAL_STEP: 1252150 | > loss_text_ce: 0.03251495584845543 (0.03134126546255987) | > loss_mel_ce: 2.7632243633270264 (nan) | > loss: 0.01109420508146286 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13711295589393405) | > loader_time: 0.0046 (0.004651773728610704)  --> TIME: 2025-02-26 13:07:12 -- STEP: 33246/36938 -- GLOBAL_STEP: 1252200 | > loss_text_ce: 0.027250949293375015 (0.031341665725617064) | > loss_mel_ce: 3.1090478897094727 (nan) | > loss: 0.012445631437003613 (nan) | > current_lr: 5e-06 | > step_time: 0.1778 (0.13711096344316498) | > loader_time: 0.0049 (0.004651767685619447)  --> TIME: 2025-02-26 13:07:21 -- STEP: 33296/36938 -- GLOBAL_STEP: 1252250 | > loss_text_ce: 0.029999082908034325 (0.03134240257496526) | > loss_mel_ce: 2.591029644012451 (nan) | > loss: 0.010400908067822456 (nan) | > current_lr: 5e-06 | > step_time: 0.1908 (0.13710740734212093) | > loader_time: 0.0049 (0.004651844966932863)  --> TIME: 2025-02-26 13:07:30 -- STEP: 33346/36938 -- GLOBAL_STEP: 1252300 | > loss_text_ce: 0.03639429807662964 (0.03134290827548553) | > loss_mel_ce: 3.056535243988037 (nan) | > loss: 0.012273530475795269 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.13709913598949808) | > loader_time: 0.0048 (0.00465181260248225)  --> TIME: 2025-02-26 13:07:40 -- STEP: 33396/36938 -- GLOBAL_STEP: 1252350 | > loss_text_ce: 0.022899853065609932 (0.03134186370262289) | > loss_mel_ce: 2.484056234359741 (nan) | > loss: 0.009948238730430603 (nan) | > current_lr: 5e-06 | > step_time: 0.1911 (0.13711128358084698) | > loader_time: 0.0049 (0.004651862385030518)  --> TIME: 2025-02-26 13:07:49 -- STEP: 33446/36938 -- GLOBAL_STEP: 1252400 | > loss_text_ce: 0.04085562005639076 (0.03134206279411169) | > loss_mel_ce: 1.5759201049804688 (nan) | > loss: 0.006415777374058962 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13710908839728628) | > loader_time: 0.0048 (0.004651909395459791)  --> TIME: 2025-02-26 13:07:59 -- STEP: 33496/36938 -- GLOBAL_STEP: 1252450 | > loss_text_ce: 0.02286948822438717 (0.03134162134232829) | > loss_mel_ce: 2.2797553539276123 (nan) | > loss: 0.009137401357293129 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.13710653660176036) | > loader_time: 0.0046 (0.004651867228702269)  --> TIME: 2025-02-26 13:08:08 -- STEP: 33546/36938 -- GLOBAL_STEP: 1252500 | > loss_text_ce: 0.02421111799776554 (0.03134058191011679) | > loss_mel_ce: 2.4685699939727783 (nan) | > loss: 0.009891989640891552 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13711346690417162) | > loader_time: 0.0047 (0.00465194480915099)  --> TIME: 2025-02-26 13:08:17 -- STEP: 33596/36938 -- GLOBAL_STEP: 1252550 | > loss_text_ce: 0.030776845291256905 (0.03134010047943178) | > loss_mel_ce: 2.9624509811401367 (nan) | > loss: 0.011877888813614845 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1371084179306523) | > loader_time: 0.0045 (0.004651897378529307)  --> TIME: 2025-02-26 13:08:27 -- STEP: 33646/36938 -- GLOBAL_STEP: 1252600 | > loss_text_ce: 0.03222540393471718 (0.031339801358648514) | > loss_mel_ce: 0.03009033203125 (nan) | > loss: 0.0002472846827004105 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.13710311851836993) | > loader_time: 0.0046 (0.00465184087696039)  --> TIME: 2025-02-26 13:08:36 -- STEP: 33696/36938 -- GLOBAL_STEP: 1252650 | > loss_text_ce: 0.031277094036340714 (0.03134130881640186) | > loss_mel_ce: 3.2035129070281982 (nan) | > loss: 0.01283646933734417 (nan) | > current_lr: 5e-06 | > step_time: 0.1921 (0.1371058638703782) | > loader_time: 0.0049 (0.004651815951549756)  --> TIME: 2025-02-26 13:08:45 -- STEP: 33746/36938 -- GLOBAL_STEP: 1252700 | > loss_text_ce: 0.033589113503694534 (0.03134027999783384) | > loss_mel_ce: 2.522190809249878 (nan) | > loss: 0.010141984559595585 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13710858780467386) | > loader_time: 0.0046 (0.004651797006417472)  --> TIME: 2025-02-26 13:08:55 -- STEP: 33796/36938 -- GLOBAL_STEP: 1252750 | > loss_text_ce: 0.037160761654376984 (0.03134153296811706) | > loss_mel_ce: 3.0899758338928223 (nan) | > loss: 0.012409273535013199 (nan) | > current_lr: 5e-06 | > step_time: 0.1767 (0.13711704291003754) | > loader_time: 0.0048 (0.004651917185440799)  --> TIME: 2025-02-26 13:09:05 -- STEP: 33846/36938 -- GLOBAL_STEP: 1252800 | > loss_text_ce: 0.03422772139310837 (0.03134079606030465) | > loss_mel_ce: 3.52569580078125 (nan) | > loss: 0.014126680791378021 (nan) | > current_lr: 5e-06 | > step_time: 0.1819 (0.1371161787883114) | > loader_time: 0.0048 (0.004651854719132556)  --> TIME: 2025-02-26 13:09:14 -- STEP: 33896/36938 -- GLOBAL_STEP: 1252850 | > loss_text_ce: 0.03475202992558479 (0.031341020584758496) | > loss_mel_ce: 2.0384185314178467 (nan) | > loss: 0.008226867765188217 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13710978377503982) | > loader_time: 0.0046 (0.004651793351509885)  --> TIME: 2025-02-26 13:09:23 -- STEP: 33946/36938 -- GLOBAL_STEP: 1252900 | > loss_text_ce: 0.03657068684697151 (0.03134044064678285) | > loss_mel_ce: 2.6503348350524902 (nan) | > loss: 0.010662324726581573 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13711568337608246) | > loader_time: 0.0047 (0.004651776222869979)  --> TIME: 2025-02-26 13:09:32 -- STEP: 33996/36938 -- GLOBAL_STEP: 1252950 | > loss_text_ce: 0.019272658973932266 (0.0313411235633258) | > loss_mel_ce: 2.1292083263397217 (nan) | > loss: 0.008525718003511429 (nan) | > current_lr: 5e-06 | > step_time: 0.1076 (0.1371069898085245) | > loader_time: 0.0045 (0.004651718433358569)  --> TIME: 2025-02-26 13:09:41 -- STEP: 34046/36938 -- GLOBAL_STEP: 1253000 | > loss_text_ce: 0.030899643898010254 (0.0313385999075326) | > loss_mel_ce: 2.305758476257324 (nan) | > loss: 0.009272453375160694 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13710003448752472) | > loader_time: 0.0046 (0.0046517118502607075)  --> TIME: 2025-02-26 13:09:51 -- STEP: 34096/36938 -- GLOBAL_STEP: 1253050 | > loss_text_ce: 0.0318877138197422 (0.0313385959072018) | > loss_mel_ce: 2.1168532371520996 (nan) | > loss: 0.008526750840246677 (nan) | > current_lr: 5e-06 | > step_time: 0.1268 (0.1370978944811217) | > loader_time: 0.0046 (0.0046516330112934216)  --> TIME: 2025-02-26 13:10:00 -- STEP: 34146/36938 -- GLOBAL_STEP: 1253100 | > loss_text_ce: 0.031015167012810707 (0.031338952615024625) | > loss_mel_ce: 3.076112747192383 (nan) | > loss: 0.012329873628914356 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13710148519674445) | > loader_time: 0.0046 (0.004651603984726621)  --> TIME: 2025-02-26 13:10:09 -- STEP: 34196/36938 -- GLOBAL_STEP: 1253150 | > loss_text_ce: 0.02674725279211998 (0.03133817542396243) | > loss_mel_ce: 3.0322265625 (nan) | > loss: 0.012138785794377327 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13709711573921185) | > loader_time: 0.0044 (0.0046515229961748995)  --> TIME: 2025-02-26 13:10:19 -- STEP: 34246/36938 -- GLOBAL_STEP: 1253200 | > loss_text_ce: 0.0337604358792305 (0.03134005695348048) | > loss_mel_ce: 2.564168691635132 (nan) | > loss: 0.010309243574738503 (nan) | > current_lr: 5e-06 | > step_time: 0.1077 (0.13709680914203462) | > loader_time: 0.0044 (0.004651555055380901)  --> TIME: 2025-02-26 13:10:28 -- STEP: 34296/36938 -- GLOBAL_STEP: 1253250 | > loss_text_ce: 0.025126716122031212 (0.031337969786383126) | > loss_mel_ce: 2.8953261375427246 (nan) | > loss: 0.011589099653065205 (nan) | > current_lr: 5e-06 | > step_time: 0.1864 (0.13709057264459176) | > loader_time: 0.0055 (0.0046516001808468025)  --> TIME: 2025-02-26 13:10:37 -- STEP: 34346/36938 -- GLOBAL_STEP: 1253300 | > loss_text_ce: 0.02224661223590374 (0.031337785181145654) | > loss_mel_ce: 0.34614259004592896 (nan) | > loss: 0.0014618620043620467 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13708088223466092) | > loader_time: 0.0044 (0.0046515102427569155)  --> TIME: 2025-02-26 13:10:46 -- STEP: 34396/36938 -- GLOBAL_STEP: 1253350 | > loss_text_ce: 0.030252952128648758 (0.031337221257392606) | > loss_mel_ce: 2.9750349521636963 (nan) | > loss: 0.011925746686756611 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13708655255965632) | > loader_time: 0.0046 (0.004651508063462614)  --> TIME: 2025-02-26 13:10:55 -- STEP: 34446/36938 -- GLOBAL_STEP: 1253400 | > loss_text_ce: 0.033484335988759995 (0.03133674842812809) | > loss_mel_ce: 1.7678980827331543 (nan) | > loss: 0.00714834313839674 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.1370853152153536) | > loader_time: 0.0045 (0.004651450490671893)  --> TIME: 2025-02-26 13:11:05 -- STEP: 34496/36938 -- GLOBAL_STEP: 1253450 | > loss_text_ce: 0.025650281459093094 (0.031335525829645806) | > loss_mel_ce: 1.9389992952346802 (nan) | > loss: 0.007796228863298893 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13708283630560783) | > loader_time: 0.0046 (0.004651447955055876)  --> TIME: 2025-02-26 13:11:14 -- STEP: 34546/36938 -- GLOBAL_STEP: 1253500 | > loss_text_ce: 0.026664985343813896 (0.031335049671878716) | > loss_mel_ce: 2.6067075729370117 (nan) | > loss: 0.010449891909956932 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.13707939235984448) | > loader_time: 0.0047 (0.004651426364887843)  --> TIME: 2025-02-26 13:11:23 -- STEP: 34596/36938 -- GLOBAL_STEP: 1253550 | > loss_text_ce: 0.028546515852212906 (0.0313351614700122) | > loss_mel_ce: 3.082505464553833 (nan) | > loss: 0.012345445342361927 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.13708053027043893) | > loader_time: 0.0048 (0.00465138858695265)  --> TIME: 2025-02-26 13:11:33 -- STEP: 34646/36938 -- GLOBAL_STEP: 1253600 | > loss_text_ce: 0.03441765531897545 (0.031334563047070774) | > loss_mel_ce: 2.6243882179260254 (nan) | > loss: 0.010550817474722862 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13708483222477338) | > loader_time: 0.0046 (0.0046513829104366045)  --> TIME: 2025-02-26 13:11:42 -- STEP: 34696/36938 -- GLOBAL_STEP: 1253650 | > loss_text_ce: 0.028671640902757645 (0.0313348013776064) | > loss_mel_ce: 2.8360958099365234 (nan) | > loss: 0.011368125677108765 (nan) | > current_lr: 5e-06 | > step_time: 0.193 (0.13708243147139934) | > loader_time: 0.0049 (0.004651472862343467)  --> TIME: 2025-02-26 13:11:52 -- STEP: 34746/36938 -- GLOBAL_STEP: 1253700 | > loss_text_ce: 0.03838246315717697 (0.03133501108050548) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13708239357712787) | > loader_time: 0.0046 (0.004651501602386264)  --> TIME: 2025-02-26 13:12:01 -- STEP: 34796/36938 -- GLOBAL_STEP: 1253750 | > loss_text_ce: 0.023028960451483727 (0.03133444582060062) | > loss_mel_ce: 2.9096131324768066 (nan) | > loss: 0.011637468822300434 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.1370834560623955) | > loader_time: 0.0046 (0.0046514466598480755)  --> TIME: 2025-02-26 13:12:10 -- STEP: 34846/36938 -- GLOBAL_STEP: 1253800 | > loss_text_ce: 0.03585486486554146 (0.03133338673951081) | > loss_mel_ce: 0.9701085090637207 (nan) | > loss: 0.003991918172687292 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13707876223233628) | > loader_time: 0.0045 (0.004651370411426037)  --> TIME: 2025-02-26 13:12:19 -- STEP: 34896/36938 -- GLOBAL_STEP: 1253850 | > loss_text_ce: 0.032721687108278275 (0.03133441498866335) | > loss_mel_ce: 3.1952602863311768 (nan) | > loss: 0.012809453532099724 (nan) | > current_lr: 5e-06 | > step_time: 0.1797 (0.13708129504004538) | > loader_time: 0.0048 (0.004651429530460306)  --> TIME: 2025-02-26 13:12:29 -- STEP: 34946/36938 -- GLOBAL_STEP: 1253900 | > loss_text_ce: 0.032876282930374146 (0.03133259869443621) | > loss_mel_ce: 2.8765335083007812 (nan) | > loss: 0.011545277200639248 (nan) | > current_lr: 5e-06 | > step_time: 0.1812 (0.1370774592121395) | > loader_time: 0.0048 (0.004651392126358991)  --> TIME: 2025-02-26 13:12:38 -- STEP: 34996/36938 -- GLOBAL_STEP: 1253950 | > loss_text_ce: 0.028476785868406296 (0.03133110507953301) | > loss_mel_ce: 3.0082342624664307 (nan) | > loss: 0.012050441466271877 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.1370705427040513) | > loader_time: 0.0046 (0.004651325050660949)  --> TIME: 2025-02-26 13:12:47 -- STEP: 35046/36938 -- GLOBAL_STEP: 1254000 | > loss_text_ce: 0.03536633029580116 (0.03133285094019565) | > loss_mel_ce: 0.13737136125564575 (nan) | > loss: 0.0006854670355096459 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.1370642106908433) | > loader_time: 0.0046 (0.004651258799036911)  --> TIME: 2025-02-26 13:12:56 -- STEP: 35096/36938 -- GLOBAL_STEP: 1254050 | > loss_text_ce: 0.04014710709452629 (0.031334668144892004) | > loss_mel_ce: 0.2397257536649704 (nan) | > loss: 0.0011106067104265094 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13706773825894916) | > loader_time: 0.0045 (0.004651248142554588)  --> TIME: 2025-02-26 13:13:05 -- STEP: 35146/36938 -- GLOBAL_STEP: 1254100 | > loss_text_ce: 0.029940828680992126 (0.031333004696609) | > loss_mel_ce: 2.876924991607666 (nan) | > loss: 0.01153518259525299 (nan) | > current_lr: 5e-06 | > step_time: 0.156 (0.13705952063178178) | > loader_time: 0.0047 (0.004651183742305454)  --> TIME: 2025-02-26 13:13:15 -- STEP: 35196/36938 -- GLOBAL_STEP: 1254150 | > loss_text_ce: 0.03075787052512169 (0.03133559585731268) | > loss_mel_ce: 2.559373140335083 (nan) | > loss: 0.010278298519551754 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.13706298782305396) | > loader_time: 0.0048 (0.00465117287047817)  --> TIME: 2025-02-26 13:13:24 -- STEP: 35246/36938 -- GLOBAL_STEP: 1254200 | > loss_text_ce: 0.025133898481726646 (0.031334438107205376) | > loss_mel_ce: 2.814567804336548 (nan) | > loss: 0.011268657632172108 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13706964388652815) | > loader_time: 0.0045 (0.0046512243229962745)  --> TIME: 2025-02-26 13:13:34 -- STEP: 35296/36938 -- GLOBAL_STEP: 1254250 | > loss_text_ce: 0.02783074602484703 (0.03133324931883838) | > loss_mel_ce: 2.4092493057250977 (nan) | > loss: 0.009670953266322613 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13706818817424757) | > loader_time: 0.0046 (0.0046512125666208414)  --> TIME: 2025-02-26 13:13:44 -- STEP: 35346/36938 -- GLOBAL_STEP: 1254300 | > loss_text_ce: 0.03010793775320053 (0.031333408555559245) | > loss_mel_ce: 1.6915109157562256 (nan) | > loss: 0.006831821519881487 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13708086437532174) | > loader_time: 0.0067 (0.004651358365972091)  --> TIME: 2025-02-26 13:13:53 -- STEP: 35396/36938 -- GLOBAL_STEP: 1254350 | > loss_text_ce: 0.025299804285168648 (0.03133308300062537) | > loss_mel_ce: 0.05950927734375 (nan) | > loss: 0.00033654397702775896 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.1370792026330625) | > loader_time: 0.0047 (0.0046513733291022345)  --> TIME: 2025-02-26 13:14:02 -- STEP: 35446/36938 -- GLOBAL_STEP: 1254400 | > loss_text_ce: 0.02913696877658367 (0.03133415057552366) | > loss_mel_ce: 2.7183837890625 (nan) | > loss: 0.010902860201895237 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13707728409601577) | > loader_time: 0.0046 (0.004651369981529512)  --> TIME: 2025-02-26 13:14:12 -- STEP: 35496/36938 -- GLOBAL_STEP: 1254450 | > loss_text_ce: 0.028188249096274376 (0.03133316046162097) | > loss_mel_ce: 2.9873945713043213 (nan) | > loss: 0.011966599151492119 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.13708033967668196) | > loader_time: 0.0047 (0.004651359429573288)  --> TIME: 2025-02-26 13:14:21 -- STEP: 35546/36938 -- GLOBAL_STEP: 1254500 | > loss_text_ce: 0.026393476873636246 (0.03133162094570386) | > loss_mel_ce: 2.879242181777954 (nan) | > loss: 0.011530300602316856 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13707576510754924) | > loader_time: 0.0045 (0.004651254635847706)  --> TIME: 2025-02-26 13:14:30 -- STEP: 35596/36938 -- GLOBAL_STEP: 1254550 | > loss_text_ce: 0.022297367453575134 (0.031332476423855775) | > loss_mel_ce: 2.7231078147888184 (nan) | > loss: 0.010894465260207653 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13708103519568834) | > loader_time: 0.0046 (0.004651254576931555)  --> TIME: 2025-02-26 13:14:39 -- STEP: 35646/36938 -- GLOBAL_STEP: 1254600 | > loss_text_ce: 0.026016904041171074 (0.0313314690324729) | > loss_mel_ce: 2.9686310291290283 (nan) | > loss: 0.011883524246513844 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.1370716734659659) | > loader_time: 0.0045 (0.004651195538911465)  --> TIME: 2025-02-26 13:14:49 -- STEP: 35696/36938 -- GLOBAL_STEP: 1254650 | > loss_text_ce: 0.036299191415309906 (0.03133014113603315) | > loss_mel_ce: 3.328540325164795 (nan) | > loss: 0.013352538459002972 (nan) | > current_lr: 5e-06 | > step_time: 0.1809 (0.13707560021638876) | > loader_time: 0.0049 (0.004651192236726772)  --> TIME: 2025-02-26 13:14:59 -- STEP: 35746/36938 -- GLOBAL_STEP: 1254700 | > loss_text_ce: 0.03887598589062691 (0.0313328405341457) | > loss_mel_ce: 3.200254201889038 (nan) | > loss: 0.012853692285716534 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13708023850748172) | > loader_time: 0.0045 (0.0046512848221181135)  --> TIME: 2025-02-26 13:15:08 -- STEP: 35796/36938 -- GLOBAL_STEP: 1254750 | > loss_text_ce: 0.03996633365750313 (0.03133126446583736) | > loss_mel_ce: 2.922438383102417 (nan) | > loss: 0.01175557542592287 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13707755963780124) | > loader_time: 0.0046 (0.004651235660263532)  --> TIME: 2025-02-26 13:15:18 -- STEP: 35846/36938 -- GLOBAL_STEP: 1254800 | > loss_text_ce: 0.03531744331121445 (0.03133304483028203) | > loss_mel_ce: 3.2663540840148926 (nan) | > loss: 0.013101872056722641 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.1370889942194932) | > loader_time: 0.0046 (0.004651209143185414)  --> TIME: 2025-02-26 13:15:27 -- STEP: 35896/36938 -- GLOBAL_STEP: 1254850 | > loss_text_ce: 0.031139668077230453 (0.03133242466613465) | > loss_mel_ce: 3.1544370651245117 (nan) | > loss: 0.0126411784440279 (nan) | > current_lr: 5e-06 | > step_time: 0.1576 (0.13709170782595193) | > loader_time: 0.0046 (0.004651306605110558)  --> TIME: 2025-02-26 13:15:37 -- STEP: 35946/36938 -- GLOBAL_STEP: 1254900 | > loss_text_ce: 0.036204222589731216 (0.031330891535624086) | > loss_mel_ce: 2.237123966217041 (nan) | > loss: 0.009021144360303879 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13709686353370623) | > loader_time: 0.0045 (0.0046512190025994665)  --> TIME: 2025-02-26 13:15:46 -- STEP: 35996/36938 -- GLOBAL_STEP: 1254950 | > loss_text_ce: 0.026431849226355553 (0.03133190700795781) | > loss_mel_ce: 2.262600898742676 (nan) | > loss: 0.009083463810384274 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13709556107945384) | > loader_time: 0.0045 (0.004651186903101391)  --> TIME: 2025-02-26 13:15:56 -- STEP: 36046/36938 -- GLOBAL_STEP: 1255000 | > loss_text_ce: 0.02510959282517433 (0.03133034673154117) | > loss_mel_ce: 2.529773235321045 (nan) | > loss: 0.01013842411339283 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.13710042982937914) | > loader_time: 0.0047 (0.004651152789311625)  --> TIME: 2025-02-26 13:16:05 -- STEP: 36096/36938 -- GLOBAL_STEP: 1255050 | > loss_text_ce: 0.029859555885195732 (0.03132979604005119) | > loss_mel_ce: 2.881248712539673 (nan) | > loss: 0.01155201718211174 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13710085023194452) | > loader_time: 0.0048 (0.004651165534312832)  --> TIME: 2025-02-26 13:16:14 -- STEP: 36146/36938 -- GLOBAL_STEP: 1255100 | > loss_text_ce: 0.03446440398693085 (0.031330291233759244) | > loss_mel_ce: 2.7302610874176025 (nan) | > loss: 0.010971133597195148 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13710038470541122) | > loader_time: 0.0045 (0.00465111856355562)  --> TIME: 2025-02-26 13:16:23 -- STEP: 36196/36938 -- GLOBAL_STEP: 1255150 | > loss_text_ce: 0.02851278707385063 (0.03133007828089893) | > loss_mel_ce: 2.0289459228515625 (nan) | > loss: 0.008164518512785435 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.1370920266996015) | > loader_time: 0.0046 (0.004651106033605521)  --> TIME: 2025-02-26 13:16:33 -- STEP: 36246/36938 -- GLOBAL_STEP: 1255200 | > loss_text_ce: 0.03345341235399246 (0.031329867045706165) | > loss_mel_ce: 2.9456756114959717 (nan) | > loss: 0.01182194147258997 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.1370939037325305) | > loader_time: 0.0045 (0.004651051354856858)  --> TIME: 2025-02-26 13:16:42 -- STEP: 36296/36938 -- GLOBAL_STEP: 1255250 | > loss_text_ce: 0.029600653797388077 (0.031332083164994404) | > loss_mel_ce: 2.492551803588867 (nan) | > loss: 0.010008541867136955 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13709906337411934) | > loader_time: 0.0047 (0.004651054296562882)  --> TIME: 2025-02-26 13:16:52 -- STEP: 36346/36938 -- GLOBAL_STEP: 1255300 | > loss_text_ce: 0.029258867725729942 (0.03133107651141564) | > loss_mel_ce: 0.011474609375 (nan) | > loss: 0.00016164079715963453 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13710180113719972) | > loader_time: 0.0045 (0.00465108856582577)  --> TIME: 2025-02-26 13:17:01 -- STEP: 36396/36938 -- GLOBAL_STEP: 1255350 | > loss_text_ce: 0.02553817257285118 (0.03133087783801607) | > loss_mel_ce: 2.2702670097351074 (nan) | > loss: 0.009110338985919952 (nan) | > current_lr: 5e-06 | > step_time: 0.1789 (0.13710230364879597) | > loader_time: 0.0049 (0.004651208764472507)  --> TIME: 2025-02-26 13:17:10 -- STEP: 36446/36938 -- GLOBAL_STEP: 1255400 | > loss_text_ce: 0.026671290397644043 (0.031329812824582505) | > loss_mel_ce: 3.5161609649658203 (nan) | > loss: 0.014058859087526798 (nan) | > current_lr: 5e-06 | > step_time: 0.1797 (0.137103881413605) | > loader_time: 0.0048 (0.004651281709748985)  --> TIME: 2025-02-26 13:17:20 -- STEP: 36496/36938 -- GLOBAL_STEP: 1255450 | > loss_text_ce: 0.03763069584965706 (0.03133005261553564) | > loss_mel_ce: 2.7420284748077393 (nan) | > loss: 0.011030394583940506 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13710522324382432) | > loader_time: 0.0046 (0.004651271456796197)  --> TIME: 2025-02-26 13:17:29 -- STEP: 36546/36938 -- GLOBAL_STEP: 1255500 | > loss_text_ce: 0.03537771850824356 (0.03133057903036721) | > loss_mel_ce: 1.7118661403656006 (nan) | > loss: 0.006933507975190878 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13709832879179495) | > loader_time: 0.0047 (0.004651198427332555)  --> TIME: 2025-02-26 13:17:38 -- STEP: 36596/36938 -- GLOBAL_STEP: 1255550 | > loss_text_ce: 0.037896446883678436 (0.031330288858777124) | > loss_mel_ce: 3.36688494682312 (nan) | > loss: 0.013511037454009056 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13710351632792894) | > loader_time: 0.0045 (0.004651132164424659)  --> TIME: 2025-02-26 13:17:48 -- STEP: 36646/36938 -- GLOBAL_STEP: 1255600 | > loss_text_ce: 0.0318048819899559 (0.03133087187262311) | > loss_mel_ce: 2.9672558307647705 (nan) | > loss: 0.011901034973561764 (nan) | > current_lr: 5e-06 | > step_time: 0.1576 (0.1371047086788039) | > loader_time: 0.0046 (0.0046510589127326295)  --> TIME: 2025-02-26 13:17:57 -- STEP: 36696/36938 -- GLOBAL_STEP: 1255650 | > loss_text_ce: 0.022254349663853645 (0.03133288007348108) | > loss_mel_ce: 1.988716959953308 (nan) | > loss: 0.007980044931173325 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13710295802527572) | > loader_time: 0.0045 (0.004651090535874288)  --> TIME: 2025-02-26 13:18:06 -- STEP: 36746/36938 -- GLOBAL_STEP: 1255700 | > loss_text_ce: 0.03543335944414139 (0.03133375385311306) | > loss_mel_ce: 2.801865577697754 (nan) | > loss: 0.011259122751653194 (nan) | > current_lr: 5e-06 | > step_time: 0.1606 (0.13710407903079777) | > loader_time: 0.0046 (0.004651065456168412)  --> TIME: 2025-02-26 13:18:16 -- STEP: 36796/36938 -- GLOBAL_STEP: 1255750 | > loss_text_ce: 0.027947580441832542 (0.031332897789766286) | > loss_mel_ce: 2.1041910648345947 (nan) | > loss: 0.008460868149995804 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13710524927910123) | > loader_time: 0.0046 (0.004651045543964234)  --> TIME: 2025-02-26 13:18:25 -- STEP: 36846/36938 -- GLOBAL_STEP: 1255800 | > loss_text_ce: 0.02827322669327259 (0.03133196027421758) | > loss_mel_ce: 1.9874045848846436 (nan) | > loss: 0.007998722605407238 (nan) | > current_lr: 5e-06 | > step_time: 0.1443 (0.13710322517503043) | > loader_time: 0.0047 (0.004650990440022016)  --> TIME: 2025-02-26 13:18:35 -- STEP: 36896/36938 -- GLOBAL_STEP: 1255850 | > loss_text_ce: 0.03600245714187622 (0.031332413442648636) | > loss_mel_ce: 2.881465196609497 (nan) | > loss: 0.011577253229916096 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.1371142466496816) | > loader_time: 0.0054 (0.0046510742419921735)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.020984176361914908 (+0.00037125549694098925) | > avg_loss_text_ce: 0.03056827239175834 (-0.00016059890610746821) | > avg_loss_mel_ce: 2.4626103940576614 (-0.004208693114836404) | > avg_loss: 2.4931786615069553 (-0.004369294289316539) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_1255892.pth  > EPOCH: 35/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-26 13:19:08)   --> TIME: 2025-02-26 13:19:10 -- STEP: 8/36938 -- GLOBAL_STEP: 1255900 | > loss_text_ce: 0.030756212770938873 (0.032245887676253915) | > loss_mel_ce: 2.5747597217559814 (2.7402754426002502) | > loss: 0.010339349508285522 (0.01100206933915615) | > current_lr: 5e-06 | > step_time: 0.1187 (0.15189483761787415) | > loader_time: 0.0045 (0.005723178386688232)  --> TIME: 2025-02-26 13:19:19 -- STEP: 58/36938 -- GLOBAL_STEP: 1255950 | > loss_text_ce: 0.029552167281508446 (0.031000338045173673) | > loss_mel_ce: 3.0706379413604736 (2.384225793696683) | > loss: 0.012302342802286148 (0.009584231258988187) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13712569351853987) | > loader_time: 0.0047 (0.005136699512087065)  --> TIME: 2025-02-26 13:19:28 -- STEP: 108/36938 -- GLOBAL_STEP: 1256000 | > loss_text_ce: 0.02928883023560047 (0.03096970453582428) | > loss_mel_ce: 2.230757236480713 (2.5002186429445397) | > loss: 0.00896843709051609 (0.01004439882591315) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13561812153568964) | > loader_time: 0.0045 (0.005008741661354345)  --> TIME: 2025-02-26 13:19:38 -- STEP: 158/36938 -- GLOBAL_STEP: 1256050 | > loss_text_ce: 0.0314447246491909 (0.03081804994918123) | > loss_mel_ce: 0.08822937309741974 (nan) | > loss: 0.0004748972423840314 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13633001756064492) | > loader_time: 0.0045 (0.004915812347508684)  --> TIME: 2025-02-26 13:19:47 -- STEP: 208/36938 -- GLOBAL_STEP: 1256100 | > loss_text_ce: 0.028692757710814476 (0.031113648822960947) | > loss_mel_ce: 3.014564037322998 (nan) | > loss: 0.012076416052877903 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.1362304286314891) | > loader_time: 0.0047 (0.004867677505199726)  --> TIME: 2025-02-26 13:19:56 -- STEP: 258/36938 -- GLOBAL_STEP: 1256150 | > loss_text_ce: 0.021855227649211884 (0.031015797806975914) | > loss_mel_ce: 2.727918863296509 (nan) | > loss: 0.01091180182993412 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13595945705739096) | > loader_time: 0.0046 (0.004850438398908275)  --> TIME: 2025-02-26 13:20:05 -- STEP: 308/36938 -- GLOBAL_STEP: 1256200 | > loss_text_ce: 0.03412866219878197 (0.03106591563260594) | > loss_mel_ce: 1.7027451992034912 (nan) | > loss: 0.006892357021570206 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1357942355143558) | > loader_time: 0.0046 (0.0048448396967603)  --> TIME: 2025-02-26 13:20:14 -- STEP: 358/36938 -- GLOBAL_STEP: 1256250 | > loss_text_ce: 0.03821877762675285 (0.031184862024416136) | > loss_mel_ce: 2.6497387886047363 (nan) | > loss: 0.010666498914361 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13533002904007532) | > loader_time: 0.0046 (0.004849511151873195)  --> TIME: 2025-02-26 13:20:24 -- STEP: 408/36938 -- GLOBAL_STEP: 1256300 | > loss_text_ce: 0.036275383085012436 (0.031091215180269645) | > loss_mel_ce: 2.7784664630889893 (nan) | > loss: 0.011169611476361752 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.1363313916851491) | > loader_time: 0.0049 (0.0048389744524862215)  --> TIME: 2025-02-26 13:20:34 -- STEP: 458/36938 -- GLOBAL_STEP: 1256350 | > loss_text_ce: 0.032046690583229065 (0.03106292104551886) | > loss_mel_ce: 3.008437395095825 (nan) | > loss: 0.01206541433930397 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13687930721382885) | > loader_time: 0.0047 (0.004842950787606716)  --> TIME: 2025-02-26 13:20:43 -- STEP: 508/36938 -- GLOBAL_STEP: 1256400 | > loss_text_ce: 0.040012702345848083 (0.031087143876494503) | > loss_mel_ce: 2.0881662368774414 (nan) | > loss: 0.008445154875516891 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13703515820615858) | > loader_time: 0.0045 (0.004823636351607914)  --> TIME: 2025-02-26 13:20:52 -- STEP: 558/36938 -- GLOBAL_STEP: 1256450 | > loss_text_ce: 0.027704784646630287 (0.03121397350363994) | > loss_mel_ce: 2.9405882358551025 (nan) | > loss: 0.011778941377997398 (nan) | > current_lr: 5e-06 | > step_time: 0.1901 (0.13697304905101798) | > loader_time: 0.005 (0.004808249866663341)  --> TIME: 2025-02-26 13:21:02 -- STEP: 608/36938 -- GLOBAL_STEP: 1256500 | > loss_text_ce: 0.033492207527160645 (0.031201901651097853) | > loss_mel_ce: 2.764397621154785 (nan) | > loss: 0.011102737858891487 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.1368371609010194) | > loader_time: 0.0046 (0.004803742233075592)  --> TIME: 2025-02-26 13:21:11 -- STEP: 658/36938 -- GLOBAL_STEP: 1256550 | > loss_text_ce: 0.03443518280982971 (0.031188696697863855) | > loss_mel_ce: 2.1848957538604736 (nan) | > loss: 0.008806869387626648 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.1368421149471249) | > loader_time: 0.0047 (0.004791442384111119)  --> TIME: 2025-02-26 13:21:20 -- STEP: 708/36938 -- GLOBAL_STEP: 1256600 | > loss_text_ce: 0.02603747323155403 (0.031170182018198787) | > loss_mel_ce: 2.8359053134918213 (nan) | > loss: 0.011356916278600693 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.13674661230906268) | > loader_time: 0.005 (0.004789840366880769)  --> TIME: 2025-02-26 13:21:30 -- STEP: 758/36938 -- GLOBAL_STEP: 1256650 | > loss_text_ce: 0.043215394020080566 (0.031199322815610976) | > loss_mel_ce: 1.6658775806427002 (nan) | > loss: 0.006782115437090397 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13692542368000626) | > loader_time: 0.0046 (0.004784550704553448)  --> TIME: 2025-02-26 13:21:39 -- STEP: 808/36938 -- GLOBAL_STEP: 1256700 | > loss_text_ce: 0.030843030661344528 (0.03121495211732085) | > loss_mel_ce: 2.795130729675293 (nan) | > loss: 0.011214181780815125 (nan) | > current_lr: 5e-06 | > step_time: 0.1777 (0.13696442441184942) | > loader_time: 0.0048 (0.004784610897007555)  --> TIME: 2025-02-26 13:21:49 -- STEP: 858/36938 -- GLOBAL_STEP: 1256750 | > loss_text_ce: 0.031228244304656982 (0.0312524591616001) | > loss_mel_ce: 0.0250244140625 (nan) | > loss: 0.00022322485165204853 (nan) | > current_lr: 5e-06 | > step_time: 0.1566 (0.13729252542927012) | > loader_time: 0.0047 (0.004790533950556686)  --> TIME: 2025-02-26 13:21:58 -- STEP: 908/36938 -- GLOBAL_STEP: 1256800 | > loss_text_ce: 0.026373684406280518 (0.03124450929032121) | > loss_mel_ce: 2.9658572673797607 (nan) | > loss: 0.011873932555317879 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13724330579656882) | > loader_time: 0.0045 (0.004795175554468768)  --> TIME: 2025-02-26 13:22:08 -- STEP: 958/36938 -- GLOBAL_STEP: 1256850 | > loss_text_ce: 0.03264518827199936 (0.03114942557009283) | > loss_mel_ce: 2.7460317611694336 (nan) | > loss: 0.01102649699896574 (nan) | > current_lr: 5e-06 | > step_time: 0.1583 (0.13724314071440263) | > loader_time: 0.0047 (0.004790533559556296)  --> TIME: 2025-02-26 13:22:17 -- STEP: 1008/36938 -- GLOBAL_STEP: 1256900 | > loss_text_ce: 0.022747112438082695 (0.03116402456477757) | > loss_mel_ce: 0.08723718672990799 (nan) | > loss: 0.00043644566903822124 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1372956340275115) | > loader_time: 0.0047 (0.004787180868406144)  --> TIME: 2025-02-26 13:22:27 -- STEP: 1058/36938 -- GLOBAL_STEP: 1256950 | > loss_text_ce: 0.03879443556070328 (0.031192524627991908) | > loss_mel_ce: 3.29561710357666 (nan) | > loss: 0.013231792487204075 (nan) | > current_lr: 5e-06 | > step_time: 0.1501 (0.13743234987745676) | > loader_time: 0.0047 (0.004787325633721451)  --> TIME: 2025-02-26 13:22:36 -- STEP: 1108/36938 -- GLOBAL_STEP: 1257000 | > loss_text_ce: 0.03244037553668022 (0.031207669580133383) | > loss_mel_ce: 2.9180660247802734 (nan) | > loss: 0.011708359234035015 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13741314970629326) | > loader_time: 0.0046 (0.0047883285942490784)  --> TIME: 2025-02-26 13:22:45 -- STEP: 1158/36938 -- GLOBAL_STEP: 1257050 | > loss_text_ce: 0.024559397250413895 (0.03128389392807336) | > loss_mel_ce: 2.3374922275543213 (nan) | > loss: 0.009373221546411514 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13716724746585543) | > loader_time: 0.0046 (0.004790164851977622)  --> TIME: 2025-02-26 13:22:54 -- STEP: 1208/36938 -- GLOBAL_STEP: 1257100 | > loss_text_ce: 0.0342009961605072 (0.03124193209747708) | > loss_mel_ce: 3.3265724182128906 (nan) | > loss: 0.013336403295397758 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.13702562707938937) | > loader_time: 0.0046 (0.004790816678116652)  --> TIME: 2025-02-26 13:23:04 -- STEP: 1258/36938 -- GLOBAL_STEP: 1257150 | > loss_text_ce: 0.029986117035150528 (0.03128629002500987) | > loss_mel_ce: 2.942131996154785 (nan) | > loss: 0.011794120073318481 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.1372060298161594) | > loader_time: 0.0047 (0.004789960971886856)  --> TIME: 2025-02-26 13:23:13 -- STEP: 1308/36938 -- GLOBAL_STEP: 1257200 | > loss_text_ce: 0.02734469063580036 (0.03129054618634812) | > loss_mel_ce: 3.0493199825286865 (nan) | > loss: 0.012208987958729267 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13735211551736287) | > loader_time: 0.0046 (0.004785635967137979)  --> TIME: 2025-02-26 13:23:23 -- STEP: 1358/36938 -- GLOBAL_STEP: 1257250 | > loss_text_ce: 0.02755914069712162 (0.03130021460480519) | > loss_mel_ce: 2.735504627227783 (nan) | > loss: 0.010964538902044296 (nan) | > current_lr: 5e-06 | > step_time: 0.1664 (0.1375534738110868) | > loader_time: 0.0046 (0.00478473239864973)  --> TIME: 2025-02-26 13:23:33 -- STEP: 1408/36938 -- GLOBAL_STEP: 1257300 | > loss_text_ce: 0.030239567160606384 (0.03132675684967337) | > loss_mel_ce: 3.187732458114624 (nan) | > loss: 0.012769730761647224 (nan) | > current_lr: 5e-06 | > step_time: 0.1478 (0.13763657669452142) | > loader_time: 0.0047 (0.004781776023182)  --> TIME: 2025-02-26 13:23:42 -- STEP: 1458/36938 -- GLOBAL_STEP: 1257350 | > loss_text_ce: 0.03520570322871208 (0.031311332806433875) | > loss_mel_ce: 3.1980886459350586 (nan) | > loss: 0.01283053308725357 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13752426433955722) | > loader_time: 0.0046 (0.004777618069380559)  --> TIME: 2025-02-26 13:23:51 -- STEP: 1508/36938 -- GLOBAL_STEP: 1257400 | > loss_text_ce: 0.04814471676945686 (0.03129739770653414) | > loss_mel_ce: 3.420719623565674 (nan) | > loss: 0.013765336014330387 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.13744893804469246) | > loader_time: 0.0047 (0.0047722438286090674)  --> TIME: 2025-02-26 13:24:00 -- STEP: 1558/36938 -- GLOBAL_STEP: 1257450 | > loss_text_ce: 0.03713760897517204 (0.03129538118095639) | > loss_mel_ce: 3.4396588802337646 (nan) | > loss: 0.013796811923384666 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13738610762839773) | > loader_time: 0.0046 (0.004767594043648442)  --> TIME: 2025-02-26 13:24:10 -- STEP: 1608/36938 -- GLOBAL_STEP: 1257500 | > loss_text_ce: 0.02960161678493023 (0.03131709689336978) | > loss_mel_ce: 3.0089380741119385 (nan) | > loss: 0.012057697400450706 (nan) | > current_lr: 5e-06 | > step_time: 0.1781 (0.13744930352144596) | > loader_time: 0.0048 (0.004763551612398515)  --> TIME: 2025-02-26 13:24:19 -- STEP: 1658/36938 -- GLOBAL_STEP: 1257550 | > loss_text_ce: 0.026179619133472443 (0.03129446950984893) | > loss_mel_ce: 2.556847333908081 (nan) | > loss: 0.010250107385218143 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13730926781137853) | > loader_time: 0.0045 (0.004762424778449375)  --> TIME: 2025-02-26 13:24:28 -- STEP: 1708/36938 -- GLOBAL_STEP: 1257600 | > loss_text_ce: 0.04236964136362076 (0.03127936417055891) | > loss_mel_ce: 1.952172875404358 (nan) | > loss: 0.007914851419627666 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13733202777366754) | > loader_time: 0.0044 (0.004759755290922567)  --> TIME: 2025-02-26 13:24:38 -- STEP: 1758/36938 -- GLOBAL_STEP: 1257650 | > loss_text_ce: 0.03785032778978348 (0.03125861164794826) | > loss_mel_ce: 2.4220404624938965 (nan) | > loss: 0.00976147223263979 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.13734711667647836) | > loader_time: 0.0047 (0.0047556087289925145)  --> TIME: 2025-02-26 13:24:47 -- STEP: 1808/36938 -- GLOBAL_STEP: 1257700 | > loss_text_ce: 0.02664487063884735 (0.031227477976075644) | > loss_mel_ce: 3.0084855556488037 (nan) | > loss: 0.012044169008731842 (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.13739133342704968) | > loader_time: 0.0049 (0.004752516087177577)  --> TIME: 2025-02-26 13:24:57 -- STEP: 1858/36938 -- GLOBAL_STEP: 1257750 | > loss_text_ce: 0.03346186503767967 (0.031199615547459526) | > loss_mel_ce: 2.7721333503723145 (nan) | > loss: 0.011133315041661263 (nan) | > current_lr: 5e-06 | > step_time: 0.1763 (0.13747412896643948) | > loader_time: 0.0048 (0.004750383682990096)  --> TIME: 2025-02-26 13:25:06 -- STEP: 1908/36938 -- GLOBAL_STEP: 1257800 | > loss_text_ce: 0.02934536151587963 (0.031188992996733268) | > loss_mel_ce: 0.9628391265869141 (nan) | > loss: 0.003937240224331617 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13741314423659085) | > loader_time: 0.0045 (0.004747955804101073)  --> TIME: 2025-02-26 13:25:15 -- STEP: 1958/36938 -- GLOBAL_STEP: 1257850 | > loss_text_ce: 0.033662181347608566 (0.03119317047722231) | > loss_mel_ce: 2.007730484008789 (nan) | > loss: 0.008100764825940132 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13750584266767824) | > loader_time: 0.0046 (0.004745315482594503)  --> TIME: 2025-02-26 13:25:25 -- STEP: 2008/36938 -- GLOBAL_STEP: 1257900 | > loss_text_ce: 0.037690144032239914 (0.031180588181123136) | > loss_mel_ce: 2.3687796592712402 (nan) | > loss: 0.00954948365688324 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.137456304998512) | > loader_time: 0.0046 (0.0047440433882147155)  --> TIME: 2025-02-26 13:25:34 -- STEP: 2058/36938 -- GLOBAL_STEP: 1257950 | > loss_text_ce: 0.031471479684114456 (0.03115268410848717) | > loss_mel_ce: 2.137115478515625 (nan) | > loss: 0.008605504408478737 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13737098839354836) | > loader_time: 0.0045 (0.004742945132612367)  --> TIME: 2025-02-26 13:25:43 -- STEP: 2108/36938 -- GLOBAL_STEP: 1258000 | > loss_text_ce: 0.03160381317138672 (0.031147220647368957) | > loss_mel_ce: 2.5179948806762695 (nan) | > loss: 0.010117455385625362 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.1373839413418491) | > loader_time: 0.0045 (0.004740383186195559)  --> TIME: 2025-02-26 13:25:52 -- STEP: 2158/36938 -- GLOBAL_STEP: 1258050 | > loss_text_ce: 0.029172217473387718 (0.03114607396420776) | > loss_mel_ce: 2.2280170917510986 (nan) | > loss: 0.00895710103213787 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13732974975610696) | > loader_time: 0.0045 (0.004737628069710575)  --> TIME: 2025-02-26 13:26:02 -- STEP: 2208/36938 -- GLOBAL_STEP: 1258100 | > loss_text_ce: 0.0362260565161705 (0.03114775987848151) | > loss_mel_ce: 2.1595051288604736 (nan) | > loss: 0.008713219314813614 (nan) | > current_lr: 5e-06 | > step_time: 0.1271 (0.13731203822122115) | > loader_time: 0.0045 (0.004735779611096865)  --> TIME: 2025-02-26 13:26:11 -- STEP: 2258/36938 -- GLOBAL_STEP: 1258150 | > loss_text_ce: 0.03236596658825874 (0.031180035614922445) | > loss_mel_ce: 2.6075563430786133 (nan) | > loss: 0.0104758832603693 (nan) | > current_lr: 5e-06 | > step_time: 0.1626 (0.1374314796407209) | > loader_time: 0.0047 (0.004734052505189066)  --> TIME: 2025-02-26 13:26:21 -- STEP: 2308/36938 -- GLOBAL_STEP: 1258200 | > loss_text_ce: 0.028938446193933487 (0.031167779012100563) | > loss_mel_ce: 2.8587770462036133 (nan) | > loss: 0.011459189467132092 (nan) | > current_lr: 5e-06 | > step_time: 0.1444 (0.13742079871159496) | > loader_time: 0.0046 (0.004733061150000448)  --> TIME: 2025-02-26 13:26:30 -- STEP: 2358/36938 -- GLOBAL_STEP: 1258250 | > loss_text_ce: 0.025098130106925964 (0.031192999602507925) | > loss_mel_ce: 2.348360776901245 (nan) | > loss: 0.0094184884801507 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13747088478418398) | > loader_time: 0.0046 (0.004732199803159021)  --> TIME: 2025-02-26 13:26:39 -- STEP: 2408/36938 -- GLOBAL_STEP: 1258300 | > loss_text_ce: 0.030182048678398132 (0.031221086169404274) | > loss_mel_ce: 1.662165880203247 (nan) | > loss: 0.006715666502714157 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13736060786484883) | > loader_time: 0.0045 (0.004731142738728815)  --> TIME: 2025-02-26 13:26:49 -- STEP: 2458/36938 -- GLOBAL_STEP: 1258350 | > loss_text_ce: 0.030485397204756737 (0.031224191934476522) | > loss_mel_ce: 2.2683603763580322 (nan) | > loss: 0.009122404269874096 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.1373533938743897) | > loader_time: 0.0046 (0.004729935558371846)  --> TIME: 2025-02-26 13:26:58 -- STEP: 2508/36938 -- GLOBAL_STEP: 1258400 | > loss_text_ce: 0.03644607961177826 (0.03126358514001063) | > loss_mel_ce: 2.2489466667175293 (nan) | > loss: 0.009069018997251987 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13737348801021193) | > loader_time: 0.0046 (0.0047283784814618505)  --> TIME: 2025-02-26 13:27:07 -- STEP: 2558/36938 -- GLOBAL_STEP: 1258450 | > loss_text_ce: 0.03700714558362961 (0.031262457316145666) | > loss_mel_ce: 2.026675224304199 (nan) | > loss: 0.008189216256141663 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.13727916508004293) | > loader_time: 0.0047 (0.004726904551436688)  --> TIME: 2025-02-26 13:27:16 -- STEP: 2608/36938 -- GLOBAL_STEP: 1258500 | > loss_text_ce: 0.021483534947037697 (0.03126120180722914) | > loss_mel_ce: 2.8132951259613037 (nan) | > loss: 0.011249122209846973 (nan) | > current_lr: 5e-06 | > step_time: 0.1777 (0.13721616264501232) | > loader_time: 0.0049 (0.004724989090960442)  --> TIME: 2025-02-26 13:27:26 -- STEP: 2658/36938 -- GLOBAL_STEP: 1258550 | > loss_text_ce: 0.04540577158331871 (0.031273054431330584) | > loss_mel_ce: 1.715019702911377 (nan) | > loss: 0.006985815707594156 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13729086139311547) | > loader_time: 0.0045 (0.004723951276573937)  --> TIME: 2025-02-26 13:27:35 -- STEP: 2708/36938 -- GLOBAL_STEP: 1258600 | > loss_text_ce: 0.023599039763212204 (0.03126133508165879) | > loss_mel_ce: 2.6532626152038574 (nan) | > loss: 0.01062246784567833 (nan) | > current_lr: 5e-06 | > step_time: 0.1577 (0.13731519827286273) | > loader_time: 0.0047 (0.004722854675535255)  --> TIME: 2025-02-26 13:27:45 -- STEP: 2758/36938 -- GLOBAL_STEP: 1258650 | > loss_text_ce: 0.03632361441850662 (0.031270340345043474) | > loss_mel_ce: 2.8048858642578125 (nan) | > loss: 0.011274641379714012 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.137378821864) | > loader_time: 0.0046 (0.004721890332648676)  --> TIME: 2025-02-26 13:27:54 -- STEP: 2808/36938 -- GLOBAL_STEP: 1258700 | > loss_text_ce: 0.03573353961110115 (0.03125920868810383) | > loss_mel_ce: 2.90200138092041 (nan) | > loss: 0.01165767852216959 (nan) | > current_lr: 5e-06 | > step_time: 0.1519 (0.137354750663806) | > loader_time: 0.0046 (0.004720982493158749)  --> TIME: 2025-02-26 13:28:03 -- STEP: 2858/36938 -- GLOBAL_STEP: 1258750 | > loss_text_ce: 0.02783927321434021 (0.03126193355101548) | > loss_mel_ce: 2.9114458560943604 (nan) | > loss: 0.011663829907774925 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13733360673932427) | > loader_time: 0.0045 (0.004719431955886094)  --> TIME: 2025-02-26 13:28:13 -- STEP: 2908/36938 -- GLOBAL_STEP: 1258800 | > loss_text_ce: 0.035872697830200195 (0.031262833219626224) | > loss_mel_ce: 1.929534912109375 (nan) | > loss: 0.007799237035214901 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13732191776802963) | > loader_time: 0.0045 (0.0047181975070843115)  --> TIME: 2025-02-26 13:28:22 -- STEP: 2958/36938 -- GLOBAL_STEP: 1258850 | > loss_text_ce: 0.040146831423044205 (0.031265237599112594) | > loss_mel_ce: 2.9710655212402344 (nan) | > loss: 0.01194925606250763 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13728670770852766) | > loader_time: 0.0047 (0.004717330903600722)  --> TIME: 2025-02-26 13:28:31 -- STEP: 3008/36938 -- GLOBAL_STEP: 1258900 | > loss_text_ce: 0.030436532571911812 (0.031258526585124305) | > loss_mel_ce: 0.4935188293457031 (nan) | > loss: 0.0020791878923773766 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13731681412838873) | > loader_time: 0.0046 (0.004716241613347471)  --> TIME: 2025-02-26 13:28:41 -- STEP: 3058/36938 -- GLOBAL_STEP: 1258950 | > loss_text_ce: 0.01982138119637966 (0.031233972505568267) | > loss_mel_ce: 2.2260308265686035 (nan) | > loss: 0.008912112563848495 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13736463497478715) | > loader_time: 0.0046 (0.004715079572015209)  --> TIME: 2025-02-26 13:28:50 -- STEP: 3108/36938 -- GLOBAL_STEP: 1259000 | > loss_text_ce: 0.029230894520878792 (0.031219489763981107) | > loss_mel_ce: 3.0132200717926025 (nan) | > loss: 0.012073218822479248 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13730486594572022) | > loader_time: 0.0046 (0.004714165798638927)  --> TIME: 2025-02-26 13:28:59 -- STEP: 3158/36938 -- GLOBAL_STEP: 1259050 | > loss_text_ce: 0.03179279342293739 (0.031209740536482084) | > loss_mel_ce: 2.6170098781585693 (nan) | > loss: 0.010511122643947601 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.1373230104887311) | > loader_time: 0.0047 (0.004715133498483553)  --> TIME: 2025-02-26 13:29:09 -- STEP: 3208/36938 -- GLOBAL_STEP: 1259100 | > loss_text_ce: 0.03333697095513344 (0.031226124985206268) | > loss_mel_ce: 2.0170984268188477 (nan) | > loss: 0.008136648684740067 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13731270642054647) | > loader_time: 0.0045 (0.004715683157009968)  --> TIME: 2025-02-26 13:29:18 -- STEP: 3258/36938 -- GLOBAL_STEP: 1259150 | > loss_text_ce: 0.02845648303627968 (0.031231180679263476) | > loss_mel_ce: 3.072216749191284 (nan) | > loss: 0.012304259464144707 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13732948875485793) | > loader_time: 0.0045 (0.004715234608354557)  --> TIME: 2025-02-26 13:29:27 -- STEP: 3308/36938 -- GLOBAL_STEP: 1259200 | > loss_text_ce: 0.02747052162885666 (0.03122197598260587) | > loss_mel_ce: 0.1715436726808548 (nan) | > loss: 0.0007897389004938304 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1372602665582909) | > loader_time: 0.0046 (0.004716750860502523)  --> TIME: 2025-02-26 13:29:37 -- STEP: 3358/36938 -- GLOBAL_STEP: 1259250 | > loss_text_ce: 0.02469429187476635 (0.031210208589853454) | > loss_mel_ce: 2.9006214141845703 (nan) | > loss: 0.011608395725488663 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13729391742704028) | > loader_time: 0.0047 (0.004717111090523204)  --> TIME: 2025-02-26 13:29:46 -- STEP: 3408/36938 -- GLOBAL_STEP: 1259300 | > loss_text_ce: 0.02840578183531761 (0.031242334694621288) | > loss_mel_ce: 2.309450626373291 (nan) | > loss: 0.009277207776904106 (nan) | > current_lr: 5e-06 | > step_time: 0.1518 (0.13723243870925458) | > loader_time: 0.0048 (0.004717190360799081)  --> TIME: 2025-02-26 13:29:55 -- STEP: 3458/36938 -- GLOBAL_STEP: 1259350 | > loss_text_ce: 0.028006291016936302 (0.03123918715880581) | > loss_mel_ce: 2.579558849334717 (nan) | > loss: 0.010347481817007065 (nan) | > current_lr: 5e-06 | > step_time: 0.1846 (0.13723648552814618) | > loader_time: 0.0048 (0.004717312912640369)  --> TIME: 2025-02-26 13:30:04 -- STEP: 3508/36938 -- GLOBAL_STEP: 1259400 | > loss_text_ce: 0.03192874789237976 (0.031228049664345066) | > loss_mel_ce: 0.6275970339775085 (nan) | > loss: 0.0026171659119427204 (nan) | > current_lr: 5e-06 | > step_time: 0.107 (0.13716108130212534) | > loader_time: 0.0046 (0.004717832280401492)  --> TIME: 2025-02-26 13:30:14 -- STEP: 3558/36938 -- GLOBAL_STEP: 1259450 | > loss_text_ce: 0.03680582717061043 (0.03124184372582008) | > loss_mel_ce: 3.3093745708465576 (nan) | > loss: 0.013278494589030743 (nan) | > current_lr: 5e-06 | > step_time: 0.1629 (0.13719223261281585) | > loader_time: 0.0046 (0.0047184102919880355)  --> TIME: 2025-02-26 13:30:23 -- STEP: 3608/36938 -- GLOBAL_STEP: 1259500 | > loss_text_ce: 0.03562458977103233 (0.031247781246060112) | > loss_mel_ce: 3.0154097080230713 (nan) | > loss: 0.012107279151678085 (nan) | > current_lr: 5e-06 | > step_time: 0.1921 (0.13729792278251748) | > loader_time: 0.0048 (0.004719684697571984)  --> TIME: 2025-02-26 13:30:33 -- STEP: 3658/36938 -- GLOBAL_STEP: 1259550 | > loss_text_ce: 0.03369663283228874 (0.031238416977687075) | > loss_mel_ce: 1.5482701063156128 (nan) | > loss: 0.006277646403759718 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.1372908176523275) | > loader_time: 0.0047 (0.0047189735075761335)  --> TIME: 2025-02-26 13:30:42 -- STEP: 3708/36938 -- GLOBAL_STEP: 1259600 | > loss_text_ce: 0.030472587794065475 (0.03123399844953739) | > loss_mel_ce: 3.3284108638763428 (nan) | > loss: 0.01332890335470438 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.13724786287501145) | > loader_time: 0.0048 (0.004717855777555298)  --> TIME: 2025-02-26 13:30:51 -- STEP: 3758/36938 -- GLOBAL_STEP: 1259650 | > loss_text_ce: 0.02681240253150463 (0.031227060000869677) | > loss_mel_ce: 0.6359950304031372 (nan) | > loss: 0.002630188362672925 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13727997394874408) | > loader_time: 0.0046 (0.004717820308638608)  --> TIME: 2025-02-26 13:31:01 -- STEP: 3808/36938 -- GLOBAL_STEP: 1259700 | > loss_text_ce: 0.028124652802944183 (0.031231683664740002) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13726005027023677) | > loader_time: 0.0046 (0.004717386883346973)  --> TIME: 2025-02-26 13:31:10 -- STEP: 3858/36938 -- GLOBAL_STEP: 1259750 | > loss_text_ce: 0.018323730677366257 (0.031231335684137627) | > loss_mel_ce: 2.4077274799346924 (nan) | > loss: 0.009627187624573708 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13726986498953822) | > loader_time: 0.0044 (0.0047171657230027165)  --> TIME: 2025-02-26 13:31:20 -- STEP: 3908/36938 -- GLOBAL_STEP: 1259800 | > loss_text_ce: 0.03401866555213928 (0.031235554398770567) | > loss_mel_ce: 2.580632448196411 (nan) | > loss: 0.010375600308179855 (nan) | > current_lr: 5e-06 | > step_time: 0.1619 (0.1373248420571841) | > loader_time: 0.0048 (0.004717132879255249)  --> TIME: 2025-02-26 13:31:29 -- STEP: 3958/36938 -- GLOBAL_STEP: 1259850 | > loss_text_ce: 0.0304829441010952 (0.031236620880371647) | > loss_mel_ce: 2.738067150115967 (nan) | > loss: 0.010986310429871082 (nan) | > current_lr: 5e-06 | > step_time: 0.1888 (0.13733466265477448) | > loader_time: 0.0048 (0.004716309710787425)  --> TIME: 2025-02-26 13:31:39 -- STEP: 4008/36938 -- GLOBAL_STEP: 1259900 | > loss_text_ce: 0.03200152888894081 (0.03121558748610402) | > loss_mel_ce: 2.7061245441436768 (nan) | > loss: 0.010865580290555954 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.13734007696667588) | > loader_time: 0.0046 (0.004715211020258372)  --> TIME: 2025-02-26 13:31:48 -- STEP: 4058/36938 -- GLOBAL_STEP: 1259950 | > loss_text_ce: 0.027900464832782745 (0.031214339324232343) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13732839641223224) | > loader_time: 0.0046 (0.004714285111180545)  --> TIME: 2025-02-26 13:31:57 -- STEP: 4108/36938 -- GLOBAL_STEP: 1260000 | > loss_text_ce: 0.03307598829269409 (0.031227083017273373) | > loss_mel_ce: 2.9904186725616455 (nan) | > loss: 0.011997995898127556 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.1373178526512487) | > loader_time: 0.0047 (0.004714228162050477) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1260000.pth  --> TIME: 2025-02-26 13:32:13 -- STEP: 4158/36938 -- GLOBAL_STEP: 1260050 | > loss_text_ce: 0.030261849984526634 (0.031228959752795972) | > loss_mel_ce: 2.863426923751831 (nan) | > loss: 0.01148289255797863 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.13731233846573598) | > loader_time: 0.0048 (0.004715637497858337)  --> TIME: 2025-02-26 13:32:22 -- STEP: 4208/36938 -- GLOBAL_STEP: 1260100 | > loss_text_ce: 0.028877751901745796 (0.031230685082459034) | > loss_mel_ce: 1.2526723146438599 (nan) | > loss: 0.005085516255348921 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13728274501321883) | > loader_time: 0.0046 (0.004715528721591817)  --> TIME: 2025-02-26 13:32:31 -- STEP: 4258/36938 -- GLOBAL_STEP: 1260150 | > loss_text_ce: 0.024629637598991394 (0.031202318830990436) | > loss_mel_ce: 2.6262621879577637 (nan) | > loss: 0.010519412346184254 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.13725841790087737) | > loader_time: 0.0049 (0.004715021365458217)  --> TIME: 2025-02-26 13:32:40 -- STEP: 4308/36938 -- GLOBAL_STEP: 1260200 | > loss_text_ce: 0.033814817667007446 (0.031193492159735138) | > loss_mel_ce: 2.785306453704834 (nan) | > loss: 0.011186989955604076 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13718106681368583) | > loader_time: 0.0047 (0.004714117962227113)  --> TIME: 2025-02-26 13:32:49 -- STEP: 4358/36938 -- GLOBAL_STEP: 1260250 | > loss_text_ce: 0.026041673496365547 (0.03119008811707891) | > loss_mel_ce: 1.92474365234375 (nan) | > loss: 0.007741211913526058 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13717458558881312) | > loader_time: 0.0047 (0.004713706655095286)  --> TIME: 2025-02-26 13:32:58 -- STEP: 4408/36938 -- GLOBAL_STEP: 1260300 | > loss_text_ce: 0.02921895682811737 (0.031190574716585637) | > loss_mel_ce: 2.433938503265381 (nan) | > loss: 0.009774434380233288 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13715210567801475) | > loader_time: 0.0045 (0.004712790358953596)  --> TIME: 2025-02-26 13:33:07 -- STEP: 4458/36938 -- GLOBAL_STEP: 1260350 | > loss_text_ce: 0.03675799071788788 (0.03119084881990482) | > loss_mel_ce: 3.1164748668670654 (nan) | > loss: 0.012512829154729843 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.13709971978558497) | > loader_time: 0.0049 (0.004712179296484119)  --> TIME: 2025-02-26 13:33:16 -- STEP: 4508/36938 -- GLOBAL_STEP: 1260400 | > loss_text_ce: 0.026233887299895287 (0.03119465300752168) | > loss_mel_ce: 1.6223862171173096 (nan) | > loss: 0.006542143877595663 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13703803008070212) | > loader_time: 0.0046 (0.004711140016591327)  --> TIME: 2025-02-26 13:33:25 -- STEP: 4558/36938 -- GLOBAL_STEP: 1260450 | > loss_text_ce: 0.03055982105433941 (0.031200933606887343) | > loss_mel_ce: 2.7854325771331787 (nan) | > loss: 0.011174573563039303 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1369505721573872) | > loader_time: 0.0046 (0.004710152987170082)  --> TIME: 2025-02-26 13:33:35 -- STEP: 4608/36938 -- GLOBAL_STEP: 1260500 | > loss_text_ce: 0.0328768789768219 (0.031207457933406783) | > loss_mel_ce: 2.9994704723358154 (nan) | > loss: 0.012033125385642052 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.13698385764534265) | > loader_time: 0.0046 (0.00471003022458818)  --> TIME: 2025-02-26 13:33:44 -- STEP: 4658/36938 -- GLOBAL_STEP: 1260550 | > loss_text_ce: 0.030210236087441444 (0.031186242498890928) | > loss_mel_ce: 2.9993529319763184 (nan) | > loss: 0.012022077105939388 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13697373867034904) | > loader_time: 0.0046 (0.0047098571724930515)  --> TIME: 2025-02-26 13:33:54 -- STEP: 4708/36938 -- GLOBAL_STEP: 1260600 | > loss_text_ce: 0.03379078954458237 (0.03118902859918953) | > loss_mel_ce: 2.8734748363494873 (nan) | > loss: 0.011536769568920135 (nan) | > current_lr: 5e-06 | > step_time: 0.1829 (0.13701423559537246) | > loader_time: 0.0048 (0.0047097680623387475)  --> TIME: 2025-02-26 13:34:03 -- STEP: 4758/36938 -- GLOBAL_STEP: 1260650 | > loss_text_ce: 0.029075738042593002 (0.031184673226528917) | > loss_mel_ce: 2.707828998565674 (nan) | > loss: 0.010860733687877655 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13703653104869498) | > loader_time: 0.0047 (0.004710222453116964)  --> TIME: 2025-02-26 13:34:13 -- STEP: 4808/36938 -- GLOBAL_STEP: 1260700 | > loss_text_ce: 0.03390238806605339 (0.031184578931754085) | > loss_mel_ce: 3.0687999725341797 (nan) | > loss: 0.012312311679124832 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1370731969046315) | > loader_time: 0.0046 (0.00471019516768748)  --> TIME: 2025-02-26 13:34:22 -- STEP: 4858/36938 -- GLOBAL_STEP: 1260750 | > loss_text_ce: 0.023984121158719063 (0.031201176233965424) | > loss_mel_ce: 2.816459894180298 (nan) | > loss: 0.011271604336798191 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.13705261168827393) | > loader_time: 0.0047 (0.004709795405708637)  --> TIME: 2025-02-26 13:34:32 -- STEP: 4908/36938 -- GLOBAL_STEP: 1260800 | > loss_text_ce: 0.023596661165356636 (0.0311912789885046) | > loss_mel_ce: 2.5415596961975098 (nan) | > loss: 0.010179192759096622 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13710006000360248) | > loader_time: 0.0046 (0.004710097233260062)  --> TIME: 2025-02-26 13:34:41 -- STEP: 4958/36938 -- GLOBAL_STEP: 1260850 | > loss_text_ce: 0.04014044255018234 (0.031211959280204402) | > loss_mel_ce: 2.914555311203003 (nan) | > loss: 0.011724983341991901 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13711415192156656) | > loader_time: 0.0046 (0.004709318310274799)  --> TIME: 2025-02-26 13:34:50 -- STEP: 5008/36938 -- GLOBAL_STEP: 1260900 | > loss_text_ce: 0.02586735412478447 (0.031214224933549937) | > loss_mel_ce: 2.5335497856140137 (nan) | > loss: 0.010156418196856976 (nan) | > current_lr: 5e-06 | > step_time: 0.1464 (0.1371099877471741) | > loader_time: 0.0047 (0.004709099952024378)  --> TIME: 2025-02-26 13:35:00 -- STEP: 5058/36938 -- GLOBAL_STEP: 1260950 | > loss_text_ce: 0.03269397094845772 (0.031219729255684576) | > loss_mel_ce: 2.425342082977295 (nan) | > loss: 0.009754111059010029 (nan) | > current_lr: 5e-06 | > step_time: 0.1823 (0.13710241371485124) | > loader_time: 0.0048 (0.004709251080637807)  --> TIME: 2025-02-26 13:35:09 -- STEP: 5108/36938 -- GLOBAL_STEP: 1261000 | > loss_text_ce: 0.0255669467151165 (0.031225346537677414) | > loss_mel_ce: 2.4724252223968506 (nan) | > loss: 0.009912668727338314 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13711285969197232) | > loader_time: 0.0046 (0.004709697553863151)  --> TIME: 2025-02-26 13:35:18 -- STEP: 5158/36938 -- GLOBAL_STEP: 1261050 | > loss_text_ce: 0.040858324617147446 (0.031221320067025736) | > loss_mel_ce: 2.2406723499298096 (nan) | > loss: 0.009053693152964115 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.1370998488326736) | > loader_time: 0.0045 (0.004709164085846703)  --> TIME: 2025-02-26 13:35:27 -- STEP: 5208/36938 -- GLOBAL_STEP: 1261100 | > loss_text_ce: 0.02554895170032978 (0.031215832165453875) | > loss_mel_ce: 2.501251697540283 (nan) | > loss: 0.01002698764204979 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.13705983858687165) | > loader_time: 0.0046 (0.004708467128639386)  --> TIME: 2025-02-26 13:35:37 -- STEP: 5258/36938 -- GLOBAL_STEP: 1261150 | > loss_text_ce: 0.030369099229574203 (0.031218710050097388) | > loss_mel_ce: 2.442814588546753 (nan) | > loss: 0.009814221411943436 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1371403689981185) | > loader_time: 0.0046 (0.004708576900689662)  --> TIME: 2025-02-26 13:35:47 -- STEP: 5308/36938 -- GLOBAL_STEP: 1261200 | > loss_text_ce: 0.027882670983672142 (0.03121908071198936) | > loss_mel_ce: 2.080246686935425 (nan) | > loss: 0.008365592919290066 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13714239582947133) | > loader_time: 0.0046 (0.004708266563674239)  --> TIME: 2025-02-26 13:35:56 -- STEP: 5358/36938 -- GLOBAL_STEP: 1261250 | > loss_text_ce: 0.029835326597094536 (0.031217581877215867) | > loss_mel_ce: 0.0919189453125 (nan) | > loss: 0.0004831519036088139 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1371379478870373) | > loader_time: 0.0046 (0.004707875737684695)  --> TIME: 2025-02-26 13:36:05 -- STEP: 5408/36938 -- GLOBAL_STEP: 1261300 | > loss_text_ce: 0.03466638922691345 (0.031215410570740808) | > loss_mel_ce: 1.8449598550796509 (nan) | > loss: 0.007458834908902645 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.1371068670111291) | > loader_time: 0.0046 (0.004707343082808877)  --> TIME: 2025-02-26 13:36:14 -- STEP: 5458/36938 -- GLOBAL_STEP: 1261350 | > loss_text_ce: 0.029292434453964233 (0.03120955278139363) | > loss_mel_ce: 3.410939931869507 (nan) | > loss: 0.013651716522872448 (nan) | > current_lr: 5e-06 | > step_time: 0.1909 (0.13707163090144955) | > loader_time: 0.0049 (0.004707363683603405)  --> TIME: 2025-02-26 13:36:24 -- STEP: 5508/36938 -- GLOBAL_STEP: 1261400 | > loss_text_ce: 0.033855605870485306 (0.031222658677901836) | > loss_mel_ce: 2.877931594848633 (nan) | > loss: 0.011554712429642677 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13709786021371786) | > loader_time: 0.0047 (0.004707178215921653)  --> TIME: 2025-02-26 13:36:33 -- STEP: 5558/36938 -- GLOBAL_STEP: 1261450 | > loss_text_ce: 0.03030061349272728 (0.031229252877406268) | > loss_mel_ce: 2.1960911750793457 (nan) | > loss: 0.008834888227283955 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13709683056227687) | > loader_time: 0.0046 (0.004708274399989703)  --> TIME: 2025-02-26 13:36:43 -- STEP: 5608/36938 -- GLOBAL_STEP: 1261500 | > loss_text_ce: 0.03467116132378578 (0.031223831550827995) | > loss_mel_ce: 0.10205078125 (nan) | > loss: 0.000542547379154712 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13714764261891899) | > loader_time: 0.0046 (0.0047080506611822575)  --> TIME: 2025-02-26 13:36:52 -- STEP: 5658/36938 -- GLOBAL_STEP: 1261550 | > loss_text_ce: 0.02955246903002262 (0.031220893288336084) | > loss_mel_ce: 2.64559006690979 (nan) | > loss: 0.010615645907819271 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.1371913244463855) | > loader_time: 0.0046 (0.004708027999786231)  --> TIME: 2025-02-26 13:37:02 -- STEP: 5708/36938 -- GLOBAL_STEP: 1261600 | > loss_text_ce: 0.02805406227707863 (0.031213302273445567) | > loss_mel_ce: 2.8592841625213623 (nan) | > loss: 0.01145769190043211 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13717082299464345) | > loader_time: 0.0047 (0.004708382535331628)  --> TIME: 2025-02-26 13:37:11 -- STEP: 5758/36938 -- GLOBAL_STEP: 1261650 | > loss_text_ce: 0.03481496870517731 (0.031207006616655725) | > loss_mel_ce: 2.702399730682373 (nan) | > loss: 0.010861963964998722 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1371544063733075) | > loader_time: 0.0048 (0.004707994085088615)  --> TIME: 2025-02-26 13:37:21 -- STEP: 5808/36938 -- GLOBAL_STEP: 1261700 | > loss_text_ce: 0.029105400666594505 (0.03121357297312506) | > loss_mel_ce: 2.8495237827301025 (nan) | > loss: 0.011423132382333279 (nan) | > current_lr: 5e-06 | > step_time: 0.1667 (0.13720209260288968) | > loader_time: 0.0047 (0.004708294369300854)  --> TIME: 2025-02-26 13:37:30 -- STEP: 5858/36938 -- GLOBAL_STEP: 1261750 | > loss_text_ce: 0.03379829600453377 (0.031208062093430234) | > loss_mel_ce: 2.8682708740234375 (nan) | > loss: 0.011516148224473 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.1372379908263782) | > loader_time: 0.0048 (0.004708879431102326)  --> TIME: 2025-02-26 13:37:39 -- STEP: 5908/36938 -- GLOBAL_STEP: 1261800 | > loss_text_ce: 0.035245876759290695 (0.031213800642436382) | > loss_mel_ce: 2.41703200340271 (nan) | > loss: 0.009731261990964413 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13721630681424632) | > loader_time: 0.0047 (0.004708467743730894)  --> TIME: 2025-02-26 13:37:49 -- STEP: 5958/36938 -- GLOBAL_STEP: 1261850 | > loss_text_ce: 0.026844238862395287 (0.031215371544491336) | > loss_mel_ce: 2.8439598083496094 (nan) | > loss: 0.01139208022505045 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.1372272434071349) | > loader_time: 0.0047 (0.004708548566966133)  --> TIME: 2025-02-26 13:37:58 -- STEP: 6008/36938 -- GLOBAL_STEP: 1261900 | > loss_text_ce: 0.029228996485471725 (0.03122395878244124) | > loss_mel_ce: 3.7326788902282715 (nan) | > loss: 0.01492820680141449 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13719002638612354) | > loader_time: 0.0046 (0.004708009458254873)  --> TIME: 2025-02-26 13:38:07 -- STEP: 6058/36938 -- GLOBAL_STEP: 1261950 | > loss_text_ce: 0.030315449461340904 (0.03121873955468327) | > loss_mel_ce: 3.077244758605957 (nan) | > loss: 0.012331588193774223 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13719186612819007) | > loader_time: 0.0047 (0.0047078903219651795)  --> TIME: 2025-02-26 13:38:17 -- STEP: 6108/36938 -- GLOBAL_STEP: 1262000 | > loss_text_ce: 0.0410621240735054 (0.031221349660969542) | > loss_mel_ce: 2.3330140113830566 (nan) | > loss: 0.009420937858521938 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13722068719326747) | > loader_time: 0.0047 (0.004707772901968458)  --> TIME: 2025-02-26 13:38:26 -- STEP: 6158/36938 -- GLOBAL_STEP: 1262050 | > loss_text_ce: 0.021459180861711502 (0.031221201281435625) | > loss_mel_ce: 2.175291061401367 (nan) | > loss: 0.008717263117432594 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13725872324751165) | > loader_time: 0.0046 (0.004707654330125669)  --> TIME: 2025-02-26 13:38:36 -- STEP: 6208/36938 -- GLOBAL_STEP: 1262100 | > loss_text_ce: 0.0378136932849884 (0.03122344377559261) | > loss_mel_ce: 3.1041831970214844 (nan) | > loss: 0.012468242086470127 (nan) | > current_lr: 5e-06 | > step_time: 0.1922 (0.1372535882767325) | > loader_time: 0.0049 (0.0047072417189165374)  --> TIME: 2025-02-26 13:38:45 -- STEP: 6258/36938 -- GLOBAL_STEP: 1262150 | > loss_text_ce: 0.028772082179784775 (0.031212659601953175) | > loss_mel_ce: 3.038565158843994 (nan) | > loss: 0.012171974405646324 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.13719238610875653) | > loader_time: 0.0049 (0.004707393611189873)  --> TIME: 2025-02-26 13:38:54 -- STEP: 6308/36938 -- GLOBAL_STEP: 1262200 | > loss_text_ce: 0.03221156448125839 (0.031216334676329883) | > loss_mel_ce: 2.6785390377044678 (nan) | > loss: 0.010756947100162506 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13722417195083644) | > loader_time: 0.0046 (0.004707830875979689)  --> TIME: 2025-02-26 13:39:04 -- STEP: 6358/36938 -- GLOBAL_STEP: 1262250 | > loss_text_ce: 0.024272650480270386 (0.031203145945673815) | > loss_mel_ce: 2.649168014526367 (nan) | > loss: 0.01060889195650816 (nan) | > current_lr: 5e-06 | > step_time: 0.1515 (0.13724168882912943) | > loader_time: 0.0047 (0.004707667091876002)  --> TIME: 2025-02-26 13:39:13 -- STEP: 6408/36938 -- GLOBAL_STEP: 1262300 | > loss_text_ce: 0.02595943957567215 (0.031203131497658566) | > loss_mel_ce: 2.1915764808654785 (nan) | > loss: 0.008799746632575989 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13726079434193586) | > loader_time: 0.0045 (0.004707400346367837)  --> TIME: 2025-02-26 13:39:23 -- STEP: 6458/36938 -- GLOBAL_STEP: 1262350 | > loss_text_ce: 0.025956859812140465 (0.031191793458606083) | > loss_mel_ce: 2.551961660385132 (nan) | > loss: 0.010229836218059063 (nan) | > current_lr: 5e-06 | > step_time: 0.1445 (0.13727961464465716) | > loader_time: 0.0047 (0.004707047835190834)  --> TIME: 2025-02-26 13:39:32 -- STEP: 6508/36938 -- GLOBAL_STEP: 1262400 | > loss_text_ce: 0.02083732560276985 (0.031193227513327516) | > loss_mel_ce: 2.194361686706543 (nan) | > loss: 0.008790472522377968 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13729130818789587) | > loader_time: 0.0046 (0.0047067470102128365)  --> TIME: 2025-02-26 13:39:42 -- STEP: 6558/36938 -- GLOBAL_STEP: 1262450 | > loss_text_ce: 0.031632013618946075 (0.031185711969028734) | > loss_mel_ce: 3.0728418827056885 (nan) | > loss: 0.01231934130191803 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13733791112536192) | > loader_time: 0.0047 (0.004706643855800773)  --> TIME: 2025-02-26 13:39:51 -- STEP: 6608/36938 -- GLOBAL_STEP: 1262500 | > loss_text_ce: 0.037082720547914505 (0.031189185405120758) | > loss_mel_ce: 3.05997896194458 (nan) | > loss: 0.012289928272366524 (nan) | > current_lr: 5e-06 | > step_time: 0.1891 (0.13731815371905892) | > loader_time: 0.0049 (0.004706093748314315)  --> TIME: 2025-02-26 13:40:01 -- STEP: 6658/36938 -- GLOBAL_STEP: 1262550 | > loss_text_ce: 0.03579061105847359 (0.031187901052918562) | > loss_mel_ce: 2.823251724243164 (nan) | > loss: 0.011345406994223595 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13733292804103794) | > loader_time: 0.0046 (0.0047058321828776116)  --> TIME: 2025-02-26 13:40:10 -- STEP: 6708/36938 -- GLOBAL_STEP: 1262600 | > loss_text_ce: 0.036672838032245636 (0.031194522402643036) | > loss_mel_ce: 3.2045652866363525 (nan) | > loss: 0.012862056493759155 (nan) | > current_lr: 5e-06 | > step_time: 0.1769 (0.13735677141054217) | > loader_time: 0.0048 (0.004705628363518879)  --> TIME: 2025-02-26 13:40:20 -- STEP: 6758/36938 -- GLOBAL_STEP: 1262650 | > loss_text_ce: 0.02624949812889099 (0.03119475848641503) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.13733748696054277) | > loader_time: 0.0046 (0.0047052012071273465)  --> TIME: 2025-02-26 13:40:29 -- STEP: 6808/36938 -- GLOBAL_STEP: 1262700 | > loss_text_ce: 0.03187509998679161 (0.03119232903191434) | > loss_mel_ce: 2.6633377075195312 (nan) | > loss: 0.010695289820432663 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13737652591756866) | > loader_time: 0.0047 (0.004705406594080295)  --> TIME: 2025-02-26 13:40:39 -- STEP: 6858/36938 -- GLOBAL_STEP: 1262750 | > loss_text_ce: 0.02990270033478737 (0.03119277791018263) | > loss_mel_ce: 1.7191799879074097 (nan) | > loss: 0.00694080488756299 (nan) | > current_lr: 5e-06 | > step_time: 0.1502 (0.13737669273380912) | > loader_time: 0.0047 (0.004704987352671011)  --> TIME: 2025-02-26 13:40:48 -- STEP: 6908/36938 -- GLOBAL_STEP: 1262800 | > loss_text_ce: 0.026756666600704193 (0.031191193539655736) | > loss_mel_ce: 2.653841733932495 (nan) | > loss: 0.010637296363711357 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13733929333551545) | > loader_time: 0.0045 (0.004704630989246979)  --> TIME: 2025-02-26 13:40:57 -- STEP: 6958/36938 -- GLOBAL_STEP: 1262850 | > loss_text_ce: 0.03287777677178383 (0.031198218589598828) | > loss_mel_ce: 2.503580093383789 (nan) | > loss: 0.010065308772027493 (nan) | > current_lr: 5e-06 | > step_time: 0.1867 (0.1373249373239977) | > loader_time: 0.0049 (0.004704423490767147)  --> TIME: 2025-02-26 13:41:07 -- STEP: 7008/36938 -- GLOBAL_STEP: 1262900 | > loss_text_ce: 0.017349082976579666 (0.0311953118557353) | > loss_mel_ce: 1.9901267290115356 (nan) | > loss: 0.007966174744069576 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13733592323244445) | > loader_time: 0.0046 (0.004704473089409727)  --> TIME: 2025-02-26 13:41:16 -- STEP: 7058/36938 -- GLOBAL_STEP: 1262950 | > loss_text_ce: 0.03257625550031662 (0.031202164523991792) | > loss_mel_ce: 2.4558990001678467 (nan) | > loss: 0.009874902665615082 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13732418591052645) | > loader_time: 0.0046 (0.00470378909647446)  --> TIME: 2025-02-26 13:41:26 -- STEP: 7108/36938 -- GLOBAL_STEP: 1263000 | > loss_text_ce: 0.04178040474653244 (0.031199565068721687) | > loss_mel_ce: 2.8068699836730957 (nan) | > loss: 0.011304168961942196 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13737043428099033) | > loader_time: 0.0046 (0.004704235072342528)  --> TIME: 2025-02-26 13:41:35 -- STEP: 7158/36938 -- GLOBAL_STEP: 1263050 | > loss_text_ce: 0.02351101115345955 (0.031198610081440193) | > loss_mel_ce: 1.907092571258545 (nan) | > loss: 0.007661126088351011 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.1373487106745855) | > loader_time: 0.0046 (0.004703953666505799)  --> TIME: 2025-02-26 13:41:44 -- STEP: 7208/36938 -- GLOBAL_STEP: 1263100 | > loss_text_ce: 0.0265812985599041 (0.03119288465229343) | > loss_mel_ce: 3.004441738128662 (nan) | > loss: 0.012027869932353497 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.13735303458045525) | > loader_time: 0.0047 (0.004703961155655364)  --> TIME: 2025-02-26 13:41:53 -- STEP: 7258/36938 -- GLOBAL_STEP: 1263150 | > loss_text_ce: 0.035636480897665024 (0.031194836384757698) | > loss_mel_ce: 2.4448251724243164 (nan) | > loss: 0.009843101724982262 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13732300805927286) | > loader_time: 0.0046 (0.004703689685830976)  --> TIME: 2025-02-26 13:42:03 -- STEP: 7308/36938 -- GLOBAL_STEP: 1263200 | > loss_text_ce: 0.023541757836937904 (0.03119897429276204) | > loss_mel_ce: 1.6871861219406128 (nan) | > loss: 0.006788603495806456 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13729341700877018) | > loader_time: 0.0046 (0.0047030697510943055)  --> TIME: 2025-02-26 13:42:12 -- STEP: 7358/36938 -- GLOBAL_STEP: 1263250 | > loss_text_ce: 0.032393768429756165 (0.03120141611749687) | > loss_mel_ce: 2.7571027278900146 (nan) | > loss: 0.011069430969655514 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.1372879007712238) | > loader_time: 0.0046 (0.00470258179291591)  --> TIME: 2025-02-26 13:42:21 -- STEP: 7408/36938 -- GLOBAL_STEP: 1263300 | > loss_text_ce: 0.030888454988598824 (0.031204504138924855) | > loss_mel_ce: 2.517498731613159 (nan) | > loss: 0.010112648829817772 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.1373226625872999) | > loader_time: 0.0047 (0.004702341575859415)  --> TIME: 2025-02-26 13:42:31 -- STEP: 7458/36938 -- GLOBAL_STEP: 1263350 | > loss_text_ce: 0.02667151391506195 (0.031203660642447006) | > loss_mel_ce: 3.253506898880005 (nan) | > loss: 0.013016581535339355 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13733998841720235) | > loader_time: 0.0047 (0.004702072835345774)  --> TIME: 2025-02-26 13:42:40 -- STEP: 7508/36938 -- GLOBAL_STEP: 1263400 | > loss_text_ce: 0.04071914404630661 (0.031199879859230305) | > loss_mel_ce: 2.0115966796875 (nan) | > loss: 0.008144110441207886 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.1373333517515604) | > loader_time: 0.0047 (0.004701713456330158)  --> TIME: 2025-02-26 13:42:49 -- STEP: 7558/36938 -- GLOBAL_STEP: 1263450 | > loss_text_ce: 0.02653585374355316 (0.03121096845844392) | > loss_mel_ce: 2.7425537109375 (nan) | > loss: 0.010988450609147549 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.137312463207452) | > loader_time: 0.0046 (0.004701226279356375)  --> TIME: 2025-02-26 13:42:59 -- STEP: 7608/36938 -- GLOBAL_STEP: 1263500 | > loss_text_ce: 0.03006783127784729 (0.031212116215088185) | > loss_mel_ce: 3.188368082046509 (nan) | > loss: 0.01277157198637724 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.1373185318288494) | > loader_time: 0.0046 (0.004700949139399466)  --> TIME: 2025-02-26 13:43:08 -- STEP: 7658/36938 -- GLOBAL_STEP: 1263550 | > loss_text_ce: 0.029133634641766548 (0.03120802720066392) | > loss_mel_ce: 2.070239782333374 (nan) | > loss: 0.008330847136676311 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.1373082053477662) | > loader_time: 0.0046 (0.004700519111455949)  --> TIME: 2025-02-26 13:43:18 -- STEP: 7708/36938 -- GLOBAL_STEP: 1263600 | > loss_text_ce: 0.037210430949926376 (0.03121279332096645) | > loss_mel_ce: 0.025390625 (nan) | > loss: 0.0002484169090166688 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13732616237611694) | > loader_time: 0.0046 (0.00470037406506658)  --> TIME: 2025-02-26 13:43:27 -- STEP: 7758/36938 -- GLOBAL_STEP: 1263650 | > loss_text_ce: 0.03470095992088318 (0.031217422056631073) | > loss_mel_ce: 2.3110830783843994 (nan) | > loss: 0.009308666922152042 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13730206063003342) | > loader_time: 0.0045 (0.004699883893704576)  --> TIME: 2025-02-26 13:43:36 -- STEP: 7808/36938 -- GLOBAL_STEP: 1263700 | > loss_text_ce: 0.03147604316473007 (0.031218846824037336) | > loss_mel_ce: 3.0458626747131348 (nan) | > loss: 0.012211661785840988 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.1372760021906409) | > loader_time: 0.0046 (0.0046998299047595)  --> TIME: 2025-02-26 13:43:45 -- STEP: 7858/36938 -- GLOBAL_STEP: 1263750 | > loss_text_ce: 0.032534923404455185 (0.03121223802599829) | > loss_mel_ce: 3.079533576965332 (nan) | > loss: 0.012349478900432587 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13726780013474976) | > loader_time: 0.0047 (0.004699748476882339)  --> TIME: 2025-02-26 13:43:54 -- STEP: 7908/36938 -- GLOBAL_STEP: 1263800 | > loss_text_ce: 0.03257482871413231 (0.031213241772510437) | > loss_mel_ce: 2.5751953125 (nan) | > loss: 0.010348294861614704 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13723391617074077) | > loader_time: 0.0046 (0.0046997383862477414)  --> TIME: 2025-02-26 13:44:04 -- STEP: 7958/36938 -- GLOBAL_STEP: 1263850 | > loss_text_ce: 0.029943406581878662 (0.031206322121886832) | > loss_mel_ce: 2.6679208278656006 (nan) | > loss: 0.010705810971558094 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13725150572232026) | > loader_time: 0.0047 (0.0046996908530603995)  --> TIME: 2025-02-26 13:44:13 -- STEP: 8008/36938 -- GLOBAL_STEP: 1263900 | > loss_text_ce: 0.04883342236280441 (0.031218775689227894) | > loss_mel_ce: 2.814894914627075 (nan) | > loss: 0.011364001780748367 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13723951271483012) | > loader_time: 0.0045 (0.004699474477863201)  --> TIME: 2025-02-26 13:44:23 -- STEP: 8058/36938 -- GLOBAL_STEP: 1263950 | > loss_text_ce: 0.03150711581110954 (0.031217778770294222) | > loss_mel_ce: 2.5113251209259033 (nan) | > loss: 0.010090604424476624 (nan) | > current_lr: 5e-06 | > step_time: 0.1582 (0.13723507673694837) | > loader_time: 0.0046 (0.004699176492215252)  --> TIME: 2025-02-26 13:44:32 -- STEP: 8108/36938 -- GLOBAL_STEP: 1264000 | > loss_text_ce: 0.03494127094745636 (0.03120921193596784) | > loss_mel_ce: 2.9000916481018066 (nan) | > loss: 0.011646956205368042 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13720576040957694) | > loader_time: 0.0045 (0.004698516937918287)  --> TIME: 2025-02-26 13:44:41 -- STEP: 8158/36938 -- GLOBAL_STEP: 1264050 | > loss_text_ce: 0.03309513255953789 (0.03120570356968347) | > loss_mel_ce: 3.0193376541137695 (nan) | > loss: 0.012112828902900219 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.1371921422756839) | > loader_time: 0.0047 (0.0046978092976836736)  --> TIME: 2025-02-26 13:44:50 -- STEP: 8208/36938 -- GLOBAL_STEP: 1264100 | > loss_text_ce: 0.024965383112430573 (0.031209160339361393) | > loss_mel_ce: 2.1301472187042236 (nan) | > loss: 0.008552034385502338 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1371781363008549) | > loader_time: 0.0045 (0.004697172555774725)  --> TIME: 2025-02-26 13:44:59 -- STEP: 8258/36938 -- GLOBAL_STEP: 1264150 | > loss_text_ce: 0.03386223316192627 (0.031209653407627616) | > loss_mel_ce: 2.4048268795013428 (nan) | > loss: 0.009677338413894176 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.1371629559465402) | > loader_time: 0.0046 (0.0046968532549783336)  --> TIME: 2025-02-26 13:45:08 -- STEP: 8308/36938 -- GLOBAL_STEP: 1264200 | > loss_text_ce: 0.026428932324051857 (0.03120327865539885) | > loss_mel_ce: 2.998993158340454 (nan) | > loss: 0.012005643919110298 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.13714075940815842) | > loader_time: 0.0047 (0.004696699105952858)  --> TIME: 2025-02-26 13:45:18 -- STEP: 8358/36938 -- GLOBAL_STEP: 1264250 | > loss_text_ce: 0.02534673921763897 (0.031194819961136322) | > loss_mel_ce: 2.2260093688964844 (nan) | > loss: 0.008933953009545803 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.1371198273583674) | > loader_time: 0.0046 (0.004696437205159212)  --> TIME: 2025-02-26 13:45:27 -- STEP: 8408/36938 -- GLOBAL_STEP: 1264300 | > loss_text_ce: 0.025788437575101852 (0.031192207318836117) | > loss_mel_ce: 2.299081563949585 (nan) | > loss: 0.009225675836205482 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13708600211892108) | > loader_time: 0.0047 (0.004695849714678414)  --> TIME: 2025-02-26 13:45:36 -- STEP: 8458/36938 -- GLOBAL_STEP: 1264350 | > loss_text_ce: 0.021587494760751724 (0.031193596724051705) | > loss_mel_ce: 0.2041015625 (nan) | > loss: 0.000895591510925442 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13706784345320602) | > loader_time: 0.0046 (0.0046953315513865055)  --> TIME: 2025-02-26 13:45:45 -- STEP: 8508/36938 -- GLOBAL_STEP: 1264400 | > loss_text_ce: 0.01804516464471817 (0.0311859057827397) | > loss_mel_ce: 1.664689540863037 (nan) | > loss: 0.00667751906439662 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13707410560620475) | > loader_time: 0.0045 (0.004695469665034126)  --> TIME: 2025-02-26 13:45:54 -- STEP: 8558/36938 -- GLOBAL_STEP: 1264450 | > loss_text_ce: 0.03257026895880699 (0.03118267997735418) | > loss_mel_ce: 0.0791015625 (nan) | > loss: 0.000443142227595672 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.1370684568445715) | > loader_time: 0.0046 (0.004695272969430658)  --> TIME: 2025-02-26 13:46:04 -- STEP: 8608/36938 -- GLOBAL_STEP: 1264500 | > loss_text_ce: 0.02711917832493782 (0.031181149024145906) | > loss_mel_ce: 2.6184141635894775 (nan) | > loss: 0.010498148389160633 (nan) | > current_lr: 5e-06 | > step_time: 0.1071 (0.1370364683212845) | > loader_time: 0.0045 (0.004695085649375121)  --> TIME: 2025-02-26 13:46:13 -- STEP: 8658/36938 -- GLOBAL_STEP: 1264550 | > loss_text_ce: 0.03099030628800392 (0.031177830827080602) | > loss_mel_ce: 1.2509714365005493 (nan) | > loss: 0.005087150260806084 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1370137497304246) | > loader_time: 0.0046 (0.004694828537711634)  --> TIME: 2025-02-26 13:46:22 -- STEP: 8708/36938 -- GLOBAL_STEP: 1264600 | > loss_text_ce: 0.034137193113565445 (0.031172575031440986) | > loss_mel_ce: 2.1058592796325684 (nan) | > loss: 0.008492049761116505 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.1370010726157273) | > loader_time: 0.0048 (0.004694654791520489)  --> TIME: 2025-02-26 13:46:32 -- STEP: 8758/36938 -- GLOBAL_STEP: 1264650 | > loss_text_ce: 0.027763821184635162 (0.031177504946904997) | > loss_mel_ce: 2.9749250411987305 (nan) | > loss: 0.011915433220565319 (nan) | > current_lr: 5e-06 | > step_time: 0.1884 (0.1370261822369688) | > loader_time: 0.0048 (0.0046945266403490785)  --> TIME: 2025-02-26 13:46:41 -- STEP: 8808/36938 -- GLOBAL_STEP: 1264700 | > loss_text_ce: 0.03976612538099289 (0.031177191411633123) | > loss_mel_ce: 2.9948670864105225 (nan) | > loss: 0.012042195536196232 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.1370175618620812) | > loader_time: 0.0049 (0.004694656444397097)  --> TIME: 2025-02-26 13:46:50 -- STEP: 8858/36938 -- GLOBAL_STEP: 1264750 | > loss_text_ce: 0.029959244653582573 (0.03117366740070024) | > loss_mel_ce: 2.9967050552368164 (nan) | > loss: 0.012010573409497738 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13703307558543323) | > loader_time: 0.0046 (0.004694103306425577)  --> TIME: 2025-02-26 13:47:00 -- STEP: 8908/36938 -- GLOBAL_STEP: 1264800 | > loss_text_ce: 0.03688046708703041 (0.03117428974427876) | > loss_mel_ce: 3.572709083557129 (nan) | > loss: 0.014323769137263298 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.1370532688507865) | > loader_time: 0.0049 (0.004693804431721708)  --> TIME: 2025-02-26 13:47:09 -- STEP: 8958/36938 -- GLOBAL_STEP: 1264850 | > loss_text_ce: 0.03545794263482094 (0.031185595244443715) | > loss_mel_ce: 0.302734375 (nan) | > loss: 0.0013420330360531807 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13705874040313643) | > loader_time: 0.0046 (0.00469353761117248)  --> TIME: 2025-02-26 13:47:18 -- STEP: 9008/36938 -- GLOBAL_STEP: 1264900 | > loss_text_ce: 0.03148002177476883 (0.031178880831124158) | > loss_mel_ce: 3.028778314590454 (nan) | > loss: 0.012143882922828197 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.1370426417455792) | > loader_time: 0.0047 (0.004693021068141037)  --> TIME: 2025-02-26 13:47:28 -- STEP: 9058/36938 -- GLOBAL_STEP: 1264950 | > loss_text_ce: 0.03580719977617264 (0.031180190179907464) | > loss_mel_ce: 2.2399280071258545 (nan) | > loss: 0.00903069507330656 (nan) | > current_lr: 5e-06 | > step_time: 0.1591 (0.13704093146413648) | > loader_time: 0.0047 (0.004692527336588904)  --> TIME: 2025-02-26 13:47:37 -- STEP: 9108/36938 -- GLOBAL_STEP: 1265000 | > loss_text_ce: 0.022218424826860428 (0.031174278389016277) | > loss_mel_ce: 2.791968584060669 (nan) | > loss: 0.01116740982979536 (nan) | > current_lr: 5e-06 | > step_time: 0.1824 (0.13701962903763681) | > loader_time: 0.0047 (0.004691872541245829)  --> TIME: 2025-02-26 13:47:46 -- STEP: 9158/36938 -- GLOBAL_STEP: 1265050 | > loss_text_ce: 0.032686784863471985 (0.031180773080078696) | > loss_mel_ce: 1.251017689704895 (nan) | > loss: 0.005094065796583891 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13703156100737793) | > loader_time: 0.0045 (0.004691594863828273)  --> TIME: 2025-02-26 13:47:56 -- STEP: 9208/36938 -- GLOBAL_STEP: 1265100 | > loss_text_ce: 0.043618153780698776 (0.03118422508505146) | > loss_mel_ce: 2.4077999591827393 (nan) | > loss: 0.009727850556373596 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.13704090960951706) | > loader_time: 0.0046 (0.004691238744066225)  --> TIME: 2025-02-26 13:48:05 -- STEP: 9258/36938 -- GLOBAL_STEP: 1265150 | > loss_text_ce: 0.03424479439854622 (0.031184789204966392) | > loss_mel_ce: 2.4088542461395264 (nan) | > loss: 0.009694837965071201 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.1370499462925814) | > loader_time: 0.0047 (0.004690878770860783)  --> TIME: 2025-02-26 13:48:14 -- STEP: 9308/36938 -- GLOBAL_STEP: 1265200 | > loss_text_ce: 0.026904704049229622 (0.031182669993044945) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1370493931016239) | > loader_time: 0.0045 (0.004690326791901812)  --> TIME: 2025-02-26 13:48:24 -- STEP: 9358/36938 -- GLOBAL_STEP: 1265250 | > loss_text_ce: 0.03593331575393677 (0.031180766667692202) | > loss_mel_ce: 2.885303020477295 (nan) | > loss: 0.011592208407819271 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.1370495938164324) | > loader_time: 0.0047 (0.0046897626478566365)  --> TIME: 2025-02-26 13:48:33 -- STEP: 9408/36938 -- GLOBAL_STEP: 1265300 | > loss_text_ce: 0.04163236916065216 (0.031181788331746663) | > loss_mel_ce: 2.351283550262451 (nan) | > loss: 0.009495698846876621 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.1370719769069938) | > loader_time: 0.0045 (0.004689549710474839)  --> TIME: 2025-02-26 13:48:43 -- STEP: 9458/36938 -- GLOBAL_STEP: 1265350 | > loss_text_ce: 0.04062088951468468 (0.031188880396681145) | > loss_mel_ce: 2.695359706878662 (nan) | > loss: 0.01085706613957882 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13706344644623536) | > loader_time: 0.0046 (0.004689258333239635)  --> TIME: 2025-02-26 13:48:52 -- STEP: 9508/36938 -- GLOBAL_STEP: 1265400 | > loss_text_ce: 0.02744910679757595 (0.03119005114278927) | > loss_mel_ce: 2.5099432468414307 (nan) | > loss: 0.010069018229842186 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13704051259790678) | > loader_time: 0.0045 (0.00468913915531444)  --> TIME: 2025-02-26 13:49:01 -- STEP: 9558/36938 -- GLOBAL_STEP: 1265450 | > loss_text_ce: 0.036248620599508286 (0.031195697025030545) | > loss_mel_ce: 2.225266456604004 (nan) | > loss: 0.008974267169833183 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1370329576497496) | > loader_time: 0.0045 (0.004688675020947853)  --> TIME: 2025-02-26 13:49:10 -- STEP: 9608/36938 -- GLOBAL_STEP: 1265500 | > loss_text_ce: 0.03706413879990578 (0.031186535077531687) | > loss_mel_ce: 1.78857421875 (nan) | > loss: 0.007244597189128399 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.137016266286621) | > loader_time: 0.0045 (0.0046882640809242405)  --> TIME: 2025-02-26 13:49:19 -- STEP: 9658/36938 -- GLOBAL_STEP: 1265550 | > loss_text_ce: 0.023531228303909302 (0.031188566197476696) | > loss_mel_ce: 1.770021677017212 (nan) | > loss: 0.00711727375164628 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13697356579342723) | > loader_time: 0.0045 (0.004687614854657378)  --> TIME: 2025-02-26 13:49:28 -- STEP: 9708/36938 -- GLOBAL_STEP: 1265600 | > loss_text_ce: 0.03062400035560131 (0.031184026455145217) | > loss_mel_ce: 2.9205617904663086 (nan) | > loss: 0.011711055412888527 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.1369723980447416) | > loader_time: 0.0046 (0.004687947061850677)  --> TIME: 2025-02-26 13:49:38 -- STEP: 9758/36938 -- GLOBAL_STEP: 1265650 | > loss_text_ce: 0.03392620012164116 (0.031181256999274095) | > loss_mel_ce: 1.7289018630981445 (nan) | > loss: 0.006995350122451782 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13700739556488656) | > loader_time: 0.0046 (0.00468805012680462)  --> TIME: 2025-02-26 13:49:47 -- STEP: 9808/36938 -- GLOBAL_STEP: 1265700 | > loss_text_ce: 0.03334323689341545 (0.031190546452911767) | > loss_mel_ce: 2.725924491882324 (nan) | > loss: 0.010949475690722466 (nan) | > current_lr: 5e-06 | > step_time: 0.196 (0.13701118589614364) | > loader_time: 0.0048 (0.004688030549402515)  --> TIME: 2025-02-26 13:49:57 -- STEP: 9858/36938 -- GLOBAL_STEP: 1265750 | > loss_text_ce: 0.037048861384391785 (0.031196578265126028) | > loss_mel_ce: 2.184816837310791 (nan) | > loss: 0.008816927671432495 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.1369907359705417) | > loader_time: 0.0045 (0.00468784656551744)  --> TIME: 2025-02-26 13:50:06 -- STEP: 9908/36938 -- GLOBAL_STEP: 1265800 | > loss_text_ce: 0.040346648544073105 (0.031203014162028062) | > loss_mel_ce: 2.7449605464935303 (nan) | > loss: 0.011052806861698627 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13697330627799464) | > loader_time: 0.0046 (0.004687657965543419)  --> TIME: 2025-02-26 13:50:15 -- STEP: 9958/36938 -- GLOBAL_STEP: 1265850 | > loss_text_ce: 0.027528412640094757 (0.031199889585355595) | > loss_mel_ce: 3.102555990219116 (nan) | > loss: 0.01242097094655037 (nan) | > current_lr: 5e-06 | > step_time: 0.1757 (0.1369774765124594) | > loader_time: 0.0048 (0.004687511458840309)  --> TIME: 2025-02-26 13:50:24 -- STEP: 10008/36938 -- GLOBAL_STEP: 1265900 | > loss_text_ce: 0.03454141691327095 (0.031204917282852497) | > loss_mel_ce: 2.4104461669921875 (nan) | > loss: 0.009702332317829132 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1369540548915392) | > loader_time: 0.0046 (0.004687127116010423)  --> TIME: 2025-02-26 13:50:34 -- STEP: 10058/36938 -- GLOBAL_STEP: 1265950 | > loss_text_ce: 0.022196361795067787 (0.03120271013594082) | > loss_mel_ce: 1.91064453125 (nan) | > loss: 0.007670004386454821 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13695612127251375) | > loader_time: 0.0045 (0.004686908921988714)  --> TIME: 2025-02-26 13:50:43 -- STEP: 10108/36938 -- GLOBAL_STEP: 1266000 | > loss_text_ce: 0.030942101031541824 (0.031199233825467788) | > loss_mel_ce: 2.831368923187256 (nan) | > loss: 0.011358378455042839 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13694603180385279) | > loader_time: 0.0045 (0.00468656098743553)  --> TIME: 2025-02-26 13:50:52 -- STEP: 10158/36938 -- GLOBAL_STEP: 1266050 | > loss_text_ce: 0.03760932385921478 (0.03119545329175487) | > loss_mel_ce: 2.736236572265625 (nan) | > loss: 0.01100732572376728 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13694892919633916) | > loader_time: 0.0046 (0.00468638039968406)  --> TIME: 2025-02-26 13:51:01 -- STEP: 10208/36938 -- GLOBAL_STEP: 1266100 | > loss_text_ce: 0.021161353215575218 (0.031186916248388978) | > loss_mel_ce: 2.741529703140259 (nan) | > loss: 0.010963059961795807 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.1369475794950248) | > loader_time: 0.0048 (0.00468608241843578)  --> TIME: 2025-02-26 13:51:11 -- STEP: 10258/36938 -- GLOBAL_STEP: 1266150 | > loss_text_ce: 0.03311436250805855 (0.03118830909614784) | > loss_mel_ce: 2.959451198577881 (nan) | > loss: 0.011875261552631855 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13693751357270612) | > loader_time: 0.0046 (0.004685762635586336)  --> TIME: 2025-02-26 13:51:20 -- STEP: 10308/36938 -- GLOBAL_STEP: 1266200 | > loss_text_ce: 0.03206751495599747 (0.031194059159350113) | > loss_mel_ce: 1.9744185209274292 (nan) | > loss: 0.007962246425449848 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13693950691563597) | > loader_time: 0.0047 (0.0046854845349715455)  --> TIME: 2025-02-26 13:51:30 -- STEP: 10358/36938 -- GLOBAL_STEP: 1266250 | > loss_text_ce: 0.02555636130273342 (0.03119374830126519) | > loss_mel_ce: 3.1117255687713623 (nan) | > loss: 0.012449531815946102 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13694834633208658) | > loader_time: 0.0046 (0.004685417246371988)  --> TIME: 2025-02-26 13:51:39 -- STEP: 10408/36938 -- GLOBAL_STEP: 1266300 | > loss_text_ce: 0.02572232484817505 (0.03119294571051007) | > loss_mel_ce: 2.776907444000244 (nan) | > loss: 0.011121546849608421 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13695120062395558) | > loader_time: 0.0046 (0.0046851287705452875)  --> TIME: 2025-02-26 13:51:48 -- STEP: 10458/36938 -- GLOBAL_STEP: 1266350 | > loss_text_ce: 0.02711629495024681 (0.03119212117645984) | > loss_mel_ce: 2.4365365505218506 (nan) | > loss: 0.009776400402188301 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.13694147578980073) | > loader_time: 0.0047 (0.004684904379013906)  --> TIME: 2025-02-26 13:51:58 -- STEP: 10508/36938 -- GLOBAL_STEP: 1266400 | > loss_text_ce: 0.023295091465115547 (0.031187780065476893) | > loss_mel_ce: 1.9621179103851318 (nan) | > loss: 0.007878622971475124 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1369689917401027) | > loader_time: 0.0045 (0.004684748057861665)  --> TIME: 2025-02-26 13:52:07 -- STEP: 10558/36938 -- GLOBAL_STEP: 1266450 | > loss_text_ce: 0.031538818031549454 (0.031184607090862123) | > loss_mel_ce: 2.2357895374298096 (nan) | > loss: 0.008997335098683834 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13695031270133542) | > loader_time: 0.0045 (0.004684277727062386)  --> TIME: 2025-02-26 13:52:16 -- STEP: 10608/36938 -- GLOBAL_STEP: 1266500 | > loss_text_ce: 0.044092271476984024 (0.031177918839589373) | > loss_mel_ce: 2.760216236114502 (nan) | > loss: 0.011128208599984646 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13693556128853562) | > loader_time: 0.0045 (0.004683963898260296)  --> TIME: 2025-02-26 13:52:26 -- STEP: 10658/36938 -- GLOBAL_STEP: 1266550 | > loss_text_ce: 0.024157200008630753 (0.031177294621973554) | > loss_mel_ce: 2.7068698406219482 (nan) | > loss: 0.010837409645318985 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1369317958582672) | > loader_time: 0.0046 (0.004683741800197616)  --> TIME: 2025-02-26 13:52:35 -- STEP: 10708/36938 -- GLOBAL_STEP: 1266600 | > loss_text_ce: 0.03918854519724846 (0.031177854133602193) | > loss_mel_ce: 2.9355580806732178 (nan) | > loss: 0.011804550886154175 (nan) | > current_lr: 5e-06 | > step_time: 0.156 (0.13692355743753018) | > loader_time: 0.0047 (0.004683493098350512)  --> TIME: 2025-02-26 13:52:44 -- STEP: 10758/36938 -- GLOBAL_STEP: 1266650 | > loss_text_ce: 0.032665446400642395 (0.03117676388934951) | > loss_mel_ce: 2.8374123573303223 (nan) | > loss: 0.011389198713004589 (nan) | > current_lr: 5e-06 | > step_time: 0.1443 (0.13691486364227373) | > loader_time: 0.0047 (0.004683263484907306)  --> TIME: 2025-02-26 13:52:53 -- STEP: 10808/36938 -- GLOBAL_STEP: 1266700 | > loss_text_ce: 0.036177393049001694 (0.031175311997589383) | > loss_mel_ce: 0.6200343370437622 (nan) | > loss: 0.002604014938697219 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13690367510899343) | > loader_time: 0.0044 (0.0046831050420319065)  --> TIME: 2025-02-26 13:53:03 -- STEP: 10858/36938 -- GLOBAL_STEP: 1266750 | > loss_text_ce: 0.024157201871275902 (0.03117111652796524) | > loss_mel_ce: 3.239654064178467 (nan) | > loss: 0.012951632961630821 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13690316082035142) | > loader_time: 0.0046 (0.004682729841047138)  --> TIME: 2025-02-26 13:53:12 -- STEP: 10908/36938 -- GLOBAL_STEP: 1266800 | > loss_text_ce: 0.03144821897149086 (0.031170543406854028) | > loss_mel_ce: 3.2658791542053223 (nan) | > loss: 0.013084633275866508 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.13691247523727096) | > loader_time: 0.0049 (0.004683085684816511)  --> TIME: 2025-02-26 13:53:22 -- STEP: 10958/36938 -- GLOBAL_STEP: 1266850 | > loss_text_ce: 0.03722988814115524 (0.03117072081729107) | > loss_mel_ce: 1.9924629926681519 (nan) | > loss: 0.008054337464272976 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13692686855412234) | > loader_time: 0.0045 (0.004682967884302518)  --> TIME: 2025-02-26 13:53:31 -- STEP: 11008/36938 -- GLOBAL_STEP: 1266900 | > loss_text_ce: 0.030458709225058556 (0.031172252848062833) | > loss_mel_ce: 2.957679271697998 (nan) | > loss: 0.01185769122093916 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13692976781275387) | > loader_time: 0.0046 (0.00468268572504436)  --> TIME: 2025-02-26 13:53:40 -- STEP: 11058/36938 -- GLOBAL_STEP: 1266950 | > loss_text_ce: 0.024298841133713722 (0.031167649703147834) | > loss_mel_ce: 2.0304794311523438 (nan) | > loss: 0.008153882808983326 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13692446566982064) | > loader_time: 0.0045 (0.004682379533033242)  --> TIME: 2025-02-26 13:53:50 -- STEP: 11108/36938 -- GLOBAL_STEP: 1267000 | > loss_text_ce: 0.03706106171011925 (0.03117051407073377) | > loss_mel_ce: 1.8625744581222534 (nan) | > loss: 0.007538236677646637 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13692644138665874) | > loader_time: 0.0046 (0.004682615415458402)  --> TIME: 2025-02-26 13:53:59 -- STEP: 11158/36938 -- GLOBAL_STEP: 1267050 | > loss_text_ce: 0.025309717282652855 (0.031171485654382755) | > loss_mel_ce: 3.0926716327667236 (nan) | > loss: 0.012372942641377449 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.136927939845918) | > loader_time: 0.0048 (0.004682424013322756)  --> TIME: 2025-02-26 13:54:09 -- STEP: 11208/36938 -- GLOBAL_STEP: 1267100 | > loss_text_ce: 0.022919608280062675 (0.031172193200935717) | > loss_mel_ce: 2.8562498092651367 (nan) | > loss: 0.011425276286900043 (nan) | > current_lr: 5e-06 | > step_time: 0.1966 (0.13693576374707458) | > loader_time: 0.0049 (0.004682344849155575)  --> TIME: 2025-02-26 13:54:18 -- STEP: 11258/36938 -- GLOBAL_STEP: 1267150 | > loss_text_ce: 0.02876368537545204 (0.031175148577394122) | > loss_mel_ce: 2.854450225830078 (nan) | > loss: 0.011441325768828392 (nan) | > current_lr: 5e-06 | > step_time: 0.1572 (0.1369449676385592) | > loader_time: 0.0046 (0.0046820567500889644)  --> TIME: 2025-02-26 13:54:27 -- STEP: 11308/36938 -- GLOBAL_STEP: 1267200 | > loss_text_ce: 0.02649506740272045 (0.031169303953463137) | > loss_mel_ce: 2.30609130859375 (nan) | > loss: 0.009256294928491116 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13694249095113903) | > loader_time: 0.0045 (0.00468189652243559)  --> TIME: 2025-02-26 13:54:37 -- STEP: 11358/36938 -- GLOBAL_STEP: 1267250 | > loss_text_ce: 0.03690391033887863 (0.031173827434603157) | > loss_mel_ce: 2.73726749420166 (nan) | > loss: 0.011008617468178272 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13694855438023548) | > loader_time: 0.0047 (0.004681654349244947)  --> TIME: 2025-02-26 13:54:46 -- STEP: 11408/36938 -- GLOBAL_STEP: 1267300 | > loss_text_ce: 0.03605318069458008 (0.03117715566076841) | > loss_mel_ce: 3.0520005226135254 (nan) | > loss: 0.012254182249307632 (nan) | > current_lr: 5e-06 | > step_time: 0.1923 (0.13694973315045716) | > loader_time: 0.0048 (0.004681299394845605)  --> TIME: 2025-02-26 13:54:56 -- STEP: 11458/36938 -- GLOBAL_STEP: 1267350 | > loss_text_ce: 0.033434685319662094 (0.031177021186395013) | > loss_mel_ce: 3.344670057296753 (nan) | > loss: 0.013405177742242813 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13695563021953047) | > loader_time: 0.0047 (0.004681005467917392)  --> TIME: 2025-02-26 13:55:05 -- STEP: 11508/36938 -- GLOBAL_STEP: 1267400 | > loss_text_ce: 0.02672860026359558 (0.031179003294157234) | > loss_mel_ce: 1.505116581916809 (nan) | > loss: 0.006078751292079687 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.13693857091476794) | > loader_time: 0.0048 (0.0046807043785264775)  --> TIME: 2025-02-26 13:55:14 -- STEP: 11558/36938 -- GLOBAL_STEP: 1267450 | > loss_text_ce: 0.03458468243479729 (0.031174440048879974) | > loss_mel_ce: 2.944190502166748 (nan) | > loss: 0.01182053703814745 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.1369599642112489) | > loader_time: 0.0046 (0.004680429327589008)  --> TIME: 2025-02-26 13:55:23 -- STEP: 11608/36938 -- GLOBAL_STEP: 1267500 | > loss_text_ce: 0.031138082966208458 (0.031175147830688376) | > loss_mel_ce: 2.698082208633423 (nan) | > loss: 0.010830240324139595 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13692383863530008) | > loader_time: 0.0044 (0.00468028096968677)  --> TIME: 2025-02-26 13:55:32 -- STEP: 11658/36938 -- GLOBAL_STEP: 1267550 | > loss_text_ce: 0.027866793796420097 (0.0311745563591483) | > loss_mel_ce: 1.7388222217559814 (nan) | > loss: 0.0070106713101267815 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13691339024059382) | > loader_time: 0.0045 (0.004679791819764998)  --> TIME: 2025-02-26 13:55:42 -- STEP: 11708/36938 -- GLOBAL_STEP: 1267600 | > loss_text_ce: 0.025744659826159477 (0.031175247191581636) | > loss_mel_ce: 3.0166265964508057 (nan) | > loss: 0.012072902172803879 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13692203914504236) | > loader_time: 0.0046 (0.00467941259661752)  --> TIME: 2025-02-26 13:55:51 -- STEP: 11758/36938 -- GLOBAL_STEP: 1267650 | > loss_text_ce: 0.03174830973148346 (0.031178682400865753) | > loss_mel_ce: 3.305403232574463 (nan) | > loss: 0.01324266567826271 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.13693231880553353) | > loader_time: 0.0046 (0.0046792546184842645)  --> TIME: 2025-02-26 13:56:01 -- STEP: 11808/36938 -- GLOBAL_STEP: 1267700 | > loss_text_ce: 0.03189636021852493 (0.03117737362972401) | > loss_mel_ce: 3.28289532661438 (nan) | > loss: 0.01315393578261137 (nan) | > current_lr: 5e-06 | > step_time: 0.179 (0.1369521032140513) | > loader_time: 0.0048 (0.0046790439665801995)  --> TIME: 2025-02-26 13:56:10 -- STEP: 11858/36938 -- GLOBAL_STEP: 1267750 | > loss_text_ce: 0.026920262724161148 (0.03117910151965672) | > loss_mel_ce: 2.6400673389434814 (nan) | > loss: 0.010583285242319107 (nan) | > current_lr: 5e-06 | > step_time: 0.1465 (0.1369424540631832) | > loader_time: 0.0046 (0.0046784714515262565)  --> TIME: 2025-02-26 13:56:19 -- STEP: 11908/36938 -- GLOBAL_STEP: 1267800 | > loss_text_ce: 0.0331641286611557 (0.031174101728741382) | > loss_mel_ce: 2.748643398284912 (nan) | > loss: 0.011038918979465961 (nan) | > current_lr: 5e-06 | > step_time: 0.1533 (0.136938140502023) | > loader_time: 0.0046 (0.004678027558574976)  --> TIME: 2025-02-26 13:56:28 -- STEP: 11958/36938 -- GLOBAL_STEP: 1267850 | > loss_text_ce: 0.030212122946977615 (0.03117907040597173) | > loss_mel_ce: 2.653759241104126 (nan) | > loss: 0.010650680400431156 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13691336673166235) | > loader_time: 0.0046 (0.004677600337422624)  --> TIME: 2025-02-26 13:56:38 -- STEP: 12008/36938 -- GLOBAL_STEP: 1267900 | > loss_text_ce: 0.030502362176775932 (0.031180917564981182) | > loss_mel_ce: 2.579439163208008 (nan) | > loss: 0.010356911458075047 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.1369104248376631) | > loader_time: 0.0046 (0.004677287623693261)  --> TIME: 2025-02-26 13:56:47 -- STEP: 12058/36938 -- GLOBAL_STEP: 1267950 | > loss_text_ce: 0.027835121378302574 (0.03118292785348978) | > loss_mel_ce: 2.5379345417022705 (nan) | > loss: 0.010181626304984093 (nan) | > current_lr: 5e-06 | > step_time: 0.1712 (0.13693554756514506) | > loader_time: 0.0048 (0.004677193618409951)  --> TIME: 2025-02-26 13:56:56 -- STEP: 12108/36938 -- GLOBAL_STEP: 1268000 | > loss_text_ce: 0.023614060133695602 (0.031179933146711143) | > loss_mel_ce: 2.0755107402801514 (nan) | > loss: 0.008329860866069794 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13691758987412955) | > loader_time: 0.0045 (0.0046768983795889285)  --> TIME: 2025-02-26 13:57:06 -- STEP: 12158/36938 -- GLOBAL_STEP: 1268050 | > loss_text_ce: 0.023580212146043777 (0.03117908572448413) | > loss_mel_ce: 2.551143169403076 (nan) | > loss: 0.010217157192528248 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.1369138161472087) | > loader_time: 0.0047 (0.004676598136921922)  --> TIME: 2025-02-26 13:57:15 -- STEP: 12208/36938 -- GLOBAL_STEP: 1268100 | > loss_text_ce: 0.03603591024875641 (0.03118252893474972) | > loss_mel_ce: 1.023231863975525 (nan) | > loss: 0.004203443881124258 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13689517816662988) | > loader_time: 0.0045 (0.004676271762166692)  --> TIME: 2025-02-26 13:57:24 -- STEP: 12258/36938 -- GLOBAL_STEP: 1268150 | > loss_text_ce: 0.026267223060131073 (0.031181604183798818) | > loss_mel_ce: 2.302616596221924 (nan) | > loss: 0.009241603314876556 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13686215583837405) | > loader_time: 0.0045 (0.004675904851419561)  --> TIME: 2025-02-26 13:57:33 -- STEP: 12308/36938 -- GLOBAL_STEP: 1268200 | > loss_text_ce: 0.036166947335004807 (0.031178613522413582) | > loss_mel_ce: 2.120551109313965 (nan) | > loss: 0.008558405563235283 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.1368618177096879) | > loader_time: 0.0046 (0.004675669777807866)  --> TIME: 2025-02-26 13:57:42 -- STEP: 12358/36938 -- GLOBAL_STEP: 1268250 | > loss_text_ce: 0.03523477539420128 (0.031179041785525198) | > loss_mel_ce: 3.0675160884857178 (nan) | > loss: 0.012312503531575203 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.13686131537780577) | > loader_time: 0.0048 (0.0046754633460081)  --> TIME: 2025-02-26 13:57:51 -- STEP: 12408/36938 -- GLOBAL_STEP: 1268300 | > loss_text_ce: 0.029532888904213905 (0.031182306318551067) | > loss_mel_ce: 1.646650791168213 (nan) | > loss: 0.006651523057371378 (nan) | > current_lr: 5e-06 | > step_time: 0.1262 (0.1368481698803257) | > loader_time: 0.0046 (0.004675447345164259)  --> TIME: 2025-02-26 13:58:00 -- STEP: 12458/36938 -- GLOBAL_STEP: 1268350 | > loss_text_ce: 0.021936070173978806 (0.031180286383114885) | > loss_mel_ce: 2.841548204421997 (nan) | > loss: 0.011363034136593342 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13682202662065182) | > loader_time: 0.0046 (0.0046755263387802766)  --> TIME: 2025-02-26 13:58:09 -- STEP: 12508/36938 -- GLOBAL_STEP: 1268400 | > loss_text_ce: 0.030436614528298378 (0.031178389482547426) | > loss_mel_ce: 2.8870084285736084 (nan) | > loss: 0.011577162891626358 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.1368088756348712) | > loader_time: 0.0047 (0.004675386239440055)  --> TIME: 2025-02-26 13:58:18 -- STEP: 12558/36938 -- GLOBAL_STEP: 1268450 | > loss_text_ce: 0.026151522994041443 (0.03117751902168661) | > loss_mel_ce: 2.771617889404297 (nan) | > loss: 0.01110226009041071 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13680017155852198) | > loader_time: 0.0047 (0.004675268462403298)  --> TIME: 2025-02-26 13:58:28 -- STEP: 12608/36938 -- GLOBAL_STEP: 1268500 | > loss_text_ce: 0.02764294110238552 (0.031172871708809075) | > loss_mel_ce: 3.158447265625 (nan) | > loss: 0.01264321617782116 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.13680832846288782) | > loader_time: 0.0046 (0.0046752076879673235)  --> TIME: 2025-02-26 13:58:37 -- STEP: 12658/36938 -- GLOBAL_STEP: 1268550 | > loss_text_ce: 0.028702788054943085 (0.031176993784139042) | > loss_mel_ce: 3.353058338165283 (nan) | > loss: 0.013419687747955322 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.1368038864153828) | > loader_time: 0.0047 (0.004675126203825397)  --> TIME: 2025-02-26 13:58:47 -- STEP: 12708/36938 -- GLOBAL_STEP: 1268600 | > loss_text_ce: 0.03097890317440033 (0.031174382697840706) | > loss_mel_ce: 3.116834878921509 (nan) | > loss: 0.012491324916481972 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.13680928161319145) | > loader_time: 0.0048 (0.004675010558678585)  --> TIME: 2025-02-26 13:58:56 -- STEP: 12758/36938 -- GLOBAL_STEP: 1268650 | > loss_text_ce: 0.03127496317028999 (0.031174065751131783) | > loss_mel_ce: 2.8247084617614746 (nan) | > loss: 0.011333268135786057 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.13682704118374941) | > loader_time: 0.0046 (0.004674999630550241)  --> TIME: 2025-02-26 13:59:05 -- STEP: 12808/36938 -- GLOBAL_STEP: 1268700 | > loss_text_ce: 0.0411943681538105 (0.031176737782273155) | > loss_mel_ce: 0.00396728515625 (nan) | > loss: 0.00017921291873790324 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.1368273536649667) | > loader_time: 0.0046 (0.004674728682903402)  --> TIME: 2025-02-26 13:59:15 -- STEP: 12858/36938 -- GLOBAL_STEP: 1268750 | > loss_text_ce: 0.02272571250796318 (0.031179324285063346) | > loss_mel_ce: 2.916508436203003 (nan) | > loss: 0.011663628742098808 (nan) | > current_lr: 5e-06 | > step_time: 0.2051 (0.1368387050475874) | > loader_time: 0.0049 (0.004674508794504772)  --> TIME: 2025-02-26 13:59:24 -- STEP: 12908/36938 -- GLOBAL_STEP: 1268800 | > loss_text_ce: 0.03368774428963661 (0.03117866217788468) | > loss_mel_ce: 2.521209239959717 (nan) | > loss: 0.010138480924069881 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13684618087017333) | > loader_time: 0.0045 (0.004674238393208461)  --> TIME: 2025-02-26 13:59:34 -- STEP: 12958/36938 -- GLOBAL_STEP: 1268850 | > loss_text_ce: 0.02406047284603119 (0.03117250145822224) | > loss_mel_ce: 2.5245845317840576 (nan) | > loss: 0.0101136714220047 (nan) | > current_lr: 5e-06 | > step_time: 0.1457 (0.1368385877372271) | > loader_time: 0.0046 (0.004674064872036337)  --> TIME: 2025-02-26 13:59:43 -- STEP: 13008/36938 -- GLOBAL_STEP: 1268900 | > loss_text_ce: 0.03662344068288803 (0.031178445955993315) | > loss_mel_ce: 2.718486785888672 (nan) | > loss: 0.010932978242635727 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13682808117599662) | > loader_time: 0.0045 (0.004673830697457282)  --> TIME: 2025-02-26 13:59:52 -- STEP: 13058/36938 -- GLOBAL_STEP: 1268950 | > loss_text_ce: 0.035084404051303864 (0.03117962254573338) | > loss_mel_ce: 2.4537575244903564 (nan) | > loss: 0.009876357391476631 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.1368268682062398) | > loader_time: 0.0045 (0.004673553966493582)  --> TIME: 2025-02-26 14:00:02 -- STEP: 13108/36938 -- GLOBAL_STEP: 1269000 | > loss_text_ce: 0.03155000880360603 (0.03117982001900151) | > loss_mel_ce: 2.078392505645752 (nan) | > loss: 0.008372788317501545 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13684834788621913) | > loader_time: 0.0046 (0.004673470838207417)  --> TIME: 2025-02-26 14:00:11 -- STEP: 13158/36938 -- GLOBAL_STEP: 1269050 | > loss_text_ce: 0.028537621721625328 (0.031176909556968262) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13683644959513122) | > loader_time: 0.0046 (0.004673139377465847)  --> TIME: 2025-02-26 14:00:20 -- STEP: 13208/36938 -- GLOBAL_STEP: 1269100 | > loss_text_ce: 0.02234845794737339 (0.031176038013293397) | > loss_mel_ce: 2.449476718902588 (nan) | > loss: 0.009808829985558987 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.1368239274319558) | > loader_time: 0.0045 (0.0046727279509146315)  --> TIME: 2025-02-26 14:00:30 -- STEP: 13258/36938 -- GLOBAL_STEP: 1269150 | > loss_text_ce: 0.033279869705438614 (0.03117823848281119) | > loss_mel_ce: 3.149315595626831 (nan) | > loss: 0.01262934785336256 (nan) | > current_lr: 5e-06 | > step_time: 0.1908 (0.1368529583681653) | > loader_time: 0.0049 (0.004672707898491986)  --> TIME: 2025-02-26 14:00:39 -- STEP: 13308/36938 -- GLOBAL_STEP: 1269200 | > loss_text_ce: 0.03635621815919876 (0.031181419967022332) | > loss_mel_ce: 2.3020260334014893 (nan) | > loss: 0.00927929487079382 (nan) | > current_lr: 5e-06 | > step_time: 0.1075 (0.1368564044637367) | > loader_time: 0.0044 (0.004672485337373472)  --> TIME: 2025-02-26 14:00:49 -- STEP: 13358/36938 -- GLOBAL_STEP: 1269250 | > loss_text_ce: 0.030076226219534874 (0.031181473921575047) | > loss_mel_ce: 2.6313745975494385 (nan) | > loss: 0.010561313480138779 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13685274281211188) | > loader_time: 0.0046 (0.00467243216355094)  --> TIME: 2025-02-26 14:00:58 -- STEP: 13408/36938 -- GLOBAL_STEP: 1269300 | > loss_text_ce: 0.033208105713129044 (0.031181460873600048) | > loss_mel_ce: 2.8131816387176514 (nan) | > loss: 0.0112951984629035 (nan) | > current_lr: 5e-06 | > step_time: 0.1583 (0.13687135309407694) | > loader_time: 0.0047 (0.00467231595857615)  --> TIME: 2025-02-26 14:01:08 -- STEP: 13458/36938 -- GLOBAL_STEP: 1269350 | > loss_text_ce: 0.027187835425138474 (0.031185112503765672) | > loss_mel_ce: 2.6022205352783203 (nan) | > loss: 0.010434160940349102 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13687382544782395) | > loader_time: 0.0046 (0.0046722367926291496)  --> TIME: 2025-02-26 14:01:17 -- STEP: 13508/36938 -- GLOBAL_STEP: 1269400 | > loss_text_ce: 0.044552505016326904 (0.031184774234067455) | > loss_mel_ce: 2.298335552215576 (nan) | > loss: 0.009297175332903862 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13687870124405133) | > loader_time: 0.0045 (0.004672120476672721)  --> TIME: 2025-02-26 14:01:26 -- STEP: 13558/36938 -- GLOBAL_STEP: 1269450 | > loss_text_ce: 0.036482494324445724 (0.031186551523526368) | > loss_mel_ce: 3.5793838500976562 (nan) | > loss: 0.014348677359521389 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.1368854599446386) | > loader_time: 0.0045 (0.004671914877485669)  --> TIME: 2025-02-26 14:01:36 -- STEP: 13608/36938 -- GLOBAL_STEP: 1269500 | > loss_text_ce: 0.03459256887435913 (0.031189308157872346) | > loss_mel_ce: 2.864894390106201 (nan) | > loss: 0.01150590181350708 (nan) | > current_lr: 5e-06 | > step_time: 0.1268 (0.13688098408417568) | > loader_time: 0.0046 (0.004671665393626384)  --> TIME: 2025-02-26 14:01:45 -- STEP: 13658/36938 -- GLOBAL_STEP: 1269550 | > loss_text_ce: 0.03740117326378822 (0.031188964582655108) | > loss_mel_ce: 1.8725495338439941 (nan) | > loss: 0.0075791701674461365 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13687552952246196) | > loader_time: 0.0045 (0.004671365140489533)  --> TIME: 2025-02-26 14:01:54 -- STEP: 13708/36938 -- GLOBAL_STEP: 1269600 | > loss_text_ce: 0.019799865782260895 (0.0311890586560197) | > loss_mel_ce: 1.5951184034347534 (nan) | > loss: 0.00640840595588088 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13687636925661406) | > loader_time: 0.0045 (0.004671277563670019)  --> TIME: 2025-02-26 14:02:04 -- STEP: 13758/36938 -- GLOBAL_STEP: 1269650 | > loss_text_ce: 0.028524838387966156 (0.031189927409631432) | > loss_mel_ce: 2.6058976650238037 (nan) | > loss: 0.010454058647155762 (nan) | > current_lr: 5e-06 | > step_time: 0.1867 (0.1368779844973629) | > loader_time: 0.0047 (0.004671201679591847)  --> TIME: 2025-02-26 14:02:13 -- STEP: 13808/36938 -- GLOBAL_STEP: 1269700 | > loss_text_ce: 0.028990570455789566 (0.03118911020514132) | > loss_mel_ce: 2.7292962074279785 (nan) | > loss: 0.010945582762360573 (nan) | > current_lr: 5e-06 | > step_time: 0.1705 (0.1368828635675742) | > loader_time: 0.0048 (0.004671128796950962)  --> TIME: 2025-02-26 14:02:22 -- STEP: 13858/36938 -- GLOBAL_STEP: 1269750 | > loss_text_ce: 0.03632907196879387 (0.031192513517774665) | > loss_mel_ce: 2.742515802383423 (nan) | > loss: 0.01102716289460659 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13686802793605474) | > loader_time: 0.0048 (0.004670859793939529)  --> TIME: 2025-02-26 14:02:31 -- STEP: 13908/36938 -- GLOBAL_STEP: 1269800 | > loss_text_ce: 0.04881177842617035 (0.031193871226181547) | > loss_mel_ce: 1.5604451894760132 (nan) | > loss: 0.006385940592736006 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13686245931850824) | > loader_time: 0.0044 (0.004670517126444871)  --> TIME: 2025-02-26 14:02:41 -- STEP: 13958/36938 -- GLOBAL_STEP: 1269850 | > loss_text_ce: 0.0274520106613636 (0.03119371224496088) | > loss_mel_ce: 2.811084270477295 (nan) | > loss: 0.011264033615589142 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13685961332647742) | > loader_time: 0.0046 (0.004670208958162411)  --> TIME: 2025-02-26 14:02:50 -- STEP: 14008/36938 -- GLOBAL_STEP: 1269900 | > loss_text_ce: 0.03209396079182625 (0.031194202195752477) | > loss_mel_ce: 2.792633533477783 (nan) | > loss: 0.011209236457943916 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.1368596512954073) | > loader_time: 0.0048 (0.004670257639163288)  --> TIME: 2025-02-26 14:02:59 -- STEP: 14058/36938 -- GLOBAL_STEP: 1269950 | > loss_text_ce: 0.02678261697292328 (0.031192404170782825) | > loss_mel_ce: 2.018847703933716 (nan) | > loss: 0.00811758078634739 (nan) | > current_lr: 5e-06 | > step_time: 0.1121 (0.13686350177507614) | > loader_time: 0.0045 (0.0046704368174779855)  --> TIME: 2025-02-26 14:03:09 -- STEP: 14108/36938 -- GLOBAL_STEP: 1270000 | > loss_text_ce: 0.03554513677954674 (0.031187451336237626) | > loss_mel_ce: 3.0008296966552734 (nan) | > loss: 0.012049106881022453 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.13686970465237977) | > loader_time: 0.0048 (0.004670342220812566) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1270000.pth  --> TIME: 2025-02-26 14:03:25 -- STEP: 14158/36938 -- GLOBAL_STEP: 1270050 | > loss_text_ce: 0.028762584552168846 (0.031186221472470108) | > loss_mel_ce: 2.5684149265289307 (nan) | > loss: 0.010306260548532009 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13686729972856565) | > loader_time: 0.0045 (0.004670359452138896)  --> TIME: 2025-02-26 14:03:34 -- STEP: 14208/36938 -- GLOBAL_STEP: 1270100 | > loss_text_ce: 0.03920789808034897 (0.031186063659284585) | > loss_mel_ce: 2.13348388671875 (nan) | > loss: 0.008621793240308762 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13686171257952287) | > loader_time: 0.0046 (0.0046701317334228755)  --> TIME: 2025-02-26 14:03:43 -- STEP: 14258/36938 -- GLOBAL_STEP: 1270150 | > loss_text_ce: 0.024870289489626884 (0.031182324645803304) | > loss_mel_ce: 2.1954970359802246 (nan) | > loss: 0.00881098210811615 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.1368505557601088) | > loader_time: 0.0046 (0.00467006184305979)  --> TIME: 2025-02-26 14:03:52 -- STEP: 14308/36938 -- GLOBAL_STEP: 1270200 | > loss_text_ce: 0.029520859941840172 (0.031182956842050538) | > loss_mel_ce: 1.5904221534729004 (nan) | > loss: 0.006428345572203398 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1368495997084162) | > loader_time: 0.0045 (0.004669951549414847)  --> TIME: 2025-02-26 14:04:01 -- STEP: 14358/36938 -- GLOBAL_STEP: 1270250 | > loss_text_ce: 0.02938195876777172 (0.031183723043826828) | > loss_mel_ce: 2.951447010040283 (nan) | > loss: 0.01182868704199791 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13684464311314104) | > loader_time: 0.0046 (0.0046697472908284595)  --> TIME: 2025-02-26 14:04:10 -- STEP: 14408/36938 -- GLOBAL_STEP: 1270300 | > loss_text_ce: 0.018562713637948036 (0.031177726245422165) | > loss_mel_ce: 1.0637348890304565 (nan) | > loss: 0.004294831771403551 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13682955817869133) | > loader_time: 0.0046 (0.004669712756091785)  --> TIME: 2025-02-26 14:04:19 -- STEP: 14458/36938 -- GLOBAL_STEP: 1270350 | > loss_text_ce: 0.026071913540363312 (0.03118007596336622) | > loss_mel_ce: 0.007940634153783321 (nan) | > loss: 0.00013497043983079493 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13682213065732732) | > loader_time: 0.0045 (0.004669585355272393)  --> TIME: 2025-02-26 14:04:29 -- STEP: 14508/36938 -- GLOBAL_STEP: 1270400 | > loss_text_ce: 0.029712798073887825 (0.03117641207719952) | > loss_mel_ce: 1.3773525953292847 (nan) | > loss: 0.005583593156188726 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13684059231281934) | > loader_time: 0.0045 (0.004669536481325635)  --> TIME: 2025-02-26 14:04:38 -- STEP: 14558/36938 -- GLOBAL_STEP: 1270450 | > loss_text_ce: 0.034647539258003235 (0.031174538166138493) | > loss_mel_ce: 2.9956324100494385 (nan) | > loss: 0.01202492043375969 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13684619288635153) | > loader_time: 0.0046 (0.004669352946233607)  --> TIME: 2025-02-26 14:04:48 -- STEP: 14608/36938 -- GLOBAL_STEP: 1270500 | > loss_text_ce: 0.02965736947953701 (0.031175846399151323) | > loss_mel_ce: 0.8943704962730408 (nan) | > loss: 0.0036667773965746164 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13684824784770863) | > loader_time: 0.0045 (0.00466921116018347)  --> TIME: 2025-02-26 14:04:57 -- STEP: 14658/36938 -- GLOBAL_STEP: 1270550 | > loss_text_ce: 0.034574154764413834 (0.031174604429422836) | > loss_mel_ce: 2.5385162830352783 (nan) | > loss: 0.010210677050054073 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.136850024557875) | > loader_time: 0.0049 (0.004669079759108249)  --> TIME: 2025-02-26 14:05:06 -- STEP: 14708/36938 -- GLOBAL_STEP: 1270600 | > loss_text_ce: 0.031585197895765305 (0.03117377817171566) | > loss_mel_ce: 2.6724276542663574 (nan) | > loss: 0.010730210691690445 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.1368462706604244) | > loader_time: 0.0047 (0.004668960922724657)  --> TIME: 2025-02-26 14:05:15 -- STEP: 14758/36938 -- GLOBAL_STEP: 1270650 | > loss_text_ce: 0.033146198838949203 (0.031175786419175264) | > loss_mel_ce: 2.62648344039917 (nan) | > loss: 0.010554086416959763 (nan) | > current_lr: 5e-06 | > step_time: 0.1271 (0.13683412060179956) | > loader_time: 0.0046 (0.004668783698887787)  --> TIME: 2025-02-26 14:05:25 -- STEP: 14808/36938 -- GLOBAL_STEP: 1270700 | > loss_text_ce: 0.015359090641140938 (0.031170317187750853) | > loss_mel_ce: 1.2079263925552368 (nan) | > loss: 0.004854307509958744 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13684037540295158) | > loader_time: 0.0044 (0.0046686842788045005)  --> TIME: 2025-02-26 14:05:34 -- STEP: 14858/36938 -- GLOBAL_STEP: 1270750 | > loss_text_ce: 0.03496021032333374 (0.03117307545860428) | > loss_mel_ce: 2.328977584838867 (nan) | > loss: 0.009380706585943699 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.1368448958452755) | > loader_time: 0.0049 (0.004668522689846032)  --> TIME: 2025-02-26 14:05:43 -- STEP: 14908/36938 -- GLOBAL_STEP: 1270800 | > loss_text_ce: 0.03688334673643112 (0.031169684307633748) | > loss_mel_ce: 1.427241325378418 (nan) | > loss: 0.005810019094496965 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13684312077671476) | > loader_time: 0.005 (0.0046694554270577185)  --> TIME: 2025-02-26 14:05:53 -- STEP: 14958/36938 -- GLOBAL_STEP: 1270850 | > loss_text_ce: 0.03263978660106659 (0.03117389636495785) | > loss_mel_ce: 2.508460760116577 (nan) | > loss: 0.01008373312652111 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13686207182698454) | > loader_time: 0.0047 (0.0046704305431265435)  --> TIME: 2025-02-26 14:06:02 -- STEP: 15008/36938 -- GLOBAL_STEP: 1270900 | > loss_text_ce: 0.032625000923871994 (0.031172893588929614) | > loss_mel_ce: 3.2337729930877686 (nan) | > loss: 0.012961897067725658 (nan) | > current_lr: 5e-06 | > step_time: 0.1759 (0.13686603550781345) | > loader_time: 0.0051 (0.004670766069055361)  --> TIME: 2025-02-26 14:06:11 -- STEP: 15058/36938 -- GLOBAL_STEP: 1270950 | > loss_text_ce: 0.03205326944589615 (0.03117056263998886) | > loss_mel_ce: 2.6360011100769043 (nan) | > loss: 0.010587518103420734 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13684475268322052) | > loader_time: 0.0045 (0.004670959463282902)  --> TIME: 2025-02-26 14:06:21 -- STEP: 15108/36938 -- GLOBAL_STEP: 1271000 | > loss_text_ce: 0.024962691590189934 (0.03116741527306143) | > loss_mel_ce: 2.676044225692749 (nan) | > loss: 0.01071828231215477 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13685838141578982) | > loader_time: 0.0045 (0.004671190777309737)  --> TIME: 2025-02-26 14:06:30 -- STEP: 15158/36938 -- GLOBAL_STEP: 1271050 | > loss_text_ce: 0.03193860873579979 (0.03116789375399764) | > loss_mel_ce: 1.7849445343017578 (nan) | > loss: 0.007209853734821081 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13684631620835713) | > loader_time: 0.0045 (0.004671388336812805)  --> TIME: 2025-02-26 14:06:40 -- STEP: 15208/36938 -- GLOBAL_STEP: 1271100 | > loss_text_ce: 0.02415219321846962 (0.031168387490883184) | > loss_mel_ce: 2.8817272186279297 (nan) | > loss: 0.011531268246471882 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13687052411947825) | > loader_time: 0.0049 (0.004671589284743335)  --> TIME: 2025-02-26 14:06:49 -- STEP: 15258/36938 -- GLOBAL_STEP: 1271150 | > loss_text_ce: 0.03087335079908371 (0.0311704659582416) | > loss_mel_ce: 2.989459276199341 (nan) | > loss: 0.011985447257757187 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.1368697183850024) | > loader_time: 0.0046 (0.004671606937512948)  --> TIME: 2025-02-26 14:06:58 -- STEP: 15308/36938 -- GLOBAL_STEP: 1271200 | > loss_text_ce: 0.036295827478170395 (0.03117249097835944) | > loss_mel_ce: 2.335148811340332 (nan) | > loss: 0.009410494938492775 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.1368549554130571) | > loader_time: 0.0046 (0.0046716972870150105)  --> TIME: 2025-02-26 14:07:08 -- STEP: 15358/36938 -- GLOBAL_STEP: 1271250 | > loss_text_ce: 0.03963889926671982 (0.03117462881883336) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13685378717528573) | > loader_time: 0.0045 (0.004671667932082404)  --> TIME: 2025-02-26 14:07:17 -- STEP: 15408/36938 -- GLOBAL_STEP: 1271300 | > loss_text_ce: 0.028356732800602913 (0.03117507653508427) | > loss_mel_ce: 2.55562162399292 (nan) | > loss: 0.010253882966935635 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.1368579991850405) | > loader_time: 0.0045 (0.004671540943014035)  --> TIME: 2025-02-26 14:07:27 -- STEP: 15458/36938 -- GLOBAL_STEP: 1271350 | > loss_text_ce: 0.02886098064482212 (0.03117280803288967) | > loss_mel_ce: 2.891542673110962 (nan) | > loss: 0.011588904075324535 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.13687626315144177) | > loader_time: 0.0047 (0.004671432065352965)  --> TIME: 2025-02-26 14:07:36 -- STEP: 15508/36938 -- GLOBAL_STEP: 1271400 | > loss_text_ce: 0.02417696826159954 (0.03117242252624957) | > loss_mel_ce: 2.821847677230835 (nan) | > loss: 0.011293749324977398 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.1368690043157423) | > loader_time: 0.0046 (0.0046714407314736375)  --> TIME: 2025-02-26 14:07:45 -- STEP: 15558/36938 -- GLOBAL_STEP: 1271450 | > loss_text_ce: 0.028684722259640694 (0.03117229076289623) | > loss_mel_ce: 2.9889214038848877 (nan) | > loss: 0.011974628083407879 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.13686538996158915) | > loader_time: 0.0047 (0.004671431243657173)  --> TIME: 2025-02-26 14:07:55 -- STEP: 15608/36938 -- GLOBAL_STEP: 1271500 | > loss_text_ce: 0.027168644592165947 (0.03117565803144124) | > loss_mel_ce: 1.8121551275253296 (nan) | > loss: 0.007298904005438089 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13689137585098374) | > loader_time: 0.0045 (0.00467161323277538)  --> TIME: 2025-02-26 14:08:04 -- STEP: 15658/36938 -- GLOBAL_STEP: 1271550 | > loss_text_ce: 0.02488502487540245 (0.031175949127562157) | > loss_mel_ce: 1.7293294668197632 (nan) | > loss: 0.006961169186979532 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13689721961453277) | > loader_time: 0.0045 (0.00467180943851468)  --> TIME: 2025-02-26 14:08:14 -- STEP: 15708/36938 -- GLOBAL_STEP: 1271600 | > loss_text_ce: 0.03740089014172554 (0.03117331236546093) | > loss_mel_ce: 2.6280202865600586 (nan) | > loss: 0.010577068664133549 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.13690881813213024) | > loader_time: 0.0047 (0.0046718059862300475)  --> TIME: 2025-02-26 14:08:24 -- STEP: 15758/36938 -- GLOBAL_STEP: 1271650 | > loss_text_ce: 0.028896450996398926 (0.031174611834981058) | > loss_mel_ce: 2.925802230834961 (nan) | > loss: 0.011724994517862797 (nan) | > current_lr: 5e-06 | > step_time: 0.182 (0.13691575832212038) | > loader_time: 0.0048 (0.004672434354497172)  --> TIME: 2025-02-26 14:08:33 -- STEP: 15808/36938 -- GLOBAL_STEP: 1271700 | > loss_text_ce: 0.03909534588456154 (0.031174591035833817) | > loss_mel_ce: 2.2129688262939453 (nan) | > loss: 0.008936763741075993 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.1369281562143253) | > loader_time: 0.0045 (0.0046725022768684685)  --> TIME: 2025-02-26 14:08:42 -- STEP: 15858/36938 -- GLOBAL_STEP: 1271750 | > loss_text_ce: 0.03398551791906357 (0.031172817621109006) | > loss_mel_ce: 2.601125478744507 (nan) | > loss: 0.010456790216267109 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13692883629413427) | > loader_time: 0.0046 (0.00467263584796079)  --> TIME: 2025-02-26 14:08:52 -- STEP: 15908/36938 -- GLOBAL_STEP: 1271800 | > loss_text_ce: 0.03121369704604149 (0.03117242844802016) | > loss_mel_ce: 2.634084701538086 (nan) | > loss: 0.010576581582427025 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.13692560476485852) | > loader_time: 0.0047 (0.004672532468876824)  --> TIME: 2025-02-26 14:09:01 -- STEP: 15958/36938 -- GLOBAL_STEP: 1271850 | > loss_text_ce: 0.024698659777641296 (0.03117044359925999) | > loss_mel_ce: 2.807464599609375 (nan) | > loss: 0.011238744482398033 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13693409084691094) | > loader_time: 0.0049 (0.0046725364119236485)  --> TIME: 2025-02-26 14:09:11 -- STEP: 16008/36938 -- GLOBAL_STEP: 1271900 | > loss_text_ce: 0.038932137191295624 (0.031171170968459047) | > loss_mel_ce: 3.1111841201782227 (nan) | > loss: 0.012500462122261524 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13694944087116223) | > loader_time: 0.0046 (0.00467254116438675)  --> TIME: 2025-02-26 14:09:20 -- STEP: 16058/36938 -- GLOBAL_STEP: 1271950 | > loss_text_ce: 0.03431330993771553 (0.03117173245281765) | > loss_mel_ce: 3.0303561687469482 (nan) | > loss: 0.012161387130618095 (nan) | > current_lr: 5e-06 | > step_time: 0.1887 (0.13693845463594914) | > loader_time: 0.0049 (0.004672312056503073)  --> TIME: 2025-02-26 14:09:29 -- STEP: 16108/36938 -- GLOBAL_STEP: 1272000 | > loss_text_ce: 0.025215350091457367 (0.031169909429751477) | > loss_mel_ce: 2.158177614212036 (nan) | > loss: 0.008664258755743504 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13694025641610225) | > loader_time: 0.0045 (0.0046722184554915365)  --> TIME: 2025-02-26 14:09:39 -- STEP: 16158/36938 -- GLOBAL_STEP: 1272050 | > loss_text_ce: 0.03196800872683525 (0.03117101796153301) | > loss_mel_ce: 2.6389060020446777 (nan) | > loss: 0.010598707012832165 (nan) | > current_lr: 5e-06 | > step_time: 0.1867 (0.136936214569551) | > loader_time: 0.0048 (0.004672340155090486)  --> TIME: 2025-02-26 14:09:48 -- STEP: 16208/36938 -- GLOBAL_STEP: 1272100 | > loss_text_ce: 0.028296729549765587 (0.031170410155408324) | > loss_mel_ce: 2.762042760848999 (nan) | > loss: 0.011072776280343533 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.13694073368683998) | > loader_time: 0.0046 (0.004672368593065308)  --> TIME: 2025-02-26 14:09:57 -- STEP: 16258/36938 -- GLOBAL_STEP: 1272150 | > loss_text_ce: 0.028302298858761787 (0.031164612363921276) | > loss_mel_ce: 2.8703725337982178 (nan) | > loss: 0.011502678506076336 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13693874029527245) | > loader_time: 0.0046 (0.004672284436586638)  --> TIME: 2025-02-26 14:10:06 -- STEP: 16308/36938 -- GLOBAL_STEP: 1272200 | > loss_text_ce: 0.03995886445045471 (0.03116518734812213) | > loss_mel_ce: 2.4325358867645264 (nan) | > loss: 0.00981148798018694 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13692710833541338) | > loader_time: 0.0045 (0.004672370633571378)  --> TIME: 2025-02-26 14:10:16 -- STEP: 16358/36938 -- GLOBAL_STEP: 1272250 | > loss_text_ce: 0.03311320021748543 (0.031166541775961135) | > loss_mel_ce: 0.00775222759693861 (nan) | > loss: 0.00016216440417338163 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13695305521002282) | > loader_time: 0.0045 (0.0046723321825356936)  --> TIME: 2025-02-26 14:10:25 -- STEP: 16408/36938 -- GLOBAL_STEP: 1272300 | > loss_text_ce: 0.0322536900639534 (0.031166703018236337) | > loss_mel_ce: 2.492215633392334 (nan) | > loss: 0.01001773588359356 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13694941821707474) | > loader_time: 0.0046 (0.004672235116791221)  --> TIME: 2025-02-26 14:10:35 -- STEP: 16458/36938 -- GLOBAL_STEP: 1272350 | > loss_text_ce: 0.031589213758707047 (0.031168120708189816) | > loss_mel_ce: 2.9097049236297607 (nan) | > loss: 0.011671802960336208 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13693992550368175) | > loader_time: 0.0048 (0.004672225574224998)  --> TIME: 2025-02-26 14:10:44 -- STEP: 16508/36938 -- GLOBAL_STEP: 1272400 | > loss_text_ce: 0.03382125124335289 (0.03117063640617286) | > loss_mel_ce: 2.652573585510254 (nan) | > loss: 0.01066029816865921 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13694490863880024) | > loader_time: 0.0046 (0.004672194786617125)  --> TIME: 2025-02-26 14:10:53 -- STEP: 16558/36938 -- GLOBAL_STEP: 1272450 | > loss_text_ce: 0.030993998050689697 (0.031168524190562572) | > loss_mel_ce: 1.9294273853302002 (nan) | > loss: 0.0077794501557946205 (nan) | > current_lr: 5e-06 | > step_time: 0.1514 (0.1369493228684506) | > loader_time: 0.0049 (0.004672264041985872)  --> TIME: 2025-02-26 14:11:02 -- STEP: 16608/36938 -- GLOBAL_STEP: 1272500 | > loss_text_ce: 0.027083134278655052 (0.03116876183961239) | > loss_mel_ce: 3.0569963455200195 (nan) | > loss: 0.012238411232829094 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.1369288841079427) | > loader_time: 0.0045 (0.004672307973307676)  --> TIME: 2025-02-26 14:11:12 -- STEP: 16658/36938 -- GLOBAL_STEP: 1272550 | > loss_text_ce: 0.03029053471982479 (0.03117125617397905) | > loss_mel_ce: 3.202526569366455 (nan) | > loss: 0.012828640639781952 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13692240064378022) | > loader_time: 0.0045 (0.004672133503218971)  --> TIME: 2025-02-26 14:11:21 -- STEP: 16708/36938 -- GLOBAL_STEP: 1272600 | > loss_text_ce: 0.03587224707007408 (0.031176785828241575) | > loss_mel_ce: 1.7250909805297852 (nan) | > loss: 0.006987949833273888 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13691941837617214) | > loader_time: 0.0045 (0.004672016285811135)  --> TIME: 2025-02-26 14:11:30 -- STEP: 16758/36938 -- GLOBAL_STEP: 1272650 | > loss_text_ce: 0.023745501413941383 (0.031176650774792893) | > loss_mel_ce: 2.3517417907714844 (nan) | > loss: 0.009426537901163101 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13691179846818152) | > loader_time: 0.0047 (0.004671856688064552)  --> TIME: 2025-02-26 14:11:39 -- STEP: 16808/36938 -- GLOBAL_STEP: 1272700 | > loss_text_ce: 0.03822212666273117 (0.03117978765288132) | > loss_mel_ce: 1.917661190032959 (nan) | > loss: 0.007761442102491856 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13691565422771232) | > loader_time: 0.0047 (0.004671752608883439)  --> TIME: 2025-02-26 14:11:48 -- STEP: 16858/36938 -- GLOBAL_STEP: 1272750 | > loss_text_ce: 0.037490468472242355 (0.03117994959984912) | > loss_mel_ce: 1.4989514350891113 (nan) | > loss: 0.006096992176026106 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.13690340893734473) | > loader_time: 0.0045 (0.004671676810320364)  --> TIME: 2025-02-26 14:11:58 -- STEP: 16908/36938 -- GLOBAL_STEP: 1272800 | > loss_text_ce: 0.03035525046288967 (0.031180308452418055) | > loss_mel_ce: 3.128117561340332 (nan) | > loss: 0.012533622793853283 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.1368906040098023) | > loader_time: 0.0048 (0.004671572722355353)  --> TIME: 2025-02-26 14:12:07 -- STEP: 16958/36938 -- GLOBAL_STEP: 1272850 | > loss_text_ce: 0.03167625144124031 (0.031179066930308474) | > loss_mel_ce: 3.210880994796753 (nan) | > loss: 0.012867291457951069 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13688910581098607) | > loader_time: 0.0048 (0.0046715343711314005)  --> TIME: 2025-02-26 14:12:16 -- STEP: 17008/36938 -- GLOBAL_STEP: 1272900 | > loss_text_ce: 0.03974118083715439 (0.03117907384624276) | > loss_mel_ce: 1.2317465543746948 (nan) | > loss: 0.0050455862656235695 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13688016767672315) | > loader_time: 0.0046 (0.004671599964768055)  --> TIME: 2025-02-26 14:12:25 -- STEP: 17058/36938 -- GLOBAL_STEP: 1272950 | > loss_text_ce: 0.03482941910624504 (0.031181503474487) | > loss_mel_ce: 0.12451171875 (nan) | > loss: 0.0006323061534203589 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13688012157043938) | > loader_time: 0.0045 (0.004671503390834259)  --> TIME: 2025-02-26 14:12:34 -- STEP: 17108/36938 -- GLOBAL_STEP: 1273000 | > loss_text_ce: 0.03677329793572426 (0.031181473799288515) | > loss_mel_ce: 2.6412532329559326 (nan) | > loss: 0.010627089999616146 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.136865600682264) | > loader_time: 0.0045 (0.004671233904983271)  --> TIME: 2025-02-26 14:12:44 -- STEP: 17158/36938 -- GLOBAL_STEP: 1273050 | > loss_text_ce: 0.02626745216548443 (0.0311783966429767) | > loss_mel_ce: 3.2627110481262207 (nan) | > loss: 0.013051503337919712 (nan) | > current_lr: 5e-06 | > step_time: 0.1352 (0.13687092672348714) | > loader_time: 0.0047 (0.004671220735509237)  --> TIME: 2025-02-26 14:12:53 -- STEP: 17208/36938 -- GLOBAL_STEP: 1273100 | > loss_text_ce: 0.025194844231009483 (0.0311782477023158) | > loss_mel_ce: 2.8130710124969482 (nan) | > loss: 0.011262960731983185 (nan) | > current_lr: 5e-06 | > step_time: 0.1566 (0.1368863135368424) | > loader_time: 0.0046 (0.0046711237499293765)  --> TIME: 2025-02-26 14:13:02 -- STEP: 17258/36938 -- GLOBAL_STEP: 1273150 | > loss_text_ce: 0.025608709082007408 (0.031175781925055504) | > loss_mel_ce: 2.508608818054199 (nan) | > loss: 0.010056419298052788 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1368713757712455) | > loader_time: 0.0046 (0.004671042453705299)  --> TIME: 2025-02-26 14:13:11 -- STEP: 17308/36938 -- GLOBAL_STEP: 1273200 | > loss_text_ce: 0.03900376334786415 (0.031174377979226855) | > loss_mel_ce: 2.124199390411377 (nan) | > loss: 0.008584140799939632 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13684994417606697) | > loader_time: 0.0045 (0.004670981408157326)  --> TIME: 2025-02-26 14:13:21 -- STEP: 17358/36938 -- GLOBAL_STEP: 1273250 | > loss_text_ce: 0.02599574252963066 (0.03117514689542978) | > loss_mel_ce: 2.3208677768707275 (nan) | > loss: 0.009312951005995274 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13686344181012028) | > loader_time: 0.0047 (0.004671122459431836)  --> TIME: 2025-02-26 14:13:31 -- STEP: 17408/36938 -- GLOBAL_STEP: 1273300 | > loss_text_ce: 0.02459622733294964 (0.03117558819303668) | > loss_mel_ce: 2.8346354961395264 (nan) | > loss: 0.01134615857154131 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13688724967377175) | > loader_time: 0.0045 (0.004671130219803134)  --> TIME: 2025-02-26 14:13:40 -- STEP: 17458/36938 -- GLOBAL_STEP: 1273350 | > loss_text_ce: 0.0389200784265995 (0.031178879916105885) | > loss_mel_ce: 3.0847251415252686 (nan) | > loss: 0.012395418249070644 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13687438061589974) | > loader_time: 0.0045 (0.004670907861384235)  --> TIME: 2025-02-26 14:13:49 -- STEP: 17508/36938 -- GLOBAL_STEP: 1273400 | > loss_text_ce: 0.028309792280197144 (0.031177573453650728) | > loss_mel_ce: 3.0541625022888184 (nan) | > loss: 0.012232033535838127 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13688455169702596) | > loader_time: 0.0046 (0.004670861147570504)  --> TIME: 2025-02-26 14:13:59 -- STEP: 17558/36938 -- GLOBAL_STEP: 1273450 | > loss_text_ce: 0.04333605244755745 (0.03117934454172835) | > loss_mel_ce: 2.4903910160064697 (nan) | > loss: 0.010054473765194416 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1368843901332549) | > loader_time: 0.0045 (0.004670671279317138)  --> TIME: 2025-02-26 14:14:08 -- STEP: 17608/36938 -- GLOBAL_STEP: 1273500 | > loss_text_ce: 0.030046837404370308 (0.031181671024661524) | > loss_mel_ce: 2.9948019981384277 (nan) | > loss: 0.012003369629383087 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.1368965664705004) | > loader_time: 0.0049 (0.004670751292507301)  --> TIME: 2025-02-26 14:14:17 -- STEP: 17658/36938 -- GLOBAL_STEP: 1273550 | > loss_text_ce: 0.03880727291107178 (0.031183812458197673) | > loss_mel_ce: 2.8458969593048096 (nan) | > loss: 0.0114472396671772 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13688018875843644) | > loader_time: 0.0045 (0.004670653233564318)  --> TIME: 2025-02-26 14:14:27 -- STEP: 17708/36938 -- GLOBAL_STEP: 1273600 | > loss_text_ce: 0.02401137538254261 (0.03118166523344558) | > loss_mel_ce: 2.808443307876587 (nan) | > loss: 0.011239900253713131 (nan) | > current_lr: 5e-06 | > step_time: 0.1751 (0.13688443272275927) | > loader_time: 0.0048 (0.004670730799349052)  --> TIME: 2025-02-26 14:14:36 -- STEP: 17758/36938 -- GLOBAL_STEP: 1273650 | > loss_text_ce: 0.02332169935107231 (0.03118035444830997) | > loss_mel_ce: 1.4470510482788086 (nan) | > loss: 0.005834813229739666 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.13688522737726272) | > loader_time: 0.0046 (0.004670636411426917)  --> TIME: 2025-02-26 14:14:45 -- STEP: 17808/36938 -- GLOBAL_STEP: 1273700 | > loss_text_ce: 0.036163218319416046 (0.031183390053715447) | > loss_mel_ce: 1.800933837890625 (nan) | > loss: 0.007290068082511425 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1368839533636407) | > loader_time: 0.0045 (0.0046706203662481545)  --> TIME: 2025-02-26 14:14:55 -- STEP: 17858/36938 -- GLOBAL_STEP: 1273750 | > loss_text_ce: 0.029994186013936996 (0.03117922118796427) | > loss_mel_ce: 2.8235113620758057 (nan) | > loss: 0.011323435232043266 (nan) | > current_lr: 5e-06 | > step_time: 0.1576 (0.1368788842698845) | > loader_time: 0.0046 (0.004670785621320932)  --> TIME: 2025-02-26 14:15:04 -- STEP: 17908/36938 -- GLOBAL_STEP: 1273800 | > loss_text_ce: 0.027452442795038223 (0.031177820240652708) | > loss_mel_ce: 0.0147705078125 (nan) | > loss: 0.0001675514067756012 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13688771685292925) | > loader_time: 0.0046 (0.004670664112243576)  --> TIME: 2025-02-26 14:15:13 -- STEP: 17958/36938 -- GLOBAL_STEP: 1273850 | > loss_text_ce: 0.03115442767739296 (0.031178481235586985) | > loss_mel_ce: 2.622312068939209 (nan) | > loss: 0.01052962988615036 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.13687643776201247) | > loader_time: 0.0047 (0.004670515863915852)  --> TIME: 2025-02-26 14:15:23 -- STEP: 18008/36938 -- GLOBAL_STEP: 1273900 | > loss_text_ce: 0.04255925863981247 (0.03117987169090391) | > loss_mel_ce: 2.4799461364746094 (nan) | > loss: 0.010009942576289177 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13687641765741937) | > loader_time: 0.0044 (0.0046703647052590246)  --> TIME: 2025-02-26 14:15:32 -- STEP: 18058/36938 -- GLOBAL_STEP: 1273950 | > loss_text_ce: 0.040739692747592926 (0.031180083959624683) | > loss_mel_ce: 2.1380856037139893 (nan) | > loss: 0.008646133355796337 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.13687919078442026) | > loader_time: 0.0048 (0.004670191344555743)  --> TIME: 2025-02-26 14:15:41 -- STEP: 18108/36938 -- GLOBAL_STEP: 1274000 | > loss_text_ce: 0.033595357090234756 (0.031183134364192563) | > loss_mel_ce: 2.3907554149627686 (nan) | > loss: 0.009620440192520618 (nan) | > current_lr: 5e-06 | > step_time: 0.1606 (0.13687026570484256) | > loader_time: 0.0047 (0.00466997537334838)  --> TIME: 2025-02-26 14:15:51 -- STEP: 18158/36938 -- GLOBAL_STEP: 1274050 | > loss_text_ce: 0.03403514623641968 (0.03118536254845093) | > loss_mel_ce: 2.6549673080444336 (nan) | > loss: 0.010670645162463188 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.1368705393367567) | > loader_time: 0.0045 (0.004669727070082874)  --> TIME: 2025-02-26 14:16:00 -- STEP: 18208/36938 -- GLOBAL_STEP: 1274100 | > loss_text_ce: 0.027485433965921402 (0.031180947738321526) | > loss_mel_ce: 2.5582027435302734 (nan) | > loss: 0.010260667651891708 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.13688084337801434) | > loader_time: 0.0046 (0.004669588419276494)  --> TIME: 2025-02-26 14:16:09 -- STEP: 18258/36938 -- GLOBAL_STEP: 1274150 | > loss_text_ce: 0.026917925104498863 (0.031187590610928434) | > loss_mel_ce: 2.8716766834259033 (nan) | > loss: 0.011502359993755817 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13688677757642195) | > loader_time: 0.0046 (0.0046694492612118395)  --> TIME: 2025-02-26 14:16:19 -- STEP: 18308/36938 -- GLOBAL_STEP: 1274200 | > loss_text_ce: 0.03650495409965515 (0.03118695620071369) | > loss_mel_ce: 2.521648645401001 (nan) | > loss: 0.010151403956115246 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13687589270385667) | > loader_time: 0.0045 (0.004669357341060409)  --> TIME: 2025-02-26 14:16:28 -- STEP: 18358/36938 -- GLOBAL_STEP: 1274250 | > loss_text_ce: 0.035099923610687256 (0.031186454089709312) | > loss_mel_ce: 0.2605966627597809 (nan) | > loss: 0.0011733992723748088 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.1368757852841493) | > loader_time: 0.0046 (0.004669303415595385)  --> TIME: 2025-02-26 14:16:37 -- STEP: 18408/36938 -- GLOBAL_STEP: 1274300 | > loss_text_ce: 0.03661146014928818 (0.031183939529335968) | > loss_mel_ce: 2.4453227519989014 (nan) | > loss: 0.009848945774137974 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13687568483327783) | > loader_time: 0.0046 (0.004669253539127235)  --> TIME: 2025-02-26 14:16:47 -- STEP: 18458/36938 -- GLOBAL_STEP: 1274350 | > loss_text_ce: 0.028906792402267456 (0.031182926416502856) | > loss_mel_ce: 2.4647562503814697 (nan) | > loss: 0.009895488619804382 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.1368841435956029) | > loader_time: 0.0046 (0.004669193315252801)  --> TIME: 2025-02-26 14:16:56 -- STEP: 18508/36938 -- GLOBAL_STEP: 1274400 | > loss_text_ce: 0.026956507936120033 (0.03118378324089973) | > loss_mel_ce: 2.8563480377197266 (nan) | > loss: 0.011441685259342194 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13687955779237818) | > loader_time: 0.0046 (0.004669167115875359)  --> TIME: 2025-02-26 14:17:05 -- STEP: 18558/36938 -- GLOBAL_STEP: 1274450 | > loss_text_ce: 0.03997525945305824 (0.031184486119912286) | > loss_mel_ce: 2.4482500553131104 (nan) | > loss: 0.009873909875750542 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.13688669927753053) | > loader_time: 0.0046 (0.0046690460396861195)  --> TIME: 2025-02-26 14:17:15 -- STEP: 18608/36938 -- GLOBAL_STEP: 1274500 | > loss_text_ce: 0.03953415900468826 (0.03118189044590233) | > loss_mel_ce: 2.449528932571411 (nan) | > loss: 0.009877234697341919 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13688958995524694) | > loader_time: 0.0046 (0.004668972495805556)  --> TIME: 2025-02-26 14:17:24 -- STEP: 18658/36938 -- GLOBAL_STEP: 1274550 | > loss_text_ce: 0.022412622347474098 (0.031183073787063188) | > loss_mel_ce: 2.820570945739746 (nan) | > loss: 0.01128168124705553 (nan) | > current_lr: 5e-06 | > step_time: 0.1626 (0.13689345070909142) | > loader_time: 0.0046 (0.004668920622056963)  --> TIME: 2025-02-26 14:17:34 -- STEP: 18708/36938 -- GLOBAL_STEP: 1274600 | > loss_text_ce: 0.031223317608237267 (0.031181386545115698) | > loss_mel_ce: 2.801884651184082 (nan) | > loss: 0.011242492124438286 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.1368876374224093) | > loader_time: 0.0046 (0.004668917886869795)  --> TIME: 2025-02-26 14:17:43 -- STEP: 18758/36938 -- GLOBAL_STEP: 1274650 | > loss_text_ce: 0.028996137902140617 (0.03117856138006214) | > loss_mel_ce: 2.830261707305908 (nan) | > loss: 0.011346261948347092 (nan) | > current_lr: 5e-06 | > step_time: 0.1447 (0.13687501197456592) | > loader_time: 0.0046 (0.00466867419292783)  --> TIME: 2025-02-26 14:17:52 -- STEP: 18808/36938 -- GLOBAL_STEP: 1274700 | > loss_text_ce: 0.030036484822630882 (0.031176682559160108) | > loss_mel_ce: 2.4672064781188965 (nan) | > loss: 0.009909695014357567 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13687929650511124) | > loader_time: 0.0046 (0.0046685321160652585)  --> TIME: 2025-02-26 14:18:01 -- STEP: 18858/36938 -- GLOBAL_STEP: 1274750 | > loss_text_ce: 0.027074867859482765 (0.03117658513402477) | > loss_mel_ce: 2.643475294113159 (nan) | > loss: 0.010597421787679195 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.13687289653723034) | > loader_time: 0.0047 (0.004668432931174817)  --> TIME: 2025-02-26 14:18:11 -- STEP: 18908/36938 -- GLOBAL_STEP: 1274800 | > loss_text_ce: 0.029646696522831917 (0.031176777096935723) | > loss_mel_ce: 3.1734068393707275 (nan) | > loss: 0.012710530310869217 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.1368738194462205) | > loader_time: 0.0047 (0.00466837680236415)  --> TIME: 2025-02-26 14:18:20 -- STEP: 18958/36938 -- GLOBAL_STEP: 1274850 | > loss_text_ce: 0.029211977496743202 (0.03117663306104039) | > loss_mel_ce: 1.5625510215759277 (nan) | > loss: 0.006316520273685455 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1368675525963889) | > loader_time: 0.0045 (0.00466840600952315)  --> TIME: 2025-02-26 14:18:29 -- STEP: 19008/36938 -- GLOBAL_STEP: 1274900 | > loss_text_ce: 0.03136363998055458 (0.03117861290307879) | > loss_mel_ce: 0.9135165214538574 (nan) | > loss: 0.0037495247088372707 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.1368600753847795) | > loader_time: 0.0045 (0.004668231652140208)  --> TIME: 2025-02-26 14:18:38 -- STEP: 19058/36938 -- GLOBAL_STEP: 1274950 | > loss_text_ce: 0.03467841446399689 (0.03117996789882505) | > loss_mel_ce: 2.5198845863342285 (nan) | > loss: 0.010137155652046204 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13685108245713865) | > loader_time: 0.0046 (0.004668050415806882)  --> TIME: 2025-02-26 14:18:47 -- STEP: 19108/36938 -- GLOBAL_STEP: 1275000 | > loss_text_ce: 0.02936510182917118 (0.03117925652394628) | > loss_mel_ce: 2.100051164627075 (nan) | > loss: 0.008450064808130264 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13684884356933305) | > loader_time: 0.0047 (0.004668078301265838)  --> TIME: 2025-02-26 14:18:57 -- STEP: 19158/36938 -- GLOBAL_STEP: 1275050 | > loss_text_ce: 0.025748388841748238 (0.031179157678986418) | > loss_mel_ce: 2.907519817352295 (nan) | > loss: 0.011639954522252083 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13684312334537532) | > loader_time: 0.0046 (0.004667985326052324)  --> TIME: 2025-02-26 14:19:06 -- STEP: 19208/36938 -- GLOBAL_STEP: 1275100 | > loss_text_ce: 0.03414030373096466 (0.03117979492133764) | > loss_mel_ce: 1.994124174118042 (nan) | > loss: 0.008048669435083866 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13685743502654996) | > loader_time: 0.0046 (0.0046682110407708826)  --> TIME: 2025-02-26 14:19:16 -- STEP: 19258/36938 -- GLOBAL_STEP: 1275150 | > loss_text_ce: 0.0314674936234951 (0.031179831418198047) | > loss_mel_ce: 2.297427177429199 (nan) | > loss: 0.009241646155714989 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13686129896796) | > loader_time: 0.0045 (0.004668180067747939)  --> TIME: 2025-02-26 14:19:25 -- STEP: 19308/36938 -- GLOBAL_STEP: 1275200 | > loss_text_ce: 0.03251427784562111 (0.031177145588903027) | > loss_mel_ce: 2.722141981124878 (nan) | > loss: 0.010931176133453846 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13684504484572155) | > loader_time: 0.0045 (0.00466804440677943)  --> TIME: 2025-02-26 14:19:34 -- STEP: 19358/36938 -- GLOBAL_STEP: 1275250 | > loss_text_ce: 0.03013836406171322 (0.031178112712763127) | > loss_mel_ce: 0.4032244086265564 (nan) | > loss: 0.0017196936532855034 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.1368427943291649) | > loader_time: 0.0045 (0.004667843973258508)  --> TIME: 2025-02-26 14:19:43 -- STEP: 19408/36938 -- GLOBAL_STEP: 1275300 | > loss_text_ce: 0.02697412483394146 (0.031179915190642123) | > loss_mel_ce: 2.841937303543091 (nan) | > loss: 0.01138457003980875 (nan) | > current_lr: 5e-06 | > step_time: 0.1488 (0.1368473719955962) | > loader_time: 0.0047 (0.004667862340721917)  --> TIME: 2025-02-26 14:19:53 -- STEP: 19458/36938 -- GLOBAL_STEP: 1275350 | > loss_text_ce: 0.03841618821024895 (0.03118119641584417) | > loss_mel_ce: 6.395833492279053 (nan) | > loss: 0.025532739236950874 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.1368392028327294) | > loader_time: 0.0047 (0.004667856781734368)  --> TIME: 2025-02-26 14:20:02 -- STEP: 19508/36938 -- GLOBAL_STEP: 1275400 | > loss_text_ce: 0.03087659738957882 (0.0311802468827479) | > loss_mel_ce: 2.220170021057129 (nan) | > loss: 0.00893272552639246 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1368312209581824) | > loader_time: 0.0045 (0.004667840166269859)  --> TIME: 2025-02-26 14:20:11 -- STEP: 19558/36938 -- GLOBAL_STEP: 1275450 | > loss_text_ce: 0.022712720558047295 (0.031178398945998633) | > loss_mel_ce: 0.021677840501070023 (nan) | > loss: 0.00017615302931517363 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1368320732226898) | > loader_time: 0.0046 (0.004667660699737663)  --> TIME: 2025-02-26 14:20:21 -- STEP: 19608/36938 -- GLOBAL_STEP: 1275500 | > loss_text_ce: 0.03004910796880722 (0.03118104854695916) | > loss_mel_ce: 2.9681739807128906 (nan) | > loss: 0.011897711083292961 (nan) | > current_lr: 5e-06 | > step_time: 0.163 (0.1368417680287836) | > loader_time: 0.0048 (0.004667657241681208)  --> TIME: 2025-02-26 14:20:30 -- STEP: 19658/36938 -- GLOBAL_STEP: 1275550 | > loss_text_ce: 0.031037403270602226 (0.031183689531469105) | > loss_mel_ce: 3.0383262634277344 (nan) | > loss: 0.01218001451343298 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.13683431769984303) | > loader_time: 0.0047 (0.004667669349726287)  --> TIME: 2025-02-26 14:20:39 -- STEP: 19708/36938 -- GLOBAL_STEP: 1275600 | > loss_text_ce: 0.03212849795818329 (0.031183205877773065) | > loss_mel_ce: 2.7622973918914795 (nan) | > loss: 0.011088992469012737 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13682693010753677) | > loader_time: 0.0046 (0.004667538173401444)  --> TIME: 2025-02-26 14:20:48 -- STEP: 19758/36938 -- GLOBAL_STEP: 1275650 | > loss_text_ce: 0.03441048413515091 (0.031184202970261716) | > loss_mel_ce: 2.3846888542175293 (nan) | > loss: 0.009599600918591022 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13681781372810128) | > loader_time: 0.0045 (0.00466763047701868)  --> TIME: 2025-02-26 14:20:57 -- STEP: 19808/36938 -- GLOBAL_STEP: 1275700 | > loss_text_ce: 0.033413417637348175 (0.03118447725092564) | > loss_mel_ce: 3.0975496768951416 (nan) | > loss: 0.01242445781826973 (nan) | > current_lr: 5e-06 | > step_time: 0.1501 (0.1368117093293653) | > loader_time: 0.0047 (0.0046676152621987045)  --> TIME: 2025-02-26 14:21:06 -- STEP: 19858/36938 -- GLOBAL_STEP: 1275750 | > loss_text_ce: 0.027441810816526413 (0.03118603144266685) | > loss_mel_ce: 2.7712137699127197 (nan) | > loss: 0.011105776764452457 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13680947585724915) | > loader_time: 0.0045 (0.004667579209243835)  --> TIME: 2025-02-26 14:21:16 -- STEP: 19908/36938 -- GLOBAL_STEP: 1275800 | > loss_text_ce: 0.029250307008624077 (0.031185927105616948) | > loss_mel_ce: 3.2412431240081787 (nan) | > loss: 0.012978149577975273 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13680932048527797) | > loader_time: 0.0046 (0.00466749766085169)  --> TIME: 2025-02-26 14:21:25 -- STEP: 19958/36938 -- GLOBAL_STEP: 1275850 | > loss_text_ce: 0.031924013048410416 (0.031182100330431384) | > loss_mel_ce: 1.1630818843841553 (nan) | > loss: 0.004742087330669165 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13679046878858642) | > loader_time: 0.0045 (0.0046675464459251)  --> TIME: 2025-02-26 14:21:34 -- STEP: 20008/36938 -- GLOBAL_STEP: 1275900 | > loss_text_ce: 0.0264423880726099 (0.03118439503448842) | > loss_mel_ce: 2.7855284214019775 (nan) | > loss: 0.011158614419400692 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13679498043216615) | > loader_time: 0.0046 (0.004667430138978789)  --> TIME: 2025-02-26 14:21:43 -- STEP: 20058/36938 -- GLOBAL_STEP: 1275950 | > loss_text_ce: 0.0325649157166481 (0.03118469432444503) | > loss_mel_ce: 2.7838058471679688 (nan) | > loss: 0.011176074855029583 (nan) | > current_lr: 5e-06 | > step_time: 0.1662 (0.13678043452534913) | > loader_time: 0.0047 (0.004667298020459741)  --> TIME: 2025-02-26 14:21:52 -- STEP: 20108/36938 -- GLOBAL_STEP: 1276000 | > loss_text_ce: 0.03691495954990387 (0.031186677201437803) | > loss_mel_ce: 2.893282175064087 (nan) | > loss: 0.011627767235040665 (nan) | > current_lr: 5e-06 | > step_time: 0.1494 (0.13677682384762613) | > loader_time: 0.0046 (0.004667575906659052)  --> TIME: 2025-02-26 14:22:02 -- STEP: 20158/36938 -- GLOBAL_STEP: 1276050 | > loss_text_ce: 0.020200194790959358 (0.031184472779054336) | > loss_mel_ce: 1.252628207206726 (nan) | > loss: 0.005050906911492348 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13677408372566952) | > loader_time: 0.0044 (0.004667379231854592)  --> TIME: 2025-02-26 14:22:11 -- STEP: 20208/36938 -- GLOBAL_STEP: 1276100 | > loss_text_ce: 0.039935238659381866 (0.031184099246985605) | > loss_mel_ce: 2.644873857498169 (nan) | > loss: 0.010654005222022533 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13677521849896143) | > loader_time: 0.0046 (0.004667294964560031)  --> TIME: 2025-02-26 14:22:20 -- STEP: 20258/36938 -- GLOBAL_STEP: 1276150 | > loss_text_ce: 0.030827337875962257 (0.031183813322208234) | > loss_mel_ce: 3.2148611545562744 (nan) | > loss: 0.012879717163741589 (nan) | > current_lr: 5e-06 | > step_time: 0.1667 (0.1367779528945226) | > loader_time: 0.0047 (0.004667178807035921)  --> TIME: 2025-02-26 14:22:30 -- STEP: 20308/36938 -- GLOBAL_STEP: 1276200 | > loss_text_ce: 0.021818041801452637 (0.031183739009390154) | > loss_mel_ce: 2.576258420944214 (nan) | > loss: 0.010309827513992786 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13678676499281653) | > loader_time: 0.0045 (0.00466709810141997)  --> TIME: 2025-02-26 14:22:39 -- STEP: 20358/36938 -- GLOBAL_STEP: 1276250 | > loss_text_ce: 0.03444933146238327 (0.03118542862542963) | > loss_mel_ce: 2.0415263175964355 (nan) | > loss: 0.008237998932600021 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.1367768335932655) | > loader_time: 0.0046 (0.004666941914744366)  --> TIME: 2025-02-26 14:22:48 -- STEP: 20408/36938 -- GLOBAL_STEP: 1276300 | > loss_text_ce: 0.04201739281415939 (0.031186329650142106) | > loss_mel_ce: 3.2969226837158203 (nan) | > loss: 0.013249763287603855 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1367876052108757) | > loader_time: 0.0045 (0.004666858820389953)  --> TIME: 2025-02-26 14:22:58 -- STEP: 20458/36938 -- GLOBAL_STEP: 1276350 | > loss_text_ce: 0.03228380158543587 (0.031184286096965743) | > loss_mel_ce: 1.8835313320159912 (nan) | > loss: 0.007602441590279341 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13678425710704004) | > loader_time: 0.0047 (0.004666862686844148)  --> TIME: 2025-02-26 14:23:07 -- STEP: 20508/36938 -- GLOBAL_STEP: 1276400 | > loss_text_ce: 0.03769662603735924 (0.031185981067644464) | > loss_mel_ce: 3.054323434829712 (nan) | > loss: 0.012269921600818634 (nan) | > current_lr: 5e-06 | > step_time: 0.1734 (0.13678339545155233) | > loader_time: 0.0047 (0.004666686592823754)  --> TIME: 2025-02-26 14:23:17 -- STEP: 20558/36938 -- GLOBAL_STEP: 1276450 | > loss_text_ce: 0.030308615416288376 (0.031184338335591484) | > loss_mel_ce: 2.912362813949585 (nan) | > loss: 0.011677268892526627 (nan) | > current_lr: 5e-06 | > step_time: 0.1781 (0.1367893458479675) | > loader_time: 0.005 (0.004666605062064437)  --> TIME: 2025-02-26 14:23:26 -- STEP: 20608/36938 -- GLOBAL_STEP: 1276500 | > loss_text_ce: 0.031712889671325684 (0.031184757794266188) | > loss_mel_ce: 2.309647798538208 (nan) | > loss: 0.009291114285588264 (nan) | > current_lr: 5e-06 | > step_time: 0.1651 (0.13679584469761893) | > loader_time: 0.0048 (0.004666520942071937)  --> TIME: 2025-02-26 14:23:35 -- STEP: 20658/36938 -- GLOBAL_STEP: 1276550 | > loss_text_ce: 0.02936498261988163 (0.031184161803011146) | > loss_mel_ce: 1.8606168031692505 (nan) | > loss: 0.007499928120523691 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13678917155190543) | > loader_time: 0.0045 (0.004666354524880354)  --> TIME: 2025-02-26 14:23:45 -- STEP: 20708/36938 -- GLOBAL_STEP: 1276600 | > loss_text_ce: 0.024430841207504272 (0.03118669329946352) | > loss_mel_ce: 2.690354108810425 (nan) | > loss: 0.010772956535220146 (nan) | > current_lr: 5e-06 | > step_time: 0.1556 (0.13679816563064484) | > loader_time: 0.0046 (0.004666251210096084)  --> TIME: 2025-02-26 14:23:54 -- STEP: 20758/36938 -- GLOBAL_STEP: 1276650 | > loss_text_ce: 0.03878389671444893 (0.031185944345022837) | > loss_mel_ce: 2.8777942657470703 (nan) | > loss: 0.011573723517358303 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13679830784948815) | > loader_time: 0.0046 (0.0046661890751020455)  --> TIME: 2025-02-26 14:24:03 -- STEP: 20808/36938 -- GLOBAL_STEP: 1276700 | > loss_text_ce: 0.03280453011393547 (0.03118653739599339) | > loss_mel_ce: 2.932162284851074 (nan) | > loss: 0.011765741743147373 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13679159809332153) | > loader_time: 0.0046 (0.004666367765574749)  --> TIME: 2025-02-26 14:24:13 -- STEP: 20858/36938 -- GLOBAL_STEP: 1276750 | > loss_text_ce: 0.038618188351392746 (0.031185312548057374) | > loss_mel_ce: 3.1559250354766846 (nan) | > loss: 0.012676759622991085 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.13679823702597188) | > loader_time: 0.0048 (0.004666275598136614)  --> TIME: 2025-02-26 14:24:22 -- STEP: 20908/36938 -- GLOBAL_STEP: 1276800 | > loss_text_ce: 0.03621400147676468 (0.031185252842793942) | > loss_mel_ce: 3.2200827598571777 (nan) | > loss: 0.012921813875436783 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13679574054537735) | > loader_time: 0.0046 (0.004666260934500917)  --> TIME: 2025-02-26 14:24:31 -- STEP: 20958/36938 -- GLOBAL_STEP: 1276850 | > loss_text_ce: 0.029386984184384346 (0.031182277448141553) | > loss_mel_ce: 2.2667040824890137 (nan) | > loss: 0.009111473336815834 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13679890915209658) | > loader_time: 0.0046 (0.0046663701915367695)  --> TIME: 2025-02-26 14:24:41 -- STEP: 21008/36938 -- GLOBAL_STEP: 1276900 | > loss_text_ce: 0.047008056193590164 (0.031183667404473227) | > loss_mel_ce: 0.1456298828125 (nan) | > loss: 0.0007644363213330507 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13680066387342826) | > loader_time: 0.0045 (0.00466622172742059)  --> TIME: 2025-02-26 14:24:50 -- STEP: 21058/36938 -- GLOBAL_STEP: 1276950 | > loss_text_ce: 0.03898198902606964 (0.031186069295039216) | > loss_mel_ce: 2.8343610763549805 (nan) | > loss: 0.01140215527266264 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13679052523505084) | > loader_time: 0.0046 (0.004665985169919273)  --> TIME: 2025-02-26 14:24:59 -- STEP: 21108/36938 -- GLOBAL_STEP: 1277000 | > loss_text_ce: 0.03173290193080902 (0.03118554092635699) | > loss_mel_ce: 2.815537691116333 (nan) | > loss: 0.011298692785203457 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13680133629595198) | > loader_time: 0.0047 (0.004665893622368181)  --> TIME: 2025-02-26 14:25:09 -- STEP: 21158/36938 -- GLOBAL_STEP: 1277050 | > loss_text_ce: 0.023508893325924873 (0.031187964879514772) | > loss_mel_ce: 2.378767251968384 (nan) | > loss: 0.009532841853797436 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13681120224108992) | > loader_time: 0.0047 (0.0046658848125076475)  --> TIME: 2025-02-26 14:25:19 -- STEP: 21208/36938 -- GLOBAL_STEP: 1277100 | > loss_text_ce: 0.02915683202445507 (0.03118475911703547) | > loss_mel_ce: 2.8742270469665527 (nan) | > loss: 0.011521365493535995 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13682490086969165) | > loader_time: 0.0048 (0.0046658501315683335)  --> TIME: 2025-02-26 14:25:28 -- STEP: 21258/36938 -- GLOBAL_STEP: 1277150 | > loss_text_ce: 0.04052123427391052 (0.031183770437609146) | > loss_mel_ce: 2.6642513275146484 (nan) | > loss: 0.0107332244515419 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.1368286710028163) | > loader_time: 0.0046 (0.004665775675463611)  --> TIME: 2025-02-26 14:25:37 -- STEP: 21308/36938 -- GLOBAL_STEP: 1277200 | > loss_text_ce: 0.035922057926654816 (0.031183717825775368) | > loss_mel_ce: 2.945446014404297 (nan) | > loss: 0.011830826289951801 (nan) | > current_lr: 5e-06 | > step_time: 0.1832 (0.13681789594316915) | > loader_time: 0.0048 (0.0046657033142954735)  --> TIME: 2025-02-26 14:25:47 -- STEP: 21358/36938 -- GLOBAL_STEP: 1277250 | > loss_text_ce: 0.03250535577535629 (0.031182349255629974) | > loss_mel_ce: 2.4521806240081787 (nan) | > loss: 0.00985986553132534 (nan) | > current_lr: 5e-06 | > step_time: 0.1565 (0.1368248389559589) | > loader_time: 0.0046 (0.004665632184965672)  --> TIME: 2025-02-26 14:25:56 -- STEP: 21408/36938 -- GLOBAL_STEP: 1277300 | > loss_text_ce: 0.03717277571558952 (0.031183549567276786) | > loss_mel_ce: 1.9289289712905884 (nan) | > loss: 0.007801991421729326 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13682515830113387) | > loader_time: 0.0045 (0.004665824485822988)  --> TIME: 2025-02-26 14:26:06 -- STEP: 21458/36938 -- GLOBAL_STEP: 1277350 | > loss_text_ce: 0.025671200826764107 (0.031185316109920622) | > loss_mel_ce: 2.279158115386963 (nan) | > loss: 0.009146148338913918 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13682994936750087) | > loader_time: 0.0045 (0.004665775171963085)  --> TIME: 2025-02-26 14:26:15 -- STEP: 21508/36938 -- GLOBAL_STEP: 1277400 | > loss_text_ce: 0.030467471107840538 (0.031186246126439856) | > loss_mel_ce: 2.5972609519958496 (nan) | > loss: 0.010427494533360004 (nan) | > current_lr: 5e-06 | > step_time: 0.1268 (0.13683453571936083) | > loader_time: 0.0046 (0.0046658193131727446)  --> TIME: 2025-02-26 14:26:24 -- STEP: 21558/36938 -- GLOBAL_STEP: 1277450 | > loss_text_ce: 0.03560931235551834 (0.03118776641018853) | > loss_mel_ce: 2.7298150062561035 (nan) | > loss: 0.010973906144499779 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13682513002982566) | > loader_time: 0.0046 (0.004665782206332094)  --> TIME: 2025-02-26 14:26:34 -- STEP: 21608/36938 -- GLOBAL_STEP: 1277500 | > loss_text_ce: 0.0315006822347641 (0.031187255339659426) | > loss_mel_ce: 2.475210189819336 (nan) | > loss: 0.00994726549834013 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13682910374075063) | > loader_time: 0.0048 (0.004666011307425762)  --> TIME: 2025-02-26 14:26:43 -- STEP: 21658/36938 -- GLOBAL_STEP: 1277550 | > loss_text_ce: 0.03364355117082596 (0.031184689860232987) | > loss_mel_ce: 2.8239548206329346 (nan) | > loss: 0.011339676566421986 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13682629603275742) | > loader_time: 0.0046 (0.004666137182323939)  --> TIME: 2025-02-26 14:26:52 -- STEP: 21708/36938 -- GLOBAL_STEP: 1277600 | > loss_text_ce: 0.02448219060897827 (0.031183872583892144) | > loss_mel_ce: 2.087954521179199 (nan) | > loss: 0.008382686413824558 (nan) | > current_lr: 5e-06 | > step_time: 0.1718 (0.13682646783913413) | > loader_time: 0.0047 (0.004666040357533818)  --> TIME: 2025-02-26 14:27:01 -- STEP: 21758/36938 -- GLOBAL_STEP: 1277650 | > loss_text_ce: 0.03542715311050415 (0.03118551583270315) | > loss_mel_ce: 1.5502420663833618 (nan) | > loss: 0.006292338948696852 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13682250520062886) | > loader_time: 0.0047 (0.004666217614517595)  --> TIME: 2025-02-26 14:27:11 -- STEP: 21808/36938 -- GLOBAL_STEP: 1277700 | > loss_text_ce: 0.03351499140262604 (0.031187386822831082) | > loss_mel_ce: 2.7944769859313965 (nan) | > loss: 0.011222191154956818 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.13682703935129162) | > loader_time: 0.0048 (0.004666351541734651)  --> TIME: 2025-02-26 14:27:21 -- STEP: 21858/36938 -- GLOBAL_STEP: 1277750 | > loss_text_ce: 0.030880805104970932 (0.03118782430154879) | > loss_mel_ce: 2.19207763671875 (nan) | > loss: 0.008821263909339905 (nan) | > current_lr: 5e-06 | > step_time: 0.1626 (0.1368427254250721) | > loader_time: 0.0047 (0.004666584791776304)  --> TIME: 2025-02-26 14:27:30 -- STEP: 21908/36938 -- GLOBAL_STEP: 1277800 | > loss_text_ce: 0.03278251737356186 (0.031186079834058196) | > loss_mel_ce: 3.0606749057769775 (nan) | > loss: 0.012275625951588154 (nan) | > current_lr: 5e-06 | > step_time: 0.2022 (0.13683455604607522) | > loader_time: 0.0051 (0.004666643310729337)  --> TIME: 2025-02-26 14:27:39 -- STEP: 21958/36938 -- GLOBAL_STEP: 1277850 | > loss_text_ce: 0.039010848850011826 (0.031184957042275054) | > loss_mel_ce: 2.570126533508301 (nan) | > loss: 0.01035371981561184 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13683413131302813) | > loader_time: 0.0046 (0.004666755755141567)  --> TIME: 2025-02-26 14:27:49 -- STEP: 22008/36938 -- GLOBAL_STEP: 1277900 | > loss_text_ce: 0.026844140142202377 (0.031184474625695815) | > loss_mel_ce: 2.897326707839966 (nan) | > loss: 0.011603852733969688 (nan) | > current_lr: 5e-06 | > step_time: 0.1877 (0.13683747846574962) | > loader_time: 0.0049 (0.00466690744672501)  --> TIME: 2025-02-26 14:27:58 -- STEP: 22058/36938 -- GLOBAL_STEP: 1277950 | > loss_text_ce: 0.026032032445073128 (0.031185392254929913) | > loss_mel_ce: 2.847470998764038 (nan) | > loss: 0.011402790434658527 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13683896461870385) | > loader_time: 0.0046 (0.004667062395794311)  --> TIME: 2025-02-26 14:28:08 -- STEP: 22108/36938 -- GLOBAL_STEP: 1278000 | > loss_text_ce: 0.028723077848553658 (0.031184949787227117) | > loss_mel_ce: 0.5933600664138794 (nan) | > loss: 0.0024685838725417852 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13684952012103999) | > loader_time: 0.0044 (0.004667192055861113)  --> TIME: 2025-02-26 14:28:17 -- STEP: 22158/36938 -- GLOBAL_STEP: 1278050 | > loss_text_ce: 0.02584184892475605 (0.03118573144009829) | > loss_mel_ce: 2.5889792442321777 (nan) | > loss: 0.010376275517046452 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.13685061101158533) | > loader_time: 0.005 (0.0046671987903366555)  --> TIME: 2025-02-26 14:28:27 -- STEP: 22208/36938 -- GLOBAL_STEP: 1278100 | > loss_text_ce: 0.03091641142964363 (0.031186424665411695) | > loss_mel_ce: 2.9574391841888428 (nan) | > loss: 0.011858554556965828 (nan) | > current_lr: 5e-06 | > step_time: 0.187 (0.13686651299554609) | > loader_time: 0.0049 (0.004667114541578689)  --> TIME: 2025-02-26 14:28:36 -- STEP: 22258/36938 -- GLOBAL_STEP: 1278150 | > loss_text_ce: 0.04151585325598717 (0.031187276445004165) | > loss_mel_ce: 2.414755344390869 (nan) | > loss: 0.009747108444571495 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13687356564434988) | > loader_time: 0.0046 (0.004667079023449797)  --> TIME: 2025-02-26 14:28:46 -- STEP: 22308/36938 -- GLOBAL_STEP: 1278200 | > loss_text_ce: 0.027518924325704575 (0.031185303828861886) | > loss_mel_ce: 1.9072142839431763 (nan) | > loss: 0.007677513174712658 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13687812946422878) | > loader_time: 0.0047 (0.004667051113782557)  --> TIME: 2025-02-26 14:28:55 -- STEP: 22358/36938 -- GLOBAL_STEP: 1278250 | > loss_text_ce: 0.047026585787534714 (0.031187923802520248) | > loss_mel_ce: 2.1786115169525146 (nan) | > loss: 0.008831897750496864 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13687447604131533) | > loader_time: 0.0046 (0.004666922013313215)  --> TIME: 2025-02-26 14:29:04 -- STEP: 22408/36938 -- GLOBAL_STEP: 1278300 | > loss_text_ce: 0.03901631012558937 (0.031188279364358803) | > loss_mel_ce: 2.7026572227478027 (nan) | > loss: 0.010879657231271267 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13687762682968202) | > loader_time: 0.0045 (0.004666814609154406)  --> TIME: 2025-02-26 14:29:14 -- STEP: 22458/36938 -- GLOBAL_STEP: 1278350 | > loss_text_ce: 0.02902963012456894 (0.031190447061708218) | > loss_mel_ce: 1.8000437021255493 (nan) | > loss: 0.007258228026330471 (nan) | > current_lr: 5e-06 | > step_time: 0.1567 (0.1368869247945253) | > loader_time: 0.0047 (0.004666787432120994)  --> TIME: 2025-02-26 14:29:23 -- STEP: 22508/36938 -- GLOBAL_STEP: 1278400 | > loss_text_ce: 0.010028486140072346 (0.0311914639775374) | > loss_mel_ce: 1.608553171157837 (nan) | > loss: 0.006422943435609341 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13688009265516987) | > loader_time: 0.0044 (0.0046668078625246474)  --> TIME: 2025-02-26 14:29:32 -- STEP: 22558/36938 -- GLOBAL_STEP: 1278450 | > loss_text_ce: 0.0376523956656456 (0.031193911987237533) | > loss_mel_ce: 2.2185211181640625 (nan) | > loss: 0.008953070268034935 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1368760305343448) | > loader_time: 0.0045 (0.00466670934185192)  --> TIME: 2025-02-26 14:29:41 -- STEP: 22608/36938 -- GLOBAL_STEP: 1278500 | > loss_text_ce: 0.025865260511636734 (0.03119500975646224) | > loss_mel_ce: 2.8676109313964844 (nan) | > loss: 0.011482049711048603 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.1368742723542584) | > loader_time: 0.0048 (0.004666617141492112)  --> TIME: 2025-02-26 14:29:51 -- STEP: 22658/36938 -- GLOBAL_STEP: 1278550 | > loss_text_ce: 0.03319033235311508 (0.03119624026779996) | > loss_mel_ce: 3.270885944366455 (nan) | > loss: 0.013111414387822151 (nan) | > current_lr: 5e-06 | > step_time: 0.1456 (0.13687215626676477) | > loader_time: 0.0046 (0.004666563260585765)  --> TIME: 2025-02-26 14:30:00 -- STEP: 22708/36938 -- GLOBAL_STEP: 1278600 | > loss_text_ce: 0.021103573963046074 (0.031196922427202452) | > loss_mel_ce: 2.299839496612549 (nan) | > loss: 0.009210092015564442 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1368829138223946) | > loader_time: 0.0045 (0.004666544464202696)  --> TIME: 2025-02-26 14:30:10 -- STEP: 22758/36938 -- GLOBAL_STEP: 1278650 | > loss_text_ce: 0.02918328158557415 (0.031196286084274622) | > loss_mel_ce: 2.019559144973755 (nan) | > loss: 0.008129931055009365 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13688204917895974) | > loader_time: 0.0045 (0.004666547949593392)  --> TIME: 2025-02-26 14:30:19 -- STEP: 22808/36938 -- GLOBAL_STEP: 1278700 | > loss_text_ce: 0.027347518131136894 (0.031196610956844464) | > loss_mel_ce: 3.0708138942718506 (nan) | > loss: 0.012294292449951172 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13689410376030278) | > loader_time: 0.0045 (0.004666410195789826)  --> TIME: 2025-02-26 14:30:28 -- STEP: 22858/36938 -- GLOBAL_STEP: 1278750 | > loss_text_ce: 0.021298881620168686 (0.03119788831935117) | > loss_mel_ce: 2.116225004196167 (nan) | > loss: 0.00848223827779293 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13688605794840703) | > loader_time: 0.0044 (0.00466617150448136)  --> TIME: 2025-02-26 14:30:38 -- STEP: 22908/36938 -- GLOBAL_STEP: 1278800 | > loss_text_ce: 0.02621428482234478 (0.031200397467692452) | > loss_mel_ce: 1.7011735439300537 (nan) | > loss: 0.006854713894426823 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13688537994146321) | > loader_time: 0.0046 (0.004665967690418022)  --> TIME: 2025-02-26 14:30:47 -- STEP: 22958/36938 -- GLOBAL_STEP: 1278850 | > loss_text_ce: 0.03124629519879818 (0.03120008928103983) | > loss_mel_ce: 1.377652883529663 (nan) | > loss: 0.0055908700451254845 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.1368916681579456) | > loader_time: 0.0046 (0.004665886310036023)  --> TIME: 2025-02-26 14:30:57 -- STEP: 23008/36938 -- GLOBAL_STEP: 1278900 | > loss_text_ce: 0.036785900592803955 (0.031200769334796416) | > loss_mel_ce: 2.7176239490509033 (nan) | > loss: 0.0109301982447505 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13689314757418083) | > loader_time: 0.0045 (0.00466578616469226)  --> TIME: 2025-02-26 14:31:06 -- STEP: 23058/36938 -- GLOBAL_STEP: 1278950 | > loss_text_ce: 0.032483089715242386 (0.031201894894036786) | > loss_mel_ce: 2.8682162761688232 (nan) | > loss: 0.011510712094604969 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13689216689375264) | > loader_time: 0.0045 (0.004665639117374318)  --> TIME: 2025-02-26 14:31:16 -- STEP: 23108/36938 -- GLOBAL_STEP: 1279000 | > loss_text_ce: 0.03868831694126129 (0.03120191936181742) | > loss_mel_ce: 1.281654953956604 (nan) | > loss: 0.005239457823336124 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13690321824532903) | > loader_time: 0.0046 (0.004665499154890355)  --> TIME: 2025-02-26 14:31:25 -- STEP: 23158/36938 -- GLOBAL_STEP: 1279050 | > loss_text_ce: 0.03397921845316887 (0.031202318608263446) | > loss_mel_ce: 3.1162421703338623 (nan) | > loss: 0.012500879354774952 (nan) | > current_lr: 5e-06 | > step_time: 0.1466 (0.13690811186135787) | > loader_time: 0.0047 (0.004665447306845064)  --> TIME: 2025-02-26 14:31:34 -- STEP: 23208/36938 -- GLOBAL_STEP: 1279100 | > loss_text_ce: 0.027559569105505943 (0.031202972135758587) | > loss_mel_ce: 1.343815565109253 (nan) | > loss: 0.005441965069621801 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.136901692444602) | > loader_time: 0.0045 (0.004665322034205273)  --> TIME: 2025-02-26 14:31:44 -- STEP: 23258/36938 -- GLOBAL_STEP: 1279150 | > loss_text_ce: 0.04065774381160736 (0.031200421236548988) | > loss_mel_ce: 2.7749338150024414 (nan) | > loss: 0.01117298286408186 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.1369029228783032) | > loader_time: 0.0046 (0.004665269980027512)  --> TIME: 2025-02-26 14:31:54 -- STEP: 23308/36938 -- GLOBAL_STEP: 1279200 | > loss_text_ce: 0.03539012745022774 (0.03120348488135858) | > loss_mel_ce: 3.027040719985962 (nan) | > loss: 0.012152504175901413 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.13692382483861243) | > loader_time: 0.0045 (0.004665197251242781)  --> TIME: 2025-02-26 14:32:03 -- STEP: 23358/36938 -- GLOBAL_STEP: 1279250 | > loss_text_ce: 0.017187681049108505 (0.03120093203608819) | > loss_mel_ce: 2.423970937728882 (nan) | > loss: 0.00968713779002428 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13692467290659607) | > loader_time: 0.0045 (0.004665089608953555)  --> TIME: 2025-02-26 14:32:12 -- STEP: 23408/36938 -- GLOBAL_STEP: 1279300 | > loss_text_ce: 0.0376405268907547 (0.031201752515292336) | > loss_mel_ce: 2.7864391803741455 (nan) | > loss: 0.011206666007637978 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.13692483061482785) | > loader_time: 0.0046 (0.004664992469268047)  --> TIME: 2025-02-26 14:32:22 -- STEP: 23458/36938 -- GLOBAL_STEP: 1279350 | > loss_text_ce: 0.026971420273184776 (0.031204626890921617) | > loss_mel_ce: 2.9443938732147217 (nan) | > loss: 0.011791132390499115 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1369290375040765) | > loader_time: 0.0046 (0.00466485394064711)  --> TIME: 2025-02-26 14:32:31 -- STEP: 23508/36938 -- GLOBAL_STEP: 1279400 | > loss_text_ce: 0.02718513272702694 (0.031201823535501286) | > loss_mel_ce: 3.0551624298095703 (nan) | > loss: 0.012231539003551006 (nan) | > current_lr: 5e-06 | > step_time: 0.1454 (0.1369327014251611) | > loader_time: 0.0045 (0.00466470923658249)  --> TIME: 2025-02-26 14:32:41 -- STEP: 23558/36938 -- GLOBAL_STEP: 1279450 | > loss_text_ce: 0.031758129596710205 (0.03120144485543504) | > loss_mel_ce: 2.4942779541015625 (nan) | > loss: 0.010023953393101692 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13694561093537072) | > loader_time: 0.0045 (0.004664656028792882)  --> TIME: 2025-02-26 14:32:50 -- STEP: 23608/36938 -- GLOBAL_STEP: 1279500 | > loss_text_ce: 0.03594173491001129 (0.03119961030107315) | > loss_mel_ce: 3.0232200622558594 (nan) | > loss: 0.012139531783759594 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.13693923655908743) | > loader_time: 0.0047 (0.004664536695567585)  --> TIME: 2025-02-26 14:32:59 -- STEP: 23658/36938 -- GLOBAL_STEP: 1279550 | > loss_text_ce: 0.0263136588037014 (0.03119848008339307) | > loss_mel_ce: 2.949772357940674 (nan) | > loss: 0.01180986501276493 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.13693758943335083) | > loader_time: 0.0049 (0.004664400170284539)  --> TIME: 2025-02-26 14:33:09 -- STEP: 23708/36938 -- GLOBAL_STEP: 1279600 | > loss_text_ce: 0.03578835353255272 (0.03119876657258132) | > loss_mel_ce: 2.9625542163848877 (nan) | > loss: 0.01189818512648344 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.13692768886798518) | > loader_time: 0.0048 (0.00466424412805393)  --> TIME: 2025-02-26 14:33:18 -- STEP: 23758/36938 -- GLOBAL_STEP: 1279650 | > loss_text_ce: 0.02395155467092991 (0.031198726457725017) | > loss_mel_ce: 2.6528360843658447 (nan) | > loss: 0.010622173547744751 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13692794751113432) | > loader_time: 0.0046 (0.00466413712278741)  --> TIME: 2025-02-26 14:33:27 -- STEP: 23808/36938 -- GLOBAL_STEP: 1279700 | > loss_text_ce: 0.03359398990869522 (0.031197790865311017) | > loss_mel_ce: 1.6890869140625 (nan) | > loss: 0.006836035754531622 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13692083149667714) | > loader_time: 0.0045 (0.0046640855750890034)  --> TIME: 2025-02-26 14:33:36 -- STEP: 23858/36938 -- GLOBAL_STEP: 1279750 | > loss_text_ce: 0.025439295917749405 (0.031200894732381138) | > loss_mel_ce: 3.173168659210205 (nan) | > loss: 0.01269288919866085 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.13692097901118766) | > loader_time: 0.0046 (0.004663936099898962)  --> TIME: 2025-02-26 14:33:46 -- STEP: 23908/36938 -- GLOBAL_STEP: 1279800 | > loss_text_ce: 0.033026695251464844 (0.03120205259363976) | > loss_mel_ce: 2.011449098587036 (nan) | > loss: 0.008112999610602856 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13692162536310273) | > loader_time: 0.0044 (0.004663763844849198)  --> TIME: 2025-02-26 14:33:55 -- STEP: 23958/36938 -- GLOBAL_STEP: 1279850 | > loss_text_ce: 0.02092629112303257 (0.03120194367919174) | > loss_mel_ce: 1.6474387645721436 (nan) | > loss: 0.006620496511459351 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13692477151660712) | > loader_time: 0.0045 (0.004663615276901265)  --> TIME: 2025-02-26 14:34:05 -- STEP: 24008/36938 -- GLOBAL_STEP: 1279900 | > loss_text_ce: 0.031146125867962837 (0.031199820955156395) | > loss_mel_ce: 3.0838427543640137 (nan) | > loss: 0.012361067347228527 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.1369249840173591) | > loader_time: 0.0047 (0.004663553268899111)  --> TIME: 2025-02-26 14:34:14 -- STEP: 24058/36938 -- GLOBAL_STEP: 1279950 | > loss_text_ce: 0.03258587047457695 (0.031200320259299016) | > loss_mel_ce: 3.0363433361053467 (nan) | > loss: 0.012178291566669941 (nan) | > current_lr: 5e-06 | > step_time: 0.1532 (0.13692603480717777) | > loader_time: 0.0046 (0.004663436725377646)  --> TIME: 2025-02-26 14:34:23 -- STEP: 24108/36938 -- GLOBAL_STEP: 1280000 | > loss_text_ce: 0.040338948369026184 (0.03120072424850316) | > loss_mel_ce: 1.3894144296646118 (nan) | > loss: 0.005673625040799379 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13692809300417133) | > loader_time: 0.0044 (0.004663330248305293) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1280000.pth  --> TIME: 2025-02-26 14:34:39 -- STEP: 24158/36938 -- GLOBAL_STEP: 1280050 | > loss_text_ce: 0.029900092631578445 (0.031198841070222617) | > loss_mel_ce: 1.1067253351211548 (nan) | > loss: 0.004510418511927128 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13692561502793793) | > loader_time: 0.0045 (0.004663247197202724)  --> TIME: 2025-02-26 14:34:48 -- STEP: 24208/36938 -- GLOBAL_STEP: 1280100 | > loss_text_ce: 0.033097781240940094 (0.0312008129665782) | > loss_mel_ce: 3.029538869857788 (nan) | > loss: 0.01215332094579935 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.13691894827264073) | > loader_time: 0.0046 (0.004663146003066335)  --> TIME: 2025-02-26 14:34:57 -- STEP: 24258/36938 -- GLOBAL_STEP: 1280150 | > loss_text_ce: 0.04124194011092186 (0.031201368464625838) | > loss_mel_ce: 2.597721576690674 (nan) | > loss: 0.010472077876329422 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13691771926780802) | > loader_time: 0.0045 (0.0046630189546365865)  --> TIME: 2025-02-26 14:35:07 -- STEP: 24308/36938 -- GLOBAL_STEP: 1280200 | > loss_text_ce: 0.026737501844763756 (0.031201542958326646) | > loss_mel_ce: 2.529512643814087 (nan) | > loss: 0.01014384999871254 (nan) | > current_lr: 5e-06 | > step_time: 0.1617 (0.13692035018093227) | > loader_time: 0.0047 (0.004663021083487777)  --> TIME: 2025-02-26 14:35:16 -- STEP: 24358/36938 -- GLOBAL_STEP: 1280250 | > loss_text_ce: 0.02220235951244831 (0.03120048300364118) | > loss_mel_ce: 2.988647222518921 (nan) | > loss: 0.011947816237807274 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13691662364879517) | > loader_time: 0.0045 (0.004662966422821662)  --> TIME: 2025-02-26 14:35:25 -- STEP: 24408/36938 -- GLOBAL_STEP: 1280300 | > loss_text_ce: 0.03600965812802315 (0.03120256181069449) | > loss_mel_ce: 2.3917510509490967 (nan) | > loss: 0.009633971378207207 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13691208044922332) | > loader_time: 0.0045 (0.004662821807458087)  --> TIME: 2025-02-26 14:35:34 -- STEP: 24458/36938 -- GLOBAL_STEP: 1280350 | > loss_text_ce: 0.02345220185816288 (0.0312001218062611) | > loss_mel_ce: 2.842258930206299 (nan) | > loss: 0.011371870525181293 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.13691394326536538) | > loader_time: 0.0047 (0.0046628047618854866)  --> TIME: 2025-02-26 14:35:43 -- STEP: 24508/36938 -- GLOBAL_STEP: 1280400 | > loss_text_ce: 0.02792578563094139 (0.03120080288203188) | > loss_mel_ce: 2.459068536758423 (nan) | > loss: 0.009869025088846684 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.1369000304595514) | > loader_time: 0.0046 (0.0046626990452780705)  --> TIME: 2025-02-26 14:35:53 -- STEP: 24558/36938 -- GLOBAL_STEP: 1280450 | > loss_text_ce: 0.04648510739207268 (0.03120549915329451) | > loss_mel_ce: 2.4287116527557373 (nan) | > loss: 0.009822210296988487 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13689864695348866) | > loader_time: 0.0046 (0.004662705793939143)  --> TIME: 2025-02-26 14:36:02 -- STEP: 24608/36938 -- GLOBAL_STEP: 1280500 | > loss_text_ce: 0.04387693479657173 (0.031203643846398548) | > loss_mel_ce: 2.875586748123169 (nan) | > loss: 0.0115851741284132 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13690107325467402) | > loader_time: 0.0047 (0.004662671309298727)  --> TIME: 2025-02-26 14:36:11 -- STEP: 24658/36938 -- GLOBAL_STEP: 1280550 | > loss_text_ce: 0.02809733897447586 (0.031201829769054803) | > loss_mel_ce: 2.8650455474853516 (nan) | > loss: 0.011480726301670074 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13690504040371163) | > loader_time: 0.0047 (0.0046627285397538845)  --> TIME: 2025-02-26 14:36:21 -- STEP: 24708/36938 -- GLOBAL_STEP: 1280600 | > loss_text_ce: 0.04890192300081253 (0.03120393460607876) | > loss_mel_ce: 3.288994312286377 (nan) | > loss: 0.013245620764791965 (nan) | > current_lr: 5e-06 | > step_time: 0.1824 (0.13689893122261265) | > loader_time: 0.0049 (0.0046627049077882735)  --> TIME: 2025-02-26 14:36:30 -- STEP: 24758/36938 -- GLOBAL_STEP: 1280650 | > loss_text_ce: 0.028439054265618324 (0.031200900663709678) | > loss_mel_ce: 3.1864943504333496 (nan) | > loss: 0.012757672928273678 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13689432094859633) | > loader_time: 0.0046 (0.0046626639314147865)  --> TIME: 2025-02-26 14:36:39 -- STEP: 24808/36938 -- GLOBAL_STEP: 1280700 | > loss_text_ce: 0.02790536731481552 (0.031199841582604054) | > loss_mel_ce: 1.70111083984375 (nan) | > loss: 0.006861175876110792 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.1368969461391987) | > loader_time: 0.0047 (0.00466264712737474)  --> TIME: 2025-02-26 14:36:48 -- STEP: 24858/36938 -- GLOBAL_STEP: 1280750 | > loss_text_ce: 0.03043081983923912 (0.031197415764151123) | > loss_mel_ce: 2.353982448577881 (nan) | > loss: 0.009461957961320877 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13688842441159219) | > loader_time: 0.0047 (0.004662584516123336)  --> TIME: 2025-02-26 14:36:58 -- STEP: 24908/36938 -- GLOBAL_STEP: 1280800 | > loss_text_ce: 0.02508648857474327 (0.031198213866792033) | > loss_mel_ce: 2.6927261352539062 (nan) | > loss: 0.010784970596432686 (nan) | > current_lr: 5e-06 | > step_time: 0.1888 (0.13688618428101776) | > loader_time: 0.0049 (0.0046625272676728965)  --> TIME: 2025-02-26 14:37:07 -- STEP: 24958/36938 -- GLOBAL_STEP: 1280850 | > loss_text_ce: 0.034536443650722504 (0.031197289976694596) | > loss_mel_ce: 3.1901371479034424 (nan) | > loss: 0.012796323746442795 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13689185292190223) | > loader_time: 0.0046 (0.004662688491684319)  --> TIME: 2025-02-26 14:37:17 -- STEP: 25008/36938 -- GLOBAL_STEP: 1280900 | > loss_text_ce: 0.034200519323349 (0.03119714565274812) | > loss_mel_ce: 3.0329172611236572 (nan) | > loss: 0.012171102687716484 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.13689191139857895) | > loader_time: 0.0047 (0.004662722778183006)  --> TIME: 2025-02-26 14:37:26 -- STEP: 25058/36938 -- GLOBAL_STEP: 1280950 | > loss_text_ce: 0.027991916984319687 (0.031197414263682704) | > loss_mel_ce: 2.698777675628662 (nan) | > loss: 0.01082051545381546 (nan) | > current_lr: 5e-06 | > step_time: 0.1628 (0.13689250603136233) | > loader_time: 0.0046 (0.004662657718398103)  --> TIME: 2025-02-26 14:37:35 -- STEP: 25108/36938 -- GLOBAL_STEP: 1281000 | > loss_text_ce: 0.045754916965961456 (0.03119580100983517) | > loss_mel_ce: 0.7080078125 (nan) | > loss: 0.00299112219363451 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13689403830581875) | > loader_time: 0.0045 (0.004662568770112796)  --> TIME: 2025-02-26 14:37:44 -- STEP: 25158/36938 -- GLOBAL_STEP: 1281050 | > loss_text_ce: 0.02931866981089115 (0.031195716341815744) | > loss_mel_ce: 3.038022041320801 (nan) | > loss: 0.012171987444162369 (nan) | > current_lr: 5e-06 | > step_time: 0.1758 (0.13689197837401376) | > loader_time: 0.0048 (0.004662515874678115)  --> TIME: 2025-02-26 14:37:54 -- STEP: 25208/36938 -- GLOBAL_STEP: 1281100 | > loss_text_ce: 0.029778344556689262 (0.03119449548867228) | > loss_mel_ce: 2.961613178253174 (nan) | > loss: 0.01187060121446848 (nan) | > current_lr: 5e-06 | > step_time: 0.1868 (0.13689658297352925) | > loader_time: 0.0049 (0.004662576723537589)  --> TIME: 2025-02-26 14:38:03 -- STEP: 25258/36938 -- GLOBAL_STEP: 1281150 | > loss_text_ce: 0.026312226429581642 (0.031194777987951675) | > loss_mel_ce: 2.689852476119995 (nan) | > loss: 0.01077843178063631 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.13688973606845736) | > loader_time: 0.0049 (0.004662541069207751)  --> TIME: 2025-02-26 14:38:12 -- STEP: 25308/36938 -- GLOBAL_STEP: 1281200 | > loss_text_ce: 0.03300286829471588 (0.031196209784215626) | > loss_mel_ce: 2.205862283706665 (nan) | > loss: 0.008884386159479618 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.13688376274119435) | > loader_time: 0.0048 (0.00466252775088255)  --> TIME: 2025-02-26 14:38:21 -- STEP: 25358/36938 -- GLOBAL_STEP: 1281250 | > loss_text_ce: 0.06283921748399734 (0.03119665224853469) | > loss_mel_ce: 2.164677858352661 (nan) | > loss: 0.008839353919029236 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13688209272350493) | > loader_time: 0.0045 (0.004662747300003052)  --> TIME: 2025-02-26 14:38:31 -- STEP: 25408/36938 -- GLOBAL_STEP: 1281300 | > loss_text_ce: 0.02874957211315632 (0.03119907181907061) | > loss_mel_ce: 1.259765625 (nan) | > loss: 0.005113156046718359 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1368798397537743) | > loader_time: 0.0066 (0.004662886468379248)  --> TIME: 2025-02-26 14:38:40 -- STEP: 25458/36938 -- GLOBAL_STEP: 1281350 | > loss_text_ce: 0.025784827768802643 (0.03119796063717034) | > loss_mel_ce: 3.140402317047119 (nan) | > loss: 0.012564235366880894 (nan) | > current_lr: 5e-06 | > step_time: 0.1797 (0.1368867642413663) | > loader_time: 0.0048 (0.004663318369860381)  --> TIME: 2025-02-26 14:38:49 -- STEP: 25508/36938 -- GLOBAL_STEP: 1281400 | > loss_text_ce: 0.029770126566290855 (0.031197344400651694) | > loss_mel_ce: 2.892578125 (nan) | > loss: 0.0115966210141778 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13687719402594406) | > loader_time: 0.0045 (0.004663450900749728)  --> TIME: 2025-02-26 14:38:58 -- STEP: 25558/36938 -- GLOBAL_STEP: 1281450 | > loss_text_ce: 0.03107917681336403 (0.031199532836924605) | > loss_mel_ce: 0.14731749892234802 (nan) | > loss: 0.000707923318259418 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1368699794607047) | > loader_time: 0.0045 (0.004663357666395316)  --> TIME: 2025-02-26 14:39:08 -- STEP: 25608/36938 -- GLOBAL_STEP: 1281500 | > loss_text_ce: 0.0253097265958786 (0.03119862066309519) | > loss_mel_ce: 2.3883702754974365 (nan) | > loss: 0.009578095749020576 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13687635325558756) | > loader_time: 0.0046 (0.0046635132698445)  --> TIME: 2025-02-26 14:39:17 -- STEP: 25658/36938 -- GLOBAL_STEP: 1281550 | > loss_text_ce: 0.027477605268359184 (0.031195433600989007) | > loss_mel_ce: 2.582537889480591 (nan) | > loss: 0.01035720482468605 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13687466738777673) | > loader_time: 0.0046 (0.004663490052252546)  --> TIME: 2025-02-26 14:39:27 -- STEP: 25708/36938 -- GLOBAL_STEP: 1281600 | > loss_text_ce: 0.030943861231207848 (0.03119412637285985) | > loss_mel_ce: 2.6515724658966064 (nan) | > loss: 0.010644907131791115 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13687351128538283) | > loader_time: 0.0047 (0.004663604228033193)  --> TIME: 2025-02-26 14:39:36 -- STEP: 25758/36938 -- GLOBAL_STEP: 1281650 | > loss_text_ce: 0.02836759202182293 (0.031191490449923056) | > loss_mel_ce: 0.19805908203125 (nan) | > loss: 0.0008985185995697975 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1368676067481585) | > loader_time: 0.0062 (0.004663643004666717)  --> TIME: 2025-02-26 14:39:45 -- STEP: 25808/36938 -- GLOBAL_STEP: 1281700 | > loss_text_ce: 0.02436494454741478 (0.031190230573197832) | > loss_mel_ce: 1.309091567993164 (nan) | > loss: 0.00529149454087019 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.1368609747473528) | > loader_time: 0.0046 (0.0046638231227949)  --> TIME: 2025-02-26 14:39:54 -- STEP: 25858/36938 -- GLOBAL_STEP: 1281750 | > loss_text_ce: 0.04279840365052223 (0.03119018982143024) | > loss_mel_ce: 2.519775629043579 (nan) | > loss: 0.010168945416808128 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13685732329838687) | > loader_time: 0.0045 (0.004663779717308963)  --> TIME: 2025-02-26 14:40:03 -- STEP: 25908/36938 -- GLOBAL_STEP: 1281800 | > loss_text_ce: 0.05152542144060135 (0.031188790547096827) | > loss_mel_ce: 3.192687749862671 (nan) | > loss: 0.012873861938714981 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.13684527474051936) | > loader_time: 0.0045 (0.004663830151689676)  --> TIME: 2025-02-26 14:40:12 -- STEP: 25958/36938 -- GLOBAL_STEP: 1281850 | > loss_text_ce: 0.021788183599710464 (0.031186721920987197) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13684634263530734) | > loader_time: 0.0046 (0.004663814068465117)  --> TIME: 2025-02-26 14:40:21 -- STEP: 26008/36938 -- GLOBAL_STEP: 1281900 | > loss_text_ce: 0.036584436893463135 (0.031185584855941997) | > loss_mel_ce: 2.3091208934783936 (nan) | > loss: 0.009308354929089546 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.13684002455107377) | > loader_time: 0.0046 (0.004663778805285346)  --> TIME: 2025-02-26 14:40:30 -- STEP: 26058/36938 -- GLOBAL_STEP: 1281950 | > loss_text_ce: 0.03385402634739876 (0.031187131498342072) | > loss_mel_ce: 2.6480910778045654 (nan) | > loss: 0.0106426402926445 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13683867397801702) | > loader_time: 0.0047 (0.004663767694960288)  --> TIME: 2025-02-26 14:40:40 -- STEP: 26108/36938 -- GLOBAL_STEP: 1282000 | > loss_text_ce: 0.03239886090159416 (0.03118771909749938) | > loss_mel_ce: 2.8520991802215576 (nan) | > loss: 0.011446421965956688 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13683597499458172) | > loader_time: 0.0047 (0.004663737422568457)  --> TIME: 2025-02-26 14:40:49 -- STEP: 26158/36938 -- GLOBAL_STEP: 1282050 | > loss_text_ce: 0.02718696929514408 (0.031186049652063497) | > loss_mel_ce: 2.9921672344207764 (nan) | > loss: 0.011981564573943615 (nan) | > current_lr: 5e-06 | > step_time: 0.1897 (0.1368493177500857) | > loader_time: 0.0049 (0.004663914184576122)  --> TIME: 2025-02-26 14:40:59 -- STEP: 26208/36938 -- GLOBAL_STEP: 1282100 | > loss_text_ce: 0.031849805265665054 (0.031186199022770575) | > loss_mel_ce: 2.910963296890259 (nan) | > loss: 0.01167783048003912 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13685052639034942) | > loader_time: 0.0046 (0.004663869956910805)  --> TIME: 2025-02-26 14:41:08 -- STEP: 26258/36938 -- GLOBAL_STEP: 1282150 | > loss_text_ce: 0.03776070848107338 (0.03118551996025032) | > loss_mel_ce: 2.840083360671997 (nan) | > loss: 0.01142001710832119 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13684461147480642) | > loader_time: 0.0045 (0.004663775731534958)  --> TIME: 2025-02-26 14:41:17 -- STEP: 26308/36938 -- GLOBAL_STEP: 1282200 | > loss_text_ce: 0.026982463896274567 (0.031185822445658245) | > loss_mel_ce: 2.505030393600464 (nan) | > loss: 0.010047671385109425 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.13684730257363736) | > loader_time: 0.0046 (0.004663769146111711)  --> TIME: 2025-02-26 14:41:27 -- STEP: 26358/36938 -- GLOBAL_STEP: 1282250 | > loss_text_ce: 0.028928954154253006 (0.03118324241967623) | > loss_mel_ce: 2.1083309650421143 (nan) | > loss: 0.008481190539896488 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.1368470468835174) | > loader_time: 0.0046 (0.0046637949591495365)  --> TIME: 2025-02-26 14:41:36 -- STEP: 26408/36938 -- GLOBAL_STEP: 1282300 | > loss_text_ce: 0.03693896532058716 (0.03118504296937606) | > loss_mel_ce: 2.526942729949951 (nan) | > loss: 0.010174133814871311 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13684115873723632) | > loader_time: 0.0046 (0.004663759923205138)  --> TIME: 2025-02-26 14:41:46 -- STEP: 26458/36938 -- GLOBAL_STEP: 1282350 | > loss_text_ce: 0.031146451830863953 (0.031184360195459306) | > loss_mel_ce: 1.8910506963729858 (nan) | > loss: 0.007627766579389572 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.13685422301770025) | > loader_time: 0.0046 (0.004663891033193055)  --> TIME: 2025-02-26 14:41:55 -- STEP: 26508/36938 -- GLOBAL_STEP: 1282400 | > loss_text_ce: 0.022928981110453606 (0.031182063083416377) | > loss_mel_ce: 0.16165751218795776 (nan) | > loss: 0.0007324861362576485 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13685602696211394) | > loader_time: 0.0046 (0.004663894641304013)  --> TIME: 2025-02-26 14:42:05 -- STEP: 26558/36938 -- GLOBAL_STEP: 1282450 | > loss_text_ce: 0.025301625952124596 (0.031182609424192802) | > loss_mel_ce: 2.7290687561035156 (nan) | > loss: 0.010930041782557964 (nan) | > current_lr: 5e-06 | > step_time: 0.187 (0.13685538222231572) | > loader_time: 0.0049 (0.0046638900934370285)  --> TIME: 2025-02-26 14:42:14 -- STEP: 26608/36938 -- GLOBAL_STEP: 1282500 | > loss_text_ce: 0.03360256925225258 (0.031182765012885034) | > loss_mel_ce: 2.4317667484283447 (nan) | > loss: 0.009783212095499039 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13685381613219294) | > loader_time: 0.0046 (0.0046639107414131725)  --> TIME: 2025-02-26 14:42:23 -- STEP: 26658/36938 -- GLOBAL_STEP: 1282550 | > loss_text_ce: 0.037050891667604446 (0.031183092573927203) | > loss_mel_ce: 3.00414776802063 (nan) | > loss: 0.0120682492852211 (nan) | > current_lr: 5e-06 | > step_time: 0.1615 (0.13685592572770813) | > loader_time: 0.0047 (0.0046639125929723415)  --> TIME: 2025-02-26 14:42:32 -- STEP: 26708/36938 -- GLOBAL_STEP: 1282600 | > loss_text_ce: 0.03498731926083565 (0.03118445546419557) | > loss_mel_ce: 0.10603637993335724 (nan) | > loss: 0.0005596178816631436 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.13685153238391654) | > loader_time: 0.0046 (0.004663912437982251)  --> TIME: 2025-02-26 14:42:41 -- STEP: 26758/36938 -- GLOBAL_STEP: 1282650 | > loss_text_ce: 0.029746446758508682 (0.03118355041368404) | > loss_mel_ce: 2.15887451171875 (nan) | > loss: 0.008685004897415638 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13683822044690777) | > loader_time: 0.0046 (0.004663923047068052)  --> TIME: 2025-02-26 14:42:51 -- STEP: 26808/36938 -- GLOBAL_STEP: 1282700 | > loss_text_ce: 0.031201209872961044 (0.031185878257933366) | > loss_mel_ce: 2.557642698287964 (nan) | > loss: 0.010273190215229988 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.13684874122906232) | > loader_time: 0.0045 (0.0046640167357991655)  --> TIME: 2025-02-26 14:43:00 -- STEP: 26858/36938 -- GLOBAL_STEP: 1282750 | > loss_text_ce: 0.03131328523159027 (0.03118623646800222) | > loss_mel_ce: 3.052231788635254 (nan) | > loss: 0.012236290611326694 (nan) | > current_lr: 5e-06 | > step_time: 0.1764 (0.1368486759729918) | > loader_time: 0.0048 (0.00466407059966267)  --> TIME: 2025-02-26 14:43:09 -- STEP: 26908/36938 -- GLOBAL_STEP: 1282800 | > loss_text_ce: 0.016581648960709572 (0.031185964111892738) | > loss_mel_ce: 0.02028656005859375 (nan) | > loss: 0.0001463024236727506 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13683660609487877) | > loader_time: 0.0045 (0.004664037439219536)  --> TIME: 2025-02-26 14:43:19 -- STEP: 26958/36938 -- GLOBAL_STEP: 1282850 | > loss_text_ce: 0.028568217530846596 (0.031186309433112083) | > loss_mel_ce: 2.7611136436462402 (nan) | > loss: 0.01107016671448946 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.136845091735497) | > loader_time: 0.0045 (0.0046640062501959205)  --> TIME: 2025-02-26 14:43:29 -- STEP: 27008/36938 -- GLOBAL_STEP: 1282900 | > loss_text_ce: 0.04047197848558426 (0.031187540859050757) | > loss_mel_ce: 2.964992046356201 (nan) | > loss: 0.011926445178687572 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.1368493562679015) | > loader_time: 0.0046 (0.0046639990997257574)  --> TIME: 2025-02-26 14:43:38 -- STEP: 27058/36938 -- GLOBAL_STEP: 1282950 | > loss_text_ce: 0.03039414994418621 (0.03118941025453658) | > loss_mel_ce: 2.8823416233062744 (nan) | > loss: 0.011558475904166698 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13685541809755092) | > loader_time: 0.0046 (0.004664074388608742)  --> TIME: 2025-02-26 14:43:47 -- STEP: 27108/36938 -- GLOBAL_STEP: 1283000 | > loss_text_ce: 0.02536499872803688 (0.031189903961152385) | > loss_mel_ce: 1.9523330926895142 (nan) | > loss: 0.007848008535802364 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13685706921413243) | > loader_time: 0.0045 (0.004664153287205599)  --> TIME: 2025-02-26 14:43:57 -- STEP: 27158/36938 -- GLOBAL_STEP: 1283050 | > loss_text_ce: 0.02894783392548561 (0.031189413496997984) | > loss_mel_ce: 2.6237990856170654 (nan) | > loss: 0.010526774451136589 (nan) | > current_lr: 5e-06 | > step_time: 0.1911 (0.13686503283501167) | > loader_time: 0.0049 (0.004664155658930054)  --> TIME: 2025-02-26 14:44:06 -- STEP: 27208/36938 -- GLOBAL_STEP: 1283100 | > loss_text_ce: 0.039188627153635025 (0.031188493216630996) | > loss_mel_ce: 2.0095202922821045 (nan) | > loss: 0.008129797875881195 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.13686598874617603) | > loader_time: 0.0049 (0.004664170000701011)  --> TIME: 2025-02-26 14:44:16 -- STEP: 27258/36938 -- GLOBAL_STEP: 1283150 | > loss_text_ce: 0.021054057404398918 (0.031189213264702125) | > loss_mel_ce: 1.1969375610351562 (nan) | > loss: 0.00483330013230443 (nan) | > current_lr: 5e-06 | > step_time: 0.1449 (0.13686975914818497) | > loader_time: 0.0046 (0.00466420419743231)  --> TIME: 2025-02-26 14:44:25 -- STEP: 27308/36938 -- GLOBAL_STEP: 1283200 | > loss_text_ce: 0.035744473338127136 (0.031190047363345616) | > loss_mel_ce: 2.9198997020721436 (nan) | > loss: 0.011728746816515923 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.13687356690527924) | > loader_time: 0.0046 (0.004664197863152282)  --> TIME: 2025-02-26 14:44:35 -- STEP: 27358/36938 -- GLOBAL_STEP: 1283250 | > loss_text_ce: 0.03399047628045082 (0.031190024501063934) | > loss_mel_ce: 1.528436541557312 (nan) | > loss: 0.006200107745826244 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.1368710974676498) | > loader_time: 0.0049 (0.004664205382359829)  --> TIME: 2025-02-26 14:44:44 -- STEP: 27408/36938 -- GLOBAL_STEP: 1283300 | > loss_text_ce: 0.0197809599339962 (0.031188323685345645) | > loss_mel_ce: 2.3992180824279785 (nan) | > loss: 0.009599202312529087 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13688376833838617) | > loader_time: 0.0046 (0.004664207115482286)  --> TIME: 2025-02-26 14:44:54 -- STEP: 27458/36938 -- GLOBAL_STEP: 1283350 | > loss_text_ce: 0.03173963353037834 (0.03119075129488879) | > loss_mel_ce: 2.4305431842803955 (nan) | > loss: 0.00977096427232027 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13688409309358038) | > loader_time: 0.0045 (0.004664154825176494)  --> TIME: 2025-02-26 14:45:03 -- STEP: 27508/36938 -- GLOBAL_STEP: 1283400 | > loss_text_ce: 0.0337892547249794 (0.03119151758758512) | > loss_mel_ce: 0.4634033143520355 (nan) | > loss: 0.0019729863852262497 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13688726381487987) | > loader_time: 0.0045 (0.004664079158722457)  --> TIME: 2025-02-26 14:45:13 -- STEP: 27558/36938 -- GLOBAL_STEP: 1283450 | > loss_text_ce: 0.036703988909721375 (0.03119019551333239) | > loss_mel_ce: 2.741997480392456 (nan) | > loss: 0.0110265938565135 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13688459883615897) | > loader_time: 0.0046 (0.004663971072747227)  --> TIME: 2025-02-26 14:45:22 -- STEP: 27608/36938 -- GLOBAL_STEP: 1283500 | > loss_text_ce: 0.02767924778163433 (0.03119101579312799) | > loss_mel_ce: 2.39400053024292 (nan) | > loss: 0.00960984081029892 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13688342645008714) | > loader_time: 0.0045 (0.0046638805377106456)  --> TIME: 2025-02-26 14:45:31 -- STEP: 27658/36938 -- GLOBAL_STEP: 1283550 | > loss_text_ce: 0.03339802473783493 (0.031189726573520905) | > loss_mel_ce: 3.620706796646118 (nan) | > loss: 0.014500416815280914 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13689291412551574) | > loader_time: 0.0047 (0.004663875730722614)  --> TIME: 2025-02-26 14:45:41 -- STEP: 27708/36938 -- GLOBAL_STEP: 1283600 | > loss_text_ce: 0.03638388216495514 (0.031193085977166597) | > loss_mel_ce: 3.247523307800293 (nan) | > loss: 0.013031378388404846 (nan) | > current_lr: 5e-06 | > step_time: 0.1928 (0.13689619617681087) | > loader_time: 0.0049 (0.004663839112357894)  --> TIME: 2025-02-26 14:45:50 -- STEP: 27758/36938 -- GLOBAL_STEP: 1283650 | > loss_text_ce: 0.024590108543634415 (0.031191962097571302) | > loss_mel_ce: 2.618147373199463 (nan) | > loss: 0.010487053543329239 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.1369006737430001) | > loader_time: 0.0044 (0.0046637744447993726)  --> TIME: 2025-02-26 14:46:00 -- STEP: 27808/36938 -- GLOBAL_STEP: 1283700 | > loss_text_ce: 0.03209796920418739 (0.03119236197503058) | > loss_mel_ce: 1.0133390426635742 (nan) | > loss: 0.004148559644818306 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13689847584595208) | > loader_time: 0.0044 (0.004663719440905726)  --> TIME: 2025-02-26 14:46:09 -- STEP: 27858/36938 -- GLOBAL_STEP: 1283750 | > loss_text_ce: 0.02348938211798668 (0.031192377469790502) | > loss_mel_ce: 1.3176761865615845 (nan) | > loss: 0.005322085693478584 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13689300727926418) | > loader_time: 0.0045 (0.004663704781686642)  --> TIME: 2025-02-26 14:46:18 -- STEP: 27908/36938 -- GLOBAL_STEP: 1283800 | > loss_text_ce: 0.03246704116463661 (0.031194003220022602) | > loss_mel_ce: 2.6406381130218506 (nan) | > loss: 0.010607561096549034 (nan) | > current_lr: 5e-06 | > step_time: 0.1502 (0.1369011548301187) | > loader_time: 0.0047 (0.004663975093226963)  --> TIME: 2025-02-26 14:46:28 -- STEP: 27958/36938 -- GLOBAL_STEP: 1283850 | > loss_text_ce: 0.036094605922698975 (0.031196395787657948) | > loss_mel_ce: 2.7248833179473877 (nan) | > loss: 0.010956262238323689 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.1369041488255732) | > loader_time: 0.0046 (0.004663972061875428)  --> TIME: 2025-02-26 14:46:37 -- STEP: 28008/36938 -- GLOBAL_STEP: 1283900 | > loss_text_ce: 0.027568260207772255 (0.031194482348659534) | > loss_mel_ce: 2.7060177326202393 (nan) | > loss: 0.010847564786672592 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13690300270919306) | > loader_time: 0.0046 (0.004663924810308175)  --> TIME: 2025-02-26 14:46:47 -- STEP: 28058/36938 -- GLOBAL_STEP: 1283950 | > loss_text_ce: 0.030333681032061577 (0.031195142489910222) | > loss_mel_ce: 2.0914883613586426 (nan) | > loss: 0.008419929072260857 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13690486645288988) | > loader_time: 0.0045 (0.004663873537955306)  --> TIME: 2025-02-26 14:46:56 -- STEP: 28108/36938 -- GLOBAL_STEP: 1284000 | > loss_text_ce: 0.030432147905230522 (0.03119255608665119) | > loss_mel_ce: 3.334181547164917 (nan) | > loss: 0.013351642526686192 (nan) | > current_lr: 5e-06 | > step_time: 0.1875 (0.13691064190759394) | > loader_time: 0.0049 (0.004664016037996991)  --> TIME: 2025-02-26 14:47:05 -- STEP: 28158/36938 -- GLOBAL_STEP: 1284050 | > loss_text_ce: 0.025158336386084557 (0.03119305428908004) | > loss_mel_ce: 2.6553292274475098 (nan) | > loss: 0.010636855848133564 (nan) | > current_lr: 5e-06 | > step_time: 0.1763 (0.13690691009842312) | > loader_time: 0.0048 (0.0046640792026108905)  --> TIME: 2025-02-26 14:47:15 -- STEP: 28208/36938 -- GLOBAL_STEP: 1284100 | > loss_text_ce: 0.02925926446914673 (0.03119341491890482) | > loss_mel_ce: 3.0557119846343994 (nan) | > loss: 0.012241950258612633 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.13689947954072115) | > loader_time: 0.0048 (0.004664141120588748)  --> TIME: 2025-02-26 14:47:24 -- STEP: 28258/36938 -- GLOBAL_STEP: 1284150 | > loss_text_ce: 0.04000012204051018 (0.0311945053946929) | > loss_mel_ce: 2.932025194168091 (nan) | > loss: 0.011793752200901508 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13689391835242617) | > loader_time: 0.0046 (0.004664151419988444)  --> TIME: 2025-02-26 14:47:33 -- STEP: 28308/36938 -- GLOBAL_STEP: 1284200 | > loss_text_ce: 0.03957454487681389 (0.03119476103181486) | > loss_mel_ce: 2.481389045715332 (nan) | > loss: 0.010003824718296528 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13689379218238817) | > loader_time: 0.0048 (0.004664180085738832)  --> TIME: 2025-02-26 14:47:42 -- STEP: 28358/36938 -- GLOBAL_STEP: 1284250 | > loss_text_ce: 0.03502633050084114 (0.031195621816426315) | > loss_mel_ce: 3.1362221240997314 (nan) | > loss: 0.012584320269525051 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13689322578057017) | > loader_time: 0.0046 (0.004664199898244426)  --> TIME: 2025-02-26 14:47:52 -- STEP: 28408/36938 -- GLOBAL_STEP: 1284300 | > loss_text_ce: 0.029750900343060493 (0.03119438935047757) | > loss_mel_ce: 2.629471778869629 (nan) | > loss: 0.01055247150361538 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13688838413922563) | > loader_time: 0.0047 (0.004664232649624528)  --> TIME: 2025-02-26 14:48:01 -- STEP: 28458/36938 -- GLOBAL_STEP: 1284350 | > loss_text_ce: 0.03561604768037796 (0.031192213242662928) | > loss_mel_ce: 2.9778637886047363 (nan) | > loss: 0.011958254501223564 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.13688837576250942) | > loader_time: 0.0047 (0.0046643321165105505)  --> TIME: 2025-02-26 14:48:11 -- STEP: 28508/36938 -- GLOBAL_STEP: 1284400 | > loss_text_ce: 0.02591753751039505 (0.031192532651314823) | > loss_mel_ce: 2.2757961750030518 (nan) | > loss: 0.009133785031735897 (nan) | > current_lr: 5e-06 | > step_time: 0.1263 (0.13689563203965263) | > loader_time: 0.0046 (0.004664463499777574)  --> TIME: 2025-02-26 14:48:20 -- STEP: 28558/36938 -- GLOBAL_STEP: 1284450 | > loss_text_ce: 0.030624035745859146 (0.03119187106708602) | > loss_mel_ce: 2.67536997795105 (nan) | > loss: 0.010738072916865349 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13689391108251814) | > loader_time: 0.0045 (0.004664477551298132)  --> TIME: 2025-02-26 14:48:29 -- STEP: 28608/36938 -- GLOBAL_STEP: 1284500 | > loss_text_ce: 0.02809915691614151 (0.031190753983138645) | > loss_mel_ce: 2.960413932800293 (nan) | > loss: 0.011859179474413395 (nan) | > current_lr: 5e-06 | > step_time: 0.187 (0.1368985125165792) | > loader_time: 0.0049 (0.004664523972894217)  --> TIME: 2025-02-26 14:48:39 -- STEP: 28658/36938 -- GLOBAL_STEP: 1284550 | > loss_text_ce: 0.02896174043416977 (0.031193886489845737) | > loss_mel_ce: 2.8557796478271484 (nan) | > loss: 0.01144738681614399 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13689887724981642) | > loader_time: 0.0046 (0.0046645145754361525)  --> TIME: 2025-02-26 14:48:48 -- STEP: 28708/36938 -- GLOBAL_STEP: 1284600 | > loss_text_ce: 0.033710867166519165 (0.03119736239822109) | > loss_mel_ce: 3.1155261993408203 (nan) | > loss: 0.012496973387897015 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13689554510000992) | > loader_time: 0.0046 (0.004664389946275652)  --> TIME: 2025-02-26 14:48:58 -- STEP: 28758/36938 -- GLOBAL_STEP: 1284650 | > loss_text_ce: 0.035101741552352905 (0.03119644075584814) | > loss_mel_ce: 2.0495593547821045 (nan) | > loss: 0.008272464387118816 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.13690735511334098) | > loader_time: 0.0046 (0.004664346102141394)  --> TIME: 2025-02-26 14:49:07 -- STEP: 28808/36938 -- GLOBAL_STEP: 1284700 | > loss_text_ce: 0.027868937700986862 (0.031198417646682394) | > loss_mel_ce: 2.509006977081299 (nan) | > loss: 0.010066968388855457 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.1369032195229494) | > loader_time: 0.0046 (0.004664366434296409)  --> TIME: 2025-02-26 14:49:16 -- STEP: 28858/36938 -- GLOBAL_STEP: 1284750 | > loss_text_ce: 0.03301527351140976 (0.031195260738843322) | > loss_mel_ce: 2.50142765045166 (nan) | > loss: 0.010057313367724419 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13690239180728098) | > loader_time: 0.0046 (0.00466430543308253)  --> TIME: 2025-02-26 14:49:26 -- STEP: 28908/36938 -- GLOBAL_STEP: 1284800 | > loss_text_ce: 0.02708473429083824 (0.031192517884282348) | > loss_mel_ce: 2.1648876667022705 (nan) | > loss: 0.008698304183781147 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.1369032895678852) | > loader_time: 0.0047 (0.004664332907576238)  --> TIME: 2025-02-26 14:49:35 -- STEP: 28958/36938 -- GLOBAL_STEP: 1284850 | > loss_text_ce: 0.023108236491680145 (0.03119309986375159) | > loss_mel_ce: 1.1784471273422241 (nan) | > loss: 0.0047680772840976715 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.1368965240331374) | > loader_time: 0.0046 (0.0046643080389608795)  --> TIME: 2025-02-26 14:49:44 -- STEP: 29008/36938 -- GLOBAL_STEP: 1284900 | > loss_text_ce: 0.02802998758852482 (0.031191127564127768) | > loss_mel_ce: 2.8865156173706055 (nan) | > loss: 0.011565657332539558 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13690106884688907) | > loader_time: 0.0046 (0.004664349797609653)  --> TIME: 2025-02-26 14:49:54 -- STEP: 29058/36938 -- GLOBAL_STEP: 1284950 | > loss_text_ce: 0.029995646327733994 (0.03118962749857547) | > loss_mel_ce: 3.1490275859832764 (nan) | > loss: 0.012615172192454338 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13690648223160978) | > loader_time: 0.0046 (0.004664424921445713)  --> TIME: 2025-02-26 14:50:03 -- STEP: 29108/36938 -- GLOBAL_STEP: 1285000 | > loss_text_ce: 0.033509254455566406 (0.031185738523822844) | > loss_mel_ce: 2.668911933898926 (nan) | > loss: 0.010723894461989403 (nan) | > current_lr: 5e-06 | > step_time: 0.1886 (0.13690432334021602) | > loader_time: 0.0049 (0.004664614573438428)  --> TIME: 2025-02-26 14:50:13 -- STEP: 29158/36938 -- GLOBAL_STEP: 1285050 | > loss_text_ce: 0.03353822976350784 (0.031184698909964954) | > loss_mel_ce: 2.958209753036499 (nan) | > loss: 0.011872016824781895 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13691510416792707) | > loader_time: 0.0046 (0.0046646061629798605)  --> TIME: 2025-02-26 14:50:22 -- STEP: 29208/36938 -- GLOBAL_STEP: 1285100 | > loss_text_ce: 0.03048356994986534 (0.03118276134109999) | > loss_mel_ce: 2.9974687099456787 (nan) | > loss: 0.012015683576464653 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13691668383424102) | > loader_time: 0.0046 (0.00466458869613836)  --> TIME: 2025-02-26 14:50:31 -- STEP: 29258/36938 -- GLOBAL_STEP: 1285150 | > loss_text_ce: 0.03706202283501625 (0.031185471300332133) | > loss_mel_ce: 2.5975077152252197 (nan) | > loss: 0.010454641655087471 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13691295330030145) | > loader_time: 0.0047 (0.0046645803912435755)  --> TIME: 2025-02-26 14:50:40 -- STEP: 29308/36938 -- GLOBAL_STEP: 1285200 | > loss_text_ce: 0.02732354775071144 (0.031184707640326926) | > loss_mel_ce: 3.337181329727173 (nan) | > loss: 0.01335121039301157 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.136906643038771) | > loader_time: 0.0047 (0.004664549279932042)  --> TIME: 2025-02-26 14:50:50 -- STEP: 29358/36938 -- GLOBAL_STEP: 1285250 | > loss_text_ce: 0.03220932558178902 (0.031184961426295347) | > loss_mel_ce: 2.4785172939300537 (nan) | > loss: 0.009963201358914375 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.13690687105403757) | > loader_time: 0.0047 (0.004664506011766389)  --> TIME: 2025-02-26 14:50:59 -- STEP: 29408/36938 -- GLOBAL_STEP: 1285300 | > loss_text_ce: 0.02875695936381817 (0.031183414390172318) | > loss_mel_ce: 2.4854676723480225 (nan) | > loss: 0.009977081790566444 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13690371948402785) | > loader_time: 0.0046 (0.004664421803089911)  --> TIME: 2025-02-26 14:51:08 -- STEP: 29458/36938 -- GLOBAL_STEP: 1285350 | > loss_text_ce: 0.03683904558420181 (0.031180535269432874) | > loss_mel_ce: 1.3333650827407837 (nan) | > loss: 0.0054373182356357574 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13689421713898525) | > loader_time: 0.0046 (0.004664333614994479)  --> TIME: 2025-02-26 14:51:17 -- STEP: 29508/36938 -- GLOBAL_STEP: 1285400 | > loss_text_ce: 0.03537391871213913 (0.031179636222176167) | > loss_mel_ce: 2.613337755203247 (nan) | > loss: 0.01051076129078865 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13689135683411843) | > loader_time: 0.0047 (0.004664284686530264)  --> TIME: 2025-02-26 14:51:27 -- STEP: 29558/36938 -- GLOBAL_STEP: 1285450 | > loss_text_ce: 0.030979570001363754 (0.031179464522646347) | > loss_mel_ce: 0.0786946639418602 (nan) | > loss: 0.00043521521729417145 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.1368929856087602) | > loader_time: 0.0045 (0.00466422109805854)  --> TIME: 2025-02-26 14:51:36 -- STEP: 29608/36938 -- GLOBAL_STEP: 1285500 | > loss_text_ce: 0.0394282341003418 (0.03118130403403616) | > loss_mel_ce: 2.1834659576416016 (nan) | > loss: 0.008821009658277035 (nan) | > current_lr: 5e-06 | > step_time: 0.1357 (0.136894547369183) | > loader_time: 0.0046 (0.004664175166081328)  --> TIME: 2025-02-26 14:51:45 -- STEP: 29658/36938 -- GLOBAL_STEP: 1285550 | > loss_text_ce: 0.02676353231072426 (0.031179923241350962) | > loss_mel_ce: 2.7650320529937744 (nan) | > loss: 0.011078554205596447 (nan) | > current_lr: 5e-06 | > step_time: 0.1518 (0.13688799496309254) | > loader_time: 0.0047 (0.0046641788846668395)  --> TIME: 2025-02-26 14:51:54 -- STEP: 29708/36938 -- GLOBAL_STEP: 1285600 | > loss_text_ce: 0.02303135022521019 (0.031178711556050925) | > loss_mel_ce: 2.9604580402374268 (nan) | > loss: 0.011839243583381176 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13688287108843658) | > loader_time: 0.0046 (0.004664085162380907)  --> TIME: 2025-02-26 14:52:04 -- STEP: 29758/36938 -- GLOBAL_STEP: 1285650 | > loss_text_ce: 0.023510003462433815 (0.03117850929121487) | > loss_mel_ce: 2.9863178730010986 (nan) | > loss: 0.011943762190639973 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.13688514120308443) | > loader_time: 0.0047 (0.00466404270281334)  --> TIME: 2025-02-26 14:52:13 -- STEP: 29808/36938 -- GLOBAL_STEP: 1285700 | > loss_text_ce: 0.043545618653297424 (0.031177224349951644) | > loss_mel_ce: 2.7338883876800537 (nan) | > loss: 0.011021564714610577 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1368803672584487) | > loader_time: 0.0045 (0.004663967128025188)  --> TIME: 2025-02-26 14:52:22 -- STEP: 29858/36938 -- GLOBAL_STEP: 1285750 | > loss_text_ce: 0.02673286944627762 (0.031177651513005073) | > loss_mel_ce: 2.1320950984954834 (nan) | > loss: 0.008566778153181076 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13687729262979895) | > loader_time: 0.0046 (0.004663867723343684)  --> TIME: 2025-02-26 14:52:31 -- STEP: 29908/36938 -- GLOBAL_STEP: 1285800 | > loss_text_ce: 0.024799445644021034 (0.03117866360815834) | > loss_mel_ce: 1.3905181884765625 (nan) | > loss: 0.005616340320557356 (nan) | > current_lr: 5e-06 | > step_time: 0.1613 (0.1368724502582229) | > loader_time: 0.0048 (0.004663831555972459)  --> TIME: 2025-02-26 14:52:40 -- STEP: 29958/36938 -- GLOBAL_STEP: 1285850 | > loss_text_ce: 0.027489211410284042 (0.031178793103136766) | > loss_mel_ce: 3.078568458557129 (nan) | > loss: 0.012325625866651535 (nan) | > current_lr: 5e-06 | > step_time: 0.157 (0.13686314418785614) | > loader_time: 0.0047 (0.004663816089822367)  --> TIME: 2025-02-26 14:52:50 -- STEP: 30008/36938 -- GLOBAL_STEP: 1285900 | > loss_text_ce: 0.02863989770412445 (0.031180159770945695) | > loss_mel_ce: 2.734717607498169 (nan) | > loss: 0.010965704917907715 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.1368654140851494) | > loader_time: 0.0045 (0.00466375178859822)  --> TIME: 2025-02-26 14:52:59 -- STEP: 30058/36938 -- GLOBAL_STEP: 1285950 | > loss_text_ce: 0.031364552676677704 (0.03118095867805145) | > loss_mel_ce: 3.1477465629577637 (nan) | > loss: 0.012615520507097244 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13686242254442388) | > loader_time: 0.0045 (0.004663800144633364)  --> TIME: 2025-02-26 14:53:08 -- STEP: 30108/36938 -- GLOBAL_STEP: 1286000 | > loss_text_ce: 0.034619834274053574 (0.031180129314132977) | > loss_mel_ce: 3.0487446784973145 (nan) | > loss: 0.012235574424266815 (nan) | > current_lr: 5e-06 | > step_time: 0.1885 (0.13686572633544133) | > loader_time: 0.0048 (0.004663692300532176)  --> TIME: 2025-02-26 14:53:18 -- STEP: 30158/36938 -- GLOBAL_STEP: 1286050 | > loss_text_ce: 0.011235254816710949 (0.03117943757853331) | > loss_mel_ce: 1.092315673828125 (nan) | > loss: 0.00437917048111558 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1368711004760636) | > loader_time: 0.0045 (0.004663614033308713)  --> TIME: 2025-02-26 14:53:27 -- STEP: 30208/36938 -- GLOBAL_STEP: 1286100 | > loss_text_ce: 0.02386460080742836 (0.03117791902942542) | > loss_mel_ce: 2.70174503326416 (nan) | > loss: 0.010815911926329136 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13687624852567679) | > loader_time: 0.0045 (0.0046635328444762396)  --> TIME: 2025-02-26 14:53:37 -- STEP: 30258/36938 -- GLOBAL_STEP: 1286150 | > loss_text_ce: 0.025857221335172653 (0.03118028117205769) | > loss_mel_ce: 2.6381478309631348 (nan) | > loss: 0.010571449063718319 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.1368764745589488) | > loader_time: 0.0046 (0.004663491723431666)  --> TIME: 2025-02-26 14:53:46 -- STEP: 30308/36938 -- GLOBAL_STEP: 1286200 | > loss_text_ce: 0.03348330780863762 (0.031181711110784098) | > loss_mel_ce: 2.3993406295776367 (nan) | > loss: 0.009654063731431961 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13687385395867432) | > loader_time: 0.0046 (0.004663488473775123)  --> TIME: 2025-02-26 14:53:55 -- STEP: 30358/36938 -- GLOBAL_STEP: 1286250 | > loss_text_ce: 0.031724732369184494 (0.031184368106469754) | > loss_mel_ce: 2.620565891265869 (nan) | > loss: 0.010524963028728962 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.1368691162108372) | > loader_time: 0.0046 (0.004663434697121054)  --> TIME: 2025-02-26 14:54:04 -- STEP: 30408/36938 -- GLOBAL_STEP: 1286300 | > loss_text_ce: 0.033173851668834686 (0.031180971862576266) | > loss_mel_ce: 3.2671947479248047 (nan) | > loss: 0.013096701353788376 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.13686682659053598) | > loader_time: 0.0047 (0.004663426925934634)  --> TIME: 2025-02-26 14:54:14 -- STEP: 30458/36938 -- GLOBAL_STEP: 1286350 | > loss_text_ce: 0.04246746748685837 (0.031179950377681342) | > loss_mel_ce: 2.8372349739074707 (nan) | > loss: 0.011427391320466995 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13686700016446077) | > loader_time: 0.0046 (0.004663369458192438)  --> TIME: 2025-02-26 14:54:22 -- STEP: 30508/36938 -- GLOBAL_STEP: 1286400 | > loss_text_ce: 0.040986936539411545 (0.031181291271748655) | > loss_mel_ce: 3.072755813598633 (nan) | > loss: 0.012356122955679893 (nan) | > current_lr: 5e-06 | > step_time: 0.1077 (0.13685377723066922) | > loader_time: 0.0044 (0.004663272189706337)  --> TIME: 2025-02-26 14:54:32 -- STEP: 30558/36938 -- GLOBAL_STEP: 1286450 | > loss_text_ce: 0.03419886529445648 (0.031179664410843105) | > loss_mel_ce: 2.338454484939575 (nan) | > loss: 0.009415291249752045 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.13685034882437946) | > loader_time: 0.0048 (0.004663208429940837)  --> TIME: 2025-02-26 14:54:41 -- STEP: 30608/36938 -- GLOBAL_STEP: 1286500 | > loss_text_ce: 0.03188467025756836 (0.031179614462194773) | > loss_mel_ce: 2.766740083694458 (nan) | > loss: 0.011105654761195183 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13684913881368455) | > loader_time: 0.0046 (0.004663158790390929)  --> TIME: 2025-02-26 14:54:50 -- STEP: 30658/36938 -- GLOBAL_STEP: 1286550 | > loss_text_ce: 0.02976961061358452 (0.031177894686715644) | > loss_mel_ce: 2.8446993827819824 (nan) | > loss: 0.011406623758375645 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13684803170851145) | > loader_time: 0.0046 (0.004663099094149196)  --> TIME: 2025-02-26 14:54:59 -- STEP: 30708/36938 -- GLOBAL_STEP: 1286600 | > loss_text_ce: 0.039918120950460434 (0.031177338657220614) | > loss_mel_ce: 2.844416618347168 (nan) | > loss: 0.011445773765444756 (nan) | > current_lr: 5e-06 | > step_time: 0.1662 (0.13684476122223235) | > loader_time: 0.0046 (0.004663015298581234)  --> TIME: 2025-02-26 14:55:09 -- STEP: 30758/36938 -- GLOBAL_STEP: 1286650 | > loss_text_ce: 0.0349319726228714 (0.03117918914800577) | > loss_mel_ce: 2.6437716484069824 (nan) | > loss: 0.010629776865243912 (nan) | > current_lr: 5e-06 | > step_time: 0.1797 (0.13684775731027035) | > loader_time: 0.0048 (0.004662965408916501)  --> TIME: 2025-02-26 14:55:19 -- STEP: 30808/36938 -- GLOBAL_STEP: 1286700 | > loss_text_ce: 0.031992580741643906 (0.031178546130731687) | > loss_mel_ce: 2.4323558807373047 (nan) | > loss: 0.009779161773622036 (nan) | > current_lr: 5e-06 | > step_time: 0.1579 (0.13685638067692005) | > loader_time: 0.0047 (0.004662921152558115)  --> TIME: 2025-02-26 14:55:28 -- STEP: 30858/36938 -- GLOBAL_STEP: 1286750 | > loss_text_ce: 0.024825045838952065 (0.031178809195175995) | > loss_mel_ce: 2.2753422260284424 (nan) | > loss: 0.009127648547291756 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1368604317047677) | > loader_time: 0.0045 (0.004662853767963147)  --> TIME: 2025-02-26 14:55:37 -- STEP: 30908/36938 -- GLOBAL_STEP: 1286800 | > loss_text_ce: 0.025765370577573776 (0.031178282209190767) | > loss_mel_ce: 2.554053783416748 (nan) | > loss: 0.010237378068268299 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13686111338375878) | > loader_time: 0.0045 (0.004662760374415499)  --> TIME: 2025-02-26 14:55:47 -- STEP: 30958/36938 -- GLOBAL_STEP: 1286850 | > loss_text_ce: 0.03865458443760872 (0.0311779717065084) | > loss_mel_ce: 2.7829532623291016 (nan) | > loss: 0.011196857318282127 (nan) | > current_lr: 5e-06 | > step_time: 0.1799 (0.1368638069502291) | > loader_time: 0.005 (0.004662683917475047)  --> TIME: 2025-02-26 14:55:56 -- STEP: 31008/36938 -- GLOBAL_STEP: 1286900 | > loss_text_ce: 0.03656955808401108 (0.031178113278579424) | > loss_mel_ce: 0.0172119140625 (nan) | > loss: 0.00021341854881029576 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13686930672702918) | > loader_time: 0.0047 (0.004662624691969558)  --> TIME: 2025-02-26 14:56:06 -- STEP: 31058/36938 -- GLOBAL_STEP: 1286950 | > loss_text_ce: 0.05349833518266678 (0.0311762000627454) | > loss_mel_ce: 2.2401678562164307 (nan) | > loss: 0.009101849980652332 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13686305112929043) | > loader_time: 0.0044 (0.00466249580046757)  --> TIME: 2025-02-26 14:56:15 -- STEP: 31108/36938 -- GLOBAL_STEP: 1287000 | > loss_text_ce: 0.02806548774242401 (0.031174272357761504) | > loss_mel_ce: 3.0029289722442627 (nan) | > loss: 0.012027756310999393 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13686634457416416) | > loader_time: 0.0045 (0.004662424398753056)  --> TIME: 2025-02-26 14:56:24 -- STEP: 31158/36938 -- GLOBAL_STEP: 1287050 | > loss_text_ce: 0.026805179193615913 (0.031173330891692812) | > loss_mel_ce: 1.8466527462005615 (nan) | > loss: 0.007434357423335314 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13686485637618537) | > loader_time: 0.0046 (0.004662332642529063)  --> TIME: 2025-02-26 14:56:33 -- STEP: 31208/36938 -- GLOBAL_STEP: 1287100 | > loss_text_ce: 0.030764520168304443 (0.03117339146208072) | > loss_mel_ce: 2.3843417167663574 (nan) | > loss: 0.009583755396306515 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13686592522048874) | > loader_time: 0.0045 (0.004662253976754392)  --> TIME: 2025-02-26 14:56:43 -- STEP: 31258/36938 -- GLOBAL_STEP: 1287150 | > loss_text_ce: 0.03167954832315445 (0.03117381378560062) | > loss_mel_ce: 2.9604990482330322 (nan) | > loss: 0.011873725801706314 (nan) | > current_lr: 5e-06 | > step_time: 0.1819 (0.1368641772252814) | > loader_time: 0.0048 (0.004662154381239599)  --> TIME: 2025-02-26 14:56:52 -- STEP: 31308/36938 -- GLOBAL_STEP: 1287200 | > loss_text_ce: 0.024460820481181145 (0.031174283161892885) | > loss_mel_ce: 2.1999971866607666 (nan) | > loss: 0.008827215060591698 (nan) | > current_lr: 5e-06 | > step_time: 0.156 (0.13686489630159773) | > loader_time: 0.0048 (0.004662108951347788)  --> TIME: 2025-02-26 14:57:01 -- STEP: 31358/36938 -- GLOBAL_STEP: 1287250 | > loss_text_ce: 0.022576507180929184 (0.031176013233513974) | > loss_mel_ce: 2.7126517295837402 (nan) | > loss: 0.010854081250727177 (nan) | > current_lr: 5e-06 | > step_time: 0.182 (0.13686433948647275) | > loader_time: 0.0049 (0.004662055333313445)  --> TIME: 2025-02-26 14:57:11 -- STEP: 31408/36938 -- GLOBAL_STEP: 1287300 | > loss_text_ce: 0.03052387200295925 (0.03117356968510937) | > loss_mel_ce: 2.6755008697509766 (nan) | > loss: 0.010738193988800049 (nan) | > current_lr: 5e-06 | > step_time: 0.1867 (0.13686484033910845) | > loader_time: 0.0049 (0.004661999327821838)  --> TIME: 2025-02-26 14:57:20 -- STEP: 31458/36938 -- GLOBAL_STEP: 1287350 | > loss_text_ce: 0.029443081468343735 (0.03117412481965851) | > loss_mel_ce: 3.1188292503356934 (nan) | > loss: 0.01249314472079277 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.1368705601919267) | > loader_time: 0.0045 (0.004661978833423275)  --> TIME: 2025-02-26 14:57:30 -- STEP: 31508/36938 -- GLOBAL_STEP: 1287400 | > loss_text_ce: 0.03291799873113632 (0.031173720545018036) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13686628114364488) | > loader_time: 0.0045 (0.004661980128702902)  --> TIME: 2025-02-26 14:57:39 -- STEP: 31558/36938 -- GLOBAL_STEP: 1287450 | > loss_text_ce: 0.03050326369702816 (0.031173837979195082) | > loss_mel_ce: 2.947735071182251 (nan) | > loss: 0.011818407103419304 (nan) | > current_lr: 5e-06 | > step_time: 0.1857 (0.1368658734830557) | > loader_time: 0.0048 (0.004661879360286699)  --> TIME: 2025-02-26 14:57:48 -- STEP: 31608/36938 -- GLOBAL_STEP: 1287500 | > loss_text_ce: 0.03059719316661358 (0.031174388961491474) | > loss_mel_ce: 2.7858076095581055 (nan) | > loss: 0.011176209896802902 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.1368681619708011) | > loader_time: 0.0047 (0.004661802497582292)  --> TIME: 2025-02-26 14:57:57 -- STEP: 31658/36938 -- GLOBAL_STEP: 1287550 | > loss_text_ce: 0.022408640012145042 (0.031175740881635258) | > loss_mel_ce: 2.3824145793914795 (nan) | > loss: 0.009542950429022312 (nan) | > current_lr: 5e-06 | > step_time: 0.1583 (0.13686448217786376) | > loader_time: 0.0047 (0.004661709467469539)  --> TIME: 2025-02-26 14:58:07 -- STEP: 31708/36938 -- GLOBAL_STEP: 1287600 | > loss_text_ce: 0.0322064645588398 (0.031177521606703295) | > loss_mel_ce: 3.034069061279297 (nan) | > loss: 0.012167761102318764 (nan) | > current_lr: 5e-06 | > step_time: 0.1664 (0.13685963785300978) | > loader_time: 0.0049 (0.004661609625115522)  --> TIME: 2025-02-26 14:58:16 -- STEP: 31758/36938 -- GLOBAL_STEP: 1287650 | > loss_text_ce: 0.02598302997648716 (0.031175975898975524) | > loss_mel_ce: 2.0112669467926025 (nan) | > loss: 0.008084326051175594 (nan) | > current_lr: 5e-06 | > step_time: 0.1726 (0.1368583452191821) | > loader_time: 0.0047 (0.004661541387801909)  --> TIME: 2025-02-26 14:58:25 -- STEP: 31808/36938 -- GLOBAL_STEP: 1287700 | > loss_text_ce: 0.03205868601799011 (0.031173811100922964) | > loss_mel_ce: 2.906672239303589 (nan) | > loss: 0.011661631055176258 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1368579519481608) | > loader_time: 0.0046 (0.004661451762830673)  --> TIME: 2025-02-26 14:58:34 -- STEP: 31858/36938 -- GLOBAL_STEP: 1287750 | > loss_text_ce: 0.031134378165006638 (0.031174779232412252) | > loss_mel_ce: 0.04046630859375 (nan) | > loss: 0.0002841297537088394 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13685050768938767) | > loader_time: 0.0045 (0.004661292737617964)  --> TIME: 2025-02-26 14:58:44 -- STEP: 31908/36938 -- GLOBAL_STEP: 1287800 | > loss_text_ce: 0.03189808875322342 (0.03117447572189839) | > loss_mel_ce: 1.5965834856033325 (nan) | > loss: 0.006462228950113058 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13684783474132076) | > loader_time: 0.0046 (0.004661222986369764)  --> TIME: 2025-02-26 14:58:53 -- STEP: 31958/36938 -- GLOBAL_STEP: 1287850 | > loss_text_ce: 0.03451450914144516 (0.031175944754795914) | > loss_mel_ce: 2.881892681121826 (nan) | > loss: 0.011573044583201408 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13684214162650493) | > loader_time: 0.0047 (0.004661135876743458)  --> TIME: 2025-02-26 14:59:02 -- STEP: 32008/36938 -- GLOBAL_STEP: 1287900 | > loss_text_ce: 0.03242674842476845 (0.03117384626886835) | > loss_mel_ce: 3.0140380859375 (nan) | > loss: 0.012089147232472897 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.1368405874730112) | > loader_time: 0.0046 (0.004661151928414194)  --> TIME: 2025-02-26 14:59:12 -- STEP: 32058/36938 -- GLOBAL_STEP: 1287950 | > loss_text_ce: 0.039298735558986664 (0.0311724973586096) | > loss_mel_ce: 2.5575642585754395 (nan) | > loss: 0.010305012576282024 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13684856468461062) | > loader_time: 0.0046 (0.0046611371106682845)  --> TIME: 2025-02-26 14:59:21 -- STEP: 32108/36938 -- GLOBAL_STEP: 1288000 | > loss_text_ce: 0.0375853106379509 (0.03117244790103512) | > loss_mel_ce: 2.6223342418670654 (nan) | > loss: 0.010555236600339413 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1368504237368291) | > loader_time: 0.0045 (0.00466104846257888)  --> TIME: 2025-02-26 14:59:30 -- STEP: 32158/36938 -- GLOBAL_STEP: 1288050 | > loss_text_ce: 0.03463487699627876 (0.031173803438714308) | > loss_mel_ce: 3.416444778442383 (nan) | > loss: 0.0136947613209486 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.1368424120408416) | > loader_time: 0.0045 (0.004660911135682682)  --> TIME: 2025-02-26 14:59:40 -- STEP: 32208/36938 -- GLOBAL_STEP: 1288100 | > loss_text_ce: 0.04089263826608658 (0.0311742719344439) | > loss_mel_ce: 2.4970502853393555 (nan) | > loss: 0.01007120218127966 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13684571682074406) | > loader_time: 0.0046 (0.004660845328431397)  --> TIME: 2025-02-26 14:59:49 -- STEP: 32258/36938 -- GLOBAL_STEP: 1288150 | > loss_text_ce: 0.018060075119137764 (0.031174794064215836) | > loss_mel_ce: 1.7723642587661743 (nan) | > loss: 0.007104858756065369 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.13684740069632556) | > loader_time: 0.0045 (0.004660806288403696)  --> TIME: 2025-02-26 14:59:58 -- STEP: 32308/36938 -- GLOBAL_STEP: 1288200 | > loss_text_ce: 0.024574611335992813 (0.031173920365335474) | > loss_mel_ce: 2.2398979663848877 (nan) | > loss: 0.008986002765595913 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13684614496950162) | > loader_time: 0.0047 (0.004660751466277593)  --> TIME: 2025-02-26 15:00:08 -- STEP: 32358/36938 -- GLOBAL_STEP: 1288250 | > loss_text_ce: 0.03461889550089836 (0.031174830441514034) | > loss_mel_ce: 2.8527698516845703 (nan) | > loss: 0.011457892134785652 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.1368449621113914) | > loader_time: 0.0046 (0.004660925852500089)  --> TIME: 2025-02-26 15:00:17 -- STEP: 32408/36938 -- GLOBAL_STEP: 1288300 | > loss_text_ce: 0.048266563564538956 (0.031178207945103214) | > loss_mel_ce: 2.301694631576538 (nan) | > loss: 0.009325243532657623 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13684337074684252) | > loader_time: 0.0045 (0.004660880645096519)  --> TIME: 2025-02-26 15:00:26 -- STEP: 32458/36938 -- GLOBAL_STEP: 1288350 | > loss_text_ce: 0.024439165368676186 (0.03117812097167748) | > loss_mel_ce: 2.728153944015503 (nan) | > loss: 0.010922988876700401 (nan) | > current_lr: 5e-06 | > step_time: 0.1349 (0.13684977893211173) | > loader_time: 0.0048 (0.0046608720324353365)  --> TIME: 2025-02-26 15:00:35 -- STEP: 32508/36938 -- GLOBAL_STEP: 1288400 | > loss_text_ce: 0.0404818132519722 (0.031176727250823693) | > loss_mel_ce: 1.652742624282837 (nan) | > loss: 0.006719144992530346 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.1368434706399609) | > loader_time: 0.0044 (0.00466081963205035)  --> TIME: 2025-02-26 15:00:45 -- STEP: 32558/36938 -- GLOBAL_STEP: 1288450 | > loss_text_ce: 0.026458391919732094 (0.03117497407726157) | > loss_mel_ce: 1.3550164699554443 (nan) | > loss: 0.005482043605297804 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13684127077993963) | > loader_time: 0.0047 (0.0046608466335783475)  --> TIME: 2025-02-26 15:00:54 -- STEP: 32608/36938 -- GLOBAL_STEP: 1288500 | > loss_text_ce: 0.03280147537589073 (0.031174813462091135) | > loss_mel_ce: 2.7536780834198 (nan) | > loss: 0.011057458817958832 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13683368675812907) | > loader_time: 0.0045 (0.00466089522405739)  --> TIME: 2025-02-26 15:01:03 -- STEP: 32658/36938 -- GLOBAL_STEP: 1288550 | > loss_text_ce: 0.02791835181415081 (0.031174044963280394) | > loss_mel_ce: 2.49539852142334 (nan) | > loss: 0.010013163089752197 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.1368297141446175) | > loader_time: 0.0047 (0.004660931765992987)  --> TIME: 2025-02-26 15:01:12 -- STEP: 32708/36938 -- GLOBAL_STEP: 1288600 | > loss_text_ce: 0.02990577556192875 (0.03117145574327057) | > loss_mel_ce: 2.7720601558685303 (nan) | > loss: 0.011118913069367409 (nan) | > current_lr: 5e-06 | > step_time: 0.2017 (0.1368313960214402) | > loader_time: 0.0049 (0.004661048014093669)  --> TIME: 2025-02-26 15:01:22 -- STEP: 32758/36938 -- GLOBAL_STEP: 1288650 | > loss_text_ce: 0.026921121403574944 (0.031171201488758363) | > loss_mel_ce: 1.2332245111465454 (nan) | > loss: 0.005000578239560127 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13683420417873468) | > loader_time: 0.0046 (0.004661070171661037)  --> TIME: 2025-02-26 15:01:31 -- STEP: 32808/36938 -- GLOBAL_STEP: 1288700 | > loss_text_ce: 0.028239667415618896 (0.031173380820964727) | > loss_mel_ce: 2.7177584171295166 (nan) | > loss: 0.010896818712353706 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13682889822807834) | > loader_time: 0.0046 (0.004661085910257577)  --> TIME: 2025-02-26 15:01:40 -- STEP: 32858/36938 -- GLOBAL_STEP: 1288750 | > loss_text_ce: 0.02662035822868347 (0.031173740686038464) | > loss_mel_ce: 1.4931035041809082 (nan) | > loss: 0.006030650809407234 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13682245611019805) | > loader_time: 0.0048 (0.004661056453946979)  --> TIME: 2025-02-26 15:01:49 -- STEP: 32908/36938 -- GLOBAL_STEP: 1288800 | > loss_text_ce: 0.036236293613910675 (0.031175207301990825) | > loss_mel_ce: 3.4074156284332275 (nan) | > loss: 0.013665285892784595 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13682408652764086) | > loader_time: 0.0047 (0.004661147477372614)  --> TIME: 2025-02-26 15:01:59 -- STEP: 32958/36938 -- GLOBAL_STEP: 1288850 | > loss_text_ce: 0.03164432570338249 (0.031174390438966255) | > loss_mel_ce: 1.7569221258163452 (nan) | > loss: 0.0070974864065647125 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.1368316727157966) | > loader_time: 0.0046 (0.004661153391355331)  --> TIME: 2025-02-26 15:02:08 -- STEP: 33008/36938 -- GLOBAL_STEP: 1288900 | > loss_text_ce: 0.031154872849583626 (0.03117482850692739) | > loss_mel_ce: 2.8929789066314697 (nan) | > loss: 0.011603706516325474 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.1368291034420545) | > loader_time: 0.0048 (0.004661158861261024)  --> TIME: 2025-02-26 15:02:17 -- STEP: 33058/36938 -- GLOBAL_STEP: 1288950 | > loss_text_ce: 0.029363498091697693 (0.031174005656099995) | > loss_mel_ce: 2.880706787109375 (nan) | > loss: 0.011547897942364216 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13682757910947088) | > loader_time: 0.0048 (0.004661209426490743)  --> TIME: 2025-02-26 15:02:26 -- STEP: 33108/36938 -- GLOBAL_STEP: 1289000 | > loss_text_ce: 0.013333619572222233 (0.03117191658404354) | > loss_mel_ce: 2.1838433742523193 (nan) | > loss: 0.008718956261873245 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13682047774484388) | > loader_time: 0.0045 (0.004661170197984943)  --> TIME: 2025-02-26 15:02:36 -- STEP: 33158/36938 -- GLOBAL_STEP: 1289050 | > loss_text_ce: 0.027698053047060966 (0.03117085815828029) | > loss_mel_ce: 2.5642716884613037 (nan) | > loss: 0.010285594500601292 (nan) | > current_lr: 5e-06 | > step_time: 0.1577 (0.1368233390348844) | > loader_time: 0.0046 (0.004661112817034515)  --> TIME: 2025-02-26 15:02:45 -- STEP: 33208/36938 -- GLOBAL_STEP: 1289100 | > loss_text_ce: 0.04655371233820915 (0.031168600380269466) | > loss_mel_ce: 3.0351500511169434 (nan) | > loss: 0.012228983454406261 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13683270244275622) | > loader_time: 0.0046 (0.0046610850594009535)  --> TIME: 2025-02-26 15:02:55 -- STEP: 33258/36938 -- GLOBAL_STEP: 1289150 | > loss_text_ce: 0.029632486402988434 (0.031166786178310268) | > loss_mel_ce: 2.620236873626709 (nan) | > loss: 0.010515355505049229 (nan) | > current_lr: 5e-06 | > step_time: 0.1866 (0.13683894217655948) | > loader_time: 0.0048 (0.00466105206600991)  --> TIME: 2025-02-26 15:03:04 -- STEP: 33308/36938 -- GLOBAL_STEP: 1289200 | > loss_text_ce: 0.03237925097346306 (0.031166602912102482) | > loss_mel_ce: 3.073948860168457 (nan) | > loss: 0.012326698750257492 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.1368283225921106) | > loader_time: 0.0045 (0.0046609425381536115)  --> TIME: 2025-02-26 15:03:13 -- STEP: 33358/36938 -- GLOBAL_STEP: 1289250 | > loss_text_ce: 0.03453097864985466 (0.031167509035113578) | > loss_mel_ce: 2.625328779220581 (nan) | > loss: 0.010554999113082886 (nan) | > current_lr: 5e-06 | > step_time: 0.1821 (0.13682673949333096) | > loader_time: 0.0048 (0.004660885692378899)  --> TIME: 2025-02-26 15:03:22 -- STEP: 33408/36938 -- GLOBAL_STEP: 1289300 | > loss_text_ce: 0.034032609313726425 (0.031167772400190242) | > loss_mel_ce: 2.8043787479400635 (nan) | > loss: 0.011263537220656872 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.1368205269220577) | > loader_time: 0.0046 (0.004661131514614539)  --> TIME: 2025-02-26 15:03:32 -- STEP: 33458/36938 -- GLOBAL_STEP: 1289350 | > loss_text_ce: 0.032933980226516724 (0.03116834981435913) | > loss_mel_ce: 2.073946237564087 (nan) | > loss: 0.008360636420547962 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13682423391005277) | > loader_time: 0.0045 (0.004661069447058286)  --> TIME: 2025-02-26 15:03:41 -- STEP: 33508/36938 -- GLOBAL_STEP: 1289400 | > loss_text_ce: 0.03010360337793827 (0.031168488690735664) | > loss_mel_ce: 2.276862144470215 (nan) | > loss: 0.009154627099633217 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13681900180352422) | > loader_time: 0.0047 (0.004660949596602178)  --> TIME: 2025-02-26 15:03:50 -- STEP: 33558/36938 -- GLOBAL_STEP: 1289450 | > loss_text_ce: 0.03678775206208229 (0.03116807452724884) | > loss_mel_ce: 3.1362459659576416 (nan) | > loss: 0.012591403909027576 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13681288094875546) | > loader_time: 0.0045 (0.004660851296526519)  --> TIME: 2025-02-26 15:03:59 -- STEP: 33608/36938 -- GLOBAL_STEP: 1289500 | > loss_text_ce: 0.026612451300024986 (0.031168236160773667) | > loss_mel_ce: 2.6581368446350098 (nan) | > loss: 0.010653767734766006 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13681312455531697) | > loader_time: 0.0047 (0.004660819441441864)  --> TIME: 2025-02-26 15:04:08 -- STEP: 33658/36938 -- GLOBAL_STEP: 1289550 | > loss_text_ce: 0.03200799226760864 (0.031170867268593527) | > loss_mel_ce: 2.9624812602996826 (nan) | > loss: 0.011882894672453403 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.13680663198220283) | > loader_time: 0.0047 (0.004660786575964735)  --> TIME: 2025-02-26 15:04:18 -- STEP: 33708/36938 -- GLOBAL_STEP: 1289600 | > loss_text_ce: 0.03533485159277916 (0.0311715910442185) | > loss_mel_ce: 2.5025360584259033 (nan) | > loss: 0.010070917196571827 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13681454942562268) | > loader_time: 0.0045 (0.004660854238313416)  --> TIME: 2025-02-26 15:04:27 -- STEP: 33758/36938 -- GLOBAL_STEP: 1289650 | > loss_text_ce: 0.03632187843322754 (0.03117053244166706) | > loss_mel_ce: 2.594308376312256 (nan) | > loss: 0.01043900940567255 (nan) | > current_lr: 5e-06 | > step_time: 0.1825 (0.13681551941735448) | > loader_time: 0.0048 (0.004660842514580464)  --> TIME: 2025-02-26 15:04:37 -- STEP: 33808/36938 -- GLOBAL_STEP: 1289700 | > loss_text_ce: 0.0353609137237072 (0.03117261789483492) | > loss_mel_ce: 3.0130615234375 (nan) | > loss: 0.01209691446274519 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13681488607203993) | > loader_time: 0.0046 (0.004660838371309324)  --> TIME: 2025-02-26 15:04:46 -- STEP: 33858/36938 -- GLOBAL_STEP: 1289750 | > loss_text_ce: 0.028625311329960823 (0.03117352110812963) | > loss_mel_ce: 2.843627691268921 (nan) | > loss: 0.011397830210626125 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.13680836276145047) | > loader_time: 0.0048 (0.004660907248835491)  --> TIME: 2025-02-26 15:04:55 -- STEP: 33908/36938 -- GLOBAL_STEP: 1289800 | > loss_text_ce: 0.031967852264642715 (0.031173296914968437) | > loss_mel_ce: 2.3863918781280518 (nan) | > loss: 0.009596666321158409 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13680754789164792) | > loader_time: 0.0046 (0.004660913934962994)  --> TIME: 2025-02-26 15:05:04 -- STEP: 33958/36938 -- GLOBAL_STEP: 1289850 | > loss_text_ce: 0.030695676803588867 (0.0311731602338919) | > loss_mel_ce: 2.6162397861480713 (nan) | > loss: 0.01050371304154396 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13680511695124467) | > loader_time: 0.0045 (0.004660906397950196)  --> TIME: 2025-02-26 15:05:14 -- STEP: 34008/36938 -- GLOBAL_STEP: 1289900 | > loss_text_ce: 0.03584054857492447 (0.031171337622838934) | > loss_mel_ce: 2.542663335800171 (nan) | > loss: 0.010232158936560154 (nan) | > current_lr: 5e-06 | > step_time: 0.1528 (0.1368035266144759) | > loader_time: 0.0045 (0.004660846555521815)  --> TIME: 2025-02-26 15:05:23 -- STEP: 34058/36938 -- GLOBAL_STEP: 1289950 | > loss_text_ce: 0.022550338879227638 (0.031170596222122662) | > loss_mel_ce: 1.7817305326461792 (nan) | > loss: 0.007159844972193241 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1368102898326946) | > loader_time: 0.0044 (0.0046607896329456215)  --> TIME: 2025-02-26 15:05:33 -- STEP: 34108/36938 -- GLOBAL_STEP: 1290000 | > loss_text_ce: 0.03094680793583393 (0.03116981717919527) | > loss_mel_ce: 2.4447526931762695 (nan) | > loss: 0.009824204258620739 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.1368131386558493) | > loader_time: 0.0045 (0.004660778543629512) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1290000.pth  --> TIME: 2025-02-26 15:05:48 -- STEP: 34158/36938 -- GLOBAL_STEP: 1290050 | > loss_text_ce: 0.025302402675151825 (0.031168128773555882) | > loss_mel_ce: 2.312934398651123 (nan) | > loss: 0.009278718382120132 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13681076793608152) | > loader_time: 0.0047 (0.004660810796904155)  --> TIME: 2025-02-26 15:05:57 -- STEP: 34208/36938 -- GLOBAL_STEP: 1290100 | > loss_text_ce: 0.036909621208906174 (0.03116612390250663) | > loss_mel_ce: 2.616593599319458 (nan) | > loss: 0.010529775172472 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13681027847294314) | > loader_time: 0.0046 (0.0046608134462969185)  --> TIME: 2025-02-26 15:06:06 -- STEP: 34258/36938 -- GLOBAL_STEP: 1290150 | > loss_text_ce: 0.030836664140224457 (0.03116791815255519) | > loss_mel_ce: 2.7402865886688232 (nan) | > loss: 0.010996521450579166 (nan) | > current_lr: 5e-06 | > step_time: 0.1662 (0.13680383867345897) | > loader_time: 0.0047 (0.004660794297758966)  --> TIME: 2025-02-26 15:06:15 -- STEP: 34308/36938 -- GLOBAL_STEP: 1290200 | > loss_text_ce: 0.041099462658166885 (0.03116913081309761) | > loss_mel_ce: 1.72397780418396 (nan) | > loss: 0.007004274986684322 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13679855786045933) | > loader_time: 0.0045 (0.004660750923977291)  --> TIME: 2025-02-26 15:06:24 -- STEP: 34358/36938 -- GLOBAL_STEP: 1290250 | > loss_text_ce: 0.033823881298303604 (0.031169534120644733) | > loss_mel_ce: 2.809447765350342 (nan) | > loss: 0.011282824911177158 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13678775208953597) | > loader_time: 0.0045 (0.004660624613673884)  --> TIME: 2025-02-26 15:06:33 -- STEP: 34408/36938 -- GLOBAL_STEP: 1290300 | > loss_text_ce: 0.02990339882671833 (0.031168031846256473) | > loss_mel_ce: 2.6176936626434326 (nan) | > loss: 0.010506338439881802 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.1367844944012997) | > loader_time: 0.0046 (0.004660540903703828)  --> TIME: 2025-02-26 15:06:43 -- STEP: 34458/36938 -- GLOBAL_STEP: 1290350 | > loss_text_ce: 0.03319023177027702 (0.031168554054612205) | > loss_mel_ce: 2.2226173877716064 (nan) | > loss: 0.00895161833614111 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13677988222439835) | > loader_time: 0.0045 (0.004660460577941559)  --> TIME: 2025-02-26 15:06:52 -- STEP: 34508/36938 -- GLOBAL_STEP: 1290400 | > loss_text_ce: 0.030499810352921486 (0.0311692243502016) | > loss_mel_ce: 2.688469409942627 (nan) | > loss: 0.010789560154080391 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13677810124730816) | > loader_time: 0.0046 (0.0046603860605822596)  --> TIME: 2025-02-26 15:07:01 -- STEP: 34558/36938 -- GLOBAL_STEP: 1290450 | > loss_text_ce: 0.028013957664370537 (0.031170301594640418) | > loss_mel_ce: 2.3178162574768066 (nan) | > loss: 0.009308850392699242 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13678101357972727) | > loader_time: 0.0048 (0.0046605109216383035)  --> TIME: 2025-02-26 15:07:11 -- STEP: 34608/36938 -- GLOBAL_STEP: 1290500 | > loss_text_ce: 0.030509481206536293 (0.031172338681221455) | > loss_mel_ce: 3.018996000289917 (nan) | > loss: 0.01210121251642704 (nan) | > current_lr: 5e-06 | > step_time: 0.1869 (0.13678356964473554) | > loader_time: 0.0049 (0.004660530473091374)  --> TIME: 2025-02-26 15:07:21 -- STEP: 34658/36938 -- GLOBAL_STEP: 1290550 | > loss_text_ce: 0.0321170836687088 (0.03117247754976113) | > loss_mel_ce: 4.32496976852417 (nan) | > loss: 0.0172900278121233 (nan) | > current_lr: 5e-06 | > step_time: 0.1875 (0.13679413381619818) | > loader_time: 0.0049 (0.004660650961355988)  --> TIME: 2025-02-26 15:07:30 -- STEP: 34708/36938 -- GLOBAL_STEP: 1290600 | > loss_text_ce: 0.029478594660758972 (0.031172762210294646) | > loss_mel_ce: 2.859318971633911 (nan) | > loss: 0.011463482864201069 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13678711554525214) | > loader_time: 0.0047 (0.004660704889590915)  --> TIME: 2025-02-26 15:07:39 -- STEP: 34758/36938 -- GLOBAL_STEP: 1290650 | > loss_text_ce: 0.043075092136859894 (0.03117271696364031) | > loss_mel_ce: 2.9744653701782227 (nan) | > loss: 0.011974367313086987 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.1367848645187275) | > loader_time: 0.0045 (0.004660674312788418)  --> TIME: 2025-02-26 15:07:48 -- STEP: 34808/36938 -- GLOBAL_STEP: 1290700 | > loss_text_ce: 0.04616284370422363 (0.031173985532530542) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13677892722471982) | > loader_time: 0.0046 (0.0046606874759102185)  --> TIME: 2025-02-26 15:07:57 -- STEP: 34858/36938 -- GLOBAL_STEP: 1290750 | > loss_text_ce: 0.04133641719818115 (0.031173618792076002) | > loss_mel_ce: 0.9231048822402954 (nan) | > loss: 0.003827148349955678 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13677718544438633) | > loader_time: 0.0045 (0.0046606867440192965)  --> TIME: 2025-02-26 15:08:07 -- STEP: 34908/36938 -- GLOBAL_STEP: 1290800 | > loss_text_ce: 0.02460574172437191 (0.031174757929856686) | > loss_mel_ce: 3.0880682468414307 (nan) | > loss: 0.012351881712675095 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13679232951277157) | > loader_time: 0.0047 (0.004660757707814574)  --> TIME: 2025-02-26 15:08:16 -- STEP: 34958/36938 -- GLOBAL_STEP: 1290850 | > loss_text_ce: 0.03145290166139603 (0.031173552810963345) | > loss_mel_ce: 2.4125711917877197 (nan) | > loss: 0.009698509238660336 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13678970999968446) | > loader_time: 0.0046 (0.004660764857133429)  --> TIME: 2025-02-26 15:08:25 -- STEP: 35008/36938 -- GLOBAL_STEP: 1290900 | > loss_text_ce: 0.035862356424331665 (0.03117363106156529) | > loss_mel_ce: 1.8783468008041382 (nan) | > loss: 0.007596068549901247 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13678402250029936) | > loader_time: 0.0048 (0.004660887292959108)  --> TIME: 2025-02-26 15:08:35 -- STEP: 35058/36938 -- GLOBAL_STEP: 1290950 | > loss_text_ce: 0.032442186027765274 (0.031174119900563724) | > loss_mel_ce: 2.921849250793457 (nan) | > loss: 0.011723378673195839 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.13679236303862988) | > loader_time: 0.005 (0.004660936986205146)  --> TIME: 2025-02-26 15:08:44 -- STEP: 35108/36938 -- GLOBAL_STEP: 1291000 | > loss_text_ce: 0.031101005151867867 (0.031174791242107723) | > loss_mel_ce: 1.9750348329544067 (nan) | > loss: 0.007960857823491096 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13679047092376048) | > loader_time: 0.0046 (0.004660979882723025)  --> TIME: 2025-02-26 15:08:54 -- STEP: 35158/36938 -- GLOBAL_STEP: 1291050 | > loss_text_ce: 0.03477831184864044 (0.03117641452441361) | > loss_mel_ce: 0.019775390625 (nan) | > loss: 0.0002164829638786614 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13678742055227308) | > loader_time: 0.0046 (0.004661147325503674)  --> TIME: 2025-02-26 15:09:03 -- STEP: 35208/36938 -- GLOBAL_STEP: 1291100 | > loss_text_ce: 0.03500848636031151 (0.031175104432253235) | > loss_mel_ce: 3.242549419403076 (nan) | > loss: 0.013006182387471199 (nan) | > current_lr: 5e-06 | > step_time: 0.1558 (0.13679080187827475) | > loader_time: 0.0047 (0.004661251810078592)  --> TIME: 2025-02-26 15:09:13 -- STEP: 35258/36938 -- GLOBAL_STEP: 1291150 | > loss_text_ce: 0.02610093727707863 (0.031174472121469954) | > loss_mel_ce: 2.8787317276000977 (nan) | > loss: 0.011527114547789097 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13679527658904736) | > loader_time: 0.0046 (0.004661324710013908)  --> TIME: 2025-02-26 15:09:22 -- STEP: 35308/36938 -- GLOBAL_STEP: 1291200 | > loss_text_ce: 0.03617975488305092 (0.031175224867559588) | > loss_mel_ce: 2.2779901027679443 (nan) | > loss: 0.009183214046061039 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.13680083691310815) | > loader_time: 0.0046 (0.004661301686267557)  --> TIME: 2025-02-26 15:09:32 -- STEP: 35358/36938 -- GLOBAL_STEP: 1291250 | > loss_text_ce: 0.035208892077207565 (0.03117420262038068) | > loss_mel_ce: 2.7534990310668945 (nan) | > loss: 0.011066301725804806 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13680558819206862) | > loader_time: 0.0045 (0.0046612726589478565)  --> TIME: 2025-02-26 15:09:41 -- STEP: 35408/36938 -- GLOBAL_STEP: 1291300 | > loss_text_ce: 0.04033001884818077 (0.03117590277590732) | > loss_mel_ce: 2.734647274017334 (nan) | > loss: 0.011011814698576927 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.13681465388691083) | > loader_time: 0.0048 (0.0046612885180932866)  --> TIME: 2025-02-26 15:09:51 -- STEP: 35458/36938 -- GLOBAL_STEP: 1291350 | > loss_text_ce: 0.027111787348985672 (0.031175680562215667) | > loss_mel_ce: 2.6188430786132812 (nan) | > loss: 0.010499821044504642 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.1368137846219508) | > loader_time: 0.0049 (0.00466132290412125)  --> TIME: 2025-02-26 15:10:00 -- STEP: 35508/36938 -- GLOBAL_STEP: 1291400 | > loss_text_ce: 0.03397824615240097 (0.031176129730701174) | > loss_mel_ce: 2.4969065189361572 (nan) | > loss: 0.010043193586170673 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13681665526540823) | > loader_time: 0.0046 (0.004661363075213776)  --> TIME: 2025-02-26 15:10:09 -- STEP: 35558/36938 -- GLOBAL_STEP: 1291450 | > loss_text_ce: 0.031998954713344574 (0.031176184688028092) | > loss_mel_ce: 0.0016017655143514276 (nan) | > loss: 0.00013333620154298842 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1368067852054381) | > loader_time: 0.0045 (0.004661351611639812)  --> TIME: 2025-02-26 15:10:18 -- STEP: 35608/36938 -- GLOBAL_STEP: 1291500 | > loss_text_ce: 0.0390506386756897 (0.031177733166158364) | > loss_mel_ce: 2.182244300842285 (nan) | > loss: 0.00881466269493103 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13680143218125107) | > loader_time: 0.0048 (0.0046613499425122015)  --> TIME: 2025-02-26 15:10:27 -- STEP: 35658/36938 -- GLOBAL_STEP: 1291550 | > loss_text_ce: 0.04101189598441124 (0.031177454981586205) | > loss_mel_ce: 2.990217924118042 (nan) | > loss: 0.012028690427541733 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13680390058908262) | > loader_time: 0.0046 (0.004661354676814196)  --> TIME: 2025-02-26 15:10:36 -- STEP: 35708/36938 -- GLOBAL_STEP: 1291600 | > loss_text_ce: 0.029885662719607353 (0.031178352053122686) | > loss_mel_ce: 1.7949090003967285 (nan) | > loss: 0.007241249084472656 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.1367955242344257) | > loader_time: 0.0046 (0.00466133477346777)  --> TIME: 2025-02-26 15:10:46 -- STEP: 35758/36938 -- GLOBAL_STEP: 1291650 | > loss_text_ce: 0.025584062561392784 (0.03117875463184809) | > loss_mel_ce: 2.73952317237854 (nan) | > loss: 0.010972647927701473 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13679329749821653) | > loader_time: 0.0045 (0.004661256517964671)  --> TIME: 2025-02-26 15:10:55 -- STEP: 35808/36938 -- GLOBAL_STEP: 1291700 | > loss_text_ce: 0.031084828078746796 (0.031177639728203962) | > loss_mel_ce: 2.337259531021118 (nan) | > loss: 0.009398192167282104 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.13679204893176108) | > loader_time: 0.0046 (0.004661187170018431)  --> TIME: 2025-02-26 15:11:04 -- STEP: 35858/36938 -- GLOBAL_STEP: 1291750 | > loss_text_ce: 0.04180604964494705 (0.031178458745621106) | > loss_mel_ce: 2.4757080078125 (nan) | > loss: 0.009990135207772255 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13678699107156544) | > loader_time: 0.0046 (0.004661115601893999)  --> TIME: 2025-02-26 15:11:14 -- STEP: 35908/36938 -- GLOBAL_STEP: 1291800 | > loss_text_ce: 0.03135489299893379 (0.031179904409109387) | > loss_mel_ce: 2.798091173171997 (nan) | > loss: 0.011227961629629135 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13678968017797394) | > loader_time: 0.0049 (0.004661123497894826)  --> TIME: 2025-02-26 15:11:23 -- STEP: 35958/36938 -- GLOBAL_STEP: 1291850 | > loss_text_ce: 0.027252333238720894 (0.031179309292750857) | > loss_mel_ce: 2.7368524074554443 (nan) | > loss: 0.01096867024898529 (nan) | > current_lr: 5e-06 | > step_time: 0.1844 (0.1367989026026199) | > loader_time: 0.0049 (0.0046611786472153275)  --> TIME: 2025-02-26 15:11:32 -- STEP: 36008/36938 -- GLOBAL_STEP: 1291900 | > loss_text_ce: 0.029963094741106033 (0.03118013253184505) | > loss_mel_ce: 0.05511474609375 (nan) | > loss: 0.00033761048689484596 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13679590095575334) | > loader_time: 0.0046 (0.004661120452819381)  --> TIME: 2025-02-26 15:11:41 -- STEP: 36058/36938 -- GLOBAL_STEP: 1291950 | > loss_text_ce: 0.0342840813100338 (0.031179343928375874) | > loss_mel_ce: 1.5092989206314087 (nan) | > loss: 0.006125329993665218 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13678656619376295) | > loader_time: 0.0053 (0.00466107377276533)  --> TIME: 2025-02-26 15:11:51 -- STEP: 36108/36938 -- GLOBAL_STEP: 1292000 | > loss_text_ce: 0.0338885635137558 (0.03117930367781926) | > loss_mel_ce: 1.5905499458312988 (nan) | > loss: 0.006446185056120157 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.136789883465364) | > loader_time: 0.0046 (0.004661063128720986)  --> TIME: 2025-02-26 15:12:00 -- STEP: 36158/36938 -- GLOBAL_STEP: 1292050 | > loss_text_ce: 0.02778438664972782 (0.031179920367722852) | > loss_mel_ce: 2.7438573837280273 (nan) | > loss: 0.010998578742146492 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13678951747808216) | > loader_time: 0.0045 (0.004661037889068368)  --> TIME: 2025-02-26 15:12:10 -- STEP: 36208/36938 -- GLOBAL_STEP: 1292100 | > loss_text_ce: 0.02663476951420307 (0.031179958341999818) | > loss_mel_ce: 2.7810750007629395 (nan) | > loss: 0.011141705326735973 (nan) | > current_lr: 5e-06 | > step_time: 0.1503 (0.13679331232897768) | > loader_time: 0.0047 (0.004661018072478887)  --> TIME: 2025-02-26 15:12:19 -- STEP: 36258/36938 -- GLOBAL_STEP: 1292150 | > loss_text_ce: 0.029939968138933182 (0.03118097191426624) | > loss_mel_ce: 2.7838618755340576 (nan) | > loss: 0.011165880598127842 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.1367887061064004) | > loader_time: 0.0045 (0.004660997587226361)  --> TIME: 2025-02-26 15:12:28 -- STEP: 36308/36938 -- GLOBAL_STEP: 1292200 | > loss_text_ce: 0.03804831579327583 (0.031180347542949387) | > loss_mel_ce: 3.0232372283935547 (nan) | > loss: 0.01214795932173729 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.1367865611872014) | > loader_time: 0.0046 (0.004660912871794144)  --> TIME: 2025-02-26 15:12:37 -- STEP: 36358/36938 -- GLOBAL_STEP: 1292250 | > loss_text_ce: 0.03348420187830925 (0.031178810682746732) | > loss_mel_ce: 3.389294147491455 (nan) | > loss: 0.01358245499432087 (nan) | > current_lr: 5e-06 | > step_time: 0.168 (0.1367882888659019) | > loader_time: 0.0049 (0.0046608565670604694)  --> TIME: 2025-02-26 15:12:47 -- STEP: 36408/36938 -- GLOBAL_STEP: 1292300 | > loss_text_ce: 0.03616584837436676 (0.031178609483953713) | > loss_mel_ce: 3.2023606300354004 (nan) | > loss: 0.012851296924054623 (nan) | > current_lr: 5e-06 | > step_time: 0.163 (0.136789008133544) | > loader_time: 0.0046 (0.0046607750872940945)  --> TIME: 2025-02-26 15:12:56 -- STEP: 36458/36938 -- GLOBAL_STEP: 1292350 | > loss_text_ce: 0.033707428723573685 (0.031178150078890448) | > loss_mel_ce: 2.414703369140625 (nan) | > loss: 0.009715916588902473 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13679007782455574) | > loader_time: 0.0047 (0.004660705458320693)  --> TIME: 2025-02-26 15:13:05 -- STEP: 36508/36938 -- GLOBAL_STEP: 1292400 | > loss_text_ce: 0.036611150950193405 (0.031176851410969767) | > loss_mel_ce: 3.048029661178589 (nan) | > loss: 0.012240638956427574 (nan) | > current_lr: 5e-06 | > step_time: 0.1869 (0.13678933171488258) | > loader_time: 0.0048 (0.004660656186514751)  --> TIME: 2025-02-26 15:13:15 -- STEP: 36558/36938 -- GLOBAL_STEP: 1292450 | > loss_text_ce: 0.030436545610427856 (0.03117602926547147) | > loss_mel_ce: 2.6758642196655273 (nan) | > loss: 0.010739289224147797 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13679242024530475) | > loader_time: 0.0045 (0.004660622329718147)  --> TIME: 2025-02-26 15:13:24 -- STEP: 36608/36938 -- GLOBAL_STEP: 1292500 | > loss_text_ce: 0.03271102532744408 (0.031174393249956495) | > loss_mel_ce: 2.802938222885132 (nan) | > loss: 0.011252577416598797 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.1367918179350925) | > loader_time: 0.0048 (0.004660588591457219)  --> TIME: 2025-02-26 15:13:34 -- STEP: 36658/36938 -- GLOBAL_STEP: 1292550 | > loss_text_ce: 0.021318022161722183 (0.03117523077521525) | > loss_mel_ce: 1.0347644090652466 (nan) | > loss: 0.004190803971141577 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.1367972324038757) | > loader_time: 0.0045 (0.004660556337057984)  --> TIME: 2025-02-26 15:13:43 -- STEP: 36708/36938 -- GLOBAL_STEP: 1292600 | > loss_text_ce: 0.03067900612950325 (0.03117558250521984) | > loss_mel_ce: 2.4173712730407715 (nan) | > loss: 0.009714486077427864 (nan) | > current_lr: 5e-06 | > step_time: 0.1905 (0.1368046097016561) | > loader_time: 0.0048 (0.004660542389010343)  --> TIME: 2025-02-26 15:13:53 -- STEP: 36758/36938 -- GLOBAL_STEP: 1292650 | > loss_text_ce: 0.03079339861869812 (0.0311751167385533) | > loss_mel_ce: 2.940873622894287 (nan) | > loss: 0.01179233007133007 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13680411512427912) | > loader_time: 0.0046 (0.004660528932940106)  --> TIME: 2025-02-26 15:14:02 -- STEP: 36808/36938 -- GLOBAL_STEP: 1292700 | > loss_text_ce: 0.019625991582870483 (0.031175380420408152) | > loss_mel_ce: 3.138831377029419 (nan) | > loss: 0.012533561326563358 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.13680437487691563) | > loader_time: 0.005 (0.004660510571203465)  --> TIME: 2025-02-26 15:14:12 -- STEP: 36858/36938 -- GLOBAL_STEP: 1292750 | > loss_text_ce: 0.022037804126739502 (0.031176443747463343) | > loss_mel_ce: 1.1688441038131714 (nan) | > loss: 0.004725722596049309 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1368131274298311) | > loader_time: 0.0045 (0.00466053524940525)  --> TIME: 2025-02-26 15:14:21 -- STEP: 36908/36938 -- GLOBAL_STEP: 1292800 | > loss_text_ce: 0.03459811210632324 (0.031177049505203263) | > loss_mel_ce: 2.717708110809326 (nan) | > loss: 0.010921850800514221 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.136817839198114) | > loader_time: 0.0046 (0.0046605349904836535)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.020589867440780805 (-0.0003943089211341025) | > avg_loss_text_ce: 0.03043369046126557 (-0.0001345819304927713) | > avg_loss_mel_ce: 2.4551523929775345 (-0.0074580010801268415) | > avg_loss: 2.485586086417191 (-0.0075925750897645194) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_1292830.pth  > EPOCH: 36/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-26 15:14:53)   --> TIME: 2025-02-26 15:14:57 -- STEP: 20/36938 -- GLOBAL_STEP: 1292850 | > loss_text_ce: 0.02536868490278721 (0.031065749097615482) | > loss_mel_ce: 2.736485004425049 (2.3611685872077937) | > loss: 0.010959737934172153 (0.009492993912135716) | > current_lr: 5e-06 | > step_time: 0.1849 (0.1403917670249939) | > loader_time: 0.005 (0.004873156547546387)  --> TIME: 2025-02-26 15:15:06 -- STEP: 70/36938 -- GLOBAL_STEP: 1292900 | > loss_text_ce: 0.030150165781378746 (0.030288664091910635) | > loss_mel_ce: 2.4928066730499268 (nan) | > loss: 0.010011734440922737 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13382124900817882) | > loader_time: 0.0048 (0.004747911861964635)  --> TIME: 2025-02-26 15:15:15 -- STEP: 120/36938 -- GLOBAL_STEP: 1292950 | > loss_text_ce: 0.030476311221718788 (0.030891806461537878) | > loss_mel_ce: 2.0875916481018066 (nan) | > loss: 0.008405032567679882 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13519172867139187) | > loader_time: 0.0046 (0.004724160830179851)  --> TIME: 2025-02-26 15:15:25 -- STEP: 170/36938 -- GLOBAL_STEP: 1293000 | > loss_text_ce: 0.0262263435870409 (0.030472728390904032) | > loss_mel_ce: 2.7601752281188965 (nan) | > loss: 0.01105714961886406 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1357612525715547) | > loader_time: 0.0047 (0.004702554029576921)  --> TIME: 2025-02-26 15:15:34 -- STEP: 220/36938 -- GLOBAL_STEP: 1293050 | > loss_text_ce: 0.028525756672024727 (0.030405305410650642) | > loss_mel_ce: 2.577484369277954 (nan) | > loss: 0.010341310873627663 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13585035800933834) | > loader_time: 0.0046 (0.0046819513494318205)  --> TIME: 2025-02-26 15:15:43 -- STEP: 270/36938 -- GLOBAL_STEP: 1293100 | > loss_text_ce: 0.024678289890289307 (0.03039251348624627) | > loss_mel_ce: 2.6220033168792725 (nan) | > loss: 0.010502705350518227 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13566438533641673) | > loader_time: 0.0045 (0.004676191012064618)  --> TIME: 2025-02-26 15:15:53 -- STEP: 320/36938 -- GLOBAL_STEP: 1293150 | > loss_text_ce: 0.042437177151441574 (0.03062738754088059) | > loss_mel_ce: 2.4121038913726807 (nan) | > loss: 0.00974024273455143 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.137200153619051) | > loader_time: 0.0046 (0.004671398550271989)  --> TIME: 2025-02-26 15:16:02 -- STEP: 370/36938 -- GLOBAL_STEP: 1293200 | > loss_text_ce: 0.03553866595029831 (0.030738087617666335) | > loss_mel_ce: 2.711712121963501 (nan) | > loss: 0.01090178918093443 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.137007534826124) | > loader_time: 0.0048 (0.004676976719418083)  --> TIME: 2025-02-26 15:16:12 -- STEP: 420/36938 -- GLOBAL_STEP: 1293250 | > loss_text_ce: 0.0329057052731514 (0.03083978530817798) | > loss_mel_ce: 3.0798532962799072 (nan) | > loss: 0.01235221978276968 (nan) | > current_lr: 5e-06 | > step_time: 0.1982 (0.13691968861080347) | > loader_time: 0.0049 (0.004666439692179356)  --> TIME: 2025-02-26 15:16:21 -- STEP: 470/36938 -- GLOBAL_STEP: 1293300 | > loss_text_ce: 0.028215596452355385 (0.030841136458230778) | > loss_mel_ce: 2.590785264968872 (nan) | > loss: 0.010392861440777779 (nan) | > current_lr: 5e-06 | > step_time: 0.1809 (0.1377868332761399) | > loader_time: 0.0049 (0.004663792062312994)  --> TIME: 2025-02-26 15:16:31 -- STEP: 520/36938 -- GLOBAL_STEP: 1293350 | > loss_text_ce: 0.036336805671453476 (0.030720300438742226) | > loss_mel_ce: 2.4151225090026855 (nan) | > loss: 0.009728014469146729 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13748015440427339) | > loader_time: 0.0045 (0.004663521509904124)  --> TIME: 2025-02-26 15:16:40 -- STEP: 570/36938 -- GLOBAL_STEP: 1293400 | > loss_text_ce: 0.02630946971476078 (0.030700489808396812) | > loss_mel_ce: 2.678652048110962 (nan) | > loss: 0.01073397509753704 (nan) | > current_lr: 5e-06 | > step_time: 0.1691 (0.13700394212153916) | > loader_time: 0.0047 (0.004658227636103043)  --> TIME: 2025-02-26 15:16:49 -- STEP: 620/36938 -- GLOBAL_STEP: 1293450 | > loss_text_ce: 0.02174440398812294 (0.03074164502472887) | > loss_mel_ce: 2.139225482940674 (nan) | > loss: 0.008575278334319592 (nan) | > current_lr: 5e-06 | > step_time: 0.1263 (0.13658653843787405) | > loader_time: 0.0047 (0.0046554880757485664)  --> TIME: 2025-02-26 15:16:58 -- STEP: 670/36938 -- GLOBAL_STEP: 1293500 | > loss_text_ce: 0.021914737299084663 (0.03071876091660181) | > loss_mel_ce: 1.952324628829956 (nan) | > loss: 0.007834283635020256 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13678512964675676) | > loader_time: 0.0045 (0.004654909603631318)  --> TIME: 2025-02-26 15:17:08 -- STEP: 720/36938 -- GLOBAL_STEP: 1293550 | > loss_text_ce: 0.036364324390888214 (0.030796211316353746) | > loss_mel_ce: 0.010223388671875 (nan) | > loss: 0.00018487188208382577 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13696586721473258) | > loader_time: 0.0045 (0.004657754633161757)  --> TIME: 2025-02-26 15:17:17 -- STEP: 770/36938 -- GLOBAL_STEP: 1293600 | > loss_text_ce: 0.02676885761320591 (0.030803220731194143) | > loss_mel_ce: 2.9422905445098877 (nan) | > loss: 0.011781983077526093 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.13666724415568557) | > loader_time: 0.0048 (0.004655667713710242)  --> TIME: 2025-02-26 15:17:26 -- STEP: 820/36938 -- GLOBAL_STEP: 1293650 | > loss_text_ce: 0.03518266603350639 (0.0308265352190086) | > loss_mel_ce: 3.35359787940979 (nan) | > loss: 0.013447542674839497 (nan) | > current_lr: 5e-06 | > step_time: 0.188 (0.13649173538859288) | > loader_time: 0.0048 (0.004655778698804906)  --> TIME: 2025-02-26 15:17:35 -- STEP: 870/36938 -- GLOBAL_STEP: 1293700 | > loss_text_ce: 0.030601076781749725 (0.030816274256199257) | > loss_mel_ce: 3.1196858882904053 (nan) | > loss: 0.01250113919377327 (nan) | > current_lr: 5e-06 | > step_time: 0.1889 (0.13651432689579043) | > loader_time: 0.0049 (0.00465457658658083)  --> TIME: 2025-02-26 15:17:45 -- STEP: 920/36938 -- GLOBAL_STEP: 1293750 | > loss_text_ce: 0.032602373510599136 (0.030800289334729315) | > loss_mel_ce: 2.7767577171325684 (nan) | > loss: 0.01114825438708067 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13666740915049658) | > loader_time: 0.0046 (0.004655363507892779)  --> TIME: 2025-02-26 15:17:54 -- STEP: 970/36938 -- GLOBAL_STEP: 1293800 | > loss_text_ce: 0.02847321145236492 (0.03083958196232921) | > loss_mel_ce: 3.0829241275787354 (nan) | > loss: 0.012346815317869186 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13669036073782995) | > loader_time: 0.0046 (0.00465612976821428)  --> TIME: 2025-02-26 15:18:04 -- STEP: 1020/36938 -- GLOBAL_STEP: 1293850 | > loss_text_ce: 0.02924187295138836 (0.03088010354332772) | > loss_mel_ce: 2.249065399169922 (nan) | > loss: 0.009040902368724346 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13682811821208266) | > loader_time: 0.0045 (0.004653982087677605)  --> TIME: 2025-02-26 15:18:13 -- STEP: 1070/36938 -- GLOBAL_STEP: 1293900 | > loss_text_ce: 0.02973615750670433 (0.030871653790959967) | > loss_mel_ce: 2.7445003986358643 (nan) | > loss: 0.01100887544453144 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13679747068993386) | > loader_time: 0.0048 (0.004651606417147917)  --> TIME: 2025-02-26 15:18:22 -- STEP: 1120/36938 -- GLOBAL_STEP: 1293950 | > loss_text_ce: 0.031228667125105858 (0.03086986495597687) | > loss_mel_ce: 3.180912733078003 (nan) | > loss: 0.012746593914926052 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13694791985409585) | > loader_time: 0.0046 (0.004649672976561959)  --> TIME: 2025-02-26 15:18:32 -- STEP: 1170/36938 -- GLOBAL_STEP: 1294000 | > loss_text_ce: 0.02424716390669346 (0.030838423745276837) | > loss_mel_ce: 2.8535449504852295 (nan) | > loss: 0.011419810354709625 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13691132068634015) | > loader_time: 0.0046 (0.004647799638601453)  --> TIME: 2025-02-26 15:18:41 -- STEP: 1220/36938 -- GLOBAL_STEP: 1294050 | > loss_text_ce: 0.03492022305727005 (0.030842473170887983) | > loss_mel_ce: 3.118655204772949 (nan) | > loss: 0.012514188885688782 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.13705197689963158) | > loader_time: 0.0046 (0.004648487489731589)  --> TIME: 2025-02-26 15:18:51 -- STEP: 1270/36938 -- GLOBAL_STEP: 1294100 | > loss_text_ce: 0.03720039874315262 (0.030870992548793084) | > loss_mel_ce: 0.0732828751206398 (nan) | > loss: 0.00043842571903951466 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1372070389469777) | > loader_time: 0.0046 (0.004648429202282526)  --> TIME: 2025-02-26 15:19:00 -- STEP: 1320/36938 -- GLOBAL_STEP: 1294150 | > loss_text_ce: 0.032933808863162994 (0.030850818371541345) | > loss_mel_ce: 2.4255757331848145 (nan) | > loss: 0.009755990467965603 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13707043149254516) | > loader_time: 0.0046 (0.004647910233699917)  --> TIME: 2025-02-26 15:19:09 -- STEP: 1370/36938 -- GLOBAL_STEP: 1294200 | > loss_text_ce: 0.03525590896606445 (0.030909529504413804) | > loss_mel_ce: 2.3912742137908936 (nan) | > loss: 0.009629088453948498 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13678659835871107) | > loader_time: 0.0046 (0.0046479475759241716)  --> TIME: 2025-02-26 15:19:18 -- STEP: 1420/36938 -- GLOBAL_STEP: 1294250 | > loss_text_ce: 0.0379992350935936 (0.03090975861799654) | > loss_mel_ce: 2.8717732429504395 (nan) | > loss: 0.011546716094017029 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13678487002010065) | > loader_time: 0.0046 (0.004653531564793122)  --> TIME: 2025-02-26 15:19:28 -- STEP: 1470/36938 -- GLOBAL_STEP: 1294300 | > loss_text_ce: 0.03659215196967125 (0.030911125770459572) | > loss_mel_ce: 2.2980527877807617 (nan) | > loss: 0.009264464490115643 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.13683031805518522) | > loader_time: 0.0047 (0.004652746031884438)  --> TIME: 2025-02-26 15:19:37 -- STEP: 1520/36938 -- GLOBAL_STEP: 1294350 | > loss_text_ce: 0.04260881990194321 (0.030912084669781556) | > loss_mel_ce: 3.376316785812378 (nan) | > loss: 0.013567165471613407 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1370240250700397) | > loader_time: 0.0054 (0.004660201386401533)  --> TIME: 2025-02-26 15:19:47 -- STEP: 1570/36938 -- GLOBAL_STEP: 1294400 | > loss_text_ce: 0.030498037114739418 (0.030915105044485847) | > loss_mel_ce: 1.1138237714767456 (nan) | > loss: 0.0045409598387777805 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.137067652052375) | > loader_time: 0.0046 (0.004664807714474435)  --> TIME: 2025-02-26 15:19:56 -- STEP: 1620/36938 -- GLOBAL_STEP: 1294450 | > loss_text_ce: 0.03892660140991211 (0.030950674779690157) | > loss_mel_ce: 3.007554769515991 (nan) | > loss: 0.01208921242505312 (nan) | > current_lr: 5e-06 | > step_time: 0.1721 (0.13721718773429759) | > loader_time: 0.0056 (0.004666480458812954)  --> TIME: 2025-02-26 15:20:06 -- STEP: 1670/36938 -- GLOBAL_STEP: 1294500 | > loss_text_ce: 0.031001461669802666 (0.03094863284345783) | > loss_mel_ce: 2.86838698387146 (nan) | > loss: 0.011505510658025742 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.13716813761317076) | > loader_time: 0.0047 (0.004667785638820631)  --> TIME: 2025-02-26 15:20:15 -- STEP: 1720/36938 -- GLOBAL_STEP: 1294550 | > loss_text_ce: 0.028773993253707886 (0.0309540796494328) | > loss_mel_ce: 2.996122360229492 (nan) | > loss: 0.01200355775654316 (nan) | > current_lr: 5e-06 | > step_time: 0.1894 (0.1370973458123761) | > loader_time: 0.0049 (0.004668518831563556)  --> TIME: 2025-02-26 15:20:24 -- STEP: 1770/36938 -- GLOBAL_STEP: 1294600 | > loss_text_ce: 0.03357210382819176 (0.03097558838478421) | > loss_mel_ce: 2.9313766956329346 (nan) | > loss: 0.011765670962631702 (nan) | > current_lr: 5e-06 | > step_time: 0.1896 (0.137241181411312) | > loader_time: 0.0048 (0.0046687563933895185)  --> TIME: 2025-02-26 15:20:34 -- STEP: 1820/36938 -- GLOBAL_STEP: 1294650 | > loss_text_ce: 0.03713798522949219 (0.030978621187834787) | > loss_mel_ce: 2.4234700202941895 (nan) | > loss: 0.009764318354427814 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13714397071482054) | > loader_time: 0.0047 (0.004672482249500992)  --> TIME: 2025-02-26 15:20:43 -- STEP: 1870/36938 -- GLOBAL_STEP: 1294700 | > loss_text_ce: 0.03237808868288994 (0.030983068567406843) | > loss_mel_ce: 2.5149142742156982 (nan) | > loss: 0.01010830421000719 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13715018703338303) | > loader_time: 0.0045 (0.004672339765783304)  --> TIME: 2025-02-26 15:20:52 -- STEP: 1920/36938 -- GLOBAL_STEP: 1294750 | > loss_text_ce: 0.034205593168735504 (0.031002712974441238) | > loss_mel_ce: 2.1578481197357178 (nan) | > loss: 0.008698626421391964 (nan) | > current_lr: 5e-06 | > step_time: 0.1975 (0.13712104273339118) | > loader_time: 0.0049 (0.004674442609151209)  --> TIME: 2025-02-26 15:21:01 -- STEP: 1970/36938 -- GLOBAL_STEP: 1294800 | > loss_text_ce: 0.03110966831445694 (0.031054473942533363) | > loss_mel_ce: 2.77699613571167 (nan) | > loss: 0.011143277399241924 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.1370713957675218) | > loader_time: 0.005 (0.0046760680106690735)  --> TIME: 2025-02-26 15:21:11 -- STEP: 2020/36938 -- GLOBAL_STEP: 1294850 | > loss_text_ce: 0.04413343966007233 (0.031042384666086425) | > loss_mel_ce: 2.8728342056274414 (nan) | > loss: 0.01157526858150959 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.1371042104050665) | > loader_time: 0.0048 (0.004675787392229141)  --> TIME: 2025-02-26 15:21:20 -- STEP: 2070/36938 -- GLOBAL_STEP: 1294900 | > loss_text_ce: 0.036172039806842804 (0.03104217792028822) | > loss_mel_ce: 3.2400553226470947 (nan) | > loss: 0.013000903651118279 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.1371001699696417) | > loader_time: 0.0047 (0.004676988504935006)  --> TIME: 2025-02-26 15:21:29 -- STEP: 2120/36938 -- GLOBAL_STEP: 1294950 | > loss_text_ce: 0.03462949022650719 (0.031074276533118398) | > loss_mel_ce: 2.8494131565093994 (nan) | > loss: 0.01144461426883936 (nan) | > current_lr: 5e-06 | > step_time: 0.1777 (0.13703434849685092) | > loader_time: 0.0047 (0.004675729882042368)  --> TIME: 2025-02-26 15:21:39 -- STEP: 2170/36938 -- GLOBAL_STEP: 1295000 | > loss_text_ce: 0.02456226386129856 (0.031051271270600058) | > loss_mel_ce: 3.229921340942383 (nan) | > loss: 0.012914618477225304 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13694975376129154) | > loader_time: 0.0045 (0.004676590097664693)  --> TIME: 2025-02-26 15:21:48 -- STEP: 2220/36938 -- GLOBAL_STEP: 1295050 | > loss_text_ce: 0.03273865208029747 (0.031035798639562483) | > loss_mel_ce: 2.975184202194214 (nan) | > loss: 0.011936202645301819 (nan) | > current_lr: 5e-06 | > step_time: 0.1811 (0.13694611304515122) | > loader_time: 0.0058 (0.004678496584161986)  --> TIME: 2025-02-26 15:21:57 -- STEP: 2270/36938 -- GLOBAL_STEP: 1295100 | > loss_text_ce: 0.034485526382923126 (0.03103909141338935) | > loss_mel_ce: 2.905073642730713 (nan) | > loss: 0.01166491862386465 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13675979916744835) | > loader_time: 0.0046 (0.004677861797652057)  --> TIME: 2025-02-26 15:22:06 -- STEP: 2320/36938 -- GLOBAL_STEP: 1295150 | > loss_text_ce: 0.03320436552166939 (0.031044154553593876) | > loss_mel_ce: 2.8830149173736572 (nan) | > loss: 0.011572299525141716 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1366054678785391) | > loader_time: 0.0045 (0.004678368054587266)  --> TIME: 2025-02-26 15:22:15 -- STEP: 2370/36938 -- GLOBAL_STEP: 1295200 | > loss_text_ce: 0.03307264670729637 (0.031050811665140503) | > loss_mel_ce: 2.9030067920684814 (nan) | > loss: 0.011651109904050827 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13661808585315827) | > loader_time: 0.0048 (0.004677374755279929)  --> TIME: 2025-02-26 15:22:25 -- STEP: 2420/36938 -- GLOBAL_STEP: 1295250 | > loss_text_ce: 0.03391481935977936 (0.03102408059457055) | > loss_mel_ce: 2.9982004165649414 (nan) | > loss: 0.012032204307615757 (nan) | > current_lr: 5e-06 | > step_time: 0.1664 (0.1366620525840887) | > loader_time: 0.0056 (0.004677677253061094)  --> TIME: 2025-02-26 15:22:34 -- STEP: 2470/36938 -- GLOBAL_STEP: 1295300 | > loss_text_ce: 0.028326796367764473 (0.03100538722084843) | > loss_mel_ce: 2.818852424621582 (nan) | > loss: 0.011298330500721931 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.1366350692293424) | > loader_time: 0.0046 (0.004680262789552514)  --> TIME: 2025-02-26 15:22:43 -- STEP: 2520/36938 -- GLOBAL_STEP: 1295350 | > loss_text_ce: 0.04021270573139191 (0.03102213922400205) | > loss_mel_ce: 1.5227779150009155 (nan) | > loss: 0.006202344316989183 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.1365814763402185) | > loader_time: 0.0044 (0.004679595004944575)  --> TIME: 2025-02-26 15:22:52 -- STEP: 2570/36938 -- GLOBAL_STEP: 1295400 | > loss_text_ce: 0.04361659288406372 (0.03106590569425186) | > loss_mel_ce: 1.1547070741653442 (nan) | > loss: 0.004755252972245216 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13657224400961931) | > loader_time: 0.0045 (0.004678577578949092)  --> TIME: 2025-02-26 15:23:01 -- STEP: 2620/36938 -- GLOBAL_STEP: 1295450 | > loss_text_ce: 0.030490415170788765 (0.031062677858809704) | > loss_mel_ce: 0.51324462890625 (nan) | > loss: 0.0021576788276433945 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13656441391879393) | > loader_time: 0.0046 (0.0046804629209387365)  --> TIME: 2025-02-26 15:23:11 -- STEP: 2670/36938 -- GLOBAL_STEP: 1295500 | > loss_text_ce: 0.026166867464780807 (0.03106253904810224) | > loss_mel_ce: 1.2614659070968628 (nan) | > loss: 0.005109654273837805 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13651470271835642) | > loader_time: 0.0045 (0.00467980283029963)  --> TIME: 2025-02-26 15:23:20 -- STEP: 2720/36938 -- GLOBAL_STEP: 1295550 | > loss_text_ce: 0.030322594568133354 (0.031042775142636588) | > loss_mel_ce: 0.7678551077842712 (nan) | > loss: 0.0031673721969127655 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.1364276277668338) | > loader_time: 0.0047 (0.004679600368527803)  --> TIME: 2025-02-26 15:23:29 -- STEP: 2770/36938 -- GLOBAL_STEP: 1295600 | > loss_text_ce: 0.0309447068721056 (0.031050708787078676) | > loss_mel_ce: 3.4964652061462402 (nan) | > loss: 0.013997660018503666 (nan) | > current_lr: 5e-06 | > step_time: 0.1786 (0.13650364548721056) | > loader_time: 0.0048 (0.0046826089762608495)  --> TIME: 2025-02-26 15:23:39 -- STEP: 2820/36938 -- GLOBAL_STEP: 1295650 | > loss_text_ce: 0.033530592918395996 (0.031047157483201818) | > loss_mel_ce: 2.7410004138946533 (nan) | > loss: 0.011010044254362583 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13655280855530558) | > loader_time: 0.0046 (0.004682952715150004)  --> TIME: 2025-02-26 15:23:48 -- STEP: 2870/36938 -- GLOBAL_STEP: 1295700 | > loss_text_ce: 0.036815568804740906 (0.031052513365091867) | > loss_mel_ce: 2.7180137634277344 (nan) | > loss: 0.010931863449513912 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13663892862273444) | > loader_time: 0.0047 (0.004684190633820323)  --> TIME: 2025-02-26 15:23:57 -- STEP: 2920/36938 -- GLOBAL_STEP: 1295750 | > loss_text_ce: 0.02934011071920395 (0.031049613564989646) | > loss_mel_ce: 2.5609498023986816 (nan) | > loss: 0.010278928093612194 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13655612493214558) | > loader_time: 0.0045 (0.004684195534823692)  --> TIME: 2025-02-26 15:24:07 -- STEP: 2970/36938 -- GLOBAL_STEP: 1295800 | > loss_text_ce: 0.02525453083217144 (0.031044217866064622) | > loss_mel_ce: 2.714611768722534 (nan) | > loss: 0.010872486047446728 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13658130449879477) | > loader_time: 0.0065 (0.004684669160682345)  --> TIME: 2025-02-26 15:24:16 -- STEP: 3020/36938 -- GLOBAL_STEP: 1295850 | > loss_text_ce: 0.020715203136205673 (0.031052549197246006) | > loss_mel_ce: 0.14732517302036285 (nan) | > loss: 0.0006668269634246826 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.136626519509499) | > loader_time: 0.0045 (0.0046848635010371936)  --> TIME: 2025-02-26 15:24:25 -- STEP: 3070/36938 -- GLOBAL_STEP: 1295900 | > loss_text_ce: 0.03152380511164665 (0.031062645104437677) | > loss_mel_ce: 2.7299282550811768 (nan) | > loss: 0.010958143509924412 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13660115015234967) | > loader_time: 0.0046 (0.004685503885101417)  --> TIME: 2025-02-26 15:24:35 -- STEP: 3120/36938 -- GLOBAL_STEP: 1295950 | > loss_text_ce: 0.023758133873343468 (0.031056823645037812) | > loss_mel_ce: 2.0253207683563232 (nan) | > loss: 0.008131266571581364 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1365563026605511) | > loader_time: 0.0045 (0.004684885877829339)  --> TIME: 2025-02-26 15:24:44 -- STEP: 3170/36938 -- GLOBAL_STEP: 1296000 | > loss_text_ce: 0.028114141896367073 (0.03104284560155088) | > loss_mel_ce: 2.057210922241211 (nan) | > loss: 0.008275100030004978 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13651528508881317) | > loader_time: 0.0046 (0.004686774515579181)  --> TIME: 2025-02-26 15:24:53 -- STEP: 3220/36938 -- GLOBAL_STEP: 1296050 | > loss_text_ce: 0.039194948971271515 (0.031051626887392277) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13654478159010075) | > loader_time: 0.0045 (0.0046873234073567865)  --> TIME: 2025-02-26 15:25:03 -- STEP: 3270/36938 -- GLOBAL_STEP: 1296100 | > loss_text_ce: 0.03610553964972496 (0.031026838688515926) | > loss_mel_ce: 2.7945973873138428 (nan) | > loss: 0.011232948862016201 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13662993630867157) | > loader_time: 0.0046 (0.004687131435499284)  --> TIME: 2025-02-26 15:25:12 -- STEP: 3320/36938 -- GLOBAL_STEP: 1296150 | > loss_text_ce: 0.04083694517612457 (0.03103664153401392) | > loss_mel_ce: 2.6346824169158936 (nan) | > loss: 0.010617141611874104 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13661998144115345) | > loader_time: 0.0047 (0.0046864604375448675)  --> TIME: 2025-02-26 15:25:21 -- STEP: 3370/36938 -- GLOBAL_STEP: 1296200 | > loss_text_ce: 0.03621635586023331 (0.031033077994665512) | > loss_mel_ce: 2.520286798477173 (nan) | > loss: 0.010144853964447975 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13658620681536332) | > loader_time: 0.0045 (0.004685301879746989)  --> TIME: 2025-02-26 15:25:31 -- STEP: 3420/36938 -- GLOBAL_STEP: 1296250 | > loss_text_ce: 0.026267651468515396 (0.031030415914059556) | > loss_mel_ce: 2.005283832550049 (nan) | > loss: 0.00806171353906393 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13661350869295907) | > loader_time: 0.0044 (0.004683982489401849)  --> TIME: 2025-02-26 15:25:40 -- STEP: 3470/36938 -- GLOBAL_STEP: 1296300 | > loss_text_ce: 0.033414795994758606 (0.031035093030750065) | > loss_mel_ce: 0.7411770224571228 (nan) | > loss: 0.0030737770721316338 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13664024563275445) | > loader_time: 0.0045 (0.004683105677623928)  --> TIME: 2025-02-26 15:25:49 -- STEP: 3520/36938 -- GLOBAL_STEP: 1296350 | > loss_text_ce: 0.025678839534521103 (0.03103242186691866) | > loss_mel_ce: 2.51981258392334 (nan) | > loss: 0.010101157240569592 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13668373619968255) | > loader_time: 0.0045 (0.004683545096354065)  --> TIME: 2025-02-26 15:25:59 -- STEP: 3570/36938 -- GLOBAL_STEP: 1296400 | > loss_text_ce: 0.03417297452688217 (0.031039385370263198) | > loss_mel_ce: 2.689317464828491 (nan) | > loss: 0.010807502083480358 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.13667957175011736) | > loader_time: 0.0049 (0.004682864328058508)  --> TIME: 2025-02-26 15:26:08 -- STEP: 3620/36938 -- GLOBAL_STEP: 1296450 | > loss_text_ce: 0.03443511947989464 (0.031019963909877022) | > loss_mel_ce: 2.5037074089050293 (nan) | > loss: 0.010071994736790657 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13665219584881277) | > loader_time: 0.0045 (0.00468271397753975)  --> TIME: 2025-02-26 15:26:17 -- STEP: 3670/36938 -- GLOBAL_STEP: 1296500 | > loss_text_ce: 0.02353499084711075 (0.031013508044355816) | > loss_mel_ce: 1.9748739004135132 (nan) | > loss: 0.007930194959044456 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.1367070766495751) | > loader_time: 0.0046 (0.004682751767317034)  --> TIME: 2025-02-26 15:26:27 -- STEP: 3720/36938 -- GLOBAL_STEP: 1296550 | > loss_text_ce: 0.02344970405101776 (0.03101325255683473) | > loss_mel_ce: 2.602350950241089 (nan) | > loss: 0.010419844649732113 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13678061616036233) | > loader_time: 0.0045 (0.004682979212012356)  --> TIME: 2025-02-26 15:26:37 -- STEP: 3770/36938 -- GLOBAL_STEP: 1296600 | > loss_text_ce: 0.033860500901937485 (0.031008431688198123) | > loss_mel_ce: 3.0676932334899902 (nan) | > loss: 0.012307753786444664 (nan) | > current_lr: 5e-06 | > step_time: 0.1582 (0.136788864793449) | > loader_time: 0.005 (0.004684449443766559)  --> TIME: 2025-02-26 15:26:46 -- STEP: 3820/36938 -- GLOBAL_STEP: 1296650 | > loss_text_ce: 0.0315060131251812 (0.031011695838752764) | > loss_mel_ce: 2.9527156352996826 (nan) | > loss: 0.011842150241136551 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.13690394075753215) | > loader_time: 0.0053 (0.0046850228184805)  --> TIME: 2025-02-26 15:26:56 -- STEP: 3870/36938 -- GLOBAL_STEP: 1296700 | > loss_text_ce: 0.025448758155107498 (0.031006722502381533) | > loss_mel_ce: 3.1208884716033936 (nan) | > loss: 0.012485465966165066 (nan) | > current_lr: 5e-06 | > step_time: 0.1812 (0.13693487927587186) | > loader_time: 0.0047 (0.004685380662134465)  --> TIME: 2025-02-26 15:27:05 -- STEP: 3920/36938 -- GLOBAL_STEP: 1296750 | > loss_text_ce: 0.03615075722336769 (0.03101001315517351) | > loss_mel_ce: 2.9423956871032715 (nan) | > loss: 0.0118196289986372 (nan) | > current_lr: 5e-06 | > step_time: 0.1867 (0.13691513781644898) | > loader_time: 0.0049 (0.004685663264624939)  --> TIME: 2025-02-26 15:27:14 -- STEP: 3970/36938 -- GLOBAL_STEP: 1296800 | > loss_text_ce: 0.017431488260626793 (0.0310063588259371) | > loss_mel_ce: 2.3174171447753906 (nan) | > loss: 0.009265272878110409 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13687123465598117) | > loader_time: 0.0045 (0.004686239864724121)  --> TIME: 2025-02-26 15:27:24 -- STEP: 4020/36938 -- GLOBAL_STEP: 1296850 | > loss_text_ce: 0.021325718611478806 (0.03100683928828742) | > loss_mel_ce: 2.34010910987854 (nan) | > loss: 0.009370774030685425 (nan) | > current_lr: 5e-06 | > step_time: 0.1453 (0.13685921691543465) | > loader_time: 0.0047 (0.004687125113473022)  --> TIME: 2025-02-26 15:27:33 -- STEP: 4070/36938 -- GLOBAL_STEP: 1296900 | > loss_text_ce: 0.03198739513754845 (0.03100891902251055) | > loss_mel_ce: 2.3699753284454346 (nan) | > loss: 0.009531598538160324 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13685373855750194) | > loader_time: 0.0046 (0.004688425438995748)  --> TIME: 2025-02-26 15:27:43 -- STEP: 4120/36938 -- GLOBAL_STEP: 1296950 | > loss_text_ce: 0.03277067467570305 (0.031018371278989734) | > loss_mel_ce: 3.0595147609710693 (nan) | > loss: 0.012270974926650524 (nan) | > current_lr: 5e-06 | > step_time: 0.1898 (0.1369266858378664) | > loader_time: 0.0047 (0.004688184353911773)  --> TIME: 2025-02-26 15:27:52 -- STEP: 4170/36938 -- GLOBAL_STEP: 1297000 | > loss_text_ce: 0.034572113305330276 (0.031013273619052817) | > loss_mel_ce: 2.424269199371338 (nan) | > loss: 0.00975730735808611 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13698583518286728) | > loader_time: 0.0045 (0.004687396685282401)  --> TIME: 2025-02-26 15:28:02 -- STEP: 4220/36938 -- GLOBAL_STEP: 1297050 | > loss_text_ce: 0.029849804937839508 (0.031019539733624785) | > loss_mel_ce: 2.692026376724243 (nan) | > loss: 0.010801096446812153 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13700935846256457) | > loader_time: 0.0048 (0.004686564630806741)  --> TIME: 2025-02-26 15:28:11 -- STEP: 4270/36938 -- GLOBAL_STEP: 1297100 | > loss_text_ce: 0.030263034626841545 (0.031016883347183465) | > loss_mel_ce: 1.4929152727127075 (nan) | > loss: 0.006044358946382999 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13704115071397233) | > loader_time: 0.0046 (0.004687983928296283)  --> TIME: 2025-02-26 15:28:21 -- STEP: 4320/36938 -- GLOBAL_STEP: 1297150 | > loss_text_ce: 0.032858241349458694 (0.031024586255619026) | > loss_mel_ce: 2.6300246715545654 (nan) | > loss: 0.010566995479166508 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.1370926447488647) | > loader_time: 0.0045 (0.004688568192499668)  --> TIME: 2025-02-26 15:28:30 -- STEP: 4370/36938 -- GLOBAL_STEP: 1297200 | > loss_text_ce: 0.037350255995988846 (0.031034381557000244) | > loss_mel_ce: 2.8266544342041016 (nan) | > loss: 0.01136509794741869 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13709633475855817) | > loader_time: 0.0048 (0.004687937064356347)  --> TIME: 2025-02-26 15:28:40 -- STEP: 4420/36938 -- GLOBAL_STEP: 1297250 | > loss_text_ce: 0.030286496505141258 (0.03103798834282517) | > loss_mel_ce: 2.497830390930176 (nan) | > loss: 0.010032210499048233 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13713467967995732) | > loader_time: 0.0045 (0.004688177119552831)  --> TIME: 2025-02-26 15:28:49 -- STEP: 4470/36938 -- GLOBAL_STEP: 1297300 | > loss_text_ce: 0.03965315595269203 (0.03102900930628247) | > loss_mel_ce: 2.793668270111084 (nan) | > loss: 0.011243339627981186 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.13717759935647886) | > loader_time: 0.0049 (0.004688491170571699)  --> TIME: 2025-02-26 15:28:59 -- STEP: 4520/36938 -- GLOBAL_STEP: 1297350 | > loss_text_ce: 0.03479565307497978 (0.031031073506435265) | > loss_mel_ce: 3.1515355110168457 (nan) | > loss: 0.01264417264610529 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13717922373155608) | > loader_time: 0.0044 (0.004688706651198135)  --> TIME: 2025-02-26 15:29:08 -- STEP: 4570/36938 -- GLOBAL_STEP: 1297400 | > loss_text_ce: 0.0357922688126564 (0.031030818794933464) | > loss_mel_ce: 2.829777240753174 (nan) | > loss: 0.0113713089376688 (nan) | > current_lr: 5e-06 | > step_time: 0.1798 (0.13718924788468811) | > loader_time: 0.0049 (0.004688243636417188)  --> TIME: 2025-02-26 15:29:17 -- STEP: 4620/36938 -- GLOBAL_STEP: 1297450 | > loss_text_ce: 0.02046293392777443 (0.031027717304565175) | > loss_mel_ce: 1.7767120599746704 (nan) | > loss: 0.007131647318601608 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.1371634668602059) | > loader_time: 0.0046 (0.004687778155008961)  --> TIME: 2025-02-26 15:29:27 -- STEP: 4670/36938 -- GLOBAL_STEP: 1297500 | > loss_text_ce: 0.026558391749858856 (0.03103041820670827) | > loss_mel_ce: 2.568542718887329 (nan) | > loss: 0.010298021137714386 (nan) | > current_lr: 5e-06 | > step_time: 0.189 (0.13723664651335724) | > loader_time: 0.0052 (0.004689025419427198)  --> TIME: 2025-02-26 15:29:36 -- STEP: 4720/36938 -- GLOBAL_STEP: 1297550 | > loss_text_ce: 0.03080974705517292 (0.031032688235587802) | > loss_mel_ce: 2.3297839164733887 (nan) | > loss: 0.009367435239255428 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13722877921694446) | > loader_time: 0.0046 (0.0046883821487426845)  --> TIME: 2025-02-26 15:29:45 -- STEP: 4770/36938 -- GLOBAL_STEP: 1297600 | > loss_text_ce: 0.02411431260406971 (0.031027077186649637) | > loss_mel_ce: 1.7915111780166626 (nan) | > loss: 0.007204863708466291 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.137177848965867) | > loader_time: 0.0046 (0.004687663094302652)  --> TIME: 2025-02-26 15:29:55 -- STEP: 4820/36938 -- GLOBAL_STEP: 1297650 | > loss_text_ce: 0.028182467445731163 (0.03104146723364986) | > loss_mel_ce: 2.3540282249450684 (nan) | > loss: 0.00945321749895811 (nan) | > current_lr: 5e-06 | > step_time: 0.1752 (0.13719543228505574) | > loader_time: 0.0048 (0.004687527749548325)  --> TIME: 2025-02-26 15:30:04 -- STEP: 4870/36938 -- GLOBAL_STEP: 1297700 | > loss_text_ce: 0.039457883685827255 (0.03103980747332655) | > loss_mel_ce: 2.6875627040863037 (nan) | > loss: 0.01082151010632515 (nan) | > current_lr: 5e-06 | > step_time: 0.1719 (0.13723121433532234) | > loader_time: 0.0048 (0.004688144072859696)  --> TIME: 2025-02-26 15:30:14 -- STEP: 4920/36938 -- GLOBAL_STEP: 1297750 | > loss_text_ce: 0.036585550755262375 (0.0310306090942955) | > loss_mel_ce: 2.8082315921783447 (nan) | > loss: 0.011288957670331001 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13721100101626046) | > loader_time: 0.0046 (0.004687787943739245)  --> TIME: 2025-02-26 15:30:23 -- STEP: 4970/36938 -- GLOBAL_STEP: 1297800 | > loss_text_ce: 0.0325518436729908 (0.031034160551373327) | > loss_mel_ce: 2.930553436279297 (nan) | > loss: 0.011758354492485523 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13718449076416536) | > loader_time: 0.0046 (0.004687254337717601)  --> TIME: 2025-02-26 15:30:32 -- STEP: 5020/36938 -- GLOBAL_STEP: 1297850 | > loss_text_ce: 0.022547828033566475 (0.0310331733654294) | > loss_mel_ce: 3.243288993835449 (nan) | > loss: 0.012959670275449753 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13721935017650375) | > loader_time: 0.0046 (0.00468723626725703)  --> TIME: 2025-02-26 15:30:42 -- STEP: 5070/36938 -- GLOBAL_STEP: 1297900 | > loss_text_ce: 0.03537062183022499 (0.031038001722140014) | > loss_mel_ce: 3.056363821029663 (nan) | > loss: 0.012268788181245327 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.13722432477939794) | > loader_time: 0.0046 (0.004686915239638839)  --> TIME: 2025-02-26 15:30:51 -- STEP: 5120/36938 -- GLOBAL_STEP: 1297950 | > loss_text_ce: 0.03488237410783768 (0.031050530306856672) | > loss_mel_ce: 1.6739959716796875 (nan) | > loss: 0.006781263742595911 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13720347392372811) | > loader_time: 0.0048 (0.004686415335163485)  --> TIME: 2025-02-26 15:31:00 -- STEP: 5170/36938 -- GLOBAL_STEP: 1298000 | > loss_text_ce: 0.029777219519019127 (0.031051940911015354) | > loss_mel_ce: 2.3617520332336426 (nan) | > loss: 0.009490196593105793 (nan) | > current_lr: 5e-06 | > step_time: 0.1615 (0.1371992685808667) | > loader_time: 0.0048 (0.004686199950065107)  --> TIME: 2025-02-26 15:31:10 -- STEP: 5220/36938 -- GLOBAL_STEP: 1298050 | > loss_text_ce: 0.03314558044075966 (0.03105583553045446) | > loss_mel_ce: 2.7103395462036133 (nan) | > loss: 0.010886847041547298 (nan) | > current_lr: 5e-06 | > step_time: 0.1881 (0.13719374571723514) | > loader_time: 0.0056 (0.004685493630010974)  --> TIME: 2025-02-26 15:31:19 -- STEP: 5270/36938 -- GLOBAL_STEP: 1298100 | > loss_text_ce: 0.027974026277661324 (0.031050225507566948) | > loss_mel_ce: 2.714308261871338 (nan) | > loss: 0.010882074013352394 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13720186525775555) | > loader_time: 0.0045 (0.004685033838029617)  --> TIME: 2025-02-26 15:31:29 -- STEP: 5320/36938 -- GLOBAL_STEP: 1298150 | > loss_text_ce: 0.04286705330014229 (0.03105467549171579) | > loss_mel_ce: 2.850435733795166 (nan) | > loss: 0.011481360532343388 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.1372245466350615) | > loader_time: 0.0045 (0.004684414092759452)  --> TIME: 2025-02-26 15:31:38 -- STEP: 5370/36938 -- GLOBAL_STEP: 1298200 | > loss_text_ce: 0.03125619515776634 (0.03105576107333964) | > loss_mel_ce: 0.0777587890625 (nan) | > loss: 0.0004325991903897375 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.1371793423729016) | > loader_time: 0.0046 (0.004683807886290607)  --> TIME: 2025-02-26 15:31:47 -- STEP: 5420/36938 -- GLOBAL_STEP: 1298250 | > loss_text_ce: 0.028568778187036514 (0.031065587656971108) | > loss_mel_ce: 2.3680202960968018 (nan) | > loss: 0.009510274976491928 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13714832714123057) | > loader_time: 0.0047 (0.004683564773784803)  --> TIME: 2025-02-26 15:31:56 -- STEP: 5470/36938 -- GLOBAL_STEP: 1298300 | > loss_text_ce: 0.038018085062503815 (0.031052856664028582) | > loss_mel_ce: 2.629819631576538 (nan) | > loss: 0.01058665756136179 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13714943921544231) | > loader_time: 0.0046 (0.0046838014173769356)  --> TIME: 2025-02-26 15:32:05 -- STEP: 5520/36938 -- GLOBAL_STEP: 1298350 | > loss_text_ce: 0.021250009536743164 (0.031060337982700626) | > loss_mel_ce: 1.343820333480835 (nan) | > loss: 0.005416946019977331 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1371416820564133) | > loader_time: 0.0045 (0.004683218615642507)  --> TIME: 2025-02-26 15:32:15 -- STEP: 5570/36938 -- GLOBAL_STEP: 1298400 | > loss_text_ce: 0.029404209926724434 (0.031059852034636043) | > loss_mel_ce: 2.6558024883270264 (nan) | > loss: 0.010655582882463932 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13712226013306889) | > loader_time: 0.0047 (0.0046826210227432245)  --> TIME: 2025-02-26 15:32:24 -- STEP: 5620/36938 -- GLOBAL_STEP: 1298450 | > loss_text_ce: 0.022281911224126816 (0.031054478793428198) | > loss_mel_ce: 2.430178165435791 (nan) | > loss: 0.009731984697282314 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.1371322336145994) | > loader_time: 0.0046 (0.004682516203232097)  --> TIME: 2025-02-26 15:32:34 -- STEP: 5670/36938 -- GLOBAL_STEP: 1298500 | > loss_text_ce: 0.03555484488606453 (0.031057635294782684) | > loss_mel_ce: 2.987687826156616 (nan) | > loss: 0.011996995657682419 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.13716940261699553) | > loader_time: 0.0047 (0.004682533577005716)  --> TIME: 2025-02-26 15:32:43 -- STEP: 5720/36938 -- GLOBAL_STEP: 1298550 | > loss_text_ce: 0.02686776965856552 (0.031055325492272166) | > loss_mel_ce: 2.772235155105591 (nan) | > loss: 0.011107551865279675 (nan) | > current_lr: 5e-06 | > step_time: 0.1434 (0.13720352036969669) | > loader_time: 0.0047 (0.004682722208383217)  --> TIME: 2025-02-26 15:32:52 -- STEP: 5770/36938 -- GLOBAL_STEP: 1298600 | > loss_text_ce: 0.0317765437066555 (0.031051357274316575) | > loss_mel_ce: 2.6672098636627197 (nan) | > loss: 0.01071026362478733 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.13715756270046697) | > loader_time: 0.0047 (0.004682149548257881)  --> TIME: 2025-02-26 15:33:02 -- STEP: 5820/36938 -- GLOBAL_STEP: 1298650 | > loss_text_ce: 0.026171958073973656 (0.031053072869732874) | > loss_mel_ce: 2.804281711578369 (nan) | > loss: 0.01123195979744196 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13717559047581024) | > loader_time: 0.0053 (0.00468278451473854)  --> TIME: 2025-02-26 15:33:11 -- STEP: 5870/36938 -- GLOBAL_STEP: 1298700 | > loss_text_ce: 0.028205012902617455 (0.031055334346365735) | > loss_mel_ce: 2.2252838611602783 (nan) | > loss: 0.008942415937781334 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.1371993860556689) | > loader_time: 0.0048 (0.004683736601314529)  --> TIME: 2025-02-26 15:33:20 -- STEP: 5920/36938 -- GLOBAL_STEP: 1298750 | > loss_text_ce: 0.03752981871366501 (0.03105297465321004) | > loss_mel_ce: 2.687795400619507 (nan) | > loss: 0.010814783163368702 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.13717145231124553) | > loader_time: 0.0047 (0.0046842456266686875)  --> TIME: 2025-02-26 15:33:30 -- STEP: 5970/36938 -- GLOBAL_STEP: 1298800 | > loss_text_ce: 0.03165555000305176 (0.031050569926493084) | > loss_mel_ce: 1.975000023841858 (nan) | > loss: 0.007962919771671295 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13717812249808295) | > loader_time: 0.0045 (0.0046844755185509205)  --> TIME: 2025-02-26 15:33:39 -- STEP: 6020/36938 -- GLOBAL_STEP: 1298850 | > loss_text_ce: 0.03572883829474449 (0.03105899372039072) | > loss_mel_ce: 2.6705102920532227 (nan) | > loss: 0.010739045217633247 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13716032211962767) | > loader_time: 0.0049 (0.004684441826272266)  --> TIME: 2025-02-26 15:33:48 -- STEP: 6070/36938 -- GLOBAL_STEP: 1298900 | > loss_text_ce: 0.03397631272673607 (0.03105869837906181) | > loss_mel_ce: 3.0858192443847656 (nan) | > loss: 0.012380141764879227 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.13716346215966146) | > loader_time: 0.005 (0.004684724328545425)  --> TIME: 2025-02-26 15:33:58 -- STEP: 6120/36938 -- GLOBAL_STEP: 1298950 | > loss_text_ce: 0.021124690771102905 (0.03105126536965638) | > loss_mel_ce: 0.0054267882369458675 (nan) | > loss: 0.00010536301851971075 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13716930217213144) | > loader_time: 0.0046 (0.004684868474411825)  --> TIME: 2025-02-26 15:34:07 -- STEP: 6170/36938 -- GLOBAL_STEP: 1299000 | > loss_text_ce: 0.035866063088178635 (0.031061908061581406) | > loss_mel_ce: 3.3570058345794678 (nan) | > loss: 0.013463778421282768 (nan) | > current_lr: 5e-06 | > step_time: 0.1822 (0.1371801990743978) | > loader_time: 0.0048 (0.004685317136866558)  --> TIME: 2025-02-26 15:34:17 -- STEP: 6220/36938 -- GLOBAL_STEP: 1299050 | > loss_text_ce: 0.0390947200357914 (0.0310539879292621) | > loss_mel_ce: 2.83815598487854 (nan) | > loss: 0.011417661793529987 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.13718778389443198) | > loader_time: 0.0048 (0.004685501883650914)  --> TIME: 2025-02-26 15:34:26 -- STEP: 6270/36938 -- GLOBAL_STEP: 1299100 | > loss_text_ce: 0.029537511989474297 (0.03106530134000205) | > loss_mel_ce: 2.9880619049072266 (nan) | > loss: 0.011974601075053215 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13719530352755258) | > loader_time: 0.0047 (0.004685360659062401)  --> TIME: 2025-02-26 15:34:35 -- STEP: 6320/36938 -- GLOBAL_STEP: 1299150 | > loss_text_ce: 0.02956482581794262 (0.031071799738432836) | > loss_mel_ce: 2.4617509841918945 (nan) | > loss: 0.009886174462735653 (nan) | > current_lr: 5e-06 | > step_time: 0.1822 (0.13716374258451872) | > loader_time: 0.0048 (0.00468506545205661)  --> TIME: 2025-02-26 15:34:44 -- STEP: 6370/36938 -- GLOBAL_STEP: 1299200 | > loss_text_ce: 0.03186291456222534 (0.031068746275685177) | > loss_mel_ce: 1.554040551185608 (nan) | > loss: 0.006293267942965031 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13716871147065984) | > loader_time: 0.0047 (0.004684955022024414)  --> TIME: 2025-02-26 15:34:54 -- STEP: 6420/36938 -- GLOBAL_STEP: 1299250 | > loss_text_ce: 0.041221845895051956 (0.031068231676621042) | > loss_mel_ce: 3.009326696395874 (nan) | > loss: 0.012105352245271206 (nan) | > current_lr: 5e-06 | > step_time: 0.1981 (0.13717744008774377) | > loader_time: 0.0055 (0.004685635915797835)  --> TIME: 2025-02-26 15:35:03 -- STEP: 6470/36938 -- GLOBAL_STEP: 1299300 | > loss_text_ce: 0.016105463728308678 (0.03105976656223237) | > loss_mel_ce: 1.570791482925415 (nan) | > loss: 0.006297210231423378 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13719886285260108) | > loader_time: 0.0047 (0.004686343319817727)  --> TIME: 2025-02-26 15:35:13 -- STEP: 6520/36938 -- GLOBAL_STEP: 1299350 | > loss_text_ce: 0.02354555018246174 (0.031052151033540535) | > loss_mel_ce: 2.691340684890747 (nan) | > loss: 0.01077335886657238 (nan) | > current_lr: 5e-06 | > step_time: 0.1469 (0.13720026005265204) | > loader_time: 0.0049 (0.004686999284416633)  --> TIME: 2025-02-26 15:35:22 -- STEP: 6570/36938 -- GLOBAL_STEP: 1299400 | > loss_text_ce: 0.023125478997826576 (0.03104184936289782) | > loss_mel_ce: 1.6875697374343872 (nan) | > loss: 0.006788473576307297 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13715253914146527) | > loader_time: 0.0048 (0.004686994639705857)  --> TIME: 2025-02-26 15:35:31 -- STEP: 6620/36938 -- GLOBAL_STEP: 1299450 | > loss_text_ce: 0.030671095475554466 (0.031049133093571823) | > loss_mel_ce: 2.724787950515747 (nan) | > loss: 0.010934362187981606 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.1371578145603404) | > loader_time: 0.0046 (0.004687066596800474)  --> TIME: 2025-02-26 15:35:40 -- STEP: 6670/36938 -- GLOBAL_STEP: 1299500 | > loss_text_ce: 0.03043060190975666 (0.03104412595488723) | > loss_mel_ce: 2.2443060874938965 (nan) | > loss: 0.009026733227074146 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.13713559146406462) | > loader_time: 0.0046 (0.0046871046969915705)  --> TIME: 2025-02-26 15:35:50 -- STEP: 6720/36938 -- GLOBAL_STEP: 1299550 | > loss_text_ce: 0.03138000890612602 (0.0310533772527595) | > loss_mel_ce: 2.8565657138824463 (nan) | > loss: 0.011460102163255215 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.1371395889137478) | > loader_time: 0.0052 (0.00468750798276493)  --> TIME: 2025-02-26 15:35:59 -- STEP: 6770/36938 -- GLOBAL_STEP: 1299600 | > loss_text_ce: 0.033458732068538666 (0.031057686719113475) | > loss_mel_ce: 2.8488428592681885 (nan) | > loss: 0.011437705717980862 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13714400242843566) | > loader_time: 0.0045 (0.0046880362481098985)  --> TIME: 2025-02-26 15:36:08 -- STEP: 6820/36938 -- GLOBAL_STEP: 1299650 | > loss_text_ce: 0.03624627739191055 (0.031062850249860205) | > loss_mel_ce: 2.0763185024261475 (nan) | > loss: 0.008383193984627724 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.1371357208472898) | > loader_time: 0.0045 (0.0046881142949079194)  --> TIME: 2025-02-26 15:36:17 -- STEP: 6870/36938 -- GLOBAL_STEP: 1299700 | > loss_text_ce: 0.028994934633374214 (0.031069626004924508) | > loss_mel_ce: 2.7682220935821533 (nan) | > loss: 0.01110006868839264 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13712199102149136) | > loader_time: 0.0046 (0.004687644022868022)  --> TIME: 2025-02-26 15:36:26 -- STEP: 6920/36938 -- GLOBAL_STEP: 1299750 | > loss_text_ce: 0.02824966423213482 (0.031063122758668755) | > loss_mel_ce: 1.760316252708435 (nan) | > loss: 0.007097484078258276 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.1370680190235208) | > loader_time: 0.0048 (0.004687907757786663)  --> TIME: 2025-02-26 15:36:36 -- STEP: 6970/36938 -- GLOBAL_STEP: 1299800 | > loss_text_ce: 0.02720767632126808 (0.031071455342831407) | > loss_mel_ce: 3.084373950958252 (nan) | > loss: 0.01234754640609026 (nan) | > current_lr: 5e-06 | > step_time: 0.1818 (0.13706845139157361) | > loader_time: 0.0048 (0.004687726548959746)  --> TIME: 2025-02-26 15:36:45 -- STEP: 7020/36938 -- GLOBAL_STEP: 1299850 | > loss_text_ce: 0.033365219831466675 (0.031069065095389797) | > loss_mel_ce: 0.02660624124109745 (nan) | > loss: 0.00023798199254088104 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.137065469505441) | > loader_time: 0.0045 (0.004687440055727625)  --> TIME: 2025-02-26 15:36:54 -- STEP: 7070/36938 -- GLOBAL_STEP: 1299900 | > loss_text_ce: 0.03510395064949989 (0.031064337041081472) | > loss_mel_ce: 2.70944881439209 (nan) | > loss: 0.010891083627939224 (nan) | > current_lr: 5e-06 | > step_time: 0.1873 (0.1370659599884234) | > loader_time: 0.0049 (0.004687966383161105)  --> TIME: 2025-02-26 15:37:04 -- STEP: 7120/36938 -- GLOBAL_STEP: 1299950 | > loss_text_ce: 0.035665642470121384 (0.031070177569027906) | > loss_mel_ce: 2.3897738456726074 (nan) | > loss: 0.00962476059794426 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13705569036891016) | > loader_time: 0.0045 (0.0046888374545601035)  --> TIME: 2025-02-26 15:37:13 -- STEP: 7170/36938 -- GLOBAL_STEP: 1300000 | > loss_text_ce: 0.02684786543250084 (0.031074693036945003) | > loss_mel_ce: 2.6556215286254883 (nan) | > loss: 0.010644719935953617 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13709353506149513) | > loader_time: 0.0054 (0.004689830849193968) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1300000.pth  --> TIME: 2025-02-26 15:37:29 -- STEP: 7220/36938 -- GLOBAL_STEP: 1300050 | > loss_text_ce: 0.022858792915940285 (0.031079942299249142) | > loss_mel_ce: 2.089179515838623 (nan) | > loss: 0.008381105028092861 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.137066771713321) | > loader_time: 0.0046 (0.004690461905048831)  --> TIME: 2025-02-26 15:37:38 -- STEP: 7270/36938 -- GLOBAL_STEP: 1300100 | > loss_text_ce: 0.03825756162405014 (0.031089631482533485) | > loss_mel_ce: 2.5848255157470703 (nan) | > loss: 0.01040906086564064 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13707299239041953) | > loader_time: 0.0046 (0.004690180979207923)  --> TIME: 2025-02-26 15:37:47 -- STEP: 7320/36938 -- GLOBAL_STEP: 1300150 | > loss_text_ce: 0.03413586691021919 (0.031091468735557314) | > loss_mel_ce: 3.2936222553253174 (nan) | > loss: 0.013205389492213726 (nan) | > current_lr: 5e-06 | > step_time: 0.1825 (0.13704743313659143) | > loader_time: 0.0048 (0.00468977653915114)  --> TIME: 2025-02-26 15:37:57 -- STEP: 7370/36938 -- GLOBAL_STEP: 1300200 | > loss_text_ce: 0.03322962671518326 (0.03109672020304458) | > loss_mel_ce: 2.9779725074768066 (nan) | > loss: 0.011949215084314346 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.1370777288964846) | > loader_time: 0.0047 (0.004690302599074888)  --> TIME: 2025-02-26 15:38:06 -- STEP: 7420/36938 -- GLOBAL_STEP: 1300250 | > loss_text_ce: 0.026371516287326813 (0.03109549876701619) | > loss_mel_ce: 2.731523275375366 (nan) | > loss: 0.010944027453660965 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13706623354369477) | > loader_time: 0.0046 (0.004690699789401983)  --> TIME: 2025-02-26 15:38:15 -- STEP: 7470/36938 -- GLOBAL_STEP: 1300300 | > loss_text_ce: 0.03282882645726204 (0.031096889579686097) | > loss_mel_ce: 2.478100538253784 (nan) | > loss: 0.009964006021618843 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13706347805110067) | > loader_time: 0.0046 (0.004691349614257001)  --> TIME: 2025-02-26 15:38:25 -- STEP: 7520/36938 -- GLOBAL_STEP: 1300350 | > loss_text_ce: 0.02532031200826168 (0.031099139742350127) | > loss_mel_ce: 2.3659472465515137 (nan) | > loss: 0.009489157237112522 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.13711012620875235) | > loader_time: 0.0047 (0.004691715665320138)  --> TIME: 2025-02-26 15:38:34 -- STEP: 7570/36938 -- GLOBAL_STEP: 1300400 | > loss_text_ce: 0.03281255438923836 (0.03110646730648105) | > loss_mel_ce: 2.387784719467163 (nan) | > loss: 0.009605545550584793 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13709470791104977) | > loader_time: 0.0045 (0.004691544864105271)  --> TIME: 2025-02-26 15:38:44 -- STEP: 7620/36938 -- GLOBAL_STEP: 1300450 | > loss_text_ce: 0.017163829877972603 (0.031098520189518808) | > loss_mel_ce: 1.6081429719924927 (nan) | > loss: 0.006449630483984947 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.1370758048818499) | > loader_time: 0.0045 (0.004691172428331356)  --> TIME: 2025-02-26 15:38:53 -- STEP: 7670/36938 -- GLOBAL_STEP: 1300500 | > loss_text_ce: 0.042710818350315094 (0.031094943244262773) | > loss_mel_ce: 1.3873194456100464 (nan) | > loss: 0.005674723535776138 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.13707653444158419) | > loader_time: 0.0046 (0.004691316811015745)  --> TIME: 2025-02-26 15:39:02 -- STEP: 7720/36938 -- GLOBAL_STEP: 1300550 | > loss_text_ce: 0.02519361674785614 (0.03109134538593784) | > loss_mel_ce: 2.826037883758545 (nan) | > loss: 0.011314411647617817 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13706183501475888) | > loader_time: 0.0046 (0.0046912329493409045)  --> TIME: 2025-02-26 15:39:12 -- STEP: 7770/36938 -- GLOBAL_STEP: 1300600 | > loss_text_ce: 0.02994251623749733 (0.031095323666925405) | > loss_mel_ce: 2.7006795406341553 (nan) | > loss: 0.010835802182555199 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13708256352668927) | > loader_time: 0.0047 (0.004691444891606831)  --> TIME: 2025-02-26 15:39:21 -- STEP: 7820/36938 -- GLOBAL_STEP: 1300650 | > loss_text_ce: 0.021908797323703766 (0.031087142210382293) | > loss_mel_ce: 2.709813356399536 (nan) | > loss: 0.010840167291462421 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.13709258392948684) | > loader_time: 0.0049 (0.004691868913752959)  --> TIME: 2025-02-26 15:39:30 -- STEP: 7870/36938 -- GLOBAL_STEP: 1300700 | > loss_text_ce: 0.03065546602010727 (0.03109050780698051) | > loss_mel_ce: 3.3007993698120117 (nan) | > loss: 0.013220059685409069 (nan) | > current_lr: 5e-06 | > step_time: 0.1781 (0.13709970244610076) | > loader_time: 0.0048 (0.0046915579629823)  --> TIME: 2025-02-26 15:39:40 -- STEP: 7920/36938 -- GLOBAL_STEP: 1300750 | > loss_text_ce: 0.032947883009910583 (0.031087221017441328) | > loss_mel_ce: 1.6512987613677979 (nan) | > loss: 0.006683518644422293 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.1371152522286989) | > loader_time: 0.0045 (0.004691884794620558)  --> TIME: 2025-02-26 15:39:49 -- STEP: 7970/36938 -- GLOBAL_STEP: 1300800 | > loss_text_ce: 0.02245965413749218 (0.03107799403778414) | > loss_mel_ce: 1.7288962602615356 (nan) | > loss: 0.006949825212359428 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13712308382898286) | > loader_time: 0.0046 (0.00469147007318782)  --> TIME: 2025-02-26 15:39:59 -- STEP: 8020/36938 -- GLOBAL_STEP: 1300850 | > loss_text_ce: 0.03101840615272522 (0.031085456840745047) | > loss_mel_ce: 2.7063848972320557 (nan) | > loss: 0.010862712748348713 (nan) | > current_lr: 5e-06 | > step_time: 0.1721 (0.13712413120745479) | > loader_time: 0.0049 (0.004691756217557004)  --> TIME: 2025-02-26 15:40:08 -- STEP: 8070/36938 -- GLOBAL_STEP: 1300900 | > loss_text_ce: 0.029615303501486778 (0.031082388921727367) | > loss_mel_ce: 3.5401785373687744 (nan) | > loss: 0.014165849424898624 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.1371415435159965) | > loader_time: 0.0045 (0.004692237350577321)  --> TIME: 2025-02-26 15:40:17 -- STEP: 8120/36938 -- GLOBAL_STEP: 1300950 | > loss_text_ce: 0.037202488631010056 (0.031084548560283125) | > loss_mel_ce: 2.9343082904815674 (nan) | > loss: 0.011791710741817951 (nan) | > current_lr: 5e-06 | > step_time: 0.1891 (0.1371327788665382) | > loader_time: 0.0056 (0.004692766143770646)  --> TIME: 2025-02-26 15:40:27 -- STEP: 8170/36938 -- GLOBAL_STEP: 1301000 | > loss_text_ce: 0.025950774550437927 (0.031082060455368557) | > loss_mel_ce: 1.0379550457000732 (nan) | > loss: 0.004221848677843809 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13713690093564632) | > loader_time: 0.0045 (0.004693862099104738)  --> TIME: 2025-02-26 15:40:36 -- STEP: 8220/36938 -- GLOBAL_STEP: 1301050 | > loss_text_ce: 0.03357449546456337 (0.031090368521841664) | > loss_mel_ce: 2.253386974334717 (nan) | > loss: 0.009075244888663292 (nan) | > current_lr: 5e-06 | > step_time: 0.1852 (0.1371567418395463) | > loader_time: 0.0049 (0.004694531549792515)  --> TIME: 2025-02-26 15:40:46 -- STEP: 8270/36938 -- GLOBAL_STEP: 1301100 | > loss_text_ce: 0.03647220879793167 (0.031096685287106823) | > loss_mel_ce: 2.937997341156006 (nan) | > loss: 0.011803451925516129 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1371529919925312) | > loader_time: 0.0048 (0.004694362332483594)  --> TIME: 2025-02-26 15:40:55 -- STEP: 8320/36938 -- GLOBAL_STEP: 1301150 | > loss_text_ce: 0.02432936802506447 (0.031090398706258453) | > loss_mel_ce: 3.0046610832214355 (nan) | > loss: 0.012019804678857327 (nan) | > current_lr: 5e-06 | > step_time: 0.1906 (0.13717576070473797) | > loader_time: 0.0049 (0.004694580143460866)  --> TIME: 2025-02-26 15:41:05 -- STEP: 8370/36938 -- GLOBAL_STEP: 1301200 | > loss_text_ce: 0.022619009017944336 (0.03108834995050317) | > loss_mel_ce: 1.4963867664337158 (nan) | > loss: 0.006027800962328911 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13718478673960055) | > loader_time: 0.0048 (0.004695129565440631)  --> TIME: 2025-02-26 15:41:14 -- STEP: 8420/36938 -- GLOBAL_STEP: 1301250 | > loss_text_ce: 0.042122531682252884 (0.03108219905123734) | > loss_mel_ce: 3.1412923336029053 (nan) | > loss: 0.012632600031793118 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13718371187423067) | > loader_time: 0.0046 (0.004695525390235381)  --> TIME: 2025-02-26 15:41:24 -- STEP: 8470/36938 -- GLOBAL_STEP: 1301300 | > loss_text_ce: 0.036392807960510254 (0.03107830436478029) | > loss_mel_ce: 4.879687309265137 (nan) | > loss: 0.019508255645632744 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13720075470497692) | > loader_time: 0.0045 (0.004695525585971733)  --> TIME: 2025-02-26 15:41:33 -- STEP: 8520/36938 -- GLOBAL_STEP: 1301350 | > loss_text_ce: 0.03106270171701908 (0.031082160138752434) | > loss_mel_ce: 2.4263854026794434 (nan) | > loss: 0.009751778095960617 (nan) | > current_lr: 5e-06 | > step_time: 0.1621 (0.13719930755140686) | > loader_time: 0.0046 (0.004695417455664267)  --> TIME: 2025-02-26 15:41:42 -- STEP: 8570/36938 -- GLOBAL_STEP: 1301400 | > loss_text_ce: 0.03558918833732605 (0.031086900613467396) | > loss_mel_ce: 3.3019165992736816 (nan) | > loss: 0.013244071044027805 (nan) | > current_lr: 5e-06 | > step_time: 0.1717 (0.13719940422018098) | > loader_time: 0.0056 (0.004695462763239931)  --> TIME: 2025-02-26 15:41:52 -- STEP: 8620/36938 -- GLOBAL_STEP: 1301450 | > loss_text_ce: 0.04972072318196297 (0.031091830578552703) | > loss_mel_ce: 1.7923370599746704 (nan) | > loss: 0.00730975391343236 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13719897566151507) | > loader_time: 0.0047 (0.004695712773263319)  --> TIME: 2025-02-26 15:42:02 -- STEP: 8670/36938 -- GLOBAL_STEP: 1301500 | > loss_text_ce: 0.027281824499368668 (0.031094572712767186) | > loss_mel_ce: 2.9817707538604736 (nan) | > loss: 0.011940685100853443 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13723843460677043) | > loader_time: 0.0046 (0.004695939990200939)  --> TIME: 2025-02-26 15:42:11 -- STEP: 8720/36938 -- GLOBAL_STEP: 1301550 | > loss_text_ce: 0.03138262778520584 (0.031096103976570377) | > loss_mel_ce: 3.0684916973114014 (nan) | > loss: 0.012301089242100716 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.13726576323356152) | > loader_time: 0.0048 (0.004696096766979326)  --> TIME: 2025-02-26 15:42:20 -- STEP: 8770/36938 -- GLOBAL_STEP: 1301600 | > loss_text_ce: 0.03600645065307617 (0.031099387085513384) | > loss_mel_ce: 2.6366515159606934 (nan) | > loss: 0.010605785995721817 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1372397327912443) | > loader_time: 0.0047 (0.004696622677784557)  --> TIME: 2025-02-26 15:42:30 -- STEP: 8820/36938 -- GLOBAL_STEP: 1301650 | > loss_text_ce: 0.02653650939464569 (0.031091950224214095) | > loss_mel_ce: 3.561631917953491 (nan) | > loss: 0.01423876453191042 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.1372585502881857) | > loader_time: 0.0045 (0.004697406481182768)  --> TIME: 2025-02-26 15:42:39 -- STEP: 8870/36938 -- GLOBAL_STEP: 1301700 | > loss_text_ce: 0.03333242982625961 (0.031085193929343308) | > loss_mel_ce: 2.739523410797119 (nan) | > loss: 0.011003396473824978 (nan) | > current_lr: 5e-06 | > step_time: 0.1937 (0.13725298886369064) | > loader_time: 0.0049 (0.004698568858046679)  --> TIME: 2025-02-26 15:42:48 -- STEP: 8920/36938 -- GLOBAL_STEP: 1301750 | > loss_text_ce: 0.03358939290046692 (0.03108307598743404) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.1372358317332422) | > loader_time: 0.0052 (0.00469877361716711)  --> TIME: 2025-02-26 15:42:57 -- STEP: 8970/36938 -- GLOBAL_STEP: 1301800 | > loss_text_ce: 0.032602403312921524 (0.03108017203536138) | > loss_mel_ce: 3.075270652770996 (nan) | > loss: 0.012332829646766186 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13720259663784962) | > loader_time: 0.0046 (0.004699060510233495)  --> TIME: 2025-02-26 15:43:07 -- STEP: 9020/36938 -- GLOBAL_STEP: 1301850 | > loss_text_ce: 0.031273700296878815 (0.031078868232638536) | > loss_mel_ce: 2.472092390060425 (nan) | > loss: 0.00993399228900671 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13718170670342328) | > loader_time: 0.0046 (0.004698913351130859)  --> TIME: 2025-02-26 15:43:16 -- STEP: 9070/36938 -- GLOBAL_STEP: 1301900 | > loss_text_ce: 0.029520198702812195 (0.031071588519935998) | > loss_mel_ce: 1.0583683252334595 (nan) | > loss: 0.004317018203437328 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13716589437167243) | > loader_time: 0.0045 (0.004698939780810442)  --> TIME: 2025-02-26 15:43:25 -- STEP: 9120/36938 -- GLOBAL_STEP: 1301950 | > loss_text_ce: 0.03115895204246044 (0.031068267813871467) | > loss_mel_ce: 2.7987208366394043 (nan) | > loss: 0.01122968178242445 (nan) | > current_lr: 5e-06 | > step_time: 0.1689 (0.1371740991609143) | > loader_time: 0.0046 (0.004698685805002849)  --> TIME: 2025-02-26 15:43:35 -- STEP: 9170/36938 -- GLOBAL_STEP: 1302000 | > loss_text_ce: 0.02344440668821335 (0.03106700663486315) | > loss_mel_ce: 2.864940881729126 (nan) | > loss: 0.011461847461760044 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.1371711259059848) | > loader_time: 0.0048 (0.0046984308808554484)  --> TIME: 2025-02-26 15:43:44 -- STEP: 9220/36938 -- GLOBAL_STEP: 1302050 | > loss_text_ce: 0.03034309484064579 (0.03106916487911906) | > loss_mel_ce: 3.659254789352417 (nan) | > loss: 0.014641262590885162 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13718763814831034) | > loader_time: 0.0047 (0.004698280915779561)  --> TIME: 2025-02-26 15:43:54 -- STEP: 9270/36938 -- GLOBAL_STEP: 1302100 | > loss_text_ce: 0.041743792593479156 (0.031071347058092634) | > loss_mel_ce: 3.1902530193328857 (nan) | > loss: 0.012825384736061096 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13719233250026475) | > loader_time: 0.0046 (0.004698870122882797)  --> TIME: 2025-02-26 15:44:03 -- STEP: 9320/36938 -- GLOBAL_STEP: 1302150 | > loss_text_ce: 0.031600262969732285 (0.0310685308478261) | > loss_mel_ce: 2.731935977935791 (nan) | > loss: 0.010966414585709572 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.1372258517363557) | > loader_time: 0.0049 (0.004699669707998188)  --> TIME: 2025-02-26 15:44:13 -- STEP: 9370/36938 -- GLOBAL_STEP: 1302200 | > loss_text_ce: 0.027127718552947044 (0.031069416229552494) | > loss_mel_ce: 3.2152037620544434 (nan) | > loss: 0.012866395525634289 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13721392162327115) | > loader_time: 0.0047 (0.004700159873880979)  --> TIME: 2025-02-26 15:44:22 -- STEP: 9420/36938 -- GLOBAL_STEP: 1302250 | > loss_text_ce: 0.02740718051791191 (0.031072105570733862) | > loss_mel_ce: 2.141378879547119 (nan) | > loss: 0.00860629417002201 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13725113729509367) | > loader_time: 0.0046 (0.0047004035577146355)  --> TIME: 2025-02-26 15:44:32 -- STEP: 9470/36938 -- GLOBAL_STEP: 1302300 | > loss_text_ce: 0.029027925804257393 (0.031075210887997744) | > loss_mel_ce: 2.2167460918426514 (nan) | > loss: 0.008911802433431149 (nan) | > current_lr: 5e-06 | > step_time: 0.1265 (0.1372771804410783) | > loader_time: 0.0046 (0.004700324452288423)  --> TIME: 2025-02-26 15:44:41 -- STEP: 9520/36938 -- GLOBAL_STEP: 1302350 | > loss_text_ce: 0.028354298323392868 (0.031080025936351667) | > loss_mel_ce: 2.605968952178955 (nan) | > loss: 0.0104536646977067 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13727728196051983) | > loader_time: 0.0047 (0.004700486124062739)  --> TIME: 2025-02-26 15:44:51 -- STEP: 9570/36938 -- GLOBAL_STEP: 1302400 | > loss_text_ce: 0.024091949686408043 (0.03108250334695402) | > loss_mel_ce: 2.584296703338623 (nan) | > loss: 0.010350748896598816 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.1372735913395262) | > loader_time: 0.0047 (0.0047007094605464785)  --> TIME: 2025-02-26 15:45:00 -- STEP: 9620/36938 -- GLOBAL_STEP: 1302450 | > loss_text_ce: 0.032921791076660156 (0.031077234590282807) | > loss_mel_ce: 3.049105405807495 (nan) | > loss: 0.012230267748236656 (nan) | > current_lr: 5e-06 | > step_time: 0.1879 (0.1372731838305629) | > loader_time: 0.0051 (0.004701420125743206)  --> TIME: 2025-02-26 15:45:09 -- STEP: 9670/36938 -- GLOBAL_STEP: 1302500 | > loss_text_ce: 0.036991946399211884 (0.03107842096120828) | > loss_mel_ce: 2.826749086380005 (nan) | > loss: 0.011364052072167397 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.1372496541801385) | > loader_time: 0.0046 (0.004700986651131626)  --> TIME: 2025-02-26 15:45:19 -- STEP: 9720/36938 -- GLOBAL_STEP: 1302550 | > loss_text_ce: 0.02981453575193882 (0.031086782588328355) | > loss_mel_ce: 2.6719706058502197 (nan) | > loss: 0.01072136964648962 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.13724720085599323) | > loader_time: 0.0048 (0.00470090376006232)  --> TIME: 2025-02-26 15:45:28 -- STEP: 9770/36938 -- GLOBAL_STEP: 1302600 | > loss_text_ce: 0.03113008476793766 (0.03108824590886414) | > loss_mel_ce: 1.433061122894287 (nan) | > loss: 0.005810282658785582 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13723358156249155) | > loader_time: 0.0046 (0.004701037904745233)  --> TIME: 2025-02-26 15:45:37 -- STEP: 9820/36938 -- GLOBAL_STEP: 1302650 | > loss_text_ce: 0.02743358165025711 (0.0310850964172491) | > loss_mel_ce: 0.01953125 (nan) | > loss: 0.00018636838649399579 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13722889556904222) | > loader_time: 0.0046 (0.00470098524132474)  --> TIME: 2025-02-26 15:45:46 -- STEP: 9870/36938 -- GLOBAL_STEP: 1302700 | > loss_text_ce: 0.035511210560798645 (0.031080909919186203) | > loss_mel_ce: 0.9452880620956421 (nan) | > loss: 0.003892060834914446 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13721593933627269) | > loader_time: 0.0047 (0.004700902047007516)  --> TIME: 2025-02-26 15:45:56 -- STEP: 9920/36938 -- GLOBAL_STEP: 1302750 | > loss_text_ce: 0.025018535554409027 (0.03108441974693579) | > loss_mel_ce: 2.283827543258667 (nan) | > loss: 0.009162087924778461 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13724017258613383) | > loader_time: 0.0046 (0.004701316885409812)  --> TIME: 2025-02-26 15:46:05 -- STEP: 9970/36938 -- GLOBAL_STEP: 1302800 | > loss_text_ce: 0.032787155359983444 (0.031083384047868617) | > loss_mel_ce: 3.336052179336548 (nan) | > loss: 0.013368410989642143 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13724508687270962) | > loader_time: 0.0049 (0.004701060708331961)  --> TIME: 2025-02-26 15:46:15 -- STEP: 10020/36938 -- GLOBAL_STEP: 1302850 | > loss_text_ce: 0.03179587423801422 (0.031080880327272915) | > loss_mel_ce: 3.1499109268188477 (nan) | > loss: 0.012625821866095066 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.13722780757321618) | > loader_time: 0.0047 (0.004701320354096186)  --> TIME: 2025-02-26 15:46:24 -- STEP: 10070/36938 -- GLOBAL_STEP: 1302900 | > loss_text_ce: 0.032922640442848206 (0.03107698144855453) | > loss_mel_ce: 3.1786162853240967 (nan) | > loss: 0.012744203209877014 (nan) | > current_lr: 5e-06 | > step_time: 0.1703 (0.1372451615546646) | > loader_time: 0.0048 (0.0047013869233495965)  --> TIME: 2025-02-26 15:46:34 -- STEP: 10120/36938 -- GLOBAL_STEP: 1302950 | > loss_text_ce: 0.024666408076882362 (0.031081344301558355) | > loss_mel_ce: 0.042877197265625 (nan) | > loss: 0.00026803018408827484 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13727095534679015) | > loader_time: 0.0047 (0.004701865237692122)  --> TIME: 2025-02-26 15:46:43 -- STEP: 10170/36938 -- GLOBAL_STEP: 1303000 | > loss_text_ce: 0.02937069907784462 (0.03107829458275288) | > loss_mel_ce: 2.798382520675659 (nan) | > loss: 0.011221243999898434 (nan) | > current_lr: 5e-06 | > step_time: 0.1353 (0.1372761747947256) | > loader_time: 0.0046 (0.004701661829165411)  --> TIME: 2025-02-26 15:46:53 -- STEP: 10220/36938 -- GLOBAL_STEP: 1303050 | > loss_text_ce: 0.0352642796933651 (0.031080282739511238) | > loss_mel_ce: 2.942377805709839 (nan) | > loss: 0.011816040612757206 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13728728012096209) | > loader_time: 0.0048 (0.004702237534196402)  --> TIME: 2025-02-26 15:47:02 -- STEP: 10270/36938 -- GLOBAL_STEP: 1303100 | > loss_text_ce: 0.0450814887881279 (0.031084963752314114) | > loss_mel_ce: 2.8994293212890625 (nan) | > loss: 0.011684566736221313 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.13728555613854876) | > loader_time: 0.0047 (0.004702366477906871)  --> TIME: 2025-02-26 15:47:12 -- STEP: 10320/36938 -- GLOBAL_STEP: 1303150 | > loss_text_ce: 0.0236642025411129 (0.0310824224614365) | > loss_mel_ce: 1.1981849670410156 (nan) | > loss: 0.004848608281463385 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13729541149250346) | > loader_time: 0.0045 (0.004702269753744428)  --> TIME: 2025-02-26 15:47:21 -- STEP: 10370/36938 -- GLOBAL_STEP: 1303200 | > loss_text_ce: 0.0243282001465559 (0.03108758937285471) | > loss_mel_ce: 2.1111135482788086 (nan) | > loss: 0.008473975583910942 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13727192756869921) | > loader_time: 0.0045 (0.0047022475570928784)  --> TIME: 2025-02-26 15:47:30 -- STEP: 10420/36938 -- GLOBAL_STEP: 1303250 | > loss_text_ce: 0.017078664153814316 (0.031085249229518414) | > loss_mel_ce: 0.10491943359375 (nan) | > loss: 0.00048411948955617845 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13726635521524513) | > loader_time: 0.0045 (0.004702090965351527)  --> TIME: 2025-02-26 15:47:39 -- STEP: 10470/36938 -- GLOBAL_STEP: 1303300 | > loss_text_ce: 0.037157438695430756 (0.031084812054216662) | > loss_mel_ce: 2.85486102104187 (nan) | > loss: 0.011476264335215092 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13727467617993175) | > loader_time: 0.0052 (0.004702712676586598)  --> TIME: 2025-02-26 15:47:49 -- STEP: 10520/36938 -- GLOBAL_STEP: 1303350 | > loss_text_ce: 0.027467748150229454 (0.03108860332626821) | > loss_mel_ce: 2.30861496925354 (nan) | > loss: 0.009270169772207737 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13726362685740265) | > loader_time: 0.0047 (0.004702590354948438)  --> TIME: 2025-02-26 15:47:58 -- STEP: 10570/36938 -- GLOBAL_STEP: 1303400 | > loss_text_ce: 0.03230321779847145 (0.031086521230676796) | > loss_mel_ce: 0.5888846516609192 (nan) | > loss: 0.0024650313425809145 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13725674862893264) | > loader_time: 0.0049 (0.004702660714892021)  --> TIME: 2025-02-26 15:48:07 -- STEP: 10620/36938 -- GLOBAL_STEP: 1303450 | > loss_text_ce: 0.029797643423080444 (0.031089192193629406) | > loss_mel_ce: 3.0006110668182373 (nan) | > loss: 0.012025431729853153 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.1372438192367558) | > loader_time: 0.0047 (0.004703319746222198)  --> TIME: 2025-02-26 15:48:17 -- STEP: 10670/36938 -- GLOBAL_STEP: 1303500 | > loss_text_ce: 0.024626612663269043 (0.03109166157993138) | > loss_mel_ce: 2.540639638900757 (nan) | > loss: 0.010179627686738968 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.1372595019505872) | > loader_time: 0.0047 (0.004703450627492313)  --> TIME: 2025-02-26 15:48:26 -- STEP: 10720/36938 -- GLOBAL_STEP: 1303550 | > loss_text_ce: 0.032758716493844986 (0.031091106145680922) | > loss_mel_ce: 2.4821605682373047 (nan) | > loss: 0.00997983943670988 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.1372710781310926) | > loader_time: 0.0047 (0.004703473622229558)  --> TIME: 2025-02-26 15:48:36 -- STEP: 10770/36938 -- GLOBAL_STEP: 1303600 | > loss_text_ce: 0.032827626913785934 (0.03109178505552554) | > loss_mel_ce: 3.108083963394165 (nan) | > loss: 0.012463935650885105 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.13728421163868934) | > loader_time: 0.0056 (0.004703688532971845)  --> TIME: 2025-02-26 15:48:45 -- STEP: 10820/36938 -- GLOBAL_STEP: 1303650 | > loss_text_ce: 0.030175084248185158 (0.031095491041465494) | > loss_mel_ce: 3.182851552963257 (nan) | > loss: 0.012750105932354927 (nan) | > current_lr: 5e-06 | > step_time: 0.186 (0.13729657580363394) | > loader_time: 0.0049 (0.0047035892877913595)  --> TIME: 2025-02-26 15:48:54 -- STEP: 10870/36938 -- GLOBAL_STEP: 1303700 | > loss_text_ce: 0.03241271525621414 (0.031095241473279998) | > loss_mel_ce: 3.012812376022339 (nan) | > loss: 0.012084227055311203 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13728451467810648) | > loader_time: 0.0047 (0.004703593144421188)  --> TIME: 2025-02-26 15:49:04 -- STEP: 10920/36938 -- GLOBAL_STEP: 1303750 | > loss_text_ce: 0.027528943493962288 (0.0310921734404467) | > loss_mel_ce: 2.8905563354492188 (nan) | > loss: 0.011579704470932484 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13729555133061613) | > loader_time: 0.0046 (0.004703500703140923)  --> TIME: 2025-02-26 15:49:13 -- STEP: 10970/36938 -- GLOBAL_STEP: 1303800 | > loss_text_ce: 0.030048944056034088 (0.031087098595556365) | > loss_mel_ce: 2.9411303997039795 (nan) | > loss: 0.01179039478302002 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13728560710235924) | > loader_time: 0.0049 (0.004703689077928477)  --> TIME: 2025-02-26 15:49:22 -- STEP: 11020/36938 -- GLOBAL_STEP: 1303850 | > loss_text_ce: 0.02853146195411682 (0.031087970291222098) | > loss_mel_ce: 2.698460578918457 (nan) | > loss: 0.010821398347616196 (nan) | > current_lr: 5e-06 | > step_time: 0.1435 (0.13729325323053007) | > loader_time: 0.0047 (0.004703637086760967)  --> TIME: 2025-02-26 15:49:32 -- STEP: 11070/36938 -- GLOBAL_STEP: 1303900 | > loss_text_ce: 0.033641260117292404 (0.031092680703222672) | > loss_mel_ce: 2.416363000869751 (nan) | > loss: 0.009722240269184113 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13730027729472127) | > loader_time: 0.0047 (0.004704856721797058)  --> TIME: 2025-02-26 15:49:41 -- STEP: 11120/36938 -- GLOBAL_STEP: 1303950 | > loss_text_ce: 0.030219005420804024 (0.031093442750501736) | > loss_mel_ce: 2.863042116165161 (nan) | > loss: 0.01148119568824768 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.1372957126270959) | > loader_time: 0.0067 (0.004705358280552363)  --> TIME: 2025-02-26 15:49:51 -- STEP: 11170/36938 -- GLOBAL_STEP: 1304000 | > loss_text_ce: 0.03293522447347641 (0.031092925302871072) | > loss_mel_ce: 2.20527720451355 (nan) | > loss: 0.008881795220077038 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13729817856606824) | > loader_time: 0.0048 (0.004705823118812598)  --> TIME: 2025-02-26 15:50:00 -- STEP: 11220/36938 -- GLOBAL_STEP: 1304050 | > loss_text_ce: 0.01934976503252983 (0.031089879740809827) | > loss_mel_ce: 2.3564157485961914 (nan) | > loss: 0.00942764151841402 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1372801314913232) | > loader_time: 0.0047 (0.004705668706944914)  --> TIME: 2025-02-26 15:50:09 -- STEP: 11270/36938 -- GLOBAL_STEP: 1304100 | > loss_text_ce: 0.03338192403316498 (0.031087848085483673) | > loss_mel_ce: 3.1736388206481934 (nan) | > loss: 0.012726273387670517 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.1372704001581557) | > loader_time: 0.0046 (0.004706218122060883)  --> TIME: 2025-02-26 15:50:18 -- STEP: 11320/36938 -- GLOBAL_STEP: 1304150 | > loss_text_ce: 0.02379295602440834 (0.031091019974086868) | > loss_mel_ce: 3.143867254257202 (nan) | > loss: 0.012570081278681755 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13725910658549978) | > loader_time: 0.0053 (0.004706493346514207)  --> TIME: 2025-02-26 15:50:27 -- STEP: 11370/36938 -- GLOBAL_STEP: 1304200 | > loss_text_ce: 0.03303876519203186 (0.031090059116640003) | > loss_mel_ce: 1.8759690523147583 (nan) | > loss: 0.0075754281133413315 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13724498950082625) | > loader_time: 0.0052 (0.0047075093788542756)  --> TIME: 2025-02-26 15:50:37 -- STEP: 11420/36938 -- GLOBAL_STEP: 1304250 | > loss_text_ce: 0.03514190763235092 (0.031093325163686285) | > loss_mel_ce: 2.378237724304199 (nan) | > loss: 0.009576903656125069 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.1372466249557806) | > loader_time: 0.0056 (0.0047080481741348925)  --> TIME: 2025-02-26 15:50:46 -- STEP: 11470/36938 -- GLOBAL_STEP: 1304300 | > loss_text_ce: 0.030373884364962578 (0.03109789193764576) | > loss_mel_ce: 2.0979092121124268 (nan) | > loss: 0.00844556838274002 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.13721841302454357) | > loader_time: 0.0047 (0.004708098242775488)  --> TIME: 2025-02-26 15:50:55 -- STEP: 11520/36938 -- GLOBAL_STEP: 1304350 | > loss_text_ce: 0.028947999700903893 (0.031101532731716158) | > loss_mel_ce: 2.818570852279663 (nan) | > loss: 0.011299679055809975 (nan) | > current_lr: 5e-06 | > step_time: 0.1835 (0.13721125276966267) | > loader_time: 0.0056 (0.0047087388940983245)  --> TIME: 2025-02-26 15:51:04 -- STEP: 11570/36938 -- GLOBAL_STEP: 1304400 | > loss_text_ce: 0.027857691049575806 (0.03109702586358157) | > loss_mel_ce: 2.6981258392333984 (nan) | > loss: 0.010817395523190498 (nan) | > current_lr: 5e-06 | > step_time: 0.1958 (0.13720734607792187) | > loader_time: 0.005 (0.004709090316924114)  --> TIME: 2025-02-26 15:51:13 -- STEP: 11620/36938 -- GLOBAL_STEP: 1304450 | > loss_text_ce: 0.03218959644436836 (0.03109884357870125) | > loss_mel_ce: 2.9011499881744385 (nan) | > loss: 0.011640236712992191 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13719339303018346) | > loader_time: 0.0048 (0.004709347718348792)  --> TIME: 2025-02-26 15:51:22 -- STEP: 11670/36938 -- GLOBAL_STEP: 1304500 | > loss_text_ce: 0.03415372222661972 (0.031097804569426186) | > loss_mel_ce: 3.0239968299865723 (nan) | > loss: 0.012135518714785576 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.1371618942409477) | > loader_time: 0.0046 (0.004709580767941239)  --> TIME: 2025-02-26 15:51:32 -- STEP: 11720/36938 -- GLOBAL_STEP: 1304550 | > loss_text_ce: 0.03439504653215408 (0.031094460993760453) | > loss_mel_ce: 2.094877004623413 (nan) | > loss: 0.008449492044746876 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13716940304118258) | > loader_time: 0.0045 (0.004709520580011816)  --> TIME: 2025-02-26 15:51:41 -- STEP: 11770/36938 -- GLOBAL_STEP: 1304600 | > loss_text_ce: 0.029177134856581688 (0.031096919947806285) | > loss_mel_ce: 1.8479584455490112 (nan) | > loss: 0.007448951248079538 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13719166716783904) | > loader_time: 0.0046 (0.004710084316380048)  --> TIME: 2025-02-26 15:51:51 -- STEP: 11820/36938 -- GLOBAL_STEP: 1304650 | > loss_text_ce: 0.025830525904893875 (0.031097644172347262) | > loss_mel_ce: 2.516602039337158 (nan) | > loss: 0.010089018382132053 (nan) | > current_lr: 5e-06 | > step_time: 0.1613 (0.13721906311822665) | > loader_time: 0.0047 (0.00471025963523463)  --> TIME: 2025-02-26 15:52:01 -- STEP: 11870/36938 -- GLOBAL_STEP: 1304700 | > loss_text_ce: 0.03489026054739952 (0.031095486680765987) | > loss_mel_ce: 3.0983543395996094 (nan) | > loss: 0.012433510273694992 (nan) | > current_lr: 5e-06 | > step_time: 0.1889 (0.13725488260200114) | > loader_time: 0.0051 (0.004710180215104427)  --> TIME: 2025-02-26 15:52:10 -- STEP: 11920/36938 -- GLOBAL_STEP: 1304750 | > loss_text_ce: 0.030583571642637253 (0.03109657421275261) | > loss_mel_ce: 2.740668535232544 (nan) | > loss: 0.01099703274667263 (nan) | > current_lr: 5e-06 | > step_time: 0.1846 (0.13723745282064387) | > loader_time: 0.0048 (0.004710006053815727)  --> TIME: 2025-02-26 15:52:19 -- STEP: 11970/36938 -- GLOBAL_STEP: 1304800 | > loss_text_ce: 0.02923942171037197 (0.03109839071162549) | > loss_mel_ce: 0.3890574276447296 (nan) | > loss: 0.0016599082155153155 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13722145141117192) | > loader_time: 0.0046 (0.004709847907573833)  --> TIME: 2025-02-26 15:52:29 -- STEP: 12020/36938 -- GLOBAL_STEP: 1304850 | > loss_text_ce: 0.03117319941520691 (0.031103131271660103) | > loss_mel_ce: 2.8469250202178955 (nan) | > loss: 0.011421025730669498 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13723070476693874) | > loader_time: 0.0047 (0.004710119893269191)  --> TIME: 2025-02-26 15:52:39 -- STEP: 12070/36938 -- GLOBAL_STEP: 1304900 | > loss_text_ce: 0.023617396131157875 (0.03110482244904353) | > loss_mel_ce: 2.525373697280884 (nan) | > loss: 0.010115045122802258 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13726894067355033) | > loader_time: 0.0048 (0.0047100764221658965)  --> TIME: 2025-02-26 15:52:48 -- STEP: 12120/36938 -- GLOBAL_STEP: 1304950 | > loss_text_ce: 0.046434614807367325 (0.031109199550835777) | > loss_mel_ce: 3.081022024154663 (nan) | > loss: 0.0124105429276824 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.1372702600342216) | > loader_time: 0.0046 (0.004709914198803014)  --> TIME: 2025-02-26 15:52:58 -- STEP: 12170/36938 -- GLOBAL_STEP: 1305000 | > loss_text_ce: 0.034171704202890396 (0.031105477528091008) | > loss_mel_ce: 2.768484592437744 (nan) | > loss: 0.011121653020381927 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.1372934814153783) | > loader_time: 0.0047 (0.0047100874862419945)  --> TIME: 2025-02-26 15:53:07 -- STEP: 12220/36938 -- GLOBAL_STEP: 1305050 | > loss_text_ce: 0.03172686696052551 (0.031110499840635204) | > loss_mel_ce: 2.5728180408477783 (nan) | > loss: 0.010335495695471764 (nan) | > current_lr: 5e-06 | > step_time: 0.1679 (0.13730830789197723) | > loader_time: 0.0046 (0.004709966795339113)  --> TIME: 2025-02-26 15:53:16 -- STEP: 12270/36938 -- GLOBAL_STEP: 1305100 | > loss_text_ce: 0.03688251972198486 (0.031112464780432718) | > loss_mel_ce: 2.4123644828796387 (nan) | > loss: 0.009719233959913254 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13730019366925816) | > loader_time: 0.0046 (0.004709724692085827)  --> TIME: 2025-02-26 15:53:25 -- STEP: 12320/36938 -- GLOBAL_STEP: 1305150 | > loss_text_ce: 0.030093638226389885 (0.031109417181208052) | > loss_mel_ce: 2.5937397480010986 (nan) | > loss: 0.010412038303911686 (nan) | > current_lr: 5e-06 | > step_time: 0.1697 (0.13728663849753264) | > loader_time: 0.0049 (0.004709547932271812)  --> TIME: 2025-02-26 15:53:35 -- STEP: 12370/36938 -- GLOBAL_STEP: 1305200 | > loss_text_ce: 0.02458665706217289 (0.031110711242904227) | > loss_mel_ce: 2.608442783355713 (nan) | > loss: 0.010448530316352844 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.1372973908506874) | > loader_time: 0.0046 (0.004709472902956349)  --> TIME: 2025-02-26 15:53:44 -- STEP: 12420/36938 -- GLOBAL_STEP: 1305250 | > loss_text_ce: 0.03240547701716423 (0.03111751426893156) | > loss_mel_ce: 2.5139105319976807 (nan) | > loss: 0.010104428976774216 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.13728873219083132) | > loader_time: 0.0047 (0.004709918045959204)  --> TIME: 2025-02-26 15:53:53 -- STEP: 12470/36938 -- GLOBAL_STEP: 1305300 | > loss_text_ce: 0.02180582657456398 (0.03111237765953793) | > loss_mel_ce: 0.8546162843704224 (nan) | > loss: 0.0034778655972331762 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13727912830178465) | > loader_time: 0.0045 (0.0047100856768769244)  --> TIME: 2025-02-26 15:54:03 -- STEP: 12520/36938 -- GLOBAL_STEP: 1305350 | > loss_text_ce: 0.033111222088336945 (0.03110700896041045) | > loss_mel_ce: 2.776064395904541 (nan) | > loss: 0.011147523298859596 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.1372756771195814) | > loader_time: 0.0053 (0.004710253701804144)  --> TIME: 2025-02-26 15:54:12 -- STEP: 12570/36938 -- GLOBAL_STEP: 1305400 | > loss_text_ce: 0.028706466779112816 (0.031111136695048523) | > loss_mel_ce: 0.8797143697738647 (nan) | > loss: 0.0036048449110239744 (nan) | > current_lr: 5e-06 | > step_time: 0.1569 (0.1372848847790571) | > loader_time: 0.0054 (0.004710473005223458)  --> TIME: 2025-02-26 15:54:22 -- STEP: 12620/36938 -- GLOBAL_STEP: 1305450 | > loss_text_ce: 0.034433238208293915 (0.03111831693446615) | > loss_mel_ce: 3.0358688831329346 (nan) | > loss: 0.012183738872408867 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13729160364758586) | > loader_time: 0.0046 (0.004710696389671341)  --> TIME: 2025-02-26 15:54:31 -- STEP: 12670/36938 -- GLOBAL_STEP: 1305500 | > loss_text_ce: 0.02214767225086689 (0.03111616911737053) | > loss_mel_ce: 0.2920774519443512 (nan) | > loss: 0.0012469252105802298 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.1373111353018385) | > loader_time: 0.0044 (0.004710673683726494)  --> TIME: 2025-02-26 15:54:41 -- STEP: 12720/36938 -- GLOBAL_STEP: 1305550 | > loss_text_ce: 0.033500783145427704 (0.031112580354627127) | > loss_mel_ce: 2.546461820602417 (nan) | > loss: 0.01023794710636139 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13731320884617626) | > loader_time: 0.0046 (0.004711134085115369)  --> TIME: 2025-02-26 15:54:50 -- STEP: 12770/36938 -- GLOBAL_STEP: 1305600 | > loss_text_ce: 0.03383727744221687 (0.031115951492635647) | > loss_mel_ce: 1.862572193145752 (nan) | > loss: 0.00752543518319726 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13731978047892934) | > loader_time: 0.0047 (0.004710823442194714)  --> TIME: 2025-02-26 15:55:00 -- STEP: 12820/36938 -- GLOBAL_STEP: 1305650 | > loss_text_ce: 0.033832669258117676 (0.031113107082929446) | > loss_mel_ce: 2.23808217048645 (nan) | > loss: 0.009015535935759544 (nan) | > current_lr: 5e-06 | > step_time: 0.1709 (0.13732291407964523) | > loader_time: 0.0055 (0.004710908427067348)  --> TIME: 2025-02-26 15:55:09 -- STEP: 12870/36938 -- GLOBAL_STEP: 1305700 | > loss_text_ce: 0.026760458946228027 (0.031110666959351847) | > loss_mel_ce: 3.067269802093506 (nan) | > loss: 0.01227789930999279 (nan) | > current_lr: 5e-06 | > step_time: 0.1796 (0.1373196753588589) | > loader_time: 0.0047 (0.004711214979211748)  --> TIME: 2025-02-26 15:55:18 -- STEP: 12920/36938 -- GLOBAL_STEP: 1305750 | > loss_text_ce: 0.01975717581808567 (0.03110796254807247) | > loss_mel_ce: 2.5430965423583984 (nan) | > loss: 0.01017005555331707 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13732853696073155) | > loader_time: 0.0052 (0.004711040310077236)  --> TIME: 2025-02-26 15:55:28 -- STEP: 12970/36938 -- GLOBAL_STEP: 1305800 | > loss_text_ce: 0.028632143512368202 (0.031104750233237777) | > loss_mel_ce: 2.2727458477020264 (nan) | > loss: 0.009132453240454197 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13732729243055697) | > loader_time: 0.0046 (0.004711319505019453)  --> TIME: 2025-02-26 15:55:37 -- STEP: 13020/36938 -- GLOBAL_STEP: 1305850 | > loss_text_ce: 0.03652477264404297 (0.03110708986085692) | > loss_mel_ce: 2.6816928386688232 (nan) | > loss: 0.010786578059196472 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13733547935104942) | > loader_time: 0.0047 (0.004711578536875948)  --> TIME: 2025-02-26 15:55:46 -- STEP: 13070/36938 -- GLOBAL_STEP: 1305900 | > loss_text_ce: 0.042714230716228485 (0.031108837468761953) | > loss_mel_ce: 2.344026565551758 (nan) | > loss: 0.009471193887293339 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13732999359822573) | > loader_time: 0.0046 (0.004711425769575309)  --> TIME: 2025-02-26 15:55:56 -- STEP: 13120/36938 -- GLOBAL_STEP: 1305950 | > loss_text_ce: 0.024511169642210007 (0.03110793538174376) | > loss_mel_ce: 2.1269123554229736 (nan) | > loss: 0.008537394925951958 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.1373317105195869) | > loader_time: 0.0046 (0.004711206747991257)  --> TIME: 2025-02-26 15:56:05 -- STEP: 13170/36938 -- GLOBAL_STEP: 1306000 | > loss_text_ce: 0.03475043177604675 (0.031109121815572318) | > loss_mel_ce: 2.933863639831543 (nan) | > loss: 0.011780215427279472 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.13731629517192417) | > loader_time: 0.0048 (0.004711304040414935)  --> TIME: 2025-02-26 15:56:14 -- STEP: 13220/36938 -- GLOBAL_STEP: 1306050 | > loss_text_ce: 0.03661055490374565 (0.031107351663712632) | > loss_mel_ce: 1.9508836269378662 (nan) | > loss: 0.007886881940066814 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.1373175208940085) | > loader_time: 0.0048 (0.004711565866773297)  --> TIME: 2025-02-26 15:56:24 -- STEP: 13270/36938 -- GLOBAL_STEP: 1306100 | > loss_text_ce: 0.028341390192508698 (0.031107249067353748) | > loss_mel_ce: 2.9430081844329834 (nan) | > loss: 0.011791069991886616 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1373239047741115) | > loader_time: 0.0064 (0.0047119941941399755)  --> TIME: 2025-02-26 15:56:33 -- STEP: 13320/36938 -- GLOBAL_STEP: 1306150 | > loss_text_ce: 0.02760050632059574 (0.03110881374411945) | > loss_mel_ce: 2.5479345321655273 (nan) | > loss: 0.010220377705991268 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13730423139022255) | > loader_time: 0.0047 (0.00471227210205236)  --> TIME: 2025-02-26 15:56:42 -- STEP: 13370/36938 -- GLOBAL_STEP: 1306200 | > loss_text_ce: 0.03520640730857849 (0.03110033282773289) | > loss_mel_ce: 2.3039236068725586 (nan) | > loss: 0.009282262064516544 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13731546838870204) | > loader_time: 0.0046 (0.0047124692906883765)  --> TIME: 2025-02-26 15:56:52 -- STEP: 13420/36938 -- GLOBAL_STEP: 1306250 | > loss_text_ce: 0.0379401370882988 (0.031095111369531483) | > loss_mel_ce: 2.8223838806152344 (nan) | > loss: 0.011350492015480995 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13730839943921305) | > loader_time: 0.0046 (0.004712218339325932)  --> TIME: 2025-02-26 15:57:01 -- STEP: 13470/36938 -- GLOBAL_STEP: 1306300 | > loss_text_ce: 0.027161022648215294 (0.031096425551766255) | > loss_mel_ce: 1.431432843208313 (nan) | > loss: 0.00578807108104229 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13732667787037853) | > loader_time: 0.0044 (0.004712384191370026)  --> TIME: 2025-02-26 15:57:11 -- STEP: 13520/36938 -- GLOBAL_STEP: 1306350 | > loss_text_ce: 0.036735258996486664 (0.031100514297684952) | > loss_mel_ce: 2.8852427005767822 (nan) | > loss: 0.011595151387155056 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13732188488604735) | > loader_time: 0.0047 (0.004712585813900399)  --> TIME: 2025-02-26 15:57:20 -- STEP: 13570/36938 -- GLOBAL_STEP: 1306400 | > loss_text_ce: 0.022435808554291725 (0.03110311000876255) | > loss_mel_ce: 3.014105796813965 (nan) | > loss: 0.012049769051373005 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13731357439880235) | > loader_time: 0.0045 (0.0047122886976777335)  --> TIME: 2025-02-26 15:57:29 -- STEP: 13620/36938 -- GLOBAL_STEP: 1306450 | > loss_text_ce: 0.024962574243545532 (0.031099691228030274) | > loss_mel_ce: 0.07589340209960938 (nan) | > loss: 0.0004002221394330263 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13730131982881733) | > loader_time: 0.0047 (0.004712182292153982)  --> TIME: 2025-02-26 15:57:38 -- STEP: 13670/36938 -- GLOBAL_STEP: 1306500 | > loss_text_ce: 0.03769606351852417 (0.03109993914989775) | > loss_mel_ce: 2.623384952545166 (nan) | > loss: 0.010559845715761185 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.1373125390755316) | > loader_time: 0.005 (0.0047120871470631796)  --> TIME: 2025-02-26 15:57:48 -- STEP: 13720/36938 -- GLOBAL_STEP: 1306550 | > loss_text_ce: 0.03398440405726433 (0.03110159563189834) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13731635092993835) | > loader_time: 0.0046 (0.004712139656522854)  --> TIME: 2025-02-26 15:57:57 -- STEP: 13770/36938 -- GLOBAL_STEP: 1306600 | > loss_text_ce: 0.037939153611660004 (0.031101866957806618) | > loss_mel_ce: 0.2279401570558548 (nan) | > loss: 0.0010550767183303833 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13731270806029267) | > loader_time: 0.0055 (0.004712294527994323)  --> TIME: 2025-02-26 15:58:06 -- STEP: 13820/36938 -- GLOBAL_STEP: 1306650 | > loss_text_ce: 0.036672063171863556 (0.031106156908318014) | > loss_mel_ce: 0.1335880011320114 (nan) | > loss: 0.0006756352377124131 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13729261939673948) | > loader_time: 0.0047 (0.004712149030737845)  --> TIME: 2025-02-26 15:58:16 -- STEP: 13870/36938 -- GLOBAL_STEP: 1306700 | > loss_text_ce: 0.0210709348320961 (0.031103601373890555) | > loss_mel_ce: 2.4114997386932373 (nan) | > loss: 0.009653058834373951 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1372936121748775) | > loader_time: 0.0046 (0.004712105794355152)  --> TIME: 2025-02-26 15:58:25 -- STEP: 13920/36938 -- GLOBAL_STEP: 1306750 | > loss_text_ce: 0.031106846407055855 (0.03110055691173204) | > loss_mel_ce: 2.3345844745635986 (nan) | > loss: 0.009387664496898651 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13729216101868363) | > loader_time: 0.0046 (0.004712125967288815)  --> TIME: 2025-02-26 15:58:34 -- STEP: 13970/36938 -- GLOBAL_STEP: 1306800 | > loss_text_ce: 0.0360259972512722 (0.031099860979285444) | > loss_mel_ce: 2.7822651863098145 (nan) | > loss: 0.011183695867657661 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.1372812030822269) | > loader_time: 0.0046 (0.004711792344439406)  --> TIME: 2025-02-26 15:58:43 -- STEP: 14020/36938 -- GLOBAL_STEP: 1306850 | > loss_text_ce: 0.02819501981139183 (0.031101807345504662) | > loss_mel_ce: 2.633948802947998 (nan) | > loss: 0.010564062744379044 (nan) | > current_lr: 5e-06 | > step_time: 0.1764 (0.13727141977546856) | > loader_time: 0.0048 (0.00471145213925717)  --> TIME: 2025-02-26 15:58:53 -- STEP: 14070/36938 -- GLOBAL_STEP: 1306900 | > loss_text_ce: 0.034831833094358444 (0.031099929383015305) | > loss_mel_ce: 2.980334997177124 (nan) | > loss: 0.011964947916567326 (nan) | > current_lr: 5e-06 | > step_time: 0.1537 (0.13729188621849012) | > loader_time: 0.0046 (0.0047112151419091)  --> TIME: 2025-02-26 15:59:02 -- STEP: 14120/36938 -- GLOBAL_STEP: 1306950 | > loss_text_ce: 0.04351974278688431 (0.031101816095525224) | > loss_mel_ce: 1.0301069021224976 (nan) | > loss: 0.004260423593223095 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13728979695957677) | > loader_time: 0.0047 (0.004710931683397134)  --> TIME: 2025-02-26 15:59:12 -- STEP: 14170/36938 -- GLOBAL_STEP: 1307000 | > loss_text_ce: 0.03254147619009018 (0.03110223608538403) | > loss_mel_ce: 0.018390655517578125 (nan) | > loss: 0.0002021116524701938 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13728926186840856) | > loader_time: 0.0046 (0.004710664039105679)  --> TIME: 2025-02-26 15:59:21 -- STEP: 14220/36938 -- GLOBAL_STEP: 1307050 | > loss_text_ce: 0.028520068153738976 (0.031102456860362164) | > loss_mel_ce: 2.613905429840088 (nan) | > loss: 0.010485815815627575 (nan) | > current_lr: 5e-06 | > step_time: 0.1734 (0.1372963187060776) | > loader_time: 0.0049 (0.004710455718758045)  --> TIME: 2025-02-26 15:59:31 -- STEP: 14270/36938 -- GLOBAL_STEP: 1307100 | > loss_text_ce: 0.02671009488403797 (0.03110479215543767) | > loss_mel_ce: 2.701158046722412 (nan) | > loss: 0.01082487404346466 (nan) | > current_lr: 5e-06 | > step_time: 0.1933 (0.13730817930466135) | > loader_time: 0.0049 (0.004710388734947133)  --> TIME: 2025-02-26 15:59:41 -- STEP: 14320/36938 -- GLOBAL_STEP: 1307150 | > loss_text_ce: 0.0347607284784317 (0.031104461253977086) | > loss_mel_ce: 2.5917723178863525 (nan) | > loss: 0.010422750376164913 (nan) | > current_lr: 5e-06 | > step_time: 0.1734 (0.13732086470007207) | > loader_time: 0.0048 (0.004710243534109422)  --> TIME: 2025-02-26 15:59:50 -- STEP: 14370/36938 -- GLOBAL_STEP: 1307200 | > loss_text_ce: 0.030239049345254898 (0.031102868729311825) | > loss_mel_ce: 2.993393659591675 (nan) | > loss: 0.01199854351580143 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.1373254047472107) | > loader_time: 0.0047 (0.004710042451766912)  --> TIME: 2025-02-26 16:00:00 -- STEP: 14420/36938 -- GLOBAL_STEP: 1307250 | > loss_text_ce: 0.03128032758831978 (0.031101572177671257) | > loss_mel_ce: 2.9203760623931885 (nan) | > loss: 0.011712922714650631 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.13733137477287438) | > loader_time: 0.0048 (0.00470985240314603)  --> TIME: 2025-02-26 16:00:09 -- STEP: 14470/36938 -- GLOBAL_STEP: 1307300 | > loss_text_ce: 0.034549832344055176 (0.031096288991310655) | > loss_mel_ce: 2.5941684246063232 (nan) | > loss: 0.010431422851979733 (nan) | > current_lr: 5e-06 | > step_time: 0.1837 (0.13734425035277603) | > loader_time: 0.0048 (0.004710695837478245)  --> TIME: 2025-02-26 16:00:18 -- STEP: 14520/36938 -- GLOBAL_STEP: 1307350 | > loss_text_ce: 0.025548264384269714 (0.031093686054840416) | > loss_mel_ce: 2.5238468647003174 (nan) | > loss: 0.01011664792895317 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13732497332509844) | > loader_time: 0.0045 (0.004711277737761965)  --> TIME: 2025-02-26 16:00:27 -- STEP: 14570/36938 -- GLOBAL_STEP: 1307400 | > loss_text_ce: 0.029302923008799553 (0.031098057897972403) | > loss_mel_ce: 2.5855250358581543 (nan) | > loss: 0.01037630159407854 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.13731688130940944) | > loader_time: 0.0049 (0.00471137708887793)  --> TIME: 2025-02-26 16:00:37 -- STEP: 14620/36938 -- GLOBAL_STEP: 1307450 | > loss_text_ce: 0.02838125079870224 (0.03109466215925579) | > loss_mel_ce: 2.645416021347046 (nan) | > loss: 0.010610307566821575 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.1373233381928885) | > loader_time: 0.0047 (0.0047112337215491195)  --> TIME: 2025-02-26 16:00:46 -- STEP: 14670/36938 -- GLOBAL_STEP: 1307500 | > loss_text_ce: 0.027568155899643898 (0.031091760834025112) | > loss_mel_ce: 3.327878475189209 (nan) | > loss: 0.013315265066921711 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13733349881695464) | > loader_time: 0.0046 (0.004711128223811817)  --> TIME: 2025-02-26 16:00:56 -- STEP: 14720/36938 -- GLOBAL_STEP: 1307550 | > loss_text_ce: 0.033113040030002594 (0.031094262792549156) | > loss_mel_ce: 0.03270721435546875 (nan) | > loss: 0.00026119151152670383 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13734614565644532) | > loader_time: 0.0045 (0.004711247721443983)  --> TIME: 2025-02-26 16:01:05 -- STEP: 14770/36938 -- GLOBAL_STEP: 1307600 | > loss_text_ce: 0.029922358691692352 (0.031094202099048916) | > loss_mel_ce: 2.7854230403900146 (nan) | > loss: 0.011172005906701088 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13732332820091633) | > loader_time: 0.0047 (0.0047113150781929545)  --> TIME: 2025-02-26 16:01:15 -- STEP: 14820/36938 -- GLOBAL_STEP: 1307650 | > loss_text_ce: 0.032088134437799454 (0.03109314997533434) | > loss_mel_ce: 0.03076171875 (nan) | > loss: 0.0002494041691534221 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13734306389825335) | > loader_time: 0.0046 (0.004711517438232152)  --> TIME: 2025-02-26 16:01:24 -- STEP: 14870/36938 -- GLOBAL_STEP: 1307700 | > loss_text_ce: 0.020636506378650665 (0.03109418146785727) | > loss_mel_ce: 2.1614224910736084 (nan) | > loss: 0.00865896511822939 (nan) | > current_lr: 5e-06 | > step_time: 0.1566 (0.13735588002541552) | > loader_time: 0.0046 (0.004711641027531348)  --> TIME: 2025-02-26 16:01:33 -- STEP: 14920/36938 -- GLOBAL_STEP: 1307750 | > loss_text_ce: 0.03117639198899269 (0.031092204789920704) | > loss_mel_ce: 1.0965697765350342 (nan) | > loss: 0.0044751837849617004 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13734990927233395) | > loader_time: 0.0046 (0.004712098405444247)  --> TIME: 2025-02-26 16:01:43 -- STEP: 14970/36938 -- GLOBAL_STEP: 1307800 | > loss_text_ce: 0.03161286935210228 (0.031094077372277727) | > loss_mel_ce: 3.175297260284424 (nan) | > loss: 0.012725834734737873 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.1373635394300864) | > loader_time: 0.0053 (0.004712398178034309)  --> TIME: 2025-02-26 16:01:52 -- STEP: 15020/36938 -- GLOBAL_STEP: 1307850 | > loss_text_ce: 0.02452871948480606 (0.031094282965291136) | > loss_mel_ce: 2.5925726890563965 (nan) | > loss: 0.010385323315858841 (nan) | > current_lr: 5e-06 | > step_time: 0.1452 (0.13735803840322222) | > loader_time: 0.0048 (0.004712954389747363)  --> TIME: 2025-02-26 16:02:02 -- STEP: 15070/36938 -- GLOBAL_STEP: 1307900 | > loss_text_ce: 0.025073163211345673 (0.031099376534980086) | > loss_mel_ce: 3.01448917388916 (nan) | > loss: 0.012061755172908306 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13736126313718905) | > loader_time: 0.0046 (0.00471336931761478)  --> TIME: 2025-02-26 16:02:11 -- STEP: 15120/36938 -- GLOBAL_STEP: 1307950 | > loss_text_ce: 0.0331820584833622 (0.03110051436612146) | > loss_mel_ce: 2.398519515991211 (nan) | > loss: 0.00964961014688015 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13736898843888826) | > loader_time: 0.0046 (0.0047132211387472586)  --> TIME: 2025-02-26 16:02:20 -- STEP: 15170/36938 -- GLOBAL_STEP: 1308000 | > loss_text_ce: 0.03559020906686783 (0.031099933836776113) | > loss_mel_ce: 2.5182697772979736 (nan) | > loss: 0.010134365409612656 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13735531537880083) | > loader_time: 0.0046 (0.0047129492109043245)  --> TIME: 2025-02-26 16:02:30 -- STEP: 15220/36938 -- GLOBAL_STEP: 1308050 | > loss_text_ce: 0.02871057018637657 (0.03109746243622345) | > loss_mel_ce: 2.0234897136688232 (nan) | > loss: 0.008143652230501175 (nan) | > current_lr: 5e-06 | > step_time: 0.1444 (0.13734734672441062) | > loader_time: 0.0053 (0.004712918506501986)  --> TIME: 2025-02-26 16:02:39 -- STEP: 15270/36938 -- GLOBAL_STEP: 1308100 | > loss_text_ce: 0.028138622641563416 (0.031093752698956964) | > loss_mel_ce: 2.2460720539093018 (nan) | > loss: 0.009024646133184433 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.13733382467046254) | > loader_time: 0.0048 (0.004712689664524658)  --> TIME: 2025-02-26 16:02:48 -- STEP: 15320/36938 -- GLOBAL_STEP: 1308150 | > loss_text_ce: 0.029851753264665604 (0.03109136797291988) | > loss_mel_ce: 1.906171202659607 (nan) | > loss: 0.007682631257921457 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13732069674733358) | > loader_time: 0.0052 (0.004712845560153511)  --> TIME: 2025-02-26 16:02:57 -- STEP: 15370/36938 -- GLOBAL_STEP: 1308200 | > loss_text_ce: 0.03365562856197357 (0.031092190827263028) | > loss_mel_ce: 0.8076615929603577 (nan) | > loss: 0.0033385606948286295 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13732512681299575) | > loader_time: 0.0046 (0.004712895782411115)  --> TIME: 2025-02-26 16:03:07 -- STEP: 15420/36938 -- GLOBAL_STEP: 1308250 | > loss_text_ce: 0.028555206954479218 (0.03109053463775341) | > loss_mel_ce: 2.738753318786621 (nan) | > loss: 0.010981383733451366 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13732186778510072) | > loader_time: 0.0047 (0.004712941751715097)  --> TIME: 2025-02-26 16:03:16 -- STEP: 15470/36938 -- GLOBAL_STEP: 1308300 | > loss_text_ce: 0.029353968799114227 (0.031091956192002654) | > loss_mel_ce: 1.9646347761154175 (nan) | > loss: 0.007912654429674149 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.1373143484766584) | > loader_time: 0.0047 (0.004712757743707202)  --> TIME: 2025-02-26 16:03:26 -- STEP: 15520/36938 -- GLOBAL_STEP: 1308350 | > loss_text_ce: 0.03620607405900955 (0.031091127764510434) | > loss_mel_ce: 3.237985134124756 (nan) | > loss: 0.01299282256513834 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.13733104336814728) | > loader_time: 0.0047 (0.004713038040190606)  --> TIME: 2025-02-26 16:03:35 -- STEP: 15570/36938 -- GLOBAL_STEP: 1308400 | > loss_text_ce: 0.022854089736938477 (0.031091223638588485) | > loss_mel_ce: 2.9236092567443848 (nan) | > loss: 0.011692315340042114 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13733920190452173) | > loader_time: 0.0045 (0.00471323371010094)  --> TIME: 2025-02-26 16:03:45 -- STEP: 15620/36938 -- GLOBAL_STEP: 1308450 | > loss_text_ce: 0.03022017888724804 (0.03109129220271118) | > loss_mel_ce: 2.6814959049224854 (nan) | > loss: 0.010760779492557049 (nan) | > current_lr: 5e-06 | > step_time: 0.1501 (0.1373507764519553) | > loader_time: 0.0046 (0.004713563699270463)  --> TIME: 2025-02-26 16:03:54 -- STEP: 15670/36938 -- GLOBAL_STEP: 1308500 | > loss_text_ce: 0.026240436360239983 (0.031091208975160663) | > loss_mel_ce: 2.0344696044921875 (nan) | > loss: 0.008177421055734158 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13734568895316784) | > loader_time: 0.0046 (0.004714512840227155)  --> TIME: 2025-02-26 16:04:04 -- STEP: 15720/36938 -- GLOBAL_STEP: 1308550 | > loss_text_ce: 0.022651175037026405 (0.03108970315311739) | > loss_mel_ce: 2.3697705268859863 (nan) | > loss: 0.009493737481534481 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13735621489641264) | > loader_time: 0.0046 (0.004714467916779824)  --> TIME: 2025-02-26 16:04:13 -- STEP: 15770/36938 -- GLOBAL_STEP: 1308600 | > loss_text_ce: 0.034733522683382034 (0.031091462237820663) | > loss_mel_ce: 2.9519171714782715 (nan) | > loss: 0.011851789429783821 (nan) | > current_lr: 5e-06 | > step_time: 0.1881 (0.13734941412998838) | > loader_time: 0.0048 (0.004714272577095857)  --> TIME: 2025-02-26 16:04:22 -- STEP: 15820/36938 -- GLOBAL_STEP: 1308650 | > loss_text_ce: 0.0273640938103199 (0.03109124291013554) | > loss_mel_ce: 2.925835609436035 (nan) | > loss: 0.011719047091901302 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13735897577215783) | > loader_time: 0.0047 (0.0047140887805393545)  --> TIME: 2025-02-26 16:04:32 -- STEP: 15870/36938 -- GLOBAL_STEP: 1308700 | > loss_text_ce: 0.02943800389766693 (0.031093000430876663) | > loss_mel_ce: 2.9881272315979004 (nan) | > loss: 0.011974466033279896 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13736502395610597) | > loader_time: 0.0047 (0.004714023127973472)  --> TIME: 2025-02-26 16:04:41 -- STEP: 15920/36938 -- GLOBAL_STEP: 1308750 | > loss_text_ce: 0.027180060744285583 (0.0310906558116809) | > loss_mel_ce: 2.2187108993530273 (nan) | > loss: 0.008912266232073307 (nan) | > current_lr: 5e-06 | > step_time: 0.1606 (0.13735819455966225) | > loader_time: 0.005 (0.004714017537370977)  --> TIME: 2025-02-26 16:04:50 -- STEP: 15970/36938 -- GLOBAL_STEP: 1308800 | > loss_text_ce: 0.02545055001974106 (0.031090512676905133) | > loss_mel_ce: 2.5003066062927246 (nan) | > loss: 0.010022846050560474 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.13734424842769924) | > loader_time: 0.0046 (0.004713926348151733)  --> TIME: 2025-02-26 16:04:59 -- STEP: 16020/36938 -- GLOBAL_STEP: 1308850 | > loss_text_ce: 0.03165171295404434 (0.031087444932112813) | > loss_mel_ce: 3.0736396312713623 (nan) | > loss: 0.012322585098445415 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.13735247174749904) | > loader_time: 0.0048 (0.004713827111152504)  --> TIME: 2025-02-26 16:05:09 -- STEP: 16070/36938 -- GLOBAL_STEP: 1308900 | > loss_text_ce: 0.04211564362049103 (0.031085745781525627) | > loss_mel_ce: 2.647024393081665 (nan) | > loss: 0.010671191848814487 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1373352588595401) | > loader_time: 0.0045 (0.004714086891036614)  --> TIME: 2025-02-26 16:05:18 -- STEP: 16120/36938 -- GLOBAL_STEP: 1308950 | > loss_text_ce: 0.03215615451335907 (0.031084445595080334) | > loss_mel_ce: 2.643143653869629 (nan) | > loss: 0.010616269893944263 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13732347044696233) | > loader_time: 0.0053 (0.004713952008902857)  --> TIME: 2025-02-26 16:05:27 -- STEP: 16170/36938 -- GLOBAL_STEP: 1309000 | > loss_text_ce: 0.03046511858701706 (0.03108710522844641) | > loss_mel_ce: 3.1104462146759033 (nan) | > loss: 0.01246393471956253 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.1373414125719696) | > loader_time: 0.0053 (0.004714317917455158)  --> TIME: 2025-02-26 16:05:37 -- STEP: 16220/36938 -- GLOBAL_STEP: 1309050 | > loss_text_ce: 0.03134583309292793 (0.03108772382188391) | > loss_mel_ce: 2.9107582569122314 (nan) | > loss: 0.011675016954541206 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.13735962791595688) | > loader_time: 0.0049 (0.004714389529386939)  --> TIME: 2025-02-26 16:05:46 -- STEP: 16270/36938 -- GLOBAL_STEP: 1309100 | > loss_text_ce: 0.03375236690044403 (0.031090704127848513) | > loss_mel_ce: 2.9218993186950684 (nan) | > loss: 0.011728777550160885 (nan) | > current_lr: 5e-06 | > step_time: 0.1262 (0.13735321312501617) | > loader_time: 0.0046 (0.0047144219123591595)  --> TIME: 2025-02-26 16:05:56 -- STEP: 16320/36938 -- GLOBAL_STEP: 1309150 | > loss_text_ce: 0.02693898230791092 (0.03109358975072574) | > loss_mel_ce: 2.490511894226074 (nan) | > loss: 0.009989884681999683 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13736545623225271) | > loader_time: 0.0046 (0.004714315443062294)  --> TIME: 2025-02-26 16:06:05 -- STEP: 16370/36938 -- GLOBAL_STEP: 1309200 | > loss_text_ce: 0.03239595517516136 (0.03109336226083971) | > loss_mel_ce: 2.6963484287261963 (nan) | > loss: 0.0108283506706357 (nan) | > current_lr: 5e-06 | > step_time: 0.1846 (0.13736428671226844) | > loader_time: 0.0058 (0.00471440834029096)  --> TIME: 2025-02-26 16:06:15 -- STEP: 16420/36938 -- GLOBAL_STEP: 1309250 | > loss_text_ce: 0.03951501473784447 (0.03109392361084723) | > loss_mel_ce: 3.0390703678131104 (nan) | > loss: 0.012216608971357346 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.1373588006185696) | > loader_time: 0.0047 (0.004714865341836999)  --> TIME: 2025-02-26 16:06:24 -- STEP: 16470/36938 -- GLOBAL_STEP: 1309300 | > loss_text_ce: 0.02932872250676155 (0.03109490420741697) | > loss_mel_ce: 2.4627413749694824 (nan) | > loss: 0.009889167733490467 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13735673126482378) | > loader_time: 0.0046 (0.00471494278910669)  --> TIME: 2025-02-26 16:06:33 -- STEP: 16520/36938 -- GLOBAL_STEP: 1309350 | > loss_text_ce: 0.027085181325674057 (0.031093185929441713) | > loss_mel_ce: 2.4270904064178467 (nan) | > loss: 0.009738791733980179 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13734004660610952) | > loader_time: 0.0052 (0.004715020301555585)  --> TIME: 2025-02-26 16:06:42 -- STEP: 16570/36938 -- GLOBAL_STEP: 1309400 | > loss_text_ce: 0.04813771694898605 (0.031089548971650106) | > loss_mel_ce: 2.27646803855896 (nan) | > loss: 0.00922462623566389 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1373488186063284) | > loader_time: 0.0045 (0.004714858409649052)  --> TIME: 2025-02-26 16:06:51 -- STEP: 16620/36938 -- GLOBAL_STEP: 1309450 | > loss_text_ce: 0.02794894203543663 (0.031087858463658826) | > loss_mel_ce: 1.0530065298080444 (nan) | > loss: 0.0042895060032606125 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13733270363400224) | > loader_time: 0.0045 (0.0047145809794447565)  --> TIME: 2025-02-26 16:07:01 -- STEP: 16670/36938 -- GLOBAL_STEP: 1309500 | > loss_text_ce: 0.025306735187768936 (0.031089063788790436) | > loss_mel_ce: 2.54404354095459 (nan) | > loss: 0.010195834562182426 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1373442531132593) | > loader_time: 0.0045 (0.0047145646849862605)  --> TIME: 2025-02-26 16:07:10 -- STEP: 16720/36938 -- GLOBAL_STEP: 1309550 | > loss_text_ce: 0.03241851553320885 (0.03108969561239865) | > loss_mel_ce: 2.955491065979004 (nan) | > loss: 0.011856785044074059 (nan) | > current_lr: 5e-06 | > step_time: 0.1877 (0.1373492954735546) | > loader_time: 0.0048 (0.00471447553646049)  --> TIME: 2025-02-26 16:07:20 -- STEP: 16770/36938 -- GLOBAL_STEP: 1309600 | > loss_text_ce: 0.03766609728336334 (0.031087705228015666) | > loss_mel_ce: 1.8590359687805176 (nan) | > loss: 0.007526596076786518 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.1373404158817985) | > loader_time: 0.0045 (0.004714174645957856)  --> TIME: 2025-02-26 16:07:29 -- STEP: 16820/36938 -- GLOBAL_STEP: 1309650 | > loss_text_ce: 0.03735475242137909 (0.031086285619229062) | > loss_mel_ce: 2.6882081031799316 (nan) | > loss: 0.010815725661814213 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.13733770295924394) | > loader_time: 0.005 (0.004713886260419316)  --> TIME: 2025-02-26 16:07:38 -- STEP: 16870/36938 -- GLOBAL_STEP: 1309700 | > loss_text_ce: 0.030968591570854187 (0.03108479308575021) | > loss_mel_ce: 2.9469783306121826 (nan) | > loss: 0.011817250400781631 (nan) | > current_lr: 5e-06 | > step_time: 0.1435 (0.13734520846877313) | > loader_time: 0.0046 (0.004714016399937525)  --> TIME: 2025-02-26 16:07:47 -- STEP: 16920/36938 -- GLOBAL_STEP: 1309750 | > loss_text_ce: 0.037637703120708466 (0.031084659143436884) | > loss_mel_ce: 2.4817287921905518 (nan) | > loss: 0.00999748706817627 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.1373184643175017) | > loader_time: 0.0048 (0.004713961630002804)  --> TIME: 2025-02-26 16:07:57 -- STEP: 16970/36938 -- GLOBAL_STEP: 1309800 | > loss_text_ce: 0.03598200902342796 (0.031083292331882412) | > loss_mel_ce: 2.6778452396392822 (nan) | > loss: 0.01076915580779314 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13732013490246367) | > loader_time: 0.0047 (0.0047137617011174)  --> TIME: 2025-02-26 16:08:07 -- STEP: 17020/36938 -- GLOBAL_STEP: 1309850 | > loss_text_ce: 0.02456018701195717 (0.03108449173909552) | > loss_mel_ce: 2.9181504249572754 (nan) | > loss: 0.011677423492074013 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.13734449616050015) | > loader_time: 0.0047 (0.004713665570551562)  --> TIME: 2025-02-26 16:08:16 -- STEP: 17070/36938 -- GLOBAL_STEP: 1309900 | > loss_text_ce: 0.028931433334946632 (0.031083223204154086) | > loss_mel_ce: 1.909256935119629 (nan) | > loss: 0.007691224105656147 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13733113440562184) | > loader_time: 0.0046 (0.004714012313601399)  --> TIME: 2025-02-26 16:08:25 -- STEP: 17120/36938 -- GLOBAL_STEP: 1309950 | > loss_text_ce: 0.04219239205121994 (0.03107816712162675) | > loss_mel_ce: 1.9814707040786743 (nan) | > loss: 0.00803040899336338 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13733555862558192) | > loader_time: 0.0048 (0.004714359134157105)  --> TIME: 2025-02-26 16:08:35 -- STEP: 17170/36938 -- GLOBAL_STEP: 1310000 | > loss_text_ce: 0.022425413131713867 (0.031077039479042078) | > loss_mel_ce: 2.4166243076324463 (nan) | > loss: 0.009678768925368786 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13733644010580442) | > loader_time: 0.0047 (0.004714669789695829) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1310000.pth  --> TIME: 2025-02-26 16:08:50 -- STEP: 17220/36938 -- GLOBAL_STEP: 1310050 | > loss_text_ce: 0.03390974551439285 (0.031080628744122096) | > loss_mel_ce: 3.2101263999938965 (nan) | > loss: 0.012873160652816296 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.1373379091352532) | > loader_time: 0.0046 (0.004715019249888363)  --> TIME: 2025-02-26 16:08:59 -- STEP: 17270/36938 -- GLOBAL_STEP: 1310100 | > loss_text_ce: 0.032961953431367874 (0.03107873582058453) | > loss_mel_ce: 3.2813045978546143 (nan) | > loss: 0.01315185148268938 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.13732534385376288) | > loader_time: 0.005 (0.004714711636011468)  --> TIME: 2025-02-26 16:09:08 -- STEP: 17320/36938 -- GLOBAL_STEP: 1310150 | > loss_text_ce: 0.037512924522161484 (0.031078368308308683) | > loss_mel_ce: 1.3537914752960205 (nan) | > loss: 0.005521049257367849 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13732626824951571) | > loader_time: 0.0054 (0.004715055667775858)  --> TIME: 2025-02-26 16:09:18 -- STEP: 17370/36938 -- GLOBAL_STEP: 1310200 | > loss_text_ce: 0.02148643136024475 (0.031075401977079916) | > loss_mel_ce: 1.6707215309143066 (nan) | > loss: 0.006715111434459686 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13731889915521156) | > loader_time: 0.0047 (0.0047154390063859335)  --> TIME: 2025-02-26 16:09:27 -- STEP: 17420/36938 -- GLOBAL_STEP: 1310250 | > loss_text_ce: 0.027579164132475853 (0.031078092587671226) | > loss_mel_ce: 2.4477956295013428 (nan) | > loss: 0.009822916239500046 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13732324874223115) | > loader_time: 0.0046 (0.004715759089017685)  --> TIME: 2025-02-26 16:09:37 -- STEP: 17470/36938 -- GLOBAL_STEP: 1310300 | > loss_text_ce: 0.0351068340241909 (0.03107368325149298) | > loss_mel_ce: 3.0013954639434814 (nan) | > loss: 0.012049613520503044 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13732556823191075) | > loader_time: 0.0047 (0.004715950746978404)  --> TIME: 2025-02-26 16:09:46 -- STEP: 17520/36938 -- GLOBAL_STEP: 1310350 | > loss_text_ce: 0.03957492113113403 (0.031075984221983054) | > loss_mel_ce: 2.7001771926879883 (nan) | > loss: 0.01087203249335289 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1373240420671353) | > loader_time: 0.0046 (0.004716023435331348)  --> TIME: 2025-02-26 16:09:55 -- STEP: 17570/36938 -- GLOBAL_STEP: 1310400 | > loss_text_ce: 0.03481800854206085 (0.031075937900051906) | > loss_mel_ce: 3.181523323059082 (nan) | > loss: 0.012763259932398796 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.1373283292599946) | > loader_time: 0.005 (0.004716103878900529)  --> TIME: 2025-02-26 16:10:04 -- STEP: 17620/36938 -- GLOBAL_STEP: 1310450 | > loss_text_ce: 0.02401980571448803 (0.03107567477393071) | > loss_mel_ce: 2.8535006046295166 (nan) | > loss: 0.01141873188316822 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13731843793297424) | > loader_time: 0.0046 (0.004716112556847192)  --> TIME: 2025-02-26 16:10:14 -- STEP: 17670/36938 -- GLOBAL_STEP: 1310500 | > loss_text_ce: 0.023442696779966354 (0.03107256385148305) | > loss_mel_ce: 2.2834625244140625 (nan) | > loss: 0.009154386818408966 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13733036929628675) | > loader_time: 0.0046 (0.0047164107421460285)  --> TIME: 2025-02-26 16:10:23 -- STEP: 17720/36938 -- GLOBAL_STEP: 1310550 | > loss_text_ce: 0.02964889071881771 (0.031070136786837965) | > loss_mel_ce: 3.3251190185546875 (nan) | > loss: 0.013312571682035923 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.13732548327801153) | > loader_time: 0.0049 (0.004716584106600288)  --> TIME: 2025-02-26 16:10:33 -- STEP: 17770/36938 -- GLOBAL_STEP: 1310600 | > loss_text_ce: 0.0328228622674942 (0.031065627080731652) | > loss_mel_ce: 2.7748231887817383 (nan) | > loss: 0.011141452938318253 (nan) | > current_lr: 5e-06 | > step_time: 0.1511 (0.13732174053310173) | > loader_time: 0.0047 (0.004716628833506684)  --> TIME: 2025-02-26 16:10:42 -- STEP: 17820/36938 -- GLOBAL_STEP: 1310650 | > loss_text_ce: 0.025775428861379623 (0.03106666177027262) | > loss_mel_ce: 2.37703537940979 (nan) | > loss: 0.009534964337944984 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13731234968712) | > loader_time: 0.0045 (0.004716455494915975)  --> TIME: 2025-02-26 16:10:52 -- STEP: 17870/36938 -- GLOBAL_STEP: 1310700 | > loss_text_ce: 0.033679742366075516 (0.031065676131227876) | > loss_mel_ce: 2.5372071266174316 (nan) | > loss: 0.010201932862401009 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13733686788427646) | > loader_time: 0.0045 (0.004716320203280515)  --> TIME: 2025-02-26 16:11:01 -- STEP: 17920/36938 -- GLOBAL_STEP: 1310750 | > loss_text_ce: 0.023244647309184074 (0.031066623267906834) | > loss_mel_ce: 2.7611653804779053 (nan) | > loss: 0.011049246415495872 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.13735007627734086) | > loader_time: 0.0047 (0.004716361061270721)  --> TIME: 2025-02-26 16:11:11 -- STEP: 17970/36938 -- GLOBAL_STEP: 1310800 | > loss_text_ce: 0.03075955994427204 (0.031065746896957763) | > loss_mel_ce: 3.16829776763916 (nan) | > loss: 0.01269467268139124 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13735571287045797) | > loader_time: 0.0053 (0.0047167076823043785)  --> TIME: 2025-02-26 16:11:20 -- STEP: 18020/36938 -- GLOBAL_STEP: 1310850 | > loss_text_ce: 0.023862147703766823 (0.03106498255541178) | > loss_mel_ce: 2.051826238632202 (nan) | > loss: 0.008236858993768692 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13734240341398182) | > loader_time: 0.0048 (0.004716840912313996)  --> TIME: 2025-02-26 16:11:29 -- STEP: 18070/36938 -- GLOBAL_STEP: 1310900 | > loss_text_ce: 0.024548355489969254 (0.031065531626939525) | > loss_mel_ce: 2.232872247695923 (nan) | > loss: 0.008958018384873867 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.137351343403485) | > loader_time: 0.0045 (0.004716843112635202)  --> TIME: 2025-02-26 16:11:39 -- STEP: 18120/36938 -- GLOBAL_STEP: 1310950 | > loss_text_ce: 0.028247497975826263 (0.031065844149415114) | > loss_mel_ce: 3.466933250427246 (nan) | > loss: 0.013869766145944595 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13734480543904978) | > loader_time: 0.0046 (0.004716672092084031)  --> TIME: 2025-02-26 16:11:48 -- STEP: 18170/36938 -- GLOBAL_STEP: 1311000 | > loss_text_ce: 0.028466884046792984 (0.031066815189793748) | > loss_mel_ce: 2.3642964363098145 (nan) | > loss: 0.009495093487203121 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.1373428668172788) | > loader_time: 0.0048 (0.004716725367796409)  --> TIME: 2025-02-26 16:11:57 -- STEP: 18220/36938 -- GLOBAL_STEP: 1311050 | > loss_text_ce: 0.03373342752456665 (0.031068424422459516) | > loss_mel_ce: 2.0319972038269043 (nan) | > loss: 0.008197343908250332 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13733808646217757) | > loader_time: 0.0045 (0.004717013877780687)  --> TIME: 2025-02-26 16:12:07 -- STEP: 18270/36938 -- GLOBAL_STEP: 1311100 | > loss_text_ce: 0.04007367044687271 (0.031067113941723963) | > loss_mel_ce: 2.123781681060791 (nan) | > loss: 0.008586728014051914 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13733794727022652) | > loader_time: 0.0046 (0.004717005793357949)  --> TIME: 2025-02-26 16:12:16 -- STEP: 18320/36938 -- GLOBAL_STEP: 1311150 | > loss_text_ce: 0.041349105536937714 (0.03106532131050262) | > loss_mel_ce: 2.454308032989502 (nan) | > loss: 0.00990340206772089 (nan) | > current_lr: 5e-06 | > step_time: 0.1577 (0.13733215395800383) | > loader_time: 0.0047 (0.004716864371403826)  --> TIME: 2025-02-26 16:12:25 -- STEP: 18370/36938 -- GLOBAL_STEP: 1311200 | > loss_text_ce: 0.026085171848535538 (0.031064255676644455) | > loss_mel_ce: 2.926974296569824 (nan) | > loss: 0.011718490161001682 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.13733323220007404) | > loader_time: 0.0047 (0.0047166692087944615)  --> TIME: 2025-02-26 16:12:35 -- STEP: 18420/36938 -- GLOBAL_STEP: 1311250 | > loss_text_ce: 0.022983405739068985 (0.031062057422095835) | > loss_mel_ce: 2.7208900451660156 (nan) | > loss: 0.010888386517763138 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13733683543666025) | > loader_time: 0.0053 (0.004716791754566208)  --> TIME: 2025-02-26 16:12:44 -- STEP: 18470/36938 -- GLOBAL_STEP: 1311300 | > loss_text_ce: 0.02389039658010006 (0.03106468766836957) | > loss_mel_ce: 2.4773173332214355 (nan) | > loss: 0.00992542877793312 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13733648938879672) | > loader_time: 0.0048 (0.004716629922615851)  --> TIME: 2025-02-26 16:12:54 -- STEP: 18520/36938 -- GLOBAL_STEP: 1311350 | > loss_text_ce: 0.030419273301959038 (0.031063111938162028) | > loss_mel_ce: 2.92345929145813 (nan) | > loss: 0.011721741408109665 (nan) | > current_lr: 5e-06 | > step_time: 0.1817 (0.1373539534938514) | > loader_time: 0.0048 (0.004716518926826431)  --> TIME: 2025-02-26 16:13:03 -- STEP: 18570/36938 -- GLOBAL_STEP: 1311400 | > loss_text_ce: 0.027784502133727074 (0.0310649444735782) | > loss_mel_ce: 2.717825174331665 (nan) | > loss: 0.010895277373492718 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13735875853557497) | > loader_time: 0.0046 (0.004716533079029191)  --> TIME: 2025-02-26 16:13:12 -- STEP: 18620/36938 -- GLOBAL_STEP: 1311450 | > loss_text_ce: 0.029218409210443497 (0.031064234694283932) | > loss_mel_ce: 0.092041015625 (nan) | > loss: 0.00048118820996023715 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13735924927171633) | > loader_time: 0.0055 (0.004716798813591424)  --> TIME: 2025-02-26 16:13:22 -- STEP: 18670/36938 -- GLOBAL_STEP: 1311500 | > loss_text_ce: 0.02172064036130905 (0.03106346559298265) | > loss_mel_ce: 0.0189208984375 (nan) | > loss: 0.00016127595154102892 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.1373530002347783) | > loader_time: 0.0045 (0.0047168812652162604)  --> TIME: 2025-02-26 16:13:31 -- STEP: 18720/36938 -- GLOBAL_STEP: 1311550 | > loss_text_ce: 0.031546685844659805 (0.03106217762727975) | > loss_mel_ce: 2.6753151416778564 (nan) | > loss: 0.010741515085101128 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.1373476689315247) | > loader_time: 0.0047 (0.004716860381965935)  --> TIME: 2025-02-26 16:13:41 -- STEP: 18770/36938 -- GLOBAL_STEP: 1311600 | > loss_text_ce: 0.03324244171380997 (0.031060414713969652) | > loss_mel_ce: 2.1182312965393066 (nan) | > loss: 0.0085375951603055 (nan) | > current_lr: 5e-06 | > step_time: 0.1766 (0.13735579252878122) | > loader_time: 0.0048 (0.004716915721009214)  --> TIME: 2025-02-26 16:13:50 -- STEP: 18820/36938 -- GLOBAL_STEP: 1311650 | > loss_text_ce: 0.027783628553152084 (0.03106129782108093) | > loss_mel_ce: 3.178419828414917 (nan) | > loss: 0.012723030522465706 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.13734737984305578) | > loader_time: 0.0048 (0.0047170658293996396)  --> TIME: 2025-02-26 16:13:59 -- STEP: 18870/36938 -- GLOBAL_STEP: 1311700 | > loss_text_ce: 0.02670665644109249 (0.031060080504006393) | > loss_mel_ce: 2.8329954147338867 (nan) | > loss: 0.011348024941980839 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13734309845142503) | > loader_time: 0.0046 (0.004716964569051339)  --> TIME: 2025-02-26 16:14:08 -- STEP: 18920/36938 -- GLOBAL_STEP: 1311750 | > loss_text_ce: 0.02607053518295288 (0.03105577457337847) | > loss_mel_ce: 2.496727466583252 (nan) | > loss: 0.010011103935539722 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13733872512934295) | > loader_time: 0.0047 (0.004716881914320224)  --> TIME: 2025-02-26 16:14:17 -- STEP: 18970/36938 -- GLOBAL_STEP: 1311800 | > loss_text_ce: 0.025883078575134277 (0.031054945536935774) | > loss_mel_ce: 2.7526819705963135 (nan) | > loss: 0.011026051826775074 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.13732328582073902) | > loader_time: 0.0047 (0.004716615696988957)  --> TIME: 2025-02-26 16:14:27 -- STEP: 19020/36938 -- GLOBAL_STEP: 1311850 | > loss_text_ce: 0.027615202590823174 (0.031053634458461425) | > loss_mel_ce: 2.9357008934020996 (nan) | > loss: 0.011759191751480103 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13733306861199515) | > loader_time: 0.0047 (0.00471681891681014)  --> TIME: 2025-02-26 16:14:36 -- STEP: 19070/36938 -- GLOBAL_STEP: 1311900 | > loss_text_ce: 0.02995823509991169 (0.031054287048532393) | > loss_mel_ce: 2.6312475204467773 (nan) | > loss: 0.01056034117937088 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.13732402510960548) | > loader_time: 0.0051 (0.0047167828649391525)  --> TIME: 2025-02-26 16:14:45 -- STEP: 19120/36938 -- GLOBAL_STEP: 1311950 | > loss_text_ce: 0.020647330209612846 (0.03105126878721094) | > loss_mel_ce: 1.725568175315857 (nan) | > loss: 0.006929426919668913 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13731571063087564) | > loader_time: 0.0046 (0.004716713021988617)  --> TIME: 2025-02-26 16:14:54 -- STEP: 19170/36938 -- GLOBAL_STEP: 1312000 | > loss_text_ce: 0.03699873760342598 (0.031052849549910926) | > loss_mel_ce: 2.9059643745422363 (nan) | > loss: 0.011678425595164299 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.13731320306044664) | > loader_time: 0.0049 (0.004716639774941348)  --> TIME: 2025-02-26 16:15:04 -- STEP: 19220/36938 -- GLOBAL_STEP: 1312050 | > loss_text_ce: 0.026635123416781425 (0.03105309254767976) | > loss_mel_ce: 2.7208354473114014 (nan) | > loss: 0.010902661830186844 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13731400799180682) | > loader_time: 0.0046 (0.004716570320288145)  --> TIME: 2025-02-26 16:15:13 -- STEP: 19270/36938 -- GLOBAL_STEP: 1312100 | > loss_text_ce: 0.02915055863559246 (0.031050734801704116) | > loss_mel_ce: 2.345663547515869 (nan) | > loss: 0.00942386593669653 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.1373214733557315) | > loader_time: 0.0046 (0.004716376783195747)  --> TIME: 2025-02-26 16:15:22 -- STEP: 19320/36938 -- GLOBAL_STEP: 1312150 | > loss_text_ce: 0.025447716936469078 (0.031054921250879056) | > loss_mel_ce: 2.1254260540008545 (nan) | > loss: 0.008535213768482208 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1373101922666067) | > loader_time: 0.0052 (0.004716457664102721)  --> TIME: 2025-02-26 16:15:32 -- STEP: 19370/36938 -- GLOBAL_STEP: 1312200 | > loss_text_ce: 0.034423112869262695 (0.031053353885018495) | > loss_mel_ce: 3.055415153503418 (nan) | > loss: 0.012261264026165009 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.13730536789763667) | > loader_time: 0.0046 (0.004716184290648094)  --> TIME: 2025-02-26 16:15:41 -- STEP: 19420/36938 -- GLOBAL_STEP: 1312250 | > loss_text_ce: 0.02519037202000618 (0.031052082159146895) | > loss_mel_ce: 1.4097769260406494 (nan) | > loss: 0.0056943148374557495 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13730625298438215) | > loader_time: 0.0046 (0.004716092243744562)  --> TIME: 2025-02-26 16:15:51 -- STEP: 19470/36938 -- GLOBAL_STEP: 1312300 | > loss_text_ce: 0.03397674486041069 (0.03105209299421075) | > loss_mel_ce: 3.02075457572937 (nan) | > loss: 0.012121950276196003 (nan) | > current_lr: 5e-06 | > step_time: 0.1855 (0.13731416236699184) | > loader_time: 0.0048 (0.004716217083139888)  --> TIME: 2025-02-26 16:16:00 -- STEP: 19520/36938 -- GLOBAL_STEP: 1312350 | > loss_text_ce: 0.01401100680232048 (0.03105213169645915) | > loss_mel_ce: 0.870776355266571 (nan) | > loss: 0.003511061193421483 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.1373072420598057) | > loader_time: 0.0047 (0.004715999472336662)  --> TIME: 2025-02-26 16:16:09 -- STEP: 19570/36938 -- GLOBAL_STEP: 1312400 | > loss_text_ce: 0.03339529410004616 (0.031050874470358578) | > loss_mel_ce: 2.723289966583252 (nan) | > loss: 0.010939227417111397 (nan) | > current_lr: 5e-06 | > step_time: 0.188 (0.13729903600504678) | > loader_time: 0.0048 (0.004715905837642308)  --> TIME: 2025-02-26 16:16:18 -- STEP: 19620/36938 -- GLOBAL_STEP: 1312450 | > loss_text_ce: 0.025812363252043724 (0.031050448311221283) | > loss_mel_ce: 2.2779035568237305 (nan) | > loss: 0.009141730144619942 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13728066453146312) | > loader_time: 0.0046 (0.004715705659168301)  --> TIME: 2025-02-26 16:16:27 -- STEP: 19670/36938 -- GLOBAL_STEP: 1312500 | > loss_text_ce: 0.029975559562444687 (0.031048148942862255) | > loss_mel_ce: 3.1225032806396484 (nan) | > loss: 0.012509837746620178 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.13727634608836525) | > loader_time: 0.0048 (0.004715688118306737)  --> TIME: 2025-02-26 16:16:36 -- STEP: 19720/36938 -- GLOBAL_STEP: 1312550 | > loss_text_ce: 0.030203158035874367 (0.03104801905916675) | > loss_mel_ce: 0.00667572021484375 (nan) | > loss: 0.0001463447551941499 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13727067575977053) | > loader_time: 0.0045 (0.004715596553520034)  --> TIME: 2025-02-26 16:16:46 -- STEP: 19770/36938 -- GLOBAL_STEP: 1312600 | > loss_text_ce: 0.028837434947490692 (0.031047240744420734) | > loss_mel_ce: 3.1356513500213623 (nan) | > loss: 0.012557496316730976 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.1372718966362996) | > loader_time: 0.0053 (0.0047157966915741985)  --> TIME: 2025-02-26 16:16:55 -- STEP: 19820/36938 -- GLOBAL_STEP: 1312650 | > loss_text_ce: 0.0339786633849144 (0.03104703150530779) | > loss_mel_ce: 1.1761471033096313 (nan) | > loss: 0.004802086856216192 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13726524231291967) | > loader_time: 0.0046 (0.004715615204196135)  --> TIME: 2025-02-26 16:17:04 -- STEP: 19870/36938 -- GLOBAL_STEP: 1312700 | > loss_text_ce: 0.02710374817252159 (0.031045901337352584) | > loss_mel_ce: 2.1561450958251953 (nan) | > loss: 0.008663685992360115 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13724738308455922) | > loader_time: 0.0045 (0.004715507151676148)  --> TIME: 2025-02-26 16:17:14 -- STEP: 19920/36938 -- GLOBAL_STEP: 1312750 | > loss_text_ce: 0.027995631098747253 (0.031045166599465136) | > loss_mel_ce: 2.3631439208984375 (nan) | > loss: 0.009488649666309357 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13727675530326464) | > loader_time: 0.0053 (0.00471605650153024)  --> TIME: 2025-02-26 16:17:23 -- STEP: 19970/36938 -- GLOBAL_STEP: 1312800 | > loss_text_ce: 0.024001793935894966 (0.03104636251966529) | > loss_mel_ce: 1.894320011138916 (nan) | > loss: 0.007612388581037521 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.1372868693824051) | > loader_time: 0.0048 (0.0047163683352138275)  --> TIME: 2025-02-26 16:17:33 -- STEP: 20020/36938 -- GLOBAL_STEP: 1312850 | > loss_text_ce: 0.028737105429172516 (0.031043201790876933) | > loss_mel_ce: 2.964350938796997 (nan) | > loss: 0.011877333745360374 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1372844006869942) | > loader_time: 0.0045 (0.004716320876236779)  --> TIME: 2025-02-26 16:17:42 -- STEP: 20070/36938 -- GLOBAL_STEP: 1312900 | > loss_text_ce: 0.022449765354394913 (0.031037780029005934) | > loss_mel_ce: 2.07496976852417 (nan) | > loss: 0.008323093876242638 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13729082973786277) | > loader_time: 0.0046 (0.004716161595806866)  --> TIME: 2025-02-26 16:17:51 -- STEP: 20120/36938 -- GLOBAL_STEP: 1312950 | > loss_text_ce: 0.03414090722799301 (0.031038643365585157) | > loss_mel_ce: 2.4523487091064453 (nan) | > loss: 0.00986702274531126 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13728185605339255) | > loader_time: 0.0046 (0.004716151573548949)  --> TIME: 2025-02-26 16:18:00 -- STEP: 20170/36938 -- GLOBAL_STEP: 1313000 | > loss_text_ce: 0.025503840297460556 (0.031036277568954964) | > loss_mel_ce: 2.7087857723236084 (nan) | > loss: 0.010850355960428715 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13726960683613704) | > loader_time: 0.0045 (0.00471608030009989)  --> TIME: 2025-02-26 16:18:10 -- STEP: 20220/36938 -- GLOBAL_STEP: 1313050 | > loss_text_ce: 0.025580279529094696 (0.031035328462156148) | > loss_mel_ce: 2.432709217071533 (nan) | > loss: 0.009755116887390614 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13727255900465116) | > loader_time: 0.0052 (0.00471620986534743)  --> TIME: 2025-02-26 16:18:19 -- STEP: 20270/36938 -- GLOBAL_STEP: 1313100 | > loss_text_ce: 0.029156876727938652 (0.0310346250681893) | > loss_mel_ce: 2.4494669437408447 (nan) | > loss: 0.009835809469223022 (nan) | > current_lr: 5e-06 | > step_time: 0.1874 (0.13727886549757276) | > loader_time: 0.0078 (0.004716481548491857)  --> TIME: 2025-02-26 16:18:29 -- STEP: 20320/36938 -- GLOBAL_STEP: 1313150 | > loss_text_ce: 0.027699168771505356 (0.031033676853366783) | > loss_mel_ce: 3.018383264541626 (nan) | > loss: 0.012087630107998848 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.1372831680760614) | > loader_time: 0.0046 (0.00471633926151304)  --> TIME: 2025-02-26 16:18:38 -- STEP: 20370/36938 -- GLOBAL_STEP: 1313200 | > loss_text_ce: 0.030478715896606445 (0.031030943729422316) | > loss_mel_ce: 2.6277668476104736 (nan) | > loss: 0.010548594407737255 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13728594365632635) | > loader_time: 0.0047 (0.004716217710975327)  --> TIME: 2025-02-26 16:18:48 -- STEP: 20420/36938 -- GLOBAL_STEP: 1313250 | > loss_text_ce: 0.03428658843040466 (0.031029241127221975) | > loss_mel_ce: 2.7716641426086426 (nan) | > loss: 0.01113472506403923 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.1372876377227145) | > loader_time: 0.0049 (0.004716143411940841)  --> TIME: 2025-02-26 16:18:57 -- STEP: 20470/36938 -- GLOBAL_STEP: 1313300 | > loss_text_ce: 0.03384855017066002 (0.031031795790711074) | > loss_mel_ce: 2.317626953125 (nan) | > loss: 0.009331252425909042 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1372921345463372) | > loader_time: 0.0046 (0.004716034429388957)  --> TIME: 2025-02-26 16:19:07 -- STEP: 20520/36938 -- GLOBAL_STEP: 1313350 | > loss_text_ce: 0.026949048042297363 (0.031031473792161698) | > loss_mel_ce: 2.749743700027466 (nan) | > loss: 0.011018622666597366 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13729925657573538) | > loader_time: 0.0046 (0.004715989591085407)  --> TIME: 2025-02-26 16:19:16 -- STEP: 20570/36938 -- GLOBAL_STEP: 1313400 | > loss_text_ce: 0.02900376170873642 (0.031031928389882384) | > loss_mel_ce: 2.8286333084106445 (nan) | > loss: 0.011339830234646797 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1373023301812129) | > loader_time: 0.0045 (0.004715969948497377)  --> TIME: 2025-02-26 16:19:25 -- STEP: 20620/36938 -- GLOBAL_STEP: 1313450 | > loss_text_ce: 0.02306666597723961 (0.03103418474566195) | > loss_mel_ce: 2.2002017498016357 (nan) | > loss: 0.008822494186460972 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1373030839905245) | > loader_time: 0.0047 (0.004715992881763783)  --> TIME: 2025-02-26 16:19:35 -- STEP: 20670/36938 -- GLOBAL_STEP: 1313500 | > loss_text_ce: 0.033263321965932846 (0.031034823468456926) | > loss_mel_ce: 2.9815075397491455 (nan) | > loss: 0.01196337677538395 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.13730034076135175) | > loader_time: 0.0049 (0.004716063791504774)  --> TIME: 2025-02-26 16:19:44 -- STEP: 20720/36938 -- GLOBAL_STEP: 1313550 | > loss_text_ce: 0.019183609634637833 (0.03103461848714831) | > loss_mel_ce: 2.603438377380371 (nan) | > loss: 0.01040723081678152 (nan) | > current_lr: 5e-06 | > step_time: 0.1447 (0.13730195944833945) | > loader_time: 0.0053 (0.0047161293996346755)  --> TIME: 2025-02-26 16:19:53 -- STEP: 20770/36938 -- GLOBAL_STEP: 1313600 | > loss_text_ce: 0.039121534675359726 (0.031035469245653445) | > loss_mel_ce: 1.7101916074752808 (nan) | > loss: 0.006941718980669975 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13729637392335184) | > loader_time: 0.0045 (0.004716104776972906)  --> TIME: 2025-02-26 16:20:03 -- STEP: 20820/36938 -- GLOBAL_STEP: 1313650 | > loss_text_ce: 0.03137649595737457 (0.03103323417866028) | > loss_mel_ce: 2.67984676361084 (nan) | > loss: 0.010758823715150356 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13730040644362618) | > loader_time: 0.0047 (0.004716154649553086)  --> TIME: 2025-02-26 16:20:12 -- STEP: 20870/36938 -- GLOBAL_STEP: 1313700 | > loss_text_ce: 0.028838032856583595 (0.031032059420521358) | > loss_mel_ce: 1.157202959060669 (nan) | > loss: 0.004706512205302715 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1373019288131137) | > loader_time: 0.0047 (0.004716044118887604)  --> TIME: 2025-02-26 16:20:22 -- STEP: 20920/36938 -- GLOBAL_STEP: 1313750 | > loss_text_ce: 0.030940579250454903 (0.031030905858104262) | > loss_mel_ce: 2.317857265472412 (nan) | > loss: 0.009320626966655254 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.1373104146288185) | > loader_time: 0.0047 (0.004715910194936704)  --> TIME: 2025-02-26 16:20:31 -- STEP: 20970/36938 -- GLOBAL_STEP: 1313800 | > loss_text_ce: 0.028766082599759102 (0.031033471105402645) | > loss_mel_ce: 2.4588589668273926 (nan) | > loss: 0.00987152848392725 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13730552223789513) | > loader_time: 0.0046 (0.004715736684312565)  --> TIME: 2025-02-26 16:20:41 -- STEP: 21020/36938 -- GLOBAL_STEP: 1313850 | > loss_text_ce: 0.033084869384765625 (0.031030498023191865) | > loss_mel_ce: 1.0356476306915283 (nan) | > loss: 0.004241002257913351 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.13731394129407404) | > loader_time: 0.0045 (0.004715700210785424)  --> TIME: 2025-02-26 16:20:50 -- STEP: 21070/36938 -- GLOBAL_STEP: 1313900 | > loss_text_ce: 0.03605177253484726 (0.031028855405261676) | > loss_mel_ce: 2.93178391456604 (nan) | > loss: 0.011777126230299473 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.13731211108735908) | > loader_time: 0.0047 (0.004715698920667696)  --> TIME: 2025-02-26 16:21:00 -- STEP: 21120/36938 -- GLOBAL_STEP: 1313950 | > loss_text_ce: 0.034713488072156906 (0.031029645320350855) | > loss_mel_ce: 3.1700949668884277 (nan) | > loss: 0.012717494741082191 (nan) | > current_lr: 5e-06 | > step_time: 0.1859 (0.13732752109115756) | > loader_time: 0.0052 (0.004716094696160507)  --> TIME: 2025-02-26 16:21:09 -- STEP: 21170/36938 -- GLOBAL_STEP: 1314000 | > loss_text_ce: 0.03215599060058594 (0.031028610880780978) | > loss_mel_ce: 1.6942065954208374 (nan) | > loss: 0.006850645411759615 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13732547615854912) | > loader_time: 0.0054 (0.004716428338664838)  --> TIME: 2025-02-26 16:21:18 -- STEP: 21220/36938 -- GLOBAL_STEP: 1314050 | > loss_text_ce: 0.03285108134150505 (0.03102662650970119) | > loss_mel_ce: 0.004888391587883234 (nan) | > loss: 0.0001497598277637735 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13732213123467132) | > loader_time: 0.0045 (0.00471647925021847)  --> TIME: 2025-02-26 16:21:27 -- STEP: 21270/36938 -- GLOBAL_STEP: 1314100 | > loss_text_ce: 0.025139231234788895 (0.03102380670248047) | > loss_mel_ce: 1.3321638107299805 (nan) | > loss: 0.005386123433709145 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.1373077486847059) | > loader_time: 0.0046 (0.004716470592392949)  --> TIME: 2025-02-26 16:21:37 -- STEP: 21320/36938 -- GLOBAL_STEP: 1314150 | > loss_text_ce: 0.035700008273124695 (0.031023875137749103) | > loss_mel_ce: 2.776719808578491 (nan) | > loss: 0.011160396970808506 (nan) | > current_lr: 5e-06 | > step_time: 0.1713 (0.13731200759898424) | > loader_time: 0.0058 (0.004716543039729836)  --> TIME: 2025-02-26 16:21:46 -- STEP: 21370/36938 -- GLOBAL_STEP: 1314200 | > loss_text_ce: 0.033033501356840134 (0.03102331410912177) | > loss_mel_ce: 2.4150969982147217 (nan) | > loss: 0.009714804589748383 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.13731738302715468) | > loader_time: 0.0047 (0.004716743483518839)  --> TIME: 2025-02-26 16:21:56 -- STEP: 21420/36938 -- GLOBAL_STEP: 1314250 | > loss_text_ce: 0.0297608133405447 (0.031023096456257653) | > loss_mel_ce: 2.6173434257507324 (nan) | > loss: 0.010504382662475109 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1373154422704801) | > loader_time: 0.0046 (0.004716634204964221)  --> TIME: 2025-02-26 16:22:04 -- STEP: 21470/36938 -- GLOBAL_STEP: 1314300 | > loss_text_ce: 0.023188721388578415 (0.03102121375102947) | > loss_mel_ce: 2.2015018463134766 (nan) | > loss: 0.008828138001263142 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13729944382371242) | > loader_time: 0.0046 (0.00471657525121637)  --> TIME: 2025-02-26 16:22:14 -- STEP: 21520/36938 -- GLOBAL_STEP: 1314350 | > loss_text_ce: 0.032730408012866974 (0.031020510748208788) | > loss_mel_ce: 1.8017829656600952 (nan) | > loss: 0.0072798156179487705 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1372886240703906) | > loader_time: 0.0045 (0.004716325825474951)  --> TIME: 2025-02-26 16:22:23 -- STEP: 21570/36938 -- GLOBAL_STEP: 1314400 | > loss_text_ce: 0.029201261699199677 (0.031018748874683234) | > loss_mel_ce: 2.936640739440918 (nan) | > loss: 0.011769214645028114 (nan) | > current_lr: 5e-06 | > step_time: 0.1703 (0.13728037816703315) | > loader_time: 0.0048 (0.00471610947787125)  --> TIME: 2025-02-26 16:22:32 -- STEP: 21620/36938 -- GLOBAL_STEP: 1314450 | > loss_text_ce: 0.03445464372634888 (0.031017261810314726) | > loss_mel_ce: 2.924821138381958 (nan) | > loss: 0.011743158102035522 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.13727328169687675) | > loader_time: 0.0048 (0.004716083355020515)  --> TIME: 2025-02-26 16:22:41 -- STEP: 21670/36938 -- GLOBAL_STEP: 1314500 | > loss_text_ce: 0.037268321961164474 (0.031019772014683384) | > loss_mel_ce: 2.674532890319824 (nan) | > loss: 0.01076111663132906 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.1372693265810172) | > loader_time: 0.005 (0.0047162893349859064)  --> TIME: 2025-02-26 16:22:50 -- STEP: 21720/36938 -- GLOBAL_STEP: 1314550 | > loss_text_ce: 0.031900495290756226 (0.031016697634301368) | > loss_mel_ce: 2.7740514278411865 (nan) | > loss: 0.011134729720652103 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13726766169839555) | > loader_time: 0.0046 (0.004716159998821714)  --> TIME: 2025-02-26 16:23:00 -- STEP: 21770/36938 -- GLOBAL_STEP: 1314600 | > loss_text_ce: 0.031196052208542824 (0.031015680847859776) | > loss_mel_ce: 2.8345096111297607 (nan) | > loss: 0.011371849104762077 (nan) | > current_lr: 5e-06 | > step_time: 0.1846 (0.1372663294770706) | > loader_time: 0.0051 (0.004716065886045836)  --> TIME: 2025-02-26 16:23:09 -- STEP: 21820/36938 -- GLOBAL_STEP: 1314650 | > loss_text_ce: 0.03326480835676193 (0.031011348904437436) | > loss_mel_ce: 0.9982528686523438 (nan) | > loss: 0.004093324299901724 (nan) | > current_lr: 5e-06 | > step_time: 0.1894 (0.1372716503554147) | > loader_time: 0.0049 (0.00471597694673196)  --> TIME: 2025-02-26 16:23:19 -- STEP: 21870/36938 -- GLOBAL_STEP: 1314700 | > loss_text_ce: 0.03132249414920807 (0.03101208472426829) | > loss_mel_ce: 1.962086796760559 (nan) | > loss: 0.007910354994237423 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13727180212122866) | > loader_time: 0.0046 (0.0047159651179372415)  --> TIME: 2025-02-26 16:23:28 -- STEP: 21920/36938 -- GLOBAL_STEP: 1314750 | > loss_text_ce: 0.030842814594507217 (0.031014368433231677) | > loss_mel_ce: 2.7697064876556396 (nan) | > loss: 0.01111329160630703 (nan) | > current_lr: 5e-06 | > step_time: 0.1667 (0.1372710292900561) | > loader_time: 0.0053 (0.004716235508013806)  --> TIME: 2025-02-26 16:23:38 -- STEP: 21970/36938 -- GLOBAL_STEP: 1314800 | > loss_text_ce: 0.02997589483857155 (0.031013599539478298) | > loss_mel_ce: 2.709057569503784 (nan) | > loss: 0.010869180783629417 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.13728044036306092) | > loader_time: 0.0053 (0.004716623952619907)  --> TIME: 2025-02-26 16:23:47 -- STEP: 22020/36938 -- GLOBAL_STEP: 1314850 | > loss_text_ce: 0.028901895508170128 (0.031013303137190647) | > loss_mel_ce: 2.7475686073303223 (nan) | > loss: 0.011017740704119205 (nan) | > current_lr: 5e-06 | > step_time: 0.156 (0.1372775751173653) | > loader_time: 0.0047 (0.004716500880824341)  --> TIME: 2025-02-26 16:23:56 -- STEP: 22070/36938 -- GLOBAL_STEP: 1314900 | > loss_text_ce: 0.02586505375802517 (0.031014403538364033) | > loss_mel_ce: 0.08243051916360855 (nan) | > loss: 0.000429744366556406 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.137276586440552) | > loader_time: 0.0047 (0.004716321241277671)  --> TIME: 2025-02-26 16:24:05 -- STEP: 22120/36938 -- GLOBAL_STEP: 1314950 | > loss_text_ce: 0.027485519647598267 (0.031015367563948654) | > loss_mel_ce: 2.1014606952667236 (nan) | > loss: 0.008448200300335884 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13727891098094844) | > loader_time: 0.0045 (0.004716294615867992)  --> TIME: 2025-02-26 16:24:15 -- STEP: 22170/36938 -- GLOBAL_STEP: 1315000 | > loss_text_ce: 0.030977727845311165 (0.031016908234752377) | > loss_mel_ce: 3.2492995262145996 (nan) | > loss: 0.013016973622143269 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13727778590472853) | > loader_time: 0.0046 (0.004716242666334847)  --> TIME: 2025-02-26 16:24:24 -- STEP: 22220/36938 -- GLOBAL_STEP: 1315050 | > loss_text_ce: 0.03562120720744133 (0.031016868732795858) | > loss_mel_ce: 2.407785415649414 (nan) | > loss: 0.009696057997643948 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13728249489348981) | > loader_time: 0.0048 (0.004716342596402592)  --> TIME: 2025-02-26 16:24:34 -- STEP: 22270/36938 -- GLOBAL_STEP: 1315100 | > loss_text_ce: 0.027232669293880463 (0.03101434372332019) | > loss_mel_ce: 3.0564112663269043 (nan) | > loss: 0.012236682698130608 (nan) | > current_lr: 5e-06 | > step_time: 0.1821 (0.1372880083592842) | > loader_time: 0.0049 (0.004716445963961917)  --> TIME: 2025-02-26 16:24:43 -- STEP: 22320/36938 -- GLOBAL_STEP: 1315150 | > loss_text_ce: 0.031024204567074776 (0.031012249101741125) | > loss_mel_ce: 3.034130096435547 (nan) | > loss: 0.012163311243057251 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.13728697352298216) | > loader_time: 0.0048 (0.004716471969867708)  --> TIME: 2025-02-26 16:24:52 -- STEP: 22370/36938 -- GLOBAL_STEP: 1315200 | > loss_text_ce: 0.031145311892032623 (0.031013662201195464) | > loss_mel_ce: 3.2984113693237305 (nan) | > loss: 0.013212527148425579 (nan) | > current_lr: 5e-06 | > step_time: 0.206 (0.1372865100603947) | > loader_time: 0.0049 (0.004716396353103046)  --> TIME: 2025-02-26 16:25:02 -- STEP: 22420/36938 -- GLOBAL_STEP: 1315250 | > loss_text_ce: 0.03582685813307762 (0.031015693274202834) | > loss_mel_ce: 2.9158754348754883 (nan) | > loss: 0.011713105253875256 (nan) | > current_lr: 5e-06 | > step_time: 0.1816 (0.13728903218319494) | > loader_time: 0.0049 (0.004716460232645372)  --> TIME: 2025-02-26 16:25:11 -- STEP: 22470/36938 -- GLOBAL_STEP: 1315300 | > loss_text_ce: 0.03856131434440613 (0.031015813508094632) | > loss_mel_ce: 2.631351947784424 (nan) | > loss: 0.010594894178211689 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1372798261188321) | > loader_time: 0.0046 (0.004716498224693897)  --> TIME: 2025-02-26 16:25:20 -- STEP: 22520/36938 -- GLOBAL_STEP: 1315350 | > loss_text_ce: 0.028836527839303017 (0.031014824518933892) | > loss_mel_ce: 3.088120460510254 (nan) | > loss: 0.012368877418339252 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13728240583336748) | > loader_time: 0.0045 (0.004716668808650821)  --> TIME: 2025-02-26 16:25:30 -- STEP: 22570/36938 -- GLOBAL_STEP: 1315400 | > loss_text_ce: 0.024984098970890045 (0.031012147294062795) | > loss_mel_ce: 2.705690860748291 (nan) | > loss: 0.010836012661457062 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13728392325939157) | > loader_time: 0.0046 (0.004716618556494644)  --> TIME: 2025-02-26 16:25:39 -- STEP: 22620/36938 -- GLOBAL_STEP: 1315450 | > loss_text_ce: 0.020183173939585686 (0.03101289331053312) | > loss_mel_ce: 1.2523651123046875 (nan) | > loss: 0.005049795377999544 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13729311632540844) | > loader_time: 0.0046 (0.004716571288020308)  --> TIME: 2025-02-26 16:25:49 -- STEP: 22670/36938 -- GLOBAL_STEP: 1315500 | > loss_text_ce: 0.018611367791891098 (0.03101319932260924) | > loss_mel_ce: 2.160991907119751 (nan) | > loss: 0.008649219758808613 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13729054780589017) | > loader_time: 0.0047 (0.004716422003589138)  --> TIME: 2025-02-26 16:25:58 -- STEP: 22720/36938 -- GLOBAL_STEP: 1315550 | > loss_text_ce: 0.0346333384513855 (0.03101434835140974) | > loss_mel_ce: 3.44685435295105 (nan) | > loss: 0.013815428130328655 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13729714830996254) | > loader_time: 0.0046 (0.00471626672316607)  --> TIME: 2025-02-26 16:26:08 -- STEP: 22770/36938 -- GLOBAL_STEP: 1315600 | > loss_text_ce: 0.02518235705792904 (0.031012354989453896) | > loss_mel_ce: 2.538548469543457 (nan) | > loss: 0.010173534974455833 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13729297202775775) | > loader_time: 0.0045 (0.004716126951252013)  --> TIME: 2025-02-26 16:26:17 -- STEP: 22820/36938 -- GLOBAL_STEP: 1315650 | > loss_text_ce: 0.0361822210252285 (0.031014928214846214) | > loss_mel_ce: 2.972628355026245 (nan) | > loss: 0.011939724907279015 (nan) | > current_lr: 5e-06 | > step_time: 0.1963 (0.13730559212193977) | > loader_time: 0.0049 (0.004716201083478349)  --> TIME: 2025-02-26 16:26:27 -- STEP: 22870/36938 -- GLOBAL_STEP: 1315700 | > loss_text_ce: 0.032700151205062866 (0.031014960702544524) | > loss_mel_ce: 2.483038902282715 (nan) | > loss: 0.009983091615140438 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13730996996750683) | > loader_time: 0.0046 (0.004716315423759474)  --> TIME: 2025-02-26 16:26:36 -- STEP: 22920/36938 -- GLOBAL_STEP: 1315750 | > loss_text_ce: 0.030093099921941757 (0.031016695183087434) | > loss_mel_ce: 3.3548941612243652 (nan) | > loss: 0.013432490639388561 (nan) | > current_lr: 5e-06 | > step_time: 0.1931 (0.13731864100975466) | > loader_time: 0.0049 (0.004716482160395162)  --> TIME: 2025-02-26 16:26:46 -- STEP: 22970/36938 -- GLOBAL_STEP: 1315800 | > loss_text_ce: 0.03300976753234863 (0.031016903223237125) | > loss_mel_ce: 2.5901782512664795 (nan) | > loss: 0.010409477166831493 (nan) | > current_lr: 5e-06 | > step_time: 0.1436 (0.1373188501552334) | > loader_time: 0.0048 (0.004716402393452353)  --> TIME: 2025-02-26 16:26:55 -- STEP: 23020/36938 -- GLOBAL_STEP: 1315850 | > loss_text_ce: 0.031831175088882446 (0.031017261527545043) | > loss_mel_ce: 3.279963731765747 (nan) | > loss: 0.01314204465597868 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.13731185037296376) | > loader_time: 0.0046 (0.004716353712653638)  --> TIME: 2025-02-26 16:27:04 -- STEP: 23070/36938 -- GLOBAL_STEP: 1315900 | > loss_text_ce: 0.03230230137705803 (0.031017242663301362) | > loss_mel_ce: 2.5403692722320557 (nan) | > loss: 0.010209014639258385 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1373065886065509) | > loader_time: 0.0046 (0.004716223961458528)  --> TIME: 2025-02-26 16:27:14 -- STEP: 23120/36938 -- GLOBAL_STEP: 1315950 | > loss_text_ce: 0.03552929684519768 (0.031018059701734515) | > loss_mel_ce: 2.8201029300689697 (nan) | > loss: 0.011331874877214432 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13730850120729193) | > loader_time: 0.0048 (0.004716112786923332)  --> TIME: 2025-02-26 16:27:23 -- STEP: 23170/36938 -- GLOBAL_STEP: 1316000 | > loss_text_ce: 0.03071938268840313 (0.03101779420397465) | > loss_mel_ce: 1.8099123239517212 (nan) | > loss: 0.007304094731807709 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13731892647082578) | > loader_time: 0.0045 (0.004715970965053297)  --> TIME: 2025-02-26 16:27:33 -- STEP: 23220/36938 -- GLOBAL_STEP: 1316050 | > loss_text_ce: 0.03630847483873367 (0.031021031504176063) | > loss_mel_ce: 2.5658016204833984 (nan) | > loss: 0.010325834155082703 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.1373139180981011) | > loader_time: 0.0046 (0.004715785427816379)  --> TIME: 2025-02-26 16:27:42 -- STEP: 23270/36938 -- GLOBAL_STEP: 1316100 | > loss_text_ce: 0.031168129295110703 (0.03101960627960868) | > loss_mel_ce: 2.811323404312134 (nan) | > loss: 0.011279729194939137 (nan) | > current_lr: 5e-06 | > step_time: 0.1909 (0.13730387716371165) | > loader_time: 0.0049 (0.004715644652412149)  --> TIME: 2025-02-26 16:27:51 -- STEP: 23320/36938 -- GLOBAL_STEP: 1316150 | > loss_text_ce: 0.03046194277703762 (0.03102069535471601) | > loss_mel_ce: 2.292003870010376 (nan) | > loss: 0.009216135367751122 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13730161763587062) | > loader_time: 0.0045 (0.004715555691596386)  --> TIME: 2025-02-26 16:28:00 -- STEP: 23370/36938 -- GLOBAL_STEP: 1316200 | > loss_text_ce: 0.03161259740591049 (0.03102216756805797) | > loss_mel_ce: 3.1018803119659424 (nan) | > loss: 0.01243449654430151 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.1373017018913556) | > loader_time: 0.0047 (0.004715403237300004)  --> TIME: 2025-02-26 16:28:10 -- STEP: 23420/36938 -- GLOBAL_STEP: 1316250 | > loss_text_ce: 0.030904902145266533 (0.031022470820230692) | > loss_mel_ce: 2.6200084686279297 (nan) | > loss: 0.010519498959183693 (nan) | > current_lr: 5e-06 | > step_time: 0.1436 (0.137304831670961) | > loader_time: 0.0047 (0.004715266653251498)  --> TIME: 2025-02-26 16:28:19 -- STEP: 23470/36938 -- GLOBAL_STEP: 1316300 | > loss_text_ce: 0.037812333554029465 (0.031018646492201773) | > loss_mel_ce: 2.2938308715820312 (nan) | > loss: 0.00925255287438631 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13729846283482305) | > loader_time: 0.0046 (0.00471509652895577)  --> TIME: 2025-02-26 16:28:29 -- STEP: 23520/36938 -- GLOBAL_STEP: 1316350 | > loss_text_ce: 0.030896088108420372 (0.03101896983416922) | > loss_mel_ce: 2.6008036136627197 (nan) | > loss: 0.010443253442645073 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.13730635829523266) | > loader_time: 0.0048 (0.004715155683407175)  --> TIME: 2025-02-26 16:28:38 -- STEP: 23570/36938 -- GLOBAL_STEP: 1316400 | > loss_text_ce: 0.024668211117386818 (0.031017030588413225) | > loss_mel_ce: 4.077216148376465 (nan) | > loss: 0.016277320683002472 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.1373058518262851) | > loader_time: 0.0047 (0.0047152069700335825)  --> TIME: 2025-02-26 16:28:47 -- STEP: 23620/36938 -- GLOBAL_STEP: 1316450 | > loss_text_ce: 0.032861724495887756 (0.03101611609178754) | > loss_mel_ce: 3.179226875305176 (nan) | > loss: 0.012746384367346764 (nan) | > current_lr: 5e-06 | > step_time: 0.2037 (0.13730805502616134) | > loader_time: 0.0048 (0.0047152020586031605)  --> TIME: 2025-02-26 16:28:57 -- STEP: 23670/36938 -- GLOBAL_STEP: 1316500 | > loss_text_ce: 0.03534373641014099 (0.031015658090980964) | > loss_mel_ce: 2.9257924556732178 (nan) | > loss: 0.0117505406960845 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.13731832731913318) | > loader_time: 0.0047 (0.004715202909307642)  --> TIME: 2025-02-26 16:29:06 -- STEP: 23720/36938 -- GLOBAL_STEP: 1316550 | > loss_text_ce: 0.035005614161491394 (0.031015956050969195) | > loss_mel_ce: 2.770496129989624 (nan) | > loss: 0.011132943443953991 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13731180962181375) | > loader_time: 0.0047 (0.004715082798197146)  --> TIME: 2025-02-26 16:29:15 -- STEP: 23770/36938 -- GLOBAL_STEP: 1316600 | > loss_text_ce: 0.031383056193590164 (0.03101520959309823) | > loss_mel_ce: 3.028026819229126 (nan) | > loss: 0.012140516191720963 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13730801459766961) | > loader_time: 0.0047 (0.00471496781632838)  --> TIME: 2025-02-26 16:29:25 -- STEP: 23820/36938 -- GLOBAL_STEP: 1316650 | > loss_text_ce: 0.030260592699050903 (0.0310139589346351) | > loss_mel_ce: 1.8141072988510132 (nan) | > loss: 0.0073189204558730125 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13730913754974827) | > loader_time: 0.0046 (0.004715174261408645)  --> TIME: 2025-02-26 16:29:34 -- STEP: 23870/36938 -- GLOBAL_STEP: 1316700 | > loss_text_ce: 0.028371447697281837 (0.031012299207866717) | > loss_mel_ce: 2.567967176437378 (nan) | > loss: 0.010302931070327759 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.13730503814610934) | > loader_time: 0.0046 (0.004715079745838835)  --> TIME: 2025-02-26 16:29:43 -- STEP: 23920/36938 -- GLOBAL_STEP: 1316750 | > loss_text_ce: 0.023533815518021584 (0.031010258570001975) | > loss_mel_ce: 2.4495623111724854 (nan) | > loss: 0.009813874028623104 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.1372973784854177) | > loader_time: 0.0046 (0.004715037345886245)  --> TIME: 2025-02-26 16:29:52 -- STEP: 23970/36938 -- GLOBAL_STEP: 1316800 | > loss_text_ce: 0.04269767180085182 (0.031010813728319913) | > loss_mel_ce: 2.697073459625244 (nan) | > loss: 0.010872107930481434 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13729693253039288) | > loader_time: 0.0045 (0.0047148626647395536)  --> TIME: 2025-02-26 16:30:02 -- STEP: 24020/36938 -- GLOBAL_STEP: 1316850 | > loss_text_ce: 0.045068979263305664 (0.03101300607271784) | > loss_mel_ce: 2.5333199501037598 (nan) | > loss: 0.010231702588498592 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13729941333759602) | > loader_time: 0.0047 (0.004714696820232905)  --> TIME: 2025-02-26 16:30:11 -- STEP: 24070/36938 -- GLOBAL_STEP: 1316900 | > loss_text_ce: 0.024167034775018692 (0.03101104083523235) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.1372925594133311) | > loader_time: 0.0044 (0.0047148549413106675)  --> TIME: 2025-02-26 16:30:20 -- STEP: 24120/36938 -- GLOBAL_STEP: 1316950 | > loss_text_ce: 0.03461276367306709 (0.031011113849164267) | > loss_mel_ce: 2.3872203826904297 (nan) | > loss: 0.009610448963940144 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13729435202494153) | > loader_time: 0.0045 (0.004714670040911918)  --> TIME: 2025-02-26 16:30:29 -- STEP: 24170/36938 -- GLOBAL_STEP: 1317000 | > loss_text_ce: 0.024598145857453346 (0.03101422984929841) | > loss_mel_ce: 2.303306818008423 (nan) | > loss: 0.009237718768417835 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1372863814709446) | > loader_time: 0.0045 (0.004714635131671811)  --> TIME: 2025-02-26 16:30:39 -- STEP: 24220/36938 -- GLOBAL_STEP: 1317050 | > loss_text_ce: 0.031240615993738174 (0.03101364555611013) | > loss_mel_ce: 2.8343749046325684 (nan) | > loss: 0.011371491476893425 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13729515612174578) | > loader_time: 0.0046 (0.004714703678001967)  --> TIME: 2025-02-26 16:30:48 -- STEP: 24270/36938 -- GLOBAL_STEP: 1317100 | > loss_text_ce: 0.03056202083826065 (0.03101124662098555) | > loss_mel_ce: 2.902496576309204 (nan) | > loss: 0.011639121919870377 (nan) | > current_lr: 5e-06 | > step_time: 0.1846 (0.1372952301216356) | > loader_time: 0.0048 (0.0047144858297812646)  --> TIME: 2025-02-26 16:30:58 -- STEP: 24320/36938 -- GLOBAL_STEP: 1317150 | > loss_text_ce: 0.03294135630130768 (0.031012332949275762) | > loss_mel_ce: 2.9079737663269043 (nan) | > loss: 0.011670298874378204 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13730018642779968) | > loader_time: 0.0047 (0.004714365793686178)  --> TIME: 2025-02-26 16:31:07 -- STEP: 24370/36938 -- GLOBAL_STEP: 1317200 | > loss_text_ce: 0.02607686258852482 (0.031011373818663604) | > loss_mel_ce: 1.1403019428253174 (nan) | > loss: 0.00462848786264658 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13730282179879755) | > loader_time: 0.0052 (0.004714416117270962)  --> TIME: 2025-02-26 16:31:16 -- STEP: 24420/36938 -- GLOBAL_STEP: 1317250 | > loss_text_ce: 0.035920921713113785 (0.031012212925717954) | > loss_mel_ce: 1.6803646087646484 (nan) | > loss: 0.0068106576800346375 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.13729044719371072) | > loader_time: 0.0048 (0.0047145219731780106)  --> TIME: 2025-02-26 16:31:25 -- STEP: 24470/36938 -- GLOBAL_STEP: 1317300 | > loss_text_ce: 0.02079685404896736 (0.031013267295085682) | > loss_mel_ce: 2.3061954975128174 (nan) | > loss: 0.009234096854925156 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1372790560509941) | > loader_time: 0.0047 (0.004714449425351646)  --> TIME: 2025-02-26 16:31:35 -- STEP: 24520/36938 -- GLOBAL_STEP: 1317350 | > loss_text_ce: 0.029915563762187958 (0.031011557809516276) | > loss_mel_ce: 2.896939754486084 (nan) | > loss: 0.011614506132900715 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13728624558954342) | > loader_time: 0.0048 (0.0047143832991601705)  --> TIME: 2025-02-26 16:31:44 -- STEP: 24570/36938 -- GLOBAL_STEP: 1317400 | > loss_text_ce: 0.025418201461434364 (0.03101205032555357) | > loss_mel_ce: 2.743993043899536 (nan) | > loss: 0.010989728383719921 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13728796198545554) | > loader_time: 0.0055 (0.004714630132387177)  --> TIME: 2025-02-26 16:31:54 -- STEP: 24620/36938 -- GLOBAL_STEP: 1317450 | > loss_text_ce: 0.03277057036757469 (0.031011080719336902) | > loss_mel_ce: 2.356032371520996 (nan) | > loss: 0.009479377418756485 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1372827002616736) | > loader_time: 0.0053 (0.004714913381782623)  --> TIME: 2025-02-26 16:32:03 -- STEP: 24670/36938 -- GLOBAL_STEP: 1317500 | > loss_text_ce: 0.023576395586133003 (0.03101120826474534) | > loss_mel_ce: 2.5390625 (nan) | > loss: 0.010169203393161297 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13728696251379316) | > loader_time: 0.0046 (0.004715130103953622)  --> TIME: 2025-02-26 16:32:13 -- STEP: 24720/36938 -- GLOBAL_STEP: 1317550 | > loss_text_ce: 0.026871446520090103 (0.031011551670492007) | > loss_mel_ce: 2.4011781215667725 (nan) | > loss: 0.009635117836296558 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.1372919388187732) | > loader_time: 0.0046 (0.004715109295829611)  --> TIME: 2025-02-26 16:32:22 -- STEP: 24770/36938 -- GLOBAL_STEP: 1317600 | > loss_text_ce: 0.03605656698346138 (0.031012058726989023) | > loss_mel_ce: 2.551844835281372 (nan) | > loss: 0.010269450023770332 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13729101906969984) | > loader_time: 0.0046 (0.004715335162166256)  --> TIME: 2025-02-26 16:32:31 -- STEP: 24820/36938 -- GLOBAL_STEP: 1317650 | > loss_text_ce: 0.031039204448461533 (0.03100982672402802) | > loss_mel_ce: 3.171977996826172 (nan) | > loss: 0.012710386887192726 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.1372866624694586) | > loader_time: 0.0047 (0.004715535805938142)  --> TIME: 2025-02-26 16:32:41 -- STEP: 24870/36938 -- GLOBAL_STEP: 1317700 | > loss_text_ce: 0.028059545904397964 (0.0310099908443573) | > loss_mel_ce: 2.409804582595825 (nan) | > loss: 0.009674063883721828 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.1372988145707062) | > loader_time: 0.0046 (0.004715648241967248)  --> TIME: 2025-02-26 16:32:50 -- STEP: 24920/36938 -- GLOBAL_STEP: 1317750 | > loss_text_ce: 0.034885015338659286 (0.03100891998131127) | > loss_mel_ce: 2.42863392829895 (nan) | > loss: 0.009775868616998196 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.13729993523028158) | > loader_time: 0.0048 (0.004715862310525894)  --> TIME: 2025-02-26 16:32:59 -- STEP: 24970/36938 -- GLOBAL_STEP: 1317800 | > loss_text_ce: 0.02820722945034504 (0.031010887202421487) | > loss_mel_ce: 0.7585693597793579 (nan) | > loss: 0.00312212947756052 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.137296122945113) | > loader_time: 0.0046 (0.004715814149327222)  --> TIME: 2025-02-26 16:33:09 -- STEP: 25020/36938 -- GLOBAL_STEP: 1317850 | > loss_text_ce: 0.03748307377099991 (0.031013045126466538) | > loss_mel_ce: 2.314971923828125 (nan) | > loss: 0.009335138835012913 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.1372938456581068) | > loader_time: 0.0046 (0.004715938791096648)  --> TIME: 2025-02-26 16:33:18 -- STEP: 25070/36938 -- GLOBAL_STEP: 1317900 | > loss_text_ce: 0.031015073880553246 (0.031016417156338787) | > loss_mel_ce: 1.6129001379013062 (nan) | > loss: 0.006523473188281059 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.1373000138967113) | > loader_time: 0.0045 (0.004716134727548035)  --> TIME: 2025-02-26 16:33:28 -- STEP: 25120/36938 -- GLOBAL_STEP: 1317950 | > loss_text_ce: 0.02662547118961811 (0.031015997992273846) | > loss_mel_ce: 3.1199655532836914 (nan) | > loss: 0.012486472725868225 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.1373068506550626) | > loader_time: 0.0055 (0.004716211168249711)  --> TIME: 2025-02-26 16:33:37 -- STEP: 25170/36938 -- GLOBAL_STEP: 1318000 | > loss_text_ce: 0.031584855169057846 (0.03101683401339493) | > loss_mel_ce: 2.5490005016326904 (nan) | > loss: 0.010240417905151844 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.1373027256096824) | > loader_time: 0.0048 (0.004716144178327618)  --> TIME: 2025-02-26 16:33:46 -- STEP: 25220/36938 -- GLOBAL_STEP: 1318050 | > loss_text_ce: 0.021262919530272484 (0.031019864590952288) | > loss_mel_ce: 2.7036919593811035 (nan) | > loss: 0.010813313536345959 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13730292743770475) | > loader_time: 0.0045 (0.004716031566486774)  --> TIME: 2025-02-26 16:33:56 -- STEP: 25270/36938 -- GLOBAL_STEP: 1318100 | > loss_text_ce: 0.03275373950600624 (0.031019990225398097) | > loss_mel_ce: 2.09627103805542 (nan) | > loss: 0.008448511362075806 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13731152029154342) | > loader_time: 0.0045 (0.004715872310959973)  --> TIME: 2025-02-26 16:34:05 -- STEP: 25320/36938 -- GLOBAL_STEP: 1318150 | > loss_text_ce: 0.03214595094323158 (0.03102067432419278) | > loss_mel_ce: 2.5846898555755615 (nan) | > loss: 0.01038426999002695 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.13731610347497583) | > loader_time: 0.0048 (0.00471579876932865)  --> TIME: 2025-02-26 16:34:15 -- STEP: 25370/36938 -- GLOBAL_STEP: 1318200 | > loss_text_ce: 0.034556496888399124 (0.031020706783940484) | > loss_mel_ce: 1.9033454656600952 (nan) | > loss: 0.007690087426453829 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13731963566827704) | > loader_time: 0.0049 (0.004715643410237237)  --> TIME: 2025-02-26 16:34:24 -- STEP: 25420/36938 -- GLOBAL_STEP: 1318250 | > loss_text_ce: 0.03758377209305763 (0.031021094771415687) | > loss_mel_ce: 2.461541175842285 (nan) | > loss: 0.009917163290083408 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13731236832991503) | > loader_time: 0.0048 (0.00471548146848882)  --> TIME: 2025-02-26 16:34:34 -- STEP: 25470/36938 -- GLOBAL_STEP: 1318300 | > loss_text_ce: 0.03718503564596176 (0.031019466810702143) | > loss_mel_ce: 2.786163568496704 (nan) | > loss: 0.01120376493781805 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.1373186989395943) | > loader_time: 0.0046 (0.004715394543159891)  --> TIME: 2025-02-26 16:34:43 -- STEP: 25520/36938 -- GLOBAL_STEP: 1318350 | > loss_text_ce: 0.035978663712739944 (0.031019839341131835) | > loss_mel_ce: 2.4121038913726807 (nan) | > loss: 0.009714613668620586 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13732318335379048) | > loader_time: 0.0046 (0.004715509941585393)  --> TIME: 2025-02-26 16:34:53 -- STEP: 25570/36938 -- GLOBAL_STEP: 1318400 | > loss_text_ce: 0.029845263808965683 (0.03101861800067491) | > loss_mel_ce: 2.1664249897003174 (nan) | > loss: 0.008715358562767506 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13732418241004676) | > loader_time: 0.0046 (0.004715573335467821)  --> TIME: 2025-02-26 16:35:02 -- STEP: 25620/36938 -- GLOBAL_STEP: 1318450 | > loss_text_ce: 0.02646220475435257 (0.031016758089604482) | > loss_mel_ce: 2.662506103515625 (nan) | > loss: 0.010670510120689869 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.1373286517405292) | > loader_time: 0.0046 (0.004715520101632117)  --> TIME: 2025-02-26 16:35:11 -- STEP: 25670/36938 -- GLOBAL_STEP: 1318500 | > loss_text_ce: 0.033407747745513916 (0.031017725329184388) | > loss_mel_ce: 1.2253824472427368 (nan) | > loss: 0.004995199851691723 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.1373295538426361) | > loader_time: 0.0046 (0.004715400230666295)  --> TIME: 2025-02-26 16:35:21 -- STEP: 25720/36938 -- GLOBAL_STEP: 1318550 | > loss_text_ce: 0.025969121605157852 (0.0310188834018452) | > loss_mel_ce: 1.8562796115875244 (nan) | > loss: 0.0074692415073513985 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13732691288550442) | > loader_time: 0.0052 (0.004715363345287087)  --> TIME: 2025-02-26 16:35:30 -- STEP: 25770/36938 -- GLOBAL_STEP: 1318600 | > loss_text_ce: 0.027201205492019653 (0.031018938388831574) | > loss_mel_ce: 2.6734135150909424 (nan) | > loss: 0.010716725140810013 (nan) | > current_lr: 5e-06 | > step_time: 0.1725 (0.1373328477132662) | > loader_time: 0.0046 (0.004715332200372694)  --> TIME: 2025-02-26 16:35:40 -- STEP: 25820/36938 -- GLOBAL_STEP: 1318650 | > loss_text_ce: 0.03652608022093773 (0.031017912811571537) | > loss_mel_ce: 2.221985101699829 (nan) | > loss: 0.008962346240878105 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.1373434626529777) | > loader_time: 0.0049 (0.004715305266686881)  --> TIME: 2025-02-26 16:35:49 -- STEP: 25870/36938 -- GLOBAL_STEP: 1318700 | > loss_text_ce: 0.02375791408121586 (0.03101824156117021) | > loss_mel_ce: 2.496920585632324 (nan) | > loss: 0.01000269316136837 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1373501206255332) | > loader_time: 0.0045 (0.004715276243698665)  --> TIME: 2025-02-26 16:35:59 -- STEP: 25920/36938 -- GLOBAL_STEP: 1318750 | > loss_text_ce: 0.021488258615136147 (0.03101695241829823) | > loss_mel_ce: 1.265433669090271 (nan) | > loss: 0.005106833763420582 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13734625393961536) | > loader_time: 0.0045 (0.004715392646598235)  --> TIME: 2025-02-26 16:36:08 -- STEP: 25970/36938 -- GLOBAL_STEP: 1318800 | > loss_text_ce: 0.03515813499689102 (0.031015364032864146) | > loss_mel_ce: 2.6962990760803223 (nan) | > loss: 0.010839116759598255 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.13735062636271347) | > loader_time: 0.0055 (0.004715387647685344)  --> TIME: 2025-02-26 16:36:18 -- STEP: 26020/36938 -- GLOBAL_STEP: 1318850 | > loss_text_ce: 0.03206339478492737 (0.031015325717658068) | > loss_mel_ce: 2.856496572494507 (nan) | > loss: 0.011462540365755558 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.1373633740499863) | > loader_time: 0.0046 (0.004715481947972908)  --> TIME: 2025-02-26 16:36:27 -- STEP: 26070/36938 -- GLOBAL_STEP: 1318900 | > loss_text_ce: 0.03762689605355263 (0.03101481409846313) | > loss_mel_ce: 3.1305136680603027 (nan) | > loss: 0.012571987695991993 (nan) | > current_lr: 5e-06 | > step_time: 0.165 (0.1373628367854556) | > loader_time: 0.0049 (0.004715377936017533)  --> TIME: 2025-02-26 16:36:37 -- STEP: 26120/36938 -- GLOBAL_STEP: 1318950 | > loss_text_ce: 0.026887180283665657 (0.031013164426140192) | > loss_mel_ce: 2.759263515472412 (nan) | > loss: 0.011056154035031796 (nan) | > current_lr: 5e-06 | > step_time: 0.2 (0.13736443069590745) | > loader_time: 0.0049 (0.004715418715206814)  --> TIME: 2025-02-26 16:36:46 -- STEP: 26170/36938 -- GLOBAL_STEP: 1319000 | > loss_text_ce: 0.03223145380616188 (0.03101237832949727) | > loss_mel_ce: 2.444837808609009 (nan) | > loss: 0.009829641319811344 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.1373598850634321) | > loader_time: 0.0048 (0.004715340284158773)  --> TIME: 2025-02-26 16:36:55 -- STEP: 26220/36938 -- GLOBAL_STEP: 1319050 | > loss_text_ce: 0.02855304442346096 (0.031012593716571946) | > loss_mel_ce: 2.6521191596984863 (nan) | > loss: 0.010637588798999786 (nan) | > current_lr: 5e-06 | > step_time: 0.1706 (0.1373608424952686) | > loader_time: 0.0046 (0.004715237619311285)  --> TIME: 2025-02-26 16:37:05 -- STEP: 26270/36938 -- GLOBAL_STEP: 1319100 | > loss_text_ce: 0.03451261669397354 (0.03101004084386428) | > loss_mel_ce: 3.1716206073760986 (nan) | > loss: 0.012722751125693321 (nan) | > current_lr: 5e-06 | > step_time: 0.1872 (0.13736769998758644) | > loader_time: 0.0051 (0.004715101765188656)  --> TIME: 2025-02-26 16:37:14 -- STEP: 26320/36938 -- GLOBAL_STEP: 1319150 | > loss_text_ce: 0.035378556698560715 (0.03101229038373298) | > loss_mel_ce: 3.306422233581543 (nan) | > loss: 0.013261114247143269 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13736330635822874) | > loader_time: 0.0046 (0.004714906179433909)  --> TIME: 2025-02-26 16:37:24 -- STEP: 26370/36938 -- GLOBAL_STEP: 1319200 | > loss_text_ce: 0.031930990517139435 (0.03101233389609305) | > loss_mel_ce: 2.757341146469116 (nan) | > loss: 0.011068540625274181 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.1373657500242061) | > loader_time: 0.0052 (0.004714986624300014)  --> TIME: 2025-02-26 16:37:33 -- STEP: 26420/36938 -- GLOBAL_STEP: 1319250 | > loss_text_ce: 0.036720551550388336 (0.031013296914366578) | > loss_mel_ce: 1.1511355638504028 (nan) | > loss: 0.004713715054094791 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13737071914983137) | > loader_time: 0.0045 (0.004715244658150349)  --> TIME: 2025-02-26 16:37:42 -- STEP: 26470/36938 -- GLOBAL_STEP: 1319300 | > loss_text_ce: 0.04331263154745102 (0.031012978683136794) | > loss_mel_ce: 2.290010690689087 (nan) | > loss: 0.009259219281375408 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13736042824158845) | > loader_time: 0.0045 (0.004715062926119133)  --> TIME: 2025-02-26 16:37:52 -- STEP: 26520/36938 -- GLOBAL_STEP: 1319350 | > loss_text_ce: 0.03470345586538315 (0.031014193857417386) | > loss_mel_ce: 1.96539306640625 (nan) | > loss: 0.007936891168355942 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.1373734170135494) | > loader_time: 0.0045 (0.00471496898484267)  --> TIME: 2025-02-26 16:38:01 -- STEP: 26570/36938 -- GLOBAL_STEP: 1319400 | > loss_text_ce: 0.03382672369480133 (0.031013847137831143) | > loss_mel_ce: 2.4273085594177246 (nan) | > loss: 0.009766411036252975 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.13737469544121617) | > loader_time: 0.0049 (0.004714862771800562)  --> TIME: 2025-02-26 16:38:11 -- STEP: 26620/36938 -- GLOBAL_STEP: 1319450 | > loss_text_ce: 0.030905820429325104 (0.03101333896129215) | > loss_mel_ce: 3.0221822261810303 (nan) | > loss: 0.012115429155528545 (nan) | > current_lr: 5e-06 | > step_time: 0.1719 (0.13737566513194638) | > loader_time: 0.0048 (0.004714701562604905)  --> TIME: 2025-02-26 16:38:20 -- STEP: 26670/36938 -- GLOBAL_STEP: 1319500 | > loss_text_ce: 0.03734692186117172 (0.031015175996389466) | > loss_mel_ce: 1.8227425813674927 (nan) | > loss: 0.0073813083581626415 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13737635713504887) | > loader_time: 0.0047 (0.004714542960333931)  --> TIME: 2025-02-26 16:38:29 -- STEP: 26720/36938 -- GLOBAL_STEP: 1319550 | > loss_text_ce: 0.029726659879088402 (0.03101717782789607) | > loss_mel_ce: 2.450824737548828 (nan) | > loss: 0.00984345842152834 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.1373731590405894) | > loader_time: 0.0048 (0.004714414932413739)  --> TIME: 2025-02-26 16:38:39 -- STEP: 26770/36938 -- GLOBAL_STEP: 1319600 | > loss_text_ce: 0.025729618966579437 (0.0310175581512709) | > loss_mel_ce: 1.7569308280944824 (nan) | > loss: 0.007074050139635801 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.1373750760032458) | > loader_time: 0.0045 (0.004714282715903952)  --> TIME: 2025-02-26 16:38:48 -- STEP: 26820/36938 -- GLOBAL_STEP: 1319650 | > loss_text_ce: 0.027420006692409515 (0.031016605461223257) | > loss_mel_ce: 1.9545549154281616 (nan) | > loss: 0.007864980027079582 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13737571571941623) | > loader_time: 0.0046 (0.004714204089842488)  --> TIME: 2025-02-26 16:38:58 -- STEP: 26870/36938 -- GLOBAL_STEP: 1319700 | > loss_text_ce: 0.025069937109947205 (0.031016446193621244) | > loss_mel_ce: 2.3963608741760254 (nan) | > loss: 0.009608852677047253 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13738140484902334) | > loader_time: 0.0046 (0.004714299765580835)  --> TIME: 2025-02-26 16:39:07 -- STEP: 26920/36938 -- GLOBAL_STEP: 1319750 | > loss_text_ce: 0.031251709908246994 (0.031019199393166542) | > loss_mel_ce: 2.640418767929077 (nan) | > loss: 0.010601866990327835 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.1373814746493908) | > loader_time: 0.0046 (0.0047145262781259556)  --> TIME: 2025-02-26 16:39:16 -- STEP: 26970/36938 -- GLOBAL_STEP: 1319800 | > loss_text_ce: 0.02184062823653221 (0.031017138279938548) | > loss_mel_ce: 2.829174757003784 (nan) | > loss: 0.011313553899526596 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13737520453750265) | > loader_time: 0.0052 (0.004714422797025216)  --> TIME: 2025-02-26 16:39:25 -- STEP: 27020/36938 -- GLOBAL_STEP: 1319850 | > loss_text_ce: 0.029081234708428383 (0.031017877683208926) | > loss_mel_ce: 1.193490743637085 (nan) | > loss: 0.004851476289331913 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13737523671346957) | > loader_time: 0.0046 (0.004714637460927462)  --> TIME: 2025-02-26 16:39:35 -- STEP: 27070/36938 -- GLOBAL_STEP: 1319900 | > loss_text_ce: 0.02894274704158306 (0.031016315848473357) | > loss_mel_ce: 3.1330814361572266 (nan) | > loss: 0.012547716498374939 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.1373798169374534) | > loader_time: 0.0046 (0.004714573526188694)  --> TIME: 2025-02-26 16:39:44 -- STEP: 27120/36938 -- GLOBAL_STEP: 1319950 | > loss_text_ce: 0.0340690091252327 (0.031017015591658002) | > loss_mel_ce: 2.5378270149230957 (nan) | > loss: 0.010205937549471855 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.1373804496804503) | > loader_time: 0.0052 (0.004714573968124968)  --> TIME: 2025-02-26 16:39:54 -- STEP: 27170/36938 -- GLOBAL_STEP: 1320000 | > loss_text_ce: 0.029607616364955902 (0.031015631344298262) | > loss_mel_ce: 2.6931819915771484 (nan) | > loss: 0.010804721154272556 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.1373822916330623) | > loader_time: 0.0049 (0.004714636492035975) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1320000.pth  --> TIME: 2025-02-26 16:40:09 -- STEP: 27220/36938 -- GLOBAL_STEP: 1320050 | > loss_text_ce: 0.01928730122745037 (0.03101409015548193) | > loss_mel_ce: 2.527453899383545 (nan) | > loss: 0.010106116533279419 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.1373738531852609) | > loader_time: 0.0053 (0.004714840129401052)  --> TIME: 2025-02-26 16:40:18 -- STEP: 27270/36938 -- GLOBAL_STEP: 1320100 | > loss_text_ce: 0.033988431096076965 (0.03101781909256698) | > loss_mel_ce: 3.065263509750366 (nan) | > loss: 0.012298619374632835 (nan) | > current_lr: 5e-06 | > step_time: 0.1866 (0.13736483621474715) | > loader_time: 0.0048 (0.004714810397977325)  --> TIME: 2025-02-26 16:40:27 -- STEP: 27320/36938 -- GLOBAL_STEP: 1320150 | > loss_text_ce: 0.035758163779973984 (0.03101923690613163) | > loss_mel_ce: 2.9199376106262207 (nan) | > loss: 0.011728952638804913 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.1373541134866387) | > loader_time: 0.0046 (0.0047146286919036865)  --> TIME: 2025-02-26 16:40:36 -- STEP: 27370/36938 -- GLOBAL_STEP: 1320200 | > loss_text_ce: 0.03471831604838371 (0.03101940360801566) | > loss_mel_ce: 2.985562324523926 (nan) | > loss: 0.011985241435468197 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13735172007264032) | > loader_time: 0.0052 (0.00471449064694491)  --> TIME: 2025-02-26 16:40:46 -- STEP: 27420/36938 -- GLOBAL_STEP: 1320250 | > loss_text_ce: 0.024916941300034523 (0.031019831747380454) | > loss_mel_ce: 2.2086777687072754 (nan) | > loss: 0.008863471448421478 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13735724537151564) | > loader_time: 0.0045 (0.004714387059646502)  --> TIME: 2025-02-26 16:40:55 -- STEP: 27470/36938 -- GLOBAL_STEP: 1320300 | > loss_text_ce: 0.037921879440546036 (0.03101784655540194) | > loss_mel_ce: 2.8502707481384277 (nan) | > loss: 0.011461082845926285 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.1373564340524062) | > loader_time: 0.0047 (0.0047142729396424474)  --> TIME: 2025-02-26 16:41:04 -- STEP: 27520/36938 -- GLOBAL_STEP: 1320350 | > loss_text_ce: 0.028041992336511612 (0.03101730690917087) | > loss_mel_ce: 1.7242405414581299 (nan) | > loss: 0.006953502539545298 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.13735360978820818) | > loader_time: 0.0054 (0.004714482884074384)  --> TIME: 2025-02-26 16:41:14 -- STEP: 27570/36938 -- GLOBAL_STEP: 1320400 | > loss_text_ce: 0.030745670199394226 (0.031018052167993203) | > loss_mel_ce: 2.315265655517578 (nan) | > loss: 0.009309569373726845 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13735241309689145) | > loader_time: 0.0045 (0.004714639886462451)  --> TIME: 2025-02-26 16:41:23 -- STEP: 27620/36938 -- GLOBAL_STEP: 1320450 | > loss_text_ce: 0.03205613046884537 (0.03101767510355091) | > loss_mel_ce: 3.0432581901550293 (nan) | > loss: 0.01220362912863493 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.137351315164806) | > loader_time: 0.0048 (0.0047148427697388355)  --> TIME: 2025-02-26 16:41:33 -- STEP: 27670/36938 -- GLOBAL_STEP: 1320500 | > loss_text_ce: 0.0478118471801281 (0.03101953824899248) | > loss_mel_ce: 1.7161529064178467 (nan) | > loss: 0.006999860517680645 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1373554905528473) | > loader_time: 0.0045 (0.004714955764856001)  --> TIME: 2025-02-26 16:41:42 -- STEP: 27720/36938 -- GLOBAL_STEP: 1320550 | > loss_text_ce: 0.033874060958623886 (0.031021180578801797) | > loss_mel_ce: 2.277636766433716 (nan) | > loss: 0.009172662161290646 (nan) | > current_lr: 5e-06 | > step_time: 0.1046 (0.1373590686706563) | > loader_time: 0.0045 (0.00471487348041839)  --> TIME: 2025-02-26 16:41:52 -- STEP: 27770/36938 -- GLOBAL_STEP: 1320600 | > loss_text_ce: 0.03888581320643425 (0.031019478196497547) | > loss_mel_ce: 1.8096822500228882 (nan) | > loss: 0.0073355878703296185 (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.1373621194449335) | > loader_time: 0.0046 (0.004714869199401851)  --> TIME: 2025-02-26 16:42:01 -- STEP: 27820/36938 -- GLOBAL_STEP: 1320650 | > loss_text_ce: 0.024996966123580933 (0.03101825724361902) | > loss_mel_ce: 2.870694637298584 (nan) | > loss: 0.011490840464830399 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.1373654343000321) | > loader_time: 0.0054 (0.004714913551624022)  --> TIME: 2025-02-26 16:42:10 -- STEP: 27870/36938 -- GLOBAL_STEP: 1320700 | > loss_text_ce: 0.02427670918405056 (0.031017699207513082) | > loss_mel_ce: 1.3450261354446411 (nan) | > loss: 0.005433741956949234 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13736430661927082) | > loader_time: 0.0046 (0.004714944356652131)  --> TIME: 2025-02-26 16:42:20 -- STEP: 27920/36938 -- GLOBAL_STEP: 1320750 | > loss_text_ce: 0.03567996248602867 (0.031019186940148238) | > loss_mel_ce: 2.6441283226013184 (nan) | > loss: 0.010634160600602627 (nan) | > current_lr: 5e-06 | > step_time: 0.1833 (0.13737033647418198) | > loader_time: 0.0048 (0.0047149897475639016)  --> TIME: 2025-02-26 16:42:29 -- STEP: 27970/36938 -- GLOBAL_STEP: 1320800 | > loss_text_ce: 0.029596664011478424 (0.031019979084365424) | > loss_mel_ce: 3.1453936100006104 (nan) | > loss: 0.012599168345332146 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13737397379733515) | > loader_time: 0.0047 (0.004715123907940977)  --> TIME: 2025-02-26 16:42:39 -- STEP: 28020/36938 -- GLOBAL_STEP: 1320850 | > loss_text_ce: 0.02384045720100403 (0.03101868983730408) | > loss_mel_ce: 2.44869065284729 (nan) | > loss: 0.00981163140386343 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.13736272705698765) | > loader_time: 0.0047 (0.0047149843951789765)  --> TIME: 2025-02-26 16:42:48 -- STEP: 28070/36938 -- GLOBAL_STEP: 1320900 | > loss_text_ce: 0.020633220672607422 (0.031018924938649533) | > loss_mel_ce: 0.004783948417752981 (nan) | > loss: 0.00010086179099744186 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13735464225683258) | > loader_time: 0.0047 (0.004714830974093713)  --> TIME: 2025-02-26 16:42:57 -- STEP: 28120/36938 -- GLOBAL_STEP: 1320950 | > loss_text_ce: 0.0213057529181242 (0.031019984892633072) | > loss_mel_ce: 3.1142828464508057 (nan) | > loss: 0.012442813254892826 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.1373580228468415) | > loader_time: 0.0045 (0.004714729351814255)  --> TIME: 2025-02-26 16:43:06 -- STEP: 28170/36938 -- GLOBAL_STEP: 1321000 | > loss_text_ce: 0.023705963045358658 (0.031018658861667343) | > loss_mel_ce: 0.0625 (nan) | > loss: 0.00034208715078420937 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13735556503454166) | > loader_time: 0.0045 (0.004714640337057405)  --> TIME: 2025-02-26 16:43:16 -- STEP: 28220/36938 -- GLOBAL_STEP: 1321050 | > loss_text_ce: 0.03334784135222435 (0.031018754837223977) | > loss_mel_ce: 3.002277135848999 (nan) | > loss: 0.012046131305396557 (nan) | > current_lr: 5e-06 | > step_time: 0.1891 (0.13736705241010516) | > loader_time: 0.0049 (0.004714713579876739)  --> TIME: 2025-02-26 16:43:26 -- STEP: 28270/36938 -- GLOBAL_STEP: 1321100 | > loss_text_ce: 0.03507325425744057 (0.03101633686370038) | > loss_mel_ce: 3.1302874088287354 (nan) | > loss: 0.012560956180095673 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.13738022507565165) | > loader_time: 0.0047 (0.004714640341422412)  --> TIME: 2025-02-26 16:43:35 -- STEP: 28320/36938 -- GLOBAL_STEP: 1321150 | > loss_text_ce: 0.030541755259037018 (0.031017903487972842) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13738116595031022) | > loader_time: 0.0046 (0.004714534941029431)  --> TIME: 2025-02-26 16:43:44 -- STEP: 28370/36938 -- GLOBAL_STEP: 1321200 | > loss_text_ce: 0.023282365873456 (0.031016691199316392) | > loss_mel_ce: 2.5654938220977783 (nan) | > loss: 0.010272921994328499 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13737019550476612) | > loader_time: 0.0046 (0.0047144849997067815)  --> TIME: 2025-02-26 16:43:53 -- STEP: 28420/36938 -- GLOBAL_STEP: 1321250 | > loss_text_ce: 0.034731123596429825 (0.03101657899037622) | > loss_mel_ce: 2.9868929386138916 (nan) | > loss: 0.011990572325885296 (nan) | > current_lr: 5e-06 | > step_time: 0.1262 (0.13736673128764124) | > loader_time: 0.0048 (0.0047143597069296845)  --> TIME: 2025-02-26 16:44:03 -- STEP: 28470/36938 -- GLOBAL_STEP: 1321300 | > loss_text_ce: 0.0218746829777956 (0.031018242782291285) | > loss_mel_ce: 2.790799140930176 (nan) | > loss: 0.01116140466183424 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13737401895285084) | > loader_time: 0.0047 (0.004714457143764158)  --> TIME: 2025-02-26 16:44:12 -- STEP: 28520/36938 -- GLOBAL_STEP: 1321350 | > loss_text_ce: 0.014980688691139221 (0.031018408851111557) | > loss_mel_ce: 2.187635898590088 (nan) | > loss: 0.008740543387830257 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13737058400438873) | > loader_time: 0.0049 (0.004714383316575077)  --> TIME: 2025-02-26 16:44:22 -- STEP: 28570/36938 -- GLOBAL_STEP: 1321400 | > loss_text_ce: 0.029334187507629395 (0.031018698665042532) | > loss_mel_ce: 2.44958233833313 (nan) | > loss: 0.009836971759796143 (nan) | > current_lr: 5e-06 | > step_time: 0.1832 (0.13736876500916992) | > loader_time: 0.005 (0.0047144397973263395)  --> TIME: 2025-02-26 16:44:31 -- STEP: 28620/36938 -- GLOBAL_STEP: 1321450 | > loss_text_ce: 0.03796083480119705 (0.03101770627568785) | > loss_mel_ce: 3.040398597717285 (nan) | > loss: 0.012215712107717991 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13737191335876553) | > loader_time: 0.0048 (0.004714724169503901)  --> TIME: 2025-02-26 16:44:41 -- STEP: 28670/36938 -- GLOBAL_STEP: 1321500 | > loss_text_ce: 0.02944311499595642 (0.031018485369360225) | > loss_mel_ce: 2.205946207046509 (nan) | > loss: 0.008870592340826988 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13738291825739082) | > loader_time: 0.0047 (0.004714681273102154)  --> TIME: 2025-02-26 16:44:51 -- STEP: 28720/36938 -- GLOBAL_STEP: 1321550 | > loss_text_ce: 0.018969213590025902 (0.03101965964957942) | > loss_mel_ce: 1.9478648900985718 (nan) | > loss: 0.007804897613823414 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.1373907557021925) | > loader_time: 0.0045 (0.00471462322642878)  --> TIME: 2025-02-26 16:45:00 -- STEP: 28770/36938 -- GLOBAL_STEP: 1321600 | > loss_text_ce: 0.03607591614127159 (0.031020550389971934) | > loss_mel_ce: 1.409055471420288 (nan) | > loss: 0.005734649021178484 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13739143410696786) | > loader_time: 0.0045 (0.004714470105242489)  --> TIME: 2025-02-26 16:45:09 -- STEP: 28820/36938 -- GLOBAL_STEP: 1321650 | > loss_text_ce: 0.02910049445927143 (0.031019898595502967) | > loss_mel_ce: 2.562870740890503 (nan) | > loss: 0.01028560008853674 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.1373915002546223) | > loader_time: 0.0046 (0.004714368938324288)  --> TIME: 2025-02-26 16:45:19 -- STEP: 28870/36938 -- GLOBAL_STEP: 1321700 | > loss_text_ce: 0.030223343521356583 (0.031018996241515044) | > loss_mel_ce: 2.8452959060668945 (nan) | > loss: 0.011410791426897049 (nan) | > current_lr: 5e-06 | > step_time: 0.144 (0.1373838961888303) | > loader_time: 0.0047 (0.004714270665400343)  --> TIME: 2025-02-26 16:45:28 -- STEP: 28920/36938 -- GLOBAL_STEP: 1321750 | > loss_text_ce: 0.03282476216554642 (0.031016778329581815) | > loss_mel_ce: 1.9467672109603882 (nan) | > loss: 0.007855524308979511 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13737713400258905) | > loader_time: 0.0046 (0.004714250737700736)  --> TIME: 2025-02-26 16:45:37 -- STEP: 28970/36938 -- GLOBAL_STEP: 1321800 | > loss_text_ce: 0.028241602703928947 (0.031015410100251006) | > loss_mel_ce: 2.206467866897583 (nan) | > loss: 0.008867895230650902 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.1373789740793847) | > loader_time: 0.005 (0.004714194683934822)  --> TIME: 2025-02-26 16:45:47 -- STEP: 29020/36938 -- GLOBAL_STEP: 1321850 | > loss_text_ce: 0.02500581368803978 (0.031017911870406104) | > loss_mel_ce: 2.176135540008545 (nan) | > loss: 0.00873468816280365 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13738445424770526) | > loader_time: 0.0051 (0.0047143037199234815)  --> TIME: 2025-02-26 16:45:56 -- STEP: 29070/36938 -- GLOBAL_STEP: 1321900 | > loss_text_ce: 0.03908896446228027 (0.031018597727188457) | > loss_mel_ce: 2.475756883621216 (nan) | > loss: 0.009979547932744026 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13738079785398352) | > loader_time: 0.0046 (0.004714348499086841)  --> TIME: 2025-02-26 16:46:05 -- STEP: 29120/36938 -- GLOBAL_STEP: 1321950 | > loss_text_ce: 0.019335201010107994 (0.031017500689287752) | > loss_mel_ce: 2.6400818824768066 (nan) | > loss: 0.010553243570029736 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.13738311787882845) | > loader_time: 0.0047 (0.0047143239814501706)  --> TIME: 2025-02-26 16:46:15 -- STEP: 29170/36938 -- GLOBAL_STEP: 1322000 | > loss_text_ce: 0.03504306823015213 (0.031017409515352253) | > loss_mel_ce: 3.191908836364746 (nan) | > loss: 0.012805365025997162 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.13738002079906972) | > loader_time: 0.0049 (0.004714283405365435)  --> TIME: 2025-02-26 16:46:24 -- STEP: 29220/36938 -- GLOBAL_STEP: 1322050 | > loss_text_ce: 0.030784429982304573 (0.031017360346319127) | > loss_mel_ce: 3.104856014251709 (nan) | > loss: 0.012443018145859241 (nan) | > current_lr: 5e-06 | > step_time: 0.1473 (0.13738817985868726) | > loader_time: 0.0053 (0.00471443664856272)  --> TIME: 2025-02-26 16:46:34 -- STEP: 29270/36938 -- GLOBAL_STEP: 1322100 | > loss_text_ce: 0.02469680644571781 (0.031017120716298526) | > loss_mel_ce: 1.752648949623108 (nan) | > loss: 0.0070529598742723465 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13739607468537446) | > loader_time: 0.0045 (0.004714505094939399)  --> TIME: 2025-02-26 16:46:43 -- STEP: 29320/36938 -- GLOBAL_STEP: 1322150 | > loss_text_ce: 0.03436516597867012 (0.03101611093673245) | > loss_mel_ce: 3.160947322845459 (nan) | > loss: 0.012679812498390675 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.13739650651182192) | > loader_time: 0.0048 (0.004714468727333013)  --> TIME: 2025-02-26 16:46:52 -- STEP: 29370/36938 -- GLOBAL_STEP: 1322200 | > loss_text_ce: 0.025491615757346153 (0.031017408344722146) | > loss_mel_ce: 3.0444445610046387 (nan) | > loss: 0.012182287871837616 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.1373914223363441) | > loader_time: 0.0046 (0.00471435389552834)  --> TIME: 2025-02-26 16:47:02 -- STEP: 29420/36938 -- GLOBAL_STEP: 1322250 | > loss_text_ce: 0.02497350424528122 (0.03101916004565527) | > loss_mel_ce: 0.5411427617073059 (nan) | > loss: 0.0022464932408183813 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13738985863127234) | > loader_time: 0.0047 (0.00471431049337849)  --> TIME: 2025-02-26 16:47:11 -- STEP: 29470/36938 -- GLOBAL_STEP: 1322300 | > loss_text_ce: 0.024164270609617233 (0.031018433126475262) | > loss_mel_ce: 0.2523193359375 (nan) | > loss: 0.0010971571318805218 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13739529061325373) | > loader_time: 0.0045 (0.004714238267451349)  --> TIME: 2025-02-26 16:47:21 -- STEP: 29520/36938 -- GLOBAL_STEP: 1322350 | > loss_text_ce: 0.022978803142905235 (0.031016697482676638) | > loss_mel_ce: 0.8265303373336792 (nan) | > loss: 0.0033710680436342955 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13740781922004242) | > loader_time: 0.0046 (0.004714195725071414)  --> TIME: 2025-02-26 16:47:30 -- STEP: 29570/36938 -- GLOBAL_STEP: 1322400 | > loss_text_ce: 0.027532929554581642 (0.03101654801770319) | > loss_mel_ce: 2.6248254776000977 (nan) | > loss: 0.010525232180953026 (nan) | > current_lr: 5e-06 | > step_time: 0.1442 (0.13740120413015358) | > loader_time: 0.0047 (0.004714111722237297)  --> TIME: 2025-02-26 16:47:40 -- STEP: 29620/36938 -- GLOBAL_STEP: 1322450 | > loss_text_ce: 0.032936666160821915 (0.031019033410898955) | > loss_mel_ce: 2.2795588970184326 (nan) | > loss: 0.009176569990813732 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.13739917453763117) | > loader_time: 0.0054 (0.004714110395056762)  --> TIME: 2025-02-26 16:47:49 -- STEP: 29670/36938 -- GLOBAL_STEP: 1322500 | > loss_text_ce: 0.02719656191766262 (0.031017853177545254) | > loss_mel_ce: 3.0733814239501953 (nan) | > loss: 0.012303882278501987 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13739375411600127) | > loader_time: 0.0047 (0.004714095765265882)  --> TIME: 2025-02-26 16:47:58 -- STEP: 29720/36938 -- GLOBAL_STEP: 1322550 | > loss_text_ce: 0.03197170048952103 (0.03101843087272756) | > loss_mel_ce: 2.7364273071289062 (nan) | > loss: 0.01098571065813303 (nan) | > current_lr: 5e-06 | > step_time: 0.1438 (0.13738821805084844) | > loader_time: 0.0047 (0.004714036413028471)  --> TIME: 2025-02-26 16:48:07 -- STEP: 29770/36938 -- GLOBAL_STEP: 1322600 | > loss_text_ce: 0.03706800565123558 (0.031017972318862892) | > loss_mel_ce: 3.0679008960723877 (nan) | > loss: 0.012321305461227894 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13738998288790957) | > loader_time: 0.0047 (0.004713990194188071)  --> TIME: 2025-02-26 16:48:17 -- STEP: 29820/36938 -- GLOBAL_STEP: 1322650 | > loss_text_ce: 0.025295861065387726 (0.03101738740528898) | > loss_mel_ce: 2.9173312187194824 (nan) | > loss: 0.011677092872560024 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13738999699362783) | > loader_time: 0.0048 (0.004713956067259899)  --> TIME: 2025-02-26 16:48:26 -- STEP: 29870/36938 -- GLOBAL_STEP: 1322700 | > loss_text_ce: 0.03243713453412056 (0.031018460944393457) | > loss_mel_ce: 3.25563645362854 (nan) | > loss: 0.01304791122674942 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13739042317224556) | > loader_time: 0.0047 (0.004713846721655572)  --> TIME: 2025-02-26 16:48:35 -- STEP: 29920/36938 -- GLOBAL_STEP: 1322750 | > loss_text_ce: 0.025160692632198334 (0.031019367726007975) | > loss_mel_ce: 1.206085205078125 (nan) | > loss: 0.004885896574705839 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13739051763067725) | > loader_time: 0.0045 (0.004713708098559472)  --> TIME: 2025-02-26 16:48:45 -- STEP: 29970/36938 -- GLOBAL_STEP: 1322800 | > loss_text_ce: 0.03258834779262543 (0.031019686964179848) | > loss_mel_ce: 2.758694648742676 (nan) | > loss: 0.011076520197093487 (nan) | > current_lr: 5e-06 | > step_time: 0.1628 (0.1373866738579356) | > loader_time: 0.0047 (0.004713567861684943)  --> TIME: 2025-02-26 16:48:54 -- STEP: 30020/36938 -- GLOBAL_STEP: 1322850 | > loss_text_ce: 0.042421188205480576 (0.031019915734820422) | > loss_mel_ce: 0.1798095703125 (nan) | > loss: 0.0008818681817501783 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13738249535563604) | > loader_time: 0.0045 (0.0047135312028284)  --> TIME: 2025-02-26 16:49:04 -- STEP: 30070/36938 -- GLOBAL_STEP: 1322900 | > loss_text_ce: 0.027131477370858192 (0.031020666056569376) | > loss_mel_ce: 2.499140501022339 (nan) | > loss: 0.01002488937228918 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13739022759848724) | > loader_time: 0.0046 (0.004713562433213002)  --> TIME: 2025-02-26 16:49:13 -- STEP: 30120/36938 -- GLOBAL_STEP: 1322950 | > loss_text_ce: 0.03136640787124634 (0.031022658870817666) | > loss_mel_ce: 2.981257438659668 (nan) | > loss: 0.011954857036471367 (nan) | > current_lr: 5e-06 | > step_time: 0.1701 (0.13739549714730373) | > loader_time: 0.0049 (0.004713555667821488)  --> TIME: 2025-02-26 16:49:22 -- STEP: 30170/36938 -- GLOBAL_STEP: 1323000 | > loss_text_ce: 0.028216928243637085 (0.031023163899420003) | > loss_mel_ce: 2.5183279514312744 (nan) | > loss: 0.010105337016284466 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13739680269517424) | > loader_time: 0.0047 (0.004713565954752715)  --> TIME: 2025-02-26 16:49:32 -- STEP: 30220/36938 -- GLOBAL_STEP: 1323050 | > loss_text_ce: 0.027964018285274506 (0.031023807748862493) | > loss_mel_ce: 2.967763900756836 (nan) | > loss: 0.011887810193002224 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.13739640963621735) | > loader_time: 0.0049 (0.004713562219684145)  --> TIME: 2025-02-26 16:49:42 -- STEP: 30270/36938 -- GLOBAL_STEP: 1323100 | > loss_text_ce: 0.03318670019507408 (0.031024113643439415) | > loss_mel_ce: 2.8723723888397217 (nan) | > loss: 0.011529996991157532 (nan) | > current_lr: 5e-06 | > step_time: 0.1862 (0.1374080470028101) | > loader_time: 0.0049 (0.004713721695929279)  --> TIME: 2025-02-26 16:49:51 -- STEP: 30320/36938 -- GLOBAL_STEP: 1323150 | > loss_text_ce: 0.03140461444854736 (0.03102312931612178) | > loss_mel_ce: 3.0143628120422363 (nan) | > loss: 0.01208637934178114 (nan) | > current_lr: 5e-06 | > step_time: 0.1886 (0.13741025686421127) | > loader_time: 0.0051 (0.004713789076792549)  --> TIME: 2025-02-26 16:50:00 -- STEP: 30370/36938 -- GLOBAL_STEP: 1323200 | > loss_text_ce: 0.032903041690588 (0.031022341809719113) | > loss_mel_ce: 2.900097131729126 (nan) | > loss: 0.011638890020549297 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.13740572558055217) | > loader_time: 0.0047 (0.004713980649936203)  --> TIME: 2025-02-26 16:50:10 -- STEP: 30420/36938 -- GLOBAL_STEP: 1323250 | > loss_text_ce: 0.03419636934995651 (0.03102358398866582) | > loss_mel_ce: 1.3983776569366455 (nan) | > loss: 0.0056848181411623955 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13740167180461818) | > loader_time: 0.0046 (0.004713943857334712)  --> TIME: 2025-02-26 16:50:19 -- STEP: 30470/36938 -- GLOBAL_STEP: 1323300 | > loss_text_ce: 0.023020002990961075 (0.031025466105060465) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13739892710299637) | > loader_time: 0.0047 (0.004713907075937686)  --> TIME: 2025-02-26 16:50:28 -- STEP: 30520/36938 -- GLOBAL_STEP: 1323350 | > loss_text_ce: 0.02456909418106079 (0.03102486292592971) | > loss_mel_ce: 2.635305643081665 (nan) | > loss: 0.010555058717727661 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.1374039891584811) | > loader_time: 0.0047 (0.004714107271259468)  --> TIME: 2025-02-26 16:50:37 -- STEP: 30570/36938 -- GLOBAL_STEP: 1323400 | > loss_text_ce: 0.031282976269721985 (0.031021432498470558) | > loss_mel_ce: 2.615114688873291 (nan) | > loss: 0.010501578450202942 (nan) | > current_lr: 5e-06 | > step_time: 0.1719 (0.1373953595362637) | > loader_time: 0.0049 (0.004714080310156441)  --> TIME: 2025-02-26 16:50:47 -- STEP: 30620/36938 -- GLOBAL_STEP: 1323450 | > loss_text_ce: 0.02807942032814026 (0.031021608247930838) | > loss_mel_ce: 2.5888495445251465 (nan) | > loss: 0.010384639725089073 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13739894314733983) | > loader_time: 0.0047 (0.0047142159775298694)  --> TIME: 2025-02-26 16:50:56 -- STEP: 30670/36938 -- GLOBAL_STEP: 1323500 | > loss_text_ce: 0.032850902527570724 (0.031022195750467885) | > loss_mel_ce: 2.9251060485839844 (nan) | > loss: 0.011737925000488758 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13739719790435803) | > loader_time: 0.0045 (0.004714328806603208)  --> TIME: 2025-02-26 16:51:05 -- STEP: 30720/36938 -- GLOBAL_STEP: 1323550 | > loss_text_ce: 0.03751777485013008 (0.031023135024073174) | > loss_mel_ce: 2.601916551589966 (nan) | > loss: 0.0104739461094141 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1373940423053361) | > loader_time: 0.0045 (0.00471419893049947)  --> TIME: 2025-02-26 16:51:15 -- STEP: 30770/36938 -- GLOBAL_STEP: 1323600 | > loss_text_ce: 0.02933654747903347 (0.031021796906900254) | > loss_mel_ce: 2.6317899227142334 (nan) | > loss: 0.010560025461018085 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13739224027262187) | > loader_time: 0.0049 (0.004714169260507921)  --> TIME: 2025-02-26 16:51:24 -- STEP: 30820/36938 -- GLOBAL_STEP: 1323650 | > loss_text_ce: 0.03072156198322773 (0.03102372488563567) | > loss_mel_ce: 2.6248676776885986 (nan) | > loss: 0.010538053698837757 (nan) | > current_lr: 5e-06 | > step_time: 0.1511 (0.13739442860902068) | > loader_time: 0.0048 (0.004714171334105154)  --> TIME: 2025-02-26 16:51:33 -- STEP: 30870/36938 -- GLOBAL_STEP: 1323700 | > loss_text_ce: 0.02653607539832592 (0.0310232605975499) | > loss_mel_ce: 2.6376256942749023 (nan) | > loss: 0.010572071187198162 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.1373922248855143) | > loader_time: 0.0048 (0.004714105984212374)  --> TIME: 2025-02-26 16:51:43 -- STEP: 30920/36938 -- GLOBAL_STEP: 1323750 | > loss_text_ce: 0.03282622620463371 (0.03102476449161021) | > loss_mel_ce: 2.7224185466766357 (nan) | > loss: 0.010933511890470982 (nan) | > current_lr: 5e-06 | > step_time: 0.1872 (0.1373917229579597) | > loader_time: 0.0048 (0.0047140913284148935)  --> TIME: 2025-02-26 16:51:52 -- STEP: 30970/36938 -- GLOBAL_STEP: 1323800 | > loss_text_ce: 0.059107664972543716 (0.03102684054096086) | > loss_mel_ce: 0.0034653800539672375 (nan) | > loss: 0.00024830576148815453 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13738936041492142) | > loader_time: 0.0046 (0.004714121740019102)  --> TIME: 2025-02-26 16:52:01 -- STEP: 31020/36938 -- GLOBAL_STEP: 1323850 | > loss_text_ce: 0.043998703360557556 (0.03102842034969592) | > loss_mel_ce: 2.7726447582244873 (nan) | > loss: 0.011177157051861286 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13738556496947732) | > loader_time: 0.0047 (0.004714157910288587)  --> TIME: 2025-02-26 16:52:11 -- STEP: 31070/36938 -- GLOBAL_STEP: 1323900 | > loss_text_ce: 0.037753570824861526 (0.031026473228730044) | > loss_mel_ce: 3.1842803955078125 (nan) | > loss: 0.012785850092768669 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.13739482173207393) | > loader_time: 0.0046 (0.004714209741052816)  --> TIME: 2025-02-26 16:52:21 -- STEP: 31120/36938 -- GLOBAL_STEP: 1323950 | > loss_text_ce: 0.0337093360722065 (0.031025645185314636) | > loss_mel_ce: 3.1379284858703613 (nan) | > loss: 0.012585865333676338 (nan) | > current_lr: 5e-06 | > step_time: 0.1846 (0.1373962038740077) | > loader_time: 0.005 (0.004714152094331085)  --> TIME: 2025-02-26 16:52:30 -- STEP: 31170/36938 -- GLOBAL_STEP: 1324000 | > loss_text_ce: 0.0321660190820694 (0.031026737975661176) | > loss_mel_ce: 2.0683391094207764 (nan) | > loss: 0.008335337974131107 (nan) | > current_lr: 5e-06 | > step_time: 0.1872 (0.13739539241423446) | > loader_time: 0.0049 (0.004714215945310244)  --> TIME: 2025-02-26 16:52:39 -- STEP: 31220/36938 -- GLOBAL_STEP: 1324050 | > loss_text_ce: 0.03559994697570801 (0.031027779779602783) | > loss_mel_ce: 2.0667431354522705 (nan) | > loss: 0.008342632092535496 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13739186924745433) | > loader_time: 0.0048 (0.004714208409848861)  --> TIME: 2025-02-26 16:52:48 -- STEP: 31270/36938 -- GLOBAL_STEP: 1324100 | > loss_text_ce: 0.03962264955043793 (0.03102926356657782) | > loss_mel_ce: 2.3030600547790527 (nan) | > loss: 0.009296360425651073 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.1373922431510881) | > loader_time: 0.0045 (0.004714131263791482)  --> TIME: 2025-02-26 16:52:58 -- STEP: 31320/36938 -- GLOBAL_STEP: 1324150 | > loss_text_ce: 0.03057742305099964 (0.03103020456464935) | > loss_mel_ce: 2.2440176010131836 (nan) | > loss: 0.009026171639561653 (nan) | > current_lr: 5e-06 | > step_time: 0.1773 (0.1373945778813838) | > loader_time: 0.0048 (0.004714261313203362)  --> TIME: 2025-02-26 16:53:07 -- STEP: 31370/36938 -- GLOBAL_STEP: 1324200 | > loss_text_ce: 0.027278954163193703 (0.031030615654162135) | > loss_mel_ce: 3.07077956199646 (nan) | > loss: 0.012293883599340916 (nan) | > current_lr: 5e-06 | > step_time: 0.1688 (0.1373979030842592) | > loader_time: 0.0049 (0.0047143606612133)  --> TIME: 2025-02-26 16:53:16 -- STEP: 31420/36938 -- GLOBAL_STEP: 1324250 | > loss_text_ce: 0.035190094262361526 (0.031028879350904946) | > loss_mel_ce: 2.9324779510498047 (nan) | > loss: 0.011776461265981197 (nan) | > current_lr: 5e-06 | > step_time: 0.1514 (0.1373889701826247) | > loader_time: 0.0046 (0.004714201469469966)  --> TIME: 2025-02-26 16:53:26 -- STEP: 31470/36938 -- GLOBAL_STEP: 1324300 | > loss_text_ce: 0.028290463611483574 (0.031030380839175947) | > loss_mel_ce: 2.326241970062256 (nan) | > loss: 0.009343383833765984 (nan) | > current_lr: 5e-06 | > step_time: 0.1721 (0.13738673434849782) | > loader_time: 0.0048 (0.0047141823497392965)  --> TIME: 2025-02-26 16:53:35 -- STEP: 31520/36938 -- GLOBAL_STEP: 1324350 | > loss_text_ce: 0.0357319600880146 (0.031028695665296938) | > loss_mel_ce: 2.7500317096710205 (nan) | > loss: 0.011054618284106255 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13739068566391138) | > loader_time: 0.0046 (0.004714093474567254)  --> TIME: 2025-02-26 16:53:44 -- STEP: 31570/36938 -- GLOBAL_STEP: 1324400 | > loss_text_ce: 0.030544180423021317 (0.03103027863560453) | > loss_mel_ce: 3.124228000640869 (nan) | > loss: 0.01251893863081932 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13737872749481897) | > loader_time: 0.0047 (0.004714034583170595)  --> TIME: 2025-02-26 16:53:54 -- STEP: 31620/36938 -- GLOBAL_STEP: 1324450 | > loss_text_ce: 0.027904121205210686 (0.031030021262311556) | > loss_mel_ce: 2.8397457599639893 (nan) | > loss: 0.011379563249647617 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13738256177142005) | > loader_time: 0.0047 (0.004714118476761526)  --> TIME: 2025-02-26 16:54:03 -- STEP: 31670/36938 -- GLOBAL_STEP: 1324500 | > loss_text_ce: 0.032905928790569305 (0.031028942306333976) | > loss_mel_ce: 2.11847186088562 (nan) | > loss: 0.008537213318049908 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13738408308507796) | > loader_time: 0.0048 (0.004714340684840775)  --> TIME: 2025-02-26 16:54:13 -- STEP: 31720/36938 -- GLOBAL_STEP: 1324550 | > loss_text_ce: 0.03167194500565529 (0.031028987176250668) | > loss_mel_ce: 2.496074914932251 (nan) | > loss: 0.010030742734670639 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13738500649510965) | > loader_time: 0.0046 (0.004714629914691454)  --> TIME: 2025-02-26 16:54:22 -- STEP: 31770/36938 -- GLOBAL_STEP: 1324600 | > loss_text_ce: 0.027996113523840904 (0.031027182864188225) | > loss_mel_ce: 2.6069841384887695 (nan) | > loss: 0.010456271469593048 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13738534225395255) | > loader_time: 0.0047 (0.0047145697017642214)  --> TIME: 2025-02-26 16:54:31 -- STEP: 31820/36938 -- GLOBAL_STEP: 1324650 | > loss_text_ce: 0.030926786363124847 (0.03102726227632243) | > loss_mel_ce: 1.8265554904937744 (nan) | > loss: 0.007370961830019951 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1373846402863582) | > loader_time: 0.0046 (0.004714490788901564)  --> TIME: 2025-02-26 16:54:41 -- STEP: 31870/36938 -- GLOBAL_STEP: 1324700 | > loss_text_ce: 0.032203711569309235 (0.031024662792436594) | > loss_mel_ce: 1.2349090576171875 (nan) | > loss: 0.005028225481510162 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13738197561412888) | > loader_time: 0.0045 (0.0047143603702829)  --> TIME: 2025-02-26 16:54:50 -- STEP: 31920/36938 -- GLOBAL_STEP: 1324750 | > loss_text_ce: 0.017433058470487595 (0.03102204474282946) | > loss_mel_ce: 2.826976776123047 (nan) | > loss: 0.011287341825664043 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13737932493662602) | > loader_time: 0.0046 (0.0047142212477543155)  --> TIME: 2025-02-26 16:54:59 -- STEP: 31970/36938 -- GLOBAL_STEP: 1324800 | > loss_text_ce: 0.025765933096408844 (0.031020992887906978) | > loss_mel_ce: 2.3786702156066895 (nan) | > loss: 0.009541413746774197 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13737667998485853) | > loader_time: 0.0046 (0.004714074297401746)  --> TIME: 2025-02-26 16:55:08 -- STEP: 32020/36938 -- GLOBAL_STEP: 1324850 | > loss_text_ce: 0.03479557856917381 (0.031020030961633728) | > loss_mel_ce: 3.371832847595215 (nan) | > loss: 0.01351836696267128 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1373765611186895) | > loader_time: 0.0046 (0.0047140463823679615)  --> TIME: 2025-02-26 16:55:18 -- STEP: 32070/36938 -- GLOBAL_STEP: 1324900 | > loss_text_ce: 0.03491950035095215 (0.03101882618366406) | > loss_mel_ce: 0.12333450466394424 (nan) | > loss: 0.0006279920926317573 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13737411445496178) | > loader_time: 0.0046 (0.004714126404029854)  --> TIME: 2025-02-26 16:55:27 -- STEP: 32120/36938 -- GLOBAL_STEP: 1324950 | > loss_text_ce: 0.026793045923113823 (0.03101846803695017) | > loss_mel_ce: 2.723027229309082 (nan) | > loss: 0.01091198530048132 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.13737668858221555) | > loader_time: 0.0048 (0.004714110519639597)  --> TIME: 2025-02-26 16:55:37 -- STEP: 32170/36938 -- GLOBAL_STEP: 1325000 | > loss_text_ce: 0.03425028175115585 (0.031018600708370484) | > loss_mel_ce: 3.1398990154266357 (nan) | > loss: 0.012595831416547298 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.1373753289387822) | > loader_time: 0.0047 (0.004714017489477474)  --> TIME: 2025-02-26 16:55:46 -- STEP: 32220/36938 -- GLOBAL_STEP: 1325050 | > loss_text_ce: 0.037811316549777985 (0.03101891432956733) | > loss_mel_ce: 1.8580807447433472 (nan) | > loss: 0.007523381616920233 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.1373681389123721) | > loader_time: 0.0046 (0.004714003103553413)  --> TIME: 2025-02-26 16:55:55 -- STEP: 32270/36938 -- GLOBAL_STEP: 1325100 | > loss_text_ce: 0.033267803490161896 (0.031018586482648942) | > loss_mel_ce: 2.9309091567993164 (nan) | > loss: 0.011762607842683792 (nan) | > current_lr: 5e-06 | > step_time: 0.1625 (0.13737269192610632) | > loader_time: 0.0047 (0.00471397479843127)  --> TIME: 2025-02-26 16:56:05 -- STEP: 32320/36938 -- GLOBAL_STEP: 1325150 | > loss_text_ce: 0.03518117219209671 (0.031020459435931475) | > loss_mel_ce: 2.496260404586792 (nan) | > loss: 0.01004540454596281 (nan) | > current_lr: 5e-06 | > step_time: 0.1733 (0.13737818262364435) | > loader_time: 0.0049 (0.004714011666503298)  --> TIME: 2025-02-26 16:56:14 -- STEP: 32370/36938 -- GLOBAL_STEP: 1325200 | > loss_text_ce: 0.03196193277835846 (0.031021354460437398) | > loss_mel_ce: 2.7549118995666504 (nan) | > loss: 0.011059023439884186 (nan) | > current_lr: 5e-06 | > step_time: 0.1832 (0.13737468653930032) | > loader_time: 0.0048 (0.004713974567633718)  --> TIME: 2025-02-26 16:56:23 -- STEP: 32420/36938 -- GLOBAL_STEP: 1325250 | > loss_text_ce: 0.030705651268363 (0.031021235927202285) | > loss_mel_ce: 3.1445260047912598 (nan) | > loss: 0.012600126676261425 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.13737087396101458) | > loader_time: 0.0048 (0.004713988782152006)  --> TIME: 2025-02-26 16:56:32 -- STEP: 32470/36938 -- GLOBAL_STEP: 1325300 | > loss_text_ce: 0.040409740060567856 (0.031021658200721856) | > loss_mel_ce: 2.989964246749878 (nan) | > loss: 0.012025294825434685 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13736174742478513) | > loader_time: 0.0047 (0.00471395494609678)  --> TIME: 2025-02-26 16:56:42 -- STEP: 32520/36938 -- GLOBAL_STEP: 1325350 | > loss_text_ce: 0.045863740146160126 (0.031021411211884536) | > loss_mel_ce: 1.6543160676956177 (nan) | > loss: 0.00674674566835165 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13736462389997428) | > loader_time: 0.0046 (0.0047140219701319016)  --> TIME: 2025-02-26 16:56:51 -- STEP: 32570/36938 -- GLOBAL_STEP: 1325400 | > loss_text_ce: 0.025089997798204422 (0.031019195649211857) | > loss_mel_ce: 2.9645895957946777 (nan) | > loss: 0.011863808147609234 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.13737065513916363) | > loader_time: 0.0049 (0.004714050672154592)  --> TIME: 2025-02-26 16:57:00 -- STEP: 32620/36938 -- GLOBAL_STEP: 1325450 | > loss_text_ce: 0.03451225534081459 (0.031018762485435943) | > loss_mel_ce: 3.1511433124542236 (nan) | > loss: 0.012641491368412971 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.13736567248748294) | > loader_time: 0.0049 (0.004713905822419462)  --> TIME: 2025-02-26 16:57:10 -- STEP: 32670/36938 -- GLOBAL_STEP: 1325500 | > loss_text_ce: 0.032428618520498276 (0.03101842818976436) | > loss_mel_ce: 2.167163372039795 (nan) | > loss: 0.008728539571166039 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.13737270343110122) | > loader_time: 0.0045 (0.004713933599954882)  --> TIME: 2025-02-26 16:57:19 -- STEP: 32720/36938 -- GLOBAL_STEP: 1325550 | > loss_text_ce: 0.03679663687944412 (0.03101827316602389) | > loss_mel_ce: 1.7456928491592407 (nan) | > loss: 0.007073371671140194 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.1373729396565894) | > loader_time: 0.0052 (0.004714039456290561)  --> TIME: 2025-02-26 16:57:29 -- STEP: 32770/36938 -- GLOBAL_STEP: 1325600 | > loss_text_ce: 0.04318889230489731 (0.031019260322244262) | > loss_mel_ce: 2.620048761367798 (nan) | > loss: 0.010568403638899326 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13736967634492486) | > loader_time: 0.0046 (0.004714098506341972)  --> TIME: 2025-02-26 16:57:39 -- STEP: 32820/36938 -- GLOBAL_STEP: 1325650 | > loss_text_ce: 0.03269915282726288 (0.031020485158708335) | > loss_mel_ce: 2.725071668624878 (nan) | > loss: 0.010943535715341568 (nan) | > current_lr: 5e-06 | > step_time: 0.1833 (0.13738082368369345) | > loader_time: 0.0049 (0.00471415819008856)  --> TIME: 2025-02-26 16:57:48 -- STEP: 32870/36938 -- GLOBAL_STEP: 1325700 | > loss_text_ce: 0.036494456231594086 (0.03101918188885369) | > loss_mel_ce: 2.5645623207092285 (nan) | > loss: 0.010321654379367828 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13738363334608014) | > loader_time: 0.0047 (0.004714236870195005)  --> TIME: 2025-02-26 16:57:58 -- STEP: 32920/36938 -- GLOBAL_STEP: 1325750 | > loss_text_ce: 0.03134843334555626 (0.03101949021057537) | > loss_mel_ce: 3.187600612640381 (nan) | > loss: 0.012773607857525349 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13738663792610004) | > loader_time: 0.0047 (0.004714242728338802)  --> TIME: 2025-02-26 16:58:07 -- STEP: 32970/36938 -- GLOBAL_STEP: 1325800 | > loss_text_ce: 0.03906318172812462 (0.031020829361652427) | > loss_mel_ce: 3.357797384262085 (nan) | > loss: 0.013479606248438358 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.1373834042466679) | > loader_time: 0.0048 (0.004714147857293471)  --> TIME: 2025-02-26 16:58:16 -- STEP: 33020/36938 -- GLOBAL_STEP: 1325850 | > loss_text_ce: 0.034504566341638565 (0.031020992864890736) | > loss_mel_ce: 2.3611207008361816 (nan) | > loss: 0.009506450034677982 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13738369518594098) | > loader_time: 0.0048 (0.004714079721994954)  --> TIME: 2025-02-26 16:58:25 -- STEP: 33070/36938 -- GLOBAL_STEP: 1325900 | > loss_text_ce: 0.027424072846770287 (0.031021383780485346) | > loss_mel_ce: 2.682666063308716 (nan) | > loss: 0.010754326358437538 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.1373703847596158) | > loader_time: 0.0046 (0.004713950728158936)  --> TIME: 2025-02-26 16:58:34 -- STEP: 33120/36938 -- GLOBAL_STEP: 1325950 | > loss_text_ce: 0.03177385404706001 (0.03102075297922258) | > loss_mel_ce: 2.885359287261963 (nan) | > loss: 0.011575926095247269 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13736510629527066) | > loader_time: 0.0047 (0.004713840156361692)  --> TIME: 2025-02-26 16:58:43 -- STEP: 33170/36938 -- GLOBAL_STEP: 1326000 | > loss_text_ce: 0.02842220477759838 (0.031021274551022028) | > loss_mel_ce: 2.0815420150756836 (nan) | > loss: 0.00837287399917841 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13735791090061453) | > loader_time: 0.0046 (0.004713788649243062)  --> TIME: 2025-02-26 16:58:52 -- STEP: 33220/36938 -- GLOBAL_STEP: 1326050 | > loss_text_ce: 0.029037121683359146 (0.03102222049754889) | > loss_mel_ce: 2.601017713546753 (nan) | > loss: 0.010436726734042168 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.1373531773031495) | > loader_time: 0.0049 (0.00471373234507122)  --> TIME: 2025-02-26 16:59:02 -- STEP: 33270/36938 -- GLOBAL_STEP: 1326100 | > loss_text_ce: 0.03175424784421921 (0.03102152229492198) | > loss_mel_ce: 2.26352596282959 (nan) | > loss: 0.009108255617320538 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13734485787746079) | > loader_time: 0.0066 (0.004713689632375656)  --> TIME: 2025-02-26 16:59:11 -- STEP: 33320/36938 -- GLOBAL_STEP: 1326150 | > loss_text_ce: 0.033037226647138596 (0.03102089944073376) | > loss_mel_ce: 2.711618661880493 (nan) | > loss: 0.01089149247854948 (nan) | > current_lr: 5e-06 | > step_time: 0.1632 (0.137342416009409) | > loader_time: 0.0047 (0.004713730558770917)  --> TIME: 2025-02-26 16:59:20 -- STEP: 33370/36938 -- GLOBAL_STEP: 1326200 | > loss_text_ce: 0.030474809929728508 (0.031018942940557653) | > loss_mel_ce: 2.648225784301758 (nan) | > loss: 0.010629765689373016 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.13734449057083037) | > loader_time: 0.0048 (0.004713675695012936)  --> TIME: 2025-02-26 16:59:29 -- STEP: 33420/36938 -- GLOBAL_STEP: 1326250 | > loss_text_ce: 0.03223051130771637 (0.0310178428256989) | > loss_mel_ce: 2.4193859100341797 (nan) | > loss: 0.009728637523949146 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13733938169793106) | > loader_time: 0.0046 (0.004713563816337668)  --> TIME: 2025-02-26 16:59:39 -- STEP: 33470/36938 -- GLOBAL_STEP: 1326300 | > loss_text_ce: 0.03592224046587944 (0.031017976585140633) | > loss_mel_ce: 2.4063403606414795 (nan) | > loss: 0.009691518731415272 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13733896697710668) | > loader_time: 0.0048 (0.004713470151540533)  --> TIME: 2025-02-26 16:59:48 -- STEP: 33520/36938 -- GLOBAL_STEP: 1326350 | > loss_text_ce: 0.024419689550995827 (0.031017570079347065) | > loss_mel_ce: 2.5159404277801514 (nan) | > loss: 0.010080794803798199 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13733930795459992) | > loader_time: 0.0046 (0.004713339815560859)  --> TIME: 2025-02-26 16:59:58 -- STEP: 33570/36938 -- GLOBAL_STEP: 1326400 | > loss_text_ce: 0.02688097581267357 (0.031018609539829007) | > loss_mel_ce: 2.768954038619995 (nan) | > loss: 0.011094584129750729 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13734471973365617) | > loader_time: 0.0047 (0.004713310483739901)  --> TIME: 2025-02-26 17:00:07 -- STEP: 33620/36938 -- GLOBAL_STEP: 1326450 | > loss_text_ce: 0.035519614815711975 (0.031020127789223768) | > loss_mel_ce: 2.9678800106048584 (nan) | > loss: 0.011918253265321255 (nan) | > current_lr: 5e-06 | > step_time: 0.1625 (0.13734123833756565) | > loader_time: 0.0047 (0.004713241873859735)  --> TIME: 2025-02-26 17:00:16 -- STEP: 33670/36938 -- GLOBAL_STEP: 1326500 | > loss_text_ce: 0.03181539848446846 (0.031020956621902244) | > loss_mel_ce: 2.2579152584075928 (nan) | > loss: 0.009086232632398605 (nan) | > current_lr: 5e-06 | > step_time: 0.1464 (0.13733857090833435) | > loader_time: 0.0046 (0.0047131280428867025)  --> TIME: 2025-02-26 17:00:25 -- STEP: 33720/36938 -- GLOBAL_STEP: 1326550 | > loss_text_ce: 0.04004644230008125 (0.03102175706986742) | > loss_mel_ce: 2.289712905883789 (nan) | > loss: 0.009245077148079872 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13733397230841626) | > loader_time: 0.0046 (0.004712969644496749)  --> TIME: 2025-02-26 17:00:35 -- STEP: 33770/36938 -- GLOBAL_STEP: 1326600 | > loss_text_ce: 0.0280589796602726 (0.031020804822871277) | > loss_mel_ce: 4.041666507720947 (nan) | > loss: 0.01614970527589321 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1373337407136109) | > loader_time: 0.0045 (0.00471289404263892)  --> TIME: 2025-02-26 17:00:44 -- STEP: 33820/36938 -- GLOBAL_STEP: 1326650 | > loss_text_ce: 0.037567246705293655 (0.031020296925867757) | > loss_mel_ce: 2.4995055198669434 (nan) | > loss: 0.010067748837172985 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13733154270227219) | > loader_time: 0.0053 (0.004712850613258508)  --> TIME: 2025-02-26 17:00:54 -- STEP: 33870/36938 -- GLOBAL_STEP: 1326700 | > loss_text_ce: 0.037611715495586395 (0.031021844450940368) | > loss_mel_ce: 2.73677134513855 (nan) | > loss: 0.011009457521140575 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.13733589813918154) | > loader_time: 0.0046 (0.004712737314040114)  --> TIME: 2025-02-26 17:01:03 -- STEP: 33920/36938 -- GLOBAL_STEP: 1326750 | > loss_text_ce: 0.029894081875681877 (0.031020261491206805) | > loss_mel_ce: 3.0748660564422607 (nan) | > loss: 0.01232047751545906 (nan) | > current_lr: 5e-06 | > step_time: 0.1855 (0.1373346038443845) | > loader_time: 0.0051 (0.0047127045550436515)  --> TIME: 2025-02-26 17:01:12 -- STEP: 33970/36938 -- GLOBAL_STEP: 1326800 | > loss_text_ce: 0.023473989218473434 (0.031018224041944622) | > loss_mel_ce: 2.1955935955047607 (nan) | > loss: 0.00880582444369793 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.1373387883823323) | > loader_time: 0.0046 (0.004712648850733807)  --> TIME: 2025-02-26 17:01:22 -- STEP: 34020/36938 -- GLOBAL_STEP: 1326850 | > loss_text_ce: 0.03544420003890991 (0.031018276469957747) | > loss_mel_ce: 2.8525614738464355 (nan) | > loss: 0.011460340581834316 (nan) | > current_lr: 5e-06 | > step_time: 0.1775 (0.1373342790160703) | > loader_time: 0.0048 (0.004712535941131268)  --> TIME: 2025-02-26 17:01:31 -- STEP: 34070/36938 -- GLOBAL_STEP: 1326900 | > loss_text_ce: 0.037684328854084015 (0.031018705205666023) | > loss_mel_ce: 2.1233363151550293 (nan) | > loss: 0.008575479499995708 (nan) | > current_lr: 5e-06 | > step_time: 0.1612 (0.1373333044544759) | > loader_time: 0.0048 (0.004712447344749475)  --> TIME: 2025-02-26 17:01:40 -- STEP: 34120/36938 -- GLOBAL_STEP: 1326950 | > loss_text_ce: 0.03324509039521217 (0.031017985695179832) | > loss_mel_ce: 2.712362766265869 (nan) | > loss: 0.01089526992291212 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13733090319499142) | > loader_time: 0.0046 (0.00471235461978534)  --> TIME: 2025-02-26 17:01:50 -- STEP: 34170/36938 -- GLOBAL_STEP: 1327000 | > loss_text_ce: 0.03198140487074852 (0.031016578205855974) | > loss_mel_ce: 2.4854836463928223 (nan) | > loss: 0.009989941492676735 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13732874131809555) | > loader_time: 0.0052 (0.0047123540379270705)  --> TIME: 2025-02-26 17:01:59 -- STEP: 34220/36938 -- GLOBAL_STEP: 1327050 | > loss_text_ce: 0.041334886103868484 (0.031016536628185706) | > loss_mel_ce: 2.6674370765686035 (nan) | > loss: 0.010749096050858498 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.1373275171937038) | > loader_time: 0.0047 (0.004712390349247122)  --> TIME: 2025-02-26 17:02:08 -- STEP: 34270/36938 -- GLOBAL_STEP: 1327100 | > loss_text_ce: 0.03312905877828598 (0.031014053851759007) | > loss_mel_ce: 2.8553009033203125 (nan) | > loss: 0.011462024413049221 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.1373301364466213) | > loader_time: 0.0046 (0.004712520850654396)  --> TIME: 2025-02-26 17:02:18 -- STEP: 34320/36938 -- GLOBAL_STEP: 1327150 | > loss_text_ce: 0.027840785682201385 (0.031014607175476393) | > loss_mel_ce: 2.8721275329589844 (nan) | > loss: 0.011507811956107616 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.13732995752132116) | > loader_time: 0.0046 (0.004712648873840472)  --> TIME: 2025-02-26 17:02:27 -- STEP: 34370/36938 -- GLOBAL_STEP: 1327200 | > loss_text_ce: 0.027146531268954277 (0.03101368616736129) | > loss_mel_ce: 2.4763906002044678 (nan) | > loss: 0.00993467215448618 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.1373320306981046) | > loader_time: 0.0051 (0.0047126674499489655)  --> TIME: 2025-02-26 17:02:37 -- STEP: 34420/36938 -- GLOBAL_STEP: 1327250 | > loss_text_ce: 0.02861429564654827 (0.031012748792063487) | > loss_mel_ce: 0.0830078125 (nan) | > loss: 0.00044294490362517536 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13733575397421463) | > loader_time: 0.0046 (0.004712825054210826)  --> TIME: 2025-02-26 17:02:46 -- STEP: 34470/36938 -- GLOBAL_STEP: 1327300 | > loss_text_ce: 0.027790291234850883 (0.03101252501281716) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13733844613904264) | > loader_time: 0.0045 (0.004712895334441327)  --> TIME: 2025-02-26 17:02:56 -- STEP: 34520/36938 -- GLOBAL_STEP: 1327350 | > loss_text_ce: 0.036862507462501526 (0.031013962457607784) | > loss_mel_ce: 2.747224807739258 (nan) | > loss: 0.01104796677827835 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.137336550941333) | > loader_time: 0.0046 (0.004712823996814635)  --> TIME: 2025-02-26 17:03:05 -- STEP: 34570/36938 -- GLOBAL_STEP: 1327400 | > loss_text_ce: 0.03169026970863342 (0.03101433690954975) | > loss_mel_ce: 1.9458516836166382 (nan) | > loss: 0.007847389206290245 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.1373388025917189) | > loader_time: 0.0053 (0.004712943517647978)  --> TIME: 2025-02-26 17:03:15 -- STEP: 34620/36938 -- GLOBAL_STEP: 1327450 | > loss_text_ce: 0.02674889750778675 (0.031014486646469615) | > loss_mel_ce: 2.4483110904693604 (nan) | > loss: 0.009821667335927486 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.13734580172340807) | > loader_time: 0.0048 (0.0047131295840496515)  --> TIME: 2025-02-26 17:03:24 -- STEP: 34670/36938 -- GLOBAL_STEP: 1327500 | > loss_text_ce: 0.019769396632909775 (0.031015165847421814) | > loss_mel_ce: 2.058394432067871 (nan) | > loss: 0.008246682584285736 (nan) | > current_lr: 5e-06 | > step_time: 0.1118 (0.13733605472711977) | > loader_time: 0.0045 (0.004713123340513175)  --> TIME: 2025-02-26 17:03:33 -- STEP: 34720/36938 -- GLOBAL_STEP: 1327550 | > loss_text_ce: 0.02552352100610733 (0.031014851870163868) | > loss_mel_ce: 2.3477301597595215 (nan) | > loss: 0.009417673572897911 (nan) | > current_lr: 5e-06 | > step_time: 0.1744 (0.13733528542078974) | > loader_time: 0.0055 (0.004713153372162506)  --> TIME: 2025-02-26 17:03:42 -- STEP: 34770/36938 -- GLOBAL_STEP: 1327600 | > loss_text_ce: 0.02265051379799843 (0.031013924234177353) | > loss_mel_ce: 2.2761473655700684 (nan) | > loss: 0.009122214280068874 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.137336500321168) | > loader_time: 0.0045 (0.004713147928360249)  --> TIME: 2025-02-26 17:03:51 -- STEP: 34820/36938 -- GLOBAL_STEP: 1327650 | > loss_text_ce: 0.02604225091636181 (0.03101281736190241) | > loss_mel_ce: 2.2518553733825684 (nan) | > loss: 0.009039276279509068 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.1373320505168489) | > loader_time: 0.0108 (0.004713264681154944)  --> TIME: 2025-02-26 17:04:01 -- STEP: 34870/36938 -- GLOBAL_STEP: 1327700 | > loss_text_ce: 0.03165426477789879 (0.031012786816870235) | > loss_mel_ce: 2.9052624702453613 (nan) | > loss: 0.011654432862997055 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13732938097785985) | > loader_time: 0.0046 (0.004713289136697891)  --> TIME: 2025-02-26 17:04:10 -- STEP: 34920/36938 -- GLOBAL_STEP: 1327750 | > loss_text_ce: 0.0312491487711668 (0.031011724728546325) | > loss_mel_ce: 2.539829730987549 (nan) | > loss: 0.01020269375294447 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.137336570365867) | > loader_time: 0.0046 (0.004713417178303658)  --> TIME: 2025-02-26 17:04:20 -- STEP: 34970/36938 -- GLOBAL_STEP: 1327800 | > loss_text_ce: 0.04277196153998375 (0.03101038651367501) | > loss_mel_ce: 3.142847776412964 (nan) | > loss: 0.012641348876059055 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.13733906550239527) | > loader_time: 0.0049 (0.0047134346644265536)  --> TIME: 2025-02-26 17:04:29 -- STEP: 35020/36938 -- GLOBAL_STEP: 1327850 | > loss_text_ce: 0.03736424446105957 (0.03101126757884682) | > loss_mel_ce: 2.333282709121704 (nan) | > loss: 0.009407329373061657 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13734280414270686) | > loader_time: 0.0047 (0.004713469011861217)  --> TIME: 2025-02-26 17:04:39 -- STEP: 35070/36938 -- GLOBAL_STEP: 1327900 | > loss_text_ce: 0.02980334497988224 (0.0310092016280049) | > loss_mel_ce: 1.3615154027938843 (nan) | > loss: 0.005521106533706188 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13734134108312354) | > loader_time: 0.0047 (0.0047134371472791975)  --> TIME: 2025-02-26 17:04:48 -- STEP: 35120/36938 -- GLOBAL_STEP: 1327950 | > loss_text_ce: 0.040279023349285126 (0.0310100968871324) | > loss_mel_ce: 1.2214692831039429 (nan) | > loss: 0.005006938241422176 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13733544022455677) | > loader_time: 0.0045 (0.004713356040061748)  --> TIME: 2025-02-26 17:04:57 -- STEP: 35170/36938 -- GLOBAL_STEP: 1328000 | > loss_text_ce: 0.028565537184476852 (0.031010341140612924) | > loss_mel_ce: 3.2652645111083984 (nan) | > loss: 0.013070755638182163 (nan) | > current_lr: 5e-06 | > step_time: 0.1979 (0.137339620241766) | > loader_time: 0.0052 (0.004713384793975463)  --> TIME: 2025-02-26 17:05:07 -- STEP: 35220/36938 -- GLOBAL_STEP: 1328050 | > loss_text_ce: 0.02910938858985901 (0.031009516368560867) | > loss_mel_ce: 2.8720595836639404 (nan) | > loss: 0.01151257660239935 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.13734870631203167) | > loader_time: 0.0049 (0.004713379016726759)  --> TIME: 2025-02-26 17:05:17 -- STEP: 35270/36938 -- GLOBAL_STEP: 1328100 | > loss_text_ce: 0.03207698464393616 (0.031008383773068172) | > loss_mel_ce: 2.3792724609375 (nan) | > loss: 0.009568847715854645 (nan) | > current_lr: 5e-06 | > step_time: 0.172 (0.13735431888673164) | > loader_time: 0.005 (0.004713345249955846)  --> TIME: 2025-02-26 17:05:26 -- STEP: 35320/36938 -- GLOBAL_STEP: 1328150 | > loss_text_ce: 0.03137371316552162 (0.03100702825580098) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1567 (0.13735139274110977) | > loader_time: 0.0047 (0.004713275356983957)  --> TIME: 2025-02-26 17:05:35 -- STEP: 35370/36938 -- GLOBAL_STEP: 1328200 | > loss_text_ce: 0.031617458909749985 (0.03100718618075687) | > loss_mel_ce: 2.767702579498291 (nan) | > loss: 0.011108413338661194 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.13735254766039662) | > loader_time: 0.0047 (0.004713234929739174)  --> TIME: 2025-02-26 17:05:45 -- STEP: 35420/36938 -- GLOBAL_STEP: 1328250 | > loss_text_ce: 0.031223854050040245 (0.03100787479868753) | > loss_mel_ce: 2.4079389572143555 (nan) | > loss: 0.00967921782284975 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13736028840890346) | > loader_time: 0.0046 (0.0047133195003098175)  --> TIME: 2025-02-26 17:05:55 -- STEP: 35470/36938 -- GLOBAL_STEP: 1328300 | > loss_text_ce: 0.029637670144438744 (0.031007243851084217) | > loss_mel_ce: 2.390909194946289 (nan) | > loss: 0.00960534531623125 (nan) | > current_lr: 5e-06 | > step_time: 0.1626 (0.13737177200172018) | > loader_time: 0.0047 (0.004713381953329517)  --> TIME: 2025-02-26 17:06:04 -- STEP: 35520/36938 -- GLOBAL_STEP: 1328350 | > loss_text_ce: 0.03858644515275955 (0.031007709103541527) | > loss_mel_ce: 3.0465564727783203 (nan) | > loss: 0.012242631055414677 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13736867450915832) | > loader_time: 0.0053 (0.004713396721326558)  --> TIME: 2025-02-26 17:06:14 -- STEP: 35570/36938 -- GLOBAL_STEP: 1328400 | > loss_text_ce: 0.03752327710390091 (0.031007856906331198) | > loss_mel_ce: 2.510219097137451 (nan) | > loss: 0.010110089555382729 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.13737249728967552) | > loader_time: 0.0047 (0.004713466370544783)  --> TIME: 2025-02-26 17:06:23 -- STEP: 35620/36938 -- GLOBAL_STEP: 1328450 | > loss_text_ce: 0.0313471220433712 (0.03100695428790149) | > loss_mel_ce: 3.0404953956604004 (nan) | > loss: 0.012189852073788643 (nan) | > current_lr: 5e-06 | > step_time: 0.184 (0.13736553319312902) | > loader_time: 0.005 (0.004713549177543564)  --> TIME: 2025-02-26 17:06:32 -- STEP: 35670/36938 -- GLOBAL_STEP: 1328500 | > loss_text_ce: 0.01993483118712902 (0.03100765118296309) | > loss_mel_ce: 1.4750539064407349 (nan) | > loss: 0.0059324949979782104 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1373654251242988) | > loader_time: 0.0046 (0.004713632394059691)  --> TIME: 2025-02-26 17:06:42 -- STEP: 35720/36938 -- GLOBAL_STEP: 1328550 | > loss_text_ce: 0.037469834089279175 (0.03100687154178421) | > loss_mel_ce: 2.2985970973968506 (nan) | > loss: 0.009270107373595238 (nan) | > current_lr: 5e-06 | > step_time: 0.1966 (0.13737067046918175) | > loader_time: 0.0051 (0.004713800566193779)  --> TIME: 2025-02-26 17:06:51 -- STEP: 35770/36938 -- GLOBAL_STEP: 1328600 | > loss_text_ce: 0.026817074045538902 (0.031006101210429005) | > loss_mel_ce: 2.526099681854248 (nan) | > loss: 0.010130622424185276 (nan) | > current_lr: 5e-06 | > step_time: 0.1261 (0.13736692627348587) | > loader_time: 0.0046 (0.004713768548461564)  --> TIME: 2025-02-26 17:07:00 -- STEP: 35820/36938 -- GLOBAL_STEP: 1328650 | > loss_text_ce: 0.04230501502752304 (0.03100505634602305) | > loss_mel_ce: 1.4316762685775757 (nan) | > loss: 0.005849132314324379 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1373659469241321) | > loader_time: 0.0052 (0.0047137826531376764)  --> TIME: 2025-02-26 17:07:10 -- STEP: 35870/36938 -- GLOBAL_STEP: 1328700 | > loss_text_ce: 0.026073994114995003 (0.031007220297685743) | > loss_mel_ce: 2.5833325386047363 (nan) | > loss: 0.010354788042604923 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13737315528067578) | > loader_time: 0.0045 (0.004713805904287409)  --> TIME: 2025-02-26 17:07:19 -- STEP: 35920/36938 -- GLOBAL_STEP: 1328750 | > loss_text_ce: 0.037605054676532745 (0.03100714779453495) | > loss_mel_ce: 2.856602907180786 (nan) | > loss: 0.01148495264351368 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.13737591167866062) | > loader_time: 0.0054 (0.004713941231071295)  --> TIME: 2025-02-26 17:07:29 -- STEP: 35970/36938 -- GLOBAL_STEP: 1328800 | > loss_text_ce: 0.03306487202644348 (0.031010343529289698) | > loss_mel_ce: 2.3669865131378174 (nan) | > loss: 0.009524013847112656 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13737748997523452) | > loader_time: 0.0047 (0.004713934369971437)  --> TIME: 2025-02-26 17:07:38 -- STEP: 36020/36938 -- GLOBAL_STEP: 1328850 | > loss_text_ce: 0.02989736944437027 (0.031010481453943337) | > loss_mel_ce: 3.101579189300537 (nan) | > loss: 0.012426495552062988 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.13737712421395873) | > loader_time: 0.0047 (0.0047139326179775586)  --> TIME: 2025-02-26 17:07:48 -- STEP: 36070/36938 -- GLOBAL_STEP: 1328900 | > loss_text_ce: 0.02560552768409252 (0.031009885877676027) | > loss_mel_ce: 2.204833984375 (nan) | > loss: 0.008850950747728348 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13737950020957276) | > loader_time: 0.0046 (0.004713879545078016)  --> TIME: 2025-02-26 17:07:57 -- STEP: 36120/36938 -- GLOBAL_STEP: 1328950 | > loss_text_ce: 0.02678893506526947 (0.03100850137915196) | > loss_mel_ce: 1.6981472969055176 (nan) | > loss: 0.006844985298812389 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13737517178124448) | > loader_time: 0.0052 (0.004713866329140319)  --> TIME: 2025-02-26 17:08:06 -- STEP: 36170/36938 -- GLOBAL_STEP: 1329000 | > loss_text_ce: 0.030822856351733208 (0.031007824865467732) | > loss_mel_ce: 2.731142282485962 (nan) | > loss: 0.010960179381072521 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13737575863486554) | > loader_time: 0.0047 (0.004713920746714435)  --> TIME: 2025-02-26 17:08:16 -- STEP: 36220/36938 -- GLOBAL_STEP: 1329050 | > loss_text_ce: 0.03030577301979065 (0.031008097210943737) | > loss_mel_ce: 2.8293120861053467 (nan) | > loss: 0.011347690597176552 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13737807838644084) | > loader_time: 0.0046 (0.0047138189558558995)  --> TIME: 2025-02-26 17:08:25 -- STEP: 36270/36938 -- GLOBAL_STEP: 1329100 | > loss_text_ce: 0.031145431101322174 (0.03100697181371421) | > loss_mel_ce: 2.584174633026123 (nan) | > loss: 0.01037825457751751 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13737566289672795) | > loader_time: 0.0045 (0.004713907161625358)  --> TIME: 2025-02-26 17:08:35 -- STEP: 36320/36938 -- GLOBAL_STEP: 1329150 | > loss_text_ce: 0.036237675696611404 (0.03100750657991611) | > loss_mel_ce: 2.350764751434326 (nan) | > loss: 0.009472232311964035 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13737651871164283) | > loader_time: 0.0045 (0.0047139623354185076)  --> TIME: 2025-02-26 17:08:44 -- STEP: 36370/36938 -- GLOBAL_STEP: 1329200 | > loss_text_ce: 0.03209925442934036 (0.0310089299235706) | > loss_mel_ce: 2.396763324737549 (nan) | > loss: 0.009638343937695026 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13737354395905277) | > loader_time: 0.0046 (0.004713898220329724)  --> TIME: 2025-02-26 17:08:53 -- STEP: 36420/36938 -- GLOBAL_STEP: 1329250 | > loss_text_ce: 0.03661195933818817 (0.031010835107790798) | > loss_mel_ce: 0.011240641586482525 (nan) | > loss: 0.00018989128875546157 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1373722776876951) | > loader_time: 0.0046 (0.004713872413855221)  --> TIME: 2025-02-26 17:09:02 -- STEP: 36470/36938 -- GLOBAL_STEP: 1329300 | > loss_text_ce: 0.033010084182024 (0.03101008458016785) | > loss_mel_ce: 2.9389071464538574 (nan) | > loss: 0.01179332286119461 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.1373698068181535) | > loader_time: 0.0047 (0.004713773341711047)  --> TIME: 2025-02-26 17:09:12 -- STEP: 36520/36938 -- GLOBAL_STEP: 1329350 | > loss_text_ce: 0.03054368495941162 (0.031011084320499517) | > loss_mel_ce: 2.847698450088501 (nan) | > loss: 0.011421595700085163 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13736794770481886) | > loader_time: 0.0046 (0.004713683112688691)  --> TIME: 2025-02-26 17:09:21 -- STEP: 36570/36938 -- GLOBAL_STEP: 1329400 | > loss_text_ce: 0.0343313030898571 (0.031011243998795027) | > loss_mel_ce: 2.7769150733947754 (nan) | > loss: 0.011155740357935429 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13736916688872464) | > loader_time: 0.0048 (0.00471376502386756)  --> TIME: 2025-02-26 17:09:30 -- STEP: 36620/36938 -- GLOBAL_STEP: 1329450 | > loss_text_ce: 0.03723381459712982 (0.03101146967924169) | > loss_mel_ce: 2.1793718338012695 (nan) | > loss: 0.008796054869890213 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.137370472583269) | > loader_time: 0.0048 (0.004713922605561413)  --> TIME: 2025-02-26 17:09:40 -- STEP: 36670/36938 -- GLOBAL_STEP: 1329500 | > loss_text_ce: 0.033617231994867325 (0.031010379337339695) | > loss_mel_ce: 2.5505495071411133 (nan) | > loss: 0.01025463081896305 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.1373638234542879) | > loader_time: 0.0046 (0.00471387432224263)  --> TIME: 2025-02-26 17:09:49 -- STEP: 36720/36938 -- GLOBAL_STEP: 1329550 | > loss_text_ce: 0.03299080580472946 (0.031009016382116884) | > loss_mel_ce: 3.0541555881500244 (nan) | > loss: 0.012250580824911594 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.1373616795004826) | > loader_time: 0.0066 (0.004714057505260631)  --> TIME: 2025-02-26 17:09:58 -- STEP: 36770/36938 -- GLOBAL_STEP: 1329600 | > loss_text_ce: 0.0359233058989048 (0.031009817945279298) | > loss_mel_ce: 3.126678705215454 (nan) | > loss: 0.012550008483231068 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.13736289884527153) | > loader_time: 0.0048 (0.0047141768149717555)  --> TIME: 2025-02-26 17:10:08 -- STEP: 36820/36938 -- GLOBAL_STEP: 1329650 | > loss_text_ce: 0.03218277171254158 (0.03101144368067993) | > loss_mel_ce: 2.607917308807373 (nan) | > loss: 0.010476588271558285 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.137361920779971) | > loader_time: 0.0046 (0.004714104016794589)  --> TIME: 2025-02-26 17:10:17 -- STEP: 36870/36938 -- GLOBAL_STEP: 1329700 | > loss_text_ce: 0.029710518196225166 (0.031011583140081515) | > loss_mel_ce: 2.9470245838165283 (nan) | > loss: 0.011812441982328892 (nan) | > current_lr: 5e-06 | > step_time: 0.161 (0.13736543904354262) | > loader_time: 0.0047 (0.004714062752554282)  --> TIME: 2025-02-26 17:10:27 -- STEP: 36920/36938 -- GLOBAL_STEP: 1329750 | > loss_text_ce: 0.022003866732120514 (0.031012602891777527) | > loss_mel_ce: 2.5823516845703125 (nan) | > loss: 0.010334745049476624 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13736908408699003) | > loader_time: 0.0045 (0.004714094817573831)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.020602845909571905 (+1.2978468791099984e-05) | > avg_loss_text_ce: 0.030306922400941943 (-0.00012676806032362736) | > avg_loss_mel_ce: 2.453815608036397 (-0.0013367849411376476) | > avg_loss: 2.4841225297734284 (-0.0014635566437624092) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_1329768.pth  > EPOCH: 37/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-26 17:10:55)   --> TIME: 2025-02-26 17:11:01 -- STEP: 32/36938 -- GLOBAL_STEP: 1329800 | > loss_text_ce: 0.02028496004641056 (0.03084362338995561) | > loss_mel_ce: 0.03099822998046875 (2.429985962808132) | > loss: 0.0002035047218669206 (0.009765197305569016) | > current_lr: 5e-06 | > step_time: 0.117 (0.1382370740175247) | > loader_time: 0.0046 (0.0047065988183021545)  --> TIME: 2025-02-26 17:11:11 -- STEP: 82/36938 -- GLOBAL_STEP: 1329850 | > loss_text_ce: 0.0420110784471035 (0.03166071795744865) | > loss_mel_ce: 1.0706055164337158 (nan) | > loss: 0.004415145143866539 (nan) | > current_lr: 5e-06 | > step_time: 0.1074 (0.13618519538786353) | > loader_time: 0.0048 (0.004734300985568909)  --> TIME: 2025-02-26 17:11:20 -- STEP: 132/36938 -- GLOBAL_STEP: 1329900 | > loss_text_ce: 0.029861997812986374 (0.03198157164804414) | > loss_mel_ce: 2.8391940593719482 (nan) | > loss: 0.011385143734514713 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13438646721117414) | > loader_time: 0.0045 (0.004755184505925036)  --> TIME: 2025-02-26 17:11:29 -- STEP: 182/36938 -- GLOBAL_STEP: 1329950 | > loss_text_ce: 0.027037467807531357 (0.03176429033647854) | > loss_mel_ce: 3.316701889038086 (nan) | > loss: 0.013268806971609592 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.1356086521358279) | > loader_time: 0.0048 (0.004735924385406161)  --> TIME: 2025-02-26 17:11:38 -- STEP: 232/36938 -- GLOBAL_STEP: 1330000 | > loss_text_ce: 0.036943912506103516 (0.03176645715399804) | > loss_mel_ce: 2.2361338138580322 (nan) | > loss: 0.009020150639116764 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13493127452916104) | > loader_time: 0.0046 (0.004727829119254804) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1330000.pth  --> TIME: 2025-02-26 17:11:54 -- STEP: 282/36938 -- GLOBAL_STEP: 1330050 | > loss_text_ce: 0.03244635835289955 (0.03150641300603217) | > loss_mel_ce: 2.9266912937164307 (nan) | > loss: 0.011742610484361649 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.13665429159259118) | > loader_time: 0.0047 (0.0047273255409078395)  --> TIME: 2025-02-26 17:12:03 -- STEP: 332/36938 -- GLOBAL_STEP: 1330100 | > loss_text_ce: 0.03047811985015869 (0.03131698104496821) | > loss_mel_ce: 2.9937551021575928 (nan) | > loss: 0.01200092677026987 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.137230482446142) | > loader_time: 0.0049 (0.004724711538797401)  --> TIME: 2025-02-26 17:12:12 -- STEP: 382/36938 -- GLOBAL_STEP: 1330150 | > loss_text_ce: 0.03763066977262497 (0.03134470129707406) | > loss_mel_ce: 2.95145320892334 (nan) | > loss: 0.01186144445091486 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.1366070359165133) | > loader_time: 0.0045 (0.004716198481814394)  --> TIME: 2025-02-26 17:12:22 -- STEP: 432/36938 -- GLOBAL_STEP: 1330200 | > loss_text_ce: 0.04365887865424156 (0.031356273697585674) | > loss_mel_ce: 2.7277848720550537 (nan) | > loss: 0.010997793637216091 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.13648337622483572) | > loader_time: 0.0047 (0.004717261702926064)  --> TIME: 2025-02-26 17:12:31 -- STEP: 482/36938 -- GLOBAL_STEP: 1330250 | > loss_text_ce: 0.04168197512626648 (0.031371655553208345) | > loss_mel_ce: 2.818455696105957 (nan) | > loss: 0.011349753476679325 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.136177795556571) | > loader_time: 0.0045 (0.004715741917305462)  --> TIME: 2025-02-26 17:12:40 -- STEP: 532/36938 -- GLOBAL_STEP: 1330300 | > loss_text_ce: 0.044614527374506 (0.031495025132859754) | > loss_mel_ce: 2.797595500946045 (nan) | > loss: 0.011278611607849598 (nan) | > current_lr: 5e-06 | > step_time: 0.1619 (0.1361632593592307) | > loader_time: 0.0048 (0.004709195373649879)  --> TIME: 2025-02-26 17:12:49 -- STEP: 582/36938 -- GLOBAL_STEP: 1330350 | > loss_text_ce: 0.025930305942893028 (0.031342275299874336) | > loss_mel_ce: 2.9482429027557373 (nan) | > loss: 0.01180227566510439 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.13590340311174956) | > loader_time: 0.0047 (0.00470372573616578)  --> TIME: 2025-02-26 17:12:59 -- STEP: 632/36938 -- GLOBAL_STEP: 1330400 | > loss_text_ce: 0.02679101563990116 (0.031376398407229306) | > loss_mel_ce: 2.9774160385131836 (nan) | > loss: 0.011921457946300507 (nan) | > current_lr: 5e-06 | > step_time: 0.1774 (0.1359324236459371) | > loader_time: 0.0049 (0.004701228458670118)  --> TIME: 2025-02-26 17:13:08 -- STEP: 682/36938 -- GLOBAL_STEP: 1330450 | > loss_text_ce: 0.03458227217197418 (0.03129337180018031) | > loss_mel_ce: 3.2857162952423096 (nan) | > loss: 0.013175789266824722 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.1360807873286811) | > loader_time: 0.0047 (0.004699349053793984)  --> TIME: 2025-02-26 17:13:17 -- STEP: 732/36938 -- GLOBAL_STEP: 1330500 | > loss_text_ce: 0.030826441943645477 (0.031216350260643615) | > loss_mel_ce: 3.0609703063964844 (nan) | > loss: 0.0122690349817276 (nan) | > current_lr: 5e-06 | > step_time: 0.1871 (0.1363868162931641) | > loader_time: 0.0048 (0.004697287017530431)  --> TIME: 2025-02-26 17:13:27 -- STEP: 782/36938 -- GLOBAL_STEP: 1330550 | > loss_text_ce: 0.027051905170083046 (0.031134284564706) | > loss_mel_ce: 2.29769229888916 (nan) | > loss: 0.009225175715982914 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.1364981561060758) | > loader_time: 0.0045 (0.004692954480495598)  --> TIME: 2025-02-26 17:13:36 -- STEP: 832/36938 -- GLOBAL_STEP: 1330600 | > loss_text_ce: 0.028058065101504326 (0.0311161453461794) | > loss_mel_ce: 2.8364057540893555 (nan) | > loss: 0.011366920545697212 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13650265106788073) | > loader_time: 0.0046 (0.004689609775176408)  --> TIME: 2025-02-26 17:13:45 -- STEP: 882/36938 -- GLOBAL_STEP: 1330650 | > loss_text_ce: 0.025747336447238922 (0.031165642179168618) | > loss_mel_ce: 2.048196315765381 (nan) | > loss: 0.008229935541749 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13636826225419174) | > loader_time: 0.0047 (0.0046851186254938115)  --> TIME: 2025-02-26 17:13:55 -- STEP: 932/36938 -- GLOBAL_STEP: 1330700 | > loss_text_ce: 0.02249784953892231 (0.03111820161450543) | > loss_mel_ce: 2.599785804748535 (nan) | > loss: 0.010405887849628925 (nan) | > current_lr: 5e-06 | > step_time: 0.1465 (0.13629978562629286) | > loader_time: 0.0046 (0.00468242373077654)  --> TIME: 2025-02-26 17:14:04 -- STEP: 982/36938 -- GLOBAL_STEP: 1330750 | > loss_text_ce: 0.029903298243880272 (0.03104042931973874) | > loss_mel_ce: 2.7749698162078857 (nan) | > loss: 0.011130450293421745 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1360470119658897) | > loader_time: 0.0045 (0.004677880320189678)  --> TIME: 2025-02-26 17:14:13 -- STEP: 1032/36938 -- GLOBAL_STEP: 1330800 | > loss_text_ce: 0.03002595342695713 (0.0310200724025192) | > loss_mel_ce: 0.0634765625 (nan) | > loss: 0.00037104173679836094 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.1363003699816476) | > loader_time: 0.0047 (0.0046767444573631535)  --> TIME: 2025-02-26 17:14:23 -- STEP: 1082/36938 -- GLOBAL_STEP: 1330850 | > loss_text_ce: 0.02905932441353798 (0.03105816667186796) | > loss_mel_ce: 3.075754404067993 (nan) | > loss: 0.012320690788328648 (nan) | > current_lr: 5e-06 | > step_time: 0.1464 (0.1362527875494826) | > loader_time: 0.0046 (0.004675941634751067)  --> TIME: 2025-02-26 17:14:32 -- STEP: 1132/36938 -- GLOBAL_STEP: 1330900 | > loss_text_ce: 0.028837911784648895 (0.031077115447671808) | > loss_mel_ce: 2.369266986846924 (nan) | > loss: 0.009516289457678795 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.1363337520154543) | > loader_time: 0.0046 (0.00467497089305109)  --> TIME: 2025-02-26 17:14:42 -- STEP: 1182/36938 -- GLOBAL_STEP: 1330950 | > loss_text_ce: 0.032863084226846695 (0.031046809458414922) | > loss_mel_ce: 2.7470703125 (nan) | > loss: 0.011031482368707657 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1365492896581867) | > loader_time: 0.0045 (0.004675186225000365)  --> TIME: 2025-02-26 17:14:51 -- STEP: 1232/36938 -- GLOBAL_STEP: 1331000 | > loss_text_ce: 0.02686629258096218 (0.031033548505299477) | > loss_mel_ce: 2.2636802196502686 (nan) | > loss: 0.009089470840990543 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13667074871527699) | > loader_time: 0.0047 (0.004680793006698798)  --> TIME: 2025-02-26 17:15:01 -- STEP: 1282/36938 -- GLOBAL_STEP: 1331050 | > loss_text_ce: 0.02656266652047634 (0.03104419924766291) | > loss_mel_ce: 2.8815155029296875 (nan) | > loss: 0.011539993807673454 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13688003589136188) | > loader_time: 0.0047 (0.004682626255588851)  --> TIME: 2025-02-26 17:15:10 -- STEP: 1332/36938 -- GLOBAL_STEP: 1331100 | > loss_text_ce: 0.0323956273496151 (0.031056319641693312) | > loss_mel_ce: 2.807417154312134 (nan) | > loss: 0.011269099079072475 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.1369762594277438) | > loader_time: 0.0046 (0.0046827218554041355)  --> TIME: 2025-02-26 17:15:20 -- STEP: 1382/36938 -- GLOBAL_STEP: 1331150 | > loss_text_ce: 0.0360315665602684 (0.031055114663296347) | > loss_mel_ce: 2.8278164863586426 (nan) | > loss: 0.011364476755261421 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.13696350582084188) | > loader_time: 0.0048 (0.004683375013547415)  --> TIME: 2025-02-26 17:15:29 -- STEP: 1432/36938 -- GLOBAL_STEP: 1331200 | > loss_text_ce: 0.023600080981850624 (0.03103536726417822) | > loss_mel_ce: 1.697149634361267 (nan) | > loss: 0.0068283723667263985 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.13690457214190346) | > loader_time: 0.0045 (0.004681835127942384)  --> TIME: 2025-02-26 17:15:39 -- STEP: 1482/36938 -- GLOBAL_STEP: 1331250 | > loss_text_ce: 0.027116809040308 (0.03104355252459746) | > loss_mel_ce: 3.0163183212280273 (nan) | > loss: 0.01207712385803461 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.13726855257423012) | > loader_time: 0.0047 (0.00468253397587661)  --> TIME: 2025-02-26 17:15:48 -- STEP: 1532/36938 -- GLOBAL_STEP: 1331300 | > loss_text_ce: 0.01774015463888645 (0.030985514987162843) | > loss_mel_ce: 2.797013759613037 (nan) | > loss: 0.011169658973813057 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13726870524976667) | > loader_time: 0.0046 (0.004682347917681258)  --> TIME: 2025-02-26 17:15:58 -- STEP: 1582/36938 -- GLOBAL_STEP: 1331350 | > loss_text_ce: 0.03395811840891838 (0.030949639658621422) | > loss_mel_ce: 2.7575814723968506 (nan) | > loss: 0.011077539063990116 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13722302865439537) | > loader_time: 0.0045 (0.004679960487163165)  --> TIME: 2025-02-26 17:16:07 -- STEP: 1632/36938 -- GLOBAL_STEP: 1331400 | > loss_text_ce: 0.034549761563539505 (0.03095327752503092) | > loss_mel_ce: 1.4301551580429077 (nan) | > loss: 0.005812321323901415 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.1371450925282406) | > loader_time: 0.0046 (0.004679300621444093)  --> TIME: 2025-02-26 17:16:16 -- STEP: 1682/36938 -- GLOBAL_STEP: 1331450 | > loss_text_ce: 0.03111332841217518 (0.030973093820609574) | > loss_mel_ce: 2.5495266914367676 (nan) | > loss: 0.010240635834634304 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13717069282826583) | > loader_time: 0.0045 (0.004677654163165541)  --> TIME: 2025-02-26 17:16:26 -- STEP: 1732/36938 -- GLOBAL_STEP: 1331500 | > loss_text_ce: 0.03711395338177681 (0.031029349992559328) | > loss_mel_ce: 3.1669981479644775 (nan) | > loss: 0.012714731507003307 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13715230537617193) | > loader_time: 0.0046 (0.004676284470679291)  --> TIME: 2025-02-26 17:16:35 -- STEP: 1782/36938 -- GLOBAL_STEP: 1331550 | > loss_text_ce: 0.025818826630711555 (0.03101138034772743) | > loss_mel_ce: 2.867577314376831 (nan) | > loss: 0.011481731198728085 (nan) | > current_lr: 5e-06 | > step_time: 0.1828 (0.13727857488574408) | > loader_time: 0.0048 (0.004675819148653406)  --> TIME: 2025-02-26 17:16:45 -- STEP: 1832/36938 -- GLOBAL_STEP: 1331600 | > loss_text_ce: 0.028359949588775635 (0.03099030011545005) | > loss_mel_ce: 1.9702101945877075 (nan) | > loss: 0.007930834777653217 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13728482988724022) | > loader_time: 0.0045 (0.004674696896274005)  --> TIME: 2025-02-26 17:16:54 -- STEP: 1882/36938 -- GLOBAL_STEP: 1331650 | > loss_text_ce: 0.029239792376756668 (0.030980074245732094) | > loss_mel_ce: 3.637586832046509 (nan) | > loss: 0.014550900086760521 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13719377190856452) | > loader_time: 0.0046 (0.0046733752067233385)  --> TIME: 2025-02-26 17:17:03 -- STEP: 1932/36938 -- GLOBAL_STEP: 1331700 | > loss_text_ce: 0.030111195519566536 (0.030999651742717314) | > loss_mel_ce: 2.7901973724365234 (nan) | > loss: 0.011191701516509056 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1371611759035731) | > loader_time: 0.0045 (0.004672456602132087)  --> TIME: 2025-02-26 17:17:12 -- STEP: 1982/36938 -- GLOBAL_STEP: 1331750 | > loss_text_ce: 0.03716101124882698 (0.030985758152125946) | > loss_mel_ce: 2.1400227546691895 (nan) | > loss: 0.008639618754386902 (nan) | > current_lr: 5e-06 | > step_time: 0.1614 (0.13712260785665817) | > loader_time: 0.0047 (0.004670986734653928)  --> TIME: 2025-02-26 17:17:22 -- STEP: 2032/36938 -- GLOBAL_STEP: 1331800 | > loss_text_ce: 0.03585297614336014 (0.030954756150682144) | > loss_mel_ce: 2.3001251220703125 (nan) | > loss: 0.009269754402339458 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13719445302730476) | > loader_time: 0.0046 (0.004670086808092002)  --> TIME: 2025-02-26 17:17:31 -- STEP: 2082/36938 -- GLOBAL_STEP: 1331850 | > loss_text_ce: 0.032337043434381485 (0.030968350436141236) | > loss_mel_ce: 2.6309289932250977 (nan) | > loss: 0.010568516328930855 (nan) | > current_lr: 5e-06 | > step_time: 0.1517 (0.13706929310368995) | > loader_time: 0.0047 (0.004669297898079077)  --> TIME: 2025-02-26 17:17:41 -- STEP: 2132/36938 -- GLOBAL_STEP: 1331900 | > loss_text_ce: 0.026377370581030846 (0.030996218521154955) | > loss_mel_ce: 3.158336639404297 (nan) | > loss: 0.012637754902243614 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.13719793455238422) | > loader_time: 0.0047 (0.004669563184312305)  --> TIME: 2025-02-26 17:17:50 -- STEP: 2182/36938 -- GLOBAL_STEP: 1331950 | > loss_text_ce: 0.027053862810134888 (0.030996827103067823) | > loss_mel_ce: 2.2013871669769287 (nan) | > loss: 0.008843020536005497 (nan) | > current_lr: 5e-06 | > step_time: 0.1937 (0.1372384766063813) | > loader_time: 0.0048 (0.00466894458784737)  --> TIME: 2025-02-26 17:18:00 -- STEP: 2232/36938 -- GLOBAL_STEP: 1332000 | > loss_text_ce: 0.031145738437771797 (0.03095938543176552) | > loss_mel_ce: 2.9666550159454346 (nan) | > loss: 0.011896035633981228 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13731603895891534) | > loader_time: 0.0045 (0.004668694990937419)  --> TIME: 2025-02-26 17:18:09 -- STEP: 2282/36938 -- GLOBAL_STEP: 1332050 | > loss_text_ce: 0.03954894840717316 (0.03095690797611183) | > loss_mel_ce: 2.6867754459381104 (nan) | > loss: 0.01081874780356884 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13720209989288631) | > loader_time: 0.006 (0.004668672600929193)  --> TIME: 2025-02-26 17:18:18 -- STEP: 2332/36938 -- GLOBAL_STEP: 1332100 | > loss_text_ce: 0.032018471509218216 (0.03095716685751584) | > loss_mel_ce: 2.921278953552246 (nan) | > loss: 0.011719434522092342 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13723745528052644) | > loader_time: 0.0048 (0.004668439641007012)  --> TIME: 2025-02-26 17:18:28 -- STEP: 2382/36938 -- GLOBAL_STEP: 1332150 | > loss_text_ce: 0.028382863849401474 (0.030945784489348167) | > loss_mel_ce: 2.082317590713501 (nan) | > loss: 0.008375795558094978 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13727854531317768) | > loader_time: 0.0045 (0.004668267207622123)  --> TIME: 2025-02-26 17:18:37 -- STEP: 2432/36938 -- GLOBAL_STEP: 1332200 | > loss_text_ce: 0.031790297478437424 (0.030978892887345098) | > loss_mel_ce: 2.692392110824585 (nan) | > loss: 0.0108102485537529 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1372585363293949) | > loader_time: 0.0046 (0.004668145587569785)  --> TIME: 2025-02-26 17:18:46 -- STEP: 2482/36938 -- GLOBAL_STEP: 1332250 | > loss_text_ce: 0.036971237510442734 (0.030978436343518958) | > loss_mel_ce: 3.3138771057128906 (nan) | > loss: 0.013297018595039845 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13725435493647506) | > loader_time: 0.0046 (0.004667883722365234)  --> TIME: 2025-02-26 17:18:56 -- STEP: 2532/36938 -- GLOBAL_STEP: 1332300 | > loss_text_ce: 0.025991661474108696 (0.03096265267649898) | > loss_mel_ce: 2.235968828201294 (nan) | > loss: 0.008976033888757229 (nan) | > current_lr: 5e-06 | > step_time: 0.1447 (0.13720889481322998) | > loader_time: 0.0048 (0.004667514402531142)  --> TIME: 2025-02-26 17:19:05 -- STEP: 2582/36938 -- GLOBAL_STEP: 1332350 | > loss_text_ce: 0.02627340890467167 (0.030946507495932574) | > loss_mel_ce: 2.674675703048706 (nan) | > loss: 0.010718053206801414 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.1371746204510664) | > loader_time: 0.0046 (0.004666830383466244)  --> TIME: 2025-02-26 17:19:14 -- STEP: 2632/36938 -- GLOBAL_STEP: 1332400 | > loss_text_ce: 0.01595013216137886 (0.030925194784860187) | > loss_mel_ce: 1.9316610097885132 (nan) | > loss: 0.007728615775704384 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.13719758681250896) | > loader_time: 0.0046 (0.004667238623900245)  --> TIME: 2025-02-26 17:19:24 -- STEP: 2682/36938 -- GLOBAL_STEP: 1332450 | > loss_text_ce: 0.02681608870625496 (0.030917551699403526) | > loss_mel_ce: 2.4734456539154053 (nan) | > loss: 0.009921674616634846 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.1372080337281193) | > loader_time: 0.0046 (0.004668118258539543)  --> TIME: 2025-02-26 17:19:33 -- STEP: 2732/36938 -- GLOBAL_STEP: 1332500 | > loss_text_ce: 0.027873555198311806 (0.030920281695027457) | > loss_mel_ce: 3.0948362350463867 (nan) | > loss: 0.012391705997288227 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.13714912424673953) | > loader_time: 0.0046 (0.004669366434412846)  --> TIME: 2025-02-26 17:19:42 -- STEP: 2782/36938 -- GLOBAL_STEP: 1332550 | > loss_text_ce: 0.029515067115426064 (0.030943756748665736) | > loss_mel_ce: 1.2266387939453125 (nan) | > loss: 0.00498473783954978 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13714450379226967) | > loader_time: 0.0046 (0.004669001254994257)  --> TIME: 2025-02-26 17:19:51 -- STEP: 2832/36938 -- GLOBAL_STEP: 1332600 | > loss_text_ce: 0.03246358409523964 (0.030946891921596064) | > loss_mel_ce: 2.622702121734619 (nan) | > loss: 0.010536372661590576 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13713344047635267) | > loader_time: 0.0046 (0.004668261119201349)  --> TIME: 2025-02-26 17:20:01 -- STEP: 2882/36938 -- GLOBAL_STEP: 1332650 | > loss_text_ce: 0.01979220099747181 (0.030927313658771396) | > loss_mel_ce: 2.769428253173828 (nan) | > loss: 0.01106833666563034 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.1372017102635295) | > loader_time: 0.0046 (0.004667759189830717)  --> TIME: 2025-02-26 17:20:10 -- STEP: 2932/36938 -- GLOBAL_STEP: 1332700 | > loss_text_ce: 0.03821457177400589 (0.0309336061820027) | > loss_mel_ce: 3.026172161102295 (nan) | > loss: 0.012160265818238258 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.1372449958308168) | > loader_time: 0.0049 (0.004669052923522562)  --> TIME: 2025-02-26 17:20:20 -- STEP: 2982/36938 -- GLOBAL_STEP: 1332750 | > loss_text_ce: 0.03207302838563919 (0.030919173298594758) | > loss_mel_ce: 2.6779847145080566 (nan) | > loss: 0.010754197835922241 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.1372940591483368) | > loader_time: 0.0046 (0.004668986453536676)  --> TIME: 2025-02-26 17:20:29 -- STEP: 3032/36938 -- GLOBAL_STEP: 1332800 | > loss_text_ce: 0.025599779561161995 (0.030915976300501413) | > loss_mel_ce: 2.646531105041504 (nan) | > loss: 0.01060369424521923 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13728131404967284) | > loader_time: 0.0052 (0.0046687518543806926)  --> TIME: 2025-02-26 17:20:39 -- STEP: 3082/36938 -- GLOBAL_STEP: 1332850 | > loss_text_ce: 0.03308518975973129 (0.03092696742287013) | > loss_mel_ce: 2.940345048904419 (nan) | > loss: 0.011799327097833157 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13725618137159096) | > loader_time: 0.0047 (0.0046681198959300915)  --> TIME: 2025-02-26 17:20:48 -- STEP: 3132/36938 -- GLOBAL_STEP: 1332900 | > loss_text_ce: 0.024938693270087242 (0.030937796304570952) | > loss_mel_ce: 0.7811676263809204 (nan) | > loss: 0.0031988348346203566 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13721359461203378) | > loader_time: 0.0045 (0.004668136437733952)  --> TIME: 2025-02-26 17:20:57 -- STEP: 3182/36938 -- GLOBAL_STEP: 1332950 | > loss_text_ce: 0.024549616500735283 (0.03093371796114388) | > loss_mel_ce: 2.8768527507781982 (nan) | > loss: 0.011513502337038517 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.13721386843848565) | > loader_time: 0.0048 (0.004667779846359227)  --> TIME: 2025-02-26 17:21:07 -- STEP: 3232/36938 -- GLOBAL_STEP: 1333000 | > loss_text_ce: 0.03604128956794739 (0.030932146339506684) | > loss_mel_ce: 2.808558940887451 (nan) | > loss: 0.011288097128272057 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.1372519759553493) | > loader_time: 0.0046 (0.004667475598283317)  --> TIME: 2025-02-26 17:21:16 -- STEP: 3282/36938 -- GLOBAL_STEP: 1333050 | > loss_text_ce: 0.030280649662017822 (0.030923424166327002) | > loss_mel_ce: 2.995861530303955 (nan) | > loss: 0.012008501216769218 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.13731774006726755) | > loader_time: 0.0045 (0.004667561494454153)  --> TIME: 2025-02-26 17:21:25 -- STEP: 3332/36938 -- GLOBAL_STEP: 1333100 | > loss_text_ce: 0.027421874925494194 (0.030939887944409918) | > loss_mel_ce: 3.215536594390869 (nan) | > loss: 0.012868884019553661 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.13726854903929805) | > loader_time: 0.0045 (0.004666692497922004)  --> TIME: 2025-02-26 17:21:35 -- STEP: 3382/36938 -- GLOBAL_STEP: 1333150 | > loss_text_ce: 0.02762380801141262 (0.030946817110270034) | > loss_mel_ce: 2.8612682819366455 (nan) | > loss: 0.011463858187198639 (nan) | > current_lr: 5e-06 | > step_time: 0.1438 (0.13726773528500047) | > loader_time: 0.0047 (0.0046667012147548)  --> TIME: 2025-02-26 17:21:44 -- STEP: 3432/36938 -- GLOBAL_STEP: 1333200 | > loss_text_ce: 0.03836783021688461 (0.03099068538459465) | > loss_mel_ce: 0.19131602346897125 (nan) | > loss: 0.0009114439017139375 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.13733629719082857) | > loader_time: 0.0046 (0.00466659456699877)  --> TIME: 2025-02-26 17:21:53 -- STEP: 3482/36938 -- GLOBAL_STEP: 1333250 | > loss_text_ce: 0.02840254083275795 (0.030964497895135678) | > loss_mel_ce: 3.2132043838500977 (nan) | > loss: 0.01286352053284645 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.1372896852608319) | > loader_time: 0.0046 (0.004666048723140838)  --> TIME: 2025-02-26 17:22:03 -- STEP: 3532/36938 -- GLOBAL_STEP: 1333300 | > loss_text_ce: 0.022146007046103477 (0.0309601302067765) | > loss_mel_ce: 0.00101470947265625 (nan) | > loss: 9.190761193167418e-05 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13736340028918334) | > loader_time: 0.0046 (0.004665706260288006)  --> TIME: 2025-02-26 17:22:12 -- STEP: 3582/36938 -- GLOBAL_STEP: 1333350 | > loss_text_ce: 0.034600041806697845 (0.030952296265006264) | > loss_mel_ce: 2.552149534225464 (nan) | > loss: 0.010264880023896694 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.1373193463688119) | > loader_time: 0.0046 (0.004664642926636242)  --> TIME: 2025-02-26 17:22:21 -- STEP: 3632/36938 -- GLOBAL_STEP: 1333400 | > loss_text_ce: 0.040198054164648056 (0.03096362872183577) | > loss_mel_ce: 0.9728975892066956 (nan) | > loss: 0.0040202210657298565 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13727286871571892) | > loader_time: 0.0045 (0.004663703396982022)  --> TIME: 2025-02-26 17:22:31 -- STEP: 3682/36938 -- GLOBAL_STEP: 1333450 | > loss_text_ce: 0.040376871824264526 (0.03096920880678756) | > loss_mel_ce: 3.2283935546875 (nan) | > loss: 0.012971311807632446 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13725019569697425) | > loader_time: 0.0045 (0.004663251263495687)  --> TIME: 2025-02-26 17:22:40 -- STEP: 3732/36938 -- GLOBAL_STEP: 1333500 | > loss_text_ce: 0.031523093581199646 (0.03095996390707743) | > loss_mel_ce: 2.560391426086426 (nan) | > loss: 0.010285375639796257 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13728352553846992) | > loader_time: 0.0045 (0.004662689735907311)  --> TIME: 2025-02-26 17:22:49 -- STEP: 3782/36938 -- GLOBAL_STEP: 1333550 | > loss_text_ce: 0.027899404987692833 (0.030958026142506246) | > loss_mel_ce: 2.3299453258514404 (nan) | > loss: 0.009356527589261532 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.1372317895253834) | > loader_time: 0.0047 (0.00466191604741346)  --> TIME: 2025-02-26 17:22:58 -- STEP: 3832/36938 -- GLOBAL_STEP: 1333600 | > loss_text_ce: 0.02678326517343521 (0.030948023192462024) | > loss_mel_ce: 2.9489307403564453 (nan) | > loss: 0.01180838979780674 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13718336733497494) | > loader_time: 0.0047 (0.004661151287700028)  --> TIME: 2025-02-26 17:23:08 -- STEP: 3882/36938 -- GLOBAL_STEP: 1333650 | > loss_text_ce: 0.022713059559464455 (0.030939696271966667) | > loss_mel_ce: 1.5 (nan) | > loss: 0.006042512599378824 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13713054286027174) | > loader_time: 0.0045 (0.004661077693956886)  --> TIME: 2025-02-26 17:23:17 -- STEP: 3932/36938 -- GLOBAL_STEP: 1333700 | > loss_text_ce: 0.032471902668476105 (0.03092038323927485) | > loss_mel_ce: 1.0440056324005127 (nan) | > loss: 0.004271736368536949 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13712317739837881) | > loader_time: 0.0047 (0.004660699702230852)  --> TIME: 2025-02-26 17:23:26 -- STEP: 3982/36938 -- GLOBAL_STEP: 1333750 | > loss_text_ce: 0.026071295142173767 (0.030913624800470368) | > loss_mel_ce: 2.2925939559936523 (nan) | > loss: 0.009201053529977798 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13710150683961986) | > loader_time: 0.0045 (0.004660382455225138)  --> TIME: 2025-02-26 17:23:36 -- STEP: 4032/36938 -- GLOBAL_STEP: 1333800 | > loss_text_ce: 0.03465042635798454 (0.03090202007816905) | > loss_mel_ce: 2.6956465244293213 (nan) | > loss: 0.010834512300789356 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13713174514354207) | > loader_time: 0.0046 (0.0046599714883736095)  --> TIME: 2025-02-26 17:23:45 -- STEP: 4082/36938 -- GLOBAL_STEP: 1333850 | > loss_text_ce: 0.024887943640351295 (0.030912365461163504) | > loss_mel_ce: 1.3088573217391968 (nan) | > loss: 0.005292640067636967 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13712373514609952) | > loader_time: 0.0046 (0.004660188596894601)  --> TIME: 2025-02-26 17:23:54 -- STEP: 4132/36938 -- GLOBAL_STEP: 1333900 | > loss_text_ce: 0.027671176940202713 (0.030892731543424805) | > loss_mel_ce: 3.2458784580230713 (nan) | > loss: 0.012990276329219341 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.13711930758032181) | > loader_time: 0.0046 (0.004659960253771821)  --> TIME: 2025-02-26 17:24:04 -- STEP: 4182/36938 -- GLOBAL_STEP: 1333950 | > loss_text_ce: 0.03503105416893959 (0.030898432280299967) | > loss_mel_ce: 3.068101167678833 (nan) | > loss: 0.012314017862081528 (nan) | > current_lr: 5e-06 | > step_time: 0.1703 (0.1371463769249571) | > loader_time: 0.0048 (0.004659739195131777)  --> TIME: 2025-02-26 17:24:13 -- STEP: 4232/36938 -- GLOBAL_STEP: 1334000 | > loss_text_ce: 0.03058447130024433 (0.030903822076920637) | > loss_mel_ce: 3.097301483154297 (nan) | > loss: 0.012412247247993946 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13717067281104656) | > loader_time: 0.0046 (0.004659486402861328)  --> TIME: 2025-02-26 17:24:22 -- STEP: 4282/36938 -- GLOBAL_STEP: 1334050 | > loss_text_ce: 0.02719053439795971 (0.03088490791237769) | > loss_mel_ce: 2.6811373233795166 (nan) | > loss: 0.01074733305722475 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13708156622887094) | > loader_time: 0.0046 (0.0046588664074915034)  --> TIME: 2025-02-26 17:24:31 -- STEP: 4332/36938 -- GLOBAL_STEP: 1334100 | > loss_text_ce: 0.030158154666423798 (0.030862648989007715) | > loss_mel_ce: 2.6291892528533936 (nan) | > loss: 0.0105529660359025 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13708862274307282) | > loader_time: 0.0045 (0.004658488630698099)  --> TIME: 2025-02-26 17:24:41 -- STEP: 4382/36938 -- GLOBAL_STEP: 1334150 | > loss_text_ce: 0.026082657277584076 (0.03086627136222212) | > loss_mel_ce: 2.9124464988708496 (nan) | > loss: 0.011660831049084663 (nan) | > current_lr: 5e-06 | > step_time: 0.187 (0.1370897194611214) | > loader_time: 0.0048 (0.004658176440835698)  --> TIME: 2025-02-26 17:24:50 -- STEP: 4432/36938 -- GLOBAL_STEP: 1334200 | > loss_text_ce: 0.018675198778510094 (0.0308751847203283) | > loss_mel_ce: 2.6213111877441406 (nan) | > loss: 0.010476136580109596 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13707170681187372) | > loader_time: 0.0045 (0.004657815993908071)  --> TIME: 2025-02-26 17:24:59 -- STEP: 4482/36938 -- GLOBAL_STEP: 1334250 | > loss_text_ce: 0.02212502434849739 (0.030874620414505654) | > loss_mel_ce: 2.7702929973602295 (nan) | > loss: 0.011081024073064327 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13704275212507042) | > loader_time: 0.0045 (0.004657705731287637)  --> TIME: 2025-02-26 17:25:08 -- STEP: 4532/36938 -- GLOBAL_STEP: 1334300 | > loss_text_ce: 0.0348941907286644 (0.030885850965059365) | > loss_mel_ce: 2.8958024978637695 (nan) | > loss: 0.011629750020802021 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13700926330167404) | > loader_time: 0.0046 (0.004657588484849105)  --> TIME: 2025-02-26 17:25:18 -- STEP: 4582/36938 -- GLOBAL_STEP: 1334350 | > loss_text_ce: 0.028083574026823044 (0.030899004119239824) | > loss_mel_ce: 2.6832571029663086 (nan) | > loss: 0.010759289376437664 (nan) | > current_lr: 5e-06 | > step_time: 0.163 (0.13704667883323066) | > loader_time: 0.0048 (0.004657696917919751)  --> TIME: 2025-02-26 17:25:27 -- STEP: 4632/36938 -- GLOBAL_STEP: 1334400 | > loss_text_ce: 0.043277159333229065 (0.030898793020232584) | > loss_mel_ce: 3.3365187644958496 (nan) | > loss: 0.013411889784038067 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.13704860869261035) | > loader_time: 0.0047 (0.004657075761307497)  --> TIME: 2025-02-26 17:25:37 -- STEP: 4682/36938 -- GLOBAL_STEP: 1334450 | > loss_text_ce: 0.02561831846833229 (0.030884340118627958) | > loss_mel_ce: 3.0217032432556152 (nan) | > loss: 0.012092546559870243 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13705064353144378) | > loader_time: 0.0047 (0.004656666112424775)  --> TIME: 2025-02-26 17:25:45 -- STEP: 4732/36938 -- GLOBAL_STEP: 1334500 | > loss_text_ce: 0.024354485794901848 (0.030877946756031466) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.1369774331002541) | > loader_time: 0.0046 (0.004656007757452918)  --> TIME: 2025-02-26 17:25:55 -- STEP: 4782/36938 -- GLOBAL_STEP: 1334550 | > loss_text_ce: 0.03738332912325859 (0.030883214973705532) | > loss_mel_ce: 2.8990206718444824 (nan) | > loss: 0.01165239792317152 (nan) | > current_lr: 5e-06 | > step_time: 0.1693 (0.13707009971216552) | > loader_time: 0.0048 (0.004655808895913591)  --> TIME: 2025-02-26 17:26:05 -- STEP: 4832/36938 -- GLOBAL_STEP: 1334600 | > loss_text_ce: 0.026003343984484673 (0.03087966778480975) | > loss_mel_ce: 3.18115234375 (nan) | > loss: 0.012726808898150921 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.1370817173198353) | > loader_time: 0.0045 (0.00465542818142089)  --> TIME: 2025-02-26 17:26:14 -- STEP: 4882/36938 -- GLOBAL_STEP: 1334650 | > loss_text_ce: 0.030997414141893387 (0.030883236871482877) | > loss_mel_ce: 2.9562408924102783 (nan) | > loss: 0.011854121461510658 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13709403129446304) | > loader_time: 0.0045 (0.004655185902231009)  --> TIME: 2025-02-26 17:26:24 -- STEP: 4932/36938 -- GLOBAL_STEP: 1334700 | > loss_text_ce: 0.02810424007475376 (0.0308729032146505) | > loss_mel_ce: 2.6861071586608887 (nan) | > loss: 0.01077068131417036 (nan) | > current_lr: 5e-06 | > step_time: 0.1716 (0.1370930122052951) | > loader_time: 0.0048 (0.004655185165389806)  --> TIME: 2025-02-26 17:26:33 -- STEP: 4982/36938 -- GLOBAL_STEP: 1334750 | > loss_text_ce: 0.02692214399576187 (0.030861931445005278) | > loss_mel_ce: 2.0970215797424316 (nan) | > loss: 0.008428349159657955 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.1370761558168239) | > loader_time: 0.0044 (0.004654836721661486)  --> TIME: 2025-02-26 17:26:42 -- STEP: 5032/36938 -- GLOBAL_STEP: 1334800 | > loss_text_ce: 0.03657596930861473 (0.030862225888023783) | > loss_mel_ce: 3.369122266769409 (nan) | > loss: 0.013514676131308079 (nan) | > current_lr: 5e-06 | > step_time: 0.1464 (0.1370611173738921) | > loader_time: 0.0046 (0.0046545451888977485)  --> TIME: 2025-02-26 17:26:52 -- STEP: 5082/36938 -- GLOBAL_STEP: 1334850 | > loss_text_ce: 0.03400816768407822 (0.030856854959186907) | > loss_mel_ce: 2.666283369064331 (nan) | > loss: 0.010715443640947342 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.1370980207585101) | > loader_time: 0.0048 (0.004654235325662795)  --> TIME: 2025-02-26 17:27:01 -- STEP: 5132/36938 -- GLOBAL_STEP: 1334900 | > loss_text_ce: 0.02675027772784233 (0.030859585233241036) | > loss_mel_ce: 1.0316413640975952 (nan) | > loss: 0.0041999672539532185 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1370620250887728) | > loader_time: 0.0047 (0.004653947295756054)  --> TIME: 2025-02-26 17:27:10 -- STEP: 5182/36938 -- GLOBAL_STEP: 1334950 | > loss_text_ce: 0.0364103689789772 (0.030870121543822987) | > loss_mel_ce: 3.105224609375 (nan) | > loss: 0.01246680598706007 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1370656954096528) | > loader_time: 0.0046 (0.0046536825835957855)  --> TIME: 2025-02-26 17:27:19 -- STEP: 5232/36938 -- GLOBAL_STEP: 1335000 | > loss_text_ce: 0.04215281829237938 (0.03088843863838728) | > loss_mel_ce: 0.0474025197327137 (nan) | > loss: 0.00035537834628485143 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13705137440370802) | > loader_time: 0.0046 (0.004653353164319965)  --> TIME: 2025-02-26 17:27:28 -- STEP: 5282/36938 -- GLOBAL_STEP: 1335050 | > loss_text_ce: 0.037216320633888245 (0.030883534845202423) | > loss_mel_ce: 1.0969631671905518 (nan) | > loss: 0.004500712268054485 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13704111651912) | > loader_time: 0.0045 (0.004652928917844024)  --> TIME: 2025-02-26 17:27:38 -- STEP: 5332/36938 -- GLOBAL_STEP: 1335100 | > loss_text_ce: 0.019818391650915146 (0.03087747391878885) | > loss_mel_ce: 1.3765885829925537 (nan) | > loss: 0.005541298072785139 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1370240257632343) | > loader_time: 0.0045 (0.0046525670457226165)  --> TIME: 2025-02-26 17:27:47 -- STEP: 5382/36938 -- GLOBAL_STEP: 1335150 | > loss_text_ce: 0.03265523165464401 (0.030879626824703608) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.13705663178676694) | > loader_time: 0.0047 (0.004652299831366286)  --> TIME: 2025-02-26 17:27:57 -- STEP: 5432/36938 -- GLOBAL_STEP: 1335200 | > loss_text_ce: 0.02957388013601303 (0.030879106398912583) | > loss_mel_ce: 3.5403635501861572 (nan) | > loss: 0.014166419394314289 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.1370773246958955) | > loader_time: 0.0047 (0.004652235530788668)  --> TIME: 2025-02-26 17:28:06 -- STEP: 5482/36938 -- GLOBAL_STEP: 1335250 | > loss_text_ce: 0.012947892770171165 (0.030878201156527767) | > loss_mel_ce: 0.0994110107421875 (nan) | > loss: 0.0004458687035366893 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13708301244622226) | > loader_time: 0.0045 (0.00465202649028491)  --> TIME: 2025-02-26 17:28:15 -- STEP: 5532/36938 -- GLOBAL_STEP: 1335300 | > loss_text_ce: 0.02802821807563305 (0.030880830476524488) | > loss_mel_ce: 2.7675251960754395 (nan) | > loss: 0.01109346654266119 (nan) | > current_lr: 5e-06 | > step_time: 0.1614 (0.13707335686562905) | > loader_time: 0.0046 (0.004651716844982119)  --> TIME: 2025-02-26 17:28:25 -- STEP: 5582/36938 -- GLOBAL_STEP: 1335350 | > loss_text_ce: 0.029124118387699127 (0.030878343895616278) | > loss_mel_ce: 0.0048828125 (nan) | > loss: 0.00013494814629666507 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13710673741224738) | > loader_time: 0.0046 (0.00465165039665253)  --> TIME: 2025-02-26 17:28:34 -- STEP: 5632/36938 -- GLOBAL_STEP: 1335400 | > loss_text_ce: 0.04204954579472542 (0.03085522660272876) | > loss_mel_ce: 2.2766287326812744 (nan) | > loss: 0.009201104752719402 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13714447041804065) | > loader_time: 0.0047 (0.004651265261186798)  --> TIME: 2025-02-26 17:28:44 -- STEP: 5682/36938 -- GLOBAL_STEP: 1335450 | > loss_text_ce: 0.03604031354188919 (0.030856995499227033) | > loss_mel_ce: 2.7472963333129883 (nan) | > loss: 0.011044987477362156 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13711687789253615) | > loader_time: 0.0045 (0.0046505604945031465)  --> TIME: 2025-02-26 17:28:53 -- STEP: 5732/36938 -- GLOBAL_STEP: 1335500 | > loss_text_ce: 0.033530183136463165 (0.03085538482240519) | > loss_mel_ce: 3.396531343460083 (nan) | > loss: 0.013611355796456337 (nan) | > current_lr: 5e-06 | > step_time: 0.1515 (0.13710028502962693) | > loader_time: 0.0046 (0.004650118712021277)  --> TIME: 2025-02-26 17:29:02 -- STEP: 5782/36938 -- GLOBAL_STEP: 1335550 | > loss_text_ce: 0.03131506219506264 (0.03085438240152297) | > loss_mel_ce: 0.0173517856746912 (nan) | > loss: 0.0001931224251165986 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13712778037000906) | > loader_time: 0.0044 (0.004649887403292221)  --> TIME: 2025-02-26 17:29:11 -- STEP: 5832/36938 -- GLOBAL_STEP: 1335600 | > loss_text_ce: 0.03422684222459793 (0.030856623785491696) | > loss_mel_ce: 2.5448553562164307 (nan) | > loss: 0.010234453715384007 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13708630151055262) | > loader_time: 0.0045 (0.004649770971188338)  --> TIME: 2025-02-26 17:29:21 -- STEP: 5882/36938 -- GLOBAL_STEP: 1335650 | > loss_text_ce: 0.02872195653617382 (0.03086480804065427) | > loss_mel_ce: 2.069915294647217 (nan) | > loss: 0.008327926509082317 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.137100011586575) | > loader_time: 0.0045 (0.004649427584830054)  --> TIME: 2025-02-26 17:29:30 -- STEP: 5932/36938 -- GLOBAL_STEP: 1335700 | > loss_text_ce: 0.0229030791670084 (0.030876507093757517) | > loss_mel_ce: 2.4621872901916504 (nan) | > loss: 0.009861470200121403 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13709935431374246) | > loader_time: 0.0047 (0.004648980423958825)  --> TIME: 2025-02-26 17:29:39 -- STEP: 5982/36938 -- GLOBAL_STEP: 1335750 | > loss_text_ce: 0.017843322828412056 (0.030872016038608015) | > loss_mel_ce: 0.00026055972557514906 (nan) | > loss: 7.184081187006086e-05 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.1370816091676488) | > loader_time: 0.0045 (0.004648706579001131)  --> TIME: 2025-02-26 17:29:49 -- STEP: 6032/36938 -- GLOBAL_STEP: 1335800 | > loss_text_ce: 0.03464851155877113 (0.030874770990123885) | > loss_mel_ce: 2.54280948638916 (nan) | > loss: 0.010228008031845093 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1370907789911448) | > loader_time: 0.0047 (0.004649489920398608)  --> TIME: 2025-02-26 17:29:59 -- STEP: 6082/36938 -- GLOBAL_STEP: 1335850 | > loss_text_ce: 0.028335345908999443 (0.030870430929548314) | > loss_mel_ce: 3.066455602645874 (nan) | > loss: 0.012280916795134544 (nan) | > current_lr: 5e-06 | > step_time: 0.1837 (0.13718484930598515) | > loader_time: 0.0048 (0.00464976190619076)  --> TIME: 2025-02-26 17:30:08 -- STEP: 6132/36938 -- GLOBAL_STEP: 1335900 | > loss_text_ce: 0.031888868659734726 (0.03087384770592445) | > loss_mel_ce: 1.4221463203430176 (nan) | > loss: 0.005769981071352959 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13721390705208142) | > loader_time: 0.0047 (0.0046498631233494155)  --> TIME: 2025-02-26 17:30:17 -- STEP: 6182/36938 -- GLOBAL_STEP: 1335950 | > loss_text_ce: 0.03530716523528099 (0.030877344105474047) | > loss_mel_ce: 1.0254873037338257 (nan) | > loss: 0.004209502134472132 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13720323379199384) | > loader_time: 0.0046 (0.004650441121607693)  --> TIME: 2025-02-26 17:30:27 -- STEP: 6232/36938 -- GLOBAL_STEP: 1336000 | > loss_text_ce: 0.03247314691543579 (0.030884914138800894) | > loss_mel_ce: 2.101593494415283 (nan) | > loss: 0.00846851896494627 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13719262295571116) | > loader_time: 0.0046 (0.004651157173479943)  --> TIME: 2025-02-26 17:30:36 -- STEP: 6282/36938 -- GLOBAL_STEP: 1336050 | > loss_text_ce: 0.02993232198059559 (0.030881392084514566) | > loss_mel_ce: 2.2681362628936768 (nan) | > loss: 0.00911931972950697 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1371530894070281) | > loader_time: 0.0046 (0.004651215455214043)  --> TIME: 2025-02-26 17:30:45 -- STEP: 6332/36938 -- GLOBAL_STEP: 1336100 | > loss_text_ce: 0.03201931342482567 (0.030882369303079797) | > loss_mel_ce: 2.463305950164795 (nan) | > loss: 0.009902085177600384 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13714712319762623) | > loader_time: 0.0046 (0.004651334755645108)  --> TIME: 2025-02-26 17:30:54 -- STEP: 6382/36938 -- GLOBAL_STEP: 1336150 | > loss_text_ce: 0.02686523087322712 (0.03087908597200448) | > loss_mel_ce: 2.6904971599578857 (nan) | > loss: 0.010783184319734573 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13712457028798083) | > loader_time: 0.0046 (0.004651574272602876)  --> TIME: 2025-02-26 17:31:04 -- STEP: 6432/36938 -- GLOBAL_STEP: 1336200 | > loss_text_ce: 0.022929979488253593 (0.03087576281689675) | > loss_mel_ce: 1.2813925743103027 (nan) | > loss: 0.005175883416086435 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13710767686811817) | > loader_time: 0.0046 (0.004652343282652153)  --> TIME: 2025-02-26 17:31:13 -- STEP: 6482/36938 -- GLOBAL_STEP: 1336250 | > loss_text_ce: 0.0373743399977684 (0.03087900761425534) | > loss_mel_ce: 0.0759684219956398 (nan) | > loss: 0.00044977289508096874 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13707916069383835) | > loader_time: 0.0047 (0.004652542451471887)  --> TIME: 2025-02-26 17:31:22 -- STEP: 6532/36938 -- GLOBAL_STEP: 1336300 | > loss_text_ce: 0.022615233436226845 (0.03088664934753434) | > loss_mel_ce: 2.026632070541382 (nan) | > loss: 0.008131934329867363 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13704503392851983) | > loader_time: 0.0046 (0.004652428612414139)  --> TIME: 2025-02-26 17:31:31 -- STEP: 6582/36938 -- GLOBAL_STEP: 1336350 | > loss_text_ce: 0.026222333312034607 (0.030888911460681085) | > loss_mel_ce: 2.859957695007324 (nan) | > loss: 0.011453095823526382 (nan) | > current_lr: 5e-06 | > step_time: 0.1716 (0.13703054307406132) | > loader_time: 0.0048 (0.004653285897926384)  --> TIME: 2025-02-26 17:31:40 -- STEP: 6632/36938 -- GLOBAL_STEP: 1336400 | > loss_text_ce: 0.032009705901145935 (0.03087722781583031) | > loss_mel_ce: 2.466200590133667 (nan) | > loss: 0.009913532994687557 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.1370317046547393) | > loader_time: 0.0047 (0.0046537552653521065)  --> TIME: 2025-02-26 17:31:49 -- STEP: 6682/36938 -- GLOBAL_STEP: 1336450 | > loss_text_ce: 0.034142155200242996 (0.030881684464638046) | > loss_mel_ce: 2.6623246669769287 (nan) | > loss: 0.010700265876948833 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.13700903432521205) | > loader_time: 0.0047 (0.004654449354706692)  --> TIME: 2025-02-26 17:31:59 -- STEP: 6732/36938 -- GLOBAL_STEP: 1336500 | > loss_text_ce: 0.030216434970498085 (0.030865662533331128) | > loss_mel_ce: 2.712803602218628 (nan) | > loss: 0.01088500116020441 (nan) | > current_lr: 5e-06 | > step_time: 0.1532 (0.1370076557319218) | > loader_time: 0.0048 (0.004654561347077463)  --> TIME: 2025-02-26 17:32:08 -- STEP: 6782/36938 -- GLOBAL_STEP: 1336550 | > loss_text_ce: 0.02897053398191929 (0.030867992428127832) | > loss_mel_ce: 3.0751612186431885 (nan) | > loss: 0.01231798343360424 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.137031652694675) | > loader_time: 0.0048 (0.004654772195166395)  --> TIME: 2025-02-26 17:32:18 -- STEP: 6832/36938 -- GLOBAL_STEP: 1336600 | > loss_text_ce: 0.03530704975128174 (0.030867971696569727) | > loss_mel_ce: 2.8620123863220215 (nan) | > loss: 0.011497299186885357 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.1370439556987837) | > loader_time: 0.0048 (0.0046553941535168114)  --> TIME: 2025-02-26 17:32:27 -- STEP: 6882/36938 -- GLOBAL_STEP: 1336650 | > loss_text_ce: 0.023945612832903862 (0.030863888829606036) | > loss_mel_ce: 1.3274972438812256 (nan) | > loss: 0.005362868774682283 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13705260024033303) | > loader_time: 0.0045 (0.004655196894673693)  --> TIME: 2025-02-26 17:32:37 -- STEP: 6932/36938 -- GLOBAL_STEP: 1336700 | > loss_text_ce: 0.03456059470772743 (0.030868908668711323) | > loss_mel_ce: 3.0428524017333984 (nan) | > loss: 0.012211957015097141 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13707842822715993) | > loader_time: 0.0045 (0.004655458235149097)  --> TIME: 2025-02-26 17:32:46 -- STEP: 6982/36938 -- GLOBAL_STEP: 1336750 | > loss_text_ce: 0.03214457258582115 (0.030875201575205676) | > loss_mel_ce: 2.32568097114563 (nan) | > loss: 0.009356451220810413 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.1370698358704593) | > loader_time: 0.0046 (0.00465531070537672)  --> TIME: 2025-02-26 17:32:55 -- STEP: 7032/36938 -- GLOBAL_STEP: 1336800 | > loss_text_ce: 0.031889207661151886 (0.030871368045041887) | > loss_mel_ce: 1.7684539556503296 (nan) | > loss: 0.007144219242036343 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13705539594873775) | > loader_time: 0.0045 (0.004655325643302919)  --> TIME: 2025-02-26 17:33:05 -- STEP: 7082/36938 -- GLOBAL_STEP: 1336850 | > loss_text_ce: 0.02655479870736599 (0.030876944541909587) | > loss_mel_ce: 2.256843328475952 (nan) | > loss: 0.009061104618012905 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13705164175591075) | > loader_time: 0.0046 (0.0046554983958320105)  --> TIME: 2025-02-26 17:33:14 -- STEP: 7132/36938 -- GLOBAL_STEP: 1336900 | > loss_text_ce: 0.04452971741557121 (0.030876621198046492) | > loss_mel_ce: 2.0103671550750732 (nan) | > loss: 0.008154353126883507 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13707991934731925) | > loader_time: 0.0045 (0.004655707203399189)  --> TIME: 2025-02-26 17:33:24 -- STEP: 7182/36938 -- GLOBAL_STEP: 1336950 | > loss_text_ce: 0.02983052097260952 (0.03087644375267041) | > loss_mel_ce: 3.486970901489258 (nan) | > loss: 0.013955561444163322 (nan) | > current_lr: 5e-06 | > step_time: 0.1512 (0.1370709691330506) | > loader_time: 0.0047 (0.0046557304222449095)  --> TIME: 2025-02-26 17:33:33 -- STEP: 7232/36938 -- GLOBAL_STEP: 1337000 | > loss_text_ce: 0.020447321236133575 (0.030886605946721605) | > loss_mel_ce: 2.442805528640747 (nan) | > loss: 0.009774813428521156 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13710010364915445) | > loader_time: 0.0045 (0.00465581852909739)  --> TIME: 2025-02-26 17:33:43 -- STEP: 7282/36938 -- GLOBAL_STEP: 1337050 | > loss_text_ce: 0.03095969930291176 (0.03089094821350131) | > loss_mel_ce: 2.2842166423797607 (nan) | > loss: 0.00918720755726099 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13710216400682956) | > loader_time: 0.0047 (0.004656249368233338)  --> TIME: 2025-02-26 17:33:52 -- STEP: 7332/36938 -- GLOBAL_STEP: 1337100 | > loss_text_ce: 0.04323180019855499 (0.03088869303062231) | > loss_mel_ce: 2.1611294746398926 (nan) | > loss: 0.008747465908527374 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13709561169895426) | > loader_time: 0.0046 (0.0046562344339852605)  --> TIME: 2025-02-26 17:34:01 -- STEP: 7382/36938 -- GLOBAL_STEP: 1337150 | > loss_text_ce: 0.027874857187271118 (0.03089332080633407) | > loss_mel_ce: 3.069944381713867 (nan) | > loss: 0.012292934581637383 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13710941518428582) | > loader_time: 0.0047 (0.004656130819907207)  --> TIME: 2025-02-26 17:34:11 -- STEP: 7432/36938 -- GLOBAL_STEP: 1337200 | > loss_text_ce: 0.03473251685500145 (0.03090232908501429) | > loss_mel_ce: 2.853264808654785 (nan) | > loss: 0.011460307985544205 (nan) | > current_lr: 5e-06 | > step_time: 0.1514 (0.13713075103492894) | > loader_time: 0.0047 (0.004656504859708748)  --> TIME: 2025-02-26 17:34:20 -- STEP: 7482/36938 -- GLOBAL_STEP: 1337250 | > loss_text_ce: 0.03286853805184364 (0.030904827990791486) | > loss_mel_ce: 2.480332612991333 (nan) | > loss: 0.009973021224141121 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13713763085829028) | > loader_time: 0.0047 (0.004656579557570258)  --> TIME: 2025-02-26 17:34:29 -- STEP: 7532/36938 -- GLOBAL_STEP: 1337300 | > loss_text_ce: 0.03698921203613281 (0.030903061664716673) | > loss_mel_ce: 2.5803518295288086 (nan) | > loss: 0.010386274196207523 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.1371169901881775) | > loader_time: 0.0046 (0.0046564249427661795)  --> TIME: 2025-02-26 17:34:39 -- STEP: 7582/36938 -- GLOBAL_STEP: 1337350 | > loss_text_ce: 0.0342395193874836 (0.03090466473250844) | > loss_mel_ce: 2.787515640258789 (nan) | > loss: 0.011197442188858986 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13712224291170305) | > loader_time: 0.0046 (0.004656896934569662)  --> TIME: 2025-02-26 17:34:49 -- STEP: 7632/36938 -- GLOBAL_STEP: 1337400 | > loss_text_ce: 0.025761818513274193 (0.030898970305473998) | > loss_mel_ce: 2.526745319366455 (nan) | > loss: 0.010128997266292572 (nan) | > current_lr: 5e-06 | > step_time: 0.1781 (0.13717669279820288) | > loader_time: 0.0048 (0.004657538931824636)  --> TIME: 2025-02-26 17:34:58 -- STEP: 7682/36938 -- GLOBAL_STEP: 1337450 | > loss_text_ce: 0.02749783731997013 (0.030898424044318848) | > loss_mel_ce: 0.015228271484375 (nan) | > loss: 0.00016954804596025497 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13715606175248643) | > loader_time: 0.0046 (0.00465742255585544)  --> TIME: 2025-02-26 17:35:07 -- STEP: 7732/36938 -- GLOBAL_STEP: 1337500 | > loss_text_ce: 0.02171185612678528 (0.03090399355186507) | > loss_mel_ce: 2.3698344230651855 (nan) | > loss: 0.009490263648331165 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.13716024900432477) | > loader_time: 0.0046 (0.004657384711599581)  --> TIME: 2025-02-26 17:35:17 -- STEP: 7782/36938 -- GLOBAL_STEP: 1337550 | > loss_text_ce: 0.035656869411468506 (0.030906710451716796) | > loss_mel_ce: 2.855623245239258 (nan) | > loss: 0.011473334394395351 (nan) | > current_lr: 5e-06 | > step_time: 0.1844 (0.1371808897649316) | > loader_time: 0.0048 (0.004657220546337128)  --> TIME: 2025-02-26 17:35:26 -- STEP: 7832/36938 -- GLOBAL_STEP: 1337600 | > loss_text_ce: 0.034484293311834335 (0.030907727418069283) | > loss_mel_ce: 2.4372684955596924 (nan) | > loss: 0.00980854406952858 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13716909036694824) | > loader_time: 0.0047 (0.004656976680590022)  --> TIME: 2025-02-26 17:35:35 -- STEP: 7882/36938 -- GLOBAL_STEP: 1337650 | > loss_text_ce: 0.03677886724472046 (0.030909312294112683) | > loss_mel_ce: 2.28974986076355 (nan) | > loss: 0.00923225749284029 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13716092541628602) | > loader_time: 0.0045 (0.004656829558118177)  --> TIME: 2025-02-26 17:35:44 -- STEP: 7932/36938 -- GLOBAL_STEP: 1337700 | > loss_text_ce: 0.03152088448405266 (0.030908697763980617) | > loss_mel_ce: 3.0914156436920166 (nan) | > loss: 0.012392605654895306 (nan) | > current_lr: 5e-06 | > step_time: 0.1861 (0.13713934682441153) | > loader_time: 0.0049 (0.004656872482415236)  --> TIME: 2025-02-26 17:35:53 -- STEP: 7982/36938 -- GLOBAL_STEP: 1337750 | > loss_text_ce: 0.030108140781521797 (0.03089783546786435) | > loss_mel_ce: 3.017395496368408 (nan) | > loss: 0.012093269266188145 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13710653211382892) | > loader_time: 0.0047 (0.004656634394072922)  --> TIME: 2025-02-26 17:36:03 -- STEP: 8032/36938 -- GLOBAL_STEP: 1337800 | > loss_text_ce: 0.033564209938049316 (0.030894896191249386) | > loss_mel_ce: 2.717813491821289 (nan) | > loss: 0.010918165557086468 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.1371019028750551) | > loader_time: 0.0046 (0.004656869487695969)  --> TIME: 2025-02-26 17:36:12 -- STEP: 8082/36938 -- GLOBAL_STEP: 1337850 | > loss_text_ce: 0.02778521552681923 (0.030902156333845368) | > loss_mel_ce: 2.388638496398926 (nan) | > loss: 0.009588983841240406 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.1370809515405783) | > loader_time: 0.0047 (0.004656611707829459)  --> TIME: 2025-02-26 17:36:22 -- STEP: 8132/36938 -- GLOBAL_STEP: 1337900 | > loss_text_ce: 0.028746165335178375 (0.03090012228830804) | > loss_mel_ce: 2.6113507747650146 (nan) | > loss: 0.01047657523304224 (nan) | > current_lr: 5e-06 | > step_time: 0.1662 (0.1370997866913422) | > loader_time: 0.0047 (0.0046565318072422784)  --> TIME: 2025-02-26 17:36:30 -- STEP: 8182/36938 -- GLOBAL_STEP: 1337950 | > loss_text_ce: 0.020366322249174118 (0.030911859737907676) | > loss_mel_ce: 3.1083223819732666 (nan) | > loss: 0.012415432371199131 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.1370584723186565) | > loader_time: 0.0046 (0.004656494756515388)  --> TIME: 2025-02-26 17:36:40 -- STEP: 8232/36938 -- GLOBAL_STEP: 1338000 | > loss_text_ce: 0.04353930056095123 (0.030910244619008154) | > loss_mel_ce: 2.39404296875 (nan) | > loss: 0.00967294629663229 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13705872461214264) | > loader_time: 0.0046 (0.00465638126066522)  --> TIME: 2025-02-26 17:36:49 -- STEP: 8282/36938 -- GLOBAL_STEP: 1338050 | > loss_text_ce: 0.029023269191384315 (0.03090824179531927) | > loss_mel_ce: 2.72554349899292 (nan) | > loss: 0.010930820368230343 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13707163411043508) | > loader_time: 0.0046 (0.0046561682348060275)  --> TIME: 2025-02-26 17:36:58 -- STEP: 8332/36938 -- GLOBAL_STEP: 1338100 | > loss_text_ce: 0.0356004498898983 (0.03091438893385033) | > loss_mel_ce: 3.4013760089874268 (nan) | > loss: 0.013638796284794807 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1370398007902) | > loader_time: 0.0045 (0.004655753770700977)  --> TIME: 2025-02-26 17:37:07 -- STEP: 8382/36938 -- GLOBAL_STEP: 1338150 | > loss_text_ce: 0.0375387966632843 (0.03091726256531124) | > loss_mel_ce: 2.5092878341674805 (nan) | > loss: 0.010106455534696579 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13702967982677897) | > loader_time: 0.0044 (0.004655418205989407)  --> TIME: 2025-02-26 17:37:17 -- STEP: 8432/36938 -- GLOBAL_STEP: 1338200 | > loss_text_ce: 0.031635429710149765 (0.03091591973224632) | > loss_mel_ce: 2.4463295936584473 (nan) | > loss: 0.009833195246756077 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13701647245793477) | > loader_time: 0.0045 (0.004655063604720409)  --> TIME: 2025-02-26 17:37:26 -- STEP: 8482/36938 -- GLOBAL_STEP: 1338250 | > loss_text_ce: 0.02406085841357708 (0.030914302394279374) | > loss_mel_ce: 2.6178970336914062 (nan) | > loss: 0.010483960621058941 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1369975210501041) | > loader_time: 0.0045 (0.004655215122373121)  --> TIME: 2025-02-26 17:37:35 -- STEP: 8532/36938 -- GLOBAL_STEP: 1338300 | > loss_text_ce: 0.04338608682155609 (0.03091523276697198) | > loss_mel_ce: 2.287473678588867 (nan) | > loss: 0.00924944318830967 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13695871220806433) | > loader_time: 0.0045 (0.004655805597604866)  --> TIME: 2025-02-26 17:37:44 -- STEP: 8582/36938 -- GLOBAL_STEP: 1338350 | > loss_text_ce: 0.03708236664533615 (0.030914431125148297) | > loss_mel_ce: 0.3458158075809479 (nan) | > loss: 0.0015194372972473502 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1369554350021086) | > loader_time: 0.0046 (0.004656165442358952)  --> TIME: 2025-02-26 17:37:53 -- STEP: 8632/36938 -- GLOBAL_STEP: 1338400 | > loss_text_ce: 0.03472071513533592 (0.030919909588416453) | > loss_mel_ce: 1.642443299293518 (nan) | > loss: 0.0066554127261042595 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13695440425488345) | > loader_time: 0.0046 (0.004656799034016133)  --> TIME: 2025-02-26 17:38:03 -- STEP: 8682/36938 -- GLOBAL_STEP: 1338450 | > loss_text_ce: 0.027738189324736595 (0.030924701856224406) | > loss_mel_ce: 2.2517354488372803 (nan) | > loss: 0.00904553011059761 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.1369508806666701) | > loader_time: 0.0046 (0.004657011596707813)  --> TIME: 2025-02-26 17:38:12 -- STEP: 8732/36938 -- GLOBAL_STEP: 1338500 | > loss_text_ce: 0.023656565696001053 (0.030921440468783236) | > loss_mel_ce: 2.1373496055603027 (nan) | > loss: 0.008575421757996082 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13695933982095113) | > loader_time: 0.0046 (0.004657031689223166)  --> TIME: 2025-02-26 17:38:22 -- STEP: 8782/36938 -- GLOBAL_STEP: 1338550 | > loss_text_ce: 0.023792114108800888 (0.030917311438750216) | > loss_mel_ce: 2.2625739574432373 (nan) | > loss: 0.009072881191968918 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.1369740626068655) | > loader_time: 0.0045 (0.004656873675637196)  --> TIME: 2025-02-26 17:38:31 -- STEP: 8832/36938 -- GLOBAL_STEP: 1338600 | > loss_text_ce: 0.02758440189063549 (0.030920464633696163) | > loss_mel_ce: 1.2261905670166016 (nan) | > loss: 0.004975297953933477 (nan) | > current_lr: 5e-06 | > step_time: 0.1846 (0.13697509586379164) | > loader_time: 0.0049 (0.0046570841492950136)  --> TIME: 2025-02-26 17:38:41 -- STEP: 8882/36938 -- GLOBAL_STEP: 1338650 | > loss_text_ce: 0.020267745479941368 (0.030916186129847117) | > loss_mel_ce: 1.9193291664123535 (nan) | > loss: 0.007696813438087702 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13700192964068972) | > loader_time: 0.0046 (0.004657427380413019)  --> TIME: 2025-02-26 17:38:50 -- STEP: 8932/36938 -- GLOBAL_STEP: 1338700 | > loss_text_ce: 0.02468341775238514 (0.03091261631512428) | > loss_mel_ce: 2.610849380493164 (nan) | > loss: 0.010458464734256268 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13702749154304025) | > loader_time: 0.0047 (0.004657511266955237)  --> TIME: 2025-02-26 17:39:00 -- STEP: 8982/36938 -- GLOBAL_STEP: 1338750 | > loss_text_ce: 0.030007243156433105 (0.030913430227873382) | > loss_mel_ce: 2.158850908279419 (nan) | > loss: 0.008685945533216 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13705915479398872) | > loader_time: 0.0047 (0.004658217420558901)  --> TIME: 2025-02-26 17:39:10 -- STEP: 9032/36938 -- GLOBAL_STEP: 1338800 | > loss_text_ce: 0.03150472417473793 (0.03091648288055276) | > loss_mel_ce: 3.18036150932312 (nan) | > loss: 0.012745501473546028 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.13709192090165445) | > loader_time: 0.0047 (0.004658441848729552)  --> TIME: 2025-02-26 17:39:19 -- STEP: 9082/36938 -- GLOBAL_STEP: 1338850 | > loss_text_ce: 0.017664726823568344 (0.03091462944813509) | > loss_mel_ce: 7.421875 (nan) | > loss: 0.029521984979510307 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13710878699184464) | > loader_time: 0.0045 (0.004658776399612019)  --> TIME: 2025-02-26 17:39:28 -- STEP: 9132/36938 -- GLOBAL_STEP: 1338900 | > loss_text_ce: 0.026827894151210785 (0.030911458345292037) | > loss_mel_ce: 0.021183013916015625 (nan) | > loss: 0.00019051948038395494 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13706971376651364) | > loader_time: 0.0045 (0.004658625149904415)  --> TIME: 2025-02-26 17:39:37 -- STEP: 9182/36938 -- GLOBAL_STEP: 1338950 | > loss_text_ce: 0.022994743660092354 (0.030910591502523607) | > loss_mel_ce: 2.6817827224731445 (nan) | > loss: 0.010733244940638542 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13705661802659583) | > loader_time: 0.0047 (0.0046584885044052326)  --> TIME: 2025-02-26 17:39:47 -- STEP: 9232/36938 -- GLOBAL_STEP: 1339000 | > loss_text_ce: 0.028421668335795403 (0.030905524092559755) | > loss_mel_ce: 1.0338078737258911 (nan) | > loss: 0.004215196706354618 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13704411729369695) | > loader_time: 0.0045 (0.004658376375145488)  --> TIME: 2025-02-26 17:39:56 -- STEP: 9282/36938 -- GLOBAL_STEP: 1339050 | > loss_text_ce: 0.030908696353435516 (0.03090720833008284) | > loss_mel_ce: 2.8524057865142822 (nan) | > loss: 0.011441724374890327 (nan) | > current_lr: 5e-06 | > step_time: 0.1689 (0.13704136545045262) | > loader_time: 0.0048 (0.004658338685047018)  --> TIME: 2025-02-26 17:40:06 -- STEP: 9332/36938 -- GLOBAL_STEP: 1339100 | > loss_text_ce: 0.03467151150107384 (0.030908417489339327) | > loss_mel_ce: 2.5625 (nan) | > loss: 0.010306237265467644 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.1370768501326297) | > loader_time: 0.0045 (0.00465832275737268)  --> TIME: 2025-02-26 17:40:15 -- STEP: 9382/36938 -- GLOBAL_STEP: 1339150 | > loss_text_ce: 0.025258861482143402 (0.03091148947336993) | > loss_mel_ce: 2.6658425331115723 (nan) | > loss: 0.010678973980247974 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13709484941286998) | > loader_time: 0.0046 (0.004658166291337113)  --> TIME: 2025-02-26 17:40:24 -- STEP: 9432/36938 -- GLOBAL_STEP: 1339200 | > loss_text_ce: 0.02246103063225746 (0.03090549418121139) | > loss_mel_ce: 2.6444928646087646 (nan) | > loss: 0.010583150200545788 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.13706742928531657) | > loader_time: 0.0046 (0.004657856304833615)  --> TIME: 2025-02-26 17:40:33 -- STEP: 9482/36938 -- GLOBAL_STEP: 1339250 | > loss_text_ce: 0.027965489774942398 (0.030910082341414798) | > loss_mel_ce: 2.9563846588134766 (nan) | > loss: 0.01184266060590744 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13702905130295853) | > loader_time: 0.0046 (0.004657511946471587)  --> TIME: 2025-02-26 17:40:42 -- STEP: 9532/36938 -- GLOBAL_STEP: 1339300 | > loss_text_ce: 0.024711303412914276 (0.030910255349173976) | > loss_mel_ce: 1.4186792373657227 (nan) | > loss: 0.005727740470319986 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13700484496627716) | > loader_time: 0.0046 (0.004657148514482298)  --> TIME: 2025-02-26 17:40:52 -- STEP: 9582/36938 -- GLOBAL_STEP: 1339350 | > loss_text_ce: 0.03231876716017723 (0.03090889476756953) | > loss_mel_ce: 1.5993118286132812 (nan) | > loss: 0.006474724970757961 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.137012502938615) | > loader_time: 0.0044 (0.004656973474335766)  --> TIME: 2025-02-26 17:41:01 -- STEP: 9632/36938 -- GLOBAL_STEP: 1339400 | > loss_text_ce: 0.037910062819719315 (0.03090997551343724) | > loss_mel_ce: 2.6759033203125 (nan) | > loss: 0.010769100859761238 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13701893810021953) | > loader_time: 0.0057 (0.004656855103581464)  --> TIME: 2025-02-26 17:41:11 -- STEP: 9682/36938 -- GLOBAL_STEP: 1339450 | > loss_text_ce: 0.03718934953212738 (0.030915056610829317) | > loss_mel_ce: 2.7669050693511963 (nan) | > loss: 0.011127359233796597 (nan) | > current_lr: 5e-06 | > step_time: 0.152 (0.13702231068129522) | > loader_time: 0.0047 (0.004656646005921674)  --> TIME: 2025-02-26 17:41:21 -- STEP: 9732/36938 -- GLOBAL_STEP: 1339500 | > loss_text_ce: 0.036832477897405624 (0.030914290227882866) | > loss_mel_ce: 2.395932674407959 (nan) | > loss: 0.009653830900788307 (nan) | > current_lr: 5e-06 | > step_time: 0.1609 (0.1370849974891363) | > loader_time: 0.0047 (0.004656732106865373)  --> TIME: 2025-02-26 17:41:30 -- STEP: 9782/36938 -- GLOBAL_STEP: 1339550 | > loss_text_ce: 0.02760581485927105 (0.03091922716362898) | > loss_mel_ce: 1.6443825960159302 (nan) | > loss: 0.006634874735027552 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13708010243384874) | > loader_time: 0.0046 (0.004656572840107034)  --> TIME: 2025-02-26 17:41:39 -- STEP: 9832/36938 -- GLOBAL_STEP: 1339600 | > loss_text_ce: 0.024906447157263756 (0.030920749296086777) | > loss_mel_ce: 2.664494276046753 (nan) | > loss: 0.01067222561687231 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13706845726493558) | > loader_time: 0.0045 (0.004656280488673034)  --> TIME: 2025-02-26 17:41:49 -- STEP: 9882/36938 -- GLOBAL_STEP: 1339650 | > loss_text_ce: 0.02572680450975895 (0.030922585560602244) | > loss_mel_ce: 2.2274420261383057 (nan) | > loss: 0.008941146545112133 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.1370910938344396) | > loader_time: 0.0045 (0.004656132959783919)  --> TIME: 2025-02-26 17:41:58 -- STEP: 9932/36938 -- GLOBAL_STEP: 1339700 | > loss_text_ce: 0.03680676594376564 (0.030923032814753476) | > loss_mel_ce: 1.5580500364303589 (nan) | > loss: 0.006328797433525324 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13712286363433507) | > loader_time: 0.0045 (0.004656140644896967)  --> TIME: 2025-02-26 17:42:08 -- STEP: 9982/36938 -- GLOBAL_STEP: 1339750 | > loss_text_ce: 0.034281790256500244 (0.030916709927252362) | > loss_mel_ce: 2.464498281478882 (nan) | > loss: 0.009915794245898724 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.137120580080874) | > loader_time: 0.0047 (0.004655914635296182)  --> TIME: 2025-02-26 17:42:17 -- STEP: 10032/36938 -- GLOBAL_STEP: 1339800 | > loss_text_ce: 0.03398577868938446 (0.030911642892304376) | > loss_mel_ce: 2.757286548614502 (nan) | > loss: 0.011076478287577629 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.1371079546412789) | > loader_time: 0.0045 (0.004655665639294784)  --> TIME: 2025-02-26 17:42:26 -- STEP: 10082/36938 -- GLOBAL_STEP: 1339850 | > loss_text_ce: 0.02840547449886799 (0.030907691788418668) | > loss_mel_ce: 1.9066311120986938 (nan) | > loss: 0.0076787169091403484 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13713198262910725) | > loader_time: 0.0045 (0.004655381276290173)  --> TIME: 2025-02-26 17:42:36 -- STEP: 10132/36938 -- GLOBAL_STEP: 1339900 | > loss_text_ce: 0.0340699627995491 (0.030905982936703793) | > loss_mel_ce: 2.998242139816284 (nan) | > loss: 0.01203298568725586 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13715422685279105) | > loader_time: 0.0046 (0.004655296911705745)  --> TIME: 2025-02-26 17:42:46 -- STEP: 10182/36938 -- GLOBAL_STEP: 1339950 | > loss_text_ce: 0.0336671881377697 (0.03090782118265147) | > loss_mel_ce: 2.9921481609344482 (nan) | > loss: 0.012007203884422779 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13717609633399155) | > loader_time: 0.0046 (0.004655405829262359)  --> TIME: 2025-02-26 17:42:55 -- STEP: 10232/36938 -- GLOBAL_STEP: 1340000 | > loss_text_ce: 0.022474991157650948 (0.030901487997033655) | > loss_mel_ce: 2.535717487335205 (nan) | > loss: 0.010151558555662632 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13718669760777163) | > loader_time: 0.0046 (0.004655397525515952) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1340000.pth  --> TIME: 2025-02-26 17:43:11 -- STEP: 10282/36938 -- GLOBAL_STEP: 1340050 | > loss_text_ce: 0.02566101774573326 (0.030899762148124506) | > loss_mel_ce: 1.106766700744629 (nan) | > loss: 0.004493760876357555 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13718856364181406) | > loader_time: 0.0045 (0.0046560376378215135)  --> TIME: 2025-02-26 17:43:20 -- STEP: 10332/36938 -- GLOBAL_STEP: 1340100 | > loss_text_ce: 0.031802259385585785 (0.030898332128045627) | > loss_mel_ce: 2.858745813369751 (nan) | > loss: 0.011470429599285126 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13717914790642005) | > loader_time: 0.0046 (0.004655916839434212)  --> TIME: 2025-02-26 17:43:29 -- STEP: 10382/36938 -- GLOBAL_STEP: 1340150 | > loss_text_ce: 0.039004482328891754 (0.030897699716161672) | > loss_mel_ce: 2.921746015548706 (nan) | > loss: 0.011749010533094406 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13716460261770255) | > loader_time: 0.0046 (0.004655730147923023)  --> TIME: 2025-02-26 17:43:38 -- STEP: 10432/36938 -- GLOBAL_STEP: 1340200 | > loss_text_ce: 0.029418116435408592 (0.030900931504543976) | > loss_mel_ce: 0.1966450959444046 (nan) | > loss: 0.0008970762719400227 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13713612453155036) | > loader_time: 0.0046 (0.004655631384783741)  --> TIME: 2025-02-26 17:43:47 -- STEP: 10482/36938 -- GLOBAL_STEP: 1340250 | > loss_text_ce: 0.030724260956048965 (0.030899832469019013) | > loss_mel_ce: 2.938227415084839 (nan) | > loss: 0.01178155466914177 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13713612412705858) | > loader_time: 0.0046 (0.004655903224111691)  --> TIME: 2025-02-26 17:43:57 -- STEP: 10532/36938 -- GLOBAL_STEP: 1340300 | > loss_text_ce: 0.043853629380464554 (0.03090446132050061) | > loss_mel_ce: 2.630777359008789 (nan) | > loss: 0.01061361562460661 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.13715937497087952) | > loader_time: 0.0046 (0.004656023165141065)  --> TIME: 2025-02-26 17:44:06 -- STEP: 10582/36938 -- GLOBAL_STEP: 1340350 | > loss_text_ce: 0.03090319037437439 (0.03090394680258721) | > loss_mel_ce: 2.1450717449188232 (nan) | > loss: 0.008634821511805058 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13716415443305094) | > loader_time: 0.0046 (0.0046558312760524335)  --> TIME: 2025-02-26 17:44:15 -- STEP: 10632/36938 -- GLOBAL_STEP: 1340400 | > loss_text_ce: 0.030070332810282707 (0.030903061241094733) | > loss_mel_ce: 2.0932016372680664 (nan) | > loss: 0.008425682783126831 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.137134171657943) | > loader_time: 0.0046 (0.004655646663397409)  --> TIME: 2025-02-26 17:44:25 -- STEP: 10682/36938 -- GLOBAL_STEP: 1340450 | > loss_text_ce: 0.027110671624541283 (0.03090364159392798) | > loss_mel_ce: 2.9576199054718018 (nan) | > loss: 0.011844169348478317 (nan) | > current_lr: 5e-06 | > step_time: 0.1773 (0.1371474401309374) | > loader_time: 0.0048 (0.004655579372506728)  --> TIME: 2025-02-26 17:44:34 -- STEP: 10732/36938 -- GLOBAL_STEP: 1340500 | > loss_text_ce: 0.019584786146879196 (0.030896381943599793) | > loss_mel_ce: 2.129983901977539 (nan) | > loss: 0.008530035614967346 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13714517224781428) | > loader_time: 0.0045 (0.004655349845545015)  --> TIME: 2025-02-26 17:44:43 -- STEP: 10782/36938 -- GLOBAL_STEP: 1340550 | > loss_text_ce: 0.026597747579216957 (0.030889803107264375) | > loss_mel_ce: 2.2810592651367188 (nan) | > loss: 0.00915736984461546 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13714276253900742) | > loader_time: 0.0045 (0.004655129125400872)  --> TIME: 2025-02-26 17:44:53 -- STEP: 10832/36938 -- GLOBAL_STEP: 1340600 | > loss_text_ce: 0.03032545931637287 (0.030884227186018062) | > loss_mel_ce: 3.292879343032837 (nan) | > loss: 0.013187320902943611 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.13714391607213472) | > loader_time: 0.0045 (0.004655039600994965)  --> TIME: 2025-02-26 17:45:03 -- STEP: 10882/36938 -- GLOBAL_STEP: 1340650 | > loss_text_ce: 0.03325560316443443 (0.030887314073812983) | > loss_mel_ce: 2.913473606109619 (nan) | > loss: 0.011693370528519154 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13715876901528848) | > loader_time: 0.0049 (0.004655050631065506)  --> TIME: 2025-02-26 17:45:12 -- STEP: 10932/36938 -- GLOBAL_STEP: 1340700 | > loss_text_ce: 0.02944107912480831 (0.030892139790091352) | > loss_mel_ce: 2.8791539669036865 (nan) | > loss: 0.011542044579982758 (nan) | > current_lr: 5e-06 | > step_time: 0.1797 (0.13714625616890216) | > loader_time: 0.0048 (0.00465488093613795)  --> TIME: 2025-02-26 17:45:21 -- STEP: 10982/36938 -- GLOBAL_STEP: 1340750 | > loss_text_ce: 0.0327981598675251 (0.03089559333702708) | > loss_mel_ce: 2.6962029933929443 (nan) | > loss: 0.01082936953753233 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13712885427118995) | > loader_time: 0.0046 (0.004654658207626792)  --> TIME: 2025-02-26 17:45:30 -- STEP: 11032/36938 -- GLOBAL_STEP: 1340800 | > loss_text_ce: 0.03742051124572754 (0.030896954996042978) | > loss_mel_ce: 3.001505136489868 (nan) | > loss: 0.012059229426085949 (nan) | > current_lr: 5e-06 | > step_time: 0.1518 (0.13712043268167146) | > loader_time: 0.0048 (0.004654528482657072)  --> TIME: 2025-02-26 17:45:39 -- STEP: 11082/36938 -- GLOBAL_STEP: 1340850 | > loss_text_ce: 0.0349503718316555 (0.03089723072856486) | > loss_mel_ce: 2.257784605026245 (nan) | > loss: 0.009098154492676258 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.1371189102845128) | > loader_time: 0.0045 (0.0046546642497609725)  --> TIME: 2025-02-26 17:45:49 -- STEP: 11132/36938 -- GLOBAL_STEP: 1340900 | > loss_text_ce: 0.033513426780700684 (0.03089494576970998) | > loss_mel_ce: 3.4317266941070557 (nan) | > loss: 0.0137509535998106 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13712105815718229) | > loader_time: 0.0046 (0.004654885859014855)  --> TIME: 2025-02-26 17:45:58 -- STEP: 11182/36938 -- GLOBAL_STEP: 1340950 | > loss_text_ce: 0.03829481825232506 (0.03088485760860974) | > loss_mel_ce: 2.7222681045532227 (nan) | > loss: 0.010954615660011768 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13711786841561616) | > loader_time: 0.0045 (0.004654847579871473)  --> TIME: 2025-02-26 17:46:07 -- STEP: 11232/36938 -- GLOBAL_STEP: 1341000 | > loss_text_ce: 0.04225802794098854 (0.030888892444999202) | > loss_mel_ce: 1.9292335510253906 (nan) | > loss: 0.007823379710316658 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13710176607227767) | > loader_time: 0.0045 (0.004654710236777627)  --> TIME: 2025-02-26 17:46:17 -- STEP: 11282/36938 -- GLOBAL_STEP: 1341050 | > loss_text_ce: 0.030331287533044815 (0.030892360600803172) | > loss_mel_ce: 2.401709794998169 (nan) | > loss: 0.009650957770645618 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13710713073772438) | > loader_time: 0.0046 (0.004654689326266894)  --> TIME: 2025-02-26 17:46:26 -- STEP: 11332/36938 -- GLOBAL_STEP: 1341100 | > loss_text_ce: 0.030009349808096886 (0.030893051797240393) | > loss_mel_ce: 1.0646240711212158 (nan) | > loss: 0.00434378394857049 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13713610986496888) | > loader_time: 0.0045 (0.004654802936906763)  --> TIME: 2025-02-26 17:46:36 -- STEP: 11382/36938 -- GLOBAL_STEP: 1341150 | > loss_text_ce: 0.0229844618588686 (0.03089248119276413) | > loss_mel_ce: 2.8764243125915527 (nan) | > loss: 0.011505590751767159 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.1371356741319204) | > loader_time: 0.0046 (0.0046551350947319735)  --> TIME: 2025-02-26 17:46:45 -- STEP: 11432/36938 -- GLOBAL_STEP: 1341200 | > loss_text_ce: 0.02537933737039566 (0.030889250344391182) | > loss_mel_ce: 2.316478967666626 (nan) | > loss: 0.009293089620769024 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13715169797250695) | > loader_time: 0.0046 (0.004655395128077958)  --> TIME: 2025-02-26 17:46:55 -- STEP: 11482/36938 -- GLOBAL_STEP: 1341250 | > loss_text_ce: 0.03639228269457817 (0.030890680180828155) | > loss_mel_ce: 1.798067569732666 (nan) | > loss: 0.007279603276401758 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13715611715139078) | > loader_time: 0.0047 (0.004655422327140982)  --> TIME: 2025-02-26 17:47:04 -- STEP: 11532/36938 -- GLOBAL_STEP: 1341300 | > loss_text_ce: 0.029469940811395645 (0.030893288057662827) | > loss_mel_ce: 2.4929568767547607 (nan) | > loss: 0.01000963058322668 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13715298360642036) | > loader_time: 0.0045 (0.00465542518385493)  --> TIME: 2025-02-26 17:47:13 -- STEP: 11582/36938 -- GLOBAL_STEP: 1341350 | > loss_text_ce: 0.027000779286026955 (0.030896073836677883) | > loss_mel_ce: 2.699099063873291 (nan) | > loss: 0.010817856527864933 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13713464869095948) | > loader_time: 0.0045 (0.004655604493410119)  --> TIME: 2025-02-26 17:47:23 -- STEP: 11632/36938 -- GLOBAL_STEP: 1341400 | > loss_text_ce: 0.026820719242095947 (0.030893067886597802) | > loss_mel_ce: 2.595473527908325 (nan) | > loss: 0.010405929759144783 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.13714434229032263) | > loader_time: 0.0049 (0.00465574778585685)  --> TIME: 2025-02-26 17:47:32 -- STEP: 11682/36938 -- GLOBAL_STEP: 1341450 | > loss_text_ce: 0.03186724707484245 (0.030893400845573404) | > loss_mel_ce: 0.0343424491584301 (nan) | > loss: 0.00026273689582012594 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13715050210693197) | > loader_time: 0.0046 (0.0046557688668174)  --> TIME: 2025-02-26 17:47:42 -- STEP: 11732/36938 -- GLOBAL_STEP: 1341500 | > loss_text_ce: 0.027423730120062828 (0.03088951711083276) | > loss_mel_ce: 2.8291916847229004 (nan) | > loss: 0.011335775256156921 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.13717983027509276) | > loader_time: 0.0048 (0.004656018310141103)  --> TIME: 2025-02-26 17:47:51 -- STEP: 11782/36938 -- GLOBAL_STEP: 1341550 | > loss_text_ce: 0.032497845590114594 (0.030889831229981443) | > loss_mel_ce: 3.128425359725952 (nan) | > loss: 0.012543346732854843 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13717287735347375) | > loader_time: 0.0046 (0.0046559957510100575)  --> TIME: 2025-02-26 17:48:00 -- STEP: 11832/36938 -- GLOBAL_STEP: 1341600 | > loss_text_ce: 0.03917061537504196 (0.030896154115238793) | > loss_mel_ce: 2.0123393535614014 (nan) | > loss: 0.008140912279486656 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.1371570802202416) | > loader_time: 0.0045 (0.004656082556971535)  --> TIME: 2025-02-26 17:48:10 -- STEP: 11882/36938 -- GLOBAL_STEP: 1341650 | > loss_text_ce: 0.03149263188242912 (0.03089115554757637) | > loss_mel_ce: 3.0588929653167725 (nan) | > loss: 0.012263435870409012 (nan) | > current_lr: 5e-06 | > step_time: 0.1981 (0.1371798850815422) | > loader_time: 0.0062 (0.004656288463686946)  --> TIME: 2025-02-26 17:48:19 -- STEP: 11932/36938 -- GLOBAL_STEP: 1341700 | > loss_text_ce: 0.03206571191549301 (0.030887646986846914) | > loss_mel_ce: 2.9383766651153564 (nan) | > loss: 0.011787470430135727 (nan) | > current_lr: 5e-06 | > step_time: 0.1558 (0.13718804356398898) | > loader_time: 0.0047 (0.004656299444160909)  --> TIME: 2025-02-26 17:48:29 -- STEP: 11982/36938 -- GLOBAL_STEP: 1341750 | > loss_text_ce: 0.036692213267087936 (0.030885405446318938) | > loss_mel_ce: 1.8802108764648438 (nan) | > loss: 0.007606758736073971 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1371903213947653) | > loader_time: 0.0047 (0.004656339941309749)  --> TIME: 2025-02-26 17:48:38 -- STEP: 12032/36938 -- GLOBAL_STEP: 1341800 | > loss_text_ce: 0.03989202529191971 (0.03088554716022701) | > loss_mel_ce: 2.379605770111084 (nan) | > loss: 0.00960118230432272 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13715738920059029) | > loader_time: 0.0046 (0.004656290853435716)  --> TIME: 2025-02-26 17:48:47 -- STEP: 12082/36938 -- GLOBAL_STEP: 1341850 | > loss_text_ce: 0.03370130807161331 (0.030882789124260637) | > loss_mel_ce: 2.458984375 (nan) | > loss: 0.009891610592603683 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13716235484060288) | > loader_time: 0.0045 (0.004656275639646897)  --> TIME: 2025-02-26 17:48:56 -- STEP: 12132/36938 -- GLOBAL_STEP: 1341900 | > loss_text_ce: 0.02227199263870716 (0.03087685379971794) | > loss_mel_ce: 2.6975629329681396 (nan) | > loss: 0.010792996734380722 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13715515710951476) | > loader_time: 0.0047 (0.004656249369248071)  --> TIME: 2025-02-26 17:49:06 -- STEP: 12182/36938 -- GLOBAL_STEP: 1341950 | > loss_text_ce: 0.038380999118089676 (0.03088221054485499) | > loss_mel_ce: 1.962667465209961 (nan) | > loss: 0.007940669544041157 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1371796148161139) | > loader_time: 0.0045 (0.004656458034086399)  --> TIME: 2025-02-26 17:49:15 -- STEP: 12232/36938 -- GLOBAL_STEP: 1342000 | > loss_text_ce: 0.040253233164548874 (0.030878484627407683) | > loss_mel_ce: 2.65387225151062 (nan) | > loss: 0.010690974071621895 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13716771608088235) | > loader_time: 0.0045 (0.004656329292967262)  --> TIME: 2025-02-26 17:49:25 -- STEP: 12282/36938 -- GLOBAL_STEP: 1342050 | > loss_text_ce: 0.027903910726308823 (0.030875405734553545) | > loss_mel_ce: 3.071148633956909 (nan) | > loss: 0.012297827750444412 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.13718360286837528) | > loader_time: 0.0046 (0.004656336649565662)  --> TIME: 2025-02-26 17:49:34 -- STEP: 12332/36938 -- GLOBAL_STEP: 1342100 | > loss_text_ce: 0.03444027528166771 (0.030877055519818113) | > loss_mel_ce: 2.6105780601501465 (nan) | > loss: 0.010496105067431927 (nan) | > current_lr: 5e-06 | > step_time: 0.1352 (0.1371855685318829) | > loader_time: 0.0045 (0.004656251823210016)  --> TIME: 2025-02-26 17:49:44 -- STEP: 12382/36938 -- GLOBAL_STEP: 1342150 | > loss_text_ce: 0.03357455134391785 (0.03088074937322755) | > loss_mel_ce: 2.438525438308716 (nan) | > loss: 0.009809921495616436 (nan) | > current_lr: 5e-06 | > step_time: 0.1439 (0.13718768763168657) | > loader_time: 0.0046 (0.004656442358158395)  --> TIME: 2025-02-26 17:49:53 -- STEP: 12432/36938 -- GLOBAL_STEP: 1342200 | > loss_text_ce: 0.024995440617203712 (0.030881582513492628) | > loss_mel_ce: 2.2119359970092773 (nan) | > loss: 0.008876712061464787 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1372005099111852) | > loader_time: 0.0045 (0.004656523005842714)  --> TIME: 2025-02-26 17:50:03 -- STEP: 12482/36938 -- GLOBAL_STEP: 1342250 | > loss_text_ce: 0.031126825138926506 (0.030886390513006734) | > loss_mel_ce: 2.6703901290893555 (nan) | > loss: 0.01072030607610941 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13722037613878976) | > loader_time: 0.0046 (0.0046566286409461)  --> TIME: 2025-02-26 17:50:12 -- STEP: 12532/36938 -- GLOBAL_STEP: 1342300 | > loss_text_ce: 0.038054391741752625 (0.030890339934249268) | > loss_mel_ce: 3.2623422145843506 (nan) | > loss: 0.01309681311249733 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.13722196918609939) | > loader_time: 0.0048 (0.0046567036212917644)  --> TIME: 2025-02-26 17:50:22 -- STEP: 12582/36938 -- GLOBAL_STEP: 1342350 | > loss_text_ce: 0.03905513510107994 (0.030891650416197687) | > loss_mel_ce: 2.471061944961548 (nan) | > loss: 0.009960782714188099 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.13721949181369503) | > loader_time: 0.0047 (0.004656718410533797)  --> TIME: 2025-02-26 17:50:31 -- STEP: 12632/36938 -- GLOBAL_STEP: 1342400 | > loss_text_ce: 0.030538052320480347 (0.030893318102825692) | > loss_mel_ce: 2.9715993404388428 (nan) | > loss: 0.011913244612514973 (nan) | > current_lr: 5e-06 | > step_time: 0.1707 (0.13721825625037312) | > loader_time: 0.0049 (0.004656590054199785)  --> TIME: 2025-02-26 17:50:40 -- STEP: 12682/36938 -- GLOBAL_STEP: 1342450 | > loss_text_ce: 0.03981059044599533 (0.030891855757225566) | > loss_mel_ce: 0.2338358610868454 (nan) | > loss: 0.001085898606106639 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13721938470507503) | > loader_time: 0.0046 (0.004656513864131905)  --> TIME: 2025-02-26 17:50:49 -- STEP: 12732/36938 -- GLOBAL_STEP: 1342500 | > loss_text_ce: 0.03772299736738205 (0.030893845539655625) | > loss_mel_ce: 2.8471009731292725 (nan) | > loss: 0.011447714641690254 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13720427949287028) | > loader_time: 0.0045 (0.004656230714236507)  --> TIME: 2025-02-26 17:50:59 -- STEP: 12782/36938 -- GLOBAL_STEP: 1342550 | > loss_text_ce: 0.027332842350006104 (0.030895896416521504) | > loss_mel_ce: 2.635714054107666 (nan) | > loss: 0.010567646473646164 (nan) | > current_lr: 5e-06 | > step_time: 0.1697 (0.13723797366160656) | > loader_time: 0.0048 (0.004656107823737936)  --> TIME: 2025-02-26 17:51:09 -- STEP: 12832/36938 -- GLOBAL_STEP: 1342600 | > loss_text_ce: 0.030925534665584564 (0.030892352324812047) | > loss_mel_ce: 2.2494144439697266 (nan) | > loss: 0.00904896855354309 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13726376968800605) | > loader_time: 0.0046 (0.004655949511284255)  --> TIME: 2025-02-26 17:51:18 -- STEP: 12882/36938 -- GLOBAL_STEP: 1342650 | > loss_text_ce: 0.038582220673561096 (0.03089156762686392) | > loss_mel_ce: 1.35888671875 (nan) | > loss: 0.005545511841773987 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.1372559319888378) | > loader_time: 0.0047 (0.0046556920410035525)  --> TIME: 2025-02-26 17:51:27 -- STEP: 12932/36938 -- GLOBAL_STEP: 1342700 | > loss_text_ce: 0.03421685844659805 (0.03089647496265624) | > loss_mel_ce: 2.9112908840179443 (nan) | > loss: 0.011688523925840855 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.1372370158154791) | > loader_time: 0.0046 (0.004655420577382299)  --> TIME: 2025-02-26 17:51:37 -- STEP: 12982/36938 -- GLOBAL_STEP: 1342750 | > loss_text_ce: 0.022931741550564766 (0.030893204639642968) | > loss_mel_ce: 3.738163471221924 (nan) | > loss: 0.014924982562661171 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13725936045504905) | > loader_time: 0.0045 (0.004655397208294179)  --> TIME: 2025-02-26 17:51:46 -- STEP: 13032/36938 -- GLOBAL_STEP: 1342800 | > loss_text_ce: 0.027749821543693542 (0.030892847013850443) | > loss_mel_ce: 2.942573308944702 (nan) | > loss: 0.011786997318267822 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13725578945937386) | > loader_time: 0.0046 (0.004655386349260629)  --> TIME: 2025-02-26 17:51:55 -- STEP: 13082/36938 -- GLOBAL_STEP: 1342850 | > loss_text_ce: 0.029919253662228584 (0.03089279840282294) | > loss_mel_ce: 3.15516996383667 (nan) | > loss: 0.01263924315571785 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.137249178231045) | > loader_time: 0.0046 (0.004655305644164956)  --> TIME: 2025-02-26 17:52:05 -- STEP: 13132/36938 -- GLOBAL_STEP: 1342900 | > loss_text_ce: 0.029275283217430115 (0.030893315844476948) | > loss_mel_ce: 2.123549461364746 (nan) | > loss: 0.008542955853044987 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.13725561802248532) | > loader_time: 0.0046 (0.0046553626824471055)  --> TIME: 2025-02-26 17:52:14 -- STEP: 13182/36938 -- GLOBAL_STEP: 1342950 | > loss_text_ce: 0.03495359420776367 (0.03088730277622605) | > loss_mel_ce: 3.1176161766052246 (nan) | > loss: 0.012510198168456554 (nan) | > current_lr: 5e-06 | > step_time: 0.1813 (0.13726907481085163) | > loader_time: 0.0048 (0.004655345024134348)  --> TIME: 2025-02-26 17:52:24 -- STEP: 13232/36938 -- GLOBAL_STEP: 1343000 | > loss_text_ce: 0.03260178118944168 (0.030893113539674395) | > loss_mel_ce: 2.1354806423187256 (nan) | > loss: 0.008603502064943314 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1372769790304039) | > loader_time: 0.0045 (0.004655850931839898)  --> TIME: 2025-02-26 17:52:33 -- STEP: 13282/36938 -- GLOBAL_STEP: 1343050 | > loss_text_ce: 0.033806879073381424 (0.03089579658604824) | > loss_mel_ce: 2.7825472354888916 (nan) | > loss: 0.011176008731126785 (nan) | > current_lr: 5e-06 | > step_time: 0.1842 (0.13726378341626558) | > loader_time: 0.0049 (0.004655763177048854)  --> TIME: 2025-02-26 17:52:43 -- STEP: 13332/36938 -- GLOBAL_STEP: 1343100 | > loss_text_ce: 0.032452795654535294 (0.030896397501680394) | > loss_mel_ce: 2.854104995727539 (nan) | > loss: 0.011454595252871513 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13726652464350175) | > loader_time: 0.0049 (0.004655835884596682)  --> TIME: 2025-02-26 17:52:52 -- STEP: 13382/36938 -- GLOBAL_STEP: 1343150 | > loss_text_ce: 0.03476208820939064 (0.030899571663059906) | > loss_mel_ce: 3.2958099842071533 (nan) | > loss: 0.01321655698120594 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13726565355392964) | > loader_time: 0.0045 (0.004655666939971015)  --> TIME: 2025-02-26 17:53:01 -- STEP: 13432/36938 -- GLOBAL_STEP: 1343200 | > loss_text_ce: 0.03690115734934807 (0.03090750823734703) | > loss_mel_ce: 2.688389539718628 (nan) | > loss: 0.010814646258950233 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.1372653567201682) | > loader_time: 0.0045 (0.004655693988845486)  --> TIME: 2025-02-26 17:53:11 -- STEP: 13482/36938 -- GLOBAL_STEP: 1343250 | > loss_text_ce: 0.033430248498916626 (0.030908845628480078) | > loss_mel_ce: 0.01731363870203495 (nan) | > loss: 0.0002013646299019456 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.13728260795894312) | > loader_time: 0.0047 (0.0046555600698969316)  --> TIME: 2025-02-26 17:53:20 -- STEP: 13532/36938 -- GLOBAL_STEP: 1343300 | > loss_text_ce: 0.03243192285299301 (0.0309054231136735) | > loss_mel_ce: 2.95224666595459 (nan) | > loss: 0.011843962594866753 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1372646417731968) | > loader_time: 0.0046 (0.004655439843801643)  --> TIME: 2025-02-26 17:53:29 -- STEP: 13582/36938 -- GLOBAL_STEP: 1343350 | > loss_text_ce: 0.029520468786358833 (0.030909443599952405) | > loss_mel_ce: 3.3331263065338135 (nan) | > loss: 0.01334383711218834 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.13726419162160444) | > loader_time: 0.0047 (0.004655553620161075)  --> TIME: 2025-02-26 17:53:39 -- STEP: 13632/36938 -- GLOBAL_STEP: 1343400 | > loss_text_ce: 0.02993249148130417 (0.030906013792407743) | > loss_mel_ce: 3.0633723735809326 (nan) | > loss: 0.01227501966059208 (nan) | > current_lr: 5e-06 | > step_time: 0.1523 (0.1372621656863344) | > loader_time: 0.0047 (0.0046555193517129785)  --> TIME: 2025-02-26 17:53:48 -- STEP: 13682/36938 -- GLOBAL_STEP: 1343450 | > loss_text_ce: 0.03888588026165962 (0.030910928045433387) | > loss_mel_ce: 2.334357738494873 (nan) | > loss: 0.009417633526027203 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13729379039778922) | > loader_time: 0.0046 (0.004655478555127001)  --> TIME: 2025-02-26 17:53:58 -- STEP: 13732/36938 -- GLOBAL_STEP: 1343500 | > loss_text_ce: 0.03019036166369915 (0.03090906465241478) | > loss_mel_ce: 2.4259324073791504 (nan) | > loss: 0.009746519848704338 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.1372822219798353) | > loader_time: 0.0045 (0.004655406334780063)  --> TIME: 2025-02-26 17:54:07 -- STEP: 13782/36938 -- GLOBAL_STEP: 1343550 | > loss_text_ce: 0.0372164323925972 (0.030906410922402167) | > loss_mel_ce: 2.4451022148132324 (nan) | > loss: 0.009850471280515194 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13729307660515408) | > loader_time: 0.0046 (0.004655419508685325)  --> TIME: 2025-02-26 17:54:17 -- STEP: 13832/36938 -- GLOBAL_STEP: 1343600 | > loss_text_ce: 0.027842946350574493 (0.03090708974964713) | > loss_mel_ce: 2.375257968902588 (nan) | > loss: 0.00953611545264721 (nan) | > current_lr: 5e-06 | > step_time: 0.1886 (0.13729887776736105) | > loader_time: 0.0049 (0.004655401630164432)  --> TIME: 2025-02-26 17:54:26 -- STEP: 13882/36938 -- GLOBAL_STEP: 1343650 | > loss_text_ce: 0.033733487129211426 (0.03090048205843948) | > loss_mel_ce: 2.720770835876465 (nan) | > loss: 0.010930572636425495 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.1372975098813685) | > loader_time: 0.0046 (0.004655369659817332)  --> TIME: 2025-02-26 17:54:35 -- STEP: 13932/36938 -- GLOBAL_STEP: 1343700 | > loss_text_ce: 0.03526393696665764 (0.030901438697292585) | > loss_mel_ce: 2.6636712551116943 (nan) | > loss: 0.010710061527788639 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1372929671150908) | > loader_time: 0.0046 (0.00465553422238407)  --> TIME: 2025-02-26 17:54:45 -- STEP: 13982/36938 -- GLOBAL_STEP: 1343750 | > loss_text_ce: 0.022491995245218277 (0.03089920718209135) | > loss_mel_ce: 2.7578718662261963 (nan) | > loss: 0.011033190414309502 (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.13730126001210452) | > loader_time: 0.0048 (0.004655684614502091)  --> TIME: 2025-02-26 17:54:54 -- STEP: 14032/36938 -- GLOBAL_STEP: 1343800 | > loss_text_ce: 0.027840418741106987 (0.030899659625992355) | > loss_mel_ce: 1.5182538032531738 (nan) | > loss: 0.006135294679552317 (nan) | > current_lr: 5e-06 | > step_time: 0.1077 (0.13729909181323002) | > loader_time: 0.0045 (0.004655634510748197)  --> TIME: 2025-02-26 17:55:03 -- STEP: 14082/36938 -- GLOBAL_STEP: 1343850 | > loss_text_ce: 0.036383695900440216 (0.030899575564481488) | > loss_mel_ce: 2.6193089485168457 (nan) | > loss: 0.010538463480770588 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13729806316117316) | > loader_time: 0.0045 (0.004655578684661217)  --> TIME: 2025-02-26 17:55:13 -- STEP: 14132/36938 -- GLOBAL_STEP: 1343900 | > loss_text_ce: 0.027202779427170753 (0.030900115139555463) | > loss_mel_ce: 3.085211753845215 (nan) | > loss: 0.012350852601230145 (nan) | > current_lr: 5e-06 | > step_time: 0.178 (0.1373153198146633) | > loader_time: 0.0048 (0.00465564210860951)  --> TIME: 2025-02-26 17:55:22 -- STEP: 14182/36938 -- GLOBAL_STEP: 1343950 | > loss_text_ce: 0.014091824181377888 (0.030896787936032868) | > loss_mel_ce: 1.14781653881073 (nan) | > loss: 0.004610748030245304 (nan) | > current_lr: 5e-06 | > step_time: 0.1077 (0.13730607796347458) | > loader_time: 0.0045 (0.004655590801779665)  --> TIME: 2025-02-26 17:55:32 -- STEP: 14232/36938 -- GLOBAL_STEP: 1344000 | > loss_text_ce: 0.03807967156171799 (0.03089831085354608) | > loss_mel_ce: 0.0037714641075581312 (nan) | > loss: 0.0001660759444348514 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13729716320598057) | > loader_time: 0.0046 (0.004655705971152275)  --> TIME: 2025-02-26 17:55:41 -- STEP: 14282/36938 -- GLOBAL_STEP: 1344050 | > loss_text_ce: 0.03222772479057312 (0.030902316039422898) | > loss_mel_ce: 2.6010353565216064 (nan) | > loss: 0.010449456982314587 (nan) | > current_lr: 5e-06 | > step_time: 0.1517 (0.13729955683968487) | > loader_time: 0.0046 (0.00465577642985409)  --> TIME: 2025-02-26 17:55:50 -- STEP: 14332/36938 -- GLOBAL_STEP: 1344100 | > loss_text_ce: 0.0370326042175293 (0.03090430028900216) | > loss_mel_ce: 2.7852463722229004 (nan) | > loss: 0.011199519969522953 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.1373028936513869) | > loader_time: 0.0046 (0.004655783099233853)  --> TIME: 2025-02-26 17:56:00 -- STEP: 14382/36938 -- GLOBAL_STEP: 1344150 | > loss_text_ce: 0.03676743805408478 (0.03090650592065473) | > loss_mel_ce: 0.2570277750492096 (nan) | > loss: 0.0011658541625365615 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.1373123131612629) | > loader_time: 0.0046 (0.004655892751823957)  --> TIME: 2025-02-26 17:56:09 -- STEP: 14432/36938 -- GLOBAL_STEP: 1344200 | > loss_text_ce: 0.030918678268790245 (0.03091112655854527) | > loss_mel_ce: 2.98940372467041 (nan) | > loss: 0.011985406279563904 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.1373094896023923) | > loader_time: 0.0046 (0.004656114956227742)  --> TIME: 2025-02-26 17:56:19 -- STEP: 14482/36938 -- GLOBAL_STEP: 1344250 | > loss_text_ce: 0.03282133489847183 (0.03091093137330521) | > loss_mel_ce: 2.765028238296509 (nan) | > loss: 0.011102578602731228 (nan) | > current_lr: 5e-06 | > step_time: 0.1732 (0.13731293190317478) | > loader_time: 0.0047 (0.004656072809392946)  --> TIME: 2025-02-26 17:56:28 -- STEP: 14532/36938 -- GLOBAL_STEP: 1344300 | > loss_text_ce: 0.019901776686310768 (0.03090808947235187) | > loss_mel_ce: 2.2953360080718994 (nan) | > loss: 0.00918745156377554 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13729492091555062) | > loader_time: 0.0045 (0.004655903917418146)  --> TIME: 2025-02-26 17:56:37 -- STEP: 14582/36938 -- GLOBAL_STEP: 1344350 | > loss_text_ce: 0.03319922462105751 (0.03090708671472231) | > loss_mel_ce: 2.9144110679626465 (nan) | > loss: 0.011696867644786835 (nan) | > current_lr: 5e-06 | > step_time: 0.1815 (0.1372967750966312) | > loader_time: 0.0049 (0.004655980798969573)  --> TIME: 2025-02-26 17:56:46 -- STEP: 14632/36938 -- GLOBAL_STEP: 1344400 | > loss_text_ce: 0.026902751997113228 (0.0309094412204506) | > loss_mel_ce: 2.6013057231903076 (nan) | > loss: 0.010429399088025093 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.1373022574750109) | > loader_time: 0.0047 (0.004655993542037678)  --> TIME: 2025-02-26 17:56:56 -- STEP: 14682/36938 -- GLOBAL_STEP: 1344450 | > loss_text_ce: 0.022076362743973732 (0.030907759425322808) | > loss_mel_ce: 1.842997431755066 (nan) | > loss: 0.0074010868556797504 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13730139124054075) | > loader_time: 0.0045 (0.00465598599720092)  --> TIME: 2025-02-26 17:57:05 -- STEP: 14732/36938 -- GLOBAL_STEP: 1344500 | > loss_text_ce: 0.028938336297869682 (0.030904446854235865) | > loss_mel_ce: 2.3900146484375 (nan) | > loss: 0.009599019773304462 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13729620167589562) | > loader_time: 0.0046 (0.004656134401363358)  --> TIME: 2025-02-26 17:57:15 -- STEP: 14782/36938 -- GLOBAL_STEP: 1344550 | > loss_text_ce: 0.03204604238271713 (0.03090752915023347) | > loss_mel_ce: 0.013580322265625 (nan) | > loss: 0.00018105701019521803 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13730706476323296) | > loader_time: 0.0046 (0.004656330156255325)  --> TIME: 2025-02-26 17:57:24 -- STEP: 14832/36938 -- GLOBAL_STEP: 1344600 | > loss_text_ce: 0.03575845807790756 (0.030907980590249223) | > loss_mel_ce: 1.448807954788208 (nan) | > loss: 0.005891136825084686 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1373047755374105) | > loader_time: 0.0046 (0.004656459810664351)  --> TIME: 2025-02-26 17:57:33 -- STEP: 14882/36938 -- GLOBAL_STEP: 1344650 | > loss_text_ce: 0.0182318277657032 (0.03090628292507314) | > loss_mel_ce: 2.05875301361084 (nan) | > loss: 0.008242003619670868 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13730235774900595) | > loader_time: 0.0045 (0.004656465059005352)  --> TIME: 2025-02-26 17:57:42 -- STEP: 14932/36938 -- GLOBAL_STEP: 1344700 | > loss_text_ce: 0.023346008732914925 (0.03090885891254139) | > loss_mel_ce: 1.8413262367248535 (nan) | > loss: 0.007399493828415871 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.1372977610634569) | > loader_time: 0.0045 (0.004656562465398967)  --> TIME: 2025-02-26 17:57:52 -- STEP: 14982/36938 -- GLOBAL_STEP: 1344750 | > loss_text_ce: 0.03456887975335121 (0.03090734895211951) | > loss_mel_ce: 2.047001838684082 (nan) | > loss: 0.008260201662778854 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13730792896783547) | > loader_time: 0.0045 (0.004656505336463569)  --> TIME: 2025-02-26 17:58:01 -- STEP: 15032/36938 -- GLOBAL_STEP: 1344800 | > loss_text_ce: 0.03871435299515724 (0.030908267227631705) | > loss_mel_ce: 2.9234330654144287 (nan) | > loss: 0.011754553765058517 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.1373038980128474) | > loader_time: 0.0047 (0.004656372233995291)  --> TIME: 2025-02-26 17:58:11 -- STEP: 15082/36938 -- GLOBAL_STEP: 1344850 | > loss_text_ce: 0.028037576004862785 (0.03090878569114938) | > loss_mel_ce: 2.5071961879730225 (nan) | > loss: 0.010060451924800873 (nan) | > current_lr: 5e-06 | > step_time: 0.1617 (0.13731807864735093) | > loader_time: 0.0046 (0.004656262619712594)  --> TIME: 2025-02-26 17:58:20 -- STEP: 15132/36938 -- GLOBAL_STEP: 1344900 | > loss_text_ce: 0.03847740590572357 (0.03090808295470397) | > loss_mel_ce: 1.5032958984375 (nan) | > loss: 0.006118148565292358 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13731838539419994) | > loader_time: 0.0045 (0.004656435162057454)  --> TIME: 2025-02-26 17:58:29 -- STEP: 15182/36938 -- GLOBAL_STEP: 1344950 | > loss_text_ce: 0.02993916906416416 (0.03090503998520963) | > loss_mel_ce: 0.002552032470703125 (nan) | > loss: 0.0001289333449676633 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13731329578637852) | > loader_time: 0.0045 (0.004656235136198691)  --> TIME: 2025-02-26 17:58:39 -- STEP: 15232/36938 -- GLOBAL_STEP: 1345000 | > loss_text_ce: 0.03055013157427311 (0.030906103721027655) | > loss_mel_ce: 2.921049118041992 (nan) | > loss: 0.011712696403265 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13733308300563762) | > loader_time: 0.0047 (0.004656190663075245)  --> TIME: 2025-02-26 17:58:48 -- STEP: 15282/36938 -- GLOBAL_STEP: 1345050 | > loss_text_ce: 0.03756999969482422 (0.030909698232531816) | > loss_mel_ce: 2.0195484161376953 (nan) | > loss: 0.008163169026374817 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13732281368981006) | > loader_time: 0.0045 (0.00465609204814378)  --> TIME: 2025-02-26 17:58:57 -- STEP: 15332/36938 -- GLOBAL_STEP: 1345100 | > loss_text_ce: 0.030519863590598106 (0.030910114603336515) | > loss_mel_ce: 2.5628416538238525 (nan) | > loss: 0.010291118174791336 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13730889575897537) | > loader_time: 0.0047 (0.00465607189056054)  --> TIME: 2025-02-26 17:59:07 -- STEP: 15382/36938 -- GLOBAL_STEP: 1345150 | > loss_text_ce: 0.03207063302397728 (0.030909313514089775) | > loss_mel_ce: 2.863466501235962 (nan) | > loss: 0.011490227654576302 (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.13731166985050128) | > loader_time: 0.0049 (0.0046559564314928885)  --> TIME: 2025-02-26 17:59:16 -- STEP: 15432/36938 -- GLOBAL_STEP: 1345200 | > loss_text_ce: 0.030191490426659584 (0.030906081350853144) | > loss_mel_ce: 2.2881979942321777 (nan) | > loss: 0.00919995829463005 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13732067997310982) | > loader_time: 0.0046 (0.004655821342552306)  --> TIME: 2025-02-26 17:59:25 -- STEP: 15482/36938 -- GLOBAL_STEP: 1345250 | > loss_text_ce: 0.02313545159995556 (0.030907493478174255) | > loss_mel_ce: 0.8154292702674866 (nan) | > loss: 0.0033276379108428955 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.13730578466589602) | > loader_time: 0.0045 (0.004655799574975802)  --> TIME: 2025-02-26 17:59:34 -- STEP: 15532/36938 -- GLOBAL_STEP: 1345300 | > loss_text_ce: 0.03309933468699455 (0.030908418326274553) | > loss_mel_ce: 2.154921054840088 (nan) | > loss: 0.008682621642947197 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.1372942873383501) | > loader_time: 0.0047 (0.004655688502203394)  --> TIME: 2025-02-26 17:59:44 -- STEP: 15582/36938 -- GLOBAL_STEP: 1345350 | > loss_text_ce: 0.0339217334985733 (0.03091356673869868) | > loss_mel_ce: 2.970202684402466 (nan) | > loss: 0.011921129189431667 (nan) | > current_lr: 5e-06 | > step_time: 0.1642 (0.13727979368705093) | > loader_time: 0.0047 (0.004655482955377376)  --> TIME: 2025-02-26 17:59:52 -- STEP: 15632/36938 -- GLOBAL_STEP: 1345400 | > loss_text_ce: 0.03169365972280502 (0.030914008051857397) | > loss_mel_ce: 2.812382936477661 (nan) | > loss: 0.01128601934760809 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13725565573698173) | > loader_time: 0.0045 (0.004655242179774944)  --> TIME: 2025-02-26 18:00:02 -- STEP: 15682/36938 -- GLOBAL_STEP: 1345450 | > loss_text_ce: 0.027973221614956856 (0.030915551702001987) | > loss_mel_ce: 2.2852563858032227 (nan) | > loss: 0.00917948316782713 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13726324546156202) | > loader_time: 0.0046 (0.004655155641755009)  --> TIME: 2025-02-26 18:00:11 -- STEP: 15732/36938 -- GLOBAL_STEP: 1345500 | > loss_text_ce: 0.030091144144535065 (0.03091510988393869) | > loss_mel_ce: 3.144197463989258 (nan) | > loss: 0.012596383690834045 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13726519730568168) | > loader_time: 0.0045 (0.004655071229932266)  --> TIME: 2025-02-26 18:00:21 -- STEP: 15782/36938 -- GLOBAL_STEP: 1345550 | > loss_text_ce: 0.030270220711827278 (0.030913706417831054) | > loss_mel_ce: 2.189849853515625 (nan) | > loss: 0.00881000142544508 (nan) | > current_lr: 5e-06 | > step_time: 0.1767 (0.1372525127739858) | > loader_time: 0.0048 (0.004654861179357123)  --> TIME: 2025-02-26 18:00:30 -- STEP: 15832/36938 -- GLOBAL_STEP: 1345600 | > loss_text_ce: 0.031213216483592987 (0.03091216476185543) | > loss_mel_ce: 2.2413151264190674 (nan) | > loss: 0.00901797041296959 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13724333690898227) | > loader_time: 0.0046 (0.0046548589183080654)  --> TIME: 2025-02-26 18:00:39 -- STEP: 15882/36938 -- GLOBAL_STEP: 1345650 | > loss_text_ce: 0.03303641825914383 (0.03091199637245011) | > loss_mel_ce: 2.549025297164917 (nan) | > loss: 0.010246277786791325 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13722406262610448) | > loader_time: 0.0046 (0.00465466409917893)  --> TIME: 2025-02-26 18:00:49 -- STEP: 15932/36938 -- GLOBAL_STEP: 1345700 | > loss_text_ce: 0.035530056804418564 (0.030911030256897426) | > loss_mel_ce: 0.9585774540901184 (nan) | > loss: 0.003944871015846729 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13724037213987536) | > loader_time: 0.0047 (0.004654954597933818)  --> TIME: 2025-02-26 18:00:58 -- STEP: 15982/36938 -- GLOBAL_STEP: 1345750 | > loss_text_ce: 0.02953978069126606 (0.030910033497549366) | > loss_mel_ce: 2.546555280685425 (nan) | > loss: 0.0102225998416543 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13724285515568127) | > loader_time: 0.0044 (0.004655066561063408)  --> TIME: 2025-02-26 18:01:07 -- STEP: 16032/36938 -- GLOBAL_STEP: 1345800 | > loss_text_ce: 0.02781803160905838 (0.030917493735890634) | > loss_mel_ce: 2.3370089530944824 (nan) | > loss: 0.009384234435856342 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.13724763972495124) | > loader_time: 0.0046 (0.004655009258293105)  --> TIME: 2025-02-26 18:01:17 -- STEP: 16082/36938 -- GLOBAL_STEP: 1345850 | > loss_text_ce: 0.0263223759829998 (0.030913282314905203) | > loss_mel_ce: 0.0031194686889648438 (nan) | > loss: 0.00011683272168738768 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13726062552873472) | > loader_time: 0.0045 (0.0046548721520791895)  --> TIME: 2025-02-26 18:01:26 -- STEP: 16132/36938 -- GLOBAL_STEP: 1345900 | > loss_text_ce: 0.030810393393039703 (0.03090989680060973) | > loss_mel_ce: 2.2297799587249756 (nan) | > loss: 0.008970596827566624 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13725695323624854) | > loader_time: 0.0046 (0.0046550381015141)  --> TIME: 2025-02-26 18:01:36 -- STEP: 16182/36938 -- GLOBAL_STEP: 1345950 | > loss_text_ce: 0.03140050917863846 (0.03091034469330075) | > loss_mel_ce: 3.1793723106384277 (nan) | > loss: 0.012741162441670895 (nan) | > current_lr: 5e-06 | > step_time: 0.1891 (0.13725137149222386) | > loader_time: 0.0049 (0.004655089017089872)  --> TIME: 2025-02-26 18:01:45 -- STEP: 16232/36938 -- GLOBAL_STEP: 1346000 | > loss_text_ce: 0.0341828428208828 (0.030914464322137503) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1372644415096792) | > loader_time: 0.0046 (0.004655107451871792)  --> TIME: 2025-02-26 18:01:55 -- STEP: 16282/36938 -- GLOBAL_STEP: 1346050 | > loss_text_ce: 0.03143541142344475 (0.030913282430692666) | > loss_mel_ce: 3.204568862915039 (nan) | > loss: 0.01284128800034523 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.1372709397293838) | > loader_time: 0.0047 (0.004655107747804119)  --> TIME: 2025-02-26 18:02:04 -- STEP: 16332/36938 -- GLOBAL_STEP: 1346100 | > loss_text_ce: 0.02717883326113224 (0.030909598412539856) | > loss_mel_ce: 2.436999797821045 (nan) | > loss: 0.00977848656475544 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13726698047181068) | > loader_time: 0.0045 (0.0046551254868303)  --> TIME: 2025-02-26 18:02:13 -- STEP: 16382/36938 -- GLOBAL_STEP: 1346150 | > loss_text_ce: 0.0329832062125206 (0.030904704024922) | > loss_mel_ce: 2.064880847930908 (nan) | > loss: 0.00832485780119896 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13727073941005477) | > loader_time: 0.0047 (0.004655139610133113)  --> TIME: 2025-02-26 18:02:23 -- STEP: 16432/36938 -- GLOBAL_STEP: 1346200 | > loss_text_ce: 0.03760652244091034 (0.030906049967608564) | > loss_mel_ce: 2.980550765991211 (nan) | > loss: 0.011976814828813076 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13727186492552654) | > loader_time: 0.0046 (0.004655171044265166)  --> TIME: 2025-02-26 18:02:32 -- STEP: 16482/36938 -- GLOBAL_STEP: 1346250 | > loss_text_ce: 0.021821508184075356 (0.030902670774496512) | > loss_mel_ce: 2.381361484527588 (nan) | > loss: 0.009536441415548325 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13727276083075174) | > loader_time: 0.0046 (0.004655082355120157)  --> TIME: 2025-02-26 18:02:42 -- STEP: 16532/36938 -- GLOBAL_STEP: 1346300 | > loss_text_ce: 0.05150837451219559 (0.030905566842643596) | > loss_mel_ce: 1.5594545602798462 (nan) | > loss: 0.006392710842192173 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13727803993767346) | > loader_time: 0.0044 (0.004655080270801822)  --> TIME: 2025-02-26 18:02:51 -- STEP: 16582/36938 -- GLOBAL_STEP: 1346350 | > loss_text_ce: 0.030225979164242744 (0.030908363817667192) | > loss_mel_ce: 3.000925302505493 (nan) | > loss: 0.012028378434479237 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13726289823566837) | > loader_time: 0.0046 (0.004654933727343999)  --> TIME: 2025-02-26 18:03:00 -- STEP: 16632/36938 -- GLOBAL_STEP: 1346400 | > loss_text_ce: 0.03229968249797821 (0.030906238213201424) | > loss_mel_ce: 3.0292141437530518 (nan) | > loss: 0.01214886549860239 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.1372603826536588) | > loader_time: 0.0048 (0.004654979393309041)  --> TIME: 2025-02-26 18:03:09 -- STEP: 16682/36938 -- GLOBAL_STEP: 1346450 | > loss_text_ce: 0.027027461677789688 (0.030907920430339676) | > loss_mel_ce: 2.9979841709136963 (nan) | > loss: 0.012004015035927296 (nan) | > current_lr: 5e-06 | > step_time: 0.176 (0.13725814443743062) | > loader_time: 0.0048 (0.004654896558009952)  --> TIME: 2025-02-26 18:03:19 -- STEP: 16732/36938 -- GLOBAL_STEP: 1346500 | > loss_text_ce: 0.03861841931939125 (0.030907408489692954) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13726966440349903) | > loader_time: 0.0045 (0.004654877470236713)  --> TIME: 2025-02-26 18:03:28 -- STEP: 16782/36938 -- GLOBAL_STEP: 1346550 | > loss_text_ce: 0.032339852303266525 (0.030906586764376333) | > loss_mel_ce: 2.263788938522339 (nan) | > loss: 0.009111622348427773 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13726583252844166) | > loader_time: 0.0045 (0.0046548039846973765)  --> TIME: 2025-02-26 18:03:38 -- STEP: 16832/36938 -- GLOBAL_STEP: 1346600 | > loss_text_ce: 0.011289524845778942 (0.030905431532240138) | > loss_mel_ce: 2.1372342109680176 (nan) | > loss: 0.008525888435542583 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.1372818572892888) | > loader_time: 0.0044 (0.004654769151836291)  --> TIME: 2025-02-26 18:03:47 -- STEP: 16882/36938 -- GLOBAL_STEP: 1346650 | > loss_text_ce: 0.029450923204421997 (0.030903159054005026) | > loss_mel_ce: 2.174102783203125 (nan) | > loss: 0.008744261227548122 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13728949453378497) | > loader_time: 0.0045 (0.004654691394733145)  --> TIME: 2025-02-26 18:03:56 -- STEP: 16932/36938 -- GLOBAL_STEP: 1346700 | > loss_text_ce: 0.029915479943156242 (0.030899199119948564) | > loss_mel_ce: 2.963486433029175 (nan) | > loss: 0.01187857985496521 (nan) | > current_lr: 5e-06 | > step_time: 0.1447 (0.13727617475008122) | > loader_time: 0.0047 (0.004654514037658775)  --> TIME: 2025-02-26 18:04:06 -- STEP: 16982/36938 -- GLOBAL_STEP: 1346750 | > loss_text_ce: 0.03057299740612507 (0.030898204044600235) | > loss_mel_ce: 2.470885992050171 (nan) | > loss: 0.009926424361765385 (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.13728570369511087) | > loader_time: 0.0048 (0.004654419308429804)  --> TIME: 2025-02-26 18:04:15 -- STEP: 17032/36938 -- GLOBAL_STEP: 1346800 | > loss_text_ce: 0.03312577307224274 (0.030900493132007576) | > loss_mel_ce: 2.993499994277954 (nan) | > loss: 0.012010420672595501 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.137286132940836) | > loader_time: 0.0046 (0.0046542560679075995)  --> TIME: 2025-02-26 18:04:25 -- STEP: 17082/36938 -- GLOBAL_STEP: 1346850 | > loss_text_ce: 0.026674332097172737 (0.030899080836339844) | > loss_mel_ce: 2.732388734817505 (nan) | > loss: 0.0109486635774374 (nan) | > current_lr: 5e-06 | > step_time: 0.1493 (0.1372942383933325) | > loader_time: 0.0047 (0.004654256650734636)  --> TIME: 2025-02-26 18:04:35 -- STEP: 17132/36938 -- GLOBAL_STEP: 1346900 | > loss_text_ce: 0.032014161348342896 (0.030900770550953166) | > loss_mel_ce: 2.5409975051879883 (nan) | > loss: 0.010210364125669003 (nan) | > current_lr: 5e-06 | > step_time: 0.1567 (0.13731888808464834) | > loader_time: 0.0046 (0.004654278856500572)  --> TIME: 2025-02-26 18:04:45 -- STEP: 17182/36938 -- GLOBAL_STEP: 1346950 | > loss_text_ce: 0.028747228905558586 (0.030906531337706947) | > loss_mel_ce: 2.7833826541900635 (nan) | > loss: 0.011159246787428856 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13735268725434147) | > loader_time: 0.0046 (0.004654385979441151)  --> TIME: 2025-02-26 18:04:54 -- STEP: 17232/36938 -- GLOBAL_STEP: 1347000 | > loss_text_ce: 0.03220358118414879 (0.030905905748974092) | > loss_mel_ce: 2.2735183238983154 (nan) | > loss: 0.009149691089987755 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.1373479283564378) | > loader_time: 0.0046 (0.004654350940452861)  --> TIME: 2025-02-26 18:05:04 -- STEP: 17282/36938 -- GLOBAL_STEP: 1347050 | > loss_text_ce: 0.030433358624577522 (0.03090846867516344) | > loss_mel_ce: 0.0216827392578125 (nan) | > loss: 0.00020680991292465478 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13736876809835655) | > loader_time: 0.0046 (0.004654492441805791)  --> TIME: 2025-02-26 18:05:13 -- STEP: 17332/36938 -- GLOBAL_STEP: 1347100 | > loss_text_ce: 0.032862041145563126 (0.030908406249005895) | > loss_mel_ce: 3.3680896759033203 (nan) | > loss: 0.01349584013223648 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13735652097840223) | > loader_time: 0.0047 (0.004654418616049453)  --> TIME: 2025-02-26 18:05:23 -- STEP: 17382/36938 -- GLOBAL_STEP: 1347150 | > loss_text_ce: 0.031105918809771538 (0.03090965779849809) | > loss_mel_ce: 3.140775203704834 (nan) | > loss: 0.012586831115186214 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.13736126412183258) | > loader_time: 0.0047 (0.004654461557579016)  --> TIME: 2025-02-26 18:05:32 -- STEP: 17432/36938 -- GLOBAL_STEP: 1347200 | > loss_text_ce: 0.03183465823531151 (0.030907595637405073) | > loss_mel_ce: 2.1658718585968018 (nan) | > loss: 0.008721058256924152 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13734859475688993) | > loader_time: 0.0044 (0.004654291547072374)  --> TIME: 2025-02-26 18:05:41 -- STEP: 17482/36938 -- GLOBAL_STEP: 1347250 | > loss_text_ce: 0.022046618163585663 (0.030909396632337) | > loss_mel_ce: 0.9894973039627075 (nan) | > loss: 0.004014063160866499 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.13734032816996533) | > loader_time: 0.0045 (0.00465417796094471)  --> TIME: 2025-02-26 18:05:50 -- STEP: 17532/36938 -- GLOBAL_STEP: 1347300 | > loss_text_ce: 0.03842495381832123 (0.03090850991168138) | > loss_mel_ce: 3.119035243988037 (nan) | > loss: 0.012529605068266392 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13733895422524017) | > loader_time: 0.0046 (0.004654229788809584)  --> TIME: 2025-02-26 18:06:00 -- STEP: 17582/36938 -- GLOBAL_STEP: 1347350 | > loss_text_ce: 0.037557948380708694 (0.03090456606841519) | > loss_mel_ce: 1.544371247291565 (nan) | > loss: 0.0062774973921477795 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13733497134872472) | > loader_time: 0.0045 (0.0046542418747547035)  --> TIME: 2025-02-26 18:06:09 -- STEP: 17632/36938 -- GLOBAL_STEP: 1347400 | > loss_text_ce: 0.031068410724401474 (0.030904612264380626) | > loss_mel_ce: 2.535048007965088 (nan) | > loss: 0.0101830018684268 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13732565927256676) | > loader_time: 0.0045 (0.004654193084058222)  --> TIME: 2025-02-26 18:06:18 -- STEP: 17682/36938 -- GLOBAL_STEP: 1347450 | > loss_text_ce: 0.026616284623742104 (0.0309080825867086) | > loss_mel_ce: 2.5547308921813965 (nan) | > loss: 0.01024344190955162 (nan) | > current_lr: 5e-06 | > step_time: 0.1811 (0.1373195899529311) | > loader_time: 0.0048 (0.004654097376386442)  --> TIME: 2025-02-26 18:06:28 -- STEP: 17732/36938 -- GLOBAL_STEP: 1347500 | > loss_text_ce: 0.026875615119934082 (0.030908143024530467) | > loss_mel_ce: 3.091447353363037 (nan) | > loss: 0.012374297715723515 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13732657487952282) | > loader_time: 0.0046 (0.00465409293982559)  --> TIME: 2025-02-26 18:06:37 -- STEP: 17782/36938 -- GLOBAL_STEP: 1347550 | > loss_text_ce: 0.022707415744662285 (0.030911096781409474) | > loss_mel_ce: 1.5726063251495361 (nan) | > loss: 0.0063306107185781 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13732283169822856) | > loader_time: 0.0045 (0.004654058293488571)  --> TIME: 2025-02-26 18:06:46 -- STEP: 17832/36938 -- GLOBAL_STEP: 1347600 | > loss_text_ce: 0.03507465496659279 (0.030910215610179696) | > loss_mel_ce: 2.45963191986084 (nan) | > loss: 0.009899630211293697 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.1373171350543087) | > loader_time: 0.0046 (0.004654001847357423)  --> TIME: 2025-02-26 18:06:56 -- STEP: 17882/36938 -- GLOBAL_STEP: 1347650 | > loss_text_ce: 0.02344975434243679 (0.03091265383619615) | > loss_mel_ce: 3.278571128845215 (nan) | > loss: 0.013103257864713669 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13732360862882942) | > loader_time: 0.0046 (0.004653980249047155)  --> TIME: 2025-02-26 18:07:05 -- STEP: 17932/36938 -- GLOBAL_STEP: 1347700 | > loss_text_ce: 0.03884303942322731 (0.030913729422894835) | > loss_mel_ce: 1.9316507577896118 (nan) | > loss: 0.007819420658051968 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.13732544560265414) | > loader_time: 0.0046 (0.0046544304097201296)  --> TIME: 2025-02-26 18:07:14 -- STEP: 17982/36938 -- GLOBAL_STEP: 1347750 | > loss_text_ce: 0.029840081930160522 (0.030918960631919803) | > loss_mel_ce: 3.2515416145324707 (nan) | > loss: 0.013021356426179409 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13731480495986167) | > loader_time: 0.0047 (0.0046543628723175025)  --> TIME: 2025-02-26 18:07:24 -- STEP: 18032/36938 -- GLOBAL_STEP: 1347800 | > loss_text_ce: 0.03291713818907738 (0.03091999752071112) | > loss_mel_ce: 2.8738763332366943 (nan) | > loss: 0.011534894816577435 (nan) | > current_lr: 5e-06 | > step_time: 0.177 (0.137324312524977) | > loader_time: 0.0048 (0.0046543437052111525)  --> TIME: 2025-02-26 18:07:33 -- STEP: 18082/36938 -- GLOBAL_STEP: 1347850 | > loss_text_ce: 0.026362789794802666 (0.03091869119587412) | > loss_mel_ce: 2.975984573364258 (nan) | > loss: 0.011914078146219254 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13730803446858797) | > loader_time: 0.0045 (0.004654585609735605)  --> TIME: 2025-02-26 18:07:42 -- STEP: 18132/36938 -- GLOBAL_STEP: 1347900 | > loss_text_ce: 0.028050782158970833 (0.03091774071965238) | > loss_mel_ce: 3.265648365020752 (nan) | > loss: 0.013070235960185528 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.1373002154133369) | > loader_time: 0.0048 (0.00465470085653174)  --> TIME: 2025-02-26 18:07:51 -- STEP: 18182/36938 -- GLOBAL_STEP: 1347950 | > loss_text_ce: 0.024373382329940796 (0.030915537435321078) | > loss_mel_ce: 2.3253986835479736 (nan) | > loss: 0.009324492886662483 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13728495746172326) | > loader_time: 0.0047 (0.0046548061068728604)  --> TIME: 2025-02-26 18:08:00 -- STEP: 18232/36938 -- GLOBAL_STEP: 1348000 | > loss_text_ce: 0.029653387144207954 (0.030916844571288994) | > loss_mel_ce: 2.9851157665252686 (nan) | > loss: 0.011963370256125927 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13727777214257375) | > loader_time: 0.0046 (0.004654703000597182)  --> TIME: 2025-02-26 18:08:10 -- STEP: 18282/36938 -- GLOBAL_STEP: 1348050 | > loss_text_ce: 0.03845402970910072 (0.030914347398672205) | > loss_mel_ce: 2.168264389038086 (nan) | > loss: 0.008756820112466812 (nan) | > current_lr: 5e-06 | > step_time: 0.1268 (0.13727616140949575) | > loader_time: 0.0046 (0.004654610056595971)  --> TIME: 2025-02-26 18:08:19 -- STEP: 18332/36938 -- GLOBAL_STEP: 1348100 | > loss_text_ce: 0.02854681946337223 (0.030915774706595237) | > loss_mel_ce: 3.0412850379943848 (nan) | > loss: 0.012181873433291912 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.13728149541205137) | > loader_time: 0.0048 (0.0046549404315128625)  --> TIME: 2025-02-26 18:08:29 -- STEP: 18382/36938 -- GLOBAL_STEP: 1348150 | > loss_text_ce: 0.018496254459023476 (0.030914620062058272) | > loss_mel_ce: 2.4270458221435547 (nan) | > loss: 0.009704533033072948 (nan) | > current_lr: 5e-06 | > step_time: 0.1076 (0.13728502342509186) | > loader_time: 0.0045 (0.0046550241054667315)  --> TIME: 2025-02-26 18:08:38 -- STEP: 18432/36938 -- GLOBAL_STEP: 1348200 | > loss_text_ce: 0.0315166600048542 (0.030913957559202263) | > loss_mel_ce: 2.8193717002868652 (nan) | > loss: 0.011313049122691154 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13730059659832808) | > loader_time: 0.0046 (0.004655143530625438)  --> TIME: 2025-02-26 18:08:47 -- STEP: 18482/36938 -- GLOBAL_STEP: 1348250 | > loss_text_ce: 0.03573412448167801 (0.030913746701823553) | > loss_mel_ce: 2.18232798576355 (nan) | > loss: 0.008801834657788277 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13728200403749444) | > loader_time: 0.0047 (0.004655130858184483)  --> TIME: 2025-02-26 18:08:57 -- STEP: 18532/36938 -- GLOBAL_STEP: 1348300 | > loss_text_ce: 0.026183132082223892 (0.030914645489292587) | > loss_mel_ce: 2.588458776473999 (nan) | > loss: 0.010375563986599445 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13728117983978733) | > loader_time: 0.0046 (0.0046552039880668195)  --> TIME: 2025-02-26 18:09:06 -- STEP: 18582/36938 -- GLOBAL_STEP: 1348350 | > loss_text_ce: 0.03787170350551605 (0.03091509409187983) | > loss_mel_ce: 2.446643352508545 (nan) | > loss: 0.009859186597168446 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13727330423738293) | > loader_time: 0.0047 (0.0046554040426382195)  --> TIME: 2025-02-26 18:09:15 -- STEP: 18632/36938 -- GLOBAL_STEP: 1348400 | > loss_text_ce: 0.021057836711406708 (0.03091441718911945) | > loss_mel_ce: 2.3217599391937256 (nan) | > loss: 0.009296896867454052 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.1372672945017866) | > loader_time: 0.0066 (0.0046555835476961745)  --> TIME: 2025-02-26 18:09:24 -- STEP: 18682/36938 -- GLOBAL_STEP: 1348450 | > loss_text_ce: 0.028225857764482498 (0.030913477456274905) | > loss_mel_ce: 2.2377727031707764 (nan) | > loss: 0.008992058224976063 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1372608384258057) | > loader_time: 0.0046 (0.004655785178260165)  --> TIME: 2025-02-26 18:09:34 -- STEP: 18732/36938 -- GLOBAL_STEP: 1348500 | > loss_text_ce: 0.02899562008678913 (0.030913243778058888) | > loss_mel_ce: 2.312391519546509 (nan) | > loss: 0.009291218593716621 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13727277096025928) | > loader_time: 0.0045 (0.004655914481776208)  --> TIME: 2025-02-26 18:09:43 -- STEP: 18782/36938 -- GLOBAL_STEP: 1348550 | > loss_text_ce: 0.02942795678973198 (0.030915492737716144) | > loss_mel_ce: 2.1497654914855957 (nan) | > loss: 0.008647593669593334 (nan) | > current_lr: 5e-06 | > step_time: 0.1516 (0.13728103217777912) | > loader_time: 0.0047 (0.004655843242620486)  --> TIME: 2025-02-26 18:09:53 -- STEP: 18832/36938 -- GLOBAL_STEP: 1348600 | > loss_text_ce: 0.030472204089164734 (0.030918588899729763) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13727485009341592) | > loader_time: 0.0047 (0.004655925710523342)  --> TIME: 2025-02-26 18:10:02 -- STEP: 18882/36938 -- GLOBAL_STEP: 1348650 | > loss_text_ce: 0.03774058446288109 (0.030918712686614527) | > loss_mel_ce: 0.009765625 (nan) | > loss: 0.00018851671484299004 (nan) | > current_lr: 5e-06 | > step_time: 0.108 (0.13727986004478468) | > loader_time: 0.0045 (0.004655996339703222)  --> TIME: 2025-02-26 18:10:11 -- STEP: 18932/36938 -- GLOBAL_STEP: 1348700 | > loss_text_ce: 0.03490062057971954 (0.03092020076477497) | > loss_mel_ce: 0.06298828125 (nan) | > loss: 0.0003884480393026024 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13726953958422164) | > loader_time: 0.0046 (0.00465608263982921)  --> TIME: 2025-02-26 18:10:21 -- STEP: 18982/36938 -- GLOBAL_STEP: 1348750 | > loss_text_ce: 0.04082684963941574 (0.030923694700632443) | > loss_mel_ce: 1.2472398281097412 (nan) | > loss: 0.0051113758236169815 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13726631000664563) | > loader_time: 0.0046 (0.004656245341455198)  --> TIME: 2025-02-26 18:10:30 -- STEP: 19032/36938 -- GLOBAL_STEP: 1348800 | > loss_text_ce: 0.026965688914060593 (0.030924641239916225) | > loss_mel_ce: 2.63222336769104 (nan) | > loss: 0.010552337393164635 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13726577587416183) | > loader_time: 0.0046 (0.004656497923052069)  --> TIME: 2025-02-26 18:10:39 -- STEP: 19082/36938 -- GLOBAL_STEP: 1348850 | > loss_text_ce: 0.027864541858434677 (0.030926810414327783) | > loss_mel_ce: 1.095739722251892 (nan) | > loss: 0.004458747338503599 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1372645309593365) | > loader_time: 0.0046 (0.004656647438898533)  --> TIME: 2025-02-26 18:10:49 -- STEP: 19132/36938 -- GLOBAL_STEP: 1348900 | > loss_text_ce: 0.03460395336151123 (0.030928907137502976) | > loss_mel_ce: 2.8013393878936768 (nan) | > loss: 0.011253743432462215 (nan) | > current_lr: 5e-06 | > step_time: 0.1577 (0.1372683075165701) | > loader_time: 0.0046 (0.004656605595403843)  --> TIME: 2025-02-26 18:10:59 -- STEP: 19182/36938 -- GLOBAL_STEP: 1348950 | > loss_text_ce: 0.028598062694072723 (0.030928102963220565) | > loss_mel_ce: 3.0066721439361572 (nan) | > loss: 0.012044724076986313 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.1372852071881991) | > loader_time: 0.0046 (0.004656567052511654)  --> TIME: 2025-02-26 18:11:08 -- STEP: 19232/36938 -- GLOBAL_STEP: 1349000 | > loss_text_ce: 0.03787644952535629 (0.030929969361241934) | > loss_mel_ce: 2.444373846054077 (nan) | > loss: 0.00985020026564598 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13727426773150916) | > loader_time: 0.0046 (0.004656536631695239)  --> TIME: 2025-02-26 18:11:17 -- STEP: 19282/36938 -- GLOBAL_STEP: 1349050 | > loss_text_ce: 0.027000026777386665 (0.030930307323438426) | > loss_mel_ce: 1.8357950448989868 (nan) | > loss: 0.007392044644802809 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13726314592257471) | > loader_time: 0.0045 (0.004656555222075154)  --> TIME: 2025-02-26 18:11:26 -- STEP: 19332/36938 -- GLOBAL_STEP: 1349100 | > loss_text_ce: 0.02927897870540619 (0.030928587532433053) | > loss_mel_ce: 1.2259434461593628 (nan) | > loss: 0.004981041885912418 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.1372625907829466) | > loader_time: 0.0046 (0.004656641583942906)  --> TIME: 2025-02-26 18:11:35 -- STEP: 19382/36938 -- GLOBAL_STEP: 1349150 | > loss_text_ce: 0.022812852635979652 (0.03092929428068419) | > loss_mel_ce: 2.435532569885254 (nan) | > loss: 0.009755339473485947 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13725591867088774) | > loader_time: 0.0045 (0.004656570317661314)  --> TIME: 2025-02-26 18:11:45 -- STEP: 19432/36938 -- GLOBAL_STEP: 1349200 | > loss_text_ce: 0.03203930705785751 (0.03093139366423611) | > loss_mel_ce: 3.148432493209839 (nan) | > loss: 0.01262092124670744 (nan) | > current_lr: 5e-06 | > step_time: 0.1816 (0.13725816545882888) | > loader_time: 0.0049 (0.004656534250894655)  --> TIME: 2025-02-26 18:11:54 -- STEP: 19482/36938 -- GLOBAL_STEP: 1349250 | > loss_text_ce: 0.033488865941762924 (0.03093031623565959) | > loss_mel_ce: 2.6727852821350098 (nan) | > loss: 0.01073918305337429 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13724160877271144) | > loader_time: 0.0048 (0.0046565248275755216)  --> TIME: 2025-02-26 18:12:03 -- STEP: 19532/36938 -- GLOBAL_STEP: 1349300 | > loss_text_ce: 0.03355792909860611 (0.03092929187416057) | > loss_mel_ce: 2.847208261489868 (nan) | > loss: 0.011431612074375153 (nan) | > current_lr: 5e-06 | > step_time: 0.177 (0.13724570121282623) | > loader_time: 0.0048 (0.004656635308739293)  --> TIME: 2025-02-26 18:12:13 -- STEP: 19582/36938 -- GLOBAL_STEP: 1349350 | > loss_text_ce: 0.019881073385477066 (0.030927594778728265) | > loss_mel_ce: 3.0838027000427246 (nan) | > loss: 0.0123162055388093 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.1372595386136932) | > loader_time: 0.0046 (0.004656626759115048)  --> TIME: 2025-02-26 18:12:23 -- STEP: 19632/36938 -- GLOBAL_STEP: 1349400 | > loss_text_ce: 0.037974417209625244 (0.03092705679954653) | > loss_mel_ce: 2.5879063606262207 (nan) | > loss: 0.010420162230730057 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.13726877837085774) | > loader_time: 0.0047 (0.0046566831283483805)  --> TIME: 2025-02-26 18:12:32 -- STEP: 19682/36938 -- GLOBAL_STEP: 1349450 | > loss_text_ce: 0.03001222386956215 (0.030925578895102456) | > loss_mel_ce: 0.014129638671875 (nan) | > loss: 0.00017516613297630101 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13726322022943713) | > loader_time: 0.0046 (0.004656746782140632)  --> TIME: 2025-02-26 18:12:41 -- STEP: 19732/36938 -- GLOBAL_STEP: 1349500 | > loss_text_ce: 0.033846914768218994 (0.03092700538447588) | > loss_mel_ce: 2.584864854812622 (nan) | > loss: 0.010391714051365852 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.1372684705288821) | > loader_time: 0.0046 (0.0046566828689637045)  --> TIME: 2025-02-26 18:12:50 -- STEP: 19782/36938 -- GLOBAL_STEP: 1349550 | > loss_text_ce: 0.023724013939499855 (0.03092628482268478) | > loss_mel_ce: 2.7907087802886963 (nan) | > loss: 0.011168384924530983 (nan) | > current_lr: 5e-06 | > step_time: 0.1442 (0.13726197458376443) | > loader_time: 0.0046 (0.004656554196459638)  --> TIME: 2025-02-26 18:13:00 -- STEP: 19832/36938 -- GLOBAL_STEP: 1349600 | > loss_text_ce: 0.015856008976697922 (0.030924378929936033) | > loss_mel_ce: 0.0656280517578125 (nan) | > loss: 0.0003233494935557246 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13725599113997433) | > loader_time: 0.0045 (0.004656519078111969)  --> TIME: 2025-02-26 18:13:09 -- STEP: 19882/36938 -- GLOBAL_STEP: 1349650 | > loss_text_ce: 0.036392416805028915 (0.03092346591800477) | > loss_mel_ce: 3.2231364250183105 (nan) | > loss: 0.012934639118611813 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13724448533140074) | > loader_time: 0.0046 (0.0046565116573034715)  --> TIME: 2025-02-26 18:13:18 -- STEP: 19932/36938 -- GLOBAL_STEP: 1349700 | > loss_text_ce: 0.030131760984659195 (0.030922790872350444) | > loss_mel_ce: 2.957979917526245 (nan) | > loss: 0.011857586912810802 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.1372317274531124) | > loader_time: 0.0046 (0.004656374657845455)  --> TIME: 2025-02-26 18:13:27 -- STEP: 19982/36938 -- GLOBAL_STEP: 1349750 | > loss_text_ce: 0.037392012774944305 (0.03092389760014408) | > loss_mel_ce: 3.222972869873047 (nan) | > loss: 0.012937956489622593 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13723241446457426) | > loader_time: 0.0047 (0.004656448949864042)  --> TIME: 2025-02-26 18:13:37 -- STEP: 20032/36938 -- GLOBAL_STEP: 1349800 | > loss_text_ce: 0.03266730159521103 (0.030923466383997077) | > loss_mel_ce: 3.114222288131714 (nan) | > loss: 0.01248765829950571 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13724511934402533) | > loader_time: 0.0046 (0.004656498353130893)  --> TIME: 2025-02-26 18:13:46 -- STEP: 20082/36938 -- GLOBAL_STEP: 1349850 | > loss_text_ce: 0.03393036127090454 (0.030924300091612338) | > loss_mel_ce: 1.0956438779830933 (nan) | > loss: 0.00448243785649538 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13724383012835886) | > loader_time: 0.0046 (0.004656507085369533)  --> TIME: 2025-02-26 18:13:56 -- STEP: 20132/36938 -- GLOBAL_STEP: 1349900 | > loss_text_ce: 0.029676396399736404 (0.03092709353643573) | > loss_mel_ce: 0.8738644123077393 (nan) | > loss: 0.0035854794550687075 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13724029965412904) | > loader_time: 0.0047 (0.0046564908333665955)  --> TIME: 2025-02-26 18:14:05 -- STEP: 20182/36938 -- GLOBAL_STEP: 1349950 | > loss_text_ce: 0.025813676416873932 (0.030924347083318426) | > loss_mel_ce: 2.9971699714660645 (nan) | > loss: 0.011995967477560043 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13723580022831997) | > loader_time: 0.0046 (0.004656457485168352)  --> TIME: 2025-02-26 18:14:14 -- STEP: 20232/36938 -- GLOBAL_STEP: 1350000 | > loss_text_ce: 0.026778200641274452 (0.030922660106074387) | > loss_mel_ce: 2.8516578674316406 (nan) | > loss: 0.01142236590385437 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13722923476972118) | > loader_time: 0.0046 (0.004656471980801202) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1350000.pth  --> TIME: 2025-02-26 18:14:30 -- STEP: 20282/36938 -- GLOBAL_STEP: 1350050 | > loss_text_ce: 0.03935568779706955 (0.030921169955320318) | > loss_mel_ce: 2.9685590267181396 (nan) | > loss: 0.011936170049011707 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.13721926063490078) | > loader_time: 0.0047 (0.004656473627081984)  --> TIME: 2025-02-26 18:14:39 -- STEP: 20332/36938 -- GLOBAL_STEP: 1350100 | > loss_text_ce: 0.03551711142063141 (0.030922279634715232) | > loss_mel_ce: 1.348486304283142 (nan) | > loss: 0.005492077209055424 (nan) | > current_lr: 5e-06 | > step_time: 0.1491 (0.1372163397060834) | > loader_time: 0.0046 (0.0046564977914358725)  --> TIME: 2025-02-26 18:14:48 -- STEP: 20382/36938 -- GLOBAL_STEP: 1350150 | > loss_text_ce: 0.03464863449335098 (0.030924764020028857) | > loss_mel_ce: 2.123831033706665 (nan) | > loss: 0.008565396070480347 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13720881730718593) | > loader_time: 0.0046 (0.004656357101247275)  --> TIME: 2025-02-26 18:14:57 -- STEP: 20432/36938 -- GLOBAL_STEP: 1350200 | > loss_text_ce: 0.028605103492736816 (0.03092257390567721) | > loss_mel_ce: 2.7907042503356934 (nan) | > loss: 0.011187735944986343 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.1372046770765701) | > loader_time: 0.0048 (0.00465635286706821)  --> TIME: 2025-02-26 18:15:06 -- STEP: 20482/36938 -- GLOBAL_STEP: 1350250 | > loss_text_ce: 0.020561736077070236 (0.030920441027274356) | > loss_mel_ce: 2.738065242767334 (nan) | > loss: 0.010946933180093765 (nan) | > current_lr: 5e-06 | > step_time: 0.1472 (0.13719765524905533) | > loader_time: 0.0046 (0.004656441055020714)  --> TIME: 2025-02-26 18:15:16 -- STEP: 20532/36938 -- GLOBAL_STEP: 1350300 | > loss_text_ce: 0.044077225029468536 (0.030921595148981147) | > loss_mel_ce: 2.434417963027954 (nan) | > loss: 0.009835299104452133 (nan) | > current_lr: 5e-06 | > step_time: 0.1265 (0.1372147790409764) | > loader_time: 0.0046 (0.004656515041569066)  --> TIME: 2025-02-26 18:15:25 -- STEP: 20582/36938 -- GLOBAL_STEP: 1350350 | > loss_text_ce: 0.03983389213681221 (0.030918505070823346) | > loss_mel_ce: 1.9647470712661743 (nan) | > loss: 0.007954686880111694 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1372105067724518) | > loader_time: 0.0045 (0.004656626895314783)  --> TIME: 2025-02-26 18:15:34 -- STEP: 20632/36938 -- GLOBAL_STEP: 1350400 | > loss_text_ce: 0.024355722591280937 (0.030917845376724422) | > loss_mel_ce: 2.4558892250061035 (nan) | > loss: 0.009842242114245892 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13720467434737305) | > loader_time: 0.0046 (0.004656553372434515)  --> TIME: 2025-02-26 18:15:44 -- STEP: 20682/36938 -- GLOBAL_STEP: 1350450 | > loss_text_ce: 0.036124397069215775 (0.030917184156859314) | > loss_mel_ce: 3.1756532192230225 (nan) | > loss: 0.012745150364935398 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13720192274979398) | > loader_time: 0.0046 (0.004656561879721592)  --> TIME: 2025-02-26 18:15:53 -- STEP: 20732/36938 -- GLOBAL_STEP: 1350500 | > loss_text_ce: 0.028984444215893745 (0.030914424051170918) | > loss_mel_ce: 0.010166985914111137 (nan) | > loss: 0.00015536282444372773 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.1371976802322857) | > loader_time: 0.0045 (0.004656508211323179)  --> TIME: 2025-02-26 18:16:02 -- STEP: 20782/36938 -- GLOBAL_STEP: 1350550 | > loss_text_ce: 0.0349007211625576 (0.03091440024371975) | > loss_mel_ce: 2.837658166885376 (nan) | > loss: 0.01139904372394085 (nan) | > current_lr: 5e-06 | > step_time: 0.1466 (0.13719640160396085) | > loader_time: 0.0046 (0.004656454066938369)  --> TIME: 2025-02-26 18:16:11 -- STEP: 20832/36938 -- GLOBAL_STEP: 1350600 | > loss_text_ce: 0.03084949217736721 (0.03091394805713587) | > loss_mel_ce: 3.0351436138153076 (nan) | > loss: 0.012166639789938927 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.1371811796809474) | > loader_time: 0.0045 (0.004656369853678937)  --> TIME: 2025-02-26 18:16:21 -- STEP: 20882/36938 -- GLOBAL_STEP: 1350650 | > loss_text_ce: 0.02644171379506588 (0.030912124136278696) | > loss_mel_ce: 0.04859164357185364 (nan) | > loss: 0.00029775145230814815 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13717291890838296) | > loader_time: 0.0046 (0.004656349889917653)  --> TIME: 2025-02-26 18:16:30 -- STEP: 20932/36938 -- GLOBAL_STEP: 1350700 | > loss_text_ce: 0.02678179182112217 (0.03090706569585876) | > loss_mel_ce: 2.348930597305298 (nan) | > loss: 0.009427431039512157 (nan) | > current_lr: 5e-06 | > step_time: 0.1599 (0.13716644785633242) | > loader_time: 0.0048 (0.004656239629315757)  --> TIME: 2025-02-26 18:16:39 -- STEP: 20982/36938 -- GLOBAL_STEP: 1350750 | > loss_text_ce: 0.026776868849992752 (0.030907793080728215) | > loss_mel_ce: 2.9487507343292236 (nan) | > loss: 0.01180764939635992 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13717743221177733) | > loader_time: 0.0045 (0.004656114270082364)  --> TIME: 2025-02-26 18:16:48 -- STEP: 21032/36938 -- GLOBAL_STEP: 1350800 | > loss_text_ce: 0.02874324657022953 (0.030908797588735173) | > loss_mel_ce: 2.1597323417663574 (nan) | > loss: 0.008684427477419376 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13716508482698572) | > loader_time: 0.0045 (0.00465600994568263)  --> TIME: 2025-02-26 18:16:58 -- STEP: 21082/36938 -- GLOBAL_STEP: 1350850 | > loss_text_ce: 0.02504057064652443 (0.03090680359613414) | > loss_mel_ce: 2.97115421295166 (nan) | > loss: 0.011889662593603134 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13716563561215647) | > loader_time: 0.0045 (0.004655989023191791)  --> TIME: 2025-02-26 18:17:07 -- STEP: 21132/36938 -- GLOBAL_STEP: 1350900 | > loss_text_ce: 0.03251441940665245 (0.03090469682479963) | > loss_mel_ce: 2.8114213943481445 (nan) | > loss: 0.011285459622740746 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.1371569425663024) | > loader_time: 0.0046 (0.004655993573710294)  --> TIME: 2025-02-26 18:17:16 -- STEP: 21182/36938 -- GLOBAL_STEP: 1350950 | > loss_text_ce: 0.03853951022028923 (0.03090394006321596) | > loss_mel_ce: 2.959324836730957 (nan) | > loss: 0.011896287091076374 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.13715228164581977) | > loader_time: 0.0048 (0.004655819125603639)  --> TIME: 2025-02-26 18:17:25 -- STEP: 21232/36938 -- GLOBAL_STEP: 1351000 | > loss_text_ce: 0.04033539816737175 (0.030905265054833413) | > loss_mel_ce: 2.5784740447998047 (nan) | > loss: 0.010392101481556892 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13714292682510687) | > loader_time: 0.0045 (0.0046556359430678165)  --> TIME: 2025-02-26 18:17:35 -- STEP: 21282/36938 -- GLOBAL_STEP: 1351050 | > loss_text_ce: 0.031776025891304016 (0.030903152408582085) | > loss_mel_ce: 3.091594696044922 (nan) | > loss: 0.012394328601658344 (nan) | > current_lr: 5e-06 | > step_time: 0.1531 (0.13715564346483644) | > loader_time: 0.0047 (0.004655542784576942)  --> TIME: 2025-02-26 18:17:44 -- STEP: 21332/36938 -- GLOBAL_STEP: 1351100 | > loss_text_ce: 0.02246507629752159 (0.030901936288955677) | > loss_mel_ce: 2.808696985244751 (nan) | > loss: 0.011234770528972149 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.1371600012964309) | > loader_time: 0.0046 (0.0046556142465987325)  --> TIME: 2025-02-26 18:17:54 -- STEP: 21382/36938 -- GLOBAL_STEP: 1351150 | > loss_text_ce: 0.032255448400974274 (0.030900287228461255) | > loss_mel_ce: 2.6866698265075684 (nan) | > loss: 0.010789386928081512 (nan) | > current_lr: 5e-06 | > step_time: 0.1809 (0.13716840808690964) | > loader_time: 0.0048 (0.004655700014924522)  --> TIME: 2025-02-26 18:18:03 -- STEP: 21432/36938 -- GLOBAL_STEP: 1351200 | > loss_text_ce: 0.029334623366594315 (0.030900791379872193) | > loss_mel_ce: 2.3197877407073975 (nan) | > loss: 0.009321914054453373 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.13716491668604225) | > loader_time: 0.0048 (0.004655739183344025)  --> TIME: 2025-02-26 18:18:12 -- STEP: 21482/36938 -- GLOBAL_STEP: 1351250 | > loss_text_ce: 0.027547525241971016 (0.03090077495955821) | > loss_mel_ce: 0.9200955033302307 (nan) | > loss: 0.003760488471016288 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13715302653401787) | > loader_time: 0.0046 (0.004655628860934617)  --> TIME: 2025-02-26 18:18:22 -- STEP: 21532/36938 -- GLOBAL_STEP: 1351300 | > loss_text_ce: 0.035591062158346176 (0.030900897668988944) | > loss_mel_ce: 2.3950750827789307 (nan) | > loss: 0.009645501151680946 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.13716124285779832) | > loader_time: 0.0047 (0.004655639865729257)  --> TIME: 2025-02-26 18:18:31 -- STEP: 21582/36938 -- GLOBAL_STEP: 1351350 | > loss_text_ce: 0.03550782799720764 (0.03090319573737614) | > loss_mel_ce: 1.4391566514968872 (nan) | > loss: 0.005851843394339085 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13715763288016508) | > loader_time: 0.0046 (0.0046556045653126064)  --> TIME: 2025-02-26 18:18:41 -- STEP: 21632/36938 -- GLOBAL_STEP: 1351400 | > loss_text_ce: 0.02817903272807598 (0.030902284371852422) | > loss_mel_ce: 0.3067372739315033 (nan) | > loss: 0.0013290329370647669 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13716147530316836) | > loader_time: 0.0046 (0.004655640032252615)  --> TIME: 2025-02-26 18:18:50 -- STEP: 21682/36938 -- GLOBAL_STEP: 1351450 | > loss_text_ce: 0.03184834495186806 (0.03090578640995674) | > loss_mel_ce: 2.639836072921753 (nan) | > loss: 0.010601922869682312 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13716165176694456) | > loader_time: 0.0071 (0.004655892036792681)  --> TIME: 2025-02-26 18:18:59 -- STEP: 21732/36938 -- GLOBAL_STEP: 1351500 | > loss_text_ce: 0.03783714771270752 (0.03090502516133496) | > loss_mel_ce: 2.544686794281006 (nan) | > loss: 0.010248110629618168 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13715738059187177) | > loader_time: 0.0046 (0.0046559271618474285)  --> TIME: 2025-02-26 18:19:09 -- STEP: 21782/36938 -- GLOBAL_STEP: 1351550 | > loss_text_ce: 0.020397312939167023 (0.030904303194603355) | > loss_mel_ce: 2.3420515060424805 (nan) | > loss: 0.009374798275530338 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13716329240829478) | > loader_time: 0.0045 (0.00465590578828258)  --> TIME: 2025-02-26 18:19:18 -- STEP: 21832/36938 -- GLOBAL_STEP: 1351600 | > loss_text_ce: 0.02391829527914524 (0.030905174070790225) | > loss_mel_ce: 1.580296516418457 (nan) | > loss: 0.006365932058542967 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13715718252486614) | > loader_time: 0.0045 (0.004655861830527899)  --> TIME: 2025-02-26 18:19:27 -- STEP: 21882/36938 -- GLOBAL_STEP: 1351650 | > loss_text_ce: 0.027200141921639442 (0.030907268054479794) | > loss_mel_ce: 2.8568525314331055 (nan) | > loss: 0.011444654315710068 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13715245697356787) | > loader_time: 0.0048 (0.004656027880264977)  --> TIME: 2025-02-26 18:19:36 -- STEP: 21932/36938 -- GLOBAL_STEP: 1351700 | > loss_text_ce: 0.025651492178440094 (0.030906078007641054) | > loss_mel_ce: 2.347435712814331 (nan) | > loss: 0.009417013265192509 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13714370103771942) | > loader_time: 0.0047 (0.004656013826286403)  --> TIME: 2025-02-26 18:19:46 -- STEP: 21982/36938 -- GLOBAL_STEP: 1351750 | > loss_text_ce: 0.02497824653983116 (0.030907500445875445) | > loss_mel_ce: 2.111191749572754 (nan) | > loss: 0.008476865477859974 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13714652153220144) | > loader_time: 0.0045 (0.004656028578359709)  --> TIME: 2025-02-26 18:19:55 -- STEP: 22032/36938 -- GLOBAL_STEP: 1351800 | > loss_text_ce: 0.024611562490463257 (0.030907502350149354) | > loss_mel_ce: 2.257215738296509 (nan) | > loss: 0.009054870344698429 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13714628567622794) | > loader_time: 0.0046 (0.004656122400869728)  --> TIME: 2025-02-26 18:20:05 -- STEP: 22082/36938 -- GLOBAL_STEP: 1351850 | > loss_text_ce: 0.028038624674081802 (0.03090993096437816) | > loss_mel_ce: 3.542651891708374 (nan) | > loss: 0.01416940800845623 (nan) | > current_lr: 5e-06 | > step_time: 0.1451 (0.13715856141796073) | > loader_time: 0.0047 (0.0046561224047509875)  --> TIME: 2025-02-26 18:20:14 -- STEP: 22132/36938 -- GLOBAL_STEP: 1351900 | > loss_text_ce: 0.026752294972538948 (0.030910537388319972) | > loss_mel_ce: 2.34196400642395 (nan) | > loss: 0.00939966831356287 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.1371546145504982) | > loader_time: 0.0047 (0.004656259931277868)  --> TIME: 2025-02-26 18:20:23 -- STEP: 22182/36938 -- GLOBAL_STEP: 1351950 | > loss_text_ce: 0.03629221022129059 (0.030911169752997368) | > loss_mel_ce: 2.6298840045928955 (nan) | > loss: 0.010580065660178661 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.1371482184425306) | > loader_time: 0.0045 (0.004656198897295971)  --> TIME: 2025-02-26 18:20:33 -- STEP: 22232/36938 -- GLOBAL_STEP: 1352000 | > loss_text_ce: 0.036181166768074036 (0.03091214819970554) | > loss_mel_ce: 2.961575984954834 (nan) | > loss: 0.01189586240798235 (nan) | > current_lr: 5e-06 | > step_time: 0.1821 (0.1371458572164283) | > loader_time: 0.0048 (0.004656202428930304)  --> TIME: 2025-02-26 18:20:42 -- STEP: 22282/36938 -- GLOBAL_STEP: 1352050 | > loss_text_ce: 0.026491304859519005 (0.030910495703821914) | > loss_mel_ce: 1.710387110710144 (nan) | > loss: 0.006892375182360411 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13714395104613555) | > loader_time: 0.0046 (0.004656116652773324)  --> TIME: 2025-02-26 18:20:51 -- STEP: 22332/36938 -- GLOBAL_STEP: 1352100 | > loss_text_ce: 0.025298725813627243 (0.03091029181983288) | > loss_mel_ce: 1.3030853271484375 (nan) | > loss: 0.005271365400403738 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.1371381609494932) | > loader_time: 0.0046 (0.004656141939800359)  --> TIME: 2025-02-26 18:21:01 -- STEP: 22382/36938 -- GLOBAL_STEP: 1352150 | > loss_text_ce: 0.02730977162718773 (0.03091005361946047) | > loss_mel_ce: 2.29607892036438 (nan) | > loss: 0.009219796396791935 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13714243436603155) | > loader_time: 0.0046 (0.004656193691202875)  --> TIME: 2025-02-26 18:21:10 -- STEP: 22432/36938 -- GLOBAL_STEP: 1352200 | > loss_text_ce: 0.022402696311473846 (0.030912346056623137) | > loss_mel_ce: 2.0699310302734375 (nan) | > loss: 0.00830291211605072 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1371405601990371) | > loader_time: 0.0045 (0.004656151489762543)  --> TIME: 2025-02-26 18:21:19 -- STEP: 22482/36938 -- GLOBAL_STEP: 1352250 | > loss_text_ce: 0.024345839396119118 (0.03091422306730718) | > loss_mel_ce: 1.9250123500823975 (nan) | > loss: 0.007735549006611109 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13713780782282287) | > loader_time: 0.0045 (0.004656045051478332)  --> TIME: 2025-02-26 18:21:29 -- STEP: 22532/36938 -- GLOBAL_STEP: 1352300 | > loss_text_ce: 0.034514009952545166 (0.030913119485801666) | > loss_mel_ce: 2.987424373626709 (nan) | > loss: 0.011991819366812706 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.1371389665126202) | > loader_time: 0.0049 (0.0046559714327141285)  --> TIME: 2025-02-26 18:21:38 -- STEP: 22582/36938 -- GLOBAL_STEP: 1352350 | > loss_text_ce: 0.029399169608950615 (0.030913287337204418) | > loss_mel_ce: 2.0179247856140137 (nan) | > loss: 0.008124302141368389 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13714408721422047) | > loader_time: 0.0045 (0.0046559873964733545)  --> TIME: 2025-02-26 18:21:48 -- STEP: 22632/36938 -- GLOBAL_STEP: 1352400 | > loss_text_ce: 0.03657758980989456 (0.03091381683570937) | > loss_mel_ce: 1.9352355003356934 (nan) | > loss: 0.007824655622243881 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13714387055292798) | > loader_time: 0.0047 (0.004655984927926142)  --> TIME: 2025-02-26 18:21:57 -- STEP: 22682/36938 -- GLOBAL_STEP: 1352450 | > loss_text_ce: 0.018440769985318184 (0.030913218722885084) | > loss_mel_ce: 1.4531570672988892 (nan) | > loss: 0.0058396742679178715 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13714153220461936) | > loader_time: 0.0045 (0.004656004018542226)  --> TIME: 2025-02-26 18:22:06 -- STEP: 22732/36938 -- GLOBAL_STEP: 1352500 | > loss_text_ce: 0.03171989694237709 (0.030914337088240196) | > loss_mel_ce: 1.1236509084701538 (nan) | > loss: 0.004584805108606815 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.1371418545725638) | > loader_time: 0.0045 (0.004655939507354378)  --> TIME: 2025-02-26 18:22:16 -- STEP: 22782/36938 -- GLOBAL_STEP: 1352550 | > loss_text_ce: 0.03953178972005844 (0.03091411397885768) | > loss_mel_ce: 2.4187979698181152 (nan) | > loss: 0.009755277074873447 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13713948176811447) | > loader_time: 0.0045 (0.004655974311684607)  --> TIME: 2025-02-26 18:22:25 -- STEP: 22832/36938 -- GLOBAL_STEP: 1352600 | > loss_text_ce: 0.035410936921834946 (0.030910069058645483) | > loss_mel_ce: 2.5570273399353027 (nan) | > loss: 0.010287453420460224 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13713352449013708) | > loader_time: 0.0046 (0.004655868023863683)  --> TIME: 2025-02-26 18:22:35 -- STEP: 22882/36938 -- GLOBAL_STEP: 1352650 | > loss_text_ce: 0.03987186402082443 (0.030911345977414146) | > loss_mel_ce: 0.0084228515625 (nan) | > loss: 0.00019164571131113917 (nan) | > current_lr: 5e-06 | > step_time: 0.1118 (0.1371464710137227) | > loader_time: 0.0045 (0.004655873782775808)  --> TIME: 2025-02-26 18:22:44 -- STEP: 22932/36938 -- GLOBAL_STEP: 1352700 | > loss_text_ce: 0.03490229323506355 (0.030910529744387034) | > loss_mel_ce: 2.1546719074249268 (nan) | > loss: 0.008688786998391151 (nan) | > current_lr: 5e-06 | > step_time: 0.1082 (0.13713837527654776) | > loader_time: 0.0045 (0.00465579306748697)  --> TIME: 2025-02-26 18:22:53 -- STEP: 22982/36938 -- GLOBAL_STEP: 1352750 | > loss_text_ce: 0.03132887929677963 (0.030911083940264036) | > loss_mel_ce: 3.704627513885498 (nan) | > loss: 0.014825224876403809 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13714405329666607) | > loader_time: 0.0045 (0.00465568582400174)  --> TIME: 2025-02-26 18:23:02 -- STEP: 23032/36938 -- GLOBAL_STEP: 1352800 | > loss_text_ce: 0.0289822556078434 (0.030912369670115566) | > loss_mel_ce: 2.9393222332000732 (nan) | > loss: 0.01177898608148098 (nan) | > current_lr: 5e-06 | > step_time: 0.184 (0.13714147459802206) | > loader_time: 0.0049 (0.004655586602829434)  --> TIME: 2025-02-26 18:23:12 -- STEP: 23082/36938 -- GLOBAL_STEP: 1352850 | > loss_text_ce: 0.026808474212884903 (0.030913485198721256) | > loss_mel_ce: 2.7786881923675537 (nan) | > loss: 0.011132923886179924 (nan) | > current_lr: 5e-06 | > step_time: 0.1492 (0.13714430415023915) | > loader_time: 0.0047 (0.0046555425356190625)  --> TIME: 2025-02-26 18:23:21 -- STEP: 23132/36938 -- GLOBAL_STEP: 1352900 | > loss_text_ce: 0.04259994253516197 (0.030916533375750097) | > loss_mel_ce: 0.1809539794921875 (nan) | > loss: 0.0008871188038028777 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1371524888978942) | > loader_time: 0.0045 (0.00465561560068949)  --> TIME: 2025-02-26 18:23:31 -- STEP: 23182/36938 -- GLOBAL_STEP: 1352950 | > loss_text_ce: 0.02671239525079727 (0.030915812298305347) | > loss_mel_ce: 2.6112606525421143 (nan) | > loss: 0.010468147695064545 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13715475107039374) | > loader_time: 0.0045 (0.004655602401823563)  --> TIME: 2025-02-26 18:23:40 -- STEP: 23232/36938 -- GLOBAL_STEP: 1353000 | > loss_text_ce: 0.02439790405333042 (0.03091857776799211) | > loss_mel_ce: 2.776811361312866 (nan) | > loss: 0.011115910485386848 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13716177412465166) | > loader_time: 0.0047 (0.004655553279418905)  --> TIME: 2025-02-26 18:23:50 -- STEP: 23282/36938 -- GLOBAL_STEP: 1353050 | > loss_text_ce: 0.02867898903787136 (0.030917585925354693) | > loss_mel_ce: 2.792292594909668 (nan) | > loss: 0.01119433157145977 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.13717003215189694) | > loader_time: 0.0047 (0.0046557527710385516)  --> TIME: 2025-02-26 18:23:59 -- STEP: 23332/36938 -- GLOBAL_STEP: 1353100 | > loss_text_ce: 0.035141900181770325 (0.030918684092889327) | > loss_mel_ce: 1.6401759386062622 (nan) | > loss: 0.0066480874083936214 (nan) | > current_lr: 5e-06 | > step_time: 0.1454 (0.1371691774263312) | > loader_time: 0.0046 (0.004655719109961444)  --> TIME: 2025-02-26 18:24:09 -- STEP: 23382/36938 -- GLOBAL_STEP: 1353150 | > loss_text_ce: 0.03670234978199005 (0.030919420443601586) | > loss_mel_ce: 2.149190664291382 (nan) | > loss: 0.008674179203808308 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1371650735465825) | > loader_time: 0.0046 (0.00465576963381448)  --> TIME: 2025-02-26 18:24:18 -- STEP: 23432/36938 -- GLOBAL_STEP: 1353200 | > loss_text_ce: 0.022401051595807076 (0.030920201780188814) | > loss_mel_ce: 0.005248038563877344 (nan) | > loss: 0.00010971861775033176 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13716388467067084) | > loader_time: 0.0046 (0.004655792378207116)  --> TIME: 2025-02-26 18:24:28 -- STEP: 23482/36938 -- GLOBAL_STEP: 1353250 | > loss_text_ce: 0.02664540708065033 (0.030921801710237957) | > loss_mel_ce: 2.4846067428588867 (nan) | > loss: 0.00996528659015894 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13717361300048977) | > loader_time: 0.0045 (0.004655788982658746)  --> TIME: 2025-02-26 18:24:37 -- STEP: 23532/36938 -- GLOBAL_STEP: 1353300 | > loss_text_ce: 0.02704036422073841 (0.030922556514554125) | > loss_mel_ce: 2.5344321727752686 (nan) | > loss: 0.010164574719965458 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13717959762207096) | > loader_time: 0.0047 (0.004655950454025872)  --> TIME: 2025-02-26 18:24:46 -- STEP: 23582/36938 -- GLOBAL_STEP: 1353350 | > loss_text_ce: 0.03524848446249962 (0.030922011127926075) | > loss_mel_ce: 2.9272212982177734 (nan) | > loss: 0.011755833402276039 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.1371760827552137) | > loader_time: 0.0047 (0.004656108480587262)  --> TIME: 2025-02-26 18:24:56 -- STEP: 23632/36938 -- GLOBAL_STEP: 1353400 | > loss_text_ce: 0.034687865525484085 (0.03092328288793622) | > loss_mel_ce: 3.0124900341033936 (nan) | > loss: 0.012091975659132004 (nan) | > current_lr: 5e-06 | > step_time: 0.1913 (0.13718351922530642) | > loader_time: 0.0049 (0.0046562158585240375)  --> TIME: 2025-02-26 18:25:05 -- STEP: 23682/36938 -- GLOBAL_STEP: 1353450 | > loss_text_ce: 0.025088073685765266 (0.030922377336648064) | > loss_mel_ce: 2.045516014099121 (nan) | > loss: 0.008216683752834797 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.1371843359597406) | > loader_time: 0.0046 (0.004656253417957153)  --> TIME: 2025-02-26 18:25:15 -- STEP: 23732/36938 -- GLOBAL_STEP: 1353500 | > loss_text_ce: 0.021371979266405106 (0.030921628256767926) | > loss_mel_ce: 1.113911747932434 (nan) | > loss: 0.004505094140768051 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.1371783035994568) | > loader_time: 0.0045 (0.00465636482010518)  --> TIME: 2025-02-26 18:25:24 -- STEP: 23782/36938 -- GLOBAL_STEP: 1353550 | > loss_text_ce: 0.03674011304974556 (0.03092165766895953) | > loss_mel_ce: 1.169921875 (nan) | > loss: 0.004788341466337442 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.1371790257788265) | > loader_time: 0.0045 (0.004656441838674537)  --> TIME: 2025-02-26 18:25:34 -- STEP: 23832/36938 -- GLOBAL_STEP: 1353600 | > loss_text_ce: 0.024572791531682014 (0.03092064165834824) | > loss_mel_ce: 2.327411413192749 (nan) | > loss: 0.009333271533250809 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13718506537775377) | > loader_time: 0.0046 (0.004656581930283453)  --> TIME: 2025-02-26 18:25:43 -- STEP: 23882/36938 -- GLOBAL_STEP: 1353650 | > loss_text_ce: 0.03730405122041702 (0.030920168356539424) | > loss_mel_ce: 2.836702823638916 (nan) | > loss: 0.011404789984226227 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13718293419176458) | > loader_time: 0.0046 (0.004656649606230468)  --> TIME: 2025-02-26 18:25:52 -- STEP: 23932/36938 -- GLOBAL_STEP: 1353700 | > loss_text_ce: 0.03293401747941971 (0.030918935763234956) | > loss_mel_ce: 2.170177698135376 (nan) | > loss: 0.008742506615817547 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13717666210068918) | > loader_time: 0.0046 (0.004656632728009202)  --> TIME: 2025-02-26 18:26:01 -- STEP: 23982/36938 -- GLOBAL_STEP: 1353750 | > loss_text_ce: 0.034541066735982895 (0.030919503369737922) | > loss_mel_ce: 2.193737506866455 (nan) | > loss: 0.008842376060783863 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13716457913291813) | > loader_time: 0.0047 (0.004656623883359115)  --> TIME: 2025-02-26 18:26:11 -- STEP: 24032/36938 -- GLOBAL_STEP: 1353800 | > loss_text_ce: 0.03691425174474716 (0.030919267791581457) | > loss_mel_ce: 2.378896474838257 (nan) | > loss: 0.00958655122667551 (nan) | > current_lr: 5e-06 | > step_time: 0.1732 (0.13718446678470728) | > loader_time: 0.0049 (0.00465666171356777)  --> TIME: 2025-02-26 18:26:20 -- STEP: 24082/36938 -- GLOBAL_STEP: 1353850 | > loss_text_ce: 0.038978755474090576 (0.03092077133821529) | > loss_mel_ce: 2.150557279586792 (nan) | > loss: 0.008688636124134064 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13718812784060663) | > loader_time: 0.0045 (0.00465659989875848)  --> TIME: 2025-02-26 18:26:30 -- STEP: 24132/36938 -- GLOBAL_STEP: 1353900 | > loss_text_ce: 0.03369772806763649 (0.030921684473725338) | > loss_mel_ce: 2.836399793624878 (nan) | > loss: 0.011389276944100857 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.1371913753930309) | > loader_time: 0.0048 (0.004656551984062674)  --> TIME: 2025-02-26 18:26:39 -- STEP: 24182/36938 -- GLOBAL_STEP: 1353950 | > loss_text_ce: 0.028047893196344376 (0.030924975963989697) | > loss_mel_ce: 2.2439968585968018 (nan) | > loss: 0.009016050957143307 (nan) | > current_lr: 5e-06 | > step_time: 0.1078 (0.13719074125276404) | > loader_time: 0.0046 (0.004656598315760702)  --> TIME: 2025-02-26 18:26:48 -- STEP: 24232/36938 -- GLOBAL_STEP: 1354000 | > loss_text_ce: 0.03547308221459389 (0.030922481723906256) | > loss_mel_ce: 2.4753310680389404 (nan) | > loss: 0.009963509626686573 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1371900706686322) | > loader_time: 0.0046 (0.004656602394545416)  --> TIME: 2025-02-26 18:26:58 -- STEP: 24282/36938 -- GLOBAL_STEP: 1354050 | > loss_text_ce: 0.02891073003411293 (0.03091997964712765) | > loss_mel_ce: 2.8817310333251953 (nan) | > loss: 0.011550165712833405 (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.13719494728091094) | > loader_time: 0.0048 (0.00465651749877263)  --> TIME: 2025-02-26 18:27:07 -- STEP: 24332/36938 -- GLOBAL_STEP: 1354100 | > loss_text_ce: 0.038898125290870667 (0.03091895473416626) | > loss_mel_ce: 2.8044803142547607 (nan) | > loss: 0.011283248662948608 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13719775249313962) | > loader_time: 0.0045 (0.004656438341114092)  --> TIME: 2025-02-26 18:27:16 -- STEP: 24382/36938 -- GLOBAL_STEP: 1354150 | > loss_text_ce: 0.03055202029645443 (0.03091792105718485) | > loss_mel_ce: 1.8692537546157837 (nan) | > loss: 0.0075389123521745205 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.1371895391275537) | > loader_time: 0.0046 (0.0046562899049845344)  --> TIME: 2025-02-26 18:27:26 -- STEP: 24432/36938 -- GLOBAL_STEP: 1354200 | > loss_text_ce: 0.04508752375841141 (0.03091817279565471) | > loss_mel_ce: 1.261684536933899 (nan) | > loss: 0.005185604095458984 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.1372024464642358) | > loader_time: 0.0045 (0.004656335976909161)  --> TIME: 2025-02-26 18:27:35 -- STEP: 24482/36938 -- GLOBAL_STEP: 1354250 | > loss_text_ce: 0.0367233045399189 (0.030920545029360424) | > loss_mel_ce: 3.015899419784546 (nan) | > loss: 0.012113583274185658 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13719678928177864) | > loader_time: 0.0048 (0.004656333664684596)  --> TIME: 2025-02-26 18:27:45 -- STEP: 24532/36938 -- GLOBAL_STEP: 1354300 | > loss_text_ce: 0.017395049333572388 (0.030923201458789976) | > loss_mel_ce: 2.0204288959503174 (nan) | > loss: 0.00808660313487053 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13719805238803878) | > loader_time: 0.0046 (0.004656307696907385)  --> TIME: 2025-02-26 18:27:54 -- STEP: 24582/36938 -- GLOBAL_STEP: 1354350 | > loss_text_ce: 0.043171729892492294 (0.03092447774670472) | > loss_mel_ce: 2.02520489692688 (nan) | > loss: 0.008207843638956547 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13720113197593775) | > loader_time: 0.0046 (0.004656218869450819)  --> TIME: 2025-02-26 18:28:03 -- STEP: 24632/36938 -- GLOBAL_STEP: 1354400 | > loss_text_ce: 0.027504760771989822 (0.030924169089680383) | > loss_mel_ce: 0.27490234375 (nan) | > loss: 0.0012000283459201455 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13719701827946748) | > loader_time: 0.0045 (0.004656096515658607)  --> TIME: 2025-02-26 18:28:13 -- STEP: 24682/36938 -- GLOBAL_STEP: 1354450 | > loss_text_ce: 0.03679557889699936 (0.030924403307452) | > loss_mel_ce: 2.4330475330352783 (nan) | > loss: 0.009800965897738934 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13719745812873052) | > loader_time: 0.0044 (0.004656004901836833)  --> TIME: 2025-02-26 18:28:22 -- STEP: 24732/36938 -- GLOBAL_STEP: 1354500 | > loss_text_ce: 0.02113492600619793 (0.030923357725821398) | > loss_mel_ce: 2.5679075717926025 (nan) | > loss: 0.010273978114128113 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.13719413009793346) | > loader_time: 0.0046 (0.004655897935462882)  --> TIME: 2025-02-26 18:28:31 -- STEP: 24782/36938 -- GLOBAL_STEP: 1354550 | > loss_text_ce: 0.03326518088579178 (0.030924297808569163) | > loss_mel_ce: 2.739940881729126 (nan) | > loss: 0.011004786007106304 (nan) | > current_lr: 5e-06 | > step_time: 0.1776 (0.13718844939736544) | > loader_time: 0.0048 (0.004655760403031938)  --> TIME: 2025-02-26 18:28:41 -- STEP: 24832/36938 -- GLOBAL_STEP: 1354600 | > loss_text_ce: 0.03693613037467003 (0.030927357145441245) | > loss_mel_ce: 3.070357084274292 (nan) | > loss: 0.012330529280006886 (nan) | > current_lr: 5e-06 | > step_time: 0.1452 (0.13718846147476557) | > loader_time: 0.0046 (0.004655687445678653)  --> TIME: 2025-02-26 18:28:50 -- STEP: 24882/36938 -- GLOBAL_STEP: 1354650 | > loss_text_ce: 0.030284341424703598 (0.030926863104767496) | > loss_mel_ce: 2.567864179611206 (nan) | > loss: 0.010310114361345768 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13719676158194433) | > loader_time: 0.0044 (0.004655639071832863)  --> TIME: 2025-02-26 18:29:00 -- STEP: 24932/36938 -- GLOBAL_STEP: 1354700 | > loss_text_ce: 0.029423274099826813 (0.03092760049035949) | > loss_mel_ce: 2.9477434158325195 (nan) | > loss: 0.011814153753221035 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13721698498660712) | > loader_time: 0.0046 (0.004655666036129986)  --> TIME: 2025-02-26 18:29:10 -- STEP: 24982/36938 -- GLOBAL_STEP: 1354750 | > loss_text_ce: 0.0412033349275589 (0.030929570121436527) | > loss_mel_ce: 2.4461793899536133 (nan) | > loss: 0.009870566427707672 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.1372293969206947) | > loader_time: 0.0049 (0.004655875528644166)  --> TIME: 2025-02-26 18:29:19 -- STEP: 25032/36938 -- GLOBAL_STEP: 1354800 | > loss_text_ce: 0.02720232680439949 (0.030927568349638872) | > loss_mel_ce: 2.7123453617095947 (nan) | > loss: 0.0108712213113904 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13722959182711075) | > loader_time: 0.0046 (0.0046559037702005)  --> TIME: 2025-02-26 18:29:29 -- STEP: 25082/36938 -- GLOBAL_STEP: 1354850 | > loss_text_ce: 0.04410793259739876 (0.030928856456256017) | > loss_mel_ce: 2.6804065704345703 (nan) | > loss: 0.01081156637519598 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.1372264993598541) | > loader_time: 0.0046 (0.004656072296353668)  --> TIME: 2025-02-26 18:29:38 -- STEP: 25132/36938 -- GLOBAL_STEP: 1354900 | > loss_text_ce: 0.036742959171533585 (0.030930546445447574) | > loss_mel_ce: 2.6603360176086426 (nan) | > loss: 0.01070269476622343 (nan) | > current_lr: 5e-06 | > step_time: 0.1705 (0.13722215463605386) | > loader_time: 0.0049 (0.004656125202160732)  --> TIME: 2025-02-26 18:29:47 -- STEP: 25182/36938 -- GLOBAL_STEP: 1354950 | > loss_text_ce: 0.02773790992796421 (0.03092894074747451) | > loss_mel_ce: 3.5753631591796875 (nan) | > loss: 0.014298020862042904 (nan) | > current_lr: 5e-06 | > step_time: 0.1878 (0.13722864701573365) | > loader_time: 0.0048 (0.004656187280481145)  --> TIME: 2025-02-26 18:29:57 -- STEP: 25232/36938 -- GLOBAL_STEP: 1355000 | > loss_text_ce: 0.015296310186386108 (0.030929101994557163) | > loss_mel_ce: 2.6491734981536865 (nan) | > loss: 0.010573293082416058 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1372287881926171) | > loader_time: 0.0046 (0.0046562824962860195)  --> TIME: 2025-02-26 18:30:06 -- STEP: 25282/36938 -- GLOBAL_STEP: 1355050 | > loss_text_ce: 0.026422709226608276 (0.030928934642885337) | > loss_mel_ce: 3.1092135906219482 (nan) | > loss: 0.012443002313375473 (nan) | > current_lr: 5e-06 | > step_time: 0.1997 (0.13723232809119001) | > loader_time: 0.0049 (0.0046564817107956295)  --> TIME: 2025-02-26 18:30:15 -- STEP: 25332/36938 -- GLOBAL_STEP: 1355100 | > loss_text_ce: 0.0355471596121788 (0.03092867332076753) | > loss_mel_ce: 1.8657214641571045 (nan) | > loss: 0.007544717285782099 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13722807668724632) | > loader_time: 0.0046 (0.004656504007306643)  --> TIME: 2025-02-26 18:30:24 -- STEP: 25382/36938 -- GLOBAL_STEP: 1355150 | > loss_text_ce: 0.02864881418645382 (0.03092926565648201) | > loss_mel_ce: 1.8541914224624634 (nan) | > loss: 0.007471588905900717 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13722173508792837) | > loader_time: 0.0046 (0.004656668541965141)  --> TIME: 2025-02-26 18:30:34 -- STEP: 25432/36938 -- GLOBAL_STEP: 1355200 | > loss_text_ce: 0.025879405438899994 (0.03093059911624727) | > loss_mel_ce: 2.8150851726531982 (nan) | > loss: 0.011273669078946114 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1372197351953663) | > loader_time: 0.0045 (0.004656809564655245)  --> TIME: 2025-02-26 18:30:43 -- STEP: 25482/36938 -- GLOBAL_STEP: 1355250 | > loss_text_ce: 0.02933059073984623 (0.030930260079976797) | > loss_mel_ce: 2.229998826980591 (nan) | > loss: 0.008965592831373215 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13722508974047498) | > loader_time: 0.0047 (0.004656794793320805)  --> TIME: 2025-02-26 18:30:53 -- STEP: 25532/36938 -- GLOBAL_STEP: 1355300 | > loss_text_ce: 0.022375792264938354 (0.03093160881173039) | > loss_mel_ce: 0.13287353515625 (nan) | > loss: 0.0006160687771625817 (nan) | > current_lr: 5e-06 | > step_time: 0.1434 (0.13722677612199985) | > loader_time: 0.0047 (0.004656806702563774)  --> TIME: 2025-02-26 18:31:02 -- STEP: 25582/36938 -- GLOBAL_STEP: 1355350 | > loss_text_ce: 0.035870227962732315 (0.030931611176360736) | > loss_mel_ce: 3.1854403018951416 (nan) | > loss: 0.012782979756593704 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13722203032478797) | > loader_time: 0.0046 (0.004656865732653611)  --> TIME: 2025-02-26 18:31:11 -- STEP: 25632/36938 -- GLOBAL_STEP: 1355400 | > loss_text_ce: 0.037435442209243774 (0.030933748571109573) | > loss_mel_ce: 2.936012029647827 (nan) | > loss: 0.011799396015703678 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.13721466575110783) | > loader_time: 0.0048 (0.004656832688458998)  --> TIME: 2025-02-26 18:31:20 -- STEP: 25682/36938 -- GLOBAL_STEP: 1355450 | > loss_text_ce: 0.020400620996952057 (0.03093256213930432) | > loss_mel_ce: 2.5615077018737793 (nan) | > loss: 0.010245667770504951 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13721361511936792) | > loader_time: 0.0045 (0.004656881467639335)  --> TIME: 2025-02-26 18:31:30 -- STEP: 25732/36938 -- GLOBAL_STEP: 1355500 | > loss_text_ce: 0.03634779155254364 (0.030934199104037063) | > loss_mel_ce: 2.736210346221924 (nan) | > loss: 0.011002215556800365 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.13722926700687701) | > loader_time: 0.0046 (0.00465691673353525)  --> TIME: 2025-02-26 18:31:40 -- STEP: 25782/36938 -- GLOBAL_STEP: 1355550 | > loss_text_ce: 0.038788363337516785 (0.030932995539766743) | > loss_mel_ce: 2.3475341796875 (nan) | > loss: 0.009469534270465374 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13723347987134268) | > loader_time: 0.0045 (0.00465687392486851)  --> TIME: 2025-02-26 18:31:49 -- STEP: 25832/36938 -- GLOBAL_STEP: 1355600 | > loss_text_ce: 0.031179362908005714 (0.030935233055496975) | > loss_mel_ce: 0.0562947578728199 (nan) | > loss: 0.0003471195523161441 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.1372324567843077) | > loader_time: 0.0045 (0.004656788336673209)  --> TIME: 2025-02-26 18:31:58 -- STEP: 25882/36938 -- GLOBAL_STEP: 1355650 | > loss_text_ce: 0.03385156765580177 (0.030933971072878683) | > loss_mel_ce: 1.0899497270584106 (nan) | > loss: 0.004459529649466276 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13723115041850725) | > loader_time: 0.0046 (0.0046566890312414655)  --> TIME: 2025-02-26 18:32:07 -- STEP: 25932/36938 -- GLOBAL_STEP: 1355700 | > loss_text_ce: 0.0429125539958477 (0.030932828749421697) | > loss_mel_ce: 3.5495800971984863 (nan) | > loss: 0.01425592415034771 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13722168964385575) | > loader_time: 0.0046 (0.004656668450754027)  --> TIME: 2025-02-26 18:32:17 -- STEP: 25982/36938 -- GLOBAL_STEP: 1355750 | > loss_text_ce: 0.04420328885316849 (0.03093470471604156) | > loss_mel_ce: 2.911170482635498 (nan) | > loss: 0.011727673932909966 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13722238269214407) | > loader_time: 0.0047 (0.004656722938185811)  --> TIME: 2025-02-26 18:32:26 -- STEP: 26032/36938 -- GLOBAL_STEP: 1355800 | > loss_text_ce: 0.046088479459285736 (0.030934933279411287) | > loss_mel_ce: 3.0904107093811035 (nan) | > loss: 0.01244642585515976 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13720890554498044) | > loader_time: 0.0047 (0.0046566480149062145)  --> TIME: 2025-02-26 18:32:35 -- STEP: 26082/36938 -- GLOBAL_STEP: 1355850 | > loss_text_ce: 0.03383736312389374 (0.03093612336818011) | > loss_mel_ce: 2.5962986946105957 (nan) | > loss: 0.01043704804033041 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13721206429344435) | > loader_time: 0.0046 (0.004656629578466288)  --> TIME: 2025-02-26 18:32:45 -- STEP: 26132/36938 -- GLOBAL_STEP: 1355900 | > loss_text_ce: 0.022748738527297974 (0.030933358761614228) | > loss_mel_ce: 3.1127774715423584 (nan) | > loss: 0.012442564591765404 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.1372150774299861) | > loader_time: 0.0047 (0.0046565699737907305)  --> TIME: 2025-02-26 18:32:54 -- STEP: 26182/36938 -- GLOBAL_STEP: 1355950 | > loss_text_ce: 0.026334920898079872 (0.03093549104673999) | > loss_mel_ce: 2.3757402896881104 (nan) | > loss: 0.00953204557299614 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13720807549459993) | > loader_time: 0.0046 (0.004656774376121835)  --> TIME: 2025-02-26 18:33:03 -- STEP: 26232/36938 -- GLOBAL_STEP: 1356000 | > loss_text_ce: 0.032927267253398895 (0.03093555203483679) | > loss_mel_ce: 2.5961973667144775 (nan) | > loss: 0.010433034971356392 (nan) | > current_lr: 5e-06 | > step_time: 0.1773 (0.13721506864277722) | > loader_time: 0.0048 (0.004656778235521283)  --> TIME: 2025-02-26 18:33:13 -- STEP: 26282/36938 -- GLOBAL_STEP: 1356050 | > loss_text_ce: 0.032214973121881485 (0.03093607390774491) | > loss_mel_ce: 2.829195976257324 (nan) | > loss: 0.011354805901646614 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.1372166779494676) | > loader_time: 0.0047 (0.004656841598701168)  --> TIME: 2025-02-26 18:33:22 -- STEP: 26332/36938 -- GLOBAL_STEP: 1356100 | > loss_text_ce: 0.03229311853647232 (0.030936564960861897) | > loss_mel_ce: 2.607879161834717 (nan) | > loss: 0.010476874187588692 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13720988435946502) | > loader_time: 0.0046 (0.004656720945542836)  --> TIME: 2025-02-26 18:33:31 -- STEP: 26382/36938 -- GLOBAL_STEP: 1356150 | > loss_text_ce: 0.033952005207538605 (0.03093454877748652) | > loss_mel_ce: 2.281001091003418 (nan) | > loss: 0.00918632186949253 (nan) | > current_lr: 5e-06 | > step_time: 0.1767 (0.13721094698280828) | > loader_time: 0.0047 (0.004656638461819971)  --> TIME: 2025-02-26 18:33:41 -- STEP: 26432/36938 -- GLOBAL_STEP: 1356200 | > loss_text_ce: 0.02676224149763584 (0.030933413141601987) | > loss_mel_ce: 3.0545461177825928 (nan) | > loss: 0.012227415107190609 (nan) | > current_lr: 5e-06 | > step_time: 0.1264 (0.13721179354205154) | > loader_time: 0.0046 (0.004656585542255103)  --> TIME: 2025-02-26 18:33:50 -- STEP: 26482/36938 -- GLOBAL_STEP: 1356250 | > loss_text_ce: 0.027693774551153183 (0.03093750288328564) | > loss_mel_ce: 2.7132742404937744 (nan) | > loss: 0.010876857675611973 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13721801953574558) | > loader_time: 0.0045 (0.004656489562901849)  --> TIME: 2025-02-26 18:33:59 -- STEP: 26532/36938 -- GLOBAL_STEP: 1356300 | > loss_text_ce: 0.02737494558095932 (0.030936980172109288) | > loss_mel_ce: 3.1611878871917725 (nan) | > loss: 0.012653027661144733 (nan) | > current_lr: 5e-06 | > step_time: 0.1574 (0.1372159191445305) | > loader_time: 0.0046 (0.0046564137685825464)  --> TIME: 2025-02-26 18:34:08 -- STEP: 26582/36938 -- GLOBAL_STEP: 1356350 | > loss_text_ce: 0.025812778621912003 (0.030936187662987395) | > loss_mel_ce: 2.792900562286377 (nan) | > loss: 0.011185371316969395 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13720409967552835) | > loader_time: 0.0047 (0.004656284211155661)  --> TIME: 2025-02-26 18:34:17 -- STEP: 26632/36938 -- GLOBAL_STEP: 1356400 | > loss_text_ce: 0.026888158172369003 (0.030935646835259265) | > loss_mel_ce: 2.6024277210235596 (nan) | > loss: 0.010433793999254704 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13718927701552938) | > loader_time: 0.0046 (0.004656395492484949)  --> TIME: 2025-02-26 18:34:27 -- STEP: 26682/36938 -- GLOBAL_STEP: 1356450 | > loss_text_ce: 0.023465855047106743 (0.030935194225089387) | > loss_mel_ce: 2.6176583766937256 (nan) | > loss: 0.010480652563273907 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13718464392234564) | > loader_time: 0.0045 (0.0046565083761853495)  --> TIME: 2025-02-26 18:34:36 -- STEP: 26732/36938 -- GLOBAL_STEP: 1356500 | > loss_text_ce: 0.02788296900689602 (0.030936141644311015) | > loss_mel_ce: 2.41689395904541 (nan) | > loss: 0.009701496921479702 (nan) | > current_lr: 5e-06 | > step_time: 0.168 (0.1371966147608296) | > loader_time: 0.0051 (0.004656570178563765)  --> TIME: 2025-02-26 18:34:46 -- STEP: 26782/36938 -- GLOBAL_STEP: 1356550 | > loss_text_ce: 0.03031506948173046 (0.030935363328631275) | > loss_mel_ce: 2.6789164543151855 (nan) | > loss: 0.010750919580459595 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13719365015635898) | > loader_time: 0.0046 (0.00465665798494915)  --> TIME: 2025-02-26 18:34:55 -- STEP: 26832/36938 -- GLOBAL_STEP: 1356600 | > loss_text_ce: 0.02816099300980568 (0.030934740916982575) | > loss_mel_ce: 2.484811782836914 (nan) | > loss: 0.00997211504727602 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13719259647649104) | > loader_time: 0.0046 (0.004656661806109405)  --> TIME: 2025-02-26 18:35:04 -- STEP: 26882/36938 -- GLOBAL_STEP: 1356650 | > loss_text_ce: 0.04519890621304512 (0.030931306402281766) | > loss_mel_ce: 2.603510856628418 (nan) | > loss: 0.010510753840208054 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13718800982516438) | > loader_time: 0.0047 (0.004656829362858897)  --> TIME: 2025-02-26 18:35:14 -- STEP: 26932/36938 -- GLOBAL_STEP: 1356700 | > loss_text_ce: 0.025870846584439278 (0.030930191726292668) | > loss_mel_ce: 2.102320671081543 (nan) | > loss: 0.00844520516693592 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.13719337347374208) | > loader_time: 0.0047 (0.004656770644350099)  --> TIME: 2025-02-26 18:35:23 -- STEP: 26982/36938 -- GLOBAL_STEP: 1356750 | > loss_text_ce: 0.029898205772042274 (0.030930363793918347) | > loss_mel_ce: 2.1152875423431396 (nan) | > loss: 0.008512642234563828 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13718984567264544) | > loader_time: 0.0045 (0.004656968146096455)  --> TIME: 2025-02-26 18:35:32 -- STEP: 27032/36938 -- GLOBAL_STEP: 1356800 | > loss_text_ce: 0.03825956955552101 (0.03092990639712482) | > loss_mel_ce: 0.19206809997558594 (nan) | > loss: 0.0009139987523667514 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13718742231851316) | > loader_time: 0.0045 (0.004656830617893932)  --> TIME: 2025-02-26 18:35:42 -- STEP: 27082/36938 -- GLOBAL_STEP: 1356850 | > loss_text_ce: 0.037558309733867645 (0.030931934905606064) | > loss_mel_ce: 2.185473680496216 (nan) | > loss: 0.008821556344628334 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13719407865580444) | > loader_time: 0.0046 (0.00465672400508142)  --> TIME: 2025-02-26 18:35:51 -- STEP: 27132/36938 -- GLOBAL_STEP: 1356900 | > loss_text_ce: 0.032377153635025024 (0.03092968461791828) | > loss_mel_ce: 0.9704834222793579 (nan) | > loss: 0.003979605622589588 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.1371949637215621) | > loader_time: 0.0048 (0.004656825245929357)  --> TIME: 2025-02-26 18:36:00 -- STEP: 27182/36938 -- GLOBAL_STEP: 1356950 | > loss_text_ce: 0.037948161363601685 (0.030929337328652915) | > loss_mel_ce: 0.6802825927734375 (nan) | > loss: 0.0028501220513135195 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13719318770176392) | > loader_time: 0.0045 (0.0046567686977138)  --> TIME: 2025-02-26 18:36:10 -- STEP: 27232/36938 -- GLOBAL_STEP: 1357000 | > loss_text_ce: 0.025482438504695892 (0.030930629754527227) | > loss_mel_ce: 2.1539382934570312 (nan) | > loss: 0.008648495189845562 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13719889247564482) | > loader_time: 0.0046 (0.004656746712120371)  --> TIME: 2025-02-26 18:36:19 -- STEP: 27282/36938 -- GLOBAL_STEP: 1357050 | > loss_text_ce: 0.017885928973555565 (0.03092993612285258) | > loss_mel_ce: 2.493595600128174 (nan) | > loss: 0.009966197423636913 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13719030887218356) | > loader_time: 0.0045 (0.004656633029680735)  --> TIME: 2025-02-26 18:36:28 -- STEP: 27332/36938 -- GLOBAL_STEP: 1357100 | > loss_text_ce: 0.028118683025240898 (0.03093016426950667) | > loss_mel_ce: 2.8042807579040527 (nan) | > loss: 0.011239680461585522 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.13718855861652324) | > loader_time: 0.0049 (0.004656679953381537)  --> TIME: 2025-02-26 18:36:38 -- STEP: 27382/36938 -- GLOBAL_STEP: 1357150 | > loss_text_ce: 0.03282160684466362 (0.030931463110024003) | > loss_mel_ce: 2.529939651489258 (nan) | > loss: 0.010169688612222672 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13718587372999702) | > loader_time: 0.0046 (0.004656708951881374)  --> TIME: 2025-02-26 18:36:47 -- STEP: 27432/36938 -- GLOBAL_STEP: 1357200 | > loss_text_ce: 0.03625805303454399 (0.030928697696852053) | > loss_mel_ce: 0.6863759160041809 (nan) | > loss: 0.0028675952926278114 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13718643914022696) | > loader_time: 0.0045 (0.004656598610723613)  --> TIME: 2025-02-26 18:36:56 -- STEP: 27482/36938 -- GLOBAL_STEP: 1357250 | > loss_text_ce: 0.02381937764585018 (0.030930234073631373) | > loss_mel_ce: 2.7909646034240723 (nan) | > loss: 0.011169778183102608 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13718289905097183) | > loader_time: 0.0045 (0.004656472465338002)  --> TIME: 2025-02-26 18:37:05 -- STEP: 27532/36938 -- GLOBAL_STEP: 1357300 | > loss_text_ce: 0.03446108102798462 (0.030929518698128246) | > loss_mel_ce: 2.9497592449188232 (nan) | > loss: 0.011842144653201103 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13717399576315356) | > loader_time: 0.0046 (0.004656455734346575)  --> TIME: 2025-02-26 18:37:15 -- STEP: 27582/36938 -- GLOBAL_STEP: 1357350 | > loss_text_ce: 0.0186533834785223 (0.030930391971871293) | > loss_mel_ce: 2.3893837928771973 (nan) | > loss: 0.009555703960359097 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1371785543367911) | > loader_time: 0.0046 (0.0046565040322558695)  --> TIME: 2025-02-26 18:37:24 -- STEP: 27632/36938 -- GLOBAL_STEP: 1357400 | > loss_text_ce: 0.03329835459589958 (0.030931474576370795) | > loss_mel_ce: 3.078173875808716 (nan) | > loss: 0.012347112409770489 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13718351706232518) | > loader_time: 0.0045 (0.004656438114467751)  --> TIME: 2025-02-26 18:37:34 -- STEP: 27682/36938 -- GLOBAL_STEP: 1357450 | > loss_text_ce: 0.027395447716116905 (0.030930991093571246) | > loss_mel_ce: 2.7145700454711914 (nan) | > loss: 0.010880815796554089 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.1371909660907516) | > loader_time: 0.0047 (0.004656320827112869)  --> TIME: 2025-02-26 18:37:43 -- STEP: 27732/36938 -- GLOBAL_STEP: 1357500 | > loss_text_ce: 0.02620004117488861 (0.030930895177630945) | > loss_mel_ce: 3.1472043991088867 (nan) | > loss: 0.012592875398695469 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.13718761619544148) | > loader_time: 0.0047 (0.0046562193861328025)  --> TIME: 2025-02-26 18:37:52 -- STEP: 27782/36938 -- GLOBAL_STEP: 1357550 | > loss_text_ce: 0.03328985720872879 (0.030930656867051366) | > loss_mel_ce: 2.565443515777588 (nan) | > loss: 0.01031243521720171 (nan) | > current_lr: 5e-06 | > step_time: 0.1082 (0.13718012896326193) | > loader_time: 0.0045 (0.0046561863036042594)  --> TIME: 2025-02-26 18:38:01 -- STEP: 27832/36938 -- GLOBAL_STEP: 1357600 | > loss_text_ce: 0.02692308835685253 (0.030931071042044785) | > loss_mel_ce: 2.564176082611084 (nan) | > loss: 0.010282140225172043 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13717623388640407) | > loader_time: 0.0046 (0.004656378643442122)  --> TIME: 2025-02-26 18:38:10 -- STEP: 27882/36938 -- GLOBAL_STEP: 1357650 | > loss_text_ce: 0.031663928180933 (0.030931667769433308) | > loss_mel_ce: 1.5313985347747803 (nan) | > loss: 0.0062026288360357285 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1371677276368337) | > loader_time: 0.0046 (0.004656275053566749)  --> TIME: 2025-02-26 18:38:20 -- STEP: 27932/36938 -- GLOBAL_STEP: 1357700 | > loss_text_ce: 0.02341046743094921 (0.030931167698607456) | > loss_mel_ce: 1.3383413553237915 (nan) | > loss: 0.005403777118772268 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13717255015504964) | > loader_time: 0.0045 (0.004656355624790931)  --> TIME: 2025-02-26 18:38:29 -- STEP: 27982/36938 -- GLOBAL_STEP: 1357750 | > loss_text_ce: 0.02996640093624592 (0.03093037236193741) | > loss_mel_ce: 2.4846296310424805 (nan) | > loss: 0.00997855607420206 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13716621501511664) | > loader_time: 0.0047 (0.0046563500136543685)  --> TIME: 2025-02-26 18:38:39 -- STEP: 28032/36938 -- GLOBAL_STEP: 1357800 | > loss_text_ce: 0.033636484295129776 (0.03093036266319072) | > loss_mel_ce: 2.8002266883850098 (nan) | > loss: 0.011245490051805973 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13716552877739077) | > loader_time: 0.0045 (0.004656351983683277)  --> TIME: 2025-02-26 18:38:48 -- STEP: 28082/36938 -- GLOBAL_STEP: 1357850 | > loss_text_ce: 0.030318791046738625 (0.030928778615182218) | > loss_mel_ce: 2.026602029800415 (nan) | > loss: 0.00816238485276699 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13717183398772478) | > loader_time: 0.0046 (0.004656277459517256)  --> TIME: 2025-02-26 18:38:57 -- STEP: 28132/36938 -- GLOBAL_STEP: 1357900 | > loss_text_ce: 0.027390453964471817 (0.030929310130357043) | > loss_mel_ce: 1.4220755100250244 (nan) | > loss: 0.005751849617809057 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13717234057551544) | > loader_time: 0.0046 (0.004656402913533843)  --> TIME: 2025-02-26 18:39:07 -- STEP: 28182/36938 -- GLOBAL_STEP: 1357950 | > loss_text_ce: 0.0384652279317379 (0.030929907362596293) | > loss_mel_ce: 0.7595805525779724 (nan) | > loss: 0.0031668483279645443 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13717334331896539) | > loader_time: 0.0046 (0.004656409939798467)  --> TIME: 2025-02-26 18:39:16 -- STEP: 28232/36938 -- GLOBAL_STEP: 1358000 | > loss_text_ce: 0.02157672867178917 (0.030929021944064796) | > loss_mel_ce: 2.0938613414764404 (nan) | > loss: 0.008394595235586166 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13717743798300713) | > loader_time: 0.0045 (0.004656425082132934)  --> TIME: 2025-02-26 18:39:26 -- STEP: 28282/36938 -- GLOBAL_STEP: 1358050 | > loss_text_ce: 0.019364649429917336 (0.030930063413173146) | > loss_mel_ce: 2.6545138359069824 (nan) | > loss: 0.010610628873109818 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1371788729553076) | > loader_time: 0.0045 (0.004656369451265592)  --> TIME: 2025-02-26 18:39:35 -- STEP: 28332/36938 -- GLOBAL_STEP: 1358100 | > loss_text_ce: 0.03317887708544731 (0.030930177909794118) | > loss_mel_ce: 3.0089364051818848 (nan) | > loss: 0.012071886099874973 (nan) | > current_lr: 5e-06 | > step_time: 0.186 (0.1371831639782776) | > loader_time: 0.0048 (0.004656311500616082)  --> TIME: 2025-02-26 18:39:45 -- STEP: 28382/36938 -- GLOBAL_STEP: 1358150 | > loss_text_ce: 0.020822986960411072 (0.030930601940408593) | > loss_mel_ce: 2.6560163497924805 (nan) | > loss: 0.010622379370033741 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13718241856807262) | > loader_time: 0.0045 (0.00465623562620402)  --> TIME: 2025-02-26 18:39:54 -- STEP: 28432/36938 -- GLOBAL_STEP: 1358200 | > loss_text_ce: 0.02846635691821575 (0.03093288847794334) | > loss_mel_ce: 1.198886752128601 (nan) | > loss: 0.0048704491928219795 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13717699462634816) | > loader_time: 0.0045 (0.004656189904832796)  --> TIME: 2025-02-26 18:40:03 -- STEP: 28482/36938 -- GLOBAL_STEP: 1358250 | > loss_text_ce: 0.033216413110494614 (0.030932374651145587) | > loss_mel_ce: 2.3451731204986572 (nan) | > loss: 0.009438054636120796 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.13717689907422656) | > loader_time: 0.0048 (0.004656093733823481)  --> TIME: 2025-02-26 18:40:13 -- STEP: 28532/36938 -- GLOBAL_STEP: 1358300 | > loss_text_ce: 0.023632751777768135 (0.030932266878859943) | > loss_mel_ce: 2.338820695877075 (nan) | > loss: 0.009374815970659256 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13717788508843107) | > loader_time: 0.0046 (0.004656157611580579)  --> TIME: 2025-02-26 18:40:22 -- STEP: 28582/36938 -- GLOBAL_STEP: 1358350 | > loss_text_ce: 0.018698086962103844 (0.03092965294023595) | > loss_mel_ce: 2.57814359664917 (nan) | > loss: 0.010304927825927734 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13717466389671532) | > loader_time: 0.0046 (0.0046561335960117346)  --> TIME: 2025-02-26 18:40:31 -- STEP: 28632/36938 -- GLOBAL_STEP: 1358400 | > loss_text_ce: 0.02057097852230072 (0.030927855172015635) | > loss_mel_ce: 2.552396535873413 (nan) | > loss: 0.010210189037024975 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.13716755433933844) | > loader_time: 0.0046 (0.0046561253440549795)  --> TIME: 2025-02-26 18:40:40 -- STEP: 28682/36938 -- GLOBAL_STEP: 1358450 | > loss_text_ce: 0.023469306528568268 (0.030927973566641776) | > loss_mel_ce: 1.0900458097457886 (nan) | > loss: 0.004418711178004742 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13717241193632815) | > loader_time: 0.0045 (0.004656080346452933)  --> TIME: 2025-02-26 18:40:50 -- STEP: 28732/36938 -- GLOBAL_STEP: 1358500 | > loss_text_ce: 0.04155084490776062 (0.03092904332560722) | > loss_mel_ce: 0.053466796875 (nan) | > loss: 0.0003770541516132653 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13717489133517688) | > loader_time: 0.0046 (0.004656067477715757)  --> TIME: 2025-02-26 18:40:59 -- STEP: 28782/36938 -- GLOBAL_STEP: 1358550 | > loss_text_ce: 0.03278513252735138 (0.030929478434526106) | > loss_mel_ce: 2.435483455657959 (nan) | > loss: 0.009794717654585838 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.1371746819442267) | > loader_time: 0.0047 (0.00465607844418923)  --> TIME: 2025-02-26 18:41:09 -- STEP: 28832/36938 -- GLOBAL_STEP: 1358600 | > loss_text_ce: 0.03169754147529602 (0.030930566424748767) | > loss_mel_ce: 2.563161611557007 (nan) | > loss: 0.010297060944139957 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.137179781805793) | > loader_time: 0.0045 (0.004656120316103729)  --> TIME: 2025-02-26 18:41:18 -- STEP: 28882/36938 -- GLOBAL_STEP: 1358650 | > loss_text_ce: 0.03353450074791908 (0.030930392334997916) | > loss_mel_ce: 3.1052680015563965 (nan) | > loss: 0.012455565854907036 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13718603071054541) | > loader_time: 0.0045 (0.004656056858860747)  --> TIME: 2025-02-26 18:41:28 -- STEP: 28932/36938 -- GLOBAL_STEP: 1358700 | > loss_text_ce: 0.029612809419631958 (0.03092999340664465) | > loss_mel_ce: 2.1266887187957764 (nan) | > loss: 0.00855675246566534 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13718486027517712) | > loader_time: 0.0045 (0.00465613774173439)  --> TIME: 2025-02-26 18:41:38 -- STEP: 28982/36938 -- GLOBAL_STEP: 1358750 | > loss_text_ce: 0.03682642802596092 (0.030929150527103298) | > loss_mel_ce: 2.9012622833251953 (nan) | > loss: 0.01165908295661211 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.1371966999976093) | > loader_time: 0.0047 (0.004656112142234713)  --> TIME: 2025-02-26 18:41:47 -- STEP: 29032/36938 -- GLOBAL_STEP: 1358800 | > loss_text_ce: 0.03346763923764229 (0.03093028527735002) | > loss_mel_ce: 3.0134708881378174 (nan) | > loss: 0.012091025710105896 (nan) | > current_lr: 5e-06 | > step_time: 0.1928 (0.13719359113349372) | > loader_time: 0.0049 (0.0046560861381757645)  --> TIME: 2025-02-26 18:41:56 -- STEP: 29082/36938 -- GLOBAL_STEP: 1358850 | > loss_text_ce: 0.037192098796367645 (0.030931557004425365) | > loss_mel_ce: 3.081782102584839 (nan) | > loss: 0.012376882135868073 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13719491018782182) | > loader_time: 0.0046 (0.004655982521473709)  --> TIME: 2025-02-26 18:42:06 -- STEP: 29132/36938 -- GLOBAL_STEP: 1358900 | > loss_text_ce: 0.03573182597756386 (0.03093103386456811) | > loss_mel_ce: 2.5732853412628174 (nan) | > loss: 0.010353242978453636 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.1372009068011371) | > loader_time: 0.0046 (0.004655957459323128)  --> TIME: 2025-02-26 18:42:15 -- STEP: 29182/36938 -- GLOBAL_STEP: 1358950 | > loss_text_ce: 0.030812649056315422 (0.030932352163065963) | > loss_mel_ce: 2.4786832332611084 (nan) | > loss: 0.009958317503333092 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13719358120516104) | > loader_time: 0.0045 (0.004655895301129148)  --> TIME: 2025-02-26 18:42:25 -- STEP: 29232/36938 -- GLOBAL_STEP: 1359000 | > loss_text_ce: 0.03650912269949913 (0.030932496332908944) | > loss_mel_ce: 2.810412883758545 (nan) | > loss: 0.011297309771180153 (nan) | > current_lr: 5e-06 | > step_time: 0.1453 (0.13720167397669633) | > loader_time: 0.0047 (0.004655854757130945)  --> TIME: 2025-02-26 18:42:34 -- STEP: 29282/36938 -- GLOBAL_STEP: 1359050 | > loss_text_ce: 0.027952952310442924 (0.03093175213512934) | > loss_mel_ce: 2.7398884296417236 (nan) | > loss: 0.010983497835695744 (nan) | > current_lr: 5e-06 | > step_time: 0.1508 (0.13721053590214866) | > loader_time: 0.0047 (0.004655776685984764)  --> TIME: 2025-02-26 18:42:44 -- STEP: 29332/36938 -- GLOBAL_STEP: 1359100 | > loss_text_ce: 0.030911099165678024 (0.0309324671226223) | > loss_mel_ce: 2.710874080657959 (nan) | > loss: 0.010880100540816784 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1372180836401955) | > loader_time: 0.0045 (0.004655761509365605)  --> TIME: 2025-02-26 18:42:53 -- STEP: 29382/36938 -- GLOBAL_STEP: 1359150 | > loss_text_ce: 0.02556905522942543 (0.030931157787953547) | > loss_mel_ce: 3.076282262802124 (nan) | > loss: 0.012308933772146702 (nan) | > current_lr: 5e-06 | > step_time: 0.1882 (0.13722744838268341) | > loader_time: 0.0049 (0.004655782055492877)  --> TIME: 2025-02-26 18:43:03 -- STEP: 29432/36938 -- GLOBAL_STEP: 1359200 | > loss_text_ce: 0.029413148760795593 (0.030929228127984664) | > loss_mel_ce: 2.868738889694214 (nan) | > loss: 0.011500604450702667 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.1372375660258212) | > loader_time: 0.0046 (0.004655827975979493)  --> TIME: 2025-02-26 18:43:12 -- STEP: 29482/36938 -- GLOBAL_STEP: 1359250 | > loss_text_ce: 0.03724018856883049 (0.030927992343947316) | > loss_mel_ce: 2.5069189071655273 (nan) | > loss: 0.0100958701223135 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.1372333575876827) | > loader_time: 0.0046 (0.004655736295413974)  --> TIME: 2025-02-26 18:43:22 -- STEP: 29532/36938 -- GLOBAL_STEP: 1359300 | > loss_text_ce: 0.03167560324072838 (0.030927818412814938) | > loss_mel_ce: 2.979935646057129 (nan) | > loss: 0.011950839310884476 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.13722890843128122) | > loader_time: 0.0048 (0.004655709373875053)  --> TIME: 2025-02-26 18:43:31 -- STEP: 29582/36938 -- GLOBAL_STEP: 1359350 | > loss_text_ce: 0.03011341020464897 (0.030926498268834764) | > loss_mel_ce: 3.112382411956787 (nan) | > loss: 0.01247022207826376 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13722988079595186) | > loader_time: 0.0046 (0.004655719069371979)  --> TIME: 2025-02-26 18:43:40 -- STEP: 29632/36938 -- GLOBAL_STEP: 1359400 | > loss_text_ce: 0.022562775760889053 (0.030925612150761596) | > loss_mel_ce: 2.4519219398498535 (nan) | > loss: 0.009819384664297104 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13722431859238693) | > loader_time: 0.0046 (0.0046557026792550795)  --> TIME: 2025-02-26 18:43:50 -- STEP: 29682/36938 -- GLOBAL_STEP: 1359450 | > loss_text_ce: 0.027504682540893555 (0.030925081401881882) | > loss_mel_ce: 2.5473103523254395 (nan) | > loss: 0.010217520408332348 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13723386379033822) | > loader_time: 0.0046 (0.004655702256600512)  --> TIME: 2025-02-26 18:43:59 -- STEP: 29732/36938 -- GLOBAL_STEP: 1359500 | > loss_text_ce: 0.029857933521270752 (0.030924563423972123) | > loss_mel_ce: 2.480154514312744 (nan) | > loss: 0.00996036734431982 (nan) | > current_lr: 5e-06 | > step_time: 0.1789 (0.1372286261419049) | > loader_time: 0.0049 (0.004655603884239288)  --> TIME: 2025-02-26 18:44:08 -- STEP: 29782/36938 -- GLOBAL_STEP: 1359550 | > loss_text_ce: 0.0347202830016613 (0.030923951066062015) | > loss_mel_ce: 0.6882339119911194 (nan) | > loss: 0.0028688660822808743 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13723141270786493) | > loader_time: 0.0047 (0.004655589611308857)  --> TIME: 2025-02-26 18:44:18 -- STEP: 29832/36938 -- GLOBAL_STEP: 1359600 | > loss_text_ce: 0.03421560302376747 (0.030922792960348174) | > loss_mel_ce: 2.592050313949585 (nan) | > loss: 0.010421691462397575 (nan) | > current_lr: 5e-06 | > step_time: 0.1687 (0.1372259504819353) | > loader_time: 0.0048 (0.004655521735648857)  --> TIME: 2025-02-26 18:44:27 -- STEP: 29882/36938 -- GLOBAL_STEP: 1359650 | > loss_text_ce: 0.0283247921615839 (0.030922741029442966) | > loss_mel_ce: 2.7755563259124756 (nan) | > loss: 0.011126513592898846 (nan) | > current_lr: 5e-06 | > step_time: 0.1793 (0.13722319297702273) | > loader_time: 0.0047 (0.0046554754859240415)  --> TIME: 2025-02-26 18:44:36 -- STEP: 29932/36938 -- GLOBAL_STEP: 1359700 | > loss_text_ce: 0.03559251129627228 (0.030922970962140032) | > loss_mel_ce: 2.887479543685913 (nan) | > loss: 0.011599493212997913 (nan) | > current_lr: 5e-06 | > step_time: 0.1263 (0.13721936726879014) | > loader_time: 0.0046 (0.004655437945491168)  --> TIME: 2025-02-26 18:44:45 -- STEP: 29982/36938 -- GLOBAL_STEP: 1359750 | > loss_text_ce: 0.026523370295763016 (0.03092389388603079) | > loss_mel_ce: 2.7001705169677734 (nan) | > loss: 0.01082021463662386 (nan) | > current_lr: 5e-06 | > step_time: 0.1842 (0.13721979605126391) | > loader_time: 0.0048 (0.004655356611056258)  --> TIME: 2025-02-26 18:44:55 -- STEP: 30032/36938 -- GLOBAL_STEP: 1359800 | > loss_text_ce: 0.03336099535226822 (0.03092397983913713) | > loss_mel_ce: 1.35888671875 (nan) | > loss: 0.005524792708456516 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1372267483839263) | > loader_time: 0.0045 (0.004655319004536421)  --> TIME: 2025-02-26 18:45:04 -- STEP: 30082/36938 -- GLOBAL_STEP: 1359850 | > loss_text_ce: 0.026634901762008667 (0.030925759897630422) | > loss_mel_ce: 3.055050849914551 (nan) | > loss: 0.01222891267389059 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13721319275581875) | > loader_time: 0.0046 (0.0046552853194031735)  --> TIME: 2025-02-26 18:45:13 -- STEP: 30132/36938 -- GLOBAL_STEP: 1359900 | > loss_text_ce: 0.031850166618824005 (0.030925459392547067) | > loss_mel_ce: 2.7534403800964355 (nan) | > loss: 0.01105274073779583 (nan) | > current_lr: 5e-06 | > step_time: 0.1799 (0.13721438268733238) | > loader_time: 0.0048 (0.0046552416813985835)  --> TIME: 2025-02-26 18:45:23 -- STEP: 30182/36938 -- GLOBAL_STEP: 1359950 | > loss_text_ce: 0.03213261812925339 (0.030924054560581143) | > loss_mel_ce: 2.5182271003723145 (nan) | > loss: 0.01012047566473484 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.13721879535289644) | > loader_time: 0.0046 (0.004655219034396269)  --> TIME: 2025-02-26 18:45:33 -- STEP: 30232/36938 -- GLOBAL_STEP: 1360000 | > loss_text_ce: 0.02417215146124363 (0.030922908220063328) | > loss_mel_ce: 2.942570924758911 (nan) | > loss: 0.011772789992392063 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.1372265936079763) | > loader_time: 0.0046 (0.004655216816841326) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1360000.pth  --> TIME: 2025-02-26 18:45:49 -- STEP: 30282/36938 -- GLOBAL_STEP: 1360050 | > loss_text_ce: 0.022439710795879364 (0.030920838273255404) | > loss_mel_ce: 2.087547540664673 (nan) | > loss: 0.008372966200113297 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13723035570762632) | > loader_time: 0.0047 (0.004655382047594857)  --> TIME: 2025-02-26 18:45:58 -- STEP: 30332/36938 -- GLOBAL_STEP: 1360100 | > loss_text_ce: 0.03249749541282654 (0.03091969307482937) | > loss_mel_ce: 2.165618896484375 (nan) | > loss: 0.008722684346139431 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13722645628818422) | > loader_time: 0.0047 (0.0046554147120312995)  --> TIME: 2025-02-26 18:46:07 -- STEP: 30382/36938 -- GLOBAL_STEP: 1360150 | > loss_text_ce: 0.01683375984430313 (0.03092066769516786) | > loss_mel_ce: 1.6274847984313965 (nan) | > loss: 0.006525074131786823 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1372301656916867) | > loader_time: 0.0045 (0.004655397555911418)  --> TIME: 2025-02-26 18:46:16 -- STEP: 30432/36938 -- GLOBAL_STEP: 1360200 | > loss_text_ce: 0.025966975837945938 (0.030920118506177405) | > loss_mel_ce: 2.1677143573760986 (nan) | > loss: 0.008705085143446922 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13721967157912512) | > loader_time: 0.0046 (0.004655328646819282)  --> TIME: 2025-02-26 18:46:26 -- STEP: 30482/36938 -- GLOBAL_STEP: 1360250 | > loss_text_ce: 0.03254620358347893 (0.03092294275248212) | > loss_mel_ce: 2.6870336532592773 (nan) | > loss: 0.010791984386742115 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13722373516131903) | > loader_time: 0.0046 (0.0046553385475943945)  --> TIME: 2025-02-26 18:46:35 -- STEP: 30532/36938 -- GLOBAL_STEP: 1360300 | > loss_text_ce: 0.02555064670741558 (0.030924903155274838) | > loss_mel_ce: 2.4442481994628906 (nan) | > loss: 0.009800789877772331 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1372262120684164) | > loader_time: 0.0045 (0.004655295573726435)  --> TIME: 2025-02-26 18:46:45 -- STEP: 30582/36938 -- GLOBAL_STEP: 1360350 | > loss_text_ce: 0.028561849147081375 (0.03092372915291232) | > loss_mel_ce: 3.036975383758545 (nan) | > loss: 0.012164831161499023 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13723131921306622) | > loader_time: 0.0046 (0.004655281788433863)  --> TIME: 2025-02-26 18:46:54 -- STEP: 30632/36938 -- GLOBAL_STEP: 1360400 | > loss_text_ce: 0.03267710283398628 (0.030919905078838645) | > loss_mel_ce: 2.7809388637542725 (nan) | > loss: 0.011165143921971321 (nan) | > current_lr: 5e-06 | > step_time: 0.1689 (0.13722383365770413) | > loader_time: 0.0048 (0.004655201080481154)  --> TIME: 2025-02-26 18:47:03 -- STEP: 30682/36938 -- GLOBAL_STEP: 1360450 | > loss_text_ce: 0.0383288599550724 (0.030920211084085766) | > loss_mel_ce: 1.5195661783218384 (nan) | > loss: 0.006182123441249132 (nan) | > current_lr: 5e-06 | > step_time: 0.1121 (0.13722887859607896) | > loader_time: 0.0044 (0.00465519662460068)  --> TIME: 2025-02-26 18:47:13 -- STEP: 30732/36938 -- GLOBAL_STEP: 1360500 | > loss_text_ce: 0.02966916933655739 (0.030921317381104523) | > loss_mel_ce: 2.2651000022888184 (nan) | > loss: 0.0091062281280756 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13722800495536555) | > loader_time: 0.0045 (0.004655184882949839)  --> TIME: 2025-02-26 18:47:22 -- STEP: 30782/36938 -- GLOBAL_STEP: 1360550 | > loss_text_ce: 0.026496965438127518 (0.03092185326719809) | > loss_mel_ce: 0.8194335699081421 (nan) | > loss: 0.0033568672370165586 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1372247876948078) | > loader_time: 0.0046 (0.004655126064239302)  --> TIME: 2025-02-26 18:47:31 -- STEP: 30832/36938 -- GLOBAL_STEP: 1360600 | > loss_text_ce: 0.03790612146258354 (0.03092378467089652) | > loss_mel_ce: 0.06874694675207138 (nan) | > loss: 0.00042322647641412914 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13722859573568238) | > loader_time: 0.0046 (0.004655148329422999)  --> TIME: 2025-02-26 18:47:41 -- STEP: 30882/36938 -- GLOBAL_STEP: 1360650 | > loss_text_ce: 0.03490540012717247 (0.030924527755033803) | > loss_mel_ce: 2.493472099304199 (nan) | > loss: 0.01003324519842863 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.1372396241861182) | > loader_time: 0.0048 (0.004655209216697237)  --> TIME: 2025-02-26 18:47:51 -- STEP: 30932/36938 -- GLOBAL_STEP: 1360700 | > loss_text_ce: 0.02792268618941307 (0.03092148859138395) | > loss_mel_ce: 2.212984323501587 (nan) | > loss: 0.008892488665878773 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1372411092286813) | > loader_time: 0.0045 (0.004655155060471022)  --> TIME: 2025-02-26 18:48:00 -- STEP: 30982/36938 -- GLOBAL_STEP: 1360750 | > loss_text_ce: 0.027630312368273735 (0.030919534104486817) | > loss_mel_ce: 0.9697221517562866 (nan) | > loss: 0.00395774794742465 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13724609087284587) | > loader_time: 0.0045 (0.004655208252743087)  --> TIME: 2025-02-26 18:48:10 -- STEP: 31032/36938 -- GLOBAL_STEP: 1360800 | > loss_text_ce: 0.03852849453687668 (0.03091973747898298) | > loss_mel_ce: 3.0796890258789062 (nan) | > loss: 0.012373879551887512 (nan) | > current_lr: 5e-06 | > step_time: 0.1908 (0.13726238511861757) | > loader_time: 0.0049 (0.004655250256194915)  --> TIME: 2025-02-26 18:48:20 -- STEP: 31082/36938 -- GLOBAL_STEP: 1360850 | > loss_text_ce: 0.026571227237582207 (0.030920573602187276) | > loss_mel_ce: 2.66180419921875 (nan) | > loss: 0.01066815759986639 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13726183234567407) | > loader_time: 0.0045 (0.00465522967022469)  --> TIME: 2025-02-26 18:48:29 -- STEP: 31132/36938 -- GLOBAL_STEP: 1360900 | > loss_text_ce: 0.03482193499803543 (0.030918615305766107) | > loss_mel_ce: 2.3454771041870117 (nan) | > loss: 0.009445631876587868 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1372535078549727) | > loader_time: 0.0046 (0.0046551631239222235)  --> TIME: 2025-02-26 18:48:38 -- STEP: 31182/36938 -- GLOBAL_STEP: 1360950 | > loss_text_ce: 0.032788991928100586 (0.030916161687499763) | > loss_mel_ce: 3.1645965576171875 (nan) | > loss: 0.012688038870692253 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13725225208706326) | > loader_time: 0.0046 (0.004655198475615835)  --> TIME: 2025-02-26 18:48:47 -- STEP: 31232/36938 -- GLOBAL_STEP: 1361000 | > loss_text_ce: 0.03693129122257233 (0.03091508641260648) | > loss_mel_ce: 2.719148874282837 (nan) | > loss: 0.010936826467514038 (nan) | > current_lr: 5e-06 | > step_time: 0.1793 (0.13725022848725887) | > loader_time: 0.0048 (0.0046551642695166874)  --> TIME: 2025-02-26 18:48:57 -- STEP: 31282/36938 -- GLOBAL_STEP: 1361050 | > loss_text_ce: 0.03023628145456314 (0.030914256147442233) | > loss_mel_ce: 3.0879006385803223 (nan) | > loss: 0.012373560108244419 (nan) | > current_lr: 5e-06 | > step_time: 0.185 (0.13725007532289213) | > loader_time: 0.0048 (0.004655115882283792)  --> TIME: 2025-02-26 18:49:06 -- STEP: 31332/36938 -- GLOBAL_STEP: 1361100 | > loss_text_ce: 0.034692056477069855 (0.03091380541266598) | > loss_mel_ce: 2.768219470977783 (nan) | > loss: 0.011122665368020535 (nan) | > current_lr: 5e-06 | > step_time: 0.1858 (0.1372461605312463) | > loader_time: 0.0049 (0.00465508404780084)  --> TIME: 2025-02-26 18:49:15 -- STEP: 31382/36938 -- GLOBAL_STEP: 1361150 | > loss_text_ce: 0.030614839866757393 (0.030914350332954962) | > loss_mel_ce: 2.8340439796447754 (nan) | > loss: 0.011367694474756718 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.1372466129810219) | > loader_time: 0.0046 (0.004655036413603474)  --> TIME: 2025-02-26 18:49:25 -- STEP: 31432/36938 -- GLOBAL_STEP: 1361200 | > loss_text_ce: 0.03213542327284813 (0.03091576499767273) | > loss_mel_ce: 3.2398159503936768 (nan) | > loss: 0.012983934953808784 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.1372583313627244) | > loader_time: 0.0046 (0.004655064783135134)  --> TIME: 2025-02-26 18:49:34 -- STEP: 31482/36938 -- GLOBAL_STEP: 1361250 | > loss_text_ce: 0.03410134091973305 (0.030916043462029523) | > loss_mel_ce: 3.3133535385131836 (nan) | > loss: 0.013283551670610905 (nan) | > current_lr: 5e-06 | > step_time: 0.1874 (0.13725831596333637) | > loader_time: 0.0049 (0.00465504292815924)  --> TIME: 2025-02-26 18:49:44 -- STEP: 31532/36938 -- GLOBAL_STEP: 1361300 | > loss_text_ce: 0.03023918718099594 (0.030916145656136887) | > loss_mel_ce: 1.9760589599609375 (nan) | > loss: 0.007961500436067581 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.1372524288256231) | > loader_time: 0.0046 (0.004654997022385118)  --> TIME: 2025-02-26 18:49:53 -- STEP: 31582/36938 -- GLOBAL_STEP: 1361350 | > loss_text_ce: 0.03435659036040306 (0.030916818606810358) | > loss_mel_ce: 3.095780611038208 (nan) | > loss: 0.012421180494129658 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13725939849203278) | > loader_time: 0.0046 (0.004654979979091279)  --> TIME: 2025-02-26 18:50:03 -- STEP: 31632/36938 -- GLOBAL_STEP: 1361400 | > loss_text_ce: 0.0405481792986393 (0.030915789472631218) | > loss_mel_ce: 1.0895742177963257 (nan) | > loss: 0.004484612960368395 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1372579289510732) | > loader_time: 0.0046 (0.00465494407869433)  --> TIME: 2025-02-26 18:50:12 -- STEP: 31682/36938 -- GLOBAL_STEP: 1361450 | > loss_text_ce: 0.031222108751535416 (0.030915368594756933) | > loss_mel_ce: 0.12526652216911316 (nan) | > loss: 0.0006209866260178387 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.1372699896860766) | > loader_time: 0.0046 (0.004654963467580307)  --> TIME: 2025-02-26 18:50:22 -- STEP: 31732/36938 -- GLOBAL_STEP: 1361500 | > loss_text_ce: 0.03762511536478996 (0.03091408524665308) | > loss_mel_ce: 2.6986751556396484 (nan) | > loss: 0.010858334600925446 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1372699605536508) | > loader_time: 0.0045 (0.004654936489625504)  --> TIME: 2025-02-26 18:50:31 -- STEP: 31782/36938 -- GLOBAL_STEP: 1361550 | > loss_text_ce: 0.025575468316674232 (0.03091415252581545) | > loss_mel_ce: 2.6709272861480713 (nan) | > loss: 0.01070040836930275 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13726746033754839) | > loader_time: 0.0045 (0.004654883880776056)  --> TIME: 2025-02-26 18:50:41 -- STEP: 31832/36938 -- GLOBAL_STEP: 1361600 | > loss_text_ce: 0.02959718555212021 (0.030914402147298608) | > loss_mel_ce: 2.415271282196045 (nan) | > loss: 0.009701860137283802 (nan) | > current_lr: 5e-06 | > step_time: 0.1618 (0.13727115348943744) | > loader_time: 0.0047 (0.004654845173678272)  --> TIME: 2025-02-26 18:50:50 -- STEP: 31882/36938 -- GLOBAL_STEP: 1361650 | > loss_text_ce: 0.03723602369427681 (0.030914960355576122) | > loss_mel_ce: 2.6494216918945312 (nan) | > loss: 0.010661340318620205 (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.1372694079575757) | > loader_time: 0.0046 (0.004654796372826354)  --> TIME: 2025-02-26 18:50:59 -- STEP: 31932/36938 -- GLOBAL_STEP: 1361700 | > loss_text_ce: 0.031074080616235733 (0.030915201258730478) | > loss_mel_ce: 2.9182374477386475 (nan) | > loss: 0.011703617870807648 (nan) | > current_lr: 5e-06 | > step_time: 0.1887 (0.13726837157486185) | > loader_time: 0.0049 (0.004654806814261805)  --> TIME: 2025-02-26 18:51:09 -- STEP: 31982/36938 -- GLOBAL_STEP: 1361750 | > loss_text_ce: 0.031771883368492126 (0.03091395200852713) | > loss_mel_ce: 2.716249465942383 (nan) | > loss: 0.010904847644269466 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13727299710553226) | > loader_time: 0.0045 (0.004654788097247749)  --> TIME: 2025-02-26 18:51:18 -- STEP: 32032/36938 -- GLOBAL_STEP: 1361800 | > loss_text_ce: 0.023228375241160393 (0.030913234463477452) | > loss_mel_ce: 2.8398163318634033 (nan) | > loss: 0.011361288838088512 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13727647956315414) | > loader_time: 0.0045 (0.0046547662976619214)  --> TIME: 2025-02-26 18:51:27 -- STEP: 32082/36938 -- GLOBAL_STEP: 1361850 | > loss_text_ce: 0.02176055870950222 (0.030913635750526622) | > loss_mel_ce: 0.061767578125 (nan) | > loss: 0.0003314608766231686 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13726720607947182) | > loader_time: 0.0046 (0.004654671744391847)  --> TIME: 2025-02-26 18:51:37 -- STEP: 32132/36938 -- GLOBAL_STEP: 1361900 | > loss_text_ce: 0.029846495017409325 (0.030911641648034813) | > loss_mel_ce: 2.8160512447357178 (nan) | > loss: 0.011293245479464531 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.1372607957229019) | > loader_time: 0.0045 (0.004654594788764417)  --> TIME: 2025-02-26 18:51:46 -- STEP: 32182/36938 -- GLOBAL_STEP: 1361950 | > loss_text_ce: 0.0313674733042717 (0.03091055852704879) | > loss_mel_ce: 2.0617151260375977 (nan) | > loss: 0.008305883966386318 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.1372587473602426) | > loader_time: 0.0047 (0.004654561019019338)  --> TIME: 2025-02-26 18:51:55 -- STEP: 32232/36938 -- GLOBAL_STEP: 1362000 | > loss_text_ce: 0.027114765718579292 (0.030911361801995805) | > loss_mel_ce: 2.467489242553711 (nan) | > loss: 0.00989922322332859 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.1372544208925676) | > loader_time: 0.0047 (0.004654491286503807)  --> TIME: 2025-02-26 18:52:05 -- STEP: 32282/36938 -- GLOBAL_STEP: 1362050 | > loss_text_ce: 0.03234061971306801 (0.030911393458995663) | > loss_mel_ce: 2.3968429565429688 (nan) | > loss: 0.0096396179869771 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13725760115115254) | > loader_time: 0.0046 (0.004654452715226675)  --> TIME: 2025-02-26 18:52:14 -- STEP: 32332/36938 -- GLOBAL_STEP: 1362100 | > loss_text_ce: 0.036958422511816025 (0.03091262835448373) | > loss_mel_ce: 1.6336562633514404 (nan) | > loss: 0.006629423703998327 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.13725737011197994) | > loader_time: 0.0047 (0.004654421253869767)  --> TIME: 2025-02-26 18:52:23 -- STEP: 32382/36938 -- GLOBAL_STEP: 1362150 | > loss_text_ce: 0.03646005690097809 (0.03091094412247113) | > loss_mel_ce: 3.130525827407837 (nan) | > loss: 0.012567405588924885 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.13725490446669542) | > loader_time: 0.0046 (0.004654364738725863)  --> TIME: 2025-02-26 18:52:33 -- STEP: 32432/36938 -- GLOBAL_STEP: 1362200 | > loss_text_ce: 0.037792954593896866 (0.03091083960729404) | > loss_mel_ce: 3.0669004917144775 (nan) | > loss: 0.01232021301984787 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13725328617455712) | > loader_time: 0.0045 (0.004654245088120414)  --> TIME: 2025-02-26 18:52:42 -- STEP: 32482/36938 -- GLOBAL_STEP: 1362250 | > loss_text_ce: 0.025362912565469742 (0.030910879389979443) | > loss_mel_ce: 2.9642302989959717 (nan) | > loss: 0.01186346635222435 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13724940777059674) | > loader_time: 0.0045 (0.0046541185759534885)  --> TIME: 2025-02-26 18:52:51 -- STEP: 32532/36938 -- GLOBAL_STEP: 1362300 | > loss_text_ce: 0.030892107635736465 (0.030909212643120763) | > loss_mel_ce: 2.9494378566741943 (nan) | > loss: 0.011826707050204277 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.1372556223518115) | > loader_time: 0.0046 (0.004654060741873885)  --> TIME: 2025-02-26 18:53:01 -- STEP: 32582/36938 -- GLOBAL_STEP: 1362350 | > loss_text_ce: 0.037115428596735 (0.030908984706620075) | > loss_mel_ce: 0.16228027641773224 (nan) | > loss: 0.0007912528235465288 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13725776231410955) | > loader_time: 0.0046 (0.004654054446785586)  --> TIME: 2025-02-26 18:53:10 -- STEP: 32632/36938 -- GLOBAL_STEP: 1362400 | > loss_text_ce: 0.037792056798934937 (0.03091064641981304) | > loss_mel_ce: 0.6410624384880066 (nan) | > loss: 0.002693867078050971 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13725323687583876) | > loader_time: 0.0047 (0.00465399944539687)  --> TIME: 2025-02-26 18:53:19 -- STEP: 32682/36938 -- GLOBAL_STEP: 1362450 | > loss_text_ce: 0.02648916095495224 (0.030909415473098584) | > loss_mel_ce: 2.740222930908203 (nan) | > loss: 0.010979017242789268 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13724718326731625) | > loader_time: 0.0045 (0.004653875367176445)  --> TIME: 2025-02-26 18:53:28 -- STEP: 32732/36938 -- GLOBAL_STEP: 1362500 | > loss_text_ce: 0.021579205989837646 (0.030911432754657645) | > loss_mel_ce: 1.9651540517807007 (nan) | > loss: 0.007883862592279911 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13723197748735766) | > loader_time: 0.0045 (0.004653797331174472)  --> TIME: 2025-02-26 18:53:37 -- STEP: 32782/36938 -- GLOBAL_STEP: 1362550 | > loss_text_ce: 0.023363865911960602 (0.030911171856339643) | > loss_mel_ce: 1.5905863046646118 (nan) | > loss: 0.0064045642502605915 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13722914058405172) | > loader_time: 0.0045 (0.004653756232020475)  --> TIME: 2025-02-26 18:53:46 -- STEP: 32832/36938 -- GLOBAL_STEP: 1362600 | > loss_text_ce: 0.031243912875652313 (0.03091075264588479) | > loss_mel_ce: 2.957581043243408 (nan) | > loss: 0.011860416270792484 (nan) | > current_lr: 5e-06 | > step_time: 0.179 (0.13723518354473135) | > loader_time: 0.0048 (0.004653698381682803)  --> TIME: 2025-02-26 18:53:56 -- STEP: 32882/36938 -- GLOBAL_STEP: 1362650 | > loss_text_ce: 0.030351970344781876 (0.030910570887548165) | > loss_mel_ce: 3.0078125 (nan) | > loss: 0.012056208215653896 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13722913956657948) | > loader_time: 0.0045 (0.004653584550356707)  --> TIME: 2025-02-26 18:54:05 -- STEP: 32932/36938 -- GLOBAL_STEP: 1362700 | > loss_text_ce: 0.01727011799812317 (0.030911488502207072) | > loss_mel_ce: 2.1109485626220703 (nan) | > loss: 0.008445312269032001 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.1372259544107048) | > loader_time: 0.0046 (0.004653555472624268)  --> TIME: 2025-02-26 18:54:14 -- STEP: 32982/36938 -- GLOBAL_STEP: 1362750 | > loss_text_ce: 0.03184156492352486 (0.030912297059277023) | > loss_mel_ce: 2.2923226356506348 (nan) | > loss: 0.009222874417901039 (nan) | > current_lr: 5e-06 | > step_time: 0.1565 (0.13722912330261802) | > loader_time: 0.0047 (0.0046536119413176195)  --> TIME: 2025-02-26 18:54:24 -- STEP: 33032/36938 -- GLOBAL_STEP: 1362800 | > loss_text_ce: 0.03453812375664711 (0.03091072254897325) | > loss_mel_ce: 2.3041751384735107 (nan) | > loss: 0.009280608035624027 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.1372296465355522) | > loader_time: 0.0045 (0.0046536388192808375)  --> TIME: 2025-02-26 18:54:33 -- STEP: 33082/36938 -- GLOBAL_STEP: 1362850 | > loss_text_ce: 0.02636644057929516 (0.030908842447633457) | > loss_mel_ce: 2.2293310165405273 (nan) | > loss: 0.00895118061453104 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.13722519162242863) | > loader_time: 0.0047 (0.004653650301340637)  --> TIME: 2025-02-26 18:54:42 -- STEP: 33132/36938 -- GLOBAL_STEP: 1362900 | > loss_text_ce: 0.03657687455415726 (0.03090809034592333) | > loss_mel_ce: 3.176464796066284 (nan) | > loss: 0.012750166468322277 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.1372225262124814) | > loader_time: 0.0046 (0.004653650760420348)  --> TIME: 2025-02-26 18:54:51 -- STEP: 33182/36938 -- GLOBAL_STEP: 1362950 | > loss_text_ce: 0.028542974963784218 (0.030908638611688494) | > loss_mel_ce: 2.133744478225708 (nan) | > loss: 0.008580506779253483 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.1372235862020758) | > loader_time: 0.0045 (0.004653560914798409)  --> TIME: 2025-02-26 18:55:01 -- STEP: 33232/36938 -- GLOBAL_STEP: 1363000 | > loss_text_ce: 0.03300332650542259 (0.03091008945274179) | > loss_mel_ce: 3.6129848957061768 (nan) | > loss: 0.014468207955360413 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.13722888521265245) | > loader_time: 0.0048 (0.0046534826750357985)  --> TIME: 2025-02-26 18:55:11 -- STEP: 33282/36938 -- GLOBAL_STEP: 1363050 | > loss_text_ce: 0.027274277061223984 (0.030908249226564367) | > loss_mel_ce: 2.9817922115325928 (nan) | > loss: 0.01194074098020792 (nan) | > current_lr: 5e-06 | > step_time: 0.1943 (0.1372344635561748) | > loader_time: 0.005 (0.004653446405425766)  --> TIME: 2025-02-26 18:55:20 -- STEP: 33332/36938 -- GLOBAL_STEP: 1363100 | > loss_text_ce: 0.02898580953478813 (0.030907399455093444) | > loss_mel_ce: 2.4750583171844482 (nan) | > loss: 0.009936682879924774 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13723197011090285) | > loader_time: 0.0044 (0.004653442596940008)  --> TIME: 2025-02-26 18:55:29 -- STEP: 33382/36938 -- GLOBAL_STEP: 1363150 | > loss_text_ce: 0.02936381660401821 (0.030909074691555745) | > loss_mel_ce: 2.298551321029663 (nan) | > loss: 0.009237758815288544 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.1372359236221888) | > loader_time: 0.0045 (0.0046534235871264415)  --> TIME: 2025-02-26 18:55:39 -- STEP: 33432/36938 -- GLOBAL_STEP: 1363200 | > loss_text_ce: 0.027233943343162537 (0.030909756819992147) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13723559207281702) | > loader_time: 0.0045 (0.004653352824202207)  --> TIME: 2025-02-26 18:55:48 -- STEP: 33482/36938 -- GLOBAL_STEP: 1363250 | > loss_text_ce: 0.024103770032525063 (0.030910067319151018) | > loss_mel_ce: 2.377215623855591 (nan) | > loss: 0.009529045782983303 (nan) | > current_lr: 5e-06 | > step_time: 0.1721 (0.13723636737370112) | > loader_time: 0.0048 (0.0046532620281936905)  --> TIME: 2025-02-26 18:55:58 -- STEP: 33532/36938 -- GLOBAL_STEP: 1363300 | > loss_text_ce: 0.02889510802924633 (0.03090852720098692) | > loss_mel_ce: 3.0721421241760254 (nan) | > loss: 0.012305703945457935 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.13724030341166768) | > loader_time: 0.0046 (0.004653240820110991)  --> TIME: 2025-02-26 18:56:07 -- STEP: 33582/36938 -- GLOBAL_STEP: 1363350 | > loss_text_ce: 0.03540702164173126 (0.03090850052795783) | > loss_mel_ce: 1.1235064268112183 (nan) | > loss: 0.004598863422870636 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13724569820404814) | > loader_time: 0.0045 (0.004653182835386081)  --> TIME: 2025-02-26 18:56:17 -- STEP: 33632/36938 -- GLOBAL_STEP: 1363400 | > loss_text_ce: 0.031892985105514526 (0.030907470257380148) | > loss_mel_ce: 2.727722406387329 (nan) | > loss: 0.010950855910778046 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.1372482016271446) | > loader_time: 0.0046 (0.004653108782080885)  --> TIME: 2025-02-26 18:56:26 -- STEP: 33682/36938 -- GLOBAL_STEP: 1363450 | > loss_text_ce: 0.021652162075042725 (0.030904915512795325) | > loss_mel_ce: 1.9715576171875 (nan) | > loss: 0.007909563370049 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13724813318459111) | > loader_time: 0.0044 (0.004653011950537075)  --> TIME: 2025-02-26 18:56:35 -- STEP: 33732/36938 -- GLOBAL_STEP: 1363500 | > loss_text_ce: 0.029242217540740967 (0.03090461012127635) | > loss_mel_ce: 2.276175022125244 (nan) | > loss: 0.009148482233285904 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13723919868864898) | > loader_time: 0.0046 (0.0046529727489318)  --> TIME: 2025-02-26 18:56:45 -- STEP: 33782/36938 -- GLOBAL_STEP: 1363550 | > loss_text_ce: 0.032420895993709564 (0.030904600185328736) | > loss_mel_ce: 2.7339844703674316 (nan) | > loss: 0.010977800004184246 (nan) | > current_lr: 5e-06 | > step_time: 0.1537 (0.1372414310874357) | > loader_time: 0.0046 (0.004652943134618385)  --> TIME: 2025-02-26 18:56:54 -- STEP: 33832/36938 -- GLOBAL_STEP: 1363600 | > loss_text_ce: 0.01998218521475792 (0.030902518194480862) | > loss_mel_ce: 1.5951303243637085 (nan) | > loss: 0.006409177090972662 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13723815013877905) | > loader_time: 0.0045 (0.004652835539707077)  --> TIME: 2025-02-26 18:57:03 -- STEP: 33882/36938 -- GLOBAL_STEP: 1363650 | > loss_text_ce: 0.0314367413520813 (0.0309020771704272) | > loss_mel_ce: 2.3326706886291504 (nan) | > loss: 0.009381379000842571 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.137243606899121) | > loader_time: 0.0046 (0.004652762066817643)  --> TIME: 2025-02-26 18:57:12 -- STEP: 33932/36938 -- GLOBAL_STEP: 1363700 | > loss_text_ce: 0.03772644326090813 (0.030903468550178816) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.1372386023540969) | > loader_time: 0.0046 (0.004652648373733885)  --> TIME: 2025-02-26 18:57:22 -- STEP: 33982/36938 -- GLOBAL_STEP: 1363750 | > loss_text_ce: 0.06333146244287491 (0.030904386388543002) | > loss_mel_ce: 2.2093505859375 (nan) | > loss: 0.009018579497933388 (nan) | > current_lr: 5e-06 | > step_time: 0.1116 (0.13723522074948855) | > loader_time: 0.0045 (0.004652520400837525)  --> TIME: 2025-02-26 18:57:31 -- STEP: 34032/36938 -- GLOBAL_STEP: 1363800 | > loss_text_ce: 0.027652869001030922 (0.030903096324012173) | > loss_mel_ce: 2.980363130569458 (nan) | > loss: 0.011936572380363941 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.13723029936766556) | > loader_time: 0.0046 (0.004652403004302847)  --> TIME: 2025-02-26 18:57:40 -- STEP: 34082/36938 -- GLOBAL_STEP: 1363850 | > loss_text_ce: 0.036846891045570374 (0.030904661430680252) | > loss_mel_ce: 2.944077730178833 (nan) | > loss: 0.011829067021608353 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13722929741307657) | > loader_time: 0.0046 (0.004652299789201267)  --> TIME: 2025-02-26 18:57:49 -- STEP: 34132/36938 -- GLOBAL_STEP: 1363900 | > loss_text_ce: 0.029948124662041664 (0.030904424860974174) | > loss_mel_ce: 2.7956788539886475 (nan) | > loss: 0.011212806217372417 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.1372284685393269) | > loader_time: 0.0046 (0.004652272100033956)  --> TIME: 2025-02-26 18:57:59 -- STEP: 34182/36938 -- GLOBAL_STEP: 1363950 | > loss_text_ce: 0.028744852170348167 (0.030903138870563108) | > loss_mel_ce: 3.1840808391571045 (nan) | > loss: 0.01274930965155363 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.13722495578364857) | > loader_time: 0.0046 (0.0046522058644628585)  --> TIME: 2025-02-26 18:58:08 -- STEP: 34232/36938 -- GLOBAL_STEP: 1364000 | > loss_text_ce: 0.0253235325217247 (0.03090327585162388) | > loss_mel_ce: 2.739199161529541 (nan) | > loss: 0.01097032893449068 (nan) | > current_lr: 5e-06 | > step_time: 0.1614 (0.1372247624269111) | > loader_time: 0.0046 (0.004652099154992304)  --> TIME: 2025-02-26 18:58:18 -- STEP: 34282/36938 -- GLOBAL_STEP: 1364050 | > loss_text_ce: 0.02306174486875534 (0.030903086407647665) | > loss_mel_ce: 3.030304193496704 (nan) | > loss: 0.01211653184145689 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.1372289669276535) | > loader_time: 0.0046 (0.004652103933484895)  --> TIME: 2025-02-26 18:58:27 -- STEP: 34332/36938 -- GLOBAL_STEP: 1364100 | > loss_text_ce: 0.015816638246178627 (0.03090069872968647) | > loss_mel_ce: 2.511497974395752 (nan) | > loss: 0.010029027238488197 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13722872022952862) | > loader_time: 0.0045 (0.004652016711210036)  --> TIME: 2025-02-26 18:58:36 -- STEP: 34382/36938 -- GLOBAL_STEP: 1364150 | > loss_text_ce: 0.029717573896050453 (0.030900405355522267) | > loss_mel_ce: 3.0593807697296143 (nan) | > loss: 0.012258327566087246 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13723092012314828) | > loader_time: 0.0045 (0.0046519232381528)  --> TIME: 2025-02-26 18:58:46 -- STEP: 34432/36938 -- GLOBAL_STEP: 1364200 | > loss_text_ce: 0.029839985072612762 (0.03090100390283191) | > loss_mel_ce: 2.778506278991699 (nan) | > loss: 0.011144232004880905 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.1372336368578503) | > loader_time: 0.0048 (0.004651871097852294)  --> TIME: 2025-02-26 18:58:55 -- STEP: 34482/36938 -- GLOBAL_STEP: 1364250 | > loss_text_ce: 0.02903895638883114 (0.030900366705044525) | > loss_mel_ce: 2.6262662410736084 (nan) | > loss: 0.010536925867199898 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13722786772147957) | > loader_time: 0.0046 (0.00465189517287627)  --> TIME: 2025-02-26 18:59:04 -- STEP: 34532/36938 -- GLOBAL_STEP: 1364300 | > loss_text_ce: 0.03536691889166832 (0.030900447457567284) | > loss_mel_ce: 2.9571573734283447 (nan) | > loss: 0.011875097639858723 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13722774702276583) | > loader_time: 0.0046 (0.00465187090346009)  --> TIME: 2025-02-26 18:59:14 -- STEP: 34582/36938 -- GLOBAL_STEP: 1364350 | > loss_text_ce: 0.04039042070508003 (0.030900823806663136) | > loss_mel_ce: 3.0197343826293945 (nan) | > loss: 0.012143353000283241 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13722974824130327) | > loader_time: 0.0046 (0.004651780312143064)  --> TIME: 2025-02-26 18:59:23 -- STEP: 34632/36938 -- GLOBAL_STEP: 1364400 | > loss_text_ce: 0.033439744263887405 (0.03090196731863369) | > loss_mel_ce: 3.0287325382232666 (nan) | > loss: 0.012151477858424187 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13722409647698897) | > loader_time: 0.0045 (0.004651648545546068)  --> TIME: 2025-02-26 18:59:32 -- STEP: 34682/36938 -- GLOBAL_STEP: 1364450 | > loss_text_ce: 0.019338015466928482 (0.030902588894101687) | > loss_mel_ce: 0.0721588134765625 (nan) | > loss: 0.00036308265407569706 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13722165201054007) | > loader_time: 0.0046 (0.004651560921424541)  --> TIME: 2025-02-26 18:59:41 -- STEP: 34732/36938 -- GLOBAL_STEP: 1364500 | > loss_text_ce: 0.03324594348669052 (0.030903432037378657) | > loss_mel_ce: 0.20894241333007812 (nan) | > loss: 0.0009610650013200939 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.1372178482059273) | > loader_time: 0.0046 (0.004651472087446248)  --> TIME: 2025-02-26 18:59:50 -- STEP: 34782/36938 -- GLOBAL_STEP: 1364550 | > loss_text_ce: 0.03224276378750801 (0.03090194864936512) | > loss_mel_ce: 2.77579927444458 (nan) | > loss: 0.011143025010824203 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.13721417502655958) | > loader_time: 0.0046 (0.004651423766253676)  --> TIME: 2025-02-26 19:00:00 -- STEP: 34832/36938 -- GLOBAL_STEP: 1364600 | > loss_text_ce: 0.033191777765750885 (0.030903909760288815) | > loss_mel_ce: 3.1800167560577393 (nan) | > loss: 0.012750827707350254 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.13721036188882657) | > loader_time: 0.0046 (0.004651460172005284)  --> TIME: 2025-02-26 19:00:09 -- STEP: 34882/36938 -- GLOBAL_STEP: 1364650 | > loss_text_ce: 0.03244301304221153 (0.03090596685978889) | > loss_mel_ce: 1.7180441617965698 (nan) | > loss: 0.0069463783875107765 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13721236122644748) | > loader_time: 0.0046 (0.004651450378127497)  --> TIME: 2025-02-26 19:00:19 -- STEP: 34932/36938 -- GLOBAL_STEP: 1364700 | > loss_text_ce: 0.03814241662621498 (0.030904246274034004) | > loss_mel_ce: 3.305314064025879 (nan) | > loss: 0.013267685659229755 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13721917061328898) | > loader_time: 0.0046 (0.00465146669827688)  --> TIME: 2025-02-26 19:00:28 -- STEP: 34982/36938 -- GLOBAL_STEP: 1364750 | > loss_text_ce: 0.031297020614147186 (0.030905428991028612) | > loss_mel_ce: 2.833301544189453 (nan) | > loss: 0.011367455124855042 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13722164100237677) | > loader_time: 0.0046 (0.0046514295589779936)  --> TIME: 2025-02-26 19:00:38 -- STEP: 35032/36938 -- GLOBAL_STEP: 1364800 | > loss_text_ce: 0.018783198669552803 (0.03090555746689227) | > loss_mel_ce: 1.1456960439682007 (nan) | > loss: 0.004620949737727642 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13722423954923454) | > loader_time: 0.0046 (0.004651412337194911)  --> TIME: 2025-02-26 19:00:47 -- STEP: 35082/36938 -- GLOBAL_STEP: 1364850 | > loss_text_ce: 0.027455369010567665 (0.030906709842144938) | > loss_mel_ce: 2.3318777084350586 (nan) | > loss: 0.009362433105707169 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.13722305556508582) | > loader_time: 0.0046 (0.004651378955953052)  --> TIME: 2025-02-26 19:00:56 -- STEP: 35132/36938 -- GLOBAL_STEP: 1364900 | > loss_text_ce: 0.021059097722172737 (0.030906829903759873) | > loss_mel_ce: 1.5785259008407593 (nan) | > loss: 0.006347560323774815 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13722469490290645) | > loader_time: 0.0046 (0.004651365268750265)  --> TIME: 2025-02-26 19:01:06 -- STEP: 35182/36938 -- GLOBAL_STEP: 1364950 | > loss_text_ce: 0.037236522883176804 (0.030906469982613385) | > loss_mel_ce: 2.623645067214966 (nan) | > loss: 0.010559054091572762 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13722632346853272) | > loader_time: 0.0066 (0.004651393012660018)  --> TIME: 2025-02-26 19:01:15 -- STEP: 35232/36938 -- GLOBAL_STEP: 1365000 | > loss_text_ce: 0.02131834253668785 (0.030904687979588705) | > loss_mel_ce: 2.607339382171631 (nan) | > loss: 0.010431182570755482 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13722341241049238) | > loader_time: 0.0045 (0.004651420170290696)  --> TIME: 2025-02-26 19:01:24 -- STEP: 35282/36938 -- GLOBAL_STEP: 1365050 | > loss_text_ce: 0.03634192422032356 (0.030904032254293726) | > loss_mel_ce: 3.132183313369751 (nan) | > loss: 0.012573513202369213 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.1372202594198732) | > loader_time: 0.0046 (0.004651351382109208)  --> TIME: 2025-02-26 19:01:34 -- STEP: 35332/36938 -- GLOBAL_STEP: 1365100 | > loss_text_ce: 0.02641061507165432 (0.030905836149782356) | > loss_mel_ce: 2.431861639022827 (nan) | > loss: 0.009755048900842667 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.1372219764010801) | > loader_time: 0.0046 (0.004651300063371304)  --> TIME: 2025-02-26 19:01:43 -- STEP: 35382/36938 -- GLOBAL_STEP: 1365150 | > loss_text_ce: 0.029000496491789818 (0.03090754386762389) | > loss_mel_ce: 0.1044921875 (nan) | > loss: 0.0005297329043969512 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13721608267234053) | > loader_time: 0.0046 (0.0046512538289331055)  --> TIME: 2025-02-26 19:01:52 -- STEP: 35432/36938 -- GLOBAL_STEP: 1365200 | > loss_text_ce: 0.03562166541814804 (0.030904956360320864) | > loss_mel_ce: 3.03814697265625 (nan) | > loss: 0.012197495438158512 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13721686011513626) | > loader_time: 0.0046 (0.0046512227708153725)  --> TIME: 2025-02-26 19:02:02 -- STEP: 35482/36938 -- GLOBAL_STEP: 1365250 | > loss_text_ce: 0.03859391808509827 (0.030907632593284093) | > loss_mel_ce: 2.7740097045898438 (nan) | > loss: 0.01116112619638443 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13722646727609367) | > loader_time: 0.0047 (0.004651235489920261)  --> TIME: 2025-02-26 19:02:12 -- STEP: 35532/36938 -- GLOBAL_STEP: 1365300 | > loss_text_ce: 0.0321236215531826 (0.030906170542816042) | > loss_mel_ce: 3.2453975677490234 (nan) | > loss: 0.013006037101149559 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.13723462931438415) | > loader_time: 0.005 (0.004651281058781584)  --> TIME: 2025-02-26 19:02:21 -- STEP: 35582/36938 -- GLOBAL_STEP: 1365350 | > loss_text_ce: 0.029716400429606438 (0.03090689570057391) | > loss_mel_ce: 2.424494981765747 (nan) | > loss: 0.00973893515765667 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.1372352673749966) | > loader_time: 0.0045 (0.004651250897394071)  --> TIME: 2025-02-26 19:02:31 -- STEP: 35632/36938 -- GLOBAL_STEP: 1365400 | > loss_text_ce: 0.03507054224610329 (0.030906756216978807) | > loss_mel_ce: 2.2101387977600098 (nan) | > loss: 0.008909561671316624 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13724528756920895) | > loader_time: 0.0047 (0.004651270241388352)  --> TIME: 2025-02-26 19:02:40 -- STEP: 35682/36938 -- GLOBAL_STEP: 1365450 | > loss_text_ce: 0.024768874049186707 (0.03090830546360636) | > loss_mel_ce: 3.1093814373016357 (nan) | > loss: 0.012437105178833008 (nan) | > current_lr: 5e-06 | > step_time: 0.1784 (0.13724787265595043) | > loader_time: 0.0048 (0.004651209757631516)  --> TIME: 2025-02-26 19:02:50 -- STEP: 35732/36938 -- GLOBAL_STEP: 1365500 | > loss_text_ce: 0.03197454288601875 (0.030905204518328774) | > loss_mel_ce: 2.979379653930664 (nan) | > loss: 0.011949818581342697 (nan) | > current_lr: 5e-06 | > step_time: 0.1651 (0.13725672543655892) | > loader_time: 0.0047 (0.00465114613362953)  --> TIME: 2025-02-26 19:02:59 -- STEP: 35782/36938 -- GLOBAL_STEP: 1365550 | > loss_text_ce: 0.029790407046675682 (0.030905960522447144) | > loss_mel_ce: 2.206186056137085 (nan) | > loss: 0.008872922509908676 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13725897205208284) | > loader_time: 0.0045 (0.004651046573506075)  --> TIME: 2025-02-26 19:03:09 -- STEP: 35832/36938 -- GLOBAL_STEP: 1365600 | > loss_text_ce: 0.02681560628116131 (0.030904906789247604) | > loss_mel_ce: 2.7845163345336914 (nan) | > loss: 0.011156080290675163 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.1372647032151651) | > loader_time: 0.0045 (0.004650979741765701)  --> TIME: 2025-02-26 19:03:18 -- STEP: 35882/36938 -- GLOBAL_STEP: 1365650 | > loss_text_ce: 0.022938083857297897 (0.030903606831166348) | > loss_mel_ce: 0.064453125 (nan) | > loss: 0.00034679053351283073 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13726286939683863) | > loader_time: 0.0045 (0.004651017209225205)  --> TIME: 2025-02-26 19:03:28 -- STEP: 35932/36938 -- GLOBAL_STEP: 1365700 | > loss_text_ce: 0.02858213149011135 (0.03090484879890107) | > loss_mel_ce: 3.132577419281006 (nan) | > loss: 0.012544284574687481 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13725898774896503) | > loader_time: 0.0046 (0.004650980940802537)  --> TIME: 2025-02-26 19:03:37 -- STEP: 35982/36938 -- GLOBAL_STEP: 1365750 | > loss_text_ce: 0.03881797567009926 (0.030903166628806465) | > loss_mel_ce: 3.4282989501953125 (nan) | > loss: 0.0137584013864398 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13726470634224294) | > loader_time: 0.0045 (0.004650940293965185)  --> TIME: 2025-02-26 19:03:47 -- STEP: 36032/36938 -- GLOBAL_STEP: 1365800 | > loss_text_ce: 0.024422122165560722 (0.03090346613226332) | > loss_mel_ce: 0.996961236000061 (nan) | > loss: 0.004053108859807253 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13726827483723317) | > loader_time: 0.0045 (0.004650897298464761)  --> TIME: 2025-02-26 19:03:56 -- STEP: 36082/36938 -- GLOBAL_STEP: 1365850 | > loss_text_ce: 0.04011647775769234 (0.030903263398225765) | > loss_mel_ce: 2.4767847061157227 (nan) | > loss: 0.009987703524529934 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13727215909553342) | > loader_time: 0.0046 (0.004650856543192536)  --> TIME: 2025-02-26 19:04:06 -- STEP: 36132/36938 -- GLOBAL_STEP: 1365900 | > loss_text_ce: 0.030645115301012993 (0.03090187874538099) | > loss_mel_ce: 2.7495031356811523 (nan) | > loss: 0.011032335460186005 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.1372772034888208) | > loader_time: 0.0045 (0.004650803924359097)  --> TIME: 2025-02-26 19:04:15 -- STEP: 36182/36938 -- GLOBAL_STEP: 1365950 | > loss_text_ce: 0.03001546487212181 (0.03090354343785638) | > loss_mel_ce: 2.9684391021728516 (nan) | > loss: 0.011898630298674107 (nan) | > current_lr: 5e-06 | > step_time: 0.1884 (0.13728149512086632) | > loader_time: 0.0048 (0.0046507482155461405)  --> TIME: 2025-02-26 19:04:25 -- STEP: 36232/36938 -- GLOBAL_STEP: 1366000 | > loss_text_ce: 0.03234909102320671 (0.03090453893468043) | > loss_mel_ce: 2.5270955562591553 (nan) | > loss: 0.010156527161598206 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.1372876526057034) | > loader_time: 0.0046 (0.004650728990500473)  --> TIME: 2025-02-26 19:04:35 -- STEP: 36282/36938 -- GLOBAL_STEP: 1366050 | > loss_text_ce: 0.032745759934186935 (0.03090376256813705) | > loss_mel_ce: 3.377224922180176 (nan) | > loss: 0.013531630858778954 (nan) | > current_lr: 5e-06 | > step_time: 0.182 (0.1373037604839835) | > loader_time: 0.0048 (0.00465074789891411)  --> TIME: 2025-02-26 19:04:44 -- STEP: 36332/36938 -- GLOBAL_STEP: 1366100 | > loss_text_ce: 0.03295621648430824 (0.030904953005983785) | > loss_mel_ce: 2.8136236667633057 (nan) | > loss: 0.011295951902866364 (nan) | > current_lr: 5e-06 | > step_time: 0.1659 (0.1373018072090587) | > loader_time: 0.0048 (0.00465072003883975)  --> TIME: 2025-02-26 19:04:54 -- STEP: 36382/36938 -- GLOBAL_STEP: 1366150 | > loss_text_ce: 0.03497995436191559 (0.030905217952522723) | > loss_mel_ce: 2.198105573654175 (nan) | > loss: 0.008861451409757137 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13730749767534609) | > loader_time: 0.0046 (0.004650681016601249)  --> TIME: 2025-02-26 19:05:03 -- STEP: 36432/36938 -- GLOBAL_STEP: 1366200 | > loss_text_ce: 0.0304759182035923 (0.030905896890392102) | > loss_mel_ce: 2.462357759475708 (nan) | > loss: 0.009892197325825691 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.13730499569161242) | > loader_time: 0.0046 (0.004650598242191764)  --> TIME: 2025-02-26 19:05:13 -- STEP: 36482/36938 -- GLOBAL_STEP: 1366250 | > loss_text_ce: 0.027574317529797554 (0.03090649322865429) | > loss_mel_ce: 1.3126615285873413 (nan) | > loss: 0.005318396259099245 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13730592639043152) | > loader_time: 0.0045 (0.004650540162607154)  --> TIME: 2025-02-26 19:05:22 -- STEP: 36532/36938 -- GLOBAL_STEP: 1366300 | > loss_text_ce: 0.02697296254336834 (0.030906855115346993) | > loss_mel_ce: 3.0300731658935547 (nan) | > loss: 0.012131135910749435 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13730365527390004) | > loader_time: 0.0046 (0.004650489486191859)  --> TIME: 2025-02-26 19:05:31 -- STEP: 36582/36938 -- GLOBAL_STEP: 1366350 | > loss_text_ce: 0.028058741241693497 (0.030906028716148923) | > loss_mel_ce: 2.525026798248291 (nan) | > loss: 0.010131292976439 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.13730324470197022) | > loader_time: 0.0047 (0.00465052328312561)  --> TIME: 2025-02-26 19:05:40 -- STEP: 36632/36938 -- GLOBAL_STEP: 1366400 | > loss_text_ce: 0.03287907689809799 (0.030905581612457635) | > loss_mel_ce: 2.923567056655884 (nan) | > loss: 0.011731930077075958 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.1372941963587752) | > loader_time: 0.0047 (0.004650595361787063)  --> TIME: 2025-02-26 19:05:49 -- STEP: 36682/36938 -- GLOBAL_STEP: 1366450 | > loss_text_ce: 0.031165948137640953 (0.03090472188316394) | > loss_mel_ce: 2.6693780422210693 (nan) | > loss: 0.010716444812715054 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13729151664334596) | > loader_time: 0.0048 (0.00465058749377626)  --> TIME: 2025-02-26 19:05:59 -- STEP: 36732/36938 -- GLOBAL_STEP: 1366500 | > loss_text_ce: 0.02581433393061161 (0.030904184894812337) | > loss_mel_ce: 2.9419209957122803 (nan) | > loss: 0.011776727624237537 (nan) | > current_lr: 5e-06 | > step_time: 0.2123 (0.13728891525810502) | > loader_time: 0.0049 (0.004650571254633424)  --> TIME: 2025-02-26 19:06:08 -- STEP: 36782/36938 -- GLOBAL_STEP: 1366550 | > loss_text_ce: 0.03732457011938095 (0.03090543695020337) | > loss_mel_ce: 2.6587560176849365 (nan) | > loss: 0.010698733851313591 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13728885475363053) | > loader_time: 0.0046 (0.004650682047253342)  --> TIME: 2025-02-26 19:06:18 -- STEP: 36832/36938 -- GLOBAL_STEP: 1366600 | > loss_text_ce: 0.027450375258922577 (0.030905878117946636) | > loss_mel_ce: 2.826528310775757 (nan) | > loss: 0.011325312778353691 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.1372912996921909) | > loader_time: 0.0047 (0.004650671620870229)  --> TIME: 2025-02-26 19:06:27 -- STEP: 36882/36938 -- GLOBAL_STEP: 1366650 | > loss_text_ce: 0.031520381569862366 (0.030906605740281774) | > loss_mel_ce: 2.966438055038452 (nan) | > loss: 0.01189666148275137 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13729371431624735) | > loader_time: 0.0046 (0.004650641758564352)  --> TIME: 2025-02-26 19:06:36 -- STEP: 36932/36938 -- GLOBAL_STEP: 1366700 | > loss_text_ce: 0.035906970500946045 (0.03090575249772549) | > loss_mel_ce: 2.786827325820923 (nan) | > loss: 0.011201327666640282 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.13728702979860788) | > loader_time: 0.0045 (0.004650512825405937)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.02013330294354127 (-0.00046954296603063383) | > avg_loss_text_ce: 0.03014838949104052 (-0.00015853290990142446) | > avg_loss_mel_ce: 2.4532970523480158 (-0.000518555688381106) | > avg_loss: 2.4834454395983485 (-0.0006770901750798863) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_1366706.pth  > EPOCH: 38/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-26 19:07:02)   --> TIME: 2025-02-26 19:07:10 -- STEP: 44/36938 -- GLOBAL_STEP: 1366750 | > loss_text_ce: 0.0317615270614624 (0.030796452891081568) | > loss_mel_ce: 0.03708648681640625 (nan) | > loss: 0.00027320641675032675 (nan) | > current_lr: 5e-06 | > step_time: 0.1353 (0.13883028789000074) | > loader_time: 0.0048 (0.004930951378562233)  --> TIME: 2025-02-26 19:07:19 -- STEP: 94/36938 -- GLOBAL_STEP: 1366800 | > loss_text_ce: 0.03122495301067829 (0.03126166507284692) | > loss_mel_ce: 2.8655846118927 (nan) | > loss: 0.011495276354253292 (nan) | > current_lr: 5e-06 | > step_time: 0.1483 (0.13481278115130488) | > loader_time: 0.0048 (0.004881681279933199)  --> TIME: 2025-02-26 19:07:29 -- STEP: 144/36938 -- GLOBAL_STEP: 1366850 | > loss_text_ce: 0.027402089908719063 (0.030644615452426176) | > loss_mel_ce: 2.880040407180786 (nan) | > loss: 0.011537470854818821 (nan) | > current_lr: 5e-06 | > step_time: 0.1624 (0.13444068200058412) | > loader_time: 0.0047 (0.004839956760406493)  --> TIME: 2025-02-26 19:07:38 -- STEP: 194/36938 -- GLOBAL_STEP: 1366900 | > loss_text_ce: 0.038257285952568054 (0.03075121086302031) | > loss_mel_ce: 2.715235710144043 (nan) | > loss: 0.010926560498774052 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13618631829920502) | > loader_time: 0.0046 (0.004810602394575929)  --> TIME: 2025-02-26 19:07:47 -- STEP: 244/36938 -- GLOBAL_STEP: 1366950 | > loss_text_ce: 0.02505834586918354 (0.03074985911275764) | > loss_mel_ce: 3.1692726612091064 (nan) | > loss: 0.012675916776061058 (nan) | > current_lr: 5e-06 | > step_time: 0.1268 (0.13550421351292094) | > loader_time: 0.0047 (0.004782702102035775)  --> TIME: 2025-02-26 19:07:56 -- STEP: 294/36938 -- GLOBAL_STEP: 1367000 | > loss_text_ce: 0.03057241626083851 (0.030606666880780134) | > loss_mel_ce: 2.9244191646575928 (nan) | > loss: 0.011726157739758492 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.13505259500879824) | > loader_time: 0.0052 (0.00478504790740759)  --> TIME: 2025-02-26 19:08:06 -- STEP: 344/36938 -- GLOBAL_STEP: 1367050 | > loss_text_ce: 0.024259233847260475 (0.030686785193528375) | > loss_mel_ce: 2.604872465133667 (nan) | > loss: 0.01043306291103363 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13501610866812763) | > loader_time: 0.0047 (0.004788757063621697)  --> TIME: 2025-02-26 19:08:15 -- STEP: 394/36938 -- GLOBAL_STEP: 1367100 | > loss_text_ce: 0.025243334472179413 (0.03072908055746389) | > loss_mel_ce: 2.8566277027130127 (nan) | > loss: 0.011435996741056442 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13495166834235786) | > loader_time: 0.0046 (0.0047761213960986456)  --> TIME: 2025-02-26 19:08:24 -- STEP: 444/36938 -- GLOBAL_STEP: 1367150 | > loss_text_ce: 0.026791438460350037 (0.030812695037221182) | > loss_mel_ce: 2.1656293869018555 (nan) | > loss: 0.008700083009898663 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13528158321036926) | > loader_time: 0.0046 (0.004776254967526276)  --> TIME: 2025-02-26 19:08:34 -- STEP: 494/36938 -- GLOBAL_STEP: 1367200 | > loss_text_ce: 0.03615190461277962 (0.03079018749575685) | > loss_mel_ce: 0.52294921875 (nan) | > loss: 0.0022186553105711937 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13595630331078018) | > loader_time: 0.0045 (0.004766650528077657)  --> TIME: 2025-02-26 19:08:43 -- STEP: 544/36938 -- GLOBAL_STEP: 1367250 | > loss_text_ce: 0.02302255854010582 (0.03072442445280852) | > loss_mel_ce: 2.2513926029205322 (nan) | > loss: 0.009025458246469498 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1361765734413089) | > loader_time: 0.0045 (0.004753581303007465)  --> TIME: 2025-02-26 19:08:53 -- STEP: 594/36938 -- GLOBAL_STEP: 1367300 | > loss_text_ce: 0.024766726419329643 (0.03062321544038477) | > loss_mel_ce: 2.3174171447753906 (nan) | > loss: 0.009294380433857441 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13621699127685324) | > loader_time: 0.0046 (0.004758206280795013)  --> TIME: 2025-02-26 19:09:02 -- STEP: 644/36938 -- GLOBAL_STEP: 1367350 | > loss_text_ce: 0.03465026617050171 (0.030738445778336863) | > loss_mel_ce: 3.1866843700408936 (nan) | > loss: 0.012783074751496315 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13607712376932166) | > loader_time: 0.0046 (0.004756970805411015)  --> TIME: 2025-02-26 19:09:11 -- STEP: 694/36938 -- GLOBAL_STEP: 1367400 | > loss_text_ce: 0.038638755679130554 (0.030812093581446053) | > loss_mel_ce: 2.2162368297576904 (nan) | > loss: 0.008947920054197311 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.1358683772320704) | > loader_time: 0.0053 (0.004756345872576711)  --> TIME: 2025-02-26 19:09:20 -- STEP: 744/36938 -- GLOBAL_STEP: 1367450 | > loss_text_ce: 0.023202190175652504 (0.03079023921702017) | > loss_mel_ce: 2.983086109161377 (nan) | > loss: 0.011929715983569622 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13576976746641148) | > loader_time: 0.0047 (0.004747505149533671)  --> TIME: 2025-02-26 19:09:29 -- STEP: 794/36938 -- GLOBAL_STEP: 1367500 | > loss_text_ce: 0.026568260043859482 (0.030743652248581502) | > loss_mel_ce: 2.3473012447357178 (nan) | > loss: 0.009420117363333702 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13586867726419646) | > loader_time: 0.0046 (0.004748528189863305)  --> TIME: 2025-02-26 19:09:39 -- STEP: 844/36938 -- GLOBAL_STEP: 1367550 | > loss_text_ce: 0.028815867379307747 (0.030816158054670167) | > loss_mel_ce: 2.6987242698669434 (nan) | > loss: 0.010823572985827923 (nan) | > current_lr: 5e-06 | > step_time: 0.1995 (0.13617609264726296) | > loader_time: 0.0049 (0.004745914472787866)  --> TIME: 2025-02-26 19:09:48 -- STEP: 894/36938 -- GLOBAL_STEP: 1367600 | > loss_text_ce: 0.03652113676071167 (0.03083770125541811) | > loss_mel_ce: 2.488849401473999 (nan) | > loss: 0.010021312162280083 (nan) | > current_lr: 5e-06 | > step_time: 0.1846 (0.13621363597161526) | > loader_time: 0.0048 (0.004747212866542057)  --> TIME: 2025-02-26 19:09:57 -- STEP: 944/36938 -- GLOBAL_STEP: 1367650 | > loss_text_ce: 0.038477301597595215 (0.030908502714305928) | > loss_mel_ce: 2.404021978378296 (nan) | > loss: 0.009692457504570484 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.1360947381136778) | > loader_time: 0.0046 (0.004743602821382426)  --> TIME: 2025-02-26 19:10:07 -- STEP: 994/36938 -- GLOBAL_STEP: 1367700 | > loss_text_ce: 0.029844891279935837 (0.0309786247529869) | > loss_mel_ce: 2.7205398082733154 (nan) | > loss: 0.010914226062595844 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.1361709581294527) | > loader_time: 0.0046 (0.004738340675231197)  --> TIME: 2025-02-26 19:10:16 -- STEP: 1044/36938 -- GLOBAL_STEP: 1367750 | > loss_text_ce: 0.032201677560806274 (0.030997129705809486) | > loss_mel_ce: 0.160614013671875 (nan) | > loss: 0.0007651416817680001 (nan) | > current_lr: 5e-06 | > step_time: 0.1825 (0.13610644518643933) | > loader_time: 0.0049 (0.0047357591632682244)  --> TIME: 2025-02-26 19:10:25 -- STEP: 1094/36938 -- GLOBAL_STEP: 1367800 | > loss_text_ce: 0.023180615156888962 (0.03094076254478293) | > loss_mel_ce: 1.0243358612060547 (nan) | > loss: 0.004156811628490686 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13613262625455397) | > loader_time: 0.0045 (0.004735382644960171)  --> TIME: 2025-02-26 19:10:35 -- STEP: 1144/36938 -- GLOBAL_STEP: 1367850 | > loss_text_ce: 0.032229118049144745 (0.030914864181280397) | > loss_mel_ce: 2.38722562789917 (nan) | > loss: 0.009601011872291565 (nan) | > current_lr: 5e-06 | > step_time: 0.1261 (0.1361758917361704) | > loader_time: 0.0046 (0.004733902174276072)  --> TIME: 2025-02-26 19:10:44 -- STEP: 1194/36938 -- GLOBAL_STEP: 1367900 | > loss_text_ce: 0.02929014153778553 (0.03088463001558479) | > loss_mel_ce: 2.8472518920898438 (nan) | > loss: 0.011414850130677223 (nan) | > current_lr: 5e-06 | > step_time: 0.1829 (0.13614886010711494) | > loader_time: 0.0048 (0.004730500567897879)  --> TIME: 2025-02-26 19:10:53 -- STEP: 1244/36938 -- GLOBAL_STEP: 1367950 | > loss_text_ce: 0.027989983558654785 (0.030953441685143485) | > loss_mel_ce: 3.30320143699646 (nan) | > loss: 0.01321901474148035 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13616857018884715) | > loader_time: 0.0046 (0.00472662284059923)  --> TIME: 2025-02-26 19:11:02 -- STEP: 1294/36938 -- GLOBAL_STEP: 1368000 | > loss_text_ce: 0.026978353038430214 (0.030941603677937475) | > loss_mel_ce: 2.7253620624542236 (nan) | > loss: 0.01092198584228754 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1360421329967019) | > loader_time: 0.0048 (0.004724250326200838)  --> TIME: 2025-02-26 19:11:12 -- STEP: 1344/36938 -- GLOBAL_STEP: 1368050 | > loss_text_ce: 0.028024112805724144 (0.03093003078884933) | > loss_mel_ce: 2.3871254920959473 (nan) | > loss: 0.009583927690982819 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.1361978907315502) | > loader_time: 0.0046 (0.0047276646253608555)  --> TIME: 2025-02-26 19:11:22 -- STEP: 1394/36938 -- GLOBAL_STEP: 1368100 | > loss_text_ce: 0.03272401914000511 (0.03086635859354454) | > loss_mel_ce: 2.83465313911438 (nan) | > loss: 0.011378481052815914 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13637347542914616) | > loader_time: 0.0047 (0.004725754859628771)  --> TIME: 2025-02-26 19:11:31 -- STEP: 1444/36938 -- GLOBAL_STEP: 1368150 | > loss_text_ce: 0.03255533054471016 (0.030894547356473193) | > loss_mel_ce: 2.806558609008789 (nan) | > loss: 0.01126632560044527 (nan) | > current_lr: 5e-06 | > step_time: 0.1573 (0.136421540932642) | > loader_time: 0.0046 (0.004725672201436641)  --> TIME: 2025-02-26 19:11:41 -- STEP: 1494/36938 -- GLOBAL_STEP: 1368200 | > loss_text_ce: 0.03614271432161331 (0.030865970901755923) | > loss_mel_ce: 3.118910789489746 (nan) | > loss: 0.01252005435526371 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13656211282354747) | > loader_time: 0.0045 (0.004727388162370351)  --> TIME: 2025-02-26 19:11:50 -- STEP: 1544/36938 -- GLOBAL_STEP: 1368250 | > loss_text_ce: 0.03455115109682083 (0.030898861018844297) | > loss_mel_ce: 0.0074253082275390625 (nan) | > loss: 0.00016657325613778085 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.13654853241431264) | > loader_time: 0.0046 (0.004725583013475257)  --> TIME: 2025-02-26 19:12:00 -- STEP: 1594/36938 -- GLOBAL_STEP: 1368300 | > loss_text_ce: 0.028664199635386467 (0.03091438248206673) | > loss_mel_ce: 3.6634521484375 (nan) | > loss: 0.014651255682110786 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13668609742388352) | > loader_time: 0.0046 (0.004723536474641927)  --> TIME: 2025-02-26 19:12:09 -- STEP: 1644/36938 -- GLOBAL_STEP: 1368350 | > loss_text_ce: 0.029271220788359642 (0.030892354032305724) | > loss_mel_ce: 1.91519033908844 (nan) | > loss: 0.0077161178924143314 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.1368165342477115) | > loader_time: 0.0046 (0.004722206696976715)  --> TIME: 2025-02-26 19:12:19 -- STEP: 1694/36938 -- GLOBAL_STEP: 1368400 | > loss_text_ce: 0.027995875105261803 (0.030879044402258773) | > loss_mel_ce: 2.057896375656128 (nan) | > loss: 0.00827735010534525 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13700501189744158) | > loader_time: 0.0046 (0.00472292947938057)  --> TIME: 2025-02-26 19:12:28 -- STEP: 1744/36938 -- GLOBAL_STEP: 1368450 | > loss_text_ce: 0.04005580022931099 (0.030874964540264647) | > loss_mel_ce: 2.572021961212158 (nan) | > loss: 0.010365388356149197 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.13676193547905027) | > loader_time: 0.0049 (0.004720764832759126)  --> TIME: 2025-02-26 19:12:37 -- STEP: 1794/36938 -- GLOBAL_STEP: 1368500 | > loss_text_ce: 0.034670982509851456 (0.030861394016286035) | > loss_mel_ce: 2.1691646575927734 (nan) | > loss: 0.00874538067728281 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.1367055658778484) | > loader_time: 0.0047 (0.004722533284488201)  --> TIME: 2025-02-26 19:12:46 -- STEP: 1844/36938 -- GLOBAL_STEP: 1368550 | > loss_text_ce: 0.03539029881358147 (0.030866556018570725) | > loss_mel_ce: 2.976787567138672 (nan) | > loss: 0.011953087523579597 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.1368118639623265) | > loader_time: 0.0047 (0.004723036883968597)  --> TIME: 2025-02-26 19:12:55 -- STEP: 1894/36938 -- GLOBAL_STEP: 1368600 | > loss_text_ce: 0.02798372693359852 (0.03087221896433229) | > loss_mel_ce: 2.9686899185180664 (nan) | > loss: 0.01189156249165535 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13672480575638046) | > loader_time: 0.0046 (0.004723709009768723)  --> TIME: 2025-02-26 19:13:05 -- STEP: 1944/36938 -- GLOBAL_STEP: 1368650 | > loss_text_ce: 0.031208859756588936 (0.030847359234045562) | > loss_mel_ce: 3.133845090866089 (nan) | > loss: 0.012559738010168076 (nan) | > current_lr: 5e-06 | > step_time: 0.1689 (0.1367720935815645) | > loader_time: 0.0048 (0.004722256596686913)  --> TIME: 2025-02-26 19:13:14 -- STEP: 1994/36938 -- GLOBAL_STEP: 1368700 | > loss_text_ce: 0.032811906188726425 (0.030863879837932713) | > loss_mel_ce: 2.553670883178711 (nan) | > loss: 0.010263821110129356 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13675386843494797) | > loader_time: 0.0046 (0.004721873622002792)  --> TIME: 2025-02-26 19:13:24 -- STEP: 2044/36938 -- GLOBAL_STEP: 1368750 | > loss_text_ce: 0.03094499371945858 (0.03083752256493702) | > loss_mel_ce: 3.039577007293701 (nan) | > loss: 0.012184612452983856 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.1367723280436371) | > loader_time: 0.0046 (0.004722643971676466)  --> TIME: 2025-02-26 19:13:33 -- STEP: 2094/36938 -- GLOBAL_STEP: 1368800 | > loss_text_ce: 0.03067612089216709 (0.030806215475077645) | > loss_mel_ce: 2.8622164726257324 (nan) | > loss: 0.011479733511805534 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13670638103995178) | > loader_time: 0.0044 (0.0047207597105869735)  --> TIME: 2025-02-26 19:13:42 -- STEP: 2144/36938 -- GLOBAL_STEP: 1368850 | > loss_text_ce: 0.032181888818740845 (0.03078309669890858) | > loss_mel_ce: 2.3445892333984375 (nan) | > loss: 0.009431632235646248 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13681969057712973) | > loader_time: 0.0045 (0.004718866922072512)  --> TIME: 2025-02-26 19:13:52 -- STEP: 2194/36938 -- GLOBAL_STEP: 1368900 | > loss_text_ce: 0.026715561747550964 (0.03074057353363324) | > loss_mel_ce: 1.4137582778930664 (nan) | > loss: 0.005716165993362665 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13674056997273107) | > loader_time: 0.0046 (0.004715694116263796)  --> TIME: 2025-02-26 19:14:01 -- STEP: 2244/36938 -- GLOBAL_STEP: 1368950 | > loss_text_ce: 0.018247205764055252 (0.030776672943457033) | > loss_mel_ce: 2.6459262371063232 (nan) | > loss: 0.010572116822004318 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1367547852045285) | > loader_time: 0.0046 (0.004716265010323754)  --> TIME: 2025-02-26 19:14:10 -- STEP: 2294/36938 -- GLOBAL_STEP: 1369000 | > loss_text_ce: 0.033067792654037476 (0.030774367445431904) | > loss_mel_ce: 2.9409584999084473 (nan) | > loss: 0.011801691725850105 (nan) | > current_lr: 5e-06 | > step_time: 0.1523 (0.13657254952394945) | > loader_time: 0.0066 (0.004714309597597396)  --> TIME: 2025-02-26 19:14:19 -- STEP: 2344/36938 -- GLOBAL_STEP: 1369050 | > loss_text_ce: 0.050206296145915985 (0.03078861839500329) | > loss_mel_ce: 0.0072021484375 (nan) | > loss: 0.00022781129518989474 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13646943038234097) | > loader_time: 0.0045 (0.0047115100732028695)  --> TIME: 2025-02-26 19:14:28 -- STEP: 2394/36938 -- GLOBAL_STEP: 1369100 | > loss_text_ce: 0.02358226291835308 (0.030829106559455643) | > loss_mel_ce: 2.3771872520446777 (nan) | > loss: 0.009526863694190979 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13632830272840243) | > loader_time: 0.0045 (0.004708911143647899)  --> TIME: 2025-02-26 19:14:37 -- STEP: 2444/36938 -- GLOBAL_STEP: 1369150 | > loss_text_ce: 0.033702924847602844 (0.030845319709714665) | > loss_mel_ce: 2.8476524353027344 (nan) | > loss: 0.011433950625360012 (nan) | > current_lr: 5e-06 | > step_time: 0.1813 (0.13641464759011865) | > loader_time: 0.0048 (0.004707340913201317)  --> TIME: 2025-02-26 19:14:46 -- STEP: 2494/36938 -- GLOBAL_STEP: 1369200 | > loss_text_ce: 0.05161258950829506 (0.030847755920234546) | > loss_mel_ce: 2.3990070819854736 (nan) | > loss: 0.009724682196974754 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.1364159078337996) | > loader_time: 0.0045 (0.004704907691851557)  --> TIME: 2025-02-26 19:14:56 -- STEP: 2544/36938 -- GLOBAL_STEP: 1369250 | > loss_text_ce: 0.03111143223941326 (0.030828604905180773) | > loss_mel_ce: 2.930271863937378 (nan) | > loss: 0.011751522310078144 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.13642083080309705) | > loader_time: 0.0048 (0.0047028614852413445)  --> TIME: 2025-02-26 19:15:05 -- STEP: 2594/36938 -- GLOBAL_STEP: 1369300 | > loss_text_ce: 0.02757108211517334 (0.0307977048981053) | > loss_mel_ce: 2.8483870029449463 (nan) | > loss: 0.011412532068789005 (nan) | > current_lr: 5e-06 | > step_time: 0.187 (0.13651534411754584) | > loader_time: 0.0048 (0.004702968972411263)  --> TIME: 2025-02-26 19:15:15 -- STEP: 2644/36938 -- GLOBAL_STEP: 1369350 | > loss_text_ce: 0.03324118256568909 (0.030792424150597467) | > loss_mel_ce: 2.1780824661254883 (nan) | > loss: 0.008775094524025917 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13653187627330637) | > loader_time: 0.0045 (0.004701840985741092)  --> TIME: 2025-02-26 19:15:24 -- STEP: 2694/36938 -- GLOBAL_STEP: 1369400 | > loss_text_ce: 0.03739745914936066 (0.030794131785628415) | > loss_mel_ce: 3.440094232559204 (nan) | > loss: 0.013799570500850677 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.13643087381420949) | > loader_time: 0.0046 (0.004699953060815668)  --> TIME: 2025-02-26 19:15:33 -- STEP: 2744/36938 -- GLOBAL_STEP: 1369450 | > loss_text_ce: 0.04341672733426094 (0.030804127364137016) | > loss_mel_ce: 2.738055944442749 (nan) | > loss: 0.01103759091347456 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13648862576345638) | > loader_time: 0.0044 (0.004701435392174012)  --> TIME: 2025-02-26 19:15:43 -- STEP: 2794/36938 -- GLOBAL_STEP: 1369500 | > loss_text_ce: 0.016576049849390984 (0.03081108418139299) | > loss_mel_ce: 0.08780884742736816 (nan) | > loss: 0.00041422582580707967 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13661375554357133) | > loader_time: 0.0045 (0.004701163227761232)  --> TIME: 2025-02-26 19:15:52 -- STEP: 2844/36938 -- GLOBAL_STEP: 1369550 | > loss_text_ce: 0.024169592186808586 (0.030773082439906885) | > loss_mel_ce: 2.606217861175537 (nan) | > loss: 0.010438046418130398 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13660726885922983) | > loader_time: 0.0045 (0.0047003874128210285)  --> TIME: 2025-02-26 19:16:02 -- STEP: 2894/36938 -- GLOBAL_STEP: 1369600 | > loss_text_ce: 0.035337839275598526 (0.030775094885281247) | > loss_mel_ce: 3.013960123062134 (nan) | > loss: 0.012100390158593655 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13663049681728734) | > loader_time: 0.0046 (0.004699025552851133)  --> TIME: 2025-02-26 19:16:11 -- STEP: 2944/36938 -- GLOBAL_STEP: 1369650 | > loss_text_ce: 0.032174598425626755 (0.030750245838767707) | > loss_mel_ce: 2.7148995399475098 (nan) | > loss: 0.010901088826358318 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13657026252020935) | > loader_time: 0.0045 (0.004700318221812667)  --> TIME: 2025-02-26 19:16:20 -- STEP: 2994/36938 -- GLOBAL_STEP: 1369700 | > loss_text_ce: 0.021908242255449295 (0.030733820264344024) | > loss_mel_ce: 2.309995174407959 (nan) | > loss: 0.00925358571112156 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13656428486168462) | > loader_time: 0.0046 (0.004699902528114298)  --> TIME: 2025-02-26 19:16:30 -- STEP: 3044/36938 -- GLOBAL_STEP: 1369750 | > loss_text_ce: 0.03476306423544884 (0.030744283066012464) | > loss_mel_ce: 2.259183883666992 (nan) | > loss: 0.009102964773774147 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.1365918233579463) | > loader_time: 0.0046 (0.004700637913251528)  --> TIME: 2025-02-26 19:16:39 -- STEP: 3094/36938 -- GLOBAL_STEP: 1369800 | > loss_text_ce: 0.030701545998454094 (0.030742207696178536) | > loss_mel_ce: 2.6683719158172607 (nan) | > loss: 0.010710610076785088 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.1366078499909281) | > loader_time: 0.0048 (0.004701157732016063)  --> TIME: 2025-02-26 19:16:48 -- STEP: 3144/36938 -- GLOBAL_STEP: 1369850 | > loss_text_ce: 0.029599661007523537 (0.030747628149886925) | > loss_mel_ce: 2.1913278102874756 (nan) | > loss: 0.008813205175101757 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13661376680733242) | > loader_time: 0.0046 (0.004700886839218734)  --> TIME: 2025-02-26 19:16:58 -- STEP: 3194/36938 -- GLOBAL_STEP: 1369900 | > loss_text_ce: 0.02625538781285286 (0.030737780462880487) | > loss_mel_ce: 2.401425361633301 (nan) | > loss: 0.009633653797209263 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13668598155342299) | > loader_time: 0.0045 (0.004703429242410591)  --> TIME: 2025-02-26 19:17:07 -- STEP: 3244/36938 -- GLOBAL_STEP: 1369950 | > loss_text_ce: 0.017029259353876114 (0.03073284138739339) | > loss_mel_ce: 2.4032390117645264 (nan) | > loss: 0.009604239836335182 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13667976371163015) | > loader_time: 0.0046 (0.004704841923625495)  --> TIME: 2025-02-26 19:17:17 -- STEP: 3294/36938 -- GLOBAL_STEP: 1370000 | > loss_text_ce: 0.02692163921892643 (0.030737979855822682) | > loss_mel_ce: 1.1894587278366089 (nan) | > loss: 0.004826906602829695 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.1367036932225503) | > loader_time: 0.0048 (0.004705921620691488) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1370000.pth  --> TIME: 2025-02-26 19:17:32 -- STEP: 3344/36938 -- GLOBAL_STEP: 1370050 | > loss_text_ce: 0.03433595225214958 (0.030727899524452318) | > loss_mel_ce: 2.6720528602600098 (nan) | > loss: 0.010739638470113277 (nan) | > current_lr: 5e-06 | > step_time: 0.1821 (0.13673021719216094) | > loader_time: 0.0049 (0.004707033197845578)  --> TIME: 2025-02-26 19:17:42 -- STEP: 3394/36938 -- GLOBAL_STEP: 1370100 | > loss_text_ce: 0.02293528988957405 (0.030735792129524996) | > loss_mel_ce: 2.5893948078155518 (nan) | > loss: 0.010366390459239483 (nan) | > current_lr: 5e-06 | > step_time: 0.1446 (0.13672530103165273) | > loader_time: 0.0048 (0.004706643930096877)  --> TIME: 2025-02-26 19:17:51 -- STEP: 3444/36938 -- GLOBAL_STEP: 1370150 | > loss_text_ce: 0.0334504134953022 (0.030725401903542664) | > loss_mel_ce: 2.630391836166382 (nan) | > loss: 0.010570802725851536 (nan) | > current_lr: 5e-06 | > step_time: 0.1629 (0.13672197515817677) | > loader_time: 0.0048 (0.004706938438437685)  --> TIME: 2025-02-26 19:18:00 -- STEP: 3494/36938 -- GLOBAL_STEP: 1370200 | > loss_text_ce: 0.03508961200714111 (0.030737309101979665) | > loss_mel_ce: 2.62798810005188 (nan) | > loss: 0.010567769408226013 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13670709935610212) | > loader_time: 0.0045 (0.004706717383609072)  --> TIME: 2025-02-26 19:18:09 -- STEP: 3544/36938 -- GLOBAL_STEP: 1370250 | > loss_text_ce: 0.031108476221561432 (0.030749505470374403) | > loss_mel_ce: 0.1339874267578125 (nan) | > loss: 0.000655142473988235 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13666507917925078) | > loader_time: 0.0046 (0.004707373695502706)  --> TIME: 2025-02-26 19:18:19 -- STEP: 3594/36938 -- GLOBAL_STEP: 1370300 | > loss_text_ce: 0.03189100697636604 (0.030741570497726693) | > loss_mel_ce: 2.331122398376465 (nan) | > loss: 0.009377038106322289 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13664732870952048) | > loader_time: 0.0046 (0.004706904167722984)  --> TIME: 2025-02-26 19:18:28 -- STEP: 3644/36938 -- GLOBAL_STEP: 1370350 | > loss_text_ce: 0.03192349523305893 (0.030747851377310134) | > loss_mel_ce: 2.291402578353882 (nan) | > loss: 0.009219548664987087 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.1366864796134951) | > loader_time: 0.0046 (0.004706217284783579)  --> TIME: 2025-02-26 19:18:38 -- STEP: 3694/36938 -- GLOBAL_STEP: 1370400 | > loss_text_ce: 0.027525339275598526 (0.030733478324145934) | > loss_mel_ce: 2.507958173751831 (nan) | > loss: 0.010061443783342838 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.1367401766273604) | > loader_time: 0.0046 (0.00470525429709641)  --> TIME: 2025-02-26 19:18:47 -- STEP: 3744/36938 -- GLOBAL_STEP: 1370450 | > loss_text_ce: 0.035133007913827896 (0.030719104254295908) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13678528094648237) | > loader_time: 0.0045 (0.004704227495906704)  --> TIME: 2025-02-26 19:18:57 -- STEP: 3794/36938 -- GLOBAL_STEP: 1370500 | > loss_text_ce: 0.02961437962949276 (0.030707313570659853) | > loss_mel_ce: 2.675004243850708 (nan) | > loss: 0.010732614435255527 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13683761378998566) | > loader_time: 0.0046 (0.004704347207286077)  --> TIME: 2025-02-26 19:19:06 -- STEP: 3844/36938 -- GLOBAL_STEP: 1370550 | > loss_text_ce: 0.03884217143058777 (0.030721914338608904) | > loss_mel_ce: 2.5152804851531982 (nan) | > loss: 0.010135408490896225 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.1368462213864554) | > loader_time: 0.0046 (0.004704724737558359)  --> TIME: 2025-02-26 19:19:15 -- STEP: 3894/36938 -- GLOBAL_STEP: 1370600 | > loss_text_ce: 0.02725822851061821 (0.03073987591835662) | > loss_mel_ce: 2.6991512775421143 (nan) | > loss: 0.010819085873663425 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13679647335469758) | > loader_time: 0.0045 (0.004704620449985064)  --> TIME: 2025-02-26 19:19:24 -- STEP: 3944/36938 -- GLOBAL_STEP: 1370650 | > loss_text_ce: 0.030576085671782494 (0.03074058031750197) | > loss_mel_ce: 2.7092092037200928 (nan) | > loss: 0.01087216380983591 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13676518987690406) | > loader_time: 0.0046 (0.004704961670339717)  --> TIME: 2025-02-26 19:19:34 -- STEP: 3994/36938 -- GLOBAL_STEP: 1370700 | > loss_text_ce: 0.02959904633462429 (0.03074292796041348) | > loss_mel_ce: 2.97619366645813 (nan) | > loss: 0.011927749030292034 (nan) | > current_lr: 5e-06 | > step_time: 0.1599 (0.13678070940133213) | > loader_time: 0.0046 (0.00470533440216936)  --> TIME: 2025-02-26 19:19:44 -- STEP: 4044/36938 -- GLOBAL_STEP: 1370750 | > loss_text_ce: 0.03130507469177246 (0.03074428693664976) | > loss_mel_ce: 3.1075122356414795 (nan) | > loss: 0.012455624528229237 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.1368833160070231) | > loader_time: 0.0046 (0.004706266489981666)  --> TIME: 2025-02-26 19:19:53 -- STEP: 4094/36938 -- GLOBAL_STEP: 1370800 | > loss_text_ce: 0.03637290745973587 (0.030752452134514618) | > loss_mel_ce: 3.0569396018981934 (nan) | > loss: 0.012275050394237041 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13686553373182997) | > loader_time: 0.0046 (0.00470599187078509)  --> TIME: 2025-02-26 19:20:02 -- STEP: 4144/36938 -- GLOBAL_STEP: 1370850 | > loss_text_ce: 0.020616095513105392 (0.03073365113951453) | > loss_mel_ce: 2.152008533477783 (nan) | > loss: 0.008621526882052422 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.1368828909507584) | > loader_time: 0.0046 (0.00470537735458507)  --> TIME: 2025-02-26 19:20:12 -- STEP: 4194/36938 -- GLOBAL_STEP: 1370900 | > loss_text_ce: 0.038831982761621475 (0.030735286453656203) | > loss_mel_ce: 2.4482421875 (nan) | > loss: 0.009869342669844627 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13691813427320654) | > loader_time: 0.0045 (0.004705014658588201)  --> TIME: 2025-02-26 19:20:21 -- STEP: 4244/36938 -- GLOBAL_STEP: 1370950 | > loss_text_ce: 0.03356913477182388 (0.030722549587618084) | > loss_mel_ce: 2.7839150428771973 (nan) | > loss: 0.011180493049323559 (nan) | > current_lr: 5e-06 | > step_time: 0.1719 (0.1369470447217592) | > loader_time: 0.0047 (0.004705628002734821)  --> TIME: 2025-02-26 19:20:31 -- STEP: 4294/36938 -- GLOBAL_STEP: 1371000 | > loss_text_ce: 0.03302869573235512 (0.03072658548210533) | > loss_mel_ce: 2.5981383323669434 (nan) | > loss: 0.010441139340400696 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.13697625244835276) | > loader_time: 0.0048 (0.004705438238551691)  --> TIME: 2025-02-26 19:20:40 -- STEP: 4344/36938 -- GLOBAL_STEP: 1371050 | > loss_text_ce: 0.02980298362672329 (0.030724563985989252) | > loss_mel_ce: 3.0659048557281494 (nan) | > loss: 0.012284555472433567 (nan) | > current_lr: 5e-06 | > step_time: 0.1814 (0.1370042732078062) | > loader_time: 0.0048 (0.00470556112942775)  --> TIME: 2025-02-26 19:20:50 -- STEP: 4394/36938 -- GLOBAL_STEP: 1371100 | > loss_text_ce: 0.03021307662129402 (0.03073782051528109) | > loss_mel_ce: 2.081484079360962 (nan) | > loss: 0.00837975088506937 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.13708891665658737) | > loader_time: 0.0046 (0.004704997167296449)  --> TIME: 2025-02-26 19:21:00 -- STEP: 4444/36938 -- GLOBAL_STEP: 1371150 | > loss_text_ce: 0.03745608776807785 (0.030736142767150657) | > loss_mel_ce: 1.5232106447219849 (nan) | > loss: 0.006193121895194054 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13714472856959334) | > loader_time: 0.0046 (0.004704661620403607)  --> TIME: 2025-02-26 19:21:09 -- STEP: 4494/36938 -- GLOBAL_STEP: 1371200 | > loss_text_ce: 0.025154756382107735 (0.030741585375269453) | > loss_mel_ce: 0.02587890625 (nan) | > loss: 0.00020251453679520637 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13715742915059656) | > loader_time: 0.0046 (0.004704533018002688)  --> TIME: 2025-02-26 19:21:19 -- STEP: 4544/36938 -- GLOBAL_STEP: 1371250 | > loss_text_ce: 0.030958110466599464 (0.03074355414165938) | > loss_mel_ce: 2.8787848949432373 (nan) | > loss: 0.011546600610017776 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13720046466504976) | > loader_time: 0.0046 (0.004707153886556634)  --> TIME: 2025-02-26 19:21:28 -- STEP: 4594/36938 -- GLOBAL_STEP: 1371300 | > loss_text_ce: 0.02445048838853836 (0.03074171577434716) | > loss_mel_ce: 3.097022771835327 (nan) | > loss: 0.012386799789965153 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13725843406730348) | > loader_time: 0.0046 (0.004707515525153588)  --> TIME: 2025-02-26 19:21:38 -- STEP: 4644/36938 -- GLOBAL_STEP: 1371350 | > loss_text_ce: 0.03457111120223999 (0.030756415948063186) | > loss_mel_ce: 2.2087409496307373 (nan) | > loss: 0.008902032859623432 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13729178315087867) | > loader_time: 0.0046 (0.004708907041541477)  --> TIME: 2025-02-26 19:21:47 -- STEP: 4694/36938 -- GLOBAL_STEP: 1371400 | > loss_text_ce: 0.032470252364873886 (0.030762920428732507) | > loss_mel_ce: 2.642515182495117 (nan) | > loss: 0.010615021921694279 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13725112527799296) | > loader_time: 0.0046 (0.004709233260429014)  --> TIME: 2025-02-26 19:21:57 -- STEP: 4744/36938 -- GLOBAL_STEP: 1371450 | > loss_text_ce: 0.03780541568994522 (0.030754249231987215) | > loss_mel_ce: 2.488440752029419 (nan) | > loss: 0.010024786926805973 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.13733490231065246) | > loader_time: 0.0046 (0.0047095252128716986)  --> TIME: 2025-02-26 19:22:06 -- STEP: 4794/36938 -- GLOBAL_STEP: 1371500 | > loss_text_ce: 0.03185921907424927 (0.030751928154800302) | > loss_mel_ce: 2.9573333263397217 (nan) | > loss: 0.011861875653266907 (nan) | > current_lr: 5e-06 | > step_time: 0.1502 (0.1373452717530809) | > loader_time: 0.0047 (0.004709487862521341)  --> TIME: 2025-02-26 19:22:16 -- STEP: 4844/36938 -- GLOBAL_STEP: 1371550 | > loss_text_ce: 0.02964668720960617 (0.030767963362456255) | > loss_mel_ce: 2.50301456451416 (nan) | > loss: 0.010050243698060513 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.1373570131723982) | > loader_time: 0.0047 (0.004709653279488005)  --> TIME: 2025-02-26 19:22:25 -- STEP: 4894/36938 -- GLOBAL_STEP: 1371600 | > loss_text_ce: 0.03225335106253624 (0.030777747782794466) | > loss_mel_ce: 3.0660457611083984 (nan) | > loss: 0.012294838204979897 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.13737283519788493) | > loader_time: 0.0047 (0.00470945077084697)  --> TIME: 2025-02-26 19:22:35 -- STEP: 4944/36938 -- GLOBAL_STEP: 1371650 | > loss_text_ce: 0.0387125238776207 (0.030780270383738487) | > loss_mel_ce: 2.0450236797332764 (nan) | > loss: 0.008268794976174831 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13736234991102894) | > loader_time: 0.0046 (0.004709107686786597)  --> TIME: 2025-02-26 19:22:44 -- STEP: 4994/36938 -- GLOBAL_STEP: 1371700 | > loss_text_ce: 0.04027261584997177 (0.030794293325600276) | > loss_mel_ce: 2.7648608684539795 (nan) | > loss: 0.011131483130156994 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1373820228962404) | > loader_time: 0.0045 (0.0047092664513914565)  --> TIME: 2025-02-26 19:22:54 -- STEP: 5044/36938 -- GLOBAL_STEP: 1371750 | > loss_text_ce: 0.04157290235161781 (0.03078904894086799) | > loss_mel_ce: 2.5559580326080322 (nan) | > loss: 0.01030766312032938 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13739461099033184) | > loader_time: 0.0046 (0.004708914507201137)  --> TIME: 2025-02-26 19:23:03 -- STEP: 5094/36938 -- GLOBAL_STEP: 1371800 | > loss_text_ce: 0.020062291994690895 (0.030772104548911255) | > loss_mel_ce: 2.0284485816955566 (nan) | > loss: 0.00812901183962822 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13737439117012237) | > loader_time: 0.0046 (0.004708451198792059)  --> TIME: 2025-02-26 19:23:12 -- STEP: 5144/36938 -- GLOBAL_STEP: 1371850 | > loss_text_ce: 0.03200141713023186 (0.030764696931663442) | > loss_mel_ce: 2.7238237857818604 (nan) | > loss: 0.010935815051198006 (nan) | > current_lr: 5e-06 | > step_time: 0.1898 (0.137414073962631) | > loader_time: 0.0048 (0.004708255570190694)  --> TIME: 2025-02-26 19:23:22 -- STEP: 5194/36938 -- GLOBAL_STEP: 1371900 | > loss_text_ce: 0.02477961778640747 (0.030760733180843984) | > loss_mel_ce: 2.793381690979004 (nan) | > loss: 0.011183179914951324 (nan) | > current_lr: 5e-06 | > step_time: 0.1503 (0.13738912038725948) | > loader_time: 0.0047 (0.004708665859529412)  --> TIME: 2025-02-26 19:23:31 -- STEP: 5244/36938 -- GLOBAL_STEP: 1371950 | > loss_text_ce: 0.04374217242002487 (0.030762591372612752) | > loss_mel_ce: 2.925663709640503 (nan) | > loss: 0.011783357709646225 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1373356893533606) | > loader_time: 0.0053 (0.00470871101527793)  --> TIME: 2025-02-26 19:23:40 -- STEP: 5294/36938 -- GLOBAL_STEP: 1372000 | > loss_text_ce: 0.02457253634929657 (0.030764792578926076) | > loss_mel_ce: 2.6129260063171387 (nan) | > loss: 0.010466264560818672 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.1373817930592647) | > loader_time: 0.0047 (0.004708368372007463)  --> TIME: 2025-02-26 19:23:50 -- STEP: 5344/36938 -- GLOBAL_STEP: 1372050 | > loss_text_ce: 0.031008878722786903 (0.03076802027825835) | > loss_mel_ce: 2.108570098876953 (nan) | > loss: 0.008490393869578838 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13737278114892726) | > loader_time: 0.0047 (0.00470775437212277)  --> TIME: 2025-02-26 19:23:59 -- STEP: 5394/36938 -- GLOBAL_STEP: 1372100 | > loss_text_ce: 0.030445558950304985 (0.030781721036700262) | > loss_mel_ce: 1.6165226697921753 (nan) | > loss: 0.006535588763654232 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13734643194999854) | > loader_time: 0.0046 (0.004707125279034433)  --> TIME: 2025-02-26 19:24:08 -- STEP: 5444/36938 -- GLOBAL_STEP: 1372150 | > loss_text_ce: 0.03489377722144127 (0.030775875753321406) | > loss_mel_ce: 2.4545018672943115 (nan) | > loss: 0.009878554381430149 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13734550694172004) | > loader_time: 0.0046 (0.0047073227180268)  --> TIME: 2025-02-26 19:24:17 -- STEP: 5494/36938 -- GLOBAL_STEP: 1372200 | > loss_text_ce: 0.03487434983253479 (0.030771043696781485) | > loss_mel_ce: 0.4305664002895355 (nan) | > loss: 0.0018469871720299125 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13733181968184838) | > loader_time: 0.0045 (0.004707247029663138)  --> TIME: 2025-02-26 19:24:27 -- STEP: 5544/36938 -- GLOBAL_STEP: 1372250 | > loss_text_ce: 0.022869838401675224 (0.030769297550902748) | > loss_mel_ce: 2.791705369949341 (nan) | > loss: 0.011168950237333775 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13730489657210718) | > loader_time: 0.0046 (0.004706419262535138)  --> TIME: 2025-02-26 19:24:36 -- STEP: 5594/36938 -- GLOBAL_STEP: 1372300 | > loss_text_ce: 0.031056324020028114 (0.030774053605889366) | > loss_mel_ce: 0.12506103515625 (nan) | > loss: 0.0006195133901201189 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.13735104242221835) | > loader_time: 0.0046 (0.00470581982107644)  --> TIME: 2025-02-26 19:24:46 -- STEP: 5644/36938 -- GLOBAL_STEP: 1372350 | > loss_text_ce: 0.026294920593500137 (0.030768589910501278) | > loss_mel_ce: 2.8983051776885986 (nan) | > loss: 0.011605557054281235 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13736925937874256) | > loader_time: 0.0046 (0.004706683858415923)  --> TIME: 2025-02-26 19:24:55 -- STEP: 5694/36938 -- GLOBAL_STEP: 1372400 | > loss_text_ce: 0.025180283933877945 (0.030779723534841402) | > loss_mel_ce: 2.611492156982422 (nan) | > loss: 0.010462986305356026 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13736688977674621) | > loader_time: 0.0047 (0.004706744147804199)  --> TIME: 2025-02-26 19:25:04 -- STEP: 5744/36938 -- GLOBAL_STEP: 1372450 | > loss_text_ce: 0.03933658450841904 (0.030783387022462614) | > loss_mel_ce: 2.5032172203063965 (nan) | > loss: 0.01008949987590313 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.1373561734442589) | > loader_time: 0.0045 (0.004706596182581455)  --> TIME: 2025-02-26 19:25:14 -- STEP: 5794/36938 -- GLOBAL_STEP: 1372500 | > loss_text_ce: 0.02919434942305088 (0.030789180998720966) | > loss_mel_ce: 0.11376953125 (nan) | > loss: 0.0005673170671798289 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13731505432991356) | > loader_time: 0.0046 (0.004706223627432173)  --> TIME: 2025-02-26 19:25:23 -- STEP: 5844/36938 -- GLOBAL_STEP: 1372550 | > loss_text_ce: 0.034820765256881714 (0.030789817361005573) | > loss_mel_ce: 1.2634490728378296 (nan) | > loss: 0.0051518650725483894 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.137315724746239) | > loader_time: 0.0048 (0.004706008151979351)  --> TIME: 2025-02-26 19:25:32 -- STEP: 5894/36938 -- GLOBAL_STEP: 1372600 | > loss_text_ce: 0.021714016795158386 (0.03078540961150981) | > loss_mel_ce: 2.5890188217163086 (nan) | > loss: 0.010360050946474075 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13729839370256322) | > loader_time: 0.0046 (0.004705638896824361)  --> TIME: 2025-02-26 19:25:42 -- STEP: 5944/36938 -- GLOBAL_STEP: 1372650 | > loss_text_ce: 0.031414829194545746 (0.03078569512933989) | > loss_mel_ce: 2.4582133293151855 (nan) | > loss: 0.009879477322101593 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.1373051125380741) | > loader_time: 0.0047 (0.004705704959212226)  --> TIME: 2025-02-26 19:25:51 -- STEP: 5994/36938 -- GLOBAL_STEP: 1372700 | > loss_text_ce: 0.031227823346853256 (0.03079368227944872) | > loss_mel_ce: 2.5385959148406982 (nan) | > loss: 0.010197713971138 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.13732104835249273) | > loader_time: 0.0047 (0.004705543911055335)  --> TIME: 2025-02-26 19:26:00 -- STEP: 6044/36938 -- GLOBAL_STEP: 1372750 | > loss_text_ce: 0.027948835864663124 (0.03078974490756604) | > loss_mel_ce: 0.8802903890609741 (nan) | > loss: 0.0036041243001818657 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13730117585941426) | > loader_time: 0.0046 (0.004705833568232192)  --> TIME: 2025-02-26 19:26:10 -- STEP: 6094/36938 -- GLOBAL_STEP: 1372800 | > loss_text_ce: 0.030123412609100342 (0.030782340750988287) | > loss_mel_ce: 3.1789493560791016 (nan) | > loss: 0.012734416872262955 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.1372999326886209) | > loader_time: 0.005 (0.0047056376836242956)  --> TIME: 2025-02-26 19:26:19 -- STEP: 6144/36938 -- GLOBAL_STEP: 1372850 | > loss_text_ce: 0.03691611438989639 (0.030802444128312345) | > loss_mel_ce: 2.154874324798584 (nan) | > loss: 0.008697581477463245 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13727796392049604) | > loader_time: 0.0046 (0.004705135012045517)  --> TIME: 2025-02-26 19:26:28 -- STEP: 6194/36938 -- GLOBAL_STEP: 1372900 | > loss_text_ce: 0.022932911291718483 (0.030799693890784414) | > loss_mel_ce: 1.9554654359817505 (nan) | > loss: 0.007850787602365017 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13725937240386235) | > loader_time: 0.0045 (0.004704886880812449)  --> TIME: 2025-02-26 19:26:38 -- STEP: 6244/36938 -- GLOBAL_STEP: 1372950 | > loss_text_ce: 0.028989724814891815 (0.030800624181379844) | > loss_mel_ce: 2.960906744003296 (nan) | > loss: 0.011864669620990753 (nan) | > current_lr: 5e-06 | > step_time: 0.1874 (0.13727946001928926) | > loader_time: 0.0049 (0.0047054792504368816)  --> TIME: 2025-02-26 19:26:47 -- STEP: 6294/36938 -- GLOBAL_STEP: 1373000 | > loss_text_ce: 0.025316189974546432 (0.03080517115100418) | > loss_mel_ce: 1.2692646980285645 (nan) | > loss: 0.005137226078659296 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13725107402546952) | > loader_time: 0.0045 (0.004705107018877162)  --> TIME: 2025-02-26 19:26:56 -- STEP: 6344/36938 -- GLOBAL_STEP: 1373050 | > loss_text_ce: 0.031862612813711166 (0.03080419554817872) | > loss_mel_ce: 3.013603687286377 (nan) | > loss: 0.012085183523595333 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.13722675944787702) | > loader_time: 0.0046 (0.004704518095692368)  --> TIME: 2025-02-26 19:27:05 -- STEP: 6394/36938 -- GLOBAL_STEP: 1373100 | > loss_text_ce: 0.031884610652923584 (0.030794823619143492) | > loss_mel_ce: 2.961005687713623 (nan) | > loss: 0.011876549571752548 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13721532571080863) | > loader_time: 0.0046 (0.0047039793997136255)  --> TIME: 2025-02-26 19:27:15 -- STEP: 6444/36938 -- GLOBAL_STEP: 1373150 | > loss_text_ce: 0.03231565281748772 (0.030806177167983843) | > loss_mel_ce: 3.6616320610046387 (nan) | > loss: 0.014658523723483086 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.1372202116461885) | > loader_time: 0.0046 (0.004703841062897428)  --> TIME: 2025-02-26 19:27:24 -- STEP: 6494/36938 -- GLOBAL_STEP: 1373200 | > loss_text_ce: 0.03912724182009697 (0.030807957926281496) | > loss_mel_ce: 2.901379346847534 (nan) | > loss: 0.011668678373098373 (nan) | > current_lr: 5e-06 | > step_time: 0.1765 (0.13719835213818937) | > loader_time: 0.005 (0.004703963650972779)  --> TIME: 2025-02-26 19:27:33 -- STEP: 6544/36938 -- GLOBAL_STEP: 1373250 | > loss_text_ce: 0.0487065352499485 (0.03081325785920689) | > loss_mel_ce: 2.281492233276367 (nan) | > loss: 0.009246820583939552 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13719378444559735) | > loader_time: 0.0048 (0.004704456712622536)  --> TIME: 2025-02-26 19:27:42 -- STEP: 6594/36938 -- GLOBAL_STEP: 1373300 | > loss_text_ce: 0.02403225563466549 (0.030809855945623025) | > loss_mel_ce: 2.610558032989502 (nan) | > loss: 0.010454724542796612 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13715311086716592) | > loader_time: 0.0045 (0.004704182893534981)  --> TIME: 2025-02-26 19:27:52 -- STEP: 6644/36938 -- GLOBAL_STEP: 1373350 | > loss_text_ce: 0.028229419142007828 (0.030809851698880182) | > loss_mel_ce: 2.942574977874756 (nan) | > loss: 0.011788907460868359 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13716557645281013) | > loader_time: 0.0045 (0.004703815947951238)  --> TIME: 2025-02-26 19:28:01 -- STEP: 6694/36938 -- GLOBAL_STEP: 1373400 | > loss_text_ce: 0.03342128545045853 (0.030804287290803294) | > loss_mel_ce: 2.980226755142212 (nan) | > loss: 0.011958921328186989 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.1371355882359152) | > loader_time: 0.0046 (0.004703052655810723)  --> TIME: 2025-02-26 19:28:10 -- STEP: 6744/36938 -- GLOBAL_STEP: 1373450 | > loss_text_ce: 0.04146591201424599 (0.03080685876424768) | > loss_mel_ce: 2.921895742416382 (nan) | > loss: 0.011759372428059578 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13712070276021704) | > loader_time: 0.0047 (0.004703230245540455)  --> TIME: 2025-02-26 19:28:19 -- STEP: 6794/36938 -- GLOBAL_STEP: 1373500 | > loss_text_ce: 0.03432595729827881 (0.030810107053431617) | > loss_mel_ce: 1.3436373472213745 (nan) | > loss: 0.00546810869127512 (nan) | > current_lr: 5e-06 | > step_time: 0.1083 (0.1371324510829818) | > loader_time: 0.0045 (0.004703501620501839)  --> TIME: 2025-02-26 19:28:29 -- STEP: 6844/36938 -- GLOBAL_STEP: 1373550 | > loss_text_ce: 0.022775446996092796 (0.03080814322516592) | > loss_mel_ce: 2.1722676753997803 (nan) | > loss: 0.008710488677024841 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.1371249616807977) | > loader_time: 0.0046 (0.004703062379102436)  --> TIME: 2025-02-26 19:28:37 -- STEP: 6894/36938 -- GLOBAL_STEP: 1373600 | > loss_text_ce: 0.02065892703831196 (0.030804846222880683) | > loss_mel_ce: 0.09865379333496094 (nan) | > loss: 0.0004734632093459368 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13707877453908357) | > loader_time: 0.0045 (0.004702307398436161)  --> TIME: 2025-02-26 19:28:47 -- STEP: 6944/36938 -- GLOBAL_STEP: 1373650 | > loss_text_ce: 0.03504200652241707 (0.030798757481517502) | > loss_mel_ce: 2.207763671875 (nan) | > loss: 0.008900023065507412 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13706715796400357) | > loader_time: 0.0045 (0.004702167061891449)  --> TIME: 2025-02-26 19:28:56 -- STEP: 6994/36938 -- GLOBAL_STEP: 1373700 | > loss_text_ce: 0.032621704041957855 (0.030783136833452747) | > loss_mel_ce: 2.33642578125 (nan) | > loss: 0.009400982409715652 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13708200034053494) | > loader_time: 0.0045 (0.004701867797629054)  --> TIME: 2025-02-26 19:29:05 -- STEP: 7044/36938 -- GLOBAL_STEP: 1373750 | > loss_text_ce: 0.03842667117714882 (0.030788346121409217) | > loss_mel_ce: 3.081780433654785 (nan) | > loss: 0.012381774373352528 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1370840347131367) | > loader_time: 0.0047 (0.004701586219142091)  --> TIME: 2025-02-26 19:29:15 -- STEP: 7094/36938 -- GLOBAL_STEP: 1373800 | > loss_text_ce: 0.03845413029193878 (0.03078705034147048) | > loss_mel_ce: 2.414062976837158 (nan) | > loss: 0.009732211008667946 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13707745011369749) | > loader_time: 0.0046 (0.00470117427679666)  --> TIME: 2025-02-26 19:29:24 -- STEP: 7144/36938 -- GLOBAL_STEP: 1373850 | > loss_text_ce: 0.017275437712669373 (0.030786202226067743) | > loss_mel_ce: 0.7918539047241211 (nan) | > loss: 0.003210830967873335 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1370610511983767) | > loader_time: 0.0046 (0.004700503550878722)  --> TIME: 2025-02-26 19:29:34 -- STEP: 7194/36938 -- GLOBAL_STEP: 1373900 | > loss_text_ce: 0.03272773697972298 (0.030793612231266126) | > loss_mel_ce: 2.586348295211792 (nan) | > loss: 0.010393159464001656 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1371060029513183) | > loader_time: 0.0045 (0.0047002062056506946)  --> TIME: 2025-02-26 19:29:43 -- STEP: 7244/36938 -- GLOBAL_STEP: 1373950 | > loss_text_ce: 0.021310433745384216 (0.030792944296815988) | > loss_mel_ce: 2.726991653442383 (nan) | > loss: 0.010905960574746132 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1371224457094648) | > loader_time: 0.0046 (0.004700387728800131)  --> TIME: 2025-02-26 19:29:53 -- STEP: 7294/36938 -- GLOBAL_STEP: 1374000 | > loss_text_ce: 0.0331861786544323 (0.03079939319991946) | > loss_mel_ce: 2.8461408615112305 (nan) | > loss: 0.011425902135670185 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13714774138050834) | > loader_time: 0.0046 (0.00470027601290706)  --> TIME: 2025-02-26 19:30:02 -- STEP: 7344/36938 -- GLOBAL_STEP: 1374050 | > loss_text_ce: 0.03223767504096031 (0.030793536063598182) | > loss_mel_ce: 2.9660565853118896 (nan) | > loss: 0.011897994205355644 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13714837359187423) | > loader_time: 0.0046 (0.004699953403607976)  --> TIME: 2025-02-26 19:30:11 -- STEP: 7394/36938 -- GLOBAL_STEP: 1374100 | > loss_text_ce: 0.03874870017170906 (0.03080285364957937) | > loss_mel_ce: 3.0214362144470215 (nan) | > loss: 0.012143591418862343 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13715133758567652) | > loader_time: 0.0046 (0.004699752141825106)  --> TIME: 2025-02-26 19:30:21 -- STEP: 7444/36938 -- GLOBAL_STEP: 1374150 | > loss_text_ce: 0.03357931599020958 (0.030801824315799895) | > loss_mel_ce: 2.3021228313446045 (nan) | > loss: 0.009268660098314285 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13716705405282986) | > loader_time: 0.0046 (0.004699643391040905)  --> TIME: 2025-02-26 19:30:30 -- STEP: 7494/36938 -- GLOBAL_STEP: 1374200 | > loss_text_ce: 0.03309067338705063 (0.030794096981382288) | > loss_mel_ce: 2.687870740890503 (nan) | > loss: 0.010797466151416302 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13716578687195) | > loader_time: 0.0047 (0.004699524479100718)  --> TIME: 2025-02-26 19:30:40 -- STEP: 7544/36938 -- GLOBAL_STEP: 1374250 | > loss_text_ce: 0.02639966830611229 (0.03079873648514598) | > loss_mel_ce: 2.4033799171447754 (nan) | > loss: 0.009641982614994049 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13715839218486575) | > loader_time: 0.0045 (0.004699365489682638)  --> TIME: 2025-02-26 19:30:49 -- STEP: 7594/36938 -- GLOBAL_STEP: 1374300 | > loss_text_ce: 0.033284224569797516 (0.030805137911544118) | > loss_mel_ce: 4.3800225257873535 (nan) | > loss: 0.017513122409582138 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.1371099426773373) | > loader_time: 0.0045 (0.0046990201571818655)  --> TIME: 2025-02-26 19:30:58 -- STEP: 7644/36938 -- GLOBAL_STEP: 1374350 | > loss_text_ce: 0.025337129831314087 (0.030802363538329963) | > loss_mel_ce: 3.1828291416168213 (nan) | > loss: 0.012730819173157215 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13712932401025357) | > loader_time: 0.0045 (0.004699047637322656)  --> TIME: 2025-02-26 19:31:08 -- STEP: 7694/36938 -- GLOBAL_STEP: 1374400 | > loss_text_ce: 0.03558504208922386 (0.030799700966681526) | > loss_mel_ce: 3.2568359375 (nan) | > loss: 0.013065163046121597 (nan) | > current_lr: 5e-06 | > step_time: 0.1692 (0.1371385644000945) | > loader_time: 0.0048 (0.004698728133028835)  --> TIME: 2025-02-26 19:31:17 -- STEP: 7744/36938 -- GLOBAL_STEP: 1374450 | > loss_text_ce: 0.02933328039944172 (0.03080017437695618) | > loss_mel_ce: 2.4492363929748535 (nan) | > loss: 0.009835594333708286 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13717628405852794) | > loader_time: 0.0045 (0.0046984407712113)  --> TIME: 2025-02-26 19:31:27 -- STEP: 7794/36938 -- GLOBAL_STEP: 1374500 | > loss_text_ce: 0.038905564695596695 (0.030796563983023786) | > loss_mel_ce: 2.8499701023101807 (nan) | > loss: 0.011463792994618416 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13716101046859083) | > loader_time: 0.0046 (0.004698159696507054)  --> TIME: 2025-02-26 19:31:36 -- STEP: 7844/36938 -- GLOBAL_STEP: 1374550 | > loss_text_ce: 0.03519025444984436 (0.03080278197609739) | > loss_mel_ce: 2.0143513679504395 (nan) | > loss: 0.00813310220837593 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13715742073273107) | > loader_time: 0.0045 (0.004698284483272063)  --> TIME: 2025-02-26 19:31:45 -- STEP: 7894/36938 -- GLOBAL_STEP: 1374600 | > loss_text_ce: 0.02719113603234291 (0.030807523538689086) | > loss_mel_ce: 2.5718636512756348 (nan) | > loss: 0.010313710197806358 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.13713256204946797) | > loader_time: 0.0048 (0.004697997750648847)  --> TIME: 2025-02-26 19:31:55 -- STEP: 7944/36938 -- GLOBAL_STEP: 1374650 | > loss_text_ce: 0.03260590881109238 (0.030811569470797797) | > loss_mel_ce: 2.428600549697876 (nan) | > loss: 0.009766693226993084 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13715851246890506) | > loader_time: 0.0047 (0.004697451028458923)  --> TIME: 2025-02-26 19:32:04 -- STEP: 7994/36938 -- GLOBAL_STEP: 1374700 | > loss_text_ce: 0.029533328488469124 (0.030805145554833557) | > loss_mel_ce: 2.6955933570861816 (nan) | > loss: 0.010813995264470577 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.13714168478555652) | > loader_time: 0.0047 (0.004697819366913207)  --> TIME: 2025-02-26 19:32:13 -- STEP: 8044/36938 -- GLOBAL_STEP: 1374750 | > loss_text_ce: 0.026707004755735397 (0.03081076146436421) | > loss_mel_ce: 2.6165616512298584 (nan) | > loss: 0.010489162057638168 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13715153917751802) | > loader_time: 0.0047 (0.004697804217431058)  --> TIME: 2025-02-26 19:32:22 -- STEP: 8094/36938 -- GLOBAL_STEP: 1374800 | > loss_text_ce: 0.031123992055654526 (0.030812240455392586) | > loss_mel_ce: 3.050184488296509 (nan) | > loss: 0.012227415107190609 (nan) | > current_lr: 5e-06 | > step_time: 0.1573 (0.1371346647308765) | > loader_time: 0.0047 (0.004697646569227764)  --> TIME: 2025-02-26 19:32:32 -- STEP: 8144/36938 -- GLOBAL_STEP: 1374850 | > loss_text_ce: 0.03372093662619591 (0.03081556045988205) | > loss_mel_ce: 0.8940836787223816 (nan) | > loss: 0.0036817644722759724 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.13713900636596196) | > loader_time: 0.0048 (0.004697412428078363)  --> TIME: 2025-02-26 19:32:41 -- STEP: 8194/36938 -- GLOBAL_STEP: 1374900 | > loss_text_ce: 0.02430812083184719 (0.030815267563275385) | > loss_mel_ce: 2.5418176651000977 (nan) | > loss: 0.010183040052652359 (nan) | > current_lr: 5e-06 | > step_time: 0.172 (0.13713179625096175) | > loader_time: 0.0049 (0.004697684808739945)  --> TIME: 2025-02-26 19:32:50 -- STEP: 8244/36938 -- GLOBAL_STEP: 1374950 | > loss_text_ce: 0.031566157937049866 (0.030814898803591555) | > loss_mel_ce: 2.955488443374634 (nan) | > loss: 0.011853392235934734 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.13712724819164882) | > loader_time: 0.0047 (0.00469781214724695)  --> TIME: 2025-02-26 19:33:00 -- STEP: 8294/36938 -- GLOBAL_STEP: 1375000 | > loss_text_ce: 0.029193712398409843 (0.030822852309339586) | > loss_mel_ce: 2.9862635135650635 (nan) | > loss: 0.0119661008939147 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.13711929097071004) | > loader_time: 0.0047 (0.004697777835725465)  --> TIME: 2025-02-26 19:33:09 -- STEP: 8344/36938 -- GLOBAL_STEP: 1375050 | > loss_text_ce: 0.023910634219646454 (0.030818756111903777) | > loss_mel_ce: 2.6268203258514404 (nan) | > loss: 0.010518773458898067 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1371255504719253) | > loader_time: 0.0046 (0.004697558349525274)  --> TIME: 2025-02-26 19:33:18 -- STEP: 8394/36938 -- GLOBAL_STEP: 1375100 | > loss_text_ce: 0.029266467317938805 (0.03081710348675755) | > loss_mel_ce: 2.175539970397949 (nan) | > loss: 0.008749231696128845 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13711856426668972) | > loader_time: 0.0046 (0.004697632727124242)  --> TIME: 2025-02-26 19:33:28 -- STEP: 8444/36938 -- GLOBAL_STEP: 1375150 | > loss_text_ce: 0.027833232656121254 (0.03081427616619142) | > loss_mel_ce: 0.7504196166992188 (nan) | > loss: 0.0030883050058037043 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13711038086438107) | > loader_time: 0.0053 (0.004697759475599606)  --> TIME: 2025-02-26 19:33:37 -- STEP: 8494/36938 -- GLOBAL_STEP: 1375200 | > loss_text_ce: 0.030447615310549736 (0.030809378501861998) | > loss_mel_ce: 2.60182785987854 (nan) | > loss: 0.01044553890824318 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.13711200513810587) | > loader_time: 0.0046 (0.004697769227689219)  --> TIME: 2025-02-26 19:33:46 -- STEP: 8544/36938 -- GLOBAL_STEP: 1375250 | > loss_text_ce: 0.02604570984840393 (0.030809532442219095) | > loss_mel_ce: 2.160388231277466 (nan) | > loss: 0.008676325902342796 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13712092299251577) | > loader_time: 0.0047 (0.004698126336162028)  --> TIME: 2025-02-26 19:33:56 -- STEP: 8594/36938 -- GLOBAL_STEP: 1375300 | > loss_text_ce: 0.029292460530996323 (0.030802551638321986) | > loss_mel_ce: 3.0003409385681152 (nan) | > loss: 0.012022354640066624 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1371366918877671) | > loader_time: 0.0046 (0.004698541737557143)  --> TIME: 2025-02-26 19:34:05 -- STEP: 8644/36938 -- GLOBAL_STEP: 1375350 | > loss_text_ce: 0.03288640081882477 (0.030808792647994857) | > loss_mel_ce: 3.037719249725342 (nan) | > loss: 0.01218494400382042 (nan) | > current_lr: 5e-06 | > step_time: 0.1574 (0.13709244591281844) | > loader_time: 0.0047 (0.004698461870637461)  --> TIME: 2025-02-26 19:34:15 -- STEP: 8694/36938 -- GLOBAL_STEP: 1375400 | > loss_text_ce: 0.0359160378575325 (0.030808271561910607) | > loss_mel_ce: 3.11328125 (nan) | > loss: 0.012496815994381905 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.1371204997030438) | > loader_time: 0.0045 (0.004698340169363876)  --> TIME: 2025-02-26 19:34:24 -- STEP: 8744/36938 -- GLOBAL_STEP: 1375450 | > loss_text_ce: 0.027174603193998337 (0.030804812116804043) | > loss_mel_ce: 0.9551566243171692 (nan) | > loss: 0.0038981400430202484 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.1371030872116809) | > loader_time: 0.0046 (0.004698037910330568)  --> TIME: 2025-02-26 19:34:33 -- STEP: 8794/36938 -- GLOBAL_STEP: 1375500 | > loss_text_ce: 0.03376064449548721 (0.03080846280772862) | > loss_mel_ce: 0.6560450792312622 (nan) | > loss: 0.0027373244520276785 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13710447837473064) | > loader_time: 0.0046 (0.004697627877007046)  --> TIME: 2025-02-26 19:34:42 -- STEP: 8844/36938 -- GLOBAL_STEP: 1375550 | > loss_text_ce: 0.029945174232125282 (0.03081162288363337) | > loss_mel_ce: 0.9628819227218628 (nan) | > loss: 0.0039397901855409145 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13709701266778357) | > loader_time: 0.0046 (0.0046972763155553945)  --> TIME: 2025-02-26 19:34:52 -- STEP: 8894/36938 -- GLOBAL_STEP: 1375600 | > loss_text_ce: 0.034124668687582016 (0.030811547387470075) | > loss_mel_ce: 1.471810221672058 (nan) | > loss: 0.005975932348519564 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13709792550397756) | > loader_time: 0.0045 (0.004697204118689625)  --> TIME: 2025-02-26 19:35:01 -- STEP: 8944/36938 -- GLOBAL_STEP: 1375650 | > loss_text_ce: 0.03561781719326973 (0.0308056874201743) | > loss_mel_ce: 2.8696112632751465 (nan) | > loss: 0.011528687551617622 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13706122098860496) | > loader_time: 0.0047 (0.004697408520471663)  --> TIME: 2025-02-26 19:35:10 -- STEP: 8994/36938 -- GLOBAL_STEP: 1375700 | > loss_text_ce: 0.0363345667719841 (0.03081092192719539) | > loss_mel_ce: 2.396909713745117 (nan) | > loss: 0.009655731730163097 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.1370488883443271) | > loader_time: 0.0047 (0.004697272638122022)  --> TIME: 2025-02-26 19:35:19 -- STEP: 9044/36938 -- GLOBAL_STEP: 1375750 | > loss_text_ce: 0.035250045359134674 (0.030813477124214794) | > loss_mel_ce: 2.412966728210449 (nan) | > loss: 0.009715146385133266 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13705003388104142) | > loader_time: 0.0046 (0.004697198680099919)  --> TIME: 2025-02-26 19:35:28 -- STEP: 9094/36938 -- GLOBAL_STEP: 1375800 | > loss_text_ce: 0.03748748078942299 (0.030818403483031935) | > loss_mel_ce: 2.8324644565582275 (nan) | > loss: 0.01138869859278202 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1370141716718624) | > loader_time: 0.0046 (0.004696829832331667)  --> TIME: 2025-02-26 19:35:37 -- STEP: 9144/36938 -- GLOBAL_STEP: 1375850 | > loss_text_ce: 0.027779215946793556 (0.03082487761667968) | > loss_mel_ce: 2.592451810836792 (nan) | > loss: 0.010397742502391338 (nan) | > current_lr: 5e-06 | > step_time: 0.1798 (0.13698636695468858) | > loader_time: 0.0048 (0.00469643959640311)  --> TIME: 2025-02-26 19:35:46 -- STEP: 9194/36938 -- GLOBAL_STEP: 1375900 | > loss_text_ce: 0.035511039197444916 (0.030816470820246855) | > loss_mel_ce: 3.250910520553589 (nan) | > loss: 0.013041355647146702 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13696373945634135) | > loader_time: 0.0046 (0.0046965195247342445)  --> TIME: 2025-02-26 19:35:56 -- STEP: 9244/36938 -- GLOBAL_STEP: 1375950 | > loss_text_ce: 0.034073349088430405 (0.030822502735080155) | > loss_mel_ce: 3.307586431503296 (nan) | > loss: 0.0132605554535985 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13697091997159838) | > loader_time: 0.0046 (0.004696608466639883)  --> TIME: 2025-02-26 19:36:05 -- STEP: 9294/36938 -- GLOBAL_STEP: 1376000 | > loss_text_ce: 0.029944362118840218 (0.03082214024063013) | > loss_mel_ce: 1.1997586488723755 (nan) | > loss: 0.004879774525761604 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13699751993597598) | > loader_time: 0.0066 (0.004696743781268554)  --> TIME: 2025-02-26 19:36:15 -- STEP: 9344/36938 -- GLOBAL_STEP: 1376050 | > loss_text_ce: 0.02949388138949871 (0.03081365871872699) | > loss_mel_ce: 2.5131113529205322 (nan) | > loss: 0.01008970383554697 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.1369847114363765) | > loader_time: 0.0047 (0.00469653201225688)  --> TIME: 2025-02-26 19:36:24 -- STEP: 9394/36938 -- GLOBAL_STEP: 1376100 | > loss_text_ce: 0.027918638661503792 (0.03081172221418163) | > loss_mel_ce: 3.1980323791503906 (nan) | > loss: 0.012801393866539001 (nan) | > current_lr: 5e-06 | > step_time: 0.1829 (0.13696845858965617) | > loader_time: 0.0048 (0.004696164355522701)  --> TIME: 2025-02-26 19:36:33 -- STEP: 9444/36938 -- GLOBAL_STEP: 1376150 | > loss_text_ce: 0.03459260240197182 (0.030809800901179223) | > loss_mel_ce: 2.857123851776123 (nan) | > loss: 0.011475065723061562 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.1369722575668766) | > loader_time: 0.0046 (0.004696061453643589)  --> TIME: 2025-02-26 19:36:42 -- STEP: 9494/36938 -- GLOBAL_STEP: 1376200 | > loss_text_ce: 0.034525591880083084 (0.030815444794376166) | > loss_mel_ce: 2.835345983505249 (nan) | > loss: 0.011388380080461502 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1369642034088262) | > loader_time: 0.0049 (0.004695804866961963)  --> TIME: 2025-02-26 19:36:52 -- STEP: 9544/36938 -- GLOBAL_STEP: 1376250 | > loss_text_ce: 0.034836266189813614 (0.030813405340891067) | > loss_mel_ce: 2.2150590419769287 (nan) | > loss: 0.008928156457841396 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13697801826385972) | > loader_time: 0.0045 (0.004695651742059071)  --> TIME: 2025-02-26 19:37:01 -- STEP: 9594/36938 -- GLOBAL_STEP: 1376300 | > loss_text_ce: 0.025456760078668594 (0.030806192455606327) | > loss_mel_ce: 2.5864105224609375 (nan) | > loss: 0.010364552959799767 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.136977572577284) | > loader_time: 0.0045 (0.004695457892291689)  --> TIME: 2025-02-26 19:37:11 -- STEP: 9644/36938 -- GLOBAL_STEP: 1376350 | > loss_text_ce: 0.03028949722647667 (0.030809974401211193) | > loss_mel_ce: 2.9532089233398438 (nan) | > loss: 0.011839279904961586 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.13697660110245047) | > loader_time: 0.0053 (0.0046952851379328305)  --> TIME: 2025-02-26 19:37:20 -- STEP: 9694/36938 -- GLOBAL_STEP: 1376400 | > loss_text_ce: 0.03819480165839195 (0.030808780384900394) | > loss_mel_ce: 1.6689966917037964 (nan) | > loss: 0.006774569861590862 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13697802678614504) | > loader_time: 0.0045 (0.0046951359563300035)  --> TIME: 2025-02-26 19:37:29 -- STEP: 9744/36938 -- GLOBAL_STEP: 1376450 | > loss_text_ce: 0.03254522383213043 (0.030809312246239794) | > loss_mel_ce: 2.607072353363037 (nan) | > loss: 0.01047467440366745 (nan) | > current_lr: 5e-06 | > step_time: 0.1439 (0.13696279069668785) | > loader_time: 0.0047 (0.00469491932975446)  --> TIME: 2025-02-26 19:37:38 -- STEP: 9794/36938 -- GLOBAL_STEP: 1376500 | > loss_text_ce: 0.03773460164666176 (0.030802668895582315) | > loss_mel_ce: 2.534335136413574 (nan) | > loss: 0.010206625796854496 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13696067669646297) | > loader_time: 0.0047 (0.004694687850333661)  --> TIME: 2025-02-26 19:37:47 -- STEP: 9844/36938 -- GLOBAL_STEP: 1376550 | > loss_text_ce: 0.026736371219158173 (0.030802032219080275) | > loss_mel_ce: 3.084665298461914 (nan) | > loss: 0.01234683208167553 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13693637963670524) | > loader_time: 0.0053 (0.004694591712680425)  --> TIME: 2025-02-26 19:37:57 -- STEP: 9894/36938 -- GLOBAL_STEP: 1376600 | > loss_text_ce: 0.033097926527261734 (0.0308089429344692) | > loss_mel_ce: 2.7325167655944824 (nan) | > loss: 0.010974662378430367 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13694290124890535) | > loader_time: 0.0046 (0.004694794630170594)  --> TIME: 2025-02-26 19:38:06 -- STEP: 9944/36938 -- GLOBAL_STEP: 1376650 | > loss_text_ce: 0.038104213774204254 (0.030808542630682403) | > loss_mel_ce: 2.0791914463043213 (nan) | > loss: 0.008401967585086823 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13693261400692378) | > loader_time: 0.0046 (0.00469480705472223)  --> TIME: 2025-02-26 19:38:15 -- STEP: 9994/36938 -- GLOBAL_STEP: 1376700 | > loss_text_ce: 0.04878873750567436 (0.03080873242442414) | > loss_mel_ce: 2.4452407360076904 (nan) | > loss: 0.009896943345665932 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13691818726069988) | > loader_time: 0.0045 (0.004694426706225176)  --> TIME: 2025-02-26 19:38:24 -- STEP: 10044/36938 -- GLOBAL_STEP: 1376750 | > loss_text_ce: 0.020847320556640625 (0.030798931006744225) | > loss_mel_ce: 1.9872084856033325 (nan) | > loss: 0.007968475110828876 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1369080922604561) | > loader_time: 0.0046 (0.004694574242899138)  --> TIME: 2025-02-26 19:38:34 -- STEP: 10094/36938 -- GLOBAL_STEP: 1376800 | > loss_text_ce: 0.02387448027729988 (0.030798511234192725) | > loss_mel_ce: 2.2901124954223633 (nan) | > loss: 0.009182488545775414 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13690255846579524) | > loader_time: 0.0045 (0.004694857950703775)  --> TIME: 2025-02-26 19:38:43 -- STEP: 10144/36938 -- GLOBAL_STEP: 1376850 | > loss_text_ce: 0.029138967394828796 (0.03079785433416446) | > loss_mel_ce: 3.2079038619995117 (nan) | > loss: 0.012845409102737904 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13688630403874472) | > loader_time: 0.0046 (0.004694720571537516)  --> TIME: 2025-02-26 19:38:52 -- STEP: 10194/36938 -- GLOBAL_STEP: 1376900 | > loss_text_ce: 0.03425295278429985 (0.030804568961224698) | > loss_mel_ce: 0.12338979542255402 (nan) | > loss: 0.0006255665211938322 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.1368835489025349) | > loader_time: 0.0046 (0.004694773562870861)  --> TIME: 2025-02-26 19:39:02 -- STEP: 10244/36938 -- GLOBAL_STEP: 1376950 | > loss_text_ce: 0.03196822479367256 (0.03080551633100385) | > loss_mel_ce: 3.0861663818359375 (nan) | > loss: 0.012373549863696098 (nan) | > current_lr: 5e-06 | > step_time: 0.1689 (0.13689924966818875) | > loader_time: 0.0047 (0.004695206775836594)  --> TIME: 2025-02-26 19:39:11 -- STEP: 10294/36938 -- GLOBAL_STEP: 1377000 | > loss_text_ce: 0.03303801640868187 (0.03080983516974228) | > loss_mel_ce: 2.953693151473999 (nan) | > loss: 0.01185210794210434 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.13688859843225784) | > loader_time: 0.0047 (0.004695315302657429)  --> TIME: 2025-02-26 19:39:20 -- STEP: 10344/36938 -- GLOBAL_STEP: 1377050 | > loss_text_ce: 0.03617905452847481 (0.03081107130989055) | > loss_mel_ce: 2.766690492630005 (nan) | > loss: 0.011122498661279678 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.1368961800970229) | > loader_time: 0.0047 (0.004695236982528513)  --> TIME: 2025-02-26 19:39:29 -- STEP: 10394/36938 -- GLOBAL_STEP: 1377100 | > loss_text_ce: 0.02987750992178917 (0.03081531865555956) | > loss_mel_ce: 3.449805498123169 (nan) | > loss: 0.013808266259729862 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.13688736132114798) | > loader_time: 0.0048 (0.004695052157371409)  --> TIME: 2025-02-26 19:39:39 -- STEP: 10444/36938 -- GLOBAL_STEP: 1377150 | > loss_text_ce: 0.0351535938680172 (0.030809009486929936) | > loss_mel_ce: 3.2785537242889404 (nan) | > loss: 0.013149633072316647 (nan) | > current_lr: 5e-06 | > step_time: 0.1528 (0.13688843634246414) | > loader_time: 0.0047 (0.004695083596431555)  --> TIME: 2025-02-26 19:39:48 -- STEP: 10494/36938 -- GLOBAL_STEP: 1377200 | > loss_text_ce: 0.032945722341537476 (0.030808073414547492) | > loss_mel_ce: 2.783482074737549 (nan) | > loss: 0.011176301166415215 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.13688591143642864) | > loader_time: 0.0046 (0.004695195594650935)  --> TIME: 2025-02-26 19:39:58 -- STEP: 10544/36938 -- GLOBAL_STEP: 1377250 | > loss_text_ce: 0.032360050827264786 (0.030810307838642415) | > loss_mel_ce: 2.733811140060425 (nan) | > loss: 0.01097687054425478 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.1369015704699103) | > loader_time: 0.0046 (0.00469516991031371)  --> TIME: 2025-02-26 19:40:07 -- STEP: 10594/36938 -- GLOBAL_STEP: 1377300 | > loss_text_ce: 0.032836366444826126 (0.030811269026761637) | > loss_mel_ce: 2.5338876247406006 (nan) | > loss: 0.010185413993895054 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13688606515524) | > loader_time: 0.0046 (0.0046948307929274135)  --> TIME: 2025-02-26 19:40:16 -- STEP: 10644/36938 -- GLOBAL_STEP: 1377350 | > loss_text_ce: 0.02974831312894821 (0.030813789414171913) | > loss_mel_ce: 2.7182679176330566 (nan) | > loss: 0.01090482622385025 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13688496838203768) | > loader_time: 0.0046 (0.004694558094874777)  --> TIME: 2025-02-26 19:40:25 -- STEP: 10694/36938 -- GLOBAL_STEP: 1377400 | > loss_text_ce: 0.03297119215130806 (0.030811114703418803) | > loss_mel_ce: 1.2953053712844849 (nan) | > loss: 0.005270938854664564 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13686642625386777) | > loader_time: 0.0044 (0.004693998250645632)  --> TIME: 2025-02-26 19:40:35 -- STEP: 10744/36938 -- GLOBAL_STEP: 1377450 | > loss_text_ce: 0.02936655469238758 (0.030818496799815586) | > loss_mel_ce: 2.354844808578491 (nan) | > loss: 0.009461156092584133 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13688400745036855) | > loader_time: 0.0044 (0.004693700076924827)  --> TIME: 2025-02-26 19:40:44 -- STEP: 10794/36938 -- GLOBAL_STEP: 1377500 | > loss_text_ce: 0.03376515954732895 (0.030820283712034057) | > loss_mel_ce: 2.1942031383514404 (nan) | > loss: 0.008841143921017647 (nan) | > current_lr: 5e-06 | > step_time: 0.1572 (0.13687452188526872) | > loader_time: 0.0048 (0.004693205630048174)  --> TIME: 2025-02-26 19:40:53 -- STEP: 10844/36938 -- GLOBAL_STEP: 1377550 | > loss_text_ce: 0.023825088515877724 (0.03081810350285253) | > loss_mel_ce: 2.2749712467193604 (nan) | > loss: 0.009122208692133427 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13688200536314277) | > loader_time: 0.0046 (0.004693023088188574)  --> TIME: 2025-02-26 19:41:03 -- STEP: 10894/36938 -- GLOBAL_STEP: 1377600 | > loss_text_ce: 0.02944055199623108 (0.030823034578269203) | > loss_mel_ce: 2.2990810871124268 (nan) | > loss: 0.009240166284143925 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13687648794203325) | > loader_time: 0.0046 (0.004692733999083913)  --> TIME: 2025-02-26 19:41:12 -- STEP: 10944/36938 -- GLOBAL_STEP: 1377650 | > loss_text_ce: 0.036222055554389954 (0.03081658219706542) | > loss_mel_ce: 1.3257036209106445 (nan) | > loss: 0.005404467228800058 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13687201049069977) | > loader_time: 0.0046 (0.004692614405301601)  --> TIME: 2025-02-26 19:41:21 -- STEP: 10994/36938 -- GLOBAL_STEP: 1377700 | > loss_text_ce: 0.03252869099378586 (0.030818091706712153) | > loss_mel_ce: 1.9685872793197632 (nan) | > loss: 0.007940936833620071 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13685838029409403) | > loader_time: 0.0046 (0.004692401954514884)  --> TIME: 2025-02-26 19:41:31 -- STEP: 11044/36938 -- GLOBAL_STEP: 1377750 | > loss_text_ce: 0.03115399368107319 (0.030818476162129682) | > loss_mel_ce: 3.130747079849243 (nan) | > loss: 0.012547226622700691 (nan) | > current_lr: 5e-06 | > step_time: 0.1519 (0.13687256478348045) | > loader_time: 0.0048 (0.004692352366421563)  --> TIME: 2025-02-26 19:41:40 -- STEP: 11094/36938 -- GLOBAL_STEP: 1377800 | > loss_text_ce: 0.029388325288891792 (0.030816382878196897) | > loss_mel_ce: 2.1926112174987793 (nan) | > loss: 0.0088174594566226 (nan) | > current_lr: 5e-06 | > step_time: 0.1782 (0.1368780746446213) | > loader_time: 0.0048 (0.0046920800436847775)  --> TIME: 2025-02-26 19:41:49 -- STEP: 11144/36938 -- GLOBAL_STEP: 1377850 | > loss_text_ce: 0.030483614653348923 (0.03082243916760777) | > loss_mel_ce: 2.493417501449585 (nan) | > loss: 0.01001548208296299 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.1368729036999371) | > loader_time: 0.0046 (0.004691697715942778)  --> TIME: 2025-02-26 19:41:58 -- STEP: 11194/36938 -- GLOBAL_STEP: 1377900 | > loss_text_ce: 0.030595622956752777 (0.03082481353128206) | > loss_mel_ce: 3.143786668777466 (nan) | > loss: 0.012596755288541317 (nan) | > current_lr: 5e-06 | > step_time: 0.1822 (0.1368645408312429) | > loader_time: 0.0047 (0.004691397566571307)  --> TIME: 2025-02-26 19:42:08 -- STEP: 11244/36938 -- GLOBAL_STEP: 1377950 | > loss_text_ce: 0.035481687635183334 (0.030818415846863158) | > loss_mel_ce: 2.3238372802734375 (nan) | > loss: 0.009362377226352692 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13684265847224833) | > loader_time: 0.0045 (0.004691009354396315)  --> TIME: 2025-02-26 19:42:17 -- STEP: 11294/36938 -- GLOBAL_STEP: 1378000 | > loss_text_ce: 0.03839719295501709 (0.03082184496650524) | > loss_mel_ce: 2.155242919921875 (nan) | > loss: 0.008704922161996365 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13685214770712145) | > loader_time: 0.0046 (0.0046906621768155205)  --> TIME: 2025-02-26 19:42:27 -- STEP: 11344/36938 -- GLOBAL_STEP: 1378050 | > loss_text_ce: 0.03966614231467247 (0.03081880818485323) | > loss_mel_ce: 1.9742697477340698 (nan) | > loss: 0.007991809397935867 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13687018928221872) | > loader_time: 0.0044 (0.004690539244199867)  --> TIME: 2025-02-26 19:42:36 -- STEP: 11394/36938 -- GLOBAL_STEP: 1378100 | > loss_text_ce: 0.03712527081370354 (0.030816249135386005) | > loss_mel_ce: 2.3150346279144287 (nan) | > loss: 0.009333969093859196 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13687268062204566) | > loader_time: 0.0045 (0.00469027344795914)  --> TIME: 2025-02-26 19:42:45 -- STEP: 11444/36938 -- GLOBAL_STEP: 1378150 | > loss_text_ce: 0.032291848212480545 (0.030811733615297234) | > loss_mel_ce: 2.703883171081543 (nan) | > loss: 0.010857838205993176 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.1368548763835821) | > loader_time: 0.0047 (0.004690130746102115)  --> TIME: 2025-02-26 19:42:54 -- STEP: 11494/36938 -- GLOBAL_STEP: 1378200 | > loss_text_ce: 0.028617648407816887 (0.030810434665279413) | > loss_mel_ce: 2.2606589794158936 (nan) | > loss: 0.009084431454539299 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13684290814486874) | > loader_time: 0.0046 (0.004690015110655105)  --> TIME: 2025-02-26 19:43:03 -- STEP: 11544/36938 -- GLOBAL_STEP: 1378250 | > loss_text_ce: 0.028273075819015503 (0.030810392188629212) | > loss_mel_ce: 0.035640716552734375 (nan) | > loss: 0.0002536261745262891 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13681314940346867) | > loader_time: 0.0045 (0.004690113244416915)  --> TIME: 2025-02-26 19:43:13 -- STEP: 11594/36938 -- GLOBAL_STEP: 1378300 | > loss_text_ce: 0.027516795322299004 (0.030806498351168388) | > loss_mel_ce: 2.511387586593628 (nan) | > loss: 0.010075017809867859 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1368228998475393) | > loader_time: 0.0045 (0.004690262270689404)  --> TIME: 2025-02-26 19:43:22 -- STEP: 11644/36938 -- GLOBAL_STEP: 1378350 | > loss_text_ce: 0.027138318866491318 (0.030807416531846328) | > loss_mel_ce: 1.2745399475097656 (nan) | > loss: 0.005165390204638243 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.1368292428013063) | > loader_time: 0.0047 (0.004690548002576577)  --> TIME: 2025-02-26 19:43:31 -- STEP: 11694/36938 -- GLOBAL_STEP: 1378400 | > loss_text_ce: 0.024406569078564644 (0.03080584680198367) | > loss_mel_ce: 2.720813035964966 (nan) | > loss: 0.010893729515373707 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.1368285347217781) | > loader_time: 0.0046 (0.004690233552396805)  --> TIME: 2025-02-26 19:43:41 -- STEP: 11744/36938 -- GLOBAL_STEP: 1378450 | > loss_text_ce: 0.03637110814452171 (0.030806632623903082) | > loss_mel_ce: 2.154425621032715 (nan) | > loss: 0.008693638257682323 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13682173984252166) | > loader_time: 0.0046 (0.004689940883288297)  --> TIME: 2025-02-26 19:43:50 -- STEP: 11794/36938 -- GLOBAL_STEP: 1378500 | > loss_text_ce: 0.02892865613102913 (0.030804052797574293) | > loss_mel_ce: 3.1727330684661865 (nan) | > loss: 0.012705007568001747 (nan) | > current_lr: 5e-06 | > step_time: 0.1708 (0.13681635232786035) | > loader_time: 0.0047 (0.004689596175177163)  --> TIME: 2025-02-26 19:43:59 -- STEP: 11844/36938 -- GLOBAL_STEP: 1378550 | > loss_text_ce: 0.028077758848667145 (0.03080320132341788) | > loss_mel_ce: 2.905505657196045 (nan) | > loss: 0.011641205288469791 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13682545394923257) | > loader_time: 0.0047 (0.0046893296029188805)  --> TIME: 2025-02-26 19:44:08 -- STEP: 11894/36938 -- GLOBAL_STEP: 1378600 | > loss_text_ce: 0.026128659024834633 (0.030796728641178748) | > loss_mel_ce: 2.9336023330688477 (nan) | > loss: 0.011744965799152851 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13679372933565628) | > loader_time: 0.0046 (0.00468880109512771)  --> TIME: 2025-02-26 19:44:17 -- STEP: 11944/36938 -- GLOBAL_STEP: 1378650 | > loss_text_ce: 0.037408266216516495 (0.030799434761478076) | > loss_mel_ce: 2.565640926361084 (nan) | > loss: 0.010329561308026314 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13678261158614885) | > loader_time: 0.0045 (0.004688389873792115)  --> TIME: 2025-02-26 19:44:26 -- STEP: 11994/36938 -- GLOBAL_STEP: 1378700 | > loss_text_ce: 0.0353679396212101 (0.030804595984788748) | > loss_mel_ce: 2.1186530590057373 (nan) | > loss: 0.008547702804207802 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13676477222019645) | > loader_time: 0.0045 (0.004687938826152151)  --> TIME: 2025-02-26 19:44:35 -- STEP: 12044/36938 -- GLOBAL_STEP: 1378750 | > loss_text_ce: 0.03401966392993927 (0.030798916974513535) | > loss_mel_ce: 1.499652624130249 (nan) | > loss: 0.006086001638323069 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13675213315406964) | > loader_time: 0.0045 (0.004687545347831396)  --> TIME: 2025-02-26 19:44:45 -- STEP: 12094/36938 -- GLOBAL_STEP: 1378800 | > loss_text_ce: 0.03602940961718559 (0.03079793853609793) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.1367665122949886) | > loader_time: 0.0045 (0.00468730364670653)  --> TIME: 2025-02-26 19:44:54 -- STEP: 12144/36938 -- GLOBAL_STEP: 1378850 | > loss_text_ce: 0.028478315100073814 (0.030803787487814243) | > loss_mel_ce: 1.9875768423080444 (nan) | > loss: 0.008000219240784645 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13678740194663752) | > loader_time: 0.0045 (0.004687211180550021)  --> TIME: 2025-02-26 19:45:04 -- STEP: 12194/36938 -- GLOBAL_STEP: 1378900 | > loss_text_ce: 0.03215692564845085 (0.03080662601985305) | > loss_mel_ce: 2.8705832958221436 (nan) | > loss: 0.011518810875713825 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13680180444353024) | > loader_time: 0.0045 (0.004687020656259753)  --> TIME: 2025-02-26 19:45:13 -- STEP: 12244/36938 -- GLOBAL_STEP: 1378950 | > loss_text_ce: 0.03205496072769165 (0.030805298927727185) | > loss_mel_ce: 2.642099142074585 (nan) | > loss: 0.01061172317713499 (nan) | > current_lr: 5e-06 | > step_time: 0.1352 (0.13680872246160633) | > loader_time: 0.0046 (0.004686744238028796)  --> TIME: 2025-02-26 19:45:22 -- STEP: 12294/36938 -- GLOBAL_STEP: 1379000 | > loss_text_ce: 0.044628776609897614 (0.03081067306343639) | > loss_mel_ce: 0.00860595703125 (nan) | > loss: 0.00021124896011315286 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13677997301705885) | > loader_time: 0.0045 (0.004686569263823806)  --> TIME: 2025-02-26 19:45:31 -- STEP: 12344/36938 -- GLOBAL_STEP: 1379050 | > loss_text_ce: 0.031380198895931244 (0.030812160024444778) | > loss_mel_ce: 3.3788280487060547 (nan) | > loss: 0.013532573357224464 (nan) | > current_lr: 5e-06 | > step_time: 0.1577 (0.1367765363747634) | > loader_time: 0.0046 (0.004686266608772877)  --> TIME: 2025-02-26 19:45:41 -- STEP: 12394/36938 -- GLOBAL_STEP: 1379100 | > loss_text_ce: 0.03322172537446022 (0.030815035471900377) | > loss_mel_ce: 2.1228549480438232 (nan) | > loss: 0.008555860258638859 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13678262028671906) | > loader_time: 0.0046 (0.004686388773977433)  --> TIME: 2025-02-26 19:45:50 -- STEP: 12444/36938 -- GLOBAL_STEP: 1379150 | > loss_text_ce: 0.03210481256246567 (0.030817071196082194) | > loss_mel_ce: 2.5036916732788086 (nan) | > loss: 0.010062684305012226 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.1367920279426237) | > loader_time: 0.0047 (0.004686313842661883)  --> TIME: 2025-02-26 19:46:00 -- STEP: 12494/36938 -- GLOBAL_STEP: 1379200 | > loss_text_ce: 0.034525446593761444 (0.030817104353683795) | > loss_mel_ce: 2.618516683578491 (nan) | > loss: 0.01052794512361288 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13680091907219774) | > loader_time: 0.0046 (0.004686332558066624)  --> TIME: 2025-02-26 19:46:09 -- STEP: 12544/36938 -- GLOBAL_STEP: 1379250 | > loss_text_ce: 0.02469288557767868 (0.030813783452351938) | > loss_mel_ce: 1.558761477470398 (nan) | > loss: 0.006283549591898918 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.1367997086930038) | > loader_time: 0.0045 (0.004686335329802706)  --> TIME: 2025-02-26 19:46:18 -- STEP: 12594/36938 -- GLOBAL_STEP: 1379300 | > loss_text_ce: 0.0436653308570385 (0.030818788255749) | > loss_mel_ce: 2.93468976020813 (nan) | > loss: 0.011818869970738888 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13678331760938453) | > loader_time: 0.0045 (0.004686029710447105)  --> TIME: 2025-02-26 19:46:28 -- STEP: 12644/36938 -- GLOBAL_STEP: 1379350 | > loss_text_ce: 0.031045524403452873 (0.030817422390040698) | > loss_mel_ce: 2.2428276538848877 (nan) | > loss: 0.009023306891322136 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.136792295343399) | > loader_time: 0.0047 (0.004685776533864826)  --> TIME: 2025-02-26 19:46:37 -- STEP: 12694/36938 -- GLOBAL_STEP: 1379400 | > loss_text_ce: 0.032752715051174164 (0.03082268131811454) | > loss_mel_ce: 1.8271065950393677 (nan) | > loss: 0.007380394730716944 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13680275190490784) | > loader_time: 0.0045 (0.004685541297650588)  --> TIME: 2025-02-26 19:46:46 -- STEP: 12744/36938 -- GLOBAL_STEP: 1379450 | > loss_text_ce: 0.03148045390844345 (0.03082493080231972) | > loss_mel_ce: 1.249685287475586 (nan) | > loss: 0.005083991214632988 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.13679705050406235) | > loader_time: 0.0046 (0.004685266992242133)  --> TIME: 2025-02-26 19:46:56 -- STEP: 12794/36938 -- GLOBAL_STEP: 1379500 | > loss_text_ce: 0.02564804069697857 (0.030822103139177857) | > loss_mel_ce: 2.766528367996216 (nan) | > loss: 0.011080065742135048 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.1367953366109207) | > loader_time: 0.0046 (0.004685342209365636)  --> TIME: 2025-02-26 19:47:05 -- STEP: 12844/36938 -- GLOBAL_STEP: 1379550 | > loss_text_ce: 0.01981416903436184 (0.030818651972105943) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13680115560251288) | > loader_time: 0.0045 (0.004685303144683677)  --> TIME: 2025-02-26 19:47:14 -- STEP: 12894/36938 -- GLOBAL_STEP: 1379600 | > loss_text_ce: 0.03181670606136322 (0.030823046734176053) | > loss_mel_ce: 2.1839728355407715 (nan) | > loss: 0.008792816661298275 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13678739508011586) | > loader_time: 0.0046 (0.004684969345987451)  --> TIME: 2025-02-26 19:47:23 -- STEP: 12944/36938 -- GLOBAL_STEP: 1379650 | > loss_text_ce: 0.03162737935781479 (0.030821309557037223) | > loss_mel_ce: 2.695603847503662 (nan) | > loss: 0.010822347365319729 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13679334495506754) | > loader_time: 0.0045 (0.004685047972482298)  --> TIME: 2025-02-26 19:47:33 -- STEP: 12994/36938 -- GLOBAL_STEP: 1379700 | > loss_text_ce: 0.03976382315158844 (0.030822773322541016) | > loss_mel_ce: 2.6880345344543457 (nan) | > loss: 0.01082459744066 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.1368142098156294) | > loader_time: 0.0046 (0.004685177112407337)  --> TIME: 2025-02-26 19:47:42 -- STEP: 13044/36938 -- GLOBAL_STEP: 1379750 | > loss_text_ce: 0.026518160477280617 (0.03082125830300991) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.1368085032078216) | > loader_time: 0.0046 (0.004684968891775514)  --> TIME: 2025-02-26 19:47:52 -- STEP: 13094/36938 -- GLOBAL_STEP: 1379800 | > loss_text_ce: 0.04580282047390938 (0.0308247787107766) | > loss_mel_ce: 2.912114143371582 (nan) | > loss: 0.011737766675651073 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1368200094513776) | > loader_time: 0.0045 (0.004685082471296071)  --> TIME: 2025-02-26 19:48:01 -- STEP: 13144/36938 -- GLOBAL_STEP: 1379850 | > loss_text_ce: 0.02683619223535061 (0.030825018087896273) | > loss_mel_ce: 2.364940643310547 (nan) | > loss: 0.009491178207099438 (nan) | > current_lr: 5e-06 | > step_time: 0.1833 (0.13680897425458893) | > loader_time: 0.0048 (0.00468486371522118)  --> TIME: 2025-02-26 19:48:11 -- STEP: 13194/36938 -- GLOBAL_STEP: 1379900 | > loss_text_ce: 0.030488526448607445 (0.03082728646828317) | > loss_mel_ce: 2.677607536315918 (nan) | > loss: 0.010746413841843605 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.1368063547872537) | > loader_time: 0.0045 (0.004684998896224283)  --> TIME: 2025-02-26 19:48:20 -- STEP: 13244/36938 -- GLOBAL_STEP: 1379950 | > loss_text_ce: 0.02692502737045288 (0.03082949362584955) | > loss_mel_ce: 2.5493927001953125 (nan) | > loss: 0.01022348366677761 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1368041309083354) | > loader_time: 0.0046 (0.004684957608972157)  --> TIME: 2025-02-26 19:48:29 -- STEP: 13294/36938 -- GLOBAL_STEP: 1380000 | > loss_text_ce: 0.029654690995812416 (0.03082742677166177) | > loss_mel_ce: 2.888533115386963 (nan) | > loss: 0.01158011145889759 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.1368029742415235) | > loader_time: 0.0054 (0.004685026336401363) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1380000.pth  --> TIME: 2025-02-26 19:48:45 -- STEP: 13344/36938 -- GLOBAL_STEP: 1380050 | > loss_text_ce: 0.02638796530663967 (0.03082391683710739) | > loss_mel_ce: 2.4838054180145264 (nan) | > loss: 0.009961085394024849 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13680122585462515) | > loader_time: 0.0046 (0.004684949896747283)  --> TIME: 2025-02-26 19:48:54 -- STEP: 13394/36938 -- GLOBAL_STEP: 1380100 | > loss_text_ce: 0.034024424850940704 (0.03082021941066932) | > loss_mel_ce: 2.966973066329956 (nan) | > loss: 0.011908721178770065 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13679707365390673) | > loader_time: 0.0044 (0.004685002297487893)  --> TIME: 2025-02-26 19:49:03 -- STEP: 13444/36938 -- GLOBAL_STEP: 1380150 | > loss_text_ce: 0.03433047607541084 (0.03081975577050547) | > loss_mel_ce: 2.3462648391723633 (nan) | > loss: 0.009446807205677032 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13677835652460138) | > loader_time: 0.0048 (0.004684927633070465)  --> TIME: 2025-02-26 19:49:12 -- STEP: 13494/36938 -- GLOBAL_STEP: 1380200 | > loss_text_ce: 0.028401624411344528 (0.03081040884192403) | > loss_mel_ce: 3.052502155303955 (nan) | > loss: 0.012225809507071972 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.13677219638157384) | > loader_time: 0.0047 (0.0046846884806209)  --> TIME: 2025-02-26 19:49:21 -- STEP: 13544/36938 -- GLOBAL_STEP: 1380250 | > loss_text_ce: 0.037039902061223984 (0.03081282002235803) | > loss_mel_ce: 2.873861312866211 (nan) | > loss: 0.011551196686923504 (nan) | > current_lr: 5e-06 | > step_time: 0.1721 (0.13677297084555504) | > loader_time: 0.0048 (0.004684576957238492)  --> TIME: 2025-02-26 19:49:30 -- STEP: 13594/36938 -- GLOBAL_STEP: 1380300 | > loss_text_ce: 0.03115377575159073 (0.03081619257072574) | > loss_mel_ce: 2.837554931640625 (nan) | > loss: 0.011383765377104282 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13677083018318292) | > loader_time: 0.0046 (0.004684350640349009)  --> TIME: 2025-02-26 19:49:40 -- STEP: 13644/36938 -- GLOBAL_STEP: 1380350 | > loss_text_ce: 0.03168810158967972 (0.030816787456297973) | > loss_mel_ce: 2.265763759613037 (nan) | > loss: 0.009116873145103455 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.13678097335238998) | > loader_time: 0.0049 (0.004684187683741832)  --> TIME: 2025-02-26 19:49:49 -- STEP: 13694/36938 -- GLOBAL_STEP: 1380400 | > loss_text_ce: 0.033497076481580734 (0.030811398596820718) | > loss_mel_ce: 1.0549862384796143 (nan) | > loss: 0.004319378640502691 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.13678301855371222) | > loader_time: 0.0048 (0.004683933868118514)  --> TIME: 2025-02-26 19:49:58 -- STEP: 13744/36938 -- GLOBAL_STEP: 1380450 | > loss_text_ce: 0.024441950023174286 (0.030807336345526397) | > loss_mel_ce: 1.1533541679382324 (nan) | > loss: 0.004673794377595186 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13677696580104218) | > loader_time: 0.0045 (0.004683861355009976)  --> TIME: 2025-02-26 19:50:08 -- STEP: 13794/36938 -- GLOBAL_STEP: 1380500 | > loss_text_ce: 0.02778702974319458 (0.030806507648703272) | > loss_mel_ce: 2.640256643295288 (nan) | > loss: 0.010587475262582302 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13677303297063106) | > loader_time: 0.0046 (0.004684006699413726)  --> TIME: 2025-02-26 19:50:17 -- STEP: 13844/36938 -- GLOBAL_STEP: 1380550 | > loss_text_ce: 0.02914908342063427 (0.030807387864804317) | > loss_mel_ce: 2.761936664581299 (nan) | > loss: 0.01107573788613081 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13677118240288744) | > loader_time: 0.0046 (0.004684086446643619)  --> TIME: 2025-02-26 19:50:26 -- STEP: 13894/36938 -- GLOBAL_STEP: 1380600 | > loss_text_ce: 0.03204061836004257 (0.030811528792427576) | > loss_mel_ce: 1.3113514184951782 (nan) | > loss: 0.005330921150743961 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13677075358111246) | > loader_time: 0.0045 (0.0046838257009732395)  --> TIME: 2025-02-26 19:50:35 -- STEP: 13944/36938 -- GLOBAL_STEP: 1380650 | > loss_text_ce: 0.03263014554977417 (0.03081562708113802) | > loss_mel_ce: 3.1204328536987305 (nan) | > loss: 0.012512155808508396 (nan) | > current_lr: 5e-06 | > step_time: 0.1701 (0.13675998988263785) | > loader_time: 0.0048 (0.004683719222325418)  --> TIME: 2025-02-26 19:50:45 -- STEP: 13994/36938 -- GLOBAL_STEP: 1380700 | > loss_text_ce: 0.03649589791893959 (0.030813508301151298) | > loss_mel_ce: 2.664978265762329 (nan) | > loss: 0.01072013657540083 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.1367537414105913) | > loader_time: 0.0045 (0.004683602583720157)  --> TIME: 2025-02-26 19:50:54 -- STEP: 14044/36938 -- GLOBAL_STEP: 1380750 | > loss_text_ce: 0.030900167301297188 (0.030809406309681157) | > loss_mel_ce: 2.654728651046753 (nan) | > loss: 0.010657258331775665 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13672986910982238) | > loader_time: 0.0047 (0.004683319471799848)  --> TIME: 2025-02-26 19:51:03 -- STEP: 14094/36938 -- GLOBAL_STEP: 1380800 | > loss_text_ce: 0.02492944896221161 (0.030804922774765015) | > loss_mel_ce: 2.3523707389831543 (nan) | > loss: 0.009433731436729431 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1367357249494754) | > loader_time: 0.0045 (0.004683125233910739)  --> TIME: 2025-02-26 19:51:12 -- STEP: 14144/36938 -- GLOBAL_STEP: 1380850 | > loss_text_ce: 0.03246736153960228 (0.030807070190990892) | > loss_mel_ce: 2.979787826538086 (nan) | > loss: 0.011953393928706646 (nan) | > current_lr: 5e-06 | > step_time: 0.1908 (0.13673668701994002) | > loader_time: 0.0048 (0.00468333048653281)  --> TIME: 2025-02-26 19:51:22 -- STEP: 14194/36938 -- GLOBAL_STEP: 1380900 | > loss_text_ce: 0.03324109688401222 (0.030809556161996612) | > loss_mel_ce: 2.935835838317871 (nan) | > loss: 0.011782051995396614 (nan) | > current_lr: 5e-06 | > step_time: 0.1993 (0.136750525784557) | > loader_time: 0.0048 (0.004683437743152705)  --> TIME: 2025-02-26 19:51:31 -- STEP: 14244/36938 -- GLOBAL_STEP: 1380950 | > loss_text_ce: 0.029313277453184128 (0.030805046507530073) | > loss_mel_ce: 0.6033380627632141 (nan) | > loss: 0.0025105213280767202 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13673118325104708) | > loader_time: 0.0046 (0.004683318365092245)  --> TIME: 2025-02-26 19:51:40 -- STEP: 14294/36938 -- GLOBAL_STEP: 1381000 | > loss_text_ce: 0.026869997382164 (0.03080543454963731) | > loss_mel_ce: 0.05614471435546875 (nan) | > loss: 0.0003294234920758754 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13672448660721281) | > loader_time: 0.0046 (0.004683116774334199)  --> TIME: 2025-02-26 19:51:50 -- STEP: 14344/36938 -- GLOBAL_STEP: 1381050 | > loss_text_ce: 0.03321739286184311 (0.030805986591475057) | > loss_mel_ce: 2.8884005546569824 (nan) | > loss: 0.011593722738325596 (nan) | > current_lr: 5e-06 | > step_time: 0.1343 (0.13673279866185648) | > loader_time: 0.0046 (0.004682919763681035)  --> TIME: 2025-02-26 19:51:59 -- STEP: 14394/36938 -- GLOBAL_STEP: 1381100 | > loss_text_ce: 0.024562863633036613 (0.03080759434800968) | > loss_mel_ce: 1.499267578125 (nan) | > loss: 0.0060469466261565685 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13673509972715364) | > loader_time: 0.0045 (0.004682695516136055)  --> TIME: 2025-02-26 19:52:08 -- STEP: 14444/36938 -- GLOBAL_STEP: 1381150 | > loss_text_ce: 0.03023059107363224 (0.030804990709799786) | > loss_mel_ce: 2.5872137546539307 (nan) | > loss: 0.010386683978140354 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13672286653347063) | > loader_time: 0.0047 (0.0046825024996461646)  --> TIME: 2025-02-26 19:52:17 -- STEP: 14494/36938 -- GLOBAL_STEP: 1381200 | > loss_text_ce: 0.024362750351428986 (0.030808900315373648) | > loss_mel_ce: 2.4855473041534424 (nan) | > loss: 0.009959961287677288 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13670124903339825) | > loader_time: 0.0045 (0.00468220739705459)  --> TIME: 2025-02-26 19:52:26 -- STEP: 14544/36938 -- GLOBAL_STEP: 1381250 | > loss_text_ce: 0.03118644654750824 (0.030810827349017533) | > loss_mel_ce: 2.1731770038604736 (nan) | > loss: 0.008747474290430546 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13669949578176693) | > loader_time: 0.0046 (0.004682423553057649)  --> TIME: 2025-02-26 19:52:36 -- STEP: 14594/36938 -- GLOBAL_STEP: 1381300 | > loss_text_ce: 0.03742978349328041 (0.03081288462323764) | > loss_mel_ce: 2.846324920654297 (nan) | > loss: 0.011443471536040306 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13670449459799497) | > loader_time: 0.0046 (0.0046824120129071445)  --> TIME: 2025-02-26 19:52:45 -- STEP: 14644/36938 -- GLOBAL_STEP: 1381350 | > loss_text_ce: 0.03408098965883255 (0.030815737422838566) | > loss_mel_ce: 0.7961580157279968 (nan) | > loss: 0.003294599475339055 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13670329318545463) | > loader_time: 0.0046 (0.004682341451626682)  --> TIME: 2025-02-26 19:52:54 -- STEP: 14694/36938 -- GLOBAL_STEP: 1381400 | > loss_text_ce: 0.030292751267552376 (0.03082012569486671) | > loss_mel_ce: 2.5264575481414795 (nan) | > loss: 0.010145834647119045 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13669082130950352) | > loader_time: 0.0045 (0.00468204564037042)  --> TIME: 2025-02-26 19:53:03 -- STEP: 14744/36938 -- GLOBAL_STEP: 1381450 | > loss_text_ce: 0.03688950836658478 (0.030816491650621097) | > loss_mel_ce: 1.963281273841858 (nan) | > loss: 0.00793718546628952 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13668077547008203) | > loader_time: 0.0045 (0.00468176216841134)  --> TIME: 2025-02-26 19:53:13 -- STEP: 14794/36938 -- GLOBAL_STEP: 1381500 | > loss_text_ce: 0.020263463258743286 (0.03081104524197666) | > loss_mel_ce: 2.710519313812256 (nan) | > loss: 0.01083644013851881 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13669860604745088) | > loader_time: 0.0046 (0.004681608314947375)  --> TIME: 2025-02-26 19:53:23 -- STEP: 14844/36938 -- GLOBAL_STEP: 1381550 | > loss_text_ce: 0.038563959300518036 (0.03081047891516703) | > loss_mel_ce: 2.086132764816284 (nan) | > loss: 0.008431336842477322 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13670767524498773) | > loader_time: 0.0055 (0.0046819391612960524)  --> TIME: 2025-02-26 19:53:32 -- STEP: 14894/36938 -- GLOBAL_STEP: 1381600 | > loss_text_ce: 0.031614091247320175 (0.030806097388405083) | > loss_mel_ce: 0.914003312587738 (nan) | > loss: 0.0037524502258747816 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13670869036554537) | > loader_time: 0.0047 (0.004681886547089154)  --> TIME: 2025-02-26 19:53:41 -- STEP: 14944/36938 -- GLOBAL_STEP: 1381650 | > loss_text_ce: 0.025426791980862617 (0.030806472567939627) | > loss_mel_ce: 2.771623373031616 (nan) | > loss: 0.011099406518042088 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.13671320472590698) | > loader_time: 0.0047 (0.004681599599646316)  --> TIME: 2025-02-26 19:53:51 -- STEP: 14994/36938 -- GLOBAL_STEP: 1381700 | > loss_text_ce: 0.02842657081782818 (0.030804185954598235) | > loss_mel_ce: 2.6479618549346924 (nan) | > loss: 0.010620590299367905 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.13671868092635028) | > loader_time: 0.0049 (0.004681545845012422)  --> TIME: 2025-02-26 19:54:00 -- STEP: 15044/36938 -- GLOBAL_STEP: 1381750 | > loss_text_ce: 0.03277476876974106 (0.030805027717377767) | > loss_mel_ce: 3.119624614715576 (nan) | > loss: 0.012509522028267384 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.1367335094951692) | > loader_time: 0.0047 (0.0046814809578335625)  --> TIME: 2025-02-26 19:54:10 -- STEP: 15094/36938 -- GLOBAL_STEP: 1381800 | > loss_text_ce: 0.040923312306404114 (0.030806145219042715) | > loss_mel_ce: 1.40673828125 (nan) | > loss: 0.005744689144194126 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13674753412310164) | > loader_time: 0.0046 (0.004681363822261738)  --> TIME: 2025-02-26 19:54:20 -- STEP: 15144/36938 -- GLOBAL_STEP: 1381850 | > loss_text_ce: 0.026356181129813194 (0.030807701608387982) | > loss_mel_ce: 2.7733139991760254 (nan) | > loss: 0.01110980287194252 (nan) | > current_lr: 5e-06 | > step_time: 0.1532 (0.13677216438912135) | > loader_time: 0.0052 (0.0046813853569151855)  --> TIME: 2025-02-26 19:54:30 -- STEP: 15194/36938 -- GLOBAL_STEP: 1381900 | > loss_text_ce: 0.02630426175892353 (0.03080767841273628) | > loss_mel_ce: 2.968388795852661 (nan) | > loss: 0.01188370306044817 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.1368029450089805) | > loader_time: 0.0048 (0.004681343057523753)  --> TIME: 2025-02-26 19:54:39 -- STEP: 15244/36938 -- GLOBAL_STEP: 1381950 | > loss_text_ce: 0.03462451696395874 (0.030808830554520102) | > loss_mel_ce: 3.071495532989502 (nan) | > loss: 0.012325874529778957 (nan) | > current_lr: 5e-06 | > step_time: 0.1617 (0.13679621107651496) | > loader_time: 0.0049 (0.004681229388047585)  --> TIME: 2025-02-26 19:54:48 -- STEP: 15294/36938 -- GLOBAL_STEP: 1382000 | > loss_text_ce: 0.028616704046726227 (0.03080765524566343) | > loss_mel_ce: 2.492313861846924 (nan) | > loss: 0.010003693401813507 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.1368005792970237) | > loader_time: 0.0048 (0.004681108947889612)  --> TIME: 2025-02-26 19:54:58 -- STEP: 15344/36938 -- GLOBAL_STEP: 1382050 | > loss_text_ce: 0.03197349235415459 (0.03081023020564692) | > loss_mel_ce: 2.3406264781951904 (nan) | > loss: 0.009415079839527607 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13680122659092053) | > loader_time: 0.0046 (0.004681025589966818)  --> TIME: 2025-02-26 19:55:07 -- STEP: 15394/36938 -- GLOBAL_STEP: 1382100 | > loss_text_ce: 0.03623383864760399 (0.03081357059431609) | > loss_mel_ce: 2.751190423965454 (nan) | > loss: 0.011061208322644234 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13680491773311226) | > loader_time: 0.0046 (0.0046808803733547335)  --> TIME: 2025-02-26 19:55:17 -- STEP: 15444/36938 -- GLOBAL_STEP: 1382150 | > loss_text_ce: 0.030853038653731346 (0.030813875843630174) | > loss_mel_ce: 3.332284688949585 (nan) | > loss: 0.013345785439014435 (nan) | > current_lr: 5e-06 | > step_time: 0.1953 (0.13682352297048586) | > loader_time: 0.0049 (0.004680779847231796)  --> TIME: 2025-02-26 19:55:26 -- STEP: 15494/36938 -- GLOBAL_STEP: 1382200 | > loss_text_ce: 0.0350114069879055 (0.030815483389906292) | > loss_mel_ce: 2.88230299949646 (nan) | > loss: 0.011576645076274872 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.13681855258040565) | > loader_time: 0.0045 (0.004680746645254693)  --> TIME: 2025-02-26 19:55:35 -- STEP: 15544/36938 -- GLOBAL_STEP: 1382250 | > loss_text_ce: 0.026532335206866264 (0.03081508004867505) | > loss_mel_ce: 2.258629083633423 (nan) | > loss: 0.009068101644515991 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13680947127565574) | > loader_time: 0.0046 (0.004680536652862952)  --> TIME: 2025-02-26 19:55:45 -- STEP: 15594/36938 -- GLOBAL_STEP: 1382300 | > loss_text_ce: 0.015299553982913494 (0.030817281037695182) | > loss_mel_ce: 2.2358381748199463 (nan) | > loss: 0.00893308687955141 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13683065928387883) | > loader_time: 0.0045 (0.004680413595602991)  --> TIME: 2025-02-26 19:55:54 -- STEP: 15644/36938 -- GLOBAL_STEP: 1382350 | > loss_text_ce: 0.03228974714875221 (0.030820249914060932) | > loss_mel_ce: 2.4791533946990967 (nan) | > loss: 0.009966044686734676 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.1368241001047221) | > loader_time: 0.0047 (0.0046801951436916175)  --> TIME: 2025-02-26 19:56:03 -- STEP: 15694/36938 -- GLOBAL_STEP: 1382400 | > loss_text_ce: 0.034285567700862885 (0.030823765423092377) | > loss_mel_ce: 1.78265380859375 (nan) | > loss: 0.007210077252238989 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.1368157537906844) | > loader_time: 0.0046 (0.004680065025839077)  --> TIME: 2025-02-26 19:56:13 -- STEP: 15744/36938 -- GLOBAL_STEP: 1382450 | > loss_text_ce: 0.03407244011759758 (0.030824526418078542) | > loss_mel_ce: 2.745709180831909 (nan) | > loss: 0.01103087980300188 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13682089627879446) | > loader_time: 0.0046 (0.004679965551912318)  --> TIME: 2025-02-26 19:56:22 -- STEP: 15794/36938 -- GLOBAL_STEP: 1382500 | > loss_text_ce: 0.028023967519402504 (0.03082234715963625) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13681542505052002) | > loader_time: 0.0046 (0.004679864760484987)  --> TIME: 2025-02-26 19:56:31 -- STEP: 15844/36938 -- GLOBAL_STEP: 1382550 | > loss_text_ce: 0.03113238513469696 (0.0308156994471105) | > loss_mel_ce: 3.4296035766601562 (nan) | > loss: 0.013733080588281155 (nan) | > current_lr: 5e-06 | > step_time: 0.1871 (0.1368267317367186) | > loader_time: 0.0048 (0.004680009358942271)  --> TIME: 2025-02-26 19:56:41 -- STEP: 15894/36938 -- GLOBAL_STEP: 1382600 | > loss_text_ce: 0.03279370814561844 (0.030815989093551967) | > loss_mel_ce: 3.341506004333496 (nan) | > loss: 0.013390079140663147 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13682451484577474) | > loader_time: 0.0046 (0.004680011967555451)  --> TIME: 2025-02-26 19:56:50 -- STEP: 15944/36938 -- GLOBAL_STEP: 1382650 | > loss_text_ce: 0.04320962354540825 (0.030819480706880602) | > loss_mel_ce: 1.6755502223968506 (nan) | > loss: 0.006820476148277521 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13682409928063943) | > loader_time: 0.0044 (0.004679844179646327)  --> TIME: 2025-02-26 19:56:59 -- STEP: 15994/36938 -- GLOBAL_STEP: 1382700 | > loss_text_ce: 0.026574749499559402 (0.0308196995723762) | > loss_mel_ce: 1.001969337463379 (nan) | > loss: 0.004081524442881346 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13682319609033974) | > loader_time: 0.0046 (0.004679576447565483)  --> TIME: 2025-02-26 19:57:09 -- STEP: 16044/36938 -- GLOBAL_STEP: 1382750 | > loss_text_ce: 0.027500614523887634 (0.030824733958355534) | > loss_mel_ce: 1.1993340253829956 (nan) | > loss: 0.004868391901254654 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13683070569702965) | > loader_time: 0.0046 (0.004679544864821945)  --> TIME: 2025-02-26 19:57:18 -- STEP: 16094/36938 -- GLOBAL_STEP: 1382800 | > loss_text_ce: 0.0277976393699646 (0.030827250922874413) | > loss_mel_ce: 2.964332342147827 (nan) | > loss: 0.011873533017933369 (nan) | > current_lr: 5e-06 | > step_time: 0.1697 (0.1368253635029862) | > loader_time: 0.0046 (0.004679320302086041)  --> TIME: 2025-02-26 19:57:27 -- STEP: 16144/36938 -- GLOBAL_STEP: 1382850 | > loss_text_ce: 0.02667517401278019 (0.030822230919762224) | > loss_mel_ce: 3.250823974609375 (nan) | > loss: 0.013005949556827545 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13682681369356192) | > loader_time: 0.0046 (0.004679120183581775)  --> TIME: 2025-02-26 19:57:37 -- STEP: 16194/36938 -- GLOBAL_STEP: 1382900 | > loss_text_ce: 0.03335745260119438 (0.030823555817946973) | > loss_mel_ce: 2.660749912261963 (nan) | > loss: 0.01069090235978365 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.13683864426079917) | > loader_time: 0.0046 (0.004679187707169598)  --> TIME: 2025-02-26 19:57:46 -- STEP: 16244/36938 -- GLOBAL_STEP: 1382950 | > loss_text_ce: 0.035559508949518204 (0.030827757031034162) | > loss_mel_ce: 2.696408271789551 (nan) | > loss: 0.010841142386198044 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13682862008859675) | > loader_time: 0.0046 (0.004679001044005914)  --> TIME: 2025-02-26 19:57:56 -- STEP: 16294/36938 -- GLOBAL_STEP: 1383000 | > loss_text_ce: 0.029819216579198837 (0.030823788110247264) | > loss_mel_ce: 2.4903159141540527 (nan) | > loss: 0.010000537149608135 (nan) | > current_lr: 5e-06 | > step_time: 0.1511 (0.13683427430756118) | > loader_time: 0.0053 (0.004679084760624472)  --> TIME: 2025-02-26 19:58:05 -- STEP: 16344/36938 -- GLOBAL_STEP: 1383050 | > loss_text_ce: 0.03627875819802284 (0.030822083504487157) | > loss_mel_ce: 2.3343505859375 (nan) | > loss: 0.00940725952386856 (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.13683260932776276) | > loader_time: 0.0046 (0.0046789256661649145)  --> TIME: 2025-02-26 19:58:14 -- STEP: 16394/36938 -- GLOBAL_STEP: 1383100 | > loss_text_ce: 0.035711776465177536 (0.030824460664281628) | > loss_mel_ce: 0.018463134765625 (nan) | > loss: 0.00021497982379514724 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13681866960640715) | > loader_time: 0.0045 (0.004678637672811527)  --> TIME: 2025-02-26 19:58:24 -- STEP: 16444/36938 -- GLOBAL_STEP: 1383150 | > loss_text_ce: 0.032652080059051514 (0.0308230161634361) | > loss_mel_ce: 1.0684013366699219 (nan) | > loss: 0.004369260277599096 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13683015164653936) | > loader_time: 0.0046 (0.00467845951951181)  --> TIME: 2025-02-26 19:58:33 -- STEP: 16494/36938 -- GLOBAL_STEP: 1383200 | > loss_text_ce: 0.027355538681149483 (0.03082026729072932) | > loss_mel_ce: 3.0705814361572266 (nan) | > loss: 0.012293401174247265 (nan) | > current_lr: 5e-06 | > step_time: 0.162 (0.13684417692057035) | > loader_time: 0.0047 (0.004678294646229719)  --> TIME: 2025-02-26 19:58:43 -- STEP: 16544/36938 -- GLOBAL_STEP: 1383250 | > loss_text_ce: 0.03159508854150772 (0.030822211039030555) | > loss_mel_ce: 2.6136887073516846 (nan) | > loss: 0.010497158393263817 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13685553443097537) | > loader_time: 0.0048 (0.004678186756960444)  --> TIME: 2025-02-26 19:58:52 -- STEP: 16594/36938 -- GLOBAL_STEP: 1383300 | > loss_text_ce: 0.02452610619366169 (0.030821389230088293) | > loss_mel_ce: 2.119230270385742 (nan) | > loss: 0.00850697048008442 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13686021201593634) | > loader_time: 0.0046 (0.0046780346473527575)  --> TIME: 2025-02-26 19:59:02 -- STEP: 16644/36938 -- GLOBAL_STEP: 1383350 | > loss_text_ce: 0.030592383816838264 (0.03082127326366873) | > loss_mel_ce: 2.113959550857544 (nan) | > loss: 0.008510127663612366 (nan) | > current_lr: 5e-06 | > step_time: 0.1567 (0.13685826086648487) | > loader_time: 0.0047 (0.004677834003139552)  --> TIME: 2025-02-26 19:59:11 -- STEP: 16694/36938 -- GLOBAL_STEP: 1383400 | > loss_text_ce: 0.02333528734743595 (0.030819398175616613) | > loss_mel_ce: 3.5446605682373047 (nan) | > loss: 0.014158714562654495 (nan) | > current_lr: 5e-06 | > step_time: 0.1453 (0.13684959064290342) | > loader_time: 0.0046 (0.004677579404923039)  --> TIME: 2025-02-26 19:59:20 -- STEP: 16744/36938 -- GLOBAL_STEP: 1383450 | > loss_text_ce: 0.025723397731781006 (0.03081437528306485) | > loss_mel_ce: 2.6242623329162598 (nan) | > loss: 0.010515817441046238 (nan) | > current_lr: 5e-06 | > step_time: 0.1706 (0.13683748977126217) | > loader_time: 0.0048 (0.004677315534044001)  --> TIME: 2025-02-26 19:59:29 -- STEP: 16794/36938 -- GLOBAL_STEP: 1383500 | > loss_text_ce: 0.0404236875474453 (0.030814394464178833) | > loss_mel_ce: 2.0197250843048096 (nan) | > loss: 0.008175193332135677 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13682525095747147) | > loader_time: 0.0045 (0.004677065571276978)  --> TIME: 2025-02-26 19:59:38 -- STEP: 16844/36938 -- GLOBAL_STEP: 1383550 | > loss_text_ce: 0.020567016676068306 (0.03081381026141449) | > loss_mel_ce: 2.7991504669189453 (nan) | > loss: 0.0111893555149436 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13682565263602345) | > loader_time: 0.0046 (0.004676859131393867)  --> TIME: 2025-02-26 19:59:48 -- STEP: 16894/36938 -- GLOBAL_STEP: 1383600 | > loss_text_ce: 0.03580566495656967 (0.03081436919973907) | > loss_mel_ce: 3.121253490447998 (nan) | > loss: 0.012528013437986374 (nan) | > current_lr: 5e-06 | > step_time: 0.1516 (0.13683016570881465) | > loader_time: 0.0052 (0.004677193170752757)  --> TIME: 2025-02-26 19:59:57 -- STEP: 16944/36938 -- GLOBAL_STEP: 1383650 | > loss_text_ce: 0.02506488747894764 (0.030808061454285718) | > loss_mel_ce: 3.5346028804779053 (nan) | > loss: 0.014125666581094265 (nan) | > current_lr: 5e-06 | > step_time: 0.1443 (0.13683059787671473) | > loader_time: 0.0046 (0.004677047754257119)  --> TIME: 2025-02-26 20:00:07 -- STEP: 16994/36938 -- GLOBAL_STEP: 1383700 | > loss_text_ce: 0.029654061421751976 (0.030810352932269157) | > loss_mel_ce: 1.7275516986846924 (nan) | > loss: 0.0069730388931930065 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13683348322414154) | > loader_time: 0.0047 (0.004677323940376335)  --> TIME: 2025-02-26 20:00:16 -- STEP: 17044/36938 -- GLOBAL_STEP: 1383750 | > loss_text_ce: 0.023924600332975388 (0.030807970736779676) | > loss_mel_ce: 2.683197498321533 (nan) | > loss: 0.01074254885315895 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1368396977560256) | > loader_time: 0.0047 (0.004677799519617455)  --> TIME: 2025-02-26 20:00:25 -- STEP: 17094/36938 -- GLOBAL_STEP: 1383800 | > loss_text_ce: 0.027585050091147423 (0.030808096730894616) | > loss_mel_ce: 2.3207578659057617 (nan) | > loss: 0.009318821132183075 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13682664364028926) | > loader_time: 0.0047 (0.004677880894054083)  --> TIME: 2025-02-26 20:00:35 -- STEP: 17144/36938 -- GLOBAL_STEP: 1383850 | > loss_text_ce: 0.029678791761398315 (0.03080850374597314) | > loss_mel_ce: 2.3992154598236084 (nan) | > loss: 0.0096384696662426 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13683082293362717) | > loader_time: 0.0046 (0.004677938611163655)  --> TIME: 2025-02-26 20:00:44 -- STEP: 17194/36938 -- GLOBAL_STEP: 1383900 | > loss_text_ce: 0.03499593958258629 (0.030807478633010834) | > loss_mel_ce: 0.0470733642578125 (nan) | > loss: 0.00032567186281085014 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13683070803126798) | > loader_time: 0.0045 (0.004678130038910266)  --> TIME: 2025-02-26 20:00:53 -- STEP: 17244/36938 -- GLOBAL_STEP: 1383950 | > loss_text_ce: 0.037104640156030655 (0.030808490317046328) | > loss_mel_ce: 0.079742431640625 (nan) | > loss: 0.00046367887989617884 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13683504813288752) | > loader_time: 0.0047 (0.004678086749474833)  --> TIME: 2025-02-26 20:01:02 -- STEP: 17294/36938 -- GLOBAL_STEP: 1384000 | > loss_text_ce: 0.025464724749326706 (0.03080571515965899) | > loss_mel_ce: 2.7900390625 (nan) | > loss: 0.011172634549438953 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13682572585496666) | > loader_time: 0.0047 (0.004678177174576688)  --> TIME: 2025-02-26 20:01:12 -- STEP: 17344/36938 -- GLOBAL_STEP: 1384050 | > loss_text_ce: 0.025846023112535477 (0.030805674000331528) | > loss_mel_ce: 2.8941357135772705 (nan) | > loss: 0.011587229557335377 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13682339997529114) | > loader_time: 0.0048 (0.004678210539131608)  --> TIME: 2025-02-26 20:01:21 -- STEP: 17394/36938 -- GLOBAL_STEP: 1384100 | > loss_text_ce: 0.03424755856394768 (0.030806869481758436) | > loss_mel_ce: 2.6326730251312256 (nan) | > loss: 0.010583018884062767 (nan) | > current_lr: 5e-06 | > step_time: 0.1566 (0.13682110026963665) | > loader_time: 0.0047 (0.004678425301460499)  --> TIME: 2025-02-26 20:01:30 -- STEP: 17444/36938 -- GLOBAL_STEP: 1384150 | > loss_text_ce: 0.030186917632818222 (0.03080835084356021) | > loss_mel_ce: 2.480318069458008 (nan) | > loss: 0.009962322190403938 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13683122831100566) | > loader_time: 0.0045 (0.004678276489426653)  --> TIME: 2025-02-26 20:01:40 -- STEP: 17494/36938 -- GLOBAL_STEP: 1384200 | > loss_text_ce: 0.03126777336001396 (0.03080553543963461) | > loss_mel_ce: 1.7631356716156006 (nan) | > loss: 0.007120648864656687 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13682211332625366) | > loader_time: 0.0046 (0.0046780870698637025)  --> TIME: 2025-02-26 20:01:49 -- STEP: 17544/36938 -- GLOBAL_STEP: 1384250 | > loss_text_ce: 0.030552934855222702 (0.03080732798117509) | > loss_mel_ce: 2.918511152267456 (nan) | > loss: 0.011702635325491428 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.13683567386047513) | > loader_time: 0.0047 (0.004678017667478165)  --> TIME: 2025-02-26 20:01:59 -- STEP: 17594/36938 -- GLOBAL_STEP: 1384300 | > loss_text_ce: 0.03866102546453476 (0.030810311405289677) | > loss_mel_ce: 2.9337480068206787 (nan) | > loss: 0.011795274913311005 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13683197080133666) | > loader_time: 0.0045 (0.004677953307596395)  --> TIME: 2025-02-26 20:02:08 -- STEP: 17644/36938 -- GLOBAL_STEP: 1384350 | > loss_text_ce: 0.03561526909470558 (0.030808433703196695) | > loss_mel_ce: 2.38944149017334 (nan) | > loss: 0.009623241610825062 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13682076865327902) | > loader_time: 0.0044 (0.0046777711031361895)  --> TIME: 2025-02-26 20:02:17 -- STEP: 17694/36938 -- GLOBAL_STEP: 1384400 | > loss_text_ce: 0.027683177962899208 (0.03080631036905004) | > loss_mel_ce: 3.25696063041687 (nan) | > loss: 0.013034301809966564 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.1368131195069221) | > loader_time: 0.0046 (0.004677604790850668)  --> TIME: 2025-02-26 20:02:26 -- STEP: 17744/36938 -- GLOBAL_STEP: 1384450 | > loss_text_ce: 0.03113226592540741 (0.03080257893989985) | > loss_mel_ce: 3.191573143005371 (nan) | > loss: 0.01278851367533207 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.1368114250914484) | > loader_time: 0.0047 (0.004677479927124915)  --> TIME: 2025-02-26 20:02:36 -- STEP: 17794/36938 -- GLOBAL_STEP: 1384500 | > loss_text_ce: 0.029741108417510986 (0.03079632001313895) | > loss_mel_ce: 2.734872341156006 (nan) | > loss: 0.010970688425004482 (nan) | > current_lr: 5e-06 | > step_time: 0.1528 (0.13683170197424585) | > loader_time: 0.0047 (0.004677546832271284)  --> TIME: 2025-02-26 20:02:45 -- STEP: 17844/36938 -- GLOBAL_STEP: 1384550 | > loss_text_ce: 0.024250302463769913 (0.030797273534261572) | > loss_mel_ce: 0.0272216796875 (nan) | > loss: 0.00020425391267053783 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13683991093080786) | > loader_time: 0.0045 (0.004677665444720205)  --> TIME: 2025-02-26 20:02:55 -- STEP: 17894/36938 -- GLOBAL_STEP: 1384600 | > loss_text_ce: 0.028554756194353104 (0.030798553263448356) | > loss_mel_ce: 2.955686330795288 (nan) | > loss: 0.01184222660958767 (nan) | > current_lr: 5e-06 | > step_time: 0.1506 (0.13683985620553862) | > loader_time: 0.0046 (0.004677485630900506)  --> TIME: 2025-02-26 20:03:04 -- STEP: 17944/36938 -- GLOBAL_STEP: 1384650 | > loss_text_ce: 0.021978186443448067 (0.03079853411030536) | > loss_mel_ce: 2.7400383949279785 (nan) | > loss: 0.010960383340716362 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13684665165572507) | > loader_time: 0.0045 (0.004677251333431436)  --> TIME: 2025-02-26 20:03:13 -- STEP: 17994/36938 -- GLOBAL_STEP: 1384700 | > loss_text_ce: 0.022094108164310455 (0.030798859457417362) | > loss_mel_ce: 2.7176952362060547 (nan) | > loss: 0.010872180573642254 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.1368417812610076) | > loader_time: 0.0046 (0.004677026632482172)  --> TIME: 2025-02-26 20:03:23 -- STEP: 18044/36938 -- GLOBAL_STEP: 1384750 | > loss_text_ce: 0.03360036388039589 (0.03080025067638293) | > loss_mel_ce: 3.442380905151367 (nan) | > loss: 0.013793577440083027 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.1368608232533895) | > loader_time: 0.0046 (0.004676943157232417)  --> TIME: 2025-02-26 20:03:32 -- STEP: 18094/36938 -- GLOBAL_STEP: 1384800 | > loss_text_ce: 0.040703147649765015 (0.030801762333001827) | > loss_mel_ce: 2.57452130317688 (nan) | > loss: 0.010377874597907066 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13685413505591706) | > loader_time: 0.0045 (0.0046768070940709505)  --> TIME: 2025-02-26 20:03:42 -- STEP: 18144/36938 -- GLOBAL_STEP: 1384850 | > loss_text_ce: 0.028085073456168175 (0.030795687579091367) | > loss_mel_ce: 2.1572558879852295 (nan) | > loss: 0.008671987801790237 (nan) | > current_lr: 5e-06 | > step_time: 0.1081 (0.13688088083603583) | > loader_time: 0.0047 (0.004676870607509821)  --> TIME: 2025-02-26 20:03:52 -- STEP: 18194/36938 -- GLOBAL_STEP: 1384900 | > loss_text_ce: 0.025136156007647514 (0.030797693741713497) | > loss_mel_ce: 2.6139438152313232 (nan) | > loss: 0.010472540743649006 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.1368898101543296) | > loader_time: 0.0047 (0.004676812020346168)  --> TIME: 2025-02-26 20:04:01 -- STEP: 18244/36938 -- GLOBAL_STEP: 1384950 | > loss_text_ce: 0.026117287576198578 (0.030797450795435423) | > loss_mel_ce: 2.5197792053222656 (nan) | > loss: 0.010102764703333378 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1368987684421332) | > loader_time: 0.0047 (0.00467673854277966)  --> TIME: 2025-02-26 20:04:11 -- STEP: 18294/36938 -- GLOBAL_STEP: 1385000 | > loss_text_ce: 0.02916184440255165 (0.030796310253561675) | > loss_mel_ce: 2.790604591369629 (nan) | > loss: 0.011189550161361694 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.1369091064093007) | > loader_time: 0.0046 (0.004676649723467941)  --> TIME: 2025-02-26 20:04:21 -- STEP: 18344/36938 -- GLOBAL_STEP: 1385050 | > loss_text_ce: 0.030998609960079193 (0.030796541950249706) | > loss_mel_ce: 2.7328054904937744 (nan) | > loss: 0.010967477224767208 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13693118420325134) | > loader_time: 0.0045 (0.0046765959605971075)  --> TIME: 2025-02-26 20:04:30 -- STEP: 18394/36938 -- GLOBAL_STEP: 1385100 | > loss_text_ce: 0.030309034511446953 (0.030795587914810227) | > loss_mel_ce: 2.3521242141723633 (nan) | > loss: 0.009454100392758846 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.1369297923867541) | > loader_time: 0.0045 (0.004676390759670391)  --> TIME: 2025-02-26 20:04:39 -- STEP: 18444/36938 -- GLOBAL_STEP: 1385150 | > loss_text_ce: 0.032841846346855164 (0.030791639175315102) | > loss_mel_ce: 2.1789779663085938 (nan) | > loss: 0.008777063339948654 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13692067930326626) | > loader_time: 0.0045 (0.004676187511535923)  --> TIME: 2025-02-26 20:04:49 -- STEP: 18494/36938 -- GLOBAL_STEP: 1385200 | > loss_text_ce: 0.027670055627822876 (0.030794000028559757) | > loss_mel_ce: 2.4198265075683594 (nan) | > loss: 0.009712289087474346 (nan) | > current_lr: 5e-06 | > step_time: 0.1873 (0.1369308994127581) | > loader_time: 0.0048 (0.004676017784951146)  --> TIME: 2025-02-26 20:04:58 -- STEP: 18544/36938 -- GLOBAL_STEP: 1385250 | > loss_text_ce: 0.028942443430423737 (0.030794901946173635) | > loss_mel_ce: 2.588791608810425 (nan) | > loss: 0.010387834161520004 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13693393846592905) | > loader_time: 0.0045 (0.004675834663887288)  --> TIME: 2025-02-26 20:05:07 -- STEP: 18594/36938 -- GLOBAL_STEP: 1385300 | > loss_text_ce: 0.02738300897181034 (0.030795237251537468) | > loss_mel_ce: 1.551998257637024 (nan) | > loss: 0.006267386022955179 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13691724186573462) | > loader_time: 0.0044 (0.004675618920311361)  --> TIME: 2025-02-26 20:05:16 -- STEP: 18644/36938 -- GLOBAL_STEP: 1385350 | > loss_text_ce: 0.033523548394441605 (0.03079421678870197) | > loss_mel_ce: 1.4562187194824219 (nan) | > loss: 0.005911676213145256 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13691672798493137) | > loader_time: 0.0046 (0.004675370190077602)  --> TIME: 2025-02-26 20:05:26 -- STEP: 18694/36938 -- GLOBAL_STEP: 1385400 | > loss_text_ce: 0.030893675982952118 (0.03079327291628078) | > loss_mel_ce: 0.6897159218788147 (nan) | > loss: 0.0028595621697604656 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13693331469440653) | > loader_time: 0.0047 (0.004675300947495219)  --> TIME: 2025-02-26 20:05:35 -- STEP: 18744/36938 -- GLOBAL_STEP: 1385450 | > loss_text_ce: 0.02587282285094261 (0.030793513772903642) | > loss_mel_ce: 2.509071111679077 (nan) | > loss: 0.010059301741421223 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13693775588307092) | > loader_time: 0.0046 (0.004675061515493879)  --> TIME: 2025-02-26 20:05:45 -- STEP: 18794/36938 -- GLOBAL_STEP: 1385500 | > loss_text_ce: 0.03344201669096947 (0.0307932465177999) | > loss_mel_ce: 2.894155502319336 (nan) | > loss: 0.011617450974881649 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.1369368633945358) | > loader_time: 0.0045 (0.004674874532345805)  --> TIME: 2025-02-26 20:05:54 -- STEP: 18844/36938 -- GLOBAL_STEP: 1385550 | > loss_text_ce: 0.029923025518655777 (0.03079438225005487) | > loss_mel_ce: 2.4690794944763184 (nan) | > loss: 0.009916677139699459 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13693412473515046) | > loader_time: 0.0047 (0.004674688781858939)  --> TIME: 2025-02-26 20:06:03 -- STEP: 18894/36938 -- GLOBAL_STEP: 1385600 | > loss_text_ce: 0.0378570631146431 (0.03079475246421989) | > loss_mel_ce: 2.9627747535705566 (nan) | > loss: 0.011907270178198814 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13692076441925452) | > loader_time: 0.0046 (0.004674475698025888)  --> TIME: 2025-02-26 20:06:12 -- STEP: 18944/36938 -- GLOBAL_STEP: 1385650 | > loss_text_ce: 0.03327349200844765 (0.030795975713551117) | > loss_mel_ce: 2.99267315864563 (nan) | > loss: 0.012007725425064564 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13691564120755004) | > loader_time: 0.0045 (0.004674260756252592)  --> TIME: 2025-02-26 20:06:22 -- STEP: 18994/36938 -- GLOBAL_STEP: 1385700 | > loss_text_ce: 0.02112027443945408 (0.030796899700049347) | > loss_mel_ce: 2.7623634338378906 (nan) | > loss: 0.01104557141661644 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13691473638583357) | > loader_time: 0.0045 (0.004674052883351847)  --> TIME: 2025-02-26 20:06:31 -- STEP: 19044/36938 -- GLOBAL_STEP: 1385750 | > loss_text_ce: 0.03765642270445824 (0.030797378567647558) | > loss_mel_ce: 2.3282058238983154 (nan) | > loss: 0.009388342499732971 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13691178990171576) | > loader_time: 0.0045 (0.004673817708096173)  --> TIME: 2025-02-26 20:06:40 -- STEP: 19094/36938 -- GLOBAL_STEP: 1385800 | > loss_text_ce: 0.03623146936297417 (0.030795366598435216) | > loss_mel_ce: 2.649751901626587 (nan) | > loss: 0.010658664628863335 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1369141718473961) | > loader_time: 0.0046 (0.004673774446931155)  --> TIME: 2025-02-26 20:06:50 -- STEP: 19144/36938 -- GLOBAL_STEP: 1385850 | > loss_text_ce: 0.021239586174488068 (0.030795388295949768) | > loss_mel_ce: 2.2346885204315186 (nan) | > loss: 0.008952096104621887 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.13691592314877982) | > loader_time: 0.0046 (0.004673598042303586)  --> TIME: 2025-02-26 20:06:59 -- STEP: 19194/36938 -- GLOBAL_STEP: 1385900 | > loss_text_ce: 0.03812564164400101 (0.030792638558431108) | > loss_mel_ce: 1.9995535612106323 (nan) | > loss: 0.008086029440164566 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.1369214613412562) | > loader_time: 0.0046 (0.004673482925305339)  --> TIME: 2025-02-26 20:07:08 -- STEP: 19244/36938 -- GLOBAL_STEP: 1385950 | > loss_text_ce: 0.03448142111301422 (0.030795061827414243) | > loss_mel_ce: 2.989889144897461 (nan) | > loss: 0.01200147159397602 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.1369173459862008) | > loader_time: 0.0046 (0.0046733330723848394)  --> TIME: 2025-02-26 20:07:18 -- STEP: 19294/36938 -- GLOBAL_STEP: 1386000 | > loss_text_ce: 0.03105049580335617 (0.030794828989779248) | > loss_mel_ce: 2.3585219383239746 (nan) | > loss: 0.009482430294156075 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1369196870505681) | > loader_time: 0.0045 (0.004673221524767379)  --> TIME: 2025-02-26 20:07:27 -- STEP: 19344/36938 -- GLOBAL_STEP: 1386050 | > loss_text_ce: 0.03239698335528374 (0.030795246200701308) | > loss_mel_ce: 3.027822494506836 (nan) | > loss: 0.012143729254603386 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.13690493745780175) | > loader_time: 0.0046 (0.00467312105486865)  --> TIME: 2025-02-26 20:07:36 -- STEP: 19394/36938 -- GLOBAL_STEP: 1386100 | > loss_text_ce: 0.026222538203001022 (0.030793563668972717) | > loss_mel_ce: 1.5213401317596436 (nan) | > loss: 0.006141122430562973 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13690471945441346) | > loader_time: 0.0044 (0.004672869303871391)  --> TIME: 2025-02-26 20:07:45 -- STEP: 19444/36938 -- GLOBAL_STEP: 1386150 | > loss_text_ce: 0.037519410252571106 (0.030796012554808756) | > loss_mel_ce: 1.6034660339355469 (nan) | > loss: 0.006511847488582134 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.1369022276391221) | > loader_time: 0.0045 (0.00467258694240082)  --> TIME: 2025-02-26 20:07:54 -- STEP: 19494/36938 -- GLOBAL_STEP: 1386200 | > loss_text_ce: 0.03581796959042549 (0.03079778751037056) | > loss_mel_ce: 2.3058156967163086 (nan) | > loss: 0.009292197413742542 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.13689057821125303) | > loader_time: 0.0046 (0.00467239438148234)  --> TIME: 2025-02-26 20:08:04 -- STEP: 19544/36938 -- GLOBAL_STEP: 1386250 | > loss_text_ce: 0.033531852066516876 (0.030791790122992567) | > loss_mel_ce: 3.2761013507843018 (nan) | > loss: 0.013133466243743896 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.1368986389527692) | > loader_time: 0.0045 (0.004672256713512737)  --> TIME: 2025-02-26 20:08:13 -- STEP: 19594/36938 -- GLOBAL_STEP: 1386300 | > loss_text_ce: 0.025871744379401207 (0.030792873546003345) | > loss_mel_ce: 2.7944295406341553 (nan) | > loss: 0.011191672645509243 (nan) | > current_lr: 5e-06 | > step_time: 0.1769 (0.1369066611959023) | > loader_time: 0.0047 (0.004672053384211426)  --> TIME: 2025-02-26 20:08:23 -- STEP: 19644/36938 -- GLOBAL_STEP: 1386350 | > loss_text_ce: 0.02980194054543972 (0.030791827849280466) | > loss_mel_ce: 3.0584542751312256 (nan) | > loss: 0.012254985049366951 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.1369174946667685) | > loader_time: 0.0048 (0.004671875740160791)  --> TIME: 2025-02-26 20:08:32 -- STEP: 19694/36938 -- GLOBAL_STEP: 1386400 | > loss_text_ce: 0.037831418216228485 (0.030792181435965824) | > loss_mel_ce: 2.0240488052368164 (nan) | > loss: 0.00818206463009119 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13692037169674529) | > loader_time: 0.0046 (0.004671724820555891)  --> TIME: 2025-02-26 20:08:42 -- STEP: 19744/36938 -- GLOBAL_STEP: 1386450 | > loss_text_ce: 0.03338876739144325 (0.03079412882840332) | > loss_mel_ce: 3.254775285720825 (nan) | > loss: 0.013048270717263222 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13692207165919826) | > loader_time: 0.0045 (0.004671532292420121)  --> TIME: 2025-02-26 20:08:51 -- STEP: 19794/36938 -- GLOBAL_STEP: 1386500 | > loss_text_ce: 0.02906973846256733 (0.03079459338151204) | > loss_mel_ce: 2.683513641357422 (nan) | > loss: 0.010764219798147678 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13691382675781863) | > loader_time: 0.0047 (0.004671415090681359)  --> TIME: 2025-02-26 20:09:00 -- STEP: 19844/36938 -- GLOBAL_STEP: 1386550 | > loss_text_ce: 0.02189846895635128 (0.03079262155904968) | > loss_mel_ce: 2.1931135654449463 (nan) | > loss: 0.008789731189608574 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13691733935215275) | > loader_time: 0.0046 (0.004671296208790532)  --> TIME: 2025-02-26 20:09:09 -- STEP: 19894/36938 -- GLOBAL_STEP: 1386600 | > loss_text_ce: 0.03390251472592354 (0.030793219345399015) | > loss_mel_ce: 0.8660888671875 (nan) | > loss: 0.0035713945981115103 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13690385150228745) | > loader_time: 0.0045 (0.004671075769054165)  --> TIME: 2025-02-26 20:09:19 -- STEP: 19944/36938 -- GLOBAL_STEP: 1386650 | > loss_text_ce: 0.025991098955273628 (0.030793027649320712) | > loss_mel_ce: 2.575329065322876 (nan) | > loss: 0.010322700254619122 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1369074948977815) | > loader_time: 0.0045 (0.004670876410415838)  --> TIME: 2025-02-26 20:09:28 -- STEP: 19994/36938 -- GLOBAL_STEP: 1386700 | > loss_text_ce: 0.02011967822909355 (0.030792771289402696) | > loss_mel_ce: 4.0078125 (nan) | > loss: 0.015983859077095985 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13691504300684088) | > loader_time: 0.0046 (0.004670712200655889)  --> TIME: 2025-02-26 20:09:37 -- STEP: 20044/36938 -- GLOBAL_STEP: 1386750 | > loss_text_ce: 0.03495655581355095 (0.030795047146984098) | > loss_mel_ce: 3.49167799949646 (nan) | > loss: 0.013994581997394562 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13690914073596608) | > loader_time: 0.0046 (0.004670525460727576)  --> TIME: 2025-02-26 20:09:47 -- STEP: 20094/36938 -- GLOBAL_STEP: 1386800 | > loss_text_ce: 0.038029249757528305 (0.030795583441274907) | > loss_mel_ce: 2.7596042156219482 (nan) | > loss: 0.011101720854640007 (nan) | > current_lr: 5e-06 | > step_time: 0.1873 (0.13691799142556024) | > loader_time: 0.0048 (0.004670408551130784)  --> TIME: 2025-02-26 20:09:56 -- STEP: 20144/36938 -- GLOBAL_STEP: 1386850 | > loss_text_ce: 0.036872897297143936 (0.03079652561891858) | > loss_mel_ce: 3.099090099334717 (nan) | > loss: 0.012444297783076763 (nan) | > current_lr: 5e-06 | > step_time: 0.1445 (0.13691955026035021) | > loader_time: 0.0046 (0.004670307916057209)  --> TIME: 2025-02-26 20:10:06 -- STEP: 20194/36938 -- GLOBAL_STEP: 1386900 | > loss_text_ce: 0.025391293689608574 (0.03079610376062056) | > loss_mel_ce: 2.9462063312530518 (nan) | > loss: 0.011792055331170559 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13691267885051556) | > loader_time: 0.0045 (0.004670341545914234)  --> TIME: 2025-02-26 20:10:15 -- STEP: 20244/36938 -- GLOBAL_STEP: 1386950 | > loss_text_ce: 0.03376571461558342 (0.03079442088376742) | > loss_mel_ce: 3.3100531101226807 (nan) | > loss: 0.013269123621284962 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13693135126474223) | > loader_time: 0.0046 (0.0046703063365240525)  --> TIME: 2025-02-26 20:10:25 -- STEP: 20294/36938 -- GLOBAL_STEP: 1387000 | > loss_text_ce: 0.04575555771589279 (0.030791186706358574) | > loss_mel_ce: 2.123994827270508 (nan) | > loss: 0.008610120974481106 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13694022691483673) | > loader_time: 0.0045 (0.0046702446086519245)  --> TIME: 2025-02-26 20:10:34 -- STEP: 20344/36938 -- GLOBAL_STEP: 1387050 | > loss_text_ce: 0.026446504518389702 (0.030792572018434956) | > loss_mel_ce: 2.52502179145813 (nan) | > loss: 0.010124875232577324 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13693505872481962) | > loader_time: 0.0045 (0.004670042434734386)  --> TIME: 2025-02-26 20:10:43 -- STEP: 20394/36938 -- GLOBAL_STEP: 1387100 | > loss_text_ce: 0.03042660467326641 (0.030790620695601758) | > loss_mel_ce: 2.757723331451416 (nan) | > loss: 0.01106408704072237 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13693613504280944) | > loader_time: 0.0045 (0.004669933783443365)  --> TIME: 2025-02-26 20:10:53 -- STEP: 20444/36938 -- GLOBAL_STEP: 1387150 | > loss_text_ce: 0.027064071968197823 (0.030791543681946937) | > loss_mel_ce: 2.4031739234924316 (nan) | > loss: 0.009643802419304848 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13693563101040254) | > loader_time: 0.0045 (0.004669799074177821)  --> TIME: 2025-02-26 20:11:02 -- STEP: 20494/36938 -- GLOBAL_STEP: 1387200 | > loss_text_ce: 0.02900933288037777 (0.03079348429762564) | > loss_mel_ce: 2.4391722679138184 (nan) | > loss: 0.009794372133910656 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13692964505390817) | > loader_time: 0.0046 (0.004669563577409437)  --> TIME: 2025-02-26 20:11:11 -- STEP: 20544/36938 -- GLOBAL_STEP: 1387250 | > loss_text_ce: 0.02412603795528412 (0.030793252040180002) | > loss_mel_ce: 2.7781291007995605 (nan) | > loss: 0.011120060458779335 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.1369315057621571) | > loader_time: 0.0046 (0.004669383005746794)  --> TIME: 2025-02-26 20:11:21 -- STEP: 20594/36938 -- GLOBAL_STEP: 1387300 | > loss_text_ce: 0.03404736518859863 (0.030791387052089007) | > loss_mel_ce: 0.008677165023982525 (nan) | > loss: 0.000169541803188622 (nan) | > current_lr: 5e-06 | > step_time: 0.1571 (0.13694270157033772) | > loader_time: 0.0046 (0.0046692240338910345)  --> TIME: 2025-02-26 20:11:30 -- STEP: 20644/36938 -- GLOBAL_STEP: 1387350 | > loss_text_ce: 0.021661121398210526 (0.030794188024544197) | > loss_mel_ce: 2.199462890625 (nan) | > loss: 0.00881398469209671 (nan) | > current_lr: 5e-06 | > step_time: 0.1116 (0.13694277049450704) | > loader_time: 0.0045 (0.00466908560407136)  --> TIME: 2025-02-26 20:11:40 -- STEP: 20694/36938 -- GLOBAL_STEP: 1387400 | > loss_text_ce: 0.04381499066948891 (0.030795647182675655) | > loss_mel_ce: 2.632878541946411 (nan) | > loss: 0.01062180008739233 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13693944984119766) | > loader_time: 0.0046 (0.00466894950223358)  --> TIME: 2025-02-26 20:11:49 -- STEP: 20744/36938 -- GLOBAL_STEP: 1387450 | > loss_text_ce: 0.03654729574918747 (0.03079713756023075) | > loss_mel_ce: 2.1563100814819336 (nan) | > loss: 0.008701815269887447 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13692465972771703) | > loader_time: 0.0045 (0.004668762865004001)  --> TIME: 2025-02-26 20:11:58 -- STEP: 20794/36938 -- GLOBAL_STEP: 1387500 | > loss_text_ce: 0.025417570024728775 (0.03079953949772268) | > loss_mel_ce: 2.579339027404785 (nan) | > loss: 0.010336336679756641 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.1369221253705577) | > loader_time: 0.0045 (0.004668619984260505)  --> TIME: 2025-02-26 20:12:07 -- STEP: 20844/36938 -- GLOBAL_STEP: 1387550 | > loss_text_ce: 0.031439028680324554 (0.03079802055845944) | > loss_mel_ce: 2.1977906227111816 (nan) | > loss: 0.008846149779856205 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.13691823001192524) | > loader_time: 0.0046 (0.0046684796991789245)  --> TIME: 2025-02-26 20:12:16 -- STEP: 20894/36938 -- GLOBAL_STEP: 1387600 | > loss_text_ce: 0.032806284725666046 (0.0307961621937434) | > loss_mel_ce: 0.5355393886566162 (nan) | > loss: 0.0022553401067852974 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13691813623755753) | > loader_time: 0.0045 (0.00466837359921628)  --> TIME: 2025-02-26 20:12:26 -- STEP: 20944/36938 -- GLOBAL_STEP: 1387650 | > loss_text_ce: 0.02787976898252964 (0.030794657373706066) | > loss_mel_ce: 1.8131103515625 (nan) | > loss: 0.007305516395717859 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.13692142998243362) | > loader_time: 0.0047 (0.004668513015538699)  --> TIME: 2025-02-26 20:12:35 -- STEP: 20994/36938 -- GLOBAL_STEP: 1387700 | > loss_text_ce: 0.02599082700908184 (0.030798240361344483) | > loss_mel_ce: 3.1643929481506348 (nan) | > loss: 0.012660253793001175 (nan) | > current_lr: 5e-06 | > step_time: 0.1831 (0.13691964840632365) | > loader_time: 0.0048 (0.00466837229541997)  --> TIME: 2025-02-26 20:12:44 -- STEP: 21044/36938 -- GLOBAL_STEP: 1387750 | > loss_text_ce: 0.024666709825396538 (0.030797720266851444) | > loss_mel_ce: 2.603438138961792 (nan) | > loss: 0.010428988374769688 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.1369189492334234) | > loader_time: 0.0046 (0.004668621594541943)  --> TIME: 2025-02-26 20:12:54 -- STEP: 21094/36938 -- GLOBAL_STEP: 1387800 | > loss_text_ce: 0.032866187393665314 (0.03079770294962392) | > loss_mel_ce: 2.849696159362793 (nan) | > loss: 0.01143874041736126 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.13692802260292125) | > loader_time: 0.0054 (0.004668658453558946)  --> TIME: 2025-02-26 20:13:03 -- STEP: 21144/36938 -- GLOBAL_STEP: 1387850 | > loss_text_ce: 0.029093792662024498 (0.030798976389343966) | > loss_mel_ce: 3.3948142528533936 (nan) | > loss: 0.013586937449872494 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13691746135394922) | > loader_time: 0.0047 (0.004668831949291732)  --> TIME: 2025-02-26 20:13:12 -- STEP: 21194/36938 -- GLOBAL_STEP: 1387900 | > loss_text_ce: 0.03618753328919411 (0.0307974737979543) | > loss_mel_ce: 1.0697460174560547 (nan) | > loss: 0.004388625267893076 (nan) | > current_lr: 5e-06 | > step_time: 0.1858 (0.13691298752626677) | > loader_time: 0.0048 (0.004668953498152044)  --> TIME: 2025-02-26 20:13:22 -- STEP: 21244/36938 -- GLOBAL_STEP: 1387950 | > loss_text_ce: 0.02626463584601879 (0.0307965601618056) | > loss_mel_ce: 2.969316005706787 (nan) | > loss: 0.011887225322425365 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.13691285558394287) | > loader_time: 0.0048 (0.004669009045543603)  --> TIME: 2025-02-26 20:13:31 -- STEP: 21294/36938 -- GLOBAL_STEP: 1388000 | > loss_text_ce: 0.026010662317276 (0.0308020927215647) | > loss_mel_ce: 3.3225739002227783 (nan) | > loss: 0.013288035057485104 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13691354411481652) | > loader_time: 0.0048 (0.004669197346667691)  --> TIME: 2025-02-26 20:13:40 -- STEP: 21344/36938 -- GLOBAL_STEP: 1388050 | > loss_text_ce: 0.03489705175161362 (0.030802835643695868) | > loss_mel_ce: 2.52821683883667 (nan) | > loss: 0.010171087458729744 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13691832234655785) | > loader_time: 0.0046 (0.004669333080659197)  --> TIME: 2025-02-26 20:13:50 -- STEP: 21394/36938 -- GLOBAL_STEP: 1388100 | > loss_text_ce: 0.03551625460386276 (0.030802379360130302) | > loss_mel_ce: 2.234910488128662 (nan) | > loss: 0.009009630419313908 (nan) | > current_lr: 5e-06 | > step_time: 0.1522 (0.1369224525427417) | > loader_time: 0.0046 (0.004669337481263073)  --> TIME: 2025-02-26 20:13:59 -- STEP: 21444/36938 -- GLOBAL_STEP: 1388150 | > loss_text_ce: 0.024659130722284317 (0.03080003566329357) | > loss_mel_ce: 2.8437845706939697 (nan) | > loss: 0.011382713913917542 (nan) | > current_lr: 5e-06 | > step_time: 0.1819 (0.136927774272244) | > loader_time: 0.0048 (0.004669293052468393)  --> TIME: 2025-02-26 20:14:09 -- STEP: 21494/36938 -- GLOBAL_STEP: 1388200 | > loss_text_ce: 0.02244310826063156 (0.03080084850669591) | > loss_mel_ce: 2.740453004837036 (nan) | > loss: 0.01096387393772602 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.1369288113546053) | > loader_time: 0.0046 (0.004669424122495946)  --> TIME: 2025-02-26 20:14:18 -- STEP: 21544/36938 -- GLOBAL_STEP: 1388250 | > loss_text_ce: 0.02283615618944168 (0.030798347269910608) | > loss_mel_ce: 2.8458433151245117 (nan) | > loss: 0.01138364989310503 (nan) | > current_lr: 5e-06 | > step_time: 0.1704 (0.13692276303183004) | > loader_time: 0.0047 (0.0046693160327623716)  --> TIME: 2025-02-26 20:14:27 -- STEP: 21594/36938 -- GLOBAL_STEP: 1388300 | > loss_text_ce: 0.03903517499566078 (0.030798778136432368) | > loss_mel_ce: 2.0272936820983887 (nan) | > loss: 0.008199717849493027 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13692028824793104) | > loader_time: 0.0046 (0.00466924314533348)  --> TIME: 2025-02-26 20:14:37 -- STEP: 21644/36938 -- GLOBAL_STEP: 1388350 | > loss_text_ce: 0.03289702907204628 (0.030799715127894898) | > loss_mel_ce: 2.336106538772583 (nan) | > loss: 0.0094008082523942 (nan) | > current_lr: 5e-06 | > step_time: 0.1871 (0.13692137047667763) | > loader_time: 0.0049 (0.0046694255804887695)  --> TIME: 2025-02-26 20:14:46 -- STEP: 21694/36938 -- GLOBAL_STEP: 1388400 | > loss_text_ce: 0.018283052369952202 (0.030797866565833294) | > loss_mel_ce: 2.4631454944610596 (nan) | > loss: 0.009846939705312252 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1369137510655942) | > loader_time: 0.0051 (0.00466963773808127)  --> TIME: 2025-02-26 20:14:55 -- STEP: 21744/36938 -- GLOBAL_STEP: 1388450 | > loss_text_ce: 0.03268711641430855 (0.030800150924454237) | > loss_mel_ce: 2.0763509273529053 (nan) | > loss: 0.008369199000298977 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13691740880519582) | > loader_time: 0.0045 (0.0046696507312340855)  --> TIME: 2025-02-26 20:15:05 -- STEP: 21794/36938 -- GLOBAL_STEP: 1388500 | > loss_text_ce: 0.04301329702138901 (0.03079899060029669) | > loss_mel_ce: 2.908184766769409 (nan) | > loss: 0.011711103841662407 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13691791630867764) | > loader_time: 0.0047 (0.0046698167213138075)  --> TIME: 2025-02-26 20:15:14 -- STEP: 21844/36938 -- GLOBAL_STEP: 1388550 | > loss_text_ce: 0.029753508046269417 (0.030796118550221033) | > loss_mel_ce: 3.1511876583099365 (nan) | > loss: 0.012622782960534096 (nan) | > current_lr: 5e-06 | > step_time: 0.1718 (0.13692566549967933) | > loader_time: 0.0048 (0.0046701052755995925)  --> TIME: 2025-02-26 20:15:23 -- STEP: 21894/36938 -- GLOBAL_STEP: 1388600 | > loss_text_ce: 0.015354979783296585 (0.030793540059113528) | > loss_mel_ce: 0.1355794221162796 (nan) | > loss: 0.0005989460623823106 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1369272455882079) | > loader_time: 0.0047 (0.004670123079546805)  --> TIME: 2025-02-26 20:15:33 -- STEP: 21944/36938 -- GLOBAL_STEP: 1388650 | > loss_text_ce: 0.02899871952831745 (0.03079057777030298) | > loss_mel_ce: 3.236973524093628 (nan) | > loss: 0.012960207648575306 (nan) | > current_lr: 5e-06 | > step_time: 0.2021 (0.1369415394828743) | > loader_time: 0.0048 (0.004670208468734442)  --> TIME: 2025-02-26 20:15:43 -- STEP: 21994/36938 -- GLOBAL_STEP: 1388700 | > loss_text_ce: 0.03359919413924217 (0.030791301859552547) | > loss_mel_ce: 2.1663424968719482 (nan) | > loss: 0.008729927241802216 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.1369464809356677) | > loader_time: 0.0046 (0.0046703432585593055)  --> TIME: 2025-02-26 20:15:52 -- STEP: 22044/36938 -- GLOBAL_STEP: 1388750 | > loss_text_ce: 0.03914562985301018 (0.0307893199914633) | > loss_mel_ce: 3.06036639213562 (nan) | > loss: 0.012299652211368084 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13695689431770083) | > loader_time: 0.0046 (0.004670395119569616)  --> TIME: 2025-02-26 20:16:02 -- STEP: 22094/36938 -- GLOBAL_STEP: 1388800 | > loss_text_ce: 0.0302802175283432 (0.03078813913647202) | > loss_mel_ce: 3.087726593017578 (nan) | > loss: 0.012373043224215508 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13695877364742184) | > loader_time: 0.0048 (0.004670436354021009)  --> TIME: 2025-02-26 20:16:11 -- STEP: 22144/36938 -- GLOBAL_STEP: 1388850 | > loss_text_ce: 0.023644961416721344 (0.030787702006330093) | > loss_mel_ce: 2.998647928237915 (nan) | > loss: 0.01199322659522295 (nan) | > current_lr: 5e-06 | > step_time: 0.1449 (0.13696483428983036) | > loader_time: 0.0048 (0.004670600283008081)  --> TIME: 2025-02-26 20:16:21 -- STEP: 22194/36938 -- GLOBAL_STEP: 1388900 | > loss_text_ce: 0.02858017198741436 (0.030788126479213432) | > loss_mel_ce: 2.863004207611084 (nan) | > loss: 0.011474542319774628 (nan) | > current_lr: 5e-06 | > step_time: 0.1878 (0.1369809452194222) | > loader_time: 0.0049 (0.004670938210540208)  --> TIME: 2025-02-26 20:16:30 -- STEP: 22244/36938 -- GLOBAL_STEP: 1388950 | > loss_text_ce: 0.02791050635278225 (0.030786582949702605) | > loss_mel_ce: 2.595592737197876 (nan) | > loss: 0.010410727933049202 (nan) | > current_lr: 5e-06 | > step_time: 0.161 (0.1369818222863583) | > loader_time: 0.0047 (0.004671025364074397)  --> TIME: 2025-02-26 20:16:40 -- STEP: 22294/36938 -- GLOBAL_STEP: 1389000 | > loss_text_ce: 0.024876423180103302 (0.03078537189947763) | > loss_mel_ce: 3.085824728012085 (nan) | > loss: 0.012344053015112877 (nan) | > current_lr: 5e-06 | > step_time: 0.1621 (0.13699037144085133) | > loader_time: 0.0053 (0.004671221144308229)  --> TIME: 2025-02-26 20:16:50 -- STEP: 22344/36938 -- GLOBAL_STEP: 1389050 | > loss_text_ce: 0.03206288069486618 (0.03078665010766714) | > loss_mel_ce: 2.675877332687378 (nan) | > loss: 0.010745794512331486 (nan) | > current_lr: 5e-06 | > step_time: 0.1519 (0.13699775416706844) | > loader_time: 0.0046 (0.004671161954991063)  --> TIME: 2025-02-26 20:16:59 -- STEP: 22394/36938 -- GLOBAL_STEP: 1389100 | > loss_text_ce: 0.033477094024419785 (0.03078720471404181) | > loss_mel_ce: 0.01877339743077755 (nan) | > loss: 0.0002073432260658592 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13699347604286777) | > loader_time: 0.0046 (0.004671163172277674)  --> TIME: 2025-02-26 20:17:08 -- STEP: 22444/36938 -- GLOBAL_STEP: 1389150 | > loss_text_ce: 0.02478710189461708 (0.030790576428941636) | > loss_mel_ce: 2.459368944168091 (nan) | > loss: 0.009857762604951859 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.1369985263098208) | > loader_time: 0.0046 (0.00467115504668157)  --> TIME: 2025-02-26 20:17:18 -- STEP: 22494/36938 -- GLOBAL_STEP: 1389200 | > loss_text_ce: 0.024058327078819275 (0.030792008109358105) | > loss_mel_ce: 2.354346990585327 (nan) | > loss: 0.009438117034733295 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.13700058769308562) | > loader_time: 0.005 (0.004671308287750004)  --> TIME: 2025-02-26 20:17:27 -- STEP: 22544/36938 -- GLOBAL_STEP: 1389250 | > loss_text_ce: 0.03436897695064545 (0.030794635471983557) | > loss_mel_ce: 3.1998980045318604 (nan) | > loss: 0.012834393419325352 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.13699535120244313) | > loader_time: 0.0047 (0.004671469859572632)  --> TIME: 2025-02-26 20:17:36 -- STEP: 22594/36938 -- GLOBAL_STEP: 1389300 | > loss_text_ce: 0.03268849477171898 (0.03079487051370196) | > loss_mel_ce: 2.794452428817749 (nan) | > loss: 0.011218813247978687 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13698522882291952) | > loader_time: 0.0046 (0.004671445502383904)  --> TIME: 2025-02-26 20:17:46 -- STEP: 22644/36938 -- GLOBAL_STEP: 1389350 | > loss_text_ce: 0.025936944410204887 (0.030793321222333223) | > loss_mel_ce: 2.983642578125 (nan) | > loss: 0.011942775920033455 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1369988005335234) | > loader_time: 0.0045 (0.0046715636784860435)  --> TIME: 2025-02-26 20:17:55 -- STEP: 22694/36938 -- GLOBAL_STEP: 1389400 | > loss_text_ce: 0.029774699360132217 (0.030793628635491) | > loss_mel_ce: 2.9863643646240234 (nan) | > loss: 0.011968806385993958 (nan) | > current_lr: 5e-06 | > step_time: 0.1434 (0.13699385363845396) | > loader_time: 0.0054 (0.004671689906582114)  --> TIME: 2025-02-26 20:18:04 -- STEP: 22744/36938 -- GLOBAL_STEP: 1389450 | > loss_text_ce: 0.027973592281341553 (0.030793975753429747) | > loss_mel_ce: 3.069808006286621 (nan) | > loss: 0.01229278463870287 (nan) | > current_lr: 5e-06 | > step_time: 0.1846 (0.13699828414819817) | > loader_time: 0.0048 (0.004671728898218456)  --> TIME: 2025-02-26 20:18:14 -- STEP: 22794/36938 -- GLOBAL_STEP: 1389500 | > loss_text_ce: 0.026076367124915123 (0.03079479663184023) | > loss_mel_ce: 2.933103322982788 (nan) | > loss: 0.011742777191102505 (nan) | > current_lr: 5e-06 | > step_time: 0.1812 (0.13699562277429575) | > loader_time: 0.0049 (0.004671846626829313)  --> TIME: 2025-02-26 20:18:23 -- STEP: 22844/36938 -- GLOBAL_STEP: 1389550 | > loss_text_ce: 0.04219770431518555 (0.030797583139392248) | > loss_mel_ce: 3.467116594314575 (nan) | > loss: 0.013925851322710514 (nan) | > current_lr: 5e-06 | > step_time: 0.1267 (0.1370051725172503) | > loader_time: 0.0046 (0.004671840706549543)  --> TIME: 2025-02-26 20:18:32 -- STEP: 22894/36938 -- GLOBAL_STEP: 1389600 | > loss_text_ce: 0.013391081243753433 (0.03079651927893503) | > loss_mel_ce: 2.157999038696289 (nan) | > loss: 0.008616628125309944 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13700085524816322) | > loader_time: 0.0052 (0.004671943388704202)  --> TIME: 2025-02-26 20:18:42 -- STEP: 22944/36938 -- GLOBAL_STEP: 1389650 | > loss_text_ce: 0.03425897657871246 (0.030798632260532935) | > loss_mel_ce: 2.29998779296875 (nan) | > loss: 0.009262884967029095 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13699200042787238) | > loader_time: 0.0045 (0.004671936940448546)  --> TIME: 2025-02-26 20:18:51 -- STEP: 22994/36938 -- GLOBAL_STEP: 1389700 | > loss_text_ce: 0.030113518238067627 (0.030796469454014017) | > loss_mel_ce: 2.115381956100464 (nan) | > loss: 0.00851387158036232 (nan) | > current_lr: 5e-06 | > step_time: 0.1881 (0.136990433051565) | > loader_time: 0.0048 (0.004671827652065126)  --> TIME: 2025-02-26 20:19:00 -- STEP: 23044/36938 -- GLOBAL_STEP: 1389750 | > loss_text_ce: 0.045823853462934494 (0.03079581599388914) | > loss_mel_ce: 1.4735219478607178 (nan) | > loss: 0.006029149983078241 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1369816935676986) | > loader_time: 0.0045 (0.0046719040148085565)  --> TIME: 2025-02-26 20:19:09 -- STEP: 23094/36938 -- GLOBAL_STEP: 1389800 | > loss_text_ce: 0.02330920845270157 (0.030795444650977177) | > loss_mel_ce: 1.0273696184158325 (nan) | > loss: 0.004169360734522343 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.1369795079787522) | > loader_time: 0.0047 (0.004671860579506099)  --> TIME: 2025-02-26 20:19:18 -- STEP: 23144/36938 -- GLOBAL_STEP: 1389850 | > loss_text_ce: 0.015971709042787552 (0.03079371912007883) | > loss_mel_ce: 1.579900860786438 (nan) | > loss: 0.006332827731966972 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13697305890337913) | > loader_time: 0.0046 (0.004672002986027132)  --> TIME: 2025-02-26 20:19:28 -- STEP: 23194/36938 -- GLOBAL_STEP: 1389900 | > loss_text_ce: 0.028391459956765175 (0.030791380342676384) | > loss_mel_ce: 0.068603515625 (nan) | > loss: 0.0003849007189273834 (nan) | > current_lr: 5e-06 | > step_time: 0.1084 (0.13696895996805022) | > loader_time: 0.0044 (0.004671838331440455)  --> TIME: 2025-02-26 20:19:37 -- STEP: 23244/36938 -- GLOBAL_STEP: 1389950 | > loss_text_ce: 0.03714267536997795 (0.03079061373146954) | > loss_mel_ce: 2.883120059967041 (nan) | > loss: 0.011588345281779766 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.1369676525345411) | > loader_time: 0.0046 (0.004671841629030289)  --> TIME: 2025-02-26 20:19:46 -- STEP: 23294/36938 -- GLOBAL_STEP: 1390000 | > loss_text_ce: 0.026729200035333633 (0.030790035259627094) | > loss_mel_ce: 2.5835649967193604 (nan) | > loss: 0.010358310304582119 (nan) | > current_lr: 5e-06 | > step_time: 0.1494 (0.13697498386342558) | > loader_time: 0.0047 (0.004671899885683807) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1390000.pth  --> TIME: 2025-02-26 20:20:02 -- STEP: 23344/36938 -- GLOBAL_STEP: 1390050 | > loss_text_ce: 0.027121197432279587 (0.03078802395015562) | > loss_mel_ce: 0.696185290813446 (nan) | > loss: 0.002870263997465372 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1369792178855368) | > loader_time: 0.0046 (0.004671983088917569)  --> TIME: 2025-02-26 20:20:12 -- STEP: 23394/36938 -- GLOBAL_STEP: 1390100 | > loss_text_ce: 0.027536863461136818 (0.03078797299519096) | > loss_mel_ce: 2.671830892562866 (nan) | > loss: 0.01071177702397108 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13698618026985265) | > loader_time: 0.0047 (0.004671995870331432)  --> TIME: 2025-02-26 20:20:21 -- STEP: 23444/36938 -- GLOBAL_STEP: 1390150 | > loss_text_ce: 0.02651086263358593 (0.030788898394386522) | > loss_mel_ce: 1.6983129978179932 (nan) | > loss: 0.0068445391952991486 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13697640955031298) | > loader_time: 0.0045 (0.0046721524578824105)  --> TIME: 2025-02-26 20:20:30 -- STEP: 23494/36938 -- GLOBAL_STEP: 1390200 | > loss_text_ce: 0.0263080932199955 (0.030790958296937457) | > loss_mel_ce: 2.512946367263794 (nan) | > loss: 0.010076407343149185 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1369837333448322) | > loader_time: 0.0045 (0.004672104768208305)  --> TIME: 2025-02-26 20:20:39 -- STEP: 23544/36938 -- GLOBAL_STEP: 1390250 | > loss_text_ce: 0.030683070421218872 (0.030789435358215347) | > loss_mel_ce: 2.8731682300567627 (nan) | > loss: 0.011523219756782055 (nan) | > current_lr: 5e-06 | > step_time: 0.1767 (0.13698063210042877) | > loader_time: 0.0048 (0.0046722149103626725)  --> TIME: 2025-02-26 20:20:49 -- STEP: 23594/36938 -- GLOBAL_STEP: 1390300 | > loss_text_ce: 0.02395852468907833 (0.03079072436336511) | > loss_mel_ce: 1.0824730396270752 (nan) | > loss: 0.004390602000057697 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.1369813818380663) | > loader_time: 0.0044 (0.00467229663512257)  --> TIME: 2025-02-26 20:20:58 -- STEP: 23644/36938 -- GLOBAL_STEP: 1390350 | > loss_text_ce: 0.031079206615686417 (0.03078953759248693) | > loss_mel_ce: 2.6862642765045166 (nan) | > loss: 0.010783109813928604 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13698586005282132) | > loader_time: 0.0046 (0.004672218510552397)  --> TIME: 2025-02-26 20:21:08 -- STEP: 23694/36938 -- GLOBAL_STEP: 1390400 | > loss_text_ce: 0.02867957018315792 (0.03078645393408418) | > loss_mel_ce: 2.703972578048706 (nan) | > loss: 0.010843858122825623 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13698761456322953) | > loader_time: 0.0046 (0.004672134376391281)  --> TIME: 2025-02-26 20:21:17 -- STEP: 23744/36938 -- GLOBAL_STEP: 1390450 | > loss_text_ce: 0.032987646758556366 (0.030788369293248866) | > loss_mel_ce: 3.3161184787750244 (nan) | > loss: 0.0132901044562459 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.1369810042014192) | > loader_time: 0.0046 (0.004671976050597954)  --> TIME: 2025-02-26 20:21:26 -- STEP: 23794/36938 -- GLOBAL_STEP: 1390500 | > loss_text_ce: 0.026002341881394386 (0.0307877812354874) | > loss_mel_ce: 2.890625 (nan) | > loss: 0.011573919095098972 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13697869819563635) | > loader_time: 0.0045 (0.0046718611760911775)  --> TIME: 2025-02-26 20:21:36 -- STEP: 23844/36938 -- GLOBAL_STEP: 1390550 | > loss_text_ce: 0.026870576664805412 (0.03078599326144041) | > loss_mel_ce: 2.8180155754089355 (nan) | > loss: 0.011289231479167938 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13698206451230874) | > loader_time: 0.0046 (0.004671758532304128)  --> TIME: 2025-02-26 20:21:45 -- STEP: 23894/36938 -- GLOBAL_STEP: 1390600 | > loss_text_ce: 0.030001353472471237 (0.03078670746775758) | > loss_mel_ce: 2.5362236499786377 (nan) | > loss: 0.010183433070778847 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.13698951373032953) | > loader_time: 0.0047 (0.004671684735947053)  --> TIME: 2025-02-26 20:21:55 -- STEP: 23944/36938 -- GLOBAL_STEP: 1390650 | > loss_text_ce: 0.02341042272746563 (0.03078515048966423) | > loss_mel_ce: 1.8607701063156128 (nan) | > loss: 0.0074769072234630585 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13698908733437515) | > loader_time: 0.0046 (0.0046715789262642225)  --> TIME: 2025-02-26 20:22:04 -- STEP: 23994/36938 -- GLOBAL_STEP: 1390700 | > loss_text_ce: 0.02556166797876358 (0.030786315954514754) | > loss_mel_ce: 3.034414052963257 (nan) | > loss: 0.012142760679125786 (nan) | > current_lr: 5e-06 | > step_time: 0.1717 (0.1369829159771862) | > loader_time: 0.0047 (0.004671397959976339)  --> TIME: 2025-02-26 20:22:13 -- STEP: 24044/36938 -- GLOBAL_STEP: 1390750 | > loss_text_ce: 0.03087463229894638 (0.030782108332380317) | > loss_mel_ce: 1.9947987794876099 (nan) | > loss: 0.008038386702537537 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.13698849105295666) | > loader_time: 0.0049 (0.004671305859273328)  --> TIME: 2025-02-26 20:22:23 -- STEP: 24094/36938 -- GLOBAL_STEP: 1390800 | > loss_text_ce: 0.026196712628006935 (0.03078195158695777) | > loss_mel_ce: 2.9671823978424072 (nan) | > loss: 0.011878489516675472 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.1369964537050981) | > loader_time: 0.0046 (0.004671338376953771)  --> TIME: 2025-02-26 20:22:32 -- STEP: 24144/36938 -- GLOBAL_STEP: 1390850 | > loss_text_ce: 0.030311813578009605 (0.030783232789102603) | > loss_mel_ce: 2.9271914958953857 (nan) | > loss: 0.011736124753952026 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.13699867288508702) | > loader_time: 0.0047 (0.004671352501338954)  --> TIME: 2025-02-26 20:22:41 -- STEP: 24194/36938 -- GLOBAL_STEP: 1390900 | > loss_text_ce: 0.04042087495326996 (0.030786754100503345) | > loss_mel_ce: 2.8523881435394287 (nan) | > loss: 0.011479401029646397 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13699866035925404) | > loader_time: 0.0046 (0.004671468334256329)  --> TIME: 2025-02-26 20:22:51 -- STEP: 24244/36938 -- GLOBAL_STEP: 1390950 | > loss_text_ce: 0.03429947420954704 (0.0307890250455077) | > loss_mel_ce: 2.5674595832824707 (nan) | > loss: 0.010324440896511078 (nan) | > current_lr: 5e-06 | > step_time: 0.1438 (0.1369975288990927) | > loader_time: 0.0055 (0.004671633037044115)  --> TIME: 2025-02-26 20:23:00 -- STEP: 24294/36938 -- GLOBAL_STEP: 1391000 | > loss_text_ce: 0.028945056721568108 (0.030789886831799753) | > loss_mel_ce: 3.087501049041748 (nan) | > loss: 0.012366849929094315 (nan) | > current_lr: 5e-06 | > step_time: 0.1469 (0.13699864851473653) | > loader_time: 0.0046 (0.004671768052024271)  --> TIME: 2025-02-26 20:23:09 -- STEP: 24344/36938 -- GLOBAL_STEP: 1391050 | > loss_text_ce: 0.026812728494405746 (0.030789636941954905) | > loss_mel_ce: 2.7952263355255127 (nan) | > loss: 0.011198568157851696 (nan) | > current_lr: 5e-06 | > step_time: 0.1662 (0.1369965191327005) | > loader_time: 0.0068 (0.0046720546482034745)  --> TIME: 2025-02-26 20:23:19 -- STEP: 24394/36938 -- GLOBAL_STEP: 1391100 | > loss_text_ce: 0.02213629148900509 (0.03079097086326313) | > loss_mel_ce: 2.670870780944824 (nan) | > loss: 0.010686536319553852 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.13699631576822796) | > loader_time: 0.0047 (0.004672006005936691)  --> TIME: 2025-02-26 20:23:28 -- STEP: 24444/36938 -- GLOBAL_STEP: 1391150 | > loss_text_ce: 0.028758039698004723 (0.030790403906934598) | > loss_mel_ce: 2.682802200317383 (nan) | > loss: 0.01076016016304493 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.13698970125459403) | > loader_time: 0.0056 (0.004672035231096351)  --> TIME: 2025-02-26 20:23:37 -- STEP: 24494/36938 -- GLOBAL_STEP: 1391200 | > loss_text_ce: 0.024970104917883873 (0.0307869657472704) | > loss_mel_ce: 2.922842502593994 (nan) | > loss: 0.011697669513523579 (nan) | > current_lr: 5e-06 | > step_time: 0.157 (0.1369786719219972) | > loader_time: 0.0046 (0.00467201481159828)  --> TIME: 2025-02-26 20:23:46 -- STEP: 24544/36938 -- GLOBAL_STEP: 1391250 | > loss_text_ce: 0.03507888317108154 (0.03078666020558817) | > loss_mel_ce: 2.156704902648926 (nan) | > loss: 0.008697555400431156 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13697722911679402) | > loader_time: 0.0046 (0.0046720470470586395)  --> TIME: 2025-02-26 20:23:56 -- STEP: 24594/36938 -- GLOBAL_STEP: 1391300 | > loss_text_ce: 0.030411455780267715 (0.03078667948949414) | > loss_mel_ce: 3.188629388809204 (nan) | > loss: 0.012773972004652023 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1369789836852069) | > loader_time: 0.0046 (0.0046722340256138105)  --> TIME: 2025-02-26 20:24:05 -- STEP: 24644/36938 -- GLOBAL_STEP: 1391350 | > loss_text_ce: 0.02263309620320797 (0.03078792046089598) | > loss_mel_ce: 1.9346177577972412 (nan) | > loss: 0.00776686891913414 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13698598680084567) | > loader_time: 0.0044 (0.0046720830985157325)  --> TIME: 2025-02-26 20:24:14 -- STEP: 24694/36938 -- GLOBAL_STEP: 1391400 | > loss_text_ce: 0.02892659418284893 (0.030788454159961048) | > loss_mel_ce: 2.826061725616455 (nan) | > loss: 0.011329319328069687 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13697521914603258) | > loader_time: 0.0046 (0.004671926777247154)  --> TIME: 2025-02-26 20:24:23 -- STEP: 24744/36938 -- GLOBAL_STEP: 1391450 | > loss_text_ce: 0.031600311398506165 (0.030787369618623647) | > loss_mel_ce: 2.484415292739868 (nan) | > loss: 0.00998418964445591 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1369684330861323) | > loader_time: 0.0045 (0.004671804965833528)  --> TIME: 2025-02-26 20:24:33 -- STEP: 24794/36938 -- GLOBAL_STEP: 1391500 | > loss_text_ce: 0.027953030541539192 (0.030787614645689423) | > loss_mel_ce: 2.592505693435669 (nan) | > loss: 0.01039864681661129 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13697020209604013) | > loader_time: 0.0046 (0.004671726052180461)  --> TIME: 2025-02-26 20:24:42 -- STEP: 24844/36938 -- GLOBAL_STEP: 1391550 | > loss_text_ce: 0.03072221763432026 (0.030786736015965304) | > loss_mel_ce: 2.673274278640747 (nan) | > loss: 0.010730145499110222 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.13697106887936936) | > loader_time: 0.0046 (0.00467155888890168)  --> TIME: 2025-02-26 20:24:52 -- STEP: 24894/36938 -- GLOBAL_STEP: 1391600 | > loss_text_ce: 0.035509027540683746 (0.030786557987888576) | > loss_mel_ce: 2.194091796875 (nan) | > loss: 0.008847623132169247 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13697466875565056) | > loader_time: 0.0046 (0.004671458001978434)  --> TIME: 2025-02-26 20:25:01 -- STEP: 24944/36938 -- GLOBAL_STEP: 1391650 | > loss_text_ce: 0.03160035237669945 (0.030782936965005593) | > loss_mel_ce: 2.772841215133667 (nan) | > loss: 0.011128736659884453 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13698156800233455) | > loader_time: 0.0046 (0.00467134684305148)  --> TIME: 2025-02-26 20:25:11 -- STEP: 24994/36938 -- GLOBAL_STEP: 1391700 | > loss_text_ce: 0.03556695580482483 (0.03078465030725794) | > loss_mel_ce: 3.024059534072876 (nan) | > loss: 0.012141375802457333 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.13699576837917188) | > loader_time: 0.0046 (0.004671277223019116)  --> TIME: 2025-02-26 20:25:20 -- STEP: 25044/36938 -- GLOBAL_STEP: 1391750 | > loss_text_ce: 0.03246674686670303 (0.03078785752484521) | > loss_mel_ce: 3.175502061843872 (nan) | > loss: 0.012730034999549389 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13700525585673323) | > loader_time: 0.0046 (0.004671188583961975)  --> TIME: 2025-02-26 20:25:30 -- STEP: 25094/36938 -- GLOBAL_STEP: 1391800 | > loss_text_ce: 0.026918116956949234 (0.03078935606841369) | > loss_mel_ce: 2.2876534461975098 (nan) | > loss: 0.00918480847030878 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.13701521802102534) | > loader_time: 0.0046 (0.004671058151611169)  --> TIME: 2025-02-26 20:25:39 -- STEP: 25144/36938 -- GLOBAL_STEP: 1391850 | > loss_text_ce: 0.04150451719760895 (0.030791232886554824) | > loss_mel_ce: 3.084061622619629 (nan) | > loss: 0.012403041124343872 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.13701469269369337) | > loader_time: 0.0046 (0.004670913151939162)  --> TIME: 2025-02-26 20:25:49 -- STEP: 25194/36938 -- GLOBAL_STEP: 1391900 | > loss_text_ce: 0.014466061256825924 (0.030790441751532886) | > loss_mel_ce: 1.8692899942398071 (nan) | > loss: 0.007475222926586866 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13701932352857552) | > loader_time: 0.0046 (0.004670910052839813)  --> TIME: 2025-02-26 20:25:58 -- STEP: 25244/36938 -- GLOBAL_STEP: 1391950 | > loss_text_ce: 0.036063216626644135 (0.030791117166504855) | > loss_mel_ce: 2.8470544815063477 (nan) | > loss: 0.0114409439265728 (nan) | > current_lr: 5e-06 | > step_time: 0.1515 (0.13701663590935856) | > loader_time: 0.0046 (0.004670790391760906)  --> TIME: 2025-02-26 20:26:07 -- STEP: 25294/36938 -- GLOBAL_STEP: 1392000 | > loss_text_ce: 0.031166404485702515 (0.030790435196124875) | > loss_mel_ce: 2.667893409729004 (nan) | > loss: 0.010710555128753185 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13701521867705546) | > loader_time: 0.0046 (0.004670718842234405)  --> TIME: 2025-02-26 20:26:17 -- STEP: 25344/36938 -- GLOBAL_STEP: 1392050 | > loss_text_ce: 0.028135616332292557 (0.030790706018935927) | > loss_mel_ce: 2.5364913940429688 (nan) | > loss: 0.01017709169536829 (nan) | > current_lr: 5e-06 | > step_time: 0.1618 (0.13702297666006677) | > loader_time: 0.0046 (0.004670649428259235)  --> TIME: 2025-02-26 20:26:26 -- STEP: 25394/36938 -- GLOBAL_STEP: 1392100 | > loss_text_ce: 0.031280577182769775 (0.030790230041211942) | > loss_mel_ce: 4.78125 (nan) | > loss: 0.0190973449498415 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.1370209874041755) | > loader_time: 0.0045 (0.004670519992912186)  --> TIME: 2025-02-26 20:26:36 -- STEP: 25444/36938 -- GLOBAL_STEP: 1392150 | > loss_text_ce: 0.029054047539830208 (0.030788654987979926) | > loss_mel_ce: 2.856585741043091 (nan) | > loss: 0.011450952850282192 (nan) | > current_lr: 5e-06 | > step_time: 0.1689 (0.13702352345692614) | > loader_time: 0.0048 (0.004670482136471956)  --> TIME: 2025-02-26 20:26:45 -- STEP: 25494/36938 -- GLOBAL_STEP: 1392200 | > loss_text_ce: 0.02376222051680088 (0.03078667403650594) | > loss_mel_ce: 2.111976385116577 (nan) | > loss: 0.00847515370696783 (nan) | > current_lr: 5e-06 | > step_time: 0.1626 (0.13702839892116556) | > loader_time: 0.0047 (0.004670566088023388)  --> TIME: 2025-02-26 20:26:55 -- STEP: 25544/36938 -- GLOBAL_STEP: 1392250 | > loss_text_ce: 0.035960640758275986 (0.030786857088616348) | > loss_mel_ce: 2.6882224082946777 (nan) | > loss: 0.010810251347720623 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13703589882926878) | > loader_time: 0.0046 (0.004670539041914017)  --> TIME: 2025-02-26 20:27:04 -- STEP: 25594/36938 -- GLOBAL_STEP: 1392300 | > loss_text_ce: 0.04107725992798805 (0.030786367780007512) | > loss_mel_ce: 2.1106719970703125 (nan) | > loss: 0.008538688533008099 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.1370344335160767) | > loader_time: 0.0046 (0.004670393795783405)  --> TIME: 2025-02-26 20:27:13 -- STEP: 25644/36938 -- GLOBAL_STEP: 1392350 | > loss_text_ce: 0.044110655784606934 (0.03078901552599338) | > loss_mel_ce: 0.6403285264968872 (nan) | > loss: 0.0027160285972058773 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13703078296511512) | > loader_time: 0.0044 (0.004670258357509631)  --> TIME: 2025-02-26 20:27:22 -- STEP: 25694/36938 -- GLOBAL_STEP: 1392400 | > loss_text_ce: 0.027259623631834984 (0.0307889177123121) | > loss_mel_ce: 3.043814182281494 (nan) | > loss: 0.012186801061034203 (nan) | > current_lr: 5e-06 | > step_time: 0.184 (0.13703311369318424) | > loader_time: 0.0048 (0.004670180745131134)  --> TIME: 2025-02-26 20:27:32 -- STEP: 25744/36938 -- GLOBAL_STEP: 1392450 | > loss_text_ce: 0.023404914885759354 (0.030789332269473724) | > loss_mel_ce: 2.874189853668213 (nan) | > loss: 0.011498392559587955 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.1370289277463625) | > loader_time: 0.0045 (0.004670114066009576)  --> TIME: 2025-02-26 20:27:41 -- STEP: 25794/36938 -- GLOBAL_STEP: 1392500 | > loss_text_ce: 0.029655231162905693 (0.0307896829485098) | > loss_mel_ce: 2.7250516414642334 (nan) | > loss: 0.010931377299129963 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.13702620880043948) | > loader_time: 0.0048 (0.004670083592120486)  --> TIME: 2025-02-26 20:27:50 -- STEP: 25844/36938 -- GLOBAL_STEP: 1392550 | > loss_text_ce: 0.01708402670919895 (0.03078843738692813) | > loss_mel_ce: 2.6254236698150635 (nan) | > loss: 0.010486142709851265 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13702339007747183) | > loader_time: 0.0046 (0.004670164751537895)  --> TIME: 2025-02-26 20:28:00 -- STEP: 25894/36938 -- GLOBAL_STEP: 1392600 | > loss_text_ce: 0.037004828453063965 (0.03078601295505036) | > loss_mel_ce: 3.0978105068206787 (nan) | > loss: 0.012439743615686893 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.1370213091801603) | > loader_time: 0.0048 (0.004670105432003751)  --> TIME: 2025-02-26 20:28:09 -- STEP: 25944/36938 -- GLOBAL_STEP: 1392650 | > loss_text_ce: 0.029119230806827545 (0.03078431084080729) | > loss_mel_ce: 2.152118682861328 (nan) | > loss: 0.00865570642054081 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13702404790368783) | > loader_time: 0.0048 (0.004670152335559525)  --> TIME: 2025-02-26 20:28:18 -- STEP: 25994/36938 -- GLOBAL_STEP: 1392700 | > loss_text_ce: 0.020818039774894714 (0.030784154661170172) | > loss_mel_ce: 2.224334239959717 (nan) | > loss: 0.008909334428608418 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13701901985625567) | > loader_time: 0.0045 (0.00467031288396454)  --> TIME: 2025-02-26 20:28:27 -- STEP: 26044/36938 -- GLOBAL_STEP: 1392750 | > loss_text_ce: 0.029502343386411667 (0.030785845586556736) | > loss_mel_ce: 3.2320303916931152 (nan) | > loss: 0.01294259075075388 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.13701845739609644) | > loader_time: 0.0047 (0.004670196708455989)  --> TIME: 2025-02-26 20:28:37 -- STEP: 26094/36938 -- GLOBAL_STEP: 1392800 | > loss_text_ce: 0.033001940697431564 (0.03078523903863232) | > loss_mel_ce: 2.180121660232544 (nan) | > loss: 0.008782236836850643 (nan) | > current_lr: 5e-06 | > step_time: 0.1447 (0.13702719022627144) | > loader_time: 0.0047 (0.004670141144731534)  --> TIME: 2025-02-26 20:28:46 -- STEP: 26144/36938 -- GLOBAL_STEP: 1392850 | > loss_text_ce: 0.03817559778690338 (0.03078507513106436) | > loss_mel_ce: 2.513266086578369 (nan) | > loss: 0.010124769061803818 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13702755746964115) | > loader_time: 0.0046 (0.004670152848762001)  --> TIME: 2025-02-26 20:28:56 -- STEP: 26194/36938 -- GLOBAL_STEP: 1392900 | > loss_text_ce: 0.033372070640325546 (0.03078585234580885) | > loss_mel_ce: 2.319333791732788 (nan) | > loss: 0.009336135350167751 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13702664108979887) | > loader_time: 0.0046 (0.004670029279247651)  --> TIME: 2025-02-26 20:29:05 -- STEP: 26244/36938 -- GLOBAL_STEP: 1392950 | > loss_text_ce: 0.024688756093382835 (0.030784812502700258) | > loss_mel_ce: 2.7208163738250732 (nan) | > loss: 0.01089486200362444 (nan) | > current_lr: 5e-06 | > step_time: 0.1522 (0.13701887102393182) | > loader_time: 0.0047 (0.004670040588556253)  --> TIME: 2025-02-26 20:29:14 -- STEP: 26294/36938 -- GLOBAL_STEP: 1393000 | > loss_text_ce: 0.041269879788160324 (0.030784602764974207) | > loss_mel_ce: 0.1941048800945282 (nan) | > loss: 0.0009340269025415182 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13702544975962727) | > loader_time: 0.0046 (0.004670054828965595)  --> TIME: 2025-02-26 20:29:24 -- STEP: 26344/36938 -- GLOBAL_STEP: 1393050 | > loss_text_ce: 0.03678511828184128 (0.030784836881376166) | > loss_mel_ce: 0.03057861328125 (nan) | > loss: 0.00026731641264632344 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.1370232393232634) | > loader_time: 0.0046 (0.004669965661987517)  --> TIME: 2025-02-26 20:29:33 -- STEP: 26394/36938 -- GLOBAL_STEP: 1393100 | > loss_text_ce: 0.03937556594610214 (0.030785189279130477) | > loss_mel_ce: 1.9773746728897095 (nan) | > loss: 0.008002977818250656 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.1370248953176204) | > loader_time: 0.0045 (0.004669942837407533)  --> TIME: 2025-02-26 20:29:42 -- STEP: 26444/36938 -- GLOBAL_STEP: 1393150 | > loss_text_ce: 0.047335099428892136 (0.030787499809278855) | > loss_mel_ce: 3.0375845432281494 (nan) | > loss: 0.012241745367646217 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.1370260341141133) | > loader_time: 0.0045 (0.004669873468491095)  --> TIME: 2025-02-26 20:29:52 -- STEP: 26494/36938 -- GLOBAL_STEP: 1393200 | > loss_text_ce: 0.02474365569651127 (0.03078573240375906) | > loss_mel_ce: 2.1530134677886963 (nan) | > loss: 0.008641893044114113 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13702036119222008) | > loader_time: 0.0047 (0.00467010370297569)  --> TIME: 2025-02-26 20:30:01 -- STEP: 26544/36938 -- GLOBAL_STEP: 1393250 | > loss_text_ce: 0.04350829869508743 (0.030785166084436717) | > loss_mel_ce: 2.422405481338501 (nan) | > loss: 0.009785372763872147 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13702472562959606) | > loader_time: 0.0046 (0.00467008492399082)  --> TIME: 2025-02-26 20:30:11 -- STEP: 26594/36938 -- GLOBAL_STEP: 1393300 | > loss_text_ce: 0.029038982465863228 (0.03078645004812694) | > loss_mel_ce: 3.12795090675354 (nan) | > loss: 0.012527737766504288 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.1370310100153937) | > loader_time: 0.0048 (0.004670454970408076)  --> TIME: 2025-02-26 20:30:20 -- STEP: 26644/36938 -- GLOBAL_STEP: 1393350 | > loss_text_ce: 0.03442440181970596 (0.0307852500431331) | > loss_mel_ce: 2.8508286476135254 (nan) | > loss: 0.011449417099356651 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.137028174820185) | > loader_time: 0.0047 (0.004670498240836988)  --> TIME: 2025-02-26 20:30:29 -- STEP: 26694/36938 -- GLOBAL_STEP: 1393400 | > loss_text_ce: 0.025938712060451508 (0.030785598803352472) | > loss_mel_ce: 2.1370739936828613 (nan) | > loss: 0.008583384566009045 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13703547427384105) | > loader_time: 0.0046 (0.004670616141899689)  --> TIME: 2025-02-26 20:30:39 -- STEP: 26744/36938 -- GLOBAL_STEP: 1393450 | > loss_text_ce: 0.030768420547246933 (0.030785770791838644) | > loss_mel_ce: 2.188016414642334 (nan) | > loss: 0.008804702199995518 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13703858972736968) | > loader_time: 0.0045 (0.004670757895060875)  --> TIME: 2025-02-26 20:30:48 -- STEP: 26794/36938 -- GLOBAL_STEP: 1393500 | > loss_text_ce: 0.030482547357678413 (0.030784752773116267) | > loss_mel_ce: 2.816755533218384 (nan) | > loss: 0.01129856426268816 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.13703663838375174) | > loader_time: 0.0046 (0.004670728647309369)  --> TIME: 2025-02-26 20:30:58 -- STEP: 26844/36938 -- GLOBAL_STEP: 1393550 | > loss_text_ce: 0.036871135234832764 (0.030785383742019966) | > loss_mel_ce: 2.6636385917663574 (nan) | > loss: 0.01071630883961916 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13704035735453687) | > loader_time: 0.0046 (0.0046707066848725365)  --> TIME: 2025-02-26 20:31:07 -- STEP: 26894/36938 -- GLOBAL_STEP: 1393600 | > loss_text_ce: 0.037072647362947464 (0.030785075176761132) | > loss_mel_ce: 3.3206379413604736 (nan) | > loss: 0.013324249535799026 (nan) | > current_lr: 5e-06 | > step_time: 0.1831 (0.13705018100006713) | > loader_time: 0.0048 (0.004670730361958338)  --> TIME: 2025-02-26 20:31:17 -- STEP: 26944/36938 -- GLOBAL_STEP: 1393650 | > loss_text_ce: 0.028632521629333496 (0.030784460289329546) | > loss_mel_ce: 1.8919596672058105 (nan) | > loss: 0.0076213981956243515 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1370528399625762) | > loader_time: 0.0045 (0.004670683135258616)  --> TIME: 2025-02-26 20:31:26 -- STEP: 26994/36938 -- GLOBAL_STEP: 1393700 | > loss_text_ce: 0.02299114130437374 (0.03078412822487186) | > loss_mel_ce: 2.095912456512451 (nan) | > loss: 0.008408348076045513 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13705247006010474) | > loader_time: 0.0045 (0.004670692194989323)  --> TIME: 2025-02-26 20:31:35 -- STEP: 27044/36938 -- GLOBAL_STEP: 1393750 | > loss_text_ce: 0.028885358944535255 (0.030783890756159095) | > loss_mel_ce: 2.4251527786254883 (nan) | > loss: 0.009738246910274029 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13704956576697683) | > loader_time: 0.0046 (0.004670762783005967)  --> TIME: 2025-02-26 20:31:45 -- STEP: 27094/36938 -- GLOBAL_STEP: 1393800 | > loss_text_ce: 0.03299577161669731 (0.030785317894106674) | > loss_mel_ce: 2.0665335655212402 (nan) | > loss: 0.008331465534865856 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13705789803525936) | > loader_time: 0.0046 (0.004670780488395287)  --> TIME: 2025-02-26 20:31:54 -- STEP: 27144/36938 -- GLOBAL_STEP: 1393850 | > loss_text_ce: 0.031081587076187134 (0.03078472297045427) | > loss_mel_ce: 2.9545047283172607 (nan) | > loss: 0.01184756588190794 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13705685403611997) | > loader_time: 0.0047 (0.00467076002585302)  --> TIME: 2025-02-26 20:32:04 -- STEP: 27194/36938 -- GLOBAL_STEP: 1393900 | > loss_text_ce: 0.02887447364628315 (0.03078454540772372) | > loss_mel_ce: 3.0594582557678223 (nan) | > loss: 0.012255288660526276 (nan) | > current_lr: 5e-06 | > step_time: 0.1487 (0.1370633839421168) | > loader_time: 0.0068 (0.004670968939120562)  --> TIME: 2025-02-26 20:32:13 -- STEP: 27244/36938 -- GLOBAL_STEP: 1393950 | > loss_text_ce: 0.026162533089518547 (0.03078364864332693) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13706064986475575) | > loader_time: 0.0045 (0.004670982720690536)  --> TIME: 2025-02-26 20:32:22 -- STEP: 27294/36938 -- GLOBAL_STEP: 1394000 | > loss_text_ce: 0.026930013671517372 (0.030785125614668493) | > loss_mel_ce: 2.779762029647827 (nan) | > loss: 0.011137668043375015 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.1370610868641687) | > loader_time: 0.0046 (0.0046710215217946535)  --> TIME: 2025-02-26 20:32:32 -- STEP: 27344/36938 -- GLOBAL_STEP: 1394050 | > loss_text_ce: 0.029138272628188133 (0.03078342620088204) | > loss_mel_ce: 1.9520456790924072 (nan) | > loss: 0.007861841470003128 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13706424969235997) | > loader_time: 0.0044 (0.004670926811682551)  --> TIME: 2025-02-26 20:32:41 -- STEP: 27394/36938 -- GLOBAL_STEP: 1394100 | > loss_text_ce: 0.028652917593717575 (0.030783334424713225) | > loss_mel_ce: 2.609928607940674 (nan) | > loss: 0.010470562614500523 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13705899722908493) | > loader_time: 0.0047 (0.0046710375583005675)  --> TIME: 2025-02-26 20:32:50 -- STEP: 27444/36938 -- GLOBAL_STEP: 1394150 | > loss_text_ce: 0.03808547183871269 (0.030784439061562835) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1370553002032925) | > loader_time: 0.0045 (0.004671022341561)  --> TIME: 2025-02-26 20:32:59 -- STEP: 27494/36938 -- GLOBAL_STEP: 1394200 | > loss_text_ce: 0.016309600323438644 (0.030784471583485382) | > loss_mel_ce: 0.990132212638855 (nan) | > loss: 0.003993817139416933 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13705131039321544) | > loader_time: 0.0045 (0.004671087106843603)  --> TIME: 2025-02-26 20:33:09 -- STEP: 27544/36938 -- GLOBAL_STEP: 1394250 | > loss_text_ce: 0.029376253485679626 (0.03078510428559656) | > loss_mel_ce: 2.614025354385376 (nan) | > loss: 0.0104896891862154 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13704514446350016) | > loader_time: 0.0047 (0.004671253707566102)  --> TIME: 2025-02-26 20:33:18 -- STEP: 27594/36938 -- GLOBAL_STEP: 1394300 | > loss_text_ce: 0.028616705909371376 (0.030784029540075072) | > loss_mel_ce: 2.670016050338745 (nan) | > loss: 0.01070886105298996 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13705237999406228) | > loader_time: 0.0046 (0.004671218352550411)  --> TIME: 2025-02-26 20:33:27 -- STEP: 27644/36938 -- GLOBAL_STEP: 1394350 | > loss_text_ce: 0.027536578476428986 (0.03078284665221296) | > loss_mel_ce: 2.4180185794830322 (nan) | > loss: 0.009704585187137127 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13705086608885972) | > loader_time: 0.0047 (0.004671282549874521)  --> TIME: 2025-02-26 20:33:37 -- STEP: 27694/36938 -- GLOBAL_STEP: 1394400 | > loss_text_ce: 0.03211381286382675 (0.03078173401324581) | > loss_mel_ce: 2.840372323989868 (nan) | > loss: 0.011398755013942719 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13705582505762937) | > loader_time: 0.0046 (0.004671308265452426)  --> TIME: 2025-02-26 20:33:46 -- STEP: 27744/36938 -- GLOBAL_STEP: 1394450 | > loss_text_ce: 0.02316664159297943 (0.030781584561106502) | > loss_mel_ce: 2.797222137451172 (nan) | > loss: 0.011192020028829575 (nan) | > current_lr: 5e-06 | > step_time: 0.1783 (0.13706159661282385) | > loader_time: 0.0048 (0.004671393578921504)  --> TIME: 2025-02-26 20:33:56 -- STEP: 27794/36938 -- GLOBAL_STEP: 1394500 | > loss_text_ce: 0.027406055480241776 (0.030778541498416544) | > loss_mel_ce: 2.5365536212921143 (nan) | > loss: 0.010174443013966084 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13705681225048708) | > loader_time: 0.0045 (0.004671485876791722)  --> TIME: 2025-02-26 20:34:05 -- STEP: 27844/36938 -- GLOBAL_STEP: 1394550 | > loss_text_ce: 0.03608168289065361 (0.030777632423947876) | > loss_mel_ce: 2.5555949211120605 (nan) | > loss: 0.010284432210028172 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13704752250745994) | > loader_time: 0.0046 (0.004671461742134248)  --> TIME: 2025-02-26 20:34:14 -- STEP: 27894/36938 -- GLOBAL_STEP: 1394600 | > loss_text_ce: 0.03287958726286888 (0.03077554594713818) | > loss_mel_ce: 2.235750436782837 (nan) | > loss: 0.009002500213682652 (nan) | > current_lr: 5e-06 | > step_time: 0.1693 (0.13705038473906545) | > loader_time: 0.0048 (0.004671665180573087)  --> TIME: 2025-02-26 20:34:23 -- STEP: 27944/36938 -- GLOBAL_STEP: 1394650 | > loss_text_ce: 0.03217129036784172 (0.03077714056035689) | > loss_mel_ce: 2.972583770751953 (nan) | > loss: 0.011923631653189659 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13704470644664832) | > loader_time: 0.0047 (0.004671609746396224)  --> TIME: 2025-02-26 20:34:32 -- STEP: 27994/36938 -- GLOBAL_STEP: 1394700 | > loss_text_ce: 0.03194887936115265 (0.03077643971898927) | > loss_mel_ce: 2.4377808570861816 (nan) | > loss: 0.00980051513761282 (nan) | > current_lr: 5e-06 | > step_time: 0.1814 (0.13704065653292846) | > loader_time: 0.0048 (0.004671703511206967)  --> TIME: 2025-02-26 20:34:42 -- STEP: 28044/36938 -- GLOBAL_STEP: 1394750 | > loss_text_ce: 0.03424340486526489 (0.030777726038522493) | > loss_mel_ce: 2.383634090423584 (nan) | > loss: 0.009594752453267574 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13703945164775105) | > loader_time: 0.0045 (0.004671791126581497)  --> TIME: 2025-02-26 20:34:51 -- STEP: 28094/36938 -- GLOBAL_STEP: 1394800 | > loss_text_ce: 0.013346153311431408 (0.030777034548922708) | > loss_mel_ce: 1.9368865489959717 (nan) | > loss: 0.007739019114524126 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13703936724134336) | > loader_time: 0.0045 (0.0046717806915426655)  --> TIME: 2025-02-26 20:35:01 -- STEP: 28144/36938 -- GLOBAL_STEP: 1394850 | > loss_text_ce: 0.026651043444871902 (0.030776105704667697) | > loss_mel_ce: 2.394862651824951 (nan) | > loss: 0.009609182365238667 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.1370481629985659) | > loader_time: 0.0048 (0.004671753977699134)  --> TIME: 2025-02-26 20:35:10 -- STEP: 28194/36938 -- GLOBAL_STEP: 1394900 | > loss_text_ce: 0.03119218349456787 (0.030777964370976345) | > loss_mel_ce: 3.1101341247558594 (nan) | > loss: 0.012465582229197025 (nan) | > current_lr: 5e-06 | > step_time: 0.1624 (0.13704399512221319) | > loader_time: 0.0047 (0.004671730123835021)  --> TIME: 2025-02-26 20:35:19 -- STEP: 28244/36938 -- GLOBAL_STEP: 1394950 | > loss_text_ce: 0.029620995745062828 (0.030775780568280918) | > loss_mel_ce: 0.200836181640625 (nan) | > loss: 0.0009145126678049564 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.1370387750701121) | > loader_time: 0.0048 (0.004671655469736933)  --> TIME: 2025-02-26 20:35:28 -- STEP: 28294/36938 -- GLOBAL_STEP: 1395000 | > loss_text_ce: 0.03293667733669281 (0.030774389897356168) | > loss_mel_ce: 3.4726738929748535 (nan) | > loss: 0.013911153189837933 (nan) | > current_lr: 5e-06 | > step_time: 0.1874 (0.1370339250449987) | > loader_time: 0.0048 (0.004671612383831082)  --> TIME: 2025-02-26 20:35:38 -- STEP: 28344/36938 -- GLOBAL_STEP: 1395050 | > loss_text_ce: 0.02927526645362377 (0.030775964367267154) | > loss_mel_ce: 2.67844295501709 (nan) | > loss: 0.0107449134811759 (nan) | > current_lr: 5e-06 | > step_time: 0.1688 (0.1370411038466047) | > loader_time: 0.0048 (0.004671507422748414)  --> TIME: 2025-02-26 20:35:47 -- STEP: 28394/36938 -- GLOBAL_STEP: 1395100 | > loss_text_ce: 0.03753771632909775 (0.030778835893859807) | > loss_mel_ce: 2.893130302429199 (nan) | > loss: 0.011629636399447918 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.1370351806009063) | > loader_time: 0.0046 (0.004671511427066597)  --> TIME: 2025-02-26 20:35:56 -- STEP: 28444/36938 -- GLOBAL_STEP: 1395150 | > loss_text_ce: 0.034653160721063614 (0.030778632048759625) | > loss_mel_ce: 1.112213134765625 (nan) | > loss: 0.0045510572381317616 (nan) | > current_lr: 5e-06 | > step_time: 0.1084 (0.13702856071084973) | > loader_time: 0.0045 (0.004671476298353544)  --> TIME: 2025-02-26 20:36:06 -- STEP: 28494/36938 -- GLOBAL_STEP: 1395200 | > loss_text_ce: 0.030103716999292374 (0.0307795384665183) | > loss_mel_ce: 2.749793529510498 (nan) | > loss: 0.011031338945031166 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.1370318167383845) | > loader_time: 0.0053 (0.004671516456612042)  --> TIME: 2025-02-26 20:36:15 -- STEP: 28544/36938 -- GLOBAL_STEP: 1395250 | > loss_text_ce: 0.03473992645740509 (0.030779653904879814) | > loss_mel_ce: 2.214299201965332 (nan) | > loss: 0.008924758993089199 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1370293278531121) | > loader_time: 0.0045 (0.004671427742849557)  --> TIME: 2025-02-26 20:36:24 -- STEP: 28594/36938 -- GLOBAL_STEP: 1395300 | > loss_text_ce: 0.029380034655332565 (0.030777341437496664) | > loss_mel_ce: 2.5092711448669434 (nan) | > loss: 0.010074013844132423 (nan) | > current_lr: 5e-06 | > step_time: 0.161 (0.1370307456499026) | > loader_time: 0.0048 (0.004671367988925291)  --> TIME: 2025-02-26 20:36:33 -- STEP: 28644/36938 -- GLOBAL_STEP: 1395350 | > loss_text_ce: 0.04065132141113281 (0.030776798185742055) | > loss_mel_ce: 2.651378631591797 (nan) | > loss: 0.010682659223675728 (nan) | > current_lr: 5e-06 | > step_time: 0.1266 (0.13702740152027792) | > loader_time: 0.0078 (0.004671390330287075)  --> TIME: 2025-02-26 20:36:42 -- STEP: 28694/36938 -- GLOBAL_STEP: 1395400 | > loss_text_ce: 0.035171233117580414 (0.030774195528531865) | > loss_mel_ce: 2.5637826919555664 (nan) | > loss: 0.010313309729099274 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13701069536978308) | > loader_time: 0.0046 (0.004671352511375573)  --> TIME: 2025-02-26 20:36:51 -- STEP: 28744/36938 -- GLOBAL_STEP: 1395450 | > loss_text_ce: 0.03191765025258064 (0.03077587486798982) | > loss_mel_ce: 3.120473623275757 (nan) | > loss: 0.012509489431977272 (nan) | > current_lr: 5e-06 | > step_time: 0.1905 (0.13700843004174504) | > loader_time: 0.0049 (0.004671469666716987)  --> TIME: 2025-02-26 20:37:01 -- STEP: 28794/36938 -- GLOBAL_STEP: 1395500 | > loss_text_ce: 0.03507554531097412 (0.030776707006398694) | > loss_mel_ce: 2.5078327655792236 (nan) | > loss: 0.010090906172990799 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.13700295251434536) | > loader_time: 0.0046 (0.004671471445330439)  --> TIME: 2025-02-26 20:37:10 -- STEP: 28844/36938 -- GLOBAL_STEP: 1395550 | > loss_text_ce: 0.01894341967999935 (0.03077667297173478) | > loss_mel_ce: 2.240635395050049 (nan) | > loss: 0.008966582827270031 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.1369949881271037) | > loader_time: 0.0046 (0.004671375582211894)  --> TIME: 2025-02-26 20:37:19 -- STEP: 28894/36938 -- GLOBAL_STEP: 1395600 | > loss_text_ce: 0.03719117119908333 (0.030778750312082877) | > loss_mel_ce: 2.958782434463501 (nan) | > loss: 0.011888784356415272 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13699633926473048) | > loader_time: 0.0046 (0.004671328115803224)  --> TIME: 2025-02-26 20:37:28 -- STEP: 28944/36938 -- GLOBAL_STEP: 1395650 | > loss_text_ce: 0.027953945100307465 (0.030779202212325347) | > loss_mel_ce: 2.628662347793579 (nan) | > loss: 0.0105421282351017 (nan) | > current_lr: 5e-06 | > step_time: 0.1906 (0.13699905818144062) | > loader_time: 0.0049 (0.004671493811855143)  --> TIME: 2025-02-26 20:37:38 -- STEP: 28994/36938 -- GLOBAL_STEP: 1395700 | > loss_text_ce: 0.03386975824832916 (0.030781004253370537) | > loss_mel_ce: 2.2171499729156494 (nan) | > loss: 0.008932618424296379 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13699920858162387) | > loader_time: 0.0045 (0.004671482026120623)  --> TIME: 2025-02-26 20:37:47 -- STEP: 29044/36938 -- GLOBAL_STEP: 1395750 | > loss_text_ce: 0.02867223136126995 (0.03078017274999762) | > loss_mel_ce: 2.583638906478882 (nan) | > loss: 0.010366315022110939 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1370064342929856) | > loader_time: 0.0045 (0.00467140655546474)  --> TIME: 2025-02-26 20:37:56 -- STEP: 29094/36938 -- GLOBAL_STEP: 1395800 | > loss_text_ce: 0.03184784576296806 (0.03077903623947369) | > loss_mel_ce: 2.309309244155884 (nan) | > loss: 0.009290306828916073 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13699900163214146) | > loader_time: 0.0047 (0.004671425969197337)  --> TIME: 2025-02-26 20:38:06 -- STEP: 29144/36938 -- GLOBAL_STEP: 1395850 | > loss_text_ce: 0.031488366425037384 (0.030779970074847725) | > loss_mel_ce: 3.118948221206665 (nan) | > loss: 0.012501733377575874 (nan) | > current_lr: 5e-06 | > step_time: 0.168 (0.13700330018539192) | > loader_time: 0.0047 (0.004671423236583577)  --> TIME: 2025-02-26 20:38:15 -- STEP: 29194/36938 -- GLOBAL_STEP: 1395900 | > loss_text_ce: 0.03319535031914711 (0.030778772392659644) | > loss_mel_ce: 2.6918647289276123 (nan) | > loss: 0.010813730768859386 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13700485197642184) | > loader_time: 0.0046 (0.0046714346743842)  --> TIME: 2025-02-26 20:38:25 -- STEP: 29244/36938 -- GLOBAL_STEP: 1395950 | > loss_text_ce: 0.04563359543681145 (0.03077967897166894) | > loss_mel_ce: 2.582317590713501 (nan) | > loss: 0.010428378358483315 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13700869026009407) | > loader_time: 0.0046 (0.004671479442217111)  --> TIME: 2025-02-26 20:38:34 -- STEP: 29294/36938 -- GLOBAL_STEP: 1396000 | > loss_text_ce: 0.029772378504276276 (0.03077984640613285) | > loss_mel_ce: 1.4747161865234375 (nan) | > loss: 0.005970193073153496 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.1370080292994015) | > loader_time: 0.0053 (0.0046715906571876435)  --> TIME: 2025-02-26 20:38:43 -- STEP: 29344/36938 -- GLOBAL_STEP: 1396050 | > loss_text_ce: 0.02680254355072975 (0.030778410937100712) | > loss_mel_ce: 1.9992752075195312 (nan) | > loss: 0.008039991371333599 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13700554733578094) | > loader_time: 0.0046 (0.004671674883937634)  --> TIME: 2025-02-26 20:38:53 -- STEP: 29394/36938 -- GLOBAL_STEP: 1396100 | > loss_text_ce: 0.03117281198501587 (0.03077849881781378) | > loss_mel_ce: 2.135143280029297 (nan) | > loss: 0.008596492931246758 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.1370037960066908) | > loader_time: 0.0046 (0.004671639160630725)  --> TIME: 2025-02-26 20:39:02 -- STEP: 29444/36938 -- GLOBAL_STEP: 1396150 | > loss_text_ce: 0.033814847469329834 (0.030778957991883578) | > loss_mel_ce: 3.1011457443237305 (nan) | > loss: 0.012440321035683155 (nan) | > current_lr: 5e-06 | > step_time: 0.2038 (0.13701568457471844) | > loader_time: 0.0049 (0.004671658118641458)  --> TIME: 2025-02-26 20:39:12 -- STEP: 29494/36938 -- GLOBAL_STEP: 1396200 | > loss_text_ce: 0.03213226795196533 (0.03078138984233227) | > loss_mel_ce: 3.3561291694641113 (nan) | > loss: 0.013445482589304447 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13701878191697886) | > loader_time: 0.0046 (0.0046717065176236664)  --> TIME: 2025-02-26 20:39:22 -- STEP: 29544/36938 -- GLOBAL_STEP: 1396250 | > loss_text_ce: 0.02653629332780838 (0.030781926871989183) | > loss_mel_ce: 0.13206176459789276 (nan) | > loss: 0.0006293574115261436 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13702399549754268) | > loader_time: 0.0045 (0.00467166509565028)  --> TIME: 2025-02-26 20:39:31 -- STEP: 29594/36938 -- GLOBAL_STEP: 1396300 | > loss_text_ce: 0.03195468336343765 (0.030782266107376572) | > loss_mel_ce: 0.8763649463653564 (nan) | > loss: 0.0036044432781636715 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.13701950537801671) | > loader_time: 0.0048 (0.004671645324004784)  --> TIME: 2025-02-26 20:39:40 -- STEP: 29644/36938 -- GLOBAL_STEP: 1396350 | > loss_text_ce: 0.029306944459676743 (0.030781733023039562) | > loss_mel_ce: 1.7436727285385132 (nan) | > loss: 0.0070356340147554874 (nan) | > current_lr: 5e-06 | > step_time: 0.1466 (0.13701748323575813) | > loader_time: 0.0046 (0.004671759208740458)  --> TIME: 2025-02-26 20:39:50 -- STEP: 29694/36938 -- GLOBAL_STEP: 1396400 | > loss_text_ce: 0.036863334476947784 (0.03078097784607253) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13702318626132118) | > loader_time: 0.0046 (0.004671872364693467)  --> TIME: 2025-02-26 20:39:59 -- STEP: 29744/36938 -- GLOBAL_STEP: 1396450 | > loss_text_ce: 0.03250545635819435 (0.030780237457248935) | > loss_mel_ce: 2.792670726776123 (nan) | > loss: 0.011211017146706581 (nan) | > current_lr: 5e-06 | > step_time: 0.1511 (0.13701798456243724) | > loader_time: 0.0047 (0.0046719510815616885)  --> TIME: 2025-02-26 20:40:08 -- STEP: 29794/36938 -- GLOBAL_STEP: 1396500 | > loss_text_ce: 0.028158118948340416 (0.030778452626844546) | > loss_mel_ce: 2.7490153312683105 (nan) | > loss: 0.011020530015230179 (nan) | > current_lr: 5e-06 | > step_time: 0.1457 (0.13701838870700697) | > loader_time: 0.0046 (0.004672034223535655)  --> TIME: 2025-02-26 20:40:18 -- STEP: 29844/36938 -- GLOBAL_STEP: 1396550 | > loss_text_ce: 0.03100641444325447 (0.030778401800511945) | > loss_mel_ce: 2.8808727264404297 (nan) | > loss: 0.011555076576769352 (nan) | > current_lr: 5e-06 | > step_time: 0.1442 (0.13702275310120776) | > loader_time: 0.0047 (0.004672100558036173)  --> TIME: 2025-02-26 20:40:27 -- STEP: 29894/36938 -- GLOBAL_STEP: 1396600 | > loss_text_ce: 0.01956930384039879 (0.03077736070859451) | > loss_mel_ce: 2.8624684810638428 (nan) | > loss: 0.011436658911406994 (nan) | > current_lr: 5e-06 | > step_time: 0.1689 (0.13702837251824274) | > loader_time: 0.0048 (0.004672178338744331)  --> TIME: 2025-02-26 20:40:37 -- STEP: 29944/36938 -- GLOBAL_STEP: 1396650 | > loss_text_ce: 0.02925078757107258 (0.030776652234329384) | > loss_mel_ce: 2.8654143810272217 (nan) | > loss: 0.011486767791211605 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13702832518686522) | > loader_time: 0.0048 (0.004672145042513711)  --> TIME: 2025-02-26 20:40:46 -- STEP: 29994/36938 -- GLOBAL_STEP: 1396700 | > loss_text_ce: 0.03310178592801094 (0.03077576022629384) | > loss_mel_ce: 0.9112266898155212 (nan) | > loss: 0.0037473354022949934 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13703435450750734) | > loader_time: 0.0045 (0.004672155059432404)  --> TIME: 2025-02-26 20:40:56 -- STEP: 30044/36938 -- GLOBAL_STEP: 1396750 | > loss_text_ce: 0.03253123164176941 (0.03077734184786248) | > loss_mel_ce: 4.381249904632568 (nan) | > loss: 0.01751500554382801 (nan) | > current_lr: 5e-06 | > step_time: 0.162 (0.13703986917064687) | > loader_time: 0.0047 (0.004672150719167395)  --> TIME: 2025-02-26 20:41:05 -- STEP: 30094/36938 -- GLOBAL_STEP: 1396800 | > loss_text_ce: 0.030019201338291168 (0.030776142026769687) | > loss_mel_ce: 3.398299217224121 (nan) | > loss: 0.013604438863694668 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.13704160066778298) | > loader_time: 0.0047 (0.0046721173733455055)  --> TIME: 2025-02-26 20:41:14 -- STEP: 30144/36938 -- GLOBAL_STEP: 1396850 | > loss_text_ce: 0.03546140715479851 (0.030777970635212412) | > loss_mel_ce: 3.0301787853240967 (nan) | > loss: 0.01216524001210928 (nan) | > current_lr: 5e-06 | > step_time: 0.1441 (0.1370372889195673) | > loader_time: 0.0046 (0.00467204117471246)  --> TIME: 2025-02-26 20:41:24 -- STEP: 30194/36938 -- GLOBAL_STEP: 1396900 | > loss_text_ce: 0.019496824592351913 (0.030778653431657744) | > loss_mel_ce: 2.286540985107422 (nan) | > loss: 0.009150944650173187 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.1370427878067083) | > loader_time: 0.0045 (0.004671967873677855)  --> TIME: 2025-02-26 20:41:33 -- STEP: 30244/36938 -- GLOBAL_STEP: 1396950 | > loss_text_ce: 0.026521537452936172 (0.030780824550348414) | > loss_mel_ce: 3.050426483154297 (nan) | > loss: 0.012210111133754253 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.13704645580092106) | > loader_time: 0.0046 (0.00467194143607336)  --> TIME: 2025-02-26 20:41:42 -- STEP: 30294/36938 -- GLOBAL_STEP: 1397000 | > loss_text_ce: 0.026925483718514442 (0.030781341829642327) | > loss_mel_ce: 3.004995107650757 (nan) | > loss: 0.0120314322412014 (nan) | > current_lr: 5e-06 | > step_time: 0.1531 (0.13704068442575443) | > loader_time: 0.0046 (0.004672003766684016)  --> TIME: 2025-02-26 20:41:52 -- STEP: 30344/36938 -- GLOBAL_STEP: 1397050 | > loss_text_ce: 0.026746639981865883 (0.030781853346533078) | > loss_mel_ce: 2.287242889404297 (nan) | > loss: 0.009182498790323734 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13704686754302514) | > loader_time: 0.0046 (0.004672124317947623)  --> TIME: 2025-02-26 20:42:01 -- STEP: 30394/36938 -- GLOBAL_STEP: 1397100 | > loss_text_ce: 0.037261396646499634 (0.030781113926142686) | > loss_mel_ce: 2.6740596294403076 (nan) | > loss: 0.010759211145341396 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13703965199435464) | > loader_time: 0.0045 (0.0046720789115335)  --> TIME: 2025-02-26 20:42:10 -- STEP: 30444/36938 -- GLOBAL_STEP: 1397150 | > loss_text_ce: 0.031133955344557762 (0.03077890810751209) | > loss_mel_ce: 2.121305227279663 (nan) | > loss: 0.008541425690054893 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13703962871145434) | > loader_time: 0.0052 (0.004672078293141396)  --> TIME: 2025-02-26 20:42:20 -- STEP: 30494/36938 -- GLOBAL_STEP: 1397200 | > loss_text_ce: 0.04426198452711105 (0.030780703772090662) | > loss_mel_ce: 2.5832629203796387 (nan) | > loss: 0.010426686145365238 (nan) | > current_lr: 5e-06 | > step_time: 0.1271 (0.1370456241549636) | > loader_time: 0.0053 (0.0046722354862020294)  --> TIME: 2025-02-26 20:42:29 -- STEP: 30544/36938 -- GLOBAL_STEP: 1397250 | > loss_text_ce: 0.03539493307471275 (0.03077845179688733) | > loss_mel_ce: 2.498065233230591 (nan) | > loss: 0.010053413920104504 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13703823228541964) | > loader_time: 0.0048 (0.004672294709931005)  --> TIME: 2025-02-26 20:42:38 -- STEP: 30594/36938 -- GLOBAL_STEP: 1397300 | > loss_text_ce: 0.02499634400010109 (0.030781702022669416) | > loss_mel_ce: 2.4655089378356934 (nan) | > loss: 0.009882957674562931 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13703839212506155) | > loader_time: 0.0046 (0.004672425349818393)  --> TIME: 2025-02-26 20:42:47 -- STEP: 30644/36938 -- GLOBAL_STEP: 1397350 | > loss_text_ce: 0.03610294684767723 (0.03078118564494208) | > loss_mel_ce: 3.111863136291504 (nan) | > loss: 0.012491930276155472 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.1370322251892528) | > loader_time: 0.0045 (0.004672405435310432)  --> TIME: 2025-02-26 20:42:57 -- STEP: 30694/36938 -- GLOBAL_STEP: 1397400 | > loss_text_ce: 0.055816229432821274 (0.03078011217103029) | > loss_mel_ce: 2.808759927749634 (nan) | > loss: 0.011367365717887878 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13702811129576306) | > loader_time: 0.0045 (0.004672298091486507)  --> TIME: 2025-02-26 20:43:06 -- STEP: 30744/36938 -- GLOBAL_STEP: 1397450 | > loss_text_ce: 0.027476025745272636 (0.03078114109195542) | > loss_mel_ce: 2.6361782550811768 (nan) | > loss: 0.010570057667791843 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13702501249971125) | > loader_time: 0.0045 (0.004672266769744199)  --> TIME: 2025-02-26 20:43:15 -- STEP: 30794/36938 -- GLOBAL_STEP: 1397500 | > loss_text_ce: 0.03293794393539429 (0.03077899481132619) | > loss_mel_ce: 2.9388580322265625 (nan) | > loss: 0.011792842298746109 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.1370245829072657) | > loader_time: 0.0048 (0.004672170908658427)  --> TIME: 2025-02-26 20:43:25 -- STEP: 30844/36938 -- GLOBAL_STEP: 1397550 | > loss_text_ce: 0.03823448717594147 (0.030781334786485762) | > loss_mel_ce: 2.446206569671631 (nan) | > loss: 0.009858893230557442 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13702566373087116) | > loader_time: 0.0046 (0.0046721021190040315)  --> TIME: 2025-02-26 20:43:34 -- STEP: 30894/36938 -- GLOBAL_STEP: 1397600 | > loss_text_ce: 0.030463652685284615 (0.030782039124100085) | > loss_mel_ce: 3.0404250621795654 (nan) | > loss: 0.012186067178845406 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.13702381109430567) | > loader_time: 0.0047 (0.004672052521161409)  --> TIME: 2025-02-26 20:43:43 -- STEP: 30944/36938 -- GLOBAL_STEP: 1397650 | > loss_text_ce: 0.032655347138643265 (0.030780262226968273) | > loss_mel_ce: 2.524421453475952 (nan) | > loss: 0.010147130116820335 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13702168960544314) | > loader_time: 0.0046 (0.004671980770383779)  --> TIME: 2025-02-26 20:43:52 -- STEP: 30994/36938 -- GLOBAL_STEP: 1397700 | > loss_text_ce: 0.0353991761803627 (0.030779622875933714) | > loss_mel_ce: 3.19844651222229 (nan) | > loss: 0.012832721695303917 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.1370087617319199) | > loader_time: 0.0047 (0.004671937743794534)  --> TIME: 2025-02-26 20:44:02 -- STEP: 31044/36938 -- GLOBAL_STEP: 1397750 | > loss_text_ce: 0.03576764091849327 (0.030779697098585645) | > loss_mel_ce: 2.9435267448425293 (nan) | > loss: 0.011822597123682499 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.1370127258219695) | > loader_time: 0.0046 (0.004671927925975438)  --> TIME: 2025-02-26 20:44:11 -- STEP: 31094/36938 -- GLOBAL_STEP: 1397800 | > loss_text_ce: 0.02805292047560215 (0.03077684999281251) | > loss_mel_ce: 2.4903969764709473 (nan) | > loss: 0.009993849322199821 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13701245136381363) | > loader_time: 0.0047 (0.0046719028197232315)  --> TIME: 2025-02-26 20:44:21 -- STEP: 31144/36938 -- GLOBAL_STEP: 1397850 | > loss_text_ce: 0.03486430272459984 (0.030775564398276004) | > loss_mel_ce: 2.9543368816375732 (nan) | > loss: 0.011861910112202168 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.13702111751139845) | > loader_time: 0.0048 (0.004671880695466489)  --> TIME: 2025-02-26 20:44:30 -- STEP: 31194/36938 -- GLOBAL_STEP: 1397900 | > loss_text_ce: 0.029789192602038383 (0.030776251226832807) | > loss_mel_ce: 1.9528988599777222 (nan) | > loss: 0.007867810316383839 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13702911824502104) | > loader_time: 0.0046 (0.0046718285512853935)  --> TIME: 2025-02-26 20:44:39 -- STEP: 31244/36938 -- GLOBAL_STEP: 1397950 | > loss_text_ce: 0.024804212152957916 (0.030774895063586646) | > loss_mel_ce: 2.551471710205078 (nan) | > loss: 0.010223316960036755 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.1370266659924306) | > loader_time: 0.0045 (0.004671727400738096)  --> TIME: 2025-02-26 20:44:49 -- STEP: 31294/36938 -- GLOBAL_STEP: 1398000 | > loss_text_ce: 0.025925027206540108 (0.03077431659304908) | > loss_mel_ce: 3.985243082046509 (nan) | > loss: 0.015917334705591202 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.13702840119221413) | > loader_time: 0.0047 (0.004671634062567547)  --> TIME: 2025-02-26 20:44:58 -- STEP: 31344/36938 -- GLOBAL_STEP: 1398050 | > loss_text_ce: 0.03160065785050392 (0.03077446116134108) | > loss_mel_ce: 2.9055228233337402 (nan) | > loss: 0.011655252426862717 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13701888935611123) | > loader_time: 0.0048 (0.004671508671678779)  --> TIME: 2025-02-26 20:45:07 -- STEP: 31394/36938 -- GLOBAL_STEP: 1398100 | > loss_text_ce: 0.0378418043255806 (0.0307746200757237) | > loss_mel_ce: 2.616736888885498 (nan) | > loss: 0.010534043423831463 (nan) | > current_lr: 5e-06 | > step_time: 0.1071 (0.13701058631718982) | > loader_time: 0.0045 (0.004671415477951239)  --> TIME: 2025-02-26 20:45:16 -- STEP: 31444/36938 -- GLOBAL_STEP: 1398150 | > loss_text_ce: 0.03356989100575447 (0.030775927129892765) | > loss_mel_ce: 3.609375 (nan) | > loss: 0.014456131495535374 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13701668868097175) | > loader_time: 0.0046 (0.004671387894738046)  --> TIME: 2025-02-26 20:45:26 -- STEP: 31494/36938 -- GLOBAL_STEP: 1398200 | > loss_text_ce: 0.027423091232776642 (0.030777351782481133) | > loss_mel_ce: 2.432910919189453 (nan) | > loss: 0.009763230569660664 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.137015108434771) | > loader_time: 0.0046 (0.004671434709032117)  --> TIME: 2025-02-26 20:45:35 -- STEP: 31544/36938 -- GLOBAL_STEP: 1398250 | > loss_text_ce: 0.034861329942941666 (0.03077813369533351) | > loss_mel_ce: 2.9406139850616455 (nan) | > loss: 0.011807442642748356 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.13701469161895782) | > loader_time: 0.0048 (0.004671327578909597)  --> TIME: 2025-02-26 20:45:44 -- STEP: 31594/36938 -- GLOBAL_STEP: 1398300 | > loss_text_ce: 0.025777161121368408 (0.03077758833620186) | > loss_mel_ce: 2.860476016998291 (nan) | > loss: 0.011453386396169662 (nan) | > current_lr: 5e-06 | > step_time: 0.1791 (0.1370133363031681) | > loader_time: 0.005 (0.004671393206295071)  --> TIME: 2025-02-26 20:45:54 -- STEP: 31644/36938 -- GLOBAL_STEP: 1398350 | > loss_text_ce: 0.035458631813526154 (0.03077806473872162) | > loss_mel_ce: 0.06437377631664276 (nan) | > loss: 0.0003961603797506541 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.1370126175120979) | > loader_time: 0.0046 (0.0046713697881678165)  --> TIME: 2025-02-26 20:46:03 -- STEP: 31694/36938 -- GLOBAL_STEP: 1398400 | > loss_text_ce: 0.01807977631688118 (0.03077750749973426) | > loss_mel_ce: 2.1944758892059326 (nan) | > loss: 0.008779983036220074 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.137004326854798) | > loader_time: 0.0046 (0.004671230514455601)  --> TIME: 2025-02-26 20:46:12 -- STEP: 31744/36938 -- GLOBAL_STEP: 1398450 | > loss_text_ce: 0.028445487841963768 (0.030776007817196162) | > loss_mel_ce: 2.813446521759033 (nan) | > loss: 0.011277349665760994 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.13699957931925116) | > loader_time: 0.0046 (0.004671209769683986)  --> TIME: 2025-02-26 20:46:21 -- STEP: 31794/36938 -- GLOBAL_STEP: 1398500 | > loss_text_ce: 0.02506583370268345 (0.030776867632380196) | > loss_mel_ce: 2.662808656692505 (nan) | > loss: 0.010666169226169586 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13700655529227287) | > loader_time: 0.0046 (0.004671151520898904)  --> TIME: 2025-02-26 20:46:31 -- STEP: 31844/36938 -- GLOBAL_STEP: 1398550 | > loss_text_ce: 0.02984386309981346 (0.030777536575279325) | > loss_mel_ce: 2.8923120498657227 (nan) | > loss: 0.011595857329666615 (nan) | > current_lr: 5e-06 | > step_time: 0.1738 (0.13700801888807315) | > loader_time: 0.0048 (0.004671082516408963)  --> TIME: 2025-02-26 20:46:40 -- STEP: 31894/36938 -- GLOBAL_STEP: 1398600 | > loss_text_ce: 0.024061093106865883 (0.03077438021552747) | > loss_mel_ce: 2.3499915599823 (nan) | > loss: 0.009420844726264477 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13700420137198502) | > loader_time: 0.0046 (0.004670998598181169)  --> TIME: 2025-02-26 20:46:49 -- STEP: 31944/36938 -- GLOBAL_STEP: 1398650 | > loss_text_ce: 0.027704734355211258 (0.030775365613317784) | > loss_mel_ce: 1.5453038215637207 (nan) | > loss: 0.006242097821086645 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13699648075315274) | > loader_time: 0.0046 (0.004671022516129286)  --> TIME: 2025-02-26 20:46:59 -- STEP: 31994/36938 -- GLOBAL_STEP: 1398700 | > loss_text_ce: 0.03555121272802353 (0.030772535861068383) | > loss_mel_ce: 2.7706823348999023 (nan) | > loss: 0.011135848239064217 (nan) | > current_lr: 5e-06 | > step_time: 0.1634 (0.13700512512673607) | > loader_time: 0.0047 (0.00467104169438167)  --> TIME: 2025-02-26 20:47:08 -- STEP: 32044/36938 -- GLOBAL_STEP: 1398750 | > loss_text_ce: 0.030697746202349663 (0.03077006132244906) | > loss_mel_ce: 1.670997142791748 (nan) | > loss: 0.0067527578212320805 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13700181411723783) | > loader_time: 0.0046 (0.004671074867605677)  --> TIME: 2025-02-26 20:47:17 -- STEP: 32094/36938 -- GLOBAL_STEP: 1398800 | > loss_text_ce: 0.025071216747164726 (0.030771141848377447) | > loss_mel_ce: 2.5090153217315674 (nan) | > loss: 0.010055899620056152 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13700164082028624) | > loader_time: 0.0045 (0.004671144338384218)  --> TIME: 2025-02-26 20:47:26 -- STEP: 32144/36938 -- GLOBAL_STEP: 1398850 | > loss_text_ce: 0.023284677416086197 (0.030771335638666544) | > loss_mel_ce: 2.0571463108062744 (nan) | > loss: 0.008255679160356522 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13699806099984846) | > loader_time: 0.0044 (0.0046710500511261575)  --> TIME: 2025-02-26 20:47:36 -- STEP: 32194/36938 -- GLOBAL_STEP: 1398900 | > loss_text_ce: 0.03314211592078209 (0.03077102318347843) | > loss_mel_ce: 2.717564344406128 (nan) | > loss: 0.010915501974523067 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.1369986024607632) | > loader_time: 0.0046 (0.00467100956280454)  --> TIME: 2025-02-26 20:47:45 -- STEP: 32244/36938 -- GLOBAL_STEP: 1398950 | > loss_text_ce: 0.031399037688970566 (0.030770914341262303) | > loss_mel_ce: 2.7641215324401855 (nan) | > loss: 0.011093336157500744 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13699590432231434) | > loader_time: 0.0046 (0.0046710933856375)  --> TIME: 2025-02-26 20:47:54 -- STEP: 32294/36938 -- GLOBAL_STEP: 1399000 | > loss_text_ce: 0.028226114809513092 (0.030770107028191933) | > loss_mel_ce: 0.006766987033188343 (nan) | > loss: 0.00013886152009945363 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13699780189268368) | > loader_time: 0.0045 (0.0046710137900991995)  --> TIME: 2025-02-26 20:48:04 -- STEP: 32344/36938 -- GLOBAL_STEP: 1399050 | > loss_text_ce: 0.03004498966038227 (0.03077072684812105) | > loss_mel_ce: 2.8293142318725586 (nan) | > loss: 0.011346664279699326 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13700039947560858) | > loader_time: 0.0046 (0.004670922410626715)  --> TIME: 2025-02-26 20:48:13 -- STEP: 32394/36938 -- GLOBAL_STEP: 1399100 | > loss_text_ce: 0.03985514119267464 (0.030770671509649385) | > loss_mel_ce: 0.03931541368365288 (nan) | > loss: 0.00031416889396496117 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13699502850591727) | > loader_time: 0.0046 (0.004670839512238517)  --> TIME: 2025-02-26 20:48:23 -- STEP: 32444/36938 -- GLOBAL_STEP: 1399150 | > loss_text_ce: 0.03304161876440048 (0.030771037726111904) | > loss_mel_ce: 2.5559439659118652 (nan) | > loss: 0.01027375366538763 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13700195490366102) | > loader_time: 0.0046 (0.004670873264131209)  --> TIME: 2025-02-26 20:48:32 -- STEP: 32494/36938 -- GLOBAL_STEP: 1399200 | > loss_text_ce: 0.02794637903571129 (0.030770185214604402) | > loss_mel_ce: 2.7319560050964355 (nan) | > loss: 0.0109519949182868 (nan) | > current_lr: 5e-06 | > step_time: 0.1733 (0.1369990266608701) | > loader_time: 0.0048 (0.004670852513330027)  --> TIME: 2025-02-26 20:48:41 -- STEP: 32544/36938 -- GLOBAL_STEP: 1399250 | > loss_text_ce: 0.03205849975347519 (0.03076899408545899) | > loss_mel_ce: 0.6997255682945251 (nan) | > loss: 0.0029039049986749887 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13699557305466717) | > loader_time: 0.0045 (0.004670716653648502)  --> TIME: 2025-02-26 20:48:51 -- STEP: 32594/36938 -- GLOBAL_STEP: 1399300 | > loss_text_ce: 0.03851202875375748 (0.030771090996564193) | > loss_mel_ce: 2.7813847064971924 (nan) | > loss: 0.011190067045390606 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13699645797716523) | > loader_time: 0.0046 (0.004670620227932402)  --> TIME: 2025-02-26 20:49:00 -- STEP: 32644/36938 -- GLOBAL_STEP: 1399350 | > loss_text_ce: 0.026713086292147636 (0.030770229117985572) | > loss_mel_ce: 3.066054105758667 (nan) | > loss: 0.01227288693189621 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.13699476622641335) | > loader_time: 0.0047 (0.00467060696419921)  --> TIME: 2025-02-26 20:49:09 -- STEP: 32694/36938 -- GLOBAL_STEP: 1399400 | > loss_text_ce: 0.037826452404260635 (0.030769644471197826) | > loss_mel_ce: 2.659268379211426 (nan) | > loss: 0.010702758096158504 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13700090860922468) | > loader_time: 0.0045 (0.004670555076592126)  --> TIME: 2025-02-26 20:49:19 -- STEP: 32744/36938 -- GLOBAL_STEP: 1399450 | > loss_text_ce: 0.02368004247546196 (0.03077188847203846) | > loss_mel_ce: 1.9969037771224976 (nan) | > loss: 0.008018190041184425 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.1369995999068305) | > loader_time: 0.0044 (0.004670507825443228)  --> TIME: 2025-02-26 20:49:28 -- STEP: 32794/36938 -- GLOBAL_STEP: 1399500 | > loss_text_ce: 0.03174997493624687 (0.030772226246929445) | > loss_mel_ce: 3.292370080947876 (nan) | > loss: 0.013190953060984612 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13699615189249484) | > loader_time: 0.0048 (0.004670416682844103)  --> TIME: 2025-02-26 20:49:38 -- STEP: 32844/36938 -- GLOBAL_STEP: 1399550 | > loss_text_ce: 0.027166716754436493 (0.030770456449629106) | > loss_mel_ce: 2.394707441329956 (nan) | > loss: 0.009610611945390701 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.1370041845864303) | > loader_time: 0.0046 (0.004670385219135877)  --> TIME: 2025-02-26 20:49:47 -- STEP: 32894/36938 -- GLOBAL_STEP: 1399600 | > loss_text_ce: 0.03800717368721962 (0.030770866017526013) | > loss_mel_ce: 0.0391693115234375 (nan) | > loss: 0.0003062558826059103 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13701139328044584) | > loader_time: 0.0046 (0.004670327410057501)  --> TIME: 2025-02-26 20:49:57 -- STEP: 32944/36938 -- GLOBAL_STEP: 1399650 | > loss_text_ce: 0.0202733613550663 (0.030771800456622304) | > loss_mel_ce: 2.299046039581299 (nan) | > loss: 0.009203649125993252 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13701090074789898) | > loader_time: 0.0045 (0.0046702371950923085)  --> TIME: 2025-02-26 20:50:06 -- STEP: 32994/36938 -- GLOBAL_STEP: 1399700 | > loss_text_ce: 0.03600427880883217 (0.0307706508495397) | > loss_mel_ce: 2.2885243892669678 (nan) | > loss: 0.009224320761859417 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13701528897175475) | > loader_time: 0.0074 (0.004670334886968405)  --> TIME: 2025-02-26 20:50:16 -- STEP: 33044/36938 -- GLOBAL_STEP: 1399750 | > loss_text_ce: 0.03587653487920761 (0.03076957457889914) | > loss_mel_ce: 3.2443015575408936 (nan) | > loss: 0.01301658060401678 (nan) | > current_lr: 5e-06 | > step_time: 0.1634 (0.1370236212138545) | > loader_time: 0.0048 (0.004670387960320138)  --> TIME: 2025-02-26 20:50:25 -- STEP: 33094/36938 -- GLOBAL_STEP: 1399800 | > loss_text_ce: 0.02421090565621853 (0.030768477390498036) | > loss_mel_ce: 2.9059932231903076 (nan) | > loss: 0.011627795174717903 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.1370241701145477) | > loader_time: 0.005 (0.004670375588077535)  --> TIME: 2025-02-26 20:50:35 -- STEP: 33144/36938 -- GLOBAL_STEP: 1399850 | > loss_text_ce: 0.027030307799577713 (0.030769898443930264) | > loss_mel_ce: 3.0748140811920166 (nan) | > loss: 0.012308906763792038 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.1370342794518941) | > loader_time: 0.0046 (0.00467037488491732)  --> TIME: 2025-02-26 20:50:44 -- STEP: 33194/36938 -- GLOBAL_STEP: 1399900 | > loss_text_ce: 0.028222285211086273 (0.030768748151970924) | > loss_mel_ce: 3.398043394088745 (nan) | > loss: 0.01359629351645708 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13703878588681467) | > loader_time: 0.0045 (0.004670330169021768)  --> TIME: 2025-02-26 20:50:54 -- STEP: 33244/36938 -- GLOBAL_STEP: 1399950 | > loss_text_ce: 0.027758436277508736 (0.030768327508320623) | > loss_mel_ce: 0.03021240234375 (nan) | > loss: 0.00023004302056506276 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.13703969177277894) | > loader_time: 0.0046 (0.004670268332337151)  --> TIME: 2025-02-26 20:51:03 -- STEP: 33294/36938 -- GLOBAL_STEP: 1400000 | > loss_text_ce: 0.04133186861872673 (0.03076858268687351) | > loss_mel_ce: 1.3598800897598267 (nan) | > loss: 0.005560365505516529 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13704478637573766) | > loader_time: 0.0046 (0.0046702689678237794) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1400000.pth  --> TIME: 2025-02-26 20:51:19 -- STEP: 33344/36938 -- GLOBAL_STEP: 1400050 | > loss_text_ce: 0.02736605890095234 (0.03076749902309689) | > loss_mel_ce: 2.9022819995880127 (nan) | > loss: 0.011625588871538639 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.1370536274635988) | > loader_time: 0.0046 (0.004670332073307315)  --> TIME: 2025-02-26 20:51:29 -- STEP: 33394/36938 -- GLOBAL_STEP: 1400100 | > loss_text_ce: 0.03997042402625084 (0.030769667506557408) | > loss_mel_ce: 2.052457809448242 (nan) | > loss: 0.00830328743904829 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.1370514587291665) | > loader_time: 0.0045 (0.004670264849857266)  --> TIME: 2025-02-26 20:51:38 -- STEP: 33444/36938 -- GLOBAL_STEP: 1400150 | > loss_text_ce: 0.03450190648436546 (0.03077205074605467) | > loss_mel_ce: 1.3369954824447632 (nan) | > loss: 0.005442450288683176 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13704437952751222) | > loader_time: 0.0046 (0.00467019331482321)  --> TIME: 2025-02-26 20:51:47 -- STEP: 33494/36938 -- GLOBAL_STEP: 1400200 | > loss_text_ce: 0.02655107155442238 (0.030773214634484857) | > loss_mel_ce: 2.1300384998321533 (nan) | > loss: 0.008557895198464394 (nan) | > current_lr: 5e-06 | > step_time: 0.162 (0.137053184354955) | > loader_time: 0.0047 (0.004670197923710054)  --> TIME: 2025-02-26 20:51:57 -- STEP: 33544/36938 -- GLOBAL_STEP: 1400250 | > loss_text_ce: 0.027450188994407654 (0.030771367060452675) | > loss_mel_ce: 2.5227746963500977 (nan) | > loss: 0.010119940154254436 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13705172366468324) | > loader_time: 0.0045 (0.004670160775704347)  --> TIME: 2025-02-26 20:52:06 -- STEP: 33594/36938 -- GLOBAL_STEP: 1400300 | > loss_text_ce: 0.029635464772582054 (0.030771322798962712) | > loss_mel_ce: 2.748347520828247 (nan) | > loss: 0.011023742146790028 (nan) | > current_lr: 5e-06 | > step_time: 0.1898 (0.13706325683422407) | > loader_time: 0.0049 (0.0046702870132029365)  --> TIME: 2025-02-26 20:52:16 -- STEP: 33644/36938 -- GLOBAL_STEP: 1400350 | > loss_text_ce: 0.04176272451877594 (0.030771585134974803) | > loss_mel_ce: 2.6349570751190186 (nan) | > loss: 0.010621905326843262 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13706485221820056) | > loader_time: 0.0045 (0.004670271364946089)  --> TIME: 2025-02-26 20:52:25 -- STEP: 33694/36938 -- GLOBAL_STEP: 1400400 | > loss_text_ce: 0.03710286691784859 (0.030772188982665424) | > loss_mel_ce: 2.5793368816375732 (nan) | > loss: 0.010382697917521 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.1370579609356074) | > loader_time: 0.0046 (0.004670336903567388)  --> TIME: 2025-02-26 20:52:34 -- STEP: 33744/36938 -- GLOBAL_STEP: 1400450 | > loss_text_ce: 0.030941924080252647 (0.030771716792147345) | > loss_mel_ce: 2.835787057876587 (nan) | > loss: 0.0113759096711874 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13705579068859827) | > loader_time: 0.0046 (0.004670322612599215)  --> TIME: 2025-02-26 20:52:43 -- STEP: 33794/36938 -- GLOBAL_STEP: 1400500 | > loss_text_ce: 0.021822955459356308 (0.030771300121453404) | > loss_mel_ce: 0.590087890625 (nan) | > loss: 0.00242821779102087 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.1370531582173407) | > loader_time: 0.0046 (0.004670335525887013)  --> TIME: 2025-02-26 20:52:53 -- STEP: 33844/36938 -- GLOBAL_STEP: 1400550 | > loss_text_ce: 0.03744380176067352 (0.030770493489664794) | > loss_mel_ce: 2.7013707160949707 (nan) | > loss: 0.010868312790989876 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13705134419963333) | > loader_time: 0.0045 (0.0046702801596539365)  --> TIME: 2025-02-26 20:53:02 -- STEP: 33894/36938 -- GLOBAL_STEP: 1400600 | > loss_text_ce: 0.022829722613096237 (0.030770160580443434) | > loss_mel_ce: 2.406662702560425 (nan) | > loss: 0.009640843607485294 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13705108931347937) | > loader_time: 0.0046 (0.004670283312838913)  --> TIME: 2025-02-26 20:53:11 -- STEP: 33944/36938 -- GLOBAL_STEP: 1400650 | > loss_text_ce: 0.0356215164065361 (0.030767670191466975) | > loss_mel_ce: 1.8875640630722046 (nan) | > loss: 0.0076316893100738525 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13704917482433096) | > loader_time: 0.0046 (0.004670263832817379)  --> TIME: 2025-02-26 20:53:21 -- STEP: 33994/36938 -- GLOBAL_STEP: 1400700 | > loss_text_ce: 0.042619310319423676 (0.030769109824347295) | > loss_mel_ce: 2.8719072341918945 (nan) | > loss: 0.011565581895411015 (nan) | > current_lr: 5e-06 | > step_time: 0.1582 (0.13705323234392613) | > loader_time: 0.0048 (0.004670291821690266)  --> TIME: 2025-02-26 20:53:30 -- STEP: 34044/36938 -- GLOBAL_STEP: 1400750 | > loss_text_ce: 0.030339442193508148 (0.030766999556701813) | > loss_mel_ce: 2.6793785095214844 (nan) | > loss: 0.010752850212156773 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.1370481151473367) | > loader_time: 0.0045 (0.004670204461888475)  --> TIME: 2025-02-26 20:53:40 -- STEP: 34094/36938 -- GLOBAL_STEP: 1400800 | > loss_text_ce: 0.05025627464056015 (0.03076768356335841) | > loss_mel_ce: 2.0220675468444824 (nan) | > loss: 0.008223507553339005 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.1370517516923593) | > loader_time: 0.0046 (0.004670284609364693)  --> TIME: 2025-02-26 20:53:49 -- STEP: 34144/36938 -- GLOBAL_STEP: 1400850 | > loss_text_ce: 0.03344859927892685 (0.030767258416308445) | > loss_mel_ce: 2.2986721992492676 (nan) | > loss: 0.009254448115825653 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13706082933091104) | > loader_time: 0.0045 (0.004670303520914993)  --> TIME: 2025-02-26 20:53:58 -- STEP: 34194/36938 -- GLOBAL_STEP: 1400900 | > loss_text_ce: 0.029382988810539246 (0.030766860810679617) | > loss_mel_ce: 2.739436388015747 (nan) | > loss: 0.010987378656864166 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13705782141610054) | > loader_time: 0.0045 (0.004670247631672083)  --> TIME: 2025-02-26 20:54:08 -- STEP: 34244/36938 -- GLOBAL_STEP: 1400950 | > loss_text_ce: 0.03345302864909172 (0.0307672409250147) | > loss_mel_ce: 2.8421144485473633 (nan) | > loss: 0.011410982348024845 (nan) | > current_lr: 5e-06 | > step_time: 0.1881 (0.13705360061612015) | > loader_time: 0.0049 (0.0046701797145700516)  --> TIME: 2025-02-26 20:54:17 -- STEP: 34294/36938 -- GLOBAL_STEP: 1401000 | > loss_text_ce: 0.028424901887774467 (0.03076745238505831) | > loss_mel_ce: 2.7661945819854736 (nan) | > loss: 0.011089760810136795 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13706303140599585) | > loader_time: 0.0046 (0.004670172222377945)  --> TIME: 2025-02-26 20:54:27 -- STEP: 34344/36938 -- GLOBAL_STEP: 1401050 | > loss_text_ce: 0.03391406685113907 (0.03076952275028082) | > loss_mel_ce: 2.9784562587738037 (nan) | > loss: 0.011953851208090782 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13706066187461755) | > loader_time: 0.0046 (0.00467012484201929)  --> TIME: 2025-02-26 20:54:36 -- STEP: 34394/36938 -- GLOBAL_STEP: 1401100 | > loss_text_ce: 0.029385603964328766 (0.030768451742965327) | > loss_mel_ce: 2.646162509918213 (nan) | > loss: 0.010617255233228207 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13706162389422774) | > loader_time: 0.0045 (0.0046700561657307986)  --> TIME: 2025-02-26 20:54:46 -- STEP: 34444/36938 -- GLOBAL_STEP: 1401150 | > loss_text_ce: 0.0336587093770504 (0.030768183984593838) | > loss_mel_ce: 3.133133888244629 (nan) | > loss: 0.012566638179123402 (nan) | > current_lr: 5e-06 | > step_time: 0.1869 (0.13707000805572864) | > loader_time: 0.0049 (0.004670006987137912)  --> TIME: 2025-02-26 20:54:55 -- STEP: 34494/36938 -- GLOBAL_STEP: 1401200 | > loss_text_ce: 0.03528371825814247 (0.03076693358309583) | > loss_mel_ce: 2.0418903827667236 (nan) | > loss: 0.008242755196988583 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13707065757146242) | > loader_time: 0.0046 (0.004669956589475017)  --> TIME: 2025-02-26 20:55:04 -- STEP: 34544/36938 -- GLOBAL_STEP: 1401250 | > loss_text_ce: 0.029145020991563797 (0.030765181587495766) | > loss_mel_ce: 2.6078903675079346 (nan) | > loss: 0.010464427061378956 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13706572511893098) | > loader_time: 0.0046 (0.004669851785238394)  --> TIME: 2025-02-26 20:55:14 -- STEP: 34594/36938 -- GLOBAL_STEP: 1401300 | > loss_text_ce: 0.023569712415337563 (0.0307648865214466) | > loss_mel_ce: 3.117093801498413 (nan) | > loss: 0.012462951242923737 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13707105571178863) | > loader_time: 0.0046 (0.004669919347269606)  --> TIME: 2025-02-26 20:55:23 -- STEP: 34644/36938 -- GLOBAL_STEP: 1401350 | > loss_text_ce: 0.019747672602534294 (0.03076496553938639) | > loss_mel_ce: 1.5652414560317993 (nan) | > loss: 0.006289639510214329 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13706852949019507) | > loader_time: 0.0046 (0.004669888515613358)  --> TIME: 2025-02-26 20:55:32 -- STEP: 34694/36938 -- GLOBAL_STEP: 1401400 | > loss_text_ce: 0.027383925393223763 (0.03076530106020413) | > loss_mel_ce: 3.343491315841675 (nan) | > loss: 0.013376490212976933 (nan) | > current_lr: 5e-06 | > step_time: 0.1268 (0.13706835195737588) | > loader_time: 0.0046 (0.004669874664300485)  --> TIME: 2025-02-26 20:55:42 -- STEP: 34744/36938 -- GLOBAL_STEP: 1401450 | > loss_text_ce: 0.02957487665116787 (0.030764959581608988) | > loss_mel_ce: 3.0989274978637695 (nan) | > loss: 0.012414692901074886 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1370695504811747) | > loader_time: 0.0046 (0.004669910829901121)  --> TIME: 2025-02-26 20:55:51 -- STEP: 34794/36938 -- GLOBAL_STEP: 1401500 | > loss_text_ce: 0.0365101657807827 (0.030764276356076394) | > loss_mel_ce: 3.179866313934326 (nan) | > loss: 0.012763399630784988 (nan) | > current_lr: 5e-06 | > step_time: 0.1537 (0.13706076612498463) | > loader_time: 0.0046 (0.0046698767583713005)  --> TIME: 2025-02-26 20:56:00 -- STEP: 34844/36938 -- GLOBAL_STEP: 1401550 | > loss_text_ce: 0.02663537859916687 (0.03076460012275106) | > loss_mel_ce: 2.0791070461273193 (nan) | > loss: 0.008356121368706226 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.1370615742740575) | > loader_time: 0.0047 (0.004669860951348436)  --> TIME: 2025-02-26 20:56:10 -- STEP: 34894/36938 -- GLOBAL_STEP: 1401600 | > loss_text_ce: 0.02425987645983696 (0.030766592207578512) | > loss_mel_ce: 1.6873213052749634 (nan) | > loss: 0.00679198931902647 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13706441860168525) | > loader_time: 0.0047 (0.004669829269545181)  --> TIME: 2025-02-26 20:56:19 -- STEP: 34944/36938 -- GLOBAL_STEP: 1401650 | > loss_text_ce: 0.033493176102638245 (0.030766479141122478) | > loss_mel_ce: 3.0511298179626465 (nan) | > loss: 0.012240568175911903 (nan) | > current_lr: 5e-06 | > step_time: 0.1853 (0.1370602346915809) | > loader_time: 0.0048 (0.0046698010489071755)  --> TIME: 2025-02-26 20:56:28 -- STEP: 34994/36938 -- GLOBAL_STEP: 1401700 | > loss_text_ce: 0.03861106559634209 (0.03076791925741531) | > loss_mel_ce: 3.238213300704956 (nan) | > loss: 0.0130032729357481 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13705751111986278) | > loader_time: 0.0046 (0.004669831474556238)  --> TIME: 2025-02-26 20:56:38 -- STEP: 35044/36938 -- GLOBAL_STEP: 1401750 | > loss_text_ce: 0.02956484816968441 (0.030768689468965722) | > loss_mel_ce: 2.822119951248169 (nan) | > loss: 0.0113162100315094 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.1370629017694061) | > loader_time: 0.0047 (0.004669806379227462)  --> TIME: 2025-02-26 20:56:47 -- STEP: 35094/36938 -- GLOBAL_STEP: 1401800 | > loss_text_ce: 0.022063473239541054 (0.03076997860879371) | > loss_mel_ce: 0.0034027099609375 (nan) | > loss: 0.00010105629189638421 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.13705927591796319) | > loader_time: 0.0047 (0.004669785302555074)  --> TIME: 2025-02-26 20:56:57 -- STEP: 35144/36938 -- GLOBAL_STEP: 1401850 | > loss_text_ce: 0.028363769873976707 (0.03076958472570662) | > loss_mel_ce: 2.9343974590301514 (nan) | > loss: 0.011756989173591137 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13706984502048766) | > loader_time: 0.0047 (0.004669800031003346)  --> TIME: 2025-02-26 20:57:06 -- STEP: 35194/36938 -- GLOBAL_STEP: 1401900 | > loss_text_ce: 0.022635573521256447 (0.030769074689352895) | > loss_mel_ce: 2.3666293621063232 (nan) | > loss: 0.009481210261583328 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.1370669719122458) | > loader_time: 0.0045 (0.004669783345310439)  --> TIME: 2025-02-26 20:57:15 -- STEP: 35244/36938 -- GLOBAL_STEP: 1401950 | > loss_text_ce: 0.02843041531741619 (0.030768916604084574) | > loss_mel_ce: 3.030097484588623 (nan) | > loss: 0.012137016281485558 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1370692015736009) | > loader_time: 0.0046 (0.004669733525621357)  --> TIME: 2025-02-26 20:57:25 -- STEP: 35294/36938 -- GLOBAL_STEP: 1402000 | > loss_text_ce: 0.028177490457892418 (0.030769115199880126) | > loss_mel_ce: 3.238590955734253 (nan) | > loss: 0.012963367626070976 (nan) | > current_lr: 5e-06 | > step_time: 0.1856 (0.1370728437560231) | > loader_time: 0.0049 (0.004669679395401458)  --> TIME: 2025-02-26 20:57:34 -- STEP: 35344/36938 -- GLOBAL_STEP: 1402050 | > loss_text_ce: 0.035106584429740906 (0.030768820835652546) | > loss_mel_ce: 3.3539459705352783 (nan) | > loss: 0.013448622077703476 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13707197209649136) | > loader_time: 0.0046 (0.0046695888636068055)  --> TIME: 2025-02-26 20:57:44 -- STEP: 35394/36938 -- GLOBAL_STEP: 1402100 | > loss_text_ce: 0.029957100749015808 (0.030768544227050602) | > loss_mel_ce: 2.6884448528289795 (nan) | > loss: 0.01078731007874012 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.13707833724715346) | > loader_time: 0.0049 (0.004669647099049337)  --> TIME: 2025-02-26 20:57:53 -- STEP: 35444/36938 -- GLOBAL_STEP: 1402150 | > loss_text_ce: 0.029454031959176064 (0.030768588862453122) | > loss_mel_ce: 2.461765766143799 (nan) | > loss: 0.009885793551802635 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.1370745600379775) | > loader_time: 0.0049 (0.004669664722979047)  --> TIME: 2025-02-26 20:58:03 -- STEP: 35494/36938 -- GLOBAL_STEP: 1402200 | > loss_text_ce: 0.025318095460534096 (0.03076852469968692) | > loss_mel_ce: 2.584663152694702 (nan) | > loss: 0.010357068851590157 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.1370784811969018) | > loader_time: 0.0046 (0.004669601671284767)  --> TIME: 2025-02-26 20:58:12 -- STEP: 35544/36938 -- GLOBAL_STEP: 1402250 | > loss_text_ce: 0.032503269612789154 (0.030768435715052393) | > loss_mel_ce: 2.6264536380767822 (nan) | > loss: 0.010551417246460915 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.13708271884768097) | > loader_time: 0.0048 (0.004669572234878194)  --> TIME: 2025-02-26 20:58:22 -- STEP: 35594/36938 -- GLOBAL_STEP: 1402300 | > loss_text_ce: 0.027673929929733276 (0.030769043633722258) | > loss_mel_ce: 2.527453899383545 (nan) | > loss: 0.010139396414160728 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13708360460947802) | > loader_time: 0.0045 (0.004669527428240081)  --> TIME: 2025-02-26 20:58:31 -- STEP: 35644/36938 -- GLOBAL_STEP: 1402350 | > loss_text_ce: 0.0271595511585474 (0.030767852663365174) | > loss_mel_ce: 3.156550168991089 (nan) | > loss: 0.01263376884162426 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13708782143871778) | > loader_time: 0.0046 (0.0046695017504272155)  --> TIME: 2025-02-26 20:58:41 -- STEP: 35694/36938 -- GLOBAL_STEP: 1402400 | > loss_text_ce: 0.027754465118050575 (0.030768047583487678) | > loss_mel_ce: 2.605738639831543 (nan) | > loss: 0.010450370609760284 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13709290091608306) | > loader_time: 0.0047 (0.004669530208540855)  --> TIME: 2025-02-26 20:58:50 -- STEP: 35744/36938 -- GLOBAL_STEP: 1402450 | > loss_text_ce: 0.03960057348012924 (0.030766961935138122) | > loss_mel_ce: 3.1070444583892822 (nan) | > loss: 0.012486687861382961 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.1370881106987971) | > loader_time: 0.0046 (0.004669542918807198)  --> TIME: 2025-02-26 20:59:00 -- STEP: 35794/36938 -- GLOBAL_STEP: 1402500 | > loss_text_ce: 0.03385575860738754 (0.030765485295918868) | > loss_mel_ce: 2.814178943634033 (nan) | > loss: 0.011301725171506405 (nan) | > current_lr: 5e-06 | > step_time: 0.1612 (0.13709707948730995) | > loader_time: 0.0048 (0.004669561741531542)  --> TIME: 2025-02-26 20:59:09 -- STEP: 35844/36938 -- GLOBAL_STEP: 1402550 | > loss_text_ce: 0.03233781456947327 (0.030763736504567395) | > loss_mel_ce: 2.7975668907165527 (nan) | > loss: 0.011229781433939934 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13709631927736443) | > loader_time: 0.0052 (0.004669514142459919)  --> TIME: 2025-02-26 20:59:18 -- STEP: 35894/36938 -- GLOBAL_STEP: 1402600 | > loss_text_ce: 0.034584179520606995 (0.03076283196971492) | > loss_mel_ce: 2.7507357597351074 (nan) | > loss: 0.011052858084440231 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.1370945036055939) | > loader_time: 0.0045 (0.0046694510400331945)  --> TIME: 2025-02-26 20:59:28 -- STEP: 35944/36938 -- GLOBAL_STEP: 1402650 | > loss_text_ce: 0.04077337682247162 (0.030763539023431107) | > loss_mel_ce: 2.8082332611083984 (nan) | > loss: 0.011305582709610462 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13709953035964595) | > loader_time: 0.0046 (0.004669437861087051)  --> TIME: 2025-02-26 20:59:37 -- STEP: 35994/36938 -- GLOBAL_STEP: 1402700 | > loss_text_ce: 0.02989358641207218 (0.030762785832090394) | > loss_mel_ce: 2.6950137615203857 (nan) | > loss: 0.010813125409185886 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.1370958294107803) | > loader_time: 0.0046 (0.004669405456621839)  --> TIME: 2025-02-26 20:59:46 -- STEP: 36044/36938 -- GLOBAL_STEP: 1402750 | > loss_text_ce: 0.026523252949118614 (0.03076205376292306) | > loss_mel_ce: 2.725886821746826 (nan) | > loss: 0.010922263376414776 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13709432503820326) | > loader_time: 0.0046 (0.004669412413258351)  --> TIME: 2025-02-26 20:59:55 -- STEP: 36094/36938 -- GLOBAL_STEP: 1402800 | > loss_text_ce: 0.03874455764889717 (0.030763240703126398) | > loss_mel_ce: 3.160379648208618 (nan) | > loss: 0.01269493717700243 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13709419524167127) | > loader_time: 0.0046 (0.004669561494174243)  --> TIME: 2025-02-26 21:00:05 -- STEP: 36144/36938 -- GLOBAL_STEP: 1402850 | > loss_text_ce: 0.026145324110984802 (0.03076373047090811) | > loss_mel_ce: 2.372596263885498 (nan) | > loss: 0.009518816135823727 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.13709656142882565) | > loader_time: 0.0047 (0.004669474356202984)  --> TIME: 2025-02-26 21:00:15 -- STEP: 36194/36938 -- GLOBAL_STEP: 1402900 | > loss_text_ce: 0.03678629547357559 (0.03076377320168189) | > loss_mel_ce: 3.0952908992767334 (nan) | > loss: 0.01242887880653143 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.137105282350175) | > loader_time: 0.0046 (0.004669545190455678)  --> TIME: 2025-02-26 21:00:24 -- STEP: 36244/36938 -- GLOBAL_STEP: 1402950 | > loss_text_ce: 0.03156748041510582 (0.03076290573723374) | > loss_mel_ce: 1.5699942111968994 (nan) | > loss: 0.006355403922498226 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13710772937897553) | > loader_time: 0.0045 (0.00466949416633055)  --> TIME: 2025-02-26 21:00:33 -- STEP: 36294/36938 -- GLOBAL_STEP: 1403000 | > loss_text_ce: 0.021836232393980026 (0.03076313895717578) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.137108648815925) | > loader_time: 0.0046 (0.0046694473030745145)  --> TIME: 2025-02-26 21:00:43 -- STEP: 36344/36938 -- GLOBAL_STEP: 1403050 | > loss_text_ce: 0.030324634164571762 (0.030762234018295444) | > loss_mel_ce: 2.57102108001709 (nan) | > loss: 0.01032280083745718 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13711598251796578) | > loader_time: 0.0045 (0.004669392624537452)  --> TIME: 2025-02-26 21:00:52 -- STEP: 36394/36938 -- GLOBAL_STEP: 1403100 | > loss_text_ce: 0.020847652107477188 (0.03076251803398008) | > loss_mel_ce: 2.709836721420288 (nan) | > loss: 0.010836048983037472 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.1371148618727065) | > loader_time: 0.0046 (0.004669335652702695)  --> TIME: 2025-02-26 21:01:01 -- STEP: 36444/36938 -- GLOBAL_STEP: 1403150 | > loss_text_ce: 0.028924768790602684 (0.030761092074107352) | > loss_mel_ce: 2.446791887283325 (nan) | > loss: 0.00982427317649126 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.1371089032224672) | > loader_time: 0.0046 (0.004669250719451255)  --> TIME: 2025-02-26 21:01:11 -- STEP: 36494/36938 -- GLOBAL_STEP: 1403200 | > loss_text_ce: 0.029147908091545105 (0.030761828527026303) | > loss_mel_ce: 2.7522425651550293 (nan) | > loss: 0.01103726401925087 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.1371125882375013) | > loader_time: 0.0046 (0.004669219923455842)  --> TIME: 2025-02-26 21:01:20 -- STEP: 36544/36938 -- GLOBAL_STEP: 1403250 | > loss_text_ce: 0.03306764364242554 (0.03075960170805211) | > loss_mel_ce: 2.025266170501709 (nan) | > loss: 0.00816799234598875 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.1371073169395631) | > loader_time: 0.0045 (0.004669137423019715)  --> TIME: 2025-02-26 21:01:30 -- STEP: 36594/36938 -- GLOBAL_STEP: 1403300 | > loss_text_ce: 0.03759744018316269 (0.030760526374289752) | > loss_mel_ce: 2.184377908706665 (nan) | > loss: 0.00881736259907484 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13711042209890603) | > loader_time: 0.0046 (0.004669097712074237)  --> TIME: 2025-02-26 21:01:39 -- STEP: 36644/36938 -- GLOBAL_STEP: 1403350 | > loss_text_ce: 0.033014729619026184 (0.030759043789805398) | > loss_mel_ce: 3.1343750953674316 (nan) | > loss: 0.012569008395075798 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13710555640153027) | > loader_time: 0.0044 (0.0046690127602598145)  --> TIME: 2025-02-26 21:01:48 -- STEP: 36694/36938 -- GLOBAL_STEP: 1403400 | > loss_text_ce: 0.03469071164727211 (0.030759444215792353) | > loss_mel_ce: 2.7048118114471436 (nan) | > loss: 0.0108710415661335 (nan) | > current_lr: 5e-06 | > step_time: 0.1443 (0.13710281026586382) | > loader_time: 0.0048 (0.004669005957755415)  --> TIME: 2025-02-26 21:01:57 -- STEP: 36744/36938 -- GLOBAL_STEP: 1403450 | > loss_text_ce: 0.034193988889455795 (0.030760210260650813) | > loss_mel_ce: 0.0066165924072265625 (nan) | > loss: 0.00016194676572922617 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13709977986167504) | > loader_time: 0.0045 (0.004668950132624998)  --> TIME: 2025-02-26 21:02:07 -- STEP: 36794/36938 -- GLOBAL_STEP: 1403500 | > loss_text_ce: 0.02989930845797062 (0.030761448938416043) | > loss_mel_ce: 3.2554969787597656 (nan) | > loss: 0.013037287630140781 (nan) | > current_lr: 5e-06 | > step_time: 0.1783 (0.1371043428828072) | > loader_time: 0.0048 (0.0046689174496273075)  --> TIME: 2025-02-26 21:02:16 -- STEP: 36844/36938 -- GLOBAL_STEP: 1403550 | > loss_text_ce: 0.02728922851383686 (0.03076090681862413) | > loss_mel_ce: 2.5872275829315186 (nan) | > loss: 0.0103750666603446 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13710113725371448) | > loader_time: 0.0047 (0.004668831275218937)  --> TIME: 2025-02-26 21:02:25 -- STEP: 36894/36938 -- GLOBAL_STEP: 1403600 | > loss_text_ce: 0.02872895821928978 (0.030763068487795) | > loss_mel_ce: 2.0009562969207764 (nan) | > loss: 0.008054306730628014 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13710276135236812) | > loader_time: 0.0047 (0.004668821220689554)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.020926671453041615 (+0.0007933685095003438) | > avg_loss_text_ce: 0.029996377001009364 (-0.00015201249003115452) | > avg_loss_mel_ce: 2.448265916935289 (-0.005031135412726595) | > avg_loss: 2.478262291391295 (-0.005183148207053456) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_1403644.pth  > EPOCH: 39/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-26 21:02:59)   --> TIME: 2025-02-26 21:03:00 -- STEP: 6/36938 -- GLOBAL_STEP: 1403650 | > loss_text_ce: 0.04057375714182854 (0.030610020272433758) | > loss_mel_ce: 2.8565802574157715 (2.617476145426432) | > loss: 0.011496643535792828 (0.010508279083296657) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13308310508728027) | > loader_time: 0.0043 (0.0043947696685791016)  --> TIME: 2025-02-26 21:03:09 -- STEP: 56/36938 -- GLOBAL_STEP: 1403700 | > loss_text_ce: 0.03429239243268967 (0.03052432116653238) | > loss_mel_ce: 2.707967519760132 (nan) | > loss: 0.010881984606385231 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13277888298034674) | > loader_time: 0.0047 (0.004727742501667569)  --> TIME: 2025-02-26 21:03:19 -- STEP: 106/36938 -- GLOBAL_STEP: 1403750 | > loss_text_ce: 0.028291983529925346 (0.031055293116226512) | > loss_mel_ce: 2.5912513732910156 (nan) | > loss: 0.010395013727247715 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13437843547677097) | > loader_time: 0.0047 (0.004697817676472214)  --> TIME: 2025-02-26 21:03:28 -- STEP: 156/36938 -- GLOBAL_STEP: 1403800 | > loss_text_ce: 0.021627604961395264 (0.03132388031540009) | > loss_mel_ce: 2.258035659790039 (nan) | > loss: 0.009046283550560474 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.1345784251506512) | > loader_time: 0.0047 (0.004758960161453638)  --> TIME: 2025-02-26 21:03:37 -- STEP: 206/36938 -- GLOBAL_STEP: 1403850 | > loss_text_ce: 0.028458645567297935 (0.03134734422257803) | > loss_mel_ce: 2.5679855346679688 (nan) | > loss: 0.010303350165486336 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13460107914452418) | > loader_time: 0.0047 (0.004750707774486356)  --> TIME: 2025-02-26 21:03:46 -- STEP: 256/36938 -- GLOBAL_STEP: 1403900 | > loss_text_ce: 0.03057241067290306 (0.031202763540932224) | > loss_mel_ce: 2.079470157623291 (nan) | > loss: 0.008373185060918331 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13400751352310186) | > loader_time: 0.0046 (0.004721850156784058)  --> TIME: 2025-02-26 21:03:55 -- STEP: 306/36938 -- GLOBAL_STEP: 1403950 | > loss_text_ce: 0.03349093347787857 (0.03133307457931975) | > loss_mel_ce: 0.1304677277803421 (nan) | > loss: 0.0006506296922452748 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.1346126849355261) | > loader_time: 0.0047 (0.004722793117847317)  --> TIME: 2025-02-26 21:04:05 -- STEP: 356/36938 -- GLOBAL_STEP: 1404000 | > loss_text_ce: 0.022143252193927765 (0.031325120523341785) | > loss_mel_ce: 1.5779756307601929 (nan) | > loss: 0.0063496786169707775 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13484806425116022) | > loader_time: 0.0045 (0.0047215618444292696)  --> TIME: 2025-02-26 21:04:14 -- STEP: 406/36938 -- GLOBAL_STEP: 1404050 | > loss_text_ce: 0.02700752392411232 (0.03133180116078463) | > loss_mel_ce: 2.838623046875 (nan) | > loss: 0.0113715510815382 (nan) | > current_lr: 5e-06 | > step_time: 0.1718 (0.13505845527930785) | > loader_time: 0.0048 (0.004719397117351661)  --> TIME: 2025-02-26 21:04:23 -- STEP: 456/36938 -- GLOBAL_STEP: 1404100 | > loss_text_ce: 0.022271014750003815 (0.03132778337315179) | > loss_mel_ce: 0.014750163070857525 (nan) | > loss: 0.0001469094422645867 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13512431634099836) | > loader_time: 0.0047 (0.004725984314031768)  --> TIME: 2025-02-26 21:04:33 -- STEP: 506/36938 -- GLOBAL_STEP: 1404150 | > loss_text_ce: 0.02913827821612358 (0.03142973704248842) | > loss_mel_ce: 2.581071138381958 (nan) | > loss: 0.010357975028455257 (nan) | > current_lr: 5e-06 | > step_time: 0.1759 (0.1354919262083149) | > loader_time: 0.0055 (0.004732203577818135)  --> TIME: 2025-02-26 21:04:42 -- STEP: 556/36938 -- GLOBAL_STEP: 1404200 | > loss_text_ce: 0.024471284821629524 (0.03137772266833421) | > loss_mel_ce: 2.660881757736206 (nan) | > loss: 0.010656163096427917 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.13542271861069474) | > loader_time: 0.0047 (0.004728790238606843)  --> TIME: 2025-02-26 21:04:51 -- STEP: 606/36938 -- GLOBAL_STEP: 1404250 | > loss_text_ce: 0.043960146605968475 (0.03138664913004136) | > loss_mel_ce: 2.8236520290374756 (nan) | > loss: 0.011379414238035679 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13553333361156683) | > loader_time: 0.0048 (0.00472977570574669)  --> TIME: 2025-02-26 21:05:01 -- STEP: 656/36938 -- GLOBAL_STEP: 1404300 | > loss_text_ce: 0.035080667585134506 (0.03126722230524872) | > loss_mel_ce: 1.8101834058761597 (nan) | > loss: 0.007322476711124182 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.13575873723844187) | > loader_time: 0.0054 (0.004728738491128127)  --> TIME: 2025-02-26 21:05:10 -- STEP: 706/36938 -- GLOBAL_STEP: 1404350 | > loss_text_ce: 0.02778800204396248 (0.03123990216400614) | > loss_mel_ce: 2.249894142150879 (nan) | > loss: 0.00903842132538557 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13613637479776686) | > loader_time: 0.0046 (0.004743074898003849)  --> TIME: 2025-02-26 21:05:20 -- STEP: 756/36938 -- GLOBAL_STEP: 1404400 | > loss_text_ce: 0.03350714594125748 (0.031214291242163197) | > loss_mel_ce: 3.0543389320373535 (nan) | > loss: 0.012253358028829098 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13630726381584432) | > loader_time: 0.0047 (0.004740969529227595)  --> TIME: 2025-02-26 21:05:29 -- STEP: 806/36938 -- GLOBAL_STEP: 1404450 | > loss_text_ce: 0.030914418399333954 (0.03112652269480914) | > loss_mel_ce: 2.4690937995910645 (nan) | > loss: 0.009920667856931686 (nan) | > current_lr: 5e-06 | > step_time: 0.1591 (0.1363952819526046) | > loader_time: 0.0046 (0.004744870786927167)  --> TIME: 2025-02-26 21:05:39 -- STEP: 856/36938 -- GLOBAL_STEP: 1404500 | > loss_text_ce: 0.026764221489429474 (0.03114929015391343) | > loss_mel_ce: 2.4063470363616943 (nan) | > loss: 0.00965520367026329 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13653253199898172) | > loader_time: 0.0046 (0.004737727831457248)  --> TIME: 2025-02-26 21:05:48 -- STEP: 906/36938 -- GLOBAL_STEP: 1404550 | > loss_text_ce: 0.026568565517663956 (0.031218309220642067) | > loss_mel_ce: 0.109100341796875 (nan) | > loss: 0.000538368709385395 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.1365765646330304) | > loader_time: 0.0045 (0.004730965391163255)  --> TIME: 2025-02-26 21:05:57 -- STEP: 956/36938 -- GLOBAL_STEP: 1404600 | > loss_text_ce: 0.011609978042542934 (0.03121997755523539) | > loss_mel_ce: 1.114335298538208 (nan) | > loss: 0.0044680372811853886 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13671408886690015) | > loader_time: 0.0046 (0.00472671940735692)  --> TIME: 2025-02-26 21:06:07 -- STEP: 1006/36938 -- GLOBAL_STEP: 1404650 | > loss_text_ce: 0.029158618301153183 (0.031215715759931872) | > loss_mel_ce: 2.315068244934082 (nan) | > loss: 0.009302487596869469 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13686911889146378) | > loader_time: 0.0046 (0.004724008425566583)  --> TIME: 2025-02-26 21:06:16 -- STEP: 1056/36938 -- GLOBAL_STEP: 1404700 | > loss_text_ce: 0.01540743000805378 (0.031165378666518143) | > loss_mel_ce: 1.876316785812378 (nan) | > loss: 0.007506842724978924 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1369541664466713) | > loader_time: 0.0045 (0.004727040953708406)  --> TIME: 2025-02-26 21:06:26 -- STEP: 1106/36938 -- GLOBAL_STEP: 1404750 | > loss_text_ce: 0.034184809774160385 (0.03109931555251586) | > loss_mel_ce: 3.1231536865234375 (nan) | > loss: 0.012529121711850166 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13685103480251093) | > loader_time: 0.0054 (0.004726321744660129)  --> TIME: 2025-02-26 21:06:35 -- STEP: 1156/36938 -- GLOBAL_STEP: 1404800 | > loss_text_ce: 0.024125706404447556 (0.03106020385831285) | > loss_mel_ce: 2.574953317642212 (nan) | > loss: 0.010313807055354118 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13663043014730952) | > loader_time: 0.0046 (0.004723204873424903)  --> TIME: 2025-02-26 21:06:44 -- STEP: 1206/36938 -- GLOBAL_STEP: 1404850 | > loss_text_ce: 0.0236927829682827 (0.031044234569565968) | > loss_mel_ce: 0.1241302490234375 (nan) | > loss: 0.0005865994025953114 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13670306379719924) | > loader_time: 0.0046 (0.004719221572180085)  --> TIME: 2025-02-26 21:06:54 -- STEP: 1256/36938 -- GLOBAL_STEP: 1404900 | > loss_text_ce: 0.03263840451836586 (0.030987755190818363) | > loss_mel_ce: 1.4489034414291382 (nan) | > loss: 0.005879134871065617 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.13688015615104845) | > loader_time: 0.0053 (0.0047202011582198325)  --> TIME: 2025-02-26 21:07:03 -- STEP: 1306/36938 -- GLOBAL_STEP: 1404950 | > loss_text_ce: 0.03300570324063301 (0.030965985109884103) | > loss_mel_ce: 3.364591121673584 (nan) | > loss: 0.013482527807354927 (nan) | > current_lr: 5e-06 | > step_time: 0.187 (0.13687017902298312) | > loader_time: 0.0055 (0.004724694430189138)  --> TIME: 2025-02-26 21:07:12 -- STEP: 1356/36938 -- GLOBAL_STEP: 1405000 | > loss_text_ce: 0.041007865220308304 (0.030927571083438257) | > loss_mel_ce: 2.8444440364837646 (nan) | > loss: 0.01145020592957735 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13683671989975413) | > loader_time: 0.0046 (0.004724040319785949)  --> TIME: 2025-02-26 21:07:22 -- STEP: 1406/36938 -- GLOBAL_STEP: 1405050 | > loss_text_ce: 0.030726689845323563 (0.030916925465388708) | > loss_mel_ce: 2.7385895252227783 (nan) | > loss: 0.010989350266754627 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13718968417871724) | > loader_time: 0.0046 (0.004725146259725676)  --> TIME: 2025-02-26 21:07:31 -- STEP: 1456/36938 -- GLOBAL_STEP: 1405100 | > loss_text_ce: 0.024934781715273857 (0.030896508120829343) | > loss_mel_ce: 2.5790388584136963 (nan) | > loss: 0.01033322885632515 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13711911079647776) | > loader_time: 0.0052 (0.004724552343179881)  --> TIME: 2025-02-26 21:07:41 -- STEP: 1506/36938 -- GLOBAL_STEP: 1405150 | > loss_text_ce: 0.0313730426132679 (0.030885412406601795) | > loss_mel_ce: 2.5490434169769287 (nan) | > loss: 0.010239748284220695 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13716166728679552) | > loader_time: 0.0045 (0.004730576696307206)  --> TIME: 2025-02-26 21:07:50 -- STEP: 1556/36938 -- GLOBAL_STEP: 1405200 | > loss_text_ce: 0.031497783958911896 (0.03089029690344126) | > loss_mel_ce: 2.9743897914886475 (nan) | > loss: 0.011928125284612179 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13723877876767152) | > loader_time: 0.0048 (0.004730011932647614)  --> TIME: 2025-02-26 21:08:00 -- STEP: 1606/36938 -- GLOBAL_STEP: 1405250 | > loss_text_ce: 0.03394771367311478 (0.030920153185098918) | > loss_mel_ce: 2.3890180587768555 (nan) | > loss: 0.009614944458007812 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1372256031072006) | > loader_time: 0.0045 (0.0047373449520334236)  --> TIME: 2025-02-26 21:08:09 -- STEP: 1656/36938 -- GLOBAL_STEP: 1405300 | > loss_text_ce: 0.036995623260736465 (0.03092008471828638) | > loss_mel_ce: 2.5475261211395264 (nan) | > loss: 0.01025603897869587 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.1370795826981032) | > loader_time: 0.0044 (0.004738244889439007)  --> TIME: 2025-02-26 21:08:18 -- STEP: 1706/36938 -- GLOBAL_STEP: 1405350 | > loss_text_ce: 0.030045168474316597 (0.030926894729002733) | > loss_mel_ce: 2.725904941558838 (nan) | > loss: 0.010936311446130276 (nan) | > current_lr: 5e-06 | > step_time: 0.168 (0.137072279193337) | > loader_time: 0.0048 (0.0047387798673801836)  --> TIME: 2025-02-26 21:08:27 -- STEP: 1756/36938 -- GLOBAL_STEP: 1405400 | > loss_text_ce: 0.024943863973021507 (0.030934494461308408) | > loss_mel_ce: 2.9749176502227783 (nan) | > loss: 0.011904212646186352 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.1369706018910592) | > loader_time: 0.0045 (0.004739185129048331)  --> TIME: 2025-02-26 21:08:36 -- STEP: 1806/36938 -- GLOBAL_STEP: 1405450 | > loss_text_ce: 0.02229924127459526 (0.03092101224412438) | > loss_mel_ce: 2.337620496749878 (nan) | > loss: 0.009364761412143707 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13689012252875216) | > loader_time: 0.0044 (0.004739688744444642)  --> TIME: 2025-02-26 21:08:46 -- STEP: 1856/36938 -- GLOBAL_STEP: 1405500 | > loss_text_ce: 0.01934678480029106 (0.030934101048403356) | > loss_mel_ce: 0.0087890625 (nan) | > loss: 0.00011165019532199949 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13687199726700786) | > loader_time: 0.0045 (0.004743212127480004)  --> TIME: 2025-02-26 21:08:55 -- STEP: 1906/36938 -- GLOBAL_STEP: 1405550 | > loss_text_ce: 0.02892271615564823 (0.030936280144759673) | > loss_mel_ce: 1.666595458984375 (nan) | > loss: 0.006728246808052063 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13690256459515343) | > loader_time: 0.0051 (0.004740696138500033)  --> TIME: 2025-02-26 21:09:04 -- STEP: 1956/36938 -- GLOBAL_STEP: 1405600 | > loss_text_ce: 0.02383141778409481 (0.030900988444983044) | > loss_mel_ce: 2.454453468322754 (nan) | > loss: 0.009834464639425278 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13682698155229564) | > loader_time: 0.0045 (0.004744518753941065)  --> TIME: 2025-02-26 21:09:14 -- STEP: 2006/36938 -- GLOBAL_STEP: 1405650 | > loss_text_ce: 0.031283240765333176 (0.03090990193054481) | > loss_mel_ce: 2.819688558578491 (nan) | > loss: 0.011313380673527718 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13691580842285309) | > loader_time: 0.0048 (0.0047412183681727555)  --> TIME: 2025-02-26 21:09:23 -- STEP: 2056/36938 -- GLOBAL_STEP: 1405700 | > loss_text_ce: 0.028705064207315445 (0.030878465915287225) | > loss_mel_ce: 2.0966074466705322 (nan) | > loss: 0.008433780632913113 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13691866710955994) | > loader_time: 0.0046 (0.004740516151435636)  --> TIME: 2025-02-26 21:09:33 -- STEP: 2106/36938 -- GLOBAL_STEP: 1405750 | > loss_text_ce: 0.02548827975988388 (0.030880577714560094) | > loss_mel_ce: 2.6688625812530518 (nan) | > loss: 0.0106918690726161 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13703970759682507) | > loader_time: 0.0046 (0.0047397665601623395)  --> TIME: 2025-02-26 21:09:42 -- STEP: 2156/36938 -- GLOBAL_STEP: 1405800 | > loss_text_ce: 0.03996286168694496 (0.030892013477987448) | > loss_mel_ce: 2.599865198135376 (nan) | > loss: 0.010475508868694305 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13714218250232194) | > loader_time: 0.0046 (0.004739073300405859)  --> TIME: 2025-02-26 21:09:52 -- STEP: 2206/36938 -- GLOBAL_STEP: 1405850 | > loss_text_ce: 0.02974657528102398 (0.03087700090133809) | > loss_mel_ce: 3.363162040710449 (nan) | > loss: 0.013463923707604408 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13716885671762585) | > loader_time: 0.0045 (0.004738210010615017)  --> TIME: 2025-02-26 21:10:01 -- STEP: 2256/36938 -- GLOBAL_STEP: 1405900 | > loss_text_ce: 0.021816741675138474 (0.03088998959601216) | > loss_mel_ce: 1.6348377466201782 (nan) | > loss: 0.006574026308953762 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.1372223036720398) | > loader_time: 0.0045 (0.004736158441989969)  --> TIME: 2025-02-26 21:10:10 -- STEP: 2306/36938 -- GLOBAL_STEP: 1405950 | > loss_text_ce: 0.03516633063554764 (0.030866797918135562) | > loss_mel_ce: 2.4736735820770264 (nan) | > loss: 0.009955714456737041 (nan) | > current_lr: 5e-06 | > step_time: 0.1264 (0.13711458429086767) | > loader_time: 0.0052 (0.004736951197734202)  --> TIME: 2025-02-26 21:10:19 -- STEP: 2356/36938 -- GLOBAL_STEP: 1406000 | > loss_text_ce: 0.03338916599750519 (0.03082263087442679) | > loss_mel_ce: 2.4695870876312256 (nan) | > loss: 0.009932445362210274 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13698428717452946) | > loader_time: 0.0044 (0.004736955886380972)  --> TIME: 2025-02-26 21:10:28 -- STEP: 2406/36938 -- GLOBAL_STEP: 1406050 | > loss_text_ce: 0.016080418601632118 (0.030834670092659697) | > loss_mel_ce: 2.1049585342407227 (nan) | > loss: 0.008416821248829365 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13694492292126725) | > loader_time: 0.0046 (0.004735524517639607)  --> TIME: 2025-02-26 21:10:38 -- STEP: 2456/36938 -- GLOBAL_STEP: 1406100 | > loss_text_ce: 0.03464905172586441 (0.030839879022522887) | > loss_mel_ce: 2.689816951751709 (nan) | > loss: 0.010811373591423035 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13706685258045267) | > loader_time: 0.0049 (0.00473446508183929)  --> TIME: 2025-02-26 21:10:48 -- STEP: 2506/36938 -- GLOBAL_STEP: 1406150 | > loss_text_ce: 0.02729720249772072 (0.030865802681586407) | > loss_mel_ce: 3.114168167114258 (nan) | > loss: 0.012466133572161198 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13713298049433567) | > loader_time: 0.0046 (0.004733483884587627)  --> TIME: 2025-02-26 21:10:57 -- STEP: 2556/36938 -- GLOBAL_STEP: 1406200 | > loss_text_ce: 0.024969352409243584 (0.030853342703588783) | > loss_mel_ce: 2.4029412269592285 (nan) | > loss: 0.009634566493332386 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13715449530567195) | > loader_time: 0.0046 (0.004731312137627629)  --> TIME: 2025-02-26 21:11:07 -- STEP: 2606/36938 -- GLOBAL_STEP: 1406250 | > loss_text_ce: 0.027043994516134262 (0.030869239055660944) | > loss_mel_ce: 3.0597026348114014 (nan) | > loss: 0.012248995713889599 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13717287408326664) | > loader_time: 0.0045 (0.004729891134424556)  --> TIME: 2025-02-26 21:11:16 -- STEP: 2656/36938 -- GLOBAL_STEP: 1406300 | > loss_text_ce: 0.034453172236680984 (0.030866983419575965) | > loss_mel_ce: 3.199122190475464 (nan) | > loss: 0.01283164881169796 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.13711403597550242) | > loader_time: 0.0047 (0.004728187638592995)  --> TIME: 2025-02-26 21:11:25 -- STEP: 2706/36938 -- GLOBAL_STEP: 1406350 | > loss_text_ce: 0.02794688008725643 (0.030863113051551583) | > loss_mel_ce: 1.8238179683685303 (nan) | > loss: 0.007348273415118456 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.1370909812268911) | > loader_time: 0.0045 (0.004730154916434827)  --> TIME: 2025-02-26 21:11:34 -- STEP: 2756/36938 -- GLOBAL_STEP: 1406400 | > loss_text_ce: 0.027522558346390724 (0.030867156573315794) | > loss_mel_ce: 2.513127565383911 (nan) | > loss: 0.010081945918500423 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13705200121260894) | > loader_time: 0.0045 (0.004727514630997287)  --> TIME: 2025-02-26 21:11:43 -- STEP: 2806/36938 -- GLOBAL_STEP: 1406450 | > loss_text_ce: 0.02882601134479046 (0.03084442056104785) | > loss_mel_ce: 2.1123061180114746 (nan) | > loss: 0.008496556431055069 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.1370062671723916) | > loader_time: 0.0046 (0.004726125696091152)  --> TIME: 2025-02-26 21:11:53 -- STEP: 2856/36938 -- GLOBAL_STEP: 1406500 | > loss_text_ce: 0.028740568086504936 (0.03084512098355829) | > loss_mel_ce: 3.326955556869507 (nan) | > loss: 0.013316255994141102 (nan) | > current_lr: 5e-06 | > step_time: 0.206 (0.1370508345235299) | > loader_time: 0.0051 (0.004727871144185201)  --> TIME: 2025-02-26 21:12:02 -- STEP: 2906/36938 -- GLOBAL_STEP: 1406550 | > loss_text_ce: 0.02895345166325569 (0.030835526946647488) | > loss_mel_ce: 2.70009708404541 (nan) | > loss: 0.010829566977918148 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.1370481964808375) | > loader_time: 0.0048 (0.004726799783096248)  --> TIME: 2025-02-26 21:12:11 -- STEP: 2956/36938 -- GLOBAL_STEP: 1406600 | > loss_text_ce: 0.029534509405493736 (0.03083399454502235) | > loss_mel_ce: 3.1976585388183594 (nan) | > loss: 0.012806322425603867 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13705524499748703) | > loader_time: 0.0047 (0.004724752273869276)  --> TIME: 2025-02-26 21:12:21 -- STEP: 3006/36938 -- GLOBAL_STEP: 1406650 | > loss_text_ce: 0.027239086106419563 (0.030824114733976043) | > loss_mel_ce: 2.3995730876922607 (nan) | > loss: 0.009630207903683186 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13706376984051252) | > loader_time: 0.0046 (0.00472314692463623)  --> TIME: 2025-02-26 21:12:30 -- STEP: 3056/36938 -- GLOBAL_STEP: 1406700 | > loss_text_ce: 0.03441844508051872 (0.030809232214811934) | > loss_mel_ce: 2.634751319885254 (nan) | > loss: 0.010591943748295307 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13697712084385782) | > loader_time: 0.0046 (0.004721499004289105)  --> TIME: 2025-02-26 21:12:39 -- STEP: 3106/36938 -- GLOBAL_STEP: 1406750 | > loss_text_ce: 0.04078487306833267 (0.03082028793378542) | > loss_mel_ce: 2.148585319519043 (nan) | > loss: 0.00868797767907381 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.13700363341255983) | > loader_time: 0.0046 (0.004719893239961451)  --> TIME: 2025-02-26 21:12:49 -- STEP: 3156/36938 -- GLOBAL_STEP: 1406800 | > loss_text_ce: 0.0229649655520916 (0.030816602724851008) | > loss_mel_ce: 1.1469208002090454 (nan) | > loss: 0.0046424041502177715 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13705016247673224) | > loader_time: 0.0046 (0.0047197187052631095)  --> TIME: 2025-02-26 21:12:58 -- STEP: 3206/36938 -- GLOBAL_STEP: 1406850 | > loss_text_ce: 0.03526262566447258 (0.030810413263965903) | > loss_mel_ce: 2.669478178024292 (nan) | > loss: 0.010733098722994328 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13703577293876745) | > loader_time: 0.0045 (0.00472095290795013)  --> TIME: 2025-02-26 21:13:08 -- STEP: 3256/36938 -- GLOBAL_STEP: 1406900 | > loss_text_ce: 0.032427828758955 (0.03081412386796746) | > loss_mel_ce: 2.7771623134613037 (nan) | > loss: 0.011149168014526367 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.13712091227714243) | > loader_time: 0.0047 (0.004722353720840692)  --> TIME: 2025-02-26 21:13:17 -- STEP: 3306/36938 -- GLOBAL_STEP: 1406950 | > loss_text_ce: 0.02938319928944111 (0.030820806808228827) | > loss_mel_ce: 2.4863929748535156 (nan) | > loss: 0.009983239695429802 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.13704099787846566) | > loader_time: 0.0048 (0.004721378530074211)  --> TIME: 2025-02-26 21:13:26 -- STEP: 3356/36938 -- GLOBAL_STEP: 1407000 | > loss_text_ce: 0.03050055168569088 (0.030845643681639424) | > loss_mel_ce: 2.738642454147339 (nan) | > loss: 0.010988663882017136 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13707536650783855) | > loader_time: 0.0048 (0.00472044596655006)  --> TIME: 2025-02-26 21:13:35 -- STEP: 3406/36938 -- GLOBAL_STEP: 1407050 | > loss_text_ce: 0.0297829769551754 (0.030824046817138413) | > loss_mel_ce: 2.493764877319336 (nan) | > loss: 0.010014079511165619 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.1370884609866411) | > loader_time: 0.0046 (0.004720427257764064)  --> TIME: 2025-02-26 21:13:44 -- STEP: 3456/36938 -- GLOBAL_STEP: 1407100 | > loss_text_ce: 0.032158516347408295 (0.03081974024664507) | > loss_mel_ce: 2.1118040084838867 (nan) | > loss: 0.008507789112627506 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13702693071078387) | > loader_time: 0.0045 (0.004719593872626606)  --> TIME: 2025-02-26 21:13:54 -- STEP: 3506/36938 -- GLOBAL_STEP: 1407150 | > loss_text_ce: 0.036868102848529816 (0.030824246903465975) | > loss_mel_ce: 2.8078107833862305 (nan) | > loss: 0.011288409121334553 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.1370001687231027) | > loader_time: 0.0046 (0.004718288787759089)  --> TIME: 2025-02-26 21:14:03 -- STEP: 3556/36938 -- GLOBAL_STEP: 1407200 | > loss_text_ce: 0.03260102868080139 (0.030824499270230264) | > loss_mel_ce: 2.1060574054718018 (nan) | > loss: 0.008486740291118622 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.1370139671793402) | > loader_time: 0.0045 (0.0047177788377210286)  --> TIME: 2025-02-26 21:14:12 -- STEP: 3606/36938 -- GLOBAL_STEP: 1407250 | > loss_text_ce: 0.024136928841471672 (0.030816833511629223) | > loss_mel_ce: 2.5899224281311035 (nan) | > loss: 0.01037325244396925 (nan) | > current_lr: 5e-06 | > step_time: 0.1614 (0.13702413338921945) | > loader_time: 0.0046 (0.0047193179841974985)  --> TIME: 2025-02-26 21:14:22 -- STEP: 3656/36938 -- GLOBAL_STEP: 1407300 | > loss_text_ce: 0.034092724323272705 (0.0308077274396286) | > loss_mel_ce: 2.4447669982910156 (nan) | > loss: 0.009836745448410511 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13698742061005628) | > loader_time: 0.0046 (0.0047192533767457965)  --> TIME: 2025-02-26 21:14:31 -- STEP: 3706/36938 -- GLOBAL_STEP: 1407350 | > loss_text_ce: 0.043069589883089066 (0.030826078910537413) | > loss_mel_ce: 3.140721082687378 (nan) | > loss: 0.012634091079235077 (nan) | > current_lr: 5e-06 | > step_time: 0.1873 (0.13702794943252128) | > loader_time: 0.0048 (0.004719741590204445)  --> TIME: 2025-02-26 21:14:41 -- STEP: 3756/36938 -- GLOBAL_STEP: 1407400 | > loss_text_ce: 0.029452865943312645 (0.03082670166040151) | > loss_mel_ce: 2.7588884830474854 (nan) | > loss: 0.011064846999943256 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.1370870326273739) | > loader_time: 0.0046 (0.004719973308717593)  --> TIME: 2025-02-26 21:14:50 -- STEP: 3806/36938 -- GLOBAL_STEP: 1407450 | > loss_text_ce: 0.02745511196553707 (0.03082684107458574) | > loss_mel_ce: 2.378275156021118 (nan) | > loss: 0.009546549059450626 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13704404082977334) | > loader_time: 0.0047 (0.0047187043439070155)  --> TIME: 2025-02-26 21:14:59 -- STEP: 3856/36938 -- GLOBAL_STEP: 1407500 | > loss_text_ce: 0.02994365617632866 (0.030834465419740272) | > loss_mel_ce: 2.6631226539611816 (nan) | > loss: 0.01068677194416523 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13697242681168922) | > loader_time: 0.0045 (0.004718902818394879)  --> TIME: 2025-02-26 21:15:08 -- STEP: 3906/36938 -- GLOBAL_STEP: 1407550 | > loss_text_ce: 0.03124944679439068 (0.030818573829393188) | > loss_mel_ce: 2.89487624168396 (nan) | > loss: 0.011611610651016235 (nan) | > current_lr: 5e-06 | > step_time: 0.1879 (0.1369315851546531) | > loader_time: 0.0048 (0.004717380586673939)  --> TIME: 2025-02-26 21:15:17 -- STEP: 3956/36938 -- GLOBAL_STEP: 1407600 | > loss_text_ce: 0.029899965971708298 (0.03082183552684622) | > loss_mel_ce: 0.9269287586212158 (nan) | > loss: 0.003796939505264163 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13693515843400103) | > loader_time: 0.0047 (0.004716048467267739)  --> TIME: 2025-02-26 21:15:27 -- STEP: 4006/36938 -- GLOBAL_STEP: 1407650 | > loss_text_ce: 0.03616635873913765 (0.030816432719336963) | > loss_mel_ce: 2.483919858932495 (nan) | > loss: 0.010000343434512615 (nan) | > current_lr: 5e-06 | > step_time: 0.1864 (0.13696262340098103) | > loader_time: 0.0051 (0.004715790643849123)  --> TIME: 2025-02-26 21:15:36 -- STEP: 4056/36938 -- GLOBAL_STEP: 1407700 | > loss_text_ce: 0.027899980545043945 (0.03081335342251301) | > loss_mel_ce: 2.931838035583496 (nan) | > loss: 0.011744992807507515 (nan) | > current_lr: 5e-06 | > step_time: 0.1689 (0.13701356297884237) | > loader_time: 0.0054 (0.004718188468286023)  --> TIME: 2025-02-26 21:15:46 -- STEP: 4106/36938 -- GLOBAL_STEP: 1407750 | > loss_text_ce: 0.030229460448026657 (0.03078442553776832) | > loss_mel_ce: 2.799945592880249 (nan) | > loss: 0.01123085431754589 (nan) | > current_lr: 5e-06 | > step_time: 0.1821 (0.1370343749138767) | > loader_time: 0.0048 (0.004720297721439131)  --> TIME: 2025-02-26 21:15:56 -- STEP: 4156/36938 -- GLOBAL_STEP: 1407800 | > loss_text_ce: 0.03133196383714676 (0.030785773867558368) | > loss_mel_ce: 2.8279356956481934 (nan) | > loss: 0.011346301063895226 (nan) | > current_lr: 5e-06 | > step_time: 0.1689 (0.13710232619716073) | > loader_time: 0.0048 (0.004720257898152606)  --> TIME: 2025-02-26 21:16:05 -- STEP: 4206/36938 -- GLOBAL_STEP: 1407850 | > loss_text_ce: 0.023211313411593437 (0.030781634256227763) | > loss_mel_ce: 2.1079092025756836 (nan) | > loss: 0.008456828072667122 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13706336892110307) | > loader_time: 0.0045 (0.004719101810137873)  --> TIME: 2025-02-26 21:16:14 -- STEP: 4256/36938 -- GLOBAL_STEP: 1407900 | > loss_text_ce: 0.030804194509983063 (0.030775159327146997) | > loss_mel_ce: 3.2973999977111816 (nan) | > loss: 0.01320715993642807 (nan) | > current_lr: 5e-06 | > step_time: 0.1522 (0.13709601933570736) | > loader_time: 0.0071 (0.0047189979849005)  --> TIME: 2025-02-26 21:16:24 -- STEP: 4306/36938 -- GLOBAL_STEP: 1407950 | > loss_text_ce: 0.036661192774772644 (0.030762267577355052) | > loss_mel_ce: 1.8508552312850952 (nan) | > loss: 0.007490144576877356 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13710013567653864) | > loader_time: 0.0046 (0.004718295319712656)  --> TIME: 2025-02-26 21:16:33 -- STEP: 4356/36938 -- GLOBAL_STEP: 1408000 | > loss_text_ce: 0.01929493062198162 (0.030761783114096528) | > loss_mel_ce: 0.8003926873207092 (nan) | > loss: 0.0032527288421988487 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13712009713669462) | > loader_time: 0.0045 (0.004717220014339188)  --> TIME: 2025-02-26 21:16:43 -- STEP: 4406/36938 -- GLOBAL_STEP: 1408050 | > loss_text_ce: 0.029863392934203148 (0.03074961095093781) | > loss_mel_ce: 2.4888198375701904 (nan) | > loss: 0.00999477505683899 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.1372219715343515) | > loader_time: 0.0046 (0.004717422516520206)  --> TIME: 2025-02-26 21:16:53 -- STEP: 4456/36938 -- GLOBAL_STEP: 1408100 | > loss_text_ce: 0.03129086643457413 (0.03075500651780195) | > loss_mel_ce: 2.584369421005249 (nan) | > loss: 0.010379604995250702 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.137263540412622) | > loader_time: 0.0051 (0.004718118659897797)  --> TIME: 2025-02-26 21:17:02 -- STEP: 4506/36938 -- GLOBAL_STEP: 1408150 | > loss_text_ce: 0.031807634979486465 (0.030749737629208084) | > loss_mel_ce: 3.6471304893493652 (nan) | > loss: 0.014598961919546127 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13727831189705436) | > loader_time: 0.0045 (0.004717672395007735)  --> TIME: 2025-02-26 21:17:12 -- STEP: 4556/36938 -- GLOBAL_STEP: 1408200 | > loss_text_ce: 0.03155026212334633 (0.030757759016209966) | > loss_mel_ce: 2.4276366233825684 (nan) | > loss: 0.009758678264915943 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13731743783883826) | > loader_time: 0.0046 (0.004718141873956658)  --> TIME: 2025-02-26 21:17:21 -- STEP: 4606/36938 -- GLOBAL_STEP: 1408250 | > loss_text_ce: 0.03274589404463768 (0.030768749379333313) | > loss_mel_ce: 0.13360023498535156 (nan) | > loss: 0.0006601037457585335 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13725056199990202) | > loader_time: 0.0046 (0.0047186069575486465)  --> TIME: 2025-02-26 21:17:30 -- STEP: 4656/36938 -- GLOBAL_STEP: 1408300 | > loss_text_ce: 0.03180135041475296 (0.030760317691601813) | > loss_mel_ce: 2.538944959640503 (nan) | > loss: 0.010201375000178814 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13719625171926833) | > loader_time: 0.0046 (0.0047177019500240745)  --> TIME: 2025-02-26 21:17:39 -- STEP: 4706/36938 -- GLOBAL_STEP: 1408350 | > loss_text_ce: 0.034456606954336166 (0.030756264234328215) | > loss_mel_ce: 2.7003567218780518 (nan) | > loss: 0.010852433741092682 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13717344812368057) | > loader_time: 0.0048 (0.004716421409814441)  --> TIME: 2025-02-26 21:17:49 -- STEP: 4756/36938 -- GLOBAL_STEP: 1408400 | > loss_text_ce: 0.028483044356107712 (0.030753815964904525) | > loss_mel_ce: 2.71627140045166 (nan) | > loss: 0.010891883634030819 (nan) | > current_lr: 5e-06 | > step_time: 0.1894 (0.13725219106754402) | > loader_time: 0.0048 (0.004715920697750407)  --> TIME: 2025-02-26 21:17:58 -- STEP: 4806/36938 -- GLOBAL_STEP: 1408450 | > loss_text_ce: 0.03266213461756706 (0.03075400730645376) | > loss_mel_ce: 1.3750814199447632 (nan) | > loss: 0.005586284212768078 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13727494958932315) | > loader_time: 0.0046 (0.0047148377310966706)  --> TIME: 2025-02-26 21:18:08 -- STEP: 4856/36938 -- GLOBAL_STEP: 1408500 | > loss_text_ce: 0.02878447063267231 (0.030762394285673365) | > loss_mel_ce: 2.442770004272461 (nan) | > loss: 0.009807756170630455 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13727955753367252) | > loader_time: 0.0047 (0.0047140304103518126)  --> TIME: 2025-02-26 21:18:17 -- STEP: 4906/36938 -- GLOBAL_STEP: 1408550 | > loss_text_ce: 0.03692310303449631 (0.030757206396639796) | > loss_mel_ce: 2.8802878856658936 (nan) | > loss: 0.011576235294342041 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.13728853726066273) | > loader_time: 0.0046 (0.004713644312684788)  --> TIME: 2025-02-26 21:18:26 -- STEP: 4956/36938 -- GLOBAL_STEP: 1408600 | > loss_text_ce: 0.029395785182714462 (0.030748749702614136) | > loss_mel_ce: 2.842933416366577 (nan) | > loss: 0.011398132890462875 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13726200227106244) | > loader_time: 0.0047 (0.0047134008496109644)  --> TIME: 2025-02-26 21:18:35 -- STEP: 5006/36938 -- GLOBAL_STEP: 1408650 | > loss_text_ce: 0.031008867546916008 (0.030747991110876505) | > loss_mel_ce: 3.36100697517395 (nan) | > loss: 0.013460381887853146 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13726135558525726) | > loader_time: 0.0046 (0.0047146011437123716)  --> TIME: 2025-02-26 21:18:45 -- STEP: 5056/36938 -- GLOBAL_STEP: 1408700 | > loss_text_ce: 0.029126087203621864 (0.030737549281875085) | > loss_mel_ce: 2.4061362743377686 (nan) | > loss: 0.009663740172982216 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13726442212922652) | > loader_time: 0.0045 (0.004714188224907151)  --> TIME: 2025-02-26 21:18:54 -- STEP: 5106/36938 -- GLOBAL_STEP: 1408750 | > loss_text_ce: 0.03736475110054016 (0.030746409689764612) | > loss_mel_ce: 1.264309287071228 (nan) | > loss: 0.0051653734408319 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13728329507780665) | > loader_time: 0.0047 (0.004715679068216087)  --> TIME: 2025-02-26 21:19:04 -- STEP: 5156/36938 -- GLOBAL_STEP: 1408800 | > loss_text_ce: 0.03342965990304947 (0.030743973725059926) | > loss_mel_ce: 2.9051990509033203 (nan) | > loss: 0.0116612259298563 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.13728410944261663) | > loader_time: 0.0049 (0.004716111949657251)  --> TIME: 2025-02-26 21:19:13 -- STEP: 5206/36938 -- GLOBAL_STEP: 1408850 | > loss_text_ce: 0.025637248530983925 (0.030746699696355916) | > loss_mel_ce: 3.2123003005981445 (nan) | > loss: 0.012848959304392338 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13729046639505788) | > loader_time: 0.0051 (0.0047151919653266195)  --> TIME: 2025-02-26 21:19:23 -- STEP: 5256/36938 -- GLOBAL_STEP: 1408900 | > loss_text_ce: 0.026486551389098167 (0.030740268301478156) | > loss_mel_ce: 2.605217695236206 (nan) | > loss: 0.010443272069096565 (nan) | > current_lr: 5e-06 | > step_time: 0.1355 (0.1373402818334337) | > loader_time: 0.0046 (0.004715540728431113)  --> TIME: 2025-02-26 21:19:32 -- STEP: 5306/36938 -- GLOBAL_STEP: 1408950 | > loss_text_ce: 0.03389879688620567 (0.030734543587669836) | > loss_mel_ce: 1.8332061767578125 (nan) | > loss: 0.007409147452563047 (nan) | > current_lr: 5e-06 | > step_time: 0.1123 (0.1373605716915971) | > loader_time: 0.0045 (0.0047148063054320045)  --> TIME: 2025-02-26 21:19:42 -- STEP: 5356/36938 -- GLOBAL_STEP: 1409000 | > loss_text_ce: 0.03494322672486305 (0.03072759867548425) | > loss_mel_ce: 2.502744197845459 (nan) | > loss: 0.010070188902318478 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13739114478953826) | > loader_time: 0.0045 (0.0047149553594524945)  --> TIME: 2025-02-26 21:19:51 -- STEP: 5406/36938 -- GLOBAL_STEP: 1409050 | > loss_text_ce: 0.026887113228440285 (0.03072777004151726) | > loss_mel_ce: 2.3525443077087402 (nan) | > loss: 0.009442188777029514 (nan) | > current_lr: 5e-06 | > step_time: 0.1791 (0.13740737849237997) | > loader_time: 0.0048 (0.004714657278621878)  --> TIME: 2025-02-26 21:20:01 -- STEP: 5456/36938 -- GLOBAL_STEP: 1409100 | > loss_text_ce: 0.04003247991204262 (0.03072842948949183) | > loss_mel_ce: 3.2140700817108154 (nan) | > loss: 0.012913106009364128 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.13741677995126608) | > loader_time: 0.0049 (0.00471535325050352)  --> TIME: 2025-02-26 21:20:10 -- STEP: 5506/36938 -- GLOBAL_STEP: 1409150 | > loss_text_ce: 0.03158341348171234 (0.030718512055550352) | > loss_mel_ce: 3.113996744155884 (nan) | > loss: 0.012482461519539356 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13737965704006486) | > loader_time: 0.0046 (0.004714480799152923)  --> TIME: 2025-02-26 21:20:19 -- STEP: 5556/36938 -- GLOBAL_STEP: 1409200 | > loss_text_ce: 0.03359544649720192 (0.030722521221509258) | > loss_mel_ce: 2.994398593902588 (nan) | > loss: 0.01201585028320551 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.1373856898861191) | > loader_time: 0.0046 (0.004714195465681083)  --> TIME: 2025-02-26 21:20:28 -- STEP: 5606/36938 -- GLOBAL_STEP: 1409250 | > loss_text_ce: 0.02408861368894577 (0.030727460301111384) | > loss_mel_ce: 0.081939697265625 (nan) | > loss: 0.00042074729572050273 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.1373780757837366) | > loader_time: 0.0048 (0.004713976378956288)  --> TIME: 2025-02-26 21:20:38 -- STEP: 5656/36938 -- GLOBAL_STEP: 1409300 | > loss_text_ce: 0.02851361222565174 (0.0307334804756941) | > loss_mel_ce: 2.4032647609710693 (nan) | > loss: 0.009649914689362049 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13740675220900908) | > loader_time: 0.0052 (0.004714118751474596)  --> TIME: 2025-02-26 21:20:47 -- STEP: 5706/36938 -- GLOBAL_STEP: 1409350 | > loss_text_ce: 0.032896608114242554 (0.03072739061692196) | > loss_mel_ce: 2.589799165725708 (nan) | > loss: 0.01040752325206995 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.1374205845596495) | > loader_time: 0.0065 (0.004715302556262177)  --> TIME: 2025-02-26 21:20:57 -- STEP: 5756/36938 -- GLOBAL_STEP: 1409400 | > loss_text_ce: 0.02817087434232235 (0.03073038236345498) | > loss_mel_ce: 2.432859420776367 (nan) | > loss: 0.009765993803739548 (nan) | > current_lr: 5e-06 | > step_time: 0.1434 (0.13737793893264025) | > loader_time: 0.0046 (0.00471506481621311)  --> TIME: 2025-02-26 21:21:06 -- STEP: 5806/36938 -- GLOBAL_STEP: 1409450 | > loss_text_ce: 0.031165244057774544 (0.030726186916536242) | > loss_mel_ce: 2.5878074169158936 (nan) | > loss: 0.010392749682068825 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13739789074304148) | > loader_time: 0.0046 (0.004715109627040685)  --> TIME: 2025-02-26 21:21:15 -- STEP: 5856/36938 -- GLOBAL_STEP: 1409500 | > loss_text_ce: 0.027813037857413292 (0.03072560832917161) | > loss_mel_ce: 3.7740886211395264 (nan) | > loss: 0.015086911618709564 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13736697253796593) | > loader_time: 0.0045 (0.004714019800144438)  --> TIME: 2025-02-26 21:21:24 -- STEP: 5906/36938 -- GLOBAL_STEP: 1409550 | > loss_text_ce: 0.027771946042776108 (0.03073115127279098) | > loss_mel_ce: 2.6524786949157715 (nan) | > loss: 0.010635916143655777 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.1373584403290897) | > loader_time: 0.0046 (0.004713548913148928)  --> TIME: 2025-02-26 21:21:34 -- STEP: 5956/36938 -- GLOBAL_STEP: 1409600 | > loss_text_ce: 0.03069036640226841 (0.03072643703752922) | > loss_mel_ce: 1.798559308052063 (nan) | > loss: 0.007258927449584007 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13734487461675496) | > loader_time: 0.0044 (0.004713245451730528)  --> TIME: 2025-02-26 21:21:43 -- STEP: 6006/36938 -- GLOBAL_STEP: 1409650 | > loss_text_ce: 0.030318137258291245 (0.030722684142264454) | > loss_mel_ce: 2.491694211959839 (nan) | > loss: 0.01000798586755991 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.137339496271157) | > loader_time: 0.0045 (0.004712225912095991)  --> TIME: 2025-02-26 21:21:52 -- STEP: 6056/36938 -- GLOBAL_STEP: 1409700 | > loss_text_ce: 0.017783284187316895 (0.030718315399499166) | > loss_mel_ce: 2.6414473056793213 (nan) | > loss: 0.010552503168582916 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13734036943871575) | > loader_time: 0.0046 (0.004711434934066783)  --> TIME: 2025-02-26 21:22:02 -- STEP: 6106/36938 -- GLOBAL_STEP: 1409750 | > loss_text_ce: 0.022787529975175858 (0.0307140905608945) | > loss_mel_ce: 2.9375412464141846 (nan) | > loss: 0.011747336946427822 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.1373365421666891) | > loader_time: 0.0045 (0.00471233208765951)  --> TIME: 2025-02-26 21:22:11 -- STEP: 6156/36938 -- GLOBAL_STEP: 1409800 | > loss_text_ce: 0.0380534790456295 (0.030712223555795882) | > loss_mel_ce: 2.323667049407959 (nan) | > loss: 0.009371907450258732 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13730106100469075) | > loader_time: 0.0053 (0.0047123546024047105)  --> TIME: 2025-02-26 21:22:20 -- STEP: 6206/36938 -- GLOBAL_STEP: 1409850 | > loss_text_ce: 0.03216896578669548 (0.030707039728957608) | > loss_mel_ce: 2.271841526031494 (nan) | > loss: 0.009142898954451084 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13728990183697184) | > loader_time: 0.0045 (0.0047131552836067216)  --> TIME: 2025-02-26 21:22:29 -- STEP: 6256/36938 -- GLOBAL_STEP: 1409900 | > loss_text_ce: 0.02972966618835926 (0.030709049581666776) | > loss_mel_ce: 2.9100754261016846 (nan) | > loss: 0.011665893718600273 (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.13729018960005185) | > loader_time: 0.0055 (0.004713179243495065)  --> TIME: 2025-02-26 21:22:39 -- STEP: 6306/36938 -- GLOBAL_STEP: 1409950 | > loss_text_ce: 0.022037601098418236 (0.03070501398999342) | > loss_mel_ce: 3.385929822921753 (nan) | > loss: 0.013523681089282036 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1372990283064716) | > loader_time: 0.0045 (0.004713641625527435)  --> TIME: 2025-02-26 21:22:48 -- STEP: 6356/36938 -- GLOBAL_STEP: 1410000 | > loss_text_ce: 0.027168462052941322 (0.030698575378941283) | > loss_mel_ce: 2.1696841716766357 (nan) | > loss: 0.008717670105397701 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13724008118153508) | > loader_time: 0.0045 (0.004714012558614757) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1410000.pth  --> TIME: 2025-02-26 21:23:03 -- STEP: 6406/36938 -- GLOBAL_STEP: 1410050 | > loss_text_ce: 0.03063560649752617 (0.03069386988804977) | > loss_mel_ce: 2.82796049118042 (nan) | > loss: 0.011343635618686676 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13722455408511344) | > loader_time: 0.0045 (0.004714843819672092)  --> TIME: 2025-02-26 21:23:12 -- STEP: 6456/36938 -- GLOBAL_STEP: 1410100 | > loss_text_ce: 0.04032324254512787 (0.030689951796961726) | > loss_mel_ce: 0.003063201904296875 (nan) | > loss: 0.00017216843843925744 (nan) | > current_lr: 5e-06 | > step_time: 0.1121 (0.13722140173043434) | > loader_time: 0.0053 (0.004714654908599736)  --> TIME: 2025-02-26 21:23:21 -- STEP: 6506/36938 -- GLOBAL_STEP: 1410150 | > loss_text_ce: 0.033544205129146576 (0.030693754160137147) | > loss_mel_ce: 2.8793747425079346 (nan) | > loss: 0.011559203267097473 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13718159862785648) | > loader_time: 0.0047 (0.004713641582105531)  --> TIME: 2025-02-26 21:23:31 -- STEP: 6556/36938 -- GLOBAL_STEP: 1410200 | > loss_text_ce: 0.03464749827980995 (0.03069944033976542) | > loss_mel_ce: 2.7103793621063232 (nan) | > loss: 0.010892963968217373 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13718380758571755) | > loader_time: 0.0045 (0.0047126707323736335)  --> TIME: 2025-02-26 21:23:39 -- STEP: 6606/36938 -- GLOBAL_STEP: 1410250 | > loss_text_ce: 0.03839070349931717 (0.03069697921436428) | > loss_mel_ce: 3.1379458904266357 (nan) | > loss: 0.01260451041162014 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.1371006004889008) | > loader_time: 0.0045 (0.004711409066974325)  --> TIME: 2025-02-26 21:23:49 -- STEP: 6656/36938 -- GLOBAL_STEP: 1410300 | > loss_text_ce: 0.031650837510824203 (0.030699506507097303) | > loss_mel_ce: 2.296990156173706 (nan) | > loss: 0.00924063939601183 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.1371035327275208) | > loader_time: 0.0045 (0.00471133695772059)  --> TIME: 2025-02-26 21:23:58 -- STEP: 6706/36938 -- GLOBAL_STEP: 1410350 | > loss_text_ce: 0.029025403782725334 (0.030703978718983668) | > loss_mel_ce: 3.030609130859375 (nan) | > loss: 0.01214140746742487 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13711589242393013) | > loader_time: 0.0045 (0.0047114510839105376)  --> TIME: 2025-02-26 21:24:07 -- STEP: 6756/36938 -- GLOBAL_STEP: 1410400 | > loss_text_ce: 0.03196457028388977 (0.030698799908512958) | > loss_mel_ce: 1.7435752153396606 (nan) | > loss: 0.007045792881399393 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.13709391645597452) | > loader_time: 0.0048 (0.004712178553258805)  --> TIME: 2025-02-26 21:24:17 -- STEP: 6806/36938 -- GLOBAL_STEP: 1410450 | > loss_text_ce: 0.03843370079994202 (0.03070440462057249) | > loss_mel_ce: 2.478888988494873 (nan) | > loss: 0.009989377111196518 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.13712066515312385) | > loader_time: 0.0055 (0.004712178992272237)  --> TIME: 2025-02-26 21:24:26 -- STEP: 6856/36938 -- GLOBAL_STEP: 1410500 | > loss_text_ce: 0.029757559299468994 (0.03070197627586807) | > loss_mel_ce: 3.2496180534362793 (nan) | > loss: 0.013013395480811596 (nan) | > current_lr: 5e-06 | > step_time: 0.1833 (0.13710092653193215) | > loader_time: 0.0056 (0.004711919445696596)  --> TIME: 2025-02-26 21:24:36 -- STEP: 6906/36938 -- GLOBAL_STEP: 1410550 | > loss_text_ce: 0.03037453070282936 (0.030696555856257125) | > loss_mel_ce: 0.2356719970703125 (nan) | > loss: 0.0010557402856647968 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13713493086171) | > loader_time: 0.0047 (0.004712873806029558)  --> TIME: 2025-02-26 21:24:45 -- STEP: 6956/36938 -- GLOBAL_STEP: 1410600 | > loss_text_ce: 0.03288544714450836 (0.03069769044630572) | > loss_mel_ce: 2.194789409637451 (nan) | > loss: 0.008839980699121952 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13710651259233014) | > loader_time: 0.0047 (0.004713038969067311)  --> TIME: 2025-02-26 21:24:54 -- STEP: 7006/36938 -- GLOBAL_STEP: 1410650 | > loss_text_ce: 0.030724503099918365 (0.030689722010262297) | > loss_mel_ce: 2.18035626411438 (nan) | > loss: 0.00877413060516119 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.1370801514909493) | > loader_time: 0.0045 (0.004713291785664577)  --> TIME: 2025-02-26 21:25:03 -- STEP: 7056/36938 -- GLOBAL_STEP: 1410700 | > loss_text_ce: 0.03456706553697586 (0.030694541984655876) | > loss_mel_ce: 3.178859233856201 (nan) | > loss: 0.012751692906022072 (nan) | > current_lr: 5e-06 | > step_time: 0.1724 (0.13706461950089036) | > loader_time: 0.0048 (0.004714179471506798)  --> TIME: 2025-02-26 21:25:12 -- STEP: 7106/36938 -- GLOBAL_STEP: 1410750 | > loss_text_ce: 0.02573796734213829 (0.03070007726317337) | > loss_mel_ce: 2.692606210708618 (nan) | > loss: 0.010787080973386765 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13706601763187562) | > loader_time: 0.0046 (0.004715215378065268)  --> TIME: 2025-02-26 21:25:22 -- STEP: 7156/36938 -- GLOBAL_STEP: 1410800 | > loss_text_ce: 0.035150911659002304 (0.030708609058278574) | > loss_mel_ce: 2.8689470291137695 (nan) | > loss: 0.01152419950813055 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13709468742267106) | > loader_time: 0.0046 (0.004715931022413728)  --> TIME: 2025-02-26 21:25:32 -- STEP: 7206/36938 -- GLOBAL_STEP: 1410850 | > loss_text_ce: 0.032138943672180176 (0.030700866550686885) | > loss_mel_ce: 2.6786046028137207 (nan) | > loss: 0.010756919160485268 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.1371329404763163) | > loader_time: 0.0046 (0.004715531824563472)  --> TIME: 2025-02-26 21:25:41 -- STEP: 7256/36938 -- GLOBAL_STEP: 1410900 | > loss_text_ce: 0.03421520069241524 (0.03069775246490356) | > loss_mel_ce: 3.118443727493286 (nan) | > loss: 0.012510552071034908 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13713955560487487) | > loader_time: 0.0052 (0.004715630507390292)  --> TIME: 2025-02-26 21:25:50 -- STEP: 7306/36938 -- GLOBAL_STEP: 1410950 | > loss_text_ce: 0.03303312882781029 (0.030693705073873163) | > loss_mel_ce: 3.0117013454437256 (nan) | > loss: 0.012082280591130257 (nan) | > current_lr: 5e-06 | > step_time: 0.1828 (0.13713046524121894) | > loader_time: 0.0048 (0.004715479728851168)  --> TIME: 2025-02-26 21:26:00 -- STEP: 7356/36938 -- GLOBAL_STEP: 1411000 | > loss_text_ce: 0.03013121522963047 (0.030695285352353104) | > loss_mel_ce: 2.6068572998046875 (nan) | > loss: 0.010464239865541458 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.1371518292136646) | > loader_time: 0.0045 (0.0047158337015376795)  --> TIME: 2025-02-26 21:26:09 -- STEP: 7406/36938 -- GLOBAL_STEP: 1411050 | > loss_text_ce: 0.03659218177199364 (0.030693206284220335) | > loss_mel_ce: 2.8650546073913574 (nan) | > loss: 0.011514471843838692 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13714049479267865) | > loader_time: 0.0048 (0.0047155136100156874)  --> TIME: 2025-02-26 21:26:19 -- STEP: 7456/36938 -- GLOBAL_STEP: 1411100 | > loss_text_ce: 0.03326783701777458 (0.030692472179818003) | > loss_mel_ce: 3.3688225746154785 (nan) | > loss: 0.013500358909368515 (nan) | > current_lr: 5e-06 | > step_time: 0.1788 (0.13715117213679898) | > loader_time: 0.0048 (0.004715552817342594)  --> TIME: 2025-02-26 21:26:28 -- STEP: 7506/36938 -- GLOBAL_STEP: 1411150 | > loss_text_ce: 0.031141826882958412 (0.03069277286188081) | > loss_mel_ce: 2.7032840251922607 (nan) | > loss: 0.010850896127521992 (nan) | > current_lr: 5e-06 | > step_time: 0.1565 (0.13711848187186035) | > loader_time: 0.0046 (0.0047150974174578544)  --> TIME: 2025-02-26 21:26:37 -- STEP: 7556/36938 -- GLOBAL_STEP: 1411200 | > loss_text_ce: 0.023741263896226883 (0.03069447613330612) | > loss_mel_ce: 2.933774948120117 (nan) | > loss: 0.01173617597669363 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13711736109472475) | > loader_time: 0.0054 (0.004715733107843976)  --> TIME: 2025-02-26 21:26:46 -- STEP: 7606/36938 -- GLOBAL_STEP: 1411250 | > loss_text_ce: 0.02097698487341404 (0.030697954170133546) | > loss_mel_ce: 1.5871541500091553 (nan) | > loss: 0.006381473038345575 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13715054037192823) | > loader_time: 0.0047 (0.0047161040919221145)  --> TIME: 2025-02-26 21:26:56 -- STEP: 7656/36938 -- GLOBAL_STEP: 1411300 | > loss_text_ce: 0.029759598895907402 (0.030695340124882422) | > loss_mel_ce: 3.166656732559204 (nan) | > loss: 0.012684192508459091 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.137154753025048) | > loader_time: 0.0046 (0.004715713961371036)  --> TIME: 2025-02-26 21:27:05 -- STEP: 7706/36938 -- GLOBAL_STEP: 1411350 | > loss_text_ce: 0.030108582228422165 (0.030689899994086018) | > loss_mel_ce: 1.5433708429336548 (nan) | > loss: 0.006243966519832611 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.1371780370632571) | > loader_time: 0.0046 (0.004715073179833561)  --> TIME: 2025-02-26 21:27:15 -- STEP: 7756/36938 -- GLOBAL_STEP: 1411400 | > loss_text_ce: 0.030289344489574432 (0.030688036527857378) | > loss_mel_ce: 2.4977385997772217 (nan) | > loss: 0.010031857527792454 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13719022894962998) | > loader_time: 0.0046 (0.004714429532208943)  --> TIME: 2025-02-26 21:27:24 -- STEP: 7806/36938 -- GLOBAL_STEP: 1411450 | > loss_text_ce: 0.03205571696162224 (0.030684705109001658) | > loss_mel_ce: 2.384685516357422 (nan) | > loss: 0.009590242989361286 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.1372222911386092) | > loader_time: 0.0046 (0.00471383850909853)  --> TIME: 2025-02-26 21:27:34 -- STEP: 7856/36938 -- GLOBAL_STEP: 1411500 | > loss_text_ce: 0.03315099701285362 (0.03068397149880836) | > loss_mel_ce: 3.0570907592773438 (nan) | > loss: 0.012262864038348198 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13722925267491373) | > loader_time: 0.0047 (0.004713198348363865)  --> TIME: 2025-02-26 21:27:43 -- STEP: 7906/36938 -- GLOBAL_STEP: 1411550 | > loss_text_ce: 0.031577277928590775 (0.030680709470783437) | > loss_mel_ce: 2.7167229652404785 (nan) | > loss: 0.01090595405548811 (nan) | > current_lr: 5e-06 | > step_time: 0.1352 (0.13720717190067616) | > loader_time: 0.0046 (0.004713058954186275)  --> TIME: 2025-02-26 21:27:52 -- STEP: 7956/36938 -- GLOBAL_STEP: 1411600 | > loss_text_ce: 0.0344148650765419 (0.03068723634216427) | > loss_mel_ce: 3.1185128688812256 (nan) | > loss: 0.012511618435382843 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13720878567630798) | > loader_time: 0.0046 (0.004712939442192964)  --> TIME: 2025-02-26 21:28:02 -- STEP: 8006/36938 -- GLOBAL_STEP: 1411650 | > loss_text_ce: 0.029367968440055847 (0.030687406500687443) | > loss_mel_ce: 1.8013149499893188 (nan) | > loss: 0.007264615036547184 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13722007619479848) | > loader_time: 0.0048 (0.004712708586607735)  --> TIME: 2025-02-26 21:28:11 -- STEP: 8056/36938 -- GLOBAL_STEP: 1411700 | > loss_text_ce: 0.02814790979027748 (0.030690173850632472) | > loss_mel_ce: 2.7074222564697266 (nan) | > loss: 0.010855438187718391 (nan) | > current_lr: 5e-06 | > step_time: 0.1905 (0.1372134611232506) | > loader_time: 0.0049 (0.004712115658785618)  --> TIME: 2025-02-26 21:28:21 -- STEP: 8106/36938 -- GLOBAL_STEP: 1411750 | > loss_text_ce: 0.031899455934762955 (0.03069318640796073) | > loss_mel_ce: 2.610490560531616 (nan) | > loss: 0.010485675185918808 (nan) | > current_lr: 5e-06 | > step_time: 0.1819 (0.13724798927358806) | > loader_time: 0.0049 (0.004711820024518099)  --> TIME: 2025-02-26 21:28:30 -- STEP: 8156/36938 -- GLOBAL_STEP: 1411800 | > loss_text_ce: 0.03973982110619545 (0.03069639125829284) | > loss_mel_ce: 0.028900146484375 (nan) | > loss: 0.0002723808283917606 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13727073873474907) | > loader_time: 0.0045 (0.004711583965604571)  --> TIME: 2025-02-26 21:28:40 -- STEP: 8206/36938 -- GLOBAL_STEP: 1411850 | > loss_text_ce: 0.030931968241930008 (0.030693205870649828) | > loss_mel_ce: 2.847623586654663 (nan) | > loss: 0.01142283994704485 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.13728103653965562) | > loader_time: 0.0046 (0.004710939405256709)  --> TIME: 2025-02-26 21:28:49 -- STEP: 8256/36938 -- GLOBAL_STEP: 1411900 | > loss_text_ce: 0.026891939342021942 (0.030695114629487544) | > loss_mel_ce: 0.04012298583984375 (nan) | > loss: 0.00026593226357363164 (nan) | > current_lr: 5e-06 | > step_time: 0.1519 (0.13723147351496814) | > loader_time: 0.0046 (0.004710121758917483)  --> TIME: 2025-02-26 21:28:58 -- STEP: 8306/36938 -- GLOBAL_STEP: 1411950 | > loss_text_ce: 0.03459525480866432 (0.030697750776661703) | > loss_mel_ce: 1.788965106010437 (nan) | > loss: 0.0072363512590527534 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13721838957310897) | > loader_time: 0.0047 (0.004710225894770887)  --> TIME: 2025-02-26 21:29:08 -- STEP: 8356/36938 -- GLOBAL_STEP: 1412000 | > loss_text_ce: 0.026395808905363083 (0.03070520038407541) | > loss_mel_ce: 0.7652130126953125 (nan) | > loss: 0.0031413049437105656 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13724880220788385) | > loader_time: 0.0046 (0.004710186206877867)  --> TIME: 2025-02-26 21:29:17 -- STEP: 8406/36938 -- GLOBAL_STEP: 1412050 | > loss_text_ce: 0.03735072538256645 (0.030703973206118652) | > loss_mel_ce: 3.0898454189300537 (nan) | > loss: 0.012409509159624577 (nan) | > current_lr: 5e-06 | > step_time: 0.1695 (0.13726615040693033) | > loader_time: 0.0048 (0.004709820902804187)  --> TIME: 2025-02-26 21:29:26 -- STEP: 8456/36938 -- GLOBAL_STEP: 1412100 | > loss_text_ce: 0.03490192070603371 (0.03069712732385055) | > loss_mel_ce: 2.652905225753784 (nan) | > loss: 0.010665901936590672 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13725276893337746) | > loader_time: 0.0046 (0.004709867339373291)  --> TIME: 2025-02-26 21:29:35 -- STEP: 8506/36938 -- GLOBAL_STEP: 1412150 | > loss_text_ce: 0.01931731216609478 (0.03069370492402436) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1372381369142055) | > loader_time: 0.0044 (0.004709181941427267)  --> TIME: 2025-02-26 21:29:45 -- STEP: 8556/36938 -- GLOBAL_STEP: 1412200 | > loss_text_ce: 0.0403667613863945 (0.030697436081184592) | > loss_mel_ce: 0.0833333358168602 (nan) | > loss: 0.0004908734117634594 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13728255066931747) | > loader_time: 0.0045 (0.00470905178002297)  --> TIME: 2025-02-26 21:29:54 -- STEP: 8606/36938 -- GLOBAL_STEP: 1412250 | > loss_text_ce: 0.031190229579806328 (0.030693658592182455) | > loss_mel_ce: 2.6743414402008057 (nan) | > loss: 0.010736237280070782 (nan) | > current_lr: 5e-06 | > step_time: 0.15 (0.13727710520198805) | > loader_time: 0.0046 (0.004708718095522749)  --> TIME: 2025-02-26 21:30:03 -- STEP: 8656/36938 -- GLOBAL_STEP: 1412300 | > loss_text_ce: 0.03006916679441929 (0.030692469589339057) | > loss_mel_ce: 2.89111328125 (nan) | > loss: 0.01159199420362711 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.1372393209921005) | > loader_time: 0.0045 (0.004708744213673608)  --> TIME: 2025-02-26 21:30:13 -- STEP: 8706/36938 -- GLOBAL_STEP: 1412350 | > loss_text_ce: 0.04118182137608528 (0.030697424596852922) | > loss_mel_ce: 2.676165819168091 (nan) | > loss: 0.010783125646412373 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13723895083124352) | > loader_time: 0.0045 (0.004708586439110317)  --> TIME: 2025-02-26 21:30:22 -- STEP: 8756/36938 -- GLOBAL_STEP: 1412400 | > loss_text_ce: 0.03140697255730629 (0.03069206853733301) | > loss_mel_ce: 3.2938663959503174 (nan) | > loss: 0.013195529580116272 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13722781508194282) | > loader_time: 0.0047 (0.004708342842992527)  --> TIME: 2025-02-26 21:30:31 -- STEP: 8806/36938 -- GLOBAL_STEP: 1412450 | > loss_text_ce: 0.030517656356096268 (0.030693370864440846) | > loss_mel_ce: 2.3599278926849365 (nan) | > loss: 0.009485894814133644 (nan) | > current_lr: 5e-06 | > step_time: 0.1267 (0.13720527985083436) | > loader_time: 0.0045 (0.004707687068413286)  --> TIME: 2025-02-26 21:30:41 -- STEP: 8856/36938 -- GLOBAL_STEP: 1412500 | > loss_text_ce: 0.03000701777637005 (0.030690977271320682) | > loss_mel_ce: 1.9631783962249756 (nan) | > loss: 0.00790946651250124 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1372217421555359) | > loader_time: 0.0051 (0.004707789921825146)  --> TIME: 2025-02-26 21:30:50 -- STEP: 8906/36938 -- GLOBAL_STEP: 1412550 | > loss_text_ce: 0.03369797021150589 (0.030685997103664476) | > loss_mel_ce: 2.414738178253174 (nan) | > loss: 0.009716017171740532 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.1371945860732297) | > loader_time: 0.0044 (0.004707388494514253)  --> TIME: 2025-02-26 21:30:59 -- STEP: 8956/36938 -- GLOBAL_STEP: 1412600 | > loss_text_ce: 0.027994830161333084 (0.03068983135484958) | > loss_mel_ce: 1.3083949089050293 (nan) | > loss: 0.005303134210407734 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13716433704724554) | > loader_time: 0.0045 (0.004707306503663884)  --> TIME: 2025-02-26 21:31:08 -- STEP: 9006/36938 -- GLOBAL_STEP: 1412650 | > loss_text_ce: 0.03203083574771881 (0.030687242627430108) | > loss_mel_ce: 2.366987466812134 (nan) | > loss: 0.009519914165139198 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13717578404113254) | > loader_time: 0.0052 (0.004707701598541623)  --> TIME: 2025-02-26 21:31:17 -- STEP: 9056/36938 -- GLOBAL_STEP: 1412700 | > loss_text_ce: 0.027036240324378014 (0.03068941257492551) | > loss_mel_ce: 2.5460431575775146 (nan) | > loss: 0.010210633277893066 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13715803575810517) | > loader_time: 0.0045 (0.004707482752024892)  --> TIME: 2025-02-26 21:31:26 -- STEP: 9106/36938 -- GLOBAL_STEP: 1412750 | > loss_text_ce: 0.0328616239130497 (0.030687992702600937) | > loss_mel_ce: 2.0092570781707764 (nan) | > loss: 0.008103646337985992 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13713011078171491) | > loader_time: 0.0045 (0.004707310347930929)  --> TIME: 2025-02-26 21:31:36 -- STEP: 9156/36938 -- GLOBAL_STEP: 1412800 | > loss_text_ce: 0.028227584436535835 (0.030691810579243816) | > loss_mel_ce: 2.6227774620056152 (nan) | > loss: 0.010519862174987793 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.1371802252262834) | > loader_time: 0.0047 (0.004707917603617287)  --> TIME: 2025-02-26 21:31:46 -- STEP: 9206/36938 -- GLOBAL_STEP: 1412850 | > loss_text_ce: 0.04243246093392372 (0.030697464512982) | > loss_mel_ce: 1.9680942296981812 (nan) | > loss: 0.007978281006217003 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1371844622872836) | > loader_time: 0.0047 (0.004708222557662884)  --> TIME: 2025-02-26 21:31:55 -- STEP: 9256/36938 -- GLOBAL_STEP: 1412900 | > loss_text_ce: 0.02739698812365532 (0.030697652445051602) | > loss_mel_ce: 2.865537643432617 (nan) | > loss: 0.011479899287223816 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13720217182292038) | > loader_time: 0.0045 (0.004708384890543949)  --> TIME: 2025-02-26 21:32:04 -- STEP: 9306/36938 -- GLOBAL_STEP: 1412950 | > loss_text_ce: 0.03285444900393486 (0.03069701434222971) | > loss_mel_ce: 2.1297357082366943 (nan) | > loss: 0.008581708185374737 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13719489365374077) | > loader_time: 0.0045 (0.004708339162162858)  --> TIME: 2025-02-26 21:32:13 -- STEP: 9356/36938 -- GLOBAL_STEP: 1413000 | > loss_text_ce: 0.03634170442819595 (0.03069149336519613) | > loss_mel_ce: 2.280805826187134 (nan) | > loss: 0.00919503066688776 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.137163867548827) | > loader_time: 0.0046 (0.004708096735392822)  --> TIME: 2025-02-26 21:32:23 -- STEP: 9406/36938 -- GLOBAL_STEP: 1413050 | > loss_text_ce: 0.026067711412906647 (0.030696241161404823) | > loss_mel_ce: 1.1622443199157715 (nan) | > loss: 0.004715524148195982 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13717593749786577) | > loader_time: 0.0045 (0.004707974625120438)  --> TIME: 2025-02-26 21:32:32 -- STEP: 9456/36938 -- GLOBAL_STEP: 1413100 | > loss_text_ce: 0.02805745229125023 (0.03069445467297959) | > loss_mel_ce: 2.4848968982696533 (nan) | > loss: 0.009972041472792625 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13720021854499784) | > loader_time: 0.0046 (0.004707510650863883)  --> TIME: 2025-02-26 21:32:41 -- STEP: 9506/36938 -- GLOBAL_STEP: 1413150 | > loss_text_ce: 0.032827265560626984 (0.030685784674549195) | > loss_mel_ce: 2.5666730403900146 (nan) | > loss: 0.010315478779375553 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.1371802155805482) | > loader_time: 0.0045 (0.0047071869188224594)  --> TIME: 2025-02-26 21:32:51 -- STEP: 9556/36938 -- GLOBAL_STEP: 1413200 | > loss_text_ce: 0.03360127657651901 (0.030690822325406743) | > loss_mel_ce: 2.259312152862549 (nan) | > loss: 0.00909886322915554 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.1372027232238757) | > loader_time: 0.0044 (0.004707071934848085)  --> TIME: 2025-02-26 21:33:01 -- STEP: 9606/36938 -- GLOBAL_STEP: 1413250 | > loss_text_ce: 0.033994160592556 (0.030698482733659643) | > loss_mel_ce: 1.3076685667037964 (nan) | > loss: 0.005324058700352907 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13721510655627225) | > loader_time: 0.0044 (0.004706591932370987)  --> TIME: 2025-02-26 21:33:10 -- STEP: 9656/36938 -- GLOBAL_STEP: 1413300 | > loss_text_ce: 0.022902261465787888 (0.030700096518129993) | > loss_mel_ce: 2.737306833267212 (nan) | > loss: 0.010953211225569248 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.1372335326503691) | > loader_time: 0.0048 (0.004706153443951964)  --> TIME: 2025-02-26 21:33:20 -- STEP: 9706/36938 -- GLOBAL_STEP: 1413350 | > loss_text_ce: 0.038797955960035324 (0.03069623056687868) | > loss_mel_ce: 2.7375335693359375 (nan) | > loss: 0.011017188429832458 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1372448151170056) | > loader_time: 0.0051 (0.004706157892236187)  --> TIME: 2025-02-26 21:33:29 -- STEP: 9756/36938 -- GLOBAL_STEP: 1413400 | > loss_text_ce: 0.03988858684897423 (0.030698571855132827) | > loss_mel_ce: 0.2578125 (nan) | > loss: 0.0011813535820692778 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.1372320096040164) | > loader_time: 0.0047 (0.00470676928966343)  --> TIME: 2025-02-26 21:33:38 -- STEP: 9806/36938 -- GLOBAL_STEP: 1413450 | > loss_text_ce: 0.030431486666202545 (0.03070548146601456) | > loss_mel_ce: 2.5327723026275635 (nan) | > loss: 0.010171444155275822 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.13724107336759636) | > loader_time: 0.0048 (0.004706360990845983)  --> TIME: 2025-02-26 21:33:48 -- STEP: 9856/36938 -- GLOBAL_STEP: 1413500 | > loss_text_ce: 0.02606760524213314 (0.030708886938590665) | > loss_mel_ce: 2.1937777996063232 (nan) | > loss: 0.008808910846710205 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.1372404530305746) | > loader_time: 0.0051 (0.004706529610149247)  --> TIME: 2025-02-26 21:33:57 -- STEP: 9906/36938 -- GLOBAL_STEP: 1413550 | > loss_text_ce: 0.03267829492688179 (0.030708159398747843) | > loss_mel_ce: 2.764164447784424 (nan) | > loss: 0.011098583228886127 (nan) | > current_lr: 5e-06 | > step_time: 0.1896 (0.13722107120650937) | > loader_time: 0.0048 (0.0047062960341076095)  --> TIME: 2025-02-26 21:34:06 -- STEP: 9956/36938 -- GLOBAL_STEP: 1413600 | > loss_text_ce: 0.036609750241041183 (0.030712753908211793) | > loss_mel_ce: 2.5999560356140137 (nan) | > loss: 0.010462562553584576 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13722986624680483) | > loader_time: 0.0046 (0.00470632652337887)  --> TIME: 2025-02-26 21:34:15 -- STEP: 10006/36938 -- GLOBAL_STEP: 1413650 | > loss_text_ce: 0.03422192111611366 (0.03071194441167014) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13722825033674646) | > loader_time: 0.0045 (0.004706573943817279)  --> TIME: 2025-02-26 21:34:25 -- STEP: 10056/36938 -- GLOBAL_STEP: 1413700 | > loss_text_ce: 0.02669679746031761 (0.03071423420801466) | > loss_mel_ce: 2.872307062149048 (nan) | > loss: 0.011503984220325947 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.1372223150995668) | > loader_time: 0.0053 (0.004706213800510693)  --> TIME: 2025-02-26 21:34:35 -- STEP: 10106/36938 -- GLOBAL_STEP: 1413750 | > loss_text_ce: 0.029179029166698456 (0.030716992760684982) | > loss_mel_ce: 3.1486308574676514 (nan) | > loss: 0.012610357254743576 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13726803005130858) | > loader_time: 0.0048 (0.004705996931413105)  --> TIME: 2025-02-26 21:34:44 -- STEP: 10156/36938 -- GLOBAL_STEP: 1413800 | > loss_text_ce: 0.03061612695455551 (0.03071472942053336) | > loss_mel_ce: 2.4523658752441406 (nan) | > loss: 0.009853103198111057 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13726580157060173) | > loader_time: 0.0046 (0.004705581598368225)  --> TIME: 2025-02-26 21:34:54 -- STEP: 10206/36938 -- GLOBAL_STEP: 1413850 | > loss_text_ce: 0.03371036425232887 (0.030716827282433495) | > loss_mel_ce: 2.801814556121826 (nan) | > loss: 0.01125208381563425 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13729239942138338) | > loader_time: 0.0045 (0.0047054782045511025)  --> TIME: 2025-02-26 21:35:03 -- STEP: 10256/36938 -- GLOBAL_STEP: 1413900 | > loss_text_ce: 0.023692289367318153 (0.030717038458947667) | > loss_mel_ce: 2.204134225845337 (nan) | > loss: 0.008840582333505154 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13732446083877398) | > loader_time: 0.0047 (0.004705349945240918)  --> TIME: 2025-02-26 21:35:13 -- STEP: 10306/36938 -- GLOBAL_STEP: 1413950 | > loss_text_ce: 0.031220287084579468 (0.030725242256378668) | > loss_mel_ce: 2.0723328590393066 (nan) | > loss: 0.008347433060407639 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13733887274520165) | > loader_time: 0.0044 (0.0047050164829937164)  --> TIME: 2025-02-26 21:35:22 -- STEP: 10356/36938 -- GLOBAL_STEP: 1414000 | > loss_text_ce: 0.02321128360927105 (0.030716148969602577) | > loss_mel_ce: 2.5342068672180176 (nan) | > loss: 0.010148485191166401 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.1373388854266588) | > loader_time: 0.0045 (0.004705238015368961)  --> TIME: 2025-02-26 21:35:32 -- STEP: 10406/36938 -- GLOBAL_STEP: 1414050 | > loss_text_ce: 0.03213620185852051 (0.030716402989486604) | > loss_mel_ce: 1.8772826194763184 (nan) | > loss: 0.0075770593248307705 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13733318883960693) | > loader_time: 0.0045 (0.004705021364423528)  --> TIME: 2025-02-26 21:35:41 -- STEP: 10456/36938 -- GLOBAL_STEP: 1414100 | > loss_text_ce: 0.026676824316382408 (0.03071197683631864) | > loss_mel_ce: 2.85306978225708 (nan) | > loss: 0.011427566409111023 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13731976217136305) | > loader_time: 0.0045 (0.004704698589619708)  --> TIME: 2025-02-26 21:35:50 -- STEP: 10506/36938 -- GLOBAL_STEP: 1414150 | > loss_text_ce: 0.029512422159314156 (0.030717268188223575) | > loss_mel_ce: 0.7326921820640564 (nan) | > loss: 0.003024621633812785 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13731491356923492) | > loader_time: 0.0053 (0.004704977543313611)  --> TIME: 2025-02-26 21:35:59 -- STEP: 10556/36938 -- GLOBAL_STEP: 1414200 | > loss_text_ce: 0.027950789779424667 (0.030713936731740894) | > loss_mel_ce: 2.859618902206421 (nan) | > loss: 0.01145861018449068 (nan) | > current_lr: 5e-06 | > step_time: 0.1664 (0.1373100962861096) | > loader_time: 0.0056 (0.004705131957367422)  --> TIME: 2025-02-26 21:36:09 -- STEP: 10606/36938 -- GLOBAL_STEP: 1414250 | > loss_text_ce: 0.023686382919549942 (0.030710696095754223) | > loss_mel_ce: 2.0169456005096436 (nan) | > loss: 0.008097746409475803 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.1373207755927738) | > loader_time: 0.0044 (0.004705606890740781)  --> TIME: 2025-02-26 21:36:18 -- STEP: 10656/36938 -- GLOBAL_STEP: 1414300 | > loss_text_ce: 0.023691300302743912 (0.030712797840660045) | > loss_mel_ce: 3.0485591888427734 (nan) | > loss: 0.0121914716437459 (nan) | > current_lr: 5e-06 | > step_time: 0.1738 (0.13732865524363527) | > loader_time: 0.0046 (0.00470536457257225)  --> TIME: 2025-02-26 21:36:28 -- STEP: 10706/36938 -- GLOBAL_STEP: 1414350 | > loss_text_ce: 0.026441726833581924 (0.03071142620879057) | > loss_mel_ce: 2.0825586318969727 (nan) | > loss: 0.008369049988687038 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.1373320772294655) | > loader_time: 0.0045 (0.004705455845234724)  --> TIME: 2025-02-26 21:36:37 -- STEP: 10756/36938 -- GLOBAL_STEP: 1414400 | > loss_text_ce: 0.03541562333703041 (0.030705744657175765) | > loss_mel_ce: 2.8316400051116943 (nan) | > loss: 0.011377206072211266 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13731644791091951) | > loader_time: 0.0051 (0.004705806765338193)  --> TIME: 2025-02-26 21:36:46 -- STEP: 10806/36938 -- GLOBAL_STEP: 1414450 | > loss_text_ce: 0.02094119042158127 (0.03070438169978881) | > loss_mel_ce: 2.225412368774414 (nan) | > loss: 0.008914101868867874 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.1373147099145617) | > loader_time: 0.0045 (0.00470565816285143)  --> TIME: 2025-02-26 21:36:55 -- STEP: 10856/36938 -- GLOBAL_STEP: 1414500 | > loss_text_ce: 0.020987996831536293 (0.0307033970153684) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13730388991781708) | > loader_time: 0.0045 (0.004705143396522312)  --> TIME: 2025-02-26 21:37:05 -- STEP: 10906/36938 -- GLOBAL_STEP: 1414550 | > loss_text_ce: 0.03103720210492611 (0.03070339143958354) | > loss_mel_ce: 1.4367014169692993 (nan) | > loss: 0.005824360065162182 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.13730637760046857) | > loader_time: 0.0048 (0.0047051259493272254)  --> TIME: 2025-02-26 21:37:14 -- STEP: 10956/36938 -- GLOBAL_STEP: 1414600 | > loss_text_ce: 0.033423859626054764 (0.030703987688823788) | > loss_mel_ce: 2.495515823364258 (nan) | > loss: 0.010035475715994835 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.1373126365355909) | > loader_time: 0.0048 (0.004704967930004404)  --> TIME: 2025-02-26 21:37:24 -- STEP: 11006/36938 -- GLOBAL_STEP: 1414650 | > loss_text_ce: 0.02541181445121765 (0.030702310754333186) | > loss_mel_ce: 2.6026384830474854 (nan) | > loss: 0.010428771376609802 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13733504138251862) | > loader_time: 0.0048 (0.0047049883515449436)  --> TIME: 2025-02-26 21:37:33 -- STEP: 11056/36938 -- GLOBAL_STEP: 1414700 | > loss_text_ce: 0.025038795545697212 (0.030708022013407034) | > loss_mel_ce: 2.183603286743164 (nan) | > loss: 0.00876445323228836 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13731269697980764) | > loader_time: 0.0045 (0.004704768466190366)  --> TIME: 2025-02-26 21:37:43 -- STEP: 11106/36938 -- GLOBAL_STEP: 1414750 | > loss_text_ce: 0.02588174119591713 (0.030703970846882225) | > loss_mel_ce: 2.9761431217193604 (nan) | > loss: 0.011912797577679157 (nan) | > current_lr: 5e-06 | > step_time: 0.1786 (0.1373525209605482) | > loader_time: 0.0048 (0.004704771590580606)  --> TIME: 2025-02-26 21:37:53 -- STEP: 11156/36938 -- GLOBAL_STEP: 1414800 | > loss_text_ce: 0.023791780695319176 (0.03070651837634628) | > loss_mel_ce: 2.4376401901245117 (nan) | > loss: 0.00976758822798729 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13736885491132061) | > loader_time: 0.0045 (0.004704860022692904)  --> TIME: 2025-02-26 21:38:02 -- STEP: 11206/36938 -- GLOBAL_STEP: 1414850 | > loss_text_ce: 0.034810155630111694 (0.03070325151246421) | > loss_mel_ce: 2.33559513092041 (nan) | > loss: 0.009406370110809803 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13736470126901404) | > loader_time: 0.0045 (0.004704522869532882)  --> TIME: 2025-02-26 21:38:11 -- STEP: 11256/36938 -- GLOBAL_STEP: 1414900 | > loss_text_ce: 0.031434427946805954 (0.030705333778627393) | > loss_mel_ce: 2.6963460445404053 (nan) | > loss: 0.010824526660144329 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13735472223990794) | > loader_time: 0.0045 (0.0047042924161489206)  --> TIME: 2025-02-26 21:38:20 -- STEP: 11306/36938 -- GLOBAL_STEP: 1414950 | > loss_text_ce: 0.026483800262212753 (0.03070695406434224) | > loss_mel_ce: 2.9526989459991455 (nan) | > loss: 0.011822154745459557 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1373456330630871) | > loader_time: 0.0045 (0.004704367496599107)  --> TIME: 2025-02-26 21:38:29 -- STEP: 11356/36938 -- GLOBAL_STEP: 1415000 | > loss_text_ce: 0.023511020466685295 (0.030706791728213383) | > loss_mel_ce: 2.479670286178589 (nan) | > loss: 0.009933259338140488 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.1373338783320588) | > loader_time: 0.0045 (0.004703884731089815)  --> TIME: 2025-02-26 21:38:39 -- STEP: 11406/36938 -- GLOBAL_STEP: 1415050 | > loss_text_ce: 0.03224220126867294 (0.030708551861906887) | > loss_mel_ce: 2.7216811180114746 (nan) | > loss: 0.010928268544375896 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.1373566428590353) | > loader_time: 0.0046 (0.004703882951684596)  --> TIME: 2025-02-26 21:38:49 -- STEP: 11456/36938 -- GLOBAL_STEP: 1415100 | > loss_text_ce: 0.03623153269290924 (0.03070815622407551) | > loss_mel_ce: 2.6239118576049805 (nan) | > loss: 0.010556125082075596 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.1373654029555819) | > loader_time: 0.0047 (0.004704159814552206)  --> TIME: 2025-02-26 21:38:58 -- STEP: 11506/36938 -- GLOBAL_STEP: 1415150 | > loss_text_ce: 0.031149838119745255 (0.03070990160888221) | > loss_mel_ce: 2.753891706466675 (nan) | > loss: 0.011051752604544163 (nan) | > current_lr: 5e-06 | > step_time: 0.1733 (0.13734457373888387) | > loader_time: 0.0048 (0.0047039016109952396)  --> TIME: 2025-02-26 21:39:07 -- STEP: 11556/36938 -- GLOBAL_STEP: 1415200 | > loss_text_ce: 0.03327256441116333 (0.03070392620210772) | > loss_mel_ce: 1.0518231391906738 (nan) | > loss: 0.004305935464799404 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13734668004302267) | > loader_time: 0.0045 (0.0047036541626501935)  --> TIME: 2025-02-26 21:39:17 -- STEP: 11606/36938 -- GLOBAL_STEP: 1415250 | > loss_text_ce: 0.03346210718154907 (0.030702396039742685) | > loss_mel_ce: 3.026071071624756 (nan) | > loss: 0.012141005136072636 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.1373690911742825) | > loader_time: 0.0046 (0.004703466489194637)  --> TIME: 2025-02-26 21:39:26 -- STEP: 11656/36938 -- GLOBAL_STEP: 1415300 | > loss_text_ce: 0.025379642844200134 (0.0306997817686766) | > loss_mel_ce: 2.4282641410827637 (nan) | > loss: 0.009736682288348675 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1373716540424964) | > loader_time: 0.0045 (0.004703612362757544)  --> TIME: 2025-02-26 21:39:35 -- STEP: 11706/36938 -- GLOBAL_STEP: 1415350 | > loss_text_ce: 0.026660004630684853 (0.03070041895835291) | > loss_mel_ce: 2.9853675365448 (nan) | > loss: 0.011952490545809269 (nan) | > current_lr: 5e-06 | > step_time: 0.1621 (0.13736159196984224) | > loader_time: 0.0048 (0.004703688169368883)  --> TIME: 2025-02-26 21:39:45 -- STEP: 11756/36938 -- GLOBAL_STEP: 1415400 | > loss_text_ce: 0.026605751365423203 (0.03069837384603097) | > loss_mel_ce: 0.50390625 (nan) | > loss: 0.002105206483975053 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13736203443683534) | > loader_time: 0.0045 (0.00470378667410232)  --> TIME: 2025-02-26 21:39:54 -- STEP: 11806/36938 -- GLOBAL_STEP: 1415450 | > loss_text_ce: 0.03861349821090698 (0.030696479426707945) | > loss_mel_ce: 0.0729166641831398 (nan) | > loss: 0.0004425800289027393 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13737824569813853) | > loader_time: 0.0045 (0.004703337068216835)  --> TIME: 2025-02-26 21:40:04 -- STEP: 11856/36938 -- GLOBAL_STEP: 1415500 | > loss_text_ce: 0.03497414290904999 (0.030692932529732535) | > loss_mel_ce: 2.712981939315796 (nan) | > loss: 0.010904587805271149 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13739378637710795) | > loader_time: 0.0045 (0.00470329183637692)  --> TIME: 2025-02-26 21:40:13 -- STEP: 11906/36938 -- GLOBAL_STEP: 1415550 | > loss_text_ce: 0.035771388560533524 (0.0306927941669315) | > loss_mel_ce: 2.9970703125 (nan) | > loss: 0.012035086750984192 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1373961630056581) | > loader_time: 0.0048 (0.004703263064582609)  --> TIME: 2025-02-26 21:40:23 -- STEP: 11956/36938 -- GLOBAL_STEP: 1415600 | > loss_text_ce: 0.026243915781378746 (0.03069765418408051) | > loss_mel_ce: 2.90962815284729 (nan) | > loss: 0.011650286614894867 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13739057574713956) | > loader_time: 0.0045 (0.004702862866868437)  --> TIME: 2025-02-26 21:40:32 -- STEP: 12006/36938 -- GLOBAL_STEP: 1415650 | > loss_text_ce: 0.02247144840657711 (0.030696716722052573) | > loss_mel_ce: 2.775164842605591 (nan) | > loss: 0.011101732030510902 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.1373852509648398) | > loader_time: 0.0046 (0.0047023435761367124)  --> TIME: 2025-02-26 21:40:41 -- STEP: 12056/36938 -- GLOBAL_STEP: 1415700 | > loss_text_ce: 0.02289436012506485 (0.030695075138537534) | > loss_mel_ce: 2.6054441928863525 (nan) | > loss: 0.010429915972054005 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13738505869569792) | > loader_time: 0.0045 (0.004702210287899051)  --> TIME: 2025-02-26 21:40:50 -- STEP: 12106/36938 -- GLOBAL_STEP: 1415750 | > loss_text_ce: 0.030142996460199356 (0.03069141921289567) | > loss_mel_ce: 2.906266927719116 (nan) | > loss: 0.011652421206235886 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13737358521887255) | > loader_time: 0.0046 (0.0047016884893419965)  --> TIME: 2025-02-26 21:41:00 -- STEP: 12156/36938 -- GLOBAL_STEP: 1415800 | > loss_text_ce: 0.02250359207391739 (0.030691569116884095) | > loss_mel_ce: 3.0476274490356445 (nan) | > loss: 0.012183060869574547 (nan) | > current_lr: 5e-06 | > step_time: 0.1887 (0.13737762284223845) | > loader_time: 0.0047 (0.004701413775635744)  --> TIME: 2025-02-26 21:41:09 -- STEP: 12206/36938 -- GLOBAL_STEP: 1415850 | > loss_text_ce: 0.027253631502389908 (0.030695274348401105) | > loss_mel_ce: 1.946549415588379 (nan) | > loss: 0.007832552306354046 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13736706180062971) | > loader_time: 0.0046 (0.0047011064268146635)  --> TIME: 2025-02-26 21:41:18 -- STEP: 12256/36938 -- GLOBAL_STEP: 1415900 | > loss_text_ce: 0.03314089775085449 (0.030690693627424694) | > loss_mel_ce: 2.882922887802124 (nan) | > loss: 0.011571682058274746 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13735254194499816) | > loader_time: 0.0047 (0.004700787949032918)  --> TIME: 2025-02-26 21:41:28 -- STEP: 12306/36938 -- GLOBAL_STEP: 1415950 | > loss_text_ce: 0.02594761550426483 (0.030687483015392866) | > loss_mel_ce: 2.6651647090911865 (nan) | > loss: 0.010679017752408981 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13736550588637117) | > loader_time: 0.0044 (0.004700623662146647)  --> TIME: 2025-02-26 21:41:37 -- STEP: 12356/36938 -- GLOBAL_STEP: 1416000 | > loss_text_ce: 0.028090698644518852 (0.03068562723069421) | > loss_mel_ce: 2.9146432876586914 (nan) | > loss: 0.011677516624331474 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.1373609725075953) | > loader_time: 0.0047 (0.004700190930661326)  --> TIME: 2025-02-26 21:41:46 -- STEP: 12406/36938 -- GLOBAL_STEP: 1416050 | > loss_text_ce: 0.03219190612435341 (0.030684023436868164) | > loss_mel_ce: 3.077599048614502 (nan) | > loss: 0.012340441346168518 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13735872335094051) | > loader_time: 0.0045 (0.004699717716461158)  --> TIME: 2025-02-26 21:41:56 -- STEP: 12456/36938 -- GLOBAL_STEP: 1416100 | > loss_text_ce: 0.02846807800233364 (0.03068772821618973) | > loss_mel_ce: 2.7601451873779297 (nan) | > loss: 0.011065926402807236 (nan) | > current_lr: 5e-06 | > step_time: 0.1892 (0.13734973005246895) | > loader_time: 0.0048 (0.004699273586426295)  --> TIME: 2025-02-26 21:42:05 -- STEP: 12506/36938 -- GLOBAL_STEP: 1416150 | > loss_text_ce: 0.03186006844043732 (0.030685736687683252) | > loss_mel_ce: 2.3779842853546143 (nan) | > loss: 0.009562875144183636 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13736631363920898) | > loader_time: 0.0045 (0.004698910466121988)  --> TIME: 2025-02-26 21:42:15 -- STEP: 12556/36938 -- GLOBAL_STEP: 1416200 | > loss_text_ce: 0.0309744942933321 (0.030687293593109243) | > loss_mel_ce: 0.9578471779823303 (nan) | > loss: 0.0039238957688212395 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13736832758999676) | > loader_time: 0.0046 (0.004698590702143141)  --> TIME: 2025-02-26 21:42:24 -- STEP: 12606/36938 -- GLOBAL_STEP: 1416250 | > loss_text_ce: 0.028440358117222786 (0.03068295619058211) | > loss_mel_ce: 1.0109506845474243 (nan) | > loss: 0.004124567843973637 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13736482071683467) | > loader_time: 0.0045 (0.004698191051461947)  --> TIME: 2025-02-26 21:42:33 -- STEP: 12656/36938 -- GLOBAL_STEP: 1416300 | > loss_text_ce: 0.03638410195708275 (0.030679543643843094) | > loss_mel_ce: 1.995608925819397 (nan) | > loss: 0.008063464425504208 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13737428397429888) | > loader_time: 0.0045 (0.004697790169233592)  --> TIME: 2025-02-26 21:42:43 -- STEP: 12706/36938 -- GLOBAL_STEP: 1416350 | > loss_text_ce: 0.027932841330766678 (0.03067831572621074) | > loss_mel_ce: 1.9110560417175293 (nan) | > loss: 0.00769440084695816 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13736905981081998) | > loader_time: 0.0045 (0.0046974200067530355)  --> TIME: 2025-02-26 21:42:52 -- STEP: 12756/36938 -- GLOBAL_STEP: 1416400 | > loss_text_ce: 0.03705113008618355 (0.030679552027267604) | > loss_mel_ce: 2.549093246459961 (nan) | > loss: 0.01026247814297676 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.13737524940736381) | > loader_time: 0.0047 (0.004697065119385954)  --> TIME: 2025-02-26 21:43:01 -- STEP: 12806/36938 -- GLOBAL_STEP: 1416450 | > loss_text_ce: 0.031992893666028976 (0.03068192434181113) | > loss_mel_ce: 2.8305885791778564 (nan) | > loss: 0.011359451338648796 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13735303734682838) | > loader_time: 0.0047 (0.004696596232909745)  --> TIME: 2025-02-26 21:43:11 -- STEP: 12856/36938 -- GLOBAL_STEP: 1416500 | > loss_text_ce: 0.01716218702495098 (0.0306769542420844) | > loss_mel_ce: 3.0517737865448 (nan) | > loss: 0.01217831764370203 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13734962966921171) | > loader_time: 0.0044 (0.004696213742851365)  --> TIME: 2025-02-26 21:43:20 -- STEP: 12906/36938 -- GLOBAL_STEP: 1416550 | > loss_text_ce: 0.020291659981012344 (0.030672973893004375) | > loss_mel_ce: 2.167961835861206 (nan) | > loss: 0.008683545514941216 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13734480875805172) | > loader_time: 0.0045 (0.0046957689127440725)  --> TIME: 2025-02-26 21:43:29 -- STEP: 12956/36938 -- GLOBAL_STEP: 1416600 | > loss_text_ce: 0.030447006225585938 (0.030673629361601316) | > loss_mel_ce: 3.479161262512207 (nan) | > loss: 0.013927017338573933 (nan) | > current_lr: 5e-06 | > step_time: 0.161 (0.13733719312765438) | > loader_time: 0.0046 (0.004695435039935974)  --> TIME: 2025-02-26 21:43:38 -- STEP: 13006/36938 -- GLOBAL_STEP: 1416650 | > loss_text_ce: 0.040209922939538956 (0.030680900269338222) | > loss_mel_ce: 2.4698421955108643 (nan) | > loss: 0.00996052473783493 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.1373380671899595) | > loader_time: 0.0045 (0.004695081461507949)  --> TIME: 2025-02-26 21:43:48 -- STEP: 13056/36938 -- GLOBAL_STEP: 1416700 | > loss_text_ce: 0.032885827124118805 (0.03067991571907781) | > loss_mel_ce: 3.0185635089874268 (nan) | > loss: 0.01210892666131258 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.1373517826820397) | > loader_time: 0.0046 (0.004694802796139405)  --> TIME: 2025-02-26 21:43:57 -- STEP: 13106/36938 -- GLOBAL_STEP: 1416750 | > loss_text_ce: 0.02581733465194702 (0.030677523061618334) | > loss_mel_ce: 2.8098995685577393 (nan) | > loss: 0.011252845637500286 (nan) | > current_lr: 5e-06 | > step_time: 0.1864 (0.137343872563084) | > loader_time: 0.0048 (0.004694403154730272)  --> TIME: 2025-02-26 21:44:06 -- STEP: 13156/36938 -- GLOBAL_STEP: 1416800 | > loss_text_ce: 0.031745243817567825 (0.030668536010267675) | > loss_mel_ce: 2.2052178382873535 (nan) | > loss: 0.00887683779001236 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.13732912397993405) | > loader_time: 0.0045 (0.004694018620566096)  --> TIME: 2025-02-26 21:44:16 -- STEP: 13206/36938 -- GLOBAL_STEP: 1416850 | > loss_text_ce: 0.01609877496957779 (0.030667325334803153) | > loss_mel_ce: 1.228289246559143 (nan) | > loss: 0.004938047844916582 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13733175204194722) | > loader_time: 0.0045 (0.004693674658457983)  --> TIME: 2025-02-26 21:44:25 -- STEP: 13256/36938 -- GLOBAL_STEP: 1416900 | > loss_text_ce: 0.030043011531233788 (0.030663030150396484) | > loss_mel_ce: 0.7790107727050781 (nan) | > loss: 0.0032105310820043087 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13734615296934619) | > loader_time: 0.0048 (0.004693334244357976)  --> TIME: 2025-02-26 21:44:35 -- STEP: 13306/36938 -- GLOBAL_STEP: 1416950 | > loss_text_ce: 0.03676721453666687 (0.03066739932670753) | > loss_mel_ce: 2.1811068058013916 (nan) | > loss: 0.008801087737083435 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.1373582491565771) | > loader_time: 0.0046 (0.004692923336410314)  --> TIME: 2025-02-26 21:44:44 -- STEP: 13356/36938 -- GLOBAL_STEP: 1417000 | > loss_text_ce: 0.03072339855134487 (0.030668935134078656) | > loss_mel_ce: 2.112544059753418 (nan) | > loss: 0.00850502960383892 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13735611066021206) | > loader_time: 0.0062 (0.004692829576356894)  --> TIME: 2025-02-26 21:44:54 -- STEP: 13406/36938 -- GLOBAL_STEP: 1417050 | > loss_text_ce: 0.027939043939113617 (0.030663909803085032) | > loss_mel_ce: 1.6018732786178589 (nan) | > loss: 0.006467509549111128 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13736375057926564) | > loader_time: 0.0045 (0.004692463061710979)  --> TIME: 2025-02-26 21:45:03 -- STEP: 13456/36938 -- GLOBAL_STEP: 1417100 | > loss_text_ce: 0.03353459760546684 (0.03066407748584954) | > loss_mel_ce: 2.332759380340576 (nan) | > loss: 0.009390056133270264 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1373638364172987) | > loader_time: 0.0045 (0.004692203242344999)  --> TIME: 2025-02-26 21:45:13 -- STEP: 13506/36938 -- GLOBAL_STEP: 1417150 | > loss_text_ce: 0.032118938863277435 (0.030666939852165946) | > loss_mel_ce: 1.5975220203399658 (nan) | > loss: 0.0064668296836316586 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13739349560332856) | > loader_time: 0.0045 (0.00469198540089199)  --> TIME: 2025-02-26 21:45:22 -- STEP: 13556/36938 -- GLOBAL_STEP: 1417200 | > loss_text_ce: 0.03290783613920212 (0.03066492823960288) | > loss_mel_ce: 2.7607674598693848 (nan) | > loss: 0.011086013168096542 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13737613092871456) | > loader_time: 0.0047 (0.004691579377767257)  --> TIME: 2025-02-26 21:45:31 -- STEP: 13606/36938 -- GLOBAL_STEP: 1417250 | > loss_text_ce: 0.03069259598851204 (0.030666262119752197) | > loss_mel_ce: 1.1321632862091064 (nan) | > loss: 0.004614507779479027 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13737017476136237) | > loader_time: 0.0046 (0.004691445089763808)  --> TIME: 2025-02-26 21:45:40 -- STEP: 13656/36938 -- GLOBAL_STEP: 1417300 | > loss_text_ce: 0.030841343104839325 (0.030663921585465843) | > loss_mel_ce: 2.5130646228790283 (nan) | > loss: 0.01009486522525549 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.1373699516250002) | > loader_time: 0.0046 (0.004691070835005102)  --> TIME: 2025-02-26 21:45:50 -- STEP: 13706/36938 -- GLOBAL_STEP: 1417350 | > loss_text_ce: 0.03805932030081749 (0.030665188158162274) | > loss_mel_ce: 3.2067012786865234 (nan) | > loss: 0.012876034714281559 (nan) | > current_lr: 5e-06 | > step_time: 0.1667 (0.1373750998402225) | > loader_time: 0.005 (0.004691438363204777)  --> TIME: 2025-02-26 21:45:59 -- STEP: 13756/36938 -- GLOBAL_STEP: 1417400 | > loss_text_ce: 0.04050581902265549 (0.030662130523355497) | > loss_mel_ce: 1.216522455215454 (nan) | > loss: 0.00498820748180151 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13736509368599045) | > loader_time: 0.0045 (0.004691743112366048)  --> TIME: 2025-02-26 21:46:08 -- STEP: 13806/36938 -- GLOBAL_STEP: 1417450 | > loss_text_ce: 0.029314041137695312 (0.030658375750903348) | > loss_mel_ce: 3.1153955459594727 (nan) | > loss: 0.012479007244110107 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.13736273314008754) | > loader_time: 0.0047 (0.004691486011186231)  --> TIME: 2025-02-26 21:46:18 -- STEP: 13856/36938 -- GLOBAL_STEP: 1417500 | > loss_text_ce: 0.022244010120630264 (0.030658746881807857) | > loss_mel_ce: 2.720613718032837 (nan) | > loss: 0.010884356684982777 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.1373705149350746) | > loader_time: 0.0048 (0.0046917365618720785)  --> TIME: 2025-02-26 21:46:28 -- STEP: 13906/36938 -- GLOBAL_STEP: 1417550 | > loss_text_ce: 0.03544887155294418 (0.03066008806990252) | > loss_mel_ce: 3.0416476726531982 (nan) | > loss: 0.012210700660943985 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.13739301648119906) | > loader_time: 0.0047 (0.00469173375915212)  --> TIME: 2025-02-26 21:46:37 -- STEP: 13956/36938 -- GLOBAL_STEP: 1417600 | > loss_text_ce: 0.034623172134160995 (0.030653407055215673) | > loss_mel_ce: 2.172297954559326 (nan) | > loss: 0.008757623843848705 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13738361425541765) | > loader_time: 0.0045 (0.0046919666446358504)  --> TIME: 2025-02-26 21:46:46 -- STEP: 14006/36938 -- GLOBAL_STEP: 1417650 | > loss_text_ce: 0.028350478038191795 (0.030648023149087194) | > loss_mel_ce: 2.6607143878936768 (nan) | > loss: 0.01067089382559061 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13738160618165404) | > loader_time: 0.0045 (0.0046925427282807726)  --> TIME: 2025-02-26 21:46:56 -- STEP: 14056/36938 -- GLOBAL_STEP: 1417700 | > loss_text_ce: 0.029543597251176834 (0.030645835474220957) | > loss_mel_ce: 2.8266162872314453 (nan) | > loss: 0.011333968490362167 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13740009074531223) | > loader_time: 0.0047 (0.004694041435054944)  --> TIME: 2025-02-26 21:47:05 -- STEP: 14106/36938 -- GLOBAL_STEP: 1417750 | > loss_text_ce: 0.03298177942633629 (0.030644600429039628) | > loss_mel_ce: 2.280778646469116 (nan) | > loss: 0.00918158981949091 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13739796415315592) | > loader_time: 0.0047 (0.004694807652833491)  --> TIME: 2025-02-26 21:47:14 -- STEP: 14156/36938 -- GLOBAL_STEP: 1417800 | > loss_text_ce: 0.0351545587182045 (0.03064573730995253) | > loss_mel_ce: 2.594147205352783 (nan) | > loss: 0.010433738119900227 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.1373935840533378) | > loader_time: 0.0048 (0.004695206922674991)  --> TIME: 2025-02-26 21:47:24 -- STEP: 14206/36938 -- GLOBAL_STEP: 1417850 | > loss_text_ce: 0.02840059995651245 (0.030648984429382572) | > loss_mel_ce: 3.140981674194336 (nan) | > loss: 0.012576914392411709 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13740814844856608) | > loader_time: 0.0047 (0.004695307565678577)  --> TIME: 2025-02-26 21:47:34 -- STEP: 14256/36938 -- GLOBAL_STEP: 1417900 | > loss_text_ce: 0.033276259899139404 (0.03065022269054029) | > loss_mel_ce: 2.915673017501831 (nan) | > loss: 0.011702180840075016 (nan) | > current_lr: 5e-06 | > step_time: 0.1689 (0.13741990718175115) | > loader_time: 0.0051 (0.004696911650340796)  --> TIME: 2025-02-26 21:47:43 -- STEP: 14306/36938 -- GLOBAL_STEP: 1417950 | > loss_text_ce: 0.027042239904403687 (0.03065370328342377) | > loss_mel_ce: 1.5721288919448853 (nan) | > loss: 0.006345917470753193 (nan) | > current_lr: 5e-06 | > step_time: 0.17 (0.13742987432630366) | > loader_time: 0.0046 (0.004697185070651633)  --> TIME: 2025-02-26 21:47:52 -- STEP: 14356/36938 -- GLOBAL_STEP: 1418000 | > loss_text_ce: 0.03534959629178047 (0.030649052208741544) | > loss_mel_ce: 1.7912975549697876 (nan) | > loss: 0.0072486004792153835 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.1374324306733549) | > loader_time: 0.0047 (0.004697161170558728)  --> TIME: 2025-02-26 21:48:02 -- STEP: 14406/36938 -- GLOBAL_STEP: 1418050 | > loss_text_ce: 0.030320117250084877 (0.03064763659240781) | > loss_mel_ce: 3.578892230987549 (nan) | > loss: 0.014322272501885891 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.13744767748546438) | > loader_time: 0.0047 (0.004697249760879932)  --> TIME: 2025-02-26 21:48:11 -- STEP: 14456/36938 -- GLOBAL_STEP: 1418100 | > loss_text_ce: 0.043096091598272324 (0.030643333155931377) | > loss_mel_ce: 2.894181489944458 (nan) | > loss: 0.011655864305794239 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13743168123862046) | > loader_time: 0.0046 (0.004697479922004836)  --> TIME: 2025-02-26 21:48:21 -- STEP: 14506/36938 -- GLOBAL_STEP: 1418150 | > loss_text_ce: 0.034229252487421036 (0.030643011293660736) | > loss_mel_ce: 2.8820042610168457 (nan) | > loss: 0.011572356335818768 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.1374310953245959) | > loader_time: 0.0045 (0.004697822347667515)  --> TIME: 2025-02-26 21:48:30 -- STEP: 14556/36938 -- GLOBAL_STEP: 1418200 | > loss_text_ce: 0.042247381061315536 (0.030647036015557953) | > loss_mel_ce: 3.1852407455444336 (nan) | > loss: 0.012807493098080158 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.1374244957089441) | > loader_time: 0.0046 (0.004697983574166174)  --> TIME: 2025-02-26 21:48:39 -- STEP: 14606/36938 -- GLOBAL_STEP: 1418250 | > loss_text_ce: 0.03001485951244831 (0.030647343521389116) | > loss_mel_ce: 2.3038456439971924 (nan) | > loss: 0.009261351078748703 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13741532451695307) | > loader_time: 0.0047 (0.004697618346728117)  --> TIME: 2025-02-26 21:48:48 -- STEP: 14656/36938 -- GLOBAL_STEP: 1418300 | > loss_text_ce: 0.033899955451488495 (0.030651456606205798) | > loss_mel_ce: 2.964890480041504 (nan) | > loss: 0.011899963021278381 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13741503158120472) | > loader_time: 0.0046 (0.00469842712561633)  --> TIME: 2025-02-26 21:48:58 -- STEP: 14706/36938 -- GLOBAL_STEP: 1418350 | > loss_text_ce: 0.03319745883345604 (0.030651046470816845) | > loss_mel_ce: 0.246337890625 (nan) | > loss: 0.0011092673521488905 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.13740945160672527) | > loader_time: 0.0046 (0.004698644945058238)  --> TIME: 2025-02-26 21:49:07 -- STEP: 14756/36938 -- GLOBAL_STEP: 1418400 | > loss_text_ce: 0.02612392045557499 (0.03065222094966996) | > loss_mel_ce: 2.425410509109497 (nan) | > loss: 0.009728312492370605 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13741952546019517) | > loader_time: 0.0054 (0.004698809439265359)  --> TIME: 2025-02-26 21:49:16 -- STEP: 14806/36938 -- GLOBAL_STEP: 1418450 | > loss_text_ce: 0.03942510858178139 (0.030655271971381133) | > loss_mel_ce: 2.66829252243042 (nan) | > loss: 0.01074491161853075 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13741979197394177) | > loader_time: 0.0045 (0.004698602779320889)  --> TIME: 2025-02-26 21:49:26 -- STEP: 14856/36938 -- GLOBAL_STEP: 1418500 | > loss_text_ce: 0.03924117237329483 (0.030654820221875972) | > loss_mel_ce: 2.837139368057251 (nan) | > loss: 0.011414208449423313 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13742229597633845) | > loader_time: 0.0046 (0.004698465813470866)  --> TIME: 2025-02-26 21:49:36 -- STEP: 14906/36938 -- GLOBAL_STEP: 1418550 | > loss_text_ce: 0.028873508796095848 (0.030653108474353024) | > loss_mel_ce: 2.2376961708068848 (nan) | > loss: 0.008994324132800102 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13743319574432958) | > loader_time: 0.0045 (0.004698562746829765)  --> TIME: 2025-02-26 21:49:45 -- STEP: 14956/36938 -- GLOBAL_STEP: 1418600 | > loss_text_ce: 0.03876612335443497 (0.030652555501311185) | > loss_mel_ce: 2.3575949668884277 (nan) | > loss: 0.009509369730949402 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1374203572301296) | > loader_time: 0.0045 (0.00469860757344159)  --> TIME: 2025-02-26 21:49:54 -- STEP: 15006/36938 -- GLOBAL_STEP: 1418650 | > loss_text_ce: 0.03233297914266586 (0.0306550338917701) | > loss_mel_ce: 2.4794347286224365 (nan) | > loss: 0.00996733270585537 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13743125161027558) | > loader_time: 0.0047 (0.004698584846507732)  --> TIME: 2025-02-26 21:50:04 -- STEP: 15056/36938 -- GLOBAL_STEP: 1418700 | > loss_text_ce: 0.02580614574253559 (0.030658514856847795) | > loss_mel_ce: 2.662811756134033 (nan) | > loss: 0.010669119656085968 (nan) | > current_lr: 5e-06 | > step_time: 0.1438 (0.13742724350800906) | > loader_time: 0.0049 (0.004698861481406068)  --> TIME: 2025-02-26 21:50:13 -- STEP: 15106/36938 -- GLOBAL_STEP: 1418750 | > loss_text_ce: 0.031333521008491516 (0.030657819939949146) | > loss_mel_ce: 2.5839133262634277 (nan) | > loss: 0.01037796400487423 (nan) | > current_lr: 5e-06 | > step_time: 0.1813 (0.13744438300618086) | > loader_time: 0.0051 (0.004699051356482878)  --> TIME: 2025-02-26 21:50:23 -- STEP: 15156/36938 -- GLOBAL_STEP: 1418800 | > loss_text_ce: 0.024174103513360023 (0.030653844507621035) | > loss_mel_ce: 2.5468032360076904 (nan) | > loss: 0.01020229235291481 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13745705491164242) | > loader_time: 0.0045 (0.0046992057009717)  --> TIME: 2025-02-26 21:50:32 -- STEP: 15206/36938 -- GLOBAL_STEP: 1418850 | > loss_text_ce: 0.028968440368771553 (0.030657995550159634) | > loss_mel_ce: 3.1055908203125 (nan) | > loss: 0.012438727542757988 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13745710602970518) | > loader_time: 0.0046 (0.004699456414219296)  --> TIME: 2025-02-26 21:50:42 -- STEP: 15256/36938 -- GLOBAL_STEP: 1418900 | > loss_text_ce: 0.02828417904675007 (0.030659479798073187) | > loss_mel_ce: 2.286328077316284 (nan) | > loss: 0.00918496958911419 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13746054036865926) | > loader_time: 0.0044 (0.004699325836573678)  --> TIME: 2025-02-26 21:50:51 -- STEP: 15306/36938 -- GLOBAL_STEP: 1418950 | > loss_text_ce: 0.027074016630649567 (0.030662520634423953) | > loss_mel_ce: 2.0223751068115234 (nan) | > loss: 0.008132735267281532 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13744478569643717) | > loader_time: 0.0046 (0.004698966447627542)  --> TIME: 2025-02-26 21:51:00 -- STEP: 15356/36938 -- GLOBAL_STEP: 1419000 | > loss_text_ce: 0.031192908063530922 (0.030668698841541928) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13744384511493873) | > loader_time: 0.0046 (0.00469875209932058)  --> TIME: 2025-02-26 21:51:10 -- STEP: 15406/36938 -- GLOBAL_STEP: 1419050 | > loss_text_ce: 0.017500415444374084 (0.03067070428289519) | > loss_mel_ce: 0.03041839599609375 (nan) | > loss: 0.00019015402358490974 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1374525311208911) | > loader_time: 0.0045 (0.004698543506490056)  --> TIME: 2025-02-26 21:51:19 -- STEP: 15456/36938 -- GLOBAL_STEP: 1419100 | > loss_text_ce: 0.037022851407527924 (0.030675520126138552) | > loss_mel_ce: 3.2250096797943115 (nan) | > loss: 0.01294457446783781 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.13745591534960938) | > loader_time: 0.0048 (0.004698239914748946)  --> TIME: 2025-02-26 21:51:28 -- STEP: 15506/36938 -- GLOBAL_STEP: 1419150 | > loss_text_ce: 0.022469226270914078 (0.03067568156533794) | > loss_mel_ce: 2.539846420288086 (nan) | > loss: 0.010167920030653477 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13744224638503927) | > loader_time: 0.0048 (0.004697876654331779)  --> TIME: 2025-02-26 21:51:38 -- STEP: 15556/36938 -- GLOBAL_STEP: 1419200 | > loss_text_ce: 0.03505921736359596 (0.030673104822209387) | > loss_mel_ce: 2.806349039077759 (nan) | > loss: 0.011275430209934711 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13746694433072848) | > loader_time: 0.0046 (0.0046977499482195465)  --> TIME: 2025-02-26 21:51:47 -- STEP: 15606/36938 -- GLOBAL_STEP: 1419250 | > loss_text_ce: 0.02712485007941723 (0.030669158730523668) | > loss_mel_ce: 2.303795337677002 (nan) | > loss: 0.009249684400856495 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13746681065799402) | > loader_time: 0.0054 (0.004697525041402461)  --> TIME: 2025-02-26 21:51:57 -- STEP: 15656/36938 -- GLOBAL_STEP: 1419300 | > loss_text_ce: 0.034573644399642944 (0.030669415943378346) | > loss_mel_ce: 2.8593292236328125 (nan) | > loss: 0.01148374192416668 (nan) | > current_lr: 5e-06 | > step_time: 0.1451 (0.1374647068349924) | > loader_time: 0.0046 (0.004697274190160009)  --> TIME: 2025-02-26 21:52:06 -- STEP: 15706/36938 -- GLOBAL_STEP: 1419350 | > loss_text_ce: 0.0355052649974823 (0.030672664961573767) | > loss_mel_ce: 0.7072471976280212 (nan) | > loss: 0.002947430359199643 (nan) | > current_lr: 5e-06 | > step_time: 0.1577 (0.13745949999414317) | > loader_time: 0.0047 (0.0046969315329585)  --> TIME: 2025-02-26 21:52:15 -- STEP: 15756/36938 -- GLOBAL_STEP: 1419400 | > loss_text_ce: 0.024713976308703423 (0.03067297032501338) | > loss_mel_ce: 2.9372332096099854 (nan) | > loss: 0.01175375934690237 (nan) | > current_lr: 5e-06 | > step_time: 0.169 (0.13744737138660865) | > loader_time: 0.0048 (0.004696576266640995)  --> TIME: 2025-02-26 21:52:25 -- STEP: 15806/36938 -- GLOBAL_STEP: 1419450 | > loss_text_ce: 0.0275937020778656 (0.030671799969761025) | > loss_mel_ce: 2.4364936351776123 (nan) | > loss: 0.009778124280273914 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13745854681659986) | > loader_time: 0.0046 (0.004696367707989174)  --> TIME: 2025-02-26 21:52:34 -- STEP: 15856/36938 -- GLOBAL_STEP: 1419500 | > loss_text_ce: 0.03675981983542442 (0.03066923623746233) | > loss_mel_ce: 2.9528939723968506 (nan) | > loss: 0.0118637066334486 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.13744945090365435) | > loader_time: 0.0051 (0.004696082590687254)  --> TIME: 2025-02-26 21:52:43 -- STEP: 15906/36938 -- GLOBAL_STEP: 1419550 | > loss_text_ce: 0.03255515545606613 (0.030670694290467926) | > loss_mel_ce: 2.14005184173584 (nan) | > loss: 0.008621457032859325 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13744825408786232) | > loader_time: 0.0045 (0.004695790017548975)  --> TIME: 2025-02-26 21:52:53 -- STEP: 15956/36938 -- GLOBAL_STEP: 1419600 | > loss_text_ce: 0.026930715888738632 (0.030670912359593922) | > loss_mel_ce: 1.9255093336105347 (nan) | > loss: 0.007747778203338385 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13746493134351423) | > loader_time: 0.0047 (0.0046955741387197415)  --> TIME: 2025-02-26 21:53:02 -- STEP: 16006/36938 -- GLOBAL_STEP: 1419650 | > loss_text_ce: 0.025707809254527092 (0.03066526282547054) | > loss_mel_ce: 2.47188138961792 (nan) | > loss: 0.009911068715155125 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13746245262191614) | > loader_time: 0.0053 (0.004695703949642851)  --> TIME: 2025-02-26 21:53:12 -- STEP: 16056/36938 -- GLOBAL_STEP: 1419700 | > loss_text_ce: 0.03385524824261665 (0.030668527089285805) | > loss_mel_ce: 2.6220903396606445 (nan) | > loss: 0.010539467446506023 (nan) | > current_lr: 5e-06 | > step_time: 0.1835 (0.13747951524971466) | > loader_time: 0.0049 (0.004695688736753521)  --> TIME: 2025-02-26 21:53:21 -- STEP: 16106/36938 -- GLOBAL_STEP: 1419750 | > loss_text_ce: 0.027269840240478516 (0.030668393511352417) | > loss_mel_ce: 2.4500606060028076 (nan) | > loss: 0.009830676950514317 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.13747584285401984) | > loader_time: 0.0046 (0.004695436147137691)  --> TIME: 2025-02-26 21:53:31 -- STEP: 16156/36938 -- GLOBAL_STEP: 1419800 | > loss_text_ce: 0.03097948059439659 (0.030667617003147258) | > loss_mel_ce: 3.0535576343536377 (nan) | > loss: 0.012240227311849594 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.13746816575099088) | > loader_time: 0.0049 (0.0046954762377813)  --> TIME: 2025-02-26 21:53:40 -- STEP: 16206/36938 -- GLOBAL_STEP: 1419850 | > loss_text_ce: 0.03030257858335972 (0.030667058929389982) | > loss_mel_ce: 1.8118443489074707 (nan) | > loss: 0.007310106884688139 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13747130291529183) | > loader_time: 0.0053 (0.004695709849469029)  --> TIME: 2025-02-26 21:53:49 -- STEP: 16256/36938 -- GLOBAL_STEP: 1419900 | > loss_text_ce: 0.039057519286870956 (0.030670287063039155) | > loss_mel_ce: 0.8921669125556946 (nan) | > loss: 0.003695335239171982 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.1374597781315784) | > loader_time: 0.0046 (0.004695520934274784)  --> TIME: 2025-02-26 21:53:58 -- STEP: 16306/36938 -- GLOBAL_STEP: 1419950 | > loss_text_ce: 0.03293710574507713 (0.030670008324310563) | > loss_mel_ce: 3.323179006576538 (nan) | > loss: 0.013317921198904514 (nan) | > current_lr: 5e-06 | > step_time: 0.1621 (0.13745260705073956) | > loader_time: 0.0046 (0.004695387438120553)  --> TIME: 2025-02-26 21:54:08 -- STEP: 16356/36938 -- GLOBAL_STEP: 1420000 | > loss_text_ce: 0.035961609333753586 (0.030670060491641244) | > loss_mel_ce: 2.41744065284729 (nan) | > loss: 0.009735723957419395 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13745146723066318) | > loader_time: 0.0046 (0.0046950386712765945) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1420000.pth  --> TIME: 2025-02-26 21:54:23 -- STEP: 16406/36938 -- GLOBAL_STEP: 1420050 | > loss_text_ce: 0.0334811806678772 (0.03067279550863727) | > loss_mel_ce: 3.029207706451416 (nan) | > loss: 0.012153527699410915 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.1374378259278167) | > loader_time: 0.0048 (0.004694992618474727)  --> TIME: 2025-02-26 21:54:32 -- STEP: 16456/36938 -- GLOBAL_STEP: 1420100 | > loss_text_ce: 0.03668152540922165 (0.03067530087865645) | > loss_mel_ce: 0.1136474609375 (nan) | > loss: 0.0005965436575934291 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13743147335124214) | > loader_time: 0.0046 (0.004694914609179281)  --> TIME: 2025-02-26 21:54:41 -- STEP: 16506/36938 -- GLOBAL_STEP: 1420150 | > loss_text_ce: 0.030148187652230263 (0.030674704612955756) | > loss_mel_ce: 2.658295154571533 (nan) | > loss: 0.010668426752090454 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13743027861618243) | > loader_time: 0.0046 (0.004694859649027704)  --> TIME: 2025-02-26 21:54:51 -- STEP: 16556/36938 -- GLOBAL_STEP: 1420200 | > loss_text_ce: 0.02536034770309925 (0.030673687949929213) | > loss_mel_ce: 0.6984601616859436 (nan) | > loss: 0.0028723038267344236 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13744067984463576) | > loader_time: 0.0046 (0.00469478687591439)  --> TIME: 2025-02-26 21:55:00 -- STEP: 16606/36938 -- GLOBAL_STEP: 1420250 | > loss_text_ce: 0.039758749306201935 (0.030676406350275742) | > loss_mel_ce: 2.8591363430023193 (nan) | > loss: 0.011503552086651325 (nan) | > current_lr: 5e-06 | > step_time: 0.1771 (0.13745383654647925) | > loader_time: 0.0049 (0.004694789500892899)  --> TIME: 2025-02-26 21:55:10 -- STEP: 16656/36938 -- GLOBAL_STEP: 1420300 | > loss_text_ce: 0.0327037014067173 (0.030676842815728952) | > loss_mel_ce: 2.7318854331970215 (nan) | > loss: 0.010970592498779297 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.13745829192465112) | > loader_time: 0.0046 (0.004694689161839302)  --> TIME: 2025-02-26 21:55:19 -- STEP: 16706/36938 -- GLOBAL_STEP: 1420350 | > loss_text_ce: 0.03500092402100563 (0.0306745415330348) | > loss_mel_ce: 2.6298084259033203 (nan) | > loss: 0.0105746416375041 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1374492122215596) | > loader_time: 0.0046 (0.004694673953107109)  --> TIME: 2025-02-26 21:55:28 -- STEP: 16756/36938 -- GLOBAL_STEP: 1420400 | > loss_text_ce: 0.033611539751291275 (0.030673572947619115) | > loss_mel_ce: 2.8868536949157715 (nan) | > loss: 0.011589148081839085 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13742430115064683) | > loader_time: 0.0046 (0.004694378839867497)  --> TIME: 2025-02-26 21:55:37 -- STEP: 16806/36938 -- GLOBAL_STEP: 1420450 | > loss_text_ce: 0.030049096792936325 (0.030673364100816634) | > loss_mel_ce: 2.507363796234131 (nan) | > loss: 0.010069099254906178 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.1374372448167035) | > loader_time: 0.0053 (0.004694659738587148)  --> TIME: 2025-02-26 21:55:47 -- STEP: 16856/36938 -- GLOBAL_STEP: 1420500 | > loss_text_ce: 0.030981501564383507 (0.030672691470552658) | > loss_mel_ce: 2.33778977394104 (nan) | > loss: 0.009399887174367905 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.13743557496324194) | > loader_time: 0.0056 (0.004694856579517572)  --> TIME: 2025-02-26 21:55:56 -- STEP: 16906/36938 -- GLOBAL_STEP: 1420550 | > loss_text_ce: 0.03082253597676754 (0.030671999951899252) | > loss_mel_ce: 2.3951542377471924 (nan) | > loss: 0.009626892395317554 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13744714864256627) | > loader_time: 0.0046 (0.004694936403251713)  --> TIME: 2025-02-26 21:56:06 -- STEP: 16956/36938 -- GLOBAL_STEP: 1420600 | > loss_text_ce: 0.02942187897861004 (0.030668560657674405) | > loss_mel_ce: 3.067077159881592 (nan) | > loss: 0.012287694960832596 (nan) | > current_lr: 5e-06 | > step_time: 0.1503 (0.13744562112039124) | > loader_time: 0.0046 (0.004695314862138084)  --> TIME: 2025-02-26 21:56:15 -- STEP: 17006/36938 -- GLOBAL_STEP: 1420650 | > loss_text_ce: 0.038210492581129074 (0.030664469479126148) | > loss_mel_ce: 2.130263566970825 (nan) | > loss: 0.008605055510997772 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1374549095762137) | > loader_time: 0.0045 (0.004695320751868032)  --> TIME: 2025-02-26 21:56:25 -- STEP: 17056/36938 -- GLOBAL_STEP: 1420700 | > loss_text_ce: 0.03812103345990181 (0.030665011192898238) | > loss_mel_ce: 2.1187634468078613 (nan) | > loss: 0.008559065870940685 (nan) | > current_lr: 5e-06 | > step_time: 0.1123 (0.13746424345037372) | > loader_time: 0.0045 (0.0046952679809217504)  --> TIME: 2025-02-26 21:56:34 -- STEP: 17106/36938 -- GLOBAL_STEP: 1420750 | > loss_text_ce: 0.025426698848605156 (0.030663521120140282) | > loss_mel_ce: 2.740548610687256 (nan) | > loss: 0.010976092889904976 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.1374649770856657) | > loader_time: 0.0045 (0.004695190179700583)  --> TIME: 2025-02-26 21:56:43 -- STEP: 17156/36938 -- GLOBAL_STEP: 1420800 | > loss_text_ce: 0.03726118057966232 (0.03066054902026006) | > loss_mel_ce: 2.7814252376556396 (nan) | > loss: 0.011185264214873314 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.13744520398947196) | > loader_time: 0.0046 (0.004694918453290342)  --> TIME: 2025-02-26 21:56:52 -- STEP: 17206/36938 -- GLOBAL_STEP: 1420850 | > loss_text_ce: 0.022881116718053818 (0.030660660888767607) | > loss_mel_ce: 0.9826496839523315 (nan) | > loss: 0.003990201745182276 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.1374370741259412) | > loader_time: 0.0045 (0.004694897255813797)  --> TIME: 2025-02-26 21:57:02 -- STEP: 17256/36938 -- GLOBAL_STEP: 1420900 | > loss_text_ce: 0.02503022365272045 (0.03066239141232836) | > loss_mel_ce: 2.044739246368408 (nan) | > loss: 0.00821337103843689 (nan) | > current_lr: 5e-06 | > step_time: 0.1506 (0.13743887271768238) | > loader_time: 0.0047 (0.0046949147570171604)  --> TIME: 2025-02-26 21:57:11 -- STEP: 17306/36938 -- GLOBAL_STEP: 1420950 | > loss_text_ce: 0.02602849155664444 (0.030662920801988267) | > loss_mel_ce: 0.0011943368008360267 (nan) | > loss: 0.0001080271031241864 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13744134376961756) | > loader_time: 0.0047 (0.0046947750758121526)  --> TIME: 2025-02-26 21:57:21 -- STEP: 17356/36938 -- GLOBAL_STEP: 1421000 | > loss_text_ce: 0.032503269612789154 (0.030660921197282004) | > loss_mel_ce: 3.1225099563598633 (nan) | > loss: 0.01251989509910345 (nan) | > current_lr: 5e-06 | > step_time: 0.1438 (0.13744299650521816) | > loader_time: 0.0047 (0.0046949387420896676)  --> TIME: 2025-02-26 21:57:30 -- STEP: 17406/36938 -- GLOBAL_STEP: 1421050 | > loss_text_ce: 0.034185122698545456 (0.030658667087890955) | > loss_mel_ce: 2.3368523120880127 (nan) | > loss: 0.009408879093825817 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13744144909247225) | > loader_time: 0.0047 (0.004694912428967091)  --> TIME: 2025-02-26 21:57:39 -- STEP: 17456/36938 -- GLOBAL_STEP: 1421100 | > loss_text_ce: 0.030670050531625748 (0.030659265735595964) | > loss_mel_ce: 1.7915133237838745 (nan) | > loss: 0.007230886723846197 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13744958632861967) | > loader_time: 0.0046 (0.004694986067486919)  --> TIME: 2025-02-26 21:57:49 -- STEP: 17506/36938 -- GLOBAL_STEP: 1421150 | > loss_text_ce: 0.031537964940071106 (0.030657041121699202) | > loss_mel_ce: 2.303149700164795 (nan) | > loss: 0.009264633990824223 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13744768025356613) | > loader_time: 0.0046 (0.004694852300007821)  --> TIME: 2025-02-26 21:57:58 -- STEP: 17556/36938 -- GLOBAL_STEP: 1421200 | > loss_text_ce: 0.028354043141007423 (0.030655947723181906) | > loss_mel_ce: 2.646121025085449 (nan) | > loss: 0.01061299629509449 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13744855523245114) | > loader_time: 0.0045 (0.004694642687093724)  --> TIME: 2025-02-26 21:58:07 -- STEP: 17606/36938 -- GLOBAL_STEP: 1421250 | > loss_text_ce: 0.029599811881780624 (0.030657130695758223) | > loss_mel_ce: 2.0045979022979736 (nan) | > loss: 0.00807221420109272 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13743987965553872) | > loader_time: 0.0046 (0.0046944604954041605)  --> TIME: 2025-02-26 21:58:16 -- STEP: 17656/36938 -- GLOBAL_STEP: 1421300 | > loss_text_ce: 0.0341828353703022 (0.030658802935467702) | > loss_mel_ce: 3.213308334350586 (nan) | > loss: 0.01288687065243721 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13742537456883067) | > loader_time: 0.0045 (0.0046941783290995855)  --> TIME: 2025-02-26 21:58:26 -- STEP: 17706/36938 -- GLOBAL_STEP: 1421350 | > loss_text_ce: 0.03329303115606308 (0.03065871504287127) | > loss_mel_ce: 0.13600540161132812 (nan) | > loss: 0.000671819259878248 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13743486881633002) | > loader_time: 0.0046 (0.004694158460250133)  --> TIME: 2025-02-26 21:58:35 -- STEP: 17756/36938 -- GLOBAL_STEP: 1421400 | > loss_text_ce: 0.02901872619986534 (0.030661463657945635) | > loss_mel_ce: 2.7975330352783203 (nan) | > loss: 0.011216475628316402 (nan) | > current_lr: 5e-06 | > step_time: 0.189 (0.13743484507651044) | > loader_time: 0.0052 (0.004693936579236551)  --> TIME: 2025-02-26 21:58:45 -- STEP: 17806/36938 -- GLOBAL_STEP: 1421450 | > loss_text_ce: 0.03230496495962143 (0.030663977615491538) | > loss_mel_ce: 2.1725409030914307 (nan) | > loss: 0.008749389089643955 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13742562201563113) | > loader_time: 0.0046 (0.0046937312086846985)  --> TIME: 2025-02-26 21:58:54 -- STEP: 17856/36938 -- GLOBAL_STEP: 1421500 | > loss_text_ce: 0.03135276213288307 (0.030661548430338553) | > loss_mel_ce: 2.5620882511138916 (nan) | > loss: 0.010291432961821556 (nan) | > current_lr: 5e-06 | > step_time: 0.161 (0.13742004617995132) | > loader_time: 0.0047 (0.004693770988119944)  --> TIME: 2025-02-26 21:59:03 -- STEP: 17906/36938 -- GLOBAL_STEP: 1421550 | > loss_text_ce: 0.032343924045562744 (0.0306588332208547) | > loss_mel_ce: 2.567417860031128 (nan) | > loss: 0.0103165153414011 (nan) | > current_lr: 5e-06 | > step_time: 0.1679 (0.13741154433974315) | > loader_time: 0.0049 (0.004693610820239766)  --> TIME: 2025-02-26 21:59:12 -- STEP: 17956/36938 -- GLOBAL_STEP: 1421600 | > loss_text_ce: 0.03378225117921829 (0.030661716729818107) | > loss_mel_ce: 2.352020263671875 (nan) | > loss: 0.009467470459640026 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.13741569334214002) | > loader_time: 0.0045 (0.0046936834169193364)  --> TIME: 2025-02-26 21:59:22 -- STEP: 18006/36938 -- GLOBAL_STEP: 1421650 | > loss_text_ce: 0.0317302905023098 (0.030662578640650088) | > loss_mel_ce: 3.6644978523254395 (nan) | > loss: 0.01466757245361805 (nan) | > current_lr: 5e-06 | > step_time: 0.1812 (0.13741884142322489) | > loader_time: 0.0048 (0.0046935476198019235)  --> TIME: 2025-02-26 21:59:31 -- STEP: 18056/36938 -- GLOBAL_STEP: 1421700 | > loss_text_ce: 0.029316503554582596 (0.030660989193948704) | > loss_mel_ce: 1.8123711347579956 (nan) | > loss: 0.007308284752070904 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13740645675834404) | > loader_time: 0.0045 (0.004693337917116584)  --> TIME: 2025-02-26 21:59:40 -- STEP: 18106/36938 -- GLOBAL_STEP: 1421750 | > loss_text_ce: 0.029918216168880463 (0.030660089177045453) | > loss_mel_ce: 0.046630859375 (nan) | > loss: 0.000303766195429489 (nan) | > current_lr: 5e-06 | > step_time: 0.1714 (0.1374091586394546) | > loader_time: 0.0049 (0.004693557462099149)  --> TIME: 2025-02-26 21:59:50 -- STEP: 18156/36938 -- GLOBAL_STEP: 1421800 | > loss_text_ce: 0.02275501750409603 (0.03066067546543997) | > loss_mel_ce: 2.8281214237213135 (nan) | > loss: 0.011313001625239849 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.1374081949563653) | > loader_time: 0.0046 (0.004693410067758912)  --> TIME: 2025-02-26 21:59:59 -- STEP: 18206/36938 -- GLOBAL_STEP: 1421850 | > loss_text_ce: 0.023547044023871422 (0.030655238400670238) | > loss_mel_ce: 1.2122771739959717 (nan) | > loss: 0.004904064815491438 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13740533348702735) | > loader_time: 0.0047 (0.004693260785316222)  --> TIME: 2025-02-26 22:00:08 -- STEP: 18256/36938 -- GLOBAL_STEP: 1421900 | > loss_text_ce: 0.03828983008861542 (0.030654189404867242) | > loss_mel_ce: 2.0813474655151367 (nan) | > loss: 0.008411259390413761 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.1374144719605052) | > loader_time: 0.0044 (0.004693331149562603)  --> TIME: 2025-02-26 22:00:18 -- STEP: 18306/36938 -- GLOBAL_STEP: 1421950 | > loss_text_ce: 0.027942605316638947 (0.030653125512536093) | > loss_mel_ce: 2.3283448219299316 (nan) | > loss: 0.009350347332656384 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.1374163516529203) | > loader_time: 0.0049 (0.004693337806414895)  --> TIME: 2025-02-26 22:00:27 -- STEP: 18356/36938 -- GLOBAL_STEP: 1422000 | > loss_text_ce: 0.027523256838321686 (0.030655283903595336) | > loss_mel_ce: 3.2468416690826416 (nan) | > loss: 0.01299351267516613 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13741481845792816) | > loader_time: 0.0045 (0.004693209813243261)  --> TIME: 2025-02-26 22:00:36 -- STEP: 18406/36938 -- GLOBAL_STEP: 1422050 | > loss_text_ce: 0.03124651312828064 (0.030654890441782937) | > loss_mel_ce: 2.4386773109436035 (nan) | > loss: 0.009801285341382027 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.137410990468809) | > loader_time: 0.0046 (0.004693552552131672)  --> TIME: 2025-02-26 22:00:46 -- STEP: 18456/36938 -- GLOBAL_STEP: 1422100 | > loss_text_ce: 0.01867753081023693 (0.030654818411890463) | > loss_mel_ce: 0.24736015498638153 (nan) | > loss: 0.001055705128237605 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13740982938804788) | > loader_time: 0.0045 (0.0046934021800743245)  --> TIME: 2025-02-26 22:00:55 -- STEP: 18506/36938 -- GLOBAL_STEP: 1422150 | > loss_text_ce: 0.033660560846328735 (0.030656763316294886) | > loss_mel_ce: 2.856724262237549 (nan) | > loss: 0.011469782330095768 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13741607978049322) | > loader_time: 0.0047 (0.004693721817929331)  --> TIME: 2025-02-26 22:01:05 -- STEP: 18556/36938 -- GLOBAL_STEP: 1422200 | > loss_text_ce: 0.03799829259514809 (0.030659000517638445) | > loss_mel_ce: 0.185835599899292 (nan) | > loss: 0.0008882297552190721 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13741357029961812) | > loader_time: 0.0054 (0.0046939798330638514)  --> TIME: 2025-02-26 22:01:14 -- STEP: 18606/36938 -- GLOBAL_STEP: 1422250 | > loss_text_ce: 0.03320175036787987 (0.03066143496356958) | > loss_mel_ce: 0.688934326171875 (nan) | > loss: 0.0028656194917857647 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13742662483321014) | > loader_time: 0.0052 (0.004694355619500835)  --> TIME: 2025-02-26 22:01:24 -- STEP: 18656/36938 -- GLOBAL_STEP: 1422300 | > loss_text_ce: 0.028474101796746254 (0.030660951706634937) | > loss_mel_ce: 1.427832007408142 (nan) | > loss: 0.005778993014246225 (nan) | > current_lr: 5e-06 | > step_time: 0.1606 (0.13743087303014695) | > loader_time: 0.0049 (0.004694443930892526)  --> TIME: 2025-02-26 22:01:33 -- STEP: 18706/36938 -- GLOBAL_STEP: 1422350 | > loss_text_ce: 0.03524772450327873 (0.030659638942515756) | > loss_mel_ce: 1.6562391519546509 (nan) | > loss: 0.00671224994584918 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13743855980491954) | > loader_time: 0.0045 (0.004694782526470362)  --> TIME: 2025-02-26 22:01:43 -- STEP: 18756/36938 -- GLOBAL_STEP: 1422400 | > loss_text_ce: 0.027642983943223953 (0.030658445311514605) | > loss_mel_ce: 2.5310444831848145 (nan) | > loss: 0.010153521783649921 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13743755855039527) | > loader_time: 0.0052 (0.0046949561253923884)  --> TIME: 2025-02-26 22:01:52 -- STEP: 18806/36938 -- GLOBAL_STEP: 1422450 | > loss_text_ce: 0.02942792885005474 (0.030660060917111587) | > loss_mel_ce: 2.829012870788574 (nan) | > loss: 0.011343020014464855 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.1374328440457563) | > loader_time: 0.0046 (0.004694993605223714)  --> TIME: 2025-02-26 22:02:01 -- STEP: 18856/36938 -- GLOBAL_STEP: 1422500 | > loss_text_ce: 0.036979012191295624 (0.03066221640785994) | > loss_mel_ce: 2.528416872024536 (nan) | > loss: 0.01018014270812273 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13742114719213538) | > loader_time: 0.0046 (0.004695007734800402)  --> TIME: 2025-02-26 22:02:10 -- STEP: 18906/36938 -- GLOBAL_STEP: 1422550 | > loss_text_ce: 0.026410609483718872 (0.030659928755644755) | > loss_mel_ce: 2.8952479362487793 (nan) | > loss: 0.011593883857131004 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13741550395870228) | > loader_time: 0.0047 (0.004694884988374567)  --> TIME: 2025-02-26 22:02:20 -- STEP: 18956/36938 -- GLOBAL_STEP: 1422600 | > loss_text_ce: 0.02174479514360428 (0.03065683592469429) | > loss_mel_ce: 1.4837685823440552 (nan) | > loss: 0.005974260158836842 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13742382592052002) | > loader_time: 0.0044 (0.004694902260460946)  --> TIME: 2025-02-26 22:02:30 -- STEP: 19006/36938 -- GLOBAL_STEP: 1422650 | > loss_text_ce: 0.019019650295376778 (0.03065680484984243) | > loss_mel_ce: 2.8927292823791504 (nan) | > loss: 0.01155455969274044 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1374354321718531) | > loader_time: 0.0046 (0.004695095865899313)  --> TIME: 2025-02-26 22:02:39 -- STEP: 19056/36938 -- GLOBAL_STEP: 1422700 | > loss_text_ce: 0.03924287110567093 (0.03065460357891954) | > loss_mel_ce: 1.9311257600784302 (nan) | > loss: 0.007818923331797123 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13743789231296638) | > loader_time: 0.0045 (0.004695000729072385)  --> TIME: 2025-02-26 22:02:48 -- STEP: 19106/36938 -- GLOBAL_STEP: 1422750 | > loss_text_ce: 0.02155224233865738 (0.030651337161429537) | > loss_mel_ce: 2.6075706481933594 (nan) | > loss: 0.01043302845209837 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.1374315457432393) | > loader_time: 0.0048 (0.004695575099758569)  --> TIME: 2025-02-26 22:02:57 -- STEP: 19156/36938 -- GLOBAL_STEP: 1422800 | > loss_text_ce: 0.03196077048778534 (0.030649674496382805) | > loss_mel_ce: 2.3712656497955322 (nan) | > loss: 0.009536612778902054 (nan) | > current_lr: 5e-06 | > step_time: 0.1262 (0.13742793842257015) | > loader_time: 0.0047 (0.004695903000340708)  --> TIME: 2025-02-26 22:03:07 -- STEP: 19206/36938 -- GLOBAL_STEP: 1422850 | > loss_text_ce: 0.03427063301205635 (0.030648911607567247) | > loss_mel_ce: 2.0165510177612305 (nan) | > loss: 0.008138181641697884 (nan) | > current_lr: 5e-06 | > step_time: 0.1533 (0.1374270042305017) | > loader_time: 0.0046 (0.004695991284025701)  --> TIME: 2025-02-26 22:03:16 -- STEP: 19256/36938 -- GLOBAL_STEP: 1422900 | > loss_text_ce: 0.03343438729643822 (0.030647792337526347) | > loss_mel_ce: 2.936957597732544 (nan) | > loss: 0.011787270195782185 (nan) | > current_lr: 5e-06 | > step_time: 0.1699 (0.13742661599950362) | > loader_time: 0.0047 (0.004696027899263889)  --> TIME: 2025-02-26 22:03:25 -- STEP: 19306/36938 -- GLOBAL_STEP: 1422950 | > loss_text_ce: 0.0347260944545269 (0.030648604665513276) | > loss_mel_ce: 1.4007364511489868 (nan) | > loss: 0.0056962803937494755 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13742361344859294) | > loader_time: 0.0045 (0.004696140187547115)  --> TIME: 2025-02-26 22:03:35 -- STEP: 19356/36938 -- GLOBAL_STEP: 1423000 | > loss_text_ce: 0.028089161962270737 (0.030649699979874757) | > loss_mel_ce: 3.041487693786621 (nan) | > loss: 0.012180861085653305 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.1374390361771687) | > loader_time: 0.0046 (0.004696056182090331)  --> TIME: 2025-02-26 22:03:44 -- STEP: 19406/36938 -- GLOBAL_STEP: 1423050 | > loss_text_ce: 0.029948847368359566 (0.030650348544590533) | > loss_mel_ce: 2.6578409671783447 (nan) | > loss: 0.010665833950042725 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.137438828338485) | > loader_time: 0.0045 (0.004695906401245672)  --> TIME: 2025-02-26 22:03:54 -- STEP: 19456/36938 -- GLOBAL_STEP: 1423100 | > loss_text_ce: 0.03372248262166977 (0.030648578704585823) | > loss_mel_ce: 0.4591623842716217 (nan) | > loss: 0.00195589242503047 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.1374398811328164) | > loader_time: 0.0047 (0.004695681708031562)  --> TIME: 2025-02-26 22:04:03 -- STEP: 19506/36938 -- GLOBAL_STEP: 1423150 | > loss_text_ce: 0.029479598626494408 (0.030649458862187787) | > loss_mel_ce: 1.7564932107925415 (nan) | > loss: 0.007087194360792637 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13744469350464444) | > loader_time: 0.0045 (0.004695968531124833)  --> TIME: 2025-02-26 22:04:13 -- STEP: 19556/36938 -- GLOBAL_STEP: 1423200 | > loss_text_ce: 0.02668020687997341 (0.030649482167674904) | > loss_mel_ce: 1.6747430562973022 (nan) | > loss: 0.00675168028101325 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13745371574613185) | > loader_time: 0.0047 (0.004696158464029519)  --> TIME: 2025-02-26 22:04:22 -- STEP: 19606/36938 -- GLOBAL_STEP: 1423250 | > loss_text_ce: 0.03623192384839058 (0.030651536358109598) | > loss_mel_ce: 1.331399917602539 (nan) | > loss: 0.0054271104745566845 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.13745427325004506) | > loader_time: 0.0046 (0.00469633371115191)  --> TIME: 2025-02-26 22:04:32 -- STEP: 19656/36938 -- GLOBAL_STEP: 1423300 | > loss_text_ce: 0.034981224685907364 (0.030652011647049303) | > loss_mel_ce: 2.9910595417022705 (nan) | > loss: 0.012008098885416985 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.1374524754142671) | > loader_time: 0.0045 (0.004696242077640738)  --> TIME: 2025-02-26 22:04:41 -- STEP: 19706/36938 -- GLOBAL_STEP: 1423350 | > loss_text_ce: 0.025785865262150764 (0.030651829519998078) | > loss_mel_ce: 2.702606201171875 (nan) | > loss: 0.010826953686773777 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1374449110098869) | > loader_time: 0.0045 (0.004696236907268273)  --> TIME: 2025-02-26 22:04:50 -- STEP: 19756/36938 -- GLOBAL_STEP: 1423400 | > loss_text_ce: 0.032587967813014984 (0.030656337226012405) | > loss_mel_ce: 3.0034568309783936 (nan) | > loss: 0.012047797441482544 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13742812586400188) | > loader_time: 0.0047 (0.0046961892590056285)  --> TIME: 2025-02-26 22:04:59 -- STEP: 19806/36938 -- GLOBAL_STEP: 1423450 | > loss_text_ce: 0.026433587074279785 (0.03065457628807312) | > loss_mel_ce: 3.1484880447387695 (nan) | > loss: 0.012598895467817783 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13742933017634693) | > loader_time: 0.0045 (0.0046961090967508915)  --> TIME: 2025-02-26 22:05:09 -- STEP: 19856/36938 -- GLOBAL_STEP: 1423500 | > loss_text_ce: 0.03004317171871662 (0.03065600227666355) | > loss_mel_ce: 2.2994775772094727 (nan) | > loss: 0.009244129993021488 (nan) | > current_lr: 5e-06 | > step_time: 0.1695 (0.13743907131560756) | > loader_time: 0.0048 (0.00469603176370538)  --> TIME: 2025-02-26 22:05:18 -- STEP: 19906/36938 -- GLOBAL_STEP: 1423550 | > loss_text_ce: 0.02790486440062523 (0.03065695120692145) | > loss_mel_ce: 3.076443910598755 (nan) | > loss: 0.01231884490698576 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.1374521637087444) | > loader_time: 0.0052 (0.0046963680452837085)  --> TIME: 2025-02-26 22:05:28 -- STEP: 19956/36938 -- GLOBAL_STEP: 1423600 | > loss_text_ce: 0.025468315929174423 (0.030659052774796032) | > loss_mel_ce: 2.523510694503784 (nan) | > loss: 0.010114996694028378 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13744602935255304) | > loader_time: 0.0045 (0.004696489073370461)  --> TIME: 2025-02-26 22:05:37 -- STEP: 20006/36938 -- GLOBAL_STEP: 1423650 | > loss_text_ce: 0.038465715944767 (0.030659111065346512) | > loss_mel_ce: 1.4365955591201782 (nan) | > loss: 0.005853418260812759 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1374485729742456) | > loader_time: 0.0048 (0.004696501429934473)  --> TIME: 2025-02-26 22:05:46 -- STEP: 20056/36938 -- GLOBAL_STEP: 1423700 | > loss_text_ce: 0.03686147928237915 (0.030661878370577576) | > loss_mel_ce: 2.2135303020477295 (nan) | > loss: 0.008930126205086708 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13744116628317332) | > loader_time: 0.0052 (0.004696395656812403)  --> TIME: 2025-02-26 22:05:56 -- STEP: 20106/36938 -- GLOBAL_STEP: 1423750 | > loss_text_ce: 0.035703156143426895 (0.03065925627710341) | > loss_mel_ce: 2.2031655311584473 (nan) | > loss: 0.008884400129318237 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13744320221283) | > loader_time: 0.0053 (0.004696336739290689)  --> TIME: 2025-02-26 22:06:05 -- STEP: 20156/36938 -- GLOBAL_STEP: 1423800 | > loss_text_ce: 0.023490888997912407 (0.030657183456167733) | > loss_mel_ce: 0.5572656989097595 (nan) | > loss: 0.002304589841514826 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13743380156271104) | > loader_time: 0.0045 (0.004696642862801785)  --> TIME: 2025-02-26 22:06:14 -- STEP: 20206/36938 -- GLOBAL_STEP: 1423850 | > loss_text_ce: 0.03286585584282875 (0.030656893277783433) | > loss_mel_ce: 2.6460041999816895 (nan) | > loss: 0.010630437172949314 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13742993722324856) | > loader_time: 0.0046 (0.004696376427770604)  --> TIME: 2025-02-26 22:06:24 -- STEP: 20256/36938 -- GLOBAL_STEP: 1423900 | > loss_text_ce: 0.03071318380534649 (0.030658116717237102) | > loss_mel_ce: 2.7423095703125 (nan) | > loss: 0.01100405864417553 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13743419707480514) | > loader_time: 0.0046 (0.0046964251858551435)  --> TIME: 2025-02-26 22:06:33 -- STEP: 20306/36938 -- GLOBAL_STEP: 1423950 | > loss_text_ce: 0.03420533239841461 (0.03065942770363572) | > loss_mel_ce: 2.7769503593444824 (nan) | > loss: 0.011155380867421627 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13742392309473017) | > loader_time: 0.0053 (0.004696388109838921)  --> TIME: 2025-02-26 22:06:42 -- STEP: 20356/36938 -- GLOBAL_STEP: 1424000 | > loss_text_ce: 0.026518480852246284 (0.03065852068067143) | > loss_mel_ce: 0.016357421875 (nan) | > loss: 0.00017014247714541852 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.1374203642498271) | > loader_time: 0.0047 (0.004696453617888266)  --> TIME: 2025-02-26 22:06:51 -- STEP: 20406/36938 -- GLOBAL_STEP: 1424050 | > loss_text_ce: 0.028901508077979088 (0.030659153722165468) | > loss_mel_ce: 3.39054274559021 (nan) | > loss: 0.013569224625825882 (nan) | > current_lr: 5e-06 | > step_time: 0.184 (0.1374126001279697) | > loader_time: 0.0048 (0.004696339657992971)  --> TIME: 2025-02-26 22:07:00 -- STEP: 20456/36938 -- GLOBAL_STEP: 1424100 | > loss_text_ce: 0.034173090010881424 (0.030660221425949912) | > loss_mel_ce: 2.4313151836395264 (nan) | > loss: 0.009783684276044369 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.1374043190344072) | > loader_time: 0.0047 (0.0046964036471963955)  --> TIME: 2025-02-26 22:07:10 -- STEP: 20506/36938 -- GLOBAL_STEP: 1424150 | > loss_text_ce: 0.0336073637008667 (0.030658829247582255) | > loss_mel_ce: 3.4116525650024414 (nan) | > loss: 0.01367166731506586 (nan) | > current_lr: 5e-06 | > step_time: 0.1687 (0.1374002991101218) | > loader_time: 0.0055 (0.004696512436222632)  --> TIME: 2025-02-26 22:07:19 -- STEP: 20556/36938 -- GLOBAL_STEP: 1424200 | > loss_text_ce: 0.026956213638186455 (0.03065723913707152) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13740295060392094) | > loader_time: 0.0045 (0.00469675123703157)  --> TIME: 2025-02-26 22:07:29 -- STEP: 20606/36938 -- GLOBAL_STEP: 1424250 | > loss_text_ce: 0.02685142494738102 (0.03065413138778552) | > loss_mel_ce: 2.773623466491699 (nan) | > loss: 0.011112996377050877 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.137417144929507) | > loader_time: 0.0044 (0.004696833500755205)  --> TIME: 2025-02-26 22:07:38 -- STEP: 20656/36938 -- GLOBAL_STEP: 1424300 | > loss_text_ce: 0.030006496235728264 (0.030655698696394673) | > loss_mel_ce: 1.77642822265625 (nan) | > loss: 0.007168392185121775 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.137420714847991) | > loader_time: 0.0046 (0.0046969190366871975)  --> TIME: 2025-02-26 22:07:48 -- STEP: 20706/36938 -- GLOBAL_STEP: 1424350 | > loss_text_ce: 0.03525974601507187 (0.03065503147471379) | > loss_mel_ce: 2.8069581985473633 (nan) | > loss: 0.011278643272817135 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.1374271794999605) | > loader_time: 0.0046 (0.004696733154997686)  --> TIME: 2025-02-26 22:07:57 -- STEP: 20756/36938 -- GLOBAL_STEP: 1424400 | > loss_text_ce: 0.039868805557489395 (0.03065916111046748) | > loss_mel_ce: 3.0025644302368164 (nan) | > loss: 0.012073148973286152 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.13742548582250091) | > loader_time: 0.0047 (0.004696856323384799)  --> TIME: 2025-02-26 22:08:07 -- STEP: 20806/36938 -- GLOBAL_STEP: 1424450 | > loss_text_ce: 0.032207440584897995 (0.030659215415237164) | > loss_mel_ce: 2.788595199584961 (nan) | > loss: 0.011193661950528622 (nan) | > current_lr: 5e-06 | > step_time: 0.1719 (0.13743357826606145) | > loader_time: 0.0048 (0.004696888212262454)  --> TIME: 2025-02-26 22:08:16 -- STEP: 20856/36938 -- GLOBAL_STEP: 1424500 | > loss_text_ce: 0.025362610816955566 (0.03066002224530653) | > loss_mel_ce: 1.77130126953125 (nan) | > loss: 0.007129618898034096 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13741656433680843) | > loader_time: 0.0046 (0.004696771073807767)  --> TIME: 2025-02-26 22:08:25 -- STEP: 20906/36938 -- GLOBAL_STEP: 1424550 | > loss_text_ce: 0.029419193044304848 (0.03065916009192153) | > loss_mel_ce: 2.878394842147827 (nan) | > loss: 0.011538945138454437 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.13742359819497033) | > loader_time: 0.0048 (0.004696942397558901)  --> TIME: 2025-02-26 22:08:34 -- STEP: 20956/36938 -- GLOBAL_STEP: 1424600 | > loss_text_ce: 0.030178343877196312 (0.030660616940147482) | > loss_mel_ce: 2.9458601474761963 (nan) | > loss: 0.011809676885604858 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13741764792005212) | > loader_time: 0.0046 (0.00469736420122987)  --> TIME: 2025-02-26 22:08:44 -- STEP: 21006/36938 -- GLOBAL_STEP: 1424650 | > loss_text_ce: 0.027616454288363457 (0.03066113066984926) | > loss_mel_ce: 2.9663093090057373 (nan) | > loss: 0.011880658566951752 (nan) | > current_lr: 5e-06 | > step_time: 0.1697 (0.13741983970573776) | > loader_time: 0.0048 (0.004697576881669452)  --> TIME: 2025-02-26 22:08:53 -- STEP: 21056/36938 -- GLOBAL_STEP: 1424700 | > loss_text_ce: 0.03090711124241352 (0.030660799335187488) | > loss_mel_ce: 2.776761531829834 (nan) | > loss: 0.01114154327660799 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.1374138229523745) | > loader_time: 0.0052 (0.004697591281498803)  --> TIME: 2025-02-26 22:09:03 -- STEP: 21106/36938 -- GLOBAL_STEP: 1424750 | > loss_text_ce: 0.030168546363711357 (0.030658436536713577) | > loss_mel_ce: 3.0353219509124756 (nan) | > loss: 0.012164645828306675 (nan) | > current_lr: 5e-06 | > step_time: 0.1501 (0.13743194431916522) | > loader_time: 0.0046 (0.004697695960481603)  --> TIME: 2025-02-26 22:09:12 -- STEP: 21156/36938 -- GLOBAL_STEP: 1424800 | > loss_text_ce: 0.02684427611529827 (0.0306601005735645) | > loss_mel_ce: 2.116767168045044 (nan) | > loss: 0.008506394922733307 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.137420113448678) | > loader_time: 0.0045 (0.0046974540230095806)  --> TIME: 2025-02-26 22:09:21 -- STEP: 21206/36938 -- GLOBAL_STEP: 1424850 | > loss_text_ce: 0.019632959738373756 (0.030658047804059094) | > loss_mel_ce: 0.2644578218460083 (nan) | > loss: 0.0011273444397374988 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13742119244198878) | > loader_time: 0.0047 (0.004697495886124386)  --> TIME: 2025-02-26 22:09:31 -- STEP: 21256/36938 -- GLOBAL_STEP: 1424900 | > loss_text_ce: 0.02600940130650997 (0.030657210189926783) | > loss_mel_ce: 1.8806805610656738 (nan) | > loss: 0.007566230371594429 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13742671813290042) | > loader_time: 0.0045 (0.004697436345534326)  --> TIME: 2025-02-26 22:09:41 -- STEP: 21306/36938 -- GLOBAL_STEP: 1424950 | > loss_text_ce: 0.03229273110628128 (0.03065982030762405) | > loss_mel_ce: 3.0765011310577393 (nan) | > loss: 0.012336485087871552 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13743623947074438) | > loader_time: 0.0045 (0.0046976691712171835)  --> TIME: 2025-02-26 22:09:50 -- STEP: 21356/36938 -- GLOBAL_STEP: 1425000 | > loss_text_ce: 0.02552984096109867 (0.03065943678601231) | > loss_mel_ce: 2.931826591491699 (nan) | > loss: 0.011735541746020317 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1374200968332429) | > loader_time: 0.0047 (0.0046976693093184795)  --> TIME: 2025-02-26 22:09:59 -- STEP: 21406/36938 -- GLOBAL_STEP: 1425050 | > loss_text_ce: 0.03169791400432587 (0.030660583796957675) | > loss_mel_ce: 1.5162979364395142 (nan) | > loss: 0.006142840720713139 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13742249281308622) | > loader_time: 0.0051 (0.004697938160529617)  --> TIME: 2025-02-26 22:10:08 -- STEP: 21456/36938 -- GLOBAL_STEP: 1425100 | > loss_text_ce: 0.03333684056997299 (0.030658253115746124) | > loss_mel_ce: 2.411806583404541 (nan) | > loss: 0.009702950716018677 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.1374273002814394) | > loader_time: 0.0046 (0.004697831551767279)  --> TIME: 2025-02-26 22:10:18 -- STEP: 21506/36938 -- GLOBAL_STEP: 1425150 | > loss_text_ce: 0.026079948991537094 (0.030662277751504975) | > loss_mel_ce: 1.7012455463409424 (nan) | > loss: 0.006854466628283262 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13741548879749896) | > loader_time: 0.0044 (0.004697538304637089)  --> TIME: 2025-02-26 22:10:27 -- STEP: 21556/36938 -- GLOBAL_STEP: 1425200 | > loss_text_ce: 0.037406519055366516 (0.030663897605142983) | > loss_mel_ce: 2.1284546852111816 (nan) | > loss: 0.008594687096774578 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13741739791893837) | > loader_time: 0.0045 (0.004697546918663362)  --> TIME: 2025-02-26 22:10:36 -- STEP: 21606/36938 -- GLOBAL_STEP: 1425250 | > loss_text_ce: 0.021581172943115234 (0.030663944860807552) | > loss_mel_ce: 0.375 (nan) | > loss: 0.0015737349167466164 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13742304255451915) | > loader_time: 0.0051 (0.004697688241849306)  --> TIME: 2025-02-26 22:10:46 -- STEP: 21656/36938 -- GLOBAL_STEP: 1425300 | > loss_text_ce: 0.03322445973753929 (0.03066495121779653) | > loss_mel_ce: 1.913926124572754 (nan) | > loss: 0.007726788520812988 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13742576787777414) | > loader_time: 0.0045 (0.0046978501825258425)  --> TIME: 2025-02-26 22:10:55 -- STEP: 21706/36938 -- GLOBAL_STEP: 1425350 | > loss_text_ce: 0.02349940873682499 (0.030665369294761435) | > loss_mel_ce: 2.7840311527252197 (nan) | > loss: 0.011140995658934116 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13742373008986378) | > loader_time: 0.0045 (0.004697914004029275)  --> TIME: 2025-02-26 22:11:05 -- STEP: 21756/36938 -- GLOBAL_STEP: 1425400 | > loss_text_ce: 0.0313328318297863 (0.030665245356856188) | > loss_mel_ce: 2.940422296524048 (nan) | > loss: 0.011792679317295551 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.1374250259288073) | > loader_time: 0.0046 (0.004697929226008576)  --> TIME: 2025-02-26 22:11:14 -- STEP: 21806/36938 -- GLOBAL_STEP: 1425450 | > loss_text_ce: 0.025594891980290413 (0.030661738466777753) | > loss_mel_ce: 2.142813205718994 (nan) | > loss: 0.00860479474067688 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13741477743087757) | > loader_time: 0.0046 (0.004697766674009884)  --> TIME: 2025-02-26 22:11:23 -- STEP: 21856/36938 -- GLOBAL_STEP: 1425500 | > loss_text_ce: 0.021049894392490387 (0.030664477703473633) | > loss_mel_ce: 2.2956271171569824 (nan) | > loss: 0.009193163365125656 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13741516268445303) | > loader_time: 0.0045 (0.004697724238670738)  --> TIME: 2025-02-26 22:11:32 -- STEP: 21906/36938 -- GLOBAL_STEP: 1425550 | > loss_text_ce: 0.027677956968545914 (0.030665309707290894) | > loss_mel_ce: 2.7706878185272217 (nan) | > loss: 0.011104626581072807 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.1374176221384496) | > loader_time: 0.0046 (0.004697448965905167)  --> TIME: 2025-02-26 22:11:42 -- STEP: 21956/36938 -- GLOBAL_STEP: 1425600 | > loss_text_ce: 0.027345428243279457 (0.03066560988351184) | > loss_mel_ce: 1.6819514036178589 (nan) | > loss: 0.00678292429074645 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13741189954840716) | > loader_time: 0.0053 (0.0046973318040729045)  --> TIME: 2025-02-26 22:11:51 -- STEP: 22006/36938 -- GLOBAL_STEP: 1425650 | > loss_text_ce: 0.035703010857105255 (0.03066706408854545) | > loss_mel_ce: 2.677180051803589 (nan) | > loss: 0.010765409097075462 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.1374013922403672) | > loader_time: 0.0045 (0.004697164156150225)  --> TIME: 2025-02-26 22:12:00 -- STEP: 22056/36938 -- GLOBAL_STEP: 1425700 | > loss_text_ce: 0.033284418284893036 (0.030668006046736123) | > loss_mel_ce: 3.0295464992523193 (nan) | > loss: 0.012154092080891132 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13740545737168286) | > loader_time: 0.0046 (0.004697339211333499)  --> TIME: 2025-02-26 22:12:10 -- STEP: 22106/36938 -- GLOBAL_STEP: 1425750 | > loss_text_ce: 0.03418157249689102 (0.03066721893787586) | > loss_mel_ce: 2.7573604583740234 (nan) | > loss: 0.011077548377215862 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.1374164605091073) | > loader_time: 0.0045 (0.004697572437549687)  --> TIME: 2025-02-26 22:12:19 -- STEP: 22156/36938 -- GLOBAL_STEP: 1425800 | > loss_text_ce: 0.03873514011502266 (0.030666751132900207) | > loss_mel_ce: 2.9432291984558105 (nan) | > loss: 0.0118331927806139 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13742602652732078) | > loader_time: 0.0045 (0.00469782062333548)  --> TIME: 2025-02-26 22:12:29 -- STEP: 22206/36938 -- GLOBAL_STEP: 1425850 | > loss_text_ce: 0.031979337334632874 (0.03066404132304745) | > loss_mel_ce: 1.3060181140899658 (nan) | > loss: 0.00530951377004385 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.1374214357656798) | > loader_time: 0.0046 (0.004697728702466469)  --> TIME: 2025-02-26 22:12:38 -- STEP: 22256/36938 -- GLOBAL_STEP: 1425900 | > loss_text_ce: 0.021564330905675888 (0.0306655819427835) | > loss_mel_ce: 2.0971243381500244 (nan) | > loss: 0.008407494984567165 (nan) | > current_lr: 5e-06 | > step_time: 0.1073 (0.137417560168773) | > loader_time: 0.0046 (0.004697677570225758)  --> TIME: 2025-02-26 22:12:47 -- STEP: 22306/36938 -- GLOBAL_STEP: 1425950 | > loss_text_ce: 0.03429674729704857 (0.030665113806745765) | > loss_mel_ce: 2.704467296600342 (nan) | > loss: 0.010868111625313759 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13740631543840745) | > loader_time: 0.0047 (0.004697844072124883)  --> TIME: 2025-02-26 22:12:56 -- STEP: 22356/36938 -- GLOBAL_STEP: 1426000 | > loss_text_ce: 0.038425546139478683 (0.03066503959690003) | > loss_mel_ce: 0.2764231264591217 (nan) | > loss: 0.0012493995018303394 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1374077739666946) | > loader_time: 0.0045 (0.004697817257935577)  --> TIME: 2025-02-26 22:13:06 -- STEP: 22406/36938 -- GLOBAL_STEP: 1426050 | > loss_text_ce: 0.023580169305205345 (0.03066364748631294) | > loss_mel_ce: 2.1397860050201416 (nan) | > loss: 0.008584786206483841 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13740998918878444) | > loader_time: 0.0052 (0.004697685825668296)  --> TIME: 2025-02-26 22:13:15 -- STEP: 22456/36938 -- GLOBAL_STEP: 1426100 | > loss_text_ce: 0.022340726107358932 (0.030662121045912313) | > loss_mel_ce: 2.7505738735198975 (nan) | > loss: 0.011003630235791206 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13741491203593503) | > loader_time: 0.0046 (0.004697702026129343)  --> TIME: 2025-02-26 22:13:25 -- STEP: 22506/36938 -- GLOBAL_STEP: 1426150 | > loss_text_ce: 0.036904532462358475 (0.03066151037617825) | > loss_mel_ce: 3.06619930267334 (nan) | > loss: 0.01231390517205 (nan) | > current_lr: 5e-06 | > step_time: 0.1784 (0.13741916480047867) | > loader_time: 0.0069 (0.00469793972943205)  --> TIME: 2025-02-26 22:13:34 -- STEP: 22556/36938 -- GLOBAL_STEP: 1426200 | > loss_text_ce: 0.033046312630176544 (0.030658615150044314) | > loss_mel_ce: 1.1076350212097168 (nan) | > loss: 0.0045265136286616325 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13741282560360685) | > loader_time: 0.0051 (0.004698047244309893)  --> TIME: 2025-02-26 22:13:44 -- STEP: 22606/36938 -- GLOBAL_STEP: 1426250 | > loss_text_ce: 0.023492610082030296 (0.030658674623365186) | > loss_mel_ce: 2.4788525104522705 (nan) | > loss: 0.009929941035807133 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1374222084261575) | > loader_time: 0.0045 (0.004698113351862543)  --> TIME: 2025-02-26 22:13:53 -- STEP: 22656/36938 -- GLOBAL_STEP: 1426300 | > loss_text_ce: 0.02590763568878174 (0.030656067025202356) | > loss_mel_ce: 2.4673991203308105 (nan) | > loss: 0.009894074872136116 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.1374213436123846) | > loader_time: 0.0047 (0.004697969499028312)  --> TIME: 2025-02-26 22:14:03 -- STEP: 22706/36938 -- GLOBAL_STEP: 1426350 | > loss_text_ce: 0.03188347816467285 (0.030653987877610853) | > loss_mel_ce: 3.1467056274414062 (nan) | > loss: 0.0126134492456913 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.1374270375957471) | > loader_time: 0.0055 (0.004698009908036531)  --> TIME: 2025-02-26 22:14:12 -- STEP: 22756/36938 -- GLOBAL_STEP: 1426400 | > loss_text_ce: 0.03603285178542137 (0.030656074334323474) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13744017008759488) | > loader_time: 0.0046 (0.004698039756589807)  --> TIME: 2025-02-26 22:14:22 -- STEP: 22806/36938 -- GLOBAL_STEP: 1426450 | > loss_text_ce: 0.036686066538095474 (0.030657455489351584) | > loss_mel_ce: 3.0094807147979736 (nan) | > loss: 0.012087964452803135 (nan) | > current_lr: 5e-06 | > step_time: 0.1662 (0.13743326509875986) | > loader_time: 0.0048 (0.004698172782707909)  --> TIME: 2025-02-26 22:14:31 -- STEP: 22856/36938 -- GLOBAL_STEP: 1426500 | > loss_text_ce: 0.036686595529317856 (0.03065624850573368) | > loss_mel_ce: 2.316452980041504 (nan) | > loss: 0.00933785643428564 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13742775924563785) | > loader_time: 0.0046 (0.004698128864236899)  --> TIME: 2025-02-26 22:14:41 -- STEP: 22906/36938 -- GLOBAL_STEP: 1426550 | > loss_text_ce: 0.03278154507279396 (0.030655274040172317) | > loss_mel_ce: 3.150479316711426 (nan) | > loss: 0.012631988152861595 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13744301701449715) | > loader_time: 0.0045 (0.004698155915738164)  --> TIME: 2025-02-26 22:14:50 -- STEP: 22956/36938 -- GLOBAL_STEP: 1426600 | > loss_text_ce: 0.02917550690472126 (0.030654588229857692) | > loss_mel_ce: 1.57891845703125 (nan) | > loss: 0.006381325889378786 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13744423249848411) | > loader_time: 0.0046 (0.004698015875383435)  --> TIME: 2025-02-26 22:14:59 -- STEP: 23006/36938 -- GLOBAL_STEP: 1426650 | > loss_text_ce: 0.03781658783555031 (0.03065402548643022) | > loss_mel_ce: 2.834961175918579 (nan) | > loss: 0.011399911716580391 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.1374432788272013) | > loader_time: 0.0045 (0.004697811341354082)  --> TIME: 2025-02-26 22:15:09 -- STEP: 23056/36938 -- GLOBAL_STEP: 1426700 | > loss_text_ce: 0.033194489777088165 (0.03065310014240295) | > loss_mel_ce: 3.2069284915924072 (nan) | > loss: 0.012857631780207157 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.1374386533794925) | > loader_time: 0.0046 (0.004697570467392308)  --> TIME: 2025-02-26 22:15:18 -- STEP: 23106/36938 -- GLOBAL_STEP: 1426750 | > loss_text_ce: 0.021253356710076332 (0.0306533232078675) | > loss_mel_ce: 0.021968603134155273 (nan) | > loss: 0.00017151571228168905 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13743033464219276) | > loader_time: 0.0045 (0.0046973117943770685)  --> TIME: 2025-02-26 22:15:27 -- STEP: 23156/36938 -- GLOBAL_STEP: 1426800 | > loss_text_ce: 0.026994211599230766 (0.030652964961155837) | > loss_mel_ce: 2.79768967628479 (nan) | > loss: 0.011209064163267612 (nan) | > current_lr: 5e-06 | > step_time: 0.149 (0.13742613928329142) | > loader_time: 0.0048 (0.004697127969468001)  --> TIME: 2025-02-26 22:15:36 -- STEP: 23206/36938 -- GLOBAL_STEP: 1426850 | > loss_text_ce: 0.030220041051506996 (0.030651586763497368) | > loss_mel_ce: 3.124890089035034 (nan) | > loss: 0.012520278804004192 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.13742331706233382) | > loader_time: 0.0046 (0.004696975604610196)  --> TIME: 2025-02-26 22:15:46 -- STEP: 23256/36938 -- GLOBAL_STEP: 1426900 | > loss_text_ce: 0.027034403756260872 (0.030648208991596388) | > loss_mel_ce: 3.104795455932617 (nan) | > loss: 0.012427897192537785 (nan) | > current_lr: 5e-06 | > step_time: 0.1888 (0.13742236897098184) | > loader_time: 0.0048 (0.004697003149682787)  --> TIME: 2025-02-26 22:15:55 -- STEP: 23306/36938 -- GLOBAL_STEP: 1426950 | > loss_text_ce: 0.028854770585894585 (0.030649817735767946) | > loss_mel_ce: 2.269407272338867 (nan) | > loss: 0.009120088070631027 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13743472457351494) | > loader_time: 0.0048 (0.004697278723802234)  --> TIME: 2025-02-26 22:16:05 -- STEP: 23356/36938 -- GLOBAL_STEP: 1427000 | > loss_text_ce: 0.027941256761550903 (0.030649319180481378) | > loss_mel_ce: 1.1201112270355225 (nan) | > loss: 0.004555764142423868 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13743771820122247) | > loader_time: 0.0045 (0.004697094614486553)  --> TIME: 2025-02-26 22:16:14 -- STEP: 23406/36938 -- GLOBAL_STEP: 1427050 | > loss_text_ce: 0.03327998146414757 (0.030646860349060254) | > loss_mel_ce: 1.094990849494934 (nan) | > loss: 0.004477265756577253 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13742725369177097) | > loader_time: 0.0045 (0.004696874835292933)  --> TIME: 2025-02-26 22:16:23 -- STEP: 23456/36938 -- GLOBAL_STEP: 1427100 | > loss_text_ce: 0.03077744133770466 (0.030645399040549377) | > loss_mel_ce: 2.808415651321411 (nan) | > loss: 0.01126664038747549 (nan) | > current_lr: 5e-06 | > step_time: 0.188 (0.13741223410402112) | > loader_time: 0.0049 (0.004697004818493456)  --> TIME: 2025-02-26 22:16:32 -- STEP: 23506/36938 -- GLOBAL_STEP: 1427150 | > loss_text_ce: 0.033153098076581955 (0.030645013778116015) | > loss_mel_ce: 3.190211534500122 (nan) | > loss: 0.012791129760444164 (nan) | > current_lr: 5e-06 | > step_time: 0.1855 (0.13741281989441714) | > loader_time: 0.0056 (0.004697072569851064)  --> TIME: 2025-02-26 22:16:41 -- STEP: 23556/36938 -- GLOBAL_STEP: 1427200 | > loss_text_ce: 0.037510503083467484 (0.03064531181772656) | > loss_mel_ce: 3.0097832679748535 (nan) | > loss: 0.012092435732483864 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13739622178719776) | > loader_time: 0.0046 (0.0046970075450863975)  --> TIME: 2025-02-26 22:16:50 -- STEP: 23606/36938 -- GLOBAL_STEP: 1427250 | > loss_text_ce: 0.024993082508444786 (0.03064355128545731) | > loss_mel_ce: 2.9931681156158447 (nan) | > loss: 0.011976831592619419 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.13739676812054047) | > loader_time: 0.005 (0.004696792418155006)  --> TIME: 2025-02-26 22:17:00 -- STEP: 23656/36938 -- GLOBAL_STEP: 1427300 | > loss_text_ce: 0.03454365208745003 (0.030644667616719842) | > loss_mel_ce: 2.827747344970703 (nan) | > loss: 0.011358298361301422 (nan) | > current_lr: 5e-06 | > step_time: 0.1434 (0.13739893490453006) | > loader_time: 0.0047 (0.0046967451621601425)  --> TIME: 2025-02-26 22:17:09 -- STEP: 23706/36938 -- GLOBAL_STEP: 1427350 | > loss_text_ce: 0.030655764043331146 (0.030644615551516215) | > loss_mel_ce: 1.876259207725525 (nan) | > loss: 0.007567123044282198 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.13739441669732513) | > loader_time: 0.0048 (0.004696705216025017)  --> TIME: 2025-02-26 22:17:19 -- STEP: 23756/36938 -- GLOBAL_STEP: 1427400 | > loss_text_ce: 0.042151179164648056 (0.030644666513635562) | > loss_mel_ce: 2.5973329544067383 (nan) | > loss: 0.010474144481122494 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13739558073341845) | > loader_time: 0.0046 (0.004696714223725826)  --> TIME: 2025-02-26 22:17:28 -- STEP: 23806/36938 -- GLOBAL_STEP: 1427450 | > loss_text_ce: 0.029284195974469185 (0.030645940627901654) | > loss_mel_ce: 2.1719512939453125 (nan) | > loss: 0.008735062554478645 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13739545404595546) | > loader_time: 0.0046 (0.0046968623028075356)  --> TIME: 2025-02-26 22:17:37 -- STEP: 23856/36938 -- GLOBAL_STEP: 1427500 | > loss_text_ce: 0.030774764716625214 (0.0306422664831484) | > loss_mel_ce: 3.004537582397461 (nan) | > loss: 0.012044890783727169 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13739154795165967) | > loader_time: 0.0045 (0.004696863587873226)  --> TIME: 2025-02-26 22:17:46 -- STEP: 23906/36938 -- GLOBAL_STEP: 1427550 | > loss_text_ce: 0.03225788101553917 (0.030641917658967895) | > loss_mel_ce: 1.986708641052246 (nan) | > loss: 0.00801177229732275 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13738870955825985) | > loader_time: 0.0046 (0.004696963163117493)  --> TIME: 2025-02-26 22:17:56 -- STEP: 23956/36938 -- GLOBAL_STEP: 1427600 | > loss_text_ce: 0.031220778822898865 (0.03064027353477027) | > loss_mel_ce: 3.38523006439209 (nan) | > loss: 0.013557344675064087 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.13738567000300042) | > loader_time: 0.0046 (0.004696848476484424)  --> TIME: 2025-02-26 22:18:05 -- STEP: 24006/36938 -- GLOBAL_STEP: 1427650 | > loss_text_ce: 0.03264140710234642 (0.03064178916191015) | > loss_mel_ce: 3.169215679168701 (nan) | > loss: 0.012705782428383827 (nan) | > current_lr: 5e-06 | > step_time: 0.1832 (0.13739355971274272) | > loader_time: 0.0056 (0.0046969857859847815)  --> TIME: 2025-02-26 22:18:15 -- STEP: 24056/36938 -- GLOBAL_STEP: 1427700 | > loss_text_ce: 0.029365448281168938 (0.03063979693519269) | > loss_mel_ce: 2.916832208633423 (nan) | > loss: 0.011691262014210224 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13740520535571454) | > loader_time: 0.0048 (0.00469708383222097)  --> TIME: 2025-02-26 22:18:25 -- STEP: 24106/36938 -- GLOBAL_STEP: 1427750 | > loss_text_ce: 0.03788067400455475 (0.030639758421167475) | > loss_mel_ce: 2.7609496116638184 (nan) | > loss: 0.011106469668447971 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13741116775750656) | > loader_time: 0.0045 (0.004697072439685085)  --> TIME: 2025-02-26 22:18:34 -- STEP: 24156/36938 -- GLOBAL_STEP: 1427800 | > loss_text_ce: 0.033095236867666245 (0.030640775432023218) | > loss_mel_ce: 2.325589656829834 (nan) | > loss: 0.00935986079275608 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.137413923778682) | > loader_time: 0.0045 (0.004697202629434253)  --> TIME: 2025-02-26 22:18:43 -- STEP: 24206/36938 -- GLOBAL_STEP: 1427850 | > loss_text_ce: 0.026012059301137924 (0.030636760565155026) | > loss_mel_ce: 2.9994771480560303 (nan) | > loss: 0.012005909346044064 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.1374054448048867) | > loader_time: 0.0048 (0.0046971469125225256)  --> TIME: 2025-02-26 22:18:52 -- STEP: 24256/36938 -- GLOBAL_STEP: 1427900 | > loss_text_ce: 0.027423197403550148 (0.030638716808932015) | > loss_mel_ce: 2.7439558506011963 (nan) | > loss: 0.010997536592185497 (nan) | > current_lr: 5e-06 | > step_time: 0.1615 (0.13739931628619098) | > loader_time: 0.0046 (0.004696980413629382)  --> TIME: 2025-02-26 22:19:02 -- STEP: 24306/36938 -- GLOBAL_STEP: 1427950 | > loss_text_ce: 0.027382155880331993 (0.03063839502561886) | > loss_mel_ce: 2.1875693798065186 (nan) | > loss: 0.008789490908384323 (nan) | > current_lr: 5e-06 | > step_time: 0.1529 (0.13739835986441265) | > loader_time: 0.0046 (0.004697012938580505)  --> TIME: 2025-02-26 22:19:11 -- STEP: 24356/36938 -- GLOBAL_STEP: 1428000 | > loss_text_ce: 0.03262268379330635 (0.030637339098157395) | > loss_mel_ce: 2.995863676071167 (nan) | > loss: 0.012017803266644478 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13739233316312452) | > loader_time: 0.0047 (0.004696805080887436)  --> TIME: 2025-02-26 22:19:20 -- STEP: 24406/36938 -- GLOBAL_STEP: 1428050 | > loss_text_ce: 0.03771437704563141 (0.030638427157891446) | > loss_mel_ce: 2.6421959400177 (nan) | > loss: 0.010634565725922585 (nan) | > current_lr: 5e-06 | > step_time: 0.15 (0.13739319950754855) | > loader_time: 0.0046 (0.004696613070047115)  --> TIME: 2025-02-26 22:19:30 -- STEP: 24456/36938 -- GLOBAL_STEP: 1428100 | > loss_text_ce: 0.028108539059758186 (0.030637842106311028) | > loss_mel_ce: 2.85115122795105 (nan) | > loss: 0.01142563484609127 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13739505906412278) | > loader_time: 0.0046 (0.0046965877410044985)  --> TIME: 2025-02-26 22:19:39 -- STEP: 24506/36938 -- GLOBAL_STEP: 1428150 | > loss_text_ce: 0.026374775916337967 (0.0306369227034442) | > loss_mel_ce: 2.586653470993042 (nan) | > loss: 0.01036916021257639 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.1373880364637433) | > loader_time: 0.0048 (0.004696662432026828)  --> TIME: 2025-02-26 22:19:48 -- STEP: 24556/36938 -- GLOBAL_STEP: 1428200 | > loss_text_ce: 0.035287659615278244 (0.030638141725455828) | > loss_mel_ce: 2.1173818111419678 (nan) | > loss: 0.008542339317500591 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13737015610390527) | > loader_time: 0.0046 (0.004696700079352895)  --> TIME: 2025-02-26 22:19:57 -- STEP: 24606/36938 -- GLOBAL_STEP: 1428250 | > loss_text_ce: 0.02187357284128666 (0.030639621172091076) | > loss_mel_ce: 2.489133358001709 (nan) | > loss: 0.009964313358068466 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13737616978514827) | > loader_time: 0.0048 (0.004696690095378859)  --> TIME: 2025-02-26 22:20:07 -- STEP: 24656/36938 -- GLOBAL_STEP: 1428300 | > loss_text_ce: 0.035880692303180695 (0.03064019943441) | > loss_mel_ce: 3.0246388912200928 (nan) | > loss: 0.012144920416176319 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13737931399860576) | > loader_time: 0.0053 (0.004696753700393719)  --> TIME: 2025-02-26 22:20:16 -- STEP: 24706/36938 -- GLOBAL_STEP: 1428350 | > loss_text_ce: 0.035331740975379944 (0.03063991490441567) | > loss_mel_ce: 1.6219221353530884 (nan) | > loss: 0.006576404441148043 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13738977387010673) | > loader_time: 0.0045 (0.0046967768451033065)  --> TIME: 2025-02-26 22:20:26 -- STEP: 24756/36938 -- GLOBAL_STEP: 1428400 | > loss_text_ce: 0.032374076545238495 (0.030639564926557894) | > loss_mel_ce: 2.389653205871582 (nan) | > loss: 0.009611220099031925 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13738958830593798) | > loader_time: 0.0046 (0.004696674754496696)  --> TIME: 2025-02-26 22:20:35 -- STEP: 24806/36938 -- GLOBAL_STEP: 1428450 | > loss_text_ce: 0.024526625871658325 (0.03064073304765709) | > loss_mel_ce: 2.8578989505767822 (nan) | > loss: 0.011438197456300259 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13738814023466986) | > loader_time: 0.0045 (0.004696523432941908)  --> TIME: 2025-02-26 22:20:45 -- STEP: 24856/36938 -- GLOBAL_STEP: 1428500 | > loss_text_ce: 0.028020964935421944 (0.030641736958929324) | > loss_mel_ce: 1.0833717584609985 (nan) | > loss: 0.004410288762301207 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1373967640377679) | > loader_time: 0.0045 (0.0046966595303639955)  --> TIME: 2025-02-26 22:20:54 -- STEP: 24906/36938 -- GLOBAL_STEP: 1428550 | > loss_text_ce: 0.02812889777123928 (0.030643709931789005) | > loss_mel_ce: 2.698709487915039 (nan) | > loss: 0.010820787400007248 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13739771418118488) | > loader_time: 0.0047 (0.004696718805776104)  --> TIME: 2025-02-26 22:21:04 -- STEP: 24956/36938 -- GLOBAL_STEP: 1428600 | > loss_text_ce: 0.03399836644530296 (0.030644361003301146) | > loss_mel_ce: 2.9545719623565674 (nan) | > loss: 0.011859406717121601 (nan) | > current_lr: 5e-06 | > step_time: 0.1835 (0.13740614916885063) | > loader_time: 0.005 (0.004696684983090363)  --> TIME: 2025-02-26 22:21:13 -- STEP: 25006/36938 -- GLOBAL_STEP: 1428650 | > loss_text_ce: 0.029539665207266808 (0.030643868764573708) | > loss_mel_ce: 0.00708770751953125 (nan) | > loss: 0.0001453467266401276 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.1374020885870112) | > loader_time: 0.0047 (0.0046964538561709394)  --> TIME: 2025-02-26 22:21:23 -- STEP: 25056/36938 -- GLOBAL_STEP: 1428700 | > loss_text_ce: 0.028803259134292603 (0.0306433531720009) | > loss_mel_ce: 2.328742027282715 (nan) | > loss: 0.009355339221656322 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13741005916210272) | > loader_time: 0.0053 (0.004696577073742132)  --> TIME: 2025-02-26 22:21:32 -- STEP: 25106/36938 -- GLOBAL_STEP: 1428750 | > loss_text_ce: 0.018541621044278145 (0.03064086133515731) | > loss_mel_ce: 1.9646700620651245 (nan) | > loss: 0.007869888097047806 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13741512685280577) | > loader_time: 0.0046 (0.004696663523977652)  --> TIME: 2025-02-26 22:21:41 -- STEP: 25156/36938 -- GLOBAL_STEP: 1428800 | > loss_text_ce: 0.022113217040896416 (0.030639057858611034) | > loss_mel_ce: 1.3432369232177734 (nan) | > loss: 0.005418056156486273 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13740963649817994) | > loader_time: 0.0045 (0.004696576019826336)  --> TIME: 2025-02-26 22:21:50 -- STEP: 25206/36938 -- GLOBAL_STEP: 1428850 | > loss_text_ce: 0.029589135199785233 (0.03064134295038184) | > loss_mel_ce: 1.87359619140625 (nan) | > loss: 0.007552323397248983 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13739723240034277) | > loader_time: 0.0046 (0.004696319786734034)  --> TIME: 2025-02-26 22:22:00 -- STEP: 25256/36938 -- GLOBAL_STEP: 1428900 | > loss_text_ce: 0.039573125541210175 (0.030643231726789483) | > loss_mel_ce: 3.0772688388824463 (nan) | > loss: 0.012368421070277691 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13739771640183687) | > loader_time: 0.0048 (0.004696254691331479)  --> TIME: 2025-02-26 22:22:09 -- STEP: 25306/36938 -- GLOBAL_STEP: 1428950 | > loss_text_ce: 0.03752464801073074 (0.030645026308312513) | > loss_mel_ce: 2.5618207454681396 (nan) | > loss: 0.010314863175153732 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13739802364793155) | > loader_time: 0.0051 (0.004696485488054663)  --> TIME: 2025-02-26 22:22:18 -- STEP: 25356/36938 -- GLOBAL_STEP: 1429000 | > loss_text_ce: 0.0361764058470726 (0.030642569858367358) | > loss_mel_ce: 2.192868232727051 (nan) | > loss: 0.008845415897667408 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1373932608709592) | > loader_time: 0.0052 (0.004696557829857307)  --> TIME: 2025-02-26 22:22:28 -- STEP: 25406/36938 -- GLOBAL_STEP: 1429050 | > loss_text_ce: 0.01792639121413231 (0.030641620079997385) | > loss_mel_ce: 1.063670039176941 (nan) | > loss: 0.004292049445211887 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.1373969873443211) | > loader_time: 0.0047 (0.004696487113546015)  --> TIME: 2025-02-26 22:22:37 -- STEP: 25456/36938 -- GLOBAL_STEP: 1429100 | > loss_text_ce: 0.027565501630306244 (0.030639358965259565) | > loss_mel_ce: 3.3400492668151855 (nan) | > loss: 0.0133635513484478 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13740297848514477) | > loader_time: 0.0047 (0.00469687969747538)  --> TIME: 2025-02-26 22:22:46 -- STEP: 25506/36938 -- GLOBAL_STEP: 1429150 | > loss_text_ce: 0.026887716725468636 (0.030635078500886634) | > loss_mel_ce: 1.8702170848846436 (nan) | > loss: 0.007528194226324558 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1373972195434094) | > loader_time: 0.0045 (0.004696853392864296)  --> TIME: 2025-02-26 22:22:56 -- STEP: 25556/36938 -- GLOBAL_STEP: 1429200 | > loss_text_ce: 0.023108359426259995 (0.03063387286436865) | > loss_mel_ce: 1.6650574207305908 (nan) | > loss: 0.006699070800095797 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13739871598125955) | > loader_time: 0.0046 (0.004696663014857769)  --> TIME: 2025-02-26 22:23:05 -- STEP: 25606/36938 -- GLOBAL_STEP: 1429250 | > loss_text_ce: 0.03074808605015278 (0.03063414748994855) | > loss_mel_ce: 1.5802688598632812 (nan) | > loss: 0.0063929250463843346 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.1373951223542206) | > loader_time: 0.0047 (0.004696602301048967)  --> TIME: 2025-02-26 22:23:14 -- STEP: 25656/36938 -- GLOBAL_STEP: 1429300 | > loss_text_ce: 0.03463325276970863 (0.0306356628056426) | > loss_mel_ce: 2.0848891735076904 (nan) | > loss: 0.0084108030423522 (nan) | > current_lr: 5e-06 | > step_time: 0.1697 (0.13739358287828546) | > loader_time: 0.0048 (0.004696479180463547)  --> TIME: 2025-02-26 22:23:24 -- STEP: 25706/36938 -- GLOBAL_STEP: 1429350 | > loss_text_ce: 0.037794433534145355 (0.03063654803528488) | > loss_mel_ce: 2.7250564098358154 (nan) | > loss: 0.01096369419246912 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13739990293956483) | > loader_time: 0.0046 (0.0046963471619895095)  --> TIME: 2025-02-26 22:23:33 -- STEP: 25756/36938 -- GLOBAL_STEP: 1429400 | > loss_text_ce: 0.043115872889757156 (0.030636081642476755) | > loss_mel_ce: 2.427170991897583 (nan) | > loss: 0.009802726097404957 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1373885816938299) | > loader_time: 0.0044 (0.004696274223970408)  --> TIME: 2025-02-26 22:23:43 -- STEP: 25806/36938 -- GLOBAL_STEP: 1429450 | > loss_text_ce: 0.033619966357946396 (0.030637064925382325) | > loss_mel_ce: 2.4149296283721924 (nan) | > loss: 0.009716467000544071 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13740117625839876) | > loader_time: 0.0045 (0.004696346221760605)  --> TIME: 2025-02-26 22:23:52 -- STEP: 25856/36938 -- GLOBAL_STEP: 1429500 | > loss_text_ce: 0.031557414680719376 (0.03063322775006323) | > loss_mel_ce: 3.768359422683716 (nan) | > loss: 0.015079035423696041 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.13739988690998323) | > loader_time: 0.0049 (0.004696250395240731)  --> TIME: 2025-02-26 22:24:01 -- STEP: 25906/36938 -- GLOBAL_STEP: 1429550 | > loss_text_ce: 0.036530837416648865 (0.03063543576619269) | > loss_mel_ce: 3.0785725116729736 (nan) | > loss: 0.01236152183264494 (nan) | > current_lr: 5e-06 | > step_time: 0.1529 (0.1374014259203795) | > loader_time: 0.0054 (0.004696324305452241)  --> TIME: 2025-02-26 22:24:11 -- STEP: 25956/36938 -- GLOBAL_STEP: 1429600 | > loss_text_ce: 0.027756106108427048 (0.03063429551821468) | > loss_mel_ce: 2.053521156311035 (nan) | > loss: 0.008259037509560585 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1373974399549019) | > loader_time: 0.0046 (0.0046961381652538955)  --> TIME: 2025-02-26 22:24:20 -- STEP: 26006/36938 -- GLOBAL_STEP: 1429650 | > loss_text_ce: 0.027337292209267616 (0.030632252655223607) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.137402274411871) | > loader_time: 0.0045 (0.004696049965648156)  --> TIME: 2025-02-26 22:24:30 -- STEP: 26056/36938 -- GLOBAL_STEP: 1429700 | > loss_text_ce: 0.02814585156738758 (0.030632496983356038) | > loss_mel_ce: 0.11004638671875 (nan) | > loss: 0.0005483819404616952 (nan) | > current_lr: 5e-06 | > step_time: 0.1353 (0.13740549318659456) | > loader_time: 0.0045 (0.004695884995494154)  --> TIME: 2025-02-26 22:24:39 -- STEP: 26106/36938 -- GLOBAL_STEP: 1429750 | > loss_text_ce: 0.039096951484680176 (0.030631615098747476) | > loss_mel_ce: 2.5059814453125 (nan) | > loss: 0.010099517181515694 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.1373969346072536) | > loader_time: 0.0044 (0.004695684473459553)  --> TIME: 2025-02-26 22:24:48 -- STEP: 26156/36938 -- GLOBAL_STEP: 1429800 | > loss_text_ce: 0.026209814473986626 (0.030628274021831027) | > loss_mel_ce: 2.481046199798584 (nan) | > loss: 0.009949428960680962 (nan) | > current_lr: 5e-06 | > step_time: 0.1436 (0.13739949322600062) | > loader_time: 0.0046 (0.004695482092870611)  --> TIME: 2025-02-26 22:24:58 -- STEP: 26206/36938 -- GLOBAL_STEP: 1429850 | > loss_text_ce: 0.02901272475719452 (0.030628701246330835) | > loss_mel_ce: 2.5858397483825684 (nan) | > loss: 0.010376399382948875 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13740087336987367) | > loader_time: 0.0045 (0.004695543467357083)  --> TIME: 2025-02-26 22:25:07 -- STEP: 26256/36938 -- GLOBAL_STEP: 1429900 | > loss_text_ce: 0.020803358405828476 (0.030627566839421187) | > loss_mel_ce: 1.218115210533142 (nan) | > loss: 0.004916343837976456 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.1374030245796059) | > loader_time: 0.0045 (0.004695417176725496)  --> TIME: 2025-02-26 22:25:16 -- STEP: 26306/36938 -- GLOBAL_STEP: 1429950 | > loss_text_ce: 0.024399740621447563 (0.03062583091028691) | > loss_mel_ce: 3.404447078704834 (nan) | > loss: 0.013606536202132702 (nan) | > current_lr: 5e-06 | > step_time: 0.1578 (0.1374040209439506) | > loader_time: 0.0048 (0.004695228131689464)  --> TIME: 2025-02-26 22:25:26 -- STEP: 26356/36938 -- GLOBAL_STEP: 1430000 | > loss_text_ce: 0.032435107976198196 (0.030626602350784498) | > loss_mel_ce: 2.034815788269043 (nan) | > loss: 0.008203377015888691 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13741085832913988) | > loader_time: 0.0046 (0.004695149361046937) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1430000.pth  --> TIME: 2025-02-26 22:25:42 -- STEP: 26406/36938 -- GLOBAL_STEP: 1430050 | > loss_text_ce: 0.039269816130399704 (0.0306253411347666) | > loss_mel_ce: 3.8250324726104736 (nan) | > loss: 0.015334534458816051 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.1374158478321332) | > loader_time: 0.0046 (0.00469560900538434)  --> TIME: 2025-02-26 22:25:51 -- STEP: 26456/36938 -- GLOBAL_STEP: 1430100 | > loss_text_ce: 0.024206794798374176 (0.03062370907607026) | > loss_mel_ce: 2.756011962890625 (nan) | > loss: 0.01103261485695839 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13740598869749135) | > loader_time: 0.0048 (0.004695594950604723)  --> TIME: 2025-02-26 22:26:00 -- STEP: 26506/36938 -- GLOBAL_STEP: 1430150 | > loss_text_ce: 0.03482428565621376 (0.030623651337231172) | > loss_mel_ce: 2.8600082397460938 (nan) | > loss: 0.011487431824207306 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.1374034427709605) | > loader_time: 0.0046 (0.004695773907250324)  --> TIME: 2025-02-26 22:26:09 -- STEP: 26556/36938 -- GLOBAL_STEP: 1430200 | > loss_text_ce: 0.030337564647197723 (0.03062127617624141) | > loss_mel_ce: 2.8618998527526855 (nan) | > loss: 0.011477133259177208 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.1373988234380912) | > loader_time: 0.0047 (0.004695825214791058)  --> TIME: 2025-02-26 22:26:18 -- STEP: 26606/36938 -- GLOBAL_STEP: 1430250 | > loss_text_ce: 0.034956011921167374 (0.030620576743301004) | > loss_mel_ce: 2.741084575653076 (nan) | > loss: 0.01101603452116251 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13739358023898973) | > loader_time: 0.0045 (0.004695784361893116)  --> TIME: 2025-02-26 22:26:28 -- STEP: 26656/36938 -- GLOBAL_STEP: 1430300 | > loss_text_ce: 0.029538538306951523 (0.030622836764876586) | > loss_mel_ce: 1.797486424446106 (nan) | > loss: 0.007250099442899227 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.1373945680879609) | > loader_time: 0.0053 (0.004696185017476401)  --> TIME: 2025-02-26 22:26:37 -- STEP: 26706/36938 -- GLOBAL_STEP: 1430350 | > loss_text_ce: 0.029543429613113403 (0.03062282541309855) | > loss_mel_ce: 2.6623120307922363 (nan) | > loss: 0.010681966319680214 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.1373890630852323) | > loader_time: 0.0047 (0.004696218554807354)  --> TIME: 2025-02-26 22:26:46 -- STEP: 26756/36938 -- GLOBAL_STEP: 1430400 | > loss_text_ce: 0.03361087292432785 (0.030621841780354462) | > loss_mel_ce: 2.456807851791382 (nan) | > loss: 0.009882614016532898 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13739504600644167) | > loader_time: 0.0048 (0.004696259059826146)  --> TIME: 2025-02-26 22:26:56 -- STEP: 26806/36938 -- GLOBAL_STEP: 1430450 | > loss_text_ce: 0.02931976318359375 (0.03062086911247836) | > loss_mel_ce: 2.339400291442871 (nan) | > loss: 0.009399683214724064 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13738801766978884) | > loader_time: 0.0046 (0.004696385741082605)  --> TIME: 2025-02-26 22:27:05 -- STEP: 26856/36938 -- GLOBAL_STEP: 1430500 | > loss_text_ce: 0.034126266837120056 (0.03061977792483743) | > loss_mel_ce: 2.2410051822662354 (nan) | > loss: 0.00902829971164465 (nan) | > current_lr: 5e-06 | > step_time: 0.1265 (0.1373858722700837) | > loader_time: 0.0047 (0.0046965093583551994)  --> TIME: 2025-02-26 22:27:14 -- STEP: 26906/36938 -- GLOBAL_STEP: 1430550 | > loss_text_ce: 0.032117169350385666 (0.03061758607048532) | > loss_mel_ce: 2.63955020904541 (nan) | > loss: 0.010601854883134365 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13739136864259502) | > loader_time: 0.0045 (0.004696514839878974)  --> TIME: 2025-02-26 22:27:24 -- STEP: 26956/36938 -- GLOBAL_STEP: 1430600 | > loss_text_ce: 0.021058909595012665 (0.03061929606829593) | > loss_mel_ce: 1.0661154985427856 (nan) | > loss: 0.004314184654504061 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13739369322946243) | > loader_time: 0.0044 (0.004696415154901016)  --> TIME: 2025-02-26 22:27:33 -- STEP: 27006/36938 -- GLOBAL_STEP: 1430650 | > loss_text_ce: 0.02820204198360443 (0.03061768895302801) | > loss_mel_ce: 2.3319058418273926 (nan) | > loss: 0.009365508332848549 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1373939878241559) | > loader_time: 0.0046 (0.004696405438028554)  --> TIME: 2025-02-26 22:27:43 -- STEP: 27056/36938 -- GLOBAL_STEP: 1430700 | > loss_text_ce: 0.03106720559298992 (0.03061825734298329) | > loss_mel_ce: 1.3645495176315308 (nan) | > loss: 0.005538162309676409 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.13739651224797933) | > loader_time: 0.0046 (0.0046966044085749005)  --> TIME: 2025-02-26 22:27:52 -- STEP: 27106/36938 -- GLOBAL_STEP: 1430750 | > loss_text_ce: 0.035481542348861694 (0.030618344850735526) | > loss_mel_ce: 2.753998279571533 (nan) | > loss: 0.011069364845752716 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.137398320722781) | > loader_time: 0.0047 (0.00469677107699865)  --> TIME: 2025-02-26 22:28:01 -- STEP: 27156/36938 -- GLOBAL_STEP: 1430800 | > loss_text_ce: 0.035591885447502136 (0.0306158765188252) | > loss_mel_ce: 3.1804749965667725 (nan) | > loss: 0.012762170284986496 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13738992571005526) | > loader_time: 0.0046 (0.004696657378783464)  --> TIME: 2025-02-26 22:28:11 -- STEP: 27206/36938 -- GLOBAL_STEP: 1430850 | > loss_text_ce: 0.040878668427467346 (0.03061697530081081) | > loss_mel_ce: 2.8721771240234375 (nan) | > loss: 0.011559746228158474 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.1373924456878003) | > loader_time: 0.0047 (0.004696530655343006)  --> TIME: 2025-02-26 22:28:21 -- STEP: 27256/36938 -- GLOBAL_STEP: 1430900 | > loss_text_ce: 0.028057919815182686 (0.030614243520592298) | > loss_mel_ce: 3.3067190647125244 (nan) | > loss: 0.013233242556452751 (nan) | > current_lr: 5e-06 | > step_time: 0.1954 (0.13740808702403984) | > loader_time: 0.0048 (0.004696470483293404)  --> TIME: 2025-02-26 22:28:30 -- STEP: 27306/36938 -- GLOBAL_STEP: 1430950 | > loss_text_ce: 0.02516286075115204 (0.030615505151697437) | > loss_mel_ce: 2.4003243446350098 (nan) | > loss: 0.009624949656426907 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13740149283334083) | > loader_time: 0.0045 (0.0046964435361553205)  --> TIME: 2025-02-26 22:28:39 -- STEP: 27356/36938 -- GLOBAL_STEP: 1431000 | > loss_text_ce: 0.028496457263827324 (0.030616950445204117) | > loss_mel_ce: 2.6619670391082764 (nan) | > loss: 0.010676443576812744 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13741018181569553) | > loader_time: 0.0047 (0.004696523242046259)  --> TIME: 2025-02-26 22:28:49 -- STEP: 27406/36938 -- GLOBAL_STEP: 1431050 | > loss_text_ce: 0.03782008960843086 (0.03061927580462225) | > loss_mel_ce: 2.712813377380371 (nan) | > loss: 0.010915213264524937 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13740800796856262) | > loader_time: 0.0047 (0.004696568920430854)  --> TIME: 2025-02-26 22:28:58 -- STEP: 27456/36938 -- GLOBAL_STEP: 1431100 | > loss_text_ce: 0.02656400203704834 (0.03062014393862226) | > loss_mel_ce: 0.8119683265686035 (nan) | > loss: 0.0033275093883275986 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13741327262980724) | > loader_time: 0.0046 (0.004696658996991057)  --> TIME: 2025-02-26 22:29:08 -- STEP: 27506/36938 -- GLOBAL_STEP: 1431150 | > loss_text_ce: 0.026723070070147514 (0.0306197711075647) | > loss_mel_ce: 2.286731481552124 (nan) | > loss: 0.009180376306176186 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.1374165567321659) | > loader_time: 0.0048 (0.004696704817279359)  --> TIME: 2025-02-26 22:29:17 -- STEP: 27556/36938 -- GLOBAL_STEP: 1431200 | > loss_text_ce: 0.03356155753135681 (0.03062203224940816) | > loss_mel_ce: 3.271468162536621 (nan) | > loss: 0.01311519742012024 (nan) | > current_lr: 5e-06 | > step_time: 0.1877 (0.13742693430246797) | > loader_time: 0.0056 (0.004696845895822853)  --> TIME: 2025-02-26 22:29:27 -- STEP: 27606/36938 -- GLOBAL_STEP: 1431250 | > loss_text_ce: 0.030075078830122948 (0.03062165764566369) | > loss_mel_ce: 3.2216525077819824 (nan) | > loss: 0.012903681956231594 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.1374363543494483) | > loader_time: 0.0048 (0.004697031649370025)  --> TIME: 2025-02-26 22:29:37 -- STEP: 27656/36938 -- GLOBAL_STEP: 1431300 | > loss_text_ce: 0.03915857896208763 (0.030620484286686882) | > loss_mel_ce: 2.5647690296173096 (nan) | > loss: 0.010333046317100525 (nan) | > current_lr: 5e-06 | > step_time: 0.1811 (0.1374454740537265) | > loader_time: 0.0053 (0.004697015115143687)  --> TIME: 2025-02-26 22:29:46 -- STEP: 27706/36938 -- GLOBAL_STEP: 1431350 | > loss_text_ce: 0.025562167167663574 (0.030618331638945822) | > loss_mel_ce: 2.5372695922851562 (nan) | > loss: 0.010169968008995056 (nan) | > current_lr: 5e-06 | > step_time: 0.147 (0.13744518391657384) | > loader_time: 0.0046 (0.004696845646877964)  --> TIME: 2025-02-26 22:29:55 -- STEP: 27756/36938 -- GLOBAL_STEP: 1431400 | > loss_text_ce: 0.03331427648663521 (0.030619217622086345) | > loss_mel_ce: 2.2563693523406982 (nan) | > loss: 0.009086046367883682 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13743694037486862) | > loader_time: 0.0046 (0.004696657204425262)  --> TIME: 2025-02-26 22:30:05 -- STEP: 27806/36938 -- GLOBAL_STEP: 1431450 | > loss_text_ce: 0.022489266470074654 (0.030618408888272018) | > loss_mel_ce: 2.5879809856414795 (nan) | > loss: 0.010359009727835655 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13744171439319572) | > loader_time: 0.0052 (0.004696623881000542)  --> TIME: 2025-02-26 22:30:14 -- STEP: 27856/36938 -- GLOBAL_STEP: 1431500 | > loss_text_ce: 0.028884412720799446 (0.0306190024582234) | > loss_mel_ce: 1.4318474531173706 (nan) | > loss: 0.005796555429697037 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13743671655894657) | > loader_time: 0.0045 (0.004696451739495261)  --> TIME: 2025-02-26 22:30:23 -- STEP: 27906/36938 -- GLOBAL_STEP: 1431550 | > loss_text_ce: 0.03675127774477005 (0.03061876481850218) | > loss_mel_ce: 2.5947885513305664 (nan) | > loss: 0.01044261921197176 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.13742605596513888) | > loader_time: 0.0048 (0.0046965214015676545)  --> TIME: 2025-02-26 22:30:33 -- STEP: 27956/36938 -- GLOBAL_STEP: 1431600 | > loss_text_ce: 0.02847152389585972 (0.030619101217131846) | > loss_mel_ce: 2.5301718711853027 (nan) | > loss: 0.010153347626328468 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.13744273155369152) | > loader_time: 0.0047 (0.00469647075233071)  --> TIME: 2025-02-26 22:30:42 -- STEP: 28006/36938 -- GLOBAL_STEP: 1431650 | > loss_text_ce: 0.036148834973573685 (0.030618873702951185) | > loss_mel_ce: 3.2613182067871094 (nan) | > loss: 0.013085187412798405 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13744297642576137) | > loader_time: 0.0052 (0.004696361313528182)  --> TIME: 2025-02-26 22:30:52 -- STEP: 28056/36938 -- GLOBAL_STEP: 1431700 | > loss_text_ce: 0.035443637520074844 (0.030618237648572882) | > loss_mel_ce: 2.483245372772217 (nan) | > loss: 0.00999479740858078 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13744447700889725) | > loader_time: 0.0045 (0.004696347382390187)  --> TIME: 2025-02-26 22:31:01 -- STEP: 28106/36938 -- GLOBAL_STEP: 1431750 | > loss_text_ce: 0.027425521984696388 (0.03061996795561997) | > loss_mel_ce: 2.206315755844116 (nan) | > loss: 0.008864053525030613 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13744428147118126) | > loader_time: 0.0047 (0.004696470668271259)  --> TIME: 2025-02-26 22:31:10 -- STEP: 28156/36938 -- GLOBAL_STEP: 1431800 | > loss_text_ce: 0.03660118207335472 (0.03061816188437519) | > loss_mel_ce: 2.2314252853393555 (nan) | > loss: 0.00900010485202074 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13743550945266636) | > loader_time: 0.0045 (0.004696400976092956)  --> TIME: 2025-02-26 22:31:19 -- STEP: 28206/36938 -- GLOBAL_STEP: 1431850 | > loss_text_ce: 0.028957922011613846 (0.03061935980513701) | > loss_mel_ce: 1.9168627262115479 (nan) | > loss: 0.00772151118144393 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.13743803847354355) | > loader_time: 0.0046 (0.004696373498956244)  --> TIME: 2025-02-26 22:31:29 -- STEP: 28256/36938 -- GLOBAL_STEP: 1431900 | > loss_text_ce: 0.024707691743969917 (0.030619956741432107) | > loss_mel_ce: 2.876230001449585 (nan) | > loss: 0.011511658318340778 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13743656130332613) | > loader_time: 0.0045 (0.004696374731655155)  --> TIME: 2025-02-26 22:31:38 -- STEP: 28306/36938 -- GLOBAL_STEP: 1431950 | > loss_text_ce: 0.03582299128174782 (0.03061784188257065) | > loss_mel_ce: 2.3844106197357178 (nan) | > loss: 0.009604102931916714 (nan) | > current_lr: 5e-06 | > step_time: 0.1271 (0.13743298339109375) | > loader_time: 0.0045 (0.004696289482095128)  --> TIME: 2025-02-26 22:31:47 -- STEP: 28356/36938 -- GLOBAL_STEP: 1432000 | > loss_text_ce: 0.026901420205831528 (0.030618284649879354) | > loss_mel_ce: 0.9785099625587463 (nan) | > loss: 0.003989728167653084 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13743091667130725) | > loader_time: 0.0046 (0.0046962658362456455)  --> TIME: 2025-02-26 22:31:57 -- STEP: 28406/36938 -- GLOBAL_STEP: 1432050 | > loss_text_ce: 0.03324132040143013 (0.030617602729156413) | > loss_mel_ce: 2.716082811355591 (nan) | > loss: 0.010910016484558582 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13742973385147347) | > loader_time: 0.0045 (0.004696216304931868)  --> TIME: 2025-02-26 22:32:06 -- STEP: 28456/36938 -- GLOBAL_STEP: 1432100 | > loss_text_ce: 0.028437964618206024 (0.030617355286801044) | > loss_mel_ce: 3.0074949264526367 (nan) | > loss: 0.012047353200614452 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13743864703218714) | > loader_time: 0.0046 (0.004696162398155725)  --> TIME: 2025-02-26 22:32:16 -- STEP: 28506/36938 -- GLOBAL_STEP: 1432150 | > loss_text_ce: 0.03660796955227852 (0.030616077084969485) | > loss_mel_ce: 0.015167236328125 (nan) | > loss: 0.00020545718143694103 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13743957557871012) | > loader_time: 0.0047 (0.00469640036812917)  --> TIME: 2025-02-26 22:32:25 -- STEP: 28556/36938 -- GLOBAL_STEP: 1432200 | > loss_text_ce: 0.02957203984260559 (0.030616752700966927) | > loss_mel_ce: 2.9667530059814453 (nan) | > loss: 0.011890179477632046 (nan) | > current_lr: 5e-06 | > step_time: 0.1469 (0.13744120999267) | > loader_time: 0.0048 (0.004696513811969868)  --> TIME: 2025-02-26 22:32:34 -- STEP: 28606/36938 -- GLOBAL_STEP: 1432250 | > loss_text_ce: 0.026716142892837524 (0.03061750191047026) | > loss_mel_ce: 2.6416995525360107 (nan) | > loss: 0.010588952340185642 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13744238987674523) | > loader_time: 0.0045 (0.004696872495549754)  --> TIME: 2025-02-26 22:32:44 -- STEP: 28656/36938 -- GLOBAL_STEP: 1432300 | > loss_text_ce: 0.03359927237033844 (0.030616295605500495) | > loss_mel_ce: 3.271867513656616 (nan) | > loss: 0.013116932474076748 (nan) | > current_lr: 5e-06 | > step_time: 0.1757 (0.13744307714990334) | > loader_time: 0.0057 (0.004696963195824734)  --> TIME: 2025-02-26 22:32:53 -- STEP: 28706/36938 -- GLOBAL_STEP: 1432350 | > loss_text_ce: 0.028749393299221992 (0.030617569907245457) | > loss_mel_ce: 1.804262638092041 (nan) | > loss: 0.007273857481777668 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.137448423333861) | > loader_time: 0.0048 (0.004697085722542744)  --> TIME: 2025-02-26 22:33:03 -- STEP: 28756/36938 -- GLOBAL_STEP: 1432400 | > loss_text_ce: 0.028854738920927048 (0.030620038070378365) | > loss_mel_ce: 3.18343448638916 (nan) | > loss: 0.01274718064814806 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.1374568931083727) | > loader_time: 0.0053 (0.00469723768767535)  --> TIME: 2025-02-26 22:33:13 -- STEP: 28806/36938 -- GLOBAL_STEP: 1432450 | > loss_text_ce: 0.038120657205581665 (0.03062109794138366) | > loss_mel_ce: 2.750206708908081 (nan) | > loss: 0.011064792051911354 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13745731190245403) | > loader_time: 0.0045 (0.004697254256457899)  --> TIME: 2025-02-26 22:33:22 -- STEP: 28856/36938 -- GLOBAL_STEP: 1432500 | > loss_text_ce: 0.03651385009288788 (0.030621104545852946) | > loss_mel_ce: 0.8391898274421692 (nan) | > loss: 0.0034750148188322783 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13745191117419764) | > loader_time: 0.0046 (0.004697385556750055)  --> TIME: 2025-02-26 22:33:31 -- STEP: 28906/36938 -- GLOBAL_STEP: 1432550 | > loss_text_ce: 0.038159530609846115 (0.03062262419367447) | > loss_mel_ce: 2.3948709964752197 (nan) | > loss: 0.009654884226620197 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13745693186200028) | > loader_time: 0.0045 (0.004697521714564376)  --> TIME: 2025-02-26 22:33:41 -- STEP: 28956/36938 -- GLOBAL_STEP: 1432600 | > loss_text_ce: 0.027116570621728897 (0.03062210751195737) | > loss_mel_ce: 2.719435691833496 (nan) | > loss: 0.010899017564952374 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.13745832201986383) | > loader_time: 0.0048 (0.004697680465093805)  --> TIME: 2025-02-26 22:33:50 -- STEP: 29006/36938 -- GLOBAL_STEP: 1432650 | > loss_text_ce: 0.026727771386504173 (0.030621220880706082) | > loss_mel_ce: 2.5180444717407227 (nan) | > loss: 0.010098302736878395 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13746549698810606) | > loader_time: 0.0047 (0.004697744652196332)  --> TIME: 2025-02-26 22:33:59 -- STEP: 29056/36938 -- GLOBAL_STEP: 1432700 | > loss_text_ce: 0.032547686249017715 (0.030622018441240052) | > loss_mel_ce: 2.4399919509887695 (nan) | > loss: 0.00981166586279869 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.137461115175342) | > loader_time: 0.0045 (0.0046977288200466556)  --> TIME: 2025-02-26 22:34:09 -- STEP: 29106/36938 -- GLOBAL_STEP: 1432750 | > loss_text_ce: 0.019374892115592957 (0.030622725875263372) | > loss_mel_ce: 2.0777947902679443 (nan) | > loss: 0.008322102017700672 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13745661343134444) | > loader_time: 0.0045 (0.004697843228043087)  --> TIME: 2025-02-26 22:34:18 -- STEP: 29156/36938 -- GLOBAL_STEP: 1432800 | > loss_text_ce: 0.037197988480329514 (0.030622185627519213) | > loss_mel_ce: 2.922776222229004 (nan) | > loss: 0.011745930649340153 (nan) | > current_lr: 5e-06 | > step_time: 0.1572 (0.13745622843246458) | > loader_time: 0.0053 (0.004697989339706804)  --> TIME: 2025-02-26 22:34:27 -- STEP: 29206/36938 -- GLOBAL_STEP: 1432850 | > loss_text_ce: 0.027420368045568466 (0.03062255791003755) | > loss_mel_ce: 2.38981294631958 (nan) | > loss: 0.009592195972800255 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.1374548537057052) | > loader_time: 0.0046 (0.004697885781389821)  --> TIME: 2025-02-26 22:34:37 -- STEP: 29256/36938 -- GLOBAL_STEP: 1432900 | > loss_text_ce: 0.029764359816908836 (0.030624339847271184) | > loss_mel_ce: 2.486339807510376 (nan) | > loss: 0.00998454075306654 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13745753675300013) | > loader_time: 0.0045 (0.0046979205625318566)  --> TIME: 2025-02-26 22:34:46 -- STEP: 29306/36938 -- GLOBAL_STEP: 1432950 | > loss_text_ce: 0.026191052049398422 (0.030623638898794353) | > loss_mel_ce: 2.256490707397461 (nan) | > loss: 0.009058261290192604 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.13746294957870925) | > loader_time: 0.0048 (0.004697916849899518)  --> TIME: 2025-02-26 22:34:56 -- STEP: 29356/36938 -- GLOBAL_STEP: 1433000 | > loss_text_ce: 0.036414675414562225 (0.030623727327598854) | > loss_mel_ce: 3.176759958267212 (nan) | > loss: 0.01275069359689951 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.1374571789089067) | > loader_time: 0.0046 (0.0046977317045715635)  --> TIME: 2025-02-26 22:35:05 -- STEP: 29406/36938 -- GLOBAL_STEP: 1433050 | > loss_text_ce: 0.017754144966602325 (0.030622214487097142) | > loss_mel_ce: 1.0788263082504272 (nan) | > loss: 0.0043515102006495 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1374558860701243) | > loader_time: 0.0044 (0.004697800704202603)  --> TIME: 2025-02-26 22:35:14 -- STEP: 29456/36938 -- GLOBAL_STEP: 1433100 | > loss_text_ce: 0.029938120394945145 (0.030624476178868064) | > loss_mel_ce: 2.2227985858917236 (nan) | > loss: 0.008939431980252266 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13745866217314165) | > loader_time: 0.0048 (0.004697995154123334)  --> TIME: 2025-02-26 22:35:24 -- STEP: 29506/36938 -- GLOBAL_STEP: 1433150 | > loss_text_ce: 0.027045121416449547 (0.030621925419928014) | > loss_mel_ce: 3.2173476219177246 (nan) | > loss: 0.012874574400484562 (nan) | > current_lr: 5e-06 | > step_time: 0.1701 (0.13745823339923802) | > loader_time: 0.0047 (0.004697947504398717)  --> TIME: 2025-02-26 22:35:33 -- STEP: 29556/36938 -- GLOBAL_STEP: 1433200 | > loss_text_ce: 0.0344584584236145 (0.030621020270153933) | > loss_mel_ce: 3.146533727645874 (nan) | > loss: 0.012622985057532787 (nan) | > current_lr: 5e-06 | > step_time: 0.1514 (0.13745649860553577) | > loader_time: 0.0046 (0.004697928700980765)  --> TIME: 2025-02-26 22:35:43 -- STEP: 29606/36938 -- GLOBAL_STEP: 1433250 | > loss_text_ce: 0.027949949726462364 (0.030622824954742493) | > loss_mel_ce: 0.0367431640625 (nan) | > loss: 0.00025671871844679117 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1374608126957747) | > loader_time: 0.0045 (0.004697922765423468)  --> TIME: 2025-02-26 22:35:52 -- STEP: 29656/36938 -- GLOBAL_STEP: 1433300 | > loss_text_ce: 0.027473071590065956 (0.030621932635472835) | > loss_mel_ce: 2.463156223297119 (nan) | > loss: 0.009883449412882328 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13747107835513248) | > loader_time: 0.0046 (0.004698026371606757)  --> TIME: 2025-02-26 22:36:02 -- STEP: 29706/36938 -- GLOBAL_STEP: 1433350 | > loss_text_ce: 0.026888545602560043 (0.030622759081641117) | > loss_mel_ce: 2.790198802947998 (nan) | > loss: 0.011178919114172459 (nan) | > current_lr: 5e-06 | > step_time: 0.1262 (0.13747059596649988) | > loader_time: 0.0046 (0.004697992152688728)  --> TIME: 2025-02-26 22:36:11 -- STEP: 29756/36938 -- GLOBAL_STEP: 1433400 | > loss_text_ce: 0.03150570020079613 (0.03062311750250447) | > loss_mel_ce: 2.6538922786712646 (nan) | > loss: 0.010656340979039669 (nan) | > current_lr: 5e-06 | > step_time: 0.1732 (0.13747031806633603) | > loader_time: 0.0047 (0.004697926840261027)  --> TIME: 2025-02-26 22:36:20 -- STEP: 29806/36938 -- GLOBAL_STEP: 1433450 | > loss_text_ce: 0.036364033818244934 (0.030622689556901003) | > loss_mel_ce: 2.579254150390625 (nan) | > loss: 0.010379438288509846 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1374678702864139) | > loader_time: 0.0045 (0.004697863642724489)  --> TIME: 2025-02-26 22:36:29 -- STEP: 29856/36938 -- GLOBAL_STEP: 1433500 | > loss_text_ce: 0.02943653240799904 (0.030621402858182396) | > loss_mel_ce: 2.8391709327697754 (nan) | > loss: 0.011383363977074623 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.1374563474970754) | > loader_time: 0.0045 (0.0046976592795002926)  --> TIME: 2025-02-26 22:36:38 -- STEP: 29906/36938 -- GLOBAL_STEP: 1433550 | > loss_text_ce: 0.033102329820394516 (0.030620989280706192) | > loss_mel_ce: 2.7861316204071045 (nan) | > loss: 0.011187436990439892 (nan) | > current_lr: 5e-06 | > step_time: 0.1995 (0.13745381959200828) | > loader_time: 0.0051 (0.0046976535190971745)  --> TIME: 2025-02-26 22:36:48 -- STEP: 29956/36938 -- GLOBAL_STEP: 1433600 | > loss_text_ce: 0.03129555284976959 (0.030621936949050674) | > loss_mel_ce: 2.192274570465088 (nan) | > loss: 0.008823691867291927 (nan) | > current_lr: 5e-06 | > step_time: 0.1266 (0.1374581338232624) | > loader_time: 0.0045 (0.004697554467086997)  --> TIME: 2025-02-26 22:36:57 -- STEP: 30006/36938 -- GLOBAL_STEP: 1433650 | > loss_text_ce: 0.030656330287456512 (0.030625119899054875) | > loss_mel_ce: 2.603736400604248 (nan) | > loss: 0.010453940369188786 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13744998892760263) | > loader_time: 0.0045 (0.004697416850998179)  --> TIME: 2025-02-26 22:37:07 -- STEP: 30056/36938 -- GLOBAL_STEP: 1433700 | > loss_text_ce: 0.03331618383526802 (0.030626029673700903) | > loss_mel_ce: 2.7553069591522217 (nan) | > loss: 0.01106596551835537 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13745471105348483) | > loader_time: 0.0046 (0.004697538894448209)  --> TIME: 2025-02-26 22:37:16 -- STEP: 30106/36938 -- GLOBAL_STEP: 1433750 | > loss_text_ce: 0.021973570808768272 (0.03062399578141417) | > loss_mel_ce: 1.1153708696365356 (nan) | > loss: 0.00451327208429575 (nan) | > current_lr: 5e-06 | > step_time: 0.1084 (0.13745262051376353) | > loader_time: 0.0044 (0.00469760163665205)  --> TIME: 2025-02-26 22:37:26 -- STEP: 30156/36938 -- GLOBAL_STEP: 1433800 | > loss_text_ce: 0.01991274021565914 (0.030622859610880713) | > loss_mel_ce: 2.7554991245269775 (nan) | > loss: 0.011013539507985115 (nan) | > current_lr: 5e-06 | > step_time: 0.1469 (0.13746068931128858) | > loader_time: 0.0053 (0.004697621904391629)  --> TIME: 2025-02-26 22:37:35 -- STEP: 30206/36938 -- GLOBAL_STEP: 1433850 | > loss_text_ce: 0.03939202055335045 (0.030626352377418297) | > loss_mel_ce: 2.8629651069641113 (nan) | > loss: 0.011517290957272053 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13745549327686526) | > loader_time: 0.0045 (0.004697688855785783)  --> TIME: 2025-02-26 22:37:44 -- STEP: 30256/36938 -- GLOBAL_STEP: 1433900 | > loss_text_ce: 0.035058777779340744 (0.030624882430908763) | > loss_mel_ce: 2.9679324626922607 (nan) | > loss: 0.011916632764041424 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.1374582994841582) | > loader_time: 0.0046 (0.0046976923249122055)  --> TIME: 2025-02-26 22:37:54 -- STEP: 30306/36938 -- GLOBAL_STEP: 1433950 | > loss_text_ce: 0.028481101617217064 (0.030624999227718906) | > loss_mel_ce: 3.038968563079834 (nan) | > loss: 0.01217241957783699 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.13747113982828646) | > loader_time: 0.0047 (0.004697858244878871)  --> TIME: 2025-02-26 22:38:03 -- STEP: 30356/36938 -- GLOBAL_STEP: 1434000 | > loss_text_ce: 0.04076329618692398 (0.03062645046223944) | > loss_mel_ce: 3.0347602367401123 (nan) | > loss: 0.012204459868371487 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13747132071897167) | > loader_time: 0.0049 (0.004697923164528519)  --> TIME: 2025-02-26 22:38:13 -- STEP: 30406/36938 -- GLOBAL_STEP: 1434050 | > loss_text_ce: 0.02835901267826557 (0.0306255861462168) | > loss_mel_ce: 1.0343059301376343 (nan) | > loss: 0.004216924775391817 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13747140739118346) | > loader_time: 0.0045 (0.004697831447192063)  --> TIME: 2025-02-26 22:38:22 -- STEP: 30456/36938 -- GLOBAL_STEP: 1434100 | > loss_text_ce: 0.024257447570562363 (0.030628764908855584) | > loss_mel_ce: 2.2543537616729736 (nan) | > loss: 0.009042108431458473 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13746386726509513) | > loader_time: 0.0045 (0.004697718135259072)  --> TIME: 2025-02-26 22:38:32 -- STEP: 30506/36938 -- GLOBAL_STEP: 1434150 | > loss_text_ce: 0.030125442892313004 (0.030628385187613273) | > loss_mel_ce: 2.8980751037597656 (nan) | > loss: 0.011619843542575836 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.13747717091070222) | > loader_time: 0.0049 (0.004697840448083271)  --> TIME: 2025-02-26 22:38:41 -- STEP: 30556/36938 -- GLOBAL_STEP: 1434200 | > loss_text_ce: 0.026784993708133698 (0.030628760693276493) | > loss_mel_ce: 2.7131311893463135 (nan) | > loss: 0.010872683487832546 (nan) | > current_lr: 5e-06 | > step_time: 0.1692 (0.13748035887833038) | > loader_time: 0.0048 (0.004697893072850628)  --> TIME: 2025-02-26 22:38:51 -- STEP: 30606/36938 -- GLOBAL_STEP: 1434250 | > loss_text_ce: 0.03077474981546402 (0.030627285784173966) | > loss_mel_ce: 2.7574422359466553 (nan) | > loss: 0.011064354330301285 (nan) | > current_lr: 5e-06 | > step_time: 0.1925 (0.13749214544690855) | > loader_time: 0.0048 (0.004697811102653674)  --> TIME: 2025-02-26 22:39:01 -- STEP: 30656/36938 -- GLOBAL_STEP: 1434300 | > loss_text_ce: 0.024952322244644165 (0.03062802260697055) | > loss_mel_ce: 1.6630859375 (nan) | > loss: 0.006698564626276493 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13750516992676837) | > loader_time: 0.0045 (0.004697947566531637)  --> TIME: 2025-02-26 22:39:10 -- STEP: 30706/36938 -- GLOBAL_STEP: 1434350 | > loss_text_ce: 0.02465120516717434 (0.030626183786039846) | > loss_mel_ce: 1.5485804080963135 (nan) | > loss: 0.006242983043193817 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13750314539374833) | > loader_time: 0.0045 (0.004697969167432675)  --> TIME: 2025-02-26 22:39:20 -- STEP: 30756/36938 -- GLOBAL_STEP: 1434400 | > loss_text_ce: 0.027629300951957703 (0.030625776279753904) | > loss_mel_ce: 2.626802444458008 (nan) | > loss: 0.010533460415899754 (nan) | > current_lr: 5e-06 | > step_time: 0.1679 (0.13750559789242445) | > loader_time: 0.0047 (0.004697886008189323)  --> TIME: 2025-02-26 22:39:29 -- STEP: 30806/36938 -- GLOBAL_STEP: 1434450 | > loss_text_ce: 0.0401059091091156 (0.030623682231200863) | > loss_mel_ce: 1.4433422088623047 (nan) | > loss: 0.005886699538677931 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13750316426791717) | > loader_time: 0.0054 (0.0046978706912019766)  --> TIME: 2025-02-26 22:39:38 -- STEP: 30856/36938 -- GLOBAL_STEP: 1434500 | > loss_text_ce: 0.02764279767870903 (0.030623838898974156) | > loss_mel_ce: 2.235126495361328 (nan) | > loss: 0.008979243226349354 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.13749897399652225) | > loader_time: 0.0046 (0.004697795239699012)  --> TIME: 2025-02-26 22:39:47 -- STEP: 30906/36938 -- GLOBAL_STEP: 1434550 | > loss_text_ce: 0.032597865909338 (0.030624008360572875) | > loss_mel_ce: 2.8349223136901855 (nan) | > loss: 0.011379048228263855 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13750104838448957) | > loader_time: 0.0049 (0.004697767915073378)  --> TIME: 2025-02-26 22:39:57 -- STEP: 30956/36938 -- GLOBAL_STEP: 1434600 | > loss_text_ce: 0.02307852916419506 (0.03062492282554209) | > loss_mel_ce: 2.207883834838867 (nan) | > loss: 0.008853025734424591 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13750509540922973) | > loader_time: 0.0047 (0.004697947373361231)  --> TIME: 2025-02-26 22:40:06 -- STEP: 31006/36938 -- GLOBAL_STEP: 1434650 | > loss_text_ce: 0.030576156452298164 (0.030625491776176045) | > loss_mel_ce: 2.796867847442627 (nan) | > loss: 0.01122001651674509 (nan) | > current_lr: 5e-06 | > step_time: 0.178 (0.13749798525458665) | > loader_time: 0.0048 (0.004697973671431903)  --> TIME: 2025-02-26 22:40:15 -- STEP: 31056/36938 -- GLOBAL_STEP: 1434700 | > loss_text_ce: 0.03122408129274845 (0.030625380515455) | > loss_mel_ce: 2.7836952209472656 (nan) | > loss: 0.011170314624905586 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1374956029920721) | > loader_time: 0.0046 (0.004698008721111846)  --> TIME: 2025-02-26 22:40:25 -- STEP: 31106/36938 -- GLOBAL_STEP: 1434750 | > loss_text_ce: 0.029567118734121323 (0.030624671193260827) | > loss_mel_ce: 0.0811767578125 (nan) | > loss: 0.0004394598654471338 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13749985746085647) | > loader_time: 0.0046 (0.0046980535762529)  --> TIME: 2025-02-26 22:40:34 -- STEP: 31156/36938 -- GLOBAL_STEP: 1434800 | > loss_text_ce: 0.034065499901771545 (0.030625918816389715) | > loss_mel_ce: 3.170346260070801 (nan) | > loss: 0.012715920805931091 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13749837967960513) | > loader_time: 0.0045 (0.004698062688399216)  --> TIME: 2025-02-26 22:40:44 -- STEP: 31206/36938 -- GLOBAL_STEP: 1434850 | > loss_text_ce: 0.03179332613945007 (0.030628168023182623) | > loss_mel_ce: 2.7864673137664795 (nan) | > loss: 0.011183574795722961 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13750273982874928) | > loader_time: 0.0045 (0.004697957703878205)  --> TIME: 2025-02-26 22:40:53 -- STEP: 31256/36938 -- GLOBAL_STEP: 1434900 | > loss_text_ce: 0.027566634118556976 (0.030627919798781864) | > loss_mel_ce: 2.874385356903076 (nan) | > loss: 0.011515683494508266 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13750652757971354) | > loader_time: 0.0053 (0.004697920913308172)  --> TIME: 2025-02-26 22:41:03 -- STEP: 31306/36938 -- GLOBAL_STEP: 1434950 | > loss_text_ce: 0.02629222348332405 (0.030628705728607642) | > loss_mel_ce: 1.910675048828125 (nan) | > loss: 0.007686378434300423 (nan) | > current_lr: 5e-06 | > step_time: 0.1083 (0.13750783353615678) | > loader_time: 0.0045 (0.004697948928413341)  --> TIME: 2025-02-26 22:41:12 -- STEP: 31356/36938 -- GLOBAL_STEP: 1435000 | > loss_text_ce: 0.0262734554708004 (0.03062915358253003) | > loss_mel_ce: 2.9762351512908936 (nan) | > loss: 0.011914717964828014 (nan) | > current_lr: 5e-06 | > step_time: 0.1778 (0.13751273223492863) | > loader_time: 0.0048 (0.0046979402124098495)  --> TIME: 2025-02-26 22:41:22 -- STEP: 31406/36938 -- GLOBAL_STEP: 1435050 | > loss_text_ce: 0.03476175293326378 (0.030630028333151607) | > loss_mel_ce: 2.6779816150665283 (nan) | > loss: 0.010764854960143566 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13751890569297215) | > loader_time: 0.0046 (0.0046980251349180826)  --> TIME: 2025-02-26 22:41:31 -- STEP: 31456/36938 -- GLOBAL_STEP: 1435100 | > loss_text_ce: 0.0378902368247509 (0.0306300174382354) | > loss_mel_ce: 3.115415096282959 (nan) | > loss: 0.0125131169334054 (nan) | > current_lr: 5e-06 | > step_time: 0.1821 (0.13751946354809863) | > loader_time: 0.0048 (0.004697882874450986)  --> TIME: 2025-02-26 22:41:41 -- STEP: 31506/36938 -- GLOBAL_STEP: 1435150 | > loss_text_ce: 0.027002764865756035 (0.03063018797716604) | > loss_mel_ce: 2.139608383178711 (nan) | > loss: 0.00859766360372305 (nan) | > current_lr: 5e-06 | > step_time: 0.1078 (0.13752708055629143) | > loader_time: 0.0045 (0.004698131324026089)  --> TIME: 2025-02-26 22:41:50 -- STEP: 31556/36938 -- GLOBAL_STEP: 1435200 | > loss_text_ce: 0.033609431236982346 (0.030629360753104226) | > loss_mel_ce: 2.788733720779419 (nan) | > loss: 0.011199775151908398 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.13752471829476245) | > loader_time: 0.0049 (0.004698188960106438)  --> TIME: 2025-02-26 22:42:00 -- STEP: 31606/36938 -- GLOBAL_STEP: 1435250 | > loss_text_ce: 0.038004808127880096 (0.030630830304157565) | > loss_mel_ce: 2.8757684230804443 (nan) | > loss: 0.011562593281269073 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13752517490185506) | > loader_time: 0.0046 (0.004698192274420342)  --> TIME: 2025-02-26 22:42:09 -- STEP: 31656/36938 -- GLOBAL_STEP: 1435300 | > loss_text_ce: 0.03279830887913704 (0.030632709862800502) | > loss_mel_ce: 1.9774311780929565 (nan) | > loss: 0.00797710195183754 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13752838198753733) | > loader_time: 0.0046 (0.004698330551077078)  --> TIME: 2025-02-26 22:42:18 -- STEP: 31706/36938 -- GLOBAL_STEP: 1435350 | > loss_text_ce: 0.037323880940675735 (0.03063145907695801) | > loss_mel_ce: 2.8913092613220215 (nan) | > loss: 0.011621560901403427 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13752590648189458) | > loader_time: 0.0047 (0.004698372876795272)  --> TIME: 2025-02-26 22:42:28 -- STEP: 31756/36938 -- GLOBAL_STEP: 1435400 | > loss_text_ce: 0.02784930355846882 (0.030631633772231343) | > loss_mel_ce: 3.2572133541107178 (nan) | > loss: 0.013035963289439678 (nan) | > current_lr: 5e-06 | > step_time: 0.1629 (0.13752950112485526) | > loader_time: 0.0046 (0.004698290619475961)  --> TIME: 2025-02-26 22:42:37 -- STEP: 31806/36938 -- GLOBAL_STEP: 1435450 | > loss_text_ce: 0.027074851095676422 (0.030629487700495315) | > loss_mel_ce: 3.1606709957122803 (nan) | > loss: 0.012649785727262497 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13751952202749124) | > loader_time: 0.0045 (0.00469821993228077)  --> TIME: 2025-02-26 22:42:46 -- STEP: 31856/36938 -- GLOBAL_STEP: 1435500 | > loss_text_ce: 0.022031890228390694 (0.030627273621942383) | > loss_mel_ce: 2.2313156127929688 (nan) | > loss: 0.008941855281591415 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.1375196917774938) | > loader_time: 0.0047 (0.004698144138188644)  --> TIME: 2025-02-26 22:42:55 -- STEP: 31906/36938 -- GLOBAL_STEP: 1435550 | > loss_text_ce: 0.027774753049016 (0.030626769741397688) | > loss_mel_ce: 2.3157355785369873 (nan) | > loss: 0.009299645200371742 (nan) | > current_lr: 5e-06 | > step_time: 0.1271 (0.13751780282873557) | > loader_time: 0.0046 (0.004698012470414412)  --> TIME: 2025-02-26 22:43:05 -- STEP: 31956/36938 -- GLOBAL_STEP: 1435600 | > loss_text_ce: 0.039215996861457825 (0.03062710897968869) | > loss_mel_ce: 1.0583738088607788 (nan) | > loss: 0.004355515353381634 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.1375127909911392) | > loader_time: 0.0045 (0.004698064027612678)  --> TIME: 2025-02-26 22:43:14 -- STEP: 32006/36938 -- GLOBAL_STEP: 1435650 | > loss_text_ce: 0.01828474923968315 (0.030626184623003486) | > loss_mel_ce: 2.4886257648468018 (nan) | > loss: 0.009948058053851128 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13751585034514324) | > loader_time: 0.0045 (0.004698188865228726)  --> TIME: 2025-02-26 22:43:23 -- STEP: 32056/36938 -- GLOBAL_STEP: 1435700 | > loss_text_ce: 0.024933110922574997 (0.030627489744950907) | > loss_mel_ce: 2.010317802429199 (nan) | > loss: 0.00807639304548502 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.1375098487728111) | > loader_time: 0.0065 (0.004698302700003684)  --> TIME: 2025-02-26 22:43:33 -- STEP: 32106/36938 -- GLOBAL_STEP: 1435750 | > loss_text_ce: 0.025724835693836212 (0.03062644909137854) | > loss_mel_ce: 1.494621992111206 (nan) | > loss: 0.006033123005181551 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13750830946002593) | > loader_time: 0.0046 (0.004698531364626737)  --> TIME: 2025-02-26 22:43:42 -- STEP: 32156/36938 -- GLOBAL_STEP: 1435800 | > loss_text_ce: 0.03148895129561424 (0.030627552951003265) | > loss_mel_ce: 2.960012435913086 (nan) | > loss: 0.011871038004755974 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13750695920521516) | > loader_time: 0.0047 (0.004698502014993887)  --> TIME: 2025-02-26 22:43:51 -- STEP: 32206/36938 -- GLOBAL_STEP: 1435850 | > loss_text_ce: 0.031033793464303017 (0.03062676549244862) | > loss_mel_ce: 0.182586669921875 (nan) | > loss: 0.0008477003430016339 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.137505921474757) | > loader_time: 0.0046 (0.004698602433516454)  --> TIME: 2025-02-26 22:44:01 -- STEP: 32256/36938 -- GLOBAL_STEP: 1435900 | > loss_text_ce: 0.03437264263629913 (0.030626207527896604) | > loss_mel_ce: 2.6929712295532227 (nan) | > loss: 0.01082279346883297 (nan) | > current_lr: 5e-06 | > step_time: 0.1449 (0.13750443341476679) | > loader_time: 0.0046 (0.004698485246903831)  --> TIME: 2025-02-26 22:44:10 -- STEP: 32306/36938 -- GLOBAL_STEP: 1435950 | > loss_text_ce: 0.03057924285531044 (0.030627079899918384) | > loss_mel_ce: 2.5737454891204834 (nan) | > loss: 0.010334623046219349 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.1375009223107541) | > loader_time: 0.0047 (0.004698530222353658)  --> TIME: 2025-02-26 22:44:19 -- STEP: 32356/36938 -- GLOBAL_STEP: 1436000 | > loss_text_ce: 0.027988407760858536 (0.030626711630466687) | > loss_mel_ce: 2.7407779693603516 (nan) | > loss: 0.010987169109284878 (nan) | > current_lr: 5e-06 | > step_time: 0.1726 (0.13749776239934036) | > loader_time: 0.0055 (0.004698495654715545)  --> TIME: 2025-02-26 22:44:28 -- STEP: 32406/36938 -- GLOBAL_STEP: 1436050 | > loss_text_ce: 0.03588881343603134 (0.0306270177550284) | > loss_mel_ce: 2.7989039421081543 (nan) | > loss: 0.011249179020524025 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.13749674856681243) | > loader_time: 0.0054 (0.0046984524092085304)  --> TIME: 2025-02-26 22:44:38 -- STEP: 32456/36938 -- GLOBAL_STEP: 1436100 | > loss_text_ce: 0.026864519342780113 (0.03062669722816905) | > loss_mel_ce: 2.6335103511810303 (nan) | > loss: 0.010557043366134167 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13749850475056566) | > loader_time: 0.0047 (0.004698747854853362)  --> TIME: 2025-02-26 22:44:47 -- STEP: 32506/36938 -- GLOBAL_STEP: 1436150 | > loss_text_ce: 0.03257999196648598 (0.03062575268562596) | > loss_mel_ce: 2.665982246398926 (nan) | > loss: 0.010708580724895 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13749102315249737) | > loader_time: 0.0045 (0.004698713632728072)  --> TIME: 2025-02-26 22:44:56 -- STEP: 32556/36938 -- GLOBAL_STEP: 1436200 | > loss_text_ce: 0.026874825358390808 (0.030624940782042902) | > loss_mel_ce: 1.894621729850769 (nan) | > loss: 0.007624986581504345 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13748766411755517) | > loader_time: 0.0045 (0.0046986661945693994)  --> TIME: 2025-02-26 22:45:05 -- STEP: 32606/36938 -- GLOBAL_STEP: 1436250 | > loss_text_ce: 0.02966388128697872 (0.030623617890317027) | > loss_mel_ce: 2.30737566947937 (nan) | > loss: 0.009273966774344444 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13748675999755294) | > loader_time: 0.0047 (0.004698549868292298)  --> TIME: 2025-02-26 22:45:15 -- STEP: 32656/36938 -- GLOBAL_STEP: 1436300 | > loss_text_ce: 0.036886926740407944 (0.03062127164324268) | > loss_mel_ce: 1.022047758102417 (nan) | > loss: 0.004202121868729591 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13748921149504864) | > loader_time: 0.0052 (0.004698689247626167)  --> TIME: 2025-02-26 22:45:24 -- STEP: 32706/36938 -- GLOBAL_STEP: 1436350 | > loss_text_ce: 0.032711297273635864 (0.03062130072583403) | > loss_mel_ce: 0.07325280457735062 (nan) | > loss: 0.00042049249168485403 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.13749238399819932) | > loader_time: 0.0049 (0.0046987530872128185)  --> TIME: 2025-02-26 22:45:34 -- STEP: 32756/36938 -- GLOBAL_STEP: 1436400 | > loss_text_ce: 0.03445437178015709 (0.03062067929374575) | > loss_mel_ce: 2.5294032096862793 (nan) | > loss: 0.0101740388199687 (nan) | > current_lr: 5e-06 | > step_time: 0.1574 (0.137486836072916) | > loader_time: 0.0047 (0.004698911390398889)  --> TIME: 2025-02-26 22:45:43 -- STEP: 32806/36938 -- GLOBAL_STEP: 1436450 | > loss_text_ce: 0.034570418298244476 (0.030621333318789568) | > loss_mel_ce: 3.0802488327026367 (nan) | > loss: 0.012360394932329655 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.1374926425651519) | > loader_time: 0.0046 (0.004699026877681103)  --> TIME: 2025-02-26 22:45:52 -- STEP: 32856/36938 -- GLOBAL_STEP: 1436500 | > loss_text_ce: 0.03827492520213127 (0.030619445549684602) | > loss_mel_ce: 3.2825324535369873 (nan) | > loss: 0.013177808374166489 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.1374921052595817) | > loader_time: 0.0053 (0.004699135976085)  --> TIME: 2025-02-26 22:46:02 -- STEP: 32906/36938 -- GLOBAL_STEP: 1436550 | > loss_text_ce: 0.0404416024684906 (0.03061972233969004) | > loss_mel_ce: 0.0341796875 (nan) | > loss: 0.00029611625359393656 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13748537657261428) | > loader_time: 0.0045 (0.004699177534205054)  --> TIME: 2025-02-26 22:46:11 -- STEP: 32956/36938 -- GLOBAL_STEP: 1436600 | > loss_text_ce: 0.036489952355623245 (0.030620972678883807) | > loss_mel_ce: 2.7189269065856934 (nan) | > loss: 0.010934194549918175 (nan) | > current_lr: 5e-06 | > step_time: 0.1916 (0.13748524936024908) | > loader_time: 0.0049 (0.004699137231391781)  --> TIME: 2025-02-26 22:46:21 -- STEP: 33006/36938 -- GLOBAL_STEP: 1436650 | > loss_text_ce: 0.028760341927409172 (0.030620399006783875) | > loss_mel_ce: 3.0643012523651123 (nan) | > loss: 0.012274054810404778 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.13749102624280624) | > loader_time: 0.0048 (0.004699240010817505)  --> TIME: 2025-02-26 22:46:30 -- STEP: 33056/36938 -- GLOBAL_STEP: 1436700 | > loss_text_ce: 0.03086402639746666 (0.03062158051075759) | > loss_mel_ce: 3.0581936836242676 (nan) | > loss: 0.012258166447281837 (nan) | > current_lr: 5e-06 | > step_time: 0.1588 (0.1374921347055187) | > loader_time: 0.005 (0.004699279513617561)  --> TIME: 2025-02-26 22:46:39 -- STEP: 33106/36938 -- GLOBAL_STEP: 1436750 | > loss_text_ce: 0.03646700456738472 (0.03061983985035314) | > loss_mel_ce: 2.80273699760437 (nan) | > loss: 0.011266683228313923 (nan) | > current_lr: 5e-06 | > step_time: 0.1355 (0.13748911450739376) | > loader_time: 0.0046 (0.004699130883024134)  --> TIME: 2025-02-26 22:46:48 -- STEP: 33156/36938 -- GLOBAL_STEP: 1436800 | > loss_text_ce: 0.03661241754889488 (0.03062225108172882) | > loss_mel_ce: 2.711188793182373 (nan) | > loss: 0.010903974063694477 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13748219318392296) | > loader_time: 0.0046 (0.004698968477280202)  --> TIME: 2025-02-26 22:46:57 -- STEP: 33206/36938 -- GLOBAL_STEP: 1436850 | > loss_text_ce: 0.015572982840240002 (0.030621200219877773) | > loss_mel_ce: 0.00799560546875 (nan) | > loss: 9.35261559789069e-05 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1374802054651238) | > loader_time: 0.0045 (0.004698952328675563)  --> TIME: 2025-02-26 22:47:07 -- STEP: 33256/36938 -- GLOBAL_STEP: 1436900 | > loss_text_ce: 0.02647552452981472 (0.03061990570147035) | > loss_mel_ce: 1.4581202268600464 (nan) | > loss: 0.005891253240406513 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13748348463978302) | > loader_time: 0.0052 (0.004699006902503363)  --> TIME: 2025-02-26 22:47:16 -- STEP: 33306/36938 -- GLOBAL_STEP: 1436950 | > loss_text_ce: 0.03107987530529499 (0.03061959308238297) | > loss_mel_ce: 2.5175929069519043 (nan) | > loss: 0.010113781318068504 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13748582283641592) | > loader_time: 0.0046 (0.004699132696312188)  --> TIME: 2025-02-26 22:47:26 -- STEP: 33356/36938 -- GLOBAL_STEP: 1437000 | > loss_text_ce: 0.0367448590695858 (0.030618259945538365) | > loss_mel_ce: 2.4302661418914795 (nan) | > loss: 0.009789726696908474 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.13749065641048092) | > loader_time: 0.0048 (0.004699175585687173)  --> TIME: 2025-02-26 22:47:35 -- STEP: 33406/36938 -- GLOBAL_STEP: 1437050 | > loss_text_ce: 0.03114888072013855 (0.030615798424174875) | > loss_mel_ce: 2.215012311935425 (nan) | > loss: 0.008913339115679264 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.1374915510439237) | > loader_time: 0.0046 (0.004699144635751183)  --> TIME: 2025-02-26 22:47:45 -- STEP: 33456/36938 -- GLOBAL_STEP: 1437100 | > loss_text_ce: 0.03408479318022728 (0.030613079189487916) | > loss_mel_ce: 2.488347291946411 (nan) | > loss: 0.010009652003645897 (nan) | > current_lr: 5e-06 | > step_time: 0.1817 (0.13748555406322424) | > loader_time: 0.0049 (0.0046990192118926966)  --> TIME: 2025-02-26 22:47:54 -- STEP: 33506/36938 -- GLOBAL_STEP: 1437150 | > loss_text_ce: 0.03258872404694557 (0.030611646862166133) | > loss_mel_ce: 2.9722251892089844 (nan) | > loss: 0.011923865415155888 (nan) | > current_lr: 5e-06 | > step_time: 0.1837 (0.13748037023828474) | > loader_time: 0.005 (0.004698899314132134)  --> TIME: 2025-02-26 22:48:03 -- STEP: 33556/36938 -- GLOBAL_STEP: 1437200 | > loss_text_ce: 0.027806038036942482 (0.030612747765579502) | > loss_mel_ce: 2.627859354019165 (nan) | > loss: 0.010538355447351933 (nan) | > current_lr: 5e-06 | > step_time: 0.1774 (0.13748249326677156) | > loader_time: 0.0049 (0.004698817395157851)  --> TIME: 2025-02-26 22:48:13 -- STEP: 33606/36938 -- GLOBAL_STEP: 1437250 | > loss_text_ce: 0.0296779777854681 (0.030613439890087087) | > loss_mel_ce: 4.852614402770996 (nan) | > loss: 0.019374176859855652 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13748485833585686) | > loader_time: 0.0045 (0.004698703120607971)  --> TIME: 2025-02-26 22:48:22 -- STEP: 33656/36938 -- GLOBAL_STEP: 1437300 | > loss_text_ce: 0.03281193971633911 (0.03061203489575176) | > loss_mel_ce: 0.019502639770507812 (nan) | > loss: 0.00020759754988830537 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13748579897165455) | > loader_time: 0.0045 (0.004698595532845049)  --> TIME: 2025-02-26 22:48:31 -- STEP: 33706/36938 -- GLOBAL_STEP: 1437350 | > loss_text_ce: 0.03680016100406647 (0.030611917308574203) | > loss_mel_ce: 0.1485828161239624 (nan) | > loss: 0.0007356467540375888 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.1374893346002881) | > loader_time: 0.0045 (0.004698667753728095)  --> TIME: 2025-02-26 22:48:41 -- STEP: 33756/36938 -- GLOBAL_STEP: 1437400 | > loss_text_ce: 0.034725818783044815 (0.030611490260083283) | > loss_mel_ce: 2.7145650386810303 (nan) | > loss: 0.010909885168075562 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.13748607962232456) | > loader_time: 0.0047 (0.004698516414521182)  --> TIME: 2025-02-26 22:48:50 -- STEP: 33806/36938 -- GLOBAL_STEP: 1437450 | > loss_text_ce: 0.022427763789892197 (0.030609444109086597) | > loss_mel_ce: 1.8156263828277588 (nan) | > loss: 0.007293866015970707 (nan) | > current_lr: 5e-06 | > step_time: 0.1082 (0.13748973750831245) | > loader_time: 0.0044 (0.004698593701257268)  --> TIME: 2025-02-26 22:49:00 -- STEP: 33856/36938 -- GLOBAL_STEP: 1437500 | > loss_text_ce: 0.026970263570547104 (0.030608502019127213) | > loss_mel_ce: 2.752070426940918 (nan) | > loss: 0.01102793961763382 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.1374886930411399) | > loader_time: 0.0046 (0.004698475333377853)  --> TIME: 2025-02-26 22:49:09 -- STEP: 33906/36938 -- GLOBAL_STEP: 1437550 | > loss_text_ce: 0.029409484937787056 (0.030608549415843282) | > loss_mel_ce: 3.2571499347686768 (nan) | > loss: 0.013041903264820576 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.1374924558528041) | > loader_time: 0.0048 (0.004698358214668677)  --> TIME: 2025-02-26 22:49:18 -- STEP: 33956/36938 -- GLOBAL_STEP: 1437600 | > loss_text_ce: 0.024005135521292686 (0.030608116840753465) | > loss_mel_ce: 0.11480604112148285 (nan) | > loss: 0.000550838012713939 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13748747752824808) | > loader_time: 0.0047 (0.00469846380853669)  --> TIME: 2025-02-26 22:49:27 -- STEP: 34006/36938 -- GLOBAL_STEP: 1437650 | > loss_text_ce: 0.03198353573679924 (0.030607875780908657) | > loss_mel_ce: 2.655709981918335 (nan) | > loss: 0.010665451176464558 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.13748575595311083) | > loader_time: 0.0046 (0.004698652993410682)  --> TIME: 2025-02-26 22:49:37 -- STEP: 34056/36938 -- GLOBAL_STEP: 1437700 | > loss_text_ce: 0.025097178295254707 (0.03061067223690437) | > loss_mel_ce: 2.4334959983825684 (nan) | > loss: 0.00975632295012474 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13748232373083585) | > loader_time: 0.0045 (0.004698530865130016)  --> TIME: 2025-02-26 22:49:46 -- STEP: 34106/36938 -- GLOBAL_STEP: 1437750 | > loss_text_ce: 0.03168565407395363 (0.03061159677370542) | > loss_mel_ce: 2.6977500915527344 (nan) | > loss: 0.010831094346940517 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13748506005681593) | > loader_time: 0.0049 (0.004698765142945562)  --> TIME: 2025-02-26 22:49:56 -- STEP: 34156/36938 -- GLOBAL_STEP: 1437800 | > loss_text_ce: 0.025980856269598007 (0.0306098200154761) | > loss_mel_ce: 1.361146092414856 (nan) | > loss: 0.005504472181200981 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.1374873998489306) | > loader_time: 0.0046 (0.004698964692010907)  --> TIME: 2025-02-26 22:50:05 -- STEP: 34206/36938 -- GLOBAL_STEP: 1437850 | > loss_text_ce: 0.03640606626868248 (0.030610291206085202) | > loss_mel_ce: 1.9911410808563232 (nan) | > loss: 0.008045822381973267 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13749226902490835) | > loader_time: 0.0045 (0.004699113717073861)  --> TIME: 2025-02-26 22:50:14 -- STEP: 34256/36938 -- GLOBAL_STEP: 1437900 | > loss_text_ce: 0.026988539844751358 (0.030611211311311146) | > loss_mel_ce: 2.7989516258239746 (nan) | > loss: 0.011214048601686954 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13748818379816077) | > loader_time: 0.0046 (0.004698999528849249)  --> TIME: 2025-02-26 22:50:23 -- STEP: 34306/36938 -- GLOBAL_STEP: 1437950 | > loss_text_ce: 0.03729900345206261 (0.030611858270546143) | > loss_mel_ce: 1.4442073106765747 (nan) | > loss: 0.005878993775695562 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13748052563186763) | > loader_time: 0.0045 (0.004698954406635528)  --> TIME: 2025-02-26 22:50:32 -- STEP: 34356/36938 -- GLOBAL_STEP: 1438000 | > loss_text_ce: 0.046866148710250854 (0.030612179487682335) | > loss_mel_ce: 1.9960342645645142 (nan) | > loss: 0.008106747642159462 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.1374717751167234) | > loader_time: 0.0044 (0.004698851025552815)  --> TIME: 2025-02-26 22:50:42 -- STEP: 34406/36938 -- GLOBAL_STEP: 1438050 | > loss_text_ce: 0.03262985125184059 (0.030612049246797956) | > loss_mel_ce: 2.684467315673828 (nan) | > loss: 0.010782132856547832 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.137467014001691) | > loader_time: 0.0048 (0.004698718674465748)  --> TIME: 2025-02-26 22:50:51 -- STEP: 34456/36938 -- GLOBAL_STEP: 1438100 | > loss_text_ce: 0.03333725035190582 (0.030613161125246313) | > loss_mel_ce: 2.3367297649383545 (nan) | > loss: 0.009405029006302357 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.1374693813154043) | > loader_time: 0.0046 (0.004698626757609136)  --> TIME: 2025-02-26 22:51:00 -- STEP: 34506/36938 -- GLOBAL_STEP: 1438150 | > loss_text_ce: 0.03191804513335228 (0.030613951125618755) | > loss_mel_ce: 2.861732244491577 (nan) | > loss: 0.011482739821076393 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13747216941708906) | > loader_time: 0.0046 (0.00469858815122367)  --> TIME: 2025-02-26 22:51:10 -- STEP: 34556/36938 -- GLOBAL_STEP: 1438200 | > loss_text_ce: 0.02748439647257328 (0.03061444243886326) | > loss_mel_ce: 2.990980386734009 (nan) | > loss: 0.011978035792708397 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13747307250091112) | > loader_time: 0.0046 (0.004698561275290972)  --> TIME: 2025-02-26 22:51:19 -- STEP: 34606/36938 -- GLOBAL_STEP: 1438250 | > loss_text_ce: 0.02643333375453949 (0.03061446720900664) | > loss_mel_ce: 2.4343435764312744 (nan) | > loss: 0.009764987975358963 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.1374736940806332) | > loader_time: 0.0045 (0.004698688016767019)  --> TIME: 2025-02-26 22:51:29 -- STEP: 34656/36938 -- GLOBAL_STEP: 1438300 | > loss_text_ce: 0.025118641555309296 (0.030615129631668608) | > loss_mel_ce: 3.4545161724090576 (nan) | > loss: 0.013808075338602066 (nan) | > current_lr: 5e-06 | > step_time: 0.1574 (0.13747174879630739) | > loader_time: 0.0046 (0.00469869541026658)  --> TIME: 2025-02-26 22:51:38 -- STEP: 34706/36938 -- GLOBAL_STEP: 1438350 | > loss_text_ce: 0.024203021079301834 (0.0306156749169133) | > loss_mel_ce: 3.221792697906494 (nan) | > loss: 0.01288093626499176 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13746277656581668) | > loader_time: 0.0045 (0.004698626961974881)  --> TIME: 2025-02-26 22:51:47 -- STEP: 34756/36938 -- GLOBAL_STEP: 1438400 | > loss_text_ce: 0.02719372883439064 (0.030614830516044016) | > loss_mel_ce: 2.597949981689453 (nan) | > loss: 0.010417237877845764 (nan) | > current_lr: 5e-06 | > step_time: 0.1521 (0.13746465684108508) | > loader_time: 0.0046 (0.004698737126689654)  --> TIME: 2025-02-26 22:51:56 -- STEP: 34806/36938 -- GLOBAL_STEP: 1438450 | > loss_text_ce: 0.027456531301140785 (0.030615264750666685) | > loss_mel_ce: 2.8321104049682617 (nan) | > loss: 0.01134748850017786 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13746101694025503) | > loader_time: 0.0046 (0.004698908507793115)  --> TIME: 2025-02-26 22:52:06 -- STEP: 34856/36938 -- GLOBAL_STEP: 1438500 | > loss_text_ce: 0.03199058398604393 (0.030614555465154864) | > loss_mel_ce: 2.9841153621673584 (nan) | > loss: 0.011968675069510937 (nan) | > current_lr: 5e-06 | > step_time: 0.1906 (0.1374782955383865) | > loader_time: 0.005 (0.004699178400842968)  --> TIME: 2025-02-26 22:52:16 -- STEP: 34906/36938 -- GLOBAL_STEP: 1438550 | > loss_text_ce: 0.03952789306640625 (0.030614977831940702) | > loss_mel_ce: 1.9174323081970215 (nan) | > loss: 0.007765715476125479 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13747462901605814) | > loader_time: 0.0046 (0.004699143640483975)  --> TIME: 2025-02-26 22:52:25 -- STEP: 34956/36938 -- GLOBAL_STEP: 1438600 | > loss_text_ce: 0.034299615770578384 (0.030614957072981397) | > loss_mel_ce: 2.917898654937744 (nan) | > loss: 0.011715073138475418 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13747742478478467) | > loader_time: 0.0046 (0.0046993400933527186)  --> TIME: 2025-02-26 22:52:34 -- STEP: 35006/36938 -- GLOBAL_STEP: 1438650 | > loss_text_ce: 0.025640878826379776 (0.030615494818347022) | > loss_mel_ce: 2.7206387519836426 (nan) | > loss: 0.010897936299443245 (nan) | > current_lr: 5e-06 | > step_time: 0.1888 (0.13747107966289684) | > loader_time: 0.0049 (0.0046993472852250775)  --> TIME: 2025-02-26 22:52:43 -- STEP: 35056/36938 -- GLOBAL_STEP: 1438700 | > loss_text_ce: 0.025088166818022728 (0.030614426199429993) | > loss_mel_ce: 2.517989158630371 (nan) | > loss: 0.010091576725244522 (nan) | > current_lr: 5e-06 | > step_time: 0.175 (0.1374713683756785) | > loader_time: 0.0047 (0.004699335114317139)  --> TIME: 2025-02-26 22:52:53 -- STEP: 35106/36938 -- GLOBAL_STEP: 1438750 | > loss_text_ce: 0.03591484576463699 (0.030615744397439682) | > loss_mel_ce: 2.488358736038208 (nan) | > loss: 0.01001695916056633 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13747003854309303) | > loader_time: 0.0046 (0.004699377112257394)  --> TIME: 2025-02-26 22:53:02 -- STEP: 35156/36938 -- GLOBAL_STEP: 1438800 | > loss_text_ce: 0.028184594586491585 (0.03061588575343962) | > loss_mel_ce: 2.754636287689209 (nan) | > loss: 0.011042940430343151 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.13747601542189167) | > loader_time: 0.0053 (0.004699672139237554)  --> TIME: 2025-02-26 22:53:12 -- STEP: 35206/36938 -- GLOBAL_STEP: 1438850 | > loss_text_ce: 0.027831077575683594 (0.030614197424528346) | > loss_mel_ce: 2.6544270515441895 (nan) | > loss: 0.010643881745636463 (nan) | > current_lr: 5e-06 | > step_time: 0.169 (0.1374779151243527) | > loader_time: 0.0049 (0.0046998382407020855)  --> TIME: 2025-02-26 22:53:21 -- STEP: 35256/36938 -- GLOBAL_STEP: 1438900 | > loss_text_ce: 0.033719081431627274 (0.030613278510074456) | > loss_mel_ce: 2.9029018878936768 (nan) | > loss: 0.011653258465230465 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1374788519636646) | > loader_time: 0.0044 (0.004699857909310992)  --> TIME: 2025-02-26 22:53:31 -- STEP: 35306/36938 -- GLOBAL_STEP: 1438950 | > loss_text_ce: 0.031432874500751495 (0.03061414973574544) | > loss_mel_ce: 2.6118249893188477 (nan) | > loss: 0.010489119216799736 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13749544036251393) | > loader_time: 0.0069 (0.004699919397065317)  --> TIME: 2025-02-26 22:53:40 -- STEP: 35356/36938 -- GLOBAL_STEP: 1439000 | > loss_text_ce: 0.039717793464660645 (0.03061442402423422) | > loss_mel_ce: 2.1049141883850098 (nan) | > loss: 0.008510444313287735 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13748891334860694) | > loader_time: 0.005 (0.004700015884325613)  --> TIME: 2025-02-26 22:53:50 -- STEP: 35406/36938 -- GLOBAL_STEP: 1439050 | > loss_text_ce: 0.03179972246289253 (0.030614256582172435) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13748680759417736) | > loader_time: 0.0045 (0.004700074921501143)  --> TIME: 2025-02-26 22:53:59 -- STEP: 35456/36938 -- GLOBAL_STEP: 1439100 | > loss_text_ce: 0.03985387459397316 (0.030613100972421755) | > loss_mel_ce: 3.3647940158843994 (nan) | > loss: 0.013510508462786674 (nan) | > current_lr: 5e-06 | > step_time: 0.1716 (0.1374824681194905) | > loader_time: 0.0055 (0.004699949470990828)  --> TIME: 2025-02-26 22:54:08 -- STEP: 35506/36938 -- GLOBAL_STEP: 1439150 | > loss_text_ce: 0.027473589405417442 (0.030614263654193865) | > loss_mel_ce: 3.195075511932373 (nan) | > loss: 0.01278789434581995 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13748753411275874) | > loader_time: 0.0047 (0.0047001022154503375)  --> TIME: 2025-02-26 22:54:18 -- STEP: 35556/36938 -- GLOBAL_STEP: 1439200 | > loss_text_ce: 0.04916461929678917 (0.030613226229685885) | > loss_mel_ce: 2.6531553268432617 (nan) | > loss: 0.010723492130637169 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13748272874918924) | > loader_time: 0.0046 (0.004700105649465541)  --> TIME: 2025-02-26 22:54:27 -- STEP: 35606/36938 -- GLOBAL_STEP: 1439250 | > loss_text_ce: 0.030822066590189934 (0.03061419410007965) | > loss_mel_ce: 2.845738172531128 (nan) | > loss: 0.011414921842515469 (nan) | > current_lr: 5e-06 | > step_time: 0.163 (0.13748490825125617) | > loader_time: 0.0053 (0.00470003287309958)  --> TIME: 2025-02-26 22:54:37 -- STEP: 35656/36938 -- GLOBAL_STEP: 1439300 | > loss_text_ce: 0.03520769625902176 (0.030614541074557692) | > loss_mel_ce: 2.6857895851135254 (nan) | > loss: 0.010797608643770218 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13748768985686102) | > loader_time: 0.0046 (0.004700044177962389)  --> TIME: 2025-02-26 22:54:46 -- STEP: 35706/36938 -- GLOBAL_STEP: 1439350 | > loss_text_ce: 0.03210112079977989 (0.030612417566187034) | > loss_mel_ce: 2.7336671352386475 (nan) | > loss: 0.010975271463394165 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.1374899346454675) | > loader_time: 0.0046 (0.004700159810206767)  --> TIME: 2025-02-26 22:54:55 -- STEP: 35756/36938 -- GLOBAL_STEP: 1439400 | > loss_text_ce: 0.02690085582435131 (0.030611577640235074) | > loss_mel_ce: 2.7659263610839844 (nan) | > loss: 0.011082648299634457 (nan) | > current_lr: 5e-06 | > step_time: 0.1624 (0.1374882207659358) | > loader_time: 0.0048 (0.004700080235407969)  --> TIME: 2025-02-26 22:55:05 -- STEP: 35806/36938 -- GLOBAL_STEP: 1439450 | > loss_text_ce: 0.034965332597494125 (0.030611060311676968) | > loss_mel_ce: 2.7029998302459717 (nan) | > loss: 0.010864941403269768 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13748561033541856) | > loader_time: 0.0046 (0.004700041653583525)  --> TIME: 2025-02-26 22:55:14 -- STEP: 35856/36938 -- GLOBAL_STEP: 1439500 | > loss_text_ce: 0.02230449952185154 (0.030609407665464924) | > loss_mel_ce: 2.702667236328125 (nan) | > loss: 0.010813380591571331 (nan) | > current_lr: 5e-06 | > step_time: 0.1717 (0.13748247887555287) | > loader_time: 0.0048 (0.004700152230869176)  --> TIME: 2025-02-26 22:55:24 -- STEP: 35906/36938 -- GLOBAL_STEP: 1439550 | > loss_text_ce: 0.03236513212323189 (0.030610237614126814) | > loss_mel_ce: 0.8528682589530945 (nan) | > loss: 0.0035128311719745398 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13748572979346374) | > loader_time: 0.0045 (0.004700139406457398)  --> TIME: 2025-02-26 22:55:33 -- STEP: 35956/36938 -- GLOBAL_STEP: 1439600 | > loss_text_ce: 0.026068484410643578 (0.030609190088820282) | > loss_mel_ce: 2.789999008178711 (nan) | > loss: 0.01117487158626318 (nan) | > current_lr: 5e-06 | > step_time: 0.187 (0.13748625215586072) | > loader_time: 0.0049 (0.004700193038863617)  --> TIME: 2025-02-26 22:55:42 -- STEP: 36006/36938 -- GLOBAL_STEP: 1439650 | > loss_text_ce: 0.0330013632774353 (0.030608877586309476) | > loss_mel_ce: 2.891291856765747 (nan) | > loss: 0.011604338884353638 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.1374818701376719) | > loader_time: 0.0046 (0.004700198045328101)  --> TIME: 2025-02-26 22:55:51 -- STEP: 36056/36938 -- GLOBAL_STEP: 1439700 | > loss_text_ce: 0.030396440997719765 (0.030608031605270647) | > loss_mel_ce: 1.928563117980957 (nan) | > loss: 0.007773649413138628 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.1374777776519986) | > loader_time: 0.0045 (0.004700110298264229)  --> TIME: 2025-02-26 22:56:01 -- STEP: 36106/36938 -- GLOBAL_STEP: 1439750 | > loss_text_ce: 0.03276051580905914 (0.030609218674909316) | > loss_mel_ce: 2.8816802501678467 (nan) | > loss: 0.01156524196267128 (nan) | > current_lr: 5e-06 | > step_time: 0.1871 (0.13748176718051447) | > loader_time: 0.0048 (0.004700127238547945)  --> TIME: 2025-02-26 22:56:10 -- STEP: 36156/36938 -- GLOBAL_STEP: 1439800 | > loss_text_ce: 0.029471533372998238 (0.030611241720909954) | > loss_mel_ce: 2.81982421875 (nan) | > loss: 0.011306730099022388 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.13748559174066644) | > loader_time: 0.0046 (0.0047002015671433)  --> TIME: 2025-02-26 22:56:20 -- STEP: 36206/36938 -- GLOBAL_STEP: 1439850 | > loss_text_ce: 0.02273344248533249 (0.030610227269139402) | > loss_mel_ce: 2.5904288291931152 (nan) | > loss: 0.010369691997766495 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13749237870640674) | > loader_time: 0.0048 (0.004700287589643335)  --> TIME: 2025-02-26 22:56:29 -- STEP: 36256/36938 -- GLOBAL_STEP: 1439900 | > loss_text_ce: 0.03434910997748375 (0.030612350548593403) | > loss_mel_ce: 3.275078058242798 (nan) | > loss: 0.013132648542523384 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13749220433323617) | > loader_time: 0.0047 (0.0047002279077182196)  --> TIME: 2025-02-26 22:56:38 -- STEP: 36306/36938 -- GLOBAL_STEP: 1439950 | > loss_text_ce: 0.034671515226364136 (0.03061077821280147) | > loss_mel_ce: 2.457697868347168 (nan) | > loss: 0.009890355169773102 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.13748664952622158) | > loader_time: 0.0046 (0.004700336227507346)  --> TIME: 2025-02-26 22:56:48 -- STEP: 36356/36938 -- GLOBAL_STEP: 1440000 | > loss_text_ce: 0.027151580899953842 (0.030612440543529094) | > loss_mel_ce: 3.044037103652954 (nan) | > loss: 0.012187257409095764 (nan) | > current_lr: 5e-06 | > step_time: 0.1844 (0.13748363958542034) | > loader_time: 0.0049 (0.004700302212056521) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1440000.pth  --> TIME: 2025-02-26 22:57:03 -- STEP: 36406/36938 -- GLOBAL_STEP: 1440050 | > loss_text_ce: 0.029203908517956734 (0.030612762392624573) | > loss_mel_ce: 2.728994369506836 (nan) | > loss: 0.010945231653749943 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.1374884978012882) | > loader_time: 0.0048 (0.004700401441897553)  --> TIME: 2025-02-26 22:57:12 -- STEP: 36456/36938 -- GLOBAL_STEP: 1440100 | > loss_text_ce: 0.03449120372533798 (0.03061026258522928) | > loss_mel_ce: 2.4104623794555664 (nan) | > loss: 0.009702197276055813 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13748029997952102) | > loader_time: 0.0046 (0.004700334351497543)  --> TIME: 2025-02-26 22:57:21 -- STEP: 36506/36938 -- GLOBAL_STEP: 1440150 | > loss_text_ce: 0.01882624253630638 (0.03061113053656395) | > loss_mel_ce: 2.8212592601776123 (nan) | > loss: 0.011270181275904179 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13747748579253047) | > loader_time: 0.0045 (0.0047003021894895984)  --> TIME: 2025-02-26 22:57:31 -- STEP: 36556/36938 -- GLOBAL_STEP: 1440200 | > loss_text_ce: 0.02659749612212181 (0.030610701775459888) | > loss_mel_ce: 0.9561038613319397 (nan) | > loss: 0.003899608738720417 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1374782395253292) | > loader_time: 0.0045 (0.004700235940132004)  --> TIME: 2025-02-26 22:57:40 -- STEP: 36606/36938 -- GLOBAL_STEP: 1440250 | > loss_text_ce: 0.03267568349838257 (0.030611608457855738) | > loss_mel_ce: 2.7735092639923096 (nan) | > loss: 0.011135655455291271 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.13747895166456256) | > loader_time: 0.0053 (0.004700308581268181)  --> TIME: 2025-02-26 22:57:49 -- STEP: 36656/36938 -- GLOBAL_STEP: 1440300 | > loss_text_ce: 0.02298089489340782 (0.03061003654226261) | > loss_mel_ce: 2.6850178241729736 (nan) | > loss: 0.01074602734297514 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.13747414046372536) | > loader_time: 0.0055 (0.004700484493318881)  --> TIME: 2025-02-26 22:57:59 -- STEP: 36706/36938 -- GLOBAL_STEP: 1440350 | > loss_text_ce: 0.026210656389594078 (0.03061028717643965) | > loss_mel_ce: 2.309619903564453 (nan) | > loss: 0.009269168600440025 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13747826054759735) | > loader_time: 0.0045 (0.004700591692396811)  --> TIME: 2025-02-26 22:58:08 -- STEP: 36756/36938 -- GLOBAL_STEP: 1440400 | > loss_text_ce: 0.032899461686611176 (0.0306090199634114) | > loss_mel_ce: 2.49757981300354 (nan) | > loss: 0.010041584260761738 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.1374725454428841) | > loader_time: 0.0045 (0.004700664597479218)  --> TIME: 2025-02-26 22:58:17 -- STEP: 36806/36938 -- GLOBAL_STEP: 1440450 | > loss_text_ce: 0.03199717774987221 (0.030606980461802654) | > loss_mel_ce: 2.535061836242676 (nan) | > loss: 0.01018674299120903 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.13747606746840757) | > loader_time: 0.0046 (0.004700695063338055)  --> TIME: 2025-02-26 22:58:27 -- STEP: 36856/36938 -- GLOBAL_STEP: 1440500 | > loss_text_ce: 0.03607810288667679 (0.030608896286607776) | > loss_mel_ce: 1.734375 (nan) | > loss: 0.0070256078615784645 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13748145680482507) | > loader_time: 0.0045 (0.004700717023998937)  --> TIME: 2025-02-26 22:58:36 -- STEP: 36906/36938 -- GLOBAL_STEP: 1440550 | > loss_text_ce: 0.032533373683691025 (0.030608675572432194) | > loss_mel_ce: 1.8541855812072754 (nan) | > loss: 0.0074869804084300995 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13747844976803478) | > loader_time: 0.0046 (0.004700602880695058)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.02089019695130906 (-3.6474501732556425e-05) | > avg_loss_text_ce: 0.02986540448031213 (-0.00013097252069723536) | > avg_loss_mel_ce: 2.44703897096143 (-0.0012269459738591948) | > avg_loss: 2.476904373357794 (-0.0013579180335012708) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_1440582.pth  > EPOCH: 40/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-26 22:59:09)   --> TIME: 2025-02-26 22:59:13 -- STEP: 18/36938 -- GLOBAL_STEP: 1440600 | > loss_text_ce: 0.031032932922244072 (0.0326801562267873) | > loss_mel_ce: 2.698741912841797 (nan) | > loss: 0.010832441039383411 (nan) | > current_lr: 5e-06 | > step_time: 0.1355 (0.13938479953342015) | > loader_time: 0.0049 (0.004887726571824815)  --> TIME: 2025-02-26 22:59:22 -- STEP: 68/36938 -- GLOBAL_STEP: 1440650 | > loss_text_ce: 0.030870193615555763 (0.031309163033523976) | > loss_mel_ce: 2.142707347869873 (nan) | > loss: 0.00862530805170536 (nan) | > current_lr: 5e-06 | > step_time: 0.1123 (0.1369658848818611) | > loader_time: 0.0046 (0.004814168986152202)  --> TIME: 2025-02-26 22:59:31 -- STEP: 118/36938 -- GLOBAL_STEP: 1440700 | > loss_text_ce: 0.040033191442489624 (0.031102965743738716) | > loss_mel_ce: 2.115234375 (nan) | > loss: 0.008552649058401585 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13732175907846225) | > loader_time: 0.0047 (0.004797878911939719)  --> TIME: 2025-02-26 22:59:40 -- STEP: 168/36938 -- GLOBAL_STEP: 1440750 | > loss_text_ce: 0.035718511790037155 (0.031042544747747126) | > loss_mel_ce: 3.1057512760162354 (nan) | > loss: 0.01246615033596754 (nan) | > current_lr: 5e-06 | > step_time: 0.1626 (0.13639277787435616) | > loader_time: 0.0047 (0.00477589737801325)  --> TIME: 2025-02-26 22:59:50 -- STEP: 218/36938 -- GLOBAL_STEP: 1440800 | > loss_text_ce: 0.02792607806622982 (0.03096893042274708) | > loss_mel_ce: 0.0364786796271801 (nan) | > loss: 0.0002555744431447238 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13610235266729218) | > loader_time: 0.0047 (0.00476819659591815)  --> TIME: 2025-02-26 22:59:59 -- STEP: 268/36938 -- GLOBAL_STEP: 1440850 | > loss_text_ce: 0.03276713192462921 (0.03098125712348343) | > loss_mel_ce: 2.990217447280884 (nan) | > loss: 0.011995971202850342 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.13635976012073334) | > loader_time: 0.0049 (0.004761575762905293)  --> TIME: 2025-02-26 23:00:09 -- STEP: 318/36938 -- GLOBAL_STEP: 1440900 | > loss_text_ce: 0.02885359153151512 (0.03078096961813434) | > loss_mel_ce: 2.3734195232391357 (nan) | > loss: 0.00953283067792654 (nan) | > current_lr: 5e-06 | > step_time: 0.2001 (0.13691701454186575) | > loader_time: 0.0049 (0.004765117693247284)  --> TIME: 2025-02-26 23:00:18 -- STEP: 368/36938 -- GLOBAL_STEP: 1440950 | > loss_text_ce: 0.030737873166799545 (0.030613502297732655) | > loss_mel_ce: 2.796149730682373 (nan) | > loss: 0.011217808350920677 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13709594114966997) | > loader_time: 0.0048 (0.004763984809751094)  --> TIME: 2025-02-26 23:00:27 -- STEP: 418/36938 -- GLOBAL_STEP: 1441000 | > loss_text_ce: 0.03037339821457863 (0.03061171073653695) | > loss_mel_ce: 2.3183887004852295 (nan) | > loss: 0.009320484474301338 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13713692820243278) | > loader_time: 0.0046 (0.0047692699295481976)  --> TIME: 2025-02-26 23:00:37 -- STEP: 468/36938 -- GLOBAL_STEP: 1441050 | > loss_text_ce: 0.0253137294203043 (0.03065839019190106) | > loss_mel_ce: 1.2094426155090332 (nan) | > loss: 0.0048998272977769375 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.137875190657428) | > loader_time: 0.0045 (0.004768424054496307)  --> TIME: 2025-02-26 23:00:46 -- STEP: 518/36938 -- GLOBAL_STEP: 1441100 | > loss_text_ce: 0.031013166531920433 (0.030640600067581333) | > loss_mel_ce: 3.006472587585449 (nan) | > loss: 0.012053515762090683 (nan) | > current_lr: 5e-06 | > step_time: 0.1943 (0.13782953295468348) | > loader_time: 0.005 (0.0047672176913405)  --> TIME: 2025-02-26 23:00:56 -- STEP: 568/36938 -- GLOBAL_STEP: 1441150 | > loss_text_ce: 0.030233241617679596 (0.030665054358214035) | > loss_mel_ce: 3.374284029006958 (nan) | > loss: 0.013509989716112614 (nan) | > current_lr: 5e-06 | > step_time: 0.1619 (0.13767519635213918) | > loader_time: 0.0048 (0.004768824913132356)  --> TIME: 2025-02-26 23:01:05 -- STEP: 618/36938 -- GLOBAL_STEP: 1441200 | > loss_text_ce: 0.029713036492466927 (0.030714262838996826) | > loss_mel_ce: 2.623363494873047 (nan) | > loss: 0.010528082959353924 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.1381092152549224) | > loader_time: 0.0048 (0.0047745758661560216)  --> TIME: 2025-02-26 23:01:15 -- STEP: 668/36938 -- GLOBAL_STEP: 1441250 | > loss_text_ce: 0.02544892020523548 (0.030707812158413186) | > loss_mel_ce: 2.6955201625823975 (nan) | > loss: 0.010797497816383839 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13797765399167625) | > loader_time: 0.0047 (0.004777697388997335)  --> TIME: 2025-02-26 23:01:24 -- STEP: 718/36938 -- GLOBAL_STEP: 1441300 | > loss_text_ce: 0.03280102089047432 (0.030708364323871133) | > loss_mel_ce: 3.0111327171325684 (nan) | > loss: 0.012079101987183094 (nan) | > current_lr: 5e-06 | > step_time: 0.1886 (0.1382422988461251) | > loader_time: 0.0049 (0.004787711379919877)  --> TIME: 2025-02-26 23:01:34 -- STEP: 768/36938 -- GLOBAL_STEP: 1441350 | > loss_text_ce: 0.03139641508460045 (0.030663909726475442) | > loss_mel_ce: 1.8044888973236084 (nan) | > loss: 0.007285259664058685 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.138334929011762) | > loader_time: 0.0048 (0.004787342001994454)  --> TIME: 2025-02-26 23:01:43 -- STEP: 818/36938 -- GLOBAL_STEP: 1441400 | > loss_text_ce: 0.02709316276013851 (0.030680207473005015) | > loss_mel_ce: 1.1609104871749878 (nan) | > loss: 0.004714300390332937 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.13808563752395883) | > loader_time: 0.0047 (0.00478118672639059)  --> TIME: 2025-02-26 23:01:53 -- STEP: 868/36938 -- GLOBAL_STEP: 1441450 | > loss_text_ce: 0.025502463802695274 (0.030601715901866555) | > loss_mel_ce: 2.355659246444702 (nan) | > loss: 0.00944905448704958 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.1382438963459384) | > loader_time: 0.0047 (0.004786458982300653)  --> TIME: 2025-02-26 23:02:02 -- STEP: 918/36938 -- GLOBAL_STEP: 1441500 | > loss_text_ce: 0.03293878585100174 (0.030651631138098785) | > loss_mel_ce: 1.7573585510253906 (nan) | > loss: 0.007104354910552502 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13815438176032505) | > loader_time: 0.0046 (0.004787419096836601)  --> TIME: 2025-02-26 23:02:11 -- STEP: 968/36938 -- GLOBAL_STEP: 1441550 | > loss_text_ce: 0.03347298130393028 (0.03063254082413806) | > loss_mel_ce: 1.5003994703292847 (nan) | > loss: 0.006086796056479216 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.1376178508455103) | > loader_time: 0.0046 (0.00478275920733933)  --> TIME: 2025-02-26 23:02:20 -- STEP: 1018/36938 -- GLOBAL_STEP: 1441600 | > loss_text_ce: 0.01968383602797985 (0.030656254911623267) | > loss_mel_ce: 2.047725200653076 (nan) | > loss: 0.008204004727303982 (nan) | > current_lr: 5e-06 | > step_time: 0.1355 (0.13761179198918969) | > loader_time: 0.0046 (0.0047778202648949795)  --> TIME: 2025-02-26 23:02:30 -- STEP: 1068/36938 -- GLOBAL_STEP: 1441650 | > loss_text_ce: 0.02989030070602894 (0.030667689678604514) | > loss_mel_ce: 2.829416513442993 (nan) | > loss: 0.011346456594765186 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13771760753924478) | > loader_time: 0.0047 (0.00477646754475569)  --> TIME: 2025-02-26 23:02:39 -- STEP: 1118/36938 -- GLOBAL_STEP: 1441700 | > loss_text_ce: 0.03569229319691658 (0.030705787892051854) | > loss_mel_ce: 2.7390506267547607 (nan) | > loss: 0.011010885238647461 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13782852067077306) | > loader_time: 0.0046 (0.0047736065545534246)  --> TIME: 2025-02-26 23:02:49 -- STEP: 1168/36938 -- GLOBAL_STEP: 1441750 | > loss_text_ce: 0.031462814658880234 (0.03071079454592059) | > loss_mel_ce: 2.3948469161987305 (nan) | > loss: 0.009628213942050934 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1380617014349324) | > loader_time: 0.0049 (0.004772705574558213)  --> TIME: 2025-02-26 23:02:59 -- STEP: 1218/36938 -- GLOBAL_STEP: 1441800 | > loss_text_ce: 0.02792232297360897 (0.030630111195795447) | > loss_mel_ce: 2.819516181945801 (nan) | > loss: 0.011299359612166882 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.13828930302793763) | > loader_time: 0.0047 (0.004770602107243785)  --> TIME: 2025-02-26 23:03:08 -- STEP: 1268/36938 -- GLOBAL_STEP: 1441850 | > loss_text_ce: 0.024798301979899406 (0.030615197199364473) | > loss_mel_ce: 2.7266507148742676 (nan) | > loss: 0.010918449610471725 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13839548769809484) | > loader_time: 0.0047 (0.0047681705808790015)  --> TIME: 2025-02-26 23:03:18 -- STEP: 1318/36938 -- GLOBAL_STEP: 1441900 | > loss_text_ce: 0.03427477553486824 (0.030616260104005723) | > loss_mel_ce: 2.9254209995269775 (nan) | > loss: 0.011744825169444084 (nan) | > current_lr: 5e-06 | > step_time: 0.1438 (0.13855209082862507) | > loader_time: 0.0048 (0.004767617253142898)  --> TIME: 2025-02-26 23:03:27 -- STEP: 1368/36938 -- GLOBAL_STEP: 1441950 | > loss_text_ce: 0.03734882175922394 (0.03060236742736347) | > loss_mel_ce: 2.4925248622894287 (nan) | > loss: 0.01003918144851923 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13836190522762795) | > loader_time: 0.0047 (0.004765405466682038)  --> TIME: 2025-02-26 23:03:37 -- STEP: 1418/36938 -- GLOBAL_STEP: 1442000 | > loss_text_ce: 0.03285038471221924 (0.0305837619693737) | > loss_mel_ce: 3.2248611450195312 (nan) | > loss: 0.01292742695659399 (nan) | > current_lr: 5e-06 | > step_time: 0.1632 (0.13855413607044556) | > loader_time: 0.0048 (0.00476433578432027)  --> TIME: 2025-02-26 23:03:46 -- STEP: 1468/36938 -- GLOBAL_STEP: 1442050 | > loss_text_ce: 0.023800358176231384 (0.030572156414861617) | > loss_mel_ce: 2.8761584758758545 (nan) | > loss: 0.011507773771882057 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13864563563864016) | > loader_time: 0.0047 (0.004763034286551002)  --> TIME: 2025-02-26 23:03:56 -- STEP: 1518/36938 -- GLOBAL_STEP: 1442100 | > loss_text_ce: 0.03208240121603012 (0.030550443729918268) | > loss_mel_ce: 2.564476251602173 (nan) | > loss: 0.010303804650902748 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13870949503304356) | > loader_time: 0.0046 (0.004761415350892646)  --> TIME: 2025-02-26 23:04:05 -- STEP: 1568/36938 -- GLOBAL_STEP: 1442150 | > loss_text_ce: 0.035272013396024704 (0.030522100136220948) | > loss_mel_ce: 2.9032983779907227 (nan) | > loss: 0.011660994961857796 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.13867227717930009) | > loader_time: 0.0047 (0.004763462257628543)  --> TIME: 2025-02-26 23:04:15 -- STEP: 1618/36938 -- GLOBAL_STEP: 1442200 | > loss_text_ce: 0.034677162766456604 (0.03056395510372755) | > loss_mel_ce: 3.0367431640625 (nan) | > loss: 0.01218817662447691 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13869487841432904) | > loader_time: 0.0047 (0.004762029913064137)  --> TIME: 2025-02-26 23:04:24 -- STEP: 1668/36938 -- GLOBAL_STEP: 1442250 | > loss_text_ce: 0.038442209362983704 (0.030550625310915944) | > loss_mel_ce: 3.046029806137085 (nan) | > loss: 0.012239968404173851 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.1385794949474385) | > loader_time: 0.0049 (0.004761033778567967)  --> TIME: 2025-02-26 23:04:33 -- STEP: 1718/36938 -- GLOBAL_STEP: 1442300 | > loss_text_ce: 0.023612912744283676 (0.03056351259313914) | > loss_mel_ce: 2.4374592304229736 (nan) | > loss: 0.009766160510480404 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.1385498044099468) | > loader_time: 0.0047 (0.004760896090994894)  --> TIME: 2025-02-26 23:04:43 -- STEP: 1768/36938 -- GLOBAL_STEP: 1442350 | > loss_text_ce: 0.016335587948560715 (0.030568069265839302) | > loss_mel_ce: 2.423231840133667 (nan) | > loss: 0.009680823422968388 (nan) | > current_lr: 5e-06 | > step_time: 0.1844 (0.1384988605166995) | > loader_time: 0.0048 (0.00476083456121419)  --> TIME: 2025-02-26 23:04:52 -- STEP: 1818/36938 -- GLOBAL_STEP: 1442400 | > loss_text_ce: 0.017877284437417984 (0.030575160964262008) | > loss_mel_ce: 1.9639232158660889 (nan) | > loss: 0.007864288054406643 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.1386292598428508) | > loader_time: 0.0045 (0.004761138240365176)  --> TIME: 2025-02-26 23:05:02 -- STEP: 1868/36938 -- GLOBAL_STEP: 1442450 | > loss_text_ce: 0.031572550535202026 (0.030552604001880937) | > loss_mel_ce: 2.858417272567749 (nan) | > loss: 0.011468213982880116 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.138662440363294) | > loader_time: 0.0047 (0.0047606231058435235)  --> TIME: 2025-02-26 23:05:11 -- STEP: 1918/36938 -- GLOBAL_STEP: 1442500 | > loss_text_ce: 0.027708835899829865 (0.0305730639869489) | > loss_mel_ce: 2.279850721359253 (nan) | > loss: 0.00915698241442442 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13868509542208662) | > loader_time: 0.0047 (0.004760137439644252)  --> TIME: 2025-02-26 23:05:21 -- STEP: 1968/36938 -- GLOBAL_STEP: 1442550 | > loss_text_ce: 0.03633245825767517 (0.030560436148889272) | > loss_mel_ce: 2.851639747619629 (nan) | > loss: 0.011460207402706146 (nan) | > current_lr: 5e-06 | > step_time: 0.1527 (0.13868860185630943) | > loader_time: 0.0047 (0.004760219677677001)  --> TIME: 2025-02-26 23:05:31 -- STEP: 2018/36938 -- GLOBAL_STEP: 1442600 | > loss_text_ce: 0.023796478286385536 (0.03055325978019296) | > loss_mel_ce: 1.7842568159103394 (nan) | > loss: 0.00717481505125761 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13876181244259447) | > loader_time: 0.0046 (0.004759635041561071)  --> TIME: 2025-02-26 23:05:40 -- STEP: 2068/36938 -- GLOBAL_STEP: 1442650 | > loss_text_ce: 0.030050698667764664 (0.030558374345147663) | > loss_mel_ce: 2.7692978382110596 (nan) | > loss: 0.011108526960015297 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13869774157120154) | > loader_time: 0.0048 (0.004758792181771315)  --> TIME: 2025-02-26 23:05:49 -- STEP: 2118/36938 -- GLOBAL_STEP: 1442700 | > loss_text_ce: 0.03672463819384575 (0.030570373029251977) | > loss_mel_ce: 2.8812689781188965 (nan) | > loss: 0.011579340323805809 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1386801523347303) | > loader_time: 0.0046 (0.004758618708710497)  --> TIME: 2025-02-26 23:05:59 -- STEP: 2168/36938 -- GLOBAL_STEP: 1442750 | > loss_text_ce: 0.02242603525519371 (0.03058725026843389) | > loss_mel_ce: 2.923128366470337 (nan) | > loss: 0.011688709259033203 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13866945166429484) | > loader_time: 0.0048 (0.004758259687036606)  --> TIME: 2025-02-26 23:06:08 -- STEP: 2218/36938 -- GLOBAL_STEP: 1442800 | > loss_text_ce: 0.022289235144853592 (0.030588077192721306) | > loss_mel_ce: 0.0012265119003131986 (nan) | > loss: 9.331646288046613e-05 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.1386334121066027) | > loader_time: 0.0046 (0.004758132697226659)  --> TIME: 2025-02-26 23:06:17 -- STEP: 2268/36938 -- GLOBAL_STEP: 1442850 | > loss_text_ce: 0.02954156883060932 (0.03058884217653496) | > loss_mel_ce: 2.299877166748047 (nan) | > loss: 0.009243725799024105 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13849477898296655) | > loader_time: 0.0046 (0.004757408742551451)  --> TIME: 2025-02-26 23:06:27 -- STEP: 2318/36938 -- GLOBAL_STEP: 1442900 | > loss_text_ce: 0.03155438229441643 (0.030590002958978606) | > loss_mel_ce: 3.213484764099121 (nan) | > loss: 0.012877140194177628 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13852124506281419) | > loader_time: 0.0047 (0.0047575435934650995)  --> TIME: 2025-02-26 23:06:36 -- STEP: 2368/36938 -- GLOBAL_STEP: 1442950 | > loss_text_ce: 0.028863154351711273 (0.030595882272275443) | > loss_mel_ce: 2.860107421875 (nan) | > loss: 0.011464170180261135 (nan) | > current_lr: 5e-06 | > step_time: 0.1507 (0.13861661072115658) | > loader_time: 0.0046 (0.004759149474872123)  --> TIME: 2025-02-26 23:06:46 -- STEP: 2418/36938 -- GLOBAL_STEP: 1443000 | > loss_text_ce: 0.03082961216568947 (0.030597502047179082) | > loss_mel_ce: 2.609717845916748 (nan) | > loss: 0.010478363372385502 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.1386076383496042) | > loader_time: 0.0046 (0.00475839842834977)  --> TIME: 2025-02-26 23:06:55 -- STEP: 2468/36938 -- GLOBAL_STEP: 1443050 | > loss_text_ce: 0.027728505432605743 (0.030595439718281502) | > loss_mel_ce: 2.8619515895843506 (nan) | > loss: 0.011466985568404198 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13866375282865692) | > loader_time: 0.0047 (0.004757736162003276)  --> TIME: 2025-02-26 23:07:05 -- STEP: 2518/36938 -- GLOBAL_STEP: 1443100 | > loss_text_ce: 0.02477765828371048 (0.030613459965156433) | > loss_mel_ce: 2.5143496990203857 (nan) | > loss: 0.010075902566313744 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13860311746786683) | > loader_time: 0.0047 (0.004756586242990893)  --> TIME: 2025-02-26 23:07:14 -- STEP: 2568/36938 -- GLOBAL_STEP: 1443150 | > loss_text_ce: 0.027365293353796005 (0.030623074088207913) | > loss_mel_ce: 2.9261410236358643 (nan) | > loss: 0.01172026339918375 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13861121622572825) | > loader_time: 0.0046 (0.004755210746485864)  --> TIME: 2025-02-26 23:07:23 -- STEP: 2618/36938 -- GLOBAL_STEP: 1443200 | > loss_text_ce: 0.021924734115600586 (0.030607285218155388) | > loss_mel_ce: 2.693979501724243 (nan) | > loss: 0.010777398012578487 (nan) | > current_lr: 5e-06 | > step_time: 0.1262 (0.13858596134768644) | > loader_time: 0.0046 (0.0047544514158465966)  --> TIME: 2025-02-26 23:07:33 -- STEP: 2668/36938 -- GLOBAL_STEP: 1443250 | > loss_text_ce: 0.03625369071960449 (0.030589579108535596) | > loss_mel_ce: 3.2219953536987305 (nan) | > loss: 0.012929560616612434 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.13860605225927655) | > loader_time: 0.0046 (0.004753666839142073)  --> TIME: 2025-02-26 23:07:42 -- STEP: 2718/36938 -- GLOBAL_STEP: 1443300 | > loss_text_ce: 0.03347253054380417 (0.030581393062237807) | > loss_mel_ce: 2.2423455715179443 (nan) | > loss: 0.009031024761497974 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.13852573791964265) | > loader_time: 0.0047 (0.004752654201235293)  --> TIME: 2025-02-26 23:07:52 -- STEP: 2768/36938 -- GLOBAL_STEP: 1443350 | > loss_text_ce: 0.02158723585307598 (0.030556458891355946) | > loss_mel_ce: 0.2889229953289032 (nan) | > loss: 0.0012321834219619632 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13864789644762285) | > loader_time: 0.0045 (0.004751705279240029)  --> TIME: 2025-02-26 23:08:02 -- STEP: 2818/36938 -- GLOBAL_STEP: 1443400 | > loss_text_ce: 0.03522367775440216 (0.03055112760195428) | > loss_mel_ce: 2.603590250015259 (nan) | > loss: 0.010471484623849392 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.13865562996346548) | > loader_time: 0.0049 (0.00475127346723785)  --> TIME: 2025-02-26 23:08:11 -- STEP: 2868/36938 -- GLOBAL_STEP: 1443450 | > loss_text_ce: 0.03551275283098221 (0.030556544905393338) | > loss_mel_ce: 2.6509511470794678 (nan) | > loss: 0.010660571046173573 (nan) | > current_lr: 5e-06 | > step_time: 0.1679 (0.1386892404017589) | > loader_time: 0.0047 (0.004750361169897524)  --> TIME: 2025-02-26 23:08:21 -- STEP: 2918/36938 -- GLOBAL_STEP: 1443500 | > loss_text_ce: 0.03433660417795181 (0.03056348034967761) | > loss_mel_ce: 2.103595495223999 (nan) | > loss: 0.008483857847750187 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13867021873440144) | > loader_time: 0.0047 (0.004750077572483324)  --> TIME: 2025-02-26 23:08:30 -- STEP: 2968/36938 -- GLOBAL_STEP: 1443550 | > loss_text_ce: 0.028114106506109238 (0.030559183929286235) | > loss_mel_ce: 2.9510273933410645 (nan) | > loss: 0.011821990832686424 (nan) | > current_lr: 5e-06 | > step_time: 0.1705 (0.13871343212950274) | > loader_time: 0.0047 (0.004748950268059088)  --> TIME: 2025-02-26 23:08:39 -- STEP: 3018/36938 -- GLOBAL_STEP: 1443600 | > loss_text_ce: 0.022488610818982124 (0.03055244631018561) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13866916296891305) | > loader_time: 0.0046 (0.004747901711738848)  --> TIME: 2025-02-26 23:08:49 -- STEP: 3068/36938 -- GLOBAL_STEP: 1443650 | > loss_text_ce: 0.03102097660303116 (0.030545657158512356) | > loss_mel_ce: 3.1359379291534424 (nan) | > loss: 0.01256729755550623 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13870660702283882) | > loader_time: 0.0046 (0.004746943207110392)  --> TIME: 2025-02-26 23:08:58 -- STEP: 3118/36938 -- GLOBAL_STEP: 1443700 | > loss_text_ce: 0.030382422730326653 (0.03058002510683335) | > loss_mel_ce: 2.4232828617095947 (nan) | > loss: 0.009736767038702965 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13863747642925744) | > loader_time: 0.0047 (0.004745853184277313)  --> TIME: 2025-02-26 23:09:08 -- STEP: 3168/36938 -- GLOBAL_STEP: 1443750 | > loss_text_ce: 0.03043653443455696 (0.030586020194048137) | > loss_mel_ce: 2.0875332355499268 (nan) | > loss: 0.008404642343521118 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1386443664780773) | > loader_time: 0.0047 (0.00474518620305591)  --> TIME: 2025-02-26 23:09:17 -- STEP: 3218/36938 -- GLOBAL_STEP: 1443800 | > loss_text_ce: 0.030054088681936264 (0.030599772313990584) | > loss_mel_ce: 2.7643847465515137 (nan) | > loss: 0.011089043691754341 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.1386307373485484) | > loader_time: 0.0047 (0.004744255224172191)  --> TIME: 2025-02-26 23:09:26 -- STEP: 3268/36938 -- GLOBAL_STEP: 1443850 | > loss_text_ce: 0.037139132618904114 (0.030596316971324814) | > loss_mel_ce: 2.5842390060424805 (nan) | > loss: 0.010402294807136059 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13858935634538358) | > loader_time: 0.0045 (0.0047430436935098)  --> TIME: 2025-02-26 23:09:35 -- STEP: 3318/36938 -- GLOBAL_STEP: 1443900 | > loss_text_ce: 0.031225105747580528 (0.030574975665793442) | > loss_mel_ce: 1.769242763519287 (nan) | > loss: 0.007144714239984751 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13853980914306216) | > loader_time: 0.0046 (0.004744107928744564)  --> TIME: 2025-02-26 23:09:45 -- STEP: 3368/36938 -- GLOBAL_STEP: 1443950 | > loss_text_ce: 0.028272219002246857 (0.030578712287089055) | > loss_mel_ce: 2.8269026279449463 (nan) | > loss: 0.011330059729516506 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.13851368830776015) | > loader_time: 0.005 (0.0047435267096177436)  --> TIME: 2025-02-26 23:09:54 -- STEP: 3418/36938 -- GLOBAL_STEP: 1444000 | > loss_text_ce: 0.03470607101917267 (0.03057062165809478) | > loss_mel_ce: 2.9542922973632812 (nan) | > loss: 0.0118611054494977 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13850268841905888) | > loader_time: 0.0047 (0.004743610787210304)  --> TIME: 2025-02-26 23:10:04 -- STEP: 3468/36938 -- GLOBAL_STEP: 1444050 | > loss_text_ce: 0.03340886905789375 (0.03057542573977498) | > loss_mel_ce: 2.4740185737609863 (nan) | > loss: 0.009950109757483006 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13850174601927798) | > loader_time: 0.0046 (0.004743633729501306)  --> TIME: 2025-02-26 23:10:13 -- STEP: 3518/36938 -- GLOBAL_STEP: 1444100 | > loss_text_ce: 0.02432403527200222 (0.030565783617337142) | > loss_mel_ce: 1.7587019205093384 (nan) | > loss: 0.007075500208884478 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13844182479103728) | > loader_time: 0.0046 (0.004743171456593423)  --> TIME: 2025-02-26 23:10:22 -- STEP: 3568/36938 -- GLOBAL_STEP: 1444150 | > loss_text_ce: 0.03337526321411133 (0.030573256375140724) | > loss_mel_ce: 1.782382845878601 (nan) | > loss: 0.007205389905720949 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13845716122821874) | > loader_time: 0.0046 (0.004743287047463149)  --> TIME: 2025-02-26 23:10:32 -- STEP: 3618/36938 -- GLOBAL_STEP: 1444200 | > loss_text_ce: 0.022356858476996422 (0.030559043372703758) | > loss_mel_ce: 2.76082444190979 (nan) | > loss: 0.011044370010495186 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.13852119742341346) | > loader_time: 0.0047 (0.004742531436395093)  --> TIME: 2025-02-26 23:10:41 -- STEP: 3668/36938 -- GLOBAL_STEP: 1444250 | > loss_text_ce: 0.04299992322921753 (0.03054982014437132) | > loss_mel_ce: 1.2473645210266113 (nan) | > loss: 0.005120494402945042 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13851136684157775) | > loader_time: 0.0046 (0.004741700160984386)  --> TIME: 2025-02-26 23:10:51 -- STEP: 3718/36938 -- GLOBAL_STEP: 1444300 | > loss_text_ce: 0.025357317179441452 (0.03057152836625136) | > loss_mel_ce: 2.5462801456451416 (nan) | > loss: 0.010204911231994629 (nan) | > current_lr: 5e-06 | > step_time: 0.1927 (0.13856839473174487) | > loader_time: 0.0049 (0.004740918432915963)  --> TIME: 2025-02-26 23:11:00 -- STEP: 3768/36938 -- GLOBAL_STEP: 1444350 | > loss_text_ce: 0.02998206950724125 (0.030581165185680655) | > loss_mel_ce: 3.076711654663086 (nan) | > loss: 0.012328150682151318 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13852812783227159) | > loader_time: 0.0046 (0.004739666946880885)  --> TIME: 2025-02-26 23:11:10 -- STEP: 3818/36938 -- GLOBAL_STEP: 1444400 | > loss_text_ce: 0.029918786138296127 (0.030567140205757914) | > loss_mel_ce: 0.0752410888671875 (nan) | > loss: 0.00041730114025995135 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13853863192329047) | > loader_time: 0.0047 (0.004738652125399048)  --> TIME: 2025-02-26 23:11:19 -- STEP: 3868/36938 -- GLOBAL_STEP: 1444450 | > loss_text_ce: 0.03168221190571785 (0.030571595477200314) | > loss_mel_ce: 2.8521645069122314 (nan) | > loss: 0.011443836614489555 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13856362834218428) | > loader_time: 0.0047 (0.004737940421395368)  --> TIME: 2025-02-26 23:11:28 -- STEP: 3918/36938 -- GLOBAL_STEP: 1444500 | > loss_text_ce: 0.024583665654063225 (0.030568192572242116) | > loss_mel_ce: 3.29358172416687 (nan) | > loss: 0.013167323544621468 (nan) | > current_lr: 5e-06 | > step_time: 0.1817 (0.13852136849505184) | > loader_time: 0.0049 (0.004737005542893385)  --> TIME: 2025-02-26 23:11:38 -- STEP: 3968/36938 -- GLOBAL_STEP: 1444550 | > loss_text_ce: 0.030832258984446526 (0.030558753577474083) | > loss_mel_ce: 2.21110463142395 (nan) | > loss: 0.008896575309336185 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.1385299654136748) | > loader_time: 0.0046 (0.004735934758378615)  --> TIME: 2025-02-26 23:11:47 -- STEP: 4018/36938 -- GLOBAL_STEP: 1444600 | > loss_text_ce: 0.032081663608551025 (0.030570420077290328) | > loss_mel_ce: 2.51727557182312 (nan) | > loss: 0.010116497054696083 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.1385296084739611) | > loader_time: 0.0051 (0.004736072569714667)  --> TIME: 2025-02-26 23:11:57 -- STEP: 4068/36938 -- GLOBAL_STEP: 1444650 | > loss_text_ce: 0.03126610070466995 (0.030573139828423745) | > loss_mel_ce: 2.6512036323547363 (nan) | > loss: 0.010644721798598766 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13856905593281355) | > loader_time: 0.0047 (0.004735520221256344)  --> TIME: 2025-02-26 23:12:07 -- STEP: 4118/36938 -- GLOBAL_STEP: 1444700 | > loss_text_ce: 0.026787851005792618 (0.030583776800471234) | > loss_mel_ce: 2.1331851482391357 (nan) | > loss: 0.00857132114470005 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.13862076924228214) | > loader_time: 0.0045 (0.004735216346396122)  --> TIME: 2025-02-26 23:12:16 -- STEP: 4168/36938 -- GLOBAL_STEP: 1444750 | > loss_text_ce: 0.026997962966561317 (0.0305809139434048) | > loss_mel_ce: 3.1029462814331055 (nan) | > loss: 0.012420414946973324 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13855364175080853) | > loader_time: 0.0048 (0.004734551723538799)  --> TIME: 2025-02-26 23:12:25 -- STEP: 4218/36938 -- GLOBAL_STEP: 1444800 | > loss_text_ce: 0.03554116562008858 (0.030576159964739967) | > loss_mel_ce: 3.3225836753845215 (nan) | > loss: 0.013325892388820648 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.13858162887631173) | > loader_time: 0.0049 (0.0047343794120437675)  --> TIME: 2025-02-26 23:12:35 -- STEP: 4268/36938 -- GLOBAL_STEP: 1444850 | > loss_text_ce: 0.027773942798376083 (0.030573130467419548) | > loss_mel_ce: 2.2482657432556152 (nan) | > loss: 0.009031903930008411 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.1385916177089128) | > loader_time: 0.0045 (0.004734168738657443)  --> TIME: 2025-02-26 23:12:44 -- STEP: 4318/36938 -- GLOBAL_STEP: 1444900 | > loss_text_ce: 0.038193292915821075 (0.030575863145215886) | > loss_mel_ce: 3.3232076168060303 (nan) | > loss: 0.013338892720639706 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13859947978925735) | > loader_time: 0.0046 (0.004734453309073721)  --> TIME: 2025-02-26 23:12:54 -- STEP: 4368/36938 -- GLOBAL_STEP: 1444950 | > loss_text_ce: 0.03729865700006485 (0.030586987607617737) | > loss_mel_ce: 2.619891881942749 (nan) | > loss: 0.010544407181441784 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.13856942509556922) | > loader_time: 0.0047 (0.004733865296011007)  --> TIME: 2025-02-26 23:13:03 -- STEP: 4418/36938 -- GLOBAL_STEP: 1445000 | > loss_text_ce: 0.02912660501897335 (0.03060563064642814) | > loss_mel_ce: 2.268975257873535 (nan) | > loss: 0.00911945290863514 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13855631489622108) | > loader_time: 0.0047 (0.004733086171976959)  --> TIME: 2025-02-26 23:13:12 -- STEP: 4468/36938 -- GLOBAL_STEP: 1445050 | > loss_text_ce: 0.027083896100521088 (0.030610986268639753) | > loss_mel_ce: 0.031005859375 (nan) | > loss: 0.00023051491007208824 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.1385282630779542) | > loader_time: 0.0048 (0.004731845354116063)  --> TIME: 2025-02-26 23:13:22 -- STEP: 4518/36938 -- GLOBAL_STEP: 1445100 | > loss_text_ce: 0.03130751848220825 (0.030613989367645734) | > loss_mel_ce: 2.2725002765655518 (nan) | > loss: 0.009142094291746616 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.13853798397033878) | > loader_time: 0.0048 (0.004730956065755019)  --> TIME: 2025-02-26 23:13:31 -- STEP: 4568/36938 -- GLOBAL_STEP: 1445150 | > loss_text_ce: 0.028632022440433502 (0.03062792530775905) | > loss_mel_ce: 2.4674019813537598 (nan) | > loss: 0.009904896840453148 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13849300036497234) | > loader_time: 0.0045 (0.004729923402365368)  --> TIME: 2025-02-26 23:13:41 -- STEP: 4618/36938 -- GLOBAL_STEP: 1445200 | > loss_text_ce: 0.026936925947666168 (0.030623613364249146) | > loss_mel_ce: 2.3451173305511475 (nan) | > loss: 0.009412914514541626 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13850833023198572) | > loader_time: 0.0047 (0.004729183890077139)  --> TIME: 2025-02-26 23:13:50 -- STEP: 4668/36938 -- GLOBAL_STEP: 1445250 | > loss_text_ce: 0.038264598697423935 (0.030625799303513654) | > loss_mel_ce: 2.7801320552825928 (nan) | > loss: 0.011184114031493664 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13852765387516364) | > loader_time: 0.0046 (0.004728554657819237)  --> TIME: 2025-02-26 23:14:00 -- STEP: 4718/36938 -- GLOBAL_STEP: 1445300 | > loss_text_ce: 0.026373710483312607 (0.03061005387051744) | > loss_mel_ce: 2.8463544845581055 (nan) | > loss: 0.011399715207517147 (nan) | > current_lr: 5e-06 | > step_time: 0.1848 (0.13855171688738047) | > loader_time: 0.0049 (0.004727650921745997)  --> TIME: 2025-02-26 23:14:09 -- STEP: 4768/36938 -- GLOBAL_STEP: 1445350 | > loss_text_ce: 0.02496800757944584 (0.030598561617197317) | > loss_mel_ce: 1.717358112335205 (nan) | > loss: 0.006913993041962385 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13854004392687966) | > loader_time: 0.0046 (0.00472657907529165)  --> TIME: 2025-02-26 23:14:18 -- STEP: 4818/36938 -- GLOBAL_STEP: 1445400 | > loss_text_ce: 0.026541970670223236 (0.030604542835954623) | > loss_mel_ce: 2.9105823040008545 (nan) | > loss: 0.01165525522083044 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.138494208869523) | > loader_time: 0.0046 (0.0047256220619230176)  --> TIME: 2025-02-26 23:14:28 -- STEP: 4868/36938 -- GLOBAL_STEP: 1445450 | > loss_text_ce: 0.016753502190113068 (0.030603161875056004) | > loss_mel_ce: 1.913356065750122 (nan) | > loss: 0.007659165654331446 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1384644768428103) | > loader_time: 0.0046 (0.004724753226230847)  --> TIME: 2025-02-26 23:14:37 -- STEP: 4918/36938 -- GLOBAL_STEP: 1445500 | > loss_text_ce: 0.02459060773253441 (0.030590568339068105) | > loss_mel_ce: 0.41152581572532654 (nan) | > loss: 0.0017306208610534668 (nan) | > current_lr: 5e-06 | > step_time: 0.1118 (0.1384388342190686) | > loader_time: 0.0046 (0.004723828563170494)  --> TIME: 2025-02-26 23:14:46 -- STEP: 4968/36938 -- GLOBAL_STEP: 1445550 | > loss_text_ce: 0.03639872372150421 (0.03059538039315084) | > loss_mel_ce: 2.9102225303649902 (nan) | > loss: 0.01169294212013483 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13838895046768604) | > loader_time: 0.0045 (0.004723308502955908)  --> TIME: 2025-02-26 23:14:55 -- STEP: 5018/36938 -- GLOBAL_STEP: 1445600 | > loss_text_ce: 0.03458164259791374 (0.030591994017138396) | > loss_mel_ce: 2.5866594314575195 (nan) | > loss: 0.010401750914752483 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1384090029798978) | > loader_time: 0.0047 (0.004722835533929663)  --> TIME: 2025-02-26 23:15:05 -- STEP: 5068/36938 -- GLOBAL_STEP: 1445650 | > loss_text_ce: 0.03333663195371628 (0.03058711040765782) | > loss_mel_ce: 2.645899534225464 (nan) | > loss: 0.010631890036165714 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.13841785355261624) | > loader_time: 0.0047 (0.004722127127854398)  --> TIME: 2025-02-26 23:15:14 -- STEP: 5118/36938 -- GLOBAL_STEP: 1445700 | > loss_text_ce: 0.03355878219008446 (0.030583857923153615) | > loss_mel_ce: 2.1985583305358887 (nan) | > loss: 0.008857608772814274 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.138408987631133) | > loader_time: 0.0046 (0.00472135048166017)  --> TIME: 2025-02-26 23:15:24 -- STEP: 5168/36938 -- GLOBAL_STEP: 1445750 | > loss_text_ce: 0.034966208040714264 (0.0305974148853607) | > loss_mel_ce: 2.3045847415924072 (nan) | > loss: 0.009283932857215405 (nan) | > current_lr: 5e-06 | > step_time: 0.1443 (0.13841592993345103) | > loader_time: 0.0047 (0.0047205924618724015)  --> TIME: 2025-02-26 23:15:33 -- STEP: 5218/36938 -- GLOBAL_STEP: 1445800 | > loss_text_ce: 0.028272317722439766 (0.030594959003181866) | > loss_mel_ce: 2.6104416847229004 (nan) | > loss: 0.010471087880432606 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.13836836563576824) | > loader_time: 0.0046 (0.00471983777466932)  --> TIME: 2025-02-26 23:15:42 -- STEP: 5268/36938 -- GLOBAL_STEP: 1445850 | > loss_text_ce: 0.02121821604669094 (0.03058723936648721) | > loss_mel_ce: 2.550130605697632 (nan) | > loss: 0.010203765705227852 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13836554201969037) | > loader_time: 0.0047 (0.0047189423598026645)  --> TIME: 2025-02-26 23:15:52 -- STEP: 5318/36938 -- GLOBAL_STEP: 1445900 | > loss_text_ce: 0.027800647541880608 (0.030576503782204924) | > loss_mel_ce: 1.502349853515625 (nan) | > loss: 0.006072026211768389 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.1383391403533235) | > loader_time: 0.0046 (0.004718457500304257)  --> TIME: 2025-02-26 23:16:01 -- STEP: 5368/36938 -- GLOBAL_STEP: 1445950 | > loss_text_ce: 0.04066186025738716 (0.03057841410677902) | > loss_mel_ce: 3.114675760269165 (nan) | > loss: 0.012521181255578995 (nan) | > current_lr: 5e-06 | > step_time: 0.144 (0.13836056140602654) | > loader_time: 0.0046 (0.004718385714532372)  --> TIME: 2025-02-26 23:16:10 -- STEP: 5418/36938 -- GLOBAL_STEP: 1446000 | > loss_text_ce: 0.03501376882195473 (0.030579066939167912) | > loss_mel_ce: 0.6158180236816406 (nan) | > loss: 0.0025826659984886646 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13832123797069268) | > loader_time: 0.0046 (0.00471741896179076)  --> TIME: 2025-02-26 23:16:20 -- STEP: 5468/36938 -- GLOBAL_STEP: 1446050 | > loss_text_ce: 0.03139638900756836 (0.03057291155174135) | > loss_mel_ce: 2.8076171875 (nan) | > loss: 0.01126592792570591 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13832875303569858) | > loader_time: 0.0046 (0.004716583561007793)  --> TIME: 2025-02-26 23:16:29 -- STEP: 5518/36938 -- GLOBAL_STEP: 1446100 | > loss_text_ce: 0.025435395538806915 (0.030576695032531194) | > loss_mel_ce: 3.1130638122558594 (nan) | > loss: 0.012454362586140633 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13833456356571278) | > loader_time: 0.0046 (0.004715991780653346)  --> TIME: 2025-02-26 23:16:38 -- STEP: 5568/36938 -- GLOBAL_STEP: 1446150 | > loss_text_ce: 0.030739787966012955 (0.03056688261105968) | > loss_mel_ce: 3.2680580615997314 (nan) | > loss: 0.013090468011796474 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13831198596577596) | > loader_time: 0.0047 (0.004715612779061002)  --> TIME: 2025-02-26 23:16:48 -- STEP: 5618/36938 -- GLOBAL_STEP: 1446200 | > loss_text_ce: 0.04059591516852379 (0.030563795334010012) | > loss_mel_ce: 0.185322567820549 (nan) | > loss: 0.0008965019951574504 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.1383296689260644) | > loader_time: 0.0048 (0.004715692916345922)  --> TIME: 2025-02-26 23:16:57 -- STEP: 5668/36938 -- GLOBAL_STEP: 1446250 | > loss_text_ce: 0.04390105605125427 (0.030550844748501175) | > loss_mel_ce: 2.755613327026367 (nan) | > loss: 0.011109184473752975 (nan) | > current_lr: 5e-06 | > step_time: 0.1618 (0.1383429375347663) | > loader_time: 0.0048 (0.004715263717344471)  --> TIME: 2025-02-26 23:17:06 -- STEP: 5718/36938 -- GLOBAL_STEP: 1446300 | > loss_text_ce: 0.031135236844420433 (0.030560050552222915) | > loss_mel_ce: 2.944617748260498 (nan) | > loss: 0.011808544397354126 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.13828342387875608) | > loader_time: 0.0047 (0.004714844651303952)  --> TIME: 2025-02-26 23:17:16 -- STEP: 5768/36938 -- GLOBAL_STEP: 1446350 | > loss_text_ce: 0.029687557369470596 (0.030560165837771373) | > loss_mel_ce: 2.349856376647949 (nan) | > loss: 0.00944263581186533 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.13829087190092404) | > loader_time: 0.0051 (0.004714412679288655)  --> TIME: 2025-02-26 23:17:26 -- STEP: 5818/36938 -- GLOBAL_STEP: 1446400 | > loss_text_ce: 0.029667936265468597 (0.03056489176159464) | > loss_mel_ce: 2.335902690887451 (nan) | > loss: 0.009387184865772724 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.1383248677406172) | > loader_time: 0.0049 (0.004714280643509031)  --> TIME: 2025-02-26 23:17:35 -- STEP: 5868/36938 -- GLOBAL_STEP: 1446450 | > loss_text_ce: 0.028826402500271797 (0.030565287358868404) | > loss_mel_ce: 1.750343680381775 (nan) | > loss: 0.007060199044644833 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1383014489536642) | > loader_time: 0.0046 (0.0047140472912024535)  --> TIME: 2025-02-26 23:17:45 -- STEP: 5918/36938 -- GLOBAL_STEP: 1446500 | > loss_text_ce: 0.03724683076143265 (0.03056322986618233) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1435 (0.1383512926971889) | > loader_time: 0.0047 (0.004713783114292444)  --> TIME: 2025-02-26 23:17:54 -- STEP: 5968/36938 -- GLOBAL_STEP: 1446550 | > loss_text_ce: 0.026662461459636688 (0.030563719244486583) | > loss_mel_ce: 2.743518352508545 (nan) | > loss: 0.01099278125911951 (nan) | > current_lr: 5e-06 | > step_time: 0.1268 (0.138347779418125) | > loader_time: 0.0046 (0.004713427644952058)  --> TIME: 2025-02-26 23:18:04 -- STEP: 6018/36938 -- GLOBAL_STEP: 1446600 | > loss_text_ce: 0.029308659955859184 (0.030557944845741238) | > loss_mel_ce: 2.312915563583374 (nan) | > loss: 0.009294540621340275 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.1383685645727514) | > loader_time: 0.0046 (0.004713560347146637)  --> TIME: 2025-02-26 23:18:13 -- STEP: 6068/36938 -- GLOBAL_STEP: 1446650 | > loss_text_ce: 0.034548331052064896 (0.03055952018588516) | > loss_mel_ce: 2.5504984855651855 (nan) | > loss: 0.010258122347295284 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13837840475372876) | > loader_time: 0.0044 (0.004713073166562458)  --> TIME: 2025-02-26 23:18:22 -- STEP: 6118/36938 -- GLOBAL_STEP: 1446700 | > loss_text_ce: 0.031211521476507187 (0.03055719320545714) | > loss_mel_ce: 1.9173810482025146 (nan) | > loss: 0.007732510566711426 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.13835530399535803) | > loader_time: 0.0045 (0.004712930883835965)  --> TIME: 2025-02-26 23:18:32 -- STEP: 6168/36938 -- GLOBAL_STEP: 1446750 | > loss_text_ce: 0.02734164334833622 (0.030567655330173544) | > loss_mel_ce: 5.15625 (nan) | > loss: 0.020569810643792152 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13833833011362542) | > loader_time: 0.0046 (0.004712788047493677)  --> TIME: 2025-02-26 23:18:41 -- STEP: 6218/36938 -- GLOBAL_STEP: 1446800 | > loss_text_ce: 0.015526522882282734 (0.030568636104736528) | > loss_mel_ce: 2.0729498863220215 (nan) | > loss: 0.00828760489821434 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13829920789596592) | > loader_time: 0.0049 (0.004712918250267065)  --> TIME: 2025-02-26 23:18:50 -- STEP: 6268/36938 -- GLOBAL_STEP: 1446850 | > loss_text_ce: 0.030684292316436768 (0.030567584203166298) | > loss_mel_ce: 2.679847240447998 (nan) | > loss: 0.01075607817620039 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13827465081869197) | > loader_time: 0.0045 (0.0047124609798098)  --> TIME: 2025-02-26 23:19:00 -- STEP: 6318/36938 -- GLOBAL_STEP: 1446900 | > loss_text_ce: 0.035236652940511703 (0.030573505686402047) | > loss_mel_ce: 2.893730878829956 (nan) | > loss: 0.01162288710474968 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.13831618032941553) | > loader_time: 0.0049 (0.00471217434554963)  --> TIME: 2025-02-26 23:19:09 -- STEP: 6368/36938 -- GLOBAL_STEP: 1446950 | > loss_text_ce: 0.028052624315023422 (0.030571717354872654) | > loss_mel_ce: 2.576582431793213 (nan) | > loss: 0.010335853323340416 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13830259970234926) | > loader_time: 0.0047 (0.004711842042716916)  --> TIME: 2025-02-26 23:19:19 -- STEP: 6418/36938 -- GLOBAL_STEP: 1447000 | > loss_text_ce: 0.025955207645893097 (0.03057470511492422) | > loss_mel_ce: 2.0667030811309814 (nan) | > loss: 0.008304200135171413 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.1382991416166792) | > loader_time: 0.0047 (0.00471159538103808)  --> TIME: 2025-02-26 23:19:28 -- STEP: 6468/36938 -- GLOBAL_STEP: 1447050 | > loss_text_ce: 0.028310200199484825 (0.030581054378510553) | > loss_mel_ce: 3.0090513229370117 (nan) | > loss: 0.012053023092448711 (nan) | > current_lr: 5e-06 | > step_time: 0.1757 (0.13831198001689407) | > loader_time: 0.0048 (0.004711122297547682)  --> TIME: 2025-02-26 23:19:37 -- STEP: 6518/36938 -- GLOBAL_STEP: 1447100 | > loss_text_ce: 0.02736225351691246 (0.03057723665445924) | > loss_mel_ce: 2.8830935955047607 (nan) | > loss: 0.011549429036676884 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13828519430216163) | > loader_time: 0.0046 (0.004710510560580297)  --> TIME: 2025-02-26 23:19:46 -- STEP: 6568/36938 -- GLOBAL_STEP: 1447150 | > loss_text_ce: 0.03152841329574585 (0.030586908557384197) | > loss_mel_ce: 3.2974801063537598 (nan) | > loss: 0.013210352510213852 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.13824247221273037) | > loader_time: 0.0047 (0.004710579724317648)  --> TIME: 2025-02-26 23:19:56 -- STEP: 6618/36938 -- GLOBAL_STEP: 1447200 | > loss_text_ce: 0.029586050659418106 (0.030583440787593747) | > loss_mel_ce: 2.8070108890533447 (nan) | > loss: 0.011256338097155094 (nan) | > current_lr: 5e-06 | > step_time: 0.172 (0.13822829128932917) | > loader_time: 0.0046 (0.004711256102155307)  --> TIME: 2025-02-26 23:20:05 -- STEP: 6668/36938 -- GLOBAL_STEP: 1447250 | > loss_text_ce: 0.026657041162252426 (0.030596600098638217) | > loss_mel_ce: 3.1227474212646484 (nan) | > loss: 0.012497637420892715 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13820758131212624) | > loader_time: 0.0047 (0.004711178190063144)  --> TIME: 2025-02-26 23:20:14 -- STEP: 6718/36938 -- GLOBAL_STEP: 1447300 | > loss_text_ce: 0.031264908611774445 (0.030592206052478695) | > loss_mel_ce: 2.654895544052124 (nan) | > loss: 0.01065936777740717 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.13819967472471661) | > loader_time: 0.0046 (0.004710838602355082)  --> TIME: 2025-02-26 23:20:23 -- STEP: 6768/36938 -- GLOBAL_STEP: 1447350 | > loss_text_ce: 0.016489803791046143 (0.030589504004098326) | > loss_mel_ce: 2.740168571472168 (nan) | > loss: 0.01093912124633789 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13816225042579905) | > loader_time: 0.0045 (0.004710228554464135)  --> TIME: 2025-02-26 23:20:33 -- STEP: 6818/36938 -- GLOBAL_STEP: 1447400 | > loss_text_ce: 0.02405143715441227 (0.030589565734307142) | > loss_mel_ce: 2.4398844242095947 (nan) | > loss: 0.009777523577213287 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13819463821617065) | > loader_time: 0.0047 (0.004709867831023441)  --> TIME: 2025-02-26 23:20:43 -- STEP: 6868/36938 -- GLOBAL_STEP: 1447450 | > loss_text_ce: 0.03984571248292923 (0.030581788239099114) | > loss_mel_ce: 2.522934913635254 (nan) | > loss: 0.01016976498067379 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13819677262508975) | > loader_time: 0.0045 (0.004709359095689525)  --> TIME: 2025-02-26 23:20:52 -- STEP: 6918/36938 -- GLOBAL_STEP: 1447500 | > loss_text_ce: 0.03639449179172516 (0.030580202946032822) | > loss_mel_ce: 2.522522449493408 (nan) | > loss: 0.01015443354845047 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1382199194638779) | > loader_time: 0.0047 (0.004709088654144557)  --> TIME: 2025-02-26 23:21:01 -- STEP: 6968/36938 -- GLOBAL_STEP: 1447550 | > loss_text_ce: 0.02461622655391693 (0.030586738190180637) | > loss_mel_ce: 2.15625 (nan) | > loss: 0.008654231205582619 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13821365324145246) | > loader_time: 0.0049 (0.004708777510065997)  --> TIME: 2025-02-26 23:21:11 -- STEP: 7018/36938 -- GLOBAL_STEP: 1447600 | > loss_text_ce: 0.02974517270922661 (0.030571789214009062) | > loss_mel_ce: 3.1684341430664062 (nan) | > loss: 0.012691188603639603 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.1382155304588328) | > loader_time: 0.005 (0.004708523898425025)  --> TIME: 2025-02-26 23:21:20 -- STEP: 7068/36938 -- GLOBAL_STEP: 1447650 | > loss_text_ce: 0.041692741215229034 (0.030576698630405186) | > loss_mel_ce: 1.7060000896453857 (nan) | > loss: 0.006935289595276117 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13819064642658502) | > loader_time: 0.0045 (0.004708156453463488)  --> TIME: 2025-02-26 23:21:29 -- STEP: 7118/36938 -- GLOBAL_STEP: 1447700 | > loss_text_ce: 0.031483426690101624 (0.03057162929077699) | > loss_mel_ce: 2.74434232711792 (nan) | > loss: 0.011015182361006737 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13817759325880166) | > loader_time: 0.0047 (0.004707793936229697)  --> TIME: 2025-02-26 23:21:39 -- STEP: 7168/36938 -- GLOBAL_STEP: 1447750 | > loss_text_ce: 0.02984829619526863 (0.03057202108604023) | > loss_mel_ce: 2.4126250743865967 (nan) | > loss: 0.009692355059087276 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13815568710145187) | > loader_time: 0.0045 (0.004707606209974217)  --> TIME: 2025-02-26 23:21:48 -- STEP: 7218/36938 -- GLOBAL_STEP: 1447800 | > loss_text_ce: 0.03197172284126282 (0.030576810509252372) | > loss_mel_ce: 2.3713581562042236 (nan) | > loss: 0.00953702349215746 (nan) | > current_lr: 5e-06 | > step_time: 0.1349 (0.13814355059322783) | > loader_time: 0.0047 (0.004706992571887048)  --> TIME: 2025-02-26 23:21:57 -- STEP: 7268/36938 -- GLOBAL_STEP: 1447850 | > loss_text_ce: 0.0385788157582283 (0.030583436600863934) | > loss_mel_ce: 2.512338399887085 (nan) | > loss: 0.010122687555849552 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.13816238597781738) | > loader_time: 0.0046 (0.004706477095415531)  --> TIME: 2025-02-26 23:22:07 -- STEP: 7318/36938 -- GLOBAL_STEP: 1447900 | > loss_text_ce: 0.03591959923505783 (0.0305847791863499) | > loss_mel_ce: 3.1917290687561035 (nan) | > loss: 0.01280813105404377 (nan) | > current_lr: 5e-06 | > step_time: 0.1773 (0.13818978093692827) | > loader_time: 0.0048 (0.00470626207608675)  --> TIME: 2025-02-26 23:22:16 -- STEP: 7368/36938 -- GLOBAL_STEP: 1447950 | > loss_text_ce: 0.032334547489881516 (0.03058699777197643) | > loss_mel_ce: 2.619305372238159 (nan) | > loss: 0.010522381402552128 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13817806103459954) | > loader_time: 0.0045 (0.004705769166625426)  --> TIME: 2025-02-26 23:22:26 -- STEP: 7418/36938 -- GLOBAL_STEP: 1448000 | > loss_text_ce: 0.031831275671720505 (0.030587587068417334) | > loss_mel_ce: 2.7881457805633545 (nan) | > loss: 0.011190385557711124 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13818005526859747) | > loader_time: 0.0047 (0.004705288494395015)  --> TIME: 2025-02-26 23:22:35 -- STEP: 7468/36938 -- GLOBAL_STEP: 1448050 | > loss_text_ce: 0.02883078344166279 (0.030595889581885163) | > loss_mel_ce: 3.0435986518859863 (nan) | > loss: 0.012192181311547756 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13821307649954714) | > loader_time: 0.0047 (0.004705059541120987)  --> TIME: 2025-02-26 23:22:44 -- STEP: 7518/36938 -- GLOBAL_STEP: 1448100 | > loss_text_ce: 0.031137695536017418 (0.030587983115254864) | > loss_mel_ce: 0.26828277111053467 (nan) | > loss: 0.0011881765676662326 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.1381817955893389) | > loader_time: 0.0047 (0.004704708810751995)  --> TIME: 2025-02-26 23:22:54 -- STEP: 7568/36938 -- GLOBAL_STEP: 1448150 | > loss_text_ce: 0.028078535571694374 (0.030589225669499745) | > loss_mel_ce: 2.209951877593994 (nan) | > loss: 0.00888107344508171 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.1381792037040945) | > loader_time: 0.0046 (0.004704364163931039)  --> TIME: 2025-02-26 23:23:03 -- STEP: 7618/36938 -- GLOBAL_STEP: 1448200 | > loss_text_ce: 0.03792803734540939 (0.030588874786536692) | > loss_mel_ce: 2.493055820465088 (nan) | > loss: 0.010043587535619736 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13815941604675397) | > loader_time: 0.0047 (0.004704055713524385)  --> TIME: 2025-02-26 23:23:13 -- STEP: 7668/36938 -- GLOBAL_STEP: 1448250 | > loss_text_ce: 0.023341596126556396 (0.030587849024310124) | > loss_mel_ce: 2.065852642059326 (nan) | > loss: 0.008290453813970089 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13816990454991707) | > loader_time: 0.0045 (0.004703775195946107)  --> TIME: 2025-02-26 23:23:22 -- STEP: 7718/36938 -- GLOBAL_STEP: 1448300 | > loss_text_ce: 0.02383350394666195 (0.030583210856622134) | > loss_mel_ce: 0.017913818359375 (nan) | > loss: 0.00016566399426665157 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13815426004770154) | > loader_time: 0.0046 (0.0047034961814613595)  --> TIME: 2025-02-26 23:23:31 -- STEP: 7768/36938 -- GLOBAL_STEP: 1448350 | > loss_text_ce: 0.034833550453186035 (0.030578434878545904) | > loss_mel_ce: 2.8219289779663086 (nan) | > loss: 0.011336359195411205 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13814747170856928) | > loader_time: 0.0048 (0.004703162842258501)  --> TIME: 2025-02-26 23:23:41 -- STEP: 7818/36938 -- GLOBAL_STEP: 1448400 | > loss_text_ce: 0.03017331101000309 (0.030569992025794277) | > loss_mel_ce: 3.119650363922119 (nan) | > loss: 0.012499300763010979 (nan) | > current_lr: 5e-06 | > step_time: 0.1687 (0.13813469912518547) | > loader_time: 0.0048 (0.004702855053079754)  --> TIME: 2025-02-26 23:23:50 -- STEP: 7868/36938 -- GLOBAL_STEP: 1448450 | > loss_text_ce: 0.02846989594399929 (0.030569420238018383) | > loss_mel_ce: 2.873696804046631 (nan) | > loss: 0.011516534723341465 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.13813714003526611) | > loader_time: 0.0046 (0.004702450996373727)  --> TIME: 2025-02-26 23:24:00 -- STEP: 7918/36938 -- GLOBAL_STEP: 1448500 | > loss_text_ce: 0.022913115099072456 (0.030571219463105544) | > loss_mel_ce: 2.3450558185577393 (nan) | > loss: 0.009396702982485294 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13813735722712644) | > loader_time: 0.0045 (0.004702280223535221)  --> TIME: 2025-02-26 23:24:09 -- STEP: 7968/36938 -- GLOBAL_STEP: 1448550 | > loss_text_ce: 0.024324098601937294 (0.030567922080767876) | > loss_mel_ce: 2.117344856262207 (nan) | > loss: 0.00849868729710579 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.1381392244653534) | > loader_time: 0.0047 (0.0047018138998483686)  --> TIME: 2025-02-26 23:24:18 -- STEP: 8018/36938 -- GLOBAL_STEP: 1448600 | > loss_text_ce: 0.03849109634757042 (0.03055870205249445) | > loss_mel_ce: 2.6535589694976807 (nan) | > loss: 0.01068273838609457 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.1381632356936278) | > loader_time: 0.0048 (0.004701889670379872)  --> TIME: 2025-02-26 23:24:28 -- STEP: 8068/36938 -- GLOBAL_STEP: 1448650 | > loss_text_ce: 0.03652219846844673 (0.030566817780021148) | > loss_mel_ce: 2.4077768325805664 (nan) | > loss: 0.00969959981739521 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1381422050766389) | > loader_time: 0.0046 (0.004701862166172583)  --> TIME: 2025-02-26 23:24:37 -- STEP: 8118/36938 -- GLOBAL_STEP: 1448700 | > loss_text_ce: 0.03825559467077255 (0.0305665207194565) | > loss_mel_ce: 1.23689866065979 (nan) | > loss: 0.005060136318206787 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1381716806919716) | > loader_time: 0.0045 (0.004701834237173186)  --> TIME: 2025-02-26 23:24:47 -- STEP: 8168/36938 -- GLOBAL_STEP: 1448750 | > loss_text_ce: 0.029999686405062675 (0.030567706972939836) | > loss_mel_ce: 2.6890645027160645 (nan) | > loss: 0.010789938271045685 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.1381469709574305) | > loader_time: 0.0052 (0.004701725649670231)  --> TIME: 2025-02-26 23:24:56 -- STEP: 8218/36938 -- GLOBAL_STEP: 1448800 | > loss_text_ce: 0.03833579272031784 (0.030560901290044663) | > loss_mel_ce: 3.1978325843811035 (nan) | > loss: 0.012841938994824886 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13816400025702694) | > loader_time: 0.0047 (0.004702089447324147)  --> TIME: 2025-02-26 23:25:05 -- STEP: 8268/36938 -- GLOBAL_STEP: 1448850 | > loss_text_ce: 0.01998141035437584 (0.0305675203614811) | > loss_mel_ce: 2.137392520904541 (nan) | > loss: 0.008561007678508759 (nan) | > current_lr: 5e-06 | > step_time: 0.1123 (0.13813836486892117) | > loader_time: 0.0045 (0.004702114141904403)  --> TIME: 2025-02-26 23:25:15 -- STEP: 8318/36938 -- GLOBAL_STEP: 1448900 | > loss_text_ce: 0.03251659870147705 (0.030566900816839437) | > loss_mel_ce: 3.2515716552734375 (nan) | > loss: 0.013032096438109875 (nan) | > current_lr: 5e-06 | > step_time: 0.1877 (0.13814108120647048) | > loader_time: 0.0049 (0.00470244615059056)  --> TIME: 2025-02-26 23:25:24 -- STEP: 8368/36938 -- GLOBAL_STEP: 1448950 | > loss_text_ce: 0.03887716308236122 (0.03057090163328391) | > loss_mel_ce: 0.024913311004638672 (nan) | > loss: 0.00025313679361715913 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13815783690768083) | > loader_time: 0.0045 (0.004702112899227773)  --> TIME: 2025-02-26 23:25:34 -- STEP: 8418/36938 -- GLOBAL_STEP: 1449000 | > loss_text_ce: 0.03081638552248478 (0.03057223142517023) | > loss_mel_ce: 0.1696980744600296 (nan) | > loss: 0.0007956923800520599 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13816211500596126) | > loader_time: 0.0046 (0.004702255657320219)  --> TIME: 2025-02-26 23:25:43 -- STEP: 8468/36938 -- GLOBAL_STEP: 1449050 | > loss_text_ce: 0.028633369132876396 (0.030566526619659407) | > loss_mel_ce: 2.4105935096740723 (nan) | > loss: 0.00967947207391262 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13816521482530764) | > loader_time: 0.0045 (0.004702138039206475)  --> TIME: 2025-02-26 23:25:53 -- STEP: 8518/36938 -- GLOBAL_STEP: 1449100 | > loss_text_ce: 0.03140239045023918 (0.030568004265120574) | > loss_mel_ce: 2.2726192474365234 (nan) | > loss: 0.009142943657934666 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13814999842257514) | > loader_time: 0.0046 (0.0047017207417753525)  --> TIME: 2025-02-26 23:26:02 -- STEP: 8568/36938 -- GLOBAL_STEP: 1449150 | > loss_text_ce: 0.0317886620759964 (0.030565001394527502) | > loss_mel_ce: 2.3732805252075195 (nan) | > loss: 0.009543925523757935 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13812913513985037) | > loader_time: 0.0046 (0.004701353866958268)  --> TIME: 2025-02-26 23:26:11 -- STEP: 8618/36938 -- GLOBAL_STEP: 1449200 | > loss_text_ce: 0.023855654522776604 (0.030565570517410483) | > loss_mel_ce: 1.3623501062393188 (nan) | > loss: 0.005500816740095615 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13811200670422502) | > loader_time: 0.0046 (0.0047009295005205265)  --> TIME: 2025-02-26 23:26:20 -- STEP: 8668/36938 -- GLOBAL_STEP: 1449250 | > loss_text_ce: 0.03254556283354759 (0.030569309778627565) | > loss_mel_ce: 2.5642597675323486 (nan) | > loss: 0.010304783470928669 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13810231930988462) | > loader_time: 0.0047 (0.004700677346457086)  --> TIME: 2025-02-26 23:26:30 -- STEP: 8718/36938 -- GLOBAL_STEP: 1449300 | > loss_text_ce: 0.03965022414922714 (0.030570612217543848) | > loss_mel_ce: 2.754364490509033 (nan) | > loss: 0.011087360791862011 (nan) | > current_lr: 5e-06 | > step_time: 0.1814 (0.1381174480731843) | > loader_time: 0.0048 (0.004700620969136561)  --> TIME: 2025-02-26 23:26:39 -- STEP: 8768/36938 -- GLOBAL_STEP: 1449350 | > loss_text_ce: 0.022125711664557457 (0.030569765040073985) | > loss_mel_ce: 2.9725561141967773 (nan) | > loss: 0.011883658356964588 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13812223990468253) | > loader_time: 0.0046 (0.004700293179846163)  --> TIME: 2025-02-26 23:26:49 -- STEP: 8818/36938 -- GLOBAL_STEP: 1449400 | > loss_text_ce: 0.0384223498404026 (0.030570927790112) | > loss_mel_ce: 2.3946352005004883 (nan) | > loss: 0.009654990397393703 (nan) | > current_lr: 5e-06 | > step_time: 0.1617 (0.1381143671932558) | > loader_time: 0.0048 (0.004699935554025071)  --> TIME: 2025-02-26 23:26:58 -- STEP: 8868/36938 -- GLOBAL_STEP: 1449450 | > loss_text_ce: 0.03804536908864975 (0.03057609661534785) | > loss_mel_ce: 3.0143420696258545 (nan) | > loss: 0.012112649157643318 (nan) | > current_lr: 5e-06 | > step_time: 0.1679 (0.13813158473185647) | > loader_time: 0.0048 (0.004700181394963315)  --> TIME: 2025-02-26 23:27:08 -- STEP: 8918/36938 -- GLOBAL_STEP: 1449500 | > loss_text_ce: 0.037550874054431915 (0.030578875863148905) | > loss_mel_ce: 1.8456777334213257 (nan) | > loss: 0.007473130244761705 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13815417095106514) | > loader_time: 0.0047 (0.00469996100932821)  --> TIME: 2025-02-26 23:27:17 -- STEP: 8968/36938 -- GLOBAL_STEP: 1449550 | > loss_text_ce: 0.030959123745560646 (0.03058156807585823) | > loss_mel_ce: 2.2985711097717285 (nan) | > loss: 0.009244168177247047 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.1381691826655335) | > loader_time: 0.0046 (0.004699856175789335)  --> TIME: 2025-02-26 23:27:27 -- STEP: 9018/36938 -- GLOBAL_STEP: 1449600 | > loss_text_ce: 0.030067747458815575 (0.030577540501640698) | > loss_mel_ce: 2.7065834999084473 (nan) | > loss: 0.01085972785949707 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13819558466054555) | > loader_time: 0.0045 (0.004699631339035319)  --> TIME: 2025-02-26 23:27:36 -- STEP: 9068/36938 -- GLOBAL_STEP: 1449650 | > loss_text_ce: 0.023075738921761513 (0.030569968173563) | > loss_mel_ce: 2.925647020339966 (nan) | > loss: 0.011701282113790512 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13818458126250738) | > loader_time: 0.0046 (0.004699167697293986)  --> TIME: 2025-02-26 23:27:46 -- STEP: 9118/36938 -- GLOBAL_STEP: 1449700 | > loss_text_ce: 0.020883196964859962 (0.03056510743135752) | > loss_mel_ce: 2.7918739318847656 (nan) | > loss: 0.011161735281348228 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13817035307301687) | > loader_time: 0.0045 (0.004698899524937233)  --> TIME: 2025-02-26 23:27:55 -- STEP: 9168/36938 -- GLOBAL_STEP: 1449750 | > loss_text_ce: 0.035379037261009216 (0.030562038612218793) | > loss_mel_ce: 2.6029703617095947 (nan) | > loss: 0.010469640605151653 (nan) | > current_lr: 5e-06 | > step_time: 0.1519 (0.13817772391474942) | > loader_time: 0.0046 (0.004698566403272473)  --> TIME: 2025-02-26 23:28:05 -- STEP: 9218/36938 -- GLOBAL_STEP: 1449800 | > loss_text_ce: 0.03305664286017418 (0.030560300608680223) | > loss_mel_ce: 2.38571834564209 (nan) | > loss: 0.009598314762115479 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13820105021624338) | > loader_time: 0.0047 (0.004698396944694494)  --> TIME: 2025-02-26 23:28:14 -- STEP: 9268/36938 -- GLOBAL_STEP: 1449850 | > loss_text_ce: 0.0314679779112339 (0.03055241502508727) | > loss_mel_ce: 0.2679850161075592 (nan) | > loss: 0.0011883055558428168 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.138184046380007) | > loader_time: 0.0047 (0.0046982209796683225)  --> TIME: 2025-02-26 23:28:23 -- STEP: 9318/36938 -- GLOBAL_STEP: 1449900 | > loss_text_ce: 0.02502184361219406 (0.030545938639503983) | > loss_mel_ce: 3.0753393173217773 (nan) | > loss: 0.012303020805120468 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13816171939548613) | > loader_time: 0.0046 (0.0046979943987995165)  --> TIME: 2025-02-26 23:28:32 -- STEP: 9368/36938 -- GLOBAL_STEP: 1449950 | > loss_text_ce: 0.04219723120331764 (0.030543274007758314) | > loss_mel_ce: 2.500945568084717 (nan) | > loss: 0.010091837495565414 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1381548331091291) | > loader_time: 0.0046 (0.00469782337686325)  --> TIME: 2025-02-26 23:28:42 -- STEP: 9418/36938 -- GLOBAL_STEP: 1450000 | > loss_text_ce: 0.02644597925245762 (0.030542924028522055) | > loss_mel_ce: 2.0176215171813965 (nan) | > loss: 0.008111379109323025 (nan) | > current_lr: 5e-06 | > step_time: 0.1512 (0.13814584268179975) | > loader_time: 0.0047 (0.0046981326281632) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1450000.pth  --> TIME: 2025-02-26 23:28:57 -- STEP: 9468/36938 -- GLOBAL_STEP: 1450050 | > loss_text_ce: 0.035576000809669495 (0.030545121499891643) | > loss_mel_ce: 2.7368810176849365 (nan) | > loss: 0.011001814156770706 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13814404013794576) | > loader_time: 0.0046 (0.004698101508249752)  --> TIME: 2025-02-26 23:29:07 -- STEP: 9518/36938 -- GLOBAL_STEP: 1450100 | > loss_text_ce: 0.02913850173354149 (0.030545051017258316) | > loss_mel_ce: 2.7203173637390137 (nan) | > loss: 0.010910539887845516 (nan) | > current_lr: 5e-06 | > step_time: 0.1506 (0.13815902343540892) | > loader_time: 0.0046 (0.004698155281498143)  --> TIME: 2025-02-26 23:29:16 -- STEP: 9568/36938 -- GLOBAL_STEP: 1450150 | > loss_text_ce: 0.03012523613870144 (0.030551302679316033) | > loss_mel_ce: 2.5286686420440674 (nan) | > loss: 0.010153944604098797 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1381322952317564) | > loader_time: 0.0046 (0.004697801950185199)  --> TIME: 2025-02-26 23:29:25 -- STEP: 9618/36938 -- GLOBAL_STEP: 1450200 | > loss_text_ce: 0.023114439100027084 (0.030550184949572505) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13813885809057053) | > loader_time: 0.0046 (0.004697688975882498)  --> TIME: 2025-02-26 23:29:35 -- STEP: 9668/36938 -- GLOBAL_STEP: 1450250 | > loss_text_ce: 0.031077710911631584 (0.030544933572933263) | > loss_mel_ce: 1.5118132829666138 (nan) | > loss: 0.006122583989053965 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13812646204447818) | > loader_time: 0.0045 (0.004697342080718784)  --> TIME: 2025-02-26 23:29:44 -- STEP: 9718/36938 -- GLOBAL_STEP: 1450300 | > loss_text_ce: 0.027285408228635788 (0.03054653176507677) | > loss_mel_ce: 1.0413942337036133 (nan) | > loss: 0.004240792710334063 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13811798963215155) | > loader_time: 0.0045 (0.004696994584439398)  --> TIME: 2025-02-26 23:29:53 -- STEP: 9768/36938 -- GLOBAL_STEP: 1450350 | > loss_text_ce: 0.0328286848962307 (0.030548157460452113) | > loss_mel_ce: 2.7753052711486816 (nan) | > loss: 0.011143389157950878 (nan) | > current_lr: 5e-06 | > step_time: 0.1791 (0.13812980429253452) | > loader_time: 0.0048 (0.004696774956244713)  --> TIME: 2025-02-26 23:30:03 -- STEP: 9818/36938 -- GLOBAL_STEP: 1450400 | > loss_text_ce: 0.025914646685123444 (0.030543293113738487) | > loss_mel_ce: 2.6029398441314697 (nan) | > loss: 0.01043196301907301 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13811781193190711) | > loader_time: 0.0047 (0.004697445284497071)  --> TIME: 2025-02-26 23:30:12 -- STEP: 9868/36938 -- GLOBAL_STEP: 1450450 | > loss_text_ce: 0.037411075085401535 (0.03054341518391493) | > loss_mel_ce: 3.1358790397644043 (nan) | > loss: 0.012592421844601631 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1381379691849954) | > loader_time: 0.0047 (0.004697768466889166)  --> TIME: 2025-02-26 23:30:22 -- STEP: 9918/36938 -- GLOBAL_STEP: 1450500 | > loss_text_ce: 0.02875138260424137 (0.030545481745931927) | > loss_mel_ce: 3.0178449153900146 (nan) | > loss: 0.012089668773114681 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13816518590296045) | > loader_time: 0.0047 (0.004697815211796679)  --> TIME: 2025-02-26 23:30:31 -- STEP: 9968/36938 -- GLOBAL_STEP: 1450550 | > loss_text_ce: 0.030453691259026527 (0.030550639330350418) | > loss_mel_ce: 2.353801727294922 (nan) | > loss: 0.00946133118122816 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13815039728751818) | > loader_time: 0.0045 (0.004697557819788958)  --> TIME: 2025-02-26 23:30:41 -- STEP: 10018/36938 -- GLOBAL_STEP: 1450600 | > loss_text_ce: 0.03142150491476059 (0.030556050143458423) | > loss_mel_ce: 3.3762853145599365 (nan) | > loss: 0.013522646389901638 (nan) | > current_lr: 5e-06 | > step_time: 0.1573 (0.1381357177856996) | > loader_time: 0.0047 (0.0046975357514890004)  --> TIME: 2025-02-26 23:30:50 -- STEP: 10068/36938 -- GLOBAL_STEP: 1450650 | > loss_text_ce: 0.031418874859809875 (0.03055660425256871) | > loss_mel_ce: 2.5634493827819824 (nan) | > loss: 0.010297096334397793 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13811899076911993) | > loader_time: 0.0047 (0.004697253389589652)  --> TIME: 2025-02-26 23:30:59 -- STEP: 10118/36938 -- GLOBAL_STEP: 1450700 | > loss_text_ce: 0.031981781125068665 (0.0305622444481849) | > loss_mel_ce: 2.1926074028015137 (nan) | > loss: 0.008827734738588333 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13812518247056632) | > loader_time: 0.0046 (0.0046971223881697634)  --> TIME: 2025-02-26 23:31:09 -- STEP: 10168/36938 -- GLOBAL_STEP: 1450750 | > loss_text_ce: 0.03311017528176308 (0.030562565966515146) | > loss_mel_ce: 2.8470919132232666 (nan) | > loss: 0.011429374106228352 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.1381117485575004) | > loader_time: 0.0047 (0.004696858459153019)  --> TIME: 2025-02-26 23:31:18 -- STEP: 10218/36938 -- GLOBAL_STEP: 1450800 | > loss_text_ce: 0.039369117468595505 (0.030562624827747003) | > loss_mel_ce: 2.516096830368042 (nan) | > loss: 0.01014073844999075 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.13810027585988194) | > loader_time: 0.0047 (0.004697186579604426)  --> TIME: 2025-02-26 23:31:27 -- STEP: 10268/36938 -- GLOBAL_STEP: 1450850 | > loss_text_ce: 0.026560796424746513 (0.030565383350208145) | > loss_mel_ce: 2.9611713886260986 (nan) | > loss: 0.01185608096420765 (nan) | > current_lr: 5e-06 | > step_time: 0.1617 (0.13808513075450038) | > loader_time: 0.0047 (0.004696996633673075)  --> TIME: 2025-02-26 23:31:36 -- STEP: 10318/36938 -- GLOBAL_STEP: 1450900 | > loss_text_ce: 0.030762754380702972 (0.03056602213392007) | > loss_mel_ce: 2.8551788330078125 (nan) | > loss: 0.011452149599790573 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.1380788243817466) | > loader_time: 0.0046 (0.004697127937461833)  --> TIME: 2025-02-26 23:31:46 -- STEP: 10368/36938 -- GLOBAL_STEP: 1450950 | > loss_text_ce: 0.030357277020812035 (0.03056358341231833) | > loss_mel_ce: 0.0858306884765625 (nan) | > loss: 0.0004610633768606931 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13805622563777928) | > loader_time: 0.0046 (0.004697045610274809)  --> TIME: 2025-02-26 23:31:55 -- STEP: 10418/36938 -- GLOBAL_STEP: 1451000 | > loss_text_ce: 0.028593474999070168 (0.030560210056103772) | > loss_mel_ce: 2.1818506717681885 (nan) | > loss: 0.008771604858338833 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13804744017240947) | > loader_time: 0.0046 (0.004697097357048563)  --> TIME: 2025-02-26 23:32:04 -- STEP: 10468/36938 -- GLOBAL_STEP: 1451050 | > loss_text_ce: 0.03621480241417885 (0.030561280104783276) | > loss_mel_ce: 0.929964005947113 (nan) | > loss: 0.0038340431638062 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1380285576276307) | > loader_time: 0.0046 (0.004696812755422926)  --> TIME: 2025-02-26 23:32:13 -- STEP: 10518/36938 -- GLOBAL_STEP: 1451100 | > loss_text_ce: 0.02925730310380459 (0.03055948862300544) | > loss_mel_ce: 2.6012179851531982 (nan) | > loss: 0.010438394732773304 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.138011561031237) | > loader_time: 0.0045 (0.004696642611275665)  --> TIME: 2025-02-26 23:32:22 -- STEP: 10568/36938 -- GLOBAL_STEP: 1451150 | > loss_text_ce: 0.028845593333244324 (0.030556647634002565) | > loss_mel_ce: 2.5162558555603027 (nan) | > loss: 0.01009960938245058 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13799723763974905) | > loader_time: 0.0045 (0.004696487816421497)  --> TIME: 2025-02-26 23:32:32 -- STEP: 10618/36938 -- GLOBAL_STEP: 1451200 | > loss_text_ce: 0.014980697073042393 (0.03055241530677621) | > loss_mel_ce: 1.1360883712768555 (nan) | > loss: 0.004567734431475401 (nan) | > current_lr: 5e-06 | > step_time: 0.1072 (0.13796620994082193) | > loader_time: 0.0046 (0.004696207411164828)  --> TIME: 2025-02-26 23:32:41 -- STEP: 10668/36938 -- GLOBAL_STEP: 1451250 | > loss_text_ce: 0.03260964900255203 (0.03054673261624451) | > loss_mel_ce: 2.079331398010254 (nan) | > loss: 0.00838071946054697 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13794024787594644) | > loader_time: 0.0046 (0.004695903530807419)  --> TIME: 2025-02-26 23:32:50 -- STEP: 10718/36938 -- GLOBAL_STEP: 1451300 | > loss_text_ce: 0.02889345772564411 (0.030541136481036783) | > loss_mel_ce: 3.021014928817749 (nan) | > loss: 0.012102811597287655 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.13792640832761174) | > loader_time: 0.0046 (0.0046961190963463375)  --> TIME: 2025-02-26 23:32:59 -- STEP: 10768/36938 -- GLOBAL_STEP: 1451350 | > loss_text_ce: 0.0336618535220623 (0.03054170083781472) | > loss_mel_ce: 2.524975538253784 (nan) | > loss: 0.010153323411941528 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13792663743999045) | > loader_time: 0.0046 (0.0046960812665378295)  --> TIME: 2025-02-26 23:33:09 -- STEP: 10818/36938 -- GLOBAL_STEP: 1451400 | > loss_text_ce: 0.030535314232110977 (0.030542506244657555) | > loss_mel_ce: 2.3140475749969482 (nan) | > loss: 0.00930390041321516 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13793371199327387) | > loader_time: 0.0046 (0.004696307704197027)  --> TIME: 2025-02-26 23:33:18 -- STEP: 10868/36938 -- GLOBAL_STEP: 1451450 | > loss_text_ce: 0.03105100244283676 (0.030535160191117194) | > loss_mel_ce: 2.5708985328674316 (nan) | > loss: 0.010325197130441666 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.1379437692076751) | > loader_time: 0.0045 (0.004696343262756329)  --> TIME: 2025-02-26 23:33:28 -- STEP: 10918/36938 -- GLOBAL_STEP: 1451500 | > loss_text_ce: 0.025960948318243027 (0.03053871559395084) | > loss_mel_ce: 3.0490453243255615 (nan) | > loss: 0.01220240630209446 (nan) | > current_lr: 5e-06 | > step_time: 0.1816 (0.13794175651314153) | > loader_time: 0.0048 (0.00469602988652421)  --> TIME: 2025-02-26 23:33:37 -- STEP: 10968/36938 -- GLOBAL_STEP: 1451550 | > loss_text_ce: 0.026055797934532166 (0.030540829486356957) | > loss_mel_ce: 3.052992105484009 (nan) | > loss: 0.012218444608151913 (nan) | > current_lr: 5e-06 | > step_time: 0.1861 (0.13794295006125332) | > loader_time: 0.0049 (0.0046962318761256715)  --> TIME: 2025-02-26 23:33:46 -- STEP: 11018/36938 -- GLOBAL_STEP: 1451600 | > loss_text_ce: 0.021728485822677612 (0.030538324869859454) | > loss_mel_ce: 1.95867919921875 (nan) | > loss: 0.007858761586248875 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13790849608028247) | > loader_time: 0.0045 (0.00469596947704434)  --> TIME: 2025-02-26 23:33:55 -- STEP: 11068/36938 -- GLOBAL_STEP: 1451650 | > loss_text_ce: 0.03481120616197586 (0.03053932546469325) | > loss_mel_ce: 2.819196939468384 (nan) | > loss: 0.011325430124998093 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.1379066722721331) | > loader_time: 0.0047 (0.004695771918384723)  --> TIME: 2025-02-26 23:34:05 -- STEP: 11118/36938 -- GLOBAL_STEP: 1451700 | > loss_text_ce: 0.02822008542716503 (0.030537326504225235) | > loss_mel_ce: 3.2844865322113037 (nan) | > loss: 0.013145661912858486 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1379233517649369) | > loader_time: 0.0046 (0.004695771151812197)  --> TIME: 2025-02-26 23:34:14 -- STEP: 11168/36938 -- GLOBAL_STEP: 1451750 | > loss_text_ce: 0.03452165424823761 (0.0305438679257704) | > loss_mel_ce: 2.797536611557007 (nan) | > loss: 0.011238327249884605 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.1379153948159825) | > loader_time: 0.0047 (0.004695814625923158)  --> TIME: 2025-02-26 23:34:24 -- STEP: 11218/36938 -- GLOBAL_STEP: 1451800 | > loss_text_ce: 0.030497780069708824 (0.030542858113081623) | > loss_mel_ce: 2.8047866821289062 (nan) | > loss: 0.01125112920999527 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.13792140162082564) | > loader_time: 0.0046 (0.004695689939606723)  --> TIME: 2025-02-26 23:34:33 -- STEP: 11268/36938 -- GLOBAL_STEP: 1451850 | > loss_text_ce: 0.023825017735362053 (0.03055118572968298) | > loss_mel_ce: 1.0690714120864868 (nan) | > loss: 0.004336890764534473 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.1378995973589074) | > loader_time: 0.0045 (0.0046954373751374135)  --> TIME: 2025-02-26 23:34:42 -- STEP: 11318/36938 -- GLOBAL_STEP: 1451900 | > loss_text_ce: 0.020070189610123634 (0.030542422616256906) | > loss_mel_ce: 1.4667879343032837 (nan) | > loss: 0.005900231190025806 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13787658283242477) | > loader_time: 0.0045 (0.0046951890855443275)  --> TIME: 2025-02-26 23:34:51 -- STEP: 11368/36938 -- GLOBAL_STEP: 1451950 | > loss_text_ce: 0.020331449806690216 (0.030544464496068558) | > loss_mel_ce: 1.7058312892913818 (nan) | > loss: 0.006849852856248617 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13788028825847876) | > loader_time: 0.0046 (0.0046950855110834445)  --> TIME: 2025-02-26 23:35:01 -- STEP: 11418/36938 -- GLOBAL_STEP: 1452000 | > loss_text_ce: 0.03081069327890873 (0.030550809306470864) | > loss_mel_ce: 2.2238643169403076 (nan) | > loss: 0.00894712284207344 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.1378908038327279) | > loader_time: 0.0045 (0.004695113474902205)  --> TIME: 2025-02-26 23:35:10 -- STEP: 11468/36938 -- GLOBAL_STEP: 1452050 | > loss_text_ce: 0.029871085658669472 (0.03055514405778098) | > loss_mel_ce: 2.6019139289855957 (nan) | > loss: 0.010443591512739658 (nan) | > current_lr: 5e-06 | > step_time: 0.17 (0.13788830429409188) | > loader_time: 0.0048 (0.004694873379380374)  --> TIME: 2025-02-26 23:35:20 -- STEP: 11518/36938 -- GLOBAL_STEP: 1452100 | > loss_text_ce: 0.029689177870750427 (0.030550669305114506) | > loss_mel_ce: 3.135472059249878 (nan) | > loss: 0.012560163624584675 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.13788758255289116) | > loader_time: 0.0048 (0.004694556109419649)  --> TIME: 2025-02-26 23:35:29 -- STEP: 11568/36938 -- GLOBAL_STEP: 1452150 | > loss_text_ce: 0.031302593648433685 (0.030550469289969184) | > loss_mel_ce: 0.528857409954071 (nan) | > loss: 0.002222857205197215 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.1378624068256243) | > loader_time: 0.0045 (0.004694138634254345)  --> TIME: 2025-02-26 23:35:38 -- STEP: 11618/36938 -- GLOBAL_STEP: 1452200 | > loss_text_ce: 0.028538240119814873 (0.0305560435695751) | > loss_mel_ce: 3.199655771255493 (nan) | > loss: 0.012810294516384602 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.13786772280844325) | > loader_time: 0.0049 (0.004693877781129056)  --> TIME: 2025-02-26 23:35:48 -- STEP: 11668/36938 -- GLOBAL_STEP: 1452250 | > loss_text_ce: 0.02607898600399494 (0.030554484325835598) | > loss_mel_ce: 2.6063907146453857 (nan) | > loss: 0.010446309112012386 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.13786204737018284) | > loader_time: 0.0046 (0.004693860299872521)  --> TIME: 2025-02-26 23:35:57 -- STEP: 11718/36938 -- GLOBAL_STEP: 1452300 | > loss_text_ce: 0.02295163832604885 (0.030553344903724564) | > loss_mel_ce: 1.9574939012527466 (nan) | > loss: 0.007858911529183388 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13787938818465295) | > loader_time: 0.0045 (0.004693942766666514)  --> TIME: 2025-02-26 23:36:07 -- STEP: 11768/36938 -- GLOBAL_STEP: 1452350 | > loss_text_ce: 0.02556011825799942 (0.030550199469968037) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13786337111856933) | > loader_time: 0.0046 (0.004694213962814278)  --> TIME: 2025-02-26 23:36:16 -- STEP: 11818/36938 -- GLOBAL_STEP: 1452400 | > loss_text_ce: 0.021206451579928398 (0.030550529595362336) | > loss_mel_ce: 2.4300057888031006 (nan) | > loss: 0.009727032855153084 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.137868535399619) | > loader_time: 0.0047 (0.004694586902488667)  --> TIME: 2025-02-26 23:36:25 -- STEP: 11868/36938 -- GLOBAL_STEP: 1452450 | > loss_text_ce: 0.02261955849826336 (0.03054862134047179) | > loss_mel_ce: 2.691622495651245 (nan) | > loss: 0.01077080238610506 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13785985064980702) | > loader_time: 0.0046 (0.004694622455721658)  --> TIME: 2025-02-26 23:36:35 -- STEP: 11918/36938 -- GLOBAL_STEP: 1452500 | > loss_text_ce: 0.02793414331972599 (0.030543991112679914) | > loss_mel_ce: 3.097353935241699 (nan) | > loss: 0.012401937507092953 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13785871878869058) | > loader_time: 0.0047 (0.004694985451164591)  --> TIME: 2025-02-26 23:36:44 -- STEP: 11968/36938 -- GLOBAL_STEP: 1452550 | > loss_text_ce: 0.02912723645567894 (0.030541824308186048) | > loss_mel_ce: 3.1361606121063232 (nan) | > loss: 0.012560666538774967 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.13784894498593112) | > loader_time: 0.0045 (0.004694894135476452)  --> TIME: 2025-02-26 23:36:53 -- STEP: 12018/36938 -- GLOBAL_STEP: 1452600 | > loss_text_ce: 0.03016316145658493 (0.030541093335194797) | > loss_mel_ce: 2.906886577606201 (nan) | > loss: 0.011654959060251713 (nan) | > current_lr: 5e-06 | > step_time: 0.1648 (0.13785965908067754) | > loader_time: 0.0047 (0.004694897256807726)  --> TIME: 2025-02-26 23:37:03 -- STEP: 12068/36938 -- GLOBAL_STEP: 1452650 | > loss_text_ce: 0.03002694994211197 (0.030536802628046388) | > loss_mel_ce: 2.84440541267395 (nan) | > loss: 0.011406478472054005 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.1378420714435895) | > loader_time: 0.0046 (0.004694843355456383)  --> TIME: 2025-02-26 23:37:12 -- STEP: 12118/36938 -- GLOBAL_STEP: 1452700 | > loss_text_ce: 0.023543428629636765 (0.0305369504530522) | > loss_mel_ce: 3.072890281677246 (nan) | > loss: 0.012287436053156853 (nan) | > current_lr: 5e-06 | > step_time: 0.1583 (0.13783988202635467) | > loader_time: 0.0047 (0.004694789682486855)  --> TIME: 2025-02-26 23:37:21 -- STEP: 12168/36938 -- GLOBAL_STEP: 1452750 | > loss_text_ce: 0.03369775041937828 (0.030541811196560888) | > loss_mel_ce: 2.78722882270813 (nan) | > loss: 0.011194153688848019 (nan) | > current_lr: 5e-06 | > step_time: 0.1444 (0.13783015035470106) | > loader_time: 0.0046 (0.004694947241796328)  --> TIME: 2025-02-26 23:37:31 -- STEP: 12218/36938 -- GLOBAL_STEP: 1452800 | > loss_text_ce: 0.027907904237508774 (0.030539509020428655) | > loss_mel_ce: 2.4889426231384277 (nan) | > loss: 0.009987502358853817 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13785048107953465) | > loader_time: 0.0046 (0.004695073655553585)  --> TIME: 2025-02-26 23:37:40 -- STEP: 12268/36938 -- GLOBAL_STEP: 1452850 | > loss_text_ce: 0.0274982750415802 (0.03054407593148284) | > loss_mel_ce: 2.6724061965942383 (nan) | > loss: 0.010713906958699226 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.1378489166430174) | > loader_time: 0.0045 (0.004694880259942671)  --> TIME: 2025-02-26 23:37:50 -- STEP: 12318/36938 -- GLOBAL_STEP: 1452900 | > loss_text_ce: 0.03798922523856163 (0.030545463267018808) | > loss_mel_ce: 2.5224695205688477 (nan) | > loss: 0.010160550475120544 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13785453414855156) | > loader_time: 0.0046 (0.004694722848078688)  --> TIME: 2025-02-26 23:37:59 -- STEP: 12368/36938 -- GLOBAL_STEP: 1452950 | > loss_text_ce: 0.029241112992167473 (0.030548894293558088) | > loss_mel_ce: 3.085875988006592 (nan) | > loss: 0.012361576780676842 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13785336524209665) | > loader_time: 0.0047 (0.004694440290042701)  --> TIME: 2025-02-26 23:38:08 -- STEP: 12418/36938 -- GLOBAL_STEP: 1453000 | > loss_text_ce: 0.027889935299754143 (0.030543522807586633) | > loss_mel_ce: 2.0713281631469727 (nan) | > loss: 0.00833023153245449 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13786205617216943) | > loader_time: 0.0046 (0.004694481348334785)  --> TIME: 2025-02-26 23:38:18 -- STEP: 12468/36938 -- GLOBAL_STEP: 1453050 | > loss_text_ce: 0.03653254732489586 (0.030546841658006446) | > loss_mel_ce: 1.2864837646484375 (nan) | > loss: 0.005250065121799707 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13787184866491337) | > loader_time: 0.0048 (0.004694405679026599)  --> TIME: 2025-02-26 23:38:27 -- STEP: 12518/36938 -- GLOBAL_STEP: 1453100 | > loss_text_ce: 0.040393974632024765 (0.03054693712028409) | > loss_mel_ce: 2.9113337993621826 (nan) | > loss: 0.011713206768035889 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.13785954821812654) | > loader_time: 0.0046 (0.004694233841171556)  --> TIME: 2025-02-26 23:38:37 -- STEP: 12568/36938 -- GLOBAL_STEP: 1453150 | > loss_text_ce: 0.03511982783675194 (0.030550261089468005) | > loss_mel_ce: 2.4689528942108154 (nan) | > loss: 0.009936796501278877 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.13786796473303478) | > loader_time: 0.0047 (0.004694182067673209)  --> TIME: 2025-02-26 23:38:46 -- STEP: 12618/36938 -- GLOBAL_STEP: 1453200 | > loss_text_ce: 0.028996732085943222 (0.030554459481500828) | > loss_mel_ce: 2.894089460372925 (nan) | > loss: 0.01159954909235239 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.137879602031222) | > loader_time: 0.0046 (0.004694102664136699)  --> TIME: 2025-02-26 23:38:55 -- STEP: 12668/36938 -- GLOBAL_STEP: 1453250 | > loss_text_ce: 0.02930927462875843 (0.030555261384588726) | > loss_mel_ce: 2.5305709838867188 (nan) | > loss: 0.010158255696296692 (nan) | > current_lr: 5e-06 | > step_time: 0.1116 (0.1378617074951083) | > loader_time: 0.0045 (0.004693834176378229)  --> TIME: 2025-02-26 23:39:05 -- STEP: 12718/36938 -- GLOBAL_STEP: 1453300 | > loss_text_ce: 0.029336581006646156 (0.03055037908770998) | > loss_mel_ce: 2.6449880599975586 (nan) | > loss: 0.010612400248646736 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13786825012309709) | > loader_time: 0.0045 (0.004693764319752063)  --> TIME: 2025-02-26 23:39:14 -- STEP: 12768/36938 -- GLOBAL_STEP: 1453350 | > loss_text_ce: 0.03837542608380318 (0.030546660891431793) | > loss_mel_ce: 2.4699552059173584 (nan) | > loss: 0.009953693486750126 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13784410333947225) | > loader_time: 0.0045 (0.004693951579114263)  --> TIME: 2025-02-26 23:39:23 -- STEP: 12818/36938 -- GLOBAL_STEP: 1453400 | > loss_text_ce: 0.025181449949741364 (0.03054595851965852) | > loss_mel_ce: 1.8567962646484375 (nan) | > loss: 0.007468165829777718 (nan) | > current_lr: 5e-06 | > step_time: 0.1116 (0.1378366778841413) | > loader_time: 0.0045 (0.004694246393895086)  --> TIME: 2025-02-26 23:39:32 -- STEP: 12868/36938 -- GLOBAL_STEP: 1453450 | > loss_text_ce: 0.0307877566665411 (0.03054868739519823) | > loss_mel_ce: 0.6768457889556885 (nan) | > loss: 0.002808069810271263 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13781111935928977) | > loader_time: 0.0047 (0.004694247064213983)  --> TIME: 2025-02-26 23:39:41 -- STEP: 12918/36938 -- GLOBAL_STEP: 1453500 | > loss_text_ce: 0.03325173631310463 (0.030554498732983975) | > loss_mel_ce: 2.697500228881836 (nan) | > loss: 0.010836318135261536 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.13779824041032096) | > loader_time: 0.0046 (0.004694243447480526)  --> TIME: 2025-02-26 23:39:51 -- STEP: 12968/36938 -- GLOBAL_STEP: 1453550 | > loss_text_ce: 0.031058568507432938 (0.030554378816437373) | > loss_mel_ce: 5.859375 (nan) | > loss: 0.023374738171696663 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13780876313882665) | > loader_time: 0.0047 (0.004694358626682787)  --> TIME: 2025-02-26 23:40:00 -- STEP: 13018/36938 -- GLOBAL_STEP: 1453600 | > loss_text_ce: 0.02260441891849041 (0.030548836858973152) | > loss_mel_ce: 2.241642951965332 (nan) | > loss: 0.008985109627246857 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13778411805455829) | > loader_time: 0.0046 (0.00469414983445517)  --> TIME: 2025-02-26 23:40:09 -- STEP: 13068/36938 -- GLOBAL_STEP: 1453650 | > loss_text_ce: 0.02460237219929695 (0.030549402362594333) | > loss_mel_ce: 2.4970881938934326 (nan) | > loss: 0.010006709024310112 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1377854896752093) | > loader_time: 0.0046 (0.004694196402132624)  --> TIME: 2025-02-26 23:40:19 -- STEP: 13118/36938 -- GLOBAL_STEP: 1453700 | > loss_text_ce: 0.027212074026465416 (0.030553592087604126) | > loss_mel_ce: 2.001378297805786 (nan) | > loss: 0.008049962110817432 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13778328579325352) | > loader_time: 0.0046 (0.004694222858683134)  --> TIME: 2025-02-26 23:40:28 -- STEP: 13168/36938 -- GLOBAL_STEP: 1453750 | > loss_text_ce: 0.03372800722718239 (0.030555871803311232) | > loss_mel_ce: 2.676882743835449 (nan) | > loss: 0.01075639296323061 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13778158124278347) | > loader_time: 0.0047 (0.004694301476600267)  --> TIME: 2025-02-26 23:40:37 -- STEP: 13218/36938 -- GLOBAL_STEP: 1453800 | > loss_text_ce: 0.029594888910651207 (0.030558127747632464) | > loss_mel_ce: 2.4820423126220703 (nan) | > loss: 0.009966814890503883 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13778097203142656) | > loader_time: 0.0045 (0.004694401649686372)  --> TIME: 2025-02-26 23:40:47 -- STEP: 13268/36938 -- GLOBAL_STEP: 1453850 | > loss_text_ce: 0.021972129121422768 (0.030559399309600395) | > loss_mel_ce: 1.6840413808822632 (nan) | > loss: 0.006769895553588867 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13778062631539853) | > loader_time: 0.0044 (0.004694377348154922)  --> TIME: 2025-02-26 23:40:56 -- STEP: 13318/36938 -- GLOBAL_STEP: 1453900 | > loss_text_ce: 0.02879703976213932 (0.030558633996166186) | > loss_mel_ce: 2.3860156536102295 (nan) | > loss: 0.009582591243088245 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.1377751588427575) | > loader_time: 0.0049 (0.004694484826555001)  --> TIME: 2025-02-26 23:41:06 -- STEP: 13368/36938 -- GLOBAL_STEP: 1453950 | > loss_text_ce: 0.027879249304533005 (0.030559521753194378) | > loss_mel_ce: 1.6867879629135132 (nan) | > loss: 0.006804235279560089 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1377754555998421) | > loader_time: 0.0046 (0.004694334355604879)  --> TIME: 2025-02-26 23:41:15 -- STEP: 13418/36938 -- GLOBAL_STEP: 1454000 | > loss_text_ce: 0.027791954576969147 (0.0305600806631099) | > loss_mel_ce: 2.7080881595611572 (nan) | > loss: 0.010856667533516884 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13778024273100728) | > loader_time: 0.0045 (0.004694212849404234)  --> TIME: 2025-02-26 23:41:24 -- STEP: 13468/36938 -- GLOBAL_STEP: 1454050 | > loss_text_ce: 0.022505611181259155 (0.030560971532785395) | > loss_mel_ce: 2.8388774394989014 (nan) | > loss: 0.011354695074260235 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1377729355917163) | > loader_time: 0.0045 (0.004693926602186016)  --> TIME: 2025-02-26 23:41:34 -- STEP: 13518/36938 -- GLOBAL_STEP: 1454100 | > loss_text_ce: 0.03276543691754341 (0.030559296658620037) | > loss_mel_ce: 2.5778872966766357 (nan) | > loss: 0.010359733365476131 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.1377664446601474) | > loader_time: 0.005 (0.004693777308271472)  --> TIME: 2025-02-26 23:41:43 -- STEP: 13568/36938 -- GLOBAL_STEP: 1454150 | > loss_text_ce: 0.024797042831778526 (0.030558111272803903) | > loss_mel_ce: 0.0013349789660423994 (nan) | > loss: 0.00010369850497227162 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1377647909915679) | > loader_time: 0.0045 (0.004693620240772687)  --> TIME: 2025-02-26 23:41:52 -- STEP: 13618/36938 -- GLOBAL_STEP: 1454200 | > loss_text_ce: 0.02758190780878067 (0.030553908626971455) | > loss_mel_ce: 3.283414125442505 (nan) | > loss: 0.013138874433934689 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.1377457044673999) | > loader_time: 0.0047 (0.00469377515669315)  --> TIME: 2025-02-26 23:42:01 -- STEP: 13668/36938 -- GLOBAL_STEP: 1454250 | > loss_text_ce: 0.03363000974059105 (0.030548689167721445) | > loss_mel_ce: 2.6790616512298584 (nan) | > loss: 0.01076465006917715 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.1377353418944424) | > loader_time: 0.0047 (0.004693642864807834)  --> TIME: 2025-02-26 23:42:10 -- STEP: 13718/36938 -- GLOBAL_STEP: 1454300 | > loss_text_ce: 0.0259690023958683 (0.03054484348891612) | > loss_mel_ce: 2.662602186203003 (nan) | > loss: 0.01066893432289362 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.1376964874736149) | > loader_time: 0.0046 (0.0046933991061927005)  --> TIME: 2025-02-26 23:42:20 -- STEP: 13768/36938 -- GLOBAL_STEP: 1454350 | > loss_text_ce: 0.027156196534633636 (0.03054852986524021) | > loss_mel_ce: 0.7532958984375 (nan) | > loss: 0.0030970321968197823 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13771080189773666) | > loader_time: 0.0045 (0.004693341611915398)  --> TIME: 2025-02-26 23:42:29 -- STEP: 13818/36938 -- GLOBAL_STEP: 1454400 | > loss_text_ce: 0.028788605704903603 (0.03055245542959703) | > loss_mel_ce: 2.872267961502075 (nan) | > loss: 0.011512129567563534 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13769837229137008) | > loader_time: 0.0047 (0.004693171950045034)  --> TIME: 2025-02-26 23:42:38 -- STEP: 13868/36938 -- GLOBAL_STEP: 1454450 | > loss_text_ce: 0.022576019167900085 (0.030554706926162366) | > loss_mel_ce: 2.365602731704712 (nan) | > loss: 0.009476901032030582 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13770162672414807) | > loader_time: 0.0048 (0.0046929810072593985)  --> TIME: 2025-02-26 23:42:48 -- STEP: 13918/36938 -- GLOBAL_STEP: 1454500 | > loss_text_ce: 0.026337385177612305 (0.03055573060107809) | > loss_mel_ce: 1.940690279006958 (nan) | > loss: 0.007805665954947472 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.1377117729406285) | > loader_time: 0.0045 (0.004692765741037199)  --> TIME: 2025-02-26 23:42:57 -- STEP: 13968/36938 -- GLOBAL_STEP: 1454550 | > loss_text_ce: 0.026173260062932968 (0.03055558697927801) | > loss_mel_ce: 2.3307454586029053 (nan) | > loss: 0.0093528525903821 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1377168849946987) | > loader_time: 0.0045 (0.004692551486817166)  --> TIME: 2025-02-26 23:43:07 -- STEP: 14018/36938 -- GLOBAL_STEP: 1454600 | > loss_text_ce: 0.022037969902157784 (0.030557395743594406) | > loss_mel_ce: 1.7318185567855835 (nan) | > loss: 0.006959748454391956 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13770032585526187) | > loader_time: 0.0044 (0.0046922094829753794)  --> TIME: 2025-02-26 23:43:16 -- STEP: 14068/36938 -- GLOBAL_STEP: 1454650 | > loss_text_ce: 0.03335948288440704 (0.030554957839919593) | > loss_mel_ce: 2.855792760848999 (nan) | > loss: 0.011464891023933887 (nan) | > current_lr: 5e-06 | > step_time: 0.1503 (0.1376899038657305) | > loader_time: 0.0047 (0.004691916566902462)  --> TIME: 2025-02-26 23:43:25 -- STEP: 14118/36938 -- GLOBAL_STEP: 1454700 | > loss_text_ce: 0.027719127014279366 (0.030550911924101858) | > loss_mel_ce: 3.1068594455718994 (nan) | > loss: 0.012438803911209106 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13767014741728795) | > loader_time: 0.0046 (0.004691641245267955)  --> TIME: 2025-02-26 23:43:34 -- STEP: 14168/36938 -- GLOBAL_STEP: 1454750 | > loss_text_ce: 0.03001631610095501 (0.03055229109512376) | > loss_mel_ce: 3.137899398803711 (nan) | > loss: 0.012571095488965511 (nan) | > current_lr: 5e-06 | > step_time: 0.1793 (0.13767898313276328) | > loader_time: 0.0049 (0.0046914765923660165)  --> TIME: 2025-02-26 23:43:44 -- STEP: 14218/36938 -- GLOBAL_STEP: 1454800 | > loss_text_ce: 0.03605597838759422 (0.03055092529187398) | > loss_mel_ce: 3.0884318351745605 (nan) | > loss: 0.012398761697113514 (nan) | > current_lr: 5e-06 | > step_time: 0.1883 (0.13767213129866535) | > loader_time: 0.0049 (0.004691180842119643)  --> TIME: 2025-02-26 23:43:53 -- STEP: 14268/36938 -- GLOBAL_STEP: 1454850 | > loss_text_ce: 0.023773351684212685 (0.03054670152843953) | > loss_mel_ce: 2.5442347526550293 (nan) | > loss: 0.010190509259700775 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13769345735347335) | > loader_time: 0.0045 (0.0046910588434506675)  --> TIME: 2025-02-26 23:44:03 -- STEP: 14318/36938 -- GLOBAL_STEP: 1454900 | > loss_text_ce: 0.02248462662100792 (0.030543627988787497) | > loss_mel_ce: 2.432464122772217 (nan) | > loss: 0.009741860441863537 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.1376933811063596) | > loader_time: 0.0046 (0.004690816705828707)  --> TIME: 2025-02-26 23:44:12 -- STEP: 14368/36938 -- GLOBAL_STEP: 1454950 | > loss_text_ce: 0.03221101313829422 (0.0305475363493477) | > loss_mel_ce: 3.018779993057251 (nan) | > loss: 0.012107107788324356 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13770105085752657) | > loader_time: 0.0045 (0.0046906630718097555)  --> TIME: 2025-02-26 23:44:22 -- STEP: 14418/36938 -- GLOBAL_STEP: 1455000 | > loss_text_ce: 0.024490036070346832 (0.03054909452527011) | > loss_mel_ce: 1.6512044668197632 (nan) | > loss: 0.006649581715464592 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.1377088701134672) | > loader_time: 0.0046 (0.004690474222992591)  --> TIME: 2025-02-26 23:44:31 -- STEP: 14468/36938 -- GLOBAL_STEP: 1455050 | > loss_text_ce: 0.015282251872122288 (0.03054450305780246) | > loss_mel_ce: 1.1808233261108398 (nan) | > loss: 0.0047464510425925255 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13769626826715373) | > loader_time: 0.0045 (0.004690381236125149)  --> TIME: 2025-02-26 23:44:40 -- STEP: 14518/36938 -- GLOBAL_STEP: 1455100 | > loss_text_ce: 0.030421439558267593 (0.030543053281503876) | > loss_mel_ce: 3.2174904346466064 (nan) | > loss: 0.012888540513813496 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13768752699532957) | > loader_time: 0.0047 (0.004690177809487609)  --> TIME: 2025-02-26 23:44:50 -- STEP: 14568/36938 -- GLOBAL_STEP: 1455150 | > loss_text_ce: 0.024289662018418312 (0.030542552078379043) | > loss_mel_ce: 3.1529204845428467 (nan) | > loss: 0.012607977725565434 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.1376803047312683) | > loader_time: 0.0046 (0.004690221578848372)  --> TIME: 2025-02-26 23:44:59 -- STEP: 14618/36938 -- GLOBAL_STEP: 1455200 | > loss_text_ce: 0.025183983147144318 (0.0305385459298547) | > loss_mel_ce: 2.4992337226867676 (nan) | > loss: 0.010017530992627144 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13769885555535688) | > loader_time: 0.0045 (0.004690344739113596)  --> TIME: 2025-02-26 23:45:09 -- STEP: 14668/36938 -- GLOBAL_STEP: 1455250 | > loss_text_ce: 0.03641559183597565 (0.03053696252449327) | > loss_mel_ce: 2.620469808578491 (nan) | > loss: 0.010543196462094784 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13769707429106784) | > loader_time: 0.0046 (0.004690518309642891)  --> TIME: 2025-02-26 23:45:18 -- STEP: 14718/36938 -- GLOBAL_STEP: 1455300 | > loss_text_ce: 0.02834094502031803 (0.03053493853691342) | > loss_mel_ce: 2.734719753265381 (nan) | > loss: 0.010964527726173401 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13767696048987027) | > loader_time: 0.0046 (0.0046906216926331235)  --> TIME: 2025-02-26 23:45:27 -- STEP: 14768/36938 -- GLOBAL_STEP: 1455350 | > loss_text_ce: 0.04301457479596138 (0.03053110334626172) | > loss_mel_ce: 2.9687280654907227 (nan) | > loss: 0.011951359920203686 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13767686370005544) | > loader_time: 0.0046 (0.004690591540522608)  --> TIME: 2025-02-26 23:45:37 -- STEP: 14818/36938 -- GLOBAL_STEP: 1455400 | > loss_text_ce: 0.025249242782592773 (0.030535386793000153) | > loss_mel_ce: 0.8747534155845642 (nan) | > loss: 0.003571439301595092 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13768810050692595) | > loader_time: 0.0046 (0.0046905481569164435)  --> TIME: 2025-02-26 23:45:46 -- STEP: 14868/36938 -- GLOBAL_STEP: 1455450 | > loss_text_ce: 0.02994040958583355 (0.03054201665280784) | > loss_mel_ce: 2.595120668411255 (nan) | > loss: 0.010416909120976925 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.1376711639275251) | > loader_time: 0.0048 (0.004690635162622418)  --> TIME: 2025-02-26 23:45:55 -- STEP: 14918/36938 -- GLOBAL_STEP: 1455500 | > loss_text_ce: 0.028025729581713676 (0.030542159649624746) | > loss_mel_ce: 0.1344255656003952 (nan) | > loss: 0.0006446480401791632 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.1376738234872977) | > loader_time: 0.0046 (0.004690576852654513)  --> TIME: 2025-02-26 23:46:05 -- STEP: 14968/36938 -- GLOBAL_STEP: 1455550 | > loss_text_ce: 0.029311802238225937 (0.030542393956736094) | > loss_mel_ce: 2.8529694080352783 (nan) | > loss: 0.01143762469291687 (nan) | > current_lr: 5e-06 | > step_time: 0.1508 (0.13768441733799247) | > loader_time: 0.0046 (0.004690601617368656)  --> TIME: 2025-02-26 23:46:14 -- STEP: 15018/36938 -- GLOBAL_STEP: 1455600 | > loss_text_ce: 0.03851318359375 (0.030543158735553064) | > loss_mel_ce: 2.2472193241119385 (nan) | > loss: 0.00907036755234003 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1376713075211093) | > loader_time: 0.0045 (0.004690688941367299)  --> TIME: 2025-02-26 23:46:23 -- STEP: 15068/36938 -- GLOBAL_STEP: 1455650 | > loss_text_ce: 0.03204688802361488 (0.03054333202456074) | > loss_mel_ce: 1.9861167669296265 (nan) | > loss: 0.00800858624279499 (nan) | > current_lr: 5e-06 | > step_time: 0.1533 (0.1376644559966062) | > loader_time: 0.0047 (0.004690659830986061)  --> TIME: 2025-02-26 23:46:33 -- STEP: 15118/36938 -- GLOBAL_STEP: 1455700 | > loss_text_ce: 0.028732841834425926 (0.030537774413343575) | > loss_mel_ce: 2.8195157051086426 (nan) | > loss: 0.01130257360637188 (nan) | > current_lr: 5e-06 | > step_time: 0.1458 (0.13767584921396256) | > loader_time: 0.0047 (0.004690625819284349)  --> TIME: 2025-02-26 23:46:43 -- STEP: 15168/36938 -- GLOBAL_STEP: 1455750 | > loss_text_ce: 0.02589394710958004 (0.030533353207848536) | > loss_mel_ce: 2.345832586288452 (nan) | > loss: 0.009411613456904888 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.13769963988837908) | > loader_time: 0.0047 (0.004690710156527266)  --> TIME: 2025-02-26 23:46:52 -- STEP: 15218/36938 -- GLOBAL_STEP: 1455800 | > loss_text_ce: 0.02102607861161232 (0.03053123505300388) | > loss_mel_ce: 1.931315541267395 (nan) | > loss: 0.007747387979179621 (nan) | > current_lr: 5e-06 | > step_time: 0.1444 (0.13769818953450144) | > loader_time: 0.0046 (0.004690729172694462)  --> TIME: 2025-02-26 23:47:01 -- STEP: 15268/36938 -- GLOBAL_STEP: 1455850 | > loss_text_ce: 0.025391168892383575 (0.030534234611421945) | > loss_mel_ce: 2.525968551635742 (nan) | > loss: 0.010124444030225277 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.1376840633998727) | > loader_time: 0.0046 (0.004690426617728077)  --> TIME: 2025-02-26 23:47:10 -- STEP: 15318/36938 -- GLOBAL_STEP: 1455900 | > loss_text_ce: 0.023169368505477905 (0.030534931027162288) | > loss_mel_ce: 2.595078945159912 (nan) | > loss: 0.010389874689280987 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13767478391012644) | > loader_time: 0.0066 (0.004690745182239086)  --> TIME: 2025-02-26 23:47:20 -- STEP: 15368/36938 -- GLOBAL_STEP: 1455950 | > loss_text_ce: 0.03470807522535324 (0.0305371695630934) | > loss_mel_ce: 2.0302469730377197 (nan) | > loss: 0.008194265887141228 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.1376703260529491) | > loader_time: 0.0046 (0.00469058551929819)  --> TIME: 2025-02-26 23:47:29 -- STEP: 15418/36938 -- GLOBAL_STEP: 1456000 | > loss_text_ce: 0.029608355835080147 (0.030534631932374896) | > loss_mel_ce: 2.502190589904785 (nan) | > loss: 0.0100468210875988 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13767160137180068) | > loader_time: 0.0046 (0.004690519148064195)  --> TIME: 2025-02-26 23:47:38 -- STEP: 15468/36938 -- GLOBAL_STEP: 1456050 | > loss_text_ce: 0.040692225098609924 (0.030539542329366377) | > loss_mel_ce: 2.0649616718292236 (nan) | > loss: 0.008355770260095596 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13766497897645652) | > loader_time: 0.0047 (0.00469066762541937)  --> TIME: 2025-02-26 23:47:48 -- STEP: 15518/36938 -- GLOBAL_STEP: 1456100 | > loss_text_ce: 0.028777293860912323 (0.030533775031666967) | > loss_mel_ce: 2.429440498352051 (nan) | > loss: 0.00975483376532793 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13766143645864215) | > loader_time: 0.0046 (0.00469081413194314)  --> TIME: 2025-02-26 23:47:57 -- STEP: 15568/36938 -- GLOBAL_STEP: 1456150 | > loss_text_ce: 0.029465848580002785 (0.0305315972986499) | > loss_mel_ce: 2.9288926124572754 (nan) | > loss: 0.011739518493413925 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13765280668500363) | > loader_time: 0.0045 (0.004690992164464811)  --> TIME: 2025-02-26 23:48:06 -- STEP: 15618/36938 -- GLOBAL_STEP: 1456200 | > loss_text_ce: 0.037696417421102524 (0.03052886382964442) | > loss_mel_ce: 2.4332408905029297 (nan) | > loss: 0.009805306792259216 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1376519297069657) | > loader_time: 0.0045 (0.0046908710414765966)  --> TIME: 2025-02-26 23:48:16 -- STEP: 15668/36938 -- GLOBAL_STEP: 1456250 | > loss_text_ce: 0.030395623296499252 (0.030529436230160654) | > loss_mel_ce: 2.6672756671905518 (nan) | > loss: 0.01070504542440176 (nan) | > current_lr: 5e-06 | > step_time: 0.1511 (0.1376518987258061) | > loader_time: 0.0047 (0.004690653790251665)  --> TIME: 2025-02-26 23:48:25 -- STEP: 15718/36938 -- GLOBAL_STEP: 1456300 | > loss_text_ce: 0.031431008130311966 (0.03053089036698804) | > loss_mel_ce: 1.4107465744018555 (nan) | > loss: 0.005722926929593086 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13765574194787733) | > loader_time: 0.0047 (0.004690739880659044)  --> TIME: 2025-02-26 23:48:34 -- STEP: 15768/36938 -- GLOBAL_STEP: 1456350 | > loss_text_ce: 0.02704715169966221 (0.030529468440741794) | > loss_mel_ce: 1.1865776777267456 (nan) | > loss: 0.004815971944481134 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.1376525326497306) | > loader_time: 0.0047 (0.004690719536873742)  --> TIME: 2025-02-26 23:48:44 -- STEP: 15818/36938 -- GLOBAL_STEP: 1456400 | > loss_text_ce: 0.02420097030699253 (0.0305294646247529) | > loss_mel_ce: 3.0234375 (nan) | > loss: 0.012093803845345974 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13764869227754917) | > loader_time: 0.0046 (0.004690698070673247)  --> TIME: 2025-02-26 23:48:53 -- STEP: 15868/36938 -- GLOBAL_STEP: 1456450 | > loss_text_ce: 0.029946621507406235 (0.030528056067465056) | > loss_mel_ce: 3.2942991256713867 (nan) | > loss: 0.013191452249884605 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13765296205513988) | > loader_time: 0.0046 (0.0046905939964254195)  --> TIME: 2025-02-26 23:49:02 -- STEP: 15918/36938 -- GLOBAL_STEP: 1456500 | > loss_text_ce: 0.03014078550040722 (0.030530226084270647) | > loss_mel_ce: 3.022824764251709 (nan) | > loss: 0.01211494393646717 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.1376390705187735) | > loader_time: 0.0046 (0.004690402116378912)  --> TIME: 2025-02-26 23:49:12 -- STEP: 15968/36938 -- GLOBAL_STEP: 1456550 | > loss_text_ce: 0.03075682744383812 (0.030530262874653526) | > loss_mel_ce: 2.76802659034729 (nan) | > loss: 0.011106283403933048 (nan) | > current_lr: 5e-06 | > step_time: 0.1867 (0.13763697472268538) | > loader_time: 0.0048 (0.004690173990979716)  --> TIME: 2025-02-26 23:49:21 -- STEP: 16018/36938 -- GLOBAL_STEP: 1456600 | > loss_text_ce: 0.026734687387943268 (0.030530181973065562) | > loss_mel_ce: 2.822179079055786 (nan) | > loss: 0.011305213905870914 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13764191444840973) | > loader_time: 0.0046 (0.004690336293980117)  --> TIME: 2025-02-26 23:49:31 -- STEP: 16068/36938 -- GLOBAL_STEP: 1456650 | > loss_text_ce: 0.03887109458446503 (0.03052934592998711) | > loss_mel_ce: 3.1069211959838867 (nan) | > loss: 0.012483303435146809 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.13764885360992307) | > loader_time: 0.0048 (0.004690442938160074)  --> TIME: 2025-02-26 23:49:41 -- STEP: 16118/36938 -- GLOBAL_STEP: 1456700 | > loss_text_ce: 0.030791066586971283 (0.03053034284133716) | > loss_mel_ce: 2.8550736904144287 (nan) | > loss: 0.011451845057308674 (nan) | > current_lr: 5e-06 | > step_time: 0.1708 (0.13766569663165149) | > loader_time: 0.0049 (0.004690522058295012)  --> TIME: 2025-02-26 23:49:50 -- STEP: 16168/36938 -- GLOBAL_STEP: 1456750 | > loss_text_ce: 0.036531493067741394 (0.03052758920190322) | > loss_mel_ce: 2.559791326522827 (nan) | > loss: 0.01030286867171526 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13766149084559853) | > loader_time: 0.0046 (0.004690582698551176)  --> TIME: 2025-02-26 23:49:59 -- STEP: 16218/36938 -- GLOBAL_STEP: 1456800 | > loss_text_ce: 0.03178541362285614 (0.03052870335177052) | > loss_mel_ce: 2.305797815322876 (nan) | > loss: 0.009276124648749828 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13765187680405302) | > loader_time: 0.0047 (0.004690500131325519)  --> TIME: 2025-02-26 23:50:09 -- STEP: 16268/36938 -- GLOBAL_STEP: 1456850 | > loss_text_ce: 0.030434297397732735 (0.030530381930008704) | > loss_mel_ce: 2.6771903038024902 (nan) | > loss: 0.010744542814791203 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13765775282435225) | > loader_time: 0.0046 (0.004690603114241387)  --> TIME: 2025-02-26 23:50:18 -- STEP: 16318/36938 -- GLOBAL_STEP: 1456900 | > loss_text_ce: 0.028365008533000946 (0.030531902561008288) | > loss_mel_ce: 3.2835843563079834 (nan) | > loss: 0.013142656534910202 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.1376625955039088) | > loader_time: 0.0048 (0.004690545784928881)  --> TIME: 2025-02-26 23:50:27 -- STEP: 16368/36938 -- GLOBAL_STEP: 1456950 | > loss_text_ce: 0.03514561429619789 (0.030530476663967198) | > loss_mel_ce: 1.0975861549377441 (nan) | > loss: 0.004494967870414257 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13765851312835048) | > loader_time: 0.0045 (0.004690463693843457)  --> TIME: 2025-02-26 23:50:36 -- STEP: 16418/36938 -- GLOBAL_STEP: 1457000 | > loss_text_ce: 0.0301735308021307 (0.030527134063233992) | > loss_mel_ce: 3.011054754257202 (nan) | > loss: 0.012068366631865501 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13764641257060894) | > loader_time: 0.0046 (0.004690526158819261)  --> TIME: 2025-02-26 23:50:46 -- STEP: 16468/36938 -- GLOBAL_STEP: 1457050 | > loss_text_ce: 0.025007911026477814 (0.030526009227882348) | > loss_mel_ce: 1.320595622062683 (nan) | > loss: 0.005339697003364563 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13763874303029322) | > loader_time: 0.0046 (0.004690513597520385)  --> TIME: 2025-02-26 23:50:55 -- STEP: 16518/36938 -- GLOBAL_STEP: 1457100 | > loss_text_ce: 0.01844089850783348 (0.030528339053855883) | > loss_mel_ce: 2.9457194805145264 (nan) | > loss: 0.011762541718780994 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13763265252358803) | > loader_time: 0.0045 (0.004690736759544689)  --> TIME: 2025-02-26 23:51:04 -- STEP: 16568/36938 -- GLOBAL_STEP: 1457150 | > loss_text_ce: 0.031787607818841934 (0.030527477261921968) | > loss_mel_ce: 2.679497480392456 (nan) | > loss: 0.01075906865298748 (nan) | > current_lr: 5e-06 | > step_time: 0.181 (0.13764203459614513) | > loader_time: 0.0049 (0.004691049837939481)  --> TIME: 2025-02-26 23:51:13 -- STEP: 16618/36938 -- GLOBAL_STEP: 1457200 | > loss_text_ce: 0.03457140922546387 (0.03052586553799218) | > loss_mel_ce: 2.560439109802246 (nan) | > loss: 0.010297661647200584 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13761334282093174) | > loader_time: 0.0045 (0.004690725043844654)  --> TIME: 2025-02-26 23:51:23 -- STEP: 16668/36938 -- GLOBAL_STEP: 1457250 | > loss_text_ce: 0.034185636788606644 (0.030526134613718104) | > loss_mel_ce: 3.038466453552246 (nan) | > loss: 0.012193064205348492 (nan) | > current_lr: 5e-06 | > step_time: 0.1814 (0.13760502962920854) | > loader_time: 0.0048 (0.004690525112376401)  --> TIME: 2025-02-26 23:51:32 -- STEP: 16718/36938 -- GLOBAL_STEP: 1457300 | > loss_text_ce: 0.034531351178884506 (0.03052826862207781) | > loss_mel_ce: 3.4130067825317383 (nan) | > loss: 0.013680707663297653 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.1376180466830931) | > loader_time: 0.0046 (0.0046904276597869365)  --> TIME: 2025-02-26 23:51:41 -- STEP: 16768/36938 -- GLOBAL_STEP: 1457350 | > loss_text_ce: 0.029547661542892456 (0.03053017525335166) | > loss_mel_ce: 2.4137802124023438 (nan) | > loss: 0.009695746004581451 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13760245784775982) | > loader_time: 0.0046 (0.0046902296652320844)  --> TIME: 2025-02-26 23:51:51 -- STEP: 16818/36938 -- GLOBAL_STEP: 1457400 | > loss_text_ce: 0.027017120271921158 (0.030529720543816512) | > loss_mel_ce: 2.913752794265747 (nan) | > loss: 0.011669722385704517 (nan) | > current_lr: 5e-06 | > step_time: 0.1599 (0.13761639953694568) | > loader_time: 0.0046 (0.0046901239738097065)  --> TIME: 2025-02-26 23:52:00 -- STEP: 16868/36938 -- GLOBAL_STEP: 1457450 | > loss_text_ce: 0.03679785504937172 (0.03052909387546559) | > loss_mel_ce: 1.3734952211380005 (nan) | > loss: 0.005596401635557413 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.1376174498655975) | > loader_time: 0.0045 (0.004690065029421112)  --> TIME: 2025-02-26 23:52:10 -- STEP: 16918/36938 -- GLOBAL_STEP: 1457500 | > loss_text_ce: 0.04251686856150627 (0.030528877636371285) | > loss_mel_ce: 0.287109375 (nan) | > loss: 0.0013080406934022903 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13762844943425342) | > loader_time: 0.0046 (0.004690312426033042)  --> TIME: 2025-02-26 23:52:19 -- STEP: 16968/36938 -- GLOBAL_STEP: 1457550 | > loss_text_ce: 0.0260162353515625 (0.030529017138926296) | > loss_mel_ce: 3.006834030151367 (nan) | > loss: 0.012035121209919453 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.13763884257899386) | > loader_time: 0.0047 (0.004690517194987147)  --> TIME: 2025-02-26 23:52:29 -- STEP: 17018/36938 -- GLOBAL_STEP: 1457600 | > loss_text_ce: 0.028930097818374634 (0.03052977835378082) | > loss_mel_ce: 2.07985258102417 (nan) | > loss: 0.008368185721337795 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13763226588838026) | > loader_time: 0.0046 (0.004690534822780291)  --> TIME: 2025-02-26 23:52:38 -- STEP: 17068/36938 -- GLOBAL_STEP: 1457650 | > loss_text_ce: 0.029180606827139854 (0.03053246549077359) | > loss_mel_ce: 2.862861394882202 (nan) | > loss: 0.011476357467472553 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.1376402296225423) | > loader_time: 0.0047 (0.004690474834714827)  --> TIME: 2025-02-26 23:52:48 -- STEP: 17118/36938 -- GLOBAL_STEP: 1457700 | > loss_text_ce: 0.034619804471731186 (0.030530764167507708) | > loss_mel_ce: 3.0006346702575684 (nan) | > loss: 0.012044661678373814 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.1376324847335368) | > loader_time: 0.0046 (0.0046903068655241315)  --> TIME: 2025-02-26 23:52:57 -- STEP: 17168/36938 -- GLOBAL_STEP: 1457750 | > loss_text_ce: 0.032167550176382065 (0.03052958294375277) | > loss_mel_ce: 2.8305907249450684 (nan) | > loss: 0.011360151693224907 (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.13763096849438486) | > loader_time: 0.0048 (0.004690132875478074)  --> TIME: 2025-02-26 23:53:06 -- STEP: 17218/36938 -- GLOBAL_STEP: 1457800 | > loss_text_ce: 0.022216366603970528 (0.03052559822046681) | > loss_mel_ce: 1.7780640125274658 (nan) | > loss: 0.007143970113247633 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13763591288855676) | > loader_time: 0.0045 (0.004690044459904501)  --> TIME: 2025-02-26 23:53:16 -- STEP: 17268/36938 -- GLOBAL_STEP: 1457850 | > loss_text_ce: 0.030656002461910248 (0.03052387121005684) | > loss_mel_ce: 2.6219916343688965 (nan) | > loss: 0.010526380501687527 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13764158460810652) | > loader_time: 0.0046 (0.004689993379512059)  --> TIME: 2025-02-26 23:53:25 -- STEP: 17318/36938 -- GLOBAL_STEP: 1457900 | > loss_text_ce: 0.0352582223713398 (0.030522378547915605) | > loss_mel_ce: 2.507927179336548 (nan) | > loss: 0.01009200606495142 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13764792949887164) | > loader_time: 0.0046 (0.004690053901051211)  --> TIME: 2025-02-26 23:53:35 -- STEP: 17368/36938 -- GLOBAL_STEP: 1457950 | > loss_text_ce: 0.03546600043773651 (0.03051943308451264) | > loss_mel_ce: 2.8232638835906982 (nan) | > loss: 0.011344166472554207 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.13764534626398103) | > loader_time: 0.0047 (0.004689923550664219)  --> TIME: 2025-02-26 23:53:44 -- STEP: 17418/36938 -- GLOBAL_STEP: 1458000 | > loss_text_ce: 0.031256988644599915 (0.030520927753655152) | > loss_mel_ce: 2.5268139839172363 (nan) | > loss: 0.010151075199246407 (nan) | > current_lr: 5e-06 | > step_time: 0.1718 (0.13765054647568603) | > loader_time: 0.0048 (0.004689908969387277)  --> TIME: 2025-02-26 23:53:54 -- STEP: 17468/36938 -- GLOBAL_STEP: 1458050 | > loss_text_ce: 0.02460923232138157 (0.03052105227279393) | > loss_mel_ce: 0.1239013671875 (nan) | > loss: 0.0005893278284929693 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13765573255907185) | > loader_time: 0.0046 (0.004689818051523622)  --> TIME: 2025-02-26 23:54:03 -- STEP: 17518/36938 -- GLOBAL_STEP: 1458100 | > loss_text_ce: 0.030688459053635597 (0.030520127298668084) | > loss_mel_ce: 1.9962785243988037 (nan) | > loss: 0.008043520152568817 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13764095260122805) | > loader_time: 0.0046 (0.004689800465728958)  --> TIME: 2025-02-26 23:54:12 -- STEP: 17568/36938 -- GLOBAL_STEP: 1458150 | > loss_text_ce: 0.02915443852543831 (0.030520321795287905) | > loss_mel_ce: 2.2717459201812744 (nan) | > loss: 0.009130557999014854 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.13764578686223047) | > loader_time: 0.0048 (0.004689647553179889)  --> TIME: 2025-02-26 23:54:22 -- STEP: 17618/36938 -- GLOBAL_STEP: 1458200 | > loss_text_ce: 0.03416791185736656 (0.030519426108056396) | > loss_mel_ce: 2.5534584522247314 (nan) | > loss: 0.010268359445035458 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.1376425678633392) | > loader_time: 0.0046 (0.00468962976618299)  --> TIME: 2025-02-26 23:54:31 -- STEP: 17668/36938 -- GLOBAL_STEP: 1458250 | > loss_text_ce: 0.022665323689579964 (0.03052225610128384) | > loss_mel_ce: 2.8315563201904297 (nan) | > loss: 0.011326276697218418 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13765680606424083) | > loader_time: 0.0046 (0.0046895854554620245)  --> TIME: 2025-02-26 23:54:40 -- STEP: 17718/36938 -- GLOBAL_STEP: 1458300 | > loss_text_ce: 0.023652028292417526 (0.03052524649696842) | > loss_mel_ce: 2.135220527648926 (nan) | > loss: 0.008566955104470253 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.1376395174660123) | > loader_time: 0.0048 (0.004689365413444897)  --> TIME: 2025-02-26 23:54:50 -- STEP: 17768/36938 -- GLOBAL_STEP: 1458350 | > loss_text_ce: 0.03429541364312172 (0.030526049235303755) | > loss_mel_ce: 2.7190072536468506 (nan) | > loss: 0.010925804264843464 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13764436424234797) | > loader_time: 0.0045 (0.00468939945323144)  --> TIME: 2025-02-26 23:54:59 -- STEP: 17818/36938 -- GLOBAL_STEP: 1458400 | > loss_text_ce: 0.026038555428385735 (0.030526100615934724) | > loss_mel_ce: 1.5855035781860352 (nan) | > loss: 0.006395008880645037 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13763401063822103) | > loader_time: 0.0047 (0.004689358168946805)  --> TIME: 2025-02-26 23:55:08 -- STEP: 17868/36938 -- GLOBAL_STEP: 1458450 | > loss_text_ce: 0.029154615476727486 (0.03052766946666564) | > loss_mel_ce: 2.609844207763672 (nan) | > loss: 0.010472217574715614 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13763775953863325) | > loader_time: 0.0046 (0.00468932146563851)  --> TIME: 2025-02-26 23:55:18 -- STEP: 17918/36938 -- GLOBAL_STEP: 1458500 | > loss_text_ce: 0.026852624490857124 (0.030525921789240264) | > loss_mel_ce: 2.4169716835021973 (nan) | > loss: 0.009697715751826763 (nan) | > current_lr: 5e-06 | > step_time: 0.1615 (0.13762421858924134) | > loader_time: 0.0046 (0.004689286058270293)  --> TIME: 2025-02-26 23:55:27 -- STEP: 17968/36938 -- GLOBAL_STEP: 1458550 | > loss_text_ce: 0.02389768324792385 (0.03052195423647686) | > loss_mel_ce: 0.00034356117248535156 (nan) | > loss: 9.619542106520385e-05 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.137628031567814) | > loader_time: 0.0044 (0.00468912358434735)  --> TIME: 2025-02-26 23:55:36 -- STEP: 18018/36938 -- GLOBAL_STEP: 1458600 | > loss_text_ce: 0.02959691919386387 (0.030522030587214553) | > loss_mel_ce: 2.708911180496216 (nan) | > loss: 0.010867095552384853 (nan) | > current_lr: 5e-06 | > step_time: 0.1572 (0.13762027528709686) | > loader_time: 0.0047 (0.004688965717184005)  --> TIME: 2025-02-26 23:55:45 -- STEP: 18068/36938 -- GLOBAL_STEP: 1458650 | > loss_text_ce: 0.03745181858539581 (0.030525283665331946) | > loss_mel_ce: 2.4298269748687744 (nan) | > loss: 0.00979078933596611 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.137615005883438) | > loader_time: 0.0045 (0.004688870083422761)  --> TIME: 2025-02-26 23:55:55 -- STEP: 18118/36938 -- GLOBAL_STEP: 1458700 | > loss_text_ce: 0.03437359631061554 (0.030525907859123336) | > loss_mel_ce: 2.12318754196167 (nan) | > loss: 0.008561750873923302 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13761928666044598) | > loader_time: 0.0045 (0.004688738815988966)  --> TIME: 2025-02-26 23:56:04 -- STEP: 18168/36938 -- GLOBAL_STEP: 1458750 | > loss_text_ce: 0.03023114800453186 (0.03052090263351054) | > loss_mel_ce: 2.708359956741333 (nan) | > loss: 0.010867426171898842 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.1376190515852668) | > loader_time: 0.0046 (0.00468850198793602)  --> TIME: 2025-02-26 23:56:14 -- STEP: 18218/36938 -- GLOBAL_STEP: 1458800 | > loss_text_ce: 0.03613061085343361 (0.030517870709587343) | > loss_mel_ce: 2.943206548690796 (nan) | > loss: 0.011822767555713654 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.1376228305429486) | > loader_time: 0.0046 (0.004688348148771867)  --> TIME: 2025-02-26 23:56:23 -- STEP: 18268/36938 -- GLOBAL_STEP: 1458850 | > loss_text_ce: 0.025001250207424164 (0.03051567281665595) | > loss_mel_ce: 2.5001444816589355 (nan) | > loss: 0.010020419955253601 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13762477408986093) | > loader_time: 0.0047 (0.004688313016751591)  --> TIME: 2025-02-26 23:56:32 -- STEP: 18318/36938 -- GLOBAL_STEP: 1458900 | > loss_text_ce: 0.038273025304079056 (0.03051526752581818) | > loss_mel_ce: 2.947000503540039 (nan) | > loss: 0.011846324428915977 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13761499087119908) | > loader_time: 0.0046 (0.004688221693221179)  --> TIME: 2025-02-26 23:56:42 -- STEP: 18368/36938 -- GLOBAL_STEP: 1458950 | > loss_text_ce: 0.01834900677204132 (0.03051370454604599) | > loss_mel_ce: 2.7714967727661133 (nan) | > loss: 0.011070816777646542 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.1376171619972077) | > loader_time: 0.0045 (0.004687987735254861)  --> TIME: 2025-02-26 23:56:51 -- STEP: 18418/36938 -- GLOBAL_STEP: 1459000 | > loss_text_ce: 0.020071176812052727 (0.030510893180277266) | > loss_mel_ce: 2.688992977142334 (nan) | > loss: 0.010750255547463894 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1376145139995156) | > loader_time: 0.0046 (0.004687883059356375)  --> TIME: 2025-02-26 23:57:00 -- STEP: 18468/36938 -- GLOBAL_STEP: 1459050 | > loss_text_ce: 0.03245680406689644 (0.030511591925424134) | > loss_mel_ce: 2.490281581878662 (nan) | > loss: 0.01001086737960577 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13760765840254843) | > loader_time: 0.0046 (0.00468775491216677)  --> TIME: 2025-02-26 23:57:10 -- STEP: 18518/36938 -- GLOBAL_STEP: 1459100 | > loss_text_ce: 0.03401525318622589 (0.030513413424937283) | > loss_mel_ce: 2.9109954833984375 (nan) | > loss: 0.011686551384627819 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13760439935520025) | > loader_time: 0.0047 (0.004687679909593763)  --> TIME: 2025-02-26 23:57:19 -- STEP: 18568/36938 -- GLOBAL_STEP: 1459150 | > loss_text_ce: 0.03918178379535675 (0.030514564634268066) | > loss_mel_ce: 3.0421247482299805 (nan) | > loss: 0.012227407656610012 (nan) | > current_lr: 5e-06 | > step_time: 0.1268 (0.13760191855734658) | > loader_time: 0.0045 (0.004687591276513991)  --> TIME: 2025-02-26 23:57:28 -- STEP: 18618/36938 -- GLOBAL_STEP: 1459200 | > loss_text_ce: 0.03243469074368477 (0.03051740421606625) | > loss_mel_ce: 2.50312876701355 (nan) | > loss: 0.010061760433018208 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.13760739579622308) | > loader_time: 0.0048 (0.004687541729013977)  --> TIME: 2025-02-26 23:57:38 -- STEP: 18668/36938 -- GLOBAL_STEP: 1459250 | > loss_text_ce: 0.03335390239953995 (0.030521485379979674) | > loss_mel_ce: 2.7097065448760986 (nan) | > loss: 0.01088516041636467 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13760211534410208) | > loader_time: 0.0045 (0.004687415881715822)  --> TIME: 2025-02-26 23:57:47 -- STEP: 18718/36938 -- GLOBAL_STEP: 1459300 | > loss_text_ce: 0.026220280677080154 (0.030521389687348657) | > loss_mel_ce: 2.42714262008667 (nan) | > loss: 0.00973556749522686 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.1376139359260703) | > loader_time: 0.0047 (0.004687357183220649)  --> TIME: 2025-02-26 23:57:57 -- STEP: 18768/36938 -- GLOBAL_STEP: 1459350 | > loss_text_ce: 0.02965962328016758 (0.03052200053781108) | > loss_mel_ce: 2.371976375579834 (nan) | > loss: 0.00953030213713646 (nan) | > current_lr: 5e-06 | > step_time: 0.1876 (0.13763052137459017) | > loader_time: 0.0048 (0.004687240386964539)  --> TIME: 2025-02-26 23:58:06 -- STEP: 18818/36938 -- GLOBAL_STEP: 1459400 | > loss_text_ce: 0.034651730209589005 (0.030520104660190034) | > loss_mel_ce: 2.5669033527374268 (nan) | > loss: 0.010323631577193737 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13762711340897835) | > loader_time: 0.0046 (0.0046870532990110365)  --> TIME: 2025-02-26 23:58:16 -- STEP: 18868/36938 -- GLOBAL_STEP: 1459450 | > loss_text_ce: 0.04052308574318886 (0.030518850473611435) | > loss_mel_ce: 2.422917127609253 (nan) | > loss: 0.0097755566239357 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13763755155564952) | > loader_time: 0.0049 (0.004686870753373414)  --> TIME: 2025-02-26 23:58:25 -- STEP: 18918/36938 -- GLOBAL_STEP: 1459500 | > loss_text_ce: 0.027707315981388092 (0.03051815692083608) | > loss_mel_ce: 2.5302586555480957 (nan) | > loss: 0.010150658898055553 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1376396345530141) | > loader_time: 0.0045 (0.004686694327186186)  --> TIME: 2025-02-26 23:58:35 -- STEP: 18968/36938 -- GLOBAL_STEP: 1459550 | > loss_text_ce: 0.03260713815689087 (0.030521312063248204) | > loss_mel_ce: 2.910503625869751 (nan) | > loss: 0.011679011397063732 (nan) | > current_lr: 5e-06 | > step_time: 0.1866 (0.13764221841199936) | > loader_time: 0.0049 (0.004686501007550706)  --> TIME: 2025-02-26 23:58:44 -- STEP: 19018/36938 -- GLOBAL_STEP: 1459600 | > loss_text_ce: 0.04141860082745552 (0.030521698411763106) | > loss_mel_ce: 2.3804359436035156 (nan) | > loss: 0.009610534645617008 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13764340265251768) | > loader_time: 0.0045 (0.00468628768076444)  --> TIME: 2025-02-26 23:58:54 -- STEP: 19068/36938 -- GLOBAL_STEP: 1459650 | > loss_text_ce: 0.02927662990987301 (0.030522245356869926) | > loss_mel_ce: 2.000418186187744 (nan) | > loss: 0.008054344914853573 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.1376433620443877) | > loader_time: 0.0046 (0.004686076885653136)  --> TIME: 2025-02-26 23:59:03 -- STEP: 19118/36938 -- GLOBAL_STEP: 1459700 | > loss_text_ce: 0.03138258308172226 (0.030520331228156984) | > loss_mel_ce: 3.0750958919525146 (nan) | > loss: 0.01232729572802782 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.1376565924522853) | > loader_time: 0.0046 (0.004685910329969399)  --> TIME: 2025-02-26 23:59:13 -- STEP: 19168/36938 -- GLOBAL_STEP: 1459750 | > loss_text_ce: 0.03131819888949394 (0.0305229926157328) | > loss_mel_ce: 2.5416393280029297 (nan) | > loss: 0.010210149921476841 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13765027757130785) | > loader_time: 0.0045 (0.004685647934924009)  --> TIME: 2025-02-26 23:59:22 -- STEP: 19218/36938 -- GLOBAL_STEP: 1459800 | > loss_text_ce: 0.03211870416998863 (0.030521427981098186) | > loss_mel_ce: 0.7023468017578125 (nan) | > loss: 0.002914545824751258 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13764878488725846) | > loader_time: 0.0046 (0.004685381744341916)  --> TIME: 2025-02-26 23:59:31 -- STEP: 19268/36938 -- GLOBAL_STEP: 1459850 | > loss_text_ce: 0.03815240412950516 (0.030522583541499004) | > loss_mel_ce: 2.855421543121338 (nan) | > loss: 0.011482437141239643 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13763908247792273) | > loader_time: 0.0045 (0.004685087163888373)  --> TIME: 2025-02-26 23:59:40 -- STEP: 19318/36938 -- GLOBAL_STEP: 1459900 | > loss_text_ce: 0.029125411063432693 (0.030523246998661203) | > loss_mel_ce: 3.1821141242980957 (nan) | > loss: 0.012743014842271805 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.1376468634578255) | > loader_time: 0.0046 (0.004684927189790753)  --> TIME: 2025-02-26 23:59:50 -- STEP: 19368/36938 -- GLOBAL_STEP: 1459950 | > loss_text_ce: 0.03471315652132034 (0.030521248360829006) | > loss_mel_ce: 1.3422647714614868 (nan) | > loss: 0.005464198533445597 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13765005705180325) | > loader_time: 0.0045 (0.004684754365340408)  --> TIME: 2025-02-26 23:59:59 -- STEP: 19418/36938 -- GLOBAL_STEP: 1460000 | > loss_text_ce: 0.03447392210364342 (0.030519939651685916) | > loss_mel_ce: 3.0061450004577637 (nan) | > loss: 0.0120659489184618 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13765290430785831) | > loader_time: 0.0045 (0.0046846745176017885) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1460000.pth  --> TIME: 2025-02-27 00:00:15 -- STEP: 19468/36938 -- GLOBAL_STEP: 1460050 | > loss_text_ce: 0.02743060700595379 (0.03051782931915008) | > loss_mel_ce: 0.0064544677734375 (nan) | > loss: 0.00013446460070554167 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13765723780731384) | > loader_time: 0.0046 (0.00468499401596899)  --> TIME: 2025-02-27 00:00:24 -- STEP: 19518/36938 -- GLOBAL_STEP: 1460100 | > loss_text_ce: 0.03532618284225464 (0.03051689987039175) | > loss_mel_ce: 2.221278190612793 (nan) | > loss: 0.008954780176281929 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1376423398934621) | > loader_time: 0.0047 (0.004684816191999987)  --> TIME: 2025-02-27 00:00:33 -- STEP: 19568/36938 -- GLOBAL_STEP: 1460150 | > loss_text_ce: 0.028285454958677292 (0.030515540191001896) | > loss_mel_ce: 2.28385329246521 (nan) | > loss: 0.009175154380500317 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13763433741139106) | > loader_time: 0.0045 (0.004684681592178678)  --> TIME: 2025-02-27 00:00:43 -- STEP: 19618/36938 -- GLOBAL_STEP: 1460200 | > loss_text_ce: 0.027149895206093788 (0.0305171390664355) | > loss_mel_ce: 1.8249332904815674 (nan) | > loss: 0.007349536754190922 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13763843650712887) | > loader_time: 0.0045 (0.004684555103975984)  --> TIME: 2025-02-27 00:00:52 -- STEP: 19668/36938 -- GLOBAL_STEP: 1460250 | > loss_text_ce: 0.03564208373427391 (0.03051859992505607) | > loss_mel_ce: 2.2337758541107178 (nan) | > loss: 0.00900562759488821 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13763103541839142) | > loader_time: 0.0046 (0.004684394056146819)  --> TIME: 2025-02-27 00:01:01 -- STEP: 19718/36938 -- GLOBAL_STEP: 1460300 | > loss_text_ce: 0.023519473150372505 (0.030515571073061704) | > loss_mel_ce: 1.0436370372772217 (nan) | > loss: 0.004234748426824808 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.1376294713315842) | > loader_time: 0.0045 (0.004684650023770242)  --> TIME: 2025-02-27 00:01:11 -- STEP: 19768/36938 -- GLOBAL_STEP: 1460350 | > loss_text_ce: 0.02475634776055813 (0.030512894338571805) | > loss_mel_ce: 2.471874952316284 (nan) | > loss: 0.009907267987728119 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13763250454578863) | > loader_time: 0.0045 (0.004684713894131235)  --> TIME: 2025-02-27 00:01:20 -- STEP: 19818/36938 -- GLOBAL_STEP: 1460400 | > loss_text_ce: 0.035950835794210434 (0.03051014625556084) | > loss_mel_ce: 0.9960843324661255 (nan) | > loss: 0.004095377866178751 (nan) | > current_lr: 5e-06 | > step_time: 0.1513 (0.13762442672730282) | > loader_time: 0.0048 (0.004684822989323414)  --> TIME: 2025-02-27 00:01:29 -- STEP: 19868/36938 -- GLOBAL_STEP: 1460450 | > loss_text_ce: 0.03253219276666641 (0.03050756360414412) | > loss_mel_ce: 2.7256369590759277 (nan) | > loss: 0.01094511616975069 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.13763376695422908) | > loader_time: 0.0047 (0.004684880258860886)  --> TIME: 2025-02-27 00:01:39 -- STEP: 19918/36938 -- GLOBAL_STEP: 1460500 | > loss_text_ce: 0.02135077863931656 (0.030506744952640476) | > loss_mel_ce: 2.35036563873291 (nan) | > loss: 0.009411574341356754 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1376451044535588) | > loader_time: 0.0046 (0.004684957206841785)  --> TIME: 2025-02-27 00:01:49 -- STEP: 19968/36938 -- GLOBAL_STEP: 1460550 | > loss_text_ce: 0.029005182906985283 (0.030506633493902112) | > loss_mel_ce: 3.0565359592437744 (nan) | > loss: 0.012244212441146374 (nan) | > current_lr: 5e-06 | > step_time: 0.1569 (0.13765560968134288) | > loader_time: 0.0047 (0.00468512735544492)  --> TIME: 2025-02-27 00:01:58 -- STEP: 20018/36938 -- GLOBAL_STEP: 1460600 | > loss_text_ce: 0.022562554106116295 (0.03050633659802648) | > loss_mel_ce: 2.9902989864349365 (nan) | > loss: 0.011955800466239452 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.13764413476931228) | > loader_time: 0.0047 (0.004685214914300396)  --> TIME: 2025-02-27 00:02:07 -- STEP: 20068/36938 -- GLOBAL_STEP: 1460650 | > loss_text_ce: 0.030753765255212784 (0.030505410813214506) | > loss_mel_ce: 0.5969412922859192 (nan) | > loss: 0.0024908536579459906 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13765160083913344) | > loader_time: 0.0047 (0.004685394526811082)  --> TIME: 2025-02-27 00:02:17 -- STEP: 20118/36938 -- GLOBAL_STEP: 1460700 | > loss_text_ce: 0.031128082424402237 (0.030505376001784785) | > loss_mel_ce: 2.1245357990264893 (nan) | > loss: 0.008554222993552685 (nan) | > current_lr: 5e-06 | > step_time: 0.1461 (0.13765799367578363) | > loader_time: 0.0047 (0.00468547643564229)  --> TIME: 2025-02-27 00:02:27 -- STEP: 20168/36938 -- GLOBAL_STEP: 1460750 | > loss_text_ce: 0.03179701417684555 (0.030505054436921624) | > loss_mel_ce: 2.888549566268921 (nan) | > loss: 0.01158867683261633 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.1376601391985796) | > loader_time: 0.0046 (0.004685575836284567)  --> TIME: 2025-02-27 00:02:36 -- STEP: 20218/36938 -- GLOBAL_STEP: 1460800 | > loss_text_ce: 0.0264647975564003 (0.030503196514366646) | > loss_mel_ce: 2.355787515640259 (nan) | > loss: 0.009453382343053818 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13767736916113316) | > loader_time: 0.0047 (0.004685586915547808)  --> TIME: 2025-02-27 00:02:46 -- STEP: 20268/36938 -- GLOBAL_STEP: 1460850 | > loss_text_ce: 0.0319194421172142 (0.030500774423031934) | > loss_mel_ce: 2.3131656646728516 (nan) | > loss: 0.009305894374847412 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13767605230656318) | > loader_time: 0.0047 (0.004685539205986707)  --> TIME: 2025-02-27 00:02:55 -- STEP: 20318/36938 -- GLOBAL_STEP: 1460900 | > loss_text_ce: 0.033336423337459564 (0.03050231418242521) | > loss_mel_ce: 2.5041251182556152 (nan) | > loss: 0.010069292038679123 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13767165536427123) | > loader_time: 0.0045 (0.004685657091738767)  --> TIME: 2025-02-27 00:03:04 -- STEP: 20368/36938 -- GLOBAL_STEP: 1460950 | > loss_text_ce: 0.02567434310913086 (0.03049953267089741) | > loss_mel_ce: 2.7560741901397705 (nan) | > loss: 0.011038685217499733 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13766818498872194) | > loader_time: 0.0047 (0.004685766087773306)  --> TIME: 2025-02-27 00:03:13 -- STEP: 20418/36938 -- GLOBAL_STEP: 1461000 | > loss_text_ce: 0.038466740399599075 (0.03050046623830427) | > loss_mel_ce: 1.51092529296875 (nan) | > loss: 0.006148381158709526 (nan) | > current_lr: 5e-06 | > step_time: 0.1496 (0.1376611957681769) | > loader_time: 0.0047 (0.004685787965903669)  --> TIME: 2025-02-27 00:03:23 -- STEP: 20468/36938 -- GLOBAL_STEP: 1461050 | > loss_text_ce: 0.031701937317848206 (0.030501096715177093) | > loss_mel_ce: 2.031905174255371 (nan) | > loss: 0.008188918232917786 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13766031149241198) | > loader_time: 0.0047 (0.004685795316478143)  --> TIME: 2025-02-27 00:03:32 -- STEP: 20518/36938 -- GLOBAL_STEP: 1461100 | > loss_text_ce: 0.03417091444134712 (0.03049912832823794) | > loss_mel_ce: 2.060384511947632 (nan) | > loss: 0.00831172801554203 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.1376644990076764) | > loader_time: 0.0047 (0.004685782528676589)  --> TIME: 2025-02-27 00:03:42 -- STEP: 20568/36938 -- GLOBAL_STEP: 1461150 | > loss_text_ce: 0.03456347808241844 (0.030500742620425368) | > loss_mel_ce: 0.1125946044921875 (nan) | > loss: 0.0005839606747031212 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13766617127681552) | > loader_time: 0.0046 (0.004685811289828494)  --> TIME: 2025-02-27 00:03:51 -- STEP: 20618/36938 -- GLOBAL_STEP: 1461200 | > loss_text_ce: 0.032757531851530075 (0.03050650050093469) | > loss_mel_ce: 3.3738746643066406 (nan) | > loss: 0.013518382795155048 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13767815229443872) | > loader_time: 0.0046 (0.004685941532518759)  --> TIME: 2025-02-27 00:04:01 -- STEP: 20668/36938 -- GLOBAL_STEP: 1461250 | > loss_text_ce: 0.0268965195864439 (0.030506093179761464) | > loss_mel_ce: 1.5609842538833618 (nan) | > loss: 0.006301114335656166 (nan) | > current_lr: 5e-06 | > step_time: 0.1537 (0.1376760395509447) | > loader_time: 0.0047 (0.00468586572992612)  --> TIME: 2025-02-27 00:04:10 -- STEP: 20718/36938 -- GLOBAL_STEP: 1461300 | > loss_text_ce: 0.026888199150562286 (0.03050318899165628) | > loss_mel_ce: 1.52892005443573 (nan) | > loss: 0.006173843052238226 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13767072088597143) | > loader_time: 0.0045 (0.004685771305343131)  --> TIME: 2025-02-27 00:04:19 -- STEP: 20768/36938 -- GLOBAL_STEP: 1461350 | > loss_text_ce: 0.03991576284170151 (0.03050210317136244) | > loss_mel_ce: 2.691019296646118 (nan) | > loss: 0.010837044566869736 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13767503456819624) | > loader_time: 0.0046 (0.0046856568319772)  --> TIME: 2025-02-27 00:04:29 -- STEP: 20818/36938 -- GLOBAL_STEP: 1461400 | > loss_text_ce: 0.02719247341156006 (0.030501841841892228) | > loss_mel_ce: 2.1120481491088867 (nan) | > loss: 0.008489050902426243 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.1376709353970756) | > loader_time: 0.0045 (0.004685692799081054)  --> TIME: 2025-02-27 00:04:38 -- STEP: 20868/36938 -- GLOBAL_STEP: 1461450 | > loss_text_ce: 0.026798877865076065 (0.030502006089619546) | > loss_mel_ce: 2.1578621864318848 (nan) | > loss: 0.008669290691614151 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.1376567404800756) | > loader_time: 0.0045 (0.004685856954608606)  --> TIME: 2025-02-27 00:04:47 -- STEP: 20918/36938 -- GLOBAL_STEP: 1461500 | > loss_text_ce: 0.03544055297970772 (0.030503755722540226) | > loss_mel_ce: 2.8548595905303955 (nan) | > loss: 0.011469445191323757 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.1376561174224928) | > loader_time: 0.0047 (0.004686084004726884)  --> TIME: 2025-02-27 00:04:57 -- STEP: 20968/36938 -- GLOBAL_STEP: 1461550 | > loss_text_ce: 0.0277363583445549 (0.030503245142020408) | > loss_mel_ce: 2.601588010787964 (nan) | > loss: 0.01043382752686739 (nan) | > current_lr: 5e-06 | > step_time: 0.1349 (0.1376655550232285) | > loader_time: 0.0047 (0.004686269890666261)  --> TIME: 2025-02-27 00:05:06 -- STEP: 21018/36938 -- GLOBAL_STEP: 1461600 | > loss_text_ce: 0.02909187413752079 (0.030506623280041882) | > loss_mel_ce: 2.1463379859924316 (nan) | > loss: 0.0086326589807868 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13766275236955772) | > loader_time: 0.0044 (0.0046861779963168464)  --> TIME: 2025-02-27 00:05:16 -- STEP: 21068/36938 -- GLOBAL_STEP: 1461650 | > loss_text_ce: 0.025363478809595108 (0.030504781780607972) | > loss_mel_ce: 2.2273175716400146 (nan) | > loss: 0.008939211256802082 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.1376595686232093) | > loader_time: 0.0046 (0.00468612232130396)  --> TIME: 2025-02-27 00:05:25 -- STEP: 21118/36938 -- GLOBAL_STEP: 1461700 | > loss_text_ce: 0.027388600632548332 (0.03050453631234778) | > loss_mel_ce: 2.3748953342437744 (nan) | > loss: 0.009532873518764973 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.1376599062120207) | > loader_time: 0.0045 (0.0046860910927395)  --> TIME: 2025-02-27 00:05:35 -- STEP: 21168/36938 -- GLOBAL_STEP: 1461750 | > loss_text_ce: 0.028685251250863075 (0.03050586645344384) | > loss_mel_ce: 2.248100519180298 (nan) | > loss: 0.009034864604473114 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.1376704447709153) | > loader_time: 0.0046 (0.004685933379990022)  --> TIME: 2025-02-27 00:05:44 -- STEP: 21218/36938 -- GLOBAL_STEP: 1461800 | > loss_text_ce: 0.0284073855727911 (0.030505622597112803) | > loss_mel_ce: 2.8013620376586914 (nan) | > loss: 0.01122924406081438 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.137671062280887) | > loader_time: 0.0046 (0.004685774241870067)  --> TIME: 2025-02-27 00:05:53 -- STEP: 21268/36938 -- GLOBAL_STEP: 1461850 | > loss_text_ce: 0.019986450672149658 (0.03050552825238072) | > loss_mel_ce: 0.035888671875 (nan) | > loss: 0.00022172668832354248 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.1376698882435864) | > loader_time: 0.0045 (0.004685566168385688)  --> TIME: 2025-02-27 00:06:03 -- STEP: 21318/36938 -- GLOBAL_STEP: 1461900 | > loss_text_ce: 0.023673074319958687 (0.030504860897178993) | > loss_mel_ce: 2.508775472640991 (nan) | > loss: 0.010049399919807911 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13767340309859907) | > loader_time: 0.0045 (0.004685413659608737)  --> TIME: 2025-02-27 00:06:12 -- STEP: 21368/36938 -- GLOBAL_STEP: 1461950 | > loss_text_ce: 0.03222371265292168 (0.030506026959720545) | > loss_mel_ce: 2.8390932083129883 (nan) | > loss: 0.011394115164875984 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13767525678669995) | > loader_time: 0.0044 (0.004685281524493992)  --> TIME: 2025-02-27 00:06:22 -- STEP: 21418/36938 -- GLOBAL_STEP: 1462000 | > loss_text_ce: 0.029621031135320663 (0.030503979693305622) | > loss_mel_ce: 2.7487339973449707 (nan) | > loss: 0.011025219224393368 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13766398934358895) | > loader_time: 0.0046 (0.004685054418474999)  --> TIME: 2025-02-27 00:06:31 -- STEP: 21468/36938 -- GLOBAL_STEP: 1462050 | > loss_text_ce: 0.03717987611889839 (0.030504504263402593) | > loss_mel_ce: 2.0657734870910645 (nan) | > loss: 0.008345053531229496 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.13766138768404895) | > loader_time: 0.0046 (0.004684933930641035)  --> TIME: 2025-02-27 00:06:40 -- STEP: 21518/36938 -- GLOBAL_STEP: 1462100 | > loss_text_ce: 0.03657311573624611 (0.030504192642020814) | > loss_mel_ce: 2.6098740100860596 (nan) | > loss: 0.010501774959266186 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.13766774561494036) | > loader_time: 0.0046 (0.004684772962575007)  --> TIME: 2025-02-27 00:06:50 -- STEP: 21568/36938 -- GLOBAL_STEP: 1462150 | > loss_text_ce: 0.015889953821897507 (0.030502735820322716) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13766256743041622) | > loader_time: 0.0045 (0.0046845849503926364)  --> TIME: 2025-02-27 00:06:59 -- STEP: 21618/36938 -- GLOBAL_STEP: 1462200 | > loss_text_ce: 0.0227553341537714 (0.030503366919195106) | > loss_mel_ce: 2.270206928253174 (nan) | > loss: 0.00909905694425106 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13766270237920109) | > loader_time: 0.0045 (0.004684405847839389)  --> TIME: 2025-02-27 00:07:08 -- STEP: 21668/36938 -- GLOBAL_STEP: 1462250 | > loss_text_ce: 0.03304734453558922 (0.030500897191000187) | > loss_mel_ce: 2.416943073272705 (nan) | > loss: 0.00972218532115221 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.1376604754501809) | > loader_time: 0.0048 (0.004684254518839564)  --> TIME: 2025-02-27 00:07:18 -- STEP: 21718/36938 -- GLOBAL_STEP: 1462300 | > loss_text_ce: 0.02640640363097191 (0.03050234365120557) | > loss_mel_ce: 2.2190897464752197 (nan) | > loss: 0.00891069881618023 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13766838383615337) | > loader_time: 0.0063 (0.004684311479195614)  --> TIME: 2025-02-27 00:07:27 -- STEP: 21768/36938 -- GLOBAL_STEP: 1462350 | > loss_text_ce: 0.030505627393722534 (0.030500500227829686) | > loss_mel_ce: 3.2177250385284424 (nan) | > loss: 0.012889805249869823 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13767087353322935) | > loader_time: 0.0046 (0.004684154096222795)  --> TIME: 2025-02-27 00:07:37 -- STEP: 21818/36938 -- GLOBAL_STEP: 1462400 | > loss_text_ce: 0.030304815620183945 (0.030500548986678106) | > loss_mel_ce: 2.8725411891937256 (nan) | > loss: 0.011519230902194977 (nan) | > current_lr: 5e-06 | > step_time: 0.1642 (0.13767734970680603) | > loader_time: 0.0046 (0.004684029900521763)  --> TIME: 2025-02-27 00:07:46 -- STEP: 21868/36938 -- GLOBAL_STEP: 1462450 | > loss_text_ce: 0.02704177238047123 (0.030499579317143585) | > loss_mel_ce: 2.615786552429199 (nan) | > loss: 0.010487413965165615 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13767556041255619) | > loader_time: 0.0045 (0.004683917306209618)  --> TIME: 2025-02-27 00:07:56 -- STEP: 21918/36938 -- GLOBAL_STEP: 1462500 | > loss_text_ce: 0.029447302222251892 (0.03050049166932742) | > loss_mel_ce: 1.968329668045044 (nan) | > loss: 0.007927686907351017 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13767832198361885) | > loader_time: 0.0046 (0.00468402889395125)  --> TIME: 2025-02-27 00:08:05 -- STEP: 21968/36938 -- GLOBAL_STEP: 1462550 | > loss_text_ce: 0.027368320152163506 (0.030498825258275058) | > loss_mel_ce: 1.7098586559295654 (nan) | > loss: 0.006893758196383715 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13767863790527024) | > loader_time: 0.0045 (0.00468404143940164)  --> TIME: 2025-02-27 00:08:15 -- STEP: 22018/36938 -- GLOBAL_STEP: 1462600 | > loss_text_ce: 0.024003788828849792 (0.030498126694389133) | > loss_mel_ce: 2.2831153869628906 (nan) | > loss: 0.00915523525327444 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13768581933011068) | > loader_time: 0.0047 (0.004684041052967415)  --> TIME: 2025-02-27 00:08:24 -- STEP: 22068/36938 -- GLOBAL_STEP: 1462650 | > loss_text_ce: 0.033803801983594894 (0.030499917922092633) | > loss_mel_ce: 2.201910972595215 (nan) | > loss: 0.00887188408523798 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13768970528784621) | > loader_time: 0.0046 (0.004684249873348906)  --> TIME: 2025-02-27 00:08:33 -- STEP: 22118/36938 -- GLOBAL_STEP: 1462700 | > loss_text_ce: 0.034361112862825394 (0.03049952398464087) | > loss_mel_ce: 0.14890137314796448 (nan) | > loss: 0.0007272321381606162 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13768044456089037) | > loader_time: 0.0045 (0.004684165617324284)  --> TIME: 2025-02-27 00:08:43 -- STEP: 22168/36938 -- GLOBAL_STEP: 1462750 | > loss_text_ce: 0.03332867845892906 (0.030499437435553486) | > loss_mel_ce: 2.9930989742279053 (nan) | > loss: 0.012009634636342525 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13768321169235775) | > loader_time: 0.0045 (0.004684280549004328)  --> TIME: 2025-02-27 00:08:52 -- STEP: 22218/36938 -- GLOBAL_STEP: 1462800 | > loss_text_ce: 0.024682968854904175 (0.030500085347624897) | > loss_mel_ce: 2.748572587966919 (nan) | > loss: 0.011004983447492123 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.13769236992240128) | > loader_time: 0.0046 (0.004684272931883873)  --> TIME: 2025-02-27 00:09:01 -- STEP: 22268/36938 -- GLOBAL_STEP: 1462850 | > loss_text_ce: 0.02673140913248062 (0.030498617465110293) | > loss_mel_ce: 3.096670389175415 (nan) | > loss: 0.012394452467560768 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.1376845937390205) | > loader_time: 0.0047 (0.00468418712523464)  --> TIME: 2025-02-27 00:09:11 -- STEP: 22318/36938 -- GLOBAL_STEP: 1462900 | > loss_text_ce: 0.025408772751688957 (0.030498878065703162) | > loss_mel_ce: 3.2180581092834473 (nan) | > loss: 0.012870901264250278 (nan) | > current_lr: 5e-06 | > step_time: 0.162 (0.13769716611418292) | > loader_time: 0.0048 (0.004684238239838404)  --> TIME: 2025-02-27 00:09:20 -- STEP: 22368/36938 -- GLOBAL_STEP: 1462950 | > loss_text_ce: 0.027285506948828697 (0.030495520592852862) | > loss_mel_ce: 2.769118547439575 (nan) | > loss: 0.011096842586994171 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13768773723143185) | > loader_time: 0.0048 (0.004684171632721056)  --> TIME: 2025-02-27 00:09:30 -- STEP: 22418/36938 -- GLOBAL_STEP: 1463000 | > loss_text_ce: 0.033158719539642334 (0.030492808534976042) | > loss_mel_ce: 2.3787360191345215 (nan) | > loss: 0.009571012109518051 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13768667261823192) | > loader_time: 0.0047 (0.004684149681889067)  --> TIME: 2025-02-27 00:09:39 -- STEP: 22468/36938 -- GLOBAL_STEP: 1463050 | > loss_text_ce: 0.036830488592386246 (0.030495901606189913) | > loss_mel_ce: 2.642223834991455 (nan) | > loss: 0.010631168261170387 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13767671570226944) | > loader_time: 0.0045 (0.004683939676724568)  --> TIME: 2025-02-27 00:09:47 -- STEP: 22518/36938 -- GLOBAL_STEP: 1463100 | > loss_text_ce: 0.03431422635912895 (0.03049208736119527) | > loss_mel_ce: 2.2946906089782715 (nan) | > loss: 0.009242082946002483 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13766241356305617) | > loader_time: 0.0047 (0.004683771039402237)  --> TIME: 2025-02-27 00:09:57 -- STEP: 22568/36938 -- GLOBAL_STEP: 1463150 | > loss_text_ce: 0.034981898963451385 (0.030494649766049507) | > loss_mel_ce: 3.2327473163604736 (nan) | > loss: 0.012967180460691452 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13765832091932892) | > loader_time: 0.0046 (0.004683692471772335)  --> TIME: 2025-02-27 00:10:06 -- STEP: 22618/36938 -- GLOBAL_STEP: 1463200 | > loss_text_ce: 0.029547419399023056 (0.030494939778651524) | > loss_mel_ce: 2.4927988052368164 (nan) | > loss: 0.010009311139583588 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13765864326294225) | > loader_time: 0.0045 (0.00468383397017947)  --> TIME: 2025-02-27 00:10:15 -- STEP: 22668/36938 -- GLOBAL_STEP: 1463250 | > loss_text_ce: 0.025945283472537994 (0.030495733131915205) | > loss_mel_ce: 2.414442300796509 (nan) | > loss: 0.009684077464044094 (nan) | > current_lr: 5e-06 | > step_time: 0.181 (0.13766155558245) | > loader_time: 0.0049 (0.004683859779108222)  --> TIME: 2025-02-27 00:10:25 -- STEP: 22718/36938 -- GLOBAL_STEP: 1463300 | > loss_text_ce: 0.029300659894943237 (0.030496556292844635) | > loss_mel_ce: 3.0195791721343994 (nan) | > loss: 0.01209873054176569 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.1376590982521595) | > loader_time: 0.0047 (0.0046839401937935725)  --> TIME: 2025-02-27 00:10:34 -- STEP: 22768/36938 -- GLOBAL_STEP: 1463350 | > loss_text_ce: 0.03461708873510361 (0.030496817186985457) | > loss_mel_ce: 2.375671863555908 (nan) | > loss: 0.009564640000462532 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13765926818276325) | > loader_time: 0.0045 (0.004684030329016724)  --> TIME: 2025-02-27 00:10:43 -- STEP: 22818/36938 -- GLOBAL_STEP: 1463400 | > loss_text_ce: 0.04490078240633011 (0.030496888271753272) | > loss_mel_ce: 0.041748046875 (nan) | > loss: 0.00034384458558633924 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13765482428573547) | > loader_time: 0.0044 (0.004684227168460815)  --> TIME: 2025-02-27 00:10:53 -- STEP: 22868/36938 -- GLOBAL_STEP: 1463450 | > loss_text_ce: 0.025142626836895943 (0.03049646383334272) | > loss_mel_ce: 3.0768604278564453 (nan) | > loss: 0.012309537269175053 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.1376735737779581) | > loader_time: 0.0047 (0.004684263568919594)  --> TIME: 2025-02-27 00:11:03 -- STEP: 22918/36938 -- GLOBAL_STEP: 1463500 | > loss_text_ce: 0.03634736314415932 (0.03049264812957177) | > loss_mel_ce: 2.2055089473724365 (nan) | > loss: 0.008896255865693092 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.1376704419405116) | > loader_time: 0.0045 (0.004684157256670667)  --> TIME: 2025-02-27 00:11:12 -- STEP: 22968/36938 -- GLOBAL_STEP: 1463550 | > loss_text_ce: 0.03212974965572357 (0.030492714512618813) | > loss_mel_ce: 2.8338863849639893 (nan) | > loss: 0.011373080313205719 (nan) | > current_lr: 5e-06 | > step_time: 0.176 (0.13767106040601185) | > loader_time: 0.0048 (0.004684069822242365)  --> TIME: 2025-02-27 00:11:21 -- STEP: 23018/36938 -- GLOBAL_STEP: 1463600 | > loss_text_ce: 0.03238451108336449 (0.030493466307413397) | > loss_mel_ce: 2.3621232509613037 (nan) | > loss: 0.009502015076577663 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13766239555137497) | > loader_time: 0.0045 (0.004684175373169918)  --> TIME: 2025-02-27 00:11:31 -- STEP: 23068/36938 -- GLOBAL_STEP: 1463650 | > loss_text_ce: 0.03643296658992767 (0.03049463161018401) | > loss_mel_ce: 2.6681737899780273 (nan) | > loss: 0.010732566937804222 (nan) | > current_lr: 5e-06 | > step_time: 0.1512 (0.1376635561746764) | > loader_time: 0.0047 (0.004684185235529735)  --> TIME: 2025-02-27 00:11:40 -- STEP: 23118/36938 -- GLOBAL_STEP: 1463700 | > loss_text_ce: 0.03285433351993561 (0.03049528265859133) | > loss_mel_ce: 2.3799352645874023 (nan) | > loss: 0.009574562311172485 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1376480800757537) | > loader_time: 0.0046 (0.004684094419858113)  --> TIME: 2025-02-27 00:11:49 -- STEP: 23168/36938 -- GLOBAL_STEP: 1463750 | > loss_text_ce: 0.0324564054608345 (0.03049296607604351) | > loss_mel_ce: 1.6880205869674683 (nan) | > loss: 0.006827290169894695 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.1376399851265206) | > loader_time: 0.0045 (0.004684127270352162)  --> TIME: 2025-02-27 00:11:58 -- STEP: 23218/36938 -- GLOBAL_STEP: 1463800 | > loss_text_ce: 0.03605325147509575 (0.030494966651570928) | > loss_mel_ce: 2.232396125793457 (nan) | > loss: 0.009001783095300198 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13763395911686738) | > loader_time: 0.0045 (0.004684255252318666)  --> TIME: 2025-02-27 00:12:07 -- STEP: 23268/36938 -- GLOBAL_STEP: 1463850 | > loss_text_ce: 0.02949456311762333 (0.030495098792596882) | > loss_mel_ce: 2.604580879211426 (nan) | > loss: 0.010452681221067905 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.1376317608329836) | > loader_time: 0.0046 (0.004684251896269858)  --> TIME: 2025-02-27 00:12:16 -- STEP: 23318/36938 -- GLOBAL_STEP: 1463900 | > loss_text_ce: 0.028967076912522316 (0.030492020289569637) | > loss_mel_ce: 1.6448456048965454 (nan) | > loss: 0.006642113905400038 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13761656805176375) | > loader_time: 0.0046 (0.004684109775328295)  --> TIME: 2025-02-27 00:12:25 -- STEP: 23368/36938 -- GLOBAL_STEP: 1463950 | > loss_text_ce: 0.030836356803774834 (0.030488013736180705) | > loss_mel_ce: 2.672088623046875 (nan) | > loss: 0.010725893080234528 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.13761665344809637) | > loader_time: 0.0045 (0.004684167849368696)  --> TIME: 2025-02-27 00:12:35 -- STEP: 23418/36938 -- GLOBAL_STEP: 1464000 | > loss_text_ce: 0.025730865076184273 (0.030489121419707315) | > loss_mel_ce: 2.3338067531585693 (nan) | > loss: 0.009363245218992233 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13760793228419826) | > loader_time: 0.0046 (0.0046841813371798725)  --> TIME: 2025-02-27 00:12:44 -- STEP: 23468/36938 -- GLOBAL_STEP: 1464050 | > loss_text_ce: 0.024750692769885063 (0.03048825836860679) | > loss_mel_ce: 2.4234225749969482 (nan) | > loss: 0.009714974090456963 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.13760677924326048) | > loader_time: 0.0046 (0.004684022069485371)  --> TIME: 2025-02-27 00:12:54 -- STEP: 23518/36938 -- GLOBAL_STEP: 1464100 | > loss_text_ce: 0.039711546152830124 (0.030490050175203384) | > loss_mel_ce: 2.7552120685577393 (nan) | > loss: 0.011090966872870922 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13761658390339396) | > loader_time: 0.0045 (0.00468393988690015)  --> TIME: 2025-02-27 00:13:03 -- STEP: 23568/36938 -- GLOBAL_STEP: 1464150 | > loss_text_ce: 0.022692861035466194 (0.030490060182240396) | > loss_mel_ce: 1.910343885421753 (nan) | > loss: 0.007670781575143337 (nan) | > current_lr: 5e-06 | > step_time: 0.1261 (0.13761023773438072) | > loader_time: 0.0046 (0.0046838504557535614)  --> TIME: 2025-02-27 00:13:12 -- STEP: 23618/36938 -- GLOBAL_STEP: 1464200 | > loss_text_ce: 0.02807696908712387 (0.030491312459375097) | > loss_mel_ce: 2.6273438930511475 (nan) | > loss: 0.010537385009229183 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13760696845289205) | > loader_time: 0.0045 (0.004683834015036052)  --> TIME: 2025-02-27 00:13:22 -- STEP: 23668/36938 -- GLOBAL_STEP: 1464250 | > loss_text_ce: 0.024609042331576347 (0.030493052660127782) | > loss_mel_ce: 2.8401505947113037 (nan) | > loss: 0.011368094943463802 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13761284434152665) | > loader_time: 0.0046 (0.004684094996272668)  --> TIME: 2025-02-27 00:13:31 -- STEP: 23718/36938 -- GLOBAL_STEP: 1464300 | > loss_text_ce: 0.02906220406293869 (0.030495120564597224) | > loss_mel_ce: 3.9140625 (nan) | > loss: 0.015647321939468384 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.1376117897813741) | > loader_time: 0.0046 (0.004684024591330974)  --> TIME: 2025-02-27 00:13:41 -- STEP: 23768/36938 -- GLOBAL_STEP: 1464350 | > loss_text_ce: 0.02722286805510521 (0.030495322991410338) | > loss_mel_ce: 2.649078369140625 (nan) | > loss: 0.010620243847370148 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13761556156553892) | > loader_time: 0.0045 (0.004684083231448051)  --> TIME: 2025-02-27 00:13:50 -- STEP: 23818/36938 -- GLOBAL_STEP: 1464400 | > loss_text_ce: 0.027400735765695572 (0.030495079284859495) | > loss_mel_ce: 0.9888216257095337 (nan) | > loss: 0.004032628610730171 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13761590434838442) | > loader_time: 0.0046 (0.0046842424362430355)  --> TIME: 2025-02-27 00:13:59 -- STEP: 23868/36938 -- GLOBAL_STEP: 1464450 | > loss_text_ce: 0.04085339605808258 (0.03049364363902564) | > loss_mel_ce: 2.8984410762786865 (nan) | > loss: 0.011663868092000484 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13760920530470727) | > loader_time: 0.0047 (0.004684159059270462)  --> TIME: 2025-02-27 00:14:09 -- STEP: 23918/36938 -- GLOBAL_STEP: 1464500 | > loss_text_ce: 0.017383970320224762 (0.03049468933402958) | > loss_mel_ce: 2.778942584991455 (nan) | > loss: 0.011096535250544548 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13761551426609125) | > loader_time: 0.0046 (0.004684218505880477)  --> TIME: 2025-02-27 00:14:18 -- STEP: 23968/36938 -- GLOBAL_STEP: 1464550 | > loss_text_ce: 0.027912314981222153 (0.030495503444661502) | > loss_mel_ce: 2.401247501373291 (nan) | > loss: 0.009639523923397064 (nan) | > current_lr: 5e-06 | > step_time: 0.1662 (0.1376198950771972) | > loader_time: 0.0048 (0.0046842073467131635)  --> TIME: 2025-02-27 00:14:28 -- STEP: 24018/36938 -- GLOBAL_STEP: 1464600 | > loss_text_ce: 0.023114852607250214 (0.03049601809912234) | > loss_mel_ce: 3.0999042987823486 (nan) | > loss: 0.012392934411764145 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.1376277558243097) | > loader_time: 0.0046 (0.004684305248614733)  --> TIME: 2025-02-27 00:14:37 -- STEP: 24068/36938 -- GLOBAL_STEP: 1464650 | > loss_text_ce: 0.03705886751413345 (0.030496515760598964) | > loss_mel_ce: 2.7620909214019775 (nan) | > loss: 0.011107738129794598 (nan) | > current_lr: 5e-06 | > step_time: 0.1563 (0.13763119990353712) | > loader_time: 0.0047 (0.004684297868495069)  --> TIME: 2025-02-27 00:14:47 -- STEP: 24118/36938 -- GLOBAL_STEP: 1464700 | > loss_text_ce: 0.029392611235380173 (0.030495306203458146) | > loss_mel_ce: 2.460164785385132 (nan) | > loss: 0.009879196994006634 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.137631091997194) | > loader_time: 0.0046 (0.004684246854705451)  --> TIME: 2025-02-27 00:14:56 -- STEP: 24168/36938 -- GLOBAL_STEP: 1464750 | > loss_text_ce: 0.03979433327913284 (0.03049679261539239) | > loss_mel_ce: 2.749843120574951 (nan) | > loss: 0.011069990694522858 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13761781563982953) | > loader_time: 0.0046 (0.004684169278244331)  --> TIME: 2025-02-27 00:15:05 -- STEP: 24218/36938 -- GLOBAL_STEP: 1464800 | > loss_text_ce: 0.020712077617645264 (0.03049467273180104) | > loss_mel_ce: 2.14054012298584 (nan) | > loss: 0.008576398715376854 (nan) | > current_lr: 5e-06 | > step_time: 0.1077 (0.1376058108368125) | > loader_time: 0.0045 (0.0046840933216072515)  --> TIME: 2025-02-27 00:15:14 -- STEP: 24268/36938 -- GLOBAL_STEP: 1464850 | > loss_text_ce: 0.03421425819396973 (0.030493244212988848) | > loss_mel_ce: 2.499178886413574 (nan) | > loss: 0.010053147561848164 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.137606583583565) | > loader_time: 0.0045 (0.004684245987230906)  --> TIME: 2025-02-27 00:15:24 -- STEP: 24318/36938 -- GLOBAL_STEP: 1464900 | > loss_text_ce: 0.032621175050735474 (0.030493178376086956) | > loss_mel_ce: 2.9413623809814453 (nan) | > loss: 0.011801522225141525 (nan) | > current_lr: 5e-06 | > step_time: 0.1524 (0.1376152475743575) | > loader_time: 0.0046 (0.004684216431640462)  --> TIME: 2025-02-27 00:15:33 -- STEP: 24368/36938 -- GLOBAL_STEP: 1464950 | > loss_text_ce: 0.0273401141166687 (0.030495537263427623) | > loss_mel_ce: 1.8155251741409302 (nan) | > loss: 0.007312957663089037 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13761896081880284) | > loader_time: 0.0046 (0.004684207426507672)  --> TIME: 2025-02-27 00:15:43 -- STEP: 24418/36938 -- GLOBAL_STEP: 1465000 | > loss_text_ce: 0.02332989312708378 (0.030492162163756793) | > loss_mel_ce: 2.846787452697754 (nan) | > loss: 0.011389355175197124 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.1376202799430476) | > loader_time: 0.0046 (0.004684169498080907)  --> TIME: 2025-02-27 00:15:52 -- STEP: 24468/36938 -- GLOBAL_STEP: 1465050 | > loss_text_ce: 0.03167640045285225 (0.030492193529056376) | > loss_mel_ce: 3.276346206665039 (nan) | > loss: 0.01312707457691431 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.1376235498133995) | > loader_time: 0.0048 (0.004684144801245437)  --> TIME: 2025-02-27 00:16:02 -- STEP: 24518/36938 -- GLOBAL_STEP: 1465100 | > loss_text_ce: 0.036205779761075974 (0.030492343939555903) | > loss_mel_ce: 2.342501401901245 (nan) | > loss: 0.009439314715564251 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13762968100345468) | > loader_time: 0.0046 (0.004684203386170677)  --> TIME: 2025-02-27 00:16:11 -- STEP: 24568/36938 -- GLOBAL_STEP: 1465150 | > loss_text_ce: 0.034847307950258255 (0.03049183994995549) | > loss_mel_ce: 2.495147705078125 (nan) | > loss: 0.010039662942290306 (nan) | > current_lr: 5e-06 | > step_time: 0.1516 (0.13762555889802303) | > loader_time: 0.0047 (0.0046840669160866395)  --> TIME: 2025-02-27 00:16:20 -- STEP: 24618/36938 -- GLOBAL_STEP: 1465200 | > loss_text_ce: 0.032327670603990555 (0.030492173529884006) | > loss_mel_ce: 2.6756489276885986 (nan) | > loss: 0.010745939798653126 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13761997653870708) | > loader_time: 0.0046 (0.004684060446398112)  --> TIME: 2025-02-27 00:16:30 -- STEP: 24668/36938 -- GLOBAL_STEP: 1465250 | > loss_text_ce: 0.023261480033397675 (0.03048931996781531) | > loss_mel_ce: 2.507009506225586 (nan) | > loss: 0.010040759108960629 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.1376169151302598) | > loader_time: 0.0046 (0.004683892634499931)  --> TIME: 2025-02-27 00:16:39 -- STEP: 24718/36938 -- GLOBAL_STEP: 1465300 | > loss_text_ce: 0.03260349854826927 (0.030491242941939323) | > loss_mel_ce: 2.707977533340454 (nan) | > loss: 0.010875321924686432 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13762127666356125) | > loader_time: 0.0045 (0.004683764228370211)  --> TIME: 2025-02-27 00:16:49 -- STEP: 24768/36938 -- GLOBAL_STEP: 1465350 | > loss_text_ce: 0.03043859638273716 (0.030490588218761692) | > loss_mel_ce: 2.663693428039551 (nan) | > loss: 0.010691001079976559 (nan) | > current_lr: 5e-06 | > step_time: 0.1521 (0.13762586147979286) | > loader_time: 0.0046 (0.004683597807505385)  --> TIME: 2025-02-27 00:16:58 -- STEP: 24818/36938 -- GLOBAL_STEP: 1465400 | > loss_text_ce: 0.035283464938402176 (0.030489681823177926) | > loss_mel_ce: 2.4238123893737793 (nan) | > loss: 0.009758317843079567 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.1376269480715024) | > loader_time: 0.0047 (0.004683644057415402)  --> TIME: 2025-02-27 00:17:07 -- STEP: 24868/36938 -- GLOBAL_STEP: 1465450 | > loss_text_ce: 0.02975531294941902 (0.03048976781799085) | > loss_mel_ce: 2.7931456565856934 (nan) | > loss: 0.011201988905668259 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.13762471240414725) | > loader_time: 0.0046 (0.004683599990530618)  --> TIME: 2025-02-27 00:17:17 -- STEP: 24918/36938 -- GLOBAL_STEP: 1465500 | > loss_text_ce: 0.03981328755617142 (0.030489991495693144) | > loss_mel_ce: 0.1307373046875 (nan) | > loss: 0.0006767880986444652 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13763336565437337) | > loader_time: 0.0047 (0.00468355115377197)  --> TIME: 2025-02-27 00:17:26 -- STEP: 24968/36938 -- GLOBAL_STEP: 1465550 | > loss_text_ce: 0.034832555800676346 (0.0304932263856421) | > loss_mel_ce: 2.7133615016937256 (nan) | > loss: 0.010905532166361809 (nan) | > current_lr: 5e-06 | > step_time: 0.1519 (0.13762783387348249) | > loader_time: 0.0047 (0.004683438515441661)  --> TIME: 2025-02-27 00:17:35 -- STEP: 25018/36938 -- GLOBAL_STEP: 1465600 | > loss_text_ce: 0.013825876638293266 (0.030491663559729102) | > loss_mel_ce: 1.1407004594802856 (nan) | > loss: 0.004581454209983349 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13761483953957646) | > loader_time: 0.0045 (0.004683343833733095)  --> TIME: 2025-02-27 00:17:44 -- STEP: 25068/36938 -- GLOBAL_STEP: 1465650 | > loss_text_ce: 0.03036089800298214 (0.030490567496735815) | > loss_mel_ce: 2.1778242588043213 (nan) | > loss: 0.008762639947235584 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.1376052659646319) | > loader_time: 0.0045 (0.004683211533784042)  --> TIME: 2025-02-27 00:17:54 -- STEP: 25118/36938 -- GLOBAL_STEP: 1465700 | > loss_text_ce: 0.03334524482488632 (0.030492379817839803) | > loss_mel_ce: 1.9525259733200073 (nan) | > loss: 0.007880441844463348 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13761551905631372) | > loader_time: 0.0045 (0.00468322154167442)  --> TIME: 2025-02-27 00:18:04 -- STEP: 25168/36938 -- GLOBAL_STEP: 1465750 | > loss_text_ce: 0.021025240421295166 (0.030493226337211222) | > loss_mel_ce: 1.8334482908248901 (nan) | > loss: 0.007359022740274668 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.1376225506319642) | > loader_time: 0.0046 (0.0046831073650341224)  --> TIME: 2025-02-27 00:18:13 -- STEP: 25218/36938 -- GLOBAL_STEP: 1465800 | > loss_text_ce: 0.0267763901501894 (0.030492922081263054) | > loss_mel_ce: 2.79162859916687 (nan) | > loss: 0.01118414755910635 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.1376149746189699) | > loader_time: 0.0047 (0.004682980556398842)  --> TIME: 2025-02-27 00:18:22 -- STEP: 25268/36938 -- GLOBAL_STEP: 1465850 | > loss_text_ce: 0.029335753992199898 (0.030492699112409503) | > loss_mel_ce: 2.503087282180786 (nan) | > loss: 0.010049298405647278 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13761619178294326) | > loader_time: 0.0045 (0.004682946397625323)  --> TIME: 2025-02-27 00:18:32 -- STEP: 25318/36938 -- GLOBAL_STEP: 1465900 | > loss_text_ce: 0.023280063644051552 (0.03048935408400286) | > loss_mel_ce: 2.257030963897705 (nan) | > loss: 0.009048854000866413 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13761069711722418) | > loader_time: 0.0046 (0.004682798965328194)  --> TIME: 2025-02-27 00:18:41 -- STEP: 25368/36938 -- GLOBAL_STEP: 1465950 | > loss_text_ce: 0.03288428485393524 (0.030487439729769233) | > loss_mel_ce: 2.288559913635254 (nan) | > loss: 0.00921208132058382 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.13760056113339672) | > loader_time: 0.0046 (0.004682614031893695)  --> TIME: 2025-02-27 00:18:50 -- STEP: 25418/36938 -- GLOBAL_STEP: 1466000 | > loss_text_ce: 0.023908667266368866 (0.030486768175871936) | > loss_mel_ce: 3.269368886947632 (nan) | > loss: 0.013068562373518944 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13759954937681196) | > loader_time: 0.0046 (0.0046824265055394705)  --> TIME: 2025-02-27 00:19:00 -- STEP: 25468/36938 -- GLOBAL_STEP: 1466050 | > loss_text_ce: 0.030901705846190453 (0.030489192560970584) | > loss_mel_ce: 1.4231644868850708 (nan) | > loss: 0.005770104005932808 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.137606209594226) | > loader_time: 0.0045 (0.004682290754382696)  --> TIME: 2025-02-27 00:19:09 -- STEP: 25518/36938 -- GLOBAL_STEP: 1466100 | > loss_text_ce: 0.03965163975954056 (0.03048827170518596) | > loss_mel_ce: 1.4681042432785034 (nan) | > loss: 0.005983158480376005 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13759616112875372) | > loader_time: 0.0045 (0.004682067177009647)  --> TIME: 2025-02-27 00:19:18 -- STEP: 25568/36938 -- GLOBAL_STEP: 1466150 | > loss_text_ce: 0.03861216828227043 (0.030487554438131132) | > loss_mel_ce: 2.7772183418273926 (nan) | > loss: 0.011173930950462818 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13759613028716997) | > loader_time: 0.0046 (0.00468223842236222)  --> TIME: 2025-02-27 00:19:27 -- STEP: 25618/36938 -- GLOBAL_STEP: 1466200 | > loss_text_ce: 0.03156626224517822 (0.030489050403224402) | > loss_mel_ce: 3.165792226791382 (nan) | > loss: 0.012687931768596172 (nan) | > current_lr: 5e-06 | > step_time: 0.145 (0.13759318146514676) | > loader_time: 0.0047 (0.00468227457354304)  --> TIME: 2025-02-27 00:19:37 -- STEP: 25668/36938 -- GLOBAL_STEP: 1466250 | > loss_text_ce: 0.029103901237249374 (0.03048856249753617) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.13759458428215404) | > loader_time: 0.0047 (0.004682249390895081)  --> TIME: 2025-02-27 00:19:46 -- STEP: 25718/36938 -- GLOBAL_STEP: 1466300 | > loss_text_ce: 0.024750836193561554 (0.03048763381352674) | > loss_mel_ce: 2.6006667613983154 (nan) | > loss: 0.01041832473129034 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13759044660306755) | > loader_time: 0.0046 (0.004682177323299066)  --> TIME: 2025-02-27 00:19:55 -- STEP: 25768/36938 -- GLOBAL_STEP: 1466350 | > loss_text_ce: 0.03197195380926132 (0.03048794299407967) | > loss_mel_ce: 2.6091346740722656 (nan) | > loss: 0.01048058271408081 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13758750970006134) | > loader_time: 0.0046 (0.00468220474075291)  --> TIME: 2025-02-27 00:20:04 -- STEP: 25818/36938 -- GLOBAL_STEP: 1466400 | > loss_text_ce: 0.023592783138155937 (0.03048520138679054) | > loss_mel_ce: 2.4906206130981445 (nan) | > loss: 0.009977038018405437 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13757868355734731) | > loader_time: 0.0046 (0.004682211689746285)  --> TIME: 2025-02-27 00:20:13 -- STEP: 25868/36938 -- GLOBAL_STEP: 1466450 | > loss_text_ce: 0.041439276188611984 (0.030484511027655096) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13757025195510078) | > loader_time: 0.0046 (0.004682143089925044)  --> TIME: 2025-02-27 00:20:23 -- STEP: 25918/36938 -- GLOBAL_STEP: 1466500 | > loss_text_ce: 0.039693914353847504 (0.03048556213094236) | > loss_mel_ce: 2.9631688594818115 (nan) | > loss: 0.011916122399270535 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.1375726981703558) | > loader_time: 0.0047 (0.0046822646212657274)  --> TIME: 2025-02-27 00:20:33 -- STEP: 25968/36938 -- GLOBAL_STEP: 1466550 | > loss_text_ce: 0.0245798472315073 (0.03048435280791906) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13758061728907922) | > loader_time: 0.0046 (0.004682273260252936)  --> TIME: 2025-02-27 00:20:42 -- STEP: 26018/36938 -- GLOBAL_STEP: 1466600 | > loss_text_ce: 0.026112837716937065 (0.030482848130537515) | > loss_mel_ce: 2.7240803241729736 (nan) | > loss: 0.010913465172052383 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13758486287105812) | > loader_time: 0.0049 (0.004682335867138822)  --> TIME: 2025-02-27 00:20:52 -- STEP: 26068/36938 -- GLOBAL_STEP: 1466650 | > loss_text_ce: 0.028732875362038612 (0.030483316163517002) | > loss_mel_ce: 2.6137301921844482 (nan) | > loss: 0.010485964827239513 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13758975152170372) | > loader_time: 0.0048 (0.004682375524277114)  --> TIME: 2025-02-27 00:21:01 -- STEP: 26118/36938 -- GLOBAL_STEP: 1466700 | > loss_text_ce: 0.028628673404455185 (0.03048299497565167) | > loss_mel_ce: 1.880865454673767 (nan) | > loss: 0.007577358279377222 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13758729833110256) | > loader_time: 0.0047 (0.004682403965815751)  --> TIME: 2025-02-27 00:21:10 -- STEP: 26168/36938 -- GLOBAL_STEP: 1466750 | > loss_text_ce: 0.027043864130973816 (0.030482274269167952) | > loss_mel_ce: 2.9978325366973877 (nan) | > loss: 0.012003478594124317 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13758548433647283) | > loader_time: 0.0047 (0.004682457946338885)  --> TIME: 2025-02-27 00:21:20 -- STEP: 26218/36938 -- GLOBAL_STEP: 1466800 | > loss_text_ce: 0.010751498863101006 (0.030480313219807945) | > loss_mel_ce: 1.8755687475204468 (nan) | > loss: 0.007485398091375828 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1375954448608432) | > loader_time: 0.0045 (0.004682408643901972)  --> TIME: 2025-02-27 00:21:29 -- STEP: 26268/36938 -- GLOBAL_STEP: 1466850 | > loss_text_ce: 0.03199988976120949 (0.030480675530396982) | > loss_mel_ce: 3.435584783554077 (nan) | > loss: 0.013760257512331009 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13758917374572416) | > loader_time: 0.0046 (0.004682276235135811)  --> TIME: 2025-02-27 00:21:38 -- STEP: 26318/36938 -- GLOBAL_STEP: 1466900 | > loss_text_ce: 0.024106070399284363 (0.030480371591373113) | > loss_mel_ce: 2.9823336601257324 (nan) | > loss: 0.011930316686630249 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13757734752025666) | > loader_time: 0.0046 (0.004682134065469378)  --> TIME: 2025-02-27 00:21:47 -- STEP: 26368/36938 -- GLOBAL_STEP: 1466950 | > loss_text_ce: 0.03475756570696831 (0.03047950896395987) | > loss_mel_ce: 2.6438982486724854 (nan) | > loss: 0.010629587806761265 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13756618963427913) | > loader_time: 0.0047 (0.004682000961552587)  --> TIME: 2025-02-27 00:21:56 -- STEP: 26418/36938 -- GLOBAL_STEP: 1467000 | > loss_text_ce: 0.033206842839717865 (0.03047874174080789) | > loss_mel_ce: 2.6180689334869385 (nan) | > loss: 0.0105209369212389 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.1375683029122676) | > loader_time: 0.0046 (0.004681948673644848)  --> TIME: 2025-02-27 00:22:06 -- STEP: 26468/36938 -- GLOBAL_STEP: 1467050 | > loss_text_ce: 0.028321022167801857 (0.030480742915950095) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13757941662337414) | > loader_time: 0.0046 (0.004681940991766493)  --> TIME: 2025-02-27 00:22:16 -- STEP: 26518/36938 -- GLOBAL_STEP: 1467100 | > loss_text_ce: 0.026735177263617516 (0.030478469014697543) | > loss_mel_ce: 3.197035074234009 (nan) | > loss: 0.0127927390858531 (nan) | > current_lr: 5e-06 | > step_time: 0.1461 (0.13758644153021846) | > loader_time: 0.0047 (0.004681907598137478)  --> TIME: 2025-02-27 00:22:25 -- STEP: 26568/36938 -- GLOBAL_STEP: 1467150 | > loss_text_ce: 0.025746900588274002 (0.030476759952674314) | > loss_mel_ce: 2.6169698238372803 (nan) | > loss: 0.010486971586942673 (nan) | > current_lr: 5e-06 | > step_time: 0.1464 (0.1375833891020304) | > loader_time: 0.0046 (0.00468175168390632)  --> TIME: 2025-02-27 00:22:34 -- STEP: 26618/36938 -- GLOBAL_STEP: 1467200 | > loss_text_ce: 0.034518033266067505 (0.030474955191890623) | > loss_mel_ce: 1.169762372970581 (nan) | > loss: 0.004778890870511532 (nan) | > current_lr: 5e-06 | > step_time: 0.1438 (0.13757519554846415) | > loader_time: 0.0047 (0.004681688191987676)  --> TIME: 2025-02-27 00:22:43 -- STEP: 26668/36938 -- GLOBAL_STEP: 1467250 | > loss_text_ce: 0.02733541838824749 (0.03047527222632141) | > loss_mel_ce: 2.52862811088562 (nan) | > loss: 0.010142712853848934 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.13756767541585824) | > loader_time: 0.0048 (0.004681714260186618)  --> TIME: 2025-02-27 00:22:52 -- STEP: 26718/36938 -- GLOBAL_STEP: 1467300 | > loss_text_ce: 0.020582016557455063 (0.030472232689025516) | > loss_mel_ce: 1.0170201063156128 (nan) | > loss: 0.00411746883764863 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13756135141111525) | > loader_time: 0.0047 (0.0046818606447822375)  --> TIME: 2025-02-27 00:23:01 -- STEP: 26768/36938 -- GLOBAL_STEP: 1467350 | > loss_text_ce: 0.01893557980656624 (0.03047300770634062) | > loss_mel_ce: 1.4108080863952637 (nan) | > loss: 0.005673586390912533 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13755473819629752) | > loader_time: 0.0045 (0.004681880521831342)  --> TIME: 2025-02-27 00:23:11 -- STEP: 26818/36938 -- GLOBAL_STEP: 1467400 | > loss_text_ce: 0.032624565064907074 (0.030473280901199606) | > loss_mel_ce: 2.519848108291626 (nan) | > loss: 0.010128860361874104 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13755714281692227) | > loader_time: 0.0047 (0.004681916762824358)  --> TIME: 2025-02-27 00:23:20 -- STEP: 26868/36938 -- GLOBAL_STEP: 1467450 | > loss_text_ce: 0.029574286192655563 (0.030473951114904127) | > loss_mel_ce: 2.1551852226257324 (nan) | > loss: 0.00866968184709549 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.13754907301428138) | > loader_time: 0.0046 (0.004681827199588682)  --> TIME: 2025-02-27 00:23:29 -- STEP: 26918/36938 -- GLOBAL_STEP: 1467500 | > loss_text_ce: 0.029158687219023705 (0.03047492876858916) | > loss_mel_ce: 2.577420711517334 (nan) | > loss: 0.010343569330871105 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.13755110521994637) | > loader_time: 0.0067 (0.004681860393538675)  --> TIME: 2025-02-27 00:23:39 -- STEP: 26968/36938 -- GLOBAL_STEP: 1467550 | > loss_text_ce: 0.026824017986655235 (0.030474068148616806) | > loss_mel_ce: 2.3701565265655518 (nan) | > loss: 0.009511828422546387 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13755655345277368) | > loader_time: 0.0046 (0.004681820121160906)  --> TIME: 2025-02-27 00:23:49 -- STEP: 27018/36938 -- GLOBAL_STEP: 1467600 | > loss_text_ce: 0.04267238825559616 (0.030474653915565624) | > loss_mel_ce: 2.667081594467163 (nan) | > loss: 0.010752992704510689 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13755963321476525) | > loader_time: 0.0046 (0.004681827826277265)  --> TIME: 2025-02-27 00:23:58 -- STEP: 27068/36938 -- GLOBAL_STEP: 1467650 | > loss_text_ce: 0.031136270612478256 (0.030475844912706745) | > loss_mel_ce: 2.785533905029297 (nan) | > loss: 0.011177263222634792 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.137555771165044) | > loader_time: 0.0046 (0.0046817443915866995)  --> TIME: 2025-02-27 00:24:07 -- STEP: 27118/36938 -- GLOBAL_STEP: 1467700 | > loss_text_ce: 0.03175267204642296 (0.030476969044494973) | > loss_mel_ce: 0.007453250698745251 (nan) | > loss: 0.00015557906590402126 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1375545311294143) | > loader_time: 0.0046 (0.00468165894350887)  --> TIME: 2025-02-27 00:24:17 -- STEP: 27168/36938 -- GLOBAL_STEP: 1467750 | > loss_text_ce: 0.03532707691192627 (0.03047686351295706) | > loss_mel_ce: 2.953646659851074 (nan) | > loss: 0.01186100672930479 (nan) | > current_lr: 5e-06 | > step_time: 0.145 (0.13755902383823107) | > loader_time: 0.0046 (0.004681539347722253)  --> TIME: 2025-02-27 00:24:26 -- STEP: 27218/36938 -- GLOBAL_STEP: 1467800 | > loss_text_ce: 0.032438404858112335 (0.030476920346665333) | > loss_mel_ce: 2.326690673828125 (nan) | > loss: 0.009361624717712402 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13755947068958807) | > loader_time: 0.0046 (0.0046814453664030834)  --> TIME: 2025-02-27 00:24:35 -- STEP: 27268/36938 -- GLOBAL_STEP: 1467850 | > loss_text_ce: 0.03497515991330147 (0.030478446266101036) | > loss_mel_ce: 2.721679925918579 (nan) | > loss: 0.010939107276499271 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.13755688176610412) | > loader_time: 0.0048 (0.004681345277016601)  --> TIME: 2025-02-27 00:24:45 -- STEP: 27318/36938 -- GLOBAL_STEP: 1467900 | > loss_text_ce: 0.032525528222322464 (0.030478457600147363) | > loss_mel_ce: 3.0644984245300293 (nan) | > loss: 0.01228977832943201 (nan) | > current_lr: 5e-06 | > step_time: 0.1835 (0.13755436205080177) | > loader_time: 0.0048 (0.0046812563237844974)  --> TIME: 2025-02-27 00:24:54 -- STEP: 27368/36938 -- GLOBAL_STEP: 1467950 | > loss_text_ce: 0.03368111327290535 (0.030478731265147423) | > loss_mel_ce: 2.4191248416900635 (nan) | > loss: 0.009733357466757298 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13755090510730134) | > loader_time: 0.0046 (0.004681133546176961)  --> TIME: 2025-02-27 00:25:03 -- STEP: 27418/36938 -- GLOBAL_STEP: 1468000 | > loss_text_ce: 0.03025197982788086 (0.0304768579674284) | > loss_mel_ce: 1.7977585792541504 (nan) | > loss: 0.007254010532051325 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.1375513639132848) | > loader_time: 0.0046 (0.004681085755870494)  --> TIME: 2025-02-27 00:25:13 -- STEP: 27468/36938 -- GLOBAL_STEP: 1468050 | > loss_text_ce: 0.020369810983538628 (0.030478185148440026) | > loss_mel_ce: 1.9286887645721436 (nan) | > loss: 0.007734359707683325 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.13755534492727048) | > loader_time: 0.0045 (0.004680988030678834)  --> TIME: 2025-02-27 00:25:22 -- STEP: 27518/36938 -- GLOBAL_STEP: 1468100 | > loss_text_ce: 0.029947103932499886 (0.03047816754439665) | > loss_mel_ce: 2.224968910217285 (nan) | > loss: 0.00894807931035757 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.13755365116875667) | > loader_time: 0.0046 (0.004680858508160774)  --> TIME: 2025-02-27 00:25:31 -- STEP: 27568/36938 -- GLOBAL_STEP: 1468150 | > loss_text_ce: 0.026099957525730133 (0.030478256220015973) | > loss_mel_ce: 1.7612910270690918 (nan) | > loss: 0.007092821411788464 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13755526376787564) | > loader_time: 0.0046 (0.0046807297062749905)  --> TIME: 2025-02-27 00:25:41 -- STEP: 27618/36938 -- GLOBAL_STEP: 1468200 | > loss_text_ce: 0.02859324775636196 (0.0304786576374021) | > loss_mel_ce: 3.0318901538848877 (nan) | > loss: 0.012144776061177254 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.1375493545243544) | > loader_time: 0.0047 (0.004680616590251453)  --> TIME: 2025-02-27 00:25:50 -- STEP: 27668/36938 -- GLOBAL_STEP: 1468250 | > loss_text_ce: 0.023869028314948082 (0.030476087081528557) | > loss_mel_ce: 1.4978004693984985 (nan) | > loss: 0.0060383714735507965 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13755599055070267) | > loader_time: 0.0044 (0.004680621524034344)  --> TIME: 2025-02-27 00:25:59 -- STEP: 27718/36938 -- GLOBAL_STEP: 1468300 | > loss_text_ce: 0.03882886841893196 (0.03047723836394259) | > loss_mel_ce: 2.5183286666870117 (nan) | > loss: 0.010147451423108578 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.1375518870978169) | > loader_time: 0.0045 (0.004680541938100828)  --> TIME: 2025-02-27 00:26:09 -- STEP: 27768/36938 -- GLOBAL_STEP: 1468350 | > loss_text_ce: 0.032571110874414444 (0.030478409752902395) | > loss_mel_ce: 2.9291443824768066 (nan) | > loss: 0.011752840131521225 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13756140338450634) | > loader_time: 0.0046 (0.004680469894024582)  --> TIME: 2025-02-27 00:26:18 -- STEP: 27818/36938 -- GLOBAL_STEP: 1468400 | > loss_text_ce: 0.03499126434326172 (0.03048211442453879) | > loss_mel_ce: 0.279296875 (nan) | > loss: 0.001247175270691514 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13755660009414952) | > loader_time: 0.0045 (0.0046803040973785424)  --> TIME: 2025-02-27 00:26:28 -- STEP: 27868/36938 -- GLOBAL_STEP: 1468450 | > loss_text_ce: 0.03497227653861046 (0.030481111440820014) | > loss_mel_ce: 2.565211057662964 (nan) | > loss: 0.01031818799674511 (nan) | > current_lr: 5e-06 | > step_time: 0.1436 (0.13756065826702055) | > loader_time: 0.0046 (0.004680186890796136)  --> TIME: 2025-02-27 00:26:37 -- STEP: 27918/36938 -- GLOBAL_STEP: 1468500 | > loss_text_ce: 0.032127439975738525 (0.03048145251578593) | > loss_mel_ce: 2.19455623626709 (nan) | > loss: 0.008836046792566776 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13755309601398452) | > loader_time: 0.0045 (0.004680002348002385)  --> TIME: 2025-02-27 00:26:46 -- STEP: 27968/36938 -- GLOBAL_STEP: 1468550 | > loss_text_ce: 0.0309959314763546 (0.030479225391808827) | > loss_mel_ce: 2.8232383728027344 (nan) | > loss: 0.011326326988637447 (nan) | > current_lr: 5e-06 | > step_time: 0.1357 (0.13754582430664358) | > loader_time: 0.0046 (0.004679821406062794)  --> TIME: 2025-02-27 00:26:56 -- STEP: 28018/36938 -- GLOBAL_STEP: 1468600 | > loss_text_ce: 0.03444424644112587 (0.030479202673118516) | > loss_mel_ce: 2.2332818508148193 (nan) | > loss: 0.008998913690447807 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13754790557154936) | > loader_time: 0.0047 (0.004679739309315465)  --> TIME: 2025-02-27 00:27:05 -- STEP: 28068/36938 -- GLOBAL_STEP: 1468650 | > loss_text_ce: 0.030416389927268028 (0.03047815300476786) | > loss_mel_ce: 2.601057291030884 (nan) | > loss: 0.010442356579005718 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.13754989263490475) | > loader_time: 0.0046 (0.004679646496421499)  --> TIME: 2025-02-27 00:27:14 -- STEP: 28118/36938 -- GLOBAL_STEP: 1468700 | > loss_text_ce: 0.03231646865606308 (0.030478714286433246) | > loss_mel_ce: 2.3885343074798584 (nan) | > loss: 0.009606551378965378 (nan) | > current_lr: 5e-06 | > step_time: 0.1267 (0.13754868329321682) | > loader_time: 0.0045 (0.004679528321588375)  --> TIME: 2025-02-27 00:27:24 -- STEP: 28168/36938 -- GLOBAL_STEP: 1468750 | > loss_text_ce: 0.03552735596895218 (0.03048098314495903) | > loss_mel_ce: 2.7707736492156982 (nan) | > loss: 0.011136116459965706 (nan) | > current_lr: 5e-06 | > step_time: 0.1264 (0.13755035966643162) | > loader_time: 0.0046 (0.004679565815992015)  --> TIME: 2025-02-27 00:27:33 -- STEP: 28218/36938 -- GLOBAL_STEP: 1468800 | > loss_text_ce: 0.026301147416234016 (0.030478152516071916) | > loss_mel_ce: 2.8352272510528564 (nan) | > loss: 0.011355272494256496 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13753607705023152) | > loader_time: 0.0045 (0.004679607038790662)  --> TIME: 2025-02-27 00:27:42 -- STEP: 28268/36938 -- GLOBAL_STEP: 1468850 | > loss_text_ce: 0.03137325122952461 (0.03047785597214009) | > loss_mel_ce: 3.5299830436706543 (nan) | > loss: 0.014132367447018623 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.1375306885870301) | > loader_time: 0.0046 (0.004679486667888975)  --> TIME: 2025-02-27 00:27:52 -- STEP: 28318/36938 -- GLOBAL_STEP: 1468900 | > loss_text_ce: 0.03620559349656105 (0.03047660481287266) | > loss_mel_ce: 2.311873435974121 (nan) | > loss: 0.009317774325609207 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.1375336064500262) | > loader_time: 0.0046 (0.004679378534375404)  --> TIME: 2025-02-27 00:28:01 -- STEP: 28368/36938 -- GLOBAL_STEP: 1468950 | > loss_text_ce: 0.029802383854985237 (0.030480163133985116) | > loss_mel_ce: 2.392507791519165 (nan) | > loss: 0.009612342342734337 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13753144880068566) | > loader_time: 0.0045 (0.004679259797374398)  --> TIME: 2025-02-27 00:28:10 -- STEP: 28418/36938 -- GLOBAL_STEP: 1469000 | > loss_text_ce: 0.029443692415952682 (0.03047975119281917) | > loss_mel_ce: 2.506708860397339 (nan) | > loss: 0.010064098052680492 (nan) | > current_lr: 5e-06 | > step_time: 0.1888 (0.13752895255118633) | > loader_time: 0.0048 (0.004679137736389081)  --> TIME: 2025-02-27 00:28:20 -- STEP: 28468/36938 -- GLOBAL_STEP: 1469050 | > loss_text_ce: 0.02723364531993866 (0.03047872219482748) | > loss_mel_ce: 2.5949039459228516 (nan) | > loss: 0.010405308566987514 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13753883889238333) | > loader_time: 0.0047 (0.004679067769344785)  --> TIME: 2025-02-27 00:28:30 -- STEP: 28518/36938 -- GLOBAL_STEP: 1469100 | > loss_text_ce: 0.03231995180249214 (0.03047919886817202) | > loss_mel_ce: 3.401675224304199 (nan) | > loss: 0.013626965694129467 (nan) | > current_lr: 5e-06 | > step_time: 0.185 (0.13755403102401456) | > loader_time: 0.0049 (0.004679220280713214)  --> TIME: 2025-02-27 00:28:39 -- STEP: 28568/36938 -- GLOBAL_STEP: 1469150 | > loss_text_ce: 0.0356283001601696 (0.030479461665660245) | > loss_mel_ce: 2.530468702316284 (nan) | > loss: 0.010182925499975681 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13755891337944812) | > loader_time: 0.0045 (0.004679326065045374)  --> TIME: 2025-02-27 00:28:49 -- STEP: 28618/36938 -- GLOBAL_STEP: 1469200 | > loss_text_ce: 0.038440119475126266 (0.030479792328125747) | > loss_mel_ce: 2.4493229389190674 (nan) | > loss: 0.009872077032923698 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13756457038441863) | > loader_time: 0.0045 (0.004679351418146197)  --> TIME: 2025-02-27 00:28:58 -- STEP: 28668/36938 -- GLOBAL_STEP: 1469250 | > loss_text_ce: 0.04131674766540527 (0.030479930311859717) | > loss_mel_ce: 3.237812042236328 (nan) | > loss: 0.013012416660785675 (nan) | > current_lr: 5e-06 | > step_time: 0.1867 (0.13756028188227953) | > loader_time: 0.0047 (0.004679275952873218)  --> TIME: 2025-02-27 00:29:07 -- STEP: 28718/36938 -- GLOBAL_STEP: 1469300 | > loss_text_ce: 0.035692229866981506 (0.030481913878519958) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13754676899145132) | > loader_time: 0.0046 (0.004679183573417049)  --> TIME: 2025-02-27 00:29:17 -- STEP: 28768/36938 -- GLOBAL_STEP: 1469350 | > loss_text_ce: 0.032798297703266144 (0.030482270172586715) | > loss_mel_ce: 2.487413167953491 (nan) | > loss: 0.010000839829444885 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.137553997868226) | > loader_time: 0.0045 (0.004679085407095363)  --> TIME: 2025-02-27 00:29:26 -- STEP: 28818/36938 -- GLOBAL_STEP: 1469400 | > loss_text_ce: 0.02215871959924698 (0.030483044715135698) | > loss_mel_ce: 2.397583484649658 (nan) | > loss: 0.009602151811122894 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13755323585063034) | > loader_time: 0.0045 (0.004678987945439151)  --> TIME: 2025-02-27 00:29:36 -- STEP: 28868/36938 -- GLOBAL_STEP: 1469450 | > loss_text_ce: 0.03202405571937561 (0.03048505184840508) | > loss_mel_ce: 2.8415424823760986 (nan) | > loss: 0.011403042823076248 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.13756013153595953) | > loader_time: 0.0048 (0.004678899906209247)  --> TIME: 2025-02-27 00:29:45 -- STEP: 28918/36938 -- GLOBAL_STEP: 1469500 | > loss_text_ce: 0.028683582320809364 (0.03048545276612887) | > loss_mel_ce: 2.5740628242492676 (nan) | > loss: 0.01032835990190506 (nan) | > current_lr: 5e-06 | > step_time: 0.1936 (0.13757046737053477) | > loader_time: 0.0056 (0.004679018163822895)  --> TIME: 2025-02-27 00:29:55 -- STEP: 28968/36938 -- GLOBAL_STEP: 1469550 | > loss_text_ce: 0.020949458703398705 (0.03048548436014355) | > loss_mel_ce: 2.122976064682007 (nan) | > loss: 0.008507641032338142 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13757351388986527) | > loader_time: 0.0045 (0.0046791631488700535)  --> TIME: 2025-02-27 00:30:04 -- STEP: 29018/36938 -- GLOBAL_STEP: 1469600 | > loss_text_ce: 0.03108784928917885 (0.030483532519366606) | > loss_mel_ce: 2.843920946121216 (nan) | > loss: 0.01140876580029726 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13757993568763724) | > loader_time: 0.0046 (0.0046792967559896)  --> TIME: 2025-02-27 00:30:14 -- STEP: 29068/36938 -- GLOBAL_STEP: 1469650 | > loss_text_ce: 0.03202191740274429 (0.030484282558968307) | > loss_mel_ce: 3.2350821495056152 (nan) | > loss: 0.012964699417352676 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.13758526697141693) | > loader_time: 0.0047 (0.004679440402157904)  --> TIME: 2025-02-27 00:30:23 -- STEP: 29118/36938 -- GLOBAL_STEP: 1469700 | > loss_text_ce: 0.03759711980819702 (0.03048556982389907) | > loss_mel_ce: 2.93338680267334 (nan) | > loss: 0.01178961992263794 (nan) | > current_lr: 5e-06 | > step_time: 0.1573 (0.13757692211824968) | > loader_time: 0.0047 (0.004679372328815874)  --> TIME: 2025-02-27 00:30:32 -- STEP: 29168/36938 -- GLOBAL_STEP: 1469750 | > loss_text_ce: 0.03551851212978363 (0.030485345309466002) | > loss_mel_ce: 0.1367754191160202 (nan) | > loss: 0.0006837061373516917 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13757406143257989) | > loader_time: 0.0046 (0.004679416014608083)  --> TIME: 2025-02-27 00:30:42 -- STEP: 29218/36938 -- GLOBAL_STEP: 1469800 | > loss_text_ce: 0.02881615050137043 (0.030485467169225625) | > loss_mel_ce: 2.4119720458984375 (nan) | > loss: 0.009685668163001537 (nan) | > current_lr: 5e-06 | > step_time: 0.1694 (0.13757475710590747) | > loader_time: 0.0047 (0.00467954383541721)  --> TIME: 2025-02-27 00:30:51 -- STEP: 29268/36938 -- GLOBAL_STEP: 1469850 | > loss_text_ce: 0.02247200906276703 (0.030482873283269984) | > loss_mel_ce: 2.75369930267334 (nan) | > loss: 0.01101655326783657 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.13757040130158485) | > loader_time: 0.0047 (0.004679540255461777)  --> TIME: 2025-02-27 00:31:00 -- STEP: 29318/36938 -- GLOBAL_STEP: 1469900 | > loss_text_ce: 0.022498782724142075 (0.03048350544850258) | > loss_mel_ce: 2.7560536861419678 (nan) | > loss: 0.01102600246667862 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.1375651116162641) | > loader_time: 0.0047 (0.004679612056548252)  --> TIME: 2025-02-27 00:31:10 -- STEP: 29368/36938 -- GLOBAL_STEP: 1469950 | > loss_text_ce: 0.03813352435827255 (0.03048396134913387) | > loss_mel_ce: 2.6011602878570557 (nan) | > loss: 0.01047338917851448 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.13756482524366487) | > loader_time: 0.0047 (0.004679618747836454)  --> TIME: 2025-02-27 00:31:19 -- STEP: 29418/36938 -- GLOBAL_STEP: 1470000 | > loss_text_ce: 0.031757429242134094 (0.03048500885418105) | > loss_mel_ce: 2.5403218269348145 (nan) | > loss: 0.010206663981080055 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.13756406053549142) | > loader_time: 0.0047 (0.00467955941322089) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1470000.pth  --> TIME: 2025-02-27 00:31:35 -- STEP: 29468/36938 -- GLOBAL_STEP: 1470050 | > loss_text_ce: 0.032888032495975494 (0.030482750749752128) | > loss_mel_ce: 3.0505216121673584 (nan) | > loss: 0.012235753238201141 (nan) | > current_lr: 5e-06 | > step_time: 0.1866 (0.13756220578111691) | > loader_time: 0.0048 (0.004679791498850939)  --> TIME: 2025-02-27 00:31:44 -- STEP: 29518/36938 -- GLOBAL_STEP: 1470100 | > loss_text_ce: 0.03229888156056404 (0.030483141374560426) | > loss_mel_ce: 2.9221079349517822 (nan) | > loss: 0.01172383688390255 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13756262029975663) | > loader_time: 0.0046 (0.004680010222251478)  --> TIME: 2025-02-27 00:31:53 -- STEP: 29568/36938 -- GLOBAL_STEP: 1470150 | > loss_text_ce: 0.028826052322983742 (0.030482101533155625) | > loss_mel_ce: 2.6928629875183105 (nan) | > loss: 0.010800354182720184 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.13756642801872557) | > loader_time: 0.0046 (0.004679985513741375)  --> TIME: 2025-02-27 00:32:02 -- STEP: 29618/36938 -- GLOBAL_STEP: 1470200 | > loss_text_ce: 0.03519706800580025 (0.030481911597246926) | > loss_mel_ce: 2.356919288635254 (nan) | > loss: 0.009492525830864906 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13755613356552965) | > loader_time: 0.0046 (0.004679845366167944)  --> TIME: 2025-02-27 00:32:12 -- STEP: 29668/36938 -- GLOBAL_STEP: 1470250 | > loss_text_ce: 0.034353576600551605 (0.03048237777750229) | > loss_mel_ce: 3.0479965209960938 (nan) | > loss: 0.012231548316776752 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13755888078706788) | > loader_time: 0.0047 (0.0046798751587911365)  --> TIME: 2025-02-27 00:32:21 -- STEP: 29718/36938 -- GLOBAL_STEP: 1470300 | > loss_text_ce: 0.025773456320166588 (0.030482783791783288) | > loss_mel_ce: 2.8538782596588135 (nan) | > loss: 0.011427190154790878 (nan) | > current_lr: 5e-06 | > step_time: 0.1622 (0.13755695591528697) | > loader_time: 0.0047 (0.004679907586903701)  --> TIME: 2025-02-27 00:32:30 -- STEP: 29768/36938 -- GLOBAL_STEP: 1470350 | > loss_text_ce: 0.033226728439331055 (0.030482903495346474) | > loss_mel_ce: 2.608717441558838 (nan) | > loss: 0.010483905673027039 (nan) | > current_lr: 5e-06 | > step_time: 0.1615 (0.13755084935038087) | > loader_time: 0.0047 (0.004679845693578573)  --> TIME: 2025-02-27 00:32:40 -- STEP: 29818/36938 -- GLOBAL_STEP: 1470400 | > loss_text_ce: 0.02628742903470993 (0.03048421912463039) | > loss_mel_ce: 3.0913472175598145 (nan) | > loss: 0.012371566146612167 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.1375492436999852) | > loader_time: 0.0045 (0.004679792771213778)  --> TIME: 2025-02-27 00:32:49 -- STEP: 29868/36938 -- GLOBAL_STEP: 1470450 | > loss_text_ce: 0.036864254623651505 (0.03048608313133204) | > loss_mel_ce: 2.7228925228118896 (nan) | > loss: 0.010951416566967964 (nan) | > current_lr: 5e-06 | > step_time: 0.1436 (0.13755012581877474) | > loader_time: 0.0047 (0.004679718681075981)  --> TIME: 2025-02-27 00:32:59 -- STEP: 29918/36938 -- GLOBAL_STEP: 1470500 | > loss_text_ce: 0.026575710624456406 (0.030485557194709408) | > loss_mel_ce: 2.9985179901123047 (nan) | > loss: 0.012004340998828411 (nan) | > current_lr: 5e-06 | > step_time: 0.1715 (0.13755374245165772) | > loader_time: 0.0046 (0.004679710256662149)  --> TIME: 2025-02-27 00:33:08 -- STEP: 29968/36938 -- GLOBAL_STEP: 1470550 | > loss_text_ce: 0.02808395028114319 (0.030485841131325214) | > loss_mel_ce: 0.7989094853401184 (nan) | > loss: 0.0032817202154546976 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13755328265091837) | > loader_time: 0.0046 (0.004679646838239652)  --> TIME: 2025-02-27 00:33:17 -- STEP: 30018/36938 -- GLOBAL_STEP: 1470600 | > loss_text_ce: 0.034177958965301514 (0.030486312549912097) | > loss_mel_ce: 2.6965386867523193 (nan) | > loss: 0.010836178436875343 (nan) | > current_lr: 5e-06 | > step_time: 0.1842 (0.1375505203361695) | > loader_time: 0.0048 (0.004679731632011347)  --> TIME: 2025-02-27 00:33:27 -- STEP: 30068/36938 -- GLOBAL_STEP: 1470650 | > loss_text_ce: 0.03580499812960625 (0.030487101694386988) | > loss_mel_ce: 2.9424543380737305 (nan) | > loss: 0.011818489991128445 (nan) | > current_lr: 5e-06 | > step_time: 0.1812 (0.13756234495662303) | > loader_time: 0.0049 (0.0046798246360704255)  --> TIME: 2025-02-27 00:33:36 -- STEP: 30118/36938 -- GLOBAL_STEP: 1470700 | > loss_text_ce: 0.023748503997921944 (0.0304878556419392) | > loss_mel_ce: 2.8421175479888916 (nan) | > loss: 0.011372484266757965 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13755956787901225) | > loader_time: 0.0045 (0.004679783572158955)  --> TIME: 2025-02-27 00:33:46 -- STEP: 30168/36938 -- GLOBAL_STEP: 1470750 | > loss_text_ce: 0.024670440703630447 (0.030489726574391692) | > loss_mel_ce: 2.714156150817871 (nan) | > loss: 0.010868359357118607 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13755712779280788) | > loader_time: 0.0047 (0.004679657844863679)  --> TIME: 2025-02-27 00:33:55 -- STEP: 30218/36938 -- GLOBAL_STEP: 1470800 | > loss_text_ce: 0.03579296916723251 (0.030490919089792224) | > loss_mel_ce: 2.672149658203125 (nan) | > loss: 0.010745804756879807 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13754686448313522) | > loader_time: 0.0046 (0.004679649652088151)  --> TIME: 2025-02-27 00:34:04 -- STEP: 30268/36938 -- GLOBAL_STEP: 1470850 | > loss_text_ce: 0.03618490695953369 (0.030491275334586374) | > loss_mel_ce: 2.3697609901428223 (nan) | > loss: 0.009547404013574123 (nan) | > current_lr: 5e-06 | > step_time: 0.1349 (0.13754892607606084) | > loader_time: 0.0046 (0.004679568971465336)  --> TIME: 2025-02-27 00:34:13 -- STEP: 30318/36938 -- GLOBAL_STEP: 1470900 | > loss_text_ce: 0.027851123362779617 (0.030490311053990398) | > loss_mel_ce: 2.22662091255188 (nan) | > loss: 0.008946318179368973 (nan) | > current_lr: 5e-06 | > step_time: 0.1461 (0.13754949536516353) | > loader_time: 0.0047 (0.004679480724484501)  --> TIME: 2025-02-27 00:34:23 -- STEP: 30368/36938 -- GLOBAL_STEP: 1470950 | > loss_text_ce: 0.03285380080342293 (0.030490950266102566) | > loss_mel_ce: 3.027336597442627 (nan) | > loss: 0.012143613770604134 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.13755417062466194) | > loader_time: 0.0047 (0.004679408061805896)  --> TIME: 2025-02-27 00:34:32 -- STEP: 30418/36938 -- GLOBAL_STEP: 1471000 | > loss_text_ce: 0.02877340465784073 (0.030490098631014848) | > loss_mel_ce: 2.846447229385376 (nan) | > loss: 0.011409605853259563 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.13755437984324823) | > loader_time: 0.0048 (0.0046792842750993805)  --> TIME: 2025-02-27 00:34:42 -- STEP: 30468/36938 -- GLOBAL_STEP: 1471050 | > loss_text_ce: 0.03765604645013809 (0.03049118590983614) | > loss_mel_ce: 2.8282876014709473 (nan) | > loss: 0.011372792534530163 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13755430194889273) | > loader_time: 0.0046 (0.004679264829151638)  --> TIME: 2025-02-27 00:34:51 -- STEP: 30518/36938 -- GLOBAL_STEP: 1471100 | > loss_text_ce: 0.027980491518974304 (0.030491549784422103) | > loss_mel_ce: 3.280327320098877 (nan) | > loss: 0.013128207065165043 (nan) | > current_lr: 5e-06 | > step_time: 0.1688 (0.13755783884574813) | > loader_time: 0.0048 (0.004679160315287856)  --> TIME: 2025-02-27 00:35:01 -- STEP: 30568/36938 -- GLOBAL_STEP: 1471150 | > loss_text_ce: 0.03307803347706795 (0.030493928028123264) | > loss_mel_ce: 2.9837646484375 (nan) | > loss: 0.011971598491072655 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.1375659526143201) | > loader_time: 0.0046 (0.0046790833717801975)  --> TIME: 2025-02-27 00:35:11 -- STEP: 30618/36938 -- GLOBAL_STEP: 1471200 | > loss_text_ce: 0.03156061843037605 (0.030492260821856076) | > loss_mel_ce: 2.483978271484375 (nan) | > loss: 0.009982298128306866 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13757195671438352) | > loader_time: 0.0045 (0.00467915071342534)  --> TIME: 2025-02-27 00:35:20 -- STEP: 30668/36938 -- GLOBAL_STEP: 1471250 | > loss_text_ce: 0.03539976105093956 (0.030495078331936386) | > loss_mel_ce: 3.0757410526275635 (nan) | > loss: 0.012345797382295132 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13758208653101836) | > loader_time: 0.0048 (0.004679078203424995)  --> TIME: 2025-02-27 00:35:30 -- STEP: 30718/36938 -- GLOBAL_STEP: 1471300 | > loss_text_ce: 0.03183438628911972 (0.030493946000566794) | > loss_mel_ce: 2.827970266342163 (nan) | > loss: 0.011348431929945946 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.1375836612920724) | > loader_time: 0.0048 (0.0046789626977711074)  --> TIME: 2025-02-27 00:35:39 -- STEP: 30768/36938 -- GLOBAL_STEP: 1471350 | > loss_text_ce: 0.024452917277812958 (0.030490971334349172) | > loss_mel_ce: 2.466784954071045 (nan) | > loss: 0.00988586526364088 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13757852531167516) | > loader_time: 0.0045 (0.004678877555821929)  --> TIME: 2025-02-27 00:35:49 -- STEP: 30818/36938 -- GLOBAL_STEP: 1471400 | > loss_text_ce: 0.03258945420384407 (0.03049228592814202) | > loss_mel_ce: 2.833650588989258 (nan) | > loss: 0.011373968794941902 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13758692082876617) | > loader_time: 0.0045 (0.0046788451889610986)  --> TIME: 2025-02-27 00:35:58 -- STEP: 30868/36938 -- GLOBAL_STEP: 1471450 | > loss_text_ce: 0.03276187181472778 (0.030491557464281015) | > loss_mel_ce: 2.1888415813446045 (nan) | > loss: 0.00881588738411665 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13759229337518736) | > loader_time: 0.0065 (0.004678981074308157)  --> TIME: 2025-02-27 00:36:08 -- STEP: 30918/36938 -- GLOBAL_STEP: 1471500 | > loss_text_ce: 0.02732287161052227 (0.03049153968961662) | > loss_mel_ce: 2.7712676525115967 (nan) | > loss: 0.011105517856776714 (nan) | > current_lr: 5e-06 | > step_time: 0.172 (0.1375984372200604) | > loader_time: 0.0048 (0.0046790730822895785)  --> TIME: 2025-02-27 00:36:17 -- STEP: 30968/36938 -- GLOBAL_STEP: 1471550 | > loss_text_ce: 0.02671653777360916 (0.03049219396350943) | > loss_mel_ce: 2.449251651763916 (nan) | > loss: 0.009825270622968674 (nan) | > current_lr: 5e-06 | > step_time: 0.19 (0.1376017978551065) | > loader_time: 0.0048 (0.004679107975510217)  --> TIME: 2025-02-27 00:36:26 -- STEP: 31018/36938 -- GLOBAL_STEP: 1471600 | > loss_text_ce: 0.022247912362217903 (0.030493106874306727) | > loss_mel_ce: 2.58117413520813 (nan) | > loss: 0.010331041179597378 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13759285120478557) | > loader_time: 0.0046 (0.0046791617725364726)  --> TIME: 2025-02-27 00:36:36 -- STEP: 31068/36938 -- GLOBAL_STEP: 1471650 | > loss_text_ce: 0.03288298845291138 (0.03049443906806642) | > loss_mel_ce: 2.803802967071533 (nan) | > loss: 0.011256691068410873 (nan) | > current_lr: 5e-06 | > step_time: 0.1706 (0.13759693440068574) | > loader_time: 0.0048 (0.0046790525675956245)  --> TIME: 2025-02-27 00:36:45 -- STEP: 31118/36938 -- GLOBAL_STEP: 1471700 | > loss_text_ce: 0.038526080548763275 (0.030494463229529166) | > loss_mel_ce: 0.820430338382721 (nan) | > loss: 0.0034085572697222233 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1375836116641335) | > loader_time: 0.0047 (0.00467906093848723)  --> TIME: 2025-02-27 00:36:54 -- STEP: 31168/36938 -- GLOBAL_STEP: 1471750 | > loss_text_ce: 0.039656125009059906 (0.03049555563617288) | > loss_mel_ce: 0.650769054889679 (nan) | > loss: 0.0027397824451327324 (nan) | > current_lr: 5e-06 | > step_time: 0.1082 (0.13758198727205287) | > loader_time: 0.0045 (0.004679045722163976)  --> TIME: 2025-02-27 00:37:04 -- STEP: 31218/36938 -- GLOBAL_STEP: 1471800 | > loss_text_ce: 0.030823132023215294 (0.030498200920487124) | > loss_mel_ce: 2.6908814907073975 (nan) | > loss: 0.010800416581332684 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13758227851742286) | > loader_time: 0.0046 (0.004679043598946155)  --> TIME: 2025-02-27 00:37:13 -- STEP: 31268/36938 -- GLOBAL_STEP: 1471850 | > loss_text_ce: 0.039152566343545914 (0.030498297057166727) | > loss_mel_ce: 2.7331392765045166 (nan) | > loss: 0.011001158505678177 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13757687708720578) | > loader_time: 0.0047 (0.004679114293667876)  --> TIME: 2025-02-27 00:37:22 -- STEP: 31318/36938 -- GLOBAL_STEP: 1471900 | > loss_text_ce: 0.027868401259183884 (0.030497710188949253) | > loss_mel_ce: 2.48110294342041 (nan) | > loss: 0.0099562369287014 (nan) | > current_lr: 5e-06 | > step_time: 0.1779 (0.13756931404385622) | > loader_time: 0.0049 (0.004679194560368624)  --> TIME: 2025-02-27 00:37:31 -- STEP: 31368/36938 -- GLOBAL_STEP: 1471950 | > loss_text_ce: 0.036926813423633575 (0.030496705956527266) | > loss_mel_ce: 2.387439012527466 (nan) | > loss: 0.009620499797165394 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13757140139054638) | > loader_time: 0.0046 (0.004679421340589277)  --> TIME: 2025-02-27 00:37:41 -- STEP: 31418/36938 -- GLOBAL_STEP: 1472000 | > loss_text_ce: 0.040581848472356796 (0.030498395468045454) | > loss_mel_ce: 1.7331844568252563 (nan) | > loss: 0.007038755342364311 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13756505742140732) | > loader_time: 0.0045 (0.004679504475334696)  --> TIME: 2025-02-27 00:37:50 -- STEP: 31468/36938 -- GLOBAL_STEP: 1472050 | > loss_text_ce: 0.029054036363959312 (0.030497878158695993) | > loss_mel_ce: 2.200392723083496 (nan) | > loss: 0.008847011253237724 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13756325571603978) | > loader_time: 0.0045 (0.004679440391310738)  --> TIME: 2025-02-27 00:37:59 -- STEP: 31518/36938 -- GLOBAL_STEP: 1472100 | > loss_text_ce: 0.027376966550946236 (0.03049836288738987) | > loss_mel_ce: 2.8716118335723877 (nan) | > loss: 0.011503924615681171 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.1375611956647993) | > loader_time: 0.0046 (0.004679514812752993)  --> TIME: 2025-02-27 00:38:08 -- STEP: 31568/36938 -- GLOBAL_STEP: 1472150 | > loss_text_ce: 0.03596942871809006 (0.030496765695024564) | > loss_mel_ce: 3.826063394546509 (nan) | > loss: 0.01532552856951952 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13754996029000185) | > loader_time: 0.0046 (0.004679403681774452)  --> TIME: 2025-02-27 00:38:17 -- STEP: 31618/36938 -- GLOBAL_STEP: 1472200 | > loss_text_ce: 0.026961935684084892 (0.03049547358804126) | > loss_mel_ce: 2.392987012863159 (nan) | > loss: 0.009602973237633705 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.1375473007862341) | > loader_time: 0.0047 (0.004679444815132444)  --> TIME: 2025-02-27 00:38:27 -- STEP: 31668/36938 -- GLOBAL_STEP: 1472250 | > loss_text_ce: 0.026131656020879745 (0.030495361903921974) | > loss_mel_ce: 2.6168627738952637 (nan) | > loss: 0.010488073341548443 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13754891908061867) | > loader_time: 0.0045 (0.004679489454958728)  --> TIME: 2025-02-27 00:38:36 -- STEP: 31718/36938 -- GLOBAL_STEP: 1472300 | > loss_text_ce: 0.03247511014342308 (0.030495783377223733) | > loss_mel_ce: 2.6693367958068848 (nan) | > loss: 0.010721475817263126 (nan) | > current_lr: 5e-06 | > step_time: 0.1447 (0.1375509857026434) | > loader_time: 0.0047 (0.0046794449924498615)  --> TIME: 2025-02-27 00:38:46 -- STEP: 31768/36938 -- GLOBAL_STEP: 1472350 | > loss_text_ce: 0.0324280746281147 (0.03049594184127349) | > loss_mel_ce: 0.00677490234375 (nan) | > loss: 0.00015556738071609288 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13755718502491598) | > loader_time: 0.0046 (0.004679484065369788)  --> TIME: 2025-02-27 00:38:55 -- STEP: 31818/36938 -- GLOBAL_STEP: 1472400 | > loss_text_ce: 0.036418478935956955 (0.030495450330476206) | > loss_mel_ce: 2.519212007522583 (nan) | > loss: 0.010141391307115555 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13755575632253217) | > loader_time: 0.0048 (0.004679495020900076)  --> TIME: 2025-02-27 00:39:05 -- STEP: 31868/36938 -- GLOBAL_STEP: 1472450 | > loss_text_ce: 0.03370772674679756 (0.03049524141317688) | > loss_mel_ce: 2.1089329719543457 (nan) | > loss: 0.008502542972564697 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13755559071372872) | > loader_time: 0.0046 (0.004679463836499516)  --> TIME: 2025-02-27 00:39:14 -- STEP: 31918/36938 -- GLOBAL_STEP: 1472500 | > loss_text_ce: 0.024399641901254654 (0.03049501703350462) | > loss_mel_ce: 2.6715316772460938 (nan) | > loss: 0.010698141530156136 (nan) | > current_lr: 5e-06 | > step_time: 0.1776 (0.1375557509749833) | > loader_time: 0.0048 (0.004679401444196678)  --> TIME: 2025-02-27 00:39:23 -- STEP: 31968/36938 -- GLOBAL_STEP: 1472550 | > loss_text_ce: 0.03192669153213501 (0.030494488581923093) | > loss_mel_ce: 3.066692590713501 (nan) | > loss: 0.012296108528971672 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.1375535884732239) | > loader_time: 0.0048 (0.004679491905657788)  --> TIME: 2025-02-27 00:39:33 -- STEP: 32018/36938 -- GLOBAL_STEP: 1472600 | > loss_text_ce: 0.03375978767871857 (0.030493254653347502) | > loss_mel_ce: 1.792730689048767 (nan) | > loss: 0.007247978821396828 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13756042312686628) | > loader_time: 0.0046 (0.004679540206077142)  --> TIME: 2025-02-27 00:39:42 -- STEP: 32068/36938 -- GLOBAL_STEP: 1472650 | > loss_text_ce: 0.028730351477861404 (0.03049178061513057) | > loss_mel_ce: 2.525907278060913 (nan) | > loss: 0.010137451812624931 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13755901194844009) | > loader_time: 0.0047 (0.0046795141567565245)  --> TIME: 2025-02-27 00:39:51 -- STEP: 32118/36938 -- GLOBAL_STEP: 1472700 | > loss_text_ce: 0.03195781633257866 (0.030491986228388564) | > loss_mel_ce: 3.0774741172790527 (nan) | > loss: 0.012339016422629356 (nan) | > current_lr: 5e-06 | > step_time: 0.1844 (0.13755540912657138) | > loader_time: 0.0048 (0.004679491907567926)  --> TIME: 2025-02-27 00:40:00 -- STEP: 32168/36938 -- GLOBAL_STEP: 1472750 | > loss_text_ce: 0.029961002990603447 (0.03049032633771932) | > loss_mel_ce: 2.928053855895996 (nan) | > loss: 0.011738155037164688 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.13754933965766453) | > loader_time: 0.0048 (0.00467938355206559)  --> TIME: 2025-02-27 00:40:09 -- STEP: 32218/36938 -- GLOBAL_STEP: 1472800 | > loss_text_ce: 0.031005358323454857 (0.03048993049513379) | > loss_mel_ce: 3.1373066902160645 (nan) | > loss: 0.012572667561471462 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.1375406039287289) | > loader_time: 0.0046 (0.00467926594226664)  --> TIME: 2025-02-27 00:40:19 -- STEP: 32268/36938 -- GLOBAL_STEP: 1472850 | > loss_text_ce: 0.027747880667448044 (0.030488975811879575) | > loss_mel_ce: 2.5411760807037354 (nan) | > loss: 0.010194143280386925 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13753825315271165) | > loader_time: 0.0047 (0.004679191034198139)  --> TIME: 2025-02-27 00:40:28 -- STEP: 32318/36938 -- GLOBAL_STEP: 1472900 | > loss_text_ce: 0.02847541682422161 (0.030488183877528663) | > loss_mel_ce: 3.065654993057251 (nan) | > loss: 0.012278296053409576 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13753775324485937) | > loader_time: 0.0047 (0.004679254637443876)  --> TIME: 2025-02-27 00:40:38 -- STEP: 32368/36938 -- GLOBAL_STEP: 1472950 | > loss_text_ce: 0.03667140752077103 (0.030489000192861448) | > loss_mel_ce: 2.6318814754486084 (nan) | > loss: 0.010589496232569218 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13754226201044925) | > loader_time: 0.0047 (0.004679441172123242)  --> TIME: 2025-02-27 00:40:47 -- STEP: 32418/36938 -- GLOBAL_STEP: 1473000 | > loss_text_ce: 0.03435806557536125 (0.03048740394906281) | > loss_mel_ce: 2.6580324172973633 (nan) | > loss: 0.010684089735150337 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.1375393766956491) | > loader_time: 0.0048 (0.004679456116923905)  --> TIME: 2025-02-27 00:40:56 -- STEP: 32468/36938 -- GLOBAL_STEP: 1473050 | > loss_text_ce: 0.03505827113986015 (0.03048820407430149) | > loss_mel_ce: 2.797478437423706 (nan) | > loss: 0.011240225285291672 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1375390723140217) | > loader_time: 0.0045 (0.004679536502232402)  --> TIME: 2025-02-27 00:41:06 -- STEP: 32518/36938 -- GLOBAL_STEP: 1473100 | > loss_text_ce: 0.02457335591316223 (0.03048626689165127) | > loss_mel_ce: 2.2298731803894043 (nan) | > loss: 0.00894621666520834 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1375364698499423) | > loader_time: 0.0045 (0.004679667531025329)  --> TIME: 2025-02-27 00:41:15 -- STEP: 32568/36938 -- GLOBAL_STEP: 1473150 | > loss_text_ce: 0.030562466010451317 (0.030485175458433093) | > loss_mel_ce: 2.793442487716675 (nan) | > loss: 0.011206368915736675 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.1375330874709641) | > loader_time: 0.0046 (0.004679634687620067)  --> TIME: 2025-02-27 00:41:24 -- STEP: 32618/36938 -- GLOBAL_STEP: 1473200 | > loss_text_ce: 0.028619417920708656 (0.030487179131509157) | > loss_mel_ce: 2.166186571121216 (nan) | > loss: 0.008709548972547054 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.13752807381765766) | > loader_time: 0.0048 (0.004679663958003683)  --> TIME: 2025-02-27 00:41:33 -- STEP: 32668/36938 -- GLOBAL_STEP: 1473250 | > loss_text_ce: 0.03656766936182976 (0.030488082659844748) | > loss_mel_ce: 2.970939874649048 (nan) | > loss: 0.01193455420434475 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13752545050166423) | > loader_time: 0.0045 (0.004679575819739772)  --> TIME: 2025-02-27 00:41:43 -- STEP: 32718/36938 -- GLOBAL_STEP: 1473300 | > loss_text_ce: 0.03962184488773346 (0.030490861669526293) | > loss_mel_ce: 2.7961342334747314 (nan) | > loss: 0.011253001168370247 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13752943137082763) | > loader_time: 0.0047 (0.0046797011197215)  --> TIME: 2025-02-27 00:41:53 -- STEP: 32768/36938 -- GLOBAL_STEP: 1473350 | > loss_text_ce: 0.032168928533792496 (0.030490923174468776) | > loss_mel_ce: 3.0452353954315186 (nan) | > loss: 0.01221192255616188 (nan) | > current_lr: 5e-06 | > step_time: 0.1623 (0.13753486028144818) | > loader_time: 0.0047 (0.004679689955082772)  --> TIME: 2025-02-27 00:42:02 -- STEP: 32818/36938 -- GLOBAL_STEP: 1473400 | > loss_text_ce: 0.030834509059786797 (0.030490298491618696) | > loss_mel_ce: 2.999340772628784 (nan) | > loss: 0.01202450506389141 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13754458738075123) | > loader_time: 0.0047 (0.004679691334574006)  --> TIME: 2025-02-27 00:42:12 -- STEP: 32868/36938 -- GLOBAL_STEP: 1473450 | > loss_text_ce: 0.029041485860943794 (0.030490628777801634) | > loss_mel_ce: 2.705744981765747 (nan) | > loss: 0.01085232850164175 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13755177468645363) | > loader_time: 0.0047 (0.004679794531749484)  --> TIME: 2025-02-27 00:42:21 -- STEP: 32918/36938 -- GLOBAL_STEP: 1473500 | > loss_text_ce: 0.02831067331135273 (0.030489411827665583) | > loss_mel_ce: 2.4231882095336914 (nan) | > loss: 0.009728170931339264 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.1375478922404554) | > loader_time: 0.0045 (0.004679870312859572)  --> TIME: 2025-02-27 00:42:30 -- STEP: 32968/36938 -- GLOBAL_STEP: 1473550 | > loss_text_ce: 0.02042875438928604 (0.030489990276799668) | > loss_mel_ce: 2.38100528717041 (nan) | > loss: 0.009529500268399715 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13754106900756188) | > loader_time: 0.0046 (0.00467994922690245)  --> TIME: 2025-02-27 00:42:40 -- STEP: 33018/36938 -- GLOBAL_STEP: 1473600 | > loss_text_ce: 0.030047958716750145 (0.03048985912056693) | > loss_mel_ce: 2.6067638397216797 (nan) | > loss: 0.010463539510965347 (nan) | > current_lr: 5e-06 | > step_time: 0.1688 (0.13754923021318938) | > loader_time: 0.0047 (0.004680142388784911)  --> TIME: 2025-02-27 00:42:49 -- STEP: 33068/36938 -- GLOBAL_STEP: 1473650 | > loss_text_ce: 0.027088871225714684 (0.030490364469937598) | > loss_mel_ce: 3.0191116333007812 (nan) | > loss: 0.012088097631931305 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.1375402989663796) | > loader_time: 0.0048 (0.004680184365126413)  --> TIME: 2025-02-27 00:42:58 -- STEP: 33118/36938 -- GLOBAL_STEP: 1473700 | > loss_text_ce: 0.033064644783735275 (0.030492295506300348) | > loss_mel_ce: 2.808238983154297 (nan) | > loss: 0.011275014840066433 (nan) | > current_lr: 5e-06 | > step_time: 0.1436 (0.13753879545420147) | > loader_time: 0.0046 (0.004680136960733031)  --> TIME: 2025-02-27 00:43:08 -- STEP: 33168/36938 -- GLOBAL_STEP: 1473750 | > loss_text_ce: 0.032672200351953506 (0.030491855411296235) | > loss_mel_ce: 2.634366273880005 (nan) | > loss: 0.010583486407995224 (nan) | > current_lr: 5e-06 | > step_time: 0.15 (0.13753773866502533) | > loader_time: 0.0047 (0.004680150899689317)  --> TIME: 2025-02-27 00:43:17 -- STEP: 33218/36938 -- GLOBAL_STEP: 1473800 | > loss_text_ce: 0.030269231647253036 (0.03049069428872231) | > loss_mel_ce: 3.1328125 (nan) | > loss: 0.012551912106573582 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13754500640250378) | > loader_time: 0.0046 (0.00468014916432563)  --> TIME: 2025-02-27 00:43:27 -- STEP: 33268/36938 -- GLOBAL_STEP: 1473850 | > loss_text_ce: 0.028854891657829285 (0.030491759498434332) | > loss_mel_ce: 2.5579631328582764 (nan) | > loss: 0.010265151038765907 (nan) | > current_lr: 5e-06 | > step_time: 0.1945 (0.13755231629172457) | > loader_time: 0.0049 (0.0046801211829081086)  --> TIME: 2025-02-27 00:43:36 -- STEP: 33318/36938 -- GLOBAL_STEP: 1473900 | > loss_text_ce: 0.029635900631546974 (0.030492575656310354) | > loss_mel_ce: 1.6387230157852173 (nan) | > loss: 0.006620472297072411 (nan) | > current_lr: 5e-06 | > step_time: 0.1758 (0.137550834698067) | > loader_time: 0.0048 (0.004680049162508149)  --> TIME: 2025-02-27 00:43:45 -- STEP: 33368/36938 -- GLOBAL_STEP: 1473950 | > loss_text_ce: 0.031077606603503227 (0.03049235818771134) | > loss_mel_ce: 1.6694797277450562 (nan) | > loss: 0.006748243700712919 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13754215838668493) | > loader_time: 0.0045 (0.004679970870170106)  --> TIME: 2025-02-27 00:43:55 -- STEP: 33418/36938 -- GLOBAL_STEP: 1474000 | > loss_text_ce: 0.03076435998082161 (0.030489978754627174) | > loss_mel_ce: 0.014251708984375 (nan) | > loss: 0.00017863520770333707 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13755052751338737) | > loader_time: 0.0046 (0.0046800434271904124)  --> TIME: 2025-02-27 00:44:04 -- STEP: 33468/36938 -- GLOBAL_STEP: 1474050 | > loss_text_ce: 0.03364143148064613 (0.030487952179338213) | > loss_mel_ce: 2.2936296463012695 (nan) | > loss: 0.009235203266143799 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13754586224859716) | > loader_time: 0.0046 (0.004679953416530553)  --> TIME: 2025-02-27 00:44:13 -- STEP: 33518/36938 -- GLOBAL_STEP: 1474100 | > loss_text_ce: 0.027596615254878998 (0.03048680153171009) | > loss_mel_ce: 3.0084218978881836 (nan) | > loss: 0.012047693133354187 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.13754141199837017) | > loader_time: 0.0046 (0.004679913999954861)  --> TIME: 2025-02-27 00:44:23 -- STEP: 33568/36938 -- GLOBAL_STEP: 1474150 | > loss_text_ce: 0.029698405414819717 (0.03048696235475429) | > loss_mel_ce: 3.0975990295410156 (nan) | > loss: 0.01240991149097681 (nan) | > current_lr: 5e-06 | > step_time: 0.1353 (0.13754047802849192) | > loader_time: 0.0046 (0.004679842611457016)  --> TIME: 2025-02-27 00:44:32 -- STEP: 33618/36938 -- GLOBAL_STEP: 1474200 | > loss_text_ce: 0.03318397328257561 (0.0304871358829842) | > loss_mel_ce: 2.2459464073181152 (nan) | > loss: 0.009044169448316097 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13754011244441672) | > loader_time: 0.0046 (0.004679818547412463)  --> TIME: 2025-02-27 00:44:42 -- STEP: 33668/36938 -- GLOBAL_STEP: 1474250 | > loss_text_ce: 0.02772732451558113 (0.03049038430988498) | > loss_mel_ce: 2.367393732070923 (nan) | > loss: 0.009504449553787708 (nan) | > current_lr: 5e-06 | > step_time: 0.1558 (0.13754056139344678) | > loader_time: 0.0047 (0.004679800142114)  --> TIME: 2025-02-27 00:44:51 -- STEP: 33718/36938 -- GLOBAL_STEP: 1474300 | > loss_text_ce: 0.020962294191122055 (0.030490318973588417) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.1375373512677955) | > loader_time: 0.0044 (0.004679802382032696)  --> TIME: 2025-02-27 00:45:00 -- STEP: 33768/36938 -- GLOBAL_STEP: 1474350 | > loss_text_ce: 0.023423640057444572 (0.030488825717497316) | > loss_mel_ce: 2.5683891773223877 (nan) | > loss: 0.010284972377121449 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13753281909689116) | > loader_time: 0.0046 (0.004679833499777003)  --> TIME: 2025-02-27 00:45:09 -- STEP: 33818/36938 -- GLOBAL_STEP: 1474400 | > loss_text_ce: 0.033172011375427246 (0.030488419001651012) | > loss_mel_ce: 2.9593241214752197 (nan) | > loss: 0.011874984949827194 (nan) | > current_lr: 5e-06 | > step_time: 0.147 (0.13752720129539941) | > loader_time: 0.0046 (0.004679871335853944)  --> TIME: 2025-02-27 00:45:18 -- STEP: 33868/36938 -- GLOBAL_STEP: 1474450 | > loss_text_ce: 0.03061065636575222 (0.030489909863381577) | > loss_mel_ce: 2.68190598487854 (nan) | > loss: 0.010763955302536488 (nan) | > current_lr: 5e-06 | > step_time: 0.1816 (0.1375239418954398) | > loader_time: 0.0048 (0.004679787689755084)  --> TIME: 2025-02-27 00:45:28 -- STEP: 33918/36938 -- GLOBAL_STEP: 1474500 | > loss_text_ce: 0.026159895583987236 (0.03048971999023152) | > loss_mel_ce: 0.00433349609375 (nan) | > loss: 0.00012100552703486755 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13752116159670416) | > loader_time: 0.0045 (0.004679703425669452)  --> TIME: 2025-02-27 00:45:37 -- STEP: 33968/36938 -- GLOBAL_STEP: 1474550 | > loss_text_ce: 0.022515244781970978 (0.030490303752319068) | > loss_mel_ce: 2.208789110183716 (nan) | > loss: 0.008854382671415806 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13751775756288775) | > loader_time: 0.0045 (0.004679615759815906)  --> TIME: 2025-02-27 00:45:46 -- STEP: 34018/36938 -- GLOBAL_STEP: 1474600 | > loss_text_ce: 0.0310351699590683 (0.030488546990200135) | > loss_mel_ce: 2.783886671066284 (nan) | > loss: 0.011170325800776482 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13751511207522005) | > loader_time: 0.0045 (0.0046796460891500745)  --> TIME: 2025-02-27 00:45:55 -- STEP: 34068/36938 -- GLOBAL_STEP: 1474650 | > loss_text_ce: 0.024050934240221977 (0.030488171001547534) | > loss_mel_ce: 3.0822372436523438 (nan) | > loss: 0.012326541356742382 (nan) | > current_lr: 5e-06 | > step_time: 0.1853 (0.13750551092494473) | > loader_time: 0.0049 (0.004679608046903589)  --> TIME: 2025-02-27 00:46:05 -- STEP: 34118/36938 -- GLOBAL_STEP: 1474700 | > loss_text_ce: 0.031026305630803108 (0.03048790487348545) | > loss_mel_ce: 2.5069057941436768 (nan) | > loss: 0.010071160271763802 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.1375090394561513) | > loader_time: 0.0046 (0.004679665349420003)  --> TIME: 2025-02-27 00:46:14 -- STEP: 34168/36938 -- GLOBAL_STEP: 1474750 | > loss_text_ce: 0.035796377807855606 (0.030488093081560037) | > loss_mel_ce: 3.16485595703125 (nan) | > loss: 0.012701001949608326 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.1375110626178834) | > loader_time: 0.0048 (0.004679621005610981)  --> TIME: 2025-02-27 00:46:23 -- STEP: 34218/36938 -- GLOBAL_STEP: 1474800 | > loss_text_ce: 0.0371326319873333 (0.03048911344568784) | > loss_mel_ce: 2.3226945400238037 (nan) | > loss: 0.009364394471049309 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.13749863369250664) | > loader_time: 0.0046 (0.004679493960133612)  --> TIME: 2025-02-27 00:46:32 -- STEP: 34268/36938 -- GLOBAL_STEP: 1474850 | > loss_text_ce: 0.02790813148021698 (0.030490655750980377) | > loss_mel_ce: 2.5576012134552 (nan) | > loss: 0.010259957984089851 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13749639430234736) | > loader_time: 0.0047 (0.004679419028100871)  --> TIME: 2025-02-27 00:46:41 -- STEP: 34318/36938 -- GLOBAL_STEP: 1474900 | > loss_text_ce: 0.02955600619316101 (0.030491038465776218) | > loss_mel_ce: 0.9919775128364563 (nan) | > loss: 0.0040537044405937195 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.1374871235390113) | > loader_time: 0.0045 (0.0046793874573531555)  --> TIME: 2025-02-27 00:46:50 -- STEP: 34368/36938 -- GLOBAL_STEP: 1474950 | > loss_text_ce: 0.035776540637016296 (0.030492306887317938) | > loss_mel_ce: 2.2108957767486572 (nan) | > loss: 0.008915367536246777 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13748004902813207) | > loader_time: 0.0047 (0.004679396672233345)  --> TIME: 2025-02-27 00:47:00 -- STEP: 34418/36938 -- GLOBAL_STEP: 1475000 | > loss_text_ce: 0.027981573715806007 (0.030492581095148918) | > loss_mel_ce: 2.8366284370422363 (nan) | > loss: 0.011367500759661198 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.1374838851304377) | > loader_time: 0.0046 (0.0046795242245176)  --> TIME: 2025-02-27 00:47:09 -- STEP: 34468/36938 -- GLOBAL_STEP: 1475050 | > loss_text_ce: 0.02897540107369423 (0.030493846089781045) | > loss_mel_ce: 0.13484779000282288 (nan) | > loss: 0.0006500920280814171 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.137478357354961) | > loader_time: 0.0047 (0.004679479371564844)  --> TIME: 2025-02-27 00:47:18 -- STEP: 34518/36938 -- GLOBAL_STEP: 1475100 | > loss_text_ce: 0.0303814634680748 (0.03049284285500727) | > loss_mel_ce: 3.318018913269043 (nan) | > loss: 0.01328730396926403 (nan) | > current_lr: 5e-06 | > step_time: 0.1442 (0.1374772142870784) | > loader_time: 0.0047 (0.004679380034273199)  --> TIME: 2025-02-27 00:47:28 -- STEP: 34568/36938 -- GLOBAL_STEP: 1475150 | > loss_text_ce: 0.028871456161141396 (0.030493269615652626) | > loss_mel_ce: 2.313910484313965 (nan) | > loss: 0.009296754375100136 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13748327869420995) | > loader_time: 0.0045 (0.004679287412436454)  --> TIME: 2025-02-27 00:47:38 -- STEP: 34618/36938 -- GLOBAL_STEP: 1475200 | > loss_text_ce: 0.031689345836639404 (0.03049145440000238) | > loss_mel_ce: 0.65576171875 (nan) | > loss: 0.0027279804926365614 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13748768226644004) | > loader_time: 0.0045 (0.004679189031917144)  --> TIME: 2025-02-27 00:47:47 -- STEP: 34668/36938 -- GLOBAL_STEP: 1475250 | > loss_text_ce: 0.0370306596159935 (0.030494189073758886) | > loss_mel_ce: 2.6326098442077637 (nan) | > loss: 0.010593811981379986 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13748615174828804) | > loader_time: 0.0046 (0.004679075715945439)  --> TIME: 2025-02-27 00:47:56 -- STEP: 34718/36938 -- GLOBAL_STEP: 1475300 | > loss_text_ce: 0.03297312185168266 (0.030494244580941837) | > loss_mel_ce: 2.7968220710754395 (nan) | > loss: 0.011229346506297588 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.1374876806555929) | > loader_time: 0.0046 (0.004679138062015292)  --> TIME: 2025-02-27 00:48:05 -- STEP: 34768/36938 -- GLOBAL_STEP: 1475350 | > loss_text_ce: 0.032797861844301224 (0.03049446487205637) | > loss_mel_ce: 2.055469274520874 (nan) | > loss: 0.008286775089800358 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13748340257235592) | > loader_time: 0.0045 (0.004679027675649777)  --> TIME: 2025-02-27 00:48:15 -- STEP: 34818/36938 -- GLOBAL_STEP: 1475400 | > loss_text_ce: 0.025636157020926476 (0.030494373350189292) | > loss_mel_ce: 0.14891991019248962 (nan) | > loss: 0.0006926828064024448 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.1374898334628399) | > loader_time: 0.0046 (0.0046789378888151874)  --> TIME: 2025-02-27 00:48:25 -- STEP: 34868/36938 -- GLOBAL_STEP: 1475450 | > loss_text_ce: 0.041198961436748505 (0.030494293537726674) | > loss_mel_ce: 2.950644016265869 (nan) | > loss: 0.01187239307910204 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13749449341562694) | > loader_time: 0.0045 (0.0046789486829165875)  --> TIME: 2025-02-27 00:48:34 -- STEP: 34918/36938 -- GLOBAL_STEP: 1475500 | > loss_text_ce: 0.03887130320072174 (0.030494563424216373) | > loss_mel_ce: 2.4412364959716797 (nan) | > loss: 0.009841698221862316 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.1374953216591186) | > loader_time: 0.0045 (0.0046789333428282264)  --> TIME: 2025-02-27 00:48:43 -- STEP: 34968/36938 -- GLOBAL_STEP: 1475550 | > loss_text_ce: 0.024469034746289253 (0.030494244381470786) | > loss_mel_ce: 2.04736328125 (nan) | > loss: 0.00822155736386776 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13749208623635817) | > loader_time: 0.0045 (0.004678886110191944)  --> TIME: 2025-02-27 00:48:53 -- STEP: 35018/36938 -- GLOBAL_STEP: 1475600 | > loss_text_ce: 0.034199561923742294 (0.030493765367649035) | > loss_mel_ce: 2.3718347549438477 (nan) | > loss: 0.009547755122184753 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13749456889720965) | > loader_time: 0.0046 (0.004678989962757012)  --> TIME: 2025-02-27 00:49:02 -- STEP: 35068/36938 -- GLOBAL_STEP: 1475650 | > loss_text_ce: 0.035957422107458115 (0.030492821966793583) | > loss_mel_ce: 3.316602945327759 (nan) | > loss: 0.01330381166189909 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13749766222655813) | > loader_time: 0.0047 (0.004678998846576488)  --> TIME: 2025-02-27 00:49:12 -- STEP: 35118/36938 -- GLOBAL_STEP: 1475700 | > loss_text_ce: 0.022952798753976822 (0.030492546591328853) | > loss_mel_ce: 0.0035247802734375 (nan) | > loss: 0.00010506976104807109 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13750539218261232) | > loader_time: 0.0046 (0.004679086485473411)  --> TIME: 2025-02-27 00:49:21 -- STEP: 35168/36938 -- GLOBAL_STEP: 1475750 | > loss_text_ce: 0.026374369859695435 (0.030492102975444496) | > loss_mel_ce: 2.474092483520508 (nan) | > loss: 0.009922487661242485 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13749861315204592) | > loader_time: 0.0045 (0.0046790732821624685)  --> TIME: 2025-02-27 00:49:31 -- STEP: 35218/36938 -- GLOBAL_STEP: 1475800 | > loss_text_ce: 0.029825784265995026 (0.03049113761738195) | > loss_mel_ce: 3.145498752593994 (nan) | > loss: 0.012600494548678398 (nan) | > current_lr: 5e-06 | > step_time: 0.1868 (0.13750376752263793) | > loader_time: 0.0049 (0.0046790560206168355)  --> TIME: 2025-02-27 00:49:40 -- STEP: 35268/36938 -- GLOBAL_STEP: 1475850 | > loss_text_ce: 0.03671302646398544 (0.030492124937877053) | > loss_mel_ce: 2.713623046875 (nan) | > loss: 0.010914033278822899 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13749597093422689) | > loader_time: 0.0046 (0.004678989857438654)  --> TIME: 2025-02-27 00:49:50 -- STEP: 35318/36938 -- GLOBAL_STEP: 1475900 | > loss_text_ce: 0.027049269527196884 (0.030489703435673865) | > loss_mel_ce: 1.6356741189956665 (nan) | > loss: 0.006598109379410744 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13750240557582152) | > loader_time: 0.0047 (0.004678991576865066)  --> TIME: 2025-02-27 00:49:59 -- STEP: 35368/36938 -- GLOBAL_STEP: 1475950 | > loss_text_ce: 0.03245152533054352 (0.03048836720291428) | > loss_mel_ce: 1.6005783081054688 (nan) | > loss: 0.006480277515947819 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13750449162725634) | > loader_time: 0.0067 (0.004679029592155138)  --> TIME: 2025-02-27 00:50:09 -- STEP: 35418/36938 -- GLOBAL_STEP: 1476000 | > loss_text_ce: 0.036384474486112595 (0.03048844270424087) | > loss_mel_ce: 2.590458631515503 (nan) | > loss: 0.010423981584608555 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13750931529889956) | > loader_time: 0.0047 (0.004679176746684896)  --> TIME: 2025-02-27 00:50:18 -- STEP: 35468/36938 -- GLOBAL_STEP: 1476050 | > loss_text_ce: 0.03246065601706505 (0.030488691722141366) | > loss_mel_ce: 3.0368118286132812 (nan) | > loss: 0.012179653160274029 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.13751309127996875) | > loader_time: 0.0047 (0.004679197649064725)  --> TIME: 2025-02-27 00:50:28 -- STEP: 35518/36938 -- GLOBAL_STEP: 1476100 | > loss_text_ce: 0.0289450753480196 (0.030489538583395908) | > loss_mel_ce: 0.09453125298023224 (nan) | > loss: 0.0004899854538962245 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13751582945243612) | > loader_time: 0.0045 (0.004679086663592956)  --> TIME: 2025-02-27 00:50:37 -- STEP: 35568/36938 -- GLOBAL_STEP: 1476150 | > loss_text_ce: 0.03431236371397972 (0.03048950173033811) | > loss_mel_ce: 2.505340099334717 (nan) | > loss: 0.010077985934913158 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13750947854350482) | > loader_time: 0.0046 (0.004679132227794736)  --> TIME: 2025-02-27 00:50:47 -- STEP: 35618/36938 -- GLOBAL_STEP: 1476200 | > loss_text_ce: 0.036516401916742325 (0.030490129328272116) | > loss_mel_ce: 2.9433176517486572 (nan) | > loss: 0.011824739165604115 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13751761914236668) | > loader_time: 0.0047 (0.004679144503152086)  --> TIME: 2025-02-27 00:50:56 -- STEP: 35668/36938 -- GLOBAL_STEP: 1476250 | > loss_text_ce: 0.02446933276951313 (0.030488738412525963) | > loss_mel_ce: 2.8777577877044678 (nan) | > loss: 0.011516775004565716 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.13751636749160392) | > loader_time: 0.0047 (0.004679155400532811)  --> TIME: 2025-02-27 00:51:05 -- STEP: 35718/36938 -- GLOBAL_STEP: 1476300 | > loss_text_ce: 0.02413121797144413 (0.03048729817408543) | > loss_mel_ce: 2.795156717300415 (nan) | > loss: 0.011187651194632053 (nan) | > current_lr: 5e-06 | > step_time: 0.147 (0.1375116827346566) | > loader_time: 0.0046 (0.0046790284147668415)  --> TIME: 2025-02-27 00:51:14 -- STEP: 35768/36938 -- GLOBAL_STEP: 1476350 | > loss_text_ce: 0.03855027258396149 (0.030487143333492093) | > loss_mel_ce: 3.129288911819458 (nan) | > loss: 0.012570791877806187 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13751371625791678) | > loader_time: 0.0045 (0.00467893780544289)  --> TIME: 2025-02-27 00:51:24 -- STEP: 35818/36938 -- GLOBAL_STEP: 1476400 | > loss_text_ce: 0.031716905534267426 (0.030488738475939033) | > loss_mel_ce: 2.420927047729492 (nan) | > loss: 0.009732713922858238 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13751036884250972) | > loader_time: 0.0046 (0.004678961965624976)  --> TIME: 2025-02-27 00:51:33 -- STEP: 35868/36938 -- GLOBAL_STEP: 1476450 | > loss_text_ce: 0.032873354852199554 (0.030489555695422357) | > loss_mel_ce: 3.302150011062622 (nan) | > loss: 0.013234220445156097 (nan) | > current_lr: 5e-06 | > step_time: 0.1704 (0.13750911087339437) | > loader_time: 0.0048 (0.00467892570268014)  --> TIME: 2025-02-27 00:51:42 -- STEP: 35918/36938 -- GLOBAL_STEP: 1476500 | > loss_text_ce: 0.02756735309958458 (0.030489810074386697) | > loss_mel_ce: 0.47614437341690063 (nan) | > loss: 0.0019988559652119875 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13750537179667247) | > loader_time: 0.0046 (0.00467888274352921)  --> TIME: 2025-02-27 00:51:52 -- STEP: 35968/36938 -- GLOBAL_STEP: 1476550 | > loss_text_ce: 0.02744423970580101 (0.030490256401457378) | > loss_mel_ce: 2.412004232406616 (nan) | > loss: 0.009680351242423058 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.1375111092902797) | > loader_time: 0.0046 (0.0046789141047043886)  --> TIME: 2025-02-27 00:52:01 -- STEP: 36018/36938 -- GLOBAL_STEP: 1476600 | > loss_text_ce: 0.029702818021178246 (0.030490512472239013) | > loss_mel_ce: 2.1037399768829346 (nan) | > loss: 0.008466043509542942 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.137508814507901) | > loader_time: 0.0046 (0.004678844598008383)  --> TIME: 2025-02-27 00:52:10 -- STEP: 36068/36938 -- GLOBAL_STEP: 1476650 | > loss_text_ce: 0.027641238644719124 (0.030489101481360144) | > loss_mel_ce: 2.3975131511688232 (nan) | > loss: 0.009623629041016102 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13750621953745068) | > loader_time: 0.0045 (0.0046787762557296084)  --> TIME: 2025-02-27 00:52:20 -- STEP: 36118/36938 -- GLOBAL_STEP: 1476700 | > loss_text_ce: 0.02856004238128662 (0.030489657885052597) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.1375034098292749) | > loader_time: 0.0047 (0.004678753986925866)  --> TIME: 2025-02-27 00:52:29 -- STEP: 36168/36938 -- GLOBAL_STEP: 1476750 | > loss_text_ce: 0.026853792369365692 (0.030488146131093855) | > loss_mel_ce: 2.852689743041992 (nan) | > loss: 0.01142676081508398 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13749751479613553) | > loader_time: 0.0045 (0.0046786788724951745)  --> TIME: 2025-02-27 00:52:38 -- STEP: 36218/36938 -- GLOBAL_STEP: 1476800 | > loss_text_ce: 0.03797729313373566 (0.030490245365043616) | > loss_mel_ce: 2.6927032470703125 (nan) | > loss: 0.010836034081876278 (nan) | > current_lr: 5e-06 | > step_time: 0.1819 (0.13750307660948324) | > loader_time: 0.0049 (0.004678807573794083)  --> TIME: 2025-02-27 00:52:48 -- STEP: 36268/36938 -- GLOBAL_STEP: 1476850 | > loss_text_ce: 0.022008031606674194 (0.030490208996634547) | > loss_mel_ce: 2.2030298709869385 (nan) | > loss: 0.008829515427350998 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13751002871058018) | > loader_time: 0.0045 (0.004678814199749976)  --> TIME: 2025-02-27 00:52:58 -- STEP: 36318/36938 -- GLOBAL_STEP: 1476900 | > loss_text_ce: 0.028966479003429413 (0.030489932063666333) | > loss_mel_ce: 3.1425583362579346 (nan) | > loss: 0.012585416436195374 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13751779152594806) | > loader_time: 0.0047 (0.004678876752258414)  --> TIME: 2025-02-27 00:53:08 -- STEP: 36368/36938 -- GLOBAL_STEP: 1476950 | > loss_text_ce: 0.02762887254357338 (0.030492203955141014) | > loss_mel_ce: 2.572897434234619 (nan) | > loss: 0.010319549590349197 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.13752962728848375) | > loader_time: 0.0048 (0.004678912162518583)  --> TIME: 2025-02-27 00:53:17 -- STEP: 36418/36938 -- GLOBAL_STEP: 1477000 | > loss_text_ce: 0.02698954939842224 (0.03049159239989821) | > loss_mel_ce: 2.4724910259246826 (nan) | > loss: 0.00991857424378395 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.1375331190601459) | > loader_time: 0.0046 (0.004678885445340037)  --> TIME: 2025-02-27 00:53:27 -- STEP: 36468/36938 -- GLOBAL_STEP: 1477050 | > loss_text_ce: 0.031139595434069633 (0.030491845475392174) | > loss_mel_ce: 2.5418615341186523 (nan) | > loss: 0.010210322216153145 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13753446105986505) | > loader_time: 0.0046 (0.004678798588768171)  --> TIME: 2025-02-27 00:53:36 -- STEP: 36518/36938 -- GLOBAL_STEP: 1477100 | > loss_text_ce: 0.029425552114844322 (0.03049037694546493) | > loss_mel_ce: 2.7786989212036133 (nan) | > loss: 0.011143351905047894 (nan) | > current_lr: 5e-06 | > step_time: 0.1435 (0.13754148474527766) | > loader_time: 0.0047 (0.004678787775894733)  --> TIME: 2025-02-27 00:53:45 -- STEP: 36568/36938 -- GLOBAL_STEP: 1477150 | > loss_text_ce: 0.03218219056725502 (0.030489536227684966) | > loss_mel_ce: 0.1994977742433548 (nan) | > loss: 0.0009193649748340249 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.13753673142819334) | > loader_time: 0.0046 (0.0046787326314000725)  --> TIME: 2025-02-27 00:53:55 -- STEP: 36618/36938 -- GLOBAL_STEP: 1477200 | > loss_text_ce: 0.030578413978219032 (0.03048906494285278) | > loss_mel_ce: 3.0644240379333496 (nan) | > loss: 0.012281755916774273 (nan) | > current_lr: 5e-06 | > step_time: 0.1618 (0.13754153576545172) | > loader_time: 0.0068 (0.004678794789323434)  --> TIME: 2025-02-27 00:54:05 -- STEP: 36668/36938 -- GLOBAL_STEP: 1477250 | > loss_text_ce: 0.045616257935762405 (0.030488534054996862) | > loss_mel_ce: 2.355154514312744 (nan) | > loss: 0.009526868350803852 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13754760564914553) | > loader_time: 0.0046 (0.004678807966553394)  --> TIME: 2025-02-27 00:54:14 -- STEP: 36718/36938 -- GLOBAL_STEP: 1477300 | > loss_text_ce: 0.031158192083239555 (0.03048820161128059) | > loss_mel_ce: 2.5107264518737793 (nan) | > loss: 0.0100868446752429 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.1375507988099394) | > loader_time: 0.0049 (0.004678831685374468)  --> TIME: 2025-02-27 00:54:24 -- STEP: 36768/36938 -- GLOBAL_STEP: 1477350 | > loss_text_ce: 0.026444099843502045 (0.030490616268948542) | > loss_mel_ce: 2.537984609603882 (nan) | > loss: 0.010176305659115314 (nan) | > current_lr: 5e-06 | > step_time: 0.184 (0.1375530999157096) | > loader_time: 0.0049 (0.004678849348096506)  --> TIME: 2025-02-27 00:54:33 -- STEP: 36818/36938 -- GLOBAL_STEP: 1477400 | > loss_text_ce: 0.034303929656744 (0.03049185662947734) | > loss_mel_ce: 3.3909971714019775 (nan) | > loss: 0.013592465780675411 (nan) | > current_lr: 5e-06 | > step_time: 0.1861 (0.137556545051489) | > loader_time: 0.0048 (0.004678828141633532)  --> TIME: 2025-02-27 00:54:43 -- STEP: 36868/36938 -- GLOBAL_STEP: 1477450 | > loss_text_ce: 0.03485281020402908 (0.03049181109759147) | > loss_mel_ce: 2.820141553878784 (nan) | > loss: 0.011329343542456627 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13755774014914685) | > loader_time: 0.0045 (0.0046787983982918184)  --> TIME: 2025-02-27 00:54:52 -- STEP: 36918/36938 -- GLOBAL_STEP: 1477500 | > loss_text_ce: 0.029646016657352448 (0.030492007960302766) | > loss_mel_ce: 0.08740234375 (nan) | > loss: 0.0004644776345230639 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13756311530055323) | > loader_time: 0.0046 (0.004678756316670851)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.021032498614622825 (+0.00014230166331376595) | > avg_loss_text_ce: 0.0297355986762755 (-0.00012980580403662914) | > avg_loss_mel_ce: 2.443853760414785 (-0.003185210546644779) | > avg_loss: 2.473589366910483 (-0.0033150064473108642) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_1477520.pth  > EPOCH: 41/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-27 00:55:21)   --> TIME: 2025-02-27 00:55:27 -- STEP: 30/36938 -- GLOBAL_STEP: 1477550 | > loss_text_ce: 0.03537782281637192 (0.03051006468012929) | > loss_mel_ce: 1.95344078540802 (2.214665838082631) | > loss: 0.007892138324677944 (0.008909428639647861) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13113813400268554) | > loader_time: 0.0047 (0.004657729466756185)  --> TIME: 2025-02-27 00:55:36 -- STEP: 80/36938 -- GLOBAL_STEP: 1477600 | > loss_text_ce: 0.03413122519850731 (0.030809558078180997) | > loss_mel_ce: 3.1142971515655518 (nan) | > loss: 0.01249376405030489 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13128482699394226) | > loader_time: 0.0047 (0.004699292778968812)  --> TIME: 2025-02-27 00:55:45 -- STEP: 130/36938 -- GLOBAL_STEP: 1477650 | > loss_text_ce: 0.028761612251400948 (0.03004403910241448) | > loss_mel_ce: 2.6369473934173584 (nan) | > loss: 0.010578211396932602 (nan) | > current_lr: 5e-06 | > step_time: 0.1757 (0.13278613824110763) | > loader_time: 0.0048 (0.004701253084036022)  --> TIME: 2025-02-27 00:55:54 -- STEP: 180/36938 -- GLOBAL_STEP: 1477700 | > loss_text_ce: 0.02286500670015812 (0.030125607865759067) | > loss_mel_ce: 2.925179958343506 (nan) | > loss: 0.011698592454195023 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.13395984172821046) | > loader_time: 0.0047 (0.004714547263251412)  --> TIME: 2025-02-27 00:56:03 -- STEP: 230/36938 -- GLOBAL_STEP: 1477750 | > loss_text_ce: 0.03186678886413574 (0.0300958432054714) | > loss_mel_ce: 2.7886979579925537 (nan) | > loss: 0.011192717589437962 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13325633899025305) | > loader_time: 0.0045 (0.004702936048093049)  --> TIME: 2025-02-27 00:56:13 -- STEP: 280/36938 -- GLOBAL_STEP: 1477800 | > loss_text_ce: 0.028399860486388206 (0.0300753775418603) | > loss_mel_ce: 2.6837687492370605 (nan) | > loss: 0.010762575082480907 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13443171381950383) | > loader_time: 0.0046 (0.004692508493150983)  --> TIME: 2025-02-27 00:56:22 -- STEP: 330/36938 -- GLOBAL_STEP: 1477850 | > loss_text_ce: 0.025960735976696014 (0.029937093707761077) | > loss_mel_ce: 2.7650835514068604 (nan) | > loss: 0.011075573042035103 (nan) | > current_lr: 5e-06 | > step_time: 0.1862 (0.13529464765028523) | > loader_time: 0.0048 (0.0046892823594989206)  --> TIME: 2025-02-27 00:56:32 -- STEP: 380/36938 -- GLOBAL_STEP: 1477900 | > loss_text_ce: 0.036111410707235336 (0.029974535210548263) | > loss_mel_ce: 2.424879789352417 (nan) | > loss: 0.009765838272869587 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13571587424529233) | > loader_time: 0.0045 (0.004685496656518234)  --> TIME: 2025-02-27 00:56:41 -- STEP: 430/36938 -- GLOBAL_STEP: 1477950 | > loss_text_ce: 0.027401737868785858 (0.029926702432161154) | > loss_mel_ce: 2.4423701763153076 (nan) | > loss: 0.00980068277567625 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.13594023848688883) | > loader_time: 0.0046 (0.004682339069455169)  --> TIME: 2025-02-27 00:56:50 -- STEP: 480/36938 -- GLOBAL_STEP: 1478000 | > loss_text_ce: 0.025008289143443108 (0.029941388079896568) | > loss_mel_ce: 0.0016326904296875 (nan) | > loss: 0.00010571817983873188 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13618960181872056) | > loader_time: 0.0045 (0.004679919779300689)  --> TIME: 2025-02-27 00:57:00 -- STEP: 530/36938 -- GLOBAL_STEP: 1478050 | > loss_text_ce: 0.03251582756638527 (0.029875166619301965) | > loss_mel_ce: 3.3100240230560303 (nan) | > loss: 0.013264047913253307 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.13590930992702274) | > loader_time: 0.0051 (0.004678056824882076)  --> TIME: 2025-02-27 00:57:09 -- STEP: 580/36938 -- GLOBAL_STEP: 1478100 | > loss_text_ce: 0.03391740843653679 (0.02993366189686389) | > loss_mel_ce: 2.823918581008911 (nan) | > loss: 0.011340619996190071 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13652892318265206) | > loader_time: 0.0047 (0.004681811250489336)  --> TIME: 2025-02-27 00:57:19 -- STEP: 630/36938 -- GLOBAL_STEP: 1478150 | > loss_text_ce: 0.028345542028546333 (0.03001107688224505) | > loss_mel_ce: 2.5771334171295166 (nan) | > loss: 0.010339203290641308 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13671785195668554) | > loader_time: 0.0045 (0.004685633901565798)  --> TIME: 2025-02-27 00:57:28 -- STEP: 680/36938 -- GLOBAL_STEP: 1478200 | > loss_text_ce: 0.02378968708217144 (0.030052859853843556) | > loss_mel_ce: 2.973020553588867 (nan) | > loss: 0.011892104521393776 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13709814513430887) | > loader_time: 0.0046 (0.004686375225291542)  --> TIME: 2025-02-27 00:57:38 -- STEP: 730/36938 -- GLOBAL_STEP: 1478250 | > loss_text_ce: 0.03409479558467865 (0.030066972390480645) | > loss_mel_ce: 2.6898555755615234 (nan) | > loss: 0.010809327475726604 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13704715624247527) | > loader_time: 0.0046 (0.004690338814095285)  --> TIME: 2025-02-27 00:57:47 -- STEP: 780/36938 -- GLOBAL_STEP: 1478300 | > loss_text_ce: 0.02563430741429329 (0.02992857521853577) | > loss_mel_ce: 2.4569199085235596 (nan) | > loss: 0.009851406328380108 (nan) | > current_lr: 5e-06 | > step_time: 0.1883 (0.1368568331767353) | > loader_time: 0.0049 (0.004683408064719967)  --> TIME: 2025-02-27 00:57:56 -- STEP: 830/36938 -- GLOBAL_STEP: 1478350 | > loss_text_ce: 0.04214851185679436 (0.02998209810333259) | > loss_mel_ce: 1.3007510900497437 (nan) | > loss: 0.005328966770321131 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13722353096467915) | > loader_time: 0.0046 (0.0046795020620507)  --> TIME: 2025-02-27 00:58:06 -- STEP: 880/36938 -- GLOBAL_STEP: 1478400 | > loss_text_ce: 0.02354525588452816 (0.02998918863330883) | > loss_mel_ce: 2.6115224361419678 (nan) | > loss: 0.01045661885291338 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.13717042261903947) | > loader_time: 0.0049 (0.004675094918771228)  --> TIME: 2025-02-27 00:58:15 -- STEP: 930/36938 -- GLOBAL_STEP: 1478450 | > loss_text_ce: 0.029722342267632484 (0.029992763671062645) | > loss_mel_ce: 2.015847682952881 (nan) | > loss: 0.008117342367768288 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13699574573065654) | > loader_time: 0.0045 (0.004671884608525106)  --> TIME: 2025-02-27 00:58:24 -- STEP: 980/36938 -- GLOBAL_STEP: 1478500 | > loss_text_ce: 0.026144864037632942 (0.029995782010979494) | > loss_mel_ce: 2.5020012855529785 (nan) | > loss: 0.01003232691437006 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.1367375398168759) | > loader_time: 0.0047 (0.00466734633153799)  --> TIME: 2025-02-27 00:58:33 -- STEP: 1030/36938 -- GLOBAL_STEP: 1478550 | > loss_text_ce: 0.03546047955751419 (0.030048059454205835) | > loss_mel_ce: 2.325455904006958 (nan) | > loss: 0.0093687167391181 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.1367185641260981) | > loader_time: 0.0045 (0.0046642453925123525)  --> TIME: 2025-02-27 00:58:43 -- STEP: 1080/36938 -- GLOBAL_STEP: 1478600 | > loss_text_ce: 0.03230167552828789 (0.030057078676677687) | > loss_mel_ce: 2.8155674934387207 (nan) | > loss: 0.011301068589091301 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1368273794651032) | > loader_time: 0.0045 (0.00466371575991313)  --> TIME: 2025-02-27 00:58:52 -- STEP: 1130/36938 -- GLOBAL_STEP: 1478650 | > loss_text_ce: 0.03260958194732666 (0.030021095334395634) | > loss_mel_ce: 1.4613028764724731 (nan) | > loss: 0.005928224418312311 (nan) | > current_lr: 5e-06 | > step_time: 0.1078 (0.13687097097920112) | > loader_time: 0.0044 (0.00466228818471453)  --> TIME: 2025-02-27 00:59:01 -- STEP: 1180/36938 -- GLOBAL_STEP: 1478700 | > loss_text_ce: 0.02641412615776062 (0.03004894370797201) | > loss_mel_ce: 3.2916667461395264 (nan) | > loss: 0.013166988268494606 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.13686621754856437) | > loader_time: 0.0045 (0.0046598577903488965)  --> TIME: 2025-02-27 00:59:10 -- STEP: 1230/36938 -- GLOBAL_STEP: 1478750 | > loss_text_ce: 0.02888498455286026 (0.030040182561681766) | > loss_mel_ce: 1.883150339126587 (nan) | > loss: 0.007587442174553871 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13675791151155312) | > loader_time: 0.0045 (0.004657824446515346)  --> TIME: 2025-02-27 00:59:20 -- STEP: 1280/36938 -- GLOBAL_STEP: 1478800 | > loss_text_ce: 0.029688110575079918 (0.030074310173949924) | > loss_mel_ce: 2.0416276454925537 (nan) | > loss: 0.008219507522881031 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13692949004471316) | > loader_time: 0.0045 (0.004656458273530013)  --> TIME: 2025-02-27 00:59:30 -- STEP: 1330/36938 -- GLOBAL_STEP: 1478850 | > loss_text_ce: 0.026791483163833618 (0.03012354658860573) | > loss_mel_ce: 2.76875901222229 (nan) | > loss: 0.011093455366790295 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1370590163352796) | > loader_time: 0.0045 (0.004658380486911407)  --> TIME: 2025-02-27 00:59:39 -- STEP: 1380/36938 -- GLOBAL_STEP: 1478900 | > loss_text_ce: 0.031433701515197754 (0.030105044457224617) | > loss_mel_ce: 3.578726053237915 (nan) | > loss: 0.014326032251119614 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13700669416482908) | > loader_time: 0.0048 (0.004657894632090701)  --> TIME: 2025-02-27 00:59:48 -- STEP: 1430/36938 -- GLOBAL_STEP: 1478950 | > loss_text_ce: 0.028219623491168022 (0.030131195718096894) | > loss_mel_ce: 2.478095531463623 (nan) | > loss: 0.009945696219801903 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13711555554316615) | > loader_time: 0.0046 (0.004658557651759869)  --> TIME: 2025-02-27 00:59:58 -- STEP: 1480/36938 -- GLOBAL_STEP: 1479000 | > loss_text_ce: 0.03325219452381134 (0.030172263353282738) | > loss_mel_ce: 3.2758290767669678 (nan) | > loss: 0.013131275773048401 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13711902885823657) | > loader_time: 0.0047 (0.004659259319305426)  --> TIME: 2025-02-27 01:00:07 -- STEP: 1530/36938 -- GLOBAL_STEP: 1479050 | > loss_text_ce: 0.03244691714644432 (0.030167629701858448) | > loss_mel_ce: 2.711247444152832 (nan) | > loss: 0.01088767684996128 (nan) | > current_lr: 5e-06 | > step_time: 0.1705 (0.13710151466668843) | > loader_time: 0.0049 (0.004659129903207429)  --> TIME: 2025-02-27 01:00:17 -- STEP: 1580/36938 -- GLOBAL_STEP: 1479100 | > loss_text_ce: 0.02760123461484909 (0.030194463580257343) | > loss_mel_ce: 2.647003412246704 (nan) | > loss: 0.010613511316478252 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.1371184429035913) | > loader_time: 0.0046 (0.00465933823887307)  --> TIME: 2025-02-27 01:00:26 -- STEP: 1630/36938 -- GLOBAL_STEP: 1479150 | > loss_text_ce: 0.03908952698111534 (0.030252828727264893) | > loss_mel_ce: 2.749605178833008 (nan) | > loss: 0.011066249571740627 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.137046247903555) | > loader_time: 0.0046 (0.004660065334998764)  --> TIME: 2025-02-27 01:00:35 -- STEP: 1680/36938 -- GLOBAL_STEP: 1479200 | > loss_text_ce: 0.0332021601498127 (0.030269944889005272) | > loss_mel_ce: 3.1094226837158203 (nan) | > loss: 0.012470734305679798 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13704895746140278) | > loader_time: 0.0047 (0.00465958189396632)  --> TIME: 2025-02-27 01:00:44 -- STEP: 1730/36938 -- GLOBAL_STEP: 1479250 | > loss_text_ce: 0.028802039101719856 (0.030257619825337145) | > loss_mel_ce: 2.9439573287963867 (nan) | > loss: 0.011796665377914906 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.1370862687943301) | > loader_time: 0.0046 (0.004658503615098201)  --> TIME: 2025-02-27 01:00:54 -- STEP: 1780/36938 -- GLOBAL_STEP: 1479300 | > loss_text_ce: 0.021098485216498375 (0.030261873788189855) | > loss_mel_ce: 1.7340105772018433 (nan) | > loss: 0.006964718922972679 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13701681179946762) | > loader_time: 0.0046 (0.004656443435154612)  --> TIME: 2025-02-27 01:01:03 -- STEP: 1830/36938 -- GLOBAL_STEP: 1479350 | > loss_text_ce: 0.03775130957365036 (0.030287483167106825) | > loss_mel_ce: 3.0097453594207764 (nan) | > loss: 0.012093241326510906 (nan) | > current_lr: 5e-06 | > step_time: 0.1864 (0.13711583809774466) | > loader_time: 0.0048 (0.0046600301409028344)  --> TIME: 2025-02-27 01:01:13 -- STEP: 1880/36938 -- GLOBAL_STEP: 1479400 | > loss_text_ce: 0.029274050146341324 (0.030279805601891526) | > loss_mel_ce: 0.7563187479972839 (nan) | > loss: 0.0031174318864941597 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.1370703006044351) | > loader_time: 0.0045 (0.0046588883755054955)  --> TIME: 2025-02-27 01:01:22 -- STEP: 1930/36938 -- GLOBAL_STEP: 1479450 | > loss_text_ce: 0.025170190259814262 (0.030254947915769766) | > loss_mel_ce: 2.7917933464050293 (nan) | > loss: 0.011178427375853062 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.1369870762750896) | > loader_time: 0.0047 (0.004657654194016538)  --> TIME: 2025-02-27 01:01:31 -- STEP: 1980/36938 -- GLOBAL_STEP: 1479500 | > loss_text_ce: 0.025626681745052338 (0.03023147828961638) | > loss_mel_ce: 2.1624186038970947 (nan) | > loss: 0.008682719431817532 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13681054428370346) | > loader_time: 0.0046 (0.0046576615535851755)  --> TIME: 2025-02-27 01:01:40 -- STEP: 2030/36938 -- GLOBAL_STEP: 1479550 | > loss_text_ce: 0.029145531356334686 (0.030208714021452397) | > loss_mel_ce: 2.8313801288604736 (nan) | > loss: 0.011351292952895164 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13669415800442283) | > loader_time: 0.0046 (0.004657779656020299)  --> TIME: 2025-02-27 01:01:49 -- STEP: 2080/36938 -- GLOBAL_STEP: 1479600 | > loss_text_ce: 0.030714983120560646 (0.030238368710986552) | > loss_mel_ce: 0.8528326153755188 (nan) | > loss: 0.0035061414819210768 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13666838155343014) | > loader_time: 0.0045 (0.004658517929223873)  --> TIME: 2025-02-27 01:01:58 -- STEP: 2130/36938 -- GLOBAL_STEP: 1479650 | > loss_text_ce: 0.02249806560575962 (0.030232696569668996) | > loss_mel_ce: 2.4972546100616455 (nan) | > loss: 0.009999019093811512 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.13663967887000897) | > loader_time: 0.0047 (0.004658908911154308)  --> TIME: 2025-02-27 01:02:08 -- STEP: 2180/36938 -- GLOBAL_STEP: 1479700 | > loss_text_ce: 0.021030230447649956 (0.030256965604755992) | > loss_mel_ce: 1.887099266052246 (nan) | > loss: 0.007571942638605833 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13661908808104523) | > loader_time: 0.0046 (0.004658906175455919)  --> TIME: 2025-02-27 01:02:17 -- STEP: 2230/36938 -- GLOBAL_STEP: 1479750 | > loss_text_ce: 0.02962537482380867 (0.030274062867238785) | > loss_mel_ce: 1.085301399230957 (nan) | > loss: 0.004424313083291054 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.136759867689535) | > loader_time: 0.0046 (0.004659654741329998)  --> TIME: 2025-02-27 01:02:27 -- STEP: 2280/36938 -- GLOBAL_STEP: 1479800 | > loss_text_ce: 0.031550176441669464 (0.03025439555201222) | > loss_mel_ce: 3.0005292892456055 (nan) | > loss: 0.01203206181526184 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13670308903643996) | > loader_time: 0.0047 (0.004659924486227207)  --> TIME: 2025-02-27 01:02:36 -- STEP: 2330/36938 -- GLOBAL_STEP: 1479850 | > loss_text_ce: 0.04016292467713356 (0.030254971083701298) | > loss_mel_ce: 3.115975856781006 (nan) | > loss: 0.012524361722171307 (nan) | > current_lr: 5e-06 | > step_time: 0.1513 (0.1367094904568066) | > loader_time: 0.0048 (0.0046605701610254215)  --> TIME: 2025-02-27 01:02:45 -- STEP: 2380/36938 -- GLOBAL_STEP: 1479900 | > loss_text_ce: 0.025749951601028442 (0.030250112426334193) | > loss_mel_ce: 2.6519994735717773 (nan) | > loss: 0.010625990107655525 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.13666300483110605) | > loader_time: 0.0046 (0.004659214640865817)  --> TIME: 2025-02-27 01:02:55 -- STEP: 2430/36938 -- GLOBAL_STEP: 1479950 | > loss_text_ce: 0.030666282400488853 (0.03025474498401024) | > loss_mel_ce: 2.8379499912261963 (nan) | > loss: 0.011383398436009884 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.13672820589669965) | > loader_time: 0.0046 (0.004657957877641854)  --> TIME: 2025-02-27 01:03:05 -- STEP: 2480/36938 -- GLOBAL_STEP: 1480000 | > loss_text_ce: 0.032673127949237823 (0.03027303555882686) | > loss_mel_ce: 2.0671966075897217 (nan) | > loss: 0.008332816883921623 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1368939586224098) | > loader_time: 0.0046 (0.0046575030011515575) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1480000.pth  --> TIME: 2025-02-27 01:03:20 -- STEP: 2530/36938 -- GLOBAL_STEP: 1480050 | > loss_text_ce: 0.04627487063407898 (0.030270895157966455) | > loss_mel_ce: 0.076416015625 (nan) | > loss: 0.0004868686373811215 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13679329604499457) | > loader_time: 0.0047 (0.004657738859003252)  --> TIME: 2025-02-27 01:03:29 -- STEP: 2580/36938 -- GLOBAL_STEP: 1480100 | > loss_text_ce: 0.023945502936840057 (0.030270978159365036) | > loss_mel_ce: 1.7544671297073364 (nan) | > loss: 0.007057193201035261 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13670609163683542) | > loader_time: 0.0045 (0.004657688528992418)  --> TIME: 2025-02-27 01:03:38 -- STEP: 2630/36938 -- GLOBAL_STEP: 1480150 | > loss_text_ce: 0.02870212309062481 (0.030263917434946438) | > loss_mel_ce: 2.515169620513916 (nan) | > loss: 0.010094729252159595 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.13665271186103428) | > loader_time: 0.0048 (0.004658309860374544)  --> TIME: 2025-02-27 01:03:47 -- STEP: 2680/36938 -- GLOBAL_STEP: 1480200 | > loss_text_ce: 0.023313013836741447 (0.030249436932212825) | > loss_mel_ce: 2.693005323410034 (nan) | > loss: 0.010779041796922684 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.136729960655099) | > loader_time: 0.005 (0.004659508235418983)  --> TIME: 2025-02-27 01:03:57 -- STEP: 2730/36938 -- GLOBAL_STEP: 1480250 | > loss_text_ce: 0.03119557537138462 (0.03024404317468077) | > loss_mel_ce: 2.728635549545288 (nan) | > loss: 0.010951711796224117 (nan) | > current_lr: 5e-06 | > step_time: 0.1882 (0.13670887099954265) | > loader_time: 0.0048 (0.0046600200317718265)  --> TIME: 2025-02-27 01:04:06 -- STEP: 2780/36938 -- GLOBAL_STEP: 1480300 | > loss_text_ce: 0.0221252478659153 (0.030245450613417213) | > loss_mel_ce: 0.008543332107365131 (nan) | > loss: 0.00012170071568107232 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13673261018108163) | > loader_time: 0.0045 (0.004659829722891614)  --> TIME: 2025-02-27 01:04:16 -- STEP: 2830/36938 -- GLOBAL_STEP: 1480350 | > loss_text_ce: 0.03965701907873154 (0.030261653303678804) | > loss_mel_ce: 1.1787337064743042 (nan) | > loss: 0.004834883846342564 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13676857586041818) | > loader_time: 0.0045 (0.004659968888380514)  --> TIME: 2025-02-27 01:04:25 -- STEP: 2880/36938 -- GLOBAL_STEP: 1480400 | > loss_text_ce: 0.027721988037228584 (0.030261128364104984) | > loss_mel_ce: 2.6079256534576416 (nan) | > loss: 0.01045891921967268 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13670018936197004) | > loader_time: 0.0048 (0.004660025156206565)  --> TIME: 2025-02-27 01:04:34 -- STEP: 2930/36938 -- GLOBAL_STEP: 1480450 | > loss_text_ce: 0.029501520097255707 (0.030251992366089147) | > loss_mel_ce: 2.7397515773773193 (nan) | > loss: 0.010989099740982056 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.136701366356209) | > loader_time: 0.0045 (0.004660854974297538)  --> TIME: 2025-02-27 01:04:43 -- STEP: 2980/36938 -- GLOBAL_STEP: 1480500 | > loss_text_ce: 0.039140183478593826 (0.030258535750635317) | > loss_mel_ce: 2.118558406829834 (nan) | > loss: 0.008562296628952026 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13668031988528007) | > loader_time: 0.0045 (0.004660766396746545)  --> TIME: 2025-02-27 01:04:53 -- STEP: 3030/36938 -- GLOBAL_STEP: 1480550 | > loss_text_ce: 0.028600890189409256 (0.030245002308128887) | > loss_mel_ce: 2.9071762561798096 (nan) | > loss: 0.011649910360574722 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13669015377661634) | > loader_time: 0.0046 (0.004660441615793969)  --> TIME: 2025-02-27 01:05:02 -- STEP: 3080/36938 -- GLOBAL_STEP: 1480600 | > loss_text_ce: 0.03489549458026886 (0.03025449307441905) | > loss_mel_ce: 2.5156257152557373 (nan) | > loss: 0.010121116414666176 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13669641103063376) | > loader_time: 0.0045 (0.004659687001983845)  --> TIME: 2025-02-27 01:05:11 -- STEP: 3130/36938 -- GLOBAL_STEP: 1480650 | > loss_text_ce: 0.034418486058712006 (0.030259391395880963) | > loss_mel_ce: 2.8013546466827393 (nan) | > loss: 0.011253069154918194 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13669899066035432) | > loader_time: 0.0046 (0.004659455119611363)  --> TIME: 2025-02-27 01:05:21 -- STEP: 3180/36938 -- GLOBAL_STEP: 1480700 | > loss_text_ce: 0.026127494871616364 (0.030252820851995883) | > loss_mel_ce: 0.7820938229560852 (nan) | > loss: 0.0032072276808321476 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13666121337398876) | > loader_time: 0.0047 (0.004659822302044569)  --> TIME: 2025-02-27 01:05:30 -- STEP: 3230/36938 -- GLOBAL_STEP: 1480750 | > loss_text_ce: 0.024077031761407852 (0.030246723367149248) | > loss_mel_ce: 2.1961047649383545 (nan) | > loss: 0.008810245431959629 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13676696037729547) | > loader_time: 0.0046 (0.004660516700508426)  --> TIME: 2025-02-27 01:05:40 -- STEP: 3280/36938 -- GLOBAL_STEP: 1480800 | > loss_text_ce: 0.026204755529761314 (0.0302516826150212) | > loss_mel_ce: 2.2529501914978027 (nan) | > loss: 0.009044266305863857 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.1368657902246577) | > loader_time: 0.0046 (0.004661739381348218)  --> TIME: 2025-02-27 01:05:49 -- STEP: 3330/36938 -- GLOBAL_STEP: 1480850 | > loss_text_ce: 0.026719938963651657 (0.030234398267178744) | > loss_mel_ce: 2.0969576835632324 (nan) | > loss: 0.008427293039858341 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13684583268724132) | > loader_time: 0.0047 (0.004662572299395953)  --> TIME: 2025-02-27 01:05:59 -- STEP: 3380/36938 -- GLOBAL_STEP: 1480900 | > loss_text_ce: 0.03432779386639595 (0.030221628000100073) | > loss_mel_ce: 2.461052179336548 (nan) | > loss: 0.00990230217576027 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.136839871011543) | > loader_time: 0.0046 (0.004662632236819294)  --> TIME: 2025-02-27 01:06:08 -- STEP: 3430/36938 -- GLOBAL_STEP: 1480950 | > loss_text_ce: 0.03203747794032097 (0.030231539419860343) | > loss_mel_ce: 3.0542125701904297 (nan) | > loss: 0.012247025035321712 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.1368899529598901) | > loader_time: 0.0046 (0.00466243101626026)  --> TIME: 2025-02-27 01:06:18 -- STEP: 3480/36938 -- GLOBAL_STEP: 1481000 | > loss_text_ce: 0.024524500593543053 (0.030251898716375146) | > loss_mel_ce: 1.6137237548828125 (nan) | > loss: 0.006500985473394394 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13698276815743365) | > loader_time: 0.0045 (0.0046625463441870635)  --> TIME: 2025-02-27 01:06:27 -- STEP: 3530/36938 -- GLOBAL_STEP: 1481050 | > loss_text_ce: 0.03106711618602276 (0.03027395164045286) | > loss_mel_ce: 2.8056650161743164 (nan) | > loss: 0.011256874538958073 (nan) | > current_lr: 5e-06 | > step_time: 0.156 (0.13696322616706905) | > loader_time: 0.0047 (0.004661876891895323)  --> TIME: 2025-02-27 01:06:36 -- STEP: 3580/36938 -- GLOBAL_STEP: 1481100 | > loss_text_ce: 0.03274301812052727 (0.030277476764892817) | > loss_mel_ce: 2.8301265239715576 (nan) | > loss: 0.01136059407144785 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13698758346408524) | > loader_time: 0.0045 (0.0046612931363409415)  --> TIME: 2025-02-27 01:06:46 -- STEP: 3630/36938 -- GLOBAL_STEP: 1481150 | > loss_text_ce: 0.03448927029967308 (0.03028055228595686) | > loss_mel_ce: 2.6300456523895264 (nan) | > loss: 0.010573551058769226 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.13696079910950956) | > loader_time: 0.0048 (0.004660696234584836)  --> TIME: 2025-02-27 01:06:55 -- STEP: 3680/36938 -- GLOBAL_STEP: 1481200 | > loss_text_ce: 0.029633518308401108 (0.030293602719346224) | > loss_mel_ce: 2.038943290710449 (nan) | > loss: 0.008208638988435268 (nan) | > current_lr: 5e-06 | > step_time: 0.1082 (0.1369800000734958) | > loader_time: 0.0045 (0.004660229838412737)  --> TIME: 2025-02-27 01:07:05 -- STEP: 3730/36938 -- GLOBAL_STEP: 1481250 | > loss_text_ce: 0.036630358546972275 (0.030301807750635668) | > loss_mel_ce: 2.147104501724243 (nan) | > loss: 0.008665614761412144 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13706445502212222) | > loader_time: 0.0045 (0.0046605762783387995)  --> TIME: 2025-02-27 01:07:14 -- STEP: 3780/36938 -- GLOBAL_STEP: 1481300 | > loss_text_ce: 0.028401080518960953 (0.03031542426561592) | > loss_mel_ce: 3.074946880340576 (nan) | > loss: 0.0123148737475276 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13700357366491314) | > loader_time: 0.0046 (0.00466128688640695)  --> TIME: 2025-02-27 01:07:23 -- STEP: 3830/36938 -- GLOBAL_STEP: 1481350 | > loss_text_ce: 0.034255288541316986 (0.030324495439575716) | > loss_mel_ce: 2.8263602256774902 (nan) | > loss: 0.011351649649441242 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13699936773384896) | > loader_time: 0.0045 (0.0046608077639388006)  --> TIME: 2025-02-27 01:07:33 -- STEP: 3880/36938 -- GLOBAL_STEP: 1481400 | > loss_text_ce: 0.024319583550095558 (0.030314222245342724) | > loss_mel_ce: 2.751378059387207 (nan) | > loss: 0.011014673858880997 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13700462776361116) | > loader_time: 0.0047 (0.0046605025370096385)  --> TIME: 2025-02-27 01:07:42 -- STEP: 3930/36938 -- GLOBAL_STEP: 1481450 | > loss_text_ce: 0.0258073378354311 (0.0302970859989922) | > loss_mel_ce: 2.5006558895111084 (nan) | > loss: 0.010025648400187492 (nan) | > current_lr: 5e-06 | > step_time: 0.1574 (0.13701318371993654) | > loader_time: 0.0047 (0.004660377429641836)  --> TIME: 2025-02-27 01:07:52 -- STEP: 3980/36938 -- GLOBAL_STEP: 1481500 | > loss_text_ce: 0.028117500245571136 (0.030289251837018298) | > loss_mel_ce: 2.5653815269470215 (nan) | > loss: 0.010291662998497486 (nan) | > current_lr: 5e-06 | > step_time: 0.1867 (0.1370757527686849) | > loader_time: 0.0071 (0.00466075979884545)  --> TIME: 2025-02-27 01:08:01 -- STEP: 4030/36938 -- GLOBAL_STEP: 1481550 | > loss_text_ce: 0.035049811005592346 (0.030282225671499303) | > loss_mel_ce: 2.857048273086548 (nan) | > loss: 0.011476580984890461 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13711883146177217) | > loader_time: 0.0046 (0.0046617458830991955)  --> TIME: 2025-02-27 01:08:10 -- STEP: 4080/36938 -- GLOBAL_STEP: 1481600 | > loss_text_ce: 0.03590264171361923 (0.030293679475218202) | > loss_mel_ce: 2.409153699874878 (nan) | > loss: 0.009702605195343494 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13702902536766262) | > loader_time: 0.0045 (0.00466231429109386)  --> TIME: 2025-02-27 01:08:20 -- STEP: 4130/36938 -- GLOBAL_STEP: 1481650 | > loss_text_ce: 0.028131339699029922 (0.03028762305953673) | > loss_mel_ce: 2.541621208190918 (nan) | > loss: 0.010197430849075317 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13711784278509465) | > loader_time: 0.0046 (0.004662490756979286)  --> TIME: 2025-02-27 01:08:29 -- STEP: 4180/36938 -- GLOBAL_STEP: 1481700 | > loss_text_ce: 0.03142319247126579 (0.03029963228557455) | > loss_mel_ce: 1.5887356996536255 (nan) | > loss: 0.006429202388972044 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.1371144961512266) | > loader_time: 0.0045 (0.004662050984122532)  --> TIME: 2025-02-27 01:08:39 -- STEP: 4230/36938 -- GLOBAL_STEP: 1481750 | > loss_text_ce: 0.028813043609261513 (0.030303805128145015) | > loss_mel_ce: 0.06689453125 (nan) | > loss: 0.00037979197804816067 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13711497124205302) | > loader_time: 0.0044 (0.004662012884802849)  --> TIME: 2025-02-27 01:08:48 -- STEP: 4280/36938 -- GLOBAL_STEP: 1481800 | > loss_text_ce: 0.025677930563688278 (0.030291163344501954) | > loss_mel_ce: 2.7306149005889893 (nan) | > loss: 0.01093767024576664 (nan) | > current_lr: 5e-06 | > step_time: 0.1713 (0.13719164840528963) | > loader_time: 0.0048 (0.004663178686783691)  --> TIME: 2025-02-27 01:08:58 -- STEP: 4330/36938 -- GLOBAL_STEP: 1481850 | > loss_text_ce: 0.03367418795824051 (0.030289346910152774) | > loss_mel_ce: 2.2991225719451904 (nan) | > loss: 0.009257130324840546 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13718939813125006) | > loader_time: 0.0046 (0.004662571272728897)  --> TIME: 2025-02-27 01:09:07 -- STEP: 4380/36938 -- GLOBAL_STEP: 1481900 | > loss_text_ce: 0.02983367070555687 (0.0303030148497251) | > loss_mel_ce: 2.092007637023926 (nan) | > loss: 0.008420005440711975 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13721942155872915) | > loader_time: 0.0046 (0.004662834291588769)  --> TIME: 2025-02-27 01:09:17 -- STEP: 4430/36938 -- GLOBAL_STEP: 1481950 | > loss_text_ce: 0.03260110318660736 (0.030301504070464825) | > loss_mel_ce: 3.187610626220703 (nan) | > loss: 0.01277861837297678 (nan) | > current_lr: 5e-06 | > step_time: 0.1956 (0.13725396451239733) | > loader_time: 0.0048 (0.004662985026701842)  --> TIME: 2025-02-27 01:09:26 -- STEP: 4480/36938 -- GLOBAL_STEP: 1482000 | > loss_text_ce: 0.030177410691976547 (0.03030080059169059) | > loss_mel_ce: 2.6250104904174805 (nan) | > loss: 0.01053646020591259 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13723820224404398) | > loader_time: 0.0046 (0.004662774130702013)  --> TIME: 2025-02-27 01:09:35 -- STEP: 4530/36938 -- GLOBAL_STEP: 1482050 | > loss_text_ce: 0.025329776108264923 (0.030295188040380044) | > loss_mel_ce: 2.373279094696045 (nan) | > loss: 0.009518289938569069 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13725036758987125) | > loader_time: 0.0049 (0.004664176873550241)  --> TIME: 2025-02-27 01:09:45 -- STEP: 4580/36938 -- GLOBAL_STEP: 1482100 | > loss_text_ce: 0.03617669269442558 (0.030314111330558634) | > loss_mel_ce: 2.4737579822540283 (nan) | > loss: 0.009960059076547623 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.1372578874425603) | > loader_time: 0.0046 (0.004664250388416135)  --> TIME: 2025-02-27 01:09:54 -- STEP: 4630/36938 -- GLOBAL_STEP: 1482150 | > loss_text_ce: 0.027604181319475174 (0.03030803507801394) | > loss_mel_ce: 2.7147223949432373 (nan) | > loss: 0.010882249101996422 (nan) | > current_lr: 5e-06 | > step_time: 0.1496 (0.13726701983622863) | > loader_time: 0.0046 (0.004664442709146249)  --> TIME: 2025-02-27 01:10:04 -- STEP: 4680/36938 -- GLOBAL_STEP: 1482200 | > loss_text_ce: 0.025077244266867638 (0.030297716534258713) | > loss_mel_ce: 2.417717695236206 (nan) | > loss: 0.009693631902337074 (nan) | > current_lr: 5e-06 | > step_time: 0.1861 (0.13727745821333398) | > loader_time: 0.0049 (0.004664838008391546)  --> TIME: 2025-02-27 01:10:13 -- STEP: 4730/36938 -- GLOBAL_STEP: 1482250 | > loss_text_ce: 0.027876071631908417 (0.030293098159417914) | > loss_mel_ce: 2.2279317378997803 (nan) | > loss: 0.008951619267463684 (nan) | > current_lr: 5e-06 | > step_time: 0.1903 (0.13726069342510447) | > loader_time: 0.0048 (0.004665039054435095)  --> TIME: 2025-02-27 01:10:22 -- STEP: 4780/36938 -- GLOBAL_STEP: 1482300 | > loss_text_ce: 0.027313003316521645 (0.030305394356501397) | > loss_mel_ce: 2.539210796356201 (nan) | > loss: 0.010184618644416332 (nan) | > current_lr: 5e-06 | > step_time: 0.1531 (0.13725117938787926) | > loader_time: 0.0046 (0.004665148208330861)  --> TIME: 2025-02-27 01:10:32 -- STEP: 4830/36938 -- GLOBAL_STEP: 1482350 | > loss_text_ce: 0.027439096942543983 (0.03030408401963904) | > loss_mel_ce: 2.3548033237457275 (nan) | > loss: 0.009453343227505684 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.13724153579885728) | > loader_time: 0.0048 (0.0046650921089061775)  --> TIME: 2025-02-27 01:10:41 -- STEP: 4880/36938 -- GLOBAL_STEP: 1482400 | > loss_text_ce: 0.04457100108265877 (0.030315069934605143) | > loss_mel_ce: 0.8558082580566406 (nan) | > loss: 0.0035729336086660624 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13721491567424146) | > loader_time: 0.0046 (0.004665763260888269)  --> TIME: 2025-02-27 01:10:50 -- STEP: 4930/36938 -- GLOBAL_STEP: 1482450 | > loss_text_ce: 0.0264892578125 (0.030314805350903015) | > loss_mel_ce: 2.9200236797332764 (nan) | > loss: 0.011692512780427933 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.1372219642325789) | > loader_time: 0.0048 (0.004666452185377387)  --> TIME: 2025-02-27 01:10:59 -- STEP: 4980/36938 -- GLOBAL_STEP: 1482500 | > loss_text_ce: 0.01917833834886551 (0.03030876921205575) | > loss_mel_ce: 0.0055821738205850124 (nan) | > loss: 9.825600864132866e-05 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13722257451360076) | > loader_time: 0.0045 (0.004666104374161681)  --> TIME: 2025-02-27 01:11:09 -- STEP: 5030/36938 -- GLOBAL_STEP: 1482550 | > loss_text_ce: 0.03771837800741196 (0.03031919020178096) | > loss_mel_ce: 3.0566999912261963 (nan) | > loss: 0.01227943878620863 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13722635528913382) | > loader_time: 0.0045 (0.004665880554002029)  --> TIME: 2025-02-27 01:11:18 -- STEP: 5080/36938 -- GLOBAL_STEP: 1482600 | > loss_text_ce: 0.031042758375406265 (0.030309717182627337) | > loss_mel_ce: 2.891289710998535 (nan) | > loss: 0.0115965586155653 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.1372060330364654) | > loader_time: 0.0046 (0.00466573688927597)  --> TIME: 2025-02-27 01:11:28 -- STEP: 5130/36938 -- GLOBAL_STEP: 1482650 | > loss_text_ce: 0.027295252308249474 (0.030319287819283162) | > loss_mel_ce: 0.799774169921875 (nan) | > loss: 0.0032820217311382294 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13719837321640277) | > loader_time: 0.0047 (0.004665791686282975)  --> TIME: 2025-02-27 01:11:37 -- STEP: 5180/36938 -- GLOBAL_STEP: 1482700 | > loss_text_ce: 0.03007505275309086 (0.03031579668212926) | > loss_mel_ce: 3.008784770965576 (nan) | > loss: 0.012058968655765057 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13716732825091515) | > loader_time: 0.0048 (0.0046662222924839895)  --> TIME: 2025-02-27 01:11:46 -- STEP: 5230/36938 -- GLOBAL_STEP: 1482750 | > loss_text_ce: 0.028759591281414032 (0.030313454748239953) | > loss_mel_ce: 2.9113969802856445 (nan) | > loss: 0.011667287908494473 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13718303872795967) | > loader_time: 0.0046 (0.004666940751084844)  --> TIME: 2025-02-27 01:11:55 -- STEP: 5280/36938 -- GLOBAL_STEP: 1482800 | > loss_text_ce: 0.025612924247980118 (0.030317157910163085) | > loss_mel_ce: 2.332907199859619 (nan) | > loss: 0.0093592070043087 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13715983868548415) | > loader_time: 0.0048 (0.004666519571434349)  --> TIME: 2025-02-27 01:12:05 -- STEP: 5330/36938 -- GLOBAL_STEP: 1482850 | > loss_text_ce: 0.025812337175011635 (0.030314504132758102) | > loss_mel_ce: 0.0013756752014160156 (nan) | > loss: 0.00010788894724100828 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13713581401009106) | > loader_time: 0.0045 (0.004666184842251017)  --> TIME: 2025-02-27 01:12:14 -- STEP: 5380/36938 -- GLOBAL_STEP: 1482900 | > loss_text_ce: 0.04273693636059761 (0.03033694991873071) | > loss_mel_ce: 2.36391544342041 (nan) | > loss: 0.00955020822584629 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13711140550202205) | > loader_time: 0.0046 (0.004665894800845562)  --> TIME: 2025-02-27 01:12:23 -- STEP: 5430/36938 -- GLOBAL_STEP: 1482950 | > loss_text_ce: 0.028553178533911705 (0.030337845901984368) | > loss_mel_ce: 2.8777072429656982 (nan) | > loss: 0.011532780714333057 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13709440758215177) | > loader_time: 0.0046 (0.00466589703744286)  --> TIME: 2025-02-27 01:12:32 -- STEP: 5480/36938 -- GLOBAL_STEP: 1483000 | > loss_text_ce: 0.02770264819264412 (0.03034176722654977) | > loss_mel_ce: 1.9271602630615234 (nan) | > loss: 0.007757393177598715 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.1370872947421392) | > loader_time: 0.0048 (0.0046672888915904095)  --> TIME: 2025-02-27 01:12:42 -- STEP: 5530/36938 -- GLOBAL_STEP: 1483050 | > loss_text_ce: 0.034850407391786575 (0.0303429606808501) | > loss_mel_ce: 3.004321813583374 (nan) | > loss: 0.012060207314789295 (nan) | > current_lr: 5e-06 | > step_time: 0.1691 (0.13710805770501572) | > loader_time: 0.0049 (0.004667669394563796)  --> TIME: 2025-02-27 01:12:52 -- STEP: 5580/36938 -- GLOBAL_STEP: 1483100 | > loss_text_ce: 0.028782248497009277 (0.030349805806412968) | > loss_mel_ce: 2.9035747051239014 (nan) | > loss: 0.011636337265372276 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.13717699841359243) | > loader_time: 0.0049 (0.004668581656657652)  --> TIME: 2025-02-27 01:13:01 -- STEP: 5630/36938 -- GLOBAL_STEP: 1483150 | > loss_text_ce: 0.0253928042948246 (0.030352116892772653) | > loss_mel_ce: 2.863168478012085 (nan) | > loss: 0.011462545022368431 (nan) | > current_lr: 5e-06 | > step_time: 0.1723 (0.13716296476326928) | > loader_time: 0.0048 (0.004668411649565064)  --> TIME: 2025-02-27 01:13:10 -- STEP: 5680/36938 -- GLOBAL_STEP: 1483200 | > loss_text_ce: 0.030745897442102432 (0.030352413444943774) | > loss_mel_ce: 1.6283975839614868 (nan) | > loss: 0.006583902984857559 (nan) | > current_lr: 5e-06 | > step_time: 0.1457 (0.13714289270656277) | > loader_time: 0.0047 (0.004669258376242394)  --> TIME: 2025-02-27 01:13:20 -- STEP: 5730/36938 -- GLOBAL_STEP: 1483250 | > loss_text_ce: 0.027094393968582153 (0.030354097602729654) | > loss_mel_ce: 2.3103950023651123 (nan) | > loss: 0.009275752119719982 (nan) | > current_lr: 5e-06 | > step_time: 0.1118 (0.13716853563698247) | > loader_time: 0.0045 (0.004669785457963913)  --> TIME: 2025-02-27 01:13:29 -- STEP: 5780/36938 -- GLOBAL_STEP: 1483300 | > loss_text_ce: 0.03468215465545654 (0.030363354188870234) | > loss_mel_ce: 2.135131597518921 (nan) | > loss: 0.008610372431576252 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.1371325500283695) | > loader_time: 0.0045 (0.004669658081754259)  --> TIME: 2025-02-27 01:13:38 -- STEP: 5830/36938 -- GLOBAL_STEP: 1483350 | > loss_text_ce: 0.03528613969683647 (0.030368173760274102) | > loss_mel_ce: 2.608151912689209 (nan) | > loss: 0.01048983447253704 (nan) | > current_lr: 5e-06 | > step_time: 0.1667 (0.1370819065624002) | > loader_time: 0.0048 (0.004669205484014011)  --> TIME: 2025-02-27 01:13:47 -- STEP: 5880/36938 -- GLOBAL_STEP: 1483400 | > loss_text_ce: 0.038040757179260254 (0.03036859036996631) | > loss_mel_ce: 2.7038943767547607 (nan) | > loss: 0.010880695655941963 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13709696535350585) | > loader_time: 0.0048 (0.004669218079573431)  --> TIME: 2025-02-27 01:13:57 -- STEP: 5930/36938 -- GLOBAL_STEP: 1483450 | > loss_text_ce: 0.02384195104241371 (0.030359699496839185) | > loss_mel_ce: 1.842923641204834 (nan) | > loss: 0.007407800294458866 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13710527030050726) | > loader_time: 0.0046 (0.004669316623102343)  --> TIME: 2025-02-27 01:14:06 -- STEP: 5980/36938 -- GLOBAL_STEP: 1483500 | > loss_text_ce: 0.029041236266493797 (0.030370215409868284) | > loss_mel_ce: 2.6702001094818115 (nan) | > loss: 0.010711275972425938 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13706321361471643) | > loader_time: 0.0047 (0.004669482612290901)  --> TIME: 2025-02-27 01:14:15 -- STEP: 6030/36938 -- GLOBAL_STEP: 1483550 | > loss_text_ce: 0.01788128726184368 (0.030365657518659505) | > loss_mel_ce: 1.6476091146469116 (nan) | > loss: 0.006609089206904173 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13703985190510226) | > loader_time: 0.0046 (0.004670104142247535)  --> TIME: 2025-02-27 01:14:24 -- STEP: 6080/36938 -- GLOBAL_STEP: 1483600 | > loss_text_ce: 0.032383549958467484 (0.030371475016765636) | > loss_mel_ce: 2.551032543182373 (nan) | > loss: 0.010251651518046856 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.1370225010341727) | > loader_time: 0.0046 (0.004670358723715725)  --> TIME: 2025-02-27 01:14:33 -- STEP: 6130/36938 -- GLOBAL_STEP: 1483650 | > loss_text_ce: 0.02486918680369854 (0.030355436466087325) | > loss_mel_ce: 2.995054006576538 (nan) | > loss: 0.011983823031187057 (nan) | > current_lr: 5e-06 | > step_time: 0.1724 (0.13702126867814016) | > loader_time: 0.0047 (0.004670108862063229)  --> TIME: 2025-02-27 01:14:42 -- STEP: 6180/36938 -- GLOBAL_STEP: 1483700 | > loss_text_ce: 0.03525882586836815 (0.03035681681077577) | > loss_mel_ce: 3.0555245876312256 (nan) | > loss: 0.012265014462172985 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1369636485877555) | > loader_time: 0.0045 (0.0046696974621621475)  --> TIME: 2025-02-27 01:14:52 -- STEP: 6230/36938 -- GLOBAL_STEP: 1483750 | > loss_text_ce: 0.03501946106553078 (0.030348119170167276) | > loss_mel_ce: 2.5070931911468506 (nan) | > loss: 0.010087748989462852 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.13698224194742578) | > loader_time: 0.0046 (0.004669563996083849)  --> TIME: 2025-02-27 01:15:01 -- STEP: 6280/36938 -- GLOBAL_STEP: 1483800 | > loss_text_ce: 0.035546354949474335 (0.030347812613828257) | > loss_mel_ce: 3.156470537185669 (nan) | > loss: 0.01266673393547535 (nan) | > current_lr: 5e-06 | > step_time: 0.2001 (0.13696558771619438) | > loader_time: 0.0048 (0.004669181746282388)  --> TIME: 2025-02-27 01:15:10 -- STEP: 6330/36938 -- GLOBAL_STEP: 1483850 | > loss_text_ce: 0.027895888313651085 (0.030348265386152696) | > loss_mel_ce: 2.7461187839508057 (nan) | > loss: 0.011007995344698429 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13694641631549817) | > loader_time: 0.0045 (0.0046688908460957475)  --> TIME: 2025-02-27 01:15:20 -- STEP: 6380/36938 -- GLOBAL_STEP: 1483900 | > loss_text_ce: 0.024667421355843544 (0.030340616584705844) | > loss_mel_ce: 1.6063638925552368 (nan) | > loss: 0.006472346838563681 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13693409918991434) | > loader_time: 0.0045 (0.004668703535133761)  --> TIME: 2025-02-27 01:15:29 -- STEP: 6430/36938 -- GLOBAL_STEP: 1483950 | > loss_text_ce: 0.02810259908437729 (0.03033616228005957) | > loss_mel_ce: 2.933591365814209 (nan) | > loss: 0.01175275444984436 (nan) | > current_lr: 5e-06 | > step_time: 0.1516 (0.13697218494504343) | > loader_time: 0.0046 (0.004668766046942293)  --> TIME: 2025-02-27 01:15:39 -- STEP: 6480/36938 -- GLOBAL_STEP: 1484000 | > loss_text_ce: 0.03015688806772232 (0.03032687343139615) | > loss_mel_ce: 2.7775325775146484 (nan) | > loss: 0.011141625232994556 (nan) | > current_lr: 5e-06 | > step_time: 0.184 (0.1370127510141449) | > loader_time: 0.0049 (0.004668742933391046)  --> TIME: 2025-02-27 01:15:48 -- STEP: 6530/36938 -- GLOBAL_STEP: 1484050 | > loss_text_ce: 0.025146815925836563 (0.03032819772164863) | > loss_mel_ce: 2.597841262817383 (nan) | > loss: 0.010408682748675346 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.13701798499634635) | > loader_time: 0.0048 (0.004668487962135673)  --> TIME: 2025-02-27 01:15:57 -- STEP: 6580/36938 -- GLOBAL_STEP: 1484100 | > loss_text_ce: 0.0313529334962368 (0.030329086039906028) | > loss_mel_ce: 3.110621213912964 (nan) | > loss: 0.012468152679502964 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.1369944778256872) | > loader_time: 0.0048 (0.004668395026476545)  --> TIME: 2025-02-27 01:16:07 -- STEP: 6630/36938 -- GLOBAL_STEP: 1484150 | > loss_text_ce: 0.0332428477704525 (0.030348865751620956) | > loss_mel_ce: 1.9391378164291382 (nan) | > loss: 0.007826907560229301 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13698407848496305) | > loader_time: 0.0045 (0.004668104810412645)  --> TIME: 2025-02-27 01:16:16 -- STEP: 6680/36938 -- GLOBAL_STEP: 1484200 | > loss_text_ce: 0.02600936032831669 (0.030347428152973418) | > loss_mel_ce: 2.211676836013794 (nan) | > loss: 0.00887970719486475 (nan) | > current_lr: 5e-06 | > step_time: 0.1813 (0.13695656432363154) | > loader_time: 0.005 (0.00466766489480069)  --> TIME: 2025-02-27 01:16:25 -- STEP: 6730/36938 -- GLOBAL_STEP: 1484250 | > loss_text_ce: 0.040400922298431396 (0.03032763209069399) | > loss_mel_ce: 3.055272340774536 (nan) | > loss: 0.012284418568015099 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13694087839055807) | > loader_time: 0.0045 (0.0046670975692204774)  --> TIME: 2025-02-27 01:16:34 -- STEP: 6780/36938 -- GLOBAL_STEP: 1484300 | > loss_text_ce: 0.02823454514145851 (0.030327714072089847) | > loss_mel_ce: 2.464860439300537 (nan) | > loss: 0.009893234819173813 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13693422036888409) | > loader_time: 0.0045 (0.004666566356445122)  --> TIME: 2025-02-27 01:16:43 -- STEP: 6830/36938 -- GLOBAL_STEP: 1484350 | > loss_text_ce: 0.03703318163752556 (0.03032861016158936) | > loss_mel_ce: 3.085125207901001 (nan) | > loss: 0.01238951738923788 (nan) | > current_lr: 5e-06 | > step_time: 0.1796 (0.13692870195927825) | > loader_time: 0.0049 (0.004666057547573452)  --> TIME: 2025-02-27 01:16:53 -- STEP: 6880/36938 -- GLOBAL_STEP: 1484400 | > loss_text_ce: 0.02817601151764393 (0.030331271057704806) | > loss_mel_ce: 3.188854217529297 (nan) | > loss: 0.012765994295477867 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13691289279350072) | > loader_time: 0.0048 (0.004665626516175814)  --> TIME: 2025-02-27 01:17:02 -- STEP: 6930/36938 -- GLOBAL_STEP: 1484450 | > loss_text_ce: 0.028860662132501602 (0.030334380172155416) | > loss_mel_ce: 2.875072956085205 (nan) | > loss: 0.011523546651005745 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13690588140556534) | > loader_time: 0.0046 (0.00466531551245486)  --> TIME: 2025-02-27 01:17:12 -- STEP: 6980/36938 -- GLOBAL_STEP: 1484500 | > loss_text_ce: 0.026074539870023727 (0.03034356933682196) | > loss_mel_ce: 1.6613456010818481 (nan) | > loss: 0.006696111988276243 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13696313075144914) | > loader_time: 0.0045 (0.004665230304258932)  --> TIME: 2025-02-27 01:17:21 -- STEP: 7030/36938 -- GLOBAL_STEP: 1484550 | > loss_text_ce: 0.026089370250701904 (0.03034006393001882) | > loss_mel_ce: 2.5657854080200195 (nan) | > loss: 0.010285218246281147 (nan) | > current_lr: 5e-06 | > step_time: 0.1778 (0.13697748896407594) | > loader_time: 0.0048 (0.004664990539062403)  --> TIME: 2025-02-27 01:17:31 -- STEP: 7080/36938 -- GLOBAL_STEP: 1484600 | > loss_text_ce: 0.027643658220767975 (0.030342310377775482) | > loss_mel_ce: 2.2741239070892334 (nan) | > loss: 0.009133999235928059 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.13698105879422687) | > loader_time: 0.0049 (0.00466509030363653)  --> TIME: 2025-02-27 01:17:40 -- STEP: 7130/36938 -- GLOBAL_STEP: 1484650 | > loss_text_ce: 0.03656822070479393 (0.030344994151821097) | > loss_mel_ce: 2.735671281814575 (nan) | > loss: 0.011000950820744038 (nan) | > current_lr: 5e-06 | > step_time: 0.1941 (0.13693158850596351) | > loader_time: 0.0049 (0.004664889726197392)  --> TIME: 2025-02-27 01:17:49 -- STEP: 7180/36938 -- GLOBAL_STEP: 1484700 | > loss_text_ce: 0.030175169929862022 (0.030339402706338954) | > loss_mel_ce: 2.6847989559173584 (nan) | > loss: 0.010773707181215286 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13697154468812717) | > loader_time: 0.0046 (0.004665304492110983)  --> TIME: 2025-02-27 01:17:59 -- STEP: 7230/36938 -- GLOBAL_STEP: 1484750 | > loss_text_ce: 0.02932717651128769 (0.030336699892160056) | > loss_mel_ce: 2.040024995803833 (nan) | > loss: 0.008211715146899223 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13700340450387177) | > loader_time: 0.0046 (0.004665306626843539)  --> TIME: 2025-02-27 01:18:08 -- STEP: 7280/36938 -- GLOBAL_STEP: 1484800 | > loss_text_ce: 0.02760889381170273 (0.0303311317308376) | > loss_mel_ce: 2.337310791015625 (nan) | > loss: 0.009384602308273315 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13698651050473368) | > loader_time: 0.0045 (0.004665020828718658)  --> TIME: 2025-02-27 01:18:18 -- STEP: 7330/36938 -- GLOBAL_STEP: 1484850 | > loss_text_ce: 0.029454471543431282 (0.030319349688959354) | > loss_mel_ce: 2.472163438796997 (nan) | > loss: 0.009927055798470974 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13705338068906223) | > loader_time: 0.0046 (0.004665182752466127)  --> TIME: 2025-02-27 01:18:28 -- STEP: 7380/36938 -- GLOBAL_STEP: 1484900 | > loss_text_ce: 0.03530782461166382 (0.030314332757493463) | > loss_mel_ce: 3.1355652809143066 (nan) | > loss: 0.012582831084728241 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13707128747045882) | > loader_time: 0.0046 (0.0046653293981784635)  --> TIME: 2025-02-27 01:18:37 -- STEP: 7430/36938 -- GLOBAL_STEP: 1484950 | > loss_text_ce: 0.032905593514442444 (0.030313938404251672) | > loss_mel_ce: 1.9361094236373901 (nan) | > loss: 0.007813552394509315 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13713875891863953) | > loader_time: 0.0045 (0.004665491783121543)  --> TIME: 2025-02-27 01:18:47 -- STEP: 7480/36938 -- GLOBAL_STEP: 1485000 | > loss_text_ce: 0.04094281420111656 (0.030322124474223718) | > loss_mel_ce: 1.5202549695968628 (nan) | > loss: 0.0061952294781804085 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13713863705568827) | > loader_time: 0.0046 (0.00466530071222845)  --> TIME: 2025-02-27 01:18:56 -- STEP: 7530/36938 -- GLOBAL_STEP: 1485050 | > loss_text_ce: 0.030433829873800278 (0.03032602181282156) | > loss_mel_ce: 3.1179616451263428 (nan) | > loss: 0.012493633665144444 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.13711162120064313) | > loader_time: 0.0047 (0.004664825189952664)  --> TIME: 2025-02-27 01:19:06 -- STEP: 7580/36938 -- GLOBAL_STEP: 1485100 | > loss_text_ce: 0.027346083894371986 (0.030317873273433077) | > loss_mel_ce: 2.7825121879577637 (nan) | > loss: 0.011150232516229153 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13715146119487687) | > loader_time: 0.0045 (0.004664949974472721)  --> TIME: 2025-02-27 01:19:15 -- STEP: 7630/36938 -- GLOBAL_STEP: 1485150 | > loss_text_ce: 0.028096983209252357 (0.030314200196835072) | > loss_mel_ce: 3.018301486968994 (nan) | > loss: 0.01208888366818428 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.13716910747838104) | > loader_time: 0.0046 (0.0046648085507934)  --> TIME: 2025-02-27 01:19:25 -- STEP: 7680/36938 -- GLOBAL_STEP: 1485200 | > loss_text_ce: 0.036352191120386124 (0.03032069384074323) | > loss_mel_ce: 3.1688921451568604 (nan) | > loss: 0.012719224207103252 (nan) | > current_lr: 5e-06 | > step_time: 0.1692 (0.13717974399526967) | > loader_time: 0.0048 (0.004664674525459598)  --> TIME: 2025-02-27 01:19:34 -- STEP: 7730/36938 -- GLOBAL_STEP: 1485250 | > loss_text_ce: 0.03608623892068863 (0.030321979860173966) | > loss_mel_ce: 2.8141965866088867 (nan) | > loss: 0.011310647241771221 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13717445870728934) | > loader_time: 0.0045 (0.0046643991519686075)  --> TIME: 2025-02-27 01:19:43 -- STEP: 7780/36938 -- GLOBAL_STEP: 1485300 | > loss_text_ce: 0.03421778976917267 (0.03032597367661337) | > loss_mel_ce: 1.864044189453125 (nan) | > loss: 0.0075327861122787 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13721169714449663) | > loader_time: 0.0046 (0.004664616382520416)  --> TIME: 2025-02-27 01:19:53 -- STEP: 7830/36938 -- GLOBAL_STEP: 1485350 | > loss_text_ce: 0.029343996196985245 (0.030331904230946206) | > loss_mel_ce: 3.1535019874572754 (nan) | > loss: 0.01263034250587225 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.13723554924834633) | > loader_time: 0.0048 (0.004664694151506866)  --> TIME: 2025-02-27 01:20:03 -- STEP: 7880/36938 -- GLOBAL_STEP: 1485400 | > loss_text_ce: 0.027109066024422646 (0.03033138127432578) | > loss_mel_ce: 2.583930015563965 (nan) | > loss: 0.010361267253756523 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13725200455806028) | > loader_time: 0.0046 (0.004664980457518903)  --> TIME: 2025-02-27 01:20:12 -- STEP: 7930/36938 -- GLOBAL_STEP: 1485450 | > loss_text_ce: 0.030720366165041924 (0.0303330311261606) | > loss_mel_ce: 2.5408177375793457 (nan) | > loss: 0.010204517282545567 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13724123571739874) | > loader_time: 0.0047 (0.004664910455970707)  --> TIME: 2025-02-27 01:20:21 -- STEP: 7980/36938 -- GLOBAL_STEP: 1485500 | > loss_text_ce: 0.027756866067647934 (0.03034117257439608) | > loss_mel_ce: 2.8235294818878174 (nan) | > loss: 0.011314629577100277 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13723588443937806) | > loader_time: 0.0046 (0.00466487503290773)  --> TIME: 2025-02-27 01:20:31 -- STEP: 8030/36938 -- GLOBAL_STEP: 1485550 | > loss_text_ce: 0.03161303326487541 (0.030344995651811656) | > loss_mel_ce: 3.1476829051971436 (nan) | > loss: 0.012616254389286041 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13724447220081462) | > loader_time: 0.0046 (0.004664767397147772)  --> TIME: 2025-02-27 01:20:40 -- STEP: 8080/36938 -- GLOBAL_STEP: 1485600 | > loss_text_ce: 0.03417712450027466 (0.030345808799115772) | > loss_mel_ce: 2.361416816711426 (nan) | > loss: 0.009506325237452984 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.1372408181723987) | > loader_time: 0.0046 (0.004664658791948064)  --> TIME: 2025-02-27 01:20:49 -- STEP: 8130/36938 -- GLOBAL_STEP: 1485650 | > loss_text_ce: 0.02737519145011902 (0.030348281502416686) | > loss_mel_ce: 2.7020294666290283 (nan) | > loss: 0.010830971412360668 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13726849086756832) | > loader_time: 0.0048 (0.004664852932956964)  --> TIME: 2025-02-27 01:20:59 -- STEP: 8180/36938 -- GLOBAL_STEP: 1485700 | > loss_text_ce: 0.022914685308933258 (0.030347085420912114) | > loss_mel_ce: 2.3221263885498047 (nan) | > loss: 0.009305719286203384 (nan) | > current_lr: 5e-06 | > step_time: 0.1473 (0.13727566379789669) | > loader_time: 0.0047 (0.004664763844683575)  --> TIME: 2025-02-27 01:21:08 -- STEP: 8230/36938 -- GLOBAL_STEP: 1485750 | > loss_text_ce: 0.023510364815592766 (0.03035314953212659) | > loss_mel_ce: 2.5029349327087402 (nan) | > loss: 0.010025577619671822 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1372854215696115) | > loader_time: 0.0046 (0.004664944067036098)  --> TIME: 2025-02-27 01:21:17 -- STEP: 8280/36938 -- GLOBAL_STEP: 1485800 | > loss_text_ce: 0.02694774977862835 (0.030356885827650372) | > loss_mel_ce: 2.6362619400024414 (nan) | > loss: 0.010568292811512947 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13726517771177296) | > loader_time: 0.0045 (0.004664942982116175)  --> TIME: 2025-02-27 01:21:27 -- STEP: 8330/36938 -- GLOBAL_STEP: 1485850 | > loss_text_ce: 0.027529818937182426 (0.03036146023079287) | > loss_mel_ce: 2.411203145980835 (nan) | > loss: 0.009677511639893055 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.1372982703480262) | > loader_time: 0.0046 (0.004665044289963291)  --> TIME: 2025-02-27 01:21:36 -- STEP: 8380/36938 -- GLOBAL_STEP: 1485900 | > loss_text_ce: 0.027540812268853188 (0.03036651637145097) | > loss_mel_ce: 3.01623797416687 (nan) | > loss: 0.012078488245606422 (nan) | > current_lr: 5e-06 | > step_time: 0.1721 (0.137282959345133) | > loader_time: 0.0048 (0.004664923410711532)  --> TIME: 2025-02-27 01:21:46 -- STEP: 8430/36938 -- GLOBAL_STEP: 1485950 | > loss_text_ce: 0.03060506097972393 (0.030369605801774183) | > loss_mel_ce: 2.6824898719787598 (nan) | > loss: 0.01076625008136034 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13729030695674313) | > loader_time: 0.0045 (0.004664732043853428)  --> TIME: 2025-02-27 01:21:55 -- STEP: 8480/36938 -- GLOBAL_STEP: 1486000 | > loss_text_ce: 0.031738147139549255 (0.030373981935609085) | > loss_mel_ce: 2.457415819168091 (nan) | > loss: 0.009877595119178295 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13730382483522902) | > loader_time: 0.0046 (0.0046647169117657586)  --> TIME: 2025-02-27 01:22:05 -- STEP: 8530/36938 -- GLOBAL_STEP: 1486050 | > loss_text_ce: 0.029141362756490707 (0.030372045858973865) | > loss_mel_ce: 2.2714405059814453 (nan) | > loss: 0.009129294194281101 (nan) | > current_lr: 5e-06 | > step_time: 0.1651 (0.13731854512571365) | > loader_time: 0.0048 (0.004664689854468713)  --> TIME: 2025-02-27 01:22:14 -- STEP: 8580/36938 -- GLOBAL_STEP: 1486100 | > loss_text_ce: 0.024250756949186325 (0.030372208068776854) | > loss_mel_ce: 3.257686138153076 (nan) | > loss: 0.01302355993539095 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.13731359348986305) | > loader_time: 0.0048 (0.004664802662420379)  --> TIME: 2025-02-27 01:22:23 -- STEP: 8630/36938 -- GLOBAL_STEP: 1486150 | > loss_text_ce: 0.023308005183935165 (0.03036718551729948) | > loss_mel_ce: 2.586275100708008 (nan) | > loss: 0.010355489328503609 (nan) | > current_lr: 5e-06 | > step_time: 0.1612 (0.13728787310568277) | > loader_time: 0.0047 (0.004664766829922738)  --> TIME: 2025-02-27 01:22:32 -- STEP: 8680/36938 -- GLOBAL_STEP: 1486200 | > loss_text_ce: 0.03045109286904335 (0.03037127657613205) | > loss_mel_ce: 1.1922760009765625 (nan) | > loss: 0.004852091893553734 (nan) | > current_lr: 5e-06 | > step_time: 0.1441 (0.13726860326006715) | > loader_time: 0.0046 (0.004664785081889766)  --> TIME: 2025-02-27 01:22:42 -- STEP: 8730/36938 -- GLOBAL_STEP: 1486250 | > loss_text_ce: 0.01978314481675625 (0.030367850880207905) | > loss_mel_ce: 0.00986589677631855 (nan) | > loss: 0.00011765493400162086 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.1372633836796471) | > loader_time: 0.0046 (0.004664781303886418)  --> TIME: 2025-02-27 01:22:51 -- STEP: 8780/36938 -- GLOBAL_STEP: 1486300 | > loss_text_ce: 0.02429390698671341 (0.030362832860104674) | > loss_mel_ce: 2.8712949752807617 (nan) | > loss: 0.011490432545542717 (nan) | > current_lr: 5e-06 | > step_time: 0.1846 (0.13727192542004465) | > loader_time: 0.0048 (0.0046652422679040835)  --> TIME: 2025-02-27 01:23:01 -- STEP: 8830/36938 -- GLOBAL_STEP: 1486350 | > loss_text_ce: 0.027548834681510925 (0.030354776743130737) | > loss_mel_ce: 2.5314218997955322 (nan) | > loss: 0.010154645889997482 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.13726883574493381) | > loader_time: 0.0046 (0.004665521506139941)  --> TIME: 2025-02-27 01:23:10 -- STEP: 8880/36938 -- GLOBAL_STEP: 1486400 | > loss_text_ce: 0.029221713542938232 (0.030348382762115152) | > loss_mel_ce: 2.7710928916931152 (nan) | > loss: 0.011112360283732414 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13727346140522173) | > loader_time: 0.0045 (0.004665347235696809)  --> TIME: 2025-02-27 01:23:19 -- STEP: 8930/36938 -- GLOBAL_STEP: 1486450 | > loss_text_ce: 0.024325702339410782 (0.03034956953561293) | > loss_mel_ce: 2.681957960128784 (nan) | > loss: 0.01073922123759985 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.1372796274112564) | > loader_time: 0.0047 (0.004665150834625005)  --> TIME: 2025-02-27 01:23:29 -- STEP: 8980/36938 -- GLOBAL_STEP: 1486500 | > loss_text_ce: 0.027829714119434357 (0.030358472220518743) | > loss_mel_ce: 2.626499652862549 (nan) | > loss: 0.010533053427934647 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13728847216922654) | > loader_time: 0.0045 (0.004664854562627712)  --> TIME: 2025-02-27 01:23:38 -- STEP: 9030/36938 -- GLOBAL_STEP: 1486550 | > loss_text_ce: 0.02463466115295887 (0.03035647132700042) | > loss_mel_ce: 2.0340616703033447 (nan) | > loss: 0.008169430308043957 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13729105309922598) | > loader_time: 0.0045 (0.0046645304425346245)  --> TIME: 2025-02-27 01:23:48 -- STEP: 9080/36938 -- GLOBAL_STEP: 1486600 | > loss_text_ce: 0.032471008598804474 (0.030359070279198534) | > loss_mel_ce: 2.787238836288452 (nan) | > loss: 0.011189324781298637 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13731724667129053) | > loader_time: 0.0046 (0.004664378045413988)  --> TIME: 2025-02-27 01:23:57 -- STEP: 9130/36938 -- GLOBAL_STEP: 1486650 | > loss_text_ce: 0.03613713011145592 (0.030355699557294856) | > loss_mel_ce: 3.478630781173706 (nan) | > loss: 0.013947492465376854 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.1373079940236023) | > loader_time: 0.0045 (0.004664017101759977)  --> TIME: 2025-02-27 01:24:06 -- STEP: 9180/36938 -- GLOBAL_STEP: 1486700 | > loss_text_ce: 0.022501548752188683 (0.03035194927665215) | > loss_mel_ce: 2.5220775604248047 (nan) | > loss: 0.010097536258399487 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13730503975176378) | > loader_time: 0.0046 (0.004664605063810326)  --> TIME: 2025-02-27 01:24:16 -- STEP: 9230/36938 -- GLOBAL_STEP: 1486750 | > loss_text_ce: 0.029184214770793915 (0.030354615112258065) | > loss_mel_ce: 2.758845567703247 (nan) | > loss: 0.011063610203564167 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.13733331415790162) | > loader_time: 0.0046 (0.004664661979881932)  --> TIME: 2025-02-27 01:24:25 -- STEP: 9280/36938 -- GLOBAL_STEP: 1486800 | > loss_text_ce: 0.02907518483698368 (0.030360296254642375) | > loss_mel_ce: 3.0310723781585693 (nan) | > loss: 0.012143443338572979 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13732011194886853) | > loader_time: 0.0047 (0.004665174232474681)  --> TIME: 2025-02-27 01:24:35 -- STEP: 9330/36938 -- GLOBAL_STEP: 1486850 | > loss_text_ce: 0.03806459158658981 (0.03036227993680101) | > loss_mel_ce: 2.9354872703552246 (nan) | > loss: 0.011799809522926807 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.1373067895188084) | > loader_time: 0.0046 (0.004664958583835867)  --> TIME: 2025-02-27 01:24:44 -- STEP: 9380/36938 -- GLOBAL_STEP: 1486900 | > loss_text_ce: 0.0309916939586401 (0.030368040416931804) | > loss_mel_ce: 3.058393955230713 (nan) | > loss: 0.012259467504918575 (nan) | > current_lr: 5e-06 | > step_time: 0.1824 (0.13728713961298308) | > loader_time: 0.0049 (0.004664906535321455)  --> TIME: 2025-02-27 01:24:53 -- STEP: 9430/36938 -- GLOBAL_STEP: 1486950 | > loss_text_ce: 0.03257131949067116 (0.03037150001260999) | > loss_mel_ce: 3.17500376701355 (nan) | > loss: 0.012728473171591759 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.1372812855408026) | > loader_time: 0.0046 (0.0046647156789219315)  --> TIME: 2025-02-27 01:25:03 -- STEP: 9480/36938 -- GLOBAL_STEP: 1487000 | > loss_text_ce: 0.029584109783172607 (0.030371157888211063) | > loss_mel_ce: 2.557217597961426 (nan) | > loss: 0.010265086777508259 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.13728909039799203) | > loader_time: 0.0048 (0.0046646780605557525)  --> TIME: 2025-02-27 01:25:12 -- STEP: 9530/36938 -- GLOBAL_STEP: 1487050 | > loss_text_ce: 0.03532343730330467 (0.030372056660094803) | > loss_mel_ce: 3.123014211654663 (nan) | > loss: 0.012533086352050304 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.1373224315213009) | > loader_time: 0.0047 (0.004664959612322992)  --> TIME: 2025-02-27 01:25:22 -- STEP: 9580/36938 -- GLOBAL_STEP: 1487100 | > loss_text_ce: 0.023572539910674095 (0.030374860144239724) | > loss_mel_ce: 0.517626941204071 (nan) | > loss: 0.002147617284208536 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.137316384444904) | > loader_time: 0.0046 (0.004665017725281515)  --> TIME: 2025-02-27 01:25:31 -- STEP: 9630/36938 -- GLOBAL_STEP: 1487150 | > loss_text_ce: 0.03130832687020302 (0.030369986630134807) | > loss_mel_ce: 0.61041259765625 (nan) | > loss: 0.002546511823311448 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13728434371056988) | > loader_time: 0.0045 (0.004664665491533911)  --> TIME: 2025-02-27 01:25:40 -- STEP: 9680/36938 -- GLOBAL_STEP: 1487200 | > loss_text_ce: 0.02529866248369217 (0.03036670541176225) | > loss_mel_ce: 2.4502627849578857 (nan) | > loss: 0.009823656640946865 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13725909164621833) | > loader_time: 0.0045 (0.004664296355129259)  --> TIME: 2025-02-27 01:25:49 -- STEP: 9730/36938 -- GLOBAL_STEP: 1487250 | > loss_text_ce: 0.032372504472732544 (0.030362543681605812) | > loss_mel_ce: 2.812859296798706 (nan) | > loss: 0.011290603317320347 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13724083910253063) | > loader_time: 0.0045 (0.0046640914121119995)  --> TIME: 2025-02-27 01:25:58 -- STEP: 9780/36938 -- GLOBAL_STEP: 1487300 | > loss_text_ce: 0.02931634895503521 (0.030365758939124327) | > loss_mel_ce: 2.803831100463867 (nan) | > loss: 0.011242649517953396 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13725576064338005) | > loader_time: 0.0046 (0.004663879349675396)  --> TIME: 2025-02-27 01:26:08 -- STEP: 9830/36938 -- GLOBAL_STEP: 1487350 | > loss_text_ce: 0.041335608810186386 (0.03036279269950776) | > loss_mel_ce: 3.0494515895843506 (nan) | > loss: 0.012265029363334179 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.1372764198850399) | > loader_time: 0.0046 (0.0046639557166114004)  --> TIME: 2025-02-27 01:26:17 -- STEP: 9880/36938 -- GLOBAL_STEP: 1487400 | > loss_text_ce: 0.02224661223590374 (0.030359902826414965) | > loss_mel_ce: 2.698791742324829 (nan) | > loss: 0.010797771625220776 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.1372643970767499) | > loader_time: 0.0046 (0.004663961594886614)  --> TIME: 2025-02-27 01:26:27 -- STEP: 9930/36938 -- GLOBAL_STEP: 1487450 | > loss_text_ce: 0.027863822877407074 (0.030360266918781666) | > loss_mel_ce: 1.7510241270065308 (nan) | > loss: 0.007059079594910145 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13726344127789517) | > loader_time: 0.0045 (0.004663968734510702)  --> TIME: 2025-02-27 01:26:36 -- STEP: 9980/36938 -- GLOBAL_STEP: 1487500 | > loss_text_ce: 0.03347717598080635 (0.030363296605375973) | > loss_mel_ce: 2.590427875518799 (nan) | > loss: 0.010412321425974369 (nan) | > current_lr: 5e-06 | > step_time: 0.1848 (0.1372769429831804) | > loader_time: 0.0049 (0.004664006237993244)  --> TIME: 2025-02-27 01:26:45 -- STEP: 10030/36938 -- GLOBAL_STEP: 1487550 | > loss_text_ce: 0.03567301481962204 (0.03035957792719058) | > loss_mel_ce: 2.66979718208313 (nan) | > loss: 0.010735993273556232 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13727363529851905) | > loader_time: 0.0046 (0.004664028819988384)  --> TIME: 2025-02-27 01:26:54 -- STEP: 10080/36938 -- GLOBAL_STEP: 1487600 | > loss_text_ce: 0.025476302951574326 (0.030360715563680654) | > loss_mel_ce: 1.4461179971694946 (nan) | > loss: 0.005839660298079252 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.1372383094732727) | > loader_time: 0.0046 (0.00466378421064406)  --> TIME: 2025-02-27 01:27:04 -- STEP: 10130/36938 -- GLOBAL_STEP: 1487650 | > loss_text_ce: 0.03506350889801979 (0.030358602473170276) | > loss_mel_ce: 3.2886500358581543 (nan) | > loss: 0.013189340010285378 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13723111825775447) | > loader_time: 0.0046 (0.00466363039327396)  --> TIME: 2025-02-27 01:27:13 -- STEP: 10180/36938 -- GLOBAL_STEP: 1487700 | > loss_text_ce: 0.030794154852628708 (0.030363592902456413) | > loss_mel_ce: 2.0346176624298096 (nan) | > loss: 0.008196079172194004 (nan) | > current_lr: 5e-06 | > step_time: 0.1079 (0.13721747405402587) | > loader_time: 0.0058 (0.004663826955559198)  --> TIME: 2025-02-27 01:27:22 -- STEP: 10230/36938 -- GLOBAL_STEP: 1487750 | > loss_text_ce: 0.02838679775595665 (0.030366887802163442) | > loss_mel_ce: 0.60890793800354 (nan) | > loss: 0.002528947312384844 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13720458283335865) | > loader_time: 0.0047 (0.004663843493307778)  --> TIME: 2025-02-27 01:27:32 -- STEP: 10280/36938 -- GLOBAL_STEP: 1487800 | > loss_text_ce: 0.0341552197933197 (0.030362596588004515) | > loss_mel_ce: 2.3958232402801514 (nan) | > loss: 0.009642771445214748 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.13721672375378494) | > loader_time: 0.0047 (0.00466397005760251)  --> TIME: 2025-02-27 01:27:41 -- STEP: 10330/36938 -- GLOBAL_STEP: 1487850 | > loss_text_ce: 0.023367391899228096 (0.030365334206827816) | > loss_mel_ce: 2.832240581512451 (nan) | > loss: 0.011331778019666672 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.1372128365231396) | > loader_time: 0.0047 (0.004663994766966523)  --> TIME: 2025-02-27 01:27:50 -- STEP: 10380/36938 -- GLOBAL_STEP: 1487900 | > loss_text_ce: 0.02644440531730652 (0.030375491958317877) | > loss_mel_ce: 2.1280720233917236 (nan) | > loss: 0.008549668826162815 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13719928103834728) | > loader_time: 0.0045 (0.004664278351961981)  --> TIME: 2025-02-27 01:27:59 -- STEP: 10430/36938 -- GLOBAL_STEP: 1487950 | > loss_text_ce: 0.026715373620390892 (0.030376287240201055) | > loss_mel_ce: 2.9450175762176514 (nan) | > loss: 0.011792591772973537 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13717999686902288) | > loader_time: 0.0046 (0.004664775554880075)  --> TIME: 2025-02-27 01:28:09 -- STEP: 10480/36938 -- GLOBAL_STEP: 1488000 | > loss_text_ce: 0.0330631397664547 (0.030372007387176585) | > loss_mel_ce: 3.025009870529175 (nan) | > loss: 0.012135211378335953 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13720503961312006) | > loader_time: 0.0046 (0.004664956658851092)  --> TIME: 2025-02-27 01:28:19 -- STEP: 10530/36938 -- GLOBAL_STEP: 1488050 | > loss_text_ce: 0.028319159522652626 (0.030370640063548674) | > loss_mel_ce: 2.840672731399536 (nan) | > loss: 0.011384888552129269 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13722928842814497) | > loader_time: 0.0046 (0.004665102827356516)  --> TIME: 2025-02-27 01:28:29 -- STEP: 10580/36938 -- GLOBAL_STEP: 1488100 | > loss_text_ce: 0.032917317003011703 (0.030372305922976038) | > loss_mel_ce: 2.6036312580108643 (nan) | > loss: 0.010462494567036629 (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.1372567497038891) | > loader_time: 0.0046 (0.004665253563504128)  --> TIME: 2025-02-27 01:28:38 -- STEP: 10630/36938 -- GLOBAL_STEP: 1488150 | > loss_text_ce: 0.031990863382816315 (0.03036556994478521) | > loss_mel_ce: 2.8067820072174072 (nan) | > loss: 0.011264972388744354 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.1372518582940667) | > loader_time: 0.0048 (0.004665129182480006)  --> TIME: 2025-02-27 01:28:47 -- STEP: 10680/36938 -- GLOBAL_STEP: 1488200 | > loss_text_ce: 0.030845245346426964 (0.030366041201074362) | > loss_mel_ce: 1.28680419921875 (nan) | > loss: 0.005228768102824688 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13725538113144048) | > loader_time: 0.0047 (0.004665372121646605)  --> TIME: 2025-02-27 01:28:57 -- STEP: 10730/36938 -- GLOBAL_STEP: 1488250 | > loss_text_ce: 0.033448848873376846 (0.030367250865731527) | > loss_mel_ce: 2.532114028930664 (nan) | > loss: 0.010180805809795856 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1372715472064063) | > loader_time: 0.0046 (0.004665376155707509)  --> TIME: 2025-02-27 01:29:06 -- STEP: 10780/36938 -- GLOBAL_STEP: 1488300 | > loss_text_ce: 0.0311697069555521 (0.030371021875266687) | > loss_mel_ce: 1.9256856441497803 (nan) | > loss: 0.007765299174934626 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13729375689281872) | > loader_time: 0.0045 (0.004665358146179141)  --> TIME: 2025-02-27 01:29:16 -- STEP: 10830/36938 -- GLOBAL_STEP: 1488350 | > loss_text_ce: 0.03122897259891033 (0.03037153863810717) | > loss_mel_ce: 0.00335693359375 (nan) | > loss: 0.00013724567543249577 (nan) | > current_lr: 5e-06 | > step_time: 0.1349 (0.1372899844705898) | > loader_time: 0.0046 (0.004665165396608479)  --> TIME: 2025-02-27 01:29:25 -- STEP: 10880/36938 -- GLOBAL_STEP: 1488400 | > loss_text_ce: 0.03959263116121292 (0.03036773306739632) | > loss_mel_ce: 2.929590940475464 (nan) | > loss: 0.011782475747168064 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.13730287580367442) | > loader_time: 0.0047 (0.004665023395243816)  --> TIME: 2025-02-27 01:29:35 -- STEP: 10930/36938 -- GLOBAL_STEP: 1488450 | > loss_text_ce: 0.026103388518095016 (0.03036706523125358) | > loss_mel_ce: 2.952103853225708 (nan) | > loss: 0.011818284168839455 (nan) | > current_lr: 5e-06 | > step_time: 0.1821 (0.13730580691122118) | > loader_time: 0.0048 (0.004664942635818021)  --> TIME: 2025-02-27 01:29:44 -- STEP: 10980/36938 -- GLOBAL_STEP: 1488500 | > loss_text_ce: 0.031043844297528267 (0.030371583752013487) | > loss_mel_ce: 0.8885574340820312 (nan) | > loss: 0.0036492114886641502 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13731217054287218) | > loader_time: 0.0046 (0.004664912836147791)  --> TIME: 2025-02-27 01:29:53 -- STEP: 11030/36938 -- GLOBAL_STEP: 1488550 | > loss_text_ce: 0.02916727028787136 (0.03037206762325075) | > loss_mel_ce: 2.762990713119507 (nan) | > loss: 0.011079992167651653 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13731882228488246) | > loader_time: 0.0045 (0.004664762065503569)  --> TIME: 2025-02-27 01:30:03 -- STEP: 11080/36938 -- GLOBAL_STEP: 1488600 | > loss_text_ce: 0.03253048285841942 (0.03036875049360869) | > loss_mel_ce: 3.043606758117676 (nan) | > loss: 0.012206894345581532 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13730091827010496) | > loader_time: 0.0046 (0.004664532501344653)  --> TIME: 2025-02-27 01:30:12 -- STEP: 11130/36938 -- GLOBAL_STEP: 1488650 | > loss_text_ce: 0.030719945207238197 (0.030370569088339783) | > loss_mel_ce: 2.6110446453094482 (nan) | > loss: 0.010483193211257458 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.1372945785308256) | > loader_time: 0.0046 (0.0046643679139856216)  --> TIME: 2025-02-27 01:30:21 -- STEP: 11180/36938 -- GLOBAL_STEP: 1488700 | > loss_text_ce: 0.030256615951657295 (0.030373109630325697) | > loss_mel_ce: 2.9747347831726074 (nan) | > loss: 0.011924569495022297 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13730076434147453) | > loader_time: 0.0046 (0.0046642412013359336)  --> TIME: 2025-02-27 01:30:31 -- STEP: 11230/36938 -- GLOBAL_STEP: 1488750 | > loss_text_ce: 0.02349378541111946 (0.030377389631297956) | > loss_mel_ce: 1.8329581022262573 (nan) | > loss: 0.007366872858256102 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.1372907629420695) | > loader_time: 0.0046 (0.004664021077584929)  --> TIME: 2025-02-27 01:30:40 -- STEP: 11280/36938 -- GLOBAL_STEP: 1488800 | > loss_text_ce: 0.030200041830539703 (0.030373605063180133) | > loss_mel_ce: 2.3999722003936768 (nan) | > loss: 0.00964354071766138 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13730520690586626) | > loader_time: 0.0045 (0.004663927440947675)  --> TIME: 2025-02-27 01:30:50 -- STEP: 11330/36938 -- GLOBAL_STEP: 1488850 | > loss_text_ce: 0.025449594482779503 (0.030371789641831216) | > loss_mel_ce: 2.084195613861084 (nan) | > loss: 0.008371608331799507 (nan) | > current_lr: 5e-06 | > step_time: 0.1265 (0.13730176757735152) | > loader_time: 0.0046 (0.004663699260351414)  --> TIME: 2025-02-27 01:30:59 -- STEP: 11380/36938 -- GLOBAL_STEP: 1488900 | > loss_text_ce: 0.02495737187564373 (0.030374129131307034) | > loss_mel_ce: 0.00927734375 (nan) | > loss: 0.00013585206761490554 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13729400014626011) | > loader_time: 0.0045 (0.004663471073593112)  --> TIME: 2025-02-27 01:31:09 -- STEP: 11430/36938 -- GLOBAL_STEP: 1488950 | > loss_text_ce: 0.036053527146577835 (0.030370431899256856) | > loss_mel_ce: 3.143636465072632 (nan) | > loss: 0.01261781807988882 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.13732185564016214) | > loader_time: 0.0048 (0.004663410545542692)  --> TIME: 2025-02-27 01:31:18 -- STEP: 11480/36938 -- GLOBAL_STEP: 1489000 | > loss_text_ce: 0.03363461047410965 (0.03036877227049054) | > loss_mel_ce: 2.185417652130127 (nan) | > loss: 0.00880576390773058 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.1373104073860099) | > loader_time: 0.0047 (0.004663311812105081)  --> TIME: 2025-02-27 01:31:27 -- STEP: 11530/36938 -- GLOBAL_STEP: 1489050 | > loss_text_ce: 0.030303403735160828 (0.030369308327768126) | > loss_mel_ce: 2.3084819316864014 (nan) | > loss: 0.009280894882977009 (nan) | > current_lr: 5e-06 | > step_time: 0.1118 (0.13730266160377813) | > loader_time: 0.0045 (0.004663200473537268)  --> TIME: 2025-02-27 01:31:37 -- STEP: 11580/36938 -- GLOBAL_STEP: 1489100 | > loss_text_ce: 0.03502072021365166 (0.030373587057320954) | > loss_mel_ce: 2.3463969230651855 (nan) | > loss: 0.009450070559978485 (nan) | > current_lr: 5e-06 | > step_time: 0.1577 (0.1373051946439078) | > loader_time: 0.0047 (0.004663139591974714)  --> TIME: 2025-02-27 01:31:46 -- STEP: 11630/36938 -- GLOBAL_STEP: 1489150 | > loss_text_ce: 0.027654489502310753 (0.030379472445954438) | > loss_mel_ce: 2.9039530754089355 (nan) | > loss: 0.011633363552391529 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.1373024002191649) | > loader_time: 0.0046 (0.004662981406430801)  --> TIME: 2025-02-27 01:31:56 -- STEP: 11680/36938 -- GLOBAL_STEP: 1489200 | > loss_text_ce: 0.01716628298163414 (0.030374096965374197) | > loss_mel_ce: 1.3790034055709839 (nan) | > loss: 0.0055403560400009155 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.1373121113605706) | > loader_time: 0.0044 (0.004662877525368786)  --> TIME: 2025-02-27 01:32:05 -- STEP: 11730/36938 -- GLOBAL_STEP: 1489250 | > loss_text_ce: 0.027035189792513847 (0.030374234764759703) | > loss_mel_ce: 2.7134757041931152 (nan) | > loss: 0.010875044390559196 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.13733150993568416) | > loader_time: 0.0047 (0.0046628079857464495)  --> TIME: 2025-02-27 01:32:15 -- STEP: 11780/36938 -- GLOBAL_STEP: 1489300 | > loss_text_ce: 0.029273003339767456 (0.030374701594236553) | > loss_mel_ce: 2.6854946613311768 (nan) | > loss: 0.0107728885486722 (nan) | > current_lr: 5e-06 | > step_time: 0.1819 (0.137336546638017) | > loader_time: 0.0048 (0.004662725314219667)  --> TIME: 2025-02-27 01:32:24 -- STEP: 11830/36938 -- GLOBAL_STEP: 1489350 | > loss_text_ce: 0.030255906283855438 (0.03038469753558774) | > loss_mel_ce: 2.701586961746216 (nan) | > loss: 0.010840646922588348 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13732980857536436) | > loader_time: 0.0045 (0.004662496541781854)  --> TIME: 2025-02-27 01:32:33 -- STEP: 11880/36938 -- GLOBAL_STEP: 1489400 | > loss_text_ce: 0.045780766755342484 (0.030384580943198795) | > loss_mel_ce: 2.382293701171875 (nan) | > loss: 0.009635216556489468 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13732161026209794) | > loader_time: 0.0045 (0.004662302066180055)  --> TIME: 2025-02-27 01:32:43 -- STEP: 11930/36938 -- GLOBAL_STEP: 1489450 | > loss_text_ce: 0.03334416449069977 (0.030387246496905152) | > loss_mel_ce: 1.456152319908142 (nan) | > loss: 0.005910700652748346 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13732596692775245) | > loader_time: 0.0046 (0.004662198213000126)  --> TIME: 2025-02-27 01:32:52 -- STEP: 11980/36938 -- GLOBAL_STEP: 1489500 | > loss_text_ce: 0.02375791408121586 (0.030390837525160593) | > loss_mel_ce: 1.5715497732162476 (nan) | > loss: 0.00633058650419116 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13731876253484784) | > loader_time: 0.0045 (0.004662004735115589)  --> TIME: 2025-02-27 01:33:01 -- STEP: 12030/36938 -- GLOBAL_STEP: 1489550 | > loss_text_ce: 0.031928759068250656 (0.03039102386958054) | > loss_mel_ce: 2.5992074012756348 (nan) | > loss: 0.010441017337143421 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1373143966655789) | > loader_time: 0.0045 (0.004662231653804906)  --> TIME: 2025-02-27 01:33:11 -- STEP: 12080/36938 -- GLOBAL_STEP: 1489600 | > loss_text_ce: 0.03438558429479599 (0.030393341148159586) | > loss_mel_ce: 2.7209830284118652 (nan) | > loss: 0.010934003628790379 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.13734705270521005) | > loader_time: 0.0046 (0.004662532048509626)  --> TIME: 2025-02-27 01:33:20 -- STEP: 12130/36938 -- GLOBAL_STEP: 1489650 | > loss_text_ce: 0.02501368708908558 (0.030394355399896492) | > loss_mel_ce: 2.854536771774292 (nan) | > loss: 0.011426787823438644 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13733978670486716) | > loader_time: 0.0046 (0.004662448182542381)  --> TIME: 2025-02-27 01:33:30 -- STEP: 12180/36938 -- GLOBAL_STEP: 1489700 | > loss_text_ce: 0.03723403438925743 (0.030392643078367945) | > loss_mel_ce: 2.89208984375 (nan) | > loss: 0.01162430178374052 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13734276486539398) | > loader_time: 0.0046 (0.00466250983756557)  --> TIME: 2025-02-27 01:33:39 -- STEP: 12230/36938 -- GLOBAL_STEP: 1489750 | > loss_text_ce: 0.04361493140459061 (0.03039295257957086) | > loss_mel_ce: 2.4071993827819824 (nan) | > loss: 0.00972545426338911 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13735929174883396) | > loader_time: 0.0046 (0.004662640038222846)  --> TIME: 2025-02-27 01:33:49 -- STEP: 12280/36938 -- GLOBAL_STEP: 1489800 | > loss_text_ce: 0.030945774167776108 (0.030393515541544326) | > loss_mel_ce: 2.854731321334839 (nan) | > loss: 0.011451099999248981 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13737368261387078) | > loader_time: 0.0046 (0.004662748947982306)  --> TIME: 2025-02-27 01:33:58 -- STEP: 12330/36938 -- GLOBAL_STEP: 1489850 | > loss_text_ce: 0.03259727358818054 (0.03039503987868741) | > loss_mel_ce: 2.968722343444824 (nan) | > loss: 0.011909998953342438 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13738424932676402) | > loader_time: 0.0046 (0.004662703558959729)  --> TIME: 2025-02-27 01:34:08 -- STEP: 12380/36938 -- GLOBAL_STEP: 1489900 | > loss_text_ce: 0.033777959644794464 (0.030394527833624424) | > loss_mel_ce: 2.989853620529175 (nan) | > loss: 0.011998538859188557 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13740289902648642) | > loader_time: 0.0047 (0.00466280859776192)  --> TIME: 2025-02-27 01:34:17 -- STEP: 12430/36938 -- GLOBAL_STEP: 1489950 | > loss_text_ce: 0.04294053092598915 (0.030394680104656728) | > loss_mel_ce: 0.2918701171875 (nan) | > loss: 0.001328613725490868 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.13739450405608902) | > loader_time: 0.0045 (0.004663182168072133)  --> TIME: 2025-02-27 01:34:27 -- STEP: 12480/36938 -- GLOBAL_STEP: 1490000 | > loss_text_ce: 0.031949929893016815 (0.03039426129018045) | > loss_mel_ce: 2.810870409011841 (nan) | > loss: 0.011281033977866173 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13741475574863365) | > loader_time: 0.0048 (0.004663516504642314) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1490000.pth  --> TIME: 2025-02-27 01:34:42 -- STEP: 12530/36938 -- GLOBAL_STEP: 1490050 | > loss_text_ce: 0.03329664096236229 (0.030396710811420695) | > loss_mel_ce: 2.8436167240142822 (nan) | > loss: 0.011416323482990265 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13740223314128586) | > loader_time: 0.0046 (0.004663582392340195)  --> TIME: 2025-02-27 01:34:51 -- STEP: 12580/36938 -- GLOBAL_STEP: 1490100 | > loss_text_ce: 0.037689127027988434 (0.030396379172966313) | > loss_mel_ce: 0.111328125 (nan) | > loss: 0.0005913383211009204 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13737357159297783) | > loader_time: 0.0045 (0.004663333680557738)  --> TIME: 2025-02-27 01:35:00 -- STEP: 12630/36938 -- GLOBAL_STEP: 1490150 | > loss_text_ce: 0.03360849618911743 (0.030397481574645214) | > loss_mel_ce: 2.923626184463501 (nan) | > loss: 0.011735058389604092 (nan) | > current_lr: 5e-06 | > step_time: 0.175 (0.1373536019313941) | > loader_time: 0.0048 (0.00466323078197805)  --> TIME: 2025-02-27 01:35:10 -- STEP: 12680/36938 -- GLOBAL_STEP: 1490200 | > loss_text_ce: 0.0331769660115242 (0.030398199321923926) | > loss_mel_ce: 2.074805974960327 (nan) | > loss: 0.008365011774003506 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.13736073728239603) | > loader_time: 0.0047 (0.004663617340172152)  --> TIME: 2025-02-27 01:35:19 -- STEP: 12730/36938 -- GLOBAL_STEP: 1490250 | > loss_text_ce: 0.03553622215986252 (0.03039800765600789) | > loss_mel_ce: 2.883695125579834 (nan) | > loss: 0.011584252119064331 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.1373519816552208) | > loader_time: 0.0049 (0.004663682470898528)  --> TIME: 2025-02-27 01:35:28 -- STEP: 12780/36938 -- GLOBAL_STEP: 1490300 | > loss_text_ce: 0.028647782281041145 (0.030397862202272172) | > loss_mel_ce: 2.815643787384033 (nan) | > loss: 0.011286872439086437 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13734231077262982) | > loader_time: 0.0045 (0.004663868670545669)  --> TIME: 2025-02-27 01:35:37 -- STEP: 12830/36938 -- GLOBAL_STEP: 1490350 | > loss_text_ce: 0.02895105443894863 (0.030396323155253153) | > loss_mel_ce: 0.009796142578125 (nan) | > loss: 0.0001537587377242744 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13731356482903595) | > loader_time: 0.0046 (0.004663507379039808)  --> TIME: 2025-02-27 01:35:47 -- STEP: 12880/36938 -- GLOBAL_STEP: 1490400 | > loss_text_ce: 0.036068934947252274 (0.03039553885110248) | > loss_mel_ce: 2.836383819580078 (nan) | > loss: 0.011398622766137123 (nan) | > current_lr: 5e-06 | > step_time: 0.1515 (0.13732557126454062) | > loader_time: 0.0046 (0.004663394012066173)  --> TIME: 2025-02-27 01:35:56 -- STEP: 12930/36938 -- GLOBAL_STEP: 1490450 | > loss_text_ce: 0.0342530831694603 (0.030396583358442426) | > loss_mel_ce: 2.245386838912964 (nan) | > loss: 0.009046190418303013 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13732399162357486) | > loader_time: 0.0045 (0.004663414081122116)  --> TIME: 2025-02-27 01:36:05 -- STEP: 12980/36938 -- GLOBAL_STEP: 1490500 | > loss_text_ce: 0.033557113260030746 (0.03039598307089423) | > loss_mel_ce: 1.6294951438903809 (nan) | > loss: 0.006599413696676493 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13733033937372033) | > loader_time: 0.0045 (0.004663268259750883)  --> TIME: 2025-02-27 01:36:15 -- STEP: 13030/36938 -- GLOBAL_STEP: 1490550 | > loss_text_ce: 0.02986416593194008 (0.030394504690351255) | > loss_mel_ce: 2.0905919075012207 (nan) | > loss: 0.008414508774876595 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13734136366972358) | > loader_time: 0.0047 (0.004663284778229044)  --> TIME: 2025-02-27 01:36:25 -- STEP: 13080/36938 -- GLOBAL_STEP: 1490600 | > loss_text_ce: 0.038130756467580795 (0.030399502790141385) | > loss_mel_ce: 2.4831175804138184 (nan) | > loss: 0.010004954412579536 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13735441242883056) | > loader_time: 0.0046 (0.004663312945526335)  --> TIME: 2025-02-27 01:36:34 -- STEP: 13130/36938 -- GLOBAL_STEP: 1490650 | > loss_text_ce: 0.03595408424735069 (0.030400661600367793) | > loss_mel_ce: 2.79685115814209 (nan) | > loss: 0.011241290718317032 (nan) | > current_lr: 5e-06 | > step_time: 0.1875 (0.13735877102055166) | > loader_time: 0.0049 (0.0046633104104988705)  --> TIME: 2025-02-27 01:36:43 -- STEP: 13180/36938 -- GLOBAL_STEP: 1490700 | > loss_text_ce: 0.033409763127565384 (0.03040211422249465) | > loss_mel_ce: 2.506944417953491 (nan) | > loss: 0.010080771520733833 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.1373386333310739) | > loader_time: 0.0045 (0.0046631880703754125)  --> TIME: 2025-02-27 01:36:52 -- STEP: 13230/36938 -- GLOBAL_STEP: 1490750 | > loss_text_ce: 0.029947081580758095 (0.030404433805406795) | > loss_mel_ce: 3.253549814224243 (nan) | > loss: 0.013029750436544418 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.137324774995143) | > loader_time: 0.0046 (0.004663044615217189)  --> TIME: 2025-02-27 01:37:01 -- STEP: 13280/36938 -- GLOBAL_STEP: 1490800 | > loss_text_ce: 0.033175572752952576 (0.030403292938857893) | > loss_mel_ce: 2.725497245788574 (nan) | > loss: 0.010947114787995815 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1373151238424241) | > loader_time: 0.0045 (0.004663007392222644)  --> TIME: 2025-02-27 01:37:11 -- STEP: 13330/36938 -- GLOBAL_STEP: 1490850 | > loss_text_ce: 0.03416186571121216 (0.030405586522906754) | > loss_mel_ce: 3.0756442546844482 (nan) | > loss: 0.012340500950813293 (nan) | > current_lr: 5e-06 | > step_time: 0.1966 (0.1373232034302625) | > loader_time: 0.0049 (0.004662978389794382)  --> TIME: 2025-02-27 01:37:21 -- STEP: 13380/36938 -- GLOBAL_STEP: 1490900 | > loss_text_ce: 0.030403180047869682 (0.030403135072959157) | > loss_mel_ce: 2.9521894454956055 (nan) | > loss: 0.011835685931146145 (nan) | > current_lr: 5e-06 | > step_time: 0.186 (0.1373455150601405) | > loader_time: 0.0049 (0.004662911489227265)  --> TIME: 2025-02-27 01:37:30 -- STEP: 13430/36938 -- GLOBAL_STEP: 1490950 | > loss_text_ce: 0.03107615001499653 (0.030405117959728594) | > loss_mel_ce: 2.2977747917175293 (nan) | > loss: 0.00924147292971611 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13731889898707852) | > loader_time: 0.0045 (0.0046626898573125)  --> TIME: 2025-02-27 01:37:39 -- STEP: 13480/36938 -- GLOBAL_STEP: 1491000 | > loss_text_ce: 0.04468699172139168 (0.030405367343121364) | > loss_mel_ce: 2.882714033126831 (nan) | > loss: 0.011616671457886696 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13732614306738608) | > loader_time: 0.0048 (0.00466257270082877)  --> TIME: 2025-02-27 01:37:48 -- STEP: 13530/36938 -- GLOBAL_STEP: 1491050 | > loss_text_ce: 0.027940087020397186 (0.03040868622025079) | > loss_mel_ce: 3.2467200756073 (nan) | > loss: 0.012994683347642422 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.13732142659883084) | > loader_time: 0.0047 (0.0046623977835938)  --> TIME: 2025-02-27 01:37:58 -- STEP: 13580/36938 -- GLOBAL_STEP: 1491100 | > loss_text_ce: 0.03297349438071251 (0.03041349093245262) | > loss_mel_ce: 2.8964946269989014 (nan) | > loss: 0.011624874547123909 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13731165706496945) | > loader_time: 0.0045 (0.004662376141864192)  --> TIME: 2025-02-27 01:38:07 -- STEP: 13630/36938 -- GLOBAL_STEP: 1491150 | > loss_text_ce: 0.030912349000573158 (0.030413551527907024) | > loss_mel_ce: 2.3765804767608643 (nan) | > loss: 0.009553543291985989 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.1372967998507444) | > loader_time: 0.0046 (0.004662282783497119)  --> TIME: 2025-02-27 01:38:16 -- STEP: 13680/36938 -- GLOBAL_STEP: 1491200 | > loss_text_ce: 0.042393460869789124 (0.03041342127219261) | > loss_mel_ce: 2.7432286739349365 (nan) | > loss: 0.011054056696593761 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13729825999304637) | > loader_time: 0.0045 (0.004662570723316138)  --> TIME: 2025-02-27 01:38:25 -- STEP: 13730/36938 -- GLOBAL_STEP: 1491250 | > loss_text_ce: 0.038367919623851776 (0.03041151916355203) | > loss_mel_ce: 1.062968373298645 (nan) | > loss: 0.004370382055640221 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13730189753377592) | > loader_time: 0.0046 (0.0046625378526047355)  --> TIME: 2025-02-27 01:38:35 -- STEP: 13780/36938 -- GLOBAL_STEP: 1491300 | > loss_text_ce: 0.03874194622039795 (0.030410925281796387) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13730248328736355) | > loader_time: 0.0046 (0.0046626303296303856)  --> TIME: 2025-02-27 01:38:44 -- STEP: 13830/36938 -- GLOBAL_STEP: 1491350 | > loss_text_ce: 0.02679840475320816 (0.030405350895369564) | > loss_mel_ce: 0.8807508945465088 (nan) | > loss: 0.00360138644464314 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13728365661954925) | > loader_time: 0.0045 (0.0046625579343011194)  --> TIME: 2025-02-27 01:38:53 -- STEP: 13880/36938 -- GLOBAL_STEP: 1491400 | > loss_text_ce: 0.03584330901503563 (0.030402177209417014) | > loss_mel_ce: 2.74407958984375 (nan) | > loss: 0.011031441390514374 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13726777607838167) | > loader_time: 0.0046 (0.004662344606877762)  --> TIME: 2025-02-27 01:39:02 -- STEP: 13930/36938 -- GLOBAL_STEP: 1491450 | > loss_text_ce: 0.022363312542438507 (0.030397122507388057) | > loss_mel_ce: 2.238189458847046 (nan) | > loss: 0.00897044874727726 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1372507508162201) | > loader_time: 0.0047 (0.004662178800162544)  --> TIME: 2025-02-27 01:39:11 -- STEP: 13980/36938 -- GLOBAL_STEP: 1491500 | > loss_text_ce: 0.03512197360396385 (0.030395350486302583) | > loss_mel_ce: 1.818784475326538 (nan) | > loss: 0.007356771733611822 (nan) | > current_lr: 5e-06 | > step_time: 0.152 (0.13723876674117363) | > loader_time: 0.0046 (0.004662013394979292)  --> TIME: 2025-02-27 01:39:21 -- STEP: 14030/36938 -- GLOBAL_STEP: 1491550 | > loss_text_ce: 0.032184600830078125 (0.03039128217352945) | > loss_mel_ce: 3.279505729675293 (nan) | > loss: 0.013141629286110401 (nan) | > current_lr: 5e-06 | > step_time: 0.1853 (0.13724709955692707) | > loader_time: 0.0048 (0.004661918026332095)  --> TIME: 2025-02-27 01:39:30 -- STEP: 14080/36938 -- GLOBAL_STEP: 1491600 | > loss_text_ce: 0.025382529944181442 (0.030390821930334568) | > loss_mel_ce: 1.6319316625595093 (nan) | > loss: 0.006576643791049719 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.1372324553572327) | > loader_time: 0.0048 (0.004662065648219816)  --> TIME: 2025-02-27 01:39:39 -- STEP: 14130/36938 -- GLOBAL_STEP: 1491650 | > loss_text_ce: 0.03259401395916939 (0.03038612612027487) | > loss_mel_ce: 2.3502981662750244 (nan) | > loss: 0.009455922059714794 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13721106971989186) | > loader_time: 0.0046 (0.004661900931356326)  --> TIME: 2025-02-27 01:39:48 -- STEP: 14180/36938 -- GLOBAL_STEP: 1491700 | > loss_text_ce: 0.026194032281637192 (0.03039014484113993) | > loss_mel_ce: 2.7377002239227295 (nan) | > loss: 0.01096783485263586 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13719747769311483) | > loader_time: 0.0046 (0.004661843672457818)  --> TIME: 2025-02-27 01:39:57 -- STEP: 14230/36938 -- GLOBAL_STEP: 1491750 | > loss_text_ce: 0.02781001664698124 (0.030392600361484234) | > loss_mel_ce: 0.19366455078125 (nan) | > loss: 0.0008788674022071064 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13718524823621145) | > loader_time: 0.0045 (0.004661887058973829)  --> TIME: 2025-02-27 01:40:07 -- STEP: 14280/36938 -- GLOBAL_STEP: 1491800 | > loss_text_ce: 0.03393733128905296 (0.030389987426080813) | > loss_mel_ce: 3.0185933113098145 (nan) | > loss: 0.012113217264413834 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.13720414421471663) | > loader_time: 0.0048 (0.00466190516447822)  --> TIME: 2025-02-27 01:40:16 -- STEP: 14330/36938 -- GLOBAL_STEP: 1491850 | > loss_text_ce: 0.03322875499725342 (0.030390559279815457) | > loss_mel_ce: 2.5613460540771484 (nan) | > loss: 0.010295933112502098 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13721307074366557) | > loader_time: 0.0046 (0.004661808176965721)  --> TIME: 2025-02-27 01:40:26 -- STEP: 14380/36938 -- GLOBAL_STEP: 1491900 | > loss_text_ce: 0.0342777855694294 (0.030387340541585744) | > loss_mel_ce: 2.585765838623047 (nan) | > loss: 0.01039699837565422 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.1372053387267873) | > loader_time: 0.0045 (0.004661724166180408)  --> TIME: 2025-02-27 01:40:35 -- STEP: 14430/36938 -- GLOBAL_STEP: 1491950 | > loss_text_ce: 0.037087541073560715 (0.030384894985639774) | > loss_mel_ce: 2.44857120513916 (nan) | > loss: 0.009863724932074547 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13721608530706866) | > loader_time: 0.0047 (0.0046617889933014585)  --> TIME: 2025-02-27 01:40:44 -- STEP: 14480/36938 -- GLOBAL_STEP: 1492000 | > loss_text_ce: 0.027974002063274384 (0.030381583815565303) | > loss_mel_ce: 0.0079345703125 (nan) | > loss: 0.0001424943475285545 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13721259121078247) | > loader_time: 0.0045 (0.004661671002266829)  --> TIME: 2025-02-27 01:40:54 -- STEP: 14530/36938 -- GLOBAL_STEP: 1492050 | > loss_text_ce: 0.02401461824774742 (0.03038137550959305) | > loss_mel_ce: 3.0819711685180664 (nan) | > loss: 0.012325341813266277 (nan) | > current_lr: 5e-06 | > step_time: 0.1856 (0.1372105658341514) | > loader_time: 0.0048 (0.004661569690507455)  --> TIME: 2025-02-27 01:41:03 -- STEP: 14580/36938 -- GLOBAL_STEP: 1492100 | > loss_text_ce: 0.042693715542554855 (0.030384149720517985) | > loss_mel_ce: 2.675450325012207 (nan) | > loss: 0.010786286555230618 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13720622790366788) | > loader_time: 0.0045 (0.00466178778267693)  --> TIME: 2025-02-27 01:41:12 -- STEP: 14630/36938 -- GLOBAL_STEP: 1492150 | > loss_text_ce: 0.03826563432812691 (0.030384070407669227) | > loss_mel_ce: 2.488327741622925 (nan) | > loss: 0.010026165284216404 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.13719672537469338) | > loader_time: 0.0048 (0.004661485762605922)  --> TIME: 2025-02-27 01:41:22 -- STEP: 14680/36938 -- GLOBAL_STEP: 1492200 | > loss_text_ce: 0.0300611425191164 (0.030384239517387036) | > loss_mel_ce: 3.3774845600128174 (nan) | > loss: 0.013522007502615452 (nan) | > current_lr: 5e-06 | > step_time: 0.1816 (0.13720639616007468) | > loader_time: 0.0048 (0.00466134652129962)  --> TIME: 2025-02-27 01:41:31 -- STEP: 14730/36938 -- GLOBAL_STEP: 1492250 | > loss_text_ce: 0.025740377604961395 (0.030387236490164095) | > loss_mel_ce: 2.9092304706573486 (nan) | > loss: 0.011646710336208344 (nan) | > current_lr: 5e-06 | > step_time: 0.15 (0.13718700460730573) | > loader_time: 0.0046 (0.0046611692334063235)  --> TIME: 2025-02-27 01:41:40 -- STEP: 14780/36938 -- GLOBAL_STEP: 1492300 | > loss_text_ce: 0.02804330363869667 (0.03038764778258094) | > loss_mel_ce: 2.616936206817627 (nan) | > loss: 0.01049595046788454 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.1371712602201119) | > loader_time: 0.0048 (0.004661167506758686)  --> TIME: 2025-02-27 01:41:49 -- STEP: 14830/36938 -- GLOBAL_STEP: 1492350 | > loss_text_ce: 0.045931410044431686 (0.030387133030240956) | > loss_mel_ce: 0.002471923828125 (nan) | > loss: 0.0001920767390402034 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13716945218943566) | > loader_time: 0.0046 (0.004661477906271497)  --> TIME: 2025-02-27 01:41:58 -- STEP: 14880/36938 -- GLOBAL_STEP: 1492400 | > loss_text_ce: 0.030965201556682587 (0.030391420366684397) | > loss_mel_ce: 2.7542686462402344 (nan) | > loss: 0.011052515357732773 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13715582025307593) | > loader_time: 0.0046 (0.004661386900691589)  --> TIME: 2025-02-27 01:42:08 -- STEP: 14930/36938 -- GLOBAL_STEP: 1492450 | > loss_text_ce: 0.025165796279907227 (0.03039181456646625) | > loss_mel_ce: 2.209958076477051 (nan) | > loss: 0.008869539946317673 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.1371544337001181) | > loader_time: 0.0046 (0.004661340355793276)  --> TIME: 2025-02-27 01:42:17 -- STEP: 14980/36938 -- GLOBAL_STEP: 1492500 | > loss_text_ce: 0.028269194066524506 (0.030393518710065032) | > loss_mel_ce: 2.987412452697754 (nan) | > loss: 0.011966992169618607 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.13713980946585513) | > loader_time: 0.0048 (0.004661193820281099)  --> TIME: 2025-02-27 01:42:26 -- STEP: 15030/36938 -- GLOBAL_STEP: 1492550 | > loss_text_ce: 0.035478878766298294 (0.03039259752230774) | > loss_mel_ce: 2.922900438308716 (nan) | > loss: 0.011739601381123066 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.13712989890567592) | > loader_time: 0.0048 (0.004661045309233028)  --> TIME: 2025-02-27 01:42:35 -- STEP: 15080/36938 -- GLOBAL_STEP: 1492600 | > loss_text_ce: 0.03874863684177399 (0.030389825283859747) | > loss_mel_ce: 2.69036865234375 (nan) | > loss: 0.010829831473529339 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13711928957653413) | > loader_time: 0.0045 (0.004660885498441512)  --> TIME: 2025-02-27 01:42:44 -- STEP: 15130/36938 -- GLOBAL_STEP: 1492650 | > loss_text_ce: 0.03155136853456497 (0.03039037936541909) | > loss_mel_ce: 3.0290579795837402 (nan) | > loss: 0.012145276181399822 (nan) | > current_lr: 5e-06 | > step_time: 0.1621 (0.13711599080829015) | > loader_time: 0.0047 (0.004660738373307004)  --> TIME: 2025-02-27 01:42:54 -- STEP: 15180/36938 -- GLOBAL_STEP: 1492700 | > loss_text_ce: 0.03388513624668121 (0.03039096071141279) | > loss_mel_ce: 2.122138738632202 (nan) | > loss: 0.008555651642382145 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13713211435260309) | > loader_time: 0.0047 (0.004660690459023993)  --> TIME: 2025-02-27 01:43:03 -- STEP: 15230/36938 -- GLOBAL_STEP: 1492750 | > loss_text_ce: 0.026438551023602486 (0.030391397352898468) | > loss_mel_ce: 2.136901378631592 (nan) | > loss: 0.008584682829678059 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13711764439891647) | > loader_time: 0.0045 (0.00466045577198914)  --> TIME: 2025-02-27 01:43:12 -- STEP: 15280/36938 -- GLOBAL_STEP: 1492800 | > loss_text_ce: 0.03379923850297928 (0.030392165110033263) | > loss_mel_ce: 2.710104465484619 (nan) | > loss: 0.010888507589697838 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13709744324234882) | > loader_time: 0.0046 (0.004660264063255962)  --> TIME: 2025-02-27 01:43:22 -- STEP: 15330/36938 -- GLOBAL_STEP: 1492850 | > loss_text_ce: 0.031052395701408386 (0.03039305012625145) | > loss_mel_ce: 2.6591761112213135 (nan) | > loss: 0.01067551039159298 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13710248988202053) | > loader_time: 0.0045 (0.004660141600243668)  --> TIME: 2025-02-27 01:43:31 -- STEP: 15380/36938 -- GLOBAL_STEP: 1492900 | > loss_text_ce: 0.030115043744444847 (0.030393763440382575) | > loss_mel_ce: 2.45192813873291 (nan) | > loss: 0.00984937883913517 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13709404755320867) | > loader_time: 0.0046 (0.004659991658090473)  --> TIME: 2025-02-27 01:43:40 -- STEP: 15430/36938 -- GLOBAL_STEP: 1492950 | > loss_text_ce: 0.02110850252211094 (0.030394828872699634) | > loss_mel_ce: 2.4236385822296143 (nan) | > loss: 0.009701377712190151 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.13709639093741569) | > loader_time: 0.0047 (0.004660283661630363)  --> TIME: 2025-02-27 01:43:50 -- STEP: 15480/36938 -- GLOBAL_STEP: 1493000 | > loss_text_ce: 0.031098760664463043 (0.030395824017240215) | > loss_mel_ce: 2.4477179050445557 (nan) | > loss: 0.009836575016379356 (nan) | > current_lr: 5e-06 | > step_time: 0.1694 (0.13709854095481275) | > loader_time: 0.0046 (0.004660534935712211)  --> TIME: 2025-02-27 01:43:59 -- STEP: 15530/36938 -- GLOBAL_STEP: 1493050 | > loss_text_ce: 0.03472217917442322 (0.03040090479866432) | > loss_mel_ce: 2.5649573802948 (nan) | > loss: 0.010316189378499985 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1370950966286655) | > loader_time: 0.0045 (0.004660472836098373)  --> TIME: 2025-02-27 01:44:08 -- STEP: 15580/36938 -- GLOBAL_STEP: 1493100 | > loss_text_ce: 0.029283858835697174 (0.030400843652227075) | > loss_mel_ce: 3.006866455078125 (nan) | > loss: 0.012048215605318546 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13710081721148032) | > loader_time: 0.0045 (0.004660491566909)  --> TIME: 2025-02-27 01:44:18 -- STEP: 15630/36938 -- GLOBAL_STEP: 1493150 | > loss_text_ce: 0.03642594441771507 (0.03040265359144755) | > loss_mel_ce: 2.8217759132385254 (nan) | > loss: 0.011342070996761322 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13709864662155877) | > loader_time: 0.0046 (0.004660371870698655)  --> TIME: 2025-02-27 01:44:27 -- STEP: 15680/36938 -- GLOBAL_STEP: 1493200 | > loss_text_ce: 0.0314229391515255 (0.030403540721216604) | > loss_mel_ce: 2.264742851257324 (nan) | > loss: 0.009111769497394562 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13711248896255895) | > loader_time: 0.0046 (0.004660474265716526)  --> TIME: 2025-02-27 01:44:37 -- STEP: 15730/36938 -- GLOBAL_STEP: 1493250 | > loss_text_ce: 0.03243662044405937 (0.030407771088676705) | > loss_mel_ce: 2.9844460487365723 (nan) | > loss: 0.01197175681591034 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1371128076212628) | > loader_time: 0.0046 (0.004660537208028468)  --> TIME: 2025-02-27 01:44:46 -- STEP: 15780/36938 -- GLOBAL_STEP: 1493300 | > loss_text_ce: 0.0419747531414032 (0.03040893551836283) | > loss_mel_ce: 2.8997721672058105 (nan) | > loss: 0.011673599481582642 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13713028089477494) | > loader_time: 0.0046 (0.0046608106416139135)  --> TIME: 2025-02-27 01:44:56 -- STEP: 15830/36938 -- GLOBAL_STEP: 1493350 | > loss_text_ce: 0.026570625603199005 (0.03040853243768897) | > loss_mel_ce: 1.5286946296691895 (nan) | > loss: 0.006171687971800566 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13711761892105567) | > loader_time: 0.0046 (0.004661005776815614)  --> TIME: 2025-02-27 01:45:05 -- STEP: 15880/36938 -- GLOBAL_STEP: 1493400 | > loss_text_ce: 0.035445213317871094 (0.030416049458284713) | > loss_mel_ce: 2.7024521827697754 (nan) | > loss: 0.010864673182368279 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.13712119683811094) | > loader_time: 0.0048 (0.004660975452634502)  --> TIME: 2025-02-27 01:45:15 -- STEP: 15930/36938 -- GLOBAL_STEP: 1493450 | > loss_text_ce: 0.052733492106199265 (0.030418325101486673) | > loss_mel_ce: 3.1875 (nan) | > loss: 0.012858069501817226 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13713269064580688) | > loader_time: 0.0045 (0.004660898218795492)  --> TIME: 2025-02-27 01:45:24 -- STEP: 15980/36938 -- GLOBAL_STEP: 1493500 | > loss_text_ce: 0.022858738899230957 (0.030415182748957482) | > loss_mel_ce: 1.774658203125 (nan) | > loss: 0.007133004255592823 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13714600294194432) | > loader_time: 0.0045 (0.004660905183331403)  --> TIME: 2025-02-27 01:45:34 -- STEP: 16030/36938 -- GLOBAL_STEP: 1493550 | > loss_text_ce: 0.033659208565950394 (0.030414119839850696) | > loss_mel_ce: 2.606759548187256 (nan) | > loss: 0.010477853007614613 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.13716301152056293) | > loader_time: 0.0047 (0.0046607708231925496)  --> TIME: 2025-02-27 01:45:43 -- STEP: 16080/36938 -- GLOBAL_STEP: 1493600 | > loss_text_ce: 0.026776978746056557 (0.030419022058263016) | > loss_mel_ce: 3.1657495498657227 (nan) | > loss: 0.012668756768107414 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13717074400156992) | > loader_time: 0.0046 (0.004660879779810941)  --> TIME: 2025-02-27 01:45:53 -- STEP: 16130/36938 -- GLOBAL_STEP: 1493650 | > loss_text_ce: 0.03032027930021286 (0.030418412334511428) | > loss_mel_ce: 2.425161600112915 (nan) | > loss: 0.009743975475430489 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.1371841301034133) | > loader_time: 0.0045 (0.004660880011353384)  --> TIME: 2025-02-27 01:46:02 -- STEP: 16180/36938 -- GLOBAL_STEP: 1493700 | > loss_text_ce: 0.04226604476571083 (0.030416287699448367) | > loss_mel_ce: 0.0693359375 (nan) | > loss: 0.0004428650136105716 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13717535368739117) | > loader_time: 0.0046 (0.004660793170056633)  --> TIME: 2025-02-27 01:46:11 -- STEP: 16230/36938 -- GLOBAL_STEP: 1493750 | > loss_text_ce: 0.030223721638321877 (0.0304181019526815) | > loss_mel_ce: 1.5455793142318726 (nan) | > loss: 0.006253187078982592 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1371658914998524) | > loader_time: 0.0046 (0.004660726445586474)  --> TIME: 2025-02-27 01:46:21 -- STEP: 16280/36938 -- GLOBAL_STEP: 1493800 | > loss_text_ce: 0.03920896351337433 (0.030419701448194354) | > loss_mel_ce: 3.017878532409668 (nan) | > loss: 0.012131299823522568 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13716829471857706) | > loader_time: 0.0047 (0.004660794717380995)  --> TIME: 2025-02-27 01:46:30 -- STEP: 16330/36938 -- GLOBAL_STEP: 1493850 | > loss_text_ce: 0.0263996422290802 (0.030421678190229628) | > loss_mel_ce: 2.9637155532836914 (nan) | > loss: 0.01186553668230772 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.13716794545126446) | > loader_time: 0.0046 (0.0046607474498199936)  --> TIME: 2025-02-27 01:46:40 -- STEP: 16380/36938 -- GLOBAL_STEP: 1493900 | > loss_text_ce: 0.02553650550544262 (0.030422439161427676) | > loss_mel_ce: 2.9931640625 (nan) | > loss: 0.011978971771895885 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.1371706277896207) | > loader_time: 0.0048 (0.0046606542135478645)  --> TIME: 2025-02-27 01:46:49 -- STEP: 16430/36938 -- GLOBAL_STEP: 1493950 | > loss_text_ce: 0.03147449716925621 (0.03042429170658561) | > loss_mel_ce: 2.6414527893066406 (nan) | > loss: 0.010606855154037476 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.13715889655593355) | > loader_time: 0.0046 (0.0046607199342092774)  --> TIME: 2025-02-27 01:46:58 -- STEP: 16480/36938 -- GLOBAL_STEP: 1494000 | > loss_text_ce: 0.031773533672094345 (0.03042622470261316) | > loss_mel_ce: 2.350595474243164 (nan) | > loss: 0.009453846141695976 (nan) | > current_lr: 5e-06 | > step_time: 0.1794 (0.13715156500490072) | > loader_time: 0.0048 (0.004660611288640117)  --> TIME: 2025-02-27 01:47:07 -- STEP: 16530/36938 -- GLOBAL_STEP: 1494050 | > loss_text_ce: 0.03462637960910797 (0.030420195684203153) | > loss_mel_ce: 2.828543186187744 (nan) | > loss: 0.011361785233020782 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13714658586602568) | > loader_time: 0.0047 (0.00466054266175573)  --> TIME: 2025-02-27 01:47:17 -- STEP: 16580/36938 -- GLOBAL_STEP: 1494100 | > loss_text_ce: 0.02525246888399124 (0.030422288928078696) | > loss_mel_ce: 0.3422444760799408 (nan) | > loss: 0.0014583212323486805 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13715475911254438) | > loader_time: 0.0045 (0.0046611221477982625)  --> TIME: 2025-02-27 01:47:26 -- STEP: 16630/36938 -- GLOBAL_STEP: 1494150 | > loss_text_ce: 0.027596527710556984 (0.03042286915782451) | > loss_mel_ce: 3.0551774501800537 (nan) | > loss: 0.012233230285346508 (nan) | > current_lr: 5e-06 | > step_time: 0.1774 (0.13716599486781791) | > loader_time: 0.0048 (0.004661082303889614)  --> TIME: 2025-02-27 01:47:36 -- STEP: 16680/36938 -- GLOBAL_STEP: 1494200 | > loss_text_ce: 0.024485846981406212 (0.030424138249933258) | > loss_mel_ce: 2.7130954265594482 (nan) | > loss: 0.010863418690860271 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13717161152574472) | > loader_time: 0.0047 (0.004661075602904225)  --> TIME: 2025-02-27 01:47:45 -- STEP: 16730/36938 -- GLOBAL_STEP: 1494250 | > loss_text_ce: 0.0384613499045372 (0.030418893572658926) | > loss_mel_ce: 2.641448736190796 (nan) | > loss: 0.01063456479460001 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.13718329936675763) | > loader_time: 0.0047 (0.0046611226253429824)  --> TIME: 2025-02-27 01:47:55 -- STEP: 16780/36938 -- GLOBAL_STEP: 1494300 | > loss_text_ce: 0.027750790119171143 (0.030415164239852247) | > loss_mel_ce: 2.879333019256592 (nan) | > loss: 0.011536046862602234 (nan) | > current_lr: 5e-06 | > step_time: 0.1842 (0.13718801044877965) | > loader_time: 0.0049 (0.004661301066111052)  --> TIME: 2025-02-27 01:48:04 -- STEP: 16830/36938 -- GLOBAL_STEP: 1494350 | > loss_text_ce: 0.033884331583976746 (0.030414338385580036) | > loss_mel_ce: 2.9250898361206055 (nan) | > loss: 0.011741961352527142 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13718700220524058) | > loader_time: 0.0045 (0.004661281648591676)  --> TIME: 2025-02-27 01:48:13 -- STEP: 16880/36938 -- GLOBAL_STEP: 1494400 | > loss_text_ce: 0.03345995023846626 (0.030417269805773996) | > loss_mel_ce: 2.626699924468994 (nan) | > loss: 0.010556190274655819 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13718178996825212) | > loader_time: 0.0047 (0.004661155580344369)  --> TIME: 2025-02-27 01:48:23 -- STEP: 16930/36938 -- GLOBAL_STEP: 1494450 | > loss_text_ce: 0.03299342468380928 (0.030412900948465726) | > loss_mel_ce: 3.0942935943603516 (nan) | > loss: 0.012409869581460953 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13717602134384746) | > loader_time: 0.0046 (0.00466099417709559)  --> TIME: 2025-02-27 01:48:32 -- STEP: 16980/36938 -- GLOBAL_STEP: 1494500 | > loss_text_ce: 0.02531101368367672 (0.030414953434733438) | > loss_mel_ce: 2.338505506515503 (nan) | > loss: 0.009380225092172623 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1371800186917706) | > loader_time: 0.0047 (0.004661105280068526)  --> TIME: 2025-02-27 01:48:41 -- STEP: 17030/36938 -- GLOBAL_STEP: 1494550 | > loss_text_ce: 0.03176252916455269 (0.03041393521211219) | > loss_mel_ce: 3.2394301891326904 (nan) | > loss: 0.012980924919247627 (nan) | > current_lr: 5e-06 | > step_time: 0.1687 (0.13716967854861126) | > loader_time: 0.0048 (0.004661064167549114)  --> TIME: 2025-02-27 01:48:51 -- STEP: 17080/36938 -- GLOBAL_STEP: 1494600 | > loss_text_ce: 0.025927182286977768 (0.030410573170580345) | > loss_mel_ce: 2.6895806789398193 (nan) | > loss: 0.010775825008749962 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.13716971254460542) | > loader_time: 0.0048 (0.004661406998891194)  --> TIME: 2025-02-27 01:49:00 -- STEP: 17130/36938 -- GLOBAL_STEP: 1494650 | > loss_text_ce: 0.02894464135169983 (0.030410604238084976) | > loss_mel_ce: 1.6377016305923462 (nan) | > loss: 0.006613675970584154 (nan) | > current_lr: 5e-06 | > step_time: 0.1077 (0.13715887334147855) | > loader_time: 0.0044 (0.004661220969752974)  --> TIME: 2025-02-27 01:49:09 -- STEP: 17180/36938 -- GLOBAL_STEP: 1494700 | > loss_text_ce: 0.03141436725854874 (0.030411482072105724) | > loss_mel_ce: 2.425028085708618 (nan) | > loss: 0.00974778737872839 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.1371582971879306) | > loader_time: 0.0047 (0.004661143436698346)  --> TIME: 2025-02-27 01:49:18 -- STEP: 17230/36938 -- GLOBAL_STEP: 1494750 | > loss_text_ce: 0.03361217677593231 (0.03040963838455817) | > loss_mel_ce: 0.0419921875 (nan) | > loss: 0.00030001733102835715 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1371506292461599) | > loader_time: 0.0045 (0.004661088341277359)  --> TIME: 2025-02-27 01:49:28 -- STEP: 17280/36938 -- GLOBAL_STEP: 1494800 | > loss_text_ce: 0.030505217611789703 (0.030413758878394558) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13714859619460845) | > loader_time: 0.0045 (0.004660972249176774)  --> TIME: 2025-02-27 01:49:37 -- STEP: 17330/36938 -- GLOBAL_STEP: 1494850 | > loss_text_ce: 0.024716738611459732 (0.030413176672129026) | > loss_mel_ce: 1.6688283681869507 (nan) | > loss: 0.006720417644828558 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13714396802249684) | > loader_time: 0.0044 (0.004660860458963955)  --> TIME: 2025-02-27 01:49:46 -- STEP: 17380/36938 -- GLOBAL_STEP: 1494900 | > loss_text_ce: 0.023744141682982445 (0.030415142979285326) | > loss_mel_ce: 0.009108816273510456 (nan) | > loss: 0.00013036889140494168 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13714799393993093) | > loader_time: 0.0046 (0.004660974780764219)  --> TIME: 2025-02-27 01:49:56 -- STEP: 17430/36938 -- GLOBAL_STEP: 1494950 | > loss_text_ce: 0.03167343884706497 (0.030411349674280705) | > loss_mel_ce: 2.629594087600708 (nan) | > loss: 0.010560586117208004 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13713635066016658) | > loader_time: 0.0046 (0.004660913920990714)  --> TIME: 2025-02-27 01:50:05 -- STEP: 17480/36938 -- GLOBAL_STEP: 1495000 | > loss_text_ce: 0.030347775667905807 (0.030415075719947938) | > loss_mel_ce: 2.830965757369995 (nan) | > loss: 0.011354419402778149 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.13712654430206203) | > loader_time: 0.0046 (0.0046609478629699354)  --> TIME: 2025-02-27 01:50:14 -- STEP: 17530/36938 -- GLOBAL_STEP: 1495050 | > loss_text_ce: 0.027905298396945 (0.030419144379261923) | > loss_mel_ce: 2.7711775302886963 (nan) | > loss: 0.011107471771538258 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13712358087114354) | > loader_time: 0.0045 (0.004661001155394123)  --> TIME: 2025-02-27 01:50:23 -- STEP: 17580/36938 -- GLOBAL_STEP: 1495100 | > loss_text_ce: 0.025966528803110123 (0.030420640286347438) | > loss_mel_ce: 2.6728625297546387 (nan) | > loss: 0.010709640569984913 (nan) | > current_lr: 5e-06 | > step_time: 0.1537 (0.13711441402684954) | > loader_time: 0.0047 (0.00466101596786708)  --> TIME: 2025-02-27 01:50:33 -- STEP: 17630/36938 -- GLOBAL_STEP: 1495150 | > loss_text_ce: 0.03006572090089321 (0.030419840920643624) | > loss_mel_ce: 2.2198686599731445 (nan) | > loss: 0.008928311988711357 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.13712470390819764) | > loader_time: 0.0046 (0.004661118166053163)  --> TIME: 2025-02-27 01:50:42 -- STEP: 17680/36938 -- GLOBAL_STEP: 1495200 | > loss_text_ce: 0.03321543708443642 (0.030419592030069314) | > loss_mel_ce: 2.108773708343506 (nan) | > loss: 0.008499957621097565 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13710697353965134) | > loader_time: 0.0047 (0.0046610744830170485)  --> TIME: 2025-02-27 01:50:51 -- STEP: 17730/36938 -- GLOBAL_STEP: 1495250 | > loss_text_ce: 0.03581777587532997 (0.03042275611916846) | > loss_mel_ce: 2.07377552986145 (nan) | > loss: 0.008371402509510517 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13709350767205658) | > loader_time: 0.0045 (0.004661054847876514)  --> TIME: 2025-02-27 01:51:00 -- STEP: 17780/36938 -- GLOBAL_STEP: 1495300 | > loss_text_ce: 0.035647738724946976 (0.030429458951957825) | > loss_mel_ce: 3.213648557662964 (nan) | > loss: 0.012894033454358578 (nan) | > current_lr: 5e-06 | > step_time: 0.1799 (0.137100652259167) | > loader_time: 0.0049 (0.00466093760909864)  --> TIME: 2025-02-27 01:51:10 -- STEP: 17830/36938 -- GLOBAL_STEP: 1495350 | > loss_text_ce: 0.02372889406979084 (0.030429911614535142) | > loss_mel_ce: 2.645522356033325 (nan) | > loss: 0.010592266917228699 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13709908352764807) | > loader_time: 0.0046 (0.004660872535898131)  --> TIME: 2025-02-27 01:51:19 -- STEP: 17880/36938 -- GLOBAL_STEP: 1495400 | > loss_text_ce: 0.03310546651482582 (0.030430090274808665) | > loss_mel_ce: 2.833911180496216 (nan) | > loss: 0.011377050541341305 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.1371079068855963) | > loader_time: 0.0046 (0.004660833348630537)  --> TIME: 2025-02-27 01:51:29 -- STEP: 17930/36938 -- GLOBAL_STEP: 1495450 | > loss_text_ce: 0.032372236251831055 (0.030430496818262917) | > loss_mel_ce: 2.6755692958831787 (nan) | > loss: 0.010745800100266933 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13711562773666885) | > loader_time: 0.0046 (0.004660739941232654)  --> TIME: 2025-02-27 01:51:38 -- STEP: 17980/36938 -- GLOBAL_STEP: 1495500 | > loss_text_ce: 0.0262752715498209 (0.03042677205526793) | > loss_mel_ce: 2.8285462856292725 (nan) | > loss: 0.011328657157719135 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.13712149130489312) | > loader_time: 0.0047 (0.004660674966084429)  --> TIME: 2025-02-27 01:51:48 -- STEP: 18030/36938 -- GLOBAL_STEP: 1495550 | > loss_text_ce: 0.022614959627389908 (0.030428451731830994) | > loss_mel_ce: 2.2518832683563232 (nan) | > loss: 0.009025787003338337 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.13713130166778012) | > loader_time: 0.0046 (0.0046606358725431035)  --> TIME: 2025-02-27 01:51:57 -- STEP: 18080/36938 -- GLOBAL_STEP: 1495600 | > loss_text_ce: 0.029521111398935318 (0.030428364238391663) | > loss_mel_ce: 2.67478609085083 (nan) | > loss: 0.010731378570199013 (nan) | > current_lr: 5e-06 | > step_time: 0.1618 (0.13713128668280955) | > loader_time: 0.0047 (0.004660523003709013)  --> TIME: 2025-02-27 01:52:06 -- STEP: 18130/36938 -- GLOBAL_STEP: 1495650 | > loss_text_ce: 0.03161819279193878 (0.03042984127079358) | > loss_mel_ce: 2.8304240703582764 (nan) | > loss: 0.01135731115937233 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13712159802226015) | > loader_time: 0.0046 (0.004660411230845817)  --> TIME: 2025-02-27 01:52:15 -- STEP: 18180/36938 -- GLOBAL_STEP: 1495700 | > loss_text_ce: 0.03172893077135086 (0.030428834199138003) | > loss_mel_ce: 2.631559133529663 (nan) | > loss: 0.010568603873252869 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13711884492563722) | > loader_time: 0.0046 (0.004660329816102736)  --> TIME: 2025-02-27 01:52:25 -- STEP: 18230/36938 -- GLOBAL_STEP: 1495750 | > loss_text_ce: 0.02957315556704998 (0.030423439719223275) | > loss_mel_ce: 2.41660213470459 (nan) | > loss: 0.009707045741379261 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13712903274917107) | > loader_time: 0.0045 (0.0046602778819257494)  --> TIME: 2025-02-27 01:52:34 -- STEP: 18280/36938 -- GLOBAL_STEP: 1495800 | > loss_text_ce: 0.03756552189588547 (0.030421164630085873) | > loss_mel_ce: 2.9121906757354736 (nan) | > loss: 0.01170538179576397 (nan) | > current_lr: 5e-06 | > step_time: 0.1927 (0.13713833885328655) | > loader_time: 0.005 (0.004660273211268436)  --> TIME: 2025-02-27 01:52:44 -- STEP: 18330/36938 -- GLOBAL_STEP: 1495850 | > loss_text_ce: 0.031011035665869713 (0.030421072583512464) | > loss_mel_ce: 2.7461113929748535 (nan) | > loss: 0.011020327918231487 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.1371291495904349) | > loader_time: 0.0046 (0.004660115187374947)  --> TIME: 2025-02-27 01:52:53 -- STEP: 18380/36938 -- GLOBAL_STEP: 1495900 | > loss_text_ce: 0.033409491181373596 (0.03042209460959953) | > loss_mel_ce: 3.3142454624176025 (nan) | > loss: 0.013284346088767052 (nan) | > current_lr: 5e-06 | > step_time: 0.1767 (0.13713322666446467) | > loader_time: 0.0048 (0.00465995789352518)  --> TIME: 2025-02-27 01:53:03 -- STEP: 18430/36938 -- GLOBAL_STEP: 1495950 | > loss_text_ce: 0.01592613384127617 (0.030426005964110315) | > loss_mel_ce: 1.4178000688552856 (nan) | > loss: 0.005689390003681183 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13713532491540248) | > loader_time: 0.0045 (0.004659852176920862)  --> TIME: 2025-02-27 01:53:12 -- STEP: 18480/36938 -- GLOBAL_STEP: 1496000 | > loss_text_ce: 0.03147777542471886 (0.030427145771354688) | > loss_mel_ce: 2.4138102531433105 (nan) | > loss: 0.00970352441072464 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.1371410401094542) | > loader_time: 0.0046 (0.004659751160836336)  --> TIME: 2025-02-27 01:53:22 -- STEP: 18530/36938 -- GLOBAL_STEP: 1496050 | > loss_text_ce: 0.034806787967681885 (0.030428697707290328) | > loss_mel_ce: 2.873614549636841 (nan) | > loss: 0.011541355401277542 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.1371453932923249) | > loader_time: 0.0046 (0.004659576449468865)  --> TIME: 2025-02-27 01:53:31 -- STEP: 18580/36938 -- GLOBAL_STEP: 1496100 | > loss_text_ce: 0.030593667179346085 (0.030426535152066277) | > loss_mel_ce: 2.8606224060058594 (nan) | > loss: 0.011473080143332481 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13714850059238395) | > loader_time: 0.0045 (0.004659390770323963)  --> TIME: 2025-02-27 01:53:40 -- STEP: 18630/36938 -- GLOBAL_STEP: 1496150 | > loss_text_ce: 0.03422538936138153 (0.030427371986261338) | > loss_mel_ce: 3.311535120010376 (nan) | > loss: 0.013276828452944756 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13714022279361723) | > loader_time: 0.0045 (0.00465915516634751)  --> TIME: 2025-02-27 01:53:49 -- STEP: 18680/36938 -- GLOBAL_STEP: 1496200 | > loss_text_ce: 0.021101249381899834 (0.03042902812190709) | > loss_mel_ce: 2.5693912506103516 (nan) | > loss: 0.010279732756316662 (nan) | > current_lr: 5e-06 | > step_time: 0.1582 (0.1371253203656497) | > loader_time: 0.0047 (0.004658933063645954)  --> TIME: 2025-02-27 01:53:58 -- STEP: 18730/36938 -- GLOBAL_STEP: 1496250 | > loss_text_ce: 0.029598692432045937 (0.030427527186041622) | > loss_mel_ce: 1.9476165771484375 (nan) | > loss: 0.00784609280526638 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13711875934875914) | > loader_time: 0.0045 (0.004658699366604661)  --> TIME: 2025-02-27 01:54:07 -- STEP: 18780/36938 -- GLOBAL_STEP: 1496300 | > loss_text_ce: 0.04248154163360596 (0.030426313494445204) | > loss_mel_ce: 2.2119717597961426 (nan) | > loss: 0.008946243673563004 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13711196219705424) | > loader_time: 0.0046 (0.004658556428834154)  --> TIME: 2025-02-27 01:54:16 -- STEP: 18830/36938 -- GLOBAL_STEP: 1496350 | > loss_text_ce: 0.027882033959031105 (0.030426439620278464) | > loss_mel_ce: 2.2054855823516846 (nan) | > loss: 0.008862570859491825 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13710517283139437) | > loader_time: 0.0044 (0.004658372821291276)  --> TIME: 2025-02-27 01:54:26 -- STEP: 18880/36938 -- GLOBAL_STEP: 1496400 | > loss_text_ce: 0.02496226504445076 (0.030423189369715962) | > loss_mel_ce: 0.023532232269644737 (nan) | > loss: 0.00019243848510086536 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.1371019098592019) | > loader_time: 0.0045 (0.004658235596903306)  --> TIME: 2025-02-27 01:54:35 -- STEP: 18930/36938 -- GLOBAL_STEP: 1496450 | > loss_text_ce: 0.023410875350236893 (0.03042180889061757) | > loss_mel_ce: 3.09375 (nan) | > loss: 0.012369686737656593 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.1370839630178353) | > loader_time: 0.0044 (0.004658027357735214)  --> TIME: 2025-02-27 01:54:44 -- STEP: 18980/36938 -- GLOBAL_STEP: 1496500 | > loss_text_ce: 0.03389936685562134 (0.03042272277143504) | > loss_mel_ce: 2.865849256515503 (nan) | > loss: 0.011506939306855202 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13706920674528167) | > loader_time: 0.0046 (0.004657851066931034)  --> TIME: 2025-02-27 01:54:53 -- STEP: 19030/36938 -- GLOBAL_STEP: 1496550 | > loss_text_ce: 0.026536280289292336 (0.030426421188942687) | > loss_mel_ce: 2.8264360427856445 (nan) | > loss: 0.01132131926715374 (nan) | > current_lr: 5e-06 | > step_time: 0.1891 (0.13708435288868898) | > loader_time: 0.0048 (0.004657840753816899)  --> TIME: 2025-02-27 01:55:03 -- STEP: 19080/36938 -- GLOBAL_STEP: 1496600 | > loss_text_ce: 0.028262976557016373 (0.03042895748464008) | > loss_mel_ce: 2.47739315032959 (nan) | > loss: 0.009943081066012383 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1370893995092114) | > loader_time: 0.0046 (0.004657785922476337)  --> TIME: 2025-02-27 01:55:12 -- STEP: 19130/36938 -- GLOBAL_STEP: 1496650 | > loss_text_ce: 0.029922137036919594 (0.030429070814105185) | > loss_mel_ce: 3.0346834659576416 (nan) | > loss: 0.012161134742200375 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13708320401815507) | > loader_time: 0.0045 (0.004657648136898669)  --> TIME: 2025-02-27 01:55:22 -- STEP: 19180/36938 -- GLOBAL_STEP: 1496700 | > loss_text_ce: 0.03313811123371124 (0.030429371638826606) | > loss_mel_ce: 2.9001917839050293 (nan) | > loss: 0.011640198528766632 (nan) | > current_lr: 5e-06 | > step_time: 0.1448 (0.13709105479456357) | > loader_time: 0.0046 (0.004657563178705849)  --> TIME: 2025-02-27 01:55:31 -- STEP: 19230/36938 -- GLOBAL_STEP: 1496750 | > loss_text_ce: 0.03486040607094765 (0.030428783416440997) | > loss_mel_ce: 1.7070236206054688 (nan) | > loss: 0.00691223843023181 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13708004604325869) | > loader_time: 0.0046 (0.004657414823662042)  --> TIME: 2025-02-27 01:55:40 -- STEP: 19280/36938 -- GLOBAL_STEP: 1496800 | > loss_text_ce: 0.03956683725118637 (0.030429496317188918) | > loss_mel_ce: 2.126622200012207 (nan) | > loss: 0.008595988154411316 (nan) | > current_lr: 5e-06 | > step_time: 0.1771 (0.1370763976791604) | > loader_time: 0.0049 (0.004657276586872913)  --> TIME: 2025-02-27 01:55:50 -- STEP: 19330/36938 -- GLOBAL_STEP: 1496850 | > loss_text_ce: 0.036760520190000534 (0.03042825763253519) | > loss_mel_ce: 2.327085256576538 (nan) | > loss: 0.00938034150749445 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13708279363491255) | > loader_time: 0.0046 (0.004657195925527582)  --> TIME: 2025-02-27 01:55:59 -- STEP: 19380/36938 -- GLOBAL_STEP: 1496900 | > loss_text_ce: 0.026002582162618637 (0.030432089303583636) | > loss_mel_ce: 3.2629806995391846 (nan) | > loss: 0.013051521964371204 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13708294609136898) | > loader_time: 0.0045 (0.004657109098660815)  --> TIME: 2025-02-27 01:56:08 -- STEP: 19430/36938 -- GLOBAL_STEP: 1496950 | > loss_text_ce: 0.027821000665426254 (0.030434603965613555) | > loss_mel_ce: 2.5553040504455566 (nan) | > loss: 0.010250496678054333 (nan) | > current_lr: 5e-06 | > step_time: 0.1567 (0.13707942227635161) | > loader_time: 0.0046 (0.004657014251921023)  --> TIME: 2025-02-27 01:56:18 -- STEP: 19480/36938 -- GLOBAL_STEP: 1497000 | > loss_text_ce: 0.027905866503715515 (0.030436949253448493) | > loss_mel_ce: 2.868525981903076 (nan) | > loss: 0.011493777856230736 (nan) | > current_lr: 5e-06 | > step_time: 0.1911 (0.13708154289874347) | > loader_time: 0.0048 (0.004656879294824308)  --> TIME: 2025-02-27 01:56:27 -- STEP: 19530/36938 -- GLOBAL_STEP: 1497050 | > loss_text_ce: 0.027103744447231293 (0.03043564094921514) | > loss_mel_ce: 2.419020414352417 (nan) | > loss: 0.00970684178173542 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13707636429356626) | > loader_time: 0.0047 (0.004656694500616011)  --> TIME: 2025-02-27 01:56:36 -- STEP: 19580/36938 -- GLOBAL_STEP: 1497100 | > loss_text_ce: 0.02680877409875393 (0.030434668771575364) | > loss_mel_ce: 2.8428475856781006 (nan) | > loss: 0.01138752605766058 (nan) | > current_lr: 5e-06 | > step_time: 0.1908 (0.1370739954119453) | > loader_time: 0.0051 (0.004656740825193291)  --> TIME: 2025-02-27 01:56:45 -- STEP: 19630/36938 -- GLOBAL_STEP: 1497150 | > loss_text_ce: 0.028659634292125702 (0.03043716564996076) | > loss_mel_ce: 2.0532050132751465 (nan) | > loss: 0.008261367678642273 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.1370658154519202) | > loader_time: 0.0047 (0.004656538341461979)  --> TIME: 2025-02-27 01:56:55 -- STEP: 19680/36938 -- GLOBAL_STEP: 1497200 | > loss_text_ce: 0.03553711622953415 (0.030435272141400443) | > loss_mel_ce: 1.5881917476654053 (nan) | > loss: 0.00644336873665452 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13705390272586468) | > loader_time: 0.0045 (0.004656485231911269)  --> TIME: 2025-02-27 01:57:04 -- STEP: 19730/36938 -- GLOBAL_STEP: 1497250 | > loss_text_ce: 0.030909959226846695 (0.030435293020248163) | > loss_mel_ce: 2.844334840774536 (nan) | > loss: 0.011409702710807323 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.1370457378789622) | > loader_time: 0.0046 (0.0046564615383041935)  --> TIME: 2025-02-27 01:57:13 -- STEP: 19780/36938 -- GLOBAL_STEP: 1497300 | > loss_text_ce: 0.03208160400390625 (0.030437964681583985) | > loss_mel_ce: 1.3118865489959717 (nan) | > loss: 0.005333207082003355 (nan) | > current_lr: 5e-06 | > step_time: 0.1765 (0.13705002416614712) | > loader_time: 0.0049 (0.004656390811121749)  --> TIME: 2025-02-27 01:57:22 -- STEP: 19830/36938 -- GLOBAL_STEP: 1497350 | > loss_text_ce: 0.02763245813548565 (0.030434354546067056) | > loss_mel_ce: 2.955193042755127 (nan) | > loss: 0.011836609803140163 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13704054774026236) | > loader_time: 0.0045 (0.004656266023940781)  --> TIME: 2025-02-27 01:57:32 -- STEP: 19880/36938 -- GLOBAL_STEP: 1497400 | > loss_text_ce: 0.02576133981347084 (0.030430555190320428) | > loss_mel_ce: 2.6768665313720703 (nan) | > loss: 0.010724714957177639 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13703798243937243) | > loader_time: 0.0047 (0.004656213989679973)  --> TIME: 2025-02-27 01:57:41 -- STEP: 19930/36938 -- GLOBAL_STEP: 1497450 | > loss_text_ce: 0.03463143855333328 (0.030430962573909807) | > loss_mel_ce: 2.803189277648926 (nan) | > loss: 0.01126119401305914 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13704744545706551) | > loader_time: 0.0046 (0.004656237701764384)  --> TIME: 2025-02-27 01:57:51 -- STEP: 19980/36938 -- GLOBAL_STEP: 1497500 | > loss_text_ce: 0.03772776201367378 (0.03043462853126873) | > loss_mel_ce: 3.1721994876861572 (nan) | > loss: 0.01273780781775713 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.13704931814749446) | > loader_time: 0.0048 (0.0046562565578235505)  --> TIME: 2025-02-27 01:58:00 -- STEP: 20030/36938 -- GLOBAL_STEP: 1497550 | > loss_text_ce: 0.03188912197947502 (0.03043497302890274) | > loss_mel_ce: 1.9328155517578125 (nan) | > loss: 0.00779644725844264 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13704765720243767) | > loader_time: 0.0046 (0.004656166049521158)  --> TIME: 2025-02-27 01:58:09 -- STEP: 20080/36938 -- GLOBAL_STEP: 1497600 | > loss_text_ce: 0.030018934980034828 (0.03043275412918343) | > loss_mel_ce: 2.8121016025543213 (nan) | > loss: 0.01127825677394867 (nan) | > current_lr: 5e-06 | > step_time: 0.1868 (0.13703992141195623) | > loader_time: 0.0049 (0.004656089385192245)  --> TIME: 2025-02-27 01:58:19 -- STEP: 20130/36938 -- GLOBAL_STEP: 1497650 | > loss_text_ce: 0.024490948766469955 (0.030431441217591507) | > loss_mel_ce: 3.4171142578125 (nan) | > loss: 0.013657163828611374 (nan) | > current_lr: 5e-06 | > step_time: 0.1648 (0.13704063897102198) | > loader_time: 0.0047 (0.004656082175586936)  --> TIME: 2025-02-27 01:58:28 -- STEP: 20180/36938 -- GLOBAL_STEP: 1497700 | > loss_text_ce: 0.020033029839396477 (0.030429533068242975) | > loss_mel_ce: 2.240340232849121 (nan) | > loss: 0.008969736285507679 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13703384607351102) | > loader_time: 0.0045 (0.0046560351038121015)  --> TIME: 2025-02-27 01:58:37 -- STEP: 20230/36938 -- GLOBAL_STEP: 1497750 | > loss_text_ce: 0.02833840809762478 (0.030426504977762853) | > loss_mel_ce: 2.0267653465270996 (nan) | > loss: 0.008155174553394318 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.1370317070895483) | > loader_time: 0.0046 (0.0046560910569519685)  --> TIME: 2025-02-27 01:58:46 -- STEP: 20280/36938 -- GLOBAL_STEP: 1497800 | > loss_text_ce: 0.027276229113340378 (0.030426868727762266) | > loss_mel_ce: 0.665486216545105 (nan) | > loss: 0.002749057486653328 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13703057995209433) | > loader_time: 0.0046 (0.004655945700771486)  --> TIME: 2025-02-27 01:58:56 -- STEP: 20330/36938 -- GLOBAL_STEP: 1497850 | > loss_text_ce: 0.02871212735772133 (0.03042784257571131) | > loss_mel_ce: 3.073338747024536 (nan) | > loss: 0.0123097263276577 (nan) | > current_lr: 5e-06 | > step_time: 0.1858 (0.13703589941326047) | > loader_time: 0.0048 (0.004655815566401156)  --> TIME: 2025-02-27 01:59:05 -- STEP: 20380/36938 -- GLOBAL_STEP: 1497900 | > loss_text_ce: 0.02018587477505207 (0.030429896051854706) | > loss_mel_ce: 2.508373498916626 (nan) | > loss: 0.010033966973423958 (nan) | > current_lr: 5e-06 | > step_time: 0.1443 (0.13703354077671506) | > loader_time: 0.0046 (0.004655676980715867)  --> TIME: 2025-02-27 01:59:15 -- STEP: 20430/36938 -- GLOBAL_STEP: 1497950 | > loss_text_ce: 0.025016674771904945 (0.03042631652340661) | > loss_mel_ce: 2.3730530738830566 (nan) | > loss: 0.009516150690615177 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13703299795245144) | > loader_time: 0.0045 (0.00465554969309595)  --> TIME: 2025-02-27 01:59:24 -- STEP: 20480/36938 -- GLOBAL_STEP: 1498000 | > loss_text_ce: 0.030487900599837303 (0.030427876129283504) | > loss_mel_ce: 2.721891164779663 (nan) | > loss: 0.010922140441834927 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.13702738148858867) | > loader_time: 0.0047 (0.004655387136153893)  --> TIME: 2025-02-27 01:59:33 -- STEP: 20530/36938 -- GLOBAL_STEP: 1498050 | > loss_text_ce: 0.0374956876039505 (0.030423237802506154) | > loss_mel_ce: 1.8370569944381714 (nan) | > loss: 0.007438701577484608 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13704357632647804) | > loader_time: 0.0046 (0.004655523365204716)  --> TIME: 2025-02-27 01:59:43 -- STEP: 20580/36938 -- GLOBAL_STEP: 1498100 | > loss_text_ce: 0.031059324741363525 (0.03042239859793878) | > loss_mel_ce: 0.0019844903144985437 (nan) | > loss: 0.00013112626038491726 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13703551685149837) | > loader_time: 0.0046 (0.004655525079621529)  --> TIME: 2025-02-27 01:59:52 -- STEP: 20630/36938 -- GLOBAL_STEP: 1498150 | > loss_text_ce: 0.032677460461854935 (0.03042209435313474) | > loss_mel_ce: 3.7628676891326904 (nan) | > loss: 0.015061687678098679 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13703139243169932) | > loader_time: 0.0045 (0.0046555305301594265)  --> TIME: 2025-02-27 02:00:01 -- STEP: 20680/36938 -- GLOBAL_STEP: 1498200 | > loss_text_ce: 0.029508566483855247 (0.03042047782132323) | > loss_mel_ce: 3.0677406787872314 (nan) | > loss: 0.012290672399103642 (nan) | > current_lr: 5e-06 | > step_time: 0.1894 (0.1370312928807584) | > loader_time: 0.0049 (0.0046554615811176545)  --> TIME: 2025-02-27 02:00:11 -- STEP: 20730/36938 -- GLOBAL_STEP: 1498250 | > loss_text_ce: 0.03165338188409805 (0.030424774684675946) | > loss_mel_ce: 2.4361376762390137 (nan) | > loss: 0.009792822413146496 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13703736117998394) | > loader_time: 0.0045 (0.004655377484150015)  --> TIME: 2025-02-27 02:00:20 -- STEP: 20780/36938 -- GLOBAL_STEP: 1498300 | > loss_text_ce: 0.031032873317599297 (0.03042706386394587) | > loss_mel_ce: 3.1710729598999023 (nan) | > loss: 0.01270676963031292 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13703089788398434) | > loader_time: 0.0045 (0.004655318482778087)  --> TIME: 2025-02-27 02:00:29 -- STEP: 20830/36938 -- GLOBAL_STEP: 1498350 | > loss_text_ce: 0.037624187767505646 (0.0304257437915184) | > loss_mel_ce: 3.280144453048706 (nan) | > loss: 0.013165748678147793 (nan) | > current_lr: 5e-06 | > step_time: 0.1758 (0.13703821114033363) | > loader_time: 0.0048 (0.004655220115980294)  --> TIME: 2025-02-27 02:00:39 -- STEP: 20880/36938 -- GLOBAL_STEP: 1498400 | > loss_text_ce: 0.0286454726010561 (0.03042461709050244) | > loss_mel_ce: 1.8085283041000366 (nan) | > loss: 0.007290372624993324 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13703510619671494) | > loader_time: 0.0046 (0.0046550832146429145)  --> TIME: 2025-02-27 02:00:48 -- STEP: 20930/36938 -- GLOBAL_STEP: 1498450 | > loss_text_ce: 0.022969799116253853 (0.030423277961552975) | > loss_mel_ce: 2.0045721530914307 (nan) | > loss: 0.008045801892876625 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13703861566918202) | > loader_time: 0.0045 (0.004654963017649144)  --> TIME: 2025-02-27 02:00:57 -- STEP: 20980/36938 -- GLOBAL_STEP: 1498500 | > loss_text_ce: 0.03975052386522293 (0.030424439532880283) | > loss_mel_ce: 2.4857430458068848 (nan) | > loss: 0.010021801106631756 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1370356864992846) | > loader_time: 0.0045 (0.00465482596105571)  --> TIME: 2025-02-27 02:01:07 -- STEP: 21030/36938 -- GLOBAL_STEP: 1498550 | > loss_text_ce: 0.033689387142658234 (0.030423180022951406) | > loss_mel_ce: 2.692903995513916 (nan) | > loss: 0.010819816030561924 (nan) | > current_lr: 5e-06 | > step_time: 0.1605 (0.13704811679823847) | > loader_time: 0.0047 (0.0046547841980408644)  --> TIME: 2025-02-27 02:01:16 -- STEP: 21080/36938 -- GLOBAL_STEP: 1498600 | > loss_text_ce: 0.020935920998454094 (0.030420815651601027) | > loss_mel_ce: 2.4806694984436035 (nan) | > loss: 0.00992700643837452 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.137042283468953) | > loader_time: 0.0046 (0.004654738188242338)  --> TIME: 2025-02-27 02:01:25 -- STEP: 21130/36938 -- GLOBAL_STEP: 1498650 | > loss_text_ce: 0.03019603155553341 (0.030421409004121505) | > loss_mel_ce: 2.5220539569854736 (nan) | > loss: 0.010127976536750793 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13704314405637247) | > loader_time: 0.0045 (0.004654769710156247)  --> TIME: 2025-02-27 02:01:35 -- STEP: 21180/36938 -- GLOBAL_STEP: 1498700 | > loss_text_ce: 0.034990523010492325 (0.030422683522686525) | > loss_mel_ce: 2.8041799068450928 (nan) | > loss: 0.011266550049185753 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.1370527205768913) | > loader_time: 0.0046 (0.004654760513809979)  --> TIME: 2025-02-27 02:01:44 -- STEP: 21230/36938 -- GLOBAL_STEP: 1498750 | > loss_text_ce: 0.03025265596807003 (0.030424719472322306) | > loss_mel_ce: 2.644266366958618 (nan) | > loss: 0.010613171383738518 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.13705405289420494) | > loader_time: 0.0047 (0.004654669188914086)  --> TIME: 2025-02-27 02:01:54 -- STEP: 21280/36938 -- GLOBAL_STEP: 1498800 | > loss_text_ce: 0.030150162056088448 (0.030427165357925567) | > loss_mel_ce: 2.364109992980957 (nan) | > loss: 0.009501032531261444 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.1370548266105194) | > loader_time: 0.0045 (0.004654624699649966)  --> TIME: 2025-02-27 02:02:03 -- STEP: 21330/36938 -- GLOBAL_STEP: 1498850 | > loss_text_ce: 0.03436587378382683 (0.03042477520601453) | > loss_mel_ce: 2.5000195503234863 (nan) | > loss: 0.010057086125016212 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.1370487655097765) | > loader_time: 0.0045 (0.004654533428258322)  --> TIME: 2025-02-27 02:02:12 -- STEP: 21380/36938 -- GLOBAL_STEP: 1498900 | > loss_text_ce: 0.03003726154565811 (0.03042433937125943) | > loss_mel_ce: 2.8578739166259766 (nan) | > loss: 0.011459965258836746 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13704610891270863) | > loader_time: 0.0046 (0.004654431878349522)  --> TIME: 2025-02-27 02:02:22 -- STEP: 21430/36938 -- GLOBAL_STEP: 1498950 | > loss_text_ce: 0.03365633264183998 (0.030422416147226194) | > loss_mel_ce: 3.0850820541381836 (nan) | > loss: 0.012375947088003159 (nan) | > current_lr: 5e-06 | > step_time: 0.1659 (0.1370640898663149) | > loader_time: 0.0048 (0.004654392036833179)  --> TIME: 2025-02-27 02:02:31 -- STEP: 21480/36938 -- GLOBAL_STEP: 1499000 | > loss_text_ce: 0.02494780719280243 (0.03042295349597518) | > loss_mel_ce: 2.4662528038024902 (nan) | > loss: 0.009885717183351517 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13706338173628368) | > loader_time: 0.0045 (0.004654278535416686)  --> TIME: 2025-02-27 02:02:41 -- STEP: 21530/36938 -- GLOBAL_STEP: 1499050 | > loss_text_ce: 0.03426402434706688 (0.030420238203730278) | > loss_mel_ce: 2.8465416431427 (nan) | > loss: 0.011431769467890263 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.1370590978703061) | > loader_time: 0.0045 (0.00465421828346591)  --> TIME: 2025-02-27 02:02:50 -- STEP: 21580/36938 -- GLOBAL_STEP: 1499100 | > loss_text_ce: 0.03976147621870041 (0.030422832665984813) | > loss_mel_ce: 2.7269601821899414 (nan) | > loss: 0.010979055427014828 (nan) | > current_lr: 5e-06 | > step_time: 0.1866 (0.13705538151107374) | > loader_time: 0.0048 (0.004654125066903941)  --> TIME: 2025-02-27 02:02:59 -- STEP: 21630/36938 -- GLOBAL_STEP: 1499150 | > loss_text_ce: 0.030598122626543045 (0.03042187095827273) | > loss_mel_ce: 2.954559326171875 (nan) | > loss: 0.011845863424241543 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13706422757066364) | > loader_time: 0.0046 (0.004654024510377015)  --> TIME: 2025-02-27 02:03:09 -- STEP: 21680/36938 -- GLOBAL_STEP: 1499200 | > loss_text_ce: 0.026015158742666245 (0.030423300196799) | > loss_mel_ce: 2.2061502933502197 (nan) | > loss: 0.00885779969394207 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13706495165824953) | > loader_time: 0.0046 (0.004653943706702501)  --> TIME: 2025-02-27 02:03:18 -- STEP: 21730/36938 -- GLOBAL_STEP: 1499250 | > loss_text_ce: 0.03550818935036659 (0.03042325576292878) | > loss_mel_ce: 2.6015045642852783 (nan) | > loss: 0.010464336723089218 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.1370665551109689) | > loader_time: 0.0047 (0.004653977886960099)  --> TIME: 2025-02-27 02:03:28 -- STEP: 21780/36938 -- GLOBAL_STEP: 1499300 | > loss_text_ce: 0.026031749323010445 (0.030422876303424568) | > loss_mel_ce: 2.9701504707336426 (nan) | > loss: 0.011889612302184105 (nan) | > current_lr: 5e-06 | > step_time: 0.2001 (0.13707100392701602) | > loader_time: 0.0049 (0.0046540002630214685)  --> TIME: 2025-02-27 02:03:37 -- STEP: 21830/36938 -- GLOBAL_STEP: 1499350 | > loss_text_ce: 0.0182790607213974 (0.030422031655425546) | > loss_mel_ce: 1.4626237154006958 (nan) | > loss: 0.005876598879694939 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13707191880777084) | > loader_time: 0.0045 (0.004653897560685824)  --> TIME: 2025-02-27 02:03:46 -- STEP: 21880/36938 -- GLOBAL_STEP: 1499400 | > loss_text_ce: 0.029746059328317642 (0.030422169085672055) | > loss_mel_ce: 3.219019889831543 (nan) | > loss: 0.012891928665339947 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13706351534976016) | > loader_time: 0.0047 (0.004653759281639661)  --> TIME: 2025-02-27 02:03:56 -- STEP: 21930/36938 -- GLOBAL_STEP: 1499450 | > loss_text_ce: 0.029019806534051895 (0.03042466509998069) | > loss_mel_ce: 2.123870849609375 (nan) | > loss: 0.008543217554688454 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13707003142265184) | > loader_time: 0.0046 (0.004653743310329109)  --> TIME: 2025-02-27 02:04:05 -- STEP: 21980/36938 -- GLOBAL_STEP: 1499500 | > loss_text_ce: 0.03327267989516258 (0.030426916826958746) | > loss_mel_ce: 3.0345308780670166 (nan) | > loss: 0.012173824943602085 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13707444291422924) | > loader_time: 0.0046 (0.004653655864413607)  --> TIME: 2025-02-27 02:04:14 -- STEP: 22030/36938 -- GLOBAL_STEP: 1499550 | > loss_text_ce: 0.025759276002645493 (0.03042840765547337) | > loss_mel_ce: 0.3100382387638092 (nan) | > loss: 0.0013325299369171262 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.1370771941844307) | > loader_time: 0.0045 (0.004653787786074262)  --> TIME: 2025-02-27 02:04:24 -- STEP: 22080/36938 -- GLOBAL_STEP: 1499600 | > loss_text_ce: 0.032523322850465775 (0.030429113164174612) | > loss_mel_ce: 2.5273237228393555 (nan) | > loss: 0.010158124379813671 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13707210053352395) | > loader_time: 0.0046 (0.004653704900672486)  --> TIME: 2025-02-27 02:04:33 -- STEP: 22130/36938 -- GLOBAL_STEP: 1499650 | > loss_text_ce: 0.03411291167140007 (0.030430657609117163) | > loss_mel_ce: 2.6731314659118652 (nan) | > loss: 0.010743034072220325 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1370671889925245) | > loader_time: 0.0045 (0.004653720504495297)  --> TIME: 2025-02-27 02:04:42 -- STEP: 22180/36938 -- GLOBAL_STEP: 1499700 | > loss_text_ce: 0.0387415885925293 (0.030431382885669687) | > loss_mel_ce: 2.779182195663452 (nan) | > loss: 0.011182238347828388 (nan) | > current_lr: 5e-06 | > step_time: 0.1693 (0.1370611462752173) | > loader_time: 0.0046 (0.004653700619156684)  --> TIME: 2025-02-27 02:04:52 -- STEP: 22230/36938 -- GLOBAL_STEP: 1499750 | > loss_text_ce: 0.026186857372522354 (0.03043501067850479) | > loss_mel_ce: 2.8827433586120605 (nan) | > loss: 0.011543374508619308 (nan) | > current_lr: 5e-06 | > step_time: 0.1625 (0.1370672348325342) | > loader_time: 0.0046 (0.004653652240926514)  --> TIME: 2025-02-27 02:05:01 -- STEP: 22280/36938 -- GLOBAL_STEP: 1499800 | > loss_text_ce: 0.027324218302965164 (0.03043558547416237) | > loss_mel_ce: 2.612971544265747 (nan) | > loss: 0.010477364994585514 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13707260457360812) | > loader_time: 0.0045 (0.00465367056523022)  --> TIME: 2025-02-27 02:05:10 -- STEP: 22330/36938 -- GLOBAL_STEP: 1499850 | > loss_text_ce: 0.029622148722410202 (0.0304359800534292) | > loss_mel_ce: 1.8970316648483276 (nan) | > loss: 0.00764545239508152 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1370630491383135) | > loader_time: 0.0045 (0.004653748737767432)  --> TIME: 2025-02-27 02:05:19 -- STEP: 22380/36938 -- GLOBAL_STEP: 1499900 | > loss_text_ce: 0.03429063409566879 (0.03043754423512404) | > loss_mel_ce: 0.08447265625 (nan) | > loss: 0.000471282924991101 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13705171478764136) | > loader_time: 0.0046 (0.00465368667386068)  --> TIME: 2025-02-27 02:05:28 -- STEP: 22430/36938 -- GLOBAL_STEP: 1499950 | > loss_text_ce: 0.03508470579981804 (0.030435919791594566) | > loss_mel_ce: 2.4528048038482666 (nan) | > loss: 0.009872578084468842 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13703788429086838) | > loader_time: 0.0045 (0.004653554253954489)  --> TIME: 2025-02-27 02:05:37 -- STEP: 22480/36938 -- GLOBAL_STEP: 1500000 | > loss_text_ce: 0.025879811495542526 (0.030433569346625836) | > loss_mel_ce: 3.247241973876953 (nan) | > loss: 0.012988579459488392 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13703023775404508) | > loader_time: 0.0045 (0.00465352140711722) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1500000.pth  --> TIME: 2025-02-27 02:05:53 -- STEP: 22530/36938 -- GLOBAL_STEP: 1500050 | > loss_text_ce: 0.031664226204156876 (0.030434378116282884) | > loss_mel_ce: 3.305295944213867 (nan) | > loss: 0.013241905719041824 (nan) | > current_lr: 5e-06 | > step_time: 0.1763 (0.1370332622020132) | > loader_time: 0.0048 (0.00465371931493417)  --> TIME: 2025-02-27 02:06:02 -- STEP: 22580/36938 -- GLOBAL_STEP: 1500100 | > loss_text_ce: 0.03129638731479645 (0.03043439986115772) | > loss_mel_ce: 3.304658889770508 (nan) | > loss: 0.013237918727099895 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13703476155722902) | > loader_time: 0.0046 (0.004653673788633279)  --> TIME: 2025-02-27 02:06:12 -- STEP: 22630/36938 -- GLOBAL_STEP: 1500150 | > loss_text_ce: 0.03458678722381592 (0.030435926035512446) | > loss_mel_ce: 2.726478338241577 (nan) | > loss: 0.01095660775899887 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13703746855706764) | > loader_time: 0.0046 (0.004653554188167447)  --> TIME: 2025-02-27 02:06:21 -- STEP: 22680/36938 -- GLOBAL_STEP: 1500200 | > loss_text_ce: 0.031786978244781494 (0.030435509039778954) | > loss_mel_ce: 0.5280213952064514 (nan) | > loss: 0.0022214618511497974 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13703362112322887) | > loader_time: 0.0046 (0.004653468228732071)  --> TIME: 2025-02-27 02:06:30 -- STEP: 22730/36938 -- GLOBAL_STEP: 1500250 | > loss_text_ce: 0.035915181040763855 (0.03043622624438815) | > loss_mel_ce: 2.87662410736084 (nan) | > loss: 0.011557696387171745 (nan) | > current_lr: 5e-06 | > step_time: 0.1507 (0.13703661758995211) | > loader_time: 0.0046 (0.004653396482673262)  --> TIME: 2025-02-27 02:06:39 -- STEP: 22780/36938 -- GLOBAL_STEP: 1500300 | > loss_text_ce: 0.037940915673971176 (0.030436408607316997) | > loss_mel_ce: 2.1889395713806152 (nan) | > loss: 0.008836828172206879 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1370260869022841) | > loader_time: 0.0045 (0.0046534069176825685)  --> TIME: 2025-02-27 02:06:49 -- STEP: 22830/36938 -- GLOBAL_STEP: 1500350 | > loss_text_ce: 0.030767440795898438 (0.030436570162913856) | > loss_mel_ce: 2.851921319961548 (nan) | > loss: 0.011439241468906403 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13704370244051722) | > loader_time: 0.0045 (0.004653377706316364)  --> TIME: 2025-02-27 02:06:58 -- STEP: 22880/36938 -- GLOBAL_STEP: 1500400 | > loss_text_ce: 0.026377925649285316 (0.030433912758334455) | > loss_mel_ce: 2.43798828125 (nan) | > loss: 0.009779231622815132 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13702243621741186) | > loader_time: 0.0045 (0.004653191472683778)  --> TIME: 2025-02-27 02:07:07 -- STEP: 22930/36938 -- GLOBAL_STEP: 1500450 | > loss_text_ce: 0.03598293662071228 (0.03043347602247919) | > loss_mel_ce: 1.5065113306045532 (nan) | > loss: 0.006121009588241577 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13702505848265328) | > loader_time: 0.0045 (0.004653075476478839)  --> TIME: 2025-02-27 02:07:17 -- STEP: 22980/36938 -- GLOBAL_STEP: 1500500 | > loss_text_ce: 0.03426206856966019 (0.030433318116665664) | > loss_mel_ce: 2.346104145050049 (nan) | > loss: 0.009445898234844208 (nan) | > current_lr: 5e-06 | > step_time: 0.1713 (0.13702448417665267) | > loader_time: 0.0049 (0.004652995820456349)  --> TIME: 2025-02-27 02:07:26 -- STEP: 23030/36938 -- GLOBAL_STEP: 1500550 | > loss_text_ce: 0.0371256060898304 (0.03043327014755799) | > loss_mel_ce: 3.047868490219116 (nan) | > loss: 0.01224204059690237 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.137020916696533) | > loader_time: 0.0045 (0.004652985478814069)  --> TIME: 2025-02-27 02:07:35 -- STEP: 23080/36938 -- GLOBAL_STEP: 1500600 | > loss_text_ce: 0.021872729063034058 (0.03043223607771105) | > loss_mel_ce: 2.9508464336395264 (nan) | > loss: 0.011796505190432072 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13701722215945036) | > loader_time: 0.0045 (0.0046529025200954775)  --> TIME: 2025-02-27 02:07:45 -- STEP: 23130/36938 -- GLOBAL_STEP: 1500650 | > loss_text_ce: 0.033889077603816986 (0.03043100838704454) | > loss_mel_ce: 2.6969215869903564 (nan) | > loss: 0.01083655096590519 (nan) | > current_lr: 5e-06 | > step_time: 0.1862 (0.1370227643951834) | > loader_time: 0.0049 (0.004652881653237768)  --> TIME: 2025-02-27 02:07:54 -- STEP: 23180/36938 -- GLOBAL_STEP: 1500700 | > loss_text_ce: 0.03343828022480011 (0.030431908253151233) | > loss_mel_ce: 2.9546151161193848 (nan) | > loss: 0.011857355013489723 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13702546146226632) | > loader_time: 0.0045 (0.00465280171403402)  --> TIME: 2025-02-27 02:08:03 -- STEP: 23230/36938 -- GLOBAL_STEP: 1500750 | > loss_text_ce: 0.032772522419691086 (0.03043450494607557) | > loss_mel_ce: 2.609210729598999 (nan) | > loss: 0.010484061203897 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13702977580137432) | > loader_time: 0.0045 (0.004652716012233016)  --> TIME: 2025-02-27 02:08:13 -- STEP: 23280/36938 -- GLOBAL_STEP: 1500800 | > loss_text_ce: 0.03335737809538841 (0.03043496574741657) | > loss_mel_ce: 1.867451548576355 (nan) | > loss: 0.007542892824858427 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.1370439837581105) | > loader_time: 0.0047 (0.004652694389992176)  --> TIME: 2025-02-27 02:08:22 -- STEP: 23330/36938 -- GLOBAL_STEP: 1500850 | > loss_text_ce: 0.03251736983656883 (0.030435333046359574) | > loss_mel_ce: 2.2967162132263184 (nan) | > loss: 0.009242990985512733 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13703914450004911) | > loader_time: 0.0046 (0.004652598299696087)  --> TIME: 2025-02-27 02:08:32 -- STEP: 23380/36938 -- GLOBAL_STEP: 1500900 | > loss_text_ce: 0.028928028419613838 (0.03043342935235277) | > loss_mel_ce: 2.331491470336914 (nan) | > loss: 0.009366744197905064 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13704097858344122) | > loader_time: 0.0046 (0.004652555810783157)  --> TIME: 2025-02-27 02:08:41 -- STEP: 23430/36938 -- GLOBAL_STEP: 1500950 | > loss_text_ce: 0.035565100610256195 (0.03043270238889425) | > loss_mel_ce: 3.1064538955688477 (nan) | > loss: 0.012468330562114716 (nan) | > current_lr: 5e-06 | > step_time: 0.1648 (0.13703366918437662) | > loader_time: 0.0046 (0.0046524219109978695)  --> TIME: 2025-02-27 02:08:50 -- STEP: 23480/36938 -- GLOBAL_STEP: 1501000 | > loss_text_ce: 0.02931980974972248 (0.03043240990360415) | > loss_mel_ce: 2.868803024291992 (nan) | > loss: 0.01150048803538084 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.1370217235368602) | > loader_time: 0.0046 (0.004652282803793803)  --> TIME: 2025-02-27 02:08:59 -- STEP: 23530/36938 -- GLOBAL_STEP: 1501050 | > loss_text_ce: 0.030582275241613388 (0.030434804924324793) | > loss_mel_ce: 2.565398931503296 (nan) | > loss: 0.01030151266604662 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13702075173385694) | > loader_time: 0.0046 (0.00465222297705747)  --> TIME: 2025-02-27 02:09:09 -- STEP: 23580/36938 -- GLOBAL_STEP: 1501100 | > loss_text_ce: 0.027460267767310143 (0.03043857969985872) | > loss_mel_ce: 2.9239754676818848 (nan) | > loss: 0.011712047271430492 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.13703029818005033) | > loader_time: 0.005 (0.004652234444282563)  --> TIME: 2025-02-27 02:09:18 -- STEP: 23630/36938 -- GLOBAL_STEP: 1501150 | > loss_text_ce: 0.02610725909471512 (0.030439398249779155) | > loss_mel_ce: 2.5832698345184326 (nan) | > loss: 0.010354671627283096 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.1370293333426504) | > loader_time: 0.0046 (0.004652090855589191)  --> TIME: 2025-02-27 02:09:27 -- STEP: 23680/36938 -- GLOBAL_STEP: 1501200 | > loss_text_ce: 0.01615390181541443 (0.03044027877618351) | > loss_mel_ce: 2.2718613147735596 (nan) | > loss: 0.00907942559570074 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13702680522928373) | > loader_time: 0.0046 (0.004652084601489294)  --> TIME: 2025-02-27 02:09:36 -- STEP: 23730/36938 -- GLOBAL_STEP: 1501250 | > loss_text_ce: 0.02809414267539978 (0.030438840672573145) | > loss_mel_ce: 2.1643106937408447 (nan) | > loss: 0.008700019679963589 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13702249756052282) | > loader_time: 0.0047 (0.0046521195770566105)  --> TIME: 2025-02-27 02:09:46 -- STEP: 23780/36938 -- GLOBAL_STEP: 1501300 | > loss_text_ce: 0.022290702909231186 (0.030437418467759197) | > loss_mel_ce: 3.023876428604126 (nan) | > loss: 0.01208796538412571 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13702214152197353) | > loader_time: 0.0047 (0.0046521447705460735)  --> TIME: 2025-02-27 02:09:55 -- STEP: 23830/36938 -- GLOBAL_STEP: 1501350 | > loss_text_ce: 0.029643503949046135 (0.030434734335945363) | > loss_mel_ce: 2.9144210815429688 (nan) | > loss: 0.011682797223329544 (nan) | > current_lr: 5e-06 | > step_time: 0.1811 (0.1370236088936437) | > loader_time: 0.0048 (0.004652207757163057)  --> TIME: 2025-02-27 02:10:05 -- STEP: 23880/36938 -- GLOBAL_STEP: 1501400 | > loss_text_ce: 0.033463407307863235 (0.030434139036078948) | > loss_mel_ce: 1.203319787979126 (nan) | > loss: 0.0049078697338700294 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.1370425589120567) | > loader_time: 0.0045 (0.004652290697672881)  --> TIME: 2025-02-27 02:10:15 -- STEP: 23930/36938 -- GLOBAL_STEP: 1501450 | > loss_text_ce: 0.034689582884311676 (0.030434431145150594) | > loss_mel_ce: 2.557074546813965 (nan) | > loss: 0.010284779593348503 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13704942859868038) | > loader_time: 0.0045 (0.004652323754721511)  --> TIME: 2025-02-27 02:10:24 -- STEP: 23980/36938 -- GLOBAL_STEP: 1501500 | > loss_text_ce: 0.03269406408071518 (0.030435458423214046) | > loss_mel_ce: 2.5705323219299316 (nan) | > loss: 0.010330264456570148 (nan) | > current_lr: 5e-06 | > step_time: 0.1462 (0.1370511989776445) | > loader_time: 0.0046 (0.004652237782784583)  --> TIME: 2025-02-27 02:10:33 -- STEP: 24030/36938 -- GLOBAL_STEP: 1501550 | > loss_text_ce: 0.02435646951198578 (0.030433095820074498) | > loss_mel_ce: 2.2201592922210693 (nan) | > loss: 0.00890680868178606 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13704238153228238) | > loader_time: 0.0045 (0.0046521429612742925)  --> TIME: 2025-02-27 02:10:43 -- STEP: 24080/36938 -- GLOBAL_STEP: 1501600 | > loss_text_ce: 0.03568704426288605 (0.03043323549011914) | > loss_mel_ce: 2.693514108657837 (nan) | > loss: 0.010830163955688477 (nan) | > current_lr: 5e-06 | > step_time: 0.1434 (0.13704185589801424) | > loader_time: 0.0047 (0.0046523111999233115)  --> TIME: 2025-02-27 02:10:52 -- STEP: 24130/36938 -- GLOBAL_STEP: 1501650 | > loss_text_ce: 0.025617225095629692 (0.03043259486711981) | > loss_mel_ce: 2.248213768005371 (nan) | > loss: 0.009023139253258705 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13703610993893958) | > loader_time: 0.0046 (0.004652260647144168)  --> TIME: 2025-02-27 02:11:01 -- STEP: 24180/36938 -- GLOBAL_STEP: 1501700 | > loss_text_ce: 0.02502891793847084 (0.030432087645578646) | > loss_mel_ce: 4.125532627105713 (nan) | > loss: 0.01647048257291317 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13704265243658056) | > loader_time: 0.0046 (0.004652194734246809)  --> TIME: 2025-02-27 02:11:11 -- STEP: 24230/36938 -- GLOBAL_STEP: 1501750 | > loss_text_ce: 0.03098360076546669 (0.030433366009354396) | > loss_mel_ce: 2.759752035140991 (nan) | > loss: 0.011074348352849483 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13704072808215695) | > loader_time: 0.0045 (0.004652178705698012)  --> TIME: 2025-02-27 02:11:20 -- STEP: 24280/36938 -- GLOBAL_STEP: 1501800 | > loss_text_ce: 0.03342629224061966 (0.030433718422719292) | > loss_mel_ce: 3.0143909454345703 (nan) | > loss: 0.012094513513147831 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13703255971534647) | > loader_time: 0.0045 (0.00465210997292396)  --> TIME: 2025-02-27 02:11:29 -- STEP: 24330/36938 -- GLOBAL_STEP: 1501850 | > loss_text_ce: 0.028144368901848793 (0.030433037287895932) | > loss_mel_ce: 2.783203125 (nan) | > loss: 0.011156141757965088 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13704376607778096) | > loader_time: 0.0045 (0.00465210086761363)  --> TIME: 2025-02-27 02:11:39 -- STEP: 24380/36938 -- GLOBAL_STEP: 1501900 | > loss_text_ce: 0.03685816377401352 (0.030433731130610805) | > loss_mel_ce: 3.2438650131225586 (nan) | > loss: 0.013018743135035038 (nan) | > current_lr: 5e-06 | > step_time: 0.1614 (0.13704645412686048) | > loader_time: 0.0048 (0.00465206828050873)  --> TIME: 2025-02-27 02:11:48 -- STEP: 24430/36938 -- GLOBAL_STEP: 1501950 | > loss_text_ce: 0.03527195751667023 (0.030433816286669284) | > loss_mel_ce: 2.74463152885437 (nan) | > loss: 0.011031363159418106 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.1370340095403693) | > loader_time: 0.0047 (0.004652010013566284)  --> TIME: 2025-02-27 02:11:57 -- STEP: 24480/36938 -- GLOBAL_STEP: 1502000 | > loss_text_ce: 0.025602368637919426 (0.030435335827767668) | > loss_mel_ce: 2.8497605323791504 (nan) | > loss: 0.011410171166062355 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.1370301577682593) | > loader_time: 0.0046 (0.004652012787223658)  --> TIME: 2025-02-27 02:12:06 -- STEP: 24530/36938 -- GLOBAL_STEP: 1502050 | > loss_text_ce: 0.034800734370946884 (0.030437092279048717) | > loss_mel_ce: 2.638597011566162 (nan) | > loss: 0.010608721524477005 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13703476746034723) | > loader_time: 0.0047 (0.004652226909150713)  --> TIME: 2025-02-27 02:12:16 -- STEP: 24580/36938 -- GLOBAL_STEP: 1502100 | > loss_text_ce: 0.025671683251857758 (0.03043701388779194) | > loss_mel_ce: 1.2710484266281128 (nan) | > loss: 0.005145715083926916 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13703094564675347) | > loader_time: 0.0045 (0.004652184631108668)  --> TIME: 2025-02-27 02:12:25 -- STEP: 24630/36938 -- GLOBAL_STEP: 1502150 | > loss_text_ce: 0.024914171546697617 (0.03043661395555201) | > loss_mel_ce: 2.6575510501861572 (nan) | > loss: 0.010644704103469849 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1370340576415815) | > loader_time: 0.0046 (0.004652179463627001)  --> TIME: 2025-02-27 02:12:35 -- STEP: 24680/36938 -- GLOBAL_STEP: 1502200 | > loss_text_ce: 0.03826058283448219 (0.030435100175376914) | > loss_mel_ce: 1.9260607957839966 (nan) | > loss: 0.007794926408678293 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13704202598265783) | > loader_time: 0.0047 (0.0046523011967274245)  --> TIME: 2025-02-27 02:12:44 -- STEP: 24730/36938 -- GLOBAL_STEP: 1502250 | > loss_text_ce: 0.0227378960698843 (0.0304310143658996) | > loss_mel_ce: 2.393017530441284 (nan) | > loss: 0.009586332365870476 (nan) | > current_lr: 5e-06 | > step_time: 0.1703 (0.13703237020482573) | > loader_time: 0.0048 (0.004652505050147072)  --> TIME: 2025-02-27 02:12:53 -- STEP: 24780/36938 -- GLOBAL_STEP: 1502300 | > loss_text_ce: 0.05726079270243645 (0.030434779230207567) | > loss_mel_ce: 2.9368844032287598 (nan) | > loss: 0.011881529353559017 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13703710543137626) | > loader_time: 0.0046 (0.004652511842602982)  --> TIME: 2025-02-27 02:13:03 -- STEP: 24830/36938 -- GLOBAL_STEP: 1502350 | > loss_text_ce: 0.014924505725502968 (0.03043514308941626) | > loss_mel_ce: 2.502511739730835 (nan) | > loss: 0.009989826940000057 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13703546925413046) | > loader_time: 0.0047 (0.00465250599062641)  --> TIME: 2025-02-27 02:13:12 -- STEP: 24880/36938 -- GLOBAL_STEP: 1502400 | > loss_text_ce: 0.027905970811843872 (0.030436782064694922) | > loss_mel_ce: 2.7947850227355957 (nan) | > loss: 0.011201155371963978 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13703222281488248) | > loader_time: 0.0046 (0.004652528316261702)  --> TIME: 2025-02-27 02:13:21 -- STEP: 24930/36938 -- GLOBAL_STEP: 1502450 | > loss_text_ce: 0.0294698104262352 (0.03043728197341642) | > loss_mel_ce: 2.706434726715088 (nan) | > loss: 0.010856764391064644 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13703665548761504) | > loader_time: 0.0046 (0.004652578305681708)  --> TIME: 2025-02-27 02:13:31 -- STEP: 24980/36938 -- GLOBAL_STEP: 1502500 | > loss_text_ce: 0.04238998889923096 (0.03043864387914455) | > loss_mel_ce: 2.291476011276245 (nan) | > loss: 0.009261374361813068 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.1370382908537834) | > loader_time: 0.0047 (0.004652646267480919)  --> TIME: 2025-02-27 02:13:40 -- STEP: 25030/36938 -- GLOBAL_STEP: 1502550 | > loss_text_ce: 0.025629805400967598 (0.03043902900480731) | > loss_mel_ce: 1.6995097398757935 (nan) | > loss: 0.006845791824162006 (nan) | > current_lr: 5e-06 | > step_time: 0.1499 (0.13703536628200436) | > loader_time: 0.0046 (0.004652560619273492)  --> TIME: 2025-02-27 02:13:50 -- STEP: 25080/36938 -- GLOBAL_STEP: 1502600 | > loss_text_ce: 0.028105096891522408 (0.0304384026473293) | > loss_mel_ce: 2.855745553970337 (nan) | > loss: 0.011443852446973324 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13703899948220558) | > loader_time: 0.0046 (0.004652511446099552)  --> TIME: 2025-02-27 02:13:59 -- STEP: 25130/36938 -- GLOBAL_STEP: 1502650 | > loss_text_ce: 0.036269355565309525 (0.030439694590423325) | > loss_mel_ce: 1.5466628074645996 (nan) | > loss: 0.006281476933509111 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13703977631896472) | > loader_time: 0.0047 (0.004652612407606926)  --> TIME: 2025-02-27 02:14:08 -- STEP: 25180/36938 -- GLOBAL_STEP: 1502700 | > loss_text_ce: 0.030770497396588326 (0.030440985499988468) | > loss_mel_ce: 2.8018157482147217 (nan) | > loss: 0.01124042272567749 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.1370398073086583) | > loader_time: 0.005 (0.004653033176237312)  --> TIME: 2025-02-27 02:14:17 -- STEP: 25230/36938 -- GLOBAL_STEP: 1502750 | > loss_text_ce: 0.030608953908085823 (0.030440035201217852) | > loss_mel_ce: 2.4855222702026367 (nan) | > loss: 0.00998464785516262 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13703241812252098) | > loader_time: 0.0045 (0.004652934569571653)  --> TIME: 2025-02-27 02:14:27 -- STEP: 25280/36938 -- GLOBAL_STEP: 1502800 | > loss_text_ce: 0.026746027171611786 (0.030438905514893123) | > loss_mel_ce: 2.251283645629883 (nan) | > loss: 0.009039800614118576 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13703827781579248) | > loader_time: 0.0046 (0.004652951167354115)  --> TIME: 2025-02-27 02:14:37 -- STEP: 25330/36938 -- GLOBAL_STEP: 1502850 | > loss_text_ce: 0.025834612548351288 (0.03043661446119756) | > loss_mel_ce: 1.1495921611785889 (nan) | > loss: 0.004664392210543156 (nan) | > current_lr: 5e-06 | > step_time: 0.1121 (0.13704612116732795) | > loader_time: 0.0046 (0.004652888003988239)  --> TIME: 2025-02-27 02:14:46 -- STEP: 25380/36938 -- GLOBAL_STEP: 1502900 | > loss_text_ce: 0.03238152340054512 (0.030436949254475223) | > loss_mel_ce: 2.532804489135742 (nan) | > loss: 0.010179310105741024 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13704682378152752) | > loader_time: 0.0045 (0.004652818739179383)  --> TIME: 2025-02-27 02:14:55 -- STEP: 25430/36938 -- GLOBAL_STEP: 1502950 | > loss_text_ce: 0.035779062658548355 (0.03043792405577008) | > loss_mel_ce: 2.8612778186798096 (nan) | > loss: 0.011496257968246937 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13704085470042887) | > loader_time: 0.0067 (0.004653052396770554)  --> TIME: 2025-02-27 02:15:04 -- STEP: 25480/36938 -- GLOBAL_STEP: 1503000 | > loss_text_ce: 0.033846188336610794 (0.03043843449047347) | > loss_mel_ce: 2.379495620727539 (nan) | > loss: 0.009576753713190556 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13703861838233033) | > loader_time: 0.0045 (0.004653197508217601)  --> TIME: 2025-02-27 02:15:14 -- STEP: 25530/36938 -- GLOBAL_STEP: 1503050 | > loss_text_ce: 0.04286671057343483 (0.03043989008438474) | > loss_mel_ce: 2.7755584716796875 (nan) | > loss: 0.011184227652847767 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13704122042964195) | > loader_time: 0.0046 (0.00465338585005992)  --> TIME: 2025-02-27 02:15:23 -- STEP: 25580/36938 -- GLOBAL_STEP: 1503100 | > loss_text_ce: 0.023867163807153702 (0.03043907318447015) | > loss_mel_ce: 2.470604419708252 (nan) | > loss: 0.009898697026073933 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13703772012287135) | > loader_time: 0.0046 (0.004653407643790181)  --> TIME: 2025-02-27 02:15:32 -- STEP: 25630/36938 -- GLOBAL_STEP: 1503150 | > loss_text_ce: 0.022068122401833534 (0.030438707248168592) | > loss_mel_ce: 1.9656546115875244 (nan) | > loss: 0.00788778904825449 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13703669878453323) | > loader_time: 0.0046 (0.00465341546411773)  --> TIME: 2025-02-27 02:15:41 -- STEP: 25680/36938 -- GLOBAL_STEP: 1503200 | > loss_text_ce: 0.03156290575861931 (0.030436758868029264) | > loss_mel_ce: 2.2152023315429688 (nan) | > loss: 0.00891573540866375 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13702900682654412) | > loader_time: 0.0045 (0.004653437050332178)  --> TIME: 2025-02-27 02:15:51 -- STEP: 25730/36938 -- GLOBAL_STEP: 1503250 | > loss_text_ce: 0.024614350870251656 (0.030438523462351326) | > loss_mel_ce: 2.4873695373535156 (nan) | > loss: 0.00996819045394659 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.1370309003447456) | > loader_time: 0.0047 (0.004653459071557085)  --> TIME: 2025-02-27 02:16:00 -- STEP: 25780/36938 -- GLOBAL_STEP: 1503300 | > loss_text_ce: 0.02971896156668663 (0.03043865564176661) | > loss_mel_ce: 2.73184871673584 (nan) | > loss: 0.010958602651953697 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.1370312845457246) | > loader_time: 0.0046 (0.004653592549894485)  --> TIME: 2025-02-27 02:16:10 -- STEP: 25830/36938 -- GLOBAL_STEP: 1503350 | > loss_text_ce: 0.025858215987682343 (0.03043881744522049) | > loss_mel_ce: 2.396899461746216 (nan) | > loss: 0.009614118374884129 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13703638036907412) | > loader_time: 0.0046 (0.004653529090748452)  --> TIME: 2025-02-27 02:16:19 -- STEP: 25880/36938 -- GLOBAL_STEP: 1503400 | > loss_text_ce: 0.03477734699845314 (0.030439807310840044) | > loss_mel_ce: 3.0836668014526367 (nan) | > loss: 0.01237477920949459 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.13703164946535523) | > loader_time: 0.0048 (0.004653550106369933)  --> TIME: 2025-02-27 02:16:29 -- STEP: 25930/36938 -- GLOBAL_STEP: 1503450 | > loss_text_ce: 0.030393974855542183 (0.030439924494235146) | > loss_mel_ce: 0.9449442028999329 (nan) | > loss: 0.0038703896570950747 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13704068760038915) | > loader_time: 0.0046 (0.004653637408290751)  --> TIME: 2025-02-27 02:16:38 -- STEP: 25980/36938 -- GLOBAL_STEP: 1503500 | > loss_text_ce: 0.024441786110401154 (0.030440271168185814) | > loss_mel_ce: 2.6966946125030518 (nan) | > loss: 0.010798160918056965 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.13704096376207694) | > loader_time: 0.0046 (0.004653719767320159)  --> TIME: 2025-02-27 02:16:48 -- STEP: 26030/36938 -- GLOBAL_STEP: 1503550 | > loss_text_ce: 0.03778940439224243 (0.030441639372127943) | > loss_mel_ce: 2.0933773517608643 (nan) | > loss: 0.008457011543214321 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.1370575701599625) | > loader_time: 0.0047 (0.004653761499531682)  --> TIME: 2025-02-27 02:16:57 -- STEP: 26080/36938 -- GLOBAL_STEP: 1503600 | > loss_text_ce: 0.01982211507856846 (0.030441537189570204) | > loss_mel_ce: 2.535688877105713 (nan) | > loss: 0.010140917263925076 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13705949766869976) | > loader_time: 0.0067 (0.004653776405047797)  --> TIME: 2025-02-27 02:17:07 -- STEP: 26130/36938 -- GLOBAL_STEP: 1503650 | > loss_text_ce: 0.03156901150941849 (0.030442661133769814) | > loss_mel_ce: 1.3235896825790405 (nan) | > loss: 0.005377613939344883 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13706072792895638) | > loader_time: 0.0046 (0.004653871757428505)  --> TIME: 2025-02-27 02:17:16 -- STEP: 26180/36938 -- GLOBAL_STEP: 1503700 | > loss_text_ce: 0.03188752010464668 (0.030443581172874847) | > loss_mel_ce: 2.555574655532837 (nan) | > loss: 0.010267707519233227 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.13706292529612546) | > loader_time: 0.0047 (0.004654032515605244)  --> TIME: 2025-02-27 02:17:26 -- STEP: 26230/36938 -- GLOBAL_STEP: 1503750 | > loss_text_ce: 0.031769223511219025 (0.030443491201082282) | > loss_mel_ce: 0.0076268515549600124 (nan) | > loss: 0.000156333640916273 (nan) | > current_lr: 5e-06 | > step_time: 0.1271 (0.1370689806209197) | > loader_time: 0.0046 (0.004654101983672842)  --> TIME: 2025-02-27 02:17:35 -- STEP: 26280/36938 -- GLOBAL_STEP: 1503800 | > loss_text_ce: 0.03101693093776703 (0.0304448027076211) | > loss_mel_ce: 3.039193868637085 (nan) | > loss: 0.012183376587927341 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.1370689857133091) | > loader_time: 0.0046 (0.0046541979414505994)  --> TIME: 2025-02-27 02:17:44 -- STEP: 26330/36938 -- GLOBAL_STEP: 1503850 | > loss_text_ce: 0.021886572241783142 (0.030444935758722303) | > loss_mel_ce: 3.8203125 (nan) | > loss: 0.015246822498738766 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13706809666688377) | > loader_time: 0.0046 (0.004654130580712322)  --> TIME: 2025-02-27 02:17:53 -- STEP: 26380/36938 -- GLOBAL_STEP: 1503900 | > loss_text_ce: 0.030765563249588013 (0.030443250942781816) | > loss_mel_ce: 1.403977394104004 (nan) | > loss: 0.005693424493074417 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.13706082205956857) | > loader_time: 0.0046 (0.00465398399643686)  --> TIME: 2025-02-27 02:18:03 -- STEP: 26430/36938 -- GLOBAL_STEP: 1503950 | > loss_text_ce: 0.031053857877850533 (0.0304420953114368) | > loss_mel_ce: 1.5027669668197632 (nan) | > loss: 0.0060865911655128 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13706047387002318) | > loader_time: 0.0046 (0.0046539470096722485)  --> TIME: 2025-02-27 02:18:12 -- STEP: 26480/36938 -- GLOBAL_STEP: 1504000 | > loss_text_ce: 0.0256916806101799 (0.030440191952884478) | > loss_mel_ce: 2.9197146892547607 (nan) | > loss: 0.01168812159448862 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13706693755536017) | > loader_time: 0.0046 (0.004653953641563042)  --> TIME: 2025-02-27 02:18:22 -- STEP: 26530/36938 -- GLOBAL_STEP: 1504050 | > loss_text_ce: 0.02964361570775509 (0.030439503681893354) | > loss_mel_ce: 1.2828572988510132 (nan) | > loss: 0.005208337213844061 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1370623532215074) | > loader_time: 0.0045 (0.004654055112629553)  --> TIME: 2025-02-27 02:18:31 -- STEP: 26580/36938 -- GLOBAL_STEP: 1504100 | > loss_text_ce: 0.029812071472406387 (0.030439852481740165) | > loss_mel_ce: 2.463283061981201 (nan) | > loss: 0.009893235750496387 (nan) | > current_lr: 5e-06 | > step_time: 0.1499 (0.13706504727235225) | > loader_time: 0.0046 (0.004654073894606959)  --> TIME: 2025-02-27 02:18:40 -- STEP: 26630/36938 -- GLOBAL_STEP: 1504150 | > loss_text_ce: 0.03127589449286461 (0.030439918868626997) | > loss_mel_ce: 2.760758876800537 (nan) | > loss: 0.011079504154622555 (nan) | > current_lr: 5e-06 | > step_time: 0.1651 (0.13707061398535253) | > loader_time: 0.0048 (0.004654113833634804)  --> TIME: 2025-02-27 02:18:50 -- STEP: 26680/36938 -- GLOBAL_STEP: 1504200 | > loss_text_ce: 0.03688771650195122 (0.0304411775588562) | > loss_mel_ce: 3.012010097503662 (nan) | > loss: 0.01209880132228136 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.13706635658530192) | > loader_time: 0.0047 (0.004654050793783618)  --> TIME: 2025-02-27 02:18:59 -- STEP: 26730/36938 -- GLOBAL_STEP: 1504250 | > loss_text_ce: 0.030453655868768692 (0.03044087139564175) | > loss_mel_ce: 2.5879552364349365 (nan) | > loss: 0.010390512645244598 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.137067280946669) | > loader_time: 0.0046 (0.004654063439931151)  --> TIME: 2025-02-27 02:19:08 -- STEP: 26780/36938 -- GLOBAL_STEP: 1504300 | > loss_text_ce: 0.02533528581261635 (0.030440115536089437) | > loss_mel_ce: 1.8530219793319702 (nan) | > loss: 0.007453799247741699 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13706265413734522) | > loader_time: 0.0045 (0.004653951932994016)  --> TIME: 2025-02-27 02:19:18 -- STEP: 26830/36938 -- GLOBAL_STEP: 1504350 | > loss_text_ce: 0.03289009630680084 (0.03043981438806052) | > loss_mel_ce: 2.7322521209716797 (nan) | > loss: 0.010972786694765091 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1370653660582652) | > loader_time: 0.0045 (0.004653875826900698)  --> TIME: 2025-02-27 02:19:27 -- STEP: 26880/36938 -- GLOBAL_STEP: 1504400 | > loss_text_ce: 0.032841458916664124 (0.030437371782179635) | > loss_mel_ce: 2.178985357284546 (nan) | > loss: 0.008777091279625893 (nan) | > current_lr: 5e-06 | > step_time: 0.1441 (0.13706556452172225) | > loader_time: 0.0046 (0.004653816714528079)  --> TIME: 2025-02-27 02:19:36 -- STEP: 26930/36938 -- GLOBAL_STEP: 1504450 | > loss_text_ce: 0.02676403895020485 (0.030438063101067645) | > loss_mel_ce: 2.6001367568969727 (nan) | > loss: 0.01042421068996191 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1370620446093595) | > loader_time: 0.0046 (0.004653696433080447)  --> TIME: 2025-02-27 02:19:46 -- STEP: 26980/36938 -- GLOBAL_STEP: 1504500 | > loss_text_ce: 0.018926363438367844 (0.030437608314243133) | > loss_mel_ce: 1.953305959701538 (nan) | > loss: 0.007826318964362144 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13705626913492097) | > loader_time: 0.0045 (0.004653549627342275)  --> TIME: 2025-02-27 02:19:55 -- STEP: 27030/36938 -- GLOBAL_STEP: 1504550 | > loss_text_ce: 0.03625890240073204 (0.030437059914179856) | > loss_mel_ce: 2.9323792457580566 (nan) | > loss: 0.011780311353504658 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.13705196856041188) | > loader_time: 0.0046 (0.0046534477319269555)  --> TIME: 2025-02-27 02:20:04 -- STEP: 27080/36938 -- GLOBAL_STEP: 1504600 | > loss_text_ce: 0.031990792602300644 (0.030436413069744008) | > loss_mel_ce: 3.530959129333496 (nan) | > loss: 0.014138691127300262 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13704808677493102) | > loader_time: 0.0045 (0.004653421154402582)  --> TIME: 2025-02-27 02:20:13 -- STEP: 27130/36938 -- GLOBAL_STEP: 1504650 | > loss_text_ce: 0.041877664625644684 (0.030437600631795486) | > loss_mel_ce: 2.8650131225585938 (nan) | > loss: 0.011535282246768475 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13704984367355205) | > loader_time: 0.0046 (0.004653407953517557)  --> TIME: 2025-02-27 02:20:23 -- STEP: 27180/36938 -- GLOBAL_STEP: 1504700 | > loss_text_ce: 0.027669794857501984 (0.030436714122317198) | > loss_mel_ce: 2.5586063861846924 (nan) | > loss: 0.010263001546263695 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13704728751256304) | > loader_time: 0.0047 (0.004653311363470063)  --> TIME: 2025-02-27 02:20:32 -- STEP: 27230/36938 -- GLOBAL_STEP: 1504750 | > loss_text_ce: 0.03108692169189453 (0.03043772993623674) | > loss_mel_ce: 2.898754358291626 (nan) | > loss: 0.011626355350017548 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13703734136617207) | > loader_time: 0.0046 (0.004653221099550946)  --> TIME: 2025-02-27 02:20:41 -- STEP: 27280/36938 -- GLOBAL_STEP: 1504800 | > loss_text_ce: 0.03454848378896713 (0.03043828057732892) | > loss_mel_ce: 2.9895360469818115 (nan) | > loss: 0.012000336311757565 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.1370430095629266) | > loader_time: 0.0046 (0.00465319243169603)  --> TIME: 2025-02-27 02:20:50 -- STEP: 27330/36938 -- GLOBAL_STEP: 1504850 | > loss_text_ce: 0.03757714480161667 (0.030436905160644446) | > loss_mel_ce: 2.5451278686523438 (nan) | > loss: 0.010248829610645771 (nan) | > current_lr: 5e-06 | > step_time: 0.1869 (0.1370415900506433) | > loader_time: 0.0048 (0.004653212573124412)  --> TIME: 2025-02-27 02:21:00 -- STEP: 27380/36938 -- GLOBAL_STEP: 1504900 | > loss_text_ce: 0.03165840730071068 (0.03043537605499442) | > loss_mel_ce: 2.6476430892944336 (nan) | > loss: 0.010632149875164032 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13704361899043707) | > loader_time: 0.0046 (0.004653264250173663)  --> TIME: 2025-02-27 02:21:10 -- STEP: 27430/36938 -- GLOBAL_STEP: 1504950 | > loss_text_ce: 0.033734820783138275 (0.030435986261824338) | > loss_mel_ce: 0.0023956298828125 (nan) | > loss: 0.00014337481115944684 (nan) | > current_lr: 5e-06 | > step_time: 0.1522 (0.13705405876926038) | > loader_time: 0.0047 (0.0046533242134208595)  --> TIME: 2025-02-27 02:21:19 -- STEP: 27480/36938 -- GLOBAL_STEP: 1505000 | > loss_text_ce: 0.028142813593149185 (0.03043558668830531) | > loss_mel_ce: 1.564216136932373 (nan) | > loss: 0.006318884901702404 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.137051106462313) | > loader_time: 0.0045 (0.004653313994581208)  --> TIME: 2025-02-27 02:21:28 -- STEP: 27530/36938 -- GLOBAL_STEP: 1505050 | > loss_text_ce: 0.028877301141619682 (0.030436727560886116) | > loss_mel_ce: 2.8282275199890137 (nan) | > loss: 0.01133771799504757 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13705850446869508) | > loader_time: 0.0046 (0.0046533459799531635)  --> TIME: 2025-02-27 02:21:38 -- STEP: 27580/36938 -- GLOBAL_STEP: 1505100 | > loss_text_ce: 0.029789544641971588 (0.03043420937143947) | > loss_mel_ce: 2.4443283081054688 (nan) | > loss: 0.009817928075790405 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13706306263188606) | > loader_time: 0.0046 (0.004653332836823359)  --> TIME: 2025-02-27 02:21:47 -- STEP: 27630/36938 -- GLOBAL_STEP: 1505150 | > loss_text_ce: 0.02906465157866478 (0.03043275934796062) | > loss_mel_ce: 2.5934219360351562 (nan) | > loss: 0.010406693443655968 (nan) | > current_lr: 5e-06 | > step_time: 0.1518 (0.13706157443060135) | > loader_time: 0.0047 (0.00465330621965807)  --> TIME: 2025-02-27 02:21:57 -- STEP: 27680/36938 -- GLOBAL_STEP: 1505200 | > loss_text_ce: 0.03061181679368019 (0.030430337212807527) | > loss_mel_ce: 3.1077919006347656 (nan) | > loss: 0.012453983537852764 (nan) | > current_lr: 5e-06 | > step_time: 0.1705 (0.13706833426490792) | > loader_time: 0.0047 (0.004653293135538298)  --> TIME: 2025-02-27 02:22:06 -- STEP: 27730/36938 -- GLOBAL_STEP: 1505250 | > loss_text_ce: 0.03153955936431885 (0.03043033973261139) | > loss_mel_ce: 2.6433956623077393 (nan) | > loss: 0.010614823549985886 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13707372664201692) | > loader_time: 0.0046 (0.004653267545731057)  --> TIME: 2025-02-27 02:22:16 -- STEP: 27780/36938 -- GLOBAL_STEP: 1505300 | > loss_text_ce: 0.02680289000272751 (0.030427779513562087) | > loss_mel_ce: 2.6121184825897217 (nan) | > loss: 0.010471910238265991 (nan) | > current_lr: 5e-06 | > step_time: 0.1462 (0.13707537081191945) | > loader_time: 0.0046 (0.004653252312568932)  --> TIME: 2025-02-27 02:22:25 -- STEP: 27830/36938 -- GLOBAL_STEP: 1505350 | > loss_text_ce: 0.027748964726924896 (0.030429207682047352) | > loss_mel_ce: 2.6689789295196533 (nan) | > loss: 0.01070130243897438 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.13707599054769115) | > loader_time: 0.0049 (0.004653219786043297)  --> TIME: 2025-02-27 02:22:35 -- STEP: 27880/36938 -- GLOBAL_STEP: 1505400 | > loss_text_ce: 0.03329365700483322 (0.030429831993100707) | > loss_mel_ce: 1.86061429977417 (nan) | > loss: 0.0075155082158744335 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13708212355117133) | > loader_time: 0.0046 (0.004653345666280637)  --> TIME: 2025-02-27 02:22:44 -- STEP: 27930/36938 -- GLOBAL_STEP: 1505450 | > loss_text_ce: 0.0312739722430706 (0.030429343591823324) | > loss_mel_ce: 2.753621816635132 (nan) | > loss: 0.011051175184547901 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13708112749454268) | > loader_time: 0.0046 (0.004653279182947656)  --> TIME: 2025-02-27 02:22:54 -- STEP: 27980/36938 -- GLOBAL_STEP: 1505500 | > loss_text_ce: 0.039339929819107056 (0.03042919896712781) | > loss_mel_ce: 1.917484998703003 (nan) | > loss: 0.007765178568661213 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13709502733120263) | > loader_time: 0.0046 (0.004653418012991248)  --> TIME: 2025-02-27 02:23:03 -- STEP: 28030/36938 -- GLOBAL_STEP: 1505550 | > loss_text_ce: 0.04167843610048294 (0.030430147186276523) | > loss_mel_ce: 2.381763458251953 (nan) | > loss: 0.009616833180189133 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1370987452994585) | > loader_time: 0.0045 (0.004653448757426156)  --> TIME: 2025-02-27 02:23:13 -- STEP: 28080/36938 -- GLOBAL_STEP: 1505600 | > loss_text_ce: 0.03295302391052246 (0.03043236916568369) | > loss_mel_ce: 3.0801355838775635 (nan) | > loss: 0.01235352735966444 (nan) | > current_lr: 5e-06 | > step_time: 0.1911 (0.13710553731843736) | > loader_time: 0.0049 (0.004653527814778198)  --> TIME: 2025-02-27 02:23:23 -- STEP: 28130/36938 -- GLOBAL_STEP: 1505650 | > loss_text_ce: 0.03193961828947067 (0.030431962161468276) | > loss_mel_ce: 2.561448812484741 (nan) | > loss: 0.010291224345564842 (nan) | > current_lr: 5e-06 | > step_time: 0.1689 (0.13711482679297346) | > loader_time: 0.0048 (0.004653512656031862)  --> TIME: 2025-02-27 02:23:32 -- STEP: 28180/36938 -- GLOBAL_STEP: 1505700 | > loss_text_ce: 0.029138006269931793 (0.030427936446932962) | > loss_mel_ce: 0.07256317138671875 (nan) | > loss: 0.0004035761230625212 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1371200870587704) | > loader_time: 0.0046 (0.004653535818190703)  --> TIME: 2025-02-27 02:23:42 -- STEP: 28230/36938 -- GLOBAL_STEP: 1505750 | > loss_text_ce: 0.03370274603366852 (0.030426993752356975) | > loss_mel_ce: 2.959817886352539 (nan) | > loss: 0.011879051104187965 (nan) | > current_lr: 5e-06 | > step_time: 0.1491 (0.1371255165121055) | > loader_time: 0.0047 (0.004653549160553668)  --> TIME: 2025-02-27 02:23:51 -- STEP: 28280/36938 -- GLOBAL_STEP: 1505800 | > loss_text_ce: 0.034483496099710464 (0.030428094040211212) | > loss_mel_ce: 1.3868217468261719 (nan) | > loss: 0.005640100687742233 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13711618339324208) | > loader_time: 0.0045 (0.00465347379909718)  --> TIME: 2025-02-27 02:24:00 -- STEP: 28330/36938 -- GLOBAL_STEP: 1505850 | > loss_text_ce: 0.02138207107782364 (0.03042807585835599) | > loss_mel_ce: 3.4901366233825684 (nan) | > loss: 0.013934599235653877 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.137112589880557) | > loader_time: 0.0045 (0.004653450435351649)  --> TIME: 2025-02-27 02:24:09 -- STEP: 28380/36938 -- GLOBAL_STEP: 1505900 | > loss_text_ce: 0.03230752423405647 (0.030430072990401666) | > loss_mel_ce: 2.5129518508911133 (nan) | > loss: 0.010100236162543297 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13711387246658244) | > loader_time: 0.0046 (0.004653393079732181)  --> TIME: 2025-02-27 02:24:19 -- STEP: 28430/36938 -- GLOBAL_STEP: 1505950 | > loss_text_ce: 0.02417789399623871 (0.030432262127767597) | > loss_mel_ce: 1.9740654230117798 (nan) | > loss: 0.007929537445306778 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13711509582151993) | > loader_time: 0.0046 (0.004653415661482115)  --> TIME: 2025-02-27 02:24:28 -- STEP: 28480/36938 -- GLOBAL_STEP: 1506000 | > loss_text_ce: 0.0345788449048996 (0.030434558277013135) | > loss_mel_ce: 2.4756035804748535 (nan) | > loss: 0.009961041621863842 (nan) | > current_lr: 5e-06 | > step_time: 0.1699 (0.13711033376415235) | > loader_time: 0.0048 (0.004653321934884883)  --> TIME: 2025-02-27 02:24:37 -- STEP: 28530/36938 -- GLOBAL_STEP: 1506050 | > loss_text_ce: 0.030841780826449394 (0.03043617307499986) | > loss_mel_ce: 2.9323549270629883 (nan) | > loss: 0.011758717708289623 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.1371081096600377) | > loader_time: 0.0045 (0.004653216603342989)  --> TIME: 2025-02-27 02:24:47 -- STEP: 28580/36938 -- GLOBAL_STEP: 1506100 | > loss_text_ce: 0.023989761248230934 (0.030435370422373126) | > loss_mel_ce: 2.5866658687591553 (nan) | > loss: 0.010359744541347027 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13711012827924726) | > loader_time: 0.0046 (0.004653414652179083)  --> TIME: 2025-02-27 02:24:56 -- STEP: 28630/36938 -- GLOBAL_STEP: 1506150 | > loss_text_ce: 0.031862225383520126 (0.03043432909721418) | > loss_mel_ce: 1.1704007387161255 (nan) | > loss: 0.004770885221660137 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13711124824719906) | > loader_time: 0.0045 (0.0046533782457997535)  --> TIME: 2025-02-27 02:25:05 -- STEP: 28680/36938 -- GLOBAL_STEP: 1506200 | > loss_text_ce: 0.035524383187294006 (0.030435090119602364) | > loss_mel_ce: 2.644566535949707 (nan) | > loss: 0.010635281912982464 (nan) | > current_lr: 5e-06 | > step_time: 0.1701 (0.137105546661003) | > loader_time: 0.0047 (0.004653378352626793)  --> TIME: 2025-02-27 02:25:14 -- STEP: 28730/36938 -- GLOBAL_STEP: 1506250 | > loss_text_ce: 0.029034584760665894 (0.030434345623871827) | > loss_mel_ce: 2.8790371417999268 (nan) | > loss: 0.011539967730641365 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13710434550488934) | > loader_time: 0.0047 (0.0046533831477870895)  --> TIME: 2025-02-27 02:25:24 -- STEP: 28780/36938 -- GLOBAL_STEP: 1506300 | > loss_text_ce: 0.042295411229133606 (0.030433974791217216) | > loss_mel_ce: 2.2057037353515625 (nan) | > loss: 0.008920632302761078 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13711339176488252) | > loader_time: 0.0046 (0.004653506116621883)  --> TIME: 2025-02-27 02:25:33 -- STEP: 28830/36938 -- GLOBAL_STEP: 1506350 | > loss_text_ce: 0.027100982144474983 (0.030432031951273558) | > loss_mel_ce: 2.165034532546997 (nan) | > loss: 0.008698951452970505 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.13711333664349387) | > loader_time: 0.0046 (0.00465356711501099)  --> TIME: 2025-02-27 02:25:43 -- STEP: 28880/36938 -- GLOBAL_STEP: 1506400 | > loss_text_ce: 0.02652011066675186 (0.030431195053625387) | > loss_mel_ce: 2.911057233810425 (nan) | > loss: 0.011657053604722023 (nan) | > current_lr: 5e-06 | > step_time: 0.1859 (0.13711196207273793) | > loader_time: 0.0054 (0.004653544612538481)  --> TIME: 2025-02-27 02:25:52 -- STEP: 28930/36938 -- GLOBAL_STEP: 1506450 | > loss_text_ce: 0.027046646922826767 (0.030430360261455174) | > loss_mel_ce: 1.5250463485717773 (nan) | > loss: 0.006159099750220776 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13710749938831746) | > loader_time: 0.0048 (0.004653591850905805)  --> TIME: 2025-02-27 02:26:01 -- STEP: 28980/36938 -- GLOBAL_STEP: 1506500 | > loss_text_ce: 0.041822995990514755 (0.03043084863388205) | > loss_mel_ce: 2.430424690246582 (nan) | > loss: 0.00981050729751587 (nan) | > current_lr: 5e-06 | > step_time: 0.1066 (0.13710755414186215) | > loader_time: 0.0046 (0.004653682578753599)  --> TIME: 2025-02-27 02:26:10 -- STEP: 29030/36938 -- GLOBAL_STEP: 1506550 | > loss_text_ce: 0.03086020238697529 (0.030432388204869487) | > loss_mel_ce: 3.3815460205078125 (nan) | > loss: 0.013541295193135738 (nan) | > current_lr: 5e-06 | > step_time: 0.1783 (0.1371052819779942) | > loader_time: 0.0047 (0.004653714494051488)  --> TIME: 2025-02-27 02:26:20 -- STEP: 29080/36938 -- GLOBAL_STEP: 1506600 | > loss_text_ce: 0.02689424529671669 (0.030431308002535257) | > loss_mel_ce: 1.332851529121399 (nan) | > loss: 0.0053958166390657425 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13710419004360358) | > loader_time: 0.0045 (0.0046536419135832525)  --> TIME: 2025-02-27 02:26:29 -- STEP: 29130/36938 -- GLOBAL_STEP: 1506650 | > loss_text_ce: 0.034594081342220306 (0.030430201186686295) | > loss_mel_ce: 2.5587270259857178 (nan) | > loss: 0.010290957055985928 (nan) | > current_lr: 5e-06 | > step_time: 0.1452 (0.13710439468715171) | > loader_time: 0.0046 (0.0046535458713719054)  --> TIME: 2025-02-27 02:26:39 -- STEP: 29180/36938 -- GLOBAL_STEP: 1506700 | > loss_text_ce: 0.03260892257094383 (0.030429909930572763) | > loss_mel_ce: 2.36020827293396 (nan) | > loss: 0.009495306760072708 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.1371136267271827) | > loader_time: 0.0045 (0.004653648066961933)  --> TIME: 2025-02-27 02:26:48 -- STEP: 29230/36938 -- GLOBAL_STEP: 1506750 | > loss_text_ce: 0.03407040983438492 (0.030429286827351434) | > loss_mel_ce: 3.0428466796875 (nan) | > loss: 0.012209989130496979 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.1371194444043654) | > loader_time: 0.0046 (0.00465361769379636)  --> TIME: 2025-02-27 02:26:57 -- STEP: 29280/36938 -- GLOBAL_STEP: 1506800 | > loss_text_ce: 0.03731102496385574 (0.03042757535303047) | > loss_mel_ce: 1.7320324182510376 (nan) | > loss: 0.0070212045684456825 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1371132020579019) | > loader_time: 0.0046 (0.004653654031740541)  --> TIME: 2025-02-27 02:27:06 -- STEP: 29330/36938 -- GLOBAL_STEP: 1506850 | > loss_text_ce: 0.029590141028165817 (0.030429685933682452) | > loss_mel_ce: 2.9475152492523193 (nan) | > loss: 0.01181391067802906 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13710376502627938) | > loader_time: 0.0047 (0.004653619622517669)  --> TIME: 2025-02-27 02:27:16 -- STEP: 29380/36938 -- GLOBAL_STEP: 1506900 | > loss_text_ce: 0.029278172180056572 (0.030429229243779575) | > loss_mel_ce: 1.8464813232421875 (nan) | > loss: 0.0074434904381632805 (nan) | > current_lr: 5e-06 | > step_time: 0.1076 (0.13710682526017887) | > loader_time: 0.0044 (0.004653590670079935)  --> TIME: 2025-02-27 02:27:25 -- STEP: 29430/36938 -- GLOBAL_STEP: 1506950 | > loss_text_ce: 0.031282082200050354 (0.030427225721279122) | > loss_mel_ce: 2.25887131690979 (nan) | > loss: 0.009087910875678062 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13710599720093344) | > loader_time: 0.0046 (0.0046536025894040895)  --> TIME: 2025-02-27 02:27:35 -- STEP: 29480/36938 -- GLOBAL_STEP: 1507000 | > loss_text_ce: 0.02718088962137699 (0.030426279084631868) | > loss_mel_ce: 2.1404037475585938 (nan) | > loss: 0.008601526729762554 (nan) | > current_lr: 5e-06 | > step_time: 0.1265 (0.13710826581174768) | > loader_time: 0.0046 (0.004653647076970392)  --> TIME: 2025-02-27 02:27:44 -- STEP: 29530/36938 -- GLOBAL_STEP: 1507050 | > loss_text_ce: 0.02679644338786602 (0.030425852243881578) | > loss_mel_ce: 2.6546521186828613 (nan) | > loss: 0.01064066868275404 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.1371022978274659) | > loader_time: 0.005 (0.004653668355182866)  --> TIME: 2025-02-27 02:27:53 -- STEP: 29580/36938 -- GLOBAL_STEP: 1507100 | > loss_text_ce: 0.0294241514056921 (0.03042561532687106) | > loss_mel_ce: 1.1374353170394897 (nan) | > loss: 0.004630395211279392 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.1371098966111686) | > loader_time: 0.0047 (0.004653673126860384)  --> TIME: 2025-02-27 02:28:03 -- STEP: 29630/36938 -- GLOBAL_STEP: 1507150 | > loss_text_ce: 0.02980264090001583 (0.030423744421495032) | > loss_mel_ce: 3.1080574989318848 (nan) | > loss: 0.012451826594769955 (nan) | > current_lr: 5e-06 | > step_time: 0.1811 (0.13710930043283995) | > loader_time: 0.0048 (0.004653657106302363)  --> TIME: 2025-02-27 02:28:12 -- STEP: 29680/36938 -- GLOBAL_STEP: 1507200 | > loss_text_ce: 0.029208047315478325 (0.03042577544689766) | > loss_mel_ce: 2.9688680171966553 (nan) | > loss: 0.011897127144038677 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.1371041632003868) | > loader_time: 0.0049 (0.004653633797586475)  --> TIME: 2025-02-27 02:28:21 -- STEP: 29730/36938 -- GLOBAL_STEP: 1507250 | > loss_text_ce: 0.035542357712984085 (0.03042437842183424) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13710760060601382) | > loader_time: 0.0046 (0.004653684659073207)  --> TIME: 2025-02-27 02:28:31 -- STEP: 29780/36938 -- GLOBAL_STEP: 1507300 | > loss_text_ce: 0.030225107446312904 (0.0304225128834543) | > loss_mel_ce: 2.310305118560791 (nan) | > loss: 0.0092878183349967 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.1371114940431959) | > loader_time: 0.0046 (0.004653689611510542)  --> TIME: 2025-02-27 02:28:40 -- STEP: 29830/36938 -- GLOBAL_STEP: 1507350 | > loss_text_ce: 0.030185699462890625 (0.030423157631057123) | > loss_mel_ce: 2.795177936553955 (nan) | > loss: 0.011211761273443699 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13711553479935457) | > loader_time: 0.0046 (0.004653673838578041)  --> TIME: 2025-02-27 02:28:50 -- STEP: 29880/36938 -- GLOBAL_STEP: 1507400 | > loss_text_ce: 0.025782635435461998 (0.030423200792954658) | > loss_mel_ce: 2.208441734313965 (nan) | > loss: 0.008865970186889172 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13710907557243718) | > loader_time: 0.0045 (0.0046535603333030485)  --> TIME: 2025-02-27 02:28:58 -- STEP: 29930/36938 -- GLOBAL_STEP: 1507450 | > loss_text_ce: 0.02107539214193821 (0.030420733407709837) | > loss_mel_ce: 2.8229522705078125 (nan) | > loss: 0.011285824701189995 (nan) | > current_lr: 5e-06 | > step_time: 0.1082 (0.13709594859274019) | > loader_time: 0.0045 (0.0046534061830178705)  --> TIME: 2025-02-27 02:29:08 -- STEP: 29980/36938 -- GLOBAL_STEP: 1507500 | > loss_text_ce: 0.03184742480516434 (0.03041980629094104) | > loss_mel_ce: 2.2840800285339355 (nan) | > loss: 0.009190188720822334 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.1370922820261122) | > loader_time: 0.0047 (0.004653404186851932)  --> TIME: 2025-02-27 02:29:17 -- STEP: 30030/36938 -- GLOBAL_STEP: 1507550 | > loss_text_ce: 0.029611686244606972 (0.03041953165378748) | > loss_mel_ce: 2.7466046810150146 (nan) | > loss: 0.011016732081770897 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13709001515731858) | > loader_time: 0.0046 (0.004653325638213743)  --> TIME: 2025-02-27 02:29:26 -- STEP: 30080/36938 -- GLOBAL_STEP: 1507600 | > loss_text_ce: 0.035074494779109955 (0.030421806866101218) | > loss_mel_ce: 2.170896291732788 (nan) | > loss: 0.008753852918744087 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.1370826742671279) | > loader_time: 0.0045 (0.004653457647308418)  --> TIME: 2025-02-27 02:29:35 -- STEP: 30130/36938 -- GLOBAL_STEP: 1507650 | > loss_text_ce: 0.027359582483768463 (0.030422720328867288) | > loss_mel_ce: 1.846027135848999 (nan) | > loss: 0.007434074766933918 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13708097668846297) | > loader_time: 0.0046 (0.004653408366740471)  --> TIME: 2025-02-27 02:29:45 -- STEP: 30180/36938 -- GLOBAL_STEP: 1507700 | > loss_text_ce: 0.02380451001226902 (0.0304210073913579) | > loss_mel_ce: 2.811189889907837 (nan) | > loss: 0.011249978095293045 (nan) | > current_lr: 5e-06 | > step_time: 0.1086 (0.13708135298816862) | > loader_time: 0.0045 (0.004653394135360796)  --> TIME: 2025-02-27 02:29:54 -- STEP: 30230/36938 -- GLOBAL_STEP: 1507750 | > loss_text_ce: 0.04510626569390297 (0.03042031280789432) | > loss_mel_ce: 2.482008218765259 (nan) | > loss: 0.010028231889009476 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13708308078262446) | > loader_time: 0.0046 (0.00465332998016441)  --> TIME: 2025-02-27 02:30:03 -- STEP: 30280/36938 -- GLOBAL_STEP: 1507800 | > loss_text_ce: 0.035151366144418716 (0.03041945423219166) | > loss_mel_ce: 2.5949132442474365 (nan) | > loss: 0.010436765849590302 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1370803576558245) | > loader_time: 0.0046 (0.004653570548073962)  --> TIME: 2025-02-27 02:30:13 -- STEP: 30330/36938 -- GLOBAL_STEP: 1507850 | > loss_text_ce: 0.026210401207208633 (0.030419014811244365) | > loss_mel_ce: 2.713181257247925 (nan) | > loss: 0.01087060198187828 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13707674170656714) | > loader_time: 0.0047 (0.004653579261960927)  --> TIME: 2025-02-27 02:30:22 -- STEP: 30380/36938 -- GLOBAL_STEP: 1507900 | > loss_text_ce: 0.028770169243216515 (0.030420364965877845) | > loss_mel_ce: 2.66658091545105 (nan) | > loss: 0.010695838369429111 (nan) | > current_lr: 5e-06 | > step_time: 0.1583 (0.13707212792014267) | > loader_time: 0.0047 (0.004653559859604201)  --> TIME: 2025-02-27 02:30:31 -- STEP: 30430/36938 -- GLOBAL_STEP: 1507950 | > loss_text_ce: 0.023480329662561417 (0.03042134538202905) | > loss_mel_ce: 3.0232295989990234 (nan) | > loss: 0.012090119533240795 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.13706824408915802) | > loader_time: 0.0047 (0.004653561557941546)  --> TIME: 2025-02-27 02:30:40 -- STEP: 30480/36938 -- GLOBAL_STEP: 1508000 | > loss_text_ce: 0.02663634717464447 (0.0304203684372385) | > loss_mel_ce: 2.4841253757476807 (nan) | > loss: 0.009963341057300568 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1370632732712385) | > loader_time: 0.0046 (0.004653540769899952)  --> TIME: 2025-02-27 02:30:50 -- STEP: 30530/36938 -- GLOBAL_STEP: 1508050 | > loss_text_ce: 0.03305993229150772 (0.030420281320213637) | > loss_mel_ce: 2.1934518814086914 (nan) | > loss: 0.008835364133119583 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13706402908260293) | > loader_time: 0.0046 (0.004653633152349196)  --> TIME: 2025-02-27 02:30:59 -- STEP: 30580/36938 -- GLOBAL_STEP: 1508100 | > loss_text_ce: 0.02730453386902809 (0.030421211050111894) | > loss_mel_ce: 3.1596343517303467 (nan) | > loss: 0.012646583840250969 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1370653793966177) | > loader_time: 0.0049 (0.004653633459476676)  --> TIME: 2025-02-27 02:31:09 -- STEP: 30630/36938 -- GLOBAL_STEP: 1508150 | > loss_text_ce: 0.02949969284236431 (0.03042223376610842) | > loss_mel_ce: 2.809034824371338 (nan) | > loss: 0.01126402709633112 (nan) | > current_lr: 5e-06 | > step_time: 0.1788 (0.13707170433520252) | > loader_time: 0.0049 (0.0046537066533446705)  --> TIME: 2025-02-27 02:31:18 -- STEP: 30680/36938 -- GLOBAL_STEP: 1508200 | > loss_text_ce: 0.02727263793349266 (0.03042397587569224) | > loss_mel_ce: 2.6266586780548096 (nan) | > loss: 0.010531473904848099 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13707809923836034) | > loader_time: 0.0047 (0.004653798375944143)  --> TIME: 2025-02-27 02:31:27 -- STEP: 30730/36938 -- GLOBAL_STEP: 1508250 | > loss_text_ce: 0.029736412689089775 (0.03042235049112099) | > loss_mel_ce: 2.9099087715148926 (nan) | > loss: 0.01166525948792696 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.1370784027197063) | > loader_time: 0.0047 (0.004653832232025005)  --> TIME: 2025-02-27 02:31:37 -- STEP: 30780/36938 -- GLOBAL_STEP: 1508300 | > loss_text_ce: 0.03552418574690819 (0.030420751598753796) | > loss_mel_ce: 2.6998932361602783 (nan) | > loss: 0.010854831896722317 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.1370678198035472) | > loader_time: 0.0046 (0.004653825877625293)  --> TIME: 2025-02-27 02:31:45 -- STEP: 30830/36938 -- GLOBAL_STEP: 1508350 | > loss_text_ce: 0.031099222600460052 (0.03042001389939745) | > loss_mel_ce: 2.2962698936462402 (nan) | > loss: 0.009235592558979988 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13705647755009526) | > loader_time: 0.0046 (0.004653785718249096)  --> TIME: 2025-02-27 02:31:55 -- STEP: 30880/36938 -- GLOBAL_STEP: 1508400 | > loss_text_ce: 0.03137071430683136 (0.030419370971156833) | > loss_mel_ce: 2.6372573375701904 (nan) | > loss: 0.010589794255793095 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.137057273429602) | > loader_time: 0.0047 (0.004653836084153396)  --> TIME: 2025-02-27 02:32:04 -- STEP: 30930/36938 -- GLOBAL_STEP: 1508450 | > loss_text_ce: 0.026675352826714516 (0.030418465153283763) | > loss_mel_ce: 1.2799038887023926 (nan) | > loss: 0.0051848385483026505 (nan) | > current_lr: 5e-06 | > step_time: 0.1081 (0.13704680880026482) | > loader_time: 0.0045 (0.0046538619982784135)  --> TIME: 2025-02-27 02:32:13 -- STEP: 30980/36938 -- GLOBAL_STEP: 1508500 | > loss_text_ce: 0.031053641811013222 (0.030419301237728208) | > loss_mel_ce: 2.4930813312530518 (nan) | > loss: 0.010016408748924732 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13704141700090858) | > loader_time: 0.0046 (0.004654004790846062)  --> TIME: 2025-02-27 02:32:22 -- STEP: 31030/36938 -- GLOBAL_STEP: 1508550 | > loss_text_ce: 0.0387156642973423 (0.0304192752514636) | > loss_mel_ce: 2.677043914794922 (nan) | > loss: 0.010776824317872524 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13704124603431372) | > loader_time: 0.0045 (0.004653952715668135)  --> TIME: 2025-02-27 02:32:32 -- STEP: 31080/36938 -- GLOBAL_STEP: 1508600 | > loss_text_ce: 0.033576350659132004 (0.030419996890096782) | > loss_mel_ce: 1.515626072883606 (nan) | > loss: 0.006147629115730524 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1370392810960848) | > loader_time: 0.0047 (0.004654019503366407)  --> TIME: 2025-02-27 02:32:41 -- STEP: 31130/36938 -- GLOBAL_STEP: 1508650 | > loss_text_ce: 0.025564588606357574 (0.030419489921008108) | > loss_mel_ce: 3.5203683376312256 (nan) | > loss: 0.014071163721382618 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.1370344072455889) | > loader_time: 0.0047 (0.004654078287516179)  --> TIME: 2025-02-27 02:32:50 -- STEP: 31180/36938 -- GLOBAL_STEP: 1508700 | > loss_text_ce: 0.035247307270765305 (0.030419672500327795) | > loss_mel_ce: 1.8089125156402588 (nan) | > loss: 0.007318094838410616 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13703268120272097) | > loader_time: 0.0046 (0.004654135460881131)  --> TIME: 2025-02-27 02:33:00 -- STEP: 31230/36938 -- GLOBAL_STEP: 1508750 | > loss_text_ce: 0.027081193402409554 (0.030420660573584776) | > loss_mel_ce: 2.5520896911621094 (nan) | > loss: 0.010234805755317211 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13703217065910778) | > loader_time: 0.0045 (0.004654173602135668)  --> TIME: 2025-02-27 02:33:09 -- STEP: 31280/36938 -- GLOBAL_STEP: 1508800 | > loss_text_ce: 0.025734292343258858 (0.030419783337015536) | > loss_mel_ce: 2.222170352935791 (nan) | > loss: 0.008920256979763508 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13702880586199817) | > loader_time: 0.0046 (0.0046542242436153165)  --> TIME: 2025-02-27 02:33:18 -- STEP: 31330/36938 -- GLOBAL_STEP: 1508850 | > loss_text_ce: 0.0225271787494421 (0.030418074614895426) | > loss_mel_ce: 2.743877649307251 (nan) | > loss: 0.010977798141539097 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.13703160365853864) | > loader_time: 0.0046 (0.004654258445851791)  --> TIME: 2025-02-27 02:33:27 -- STEP: 31380/36938 -- GLOBAL_STEP: 1508900 | > loss_text_ce: 0.02878461591899395 (0.03041817927026707) | > loss_mel_ce: 2.9626054763793945 (nan) | > loss: 0.011870595626533031 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13701734128767318) | > loader_time: 0.0046 (0.0046542342868559645)  --> TIME: 2025-02-27 02:33:36 -- STEP: 31430/36938 -- GLOBAL_STEP: 1508950 | > loss_text_ce: 0.031581077724695206 (0.030418261134954935) | > loss_mel_ce: 3.393540382385254 (nan) | > loss: 0.01359175331890583 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13700948164336432) | > loader_time: 0.0047 (0.004654202619025281)  --> TIME: 2025-02-27 02:33:46 -- STEP: 31480/36938 -- GLOBAL_STEP: 1509000 | > loss_text_ce: 0.030892735347151756 (0.03041904051947908) | > loss_mel_ce: 2.375201463699341 (nan) | > loss: 0.009547993540763855 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.1370119242913545) | > loader_time: 0.0046 (0.004654225430615851)  --> TIME: 2025-02-27 02:33:55 -- STEP: 31530/36938 -- GLOBAL_STEP: 1509050 | > loss_text_ce: 0.0412367507815361 (0.03041960882968773) | > loss_mel_ce: 0.1687825471162796 (nan) | > loss: 0.0008334099547937512 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.1370124999735047) | > loader_time: 0.0045 (0.004654311135652317)  --> TIME: 2025-02-27 02:34:04 -- STEP: 31580/36938 -- GLOBAL_STEP: 1509100 | > loss_text_ce: 0.024093741551041603 (0.030419595991960237) | > loss_mel_ce: 2.9700169563293457 (nan) | > loss: 0.011881391517817974 (nan) | > current_lr: 5e-06 | > step_time: 0.1782 (0.13700876285034838) | > loader_time: 0.0048 (0.00465432160862496)  --> TIME: 2025-02-27 02:34:14 -- STEP: 31630/36938 -- GLOBAL_STEP: 1509150 | > loss_text_ce: 0.033023614436388016 (0.030420050492914765) | > loss_mel_ce: 2.930823564529419 (nan) | > loss: 0.011761299334466457 (nan) | > current_lr: 5e-06 | > step_time: 0.1628 (0.1370116545058796) | > loader_time: 0.0049 (0.00465434691290145)  --> TIME: 2025-02-27 02:34:23 -- STEP: 31680/36938 -- GLOBAL_STEP: 1509200 | > loss_text_ce: 0.033246930688619614 (0.030421041956611185) | > loss_mel_ce: 2.5726699829101562 (nan) | > loss: 0.01034094113856554 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13700508698068553) | > loader_time: 0.0046 (0.004654458970433561)  --> TIME: 2025-02-27 02:34:32 -- STEP: 31730/36938 -- GLOBAL_STEP: 1509250 | > loss_text_ce: 0.028031734749674797 (0.0304227486227836) | > loss_mel_ce: 2.531291961669922 (nan) | > loss: 0.010156047530472279 (nan) | > current_lr: 5e-06 | > step_time: 0.179 (0.1370140821746655) | > loader_time: 0.0048 (0.004654479718080602)  --> TIME: 2025-02-27 02:34:42 -- STEP: 31780/36938 -- GLOBAL_STEP: 1509300 | > loss_text_ce: 0.038517504930496216 (0.03042243557690266) | > loss_mel_ce: 2.54925274848938 (nan) | > loss: 0.010268930345773697 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.1370176450871618) | > loader_time: 0.0047 (0.0046546947363415)  --> TIME: 2025-02-27 02:34:51 -- STEP: 31830/36938 -- GLOBAL_STEP: 1509350 | > loss_text_ce: 0.03889720141887665 (0.030423216705927066) | > loss_mel_ce: 3.466493844985962 (nan) | > loss: 0.013910283334553242 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13701631123263913) | > loader_time: 0.0046 (0.004654685439008859)  --> TIME: 2025-02-27 02:35:01 -- STEP: 31880/36938 -- GLOBAL_STEP: 1509400 | > loss_text_ce: 0.03629717603325844 (0.030423883432341293) | > loss_mel_ce: 2.404371738433838 (nan) | > loss: 0.009685194119811058 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13701776598896923) | > loader_time: 0.0045 (0.004654670255247021)  --> TIME: 2025-02-27 02:35:10 -- STEP: 31930/36938 -- GLOBAL_STEP: 1509450 | > loss_text_ce: 0.025757767260074615 (0.03042573858258999) | > loss_mel_ce: 2.7100107669830322 (nan) | > loss: 0.010856225155293941 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13701625662539987) | > loader_time: 0.0047 (0.004654709627514109)  --> TIME: 2025-02-27 02:35:19 -- STEP: 31980/36938 -- GLOBAL_STEP: 1509500 | > loss_text_ce: 0.040762241929769516 (0.030425155602670447) | > loss_mel_ce: 2.524801254272461 (nan) | > loss: 0.010180807672441006 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13701561524616093) | > loader_time: 0.0047 (0.004654751008864591)  --> TIME: 2025-02-27 02:35:29 -- STEP: 32030/36938 -- GLOBAL_STEP: 1509550 | > loss_text_ce: 0.030997799709439278 (0.03042676022652533) | > loss_mel_ce: 2.920452833175659 (nan) | > loss: 0.011712105944752693 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1370156018093091) | > loader_time: 0.0045 (0.0046547747313362454)  --> TIME: 2025-02-27 02:35:38 -- STEP: 32080/36938 -- GLOBAL_STEP: 1509600 | > loss_text_ce: 0.03200626000761986 (0.030425767489724056) | > loss_mel_ce: 2.7238075733184814 (nan) | > loss: 0.01093576941639185 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13701634014633893) | > loader_time: 0.0046 (0.0046547861021950625)  --> TIME: 2025-02-27 02:35:47 -- STEP: 32130/36938 -- GLOBAL_STEP: 1509650 | > loss_text_ce: 0.03521861881017685 (0.030423443813202743) | > loss_mel_ce: 3.1365017890930176 (nan) | > loss: 0.012586193159222603 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13700896228840218) | > loader_time: 0.0047 (0.004654731870699356)  --> TIME: 2025-02-27 02:35:56 -- STEP: 32180/36938 -- GLOBAL_STEP: 1509700 | > loss_text_ce: 0.037322673946619034 (0.03042370352125394) | > loss_mel_ce: 2.5144760608673096 (nan) | > loss: 0.010126186534762383 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13699933135635511) | > loader_time: 0.0046 (0.004654766973906852)  --> TIME: 2025-02-27 02:36:05 -- STEP: 32230/36938 -- GLOBAL_STEP: 1509750 | > loss_text_ce: 0.026662472635507584 (0.03042167388162802) | > loss_mel_ce: 2.219334602355957 (nan) | > loss: 0.008912688121199608 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13699743770643225) | > loader_time: 0.0046 (0.004654733682692244)  --> TIME: 2025-02-27 02:36:15 -- STEP: 32280/36938 -- GLOBAL_STEP: 1509800 | > loss_text_ce: 0.033194947987794876 (0.03042129108617648) | > loss_mel_ce: 2.7971150875091553 (nan) | > loss: 0.011231389828026295 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.137000732912303) | > loader_time: 0.0047 (0.0046548304135324015)  --> TIME: 2025-02-27 02:36:24 -- STEP: 32330/36938 -- GLOBAL_STEP: 1509850 | > loss_text_ce: 0.02985432557761669 (0.03042008851464991) | > loss_mel_ce: 2.646376371383667 (nan) | > loss: 0.010619963519275188 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13699908082491905) | > loader_time: 0.0046 (0.004654865658692547)  --> TIME: 2025-02-27 02:36:34 -- STEP: 32380/36938 -- GLOBAL_STEP: 1509900 | > loss_text_ce: 0.025419600307941437 (0.030419076971998502) | > loss_mel_ce: 2.6484375 (nan) | > loss: 0.010610545054078102 (nan) | > current_lr: 5e-06 | > step_time: 0.176 (0.13700147453394174) | > loader_time: 0.0049 (0.004654910580621549)  --> TIME: 2025-02-27 02:36:43 -- STEP: 32430/36938 -- GLOBAL_STEP: 1509950 | > loss_text_ce: 0.029984062537550926 (0.030418678431941418) | > loss_mel_ce: 0.13312824070453644 (nan) | > loss: 0.0006472710520029068 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13700337196036097) | > loader_time: 0.0045 (0.004654970942475812)  --> TIME: 2025-02-27 02:36:53 -- STEP: 32480/36938 -- GLOBAL_STEP: 1510000 | > loss_text_ce: 0.03156173601746559 (0.03041734960569032) | > loss_mel_ce: 2.64259934425354 (nan) | > loss: 0.01061175111681223 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.1370126509901345) | > loader_time: 0.0047 (0.0046549978661419985) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1510000.pth  --> TIME: 2025-02-27 02:37:08 -- STEP: 32530/36938 -- GLOBAL_STEP: 1510050 | > loss_text_ce: 0.03417334705591202 (0.030416417480616745) | > loss_mel_ce: 2.9113047122955322 (nan) | > loss: 0.011688405647873878 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13701152775861572) | > loader_time: 0.0046 (0.004655013559343216)  --> TIME: 2025-02-27 02:37:18 -- STEP: 32580/36938 -- GLOBAL_STEP: 1510100 | > loss_text_ce: 0.02455795742571354 (0.030416289037819597) | > loss_mel_ce: 2.0808863639831543 (nan) | > loss: 0.008354938589036465 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13701013587163732) | > loader_time: 0.0045 (0.004654977825099252)  --> TIME: 2025-02-27 02:37:27 -- STEP: 32630/36938 -- GLOBAL_STEP: 1510150 | > loss_text_ce: 0.04031763970851898 (0.03041667576783362) | > loss_mel_ce: 1.9664896726608276 (nan) | > loss: 0.007963521406054497 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.1370163363253021) | > loader_time: 0.0047 (0.004654954892157658)  --> TIME: 2025-02-27 02:37:36 -- STEP: 32680/36938 -- GLOBAL_STEP: 1510200 | > loss_text_ce: 0.02820499613881111 (0.030415504392820116) | > loss_mel_ce: 0.08298315107822418 (nan) | > loss: 0.0004412228299770504 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13701439778545124) | > loader_time: 0.0045 (0.00465487674371122)  --> TIME: 2025-02-27 02:37:46 -- STEP: 32730/36938 -- GLOBAL_STEP: 1510250 | > loss_text_ce: 0.031481944024562836 (0.03041284891728083) | > loss_mel_ce: 2.5226919651031494 (nan) | > loss: 0.010135611519217491 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13701602559624404) | > loader_time: 0.0046 (0.004654819660146001)  --> TIME: 2025-02-27 02:37:56 -- STEP: 32780/36938 -- GLOBAL_STEP: 1510300 | > loss_text_ce: 0.03191119804978371 (0.03041317905046505) | > loss_mel_ce: 1.3003597259521484 (nan) | > loss: 0.00528678996488452 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13702427312468152) | > loader_time: 0.0045 (0.0046551029154118535)  --> TIME: 2025-02-27 02:38:05 -- STEP: 32830/36938 -- GLOBAL_STEP: 1510350 | > loss_text_ce: 0.03354313597083092 (0.03041383925867342) | > loss_mel_ce: 2.941256284713745 (nan) | > loss: 0.011804760433733463 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.1370207144580096) | > loader_time: 0.0046 (0.004655129924766062)  --> TIME: 2025-02-27 02:38:14 -- STEP: 32880/36938 -- GLOBAL_STEP: 1510400 | > loss_text_ce: 0.032330188900232315 (0.030415058767858105) | > loss_mel_ce: 0.5689720511436462 (nan) | > loss: 0.0023861201480031013 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13702205440568874) | > loader_time: 0.0046 (0.004655170397166814)  --> TIME: 2025-02-27 02:38:24 -- STEP: 32930/36938 -- GLOBAL_STEP: 1510450 | > loss_text_ce: 0.026194337755441666 (0.03041530659814265) | > loss_mel_ce: 2.175882577896118 (nan) | > loss: 0.008738401345908642 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13702583661240206) | > loader_time: 0.0046 (0.004655261753614019)  --> TIME: 2025-02-27 02:38:33 -- STEP: 32980/36938 -- GLOBAL_STEP: 1510500 | > loss_text_ce: 0.0311304721981287 (0.03041685921930142) | > loss_mel_ce: 2.7624266147613525 (nan) | > loss: 0.01108554471284151 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13702696852426727) | > loader_time: 0.0047 (0.004655371787983288)  --> TIME: 2025-02-27 02:38:43 -- STEP: 33030/36938 -- GLOBAL_STEP: 1510550 | > loss_text_ce: 0.02963520772755146 (0.030417373389526724) | > loss_mel_ce: 2.5208842754364014 (nan) | > loss: 0.010121109895408154 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13703289100555796) | > loader_time: 0.0045 (0.004655398024669629)  --> TIME: 2025-02-27 02:38:52 -- STEP: 33080/36938 -- GLOBAL_STEP: 1510600 | > loss_text_ce: 0.02727561630308628 (0.030415318469869414) | > loss_mel_ce: 2.5292680263519287 (nan) | > loss: 0.010145015083253384 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13703124980828557) | > loader_time: 0.0045 (0.0046553567718566075)  --> TIME: 2025-02-27 02:39:01 -- STEP: 33130/36938 -- GLOBAL_STEP: 1510650 | > loss_text_ce: 0.03196684643626213 (0.03041206393379649) | > loss_mel_ce: 2.13226318359375 (nan) | > loss: 0.008588215336203575 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.1370268041980334) | > loader_time: 0.0047 (0.004655328885040169)  --> TIME: 2025-02-27 02:39:10 -- STEP: 33180/36938 -- GLOBAL_STEP: 1510700 | > loss_text_ce: 0.03275792673230171 (0.03041247343655727) | > loss_mel_ce: 1.2850093841552734 (nan) | > loss: 0.005229235626757145 (nan) | > current_lr: 5e-06 | > step_time: 0.1702 (0.13702710977586577) | > loader_time: 0.0048 (0.0046553848952396535)  --> TIME: 2025-02-27 02:39:20 -- STEP: 33230/36938 -- GLOBAL_STEP: 1510750 | > loss_text_ce: 0.03263043984770775 (0.030412418458901126) | > loss_mel_ce: 2.6710205078125 (nan) | > loss: 0.010728774592280388 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.1370275916731798) | > loader_time: 0.0047 (0.004655371693792834)  --> TIME: 2025-02-27 02:39:29 -- STEP: 33280/36938 -- GLOBAL_STEP: 1510800 | > loss_text_ce: 0.03082267753779888 (0.03041304956372747) | > loss_mel_ce: 1.7925328016281128 (nan) | > loss: 0.007235537748783827 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.13702705152189562) | > loader_time: 0.0046 (0.004655338358134054)  --> TIME: 2025-02-27 02:39:39 -- STEP: 33330/36938 -- GLOBAL_STEP: 1510850 | > loss_text_ce: 0.027557071298360825 (0.030412677452169374) | > loss_mel_ce: 2.15118408203125 (nan) | > loss: 0.00864579901099205 (nan) | > current_lr: 5e-06 | > step_time: 0.1751 (0.1370295043825716) | > loader_time: 0.0047 (0.004655263640663861)  --> TIME: 2025-02-27 02:39:48 -- STEP: 33380/36938 -- GLOBAL_STEP: 1510900 | > loss_text_ce: 0.05101238563656807 (0.030413423365884866) | > loss_mel_ce: 0.9351908564567566 (nan) | > loss: 0.003913505002856255 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13703041680920605) | > loader_time: 0.0045 (0.00465527117216639)  --> TIME: 2025-02-27 02:39:57 -- STEP: 33430/36938 -- GLOBAL_STEP: 1510950 | > loss_text_ce: 0.0321018248796463 (0.03041517644415233) | > loss_mel_ce: 2.674429178237915 (nan) | > loss: 0.010740202851593494 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13702914583586792) | > loader_time: 0.0045 (0.004655220506435897)  --> TIME: 2025-02-27 02:40:06 -- STEP: 33480/36938 -- GLOBAL_STEP: 1511000 | > loss_text_ce: 0.03341416269540787 (0.03041483370994474) | > loss_mel_ce: 2.935244560241699 (nan) | > loss: 0.01178039237856865 (nan) | > current_lr: 5e-06 | > step_time: 0.1436 (0.13702486752751686) | > loader_time: 0.0046 (0.004655166445667191)  --> TIME: 2025-02-27 02:40:16 -- STEP: 33530/36938 -- GLOBAL_STEP: 1511050 | > loss_text_ce: 0.01998792216181755 (0.030414734236493834) | > loss_mel_ce: 2.4776458740234375 (nan) | > loss: 0.009911245666444302 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13702384192588815) | > loader_time: 0.0045 (0.00465508005066557)  --> TIME: 2025-02-27 02:40:25 -- STEP: 33580/36938 -- GLOBAL_STEP: 1511100 | > loss_text_ce: 0.031129388138651848 (0.030414864073433703) | > loss_mel_ce: 2.5508229732513428 (nan) | > loss: 0.01024584285914898 (nan) | > current_lr: 5e-06 | > step_time: 0.1751 (0.13702898868994806) | > loader_time: 0.0048 (0.004655063514414129)  --> TIME: 2025-02-27 02:40:35 -- STEP: 33630/36938 -- GLOBAL_STEP: 1511150 | > loss_text_ce: 0.030640052631497383 (0.030416024952557526) | > loss_mel_ce: 2.694288492202759 (nan) | > loss: 0.010813209228217602 (nan) | > current_lr: 5e-06 | > step_time: 0.1935 (0.13703253459193177) | > loader_time: 0.0049 (0.004655043284098323)  --> TIME: 2025-02-27 02:40:44 -- STEP: 33680/36938 -- GLOBAL_STEP: 1511200 | > loss_text_ce: 0.02448725700378418 (0.030415822766696902) | > loss_mel_ce: 2.8394970893859863 (nan) | > loss: 0.011365017853677273 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1370303862522453) | > loader_time: 0.0046 (0.004654964408228929)  --> TIME: 2025-02-27 02:40:53 -- STEP: 33730/36938 -- GLOBAL_STEP: 1511250 | > loss_text_ce: 0.03407144546508789 (0.03041627344126176) | > loss_mel_ce: 2.54099178314209 (nan) | > loss: 0.010218505747616291 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.1370315732639273) | > loader_time: 0.0046 (0.004654910534028321)  --> TIME: 2025-02-27 02:41:03 -- STEP: 33780/36938 -- GLOBAL_STEP: 1511300 | > loss_text_ce: 0.03771281987428665 (0.03041558221910013) | > loss_mel_ce: 1.5735796689987183 (nan) | > loss: 0.006394017953425646 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13703609079634316) | > loader_time: 0.0045 (0.004654857983879977)  --> TIME: 2025-02-27 02:41:12 -- STEP: 33830/36938 -- GLOBAL_STEP: 1511350 | > loss_text_ce: 0.027584148570895195 (0.030416970676845716) | > loss_mel_ce: 2.986042022705078 (nan) | > loss: 0.01195883471518755 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13704108026230008) | > loader_time: 0.0046 (0.00465481868341342)  --> TIME: 2025-02-27 02:41:22 -- STEP: 33880/36938 -- GLOBAL_STEP: 1511400 | > loss_text_ce: 0.0387011356651783 (0.03041833018440941) | > loss_mel_ce: 3.1307802200317383 (nan) | > loss: 0.012577307410538197 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.1370478787909157) | > loader_time: 0.0047 (0.004654784016794988)  --> TIME: 2025-02-27 02:41:31 -- STEP: 33930/36938 -- GLOBAL_STEP: 1511450 | > loss_text_ce: 0.02830449678003788 (0.030420139862178883) | > loss_mel_ce: 0.0054340362548828125 (nan) | > loss: 0.00013388306251727045 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13704680477381864) | > loader_time: 0.0045 (0.00465469717311889)  --> TIME: 2025-02-27 02:41:41 -- STEP: 33980/36938 -- GLOBAL_STEP: 1511500 | > loss_text_ce: 0.03311161324381828 (0.030420333539914676) | > loss_mel_ce: 2.637859582901001 (nan) | > loss: 0.010599092580378056 (nan) | > current_lr: 5e-06 | > step_time: 0.1886 (0.1370531651396979) | > loader_time: 0.0048 (0.004654738585902655)  --> TIME: 2025-02-27 02:41:50 -- STEP: 34030/36938 -- GLOBAL_STEP: 1511550 | > loss_text_ce: 0.028362786397337914 (0.030420618150421536) | > loss_mel_ce: 2.9533495903015137 (nan) | > loss: 0.011832192540168762 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13705348089937314) | > loader_time: 0.0047 (0.004654702494293688)  --> TIME: 2025-02-27 02:42:00 -- STEP: 34080/36938 -- GLOBAL_STEP: 1511600 | > loss_text_ce: 0.023969531059265137 (0.03042196073933359) | > loss_mel_ce: 0.071319580078125 (nan) | > loss: 0.0003781314007937908 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.1370632651686115) | > loader_time: 0.0046 (0.00465477837223404)  --> TIME: 2025-02-27 02:42:09 -- STEP: 34130/36938 -- GLOBAL_STEP: 1511650 | > loss_text_ce: 0.03200976923108101 (0.030422556554364318) | > loss_mel_ce: 2.205845832824707 (nan) | > loss: 0.008880380541086197 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13706689015742576) | > loader_time: 0.0046 (0.004654822138583192)  --> TIME: 2025-02-27 02:42:18 -- STEP: 34180/36938 -- GLOBAL_STEP: 1511700 | > loss_text_ce: 0.0324595607817173 (0.030420953110258223) | > loss_mel_ce: 2.5260162353515625 (nan) | > loss: 0.010152681730687618 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13706335685209953) | > loader_time: 0.0046 (0.004654893490080965)  --> TIME: 2025-02-27 02:42:28 -- STEP: 34230/36938 -- GLOBAL_STEP: 1511750 | > loss_text_ce: 0.027154412120580673 (0.0304202706808795) | > loss_mel_ce: 3.064697265625 (nan) | > loss: 0.012269253842532635 (nan) | > current_lr: 5e-06 | > step_time: 0.1588 (0.13706873524721885) | > loader_time: 0.0045 (0.004654870157244622)  --> TIME: 2025-02-27 02:42:38 -- STEP: 34280/36938 -- GLOBAL_STEP: 1511800 | > loss_text_ce: 0.04273725301027298 (0.03042008396568995) | > loss_mel_ce: 2.521758556365967 (nan) | > loss: 0.01017657108604908 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.13707343966568694) | > loader_time: 0.0049 (0.004654839005464727)  --> TIME: 2025-02-27 02:42:47 -- STEP: 34330/36938 -- GLOBAL_STEP: 1511850 | > loss_text_ce: 0.03906520828604698 (0.030422417108271177) | > loss_mel_ce: 2.6200544834136963 (nan) | > loss: 0.010552062653005123 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13707951686835845) | > loader_time: 0.0046 (0.004654784123403785)  --> TIME: 2025-02-27 02:42:56 -- STEP: 34380/36938 -- GLOBAL_STEP: 1511900 | > loss_text_ce: 0.02998415194451809 (0.030423446984879265) | > loss_mel_ce: 2.5475423336029053 (nan) | > loss: 0.010228280909359455 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13708145510255826) | > loader_time: 0.0045 (0.004654707701973214)  --> TIME: 2025-02-27 02:43:06 -- STEP: 34430/36938 -- GLOBAL_STEP: 1511950 | > loss_text_ce: 0.02320251800119877 (0.030419497401243906) | > loss_mel_ce: 2.6614034175872803 (nan) | > loss: 0.010653198696672916 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.1370822368442343) | > loader_time: 0.0049 (0.004654703443562141)  --> TIME: 2025-02-27 02:43:15 -- STEP: 34480/36938 -- GLOBAL_STEP: 1512000 | > loss_text_ce: 0.014399084262549877 (0.03041816698335944) | > loss_mel_ce: 1.7645024061203003 (nan) | > loss: 0.007059133145958185 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13708936735953148) | > loader_time: 0.0046 (0.004654703477695776)  --> TIME: 2025-02-27 02:43:25 -- STEP: 34530/36938 -- GLOBAL_STEP: 1512050 | > loss_text_ce: 0.03499286249279976 (0.030418780172373705) | > loss_mel_ce: 2.497385263442993 (nan) | > loss: 0.010049120523035526 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.137088514755265) | > loader_time: 0.0046 (0.004654650083321491)  --> TIME: 2025-02-27 02:43:34 -- STEP: 34580/36938 -- GLOBAL_STEP: 1512100 | > loss_text_ce: 0.02954728528857231 (0.030418448728800642) | > loss_mel_ce: 0.0791015625 (nan) | > loss: 0.0004311462689656764 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13708662103544905) | > loader_time: 0.0047 (0.004654641603720285)  --> TIME: 2025-02-27 02:43:43 -- STEP: 34630/36938 -- GLOBAL_STEP: 1512150 | > loss_text_ce: 0.03851781040430069 (0.03041893086223231) | > loss_mel_ce: 2.0844831466674805 (nan) | > loss: 0.00842460710555315 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.1370880984378769) | > loader_time: 0.0046 (0.004654623627008628)  --> TIME: 2025-02-27 02:43:53 -- STEP: 34680/36938 -- GLOBAL_STEP: 1512200 | > loss_text_ce: 0.03652704134583473 (0.03042044141149897) | > loss_mel_ce: 2.447845458984375 (nan) | > loss: 0.009858622215688229 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13709437071658395) | > loader_time: 0.0046 (0.004654622077941916)  --> TIME: 2025-02-27 02:44:02 -- STEP: 34730/36938 -- GLOBAL_STEP: 1512250 | > loss_text_ce: 0.0294487327337265 (0.030419497815084923) | > loss_mel_ce: 2.8649635314941406 (nan) | > loss: 0.01148576382547617 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.13709809886527127) | > loader_time: 0.0049 (0.004654602581576679)  --> TIME: 2025-02-27 02:44:12 -- STEP: 34780/36938 -- GLOBAL_STEP: 1512300 | > loss_text_ce: 0.023406652733683586 (0.030420283611693532) | > loss_mel_ce: 2.0968799591064453 (nan) | > loss: 0.008413836359977722 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.1370973930336957) | > loader_time: 0.0046 (0.004654549393865027)  --> TIME: 2025-02-27 02:44:21 -- STEP: 34830/36938 -- GLOBAL_STEP: 1512350 | > loss_text_ce: 0.020576629787683487 (0.03041944233401154) | > loss_mel_ce: 0.9653751850128174 (nan) | > loss: 0.003912507556378841 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13710271681036226) | > loader_time: 0.0046 (0.004654481162499211)  --> TIME: 2025-02-27 02:44:31 -- STEP: 34880/36938 -- GLOBAL_STEP: 1512400 | > loss_text_ce: 0.027869531884789467 (0.030417202697792335) | > loss_mel_ce: 2.1925594806671143 (nan) | > loss: 0.0088112261146307 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13709937178213724) | > loader_time: 0.0045 (0.004654359591936866)  --> TIME: 2025-02-27 02:44:40 -- STEP: 34930/36938 -- GLOBAL_STEP: 1512450 | > loss_text_ce: 0.026403089985251427 (0.0304185220946374) | > loss_mel_ce: 2.549848794937134 (nan) | > loss: 0.010223222896456718 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1371023242807381) | > loader_time: 0.0045 (0.004654269303082415)  --> TIME: 2025-02-27 02:44:49 -- STEP: 34980/36938 -- GLOBAL_STEP: 1512500 | > loss_text_ce: 0.02493416704237461 (0.030416882258753098) | > loss_mel_ce: 1.364105224609375 (nan) | > loss: 0.005512061528861523 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13710007046617254) | > loader_time: 0.0045 (0.004654223623106742)  --> TIME: 2025-02-27 02:44:59 -- STEP: 35030/36938 -- GLOBAL_STEP: 1512550 | > loss_text_ce: 0.023500699549913406 (0.03041831450797393) | > loss_mel_ce: 2.717843532562256 (nan) | > loss: 0.010878350585699081 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1371059267377301) | > loader_time: 0.0045 (0.004654150937510822)  --> TIME: 2025-02-27 02:45:08 -- STEP: 35080/36938 -- GLOBAL_STEP: 1512600 | > loss_text_ce: 0.029189812019467354 (0.030418358775712705) | > loss_mel_ce: 3.2965140342712402 (nan) | > loss: 0.013197238557040691 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13710143160385094) | > loader_time: 0.0046 (0.004654033860975436)  --> TIME: 2025-02-27 02:45:17 -- STEP: 35130/36938 -- GLOBAL_STEP: 1512650 | > loss_text_ce: 0.034210555255413055 (0.030417372131152313) | > loss_mel_ce: 2.318929672241211 (nan) | > loss: 0.00933785829693079 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13709429118696445) | > loader_time: 0.0045 (0.0046539552321028365)  --> TIME: 2025-02-27 02:45:26 -- STEP: 35180/36938 -- GLOBAL_STEP: 1512700 | > loss_text_ce: 0.0367274172604084 (0.030417251826071755) | > loss_mel_ce: 2.6818811893463135 (nan) | > loss: 0.010788129642605782 (nan) | > current_lr: 5e-06 | > step_time: 0.1352 (0.1370888401663668) | > loader_time: 0.0046 (0.0046538577762905405)  --> TIME: 2025-02-27 02:45:36 -- STEP: 35230/36938 -- GLOBAL_STEP: 1512750 | > loss_text_ce: 0.02762087993323803 (0.0304152156195936) | > loss_mel_ce: 2.2535173892974854 (nan) | > loss: 0.009052135981619358 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.1370944780684863) | > loader_time: 0.0048 (0.004653813721881228)  --> TIME: 2025-02-27 02:45:45 -- STEP: 35280/36938 -- GLOBAL_STEP: 1512800 | > loss_text_ce: 0.021988827735185623 (0.030414681889691174) | > loss_mel_ce: 2.4620895385742188 (nan) | > loss: 0.00985745433717966 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13709564431995006) | > loader_time: 0.0045 (0.004653760040698427)  --> TIME: 2025-02-27 02:45:55 -- STEP: 35330/36938 -- GLOBAL_STEP: 1512850 | > loss_text_ce: 0.0305727981030941 (0.030413245502938422) | > loss_mel_ce: 2.942243814468384 (nan) | > loss: 0.011796892620623112 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13709195706178326) | > loader_time: 0.0046 (0.004653694209248105)  --> TIME: 2025-02-27 02:46:04 -- STEP: 35380/36938 -- GLOBAL_STEP: 1512900 | > loss_text_ce: 0.029488472267985344 (0.030410871020275527) | > loss_mel_ce: 2.9013936519622803 (nan) | > loss: 0.011630485765635967 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.13709285179729103) | > loader_time: 0.0048 (0.004653660916823883)  --> TIME: 2025-02-27 02:46:13 -- STEP: 35430/36938 -- GLOBAL_STEP: 1512950 | > loss_text_ce: 0.024227596819400787 (0.03041006680091696) | > loss_mel_ce: 2.621542453765869 (nan) | > loss: 0.01049908809363842 (nan) | > current_lr: 5e-06 | > step_time: 0.1667 (0.13708874410806451) | > loader_time: 0.0047 (0.004653586750477357)  --> TIME: 2025-02-27 02:46:23 -- STEP: 35480/36938 -- GLOBAL_STEP: 1513000 | > loss_text_ce: 0.0282101109623909 (0.03040987535538896) | > loss_mel_ce: 2.250007152557373 (nan) | > loss: 0.009040545672178268 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13709251932388292) | > loader_time: 0.0046 (0.004653563567992678)  --> TIME: 2025-02-27 02:46:32 -- STEP: 35530/36938 -- GLOBAL_STEP: 1513050 | > loss_text_ce: 0.018361439928412437 (0.03040960232622117) | > loss_mel_ce: 2.277336597442627 (nan) | > loss: 0.009109913371503353 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13709678721032081) | > loader_time: 0.0045 (0.00465350701310352)  --> TIME: 2025-02-27 02:46:42 -- STEP: 35580/36938 -- GLOBAL_STEP: 1513100 | > loss_text_ce: 0.028504950925707817 (0.030407564475385857) | > loss_mel_ce: 1.629638671875 (nan) | > loss: 0.006579935550689697 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.1370998146897163) | > loader_time: 0.0046 (0.004653457391791439)  --> TIME: 2025-02-27 02:46:51 -- STEP: 35630/36938 -- GLOBAL_STEP: 1513150 | > loss_text_ce: 0.022075707092881203 (0.030408302717691852) | > loss_mel_ce: 2.8179008960723877 (nan) | > loss: 0.011269749142229557 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13710390577281562) | > loader_time: 0.0046 (0.004653443916774768)  --> TIME: 2025-02-27 02:47:01 -- STEP: 35680/36938 -- GLOBAL_STEP: 1513200 | > loss_text_ce: 0.026659848168492317 (0.030408462126725738) | > loss_mel_ce: 2.9167051315307617 (nan) | > loss: 0.011680020950734615 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13710333450359152) | > loader_time: 0.0045 (0.004653413587086953)  --> TIME: 2025-02-27 02:47:10 -- STEP: 35730/36938 -- GLOBAL_STEP: 1513250 | > loss_text_ce: 0.029756516218185425 (0.030405430978056154) | > loss_mel_ce: 2.7058823108673096 (nan) | > loss: 0.01085571013391018 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13709903125031672) | > loader_time: 0.0045 (0.004653358085796755)  --> TIME: 2025-02-27 02:47:19 -- STEP: 35780/36938 -- GLOBAL_STEP: 1513300 | > loss_text_ce: 0.032515279948711395 (0.030405848770068664) | > loss_mel_ce: 2.76605224609375 (nan) | > loss: 0.011105427518486977 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.13709888322983785) | > loader_time: 0.0046 (0.004653313947563549)  --> TIME: 2025-02-27 02:47:29 -- STEP: 35830/36938 -- GLOBAL_STEP: 1513350 | > loss_text_ce: 0.026470402255654335 (0.030405800221105773) | > loss_mel_ce: 0.018829345703125 (nan) | > loss: 0.00017976091476157308 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13709842947119671) | > loader_time: 0.0046 (0.004653275981150373)  --> TIME: 2025-02-27 02:47:38 -- STEP: 35880/36938 -- GLOBAL_STEP: 1513400 | > loss_text_ce: 0.025425110012292862 (0.030404564530049815) | > loss_mel_ce: 0.0732421875 (nan) | > loss: 0.00039153691614046693 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.1370969357232724) | > loader_time: 0.0047 (0.004653251430262704)  --> TIME: 2025-02-27 02:47:47 -- STEP: 35930/36938 -- GLOBAL_STEP: 1513450 | > loss_text_ce: 0.0417742095887661 (0.03040451881746096) | > loss_mel_ce: 2.6750237941741943 (nan) | > loss: 0.010780945420265198 (nan) | > current_lr: 5e-06 | > step_time: 0.1794 (0.13710044153613285) | > loader_time: 0.0047 (0.004653284644602813)  --> TIME: 2025-02-27 02:47:57 -- STEP: 35980/36938 -- GLOBAL_STEP: 1513500 | > loss_text_ce: 0.030214915052056313 (0.030402533612263197) | > loss_mel_ce: 2.8638620376586914 (nan) | > loss: 0.011484433896839619 (nan) | > current_lr: 5e-06 | > step_time: 0.1443 (0.1371023070951385) | > loader_time: 0.0046 (0.004653192348384818)  --> TIME: 2025-02-27 02:48:06 -- STEP: 36030/36938 -- GLOBAL_STEP: 1513550 | > loss_text_ce: 0.029152801260352135 (0.03040197185030536) | > loss_mel_ce: 1.4255739450454712 (nan) | > loss: 0.005772725213319063 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13710884365015136) | > loader_time: 0.0047 (0.004653143035746815)  --> TIME: 2025-02-27 02:48:16 -- STEP: 36080/36938 -- GLOBAL_STEP: 1513600 | > loss_text_ce: 0.040218010544776917 (0.030403040082683307) | > loss_mel_ce: 0.0162353515625 (nan) | > loss: 0.00022402129252441227 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13711092251367468) | > loader_time: 0.0046 (0.004653071299889152)  --> TIME: 2025-02-27 02:48:25 -- STEP: 36130/36938 -- GLOBAL_STEP: 1513650 | > loss_text_ce: 0.029789483174681664 (0.030401748748701864) | > loss_mel_ce: 2.666285514831543 (nan) | > loss: 0.010698710568249226 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13710885908410367) | > loader_time: 0.0046 (0.00465298137802054)  --> TIME: 2025-02-27 02:48:35 -- STEP: 36180/36938 -- GLOBAL_STEP: 1513700 | > loss_text_ce: 0.029507776722311974 (0.03040152861628085) | > loss_mel_ce: 2.3042383193969727 (nan) | > loss: 0.00926089845597744 (nan) | > current_lr: 5e-06 | > step_time: 0.1263 (0.1371147453554305) | > loader_time: 0.005 (0.004652900950166669)  --> TIME: 2025-02-27 02:48:44 -- STEP: 36230/36938 -- GLOBAL_STEP: 1513750 | > loss_text_ce: 0.03021073341369629 (0.03039882395513151) | > loss_mel_ce: 1.6751683950424194 (nan) | > loss: 0.006767377723008394 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13711238350290486) | > loader_time: 0.0045 (0.004652860471960809)  --> TIME: 2025-02-27 02:48:53 -- STEP: 36280/36938 -- GLOBAL_STEP: 1513800 | > loss_text_ce: 0.02960645593702793 (0.03039844346970992) | > loss_mel_ce: 3.206352949142456 (nan) | > loss: 0.012841109186410904 (nan) | > current_lr: 5e-06 | > step_time: 0.1831 (0.1371118773830506) | > loader_time: 0.0048 (0.004652831723960803)  --> TIME: 2025-02-27 02:49:03 -- STEP: 36330/36938 -- GLOBAL_STEP: 1513850 | > loss_text_ce: 0.03464310243725777 (0.03039768114106658) | > loss_mel_ce: 0.1931326687335968 (nan) | > loss: 0.0009038721327669919 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.1371114261479149) | > loader_time: 0.0046 (0.004652747253462087)  --> TIME: 2025-02-27 02:49:11 -- STEP: 36380/36938 -- GLOBAL_STEP: 1513900 | > loss_text_ce: 0.03882719576358795 (0.030397228967119865) | > loss_mel_ce: 2.2678446769714355 (nan) | > loss: 0.009153460152447224 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13709825139703724) | > loader_time: 0.0046 (0.004652623615139082)  --> TIME: 2025-02-27 02:49:21 -- STEP: 36430/36938 -- GLOBAL_STEP: 1513950 | > loss_text_ce: 0.029909741133451462 (0.03039651654958274) | > loss_mel_ce: 2.8037309646606445 (nan) | > loss: 0.011244607158005238 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.13710026271782408) | > loader_time: 0.0045 (0.004652599544928308)  --> TIME: 2025-02-27 02:49:30 -- STEP: 36480/36938 -- GLOBAL_STEP: 1514000 | > loss_text_ce: 0.027241183444857597 (0.03039681505847909) | > loss_mel_ce: 2.6592578887939453 (nan) | > loss: 0.01066071167588234 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13709365152999015) | > loader_time: 0.0047 (0.004652525321172012)  --> TIME: 2025-02-27 02:49:39 -- STEP: 36530/36938 -- GLOBAL_STEP: 1514050 | > loss_text_ce: 0.03505055233836174 (0.030397066858064502) | > loss_mel_ce: 2.7186167240142822 (nan) | > loss: 0.010927252471446991 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13708959085269762) | > loader_time: 0.0044 (0.004652458303698527)  --> TIME: 2025-02-27 02:49:49 -- STEP: 36580/36938 -- GLOBAL_STEP: 1514100 | > loss_text_ce: 0.029526153579354286 (0.030395505979108246) | > loss_mel_ce: 3.131775140762329 (nan) | > loss: 0.012544847093522549 (nan) | > current_lr: 5e-06 | > step_time: 0.1879 (0.13709332772198146) | > loader_time: 0.0049 (0.004652456288262123)  --> TIME: 2025-02-27 02:49:58 -- STEP: 36630/36938 -- GLOBAL_STEP: 1514150 | > loss_text_ce: 0.02523418515920639 (0.030395354119561947) | > loss_mel_ce: 2.569401979446411 (nan) | > loss: 0.010296176187694073 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.13708871960021646) | > loader_time: 0.0046 (0.004652378261658869)  --> TIME: 2025-02-27 02:50:08 -- STEP: 36680/36938 -- GLOBAL_STEP: 1514200 | > loss_text_ce: 0.03386513516306877 (0.030394667543515535) | > loss_mel_ce: 2.8665850162506104 (nan) | > loss: 0.011509723961353302 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.1370957430407424) | > loader_time: 0.0046 (0.0046523585444318526)  --> TIME: 2025-02-27 02:50:17 -- STEP: 36730/36938 -- GLOBAL_STEP: 1514250 | > loss_text_ce: 0.029174014925956726 (0.030393173998404813) | > loss_mel_ce: 3.030888795852661 (nan) | > loss: 0.012143107131123543 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13710096042746286) | > loader_time: 0.0046 (0.0046523889208761925)  --> TIME: 2025-02-27 02:50:27 -- STEP: 36780/36938 -- GLOBAL_STEP: 1514300 | > loss_text_ce: 0.029700476676225662 (0.030394142638039096) | > loss_mel_ce: 2.9979336261749268 (nan) | > loss: 0.012014421634376049 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13710192812079036) | > loader_time: 0.0046 (0.00465236880845388)  --> TIME: 2025-02-27 02:50:36 -- STEP: 36830/36938 -- GLOBAL_STEP: 1514350 | > loss_text_ce: 0.029564179480075836 (0.030393265850160247) | > loss_mel_ce: 2.358656167984009 (nan) | > loss: 0.009477064944803715 (nan) | > current_lr: 5e-06 | > step_time: 0.1565 (0.13709746505655526) | > loader_time: 0.0046 (0.004652348284549206)  --> TIME: 2025-02-27 02:50:45 -- STEP: 36880/36938 -- GLOBAL_STEP: 1514400 | > loss_text_ce: 0.029130496084690094 (0.030392306711102487) | > loss_mel_ce: 1.5787996053695679 (nan) | > loss: 0.006380675360560417 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.137099330020572) | > loader_time: 0.0045 (0.004652449346416194)  --> TIME: 2025-02-27 02:50:54 -- STEP: 36930/36938 -- GLOBAL_STEP: 1514450 | > loss_text_ce: 0.032805461436510086 (0.030391402206502917) | > loss_mel_ce: 2.050966739654541 (nan) | > loss: 0.008268937468528748 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13709334727389863) | > loader_time: 0.0045 (0.004652412228877418)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.02053303529720496 (-0.0004994633174178649) | > avg_loss_text_ce: 0.029628621492682406 (-0.00010697718359309327) | > avg_loss_mel_ce: 2.4349505075133697 (-0.00890325290141547) | > avg_loss: 2.4645791280977805 (-0.009010238812702465) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_1514458.pth  > EPOCH: 42/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-27 02:51:21)   --> TIME: 2025-02-27 02:51:29 -- STEP: 42/36938 -- GLOBAL_STEP: 1514500 | > loss_text_ce: 0.029963478446006775 (0.028413362268890654) | > loss_mel_ce: 2.2829489707946777 (nan) | > loss: 0.009178224951028824 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13740322703406924) | > loader_time: 0.0047 (0.004843592643737793)  --> TIME: 2025-02-27 02:51:38 -- STEP: 92/36938 -- GLOBAL_STEP: 1514550 | > loss_text_ce: 0.02917664498090744 (0.02963390658893015) | > loss_mel_ce: 2.7213492393493652 (nan) | > loss: 0.010914785787463188 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.13454346553139065) | > loader_time: 0.0049 (0.004746703997902247)  --> TIME: 2025-02-27 02:51:47 -- STEP: 142/36938 -- GLOBAL_STEP: 1514600 | > loss_text_ce: 0.0439411960542202 (0.03023551423496134) | > loss_mel_ce: 2.539924144744873 (nan) | > loss: 0.010253435000777245 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13314930150206664) | > loader_time: 0.0047 (0.004731077543446713)  --> TIME: 2025-02-27 02:51:56 -- STEP: 192/36938 -- GLOBAL_STEP: 1514650 | > loss_text_ce: 0.028576092794537544 (0.03016455909528304) | > loss_mel_ce: 2.4175498485565186 (nan) | > loss: 0.009706850163638592 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.13225078831116358) | > loader_time: 0.0049 (0.004719184090693793)  --> TIME: 2025-02-27 02:52:05 -- STEP: 242/36938 -- GLOBAL_STEP: 1514700 | > loss_text_ce: 0.02388610877096653 (0.030380689050263362) | > loss_mel_ce: 2.2208166122436523 (nan) | > loss: 0.008907550945878029 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13285844779211622) | > loader_time: 0.0046 (0.004722869100649497)  --> TIME: 2025-02-27 02:52:14 -- STEP: 292/36938 -- GLOBAL_STEP: 1514750 | > loss_text_ce: 0.029670801013708115 (0.030298916868584817) | > loss_mel_ce: 1.921019434928894 (nan) | > loss: 0.007740834727883339 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1332247240902626) | > loader_time: 0.0045 (0.0047127645309657275)  --> TIME: 2025-02-27 02:52:23 -- STEP: 342/36938 -- GLOBAL_STEP: 1514800 | > loss_text_ce: 0.02673516236245632 (0.030393534108495327) | > loss_mel_ce: 2.574324369430542 (nan) | > loss: 0.01032166462391615 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13309788913057558) | > loader_time: 0.0046 (0.004703233813681798)  --> TIME: 2025-02-27 02:52:33 -- STEP: 392/36938 -- GLOBAL_STEP: 1514850 | > loss_text_ce: 0.03240594267845154 (0.030546550627094597) | > loss_mel_ce: 2.9802815914154053 (nan) | > loss: 0.011955109424889088 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13388392694142395) | > loader_time: 0.0046 (0.004704791064165075)  --> TIME: 2025-02-27 02:52:42 -- STEP: 442/36938 -- GLOBAL_STEP: 1514900 | > loss_text_ce: 0.024128368124365807 (0.030429561906894543) | > loss_mel_ce: 1.5288753509521484 (nan) | > loss: 0.006162713281810284 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13448449767013473) | > loader_time: 0.0046 (0.004701974165385661)  --> TIME: 2025-02-27 02:52:52 -- STEP: 492/36938 -- GLOBAL_STEP: 1514950 | > loss_text_ce: 0.024342436343431473 (0.030440383622937692) | > loss_mel_ce: 2.18064284324646 (nan) | > loss: 0.008749942295253277 (nan) | > current_lr: 5e-06 | > step_time: 0.1516 (0.13496847075175455) | > loader_time: 0.0046 (0.004698220791855475)  --> TIME: 2025-02-27 02:53:01 -- STEP: 542/36938 -- GLOBAL_STEP: 1515000 | > loss_text_ce: 0.032895296812057495 (0.030421816659713254) | > loss_mel_ce: 2.8458480834960938 (nan) | > loss: 0.011423585936427116 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13538891582911297) | > loader_time: 0.0046 (0.004690993755945856)  --> TIME: 2025-02-27 02:53:11 -- STEP: 592/36938 -- GLOBAL_STEP: 1515050 | > loss_text_ce: 0.020234674215316772 (0.03040563389879173) | > loss_mel_ce: 2.3908491134643555 (nan) | > loss: 0.00956779345870018 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13574065791594012) | > loader_time: 0.0045 (0.004691561331620084)  --> TIME: 2025-02-27 02:53:20 -- STEP: 642/36938 -- GLOBAL_STEP: 1515100 | > loss_text_ce: 0.020704638212919235 (0.03048607401950198) | > loss_mel_ce: 2.7026476860046387 (nan) | > loss: 0.010806954465806484 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13608431964648468) | > loader_time: 0.0048 (0.004689963807198116)  --> TIME: 2025-02-27 02:53:30 -- STEP: 692/36938 -- GLOBAL_STEP: 1515150 | > loss_text_ce: 0.030906183645129204 (0.030483037027438854) | > loss_mel_ce: 2.59938645362854 (nan) | > loss: 0.010437670163810253 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13633585114010488) | > loader_time: 0.0045 (0.004687529767868836)  --> TIME: 2025-02-27 02:53:39 -- STEP: 742/36938 -- GLOBAL_STEP: 1515200 | > loss_text_ce: 0.033094070851802826 (0.030421054480230872) | > loss_mel_ce: 3.3257040977478027 (nan) | > loss: 0.013328565284609795 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13644086607704253) | > loader_time: 0.0046 (0.004686572802355989)  --> TIME: 2025-02-27 02:53:48 -- STEP: 792/36938 -- GLOBAL_STEP: 1515250 | > loss_text_ce: 0.025723716244101524 (0.030452949571131607) | > loss_mel_ce: 2.765411376953125 (nan) | > loss: 0.011075933463871479 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13645969436626224) | > loader_time: 0.0046 (0.004684440114281395)  --> TIME: 2025-02-27 02:53:58 -- STEP: 842/36938 -- GLOBAL_STEP: 1515300 | > loss_text_ce: 0.029135102406144142 (0.030402680516083705) | > loss_mel_ce: 2.4965858459472656 (nan) | > loss: 0.010022702626883984 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.13658526023040177) | > loader_time: 0.0048 (0.004682000345968576)  --> TIME: 2025-02-27 02:54:07 -- STEP: 892/36938 -- GLOBAL_STEP: 1515350 | > loss_text_ce: 0.030485188588500023 (0.030358729866279015) | > loss_mel_ce: 3.1155030727386475 (nan) | > loss: 0.012484081089496613 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13659486107762067) | > loader_time: 0.0047 (0.004679969875267268)  --> TIME: 2025-02-27 02:54:16 -- STEP: 942/36938 -- GLOBAL_STEP: 1515400 | > loss_text_ce: 0.035098448395729065 (0.030363189178121506) | > loss_mel_ce: 2.118424654006958 (nan) | > loss: 0.008545727469027042 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13656977078464122) | > loader_time: 0.0046 (0.004677252911205252)  --> TIME: 2025-02-27 02:54:26 -- STEP: 992/36938 -- GLOBAL_STEP: 1515450 | > loss_text_ce: 0.026182202622294426 (0.030322315983451722) | > loss_mel_ce: 3.0367517471313477 (nan) | > loss: 0.012154500000178814 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.13651707600201324) | > loader_time: 0.0048 (0.004675961550204984)  --> TIME: 2025-02-27 02:54:35 -- STEP: 1042/36938 -- GLOBAL_STEP: 1515500 | > loss_text_ce: 0.02127111330628395 (0.030299311249039628) | > loss_mel_ce: 1.5291072130203247 (nan) | > loss: 0.006152295507490635 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1364983034225434) | > loader_time: 0.0045 (0.004673216942404602)  --> TIME: 2025-02-27 02:54:44 -- STEP: 1092/36938 -- GLOBAL_STEP: 1515550 | > loss_text_ce: 0.04030214622616768 (0.030310601576814776) | > loss_mel_ce: 2.63562273979187 (nan) | > loss: 0.010618750005960464 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.1364952548519595) | > loader_time: 0.0047 (0.004672109862386963)  --> TIME: 2025-02-27 02:54:53 -- STEP: 1142/36938 -- GLOBAL_STEP: 1515600 | > loss_text_ce: 0.0306279007345438 (0.030298776757428765) | > loss_mel_ce: 3.3242688179016113 (nan) | > loss: 0.013313082978129387 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13646135104725535) | > loader_time: 0.0046 (0.004672948408043321)  --> TIME: 2025-02-27 02:55:03 -- STEP: 1192/36938 -- GLOBAL_STEP: 1515650 | > loss_text_ce: 0.03358373045921326 (0.0303353626826378) | > loss_mel_ce: 2.8633615970611572 (nan) | > loss: 0.01149581465870142 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.1363663583393865) | > loader_time: 0.0046 (0.00467343798419773)  --> TIME: 2025-02-27 02:55:12 -- STEP: 1242/36938 -- GLOBAL_STEP: 1515700 | > loss_text_ce: 0.027573270723223686 (0.03032851729904948) | > loss_mel_ce: 2.5726115703582764 (nan) | > loss: 0.010318194516003132 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13640273461979188) | > loader_time: 0.0044 (0.004673777954973841)  --> TIME: 2025-02-27 02:55:22 -- STEP: 1292/36938 -- GLOBAL_STEP: 1515750 | > loss_text_ce: 0.03005228377878666 (0.030316776605380624) | > loss_mel_ce: 3.010718822479248 (nan) | > loss: 0.01206655241549015 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13644547783553435) | > loader_time: 0.0046 (0.004672988280423283)  --> TIME: 2025-02-27 02:55:31 -- STEP: 1342/36938 -- GLOBAL_STEP: 1515800 | > loss_text_ce: 0.02753634564578533 (0.03038948746509128) | > loss_mel_ce: 3.2245261669158936 (nan) | > loss: 0.01290501095354557 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13655414453384268) | > loader_time: 0.0045 (0.004678448870356143)  --> TIME: 2025-02-27 02:55:40 -- STEP: 1392/36938 -- GLOBAL_STEP: 1515850 | > loss_text_ce: 0.0363253578543663 (0.030410422993711484) | > loss_mel_ce: 2.346092939376831 (nan) | > loss: 0.009454041719436646 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13642767654068197) | > loader_time: 0.0045 (0.004675804198473348)  --> TIME: 2025-02-27 02:55:50 -- STEP: 1442/36938 -- GLOBAL_STEP: 1515900 | > loss_text_ce: 0.03196999058127403 (0.030379388715823834) | > loss_mel_ce: 2.720597505569458 (nan) | > loss: 0.010922887362539768 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13649365475372896) | > loader_time: 0.0047 (0.004673743380256892)  --> TIME: 2025-02-27 02:55:59 -- STEP: 1492/36938 -- GLOBAL_STEP: 1515950 | > loss_text_ce: 0.028847405686974525 (0.03037985949277698) | > loss_mel_ce: 1.48101806640625 (nan) | > loss: 0.005991530138999224 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.1364814293927869) | > loader_time: 0.0045 (0.004671707709417259)  --> TIME: 2025-02-27 02:56:09 -- STEP: 1542/36938 -- GLOBAL_STEP: 1516000 | > loss_text_ce: 0.02945864386856556 (0.030401588136743363) | > loss_mel_ce: 2.526242256164551 (nan) | > loss: 0.010141671635210514 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13664880383339406) | > loader_time: 0.0049 (0.004672325384137534)  --> TIME: 2025-02-27 02:56:18 -- STEP: 1592/36938 -- GLOBAL_STEP: 1516050 | > loss_text_ce: 0.0280288215726614 (0.030414666873449694) | > loss_mel_ce: 2.42273211479187 (nan) | > loss: 0.009725241921842098 (nan) | > current_lr: 5e-06 | > step_time: 0.1583 (0.1366536503461141) | > loader_time: 0.0047 (0.004673111229086634)  --> TIME: 2025-02-27 02:56:27 -- STEP: 1642/36938 -- GLOBAL_STEP: 1516100 | > loss_text_ce: 0.0301944799721241 (0.030371447406305357) | > loss_mel_ce: 3.0384557247161865 (nan) | > loss: 0.012177184224128723 (nan) | > current_lr: 5e-06 | > step_time: 0.1661 (0.13661649020958208) | > loader_time: 0.0047 (0.004672828458258639)  --> TIME: 2025-02-27 02:56:37 -- STEP: 1692/36938 -- GLOBAL_STEP: 1516150 | > loss_text_ce: 0.038995929062366486 (0.030417088272234966) | > loss_mel_ce: 3.125088930130005 (nan) | > loss: 0.012555893510580063 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13663894360792583) | > loader_time: 0.0046 (0.004672258176420315)  --> TIME: 2025-02-27 02:56:46 -- STEP: 1742/36938 -- GLOBAL_STEP: 1516200 | > loss_text_ce: 0.02693304978311062 (0.03038918993026275) | > loss_mel_ce: 2.363553762435913 (nan) | > loss: 0.009486058726906776 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13684625767949848) | > loader_time: 0.0047 (0.004672220188495623)  --> TIME: 2025-02-27 02:56:55 -- STEP: 1792/36938 -- GLOBAL_STEP: 1516250 | > loss_text_ce: 0.01919671893119812 (0.030350275242588914) | > loss_mel_ce: 1.7252776622772217 (nan) | > loss: 0.006922517903149128 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.136742587733482) | > loader_time: 0.0045 (0.00467324855604342)  --> TIME: 2025-02-27 02:57:05 -- STEP: 1842/36938 -- GLOBAL_STEP: 1516300 | > loss_text_ce: 0.0262395478785038 (0.030363282004227693) | > loss_mel_ce: 2.85534930229187 (nan) | > loss: 0.011434877291321754 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.136899832017259) | > loader_time: 0.0045 (0.004672404220904124)  --> TIME: 2025-02-27 02:57:15 -- STEP: 1892/36938 -- GLOBAL_STEP: 1516350 | > loss_text_ce: 0.030738797038793564 (0.03036003811068313) | > loss_mel_ce: 2.8490419387817383 (nan) | > loss: 0.011427702382206917 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13690646569270196) | > loader_time: 0.0046 (0.004672994447308917)  --> TIME: 2025-02-27 02:57:24 -- STEP: 1942/36938 -- GLOBAL_STEP: 1516400 | > loss_text_ce: 0.02606099285185337 (0.030353131871881496) | > loss_mel_ce: 1.2018581628799438 (nan) | > loss: 0.004872695077210665 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13690150471597975) | > loader_time: 0.0045 (0.004673881462010738)  --> TIME: 2025-02-27 02:57:33 -- STEP: 1992/36938 -- GLOBAL_STEP: 1516450 | > loss_text_ce: 0.028261348605155945 (0.030354951090370137) | > loss_mel_ce: 2.4708876609802246 (nan) | > loss: 0.009917259216308594 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13692066923202795) | > loader_time: 0.0047 (0.004673156752643815)  --> TIME: 2025-02-27 02:57:43 -- STEP: 2042/36938 -- GLOBAL_STEP: 1516500 | > loss_text_ce: 0.03740280494093895 (0.030360780566945366) | > loss_mel_ce: 2.5826876163482666 (nan) | > loss: 0.010397184640169144 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13695454072064947) | > loader_time: 0.0046 (0.004673426228335508)  --> TIME: 2025-02-27 02:57:52 -- STEP: 2092/36938 -- GLOBAL_STEP: 1516550 | > loss_text_ce: 0.03026074729859829 (0.030343000554474312) | > loss_mel_ce: 2.4763519763946533 (nan) | > loss: 0.009946877136826515 (nan) | > current_lr: 5e-06 | > step_time: 0.1502 (0.13700492295434813) | > loader_time: 0.0047 (0.004674631254622859)  --> TIME: 2025-02-27 02:58:02 -- STEP: 2142/36938 -- GLOBAL_STEP: 1516600 | > loss_text_ce: 0.034534744918346405 (0.0303352092454483) | > loss_mel_ce: 3.0511484146118164 (nan) | > loss: 0.012244774959981441 (nan) | > current_lr: 5e-06 | > step_time: 0.1915 (0.13704489392726726) | > loader_time: 0.0049 (0.004674951680679393)  --> TIME: 2025-02-27 02:58:11 -- STEP: 2192/36938 -- GLOBAL_STEP: 1516650 | > loss_text_ce: 0.036946143954992294 (0.030302226801749564) | > loss_mel_ce: 2.593360424041748 (nan) | > loss: 0.010437725111842155 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13699477943625793) | > loader_time: 0.0045 (0.004676369093630437)  --> TIME: 2025-02-27 02:58:20 -- STEP: 2242/36938 -- GLOBAL_STEP: 1516700 | > loss_text_ce: 0.024588678032159805 (0.03031551491063716) | > loss_mel_ce: 1.958202600479126 (nan) | > loss: 0.00786822009831667 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13699790104279896) | > loader_time: 0.0046 (0.004676751214196699)  --> TIME: 2025-02-27 02:58:29 -- STEP: 2292/36938 -- GLOBAL_STEP: 1516750 | > loss_text_ce: 0.02096656523644924 (0.030322686785321688) | > loss_mel_ce: 2.380232334136963 (nan) | > loss: 0.00952856708317995 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13694323139040906) | > loader_time: 0.0045 (0.004675837711513976)  --> TIME: 2025-02-27 02:58:38 -- STEP: 2342/36938 -- GLOBAL_STEP: 1516800 | > loss_text_ce: 0.03274546191096306 (0.030321691543131073) | > loss_mel_ce: 4.136458396911621 (nan) | > loss: 0.016544459387660027 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1368422218106123) | > loader_time: 0.0045 (0.004674062781818492)  --> TIME: 2025-02-27 02:58:47 -- STEP: 2392/36938 -- GLOBAL_STEP: 1516850 | > loss_text_ce: 0.02857968956232071 (0.030324984023372414) | > loss_mel_ce: 1.8321533203125 (nan) | > loss: 0.00738386157900095 (nan) | > current_lr: 5e-06 | > step_time: 0.1558 (0.13678508329152297) | > loader_time: 0.0046 (0.004672280322747879)  --> TIME: 2025-02-27 02:58:57 -- STEP: 2442/36938 -- GLOBAL_STEP: 1516900 | > loss_text_ce: 0.027815457433462143 (0.030327925705682733) | > loss_mel_ce: 2.755298376083374 (nan) | > loss: 0.011044102720916271 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.1367284977270093) | > loader_time: 0.005 (0.0046710399796394434)  --> TIME: 2025-02-27 02:59:06 -- STEP: 2492/36938 -- GLOBAL_STEP: 1516950 | > loss_text_ce: 0.023317279294133186 (0.030326281143143105) | > loss_mel_ce: 2.7062439918518066 (nan) | > loss: 0.010831593535840511 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13672127367788148) | > loader_time: 0.0046 (0.004671986279097256)  --> TIME: 2025-02-27 02:59:16 -- STEP: 2542/36938 -- GLOBAL_STEP: 1517000 | > loss_text_ce: 0.030722307041287422 (0.030312279578192814) | > loss_mel_ce: 2.7026960849761963 (nan) | > loss: 0.010846898891031742 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.1368639058338952) | > loader_time: 0.0046 (0.00467138614924661)  --> TIME: 2025-02-27 02:59:25 -- STEP: 2592/36938 -- GLOBAL_STEP: 1517050 | > loss_text_ce: 0.011341173201799393 (0.030312135729746156) | > loss_mel_ce: 1.2738933563232422 (nan) | > loss: 0.0051001375541090965 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1368373886302669) | > loader_time: 0.0053 (0.0046728880685052735)  --> TIME: 2025-02-27 02:59:34 -- STEP: 2642/36938 -- GLOBAL_STEP: 1517100 | > loss_text_ce: 0.03415267914533615 (0.03030783228786277) | > loss_mel_ce: 2.5062482357025146 (nan) | > loss: 0.010080956853926182 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13686963548089986) | > loader_time: 0.0045 (0.0046729950359786256)  --> TIME: 2025-02-27 02:59:43 -- STEP: 2692/36938 -- GLOBAL_STEP: 1517150 | > loss_text_ce: 0.01805993728339672 (0.03029497592214545) | > loss_mel_ce: 0.7282830476760864 (nan) | > loss: 0.0029616786632686853 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13678373704162103) | > loader_time: 0.0045 (0.004673477887753036)  --> TIME: 2025-02-27 02:59:53 -- STEP: 2742/36938 -- GLOBAL_STEP: 1517200 | > loss_text_ce: 0.025709426030516624 (0.030282829248942135) | > loss_mel_ce: 3.3172922134399414 (nan) | > loss: 0.013265879824757576 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13679354013447206) | > loader_time: 0.0046 (0.004676053856168668)  --> TIME: 2025-02-27 03:00:02 -- STEP: 2792/36938 -- GLOBAL_STEP: 1517250 | > loss_text_ce: 0.03448135405778885 (0.030276566696484014) | > loss_mel_ce: 2.268071413040161 (nan) | > loss: 0.009137114509940147 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13680287939429672) | > loader_time: 0.0045 (0.004676729696869506)  --> TIME: 2025-02-27 03:00:12 -- STEP: 2842/36938 -- GLOBAL_STEP: 1517300 | > loss_text_ce: 0.032499443739652634 (0.030261081594903018) | > loss_mel_ce: 2.865034341812134 (nan) | > loss: 0.01149815134704113 (nan) | > current_lr: 5e-06 | > step_time: 0.1486 (0.1367573128209665) | > loader_time: 0.0046 (0.004676078424581244)  --> TIME: 2025-02-27 03:00:21 -- STEP: 2892/36938 -- GLOBAL_STEP: 1517350 | > loss_text_ce: 0.03533600643277168 (0.0302626531620099) | > loss_mel_ce: 2.3459489345550537 (nan) | > loss: 0.009449544362723827 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13673364530790272) | > loader_time: 0.0048 (0.004676991396095406)  --> TIME: 2025-02-27 03:00:30 -- STEP: 2942/36938 -- GLOBAL_STEP: 1517400 | > loss_text_ce: 0.031382251530885696 (0.030271614321781656) | > loss_mel_ce: 2.2741246223449707 (nan) | > loss: 0.009148837998509407 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13667471355357885) | > loader_time: 0.0045 (0.004677403143358255)  --> TIME: 2025-02-27 03:00:39 -- STEP: 2992/36938 -- GLOBAL_STEP: 1517450 | > loss_text_ce: 0.03466211259365082 (0.030267590552912078) | > loss_mel_ce: 1.949481725692749 (nan) | > loss: 0.007873587310314178 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13662727320576779) | > loader_time: 0.0046 (0.004676848889034694)  --> TIME: 2025-02-27 03:00:48 -- STEP: 3042/36938 -- GLOBAL_STEP: 1517500 | > loss_text_ce: 0.03031783737242222 (0.03027064095965827) | > loss_mel_ce: 2.5225772857666016 (nan) | > loss: 0.010130536742508411 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13653880040321179) | > loader_time: 0.0046 (0.004675795183927896)  --> TIME: 2025-02-27 03:00:58 -- STEP: 3092/36938 -- GLOBAL_STEP: 1517550 | > loss_text_ce: 0.0322597399353981 (0.030279060756729828) | > loss_mel_ce: 2.9513981342315674 (nan) | > loss: 0.01183991227298975 (nan) | > current_lr: 5e-06 | > step_time: 0.1513 (0.13662730745325463) | > loader_time: 0.0047 (0.004675685819648062)  --> TIME: 2025-02-27 03:01:07 -- STEP: 3142/36938 -- GLOBAL_STEP: 1517600 | > loss_text_ce: 0.02964102476835251 (0.030264024823112027) | > loss_mel_ce: 1.91909658908844 (nan) | > loss: 0.007733086124062538 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1366221705946026) | > loader_time: 0.0046 (0.004675520445267007)  --> TIME: 2025-02-27 03:01:16 -- STEP: 3192/36938 -- GLOBAL_STEP: 1517650 | > loss_text_ce: 0.030180739238858223 (0.03026449881473038) | > loss_mel_ce: 2.8795433044433594 (nan) | > loss: 0.011546524241566658 (nan) | > current_lr: 5e-06 | > step_time: 0.1262 (0.13662497508794794) | > loader_time: 0.0046 (0.004675510906635377)  --> TIME: 2025-02-27 03:01:26 -- STEP: 3242/36938 -- GLOBAL_STEP: 1517700 | > loss_text_ce: 0.03035116381943226 (0.03025318121791511) | > loss_mel_ce: 3.101033926010132 (nan) | > loss: 0.012426132336258888 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13666670290060398) | > loader_time: 0.0046 (0.0046756243279214215)  --> TIME: 2025-02-27 03:01:35 -- STEP: 3292/36938 -- GLOBAL_STEP: 1517750 | > loss_text_ce: 0.017368124797940254 (0.030247955614155473) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13670286547080324) | > loader_time: 0.0046 (0.004675413756874622)  --> TIME: 2025-02-27 03:01:44 -- STEP: 3342/36938 -- GLOBAL_STEP: 1517800 | > loss_text_ce: 0.025699200108647346 (0.03025702288277327) | > loss_mel_ce: 1.8582712411880493 (nan) | > loss: 0.007476073689758778 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13664824844191462) | > loader_time: 0.0048 (0.004675220829651511)  --> TIME: 2025-02-27 03:01:54 -- STEP: 3392/36938 -- GLOBAL_STEP: 1517850 | > loss_text_ce: 0.03956864774227142 (0.030260879796806653) | > loss_mel_ce: 2.2771148681640625 (nan) | > loss: 0.009193189442157745 (nan) | > current_lr: 5e-06 | > step_time: 0.1445 (0.1366901116551097) | > loader_time: 0.0053 (0.004675549648280424)  --> TIME: 2025-02-27 03:02:03 -- STEP: 3442/36938 -- GLOBAL_STEP: 1517900 | > loss_text_ce: 0.03665749356150627 (0.030253252069174136) | > loss_mel_ce: 2.8148715496063232 (nan) | > loss: 0.01131559256464243 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1366743181141087) | > loader_time: 0.0045 (0.004674862734735332)  --> TIME: 2025-02-27 03:02:13 -- STEP: 3492/36938 -- GLOBAL_STEP: 1517950 | > loss_text_ce: 0.03805657476186752 (0.030268426688509605) | > loss_mel_ce: 2.218740701675415 (nan) | > loss: 0.008955545723438263 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13670103711087853) | > loader_time: 0.0046 (0.004673958917117482)  --> TIME: 2025-02-27 03:02:22 -- STEP: 3542/36938 -- GLOBAL_STEP: 1518000 | > loss_text_ce: 0.026296846568584442 (0.03028396743295566) | > loss_mel_ce: 1.9291088581085205 (nan) | > loss: 0.007759546861052513 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13671340887187142) | > loader_time: 0.0044 (0.004673111580106117)  --> TIME: 2025-02-27 03:02:31 -- STEP: 3592/36938 -- GLOBAL_STEP: 1518050 | > loss_text_ce: 0.009136205539107323 (0.030266936143750287) | > loss_mel_ce: 1.5362491607666016 (nan) | > loss: 0.006132482085376978 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13671593636871754) | > loader_time: 0.0044 (0.004672031824732153)  --> TIME: 2025-02-27 03:02:40 -- STEP: 3642/36938 -- GLOBAL_STEP: 1518100 | > loss_text_ce: 0.028905589133501053 (0.030255249112267568) | > loss_mel_ce: 2.5789637565612793 (nan) | > loss: 0.010348688811063766 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13667847617912446) | > loader_time: 0.0046 (0.004670745000153183)  --> TIME: 2025-02-27 03:02:50 -- STEP: 3692/36938 -- GLOBAL_STEP: 1518150 | > loss_text_ce: 0.029599206522107124 (0.030263587785027073) | > loss_mel_ce: 2.1988770961761475 (nan) | > loss: 0.00884316023439169 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13667019068355338) | > loader_time: 0.0045 (0.00466975973331787)  --> TIME: 2025-02-27 03:02:59 -- STEP: 3742/36938 -- GLOBAL_STEP: 1518200 | > loss_text_ce: 0.02653205581009388 (0.03027015840430619) | > loss_mel_ce: 2.586580753326416 (nan) | > loss: 0.010369496420025826 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13674075459491034) | > loader_time: 0.0046 (0.0046692009216830616)  --> TIME: 2025-02-27 03:03:09 -- STEP: 3792/36938 -- GLOBAL_STEP: 1518250 | > loss_text_ce: 0.03589587286114693 (0.030270208542250914) | > loss_mel_ce: 2.6315183639526367 (nan) | > loss: 0.010584977455437183 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13676550475102478) | > loader_time: 0.0048 (0.0046688522593381614)  --> TIME: 2025-02-27 03:03:18 -- STEP: 3842/36938 -- GLOBAL_STEP: 1518300 | > loss_text_ce: 0.029926370829343796 (0.030272220368053472) | > loss_mel_ce: 2.514232873916626 (nan) | > loss: 0.0100958701223135 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13672217039438953) | > loader_time: 0.0046 (0.004667754225405731)  --> TIME: 2025-02-27 03:03:27 -- STEP: 3892/36938 -- GLOBAL_STEP: 1518350 | > loss_text_ce: 0.0316869355738163 (0.03025804189457831) | > loss_mel_ce: 2.0871942043304443 (nan) | > loss: 0.008408259600400925 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13671487030733076) | > loader_time: 0.0047 (0.004666873754479787)  --> TIME: 2025-02-27 03:03:37 -- STEP: 3942/36938 -- GLOBAL_STEP: 1518400 | > loss_text_ce: 0.03463028743863106 (0.030256109884862124) | > loss_mel_ce: 2.5938422679901123 (nan) | > loss: 0.01043044775724411 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13673625048013743) | > loader_time: 0.0045 (0.004666597512940482)  --> TIME: 2025-02-27 03:03:46 -- STEP: 3992/36938 -- GLOBAL_STEP: 1518450 | > loss_text_ce: 0.029498806223273277 (0.03026694876985449) | > loss_mel_ce: 3.58695650100708 (nan) | > loss: 0.0143510140478611 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13664706980059324) | > loader_time: 0.0044 (0.004666026942954519)  --> TIME: 2025-02-27 03:03:55 -- STEP: 4042/36938 -- GLOBAL_STEP: 1518500 | > loss_text_ce: 0.03208029270172119 (0.030273878243685003) | > loss_mel_ce: 2.7959887981414795 (nan) | > loss: 0.011222497560083866 (nan) | > current_lr: 5e-06 | > step_time: 0.1579 (0.1366604675107051) | > loader_time: 0.0046 (0.0046656672049724904)  --> TIME: 2025-02-27 03:04:04 -- STEP: 4092/36938 -- GLOBAL_STEP: 1518550 | > loss_text_ce: 0.03468717262148857 (0.030269644264348455) | > loss_mel_ce: 2.123120069503784 (nan) | > loss: 0.008562727831304073 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13667983224548125) | > loader_time: 0.0045 (0.004665526709831595)  --> TIME: 2025-02-27 03:04:14 -- STEP: 4142/36938 -- GLOBAL_STEP: 1518600 | > loss_text_ce: 0.025997141376137733 (0.0302699835602696) | > loss_mel_ce: 2.7984163761138916 (nan) | > loss: 0.011207991279661655 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1366655354566592) | > loader_time: 0.0045 (0.0046652275142550015)  --> TIME: 2025-02-27 03:04:23 -- STEP: 4192/36938 -- GLOBAL_STEP: 1518650 | > loss_text_ce: 0.030492544174194336 (0.030264704512434) | > loss_mel_ce: 2.933429002761841 (nan) | > loss: 0.01176159456372261 (nan) | > current_lr: 5e-06 | > step_time: 0.1659 (0.13671223921630254) | > loader_time: 0.0048 (0.004664941484691538)  --> TIME: 2025-02-27 03:04:33 -- STEP: 4242/36938 -- GLOBAL_STEP: 1518700 | > loss_text_ce: 0.031404878944158554 (0.030249900786352785) | > loss_mel_ce: 2.6796176433563232 (nan) | > loss: 0.010758026503026485 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13673609587900068) | > loader_time: 0.0049 (0.00466504062138207)  --> TIME: 2025-02-27 03:04:42 -- STEP: 4292/36938 -- GLOBAL_STEP: 1518750 | > loss_text_ce: 0.032660312950611115 (0.03024730370388664) | > loss_mel_ce: 2.7144463062286377 (nan) | > loss: 0.010901217348873615 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13670921697865474) | > loader_time: 0.0046 (0.004664915694550507)  --> TIME: 2025-02-27 03:04:51 -- STEP: 4342/36938 -- GLOBAL_STEP: 1518800 | > loss_text_ce: 0.03403123468160629 (0.030241866281383915) | > loss_mel_ce: 1.7720611095428467 (nan) | > loss: 0.007167033851146698 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13664146257072848) | > loader_time: 0.0045 (0.0046650019728714685)  --> TIME: 2025-02-27 03:05:00 -- STEP: 4392/36938 -- GLOBAL_STEP: 1518850 | > loss_text_ce: 0.02976175770163536 (0.03024485416880865) | > loss_mel_ce: 2.223119020462036 (nan) | > loss: 0.00894000381231308 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.136688504768417) | > loader_time: 0.0047 (0.004665229109901337)  --> TIME: 2025-02-27 03:05:10 -- STEP: 4442/36938 -- GLOBAL_STEP: 1518900 | > loss_text_ce: 0.025059938430786133 (0.030234936015275365) | > loss_mel_ce: 1.9495471715927124 (nan) | > loss: 0.007835743017494678 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.1366561164946131) | > loader_time: 0.0053 (0.004666924369276141)  --> TIME: 2025-02-27 03:05:19 -- STEP: 4492/36938 -- GLOBAL_STEP: 1518950 | > loss_text_ce: 0.0360061414539814 (0.030231409935274046) | > loss_mel_ce: 2.6332294940948486 (nan) | > loss: 0.010592205449938774 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13667578391485322) | > loader_time: 0.0046 (0.00466751622389389)  --> TIME: 2025-02-27 03:05:29 -- STEP: 4542/36938 -- GLOBAL_STEP: 1519000 | > loss_text_ce: 0.03706514090299606 (0.030239757262572206) | > loss_mel_ce: 2.7970821857452393 (nan) | > loss: 0.011246616952121258 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.1367052970925706) | > loader_time: 0.0047 (0.004667401681225615)  --> TIME: 2025-02-27 03:05:38 -- STEP: 4592/36938 -- GLOBAL_STEP: 1519050 | > loss_text_ce: 0.031494684517383575 (0.030230941309939214) | > loss_mel_ce: 2.935075044631958 (nan) | > loss: 0.011772102676331997 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.136700690259917) | > loader_time: 0.0045 (0.004667238802860019)  --> TIME: 2025-02-27 03:05:47 -- STEP: 4642/36938 -- GLOBAL_STEP: 1519100 | > loss_text_ce: 0.025006329640746117 (0.030226021772952735) | > loss_mel_ce: 2.7909038066864014 (nan) | > loss: 0.011174246668815613 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13667929408777285) | > loader_time: 0.0046 (0.004666670187441053)  --> TIME: 2025-02-27 03:05:56 -- STEP: 4692/36938 -- GLOBAL_STEP: 1519150 | > loss_text_ce: 0.030582360923290253 (0.030224575876883167) | > loss_mel_ce: 2.044203281402588 (nan) | > loss: 0.008233277127146721 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13658773299474958) | > loader_time: 0.0047 (0.004666088792064317)  --> TIME: 2025-02-27 03:06:05 -- STEP: 4742/36938 -- GLOBAL_STEP: 1519200 | > loss_text_ce: 0.03221447765827179 (0.030214771681996662) | > loss_mel_ce: 3.0068187713623047 (nan) | > loss: 0.012059655971825123 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13654908868442964) | > loader_time: 0.0047 (0.004665841487126741)  --> TIME: 2025-02-27 03:06:15 -- STEP: 4792/36938 -- GLOBAL_STEP: 1519250 | > loss_text_ce: 0.02849867194890976 (0.030202477106720346) | > loss_mel_ce: 2.3131120204925537 (nan) | > loss: 0.00929210614413023 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1365952707192734) | > loader_time: 0.0045 (0.004666534708019895)  --> TIME: 2025-02-27 03:06:24 -- STEP: 4842/36938 -- GLOBAL_STEP: 1519300 | > loss_text_ce: 0.021120872348546982 (0.030204954303036307) | > loss_mel_ce: 2.4275050163269043 (nan) | > loss: 0.009716769680380821 (nan) | > current_lr: 5e-06 | > step_time: 0.1831 (0.13661377851453896) | > loader_time: 0.0049 (0.004666504708266077)  --> TIME: 2025-02-27 03:06:33 -- STEP: 4892/36938 -- GLOBAL_STEP: 1519350 | > loss_text_ce: 0.03585505113005638 (0.030229089231263065) | > loss_mel_ce: 2.307915687561035 (nan) | > loss: 0.00930067803710699 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.1366254687796628) | > loader_time: 0.0047 (0.0046663110312729385)  --> TIME: 2025-02-27 03:06:43 -- STEP: 4942/36938 -- GLOBAL_STEP: 1519400 | > loss_text_ce: 0.029930660501122475 (0.03023176830908708) | > loss_mel_ce: 1.2392396926879883 (nan) | > loss: 0.0050363908521831036 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13661993263897637) | > loader_time: 0.0045 (0.004666084608342771)  --> TIME: 2025-02-27 03:06:52 -- STEP: 4992/36938 -- GLOBAL_STEP: 1519450 | > loss_text_ce: 0.02236649952828884 (0.03023002344222429) | > loss_mel_ce: 2.6390509605407715 (nan) | > loss: 0.010561181232333183 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.13660523462562976) | > loader_time: 0.0046 (0.004665637341065293)  --> TIME: 2025-02-27 03:07:01 -- STEP: 5042/36938 -- GLOBAL_STEP: 1519500 | > loss_text_ce: 0.030875541269779205 (0.030231247059055166) | > loss_mel_ce: 2.632707118988037 (nan) | > loss: 0.010569772683084011 (nan) | > current_lr: 5e-06 | > step_time: 0.1835 (0.13665062971693884) | > loader_time: 0.0049 (0.004666177586211625)  --> TIME: 2025-02-27 03:07:11 -- STEP: 5092/36938 -- GLOBAL_STEP: 1519550 | > loss_text_ce: 0.04375721514225006 (0.030228004590019245) | > loss_mel_ce: 2.7960710525512695 (nan) | > loss: 0.0112691605463624 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13662852757761806) | > loader_time: 0.0046 (0.004666313774831084)  --> TIME: 2025-02-27 03:07:20 -- STEP: 5142/36938 -- GLOBAL_STEP: 1519600 | > loss_text_ce: 0.03464122116565704 (0.03023083594662816) | > loss_mel_ce: 3.0742974281311035 (nan) | > loss: 0.012337058782577515 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.13662289089379442) | > loader_time: 0.005 (0.00466624070400861)  --> TIME: 2025-02-27 03:07:29 -- STEP: 5192/36938 -- GLOBAL_STEP: 1519650 | > loss_text_ce: 0.02996288239955902 (0.030233625407246025) | > loss_mel_ce: 3.202904462814331 (nan) | > loss: 0.012828839011490345 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.13662820581478782) | > loader_time: 0.0048 (0.004665998124929362)  --> TIME: 2025-02-27 03:07:39 -- STEP: 5242/36938 -- GLOBAL_STEP: 1519700 | > loss_text_ce: 0.04586585983633995 (0.03023110143875846) | > loss_mel_ce: 2.418862819671631 (nan) | > loss: 0.009780669584870338 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13662271602423962) | > loader_time: 0.0048 (0.004666216088723984)  --> TIME: 2025-02-27 03:07:48 -- STEP: 5292/36938 -- GLOBAL_STEP: 1519750 | > loss_text_ce: 0.028288407251238823 (0.030246958155213142) | > loss_mel_ce: 1.290497064590454 (nan) | > loss: 0.005233275704085827 (nan) | > current_lr: 5e-06 | > step_time: 0.1061 (0.13657253205731174) | > loader_time: 0.0045 (0.004666063971169876)  --> TIME: 2025-02-27 03:07:57 -- STEP: 5342/36938 -- GLOBAL_STEP: 1519800 | > loss_text_ce: 0.027972273528575897 (0.03024583861743665) | > loss_mel_ce: 2.8411872386932373 (nan) | > loss: 0.011385554447770119 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13658691675463958) | > loader_time: 0.0046 (0.004665881986700178)  --> TIME: 2025-02-27 03:08:07 -- STEP: 5392/36938 -- GLOBAL_STEP: 1519850 | > loss_text_ce: 0.02855122648179531 (0.0302362497198155) | > loss_mel_ce: 2.519015073776245 (nan) | > loss: 0.010109391063451767 (nan) | > current_lr: 5e-06 | > step_time: 0.1882 (0.13662153274616812) | > loader_time: 0.0048 (0.004665609327551182)  --> TIME: 2025-02-27 03:08:16 -- STEP: 5442/36938 -- GLOBAL_STEP: 1519900 | > loss_text_ce: 0.026605883613228798 (0.030239045434453084) | > loss_mel_ce: 2.530291795730591 (nan) | > loss: 0.010146419517695904 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13661865371415333) | > loader_time: 0.0046 (0.004665422684915904)  --> TIME: 2025-02-27 03:08:25 -- STEP: 5492/36938 -- GLOBAL_STEP: 1519950 | > loss_text_ce: 0.02546980045735836 (0.030230758491922537) | > loss_mel_ce: 2.378502368927002 (nan) | > loss: 0.009539572522044182 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.1366320049684587) | > loader_time: 0.0045 (0.0046658420423825965)  --> TIME: 2025-02-27 03:08:35 -- STEP: 5542/36938 -- GLOBAL_STEP: 1520000 | > loss_text_ce: 0.026138748973608017 (0.03023606147691043) | > loss_mel_ce: 0.0162353515625 (nan) | > loss: 0.00016815120761748403 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.1366781317684379) | > loader_time: 0.0046 (0.004665687771662632) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1520000.pth  --> TIME: 2025-02-27 03:08:50 -- STEP: 5592/36938 -- GLOBAL_STEP: 1520050 | > loss_text_ce: 0.03642557933926582 (0.030235739659487358) | > loss_mel_ce: 3.0817484855651855 (nan) | > loss: 0.012373707257211208 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.1366104894612142) | > loader_time: 0.0049 (0.004666070740280934)  --> TIME: 2025-02-27 03:08:59 -- STEP: 5642/36938 -- GLOBAL_STEP: 1520100 | > loss_text_ce: 0.03900590166449547 (0.030233101268093907) | > loss_mel_ce: 4.65625 (nan) | > loss: 0.018631968647241592 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1365941495854821) | > loader_time: 0.0045 (0.0046655318281491)  --> TIME: 2025-02-27 03:09:08 -- STEP: 5692/36938 -- GLOBAL_STEP: 1520150 | > loss_text_ce: 0.018996961414813995 (0.030228905534392676) | > loss_mel_ce: 2.2699828147888184 (nan) | > loss: 0.00908325333148241 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.1366019875586078) | > loader_time: 0.0046 (0.004665491703272053)  --> TIME: 2025-02-27 03:09:18 -- STEP: 5742/36938 -- GLOBAL_STEP: 1520200 | > loss_text_ce: 0.03170361369848251 (0.030229313941189103) | > loss_mel_ce: 2.7908732891082764 (nan) | > loss: 0.011200702749192715 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13660011927618984) | > loader_time: 0.0047 (0.004665153527334586)  --> TIME: 2025-02-27 03:09:27 -- STEP: 5792/36938 -- GLOBAL_STEP: 1520250 | > loss_text_ce: 0.031188268214464188 (0.030236429920920443) | > loss_mel_ce: 2.0040409564971924 (nan) | > loss: 0.00807630643248558 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1365937218893301) | > loader_time: 0.0045 (0.00466479274614083)  --> TIME: 2025-02-27 03:09:36 -- STEP: 5842/36938 -- GLOBAL_STEP: 1520300 | > loss_text_ce: 0.03806733712553978 (0.030234914404862662) | > loss_mel_ce: 2.3763394355773926 (nan) | > loss: 0.00958098005503416 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13656519333494332) | > loader_time: 0.0046 (0.004664692230642536)  --> TIME: 2025-02-27 03:09:45 -- STEP: 5892/36938 -- GLOBAL_STEP: 1520350 | > loss_text_ce: 0.03356122225522995 (0.030231653873442604) | > loss_mel_ce: 3.0588879585266113 (nan) | > loss: 0.012271624058485031 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13657030521892652) | > loader_time: 0.0046 (0.004664827226702264)  --> TIME: 2025-02-27 03:09:55 -- STEP: 5942/36938 -- GLOBAL_STEP: 1520400 | > loss_text_ce: 0.027803974226117134 (0.030233054954586197) | > loss_mel_ce: 0.005401611328125 (nan) | > loss: 0.00013176820357330143 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.13656247231904933) | > loader_time: 0.0048 (0.004664390105584884)  --> TIME: 2025-02-27 03:10:04 -- STEP: 5992/36938 -- GLOBAL_STEP: 1520450 | > loss_text_ce: 0.023139437660574913 (0.030235660635276843) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1365626180601693) | > loader_time: 0.0052 (0.004664421638595744)  --> TIME: 2025-02-27 03:10:13 -- STEP: 6042/36938 -- GLOBAL_STEP: 1520500 | > loss_text_ce: 0.03349706903100014 (0.030227916610687085) | > loss_mel_ce: 2.2396514415740967 (nan) | > loss: 0.009020430967211723 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13657279801423952) | > loader_time: 0.0055 (0.004665862049972824)  --> TIME: 2025-02-27 03:10:23 -- STEP: 6092/36938 -- GLOBAL_STEP: 1520550 | > loss_text_ce: 0.034525442868471146 (0.030240254819873802) | > loss_mel_ce: 2.9057183265686035 (nan) | > loss: 0.01166763436049223 (nan) | > current_lr: 5e-06 | > step_time: 0.1518 (0.13659259647581046) | > loader_time: 0.0047 (0.004666079875284822)  --> TIME: 2025-02-27 03:10:32 -- STEP: 6142/36938 -- GLOBAL_STEP: 1520600 | > loss_text_ce: 0.02635560929775238 (0.03024591124582901) | > loss_mel_ce: 2.9578428268432617 (nan) | > loss: 0.01184205710887909 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.13658331648529415) | > loader_time: 0.0048 (0.004665947162301505)  --> TIME: 2025-02-27 03:10:42 -- STEP: 6192/36938 -- GLOBAL_STEP: 1520650 | > loss_text_ce: 0.027938110753893852 (0.03025306310869864) | > loss_mel_ce: 2.807609796524048 (nan) | > loss: 0.011252175085246563 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1366326153586386) | > loader_time: 0.0045 (0.004666632960625107)  --> TIME: 2025-02-27 03:10:51 -- STEP: 6242/36938 -- GLOBAL_STEP: 1520700 | > loss_text_ce: 0.026449421420693398 (0.030255166137694252) | > loss_mel_ce: 1.4747024774551392 (nan) | > loss: 0.0059569524601101875 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13664314736161853) | > loader_time: 0.0046 (0.004667075617654876)  --> TIME: 2025-02-27 03:11:01 -- STEP: 6292/36938 -- GLOBAL_STEP: 1520750 | > loss_text_ce: 0.023033419623970985 (0.03025141191072004) | > loss_mel_ce: 3.1359050273895264 (nan) | > loss: 0.01253547053784132 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13663181565934054) | > loader_time: 0.0045 (0.004666667945507766)  --> TIME: 2025-02-27 03:11:10 -- STEP: 6342/36938 -- GLOBAL_STEP: 1520800 | > loss_text_ce: 0.023088177666068077 (0.030253888478926145) | > loss_mel_ce: 2.4337046146392822 (nan) | > loss: 0.009749178774654865 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13664828381377495) | > loader_time: 0.0045 (0.0046671896242486575)  --> TIME: 2025-02-27 03:11:19 -- STEP: 6392/36938 -- GLOBAL_STEP: 1520850 | > loss_text_ce: 0.027216462418437004 (0.030259354451049974) | > loss_mel_ce: 2.244992256164551 (nan) | > loss: 0.009016701951622963 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13666352961627368) | > loader_time: 0.0046 (0.004668411869877437)  --> TIME: 2025-02-27 03:11:29 -- STEP: 6442/36938 -- GLOBAL_STEP: 1520900 | > loss_text_ce: 0.03652564808726311 (0.030261612138590924) | > loss_mel_ce: 2.4833953380584717 (nan) | > loss: 0.009999687783420086 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13663579695197386) | > loader_time: 0.0045 (0.004667941771172414)  --> TIME: 2025-02-27 03:11:38 -- STEP: 6492/36938 -- GLOBAL_STEP: 1520950 | > loss_text_ce: 0.033057261258363724 (0.030257350779944733) | > loss_mel_ce: 2.778944730758667 (nan) | > loss: 0.011158738285303116 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13667297172311355) | > loader_time: 0.0048 (0.004668392778104)  --> TIME: 2025-02-27 03:11:47 -- STEP: 6542/36938 -- GLOBAL_STEP: 1521000 | > loss_text_ce: 0.021901177242398262 (0.03025164094789723) | > loss_mel_ce: 2.1901066303253174 (nan) | > loss: 0.008777809329330921 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13661940809930379) | > loader_time: 0.0045 (0.004668024912116671)  --> TIME: 2025-02-27 03:11:57 -- STEP: 6592/36938 -- GLOBAL_STEP: 1521050 | > loss_text_ce: 0.02762545831501484 (0.03025395778198252) | > loss_mel_ce: 2.41780424118042 (nan) | > loss: 0.009704086929559708 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13663272921321476) | > loader_time: 0.0046 (0.004668095447484748)  --> TIME: 2025-02-27 03:12:06 -- STEP: 6642/36938 -- GLOBAL_STEP: 1521100 | > loss_text_ce: 0.036375924944877625 (0.03026030601785529) | > loss_mel_ce: 3.165666103363037 (nan) | > loss: 0.012706517241895199 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13669619047652545) | > loader_time: 0.0047 (0.004668126835660828)  --> TIME: 2025-02-27 03:12:16 -- STEP: 6692/36938 -- GLOBAL_STEP: 1521150 | > loss_text_ce: 0.026562923565506935 (0.03025480558290227) | > loss_mel_ce: 1.739546298980713 (nan) | > loss: 0.007008370477706194 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13671665379050543) | > loader_time: 0.0045 (0.004667671903905529)  --> TIME: 2025-02-27 03:12:25 -- STEP: 6742/36938 -- GLOBAL_STEP: 1521200 | > loss_text_ce: 0.024083822965621948 (0.030260307073799793) | > loss_mel_ce: 2.579522132873535 (nan) | > loss: 0.010331770405173302 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13675660126572098) | > loader_time: 0.0045 (0.004667366763952064)  --> TIME: 2025-02-27 03:12:34 -- STEP: 6792/36938 -- GLOBAL_STEP: 1521250 | > loss_text_ce: 0.027342988178133965 (0.03025373280252954) | > loss_mel_ce: 0.502636730670929 (nan) | > loss: 0.002103094244375825 (nan) | > current_lr: 5e-06 | > step_time: 0.1695 (0.1367216140066919) | > loader_time: 0.0048 (0.004666668050280748)  --> TIME: 2025-02-27 03:12:44 -- STEP: 6842/36938 -- GLOBAL_STEP: 1521300 | > loss_text_ce: 0.0271591916680336 (0.030250579156977927) | > loss_mel_ce: 2.717740535736084 (nan) | > loss: 0.010892460122704506 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1367133855819703) | > loader_time: 0.0045 (0.004666114891875198)  --> TIME: 2025-02-27 03:12:53 -- STEP: 6892/36938 -- GLOBAL_STEP: 1521350 | > loss_text_ce: 0.033426184207201004 (0.030241335889333195) | > loss_mel_ce: 3.15397047996521 (nan) | > loss: 0.012648400850594044 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13669240512753816) | > loader_time: 0.0047 (0.004665808939200412)  --> TIME: 2025-02-27 03:13:02 -- STEP: 6942/36938 -- GLOBAL_STEP: 1521400 | > loss_text_ce: 0.02760680578649044 (0.030246844702891015) | > loss_mel_ce: 2.0507636070251465 (nan) | > loss: 0.008247502148151398 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13667691409914132) | > loader_time: 0.0048 (0.0046653856086786094)  --> TIME: 2025-02-27 03:13:12 -- STEP: 6992/36938 -- GLOBAL_STEP: 1521450 | > loss_text_ce: 0.02757677063345909 (0.030242581349223197) | > loss_mel_ce: 2.1693332195281982 (nan) | > loss: 0.008717897348105907 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13668468407007878) | > loader_time: 0.0046 (0.004665366640352839)  --> TIME: 2025-02-27 03:13:21 -- STEP: 7042/36938 -- GLOBAL_STEP: 1521500 | > loss_text_ce: 0.03887307643890381 (0.03024590263503967) | > loss_mel_ce: 2.699759006500244 (nan) | > loss: 0.0108675891533494 (nan) | > current_lr: 5e-06 | > step_time: 0.1887 (0.1366635355479472) | > loader_time: 0.0049 (0.004665330911488485)  --> TIME: 2025-02-27 03:13:30 -- STEP: 7092/36938 -- GLOBAL_STEP: 1521550 | > loss_text_ce: 0.0298298429697752 (0.030250787362955337) | > loss_mel_ce: 2.7347826957702637 (nan) | > loss: 0.010970684699714184 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13666580961171482) | > loader_time: 0.0044 (0.0046655802624547184)  --> TIME: 2025-02-27 03:13:40 -- STEP: 7142/36938 -- GLOBAL_STEP: 1521600 | > loss_text_ce: 0.032209452241659164 (0.030242579729507255) | > loss_mel_ce: 2.7731451988220215 (nan) | > loss: 0.011132360436022282 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.1366748660594199) | > loader_time: 0.0046 (0.0046653402716263755)  --> TIME: 2025-02-27 03:13:49 -- STEP: 7192/36938 -- GLOBAL_STEP: 1521650 | > loss_text_ce: 0.02568477764725685 (0.030234263525489465) | > loss_mel_ce: 2.536583423614502 (nan) | > loss: 0.010167731903493404 (nan) | > current_lr: 5e-06 | > step_time: 0.1769 (0.1366563013716453) | > loader_time: 0.0056 (0.004665548464612798)  --> TIME: 2025-02-27 03:13:58 -- STEP: 7242/36938 -- GLOBAL_STEP: 1521700 | > loss_text_ce: 0.03277133032679558 (0.03023007483401888) | > loss_mel_ce: 2.813181161880493 (nan) | > loss: 0.011293463408946991 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13667444634984052) | > loader_time: 0.0045 (0.004665236353248713)  --> TIME: 2025-02-27 03:14:08 -- STEP: 7292/36938 -- GLOBAL_STEP: 1521750 | > loss_text_ce: 0.043520063161849976 (0.030227275263212592) | > loss_mel_ce: 2.334186553955078 (nan) | > loss: 0.00943534355610609 (nan) | > current_lr: 5e-06 | > step_time: 0.1079 (0.13670639990452407) | > loader_time: 0.0045 (0.004665800816268646)  --> TIME: 2025-02-27 03:14:18 -- STEP: 7342/36938 -- GLOBAL_STEP: 1521800 | > loss_text_ce: 0.0279964879155159 (0.03023256897508745) | > loss_mel_ce: 2.9848179817199707 (nan) | > loss: 0.01195561420172453 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.1367680126625066) | > loader_time: 0.0046 (0.004666075236269335)  --> TIME: 2025-02-27 03:14:27 -- STEP: 7392/36938 -- GLOBAL_STEP: 1521850 | > loss_text_ce: 0.028188364580273628 (0.030228945766943526) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13677806658791247) | > loader_time: 0.0054 (0.004666304788290182)  --> TIME: 2025-02-27 03:14:36 -- STEP: 7442/36938 -- GLOBAL_STEP: 1521900 | > loss_text_ce: 0.03819894790649414 (0.030229401591743747) | > loss_mel_ce: 2.4811179637908936 (nan) | > loss: 0.009997289627790451 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.1367659886912997) | > loader_time: 0.0053 (0.00466701590992438)  --> TIME: 2025-02-27 03:14:46 -- STEP: 7492/36938 -- GLOBAL_STEP: 1521950 | > loss_text_ce: 0.02492498606443405 (0.030230715955884554) | > loss_mel_ce: 0.40021514892578125 (nan) | > loss: 0.001687064184807241 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13677557200209303) | > loader_time: 0.0046 (0.004666975775442227)  --> TIME: 2025-02-27 03:14:55 -- STEP: 7542/36938 -- GLOBAL_STEP: 1522000 | > loss_text_ce: 0.02971022017300129 (0.030232732208434306) | > loss_mel_ce: 2.38067889213562 (nan) | > loss: 0.009565036743879318 (nan) | > current_lr: 5e-06 | > step_time: 0.1528 (0.13681437747290698) | > loader_time: 0.0046 (0.004666807448570008)  --> TIME: 2025-02-27 03:15:05 -- STEP: 7592/36938 -- GLOBAL_STEP: 1522050 | > loss_text_ce: 0.039067115634679794 (0.030237644928231742) | > loss_mel_ce: 2.230757713317871 (nan) | > loss: 0.009007241576910019 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13680225026620063) | > loader_time: 0.0045 (0.004666475274917089)  --> TIME: 2025-02-27 03:15:14 -- STEP: 7642/36938 -- GLOBAL_STEP: 1522100 | > loss_text_ce: 0.023114871233701706 (0.030236238738031045) | > loss_mel_ce: 2.622546672821045 (nan) | > loss: 0.010498657822608948 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.13676225409024786) | > loader_time: 0.0046 (0.00466614551364488)  --> TIME: 2025-02-27 03:15:23 -- STEP: 7692/36938 -- GLOBAL_STEP: 1522150 | > loss_text_ce: 0.03331920877099037 (0.030237618982704724) | > loss_mel_ce: 3.1638762950897217 (nan) | > loss: 0.012687284499406815 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.136767040362782) | > loader_time: 0.0047 (0.004666187151979809)  --> TIME: 2025-02-27 03:15:32 -- STEP: 7742/36938 -- GLOBAL_STEP: 1522200 | > loss_text_ce: 0.022907324135303497 (0.030241428587168988) | > loss_mel_ce: 0.1004706472158432 (nan) | > loss: 0.0004895951715297997 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13678013804161515) | > loader_time: 0.0046 (0.004666217966801574)  --> TIME: 2025-02-27 03:15:42 -- STEP: 7792/36938 -- GLOBAL_STEP: 1522250 | > loss_text_ce: 0.024473628029227257 (0.03024543005255007) | > loss_mel_ce: 2.401870012283325 (nan) | > loss: 0.009628348052501678 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13677461867582125) | > loader_time: 0.0046 (0.004666495433333489)  --> TIME: 2025-02-27 03:15:51 -- STEP: 7842/36938 -- GLOBAL_STEP: 1522300 | > loss_text_ce: 0.03146647289395332 (0.030253758490108404) | > loss_mel_ce: 2.0212314128875732 (nan) | > loss: 0.00814562663435936 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13674625454128947) | > loader_time: 0.0047 (0.004666121660887801)  --> TIME: 2025-02-27 03:16:00 -- STEP: 7892/36938 -- GLOBAL_STEP: 1522350 | > loss_text_ce: 0.030264990404248238 (0.030262647100827655) | > loss_mel_ce: 2.206242561340332 (nan) | > loss: 0.008875031024217606 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13670788787671642) | > loader_time: 0.0045 (0.004665717610960028)  --> TIME: 2025-02-27 03:16:09 -- STEP: 7942/36938 -- GLOBAL_STEP: 1522400 | > loss_text_ce: 0.029508696869015694 (0.03025461395006806) | > loss_mel_ce: 2.7680113315582275 (nan) | > loss: 0.011101270094513893 (nan) | > current_lr: 5e-06 | > step_time: 0.184 (0.1367364580581422) | > loader_time: 0.0048 (0.004665627073683283)  --> TIME: 2025-02-27 03:16:19 -- STEP: 7992/36938 -- GLOBAL_STEP: 1522450 | > loss_text_ce: 0.033435411751270294 (0.030252609657612187) | > loss_mel_ce: 0.025146484375 (nan) | > loss: 0.00023246784985531121 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13673247252259993) | > loader_time: 0.0046 (0.004665379200850418)  --> TIME: 2025-02-27 03:16:28 -- STEP: 8042/36938 -- GLOBAL_STEP: 1522500 | > loss_text_ce: 0.02286827191710472 (0.030254850121438727) | > loss_mel_ce: 2.2799038887023926 (nan) | > loss: 0.009137985296547413 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13674248566351185) | > loader_time: 0.0045 (0.0046651652427906875)  --> TIME: 2025-02-27 03:16:37 -- STEP: 8092/36938 -- GLOBAL_STEP: 1522550 | > loss_text_ce: 0.033832330256700516 (0.030261693209284495) | > loss_mel_ce: 2.914761543273926 (nan) | > loss: 0.011700769886374474 (nan) | > current_lr: 5e-06 | > step_time: 0.1578 (0.13673373899186406) | > loader_time: 0.0046 (0.0046648539591707845)  --> TIME: 2025-02-27 03:16:47 -- STEP: 8142/36938 -- GLOBAL_STEP: 1522600 | > loss_text_ce: 0.0280467439442873 (0.030258726890699205) | > loss_mel_ce: 1.7602754831314087 (nan) | > loss: 0.007096516899764538 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13672917157418885) | > loader_time: 0.0045 (0.004664503248397929)  --> TIME: 2025-02-27 03:16:56 -- STEP: 8192/36938 -- GLOBAL_STEP: 1522650 | > loss_text_ce: 0.03111746534705162 (0.030258092858957752) | > loss_mel_ce: 2.7224032878875732 (nan) | > loss: 0.010926670394837856 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.13672737148590375) | > loader_time: 0.0048 (0.0046642050729133345)  --> TIME: 2025-02-27 03:17:05 -- STEP: 8242/36938 -- GLOBAL_STEP: 1522700 | > loss_text_ce: 0.03353945165872574 (0.030257946535903357) | > loss_mel_ce: 2.91306209564209 (nan) | > loss: 0.011692863889038563 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13670776426083417) | > loader_time: 0.0046 (0.004663796831004641)  --> TIME: 2025-02-27 03:17:15 -- STEP: 8292/36938 -- GLOBAL_STEP: 1522750 | > loss_text_ce: 0.02441592700779438 (0.03026340516586014) | > loss_mel_ce: 1.6027429103851318 (nan) | > loss: 0.006456980016082525 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13671355594256843) | > loader_time: 0.0047 (0.004663515366969696)  --> TIME: 2025-02-27 03:17:24 -- STEP: 8342/36938 -- GLOBAL_STEP: 1522800 | > loss_text_ce: 0.028558854013681412 (0.030260498144913286) | > loss_mel_ce: 2.6934103965759277 (nan) | > loss: 0.010801466181874275 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.13674615836206364) | > loader_time: 0.0046 (0.004663958305993617)  --> TIME: 2025-02-27 03:17:33 -- STEP: 8392/36938 -- GLOBAL_STEP: 1522850 | > loss_text_ce: 0.027420587837696075 (0.030262759860713324) | > loss_mel_ce: 1.7036471366882324 (nan) | > loss: 0.006869317032396793 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1367118420830445) | > loader_time: 0.0045 (0.004663429962554597)  --> TIME: 2025-02-27 03:17:42 -- STEP: 8442/36938 -- GLOBAL_STEP: 1522900 | > loss_text_ce: 0.025424182415008545 (0.030260558885267786) | > loss_mel_ce: 1.7528572082519531 (nan) | > loss: 0.007056673057377338 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13669558032884666) | > loader_time: 0.0045 (0.004663747539172338)  --> TIME: 2025-02-27 03:17:52 -- STEP: 8492/36938 -- GLOBAL_STEP: 1522950 | > loss_text_ce: 0.022659411653876305 (0.03026271004886052) | > loss_mel_ce: 3.1097936630249023 (nan) | > loss: 0.012430369853973389 (nan) | > current_lr: 5e-06 | > step_time: 0.1642 (0.13670562591633595) | > loader_time: 0.0046 (0.0046638084985263315)  --> TIME: 2025-02-27 03:18:01 -- STEP: 8542/36938 -- GLOBAL_STEP: 1523000 | > loss_text_ce: 0.025251414626836777 (0.030263462434914214) | > loss_mel_ce: 0.9284324645996094 (nan) | > loss: 0.0037844600155949593 (nan) | > current_lr: 5e-06 | > step_time: 0.1807 (0.13670855326553172) | > loader_time: 0.0049 (0.0046637042069485515)  --> TIME: 2025-02-27 03:18:10 -- STEP: 8592/36938 -- GLOBAL_STEP: 1523050 | > loss_text_ce: 0.02404753305017948 (0.030266430914866794) | > loss_mel_ce: 2.5447616577148438 (nan) | > loss: 0.010193687863647938 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13668834308450234) | > loader_time: 0.0046 (0.004663548267531457)  --> TIME: 2025-02-27 03:18:20 -- STEP: 8642/36938 -- GLOBAL_STEP: 1523100 | > loss_text_ce: 0.023708941414952278 (0.030263862115678213) | > loss_mel_ce: 1.0600436925888062 (nan) | > loss: 0.004300605971366167 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13671208807396365) | > loader_time: 0.0045 (0.004663540626725416)  --> TIME: 2025-02-27 03:18:29 -- STEP: 8692/36938 -- GLOBAL_STEP: 1523150 | > loss_text_ce: 0.025958366692066193 (0.03026901404714075) | > loss_mel_ce: 2.5428073406219482 (nan) | > loss: 0.01019351463764906 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13672578779168806) | > loader_time: 0.0046 (0.004663289745337851)  --> TIME: 2025-02-27 03:18:39 -- STEP: 8742/36938 -- GLOBAL_STEP: 1523200 | > loss_text_ce: 0.018340982496738434 (0.030264065120888223) | > loss_mel_ce: 0.06984592974185944 (nan) | > loss: 0.00034994809539057314 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13677161816082295) | > loader_time: 0.0047 (0.004663329897906534)  --> TIME: 2025-02-27 03:18:49 -- STEP: 8792/36938 -- GLOBAL_STEP: 1523250 | > loss_text_ce: 0.03134502097964287 (0.030262801179906398) | > loss_mel_ce: 2.552459478378296 (nan) | > loss: 0.01025319378823042 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13679700189552269) | > loader_time: 0.0045 (0.004663311236549026)  --> TIME: 2025-02-27 03:18:58 -- STEP: 8842/36938 -- GLOBAL_STEP: 1523300 | > loss_text_ce: 0.035445716232061386 (0.030269096151509987) | > loss_mel_ce: 2.3292007446289062 (nan) | > loss: 0.009383518248796463 (nan) | > current_lr: 5e-06 | > step_time: 0.1878 (0.13679657457533312) | > loader_time: 0.0049 (0.0046633683942179285)  --> TIME: 2025-02-27 03:19:07 -- STEP: 8892/36938 -- GLOBAL_STEP: 1523350 | > loss_text_ce: 0.029852069914340973 (0.030266530959047053) | > loss_mel_ce: 2.9117391109466553 (nan) | > loss: 0.011672982014715672 (nan) | > current_lr: 5e-06 | > step_time: 0.1797 (0.13681336233222266) | > loader_time: 0.0048 (0.004663321599732967)  --> TIME: 2025-02-27 03:19:17 -- STEP: 8942/36938 -- GLOBAL_STEP: 1523400 | > loss_text_ce: 0.029356246814131737 (0.030268749422338533) | > loss_mel_ce: 3.0366344451904297 (nan) | > loss: 0.01216663047671318 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.1368449372857029) | > loader_time: 0.0048 (0.004663767710118146)  --> TIME: 2025-02-27 03:19:27 -- STEP: 8992/36938 -- GLOBAL_STEP: 1523450 | > loss_text_ce: 0.026209382340312004 (0.03026928436495794) | > loss_mel_ce: 2.7351527214050293 (nan) | > loss: 0.010957786813378334 (nan) | > current_lr: 5e-06 | > step_time: 0.1914 (0.13684691014132885) | > loader_time: 0.0049 (0.004664152144749403)  --> TIME: 2025-02-27 03:19:36 -- STEP: 9042/36938 -- GLOBAL_STEP: 1523500 | > loss_text_ce: 0.030940767377614975 (0.030272584563639075) | > loss_mel_ce: 3.3608860969543457 (nan) | > loss: 0.013459631241858006 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13683607796603808) | > loader_time: 0.0046 (0.004664195873283289)  --> TIME: 2025-02-27 03:19:45 -- STEP: 9092/36938 -- GLOBAL_STEP: 1523550 | > loss_text_ce: 0.03766704350709915 (0.030270671452695946) | > loss_mel_ce: 3.2435731887817383 (nan) | > loss: 0.013020794838666916 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13680598752067516) | > loader_time: 0.0045 (0.004663913406257956)  --> TIME: 2025-02-27 03:19:54 -- STEP: 9142/36938 -- GLOBAL_STEP: 1523600 | > loss_text_ce: 0.031694646924734116 (0.030273994850616198) | > loss_mel_ce: 2.5355896949768066 (nan) | > loss: 0.010187637060880661 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13681806521247755) | > loader_time: 0.0045 (0.004663933447481281)  --> TIME: 2025-02-27 03:20:03 -- STEP: 9192/36938 -- GLOBAL_STEP: 1523650 | > loss_text_ce: 0.02134617604315281 (0.030276934121223932) | > loss_mel_ce: 2.291513442993164 (nan) | > loss: 0.00917801447212696 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13680062758808456) | > loader_time: 0.0048 (0.004663897375110141)  --> TIME: 2025-02-27 03:20:12 -- STEP: 9242/36938 -- GLOBAL_STEP: 1523700 | > loss_text_ce: 0.03100234642624855 (0.03028017666361769) | > loss_mel_ce: 2.203895330429077 (nan) | > loss: 0.008868642151355743 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13677793352976947) | > loader_time: 0.0047 (0.00466351647367728)  --> TIME: 2025-02-27 03:20:22 -- STEP: 9292/36938 -- GLOBAL_STEP: 1523750 | > loss_text_ce: 0.034324005246162415 (0.030267833852041962) | > loss_mel_ce: 1.842474102973938 (nan) | > loss: 0.007447612006217241 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13677834671630565) | > loader_time: 0.0047 (0.004663815413165912)  --> TIME: 2025-02-27 03:20:31 -- STEP: 9342/36938 -- GLOBAL_STEP: 1523800 | > loss_text_ce: 0.026757216081023216 (0.030267416930722962) | > loss_mel_ce: 2.4913253784179688 (nan) | > loss: 0.009992391802370548 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.13679120283365712) | > loader_time: 0.0048 (0.00466428127668574)  --> TIME: 2025-02-27 03:20:40 -- STEP: 9392/36938 -- GLOBAL_STEP: 1523850 | > loss_text_ce: 0.0363464318215847 (0.030266741262625872) | > loss_mel_ce: 1.937006950378418 (nan) | > loss: 0.007830767892301083 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13678118690619886) | > loader_time: 0.0046 (0.0046643999102168555)  --> TIME: 2025-02-27 03:20:50 -- STEP: 9442/36938 -- GLOBAL_STEP: 1523900 | > loss_text_ce: 0.02830001339316368 (0.030263670559777613) | > loss_mel_ce: 2.593644857406616 (nan) | > loss: 0.010404543951153755 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.13679396492574858) | > loader_time: 0.0048 (0.004664598494054815)  --> TIME: 2025-02-27 03:20:59 -- STEP: 9492/36938 -- GLOBAL_STEP: 1523950 | > loss_text_ce: 0.02212057262659073 (0.030262724752116) | > loss_mel_ce: 2.845264434814453 (nan) | > loss: 0.011378511786460876 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13680098019618794) | > loader_time: 0.0045 (0.004664609214197579)  --> TIME: 2025-02-27 03:21:08 -- STEP: 9542/36938 -- GLOBAL_STEP: 1524000 | > loss_text_ce: 0.0265261922031641 (0.03026199783252216) | > loss_mel_ce: 2.775995969772339 (nan) | > loss: 0.011121120303869247 (nan) | > current_lr: 5e-06 | > step_time: 0.1786 (0.1367951550590643) | > loader_time: 0.0049 (0.004664663422959736)  --> TIME: 2025-02-27 03:21:18 -- STEP: 9592/36938 -- GLOBAL_STEP: 1524050 | > loss_text_ce: 0.017033923417329788 (0.03026070693830918) | > loss_mel_ce: 0.707073986530304 (nan) | > loss: 0.0028734442312270403 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13681415707693168) | > loader_time: 0.0045 (0.004665165493347136)  --> TIME: 2025-02-27 03:21:27 -- STEP: 9642/36938 -- GLOBAL_STEP: 1524100 | > loss_text_ce: 0.03630758076906204 (0.030262332998047693) | > loss_mel_ce: 1.745920181274414 (nan) | > loss: 0.00707233278080821 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13682358968655994) | > loader_time: 0.005 (0.0046652154392336645)  --> TIME: 2025-02-27 03:21:37 -- STEP: 9692/36938 -- GLOBAL_STEP: 1524150 | > loss_text_ce: 0.031423140317201614 (0.03025481902490309) | > loss_mel_ce: 2.261777400970459 (nan) | > loss: 0.009100002236664295 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13681523434839296) | > loader_time: 0.0046 (0.004665140986590165)  --> TIME: 2025-02-27 03:21:46 -- STEP: 9742/36938 -- GLOBAL_STEP: 1524200 | > loss_text_ce: 0.029525214806199074 (0.030258458292980166) | > loss_mel_ce: 3.1771609783172607 (nan) | > loss: 0.01272494625300169 (nan) | > current_lr: 5e-06 | > step_time: 0.1511 (0.13681498694238736) | > loader_time: 0.0046 (0.004664940110615599)  --> TIME: 2025-02-27 03:21:55 -- STEP: 9792/36938 -- GLOBAL_STEP: 1524250 | > loss_text_ce: 0.03601744771003723 (0.03025324060239226) | > loss_mel_ce: 2.530083656311035 (nan) | > loss: 0.01018294133245945 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13682267410692808) | > loader_time: 0.0046 (0.0046651193908616604)  --> TIME: 2025-02-27 03:22:05 -- STEP: 9842/36938 -- GLOBAL_STEP: 1524300 | > loss_text_ce: 0.028359493240714073 (0.03025662616024721) | > loss_mel_ce: 2.7491238117218018 (nan) | > loss: 0.011021759361028671 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13681085219690395) | > loader_time: 0.0046 (0.004665268385611374)  --> TIME: 2025-02-27 03:22:14 -- STEP: 9892/36938 -- GLOBAL_STEP: 1524350 | > loss_text_ce: 0.029502494260668755 (0.030250829812839736) | > loss_mel_ce: 2.4963696002960205 (nan) | > loss: 0.010023302398622036 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.13682997436403874) | > loader_time: 0.0048 (0.004665493242084092)  --> TIME: 2025-02-27 03:22:23 -- STEP: 9942/36938 -- GLOBAL_STEP: 1524400 | > loss_text_ce: 0.032463569194078445 (0.030247425110615355) | > loss_mel_ce: 2.7415406703948975 (nan) | > loss: 0.011007953435182571 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13683242645332724) | > loader_time: 0.0046 (0.00466540082314921)  --> TIME: 2025-02-27 03:22:33 -- STEP: 9992/36938 -- GLOBAL_STEP: 1524450 | > loss_text_ce: 0.03022952191531658 (0.03024713336296224) | > loss_mel_ce: 2.8218696117401123 (nan) | > loss: 0.01131785474717617 (nan) | > current_lr: 5e-06 | > step_time: 0.1713 (0.13682665867362612) | > loader_time: 0.0049 (0.004665628278417721)  --> TIME: 2025-02-27 03:22:42 -- STEP: 10042/36938 -- GLOBAL_STEP: 1524500 | > loss_text_ce: 0.031683292239904404 (0.030248923587774732) | > loss_mel_ce: 3.0092363357543945 (nan) | > loss: 0.012067141942679882 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.13682412847155365) | > loader_time: 0.0049 (0.0046655190131075965)  --> TIME: 2025-02-27 03:22:51 -- STEP: 10092/36938 -- GLOBAL_STEP: 1524550 | > loss_text_ce: 0.033474504947662354 (0.030246773981177318) | > loss_mel_ce: 2.678088426589966 (nan) | > loss: 0.01076017040759325 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.13682123398525664) | > loader_time: 0.0048 (0.004665539560457852)  --> TIME: 2025-02-27 03:23:01 -- STEP: 10142/36938 -- GLOBAL_STEP: 1524600 | > loss_text_ce: 0.03241480141878128 (0.030252225826702107) | > loss_mel_ce: 2.886334180831909 (nan) | > loss: 0.011582338251173496 (nan) | > current_lr: 5e-06 | > step_time: 0.1748 (0.13681872973332032) | > loader_time: 0.0048 (0.004665531037333825)  --> TIME: 2025-02-27 03:23:10 -- STEP: 10192/36938 -- GLOBAL_STEP: 1524650 | > loss_text_ce: 0.02093156985938549 (0.030253358383215606) | > loss_mel_ce: 1.2339352369308472 (nan) | > loss: 0.004979630466550589 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13682000037265135) | > loader_time: 0.0045 (0.004665416184362479)  --> TIME: 2025-02-27 03:23:19 -- STEP: 10242/36938 -- GLOBAL_STEP: 1524700 | > loss_text_ce: 0.031677935272455215 (0.03025620055141782) | > loss_mel_ce: 1.6513947248458862 (nan) | > loss: 0.006678860168904066 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13682371409370933) | > loader_time: 0.0046 (0.004665304035722366)  --> TIME: 2025-02-27 03:23:29 -- STEP: 10292/36938 -- GLOBAL_STEP: 1524750 | > loss_text_ce: 0.027858871966600418 (0.030252950466529333) | > loss_mel_ce: 2.0124003887176514 (nan) | > loss: 0.008096267469227314 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13684387133828496) | > loader_time: 0.0046 (0.004665278040199862)  --> TIME: 2025-02-27 03:23:39 -- STEP: 10342/36938 -- GLOBAL_STEP: 1524800 | > loss_text_ce: 0.029557103291153908 (0.03024705417751055) | > loss_mel_ce: 1.9359104633331299 (nan) | > loss: 0.007799474988132715 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13686271978056277) | > loader_time: 0.0046 (0.0046652641685528675)  --> TIME: 2025-02-27 03:23:48 -- STEP: 10392/36938 -- GLOBAL_STEP: 1524850 | > loss_text_ce: 0.0466420017182827 (0.030244257303063765) | > loss_mel_ce: 2.3505859375 (nan) | > loss: 0.009512810036540031 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.13686177870968463) | > loader_time: 0.0044 (0.004665090245774752)  --> TIME: 2025-02-27 03:23:57 -- STEP: 10442/36938 -- GLOBAL_STEP: 1524900 | > loss_text_ce: 0.01604510471224785 (0.030243001223428163) | > loss_mel_ce: 3.453125 (nan) | > loss: 0.013766548596322536 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13687697668866486) | > loader_time: 0.0045 (0.004665179376834155)  --> TIME: 2025-02-27 03:24:06 -- STEP: 10492/36938 -- GLOBAL_STEP: 1524950 | > loss_text_ce: 0.025216754525899887 (0.030242162293268437) | > loss_mel_ce: 3.0186173915863037 (nan) | > loss: 0.012078708037734032 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.13686071912886946) | > loader_time: 0.0047 (0.004665065961533092)  --> TIME: 2025-02-27 03:24:16 -- STEP: 10542/36938 -- GLOBAL_STEP: 1525000 | > loss_text_ce: 0.02406213991343975 (0.03024024141041038) | > loss_mel_ce: 2.4817631244659424 (nan) | > loss: 0.009943751618266106 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13688655386747256) | > loader_time: 0.0048 (0.004665155900178294)  --> TIME: 2025-02-27 03:24:26 -- STEP: 10592/36938 -- GLOBAL_STEP: 1525050 | > loss_text_ce: 0.035054825246334076 (0.030239916177785858) | > loss_mel_ce: 1.5009844303131104 (nan) | > loss: 0.006095394026488066 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13691254178472875) | > loader_time: 0.0045 (0.004665215997717533)  --> TIME: 2025-02-27 03:24:35 -- STEP: 10642/36938 -- GLOBAL_STEP: 1525100 | > loss_text_ce: 0.025035502389073372 (0.030242064647450745) | > loss_mel_ce: 1.8071036338806152 (nan) | > loss: 0.007270393893122673 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.1369321928999294) | > loader_time: 0.0048 (0.004665188111884326)  --> TIME: 2025-02-27 03:24:44 -- STEP: 10692/36938 -- GLOBAL_STEP: 1525150 | > loss_text_ce: 0.029585668817162514 (0.030247121879826075) | > loss_mel_ce: 2.8435463905334473 (nan) | > loss: 0.01140131801366806 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.13691276501414043) | > loader_time: 0.0047 (0.0046650853622642085)  --> TIME: 2025-02-27 03:24:54 -- STEP: 10742/36938 -- GLOBAL_STEP: 1525200 | > loss_text_ce: 0.03290287405252457 (0.030248035874104687) | > loss_mel_ce: 2.8297245502471924 (nan) | > loss: 0.01135963387787342 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13691360225581636) | > loader_time: 0.0047 (0.004664993068621915)  --> TIME: 2025-02-27 03:25:03 -- STEP: 10792/36938 -- GLOBAL_STEP: 1525250 | > loss_text_ce: 0.02922753058373928 (0.030244573315186454) | > loss_mel_ce: 2.150592565536499 (nan) | > loss: 0.008650080300867558 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13691364782134038) | > loader_time: 0.0046 (0.004664714266584038)  --> TIME: 2025-02-27 03:25:12 -- STEP: 10842/36938 -- GLOBAL_STEP: 1525300 | > loss_text_ce: 0.034475527703762054 (0.030245548827576057) | > loss_mel_ce: 2.7121732234954834 (nan) | > loss: 0.01089940033853054 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.1368910336358109) | > loader_time: 0.0045 (0.004664416595526538)  --> TIME: 2025-02-27 03:25:22 -- STEP: 10892/36938 -- GLOBAL_STEP: 1525350 | > loss_text_ce: 0.032053109258413315 (0.030244911412663653) | > loss_mel_ce: 2.8774585723876953 (nan) | > loss: 0.011545681394636631 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1369203645168226) | > loader_time: 0.0046 (0.00466446247426543)  --> TIME: 2025-02-27 03:25:31 -- STEP: 10942/36938 -- GLOBAL_STEP: 1525400 | > loss_text_ce: 0.028942452743649483 (0.030240410322034388) | > loss_mel_ce: 1.572265625 (nan) | > loss: 0.006354000885039568 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13691165263769187) | > loader_time: 0.0045 (0.004664303201401041)  --> TIME: 2025-02-27 03:25:40 -- STEP: 10992/36938 -- GLOBAL_STEP: 1525450 | > loss_text_ce: 0.03037952072918415 (0.030240578857793438) | > loss_mel_ce: 2.6402335166931152 (nan) | > loss: 0.010597671382129192 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.1368961188036632) | > loader_time: 0.0049 (0.004664652038696376)  --> TIME: 2025-02-27 03:25:50 -- STEP: 11042/36938 -- GLOBAL_STEP: 1525500 | > loss_text_ce: 0.033495064824819565 (0.03023545365782953) | > loss_mel_ce: 0.5067367553710938 (nan) | > loss: 0.002143777208402753 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.1369066624035982) | > loader_time: 0.0046 (0.004664906641858747)  --> TIME: 2025-02-27 03:25:59 -- STEP: 11092/36938 -- GLOBAL_STEP: 1525550 | > loss_text_ce: 0.028671640902757645 (0.03023478725127099) | > loss_mel_ce: 2.9110641479492188 (nan) | > loss: 0.011665618978440762 (nan) | > current_lr: 5e-06 | > step_time: 0.1662 (0.1369124799268239) | > loader_time: 0.0048 (0.004665381268736176)  --> TIME: 2025-02-27 03:26:08 -- STEP: 11142/36938 -- GLOBAL_STEP: 1525600 | > loss_text_ce: 0.03130090609192848 (0.030233517506486996) | > loss_mel_ce: 2.803457498550415 (nan) | > loss: 0.011249042116105556 (nan) | > current_lr: 5e-06 | > step_time: 0.2032 (0.13691073242027296) | > loader_time: 0.0048 (0.004665192593109068)  --> TIME: 2025-02-27 03:26:18 -- STEP: 11192/36938 -- GLOBAL_STEP: 1525650 | > loss_text_ce: 0.03972925990819931 (0.030229250929239143) | > loss_mel_ce: 2.275956630706787 (nan) | > loss: 0.009189230389893055 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1369307840441365) | > loader_time: 0.0046 (0.004665052724617391)  --> TIME: 2025-02-27 03:26:27 -- STEP: 11242/36938 -- GLOBAL_STEP: 1525700 | > loss_text_ce: 0.030979851260781288 (0.030230981857313564) | > loss_mel_ce: 2.0911810398101807 (nan) | > loss: 0.008421273902058601 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1369389489417697) | > loader_time: 0.0046 (0.0046649113432662434)  --> TIME: 2025-02-27 03:26:37 -- STEP: 11292/36938 -- GLOBAL_STEP: 1525750 | > loss_text_ce: 0.03612793982028961 (0.0302326961620399) | > loss_mel_ce: 2.4971213340759277 (nan) | > loss: 0.010052577592432499 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13695323893756187) | > loader_time: 0.0046 (0.004664738909639954)  --> TIME: 2025-02-27 03:26:46 -- STEP: 11342/36938 -- GLOBAL_STEP: 1525800 | > loss_text_ce: 0.01917409338057041 (0.03023522711036918) | > loss_mel_ce: 0.6441551446914673 (nan) | > loss: 0.0026322591584175825 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13691776804855926) | > loader_time: 0.0045 (0.004664818859251686)  --> TIME: 2025-02-27 03:26:55 -- STEP: 11392/36938 -- GLOBAL_STEP: 1525850 | > loss_text_ce: 0.03308773413300514 (0.03024020155903548) | > loss_mel_ce: 3.0890002250671387 (nan) | > loss: 0.01238923892378807 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.1369145201372635) | > loader_time: 0.0046 (0.0046646480312508105)  --> TIME: 2025-02-27 03:27:04 -- STEP: 11442/36938 -- GLOBAL_STEP: 1525900 | > loss_text_ce: 0.01924789696931839 (0.030238282608753563) | > loss_mel_ce: 0.12200927734375 (nan) | > loss: 0.0005605443730019033 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13691263927319064) | > loader_time: 0.0047 (0.0046647770216031136)  --> TIME: 2025-02-27 03:27:14 -- STEP: 11492/36938 -- GLOBAL_STEP: 1525950 | > loss_text_ce: 0.021082090213894844 (0.030236506412402855) | > loss_mel_ce: 0.0059814453125 (nan) | > loss: 0.00010739498975453898 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.1369097518497674) | > loader_time: 0.0046 (0.00466489329223553)  --> TIME: 2025-02-27 03:27:23 -- STEP: 11542/36938 -- GLOBAL_STEP: 1526000 | > loss_text_ce: 0.03945519030094147 (0.030240881891142983) | > loss_mel_ce: 0.060564421117305756 (nan) | > loss: 0.00039690325502306223 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13692018938651251) | > loader_time: 0.0045 (0.0046652837221842665)  --> TIME: 2025-02-27 03:27:32 -- STEP: 11592/36938 -- GLOBAL_STEP: 1526050 | > loss_text_ce: 0.031211182475090027 (0.030240084242436894) | > loss_mel_ce: 2.987053632736206 (nan) | > loss: 0.011977242305874825 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13691738173828388) | > loader_time: 0.0047 (0.004665416816417884)  --> TIME: 2025-02-27 03:27:42 -- STEP: 11642/36938 -- GLOBAL_STEP: 1526100 | > loss_text_ce: 0.03273271396756172 (0.030241672720474035) | > loss_mel_ce: 2.691274881362915 (nan) | > loss: 0.01080955471843481 (nan) | > current_lr: 5e-06 | > step_time: 0.1791 (0.13693870053916374) | > loader_time: 0.0048 (0.004665767054974759)  --> TIME: 2025-02-27 03:27:51 -- STEP: 11692/36938 -- GLOBAL_STEP: 1526150 | > loss_text_ce: 0.03604379668831825 (0.03023952335435317) | > loss_mel_ce: 1.7921617031097412 (nan) | > loss: 0.007254783995449543 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13692339046581406) | > loader_time: 0.0046 (0.004665581771375545)  --> TIME: 2025-02-27 03:28:00 -- STEP: 11742/36938 -- GLOBAL_STEP: 1526200 | > loss_text_ce: 0.025812311097979546 (0.030241053185632392) | > loss_mel_ce: 2.225486993789673 (nan) | > loss: 0.00893372856080532 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.13690987950674136) | > loader_time: 0.0049 (0.004665408989697254)  --> TIME: 2025-02-27 03:28:10 -- STEP: 11792/36938 -- GLOBAL_STEP: 1526250 | > loss_text_ce: 0.03298439830541611 (0.03024664227367382) | > loss_mel_ce: 2.740628719329834 (nan) | > loss: 0.011006401851773262 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1369083192164089) | > loader_time: 0.0052 (0.0046655941688868875)  --> TIME: 2025-02-27 03:28:19 -- STEP: 11842/36938 -- GLOBAL_STEP: 1526300 | > loss_text_ce: 0.025848768651485443 (0.03025248591023519) | > loss_mel_ce: 2.5979669094085693 (nan) | > loss: 0.010411967523396015 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.13694282388228118) | > loader_time: 0.0046 (0.004665819721998264)  --> TIME: 2025-02-27 03:28:29 -- STEP: 11892/36938 -- GLOBAL_STEP: 1526350 | > loss_text_ce: 0.02774977684020996 (0.030251575496675506) | > loss_mel_ce: 2.471220016479492 (nan) | > loss: 0.009916547685861588 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13694269904535417) | > loader_time: 0.0049 (0.004666411952863348)  --> TIME: 2025-02-27 03:28:38 -- STEP: 11942/36938 -- GLOBAL_STEP: 1526400 | > loss_text_ce: 0.028920985758304596 (0.030247910727243452) | > loss_mel_ce: 5.942708492279053 (nan) | > loss: 0.02369694411754608 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13692426016693698) | > loader_time: 0.0077 (0.004666821378855783)  --> TIME: 2025-02-27 03:28:47 -- STEP: 11992/36938 -- GLOBAL_STEP: 1526450 | > loss_text_ce: 0.025988692417740822 (0.030249557050527145) | > loss_mel_ce: 2.8699498176574707 (nan) | > loss: 0.01149181928485632 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.13692265715179192) | > loader_time: 0.0047 (0.0046667935372671325)  --> TIME: 2025-02-27 03:28:56 -- STEP: 12042/36938 -- GLOBAL_STEP: 1526500 | > loss_text_ce: 0.026853838935494423 (0.03024451407118072) | > loss_mel_ce: 2.093031406402588 (nan) | > loss: 0.00841224379837513 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13691709893344972) | > loader_time: 0.0045 (0.004666725081713215)  --> TIME: 2025-02-27 03:29:06 -- STEP: 12092/36938 -- GLOBAL_STEP: 1526550 | > loss_text_ce: 0.03680242598056793 (0.03024633750579807) | > loss_mel_ce: 2.6642818450927734 (nan) | > loss: 0.010718589648604393 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13689695623510165) | > loader_time: 0.0045 (0.00466658993742145)  --> TIME: 2025-02-27 03:29:15 -- STEP: 12142/36938 -- GLOBAL_STEP: 1526600 | > loss_text_ce: 0.03163333237171173 (0.030254134151264173) | > loss_mel_ce: 2.6060707569122314 (nan) | > loss: 0.010467080399394035 (nan) | > current_lr: 5e-06 | > step_time: 0.1857 (0.13687943874208983) | > loader_time: 0.0059 (0.004666640483219686)  --> TIME: 2025-02-27 03:29:24 -- STEP: 12192/36938 -- GLOBAL_STEP: 1526650 | > loss_text_ce: 0.03293925151228905 (0.03025777426913742) | > loss_mel_ce: 3.006138801574707 (nan) | > loss: 0.012059833854436874 (nan) | > current_lr: 5e-06 | > step_time: 0.1851 (0.1369002219142879) | > loader_time: 0.0049 (0.004666804700229426)  --> TIME: 2025-02-27 03:29:34 -- STEP: 12242/36938 -- GLOBAL_STEP: 1526700 | > loss_text_ce: 0.03167719393968582 (0.030259550368333294) | > loss_mel_ce: 2.6137688159942627 (nan) | > loss: 0.01049780286848545 (nan) | > current_lr: 5e-06 | > step_time: 0.184 (0.13690969062151456) | > loader_time: 0.0048 (0.0046669561631966635)  --> TIME: 2025-02-27 03:29:43 -- STEP: 12292/36938 -- GLOBAL_STEP: 1526750 | > loss_text_ce: 0.03160788118839264 (0.030261703921749268) | > loss_mel_ce: 0.082122802734375 (nan) | > loss: 0.0004513122548814863 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.1369047976843592) | > loader_time: 0.0046 (0.0046671396585074125)  --> TIME: 2025-02-27 03:29:53 -- STEP: 12342/36938 -- GLOBAL_STEP: 1526800 | > loss_text_ce: 0.019885094836354256 (0.030265417431117915) | > loss_mel_ce: 2.6426329612731934 (nan) | > loss: 0.010565548203885555 (nan) | > current_lr: 5e-06 | > step_time: 0.1692 (0.1369321696512166) | > loader_time: 0.0049 (0.004667156327197709)  --> TIME: 2025-02-27 03:30:02 -- STEP: 12392/36938 -- GLOBAL_STEP: 1526850 | > loss_text_ce: 0.027950329706072807 (0.03026863836771715) | > loss_mel_ce: 2.565037965774536 (nan) | > loss: 0.010289636440575123 (nan) | > current_lr: 5e-06 | > step_time: 0.165 (0.1369356983626869) | > loader_time: 0.0047 (0.004667243297982025)  --> TIME: 2025-02-27 03:30:11 -- STEP: 12442/36938 -- GLOBAL_STEP: 1526900 | > loss_text_ce: 0.03806250914931297 (0.030273518549166694) | > loss_mel_ce: 3.281240940093994 (nan) | > loss: 0.013171840459108353 (nan) | > current_lr: 5e-06 | > step_time: 0.1617 (0.13691997731197633) | > loader_time: 0.0049 (0.004667108588961431)  --> TIME: 2025-02-27 03:30:21 -- STEP: 12492/36938 -- GLOBAL_STEP: 1526950 | > loss_text_ce: 0.017792239785194397 (0.03026581786765579) | > loss_mel_ce: 2.789806604385376 (nan) | > loss: 0.011141265742480755 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1369219084714341) | > loader_time: 0.0045 (0.004667048571852592)  --> TIME: 2025-02-27 03:30:30 -- STEP: 12542/36938 -- GLOBAL_STEP: 1527000 | > loss_text_ce: 0.023874511942267418 (0.030265960118074985) | > loss_mel_ce: 2.301731586456299 (nan) | > loss: 0.009228596463799477 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1369132526448196) | > loader_time: 0.0045 (0.004666826634140151)  --> TIME: 2025-02-27 03:30:39 -- STEP: 12592/36938 -- GLOBAL_STEP: 1527050 | > loss_text_ce: 0.022566789761185646 (0.030262183848581495) | > loss_mel_ce: 2.836236000061035 (nan) | > loss: 0.011344456113874912 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13689112805275042) | > loader_time: 0.0046 (0.004666502548445591)  --> TIME: 2025-02-27 03:30:48 -- STEP: 12642/36938 -- GLOBAL_STEP: 1527100 | > loss_text_ce: 0.028558140620589256 (0.03025903037390168) | > loss_mel_ce: 2.768411874771118 (nan) | > loss: 0.011099088005721569 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.1368913294827329) | > loader_time: 0.0055 (0.00466687091636688)  --> TIME: 2025-02-27 03:30:57 -- STEP: 12692/36938 -- GLOBAL_STEP: 1527150 | > loss_text_ce: 0.030702969059348106 (0.030256373914994184) | > loss_mel_ce: 2.816720485687256 (nan) | > loss: 0.011299300007522106 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.13687778871742443) | > loader_time: 0.0047 (0.004667051181107944)  --> TIME: 2025-02-27 03:31:06 -- STEP: 12742/36938 -- GLOBAL_STEP: 1527200 | > loss_text_ce: 0.02294253371655941 (0.03024994380696976) | > loss_mel_ce: 2.6903953552246094 (nan) | > loss: 0.010767214931547642 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.1368583881804491) | > loader_time: 0.0046 (0.004667113853051032)  --> TIME: 2025-02-27 03:31:16 -- STEP: 12792/36938 -- GLOBAL_STEP: 1527250 | > loss_text_ce: 0.03304838761687279 (0.030249473898237064) | > loss_mel_ce: 2.7792885303497314 (nan) | > loss: 0.011160068213939667 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.13686167666358895) | > loader_time: 0.0049 (0.004667389348121347)  --> TIME: 2025-02-27 03:31:25 -- STEP: 12842/36938 -- GLOBAL_STEP: 1527300 | > loss_text_ce: 0.024270739406347275 (0.030247089539269748) | > loss_mel_ce: 2.753065347671509 (nan) | > loss: 0.011021175421774387 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.13686592854400448) | > loader_time: 0.005 (0.004667494327103758)  --> TIME: 2025-02-27 03:31:35 -- STEP: 12892/36938 -- GLOBAL_STEP: 1527350 | > loss_text_ce: 0.03301289677619934 (0.03024301830318843) | > loss_mel_ce: 3.1315479278564453 (nan) | > loss: 0.012557781301438808 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.13686967201200348) | > loader_time: 0.0047 (0.004667816197616957)  --> TIME: 2025-02-27 03:31:44 -- STEP: 12942/36938 -- GLOBAL_STEP: 1527400 | > loss_text_ce: 0.027318622916936874 (0.03024184623453377) | > loss_mel_ce: 2.961702585220337 (nan) | > loss: 0.011861196719110012 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13687137246887188) | > loader_time: 0.0046 (0.004667740851166338)  --> TIME: 2025-02-27 03:31:53 -- STEP: 12992/36938 -- GLOBAL_STEP: 1527450 | > loss_text_ce: 0.02764781564474106 (0.030234096586211637) | > loss_mel_ce: 2.3213393688201904 (nan) | > loss: 0.009321377612650394 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13687141760816698) | > loader_time: 0.0049 (0.004667612939632592)  --> TIME: 2025-02-27 03:32:03 -- STEP: 13042/36938 -- GLOBAL_STEP: 1527500 | > loss_text_ce: 0.02694840542972088 (0.03023191017680426) | > loss_mel_ce: 2.5598254203796387 (nan) | > loss: 0.01026497595012188 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.13690281003882698) | > loader_time: 0.0046 (0.004667594505588521)  --> TIME: 2025-02-27 03:32:12 -- STEP: 13092/36938 -- GLOBAL_STEP: 1527550 | > loss_text_ce: 0.03364263474941254 (0.030231003246969815) | > loss_mel_ce: 1.961538314819336 (nan) | > loss: 0.007917385548353195 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13689844689527997) | > loader_time: 0.0047 (0.00466758519037762)  --> TIME: 2025-02-27 03:32:22 -- STEP: 13142/36938 -- GLOBAL_STEP: 1527600 | > loss_text_ce: 0.01882092095911503 (0.030230571555047134) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13688654586470955) | > loader_time: 0.0046 (0.004667570086270056)  --> TIME: 2025-02-27 03:32:31 -- STEP: 13192/36938 -- GLOBAL_STEP: 1527650 | > loss_text_ce: 0.029926959425210953 (0.030232929850740973) | > loss_mel_ce: 1.930214762687683 (nan) | > loss: 0.0077783409506082535 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13690404186399152) | > loader_time: 0.0046 (0.004667602411684233)  --> TIME: 2025-02-27 03:32:41 -- STEP: 13242/36938 -- GLOBAL_STEP: 1527700 | > loss_text_ce: 0.024716874584555626 (0.030229783832946423) | > loss_mel_ce: 1.7019380331039429 (nan) | > loss: 0.0068518053740262985 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13690007062427165) | > loader_time: 0.0048 (0.004667618972911994)  --> TIME: 2025-02-27 03:32:50 -- STEP: 13292/36938 -- GLOBAL_STEP: 1527750 | > loss_text_ce: 0.016978969797492027 (0.030231336052306698) | > loss_mel_ce: 2.9960501194000244 (nan) | > loss: 0.011956465430557728 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13689061666589686) | > loader_time: 0.0045 (0.004667665579575146)  --> TIME: 2025-02-27 03:32:59 -- STEP: 13342/36938 -- GLOBAL_STEP: 1527800 | > loss_text_ce: 0.03370368480682373 (0.03022670509891754) | > loss_mel_ce: 2.5618693828582764 (nan) | > loss: 0.010299893096089363 (nan) | > current_lr: 5e-06 | > step_time: 0.187 (0.13688552592258538) | > loader_time: 0.0049 (0.004667769788605695)  --> TIME: 2025-02-27 03:33:08 -- STEP: 13392/36938 -- GLOBAL_STEP: 1527850 | > loss_text_ce: 0.029018668457865715 (0.03022583768812365) | > loss_mel_ce: 2.7997732162475586 (nan) | > loss: 0.011225365102291107 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.1368752453240302) | > loader_time: 0.0047 (0.004667846586114629)  --> TIME: 2025-02-27 03:33:17 -- STEP: 13442/36938 -- GLOBAL_STEP: 1527900 | > loss_text_ce: 0.03357480466365814 (0.03023060898049466) | > loss_mel_ce: 2.4859678745269775 (nan) | > loss: 0.009998185560107231 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13686396378240626) | > loader_time: 0.0046 (0.004667741825601538)  --> TIME: 2025-02-27 03:33:26 -- STEP: 13492/36938 -- GLOBAL_STEP: 1527950 | > loss_text_ce: 0.022971246391534805 (0.030231192435079664) | > loss_mel_ce: 0.05743587762117386 (nan) | > loss: 0.0003190759161952883 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13685531676999588) | > loader_time: 0.0045 (0.00466755528213927)  --> TIME: 2025-02-27 03:33:35 -- STEP: 13542/36938 -- GLOBAL_STEP: 1528000 | > loss_text_ce: 0.027926551178097725 (0.030228776297995686) | > loss_mel_ce: 3.148850202560425 (nan) | > loss: 0.012606257572770119 (nan) | > current_lr: 5e-06 | > step_time: 0.1827 (0.13684380725257242) | > loader_time: 0.0048 (0.004667363355543929)  --> TIME: 2025-02-27 03:33:45 -- STEP: 13592/36938 -- GLOBAL_STEP: 1528050 | > loss_text_ce: 0.03694332018494606 (0.03022548352097385) | > loss_mel_ce: 2.3916585445404053 (nan) | > loss: 0.009637310169637203 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13682741086366568) | > loader_time: 0.0046 (0.004667195767216865)  --> TIME: 2025-02-27 03:33:54 -- STEP: 13642/36938 -- GLOBAL_STEP: 1528100 | > loss_text_ce: 0.03798436000943184 (0.030224808860281276) | > loss_mel_ce: 2.872307538986206 (nan) | > loss: 0.011548778042197227 (nan) | > current_lr: 5e-06 | > step_time: 0.1829 (0.13682520739093032) | > loader_time: 0.0048 (0.004667228800235768)  --> TIME: 2025-02-27 03:34:03 -- STEP: 13692/36938 -- GLOBAL_STEP: 1528150 | > loss_text_ce: 0.029339758679270744 (0.03022576264126687) | > loss_mel_ce: 2.808321237564087 (nan) | > loss: 0.0112605607137084 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13683159618464086) | > loader_time: 0.0048 (0.004667405022747377)  --> TIME: 2025-02-27 03:34:13 -- STEP: 13742/36938 -- GLOBAL_STEP: 1528200 | > loss_text_ce: 0.024797355756163597 (0.030224800948338237) | > loss_mel_ce: 3.187778949737549 (nan) | > loss: 0.012748319655656815 (nan) | > current_lr: 5e-06 | > step_time: 0.1695 (0.13682640186998835) | > loader_time: 0.0049 (0.004667460909751712)  --> TIME: 2025-02-27 03:34:22 -- STEP: 13792/36938 -- GLOBAL_STEP: 1528250 | > loss_text_ce: 0.04324121028184891 (0.03022484166027433) | > loss_mel_ce: 2.5416877269744873 (nan) | > loss: 0.010257655754685402 (nan) | > current_lr: 5e-06 | > step_time: 0.1529 (0.13682784975059104) | > loader_time: 0.0046 (0.004667758198321016)  --> TIME: 2025-02-27 03:34:31 -- STEP: 13842/36938 -- GLOBAL_STEP: 1528300 | > loss_text_ce: 0.023347608745098114 (0.030222606551041534) | > loss_mel_ce: 1.7529394626617432 (nan) | > loss: 0.007048758678138256 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.13682978579702143) | > loader_time: 0.0047 (0.004667582926662696)  --> TIME: 2025-02-27 03:34:41 -- STEP: 13892/36938 -- GLOBAL_STEP: 1528350 | > loss_text_ce: 0.017788661643862724 (0.030226763192507396) | > loss_mel_ce: 2.6821601390838623 (nan) | > loss: 0.010714082978665829 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13682164153550258) | > loader_time: 0.0047 (0.004667494916929636)  --> TIME: 2025-02-27 03:34:50 -- STEP: 13942/36938 -- GLOBAL_STEP: 1528400 | > loss_text_ce: 0.020283250138163567 (0.03022365468797216) | > loss_mel_ce: 1.2061767578125 (nan) | > loss: 0.00486690504476428 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13685326729455619) | > loader_time: 0.0045 (0.004667514589112977)  --> TIME: 2025-02-27 03:34:59 -- STEP: 13992/36938 -- GLOBAL_STEP: 1528450 | > loss_text_ce: 0.02487293630838394 (0.030221266708890023) | > loss_mel_ce: 0.9395034313201904 (nan) | > loss: 0.003826890606433153 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13684493480511192) | > loader_time: 0.0046 (0.004667669483564334)  --> TIME: 2025-02-27 03:35:09 -- STEP: 14042/36938 -- GLOBAL_STEP: 1528500 | > loss_text_ce: 0.027016762644052505 (0.030219402212637376) | > loss_mel_ce: 0.0086669921875 (nan) | > loss: 0.000141602213261649 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.1368405386824639) | > loader_time: 0.0046 (0.004667743932248328)  --> TIME: 2025-02-27 03:35:18 -- STEP: 14092/36938 -- GLOBAL_STEP: 1528550 | > loss_text_ce: 0.02864990197122097 (0.030214705436273435) | > loss_mel_ce: 2.465472459793091 (nan) | > loss: 0.009897311218082905 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13684322033421165) | > loader_time: 0.0045 (0.00466776278219947)  --> TIME: 2025-02-27 03:35:28 -- STEP: 14142/36938 -- GLOBAL_STEP: 1528600 | > loss_text_ce: 0.027585051953792572 (0.030216731082080627) | > loss_mel_ce: 0.31152647733688354 (nan) | > loss: 0.0013456807937473059 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13684692411135477) | > loader_time: 0.0047 (0.004667738070330863)  --> TIME: 2025-02-27 03:35:37 -- STEP: 14192/36938 -- GLOBAL_STEP: 1528650 | > loss_text_ce: 0.024442272260785103 (0.030217149215640147) | > loss_mel_ce: 2.982611894607544 (nan) | > loss: 0.011932754889130592 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.13683293138241176) | > loader_time: 0.0047 (0.004667703503282939)  --> TIME: 2025-02-27 03:35:46 -- STEP: 14242/36938 -- GLOBAL_STEP: 1528700 | > loss_text_ce: 0.032852429896593094 (0.030221758417527778) | > loss_mel_ce: 2.3432083129882812 (nan) | > loss: 0.009428813122212887 (nan) | > current_lr: 5e-06 | > step_time: 0.1566 (0.1368386863306191) | > loader_time: 0.0046 (0.0046675504551327695)  --> TIME: 2025-02-27 03:35:56 -- STEP: 14292/36938 -- GLOBAL_STEP: 1528750 | > loss_text_ce: 0.03563351184129715 (0.030222720741024542) | > loss_mel_ce: 2.4774482250213623 (nan) | > loss: 0.009972547180950642 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13684617111805042) | > loader_time: 0.0046 (0.0046676959672496115)  --> TIME: 2025-02-27 03:36:05 -- STEP: 14342/36938 -- GLOBAL_STEP: 1528800 | > loss_text_ce: 0.026760663837194443 (0.0302233880592515) | > loss_mel_ce: 2.8058125972747803 (nan) | > loss: 0.011240370571613312 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.13683906272770655) | > loader_time: 0.0048 (0.0046675967099593235)  --> TIME: 2025-02-27 03:36:14 -- STEP: 14392/36938 -- GLOBAL_STEP: 1528850 | > loss_text_ce: 0.0342598520219326 (0.030226120388646787) | > loss_mel_ce: 1.46990168094635 (nan) | > loss: 0.005968895275145769 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13682607379537515) | > loader_time: 0.0047 (0.00466761478389086)  --> TIME: 2025-02-27 03:36:24 -- STEP: 14442/36938 -- GLOBAL_STEP: 1528900 | > loss_text_ce: 0.027184896171092987 (0.030225752807716146) | > loss_mel_ce: 2.5337581634521484 (nan) | > loss: 0.010162473656237125 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.13684205962026463) | > loader_time: 0.0053 (0.004667928469443561)  --> TIME: 2025-02-27 03:36:33 -- STEP: 14492/36938 -- GLOBAL_STEP: 1528950 | > loss_text_ce: 0.03985112905502319 (0.030227774270125636) | > loss_mel_ce: 2.687145233154297 (nan) | > loss: 0.010821415111422539 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13685176206694585) | > loader_time: 0.0045 (0.004668187131020971)  --> TIME: 2025-02-27 03:36:42 -- STEP: 14542/36938 -- GLOBAL_STEP: 1529000 | > loss_text_ce: 0.030317097902297974 (0.03022736557847084) | > loss_mel_ce: 1.982862949371338 (nan) | > loss: 0.00798880960792303 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13684083874067254) | > loader_time: 0.0045 (0.004668122225853107)  --> TIME: 2025-02-27 03:36:52 -- STEP: 14592/36938 -- GLOBAL_STEP: 1529050 | > loss_text_ce: 0.031811799854040146 (0.030234972167180095) | > loss_mel_ce: 2.7597169876098633 (nan) | > loss: 0.01107749529182911 (nan) | > current_lr: 5e-06 | > step_time: 0.1492 (0.13686037223720773) | > loader_time: 0.0048 (0.004668166730226144)  --> TIME: 2025-02-27 03:37:01 -- STEP: 14642/36938 -- GLOBAL_STEP: 1529100 | > loss_text_ce: 0.028924264013767242 (0.03023310242839775) | > loss_mel_ce: 2.8534305095672607 (nan) | > loss: 0.011437916196882725 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.1368467744753802) | > loader_time: 0.0049 (0.004668160664418127)  --> TIME: 2025-02-27 03:37:10 -- STEP: 14692/36938 -- GLOBAL_STEP: 1529150 | > loss_text_ce: 0.031471963971853256 (0.030236943490654145) | > loss_mel_ce: 0.006134033203125 (nan) | > loss: 0.00014923015260137618 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13684387163865355) | > loader_time: 0.0045 (0.004668261694577107)  --> TIME: 2025-02-27 03:37:20 -- STEP: 14742/36938 -- GLOBAL_STEP: 1529200 | > loss_text_ce: 0.031754158437252045 (0.030235464224500117) | > loss_mel_ce: 2.8142032623291016 (nan) | > loss: 0.011293482966721058 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.1368504625989859) | > loader_time: 0.0048 (0.004668125432182285)  --> TIME: 2025-02-27 03:37:29 -- STEP: 14792/36938 -- GLOBAL_STEP: 1529250 | > loss_text_ce: 0.033307500183582306 (0.03023478171670796) | > loss_mel_ce: 1.921903133392334 (nan) | > loss: 0.007758772931993008 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13686255245933435) | > loader_time: 0.0044 (0.0046680159927252746)  --> TIME: 2025-02-27 03:37:38 -- STEP: 14842/36938 -- GLOBAL_STEP: 1529300 | > loss_text_ce: 0.03153583034873009 (0.030233736660242632) | > loss_mel_ce: 2.19700026512146 (nan) | > loss: 0.008843397721648216 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1368490242280181) | > loader_time: 0.0045 (0.004667758748885077)  --> TIME: 2025-02-27 03:37:48 -- STEP: 14892/36938 -- GLOBAL_STEP: 1529350 | > loss_text_ce: 0.030775176361203194 (0.03023166448106024) | > loss_mel_ce: 2.6300840377807617 (nan) | > loss: 0.010558965615928173 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.1368455614033103) | > loader_time: 0.0045 (0.0046675066585474285)  --> TIME: 2025-02-27 03:37:57 -- STEP: 14942/36938 -- GLOBAL_STEP: 1529400 | > loss_text_ce: 0.030933724716305733 (0.030233989388928392) | > loss_mel_ce: 2.5190165042877197 (nan) | > loss: 0.010118850506842136 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13683889687994785) | > loader_time: 0.0049 (0.004667294693988847)  --> TIME: 2025-02-27 03:38:06 -- STEP: 14992/36938 -- GLOBAL_STEP: 1529450 | > loss_text_ce: 0.02802281081676483 (0.03023470997717927) | > loss_mel_ce: 2.2410643100738525 (nan) | > loss: 0.009004314430058002 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13685035667399115) | > loader_time: 0.0045 (0.004667216122722942)  --> TIME: 2025-02-27 03:38:16 -- STEP: 15042/36938 -- GLOBAL_STEP: 1529500 | > loss_text_ce: 0.025436202064156532 (0.0302333027220249) | > loss_mel_ce: 2.7004880905151367 (nan) | > loss: 0.01081715989857912 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.1368546896103836) | > loader_time: 0.0048 (0.004667412361365158)  --> TIME: 2025-02-27 03:38:26 -- STEP: 15092/36938 -- GLOBAL_STEP: 1529550 | > loss_text_ce: 0.032743118703365326 (0.030233512933321813) | > loss_mel_ce: 1.8296051025390625 (nan) | > loss: 0.007390270940959454 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13687672853280314) | > loader_time: 0.0045 (0.00466768549823838)  --> TIME: 2025-02-27 03:38:35 -- STEP: 15142/36938 -- GLOBAL_STEP: 1529600 | > loss_text_ce: 0.025368191301822662 (0.030233030800495192) | > loss_mel_ce: 2.9817068576812744 (nan) | > loss: 0.011932838708162308 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.1368774876949136) | > loader_time: 0.0046 (0.004667556958191631)  --> TIME: 2025-02-27 03:38:44 -- STEP: 15192/36938 -- GLOBAL_STEP: 1529650 | > loss_text_ce: 0.035854194313287735 (0.03023492385843683) | > loss_mel_ce: 2.5774006843566895 (nan) | > loss: 0.010370058938860893 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13686871544319218) | > loader_time: 0.0046 (0.004667573253878685)  --> TIME: 2025-02-27 03:38:54 -- STEP: 15242/36938 -- GLOBAL_STEP: 1529700 | > loss_text_ce: 0.03920970484614372 (0.03023858836111086) | > loss_mel_ce: 0.8788923025131226 (nan) | > loss: 0.0036432622000575066 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1368704798371444) | > loader_time: 0.0054 (0.004667512780176886)  --> TIME: 2025-02-27 03:39:03 -- STEP: 15292/36938 -- GLOBAL_STEP: 1529750 | > loss_text_ce: 0.03502171114087105 (0.03024078142801711) | > loss_mel_ce: 1.9887855052947998 (nan) | > loss: 0.008030981756746769 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.1368904189853936) | > loader_time: 0.0047 (0.004667822038693929)  --> TIME: 2025-02-27 03:39:12 -- STEP: 15342/36938 -- GLOBAL_STEP: 1529800 | > loss_text_ce: 0.04017145186662674 (0.030240763240669752) | > loss_mel_ce: 2.6040892601013184 (nan) | > loss: 0.010493098758161068 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.136877872287769) | > loader_time: 0.0046 (0.004667666430517473)  --> TIME: 2025-02-27 03:39:22 -- STEP: 15392/36938 -- GLOBAL_STEP: 1529850 | > loss_text_ce: 0.03694871813058853 (0.03023964580907941) | > loss_mel_ce: 2.0938880443573 (nan) | > loss: 0.008455702103674412 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13687676975112484) | > loader_time: 0.0047 (0.004667662719183317)  --> TIME: 2025-02-27 03:39:31 -- STEP: 15442/36938 -- GLOBAL_STEP: 1529900 | > loss_text_ce: 0.03050462156534195 (0.03024066923558422) | > loss_mel_ce: 2.326334238052368 (nan) | > loss: 0.00935253594070673 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13685782296008206) | > loader_time: 0.0045 (0.004667401931366464)  --> TIME: 2025-02-27 03:39:40 -- STEP: 15492/36938 -- GLOBAL_STEP: 1529950 | > loss_text_ce: 0.030298933386802673 (0.03024043142867702) | > loss_mel_ce: 3.1020009517669678 (nan) | > loss: 0.01242976263165474 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.13686624724151525) | > loader_time: 0.0047 (0.004667311052681412)  --> TIME: 2025-02-27 03:39:50 -- STEP: 15542/36938 -- GLOBAL_STEP: 1530000 | > loss_text_ce: 0.03520841896533966 (0.03024294446943357) | > loss_mel_ce: 2.761852264404297 (nan) | > loss: 0.011099448427557945 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1368905557289564) | > loader_time: 0.0047 (0.004667258388424557) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1530000.pth  --> TIME: 2025-02-27 03:40:05 -- STEP: 15592/36938 -- GLOBAL_STEP: 1530050 | > loss_text_ce: 0.02724313922226429 (0.030242842656984088) | > loss_mel_ce: 2.636495590209961 (nan) | > loss: 0.01057039201259613 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1368594769944654) | > loader_time: 0.0045 (0.004667202330907849)  --> TIME: 2025-02-27 03:40:14 -- STEP: 15642/36938 -- GLOBAL_STEP: 1530100 | > loss_text_ce: 0.03248223289847374 (0.030242735156812042) | > loss_mel_ce: 2.6774661540985107 (nan) | > loss: 0.01075376383960247 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13686023653506232) | > loader_time: 0.0046 (0.004667454280446011)  --> TIME: 2025-02-27 03:40:23 -- STEP: 15692/36938 -- GLOBAL_STEP: 1530150 | > loss_text_ce: 0.027077702805399895 (0.030239417290138404) | > loss_mel_ce: 0.9321696162223816 (nan) | > loss: 0.003806537017226219 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13684943668580413) | > loader_time: 0.0045 (0.004667565116401)  --> TIME: 2025-02-27 03:40:32 -- STEP: 15742/36938 -- GLOBAL_STEP: 1530200 | > loss_text_ce: 0.0306624136865139 (0.030238548741149254) | > loss_mel_ce: 2.2830543518066406 (nan) | > loss: 0.009181415662169456 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13683708995985847) | > loader_time: 0.0045 (0.004667424001561669)  --> TIME: 2025-02-27 03:40:41 -- STEP: 15792/36938 -- GLOBAL_STEP: 1530250 | > loss_text_ce: 0.0303901769220829 (0.030238894016489954) | > loss_mel_ce: 3.3154425621032715 (nan) | > loss: 0.013277115300297737 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.136822957987119) | > loader_time: 0.0046 (0.0046672390768230975)  --> TIME: 2025-02-27 03:40:51 -- STEP: 15842/36938 -- GLOBAL_STEP: 1530300 | > loss_text_ce: 0.03200125694274902 (0.030240054574995297) | > loss_mel_ce: 2.3489913940429688 (nan) | > loss: 0.009448383934795856 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13681403602739312) | > loader_time: 0.0046 (0.004667183573597275)  --> TIME: 2025-02-27 03:41:00 -- STEP: 15892/36938 -- GLOBAL_STEP: 1530350 | > loss_text_ce: 0.03438529744744301 (0.030243032500267365) | > loss_mel_ce: 3.172273635864258 (nan) | > loss: 0.01272483728826046 (nan) | > current_lr: 5e-06 | > step_time: 0.1879 (0.13681885293290416) | > loader_time: 0.0048 (0.0046671040856865334)  --> TIME: 2025-02-27 03:41:09 -- STEP: 15942/36938 -- GLOBAL_STEP: 1530400 | > loss_text_ce: 0.03634440898895264 (0.030242256824232813) | > loss_mel_ce: 3.019737958908081 (nan) | > loss: 0.01212731096893549 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.13680608597682364) | > loader_time: 0.0046 (0.004667032319828532)  --> TIME: 2025-02-27 03:41:18 -- STEP: 15992/36938 -- GLOBAL_STEP: 1530450 | > loss_text_ce: 0.03478348255157471 (0.03024286359601913) | > loss_mel_ce: 1.132959008216858 (nan) | > loss: 0.004633898846805096 (nan) | > current_lr: 5e-06 | > step_time: 0.1118 (0.13680054525425017) | > loader_time: 0.0048 (0.004667336744568012)  --> TIME: 2025-02-27 03:41:28 -- STEP: 16042/36938 -- GLOBAL_STEP: 1530500 | > loss_text_ce: 0.026687275618314743 (0.03024027747659528) | > loss_mel_ce: 2.471776247024536 (nan) | > loss: 0.009914538823068142 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13679921081603877) | > loader_time: 0.0045 (0.004667339933110281)  --> TIME: 2025-02-27 03:41:37 -- STEP: 16092/36938 -- GLOBAL_STEP: 1530550 | > loss_text_ce: 0.02919957786798477 (0.03023944268545509) | > loss_mel_ce: 2.2349259853363037 (nan) | > loss: 0.008984626270830631 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13680230849659836) | > loader_time: 0.0046 (0.004667247138804485)  --> TIME: 2025-02-27 03:41:46 -- STEP: 16142/36938 -- GLOBAL_STEP: 1530600 | > loss_text_ce: 0.026714855805039406 (0.030240909172838896) | > loss_mel_ce: 2.9457521438598633 (nan) | > loss: 0.01179550401866436 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.13680804000292554) | > loader_time: 0.0046 (0.004667105498684348)  --> TIME: 2025-02-27 03:41:56 -- STEP: 16192/36938 -- GLOBAL_STEP: 1530650 | > loss_text_ce: 0.027547411620616913 (0.03023997488537004) | > loss_mel_ce: 2.1899428367614746 (nan) | > loss: 0.008799565024673939 (nan) | > current_lr: 5e-06 | > step_time: 0.1588 (0.13683542752042152) | > loader_time: 0.0047 (0.004667141868662939)  --> TIME: 2025-02-27 03:42:06 -- STEP: 16242/36938 -- GLOBAL_STEP: 1530700 | > loss_text_ce: 0.033177450299263 (0.030242585925814948) | > loss_mel_ce: 2.85603666305542 (nan) | > loss: 0.011465135961771011 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.13682583703620174) | > loader_time: 0.0046 (0.004667016896484032)  --> TIME: 2025-02-27 03:42:15 -- STEP: 16292/36938 -- GLOBAL_STEP: 1530750 | > loss_text_ce: 0.033391132950782776 (0.030240284407894582) | > loss_mel_ce: 3.027677297592163 (nan) | > loss: 0.01214709784835577 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13683648667580225) | > loader_time: 0.0047 (0.004667065315096739)  --> TIME: 2025-02-27 03:42:25 -- STEP: 16342/36938 -- GLOBAL_STEP: 1530800 | > loss_text_ce: 0.02738347463309765 (0.03024051903969623) | > loss_mel_ce: 2.393617630004883 (nan) | > loss: 0.009607148356735706 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13684347620968634) | > loader_time: 0.0045 (0.004667045976435488)  --> TIME: 2025-02-27 03:42:34 -- STEP: 16392/36938 -- GLOBAL_STEP: 1530850 | > loss_text_ce: 0.02695458196103573 (0.030236506268457974) | > loss_mel_ce: 2.9137089252471924 (nan) | > loss: 0.011669300496578217 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13683596531201048) | > loader_time: 0.0046 (0.004666794009648525)  --> TIME: 2025-02-27 03:42:43 -- STEP: 16442/36938 -- GLOBAL_STEP: 1530900 | > loss_text_ce: 0.024006208404898643 (0.03023642368618978) | > loss_mel_ce: 0.0031689167954027653 (nan) | > loss: 0.00010783780453493819 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.1368374288872408) | > loader_time: 0.0045 (0.004666636567556032)  --> TIME: 2025-02-27 03:42:52 -- STEP: 16492/36938 -- GLOBAL_STEP: 1530950 | > loss_text_ce: 0.038359370082616806 (0.03023677557184361) | > loss_mel_ce: 2.3065853118896484 (nan) | > loss: 0.009305336512625217 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13683621733910914) | > loader_time: 0.0047 (0.00466663009646794)  --> TIME: 2025-02-27 03:43:02 -- STEP: 16542/36938 -- GLOBAL_STEP: 1531000 | > loss_text_ce: 0.034596167504787445 (0.03023674339219286) | > loss_mel_ce: 2.874743938446045 (nan) | > loss: 0.011545001529157162 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.1368417920606227) | > loader_time: 0.0047 (0.00466663545427034)  --> TIME: 2025-02-27 03:43:11 -- STEP: 16592/36938 -- GLOBAL_STEP: 1531050 | > loss_text_ce: 0.02485453523695469 (0.030239950484659505) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1452 (0.13684478065891723) | > loader_time: 0.0053 (0.0046667293676874895)  --> TIME: 2025-02-27 03:43:21 -- STEP: 16642/36938 -- GLOBAL_STEP: 1531100 | > loss_text_ce: 0.0335187129676342 (0.030242548944060782) | > loss_mel_ce: 2.6896495819091797 (nan) | > loss: 0.010806224308907986 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.13684814939876003) | > loader_time: 0.0048 (0.0046669588597860155)  --> TIME: 2025-02-27 03:43:30 -- STEP: 16692/36938 -- GLOBAL_STEP: 1531150 | > loss_text_ce: 0.02734542265534401 (0.030245804766808118) | > loss_mel_ce: 2.5159108638763428 (nan) | > loss: 0.010092287324368954 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.13685766614486158) | > loader_time: 0.0046 (0.004666994522436624)  --> TIME: 2025-02-27 03:43:40 -- STEP: 16742/36938 -- GLOBAL_STEP: 1531200 | > loss_text_ce: 0.028593963012099266 (0.03024452391417998) | > loss_mel_ce: 1.9382712841033936 (nan) | > loss: 0.007805021479725838 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13685661651885697) | > loader_time: 0.0045 (0.004667070259146199)  --> TIME: 2025-02-27 03:43:49 -- STEP: 16792/36938 -- GLOBAL_STEP: 1531250 | > loss_text_ce: 0.027174195274710655 (0.030245951547512803) | > loss_mel_ce: 2.1163318157196045 (nan) | > loss: 0.008505976758897305 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.1368485595738113) | > loader_time: 0.0048 (0.0046668445399513)  --> TIME: 2025-02-27 03:43:58 -- STEP: 16842/36938 -- GLOBAL_STEP: 1531300 | > loss_text_ce: 0.02730695903301239 (0.030242494547726585) | > loss_mel_ce: 1.49563729763031 (nan) | > loss: 0.0060434299521148205 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13685277850055172) | > loader_time: 0.0046 (0.0046668978629833165)  --> TIME: 2025-02-27 03:44:08 -- STEP: 16892/36938 -- GLOBAL_STEP: 1531350 | > loss_text_ce: 0.026555966585874557 (0.03024722703323678) | > loss_mel_ce: 3.070526361465454 (nan) | > loss: 0.01229001022875309 (nan) | > current_lr: 5e-06 | > step_time: 0.1521 (0.1368642563845859) | > loader_time: 0.0047 (0.004666956473717956)  --> TIME: 2025-02-27 03:44:17 -- STEP: 16942/36938 -- GLOBAL_STEP: 1531400 | > loss_text_ce: 0.035508088767528534 (0.030247220196504488) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13686712295355283) | > loader_time: 0.0046 (0.004666966722673271)  --> TIME: 2025-02-27 03:44:27 -- STEP: 16992/36938 -- GLOBAL_STEP: 1531450 | > loss_text_ce: 0.026286162436008453 (0.030248092574930347) | > loss_mel_ce: 2.7591805458068848 (nan) | > loss: 0.011053440161049366 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13688125857020542) | > loader_time: 0.0046 (0.004667059863915786)  --> TIME: 2025-02-27 03:44:36 -- STEP: 17042/36938 -- GLOBAL_STEP: 1531500 | > loss_text_ce: 0.025043003261089325 (0.03024801826642266) | > loss_mel_ce: 2.636482000350952 (nan) | > loss: 0.010561607778072357 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13688377410141978) | > loader_time: 0.0053 (0.004667289379322684)  --> TIME: 2025-02-27 03:44:46 -- STEP: 17092/36938 -- GLOBAL_STEP: 1531550 | > loss_text_ce: 0.03006826899945736 (0.030245296529854082) | > loss_mel_ce: 2.1232380867004395 (nan) | > loss: 0.008544866926968098 (nan) | > current_lr: 5e-06 | > step_time: 0.1679 (0.13689598260482727) | > loader_time: 0.0047 (0.00466775156255911)  --> TIME: 2025-02-27 03:44:55 -- STEP: 17142/36938 -- GLOBAL_STEP: 1531600 | > loss_text_ce: 0.02900892123579979 (0.030242632172917894) | > loss_mel_ce: 2.739025354385376 (nan) | > loss: 0.010984263382852077 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.13689914012286325) | > loader_time: 0.0046 (0.004667652945384426)  --> TIME: 2025-02-27 03:45:05 -- STEP: 17192/36938 -- GLOBAL_STEP: 1531650 | > loss_text_ce: 0.021896323189139366 (0.03024249902013916) | > loss_mel_ce: 2.3175272941589355 (nan) | > loss: 0.009283428080379963 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.13691149636056096) | > loader_time: 0.0048 (0.004667735296718026)  --> TIME: 2025-02-27 03:45:14 -- STEP: 17242/36938 -- GLOBAL_STEP: 1531700 | > loss_text_ce: 0.02396358922123909 (0.030243618017283588) | > loss_mel_ce: 2.833934783935547 (nan) | > loss: 0.011340867727994919 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13690638475801994) | > loader_time: 0.0046 (0.004667733332602232)  --> TIME: 2025-02-27 03:45:23 -- STEP: 17292/36938 -- GLOBAL_STEP: 1531750 | > loss_text_ce: 0.033916063606739044 (0.03024109210992432) | > loss_mel_ce: 1.9541661739349365 (nan) | > loss: 0.0078892158344388 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13689170469985196) | > loader_time: 0.0045 (0.004667634175878811)  --> TIME: 2025-02-27 03:45:33 -- STEP: 17342/36938 -- GLOBAL_STEP: 1531800 | > loss_text_ce: 0.03070778027176857 (0.030241385178960356) | > loss_mel_ce: 2.9439425468444824 (nan) | > loss: 0.011804169043898582 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1368987193749369) | > loader_time: 0.0046 (0.004667629682540681)  --> TIME: 2025-02-27 03:45:42 -- STEP: 17392/36938 -- GLOBAL_STEP: 1531850 | > loss_text_ce: 0.02131466753780842 (0.030240031428872462) | > loss_mel_ce: 2.222163677215576 (nan) | > loss: 0.00890269223600626 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.1369048261598594) | > loader_time: 0.0047 (0.004667529831156349)  --> TIME: 2025-02-27 03:45:51 -- STEP: 17442/36938 -- GLOBAL_STEP: 1531900 | > loss_text_ce: 0.0207586120814085 (0.030239738158308424) | > loss_mel_ce: 2.337458848953247 (nan) | > loss: 0.009358006529510021 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.1368942164374174) | > loader_time: 0.0045 (0.004667420505370157)  --> TIME: 2025-02-27 03:46:01 -- STEP: 17492/36938 -- GLOBAL_STEP: 1531950 | > loss_text_ce: 0.0356721356511116 (0.030238861898430985) | > loss_mel_ce: 2.4821481704711914 (nan) | > loss: 0.009991351515054703 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13689498457842172) | > loader_time: 0.0045 (0.004667328406110833)  --> TIME: 2025-02-27 03:46:10 -- STEP: 17542/36938 -- GLOBAL_STEP: 1532000 | > loss_text_ce: 0.027656737715005875 (0.03023921305214109) | > loss_mel_ce: 3.01544451713562 (nan) | > loss: 0.012075799517333508 (nan) | > current_lr: 5e-06 | > step_time: 0.1837 (0.1368812251534086) | > loader_time: 0.0048 (0.004667257341144167)  --> TIME: 2025-02-27 03:46:19 -- STEP: 17592/36938 -- GLOBAL_STEP: 1532050 | > loss_text_ce: 0.023165790364146233 (0.030239828933983727) | > loss_mel_ce: 0.0587158203125 (nan) | > loss: 0.00032492706668563187 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13687562099528833) | > loader_time: 0.0046 (0.004667209014940292)  --> TIME: 2025-02-27 03:46:28 -- STEP: 17642/36938 -- GLOBAL_STEP: 1532100 | > loss_text_ce: 0.027430666610598564 (0.03024256111103348) | > loss_mel_ce: 2.834183931350708 (nan) | > loss: 0.01135561428964138 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.1368732793627654) | > loader_time: 0.0045 (0.004667125933700897)  --> TIME: 2025-02-27 03:46:38 -- STEP: 17692/36938 -- GLOBAL_STEP: 1532150 | > loss_text_ce: 0.026210738345980644 (0.030244337397728825) | > loss_mel_ce: 2.576760768890381 (nan) | > loss: 0.010329253040254116 (nan) | > current_lr: 5e-06 | > step_time: 0.1885 (0.13688353498462352) | > loader_time: 0.0049 (0.0046671401476013835)  --> TIME: 2025-02-27 03:46:47 -- STEP: 17742/36938 -- GLOBAL_STEP: 1532200 | > loss_text_ce: 0.03014412149786949 (0.030246572642764667) | > loss_mel_ce: 2.494678497314453 (nan) | > loss: 0.01001913845539093 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.136881865840689) | > loader_time: 0.0047 (0.004666948262041479)  --> TIME: 2025-02-27 03:46:56 -- STEP: 17792/36938 -- GLOBAL_STEP: 1532250 | > loss_text_ce: 0.02866954915225506 (0.03024323752857436) | > loss_mel_ce: 2.6237194538116455 (nan) | > loss: 0.0105253541842103 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13687687629847253) | > loader_time: 0.0046 (0.004666752697859746)  --> TIME: 2025-02-27 03:47:05 -- STEP: 17842/36938 -- GLOBAL_STEP: 1532300 | > loss_text_ce: 0.03422296792268753 (0.03024464593513325) | > loss_mel_ce: 2.7684433460235596 (nan) | > loss: 0.011121693067252636 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.1368700643249086) | > loader_time: 0.0047 (0.004666610384666262)  --> TIME: 2025-02-27 03:47:15 -- STEP: 17892/36938 -- GLOBAL_STEP: 1532350 | > loss_text_ce: 0.028688699007034302 (0.03024463779716709) | > loss_mel_ce: 2.450086832046509 (nan) | > loss: 0.009836411103606224 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1368685826316077) | > loader_time: 0.0045 (0.004666397922289023)  --> TIME: 2025-02-27 03:47:24 -- STEP: 17942/36938 -- GLOBAL_STEP: 1532400 | > loss_text_ce: 0.027788694947957993 (0.030245296728977182) | > loss_mel_ce: 2.9817655086517334 (nan) | > loss: 0.011942676268517971 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1368681593173037) | > loader_time: 0.0045 (0.00466620851660663)  --> TIME: 2025-02-27 03:47:33 -- STEP: 17992/36938 -- GLOBAL_STEP: 1532450 | > loss_text_ce: 0.037136949598789215 (0.03024642496820534) | > loss_mel_ce: 0.013203938491642475 (nan) | > loss: 0.000199765432626009 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13686366688680462) | > loader_time: 0.0047 (0.004666194657634879)  --> TIME: 2025-02-27 03:47:43 -- STEP: 18042/36938 -- GLOBAL_STEP: 1532500 | > loss_text_ce: 0.02997763641178608 (0.03024313158147421) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13686634887415844) | > loader_time: 0.0046 (0.0046661905221671695)  --> TIME: 2025-02-27 03:47:52 -- STEP: 18092/36938 -- GLOBAL_STEP: 1532550 | > loss_text_ce: 0.02826676517724991 (0.030244688153142962) | > loss_mel_ce: 2.6520018577575684 (nan) | > loss: 0.010635986924171448 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.13685940000142768) | > loader_time: 0.0046 (0.004666011667662753)  --> TIME: 2025-02-27 03:48:01 -- STEP: 18142/36938 -- GLOBAL_STEP: 1532600 | > loss_text_ce: 0.042557552456855774 (0.03024280540662167) | > loss_mel_ce: 2.6642117500305176 (nan) | > loss: 0.010741148144006729 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13687013348386115) | > loader_time: 0.0046 (0.004665896537981788)  --> TIME: 2025-02-27 03:48:11 -- STEP: 18192/36938 -- GLOBAL_STEP: 1532650 | > loss_text_ce: 0.037929706275463104 (0.030243858567198623) | > loss_mel_ce: 2.6622421741485596 (nan) | > loss: 0.010714968666434288 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13687699396251737) | > loader_time: 0.0047 (0.004665965940098869)  --> TIME: 2025-02-27 03:48:20 -- STEP: 18242/36938 -- GLOBAL_STEP: 1532700 | > loss_text_ce: 0.026881586760282516 (0.030247198554147373) | > loss_mel_ce: 2.7614758014678955 (nan) | > loss: 0.011064911261200905 (nan) | > current_lr: 5e-06 | > step_time: 0.1923 (0.13687653424459925) | > loader_time: 0.0049 (0.0046659125242431)  --> TIME: 2025-02-27 03:48:29 -- STEP: 18292/36938 -- GLOBAL_STEP: 1532750 | > loss_text_ce: 0.03264758735895157 (0.030245682817629026) | > loss_mel_ce: 1.9590284824371338 (nan) | > loss: 0.007903477177023888 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13687470793176668) | > loader_time: 0.0077 (0.004666003230987698)  --> TIME: 2025-02-27 03:48:39 -- STEP: 18342/36938 -- GLOBAL_STEP: 1532800 | > loss_text_ce: 0.028278449550271034 (0.030242659244777773) | > loss_mel_ce: 1.9006143808364868 (nan) | > loss: 0.007654337212443352 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.1368802789642983) | > loader_time: 0.0048 (0.004665904184968903)  --> TIME: 2025-02-27 03:48:48 -- STEP: 18392/36938 -- GLOBAL_STEP: 1532850 | > loss_text_ce: 0.025922736153006554 (0.03024415726199822) | > loss_mel_ce: 2.4611024856567383 (nan) | > loss: 0.009869148023426533 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.13688310248377877) | > loader_time: 0.0047 (0.004665826353730205)  --> TIME: 2025-02-27 03:48:58 -- STEP: 18442/36938 -- GLOBAL_STEP: 1532900 | > loss_text_ce: 0.031263355165719986 (0.030243795278235005) | > loss_mel_ce: 2.5676686763763428 (nan) | > loss: 0.010313223116099834 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.13688629186278856) | > loader_time: 0.0049 (0.004665891385468863)  --> TIME: 2025-02-27 03:49:07 -- STEP: 18492/36938 -- GLOBAL_STEP: 1532950 | > loss_text_ce: 0.037135038524866104 (0.030246052458141395) | > loss_mel_ce: 2.6939048767089844 (nan) | > loss: 0.010837460868060589 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.1368858531236705) | > loader_time: 0.0045 (0.004666052531453414)  --> TIME: 2025-02-27 03:49:16 -- STEP: 18542/36938 -- GLOBAL_STEP: 1533000 | > loss_text_ce: 0.04021550714969635 (0.030245833510404205) | > loss_mel_ce: 2.211239814758301 (nan) | > loss: 0.008934346958994865 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.1368839872936492) | > loader_time: 0.0045 (0.004666049739404384)  --> TIME: 2025-02-27 03:49:26 -- STEP: 18592/36938 -- GLOBAL_STEP: 1533050 | > loss_text_ce: 0.024476567283272743 (0.030247407243526977) | > loss_mel_ce: 2.3301849365234375 (nan) | > loss: 0.009343895129859447 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13688159699386712) | > loader_time: 0.0048 (0.004665964275034294)  --> TIME: 2025-02-27 03:49:35 -- STEP: 18642/36938 -- GLOBAL_STEP: 1533100 | > loss_text_ce: 0.029868343845009804 (0.030247302815336503) | > loss_mel_ce: 3.0145132541656494 (nan) | > loss: 0.012080879881978035 (nan) | > current_lr: 5e-06 | > step_time: 0.1493 (0.13688461617612155) | > loader_time: 0.0046 (0.004666029607619164)  --> TIME: 2025-02-27 03:49:44 -- STEP: 18692/36938 -- GLOBAL_STEP: 1533150 | > loss_text_ce: 0.0389600470662117 (0.030247006634987302) | > loss_mel_ce: 1.7507532835006714 (nan) | > loss: 0.007102037779986858 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1368855436685487) | > loader_time: 0.0045 (0.004665906835413595)  --> TIME: 2025-02-27 03:49:54 -- STEP: 18742/36938 -- GLOBAL_STEP: 1533200 | > loss_text_ce: 0.034061551094055176 (0.030248805622646455) | > loss_mel_ce: 2.218567371368408 (nan) | > loss: 0.008939003571867943 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13689201668886003) | > loader_time: 0.0045 (0.004666040768275827)  --> TIME: 2025-02-27 03:50:03 -- STEP: 18792/36938 -- GLOBAL_STEP: 1533250 | > loss_text_ce: 0.03536222502589226 (0.030248153694011695) | > loss_mel_ce: 3.05863618850708 (nan) | > loss: 0.012277772650122643 (nan) | > current_lr: 5e-06 | > step_time: 0.1794 (0.13689616850306538) | > loader_time: 0.0049 (0.0046659640146347395)  --> TIME: 2025-02-27 03:50:13 -- STEP: 18842/36938 -- GLOBAL_STEP: 1533300 | > loss_text_ce: 0.027240531519055367 (0.030245534264402595) | > loss_mel_ce: 2.839351177215576 (nan) | > loss: 0.01137536484748125 (nan) | > current_lr: 5e-06 | > step_time: 0.1591 (0.1368952129768122) | > loader_time: 0.0047 (0.004665936916044814)  --> TIME: 2025-02-27 03:50:21 -- STEP: 18892/36938 -- GLOBAL_STEP: 1533350 | > loss_text_ce: 0.028617365285754204 (0.030244942919110406) | > loss_mel_ce: 1.5895785093307495 (nan) | > loss: 0.00642141280695796 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.1368751094903129) | > loader_time: 0.0044 (0.00466568577776718)  --> TIME: 2025-02-27 03:50:31 -- STEP: 18942/36938 -- GLOBAL_STEP: 1533400 | > loss_text_ce: 0.038481585681438446 (0.03024491210744532) | > loss_mel_ce: 2.343578577041626 (nan) | > loss: 0.009452619589865208 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13687109499115666) | > loader_time: 0.0045 (0.004665781056763907)  --> TIME: 2025-02-27 03:50:40 -- STEP: 18992/36938 -- GLOBAL_STEP: 1533450 | > loss_text_ce: 0.02732713706791401 (0.030244035055563133) | > loss_mel_ce: 2.7450320720672607 (nan) | > loss: 0.011001425795257092 (nan) | > current_lr: 5e-06 | > step_time: 0.1859 (0.13688665245547496) | > loader_time: 0.0049 (0.004665719227521695)  --> TIME: 2025-02-27 03:50:49 -- STEP: 19042/36938 -- GLOBAL_STEP: 1533500 | > loss_text_ce: 0.03766559436917305 (0.030243994240124656) | > loss_mel_ce: 1.3805149793624878 (nan) | > loss: 0.005627701058983803 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.1368720202796067) | > loader_time: 0.0046 (0.004665479603941592)  --> TIME: 2025-02-27 03:50:59 -- STEP: 19092/36938 -- GLOBAL_STEP: 1533550 | > loss_text_ce: 0.03260590136051178 (0.03024457655346583) | > loss_mel_ce: 2.6676130294799805 (nan) | > loss: 0.010715154930949211 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13686825455846052) | > loader_time: 0.0045 (0.00466531550087521)  --> TIME: 2025-02-27 03:51:08 -- STEP: 19142/36938 -- GLOBAL_STEP: 1533600 | > loss_text_ce: 0.02843400463461876 (0.030244269983173706) | > loss_mel_ce: 2.4113616943359375 (nan) | > loss: 0.009681729599833488 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13688150665102689) | > loader_time: 0.0046 (0.004665365725796182)  --> TIME: 2025-02-27 03:51:18 -- STEP: 19192/36938 -- GLOBAL_STEP: 1533650 | > loss_text_ce: 0.0319899283349514 (0.030244554777809428) | > loss_mel_ce: 3.125648021697998 (nan) | > loss: 0.012530309148132801 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.13689317700514092) | > loader_time: 0.0047 (0.004665505617745187)  --> TIME: 2025-02-27 03:51:27 -- STEP: 19242/36938 -- GLOBAL_STEP: 1533700 | > loss_text_ce: 0.029018303379416466 (0.03024307653512553) | > loss_mel_ce: 2.766042470932007 (nan) | > loss: 0.011091511696577072 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1369053543066241) | > loader_time: 0.0046 (0.004665432024889247)  --> TIME: 2025-02-27 03:51:37 -- STEP: 19292/36938 -- GLOBAL_STEP: 1533750 | > loss_text_ce: 0.03425975143909454 (0.03024375617428619) | > loss_mel_ce: 2.21978497505188 (nan) | > loss: 0.008944622240960598 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13690548874445577) | > loader_time: 0.0045 (0.004665318413836674)  --> TIME: 2025-02-27 03:51:46 -- STEP: 19342/36938 -- GLOBAL_STEP: 1533800 | > loss_text_ce: 0.030323734506964684 (0.03024782235985029) | > loss_mel_ce: 2.545363664627075 (nan) | > loss: 0.010220982134342194 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13690160961731007) | > loader_time: 0.0046 (0.004665160953240496)  --> TIME: 2025-02-27 03:51:55 -- STEP: 19392/36938 -- GLOBAL_STEP: 1533850 | > loss_text_ce: 0.035298481583595276 (0.03025049645285217) | > loss_mel_ce: 2.660210371017456 (nan) | > loss: 0.010696465149521828 (nan) | > current_lr: 5e-06 | > step_time: 0.1833 (0.1368991699032858) | > loader_time: 0.0048 (0.0046651010884113525)  --> TIME: 2025-02-27 03:52:05 -- STEP: 19442/36938 -- GLOBAL_STEP: 1533900 | > loss_text_ce: 0.02504437230527401 (0.03024862758487061) | > loss_mel_ce: 2.9670896530151367 (nan) | > loss: 0.011873548850417137 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.13690495298497685) | > loader_time: 0.0048 (0.004664987708889237)  --> TIME: 2025-02-27 03:52:14 -- STEP: 19492/36938 -- GLOBAL_STEP: 1533950 | > loss_text_ce: 0.034130215644836426 (0.030247138839273536) | > loss_mel_ce: 1.7156906127929688 (nan) | > loss: 0.006943733897060156 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.136899772682108) | > loader_time: 0.0046 (0.0046650036242912)  --> TIME: 2025-02-27 03:52:23 -- STEP: 19542/36938 -- GLOBAL_STEP: 1534000 | > loss_text_ce: 0.027448726817965508 (0.030243453705105183) | > loss_mel_ce: 1.164161205291748 (nan) | > loss: 0.004728611093014479 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13689321754728095) | > loader_time: 0.0045 (0.004664841248232453)  --> TIME: 2025-02-27 03:52:32 -- STEP: 19592/36938 -- GLOBAL_STEP: 1534050 | > loss_text_ce: 0.020496830344200134 (0.030241585082451342) | > loss_mel_ce: 2.0701730251312256 (nan) | > loss: 0.008296309038996696 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1368827393051072) | > loader_time: 0.0045 (0.004664712143995763)  --> TIME: 2025-02-27 03:52:41 -- STEP: 19642/36938 -- GLOBAL_STEP: 1534100 | > loss_text_ce: 0.02583285979926586 (0.03024211275391187) | > loss_mel_ce: 2.423995018005371 (nan) | > loss: 0.009721539914608002 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13688672188524958) | > loader_time: 0.0047 (0.004664744892378672)  --> TIME: 2025-02-27 03:52:51 -- STEP: 19692/36938 -- GLOBAL_STEP: 1534150 | > loss_text_ce: 0.0223294235765934 (0.030241993998861995) | > loss_mel_ce: 2.3887879848480225 (nan) | > loss: 0.00956792663782835 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.1368852240051801) | > loader_time: 0.0046 (0.004664793492525846)  --> TIME: 2025-02-27 03:53:00 -- STEP: 19742/36938 -- GLOBAL_STEP: 1534200 | > loss_text_ce: 0.027005460113286972 (0.03024133662023795) | > loss_mel_ce: 3.0680792331695557 (nan) | > loss: 0.012282082810997963 (nan) | > current_lr: 5e-06 | > step_time: 0.1442 (0.1368851497438759) | > loader_time: 0.0047 (0.004664761838235712)  --> TIME: 2025-02-27 03:53:09 -- STEP: 19792/36938 -- GLOBAL_STEP: 1534250 | > loss_text_ce: 0.03991921991109848 (0.030241344930186636) | > loss_mel_ce: 2.724240303039551 (nan) | > loss: 0.010968887247145176 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.136872323427166) | > loader_time: 0.0046 (0.004664623650602506)  --> TIME: 2025-02-27 03:53:19 -- STEP: 19842/36938 -- GLOBAL_STEP: 1534300 | > loss_text_ce: 0.031153559684753418 (0.030240818353339992) | > loss_mel_ce: 1.7969822883605957 (nan) | > loss: 0.0072545078583061695 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13687824306559196) | > loader_time: 0.0045 (0.00466452014024503)  --> TIME: 2025-02-27 03:53:28 -- STEP: 19892/36938 -- GLOBAL_STEP: 1534350 | > loss_text_ce: 0.02958148717880249 (0.03024134558731144) | > loss_mel_ce: 2.6055188179016113 (nan) | > loss: 0.010456748306751251 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.13689068153353748) | > loader_time: 0.0046 (0.00466440742988595)  --> TIME: 2025-02-27 03:53:38 -- STEP: 19942/36938 -- GLOBAL_STEP: 1534400 | > loss_text_ce: 0.02512623928487301 (0.03023798850154547) | > loss_mel_ce: 3.0055339336395264 (nan) | > loss: 0.012026430107653141 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.13689421725098594) | > loader_time: 0.0045 (0.0046643075989859185)  --> TIME: 2025-02-27 03:53:47 -- STEP: 19992/36938 -- GLOBAL_STEP: 1534450 | > loss_text_ce: 0.0260654054582119 (0.0302352177345061) | > loss_mel_ce: 1.5883667469024658 (nan) | > loss: 0.006406476721167564 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13689745988260038) | > loader_time: 0.0046 (0.004664267573942428)  --> TIME: 2025-02-27 03:53:57 -- STEP: 20042/36938 -- GLOBAL_STEP: 1534500 | > loss_text_ce: 0.02842576988041401 (0.030234702580595706) | > loss_mel_ce: 3.048884630203247 (nan) | > loss: 0.012211550027132034 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.13690956123526402) | > loader_time: 0.0047 (0.004664235968704467)  --> TIME: 2025-02-27 03:54:06 -- STEP: 20092/36938 -- GLOBAL_STEP: 1534550 | > loss_text_ce: 0.021669192239642143 (0.03023619444251155) | > loss_mel_ce: 2.6569886207580566 (nan) | > loss: 0.010629595257341862 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13691500819538696) | > loader_time: 0.0046 (0.004664203500263539)  --> TIME: 2025-02-27 03:54:16 -- STEP: 20142/36938 -- GLOBAL_STEP: 1534600 | > loss_text_ce: 0.039071377366781235 (0.03023709508491175) | > loss_mel_ce: 2.855306625366211 (nan) | > loss: 0.011485627852380276 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1369099942045314) | > loader_time: 0.0045 (0.004664037258643261)  --> TIME: 2025-02-27 03:54:25 -- STEP: 20192/36938 -- GLOBAL_STEP: 1534650 | > loss_text_ce: 0.02403278835117817 (0.030234933169757908) | > loss_mel_ce: 2.7514846324920654 (nan) | > loss: 0.011013958603143692 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13690924871175694) | > loader_time: 0.0046 (0.004664002101508261)  --> TIME: 2025-02-27 03:54:34 -- STEP: 20242/36938 -- GLOBAL_STEP: 1534700 | > loss_text_ce: 0.025637416169047356 (0.030237001654380154) | > loss_mel_ce: 2.0103759765625 (nan) | > loss: 0.008079418912529945 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13689972689801713) | > loader_time: 0.0045 (0.004663864021632953)  --> TIME: 2025-02-27 03:54:43 -- STEP: 20292/36938 -- GLOBAL_STEP: 1534750 | > loss_text_ce: 0.030224496498703957 (0.03023622544559824) | > loss_mel_ce: 1.471665382385254 (nan) | > loss: 0.005959880538284779 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13689825246020626) | > loader_time: 0.0046 (0.004663790292155221)  --> TIME: 2025-02-27 03:54:52 -- STEP: 20342/36938 -- GLOBAL_STEP: 1534800 | > loss_text_ce: 0.030606666579842567 (0.03023704790969261) | > loss_mel_ce: 2.8458516597747803 (nan) | > loss: 0.01141451857984066 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13688609563967988) | > loader_time: 0.0046 (0.004663913360845496)  --> TIME: 2025-02-27 03:55:02 -- STEP: 20392/36938 -- GLOBAL_STEP: 1534850 | > loss_text_ce: 0.035260289907455444 (0.030233226803841322) | > loss_mel_ce: 2.488145351409912 (nan) | > loss: 0.010013515129685402 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.13688494304246945) | > loader_time: 0.0047 (0.004663846968856225)  --> TIME: 2025-02-27 03:55:11 -- STEP: 20442/36938 -- GLOBAL_STEP: 1534900 | > loss_text_ce: 0.0279624592512846 (0.0302360874149034) | > loss_mel_ce: 2.533834934234619 (nan) | > loss: 0.010165863670408726 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13689055709663472) | > loader_time: 0.0046 (0.00466389681026085)  --> TIME: 2025-02-27 03:55:21 -- STEP: 20492/36938 -- GLOBAL_STEP: 1534950 | > loss_text_ce: 0.031298935413360596 (0.030237596997465473) | > loss_mel_ce: 3.1062381267547607 (nan) | > loss: 0.01245054416358471 (nan) | > current_lr: 5e-06 | > step_time: 0.1609 (0.13688814321253998) | > loader_time: 0.0047 (0.0046637567276836475)  --> TIME: 2025-02-27 03:55:30 -- STEP: 20542/36938 -- GLOBAL_STEP: 1535000 | > loss_text_ce: 0.024205071851611137 (0.03023917313666785) | > loss_mel_ce: 2.5128750801086426 (nan) | > loss: 0.010067778639495373 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.1368895747415305) | > loader_time: 0.0046 (0.004663671923524024)  --> TIME: 2025-02-27 03:55:39 -- STEP: 20592/36938 -- GLOBAL_STEP: 1535050 | > loss_text_ce: 0.034610457718372345 (0.03024260063811077) | > loss_mel_ce: 2.8386592864990234 (nan) | > loss: 0.01140186470001936 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13688633534782346) | > loader_time: 0.0056 (0.004663594211824392)  --> TIME: 2025-02-27 03:55:48 -- STEP: 20642/36938 -- GLOBAL_STEP: 1535100 | > loss_text_ce: 0.02857384644448757 (0.03023983227482837) | > loss_mel_ce: 2.7180910110473633 (nan) | > loss: 0.010899463668465614 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13687557524320837) | > loader_time: 0.0046 (0.004663438612656389)  --> TIME: 2025-02-27 03:55:58 -- STEP: 20692/36938 -- GLOBAL_STEP: 1535150 | > loss_text_ce: 0.03256857022643089 (0.030240670032196135) | > loss_mel_ce: 2.4469947814941406 (nan) | > loss: 0.009839537553489208 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13689112507718557) | > loader_time: 0.0046 (0.004663438036988885)  --> TIME: 2025-02-27 03:56:07 -- STEP: 20742/36938 -- GLOBAL_STEP: 1535200 | > loss_text_ce: 0.034623920917510986 (0.030242300556801834) | > loss_mel_ce: 2.387186288833618 (nan) | > loss: 0.009610358625650406 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1368833671320994) | > loader_time: 0.0045 (0.0046634812925627715)  --> TIME: 2025-02-27 03:56:16 -- STEP: 20792/36938 -- GLOBAL_STEP: 1535250 | > loss_text_ce: 0.029199901968240738 (0.030243474141045664) | > loss_mel_ce: 2.2370564937591553 (nan) | > loss: 0.00899308081716299 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13688171440796396) | > loader_time: 0.0046 (0.004663415221628571)  --> TIME: 2025-02-27 03:56:25 -- STEP: 20842/36938 -- GLOBAL_STEP: 1535300 | > loss_text_ce: 0.0332179069519043 (0.030242442249556127) | > loss_mel_ce: 3.0064258575439453 (nan) | > loss: 0.012062079273164272 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.1368772947402735) | > loader_time: 0.0045 (0.004663240004725178)  --> TIME: 2025-02-27 03:56:35 -- STEP: 20892/36938 -- GLOBAL_STEP: 1535350 | > loss_text_ce: 0.031156383454799652 (0.030241749159618723) | > loss_mel_ce: 2.4765844345092773 (nan) | > loss: 0.009951353073120117 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13687562645794063) | > loader_time: 0.0046 (0.004663176139880484)  --> TIME: 2025-02-27 03:56:44 -- STEP: 20942/36938 -- GLOBAL_STEP: 1535400 | > loss_text_ce: 0.029042279347777367 (0.030240173190910418) | > loss_mel_ce: 2.3838205337524414 (nan) | > loss: 0.009574852883815765 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13687663479732975) | > loader_time: 0.0045 (0.004663029983931456)  --> TIME: 2025-02-27 03:56:53 -- STEP: 20992/36938 -- GLOBAL_STEP: 1535450 | > loss_text_ce: 0.03478796035051346 (0.03024110801259934) | > loss_mel_ce: 2.5841922760009766 (nan) | > loss: 0.010392779484391212 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13687404877756096) | > loader_time: 0.0046 (0.0046629711031550385)  --> TIME: 2025-02-27 03:57:03 -- STEP: 21042/36938 -- GLOBAL_STEP: 1535500 | > loss_text_ce: 0.03519737347960472 (0.03024247058975914) | > loss_mel_ce: 0.8530362248420715 (nan) | > loss: 0.0035247367341071367 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13688150639430244) | > loader_time: 0.0045 (0.0046627805346534405)  --> TIME: 2025-02-27 03:57:12 -- STEP: 21092/36938 -- GLOBAL_STEP: 1535550 | > loss_text_ce: 0.029771003872156143 (0.030242240662647153) | > loss_mel_ce: 2.640406608581543 (nan) | > loss: 0.01059594377875328 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13688354585556456) | > loader_time: 0.0045 (0.004662574174032032)  --> TIME: 2025-02-27 03:57:21 -- STEP: 21142/36938 -- GLOBAL_STEP: 1535600 | > loss_text_ce: 0.027671637013554573 (0.030241491851609105) | > loss_mel_ce: 0.06396484375 (nan) | > loss: 0.0003636368492152542 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13688367507663224) | > loader_time: 0.0044 (0.0046624451122946175)  --> TIME: 2025-02-27 03:57:31 -- STEP: 21192/36938 -- GLOBAL_STEP: 1535650 | > loss_text_ce: 0.026131805032491684 (0.030243729651128978) | > loss_mel_ce: 1.6450140476226807 (nan) | > loss: 0.006631531286984682 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1368891056887204) | > loader_time: 0.0046 (0.004662349567002916)  --> TIME: 2025-02-27 03:57:40 -- STEP: 21242/36938 -- GLOBAL_STEP: 1535700 | > loss_text_ce: 0.036034151911735535 (0.03024452391767411) | > loss_mel_ce: 0.0206146240234375 (nan) | > loss: 0.00022479674953501672 (nan) | > current_lr: 5e-06 | > step_time: 0.1076 (0.13687486243824917) | > loader_time: 0.0044 (0.004662190259440159)  --> TIME: 2025-02-27 03:57:49 -- STEP: 21292/36938 -- GLOBAL_STEP: 1535750 | > loss_text_ce: 0.0451652929186821 (0.03024713253655225) | > loss_mel_ce: 2.588334560394287 (nan) | > loss: 0.010450396686792374 (nan) | > current_lr: 5e-06 | > step_time: 0.1121 (0.13687678560734987) | > loader_time: 0.0045 (0.004662122344379471)  --> TIME: 2025-02-27 03:57:58 -- STEP: 21342/36938 -- GLOBAL_STEP: 1535800 | > loss_text_ce: 0.033461201936006546 (0.03024816687735782) | > loss_mel_ce: 2.844398021697998 (nan) | > loss: 0.011420076712965965 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13685651165555893) | > loader_time: 0.0045 (0.004662258211020932)  --> TIME: 2025-02-27 03:58:07 -- STEP: 21392/36938 -- GLOBAL_STEP: 1535850 | > loss_text_ce: 0.03290494903922081 (0.030249429766974083) | > loss_mel_ce: 2.423853635787964 (nan) | > loss: 0.009749041870236397 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.1368498939048384) | > loader_time: 0.0054 (0.0046622474624224385)  --> TIME: 2025-02-27 03:58:17 -- STEP: 21442/36938 -- GLOBAL_STEP: 1535900 | > loss_text_ce: 0.030879801139235497 (0.030254834516603043) | > loss_mel_ce: 2.6648852825164795 (nan) | > loss: 0.010697481222450733 (nan) | > current_lr: 5e-06 | > step_time: 0.1927 (0.13685576068010746) | > loader_time: 0.0047 (0.004662429326817725)  --> TIME: 2025-02-27 03:58:26 -- STEP: 21492/36938 -- GLOBAL_STEP: 1535950 | > loss_text_ce: 0.030642613768577576 (0.030252371293803065) | > loss_mel_ce: 2.3060262203216553 (nan) | > loss: 0.009272495284676552 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13685289229979336) | > loader_time: 0.0045 (0.004662333809639432)  --> TIME: 2025-02-27 03:58:35 -- STEP: 21542/36938 -- GLOBAL_STEP: 1536000 | > loss_text_ce: 0.02915322221815586 (0.030253078178589528) | > loss_mel_ce: 2.6873178482055664 (nan) | > loss: 0.010779647156596184 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.1368537379909328) | > loader_time: 0.0045 (0.004662267887960867)  --> TIME: 2025-02-27 03:58:45 -- STEP: 21592/36938 -- GLOBAL_STEP: 1536050 | > loss_text_ce: 0.03088769130408764 (0.030254884786205465) | > loss_mel_ce: 3.015212059020996 (nan) | > loss: 0.012087698094546795 (nan) | > current_lr: 5e-06 | > step_time: 0.157 (0.1368555899305057) | > loader_time: 0.0048 (0.004662266933463067)  --> TIME: 2025-02-27 03:58:54 -- STEP: 21642/36938 -- GLOBAL_STEP: 1536100 | > loss_text_ce: 0.03309128060936928 (0.030253226465717136) | > loss_mel_ce: 2.6065917015075684 (nan) | > loss: 0.010474933311343193 (nan) | > current_lr: 5e-06 | > step_time: 0.1455 (0.13683762414631087) | > loader_time: 0.0046 (0.004662084002204414)  --> TIME: 2025-02-27 03:59:03 -- STEP: 21692/36938 -- GLOBAL_STEP: 1536150 | > loss_text_ce: 0.03282874822616577 (0.030249273259319455) | > loss_mel_ce: 1.5488554239273071 (nan) | > loss: 0.006276524625718594 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13683628557837646) | > loader_time: 0.0045 (0.004662124978293572)  --> TIME: 2025-02-27 03:59:13 -- STEP: 21742/36938 -- GLOBAL_STEP: 1536200 | > loss_text_ce: 0.03853493556380272 (0.03024810784605255) | > loss_mel_ce: 2.3675944805145264 (nan) | > loss: 0.009548133239150047 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13684961180844485) | > loader_time: 0.0045 (0.004662138066288732)  --> TIME: 2025-02-27 03:59:22 -- STEP: 21792/36938 -- GLOBAL_STEP: 1536250 | > loss_text_ce: 0.03111656755208969 (0.030251047986138365) | > loss_mel_ce: 3.013200521469116 (nan) | > loss: 0.012080623768270016 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.13684282295416886) | > loader_time: 0.0047 (0.0046620787984299)  --> TIME: 2025-02-27 03:59:31 -- STEP: 21842/36938 -- GLOBAL_STEP: 1536300 | > loss_text_ce: 0.02898360602557659 (0.03025604332992091) | > loss_mel_ce: 0.09539413452148438 (nan) | > loss: 0.0004935624892823398 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13683519559010987) | > loader_time: 0.0045 (0.004662256157449522)  --> TIME: 2025-02-27 03:59:40 -- STEP: 21892/36938 -- GLOBAL_STEP: 1536350 | > loss_text_ce: 0.02906913496553898 (0.03025768748575592) | > loss_mel_ce: 2.244885206222534 (nan) | > loss: 0.009023629128932953 (nan) | > current_lr: 5e-06 | > step_time: 0.1888 (0.1368202708788168) | > loader_time: 0.0048 (0.004662281304200337)  --> TIME: 2025-02-27 03:59:49 -- STEP: 21942/36938 -- GLOBAL_STEP: 1536400 | > loss_text_ce: 0.028139492496848106 (0.030257877387337668) | > loss_mel_ce: 2.780923843383789 (nan) | > loss: 0.01114707812666893 (nan) | > current_lr: 5e-06 | > step_time: 0.1667 (0.13680999643680358) | > loader_time: 0.0046 (0.0046620974837565535)  --> TIME: 2025-02-27 03:59:58 -- STEP: 21992/36938 -- GLOBAL_STEP: 1536450 | > loss_text_ce: 0.01898428238928318 (0.030259808893661064) | > loss_mel_ce: 2.3967933654785156 (nan) | > loss: 0.00958641991019249 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13680365513610487) | > loader_time: 0.0046 (0.004661917404592488)  --> TIME: 2025-02-27 04:00:07 -- STEP: 22042/36938 -- GLOBAL_STEP: 1536500 | > loss_text_ce: 0.0320231057703495 (0.03026184008260126) | > loss_mel_ce: 2.6352272033691406 (nan) | > loss: 0.010584327392280102 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13680314154538914) | > loader_time: 0.0045 (0.004661766957721822)  --> TIME: 2025-02-27 04:00:17 -- STEP: 22092/36938 -- GLOBAL_STEP: 1536550 | > loss_text_ce: 0.02953595481812954 (0.030265035703277456) | > loss_mel_ce: 2.4608373641967773 (nan) | > loss: 0.009882434271275997 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13680910625989676) | > loader_time: 0.0045 (0.004661612551259361)  --> TIME: 2025-02-27 04:00:26 -- STEP: 22142/36938 -- GLOBAL_STEP: 1536600 | > loss_text_ce: 0.025292562320828438 (0.030265658742518305) | > loss_mel_ce: 2.66751766204834 (nan) | > loss: 0.010685755871236324 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13680565556410917) | > loader_time: 0.0045 (0.004661439481805346)  --> TIME: 2025-02-27 04:00:36 -- STEP: 22192/36938 -- GLOBAL_STEP: 1536650 | > loss_text_ce: 0.02992432564496994 (0.03026241221920593) | > loss_mel_ce: 2.656445264816284 (nan) | > loss: 0.010660197585821152 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.1368178283686945) | > loader_time: 0.0045 (0.004661314603056761)  --> TIME: 2025-02-27 04:00:45 -- STEP: 22242/36938 -- GLOBAL_STEP: 1536700 | > loss_text_ce: 0.03604356572031975 (0.030264705245477475) | > loss_mel_ce: 2.613813877105713 (nan) | > loss: 0.010515308007597923 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13681917929754478) | > loader_time: 0.0045 (0.004661170229960252)  --> TIME: 2025-02-27 04:00:55 -- STEP: 22292/36938 -- GLOBAL_STEP: 1536750 | > loss_text_ce: 0.03609953448176384 (0.0302626173535107) | > loss_mel_ce: 2.329432249069214 (nan) | > loss: 0.009387031197547913 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13681926020124877) | > loader_time: 0.0046 (0.004661133756505803)  --> TIME: 2025-02-27 04:01:04 -- STEP: 22342/36938 -- GLOBAL_STEP: 1536800 | > loss_text_ce: 0.02839341014623642 (0.03026226212607122) | > loss_mel_ce: 2.4927806854248047 (nan) | > loss: 0.010004660114645958 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13681755949716576) | > loader_time: 0.0046 (0.00466098598438905)  --> TIME: 2025-02-27 04:01:13 -- STEP: 22392/36938 -- GLOBAL_STEP: 1536850 | > loss_text_ce: 0.03506501019001007 (0.030264853479574593) | > loss_mel_ce: 2.48685622215271 (nan) | > loss: 0.010007624514400959 (nan) | > current_lr: 5e-06 | > step_time: 0.1267 (0.13682300727594915) | > loader_time: 0.0046 (0.004660929418453448)  --> TIME: 2025-02-27 04:01:23 -- STEP: 22442/36938 -- GLOBAL_STEP: 1536900 | > loss_text_ce: 0.02345488779246807 (0.030260386403298808) | > loss_mel_ce: 0.0004765192570630461 (nan) | > loss: 9.496590064372867e-05 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13683145120175616) | > loader_time: 0.0045 (0.004660808639553564)  --> TIME: 2025-02-27 04:01:32 -- STEP: 22492/36938 -- GLOBAL_STEP: 1536950 | > loss_text_ce: 0.029886256903409958 (0.03026112521113804) | > loss_mel_ce: 3.1136634349823 (nan) | > loss: 0.012474404647946358 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13683734412043272) | > loader_time: 0.0046 (0.0046608361531338535)  --> TIME: 2025-02-27 04:01:42 -- STEP: 22542/36938 -- GLOBAL_STEP: 1537000 | > loss_text_ce: 0.028622478246688843 (0.03025916581817264) | > loss_mel_ce: 2.751744031906128 (nan) | > loss: 0.011033200658857822 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13684093999773897) | > loader_time: 0.0046 (0.0046607105218574125)  --> TIME: 2025-02-27 04:01:52 -- STEP: 22592/36938 -- GLOBAL_STEP: 1537050 | > loss_text_ce: 0.032879214733839035 (0.030258856087668463) | > loss_mel_ce: 1.5882335901260376 (nan) | > loss: 0.006432987749576569 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13685284411434487) | > loader_time: 0.0045 (0.0046606068802791745)  --> TIME: 2025-02-27 04:02:01 -- STEP: 22642/36938 -- GLOBAL_STEP: 1537100 | > loss_text_ce: 0.03291869908571243 (0.03025728629536914) | > loss_mel_ce: 3.998955726623535 (nan) | > loss: 0.015999503433704376 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1368586582068101) | > loader_time: 0.0048 (0.0046609706354693095)  --> TIME: 2025-02-27 04:02:10 -- STEP: 22692/36938 -- GLOBAL_STEP: 1537150 | > loss_text_ce: 0.03641359508037567 (0.030257028467821388) | > loss_mel_ce: 2.453923463821411 (nan) | > loss: 0.009882290847599506 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13686110192498505) | > loader_time: 0.0045 (0.004660936001220041)  --> TIME: 2025-02-27 04:02:20 -- STEP: 22742/36938 -- GLOBAL_STEP: 1537200 | > loss_text_ce: 0.02785661071538925 (0.03025790668171624) | > loss_mel_ce: 2.6387555599212646 (nan) | > loss: 0.010581795126199722 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13685993022942247) | > loader_time: 0.0046 (0.0046608282911485025)  --> TIME: 2025-02-27 04:02:29 -- STEP: 22792/36938 -- GLOBAL_STEP: 1537250 | > loss_text_ce: 0.02601383812725544 (0.030254270182738616) | > loss_mel_ce: 2.2456090450286865 (nan) | > loss: 0.009014377370476723 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13686186419365212) | > loader_time: 0.0047 (0.004660733742394258)  --> TIME: 2025-02-27 04:02:38 -- STEP: 22842/36938 -- GLOBAL_STEP: 1537300 | > loss_text_ce: 0.03251877427101135 (0.030252344815876606) | > loss_mel_ce: 2.8253772258758545 (nan) | > loss: 0.011340858414769173 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13686110131868595) | > loader_time: 0.0047 (0.004660751207750995)  --> TIME: 2025-02-27 04:02:48 -- STEP: 22892/36938 -- GLOBAL_STEP: 1537350 | > loss_text_ce: 0.03379372879862785 (0.030253917254323175) | > loss_mel_ce: 2.7539868354797363 (nan) | > loss: 0.0110626220703125 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.136856074206546) | > loader_time: 0.0046 (0.004660710064906166)  --> TIME: 2025-02-27 04:02:57 -- STEP: 22942/36938 -- GLOBAL_STEP: 1537400 | > loss_text_ce: 0.033347103744745255 (0.030253483957417237) | > loss_mel_ce: 2.790846824645996 (nan) | > loss: 0.011207119561731815 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.1368523013071122) | > loader_time: 0.0046 (0.004660565428492314)  --> TIME: 2025-02-27 04:03:06 -- STEP: 22992/36938 -- GLOBAL_STEP: 1537450 | > loss_text_ce: 0.024843968451023102 (0.030252713064134757) | > loss_mel_ce: 3.0445001125335693 (nan) | > loss: 0.012179937213659286 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.13685367513051433) | > loader_time: 0.0053 (0.004660666642208936)  --> TIME: 2025-02-27 04:03:16 -- STEP: 23042/36938 -- GLOBAL_STEP: 1537500 | > loss_text_ce: 0.027421699836850166 (0.03025196184305251) | > loss_mel_ce: 2.8809409141540527 (nan) | > loss: 0.011541122570633888 (nan) | > current_lr: 5e-06 | > step_time: 0.1904 (0.13685747850956373) | > loader_time: 0.0048 (0.004660721175342204)  --> TIME: 2025-02-27 04:03:25 -- STEP: 23092/36938 -- GLOBAL_STEP: 1537550 | > loss_text_ce: 0.0297954510897398 (0.030254349215092492) | > loss_mel_ce: 2.7091283798217773 (nan) | > loss: 0.010868745855987072 (nan) | > current_lr: 5e-06 | > step_time: 0.1924 (0.13687161672018341) | > loader_time: 0.007 (0.004660778136098899)  --> TIME: 2025-02-27 04:03:35 -- STEP: 23142/36938 -- GLOBAL_STEP: 1537600 | > loss_text_ce: 0.030403252691030502 (0.030254795918540245) | > loss_mel_ce: 3.3074216842651367 (nan) | > loss: 0.013245337642729282 (nan) | > current_lr: 5e-06 | > step_time: 0.1783 (0.1368919637351372) | > loader_time: 0.0049 (0.004661022498978314)  --> TIME: 2025-02-27 04:03:44 -- STEP: 23192/36938 -- GLOBAL_STEP: 1537650 | > loss_text_ce: 0.03205731138586998 (0.030254374767994027) | > loss_mel_ce: 2.957454204559326 (nan) | > loss: 0.01186314132064581 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13688588080713845) | > loader_time: 0.0046 (0.004660970108309215)  --> TIME: 2025-02-27 04:03:54 -- STEP: 23242/36938 -- GLOBAL_STEP: 1537700 | > loss_text_ce: 0.033489126712083817 (0.030257957580870173) | > loss_mel_ce: 1.5120697021484375 (nan) | > loss: 0.006133170332759619 (nan) | > current_lr: 5e-06 | > step_time: 0.1885 (0.13690022316008618) | > loader_time: 0.0048 (0.004660992560416609)  --> TIME: 2025-02-27 04:04:04 -- STEP: 23292/36938 -- GLOBAL_STEP: 1537750 | > loss_text_ce: 0.0307120643556118 (0.0302576960754646) | > loss_mel_ce: 2.881890058517456 (nan) | > loss: 0.011557945981621742 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.13689948341270444) | > loader_time: 0.0047 (0.004660938176301341)  --> TIME: 2025-02-27 04:04:13 -- STEP: 23342/36938 -- GLOBAL_STEP: 1537800 | > loss_text_ce: 0.03145008534193039 (0.030256363987705148) | > loss_mel_ce: 2.470813512802124 (nan) | > loss: 0.009929617866873741 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.13689603236409872) | > loader_time: 0.0047 (0.004660998045678067)  --> TIME: 2025-02-27 04:04:22 -- STEP: 23392/36938 -- GLOBAL_STEP: 1537850 | > loss_text_ce: 0.02898855321109295 (0.030254639699904953) | > loss_mel_ce: 0.283447265625 (nan) | > loss: 0.0012398246908560395 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13688820318182632) | > loader_time: 0.0046 (0.004660848431319772)  --> TIME: 2025-02-27 04:04:31 -- STEP: 23442/36938 -- GLOBAL_STEP: 1537900 | > loss_text_ce: 0.027929360046982765 (0.030252806107761446) | > loss_mel_ce: 0.9385330677032471 (nan) | > loss: 0.003835168667137623 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13689646833983063) | > loader_time: 0.0045 (0.004660815989622308)  --> TIME: 2025-02-27 04:04:41 -- STEP: 23492/36938 -- GLOBAL_STEP: 1537950 | > loss_text_ce: 0.022310951724648476 (0.030251965726760338) | > loss_mel_ce: 0.055446624755859375 (nan) | > loss: 0.00030856180819682777 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13690057913976583) | > loader_time: 0.0046 (0.004660730160583779)  --> TIME: 2025-02-27 04:04:50 -- STEP: 23542/36938 -- GLOBAL_STEP: 1538000 | > loss_text_ce: 0.04314170777797699 (0.030253924418755233) | > loss_mel_ce: 1.3881735801696777 (nan) | > loss: 0.005679822992533445 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13690187088260988) | > loader_time: 0.0046 (0.004660615671076983)  --> TIME: 2025-02-27 04:05:00 -- STEP: 23592/36938 -- GLOBAL_STEP: 1538050 | > loss_text_ce: 0.035484135150909424 (0.0302547864091397) | > loss_mel_ce: 2.5999910831451416 (nan) | > loss: 0.010458235628902912 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.13691453657420344) | > loader_time: 0.0046 (0.0046605829992064675)  --> TIME: 2025-02-27 04:05:09 -- STEP: 23642/36938 -- GLOBAL_STEP: 1538100 | > loss_text_ce: 0.038638532161712646 (0.030255506912018823) | > loss_mel_ce: 3.3353240489959717 (nan) | > loss: 0.013388741761446 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13691732594228948) | > loader_time: 0.0047 (0.004660534622725488)  --> TIME: 2025-02-27 04:05:19 -- STEP: 23692/36938 -- GLOBAL_STEP: 1538150 | > loss_text_ce: 0.035442788153886795 (0.030255621685501825) | > loss_mel_ce: 1.9195150136947632 (nan) | > loss: 0.007757769897580147 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13691331098233897) | > loader_time: 0.0045 (0.004660370491574621)  --> TIME: 2025-02-27 04:05:28 -- STEP: 23742/36938 -- GLOBAL_STEP: 1538200 | > loss_text_ce: 0.02890910767018795 (0.03025584659128381) | > loss_mel_ce: 2.5586509704589844 (nan) | > loss: 0.010268096812069416 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13689997086538278) | > loader_time: 0.0048 (0.00466019502134775)  --> TIME: 2025-02-27 04:05:37 -- STEP: 23792/36938 -- GLOBAL_STEP: 1538250 | > loss_text_ce: 0.035708725452423096 (0.030258227370056366) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13690942850542676) | > loader_time: 0.0046 (0.004660226640159356)  --> TIME: 2025-02-27 04:05:46 -- STEP: 23842/36938 -- GLOBAL_STEP: 1538300 | > loss_text_ce: 0.031389810144901276 (0.03025375482556417) | > loss_mel_ce: 2.7549920082092285 (nan) | > loss: 0.011057071387767792 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.1369008834345436) | > loader_time: 0.0046 (0.004660006767850861)  --> TIME: 2025-02-27 04:05:56 -- STEP: 23892/36938 -- GLOBAL_STEP: 1538350 | > loss_text_ce: 0.03074340708553791 (0.0302532906202592) | > loss_mel_ce: 1.7136584520339966 (nan) | > loss: 0.0069222296588122845 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.1369036912379229) | > loader_time: 0.0046 (0.004659924937436154)  --> TIME: 2025-02-27 04:06:05 -- STEP: 23942/36938 -- GLOBAL_STEP: 1538400 | > loss_text_ce: 0.03316929563879967 (0.030252669664739634) | > loss_mel_ce: 2.2303831577301025 (nan) | > loss: 0.008982351049780846 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13691111761529284) | > loader_time: 0.0045 (0.004660030224301164)  --> TIME: 2025-02-27 04:06:14 -- STEP: 23992/36938 -- GLOBAL_STEP: 1538450 | > loss_text_ce: 0.03028201498091221 (0.03025069057098853) | > loss_mel_ce: 2.498548746109009 (nan) | > loss: 0.010035043582320213 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1369016397273638) | > loader_time: 0.0045 (0.004660031842406329)  --> TIME: 2025-02-27 04:06:24 -- STEP: 24042/36938 -- GLOBAL_STEP: 1538500 | > loss_text_ce: 0.022738344967365265 (0.030249058262010953) | > loss_mel_ce: 1.0192172527313232 (nan) | > loss: 0.004134744871407747 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1369028909959476) | > loader_time: 0.0046 (0.004660063878379814)  --> TIME: 2025-02-27 04:06:33 -- STEP: 24092/36938 -- GLOBAL_STEP: 1538550 | > loss_text_ce: 0.028824765235185623 (0.03024983963694934) | > loss_mel_ce: 2.5598092079162598 (nan) | > loss: 0.010272357612848282 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.1368989971672487) | > loader_time: 0.0046 (0.004660149002993556)  --> TIME: 2025-02-27 04:06:42 -- STEP: 24142/36938 -- GLOBAL_STEP: 1538600 | > loss_text_ce: 0.025798490270972252 (0.030250356500556654) | > loss_mel_ce: 2.166454315185547 (nan) | > loss: 0.008699417114257812 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13689648633014762) | > loader_time: 0.0045 (0.0046600928984841)  --> TIME: 2025-02-27 04:06:52 -- STEP: 24192/36938 -- GLOBAL_STEP: 1538650 | > loss_text_ce: 0.035626862198114395 (0.030251282658807544) | > loss_mel_ce: 3.029452085494995 (nan) | > loss: 0.0121630122885108 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13689626689271084) | > loader_time: 0.0047 (0.004660119208945792)  --> TIME: 2025-02-27 04:07:01 -- STEP: 24242/36938 -- GLOBAL_STEP: 1538700 | > loss_text_ce: 0.03625030443072319 (0.030252580337258517) | > loss_mel_ce: 2.7802212238311768 (nan) | > loss: 0.011176475323736668 (nan) | > current_lr: 5e-06 | > step_time: 0.1874 (0.1368981514342132) | > loader_time: 0.0048 (0.004660092990652734)  --> TIME: 2025-02-27 04:07:10 -- STEP: 24292/36938 -- GLOBAL_STEP: 1538750 | > loss_text_ce: 0.031068965792655945 (0.030253963770338576) | > loss_mel_ce: 2.744868755340576 (nan) | > loss: 0.011015626601874828 (nan) | > current_lr: 5e-06 | > step_time: 0.1679 (0.13689202982021495) | > loader_time: 0.0048 (0.004659962157490182)  --> TIME: 2025-02-27 04:07:19 -- STEP: 24342/36938 -- GLOBAL_STEP: 1538800 | > loss_text_ce: 0.03548415005207062 (0.0302526816322969) | > loss_mel_ce: 2.5490894317626953 (nan) | > loss: 0.01025624480098486 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13689386732977407) | > loader_time: 0.0045 (0.0046598429492207254)  --> TIME: 2025-02-27 04:07:29 -- STEP: 24392/36938 -- GLOBAL_STEP: 1538850 | > loss_text_ce: 0.03195788711309433 (0.030252023172772063) | > loss_mel_ce: 2.5915985107421875 (nan) | > loss: 0.010410938411951065 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.13690520504843348) | > loader_time: 0.0047 (0.004659875039782195)  --> TIME: 2025-02-27 04:07:39 -- STEP: 24442/36938 -- GLOBAL_STEP: 1538900 | > loss_text_ce: 0.030198073014616966 (0.030254221829021734) | > loss_mel_ce: 3.081559896469116 (nan) | > loss: 0.01234824676066637 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.1369159704690176) | > loader_time: 0.0046 (0.0046597985684427795)  --> TIME: 2025-02-27 04:07:48 -- STEP: 24492/36938 -- GLOBAL_STEP: 1538950 | > loss_text_ce: 0.025878965854644775 (0.030254860400903975) | > loss_mel_ce: 3.285515785217285 (nan) | > loss: 0.013140455819666386 (nan) | > current_lr: 5e-06 | > step_time: 0.1714 (0.1369061353801232) | > loader_time: 0.0049 (0.004659822967226511)  --> TIME: 2025-02-27 04:07:57 -- STEP: 24542/36938 -- GLOBAL_STEP: 1539000 | > loss_text_ce: 0.02970813773572445 (0.03025447879060293) | > loss_mel_ce: 2.152658462524414 (nan) | > loss: 0.008660185150802135 (nan) | > current_lr: 5e-06 | > step_time: 0.177 (0.13690716221050747) | > loader_time: 0.0049 (0.004659699962343328)  --> TIME: 2025-02-27 04:08:07 -- STEP: 24592/36938 -- GLOBAL_STEP: 1539050 | > loss_text_ce: 0.03903801739215851 (0.030254008118986514) | > loss_mel_ce: 2.670626163482666 (nan) | > loss: 0.010752636007964611 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13692099747909184) | > loader_time: 0.0046 (0.004659710996090465)  --> TIME: 2025-02-27 04:08:16 -- STEP: 24642/36938 -- GLOBAL_STEP: 1539100 | > loss_text_ce: 0.027525154873728752 (0.030255297948254458) | > loss_mel_ce: 0.011853617615997791 (nan) | > loss: 0.00015626498498022556 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13690723617997497) | > loader_time: 0.0045 (0.004659643392653419)  --> TIME: 2025-02-27 04:08:25 -- STEP: 24692/36938 -- GLOBAL_STEP: 1539150 | > loss_text_ce: 0.027021605521440506 (0.030254428806992625) | > loss_mel_ce: 2.577589988708496 (nan) | > loss: 0.010335761122405529 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.13690680684484194) | > loader_time: 0.0048 (0.004659912023477873)  --> TIME: 2025-02-27 04:08:35 -- STEP: 24742/36938 -- GLOBAL_STEP: 1539200 | > loss_text_ce: 0.026380004361271858 (0.03025358561268685) | > loss_mel_ce: 2.7293026447296143 (nan) | > loss: 0.010935249738395214 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.13690859475890227) | > loader_time: 0.0047 (0.004659840268736436)  --> TIME: 2025-02-27 04:08:44 -- STEP: 24792/36938 -- GLOBAL_STEP: 1539250 | > loss_text_ce: 0.03628888353705406 (0.03025392931865357) | > loss_mel_ce: 2.759611129760742 (nan) | > loss: 0.011094842106103897 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13690213626836747) | > loader_time: 0.0045 (0.00465967236660725)  --> TIME: 2025-02-27 04:08:53 -- STEP: 24842/36938 -- GLOBAL_STEP: 1539300 | > loss_text_ce: 0.02448519878089428 (0.030253615663948113) | > loss_mel_ce: 2.7674336433410645 (nan) | > loss: 0.01107904314994812 (nan) | > current_lr: 5e-06 | > step_time: 0.1588 (0.13689313966787897) | > loader_time: 0.0047 (0.0046595940026729265)  --> TIME: 2025-02-27 04:09:02 -- STEP: 24892/36938 -- GLOBAL_STEP: 1539350 | > loss_text_ce: 0.02878206968307495 (0.030252226187863128) | > loss_mel_ce: 2.357745409011841 (nan) | > loss: 0.009470348246395588 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.1368918683048855) | > loader_time: 0.0046 (0.0046595790159006745)  --> TIME: 2025-02-27 04:09:12 -- STEP: 24942/36938 -- GLOBAL_STEP: 1539400 | > loss_text_ce: 0.03884690999984741 (0.030251309224042584) | > loss_mel_ce: 3.026148557662964 (nan) | > loss: 0.012162681668996811 (nan) | > current_lr: 5e-06 | > step_time: 0.161 (0.13689421693589232) | > loader_time: 0.0047 (0.004659661762258022)  --> TIME: 2025-02-27 04:09:21 -- STEP: 24992/36938 -- GLOBAL_STEP: 1539450 | > loss_text_ce: 0.03309585154056549 (0.030251988046731786) | > loss_mel_ce: 1.1293714046478271 (nan) | > loss: 0.0046129655092954636 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13690449908929145) | > loader_time: 0.0045 (0.0046596124997212)  --> TIME: 2025-02-27 04:09:30 -- STEP: 25042/36938 -- GLOBAL_STEP: 1539500 | > loss_text_ce: 0.02695697732269764 (0.030253753019444742) | > loss_mel_ce: 0.177459716796875 (nan) | > loss: 0.0008111774222925305 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.1369007445878386) | > loader_time: 0.0046 (0.004659514516299236)  --> TIME: 2025-02-27 04:09:39 -- STEP: 25092/36938 -- GLOBAL_STEP: 1539550 | > loss_text_ce: 0.03762451186776161 (0.03025541186122848) | > loss_mel_ce: 2.303081750869751 (nan) | > loss: 0.009288517758250237 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.13689431391814472) | > loader_time: 0.0067 (0.004659555478280697)  --> TIME: 2025-02-27 04:09:49 -- STEP: 25142/36938 -- GLOBAL_STEP: 1539600 | > loss_text_ce: 0.021071137860417366 (0.03025826835010559) | > loss_mel_ce: 2.1267333030700684 (nan) | > loss: 0.008523033931851387 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13688758787776265) | > loader_time: 0.0045 (0.004659594949736379)  --> TIME: 2025-02-27 04:09:58 -- STEP: 25192/36938 -- GLOBAL_STEP: 1539650 | > loss_text_ce: 0.029160918667912483 (0.030260053825668647) | > loss_mel_ce: 2.9431474208831787 (nan) | > loss: 0.011794875375926495 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13688695195100636) | > loader_time: 0.0046 (0.004659524017684055)  --> TIME: 2025-02-27 04:10:07 -- STEP: 25242/36938 -- GLOBAL_STEP: 1539700 | > loss_text_ce: 0.02794663794338703 (0.03026250669181542) | > loss_mel_ce: 1.7597235441207886 (nan) | > loss: 0.007093929685652256 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13688797133158465) | > loader_time: 0.0045 (0.004659500281507163)  --> TIME: 2025-02-27 04:10:17 -- STEP: 25292/36938 -- GLOBAL_STEP: 1539750 | > loss_text_ce: 0.024664046242833138 (0.03026006503420465) | > loss_mel_ce: 1.705780029296875 (nan) | > loss: 0.006866842042654753 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.1368876914613789) | > loader_time: 0.0044 (0.004659419221514536)  --> TIME: 2025-02-27 04:10:26 -- STEP: 25342/36938 -- GLOBAL_STEP: 1539800 | > loss_text_ce: 0.036964498460292816 (0.03025891472487451) | > loss_mel_ce: 2.820624828338623 (nan) | > loss: 0.011339640244841576 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13688836010410727) | > loader_time: 0.0046 (0.004659487457717924)  --> TIME: 2025-02-27 04:10:36 -- STEP: 25392/36938 -- GLOBAL_STEP: 1539850 | > loss_text_ce: 0.032005321234464645 (0.030257733512797303) | > loss_mel_ce: 2.1459178924560547 (nan) | > loss: 0.008642553351819515 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13689747784185874) | > loader_time: 0.0047 (0.004659479924095577)  --> TIME: 2025-02-27 04:10:45 -- STEP: 25442/36938 -- GLOBAL_STEP: 1539900 | > loss_text_ce: 0.02299732342362404 (0.030257283017235074) | > loss_mel_ce: 2.6631662845611572 (nan) | > loss: 0.01065937988460064 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13689878086991) | > loader_time: 0.0045 (0.004659416099996929)  --> TIME: 2025-02-27 04:10:54 -- STEP: 25492/36938 -- GLOBAL_STEP: 1539950 | > loss_text_ce: 0.02833259105682373 (0.03025956166732856) | > loss_mel_ce: 0.8128280639648438 (nan) | > loss: 0.0033379392698407173 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13688933535084669) | > loader_time: 0.0045 (0.004659262964980988)  --> TIME: 2025-02-27 04:11:03 -- STEP: 25542/36938 -- GLOBAL_STEP: 1540000 | > loss_text_ce: 0.026536904275417328 (0.030258723904098908) | > loss_mel_ce: 2.6640679836273193 (nan) | > loss: 0.010677004233002663 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.13688684065221995) | > loader_time: 0.0046 (0.004659128762217457) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1540000.pth  --> TIME: 2025-02-27 04:11:19 -- STEP: 25592/36938 -- GLOBAL_STEP: 1540050 | > loss_text_ce: 0.024801841005682945 (0.030258585146139088) | > loss_mel_ce: 2.9348323345184326 (nan) | > loss: 0.01174458023160696 (nan) | > current_lr: 5e-06 | > step_time: 0.1357 (0.13689137666849333) | > loader_time: 0.0047 (0.004659218950620384)  --> TIME: 2025-02-27 04:11:28 -- STEP: 25642/36938 -- GLOBAL_STEP: 1540100 | > loss_text_ce: 0.037135422229766846 (0.03025830011714517) | > loss_mel_ce: 2.1472320556640625 (nan) | > loss: 0.008668125607073307 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.13689167238641245) | > loader_time: 0.0045 (0.004659276309488291)  --> TIME: 2025-02-27 04:11:37 -- STEP: 25692/36938 -- GLOBAL_STEP: 1540150 | > loss_text_ce: 0.02956518344581127 (0.03025694017746347) | > loss_mel_ce: 2.6752984523773193 (nan) | > loss: 0.010733585804700851 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.13688396145468654) | > loader_time: 0.0046 (0.004659313187130715)  --> TIME: 2025-02-27 04:11:47 -- STEP: 25742/36938 -- GLOBAL_STEP: 1540200 | > loss_text_ce: 0.036026474088430405 (0.030258739752127395) | > loss_mel_ce: 2.614168405532837 (nan) | > loss: 0.01051664724946022 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13687848921617254) | > loader_time: 0.0046 (0.00465921540238455)  --> TIME: 2025-02-27 04:11:56 -- STEP: 25792/36938 -- GLOBAL_STEP: 1540250 | > loss_text_ce: 0.050119951367378235 (0.03025855690710783) | > loss_mel_ce: 2.5215694904327393 (nan) | > loss: 0.010205117054283619 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13687265715180585) | > loader_time: 0.0045 (0.004659100932533611)  --> TIME: 2025-02-27 04:12:05 -- STEP: 25842/36938 -- GLOBAL_STEP: 1540300 | > loss_text_ce: 0.017732173204421997 (0.030256610185922926) | > loss_mel_ce: 2.5282304286956787 (nan) | > loss: 0.010103026404976845 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13686831720232834) | > loader_time: 0.0046 (0.004658991057348161)  --> TIME: 2025-02-27 04:12:14 -- STEP: 25892/36938 -- GLOBAL_STEP: 1540350 | > loss_text_ce: 0.03153498098254204 (0.030255233386983115) | > loss_mel_ce: 2.6354551315307617 (nan) | > loss: 0.010583294555544853 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13686182410349143) | > loader_time: 0.0046 (0.004658887407685013)  --> TIME: 2025-02-27 04:12:23 -- STEP: 25942/36938 -- GLOBAL_STEP: 1540400 | > loss_text_ce: 0.03337673842906952 (0.03025528080264934) | > loss_mel_ce: 2.8382859230041504 (nan) | > loss: 0.011395487003028393 (nan) | > current_lr: 5e-06 | > step_time: 0.1885 (0.13685890760871086) | > loader_time: 0.0048 (0.004658795526148789)  --> TIME: 2025-02-27 04:12:32 -- STEP: 25992/36938 -- GLOBAL_STEP: 1540450 | > loss_text_ce: 0.029589219018816948 (0.03025456235465509) | > loss_mel_ce: 2.505253791809082 (nan) | > loss: 0.010058901272714138 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1368531207555485) | > loader_time: 0.0045 (0.004658697549656137)  --> TIME: 2025-02-27 04:12:42 -- STEP: 26042/36938 -- GLOBAL_STEP: 1540500 | > loss_text_ce: 0.02761128544807434 (0.03025329391897328) | > loss_mel_ce: 0.2766927182674408 (nan) | > loss: 0.0012075556442141533 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13686163988853975) | > loader_time: 0.0046 (0.004658675067445446)  --> TIME: 2025-02-27 04:12:51 -- STEP: 26092/36938 -- GLOBAL_STEP: 1540550 | > loss_text_ce: 0.027638403698801994 (0.03025129615032163) | > loss_mel_ce: 2.6646077632904053 (nan) | > loss: 0.010683516971766949 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.13686366853731644) | > loader_time: 0.0047 (0.0046586255326914894)  --> TIME: 2025-02-27 04:13:00 -- STEP: 26142/36938 -- GLOBAL_STEP: 1540600 | > loss_text_ce: 0.02298770286142826 (0.03025134394291831) | > loss_mel_ce: 3.0451762676239014 (nan) | > loss: 0.01217525452375412 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.13686187317829113) | > loader_time: 0.0046 (0.004658630233343207)  --> TIME: 2025-02-27 04:13:10 -- STEP: 26192/36938 -- GLOBAL_STEP: 1540650 | > loss_text_ce: 0.026632944121956825 (0.03025368783538283) | > loss_mel_ce: 2.8582305908203125 (nan) | > loss: 0.011447872035205364 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.1368698116275847) | > loader_time: 0.0047 (0.004658626805519851)  --> TIME: 2025-02-27 04:13:19 -- STEP: 26242/36938 -- GLOBAL_STEP: 1540700 | > loss_text_ce: 0.0348832868039608 (0.03025403912286352) | > loss_mel_ce: 1.5245094299316406 (nan) | > loss: 0.006188066676259041 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13686858889951484) | > loader_time: 0.0045 (0.004658617348980207)  --> TIME: 2025-02-27 04:13:28 -- STEP: 26292/36938 -- GLOBAL_STEP: 1540750 | > loss_text_ce: 0.02594802714884281 (0.030253865586226922) | > loss_mel_ce: 2.8384594917297363 (nan) | > loss: 0.011366697028279305 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.136855316655415) | > loader_time: 0.0046 (0.004658556947529649)  --> TIME: 2025-02-27 04:13:37 -- STEP: 26342/36938 -- GLOBAL_STEP: 1540800 | > loss_text_ce: 0.033149536699056625 (0.030255722429865115) | > loss_mel_ce: 2.593039035797119 (nan) | > loss: 0.010421383194625378 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13685284616843893) | > loader_time: 0.0046 (0.004658666669657156)  --> TIME: 2025-02-27 04:13:46 -- STEP: 26392/36938 -- GLOBAL_STEP: 1540850 | > loss_text_ce: 0.030895115807652473 (0.03025411676483712) | > loss_mel_ce: 3.1543056964874268 (nan) | > loss: 0.012639687396585941 (nan) | > current_lr: 5e-06 | > step_time: 0.1628 (0.1368413928712135) | > loader_time: 0.0048 (0.004658571777288805)  --> TIME: 2025-02-27 04:13:56 -- STEP: 26442/36938 -- GLOBAL_STEP: 1540900 | > loss_text_ce: 0.024898936972022057 (0.030254809342192452) | > loss_mel_ce: 2.6346938610076904 (nan) | > loss: 0.010553940199315548 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.1368430512330696) | > loader_time: 0.0046 (0.0046585085316077825)  --> TIME: 2025-02-27 04:14:05 -- STEP: 26492/36938 -- GLOBAL_STEP: 1540950 | > loss_text_ce: 0.032407138496637344 (0.030254434623774366) | > loss_mel_ce: 3.0471205711364746 (nan) | > loss: 0.012220348231494427 (nan) | > current_lr: 5e-06 | > step_time: 0.1659 (0.1368499067944565) | > loader_time: 0.0048 (0.004658474314524171)  --> TIME: 2025-02-27 04:14:15 -- STEP: 26542/36938 -- GLOBAL_STEP: 1541000 | > loss_text_ce: 0.031146880239248276 (0.030255397098115356) | > loss_mel_ce: 2.080674886703491 (nan) | > loss: 0.008380245417356491 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13684912356154758) | > loader_time: 0.0045 (0.00465850016785299)  --> TIME: 2025-02-27 04:14:24 -- STEP: 26592/36938 -- GLOBAL_STEP: 1541050 | > loss_text_ce: 0.02916872687637806 (0.030254473554168132) | > loss_mel_ce: 3.08734393119812 (nan) | > loss: 0.01236711535602808 (nan) | > current_lr: 5e-06 | > step_time: 0.1879 (0.13684606280460335) | > loader_time: 0.0049 (0.004658481462552646)  --> TIME: 2025-02-27 04:14:33 -- STEP: 26642/36938 -- GLOBAL_STEP: 1541100 | > loss_text_ce: 0.026781026273965836 (0.030252365792729767) | > loss_mel_ce: 2.264228105545044 (nan) | > loss: 0.00909130647778511 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13685258929681748) | > loader_time: 0.0047 (0.004658574340633588)  --> TIME: 2025-02-27 04:14:42 -- STEP: 26692/36938 -- GLOBAL_STEP: 1541150 | > loss_text_ce: 0.026734329760074615 (0.03025009727749107) | > loss_mel_ce: 2.7459158897399902 (nan) | > loss: 0.011002580635249615 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13684287272024215) | > loader_time: 0.0046 (0.0046585294129554275)  --> TIME: 2025-02-27 04:14:52 -- STEP: 26742/36938 -- GLOBAL_STEP: 1541200 | > loss_text_ce: 0.023478416725993156 (0.030249223917876364) | > loss_mel_ce: 2.8819990158081055 (nan) | > loss: 0.01152967382222414 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13686250408196368) | > loader_time: 0.0046 (0.004658675792908905)  --> TIME: 2025-02-27 04:15:02 -- STEP: 26792/36938 -- GLOBAL_STEP: 1541250 | > loss_text_ce: 0.04022808000445366 (0.030249417682568905) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13685952313411287) | > loader_time: 0.0045 (0.004658713158154409)  --> TIME: 2025-02-27 04:15:11 -- STEP: 26842/36938 -- GLOBAL_STEP: 1541300 | > loss_text_ce: 0.031056052073836327 (0.030250236066273403) | > loss_mel_ce: 2.2396092414855957 (nan) | > loss: 0.009010577574372292 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13686277257748736) | > loader_time: 0.0045 (0.004658636388813494)  --> TIME: 2025-02-27 04:15:21 -- STEP: 26892/36938 -- GLOBAL_STEP: 1541350 | > loss_text_ce: 0.023259038105607033 (0.030250858934668917) | > loss_mel_ce: 2.336697816848755 (nan) | > loss: 0.009364908561110497 (nan) | > current_lr: 5e-06 | > step_time: 0.1763 (0.13687159948435632) | > loader_time: 0.0048 (0.004658587016503256)  --> TIME: 2025-02-27 04:15:30 -- STEP: 26942/36938 -- GLOBAL_STEP: 1541400 | > loss_text_ce: 0.01945219747722149 (0.030252041824453905) | > loss_mel_ce: 1.6070376634597778 (nan) | > loss: 0.006454325281083584 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13686705246223815) | > loader_time: 0.0045 (0.004658515040428998)  --> TIME: 2025-02-27 04:15:39 -- STEP: 26992/36938 -- GLOBAL_STEP: 1541450 | > loss_text_ce: 0.02341184951364994 (0.03025267473424407) | > loss_mel_ce: 2.755340576171875 (nan) | > loss: 0.011026795953512192 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13686792340480822) | > loader_time: 0.0046 (0.004658460767225316)  --> TIME: 2025-02-27 04:15:48 -- STEP: 27042/36938 -- GLOBAL_STEP: 1541500 | > loss_text_ce: 0.027547871693968773 (0.03025222875789995) | > loss_mel_ce: 2.194828510284424 (nan) | > loss: 0.008818954229354858 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13685725774032287) | > loader_time: 0.0046 (0.004658462292219995)  --> TIME: 2025-02-27 04:15:58 -- STEP: 27092/36938 -- GLOBAL_STEP: 1541550 | > loss_text_ce: 0.03594595193862915 (0.030252079820495135) | > loss_mel_ce: 2.84796142578125 (nan) | > loss: 0.011444076895713806 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13685886283120177) | > loader_time: 0.0046 (0.004658510664550261)  --> TIME: 2025-02-27 04:16:07 -- STEP: 27142/36938 -- GLOBAL_STEP: 1541600 | > loss_text_ce: 0.026851708069443703 (0.03024962572833489) | > loss_mel_ce: 0.1328125 (nan) | > loss: 0.000633588177151978 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.136855493350104) | > loader_time: 0.0045 (0.0046584597913581615)  --> TIME: 2025-02-27 04:16:16 -- STEP: 27192/36938 -- GLOBAL_STEP: 1541650 | > loss_text_ce: 0.029530281201004982 (0.030251059877000802) | > loss_mel_ce: 0.8469210267066956 (nan) | > loss: 0.003477981546893716 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13684212629252956) | > loader_time: 0.0046 (0.004658465044889136)  --> TIME: 2025-02-27 04:16:25 -- STEP: 27242/36938 -- GLOBAL_STEP: 1541700 | > loss_text_ce: 0.035785529762506485 (0.03025426864713936) | > loss_mel_ce: 1.5430452823638916 (nan) | > loss: 0.006265202071517706 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13684472544289109) | > loader_time: 0.0044 (0.004658401664435333)  --> TIME: 2025-02-27 04:16:34 -- STEP: 27292/36938 -- GLOBAL_STEP: 1541750 | > loss_text_ce: 0.025425871834158897 (0.030256375832438547) | > loss_mel_ce: 1.7416776418685913 (nan) | > loss: 0.007012316025793552 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13683586401550216) | > loader_time: 0.0045 (0.004658283174658172)  --> TIME: 2025-02-27 04:16:44 -- STEP: 27342/36938 -- GLOBAL_STEP: 1541800 | > loss_text_ce: 0.029186338186264038 (0.030255823429714383) | > loss_mel_ce: 3.039048433303833 (nan) | > loss: 0.012175534851849079 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13683969372323956) | > loader_time: 0.0046 (0.004658233490712932)  --> TIME: 2025-02-27 04:16:53 -- STEP: 27392/36938 -- GLOBAL_STEP: 1541850 | > loss_text_ce: 0.03256901726126671 (0.030255662654789697) | > loss_mel_ce: 1.7750197649002075 (nan) | > loss: 0.007172971963882446 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13684520304690537) | > loader_time: 0.0047 (0.0046581979231717354)  --> TIME: 2025-02-27 04:17:02 -- STEP: 27442/36938 -- GLOBAL_STEP: 1541900 | > loss_text_ce: 0.030592510476708412 (0.030258688106120892) | > loss_mel_ce: 2.3090553283691406 (nan) | > loss: 0.00928431749343872 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13684063068623784) | > loader_time: 0.0047 (0.004658126064765101)  --> TIME: 2025-02-27 04:17:12 -- STEP: 27492/36938 -- GLOBAL_STEP: 1541950 | > loss_text_ce: 0.0299754049628973 (0.03025596972959607) | > loss_mel_ce: 2.4219250679016113 (nan) | > loss: 0.009729764424264431 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.1368478726268811) | > loader_time: 0.0046 (0.004658139985183185)  --> TIME: 2025-02-27 04:17:22 -- STEP: 27542/36938 -- GLOBAL_STEP: 1542000 | > loss_text_ce: 0.04257582500576973 (0.03025799216970055) | > loss_mel_ce: 2.1784374713897705 (nan) | > loss: 0.008813545107841492 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1368557007499245) | > loader_time: 0.0045 (0.0046582027905110734)  --> TIME: 2025-02-27 04:17:31 -- STEP: 27592/36938 -- GLOBAL_STEP: 1542050 | > loss_text_ce: 0.026530778035521507 (0.030257835894192973) | > loss_mel_ce: 2.730940818786621 (nan) | > loss: 0.010942348279058933 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.1368512761492639) | > loader_time: 0.0046 (0.00465810261763984)  --> TIME: 2025-02-27 04:17:40 -- STEP: 27642/36938 -- GLOBAL_STEP: 1542100 | > loss_text_ce: 0.028465261682868004 (0.030257241446270827) | > loss_mel_ce: 2.5075464248657227 (nan) | > loss: 0.010063539259135723 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.1368513624296037) | > loader_time: 0.0047 (0.004658026681796104)  --> TIME: 2025-02-27 04:17:49 -- STEP: 27692/36938 -- GLOBAL_STEP: 1542150 | > loss_text_ce: 0.03315424919128418 (0.030255920000835882) | > loss_mel_ce: 2.707364797592163 (nan) | > loss: 0.010875076055526733 (nan) | > current_lr: 5e-06 | > step_time: 0.1767 (0.13685437965889055) | > loader_time: 0.0048 (0.00465800082702504)  --> TIME: 2025-02-27 04:17:59 -- STEP: 27742/36938 -- GLOBAL_STEP: 1542200 | > loss_text_ce: 0.03390597179532051 (0.030256132802625017) | > loss_mel_ce: 2.7623016834259033 (nan) | > loss: 0.011096063069999218 (nan) | > current_lr: 5e-06 | > step_time: 0.1905 (0.13685492970434585) | > loader_time: 0.0049 (0.004657977351491926)  --> TIME: 2025-02-27 04:18:08 -- STEP: 27792/36938 -- GLOBAL_STEP: 1542250 | > loss_text_ce: 0.03115789033472538 (0.03025326638130451) | > loss_mel_ce: 3.0600168704986572 (nan) | > loss: 0.012266567908227444 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.1368617020465902) | > loader_time: 0.0046 (0.00465792306861317)  --> TIME: 2025-02-27 04:18:18 -- STEP: 27842/36938 -- GLOBAL_STEP: 1542300 | > loss_text_ce: 0.027005231007933617 (0.030249905641231593) | > loss_mel_ce: 2.5789520740509033 (nan) | > loss: 0.010341101326048374 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13686016933062287) | > loader_time: 0.0045 (0.004657805158646484)  --> TIME: 2025-02-27 04:18:27 -- STEP: 27892/36938 -- GLOBAL_STEP: 1542350 | > loss_text_ce: 0.028027500957250595 (0.030250616288873742) | > loss_mel_ce: 2.0886342525482178 (nan) | > loss: 0.008399452082812786 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13685563888369365) | > loader_time: 0.0045 (0.004657678542239033)  --> TIME: 2025-02-27 04:18:36 -- STEP: 27942/36938 -- GLOBAL_STEP: 1542400 | > loss_text_ce: 0.03183342143893242 (0.030250906833067488) | > loss_mel_ce: 0.0046946206130087376 (nan) | > loss: 0.00014495255891233683 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1368562279958519) | > loader_time: 0.0045 (0.004657576321514558)  --> TIME: 2025-02-27 04:18:46 -- STEP: 27992/36938 -- GLOBAL_STEP: 1542450 | > loss_text_ce: 0.02394002676010132 (0.030249743911075842) | > loss_mel_ce: 3.1349427700042725 (nan) | > loss: 0.012535249814391136 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13685630516993752) | > loader_time: 0.0046 (0.00465749519727815)  --> TIME: 2025-02-27 04:18:55 -- STEP: 28042/36938 -- GLOBAL_STEP: 1542500 | > loss_text_ce: 0.03376651182770729 (0.030248873480626545) | > loss_mel_ce: 2.566023111343384 (nan) | > loss: 0.01031662616878748 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.13685378905208612) | > loader_time: 0.0048 (0.004657447920504668)  --> TIME: 2025-02-27 04:19:04 -- STEP: 28092/36938 -- GLOBAL_STEP: 1542550 | > loss_text_ce: 0.027571532875299454 (0.030249893672524418) | > loss_mel_ce: 2.3874425888061523 (nan) | > loss: 0.00958338938653469 (nan) | > current_lr: 5e-06 | > step_time: 0.1857 (0.13684640387498934) | > loader_time: 0.005 (0.004657429099405451)  --> TIME: 2025-02-27 04:19:14 -- STEP: 28142/36938 -- GLOBAL_STEP: 1542600 | > loss_text_ce: 0.03321726992726326 (0.03025085048298447) | > loss_mel_ce: 2.749492645263672 (nan) | > loss: 0.011042499914765358 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13685860521850565) | > loader_time: 0.0046 (0.004657437514838575)  --> TIME: 2025-02-27 04:19:24 -- STEP: 28192/36938 -- GLOBAL_STEP: 1542650 | > loss_text_ce: 0.029728345572948456 (0.030251074264260894) | > loss_mel_ce: 2.635037422180176 (nan) | > loss: 0.010574468411505222 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13687366236183182) | > loader_time: 0.0047 (0.004657463101875223)  --> TIME: 2025-02-27 04:19:33 -- STEP: 28242/36938 -- GLOBAL_STEP: 1542700 | > loss_text_ce: 0.03226175904273987 (0.030252197829763253) | > loss_mel_ce: 1.4406801462173462 (nan) | > loss: 0.005845007952302694 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13687384219744111) | > loader_time: 0.0046 (0.004657550663796437)  --> TIME: 2025-02-27 04:19:42 -- STEP: 28292/36938 -- GLOBAL_STEP: 1542750 | > loss_text_ce: 0.03849419206380844 (0.030252189244758323) | > loss_mel_ce: 3.0164246559143066 (nan) | > loss: 0.01212269440293312 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13687063776990105) | > loader_time: 0.0047 (0.004657453708966884)  --> TIME: 2025-02-27 04:19:52 -- STEP: 28342/36938 -- GLOBAL_STEP: 1542800 | > loss_text_ce: 0.030473586171865463 (0.030251483357616685) | > loss_mel_ce: 3.150465965270996 (nan) | > loss: 0.012622776441276073 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.1368806411952336) | > loader_time: 0.0046 (0.004657450555770834)  --> TIME: 2025-02-27 04:20:01 -- STEP: 28392/36938 -- GLOBAL_STEP: 1542850 | > loss_text_ce: 0.029748300090432167 (0.030251612715279704) | > loss_mel_ce: 2.715045690536499 (nan) | > loss: 0.010892040096223354 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.1368786644586286) | > loader_time: 0.0045 (0.0046573489543316055)  --> TIME: 2025-02-27 04:20:11 -- STEP: 28442/36938 -- GLOBAL_STEP: 1542900 | > loss_text_ce: 0.023375647142529488 (0.03025062726169108) | > loss_mel_ce: 2.3674283027648926 (nan) | > loss: 0.009487317875027657 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13688455613955003) | > loader_time: 0.0046 (0.004657328040797146)  --> TIME: 2025-02-27 04:20:20 -- STEP: 28492/36938 -- GLOBAL_STEP: 1542950 | > loss_text_ce: 0.027855638414621353 (0.03025302997343183) | > loss_mel_ce: 2.047969341278076 (nan) | > loss: 0.008237401023507118 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13688566604223684) | > loader_time: 0.0046 (0.004657268415387048)  --> TIME: 2025-02-27 04:20:30 -- STEP: 28542/36938 -- GLOBAL_STEP: 1543000 | > loss_text_ce: 0.028613802045583725 (0.030255547901893515) | > loss_mel_ce: 2.4261322021484375 (nan) | > loss: 0.009741055779159069 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13688621699087578) | > loader_time: 0.0045 (0.004657219908230938)  --> TIME: 2025-02-27 04:20:39 -- STEP: 28592/36938 -- GLOBAL_STEP: 1543050 | > loss_text_ce: 0.024314483627676964 (0.030256685884090286) | > loss_mel_ce: 0.010833740234375 (nan) | > loss: 0.000139477095217444 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.1368910314136877) | > loader_time: 0.0046 (0.004657195135742572)  --> TIME: 2025-02-27 04:20:48 -- STEP: 28642/36938 -- GLOBAL_STEP: 1543100 | > loss_text_ce: 0.026283513754606247 (0.03025495141365023) | > loss_mel_ce: 2.9219298362731934 (nan) | > loss: 0.011699260212481022 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.1368905479502949) | > loader_time: 0.0047 (0.004657161026874669)  --> TIME: 2025-02-27 04:20:57 -- STEP: 28692/36938 -- GLOBAL_STEP: 1543150 | > loss_text_ce: 0.030741022899746895 (0.030254930134760227) | > loss_mel_ce: 2.769327163696289 (nan) | > loss: 0.011111382395029068 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.1368850026605329) | > loader_time: 0.0045 (0.004657161505039838)  --> TIME: 2025-02-27 04:21:07 -- STEP: 28742/36938 -- GLOBAL_STEP: 1543200 | > loss_text_ce: 0.03094673343002796 (0.030253889454676712) | > loss_mel_ce: 2.6295430660247803 (nan) | > loss: 0.010557499714195728 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.1368877764626135) | > loader_time: 0.0046 (0.004657181392141705)  --> TIME: 2025-02-27 04:21:16 -- STEP: 28792/36938 -- GLOBAL_STEP: 1543250 | > loss_text_ce: 0.028392544016242027 (0.030251749576536896) | > loss_mel_ce: 2.9703927040100098 (nan) | > loss: 0.01189994253218174 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13688662637436147) | > loader_time: 0.0047 (0.0046571405869982345)  --> TIME: 2025-02-27 04:21:25 -- STEP: 28842/36938 -- GLOBAL_STEP: 1543300 | > loss_text_ce: 0.0324823372066021 (0.030250931975724493) | > loss_mel_ce: 2.2686355113983154 (nan) | > loss: 0.009131420403718948 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.1368825365704967) | > loader_time: 0.0046 (0.00465728857011175)  --> TIME: 2025-02-27 04:21:34 -- STEP: 28892/36938 -- GLOBAL_STEP: 1543350 | > loss_text_ce: 0.027932101860642433 (0.03025135354476537) | > loss_mel_ce: 1.1358534097671509 (nan) | > loss: 0.004618196748197079 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.1368734043472688) | > loader_time: 0.0047 (0.004657222213597236)  --> TIME: 2025-02-27 04:21:44 -- STEP: 28942/36938 -- GLOBAL_STEP: 1543400 | > loss_text_ce: 0.0266131442040205 (0.030251160626784367) | > loss_mel_ce: 2.9153101444244385 (nan) | > loss: 0.011674299836158752 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1368749981207928) | > loader_time: 0.0045 (0.0046572189490538575)  --> TIME: 2025-02-27 04:21:53 -- STEP: 28992/36938 -- GLOBAL_STEP: 1543450 | > loss_text_ce: 0.02293945476412773 (0.030252370023433714) | > loss_mel_ce: 2.877650499343872 (nan) | > loss: 0.011510278098285198 (nan) | > current_lr: 5e-06 | > step_time: 0.1349 (0.1368744412136931) | > loader_time: 0.0046 (0.0046571951285497995)  --> TIME: 2025-02-27 04:22:02 -- STEP: 29042/36938 -- GLOBAL_STEP: 1543500 | > loss_text_ce: 0.023337766528129578 (0.030253436457803554) | > loss_mel_ce: 2.599773645401001 (nan) | > loss: 0.010409173555672169 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13687038893673806) | > loader_time: 0.0046 (0.004657111756688804)  --> TIME: 2025-02-27 04:22:12 -- STEP: 29092/36938 -- GLOBAL_STEP: 1543550 | > loss_text_ce: 0.0307056475430727 (0.030254144248739328) | > loss_mel_ce: 2.9377009868621826 (nan) | > loss: 0.011779392138123512 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.13687929207433333) | > loader_time: 0.0046 (0.0046570591498450555)  --> TIME: 2025-02-27 04:22:21 -- STEP: 29142/36938 -- GLOBAL_STEP: 1543600 | > loss_text_ce: 0.040280263870954514 (0.030256359467379285) | > loss_mel_ce: 2.8853065967559814 (nan) | > loss: 0.011609472334384918 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13687874036944425) | > loader_time: 0.0048 (0.0046570060281123215)  --> TIME: 2025-02-27 04:22:31 -- STEP: 29192/36938 -- GLOBAL_STEP: 1543650 | > loss_text_ce: 0.015771301463246346 (0.03025536362697693) | > loss_mel_ce: 2.0273568630218506 (nan) | > loss: 0.008107651956379414 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13687974381982732) | > loader_time: 0.0045 (0.004656938534300432)  --> TIME: 2025-02-27 04:22:40 -- STEP: 29242/36938 -- GLOBAL_STEP: 1543700 | > loss_text_ce: 0.030472474172711372 (0.030254443504021882) | > loss_mel_ce: 3.003589630126953 (nan) | > loss: 0.012039929628372192 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.13687320322534294) | > loader_time: 0.0046 (0.004656831711524567)  --> TIME: 2025-02-27 04:22:49 -- STEP: 29292/36938 -- GLOBAL_STEP: 1543750 | > loss_text_ce: 0.03742886334657669 (0.030254581684172777) | > loss_mel_ce: 2.6012063026428223 (nan) | > loss: 0.010470774956047535 (nan) | > current_lr: 5e-06 | > step_time: 0.1659 (0.13686741305947456) | > loader_time: 0.0048 (0.004656762694557162)  --> TIME: 2025-02-27 04:22:58 -- STEP: 29342/36938 -- GLOBAL_STEP: 1543800 | > loss_text_ce: 0.03540129214525223 (0.03025640264826946) | > loss_mel_ce: 3.3587839603424072 (nan) | > loss: 0.013468990102410316 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.13686331355590572) | > loader_time: 0.0046 (0.004656775021600164)  --> TIME: 2025-02-27 04:23:07 -- STEP: 29392/36938 -- GLOBAL_STEP: 1543850 | > loss_text_ce: 0.023476991802453995 (0.03025607700057181) | > loss_mel_ce: 1.5227503776550293 (nan) | > loss: 0.006135823205113411 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13686795282052538) | > loader_time: 0.0046 (0.004656834313904515)  --> TIME: 2025-02-27 04:23:17 -- STEP: 29442/36938 -- GLOBAL_STEP: 1543900 | > loss_text_ce: 0.024944281205534935 (0.03025699354347649) | > loss_mel_ce: 2.6393306255340576 (nan) | > loss: 0.010572520084679127 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13686784467217245) | > loader_time: 0.0045 (0.0046567998416041055)  --> TIME: 2025-02-27 04:23:26 -- STEP: 29492/36938 -- GLOBAL_STEP: 1543950 | > loss_text_ce: 0.03223829343914986 (0.030256656318626726) | > loss_mel_ce: 1.48828125 (nan) | > loss: 0.006033807992935181 (nan) | > current_lr: 5e-06 | > step_time: 0.1072 (0.13686180771181497) | > loader_time: 0.0045 (0.0046566727445000395)  --> TIME: 2025-02-27 04:23:36 -- STEP: 29542/36938 -- GLOBAL_STEP: 1544000 | > loss_text_ce: 0.025922536849975586 (0.0302579342635879) | > loss_mel_ce: 3.84375 (nan) | > loss: 0.015355844050645828 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13687050206168583) | > loader_time: 0.0046 (0.004656943493032732)  --> TIME: 2025-02-27 04:23:45 -- STEP: 29592/36938 -- GLOBAL_STEP: 1544050 | > loss_text_ce: 0.03133134916424751 (0.03025900281001239) | > loss_mel_ce: 0.05296261981129646 (nan) | > loss: 0.0003344998985994607 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13686400396529472) | > loader_time: 0.0046 (0.004656995243560816)  --> TIME: 2025-02-27 04:23:54 -- STEP: 29642/36938 -- GLOBAL_STEP: 1544100 | > loss_text_ce: 0.02207263745367527 (0.030258436338995137) | > loss_mel_ce: 2.683237075805664 (nan) | > loss: 0.010735356248915195 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13686571335327785) | > loader_time: 0.0046 (0.00465716695376924)  --> TIME: 2025-02-27 04:24:03 -- STEP: 29692/36938 -- GLOBAL_STEP: 1544150 | > loss_text_ce: 0.03194727748632431 (0.030256878934308437) | > loss_mel_ce: 2.1637752056121826 (nan) | > loss: 0.008713185787200928 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13686133849079749) | > loader_time: 0.0046 (0.004657076421041587)  --> TIME: 2025-02-27 04:24:13 -- STEP: 29742/36938 -- GLOBAL_STEP: 1544200 | > loss_text_ce: 0.03267711400985718 (0.03025924714392567) | > loss_mel_ce: 2.650169610977173 (nan) | > loss: 0.010646218433976173 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13685979161600287) | > loader_time: 0.0046 (0.004657046667112719)  --> TIME: 2025-02-27 04:24:22 -- STEP: 29792/36938 -- GLOBAL_STEP: 1544250 | > loss_text_ce: 0.027879245579242706 (0.03026112646603681) | > loss_mel_ce: 2.4053540229797363 (nan) | > loss: 0.00965568795800209 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1368633047482523) | > loader_time: 0.0045 (0.0046569814487379384)  --> TIME: 2025-02-27 04:24:32 -- STEP: 29842/36938 -- GLOBAL_STEP: 1544300 | > loss_text_ce: 0.03335136920213699 (0.03025921212548161) | > loss_mel_ce: 2.8609135150909424 (nan) | > loss: 0.011485178954899311 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.13686200511590696) | > loader_time: 0.0046 (0.0046570301295589065)  --> TIME: 2025-02-27 04:24:41 -- STEP: 29892/36938 -- GLOBAL_STEP: 1544350 | > loss_text_ce: 0.025157414376735687 (0.030259665598848958) | > loss_mel_ce: 2.2830774784088135 (nan) | > loss: 0.009159662760794163 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.1368581252565173) | > loader_time: 0.0045 (0.004657157346711933)  --> TIME: 2025-02-27 04:24:50 -- STEP: 29942/36938 -- GLOBAL_STEP: 1544400 | > loss_text_ce: 0.02254742756485939 (0.030258260880963412) | > loss_mel_ce: 2.882671356201172 (nan) | > loss: 0.01152864657342434 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.1368520349278094) | > loader_time: 0.0046 (0.004657161800363368)  --> TIME: 2025-02-27 04:24:59 -- STEP: 29992/36938 -- GLOBAL_STEP: 1544450 | > loss_text_ce: 0.03111036866903305 (0.030258297451041326) | > loss_mel_ce: 0.09769286960363388 (nan) | > loss: 0.0005111239734105766 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13684661827522537) | > loader_time: 0.0045 (0.004657139529161814)  --> TIME: 2025-02-27 04:25:09 -- STEP: 30042/36938 -- GLOBAL_STEP: 1544500 | > loss_text_ce: 0.035856958478689194 (0.030258301779455817) | > loss_mel_ce: 2.250547170639038 (nan) | > loss: 0.00907303299754858 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13685531957784067) | > loader_time: 0.0046 (0.00465729900701934)  --> TIME: 2025-02-27 04:25:18 -- STEP: 30092/36938 -- GLOBAL_STEP: 1544550 | > loss_text_ce: 0.025474796071648598 (0.03026085456282163) | > loss_mel_ce: 2.5923047065734863 (nan) | > loss: 0.01038801483809948 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.1368592242270457) | > loader_time: 0.0045 (0.004657290153929354)  --> TIME: 2025-02-27 04:25:28 -- STEP: 30142/36938 -- GLOBAL_STEP: 1544600 | > loss_text_ce: 0.027975229546427727 (0.030259721521786857) | > loss_mel_ce: 3.134929895401001 (nan) | > loss: 0.01255121175199747 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13686659668145565) | > loader_time: 0.0045 (0.004657370363437582)  --> TIME: 2025-02-27 04:25:37 -- STEP: 30192/36938 -- GLOBAL_STEP: 1544650 | > loss_text_ce: 0.034618061035871506 (0.030260274661290942) | > loss_mel_ce: 2.64330792427063 (nan) | > loss: 0.01062669139355421 (nan) | > current_lr: 5e-06 | > step_time: 0.1567 (0.13687351599501643) | > loader_time: 0.0048 (0.0046574846265425894)  --> TIME: 2025-02-27 04:25:47 -- STEP: 30242/36938 -- GLOBAL_STEP: 1544700 | > loss_text_ce: 0.02743159979581833 (0.03026041568316782) | > loss_mel_ce: 2.9723165035247803 (nan) | > loss: 0.011903762817382812 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13687909438602178) | > loader_time: 0.0047 (0.004657549593516592)  --> TIME: 2025-02-27 04:25:56 -- STEP: 30292/36938 -- GLOBAL_STEP: 1544750 | > loss_text_ce: 0.03182952478528023 (0.030260746668621233) | > loss_mel_ce: 2.9282543659210205 (nan) | > loss: 0.011746365576982498 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.13687450989980623) | > loader_time: 0.0047 (0.0046576106546735154)  --> TIME: 2025-02-27 04:26:05 -- STEP: 30342/36938 -- GLOBAL_STEP: 1544800 | > loss_text_ce: 0.03292909264564514 (0.030260599390825527) | > loss_mel_ce: 3.003962755203247 (nan) | > loss: 0.012051159515976906 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.1368719340470821) | > loader_time: 0.0046 (0.004657689390873341)  --> TIME: 2025-02-27 04:26:15 -- STEP: 30392/36938 -- GLOBAL_STEP: 1544850 | > loss_text_ce: 0.034756310284137726 (0.030262888213583437) | > loss_mel_ce: 3.0553338527679443 (nan) | > loss: 0.012262263335287571 (nan) | > current_lr: 5e-06 | > step_time: 0.1528 (0.13687389359470556) | > loader_time: 0.0046 (0.004657620645002425)  --> TIME: 2025-02-27 04:26:24 -- STEP: 30442/36938 -- GLOBAL_STEP: 1544900 | > loss_text_ce: 0.03687465190887451 (0.03026480360828105) | > loss_mel_ce: 3.1081714630126953 (nan) | > loss: 0.012480342760682106 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13687673421580063) | > loader_time: 0.0046 (0.004657611287099896)  --> TIME: 2025-02-27 04:26:34 -- STEP: 30492/36938 -- GLOBAL_STEP: 1544950 | > loss_text_ce: 0.024456091225147247 (0.03026525856139085) | > loss_mel_ce: 1.9525951147079468 (nan) | > loss: 0.007845441810786724 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1368793982166528) | > loader_time: 0.0046 (0.004657665552249121)  --> TIME: 2025-02-27 04:26:43 -- STEP: 30542/36938 -- GLOBAL_STEP: 1545000 | > loss_text_ce: 0.02687106281518936 (0.030266401259166145) | > loss_mel_ce: 1.9864014387130737 (nan) | > loss: 0.007989177480340004 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13688272266270674) | > loader_time: 0.0045 (0.004657695050027541)  --> TIME: 2025-02-27 04:26:52 -- STEP: 30592/36938 -- GLOBAL_STEP: 1545050 | > loss_text_ce: 0.02796333096921444 (0.030265387875276013) | > loss_mel_ce: 0.706451416015625 (nan) | > loss: 0.0029143444262444973 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13687958876957096) | > loader_time: 0.0047 (0.00465765448918402)  --> TIME: 2025-02-27 04:27:02 -- STEP: 30642/36938 -- GLOBAL_STEP: 1545100 | > loss_text_ce: 0.03543383628129959 (0.030268095667028955) | > loss_mel_ce: 2.8769776821136475 (nan) | > loss: 0.01155718881636858 (nan) | > current_lr: 5e-06 | > step_time: 0.1522 (0.13688944128519795) | > loader_time: 0.0046 (0.004657683332971206)  --> TIME: 2025-02-27 04:27:11 -- STEP: 30692/36938 -- GLOBAL_STEP: 1545150 | > loss_text_ce: 0.008390368893742561 (0.030265892189966247) | > loss_mel_ce: 2.07584810256958 (nan) | > loss: 0.008270788006484509 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13688788808776536) | > loader_time: 0.0044 (0.004657715291006223)  --> TIME: 2025-02-27 04:27:21 -- STEP: 30742/36938 -- GLOBAL_STEP: 1545200 | > loss_text_ce: 0.026663120836019516 (0.030264375028706682) | > loss_mel_ce: 2.8643531799316406 (nan) | > loss: 0.011472287587821484 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.1368884381146042) | > loader_time: 0.0047 (0.00465788939586885)  --> TIME: 2025-02-27 04:27:30 -- STEP: 30792/36938 -- GLOBAL_STEP: 1545250 | > loss_text_ce: 0.03555551543831825 (0.030265035334347) | > loss_mel_ce: 3.18310284614563 (nan) | > loss: 0.012772454880177975 (nan) | > current_lr: 5e-06 | > step_time: 0.1692 (0.13688822228836203) | > loader_time: 0.0048 (0.004658090794170075)  --> TIME: 2025-02-27 04:27:39 -- STEP: 30842/36938 -- GLOBAL_STEP: 1545300 | > loss_text_ce: 0.025442209094762802 (0.030263330346746654) | > loss_mel_ce: 2.055255889892578 (nan) | > loss: 0.008256739005446434 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1368849586245397) | > loader_time: 0.0045 (0.004658025863778069)  --> TIME: 2025-02-27 04:27:48 -- STEP: 30892/36938 -- GLOBAL_STEP: 1545350 | > loss_text_ce: 0.02934466488659382 (0.03026327900611727) | > loss_mel_ce: 0.1307474821805954 (nan) | > loss: 0.0006352863274514675 (nan) | > current_lr: 5e-06 | > step_time: 0.1507 (0.13688064818553428) | > loader_time: 0.0047 (0.004657954089493077)  --> TIME: 2025-02-27 04:27:58 -- STEP: 30942/36938 -- GLOBAL_STEP: 1545400 | > loss_text_ce: 0.03515903279185295 (0.0302639445025249) | > loss_mel_ce: 3.185742139816284 (nan) | > loss: 0.012781354598701 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13687835447189364) | > loader_time: 0.0046 (0.00465809341146121)  --> TIME: 2025-02-27 04:28:06 -- STEP: 30992/36938 -- GLOBAL_STEP: 1545450 | > loss_text_ce: 0.02534927986562252 (0.03026426616621543) | > loss_mel_ce: 2.1939260959625244 (nan) | > loss: 0.008806649595499039 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13686517727209002) | > loader_time: 0.0046 (0.004658068832688926)  --> TIME: 2025-02-27 04:28:16 -- STEP: 31042/36938 -- GLOBAL_STEP: 1545500 | > loss_text_ce: 0.02449551597237587 (0.03026400022961442) | > loss_mel_ce: 2.560368537902832 (nan) | > loss: 0.010257397778332233 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13686240068854488) | > loader_time: 0.0045 (0.0046580734883346405)  --> TIME: 2025-02-27 04:28:25 -- STEP: 31092/36938 -- GLOBAL_STEP: 1545550 | > loss_text_ce: 0.02393944188952446 (0.030263469988708706) | > loss_mel_ce: 2.0267927646636963 (nan) | > loss: 0.008137826807796955 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13685708833173404) | > loader_time: 0.0046 (0.004658222052791194)  --> TIME: 2025-02-27 04:28:34 -- STEP: 31142/36938 -- GLOBAL_STEP: 1545600 | > loss_text_ce: 0.03487608954310417 (0.030263508666398493) | > loss_mel_ce: 3.032172918319702 (nan) | > loss: 0.012170829810202122 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.1368604133250249) | > loader_time: 0.0046 (0.00465826575263495)  --> TIME: 2025-02-27 04:28:44 -- STEP: 31192/36938 -- GLOBAL_STEP: 1545650 | > loss_text_ce: 0.03416365012526512 (0.03026405403346159) | > loss_mel_ce: 3.0472116470336914 (nan) | > loss: 0.012227680534124374 (nan) | > current_lr: 5e-06 | > step_time: 0.1493 (0.13685907583629406) | > loader_time: 0.0053 (0.004658194803891957)  --> TIME: 2025-02-27 04:28:53 -- STEP: 31242/36938 -- GLOBAL_STEP: 1545700 | > loss_text_ce: 0.028437426313757896 (0.030262081247718214) | > loss_mel_ce: 1.518460750579834 (nan) | > loss: 0.006138484925031662 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13686694121148765) | > loader_time: 0.0052 (0.004658284172664702)  --> TIME: 2025-02-27 04:29:03 -- STEP: 31292/36938 -- GLOBAL_STEP: 1545750 | > loss_text_ce: 0.03675699234008789 (0.030261093779093477) | > loss_mel_ce: 2.8237218856811523 (nan) | > loss: 0.011351107619702816 (nan) | > current_lr: 5e-06 | > step_time: 0.151 (0.13687167096196043) | > loader_time: 0.0046 (0.004658374429190992)  --> TIME: 2025-02-27 04:29:12 -- STEP: 31342/36938 -- GLOBAL_STEP: 1545800 | > loss_text_ce: 0.02921145409345627 (0.030260484306721538) | > loss_mel_ce: 2.6478934288024902 (nan) | > loss: 0.010623432695865631 (nan) | > current_lr: 5e-06 | > step_time: 0.1715 (0.13687060475844018) | > loader_time: 0.0047 (0.004658533378015879)  --> TIME: 2025-02-27 04:29:21 -- STEP: 31392/36938 -- GLOBAL_STEP: 1545850 | > loss_text_ce: 0.013668131083250046 (0.03026054497848507) | > loss_mel_ce: 1.6039866209030151 (nan) | > loss: 0.006419265642762184 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13686120176686073) | > loader_time: 0.0044 (0.00465847443975318)  --> TIME: 2025-02-27 04:29:31 -- STEP: 31442/36938 -- GLOBAL_STEP: 1545900 | > loss_text_ce: 0.039867471903562546 (0.030259620722847835) | > loss_mel_ce: 1.107507348060608 (nan) | > loss: 0.004553074948489666 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.1368677364825215) | > loader_time: 0.0045 (0.004658415279469712)  --> TIME: 2025-02-27 04:29:40 -- STEP: 31492/36938 -- GLOBAL_STEP: 1545950 | > loss_text_ce: 0.021382838487625122 (0.03026024182784) | > loss_mel_ce: 2.0716145038604736 (nan) | > loss: 0.008305544964969158 (nan) | > current_lr: 5e-06 | > step_time: 0.1078 (0.1368692500397783) | > loader_time: 0.0046 (0.004658485047928971)  --> TIME: 2025-02-27 04:29:50 -- STEP: 31542/36938 -- GLOBAL_STEP: 1546000 | > loss_text_ce: 0.03146985545754433 (0.030260601208257274) | > loss_mel_ce: 1.9712285995483398 (nan) | > loss: 0.007947216741740704 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.1368735269620054) | > loader_time: 0.0045 (0.004658668384850721)  --> TIME: 2025-02-27 04:29:59 -- STEP: 31592/36938 -- GLOBAL_STEP: 1546050 | > loss_text_ce: 0.02735639363527298 (0.030259266382369965) | > loss_mel_ce: 2.565913677215576 (nan) | > loss: 0.010290754958987236 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13687367663712055) | > loader_time: 0.0052 (0.004658764496607909)  --> TIME: 2025-02-27 04:30:08 -- STEP: 31642/36938 -- GLOBAL_STEP: 1546100 | > loss_text_ce: 0.02695200778543949 (0.030260254223235573) | > loss_mel_ce: 3.252669095993042 (nan) | > loss: 0.013014370575547218 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.13686982713271134) | > loader_time: 0.0046 (0.004658730222500571)  --> TIME: 2025-02-27 04:30:18 -- STEP: 31692/36938 -- GLOBAL_STEP: 1546150 | > loss_text_ce: 0.023713702335953712 (0.030258485545506763) | > loss_mel_ce: 2.5955371856689453 (nan) | > loss: 0.010393853299319744 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13687539391786796) | > loader_time: 0.0046 (0.004658679776791742)  --> TIME: 2025-02-27 04:30:27 -- STEP: 31742/36938 -- GLOBAL_STEP: 1546200 | > loss_text_ce: 0.03215944021940231 (0.030258388460011904) | > loss_mel_ce: 2.849438428878784 (nan) | > loss: 0.01143491268157959 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.1368705907425922) | > loader_time: 0.0046 (0.004658666407256032)  --> TIME: 2025-02-27 04:30:36 -- STEP: 31792/36938 -- GLOBAL_STEP: 1546250 | > loss_text_ce: 0.032329998910427094 (0.03025894535608601) | > loss_mel_ce: 2.968040943145752 (nan) | > loss: 0.011906234547495842 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13686075184173796) | > loader_time: 0.0047 (0.004658976435721307)  --> TIME: 2025-02-27 04:30:45 -- STEP: 31842/36938 -- GLOBAL_STEP: 1546300 | > loss_text_ce: 0.03817133232951164 (0.030257917681228565) | > loss_mel_ce: 2.661123037338257 (nan) | > loss: 0.0107114864513278 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13685466333307592) | > loader_time: 0.0048 (0.004658998634997759)  --> TIME: 2025-02-27 04:30:54 -- STEP: 31892/36938 -- GLOBAL_STEP: 1546350 | > loss_text_ce: 0.03403345122933388 (0.030259186153085885) | > loss_mel_ce: 2.077186107635498 (nan) | > loss: 0.008377856574952602 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13685730430482831) | > loader_time: 0.0044 (0.0046590027928157945)  --> TIME: 2025-02-27 04:31:03 -- STEP: 31942/36938 -- GLOBAL_STEP: 1546400 | > loss_text_ce: 0.04048561304807663 (0.030257969237372444) | > loss_mel_ce: 2.489941358566284 (nan) | > loss: 0.010041377507150173 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13684818573880736) | > loader_time: 0.0045 (0.004658952822824668)  --> TIME: 2025-02-27 04:31:13 -- STEP: 31992/36938 -- GLOBAL_STEP: 1546450 | > loss_text_ce: 0.03469187393784523 (0.03025627253671958) | > loss_mel_ce: 2.6522397994995117 (nan) | > loss: 0.010662428103387356 (nan) | > current_lr: 5e-06 | > step_time: 0.1504 (0.13685138731129334) | > loader_time: 0.0047 (0.004658947298901767)  --> TIME: 2025-02-27 04:31:22 -- STEP: 32042/36938 -- GLOBAL_STEP: 1546500 | > loss_text_ce: 0.02695172093808651 (0.030257634381188087) | > loss_mel_ce: 2.918955087661743 (nan) | > loss: 0.011690107174217701 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13684690290306772) | > loader_time: 0.0051 (0.004658995819972786)  --> TIME: 2025-02-27 04:31:31 -- STEP: 32092/36938 -- GLOBAL_STEP: 1546550 | > loss_text_ce: 0.03487659618258476 (0.030255973900581965) | > loss_mel_ce: 3.1552734375 (nan) | > loss: 0.012659326195716858 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13684908003933796) | > loader_time: 0.0046 (0.004659137307713484)  --> TIME: 2025-02-27 04:31:41 -- STEP: 32142/36938 -- GLOBAL_STEP: 1546600 | > loss_text_ce: 0.03605314716696739 (0.030256405715032016) | > loss_mel_ce: 2.5692529678344727 (nan) | > loss: 0.010338516905903816 (nan) | > current_lr: 5e-06 | > step_time: 0.1837 (0.13685123532200236) | > loader_time: 0.005 (0.004659100190369697)  --> TIME: 2025-02-27 04:31:50 -- STEP: 32192/36938 -- GLOBAL_STEP: 1546650 | > loss_text_ce: 0.03636663407087326 (0.030256649074926364) | > loss_mel_ce: 2.289525270462036 (nan) | > loss: 0.009229730814695358 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13684242357227855) | > loader_time: 0.0051 (0.004659125103687431)  --> TIME: 2025-02-27 04:31:59 -- STEP: 32242/36938 -- GLOBAL_STEP: 1546700 | > loss_text_ce: 0.050039418041706085 (0.030259724274040486) | > loss_mel_ce: 1.4863382577896118 (nan) | > loss: 0.006096736993640661 (nan) | > current_lr: 5e-06 | > step_time: 0.1073 (0.1368419205563437) | > loader_time: 0.0045 (0.004659234261026184)  --> TIME: 2025-02-27 04:32:08 -- STEP: 32292/36938 -- GLOBAL_STEP: 1546750 | > loss_text_ce: 0.03149554878473282 (0.030258630131154214) | > loss_mel_ce: 2.836491107940674 (nan) | > loss: 0.011380900628864765 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13684174324313622) | > loader_time: 0.0045 (0.004659352383175906)  --> TIME: 2025-02-27 04:32:18 -- STEP: 32342/36938 -- GLOBAL_STEP: 1546800 | > loss_text_ce: 0.033680181950330734 (0.03026036666325809) | > loss_mel_ce: 0.0895182266831398 (nan) | > loss: 0.000488882593344897 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13684058441507596) | > loader_time: 0.0046 (0.00465941198401014)  --> TIME: 2025-02-27 04:32:27 -- STEP: 32392/36938 -- GLOBAL_STEP: 1546850 | > loss_text_ce: 0.023270338773727417 (0.03026028170362661) | > loss_mel_ce: 0.02226766012609005 (nan) | > loss: 0.0001807063672458753 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13684212311340646) | > loader_time: 0.0046 (0.004659374780671391)  --> TIME: 2025-02-27 04:32:36 -- STEP: 32442/36938 -- GLOBAL_STEP: 1546900 | > loss_text_ce: 0.023696407675743103 (0.030260174122619086) | > loss_mel_ce: 3.1263577938079834 (nan) | > loss: 0.012500216253101826 (nan) | > current_lr: 5e-06 | > step_time: 0.1691 (0.13684003525528246) | > loader_time: 0.0048 (0.00465933965421121)  --> TIME: 2025-02-27 04:32:46 -- STEP: 32492/36938 -- GLOBAL_STEP: 1546950 | > loss_text_ce: 0.030346529558300972 (0.03025982612877831) | > loss_mel_ce: 2.7584807872772217 (nan) | > loss: 0.011066775768995285 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.13685069826162155) | > loader_time: 0.0047 (0.0046593530137272075)  --> TIME: 2025-02-27 04:32:55 -- STEP: 32542/36938 -- GLOBAL_STEP: 1547000 | > loss_text_ce: 0.03492381423711777 (0.030258877971276847) | > loss_mel_ce: 2.9446170330047607 (nan) | > loss: 0.011823575012385845 (nan) | > current_lr: 5e-06 | > step_time: 0.162 (0.13685288576376267) | > loader_time: 0.0046 (0.0046593617165024874)  --> TIME: 2025-02-27 04:33:05 -- STEP: 32592/36938 -- GLOBAL_STEP: 1547050 | > loss_text_ce: 0.028551004827022552 (0.030259220984557633) | > loss_mel_ce: 2.267042875289917 (nan) | > loss: 0.009109500795602798 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13686405743344848) | > loader_time: 0.0046 (0.004659483676513852)  --> TIME: 2025-02-27 04:33:15 -- STEP: 32642/36938 -- GLOBAL_STEP: 1547100 | > loss_text_ce: 0.030460437759757042 (0.030259058384268605) | > loss_mel_ce: 0.192718505859375 (nan) | > loss: 0.0008856307831592858 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13686641084600915) | > loader_time: 0.0046 (0.004659600296146718)  --> TIME: 2025-02-27 04:33:24 -- STEP: 32692/36938 -- GLOBAL_STEP: 1547150 | > loss_text_ce: 0.0330970361828804 (0.030260967407438825) | > loss_mel_ce: 2.815519332885742 (nan) | > loss: 0.011304033920168877 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.1368652619289312) | > loader_time: 0.0046 (0.004659562912825106)  --> TIME: 2025-02-27 04:33:33 -- STEP: 32742/36938 -- GLOBAL_STEP: 1547200 | > loss_text_ce: 0.02763999253511429 (0.030260029083816895) | > loss_mel_ce: 3.2882769107818604 (nan) | > loss: 0.013158401474356651 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.1368577789571476) | > loader_time: 0.0046 (0.004659556831351472)  --> TIME: 2025-02-27 04:33:42 -- STEP: 32792/36938 -- GLOBAL_STEP: 1547250 | > loss_text_ce: 0.03179899603128433 (0.03026179188514419) | > loss_mel_ce: 2.687211751937866 (nan) | > loss: 0.010789725929498672 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.1368558164790493) | > loader_time: 0.0047 (0.0046596490601034475)  --> TIME: 2025-02-27 04:33:51 -- STEP: 32842/36938 -- GLOBAL_STEP: 1547300 | > loss_text_ce: 0.03841817006468773 (0.030260687071743717) | > loss_mel_ce: 3.2273523807525635 (nan) | > loss: 0.012959407642483711 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13685558953839108) | > loader_time: 0.0045 (0.004659760608850806)  --> TIME: 2025-02-27 04:34:01 -- STEP: 32892/36938 -- GLOBAL_STEP: 1547350 | > loss_text_ce: 0.027342509478330612 (0.03026086023860493) | > loss_mel_ce: 2.2827625274658203 (nan) | > loss: 0.009167084470391273 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13686318676880369) | > loader_time: 0.0046 (0.004659814627576767)  --> TIME: 2025-02-27 04:34:10 -- STEP: 32942/36938 -- GLOBAL_STEP: 1547400 | > loss_text_ce: 0.031523965299129486 (0.030260896399506035) | > loss_mel_ce: 2.6594464778900146 (nan) | > loss: 0.010678454302251339 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.1368629514943082) | > loader_time: 0.0048 (0.004659824731477178)  --> TIME: 2025-02-27 04:34:20 -- STEP: 32992/36938 -- GLOBAL_STEP: 1547450 | > loss_text_ce: 0.031703732907772064 (0.030261124032595295) | > loss_mel_ce: 2.582263708114624 (nan) | > loss: 0.01037288736552 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13686813060718309) | > loader_time: 0.0046 (0.0046598954861541)  --> TIME: 2025-02-27 04:34:29 -- STEP: 33042/36938 -- GLOBAL_STEP: 1547500 | > loss_text_ce: 0.028245167806744576 (0.030261387672936776) | > loss_mel_ce: 2.571211338043213 (nan) | > loss: 0.0103153046220541 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.1368642070522752) | > loader_time: 0.0046 (0.004659891590339839)  --> TIME: 2025-02-27 04:34:39 -- STEP: 33092/36938 -- GLOBAL_STEP: 1547550 | > loss_text_ce: 0.0214459840208292 (0.03026085853300911) | > loss_mel_ce: 2.443572998046875 (nan) | > loss: 0.009781821630895138 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13686882546803214) | > loader_time: 0.0046 (0.00465998747004976)  --> TIME: 2025-02-27 04:34:48 -- STEP: 33142/36938 -- GLOBAL_STEP: 1547600 | > loss_text_ce: 0.037353113293647766 (0.03026106191900377) | > loss_mel_ce: 2.503796339035034 (nan) | > loss: 0.01008392684161663 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.1368678322123844) | > loader_time: 0.0046 (0.004659908573626462)  --> TIME: 2025-02-27 04:34:57 -- STEP: 33192/36938 -- GLOBAL_STEP: 1547650 | > loss_text_ce: 0.034423135221004486 (0.030262035627781056) | > loss_mel_ce: 2.253710985183716 (nan) | > loss: 0.00907989777624607 (nan) | > current_lr: 5e-06 | > step_time: 0.1811 (0.1368570452743235) | > loader_time: 0.0049 (0.004659818350248777)  --> TIME: 2025-02-27 04:35:06 -- STEP: 33242/36938 -- GLOBAL_STEP: 1547700 | > loss_text_ce: 0.03211356699466705 (0.030260694788683283) | > loss_mel_ce: 3.101728677749634 (nan) | > loss: 0.012435882352292538 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1368592999112167) | > loader_time: 0.0046 (0.00465991676201581)  --> TIME: 2025-02-27 04:35:16 -- STEP: 33292/36938 -- GLOBAL_STEP: 1547750 | > loss_text_ce: 0.02278032898902893 (0.0302595452813071) | > loss_mel_ce: 2.6892361640930176 (nan) | > loss: 0.010761971585452557 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13685788072589403) | > loader_time: 0.0046 (0.0046599138804271495)  --> TIME: 2025-02-27 04:35:25 -- STEP: 33342/36938 -- GLOBAL_STEP: 1547800 | > loss_text_ce: 0.03155630826950073 (0.030259790976191206) | > loss_mel_ce: 0.004364013671875 (nan) | > loss: 0.0001425409718649462 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.1368570694017564) | > loader_time: 0.0046 (0.004659826929571673)  --> TIME: 2025-02-27 04:35:34 -- STEP: 33392/36938 -- GLOBAL_STEP: 1547850 | > loss_text_ce: 0.03664577007293701 (0.03025995343539516) | > loss_mel_ce: 2.3504531383514404 (nan) | > loss: 0.009472615085542202 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.1368568587457399) | > loader_time: 0.0046 (0.004659810268142087)  --> TIME: 2025-02-27 04:35:43 -- STEP: 33442/36938 -- GLOBAL_STEP: 1547900 | > loss_text_ce: 0.03396730497479439 (0.030260798263500956) | > loss_mel_ce: 2.5648117065429688 (nan) | > loss: 0.010312615893781185 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1368514903602228) | > loader_time: 0.0046 (0.004659727211311082)  --> TIME: 2025-02-27 04:35:53 -- STEP: 33492/36938 -- GLOBAL_STEP: 1547950 | > loss_text_ce: 0.03503314033150673 (0.03026111457103075) | > loss_mel_ce: 3.1202619075775146 (nan) | > loss: 0.012521013617515564 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13684913676570815) | > loader_time: 0.0046 (0.0046596612808426155)  --> TIME: 2025-02-27 04:36:02 -- STEP: 33542/36938 -- GLOBAL_STEP: 1548000 | > loss_text_ce: 0.0265414547175169 (0.030260332948611805) | > loss_mel_ce: 1.7586039304733276 (nan) | > loss: 0.007083910517394543 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.1368470420195416) | > loader_time: 0.0048 (0.0046596532572816245)  --> TIME: 2025-02-27 04:36:11 -- STEP: 33592/36938 -- GLOBAL_STEP: 1548050 | > loss_text_ce: 0.040378108620643616 (0.030259659674647305) | > loss_mel_ce: 3.396466016769409 (nan) | > loss: 0.013638271018862724 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.1368462827609376) | > loader_time: 0.0046 (0.004659584219270045)  --> TIME: 2025-02-27 04:36:21 -- STEP: 33642/36938 -- GLOBAL_STEP: 1548100 | > loss_text_ce: 0.021895328536629677 (0.03025761247341461) | > loss_mel_ce: 1.2703043222427368 (nan) | > loss: 0.00512777641415596 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1368451405560882) | > loader_time: 0.0046 (0.004659583725761155)  --> TIME: 2025-02-27 04:36:30 -- STEP: 33692/36938 -- GLOBAL_STEP: 1548150 | > loss_text_ce: 0.025729794055223465 (0.03025846892330555) | > loss_mel_ce: 2.9155938625335693 (nan) | > loss: 0.01167192030698061 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.13684376027286146) | > loader_time: 0.0047 (0.0046597144516830865)  --> TIME: 2025-02-27 04:36:39 -- STEP: 33742/36938 -- GLOBAL_STEP: 1548200 | > loss_text_ce: 0.03698696196079254 (0.03025879614599398) | > loss_mel_ce: 3.112194299697876 (nan) | > loss: 0.012496751733124256 (nan) | > current_lr: 5e-06 | > step_time: 0.1828 (0.13684749019625875) | > loader_time: 0.0048 (0.004659723630691137)  --> TIME: 2025-02-27 04:36:48 -- STEP: 33792/36938 -- GLOBAL_STEP: 1548250 | > loss_text_ce: 0.0337427482008934 (0.03026022644221345) | > loss_mel_ce: 2.306899309158325 (nan) | > loss: 0.009288262575864792 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.1368446142455068) | > loader_time: 0.0046 (0.004659628220410509)  --> TIME: 2025-02-27 04:36:57 -- STEP: 33842/36938 -- GLOBAL_STEP: 1548300 | > loss_text_ce: 0.040244393050670624 (0.030261998640955214) | > loss_mel_ce: 2.0720036029815674 (nan) | > loss: 0.008381936699151993 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13683658101712262) | > loader_time: 0.0045 (0.004659583590976259)  --> TIME: 2025-02-27 04:37:07 -- STEP: 33892/36938 -- GLOBAL_STEP: 1548350 | > loss_text_ce: 0.026071524247527122 (0.03026193047137069) | > loss_mel_ce: 2.311227798461914 (nan) | > loss: 0.009274997748434544 (nan) | > current_lr: 5e-06 | > step_time: 0.1767 (0.1368405046604282) | > loader_time: 0.0052 (0.004659579444013185)  --> TIME: 2025-02-27 04:37:16 -- STEP: 33942/36938 -- GLOBAL_STEP: 1548400 | > loss_text_ce: 0.031842198222875595 (0.03026317373698428) | > loss_mel_ce: 1.9817898273468018 (nan) | > loss: 0.007990604266524315 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1368398168148692) | > loader_time: 0.0045 (0.004659501174871466)  --> TIME: 2025-02-27 04:37:26 -- STEP: 33992/36938 -- GLOBAL_STEP: 1548450 | > loss_text_ce: 0.025568455457687378 (0.030262500432341955) | > loss_mel_ce: 3.0306243896484375 (nan) | > loss: 0.012127750553190708 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1368480684491277) | > loader_time: 0.0046 (0.00465954125109884)  --> TIME: 2025-02-27 04:37:35 -- STEP: 34042/36938 -- GLOBAL_STEP: 1548500 | > loss_text_ce: 0.03053738735616207 (0.030261988434660964) | > loss_mel_ce: 2.880061626434326 (nan) | > loss: 0.011549996212124825 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13684816065899846) | > loader_time: 0.0046 (0.004659540406279726)  --> TIME: 2025-02-27 04:37:45 -- STEP: 34092/36938 -- GLOBAL_STEP: 1548550 | > loss_text_ce: 0.028710147365927696 (0.030260252732314002) | > loss_mel_ce: 2.1190779209136963 (nan) | > loss: 0.008522968739271164 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13684811846521636) | > loader_time: 0.0046 (0.004659529129803974)  --> TIME: 2025-02-27 04:37:54 -- STEP: 34142/36938 -- GLOBAL_STEP: 1548600 | > loss_text_ce: 0.0284466203302145 (0.03025790609800526) | > loss_mel_ce: 3.0212881565093994 (nan) | > loss: 0.012102123349905014 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.1368448977925433) | > loader_time: 0.0046 (0.004659529687873538)  --> TIME: 2025-02-27 04:38:03 -- STEP: 34192/36938 -- GLOBAL_STEP: 1548650 | > loss_text_ce: 0.030351672321558 (0.030257458658543) | > loss_mel_ce: 0.7830871343612671 (nan) | > loss: 0.003227932145819068 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13684094361490348) | > loader_time: 0.0047 (0.004659478700378535)  --> TIME: 2025-02-27 04:38:12 -- STEP: 34242/36938 -- GLOBAL_STEP: 1548700 | > loss_text_ce: 0.033719751983881 (0.030256294783250582) | > loss_mel_ce: 2.084676742553711 (nan) | > loss: 0.008406335487961769 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13683878596325866) | > loader_time: 0.0048 (0.0046595668731452405)  --> TIME: 2025-02-27 04:38:22 -- STEP: 34292/36938 -- GLOBAL_STEP: 1548750 | > loss_text_ce: 0.0331769697368145 (0.030255705825674314) | > loss_mel_ce: 2.0839920043945312 (nan) | > loss: 0.00840146467089653 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13684004414503032) | > loader_time: 0.0046 (0.004659598417103007)  --> TIME: 2025-02-27 04:38:31 -- STEP: 34342/36938 -- GLOBAL_STEP: 1548800 | > loss_text_ce: 0.03507998585700989 (0.030255926406966865) | > loss_mel_ce: 2.9622225761413574 (nan) | > loss: 0.01189405843615532 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13684494665760383) | > loader_time: 0.0045 (0.004659606514712395)  --> TIME: 2025-02-27 04:38:40 -- STEP: 34392/36938 -- GLOBAL_STEP: 1548850 | > loss_text_ce: 0.029945585876703262 (0.030254753995156536) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13683658195640158) | > loader_time: 0.0045 (0.004659542727675594)  --> TIME: 2025-02-27 04:38:49 -- STEP: 34442/36938 -- GLOBAL_STEP: 1548900 | > loss_text_ce: 0.03581906855106354 (0.03025416586435655) | > loss_mel_ce: 2.462705373764038 (nan) | > loss: 0.009914780035614967 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13682832979420934) | > loader_time: 0.0046 (0.004659460747077915)  --> TIME: 2025-02-27 04:38:58 -- STEP: 34492/36938 -- GLOBAL_STEP: 1548950 | > loss_text_ce: 0.02588091790676117 (0.030254209036986356) | > loss_mel_ce: 2.7504661083221436 (nan) | > loss: 0.011017249897122383 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.1368277690450166) | > loader_time: 0.0045 (0.004659369990538931)  --> TIME: 2025-02-27 04:39:08 -- STEP: 34542/36938 -- GLOBAL_STEP: 1549000 | > loss_text_ce: 0.036263711750507355 (0.030255922808980363) | > loss_mel_ce: 2.936845302581787 (nan) | > loss: 0.011798052117228508 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13683005481152682) | > loader_time: 0.0046 (0.004659434742839479)  --> TIME: 2025-02-27 04:39:17 -- STEP: 34592/36938 -- GLOBAL_STEP: 1549050 | > loss_text_ce: 0.033601488918066025 (0.030254491755027965) | > loss_mel_ce: 2.239654302597046 (nan) | > loss: 0.009020856581628323 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1368303145872137) | > loader_time: 0.0046 (0.004659448945886222)  --> TIME: 2025-02-27 04:39:26 -- STEP: 34642/36938 -- GLOBAL_STEP: 1549100 | > loss_text_ce: 0.04595724120736122 (0.030253719458268092) | > loss_mel_ce: 3.0939888954162598 (nan) | > loss: 0.012460105121135712 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13683001056058147) | > loader_time: 0.0048 (0.004659411930733703)  --> TIME: 2025-02-27 04:39:36 -- STEP: 34692/36938 -- GLOBAL_STEP: 1549150 | > loss_text_ce: 0.02980598621070385 (0.030252502763238053) | > loss_mel_ce: 2.191389560699463 (nan) | > loss: 0.008814267814159393 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.1368279197176737) | > loader_time: 0.0046 (0.004659324228096956)  --> TIME: 2025-02-27 04:39:45 -- STEP: 34742/36938 -- GLOBAL_STEP: 1549200 | > loss_text_ce: 0.03523048013448715 (0.030253191994729162) | > loss_mel_ce: 3.26253604888916 (nan) | > loss: 0.013086375780403614 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13683312923740507) | > loader_time: 0.0045 (0.004659244319838251)  --> TIME: 2025-02-27 04:39:55 -- STEP: 34792/36938 -- GLOBAL_STEP: 1549250 | > loss_text_ce: 0.035999421030282974 (0.03025199099268305) | > loss_mel_ce: 2.283480644226074 (nan) | > loss: 0.009204287081956863 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.1368420289888919) | > loader_time: 0.0046 (0.0046592326472890445)  --> TIME: 2025-02-27 04:40:04 -- STEP: 34842/36938 -- GLOBAL_STEP: 1549300 | > loss_text_ce: 0.03276965767145157 (0.030251686895162663) | > loss_mel_ce: 2.246159315109253 (nan) | > loss: 0.009043369442224503 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13684516586438283) | > loader_time: 0.0066 (0.00465924316538946)  --> TIME: 2025-02-27 04:40:13 -- STEP: 34892/36938 -- GLOBAL_STEP: 1549350 | > loss_text_ce: 0.030242448672652245 (0.030251136350562452) | > loss_mel_ce: 1.994098424911499 (nan) | > loss: 0.008033099584281445 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13683818418553664) | > loader_time: 0.0045 (0.0046591311436952314)  --> TIME: 2025-02-27 04:40:23 -- STEP: 34942/36938 -- GLOBAL_STEP: 1549400 | > loss_text_ce: 0.03516058996319771 (0.03025241629333021) | > loss_mel_ce: 3.045964479446411 (nan) | > loss: 0.012226687744259834 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.13683338903833703) | > loader_time: 0.0048 (0.004659136905127575)  --> TIME: 2025-02-27 04:40:32 -- STEP: 34992/36938 -- GLOBAL_STEP: 1549450 | > loss_text_ce: 0.029466349631547928 (0.030253666143700536) | > loss_mel_ce: 2.753981828689575 (nan) | > loss: 0.011045429855585098 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.13683657792397405) | > loader_time: 0.0048 (0.004659178863936596)  --> TIME: 2025-02-27 04:40:41 -- STEP: 35042/36938 -- GLOBAL_STEP: 1549500 | > loss_text_ce: 0.033083055168390274 (0.030253297875191314) | > loss_mel_ce: 2.54158091545105 (nan) | > loss: 0.010216920636594296 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13683102945440356) | > loader_time: 0.0046 (0.00465916949084316)  --> TIME: 2025-02-27 04:40:50 -- STEP: 35092/36938 -- GLOBAL_STEP: 1549550 | > loss_text_ce: 0.029868310317397118 (0.030251279768304055) | > loss_mel_ce: 2.7357430458068848 (nan) | > loss: 0.010974649339914322 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13683329541123782) | > loader_time: 0.0046 (0.004659231829015468)  --> TIME: 2025-02-27 04:41:00 -- STEP: 35142/36938 -- GLOBAL_STEP: 1549600 | > loss_text_ce: 0.03228180482983589 (0.030250740941485315) | > loss_mel_ce: 2.2026500701904297 (nan) | > loss: 0.008868778124451637 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13683089383752717) | > loader_time: 0.0046 (0.004659239531146806)  --> TIME: 2025-02-27 04:41:09 -- STEP: 35192/36938 -- GLOBAL_STEP: 1549650 | > loss_text_ce: 0.026190966367721558 (0.030252023674419044) | > loss_mel_ce: 2.924283266067505 (nan) | > loss: 0.011708231642842293 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.1368262475594863) | > loader_time: 0.0047 (0.004659267163162846)  --> TIME: 2025-02-27 04:41:18 -- STEP: 35242/36938 -- GLOBAL_STEP: 1549700 | > loss_text_ce: 0.02892123907804489 (0.0302515735181914) | > loss_mel_ce: 2.2488536834716797 (nan) | > loss: 0.009038789197802544 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13682632714596377) | > loader_time: 0.0046 (0.004659341504777987)  --> TIME: 2025-02-27 04:41:28 -- STEP: 35292/36938 -- GLOBAL_STEP: 1549750 | > loss_text_ce: 0.04920413717627525 (0.03025126618188917) | > loss_mel_ce: 1.8381614685058594 (nan) | > loss: 0.007489546202123165 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13682715842530258) | > loader_time: 0.0045 (0.004659365164670187)  --> TIME: 2025-02-27 04:41:37 -- STEP: 35342/36938 -- GLOBAL_STEP: 1549800 | > loss_text_ce: 0.022562265396118164 (0.03025149499941788) | > loss_mel_ce: 2.853863000869751 (nan) | > loss: 0.011414386332035065 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13682417955647516) | > loader_time: 0.0046 (0.0046593742941050615)  --> TIME: 2025-02-27 04:41:46 -- STEP: 35392/36938 -- GLOBAL_STEP: 1549850 | > loss_text_ce: 0.02354530431330204 (0.030250826188605532) | > loss_mel_ce: 2.094754219055176 (nan) | > loss: 0.008405950851738453 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.1368260859607804) | > loader_time: 0.0047 (0.004659343396345476)  --> TIME: 2025-02-27 04:41:56 -- STEP: 35442/36938 -- GLOBAL_STEP: 1549900 | > loss_text_ce: 0.032255496829748154 (0.03025204960295862) | > loss_mel_ce: 2.9453816413879395 (nan) | > loss: 0.011816021054983139 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13682680771492423) | > loader_time: 0.0045 (0.004659346691689329)  --> TIME: 2025-02-27 04:42:05 -- STEP: 35492/36938 -- GLOBAL_STEP: 1549950 | > loss_text_ce: 0.02156597375869751 (0.030251356095933025) | > loss_mel_ce: 1.012646198272705 (nan) | > loss: 0.004104016348719597 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13682778769276907) | > loader_time: 0.0045 (0.004659285731290343)  --> TIME: 2025-02-27 04:42:14 -- STEP: 35542/36938 -- GLOBAL_STEP: 1550000 | > loss_text_ce: 0.02868659421801567 (0.030250743503700803) | > loss_mel_ce: 2.539824962615967 (nan) | > loss: 0.010192506946623325 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.13682381536996846) | > loader_time: 0.0046 (0.004659197137416299) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1550000.pth  --> TIME: 2025-02-27 04:42:29 -- STEP: 35592/36938 -- GLOBAL_STEP: 1550050 | > loss_text_ce: 0.028358962386846542 (0.030251950309165443) | > loss_mel_ce: 2.3470041751861572 (nan) | > loss: 0.009426044300198555 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13682360608515187) | > loader_time: 0.0052 (0.0046594306695967565)  --> TIME: 2025-02-27 04:42:39 -- STEP: 35642/36938 -- GLOBAL_STEP: 1550100 | > loss_text_ce: 0.02818981744349003 (0.03025204815287657) | > loss_mel_ce: 2.816582679748535 (nan) | > loss: 0.011288780719041824 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.1368224034572425) | > loader_time: 0.0047 (0.00465952752047358)  --> TIME: 2025-02-27 04:42:48 -- STEP: 35692/36938 -- GLOBAL_STEP: 1550150 | > loss_text_ce: 0.03605473041534424 (0.030250520718877405) | > loss_mel_ce: 1.869984745979309 (nan) | > loss: 0.007563649211078882 (nan) | > current_lr: 5e-06 | > step_time: 0.1506 (0.1368199447592628) | > loader_time: 0.0046 (0.004659631708391585)  --> TIME: 2025-02-27 04:42:57 -- STEP: 35742/36938 -- GLOBAL_STEP: 1550200 | > loss_text_ce: 0.03157160058617592 (0.030250528566493937) | > loss_mel_ce: 2.9352452754974365 (nan) | > loss: 0.011773083359003067 (nan) | > current_lr: 5e-06 | > step_time: 0.1719 (0.13681271745124682) | > loader_time: 0.0047 (0.004659643864819742)  --> TIME: 2025-02-27 04:43:06 -- STEP: 35792/36938 -- GLOBAL_STEP: 1550250 | > loss_text_ce: 0.04226715490221977 (0.030250038225730913) | > loss_mel_ce: 1.4141650199890137 (nan) | > loss: 0.005779493134468794 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13680720400559923) | > loader_time: 0.0045 (0.004659744235197753)  --> TIME: 2025-02-27 04:43:15 -- STEP: 35842/36938 -- GLOBAL_STEP: 1550300 | > loss_text_ce: 0.02903912216424942 (0.030249732030165) | > loss_mel_ce: 3.83203125 (nan) | > loss: 0.015321709215641022 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13680471490452822) | > loader_time: 0.0048 (0.004659764668643605)  --> TIME: 2025-02-27 04:43:25 -- STEP: 35892/36938 -- GLOBAL_STEP: 1550350 | > loss_text_ce: 0.02473612315952778 (0.030249398888921575) | > loss_mel_ce: 2.5404276847839355 (nan) | > loss: 0.01017922256141901 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.1368054494960897) | > loader_time: 0.0047 (0.00465983666531897)  --> TIME: 2025-02-27 04:43:34 -- STEP: 35942/36938 -- GLOBAL_STEP: 1550400 | > loss_text_ce: 0.024907946586608887 (0.030250063793671306) | > loss_mel_ce: 2.529595375061035 (nan) | > loss: 0.010136919096112251 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.1368074067267413) | > loader_time: 0.0046 (0.0046598182404606996)  --> TIME: 2025-02-27 04:43:43 -- STEP: 35992/36938 -- GLOBAL_STEP: 1550450 | > loss_text_ce: 0.026804693043231964 (0.030247685202240793) | > loss_mel_ce: 2.701860189437866 (nan) | > loss: 0.01082803588360548 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.1368052253036362) | > loader_time: 0.0047 (0.004659785300155393)  --> TIME: 2025-02-27 04:43:53 -- STEP: 36042/36938 -- GLOBAL_STEP: 1550500 | > loss_text_ce: 0.030915841460227966 (0.030250113409753736) | > loss_mel_ce: 2.681225061416626 (nan) | > loss: 0.010762464255094528 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13680725162219) | > loader_time: 0.0045 (0.004659836409088099)  --> TIME: 2025-02-27 04:44:02 -- STEP: 36092/36938 -- GLOBAL_STEP: 1550550 | > loss_text_ce: 0.024263570085167885 (0.030248654197962268) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13680858669135293) | > loader_time: 0.0045 (0.004659828115273204)  --> TIME: 2025-02-27 04:44:11 -- STEP: 36142/36938 -- GLOBAL_STEP: 1550600 | > loss_text_ce: 0.031737376004457474 (0.03024962803425533) | > loss_mel_ce: 2.853557586669922 (nan) | > loss: 0.011449583806097507 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13680360150465334) | > loader_time: 0.0045 (0.0046597598208698065)  --> TIME: 2025-02-27 04:44:21 -- STEP: 36192/36938 -- GLOBAL_STEP: 1550650 | > loss_text_ce: 0.03140706941485405 (0.030249629365259156) | > loss_mel_ce: 2.926072120666504 (nan) | > loss: 0.011736029759049416 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.13680021879843884) | > loader_time: 0.0049 (0.004659706800780798)  --> TIME: 2025-02-27 04:44:30 -- STEP: 36242/36938 -- GLOBAL_STEP: 1550700 | > loss_text_ce: 0.0449313148856163 (0.030246898246917526) | > loss_mel_ce: 2.8556292057037354 (nan) | > loss: 0.011510161682963371 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1368019788590615) | > loader_time: 0.0046 (0.004659755236133738)  --> TIME: 2025-02-27 04:44:40 -- STEP: 36292/36938 -- GLOBAL_STEP: 1550750 | > loss_text_ce: 0.03176681697368622 (0.030246803078394044) | > loss_mel_ce: 2.5612261295318604 (nan) | > loss: 0.010289655067026615 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13680674486986588) | > loader_time: 0.0046 (0.004659784053029694)  --> TIME: 2025-02-27 04:44:49 -- STEP: 36342/36938 -- GLOBAL_STEP: 1550800 | > loss_text_ce: 0.03740950673818588 (0.03024672895669577) | > loss_mel_ce: 2.8374359607696533 (nan) | > loss: 0.011408117599785328 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.1368066278001785) | > loader_time: 0.0047 (0.004659751221143126)  --> TIME: 2025-02-27 04:44:59 -- STEP: 36392/36938 -- GLOBAL_STEP: 1550850 | > loss_text_ce: 0.032471369951963425 (0.0302462506177313) | > loss_mel_ce: 3.6243529319763184 (nan) | > loss: 0.014511208981275558 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13681498743801823) | > loader_time: 0.0046 (0.004659824094606703)  --> TIME: 2025-02-27 04:45:08 -- STEP: 36442/36938 -- GLOBAL_STEP: 1550900 | > loss_text_ce: 0.0307651124894619 (0.03024736372558471) | > loss_mel_ce: 2.970123529434204 (nan) | > loss: 0.011908289045095444 (nan) | > current_lr: 5e-06 | > step_time: 0.1777 (0.1368101115650968) | > loader_time: 0.0048 (0.004659741313847007)  --> TIME: 2025-02-27 04:45:17 -- STEP: 36492/36938 -- GLOBAL_STEP: 1550950 | > loss_text_ce: 0.025784077122807503 (0.03024738296442201) | > loss_mel_ce: 2.910085439682007 (nan) | > loss: 0.011650276370346546 (nan) | > current_lr: 5e-06 | > step_time: 0.1773 (0.13681286860881867) | > loader_time: 0.0048 (0.004659689055531335)  --> TIME: 2025-02-27 04:45:27 -- STEP: 36542/36938 -- GLOBAL_STEP: 1551000 | > loss_text_ce: 0.02867424674332142 (0.030246346516530187) | > loss_mel_ce: 2.498599052429199 (nan) | > loss: 0.010028863325715065 (nan) | > current_lr: 5e-06 | > step_time: 0.1512 (0.1368154475635576) | > loader_time: 0.0046 (0.004659647947069927)  --> TIME: 2025-02-27 04:45:36 -- STEP: 36592/36938 -- GLOBAL_STEP: 1551050 | > loss_text_ce: 0.03526710346341133 (0.030245032195744707) | > loss_mel_ce: 2.894474983215332 (nan) | > loss: 0.011625961400568485 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13680996818473792) | > loader_time: 0.0045 (0.004659620711883692)  --> TIME: 2025-02-27 04:45:45 -- STEP: 36642/36938 -- GLOBAL_STEP: 1551100 | > loss_text_ce: 0.034794289618730545 (0.030245405337323627) | > loss_mel_ce: 2.2866222858428955 (nan) | > loss: 0.009211971424520016 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13680409597821006) | > loader_time: 0.0046 (0.004659594487990345)  --> TIME: 2025-02-27 04:45:54 -- STEP: 36692/36938 -- GLOBAL_STEP: 1551150 | > loss_text_ce: 0.029539057984948158 (0.0302444843543859) | > loss_mel_ce: 3.03572154045105 (nan) | > loss: 0.012163733132183552 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.1368027338606845) | > loader_time: 0.0046 (0.004659557055324953)  --> TIME: 2025-02-27 04:46:04 -- STEP: 36742/36938 -- GLOBAL_STEP: 1551200 | > loss_text_ce: 0.031112780794501305 (0.030245723543255908) | > loss_mel_ce: 2.7695767879486084 (nan) | > loss: 0.011113847605884075 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13680668763853385) | > loader_time: 0.0045 (0.004659576250156954)  --> TIME: 2025-02-27 04:46:14 -- STEP: 36792/36938 -- GLOBAL_STEP: 1551250 | > loss_text_ce: 0.02525385282933712 (0.030246084231835585) | > loss_mel_ce: 2.3220736980438232 (nan) | > loss: 0.009314792230725288 (nan) | > current_lr: 5e-06 | > step_time: 0.1529 (0.1368159883357734) | > loader_time: 0.0048 (0.004659606921048121)  --> TIME: 2025-02-27 04:46:23 -- STEP: 36842/36938 -- GLOBAL_STEP: 1551300 | > loss_text_ce: 0.03565394505858421 (0.030248085806626752) | > loss_mel_ce: 2.7703115940093994 (nan) | > loss: 0.011134783737361431 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13681491741715657) | > loader_time: 0.0047 (0.004659607462072511)  --> TIME: 2025-02-27 04:46:32 -- STEP: 36892/36938 -- GLOBAL_STEP: 1551350 | > loss_text_ce: 0.036004070192575455 (0.030247272846731656) | > loss_mel_ce: 3.0764307975769043 (nan) | > loss: 0.012350932694971561 (nan) | > current_lr: 5e-06 | > step_time: 0.1709 (0.1368156939140708) | > loader_time: 0.0048 (0.004659559887504793)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.020254400697084937 (-0.00027863460012002295) | > avg_loss_text_ce: 0.029505676470033014 (-0.00012294502264939208) | > avg_loss_mel_ce: 2.4308989145378064 (-0.004051592975563345) | > avg_loss: 2.4604046046733865 (-0.00417452342439395) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_1551396.pth  > EPOCH: 43/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-27 04:47:05)   --> TIME: 2025-02-27 04:47:06 -- STEP: 4/36938 -- GLOBAL_STEP: 1551400 | > loss_text_ce: 0.024816080927848816 (0.028894139919430017) | > loss_mel_ce: 2.026547431945801 (2.442493438720703) | > loss: 0.00814033206552267 (0.009807094349525869) | > current_lr: 5e-06 | > step_time: 0.1359 (0.12769335508346558) | > loader_time: 0.0044 (0.005515158176422119)  --> TIME: 2025-02-27 04:47:15 -- STEP: 54/36938 -- GLOBAL_STEP: 1551450 | > loss_text_ce: 0.03258175030350685 (0.029445864053236112) | > loss_mel_ce: 2.501366376876831 (nan) | > loss: 0.01005535013973713 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13502556747860375) | > loader_time: 0.0048 (0.004872953450238265)  --> TIME: 2025-02-27 04:47:24 -- STEP: 104/36938 -- GLOBAL_STEP: 1551500 | > loss_text_ce: 0.039857421070337296 (0.029329613854105655) | > loss_mel_ce: 2.5155911445617676 (nan) | > loss: 0.010140669532120228 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13404571322294387) | > loader_time: 0.0068 (0.004930030841093798)  --> TIME: 2025-02-27 04:47:34 -- STEP: 154/36938 -- GLOBAL_STEP: 1551550 | > loss_text_ce: 0.033584319055080414 (0.029909208247607403) | > loss_mel_ce: 2.835545063018799 (nan) | > loss: 0.011385435238480568 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.13459970115067132) | > loader_time: 0.0048 (0.00492874678079184)  --> TIME: 2025-02-27 04:47:43 -- STEP: 204/36938 -- GLOBAL_STEP: 1551600 | > loss_text_ce: 0.029386354610323906 (0.029671890761557164) | > loss_mel_ce: 2.0674853324890137 (nan) | > loss: 0.00832091923803091 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.1348552470113718) | > loader_time: 0.0047 (0.004903255724439436)  --> TIME: 2025-02-27 04:47:52 -- STEP: 254/36938 -- GLOBAL_STEP: 1551650 | > loss_text_ce: 0.036667924374341965 (0.02984266380566781) | > loss_mel_ce: 2.859251022338867 (nan) | > loss: 0.011491741985082626 (nan) | > current_lr: 5e-06 | > step_time: 0.1494 (0.1358594059005498) | > loader_time: 0.0047 (0.004860731560414233)  --> TIME: 2025-02-27 04:48:01 -- STEP: 304/36938 -- GLOBAL_STEP: 1551700 | > loss_text_ce: 0.029495473951101303 (0.029801927365999865) | > loss_mel_ce: 1.036336898803711 (nan) | > loss: 0.004229493904858828 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.1352793025343042) | > loader_time: 0.0045 (0.004832023068478237)  --> TIME: 2025-02-27 04:48:11 -- STEP: 354/36938 -- GLOBAL_STEP: 1551750 | > loss_text_ce: 0.029279649257659912 (0.02993275254628079) | > loss_mel_ce: 2.9772329330444336 (nan) | > loss: 0.011930606327950954 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.13585434256300416) | > loader_time: 0.0046 (0.004816229060544811)  --> TIME: 2025-02-27 04:48:20 -- STEP: 404/36938 -- GLOBAL_STEP: 1551800 | > loss_text_ce: 0.03199262544512749 (0.029862817422957114) | > loss_mel_ce: 2.7346997261047363 (nan) | > loss: 0.010978939011693 (nan) | > current_lr: 5e-06 | > step_time: 0.1831 (0.1361947590761845) | > loader_time: 0.0048 (0.004804235283691113)  --> TIME: 2025-02-27 04:48:30 -- STEP: 454/36938 -- GLOBAL_STEP: 1551850 | > loss_text_ce: 0.02561536803841591 (0.02983139047916109) | > loss_mel_ce: 2.7025115489959717 (nan) | > loss: 0.010825901292264462 (nan) | > current_lr: 5e-06 | > step_time: 0.1838 (0.13617730718352192) | > loader_time: 0.0048 (0.004796003980258497)  --> TIME: 2025-02-27 04:48:39 -- STEP: 504/36938 -- GLOBAL_STEP: 1551900 | > loss_text_ce: 0.024277059361338615 (0.029766788713574882) | > loss_mel_ce: 2.2120742797851562 (nan) | > loss: 0.008874409832060337 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.1362424322537013) | > loader_time: 0.0045 (0.004781280245099757)  --> TIME: 2025-02-27 04:48:49 -- STEP: 554/36938 -- GLOBAL_STEP: 1551950 | > loss_text_ce: 0.025679761543869972 (0.02977548366822706) | > loss_mel_ce: 2.8490142822265625 (nan) | > loss: 0.011407516896724701 (nan) | > current_lr: 5e-06 | > step_time: 0.1988 (0.1369218731615087) | > loader_time: 0.0049 (0.0047788684548884155)  --> TIME: 2025-02-27 04:48:58 -- STEP: 604/36938 -- GLOBAL_STEP: 1552000 | > loss_text_ce: 0.03011489100754261 (0.029774402564280475) | > loss_mel_ce: 2.738276720046997 (nan) | > loss: 0.010985681787133217 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13686774463843043) | > loader_time: 0.0046 (0.004777082939021644)  --> TIME: 2025-02-27 04:49:07 -- STEP: 654/36938 -- GLOBAL_STEP: 1552050 | > loss_text_ce: 0.024341516196727753 (0.02972656472611728) | > loss_mel_ce: 2.292210817337036 (nan) | > loss: 0.009192668832838535 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1369164285309818) | > loader_time: 0.0045 (0.004768913674427472)  --> TIME: 2025-02-27 04:49:17 -- STEP: 704/36938 -- GLOBAL_STEP: 1552100 | > loss_text_ce: 0.02922794036567211 (0.02976779971636874) | > loss_mel_ce: 2.520460367202759 (nan) | > loss: 0.010117811150848866 (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.1372830796648155) | > loader_time: 0.0049 (0.0047657537189397)  --> TIME: 2025-02-27 04:49:26 -- STEP: 754/36938 -- GLOBAL_STEP: 1552150 | > loss_text_ce: 0.03301708400249481 (0.029773411912897934) | > loss_mel_ce: 2.820844888687134 (nan) | > loss: 0.011324849911034107 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13721032326038068) | > loader_time: 0.0045 (0.004760119579831239)  --> TIME: 2025-02-27 04:49:36 -- STEP: 804/36938 -- GLOBAL_STEP: 1552200 | > loss_text_ce: 0.03151995688676834 (0.029745551934743774) | > loss_mel_ce: 2.6937785148620605 (nan) | > loss: 0.010814676992595196 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13737715951245816) | > loader_time: 0.0046 (0.004752863105849847)  --> TIME: 2025-02-27 04:49:45 -- STEP: 854/36938 -- GLOBAL_STEP: 1552250 | > loss_text_ce: 0.03818973898887634 (0.029844137505176685) | > loss_mel_ce: 2.4141762256622314 (nan) | > loss: 0.009731611236929893 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.13737560407339275) | > loader_time: 0.0046 (0.004747910856921448)  --> TIME: 2025-02-27 04:49:55 -- STEP: 904/36938 -- GLOBAL_STEP: 1552300 | > loss_text_ce: 0.038306210190057755 (0.029899444046792928) | > loss_mel_ce: 2.4697253704071045 (nan) | > loss: 0.009952506981790066 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.1374280982840378) | > loader_time: 0.0049 (0.004741634942788998)  --> TIME: 2025-02-27 04:50:04 -- STEP: 954/36938 -- GLOBAL_STEP: 1552350 | > loss_text_ce: 0.03036470338702202 (0.029895345451001788) | > loss_mel_ce: 2.6692140102386475 (nan) | > loss: 0.010712615214288235 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13739542541263983) | > loader_time: 0.0046 (0.004737599340874698)  --> TIME: 2025-02-27 04:50:13 -- STEP: 1004/36938 -- GLOBAL_STEP: 1552400 | > loss_text_ce: 0.03248359262943268 (0.02988142294965061) | > loss_mel_ce: 2.730207681655884 (nan) | > loss: 0.010963060893118382 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.1373864939488262) | > loader_time: 0.0046 (0.004732050268773542)  --> TIME: 2025-02-27 04:50:23 -- STEP: 1054/36938 -- GLOBAL_STEP: 1552450 | > loss_text_ce: 0.03503235802054405 (0.02988256359441061) | > loss_mel_ce: 2.9253766536712646 (nan) | > loss: 0.01174765545874834 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13747611624918576) | > loader_time: 0.0046 (0.004732012748718265)  --> TIME: 2025-02-27 04:50:32 -- STEP: 1104/36938 -- GLOBAL_STEP: 1552500 | > loss_text_ce: 0.029240800067782402 (0.029926916201382985) | > loss_mel_ce: 2.3991482257843018 (nan) | > loss: 0.009636465460062027 (nan) | > current_lr: 5e-06 | > step_time: 0.1864 (0.1374538620744926) | > loader_time: 0.0055 (0.004731005948522821)  --> TIME: 2025-02-27 04:50:42 -- STEP: 1154/36938 -- GLOBAL_STEP: 1552550 | > loss_text_ce: 0.033559225499629974 (0.029976102239428093) | > loss_mel_ce: 1.5877598524093628 (nan) | > loss: 0.006433805916458368 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.1374894107854965) | > loader_time: 0.0045 (0.004730988003352284)  --> TIME: 2025-02-27 04:50:51 -- STEP: 1204/36938 -- GLOBAL_STEP: 1552600 | > loss_text_ce: 0.025397542864084244 (0.029920722820495946) | > loss_mel_ce: 2.9727590084075928 (nan) | > loss: 0.011897447519004345 (nan) | > current_lr: 5e-06 | > step_time: 0.1798 (0.1375990395134074) | > loader_time: 0.0048 (0.004733775145191691)  --> TIME: 2025-02-27 04:51:00 -- STEP: 1254/36938 -- GLOBAL_STEP: 1552650 | > loss_text_ce: 0.03341910243034363 (0.029904240457136095) | > loss_mel_ce: 3.0421719551086426 (nan) | > loss: 0.012204727157950401 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13733744583251564) | > loader_time: 0.0046 (0.004729653279366872)  --> TIME: 2025-02-27 04:51:10 -- STEP: 1304/36938 -- GLOBAL_STEP: 1552700 | > loss_text_ce: 0.025819003582000732 (0.029885637325354698) | > loss_mel_ce: 2.4332895278930664 (nan) | > loss: 0.009758368134498596 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13742412178794294) | > loader_time: 0.0045 (0.0047277506509441625)  --> TIME: 2025-02-27 04:51:19 -- STEP: 1354/36938 -- GLOBAL_STEP: 1552750 | > loss_text_ce: 0.03474986553192139 (0.029863038440090514) | > loss_mel_ce: 0.0026092529296875 (nan) | > loss: 0.00014825047401245683 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13729145150064886) | > loader_time: 0.0044 (0.004722257308452845)  --> TIME: 2025-02-27 04:51:28 -- STEP: 1404/36938 -- GLOBAL_STEP: 1552800 | > loss_text_ce: 0.026693999767303467 (0.029915109749066183) | > loss_mel_ce: 2.5457255840301514 (nan) | > loss: 0.010208015330135822 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13737445118760094) | > loader_time: 0.0046 (0.004723122486701388)  --> TIME: 2025-02-27 04:51:38 -- STEP: 1454/36938 -- GLOBAL_STEP: 1552850 | > loss_text_ce: 0.025693118572235107 (0.02991467499990206) | > loss_mel_ce: 1.061187744140625 (nan) | > loss: 0.004313019569963217 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13749046830724304) | > loader_time: 0.0045 (0.0047274073675541385)  --> TIME: 2025-02-27 04:51:47 -- STEP: 1504/36938 -- GLOBAL_STEP: 1552900 | > loss_text_ce: 0.03350762277841568 (0.029875349002627734) | > loss_mel_ce: 1.6858419179916382 (nan) | > loss: 0.0068228160962462425 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13750159661186478) | > loader_time: 0.0045 (0.004726816840628369)  --> TIME: 2025-02-27 04:51:57 -- STEP: 1554/36938 -- GLOBAL_STEP: 1552950 | > loss_text_ce: 0.02673068642616272 (0.029878259120458685) | > loss_mel_ce: 2.74606990814209 (nan) | > loss: 0.011003177613019943 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.1375451629533115) | > loader_time: 0.0047 (0.004723374294344837)  --> TIME: 2025-02-27 04:52:06 -- STEP: 1604/36938 -- GLOBAL_STEP: 1553000 | > loss_text_ce: 0.022431552410125732 (0.029883909647113776) | > loss_mel_ce: 3.126091957092285 (nan) | > loss: 0.012494142167270184 (nan) | > current_lr: 5e-06 | > step_time: 0.1977 (0.13760268465241868) | > loader_time: 0.0047 (0.004719897398627614)  --> TIME: 2025-02-27 04:52:16 -- STEP: 1654/36938 -- GLOBAL_STEP: 1553050 | > loss_text_ce: 0.02983735501766205 (0.02986922691392621) | > loss_mel_ce: 2.749359130859375 (nan) | > loss: 0.011028558015823364 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.137643074758684) | > loader_time: 0.0048 (0.004718836302336364)  --> TIME: 2025-02-27 04:52:25 -- STEP: 1704/36938 -- GLOBAL_STEP: 1553100 | > loss_text_ce: 0.02845640666782856 (0.029850396567610193) | > loss_mel_ce: 3.0435538291931152 (nan) | > loss: 0.012190517969429493 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13766409967426943) | > loader_time: 0.0046 (0.00472294766578317)  --> TIME: 2025-02-27 04:52:34 -- STEP: 1754/36938 -- GLOBAL_STEP: 1553150 | > loss_text_ce: 0.028429588302969933 (0.02981293356543069) | > loss_mel_ce: 2.7151501178741455 (nan) | > loss: 0.010887221433222294 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13754436648965818) | > loader_time: 0.0045 (0.004720194309847937)  --> TIME: 2025-02-27 04:52:44 -- STEP: 1804/36938 -- GLOBAL_STEP: 1553200 | > loss_text_ce: 0.024912768974900246 (0.029805723080585014) | > loss_mel_ce: 0.7148551940917969 (nan) | > loss: 0.002935587428510189 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1376409422267566) | > loader_time: 0.0045 (0.004718317425171719)  --> TIME: 2025-02-27 04:52:53 -- STEP: 1854/36938 -- GLOBAL_STEP: 1553250 | > loss_text_ce: 0.024324001744389534 (0.029804062430305808) | > loss_mel_ce: 2.3215529918670654 (nan) | > loss: 0.00930903572589159 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13769713047381477) | > loader_time: 0.0045 (0.004715574597849437)  --> TIME: 2025-02-27 04:53:03 -- STEP: 1904/36938 -- GLOBAL_STEP: 1553300 | > loss_text_ce: 0.03192465379834175 (0.029801123485481758) | > loss_mel_ce: 2.6847312450408936 (nan) | > loss: 0.01078038103878498 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13770861447859203) | > loader_time: 0.0046 (0.004713528171306906)  --> TIME: 2025-02-27 04:53:12 -- STEP: 1954/36938 -- GLOBAL_STEP: 1553350 | > loss_text_ce: 0.038547102361917496 (0.029828165256977845) | > loss_mel_ce: 0.2503153383731842 (nan) | > loss: 0.0011462796246632934 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13768125202663103) | > loader_time: 0.0045 (0.004712550032468274)  --> TIME: 2025-02-27 04:53:22 -- STEP: 2004/36938 -- GLOBAL_STEP: 1553400 | > loss_text_ce: 0.036546189337968826 (0.029848668419434402) | > loss_mel_ce: 3.2424938678741455 (nan) | > loss: 0.01301206462085247 (nan) | > current_lr: 5e-06 | > step_time: 0.1434 (0.13782415858761746) | > loader_time: 0.0045 (0.004709475173683707)  --> TIME: 2025-02-27 04:53:31 -- STEP: 2054/36938 -- GLOBAL_STEP: 1553450 | > loss_text_ce: 0.02943074144423008 (0.02986434276261981) | > loss_mel_ce: 2.4656691551208496 (nan) | > loss: 0.009901190176606178 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1378022042314404) | > loader_time: 0.0045 (0.0047064451999534376)  --> TIME: 2025-02-27 04:53:40 -- STEP: 2104/36938 -- GLOBAL_STEP: 1553500 | > loss_text_ce: 0.026289986446499825 (0.029873075271768936) | > loss_mel_ce: 2.5366063117980957 (nan) | > loss: 0.010170224122703075 (nan) | > current_lr: 5e-06 | > step_time: 0.151 (0.13771028641058916) | > loader_time: 0.0046 (0.004706178327930295)  --> TIME: 2025-02-27 04:53:49 -- STEP: 2154/36938 -- GLOBAL_STEP: 1553550 | > loss_text_ce: 0.02708769589662552 (0.029904744581785478) | > loss_mel_ce: 2.4062464237213135 (nan) | > loss: 0.009656088426709175 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.137581223122146) | > loader_time: 0.0045 (0.004705819689574463)  --> TIME: 2025-02-27 04:53:58 -- STEP: 2204/36938 -- GLOBAL_STEP: 1553600 | > loss_text_ce: 0.03504180908203125 (0.029936843639587853) | > loss_mel_ce: 0.6707342863082886 (nan) | > loss: 0.002800698857754469 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.1375227332547875) | > loader_time: 0.0045 (0.0047028583969697816)  --> TIME: 2025-02-27 04:54:08 -- STEP: 2254/36938 -- GLOBAL_STEP: 1553650 | > loss_text_ce: 0.029336513951420784 (0.02993664227032698) | > loss_mel_ce: 2.1469690799713135 (nan) | > loss: 0.00863613374531269 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.1375603847199027) | > loader_time: 0.0045 (0.004702388658498176)  --> TIME: 2025-02-27 04:54:18 -- STEP: 2304/36938 -- GLOBAL_STEP: 1553700 | > loss_text_ce: 0.027134036645293236 (0.02993144173564764) | > loss_mel_ce: 3.1695754528045654 (nan) | > loss: 0.012685355730354786 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13764428957882857) | > loader_time: 0.0051 (0.0047055831592943955)  --> TIME: 2025-02-27 04:54:27 -- STEP: 2354/36938 -- GLOBAL_STEP: 1553750 | > loss_text_ce: 0.03478085994720459 (0.029941359231408327) | > loss_mel_ce: 2.7103681564331055 (nan) | > loss: 0.010893449187278748 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13758462550990544) | > loader_time: 0.0046 (0.004705316755056585)  --> TIME: 2025-02-27 04:54:36 -- STEP: 2404/36938 -- GLOBAL_STEP: 1553800 | > loss_text_ce: 0.029795078560709953 (0.029960746787792182) | > loss_mel_ce: 2.7247703075408936 (nan) | > loss: 0.01093081571161747 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13742172033735198) | > loader_time: 0.0045 (0.004702803695856436)  --> TIME: 2025-02-27 04:54:45 -- STEP: 2454/36938 -- GLOBAL_STEP: 1553850 | > loss_text_ce: 0.026464853435754776 (0.029964003290680476) | > loss_mel_ce: 2.5566866397857666 (nan) | > loss: 0.010250601917505264 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13736711679286062) | > loader_time: 0.0045 (0.004701683359993315)  --> TIME: 2025-02-27 04:54:54 -- STEP: 2504/36938 -- GLOBAL_STEP: 1553900 | > loss_text_ce: 0.02649698406457901 (0.02995932791025017) | > loss_mel_ce: 2.434215784072876 (nan) | > loss: 0.009764733724296093 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.13734467122882307) | > loader_time: 0.0046 (0.004699244571570016)  --> TIME: 2025-02-27 04:55:04 -- STEP: 2554/36938 -- GLOBAL_STEP: 1553950 | > loss_text_ce: 0.028426924720406532 (0.029953054202021843) | > loss_mel_ce: 1.6703262329101562 (nan) | > loss: 0.006741084158420563 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.13738702178094747) | > loader_time: 0.0045 (0.004697375775503764)  --> TIME: 2025-02-27 04:55:13 -- STEP: 2604/36938 -- GLOBAL_STEP: 1554000 | > loss_text_ce: 0.02507851831614971 (0.029965608067432852) | > loss_mel_ce: 1.5032836198806763 (nan) | > loss: 0.006064929533749819 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.1373556583158431) | > loader_time: 0.0045 (0.004696071788828863)  --> TIME: 2025-02-27 04:55:23 -- STEP: 2654/36938 -- GLOBAL_STEP: 1554050 | > loss_text_ce: 0.026558024808764458 (0.02998663693001127) | > loss_mel_ce: 1.572348952293396 (nan) | > loss: 0.006344869267195463 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13748681248806438) | > loader_time: 0.0046 (0.004696792916597598)  --> TIME: 2025-02-27 04:55:33 -- STEP: 2704/36938 -- GLOBAL_STEP: 1554100 | > loss_text_ce: 0.025753432884812355 (0.0299697772630066) | > loss_mel_ce: 2.6916558742523193 (nan) | > loss: 0.010783371515572071 (nan) | > current_lr: 5e-06 | > step_time: 0.1693 (0.13761540251018037) | > loader_time: 0.0048 (0.00469612193530833)  --> TIME: 2025-02-27 04:55:42 -- STEP: 2754/36938 -- GLOBAL_STEP: 1554150 | > loss_text_ce: 0.02554703876376152 (0.029970380680509768) | > loss_mel_ce: 2.5556886196136475 (nan) | > loss: 0.010242999531328678 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.13763793101757443) | > loader_time: 0.0048 (0.00469683795712872)  --> TIME: 2025-02-27 04:55:51 -- STEP: 2804/36938 -- GLOBAL_STEP: 1554200 | > loss_text_ce: 0.015478708781301975 (0.029985978310268776) | > loss_mel_ce: 1.4994628429412842 (nan) | > loss: 0.006011673249304295 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.13759797564926907) | > loader_time: 0.0044 (0.004696614221907548)  --> TIME: 2025-02-27 04:56:01 -- STEP: 2854/36938 -- GLOBAL_STEP: 1554250 | > loss_text_ce: 0.019994333386421204 (0.029996514921132065) | > loss_mel_ce: 2.084699869155884 (nan) | > loss: 0.008351961150765419 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13763490179325205) | > loader_time: 0.0045 (0.004696125740259784)  --> TIME: 2025-02-27 04:56:10 -- STEP: 2904/36938 -- GLOBAL_STEP: 1554300 | > loss_text_ce: 0.031469058245420456 (0.02999219328156074) | > loss_mel_ce: 2.497546672821045 (nan) | > loss: 0.01003577746450901 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.1375840839111445) | > loader_time: 0.0053 (0.004695612126473887)  --> TIME: 2025-02-27 04:56:20 -- STEP: 2954/36938 -- GLOBAL_STEP: 1554350 | > loss_text_ce: 0.03957005962729454 (0.029995368869588045) | > loss_mel_ce: 2.3102943897247314 (nan) | > loss: 0.009324859827756882 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13766549807104955) | > loader_time: 0.0076 (0.00469523091209686)  --> TIME: 2025-02-27 04:56:29 -- STEP: 3004/36938 -- GLOBAL_STEP: 1554400 | > loss_text_ce: 0.034264519810676575 (0.029979827129573478) | > loss_mel_ce: 3.0230655670166016 (nan) | > loss: 0.01213226281106472 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13763347764148554) | > loader_time: 0.0046 (0.004695294223993666)  --> TIME: 2025-02-27 04:56:38 -- STEP: 3054/36938 -- GLOBAL_STEP: 1554450 | > loss_text_ce: 0.030857037752866745 (0.029978917146524862) | > loss_mel_ce: 2.943066120147705 (nan) | > loss: 0.011801282875239849 (nan) | > current_lr: 5e-06 | > step_time: 0.187 (0.1375815047886767) | > loader_time: 0.0047 (0.004694036666130663)  --> TIME: 2025-02-27 04:56:48 -- STEP: 3104/36938 -- GLOBAL_STEP: 1554500 | > loss_text_ce: 0.022000083699822426 (0.029972171198190234) | > loss_mel_ce: 2.4538018703460693 (nan) | > loss: 0.009824611246585846 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13762586463972473) | > loader_time: 0.0046 (0.004693570210761634)  --> TIME: 2025-02-27 04:56:57 -- STEP: 3154/36938 -- GLOBAL_STEP: 1554550 | > loss_text_ce: 0.04242995008826256 (0.029992055614138126) | > loss_mel_ce: 2.556844711303711 (nan) | > loss: 0.010314582847058773 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.1377024425308239) | > loader_time: 0.0055 (0.0046940981472453226)  --> TIME: 2025-02-27 04:57:07 -- STEP: 3204/36938 -- GLOBAL_STEP: 1554600 | > loss_text_ce: 0.02971380390226841 (0.03000166407523539) | > loss_mel_ce: 3.202728509902954 (nan) | > loss: 0.012827153317630291 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.1376796780751739) | > loader_time: 0.0046 (0.004694249969296689)  --> TIME: 2025-02-27 04:57:16 -- STEP: 3254/36938 -- GLOBAL_STEP: 1554650 | > loss_text_ce: 0.031303148716688156 (0.02999915165217626) | > loss_mel_ce: 2.641498327255249 (nan) | > loss: 0.010606355965137482 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13767679688677023) | > loader_time: 0.0046 (0.004693291563891015)  --> TIME: 2025-02-27 04:57:26 -- STEP: 3304/36938 -- GLOBAL_STEP: 1554700 | > loss_text_ce: 0.02463398315012455 (0.029986055744597662) | > loss_mel_ce: 2.0719211101531982 (nan) | > loss: 0.008319662883877754 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13771002979601876) | > loader_time: 0.0053 (0.004693724557793454)  --> TIME: 2025-02-27 04:57:35 -- STEP: 3354/36938 -- GLOBAL_STEP: 1554750 | > loss_text_ce: 0.03622094541788101 (0.029991624267915115) | > loss_mel_ce: 3.172654867172241 (nan) | > loss: 0.012733635492622852 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.13770251467459704) | > loader_time: 0.0049 (0.004692869166497603)  --> TIME: 2025-02-27 04:57:44 -- STEP: 3404/36938 -- GLOBAL_STEP: 1554800 | > loss_text_ce: 0.028737420216202736 (0.02999928460120454) | > loss_mel_ce: 2.699079751968384 (nan) | > loss: 0.010824671946465969 (nan) | > current_lr: 5e-06 | > step_time: 0.169 (0.1377065896847837) | > loader_time: 0.0046 (0.004691994218232341)  --> TIME: 2025-02-27 04:57:54 -- STEP: 3454/36938 -- GLOBAL_STEP: 1554850 | > loss_text_ce: 0.037140049040317535 (0.030005101099951144) | > loss_mel_ce: 2.51216197013855 (nan) | > loss: 0.010116279125213623 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13772934954775257) | > loader_time: 0.0045 (0.004692685279669476)  --> TIME: 2025-02-27 04:58:03 -- STEP: 3504/36938 -- GLOBAL_STEP: 1554900 | > loss_text_ce: 0.03413774073123932 (0.030007787021972047) | > loss_mel_ce: 2.2774405479431152 (nan) | > loss: 0.009172930382192135 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13770023056361255) | > loader_time: 0.0045 (0.004691562764176489)  --> TIME: 2025-02-27 04:58:12 -- STEP: 3554/36938 -- GLOBAL_STEP: 1554950 | > loss_text_ce: 0.02889416553080082 (0.03002188725556316) | > loss_mel_ce: 2.3971517086029053 (nan) | > loss: 0.009627167135477066 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13766091367671132) | > loader_time: 0.0045 (0.004691406858054581)  --> TIME: 2025-02-27 04:58:22 -- STEP: 3604/36938 -- GLOBAL_STEP: 1555000 | > loss_text_ce: 0.032168950885534286 (0.030032863455166113) | > loss_mel_ce: 2.702744483947754 (nan) | > loss: 0.010852831415832043 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.13773018002642404) | > loader_time: 0.0047 (0.004692480183071084)  --> TIME: 2025-02-27 04:58:31 -- STEP: 3654/36938 -- GLOBAL_STEP: 1555050 | > loss_text_ce: 0.03956898674368858 (0.030022816161345226) | > loss_mel_ce: 2.6604812145233154 (nan) | > loss: 0.010714485310018063 (nan) | > current_lr: 5e-06 | > step_time: 0.1565 (0.13766355650243586) | > loader_time: 0.0046 (0.00469143689494349)  --> TIME: 2025-02-27 04:58:40 -- STEP: 3704/36938 -- GLOBAL_STEP: 1555100 | > loss_text_ce: 0.019451210275292397 (0.030019448937588723) | > loss_mel_ce: 0.1576557159423828 (nan) | > loss: 0.0007028053514659405 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13769286420381374) | > loader_time: 0.0045 (0.004690102573089923)  --> TIME: 2025-02-27 04:58:50 -- STEP: 3754/36938 -- GLOBAL_STEP: 1555150 | > loss_text_ce: 0.03203192353248596 (0.030010725514190224) | > loss_mel_ce: 2.9358770847320557 (nan) | > loss: 0.011777417734265327 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.13770945985455707) | > loader_time: 0.0048 (0.004689251735099905)  --> TIME: 2025-02-27 04:58:59 -- STEP: 3804/36938 -- GLOBAL_STEP: 1555200 | > loss_text_ce: 0.027857903391122818 (0.029988664171299046) | > loss_mel_ce: 1.585166573524475 (nan) | > loss: 0.006400891114026308 (nan) | > current_lr: 5e-06 | > step_time: 0.1625 (0.13771663404037549) | > loader_time: 0.0046 (0.004688294402683064)  --> TIME: 2025-02-27 04:59:08 -- STEP: 3854/36938 -- GLOBAL_STEP: 1555250 | > loss_text_ce: 0.0235504861921072 (0.030004406117577515) | > loss_mel_ce: 2.8207850456237793 (nan) | > loss: 0.01128704659640789 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.13768677162146678) | > loader_time: 0.0048 (0.004688901485528233)  --> TIME: 2025-02-27 04:59:18 -- STEP: 3904/36938 -- GLOBAL_STEP: 1555300 | > loss_text_ce: 0.029148044064641 (0.030004216142172434) | > loss_mel_ce: 2.700687885284424 (nan) | > loss: 0.010832683183252811 (nan) | > current_lr: 5e-06 | > step_time: 0.1852 (0.1377199346535521) | > loader_time: 0.0048 (0.004688054262126073)  --> TIME: 2025-02-27 04:59:27 -- STEP: 3954/36938 -- GLOBAL_STEP: 1555350 | > loss_text_ce: 0.028977252542972565 (0.029998335010076732) | > loss_mel_ce: 3.879380464553833 (nan) | > loss: 0.015509356744587421 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13771033877969474) | > loader_time: 0.0045 (0.004686988901476934)  --> TIME: 2025-02-27 04:59:37 -- STEP: 4004/36938 -- GLOBAL_STEP: 1555400 | > loss_text_ce: 0.03185286745429039 (0.030002136818262616) | > loss_mel_ce: 2.72454833984375 (nan) | > loss: 0.010938101448118687 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1376841712545803) | > loader_time: 0.0046 (0.004688620626866872)  --> TIME: 2025-02-27 04:59:46 -- STEP: 4054/36938 -- GLOBAL_STEP: 1555450 | > loss_text_ce: 0.026176268234848976 (0.030005207041082747) | > loss_mel_ce: 2.810748815536499 (nan) | > loss: 0.011257640086114407 (nan) | > current_lr: 5e-06 | > step_time: 0.1873 (0.13770092839258458) | > loader_time: 0.0048 (0.004688441077064392)  --> TIME: 2025-02-27 04:59:55 -- STEP: 4104/36938 -- GLOBAL_STEP: 1555500 | > loss_text_ce: 0.021241992712020874 (0.030009058804966536) | > loss_mel_ce: 2.602444887161255 (nan) | > loss: 0.010411456227302551 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13768336885621463) | > loader_time: 0.0048 (0.00468858327084814)  --> TIME: 2025-02-27 05:00:05 -- STEP: 4154/36938 -- GLOBAL_STEP: 1555550 | > loss_text_ce: 0.03199388459324837 (0.030016281626998417) | > loss_mel_ce: 1.3037075996398926 (nan) | > loss: 0.005300403106957674 (nan) | > current_lr: 5e-06 | > step_time: 0.1262 (0.1377164735550715) | > loader_time: 0.0046 (0.004690157489014401)  --> TIME: 2025-02-27 05:00:14 -- STEP: 4204/36938 -- GLOBAL_STEP: 1555600 | > loss_text_ce: 0.029058026149868965 (0.030019641905929095) | > loss_mel_ce: 1.8169066905975342 (nan) | > loss: 0.007325257174670696 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13767457853375356) | > loader_time: 0.0046 (0.004689890923214226)  --> TIME: 2025-02-27 05:00:23 -- STEP: 4254/36938 -- GLOBAL_STEP: 1555650 | > loss_text_ce: 0.03846844285726547 (0.0300264490201726) | > loss_mel_ce: 3.088238000869751 (nan) | > loss: 0.012407565489411354 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.13761209202978517) | > loader_time: 0.0046 (0.004690019973410533)  --> TIME: 2025-02-27 05:00:33 -- STEP: 4304/36938 -- GLOBAL_STEP: 1555700 | > loss_text_ce: 0.029889196157455444 (0.03001773165976871) | > loss_mel_ce: 3.0369982719421387 (nan) | > loss: 0.012170189060270786 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13761493430704896) | > loader_time: 0.0047 (0.004689199995374143)  --> TIME: 2025-02-27 05:00:42 -- STEP: 4354/36938 -- GLOBAL_STEP: 1555750 | > loss_text_ce: 0.029574232175946236 (0.030031218213150274) | > loss_mel_ce: 3.0688276290893555 (nan) | > loss: 0.012295246124267578 (nan) | > current_lr: 5e-06 | > step_time: 0.1695 (0.13760406301828734) | > loader_time: 0.0047 (0.004689326240318759)  --> TIME: 2025-02-27 05:00:51 -- STEP: 4404/36938 -- GLOBAL_STEP: 1555800 | > loss_text_ce: 0.042272377759218216 (0.03003411230628635) | > loss_mel_ce: 2.4791858196258545 (nan) | > loss: 0.010005787014961243 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13755695772214352) | > loader_time: 0.0046 (0.004688094032557845)  --> TIME: 2025-02-27 05:01:01 -- STEP: 4454/36938 -- GLOBAL_STEP: 1555850 | > loss_text_ce: 0.03584190830588341 (0.030038524899158342) | > loss_mel_ce: 2.8293612003326416 (nan) | > loss: 0.011369854211807251 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.137629108895729) | > loader_time: 0.0045 (0.004687869232975958)  --> TIME: 2025-02-27 05:01:10 -- STEP: 4504/36938 -- GLOBAL_STEP: 1555900 | > loss_text_ce: 0.021969838067889214 (0.03004555079877562) | > loss_mel_ce: 0.0729573592543602 (nan) | > loss: 0.00037669524317607284 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1376443821837596) | > loader_time: 0.0048 (0.004687093661479373)  --> TIME: 2025-02-27 05:01:19 -- STEP: 4554/36938 -- GLOBAL_STEP: 1555950 | > loss_text_ce: 0.020663481205701828 (0.030048078566554035) | > loss_mel_ce: 0.12778472900390625 (nan) | > loss: 0.0005890802713111043 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13760942453661704) | > loader_time: 0.0045 (0.0046861269100224355)  --> TIME: 2025-02-27 05:01:28 -- STEP: 4604/36938 -- GLOBAL_STEP: 1556000 | > loss_text_ce: 0.03136993944644928 (0.030044818234320026) | > loss_mel_ce: 0.8894137144088745 (nan) | > loss: 0.003653903491795063 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13751874790307686) | > loader_time: 0.0046 (0.0046848757281498085)  --> TIME: 2025-02-27 05:01:37 -- STEP: 4654/36938 -- GLOBAL_STEP: 1556050 | > loss_text_ce: 0.03070921264588833 (0.030052978627073134) | > loss_mel_ce: 2.7179768085479736 (nan) | > loss: 0.010907485149800777 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.13743045420210231) | > loader_time: 0.0046 (0.004683995175166961)  --> TIME: 2025-02-27 05:01:46 -- STEP: 4704/36938 -- GLOBAL_STEP: 1556100 | > loss_text_ce: 0.03090090863406658 (0.03004341217436587) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13742247857406883) | > loader_time: 0.0045 (0.004683878906324604)  --> TIME: 2025-02-27 05:01:56 -- STEP: 4754/36938 -- GLOBAL_STEP: 1556150 | > loss_text_ce: 0.020877303555607796 (0.030041325373671476) | > loss_mel_ce: 2.471937894821167 (nan) | > loss: 0.009892124682664871 (nan) | > current_lr: 5e-06 | > step_time: 0.1451 (0.1374196371813511) | > loader_time: 0.0046 (0.004683805856074809)  --> TIME: 2025-02-27 05:02:05 -- STEP: 4804/36938 -- GLOBAL_STEP: 1556200 | > loss_text_ce: 0.025381721556186676 (0.030032643421183183) | > loss_mel_ce: 1.865047574043274 (nan) | > loss: 0.007501703687012196 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.137441886106598) | > loader_time: 0.0047 (0.00468383800179436)  --> TIME: 2025-02-27 05:02:15 -- STEP: 4854/36938 -- GLOBAL_STEP: 1556250 | > loss_text_ce: 0.030378451570868492 (0.030035276328557722) | > loss_mel_ce: 3.0192666053771973 (nan) | > loss: 0.012101766653358936 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.13743102791184653) | > loader_time: 0.0047 (0.004683228348526484)  --> TIME: 2025-02-27 05:02:24 -- STEP: 4904/36938 -- GLOBAL_STEP: 1556300 | > loss_text_ce: 0.033762067556381226 (0.030026157093758513) | > loss_mel_ce: 3.5135610103607178 (nan) | > loss: 0.01407667901366949 (nan) | > current_lr: 5e-06 | > step_time: 0.1876 (0.1374353640522893) | > loader_time: 0.0049 (0.004683110054709978)  --> TIME: 2025-02-27 05:02:33 -- STEP: 4954/36938 -- GLOBAL_STEP: 1556350 | > loss_text_ce: 0.025468504056334496 (0.030017692409230183) | > loss_mel_ce: 2.164034843444824 (nan) | > loss: 0.008688505738973618 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.1374333251807597) | > loader_time: 0.0046 (0.004682827582992658)  --> TIME: 2025-02-27 05:02:42 -- STEP: 5004/36938 -- GLOBAL_STEP: 1556400 | > loss_text_ce: 0.02231082133948803 (0.03001479458936351) | > loss_mel_ce: 2.8384313583374023 (nan) | > loss: 0.01135215163230896 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13739136330706006) | > loader_time: 0.0044 (0.0046827223279969745)  --> TIME: 2025-02-27 05:02:52 -- STEP: 5054/36938 -- GLOBAL_STEP: 1556450 | > loss_text_ce: 0.029296988621354103 (0.03000771657493353) | > loss_mel_ce: 3.1511082649230957 (nan) | > loss: 0.012620655819773674 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13744763630767823) | > loader_time: 0.0046 (0.004682888237440304)  --> TIME: 2025-02-27 05:03:01 -- STEP: 5104/36938 -- GLOBAL_STEP: 1556500 | > loss_text_ce: 0.027649952098727226 (0.030018958817440492) | > loss_mel_ce: 2.80645751953125 (nan) | > loss: 0.011246458627283573 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13745289242192948) | > loader_time: 0.0046 (0.004682483017257756)  --> TIME: 2025-02-27 05:03:11 -- STEP: 5154/36938 -- GLOBAL_STEP: 1556550 | > loss_text_ce: 0.03345702961087227 (0.030026133207269635) | > loss_mel_ce: 2.9923155307769775 (nan) | > loss: 0.012007034383714199 (nan) | > current_lr: 5e-06 | > step_time: 0.1716 (0.13741080142011278) | > loader_time: 0.0048 (0.004681859175549209)  --> TIME: 2025-02-27 05:03:20 -- STEP: 5204/36938 -- GLOBAL_STEP: 1556600 | > loss_text_ce: 0.027348410338163376 (0.030029922619613774) | > loss_mel_ce: 1.6998412609100342 (nan) | > loss: 0.00685392739251256 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13743265246905245) | > loader_time: 0.0045 (0.00468158414966779)  --> TIME: 2025-02-27 05:03:29 -- STEP: 5254/36938 -- GLOBAL_STEP: 1556650 | > loss_text_ce: 0.03364374488592148 (0.030041237772617177) | > loss_mel_ce: 2.5830941200256348 (nan) | > loss: 0.010383880697190762 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13742074065213983) | > loader_time: 0.0046 (0.004681387826159369)  --> TIME: 2025-02-27 05:03:39 -- STEP: 5304/36938 -- GLOBAL_STEP: 1556700 | > loss_text_ce: 0.031399622559547424 (0.030040427800450548) | > loss_mel_ce: 2.5505752563476562 (nan) | > loss: 0.010245933197438717 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13747223789156207) | > loader_time: 0.0046 (0.004682450812326833)  --> TIME: 2025-02-27 05:03:49 -- STEP: 5354/36938 -- GLOBAL_STEP: 1556750 | > loss_text_ce: 0.025829579681158066 (0.030043476016626325) | > loss_mel_ce: 2.271296977996826 (nan) | > loss: 0.009115582332015038 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1374893711714955) | > loader_time: 0.0053 (0.004683211974613156)  --> TIME: 2025-02-27 05:03:58 -- STEP: 5404/36938 -- GLOBAL_STEP: 1556800 | > loss_text_ce: 0.03522193431854248 (0.030049653296434022) | > loss_mel_ce: 2.772763729095459 (nan) | > loss: 0.011142801493406296 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.13751153792389154) | > loader_time: 0.0046 (0.004684038245174992)  --> TIME: 2025-02-27 05:04:07 -- STEP: 5454/36938 -- GLOBAL_STEP: 1556850 | > loss_text_ce: 0.027953313663601875 (0.03006667781281065) | > loss_mel_ce: 2.9302964210510254 (nan) | > loss: 0.011739087291061878 (nan) | > current_lr: 5e-06 | > step_time: 0.2024 (0.13750551483069065) | > loader_time: 0.0049 (0.004683458022595674)  --> TIME: 2025-02-27 05:04:17 -- STEP: 5504/36938 -- GLOBAL_STEP: 1556900 | > loss_text_ce: 0.022283339872956276 (0.03006115092437032) | > loss_mel_ce: 2.7726190090179443 (nan) | > loss: 0.011090883053839207 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.13757114155694597) | > loader_time: 0.0053 (0.004684322968471874)  --> TIME: 2025-02-27 05:04:27 -- STEP: 5554/36938 -- GLOBAL_STEP: 1556950 | > loss_text_ce: 0.03018302097916603 (0.030052519005816864) | > loss_mel_ce: 2.6452085971832275 (nan) | > loss: 0.010616634041070938 (nan) | > current_lr: 5e-06 | > step_time: 0.1871 (0.1375920131646958) | > loader_time: 0.0049 (0.0046854466038897214)  --> TIME: 2025-02-27 05:04:35 -- STEP: 5604/36938 -- GLOBAL_STEP: 1557000 | > loss_text_ce: 0.029469409957528114 (0.030054798962808958) | > loss_mel_ce: 2.74495530128479 (nan) | > loss: 0.011009622365236282 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13752689176929864) | > loader_time: 0.0045 (0.004685312998796846)  --> TIME: 2025-02-27 05:04:45 -- STEP: 5654/36938 -- GLOBAL_STEP: 1557050 | > loss_text_ce: 0.009405938908457756 (0.03006450919075999) | > loss_mel_ce: 0.5754547119140625 (nan) | > loss: 0.002320875646546483 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13752748593450942) | > loader_time: 0.0044 (0.004685703882447471)  --> TIME: 2025-02-27 05:04:55 -- STEP: 5704/36938 -- GLOBAL_STEP: 1557100 | > loss_text_ce: 0.036528099328279495 (0.030063844874512986) | > loss_mel_ce: 2.6404383182525635 (nan) | > loss: 0.010622883215546608 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13758870395002448) | > loader_time: 0.0046 (0.004686637897812461)  --> TIME: 2025-02-27 05:05:04 -- STEP: 5754/36938 -- GLOBAL_STEP: 1557150 | > loss_text_ce: 0.0346786268055439 (0.030073453673039527) | > loss_mel_ce: 2.634740114212036 (nan) | > loss: 0.010592932812869549 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.1375836615849183) | > loader_time: 0.0046 (0.004686324141445694)  --> TIME: 2025-02-27 05:05:13 -- STEP: 5804/36938 -- GLOBAL_STEP: 1557200 | > loss_text_ce: 0.03552992269396782 (0.030083632329639322) | > loss_mel_ce: 2.4696786403656006 (nan) | > loss: 0.009941304102540016 (nan) | > current_lr: 5e-06 | > step_time: 0.1704 (0.13758588502688165) | > loader_time: 0.0048 (0.004686876299626081)  --> TIME: 2025-02-27 05:05:23 -- STEP: 5854/36938 -- GLOBAL_STEP: 1557250 | > loss_text_ce: 0.037753455340862274 (0.03008402178403172) | > loss_mel_ce: 1.4337772130966187 (nan) | > loss: 0.0058394079096615314 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13755035608100802) | > loader_time: 0.0045 (0.0046875622234207595)  --> TIME: 2025-02-27 05:05:32 -- STEP: 5904/36938 -- GLOBAL_STEP: 1557300 | > loss_text_ce: 0.03874051943421364 (0.030086815882496898) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13754839524828752) | > loader_time: 0.0045 (0.00468851710722698)  --> TIME: 2025-02-27 05:05:41 -- STEP: 5954/36938 -- GLOBAL_STEP: 1557350 | > loss_text_ce: 0.024954168125987053 (0.03009044642750703) | > loss_mel_ce: 3.313981771469116 (nan) | > loss: 0.013249746523797512 (nan) | > current_lr: 5e-06 | > step_time: 0.1343 (0.13754219439432636) | > loader_time: 0.0046 (0.004689975316627817)  --> TIME: 2025-02-27 05:05:51 -- STEP: 6004/36938 -- GLOBAL_STEP: 1557400 | > loss_text_ce: 0.0325414203107357 (0.030102586489404087) | > loss_mel_ce: 2.8541743755340576 (nan) | > loss: 0.01145522203296423 (nan) | > current_lr: 5e-06 | > step_time: 0.1563 (0.13758314176053799) | > loader_time: 0.0047 (0.004690439084781793)  --> TIME: 2025-02-27 05:06:01 -- STEP: 6054/36938 -- GLOBAL_STEP: 1557450 | > loss_text_ce: 0.0302912425249815 (0.030091539328671108) | > loss_mel_ce: 1.163782000541687 (nan) | > loss: 0.0047383857890963554 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1376230621400911) | > loader_time: 0.0046 (0.004690263859698748)  --> TIME: 2025-02-27 05:06:10 -- STEP: 6104/36938 -- GLOBAL_STEP: 1557500 | > loss_text_ce: 0.035385143011808395 (0.030079396093706082) | > loss_mel_ce: 2.769583225250244 (nan) | > loss: 0.011130827479064465 (nan) | > current_lr: 5e-06 | > step_time: 0.1261 (0.13760017743273234) | > loader_time: 0.0046 (0.004689922768666578)  --> TIME: 2025-02-27 05:06:19 -- STEP: 6154/36938 -- GLOBAL_STEP: 1557550 | > loss_text_ce: 0.03074844554066658 (0.030084180434984766) | > loss_mel_ce: 2.7550861835479736 (nan) | > loss: 0.011054899543523788 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13758076976792333) | > loader_time: 0.0047 (0.004689799875340603)  --> TIME: 2025-02-27 05:06:28 -- STEP: 6204/36938 -- GLOBAL_STEP: 1557600 | > loss_text_ce: 0.025514552369713783 (0.030081285463392657) | > loss_mel_ce: 1.6239389181137085 (nan) | > loss: 0.006545451004058123 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.1375531495040198) | > loader_time: 0.0045 (0.0046895230531231345)  --> TIME: 2025-02-27 05:06:38 -- STEP: 6254/36938 -- GLOBAL_STEP: 1557650 | > loss_text_ce: 0.03313117101788521 (0.030075171016166205) | > loss_mel_ce: 2.0588037967681885 (nan) | > loss: 0.008301329798996449 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13755901793073463) | > loader_time: 0.0046 (0.004689199229875693)  --> TIME: 2025-02-27 05:06:47 -- STEP: 6304/36938 -- GLOBAL_STEP: 1557700 | > loss_text_ce: 0.02739483304321766 (0.030080432669427434) | > loss_mel_ce: 2.998763084411621 (nan) | > loss: 0.012008563615381718 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.1375572657600281) | > loader_time: 0.0046 (0.004688779828209565)  --> TIME: 2025-02-27 05:06:56 -- STEP: 6354/36938 -- GLOBAL_STEP: 1557750 | > loss_text_ce: 0.027382265776395798 (0.03007697585388563) | > loss_mel_ce: 2.78232479095459 (nan) | > loss: 0.011149631813168526 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.1375377039733766) | > loader_time: 0.0048 (0.004688181215237628)  --> TIME: 2025-02-27 05:07:06 -- STEP: 6404/36938 -- GLOBAL_STEP: 1557800 | > loss_text_ce: 0.035127270966768265 (0.030074730566816216) | > loss_mel_ce: 0.00372314453125 (nan) | > loss: 0.00015416833048220724 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13754784405641093) | > loader_time: 0.0045 (0.00468771908895289)  --> TIME: 2025-02-27 05:07:15 -- STEP: 6454/36938 -- GLOBAL_STEP: 1557850 | > loss_text_ce: 0.03556248918175697 (0.030072877392505283) | > loss_mel_ce: 2.8815460205078125 (nan) | > loss: 0.011575828306376934 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13752376157727011) | > loader_time: 0.0046 (0.004687141810624933)  --> TIME: 2025-02-27 05:07:24 -- STEP: 6504/36938 -- GLOBAL_STEP: 1557900 | > loss_text_ce: 0.025963418185710907 (0.030081020733587864) | > loss_mel_ce: 2.5596442222595215 (nan) | > loss: 0.010260348208248615 (nan) | > current_lr: 5e-06 | > step_time: 0.1455 (0.13754907521667886) | > loader_time: 0.0046 (0.0046873658638891995)  --> TIME: 2025-02-27 05:07:34 -- STEP: 6554/36938 -- GLOBAL_STEP: 1557950 | > loss_text_ce: 0.031055811792612076 (0.030084264386663548) | > loss_mel_ce: 1.8578341007232666 (nan) | > loss: 0.007495595142245293 (nan) | > current_lr: 5e-06 | > step_time: 0.1752 (0.1375326957868527) | > loader_time: 0.0048 (0.004687698832509866)  --> TIME: 2025-02-27 05:07:44 -- STEP: 6604/36938 -- GLOBAL_STEP: 1558000 | > loss_text_ce: 0.04395672678947449 (0.03008627098365075) | > loss_mel_ce: 2.7458314895629883 (nan) | > loss: 0.01107058860361576 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13757588479246038) | > loader_time: 0.0045 (0.004687689962277337)  --> TIME: 2025-02-27 05:07:53 -- STEP: 6654/36938 -- GLOBAL_STEP: 1558050 | > loss_text_ce: 0.02749083749949932 (0.03008726944414625) | > loss_mel_ce: 2.623013734817505 (nan) | > loss: 0.010517875663936138 (nan) | > current_lr: 5e-06 | > step_time: 0.1965 (0.13756166236838535) | > loader_time: 0.0049 (0.004687974823463565)  --> TIME: 2025-02-27 05:08:02 -- STEP: 6704/36938 -- GLOBAL_STEP: 1558100 | > loss_text_ce: 0.03948353976011276 (0.030084679222739077) | > loss_mel_ce: 1.7332247495651245 (nan) | > loss: 0.007034556940197945 (nan) | > current_lr: 5e-06 | > step_time: 0.1261 (0.1375001545524263) | > loader_time: 0.0046 (0.0046876674975870935)  --> TIME: 2025-02-27 05:08:11 -- STEP: 6754/36938 -- GLOBAL_STEP: 1558150 | > loss_text_ce: 0.021554788574576378 (0.030087521039103927) | > loss_mel_ce: 2.6503164768218994 (nan) | > loss: 0.010602664202451706 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.1374613090771567) | > loader_time: 0.0052 (0.004688225062175161)  --> TIME: 2025-02-27 05:08:20 -- STEP: 6804/36938 -- GLOBAL_STEP: 1558200 | > loss_text_ce: 0.0302654430270195 (0.0300889639131865) | > loss_mel_ce: 2.5256755352020264 (nan) | > loss: 0.010142623446881771 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13748840736263665) | > loader_time: 0.0046 (0.004688245383940473)  --> TIME: 2025-02-27 05:08:29 -- STEP: 6854/36938 -- GLOBAL_STEP: 1558250 | > loss_text_ce: 0.04315798357129097 (0.030094843150312443) | > loss_mel_ce: 1.1266357898712158 (nan) | > loss: 0.004642039071768522 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13746892786735718) | > loader_time: 0.0052 (0.004688539378386327)  --> TIME: 2025-02-27 05:08:39 -- STEP: 6904/36938 -- GLOBAL_STEP: 1558300 | > loss_text_ce: 0.02723412774503231 (0.03009519262224768) | > loss_mel_ce: 2.8101539611816406 (nan) | > loss: 0.011259476654231548 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13745923701194698) | > loader_time: 0.0047 (0.00468873508006922)  --> TIME: 2025-02-27 05:08:49 -- STEP: 6954/36938 -- GLOBAL_STEP: 1558350 | > loss_text_ce: 0.02856510318815708 (0.03010497808465528) | > loss_mel_ce: 2.4726719856262207 (nan) | > loss: 0.009925545193254948 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.13751059532302773) | > loader_time: 0.0047 (0.004689300475121922)  --> TIME: 2025-02-27 05:08:58 -- STEP: 7004/36938 -- GLOBAL_STEP: 1558400 | > loss_text_ce: 0.03508772701025009 (0.03010438495104987) | > loss_mel_ce: 2.7888176441192627 (nan) | > loss: 0.011205974966287613 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.137498253383071) | > loader_time: 0.0046 (0.004689635275977471)  --> TIME: 2025-02-27 05:09:07 -- STEP: 7054/36938 -- GLOBAL_STEP: 1558450 | > loss_text_ce: 0.03728428855538368 (0.030100988850487606) | > loss_mel_ce: 2.739680767059326 (nan) | > loss: 0.011019703932106495 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1374726197791912) | > loader_time: 0.0054 (0.004690733177654466)  --> TIME: 2025-02-27 05:09:17 -- STEP: 7104/36938 -- GLOBAL_STEP: 1558500 | > loss_text_ce: 0.025992609560489655 (0.030095900616321194) | > loss_mel_ce: 2.640627384185791 (nan) | > loss: 0.010581825859844685 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1374962059025837) | > loader_time: 0.0046 (0.004691040327957091)  --> TIME: 2025-02-27 05:09:26 -- STEP: 7154/36938 -- GLOBAL_STEP: 1558550 | > loss_text_ce: 0.03545546531677246 (0.03010418321854006) | > loss_mel_ce: 2.717000961303711 (nan) | > loss: 0.010922446846961975 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13748930804983486) | > loader_time: 0.0046 (0.004690660055969432)  --> TIME: 2025-02-27 05:09:35 -- STEP: 7204/36938 -- GLOBAL_STEP: 1558600 | > loss_text_ce: 0.03847885876893997 (0.030107413263844616) | > loss_mel_ce: 3.2535719871520996 (nan) | > loss: 0.013063694350421429 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.1374939569561175) | > loader_time: 0.0046 (0.0046901577449652496)  --> TIME: 2025-02-27 05:09:45 -- STEP: 7254/36938 -- GLOBAL_STEP: 1558650 | > loss_text_ce: 0.026213092729449272 (0.03011044727095028) | > loss_mel_ce: 2.120327949523926 (nan) | > loss: 0.008518020622432232 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13751303801235565) | > loader_time: 0.0045 (0.004690833350729284)  --> TIME: 2025-02-27 05:09:54 -- STEP: 7304/36938 -- GLOBAL_STEP: 1558700 | > loss_text_ce: 0.02152959257364273 (0.030115400687209967) | > loss_mel_ce: 0.010515849106013775 (nan) | > loss: 0.0001271644578082487 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.137504731316165) | > loader_time: 0.0045 (0.00469044542913239)  --> TIME: 2025-02-27 05:10:03 -- STEP: 7354/36938 -- GLOBAL_STEP: 1558750 | > loss_text_ce: 0.027606235817074776 (0.030115780297125774) | > loss_mel_ce: 3.099184274673462 (nan) | > loss: 0.012407899834215641 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13750065206217063) | > loader_time: 0.0045 (0.00469007896021848)  --> TIME: 2025-02-27 05:10:13 -- STEP: 7404/36938 -- GLOBAL_STEP: 1558800 | > loss_text_ce: 0.02358320727944374 (0.030114012623562007) | > loss_mel_ce: 1.2342591285705566 (nan) | > loss: 0.004991437774151564 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.1375012067380569) | > loader_time: 0.0044 (0.004689668688112055)  --> TIME: 2025-02-27 05:10:22 -- STEP: 7454/36938 -- GLOBAL_STEP: 1558850 | > loss_text_ce: 0.025689691305160522 (0.030107095326077835) | > loss_mel_ce: 2.6202468872070312 (nan) | > loss: 0.010499748401343822 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1375058397454814) | > loader_time: 0.0046 (0.004689331153202293)  --> TIME: 2025-02-27 05:10:31 -- STEP: 7504/36938 -- GLOBAL_STEP: 1558900 | > loss_text_ce: 0.022347671911120415 (0.030106895609693263) | > loss_mel_ce: 0.8345046639442444 (nan) | > loss: 0.0034002079628407955 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13745827086444654) | > loader_time: 0.0045 (0.004689318602527388)  --> TIME: 2025-02-27 05:10:41 -- STEP: 7554/36938 -- GLOBAL_STEP: 1558950 | > loss_text_ce: 0.03105812892317772 (0.03010018210244622) | > loss_mel_ce: 2.7645926475524902 (nan) | > loss: 0.011093853041529655 (nan) | > current_lr: 5e-06 | > step_time: 0.1668 (0.13749595704077464) | > loader_time: 0.0046 (0.004689222168663622)  --> TIME: 2025-02-27 05:10:50 -- STEP: 7604/36938 -- GLOBAL_STEP: 1559000 | > loss_text_ce: 0.03350337967276573 (0.03010506680879991) | > loss_mel_ce: 2.710383415222168 (nan) | > loss: 0.010888439603149891 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13748957165788864) | > loader_time: 0.0046 (0.004689303716190992)  --> TIME: 2025-02-27 05:10:59 -- STEP: 7654/36938 -- GLOBAL_STEP: 1559050 | > loss_text_ce: 0.03792013227939606 (0.030101011485311094) | > loss_mel_ce: 2.9447641372680664 (nan) | > loss: 0.01183605007827282 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.1374527358934035) | > loader_time: 0.0054 (0.00469050810484716)  --> TIME: 2025-02-27 05:11:08 -- STEP: 7704/36938 -- GLOBAL_STEP: 1559100 | > loss_text_ce: 0.026209043338894844 (0.030100131082432664) | > loss_mel_ce: 2.9485950469970703 (nan) | > loss: 0.011804779060184956 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1374290358364154) | > loader_time: 0.0047 (0.004690999399216876)  --> TIME: 2025-02-27 05:11:17 -- STEP: 7754/36938 -- GLOBAL_STEP: 1559150 | > loss_text_ce: 0.030574850738048553 (0.030101718366874426) | > loss_mel_ce: 2.6757497787475586 (nan) | > loss: 0.010739384219050407 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13741727747928295) | > loader_time: 0.0046 (0.0046911530805857915)  --> TIME: 2025-02-27 05:11:27 -- STEP: 7804/36938 -- GLOBAL_STEP: 1559200 | > loss_text_ce: 0.02845255471765995 (0.030099434513042082) | > loss_mel_ce: 2.5548226833343506 (nan) | > loss: 0.010251092724502087 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.1374010878731084) | > loader_time: 0.0046 (0.004691047982519439)  --> TIME: 2025-02-27 05:11:36 -- STEP: 7854/36938 -- GLOBAL_STEP: 1559250 | > loss_text_ce: 0.030201958492398262 (0.030106093474932497) | > loss_mel_ce: 1.934975266456604 (nan) | > loss: 0.007798322476446629 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13738901025623035) | > loader_time: 0.0046 (0.004691464408590868)  --> TIME: 2025-02-27 05:11:45 -- STEP: 7904/36938 -- GLOBAL_STEP: 1559300 | > loss_text_ce: 0.028623925521969795 (0.030102693921059694) | > loss_mel_ce: 2.4860334396362305 (nan) | > loss: 0.009978799149394035 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1373228651248979) | > loader_time: 0.0046 (0.00469197996473506)  --> TIME: 2025-02-27 05:11:54 -- STEP: 7954/36938 -- GLOBAL_STEP: 1559350 | > loss_text_ce: 0.03077210672199726 (0.030103215586763134) | > loss_mel_ce: 2.6930599212646484 (nan) | > loss: 0.010808858089148998 (nan) | > current_lr: 5e-06 | > step_time: 0.1605 (0.13732310689691674) | > loader_time: 0.0057 (0.004692425852538899)  --> TIME: 2025-02-27 05:12:03 -- STEP: 8004/36938 -- GLOBAL_STEP: 1559400 | > loss_text_ce: 0.03493325784802437 (0.03010230176446346) | > loss_mel_ce: 2.9536380767822266 (nan) | > loss: 0.011859411373734474 (nan) | > current_lr: 5e-06 | > step_time: 0.1832 (0.13731484583531112) | > loader_time: 0.0048 (0.004693210810080347)  --> TIME: 2025-02-27 05:12:13 -- STEP: 8054/36938 -- GLOBAL_STEP: 1559450 | > loss_text_ce: 0.027509188279509544 (0.030099046485383377) | > loss_mel_ce: 2.2229275703430176 (nan) | > loss: 0.008930305019021034 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13730536677685262) | > loader_time: 0.0046 (0.0046939709241614665)  --> TIME: 2025-02-27 05:12:22 -- STEP: 8104/36938 -- GLOBAL_STEP: 1559500 | > loss_text_ce: 0.025412332266569138 (0.030103618905420793) | > loss_mel_ce: 2.8726234436035156 (nan) | > loss: 0.011500142514705658 (nan) | > current_lr: 5e-06 | > step_time: 0.1629 (0.13731994436500386) | > loader_time: 0.0053 (0.004694505922886137)  --> TIME: 2025-02-27 05:12:31 -- STEP: 8154/36938 -- GLOBAL_STEP: 1559550 | > loss_text_ce: 0.02609136514365673 (0.030101650007533287) | > loss_mel_ce: 0.06396484375 (nan) | > loss: 0.00035736593417823315 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13731167911167486) | > loader_time: 0.0047 (0.0046946781652945115)  --> TIME: 2025-02-27 05:12:41 -- STEP: 8204/36938 -- GLOBAL_STEP: 1559600 | > loss_text_ce: 0.041172467172145844 (0.03009540308471215) | > loss_mel_ce: 2.751221179962158 (nan) | > loss: 0.011080928146839142 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13729214473563528) | > loader_time: 0.0046 (0.004695213085148527)  --> TIME: 2025-02-27 05:12:50 -- STEP: 8254/36938 -- GLOBAL_STEP: 1559650 | > loss_text_ce: 0.027357280254364014 (0.030094114554446143) | > loss_mel_ce: 2.870842695236206 (nan) | > loss: 0.011500794440507889 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1373242443688769) | > loader_time: 0.0046 (0.004695237910513562)  --> TIME: 2025-02-27 05:13:00 -- STEP: 8304/36938 -- GLOBAL_STEP: 1559700 | > loss_text_ce: 0.02375761978328228 (0.030101923287042233) | > loss_mel_ce: 3.068035125732422 (nan) | > loss: 0.012269020080566406 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13735508778076913) | > loader_time: 0.0046 (0.004695837437991696)  --> TIME: 2025-02-27 05:13:09 -- STEP: 8354/36938 -- GLOBAL_STEP: 1559750 | > loss_text_ce: 0.04044705629348755 (0.030104733434315575) | > loss_mel_ce: 2.1808016300201416 (nan) | > loss: 0.008814479224383831 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13732386206165054) | > loader_time: 0.0046 (0.004695887653514308)  --> TIME: 2025-02-27 05:13:18 -- STEP: 8404/36938 -- GLOBAL_STEP: 1559800 | > loss_text_ce: 0.03692738339304924 (0.03010243893825298) | > loss_mel_ce: 2.9499707221984863 (nan) | > loss: 0.011852771043777466 (nan) | > current_lr: 5e-06 | > step_time: 0.1923 (0.13730477421468737) | > loader_time: 0.0048 (0.004695618453336753)  --> TIME: 2025-02-27 05:13:27 -- STEP: 8454/36938 -- GLOBAL_STEP: 1559850 | > loss_text_ce: 0.026288531720638275 (0.030102619041252717) | > loss_mel_ce: 2.39859676361084 (nan) | > loss: 0.009622560814023018 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1372837210085088) | > loader_time: 0.0045 (0.004695103132953199)  --> TIME: 2025-02-27 05:13:37 -- STEP: 8504/36938 -- GLOBAL_STEP: 1559900 | > loss_text_ce: 0.029112493619322777 (0.030103781630056416) | > loss_mel_ce: 2.1111812591552734 (nan) | > loss: 0.008493229746818542 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13728336239915284) | > loader_time: 0.0045 (0.0046944707940461405)  --> TIME: 2025-02-27 05:13:46 -- STEP: 8554/36938 -- GLOBAL_STEP: 1559950 | > loss_text_ce: 0.031052980571985245 (0.030104917551175035) | > loss_mel_ce: 1.7879871129989624 (nan) | > loss: 0.007218413520604372 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13728051044017306) | > loader_time: 0.0046 (0.0046944723860638336)  --> TIME: 2025-02-27 05:13:55 -- STEP: 8604/36938 -- GLOBAL_STEP: 1560000 | > loss_text_ce: 0.027593715116381645 (0.03010735274580455) | > loss_mel_ce: 2.954446315765381 (nan) | > loss: 0.011833492666482925 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13727798383217132) | > loader_time: 0.0046 (0.00469488143810058) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1560000.pth  --> TIME: 2025-02-27 05:14:11 -- STEP: 8654/36938 -- GLOBAL_STEP: 1560050 | > loss_text_ce: 0.027727458626031876 (0.030107900417647243) | > loss_mel_ce: 2.6890134811401367 (nan) | > loss: 0.010780718177556992 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.1372653012852048) | > loader_time: 0.0052 (0.00469563154472279)  --> TIME: 2025-02-27 05:14:19 -- STEP: 8704/36938 -- GLOBAL_STEP: 1560100 | > loss_text_ce: 0.02518879994750023 (0.03010358642517103) | > loss_mel_ce: 0.026531219482421875 (nan) | > loss: 0.00020523818966466933 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13721788225366832) | > loader_time: 0.0052 (0.004695884883403779)  --> TIME: 2025-02-27 05:14:29 -- STEP: 8754/36938 -- GLOBAL_STEP: 1560150 | > loss_text_ce: 0.037864282727241516 (0.03010497518162089) | > loss_mel_ce: 3.0796961784362793 (nan) | > loss: 0.012371271848678589 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.1372162749676342) | > loader_time: 0.0046 (0.004696477704160363)  --> TIME: 2025-02-27 05:14:37 -- STEP: 8804/36938 -- GLOBAL_STEP: 1560200 | > loss_text_ce: 0.04074690118432045 (0.030108645367522852) | > loss_mel_ce: 2.178663492202759 (nan) | > loss: 0.008807185105979443 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13717344369416065) | > loader_time: 0.0045 (0.004696633478881331)  --> TIME: 2025-02-27 05:14:47 -- STEP: 8854/36938 -- GLOBAL_STEP: 1560250 | > loss_text_ce: 0.022948751226067543 (0.030116980546180377) | > loss_mel_ce: 2.871454954147339 (nan) | > loss: 0.011485729366540909 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13719272007650338) | > loader_time: 0.0047 (0.004696781866324786)  --> TIME: 2025-02-27 05:14:56 -- STEP: 8904/36938 -- GLOBAL_STEP: 1560300 | > loss_text_ce: 0.031084703281521797 (0.03012015061726608) | > loss_mel_ce: 2.109286308288574 (nan) | > loss: 0.008493536151945591 (nan) | > current_lr: 5e-06 | > step_time: 0.151 (0.13717000675115773) | > loader_time: 0.0047 (0.004696539764669915)  --> TIME: 2025-02-27 05:15:05 -- STEP: 8954/36938 -- GLOBAL_STEP: 1560350 | > loss_text_ce: 0.026864180341362953 (0.03012231409862581) | > loss_mel_ce: 2.228616237640381 (nan) | > loss: 0.008950320072472095 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13715021602804478) | > loader_time: 0.0046 (0.004696110702397057)  --> TIME: 2025-02-27 05:15:15 -- STEP: 9004/36938 -- GLOBAL_STEP: 1560400 | > loss_text_ce: 0.03269116207957268 (0.03012147725314425) | > loss_mel_ce: 1.1427905559539795 (nan) | > loss: 0.004664610140025616 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13717925180492826) | > loader_time: 0.0045 (0.004696537499319754)  --> TIME: 2025-02-27 05:15:24 -- STEP: 9054/36938 -- GLOBAL_STEP: 1560450 | > loss_text_ce: 0.03175473213195801 (0.030126861479999063) | > loss_mel_ce: 2.4685487747192383 (nan) | > loss: 0.009921839460730553 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.1371773882946592) | > loader_time: 0.0046 (0.004697085111536719)  --> TIME: 2025-02-27 05:15:34 -- STEP: 9104/36938 -- GLOBAL_STEP: 1560500 | > loss_text_ce: 0.020956775173544884 (0.030123733450341087) | > loss_mel_ce: 0.00624847412109375 (nan) | > loss: 0.0001079573412425816 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13721484961953884) | > loader_time: 0.0045 (0.004697684689649166)  --> TIME: 2025-02-27 05:15:43 -- STEP: 9154/36938 -- GLOBAL_STEP: 1560550 | > loss_text_ce: 0.03009161166846752 (0.0301206570250113) | > loss_mel_ce: 2.3975062370300293 (nan) | > loss: 0.009633325040340424 (nan) | > current_lr: 5e-06 | > step_time: 0.1875 (0.13720527574540464) | > loader_time: 0.0049 (0.004697739596134561)  --> TIME: 2025-02-27 05:15:52 -- STEP: 9204/36938 -- GLOBAL_STEP: 1560600 | > loss_text_ce: 0.023871958255767822 (0.030122325739061144) | > loss_mel_ce: 2.6629116535186768 (nan) | > loss: 0.010661840438842773 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1371718490968215) | > loader_time: 0.0045 (0.004698078044856541)  --> TIME: 2025-02-27 05:16:01 -- STEP: 9254/36938 -- GLOBAL_STEP: 1560650 | > loss_text_ce: 0.029486769810318947 (0.030125068692659133) | > loss_mel_ce: 2.9807844161987305 (nan) | > loss: 0.011945520527660847 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13715514212827457) | > loader_time: 0.0045 (0.004698097615667621)  --> TIME: 2025-02-27 05:16:11 -- STEP: 9304/36938 -- GLOBAL_STEP: 1560700 | > loss_text_ce: 0.03164505958557129 (0.030127294764490452) | > loss_mel_ce: 3.0064682960510254 (nan) | > loss: 0.012056006118655205 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.13714909771343617) | > loader_time: 0.0046 (0.004698477064035923)  --> TIME: 2025-02-27 05:16:20 -- STEP: 9354/36938 -- GLOBAL_STEP: 1560750 | > loss_text_ce: 0.03019176982343197 (0.030131811765634894) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13713798791597062) | > loader_time: 0.0045 (0.004698743696826522)  --> TIME: 2025-02-27 05:16:29 -- STEP: 9404/36938 -- GLOBAL_STEP: 1560800 | > loss_text_ce: 0.03610328957438469 (0.03013890532244699) | > loss_mel_ce: 2.897660970687866 (nan) | > loss: 0.011641922406852245 (nan) | > current_lr: 5e-06 | > step_time: 0.1747 (0.13714100598579068) | > loader_time: 0.0048 (0.004699648922425736)  --> TIME: 2025-02-27 05:16:39 -- STEP: 9454/36938 -- GLOBAL_STEP: 1560850 | > loss_text_ce: 0.03738746792078018 (0.03014105739507778) | > loss_mel_ce: 3.166982650756836 (nan) | > loss: 0.01271575503051281 (nan) | > current_lr: 5e-06 | > step_time: 0.1841 (0.13714541701955776) | > loader_time: 0.0049 (0.004700273243851851)  --> TIME: 2025-02-27 05:16:48 -- STEP: 9504/36938 -- GLOBAL_STEP: 1560900 | > loss_text_ce: 0.030997635796666145 (0.030140644353101036) | > loss_mel_ce: 2.336005449295044 (nan) | > loss: 0.009392870590090752 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13713406628430483) | > loader_time: 0.0045 (0.004700762304392715)  --> TIME: 2025-02-27 05:16:57 -- STEP: 9554/36938 -- GLOBAL_STEP: 1560950 | > loss_text_ce: 0.019465049728751183 (0.03013275522580388) | > loss_mel_ce: 1.7987611293792725 (nan) | > loss: 0.0072151836939156055 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13709430582782375) | > loader_time: 0.0045 (0.004700425406342787)  --> TIME: 2025-02-27 05:17:06 -- STEP: 9604/36938 -- GLOBAL_STEP: 1561000 | > loss_text_ce: 0.037508442997932434 (0.030134913924191674) | > loss_mel_ce: 1.8674260377883911 (nan) | > loss: 0.0075592645443975925 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1370742733604656) | > loader_time: 0.0044 (0.004699867474739858)  --> TIME: 2025-02-27 05:17:15 -- STEP: 9654/36938 -- GLOBAL_STEP: 1561050 | > loss_text_ce: 0.02785669080913067 (0.03013398883550177) | > loss_mel_ce: 3.0301740169525146 (nan) | > loss: 0.012135042808949947 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13709264723242548) | > loader_time: 0.0046 (0.004699891834049594)  --> TIME: 2025-02-27 05:17:25 -- STEP: 9704/36938 -- GLOBAL_STEP: 1561100 | > loss_text_ce: 0.028548240661621094 (0.030136890142295366) | > loss_mel_ce: 3.048422336578369 (nan) | > loss: 0.012210201472043991 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13707708115923628) | > loader_time: 0.0047 (0.004699968176249704)  --> TIME: 2025-02-27 05:17:33 -- STEP: 9754/36938 -- GLOBAL_STEP: 1561150 | > loss_text_ce: 0.030593182891607285 (0.030137642867338416) | > loss_mel_ce: 2.251979351043701 (nan) | > loss: 0.009057828225195408 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13704221207562198) | > loader_time: 0.0048 (0.004700336418167485)  --> TIME: 2025-02-27 05:17:43 -- STEP: 9804/36938 -- GLOBAL_STEP: 1561200 | > loss_text_ce: 0.030051572248339653 (0.030144073656824373) | > loss_mel_ce: 2.1821627616882324 (nan) | > loss: 0.008778627961874008 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13705410307059715) | > loader_time: 0.0045 (0.004700429728448452)  --> TIME: 2025-02-27 05:17:53 -- STEP: 9854/36938 -- GLOBAL_STEP: 1561250 | > loss_text_ce: 0.0394548662006855 (0.03014740678134898) | > loss_mel_ce: 2.3437752723693848 (nan) | > loss: 0.00945726316422224 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.1370864691152663) | > loader_time: 0.0046 (0.004700901882387634)  --> TIME: 2025-02-27 05:18:02 -- STEP: 9904/36938 -- GLOBAL_STEP: 1561300 | > loss_text_ce: 0.03043505921959877 (0.030148353897917545) | > loss_mel_ce: 2.657008647918701 (nan) | > loss: 0.010664460249245167 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13709446111384033) | > loader_time: 0.0046 (0.004700932055952468)  --> TIME: 2025-02-27 05:18:11 -- STEP: 9954/36938 -- GLOBAL_STEP: 1561350 | > loss_text_ce: 0.03452574834227562 (0.03014913718003734) | > loss_mel_ce: 2.318180561065674 (nan) | > loss: 0.009336136281490326 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13706752663279534) | > loader_time: 0.0045 (0.0047014323683515005)  --> TIME: 2025-02-27 05:18:21 -- STEP: 10004/36938 -- GLOBAL_STEP: 1561400 | > loss_text_ce: 0.03324951231479645 (0.030156613127036137) | > loss_mel_ce: 2.17828369140625 (nan) | > loss: 0.008775926195085049 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13707749770193858) | > loader_time: 0.0045 (0.004701291070562131)  --> TIME: 2025-02-27 05:18:30 -- STEP: 10054/36938 -- GLOBAL_STEP: 1561450 | > loss_text_ce: 0.023945655673742294 (0.03015544835010461) | > loss_mel_ce: 2.0209763050079346 (nan) | > loss: 0.0081147700548172 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.1370692287023532) | > loader_time: 0.0046 (0.004700893907108767)  --> TIME: 2025-02-27 05:18:39 -- STEP: 10104/36938 -- GLOBAL_STEP: 1561500 | > loss_text_ce: 0.031100310385227203 (0.030149774207152508) | > loss_mel_ce: 2.859628915786743 (nan) | > loss: 0.011471148580312729 (nan) | > current_lr: 5e-06 | > step_time: 0.169 (0.1370771814459199) | > loader_time: 0.0047 (0.00470106281275232)  --> TIME: 2025-02-27 05:18:48 -- STEP: 10154/36938 -- GLOBAL_STEP: 1561550 | > loss_text_ce: 0.017587987706065178 (0.03014913195764702) | > loss_mel_ce: 2.3719568252563477 (nan) | > loss: 0.009482321329414845 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13705992083486573) | > loader_time: 0.0046 (0.004701020282296794)  --> TIME: 2025-02-27 05:18:58 -- STEP: 10204/36938 -- GLOBAL_STEP: 1561600 | > loss_text_ce: 0.026115013286471367 (0.030146948562318737) | > loss_mel_ce: 2.5712552070617676 (nan) | > loss: 0.010307025164365768 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13704532097480376) | > loader_time: 0.0047 (0.004701206727104531)  --> TIME: 2025-02-27 05:19:07 -- STEP: 10254/36938 -- GLOBAL_STEP: 1561650 | > loss_text_ce: 0.022304141893982887 (0.030143919640467927) | > loss_mel_ce: 2.305100679397583 (nan) | > loss: 0.009235733188688755 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.1370442745070349) | > loader_time: 0.0044 (0.004701148912598333)  --> TIME: 2025-02-27 05:19:16 -- STEP: 10304/36938 -- GLOBAL_STEP: 1561700 | > loss_text_ce: 0.02844727598130703 (0.03014267362550809) | > loss_mel_ce: 2.5996460914611816 (nan) | > loss: 0.010428942739963531 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.1370373636900638) | > loader_time: 0.0047 (0.004700800068445089)  --> TIME: 2025-02-27 05:19:26 -- STEP: 10354/36938 -- GLOBAL_STEP: 1561750 | > loss_text_ce: 0.029461031779646873 (0.03014069399639017) | > loss_mel_ce: 2.9258017539978027 (nan) | > loss: 0.011727234348654747 (nan) | > current_lr: 5e-06 | > step_time: 0.194 (0.13705740869333097) | > loader_time: 0.0049 (0.004700906976254398)  --> TIME: 2025-02-27 05:19:35 -- STEP: 10404/36938 -- GLOBAL_STEP: 1561800 | > loss_text_ce: 0.023472685366868973 (0.03013587538213804) | > loss_mel_ce: 2.430109739303589 (nan) | > loss: 0.0097364392131567 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.13704432170696318) | > loader_time: 0.0044 (0.004700485329039505)  --> TIME: 2025-02-27 05:19:45 -- STEP: 10454/36938 -- GLOBAL_STEP: 1561850 | > loss_text_ce: 0.01877100020647049 (0.030130400088789793) | > loss_mel_ce: 1.5283081531524658 (nan) | > loss: 0.006139203440397978 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13706047906231553) | > loader_time: 0.0046 (0.004700272790738403)  --> TIME: 2025-02-27 05:19:54 -- STEP: 10504/36938 -- GLOBAL_STEP: 1561900 | > loss_text_ce: 0.04186393320560455 (0.030133263243437493) | > loss_mel_ce: 2.8723397254943848 (nan) | > loss: 0.011564300395548344 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13707067877031512) | > loader_time: 0.0046 (0.004700441489397617)  --> TIME: 2025-02-27 05:20:03 -- STEP: 10554/36938 -- GLOBAL_STEP: 1561950 | > loss_text_ce: 0.029681062325835228 (0.030139031338542333) | > loss_mel_ce: 2.3518238067626953 (nan) | > loss: 0.009450416080653667 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.13705095328547318) | > loader_time: 0.0046 (0.00470081596905837)  --> TIME: 2025-02-27 05:20:12 -- STEP: 10604/36938 -- GLOBAL_STEP: 1562000 | > loss_text_ce: 0.02109622210264206 (0.03013203464418942) | > loss_mel_ce: 2.132662773132324 (nan) | > loss: 0.008546663448214531 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1370345446713598) | > loader_time: 0.0045 (0.004701028631031177)  --> TIME: 2025-02-27 05:20:21 -- STEP: 10654/36938 -- GLOBAL_STEP: 1562050 | > loss_text_ce: 0.03173748403787613 (0.0301272748062649) | > loss_mel_ce: 2.3218448162078857 (nan) | > loss: 0.009339612908661366 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.137025570721749) | > loader_time: 0.0053 (0.004700711772883616)  --> TIME: 2025-02-27 05:20:31 -- STEP: 10704/36938 -- GLOBAL_STEP: 1562100 | > loss_text_ce: 0.03966827318072319 (0.030129256019624916) | > loss_mel_ce: 2.455059766769409 (nan) | > loss: 0.009899714961647987 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13704253428007296) | > loader_time: 0.0045 (0.004701219242962669)  --> TIME: 2025-02-27 05:20:41 -- STEP: 10754/36938 -- GLOBAL_STEP: 1562150 | > loss_text_ce: 0.030534105375409126 (0.030128505068294384) | > loss_mel_ce: 2.300058603286743 (nan) | > loss: 0.009248384274542332 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13706592070650675) | > loader_time: 0.0046 (0.004702094852957606)  --> TIME: 2025-02-27 05:20:50 -- STEP: 10804/36938 -- GLOBAL_STEP: 1562200 | > loss_text_ce: 0.031193647533655167 (0.03012703501980411) | > loss_mel_ce: 2.6148011684417725 (nan) | > loss: 0.010499980300664902 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1370774690021813) | > loader_time: 0.0045 (0.004701904723574526)  --> TIME: 2025-02-27 05:21:00 -- STEP: 10854/36938 -- GLOBAL_STEP: 1562250 | > loss_text_ce: 0.023241760209202766 (0.03012771280497856) | > loss_mel_ce: 2.3276872634887695 (nan) | > loss: 0.009329084306955338 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13708854163134565) | > loader_time: 0.0046 (0.004702756675936712)  --> TIME: 2025-02-27 05:21:09 -- STEP: 10904/36938 -- GLOBAL_STEP: 1562300 | > loss_text_ce: 0.03552914410829544 (0.030125637735689083) | > loss_mel_ce: 3.0973658561706543 (nan) | > loss: 0.01243212353438139 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13708328601637068) | > loader_time: 0.0045 (0.004702941665558457)  --> TIME: 2025-02-27 05:21:19 -- STEP: 10954/36938 -- GLOBAL_STEP: 1562350 | > loss_text_ce: 0.029581278562545776 (0.03012913574837182) | > loss_mel_ce: 2.52584171295166 (nan) | > loss: 0.010140568017959595 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13710201165309172) | > loader_time: 0.0046 (0.004703352567551893)  --> TIME: 2025-02-27 05:21:28 -- STEP: 11004/36938 -- GLOBAL_STEP: 1562400 | > loss_text_ce: 0.030135881155729294 (0.0301287980360544) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.1371188817003433) | > loader_time: 0.0048 (0.0047031762772410755)  --> TIME: 2025-02-27 05:21:37 -- STEP: 11054/36938 -- GLOBAL_STEP: 1562450 | > loss_text_ce: 0.032176654785871506 (0.030132419786442786) | > loss_mel_ce: 2.108232021331787 (nan) | > loss: 0.008493686094880104 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13710751275980015) | > loader_time: 0.0046 (0.004702899152770578)  --> TIME: 2025-02-27 05:21:46 -- STEP: 11104/36938 -- GLOBAL_STEP: 1562500 | > loss_text_ce: 0.041248440742492676 (0.030134078429747503) | > loss_mel_ce: 1.0860382318496704 (nan) | > loss: 0.004473360255360603 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.1370875890814951) | > loader_time: 0.0046 (0.004702727543861106)  --> TIME: 2025-02-27 05:21:56 -- STEP: 11154/36938 -- GLOBAL_STEP: 1562550 | > loss_text_ce: 0.03556782752275467 (0.030130720595978315) | > loss_mel_ce: 2.6840429306030273 (nan) | > loss: 0.01079210638999939 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13707621088587146) | > loader_time: 0.0046 (0.00470277731705465)  --> TIME: 2025-02-27 05:22:05 -- STEP: 11204/36938 -- GLOBAL_STEP: 1562600 | > loss_text_ce: 0.03839012235403061 (0.030125177976758788) | > loss_mel_ce: 1.3486430644989014 (nan) | > loss: 0.0055041005834937096 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13706004270610117) | > loader_time: 0.0063 (0.004703006077051082)  --> TIME: 2025-02-27 05:22:14 -- STEP: 11254/36938 -- GLOBAL_STEP: 1562650 | > loss_text_ce: 0.03873683512210846 (0.030120723333156126) | > loss_mel_ce: 2.897439479827881 (nan) | > loss: 0.011651493608951569 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.1370718849262465) | > loader_time: 0.0045 (0.004703195200564705)  --> TIME: 2025-02-27 05:22:23 -- STEP: 11304/36938 -- GLOBAL_STEP: 1562700 | > loss_text_ce: 0.031653132289648056 (0.030121056231988432) | > loss_mel_ce: 2.992720365524292 (nan) | > loss: 0.012001482769846916 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.1370605082901675) | > loader_time: 0.0054 (0.004703455395674928)  --> TIME: 2025-02-27 05:22:33 -- STEP: 11354/36938 -- GLOBAL_STEP: 1562750 | > loss_text_ce: 0.04653779789805412 (0.03012615601800058) | > loss_mel_ce: 2.747887134552002 (nan) | > loss: 0.011088988743722439 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13707854616740708) | > loader_time: 0.0046 (0.004703907410569736)  --> TIME: 2025-02-27 05:22:42 -- STEP: 11404/36938 -- GLOBAL_STEP: 1562800 | > loss_text_ce: 0.0264369435608387 (0.030127303215362437) | > loss_mel_ce: 1.9925379753112793 (nan) | > loss: 0.008011805824935436 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13705958704578802) | > loader_time: 0.0047 (0.00470351612555358)  --> TIME: 2025-02-27 05:22:51 -- STEP: 11454/36938 -- GLOBAL_STEP: 1562850 | > loss_text_ce: 0.027526874095201492 (0.030128367585397546) | > loss_mel_ce: 3.191957712173462 (nan) | > loss: 0.012775733135640621 (nan) | > current_lr: 5e-06 | > step_time: 0.1884 (0.13705634839750827) | > loader_time: 0.005 (0.004703435428233723)  --> TIME: 2025-02-27 05:23:00 -- STEP: 11504/36938 -- GLOBAL_STEP: 1562900 | > loss_text_ce: 0.026839355006814003 (0.03012919966320077) | > loss_mel_ce: 1.9924217462539673 (nan) | > loss: 0.008012941107153893 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13703466359116606) | > loader_time: 0.0046 (0.004702913454411261)  --> TIME: 2025-02-27 05:23:10 -- STEP: 11554/36938 -- GLOBAL_STEP: 1562950 | > loss_text_ce: 0.03186223283410072 (0.030128231081911372) | > loss_mel_ce: 2.8540098667144775 (nan) | > loss: 0.011451873928308487 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.1370271755498814) | > loader_time: 0.0046 (0.00470251372183651)  --> TIME: 2025-02-27 05:23:19 -- STEP: 11604/36938 -- GLOBAL_STEP: 1563000 | > loss_text_ce: 0.024348439648747444 (0.030126191586214615) | > loss_mel_ce: 2.5835938453674316 (nan) | > loss: 0.010348978452384472 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13702509708134974) | > loader_time: 0.0046 (0.004702510298716125)  --> TIME: 2025-02-27 05:23:28 -- STEP: 11654/36938 -- GLOBAL_STEP: 1563050 | > loss_text_ce: 0.023200856521725655 (0.030130909636074246) | > loss_mel_ce: 3.4678237438201904 (nan) | > loss: 0.013853272423148155 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13700543602559964) | > loader_time: 0.0047 (0.0047023093411949855)  --> TIME: 2025-02-27 05:23:37 -- STEP: 11704/36938 -- GLOBAL_STEP: 1563100 | > loss_text_ce: 0.027893245220184326 (0.03012688640112462) | > loss_mel_ce: 2.8517322540283203 (nan) | > loss: 0.011427085846662521 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.1370063353220406) | > loader_time: 0.0046 (0.004702189566741964)  --> TIME: 2025-02-27 05:23:47 -- STEP: 11754/36938 -- GLOBAL_STEP: 1563150 | > loss_text_ce: 0.02899724431335926 (0.030130371527799564) | > loss_mel_ce: 1.454088807106018 (nan) | > loss: 0.005885262507945299 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13700606457043504) | > loader_time: 0.0046 (0.0047028081563534494)  --> TIME: 2025-02-27 05:23:56 -- STEP: 11804/36938 -- GLOBAL_STEP: 1563200 | > loss_text_ce: 0.02520790509879589 (0.030132832523709605) | > loss_mel_ce: 2.4276769161224365 (nan) | > loss: 0.009733670391142368 (nan) | > current_lr: 5e-06 | > step_time: 0.1918 (0.13702735354317858) | > loader_time: 0.0049 (0.004703203668759173)  --> TIME: 2025-02-27 05:24:06 -- STEP: 11854/36938 -- GLOBAL_STEP: 1563250 | > loss_text_ce: 0.02637120895087719 (0.030130996173863653) | > loss_mel_ce: 2.955449104309082 (nan) | > loss: 0.011832620948553085 (nan) | > current_lr: 5e-06 | > step_time: 0.1874 (0.1370191569872968) | > loader_time: 0.0056 (0.00470372012237966)  --> TIME: 2025-02-27 05:24:15 -- STEP: 11904/36938 -- GLOBAL_STEP: 1563300 | > loss_text_ce: 0.03892601281404495 (0.03013263414592779) | > loss_mel_ce: 0.6018415093421936 (nan) | > loss: 0.0025427283253520727 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13704846911532925) | > loader_time: 0.0046 (0.00470437676275289)  --> TIME: 2025-02-27 05:24:25 -- STEP: 11954/36938 -- GLOBAL_STEP: 1563350 | > loss_text_ce: 0.0289878249168396 (0.030131465930425306) | > loss_mel_ce: 2.9617581367492676 (nan) | > loss: 0.011868040077388287 (nan) | > current_lr: 5e-06 | > step_time: 0.144 (0.1370546238187448) | > loader_time: 0.0047 (0.004704811011785558)  --> TIME: 2025-02-27 05:24:35 -- STEP: 12004/36938 -- GLOBAL_STEP: 1563400 | > loss_text_ce: 0.031012823805212975 (0.03013631745199842) | > loss_mel_ce: 1.5398237705230713 (nan) | > loss: 0.006233478896319866 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13708508399993904) | > loader_time: 0.0045 (0.004705556290660522)  --> TIME: 2025-02-27 05:24:44 -- STEP: 12054/36938 -- GLOBAL_STEP: 1563450 | > loss_text_ce: 0.03279794007539749 (0.030138676776369142) | > loss_mel_ce: 2.117929697036743 (nan) | > loss: 0.008534634485840797 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13708416595177347) | > loader_time: 0.0047 (0.004706034222032092)  --> TIME: 2025-02-27 05:24:54 -- STEP: 12104/36938 -- GLOBAL_STEP: 1563500 | > loss_text_ce: 0.02984696626663208 (0.03014040184486565) | > loss_mel_ce: 2.8340232372283936 (nan) | > loss: 0.011364564299583435 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13709334533733025) | > loader_time: 0.0046 (0.0047062541267687465)  --> TIME: 2025-02-27 05:25:03 -- STEP: 12154/36938 -- GLOBAL_STEP: 1563550 | > loss_text_ce: 0.02709570899605751 (0.030136872237849836) | > loss_mel_ce: 2.8979806900024414 (nan) | > loss: 0.01160744670778513 (nan) | > current_lr: 5e-06 | > step_time: 0.172 (0.1371022657832564) | > loader_time: 0.0048 (0.004706618325655761)  --> TIME: 2025-02-27 05:25:12 -- STEP: 12204/36938 -- GLOBAL_STEP: 1563600 | > loss_text_ce: 0.029093096032738686 (0.030137838556273115) | > loss_mel_ce: 2.952449321746826 (nan) | > loss: 0.01183151826262474 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.1371002849380683) | > loader_time: 0.0046 (0.004706802328310416)  --> TIME: 2025-02-27 05:25:22 -- STEP: 12254/36938 -- GLOBAL_STEP: 1563650 | > loss_text_ce: 0.03321607783436775 (0.03013531583544035) | > loss_mel_ce: 2.380361318588257 (nan) | > loss: 0.00957768876105547 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13710416569607822) | > loader_time: 0.0048 (0.004706609904795267)  --> TIME: 2025-02-27 05:25:31 -- STEP: 12304/36938 -- GLOBAL_STEP: 1563700 | > loss_text_ce: 0.01590397022664547 (0.030135211991641236) | > loss_mel_ce: 2.882620096206665 (nan) | > loss: 0.011502080596983433 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13713414509755267) | > loader_time: 0.0046 (0.004706829108094366)  --> TIME: 2025-02-27 05:25:41 -- STEP: 12354/36938 -- GLOBAL_STEP: 1563750 | > loss_text_ce: 0.025457903742790222 (0.030132540422460823) | > loss_mel_ce: 2.654905319213867 (nan) | > loss: 0.010636362247169018 (nan) | > current_lr: 5e-06 | > step_time: 0.1632 (0.13715577472929466) | > loader_time: 0.0047 (0.004706709346706398)  --> TIME: 2025-02-27 05:25:50 -- STEP: 12404/36938 -- GLOBAL_STEP: 1563800 | > loss_text_ce: 0.017118731513619423 (0.030131582253428175) | > loss_mel_ce: 2.050994873046875 (nan) | > loss: 0.008206800557672977 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13714329587456034) | > loader_time: 0.0045 (0.0047065491101235144)  --> TIME: 2025-02-27 05:25:59 -- STEP: 12454/36938 -- GLOBAL_STEP: 1563850 | > loss_text_ce: 0.042734984308481216 (0.03013497816240354) | > loss_mel_ce: 2.2843499183654785 (nan) | > loss: 0.009234464727342129 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13712477655305108) | > loader_time: 0.0046 (0.004706644946812472)  --> TIME: 2025-02-27 05:26:09 -- STEP: 12504/36938 -- GLOBAL_STEP: 1563900 | > loss_text_ce: 0.026743806898593903 (0.03013544545671441) | > loss_mel_ce: 2.0952064990997314 (nan) | > loss: 0.008420438505709171 (nan) | > current_lr: 5e-06 | > step_time: 0.1343 (0.13713954258483746) | > loader_time: 0.0047 (0.0047066634233685805)  --> TIME: 2025-02-27 05:26:18 -- STEP: 12554/36938 -- GLOBAL_STEP: 1563950 | > loss_text_ce: 0.03311273828148842 (0.030132699615934972) | > loss_mel_ce: 2.7864491939544678 (nan) | > loss: 0.011188738979399204 (nan) | > current_lr: 5e-06 | > step_time: 0.1855 (0.13714555623293218) | > loader_time: 0.0048 (0.004706938023283843)  --> TIME: 2025-02-27 05:26:28 -- STEP: 12604/36938 -- GLOBAL_STEP: 1564000 | > loss_text_ce: 0.03167906031012535 (0.030130538725712693) | > loss_mel_ce: 2.006765365600586 (nan) | > loss: 0.008089066483080387 (nan) | > current_lr: 5e-06 | > step_time: 0.1819 (0.13715832905329003) | > loader_time: 0.0048 (0.004706859853448507)  --> TIME: 2025-02-27 05:26:37 -- STEP: 12654/36938 -- GLOBAL_STEP: 1564050 | > loss_text_ce: 0.027738753706216812 (0.030122922855416547) | > loss_mel_ce: 2.0808258056640625 (nan) | > loss: 0.008367320522665977 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13714553806027008) | > loader_time: 0.0046 (0.0047067900074777465)  --> TIME: 2025-02-27 05:26:46 -- STEP: 12704/36938 -- GLOBAL_STEP: 1564100 | > loss_text_ce: 0.01988782174885273 (0.030118301377204476) | > loss_mel_ce: 2.235013961791992 (nan) | > loss: 0.008948023431003094 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13714360494532585) | > loader_time: 0.0046 (0.004706629052270268)  --> TIME: 2025-02-27 05:26:56 -- STEP: 12754/36938 -- GLOBAL_STEP: 1564150 | > loss_text_ce: 0.032369744032621384 (0.030120108431375733) | > loss_mel_ce: 2.5878190994262695 (nan) | > loss: 0.010397575795650482 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.1371513803457232) | > loader_time: 0.0046 (0.004706795245142629)  --> TIME: 2025-02-27 05:27:05 -- STEP: 12804/36938 -- GLOBAL_STEP: 1564200 | > loss_text_ce: 0.02753356844186783 (0.03011737351561962) | > loss_mel_ce: 1.6549278497695923 (nan) | > loss: 0.00667643453925848 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1371656650865276) | > loader_time: 0.0045 (0.004706878898218997)  --> TIME: 2025-02-27 05:27:15 -- STEP: 12854/36938 -- GLOBAL_STEP: 1564250 | > loss_text_ce: 0.02289615012705326 (0.030110279955971195) | > loss_mel_ce: 0.0255126953125 (nan) | > loss: 0.0001920985960168764 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.1371521518893351) | > loader_time: 0.0047 (0.004707110397400598)  --> TIME: 2025-02-27 05:27:24 -- STEP: 12904/36938 -- GLOBAL_STEP: 1564300 | > loss_text_ce: 0.01699831523001194 (0.030108896668046836) | > loss_mel_ce: 0.0128173828125 (nan) | > loss: 0.00011831626761704683 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13717279425704995) | > loader_time: 0.0046 (0.004707086958356033)  --> TIME: 2025-02-27 05:27:34 -- STEP: 12954/36938 -- GLOBAL_STEP: 1564350 | > loss_text_ce: 0.024144599214196205 (0.03011223968591927) | > loss_mel_ce: 1.7859221696853638 (nan) | > loss: 0.007182805333286524 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.13718504204101478) | > loader_time: 0.0047 (0.004707317818309933)  --> TIME: 2025-02-27 05:27:43 -- STEP: 13004/36938 -- GLOBAL_STEP: 1564400 | > loss_text_ce: 0.029381271451711655 (0.030109057882699) | > loss_mel_ce: 2.5559310913085938 (nan) | > loss: 0.010259176604449749 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1371640937763228) | > loader_time: 0.0046 (0.004707607625998269)  --> TIME: 2025-02-27 05:27:52 -- STEP: 13054/36938 -- GLOBAL_STEP: 1564450 | > loss_text_ce: 0.026145318523049355 (0.03011082453423917) | > loss_mel_ce: 1.8109573125839233 (nan) | > loss: 0.007290090434253216 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13718171305617732) | > loader_time: 0.0045 (0.004708100409936597)  --> TIME: 2025-02-27 05:28:02 -- STEP: 13104/36938 -- GLOBAL_STEP: 1564500 | > loss_text_ce: 0.032470863312482834 (0.030112219446443045) | > loss_mel_ce: 2.1708927154541016 (nan) | > loss: 0.008743506856262684 (nan) | > current_lr: 5e-06 | > step_time: 0.1263 (0.13718054351716427) | > loader_time: 0.0047 (0.004708045895719708)  --> TIME: 2025-02-27 05:28:11 -- STEP: 13154/36938 -- GLOBAL_STEP: 1564550 | > loss_text_ce: 0.03194744139909744 (0.030107892844640483) | > loss_mel_ce: 3.023740530014038 (nan) | > loss: 0.012125746347010136 (nan) | > current_lr: 5e-06 | > step_time: 0.1756 (0.13717445160673108) | > loader_time: 0.0049 (0.004708217200635509)  --> TIME: 2025-02-27 05:28:20 -- STEP: 13204/36938 -- GLOBAL_STEP: 1564600 | > loss_text_ce: 0.02664944902062416 (0.030107168677083525) | > loss_mel_ce: 2.590610980987549 (nan) | > loss: 0.01038595475256443 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13716113836394808) | > loader_time: 0.0046 (0.0047082720977253975)  --> TIME: 2025-02-27 05:28:30 -- STEP: 13254/36938 -- GLOBAL_STEP: 1564650 | > loss_text_ce: 0.03377475589513779 (0.03011157678461153) | > loss_mel_ce: 2.2250561714172363 (nan) | > loss: 0.008963615633547306 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13716317133358838) | > loader_time: 0.0052 (0.004708485562318919)  --> TIME: 2025-02-27 05:28:39 -- STEP: 13304/36938 -- GLOBAL_STEP: 1564700 | > loss_text_ce: 0.03154053911566734 (0.03010917795310021) | > loss_mel_ce: 1.7434570789337158 (nan) | > loss: 0.007043641526252031 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13714533490512448) | > loader_time: 0.0046 (0.0047085779622170355)  --> TIME: 2025-02-27 05:28:48 -- STEP: 13354/36938 -- GLOBAL_STEP: 1564750 | > loss_text_ce: 0.027005357667803764 (0.030108566876977595) | > loss_mel_ce: 2.5093445777893066 (nan) | > loss: 0.010064881294965744 (nan) | > current_lr: 5e-06 | > step_time: 0.1519 (0.13715636551656918) | > loader_time: 0.0047 (0.004708943885476433)  --> TIME: 2025-02-27 05:28:58 -- STEP: 13404/36938 -- GLOBAL_STEP: 1564800 | > loss_text_ce: 0.03999702259898186 (0.03010729973105632) | > loss_mel_ce: 3.066182851791382 (nan) | > loss: 0.012326112017035484 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13715857829452824) | > loader_time: 0.0046 (0.004709061634287981)  --> TIME: 2025-02-27 05:29:07 -- STEP: 13454/36938 -- GLOBAL_STEP: 1564850 | > loss_text_ce: 0.028073668479919434 (0.030110610808859566) | > loss_mel_ce: 1.8486464023590088 (nan) | > loss: 0.007447302341461182 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13716540510983719) | > loader_time: 0.0046 (0.004709778328928605)  --> TIME: 2025-02-27 05:29:17 -- STEP: 13504/36938 -- GLOBAL_STEP: 1564900 | > loss_text_ce: 0.025803105905652046 (0.030109131894640182) | > loss_mel_ce: 0.0368550606071949 (nan) | > loss: 0.0002486435405444354 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.137191406711583) | > loader_time: 0.0045 (0.0047100083416954635)  --> TIME: 2025-02-27 05:29:26 -- STEP: 13554/36938 -- GLOBAL_STEP: 1564950 | > loss_text_ce: 0.02789073809981346 (0.03011101600514763) | > loss_mel_ce: 1.5564402341842651 (nan) | > loss: 0.00628702761605382 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13717515230776914) | > loader_time: 0.0046 (0.00471003600519211)  --> TIME: 2025-02-27 05:29:35 -- STEP: 13604/36938 -- GLOBAL_STEP: 1565000 | > loss_text_ce: 0.032817430794239044 (0.03011138558847667) | > loss_mel_ce: 1.9558731317520142 (nan) | > loss: 0.007891629822552204 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13717437296956828) | > loader_time: 0.0045 (0.004710358947349001)  --> TIME: 2025-02-27 05:29:45 -- STEP: 13654/36938 -- GLOBAL_STEP: 1565050 | > loss_text_ce: 0.03461625427007675 (0.030109904780382325) | > loss_mel_ce: 2.951256513595581 (nan) | > loss: 0.01184870209544897 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13717965093489462) | > loader_time: 0.0054 (0.004710887385371567)  --> TIME: 2025-02-27 05:29:54 -- STEP: 13704/36938 -- GLOBAL_STEP: 1565100 | > loss_text_ce: 0.032730583101511 (0.030106855576043222) | > loss_mel_ce: 1.9781208038330078 (nan) | > loss: 0.007979569025337696 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13719155589391524) | > loader_time: 0.0046 (0.004711184561496168)  --> TIME: 2025-02-27 05:30:04 -- STEP: 13754/36938 -- GLOBAL_STEP: 1565150 | > loss_text_ce: 0.0240487027913332 (0.03010795313624024) | > loss_mel_ce: 0.6445515751838684 (nan) | > loss: 0.0026531757321208715 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.1371997054690657) | > loader_time: 0.0045 (0.004711156687418256)  --> TIME: 2025-02-27 05:30:13 -- STEP: 13804/36938 -- GLOBAL_STEP: 1565200 | > loss_text_ce: 0.03341670334339142 (0.030107355211705985) | > loss_mel_ce: 2.6599626541137695 (nan) | > loss: 0.010688014328479767 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.1372082717727487) | > loader_time: 0.0046 (0.004711592466856775)  --> TIME: 2025-02-27 05:30:23 -- STEP: 13854/36938 -- GLOBAL_STEP: 1565250 | > loss_text_ce: 0.029531819745898247 (0.030106592846868157) | > loss_mel_ce: 2.3326032161712646 (nan) | > loss: 0.009373552165925503 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13721058441650552) | > loader_time: 0.0045 (0.004711366240142216)  --> TIME: 2025-02-27 05:30:32 -- STEP: 13904/36938 -- GLOBAL_STEP: 1565300 | > loss_text_ce: 0.031115418300032616 (0.03010174734665963) | > loss_mel_ce: 2.950101613998413 (nan) | > loss: 0.011830227449536324 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.13722550668362488) | > loader_time: 0.0048 (0.00471146998046046)  --> TIME: 2025-02-27 05:30:42 -- STEP: 13954/36938 -- GLOBAL_STEP: 1565350 | > loss_text_ce: 0.032562967389822006 (0.030103545492075535) | > loss_mel_ce: 2.7546846866607666 (nan) | > loss: 0.011060507036745548 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.13722198206322103) | > loader_time: 0.0046 (0.004711481977093443)  --> TIME: 2025-02-27 05:30:51 -- STEP: 14004/36938 -- GLOBAL_STEP: 1565400 | > loss_text_ce: 0.032362211495637894 (0.0301022425772286) | > loss_mel_ce: 2.36286997795105 (nan) | > loss: 0.009504890069365501 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13720366114447738) | > loader_time: 0.0047 (0.00471152935120693)  --> TIME: 2025-02-27 05:31:00 -- STEP: 14054/36938 -- GLOBAL_STEP: 1565450 | > loss_text_ce: 0.02152656763792038 (0.030099043227559623) | > loss_mel_ce: 2.5467772483825684 (nan) | > loss: 0.01019168272614479 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13718431272801182) | > loader_time: 0.0045 (0.004711685435809609)  --> TIME: 2025-02-27 05:31:09 -- STEP: 14104/36938 -- GLOBAL_STEP: 1565500 | > loss_text_ce: 0.030850449576973915 (0.030100331180331456) | > loss_mel_ce: 2.8913445472717285 (nan) | > loss: 0.011596012860536575 (nan) | > current_lr: 5e-06 | > step_time: 0.1763 (0.13720153917051844) | > loader_time: 0.0048 (0.004711770007235727)  --> TIME: 2025-02-27 05:31:19 -- STEP: 14154/36938 -- GLOBAL_STEP: 1565550 | > loss_text_ce: 0.030709344893693924 (0.0300992287106044) | > loss_mel_ce: 2.115774631500244 (nan) | > loss: 0.008517793379724026 (nan) | > current_lr: 5e-06 | > step_time: 0.1749 (0.13719675288928695) | > loader_time: 0.0049 (0.004711734081234358)  --> TIME: 2025-02-27 05:31:28 -- STEP: 14204/36938 -- GLOBAL_STEP: 1565600 | > loss_text_ce: 0.014865300618112087 (0.030098351841299133) | > loss_mel_ce: 1.202819585800171 (nan) | > loss: 0.004832082893699408 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1371966289218393) | > loader_time: 0.0045 (0.0047119456390299274)  --> TIME: 2025-02-27 05:31:37 -- STEP: 14254/36938 -- GLOBAL_STEP: 1565650 | > loss_text_ce: 0.02740483172237873 (0.03009492713282826) | > loss_mel_ce: 2.276278495788574 (nan) | > loss: 0.00914160069078207 (nan) | > current_lr: 5e-06 | > step_time: 0.1127 (0.1371825569154703) | > loader_time: 0.0045 (0.0047118929403601375)  --> TIME: 2025-02-27 05:31:47 -- STEP: 14304/36938 -- GLOBAL_STEP: 1565700 | > loss_text_ce: 0.02903200499713421 (0.030094505094299853) | > loss_mel_ce: 3.2890625 (nan) | > loss: 0.013167042285203934 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.13719390022674635) | > loader_time: 0.0047 (0.0047117046328465735)  --> TIME: 2025-02-27 05:31:56 -- STEP: 14354/36938 -- GLOBAL_STEP: 1565750 | > loss_text_ce: 0.03080146573483944 (0.030093050213653678) | > loss_mel_ce: 2.776123046875 (nan) | > loss: 0.011138590052723885 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.13720840013872612) | > loader_time: 0.0047 (0.004711477939541405)  --> TIME: 2025-02-27 05:32:05 -- STEP: 14404/36938 -- GLOBAL_STEP: 1565800 | > loss_text_ce: 0.02838873118162155 (0.030090896011370532) | > loss_mel_ce: 3.0457897186279297 (nan) | > loss: 0.01219912152737379 (nan) | > current_lr: 5e-06 | > step_time: 0.1908 (0.1371965391110855) | > loader_time: 0.0048 (0.004711319955975709)  --> TIME: 2025-02-27 05:32:15 -- STEP: 14454/36938 -- GLOBAL_STEP: 1565850 | > loss_text_ce: 0.03131895884871483 (0.0300889238325671) | > loss_mel_ce: 2.7942962646484375 (nan) | > loss: 0.011212759651243687 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.137188905268862) | > loader_time: 0.0046 (0.004711000606267145)  --> TIME: 2025-02-27 05:32:24 -- STEP: 14504/36938 -- GLOBAL_STEP: 1565900 | > loss_text_ce: 0.031884513795375824 (0.03008571650165221) | > loss_mel_ce: 2.6668689250946045 (nan) | > loss: 0.010709339752793312 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13718891193113197) | > loader_time: 0.0046 (0.00471088752125879)  --> TIME: 2025-02-27 05:32:33 -- STEP: 14554/36938 -- GLOBAL_STEP: 1565950 | > loss_text_ce: 0.03761351853609085 (0.03008312219776258) | > loss_mel_ce: 3.248436689376831 (nan) | > loss: 0.013039883226156235 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13718426252142055) | > loader_time: 0.0047 (0.0047107577340295284)  --> TIME: 2025-02-27 05:32:43 -- STEP: 14604/36938 -- GLOBAL_STEP: 1566000 | > loss_text_ce: 0.026750853285193443 (0.03008075221412989) | > loss_mel_ce: 2.6524224281311035 (nan) | > loss: 0.010631640441715717 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13719590816782531) | > loader_time: 0.0046 (0.004710956310056987)  --> TIME: 2025-02-27 05:32:52 -- STEP: 14654/36938 -- GLOBAL_STEP: 1566050 | > loss_text_ce: 0.03635378181934357 (0.03008323916384318) | > loss_mel_ce: 2.7553799152374268 (nan) | > loss: 0.011078309267759323 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.1371969700315176) | > loader_time: 0.0047 (0.004710897850082443)  --> TIME: 2025-02-27 05:33:01 -- STEP: 14704/36938 -- GLOBAL_STEP: 1566100 | > loss_text_ce: 0.0359988734126091 (0.030086355455534154) | > loss_mel_ce: 1.3782848119735718 (nan) | > loss: 0.0056122369132936 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13719279707386142) | > loader_time: 0.0045 (0.004710800467429929)  --> TIME: 2025-02-27 05:33:10 -- STEP: 14754/36938 -- GLOBAL_STEP: 1566150 | > loss_text_ce: 0.0296334158629179 (0.030088059647186534) | > loss_mel_ce: 2.5643198490142822 (nan) | > loss: 0.010293466039001942 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13718612796490778) | > loader_time: 0.0045 (0.004710601907088637)  --> TIME: 2025-02-27 05:33:20 -- STEP: 14804/36938 -- GLOBAL_STEP: 1566200 | > loss_text_ce: 0.025012535974383354 (0.030088745964213178) | > loss_mel_ce: 2.3839728832244873 (nan) | > loss: 0.009559466503560543 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.13718143995308754) | > loader_time: 0.0046 (0.0047105273115605984)  --> TIME: 2025-02-27 05:33:29 -- STEP: 14854/36938 -- GLOBAL_STEP: 1566250 | > loss_text_ce: 0.03587942570447922 (0.03008958414520868) | > loss_mel_ce: 2.2899105548858643 (nan) | > loss: 0.0092293256893754 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13717211976545063) | > loader_time: 0.0046 (0.0047107620837631686)  --> TIME: 2025-02-27 05:33:38 -- STEP: 14904/36938 -- GLOBAL_STEP: 1566300 | > loss_text_ce: 0.027985401451587677 (0.030088008211927374) | > loss_mel_ce: 2.396195411682129 (nan) | > loss: 0.009619764983654022 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13717042268826263) | > loader_time: 0.0047 (0.004710670926528666)  --> TIME: 2025-02-27 05:33:48 -- STEP: 14954/36938 -- GLOBAL_STEP: 1566350 | > loss_text_ce: 0.023096716031432152 (0.030091578711799044) | > loss_mel_ce: 2.679325819015503 (nan) | > loss: 0.01072390004992485 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13718647741292495) | > loader_time: 0.0046 (0.004711190933879569)  --> TIME: 2025-02-27 05:33:57 -- STEP: 15004/36938 -- GLOBAL_STEP: 1566400 | > loss_text_ce: 0.03419411927461624 (0.030090797606785367) | > loss_mel_ce: 2.076054573059082 (nan) | > loss: 0.008374003693461418 (nan) | > current_lr: 5e-06 | > step_time: 0.1624 (0.1371827744636749) | > loader_time: 0.0046 (0.004711329539722844)  --> TIME: 2025-02-27 05:34:06 -- STEP: 15054/36938 -- GLOBAL_STEP: 1566450 | > loss_text_ce: 0.029102755710482597 (0.030092623243808708) | > loss_mel_ce: 2.168900966644287 (nan) | > loss: 0.008722237311303616 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13717980854885345) | > loader_time: 0.0052 (0.004711480718440111)  --> TIME: 2025-02-27 05:34:16 -- STEP: 15104/36938 -- GLOBAL_STEP: 1566500 | > loss_text_ce: 0.03248845413327217 (0.030094537199717813) | > loss_mel_ce: 2.7669966220855713 (nan) | > loss: 0.011109068058431149 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13718383795565972) | > loader_time: 0.0047 (0.004711910529788274)  --> TIME: 2025-02-27 05:34:25 -- STEP: 15154/36938 -- GLOBAL_STEP: 1566550 | > loss_text_ce: 0.039779745042324066 (0.03009420619239266) | > loss_mel_ce: 3.2752022743225098 (nan) | > loss: 0.013154691085219383 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.13718003359479075) | > loader_time: 0.0047 (0.00471211678594434)  --> TIME: 2025-02-27 05:34:35 -- STEP: 15204/36938 -- GLOBAL_STEP: 1566600 | > loss_text_ce: 0.03014400228857994 (0.030092200328621907) | > loss_mel_ce: 2.9263079166412354 (nan) | > loss: 0.011731952428817749 (nan) | > current_lr: 5e-06 | > step_time: 0.1851 (0.1371822988036682) | > loader_time: 0.0048 (0.004712263319687421)  --> TIME: 2025-02-27 05:34:44 -- STEP: 15254/36938 -- GLOBAL_STEP: 1566650 | > loss_text_ce: 0.03229182958602905 (0.030088107807865073) | > loss_mel_ce: 3.294546365737915 (nan) | > loss: 0.013201739639043808 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.1371777438704802) | > loader_time: 0.0047 (0.004712084572265519)  --> TIME: 2025-02-27 05:34:53 -- STEP: 15304/36938 -- GLOBAL_STEP: 1566700 | > loss_text_ce: 0.019523706287145615 (0.03008657210794099) | > loss_mel_ce: 1.8617373704910278 (nan) | > loss: 0.0074653225019574165 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13718809414084396) | > loader_time: 0.0045 (0.004712121856193927)  --> TIME: 2025-02-27 05:35:02 -- STEP: 15354/36938 -- GLOBAL_STEP: 1566750 | > loss_text_ce: 0.03145202249288559 (0.030089767326682008) | > loss_mel_ce: 2.055586576461792 (nan) | > loss: 0.008281899616122246 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.1371832532281442) | > loader_time: 0.0047 (0.0047120922661796644)  --> TIME: 2025-02-27 05:35:12 -- STEP: 15404/36938 -- GLOBAL_STEP: 1566800 | > loss_text_ce: 0.021359380334615707 (0.030084315070697917) | > loss_mel_ce: 1.5857781171798706 (nan) | > loss: 0.006377529818564653 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1371871113343785) | > loader_time: 0.0045 (0.004712024963357119)  --> TIME: 2025-02-27 05:35:21 -- STEP: 15454/36938 -- GLOBAL_STEP: 1566850 | > loss_text_ce: 0.029493825510144234 (0.03008380312411956) | > loss_mel_ce: 1.8663482666015625 (nan) | > loss: 0.00752318324521184 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.13717705422514112) | > loader_time: 0.0047 (0.004712314983381255)  --> TIME: 2025-02-27 05:35:30 -- STEP: 15504/36938 -- GLOBAL_STEP: 1566900 | > loss_text_ce: 0.02487686276435852 (0.03008434502398898) | > loss_mel_ce: 2.479886293411255 (nan) | > loss: 0.009939536452293396 (nan) | > current_lr: 5e-06 | > step_time: 0.1897 (0.13717246466192298) | > loader_time: 0.0049 (0.004712161803147132)  --> TIME: 2025-02-27 05:35:40 -- STEP: 15554/36938 -- GLOBAL_STEP: 1566950 | > loss_text_ce: 0.022831525653600693 (0.030088512214395027) | > loss_mel_ce: 1.8264950513839722 (nan) | > loss: 0.007338597904890776 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13717813929908146) | > loader_time: 0.0046 (0.0047119594837378605)  --> TIME: 2025-02-27 05:35:49 -- STEP: 15604/36938 -- GLOBAL_STEP: 1567000 | > loss_text_ce: 0.02899847738444805 (0.030090129340058672) | > loss_mel_ce: 2.324432849884033 (nan) | > loss: 0.009339013136923313 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13717214739099584) | > loader_time: 0.0046 (0.004712128220567706)  --> TIME: 2025-02-27 05:35:58 -- STEP: 15654/36938 -- GLOBAL_STEP: 1567050 | > loss_text_ce: 0.0310511514544487 (0.030091804896454088) | > loss_mel_ce: 2.508368730545044 (nan) | > loss: 0.010077063925564289 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13716588237207328) | > loader_time: 0.0045 (0.0047120847235363255)  --> TIME: 2025-02-27 05:36:07 -- STEP: 15704/36938 -- GLOBAL_STEP: 1567100 | > loss_text_ce: 0.029556993395090103 (0.03009340463276033) | > loss_mel_ce: 2.0642828941345215 (nan) | > loss: 0.008308889344334602 (nan) | > current_lr: 5e-06 | > step_time: 0.1599 (0.13715705604985606) | > loader_time: 0.0049 (0.004711764112638692)  --> TIME: 2025-02-27 05:36:17 -- STEP: 15754/36938 -- GLOBAL_STEP: 1567150 | > loss_text_ce: 0.03462928906083107 (0.030094930906172938) | > loss_mel_ce: 2.5959675312042236 (nan) | > loss: 0.010438877157866955 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.1371548537056625) | > loader_time: 0.0046 (0.0047116843489079996)  --> TIME: 2025-02-27 05:36:26 -- STEP: 15804/36938 -- GLOBAL_STEP: 1567200 | > loss_text_ce: 0.028756529092788696 (0.0300952619031768) | > loss_mel_ce: 2.373609781265259 (nan) | > loss: 0.009533200412988663 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.1371524790756855) | > loader_time: 0.0046 (0.004711631641180362)  --> TIME: 2025-02-27 05:36:35 -- STEP: 15854/36938 -- GLOBAL_STEP: 1567250 | > loss_text_ce: 0.029477640986442566 (0.03009360310879092) | > loss_mel_ce: 2.5217015743255615 (nan) | > loss: 0.010123727843165398 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13715595386662008) | > loader_time: 0.0046 (0.004711664142562931)  --> TIME: 2025-02-27 05:36:45 -- STEP: 15904/36938 -- GLOBAL_STEP: 1567300 | > loss_text_ce: 0.027998171746730804 (0.0300903457786522) | > loss_mel_ce: 2.2005982398986816 (nan) | > loss: 0.008843637071549892 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.13714881533648166) | > loader_time: 0.0047 (0.0047117458502771445)  --> TIME: 2025-02-27 05:36:54 -- STEP: 15954/36938 -- GLOBAL_STEP: 1567350 | > loss_text_ce: 0.035428471863269806 (0.03008795742519803) | > loss_mel_ce: 2.081530809402466 (nan) | > loss: 0.008400632999837399 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13714408216970195) | > loader_time: 0.0046 (0.004711864854483379)  --> TIME: 2025-02-27 05:37:03 -- STEP: 16004/36938 -- GLOBAL_STEP: 1567400 | > loss_text_ce: 0.03463699296116829 (0.0300892755640666) | > loss_mel_ce: 2.6295502185821533 (nan) | > loss: 0.01057217177003622 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13714475233654355) | > loader_time: 0.0045 (0.004712281764015445)  --> TIME: 2025-02-27 05:37:12 -- STEP: 16054/36938 -- GLOBAL_STEP: 1567450 | > loss_text_ce: 0.026426929980516434 (0.03008858057991913) | > loss_mel_ce: 0.003082275390625 (nan) | > loss: 0.00011710002581821755 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13713902832785446) | > loader_time: 0.0046 (0.004712555749160622)  --> TIME: 2025-02-27 05:37:22 -- STEP: 16104/36938 -- GLOBAL_STEP: 1567500 | > loss_text_ce: 0.02688850276172161 (0.03009105690750208) | > loss_mel_ce: 2.9149680137634277 (nan) | > loss: 0.011674035340547562 (nan) | > current_lr: 5e-06 | > step_time: 0.1621 (0.13716697531497365) | > loader_time: 0.0047 (0.004712899659204654)  --> TIME: 2025-02-27 05:37:31 -- STEP: 16154/36938 -- GLOBAL_STEP: 1567550 | > loss_text_ce: 0.028625236824154854 (0.030090139856261303) | > loss_mel_ce: 1.8498687744140625 (nan) | > loss: 0.007454341743141413 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13715367974275053) | > loader_time: 0.0046 (0.004713047756569601)  --> TIME: 2025-02-27 05:37:41 -- STEP: 16204/36938 -- GLOBAL_STEP: 1567600 | > loss_text_ce: 0.029636038467288017 (0.030087041753053184) | > loss_mel_ce: 2.2865400314331055 (nan) | > loss: 0.009191175922751427 (nan) | > current_lr: 5e-06 | > step_time: 0.1261 (0.13716193200099383) | > loader_time: 0.0046 (0.004712943014348774)  --> TIME: 2025-02-27 05:37:51 -- STEP: 16254/36938 -- GLOBAL_STEP: 1567650 | > loss_text_ce: 0.03363439068198204 (0.03008447862420471) | > loss_mel_ce: 2.861994981765747 (nan) | > loss: 0.011490593664348125 (nan) | > current_lr: 5e-06 | > step_time: 0.1517 (0.13717232381811995) | > loader_time: 0.0047 (0.00471282760727607)  --> TIME: 2025-02-27 05:38:00 -- STEP: 16304/36938 -- GLOBAL_STEP: 1567700 | > loss_text_ce: 0.03078632988035679 (0.030090212834098944) | > loss_mel_ce: 2.9970786571502686 (nan) | > loss: 0.01201533805578947 (nan) | > current_lr: 5e-06 | > step_time: 0.1581 (0.13716342841970558) | > loader_time: 0.0047 (0.004712496058863681)  --> TIME: 2025-02-27 05:38:09 -- STEP: 16354/36938 -- GLOBAL_STEP: 1567750 | > loss_text_ce: 0.041546642780303955 (0.030090407378087977) | > loss_mel_ce: 2.4830007553100586 (nan) | > loss: 0.010018046014010906 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.1371736349213367) | > loader_time: 0.0046 (0.004712312921591931)  --> TIME: 2025-02-27 05:38:19 -- STEP: 16404/36938 -- GLOBAL_STEP: 1567800 | > loss_text_ce: 0.025400375947356224 (0.03009127049149468) | > loss_mel_ce: 2.3099746704101562 (nan) | > loss: 0.009267361834645271 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1371670077847609) | > loader_time: 0.0047 (0.0047120561078826725)  --> TIME: 2025-02-27 05:38:28 -- STEP: 16454/36938 -- GLOBAL_STEP: 1567850 | > loss_text_ce: 0.03262109309434891 (0.03008960050020638) | > loss_mel_ce: 2.7745869159698486 (nan) | > loss: 0.01113971509039402 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.1371586297426911) | > loader_time: 0.0046 (0.004711803419702734)  --> TIME: 2025-02-27 05:38:37 -- STEP: 16504/36938 -- GLOBAL_STEP: 1567900 | > loss_text_ce: 0.03649245575070381 (0.030090114460691268) | > loss_mel_ce: 2.7871036529541016 (nan) | > loss: 0.011204746551811695 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13713665019812893) | > loader_time: 0.0046 (0.004711468359595418)  --> TIME: 2025-02-27 05:38:46 -- STEP: 16554/36938 -- GLOBAL_STEP: 1567950 | > loss_text_ce: 0.030994774773716927 (0.030093144172675647) | > loss_mel_ce: 2.969712495803833 (nan) | > loss: 0.01190757006406784 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13713691259648936) | > loader_time: 0.0046 (0.004711357193666051)  --> TIME: 2025-02-27 05:38:55 -- STEP: 16604/36938 -- GLOBAL_STEP: 1568000 | > loss_text_ce: 0.03347468748688698 (0.030093624216755228) | > loss_mel_ce: 2.251356601715088 (nan) | > loss: 0.009066791273653507 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13712411067825267) | > loader_time: 0.0045 (0.004711027850130618)  --> TIME: 2025-02-27 05:39:04 -- STEP: 16654/36938 -- GLOBAL_STEP: 1568050 | > loss_text_ce: 0.033908065408468246 (0.030094375375002192) | > loss_mel_ce: 2.629317045211792 (nan) | > loss: 0.010568354278802872 (nan) | > current_lr: 5e-06 | > step_time: 0.17 (0.13711768332199997) | > loader_time: 0.0047 (0.004710771906657913)  --> TIME: 2025-02-27 05:39:14 -- STEP: 16704/36938 -- GLOBAL_STEP: 1568100 | > loss_text_ce: 0.025589630007743835 (0.03009130449663295) | > loss_mel_ce: 3.042349100112915 (nan) | > loss: 0.012174361385405064 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13712730518474445) | > loader_time: 0.0045 (0.004710630595798239)  --> TIME: 2025-02-27 05:39:23 -- STEP: 16754/36938 -- GLOBAL_STEP: 1568150 | > loss_text_ce: 0.03381871059536934 (0.030090496021763044) | > loss_mel_ce: 4.696875095367432 (nan) | > loss: 0.01877259649336338 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13710924173662262) | > loader_time: 0.0046 (0.00471036923989001)  --> TIME: 2025-02-27 05:39:32 -- STEP: 16804/36938 -- GLOBAL_STEP: 1568200 | > loss_text_ce: 0.029566587880253792 (0.030091095573795995) | > loss_mel_ce: 2.8153038024902344 (nan) | > loss: 0.011289169080555439 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13711250865496338) | > loader_time: 0.0045 (0.004710158856815738)  --> TIME: 2025-02-27 05:39:41 -- STEP: 16854/36938 -- GLOBAL_STEP: 1568250 | > loss_text_ce: 0.03569106012582779 (0.030090053831841036) | > loss_mel_ce: 2.2477455139160156 (nan) | > loss: 0.009061256423592567 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.1371016650207247) | > loader_time: 0.0047 (0.00471021373090589)  --> TIME: 2025-02-27 05:39:51 -- STEP: 16904/36938 -- GLOBAL_STEP: 1568300 | > loss_text_ce: 0.022572876885533333 (0.030088443158907786) | > loss_mel_ce: 2.4281609058380127 (nan) | > loss: 0.009725134819746017 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13710350277747707) | > loader_time: 0.0046 (0.0047104290690514585)  --> TIME: 2025-02-27 05:40:01 -- STEP: 16954/36938 -- GLOBAL_STEP: 1568350 | > loss_text_ce: 0.033449698239564896 (0.030087176807020617) | > loss_mel_ce: 2.8920176029205322 (nan) | > loss: 0.011608997359871864 (nan) | > current_lr: 5e-06 | > step_time: 0.1872 (0.1371271892357257) | > loader_time: 0.0048 (0.004710638229191915)  --> TIME: 2025-02-27 05:40:10 -- STEP: 17004/36938 -- GLOBAL_STEP: 1568400 | > loss_text_ce: 0.02527683414518833 (0.030082843720778636) | > loss_mel_ce: 2.2232959270477295 (nan) | > loss: 0.008922908455133438 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.1371261972563321) | > loader_time: 0.0045 (0.004710665718860219)  --> TIME: 2025-02-27 05:40:19 -- STEP: 17054/36938 -- GLOBAL_STEP: 1568450 | > loss_text_ce: 0.023874014616012573 (0.0300852704729157) | > loss_mel_ce: 2.3331475257873535 (nan) | > loss: 0.00935326050966978 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.1371304156758715) | > loader_time: 0.0046 (0.004710641586166817)  --> TIME: 2025-02-27 05:40:29 -- STEP: 17104/36938 -- GLOBAL_STEP: 1568500 | > loss_text_ce: 0.040969714522361755 (0.030082535517814292) | > loss_mel_ce: 2.8638522624969482 (nan) | > loss: 0.011527071706950665 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13713403126979357) | > loader_time: 0.0048 (0.004710818460547444)  --> TIME: 2025-02-27 05:40:39 -- STEP: 17154/36938 -- GLOBAL_STEP: 1568550 | > loss_text_ce: 0.02733907662332058 (0.03008486629790352) | > loss_mel_ce: 2.5030534267425537 (nan) | > loss: 0.010041240602731705 (nan) | > current_lr: 5e-06 | > step_time: 0.1765 (0.13715594214020999) | > loader_time: 0.0049 (0.004710786170554742)  --> TIME: 2025-02-27 05:40:48 -- STEP: 17204/36938 -- GLOBAL_STEP: 1568600 | > loss_text_ce: 0.023700829595327377 (0.03008449110327541) | > loss_mel_ce: 1.92431640625 (nan) | > loss: 0.007730227429419756 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13713692740156777) | > loader_time: 0.0044 (0.004710806910034337)  --> TIME: 2025-02-27 05:40:57 -- STEP: 17254/36938 -- GLOBAL_STEP: 1568650 | > loss_text_ce: 0.0334356427192688 (0.030089211624256693) | > loss_mel_ce: 2.5172410011291504 (nan) | > loss: 0.010121732950210571 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.137144389303974) | > loader_time: 0.0047 (0.004710688159324014)  --> TIME: 2025-02-27 05:41:07 -- STEP: 17304/36938 -- GLOBAL_STEP: 1568700 | > loss_text_ce: 0.025949647650122643 (0.030088740837540894) | > loss_mel_ce: 2.943842649459839 (nan) | > loss: 0.011784890666604042 (nan) | > current_lr: 5e-06 | > step_time: 0.1612 (0.13715370044187997) | > loader_time: 0.0048 (0.004710572643848795)  --> TIME: 2025-02-27 05:41:16 -- STEP: 17354/36938 -- GLOBAL_STEP: 1568750 | > loss_text_ce: 0.030905673280358315 (0.030086300138091517) | > loss_mel_ce: 2.8744349479675293 (nan) | > loss: 0.011529129929840565 (nan) | > current_lr: 5e-06 | > step_time: 0.1828 (0.13716042382494295) | > loader_time: 0.0049 (0.004710405862343755)  --> TIME: 2025-02-27 05:41:26 -- STEP: 17404/36938 -- GLOBAL_STEP: 1568800 | > loss_text_ce: 0.027826938778162003 (0.030089158813931028) | > loss_mel_ce: 1.4370346069335938 (nan) | > loss: 0.005812942981719971 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13715906352510243) | > loader_time: 0.0046 (0.004710496965152753)  --> TIME: 2025-02-27 05:41:35 -- STEP: 17454/36938 -- GLOBAL_STEP: 1568850 | > loss_text_ce: 0.028172440826892853 (0.030086015906767374) | > loss_mel_ce: 2.737668752670288 (nan) | > loss: 0.01097556110471487 (nan) | > current_lr: 5e-06 | > step_time: 0.1823 (0.1371716731081199) | > loader_time: 0.0049 (0.004710612420542917)  --> TIME: 2025-02-27 05:41:45 -- STEP: 17504/36938 -- GLOBAL_STEP: 1568900 | > loss_text_ce: 0.03350081667304039 (0.030086008843111762) | > loss_mel_ce: 2.324655771255493 (nan) | > loss: 0.009357765316963196 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.1371781496940628) | > loader_time: 0.0047 (0.004710651757076407)  --> TIME: 2025-02-27 05:41:54 -- STEP: 17554/36938 -- GLOBAL_STEP: 1568950 | > loss_text_ce: 0.03015059605240822 (0.030084689579290978) | > loss_mel_ce: 2.7487564086914062 (nan) | > loss: 0.011027409695088863 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13717955926073933) | > loader_time: 0.0047 (0.0047104588888141575)  --> TIME: 2025-02-27 05:42:03 -- STEP: 17604/36938 -- GLOBAL_STEP: 1569000 | > loss_text_ce: 0.0323333740234375 (0.030085541027614165) | > loss_mel_ce: 2.74013090133667 (nan) | > loss: 0.01100184302777052 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.1371811476248933) | > loader_time: 0.0045 (0.004710234056627495)  --> TIME: 2025-02-27 05:42:13 -- STEP: 17654/36938 -- GLOBAL_STEP: 1569050 | > loss_text_ce: 0.028600158169865608 (0.030084896907483616) | > loss_mel_ce: 2.023505926132202 (nan) | > loss: 0.008143278770148754 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13717963457296622) | > loader_time: 0.0046 (0.00471006007511618)  --> TIME: 2025-02-27 05:42:22 -- STEP: 17704/36938 -- GLOBAL_STEP: 1569100 | > loss_text_ce: 0.031174195930361748 (0.03007909999711614) | > loss_mel_ce: 2.0541799068450928 (nan) | > loss: 0.00827521551400423 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13717194851630968) | > loader_time: 0.0046 (0.004710091571704432)  --> TIME: 2025-02-27 05:42:31 -- STEP: 17754/36938 -- GLOBAL_STEP: 1569150 | > loss_text_ce: 0.02506796456873417 (0.030077391079113634) | > loss_mel_ce: 2.854311227798462 (nan) | > loss: 0.01142610888928175 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.13717759756726625) | > loader_time: 0.0049 (0.0047101681063515454)  --> TIME: 2025-02-27 05:42:41 -- STEP: 17804/36938 -- GLOBAL_STEP: 1569200 | > loss_text_ce: 0.02604021318256855 (0.030075962448021232) | > loss_mel_ce: 2.856795072555542 (nan) | > loss: 0.011439823545515537 (nan) | > current_lr: 5e-06 | > step_time: 0.1869 (0.13718220258771482) | > loader_time: 0.0049 (0.004710128604093579)  --> TIME: 2025-02-27 05:42:50 -- STEP: 17854/36938 -- GLOBAL_STEP: 1569250 | > loss_text_ce: 0.024827713146805763 (0.030075017656746826) | > loss_mel_ce: 2.7142791748046875 (nan) | > loss: 0.010869472287595272 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.1371829991539877) | > loader_time: 0.0047 (0.004710040862180757)  --> TIME: 2025-02-27 05:43:00 -- STEP: 17904/36938 -- GLOBAL_STEP: 1569300 | > loss_text_ce: 0.035935815423727036 (0.030073971682049415) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.1371808153002044) | > loader_time: 0.0045 (0.004709819446619969)  --> TIME: 2025-02-27 05:43:09 -- STEP: 17954/36938 -- GLOBAL_STEP: 1569350 | > loss_text_ce: 0.025336839258670807 (0.03007218558013948) | > loss_mel_ce: 2.758108139038086 (nan) | > loss: 0.011045416817069054 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13718133324887322) | > loader_time: 0.0047 (0.004710026582524759)  --> TIME: 2025-02-27 05:43:18 -- STEP: 18004/36938 -- GLOBAL_STEP: 1569400 | > loss_text_ce: 0.03370297700166702 (0.030073726281402247) | > loss_mel_ce: 1.7219866514205933 (nan) | > loss: 0.0069670225493609905 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13717852606028627) | > loader_time: 0.0046 (0.004710207711694188)  --> TIME: 2025-02-27 05:43:27 -- STEP: 18054/36938 -- GLOBAL_STEP: 1569450 | > loss_text_ce: 0.023726217448711395 (0.03007343613590694) | > loss_mel_ce: 2.2927451133728027 (nan) | > loss: 0.009192347526550293 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13717173535628122) | > loader_time: 0.0052 (0.004710150422666429)  --> TIME: 2025-02-27 05:43:37 -- STEP: 18104/36938 -- GLOBAL_STEP: 1569500 | > loss_text_ce: 0.025527358055114746 (0.030072884848885612) | > loss_mel_ce: 1.3955026865005493 (nan) | > loss: 0.005639008246362209 (nan) | > current_lr: 5e-06 | > step_time: 0.1494 (0.13716807608524373) | > loader_time: 0.0047 (0.004710385863147792)  --> TIME: 2025-02-27 05:43:46 -- STEP: 18154/36938 -- GLOBAL_STEP: 1569550 | > loss_text_ce: 0.03730250895023346 (0.030075717359146978) | > loss_mel_ce: 2.6553657054901123 (nan) | > loss: 0.010685191489756107 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13717795868230928) | > loader_time: 0.0046 (0.004710714302491653)  --> TIME: 2025-02-27 05:43:56 -- STEP: 18204/36938 -- GLOBAL_STEP: 1569600 | > loss_text_ce: 0.026564354076981544 (0.030075195218457702) | > loss_mel_ce: 0.33014005422592163 (nan) | > loss: 0.0014154937816783786 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13717392450425692) | > loader_time: 0.0046 (0.004710839046695367)  --> TIME: 2025-02-27 05:44:05 -- STEP: 18254/36938 -- GLOBAL_STEP: 1569650 | > loss_text_ce: 0.025502461940050125 (0.030074133930284574) | > loss_mel_ce: 0.9823024272918701 (nan) | > loss: 0.0039992257952690125 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13716388696959145) | > loader_time: 0.0052 (0.004710906631024181)  --> TIME: 2025-02-27 05:44:14 -- STEP: 18304/36938 -- GLOBAL_STEP: 1569700 | > loss_text_ce: 0.028212446719408035 (0.030072706366383015) | > loss_mel_ce: 2.7165791988372803 (nan) | > loss: 0.010892031714320183 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13714627694937748) | > loader_time: 0.0046 (0.00471081091025909)  --> TIME: 2025-02-27 05:44:23 -- STEP: 18354/36938 -- GLOBAL_STEP: 1569750 | > loss_text_ce: 0.021595699712634087 (0.030070501394612294) | > loss_mel_ce: 1.0268701314926147 (nan) | > loss: 0.004160578828305006 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13715415180437432) | > loader_time: 0.0045 (0.004710762189262579)  --> TIME: 2025-02-27 05:44:32 -- STEP: 18404/36938 -- GLOBAL_STEP: 1569800 | > loss_text_ce: 0.03199024870991707 (0.03006570838210921) | > loss_mel_ce: 2.5325355529785156 (nan) | > loss: 0.01017669029533863 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13713732309326943) | > loader_time: 0.0046 (0.004710725029508226)  --> TIME: 2025-02-27 05:44:42 -- STEP: 18454/36938 -- GLOBAL_STEP: 1569850 | > loss_text_ce: 0.02942883037030697 (0.030064610037580212) | > loss_mel_ce: 2.2083282470703125 (nan) | > loss: 0.008879988454282284 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13713446613819064) | > loader_time: 0.0046 (0.004710986643425622)  --> TIME: 2025-02-27 05:44:51 -- STEP: 18504/36938 -- GLOBAL_STEP: 1569900 | > loss_text_ce: 0.03317075967788696 (0.03006655762892219) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13713210421725908) | > loader_time: 0.0053 (0.004711102302900081)  --> TIME: 2025-02-27 05:45:01 -- STEP: 18554/36938 -- GLOBAL_STEP: 1569950 | > loss_text_ce: 0.034696441143751144 (0.03006880546879243) | > loss_mel_ce: 2.871372699737549 (nan) | > loss: 0.011532020755112171 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.13716007800429675) | > loader_time: 0.0046 (0.0047113150502762335)  --> TIME: 2025-02-27 05:45:10 -- STEP: 18604/36938 -- GLOBAL_STEP: 1570000 | > loss_text_ce: 0.03681077063083649 (0.030068316423645356) | > loss_mel_ce: 2.8881187438964844 (nan) | > loss: 0.011606864631175995 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.13716534472772915) | > loader_time: 0.0048 (0.004711500177381369) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1570000.pth  --> TIME: 2025-02-27 05:45:26 -- STEP: 18654/36938 -- GLOBAL_STEP: 1570050 | > loss_text_ce: 0.03661670535802841 (0.030069985530933603) | > loss_mel_ce: 2.9190073013305664 (nan) | > loss: 0.01172866765409708 (nan) | > current_lr: 5e-06 | > step_time: 0.1274 (0.13716088379644928) | > loader_time: 0.0046 (0.004711503932442041)  --> TIME: 2025-02-27 05:45:35 -- STEP: 18704/36938 -- GLOBAL_STEP: 1570100 | > loss_text_ce: 0.028580185025930405 (0.030072460964498754) | > loss_mel_ce: 2.4402718544006348 (nan) | > loss: 0.009797032922506332 (nan) | > current_lr: 5e-06 | > step_time: 0.1499 (0.13716379495081096) | > loader_time: 0.0046 (0.004711558209004001)  --> TIME: 2025-02-27 05:45:44 -- STEP: 18754/36938 -- GLOBAL_STEP: 1570150 | > loss_text_ce: 0.024220308288931847 (0.03007644370840949) | > loss_mel_ce: 2.5491104125976562 (nan) | > loss: 0.010211629793047905 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13716300868449025) | > loader_time: 0.0048 (0.004711573675927969)  --> TIME: 2025-02-27 05:45:54 -- STEP: 18804/36938 -- GLOBAL_STEP: 1570200 | > loss_text_ce: 0.029369214549660683 (0.030075021148918815) | > loss_mel_ce: 2.6808786392211914 (nan) | > loss: 0.01075495220720768 (nan) | > current_lr: 5e-06 | > step_time: 0.1977 (0.13716684090687972) | > loader_time: 0.0048 (0.004711747980959686)  --> TIME: 2025-02-27 05:46:03 -- STEP: 18854/36938 -- GLOBAL_STEP: 1570250 | > loss_text_ce: 0.025874139741063118 (0.030074766119501453) | > loss_mel_ce: 2.7090203762054443 (nan) | > loss: 0.010852756910026073 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1371738244382755) | > loader_time: 0.0046 (0.0047120300623573075)  --> TIME: 2025-02-27 05:46:13 -- STEP: 18904/36938 -- GLOBAL_STEP: 1570300 | > loss_text_ce: 0.034937117248773575 (0.030075989522872878) | > loss_mel_ce: 3.46875 (nan) | > loss: 0.013903521001338959 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13717576412391844) | > loader_time: 0.0045 (0.004711867324689453)  --> TIME: 2025-02-27 05:46:22 -- STEP: 18954/36938 -- GLOBAL_STEP: 1570350 | > loss_text_ce: 0.03552950546145439 (0.030074242773558482) | > loss_mel_ce: 0.0115814208984375 (nan) | > loss: 0.00018694813479669392 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13718384697402844) | > loader_time: 0.0046 (0.004711697571285413)  --> TIME: 2025-02-27 05:46:32 -- STEP: 19004/36938 -- GLOBAL_STEP: 1570400 | > loss_text_ce: 0.03318211808800697 (0.03007707404620319) | > loss_mel_ce: 2.6588919162750244 (nan) | > loss: 0.010682834312319756 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13719024450697181) | > loader_time: 0.0045 (0.0047115824318413844)  --> TIME: 2025-02-27 05:46:41 -- STEP: 19054/36938 -- GLOBAL_STEP: 1570450 | > loss_text_ce: 0.028413577005267143 (0.03007695780198646) | > loss_mel_ce: 0.1490325927734375 (nan) | > loss: 0.0007041515200398862 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13718094811480078) | > loader_time: 0.0047 (0.0047114570480936025)  --> TIME: 2025-02-27 05:46:50 -- STEP: 19104/36938 -- GLOBAL_STEP: 1570500 | > loss_text_ce: 0.03277691826224327 (0.030076355999876774) | > loss_mel_ce: 2.4555280208587646 (nan) | > loss: 0.009874226525425911 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13717972781740062) | > loader_time: 0.0048 (0.004711465894477043)  --> TIME: 2025-02-27 05:47:00 -- STEP: 19154/36938 -- GLOBAL_STEP: 1570550 | > loss_text_ce: 0.02813269942998886 (0.030074568116409985) | > loss_mel_ce: 2.5229175090789795 (nan) | > loss: 0.010123215615749359 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.1371870578250247) | > loader_time: 0.0045 (0.004711361597086443)  --> TIME: 2025-02-27 05:47:09 -- STEP: 19204/36938 -- GLOBAL_STEP: 1570600 | > loss_text_ce: 0.0393109992146492 (0.030072942423444902) | > loss_mel_ce: 2.622309923171997 (nan) | > loss: 0.0105619877576828 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13718296389757567) | > loader_time: 0.0046 (0.004711806153387003)  --> TIME: 2025-02-27 05:47:18 -- STEP: 19254/36938 -- GLOBAL_STEP: 1570650 | > loss_text_ce: 0.03935656696557999 (0.030072551768715608) | > loss_mel_ce: 2.2917895317077637 (nan) | > loss: 0.009250580333173275 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.1371805460341343) | > loader_time: 0.0052 (0.0047116710399929896)  --> TIME: 2025-02-27 05:47:27 -- STEP: 19304/36938 -- GLOBAL_STEP: 1570700 | > loss_text_ce: 0.0222102589905262 (0.030073238392245686) | > loss_mel_ce: 2.3124618530273438 (nan) | > loss: 0.009264572523534298 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13717011374356225) | > loader_time: 0.0045 (0.004711690195556293)  --> TIME: 2025-02-27 05:47:37 -- STEP: 19354/36938 -- GLOBAL_STEP: 1570750 | > loss_text_ce: 0.029768453910946846 (0.03007206928482614) | > loss_mel_ce: 2.148695707321167 (nan) | > loss: 0.008644700050354004 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13717638263651183) | > loader_time: 0.0046 (0.004711830494040914)  --> TIME: 2025-02-27 05:47:46 -- STEP: 19404/36938 -- GLOBAL_STEP: 1570800 | > loss_text_ce: 0.03657044097781181 (0.030070266153211653) | > loss_mel_ce: 1.7648259401321411 (nan) | > loss: 0.00714839855208993 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13717620013006435) | > loader_time: 0.0046 (0.004712003564077696)  --> TIME: 2025-02-27 05:47:56 -- STEP: 19454/36938 -- GLOBAL_STEP: 1570850 | > loss_text_ce: 0.02060714364051819 (0.030070745123910683) | > loss_mel_ce: 1.9650909900665283 (nan) | > loss: 0.007879754528403282 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13717846319169827) | > loader_time: 0.0045 (0.0047121114521020595)  --> TIME: 2025-02-27 05:48:05 -- STEP: 19504/36938 -- GLOBAL_STEP: 1570900 | > loss_text_ce: 0.02890794724225998 (0.030071184772059705) | > loss_mel_ce: 2.754995584487915 (nan) | > loss: 0.011047237552702427 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.1371805959054938) | > loader_time: 0.0047 (0.0047119576560365325)  --> TIME: 2025-02-27 05:48:14 -- STEP: 19554/36938 -- GLOBAL_STEP: 1570950 | > loss_text_ce: 0.026804082095623016 (0.030066120669822404) | > loss_mel_ce: 2.892761707305908 (nan) | > loss: 0.011585578322410583 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13717603806619308) | > loader_time: 0.0047 (0.004711847553056124)  --> TIME: 2025-02-27 05:48:24 -- STEP: 19604/36938 -- GLOBAL_STEP: 1571000 | > loss_text_ce: 0.029925525188446045 (0.030066130799668925) | > loss_mel_ce: 1.848378300666809 (nan) | > loss: 0.007453586906194687 (nan) | > current_lr: 5e-06 | > step_time: 0.163 (0.13717471946382842) | > loader_time: 0.0046 (0.004711863119440195)  --> TIME: 2025-02-27 05:48:33 -- STEP: 19654/36938 -- GLOBAL_STEP: 1571050 | > loss_text_ce: 0.026644403114914894 (0.030068272944831494) | > loss_mel_ce: 2.4788291454315186 (nan) | > loss: 0.009942356497049332 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13717495958402637) | > loader_time: 0.0046 (0.0047117767807570345)  --> TIME: 2025-02-27 05:48:42 -- STEP: 19704/36938 -- GLOBAL_STEP: 1571100 | > loss_text_ce: 0.02766518108546734 (0.030064272163330887) | > loss_mel_ce: 2.5371997356414795 (nan) | > loss: 0.01017803605645895 (nan) | > current_lr: 5e-06 | > step_time: 0.1786 (0.13717336713711184) | > loader_time: 0.0048 (0.004711682906346341)  --> TIME: 2025-02-27 05:48:52 -- STEP: 19754/36938 -- GLOBAL_STEP: 1571150 | > loss_text_ce: 0.0320592001080513 (0.03006323349409045) | > loss_mel_ce: 2.7807118892669678 (nan) | > loss: 0.01116179022938013 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13717562864820598) | > loader_time: 0.0046 (0.004711705409431655)  --> TIME: 2025-02-27 05:49:01 -- STEP: 19804/36938 -- GLOBAL_STEP: 1571200 | > loss_text_ce: 0.027538210153579712 (0.030064023159681538) | > loss_mel_ce: 2.7967453002929688 (nan) | > loss: 0.011207475326955318 (nan) | > current_lr: 5e-06 | > step_time: 0.1517 (0.13716439032068312) | > loader_time: 0.0046 (0.004711758618498286)  --> TIME: 2025-02-27 05:49:10 -- STEP: 19854/36938 -- GLOBAL_STEP: 1571250 | > loss_text_ce: 0.02470257692039013 (0.030064137669849077) | > loss_mel_ce: 3.1590781211853027 (nan) | > loss: 0.012634051032364368 (nan) | > current_lr: 5e-06 | > step_time: 0.1996 (0.13715290792689488) | > loader_time: 0.0049 (0.004711709366448028)  --> TIME: 2025-02-27 05:49:20 -- STEP: 19904/36938 -- GLOBAL_STEP: 1571300 | > loss_text_ce: 0.029210135340690613 (0.03006367226773884) | > loss_mel_ce: 1.190682053565979 (nan) | > loss: 0.004840841982513666 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13716990597713855) | > loader_time: 0.0045 (0.004711751445792483)  --> TIME: 2025-02-27 05:49:29 -- STEP: 19954/36938 -- GLOBAL_STEP: 1571350 | > loss_text_ce: 0.0326867438852787 (0.030063451417950787) | > loss_mel_ce: 3.1660618782043457 (nan) | > loss: 0.012693447060883045 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.1371720288838636) | > loader_time: 0.0049 (0.004711933708076201)  --> TIME: 2025-02-27 05:49:38 -- STEP: 20004/36938 -- GLOBAL_STEP: 1571400 | > loss_text_ce: 0.02313391864299774 (0.030065090133372035) | > loss_mel_ce: 2.7283878326416016 (nan) | > loss: 0.010918738320469856 (nan) | > current_lr: 5e-06 | > step_time: 0.1886 (0.1371752141690503) | > loader_time: 0.0048 (0.004712199657064882)  --> TIME: 2025-02-27 05:49:48 -- STEP: 20054/36938 -- GLOBAL_STEP: 1571450 | > loss_text_ce: 0.03208575025200844 (0.03006281804735417) | > loss_mel_ce: 1.995413064956665 (nan) | > loss: 0.008045630529522896 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13717178189410947) | > loader_time: 0.0046 (0.0047121407254048595)  --> TIME: 2025-02-27 05:49:57 -- STEP: 20104/36938 -- GLOBAL_STEP: 1571500 | > loss_text_ce: 0.029373101890087128 (0.030064278106073417) | > loss_mel_ce: 3.2217752933502197 (nan) | > loss: 0.012901383452117443 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13717244363286557) | > loader_time: 0.0047 (0.004712213724673708)  --> TIME: 2025-02-27 05:50:06 -- STEP: 20154/36938 -- GLOBAL_STEP: 1571550 | > loss_text_ce: 0.03453712537884712 (0.03006094739476818) | > loss_mel_ce: 2.9692134857177734 (nan) | > loss: 0.011919645592570305 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.1371594123735288) | > loader_time: 0.0047 (0.004712149892710277)  --> TIME: 2025-02-27 05:50:15 -- STEP: 20204/36938 -- GLOBAL_STEP: 1571600 | > loss_text_ce: 0.029972607269883156 (0.030062526765261402) | > loss_mel_ce: 2.544318199157715 (nan) | > loss: 0.010215439833700657 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.1371417715275228) | > loader_time: 0.0046 (0.00471219088247189)  --> TIME: 2025-02-27 05:50:24 -- STEP: 20254/36938 -- GLOBAL_STEP: 1571650 | > loss_text_ce: 0.03427902236580849 (0.030062928963074605) | > loss_mel_ce: 1.4484832286834717 (nan) | > loss: 0.00588397728279233 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13714734369903095) | > loader_time: 0.0066 (0.0047121217835393156)  --> TIME: 2025-02-27 05:50:33 -- STEP: 20304/36938 -- GLOBAL_STEP: 1571700 | > loss_text_ce: 0.03801367059350014 (0.03006589668699712) | > loss_mel_ce: 2.8840157985687256 (nan) | > loss: 0.011595355346798897 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13713418763392202) | > loader_time: 0.0046 (0.004712180630065385)  --> TIME: 2025-02-27 05:50:43 -- STEP: 20354/36938 -- GLOBAL_STEP: 1571750 | > loss_text_ce: 0.04204479604959488 (0.03006480245854092) | > loss_mel_ce: 0.027289798483252525 (nan) | > loss: 0.00027513731038197875 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13714043038924786) | > loader_time: 0.0045 (0.004712190224635418)  --> TIME: 2025-02-27 05:50:52 -- STEP: 20404/36938 -- GLOBAL_STEP: 1571800 | > loss_text_ce: 0.029485099017620087 (0.03006072254930242) | > loss_mel_ce: 2.5774757862091064 (nan) | > loss: 0.010345082730054855 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1371331450004858) | > loader_time: 0.0046 (0.004712122441553349)  --> TIME: 2025-02-27 05:51:02 -- STEP: 20454/36938 -- GLOBAL_STEP: 1571850 | > loss_text_ce: 0.03141475468873978 (0.030064886882973135) | > loss_mel_ce: 2.5245840549468994 (nan) | > loss: 0.010142852552235126 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13713190798490463) | > loader_time: 0.0048 (0.004712237761122523)  --> TIME: 2025-02-27 05:51:11 -- STEP: 20504/36938 -- GLOBAL_STEP: 1571900 | > loss_text_ce: 0.030846482142806053 (0.03006441627364839) | > loss_mel_ce: 2.0140390396118164 (nan) | > loss: 0.008114625699818134 (nan) | > current_lr: 5e-06 | > step_time: 0.1816 (0.13713353647880644) | > loader_time: 0.0049 (0.004712298448505307)  --> TIME: 2025-02-27 05:51:20 -- STEP: 20554/36938 -- GLOBAL_STEP: 1571950 | > loss_text_ce: 0.022904014214873314 (0.030067036238227936) | > loss_mel_ce: 1.2986717224121094 (nan) | > loss: 0.005244348663836718 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.1371215561478593) | > loader_time: 0.0046 (0.004712154733725197)  --> TIME: 2025-02-27 05:51:30 -- STEP: 20604/36938 -- GLOBAL_STEP: 1572000 | > loss_text_ce: 0.031605444848537445 (0.03006990263365564) | > loss_mel_ce: 2.6788084506988525 (nan) | > loss: 0.010755611583590508 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13713365322269327) | > loader_time: 0.0046 (0.004712131481174345)  --> TIME: 2025-02-27 05:51:39 -- STEP: 20654/36938 -- GLOBAL_STEP: 1572050 | > loss_text_ce: 0.023078765720129013 (0.03007354094196676) | > loss_mel_ce: 2.879988670349121 (nan) | > loss: 0.01152010913938284 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.1371357939048698) | > loader_time: 0.0048 (0.0047121805917077295)  --> TIME: 2025-02-27 05:51:48 -- STEP: 20704/36938 -- GLOBAL_STEP: 1572100 | > loss_text_ce: 0.029668111354112625 (0.030073658617378367) | > loss_mel_ce: 3.2863309383392334 (nan) | > loss: 0.013158727437257767 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13713222740929165) | > loader_time: 0.0047 (0.004712246980235964)  --> TIME: 2025-02-27 05:51:58 -- STEP: 20754/36938 -- GLOBAL_STEP: 1572150 | > loss_text_ce: 0.0308380126953125 (0.03007376629068433) | > loss_mel_ce: 3.356006145477295 (nan) | > loss: 0.013439858332276344 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13714120239148656) | > loader_time: 0.0047 (0.004712327443141513)  --> TIME: 2025-02-27 05:52:07 -- STEP: 20804/36938 -- GLOBAL_STEP: 1572200 | > loss_text_ce: 0.023495879024267197 (0.030075048372172567) | > loss_mel_ce: 2.340776205062866 (nan) | > loss: 0.009382032789289951 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13713173885983593) | > loader_time: 0.0047 (0.004712500622171179)  --> TIME: 2025-02-27 05:52:17 -- STEP: 20854/36938 -- GLOBAL_STEP: 1572250 | > loss_text_ce: 0.04002624377608299 (0.030074493991544392) | > loss_mel_ce: 1.076572299003601 (nan) | > loss: 0.0044309464283287525 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13714190750208063) | > loader_time: 0.0045 (0.004712581120047712)  --> TIME: 2025-02-27 05:52:26 -- STEP: 20904/36938 -- GLOBAL_STEP: 1572300 | > loss_text_ce: 0.03902297839522362 (0.03007651101197699) | > loss_mel_ce: 1.5818325281143188 (nan) | > loss: 0.006431966554373503 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13713806366902243) | > loader_time: 0.0046 (0.004712416917667319)  --> TIME: 2025-02-27 05:52:35 -- STEP: 20954/36938 -- GLOBAL_STEP: 1572350 | > loss_text_ce: 0.02796768769621849 (0.03007494630908525) | > loss_mel_ce: 2.908104658126831 (nan) | > loss: 0.011651081033051014 (nan) | > current_lr: 5e-06 | > step_time: 0.178 (0.13713647679335034) | > loader_time: 0.0049 (0.004712337663385988)  --> TIME: 2025-02-27 05:52:44 -- STEP: 21004/36938 -- GLOBAL_STEP: 1572400 | > loss_text_ce: 0.03977726399898529 (0.030077770385552264) | > loss_mel_ce: 2.8345158100128174 (nan) | > loss: 0.011405925266444683 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13713580682104703) | > loader_time: 0.0045 (0.004712368449445582)  --> TIME: 2025-02-27 05:52:54 -- STEP: 21054/36938 -- GLOBAL_STEP: 1572450 | > loss_text_ce: 0.02764897607266903 (0.03008100407935822) | > loss_mel_ce: 2.6962382793426514 (nan) | > loss: 0.010809076949954033 (nan) | > current_lr: 5e-06 | > step_time: 0.1271 (0.1371407574983114) | > loader_time: 0.0053 (0.004712510382992096)  --> TIME: 2025-02-27 05:53:04 -- STEP: 21104/36938 -- GLOBAL_STEP: 1572500 | > loss_text_ce: 0.02168199233710766 (0.030083842131181586) | > loss_mel_ce: 1.973336100578308 (nan) | > loss: 0.007916739210486412 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.1371520954248488) | > loader_time: 0.0046 (0.004712652954484039)  --> TIME: 2025-02-27 05:53:13 -- STEP: 21154/36938 -- GLOBAL_STEP: 1572550 | > loss_text_ce: 0.030378276482224464 (0.030084567726908925) | > loss_mel_ce: 3.027937173843384 (nan) | > loss: 0.01213617343455553 (nan) | > current_lr: 5e-06 | > step_time: 0.1632 (0.13714449925470912) | > loader_time: 0.0048 (0.004713005139131879)  --> TIME: 2025-02-27 05:53:22 -- STEP: 21204/36938 -- GLOBAL_STEP: 1572600 | > loss_text_ce: 0.037622082978487015 (0.030085047278216804) | > loss_mel_ce: 3.710460901260376 (nan) | > loss: 0.014873345382511616 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.13715258254754814) | > loader_time: 0.0046 (0.004713093429393442)  --> TIME: 2025-02-27 05:53:32 -- STEP: 21254/36938 -- GLOBAL_STEP: 1572650 | > loss_text_ce: 0.025873595848679543 (0.030082630905797464) | > loss_mel_ce: 3.3812499046325684 (nan) | > loss: 0.013520332053303719 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13715478826962382) | > loader_time: 0.0052 (0.004713289307170452)  --> TIME: 2025-02-27 05:53:41 -- STEP: 21304/36938 -- GLOBAL_STEP: 1572700 | > loss_text_ce: 0.03324620798230171 (0.03008214099056144) | > loss_mel_ce: 2.5244040489196777 (nan) | > loss: 0.010149406269192696 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13715314781177862) | > loader_time: 0.0046 (0.004713603519560739)  --> TIME: 2025-02-27 05:53:50 -- STEP: 21354/36938 -- GLOBAL_STEP: 1572750 | > loss_text_ce: 0.01662762649357319 (0.030079860950968346) | > loss_mel_ce: 2.4556992053985596 (nan) | > loss: 0.009810821153223515 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13715522326344115) | > loader_time: 0.0046 (0.0047138339293921905)  --> TIME: 2025-02-27 05:54:00 -- STEP: 21404/36938 -- GLOBAL_STEP: 1572800 | > loss_text_ce: 0.03153108432888985 (0.03007801403961167) | > loss_mel_ce: 2.5644493103027344 (nan) | > loss: 0.010301509872078896 (nan) | > current_lr: 5e-06 | > step_time: 0.1448 (0.1371613367765425) | > loader_time: 0.0046 (0.004713958188810647)  --> TIME: 2025-02-27 05:54:09 -- STEP: 21454/36938 -- GLOBAL_STEP: 1572850 | > loss_text_ce: 0.034152861684560776 (0.030076689822114835) | > loss_mel_ce: 2.093568801879883 (nan) | > loss: 0.008443339727818966 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13715933065351324) | > loader_time: 0.0055 (0.004714126121056554)  --> TIME: 2025-02-27 05:54:18 -- STEP: 21504/36938 -- GLOBAL_STEP: 1572900 | > loss_text_ce: 0.02893190085887909 (0.030076075667217254) | > loss_mel_ce: 2.071533679962158 (nan) | > loss: 0.008335181511938572 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.13715941430113887) | > loader_time: 0.0046 (0.0047140117357706655)  --> TIME: 2025-02-27 05:54:28 -- STEP: 21554/36938 -- GLOBAL_STEP: 1572950 | > loss_text_ce: 0.033473968505859375 (0.03007280909527348) | > loss_mel_ce: 2.0052058696746826 (nan) | > loss: 0.008089999668300152 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13715570434475843) | > loader_time: 0.0047 (0.00471380798473996)  --> TIME: 2025-02-27 05:54:37 -- STEP: 21604/36938 -- GLOBAL_STEP: 1573000 | > loss_text_ce: 0.026302214711904526 (0.030069085329968825) | > loss_mel_ce: 2.700782299041748 (nan) | > loss: 0.01082176435738802 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13714720312919157) | > loader_time: 0.0045 (0.004713683884533603)  --> TIME: 2025-02-27 05:54:46 -- STEP: 21654/36938 -- GLOBAL_STEP: 1573050 | > loss_text_ce: 0.025245942175388336 (0.030067164027307143) | > loss_mel_ce: 2.532741069793701 (nan) | > loss: 0.010150742717087269 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13714599450825696) | > loader_time: 0.0046 (0.004713673290810409)  --> TIME: 2025-02-27 05:54:55 -- STEP: 21704/36938 -- GLOBAL_STEP: 1573100 | > loss_text_ce: 0.027309749275445938 (0.030068154386856612) | > loss_mel_ce: 2.908653736114502 (nan) | > loss: 0.011650648899376392 (nan) | > current_lr: 5e-06 | > step_time: 0.1605 (0.13714561360681227) | > loader_time: 0.0047 (0.0047136558802682174)  --> TIME: 2025-02-27 05:55:05 -- STEP: 21754/36938 -- GLOBAL_STEP: 1573150 | > loss_text_ce: 0.03243652358651161 (0.03006715881442152) | > loss_mel_ce: 2.7520852088928223 (nan) | > loss: 0.011049690656363964 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.13714651042424122) | > loader_time: 0.0048 (0.00471363504263717)  --> TIME: 2025-02-27 05:55:14 -- STEP: 21804/36938 -- GLOBAL_STEP: 1573200 | > loss_text_ce: 0.030397001653909683 (0.030068517476973783) | > loss_mel_ce: 3.0347938537597656 (nan) | > loss: 0.012163456529378891 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.13714091675234746) | > loader_time: 0.0046 (0.004714045310715868)  --> TIME: 2025-02-27 05:55:24 -- STEP: 21854/36938 -- GLOBAL_STEP: 1573250 | > loss_text_ce: 0.02365318313241005 (0.030067287438669363) | > loss_mel_ce: 2.783416986465454 (nan) | > loss: 0.01113916840404272 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1371530765713414) | > loader_time: 0.0046 (0.0047141011029418955)  --> TIME: 2025-02-27 05:55:33 -- STEP: 21904/36938 -- GLOBAL_STEP: 1573300 | > loss_text_ce: 0.028438927605748177 (0.0300649637867684) | > loss_mel_ce: 3.158043146133423 (nan) | > loss: 0.012644771486520767 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13714179926714487) | > loader_time: 0.0054 (0.004714260077807372)  --> TIME: 2025-02-27 05:55:42 -- STEP: 21954/36938 -- GLOBAL_STEP: 1573350 | > loss_text_ce: 0.022940482944250107 (0.030065465847745) | > loss_mel_ce: 3.030266046524048 (nan) | > loss: 0.012115899473428726 (nan) | > current_lr: 5e-06 | > step_time: 0.1456 (0.1371470077925936) | > loader_time: 0.0047 (0.004714501863013742)  --> TIME: 2025-02-27 05:55:52 -- STEP: 22004/36938 -- GLOBAL_STEP: 1573400 | > loss_text_ce: 0.03761615976691246 (0.030067259927044582) | > loss_mel_ce: 2.9419944286346436 (nan) | > loss: 0.01182385254651308 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.1371447196918669) | > loader_time: 0.0046 (0.004714432293188546)  --> TIME: 2025-02-27 05:56:01 -- STEP: 22054/36938 -- GLOBAL_STEP: 1573450 | > loss_text_ce: 0.03421410173177719 (0.030065220188131753) | > loss_mel_ce: 2.5553038120269775 (nan) | > loss: 0.010275864973664284 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1371526845381268) | > loader_time: 0.0045 (0.004714344239055447)  --> TIME: 2025-02-27 05:56:11 -- STEP: 22104/36938 -- GLOBAL_STEP: 1573500 | > loss_text_ce: 0.02873382531106472 (0.030065167685192875) | > loss_mel_ce: 3.181180953979492 (nan) | > loss: 0.0127377575263381 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13716063137465181) | > loader_time: 0.0046 (0.0047141651808676015)  --> TIME: 2025-02-27 05:56:20 -- STEP: 22154/36938 -- GLOBAL_STEP: 1573550 | > loss_text_ce: 0.03469248488545418 (0.030066510473115766) | > loss_mel_ce: 2.380918264389038 (nan) | > loss: 0.009585757739841938 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13716682799512256) | > loader_time: 0.0045 (0.004714091321110559)  --> TIME: 2025-02-27 05:56:30 -- STEP: 22204/36938 -- GLOBAL_STEP: 1573600 | > loss_text_ce: 0.02853691764175892 (0.030064638331536674) | > loss_mel_ce: 2.4835379123687744 (nan) | > loss: 0.009968551807105541 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13716610920921019) | > loader_time: 0.0045 (0.004713935210798837)  --> TIME: 2025-02-27 05:56:39 -- STEP: 22254/36938 -- GLOBAL_STEP: 1573650 | > loss_text_ce: 0.03171312063932419 (0.030069588006104042) | > loss_mel_ce: 2.0296630859375 (nan) | > loss: 0.008180064149200916 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13716512909365813) | > loader_time: 0.0046 (0.004713863045997621)  --> TIME: 2025-02-27 05:56:48 -- STEP: 22304/36938 -- GLOBAL_STEP: 1573700 | > loss_text_ce: 0.023325655609369278 (0.030072936053319173) | > loss_mel_ce: 3.053497552871704 (nan) | > loss: 0.012209616601467133 (nan) | > current_lr: 5e-06 | > step_time: 0.178 (0.13715249313899414) | > loader_time: 0.0049 (0.004713681990139084)  --> TIME: 2025-02-27 05:56:58 -- STEP: 22354/36938 -- GLOBAL_STEP: 1573750 | > loss_text_ce: 0.029713770374655724 (0.030073638225272245) | > loss_mel_ce: 3.001758337020874 (nan) | > loss: 0.01202965248376131 (nan) | > current_lr: 5e-06 | > step_time: 0.1895 (0.13716319310832603) | > loader_time: 0.0049 (0.004713838286227884)  --> TIME: 2025-02-27 05:57:07 -- STEP: 22404/36938 -- GLOBAL_STEP: 1573800 | > loss_text_ce: 0.02524215169250965 (0.030074853181209296) | > loss_mel_ce: 1.931227684020996 (nan) | > loss: 0.007763769943267107 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13717008878920406) | > loader_time: 0.0045 (0.0047137806552198114)  --> TIME: 2025-02-27 05:57:17 -- STEP: 22454/36938 -- GLOBAL_STEP: 1573850 | > loss_text_ce: 0.035050421953201294 (0.03007685812378895) | > loss_mel_ce: 2.752927541732788 (nan) | > loss: 0.011063405312597752 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13717534851707258) | > loader_time: 0.0045 (0.0047138088732928355)  --> TIME: 2025-02-27 05:57:26 -- STEP: 22504/36938 -- GLOBAL_STEP: 1573900 | > loss_text_ce: 0.03278758376836777 (0.030075199436623977) | > loss_mel_ce: 3.070626974105835 (nan) | > loss: 0.012315137311816216 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13717915510266815) | > loader_time: 0.0046 (0.00471375622530655)  --> TIME: 2025-02-27 05:57:36 -- STEP: 22554/36938 -- GLOBAL_STEP: 1573950 | > loss_text_ce: 0.038588013499975204 (0.030074631679233126) | > loss_mel_ce: 1.7909674644470215 (nan) | > loss: 0.0072601414285600185 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.1371856789497177) | > loader_time: 0.0045 (0.004713745545101989)  --> TIME: 2025-02-27 05:57:45 -- STEP: 22604/36938 -- GLOBAL_STEP: 1574000 | > loss_text_ce: 0.03237967565655708 (0.030074024610029386) | > loss_mel_ce: 3.0292890071868896 (nan) | > loss: 0.012149479240179062 (nan) | > current_lr: 5e-06 | > step_time: 0.1886 (0.137190805000872) | > loader_time: 0.0048 (0.004713684272985473)  --> TIME: 2025-02-27 05:57:54 -- STEP: 22654/36938 -- GLOBAL_STEP: 1574050 | > loss_text_ce: 0.03277135640382767 (0.030072758839010748) | > loss_mel_ce: 3.0885169506073 (nan) | > loss: 0.012386064976453781 (nan) | > current_lr: 5e-06 | > step_time: 0.1814 (0.13718263030715264) | > loader_time: 0.0055 (0.004713901124649238)  --> TIME: 2025-02-27 05:58:04 -- STEP: 22704/36938 -- GLOBAL_STEP: 1574100 | > loss_text_ce: 0.032939694821834564 (0.030072467190308177) | > loss_mel_ce: 2.8874852657318115 (nan) | > loss: 0.011588988825678825 (nan) | > current_lr: 5e-06 | > step_time: 0.1565 (0.13719037846203386) | > loader_time: 0.0047 (0.004713890122392763)  --> TIME: 2025-02-27 05:58:13 -- STEP: 22754/36938 -- GLOBAL_STEP: 1574150 | > loss_text_ce: 0.026538260281085968 (0.030072374274298336) | > loss_mel_ce: 2.737523078918457 (nan) | > loss: 0.010968498885631561 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13717856323969504) | > loader_time: 0.0046 (0.0047137681321140976)  --> TIME: 2025-02-27 05:58:22 -- STEP: 22804/36938 -- GLOBAL_STEP: 1574200 | > loss_text_ce: 0.034047503024339676 (0.030069075101149954) | > loss_mel_ce: 3.0834715366363525 (nan) | > loss: 0.012371108867228031 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13718309911212065) | > loader_time: 0.0047 (0.004713873500636579)  --> TIME: 2025-02-27 05:58:32 -- STEP: 22854/36938 -- GLOBAL_STEP: 1574250 | > loss_text_ce: 0.03368857875466347 (0.030071402301472632) | > loss_mel_ce: 2.5314929485321045 (nan) | > loss: 0.010179292410612106 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13718056803959003) | > loader_time: 0.0053 (0.004714252598863374)  --> TIME: 2025-02-27 05:58:41 -- STEP: 22904/36938 -- GLOBAL_STEP: 1574300 | > loss_text_ce: 0.03995921462774277 (0.030073602096133887) | > loss_mel_ce: 2.860607385635376 (nan) | > loss: 0.011510185897350311 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.13718299312146923) | > loader_time: 0.0047 (0.004714328333751046)  --> TIME: 2025-02-27 05:58:50 -- STEP: 22954/36938 -- GLOBAL_STEP: 1574350 | > loss_text_ce: 0.029240986332297325 (0.03007467537709788) | > loss_mel_ce: 2.657881498336792 (nan) | > loss: 0.010663185268640518 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.1371843639209095) | > loader_time: 0.0048 (0.004714434857545843)  --> TIME: 2025-02-27 05:59:00 -- STEP: 23004/36938 -- GLOBAL_STEP: 1574400 | > loss_text_ce: 0.029444832354784012 (0.030075577500184286) | > loss_mel_ce: 2.278517246246338 (nan) | > loss: 0.00915858056396246 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.1371901495169409) | > loader_time: 0.0046 (0.004714312179879144)  --> TIME: 2025-02-27 05:59:09 -- STEP: 23054/36938 -- GLOBAL_STEP: 1574450 | > loss_text_ce: 0.02536935545504093 (0.030069389231549063) | > loss_mel_ce: 2.7240447998046875 (nan) | > loss: 0.010910374112427235 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13719552480782607) | > loader_time: 0.0047 (0.0047141464335491825)  --> TIME: 2025-02-27 05:59:19 -- STEP: 23104/36938 -- GLOBAL_STEP: 1574500 | > loss_text_ce: 0.03428163379430771 (0.030071083998306457) | > loss_mel_ce: 3.1101694107055664 (nan) | > loss: 0.012477981857955456 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13719434075449727) | > loader_time: 0.0046 (0.004714269861785295)  --> TIME: 2025-02-27 05:59:28 -- STEP: 23154/36938 -- GLOBAL_STEP: 1574550 | > loss_text_ce: 0.028973914682865143 (0.03007020445358659) | > loss_mel_ce: 3.093982219696045 (nan) | > loss: 0.012392682954668999 (nan) | > current_lr: 5e-06 | > step_time: 0.176 (0.13719179705036416) | > loader_time: 0.0048 (0.004714281661755302)  --> TIME: 2025-02-27 05:59:37 -- STEP: 23204/36938 -- GLOBAL_STEP: 1574600 | > loss_text_ce: 0.040718741714954376 (0.03007121389355527) | > loss_mel_ce: 2.471426248550415 (nan) | > loss: 0.009968830272555351 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.1371926936049486) | > loader_time: 0.0046 (0.004714316621851718)  --> TIME: 2025-02-27 05:59:47 -- STEP: 23254/36938 -- GLOBAL_STEP: 1574650 | > loss_text_ce: 0.030717264860868454 (0.030070455561213308) | > loss_mel_ce: 2.6579208374023438 (nan) | > loss: 0.010669199749827385 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.13719559457097721) | > loader_time: 0.0047 (0.0047144219401040995)  --> TIME: 2025-02-27 05:59:56 -- STEP: 23304/36938 -- GLOBAL_STEP: 1574700 | > loss_text_ce: 0.028477713465690613 (0.030068595465206325) | > loss_mel_ce: 1.4236563444137573 (nan) | > loss: 0.005762436892837286 (nan) | > current_lr: 5e-06 | > step_time: 0.1528 (0.1371890038016574) | > loader_time: 0.0047 (0.004714263823692859)  --> TIME: 2025-02-27 06:00:05 -- STEP: 23354/36938 -- GLOBAL_STEP: 1574750 | > loss_text_ce: 0.034803327172994614 (0.030071425758219424) | > loss_mel_ce: 2.039768695831299 (nan) | > loss: 0.00823242962360382 (nan) | > current_lr: 5e-06 | > step_time: 0.1497 (0.13719103705283828) | > loader_time: 0.0047 (0.004714180562168871)  --> TIME: 2025-02-27 06:00:14 -- STEP: 23404/36938 -- GLOBAL_STEP: 1574800 | > loss_text_ce: 0.022494342178106308 (0.030072322327340726) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1371747869810398) | > loader_time: 0.0045 (0.00471395129852671)  --> TIME: 2025-02-27 06:00:24 -- STEP: 23454/36938 -- GLOBAL_STEP: 1574850 | > loss_text_ce: 0.020402099937200546 (0.03007089935152674) | > loss_mel_ce: 2.7509994506835938 (nan) | > loss: 0.01099762599915266 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13717495041083913) | > loader_time: 0.0046 (0.004713788640014209)  --> TIME: 2025-02-27 06:00:33 -- STEP: 23504/36938 -- GLOBAL_STEP: 1574900 | > loss_text_ce: 0.031690847128629684 (0.030072160420740096) | > loss_mel_ce: 1.3826732635498047 (nan) | > loss: 0.005612556356936693 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13717473181151044) | > loader_time: 0.0046 (0.00471393955735788)  --> TIME: 2025-02-27 06:00:42 -- STEP: 23554/36938 -- GLOBAL_STEP: 1574950 | > loss_text_ce: 0.024668965488672256 (0.0300731038043616) | > loss_mel_ce: 4.26953125 (nan) | > loss: 0.017040478065609932 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13716309300563878) | > loader_time: 0.0045 (0.0047137337345577535)  --> TIME: 2025-02-27 06:00:51 -- STEP: 23604/36938 -- GLOBAL_STEP: 1575000 | > loss_text_ce: 0.020761407911777496 (0.030074376653916338) | > loss_mel_ce: 2.293562650680542 (nan) | > loss: 0.009183826856315136 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.1371584715347865) | > loader_time: 0.0046 (0.004713914410943909)  --> TIME: 2025-02-27 06:01:00 -- STEP: 23654/36938 -- GLOBAL_STEP: 1575050 | > loss_text_ce: 0.02666485495865345 (0.030073182611511484) | > loss_mel_ce: 2.448676824569702 (nan) | > loss: 0.0098227858543396 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13715275274162123) | > loader_time: 0.0052 (0.004714118987838405)  --> TIME: 2025-02-27 06:01:10 -- STEP: 23704/36938 -- GLOBAL_STEP: 1575100 | > loss_text_ce: 0.033757857978343964 (0.03007553200112276) | > loss_mel_ce: 3.156984806060791 (nan) | > loss: 0.012661677785217762 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13717196854111194) | > loader_time: 0.0046 (0.004714391620184078)  --> TIME: 2025-02-27 06:01:19 -- STEP: 23754/36938 -- GLOBAL_STEP: 1575150 | > loss_text_ce: 0.02795712649822235 (0.030072343367425345) | > loss_mel_ce: 3.025857925415039 (nan) | > loss: 0.012118314392864704 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.1371640032606618) | > loader_time: 0.0047 (0.004714328090259486)  --> TIME: 2025-02-27 06:01:28 -- STEP: 23804/36938 -- GLOBAL_STEP: 1575200 | > loss_text_ce: 0.02910042740404606 (0.030071977222322827) | > loss_mel_ce: 2.8870978355407715 (nan) | > loss: 0.01157221570611 (nan) | > current_lr: 5e-06 | > step_time: 0.1513 (0.13715498282356162) | > loader_time: 0.0046 (0.0047141936241608725)  --> TIME: 2025-02-27 06:01:38 -- STEP: 23854/36938 -- GLOBAL_STEP: 1575250 | > loss_text_ce: 0.03013506345450878 (0.030071409257913434) | > loss_mel_ce: 2.2065086364746094 (nan) | > loss: 0.008875571191310883 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.137162978984769) | > loader_time: 0.0046 (0.004714335121512347)  --> TIME: 2025-02-27 06:01:47 -- STEP: 23904/36938 -- GLOBAL_STEP: 1575300 | > loss_text_ce: 0.01951846480369568 (0.030072938192022843) | > loss_mel_ce: 2.0980093479156494 (nan) | > loss: 0.008402888663113117 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.1371600215075171) | > loader_time: 0.0053 (0.004714461265398621)  --> TIME: 2025-02-27 06:01:57 -- STEP: 23954/36938 -- GLOBAL_STEP: 1575350 | > loss_text_ce: 0.029790891334414482 (0.03007344754948609) | > loss_mel_ce: 2.4195010662078857 (nan) | > loss: 0.00971941277384758 (nan) | > current_lr: 5e-06 | > step_time: 0.1624 (0.1371672618983341) | > loader_time: 0.0048 (0.004714569305352385)  --> TIME: 2025-02-27 06:02:06 -- STEP: 24004/36938 -- GLOBAL_STEP: 1575400 | > loss_text_ce: 0.030389565974473953 (0.030074471815784414) | > loss_mel_ce: 0.0429840087890625 (nan) | > loss: 0.0002911649935413152 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13715635406198298) | > loader_time: 0.0045 (0.004714619912737252)  --> TIME: 2025-02-27 06:02:15 -- STEP: 24054/36938 -- GLOBAL_STEP: 1575450 | > loss_text_ce: 0.03907545655965805 (0.03007642719055629) | > loss_mel_ce: 2.063619613647461 (nan) | > loss: 0.008344028145074844 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.137155193030473) | > loader_time: 0.0044 (0.00471443377439542)  --> TIME: 2025-02-27 06:02:25 -- STEP: 24104/36938 -- GLOBAL_STEP: 1575500 | > loss_text_ce: 0.017896834760904312 (0.030077040376076912) | > loss_mel_ce: 0.1393229216337204 (nan) | > loss: 0.0006238879286684096 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1371519913160645) | > loader_time: 0.0053 (0.004714660101699685)  --> TIME: 2025-02-27 06:02:34 -- STEP: 24154/36938 -- GLOBAL_STEP: 1575550 | > loss_text_ce: 0.032343968749046326 (0.030075529595743535) | > loss_mel_ce: 0.0533905029296875 (nan) | > loss: 0.0003402161819394678 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.1371466675133982) | > loader_time: 0.0046 (0.00471474001657747)  --> TIME: 2025-02-27 06:02:43 -- STEP: 24204/36938 -- GLOBAL_STEP: 1575600 | > loss_text_ce: 0.03507591038942337 (0.03007514141652754) | > loss_mel_ce: 2.282459020614624 (nan) | > loss: 0.009196567349135876 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13714665367551737) | > loader_time: 0.0046 (0.004714805997909327)  --> TIME: 2025-02-27 06:02:53 -- STEP: 24254/36938 -- GLOBAL_STEP: 1575650 | > loss_text_ce: 0.021416887640953064 (0.030074486880453916) | > loss_mel_ce: 2.2067973613739014 (nan) | > loss: 0.008842120878398418 (nan) | > current_lr: 5e-06 | > step_time: 0.1514 (0.1371532277635472) | > loader_time: 0.0047 (0.004714841794287391)  --> TIME: 2025-02-27 06:03:02 -- STEP: 24304/36938 -- GLOBAL_STEP: 1575700 | > loss_text_ce: 0.021178964525461197 (0.030073973529007553) | > loss_mel_ce: 2.471776247024536 (nan) | > loss: 0.009892679750919342 (nan) | > current_lr: 5e-06 | > step_time: 0.193 (0.13715450191003434) | > loader_time: 0.0049 (0.0047147940890894)  --> TIME: 2025-02-27 06:03:11 -- STEP: 24354/36938 -- GLOBAL_STEP: 1575750 | > loss_text_ce: 0.034599557518959045 (0.03007438845988392) | > loss_mel_ce: 2.0586466789245605 (nan) | > loss: 0.008306533098220825 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13715270980782684) | > loader_time: 0.0046 (0.004714885427220114)  --> TIME: 2025-02-27 06:03:21 -- STEP: 24404/36938 -- GLOBAL_STEP: 1575800 | > loss_text_ce: 0.02571995183825493 (0.030072619554617545) | > loss_mel_ce: 3.02479887008667 (nan) | > loss: 0.01210523396730423 (nan) | > current_lr: 5e-06 | > step_time: 0.1495 (0.13715635582260077) | > loader_time: 0.0047 (0.004715001928945016)  --> TIME: 2025-02-27 06:03:30 -- STEP: 24454/36938 -- GLOBAL_STEP: 1575850 | > loss_text_ce: 0.022333361208438873 (0.03007093534953868) | > loss_mel_ce: 2.931460380554199 (nan) | > loss: 0.011721404269337654 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.1371578037314694) | > loader_time: 0.0046 (0.004715174541378216)  --> TIME: 2025-02-27 06:03:40 -- STEP: 24504/36938 -- GLOBAL_STEP: 1575900 | > loss_text_ce: 0.028899721801280975 (0.030071522300670558) | > loss_mel_ce: 0.9010902643203735 (nan) | > loss: 0.003690436715260148 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13716417141416884) | > loader_time: 0.005 (0.004715179603083459)  --> TIME: 2025-02-27 06:03:49 -- STEP: 24554/36938 -- GLOBAL_STEP: 1575950 | > loss_text_ce: 0.0351719856262207 (0.030071289171434777) | > loss_mel_ce: 2.2194619178771973 (nan) | > loss: 0.008946960791945457 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13716273911954616) | > loader_time: 0.0046 (0.00471530228044869)  --> TIME: 2025-02-27 06:03:58 -- STEP: 24604/36938 -- GLOBAL_STEP: 1576000 | > loss_text_ce: 0.02911759354174137 (0.030072047596333577) | > loss_mel_ce: 2.5117876529693604 (nan) | > loss: 0.01008295826613903 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.13716271752711187) | > loader_time: 0.0047 (0.004715502242734989)  --> TIME: 2025-02-27 06:04:08 -- STEP: 24654/36938 -- GLOBAL_STEP: 1576050 | > loss_text_ce: 0.02235986664891243 (0.030069987693486625) | > loss_mel_ce: 3.246680736541748 (nan) | > loss: 0.012972383759915829 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1371601748816572) | > loader_time: 0.0047 (0.004715430849687568)  --> TIME: 2025-02-27 06:04:17 -- STEP: 24704/36938 -- GLOBAL_STEP: 1576100 | > loss_text_ce: 0.033039454370737076 (0.030068198045883225) | > loss_mel_ce: 2.0187885761260986 (nan) | > loss: 0.00814217608422041 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1371658477507575) | > loader_time: 0.0045 (0.0047153643684683545)  --> TIME: 2025-02-27 06:04:26 -- STEP: 24754/36938 -- GLOBAL_STEP: 1576150 | > loss_text_ce: 0.027400663122534752 (0.030066616396311575) | > loss_mel_ce: 2.5425384044647217 (nan) | > loss: 0.010198172181844711 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.137161911318874) | > loader_time: 0.0045 (0.0047152047782565544)  --> TIME: 2025-02-27 06:04:36 -- STEP: 24804/36938 -- GLOBAL_STEP: 1576200 | > loss_text_ce: 0.029217662289738655 (0.030063176678841005) | > loss_mel_ce: 2.574093818664551 (nan) | > loss: 0.01033060159534216 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.13716946598253554) | > loader_time: 0.0046 (0.004715169125882818)  --> TIME: 2025-02-27 06:04:46 -- STEP: 24854/36938 -- GLOBAL_STEP: 1576250 | > loss_text_ce: 0.028617648407816887 (0.03006333852234538) | > loss_mel_ce: 3.045779228210449 (nan) | > loss: 0.012199988588690758 (nan) | > current_lr: 5e-06 | > step_time: 0.1679 (0.13717315727854615) | > loader_time: 0.0049 (0.00471547438074425)  --> TIME: 2025-02-27 06:04:55 -- STEP: 24904/36938 -- GLOBAL_STEP: 1576300 | > loss_text_ce: 0.02859463170170784 (0.030063840293884488) | > loss_mel_ce: 2.583932399749756 (nan) | > loss: 0.010367171838879585 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13718352526895458) | > loader_time: 0.0046 (0.004715893464267641)  --> TIME: 2025-02-27 06:05:04 -- STEP: 24954/36938 -- GLOBAL_STEP: 1576350 | > loss_text_ce: 0.03043753281235695 (0.030064132683801825) | > loss_mel_ce: 2.13077974319458 (nan) | > loss: 0.008576259016990662 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13717136559238396) | > loader_time: 0.0046 (0.004716121157726955)  --> TIME: 2025-02-27 06:05:14 -- STEP: 25004/36938 -- GLOBAL_STEP: 1576400 | > loss_text_ce: 0.02456357516348362 (0.030064435855475368) | > loss_mel_ce: 2.563951015472412 (nan) | > loss: 0.010271883569657803 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13717010734062615) | > loader_time: 0.0045 (0.004716274233326631)  --> TIME: 2025-02-27 06:05:23 -- STEP: 25054/36938 -- GLOBAL_STEP: 1576450 | > loss_text_ce: 0.023467807099223137 (0.03006175327954267) | > loss_mel_ce: 2.3828532695770264 (nan) | > loss: 0.009548893198370934 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13716606124301808) | > loader_time: 0.0045 (0.004716330289441001)  --> TIME: 2025-02-27 06:05:32 -- STEP: 25104/36938 -- GLOBAL_STEP: 1576500 | > loss_text_ce: 0.03175458684563637 (0.03006096187542481) | > loss_mel_ce: 2.6812658309936523 (nan) | > loss: 0.010765953920781612 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13715871394516346) | > loader_time: 0.0046 (0.004716164333338626)  --> TIME: 2025-02-27 06:05:41 -- STEP: 25154/36938 -- GLOBAL_STEP: 1576550 | > loss_text_ce: 0.021363167092204094 (0.030059429579072688) | > loss_mel_ce: 2.616366147994995 (nan) | > loss: 0.010467180982232094 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13715176942703558) | > loader_time: 0.0047 (0.004716123876429842)  --> TIME: 2025-02-27 06:05:51 -- STEP: 25204/36938 -- GLOBAL_STEP: 1576600 | > loss_text_ce: 0.02511625923216343 (0.03005782945469655) | > loss_mel_ce: 2.7304396629333496 (nan) | > loss: 0.010934745892882347 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.1371536267936996) | > loader_time: 0.0048 (0.0047162720994747874)  --> TIME: 2025-02-27 06:06:00 -- STEP: 25254/36938 -- GLOBAL_STEP: 1576650 | > loss_text_ce: 0.0280134417116642 (0.030058972813495708) | > loss_mel_ce: 2.4118094444274902 (nan) | > loss: 0.009681837633252144 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.1371564732751948) | > loader_time: 0.0045 (0.00471615399526587)  --> TIME: 2025-02-27 06:06:10 -- STEP: 25304/36938 -- GLOBAL_STEP: 1576700 | > loss_text_ce: 0.03371932730078697 (0.030060418509502657) | > loss_mel_ce: 3.147550106048584 (nan) | > loss: 0.012624085880815983 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13716371253325996) | > loader_time: 0.0047 (0.004716092890815706)  --> TIME: 2025-02-27 06:06:19 -- STEP: 25354/36938 -- GLOBAL_STEP: 1576750 | > loss_text_ce: 0.03274298831820488 (0.030062331052919014) | > loss_mel_ce: 1.0293172597885132 (nan) | > loss: 0.00421452522277832 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.1371560003363093) | > loader_time: 0.0045 (0.0047158936629716815)  --> TIME: 2025-02-27 06:06:28 -- STEP: 25404/36938 -- GLOBAL_STEP: 1576800 | > loss_text_ce: 0.02771933749318123 (0.030060107240665553) | > loss_mel_ce: 2.6848702430725098 (nan) | > loss: 0.010764244943857193 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13715545569155446) | > loader_time: 0.0049 (0.004715878838461172)  --> TIME: 2025-02-27 06:06:37 -- STEP: 25454/36938 -- GLOBAL_STEP: 1576850 | > loss_text_ce: 0.025852903723716736 (0.0300605579760977) | > loss_mel_ce: 2.9773905277252197 (nan) | > loss: 0.01191763300448656 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13715683432285547) | > loader_time: 0.0046 (0.004715862142662212)  --> TIME: 2025-02-27 06:06:47 -- STEP: 25504/36938 -- GLOBAL_STEP: 1576900 | > loss_text_ce: 0.028103098273277283 (0.030059234664380027) | > loss_mel_ce: 2.683555841445923 (nan) | > loss: 0.010760552249848843 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.1371546477415931) | > loader_time: 0.0053 (0.004715898395556859)  --> TIME: 2025-02-27 06:06:56 -- STEP: 25554/36938 -- GLOBAL_STEP: 1576950 | > loss_text_ce: 0.03132222220301628 (0.030060790260092424) | > loss_mel_ce: 0.921875 (nan) | > loss: 0.003782528918236494 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.1371583484414693) | > loader_time: 0.0045 (0.004715728677661688)  --> TIME: 2025-02-27 06:07:05 -- STEP: 25604/36938 -- GLOBAL_STEP: 1577000 | > loss_text_ce: 0.029359634965658188 (0.030061615614634146) | > loss_mel_ce: 3.3264055252075195 (nan) | > loss: 0.013316528871655464 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13715488863929973) | > loader_time: 0.0046 (0.004715579577748265)  --> TIME: 2025-02-27 06:07:15 -- STEP: 25654/36938 -- GLOBAL_STEP: 1577050 | > loss_text_ce: 0.03965950012207031 (0.030062273624959012) | > loss_mel_ce: 2.497187614440918 (nan) | > loss: 0.01006685383617878 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13715178428235655) | > loader_time: 0.0048 (0.0047156996354129465)  --> TIME: 2025-02-27 06:07:24 -- STEP: 25704/36938 -- GLOBAL_STEP: 1577100 | > loss_text_ce: 0.027947217226028442 (0.03005946283292827) | > loss_mel_ce: 2.250825881958008 (nan) | > loss: 0.009042751044034958 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.1371551452583145) | > loader_time: 0.0046 (0.004715604821803803)  --> TIME: 2025-02-27 06:07:33 -- STEP: 25754/36938 -- GLOBAL_STEP: 1577150 | > loss_text_ce: 0.027237974107265472 (0.030061027675496347) | > loss_mel_ce: 2.2038817405700684 (nan) | > loss: 0.008853649720549583 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13715561068353346) | > loader_time: 0.0045 (0.00471584696183525)  --> TIME: 2025-02-27 06:07:43 -- STEP: 25804/36938 -- GLOBAL_STEP: 1577200 | > loss_text_ce: 0.02551981434226036 (0.030061035364271602) | > loss_mel_ce: 3.0779545307159424 (nan) | > loss: 0.012315374799072742 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13716240766121984) | > loader_time: 0.0047 (0.0047159404130780805)  --> TIME: 2025-02-27 06:07:52 -- STEP: 25854/36938 -- GLOBAL_STEP: 1577250 | > loss_text_ce: 0.03736267611384392 (0.030062094460846456) | > loss_mel_ce: 2.8357505798339844 (nan) | > loss: 0.011401243507862091 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.1371621558826134) | > loader_time: 0.0046 (0.004716000664286702)  --> TIME: 2025-02-27 06:08:01 -- STEP: 25904/36938 -- GLOBAL_STEP: 1577300 | > loss_text_ce: 0.029459979385137558 (0.03006235353228805) | > loss_mel_ce: 1.7548820972442627 (nan) | > loss: 0.007080722600221634 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1371483060958726) | > loader_time: 0.0045 (0.004716030742519768)  --> TIME: 2025-02-27 06:08:11 -- STEP: 25954/36938 -- GLOBAL_STEP: 1577350 | > loss_text_ce: 0.01567850261926651 (0.030063118777922705) | > loss_mel_ce: 0.9774017333984375 (nan) | > loss: 0.003940795082598925 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13715258024594468) | > loader_time: 0.0045 (0.0047160774513009774)  --> TIME: 2025-02-27 06:08:20 -- STEP: 26004/36938 -- GLOBAL_STEP: 1577400 | > loss_text_ce: 0.03560753911733627 (0.030063823554111156) | > loss_mel_ce: 1.9740355014801025 (nan) | > loss: 0.007974774576723576 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13715221115927045) | > loader_time: 0.0044 (0.004716104552335407)  --> TIME: 2025-02-27 06:08:29 -- STEP: 26054/36938 -- GLOBAL_STEP: 1577450 | > loss_text_ce: 0.031883109360933304 (0.03006330425244797) | > loss_mel_ce: 2.4806318283081055 (nan) | > loss: 0.00997029710561037 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13715239956436226) | > loader_time: 0.0048 (0.004716164071790297)  --> TIME: 2025-02-27 06:08:39 -- STEP: 26104/36938 -- GLOBAL_STEP: 1577500 | > loss_text_ce: 0.03284875303506851 (0.030063485700330704) | > loss_mel_ce: 2.567830801010132 (nan) | > loss: 0.010320157743990421 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13715672396166792) | > loader_time: 0.0054 (0.0047161993240966375)  --> TIME: 2025-02-27 06:08:48 -- STEP: 26154/36938 -- GLOBAL_STEP: 1577550 | > loss_text_ce: 0.027918823063373566 (0.03006258589873492) | > loss_mel_ce: 3.120584011077881 (nan) | > loss: 0.012494059279561043 (nan) | > current_lr: 5e-06 | > step_time: 0.1693 (0.1371670929018378) | > loader_time: 0.0049 (0.004716605242033206)  --> TIME: 2025-02-27 06:08:58 -- STEP: 26204/36938 -- GLOBAL_STEP: 1577600 | > loss_text_ce: 0.03733509033918381 (0.030061536149671716) | > loss_mel_ce: 2.3025758266448975 (nan) | > loss: 0.009285361506044865 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13717004408856015) | > loader_time: 0.0046 (0.004716778280097719)  --> TIME: 2025-02-27 06:09:07 -- STEP: 26254/36938 -- GLOBAL_STEP: 1577650 | > loss_text_ce: 0.0311529990285635 (0.03006129213138422) | > loss_mel_ce: 3.115931272506714 (nan) | > loss: 0.012488430365920067 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.1371690720953163) | > loader_time: 0.0047 (0.004716867556745924)  --> TIME: 2025-02-27 06:09:17 -- STEP: 26304/36938 -- GLOBAL_STEP: 1577700 | > loss_text_ce: 0.03540285304188728 (0.030060359416703025) | > loss_mel_ce: 2.4059908390045166 (nan) | > loss: 0.00968807004392147 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13716797232446565) | > loader_time: 0.0047 (0.00471684635773189)  --> TIME: 2025-02-27 06:09:26 -- STEP: 26354/36938 -- GLOBAL_STEP: 1577750 | > loss_text_ce: 0.029106736183166504 (0.030061949206548092) | > loss_mel_ce: 2.4298548698425293 (nan) | > loss: 0.009757784195244312 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.137162267732052) | > loader_time: 0.0045 (0.004716954182771485)  --> TIME: 2025-02-27 06:09:35 -- STEP: 26404/36938 -- GLOBAL_STEP: 1577800 | > loss_text_ce: 0.02785254269838333 (0.03006088460475382) | > loss_mel_ce: 3.030474901199341 (nan) | > loss: 0.012136220932006836 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13716828943183784) | > loader_time: 0.0046 (0.0047171914185597475)  --> TIME: 2025-02-27 06:09:44 -- STEP: 26454/36938 -- GLOBAL_STEP: 1577850 | > loss_text_ce: 0.022090192884206772 (0.03006275762987484) | > loss_mel_ce: 3.521477222442627 (nan) | > loss: 0.014061775989830494 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13716185257856084) | > loader_time: 0.0047 (0.004717468458336986)  --> TIME: 2025-02-27 06:09:54 -- STEP: 26504/36938 -- GLOBAL_STEP: 1577900 | > loss_text_ce: 0.02866203710436821 (0.030063919137039305) | > loss_mel_ce: 2.9447052478790283 (nan) | > loss: 0.011799076572060585 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13715638151494083) | > loader_time: 0.0046 (0.00471738711282846)  --> TIME: 2025-02-27 06:10:03 -- STEP: 26554/36938 -- GLOBAL_STEP: 1577950 | > loss_text_ce: 0.034498509019613266 (0.0300620008923273) | > loss_mel_ce: 2.8568806648254395 (nan) | > loss: 0.01147372741252184 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13715301665626223) | > loader_time: 0.0046 (0.004717378234920712)  --> TIME: 2025-02-27 06:10:12 -- STEP: 26604/36938 -- GLOBAL_STEP: 1578000 | > loss_text_ce: 0.03826047480106354 (0.030062589386059803) | > loss_mel_ce: 3.090191602706909 (nan) | > loss: 0.012414492666721344 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.1371514956729964) | > loader_time: 0.0046 (0.004717413186489243)  --> TIME: 2025-02-27 06:10:21 -- STEP: 26654/36938 -- GLOBAL_STEP: 1578050 | > loss_text_ce: 0.03013293258845806 (0.030064128486235808) | > loss_mel_ce: 2.4282379150390625 (nan) | > loss: 0.00975544098764658 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13715086605307877) | > loader_time: 0.0046 (0.004717217263111296)  --> TIME: 2025-02-27 06:10:31 -- STEP: 26704/36938 -- GLOBAL_STEP: 1578100 | > loss_text_ce: 0.029091214761137962 (0.030064830588928906) | > loss_mel_ce: 1.063658595085144 (nan) | > loss: 0.004336309153586626 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.13714851059550778) | > loader_time: 0.0047 (0.004717085570757633)  --> TIME: 2025-02-27 06:10:40 -- STEP: 26754/36938 -- GLOBAL_STEP: 1578150 | > loss_text_ce: 0.031883154064416885 (0.030062591048535953) | > loss_mel_ce: 3.2748403549194336 (nan) | > loss: 0.013121919706463814 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13716067589299766) | > loader_time: 0.0048 (0.0047169956131135225)  --> TIME: 2025-02-27 06:10:50 -- STEP: 26804/36938 -- GLOBAL_STEP: 1578200 | > loss_text_ce: 0.031073566526174545 (0.030064773965062547) | > loss_mel_ce: 2.7746095657348633 (nan) | > loss: 0.011133664287626743 (nan) | > current_lr: 5e-06 | > step_time: 0.1494 (0.1371588503988868) | > loader_time: 0.0047 (0.004717002393878781)  --> TIME: 2025-02-27 06:10:59 -- STEP: 26854/36938 -- GLOBAL_STEP: 1578250 | > loss_text_ce: 0.03409649804234505 (0.03006621183598417) | > loss_mel_ce: 1.8471347093582153 (nan) | > loss: 0.007465203758329153 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1371580649001106) | > loader_time: 0.0052 (0.004717234285996247)  --> TIME: 2025-02-27 06:11:08 -- STEP: 26904/36938 -- GLOBAL_STEP: 1578300 | > loss_text_ce: 0.034023575484752655 (0.030065823596221078) | > loss_mel_ce: 1.6629537343978882 (nan) | > loss: 0.006734037306159735 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1371614473971304) | > loader_time: 0.0046 (0.004717324014366485)  --> TIME: 2025-02-27 06:11:18 -- STEP: 26954/36938 -- GLOBAL_STEP: 1578350 | > loss_text_ce: 0.03195991739630699 (0.030066133453695555) | > loss_mel_ce: 2.3126742839813232 (nan) | > loss: 0.009304105304181576 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13715396180761072) | > loader_time: 0.0046 (0.00471742133530962)  --> TIME: 2025-02-27 06:11:27 -- STEP: 27004/36938 -- GLOBAL_STEP: 1578400 | > loss_text_ce: 0.05440060794353485 (0.03006792508454108) | > loss_mel_ce: 2.549407958984375 (nan) | > loss: 0.010332574136555195 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1371587169927906) | > loader_time: 0.0044 (0.004717601394497909)  --> TIME: 2025-02-27 06:11:37 -- STEP: 27054/36938 -- GLOBAL_STEP: 1578450 | > loss_text_ce: 0.031033772975206375 (0.03006791242881918) | > loss_mel_ce: 2.998039722442627 (nan) | > loss: 0.012020133435726166 (nan) | > current_lr: 5e-06 | > step_time: 0.1502 (0.13716240285055717) | > loader_time: 0.0048 (0.00471754456391272)  --> TIME: 2025-02-27 06:11:46 -- STEP: 27104/36938 -- GLOBAL_STEP: 1578500 | > loss_text_ce: 0.029198002070188522 (0.030068794796147786) | > loss_mel_ce: 0.462890625 (nan) | > loss: 0.0019527326803654432 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.1371570335175519) | > loader_time: 0.0047 (0.004717450769268471)  --> TIME: 2025-02-27 06:11:55 -- STEP: 27154/36938 -- GLOBAL_STEP: 1578550 | > loss_text_ce: 0.032881248742341995 (0.030068148777302205) | > loss_mel_ce: 2.119183301925659 (nan) | > loss: 0.008539939299225807 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13714954514491848) | > loader_time: 0.0046 (0.004717351858733683)  --> TIME: 2025-02-27 06:12:04 -- STEP: 27204/36938 -- GLOBAL_STEP: 1578600 | > loss_text_ce: 0.028954356908798218 (0.030068302920718503) | > loss_mel_ce: 3.0540404319763184 (nan) | > loss: 0.012234106659889221 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.1371392097355351) | > loader_time: 0.0046 (0.004717346141135349)  --> TIME: 2025-02-27 06:12:13 -- STEP: 27254/36938 -- GLOBAL_STEP: 1578650 | > loss_text_ce: 0.020424991846084595 (0.030068115989196537) | > loss_mel_ce: 2.272346019744873 (nan) | > loss: 0.009098298847675323 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13713334295539148) | > loader_time: 0.0046 (0.004717243184012445)  --> TIME: 2025-02-27 06:12:23 -- STEP: 27304/36938 -- GLOBAL_STEP: 1578700 | > loss_text_ce: 0.03210439532995224 (0.030068160879486147) | > loss_mel_ce: 2.8547470569610596 (nan) | > loss: 0.011455760337412357 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13713605607181112) | > loader_time: 0.0045 (0.004717525641589138)  --> TIME: 2025-02-27 06:12:32 -- STEP: 27354/36938 -- GLOBAL_STEP: 1578750 | > loss_text_ce: 0.026834892109036446 (0.030068570029187572) | > loss_mel_ce: 2.428439140319824 (nan) | > loss: 0.009743152186274529 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13713403809547314) | > loader_time: 0.0047 (0.004717610580858221)  --> TIME: 2025-02-27 06:12:41 -- STEP: 27404/36938 -- GLOBAL_STEP: 1578800 | > loss_text_ce: 0.029876217246055603 (0.03006912261646142) | > loss_mel_ce: 2.5324084758758545 (nan) | > loss: 0.010167797096073627 (nan) | > current_lr: 5e-06 | > step_time: 0.1114 (0.13712426526340063) | > loader_time: 0.0045 (0.004717678653812789)  --> TIME: 2025-02-27 06:12:50 -- STEP: 27454/36938 -- GLOBAL_STEP: 1578850 | > loss_text_ce: 0.026989052072167397 (0.030069702739300916) | > loss_mel_ce: 1.9642261266708374 (nan) | > loss: 0.007901648059487343 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.1371157520882215) | > loader_time: 0.0046 (0.004717607269598332)  --> TIME: 2025-02-27 06:12:59 -- STEP: 27504/36938 -- GLOBAL_STEP: 1578900 | > loss_text_ce: 0.025252636522054672 (0.030071538307786246) | > loss_mel_ce: 1.4880495071411133 (nan) | > loss: 0.006005167495459318 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13711776648100296) | > loader_time: 0.0046 (0.0047176484194101235)  --> TIME: 2025-02-27 06:13:09 -- STEP: 27554/36938 -- GLOBAL_STEP: 1578950 | > loss_text_ce: 0.028524266555905342 (0.03007129270491388) | > loss_mel_ce: 2.9704840183258057 (nan) | > loss: 0.01190082635730505 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.13712753414233742) | > loader_time: 0.0047 (0.0047177614109725485)  --> TIME: 2025-02-27 06:13:18 -- STEP: 27604/36938 -- GLOBAL_STEP: 1579000 | > loss_text_ce: 0.03388874605298042 (0.03006964053478823) | > loss_mel_ce: 2.4680159091949463 (nan) | > loss: 0.009928193874657154 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13712758639466452) | > loader_time: 0.0045 (0.004717751614098388)  --> TIME: 2025-02-27 06:13:27 -- STEP: 27654/36938 -- GLOBAL_STEP: 1579050 | > loss_text_ce: 0.025195660069584846 (0.03006745900862391) | > loss_mel_ce: 1.4031609296798706 (nan) | > loss: 0.005668082274496555 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13711595398196652) | > loader_time: 0.0053 (0.004717957243240322)  --> TIME: 2025-02-27 06:13:36 -- STEP: 27704/36938 -- GLOBAL_STEP: 1579100 | > loss_text_ce: 0.01896357722580433 (0.03006355929966796) | > loss_mel_ce: 1.6038169860839844 (nan) | > loss: 0.006439605727791786 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1371075623199972) | > loader_time: 0.0052 (0.0047181580595428785)  --> TIME: 2025-02-27 06:13:46 -- STEP: 27754/36938 -- GLOBAL_STEP: 1579150 | > loss_text_ce: 0.034505777060985565 (0.030063993478000057) | > loss_mel_ce: 2.9832046031951904 (nan) | > loss: 0.011975042521953583 (nan) | > current_lr: 5e-06 | > step_time: 0.1617 (0.1371078593608767) | > loader_time: 0.0047 (0.004718144018831977)  --> TIME: 2025-02-27 06:13:55 -- STEP: 27804/36938 -- GLOBAL_STEP: 1579200 | > loss_text_ce: 0.026478709653019905 (0.030061697774094284) | > loss_mel_ce: 2.170786142349243 (nan) | > loss: 0.008719305507838726 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1370947537636893) | > loader_time: 0.0046 (0.0047181403357388)  --> TIME: 2025-02-27 06:14:04 -- STEP: 27854/36938 -- GLOBAL_STEP: 1579250 | > loss_text_ce: 0.030591292306780815 (0.03006166476767903) | > loss_mel_ce: 2.821178436279297 (nan) | > loss: 0.01131654717028141 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.137087225503141) | > loader_time: 0.0047 (0.0047182881618989)  --> TIME: 2025-02-27 06:14:13 -- STEP: 27904/36938 -- GLOBAL_STEP: 1579300 | > loss_text_ce: 0.03271712362766266 (0.03006270773187955) | > loss_mel_ce: 2.769294023513794 (nan) | > loss: 0.01111909281462431 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13708096448707965) | > loader_time: 0.0046 (0.0047185250361441245)  --> TIME: 2025-02-27 06:14:22 -- STEP: 27954/36938 -- GLOBAL_STEP: 1579350 | > loss_text_ce: 0.02445302903652191 (0.030060489828322996) | > loss_mel_ce: 2.6234042644500732 (nan) | > loss: 0.010507370345294476 (nan) | > current_lr: 5e-06 | > step_time: 0.1871 (0.13708576981564297) | > loader_time: 0.005 (0.004718573749973711)  --> TIME: 2025-02-27 06:14:31 -- STEP: 28004/36938 -- GLOBAL_STEP: 1579400 | > loss_text_ce: 0.037760112434625626 (0.030060054708758166) | > loss_mel_ce: 2.8937551975250244 (nan) | > loss: 0.011632997542619705 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13707132672194164) | > loader_time: 0.0045 (0.004718530673569307)  --> TIME: 2025-02-27 06:14:41 -- STEP: 28054/36938 -- GLOBAL_STEP: 1579450 | > loss_text_ce: 0.032581210136413574 (0.030060853855910143) | > loss_mel_ce: 1.8718960285186768 (nan) | > loss: 0.007557449862360954 (nan) | > current_lr: 5e-06 | > step_time: 0.1486 (0.1370756479019085) | > loader_time: 0.0054 (0.004718704455473484)  --> TIME: 2025-02-27 06:14:50 -- STEP: 28104/36938 -- GLOBAL_STEP: 1579500 | > loss_text_ce: 0.032066166400909424 (0.030062181927849492) | > loss_mel_ce: 3.0525221824645996 (nan) | > loss: 0.01224043034017086 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13708514532505328) | > loader_time: 0.0053 (0.004718925872827687)  --> TIME: 2025-02-27 06:14:59 -- STEP: 28154/36938 -- GLOBAL_STEP: 1579550 | > loss_text_ce: 0.031003287062048912 (0.030062126423126865) | > loss_mel_ce: 1.2050102949142456 (nan) | > loss: 0.004904815927147865 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13708079156782718) | > loader_time: 0.0046 (0.004718996147764921)  --> TIME: 2025-02-27 06:15:09 -- STEP: 28204/36938 -- GLOBAL_STEP: 1579600 | > loss_text_ce: 0.025088854134082794 (0.030062263349407033) | > loss_mel_ce: 1.5534169673919678 (nan) | > loss: 0.006263912655413151 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13707821169840823) | > loader_time: 0.0045 (0.00471904345935597)  --> TIME: 2025-02-27 06:15:18 -- STEP: 28254/36938 -- GLOBAL_STEP: 1579650 | > loss_text_ce: 0.025470566004514694 (0.030064394181370635) | > loss_mel_ce: 2.127797842025757 (nan) | > loss: 0.00854471605271101 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13706885300059235) | > loader_time: 0.0054 (0.004719052875405635)  --> TIME: 2025-02-27 06:15:27 -- STEP: 28304/36938 -- GLOBAL_STEP: 1579700 | > loss_text_ce: 0.03266262635588646 (0.03006733413600445) | > loss_mel_ce: 3.231104612350464 (nan) | > loss: 0.012951457872986794 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13707447775580017) | > loader_time: 0.0046 (0.004719015836311348)  --> TIME: 2025-02-27 06:15:37 -- STEP: 28354/36938 -- GLOBAL_STEP: 1579750 | > loss_text_ce: 0.025920940563082695 (0.030065985866977667) | > loss_mel_ce: 2.4479713439941406 (nan) | > loss: 0.0098170330747962 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13707789178542648) | > loader_time: 0.0046 (0.004718877015130962)  --> TIME: 2025-02-27 06:15:46 -- STEP: 28404/36938 -- GLOBAL_STEP: 1579800 | > loss_text_ce: 0.025049323216080666 (0.030066401192671458) | > loss_mel_ce: 3.33203125 (nan) | > loss: 0.013321748934686184 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13707914163219803) | > loader_time: 0.0045 (0.004718852259712858)  --> TIME: 2025-02-27 06:15:56 -- STEP: 28454/36938 -- GLOBAL_STEP: 1579850 | > loss_text_ce: 0.02839316613972187 (0.03006738421602492) | > loss_mel_ce: 2.0818328857421875 (nan) | > loss: 0.00837391335517168 (nan) | > current_lr: 5e-06 | > step_time: 0.1491 (0.1370834251846754) | > loader_time: 0.0047 (0.004718811863747514)  --> TIME: 2025-02-27 06:16:05 -- STEP: 28504/36938 -- GLOBAL_STEP: 1579900 | > loss_text_ce: 0.03312119096517563 (0.030067237601772975) | > loss_mel_ce: 1.4880123138427734 (nan) | > loss: 0.006036244332790375 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.1370838514094997) | > loader_time: 0.0046 (0.004718871313034044)  --> TIME: 2025-02-27 06:16:14 -- STEP: 28554/36938 -- GLOBAL_STEP: 1579950 | > loss_text_ce: 0.03523813560605049 (0.030068115676491185) | > loss_mel_ce: 3.079108476638794 (nan) | > loss: 0.012358518317341805 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.1370775861600497) | > loader_time: 0.0047 (0.004718841595952626)  --> TIME: 2025-02-27 06:16:23 -- STEP: 28604/36938 -- GLOBAL_STEP: 1580000 | > loss_text_ce: 0.03358112648129463 (0.030069583328847776) | > loss_mel_ce: 2.7310187816619873 (nan) | > loss: 0.010970634408295155 (nan) | > current_lr: 5e-06 | > step_time: 0.1862 (0.1370704561876185) | > loader_time: 0.0048 (0.004718924732312253) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1580000.pth  --> TIME: 2025-02-27 06:16:38 -- STEP: 28654/36938 -- GLOBAL_STEP: 1580050 | > loss_text_ce: 0.031534548848867416 (0.030069847210602466) | > loss_mel_ce: 0.8691895604133606 (nan) | > loss: 0.00357430218718946 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13706360490155448) | > loader_time: 0.0045 (0.004718805462748067)  --> TIME: 2025-02-27 06:16:48 -- STEP: 28704/36938 -- GLOBAL_STEP: 1580100 | > loss_text_ce: 0.023541554808616638 (0.030070163282822613) | > loss_mel_ce: 3.084527015686035 (nan) | > loss: 0.01233360543847084 (nan) | > current_lr: 5e-06 | > step_time: 0.1446 (0.13705975222614086) | > loader_time: 0.0047 (0.00471867789559006)  --> TIME: 2025-02-27 06:16:57 -- STEP: 28754/36938 -- GLOBAL_STEP: 1580150 | > loss_text_ce: 0.03078615851700306 (0.030067416191110273) | > loss_mel_ce: 0.3599853515625 (nan) | > loss: 0.0015506807249039412 (nan) | > current_lr: 5e-06 | > step_time: 0.1807 (0.13705959400994944) | > loader_time: 0.0048 (0.00471854879866555)  --> TIME: 2025-02-27 06:17:06 -- STEP: 28804/36938 -- GLOBAL_STEP: 1580200 | > loss_text_ce: 0.025867534801363945 (0.030070921827627974) | > loss_mel_ce: 4.186849117279053 (nan) | > loss: 0.016717130318284035 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13706559262297474) | > loader_time: 0.0046 (0.004718568950473471)  --> TIME: 2025-02-27 06:17:15 -- STEP: 28854/36938 -- GLOBAL_STEP: 1580250 | > loss_text_ce: 0.018898125737905502 (0.0300704875765573) | > loss_mel_ce: 0.27634844183921814 (nan) | > loss: 0.0011716134613379836 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13705889255050047) | > loader_time: 0.0046 (0.004718417799744638)  --> TIME: 2025-02-27 06:17:25 -- STEP: 28904/36938 -- GLOBAL_STEP: 1580300 | > loss_text_ce: 0.03208640590310097 (0.030070317045962962) | > loss_mel_ce: 2.6867096424102783 (nan) | > loss: 0.0107888737693429 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13705377725725976) | > loader_time: 0.0046 (0.004718310881479254)  --> TIME: 2025-02-27 06:17:34 -- STEP: 28954/36938 -- GLOBAL_STEP: 1580350 | > loss_text_ce: 0.030951546505093575 (0.030069908755816947) | > loss_mel_ce: 2.3729248046875 (nan) | > loss: 0.009539192542433739 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13704807089533938) | > loader_time: 0.0045 (0.004718236586595817)  --> TIME: 2025-02-27 06:17:43 -- STEP: 29004/36938 -- GLOBAL_STEP: 1580400 | > loss_text_ce: 0.026476504281163216 (0.03006788229545325) | > loss_mel_ce: 2.1799473762512207 (nan) | > loss: 0.00875565130263567 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.1370570916107576) | > loader_time: 0.0046 (0.004718112174502188)  --> TIME: 2025-02-27 06:17:53 -- STEP: 29054/36938 -- GLOBAL_STEP: 1580450 | > loss_text_ce: 0.029220907017588615 (0.03006760957689947) | > loss_mel_ce: 0.894683837890625 (nan) | > loss: 0.003666288685053587 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13705638316293536) | > loader_time: 0.0045 (0.004717941752581005)  --> TIME: 2025-02-27 06:18:02 -- STEP: 29104/36938 -- GLOBAL_STEP: 1580500 | > loss_text_ce: 0.026218656450510025 (0.030066298339657363) | > loss_mel_ce: 2.9313018321990967 (nan) | > loss: 0.011736192740499973 (nan) | > current_lr: 5e-06 | > step_time: 0.173 (0.1370543126977356) | > loader_time: 0.0049 (0.004717983956714439)  --> TIME: 2025-02-27 06:18:11 -- STEP: 29154/36938 -- GLOBAL_STEP: 1580550 | > loss_text_ce: 0.032111089676618576 (0.030064321365988807) | > loss_mel_ce: 0.017181396484375 (nan) | > loss: 0.00019560511282179505 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13705491550241644) | > loader_time: 0.0047 (0.0047181949470238745)  --> TIME: 2025-02-27 06:18:21 -- STEP: 29204/36938 -- GLOBAL_STEP: 1580600 | > loss_text_ce: 0.02646193653345108 (0.030062732744838627) | > loss_mel_ce: 2.659827947616577 (nan) | > loss: 0.010659880936145782 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13705495823214972) | > loader_time: 0.0046 (0.0047182670489939165)  --> TIME: 2025-02-27 06:18:30 -- STEP: 29254/36938 -- GLOBAL_STEP: 1580650 | > loss_text_ce: 0.029200170189142227 (0.03006285431511102) | > loss_mel_ce: 2.6451499462127686 (nan) | > loss: 0.010612500831484795 (nan) | > current_lr: 5e-06 | > step_time: 0.1441 (0.13705685843346846) | > loader_time: 0.0046 (0.0047183014799339375)  --> TIME: 2025-02-27 06:18:40 -- STEP: 29304/36938 -- GLOBAL_STEP: 1580700 | > loss_text_ce: 0.0308759156614542 (0.030064668801140625) | > loss_mel_ce: 2.6368699073791504 (nan) | > loss: 0.010586293414235115 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1370613312838406) | > loader_time: 0.0052 (0.004718622385462917)  --> TIME: 2025-02-27 06:18:49 -- STEP: 29354/36938 -- GLOBAL_STEP: 1580750 | > loss_text_ce: 0.032499849796295166 (0.03006499600737595) | > loss_mel_ce: 0.014470417983829975 (nan) | > loss: 0.00018638995243236423 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13706336989100257) | > loader_time: 0.0047 (0.0047185517156644536)  --> TIME: 2025-02-27 06:18:59 -- STEP: 29404/36938 -- GLOBAL_STEP: 1580800 | > loss_text_ce: 0.029979489743709564 (0.030066809709548243) | > loss_mel_ce: 1.9490859508514404 (nan) | > loss: 0.007853434421122074 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13707083637998774) | > loader_time: 0.0046 (0.00471848268084679)  --> TIME: 2025-02-27 06:19:08 -- STEP: 29454/36938 -- GLOBAL_STEP: 1580850 | > loss_text_ce: 0.03598881512880325 (0.03006631892155574) | > loss_mel_ce: 2.270798921585083 (nan) | > loss: 0.009153920225799084 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13706427434153323) | > loader_time: 0.0046 (0.004718273034234647)  --> TIME: 2025-02-27 06:19:17 -- STEP: 29504/36938 -- GLOBAL_STEP: 1580900 | > loss_text_ce: 0.028760001063346863 (0.030064486316294756) | > loss_mel_ce: 2.93705677986145 (nan) | > loss: 0.01176911499351263 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.1370697073767729) | > loader_time: 0.0048 (0.004718159880982024)  --> TIME: 2025-02-27 06:19:27 -- STEP: 29554/36938 -- GLOBAL_STEP: 1580950 | > loss_text_ce: 0.023577965795993805 (0.030065410901709975) | > loss_mel_ce: 1.4277403354644775 (nan) | > loss: 0.005759199615567923 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13706912985994815) | > loader_time: 0.0045 (0.0047181893921337086)  --> TIME: 2025-02-27 06:19:36 -- STEP: 29604/36938 -- GLOBAL_STEP: 1581000 | > loss_text_ce: 0.028295354917645454 (0.03006632691273475) | > loss_mel_ce: 2.645555019378662 (nan) | > loss: 0.010610518045723438 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13706791228562482) | > loader_time: 0.0045 (0.004718213166083645)  --> TIME: 2025-02-27 06:19:45 -- STEP: 29654/36938 -- GLOBAL_STEP: 1581050 | > loss_text_ce: 0.027007540687918663 (0.030065193502739958) | > loss_mel_ce: 1.8574304580688477 (nan) | > loss: 0.007477929349988699 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13706718950734023) | > loader_time: 0.0047 (0.0047181619751708054)  --> TIME: 2025-02-27 06:19:54 -- STEP: 29704/36938 -- GLOBAL_STEP: 1581100 | > loss_text_ce: 0.03317735344171524 (0.030065808955414074) | > loss_mel_ce: 2.6569502353668213 (nan) | > loss: 0.010675109922885895 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.13706207785072086) | > loader_time: 0.0047 (0.004718266228861522)  --> TIME: 2025-02-27 06:20:04 -- STEP: 29754/36938 -- GLOBAL_STEP: 1581150 | > loss_text_ce: 0.03685968369245529 (0.030065258901872137) | > loss_mel_ce: 3.205658197402954 (nan) | > loss: 0.012867134995758533 (nan) | > current_lr: 5e-06 | > step_time: 0.1711 (0.13705574095132936) | > loader_time: 0.0048 (0.00471826589114723)  --> TIME: 2025-02-27 06:20:13 -- STEP: 29804/36938 -- GLOBAL_STEP: 1581200 | > loss_text_ce: 0.033342473208904266 (0.030065766162267263) | > loss_mel_ce: 2.4173362255096436 (nan) | > loss: 0.009724915958940983 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13705603181395826) | > loader_time: 0.0046 (0.004718463671413795)  --> TIME: 2025-02-27 06:20:22 -- STEP: 29854/36938 -- GLOBAL_STEP: 1581250 | > loss_text_ce: 0.02794337272644043 (0.03006369480219018) | > loss_mel_ce: 2.7003939151763916 (nan) | > loss: 0.010826735757291317 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13705296657613578) | > loader_time: 0.0053 (0.0047185034140791535)  --> TIME: 2025-02-27 06:20:32 -- STEP: 29904/36938 -- GLOBAL_STEP: 1581300 | > loss_text_ce: 0.03161713853478432 (0.03006462617775108) | > loss_mel_ce: 2.590519905090332 (nan) | > loss: 0.010405306704342365 (nan) | > current_lr: 5e-06 | > step_time: 0.1829 (0.1370583340693051) | > loader_time: 0.0057 (0.004718694196084125)  --> TIME: 2025-02-27 06:20:41 -- STEP: 29954/36938 -- GLOBAL_STEP: 1581350 | > loss_text_ce: 0.02395361289381981 (0.030064245363000456) | > loss_mel_ce: 3.1057419776916504 (nan) | > loss: 0.012419427745044231 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.13706273058987845) | > loader_time: 0.0046 (0.004718870619010816)  --> TIME: 2025-02-27 06:20:50 -- STEP: 30004/36938 -- GLOBAL_STEP: 1581400 | > loss_text_ce: 0.028896359726786613 (0.030061009108011186) | > loss_mel_ce: 2.7107555866241455 (nan) | > loss: 0.01087163481861353 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.1370520245041974) | > loader_time: 0.0046 (0.004718865438900423)  --> TIME: 2025-02-27 06:20:59 -- STEP: 30054/36938 -- GLOBAL_STEP: 1581450 | > loss_text_ce: 0.03441759571433067 (0.030060727881951146) | > loss_mel_ce: 1.0300551652908325 (nan) | > loss: 0.00422409875318408 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1370437005850224) | > loader_time: 0.0045 (0.0047188079499147825)  --> TIME: 2025-02-27 06:21:08 -- STEP: 30104/36938 -- GLOBAL_STEP: 1581500 | > loss_text_ce: 0.031191548332571983 (0.030060419817052316) | > loss_mel_ce: 2.470139741897583 (nan) | > loss: 0.009925918653607368 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13704467009047086) | > loader_time: 0.0053 (0.004718963600301101)  --> TIME: 2025-02-27 06:21:18 -- STEP: 30154/36938 -- GLOBAL_STEP: 1581550 | > loss_text_ce: 0.027679642662405968 (0.030060729700378288) | > loss_mel_ce: 2.6096179485321045 (nan) | > loss: 0.010465467348694801 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.13705212260902358) | > loader_time: 0.0049 (0.004719057331083919)  --> TIME: 2025-02-27 06:21:27 -- STEP: 30204/36938 -- GLOBAL_STEP: 1581600 | > loss_text_ce: 0.01809806004166603 (0.030061613287368585) | > loss_mel_ce: 1.910400390625 (nan) | > loss: 0.0076527721248567104 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13705164599049172) | > loader_time: 0.0047 (0.0047190738756788106)  --> TIME: 2025-02-27 06:21:37 -- STEP: 30254/36938 -- GLOBAL_STEP: 1581650 | > loss_text_ce: 0.03214379772543907 (0.03006274464499919) | > loss_mel_ce: 2.666367530822754 (nan) | > loss: 0.010708378627896309 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13705270735784178) | > loader_time: 0.0054 (0.004719180842341765)  --> TIME: 2025-02-27 06:21:46 -- STEP: 30304/36938 -- GLOBAL_STEP: 1581700 | > loss_text_ce: 0.033337488770484924 (0.03006332597921545) | > loss_mel_ce: 1.4568965435028076 (nan) | > loss: 0.0059136273339390755 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13705846387991258) | > loader_time: 0.0047 (0.004719324976426097)  --> TIME: 2025-02-27 06:21:56 -- STEP: 30354/36938 -- GLOBAL_STEP: 1581750 | > loss_text_ce: 0.025594033300876617 (0.030061169929727922) | > loss_mel_ce: 1.8614792823791504 (nan) | > loss: 0.0074883862398564816 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13705832563763293) | > loader_time: 0.0047 (0.004719406325111588)  --> TIME: 2025-02-27 06:22:05 -- STEP: 30404/36938 -- GLOBAL_STEP: 1581800 | > loss_text_ce: 0.01760539785027504 (0.030060877373457126) | > loss_mel_ce: 0.028076171875 (nan) | > loss: 0.00018127607472706586 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1370538156482043) | > loader_time: 0.0044 (0.004719336963518541)  --> TIME: 2025-02-27 06:22:14 -- STEP: 30454/36938 -- GLOBAL_STEP: 1581850 | > loss_text_ce: 0.02817419171333313 (0.0300628809458374) | > loss_mel_ce: 2.7799017429351807 (nan) | > loss: 0.011143159121274948 (nan) | > current_lr: 5e-06 | > step_time: 0.1642 (0.13705600120644343) | > loader_time: 0.0049 (0.004719370668931944)  --> TIME: 2025-02-27 06:22:24 -- STEP: 30504/36938 -- GLOBAL_STEP: 1581900 | > loss_text_ce: 0.0202080849558115 (0.030064399571618036) | > loss_mel_ce: 0.4666152000427246 (nan) | > loss: 0.0019318385748192668 (nan) | > current_lr: 5e-06 | > step_time: 0.181 (0.13706034639204903) | > loader_time: 0.0055 (0.004719574394233557)  --> TIME: 2025-02-27 06:22:33 -- STEP: 30554/36938 -- GLOBAL_STEP: 1581950 | > loss_text_ce: 0.03737876936793327 (0.030066206380584083) | > loss_mel_ce: 2.2199172973632812 (nan) | > loss: 0.00895752478390932 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.13705293100807345) | > loader_time: 0.0049 (0.004719775704849961)  --> TIME: 2025-02-27 06:22:42 -- STEP: 30604/36938 -- GLOBAL_STEP: 1582000 | > loss_text_ce: 0.0259237140417099 (0.030065625617394415) | > loss_mel_ce: 2.990086317062378 (nan) | > loss: 0.011968294158577919 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.13705942617056602) | > loader_time: 0.0046 (0.004719983922190279)  --> TIME: 2025-02-27 06:22:52 -- STEP: 30654/36938 -- GLOBAL_STEP: 1582050 | > loss_text_ce: 0.03165823593735695 (0.030067272161845675) | > loss_mel_ce: 3.0889084339141846 (nan) | > loss: 0.012383201159536839 (nan) | > current_lr: 5e-06 | > step_time: 0.156 (0.1370624016197236) | > loader_time: 0.0053 (0.004720059192179835)  --> TIME: 2025-02-27 06:23:01 -- STEP: 30704/36938 -- GLOBAL_STEP: 1582100 | > loss_text_ce: 0.02676382288336754 (0.03006782833911872) | > loss_mel_ce: 2.9642295837402344 (nan) | > loss: 0.011869022622704506 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.13705776146515466) | > loader_time: 0.0047 (0.004719963160388582)  --> TIME: 2025-02-27 06:23:11 -- STEP: 30754/36938 -- GLOBAL_STEP: 1582150 | > loss_text_ce: 0.030991043895483017 (0.030069342150083438) | > loss_mel_ce: 2.9812259674072266 (nan) | > loss: 0.011953243054449558 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13706013800814548) | > loader_time: 0.0049 (0.0047200784700592805)  --> TIME: 2025-02-27 06:23:20 -- STEP: 30804/36938 -- GLOBAL_STEP: 1582200 | > loss_text_ce: 0.03702758625149727 (0.03007041816529827) | > loss_mel_ce: 1.8979861736297607 (nan) | > loss: 0.007678626570850611 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13705605257642545) | > loader_time: 0.0046 (0.004720258822674515)  --> TIME: 2025-02-27 06:23:29 -- STEP: 30854/36938 -- GLOBAL_STEP: 1582250 | > loss_text_ce: 0.031166929751634598 (0.030070951967786603) | > loss_mel_ce: 2.370039463043213 (nan) | > loss: 0.009528597816824913 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13705672212140607) | > loader_time: 0.0046 (0.004720131368161681)  --> TIME: 2025-02-27 06:23:39 -- STEP: 30904/36938 -- GLOBAL_STEP: 1582300 | > loss_text_ce: 0.03301806375384331 (0.030072547021973976) | > loss_mel_ce: 2.8577260971069336 (nan) | > loss: 0.011471208184957504 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.1370662952561834) | > loader_time: 0.0047 (0.004720211414793361)  --> TIME: 2025-02-27 06:23:48 -- STEP: 30954/36938 -- GLOBAL_STEP: 1582350 | > loss_text_ce: 0.031883690506219864 (0.030072998538447356) | > loss_mel_ce: 2.414930820465088 (nan) | > loss: 0.009709581732749939 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13707186106356245) | > loader_time: 0.0046 (0.004720392249971791)  --> TIME: 2025-02-27 06:23:58 -- STEP: 31004/36938 -- GLOBAL_STEP: 1582400 | > loss_text_ce: 0.04257076233625412 (0.0300746411966617) | > loss_mel_ce: 1.443603515625 (nan) | > loss: 0.005897517316043377 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1370739320634545) | > loader_time: 0.0045 (0.004720341857825977)  --> TIME: 2025-02-27 06:24:07 -- STEP: 31054/36938 -- GLOBAL_STEP: 1582450 | > loss_text_ce: 0.029177632182836533 (0.03007421870626545) | > loss_mel_ce: 2.414308786392212 (nan) | > loss: 0.009696375578641891 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13706830518017013) | > loader_time: 0.0046 (0.00472039710978633)  --> TIME: 2025-02-27 06:24:16 -- STEP: 31104/36938 -- GLOBAL_STEP: 1582500 | > loss_text_ce: 0.023827338591217995 (0.030072914969445894) | > loss_mel_ce: 1.850948691368103 (nan) | > loss: 0.007439587730914354 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13707209497291564) | > loader_time: 0.0045 (0.0047203667093942)  --> TIME: 2025-02-27 06:24:26 -- STEP: 31154/36938 -- GLOBAL_STEP: 1582550 | > loss_text_ce: 0.016605312004685402 (0.030070422140795562) | > loss_mel_ce: 1.9719852209091187 (nan) | > loss: 0.007891232147812843 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13707019033485224) | > loader_time: 0.0045 (0.004720193947770421)  --> TIME: 2025-02-27 06:24:35 -- STEP: 31204/36938 -- GLOBAL_STEP: 1582600 | > loss_text_ce: 0.022006353363394737 (0.030071349478957808) | > loss_mel_ce: 2.3185739517211914 (nan) | > loss: 0.009288017638027668 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.1370641040108227) | > loader_time: 0.0045 (0.00472002770713987)  --> TIME: 2025-02-27 06:24:44 -- STEP: 31254/36938 -- GLOBAL_STEP: 1582650 | > loss_text_ce: 0.02926403470337391 (0.030070228942531602) | > loss_mel_ce: 2.601017951965332 (nan) | > loss: 0.01043762732297182 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13706338628358894) | > loader_time: 0.0046 (0.004719977416377012)  --> TIME: 2025-02-27 06:24:54 -- STEP: 31304/36938 -- GLOBAL_STEP: 1582700 | > loss_text_ce: 0.02016836777329445 (0.03006955430989132) | > loss_mel_ce: 2.3156023025512695 (nan) | > loss: 0.009268931113183498 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1370698611601638) | > loader_time: 0.0045 (0.004720046868751987)  --> TIME: 2025-02-27 06:25:03 -- STEP: 31354/36938 -- GLOBAL_STEP: 1582750 | > loss_text_ce: 0.02407425083220005 (0.03006986600168405) | > loss_mel_ce: 2.8031182289123535 (nan) | > loss: 0.011219019070267677 (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.1370736612853566) | > loader_time: 0.0048 (0.004720208321997341)  --> TIME: 2025-02-27 06:25:12 -- STEP: 31404/36938 -- GLOBAL_STEP: 1582800 | > loss_text_ce: 0.036429088562726974 (0.030067338829751787) | > loss_mel_ce: 2.562953472137451 (nan) | > loss: 0.010315011255443096 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13707181982531833) | > loader_time: 0.0044 (0.004720177624335225)  --> TIME: 2025-02-27 06:25:22 -- STEP: 31454/36938 -- GLOBAL_STEP: 1582850 | > loss_text_ce: 0.02684064581990242 (0.030070617779667866) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13707337609430076) | > loader_time: 0.0046 (0.004720122533570073)  --> TIME: 2025-02-27 06:25:31 -- STEP: 31504/36938 -- GLOBAL_STEP: 1582900 | > loss_text_ce: 0.037717919796705246 (0.030070856004876405) | > loss_mel_ce: 3.0794577598571777 (nan) | > loss: 0.012369745410978794 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13707623663682345) | > loader_time: 0.0047 (0.004720227496765644)  --> TIME: 2025-02-27 06:25:41 -- STEP: 31554/36938 -- GLOBAL_STEP: 1582950 | > loss_text_ce: 0.023779122158885002 (0.030071722031898295) | > loss_mel_ce: 2.2003347873687744 (nan) | > loss: 0.008825849741697311 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13709004942950817) | > loader_time: 0.0052 (0.004720545777683948)  --> TIME: 2025-02-27 06:25:51 -- STEP: 31604/36938 -- GLOBAL_STEP: 1583000 | > loss_text_ce: 0.0269622765481472 (0.03007096192031736) | > loss_mel_ce: 2.6179323196411133 (nan) | > loss: 0.010495614260435104 (nan) | > current_lr: 5e-06 | > step_time: 0.1574 (0.13709202314114136) | > loader_time: 0.0047 (0.004720604181319829)  --> TIME: 2025-02-27 06:26:00 -- STEP: 31654/36938 -- GLOBAL_STEP: 1583050 | > loss_text_ce: 0.03545703738927841 (0.030070130880915605) | > loss_mel_ce: 2.8112218379974365 (nan) | > loss: 0.011296345852315426 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13708755916071907) | > loader_time: 0.0053 (0.004720667989208636)  --> TIME: 2025-02-27 06:26:09 -- STEP: 31704/36938 -- GLOBAL_STEP: 1583100 | > loss_text_ce: 0.016221683472394943 (0.030071209099709752) | > loss_mel_ce: 0.029463334009051323 (nan) | > loss: 0.00018128975352738053 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13708962737355798) | > loader_time: 0.0045 (0.0047208568964406195)  --> TIME: 2025-02-27 06:26:19 -- STEP: 31754/36938 -- GLOBAL_STEP: 1583150 | > loss_text_ce: 0.03068152442574501 (0.030073635871953563) | > loss_mel_ce: 2.438828229904175 (nan) | > loss: 0.009799642488360405 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.13708826772075053) | > loader_time: 0.0047 (0.00472092446778126)  --> TIME: 2025-02-27 06:26:28 -- STEP: 31804/36938 -- GLOBAL_STEP: 1583200 | > loss_text_ce: 0.02495984546840191 (0.030074912406330916) | > loss_mel_ce: 2.8910348415374756 (nan) | > loss: 0.011571408249437809 (nan) | > current_lr: 5e-06 | > step_time: 0.1779 (0.13709543225390658) | > loader_time: 0.0048 (0.004720908278205093)  --> TIME: 2025-02-27 06:26:37 -- STEP: 31854/36938 -- GLOBAL_STEP: 1583250 | > loss_text_ce: 0.031229713931679726 (0.03007588110316666) | > loss_mel_ce: 2.428711414337158 (nan) | > loss: 0.009761671535670757 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13709192345010138) | > loader_time: 0.0047 (0.004720847740036799)  --> TIME: 2025-02-27 06:26:46 -- STEP: 31904/36938 -- GLOBAL_STEP: 1583300 | > loss_text_ce: 0.02266254648566246 (0.030076421419407407) | > loss_mel_ce: 1.905916690826416 (nan) | > loss: 0.007653092499822378 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13708389065449744) | > loader_time: 0.0047 (0.004720834733071972)  --> TIME: 2025-02-27 06:26:56 -- STEP: 31954/36938 -- GLOBAL_STEP: 1583350 | > loss_text_ce: 0.02508174069225788 (0.030075017666167183) | > loss_mel_ce: 1.723008632659912 (nan) | > loss: 0.006936866790056229 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13708407001276848) | > loader_time: 0.0046 (0.004720974462444312)  --> TIME: 2025-02-27 06:27:05 -- STEP: 32004/36938 -- GLOBAL_STEP: 1583400 | > loss_text_ce: 0.02520616166293621 (0.030076554694990296) | > loss_mel_ce: 1.9159125089645386 (nan) | > loss: 0.007702852599322796 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.13708712958258473) | > loader_time: 0.0048 (0.0047210702194658985)  --> TIME: 2025-02-27 06:27:15 -- STEP: 32054/36938 -- GLOBAL_STEP: 1583450 | > loss_text_ce: 0.022007383406162262 (0.03007404197863895) | > loss_mel_ce: 2.154147148132324 (nan) | > loss: 0.008635534904897213 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13709028640257187) | > loader_time: 0.0046 (0.004721081233808546)  --> TIME: 2025-02-27 06:27:24 -- STEP: 32104/36938 -- GLOBAL_STEP: 1583500 | > loss_text_ce: 0.026909997686743736 (0.030072289396519153) | > loss_mel_ce: 2.990678310394287 (nan) | > loss: 0.011974557302892208 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.1370984443829829) | > loader_time: 0.0047 (0.004721163841901541)  --> TIME: 2025-02-27 06:27:34 -- STEP: 32154/36938 -- GLOBAL_STEP: 1583550 | > loss_text_ce: 0.02789417654275894 (0.0300736546963137) | > loss_mel_ce: 2.8441550731658936 (nan) | > loss: 0.011397021822631359 (nan) | > current_lr: 5e-06 | > step_time: 0.1866 (0.137095309559916) | > loader_time: 0.0051 (0.0047213073289015285)  --> TIME: 2025-02-27 06:27:43 -- STEP: 32204/36938 -- GLOBAL_STEP: 1583600 | > loss_text_ce: 0.02264498919248581 (0.03007407198098633) | > loss_mel_ce: 1.9041085243225098 (nan) | > loss: 0.007645847741514444 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13710052812933032) | > loader_time: 0.0045 (0.004721477281645989)  --> TIME: 2025-02-27 06:27:53 -- STEP: 32254/36938 -- GLOBAL_STEP: 1583650 | > loss_text_ce: 0.043523676693439484 (0.03007382938513649) | > loss_mel_ce: 3.0352985858917236 (nan) | > loss: 0.012217549607157707 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13710599594774933) | > loader_time: 0.0049 (0.004721498300147036)  --> TIME: 2025-02-27 06:28:02 -- STEP: 32304/36938 -- GLOBAL_STEP: 1583700 | > loss_text_ce: 0.02869343012571335 (0.03007547081479678) | > loss_mel_ce: 2.2678542137145996 (nan) | > loss: 0.009113284759223461 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.1371049197697883) | > loader_time: 0.0049 (0.004721432363355182)  --> TIME: 2025-02-27 06:28:11 -- STEP: 32354/36938 -- GLOBAL_STEP: 1583750 | > loss_text_ce: 0.04365665838122368 (0.030073526748452893) | > loss_mel_ce: 1.778980016708374 (nan) | > loss: 0.007232686039060354 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13710162071758084) | > loader_time: 0.0046 (0.0047214120753565686)  --> TIME: 2025-02-27 06:28:21 -- STEP: 32404/36938 -- GLOBAL_STEP: 1583800 | > loss_text_ce: 0.03357310965657234 (0.03007525684064641) | > loss_mel_ce: 3.281952142715454 (nan) | > loss: 0.013156847096979618 (nan) | > current_lr: 5e-06 | > step_time: 0.1689 (0.13710447910052392) | > loader_time: 0.0048 (0.004721419250004185)  --> TIME: 2025-02-27 06:28:30 -- STEP: 32454/36938 -- GLOBAL_STEP: 1583850 | > loss_text_ce: 0.016406361013650894 (0.030075674632835312) | > loss_mel_ce: 2.0407822132110596 (nan) | > loss: 0.008163446560502052 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13710286825621612) | > loader_time: 0.0046 (0.004721285315812324)  --> TIME: 2025-02-27 06:28:40 -- STEP: 32504/36938 -- GLOBAL_STEP: 1583900 | > loss_text_ce: 0.024085845798254013 (0.0300751674150627) | > loss_mel_ce: 1.480652928352356 (nan) | > loss: 0.005971185863018036 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13711181999631813) | > loader_time: 0.0046 (0.00472119011199446)  --> TIME: 2025-02-27 06:28:49 -- STEP: 32554/36938 -- GLOBAL_STEP: 1583950 | > loss_text_ce: 0.03459876775741577 (0.03007518775288114) | > loss_mel_ce: 1.9427083730697632 (nan) | > loss: 0.007846456952393055 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13711793861979707) | > loader_time: 0.0046 (0.004721128641039346)  --> TIME: 2025-02-27 06:28:59 -- STEP: 32604/36938 -- GLOBAL_STEP: 1584000 | > loss_text_ce: 0.025263268500566483 (0.030074052881544182) | > loss_mel_ce: 0.8581515550613403 (nan) | > loss: 0.003505614586174488 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13711508384965926) | > loader_time: 0.0045 (0.004720949436342634)  --> TIME: 2025-02-27 06:29:08 -- STEP: 32654/36938 -- GLOBAL_STEP: 1584050 | > loss_text_ce: 0.03130018711090088 (0.03007518516079907) | > loss_mel_ce: 3.012272596359253 (nan) | > loss: 0.01207767054438591 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.1371085619673821) | > loader_time: 0.0047 (0.004720899444994075)  --> TIME: 2025-02-27 06:29:17 -- STEP: 32704/36938 -- GLOBAL_STEP: 1584100 | > loss_text_ce: 0.03750113397836685 (0.030075131315164885) | > loss_mel_ce: 2.9032552242279053 (nan) | > loss: 0.01166966836899519 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13711732204587915) | > loader_time: 0.0047 (0.004720836506022847)  --> TIME: 2025-02-27 06:29:27 -- STEP: 32754/36938 -- GLOBAL_STEP: 1584150 | > loss_text_ce: 0.02939623035490513 (0.03007392501575389) | > loss_mel_ce: 2.754563570022583 (nan) | > loss: 0.01104746013879776 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13711976365902706) | > loader_time: 0.0046 (0.004720679568073892)  --> TIME: 2025-02-27 06:29:36 -- STEP: 32804/36938 -- GLOBAL_STEP: 1584200 | > loss_text_ce: 0.025789348408579826 (0.030073336386298968) | > loss_mel_ce: 0.410430908203125 (nan) | > loss: 0.0017310328548774123 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13711124635327038) | > loader_time: 0.0045 (0.0047205432190166345)  --> TIME: 2025-02-27 06:29:45 -- STEP: 32854/36938 -- GLOBAL_STEP: 1584250 | > loss_text_ce: 0.02782479301095009 (0.030072307245167025) | > loss_mel_ce: 2.5889649391174316 (nan) | > loss: 0.010384087450802326 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13710793191519963) | > loader_time: 0.0044 (0.0047204095781585844)  --> TIME: 2025-02-27 06:29:54 -- STEP: 32904/36938 -- GLOBAL_STEP: 1584300 | > loss_text_ce: 0.031571779400110245 (0.03007384948728225) | > loss_mel_ce: 1.9528447389602661 (nan) | > loss: 0.007874669507145882 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13710866193541765) | > loader_time: 0.0046 (0.004720322028755283)  --> TIME: 2025-02-27 06:30:04 -- STEP: 32954/36938 -- GLOBAL_STEP: 1584350 | > loss_text_ce: 0.03287526220083237 (0.030072493510774772) | > loss_mel_ce: 3.2006051540374756 (nan) | > loss: 0.012831272557377815 (nan) | > current_lr: 5e-06 | > step_time: 0.1884 (0.13710762438369198) | > loader_time: 0.005 (0.004720237675154155)  --> TIME: 2025-02-27 06:30:13 -- STEP: 33004/36938 -- GLOBAL_STEP: 1584400 | > loss_text_ce: 0.03979072347283363 (0.03007380790260852) | > loss_mel_ce: 1.9217751026153564 (nan) | > loss: 0.007783991750329733 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13710047829268135) | > loader_time: 0.0045 (0.0047201302221739775)  --> TIME: 2025-02-27 06:30:22 -- STEP: 33054/36938 -- GLOBAL_STEP: 1584450 | > loss_text_ce: 0.02756500244140625 (0.030073379468539187) | > loss_mel_ce: 2.382114887237549 (nan) | > loss: 0.00956222228705883 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13710004334701598) | > loader_time: 0.0044 (0.0047199709659090806)  --> TIME: 2025-02-27 06:30:31 -- STEP: 33104/36938 -- GLOBAL_STEP: 1584500 | > loss_text_ce: 0.03085334040224552 (0.030072197394416616) | > loss_mel_ce: 2.8785150051116943 (nan) | > loss: 0.011545113287866116 (nan) | > current_lr: 5e-06 | > step_time: 0.1522 (0.13709290957986653) | > loader_time: 0.0047 (0.004719771743684972)  --> TIME: 2025-02-27 06:30:40 -- STEP: 33154/36938 -- GLOBAL_STEP: 1584550 | > loss_text_ce: 0.03449398651719093 (0.030072397550618767) | > loss_mel_ce: 3.2989871501922607 (nan) | > loss: 0.013228100724518299 (nan) | > current_lr: 5e-06 | > step_time: 0.1708 (0.13708885849230476) | > loader_time: 0.0049 (0.004719670959265987)  --> TIME: 2025-02-27 06:30:50 -- STEP: 33204/36938 -- GLOBAL_STEP: 1584600 | > loss_text_ce: 0.027889423072338104 (0.030074246591852906) | > loss_mel_ce: 2.659728527069092 (nan) | > loss: 0.010665151290595531 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13708529196192173) | > loader_time: 0.0046 (0.004720083519200656)  --> TIME: 2025-02-27 06:30:59 -- STEP: 33254/36938 -- GLOBAL_STEP: 1584650 | > loss_text_ce: 0.036728423088788986 (0.030073319118705307) | > loss_mel_ce: 2.4986188411712646 (nan) | > loss: 0.010060902684926987 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13708378901991772) | > loader_time: 0.0046 (0.0047202351835061015)  --> TIME: 2025-02-27 06:31:08 -- STEP: 33304/36938 -- GLOBAL_STEP: 1584700 | > loss_text_ce: 0.03191300481557846 (0.03007258993456651) | > loss_mel_ce: 3.065438985824585 (nan) | > loss: 0.012291080318391323 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.137082918751792) | > loader_time: 0.0047 (0.004720188478652636)  --> TIME: 2025-02-27 06:31:17 -- STEP: 33354/36938 -- GLOBAL_STEP: 1584750 | > loss_text_ce: 0.038197968155145645 (0.030073492774529205) | > loss_mel_ce: 2.754871129989624 (nan) | > loss: 0.011083608493208885 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13707841049533281) | > loader_time: 0.0046 (0.004720213202082873)  --> TIME: 2025-02-27 06:31:27 -- STEP: 33404/36938 -- GLOBAL_STEP: 1584800 | > loss_text_ce: 0.03551538288593292 (0.030073714052822567) | > loss_mel_ce: 2.902224063873291 (nan) | > loss: 0.011657696217298508 (nan) | > current_lr: 5e-06 | > step_time: 0.1867 (0.13708512098714082) | > loader_time: 0.0048 (0.004720345519571392)  --> TIME: 2025-02-27 06:31:36 -- STEP: 33454/36938 -- GLOBAL_STEP: 1584850 | > loss_text_ce: 0.0248325914144516 (0.03007205747864509) | > loss_mel_ce: 1.5515507459640503 (nan) | > loss: 0.006255489774048328 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13708108048875992) | > loader_time: 0.0047 (0.004720186940608956)  --> TIME: 2025-02-27 06:31:46 -- STEP: 33504/36938 -- GLOBAL_STEP: 1584900 | > loss_text_ce: 0.034644268453121185 (0.03007215269613703) | > loss_mel_ce: 2.519155979156494 (nan) | > loss: 0.010134128853678703 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13708512879260476) | > loader_time: 0.0046 (0.004720261816821306)  --> TIME: 2025-02-27 06:31:55 -- STEP: 33554/36938 -- GLOBAL_STEP: 1584950 | > loss_text_ce: 0.03326869010925293 (0.030072343127861733) | > loss_mel_ce: 2.451047658920288 (nan) | > loss: 0.009858398698270321 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13708586354445204) | > loader_time: 0.0046 (0.004720131163020967)  --> TIME: 2025-02-27 06:32:04 -- STEP: 33604/36938 -- GLOBAL_STEP: 1585000 | > loss_text_ce: 0.03140803799033165 (0.030071359652578095) | > loss_mel_ce: 2.690385580062866 (nan) | > loss: 0.010800768621265888 (nan) | > current_lr: 5e-06 | > step_time: 0.1823 (0.13708303424094787) | > loader_time: 0.0048 (0.004720151140610077)  --> TIME: 2025-02-27 06:32:13 -- STEP: 33654/36938 -- GLOBAL_STEP: 1585050 | > loss_text_ce: 0.03969566896557808 (0.03007129738657195) | > loss_mel_ce: 2.5234375 (nan) | > loss: 0.010171163827180862 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13707099821206611) | > loader_time: 0.0052 (0.004720276885685244)  --> TIME: 2025-02-27 06:32:23 -- STEP: 33704/36938 -- GLOBAL_STEP: 1585100 | > loss_text_ce: 0.03028729557991028 (0.030072283269349517) | > loss_mel_ce: 3.0131497383117676 (nan) | > loss: 0.012077131308615208 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13707358638917083) | > loader_time: 0.0045 (0.0047204511948840365)  --> TIME: 2025-02-27 06:32:32 -- STEP: 33754/36938 -- GLOBAL_STEP: 1585150 | > loss_text_ce: 0.03079366311430931 (0.030072042563258693) | > loss_mel_ce: 2.5421080589294434 (nan) | > loss: 0.010209928266704082 (nan) | > current_lr: 5e-06 | > step_time: 0.1565 (0.13707703585420375) | > loader_time: 0.0047 (0.004720491764563358)  --> TIME: 2025-02-27 06:32:41 -- STEP: 33804/36938 -- GLOBAL_STEP: 1585200 | > loss_text_ce: 0.0318865142762661 (0.03007386252593996) | > loss_mel_ce: 2.483109712600708 (nan) | > loss: 0.009980144910514355 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.1370792294053572) | > loader_time: 0.0046 (0.004720569644337086)  --> TIME: 2025-02-27 06:32:51 -- STEP: 33854/36938 -- GLOBAL_STEP: 1585250 | > loss_text_ce: 0.03659816086292267 (0.030074108112237558) | > loss_mel_ce: 3.1799209117889404 (nan) | > loss: 0.012763965874910355 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.1370801349719215) | > loader_time: 0.0046 (0.004720572241580687)  --> TIME: 2025-02-27 06:33:00 -- STEP: 33904/36938 -- GLOBAL_STEP: 1585300 | > loss_text_ce: 0.04314608871936798 (0.030075005450081724) | > loss_mel_ce: 2.428257465362549 (nan) | > loss: 0.009807157330214977 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.1370760346285963) | > loader_time: 0.0047 (0.004720613733109778)  --> TIME: 2025-02-27 06:33:10 -- STEP: 33954/36938 -- GLOBAL_STEP: 1585350 | > loss_text_ce: 0.02864697389304638 (0.03007560198936081) | > loss_mel_ce: 2.5824427604675293 (nan) | > loss: 0.010361467488110065 (nan) | > current_lr: 5e-06 | > step_time: 0.1664 (0.1370793099733964) | > loader_time: 0.0049 (0.004720657251113874)  --> TIME: 2025-02-27 06:33:19 -- STEP: 34004/36938 -- GLOBAL_STEP: 1585400 | > loss_text_ce: 0.026636607944965363 (0.030075180098128377) | > loss_mel_ce: 2.5745620727539062 (nan) | > loss: 0.01032221782952547 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.1370825653299974) | > loader_time: 0.0046 (0.004720704048721222)  --> TIME: 2025-02-27 06:33:29 -- STEP: 34054/36938 -- GLOBAL_STEP: 1585450 | > loss_text_ce: 0.031002497300505638 (0.030076029097169988) | > loss_mel_ce: 2.9244959354400635 (nan) | > loss: 0.011728169396519661 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13708770223307098) | > loader_time: 0.0046 (0.004720637975719496)  --> TIME: 2025-02-27 06:33:38 -- STEP: 34104/36938 -- GLOBAL_STEP: 1585500 | > loss_text_ce: 0.03796190023422241 (0.03007615026143207) | > loss_mel_ce: 2.3356516361236572 (nan) | > loss: 0.009419102221727371 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13709143085880127) | > loader_time: 0.0045 (0.004720563791236074)  --> TIME: 2025-02-27 06:33:48 -- STEP: 34154/36938 -- GLOBAL_STEP: 1585550 | > loss_text_ce: 0.026687435805797577 (0.030074849864724307) | > loss_mel_ce: 2.3444368839263916 (nan) | > loss: 0.009409223683178425 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13709494859152904) | > loader_time: 0.0045 (0.004720451499955469)  --> TIME: 2025-02-27 06:33:57 -- STEP: 34204/36938 -- GLOBAL_STEP: 1585600 | > loss_text_ce: 0.02910674922168255 (0.030074846781821304) | > loss_mel_ce: 2.960195302963257 (nan) | > loss: 0.011862310580909252 (nan) | > current_lr: 5e-06 | > step_time: 0.1659 (0.13708919546727064) | > loader_time: 0.0046 (0.004720302098481312)  --> TIME: 2025-02-27 06:34:06 -- STEP: 34254/36938 -- GLOBAL_STEP: 1585650 | > loss_text_ce: 0.027751831337809563 (0.030073767096085786) | > loss_mel_ce: 0.03885650634765625 (nan) | > loss: 0.0002643188345246017 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13709767703355133) | > loader_time: 0.0045 (0.004720228520320568)  --> TIME: 2025-02-27 06:34:16 -- STEP: 34304/36938 -- GLOBAL_STEP: 1585700 | > loss_text_ce: 0.028610315173864365 (0.030072775923117804) | > loss_mel_ce: 2.2578678131103516 (nan) | > loss: 0.009073326364159584 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13711134930933536) | > loader_time: 0.0045 (0.004720188371503506)  --> TIME: 2025-02-27 06:34:26 -- STEP: 34354/36938 -- GLOBAL_STEP: 1585750 | > loss_text_ce: 0.03781535103917122 (0.030071389403944264) | > loss_mel_ce: 2.867180109024048 (nan) | > loss: 0.011527759954333305 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.13711060445397602) | > loader_time: 0.0049 (0.004720200854935084)  --> TIME: 2025-02-27 06:34:35 -- STEP: 34404/36938 -- GLOBAL_STEP: 1585800 | > loss_text_ce: 0.027023887261748314 (0.030070613499263817) | > loss_mel_ce: 1.4385714530944824 (nan) | > loss: 0.005815854761749506 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.13711744122776234) | > loader_time: 0.0046 (0.004720126476527318)  --> TIME: 2025-02-27 06:34:45 -- STEP: 34454/36938 -- GLOBAL_STEP: 1585850 | > loss_text_ce: 0.030547238886356354 (0.03006942290695147) | > loss_mel_ce: 2.97149658203125 (nan) | > loss: 0.011912873014807701 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.1371257200793016) | > loader_time: 0.0046 (0.0047200592961861645)  --> TIME: 2025-02-27 06:34:55 -- STEP: 34504/36938 -- GLOBAL_STEP: 1585900 | > loss_text_ce: 0.029471447691321373 (0.030069987860873955) | > loss_mel_ce: 2.8673837184906006 (nan) | > loss: 0.011495457030832767 (nan) | > current_lr: 5e-06 | > step_time: 0.1775 (0.13713474031697512) | > loader_time: 0.0049 (0.004720030342540278)  --> TIME: 2025-02-27 06:35:04 -- STEP: 34554/36938 -- GLOBAL_STEP: 1585950 | > loss_text_ce: 0.02452862448990345 (0.0300699820230692) | > loss_mel_ce: 2.5762977600097656 (nan) | > loss: 0.01032074075192213 (nan) | > current_lr: 5e-06 | > step_time: 0.1352 (0.13713790403442105) | > loader_time: 0.0046 (0.0047199747632373015)  --> TIME: 2025-02-27 06:35:13 -- STEP: 34604/36938 -- GLOBAL_STEP: 1586000 | > loss_text_ce: 0.03161218762397766 (0.03007184713014466) | > loss_mel_ce: 3.1095728874206543 (nan) | > loss: 0.012465020641684532 (nan) | > current_lr: 5e-06 | > step_time: 0.1513 (0.13713433050500196) | > loader_time: 0.0047 (0.004719945526222238)  --> TIME: 2025-02-27 06:35:23 -- STEP: 34654/36938 -- GLOBAL_STEP: 1586050 | > loss_text_ce: 0.027603762224316597 (0.030070297864065315) | > loss_mel_ce: 2.011570453643799 (nan) | > loss: 0.008091961964964867 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13713104904313678) | > loader_time: 0.0046 (0.004719834728937744)  --> TIME: 2025-02-27 06:35:32 -- STEP: 34704/36938 -- GLOBAL_STEP: 1586100 | > loss_text_ce: 0.03215767443180084 (0.03007134224175508) | > loss_mel_ce: 2.6946604251861572 (nan) | > loss: 0.010820707306265831 (nan) | > current_lr: 5e-06 | > step_time: 0.1615 (0.1371287915249359) | > loader_time: 0.0047 (0.00471979266984936)  --> TIME: 2025-02-27 06:35:41 -- STEP: 34754/36938 -- GLOBAL_STEP: 1586150 | > loss_text_ce: 0.03112148679792881 (0.030071071400441917) | > loss_mel_ce: 1.9331411123275757 (nan) | > loss: 0.007794693112373352 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13712616875633027) | > loader_time: 0.0046 (0.004719808343543855)  --> TIME: 2025-02-27 06:35:51 -- STEP: 34804/36938 -- GLOBAL_STEP: 1586200 | > loss_text_ce: 0.026689883321523666 (0.030072921380450864) | > loss_mel_ce: 3.157362222671509 (nan) | > loss: 0.012635127641260624 (nan) | > current_lr: 5e-06 | > step_time: 0.1771 (0.13712972075127222) | > loader_time: 0.0048 (0.004719692802253806)  --> TIME: 2025-02-27 06:36:00 -- STEP: 34854/36938 -- GLOBAL_STEP: 1586250 | > loss_text_ce: 0.030247138813138008 (0.030073499937123396) | > loss_mel_ce: 2.738800287246704 (nan) | > loss: 0.010988283902406693 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.13713106201609834) | > loader_time: 0.0046 (0.004719751929295544)  --> TIME: 2025-02-27 06:36:09 -- STEP: 34904/36938 -- GLOBAL_STEP: 1586300 | > loss_text_ce: 0.03361213207244873 (0.030074495041637585) | > loss_mel_ce: 2.531745433807373 (nan) | > loss: 0.010179991833865643 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13712851355143002) | > loader_time: 0.0046 (0.004719778809994402)  --> TIME: 2025-02-27 06:36:18 -- STEP: 34954/36938 -- GLOBAL_STEP: 1586350 | > loss_text_ce: 0.031047310680150986 (0.03007359794294028) | > loss_mel_ce: 2.981262445449829 (nan) | > loss: 0.011953610926866531 (nan) | > current_lr: 5e-06 | > step_time: 0.1571 (0.13712218737870932) | > loader_time: 0.0047 (0.0047196647821714215)  --> TIME: 2025-02-27 06:36:28 -- STEP: 35004/36938 -- GLOBAL_STEP: 1586400 | > loss_text_ce: 0.03203526511788368 (0.03007389046625591) | > loss_mel_ce: 2.734137773513794 (nan) | > loss: 0.010976877994835377 (nan) | > current_lr: 5e-06 | > step_time: 0.1763 (0.13712479670161326) | > loader_time: 0.0049 (0.004719571826962258)  --> TIME: 2025-02-27 06:36:37 -- STEP: 35054/36938 -- GLOBAL_STEP: 1586450 | > loss_text_ce: 0.029039578512310982 (0.030073322615969712) | > loss_mel_ce: 2.7102742195129395 (nan) | > loss: 0.010870293714106083 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.1371271987537308) | > loader_time: 0.0046 (0.0047195050165073135)  --> TIME: 2025-02-27 06:36:47 -- STEP: 35104/36938 -- GLOBAL_STEP: 1586500 | > loss_text_ce: 0.026508068665862083 (0.03007354212767195) | > loss_mel_ce: 3.1440796852111816 (nan) | > loss: 0.012581698596477509 (nan) | > current_lr: 5e-06 | > step_time: 0.1563 (0.1371290790852673) | > loader_time: 0.0047 (0.004719423991012467)  --> TIME: 2025-02-27 06:36:56 -- STEP: 35154/36938 -- GLOBAL_STEP: 1586550 | > loss_text_ce: 0.034272439777851105 (0.03007295171390797) | > loss_mel_ce: 0.5484617948532104 (nan) | > loss: 0.002312437631189823 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1371370308254193) | > loader_time: 0.0052 (0.004719536276124603)  --> TIME: 2025-02-27 06:37:06 -- STEP: 35204/36938 -- GLOBAL_STEP: 1586600 | > loss_text_ce: 0.03253859654068947 (0.030073932284269068) | > loss_mel_ce: 2.2941489219665527 (nan) | > loss: 0.009232887998223305 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.137141891730507) | > loader_time: 0.0046 (0.004719625744084399)  --> TIME: 2025-02-27 06:37:15 -- STEP: 35254/36938 -- GLOBAL_STEP: 1586650 | > loss_text_ce: 0.029878081753849983 (0.030074835707551192) | > loss_mel_ce: 3.1071360111236572 (nan) | > loss: 0.012448469176888466 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13713935171272154) | > loader_time: 0.0046 (0.0047196367049762095)  --> TIME: 2025-02-27 06:37:24 -- STEP: 35304/36938 -- GLOBAL_STEP: 1586700 | > loss_text_ce: 0.02794254943728447 (0.030076673330253014) | > loss_mel_ce: 2.7574851512908936 (nan) | > loss: 0.01105328556150198 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.13714279421138081) | > loader_time: 0.0046 (0.0047196050755107044)  --> TIME: 2025-02-27 06:37:34 -- STEP: 35354/36938 -- GLOBAL_STEP: 1586750 | > loss_text_ce: 0.027063800022006035 (0.030076652103767382) | > loss_mel_ce: 3.1368353366851807 (nan) | > loss: 0.012555155903100967 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13714928213487276) | > loader_time: 0.0045 (0.004719492583521102)  --> TIME: 2025-02-27 06:37:43 -- STEP: 35404/36938 -- GLOBAL_STEP: 1586800 | > loss_text_ce: 0.042815905064344406 (0.030078094383185993) | > loss_mel_ce: 2.644665241241455 (nan) | > loss: 0.01066460832953453 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13714752505289105) | > loader_time: 0.0046 (0.004719334239408027)  --> TIME: 2025-02-27 06:37:52 -- STEP: 35454/36938 -- GLOBAL_STEP: 1586850 | > loss_text_ce: 0.030354605987668037 (0.03007813857384922) | > loss_mel_ce: 2.513856887817383 (nan) | > loss: 0.010096077807247639 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.1371439497113228) | > loader_time: 0.0046 (0.004719168541226571)  --> TIME: 2025-02-27 06:38:02 -- STEP: 35504/36938 -- GLOBAL_STEP: 1586900 | > loss_text_ce: 0.028020603582262993 (0.030076014171767362) | > loss_mel_ce: 2.3872172832489014 (nan) | > loss: 0.009584277868270874 (nan) | > current_lr: 5e-06 | > step_time: 0.1494 (0.13714887746393536) | > loader_time: 0.0048 (0.004719102165085478)  --> TIME: 2025-02-27 06:38:11 -- STEP: 35554/36938 -- GLOBAL_STEP: 1586950 | > loss_text_ce: 0.031190432608127594 (0.030075797975168003) | > loss_mel_ce: 1.9657872915267944 (nan) | > loss: 0.007924514822661877 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13714696695507206) | > loader_time: 0.0046 (0.00471901736699628)  --> TIME: 2025-02-27 06:38:21 -- STEP: 35604/36938 -- GLOBAL_STEP: 1587000 | > loss_text_ce: 0.029384607449173927 (0.030077132662150226) | > loss_mel_ce: 1.8595142364501953 (nan) | > loss: 0.007495630998164415 (nan) | > current_lr: 5e-06 | > step_time: 0.1566 (0.13714546619432802) | > loader_time: 0.0047 (0.004718889601912825)  --> TIME: 2025-02-27 06:38:30 -- STEP: 35654/36938 -- GLOBAL_STEP: 1587050 | > loss_text_ce: 0.030201902613043785 (0.030077175874188976) | > loss_mel_ce: 2.6567351818084717 (nan) | > loss: 0.010662449523806572 (nan) | > current_lr: 5e-06 | > step_time: 0.1857 (0.13714304188065832) | > loader_time: 0.0048 (0.00471879369098363)  --> TIME: 2025-02-27 06:38:39 -- STEP: 35704/36938 -- GLOBAL_STEP: 1587100 | > loss_text_ce: 0.03419441729784012 (0.030075066880388836) | > loss_mel_ce: 3.1062610149383545 (nan) | > loss: 0.012462126091122627 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13714615189652463) | > loader_time: 0.0046 (0.004718742555175504)  --> TIME: 2025-02-27 06:38:48 -- STEP: 35754/36938 -- GLOBAL_STEP: 1587150 | > loss_text_ce: 0.03158716484904289 (0.030076505213148315) | > loss_mel_ce: 0.00795745849609375 (nan) | > loss: 0.0001569231244502589 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13713639327435043) | > loader_time: 0.0045 (0.004718578661315638)  --> TIME: 2025-02-27 06:38:57 -- STEP: 35804/36938 -- GLOBAL_STEP: 1587200 | > loss_text_ce: 0.03152905032038689 (0.030076333366852285) | > loss_mel_ce: 2.049896240234375 (nan) | > loss: 0.008259624242782593 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13713431195192008) | > loader_time: 0.0046 (0.004718462317605963)  --> TIME: 2025-02-27 06:39:07 -- STEP: 35854/36938 -- GLOBAL_STEP: 1587250 | > loss_text_ce: 0.027517374604940414 (0.030075379525762596) | > loss_mel_ce: 2.7484548091888428 (nan) | > loss: 0.011015763506293297 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13713543138574702) | > loader_time: 0.0045 (0.004718343685054579)  --> TIME: 2025-02-27 06:39:16 -- STEP: 35904/36938 -- GLOBAL_STEP: 1587300 | > loss_text_ce: 0.03584893420338631 (0.030076483641060615) | > loss_mel_ce: 2.846263885498047 (nan) | > loss: 0.011436956003308296 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13713321899214875) | > loader_time: 0.0045 (0.004718260896025257)  --> TIME: 2025-02-27 06:39:25 -- STEP: 35954/36938 -- GLOBAL_STEP: 1587350 | > loss_text_ce: 0.031204799190163612 (0.030075243075738868) | > loss_mel_ce: 0.0999348983168602 (nan) | > loss: 0.0005203956388868392 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.1371299479385294) | > loader_time: 0.0046 (0.004718100579673368)  --> TIME: 2025-02-27 06:39:34 -- STEP: 36004/36938 -- GLOBAL_STEP: 1587400 | > loss_text_ce: 0.033836014568805695 (0.030075612974802404) | > loss_mel_ce: 2.536357879638672 (nan) | > loss: 0.010199183598160744 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1371297021002752) | > loader_time: 0.0045 (0.0047179804008783705)  --> TIME: 2025-02-27 06:39:44 -- STEP: 36054/36938 -- GLOBAL_STEP: 1587450 | > loss_text_ce: 0.032145269215106964 (0.030074685261285542) | > loss_mel_ce: 2.5806896686553955 (nan) | > loss: 0.010368392802774906 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13713134860029527) | > loader_time: 0.0046 (0.004717974494286109)  --> TIME: 2025-02-27 06:39:53 -- STEP: 36104/36938 -- GLOBAL_STEP: 1587500 | > loss_text_ce: 0.02710198611021042 (0.03007381788375552) | > loss_mel_ce: 2.468409538269043 (nan) | > loss: 0.009902823716402054 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13713472405373764) | > loader_time: 0.0046 (0.004718056043129954)  --> TIME: 2025-02-27 06:40:03 -- STEP: 36154/36938 -- GLOBAL_STEP: 1587550 | > loss_text_ce: 0.02637515403330326 (0.03007402771970484) | > loss_mel_ce: 2.984654188156128 (nan) | > loss: 0.01194852963089943 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.137135517849768) | > loader_time: 0.0047 (0.0047180677018163015)  --> TIME: 2025-02-27 06:40:12 -- STEP: 36204/36938 -- GLOBAL_STEP: 1587600 | > loss_text_ce: 0.025043973699212074 (0.030075005245980085) | > loss_mel_ce: 1.2409179210662842 (nan) | > loss: 0.005023658741265535 (nan) | > current_lr: 5e-06 | > step_time: 0.1086 (0.1371368924407919) | > loader_time: 0.0064 (0.0047181210798685445)  --> TIME: 2025-02-27 06:40:21 -- STEP: 36254/36938 -- GLOBAL_STEP: 1587650 | > loss_text_ce: 0.02967613935470581 (0.03007536003704538) | > loss_mel_ce: 2.55709171295166 (nan) | > loss: 0.010264952667057514 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.1371369123024813) | > loader_time: 0.0047 (0.00471804875523095)  --> TIME: 2025-02-27 06:40:31 -- STEP: 36304/36938 -- GLOBAL_STEP: 1587700 | > loss_text_ce: 0.03576446324586868 (0.030074093501472363) | > loss_mel_ce: 1.6220773458480835 (nan) | > loss: 0.006578737869858742 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.1371351972504711) | > loader_time: 0.0046 (0.00471802351363379)  --> TIME: 2025-02-27 06:40:40 -- STEP: 36354/36938 -- GLOBAL_STEP: 1587750 | > loss_text_ce: 0.02549469843506813 (0.030073239778799173) | > loss_mel_ce: 2.696272850036621 (nan) | > loss: 0.01080066617578268 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13713588126718085) | > loader_time: 0.0047 (0.0047180472660059295)  --> TIME: 2025-02-27 06:40:50 -- STEP: 36404/36938 -- GLOBAL_STEP: 1587800 | > loss_text_ce: 0.027388781309127808 (0.030072455463239514) | > loss_mel_ce: 2.701584815979004 (nan) | > loss: 0.010829261504113674 (nan) | > current_lr: 5e-06 | > step_time: 0.1624 (0.13713940910265482) | > loader_time: 0.0048 (0.004718022148185544)  --> TIME: 2025-02-27 06:40:59 -- STEP: 36454/36938 -- GLOBAL_STEP: 1587850 | > loss_text_ce: 0.02654489502310753 (0.030073235749300823) | > loss_mel_ce: 2.454784870147705 (nan) | > loss: 0.009846546687185764 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13714156795294827) | > loader_time: 0.0047 (0.004717981853925648)  --> TIME: 2025-02-27 06:41:08 -- STEP: 36504/36938 -- GLOBAL_STEP: 1587900 | > loss_text_ce: 0.029366496950387955 (0.030072077072364613) | > loss_mel_ce: 1.2198244333267212 (nan) | > loss: 0.004957107361406088 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13714032786576336) | > loader_time: 0.0046 (0.00471799571002597)  --> TIME: 2025-02-27 06:41:17 -- STEP: 36554/36938 -- GLOBAL_STEP: 1587950 | > loss_text_ce: 0.02899974212050438 (0.030071558545913223) | > loss_mel_ce: 3.20259952545166 (nan) | > loss: 0.012823808006942272 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.13713349510466585) | > loader_time: 0.0047 (0.004717886301153242)  --> TIME: 2025-02-27 06:41:26 -- STEP: 36604/36938 -- GLOBAL_STEP: 1588000 | > loss_text_ce: 0.030801229178905487 (0.03007205593282004) | > loss_mel_ce: 0.69635009765625 (nan) | > loss: 0.00288552138954401 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.1371275908815926) | > loader_time: 0.0046 (0.004717760230137228)  --> TIME: 2025-02-27 06:41:36 -- STEP: 36654/36938 -- GLOBAL_STEP: 1588050 | > loss_text_ce: 0.030935510993003845 (0.030071529710476108) | > loss_mel_ce: 3.2444519996643066 (nan) | > loss: 0.01299757044762373 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.1371280471633458) | > loader_time: 0.0045 (0.004717654387546433)  --> TIME: 2025-02-27 06:41:45 -- STEP: 36704/36938 -- GLOBAL_STEP: 1588100 | > loss_text_ce: 0.03403830528259277 (0.03006977049160943) | > loss_mel_ce: 2.173433303833008 (nan) | > loss: 0.008759808726608753 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13712345041600812) | > loader_time: 0.0046 (0.004717594056464106)  --> TIME: 2025-02-27 06:41:54 -- STEP: 36754/36938 -- GLOBAL_STEP: 1588150 | > loss_text_ce: 0.03392438590526581 (0.030067577681464615) | > loss_mel_ce: 2.2184066772460938 (nan) | > loss: 0.008937821723520756 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.1371221896541723) | > loader_time: 0.0045 (0.004717539954757625)  --> TIME: 2025-02-27 06:42:04 -- STEP: 36804/36938 -- GLOBAL_STEP: 1588200 | > loss_text_ce: 0.029425272718071938 (0.03006595380705943) | > loss_mel_ce: 1.8516285419464111 (nan) | > loss: 0.007464499678462744 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13712322782555308) | > loader_time: 0.0048 (0.0047175481894420205)  --> TIME: 2025-02-27 06:42:13 -- STEP: 36854/36938 -- GLOBAL_STEP: 1588250 | > loss_text_ce: 0.03053305298089981 (0.030065378068921088) | > loss_mel_ce: 0.00311279296875 (nan) | > loss: 0.00013351526285987347 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.1371203225065352) | > loader_time: 0.0046 (0.004717600645138901)  --> TIME: 2025-02-27 06:42:22 -- STEP: 36904/36938 -- GLOBAL_STEP: 1588300 | > loss_text_ce: 0.030702371150255203 (0.030064795312814203) | > loss_mel_ce: 0.1325022429227829 (nan) | > loss: 0.0006476373528130352 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13710840445342007) | > loader_time: 0.0046 (0.004717561607114821)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.020983877748546046 (+0.0007294770514611089) | > avg_loss_text_ce: 0.02937677695351367 (-0.00012889951651934295) | > avg_loss_mel_ce: 2.429188026650116 (-0.001710887887690582) | > avg_loss: 2.4585648049222377 (-0.0018397997511487851) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_1588334.pth  > EPOCH: 44/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-27 06:42:54)   --> TIME: 2025-02-27 06:42:58 -- STEP: 16/36938 -- GLOBAL_STEP: 1588350 | > loss_text_ce: 0.029800109565258026 (0.03285927837714553) | > loss_mel_ce: 2.0777790546417236 (2.3546261191368103) | > loss: 0.008363409899175167 (0.009474149002926424) | > current_lr: 5e-06 | > step_time: 0.1273 (0.130327433347702) | > loader_time: 0.0048 (0.004997909069061279)  --> TIME: 2025-02-27 06:43:07 -- STEP: 66/36938 -- GLOBAL_STEP: 1588400 | > loss_text_ce: 0.024116452783346176 (0.030675287302019016) | > loss_mel_ce: 2.094651937484741 (nan) | > loss: 0.008407811634242535 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.1366187839797049) | > loader_time: 0.0074 (0.00500137878186775)  --> TIME: 2025-02-27 06:43:16 -- STEP: 116/36938 -- GLOBAL_STEP: 1588450 | > loss_text_ce: 0.027999214828014374 (0.03087818524639668) | > loss_mel_ce: 2.2806544303894043 (nan) | > loss: 0.009161324240267277 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13595178003968864) | > loader_time: 0.0048 (0.00495487246020087)  --> TIME: 2025-02-27 06:43:25 -- STEP: 166/36938 -- GLOBAL_STEP: 1588500 | > loss_text_ce: 0.03418760374188423 (0.030542766652911543) | > loss_mel_ce: 1.9972281455993652 (nan) | > loss: 0.008061174303293228 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13393447916191753) | > loader_time: 0.0047 (0.00491147730723921)  --> TIME: 2025-02-27 06:43:34 -- STEP: 216/36938 -- GLOBAL_STEP: 1588550 | > loss_text_ce: 0.030260592699050903 (0.030416493985319027) | > loss_mel_ce: 2.9260916709899902 (nan) | > loss: 0.011731557548046112 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13381331386389558) | > loader_time: 0.0049 (0.004858788516786366)  --> TIME: 2025-02-27 06:43:43 -- STEP: 266/36938 -- GLOBAL_STEP: 1588600 | > loss_text_ce: 0.02600390650331974 (0.030340459869992464) | > loss_mel_ce: 2.832228899002075 (nan) | > loss: 0.011342193931341171 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13354893824211642) | > loader_time: 0.0047 (0.004823174691738044)  --> TIME: 2025-02-27 06:43:53 -- STEP: 316/36938 -- GLOBAL_STEP: 1588650 | > loss_text_ce: 0.025937365368008614 (0.03045622284135109) | > loss_mel_ce: 2.620309829711914 (nan) | > loss: 0.010500981472432613 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13363005843343617) | > loader_time: 0.0046 (0.004797255691093738)  --> TIME: 2025-02-27 06:44:02 -- STEP: 366/36938 -- GLOBAL_STEP: 1588700 | > loss_text_ce: 0.0214848555624485 (0.030276150279805635) | > loss_mel_ce: 0.5972164273262024 (nan) | > loss: 0.0024551639799028635 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13329368992581397) | > loader_time: 0.0047 (0.004778084859170552)  --> TIME: 2025-02-27 06:44:11 -- STEP: 416/36938 -- GLOBAL_STEP: 1588750 | > loss_text_ce: 0.02657107450067997 (0.030240096926438406) | > loss_mel_ce: 2.681550979614258 (nan) | > loss: 0.010746516287326813 (nan) | > current_lr: 5e-06 | > step_time: 0.1713 (0.13363749018082258) | > loader_time: 0.0048 (0.00476444684542143)  --> TIME: 2025-02-27 06:44:20 -- STEP: 466/36938 -- GLOBAL_STEP: 1588800 | > loss_text_ce: 0.03408775478601456 (0.030269627691178618) | > loss_mel_ce: 2.3726232051849365 (nan) | > loss: 0.00955044012516737 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.13408527916593094) | > loader_time: 0.0046 (0.004753766141735951)  --> TIME: 2025-02-27 06:44:30 -- STEP: 516/36938 -- GLOBAL_STEP: 1588850 | > loss_text_ce: 0.034810133278369904 (0.03033813479710226) | > loss_mel_ce: 2.791274309158325 (nan) | > loss: 0.011214621365070343 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.1343735581220582) | > loader_time: 0.0048 (0.004745308742966768)  --> TIME: 2025-02-27 06:44:39 -- STEP: 566/36938 -- GLOBAL_STEP: 1588900 | > loss_text_ce: 0.030470099300146103 (0.03032543722790248) | > loss_mel_ce: 3.3974382877349854 (nan) | > loss: 0.013602811843156815 (nan) | > current_lr: 5e-06 | > step_time: 0.1829 (0.13415425362940808) | > loader_time: 0.0049 (0.004740507779610036)  --> TIME: 2025-02-27 06:44:48 -- STEP: 616/36938 -- GLOBAL_STEP: 1588950 | > loss_text_ce: 0.029197873547673225 (0.030360881792526547) | > loss_mel_ce: 1.8963466882705688 (nan) | > loss: 0.0076410504989326 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1345223722519812) | > loader_time: 0.0074 (0.0047552930070208325)  --> TIME: 2025-02-27 06:44:58 -- STEP: 666/36938 -- GLOBAL_STEP: 1589000 | > loss_text_ce: 0.030605942010879517 (0.03036423712722711) | > loss_mel_ce: 2.7289700508117676 (nan) | > loss: 0.01095069944858551 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.13494652742380123) | > loader_time: 0.0053 (0.004754325291057968)  --> TIME: 2025-02-27 06:45:07 -- STEP: 716/36938 -- GLOBAL_STEP: 1589050 | > loss_text_ce: 0.02748173102736473 (0.0303676878245204) | > loss_mel_ce: 2.4931018352508545 (nan) | > loss: 0.010002316907048225 (nan) | > current_lr: 5e-06 | > step_time: 0.1502 (0.13527100845422138) | > loader_time: 0.0047 (0.004752104841797049)  --> TIME: 2025-02-27 06:45:17 -- STEP: 766/36938 -- GLOBAL_STEP: 1589100 | > loss_text_ce: 0.03082589991390705 (0.03035804178619237) | > loss_mel_ce: 1.8647335767745972 (nan) | > loss: 0.007522061467170715 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.1352020651491128) | > loader_time: 0.0046 (0.004751230655700054)  --> TIME: 2025-02-27 06:45:26 -- STEP: 816/36938 -- GLOBAL_STEP: 1589150 | > loss_text_ce: 0.037443071603775024 (0.030344777827278947) | > loss_mel_ce: 3.058143377304077 (nan) | > loss: 0.01228407397866249 (nan) | > current_lr: 5e-06 | > step_time: 0.1699 (0.1353960449204724) | > loader_time: 0.0048 (0.004751847947345066)  --> TIME: 2025-02-27 06:45:35 -- STEP: 866/36938 -- GLOBAL_STEP: 1589200 | > loss_text_ce: 0.028718212619423866 (0.03031292929169969) | > loss_mel_ce: 2.7831521034240723 (nan) | > loss: 0.011158215813338757 (nan) | > current_lr: 5e-06 | > step_time: 0.1876 (0.13508908825713534) | > loader_time: 0.005 (0.004747115712242921)  --> TIME: 2025-02-27 06:45:45 -- STEP: 916/36938 -- GLOBAL_STEP: 1589250 | > loss_text_ce: 0.03643352910876274 (0.03031827036828193) | > loss_mel_ce: 2.7100491523742676 (nan) | > loss: 0.010898740962147713 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.13531972539476944) | > loader_time: 0.0068 (0.004749496430809323)  --> TIME: 2025-02-27 06:45:54 -- STEP: 966/36938 -- GLOBAL_STEP: 1589300 | > loss_text_ce: 0.026718851178884506 (0.03025741732484559) | > loss_mel_ce: 2.7323222160339355 (nan) | > loss: 0.010948576033115387 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1352714369262473) | > loader_time: 0.0045 (0.004745924448127828)  --> TIME: 2025-02-27 06:46:03 -- STEP: 1016/36938 -- GLOBAL_STEP: 1589350 | > loss_text_ce: 0.033706989139318466 (0.030262580519087437) | > loss_mel_ce: 2.6967132091522217 (nan) | > loss: 0.01083500124514103 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13517930775176812) | > loader_time: 0.0052 (0.004742423145789801)  --> TIME: 2025-02-27 06:46:12 -- STEP: 1066/36938 -- GLOBAL_STEP: 1589400 | > loss_text_ce: 0.03056151233613491 (0.030240814917935774) | > loss_mel_ce: 3.0005083084106445 (nan) | > loss: 0.012028055265545845 (nan) | > current_lr: 5e-06 | > step_time: 0.1697 (0.13521860822280385) | > loader_time: 0.0047 (0.004741601827667982)  --> TIME: 2025-02-27 06:46:22 -- STEP: 1116/36938 -- GLOBAL_STEP: 1589450 | > loss_text_ce: 0.0325096920132637 (0.03022864778062135) | > loss_mel_ce: 2.171171188354492 (nan) | > loss: 0.008744766004383564 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13547345728857102) | > loader_time: 0.0048 (0.0047434420995814824)  --> TIME: 2025-02-27 06:46:31 -- STEP: 1166/36938 -- GLOBAL_STEP: 1589500 | > loss_text_ce: 0.02379525639116764 (0.03022315300518637) | > loss_mel_ce: 0.015289306640625 (nan) | > loss: 0.00015509747026953846 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13538119764311637) | > loader_time: 0.0045 (0.004743400405242555)  --> TIME: 2025-02-27 06:46:40 -- STEP: 1216/36938 -- GLOBAL_STEP: 1589550 | > loss_text_ce: 0.03463032841682434 (0.03019342722530526) | > loss_mel_ce: 2.661379337310791 (nan) | > loss: 0.010698451660573483 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13546732185702565) | > loader_time: 0.0046 (0.004742719233036041)  --> TIME: 2025-02-27 06:46:50 -- STEP: 1266/36938 -- GLOBAL_STEP: 1589600 | > loss_text_ce: 0.03253677859902382 (0.030193666278680354) | > loss_mel_ce: 3.0226004123687744 (nan) | > loss: 0.012123560532927513 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13567112991218488) | > loader_time: 0.0046 (0.0047411169109374575)  --> TIME: 2025-02-27 06:46:59 -- STEP: 1316/36938 -- GLOBAL_STEP: 1589650 | > loss_text_ce: 0.02735723927617073 (0.03019082726412495) | > loss_mel_ce: 2.6462907791137695 (nan) | > loss: 0.01060971524566412 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13582109933928502) | > loader_time: 0.0046 (0.0047396267438732)  --> TIME: 2025-02-27 06:47:09 -- STEP: 1366/36938 -- GLOBAL_STEP: 1589700 | > loss_text_ce: 0.024229273200035095 (0.0301570197482802) | > loss_mel_ce: 2.9255614280700684 (nan) | > loss: 0.011705519631505013 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.1357854056672504) | > loader_time: 0.0047 (0.004737184477061858)  --> TIME: 2025-02-27 06:47:18 -- STEP: 1416/36938 -- GLOBAL_STEP: 1589750 | > loss_text_ce: 0.029333585873246193 (0.03010298500211099) | > loss_mel_ce: 1.3750044107437134 (nan) | > loss: 0.00557277025654912 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13582705660054897) | > loader_time: 0.0052 (0.004735547775602604)  --> TIME: 2025-02-27 06:47:27 -- STEP: 1466/36938 -- GLOBAL_STEP: 1589800 | > loss_text_ce: 0.02674078941345215 (0.030079603718662327) | > loss_mel_ce: 2.8415684700012207 (nan) | > loss: 0.011382180266082287 (nan) | > current_lr: 5e-06 | > step_time: 0.1264 (0.1357920700010789) | > loader_time: 0.0045 (0.004732795008858421)  --> TIME: 2025-02-27 06:47:36 -- STEP: 1516/36938 -- GLOBAL_STEP: 1589850 | > loss_text_ce: 0.01719333417713642 (0.03008292090531075) | > loss_mel_ce: 2.8910272121429443 (nan) | > loss: 0.011540558189153671 (nan) | > current_lr: 5e-06 | > step_time: 0.1074 (0.13566741858434528) | > loader_time: 0.0045 (0.004729166980783665)  --> TIME: 2025-02-27 06:47:46 -- STEP: 1566/36938 -- GLOBAL_STEP: 1589900 | > loss_text_ce: 0.03807499632239342 (0.03007994422458331) | > loss_mel_ce: 2.6286754608154297 (nan) | > loss: 0.01058234367519617 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13582447296814876) | > loader_time: 0.0048 (0.00472846691940327)  --> TIME: 2025-02-27 06:47:55 -- STEP: 1616/36938 -- GLOBAL_STEP: 1589950 | > loss_text_ce: 0.04515674337744713 (0.03010860617655228) | > loss_mel_ce: 2.5295941829681396 (nan) | > loss: 0.010217266157269478 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.1358719271893546) | > loader_time: 0.0047 (0.00472848146858782)  --> TIME: 2025-02-27 06:48:05 -- STEP: 1666/36938 -- GLOBAL_STEP: 1590000 | > loss_text_ce: 0.0387968085706234 (0.030130014030345563) | > loss_mel_ce: 2.7095773220062256 (nan) | > loss: 0.010906247422099113 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.1359027415191997) | > loader_time: 0.0048 (0.004729591068528852) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1590000.pth  --> TIME: 2025-02-27 06:48:20 -- STEP: 1716/36938 -- GLOBAL_STEP: 1590050 | > loss_text_ce: 0.03032420575618744 (0.03014617448017036) | > loss_mel_ce: 1.61572265625 (nan) | > loss: 0.006531932391226292 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13588332796430247) | > loader_time: 0.0045 (0.004728959990548089)  --> TIME: 2025-02-27 06:48:29 -- STEP: 1766/36938 -- GLOBAL_STEP: 1590100 | > loss_text_ce: 0.03678669035434723 (0.030151985333610163) | > loss_mel_ce: 2.2686805725097656 (nan) | > loss: 0.009148680604994297 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13575078636441584) | > loader_time: 0.0046 (0.004726779960424959)  --> TIME: 2025-02-27 06:48:38 -- STEP: 1816/36938 -- GLOBAL_STEP: 1590150 | > loss_text_ce: 0.039223454892635345 (0.030142898011813146) | > loss_mel_ce: 2.6602859497070312 (nan) | > loss: 0.010712339542806149 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13570203051167937) | > loader_time: 0.0046 (0.004723639609004959)  --> TIME: 2025-02-27 06:48:47 -- STEP: 1866/36938 -- GLOBAL_STEP: 1590200 | > loss_text_ce: 0.03258918970823288 (0.030101040951180942) | > loss_mel_ce: 0.5595703125 (nan) | > loss: 0.0023498395457863808 (nan) | > current_lr: 5e-06 | > step_time: 0.1714 (0.13571317402879501) | > loader_time: 0.0049 (0.004722866086970062)  --> TIME: 2025-02-27 06:48:57 -- STEP: 1916/36938 -- GLOBAL_STEP: 1590250 | > loss_text_ce: 0.036038339138031006 (0.030084363128484417) | > loss_mel_ce: 1.9041532278060913 (nan) | > loss: 0.00769917294383049 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1358608666939623) | > loader_time: 0.0045 (0.004722399228798822)  --> TIME: 2025-02-27 06:49:06 -- STEP: 1966/36938 -- GLOBAL_STEP: 1590300 | > loss_text_ce: 0.04068463295698166 (0.03010093358213637) | > loss_mel_ce: 3.1650891304016113 (nan) | > loss: 0.012721325270831585 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13578519137199116) | > loader_time: 0.0045 (0.004721716849752993)  --> TIME: 2025-02-27 06:49:15 -- STEP: 2016/36938 -- GLOBAL_STEP: 1590350 | > loss_text_ce: 0.03120863065123558 (0.030082712380958367) | > loss_mel_ce: 2.44189453125 (nan) | > loss: 0.009813901968300343 (nan) | > current_lr: 5e-06 | > step_time: 0.1061 (0.13572401163123873) | > loader_time: 0.0047 (0.004720347268240794)  --> TIME: 2025-02-27 06:49:25 -- STEP: 2066/36938 -- GLOBAL_STEP: 1590400 | > loss_text_ce: 0.031558044254779816 (0.030062002161858534) | > loss_mel_ce: 2.8858752250671387 (nan) | > loss: 0.011577116325497627 (nan) | > current_lr: 5e-06 | > step_time: 0.1622 (0.13580905410234806) | > loader_time: 0.0047 (0.004722740043167455)  --> TIME: 2025-02-27 06:49:34 -- STEP: 2116/36938 -- GLOBAL_STEP: 1590450 | > loss_text_ce: 0.03237493336200714 (0.030080563278291342) | > loss_mel_ce: 2.8297035694122314 (nan) | > loss: 0.01135745458304882 (nan) | > current_lr: 5e-06 | > step_time: 0.1717 (0.13582936295489478) | > loader_time: 0.0049 (0.004722961518624783)  --> TIME: 2025-02-27 06:49:44 -- STEP: 2166/36938 -- GLOBAL_STEP: 1590500 | > loss_text_ce: 0.03111117146909237 (0.03007010847290536) | > loss_mel_ce: 3.042245388031006 (nan) | > loss: 0.012195860035717487 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13593514694291328) | > loader_time: 0.0046 (0.004725544186290067)  --> TIME: 2025-02-27 06:49:53 -- STEP: 2216/36938 -- GLOBAL_STEP: 1590550 | > loss_text_ce: 0.0326896570622921 (0.030070252873302717) | > loss_mel_ce: 2.8472542762756348 (nan) | > loss: 0.011428349651396275 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13594266105214586) | > loader_time: 0.0056 (0.0047257220917230084)  --> TIME: 2025-02-27 06:50:02 -- STEP: 2266/36938 -- GLOBAL_STEP: 1590600 | > loss_text_ce: 0.022153174504637718 (0.03005773792395143) | > loss_mel_ce: 1.4254487752914429 (nan) | > loss: 0.005744452588260174 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13597518863240252) | > loader_time: 0.0047 (0.004727023950426958)  --> TIME: 2025-02-27 06:50:12 -- STEP: 2316/36938 -- GLOBAL_STEP: 1590650 | > loss_text_ce: 0.0338410809636116 (0.030080825290264682) | > loss_mel_ce: 2.7727158069610596 (nan) | > loss: 0.011137131601572037 (nan) | > current_lr: 5e-06 | > step_time: 0.1469 (0.13601226458689467) | > loader_time: 0.0047 (0.004727467662929874)  --> TIME: 2025-02-27 06:50:21 -- STEP: 2366/36938 -- GLOBAL_STEP: 1590700 | > loss_text_ce: 0.030408302322030067 (0.030063188362029) | > loss_mel_ce: 2.6345999240875244 (nan) | > loss: 0.0105754304677248 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.13615149849063712) | > loader_time: 0.0047 (0.004729001076666867)  --> TIME: 2025-02-27 06:50:31 -- STEP: 2416/36938 -- GLOBAL_STEP: 1590750 | > loss_text_ce: 0.025104669854044914 (0.030077461359501255) | > loss_mel_ce: 2.3149056434631348 (nan) | > loss: 0.009285756386816502 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13614556785450826) | > loader_time: 0.0059 (0.004730949338698235)  --> TIME: 2025-02-27 06:50:40 -- STEP: 2466/36938 -- GLOBAL_STEP: 1590800 | > loss_text_ce: 0.02565699815750122 (0.03010330885302786) | > loss_mel_ce: 2.4398281574249268 (nan) | > loss: 0.009783671237528324 (nan) | > current_lr: 5e-06 | > step_time: 0.1259 (0.13612229758322153) | > loader_time: 0.0048 (0.004730213490909232)  --> TIME: 2025-02-27 06:50:49 -- STEP: 2516/36938 -- GLOBAL_STEP: 1590850 | > loss_text_ce: 0.03355952352285385 (0.030101377717241504) | > loss_mel_ce: 2.027320146560669 (nan) | > loss: 0.008178094401955605 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.1360321600221106) | > loader_time: 0.0047 (0.004728730608056196)  --> TIME: 2025-02-27 06:50:58 -- STEP: 2566/36938 -- GLOBAL_STEP: 1590900 | > loss_text_ce: 0.026866931468248367 (0.030102811039469712) | > loss_mel_ce: 2.2535812854766846 (nan) | > loss: 0.009049397893249989 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.1360457303091033) | > loader_time: 0.0049 (0.004728690219501548)  --> TIME: 2025-02-27 06:51:08 -- STEP: 2616/36938 -- GLOBAL_STEP: 1590950 | > loss_text_ce: 0.03410447761416435 (0.0301017452769027) | > loss_mel_ce: 2.884124994277954 (nan) | > loss: 0.011580277234315872 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13601700601592315) | > loader_time: 0.0046 (0.004728262486443245)  --> TIME: 2025-02-27 06:51:17 -- STEP: 2666/36938 -- GLOBAL_STEP: 1591000 | > loss_text_ce: 0.02732273004949093 (0.030086673752160915) | > loss_mel_ce: 2.5937678813934326 (nan) | > loss: 0.010401153936982155 (nan) | > current_lr: 5e-06 | > step_time: 0.1576 (0.13605431337063223) | > loader_time: 0.0046 (0.004726608445209753)  --> TIME: 2025-02-27 06:51:26 -- STEP: 2716/36938 -- GLOBAL_STEP: 1591050 | > loss_text_ce: 0.0243530310690403 (0.030083032198722128) | > loss_mel_ce: 2.209118127822876 (nan) | > loss: 0.00886298157274723 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.1360390868383582) | > loader_time: 0.0045 (0.00472493638697359)  --> TIME: 2025-02-27 06:51:36 -- STEP: 2766/36938 -- GLOBAL_STEP: 1591100 | > loss_text_ce: 0.031156131997704506 (0.030080471618137365) | > loss_mel_ce: 2.988895893096924 (nan) | > loss: 0.011984333395957947 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.13606432849569566) | > loader_time: 0.0048 (0.004724104078627979)  --> TIME: 2025-02-27 06:51:45 -- STEP: 2816/36938 -- GLOBAL_STEP: 1591150 | > loss_text_ce: 0.03057374432682991 (0.030115464994609778) | > loss_mel_ce: 2.8817272186279297 (nan) | > loss: 0.01155675109475851 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13610208931971665) | > loader_time: 0.0046 (0.0047227990051562155)  --> TIME: 2025-02-27 06:51:54 -- STEP: 2866/36938 -- GLOBAL_STEP: 1591200 | > loss_text_ce: 0.018365003168582916 (0.030093468562553623) | > loss_mel_ce: 2.2656919956207275 (nan) | > loss: 0.00906371884047985 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13616168465717463) | > loader_time: 0.0046 (0.004722619173165726)  --> TIME: 2025-02-27 06:52:04 -- STEP: 2916/36938 -- GLOBAL_STEP: 1591250 | > loss_text_ce: 0.03248811513185501 (0.030087656876780042) | > loss_mel_ce: 0.2152099609375 (nan) | > loss: 0.0009829289047047496 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13613027971958425) | > loader_time: 0.0047 (0.00472064330581121)  --> TIME: 2025-02-27 06:52:13 -- STEP: 2966/36938 -- GLOBAL_STEP: 1591300 | > loss_text_ce: 0.03813362121582031 (0.03010258237521691) | > loss_mel_ce: 2.9490456581115723 (nan) | > loss: 0.011853886768221855 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.13622874982117483) | > loader_time: 0.0047 (0.004721748049164334)  --> TIME: 2025-02-27 06:52:23 -- STEP: 3016/36938 -- GLOBAL_STEP: 1591350 | > loss_text_ce: 0.022773120552301407 (0.03009787891912385) | > loss_mel_ce: 1.2341333627700806 (nan) | > loss: 0.004987724591046572 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13629817575295353) | > loader_time: 0.0053 (0.004722115927096701)  --> TIME: 2025-02-27 06:52:33 -- STEP: 3066/36938 -- GLOBAL_STEP: 1591400 | > loss_text_ce: 0.0381329320371151 (0.030090313507468772) | > loss_mel_ce: 2.95950984954834 (nan) | > loss: 0.011895408853888512 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.1364022001905997) | > loader_time: 0.0049 (0.004721648694639954)  --> TIME: 2025-02-27 06:52:42 -- STEP: 3116/36938 -- GLOBAL_STEP: 1591450 | > loss_text_ce: 0.030935466289520264 (0.030086332278754156) | > loss_mel_ce: 1.1384953260421753 (nan) | > loss: 0.004640598315745592 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1364003845295643) | > loader_time: 0.0046 (0.004720132295831325)  --> TIME: 2025-02-27 06:52:51 -- STEP: 3166/36938 -- GLOBAL_STEP: 1591500 | > loss_text_ce: 0.039262719452381134 (0.030081522890760205) | > loss_mel_ce: 1.9840223789215088 (nan) | > loss: 0.00802890956401825 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1363725762388335) | > loader_time: 0.0044 (0.004718692334332343)  --> TIME: 2025-02-27 06:53:00 -- STEP: 3216/36938 -- GLOBAL_STEP: 1591550 | > loss_text_ce: 0.025035979226231575 (0.030058935688303513) | > loss_mel_ce: 1.7156305313110352 (nan) | > loss: 0.006907407194375992 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13636899113062026) | > loader_time: 0.0046 (0.004717475368608883)  --> TIME: 2025-02-27 06:53:10 -- STEP: 3266/36938 -- GLOBAL_STEP: 1591600 | > loss_text_ce: 0.03213585913181305 (0.030057888536783026) | > loss_mel_ce: 2.2146267890930176 (nan) | > loss: 0.008915726095438004 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.1364268654340335) | > loader_time: 0.0046 (0.004717478486415585)  --> TIME: 2025-02-27 06:53:19 -- STEP: 3316/36938 -- GLOBAL_STEP: 1591650 | > loss_text_ce: 0.025021130219101906 (0.03004630075360955) | > loss_mel_ce: 2.4128286838531494 (nan) | > loss: 0.009674008004367352 (nan) | > current_lr: 5e-06 | > step_time: 0.1784 (0.1363880780263747) | > loader_time: 0.0048 (0.004717675638716525)  --> TIME: 2025-02-27 06:53:29 -- STEP: 3366/36938 -- GLOBAL_STEP: 1591700 | > loss_text_ce: 0.03544392064213753 (0.030051977365912422) | > loss_mel_ce: 2.49242901802063 (nan) | > loss: 0.010031242854893208 (nan) | > current_lr: 5e-06 | > step_time: 0.1748 (0.13643724381463274) | > loader_time: 0.005 (0.004717146011597329)  --> TIME: 2025-02-27 06:53:38 -- STEP: 3416/36938 -- GLOBAL_STEP: 1591750 | > loss_text_ce: 0.039410632103681564 (0.03005721504021375) | > loss_mel_ce: 2.356712579727173 (nan) | > loss: 0.009508425369858742 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13652823652539942) | > loader_time: 0.0045 (0.004717745602270475)  --> TIME: 2025-02-27 06:53:48 -- STEP: 3466/36938 -- GLOBAL_STEP: 1591800 | > loss_text_ce: 0.03439836576581001 (0.030035544930330105) | > loss_mel_ce: 2.9699134826660156 (nan) | > loss: 0.011921873316168785 (nan) | > current_lr: 5e-06 | > step_time: 0.1984 (0.13660867078003006) | > loader_time: 0.0049 (0.004717416523924051)  --> TIME: 2025-02-27 06:53:57 -- STEP: 3516/36938 -- GLOBAL_STEP: 1591850 | > loss_text_ce: 0.019177518784999847 (0.03003988431187069) | > loss_mel_ce: 2.412338972091675 (nan) | > loss: 0.009648875333368778 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13663565773470274) | > loader_time: 0.0046 (0.00471678006635455)  --> TIME: 2025-02-27 06:54:07 -- STEP: 3566/36938 -- GLOBAL_STEP: 1591900 | > loss_text_ce: 0.026523005217313766 (0.030051649940125456) | > loss_mel_ce: 2.6960582733154297 (nan) | > loss: 0.010803895071148872 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.1365891309915736) | > loader_time: 0.0046 (0.004715854641566163)  --> TIME: 2025-02-27 06:54:16 -- STEP: 3616/36938 -- GLOBAL_STEP: 1591950 | > loss_text_ce: 0.025632845237851143 (0.030047046942935137) | > loss_mel_ce: 1.7222068309783936 (nan) | > loss: 0.006935872137546539 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13652799720257788) | > loader_time: 0.0045 (0.004716640156982216)  --> TIME: 2025-02-27 06:54:25 -- STEP: 3666/36938 -- GLOBAL_STEP: 1592000 | > loss_text_ce: 0.03276384621858597 (0.030052573807015242) | > loss_mel_ce: 2.687774419784546 (nan) | > loss: 0.010795787908136845 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13649901085971688) | > loader_time: 0.0045 (0.004715186419281485)  --> TIME: 2025-02-27 06:54:34 -- STEP: 3716/36938 -- GLOBAL_STEP: 1592050 | > loss_text_ce: 0.02411341667175293 (0.030041703770129637) | > loss_mel_ce: 2.4421799182891846 (nan) | > loss: 0.0097868787124753 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13653800972567953) | > loader_time: 0.0045 (0.0047141615746480615)  --> TIME: 2025-02-27 06:54:44 -- STEP: 3766/36938 -- GLOBAL_STEP: 1592100 | > loss_text_ce: 0.02311776392161846 (0.030054172034812098) | > loss_mel_ce: 2.9359238147735596 (nan) | > loss: 0.01174222957342863 (nan) | > current_lr: 5e-06 | > step_time: 0.1266 (0.13659610947078674) | > loader_time: 0.0046 (0.0047133988797253105)  --> TIME: 2025-02-27 06:54:53 -- STEP: 3816/36938 -- GLOBAL_STEP: 1592150 | > loss_text_ce: 0.03357386961579323 (0.03006350892997177) | > loss_mel_ce: 2.4438698291778564 (nan) | > loss: 0.009831126779317856 (nan) | > current_lr: 5e-06 | > step_time: 0.1536 (0.13659527029381352) | > loader_time: 0.007 (0.004713667388732075)  --> TIME: 2025-02-27 06:55:02 -- STEP: 3866/36938 -- GLOBAL_STEP: 1592200 | > loss_text_ce: 0.03277214989066124 (0.030077970398332685) | > loss_mel_ce: 0.7619300484657288 (nan) | > loss: 0.003153580240905285 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13658789896434653) | > loader_time: 0.0046 (0.004713603393191567)  --> TIME: 2025-02-27 06:55:12 -- STEP: 3916/36938 -- GLOBAL_STEP: 1592250 | > loss_text_ce: 0.031058404594659805 (0.0300836665748473) | > loss_mel_ce: 3.206793785095215 (nan) | > loss: 0.012848620302975178 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13667571063669995) | > loader_time: 0.0045 (0.004713456465104502)  --> TIME: 2025-02-27 06:55:22 -- STEP: 3966/36938 -- GLOBAL_STEP: 1592300 | > loss_text_ce: 0.03547066077589989 (0.030080209408458878) | > loss_mel_ce: 2.5918874740600586 (nan) | > loss: 0.010426024906337261 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13673475993382944) | > loader_time: 0.0045 (0.004713058712142194)  --> TIME: 2025-02-27 06:55:31 -- STEP: 4016/36938 -- GLOBAL_STEP: 1592350 | > loss_text_ce: 0.035369351506233215 (0.030094403589647115) | > loss_mel_ce: 3.280571222305298 (nan) | > loss: 0.013158495537936687 (nan) | > current_lr: 5e-06 | > step_time: 0.1807 (0.13673341630464525) | > loader_time: 0.0048 (0.00471251561822169)  --> TIME: 2025-02-27 06:55:40 -- STEP: 4066/36938 -- GLOBAL_STEP: 1592400 | > loss_text_ce: 0.027882615104317665 (0.030106568631825557) | > loss_mel_ce: 2.0500617027282715 (nan) | > loss: 0.008245810866355896 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.1367142490723774) | > loader_time: 0.0046 (0.004711724183738844)  --> TIME: 2025-02-27 06:55:50 -- STEP: 4116/36938 -- GLOBAL_STEP: 1592450 | > loss_text_ce: 0.040656741708517075 (0.030099080063889585) | > loss_mel_ce: 3.154169797897339 (nan) | > loss: 0.012677883729338646 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13675701890441253) | > loader_time: 0.0045 (0.004710742926806121)  --> TIME: 2025-02-27 06:55:59 -- STEP: 4166/36938 -- GLOBAL_STEP: 1592500 | > loss_text_ce: 0.027278603985905647 (0.030104974737910413) | > loss_mel_ce: 1.9248114824295044 (nan) | > loss: 0.00774639006704092 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13676205673758016) | > loader_time: 0.0045 (0.004709609071051218)  --> TIME: 2025-02-27 06:56:08 -- STEP: 4216/36938 -- GLOBAL_STEP: 1592550 | > loss_text_ce: 0.033021558076143265 (0.030113262358736567) | > loss_mel_ce: 3.158712387084961 (nan) | > loss: 0.012665610760450363 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13666988007257733) | > loader_time: 0.0046 (0.004708660621570677)  --> TIME: 2025-02-27 06:56:18 -- STEP: 4266/36938 -- GLOBAL_STEP: 1592600 | > loss_text_ce: 0.022662261500954628 (0.030100740940758475) | > loss_mel_ce: 2.1166932582855225 (nan) | > loss: 0.00848950631916523 (nan) | > current_lr: 5e-06 | > step_time: 0.1084 (0.1367382411938902) | > loader_time: 0.0046 (0.004708465700466975)  --> TIME: 2025-02-27 06:56:27 -- STEP: 4316/36938 -- GLOBAL_STEP: 1592650 | > loss_text_ce: 0.02612292766571045 (0.030096183776548192) | > loss_mel_ce: 2.2805545330047607 (nan) | > loss: 0.009153482504189014 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.13673435813083587) | > loader_time: 0.0047 (0.004707894466672366)  --> TIME: 2025-02-27 06:56:37 -- STEP: 4366/36938 -- GLOBAL_STEP: 1592700 | > loss_text_ce: 0.027138056233525276 (0.03008594112539006) | > loss_mel_ce: 2.314432144165039 (nan) | > loss: 0.009291945956647396 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13673801013651196) | > loader_time: 0.0047 (0.004707042689286287)  --> TIME: 2025-02-27 06:56:46 -- STEP: 4416/36938 -- GLOBAL_STEP: 1592750 | > loss_text_ce: 0.03658480942249298 (0.03008896029327382) | > loss_mel_ce: 2.30967116355896 (nan) | > loss: 0.009310540743172169 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13671106359233018) | > loader_time: 0.0047 (0.004706197404775064)  --> TIME: 2025-02-27 06:56:55 -- STEP: 4466/36938 -- GLOBAL_STEP: 1592800 | > loss_text_ce: 0.030835190787911415 (0.03009024289642255) | > loss_mel_ce: 2.7799770832061768 (nan) | > loss: 0.01115401741117239 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.136788990910791) | > loader_time: 0.0046 (0.004706564368125411)  --> TIME: 2025-02-27 06:57:05 -- STEP: 4516/36938 -- GLOBAL_STEP: 1592850 | > loss_text_ce: 0.01845463737845421 (0.03008100342445168) | > loss_mel_ce: 2.210906982421875 (nan) | > loss: 0.008846673183143139 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13687618856835512) | > loader_time: 0.0048 (0.004707356223603918)  --> TIME: 2025-02-27 06:57:15 -- STEP: 4566/36938 -- GLOBAL_STEP: 1592900 | > loss_text_ce: 0.03035975806415081 (0.030081486020423227) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13686220080926662) | > loader_time: 0.0045 (0.004707455739502943)  --> TIME: 2025-02-27 06:57:24 -- STEP: 4616/36938 -- GLOBAL_STEP: 1592950 | > loss_text_ce: 0.028157565742731094 (0.03007788739334211) | > loss_mel_ce: 0.7616500854492188 (nan) | > loss: 0.0031341577414423227 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13685615497610606) | > loader_time: 0.0044 (0.004707114483702741)  --> TIME: 2025-02-27 06:57:33 -- STEP: 4666/36938 -- GLOBAL_STEP: 1593000 | > loss_text_ce: 0.03189893066883087 (0.03008629316248501) | > loss_mel_ce: 2.349591016769409 (nan) | > loss: 0.009450357407331467 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.1367765127513934) | > loader_time: 0.0045 (0.004706765519055082)  --> TIME: 2025-02-27 06:57:42 -- STEP: 4716/36938 -- GLOBAL_STEP: 1593050 | > loss_text_ce: 0.03765532374382019 (0.030091020513833062) | > loss_mel_ce: 2.2841944694519043 (nan) | > loss: 0.009213690645992756 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.1367430470574196) | > loader_time: 0.0048 (0.0047056649679648255)  --> TIME: 2025-02-27 06:57:51 -- STEP: 4766/36938 -- GLOBAL_STEP: 1593100 | > loss_text_ce: 0.038178253918886185 (0.030100876609065918) | > loss_mel_ce: 2.5279760360717773 (nan) | > loss: 0.010183152742683887 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13670063914244188) | > loader_time: 0.0045 (0.004706107714673426)  --> TIME: 2025-02-27 06:58:01 -- STEP: 4816/36938 -- GLOBAL_STEP: 1593150 | > loss_text_ce: 0.029810993000864983 (0.030093452952565485) | > loss_mel_ce: 2.270702362060547 (nan) | > loss: 0.009129021316766739 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13673996351089995) | > loader_time: 0.0046 (0.0047064400293502315)  --> TIME: 2025-02-27 06:58:10 -- STEP: 4866/36938 -- GLOBAL_STEP: 1593200 | > loss_text_ce: 0.03235658258199692 (0.03010034332843478) | > loss_mel_ce: 2.764469861984253 (nan) | > loss: 0.011098518036305904 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13674694385442343) | > loader_time: 0.0045 (0.004706281230031243)  --> TIME: 2025-02-27 06:58:19 -- STEP: 4916/36938 -- GLOBAL_STEP: 1593250 | > loss_text_ce: 0.026243366301059723 (0.03010674015477912) | > loss_mel_ce: 1.9012993574142456 (nan) | > loss: 0.007648979313671589 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1367515348825542) | > loader_time: 0.0046 (0.0047056309278657275)  --> TIME: 2025-02-27 06:58:29 -- STEP: 4966/36938 -- GLOBAL_STEP: 1593300 | > loss_text_ce: 0.0320880264043808 (0.03009956718774672) | > loss_mel_ce: 3.123378276824951 (nan) | > loss: 0.012521692551672459 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1368175567372692) | > loader_time: 0.0046 (0.004705511510636214)  --> TIME: 2025-02-27 06:58:39 -- STEP: 5016/36938 -- GLOBAL_STEP: 1593350 | > loss_text_ce: 0.028341542929410934 (0.030105162565477572) | > loss_mel_ce: 2.2900726795196533 (nan) | > loss: 0.00920005701482296 (nan) | > current_lr: 5e-06 | > step_time: 0.1956 (0.13690207772658022) | > loader_time: 0.0049 (0.00470602925884666)  --> TIME: 2025-02-27 06:58:48 -- STEP: 5066/36938 -- GLOBAL_STEP: 1593400 | > loss_text_ce: 0.030705831944942474 (0.030108472546897136) | > loss_mel_ce: 2.1141695976257324 (nan) | > loss: 0.008511411026120186 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.1368781017118231) | > loader_time: 0.0045 (0.004705907937993741)  --> TIME: 2025-02-27 06:58:57 -- STEP: 5116/36938 -- GLOBAL_STEP: 1593450 | > loss_text_ce: 0.023195946589112282 (0.03010501488959497) | > loss_mel_ce: 3.098353862762451 (nan) | > loss: 0.012387103401124477 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.13681626683385048) | > loader_time: 0.0046 (0.004705154308590492)  --> TIME: 2025-02-27 06:59:07 -- STEP: 5166/36938 -- GLOBAL_STEP: 1593500 | > loss_text_ce: 0.036494117230176926 (0.03010926313285092) | > loss_mel_ce: 2.3330631256103516 (nan) | > loss: 0.009403005242347717 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.136837880278206) | > loader_time: 0.0047 (0.0047047241593440895)  --> TIME: 2025-02-27 06:59:16 -- STEP: 5216/36938 -- GLOBAL_STEP: 1593550 | > loss_text_ce: 0.04140833765268326 (0.030102382366726906) | > loss_mel_ce: 1.5238856077194214 (nan) | > loss: 0.00621148431673646 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.13680357613995028) | > loader_time: 0.0045 (0.0047040354986132155)  --> TIME: 2025-02-27 06:59:25 -- STEP: 5266/36938 -- GLOBAL_STEP: 1593600 | > loss_text_ce: 0.023278145119547844 (0.03010731876165982) | > loss_mel_ce: 2.9519245624542236 (nan) | > loss: 0.011806361377239227 (nan) | > current_lr: 5e-06 | > step_time: 0.1614 (0.13681825054855445) | > loader_time: 0.0047 (0.004703592719843378)  --> TIME: 2025-02-27 06:59:35 -- STEP: 5316/36938 -- GLOBAL_STEP: 1593650 | > loss_text_ce: 0.027082817628979683 (0.030105763058051287) | > loss_mel_ce: 2.1203112602233887 (nan) | > loss: 0.00852140597999096 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13679926650159965) | > loader_time: 0.0046 (0.0047039502181891865)  --> TIME: 2025-02-27 06:59:44 -- STEP: 5366/36938 -- GLOBAL_STEP: 1593700 | > loss_text_ce: 0.03678372874855995 (0.03009768077850958) | > loss_mel_ce: 2.7406976222991943 (nan) | > loss: 0.011021751910448074 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13681031273142688) | > loader_time: 0.0046 (0.004703614412294215)  --> TIME: 2025-02-27 06:59:53 -- STEP: 5416/36938 -- GLOBAL_STEP: 1593750 | > loss_text_ce: 0.027092529460787773 (0.03009365511675705) | > loss_mel_ce: 0.0728759765625 (nan) | > loss: 0.0003967004595324397 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.1367947803906537) | > loader_time: 0.0045 (0.00470329017723679)  --> TIME: 2025-02-27 07:00:03 -- STEP: 5466/36938 -- GLOBAL_STEP: 1593800 | > loss_text_ce: 0.02756747603416443 (0.03010384948308073) | > loss_mel_ce: 2.4583888053894043 (nan) | > loss: 0.00986490584909916 (nan) | > current_lr: 5e-06 | > step_time: 0.1513 (0.13680529821237478) | > loader_time: 0.0046 (0.004702907885730445)  --> TIME: 2025-02-27 07:00:12 -- STEP: 5516/36938 -- GLOBAL_STEP: 1593850 | > loss_text_ce: 0.02452501654624939 (0.03009856547998228) | > loss_mel_ce: 2.7368900775909424 (nan) | > loss: 0.010957996360957623 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.13684083647793685) | > loader_time: 0.0047 (0.00470326969466232)  --> TIME: 2025-02-27 07:00:22 -- STEP: 5566/36938 -- GLOBAL_STEP: 1593900 | > loss_text_ce: 0.029266666620969772 (0.03009684394577875) | > loss_mel_ce: 2.7711665630340576 (nan) | > loss: 0.011112830601632595 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13683445420232654) | > loader_time: 0.0047 (0.004704337731737294)  --> TIME: 2025-02-27 07:00:31 -- STEP: 5616/36938 -- GLOBAL_STEP: 1593950 | > loss_text_ce: 0.03655615821480751 (0.03008546320319526) | > loss_mel_ce: 1.9489531517028809 (nan) | > loss: 0.007879005745053291 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13681802140851315) | > loader_time: 0.0048 (0.004704476973609697)  --> TIME: 2025-02-27 07:00:40 -- STEP: 5666/36938 -- GLOBAL_STEP: 1594000 | > loss_text_ce: 0.030929703265428543 (0.03009310909906883) | > loss_mel_ce: 2.3308019638061523 (nan) | > loss: 0.009371952153742313 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.1367785870040049) | > loader_time: 0.0046 (0.004703932123221226)  --> TIME: 2025-02-27 07:00:49 -- STEP: 5716/36938 -- GLOBAL_STEP: 1594050 | > loss_text_ce: 0.03637603670358658 (0.03009564027990533) | > loss_mel_ce: 2.025986671447754 (nan) | > loss: 0.0081839794293046 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.1367775833715169) | > loader_time: 0.0047 (0.004703970911454443)  --> TIME: 2025-02-27 07:00:59 -- STEP: 5766/36938 -- GLOBAL_STEP: 1594100 | > loss_text_ce: 0.0255750622600317 (0.030103759445376667) | > loss_mel_ce: 2.378354787826538 (nan) | > loss: 0.009539404883980751 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13680805119631254) | > loader_time: 0.0045 (0.0047036576014692945)  --> TIME: 2025-02-27 07:01:08 -- STEP: 5816/36938 -- GLOBAL_STEP: 1594150 | > loss_text_ce: 0.029222577810287476 (0.030100840263637463) | > loss_mel_ce: 3.0252957344055176 (nan) | > loss: 0.012121104635298252 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.1367488073023675) | > loader_time: 0.0046 (0.004703054811800348)  --> TIME: 2025-02-27 07:01:17 -- STEP: 5866/36938 -- GLOBAL_STEP: 1594200 | > loss_text_ce: 0.022069785743951797 (0.030104509336023537) | > loss_mel_ce: 2.2887086868286133 (nan) | > loss: 0.009169756434857845 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13675748257355771) | > loader_time: 0.0067 (0.004703248599833109)  --> TIME: 2025-02-27 07:01:26 -- STEP: 5916/36938 -- GLOBAL_STEP: 1594250 | > loss_text_ce: 0.02761339582502842 (0.0301026476884325) | > loss_mel_ce: 1.05133056640625 (nan) | > loss: 0.0042815241031348705 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13676411074669978) | > loader_time: 0.0046 (0.0047031850166784905)  --> TIME: 2025-02-27 07:01:36 -- STEP: 5966/36938 -- GLOBAL_STEP: 1594300 | > loss_text_ce: 0.03337206691503525 (0.03009885713315069) | > loss_mel_ce: 2.2138569355010986 (nan) | > loss: 0.008917576633393764 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13672815822252232) | > loader_time: 0.0047 (0.004703242268052375)  --> TIME: 2025-02-27 07:01:45 -- STEP: 6016/36938 -- GLOBAL_STEP: 1594350 | > loss_text_ce: 0.02625204063951969 (0.030090811267461718) | > loss_mel_ce: 1.5791226625442505 (nan) | > loss: 0.0063705346547067165 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13671632868019853) | > loader_time: 0.0046 (0.004703741045074242)  --> TIME: 2025-02-27 07:01:54 -- STEP: 6066/36938 -- GLOBAL_STEP: 1594400 | > loss_text_ce: 0.04287082701921463 (0.030096379597418003) | > loss_mel_ce: 1.8067991733551025 (nan) | > loss: 0.007339960895478725 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13673233482808633) | > loader_time: 0.0047 (0.004704144423132939)  --> TIME: 2025-02-27 07:02:04 -- STEP: 6116/36938 -- GLOBAL_STEP: 1594450 | > loss_text_ce: 0.030655959621071815 (0.03008507600631577) | > loss_mel_ce: 1.4701452255249023 (nan) | > loss: 0.005955560598522425 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13673350793852262) | > loader_time: 0.0046 (0.004704439265810629)  --> TIME: 2025-02-27 07:02:13 -- STEP: 6166/36938 -- GLOBAL_STEP: 1594500 | > loss_text_ce: 0.032187674194574356 (0.03007768387439535) | > loss_mel_ce: 0.8046200275421143 (nan) | > loss: 0.0033206657972186804 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13673049124502154) | > loader_time: 0.0045 (0.004704207868263617)  --> TIME: 2025-02-27 07:02:22 -- STEP: 6216/36938 -- GLOBAL_STEP: 1594550 | > loss_text_ce: 0.02477957494556904 (0.030061905472922923) | > loss_mel_ce: 2.4546873569488525 (nan) | > loss: 0.009839154779911041 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13671640708486435) | > loader_time: 0.0047 (0.004703951848519815)  --> TIME: 2025-02-27 07:02:31 -- STEP: 6266/36938 -- GLOBAL_STEP: 1594600 | > loss_text_ce: 0.03364080190658569 (0.030054167395935028) | > loss_mel_ce: 2.5935111045837402 (nan) | > loss: 0.010425207205116749 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.1366731941300465) | > loader_time: 0.0047 (0.0047033994372112425)  --> TIME: 2025-02-27 07:02:41 -- STEP: 6316/36938 -- GLOBAL_STEP: 1594650 | > loss_text_ce: 0.026310374960303307 (0.03005475442348717) | > loss_mel_ce: 3.1016833782196045 (nan) | > loss: 0.012412674725055695 (nan) | > current_lr: 5e-06 | > step_time: 0.2004 (0.13667588852265775) | > loader_time: 0.0049 (0.004702964185384947)  --> TIME: 2025-02-27 07:02:50 -- STEP: 6366/36938 -- GLOBAL_STEP: 1594700 | > loss_text_ce: 0.017333118245005608 (0.030055579279516943) | > loss_mel_ce: 2.185771942138672 (nan) | > loss: 0.008742480538785458 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13665544009231184) | > loader_time: 0.0045 (0.004703170467464195)  --> TIME: 2025-02-27 07:02:59 -- STEP: 6416/36938 -- GLOBAL_STEP: 1594750 | > loss_text_ce: 0.024586813524365425 (0.03005333060147287) | > loss_mel_ce: 0.019287109375 (nan) | > loss: 0.00017410286818630993 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1366290933919372) | > loader_time: 0.0045 (0.004702546538557494)  --> TIME: 2025-02-27 07:03:08 -- STEP: 6466/36938 -- GLOBAL_STEP: 1594800 | > loss_text_ce: 0.021973291411995888 (0.03004812991561727) | > loss_mel_ce: 1.4798601865768433 (nan) | > loss: 0.005959657020866871 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13665888558737932) | > loader_time: 0.0045 (0.004702710308897357)  --> TIME: 2025-02-27 07:03:18 -- STEP: 6516/36938 -- GLOBAL_STEP: 1594850 | > loss_text_ce: 0.026507725939154625 (0.030040572604085585) | > loss_mel_ce: 2.606842041015625 (nan) | > loss: 0.01044980064034462 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.13666607958177213) | > loader_time: 0.005 (0.004703072260463801)  --> TIME: 2025-02-27 07:03:27 -- STEP: 6566/36938 -- GLOBAL_STEP: 1594900 | > loss_text_ce: 0.027746938169002533 (0.030037474194430938) | > loss_mel_ce: 2.212970733642578 (nan) | > loss: 0.008891737088561058 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13666138985227497) | > loader_time: 0.0046 (0.004702938971924699)  --> TIME: 2025-02-27 07:03:36 -- STEP: 6616/36938 -- GLOBAL_STEP: 1594950 | > loss_text_ce: 0.025645002722740173 (0.030039578780110706) | > loss_mel_ce: 3.155280113220215 (nan) | > loss: 0.012622719630599022 (nan) | > current_lr: 5e-06 | > step_time: 0.1709 (0.1366223381897562) | > loader_time: 0.0049 (0.00470245994420265)  --> TIME: 2025-02-27 07:03:45 -- STEP: 6666/36938 -- GLOBAL_STEP: 1595000 | > loss_text_ce: 0.023269042372703552 (0.030048054993094497) | > loss_mel_ce: 1.9267499446868896 (nan) | > loss: 0.007738171145319939 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.1366027024569831) | > loader_time: 0.0046 (0.004702164645385281)  --> TIME: 2025-02-27 07:03:55 -- STEP: 6716/36938 -- GLOBAL_STEP: 1595050 | > loss_text_ce: 0.03089982457458973 (0.030047471327746) | > loss_mel_ce: 2.7461280822753906 (nan) | > loss: 0.011019952595233917 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13659844333178414) | > loader_time: 0.0046 (0.004702414090325248)  --> TIME: 2025-02-27 07:04:04 -- STEP: 6766/36938 -- GLOBAL_STEP: 1595100 | > loss_text_ce: 0.03024042211472988 (0.03005354096212799) | > loss_mel_ce: 2.829246759414673 (nan) | > loss: 0.011347172781825066 (nan) | > current_lr: 5e-06 | > step_time: 0.1857 (0.13660385007517206) | > loader_time: 0.0048 (0.004701944275646561)  --> TIME: 2025-02-27 07:04:14 -- STEP: 6816/36938 -- GLOBAL_STEP: 1595150 | > loss_text_ce: 0.028625736013054848 (0.03004680855386615) | > loss_mel_ce: 2.833456516265869 (nan) | > loss: 0.011357469484210014 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.1366288352740206) | > loader_time: 0.0046 (0.004702020349077209)  --> TIME: 2025-02-27 07:04:23 -- STEP: 6866/36938 -- GLOBAL_STEP: 1595200 | > loss_text_ce: 0.027313709259033203 (0.030049725614993802) | > loss_mel_ce: 2.6429946422576904 (nan) | > loss: 0.010596462525427341 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.13667759716840688) | > loader_time: 0.0046 (0.004701909816129674)  --> TIME: 2025-02-27 07:04:32 -- STEP: 6916/36938 -- GLOBAL_STEP: 1595250 | > loss_text_ce: 0.0235569067299366 (0.030048276474963314) | > loss_mel_ce: 2.4711649417877197 (nan) | > loss: 0.009899690747261047 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13666136268937887) | > loader_time: 0.0046 (0.004701887444308061)  --> TIME: 2025-02-27 07:04:42 -- STEP: 6966/36938 -- GLOBAL_STEP: 1595300 | > loss_text_ce: 0.036801014095544815 (0.030050034656427627) | > loss_mel_ce: 1.1483451128005981 (nan) | > loss: 0.0047029610723257065 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.1366369209514078) | > loader_time: 0.0048 (0.004701958317336351)  --> TIME: 2025-02-27 07:04:51 -- STEP: 7016/36938 -- GLOBAL_STEP: 1595350 | > loss_text_ce: 0.02577328309416771 (0.030051097800549514) | > loss_mel_ce: 2.7288291454315186 (nan) | > loss: 0.010930962860584259 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13664386790997674) | > loader_time: 0.0046 (0.004701562387233715)  --> TIME: 2025-02-27 07:05:00 -- STEP: 7066/36938 -- GLOBAL_STEP: 1595400 | > loss_text_ce: 0.038075514137744904 (0.030055913963994898) | > loss_mel_ce: 2.7315738201141357 (nan) | > loss: 0.010990672744810581 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.13665114537602724) | > loader_time: 0.0046 (0.00470110123669699)  --> TIME: 2025-02-27 07:05:10 -- STEP: 7116/36938 -- GLOBAL_STEP: 1595450 | > loss_text_ce: 0.020489806309342384 (0.030056724935233963) | > loss_mel_ce: 1.8312586545944214 (nan) | > loss: 0.00734820868819952 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13667977251585753) | > loader_time: 0.0045 (0.004700811610187012)  --> TIME: 2025-02-27 07:05:19 -- STEP: 7166/36938 -- GLOBAL_STEP: 1595500 | > loss_text_ce: 0.019517313688993454 (0.03006653829435879) | > loss_mel_ce: 2.239182949066162 (nan) | > loss: 0.008963096886873245 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13666491178437568) | > loader_time: 0.0046 (0.0047003140237921555)  --> TIME: 2025-02-27 07:05:28 -- STEP: 7216/36938 -- GLOBAL_STEP: 1595550 | > loss_text_ce: 0.02325514703989029 (0.03007376772721201) | > loss_mel_ce: 2.823218822479248 (nan) | > loss: 0.011295531876385212 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.13663062740720344) | > loader_time: 0.0046 (0.004699928434090703)  --> TIME: 2025-02-27 07:05:38 -- STEP: 7266/36938 -- GLOBAL_STEP: 1595600 | > loss_text_ce: 0.030122537165880203 (0.0300697517883009) | > loss_mel_ce: 2.7678005695343018 (nan) | > loss: 0.011102870106697083 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.1366758444011955) | > loader_time: 0.0049 (0.004699537093206894)  --> TIME: 2025-02-27 07:05:47 -- STEP: 7316/36938 -- GLOBAL_STEP: 1595650 | > loss_text_ce: 0.02984573133289814 (0.030071901097667633) | > loss_mel_ce: 2.5794837474823 (nan) | > loss: 0.01035448256880045 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13668234726977313) | > loader_time: 0.0044 (0.004699226055332968)  --> TIME: 2025-02-27 07:05:56 -- STEP: 7366/36938 -- GLOBAL_STEP: 1595700 | > loss_text_ce: 0.029869064688682556 (0.03007333830935603) | > loss_mel_ce: 2.69783091545105 (nan) | > loss: 0.010824207216501236 (nan) | > current_lr: 5e-06 | > step_time: 0.1813 (0.13667049467515133) | > loader_time: 0.0048 (0.004698802134668954)  --> TIME: 2025-02-27 07:06:06 -- STEP: 7416/36938 -- GLOBAL_STEP: 1595750 | > loss_text_ce: 0.028273731470108032 (0.03006734155375157) | > loss_mel_ce: 0.23529052734375 (nan) | > loss: 0.0010458899196237326 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1366900947616071) | > loader_time: 0.0047 (0.00469809108310275)  --> TIME: 2025-02-27 07:06:15 -- STEP: 7466/36938 -- GLOBAL_STEP: 1595800 | > loss_text_ce: 0.039954233914613724 (0.03006827884470705) | > loss_mel_ce: 3.12292742729187 (nan) | > loss: 0.01255111861974001 (nan) | > current_lr: 5e-06 | > step_time: 0.174 (0.1366625029185555) | > loader_time: 0.0048 (0.004697470475929269)  --> TIME: 2025-02-27 07:06:25 -- STEP: 7516/36938 -- GLOBAL_STEP: 1595850 | > loss_text_ce: 0.03582487627863884 (0.030067464427084738) | > loss_mel_ce: 2.511810779571533 (nan) | > loss: 0.010109665803611279 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.1367189272278612) | > loader_time: 0.0047 (0.004697574206398414)  --> TIME: 2025-02-27 07:06:34 -- STEP: 7566/36938 -- GLOBAL_STEP: 1595900 | > loss_text_ce: 0.030091125518083572 (0.03006732927713107) | > loss_mel_ce: 2.6921136379241943 (nan) | > loss: 0.010802400298416615 (nan) | > current_lr: 5e-06 | > step_time: 0.178 (0.13671578987231536) | > loader_time: 0.0048 (0.004697254527790529)  --> TIME: 2025-02-27 07:06:43 -- STEP: 7616/36938 -- GLOBAL_STEP: 1595950 | > loss_text_ce: 0.026589900255203247 (0.030072368730467102) | > loss_mel_ce: 0.10791015625 (nan) | > loss: 0.0005337304319255054 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13671579963269395) | > loader_time: 0.0046 (0.004696864039958019)  --> TIME: 2025-02-27 07:06:53 -- STEP: 7666/36938 -- GLOBAL_STEP: 1596000 | > loss_text_ce: 0.055478692054748535 (0.030075313845231932) | > loss_mel_ce: 1.0508588552474976 (nan) | > loss: 0.0043902285397052765 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13677180640665798) | > loader_time: 0.0046 (0.0046967691893742974)  --> TIME: 2025-02-27 07:07:02 -- STEP: 7716/36938 -- GLOBAL_STEP: 1596050 | > loss_text_ce: 0.03255678340792656 (0.030081837786508348) | > loss_mel_ce: 2.7974472045898438 (nan) | > loss: 0.011230175383388996 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.1367551088951119) | > loader_time: 0.0046 (0.004696664629729065)  --> TIME: 2025-02-27 07:07:11 -- STEP: 7766/36938 -- GLOBAL_STEP: 1596100 | > loss_text_ce: 0.03192121908068657 (0.030079894548412985) | > loss_mel_ce: 3.1811728477478027 (nan) | > loss: 0.012750374153256416 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13673166752230018) | > loader_time: 0.0046 (0.0046966766961112644)  --> TIME: 2025-02-27 07:07:21 -- STEP: 7816/36938 -- GLOBAL_STEP: 1596150 | > loss_text_ce: 0.02404649928212166 (0.030071809514332276) | > loss_mel_ce: 1.512119174003601 (nan) | > loss: 0.0060958960093557835 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.1367231371031817) | > loader_time: 0.0046 (0.004697449345183587)  --> TIME: 2025-02-27 07:07:30 -- STEP: 7866/36938 -- GLOBAL_STEP: 1596200 | > loss_text_ce: 0.02891470119357109 (0.03006806504040551) | > loss_mel_ce: 2.841567039489746 (nan) | > loss: 0.011390801519155502 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13673304884057336) | > loader_time: 0.0046 (0.004697303356119243)  --> TIME: 2025-02-27 07:07:40 -- STEP: 7916/36938 -- GLOBAL_STEP: 1596250 | > loss_text_ce: 0.029142538085579872 (0.030076751142044698) | > loss_mel_ce: 0.2418878674507141 (nan) | > loss: 0.0010755175026133657 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.1367602988949568) | > loader_time: 0.0047 (0.004697441994511395)  --> TIME: 2025-02-27 07:07:49 -- STEP: 7966/36938 -- GLOBAL_STEP: 1596300 | > loss_text_ce: 0.027575068175792694 (0.030075853237002608) | > loss_mel_ce: 3.013788938522339 (nan) | > loss: 0.01206890493631363 (nan) | > current_lr: 5e-06 | > step_time: 0.1522 (0.13677319542355473) | > loader_time: 0.0047 (0.004697433225539286)  --> TIME: 2025-02-27 07:07:59 -- STEP: 8016/36938 -- GLOBAL_STEP: 1596350 | > loss_text_ce: 0.03024691715836525 (0.03007212266344075) | > loss_mel_ce: 2.9224743843078613 (nan) | > loss: 0.011717149056494236 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.13677136466174822) | > loader_time: 0.0046 (0.004697057241927121)  --> TIME: 2025-02-27 07:08:08 -- STEP: 8066/36938 -- GLOBAL_STEP: 1596400 | > loss_text_ce: 0.026898980140686035 (0.030068180685345162) | > loss_mel_ce: 2.224592685699463 (nan) | > loss: 0.008934490382671356 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13680041152609845) | > loader_time: 0.0045 (0.004696780624976083)  --> TIME: 2025-02-27 07:08:18 -- STEP: 8116/36938 -- GLOBAL_STEP: 1596450 | > loss_text_ce: 0.023440586403012276 (0.0300635830796438) | > loss_mel_ce: 2.6882736682891846 (nan) | > loss: 0.010760771110653877 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13680568996939926) | > loader_time: 0.0046 (0.004696803441947755)  --> TIME: 2025-02-27 07:08:27 -- STEP: 8166/36938 -- GLOBAL_STEP: 1596500 | > loss_text_ce: 0.02984451875090599 (0.030064518651266966) | > loss_mel_ce: 2.5049357414245605 (nan) | > loss: 0.010058652609586716 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.13683575381550675) | > loader_time: 0.0047 (0.004697069449077884)  --> TIME: 2025-02-27 07:08:36 -- STEP: 8216/36938 -- GLOBAL_STEP: 1596550 | > loss_text_ce: 0.04432031139731407 (0.030068449121958846) | > loss_mel_ce: 2.259089469909668 (nan) | > loss: 0.009140515699982643 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13681960204560004) | > loader_time: 0.0046 (0.00469706260473145)  --> TIME: 2025-02-27 07:08:46 -- STEP: 8266/36938 -- GLOBAL_STEP: 1596600 | > loss_text_ce: 0.020673397928476334 (0.030062796979306142) | > loss_mel_ce: 2.62628173828125 (nan) | > loss: 0.010503791272640228 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.1368086494373812) | > loader_time: 0.0046 (0.004697046699865425)  --> TIME: 2025-02-27 07:08:55 -- STEP: 8316/36938 -- GLOBAL_STEP: 1596650 | > loss_text_ce: 0.02233232744038105 (0.030062694211157617) | > loss_mel_ce: 2.45754337310791 (nan) | > loss: 0.009840777143836021 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13680953888322062) | > loader_time: 0.0046 (0.0046966939146785855)  --> TIME: 2025-02-27 07:09:04 -- STEP: 8366/36938 -- GLOBAL_STEP: 1596700 | > loss_text_ce: 0.030893227085471153 (0.030067144329818832) | > loss_mel_ce: 1.53125 (nan) | > loss: 0.006198981311172247 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1368053178880711) | > loader_time: 0.0046 (0.00469639405034644)  --> TIME: 2025-02-27 07:09:13 -- STEP: 8416/36938 -- GLOBAL_STEP: 1596750 | > loss_text_ce: 0.03682363033294678 (0.030067417848529475) | > loss_mel_ce: 2.9910976886749268 (nan) | > loss: 0.012015560641884804 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13678506724961398) | > loader_time: 0.0046 (0.004696175484376255)  --> TIME: 2025-02-27 07:09:22 -- STEP: 8466/36938 -- GLOBAL_STEP: 1596800 | > loss_text_ce: 0.027900299057364464 (0.03006481347764437) | > loss_mel_ce: 3.143484115600586 (nan) | > loss: 0.012584859505295753 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.1367499162873115) | > loader_time: 0.0046 (0.004696239035038859)  --> TIME: 2025-02-27 07:09:31 -- STEP: 8516/36938 -- GLOBAL_STEP: 1596850 | > loss_text_ce: 0.025372888892889023 (0.030066357950031226) | > loss_mel_ce: 2.315709352493286 (nan) | > loss: 0.00929000973701477 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1367179730118809) | > loader_time: 0.0046 (0.004696386249035156)  --> TIME: 2025-02-27 07:09:41 -- STEP: 8566/36938 -- GLOBAL_STEP: 1596900 | > loss_text_ce: 0.03278426453471184 (0.030069035848356067) | > loss_mel_ce: 3.056720495223999 (nan) | > loss: 0.012259939685463905 (nan) | > current_lr: 5e-06 | > step_time: 0.1877 (0.13671243407863828) | > loader_time: 0.0049 (0.004695981316496321)  --> TIME: 2025-02-27 07:09:50 -- STEP: 8616/36938 -- GLOBAL_STEP: 1596950 | > loss_text_ce: 0.03284819796681404 (0.03007107228224452) | > loss_mel_ce: 3.031407356262207 (nan) | > loss: 0.012159744277596474 (nan) | > current_lr: 5e-06 | > step_time: 0.1816 (0.13668330845283863) | > loader_time: 0.0049 (0.004695720133564479)  --> TIME: 2025-02-27 07:09:59 -- STEP: 8666/36938 -- GLOBAL_STEP: 1597000 | > loss_text_ce: 0.024234680458903313 (0.030071647186713506) | > loss_mel_ce: 2.447561025619507 (nan) | > loss: 0.009808714501559734 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13668593267026616) | > loader_time: 0.0045 (0.004695395275541183)  --> TIME: 2025-02-27 07:10:09 -- STEP: 8716/36938 -- GLOBAL_STEP: 1597050 | > loss_text_ce: 0.033677440136671066 (0.030077369997774505) | > loss_mel_ce: 2.873415946960449 (nan) | > loss: 0.011536085046827793 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13669705084563982) | > loader_time: 0.0046 (0.0046952134045289634)  --> TIME: 2025-02-27 07:10:18 -- STEP: 8766/36938 -- GLOBAL_STEP: 1597100 | > loss_text_ce: 0.031174926087260246 (0.030070588371210513) | > loss_mel_ce: 2.784487724304199 (nan) | > loss: 0.011173265054821968 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13668641798196562) | > loader_time: 0.0048 (0.004694872758662125)  --> TIME: 2025-02-27 07:10:27 -- STEP: 8816/36938 -- GLOBAL_STEP: 1597150 | > loss_text_ce: 0.03249736130237579 (0.03006315924216251) | > loss_mel_ce: 3.077669620513916 (nan) | > loss: 0.012341933324933052 (nan) | > current_lr: 5e-06 | > step_time: 0.1873 (0.1366936813789795) | > loader_time: 0.0049 (0.0046945064719056415)  --> TIME: 2025-02-27 07:10:37 -- STEP: 8866/36938 -- GLOBAL_STEP: 1597200 | > loss_text_ce: 0.0240542721003294 (0.030066523019215162) | > loss_mel_ce: 2.9828598499298096 (nan) | > loss: 0.011932199820876122 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13675778403713554) | > loader_time: 0.0046 (0.004694890174590351)  --> TIME: 2025-02-27 07:10:47 -- STEP: 8916/36938 -- GLOBAL_STEP: 1597250 | > loss_text_ce: 0.023504972457885742 (0.030063855546489424) | > loss_mel_ce: 2.232469320297241 (nan) | > loss: 0.008952279575169086 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1367660427959904) | > loader_time: 0.0046 (0.004694495756244269)  --> TIME: 2025-02-27 07:10:56 -- STEP: 8966/36938 -- GLOBAL_STEP: 1597300 | > loss_text_ce: 0.03404763713479042 (0.03006104518572783) | > loss_mel_ce: 3.0043721199035645 (nan) | > loss: 0.012057221494615078 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13677619106131944) | > loader_time: 0.0045 (0.004694421696072568)  --> TIME: 2025-02-27 07:11:06 -- STEP: 9016/36938 -- GLOBAL_STEP: 1597350 | > loss_text_ce: 0.04079373925924301 (0.030056458933526004) | > loss_mel_ce: 2.085353374481201 (nan) | > loss: 0.00843709148466587 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13679810541334086) | > loader_time: 0.0046 (0.004694089121594293)  --> TIME: 2025-02-27 07:11:15 -- STEP: 9066/36938 -- GLOBAL_STEP: 1597400 | > loss_text_ce: 0.02946626767516136 (0.030048890416024505) | > loss_mel_ce: 2.442373275756836 (nan) | > loss: 0.009808886796236038 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13678587695344185) | > loader_time: 0.0046 (0.00469369170893482)  --> TIME: 2025-02-27 07:11:25 -- STEP: 9116/36938 -- GLOBAL_STEP: 1597450 | > loss_text_ce: 0.03749651089310646 (0.030047651738996612) | > loss_mel_ce: 1.4049303531646729 (nan) | > loss: 0.005723916459828615 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1368086123843111) | > loader_time: 0.0066 (0.004694270847030978)  --> TIME: 2025-02-27 07:11:34 -- STEP: 9166/36938 -- GLOBAL_STEP: 1597500 | > loss_text_ce: 0.03657175973057747 (0.03004813113833261) | > loss_mel_ce: 1.1467864513397217 (nan) | > loss: 0.004695866256952286 (nan) | > current_lr: 5e-06 | > step_time: 0.1576 (0.13682948619602026) | > loader_time: 0.0046 (0.004694468767497831)  --> TIME: 2025-02-27 07:11:44 -- STEP: 9216/36938 -- GLOBAL_STEP: 1597550 | > loss_text_ce: 0.03507668897509575 (0.03005232554813928) | > loss_mel_ce: 2.902336359024048 (nan) | > loss: 0.011656401678919792 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13685010942734) | > loader_time: 0.0047 (0.004694208062978252)  --> TIME: 2025-02-27 07:11:53 -- STEP: 9266/36938 -- GLOBAL_STEP: 1597600 | > loss_text_ce: 0.03513059392571449 (0.03005562866612796) | > loss_mel_ce: 1.120158314704895 (nan) | > loss: 0.0045844800770282745 (nan) | > current_lr: 5e-06 | > step_time: 0.1257 (0.13687668491485439) | > loader_time: 0.0045 (0.004693831065651562)  --> TIME: 2025-02-27 07:12:03 -- STEP: 9316/36938 -- GLOBAL_STEP: 1597650 | > loss_text_ce: 0.027484502643346786 (0.030059315780071344) | > loss_mel_ce: 3.2880184650421143 (nan) | > loss: 0.01315675862133503 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.136879627056622) | > loader_time: 0.0048 (0.004693371638457322)  --> TIME: 2025-02-27 07:12:12 -- STEP: 9366/36938 -- GLOBAL_STEP: 1597700 | > loss_text_ce: 0.03070968948304653 (0.030055076779788525) | > loss_mel_ce: 2.9873640537261963 (nan) | > loss: 0.011976484209299088 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13688008120343603) | > loader_time: 0.0048 (0.004693226072998504)  --> TIME: 2025-02-27 07:12:22 -- STEP: 9416/36938 -- GLOBAL_STEP: 1597750 | > loss_text_ce: 0.03123723901808262 (0.0300557638327421) | > loss_mel_ce: 2.77876353263855 (nan) | > loss: 0.01115079689770937 (nan) | > current_lr: 5e-06 | > step_time: 0.1837 (0.13691132392231103) | > loader_time: 0.0048 (0.004692929547625068)  --> TIME: 2025-02-27 07:12:31 -- STEP: 9466/36938 -- GLOBAL_STEP: 1597800 | > loss_text_ce: 0.03268694505095482 (0.030058764996030305) | > loss_mel_ce: 2.8212239742279053 (nan) | > loss: 0.011325043626129627 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.13690302970418017) | > loader_time: 0.0047 (0.00469242418238005)  --> TIME: 2025-02-27 07:12:40 -- STEP: 9516/36938 -- GLOBAL_STEP: 1597850 | > loss_text_ce: 0.02655072882771492 (0.030054082368594876) | > loss_mel_ce: 3.000096082687378 (nan) | > loss: 0.012010504491627216 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13690692051061382) | > loader_time: 0.0046 (0.004692027302237306)  --> TIME: 2025-02-27 07:12:49 -- STEP: 9566/36938 -- GLOBAL_STEP: 1597900 | > loss_text_ce: 0.030776040628552437 (0.03005674744082119) | > loss_mel_ce: 1.2348490953445435 (nan) | > loss: 0.005022322293370962 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1368993495314691) | > loader_time: 0.0045 (0.004691587615008133)  --> TIME: 2025-02-27 07:12:59 -- STEP: 9616/36938 -- GLOBAL_STEP: 1597950 | > loss_text_ce: 0.02808764949440956 (0.030057414472041133) | > loss_mel_ce: 2.5601227283477783 (nan) | > loss: 0.010270676575601101 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13687499137765818) | > loader_time: 0.0046 (0.004691132590695343)  --> TIME: 2025-02-27 07:13:08 -- STEP: 9666/36938 -- GLOBAL_STEP: 1598000 | > loss_text_ce: 0.029826542362570763 (0.030060602842253755) | > loss_mel_ce: 2.3958916664123535 (nan) | > loss: 0.009625867009162903 (nan) | > current_lr: 5e-06 | > step_time: 0.1807 (0.13687730020502994) | > loader_time: 0.005 (0.004691263249565199)  --> TIME: 2025-02-27 07:13:17 -- STEP: 9716/36938 -- GLOBAL_STEP: 1598050 | > loss_text_ce: 0.03177452087402344 (0.030061001411860386) | > loss_mel_ce: 2.8880345821380615 (nan) | > loss: 0.011586545035243034 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13687044583313202) | > loader_time: 0.0046 (0.004691364319543378)  --> TIME: 2025-02-27 07:13:26 -- STEP: 9766/36938 -- GLOBAL_STEP: 1598100 | > loss_text_ce: 0.024774448946118355 (0.030069523064307862) | > loss_mel_ce: 0.6608161926269531 (nan) | > loss: 0.0027205978985875845 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1368500883892234) | > loader_time: 0.0045 (0.004690980681771501)  --> TIME: 2025-02-27 07:13:36 -- STEP: 9816/36938 -- GLOBAL_STEP: 1598150 | > loss_text_ce: 0.032072197645902634 (0.030067016182356524) | > loss_mel_ce: 2.7907369136810303 (nan) | > loss: 0.011201624758541584 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.1368531883005802) | > loader_time: 0.0046 (0.004690898708337282)  --> TIME: 2025-02-27 07:13:45 -- STEP: 9866/36938 -- GLOBAL_STEP: 1598200 | > loss_text_ce: 0.039405398070812225 (0.030067145659426104) | > loss_mel_ce: 3.1185481548309326 (nan) | > loss: 0.012531561776995659 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13685627486772048) | > loader_time: 0.0046 (0.004690721048053121)  --> TIME: 2025-02-27 07:13:55 -- STEP: 9916/36938 -- GLOBAL_STEP: 1598250 | > loss_text_ce: 0.03319841995835304 (0.030063980984260726) | > loss_mel_ce: 2.673407554626465 (nan) | > loss: 0.010740499943494797 (nan) | > current_lr: 5e-06 | > step_time: 0.1887 (0.1368774917876836) | > loader_time: 0.0049 (0.004690884678245499)  --> TIME: 2025-02-27 07:14:04 -- STEP: 9966/36938 -- GLOBAL_STEP: 1598300 | > loss_text_ce: 0.032791681587696075 (0.03005979277852182) | > loss_mel_ce: 1.971215844154358 (nan) | > loss: 0.007952411659061909 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13685198629713488) | > loader_time: 0.0046 (0.004690951260844788)  --> TIME: 2025-02-27 07:14:13 -- STEP: 10016/36938 -- GLOBAL_STEP: 1598350 | > loss_text_ce: 0.024533633142709732 (0.030047003646658663) | > loss_mel_ce: 3.2052454948425293 (nan) | > loss: 0.012816584669053555 (nan) | > current_lr: 5e-06 | > step_time: 0.1881 (0.13683252662610756) | > loader_time: 0.0049 (0.0046908415544528095)  --> TIME: 2025-02-27 07:14:22 -- STEP: 10066/36938 -- GLOBAL_STEP: 1598400 | > loss_text_ce: 0.0330851748585701 (0.03004662504422645) | > loss_mel_ce: 0.062591552734375 (nan) | > loss: 0.00037966956733725965 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13684816313576323) | > loader_time: 0.0045 (0.004691009229678592)  --> TIME: 2025-02-27 07:14:32 -- STEP: 10116/36938 -- GLOBAL_STEP: 1598450 | > loss_text_ce: 0.03522089496254921 (0.03004905179217708) | > loss_mel_ce: 2.7278478145599365 (nan) | > loss: 0.010964558459818363 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.1368390070727433) | > loader_time: 0.0047 (0.004691296907789187)  --> TIME: 2025-02-27 07:14:41 -- STEP: 10166/36938 -- GLOBAL_STEP: 1598500 | > loss_text_ce: 0.026901327073574066 (0.030042071348818773) | > loss_mel_ce: 2.264589309692383 (nan) | > loss: 0.009093216620385647 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13684209193297828) | > loader_time: 0.0064 (0.004692039033828591)  --> TIME: 2025-02-27 07:14:50 -- STEP: 10216/36938 -- GLOBAL_STEP: 1598550 | > loss_text_ce: 0.03745223581790924 (0.030040329430794186) | > loss_mel_ce: 3.1835379600524902 (nan) | > loss: 0.01278170756995678 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13685834830219942) | > loader_time: 0.0047 (0.004692130846816652)  --> TIME: 2025-02-27 07:15:00 -- STEP: 10266/36938 -- GLOBAL_STEP: 1598600 | > loss_text_ce: 0.022804196923971176 (0.030040328016498478) | > loss_mel_ce: 1.040650486946106 (nan) | > loss: 0.004220058210194111 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13683985583897904) | > loader_time: 0.0047 (0.004692456166274618)  --> TIME: 2025-02-27 07:15:09 -- STEP: 10316/36938 -- GLOBAL_STEP: 1598650 | > loss_text_ce: 0.025372322648763657 (0.03004387231966529) | > loss_mel_ce: 2.3219261169433594 (nan) | > loss: 0.009314676746726036 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13683740759413618) | > loader_time: 0.0046 (0.004692321463396657)  --> TIME: 2025-02-27 07:15:18 -- STEP: 10366/36938 -- GLOBAL_STEP: 1598700 | > loss_text_ce: 0.02640320174396038 (0.03004425972297259) | > loss_mel_ce: 0.007293701171875 (nan) | > loss: 0.00013371788372751325 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13685489961269803) | > loader_time: 0.0046 (0.004692523216801009)  --> TIME: 2025-02-27 07:15:28 -- STEP: 10416/36938 -- GLOBAL_STEP: 1598750 | > loss_text_ce: 0.03568383678793907 (0.03004051386080067) | > loss_mel_ce: 2.1081464290618896 (nan) | > loss: 0.008507263846695423 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1368570230477787) | > loader_time: 0.0046 (0.004692616756610585)  --> TIME: 2025-02-27 07:15:37 -- STEP: 10466/36938 -- GLOBAL_STEP: 1598800 | > loss_text_ce: 0.02166021801531315 (0.030036913983565845) | > loss_mel_ce: 1.2583881616592407 (nan) | > loss: 0.005079557187855244 (nan) | > current_lr: 5e-06 | > step_time: 0.1074 (0.13684491933064186) | > loader_time: 0.0044 (0.004692831345595869)  --> TIME: 2025-02-27 07:15:46 -- STEP: 10516/36938 -- GLOBAL_STEP: 1598850 | > loss_text_ce: 0.03580167144536972 (0.030043150149955327) | > loss_mel_ce: 2.2089850902557373 (nan) | > loss: 0.008907884359359741 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.1368398979708653) | > loader_time: 0.0047 (0.004693130478291222)  --> TIME: 2025-02-27 07:15:56 -- STEP: 10566/36938 -- GLOBAL_STEP: 1598900 | > loss_text_ce: 0.020120471715927124 (0.030045277979288385) | > loss_mel_ce: 0.0185546875 (nan) | > loss: 0.00015347286534961313 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.1368556745842558) | > loader_time: 0.0048 (0.004694034763071165)  --> TIME: 2025-02-27 07:16:05 -- STEP: 10616/36938 -- GLOBAL_STEP: 1598950 | > loss_text_ce: 0.030853698030114174 (0.030049364237180055) | > loss_mel_ce: 3.149970769882202 (nan) | > loss: 0.012622320093214512 (nan) | > current_lr: 5e-06 | > step_time: 0.1953 (0.13687715291437977) | > loader_time: 0.0049 (0.004694116456938025)  --> TIME: 2025-02-27 07:16:14 -- STEP: 10666/36938 -- GLOBAL_STEP: 1599000 | > loss_text_ce: 0.034793950617313385 (0.030044190423640312) | > loss_mel_ce: 2.1049911975860596 (nan) | > loss: 0.008491210639476776 (nan) | > current_lr: 5e-06 | > step_time: 0.1517 (0.13684434631897768) | > loader_time: 0.0047 (0.00469389566220865)  --> TIME: 2025-02-27 07:16:24 -- STEP: 10716/36938 -- GLOBAL_STEP: 1599050 | > loss_text_ce: 0.023599473759531975 (0.03005005651433125) | > loss_mel_ce: 2.880629539489746 (nan) | > loss: 0.011524718254804611 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13683988439275924) | > loader_time: 0.0046 (0.004694184245970655)  --> TIME: 2025-02-27 07:16:33 -- STEP: 10766/36938 -- GLOBAL_STEP: 1599100 | > loss_text_ce: 0.03237035870552063 (0.030051681116453788) | > loss_mel_ce: 2.926471471786499 (nan) | > loss: 0.011741436086595058 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13684247965354773) | > loader_time: 0.0046 (0.004694139538877784)  --> TIME: 2025-02-27 07:16:43 -- STEP: 10816/36938 -- GLOBAL_STEP: 1599150 | > loss_text_ce: 0.03237948939204216 (0.030053279775497228) | > loss_mel_ce: 2.8713319301605225 (nan) | > loss: 0.011522664688527584 (nan) | > current_lr: 5e-06 | > step_time: 0.1667 (0.1368737084096711) | > loader_time: 0.005 (0.0046943385352396835)  --> TIME: 2025-02-27 07:16:52 -- STEP: 10866/36938 -- GLOBAL_STEP: 1599200 | > loss_text_ce: 0.03420817106962204 (0.0300569297740167) | > loss_mel_ce: 2.624051570892334 (nan) | > loss: 0.010548650287091732 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13688121933799335) | > loader_time: 0.0045 (0.004694124205477097)  --> TIME: 2025-02-27 07:17:01 -- STEP: 10916/36938 -- GLOBAL_STEP: 1599250 | > loss_text_ce: 0.03221474215388298 (0.03004939971712154) | > loss_mel_ce: 2.6407997608184814 (nan) | > loss: 0.010607200674712658 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13687287621028152) | > loader_time: 0.0047 (0.004693976445440905)  --> TIME: 2025-02-27 07:17:11 -- STEP: 10966/36938 -- GLOBAL_STEP: 1599300 | > loss_text_ce: 0.028034133836627007 (0.030040661258629458) | > loss_mel_ce: 2.592848539352417 (nan) | > loss: 0.010400328785181046 (nan) | > current_lr: 5e-06 | > step_time: 0.1076 (0.1368799125454673) | > loader_time: 0.0046 (0.004693850622154482)  --> TIME: 2025-02-27 07:17:20 -- STEP: 11016/36938 -- GLOBAL_STEP: 1599350 | > loss_text_ce: 0.024822447448968887 (0.03003896127022754) | > loss_mel_ce: 7.4921875 (nan) | > loss: 0.02982940524816513 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13687681262269838) | > loader_time: 0.0049 (0.004693682092467135)  --> TIME: 2025-02-27 07:17:30 -- STEP: 11066/36938 -- GLOBAL_STEP: 1599400 | > loss_text_ce: 0.022289574146270752 (0.030044008530986075) | > loss_mel_ce: 2.0044000148773193 (nan) | > loss: 0.008042419329285622 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13690676196625998) | > loader_time: 0.0045 (0.004693614085673309)  --> TIME: 2025-02-27 07:17:39 -- STEP: 11116/36938 -- GLOBAL_STEP: 1599450 | > loss_text_ce: 0.03732861205935478 (0.030041216782147762) | > loss_mel_ce: 0.04541015625 (nan) | > loss: 0.00032832848955877125 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.13689938336664487) | > loader_time: 0.0045 (0.0046937167108294575)  --> TIME: 2025-02-27 07:17:49 -- STEP: 11166/36938 -- GLOBAL_STEP: 1599500 | > loss_text_ce: 0.03272869437932968 (0.03003888207254389) | > loss_mel_ce: 2.72479248046875 (nan) | > loss: 0.010942544788122177 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.13691065624455093) | > loader_time: 0.0048 (0.004693723527756953)  --> TIME: 2025-02-27 07:17:58 -- STEP: 11216/36938 -- GLOBAL_STEP: 1599550 | > loss_text_ce: 0.0333104133605957 (0.03003914282938197) | > loss_mel_ce: 2.159776449203491 (nan) | > loss: 0.008702726103365421 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.13691786516154958) | > loader_time: 0.0047 (0.0046937892083264345)  --> TIME: 2025-02-27 07:18:07 -- STEP: 11266/36938 -- GLOBAL_STEP: 1599600 | > loss_text_ce: 0.04600558802485466 (0.030042035349762273) | > loss_mel_ce: 0.0311279296875 (nan) | > loss: 0.0003060854214709252 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.1369158603910996) | > loader_time: 0.0046 (0.004693966997086646)  --> TIME: 2025-02-27 07:18:16 -- STEP: 11316/36938 -- GLOBAL_STEP: 1599650 | > loss_text_ce: 0.035349972546100616 (0.030041106141374153) | > loss_mel_ce: 2.337935209274292 (nan) | > loss: 0.009417799301445484 (nan) | > current_lr: 5e-06 | > step_time: 0.1606 (0.13688975897051123) | > loader_time: 0.0048 (0.004693736558922923)  --> TIME: 2025-02-27 07:18:25 -- STEP: 11366/36938 -- GLOBAL_STEP: 1599700 | > loss_text_ce: 0.030183207243680954 (0.03004113436823561) | > loss_mel_ce: 2.8954017162323 (nan) | > loss: 0.011609464883804321 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.1368610981210072) | > loader_time: 0.0047 (0.004693793805787055)  --> TIME: 2025-02-27 07:18:35 -- STEP: 11416/36938 -- GLOBAL_STEP: 1599750 | > loss_text_ce: 0.030808143317699432 (0.030037683070527642) | > loss_mel_ce: 2.031947612762451 (nan) | > loss: 0.00818553939461708 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13686564545908797) | > loader_time: 0.0045 (0.004693994592266955)  --> TIME: 2025-02-27 07:18:44 -- STEP: 11466/36938 -- GLOBAL_STEP: 1599800 | > loss_text_ce: 0.02352336049079895 (0.03003044879163383) | > loss_mel_ce: 2.5884947776794434 (nan) | > loss: 0.010365151800215244 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13686087099062377) | > loader_time: 0.0047 (0.004693732697490017)  --> TIME: 2025-02-27 07:18:53 -- STEP: 11516/36938 -- GLOBAL_STEP: 1599850 | > loss_text_ce: 0.026662303134799004 (0.030031445818300095) | > loss_mel_ce: 0.865558385848999 (nan) | > loss: 0.0035405585076659918 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13683207782865975) | > loader_time: 0.0046 (0.004693541356199667)  --> TIME: 2025-02-27 07:19:03 -- STEP: 11566/36938 -- GLOBAL_STEP: 1599900 | > loss_text_ce: 0.034393537789583206 (0.03003856626891802) | > loss_mel_ce: 2.75276517868042 (nan) | > loss: 0.011060154065489769 (nan) | > current_lr: 5e-06 | > step_time: 0.1341 (0.13686022201088258) | > loader_time: 0.0046 (0.004693501448726087)  --> TIME: 2025-02-27 07:19:12 -- STEP: 11616/36938 -- GLOBAL_STEP: 1599950 | > loss_text_ce: 0.030871571972966194 (0.03003872974931914) | > loss_mel_ce: 2.9348978996276855 (nan) | > loss: 0.011768927797675133 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13688879020861083) | > loader_time: 0.0046 (0.004693465066185975)  --> TIME: 2025-02-27 07:19:22 -- STEP: 11666/36938 -- GLOBAL_STEP: 1600000 | > loss_text_ce: 0.029794389382004738 (0.030037677156060844) | > loss_mel_ce: 2.4422924518585205 (nan) | > loss: 0.009809869341552258 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1368789262884424) | > loader_time: 0.0045 (0.004693221682118997) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1600000.pth  --> TIME: 2025-02-27 07:19:37 -- STEP: 11716/36938 -- GLOBAL_STEP: 1600050 | > loss_text_ce: 0.024671299383044243 (0.03003850179014059) | > loss_mel_ce: 1.1931533813476562 (nan) | > loss: 0.004832637961953878 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13685880687384164) | > loader_time: 0.0046 (0.004693135481802155)  --> TIME: 2025-02-27 07:19:46 -- STEP: 11766/36938 -- GLOBAL_STEP: 1600100 | > loss_text_ce: 0.034565940499305725 (0.030035167285587087) | > loss_mel_ce: 3.3938567638397217 (nan) | > loss: 0.013604852370917797 (nan) | > current_lr: 5e-06 | > step_time: 0.1499 (0.13686339427232178) | > loader_time: 0.0047 (0.004693187095671894)  --> TIME: 2025-02-27 07:19:56 -- STEP: 11816/36938 -- GLOBAL_STEP: 1600150 | > loss_text_ce: 0.02979067526757717 (0.030037215968232018) | > loss_mel_ce: 3.0178418159484863 (nan) | > loss: 0.012093780562281609 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.1368668663921438) | > loader_time: 0.0048 (0.004693244507608149)  --> TIME: 2025-02-27 07:20:05 -- STEP: 11866/36938 -- GLOBAL_STEP: 1600200 | > loss_text_ce: 0.03186284378170967 (0.030035863706291716) | > loss_mel_ce: 3.122770071029663 (nan) | > loss: 0.012518384493887424 (nan) | > current_lr: 5e-06 | > step_time: 0.1963 (0.13686507818732277) | > loader_time: 0.0048 (0.004693153855693309)  --> TIME: 2025-02-27 07:20:14 -- STEP: 11916/36938 -- GLOBAL_STEP: 1600250 | > loss_text_ce: 0.03211662545800209 (0.03004236258838367) | > loss_mel_ce: 2.5606558322906494 (nan) | > loss: 0.010288780555129051 (nan) | > current_lr: 5e-06 | > step_time: 0.1859 (0.1368755328851327) | > loader_time: 0.0051 (0.004693500705115845)  --> TIME: 2025-02-27 07:20:23 -- STEP: 11966/36938 -- GLOBAL_STEP: 1600300 | > loss_text_ce: 0.02816701866686344 (0.030045159712132456) | > loss_mel_ce: 2.9863860607147217 (nan) | > loss: 0.011962512508034706 (nan) | > current_lr: 5e-06 | > step_time: 0.163 (0.13685340720356373) | > loader_time: 0.0047 (0.0046934183277893135)  --> TIME: 2025-02-27 07:20:33 -- STEP: 12016/36938 -- GLOBAL_STEP: 1600350 | > loss_text_ce: 0.035643208771944046 (0.03004524749699055) | > loss_mel_ce: 2.927981376647949 (nan) | > loss: 0.011760415509343147 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13686381355820942) | > loader_time: 0.0054 (0.0046932917341252454)  --> TIME: 2025-02-27 07:20:42 -- STEP: 12066/36938 -- GLOBAL_STEP: 1600400 | > loss_text_ce: 0.03377051278948784 (0.03004295338887197) | > loss_mel_ce: 3.0281100273132324 (nan) | > loss: 0.01215032022446394 (nan) | > current_lr: 5e-06 | > step_time: 0.1842 (0.13685863677609295) | > loader_time: 0.0048 (0.004693285438446528)  --> TIME: 2025-02-27 07:20:51 -- STEP: 12116/36938 -- GLOBAL_STEP: 1600450 | > loss_text_ce: 0.02966351993381977 (0.03004262699778187) | > loss_mel_ce: 0.19921875 (nan) | > loss: 0.0009082630276679993 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13684467401155082) | > loader_time: 0.0046 (0.004693119311025708)  --> TIME: 2025-02-27 07:21:00 -- STEP: 12166/36938 -- GLOBAL_STEP: 1600500 | > loss_text_ce: 0.023435356095433235 (0.030042296723244776) | > loss_mel_ce: 2.3286116123199463 (nan) | > loss: 0.009333520196378231 (nan) | > current_lr: 5e-06 | > step_time: 0.1523 (0.13682424675256247) | > loader_time: 0.0047 (0.004693057727954804)  --> TIME: 2025-02-27 07:21:10 -- STEP: 12216/36938 -- GLOBAL_STEP: 1600550 | > loss_text_ce: 0.0343816801905632 (0.030045297563813394) | > loss_mel_ce: 2.651454448699951 (nan) | > loss: 0.01065808068960905 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13683952054480958) | > loader_time: 0.0052 (0.0046933626105914806)  --> TIME: 2025-02-27 07:21:20 -- STEP: 12266/36938 -- GLOBAL_STEP: 1600600 | > loss_text_ce: 0.028489986434578896 (0.03004257105779557) | > loss_mel_ce: 1.9839458465576172 (nan) | > loss: 0.0079858573153615 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13686090244198096) | > loader_time: 0.0044 (0.004693445093418309)  --> TIME: 2025-02-27 07:21:29 -- STEP: 12316/36938 -- GLOBAL_STEP: 1600650 | > loss_text_ce: 0.038047317415475845 (0.030043784913795384) | > loss_mel_ce: 3.360053539276123 (nan) | > loss: 0.013484528288245201 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13685331942006076) | > loader_time: 0.0046 (0.004693152552961166)  --> TIME: 2025-02-27 07:21:38 -- STEP: 12366/36938 -- GLOBAL_STEP: 1600700 | > loss_text_ce: 0.024203328415751457 (0.030043807551376225) | > loss_mel_ce: 2.9940104484558105 (nan) | > loss: 0.011977039277553558 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13684640315557337) | > loader_time: 0.0047 (0.004692931593997205)  --> TIME: 2025-02-27 07:21:48 -- STEP: 12416/36938 -- GLOBAL_STEP: 1600750 | > loss_text_ce: 0.023574737831950188 (0.030041801258420556) | > loss_mel_ce: 0.022369384765625 (nan) | > loss: 0.00018231796275358647 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13686127832025033) | > loader_time: 0.0046 (0.004692662699320865)  --> TIME: 2025-02-27 07:21:57 -- STEP: 12466/36938 -- GLOBAL_STEP: 1600800 | > loss_text_ce: 0.03146198019385338 (0.030041538216798218) | > loss_mel_ce: 2.9850244522094727 (nan) | > loss: 0.011970184743404388 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.1368681802119262) | > loader_time: 0.0048 (0.0046924297373156605)  --> TIME: 2025-02-27 07:22:07 -- STEP: 12516/36938 -- GLOBAL_STEP: 1600850 | > loss_text_ce: 0.02750384993851185 (0.03004114944394053) | > loss_mel_ce: 1.925612211227417 (nan) | > loss: 0.0077504608780145645 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13688142936273184) | > loader_time: 0.0046 (0.004692155985681332)  --> TIME: 2025-02-27 07:22:16 -- STEP: 12566/36938 -- GLOBAL_STEP: 1600900 | > loss_text_ce: 0.04262970760464668 (0.03004372645567936) | > loss_mel_ce: 1.8644875288009644 (nan) | > loss: 0.0075679258443415165 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13687670680532915) | > loader_time: 0.0045 (0.004691975844925251)  --> TIME: 2025-02-27 07:22:25 -- STEP: 12616/36938 -- GLOBAL_STEP: 1600950 | > loss_text_ce: 0.032618239521980286 (0.03004562630499409) | > loss_mel_ce: 0.398162841796875 (nan) | > loss: 0.0017094487557187676 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.13687146972385386) | > loader_time: 0.0049 (0.004691807809477733)  --> TIME: 2025-02-27 07:22:35 -- STEP: 12666/36938 -- GLOBAL_STEP: 1601000 | > loss_text_ce: 0.031423185020685196 (0.03004913062996046) | > loss_mel_ce: 2.468371868133545 (nan) | > loss: 0.009919822216033936 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13687385572333843) | > loader_time: 0.0046 (0.004691806126408848)  --> TIME: 2025-02-27 07:22:44 -- STEP: 12716/36938 -- GLOBAL_STEP: 1601050 | > loss_text_ce: 0.04138524830341339 (0.030046683899764136) | > loss_mel_ce: 2.9277188777923584 (nan) | > loss: 0.01178216002881527 (nan) | > current_lr: 5e-06 | > step_time: 0.1569 (0.13686708216773752) | > loader_time: 0.0067 (0.004691789363232281)  --> TIME: 2025-02-27 07:22:53 -- STEP: 12766/36938 -- GLOBAL_STEP: 1601100 | > loss_text_ce: 0.03361647203564644 (0.030046162916574706) | > loss_mel_ce: 2.1973042488098145 (nan) | > loss: 0.008852860890328884 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13686642270653784) | > loader_time: 0.0067 (0.0046921640695233744)  --> TIME: 2025-02-27 07:23:02 -- STEP: 12816/36938 -- GLOBAL_STEP: 1601150 | > loss_text_ce: 0.029744824394583702 (0.030047623599331812) | > loss_mel_ce: 2.6568617820739746 (nan) | > loss: 0.010661138221621513 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13685991052310348) | > loader_time: 0.0056 (0.004692309711607481)  --> TIME: 2025-02-27 07:23:12 -- STEP: 12866/36938 -- GLOBAL_STEP: 1601200 | > loss_text_ce: 0.03238707408308983 (0.03005167349423653) | > loss_mel_ce: 3.2912628650665283 (nan) | > loss: 0.013189087621867657 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13686291211207047) | > loader_time: 0.0047 (0.004692545152833457)  --> TIME: 2025-02-27 07:23:21 -- STEP: 12916/36938 -- GLOBAL_STEP: 1601250 | > loss_text_ce: 0.02652907744050026 (0.030057542353404697) | > loss_mel_ce: 0.013883208855986595 (nan) | > loss: 0.00016036622400861233 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.136876731271086) | > loader_time: 0.0045 (0.004692651255280631)  --> TIME: 2025-02-27 07:23:31 -- STEP: 12966/36938 -- GLOBAL_STEP: 1601300 | > loss_text_ce: 0.021693071350455284 (0.030056661472112955) | > loss_mel_ce: 2.23684024810791 (nan) | > loss: 0.008962433785200119 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13687146114527154) | > loader_time: 0.0046 (0.004692798445626695)  --> TIME: 2025-02-27 07:23:40 -- STEP: 13016/36938 -- GLOBAL_STEP: 1601350 | > loss_text_ce: 0.03759869560599327 (0.030062635789826264) | > loss_mel_ce: 2.7817273139953613 (nan) | > loss: 0.011187802068889141 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.13688781877639708) | > loader_time: 0.0049 (0.0046929343573213)  --> TIME: 2025-02-27 07:23:50 -- STEP: 13066/36938 -- GLOBAL_STEP: 1601400 | > loss_text_ce: 0.029321929439902306 (0.030061213413819947) | > loss_mel_ce: 2.7829461097717285 (nan) | > loss: 0.01115979440510273 (nan) | > current_lr: 5e-06 | > step_time: 0.1898 (0.13690279565834904) | > loader_time: 0.0048 (0.004692812033822128)  --> TIME: 2025-02-27 07:23:59 -- STEP: 13116/36938 -- GLOBAL_STEP: 1601450 | > loss_text_ce: 0.03954966366291046 (0.030059058456640485) | > loss_mel_ce: 0.8147088885307312 (nan) | > loss: 0.0033899149857461452 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.1369063007318972) | > loader_time: 0.0046 (0.004692694478443579)  --> TIME: 2025-02-27 07:24:09 -- STEP: 13166/36938 -- GLOBAL_STEP: 1601500 | > loss_text_ce: 0.02089601382613182 (0.030054446388605868) | > loss_mel_ce: 2.5925071239471436 (nan) | > loss: 0.01037064753472805 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13690957940858087) | > loader_time: 0.0045 (0.00469275863087683)  --> TIME: 2025-02-27 07:24:18 -- STEP: 13216/36938 -- GLOBAL_STEP: 1601550 | > loss_text_ce: 0.03296542167663574 (0.030051704829887962) | > loss_mel_ce: 2.417595386505127 (nan) | > loss: 0.009724448435008526 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13691969136493357) | > loader_time: 0.0047 (0.0046926717348306285)  --> TIME: 2025-02-27 07:24:27 -- STEP: 13266/36938 -- GLOBAL_STEP: 1601600 | > loss_text_ce: 0.024416396394371986 (0.030055673266851567) | > loss_mel_ce: 2.447383403778076 (nan) | > loss: 0.009808730334043503 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1369234338418482) | > loader_time: 0.0047 (0.0046925962231878894)  --> TIME: 2025-02-27 07:24:37 -- STEP: 13316/36938 -- GLOBAL_STEP: 1601650 | > loss_text_ce: 0.02802499197423458 (0.03005497858604457) | > loss_mel_ce: 2.921670436859131 (nan) | > loss: 0.011705140583217144 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13691647432702064) | > loader_time: 0.0047 (0.004692312850361984)  --> TIME: 2025-02-27 07:24:46 -- STEP: 13366/36938 -- GLOBAL_STEP: 1601700 | > loss_text_ce: 0.03230715170502663 (0.0300516335374617) | > loss_mel_ce: 2.393979787826538 (nan) | > loss: 0.009628123603761196 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13691071778665242) | > loader_time: 0.0046 (0.0046922668946211814)  --> TIME: 2025-02-27 07:24:55 -- STEP: 13416/36938 -- GLOBAL_STEP: 1601750 | > loss_text_ce: 0.03190318122506142 (0.030049544226027777) | > loss_mel_ce: 2.7746529579162598 (nan) | > loss: 0.011137128807604313 (nan) | > current_lr: 5e-06 | > step_time: 0.1601 (0.13690760314073688) | > loader_time: 0.0047 (0.00469221625217169)  --> TIME: 2025-02-27 07:25:05 -- STEP: 13466/36938 -- GLOBAL_STEP: 1601800 | > loss_text_ce: 0.02397424355149269 (0.030049238265925662) | > loss_mel_ce: 2.1580936908721924 (nan) | > loss: 0.00865899957716465 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.1369138868252033) | > loader_time: 0.0048 (0.004692208797036118)  --> TIME: 2025-02-27 07:25:14 -- STEP: 13516/36938 -- GLOBAL_STEP: 1601850 | > loss_text_ce: 0.028919486328959465 (0.03004951749020952) | > loss_mel_ce: 0.8030387163162231 (nan) | > loss: 0.0033014214131981134 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.13690970862324642) | > loader_time: 0.0046 (0.004692468691874107)  --> TIME: 2025-02-27 07:25:23 -- STEP: 13566/36938 -- GLOBAL_STEP: 1601900 | > loss_text_ce: 0.02768210880458355 (0.030054568300499015) | > loss_mel_ce: 3.71923828125 (nan) | > loss: 0.014868732541799545 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13690777414055702) | > loader_time: 0.0046 (0.0046924192012981775)  --> TIME: 2025-02-27 07:25:33 -- STEP: 13616/36938 -- GLOBAL_STEP: 1601950 | > loss_text_ce: 0.029517030343413353 (0.030050270853289998) | > loss_mel_ce: 3.2313730716705322 (nan) | > loss: 0.012940040789544582 (nan) | > current_lr: 5e-06 | > step_time: 0.1839 (0.1369122663199841) | > loader_time: 0.0052 (0.004692287110974145)  --> TIME: 2025-02-27 07:25:42 -- STEP: 13666/36938 -- GLOBAL_STEP: 1602000 | > loss_text_ce: 0.037953682243824005 (0.03004897834718681) | > loss_mel_ce: 2.6697680950164795 (nan) | > loss: 0.010744928382337093 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.1368961248044726) | > loader_time: 0.0046 (0.004692194386106353)  --> TIME: 2025-02-27 07:25:51 -- STEP: 13716/36938 -- GLOBAL_STEP: 1602050 | > loss_text_ce: 0.030784867703914642 (0.030047702122850142) | > loss_mel_ce: 2.985555410385132 (nan) | > loss: 0.011969604529440403 (nan) | > current_lr: 5e-06 | > step_time: 0.1868 (0.13689672767890107) | > loader_time: 0.005 (0.004692269383244392)  --> TIME: 2025-02-27 07:26:00 -- STEP: 13766/36938 -- GLOBAL_STEP: 1602100 | > loss_text_ce: 0.027054578065872192 (0.030051610907495075) | > loss_mel_ce: 2.5894570350646973 (nan) | > loss: 0.010382982902228832 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13687697841420465) | > loader_time: 0.0046 (0.004692023323144405)  --> TIME: 2025-02-27 07:26:09 -- STEP: 13816/36938 -- GLOBAL_STEP: 1602150 | > loss_text_ce: 0.02228628657758236 (0.03005262347417935) | > loss_mel_ce: 0.02585856057703495 (nan) | > loss: 0.00019105098908767104 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.136863326519021) | > loader_time: 0.0046 (0.00469181735389963)  --> TIME: 2025-02-27 07:26:19 -- STEP: 13866/36938 -- GLOBAL_STEP: 1602200 | > loss_text_ce: 0.03401264175772667 (0.03005389586921481) | > loss_mel_ce: 1.5719326734542847 (nan) | > loss: 0.006372799165546894 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1368766897566894) | > loader_time: 0.0046 (0.004691787909828642)  --> TIME: 2025-02-27 07:26:28 -- STEP: 13916/36938 -- GLOBAL_STEP: 1602250 | > loss_text_ce: 0.037290316075086594 (0.030050255310100367) | > loss_mel_ce: 1.9971508979797363 (nan) | > loss: 0.008073179982602596 (nan) | > current_lr: 5e-06 | > step_time: 0.1118 (0.13687130773708328) | > loader_time: 0.0047 (0.004691606573278626)  --> TIME: 2025-02-27 07:26:37 -- STEP: 13966/36938 -- GLOBAL_STEP: 1602300 | > loss_text_ce: 0.03840407729148865 (0.030047787117664974) | > loss_mel_ce: 1.7561171054840088 (nan) | > loss: 0.007121116388589144 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13686279654417607) | > loader_time: 0.0045 (0.004691360315347159)  --> TIME: 2025-02-27 07:26:47 -- STEP: 14016/36938 -- GLOBAL_STEP: 1602350 | > loss_text_ce: 0.03613261133432388 (0.030049736972817976) | > loss_mel_ce: 2.8139820098876953 (nan) | > loss: 0.011309979483485222 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.1368779392921499) | > loader_time: 0.0049 (0.004691197838821342)  --> TIME: 2025-02-27 07:26:56 -- STEP: 14066/36938 -- GLOBAL_STEP: 1602400 | > loss_text_ce: 0.02230493538081646 (0.030048756454787225) | > loss_mel_ce: 2.874321222305298 (nan) | > loss: 0.011494548991322517 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.1368564583745712) | > loader_time: 0.005 (0.004690940902495145)  --> TIME: 2025-02-27 07:27:05 -- STEP: 14116/36938 -- GLOBAL_STEP: 1602450 | > loss_text_ce: 0.020326022058725357 (0.03004800244266171) | > loss_mel_ce: 1.3180588483810425 (nan) | > loss: 0.00531105138361454 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13684997834913484) | > loader_time: 0.0046 (0.00469098114095822)  --> TIME: 2025-02-27 07:27:15 -- STEP: 14166/36938 -- GLOBAL_STEP: 1602500 | > loss_text_ce: 0.033261220902204514 (0.03004429648552349) | > loss_mel_ce: 1.597474217414856 (nan) | > loss: 0.0064711724407970905 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13687375624200473) | > loader_time: 0.0048 (0.004691485915668036)  --> TIME: 2025-02-27 07:27:25 -- STEP: 14216/36938 -- GLOBAL_STEP: 1602550 | > loss_text_ce: 0.033041343092918396 (0.030044522528082224) | > loss_mel_ce: 3.0019285678863525 (nan) | > loss: 0.012043531984090805 (nan) | > current_lr: 5e-06 | > step_time: 0.2012 (0.1368889611929279) | > loader_time: 0.0049 (0.0046915483897364005)  --> TIME: 2025-02-27 07:27:34 -- STEP: 14266/36938 -- GLOBAL_STEP: 1602600 | > loss_text_ce: 0.02888398803770542 (0.03004746322115194) | > loss_mel_ce: 2.09806752204895 (nan) | > loss: 0.008440284058451653 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13689352803221705) | > loader_time: 0.0046 (0.00469187473191227)  --> TIME: 2025-02-27 07:27:43 -- STEP: 14316/36938 -- GLOBAL_STEP: 1602650 | > loss_text_ce: 0.025995608419179916 (0.03004620100204219) | > loss_mel_ce: 1.4536830186843872 (nan) | > loss: 0.005871741101145744 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.136874578757872) | > loader_time: 0.0045 (0.004691950899823854)  --> TIME: 2025-02-27 07:27:52 -- STEP: 14366/36938 -- GLOBAL_STEP: 1602700 | > loss_text_ce: 0.03371530771255493 (0.03004258628180021) | > loss_mel_ce: 2.0940165519714355 (nan) | > loss: 0.008443380706012249 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13686557999730103) | > loader_time: 0.0045 (0.004692109849614592)  --> TIME: 2025-02-27 07:28:02 -- STEP: 14416/36938 -- GLOBAL_STEP: 1602750 | > loss_text_ce: 0.030309900641441345 (0.030040130426447743) | > loss_mel_ce: 2.4661951065063477 (nan) | > loss: 0.009906766936182976 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.1368673932009886) | > loader_time: 0.0046 (0.004691899153553792)  --> TIME: 2025-02-27 07:28:11 -- STEP: 14466/36938 -- GLOBAL_STEP: 1602800 | > loss_text_ce: 0.034818291664123535 (0.03003787737933062) | > loss_mel_ce: 3.1420116424560547 (nan) | > loss: 0.012606468051671982 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13685626761474656) | > loader_time: 0.0046 (0.004691836432798833)  --> TIME: 2025-02-27 07:28:20 -- STEP: 14516/36938 -- GLOBAL_STEP: 1602850 | > loss_text_ce: 0.039803940802812576 (0.030036451753429592) | > loss_mel_ce: 2.701239824295044 (nan) | > loss: 0.010877158492803574 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1368667995775337) | > loader_time: 0.0046 (0.004691886225164519)  --> TIME: 2025-02-27 07:28:29 -- STEP: 14566/36938 -- GLOBAL_STEP: 1602900 | > loss_text_ce: 0.020075513049960136 (0.030036681778301856) | > loss_mel_ce: 2.245781183242798 (nan) | > loss: 0.008991495706140995 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13685595296854342) | > loader_time: 0.0049 (0.0046917106953685735)  --> TIME: 2025-02-27 07:28:39 -- STEP: 14616/36938 -- GLOBAL_STEP: 1602950 | > loss_text_ce: 0.03672916069626808 (0.0300407694115777) | > loss_mel_ce: 2.5058820247650146 (nan) | > loss: 0.010089727118611336 (nan) | > current_lr: 5e-06 | > step_time: 0.1258 (0.1368550432791926) | > loader_time: 0.005 (0.004691644173602793)  --> TIME: 2025-02-27 07:28:48 -- STEP: 14666/36938 -- GLOBAL_STEP: 1603000 | > loss_text_ce: 0.03901005536317825 (0.030041022600051582) | > loss_mel_ce: 2.3469178676605225 (nan) | > loss: 0.009467968717217445 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13684866497628242) | > loader_time: 0.0046 (0.004691477981316617)  --> TIME: 2025-02-27 07:28:58 -- STEP: 14716/36938 -- GLOBAL_STEP: 1603050 | > loss_text_ce: 0.027431992813944817 (0.03003592396095004) | > loss_mel_ce: 2.9552903175354004 (nan) | > loss: 0.011836200021207333 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13686377897182309) | > loader_time: 0.0047 (0.004691648204613692)  --> TIME: 2025-02-27 07:29:07 -- STEP: 14766/36938 -- GLOBAL_STEP: 1603100 | > loss_text_ce: 0.030117811635136604 (0.03003600663818963) | > loss_mel_ce: 2.710801839828491 (nan) | > loss: 0.010876665823161602 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13685032371323191) | > loader_time: 0.0046 (0.004691385050606576)  --> TIME: 2025-02-27 07:29:16 -- STEP: 14816/36938 -- GLOBAL_STEP: 1603150 | > loss_text_ce: 0.025451499968767166 (0.030033477550042095) | > loss_mel_ce: 2.6470799446105957 (nan) | > loss: 0.0106052840128541 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.1368721696522021) | > loader_time: 0.0046 (0.004691271960220353)  --> TIME: 2025-02-27 07:29:26 -- STEP: 14866/36938 -- GLOBAL_STEP: 1603200 | > loss_text_ce: 0.03404352813959122 (0.030030186217642602) | > loss_mel_ce: 0.00213623046875 (nan) | > loss: 0.00014357047621160746 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.1368798817148708) | > loader_time: 0.0046 (0.004691253500062981)  --> TIME: 2025-02-27 07:29:35 -- STEP: 14916/36938 -- GLOBAL_STEP: 1603250 | > loss_text_ce: 0.031988468021154404 (0.030027402639184242) | > loss_mel_ce: 3.2501220703125 (nan) | > loss: 0.01302424818277359 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13689173469916924) | > loader_time: 0.0047 (0.004691484691183035)  --> TIME: 2025-02-27 07:29:45 -- STEP: 14966/36938 -- GLOBAL_STEP: 1603300 | > loss_text_ce: 0.024562686681747437 (0.030028847825804186) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.1368790973432407) | > loader_time: 0.0045 (0.004691381386284432)  --> TIME: 2025-02-27 07:29:54 -- STEP: 15016/36938 -- GLOBAL_STEP: 1603350 | > loss_text_ce: 0.03742097690701485 (0.03002909644924652) | > loss_mel_ce: 2.1492650508880615 (nan) | > loss: 0.008677326142787933 (nan) | > current_lr: 5e-06 | > step_time: 0.1786 (0.13688841090598716) | > loader_time: 0.0049 (0.004691463521521436)  --> TIME: 2025-02-27 07:30:03 -- STEP: 15066/36938 -- GLOBAL_STEP: 1603400 | > loss_text_ce: 0.031934626400470734 (0.030031204548044126) | > loss_mel_ce: 1.9736359119415283 (nan) | > loss: 0.007958614267408848 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13688645376463532) | > loader_time: 0.0048 (0.00469158707933564)  --> TIME: 2025-02-27 07:30:13 -- STEP: 15116/36938 -- GLOBAL_STEP: 1603450 | > loss_text_ce: 0.034552086144685745 (0.03002995910671581) | > loss_mel_ce: 2.747305393218994 (nan) | > loss: 0.011039117351174355 (nan) | > current_lr: 5e-06 | > step_time: 0.1821 (0.13688782662681703) | > loader_time: 0.0049 (0.00469153545369037)  --> TIME: 2025-02-27 07:30:22 -- STEP: 15166/36938 -- GLOBAL_STEP: 1603500 | > loss_text_ce: 0.030227214097976685 (0.03003050927055569) | > loss_mel_ce: 3.265138626098633 (nan) | > loss: 0.013076849281787872 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.1368755958458537) | > loader_time: 0.0047 (0.004691587437052792)  --> TIME: 2025-02-27 07:30:31 -- STEP: 15216/36938 -- GLOBAL_STEP: 1603550 | > loss_text_ce: 0.02631272003054619 (0.030029988239399823) | > loss_mel_ce: 2.563542366027832 (nan) | > loss: 0.010277203284204006 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13689287773066217) | > loader_time: 0.0046 (0.004691506911276744)  --> TIME: 2025-02-27 07:30:41 -- STEP: 15266/36938 -- GLOBAL_STEP: 1603600 | > loss_text_ce: 0.024770740419626236 (0.03002924059068553) | > loss_mel_ce: 2.922842025756836 (nan) | > loss: 0.011696876958012581 (nan) | > current_lr: 5e-06 | > step_time: 0.1468 (0.13689605741189828) | > loader_time: 0.0046 (0.004691272907639375)  --> TIME: 2025-02-27 07:30:50 -- STEP: 15316/36938 -- GLOBAL_STEP: 1603650 | > loss_text_ce: 0.031908635050058365 (0.03002862185968262) | > loss_mel_ce: 2.1873996257781982 (nan) | > loss: 0.008806779980659485 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1368861760847817) | > loader_time: 0.0045 (0.004690938657181268)  --> TIME: 2025-02-27 07:30:59 -- STEP: 15366/36938 -- GLOBAL_STEP: 1603700 | > loss_text_ce: 0.03057440184056759 (0.03002992481551846) | > loss_mel_ce: 2.8187105655670166 (nan) | > loss: 0.011306686326861382 (nan) | > current_lr: 5e-06 | > step_time: 0.1716 (0.13689672563685254) | > loader_time: 0.0048 (0.004690713859482612)  --> TIME: 2025-02-27 07:31:09 -- STEP: 15416/36938 -- GLOBAL_STEP: 1603750 | > loss_text_ce: 0.03617623448371887 (0.030030122050301658) | > loss_mel_ce: 2.63775634765625 (nan) | > loss: 0.010610844008624554 (nan) | > current_lr: 5e-06 | > step_time: 0.1623 (0.13689330065565972) | > loader_time: 0.0048 (0.004690509341699668)  --> TIME: 2025-02-27 07:31:18 -- STEP: 15466/36938 -- GLOBAL_STEP: 1603800 | > loss_text_ce: 0.03618152067065239 (0.030029883653927027) | > loss_mel_ce: 2.323538064956665 (nan) | > loss: 0.00936396699398756 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13688894203343474) | > loader_time: 0.0046 (0.004690374006017042)  --> TIME: 2025-02-27 07:31:27 -- STEP: 15516/36938 -- GLOBAL_STEP: 1603850 | > loss_text_ce: 0.026548193767666817 (0.030029636996513228) | > loss_mel_ce: 2.790555238723755 (nan) | > loss: 0.011178982444107533 (nan) | > current_lr: 5e-06 | > step_time: 0.1569 (0.13689043619301505) | > loader_time: 0.0048 (0.004690210731350943)  --> TIME: 2025-02-27 07:31:36 -- STEP: 15566/36938 -- GLOBAL_STEP: 1603900 | > loss_text_ce: 0.03809444233775139 (0.030029651582718267) | > loss_mel_ce: 1.4593075513839722 (nan) | > loss: 0.005942071788012981 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13687981553514603) | > loader_time: 0.0045 (0.004689999063539095)  --> TIME: 2025-02-27 07:31:46 -- STEP: 15616/36938 -- GLOBAL_STEP: 1603950 | > loss_text_ce: 0.02437724359333515 (0.030029559382783788) | > loss_mel_ce: 2.7414958477020264 (nan) | > loss: 0.010975687764585018 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13688306718087662) | > loader_time: 0.0047 (0.0046899963139754594)  --> TIME: 2025-02-27 07:31:55 -- STEP: 15666/36938 -- GLOBAL_STEP: 1604000 | > loss_text_ce: 0.028301293030381203 (0.03002590503914337) | > loss_mel_ce: 1.986180305480957 (nan) | > loss: 0.007993974722921848 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13687829182672498) | > loader_time: 0.0046 (0.00468987724903698)  --> TIME: 2025-02-27 07:32:04 -- STEP: 15716/36938 -- GLOBAL_STEP: 1604050 | > loss_text_ce: 0.030057549476623535 (0.03002888089268461) | > loss_mel_ce: 0.0946451798081398 (nan) | > loss: 0.0004948521382175386 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.13688561302858943) | > loader_time: 0.0047 (0.004689745819274304)  --> TIME: 2025-02-27 07:32:14 -- STEP: 15766/36938 -- GLOBAL_STEP: 1604100 | > loss_text_ce: 0.0262417234480381 (0.030027209729186367) | > loss_mel_ce: 0.0018581947078928351 (nan) | > loss: 0.00011150762293254957 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13689514738675998) | > loader_time: 0.0045 (0.0046896906381752234)  --> TIME: 2025-02-27 07:32:23 -- STEP: 15816/36938 -- GLOBAL_STEP: 1604150 | > loss_text_ce: 0.025059223175048828 (0.030027868968767215) | > loss_mel_ce: 1.6556650400161743 (nan) | > loss: 0.006669540889561176 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.1369030739269785) | > loader_time: 0.0045 (0.004689614174036747)  --> TIME: 2025-02-27 07:32:33 -- STEP: 15866/36938 -- GLOBAL_STEP: 1604200 | > loss_text_ce: 0.02732914313673973 (0.030026285153832797) | > loss_mel_ce: 2.623466730117798 (nan) | > loss: 0.01051903236657381 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.1368962035839733) | > loader_time: 0.0047 (0.004689454521430187)  --> TIME: 2025-02-27 07:32:42 -- STEP: 15916/36938 -- GLOBAL_STEP: 1604250 | > loss_text_ce: 0.030153241008520126 (0.030026273198459005) | > loss_mel_ce: 2.6466259956359863 (nan) | > loss: 0.01062214095145464 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13688255139828162) | > loader_time: 0.0046 (0.004689253733726722)  --> TIME: 2025-02-27 07:32:51 -- STEP: 15966/36938 -- GLOBAL_STEP: 1604300 | > loss_text_ce: 0.03372454643249512 (0.03002451856225238) | > loss_mel_ce: 2.3602700233459473 (nan) | > loss: 0.009499979205429554 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13687825977959145) | > loader_time: 0.0051 (0.0046896568504055155)  --> TIME: 2025-02-27 07:33:00 -- STEP: 16016/36938 -- GLOBAL_STEP: 1604350 | > loss_text_ce: 0.032179057598114014 (0.03002346409609184) | > loss_mel_ce: 2.5440826416015625 (nan) | > loss: 0.010223261080682278 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13688159935719796) | > loader_time: 0.0046 (0.004690084662232581)  --> TIME: 2025-02-27 07:33:10 -- STEP: 16066/36938 -- GLOBAL_STEP: 1604400 | > loss_text_ce: 0.026825925335288048 (0.030025270621371246) | > loss_mel_ce: 2.4493534564971924 (nan) | > loss: 0.009826108813285828 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13689372064048455) | > loader_time: 0.0047 (0.004690170703604414)  --> TIME: 2025-02-27 07:33:19 -- STEP: 16116/36938 -- GLOBAL_STEP: 1604450 | > loss_text_ce: 0.02266927994787693 (0.03002972513457844) | > loss_mel_ce: 2.1063992977142334 (nan) | > loss: 0.008448685519397259 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13689325853801373) | > loader_time: 0.0046 (0.004690090400996863)  --> TIME: 2025-02-27 07:33:29 -- STEP: 16166/36938 -- GLOBAL_STEP: 1604500 | > loss_text_ce: 0.0261954665184021 (0.03003047778240375) | > loss_mel_ce: 2.720889091491699 (nan) | > loss: 0.010901129804551601 (nan) | > current_lr: 5e-06 | > step_time: 0.1123 (0.13691600040384894) | > loader_time: 0.0045 (0.0046903132861400385)  --> TIME: 2025-02-27 07:33:38 -- STEP: 16216/36938 -- GLOBAL_STEP: 1604550 | > loss_text_ce: 0.035806234925985336 (0.030032535745614344) | > loss_mel_ce: 2.9900593757629395 (nan) | > loss: 0.012007404118776321 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13691756313048342) | > loader_time: 0.0046 (0.004690319931912882)  --> TIME: 2025-02-27 07:33:48 -- STEP: 16266/36938 -- GLOBAL_STEP: 1604600 | > loss_text_ce: 0.0328446626663208 (0.030034020460020308) | > loss_mel_ce: 2.296826124191284 (nan) | > loss: 0.009244726039469242 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13692275397264325) | > loader_time: 0.0046 (0.004690269988268814)  --> TIME: 2025-02-27 07:33:57 -- STEP: 16316/36938 -- GLOBAL_STEP: 1604650 | > loss_text_ce: 0.030047105625271797 (0.03003526364428478) | > loss_mel_ce: 1.2148107290267944 (nan) | > loss: 0.004939912352710962 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13691876335861886) | > loader_time: 0.0049 (0.0046902772812727365)  --> TIME: 2025-02-27 07:34:06 -- STEP: 16366/36938 -- GLOBAL_STEP: 1604700 | > loss_text_ce: 0.028778787702322006 (0.030034525755435033) | > loss_mel_ce: 1.426173210144043 (nan) | > loss: 0.005773619282990694 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13690736215708746) | > loader_time: 0.0047 (0.004690357281907334)  --> TIME: 2025-02-27 07:34:16 -- STEP: 16416/36938 -- GLOBAL_STEP: 1604750 | > loss_text_ce: 0.03437906876206398 (0.030033324820229596) | > loss_mel_ce: 0.7373860478401184 (nan) | > loss: 0.003062560223042965 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13690705006292297) | > loader_time: 0.0048 (0.004690447252163855)  --> TIME: 2025-02-27 07:34:25 -- STEP: 16466/36938 -- GLOBAL_STEP: 1604800 | > loss_text_ce: 0.02544211782515049 (0.03003513376797169) | > loss_mel_ce: 2.3710992336273193 (nan) | > loss: 0.009510084986686707 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13690449323310214) | > loader_time: 0.0047 (0.004690330590046879)  --> TIME: 2025-02-27 07:34:35 -- STEP: 16516/36938 -- GLOBAL_STEP: 1604850 | > loss_text_ce: 0.03146198019385338 (0.03003442294262512) | > loss_mel_ce: 2.7102794647216797 (nan) | > loss: 0.010879927314817905 (nan) | > current_lr: 5e-06 | > step_time: 0.1652 (0.13691666531025876) | > loader_time: 0.0048 (0.004690269893813441)  --> TIME: 2025-02-27 07:34:44 -- STEP: 16566/36938 -- GLOBAL_STEP: 1604900 | > loss_text_ce: 0.03689584881067276 (0.030033623463211454) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13690884903722825) | > loader_time: 0.0046 (0.004690263246289611)  --> TIME: 2025-02-27 07:34:54 -- STEP: 16616/36938 -- GLOBAL_STEP: 1604950 | > loss_text_ce: 0.029997948557138443 (0.03003075472249609) | > loss_mel_ce: 2.9365007877349854 (nan) | > loss: 0.011771821416914463 (nan) | > current_lr: 5e-06 | > step_time: 0.1809 (0.1369284216922589) | > loader_time: 0.0052 (0.0046903446252149535)  --> TIME: 2025-02-27 07:35:03 -- STEP: 16666/36938 -- GLOBAL_STEP: 1605000 | > loss_text_ce: 0.02401483803987503 (0.030026901574360786) | > loss_mel_ce: 3.1443355083465576 (nan) | > loss: 0.0125728202983737 (nan) | > current_lr: 5e-06 | > step_time: 0.1709 (0.13693807602329344) | > loader_time: 0.0048 (0.0046903807247412966)  --> TIME: 2025-02-27 07:35:12 -- STEP: 16716/36938 -- GLOBAL_STEP: 1605050 | > loss_text_ce: 0.03898525610566139 (0.030024276846307985) | > loss_mel_ce: 2.274358034133911 (nan) | > loss: 0.009179933927953243 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.13693921166363235) | > loader_time: 0.0046 (0.004690332557060219)  --> TIME: 2025-02-27 07:35:22 -- STEP: 16766/36938 -- GLOBAL_STEP: 1605100 | > loss_text_ce: 0.02880985103547573 (0.030026619992732515) | > loss_mel_ce: 2.4176230430603027 (nan) | > loss: 0.009708067402243614 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13696015614904758) | > loader_time: 0.0046 (0.004690614930345927)  --> TIME: 2025-02-27 07:35:32 -- STEP: 16816/36938 -- GLOBAL_STEP: 1605150 | > loss_text_ce: 0.02648765780031681 (0.03002554588028709) | > loss_mel_ce: 2.1694023609161377 (nan) | > loss: 0.008713849820196629 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.13697329617249934) | > loader_time: 0.0047 (0.004690736858307103)  --> TIME: 2025-02-27 07:35:41 -- STEP: 16866/36938 -- GLOBAL_STEP: 1605200 | > loss_text_ce: 0.03563757240772247 (0.03002664520669511) | > loss_mel_ce: 1.5503265857696533 (nan) | > loss: 0.006293509155511856 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13697329594347452) | > loader_time: 0.0049 (0.0046906467294347874)  --> TIME: 2025-02-27 07:35:50 -- STEP: 16916/36938 -- GLOBAL_STEP: 1605250 | > loss_text_ce: 0.03474591299891472 (0.030024021929343576) | > loss_mel_ce: 2.607919454574585 (nan) | > loss: 0.010486767627298832 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.1369522454384012) | > loader_time: 0.0046 (0.004690405098533293)  --> TIME: 2025-02-27 07:35:59 -- STEP: 16966/36938 -- GLOBAL_STEP: 1605300 | > loss_text_ce: 0.028527405112981796 (0.03002137557709803) | > loss_mel_ce: 2.6738555431365967 (nan) | > loss: 0.01072374265640974 (nan) | > current_lr: 5e-06 | > step_time: 0.1571 (0.1369482003811977) | > loader_time: 0.0047 (0.00469018924521566)  --> TIME: 2025-02-27 07:36:09 -- STEP: 17016/36938 -- GLOBAL_STEP: 1605350 | > loss_text_ce: 0.029932739213109016 (0.03002628974211949) | > loss_mel_ce: 2.976668119430542 (nan) | > loss: 0.01193095650523901 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.136948632252402) | > loader_time: 0.0049 (0.004690102837037506)  --> TIME: 2025-02-27 07:36:18 -- STEP: 17066/36938 -- GLOBAL_STEP: 1605400 | > loss_text_ce: 0.028753124177455902 (0.030023444293704925) | > loss_mel_ce: 2.167814254760742 (nan) | > loss: 0.008716537617146969 (nan) | > current_lr: 5e-06 | > step_time: 0.1442 (0.13695739362802353) | > loader_time: 0.0047 (0.0046902066529963485)  --> TIME: 2025-02-27 07:36:27 -- STEP: 17116/36938 -- GLOBAL_STEP: 1605450 | > loss_text_ce: 0.025819292291998863 (0.030025212759433706) | > loss_mel_ce: 3.3442189693450928 (nan) | > loss: 0.013373168185353279 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13695079832050344) | > loader_time: 0.0047 (0.004690130588801596)  --> TIME: 2025-02-27 07:36:37 -- STEP: 17166/36938 -- GLOBAL_STEP: 1605500 | > loss_text_ce: 0.022961728274822235 (0.030024405404231777) | > loss_mel_ce: 2.60314679145813 (nan) | > loss: 0.010421065613627434 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13697756845106968) | > loader_time: 0.0048 (0.00469017091309093)  --> TIME: 2025-02-27 07:36:47 -- STEP: 17216/36938 -- GLOBAL_STEP: 1605550 | > loss_text_ce: 0.03425292298197746 (0.03002329003405551) | > loss_mel_ce: 2.7196311950683594 (nan) | > loss: 0.010928112082183361 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13698266533837064) | > loader_time: 0.0046 (0.004690121443852152)  --> TIME: 2025-02-27 07:36:56 -- STEP: 17266/36938 -- GLOBAL_STEP: 1605600 | > loss_text_ce: 0.03135346993803978 (0.030026053213753352) | > loss_mel_ce: 2.176504135131836 (nan) | > loss: 0.008761339820921421 (nan) | > current_lr: 5e-06 | > step_time: 0.152 (0.13698997406974514) | > loader_time: 0.0047 (0.004690113645377041)  --> TIME: 2025-02-27 07:37:05 -- STEP: 17316/36938 -- GLOBAL_STEP: 1605650 | > loss_text_ce: 0.02130180597305298 (0.030028480803540224) | > loss_mel_ce: 2.5816972255706787 (nan) | > loss: 0.010329362004995346 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13697890941803234) | > loader_time: 0.0048 (0.004690011424978639)  --> TIME: 2025-02-27 07:37:15 -- STEP: 17366/36938 -- GLOBAL_STEP: 1605700 | > loss_text_ce: 0.022261178120970726 (0.030027692735522953) | > loss_mel_ce: 2.4539949893951416 (nan) | > loss: 0.009826413355767727 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13697375039495346) | > loader_time: 0.0046 (0.004689890119483842)  --> TIME: 2025-02-27 07:37:24 -- STEP: 17416/36938 -- GLOBAL_STEP: 1605750 | > loss_text_ce: 0.027513831853866577 (0.030027178509933496) | > loss_mel_ce: 2.759671449661255 (nan) | > loss: 0.0110602593049407 (nan) | > current_lr: 5e-06 | > step_time: 0.1773 (0.13698158140405975) | > loader_time: 0.0049 (0.004689926626378905)  --> TIME: 2025-02-27 07:37:34 -- STEP: 17466/36938 -- GLOBAL_STEP: 1605800 | > loss_text_ce: 0.029127737507224083 (0.030029000387557243) | > loss_mel_ce: 2.12298583984375 (nan) | > loss: 0.008540133945643902 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13698735974791407) | > loader_time: 0.0045 (0.00468987287230661)  --> TIME: 2025-02-27 07:37:43 -- STEP: 17516/36938 -- GLOBAL_STEP: 1605850 | > loss_text_ce: 0.022643500939011574 (0.030027952489489286) | > loss_mel_ce: 1.8812319040298462 (nan) | > loss: 0.007555061485618353 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13698298754444957) | > loader_time: 0.0047 (0.004689867106113387)  --> TIME: 2025-02-27 07:37:52 -- STEP: 17566/36938 -- GLOBAL_STEP: 1605900 | > loss_text_ce: 0.038995664566755295 (0.030029208119417863) | > loss_mel_ce: 2.9485676288604736 (nan) | > loss: 0.0118554113432765 (nan) | > current_lr: 5e-06 | > step_time: 0.1494 (0.13698145372238144) | > loader_time: 0.0046 (0.004689823192655923)  --> TIME: 2025-02-27 07:38:01 -- STEP: 17616/36938 -- GLOBAL_STEP: 1605950 | > loss_text_ce: 0.027925528585910797 (0.03002742971418398) | > loss_mel_ce: 3.0003750324249268 (nan) | > loss: 0.012017066590487957 (nan) | > current_lr: 5e-06 | > step_time: 0.1439 (0.13697478328099857) | > loader_time: 0.0054 (0.004689982473904374)  --> TIME: 2025-02-27 07:38:11 -- STEP: 17666/36938 -- GLOBAL_STEP: 1606000 | > loss_text_ce: 0.02455364726483822 (0.030028463401500102) | > loss_mel_ce: 1.731227993965149 (nan) | > loss: 0.006967388093471527 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.1369679857323349) | > loader_time: 0.0045 (0.004689978011990799)  --> TIME: 2025-02-27 07:38:20 -- STEP: 17716/36938 -- GLOBAL_STEP: 1606050 | > loss_text_ce: 0.03458798676729202 (0.03002837693226624) | > loss_mel_ce: 2.2679836750030518 (nan) | > loss: 0.009137189947068691 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13695325983865347) | > loader_time: 0.0045 (0.0046900200047108485)  --> TIME: 2025-02-27 07:38:29 -- STEP: 17766/36938 -- GLOBAL_STEP: 1606100 | > loss_text_ce: 0.03624727949500084 (0.030027321606394715) | > loss_mel_ce: 0.012151082046329975 (nan) | > loss: 0.0001920569920912385 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.1369474725547203) | > loader_time: 0.0049 (0.00468995064399698)  --> TIME: 2025-02-27 07:38:38 -- STEP: 17816/36938 -- GLOBAL_STEP: 1606150 | > loss_text_ce: 0.030368074774742126 (0.030026883852919455) | > loss_mel_ce: 2.7583985328674316 (nan) | > loss: 0.01106653455644846 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13695577137610554) | > loader_time: 0.0045 (0.0046898311545188155)  --> TIME: 2025-02-27 07:38:48 -- STEP: 17866/36938 -- GLOBAL_STEP: 1606200 | > loss_text_ce: 0.029589612036943436 (0.030025126446289963) | > loss_mel_ce: 2.926081657409668 (nan) | > loss: 0.011728854849934578 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13695379069787542) | > loader_time: 0.0046 (0.004689646357053005)  --> TIME: 2025-02-27 07:38:57 -- STEP: 17916/36938 -- GLOBAL_STEP: 1606250 | > loss_text_ce: 0.03833019733428955 (0.030027656231244378) | > loss_mel_ce: 2.5757296085357666 (nan) | > loss: 0.010373254306614399 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13696092345880176) | > loader_time: 0.0045 (0.00468954781278697)  --> TIME: 2025-02-27 07:39:07 -- STEP: 17966/36938 -- GLOBAL_STEP: 1606300 | > loss_text_ce: 0.028809040784835815 (0.030028329238205993) | > loss_mel_ce: 2.737900733947754 (nan) | > loss: 0.010979007929563522 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.1369805129376184) | > loader_time: 0.0048 (0.004689754694484581)  --> TIME: 2025-02-27 07:39:17 -- STEP: 18016/36938 -- GLOBAL_STEP: 1606350 | > loss_text_ce: 0.026265770196914673 (0.030028270818164907) | > loss_mel_ce: 1.7852531671524048 (nan) | > loss: 0.00718856742605567 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13699007194429746) | > loader_time: 0.0047 (0.004689885763249629)  --> TIME: 2025-02-27 07:39:26 -- STEP: 18066/36938 -- GLOBAL_STEP: 1606400 | > loss_text_ce: 0.03167923167347908 (0.03002592370072421) | > loss_mel_ce: 3.266220808029175 (nan) | > loss: 0.01308690570294857 (nan) | > current_lr: 5e-06 | > step_time: 0.1817 (0.1369883886909722) | > loader_time: 0.0048 (0.004689800809232875)  --> TIME: 2025-02-27 07:39:35 -- STEP: 18116/36938 -- GLOBAL_STEP: 1606450 | > loss_text_ce: 0.030666310340166092 (0.030027335406428917) | > loss_mel_ce: 2.9836552143096924 (nan) | > loss: 0.011961594223976135 (nan) | > current_lr: 5e-06 | > step_time: 0.1928 (0.13698562605277367) | > loader_time: 0.0048 (0.004689672512312161)  --> TIME: 2025-02-27 07:39:45 -- STEP: 18166/36938 -- GLOBAL_STEP: 1606500 | > loss_text_ce: 0.02417050115764141 (0.0300280421940079) | > loss_mel_ce: 2.702328681945801 (nan) | > loss: 0.010819441638886929 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.1369924564843507) | > loader_time: 0.0046 (0.0046898306013269)  --> TIME: 2025-02-27 07:39:54 -- STEP: 18216/36938 -- GLOBAL_STEP: 1606550 | > loss_text_ce: 0.029356755316257477 (0.030025458395571924) | > loss_mel_ce: 0.022319793701171875 (nan) | > loss: 0.00020506567670963705 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.1369933475121423) | > loader_time: 0.0048 (0.0046898422269481726)  --> TIME: 2025-02-27 07:40:03 -- STEP: 18266/36938 -- GLOBAL_STEP: 1606600 | > loss_text_ce: 0.0289375688880682 (0.030028553300652513) | > loss_mel_ce: 3.0332894325256348 (nan) | > loss: 0.012151694856584072 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.1369962908933715) | > loader_time: 0.0068 (0.0046903080189983634)  --> TIME: 2025-02-27 07:40:12 -- STEP: 18316/36938 -- GLOBAL_STEP: 1606650 | > loss_text_ce: 0.02864309959113598 (0.030028121937639887) | > loss_mel_ce: 2.757571220397949 (nan) | > loss: 0.011056406423449516 (nan) | > current_lr: 5e-06 | > step_time: 0.1461 (0.1369804069318484) | > loader_time: 0.0047 (0.00469042501691178)  --> TIME: 2025-02-27 07:40:22 -- STEP: 18366/36938 -- GLOBAL_STEP: 1606700 | > loss_text_ce: 0.022977227345108986 (0.030028682789615922) | > loss_mel_ce: 2.1509149074554443 (nan) | > loss: 0.008626556023955345 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.136986944769692) | > loader_time: 0.0046 (0.004690593096161784)  --> TIME: 2025-02-27 07:40:31 -- STEP: 18416/36938 -- GLOBAL_STEP: 1606750 | > loss_text_ce: 0.029863549396395683 (0.030029170384139423) | > loss_mel_ce: 2.60071063041687 (nan) | > loss: 0.010438786819577217 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.13697713429673097) | > loader_time: 0.0047 (0.004690576386803842)  --> TIME: 2025-02-27 07:40:40 -- STEP: 18466/36938 -- GLOBAL_STEP: 1606800 | > loss_text_ce: 0.03497156500816345 (0.03002691186198313) | > loss_mel_ce: 2.9891459941864014 (nan) | > loss: 0.01200046669691801 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13696826881930158) | > loader_time: 0.0046 (0.004690759569054661)  --> TIME: 2025-02-27 07:40:49 -- STEP: 18516/36938 -- GLOBAL_STEP: 1606850 | > loss_text_ce: 0.03365543857216835 (0.030027120375350293) | > loss_mel_ce: 2.462287425994873 (nan) | > loss: 0.009904535487294197 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.13695655572901178) | > loader_time: 0.0046 (0.004690715460057216)  --> TIME: 2025-02-27 07:40:59 -- STEP: 18566/36938 -- GLOBAL_STEP: 1606900 | > loss_text_ce: 0.03856470808386803 (0.030030109677020342) | > loss_mel_ce: 1.896270751953125 (nan) | > loss: 0.007677918765693903 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13696081483208777) | > loader_time: 0.0045 (0.004690559390905414)  --> TIME: 2025-02-27 07:41:08 -- STEP: 18616/36938 -- GLOBAL_STEP: 1606950 | > loss_text_ce: 0.02951265312731266 (0.030028888520368054) | > loss_mel_ce: 2.5529961585998535 (nan) | > loss: 0.010248051024973392 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.1369503802324914) | > loader_time: 0.0047 (0.004690406260492448)  --> TIME: 2025-02-27 07:41:17 -- STEP: 18666/36938 -- GLOBAL_STEP: 1607000 | > loss_text_ce: 0.030044931918382645 (0.03002582614510761) | > loss_mel_ce: 2.486405611038208 (nan) | > loss: 0.009985915385186672 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.1369430611246749) | > loader_time: 0.0049 (0.004690297582608544)  --> TIME: 2025-02-27 07:41:26 -- STEP: 18716/36938 -- GLOBAL_STEP: 1607050 | > loss_text_ce: 0.03475361689925194 (0.03002501608994752) | > loss_mel_ce: 2.970661163330078 (nan) | > loss: 0.011926249600946903 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13693625342517315) | > loader_time: 0.0047 (0.004690200313336556)  --> TIME: 2025-02-27 07:41:36 -- STEP: 18766/36938 -- GLOBAL_STEP: 1607100 | > loss_text_ce: 0.03069777600467205 (0.030027503201865657) | > loss_mel_ce: 2.716048002243042 (nan) | > loss: 0.010899785906076431 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13693855969519433) | > loader_time: 0.0046 (0.00469056063085605)  --> TIME: 2025-02-27 07:41:45 -- STEP: 18816/36938 -- GLOBAL_STEP: 1607150 | > loss_text_ce: 0.049994900822639465 (0.0300246101237246) | > loss_mel_ce: 1.5087890625 (nan) | > loss: 0.006185651291161776 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13694334320318602) | > loader_time: 0.0046 (0.004690528536836284)  --> TIME: 2025-02-27 07:41:54 -- STEP: 18866/36938 -- GLOBAL_STEP: 1607200 | > loss_text_ce: 0.0345950722694397 (0.030023291104191902) | > loss_mel_ce: 3.0363221168518066 (nan) | > loss: 0.012186179868876934 (nan) | > current_lr: 5e-06 | > step_time: 0.168 (0.13693677798998857) | > loader_time: 0.0048 (0.004690530152787415)  --> TIME: 2025-02-27 07:42:04 -- STEP: 18916/36938 -- GLOBAL_STEP: 1607250 | > loss_text_ce: 0.020570652559399605 (0.030020307204679225) | > loss_mel_ce: 2.358217477798462 (nan) | > loss: 0.009439636021852493 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13694164678092094) | > loader_time: 0.0045 (0.004690537394214851)  --> TIME: 2025-02-27 07:42:13 -- STEP: 18966/36938 -- GLOBAL_STEP: 1607300 | > loss_text_ce: 0.03507329523563385 (0.030019418843311323) | > loss_mel_ce: 2.4639172554016113 (nan) | > loss: 0.009916629642248154 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13695345877293785) | > loader_time: 0.0045 (0.004690652480422581)  --> TIME: 2025-02-27 07:42:23 -- STEP: 19016/36938 -- GLOBAL_STEP: 1607350 | > loss_text_ce: 0.030343234539031982 (0.03001982888054428) | > loss_mel_ce: 2.3966658115386963 (nan) | > loss: 0.009630989283323288 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13694587323562404) | > loader_time: 0.0053 (0.0046905492678077565)  --> TIME: 2025-02-27 07:42:32 -- STEP: 19066/36938 -- GLOBAL_STEP: 1607400 | > loss_text_ce: 0.03286217525601387 (0.03001954834636657) | > loss_mel_ce: 2.8580875396728516 (nan) | > loss: 0.011472023092210293 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.13695060549059557) | > loader_time: 0.0047 (0.0046906459497827535)  --> TIME: 2025-02-27 07:42:41 -- STEP: 19116/36938 -- GLOBAL_STEP: 1607450 | > loss_text_ce: 0.0249750055372715 (0.03002025137652905) | > loss_mel_ce: 2.604642629623413 (nan) | > loss: 0.010434991680085659 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.1369431283614167) | > loader_time: 0.0048 (0.004690549418027722)  --> TIME: 2025-02-27 07:42:50 -- STEP: 19166/36938 -- GLOBAL_STEP: 1607500 | > loss_text_ce: 0.02642720565199852 (0.03002271097274713) | > loss_mel_ce: 2.301058292388916 (nan) | > loss: 0.009236054494976997 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.1369306516463243) | > loader_time: 0.0045 (0.004690290318393222)  --> TIME: 2025-02-27 07:42:59 -- STEP: 19216/36938 -- GLOBAL_STEP: 1607550 | > loss_text_ce: 0.039421532303094864 (0.030025576527069935) | > loss_mel_ce: 1.807352900505066 (nan) | > loss: 0.007328470703214407 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.1369248597598107) | > loader_time: 0.0052 (0.004690231766232229)  --> TIME: 2025-02-27 07:43:09 -- STEP: 19266/36938 -- GLOBAL_STEP: 1607600 | > loss_text_ce: 0.035854652523994446 (0.03002297419864795) | > loss_mel_ce: 1.9173892736434937 (nan) | > loss: 0.00775096844881773 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13692361881809045) | > loader_time: 0.0047 (0.004690430251705309)  --> TIME: 2025-02-27 07:43:18 -- STEP: 19316/36938 -- GLOBAL_STEP: 1607650 | > loss_text_ce: 0.03513891249895096 (0.030021286524020703) | > loss_mel_ce: 2.6158809661865234 (nan) | > loss: 0.010519920848309994 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.1369186624727243) | > loader_time: 0.0047 (0.004690443538785036)  --> TIME: 2025-02-27 07:43:27 -- STEP: 19366/36938 -- GLOBAL_STEP: 1607700 | > loss_text_ce: 0.03742115944623947 (0.03002129248400637) | > loss_mel_ce: 2.6453750133514404 (nan) | > loss: 0.010646017268300056 (nan) | > current_lr: 5e-06 | > step_time: 0.1366 (0.13691822952885982) | > loader_time: 0.0046 (0.004690408657420245)  --> TIME: 2025-02-27 07:43:37 -- STEP: 19416/36938 -- GLOBAL_STEP: 1607750 | > loss_text_ce: 0.030357636511325836 (0.0300221879831099) | > loss_mel_ce: 2.6142778396606445 (nan) | > loss: 0.010494586080312729 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.1369367655923852) | > loader_time: 0.0046 (0.004690444157552837)  --> TIME: 2025-02-27 07:43:47 -- STEP: 19466/36938 -- GLOBAL_STEP: 1607800 | > loss_text_ce: 0.03392660617828369 (0.030024172246841202) | > loss_mel_ce: 3.0404350757598877 (nan) | > loss: 0.012199848890304565 (nan) | > current_lr: 5e-06 | > step_time: 0.1979 (0.136940677340152) | > loader_time: 0.0049 (0.004690365312181005)  --> TIME: 2025-02-27 07:43:56 -- STEP: 19516/36938 -- GLOBAL_STEP: 1607850 | > loss_text_ce: 0.027778014540672302 (0.030025530190572237) | > loss_mel_ce: 2.909716844558716 (nan) | > loss: 0.011656725779175758 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13693954160795568) | > loader_time: 0.0046 (0.004690356615210756)  --> TIME: 2025-02-27 07:44:05 -- STEP: 19566/36938 -- GLOBAL_STEP: 1607900 | > loss_text_ce: 0.030993904918432236 (0.030024739978281898) | > loss_mel_ce: 2.1940200328826904 (nan) | > loss: 0.008829421363770962 (nan) | > current_lr: 5e-06 | > step_time: 0.1078 (0.13694027555168098) | > loader_time: 0.0048 (0.004690511551028075)  --> TIME: 2025-02-27 07:44:14 -- STEP: 19616/36938 -- GLOBAL_STEP: 1607950 | > loss_text_ce: 0.028564168140292168 (0.03002094265341693) | > loss_mel_ce: 3.0660512447357178 (nan) | > loss: 0.012280221097171307 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13693096774340172) | > loader_time: 0.0046 (0.004690333483755857)  --> TIME: 2025-02-27 07:44:24 -- STEP: 19666/36938 -- GLOBAL_STEP: 1608000 | > loss_text_ce: 0.03180632367730141 (0.030022327432428452) | > loss_mel_ce: 3.023604154586792 (nan) | > loss: 0.01212464552372694 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.1369330725543302) | > loader_time: 0.0046 (0.0046902325416913054)  --> TIME: 2025-02-27 07:44:33 -- STEP: 19716/36938 -- GLOBAL_STEP: 1608050 | > loss_text_ce: 0.03649316728115082 (0.030026621306641955) | > loss_mel_ce: 2.7316579818725586 (nan) | > loss: 0.010984727181494236 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13693369086699636) | > loader_time: 0.0045 (0.004690068963817585)  --> TIME: 2025-02-27 07:44:42 -- STEP: 19766/36938 -- GLOBAL_STEP: 1608100 | > loss_text_ce: 0.024686098098754883 (0.03002633525135524) | > loss_mel_ce: 3.0629611015319824 (nan) | > loss: 0.012252569198608398 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13692518798226466) | > loader_time: 0.0045 (0.0046898851893539635)  --> TIME: 2025-02-27 07:44:51 -- STEP: 19816/36938 -- GLOBAL_STEP: 1608150 | > loss_text_ce: 0.03010358288884163 (0.03002664440302657) | > loss_mel_ce: 2.59240984916687 (nan) | > loss: 0.010406800545752048 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.136912340363718) | > loader_time: 0.0045 (0.00468979042578376)  --> TIME: 2025-02-27 07:45:01 -- STEP: 19866/36938 -- GLOBAL_STEP: 1608200 | > loss_text_ce: 0.024816496297717094 (0.03002480344042756) | > loss_mel_ce: 2.0138099193573 (nan) | > loss: 0.008089788258075714 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13691933089292585) | > loader_time: 0.0045 (0.004689648397905159)  --> TIME: 2025-02-27 07:45:10 -- STEP: 19916/36938 -- GLOBAL_STEP: 1608250 | > loss_text_ce: 0.03190727159380913 (0.030022936394570217) | > loss_mel_ce: 2.866008758544922 (nan) | > loss: 0.011499667540192604 (nan) | > current_lr: 5e-06 | > step_time: 0.1815 (0.1369075690510055) | > loader_time: 0.0048 (0.004689452099402637)  --> TIME: 2025-02-27 07:45:19 -- STEP: 19966/36938 -- GLOBAL_STEP: 1608300 | > loss_text_ce: 0.03224669024348259 (0.030022855333481632) | > loss_mel_ce: 2.8905141353607178 (nan) | > loss: 0.011598257347941399 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.1369033252668694) | > loader_time: 0.0045 (0.004689340539844529)  --> TIME: 2025-02-27 07:45:28 -- STEP: 20016/36938 -- GLOBAL_STEP: 1608350 | > loss_text_ce: 0.03573441132903099 (0.030021530054182087) | > loss_mel_ce: 3.2035608291625977 (nan) | > loss: 0.012854347005486488 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.1368917242276215) | > loader_time: 0.0048 (0.004689167324396991)  --> TIME: 2025-02-27 07:45:37 -- STEP: 20066/36938 -- GLOBAL_STEP: 1608400 | > loss_text_ce: 0.03228103742003441 (0.030022706031429137) | > loss_mel_ce: 2.983694314956665 (nan) | > loss: 0.011968156322836876 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13687499975333678) | > loader_time: 0.0045 (0.0046889805477708275)  --> TIME: 2025-02-27 07:45:46 -- STEP: 20116/36938 -- GLOBAL_STEP: 1608450 | > loss_text_ce: 0.034430280327796936 (0.030024138178839716) | > loss_mel_ce: 2.3586175441741943 (nan) | > loss: 0.009496222250163555 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13686937826199966) | > loader_time: 0.0045 (0.0046888186173544046)  --> TIME: 2025-02-27 07:45:55 -- STEP: 20166/36938 -- GLOBAL_STEP: 1608500 | > loss_text_ce: 0.038245879113674164 (0.030021091896470165) | > loss_mel_ce: 2.5468506813049316 (nan) | > loss: 0.010258320719003677 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13685866779814407) | > loader_time: 0.0047 (0.004688750606295848)  --> TIME: 2025-02-27 07:46:04 -- STEP: 20216/36938 -- GLOBAL_STEP: 1608550 | > loss_text_ce: 0.034331098198890686 (0.030018007789904703) | > loss_mel_ce: 1.3996829986572266 (nan) | > loss: 0.005690532270818949 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.1368540986800422) | > loader_time: 0.0046 (0.004688846166302745)  --> TIME: 2025-02-27 07:46:14 -- STEP: 20266/36938 -- GLOBAL_STEP: 1608600 | > loss_text_ce: 0.038272641599178314 (0.030018179141408544) | > loss_mel_ce: 2.7234222888946533 (nan) | > loss: 0.010959107428789139 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.1368648775894679) | > loader_time: 0.0046 (0.0046887916814159535)  --> TIME: 2025-02-27 07:46:23 -- STEP: 20316/36938 -- GLOBAL_STEP: 1608650 | > loss_text_ce: 0.02891024574637413 (0.03001789268480109) | > loss_mel_ce: 1.564415454864502 (nan) | > loss: 0.0063227214850485325 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13685163941808767) | > loader_time: 0.0045 (0.0046887110128813695)  --> TIME: 2025-02-27 07:46:32 -- STEP: 20366/36938 -- GLOBAL_STEP: 1608700 | > loss_text_ce: 0.03425345569849014 (0.030017773267434125) | > loss_mel_ce: 0.8956640958786011 (nan) | > loss: 0.003690149402245879 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13683625095475813) | > loader_time: 0.0046 (0.004688516576737539)  --> TIME: 2025-02-27 07:46:41 -- STEP: 20416/36938 -- GLOBAL_STEP: 1608750 | > loss_text_ce: 0.0266653411090374 (0.03001724131897839) | > loss_mel_ce: 3.2620737552642822 (nan) | > loss: 0.013050553388893604 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13683160469169645) | > loader_time: 0.0045 (0.004688310457340957)  --> TIME: 2025-02-27 07:46:51 -- STEP: 20466/36938 -- GLOBAL_STEP: 1608800 | > loss_text_ce: 0.026286598294973373 (0.03001747748300399) | > loss_mel_ce: 2.5057196617126465 (nan) | > loss: 0.01004764437675476 (nan) | > current_lr: 5e-06 | > step_time: 0.1782 (0.13684087097615805) | > loader_time: 0.005 (0.0046882199062159725)  --> TIME: 2025-02-27 07:47:01 -- STEP: 20516/36938 -- GLOBAL_STEP: 1608850 | > loss_text_ce: 0.02991105429828167 (0.030016455912601585) | > loss_mel_ce: 3.0196380615234375 (nan) | > loss: 0.012101385742425919 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13685001195534713) | > loader_time: 0.0046 (0.004688112690193544)  --> TIME: 2025-02-27 07:47:10 -- STEP: 20566/36938 -- GLOBAL_STEP: 1608900 | > loss_text_ce: 0.02911711111664772 (0.030014153832994862) | > loss_mel_ce: 2.850080966949463 (nan) | > loss: 0.011425389908254147 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.13686125863680146) | > loader_time: 0.0047 (0.004688105705611412)  --> TIME: 2025-02-27 07:47:19 -- STEP: 20616/36938 -- GLOBAL_STEP: 1608950 | > loss_text_ce: 0.0391143374145031 (0.030014652654618616) | > loss_mel_ce: 1.9898364543914795 (nan) | > loss: 0.008051391690969467 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13685853846969773) | > loader_time: 0.0047 (0.004688205312378038)  --> TIME: 2025-02-27 07:47:29 -- STEP: 20666/36938 -- GLOBAL_STEP: 1609000 | > loss_text_ce: 0.029335713014006615 (0.030013916819635528) | > loss_mel_ce: 2.44985032081604 (nan) | > loss: 0.009838040918111801 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13686783574712733) | > loader_time: 0.0045 (0.0046882247066469965)  --> TIME: 2025-02-27 07:47:38 -- STEP: 20716/36938 -- GLOBAL_STEP: 1609050 | > loss_text_ce: 0.029501324519515038 (0.030016534078675997) | > loss_mel_ce: 0.0287043247371912 (nan) | > loss: 0.00023097480880096555 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13686883029479344) | > loader_time: 0.0046 (0.004688171869448455)  --> TIME: 2025-02-27 07:47:48 -- STEP: 20766/36938 -- GLOBAL_STEP: 1609100 | > loss_text_ce: 0.03300366550683975 (0.03001635348086877) | > loss_mel_ce: 2.5697617530822754 (nan) | > loss: 0.010328434407711029 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13688219994303824) | > loader_time: 0.0046 (0.004688333709563942)  --> TIME: 2025-02-27 07:47:57 -- STEP: 20816/36938 -- GLOBAL_STEP: 1609150 | > loss_text_ce: 0.032798148691654205 (0.030011749485223782) | > loss_mel_ce: 1.9450984001159668 (nan) | > loss: 0.007848796434700489 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13687420186860977) | > loader_time: 0.0046 (0.004688244201328808)  --> TIME: 2025-02-27 07:48:06 -- STEP: 20866/36938 -- GLOBAL_STEP: 1609200 | > loss_text_ce: 0.034556884318590164 (0.03001060045661607) | > loss_mel_ce: 2.3471620082855225 (nan) | > loss: 0.009451266378164291 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.1368698369824841) | > loader_time: 0.0046 (0.004688212732837719)  --> TIME: 2025-02-27 07:48:16 -- STEP: 20916/36938 -- GLOBAL_STEP: 1609250 | > loss_text_ce: 0.027300138026475906 (0.030010297612222462) | > loss_mel_ce: 2.72963809967041 (nan) | > loss: 0.010940231382846832 (nan) | > current_lr: 5e-06 | > step_time: 0.1453 (0.13687143032399102) | > loader_time: 0.0049 (0.004688102990633922)  --> TIME: 2025-02-27 07:48:25 -- STEP: 20966/36938 -- GLOBAL_STEP: 1609300 | > loss_text_ce: 0.0336967371404171 (0.030008661215788195) | > loss_mel_ce: 2.3957903385162354 (nan) | > loss: 0.009640822187066078 (nan) | > current_lr: 5e-06 | > step_time: 0.1519 (0.13687470756868572) | > loader_time: 0.0048 (0.004688087929344719)  --> TIME: 2025-02-27 07:48:34 -- STEP: 21016/36938 -- GLOBAL_STEP: 1609350 | > loss_text_ce: 0.02691015601158142 (0.030006589027957276) | > loss_mel_ce: 2.251420497894287 (nan) | > loss: 0.009040994569659233 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.1368637252180681) | > loader_time: 0.0045 (0.004687918834374053)  --> TIME: 2025-02-27 07:48:44 -- STEP: 21066/36938 -- GLOBAL_STEP: 1609400 | > loss_text_ce: 0.022190365940332413 (0.030005390396451156) | > loss_mel_ce: 1.5594720840454102 (nan) | > loss: 0.006276438478380442 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13685846340053173) | > loader_time: 0.0047 (0.004687811001225596)  --> TIME: 2025-02-27 07:48:53 -- STEP: 21116/36938 -- GLOBAL_STEP: 1609450 | > loss_text_ce: 0.022099150344729424 (0.030008569289938995) | > loss_mel_ce: 2.0887677669525146 (nan) | > loss: 0.008376456797122955 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13686331463573811) | > loader_time: 0.0046 (0.004687688458618446)  --> TIME: 2025-02-27 07:49:02 -- STEP: 21166/36938 -- GLOBAL_STEP: 1609500 | > loss_text_ce: 0.025054816156625748 (0.03000724316516729) | > loss_mel_ce: 2.411362886428833 (nan) | > loss: 0.009668325074017048 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13686645477038747) | > loader_time: 0.0048 (0.0046876283243001225)  --> TIME: 2025-02-27 07:49:12 -- STEP: 21216/36938 -- GLOBAL_STEP: 1609550 | > loss_text_ce: 0.031795140355825424 (0.030007461129220144) | > loss_mel_ce: 2.536072254180908 (nan) | > loss: 0.010189950466156006 (nan) | > current_lr: 5e-06 | > step_time: 0.1448 (0.13687133478902713) | > loader_time: 0.0047 (0.004687497844613314)  --> TIME: 2025-02-27 07:49:21 -- STEP: 21266/36938 -- GLOBAL_STEP: 1609600 | > loss_text_ce: 0.024907678365707397 (0.030007836472261126) | > loss_mel_ce: 2.5006167888641357 (nan) | > loss: 0.01002192310988903 (nan) | > current_lr: 5e-06 | > step_time: 0.1264 (0.1368643165049534) | > loader_time: 0.0046 (0.004687419707223726)  --> TIME: 2025-02-27 07:49:31 -- STEP: 21316/36938 -- GLOBAL_STEP: 1609650 | > loss_text_ce: 0.03300706669688225 (0.030008031137055715) | > loss_mel_ce: 3.1108171939849854 (nan) | > loss: 0.012475494295358658 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13686940568901523) | > loader_time: 0.0046 (0.004687401574600634)  --> TIME: 2025-02-27 07:49:40 -- STEP: 21366/36938 -- GLOBAL_STEP: 1609700 | > loss_text_ce: 0.028083181008696556 (0.030009561800953703) | > loss_mel_ce: 2.928957939147949 (nan) | > loss: 0.011734290048480034 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.13686853200322885) | > loader_time: 0.0048 (0.004687246541611788)  --> TIME: 2025-02-27 07:49:50 -- STEP: 21416/36938 -- GLOBAL_STEP: 1609750 | > loss_text_ce: 0.02199246548116207 (0.030006898990363486) | > loss_mel_ce: 2.134796142578125 (nan) | > loss: 0.008558684960007668 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13687495351238615) | > loader_time: 0.0046 (0.004687125530535019)  --> TIME: 2025-02-27 07:49:59 -- STEP: 21466/36938 -- GLOBAL_STEP: 1609800 | > loss_text_ce: 0.030100395902991295 (0.030006835813809647) | > loss_mel_ce: 1.72265625 (nan) | > loss: 0.006955383811146021 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13686637849318276) | > loader_time: 0.0046 (0.004686994365129478)  --> TIME: 2025-02-27 07:50:08 -- STEP: 21516/36938 -- GLOBAL_STEP: 1609850 | > loss_text_ce: 0.028560539707541466 (0.030008011232890326) | > loss_mel_ce: 2.41995906829834 (nan) | > loss: 0.009716348722577095 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.1368625274177035) | > loader_time: 0.0045 (0.004686824338855845)  --> TIME: 2025-02-27 07:50:18 -- STEP: 21566/36938 -- GLOBAL_STEP: 1609900 | > loss_text_ce: 0.023770513013005257 (0.030007387633968943) | > loss_mel_ce: 2.397336483001709 (nan) | > loss: 0.009607568383216858 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.1368702369044502) | > loader_time: 0.0046 (0.004686763785626744)  --> TIME: 2025-02-27 07:50:27 -- STEP: 21616/36938 -- GLOBAL_STEP: 1609950 | > loss_text_ce: 0.029102206230163574 (0.030005465531069416) | > loss_mel_ce: 2.848877429962158 (nan) | > loss: 0.011420555412769318 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1368771706860137) | > loader_time: 0.0046 (0.00468671825927418)  --> TIME: 2025-02-27 07:50:37 -- STEP: 21666/36938 -- GLOBAL_STEP: 1610000 | > loss_text_ce: 0.03794553130865097 (0.030003724591354336) | > loss_mel_ce: 2.1814751625061035 (nan) | > loss: 0.008807225152850151 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.1368824545502798) | > loader_time: 0.0045 (0.004686661542622866) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1610000.pth  --> TIME: 2025-02-27 07:50:52 -- STEP: 21716/36938 -- GLOBAL_STEP: 1610050 | > loss_text_ce: 0.03355969488620758 (0.03000612554847931) | > loss_mel_ce: 2.6232197284698486 (nan) | > loss: 0.010542776435613632 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.1368944139567558) | > loader_time: 0.0047 (0.004687045144867881)  --> TIME: 2025-02-27 07:51:02 -- STEP: 21766/36938 -- GLOBAL_STEP: 1610100 | > loss_text_ce: 0.03166279196739197 (0.030005625874497437) | > loss_mel_ce: 2.9432599544525146 (nan) | > loss: 0.011805249378085136 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13690094169882183) | > loader_time: 0.0047 (0.004687086718491847)  --> TIME: 2025-02-27 07:51:11 -- STEP: 21816/36938 -- GLOBAL_STEP: 1610150 | > loss_text_ce: 0.017699725925922394 (0.030004975221757275) | > loss_mel_ce: 0.022905612364411354 (nan) | > loss: 0.00016113229503389448 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13690487785264305) | > loader_time: 0.0047 (0.004687034279424947)  --> TIME: 2025-02-27 07:51:21 -- STEP: 21866/36938 -- GLOBAL_STEP: 1610200 | > loss_text_ce: 0.031996846199035645 (0.030006576476306543) | > loss_mel_ce: 2.4025309085845947 (nan) | > loss: 0.00966082513332367 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1369022796523233) | > loader_time: 0.0045 (0.0046868741561364585)  --> TIME: 2025-02-27 07:51:30 -- STEP: 21916/36938 -- GLOBAL_STEP: 1610250 | > loss_text_ce: 0.029578212648630142 (0.030003029604827945) | > loss_mel_ce: 3.0004947185516357 (nan) | > loss: 0.012024099938571453 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13689964586506284) | > loader_time: 0.0047 (0.004686749542819053)  --> TIME: 2025-02-27 07:51:39 -- STEP: 21966/36938 -- GLOBAL_STEP: 1610300 | > loss_text_ce: 0.028618482872843742 (0.030004669345950433) | > loss_mel_ce: 2.8241751194000244 (nan) | > loss: 0.011320610530674458 (nan) | > current_lr: 5e-06 | > step_time: 0.161 (0.1369111588730001) | > loader_time: 0.0047 (0.004686649712037466)  --> TIME: 2025-02-27 07:51:48 -- STEP: 22016/36938 -- GLOBAL_STEP: 1610350 | > loss_text_ce: 0.01825525425374508 (0.030003543537299445) | > loss_mel_ce: 1.0774744749069214 (nan) | > loss: 0.004348134156316519 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13689940909010515) | > loader_time: 0.0045 (0.0046864482140991336)  --> TIME: 2025-02-27 07:51:58 -- STEP: 22066/36938 -- GLOBAL_STEP: 1610400 | > loss_text_ce: 0.050825946033000946 (0.030005923284337252) | > loss_mel_ce: 3.4156923294067383 (nan) | > loss: 0.013756025582551956 (nan) | > current_lr: 5e-06 | > step_time: 0.1537 (0.13689622007729893) | > loader_time: 0.0047 (0.004686374099432)  --> TIME: 2025-02-27 07:52:07 -- STEP: 22116/36938 -- GLOBAL_STEP: 1610450 | > loss_text_ce: 0.04092798009514809 (0.030010971308813528) | > loss_mel_ce: 3.236320972442627 (nan) | > loss: 0.013004956766963005 (nan) | > current_lr: 5e-06 | > step_time: 0.1445 (0.13689993662824823) | > loader_time: 0.0047 (0.004686333070635281)  --> TIME: 2025-02-27 07:52:16 -- STEP: 22166/36938 -- GLOBAL_STEP: 1610500 | > loss_text_ce: 0.03768688440322876 (0.03000887540027216) | > loss_mel_ce: 2.8694443702697754 (nan) | > loss: 0.01153623592108488 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13689275925032462) | > loader_time: 0.0046 (0.004686242222065277)  --> TIME: 2025-02-27 07:52:25 -- STEP: 22216/36938 -- GLOBAL_STEP: 1610550 | > loss_text_ce: 0.029245799407362938 (0.03000701651225561) | > loss_mel_ce: 2.0762014389038086 (nan) | > loss: 0.00835494976490736 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.13688901315758947) | > loader_time: 0.0047 (0.0046861069340649)  --> TIME: 2025-02-27 07:52:35 -- STEP: 22266/36938 -- GLOBAL_STEP: 1610600 | > loss_text_ce: 0.03545722737908363 (0.030008681073252295) | > loss_mel_ce: 1.8585402965545654 (nan) | > loss: 0.007515863515436649 (nan) | > current_lr: 5e-06 | > step_time: 0.1113 (0.1368853490243212) | > loader_time: 0.0047 (0.004685991024334945)  --> TIME: 2025-02-27 07:52:44 -- STEP: 22316/36938 -- GLOBAL_STEP: 1610650 | > loss_text_ce: 0.03202761337161064 (0.030008260984606263) | > loss_mel_ce: 2.412656784057617 (nan) | > loss: 0.009701129980385303 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13688348759073382) | > loader_time: 0.0048 (0.004685872022899187)  --> TIME: 2025-02-27 07:52:53 -- STEP: 22366/36938 -- GLOBAL_STEP: 1610700 | > loss_text_ce: 0.033893030136823654 (0.030008643481767676) | > loss_mel_ce: 1.912986159324646 (nan) | > loss: 0.0077257114462554455 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13688095085183752) | > loader_time: 0.0046 (0.004685894754122035)  --> TIME: 2025-02-27 07:53:03 -- STEP: 22416/36938 -- GLOBAL_STEP: 1610750 | > loss_text_ce: 0.028831901028752327 (0.030010773661982) | > loss_mel_ce: 2.9217758178710938 (nan) | > loss: 0.011708761565387249 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.13688844544474157) | > loader_time: 0.0047 (0.004685835645591907)  --> TIME: 2025-02-27 07:53:12 -- STEP: 22466/36938 -- GLOBAL_STEP: 1610800 | > loss_text_ce: 0.034363627433776855 (0.03001090880828045) | > loss_mel_ce: 2.60797381401062 (nan) | > loss: 0.010485466569662094 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13688538977185813) | > loader_time: 0.0046 (0.004685894661676422)  --> TIME: 2025-02-27 07:53:22 -- STEP: 22516/36938 -- GLOBAL_STEP: 1610850 | > loss_text_ce: 0.0283806249499321 (0.030013060823899766) | > loss_mel_ce: 2.2196359634399414 (nan) | > loss: 0.0089207012206316 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13688856971096722) | > loader_time: 0.0046 (0.004685714467631941)  --> TIME: 2025-02-27 07:53:31 -- STEP: 22566/36938 -- GLOBAL_STEP: 1610900 | > loss_text_ce: 0.029826713725924492 (0.030015942895513254) | > loss_mel_ce: 3.0104994773864746 (nan) | > loss: 0.01206478662788868 (nan) | > current_lr: 5e-06 | > step_time: 0.1874 (0.13688659235844441) | > loader_time: 0.0049 (0.004685568596088388)  --> TIME: 2025-02-27 07:53:40 -- STEP: 22616/36938 -- GLOBAL_STEP: 1610950 | > loss_text_ce: 0.031468670815229416 (0.030016012583376814) | > loss_mel_ce: 2.513059139251709 (nan) | > loss: 0.01009733323007822 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.13688309916051783) | > loader_time: 0.0048 (0.004685472801122119)  --> TIME: 2025-02-27 07:53:49 -- STEP: 22666/36938 -- GLOBAL_STEP: 1611000 | > loss_text_ce: 0.017760945484042168 (0.030015492028252964) | > loss_mel_ce: 2.7162015438079834 (nan) | > loss: 0.010849058628082275 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13687940333050203) | > loader_time: 0.0046 (0.004685520126509708)  --> TIME: 2025-02-27 07:53:59 -- STEP: 22716/36938 -- GLOBAL_STEP: 1611050 | > loss_text_ce: 0.03525788336992264 (0.030016761587296912) | > loss_mel_ce: 3.015397071838379 (nan) | > loss: 0.012105774134397507 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.13688519982767816) | > loader_time: 0.0049 (0.004685423673931007)  --> TIME: 2025-02-27 07:54:08 -- STEP: 22766/36938 -- GLOBAL_STEP: 1611100 | > loss_text_ce: 0.030231641605496407 (0.03001660790741867) | > loss_mel_ce: 2.5308773517608643 (nan) | > loss: 0.010163132101297379 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13689421448108258) | > loader_time: 0.0045 (0.004685357429017108)  --> TIME: 2025-02-27 07:54:18 -- STEP: 22816/36938 -- GLOBAL_STEP: 1611150 | > loss_text_ce: 0.030696693807840347 (0.030015320745891926) | > loss_mel_ce: 3.033949136734009 (nan) | > loss: 0.01216129306703806 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13689461212506088) | > loader_time: 0.0052 (0.004685680994562799)  --> TIME: 2025-02-27 07:54:27 -- STEP: 22866/36938 -- GLOBAL_STEP: 1611200 | > loss_text_ce: 0.034771524369716644 (0.030015195027352344) | > loss_mel_ce: 3.0927894115448 (nan) | > loss: 0.012410956434905529 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.1368862959090103) | > loader_time: 0.0047 (0.004685609649091634)  --> TIME: 2025-02-27 07:54:36 -- STEP: 22916/36938 -- GLOBAL_STEP: 1611250 | > loss_text_ce: 0.03174537792801857 (0.030014154373871022) | > loss_mel_ce: 2.626896381378174 (nan) | > loss: 0.010550166480243206 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.1368738659224319) | > loader_time: 0.0046 (0.004685443886330561)  --> TIME: 2025-02-27 07:54:45 -- STEP: 22966/36938 -- GLOBAL_STEP: 1611300 | > loss_text_ce: 0.03120841644704342 (0.030013411457408764) | > loss_mel_ce: 2.8409292697906494 (nan) | > loss: 0.01139737293124199 (nan) | > current_lr: 5e-06 | > step_time: 0.1861 (0.13687484049271884) | > loader_time: 0.0048 (0.004685332786950714)  --> TIME: 2025-02-27 07:54:54 -- STEP: 23016/36938 -- GLOBAL_STEP: 1611350 | > loss_text_ce: 0.02601839229464531 (0.030014841373650375) | > loss_mel_ce: 2.467931032180786 (nan) | > loss: 0.009896624833345413 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.136871540146568) | > loader_time: 0.0046 (0.004685206228053339)  --> TIME: 2025-02-27 07:55:03 -- STEP: 23066/36938 -- GLOBAL_STEP: 1611400 | > loss_text_ce: 0.030126454308629036 (0.030015002407011975) | > loss_mel_ce: 0.67181396484375 (nan) | > loss: 0.0027854780200868845 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13687158600471502) | > loader_time: 0.0045 (0.0046851343597138035)  --> TIME: 2025-02-27 07:55:13 -- STEP: 23116/36938 -- GLOBAL_STEP: 1611450 | > loss_text_ce: 0.024464890360832214 (0.030011949604471438) | > loss_mel_ce: 2.1785807609558105 (nan) | > loss: 0.00874224491417408 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13687950785256311) | > loader_time: 0.0045 (0.004685039255399958)  --> TIME: 2025-02-27 07:55:23 -- STEP: 23166/36938 -- GLOBAL_STEP: 1611500 | > loss_text_ce: 0.028427833691239357 (0.03001395032418227) | > loss_mel_ce: 1.939666748046875 (nan) | > loss: 0.007809899747371674 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13689514626948784) | > loader_time: 0.0048 (0.004685028284970788)  --> TIME: 2025-02-27 07:55:32 -- STEP: 23216/36938 -- GLOBAL_STEP: 1611550 | > loss_text_ce: 0.02950967662036419 (0.03001227942355948) | > loss_mel_ce: 2.7110061645507812 (nan) | > loss: 0.010875063948333263 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13689692921263966) | > loader_time: 0.0047 (0.0046849232718995226)  --> TIME: 2025-02-27 07:55:41 -- STEP: 23266/36938 -- GLOBAL_STEP: 1611600 | > loss_text_ce: 0.038711026310920715 (0.030009572299222624) | > loss_mel_ce: 2.675595283508301 (nan) | > loss: 0.010771057568490505 (nan) | > current_lr: 5e-06 | > step_time: 0.186 (0.13689824197110662) | > loader_time: 0.0049 (0.0046848879526151625)  --> TIME: 2025-02-27 07:55:51 -- STEP: 23316/36938 -- GLOBAL_STEP: 1611650 | > loss_text_ce: 0.03200729191303253 (0.030009851378887526) | > loss_mel_ce: 3.098698616027832 (nan) | > loss: 0.012423436157405376 (nan) | > current_lr: 5e-06 | > step_time: 0.1895 (0.13690479539193026) | > loader_time: 0.0048 (0.00468471408565202)  --> TIME: 2025-02-27 07:56:00 -- STEP: 23366/36938 -- GLOBAL_STEP: 1611700 | > loss_text_ce: 0.030304014682769775 (0.030009774638286954) | > loss_mel_ce: 2.680227041244507 (nan) | > loss: 0.01075607631355524 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13691230671440408) | > loader_time: 0.0047 (0.0046846127965153505)  --> TIME: 2025-02-27 07:56:10 -- STEP: 23416/36938 -- GLOBAL_STEP: 1611750 | > loss_text_ce: 0.02997247315943241 (0.03001030704362044) | > loss_mel_ce: 3.249971389770508 (nan) | > loss: 0.013015651144087315 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.13692366210397086) | > loader_time: 0.005 (0.004684576462428618)  --> TIME: 2025-02-27 07:56:19 -- STEP: 23466/36938 -- GLOBAL_STEP: 1611800 | > loss_text_ce: 0.029251500964164734 (0.03001024320388326) | > loss_mel_ce: 0.9332294464111328 (nan) | > loss: 0.003819369012489915 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13692607648393293) | > loader_time: 0.0047 (0.0046844659716863465)  --> TIME: 2025-02-27 07:56:29 -- STEP: 23516/36938 -- GLOBAL_STEP: 1611850 | > loss_text_ce: 0.03763885051012039 (0.03001155850463181) | > loss_mel_ce: 2.6508498191833496 (nan) | > loss: 0.010668606497347355 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.13693114847000704) | > loader_time: 0.0046 (0.00468443809348031)  --> TIME: 2025-02-27 07:56:38 -- STEP: 23566/36938 -- GLOBAL_STEP: 1611900 | > loss_text_ce: 0.027467936277389526 (0.030009874494531033) | > loss_mel_ce: 2.3632400035858154 (nan) | > loss: 0.009486936964094639 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13693472649704644) | > loader_time: 0.0044 (0.004684440543106648)  --> TIME: 2025-02-27 07:56:48 -- STEP: 23616/36938 -- GLOBAL_STEP: 1611950 | > loss_text_ce: 0.034449171274900436 (0.03001016333944441) | > loss_mel_ce: 0.1542256623506546 (nan) | > loss: 0.0007487097172997892 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13694632112939642) | > loader_time: 0.0048 (0.004684348830443383)  --> TIME: 2025-02-27 07:56:58 -- STEP: 23666/36938 -- GLOBAL_STEP: 1612000 | > loss_text_ce: 0.04061335325241089 (0.03000962973654483) | > loss_mel_ce: 2.626049518585205 (nan) | > loss: 0.01058199629187584 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13695311669232382) | > loader_time: 0.0047 (0.004684298739453004)  --> TIME: 2025-02-27 07:57:06 -- STEP: 23716/36938 -- GLOBAL_STEP: 1612050 | > loss_text_ce: 0.03583623841404915 (0.03000760922234112) | > loss_mel_ce: 3.135219097137451 (nan) | > loss: 0.012583553791046143 (nan) | > current_lr: 5e-06 | > step_time: 0.1815 (0.1369377199524602) | > loader_time: 0.0048 (0.004684216358103072)  --> TIME: 2025-02-27 07:57:16 -- STEP: 23766/36938 -- GLOBAL_STEP: 1612100 | > loss_text_ce: 0.025558246299624443 (0.030007635405060497) | > loss_mel_ce: 3.1531598567962646 (nan) | > loss: 0.012613961473107338 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.1369356972736062) | > loader_time: 0.0046 (0.004684136791240451)  --> TIME: 2025-02-27 07:57:25 -- STEP: 23816/36938 -- GLOBAL_STEP: 1612150 | > loss_text_ce: 0.02673901990056038 (0.030007504731200695) | > loss_mel_ce: 2.1088473796844482 (nan) | > loss: 0.00847455020993948 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13693154361731066) | > loader_time: 0.0046 (0.0046840373765793235)  --> TIME: 2025-02-27 07:57:34 -- STEP: 23866/36938 -- GLOBAL_STEP: 1612200 | > loss_text_ce: 0.027994994074106216 (0.030007278071917796) | > loss_mel_ce: 2.277050495147705 (nan) | > loss: 0.009147007018327713 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13693246084796948) | > loader_time: 0.0045 (0.004684095069807219)  --> TIME: 2025-02-27 07:57:44 -- STEP: 23916/36938 -- GLOBAL_STEP: 1612250 | > loss_text_ce: 0.029852833598852158 (0.030007892489107905) | > loss_mel_ce: 2.688194513320923 (nan) | > loss: 0.010785902850329876 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.13693521250688076) | > loader_time: 0.0051 (0.004684231336708115)  --> TIME: 2025-02-27 07:57:53 -- STEP: 23966/36938 -- GLOBAL_STEP: 1612300 | > loss_text_ce: 0.02937740460038185 (0.03000791877750885) | > loss_mel_ce: 2.89109206199646 (nan) | > loss: 0.011589165776968002 (nan) | > current_lr: 5e-06 | > step_time: 0.1262 (0.13694490732220035) | > loader_time: 0.0046 (0.00468452934986815)  --> TIME: 2025-02-27 07:58:03 -- STEP: 24016/36938 -- GLOBAL_STEP: 1612350 | > loss_text_ce: 0.03885083645582199 (0.03000754337431073) | > loss_mel_ce: 2.8977503776550293 (nan) | > loss: 0.011653180234134197 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.13694902445418752) | > loader_time: 0.0047 (0.004684546742734673)  --> TIME: 2025-02-27 07:58:12 -- STEP: 24066/36938 -- GLOBAL_STEP: 1612400 | > loss_text_ce: 0.03195139393210411 (0.03000805687584909) | > loss_mel_ce: 2.7024052143096924 (nan) | > loss: 0.010850622318685055 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.1369439825932022) | > loader_time: 0.0049 (0.004684652105626375)  --> TIME: 2025-02-27 07:58:21 -- STEP: 24116/36938 -- GLOBAL_STEP: 1612450 | > loss_text_ce: 0.025527527555823326 (0.030008178038296616) | > loss_mel_ce: 1.575822353363037 (nan) | > loss: 0.006354563403874636 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.1369454279964625) | > loader_time: 0.0046 (0.004684768124072748)  --> TIME: 2025-02-27 07:58:31 -- STEP: 24166/36938 -- GLOBAL_STEP: 1612500 | > loss_text_ce: 0.020300382748246193 (0.030007780379932753) | > loss_mel_ce: 1.2394639253616333 (nan) | > loss: 0.004999064840376377 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13695111385611752) | > loader_time: 0.0047 (0.004684800937121854)  --> TIME: 2025-02-27 07:58:40 -- STEP: 24216/36938 -- GLOBAL_STEP: 1612550 | > loss_text_ce: 0.03514643758535385 (0.030008807802568396) | > loss_mel_ce: 2.549964666366577 (nan) | > loss: 0.010258378461003304 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13694563829981674) | > loader_time: 0.0046 (0.0046849610351043)  --> TIME: 2025-02-27 07:58:49 -- STEP: 24266/36938 -- GLOBAL_STEP: 1612600 | > loss_text_ce: 0.03019273839890957 (0.03001064160107744) | > loss_mel_ce: 2.781090497970581 (nan) | > loss: 0.011155886575579643 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.1369426248885744) | > loader_time: 0.0047 (0.004684902825241922)  --> TIME: 2025-02-27 07:58:59 -- STEP: 24316/36938 -- GLOBAL_STEP: 1612650 | > loss_text_ce: 0.028050823137164116 (0.030010706006935686) | > loss_mel_ce: 3.9580078125 (nan) | > loss: 0.01581769436597824 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.1369409330053196) | > loader_time: 0.0045 (0.004684947905404739)  --> TIME: 2025-02-27 07:59:08 -- STEP: 24366/36938 -- GLOBAL_STEP: 1612700 | > loss_text_ce: 0.038943998515605927 (0.030012789746200006) | > loss_mel_ce: 3.1861963272094727 (nan) | > loss: 0.01279817707836628 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13693829024257925) | > loader_time: 0.0046 (0.004684966136735397)  --> TIME: 2025-02-27 07:59:17 -- STEP: 24416/36938 -- GLOBAL_STEP: 1612750 | > loss_text_ce: 0.03249458223581314 (0.03001166268551725) | > loss_mel_ce: 2.8033084869384766 (nan) | > loss: 0.01125318743288517 (nan) | > current_lr: 5e-06 | > step_time: 0.1121 (0.13693417002340869) | > loader_time: 0.0046 (0.0046849746164300396)  --> TIME: 2025-02-27 07:59:26 -- STEP: 24466/36938 -- GLOBAL_STEP: 1612800 | > loss_text_ce: 0.03280634805560112 (0.030010841126930348) | > loss_mel_ce: 1.9188789129257202 (nan) | > loss: 0.0077447835355997086 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1369272587510434) | > loader_time: 0.0045 (0.0046850852561700325)  --> TIME: 2025-02-27 07:59:35 -- STEP: 24516/36938 -- GLOBAL_STEP: 1612850 | > loss_text_ce: 0.02226758934557438 (0.030006254112277614) | > loss_mel_ce: 2.573676109313965 (nan) | > loss: 0.010301364585757256 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13691680242506216) | > loader_time: 0.0047 (0.004684979782580603)  --> TIME: 2025-02-27 07:59:44 -- STEP: 24566/36938 -- GLOBAL_STEP: 1612900 | > loss_text_ce: 0.037942782044410706 (0.030006167662026618) | > loss_mel_ce: 0.052978515625 (nan) | > loss: 0.00036079881829209626 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13690926819411472) | > loader_time: 0.0046 (0.004684914122145651)  --> TIME: 2025-02-27 07:59:54 -- STEP: 24616/36938 -- GLOBAL_STEP: 1612950 | > loss_text_ce: 0.03174567595124245 (0.030003371074150888) | > loss_mel_ce: 0.036407470703125 (nan) | > loss: 0.000270449003437534 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1369076952904835) | > loader_time: 0.0052 (0.004685020723782978)  --> TIME: 2025-02-27 08:00:03 -- STEP: 24666/36938 -- GLOBAL_STEP: 1613000 | > loss_text_ce: 0.039392706006765366 (0.030005097593091962) | > loss_mel_ce: 1.804530143737793 (nan) | > loss: 0.007317154668271542 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13690800213672802) | > loader_time: 0.0046 (0.00468526816973184)  --> TIME: 2025-02-27 08:00:12 -- STEP: 24716/36938 -- GLOBAL_STEP: 1613050 | > loss_text_ce: 0.028267541900277138 (0.03000500248356594) | > loss_mel_ce: 3.108247995376587 (nan) | > loss: 0.012446491047739983 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.136897158643965) | > loader_time: 0.0046 (0.004685212289038755)  --> TIME: 2025-02-27 08:00:21 -- STEP: 24766/36938 -- GLOBAL_STEP: 1613100 | > loss_text_ce: 0.030405785888433456 (0.030004288800359942) | > loss_mel_ce: 2.4114184379577637 (nan) | > loss: 0.00968977902084589 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13690339062409074) | > loader_time: 0.0047 (0.004685251063756786)  --> TIME: 2025-02-27 08:00:31 -- STEP: 24816/36938 -- GLOBAL_STEP: 1613150 | > loss_text_ce: 0.020848259329795837 (0.030004190550736526) | > loss_mel_ce: 2.2600905895233154 (nan) | > loss: 0.00905134528875351 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13690790106487907) | > loader_time: 0.0046 (0.00468547677778565)  --> TIME: 2025-02-27 08:00:40 -- STEP: 24866/36938 -- GLOBAL_STEP: 1613200 | > loss_text_ce: 0.03186699375510216 (0.030001812414441834) | > loss_mel_ce: 3.0190696716308594 (nan) | > loss: 0.01210689265280962 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.1369035035446102) | > loader_time: 0.0048 (0.004685622481974551)  --> TIME: 2025-02-27 08:00:50 -- STEP: 24916/36938 -- GLOBAL_STEP: 1613250 | > loss_text_ce: 0.029763195663690567 (0.03000192337140976) | > loss_mel_ce: 2.6727302074432373 (nan) | > loss: 0.010724181309342384 (nan) | > current_lr: 5e-06 | > step_time: 0.1343 (0.13691155162970017) | > loader_time: 0.0046 (0.004685683328126646)  --> TIME: 2025-02-27 08:00:59 -- STEP: 24966/36938 -- GLOBAL_STEP: 1613300 | > loss_text_ce: 0.03912133350968361 (0.030004031492604905) | > loss_mel_ce: 0.014923095703125 (nan) | > loss: 0.0002144620375474915 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13691910577777378) | > loader_time: 0.0048 (0.004685926378169574)  --> TIME: 2025-02-27 08:01:09 -- STEP: 25016/36938 -- GLOBAL_STEP: 1613350 | > loss_text_ce: 0.025660770013928413 (0.03000300602725987) | > loss_mel_ce: 2.620459794998169 (nan) | > loss: 0.010500478558242321 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.1369217222858487) | > loader_time: 0.0066 (0.004686075723472378)  --> TIME: 2025-02-27 08:01:18 -- STEP: 25066/36938 -- GLOBAL_STEP: 1613400 | > loss_text_ce: 0.02109765075147152 (0.030004077804586447) | > loss_mel_ce: 3.0788028240203857 (nan) | > loss: 0.012301193550229073 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.136924597800973) | > loader_time: 0.0046 (0.004686243810115685)  --> TIME: 2025-02-27 08:01:28 -- STEP: 25116/36938 -- GLOBAL_STEP: 1613450 | > loss_text_ce: 0.03755412623286247 (0.03000432050550412) | > loss_mel_ce: 2.8216488361358643 (nan) | > loss: 0.011346044018864632 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13692602954258018) | > loader_time: 0.0046 (0.004686127737613009)  --> TIME: 2025-02-27 08:01:37 -- STEP: 25166/36938 -- GLOBAL_STEP: 1613500 | > loss_text_ce: 0.02711085043847561 (0.030003510404248365) | > loss_mel_ce: 2.2062923908233643 (nan) | > loss: 0.008862711489200592 (nan) | > current_lr: 5e-06 | > step_time: 0.1828 (0.13691873813029937) | > loader_time: 0.0048 (0.00468604706584823)  --> TIME: 2025-02-27 08:01:46 -- STEP: 25216/36938 -- GLOBAL_STEP: 1613550 | > loss_text_ce: 0.03139172121882439 (0.030001833027458156) | > loss_mel_ce: 2.79923677444458 (nan) | > loss: 0.011232652701437473 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.13691276580397993) | > loader_time: 0.0046 (0.004685891792178135)  --> TIME: 2025-02-27 08:01:55 -- STEP: 25266/36938 -- GLOBAL_STEP: 1613600 | > loss_text_ce: 0.02981450967490673 (0.030000538276543847) | > loss_mel_ce: 2.909386396408081 (nan) | > loss: 0.011663496494293213 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.13691469802621048) | > loader_time: 0.0046 (0.004685853624687334)  --> TIME: 2025-02-27 08:02:05 -- STEP: 25316/36938 -- GLOBAL_STEP: 1613650 | > loss_text_ce: 0.031336843967437744 (0.030001734301091328) | > loss_mel_ce: 2.974238157272339 (nan) | > loss: 0.011926885694265366 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13691777890281304) | > loader_time: 0.0045 (0.004685846705217443)  --> TIME: 2025-02-27 08:02:14 -- STEP: 25366/36938 -- GLOBAL_STEP: 1613700 | > loss_text_ce: 0.021433250978589058 (0.030001170887884756) | > loss_mel_ce: 2.740415096282959 (nan) | > loss: 0.010959716513752937 (nan) | > current_lr: 5e-06 | > step_time: 0.1128 (0.13692069433375278) | > loader_time: 0.0046 (0.0046858428489482155)  --> TIME: 2025-02-27 08:02:24 -- STEP: 25416/36938 -- GLOBAL_STEP: 1613750 | > loss_text_ce: 0.02904820628464222 (0.03000247119105477) | > loss_mel_ce: 2.455430746078491 (nan) | > loss: 0.00985904410481453 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13692514464695485) | > loader_time: 0.0048 (0.004685921679512655)  --> TIME: 2025-02-27 08:02:33 -- STEP: 25466/36938 -- GLOBAL_STEP: 1613800 | > loss_text_ce: 0.03773752972483635 (0.030005084207656848) | > loss_mel_ce: 1.2829428911209106 (nan) | > loss: 0.0052407956682145596 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13693160752532002) | > loader_time: 0.0045 (0.004685922615715144)  --> TIME: 2025-02-27 08:02:42 -- STEP: 25516/36938 -- GLOBAL_STEP: 1613850 | > loss_text_ce: 0.03634883463382721 (0.03000589386283138) | > loss_mel_ce: 2.366983652114868 (nan) | > loss: 0.00953703373670578 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13692530474331774) | > loader_time: 0.0045 (0.0046858328284654915)  --> TIME: 2025-02-27 08:02:52 -- STEP: 25566/36938 -- GLOBAL_STEP: 1613900 | > loss_text_ce: 0.02921818383038044 (0.03000959846453975) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13693000222468624) | > loader_time: 0.0045 (0.004685755170660823)  --> TIME: 2025-02-27 08:03:01 -- STEP: 25616/36938 -- GLOBAL_STEP: 1613950 | > loss_text_ce: 0.03082066774368286 (0.030009154373403096) | > loss_mel_ce: 2.6792380809783936 (nan) | > loss: 0.01075420156121254 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13692839768512305) | > loader_time: 0.0046 (0.00468570517979881)  --> TIME: 2025-02-27 08:03:11 -- STEP: 25666/36938 -- GLOBAL_STEP: 1614000 | > loss_text_ce: 0.05325501412153244 (0.030010538849700823) | > loss_mel_ce: 2.635770797729492 (nan) | > loss: 0.01067073829472065 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13693201501027943) | > loader_time: 0.0046 (0.0046856620162551025)  --> TIME: 2025-02-27 08:03:20 -- STEP: 25716/36938 -- GLOBAL_STEP: 1614050 | > loss_text_ce: 0.038482945412397385 (0.03000822971330652) | > loss_mel_ce: 1.7374833822250366 (nan) | > loss: 0.007047485560178757 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13693587630546922) | > loader_time: 0.0046 (0.004685582093306893)  --> TIME: 2025-02-27 08:03:30 -- STEP: 25766/36938 -- GLOBAL_STEP: 1614100 | > loss_text_ce: 0.021116651594638824 (0.03000659676516149) | > loss_mel_ce: 1.5592738389968872 (nan) | > loss: 0.006271391175687313 (nan) | > current_lr: 5e-06 | > step_time: 0.1081 (0.1369383006346477) | > loader_time: 0.0046 (0.004685573674854626)  --> TIME: 2025-02-27 08:03:39 -- STEP: 25816/36938 -- GLOBAL_STEP: 1614150 | > loss_text_ce: 0.038712937384843826 (0.030008300304183267) | > loss_mel_ce: 2.3130900859832764 (nan) | > loss: 0.009332552552223206 (nan) | > current_lr: 5e-06 | > step_time: 0.1648 (0.13693097259631246) | > loader_time: 0.0048 (0.004685560301948242)  --> TIME: 2025-02-27 08:03:49 -- STEP: 25866/36938 -- GLOBAL_STEP: 1614200 | > loss_text_ce: 0.032271455973386765 (0.030004844972052955) | > loss_mel_ce: 3.117231607437134 (nan) | > loss: 0.012498028576374054 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.13694047951821936) | > loader_time: 0.0049 (0.004685574301266364)  --> TIME: 2025-02-27 08:03:58 -- STEP: 25916/36938 -- GLOBAL_STEP: 1614250 | > loss_text_ce: 0.036420878022909164 (0.0300043828705174) | > loss_mel_ce: 1.208471655845642 (nan) | > loss: 0.004940049722790718 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13693641347637966) | > loader_time: 0.0046 (0.004685511521340715)  --> TIME: 2025-02-27 08:04:07 -- STEP: 25966/36938 -- GLOBAL_STEP: 1614300 | > loss_text_ce: 0.0357433557510376 (0.030006802027726855) | > loss_mel_ce: 2.594200849533081 (nan) | > loss: 0.01043628714978695 (nan) | > current_lr: 5e-06 | > step_time: 0.1764 (0.13692828729837206) | > loader_time: 0.0048 (0.004685465556616784)  --> TIME: 2025-02-27 08:04:17 -- STEP: 26016/36938 -- GLOBAL_STEP: 1614350 | > loss_text_ce: 0.021495627239346504 (0.030006173793227207) | > loss_mel_ce: 2.238076686859131 (nan) | > loss: 0.008966556750237942 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1369387188887401) | > loader_time: 0.0046 (0.00468547042885387)  --> TIME: 2025-02-27 08:04:26 -- STEP: 26066/36938 -- GLOBAL_STEP: 1614400 | > loss_text_ce: 0.038680873811244965 (0.030008295539598346) | > loss_mel_ce: 2.579881429672241 (nan) | > loss: 0.010391120798885822 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13693108955074024) | > loader_time: 0.0047 (0.004685579152627841)  --> TIME: 2025-02-27 08:04:35 -- STEP: 26116/36938 -- GLOBAL_STEP: 1614450 | > loss_text_ce: 0.039169616997241974 (0.03000945435424457) | > loss_mel_ce: 2.6071836948394775 (nan) | > loss: 0.01050140243023634 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13693481341385622) | > loader_time: 0.0047 (0.004685476913005155)  --> TIME: 2025-02-27 08:04:45 -- STEP: 26166/36938 -- GLOBAL_STEP: 1614500 | > loss_text_ce: 0.03220878541469574 (0.030010712545459495) | > loss_mel_ce: 2.3525850772857666 (nan) | > loss: 0.009463467635214329 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13693799134615903) | > loader_time: 0.0046 (0.004685380777197013)  --> TIME: 2025-02-27 08:04:54 -- STEP: 26216/36938 -- GLOBAL_STEP: 1614550 | > loss_text_ce: 0.027841120958328247 (0.030012092515085183) | > loss_mel_ce: 3.585355043411255 (nan) | > loss: 0.014338080771267414 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13693283622425764) | > loader_time: 0.0045 (0.004685289919067135)  --> TIME: 2025-02-27 08:05:03 -- STEP: 26266/36938 -- GLOBAL_STEP: 1614600 | > loss_text_ce: 0.024239491671323776 (0.03001236056082881) | > loss_mel_ce: 2.7744009494781494 (nan) | > loss: 0.011105717159807682 (nan) | > current_lr: 5e-06 | > step_time: 0.1706 (0.13692621968537486) | > loader_time: 0.0046 (0.004685182142246766)  --> TIME: 2025-02-27 08:05:13 -- STEP: 26316/36938 -- GLOBAL_STEP: 1614650 | > loss_text_ce: 0.030771685764193535 (0.030010902069029576) | > loss_mel_ce: 2.7680468559265137 (nan) | > loss: 0.011106424033641815 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13693280785546663) | > loader_time: 0.0048 (0.004685095712250325)  --> TIME: 2025-02-27 08:05:22 -- STEP: 26366/36938 -- GLOBAL_STEP: 1614700 | > loss_text_ce: 0.03677385672926903 (0.030008746581129484) | > loss_mel_ce: 3.1262848377227783 (nan) | > loss: 0.01255182083696127 (nan) | > current_lr: 5e-06 | > step_time: 0.2009 (0.13694183394026982) | > loader_time: 0.0048 (0.004684994997135764)  --> TIME: 2025-02-27 08:05:32 -- STEP: 26416/36938 -- GLOBAL_STEP: 1614750 | > loss_text_ce: 0.03134196624159813 (0.030008920928463207) | > loss_mel_ce: 2.6582140922546387 (nan) | > loss: 0.010672842152416706 (nan) | > current_lr: 5e-06 | > step_time: 0.1523 (0.1369450061719114) | > loader_time: 0.0047 (0.004684914140395113)  --> TIME: 2025-02-27 08:05:41 -- STEP: 26466/36938 -- GLOBAL_STEP: 1614800 | > loss_text_ce: 0.028291473165154457 (0.03000489168451758) | > loss_mel_ce: 1.621614694595337 (nan) | > loss: 0.006547247059643269 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13694663301109233) | > loader_time: 0.0045 (0.004684815256898082)  --> TIME: 2025-02-27 08:05:50 -- STEP: 26516/36938 -- GLOBAL_STEP: 1614850 | > loss_text_ce: 0.03257995843887329 (0.03000563930296578) | > loss_mel_ce: 0.5555331110954285 (nan) | > loss: 0.002333782147616148 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13694591524195018) | > loader_time: 0.0046 (0.004684687775712403)  --> TIME: 2025-02-27 08:05:59 -- STEP: 26566/36938 -- GLOBAL_STEP: 1614900 | > loss_text_ce: 0.03779127821326256 (0.030006608067583523) | > loss_mel_ce: 3.051727294921875 (nan) | > loss: 0.012259994633495808 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13694323694132568) | > loader_time: 0.0047 (0.004684644767447012)  --> TIME: 2025-02-27 08:06:09 -- STEP: 26616/36938 -- GLOBAL_STEP: 1614950 | > loss_text_ce: 0.03991924226284027 (0.030008262822123603) | > loss_mel_ce: 2.3681280612945557 (nan) | > loss: 0.00955574307590723 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13694557979331284) | > loader_time: 0.0046 (0.004684532525340859)  --> TIME: 2025-02-27 08:06:18 -- STEP: 26666/36938 -- GLOBAL_STEP: 1615000 | > loss_text_ce: 0.03374280780553818 (0.03000731304367751) | > loss_mel_ce: 2.4222562313079834 (nan) | > loss: 0.009746029041707516 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13695042822497516) | > loader_time: 0.0046 (0.004684426605160483)  --> TIME: 2025-02-27 08:06:27 -- STEP: 26716/36938 -- GLOBAL_STEP: 1615050 | > loss_text_ce: 0.024830223992466927 (0.030007625488699778) | > loss_mel_ce: 1.8922895193099976 (nan) | > loss: 0.007607618346810341 (nan) | > current_lr: 5e-06 | > step_time: 0.156 (0.13694414171135283) | > loader_time: 0.0047 (0.00468441507990868)  --> TIME: 2025-02-27 08:06:37 -- STEP: 26766/36938 -- GLOBAL_STEP: 1615100 | > loss_text_ce: 0.02596687711775303 (0.030003803641332358) | > loss_mel_ce: 2.994873285293579 (nan) | > loss: 0.01198746170848608 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.13694042215466362) | > loader_time: 0.0048 (0.004684409102561202)  --> TIME: 2025-02-27 08:06:46 -- STEP: 26816/36938 -- GLOBAL_STEP: 1615150 | > loss_text_ce: 0.022680776193737984 (0.030005559968089873) | > loss_mel_ce: 2.6421194076538086 (nan) | > loss: 0.010574604384601116 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.1369480689485211) | > loader_time: 0.0048 (0.00468436420532851)  --> TIME: 2025-02-27 08:06:56 -- STEP: 26866/36938 -- GLOBAL_STEP: 1615200 | > loss_text_ce: 0.026714259758591652 (0.030004192090588504) | > loss_mel_ce: 2.3409478664398193 (nan) | > loss: 0.009395485743880272 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13696183170963883) | > loader_time: 0.0044 (0.004684347003478677)  --> TIME: 2025-02-27 08:07:05 -- STEP: 26916/36938 -- GLOBAL_STEP: 1615250 | > loss_text_ce: 0.02843419276177883 (0.030001680699747596) | > loss_mel_ce: 2.3594157695770264 (nan) | > loss: 0.009475596249103546 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13696021484680362) | > loader_time: 0.0045 (0.00468428893329517)  --> TIME: 2025-02-27 08:07:15 -- STEP: 26966/36938 -- GLOBAL_STEP: 1615300 | > loss_text_ce: 0.025231797248125076 (0.029999941887142247) | > loss_mel_ce: 2.202929735183716 (nan) | > loss: 0.00884191133081913 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.13696682715746716) | > loader_time: 0.0045 (0.004684224783344176)  --> TIME: 2025-02-27 08:07:24 -- STEP: 27016/36938 -- GLOBAL_STEP: 1615350 | > loss_text_ce: 0.03528742864727974 (0.03000048942469778) | > loss_mel_ce: 2.596637487411499 (nan) | > loss: 0.01044414658099413 (nan) | > current_lr: 5e-06 | > step_time: 0.1442 (0.13697095139866186) | > loader_time: 0.0046 (0.004684122164008638)  --> TIME: 2025-02-27 08:07:34 -- STEP: 27066/36938 -- GLOBAL_STEP: 1615400 | > loss_text_ce: 0.023240959271788597 (0.02999979539284095) | > loss_mel_ce: 2.131913185119629 (nan) | > loss: 0.00855220016092062 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.13697701546795885) | > loader_time: 0.0046 (0.0046840292963653265)  --> TIME: 2025-02-27 08:07:43 -- STEP: 27116/36938 -- GLOBAL_STEP: 1615450 | > loss_text_ce: 0.03175579383969307 (0.030000504798816118) | > loss_mel_ce: 2.6600475311279297 (nan) | > loss: 0.01068175956606865 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1369699610512358) | > loader_time: 0.0046 (0.0046838794877912555)  --> TIME: 2025-02-27 08:07:52 -- STEP: 27166/36938 -- GLOBAL_STEP: 1615500 | > loss_text_ce: 0.025769243016839027 (0.030000739774553534) | > loss_mel_ce: 3.390625 (nan) | > loss: 0.01355712115764618 (nan) | > current_lr: 5e-06 | > step_time: 0.1577 (0.13697193130200352) | > loader_time: 0.0047 (0.004683769952475902)  --> TIME: 2025-02-27 08:08:01 -- STEP: 27216/36938 -- GLOBAL_STEP: 1615550 | > loss_text_ce: 0.026724766939878464 (0.02999919908934494) | > loss_mel_ce: 1.5241750478744507 (nan) | > loss: 0.0061543649062514305 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.1369669032685563) | > loader_time: 0.0046 (0.004683627793537987)  --> TIME: 2025-02-27 08:08:11 -- STEP: 27266/36938 -- GLOBAL_STEP: 1615600 | > loss_text_ce: 0.028679562732577324 (0.030003121132264724) | > loss_mel_ce: 1.9600385427474976 (nan) | > loss: 0.007891738787293434 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.13696785014417132) | > loader_time: 0.0046 (0.004683582586701007)  --> TIME: 2025-02-27 08:08:20 -- STEP: 27316/36938 -- GLOBAL_STEP: 1615650 | > loss_text_ce: 0.02980225905776024 (0.03000203205277943) | > loss_mel_ce: 1.9968668222427368 (nan) | > loss: 0.00804233830422163 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13696861236541386) | > loader_time: 0.0045 (0.004683452681403022)  --> TIME: 2025-02-27 08:08:29 -- STEP: 27366/36938 -- GLOBAL_STEP: 1615700 | > loss_text_ce: 0.031058091670274734 (0.030003709772542925) | > loss_mel_ce: 2.5908823013305664 (nan) | > loss: 0.010404526256024837 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13695531919286147) | > loader_time: 0.0046 (0.004683267884650658)  --> TIME: 2025-02-27 08:08:38 -- STEP: 27416/36938 -- GLOBAL_STEP: 1615750 | > loss_text_ce: 0.027428356930613518 (0.03000363271504705) | > loss_mel_ce: 0.031982421875 (nan) | > loss: 0.0002357570774620399 (nan) | > current_lr: 5e-06 | > step_time: 0.1423 (0.13695236896160765) | > loader_time: 0.0047 (0.00468312962638901)  --> TIME: 2025-02-27 08:08:48 -- STEP: 27466/36938 -- GLOBAL_STEP: 1615800 | > loss_text_ce: 0.029790937900543213 (0.030002031098670254) | > loss_mel_ce: 2.4922404289245605 (nan) | > loss: 0.010008061304688454 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13695731957637874) | > loader_time: 0.0045 (0.004683001454777676)  --> TIME: 2025-02-27 08:08:57 -- STEP: 27516/36938 -- GLOBAL_STEP: 1615850 | > loss_text_ce: 0.03616072237491608 (0.030001938811197115) | > loss_mel_ce: 3.653125047683716 (nan) | > loss: 0.014640023931860924 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13696511101282663) | > loader_time: 0.0045 (0.0046828889988902965)  --> TIME: 2025-02-27 08:09:06 -- STEP: 27566/36938 -- GLOBAL_STEP: 1615900 | > loss_text_ce: 0.03017154335975647 (0.03000276281385801) | > loss_mel_ce: 2.179316282272339 (nan) | > loss: 0.008767809718847275 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13695990799698685) | > loader_time: 0.0046 (0.004682735046506254)  --> TIME: 2025-02-27 08:09:16 -- STEP: 27616/36938 -- GLOBAL_STEP: 1615950 | > loss_text_ce: 0.02786104381084442 (0.030003046428647547) | > loss_mel_ce: 2.8097774982452393 (nan) | > loss: 0.011260471306741238 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13696031060894404) | > loader_time: 0.0046 (0.004682656589070973)  --> TIME: 2025-02-27 08:09:25 -- STEP: 27666/36938 -- GLOBAL_STEP: 1616000 | > loss_text_ce: 0.024688439443707466 (0.029999801268903975) | > loss_mel_ce: 4.125 (nan) | > loss: 0.0164670180529356 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13696749540589837) | > loader_time: 0.0047 (0.0046825766916742855)  --> TIME: 2025-02-27 08:09:35 -- STEP: 27716/36938 -- GLOBAL_STEP: 1616050 | > loss_text_ce: 0.032393645495176315 (0.029998226571459986) | > loss_mel_ce: 1.92620849609375 (nan) | > loss: 0.007772231474518776 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1369655165068095) | > loader_time: 0.0046 (0.0046824456844035)  --> TIME: 2025-02-27 08:09:44 -- STEP: 27766/36938 -- GLOBAL_STEP: 1616100 | > loss_text_ce: 0.043659914284944534 (0.029997314577106495) | > loss_mel_ce: 2.59295654296875 (nan) | > loss: 0.010462764650583267 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13696227508672254) | > loader_time: 0.0046 (0.00468234008476366)  --> TIME: 2025-02-27 08:09:53 -- STEP: 27816/36938 -- GLOBAL_STEP: 1616150 | > loss_text_ce: 0.027989670634269714 (0.029997905298668005) | > loss_mel_ce: 2.9288976192474365 (nan) | > loss: 0.01173368003219366 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.13696206666835725) | > loader_time: 0.0046 (0.004682226233486447)  --> TIME: 2025-02-27 08:10:03 -- STEP: 27866/36938 -- GLOBAL_STEP: 1616200 | > loss_text_ce: 0.029383881017565727 (0.02999607114590088) | > loss_mel_ce: 2.5427350997924805 (nan) | > loss: 0.01020682230591774 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13695984603531802) | > loader_time: 0.0045 (0.004682122133813916)  --> TIME: 2025-02-27 08:10:12 -- STEP: 27916/36938 -- GLOBAL_STEP: 1616250 | > loss_text_ce: 0.03489571437239647 (0.02999508675337754) | > loss_mel_ce: 3.2980568408966064 (nan) | > loss: 0.01322600245475769 (nan) | > current_lr: 5e-06 | > step_time: 0.1905 (0.1369549932615156) | > loader_time: 0.0048 (0.004682029894112589)  --> TIME: 2025-02-27 08:10:21 -- STEP: 27966/36938 -- GLOBAL_STEP: 1616300 | > loss_text_ce: 0.03336216136813164 (0.029994542435059234) | > loss_mel_ce: 2.827313184738159 (nan) | > loss: 0.01135188713669777 (nan) | > current_lr: 5e-06 | > step_time: 0.1777 (0.1369541031023756) | > loader_time: 0.0048 (0.004682150238668362)  --> TIME: 2025-02-27 08:10:30 -- STEP: 28016/36938 -- GLOBAL_STEP: 1616350 | > loss_text_ce: 0.022403091192245483 (0.029993142684448124) | > loss_mel_ce: 2.4900238513946533 (nan) | > loss: 0.009969948790967464 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.1369521420285215) | > loader_time: 0.0046 (0.0046821926523112475)  --> TIME: 2025-02-27 08:10:40 -- STEP: 28066/36938 -- GLOBAL_STEP: 1616400 | > loss_text_ce: 0.02333267219364643 (0.029993498076702103) | > loss_mel_ce: 0.01055908203125 (nan) | > loss: 0.0001344910851912573 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13695485215733286) | > loader_time: 0.0045 (0.004682097993623831)  --> TIME: 2025-02-27 08:10:50 -- STEP: 28116/36938 -- GLOBAL_STEP: 1616450 | > loss_text_ce: 0.02800740674138069 (0.029995800222280534) | > loss_mel_ce: 2.717029094696045 (nan) | > loss: 0.010893003083765507 (nan) | > current_lr: 5e-06 | > step_time: 0.1868 (0.1369620739760735) | > loader_time: 0.005 (0.004682076080779198)  --> TIME: 2025-02-27 08:10:59 -- STEP: 28166/36938 -- GLOBAL_STEP: 1616500 | > loss_text_ce: 0.02409929409623146 (0.029996479213040567) | > loss_mel_ce: 2.380922317504883 (nan) | > loss: 0.009543737396597862 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13696510676827198) | > loader_time: 0.0045 (0.004682141712153208)  --> TIME: 2025-02-27 08:11:09 -- STEP: 28216/36938 -- GLOBAL_STEP: 1616550 | > loss_text_ce: 0.03480266407132149 (0.02999777919005829) | > loss_mel_ce: 2.56569766998291 (nan) | > loss: 0.010319446213543415 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13697315317765987) | > loader_time: 0.0046 (0.004682214817109822)  --> TIME: 2025-02-27 08:11:18 -- STEP: 28266/36938 -- GLOBAL_STEP: 1616600 | > loss_text_ce: 0.037311162799596786 (0.02999921788959425) | > loss_mel_ce: 2.428469657897949 (nan) | > loss: 0.00978484470397234 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13696714860571754) | > loader_time: 0.0046 (0.004682129375630751)  --> TIME: 2025-02-27 08:11:27 -- STEP: 28316/36938 -- GLOBAL_STEP: 1616650 | > loss_text_ce: 0.029059164226055145 (0.029998657192463526) | > loss_mel_ce: 2.956671714782715 (nan) | > loss: 0.011848139576613903 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.13697438795598046) | > loader_time: 0.0046 (0.004682156094589992)  --> TIME: 2025-02-27 08:11:37 -- STEP: 28366/36938 -- GLOBAL_STEP: 1616700 | > loss_text_ce: 0.02163912169635296 (0.029997032699144652) | > loss_mel_ce: 2.0404574871063232 (nan) | > loss: 0.008182923309504986 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13697313692525695) | > loader_time: 0.0045 (0.004682082127324507)  --> TIME: 2025-02-27 08:11:46 -- STEP: 28416/36938 -- GLOBAL_STEP: 1616750 | > loss_text_ce: 0.031135279685258865 (0.02999689561157367) | > loss_mel_ce: 2.4854085445404053 (nan) | > loss: 0.00998628605157137 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13696754703650785) | > loader_time: 0.0046 (0.00468200800923611)  --> TIME: 2025-02-27 08:11:55 -- STEP: 28466/36938 -- GLOBAL_STEP: 1616800 | > loss_text_ce: 0.033379264175891876 (0.029996353752081) | > loss_mel_ce: 2.6119935512542725 (nan) | > loss: 0.01049751229584217 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13696948637540737) | > loader_time: 0.0047 (0.004682094057638301)  --> TIME: 2025-02-27 08:12:04 -- STEP: 28516/36938 -- GLOBAL_STEP: 1616850 | > loss_text_ce: 0.019584691151976585 (0.029995572814928356) | > loss_mel_ce: 1.792978048324585 (nan) | > loss: 0.007192709483206272 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13696365763452667) | > loader_time: 0.0045 (0.0046819820278973835)  --> TIME: 2025-02-27 08:12:14 -- STEP: 28566/36938 -- GLOBAL_STEP: 1616900 | > loss_text_ce: 0.027426814660429955 (0.029993886426201398) | > loss_mel_ce: 3.007081985473633 (nan) | > loss: 0.012041701935231686 (nan) | > current_lr: 5e-06 | > step_time: 0.1556 (0.13696840009226785) | > loader_time: 0.0049 (0.004681961564617721)  --> TIME: 2025-02-27 08:12:23 -- STEP: 28616/36938 -- GLOBAL_STEP: 1616950 | > loss_text_ce: 0.03509867191314697 (0.029992980771496414) | > loss_mel_ce: 2.6471734046936035 (nan) | > loss: 0.010643936693668365 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.13696749830399446) | > loader_time: 0.0048 (0.0046820247809847325)  --> TIME: 2025-02-27 08:12:32 -- STEP: 28666/36938 -- GLOBAL_STEP: 1617000 | > loss_text_ce: 0.03446077182888985 (0.029992291344121014) | > loss_mel_ce: 2.9952304363250732 (nan) | > loss: 0.012022584676742554 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13696864034196032) | > loader_time: 0.0048 (0.004682076199392531)  --> TIME: 2025-02-27 08:12:42 -- STEP: 28716/36938 -- GLOBAL_STEP: 1617050 | > loss_text_ce: 0.03261900693178177 (0.029992868169677882) | > loss_mel_ce: 2.6172001361846924 (nan) | > loss: 0.010515156202018261 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.1369708733217267) | > loader_time: 0.0046 (0.004682257673935969)  --> TIME: 2025-02-27 08:12:51 -- STEP: 28766/36938 -- GLOBAL_STEP: 1617100 | > loss_text_ce: 0.0399254746735096 (0.029991479592429156) | > loss_mel_ce: 2.5224366188049316 (nan) | > loss: 0.01016810443252325 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13696606943440587) | > loader_time: 0.0046 (0.0046823430706162605)  --> TIME: 2025-02-27 08:13:00 -- STEP: 28816/36938 -- GLOBAL_STEP: 1617150 | > loss_text_ce: 0.028182903304696083 (0.029989727744024006) | > loss_mel_ce: 3.3964903354644775 (nan) | > loss: 0.013589974492788315 (nan) | > current_lr: 5e-06 | > step_time: 0.188 (0.13696738686216248) | > loader_time: 0.005 (0.004682312461483921)  --> TIME: 2025-02-27 08:13:10 -- STEP: 28866/36938 -- GLOBAL_STEP: 1617200 | > loss_text_ce: 0.02838241122663021 (0.02998852169631001) | > loss_mel_ce: 2.445972442626953 (nan) | > loss: 0.009818868711590767 (nan) | > current_lr: 5e-06 | > step_time: 0.1268 (0.13697989482704048) | > loader_time: 0.0046 (0.004682319762098335)  --> TIME: 2025-02-27 08:13:19 -- STEP: 28916/36938 -- GLOBAL_STEP: 1617250 | > loss_text_ce: 0.031584858894348145 (0.02998840650210864) | > loss_mel_ce: 0.0008333580917678773 (nan) | > loss: 0.00012864373275078833 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.1369690650692471) | > loader_time: 0.0045 (0.004682333625390271)  --> TIME: 2025-02-27 08:13:28 -- STEP: 28966/36938 -- GLOBAL_STEP: 1617300 | > loss_text_ce: 0.03774220123887062 (0.029991080538826737) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13696617442402978) | > loader_time: 0.0047 (0.004682411691854869)  --> TIME: 2025-02-27 08:13:38 -- STEP: 29016/36938 -- GLOBAL_STEP: 1617350 | > loss_text_ce: 0.03690866753458977 (0.029991155523547603) | > loss_mel_ce: 2.225987434387207 (nan) | > loss: 0.00897974707186222 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.1369666126121479) | > loader_time: 0.005 (0.004682548592693363)  --> TIME: 2025-02-27 08:13:47 -- STEP: 29066/36938 -- GLOBAL_STEP: 1617400 | > loss_text_ce: 0.028604567050933838 (0.029989298236730216) | > loss_mel_ce: 2.4757981300354004 (nan) | > loss: 0.009938105940818787 (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.13696351899814124) | > loader_time: 0.0049 (0.004682535250123076)  --> TIME: 2025-02-27 08:13:56 -- STEP: 29116/36938 -- GLOBAL_STEP: 1617450 | > loss_text_ce: 0.032816093415021896 (0.0299890723820989) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.1369630715597773) | > loader_time: 0.0045 (0.00468261959395869)  --> TIME: 2025-02-27 08:14:06 -- STEP: 29166/36938 -- GLOBAL_STEP: 1617500 | > loss_text_ce: 0.011712534353137016 (0.02998690091746061) | > loss_mel_ce: 1.0260313749313354 (nan) | > loss: 0.004118031822144985 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.1369676176988173) | > loader_time: 0.0045 (0.004682588878093484)  --> TIME: 2025-02-27 08:14:15 -- STEP: 29216/36938 -- GLOBAL_STEP: 1617550 | > loss_text_ce: 0.032050151377916336 (0.029989427061269384) | > loss_mel_ce: 2.3990588188171387 (nan) | > loss: 0.009647258557379246 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.1369671279489033) | > loader_time: 0.0048 (0.00468259500615043)  --> TIME: 2025-02-27 08:14:25 -- STEP: 29266/36938 -- GLOBAL_STEP: 1617600 | > loss_text_ce: 0.015648633241653442 (0.029990331718229978) | > loss_mel_ce: 1.97186279296875 (nan) | > loss: 0.007886950857937336 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13697090893036976) | > loader_time: 0.0046 (0.0046825567957301)  --> TIME: 2025-02-27 08:14:34 -- STEP: 29316/36938 -- GLOBAL_STEP: 1617650 | > loss_text_ce: 0.03668397665023804 (0.02999070826964587) | > loss_mel_ce: 2.5834453105926514 (nan) | > loss: 0.01039733923971653 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1369701198657144) | > loader_time: 0.0045 (0.004682514779417074)  --> TIME: 2025-02-27 08:14:43 -- STEP: 29366/36938 -- GLOBAL_STEP: 1617700 | > loss_text_ce: 0.026415430009365082 (0.029992456845126646) | > loss_mel_ce: 0.0016216188669204712 (nan) | > loss: 0.00011125813762191683 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.13696549043160008) | > loader_time: 0.0045 (0.004682601306026468)  --> TIME: 2025-02-27 08:14:53 -- STEP: 29416/36938 -- GLOBAL_STEP: 1617750 | > loss_text_ce: 0.030756261199712753 (0.029993718770183333) | > loss_mel_ce: 2.8253085613250732 (nan) | > loss: 0.011333591304719448 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13696866790691536) | > loader_time: 0.0046 (0.00468263893241558)  --> TIME: 2025-02-27 08:15:02 -- STEP: 29466/36938 -- GLOBAL_STEP: 1617800 | > loss_text_ce: 0.032383035868406296 (0.02999468380871354) | > loss_mel_ce: 2.211541175842285 (nan) | > loss: 0.00890446174889803 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13696890480769527) | > loader_time: 0.0049 (0.004682637277255002)  --> TIME: 2025-02-27 08:15:12 -- STEP: 29516/36938 -- GLOBAL_STEP: 1617850 | > loss_text_ce: 0.03190987929701805 (0.029992788587368705) | > loss_mel_ce: 2.8154022693634033 (nan) | > loss: 0.011298858560621738 (nan) | > current_lr: 5e-06 | > step_time: 0.1887 (0.13697202989347343) | > loader_time: 0.0048 (0.004682574666019738)  --> TIME: 2025-02-27 08:15:21 -- STEP: 29566/36938 -- GLOBAL_STEP: 1617900 | > loss_text_ce: 0.029681727290153503 (0.029993364984811702) | > loss_mel_ce: 2.657356023788452 (nan) | > loss: 0.010662849061191082 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13697149884529183) | > loader_time: 0.0048 (0.004682505783140894)  --> TIME: 2025-02-27 08:15:30 -- STEP: 29616/36938 -- GLOBAL_STEP: 1617950 | > loss_text_ce: 0.025536315515637398 (0.02999104973131468) | > loss_mel_ce: 1.7060896158218384 (nan) | > loss: 0.006871531717479229 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13697481717308735) | > loader_time: 0.0048 (0.004682509296008639)  --> TIME: 2025-02-27 08:15:40 -- STEP: 29666/36938 -- GLOBAL_STEP: 1618000 | > loss_text_ce: 0.03070070408284664 (0.029992673337041535) | > loss_mel_ce: 2.399221420288086 (nan) | > loss: 0.009642548859119415 (nan) | > current_lr: 5e-06 | > step_time: 0.1706 (0.136970718945464) | > loader_time: 0.0047 (0.004682496819952295)  --> TIME: 2025-02-27 08:15:49 -- STEP: 29716/36938 -- GLOBAL_STEP: 1618050 | > loss_text_ce: 0.028543507680296898 (0.029992828217534632) | > loss_mel_ce: 2.3445394039154053 (nan) | > loss: 0.009416996501386166 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.1369785517634687) | > loader_time: 0.0047 (0.004682466277429151)  --> TIME: 2025-02-27 08:15:59 -- STEP: 29766/36938 -- GLOBAL_STEP: 1618100 | > loss_text_ce: 0.03160589560866356 (0.029994811083210237) | > loss_mel_ce: 0.010411580093204975 (nan) | > loss: 0.00016673601930961013 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13698356986838986) | > loader_time: 0.0047 (0.004682410831037445)  --> TIME: 2025-02-27 08:16:08 -- STEP: 29816/36938 -- GLOBAL_STEP: 1618150 | > loss_text_ce: 0.021967703476548195 (0.029993527736003174) | > loss_mel_ce: 2.1985275745391846 (nan) | > loss: 0.008811489678919315 (nan) | > current_lr: 5e-06 | > step_time: 0.1615 (0.13699053944772627) | > loader_time: 0.0046 (0.004682415255215687)  --> TIME: 2025-02-27 08:16:18 -- STEP: 29866/36938 -- GLOBAL_STEP: 1618200 | > loss_text_ce: 0.03279547765851021 (0.029990170964553833) | > loss_mel_ce: 2.7747039794921875 (nan) | > loss: 0.011140870861709118 (nan) | > current_lr: 5e-06 | > step_time: 0.1793 (0.13698719524614142) | > loader_time: 0.0048 (0.004682338861230282)  --> TIME: 2025-02-27 08:16:27 -- STEP: 29916/36938 -- GLOBAL_STEP: 1618250 | > loss_text_ce: 0.0189401526004076 (0.029990405704113716) | > loss_mel_ce: 2.6294238567352295 (nan) | > loss: 0.010509382002055645 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13699419042741998) | > loader_time: 0.0048 (0.004682267026190967)  --> TIME: 2025-02-27 08:16:37 -- STEP: 29966/36938 -- GLOBAL_STEP: 1618300 | > loss_text_ce: 0.02818862907588482 (0.029989390201422945) | > loss_mel_ce: 2.2631702423095703 (nan) | > loss: 0.009092695079743862 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.1370045193793324) | > loader_time: 0.0045 (0.0046821766301292115)  --> TIME: 2025-02-27 08:16:47 -- STEP: 30016/36938 -- GLOBAL_STEP: 1618350 | > loss_text_ce: 0.026611801236867905 (0.029990234482049356) | > loss_mel_ce: 2.536648988723755 (nan) | > loss: 0.010171670466661453 (nan) | > current_lr: 5e-06 | > step_time: 0.1577 (0.137011041551004) | > loader_time: 0.0047 (0.0046821021671488074)  --> TIME: 2025-02-27 08:16:56 -- STEP: 30066/36938 -- GLOBAL_STEP: 1618400 | > loss_text_ce: 0.04449602961540222 (0.029994159355316175) | > loss_mel_ce: 2.8120861053466797 (nan) | > loss: 0.0113356439396739 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13700489107977257) | > loader_time: 0.0046 (0.004682004344698838)  --> TIME: 2025-02-27 08:17:05 -- STEP: 30116/36938 -- GLOBAL_STEP: 1618450 | > loss_text_ce: 0.02706599235534668 (0.02999439182207803) | > loss_mel_ce: 3.168104648590088 (nan) | > loss: 0.012679249048233032 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13700782807799108) | > loader_time: 0.0045 (0.0046819533496156715)  --> TIME: 2025-02-27 08:17:15 -- STEP: 30166/36938 -- GLOBAL_STEP: 1618500 | > loss_text_ce: 0.039656054228544235 (0.02999538651104161) | > loss_mel_ce: 1.3762096166610718 (nan) | > loss: 0.005618514958769083 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13700820502161412) | > loader_time: 0.0045 (0.0046819532011624036)  --> TIME: 2025-02-27 08:17:24 -- STEP: 30216/36938 -- GLOBAL_STEP: 1618550 | > loss_text_ce: 0.06487297266721725 (0.029997131832687797) | > loss_mel_ce: 1.6814613342285156 (nan) | > loss: 0.006929898634552956 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13700685651491246) | > loader_time: 0.0045 (0.004681873280502709)  --> TIME: 2025-02-27 08:17:33 -- STEP: 30266/36938 -- GLOBAL_STEP: 1618600 | > loss_text_ce: 0.03198723867535591 (0.029998499485198508) | > loss_mel_ce: 2.938744306564331 (nan) | > loss: 0.011788617819547653 (nan) | > current_lr: 5e-06 | > step_time: 0.1518 (0.1370064833896361) | > loader_time: 0.0046 (0.004681817776132894)  --> TIME: 2025-02-27 08:17:43 -- STEP: 30316/36938 -- GLOBAL_STEP: 1618650 | > loss_text_ce: 0.02106095477938652 (0.029998730185076965) | > loss_mel_ce: 2.535468578338623 (nan) | > loss: 0.010144959203898907 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.1370022263324345) | > loader_time: 0.0046 (0.0046817888400692495)  --> TIME: 2025-02-27 08:17:52 -- STEP: 30366/36938 -- GLOBAL_STEP: 1618700 | > loss_text_ce: 0.030055956915020943 (0.029998680635215626) | > loss_mel_ce: 2.065247058868408 (nan) | > loss: 0.008314695209264755 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13700169198858508) | > loader_time: 0.0048 (0.0046817668065451045)  --> TIME: 2025-02-27 08:18:01 -- STEP: 30416/36938 -- GLOBAL_STEP: 1618750 | > loss_text_ce: 0.029202435165643692 (0.029999030535301025) | > loss_mel_ce: 2.688504457473755 (nan) | > loss: 0.010784551501274109 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.137008055895647) | > loader_time: 0.0047 (0.004681743105294364)  --> TIME: 2025-02-27 08:18:11 -- STEP: 30466/36938 -- GLOBAL_STEP: 1618800 | > loss_text_ce: 0.03153301775455475 (0.029999266144803242) | > loss_mel_ce: 2.972621440887451 (nan) | > loss: 0.011921248398721218 (nan) | > current_lr: 5e-06 | > step_time: 0.1691 (0.13701401157200266) | > loader_time: 0.0046 (0.004681788442139996)  --> TIME: 2025-02-27 08:18:20 -- STEP: 30516/36938 -- GLOBAL_STEP: 1618850 | > loss_text_ce: 0.027762800455093384 (0.030000456815095607) | > loss_mel_ce: 2.645327091217041 (nan) | > loss: 0.010607500560581684 (nan) | > current_lr: 5e-06 | > step_time: 0.1623 (0.137004809277074) | > loader_time: 0.0047 (0.0046818821251134275)  --> TIME: 2025-02-27 08:18:29 -- STEP: 30566/36938 -- GLOBAL_STEP: 1618900 | > loss_text_ce: 0.02856985293328762 (0.02999871100266398) | > loss_mel_ce: 2.9462361335754395 (nan) | > loss: 0.011804786510765553 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13700049328163014) | > loader_time: 0.0046 (0.004681838804426057)  --> TIME: 2025-02-27 08:18:38 -- STEP: 30616/36938 -- GLOBAL_STEP: 1618950 | > loss_text_ce: 0.01860715076327324 (0.03000126809822878) | > loss_mel_ce: 0.030961837619543076 (nan) | > loss: 0.00019670234178192914 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13700088989432666) | > loader_time: 0.0045 (0.004681761952582842)  --> TIME: 2025-02-27 08:18:48 -- STEP: 30666/36938 -- GLOBAL_STEP: 1619000 | > loss_text_ce: 0.023979777470231056 (0.03000152629596299) | > loss_mel_ce: 2.7375340461730957 (nan) | > loss: 0.010958388447761536 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13700482094738192) | > loader_time: 0.0046 (0.004681751358449173)  --> TIME: 2025-02-27 08:18:57 -- STEP: 30716/36938 -- GLOBAL_STEP: 1619050 | > loss_text_ce: 0.03720893710851669 (0.030002288688387186) | > loss_mel_ce: 0.16517333686351776 (nan) | > loss: 0.0008031042525544763 (nan) | > current_lr: 5e-06 | > step_time: 0.1828 (0.13700599082203527) | > loader_time: 0.0048 (0.004681780812081769)  --> TIME: 2025-02-27 08:19:07 -- STEP: 30766/36938 -- GLOBAL_STEP: 1619100 | > loss_text_ce: 0.03412548080086708 (0.030003294068965798) | > loss_mel_ce: 2.641721248626709 (nan) | > loss: 0.010618440806865692 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13700622909518537) | > loader_time: 0.0046 (0.004681796833232474)  --> TIME: 2025-02-27 08:19:16 -- STEP: 30816/36938 -- GLOBAL_STEP: 1619150 | > loss_text_ce: 0.025302009657025337 (0.03000539965877856) | > loss_mel_ce: 2.591050148010254 (nan) | > loss: 0.010382350534200668 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.1370148015047052) | > loader_time: 0.0047 (0.004681765878435705)  --> TIME: 2025-02-27 08:19:26 -- STEP: 30866/36938 -- GLOBAL_STEP: 1619200 | > loss_text_ce: 0.03031158447265625 (0.030005160359568482) | > loss_mel_ce: 2.5753257274627686 (nan) | > loss: 0.010339831002056599 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.13701186559069603) | > loader_time: 0.0048 (0.004681755979981993)  --> TIME: 2025-02-27 08:19:35 -- STEP: 30916/36938 -- GLOBAL_STEP: 1619250 | > loss_text_ce: 0.03554128482937813 (0.030006009513961835) | > loss_mel_ce: 2.369910717010498 (nan) | > loss: 0.009545445442199707 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13701884176494059) | > loader_time: 0.0046 (0.004681698292559345)  --> TIME: 2025-02-27 08:19:44 -- STEP: 30966/36938 -- GLOBAL_STEP: 1619300 | > loss_text_ce: 0.04707907512784004 (0.030006084894718864) | > loss_mel_ce: 3.25675630569458 (nan) | > loss: 0.013110458850860596 (nan) | > current_lr: 5e-06 | > step_time: 0.1533 (0.13701891268530944) | > loader_time: 0.0046 (0.004681540514881838)  --> TIME: 2025-02-27 08:19:54 -- STEP: 31016/36938 -- GLOBAL_STEP: 1619350 | > loss_text_ce: 0.026488028466701508 (0.030005079688612324) | > loss_mel_ce: 2.887052536010742 (nan) | > loss: 0.011561669409275055 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.13702249734525473) | > loader_time: 0.0047 (0.004681424455664813)  --> TIME: 2025-02-27 08:20:03 -- STEP: 31066/36938 -- GLOBAL_STEP: 1619400 | > loss_text_ce: 0.025193069130182266 (0.03000292921440193) | > loss_mel_ce: 2.8685221672058105 (nan) | > loss: 0.011482996866106987 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13701854165489408) | > loader_time: 0.0047 (0.004681373152109341)  --> TIME: 2025-02-27 08:20:13 -- STEP: 31116/36938 -- GLOBAL_STEP: 1619450 | > loss_text_ce: 0.028349224478006363 (0.030002325117233792) | > loss_mel_ce: 2.941296100616455 (nan) | > loss: 0.011784307658672333 (nan) | > current_lr: 5e-06 | > step_time: 0.1607 (0.1370253667593601) | > loader_time: 0.0047 (0.004681364837748949)  --> TIME: 2025-02-27 08:20:22 -- STEP: 31166/36938 -- GLOBAL_STEP: 1619500 | > loss_text_ce: 0.029141684994101524 (0.030001838272652615) | > loss_mel_ce: 3.0005722045898438 (nan) | > loss: 0.012022675015032291 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.1370247910756077) | > loader_time: 0.0047 (0.004681446437068224)  --> TIME: 2025-02-27 08:20:32 -- STEP: 31216/36938 -- GLOBAL_STEP: 1619550 | > loss_text_ce: 0.03865249827504158 (0.030001644647374363) | > loss_mel_ce: 2.99672269821167 (nan) | > loss: 0.012045140378177166 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.13703376144888868) | > loader_time: 0.0046 (0.004681575235680125)  --> TIME: 2025-02-27 08:20:41 -- STEP: 31266/36938 -- GLOBAL_STEP: 1619600 | > loss_text_ce: 0.030263163149356842 (0.030002295796558134) | > loss_mel_ce: 2.4245948791503906 (nan) | > loss: 0.009741500951349735 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.137029186576494) | > loader_time: 0.0048 (0.004681531751421958)  --> TIME: 2025-02-27 08:20:50 -- STEP: 31316/36938 -- GLOBAL_STEP: 1619650 | > loss_text_ce: 0.032365720719099045 (0.030001162675966874) | > loss_mel_ce: 1.55859375 (nan) | > loss: 0.006313331425189972 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13703330324624818) | > loader_time: 0.0045 (0.004681461142738386)  --> TIME: 2025-02-27 08:21:00 -- STEP: 31366/36938 -- GLOBAL_STEP: 1619700 | > loss_text_ce: 0.03400314226746559 (0.030002541769661627) | > loss_mel_ce: 2.7560205459594727 (nan) | > loss: 0.011071523651480675 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13702963130978818) | > loader_time: 0.0046 (0.004681447836419759)  --> TIME: 2025-02-27 08:21:09 -- STEP: 31416/36938 -- GLOBAL_STEP: 1619750 | > loss_text_ce: 0.027247805148363113 (0.030000860191152282) | > loss_mel_ce: 0.6364455223083496 (nan) | > loss: 0.0026337038725614548 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13702939436701175) | > loader_time: 0.0053 (0.004681562410531384)  --> TIME: 2025-02-27 08:21:18 -- STEP: 31466/36938 -- GLOBAL_STEP: 1619800 | > loss_text_ce: 0.03471684083342552 (0.03000281288361108) | > loss_mel_ce: 3.039220094680786 (nan) | > loss: 0.012198163196444511 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.1370257440316809) | > loader_time: 0.0047 (0.004681617284862163)  --> TIME: 2025-02-27 08:21:27 -- STEP: 31516/36938 -- GLOBAL_STEP: 1619850 | > loss_text_ce: 0.03810463100671768 (0.030002466400716826) | > loss_mel_ce: 2.7963552474975586 (nan) | > loss: 0.011247856542468071 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13702375599992878) | > loader_time: 0.0045 (0.004681626844714755)  --> TIME: 2025-02-27 08:21:37 -- STEP: 31566/36938 -- GLOBAL_STEP: 1619900 | > loss_text_ce: 0.03659138083457947 (0.03000284738547233) | > loss_mel_ce: 2.8509693145751953 (nan) | > loss: 0.011458574794232845 (nan) | > current_lr: 5e-06 | > step_time: 0.1856 (0.1370234751983854) | > loader_time: 0.0047 (0.004681683014170738)  --> TIME: 2025-02-27 08:21:46 -- STEP: 31616/36938 -- GLOBAL_STEP: 1619950 | > loss_text_ce: 0.028434639796614647 (0.030000760131810734) | > loss_mel_ce: 3.156968832015991 (nan) | > loss: 0.012640491127967834 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13702426911124493) | > loader_time: 0.0047 (0.004681618710761103)  --> TIME: 2025-02-27 08:21:56 -- STEP: 31666/36938 -- GLOBAL_STEP: 1620000 | > loss_text_ce: 0.021655699238181114 (0.03000216152990326) | > loss_mel_ce: 2.7099742889404297 (nan) | > loss: 0.010839802213013172 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13702930407553482) | > loader_time: 0.0046 (0.004681643281410301) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1620000.pth  --> TIME: 2025-02-27 08:22:11 -- STEP: 31716/36938 -- GLOBAL_STEP: 1620050 | > loss_text_ce: 0.02859179675579071 (0.029999785692591253) | > loss_mel_ce: 2.82144832611084 (nan) | > loss: 0.011309684254229069 (nan) | > current_lr: 5e-06 | > step_time: 0.1618 (0.1370259732823623) | > loader_time: 0.0049 (0.004681645726357727)  --> TIME: 2025-02-27 08:22:20 -- STEP: 31766/36938 -- GLOBAL_STEP: 1620100 | > loss_text_ce: 0.04080524668097496 (0.030000344663673715) | > loss_mel_ce: 2.4008753299713135 (nan) | > loss: 0.0096892099827528 (nan) | > current_lr: 5e-06 | > step_time: 0.1503 (0.13702385953802365) | > loader_time: 0.0047 (0.004681608489724096)  --> TIME: 2025-02-27 08:22:30 -- STEP: 31816/36938 -- GLOBAL_STEP: 1620150 | > loss_text_ce: 0.02638627216219902 (0.03000123936811211) | > loss_mel_ce: 2.72719407081604 (nan) | > loss: 0.010926906950771809 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1370271071285842) | > loader_time: 0.0047 (0.004681596661261442)  --> TIME: 2025-02-27 08:22:39 -- STEP: 31866/36938 -- GLOBAL_STEP: 1620200 | > loss_text_ce: 0.027787886559963226 (0.030001542556307634) | > loss_mel_ce: 1.5037610530853271 (nan) | > loss: 0.006077575497329235 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13702649459816282) | > loader_time: 0.0046 (0.004681529990100255)  --> TIME: 2025-02-27 08:22:48 -- STEP: 31916/36938 -- GLOBAL_STEP: 1620250 | > loss_text_ce: 0.03168344497680664 (0.030002430398164828) | > loss_mel_ce: 1.4326378107070923 (nan) | > loss: 0.005810799077153206 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13702771736574984) | > loader_time: 0.0046 (0.004681467330161429)  --> TIME: 2025-02-27 08:22:57 -- STEP: 31966/36938 -- GLOBAL_STEP: 1620300 | > loss_text_ce: 0.02641698345541954 (0.03000266533872033) | > loss_mel_ce: 2.754349708557129 (nan) | > loss: 0.011034789495170116 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.1370215071819173) | > loader_time: 0.0046 (0.004681367797472049)  --> TIME: 2025-02-27 08:23:07 -- STEP: 32016/36938 -- GLOBAL_STEP: 1620350 | > loss_text_ce: 0.03156370669603348 (0.03000384897407229) | > loss_mel_ce: 2.670316457748413 (nan) | > loss: 0.010721747763454914 (nan) | > current_lr: 5e-06 | > step_time: 0.1564 (0.13701757338868129) | > loader_time: 0.0046 (0.004681291407730958)  --> TIME: 2025-02-27 08:23:16 -- STEP: 32066/36938 -- GLOBAL_STEP: 1620400 | > loss_text_ce: 0.029596058651804924 (0.030002661359104968) | > loss_mel_ce: 2.7857728004455566 (nan) | > loss: 0.01117209903895855 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.13701245875172263) | > loader_time: 0.0047 (0.004681202653475508)  --> TIME: 2025-02-27 08:23:25 -- STEP: 32116/36938 -- GLOBAL_STEP: 1620450 | > loss_text_ce: 0.03711938112974167 (0.030004542806883614) | > loss_mel_ce: 1.7987864017486572 (nan) | > loss: 0.007285340689122677 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13700267473213382) | > loader_time: 0.0047 (0.004681111510477862)  --> TIME: 2025-02-27 08:23:34 -- STEP: 32166/36938 -- GLOBAL_STEP: 1620500 | > loss_text_ce: 0.025153551250696182 (0.030004771168183963) | > loss_mel_ce: 2.6388590335845947 (nan) | > loss: 0.010571479797363281 (nan) | > current_lr: 5e-06 | > step_time: 0.2014 (0.13700238863627273) | > loader_time: 0.0051 (0.004681027677531598)  --> TIME: 2025-02-27 08:23:43 -- STEP: 32216/36938 -- GLOBAL_STEP: 1620550 | > loss_text_ce: 0.024510061368346214 (0.030004595961617274) | > loss_mel_ce: 2.913283586502075 (nan) | > loss: 0.011657912284135818 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13699926388213537) | > loader_time: 0.0047 (0.004680974787803722)  --> TIME: 2025-02-27 08:23:53 -- STEP: 32266/36938 -- GLOBAL_STEP: 1620600 | > loss_text_ce: 0.027537381276488304 (0.03000288435011705) | > loss_mel_ce: 2.733701467514038 (nan) | > loss: 0.010957297869026661 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.136997461444587) | > loader_time: 0.0047 (0.004680946187595264)  --> TIME: 2025-02-27 08:24:02 -- STEP: 32316/36938 -- GLOBAL_STEP: 1620650 | > loss_text_ce: 0.029876118525862694 (0.030004294141988134) | > loss_mel_ce: 2.1473677158355713 (nan) | > loss: 0.008639857172966003 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13699193280556968) | > loader_time: 0.0045 (0.0046808809126820524)  --> TIME: 2025-02-27 08:24:11 -- STEP: 32366/36938 -- GLOBAL_STEP: 1620700 | > loss_text_ce: 0.02923653833568096 (0.03000607689274742) | > loss_mel_ce: 2.7105774879455566 (nan) | > loss: 0.010872278362512589 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13699253211834528) | > loader_time: 0.0046 (0.004681007459742192)  --> TIME: 2025-02-27 08:24:21 -- STEP: 32416/36938 -- GLOBAL_STEP: 1620750 | > loss_text_ce: 0.032896336168050766 (0.030006063195827024) | > loss_mel_ce: 2.5738649368286133 (nan) | > loss: 0.010344291105866432 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.136999878057842) | > loader_time: 0.0054 (0.004681107763712144)  --> TIME: 2025-02-27 08:24:30 -- STEP: 32466/36938 -- GLOBAL_STEP: 1620800 | > loss_text_ce: 0.03179055452346802 (0.03000456314176062) | > loss_mel_ce: 2.5671796798706055 (nan) | > loss: 0.010313373990356922 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1370004490637526) | > loader_time: 0.0048 (0.004681190376338113)  --> TIME: 2025-02-27 08:24:39 -- STEP: 32516/36938 -- GLOBAL_STEP: 1620850 | > loss_text_ce: 0.03001946397125721 (0.030003087470655783) | > loss_mel_ce: 2.8420920372009277 (nan) | > loss: 0.011397268623113632 (nan) | > current_lr: 5e-06 | > step_time: 0.1921 (0.1369974746418449) | > loader_time: 0.0049 (0.004681227787612363)  --> TIME: 2025-02-27 08:24:49 -- STEP: 32566/36938 -- GLOBAL_STEP: 1620900 | > loss_text_ce: 0.026577996090054512 (0.030001840142681098) | > loss_mel_ce: 2.536959171295166 (nan) | > loss: 0.010172766633331776 (nan) | > current_lr: 5e-06 | > step_time: 0.1083 (0.13699810345899438) | > loader_time: 0.0045 (0.004681300071491849)  --> TIME: 2025-02-27 08:24:58 -- STEP: 32616/36938 -- GLOBAL_STEP: 1620950 | > loss_text_ce: 0.035404980182647705 (0.030001756901416942) | > loss_mel_ce: 2.4223921298980713 (nan) | > loss: 0.009753163903951645 (nan) | > current_lr: 5e-06 | > step_time: 0.1504 (0.13700201757287095) | > loader_time: 0.005 (0.004681272514900892)  --> TIME: 2025-02-27 08:25:08 -- STEP: 32666/36938 -- GLOBAL_STEP: 1621000 | > loss_text_ce: 0.024317974224686623 (0.030000758413625706) | > loss_mel_ce: 3.0127875804901123 (nan) | > loss: 0.012052007019519806 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1370042366369309) | > loader_time: 0.0046 (0.004681244721526786)  --> TIME: 2025-02-27 08:25:17 -- STEP: 32716/36938 -- GLOBAL_STEP: 1621050 | > loss_text_ce: 0.028535809367895126 (0.030001192070078737) | > loss_mel_ce: 2.026517629623413 (nan) | > loss: 0.008154974319040775 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.13700379840988808) | > loader_time: 0.0048 (0.004681209098856937)  --> TIME: 2025-02-27 08:25:26 -- STEP: 32766/36938 -- GLOBAL_STEP: 1621100 | > loss_text_ce: 0.03176584839820862 (0.03000020517186054) | > loss_mel_ce: 2.2907660007476807 (nan) | > loss: 0.009216397069394588 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.1369922100926468) | > loader_time: 0.0045 (0.004681135121845937)  --> TIME: 2025-02-27 08:25:35 -- STEP: 32816/36938 -- GLOBAL_STEP: 1621150 | > loss_text_ce: 0.029583662748336792 (0.029999231261624414) | > loss_mel_ce: 1.8188443183898926 (nan) | > loss: 0.00733503233641386 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.1369942131564436) | > loader_time: 0.0053 (0.004681266833374617)  --> TIME: 2025-02-27 08:25:45 -- STEP: 32866/36938 -- GLOBAL_STEP: 1621200 | > loss_text_ce: 0.03323787823319435 (0.029997884430991432) | > loss_mel_ce: 3.2344112396240234 (nan) | > loss: 0.012966862879693508 (nan) | > current_lr: 5e-06 | > step_time: 0.139 (0.13700156220413212) | > loader_time: 0.0047 (0.004681354966781964)  --> TIME: 2025-02-27 08:25:55 -- STEP: 32916/36938 -- GLOBAL_STEP: 1621250 | > loss_text_ce: 0.03214436024427414 (0.029998172620176254) | > loss_mel_ce: 2.952780246734619 (nan) | > loss: 0.011844939552247524 (nan) | > current_lr: 5e-06 | > step_time: 0.1733 (0.1370099236948101) | > loader_time: 0.0048 (0.004681330627363039)  --> TIME: 2025-02-27 08:26:04 -- STEP: 32966/36938 -- GLOBAL_STEP: 1621300 | > loss_text_ce: 0.02638726867735386 (0.029999361256298984) | > loss_mel_ce: 2.666266679763794 (nan) | > loss: 0.010685134679079056 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13700175278107257) | > loader_time: 0.0064 (0.004681322576495842)  --> TIME: 2025-02-27 08:26:13 -- STEP: 33016/36938 -- GLOBAL_STEP: 1621350 | > loss_text_ce: 0.01778940111398697 (0.029999620247140774) | > loss_mel_ce: 2.741943359375 (nan) | > loss: 0.010951320640742779 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13700039121871987) | > loader_time: 0.0052 (0.004681425440360513)  --> TIME: 2025-02-27 08:26:23 -- STEP: 33066/36938 -- GLOBAL_STEP: 1621400 | > loss_text_ce: 0.030727535486221313 (0.02999779644523011) | > loss_mel_ce: 2.749966621398926 (nan) | > loss: 0.01103450171649456 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13700477150256335) | > loader_time: 0.005 (0.004681552414711514)  --> TIME: 2025-02-27 08:26:32 -- STEP: 33116/36938 -- GLOBAL_STEP: 1621450 | > loss_text_ce: 0.025330595672130585 (0.029997578893958824) | > loss_mel_ce: 1.8405307531356812 (nan) | > loss: 0.007404211908578873 (nan) | > current_lr: 5e-06 | > step_time: 0.108 (0.1369977891481173) | > loader_time: 0.0044 (0.0046815203215019065)  --> TIME: 2025-02-27 08:26:41 -- STEP: 33166/36938 -- GLOBAL_STEP: 1621500 | > loss_text_ce: 0.02759443409740925 (0.02999652996746892) | > loss_mel_ce: 2.5920145511627197 (nan) | > loss: 0.010395273566246033 (nan) | > current_lr: 5e-06 | > step_time: 0.1774 (0.1369989958501299) | > loader_time: 0.0048 (0.004681445157248262)  --> TIME: 2025-02-27 08:26:51 -- STEP: 33216/36938 -- GLOBAL_STEP: 1621550 | > loss_text_ce: 0.039982616901397705 (0.02999924675054331) | > loss_mel_ce: 2.9015750885009766 (nan) | > loss: 0.011672848835587502 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13700045790229157) | > loader_time: 0.0046 (0.004681337682210849)  --> TIME: 2025-02-27 08:27:00 -- STEP: 33266/36938 -- GLOBAL_STEP: 1621600 | > loss_text_ce: 0.026012437418103218 (0.030000250439333666) | > loss_mel_ce: 0.006940523628145456 (nan) | > loss: 0.00013076572213321924 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13700205030683973) | > loader_time: 0.0045 (0.004681393358114407)  --> TIME: 2025-02-27 08:27:09 -- STEP: 33316/36938 -- GLOBAL_STEP: 1621650 | > loss_text_ce: 0.030385872349143028 (0.030001419569034828) | > loss_mel_ce: 2.7153921127319336 (nan) | > loss: 0.010895945131778717 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.13699783980023586) | > loader_time: 0.0046 (0.004681279205849405)  --> TIME: 2025-02-27 08:27:19 -- STEP: 33366/36938 -- GLOBAL_STEP: 1621700 | > loss_text_ce: 0.022634174674749374 (0.030003342482513195) | > loss_mel_ce: 1.587188720703125 (nan) | > loss: 0.006388186477124691 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.1369971519736159) | > loader_time: 0.0046 (0.004681282940081218)  --> TIME: 2025-02-27 08:27:28 -- STEP: 33416/36938 -- GLOBAL_STEP: 1621750 | > loss_text_ce: 0.027406223118305206 (0.030003880745206667) | > loss_mel_ce: 2.8250107765197754 (nan) | > loss: 0.0113191157579422 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13699118077027372) | > loader_time: 0.0047 (0.0046814191361377565)  --> TIME: 2025-02-27 08:27:37 -- STEP: 33466/36938 -- GLOBAL_STEP: 1621800 | > loss_text_ce: 0.02490975521504879 (0.03000427987695904) | > loss_mel_ce: 2.568018913269043 (nan) | > loss: 0.01028939988464117 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13699118723995155) | > loader_time: 0.0064 (0.004681493386359958)  --> TIME: 2025-02-27 08:27:47 -- STEP: 33516/36938 -- GLOBAL_STEP: 1621850 | > loss_text_ce: 0.027028076350688934 (0.030004275276000313) | > loss_mel_ce: 0.10624313354492188 (nan) | > loss: 0.0005288540851324797 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13699932868931275) | > loader_time: 0.0051 (0.004681598579617507)  --> TIME: 2025-02-27 08:27:56 -- STEP: 33566/36938 -- GLOBAL_STEP: 1621900 | > loss_text_ce: 0.03157465532422066 (0.030003236958508775) | > loss_mel_ce: 1.8247435092926025 (nan) | > loss: 0.007366342004388571 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.1369941898646107) | > loader_time: 0.0045 (0.004681603875723038)  --> TIME: 2025-02-27 08:28:05 -- STEP: 33616/36938 -- GLOBAL_STEP: 1621950 | > loss_text_ce: 0.028322819620370865 (0.030000362001269255) | > loss_mel_ce: 2.759472608566284 (nan) | > loss: 0.011062681674957275 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13698446747804063) | > loader_time: 0.0053 (0.004681714017410477)  --> TIME: 2025-02-27 08:28:14 -- STEP: 33666/36938 -- GLOBAL_STEP: 1622000 | > loss_text_ce: 0.020243370905518532 (0.030001339411979015) | > loss_mel_ce: 2.8362925052642822 (nan) | > loss: 0.011335460469126701 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13698925310820687) | > loader_time: 0.0045 (0.0046818653671653315)  --> TIME: 2025-02-27 08:28:23 -- STEP: 33716/36938 -- GLOBAL_STEP: 1622050 | > loss_text_ce: 0.02452877163887024 (0.029999567669583067) | > loss_mel_ce: 2.1392319202423096 (nan) | > loss: 0.008586352691054344 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1369844151499279) | > loader_time: 0.0046 (0.004681805300534366)  --> TIME: 2025-02-27 08:28:33 -- STEP: 33766/36938 -- GLOBAL_STEP: 1622100 | > loss_text_ce: 0.028728637844324112 (0.029998218861752142) | > loss_mel_ce: 2.7765512466430664 (nan) | > loss: 0.011132064275443554 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13698078362656496) | > loader_time: 0.0045 (0.004681797118814788)  --> TIME: 2025-02-27 08:28:42 -- STEP: 33816/36938 -- GLOBAL_STEP: 1622150 | > loss_text_ce: 0.03525972738862038 (0.029998778894640366) | > loss_mel_ce: 2.7058818340301514 (nan) | > loss: 0.010877546854317188 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.13698267921900717) | > loader_time: 0.0048 (0.0046816765133979605)  --> TIME: 2025-02-27 08:28:51 -- STEP: 33866/36938 -- GLOBAL_STEP: 1622200 | > loss_text_ce: 0.029691139236092567 (0.03000077544975256) | > loss_mel_ce: 3.253710985183716 (nan) | > loss: 0.013029374182224274 (nan) | > current_lr: 5e-06 | > step_time: 0.1927 (0.1369812182172631) | > loader_time: 0.0049 (0.004681594618338443)  --> TIME: 2025-02-27 08:29:01 -- STEP: 33916/36938 -- GLOBAL_STEP: 1622250 | > loss_text_ce: 0.03536660596728325 (0.03000016125554033) | > loss_mel_ce: 2.957385778427124 (nan) | > loss: 0.011876001954078674 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.13698417361663734) | > loader_time: 0.0046 (0.004681555726278177)  --> TIME: 2025-02-27 08:29:10 -- STEP: 33966/36938 -- GLOBAL_STEP: 1622300 | > loss_text_ce: 0.03119034133851528 (0.030000915859852713) | > loss_mel_ce: 3.079493522644043 (nan) | > loss: 0.012343984097242355 (nan) | > current_lr: 5e-06 | > step_time: 0.1737 (0.13699010475177711) | > loader_time: 0.0051 (0.004681588159828549)  --> TIME: 2025-02-27 08:29:19 -- STEP: 34016/36938 -- GLOBAL_STEP: 1622350 | > loss_text_ce: 0.026688432320952415 (0.029999557119413715) | > loss_mel_ce: 0.005157470703125 (nan) | > loss: 0.00012637264444492757 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13698208739986464) | > loader_time: 0.0045 (0.004681488623455333)  --> TIME: 2025-02-27 08:29:29 -- STEP: 34066/36938 -- GLOBAL_STEP: 1622400 | > loss_text_ce: 0.03341204673051834 (0.02999983701960143) | > loss_mel_ce: 2.530743360519409 (nan) | > loss: 0.010175219736993313 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.13698201602087584) | > loader_time: 0.0048 (0.004681467142106765)  --> TIME: 2025-02-27 08:29:38 -- STEP: 34116/36938 -- GLOBAL_STEP: 1622450 | > loss_text_ce: 0.03782040625810623 (0.029999564113967852) | > loss_mel_ce: 0.7396202087402344 (nan) | > loss: 0.0030850819312036037 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13698049721788988) | > loader_time: 0.0046 (0.004681481148268818)  --> TIME: 2025-02-27 08:29:47 -- STEP: 34166/36938 -- GLOBAL_STEP: 1622500 | > loss_text_ce: 0.026559650897979736 (0.030000198331433026) | > loss_mel_ce: 3.571044921875 (nan) | > loss: 0.01427620928734541 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13697684518083694) | > loader_time: 0.0044 (0.00468138334295169)  --> TIME: 2025-02-27 08:29:56 -- STEP: 34216/36938 -- GLOBAL_STEP: 1622550 | > loss_text_ce: 0.027695240452885628 (0.030000199446378852) | > loss_mel_ce: 0.0072174072265625 (nan) | > loss: 0.00013854225107934326 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.1369723915524215) | > loader_time: 0.0046 (0.0046813921906665255)  --> TIME: 2025-02-27 08:30:05 -- STEP: 34266/36938 -- GLOBAL_STEP: 1622600 | > loss_text_ce: 0.030687345191836357 (0.030001340885673536) | > loss_mel_ce: 0.7943328022956848 (nan) | > loss: 0.0032738896552473307 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1369656293911114) | > loader_time: 0.0046 (0.004681377654970943)  --> TIME: 2025-02-27 08:30:14 -- STEP: 34316/36938 -- GLOBAL_STEP: 1622650 | > loss_text_ce: 0.03841675445437431 (0.030002481457719228) | > loss_mel_ce: 2.6226806640625 (nan) | > loss: 0.010559911839663982 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1369568653728665) | > loader_time: 0.0046 (0.0046813036542572755)  --> TIME: 2025-02-27 08:30:24 -- STEP: 34366/36938 -- GLOBAL_STEP: 1622700 | > loss_text_ce: 0.038798414170742035 (0.030003498491286345) | > loss_mel_ce: 2.3646492958068848 (nan) | > loss: 0.009537491016089916 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13696617204749195) | > loader_time: 0.0047 (0.0046813264198660425)  --> TIME: 2025-02-27 08:30:34 -- STEP: 34416/36938 -- GLOBAL_STEP: 1622750 | > loss_text_ce: 0.030762847512960434 (0.030004210914093304) | > loss_mel_ce: 1.4336395263671875 (nan) | > loss: 0.0058111208491027355 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13696939533170388) | > loader_time: 0.0047 (0.004681332167610328)  --> TIME: 2025-02-27 08:30:43 -- STEP: 34466/36938 -- GLOBAL_STEP: 1622800 | > loss_text_ce: 0.028997069224715233 (0.03000418289795258) | > loss_mel_ce: 2.4791641235351562 (nan) | > loss: 0.009953022003173828 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13696284899548225) | > loader_time: 0.0045 (0.004681252592059427)  --> TIME: 2025-02-27 08:30:52 -- STEP: 34516/36938 -- GLOBAL_STEP: 1622850 | > loss_text_ce: 0.03511303290724754 (0.030004319285183635) | > loss_mel_ce: 3.0471246242523193 (nan) | > loss: 0.012231103144586086 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.1369596491933146) | > loader_time: 0.0046 (0.00468117290858886)  --> TIME: 2025-02-27 08:31:01 -- STEP: 34566/36938 -- GLOBAL_STEP: 1622900 | > loss_text_ce: 0.021947965025901794 (0.0300047129104007) | > loss_mel_ce: 2.5086288452148438 (nan) | > loss: 0.010041971690952778 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13695696275115601) | > loader_time: 0.0046 (0.004681096428461308)  --> TIME: 2025-02-27 08:31:11 -- STEP: 34616/36938 -- GLOBAL_STEP: 1622950 | > loss_text_ce: 0.03011266328394413 (0.030005582134899705) | > loss_mel_ce: 2.810253858566284 (nan) | > loss: 0.011271297000348568 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1369565445693023) | > loader_time: 0.0045 (0.004681023716347997)  --> TIME: 2025-02-27 08:31:20 -- STEP: 34666/36938 -- GLOBAL_STEP: 1623000 | > loss_text_ce: 0.03106905147433281 (0.030007585950814975) | > loss_mel_ce: 1.9165853261947632 (nan) | > loss: 0.0077287876047194 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13695406907567656) | > loader_time: 0.0046 (0.004680949020033584)  --> TIME: 2025-02-27 08:31:29 -- STEP: 34716/36938 -- GLOBAL_STEP: 1623050 | > loss_text_ce: 0.026120850816369057 (0.030007485197045766) | > loss_mel_ce: 0.005096435546875 (nan) | > loss: 0.00012387812603265047 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13694799256113407) | > loader_time: 0.0045 (0.004680818004093877)  --> TIME: 2025-02-27 08:31:38 -- STEP: 34766/36938 -- GLOBAL_STEP: 1623100 | > loss_text_ce: 0.023074736818671227 (0.03000546501055127) | > loss_mel_ce: 2.71990704536438 (nan) | > loss: 0.010884848423302174 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1369525433231321) | > loader_time: 0.0045 (0.0046807420834609785)  --> TIME: 2025-02-27 08:31:47 -- STEP: 34816/36938 -- GLOBAL_STEP: 1623150 | > loss_text_ce: 0.03162696585059166 (0.030004732941025007) | > loss_mel_ce: 2.232067108154297 (nan) | > loss: 0.008982913568615913 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13694573827225012) | > loader_time: 0.0045 (0.004680628888308975)  --> TIME: 2025-02-27 08:31:57 -- STEP: 34866/36938 -- GLOBAL_STEP: 1623200 | > loss_text_ce: 0.03029649518430233 (0.03000414742018817) | > loss_mel_ce: 2.5100698471069336 (nan) | > loss: 0.010080819949507713 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13694590965045347) | > loader_time: 0.0048 (0.004680589917590503)  --> TIME: 2025-02-27 08:32:06 -- STEP: 34916/36938 -- GLOBAL_STEP: 1623250 | > loss_text_ce: 0.03792763128876686 (0.03000590403227977) | > loss_mel_ce: 2.1041743755340576 (nan) | > loss: 0.008500405587255955 (nan) | > current_lr: 5e-06 | > step_time: 0.1793 (0.13694323700227548) | > loader_time: 0.0048 (0.0046805543019498436)  --> TIME: 2025-02-27 08:32:15 -- STEP: 34966/36938 -- GLOBAL_STEP: 1623300 | > loss_text_ce: 0.02678540162742138 (0.03000711321440065) | > loss_mel_ce: 1.6920013427734375 (nan) | > loss: 0.006820582784712315 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13694051426471654) | > loader_time: 0.0045 (0.004680463844023927)  --> TIME: 2025-02-27 08:32:24 -- STEP: 35016/36938 -- GLOBAL_STEP: 1623350 | > loss_text_ce: 0.031173374503850937 (0.030005179351113336) | > loss_mel_ce: 2.861778736114502 (nan) | > loss: 0.011479970067739487 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13693675769908936) | > loader_time: 0.0046 (0.004680405761761427)  --> TIME: 2025-02-27 08:32:34 -- STEP: 35066/36938 -- GLOBAL_STEP: 1623400 | > loss_text_ce: 0.02657589130103588 (0.030005345872304235) | > loss_mel_ce: 2.27167010307312 (nan) | > loss: 0.009120023809373379 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13694163080780358) | > loader_time: 0.0046 (0.004680479476450895)  --> TIME: 2025-02-27 08:32:43 -- STEP: 35116/36938 -- GLOBAL_STEP: 1623450 | > loss_text_ce: 0.03295382112264633 (0.03000779218164631) | > loss_mel_ce: 1.740593671798706 (nan) | > loss: 0.00703788734972477 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13693461967826503) | > loader_time: 0.0045 (0.004680517567413927)  --> TIME: 2025-02-27 08:32:52 -- STEP: 35166/36938 -- GLOBAL_STEP: 1623500 | > loss_text_ce: 0.029951395466923714 (0.03000837033683902) | > loss_mel_ce: 3.381730079650879 (nan) | > loss: 0.013538419269025326 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.13693668473765735) | > loader_time: 0.0046 (0.004680638134848995)  --> TIME: 2025-02-27 08:33:02 -- STEP: 35216/36938 -- GLOBAL_STEP: 1623550 | > loss_text_ce: 0.03445267304778099 (0.030008757812860005) | > loss_mel_ce: 3.203972101211548 (nan) | > loss: 0.01285089273005724 (nan) | > current_lr: 5e-06 | > step_time: 0.1653 (0.13694397607536637) | > loader_time: 0.0048 (0.004680649319435971)  --> TIME: 2025-02-27 08:33:12 -- STEP: 35266/36938 -- GLOBAL_STEP: 1623600 | > loss_text_ce: 0.025006525218486786 (0.03000957842293853) | > loss_mel_ce: 0.0010131895542144775 (nan) | > loss: 0.00010325284529244527 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13694949802616743) | > loader_time: 0.0045 (0.004680650047495194)  --> TIME: 2025-02-27 08:33:21 -- STEP: 35316/36938 -- GLOBAL_STEP: 1623650 | > loss_text_ce: 0.032332971692085266 (0.03001048516404899) | > loss_mel_ce: 1.6615707874298096 (nan) | > loss: 0.00672184070572257 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1369458709706664) | > loader_time: 0.0047 (0.004680641625877768)  --> TIME: 2025-02-27 08:33:30 -- STEP: 35366/36938 -- GLOBAL_STEP: 1623700 | > loss_text_ce: 0.0309151541441679 (0.030011197899217586) | > loss_mel_ce: 2.1673741340637207 (nan) | > loss: 0.008723370730876923 (nan) | > current_lr: 5e-06 | > step_time: 0.1353 (0.13695168077207726) | > loader_time: 0.0046 (0.004680573235794197)  --> TIME: 2025-02-27 08:33:40 -- STEP: 35416/36938 -- GLOBAL_STEP: 1623750 | > loss_text_ce: 0.04039321839809418 (0.030011633670913442) | > loss_mel_ce: 1.965873122215271 (nan) | > loss: 0.007961374707520008 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13694813048460092) | > loader_time: 0.0052 (0.004680516712009036)  --> TIME: 2025-02-27 08:33:49 -- STEP: 35466/36938 -- GLOBAL_STEP: 1623800 | > loss_text_ce: 0.02170395478606224 (0.030010867810323937) | > loss_mel_ce: 2.526948928833008 (nan) | > loss: 0.010113702155649662 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13694803506894102) | > loader_time: 0.0046 (0.00468052323616235)  --> TIME: 2025-02-27 08:33:58 -- STEP: 35516/36938 -- GLOBAL_STEP: 1623850 | > loss_text_ce: 0.026648972183465958 (0.030010817229587487) | > loss_mel_ce: 2.5332844257354736 (nan) | > loss: 0.010158466175198555 (nan) | > current_lr: 5e-06 | > step_time: 0.1881 (0.13695223716911553) | > loader_time: 0.0048 (0.004680468203959941)  --> TIME: 2025-02-27 08:34:08 -- STEP: 35566/36938 -- GLOBAL_STEP: 1623900 | > loss_text_ce: 0.035205088555812836 (0.03000991859784426) | > loss_mel_ce: 0.0262451171875 (nan) | > loss: 0.00024385003780480474 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13695071461243422) | > loader_time: 0.0046 (0.004680425144854736)  --> TIME: 2025-02-27 08:34:17 -- STEP: 35616/36938 -- GLOBAL_STEP: 1623950 | > loss_text_ce: 0.03331027552485466 (0.030009147468409367) | > loss_mel_ce: 2.768528461456299 (nan) | > loss: 0.011118408292531967 (nan) | > current_lr: 5e-06 | > step_time: 0.1268 (0.13695322094286216) | > loader_time: 0.0045 (0.004680405127391637)  --> TIME: 2025-02-27 08:34:26 -- STEP: 35666/36938 -- GLOBAL_STEP: 1624000 | > loss_text_ce: 0.038372281938791275 (0.030010372846778634) | > loss_mel_ce: 2.8243391513824463 (nan) | > loss: 0.011359966360032558 (nan) | > current_lr: 5e-06 | > step_time: 0.1642 (0.13694810901402035) | > loader_time: 0.0047 (0.004680299313706226)  --> TIME: 2025-02-27 08:34:36 -- STEP: 35716/36938 -- GLOBAL_STEP: 1624050 | > loss_text_ce: 0.03676078841090202 (0.030009017074801893) | > loss_mel_ce: 0.07373809814453125 (nan) | > loss: 0.00043848768109455705 (nan) | > current_lr: 5e-06 | > step_time: 0.1738 (0.13696402761247217) | > loader_time: 0.0048 (0.004680258127112743)  --> TIME: 2025-02-27 08:34:46 -- STEP: 35766/36938 -- GLOBAL_STEP: 1624100 | > loss_text_ce: 0.02712905779480934 (0.030008363499212546) | > loss_mel_ce: 2.357783555984497 (nan) | > loss: 0.009463939815759659 (nan) | > current_lr: 5e-06 | > step_time: 0.1642 (0.13697553720876002) | > loader_time: 0.0049 (0.004680185445181712)  --> TIME: 2025-02-27 08:34:56 -- STEP: 35816/36938 -- GLOBAL_STEP: 1624150 | > loss_text_ce: 0.035171400755643845 (0.03000634121342449) | > loss_mel_ce: 2.7561495304107666 (nan) | > loss: 0.011076671071350574 (nan) | > current_lr: 5e-06 | > step_time: 0.1868 (0.13697825082982987) | > loader_time: 0.0049 (0.00468008713128976)  --> TIME: 2025-02-27 08:35:05 -- STEP: 35866/36938 -- GLOBAL_STEP: 1624200 | > loss_text_ce: 0.027609212324023247 (0.030006053703938093) | > loss_mel_ce: 2.512998104095459 (nan) | > loss: 0.010081775486469269 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13698144895143996) | > loader_time: 0.0046 (0.0046799872767497586)  --> TIME: 2025-02-27 08:35:15 -- STEP: 35916/36938 -- GLOBAL_STEP: 1624250 | > loss_text_ce: 0.028938909992575645 (0.030005479181275177) | > loss_mel_ce: 2.6546223163604736 (nan) | > loss: 0.010649053379893303 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13698496085284093) | > loader_time: 0.0045 (0.004679893282981533)  --> TIME: 2025-02-27 08:35:24 -- STEP: 35966/36938 -- GLOBAL_STEP: 1624300 | > loss_text_ce: 0.02823374606668949 (0.030006866613052338) | > loss_mel_ce: 2.7817959785461426 (nan) | > loss: 0.011150912381708622 (nan) | > current_lr: 5e-06 | > step_time: 0.1807 (0.13698901450257034) | > loader_time: 0.0048 (0.0046798240115603455)  --> TIME: 2025-02-27 08:35:33 -- STEP: 36016/36938 -- GLOBAL_STEP: 1624350 | > loss_text_ce: 0.028236528858542442 (0.030008045759889638) | > loss_mel_ce: 2.871750593185425 (nan) | > loss: 0.011507886461913586 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13698361792176958) | > loader_time: 0.0045 (0.004679696797424268)  --> TIME: 2025-02-27 08:35:43 -- STEP: 36066/36938 -- GLOBAL_STEP: 1624400 | > loss_text_ce: 0.03344186767935753 (0.0300069713670111) | > loss_mel_ce: 2.610136032104492 (nan) | > loss: 0.010490388609468937 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13698242370799457) | > loader_time: 0.0044 (0.0046795993995422955)  --> TIME: 2025-02-27 08:35:52 -- STEP: 36116/36938 -- GLOBAL_STEP: 1624450 | > loss_text_ce: 0.03373071551322937 (0.0300080544783931) | > loss_mel_ce: 2.911116123199463 (nan) | > loss: 0.011685901321470737 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.1369914678867687) | > loader_time: 0.0045 (0.004679552798968794)  --> TIME: 2025-02-27 08:36:02 -- STEP: 36166/36938 -- GLOBAL_STEP: 1624500 | > loss_text_ce: 0.03412061557173729 (0.03000915263143042) | > loss_mel_ce: 3.100206136703491 (nan) | > loss: 0.012437805533409119 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13699787182400222) | > loader_time: 0.0045 (0.0046795024641357675)  --> TIME: 2025-02-27 08:36:11 -- STEP: 36216/36938 -- GLOBAL_STEP: 1624550 | > loss_text_ce: 0.02736179530620575 (0.03000886450059894) | > loss_mel_ce: 2.880354166030884 (nan) | > loss: 0.011538556776940823 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13699214211462687) | > loader_time: 0.0047 (0.004679373401086088)  --> TIME: 2025-02-27 08:36:20 -- STEP: 36266/36938 -- GLOBAL_STEP: 1624600 | > loss_text_ce: 0.027404550462961197 (0.030008256081134225) | > loss_mel_ce: 1.3844413757324219 (nan) | > loss: 0.005602563731372356 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13698930283583866) | > loader_time: 0.0046 (0.004679234247571033)  --> TIME: 2025-02-27 08:36:29 -- STEP: 36316/36938 -- GLOBAL_STEP: 1624650 | > loss_text_ce: 0.032255467027425766 (0.030006986879648735) | > loss_mel_ce: 3.56494140625 (nan) | > loss: 0.014274591580033302 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.1369861335937716) | > loader_time: 0.0045 (0.004679103079628902)  --> TIME: 2025-02-27 08:36:38 -- STEP: 36366/36938 -- GLOBAL_STEP: 1624700 | > loss_text_ce: 0.030508417636156082 (0.03000811779809504) | > loss_mel_ce: 1.937455654144287 (nan) | > loss: 0.0078093814663589 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13697908898624292) | > loader_time: 0.0045 (0.004678984473248935)  --> TIME: 2025-02-27 08:36:47 -- STEP: 36416/36938 -- GLOBAL_STEP: 1624750 | > loss_text_ce: 0.03099626488983631 (0.030007922175471897) | > loss_mel_ce: 2.2070529460906982 (nan) | > loss: 0.00888114795088768 (nan) | > current_lr: 5e-06 | > step_time: 0.1523 (0.1369750981721626) | > loader_time: 0.0047 (0.004678857196875292)  --> TIME: 2025-02-27 08:36:57 -- STEP: 36466/36938 -- GLOBAL_STEP: 1624800 | > loss_text_ce: 0.03268120065331459 (0.030007074776882022) | > loss_mel_ce: 2.2080554962158203 (nan) | > loss: 0.008891812525689602 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.1369815482616266) | > loader_time: 0.0045 (0.004678764483438687)  --> TIME: 2025-02-27 08:37:06 -- STEP: 36516/36938 -- GLOBAL_STEP: 1624850 | > loss_text_ce: 0.03600863367319107 (0.030006350599255237) | > loss_mel_ce: 2.9666099548339844 (nan) | > loss: 0.011915153823792934 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.13698043999137838) | > loader_time: 0.0046 (0.004678654205839551)  --> TIME: 2025-02-27 08:37:16 -- STEP: 36566/36938 -- GLOBAL_STEP: 1624900 | > loss_text_ce: 0.02153615839779377 (0.03000712330087709) | > loss_mel_ce: 2.5370864868164062 (nan) | > loss: 0.010153264738619328 (nan) | > current_lr: 5e-06 | > step_time: 0.1767 (0.13698181417873667) | > loader_time: 0.0048 (0.004678536835889363)  --> TIME: 2025-02-27 08:37:25 -- STEP: 36616/36938 -- GLOBAL_STEP: 1624950 | > loss_text_ce: 0.023027503862977028 (0.030007144384251346) | > loss_mel_ce: 2.8855156898498535 (nan) | > loss: 0.011541838757693768 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.1369806873842671) | > loader_time: 0.0047 (0.004678432620298317)  --> TIME: 2025-02-27 08:37:34 -- STEP: 36666/36938 -- GLOBAL_STEP: 1625000 | > loss_text_ce: 0.026503190398216248 (0.030008512205336416) | > loss_mel_ce: 2.6672117710113525 (nan) | > loss: 0.010689345188438892 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13698007694628112) | > loader_time: 0.0048 (0.004678330600655053)  --> TIME: 2025-02-27 08:37:43 -- STEP: 36716/36938 -- GLOBAL_STEP: 1625050 | > loss_text_ce: 0.022039584815502167 (0.030009295657710947) | > loss_mel_ce: 1.9661922454833984 (nan) | > loss: 0.007889809086918831 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13698063568555371) | > loader_time: 0.0045 (0.0046782290017322705)  --> TIME: 2025-02-27 08:37:53 -- STEP: 36766/36938 -- GLOBAL_STEP: 1625100 | > loss_text_ce: 0.028436416760087013 (0.03001089062316174) | > loss_mel_ce: 2.2892184257507324 (nan) | > loss: 0.009197043254971504 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13698165331637752) | > loader_time: 0.0047 (0.004678130182265062)  --> TIME: 2025-02-27 08:38:02 -- STEP: 36816/36938 -- GLOBAL_STEP: 1625150 | > loss_text_ce: 0.022683342918753624 (0.030012603592335388) | > loss_mel_ce: 2.771955728530884 (nan) | > loss: 0.011089838109910488 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13698333771863144) | > loader_time: 0.0046 (0.004678032388898475)  --> TIME: 2025-02-27 08:38:12 -- STEP: 36866/36938 -- GLOBAL_STEP: 1625200 | > loss_text_ce: 0.023301366716623306 (0.030012649443129483) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13698819597693884) | > loader_time: 0.0045 (0.004677941017541982)  --> TIME: 2025-02-27 08:38:21 -- STEP: 36916/36938 -- GLOBAL_STEP: 1625250 | > loss_text_ce: 0.03638024255633354 (0.03001264145741413) | > loss_mel_ce: 2.232102632522583 (nan) | > loss: 0.009001917205750942 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.13699126990829316) | > loader_time: 0.0048 (0.004677838300855173)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.02098453044891357 (+6.527003675246834e-07) | > avg_loss_text_ce: 0.029278555659042433 (-9.822129447123809e-05) | > avg_loss_mel_ce: 2.423865228891374 (-0.005322797758741782) | > avg_loss: 2.4531437854955715 (-0.005421019426666174) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_1625272.pth  > EPOCH: 45/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-27 08:38:50)   --> TIME: 2025-02-27 08:38:56 -- STEP: 28/36938 -- GLOBAL_STEP: 1625300 | > loss_text_ce: 0.03169773891568184 (0.030434561228113517) | > loss_mel_ce: 2.55816650390625 (2.2074639445969044) | > loss: 0.010277239605784416 (0.008880550154052409) | > current_lr: 5e-06 | > step_time: 0.1391 (0.1344208717346192) | > loader_time: 0.005 (0.004648608820778983)  --> TIME: 2025-02-27 08:39:05 -- STEP: 78/36938 -- GLOBAL_STEP: 1625350 | > loss_text_ce: 0.021570861339569092 (0.030281868500587266) | > loss_mel_ce: 1.9125715494155884 (nan) | > loss: 0.0076751685701310635 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.1332538494696984) | > loader_time: 0.0046 (0.004696097129430526)  --> TIME: 2025-02-27 08:39:14 -- STEP: 128/36938 -- GLOBAL_STEP: 1625400 | > loss_text_ce: 0.020127564668655396 (0.030292593568447046) | > loss_mel_ce: 1.7707834243774414 (nan) | > loss: 0.007106789853423834 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.13302095979452128) | > loader_time: 0.0044 (0.004675077274441719)  --> TIME: 2025-02-27 08:39:23 -- STEP: 178/36938 -- GLOBAL_STEP: 1625450 | > loss_text_ce: 0.02207261510193348 (0.02970680021000712) | > loss_mel_ce: 2.741818904876709 (nan) | > loss: 0.010967823676764965 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13447247998098302) | > loader_time: 0.0046 (0.004673325613643348)  --> TIME: 2025-02-27 08:39:32 -- STEP: 228/36938 -- GLOBAL_STEP: 1625500 | > loss_text_ce: 0.024916069582104683 (0.029722652855541622) | > loss_mel_ce: 2.6573007106781006 (nan) | > loss: 0.010643718764185905 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13419804970423369) | > loader_time: 0.0046 (0.004664660545817594)  --> TIME: 2025-02-27 08:39:41 -- STEP: 278/36938 -- GLOBAL_STEP: 1625550 | > loss_text_ce: 0.029255859553813934 (0.029622347543029476) | > loss_mel_ce: 2.651503801345825 (nan) | > loss: 0.010637936182320118 (nan) | > current_lr: 5e-06 | > step_time: 0.1558 (0.1349622594366827) | > loader_time: 0.0047 (0.004655067869227569)  --> TIME: 2025-02-27 08:39:51 -- STEP: 328/36938 -- GLOBAL_STEP: 1625600 | > loss_text_ce: 0.031208576634526253 (0.029508461426125794) | > loss_mel_ce: 2.749004602432251 (nan) | > loss: 0.011032592505216599 (nan) | > current_lr: 5e-06 | > step_time: 0.1357 (0.134987670473936) | > loader_time: 0.0046 (0.004651139422160823)  --> TIME: 2025-02-27 08:40:00 -- STEP: 378/36938 -- GLOBAL_STEP: 1625650 | > loss_text_ce: 0.025810744613409042 (0.029418395053615015) | > loss_mel_ce: 1.090316891670227 (nan) | > loss: 0.004429078195244074 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13498183154555216) | > loader_time: 0.0044 (0.004643297069287174)  --> TIME: 2025-02-27 08:40:10 -- STEP: 428/36938 -- GLOBAL_STEP: 1625700 | > loss_text_ce: 0.028811650350689888 (0.02943513255183385) | > loss_mel_ce: 2.156186103820801 (nan) | > loss: 0.008670627139508724 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.13592317004070084) | > loader_time: 0.0048 (0.004641624254601025)  --> TIME: 2025-02-27 08:40:19 -- STEP: 478/36938 -- GLOBAL_STEP: 1625750 | > loss_text_ce: 0.02807907946407795 (0.029516385158983735) | > loss_mel_ce: 2.524050712585449 (nan) | > loss: 0.01012749969959259 (nan) | > current_lr: 5e-06 | > step_time: 0.1528 (0.1365495665801617) | > loader_time: 0.0048 (0.004644147022997485)  --> TIME: 2025-02-27 08:40:29 -- STEP: 528/36938 -- GLOBAL_STEP: 1625800 | > loss_text_ce: 0.027919718995690346 (0.029682917245240373) | > loss_mel_ce: 2.3047993183135986 (nan) | > loss: 0.009256822057068348 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13634965275273161) | > loader_time: 0.0047 (0.004647188114397453)  --> TIME: 2025-02-27 08:40:38 -- STEP: 578/36938 -- GLOBAL_STEP: 1625850 | > loss_text_ce: 0.028688980266451836 (0.029709180694550377) | > loss_mel_ce: 2.55067777633667 (nan) | > loss: 0.01023558247834444 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.1365648223454563) | > loader_time: 0.0047 (0.004645814532639659)  --> TIME: 2025-02-27 08:40:47 -- STEP: 628/36938 -- GLOBAL_STEP: 1625900 | > loss_text_ce: 0.027215246111154556 (0.029743858000595765) | > loss_mel_ce: 2.6154491901397705 (nan) | > loss: 0.010486763902008533 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13684645010407556) | > loader_time: 0.0045 (0.004646487296766539)  --> TIME: 2025-02-27 08:40:57 -- STEP: 678/36938 -- GLOBAL_STEP: 1625950 | > loss_text_ce: 0.023338617756962776 (0.029789507739344794) | > loss_mel_ce: 3.0662572383880615 (nan) | > loss: 0.012260301038622856 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1370665738716237) | > loader_time: 0.0046 (0.0046445036356428)  --> TIME: 2025-02-27 08:41:06 -- STEP: 728/36938 -- GLOBAL_STEP: 1626000 | > loss_text_ce: 0.037804752588272095 (0.029741191293706525) | > loss_mel_ce: 0.21484756469726562 (nan) | > loss: 0.001002588658593595 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13711894966743804) | > loader_time: 0.0045 (0.004641203107414664)  --> TIME: 2025-02-27 08:41:16 -- STEP: 778/36938 -- GLOBAL_STEP: 1626050 | > loss_text_ce: 0.026683460921049118 (0.029775227569666114) | > loss_mel_ce: 2.6817290782928467 (nan) | > loss: 0.010747670195996761 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.13731585983139072) | > loader_time: 0.0047 (0.004640296982615647)  --> TIME: 2025-02-27 08:41:25 -- STEP: 828/36938 -- GLOBAL_STEP: 1626100 | > loss_text_ce: 0.02582141011953354 (0.02981043613182858) | > loss_mel_ce: 2.3874597549438477 (nan) | > loss: 0.009576513431966305 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13728795909651226) | > loader_time: 0.0045 (0.004637527580998367)  --> TIME: 2025-02-27 08:41:35 -- STEP: 878/36938 -- GLOBAL_STEP: 1626150 | > loss_text_ce: 0.022458484396338463 (0.02986321261509031) | > loss_mel_ce: 2.476661443710327 (nan) | > loss: 0.009917143732309341 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13730842214510477) | > loader_time: 0.0045 (0.004637033358250225)  --> TIME: 2025-02-27 08:41:43 -- STEP: 928/36938 -- GLOBAL_STEP: 1626200 | > loss_text_ce: 0.024547912180423737 (0.029792872608783817) | > loss_mel_ce: 2.965975761413574 (nan) | > loss: 0.011867158114910126 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13681336595066645) | > loader_time: 0.0048 (0.004633363207866402)  --> TIME: 2025-02-27 08:41:53 -- STEP: 978/36938 -- GLOBAL_STEP: 1626250 | > loss_text_ce: 0.027080710977315903 (0.029765551461283964) | > loss_mel_ce: 2.670293092727661 (nan) | > loss: 0.010703865438699722 (nan) | > current_lr: 5e-06 | > step_time: 0.1797 (0.13681358452467327) | > loader_time: 0.0048 (0.004631436188040575)  --> TIME: 2025-02-27 08:42:02 -- STEP: 1028/36938 -- GLOBAL_STEP: 1626300 | > loss_text_ce: 0.025810714811086655 (0.029804026176210797) | > loss_mel_ce: 2.5992608070373535 (nan) | > loss: 0.010416951030492783 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.1368185975208355) | > loader_time: 0.0045 (0.0046306581812609925)  --> TIME: 2025-02-27 08:42:11 -- STEP: 1078/36938 -- GLOBAL_STEP: 1626350 | > loss_text_ce: 0.02837217226624489 (0.029809414514501153) | > loss_mel_ce: 0.027912139892578125 (nan) | > loss: 0.0002233504637842998 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13678391457488664) | > loader_time: 0.0046 (0.004630623368031459)  --> TIME: 2025-02-27 08:42:21 -- STEP: 1128/36938 -- GLOBAL_STEP: 1626400 | > loss_text_ce: 0.02683064341545105 (0.02977725176193507) | > loss_mel_ce: 2.41860294342041 (nan) | > loss: 0.009704101830720901 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13665917156435914) | > loader_time: 0.0046 (0.004629146334127327)  --> TIME: 2025-02-27 08:42:30 -- STEP: 1178/36938 -- GLOBAL_STEP: 1626450 | > loss_text_ce: 0.03400802984833717 (0.029804592617314645) | > loss_mel_ce: 2.6528141498565674 (nan) | > loss: 0.01066199317574501 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13657026157314384) | > loader_time: 0.0046 (0.004627631961388576)  --> TIME: 2025-02-27 08:42:39 -- STEP: 1228/36938 -- GLOBAL_STEP: 1626500 | > loss_text_ce: 0.01666126400232315 (0.02974771083437258) | > loss_mel_ce: 1.3819626569747925 (nan) | > loss: 0.005550095345824957 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.1364477882556107) | > loader_time: 0.0044 (0.004626327306517557)  --> TIME: 2025-02-27 08:42:48 -- STEP: 1278/36938 -- GLOBAL_STEP: 1626550 | > loss_text_ce: 0.03687485307455063 (0.029813263594815637) | > loss_mel_ce: 2.533263921737671 (nan) | > loss: 0.010198963806033134 (nan) | > current_lr: 5e-06 | > step_time: 0.161 (0.13636781594749522) | > loader_time: 0.0046 (0.004625486842530055)  --> TIME: 2025-02-27 08:42:57 -- STEP: 1328/36938 -- GLOBAL_STEP: 1626600 | > loss_text_ce: 0.03504747152328491 (0.029832388001216396) | > loss_mel_ce: 2.9681806564331055 (nan) | > loss: 0.01191757246851921 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13624919035348532) | > loader_time: 0.0045 (0.004623847912593052)  --> TIME: 2025-02-27 08:43:06 -- STEP: 1378/36938 -- GLOBAL_STEP: 1626650 | > loss_text_ce: 0.03767257556319237 (0.029828074500735186) | > loss_mel_ce: 1.0770295858383179 (nan) | > loss: 0.0044234213419258595 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.1361688076152164) | > loader_time: 0.0047 (0.004622463044995655)  --> TIME: 2025-02-27 08:43:16 -- STEP: 1428/36938 -- GLOBAL_STEP: 1626700 | > loss_text_ce: 0.03242412582039833 (0.02981552077067301) | > loss_mel_ce: 2.3006503582000732 (nan) | > loss: 0.00925823301076889 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13649895211227803) | > loader_time: 0.0045 (0.004622965967621777)  --> TIME: 2025-02-27 08:43:25 -- STEP: 1478/36938 -- GLOBAL_STEP: 1626750 | > loss_text_ce: 0.030034691095352173 (0.029824639010496427) | > loss_mel_ce: 1.6935185194015503 (nan) | > loss: 0.00683949701488018 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13636848600694715) | > loader_time: 0.0045 (0.0046215860704930285)  --> TIME: 2025-02-27 08:43:35 -- STEP: 1528/36938 -- GLOBAL_STEP: 1626800 | > loss_text_ce: 0.02129502408206463 (0.029876649935902902) | > loss_mel_ce: 0.08837890625 (nan) | > loss: 0.0004352140531409532 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13650276117924) | > loader_time: 0.0045 (0.004622585642400199)  --> TIME: 2025-02-27 08:43:44 -- STEP: 1578/36938 -- GLOBAL_STEP: 1626850 | > loss_text_ce: 0.024724245071411133 (0.02986573254310671) | > loss_mel_ce: 2.435267925262451 (nan) | > loss: 0.009761874563992023 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.1364656843915487) | > loader_time: 0.0045 (0.004621803080627664)  --> TIME: 2025-02-27 08:43:54 -- STEP: 1628/36938 -- GLOBAL_STEP: 1626900 | > loss_text_ce: 0.04050196707248688 (0.029882608462383636) | > loss_mel_ce: 3.187091588973999 (nan) | > loss: 0.012807912193238735 (nan) | > current_lr: 5e-06 | > step_time: 0.1435 (0.13639912280169386) | > loader_time: 0.0046 (0.004620818599729035)  --> TIME: 2025-02-27 08:44:03 -- STEP: 1678/36938 -- GLOBAL_STEP: 1626950 | > loss_text_ce: 0.026052281260490417 (0.02986988476011347) | > loss_mel_ce: 1.9407672882080078 (nan) | > loss: 0.007804839871823788 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13636422796669953) | > loader_time: 0.0046 (0.004620101511549455)  --> TIME: 2025-02-27 08:44:12 -- STEP: 1728/36938 -- GLOBAL_STEP: 1627000 | > loss_text_ce: 0.02601615898311138 (0.02983820584793446) | > loss_mel_ce: 2.81037974357605 (nan) | > loss: 0.011255539953708649 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.1363826737635666) | > loader_time: 0.0045 (0.0046195638952431775)  --> TIME: 2025-02-27 08:44:22 -- STEP: 1778/36938 -- GLOBAL_STEP: 1627050 | > loss_text_ce: 0.030493134632706642 (0.029873336965209805) | > loss_mel_ce: 3.061718702316284 (nan) | > loss: 0.012270682491362095 (nan) | > current_lr: 5e-06 | > step_time: 0.1266 (0.13661612877025112) | > loader_time: 0.0045 (0.0046209700330169905)  --> TIME: 2025-02-27 08:44:31 -- STEP: 1828/36938 -- GLOBAL_STEP: 1627100 | > loss_text_ce: 0.0333075076341629 (0.029849960357898386) | > loss_mel_ce: 2.0147039890289307 (nan) | > loss: 0.008127029985189438 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.1366690600727008) | > loader_time: 0.0047 (0.004620621319151021)  --> TIME: 2025-02-27 08:44:41 -- STEP: 1878/36938 -- GLOBAL_STEP: 1627150 | > loss_text_ce: 0.03096984326839447 (0.029841082851740672) | > loss_mel_ce: 1.8265008926391602 (nan) | > loss: 0.007370916195213795 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.1366964548159714) | > loader_time: 0.0045 (0.004620317453012672)  --> TIME: 2025-02-27 08:44:50 -- STEP: 1928/36938 -- GLOBAL_STEP: 1627200 | > loss_text_ce: 0.025628721341490746 (0.02988301807723715) | > loss_mel_ce: 2.4485645294189453 (nan) | > loss: 0.009818227961659431 (nan) | > current_lr: 5e-06 | > step_time: 0.1789 (0.13661924016920876) | > loader_time: 0.0047 (0.004619368255385712)  --> TIME: 2025-02-27 08:45:00 -- STEP: 1978/36938 -- GLOBAL_STEP: 1627250 | > loss_text_ce: 0.04872392863035202 (0.029913475346884287) | > loss_mel_ce: 2.5503015518188477 (nan) | > loss: 0.010313593782484531 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13672607926194197) | > loader_time: 0.0045 (0.004619192667267795)  --> TIME: 2025-02-27 08:45:09 -- STEP: 2028/36938 -- GLOBAL_STEP: 1627300 | > loss_text_ce: 0.03971990570425987 (0.029939822097404454) | > loss_mel_ce: 1.9759974479675293 (nan) | > loss: 0.007998878136277199 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13687325523214713) | > loader_time: 0.0046 (0.004619200789246562)  --> TIME: 2025-02-27 08:45:19 -- STEP: 2078/36938 -- GLOBAL_STEP: 1627350 | > loss_text_ce: 0.024330150336027145 (0.029901887336566586) | > loss_mel_ce: 1.9625320434570312 (nan) | > loss: 0.007884373888373375 (nan) | > current_lr: 5e-06 | > step_time: 0.1434 (0.13692907947433827) | > loader_time: 0.0046 (0.0046186995575127334)  --> TIME: 2025-02-27 08:45:28 -- STEP: 2128/36938 -- GLOBAL_STEP: 1627400 | > loss_text_ce: 0.028486067429184914 (0.029895831044558507) | > loss_mel_ce: 2.8509361743927 (nan) | > loss: 0.011426279321312904 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13697201876263884) | > loader_time: 0.0044 (0.004618420636743523)  --> TIME: 2025-02-27 08:45:38 -- STEP: 2178/36938 -- GLOBAL_STEP: 1627450 | > loss_text_ce: 0.027890905737876892 (0.029883280503148806) | > loss_mel_ce: 2.1382296085357666 (nan) | > loss: 0.008595717139542103 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.13702634050826418) | > loader_time: 0.0046 (0.0046185790343061224)  --> TIME: 2025-02-27 08:45:47 -- STEP: 2228/36938 -- GLOBAL_STEP: 1627500 | > loss_text_ce: 0.03583686426281929 (0.029912608152939276) | > loss_mel_ce: 1.6380798816680908 (nan) | > loss: 0.00664252694696188 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.1369862599261575) | > loader_time: 0.0045 (0.004617856370696466)  --> TIME: 2025-02-27 08:45:56 -- STEP: 2278/36938 -- GLOBAL_STEP: 1627550 | > loss_text_ce: 0.02911038137972355 (0.029915793938480508) | > loss_mel_ce: 1.8624804019927979 (nan) | > loss: 0.007506313268095255 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13690618143676164) | > loader_time: 0.0047 (0.004617069142235269)  --> TIME: 2025-02-27 08:46:06 -- STEP: 2328/36938 -- GLOBAL_STEP: 1627600 | > loss_text_ce: 0.02974879741668701 (0.029906601492832398) | > loss_mel_ce: 2.3747804164886475 (nan) | > loss: 0.009541782550513744 (nan) | > current_lr: 5e-06 | > step_time: 0.1519 (0.13715929427917084) | > loader_time: 0.0046 (0.004617624164037269)  --> TIME: 2025-02-27 08:46:16 -- STEP: 2378/36938 -- GLOBAL_STEP: 1627650 | > loss_text_ce: 0.028857626020908356 (0.029922068334255866) | > loss_mel_ce: 3.035195827484131 (nan) | > loss: 0.012158943340182304 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.1372751387555226) | > loader_time: 0.0045 (0.004617959336135635)  --> TIME: 2025-02-27 08:46:25 -- STEP: 2428/36938 -- GLOBAL_STEP: 1627700 | > loss_text_ce: 0.03576532378792763 (0.029922587916913213) | > loss_mel_ce: 2.4408161640167236 (nan) | > loss: 0.009827705100178719 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13727713485526122) | > loader_time: 0.0045 (0.004618389700743461)  --> TIME: 2025-02-27 08:46:35 -- STEP: 2478/36938 -- GLOBAL_STEP: 1627750 | > loss_text_ce: 0.025604482740163803 (0.029911145714658093) | > loss_mel_ce: 2.4527862071990967 (nan) | > loss: 0.00983488466590643 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13734463898764232) | > loader_time: 0.0046 (0.004619619555777352)  --> TIME: 2025-02-27 08:46:44 -- STEP: 2528/36938 -- GLOBAL_STEP: 1627800 | > loss_text_ce: 0.02196304313838482 (0.0298919471128493) | > loss_mel_ce: 2.5290114879608154 (nan) | > loss: 0.010122915729880333 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1373739980066882) | > loader_time: 0.0047 (0.004620007227493227)  --> TIME: 2025-02-27 08:46:53 -- STEP: 2578/36938 -- GLOBAL_STEP: 1627850 | > loss_text_ce: 0.024866389110684395 (0.029900499530854767) | > loss_mel_ce: 3.1434733867645264 (nan) | > loss: 0.012572777457535267 (nan) | > current_lr: 5e-06 | > step_time: 0.185 (0.13733589408747296) | > loader_time: 0.0047 (0.004619193132576586)  --> TIME: 2025-02-27 08:47:02 -- STEP: 2628/36938 -- GLOBAL_STEP: 1627900 | > loss_text_ce: 0.02940479852259159 (0.029879659731997122) | > loss_mel_ce: 3.028782844543457 (nan) | > loss: 0.012135665863752365 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13721597403333297) | > loader_time: 0.0045 (0.0046186097862150645)  --> TIME: 2025-02-27 08:47:12 -- STEP: 2678/36938 -- GLOBAL_STEP: 1627950 | > loss_text_ce: 0.030797390267252922 (0.029883491461874) | > loss_mel_ce: 3.188364028930664 (nan) | > loss: 0.01277445163577795 (nan) | > current_lr: 5e-06 | > step_time: 0.1833 (0.13721026378927942) | > loader_time: 0.0048 (0.004618831051206117)  --> TIME: 2025-02-27 08:47:21 -- STEP: 2728/36938 -- GLOBAL_STEP: 1628000 | > loss_text_ce: 0.025520378723740578 (0.029879394165053145) | > loss_mel_ce: 2.338923215866089 (nan) | > loss: 0.00938271265476942 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13725701114299418) | > loader_time: 0.0047 (0.0046192845466199745)  --> TIME: 2025-02-27 08:47:30 -- STEP: 2778/36938 -- GLOBAL_STEP: 1628050 | > loss_text_ce: 0.03481164574623108 (0.02988207603763196) | > loss_mel_ce: 1.8303015232086182 (nan) | > loss: 0.0074012428522109985 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13723980093105365) | > loader_time: 0.0045 (0.004618986207345468)  --> TIME: 2025-02-27 08:47:40 -- STEP: 2828/36938 -- GLOBAL_STEP: 1628100 | > loss_text_ce: 0.03357382118701935 (0.029888911126807194) | > loss_mel_ce: 2.333705425262451 (nan) | > loss: 0.0093939658254385 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.1371779441833502) | > loader_time: 0.0045 (0.004619634809048872)  --> TIME: 2025-02-27 08:47:49 -- STEP: 2878/36938 -- GLOBAL_STEP: 1628150 | > loss_text_ce: 0.03661743924021721 (0.02988139608447117) | > loss_mel_ce: 2.935908555984497 (nan) | > loss: 0.011795738711953163 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1371110521976617) | > loader_time: 0.0045 (0.0046194272376001416)  --> TIME: 2025-02-27 08:47:58 -- STEP: 2928/36938 -- GLOBAL_STEP: 1628200 | > loss_text_ce: 0.029406649991869926 (0.02986791142777163) | > loss_mel_ce: 3.1346566677093506 (nan) | > loss: 0.012555806897580624 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13712142912742267) | > loader_time: 0.0045 (0.004620541721745257)  --> TIME: 2025-02-27 08:48:08 -- STEP: 2978/36938 -- GLOBAL_STEP: 1628250 | > loss_text_ce: 0.04235006868839264 (0.0298478570314846) | > loss_mel_ce: 3.3557941913604736 (nan) | > loss: 0.013484700582921505 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13714711797405577) | > loader_time: 0.0045 (0.004620332378441244)  --> TIME: 2025-02-27 08:48:17 -- STEP: 3028/36938 -- GLOBAL_STEP: 1628300 | > loss_text_ce: 0.03092733398079872 (0.029837775326646433) | > loss_mel_ce: 2.891045570373535 (nan) | > loss: 0.011595131829380989 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13715048935485857) | > loader_time: 0.0046 (0.004620238607121896)  --> TIME: 2025-02-27 08:48:26 -- STEP: 3078/36938 -- GLOBAL_STEP: 1628350 | > loss_text_ce: 0.028878433629870415 (0.029868640265348256) | > loss_mel_ce: 2.6654481887817383 (nan) | > loss: 0.010691773146390915 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13716926909329658) | > loader_time: 0.0046 (0.004620767553724814)  --> TIME: 2025-02-27 08:48:36 -- STEP: 3128/36938 -- GLOBAL_STEP: 1628400 | > loss_text_ce: 0.0203363299369812 (0.029863997123322075) | > loss_mel_ce: 1.544231653213501 (nan) | > loss: 0.006208603736013174 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13713190950396162) | > loader_time: 0.0045 (0.004620743910674853)  --> TIME: 2025-02-27 08:48:45 -- STEP: 3178/36938 -- GLOBAL_STEP: 1628450 | > loss_text_ce: 0.022085411474108696 (0.029857706391827547) | > loss_mel_ce: 1.4150501489639282 (nan) | > loss: 0.005702919326722622 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1372143077430225) | > loader_time: 0.0047 (0.004622121139319909)  --> TIME: 2025-02-27 08:48:55 -- STEP: 3228/36938 -- GLOBAL_STEP: 1628500 | > loss_text_ce: 0.029761599376797676 (0.02986241399891596) | > loss_mel_ce: 2.337646484375 (nan) | > loss: 0.009394477121531963 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13725116904340312) | > loader_time: 0.0047 (0.004622455866363414)  --> TIME: 2025-02-27 08:49:04 -- STEP: 3278/36938 -- GLOBAL_STEP: 1628550 | > loss_text_ce: 0.02919517457485199 (0.02986431377769412) | > loss_mel_ce: 1.9486044645309448 (nan) | > loss: 0.007848411798477173 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1372001629329591) | > loader_time: 0.0045 (0.004622018213603758)  --> TIME: 2025-02-27 08:49:13 -- STEP: 3328/36938 -- GLOBAL_STEP: 1628600 | > loss_text_ce: 0.03132577985525131 (0.02985673091503626) | > loss_mel_ce: 2.4462013244628906 (nan) | > loss: 0.009831457398831844 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13712658417912696) | > loader_time: 0.0045 (0.004620989498037556)  --> TIME: 2025-02-27 08:49:22 -- STEP: 3378/36938 -- GLOBAL_STEP: 1628650 | > loss_text_ce: 0.03188932687044144 (0.029885568858255) | > loss_mel_ce: 2.976727247238159 (nan) | > loss: 0.011938955634832382 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.13712984990480812) | > loader_time: 0.0047 (0.004621267318725584)  --> TIME: 2025-02-27 08:49:32 -- STEP: 3428/36938 -- GLOBAL_STEP: 1628700 | > loss_text_ce: 0.03019225038588047 (0.029885296683889992) | > loss_mel_ce: 1.5975303649902344 (nan) | > loss: 0.006459217052906752 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1370950904582465) | > loader_time: 0.0045 (0.004621152838859424)  --> TIME: 2025-02-27 08:49:41 -- STEP: 3478/36938 -- GLOBAL_STEP: 1628750 | > loss_text_ce: 0.025639807805418968 (0.029907706639873504) | > loss_mel_ce: 2.9170331954956055 (nan) | > loss: 0.011677274480462074 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13716124000735858) | > loader_time: 0.0046 (0.004621549335411732)  --> TIME: 2025-02-27 08:49:51 -- STEP: 3528/36938 -- GLOBAL_STEP: 1628800 | > loss_text_ce: 0.0317106619477272 (0.02989874304973894) | > loss_mel_ce: 1.1214146614074707 (nan) | > loss: 0.0045758942142128944 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13713778511466973) | > loader_time: 0.005 (0.004621615958592246)  --> TIME: 2025-02-27 08:50:00 -- STEP: 3578/36938 -- GLOBAL_STEP: 1628850 | > loss_text_ce: 0.03307771310210228 (0.029906275060705638) | > loss_mel_ce: 2.5116920471191406 (nan) | > loss: 0.010098293423652649 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13716229458239118) | > loader_time: 0.0066 (0.004623628515581499)  --> TIME: 2025-02-27 08:50:09 -- STEP: 3628/36938 -- GLOBAL_STEP: 1628900 | > loss_text_ce: 0.031415656208992004 (0.029905447137363404) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.13716762351096096) | > loader_time: 0.005 (0.004625389896653583)  --> TIME: 2025-02-27 08:50:19 -- STEP: 3678/36938 -- GLOBAL_STEP: 1628950 | > loss_text_ce: 0.033783696591854095 (0.02990059248503964) | > loss_mel_ce: 0.0097808837890625 (nan) | > loss: 0.00017287532682530582 (nan) | > current_lr: 5e-06 | > step_time: 0.1533 (0.1371233785457112) | > loader_time: 0.0047 (0.0046271912879176326)  --> TIME: 2025-02-27 08:50:28 -- STEP: 3728/36938 -- GLOBAL_STEP: 1629000 | > loss_text_ce: 0.033428896218538284 (0.02989881784046097) | > loss_mel_ce: 1.8604768514633179 (nan) | > loss: 0.007515499368309975 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13710494652838254) | > loader_time: 0.0047 (0.004627610097115642)  --> TIME: 2025-02-27 08:50:37 -- STEP: 3778/36938 -- GLOBAL_STEP: 1629050 | > loss_text_ce: 0.03863814100623131 (0.029904585453578606) | > loss_mel_ce: 0.8369320034980774 (nan) | > loss: 0.003474484896287322 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.1370589458607956) | > loader_time: 0.0047 (0.004627918868925659)  --> TIME: 2025-02-27 08:50:46 -- STEP: 3828/36938 -- GLOBAL_STEP: 1629100 | > loss_text_ce: 0.029363879933953285 (0.029895190531636084) | > loss_mel_ce: 2.16117262840271 (nan) | > loss: 0.008692605420947075 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.13704817266424474) | > loader_time: 0.0047 (0.004628304030080572)  --> TIME: 2025-02-27 08:50:55 -- STEP: 3878/36938 -- GLOBAL_STEP: 1629150 | > loss_text_ce: 0.0377490408718586 (0.02990064170087153) | > loss_mel_ce: 3.305844783782959 (nan) | > loss: 0.013268230482935905 (nan) | > current_lr: 5e-06 | > step_time: 0.1593 (0.13706567817400855) | > loader_time: 0.0046 (0.004628974731095122)  --> TIME: 2025-02-27 08:51:05 -- STEP: 3928/36938 -- GLOBAL_STEP: 1629200 | > loss_text_ce: 0.03496715798974037 (0.02989089649889994) | > loss_mel_ce: 0.9757934808731079 (nan) | > loss: 0.004010955337435007 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.13713789799548556) | > loader_time: 0.0046 (0.0046294558558104704)  --> TIME: 2025-02-27 08:51:14 -- STEP: 3978/36938 -- GLOBAL_STEP: 1629250 | > loss_text_ce: 0.028046030551195145 (0.029884540568732847) | > loss_mel_ce: 1.2284207344055176 (nan) | > loss: 0.004985979292541742 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13707057916680648) | > loader_time: 0.0045 (0.004629414065990571)  --> TIME: 2025-02-27 08:51:23 -- STEP: 4028/36938 -- GLOBAL_STEP: 1629300 | > loss_text_ce: 0.034967709332704544 (0.029900710077751927) | > loss_mel_ce: 3.04587721824646 (nan) | > loss: 0.012225575745105743 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.1370400373251469) | > loader_time: 0.0047 (0.00463007868460889)  --> TIME: 2025-02-27 08:51:33 -- STEP: 4078/36938 -- GLOBAL_STEP: 1629350 | > loss_text_ce: 0.032035376876592636 (0.029906885297521363) | > loss_mel_ce: 2.2148616313934326 (nan) | > loss: 0.008916258811950684 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13701858925082838) | > loader_time: 0.0047 (0.00463188203191453)  --> TIME: 2025-02-27 08:51:42 -- STEP: 4128/36938 -- GLOBAL_STEP: 1629400 | > loss_text_ce: 0.03657990321516991 (0.02990404675841465) | > loss_mel_ce: 2.998873710632324 (nan) | > loss: 0.012045451439917088 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13701901405818795) | > loader_time: 0.0046 (0.004632408131462657)  --> TIME: 2025-02-27 08:51:51 -- STEP: 4178/36938 -- GLOBAL_STEP: 1629450 | > loss_text_ce: 0.02768241986632347 (0.02989124644482165) | > loss_mel_ce: 2.2104990482330322 (nan) | > loss: 0.008881672285497189 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13700358584601757) | > loader_time: 0.0045 (0.004632367192985566)  --> TIME: 2025-02-27 08:52:01 -- STEP: 4228/36938 -- GLOBAL_STEP: 1629500 | > loss_text_ce: 0.02643989585340023 (0.02989717052208459) | > loss_mel_ce: 2.8755381107330322 (nan) | > loss: 0.011515786871314049 (nan) | > current_lr: 5e-06 | > step_time: 0.1667 (0.1370105731002326) | > loader_time: 0.0047 (0.0046323277302925125)  --> TIME: 2025-02-27 08:52:10 -- STEP: 4278/36938 -- GLOBAL_STEP: 1629550 | > loss_text_ce: 0.04146314412355423 (0.029895595787441363) | > loss_mel_ce: 2.7447099685668945 (nan) | > loss: 0.011056243441998959 (nan) | > current_lr: 5e-06 | > step_time: 0.1264 (0.13699823525978597) | > loader_time: 0.0047 (0.004632749586475399)  --> TIME: 2025-02-27 08:52:19 -- STEP: 4328/36938 -- GLOBAL_STEP: 1629600 | > loss_text_ce: 0.03077133186161518 (0.029880857996856913) | > loss_mel_ce: 3.177276372909546 (nan) | > loss: 0.012730348855257034 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13703500310065878) | > loader_time: 0.0046 (0.004632906420173571)  --> TIME: 2025-02-27 08:52:29 -- STEP: 4378/36938 -- GLOBAL_STEP: 1629650 | > loss_text_ce: 0.031232750043272972 (0.029884952905129637) | > loss_mel_ce: 2.512451171875 (nan) | > loss: 0.010093985125422478 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13700519527349997) | > loader_time: 0.0046 (0.004633531008424578)  --> TIME: 2025-02-27 08:52:38 -- STEP: 4428/36938 -- GLOBAL_STEP: 1629700 | > loss_text_ce: 0.0244727935642004 (0.029884603768086538) | > loss_mel_ce: 2.8770833015441895 (nan) | > loss: 0.011514111422002316 (nan) | > current_lr: 5e-06 | > step_time: 0.1915 (0.1369588740688253) | > loader_time: 0.0049 (0.0046342133183466345)  --> TIME: 2025-02-27 08:52:47 -- STEP: 4478/36938 -- GLOBAL_STEP: 1629750 | > loss_text_ce: 0.029644034802913666 (0.029880645896098897) | > loss_mel_ce: 2.7951414585113525 (nan) | > loss: 0.011209466494619846 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13692523550380362) | > loader_time: 0.0046 (0.004634686589720202)  --> TIME: 2025-02-27 08:52:56 -- STEP: 4528/36938 -- GLOBAL_STEP: 1629800 | > loss_text_ce: 0.03798219934105873 (0.029872502894580936) | > loss_mel_ce: 2.25112247467041 (nan) | > loss: 0.00908374972641468 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.1368950921844685) | > loader_time: 0.005 (0.004634655985731111)  --> TIME: 2025-02-27 08:53:05 -- STEP: 4578/36938 -- GLOBAL_STEP: 1629850 | > loss_text_ce: 0.03559030219912529 (0.02987731066910821) | > loss_mel_ce: 2.2294228076934814 (nan) | > loss: 0.008988148532807827 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.1368261285617714) | > loader_time: 0.0046 (0.00463442887839087)  --> TIME: 2025-02-27 08:53:14 -- STEP: 4628/36938 -- GLOBAL_STEP: 1629900 | > loss_text_ce: 0.028809262439608574 (0.029881626332050725) | > loss_mel_ce: 2.095564126968384 (nan) | > loss: 0.008430053479969501 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13676431751910698) | > loader_time: 0.0046 (0.004634532365906097)  --> TIME: 2025-02-27 08:53:23 -- STEP: 4678/36938 -- GLOBAL_STEP: 1629950 | > loss_text_ce: 0.02614169381558895 (0.029878095429937756) | > loss_mel_ce: 3.1586251258850098 (nan) | > loss: 0.012637964449822903 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13676526450662402) | > loader_time: 0.0046 (0.0046355026611256305)  --> TIME: 2025-02-27 08:53:33 -- STEP: 4728/36938 -- GLOBAL_STEP: 1630000 | > loss_text_ce: 0.02992590144276619 (0.029890760458401496) | > loss_mel_ce: 3.191028118133545 (nan) | > loss: 0.012781564146280289 (nan) | > current_lr: 5e-06 | > step_time: 0.1835 (0.1367966000198719) | > loader_time: 0.0049 (0.004636156175745714) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1630000.pth  --> TIME: 2025-02-27 08:53:48 -- STEP: 4778/36938 -- GLOBAL_STEP: 1630050 | > loss_text_ce: 0.028827611356973648 (0.029893933532739277) | > loss_mel_ce: 1.7529678344726562 (nan) | > loss: 0.007070617284625769 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.1367710216517432) | > loader_time: 0.0046 (0.0046374313798395765)  --> TIME: 2025-02-27 08:53:57 -- STEP: 4828/36938 -- GLOBAL_STEP: 1630100 | > loss_text_ce: 0.02136133424937725 (0.02989375483379302) | > loss_mel_ce: 0.0753580704331398 (nan) | > loss: 0.00038380720070563257 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13675738625419828) | > loader_time: 0.0046 (0.004637579929759501)  --> TIME: 2025-02-27 08:54:07 -- STEP: 4878/36938 -- GLOBAL_STEP: 1630150 | > loss_text_ce: 0.026713775470852852 (0.029894893793557212) | > loss_mel_ce: 2.910907506942749 (nan) | > loss: 0.01165722869336605 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13672872399052208) | > loader_time: 0.0045 (0.004637982719205543)  --> TIME: 2025-02-27 08:54:16 -- STEP: 4928/36938 -- GLOBAL_STEP: 1630200 | > loss_text_ce: 0.02920648828148842 (0.029902730229807346) | > loss_mel_ce: 2.0825068950653076 (nan) | > loss: 0.008379815146327019 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.1367890610323331) | > loader_time: 0.0047 (0.004638095858035142)  --> TIME: 2025-02-27 08:54:25 -- STEP: 4978/36938 -- GLOBAL_STEP: 1630250 | > loss_text_ce: 0.020285354927182198 (0.029892277500961482) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.1367855901074445) | > loader_time: 0.0048 (0.004638300070373724)  --> TIME: 2025-02-27 08:54:34 -- STEP: 5028/36938 -- GLOBAL_STEP: 1630300 | > loss_text_ce: 0.027401648461818695 (0.029892334262299588) | > loss_mel_ce: 2.6417315006256104 (nan) | > loss: 0.010591799393296242 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13673266728150135) | > loader_time: 0.0047 (0.004637912330688127)  --> TIME: 2025-02-27 08:54:44 -- STEP: 5078/36938 -- GLOBAL_STEP: 1630350 | > loss_text_ce: 0.03202774003148079 (0.02988295226032153) | > loss_mel_ce: 3.250396251678467 (nan) | > loss: 0.013025492429733276 (nan) | > current_lr: 5e-06 | > step_time: 0.1867 (0.13671955389974688) | > loader_time: 0.0048 (0.0046377569354862135)  --> TIME: 2025-02-27 08:54:53 -- STEP: 5128/36938 -- GLOBAL_STEP: 1630400 | > loss_text_ce: 0.03082508035004139 (0.02987792224388253) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.13669555230929287) | > loader_time: 0.0045 (0.004637352389961979)  --> TIME: 2025-02-27 08:55:02 -- STEP: 5178/36938 -- GLOBAL_STEP: 1630450 | > loss_text_ce: 0.026667900383472443 (0.029885356142622693) | > loss_mel_ce: 2.6825766563415527 (nan) | > loss: 0.010750970803201199 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.1366750956410474) | > loader_time: 0.0046 (0.004637296018383118)  --> TIME: 2025-02-27 08:55:11 -- STEP: 5228/36938 -- GLOBAL_STEP: 1630500 | > loss_text_ce: 0.028463300317525864 (0.029888501785664913) | > loss_mel_ce: 2.5218441486358643 (nan) | > loss: 0.010120268911123276 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13666344728739885) | > loader_time: 0.0046 (0.004636992638404791)  --> TIME: 2025-02-27 08:55:20 -- STEP: 5278/36938 -- GLOBAL_STEP: 1630550 | > loss_text_ce: 0.03776531293988228 (0.029884230051251067) | > loss_mel_ce: 0.005745999980717897 (nan) | > loss: 0.00017266394570469856 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13660857093835677) | > loader_time: 0.0046 (0.0046366752662094805)  --> TIME: 2025-02-27 08:55:30 -- STEP: 5328/36938 -- GLOBAL_STEP: 1630600 | > loss_text_ce: 0.02799965627491474 (0.029878043780517945) | > loss_mel_ce: 1.7314889430999756 (nan) | > loss: 0.006982097867876291 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13663623694543078) | > loader_time: 0.0045 (0.004636513086052614)  --> TIME: 2025-02-27 08:55:39 -- STEP: 5378/36938 -- GLOBAL_STEP: 1630650 | > loss_text_ce: 0.031396687030792236 (0.02987550421636628) | > loss_mel_ce: 2.2993264198303223 (nan) | > loss: 0.009248901158571243 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.1366379505287282) | > loader_time: 0.0044 (0.004636277448470451)  --> TIME: 2025-02-27 08:55:48 -- STEP: 5428/36938 -- GLOBAL_STEP: 1630700 | > loss_text_ce: 0.03057735413312912 (0.029867561116170168) | > loss_mel_ce: 2.482419729232788 (nan) | > loss: 0.00997221190482378 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.1366619545919724) | > loader_time: 0.0048 (0.004636100881136586)  --> TIME: 2025-02-27 08:55:58 -- STEP: 5478/36938 -- GLOBAL_STEP: 1630750 | > loss_text_ce: 0.04309511557221413 (0.02987845978793471) | > loss_mel_ce: 2.234524965286255 (nan) | > loss: 0.009038175456225872 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1366608472756556) | > loader_time: 0.0046 (0.004635773596427603)  --> TIME: 2025-02-27 08:56:07 -- STEP: 5528/36938 -- GLOBAL_STEP: 1630800 | > loss_text_ce: 0.02053580805659294 (0.029874933796699595) | > loss_mel_ce: 2.5223681926727295 (nan) | > loss: 0.010090889409184456 (nan) | > current_lr: 5e-06 | > step_time: 0.1632 (0.1366703134628529) | > loader_time: 0.0047 (0.004635412898318974)  --> TIME: 2025-02-27 08:56:16 -- STEP: 5578/36938 -- GLOBAL_STEP: 1630850 | > loss_text_ce: 0.01685575395822525 (0.029878693153345726) | > loss_mel_ce: 1.6540799140930176 (nan) | > loss: 0.006630697287619114 (nan) | > current_lr: 5e-06 | > step_time: 0.1082 (0.13667715095883917) | > loader_time: 0.0044 (0.004635063838514202)  --> TIME: 2025-02-27 08:56:26 -- STEP: 5628/36938 -- GLOBAL_STEP: 1630900 | > loss_text_ce: 0.02884657308459282 (0.029874324586852068) | > loss_mel_ce: 2.6365084648132324 (nan) | > loss: 0.010576806031167507 (nan) | > current_lr: 5e-06 | > step_time: 0.182 (0.13669406651772822) | > loader_time: 0.0048 (0.0046347286062369255)  --> TIME: 2025-02-27 08:56:35 -- STEP: 5678/36938 -- GLOBAL_STEP: 1630950 | > loss_text_ce: 0.0331207700073719 (0.029873735297391898) | > loss_mel_ce: 2.5173180103302 (nan) | > loss: 0.010120789520442486 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13668347715113846) | > loader_time: 0.0048 (0.004634466167905451)  --> TIME: 2025-02-27 08:56:45 -- STEP: 5728/36938 -- GLOBAL_STEP: 1631000 | > loss_text_ce: 0.03642683103680611 (0.02986937146069672) | > loss_mel_ce: 1.3797688484191895 (nan) | > loss: 0.005619824398308992 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13672863384032463) | > loader_time: 0.0045 (0.004634608686303288)  --> TIME: 2025-02-27 08:56:55 -- STEP: 5778/36938 -- GLOBAL_STEP: 1631050 | > loss_text_ce: 0.02791188657283783 (0.029859111651822708) | > loss_mel_ce: 2.9806394577026367 (nan) | > loss: 0.011938696727156639 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13679902910072192) | > loader_time: 0.0046 (0.004634617603786587)  --> TIME: 2025-02-27 08:57:04 -- STEP: 5828/36938 -- GLOBAL_STEP: 1631100 | > loss_text_ce: 0.03715457767248154 (0.029859379735869895) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13677134027854243) | > loader_time: 0.0045 (0.004634195635722929)  --> TIME: 2025-02-27 08:57:13 -- STEP: 5878/36938 -- GLOBAL_STEP: 1631150 | > loss_text_ce: 0.0292812492698431 (0.02986052871111992) | > loss_mel_ce: 2.678851842880249 (nan) | > loss: 0.01074656005948782 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13682418755423978) | > loader_time: 0.0047 (0.004634721784536502)  --> TIME: 2025-02-27 08:57:23 -- STEP: 5928/36938 -- GLOBAL_STEP: 1631200 | > loss_text_ce: 0.032115355134010315 (0.029858544393233308) | > loss_mel_ce: 2.0523264408111572 (nan) | > loss: 0.0082715954631567 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13681380267728918) | > loader_time: 0.0045 (0.004634316110739653)  --> TIME: 2025-02-27 08:57:32 -- STEP: 5978/36938 -- GLOBAL_STEP: 1631250 | > loss_text_ce: 0.029273400083184242 (0.029864977974885208) | > loss_mel_ce: 2.6066973209381104 (nan) | > loss: 0.0104602025821805 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13686168951587874) | > loader_time: 0.0047 (0.004634160666850219)  --> TIME: 2025-02-27 08:57:42 -- STEP: 6028/36938 -- GLOBAL_STEP: 1631300 | > loss_text_ce: 0.0339217334985733 (0.029865658374898154) | > loss_mel_ce: 1.6663308143615723 (nan) | > loss: 0.006747034378349781 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13689083897205323) | > loader_time: 0.0047 (0.0046346185613959905)  --> TIME: 2025-02-27 08:57:51 -- STEP: 6078/36938 -- GLOBAL_STEP: 1631350 | > loss_text_ce: 0.026358896866440773 (0.029862504343291688) | > loss_mel_ce: 2.7224016189575195 (nan) | > loss: 0.01090778037905693 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13689949222990302) | > loader_time: 0.0045 (0.004634738243030213)  --> TIME: 2025-02-27 08:58:01 -- STEP: 6128/36938 -- GLOBAL_STEP: 1631400 | > loss_text_ce: 0.03424526005983353 (0.02987387609282355) | > loss_mel_ce: 0.18609333038330078 (nan) | > loss: 0.0008743595099076629 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13689965643864146) | > loader_time: 0.0046 (0.004634835234510048)  --> TIME: 2025-02-27 08:58:10 -- STEP: 6178/36938 -- GLOBAL_STEP: 1631450 | > loss_text_ce: 0.03418999910354614 (0.02987148032500175) | > loss_mel_ce: 2.753065586090088 (nan) | > loss: 0.011060538701713085 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13689565623837013) | > loader_time: 0.0046 (0.004635334709540421)  --> TIME: 2025-02-27 08:58:19 -- STEP: 6228/36938 -- GLOBAL_STEP: 1631500 | > loss_text_ce: 0.02831159718334675 (0.029885967774032546) | > loss_mel_ce: 3.2154202461242676 (nan) | > loss: 0.012871951796114445 (nan) | > current_lr: 5e-06 | > step_time: 0.1984 (0.13692084075398608) | > loader_time: 0.0048 (0.004635980899470228)  --> TIME: 2025-02-27 08:58:29 -- STEP: 6278/36938 -- GLOBAL_STEP: 1631550 | > loss_text_ce: 0.03958764299750328 (0.02989237235929353) | > loss_mel_ce: 2.3016815185546875 (nan) | > loss: 0.009290752001106739 (nan) | > current_lr: 5e-06 | > step_time: 0.1793 (0.13690851026372866) | > loader_time: 0.0049 (0.004636030054350641)  --> TIME: 2025-02-27 08:58:38 -- STEP: 6328/36938 -- GLOBAL_STEP: 1631600 | > loss_text_ce: 0.032191675156354904 (0.029894451502055358) | > loss_mel_ce: 1.2416635751724243 (nan) | > loss: 0.0050549814477562904 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13684680357142207) | > loader_time: 0.0048 (0.004635766318112642)  --> TIME: 2025-02-27 08:58:47 -- STEP: 6378/36938 -- GLOBAL_STEP: 1631650 | > loss_text_ce: 0.03433658555150032 (0.02990705270065224) | > loss_mel_ce: 3.1804535388946533 (nan) | > loss: 0.012757103890180588 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.1368476394651534) | > loader_time: 0.0048 (0.004635621552633309)  --> TIME: 2025-02-27 08:58:56 -- STEP: 6428/36938 -- GLOBAL_STEP: 1631700 | > loss_text_ce: 0.023647384718060493 (0.029911561406828352) | > loss_mel_ce: 3.3570313453674316 (nan) | > loss: 0.013415392488241196 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.13687783123620562) | > loader_time: 0.0046 (0.00463588622021392)  --> TIME: 2025-02-27 08:59:06 -- STEP: 6478/36938 -- GLOBAL_STEP: 1631750 | > loss_text_ce: 0.02814485691487789 (0.02990475280893844) | > loss_mel_ce: 2.671875 (nan) | > loss: 0.010714365169405937 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13687616658453722) | > loader_time: 0.0046 (0.004636028512976487)  --> TIME: 2025-02-27 08:59:15 -- STEP: 6528/36938 -- GLOBAL_STEP: 1631800 | > loss_text_ce: 0.03149276599287987 (0.029905665274033228) | > loss_mel_ce: 2.826749801635742 (nan) | > loss: 0.011342233046889305 (nan) | > current_lr: 5e-06 | > step_time: 0.196 (0.13686752885434036) | > loader_time: 0.0049 (0.004636383962397464)  --> TIME: 2025-02-27 08:59:24 -- STEP: 6578/36938 -- GLOBAL_STEP: 1631850 | > loss_text_ce: 0.026628119871020317 (0.0299008198018877) | > loss_mel_ce: 2.2582154273986816 (nan) | > loss: 0.009066839702427387 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13687409412973303) | > loader_time: 0.0046 (0.004636873188407329)  --> TIME: 2025-02-27 08:59:34 -- STEP: 6628/36938 -- GLOBAL_STEP: 1631900 | > loss_text_ce: 0.020499128848314285 (0.029902538840178967) | > loss_mel_ce: 2.4608840942382812 (nan) | > loss: 0.009846759960055351 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13684956978514246) | > loader_time: 0.0046 (0.0046369846714643505)  --> TIME: 2025-02-27 08:59:43 -- STEP: 6678/36938 -- GLOBAL_STEP: 1631950 | > loss_text_ce: 0.042913876473903656 (0.02991574900270827) | > loss_mel_ce: 1.5024219751358032 (nan) | > loss: 0.006132285576313734 (nan) | > current_lr: 5e-06 | > step_time: 0.1077 (0.13683395608523327) | > loader_time: 0.0045 (0.004637113907052271)  --> TIME: 2025-02-27 08:59:52 -- STEP: 6728/36938 -- GLOBAL_STEP: 1632000 | > loss_text_ce: 0.03061949647963047 (0.02991283025921186) | > loss_mel_ce: 2.9213438034057617 (nan) | > loss: 0.011714139953255653 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.1368234898190727) | > loader_time: 0.0046 (0.004637222440290945)  --> TIME: 2025-02-27 09:00:01 -- STEP: 6778/36938 -- GLOBAL_STEP: 1632050 | > loss_text_ce: 0.025230256840586662 (0.02990815879654235) | > loss_mel_ce: 1.2761526107788086 (nan) | > loss: 0.005164218135178089 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13680107004700956) | > loader_time: 0.0047 (0.004637441300262406)  --> TIME: 2025-02-27 09:00:11 -- STEP: 6828/36938 -- GLOBAL_STEP: 1632100 | > loss_text_ce: 0.03555828705430031 (0.02991224108006059) | > loss_mel_ce: 2.737506628036499 (nan) | > loss: 0.01100422628223896 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13681308327184977) | > loader_time: 0.0045 (0.004638058020486127)  --> TIME: 2025-02-27 09:00:20 -- STEP: 6878/36938 -- GLOBAL_STEP: 1632150 | > loss_text_ce: 0.03985513746738434 (0.02992369058215896) | > loss_mel_ce: 3.1739230155944824 (nan) | > loss: 0.01275308895856142 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.1368001656464353) | > loader_time: 0.0046 (0.0046379408818339315)  --> TIME: 2025-02-27 09:00:29 -- STEP: 6928/36938 -- GLOBAL_STEP: 1632200 | > loss_text_ce: 0.03396905958652496 (0.02992595098198899) | > loss_mel_ce: 3.4820499420166016 (nan) | > loss: 0.013952457346022129 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.13675026843371663) | > loader_time: 0.0047 (0.004637910539228408)  --> TIME: 2025-02-27 09:00:38 -- STEP: 6978/36938 -- GLOBAL_STEP: 1632250 | > loss_text_ce: 0.029426133260130882 (0.02992170209172653) | > loss_mel_ce: 1.892657995223999 (nan) | > loss: 0.007627318147569895 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13677588955110348) | > loader_time: 0.0046 (0.004637948897480092)  --> TIME: 2025-02-27 09:00:48 -- STEP: 7028/36938 -- GLOBAL_STEP: 1632300 | > loss_text_ce: 0.034157853573560715 (0.02992000989118982) | > loss_mel_ce: 2.983445405960083 (nan) | > loss: 0.011974615976214409 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13679157486678423) | > loader_time: 0.0046 (0.004637983792467267)  --> TIME: 2025-02-27 09:00:57 -- STEP: 7078/36938 -- GLOBAL_STEP: 1632350 | > loss_text_ce: 0.028202546760439873 (0.029918748207559256) | > loss_mel_ce: 3.3327760696411133 (nan) | > loss: 0.013337217271327972 (nan) | > current_lr: 5e-06 | > step_time: 0.1952 (0.13680360185187188) | > loader_time: 0.0048 (0.004637873351287608)  --> TIME: 2025-02-27 09:01:06 -- STEP: 7128/36938 -- GLOBAL_STEP: 1632400 | > loss_text_ce: 0.03523897007107735 (0.02992039399440259) | > loss_mel_ce: 2.3374383449554443 (nan) | > loss: 0.00941538717597723 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13674249870474786) | > loader_time: 0.0047 (0.0046378703229756285)  --> TIME: 2025-02-27 09:01:15 -- STEP: 7178/36938 -- GLOBAL_STEP: 1632450 | > loss_text_ce: 0.03474953770637512 (0.02991414839625265) | > loss_mel_ce: 3.3650918006896973 (nan) | > loss: 0.01349143497645855 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13674938217050442) | > loader_time: 0.0048 (0.004637792569476918)  --> TIME: 2025-02-27 09:01:24 -- STEP: 7228/36938 -- GLOBAL_STEP: 1632500 | > loss_text_ce: 0.034258559346199036 (0.029924973899534146) | > loss_mel_ce: 3.0263736248016357 (nan) | > loss: 0.01214536651968956 (nan) | > current_lr: 5e-06 | > step_time: 0.1758 (0.13670626294236862) | > loader_time: 0.0051 (0.004637782489257271)  --> TIME: 2025-02-27 09:01:34 -- STEP: 7278/36938 -- GLOBAL_STEP: 1632550 | > loss_text_ce: 0.03488516807556152 (0.029923597514702558) | > loss_mel_ce: 2.9291398525238037 (nan) | > loss: 0.011762004345655441 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13673006292554374) | > loader_time: 0.0045 (0.0046377724820226215)  --> TIME: 2025-02-27 09:01:43 -- STEP: 7328/36938 -- GLOBAL_STEP: 1632600 | > loss_text_ce: 0.030686456710100174 (0.02992454570977121) | > loss_mel_ce: 1.6710141897201538 (nan) | > loss: 0.006752781104296446 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13670648862180762) | > loader_time: 0.0046 (0.004637629769775007)  --> TIME: 2025-02-27 09:01:52 -- STEP: 7378/36938 -- GLOBAL_STEP: 1632650 | > loss_text_ce: 0.03641591593623161 (0.029923191120192674) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1366868747156404) | > loader_time: 0.0045 (0.0046376233567516625)  --> TIME: 2025-02-27 09:02:01 -- STEP: 7428/36938 -- GLOBAL_STEP: 1632700 | > loss_text_ce: 0.03437027707695961 (0.029928162321410712) | > loss_mel_ce: 2.8851499557495117 (nan) | > loss: 0.011585397645831108 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13670768102908837) | > loader_time: 0.0046 (0.004637745194494162)  --> TIME: 2025-02-27 09:02:10 -- STEP: 7478/36938 -- GLOBAL_STEP: 1632750 | > loss_text_ce: 0.01274199690669775 (0.029917694738644866) | > loss_mel_ce: 3.34375 (nan) | > loss: 0.013319413177669048 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.13667078453705725) | > loader_time: 0.0046 (0.004637725533696842)  --> TIME: 2025-02-27 09:02:20 -- STEP: 7528/36938 -- GLOBAL_STEP: 1632800 | > loss_text_ce: 0.03445643186569214 (0.029924363537291193) | > loss_mel_ce: 1.342751145362854 (nan) | > loss: 0.005465109366923571 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1366557477004473) | > loader_time: 0.0046 (0.0046379565875917725)  --> TIME: 2025-02-27 09:02:29 -- STEP: 7578/36938 -- GLOBAL_STEP: 1632850 | > loss_text_ce: 0.023995691910386086 (0.029926100213217798) | > loss_mel_ce: 2.8981332778930664 (nan) | > loss: 0.011595750227570534 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13667176993046157) | > loader_time: 0.0046 (0.004637865883028562)  --> TIME: 2025-02-27 09:02:38 -- STEP: 7628/36938 -- GLOBAL_STEP: 1632900 | > loss_text_ce: 0.028079381212592125 (0.029918663197184323) | > loss_mel_ce: 2.53204345703125 (nan) | > loss: 0.01015921775251627 (nan) | > current_lr: 5e-06 | > step_time: 0.1618 (0.13667479227773202) | > loader_time: 0.0048 (0.004637901640463434)  --> TIME: 2025-02-27 09:02:47 -- STEP: 7678/36938 -- GLOBAL_STEP: 1632950 | > loss_text_ce: 0.043108757585287094 (0.029914189147837916) | > loss_mel_ce: 3.1845357418060303 (nan) | > loss: 0.012808113358914852 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.1366445713597187) | > loader_time: 0.0047 (0.0046378036252096395)  --> TIME: 2025-02-27 09:02:57 -- STEP: 7728/36938 -- GLOBAL_STEP: 1633000 | > loss_text_ce: 0.0333130843937397 (0.029919114365692175) | > loss_mel_ce: 2.6804823875427246 (nan) | > loss: 0.010769030079245567 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13663359893404953) | > loader_time: 0.0048 (0.004637762317993116)  --> TIME: 2025-02-27 09:03:06 -- STEP: 7778/36938 -- GLOBAL_STEP: 1633050 | > loss_text_ce: 0.02813537046313286 (0.029919460630949562) | > loss_mel_ce: 2.808002471923828 (nan) | > loss: 0.011254515498876572 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13665129433992654) | > loader_time: 0.0046 (0.004637680282994923)  --> TIME: 2025-02-27 09:03:15 -- STEP: 7828/36938 -- GLOBAL_STEP: 1633100 | > loss_text_ce: 0.03588827699422836 (0.029929179253090434) | > loss_mel_ce: 2.1881933212280273 (nan) | > loss: 0.008825721219182014 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1366492220803558) | > loader_time: 0.0046 (0.004637591529391998)  --> TIME: 2025-02-27 09:03:25 -- STEP: 7878/36938 -- GLOBAL_STEP: 1633150 | > loss_text_ce: 0.027765681967139244 (0.029929525145786093) | > loss_mel_ce: 2.536203622817993 (nan) | > loss: 0.010174482129514217 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.13664961608723084) | > loader_time: 0.0046 (0.004637410114730892)  --> TIME: 2025-02-27 09:03:34 -- STEP: 7928/36938 -- GLOBAL_STEP: 1633200 | > loss_text_ce: 0.024539398029446602 (0.029926103162619178) | > loss_mel_ce: 2.540512800216675 (nan) | > loss: 0.010178779251873493 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13665343909764058) | > loader_time: 0.0045 (0.004637134273646442)  --> TIME: 2025-02-27 09:03:43 -- STEP: 7978/36938 -- GLOBAL_STEP: 1633250 | > loss_text_ce: 0.030951781198382378 (0.029915390359858827) | > loss_mel_ce: 2.8502566814422607 (nan) | > loss: 0.011433367617428303 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.1366527064398259) | > loader_time: 0.0046 (0.004637258965498688)  --> TIME: 2025-02-27 09:03:53 -- STEP: 8028/36938 -- GLOBAL_STEP: 1633300 | > loss_text_ce: 0.032366037368774414 (0.029911876675778628) | > loss_mel_ce: 2.4995782375335693 (nan) | > loss: 0.010047398507595062 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13665553542827605) | > loader_time: 0.0046 (0.004637044819659344)  --> TIME: 2025-02-27 09:04:02 -- STEP: 8078/36938 -- GLOBAL_STEP: 1633350 | > loss_text_ce: 0.0276336669921875 (0.029908574566438306) | > loss_mel_ce: 2.569152355194092 (nan) | > loss: 0.01030470710247755 (nan) | > current_lr: 5e-06 | > step_time: 0.191 (0.1366703797165906) | > loader_time: 0.0049 (0.004636825798584331)  --> TIME: 2025-02-27 09:04:11 -- STEP: 8128/36938 -- GLOBAL_STEP: 1633400 | > loss_text_ce: 0.030120179057121277 (0.029901637609048442) | > loss_mel_ce: 1.1533203125 (nan) | > loss: 0.0046961926855146885 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13666328077945214) | > loader_time: 0.0046 (0.004636553446138922)  --> TIME: 2025-02-27 09:04:21 -- STEP: 8178/36938 -- GLOBAL_STEP: 1633450 | > loss_text_ce: 0.026056405156850815 (0.029890285550627387) | > loss_mel_ce: 2.268195390701294 (nan) | > loss: 0.009104173630475998 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.136662717377478) | > loader_time: 0.0045 (0.004636325268268453)  --> TIME: 2025-02-27 09:04:30 -- STEP: 8228/36938 -- GLOBAL_STEP: 1633500 | > loss_text_ce: 0.028405828401446342 (0.02989233863943174) | > loss_mel_ce: 2.278982400894165 (nan) | > loss: 0.009156303480267525 (nan) | > current_lr: 5e-06 | > step_time: 0.177 (0.1366655105053947) | > loader_time: 0.0049 (0.004636147182197536)  --> TIME: 2025-02-27 09:04:39 -- STEP: 8278/36938 -- GLOBAL_STEP: 1633550 | > loss_text_ce: 0.02554999850690365 (0.029893486250624397) | > loss_mel_ce: 2.4068617820739746 (nan) | > loss: 0.009652427397668362 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.1366927203247302) | > loader_time: 0.0048 (0.004635968914525048)  --> TIME: 2025-02-27 09:04:49 -- STEP: 8328/36938 -- GLOBAL_STEP: 1633600 | > loss_text_ce: 0.026160571724176407 (0.02989667862707187) | > loss_mel_ce: 2.256931781768799 (nan) | > loss: 0.009059890173375607 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13670570892170036) | > loader_time: 0.0045 (0.004635758576315242)  --> TIME: 2025-02-27 09:04:58 -- STEP: 8378/36938 -- GLOBAL_STEP: 1633650 | > loss_text_ce: 0.027751287445425987 (0.029901115363997338) | > loss_mel_ce: 2.7851016521453857 (nan) | > loss: 0.011162115260958672 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13672289990046996) | > loader_time: 0.0045 (0.0046355336171739914)  --> TIME: 2025-02-27 09:05:08 -- STEP: 8428/36938 -- GLOBAL_STEP: 1633700 | > loss_text_ce: 0.04154998064041138 (0.029898918888378687) | > loss_mel_ce: 3.010429620742798 (nan) | > loss: 0.012111030519008636 (nan) | > current_lr: 5e-06 | > step_time: 0.1515 (0.13672296917149995) | > loader_time: 0.0046 (0.004635419107807363)  --> TIME: 2025-02-27 09:05:17 -- STEP: 8478/36938 -- GLOBAL_STEP: 1633750 | > loss_text_ce: 0.04136227071285248 (0.02989295655607739) | > loss_mel_ce: 0.5306521058082581 (nan) | > loss: 0.0022698985412716866 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13669528073087933) | > loader_time: 0.0045 (0.004635064155882766)  --> TIME: 2025-02-27 09:05:26 -- STEP: 8528/36938 -- GLOBAL_STEP: 1633800 | > loss_text_ce: 0.028548387810587883 (0.02988713744285823) | > loss_mel_ce: 2.228508949279785 (nan) | > loss: 0.008956577628850937 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.13669707188574975) | > loader_time: 0.0046 (0.004634851558347118)  --> TIME: 2025-02-27 09:05:35 -- STEP: 8578/36938 -- GLOBAL_STEP: 1633850 | > loss_text_ce: 0.03328641504049301 (0.029887480170694643) | > loss_mel_ce: 2.543539524078369 (nan) | > loss: 0.010225499980151653 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1366980914386957) | > loader_time: 0.0047 (0.004634630599481057)  --> TIME: 2025-02-27 09:05:44 -- STEP: 8628/36938 -- GLOBAL_STEP: 1633900 | > loss_text_ce: 0.03393116593360901 (0.029880642957584275) | > loss_mel_ce: 2.777989387512207 (nan) | > loss: 0.011158416047692299 (nan) | > current_lr: 5e-06 | > step_time: 0.1574 (0.1366842439122704) | > loader_time: 0.0046 (0.004634338780360234)  --> TIME: 2025-02-27 09:05:54 -- STEP: 8678/36938 -- GLOBAL_STEP: 1633950 | > loss_text_ce: 0.03159007802605629 (0.029886498458683047) | > loss_mel_ce: 1.7297271490097046 (nan) | > loss: 0.006989354733377695 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13667211699634163) | > loader_time: 0.0046 (0.004634064005992176)  --> TIME: 2025-02-27 09:06:03 -- STEP: 8728/36938 -- GLOBAL_STEP: 1634000 | > loss_text_ce: 0.034965746104717255 (0.02989051044135593) | > loss_mel_ce: 2.6000423431396484 (nan) | > loss: 0.010456382296979427 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13667886643427238) | > loader_time: 0.0045 (0.00463394384991673)  --> TIME: 2025-02-27 09:06:12 -- STEP: 8778/36938 -- GLOBAL_STEP: 1634050 | > loss_text_ce: 0.021789971739053726 (0.029889408024482027) | > loss_mel_ce: 2.2348592281341553 (nan) | > loss: 0.00895495805889368 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13667280604991996) | > loader_time: 0.0045 (0.004633655877286068)  --> TIME: 2025-02-27 09:06:21 -- STEP: 8828/36938 -- GLOBAL_STEP: 1634100 | > loss_text_ce: 0.02978157252073288 (0.029886641976124784) | > loss_mel_ce: 2.895509958267212 (nan) | > loss: 0.011608300730586052 (nan) | > current_lr: 5e-06 | > step_time: 0.1613 (0.1366693683215059) | > loader_time: 0.0047 (0.004633620955255903)  --> TIME: 2025-02-27 09:06:31 -- STEP: 8878/36938 -- GLOBAL_STEP: 1634150 | > loss_text_ce: 0.0349164754152298 (0.0298824570816569) | > loss_mel_ce: 2.6756832599639893 (nan) | > loss: 0.010756348259747028 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.1366785234039664) | > loader_time: 0.0045 (0.004633816385838474)  --> TIME: 2025-02-27 09:06:40 -- STEP: 8928/36938 -- GLOBAL_STEP: 1634200 | > loss_text_ce: 0.02752566710114479 (0.029878502800269123) | > loss_mel_ce: 2.9347243309020996 (nan) | > loss: 0.011754960753023624 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.136669452792855) | > loader_time: 0.0046 (0.004633428054898426)  --> TIME: 2025-02-27 09:06:50 -- STEP: 8978/36938 -- GLOBAL_STEP: 1634250 | > loss_text_ce: 0.024395469576120377 (0.02988256085750132) | > loss_mel_ce: 2.561056613922119 (nan) | > loss: 0.010259730741381645 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13668540629333598) | > loader_time: 0.0047 (0.0046332379868206175)  --> TIME: 2025-02-27 09:06:59 -- STEP: 9028/36938 -- GLOBAL_STEP: 1634300 | > loss_text_ce: 0.023980507627129555 (0.029877064464001842) | > loss_mel_ce: 2.589092493057251 (nan) | > loss: 0.010369338095188141 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1367086622637885) | > loader_time: 0.0046 (0.004632978773138175)  --> TIME: 2025-02-27 09:07:09 -- STEP: 9078/36938 -- GLOBAL_STEP: 1634350 | > loss_text_ce: 0.03018295392394066 (0.029871816563919195) | > loss_mel_ce: 2.665414333343506 (nan) | > loss: 0.010696814395487309 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13674601108183781) | > loader_time: 0.0046 (0.00463285362649577)  --> TIME: 2025-02-27 09:07:18 -- STEP: 9128/36938 -- GLOBAL_STEP: 1634400 | > loss_text_ce: 0.03368858993053436 (0.029875412040198654) | > loss_mel_ce: 0.1877238005399704 (nan) | > loss: 0.0008786206599324942 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.1367498424416778) | > loader_time: 0.0046 (0.004632651596921863)  --> TIME: 2025-02-27 09:07:28 -- STEP: 9178/36938 -- GLOBAL_STEP: 1634450 | > loss_text_ce: 0.026180360466241837 (0.029877515684309536) | > loss_mel_ce: 3.9453125 (nan) | > loss: 0.015759892761707306 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13677256282416558) | > loader_time: 0.0045 (0.004632559807686577)  --> TIME: 2025-02-27 09:07:37 -- STEP: 9228/36938 -- GLOBAL_STEP: 1634500 | > loss_text_ce: 0.025596851482987404 (0.029883423471598464) | > loss_mel_ce: 2.3585009574890137 (nan) | > loss: 0.009460706263780594 (nan) | > current_lr: 5e-06 | > step_time: 0.1716 (0.1367517246226497) | > loader_time: 0.0047 (0.004632296089904867)  --> TIME: 2025-02-27 09:07:46 -- STEP: 9278/36938 -- GLOBAL_STEP: 1634550 | > loss_text_ce: 0.02594483271241188 (0.02988038346012577) | > loss_mel_ce: 1.9506795406341553 (nan) | > loss: 0.007843747735023499 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13674475306913667) | > loader_time: 0.0046 (0.004632178322203472)  --> TIME: 2025-02-27 09:07:56 -- STEP: 9328/36938 -- GLOBAL_STEP: 1634600 | > loss_text_ce: 0.03007013536989689 (0.029877840714278743) | > loss_mel_ce: 3.1341373920440674 (nan) | > loss: 0.012556379660964012 (nan) | > current_lr: 5e-06 | > step_time: 0.1789 (0.1367778592770138) | > loader_time: 0.0047 (0.004632270867787737)  --> TIME: 2025-02-27 09:08:05 -- STEP: 9378/36938 -- GLOBAL_STEP: 1634650 | > loss_text_ce: 0.028275782242417336 (0.029890057861681408) | > loss_mel_ce: 2.934518814086914 (nan) | > loss: 0.011757121421396732 (nan) | > current_lr: 5e-06 | > step_time: 0.1856 (0.13675467638502667) | > loader_time: 0.0048 (0.004632219395725973)  --> TIME: 2025-02-27 09:08:14 -- STEP: 9428/36938 -- GLOBAL_STEP: 1634700 | > loss_text_ce: 0.03344279155135155 (0.0298864225561896) | > loss_mel_ce: 1.728994369506836 (nan) | > loss: 0.006993798539042473 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.1367397110088934) | > loader_time: 0.0045 (0.004632101202597789)  --> TIME: 2025-02-27 09:08:23 -- STEP: 9478/36938 -- GLOBAL_STEP: 1634750 | > loss_text_ce: 0.02635500766336918 (0.029888502019147983) | > loss_mel_ce: 2.5154495239257812 (nan) | > loss: 0.010086526162922382 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13673870692803514) | > loader_time: 0.0048 (0.0046320666892540035)  --> TIME: 2025-02-27 09:08:33 -- STEP: 9528/36938 -- GLOBAL_STEP: 1634800 | > loss_text_ce: 0.04456670582294464 (0.029893175825348783) | > loss_mel_ce: 1.986379623413086 (nan) | > loss: 0.008059310726821423 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13676323598418094) | > loader_time: 0.0045 (0.004632069697207901)  --> TIME: 2025-02-27 09:08:42 -- STEP: 9578/36938 -- GLOBAL_STEP: 1634850 | > loss_text_ce: 0.029147028923034668 (0.029896464365149224) | > loss_mel_ce: 2.571545124053955 (nan) | > loss: 0.01032020803540945 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.1367825507669614) | > loader_time: 0.0046 (0.004631986571244436)  --> TIME: 2025-02-27 09:08:52 -- STEP: 9628/36938 -- GLOBAL_STEP: 1634900 | > loss_text_ce: 0.030936583876609802 (0.029898098325053094) | > loss_mel_ce: 2.737691879272461 (nan) | > loss: 0.010986621491611004 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.1367792806835357) | > loader_time: 0.0044 (0.004631847477474292)  --> TIME: 2025-02-27 09:09:01 -- STEP: 9678/36938 -- GLOBAL_STEP: 1634950 | > loss_text_ce: 0.022918352857232094 (0.02989976792376873) | > loss_mel_ce: 2.258950710296631 (nan) | > loss: 0.009055037051439285 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13679054118946932) | > loader_time: 0.0045 (0.0046318002810382545)  --> TIME: 2025-02-27 09:09:10 -- STEP: 9728/36938 -- GLOBAL_STEP: 1635000 | > loss_text_ce: 0.02626240998506546 (0.029903622045103943) | > loss_mel_ce: 2.6412386894226074 (nan) | > loss: 0.010585322976112366 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13677045286289954) | > loader_time: 0.0045 (0.004631651050754269)  --> TIME: 2025-02-27 09:09:20 -- STEP: 9778/36938 -- GLOBAL_STEP: 1635050 | > loss_text_ce: 0.03350114822387695 (0.029907016107076446) | > loss_mel_ce: 0.15483398735523224 (nan) | > loss: 0.0007473616860806942 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.1367768447417184) | > loader_time: 0.0046 (0.004631675247961711)  --> TIME: 2025-02-27 09:09:29 -- STEP: 9828/36938 -- GLOBAL_STEP: 1635100 | > loss_text_ce: 0.022548632696270943 (0.029901644893437475) | > loss_mel_ce: 1.7110368013381958 (nan) | > loss: 0.006879307795315981 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13677095282499915) | > loader_time: 0.0045 (0.00463153096200199)  --> TIME: 2025-02-27 09:09:38 -- STEP: 9878/36938 -- GLOBAL_STEP: 1635150 | > loss_text_ce: 0.03847271576523781 (0.02990139796648239) | > loss_mel_ce: 3.178281784057617 (nan) | > loss: 0.01276489906013012 (nan) | > current_lr: 5e-06 | > step_time: 0.1623 (0.136745824556067) | > loader_time: 0.0049 (0.00463137961660084)  --> TIME: 2025-02-27 09:09:47 -- STEP: 9928/36938 -- GLOBAL_STEP: 1635200 | > loss_text_ce: 0.05869094654917717 (0.02990552270845948) | > loss_mel_ce: 2.4382755756378174 (nan) | > loss: 0.00990859791636467 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.1367455116690023) | > loader_time: 0.0046 (0.004631287407048943)  --> TIME: 2025-02-27 09:09:57 -- STEP: 9978/36938 -- GLOBAL_STEP: 1635250 | > loss_text_ce: 0.028158873319625854 (0.029903559395251907) | > loss_mel_ce: 2.478762626647949 (nan) | > loss: 0.009948101826012135 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.13678129067232683) | > loader_time: 0.0046 (0.004631254113396692)  --> TIME: 2025-02-27 09:10:06 -- STEP: 10028/36938 -- GLOBAL_STEP: 1635300 | > loss_text_ce: 0.027193358168005943 (0.029901762409973394) | > loss_mel_ce: 2.0664963722229004 (nan) | > loss: 0.008308293297886848 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13678661708010126) | > loader_time: 0.0045 (0.0046311410528090844)  --> TIME: 2025-02-27 09:10:16 -- STEP: 10078/36938 -- GLOBAL_STEP: 1635350 | > loss_text_ce: 0.03231296315789223 (0.029904088855839887) | > loss_mel_ce: 2.328937530517578 (nan) | > loss: 0.009370042011141777 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13682109355264124) | > loader_time: 0.0045 (0.004631184211537151)  --> TIME: 2025-02-27 09:10:25 -- STEP: 10128/36938 -- GLOBAL_STEP: 1635400 | > loss_text_ce: 0.022839002311229706 (0.029906629200770595) | > loss_mel_ce: 2.630655288696289 (nan) | > loss: 0.010529739782214165 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.13684347283689532) | > loader_time: 0.0047 (0.004631108535221211)  --> TIME: 2025-02-27 09:10:35 -- STEP: 10178/36938 -- GLOBAL_STEP: 1635450 | > loss_text_ce: 0.02948915585875511 (0.029908333416824306) | > loss_mel_ce: 3.2317392826080322 (nan) | > loss: 0.012941382825374603 (nan) | > current_lr: 5e-06 | > step_time: 0.1717 (0.13684929131665116) | > loader_time: 0.0048 (0.004631256911955593)  --> TIME: 2025-02-27 09:10:44 -- STEP: 10228/36938 -- GLOBAL_STEP: 1635500 | > loss_text_ce: 0.026854008436203003 (0.029907484835377595) | > loss_mel_ce: 2.738156318664551 (nan) | > loss: 0.010972264222800732 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13683058658893762) | > loader_time: 0.0046 (0.00463122565344436)  --> TIME: 2025-02-27 09:10:53 -- STEP: 10278/36938 -- GLOBAL_STEP: 1635550 | > loss_text_ce: 0.03104163147509098 (0.029908124214882034) | > loss_mel_ce: 3.01427960395813 (nan) | > loss: 0.012084608897566795 (nan) | > current_lr: 5e-06 | > step_time: 0.1873 (0.13680393571959426) | > loader_time: 0.0048 (0.004631220447713236)  --> TIME: 2025-02-27 09:11:02 -- STEP: 10328/36938 -- GLOBAL_STEP: 1635600 | > loss_text_ce: 0.027936231344938278 (0.02990239462982047) | > loss_mel_ce: 3.249394178390503 (nan) | > loss: 0.013005279935896397 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.1368052201802929) | > loader_time: 0.0047 (0.004631151924938489)  --> TIME: 2025-02-27 09:11:11 -- STEP: 10378/36938 -- GLOBAL_STEP: 1635650 | > loss_text_ce: 0.02893613837659359 (0.029901298950655106) | > loss_mel_ce: 1.8255847692489624 (nan) | > loss: 0.007359210401773453 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13680522883873417) | > loader_time: 0.0046 (0.0046309935010497155)  --> TIME: 2025-02-27 09:11:21 -- STEP: 10428/36938 -- GLOBAL_STEP: 1635700 | > loss_text_ce: 0.029510408639907837 (0.029898214102726264) | > loss_mel_ce: 2.6403892040252686 (nan) | > loss: 0.010594841092824936 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.13680393239936942) | > loader_time: 0.0049 (0.0046310868352868705)  --> TIME: 2025-02-27 09:11:30 -- STEP: 10478/36938 -- GLOBAL_STEP: 1635750 | > loss_text_ce: 0.03323335945606232 (0.029897399096701537) | > loss_mel_ce: 2.60139799118042 (nan) | > loss: 0.01045488752424717 (nan) | > current_lr: 5e-06 | > step_time: 0.1766 (0.13681362985449683) | > loader_time: 0.0049 (0.0046311711099999145)  --> TIME: 2025-02-27 09:11:40 -- STEP: 10528/36938 -- GLOBAL_STEP: 1635800 | > loss_text_ce: 0.037443626672029495 (0.029901148602276207) | > loss_mel_ce: 2.4495737552642822 (nan) | > loss: 0.00986911728978157 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13684137595823126) | > loader_time: 0.0044 (0.004631424407647359)  --> TIME: 2025-02-27 09:11:49 -- STEP: 10578/36938 -- GLOBAL_STEP: 1635850 | > loss_text_ce: 0.03387831151485443 (0.029896102813665025) | > loss_mel_ce: 2.861072301864624 (nan) | > loss: 0.011487900279462337 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.1368601255683736) | > loader_time: 0.0046 (0.004631571405482516)  --> TIME: 2025-02-27 09:11:59 -- STEP: 10628/36938 -- GLOBAL_STEP: 1635900 | > loss_text_ce: 0.03846198320388794 (0.029898359060701426) | > loss_mel_ce: 3.234422445297241 (nan) | > loss: 0.012987636961042881 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.1368723475578637) | > loader_time: 0.0046 (0.004631546192434784)  --> TIME: 2025-02-27 09:12:08 -- STEP: 10678/36938 -- GLOBAL_STEP: 1635950 | > loss_text_ce: 0.03606053441762924 (0.029901990550417545) | > loss_mel_ce: 2.978325843811035 (nan) | > loss: 0.011961851269006729 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.13687010321819898) | > loader_time: 0.0048 (0.004631666660219517)  --> TIME: 2025-02-27 09:12:18 -- STEP: 10728/36938 -- GLOBAL_STEP: 1636000 | > loss_text_ce: 0.040060970932245255 (0.029898896143678983) | > loss_mel_ce: 2.6279995441436768 (nan) | > loss: 0.010587542317807674 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.13687537116047913) | > loader_time: 0.0046 (0.004631945684185488)  --> TIME: 2025-02-27 09:12:27 -- STEP: 10778/36938 -- GLOBAL_STEP: 1636050 | > loss_text_ce: 0.029179692268371582 (0.029895201398907913) | > loss_mel_ce: 2.620166778564453 (nan) | > loss: 0.01051327958703041 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.1368756510371249) | > loader_time: 0.0045 (0.004631929526768793)  --> TIME: 2025-02-27 09:12:37 -- STEP: 10828/36938 -- GLOBAL_STEP: 1636100 | > loss_text_ce: 0.028305701911449432 (0.029891903377702297) | > loss_mel_ce: 0.00417327880859375 (nan) | > loss: 0.00012888485798612237 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13689082473417435) | > loader_time: 0.0047 (0.004631909246941436)  --> TIME: 2025-02-27 09:12:46 -- STEP: 10878/36938 -- GLOBAL_STEP: 1636150 | > loss_text_ce: 0.030849793925881386 (0.029889595596082764) | > loss_mel_ce: 2.4308416843414307 (nan) | > loss: 0.00976861733943224 (nan) | > current_lr: 5e-06 | > step_time: 0.1534 (0.13689128351816526) | > loader_time: 0.0048 (0.004632285838399782)  --> TIME: 2025-02-27 09:12:55 -- STEP: 10928/36938 -- GLOBAL_STEP: 1636200 | > loss_text_ce: 0.032118458300828934 (0.02989104695589687) | > loss_mel_ce: 2.4218907356262207 (nan) | > loss: 0.009738132357597351 (nan) | > current_lr: 5e-06 | > step_time: 0.1832 (0.1368916865061881) | > loader_time: 0.0048 (0.004632529869435636)  --> TIME: 2025-02-27 09:13:05 -- STEP: 10978/36938 -- GLOBAL_STEP: 1636250 | > loss_text_ce: 0.029353909194469452 (0.02989138151550558) | > loss_mel_ce: 2.362825393676758 (nan) | > loss: 0.009492775425314903 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13689914651420093) | > loader_time: 0.0045 (0.004632743466071982)  --> TIME: 2025-02-27 09:13:14 -- STEP: 11028/36938 -- GLOBAL_STEP: 1636300 | > loss_text_ce: 0.03165018558502197 (0.029893898680992955) | > loss_mel_ce: 2.5420773029327393 (nan) | > loss: 0.010213205590844154 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13689837907162328) | > loader_time: 0.0046 (0.004632773371680199)  --> TIME: 2025-02-27 09:13:23 -- STEP: 11078/36938 -- GLOBAL_STEP: 1636350 | > loss_text_ce: 0.024222759529948235 (0.029893625494202655) | > loss_mel_ce: 2.3816628456115723 (nan) | > loss: 0.009547165594995022 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.1368883432604218) | > loader_time: 0.0047 (0.004632802103417582)  --> TIME: 2025-02-27 09:13:33 -- STEP: 11128/36938 -- GLOBAL_STEP: 1636400 | > loss_text_ce: 0.023765111342072487 (0.029889261669110722) | > loss_mel_ce: 2.618352174758911 (nan) | > loss: 0.010484592989087105 (nan) | > current_lr: 5e-06 | > step_time: 0.1517 (0.13687759567129854) | > loader_time: 0.0046 (0.00463311116014941)  --> TIME: 2025-02-27 09:13:42 -- STEP: 11178/36938 -- GLOBAL_STEP: 1636450 | > loss_text_ce: 0.02921828255057335 (0.02988739491031766) | > loss_mel_ce: 2.1958625316619873 (nan) | > loss: 0.008829685859382153 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.136859113467489) | > loader_time: 0.0045 (0.004632926750831499)  --> TIME: 2025-02-27 09:13:51 -- STEP: 11228/36938 -- GLOBAL_STEP: 1636500 | > loss_text_ce: 0.035131655633449554 (0.02988535010946484) | > loss_mel_ce: 1.1995950937271118 (nan) | > loss: 0.004899709951132536 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13686098683377362) | > loader_time: 0.0045 (0.004632765706595365)  --> TIME: 2025-02-27 09:14:00 -- STEP: 11278/36938 -- GLOBAL_STEP: 1636550 | > loss_text_ce: 0.03407210111618042 (0.029883312105237012) | > loss_mel_ce: 0.4562622010707855 (nan) | > loss: 0.0019457711605355144 (nan) | > current_lr: 5e-06 | > step_time: 0.1363 (0.1368299732607386) | > loader_time: 0.0048 (0.004632581419757866)  --> TIME: 2025-02-27 09:14:09 -- STEP: 11328/36938 -- GLOBAL_STEP: 1636600 | > loss_text_ce: 0.035760488361120224 (0.029883103249872443) | > loss_mel_ce: 2.2347300052642822 (nan) | > loss: 0.009009882807731628 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13683285924085198) | > loader_time: 0.0045 (0.004632603734900033)  --> TIME: 2025-02-27 09:14:18 -- STEP: 11378/36938 -- GLOBAL_STEP: 1636650 | > loss_text_ce: 0.04483587294816971 (0.029887571566025894) | > loss_mel_ce: 0.9952173233032227 (nan) | > loss: 0.004127195570617914 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1368058910686712) | > loader_time: 0.0046 (0.004632417295284593)  --> TIME: 2025-02-27 09:14:27 -- STEP: 11428/36938 -- GLOBAL_STEP: 1636700 | > loss_text_ce: 0.03555059805512428 (0.02988468395683152) | > loss_mel_ce: 3.013875722885132 (nan) | > loss: 0.012100898660719395 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.1368035737309048) | > loader_time: 0.0046 (0.004632415619604128)  --> TIME: 2025-02-27 09:14:37 -- STEP: 11478/36938 -- GLOBAL_STEP: 1636750 | > loss_text_ce: 0.020974405109882355 (0.02988565004236624) | > loss_mel_ce: 2.800318717956543 (nan) | > loss: 0.011195608414709568 (nan) | > current_lr: 5e-06 | > step_time: 0.1566 (0.13681457033970026) | > loader_time: 0.0063 (0.004632430035886401)  --> TIME: 2025-02-27 09:14:47 -- STEP: 11528/36938 -- GLOBAL_STEP: 1636800 | > loss_text_ce: 0.028518563136458397 (0.02988484913780076) | > loss_mel_ce: 3.0300941467285156 (nan) | > loss: 0.012137353420257568 (nan) | > current_lr: 5e-06 | > step_time: 0.1766 (0.13683648667014614) | > loader_time: 0.0048 (0.004632382881798553)  --> TIME: 2025-02-27 09:14:56 -- STEP: 11578/36938 -- GLOBAL_STEP: 1636850 | > loss_text_ce: 0.031448155641555786 (0.02988754079218983) | > loss_mel_ce: 3.4281506538391113 (nan) | > loss: 0.013728567399084568 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.1368595075475517) | > loader_time: 0.0046 (0.004632306523141008)  --> TIME: 2025-02-27 09:15:06 -- STEP: 11628/36938 -- GLOBAL_STEP: 1636900 | > loss_text_ce: 0.03668491914868355 (0.029887690394548135) | > loss_mel_ce: 2.967452049255371 (nan) | > loss: 0.011921179480850697 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.136854440578695) | > loader_time: 0.0046 (0.004632156412749904)  --> TIME: 2025-02-27 09:15:15 -- STEP: 11678/36938 -- GLOBAL_STEP: 1636950 | > loss_text_ce: 0.03224414959549904 (0.02988942686710324) | > loss_mel_ce: 2.809997320175171 (nan) | > loss: 0.011278736405074596 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13684125603174338) | > loader_time: 0.0045 (0.0046319790461390314)  --> TIME: 2025-02-27 09:15:24 -- STEP: 11728/36938 -- GLOBAL_STEP: 1637000 | > loss_text_ce: 0.03176279738545418 (0.029880523066410255) | > loss_mel_ce: 3.0103759765625 (nan) | > loss: 0.01207198016345501 (nan) | > current_lr: 5e-06 | > step_time: 0.1723 (0.13686337932883388) | > loader_time: 0.0049 (0.004631971292411144)  --> TIME: 2025-02-27 09:15:34 -- STEP: 11778/36938 -- GLOBAL_STEP: 1637050 | > loss_text_ce: 0.032444681972265244 (0.02988199056652803) | > loss_mel_ce: 2.87016224861145 (nan) | > loss: 0.011518281884491444 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13687585256038648) | > loader_time: 0.0046 (0.0046320406482506585)  --> TIME: 2025-02-27 09:15:43 -- STEP: 11828/36938 -- GLOBAL_STEP: 1637100 | > loss_text_ce: 0.03009631298482418 (0.029879191696149655) | > loss_mel_ce: 2.9028894901275635 (nan) | > loss: 0.011638833209872246 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.1368804053152426) | > loader_time: 0.0046 (0.004632013570548784)  --> TIME: 2025-02-27 09:15:53 -- STEP: 11878/36938 -- GLOBAL_STEP: 1637150 | > loss_text_ce: 0.03957369551062584 (0.029883082974697404) | > loss_mel_ce: 1.5313159227371216 (nan) | > loss: 0.006233689375221729 (nan) | > current_lr: 5e-06 | > step_time: 0.1703 (0.13687600457602153) | > loader_time: 0.0052 (0.004631914320086791)  --> TIME: 2025-02-27 09:16:02 -- STEP: 11928/36938 -- GLOBAL_STEP: 1637200 | > loss_text_ce: 0.021576985716819763 (0.029879647614845045) | > loss_mel_ce: 2.8641741275787354 (nan) | > loss: 0.011451393365859985 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.13688635952036232) | > loader_time: 0.0048 (0.004631870009609693)  --> TIME: 2025-02-27 09:16:12 -- STEP: 11978/36938 -- GLOBAL_STEP: 1637250 | > loss_text_ce: 0.02477831393480301 (0.029875120812187905) | > loss_mel_ce: 2.0152385234832764 (nan) | > loss: 0.008095305413007736 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13690759692173843) | > loader_time: 0.0045 (0.0046317868169828014)  --> TIME: 2025-02-27 09:16:21 -- STEP: 12028/36938 -- GLOBAL_STEP: 1637300 | > loss_text_ce: 0.041563134640455246 (0.02988097181485325) | > loss_mel_ce: 2.518239974975586 (nan) | > loss: 0.010157949291169643 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.13689995945828118) | > loader_time: 0.0047 (0.004631669905084989)  --> TIME: 2025-02-27 09:16:30 -- STEP: 12078/36938 -- GLOBAL_STEP: 1637350 | > loss_text_ce: 0.02871624194085598 (0.029880636339556243) | > loss_mel_ce: 2.4575250148773193 (nan) | > loss: 0.009866037406027317 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1369101533157033) | > loader_time: 0.0046 (0.00463198044182033)  --> TIME: 2025-02-27 09:16:40 -- STEP: 12128/36938 -- GLOBAL_STEP: 1637400 | > loss_text_ce: 0.037623945623636246 (0.02988469513746114) | > loss_mel_ce: 2.7891552448272705 (nan) | > loss: 0.011217378079891205 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13690429117953315) | > loader_time: 0.0046 (0.004632081944583883)  --> TIME: 2025-02-27 09:16:49 -- STEP: 12178/36938 -- GLOBAL_STEP: 1637450 | > loss_text_ce: 0.035007353872060776 (0.029881997374079882) | > loss_mel_ce: 2.1246256828308105 (nan) | > loss: 0.008569972589612007 (nan) | > current_lr: 5e-06 | > step_time: 0.1499 (0.1369114038411343) | > loader_time: 0.0049 (0.004632165287492058)  --> TIME: 2025-02-27 09:16:58 -- STEP: 12228/36938 -- GLOBAL_STEP: 1637500 | > loss_text_ce: 0.02451339177787304 (0.02987990940795034) | > loss_mel_ce: 2.956944465637207 (nan) | > loss: 0.011831182986497879 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13690463372371914) | > loader_time: 0.0046 (0.004632407089522885)  --> TIME: 2025-02-27 09:17:07 -- STEP: 12278/36938 -- GLOBAL_STEP: 1637550 | > loss_text_ce: 0.03838270902633667 (0.02988327208673239) | > loss_mel_ce: 2.4092366695404053 (nan) | > loss: 0.00971277616918087 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13687116945580277) | > loader_time: 0.0045 (0.004632849242015722)  --> TIME: 2025-02-27 09:17:16 -- STEP: 12328/36938 -- GLOBAL_STEP: 1637600 | > loss_text_ce: 0.026121456176042557 (0.029875413338841993) | > loss_mel_ce: 2.5767087936401367 (nan) | > loss: 0.010328691452741623 (nan) | > current_lr: 5e-06 | > step_time: 0.1726 (0.13685960422700974) | > loader_time: 0.0047 (0.004632855045725802)  --> TIME: 2025-02-27 09:17:26 -- STEP: 12378/36938 -- GLOBAL_STEP: 1637650 | > loss_text_ce: 0.03155653923749924 (0.029872792110722555) | > loss_mel_ce: 2.935858964920044 (nan) | > loss: 0.011775459162890911 (nan) | > current_lr: 5e-06 | > step_time: 0.171 (0.1368695242683464) | > loader_time: 0.0048 (0.004633222822034301)  --> TIME: 2025-02-27 09:17:35 -- STEP: 12428/36938 -- GLOBAL_STEP: 1637700 | > loss_text_ce: 0.03706295043230057 (0.02987022297403654) | > loss_mel_ce: 2.9716923236846924 (nan) | > loss: 0.011939505115151405 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13685429959501344) | > loader_time: 0.0051 (0.004633243324904709)  --> TIME: 2025-02-27 09:17:44 -- STEP: 12478/36938 -- GLOBAL_STEP: 1637750 | > loss_text_ce: 0.024309126660227776 (0.02986478522857452) | > loss_mel_ce: 2.8434131145477295 (nan) | > loss: 0.011379851028323174 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13683028432455113) | > loader_time: 0.0046 (0.004633159032902039)  --> TIME: 2025-02-27 09:17:53 -- STEP: 12528/36938 -- GLOBAL_STEP: 1637800 | > loss_text_ce: 0.033619605004787445 (0.029864328674873028) | > loss_mel_ce: 2.654644727706909 (nan) | > loss: 0.010667716152966022 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.1368065601069662) | > loader_time: 0.0047 (0.004633346717628619)  --> TIME: 2025-02-27 09:18:02 -- STEP: 12578/36938 -- GLOBAL_STEP: 1637850 | > loss_text_ce: 0.025281252339482307 (0.02986194432757813) | > loss_mel_ce: 2.1750426292419434 (nan) | > loss: 0.008731444366276264 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13681019561850732) | > loader_time: 0.0047 (0.004633530351235713)  --> TIME: 2025-02-27 09:18:11 -- STEP: 12628/36938 -- GLOBAL_STEP: 1637900 | > loss_text_ce: 0.03574129566550255 (0.029867747440834758) | > loss_mel_ce: 2.8721790313720703 (nan) | > loss: 0.011539367027580738 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13680319182662123) | > loader_time: 0.0048 (0.0046338161071871435)  --> TIME: 2025-02-27 09:18:21 -- STEP: 12678/36938 -- GLOBAL_STEP: 1637950 | > loss_text_ce: 0.034282393753528595 (0.02986759558458296) | > loss_mel_ce: 2.595550060272217 (nan) | > loss: 0.010435843840241432 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.1368108764020483) | > loader_time: 0.0047 (0.0046337700206072205)  --> TIME: 2025-02-27 09:18:30 -- STEP: 12728/36938 -- GLOBAL_STEP: 1638000 | > loss_text_ce: 0.031615715473890305 (0.029861674619504047) | > loss_mel_ce: 2.6798057556152344 (nan) | > loss: 0.010759609751403332 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13682377724284941) | > loader_time: 0.0045 (0.004633765880751335)  --> TIME: 2025-02-27 09:18:40 -- STEP: 12778/36938 -- GLOBAL_STEP: 1638050 | > loss_text_ce: 0.029984554275870323 (0.02986329735152068) | > loss_mel_ce: 3.10805082321167 (nan) | > loss: 0.012452521361410618 (nan) | > current_lr: 5e-06 | > step_time: 0.1789 (0.13683262048967082) | > loader_time: 0.0048 (0.004633686411528154)  --> TIME: 2025-02-27 09:18:49 -- STEP: 12828/36938 -- GLOBAL_STEP: 1638100 | > loss_text_ce: 0.023545244708657265 (0.02986390973789931) | > loss_mel_ce: 3.123943328857422 (nan) | > loss: 0.01249003503471613 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13683855433236636) | > loader_time: 0.0045 (0.004633690417277527)  --> TIME: 2025-02-27 09:18:59 -- STEP: 12878/36938 -- GLOBAL_STEP: 1638150 | > loss_text_ce: 0.03732762113213539 (0.029862755987413378) | > loss_mel_ce: 2.6990461349487305 (nan) | > loss: 0.0108586261048913 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.1368332896696935) | > loader_time: 0.0046 (0.004633914889302888)  --> TIME: 2025-02-27 09:19:08 -- STEP: 12928/36938 -- GLOBAL_STEP: 1638200 | > loss_text_ce: 0.021110817790031433 (0.029863025219948196) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13683051370674518) | > loader_time: 0.0046 (0.004634347679738938)  --> TIME: 2025-02-27 09:19:17 -- STEP: 12978/36938 -- GLOBAL_STEP: 1638250 | > loss_text_ce: 0.030514849349856377 (0.029866902837641717) | > loss_mel_ce: 2.8808188438415527 (nan) | > loss: 0.011552912183105946 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.1368399279237738) | > loader_time: 0.0046 (0.004634386494854474)  --> TIME: 2025-02-27 09:19:27 -- STEP: 13028/36938 -- GLOBAL_STEP: 1638300 | > loss_text_ce: 0.02049623243510723 (0.029868862202405602) | > loss_mel_ce: 2.061553955078125 (nan) | > loss: 0.008262104354798794 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13684120960018992) | > loader_time: 0.0046 (0.004634450413092228)  --> TIME: 2025-02-27 09:19:37 -- STEP: 13078/36938 -- GLOBAL_STEP: 1638350 | > loss_text_ce: 0.01958727277815342 (0.029866709647526992) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.13687257335389894) | > loader_time: 0.0048 (0.004634516777695443)  --> TIME: 2025-02-27 09:19:45 -- STEP: 13128/36938 -- GLOBAL_STEP: 1638400 | > loss_text_ce: 0.03204998001456261 (0.02986514102529233) | > loss_mel_ce: 1.8801454305648804 (nan) | > loss: 0.007588077802211046 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13684573057655375) | > loader_time: 0.0046 (0.0046346812150886525)  --> TIME: 2025-02-27 09:19:55 -- STEP: 13178/36938 -- GLOBAL_STEP: 1638450 | > loss_text_ce: 0.031773608177900314 (0.02986370530631774) | > loss_mel_ce: 3.185854196548462 (nan) | > loss: 0.012768364511430264 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.13684345612415305) | > loader_time: 0.0049 (0.004634904832502722)  --> TIME: 2025-02-27 09:20:04 -- STEP: 13228/36938 -- GLOBAL_STEP: 1638500 | > loss_text_ce: 0.03637409210205078 (0.029866791617355913) | > loss_mel_ce: 1.6304223537445068 (nan) | > loss: 0.006614272017031908 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13686064481374863) | > loader_time: 0.0045 (0.0046349691594876185)  --> TIME: 2025-02-27 09:20:14 -- STEP: 13278/36938 -- GLOBAL_STEP: 1638550 | > loss_text_ce: 0.029404805973172188 (0.02986270584718614) | > loss_mel_ce: 2.807347536087036 (nan) | > loss: 0.01125695463269949 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13685382425219564) | > loader_time: 0.0045 (0.004634911530275335)  --> TIME: 2025-02-27 09:20:23 -- STEP: 13328/36938 -- GLOBAL_STEP: 1638600 | > loss_text_ce: 0.04087363928556442 (0.029864255770154027) | > loss_mel_ce: 2.6771037578582764 (nan) | > loss: 0.010785624384880066 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.1368508485793218) | > loader_time: 0.0046 (0.0046349866013853155)  --> TIME: 2025-02-27 09:20:32 -- STEP: 13378/36938 -- GLOBAL_STEP: 1638650 | > loss_text_ce: 0.03686375170946121 (0.02986797154484248) | > loss_mel_ce: 3.2393672466278076 (nan) | > loss: 0.013000917620956898 (nan) | > current_lr: 5e-06 | > step_time: 0.1503 (0.13684889806512848) | > loader_time: 0.0047 (0.004634929409047051)  --> TIME: 2025-02-27 09:20:41 -- STEP: 13428/36938 -- GLOBAL_STEP: 1638700 | > loss_text_ce: 0.028706664219498634 (0.02986551785634484) | > loss_mel_ce: 1.9533517360687256 (nan) | > loss: 0.007865311577916145 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.1368396375589093) | > loader_time: 0.0044 (0.004634808297309264)  --> TIME: 2025-02-27 09:20:51 -- STEP: 13478/36938 -- GLOBAL_STEP: 1638750 | > loss_text_ce: 0.035892851650714874 (0.029863644430235835) | > loss_mel_ce: 3.0190274715423584 (nan) | > loss: 0.012122700922191143 (nan) | > current_lr: 5e-06 | > step_time: 0.1494 (0.13683000382515337) | > loader_time: 0.0047 (0.004634824841396439)  --> TIME: 2025-02-27 09:21:00 -- STEP: 13528/36938 -- GLOBAL_STEP: 1638800 | > loss_text_ce: 0.03166152536869049 (0.029869450218467465) | > loss_mel_ce: 3.0498337745666504 (nan) | > loss: 0.012228156439960003 (nan) | > current_lr: 5e-06 | > step_time: 0.1829 (0.13683046579431538) | > loader_time: 0.0051 (0.004635017081028881)  --> TIME: 2025-02-27 09:21:09 -- STEP: 13578/36938 -- GLOBAL_STEP: 1638850 | > loss_text_ce: 0.01932181976735592 (0.02987310973234476) | > loss_mel_ce: 2.5010318756103516 (nan) | > loss: 0.010001404210925102 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13682827483854496) | > loader_time: 0.0046 (0.00463500063625486)  --> TIME: 2025-02-27 09:21:18 -- STEP: 13628/36938 -- GLOBAL_STEP: 1638900 | > loss_text_ce: 0.016925707459449768 (0.029866828882881155) | > loss_mel_ce: 2.1708014011383057 (nan) | > loss: 0.008681457489728928 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.1368043821579494) | > loader_time: 0.0045 (0.0046348929020327626)  --> TIME: 2025-02-27 09:21:28 -- STEP: 13678/36938 -- GLOBAL_STEP: 1638950 | > loss_text_ce: 0.026979243382811546 (0.029862630898707585) | > loss_mel_ce: 2.736711025238037 (nan) | > loss: 0.01096702553331852 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13680973382769462) | > loader_time: 0.0046 (0.0046348187079431665)  --> TIME: 2025-02-27 09:21:37 -- STEP: 13728/36938 -- GLOBAL_STEP: 1639000 | > loss_text_ce: 0.03943105787038803 (0.029869199135930415) | > loss_mel_ce: 2.201322555541992 (nan) | > loss: 0.008891880512237549 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1368175365013404) | > loader_time: 0.0046 (0.004634792658137822)  --> TIME: 2025-02-27 09:21:46 -- STEP: 13778/36938 -- GLOBAL_STEP: 1639050 | > loss_text_ce: 0.028662027791142464 (0.029865849103868088) | > loss_mel_ce: 2.7392725944519043 (nan) | > loss: 0.010983867570757866 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13681414087896115) | > loader_time: 0.0068 (0.004634923314750573)  --> TIME: 2025-02-27 09:21:56 -- STEP: 13828/36938 -- GLOBAL_STEP: 1639100 | > loss_text_ce: 0.025848500430583954 (0.029866933205465615) | > loss_mel_ce: 2.6007237434387207 (nan) | > loss: 0.010422905907034874 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13681594170964104) | > loader_time: 0.0047 (0.004635001301317147)  --> TIME: 2025-02-27 09:22:05 -- STEP: 13878/36938 -- GLOBAL_STEP: 1639150 | > loss_text_ce: 0.030112361535429955 (0.029867182138039988) | > loss_mel_ce: 2.9743974208831787 (nan) | > loss: 0.011922658421099186 (nan) | > current_lr: 5e-06 | > step_time: 0.1617 (0.13683140458248483) | > loader_time: 0.0047 (0.004635059209906885)  --> TIME: 2025-02-27 09:22:15 -- STEP: 13928/36938 -- GLOBAL_STEP: 1639200 | > loss_text_ce: 0.0327712781727314 (0.029869522602934732) | > loss_mel_ce: 2.94856595993042 (nan) | > loss: 0.011830704286694527 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13683509590432744) | > loader_time: 0.0047 (0.004635221789719923)  --> TIME: 2025-02-27 09:22:24 -- STEP: 13978/36938 -- GLOBAL_STEP: 1639250 | > loss_text_ce: 0.03203045204281807 (0.029868366710618044) | > loss_mel_ce: 3.2515554428100586 (nan) | > loss: 0.013030103407800198 (nan) | > current_lr: 5e-06 | > step_time: 0.1541 (0.13685220346070168) | > loader_time: 0.0049 (0.0046355352839749404)  --> TIME: 2025-02-27 09:22:34 -- STEP: 14028/36938 -- GLOBAL_STEP: 1639300 | > loss_text_ce: 0.027044627815485 (0.0298681847942942) | > loss_mel_ce: 2.6355772018432617 (nan) | > loss: 0.010565959848463535 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13685033040608321) | > loader_time: 0.0047 (0.00463552061295216)  --> TIME: 2025-02-27 09:22:43 -- STEP: 14078/36938 -- GLOBAL_STEP: 1639350 | > loss_text_ce: 0.0225936621427536 (0.029866489655625196) | > loss_mel_ce: 2.2770586013793945 (nan) | > loss: 0.00912560522556305 (nan) | > current_lr: 5e-06 | > step_time: 0.1921 (0.13686722368827622) | > loader_time: 0.0049 (0.00463567311527681)  --> TIME: 2025-02-27 09:22:53 -- STEP: 14128/36938 -- GLOBAL_STEP: 1639400 | > loss_text_ce: 0.025007866322994232 (0.029863037765328) | > loss_mel_ce: 1.2213159799575806 (nan) | > loss: 0.004945729859173298 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13687379889418097) | > loader_time: 0.0045 (0.004635861158235957)  --> TIME: 2025-02-27 09:23:02 -- STEP: 14178/36938 -- GLOBAL_STEP: 1639450 | > loss_text_ce: 0.030987348407506943 (0.02986245138800179) | > loss_mel_ce: 0.0218505859375 (nan) | > loss: 0.0002096743555739522 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13689102479442672) | > loader_time: 0.0047 (0.004636041114824339)  --> TIME: 2025-02-27 09:23:12 -- STEP: 14228/36938 -- GLOBAL_STEP: 1639500 | > loss_text_ce: 0.0339769572019577 (0.029864412353458887) | > loss_mel_ce: 2.7052838802337646 (nan) | > loss: 0.01087008323520422 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.13690394923899554) | > loader_time: 0.0048 (0.004636234670064732)  --> TIME: 2025-02-27 09:23:21 -- STEP: 14278/36938 -- GLOBAL_STEP: 1639550 | > loss_text_ce: 0.030504288151860237 (0.02986117187763627) | > loss_mel_ce: 1.910753846168518 (nan) | > loss: 0.0077034058049321175 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13689437480613484) | > loader_time: 0.0046 (0.004636163370234229)  --> TIME: 2025-02-27 09:23:31 -- STEP: 14328/36938 -- GLOBAL_STEP: 1639600 | > loss_text_ce: 0.027166880667209625 (0.02985985682429516) | > loss_mel_ce: 1.47576904296875 (nan) | > loss: 0.00596403144299984 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13690627897537969) | > loader_time: 0.0046 (0.004636519168888894)  --> TIME: 2025-02-27 09:23:40 -- STEP: 14378/36938 -- GLOBAL_STEP: 1639650 | > loss_text_ce: 0.024781426414847374 (0.02985947433627088) | > loss_mel_ce: 0.05798950046300888 (nan) | > loss: 0.0003284560807514936 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1368968556928519) | > loader_time: 0.0045 (0.004636372905030229)  --> TIME: 2025-02-27 09:23:49 -- STEP: 14428/36938 -- GLOBAL_STEP: 1639700 | > loss_text_ce: 0.030658716335892677 (0.0298599034577434) | > loss_mel_ce: 2.733943462371826 (nan) | > loss: 0.010970644652843475 (nan) | > current_lr: 5e-06 | > step_time: 0.152 (0.13690410502769404) | > loader_time: 0.0048 (0.004636281855984798)  --> TIME: 2025-02-27 09:23:59 -- STEP: 14478/36938 -- GLOBAL_STEP: 1639750 | > loss_text_ce: 0.02749938890337944 (0.029860775831860108) | > loss_mel_ce: 1.9546557664871216 (nan) | > loss: 0.007865695282816887 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.1369156212351546) | > loader_time: 0.0044 (0.004636220698824959)  --> TIME: 2025-02-27 09:24:08 -- STEP: 14528/36938 -- GLOBAL_STEP: 1639800 | > loss_text_ce: 0.03306133300065994 (0.02986280629960722) | > loss_mel_ce: 2.400287389755249 (nan) | > loss: 0.009656146168708801 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13692741368441944) | > loader_time: 0.0046 (0.004636286967126274)  --> TIME: 2025-02-27 09:24:17 -- STEP: 14578/36938 -- GLOBAL_STEP: 1639850 | > loss_text_ce: 0.03665050119161606 (0.029862143807702187) | > loss_mel_ce: 3.0996148586273193 (nan) | > loss: 0.012445498257875443 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13692481484428273) | > loader_time: 0.0045 (0.004636341757793315)  --> TIME: 2025-02-27 09:24:27 -- STEP: 14628/36938 -- GLOBAL_STEP: 1639900 | > loss_text_ce: 0.028595970943570137 (0.02986052634073143) | > loss_mel_ce: 2.5454277992248535 (nan) | > loss: 0.010214379988610744 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13691218218126755) | > loader_time: 0.0046 (0.004636254374499125)  --> TIME: 2025-02-27 09:24:36 -- STEP: 14678/36938 -- GLOBAL_STEP: 1639950 | > loss_text_ce: 0.027088569477200508 (0.029865333260113012) | > loss_mel_ce: 2.421891450881958 (nan) | > loss: 0.00971817597746849 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13691467525101766) | > loader_time: 0.0045 (0.004636408701500452)  --> TIME: 2025-02-27 09:24:45 -- STEP: 14728/36938 -- GLOBAL_STEP: 1640000 | > loss_text_ce: 0.030478792265057564 (0.0298685037601086) | > loss_mel_ce: 2.7548632621765137 (nan) | > loss: 0.01105294469743967 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13692794148524456) | > loader_time: 0.0046 (0.004636598468668616) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1640000.pth  --> TIME: 2025-02-27 09:25:01 -- STEP: 14778/36938 -- GLOBAL_STEP: 1640050 | > loss_text_ce: 0.02089001052081585 (0.029866555180608066) | > loss_mel_ce: 2.847851276397705 (nan) | > loss: 0.011383894830942154 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.1369348817820936) | > loader_time: 0.0047 (0.004636917034881743)  --> TIME: 2025-02-27 09:25:10 -- STEP: 14828/36938 -- GLOBAL_STEP: 1640100 | > loss_text_ce: 0.02359490655362606 (0.02986261304918252) | > loss_mel_ce: 3.0390076637268066 (nan) | > loss: 0.012153184972703457 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.1369275994496368) | > loader_time: 0.0046 (0.004637008025892529)  --> TIME: 2025-02-27 09:25:20 -- STEP: 14878/36938 -- GLOBAL_STEP: 1640150 | > loss_text_ce: 0.03557560220360756 (0.029858752269524932) | > loss_mel_ce: 2.0335605144500732 (nan) | > loss: 0.008210858330130577 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13693927039156592) | > loader_time: 0.0047 (0.004637067525324945)  --> TIME: 2025-02-27 09:25:29 -- STEP: 14928/36938 -- GLOBAL_STEP: 1640200 | > loss_text_ce: 0.025533253327012062 (0.02985691960673556) | > loss_mel_ce: 2.990004539489746 (nan) | > loss: 0.011966420337557793 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.13692326166558288) | > loader_time: 0.0048 (0.004637031357771321)  --> TIME: 2025-02-27 09:25:39 -- STEP: 14978/36938 -- GLOBAL_STEP: 1640250 | > loss_text_ce: 0.023559827357530594 (0.029856383583656037) | > loss_mel_ce: 2.8965845108032227 (nan) | > loss: 0.011587874963879585 (nan) | > current_lr: 5e-06 | > step_time: 0.1766 (0.13694125411507016) | > loader_time: 0.0048 (0.0046371682047732414)  --> TIME: 2025-02-27 09:25:48 -- STEP: 15028/36938 -- GLOBAL_STEP: 1640300 | > loss_text_ce: 0.04529498517513275 (0.02985728540271743) | > loss_mel_ce: 3.0848774909973145 (nan) | > loss: 0.01242132019251585 (nan) | > current_lr: 5e-06 | > step_time: 0.1543 (0.13693563696041672) | > loader_time: 0.0047 (0.004637260782233182)  --> TIME: 2025-02-27 09:25:57 -- STEP: 15078/36938 -- GLOBAL_STEP: 1640350 | > loss_text_ce: 0.026303717866539955 (0.02985074007554449) | > loss_mel_ce: 0.8729782104492188 (nan) | > loss: 0.0035685792099684477 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1369303905263416) | > loader_time: 0.0046 (0.004637311775909831)  --> TIME: 2025-02-27 09:26:06 -- STEP: 15128/36938 -- GLOBAL_STEP: 1640400 | > loss_text_ce: 0.04679466784000397 (0.02985329307642967) | > loss_mel_ce: 3.1237399578094482 (nan) | > loss: 0.01258148718625307 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13692931615757906) | > loader_time: 0.0046 (0.004637353732937663)  --> TIME: 2025-02-27 09:26:16 -- STEP: 15178/36938 -- GLOBAL_STEP: 1640450 | > loss_text_ce: 0.032296158373355865 (0.029851866590785904) | > loss_mel_ce: 3.08551025390625 (nan) | > loss: 0.01237224880605936 (nan) | > current_lr: 5e-06 | > step_time: 0.1357 (0.13693809625517842) | > loader_time: 0.0046 (0.00463751402591147)  --> TIME: 2025-02-27 09:26:25 -- STEP: 15228/36938 -- GLOBAL_STEP: 1640500 | > loss_text_ce: 0.024690749123692513 (0.02985052360099548) | > loss_mel_ce: 1.6256946325302124 (nan) | > loss: 0.006549148820340633 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.1369271757629626) | > loader_time: 0.0046 (0.004637543723625172)  --> TIME: 2025-02-27 09:26:34 -- STEP: 15278/36938 -- GLOBAL_STEP: 1640550 | > loss_text_ce: 0.04045715555548668 (0.029850828491127187) | > loss_mel_ce: 2.7506675720214844 (nan) | > loss: 0.011075892485678196 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.13690338003368605) | > loader_time: 0.0046 (0.004637468905173626)  --> TIME: 2025-02-27 09:26:43 -- STEP: 15328/36938 -- GLOBAL_STEP: 1640600 | > loss_text_ce: 0.026247359812259674 (0.029853663427711726) | > loss_mel_ce: 2.808082342147827 (nan) | > loss: 0.011247340589761734 (nan) | > current_lr: 5e-06 | > step_time: 0.1771 (0.13689243188419573) | > loader_time: 0.005 (0.0046376165057024936)  --> TIME: 2025-02-27 09:26:52 -- STEP: 15378/36938 -- GLOBAL_STEP: 1640650 | > loss_text_ce: 0.026057200506329536 (0.029849461484902712) | > loss_mel_ce: 2.288748025894165 (nan) | > loss: 0.009185736067593098 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.1368816217344847) | > loader_time: 0.0046 (0.004637534293726548)  --> TIME: 2025-02-27 09:27:01 -- STEP: 15428/36938 -- GLOBAL_STEP: 1640700 | > loss_text_ce: 0.02998938038945198 (0.029850436915391902) | > loss_mel_ce: 2.056802272796631 (nan) | > loss: 0.00828091986477375 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13688246244774777) | > loader_time: 0.0046 (0.004637541380267323)  --> TIME: 2025-02-27 09:27:11 -- STEP: 15478/36938 -- GLOBAL_STEP: 1640750 | > loss_text_ce: 0.01975896768271923 (0.029846842620322733) | > loss_mel_ce: 2.2344324588775635 (nan) | > loss: 0.008945204317569733 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13687822434895772) | > loader_time: 0.0046 (0.004637545941026646)  --> TIME: 2025-02-27 09:27:20 -- STEP: 15528/36938 -- GLOBAL_STEP: 1640800 | > loss_text_ce: 0.027892863377928734 (0.02985152503866476) | > loss_mel_ce: 2.8263611793518066 (nan) | > loss: 0.011326405219733715 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.1368664828297533) | > loader_time: 0.0046 (0.004637680137851437)  --> TIME: 2025-02-27 09:27:29 -- STEP: 15578/36938 -- GLOBAL_STEP: 1640850 | > loss_text_ce: 0.029330309480428696 (0.029851082762302018) | > loss_mel_ce: 2.838345527648926 (nan) | > loss: 0.0113796666264534 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13687009039425996) | > loader_time: 0.0044 (0.004637728347783209)  --> TIME: 2025-02-27 09:27:38 -- STEP: 15628/36938 -- GLOBAL_STEP: 1640900 | > loss_text_ce: 0.0329270176589489 (0.029851006813465805) | > loss_mel_ce: 0.06367746740579605 (nan) | > loss: 0.00038335114368237555 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13687245001230322) | > loader_time: 0.0045 (0.004637690358739199)  --> TIME: 2025-02-27 09:27:48 -- STEP: 15678/36938 -- GLOBAL_STEP: 1640950 | > loss_text_ce: 0.029151126742362976 (0.02985061619634295) | > loss_mel_ce: 2.3310680389404297 (nan) | > loss: 0.009365949779748917 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13687178572764144) | > loader_time: 0.0046 (0.004637853407954078)  --> TIME: 2025-02-27 09:27:57 -- STEP: 15728/36938 -- GLOBAL_STEP: 1641000 | > loss_text_ce: 0.025481969118118286 (0.02985422940864246) | > loss_mel_ce: 2.574376344680786 (nan) | > loss: 0.010316898114979267 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.13685182584326533) | > loader_time: 0.0047 (0.004637726674487357)  --> TIME: 2025-02-27 09:28:06 -- STEP: 15778/36938 -- GLOBAL_STEP: 1641050 | > loss_text_ce: 0.03740515187382698 (0.02985573823957816) | > loss_mel_ce: 2.747281789779663 (nan) | > loss: 0.011050346307456493 (nan) | > current_lr: 5e-06 | > step_time: 0.1785 (0.13685615318993244) | > loader_time: 0.005 (0.004637739068731357)  --> TIME: 2025-02-27 09:28:15 -- STEP: 15828/36938 -- GLOBAL_STEP: 1641100 | > loss_text_ce: 0.03567570075392723 (0.029857968255019126) | > loss_mel_ce: 2.21929931640625 (nan) | > loss: 0.008948314003646374 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13686033666872022) | > loader_time: 0.0046 (0.004637730672921612)  --> TIME: 2025-02-27 09:28:24 -- STEP: 15878/36938 -- GLOBAL_STEP: 1641150 | > loss_text_ce: 0.026848632842302322 (0.029856584355379853) | > loss_mel_ce: 2.789236068725586 (nan) | > loss: 0.011174939572811127 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13683698756641688) | > loader_time: 0.0046 (0.004637627491108301)  --> TIME: 2025-02-27 09:28:34 -- STEP: 15928/36938 -- GLOBAL_STEP: 1641200 | > loss_text_ce: 0.029789112508296967 (0.02985928021117016) | > loss_mel_ce: 2.8951823711395264 (nan) | > loss: 0.011607030406594276 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.1368320941446061) | > loader_time: 0.0046 (0.004637673025691638)  --> TIME: 2025-02-27 09:28:43 -- STEP: 15978/36938 -- GLOBAL_STEP: 1641250 | > loss_text_ce: 0.02561507374048233 (0.029854494481836996) | > loss_mel_ce: 2.7603466510772705 (nan) | > loss: 0.01105540432035923 (nan) | > current_lr: 5e-06 | > step_time: 0.2046 (0.13683093413346928) | > loader_time: 0.0049 (0.004638071307283651)  --> TIME: 2025-02-27 09:28:52 -- STEP: 16028/36938 -- GLOBAL_STEP: 1641300 | > loss_text_ce: 0.016821563243865967 (0.02985539554735324) | > loss_mel_ce: 1.7164306640625 (nan) | > loss: 0.006877985782921314 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13683180281548946) | > loader_time: 0.0046 (0.004638120662193204)  --> TIME: 2025-02-27 09:29:02 -- STEP: 16078/36938 -- GLOBAL_STEP: 1641350 | > loss_text_ce: 0.03228767216205597 (0.029860270279265598) | > loss_mel_ce: 2.8602309226989746 (nan) | > loss: 0.011478248983621597 (nan) | > current_lr: 5e-06 | > step_time: 0.1757 (0.1368301772466276) | > loader_time: 0.0048 (0.004638118624672357)  --> TIME: 2025-02-27 09:29:11 -- STEP: 16128/36938 -- GLOBAL_STEP: 1641400 | > loss_text_ce: 0.03978544846177101 (0.02985940743663319) | > loss_mel_ce: 0.6529785394668579 (nan) | > loss: 0.002749063540250063 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13681533792248368) | > loader_time: 0.0046 (0.004638153882253729)  --> TIME: 2025-02-27 09:29:20 -- STEP: 16178/36938 -- GLOBAL_STEP: 1641450 | > loss_text_ce: 0.024757711216807365 (0.02986411791019526) | > loss_mel_ce: 2.529656410217285 (nan) | > loss: 0.010136564262211323 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1368182619476606) | > loader_time: 0.0046 (0.004638356587134909)  --> TIME: 2025-02-27 09:29:30 -- STEP: 16228/36938 -- GLOBAL_STEP: 1641500 | > loss_text_ce: 0.02589285559952259 (0.029866700384143522) | > loss_mel_ce: 2.7535881996154785 (nan) | > loss: 0.011029687710106373 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.13682761354225495) | > loader_time: 0.005 (0.00463850522587855)  --> TIME: 2025-02-27 09:29:39 -- STEP: 16278/36938 -- GLOBAL_STEP: 1641550 | > loss_text_ce: 0.028510546311736107 (0.02986725718978165) | > loss_mel_ce: 3.130127429962158 (nan) | > loss: 0.012534278444945812 (nan) | > current_lr: 5e-06 | > step_time: 0.1449 (0.13683109550602668) | > loader_time: 0.0047 (0.0046386490701266865)  --> TIME: 2025-02-27 09:29:48 -- STEP: 16328/36938 -- GLOBAL_STEP: 1641600 | > loss_text_ce: 0.03359097242355347 (0.02986760172468305) | > loss_mel_ce: 2.4376418590545654 (nan) | > loss: 0.009806480258703232 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13682894352485875) | > loader_time: 0.0047 (0.004638822171572425)  --> TIME: 2025-02-27 09:29:58 -- STEP: 16378/36938 -- GLOBAL_STEP: 1641650 | > loss_text_ce: 0.030798034742474556 (0.02986934204667279) | > loss_mel_ce: 1.9704525470733643 (nan) | > loss: 0.007941470481455326 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13684039484786428) | > loader_time: 0.0046 (0.004638993997745037)  --> TIME: 2025-02-27 09:30:07 -- STEP: 16428/36938 -- GLOBAL_STEP: 1641700 | > loss_text_ce: 0.03128785640001297 (0.029872191743035157) | > loss_mel_ce: 2.4091997146606445 (nan) | > loss: 0.009684475138783455 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13683075670607386) | > loader_time: 0.0045 (0.004639245615058892)  --> TIME: 2025-02-27 09:30:17 -- STEP: 16478/36938 -- GLOBAL_STEP: 1641750 | > loss_text_ce: 0.03193899244070053 (0.029870367182894816) | > loss_mel_ce: 1.2278637886047363 (nan) | > loss: 0.004999218042939901 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13684577589721486) | > loader_time: 0.0046 (0.004639274852052151)  --> TIME: 2025-02-27 09:30:26 -- STEP: 16528/36938 -- GLOBAL_STEP: 1641800 | > loss_text_ce: 0.028536846861243248 (0.029869027632293814) | > loss_mel_ce: 2.3929879665374756 (nan) | > loss: 0.009609226137399673 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13685440192033385) | > loader_time: 0.0046 (0.004639311413220252)  --> TIME: 2025-02-27 09:30:35 -- STEP: 16578/36938 -- GLOBAL_STEP: 1641850 | > loss_text_ce: 0.026545358821749687 (0.029867310457699883) | > loss_mel_ce: 0.8134765625 (nan) | > loss: 0.003333420492708683 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.13683801254470856) | > loader_time: 0.0046 (0.004639323060597961)  --> TIME: 2025-02-27 09:30:45 -- STEP: 16628/36938 -- GLOBAL_STEP: 1641900 | > loss_text_ce: 0.03606104478240013 (0.029869516363636505) | > loss_mel_ce: 2.367671012878418 (nan) | > loss: 0.00953861977905035 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.1368409094398692) | > loader_time: 0.0047 (0.004639343714123301)  --> TIME: 2025-02-27 09:30:54 -- STEP: 16678/36938 -- GLOBAL_STEP: 1641950 | > loss_text_ce: 0.031060585752129555 (0.029868957206725465) | > loss_mel_ce: 2.2361621856689453 (nan) | > loss: 0.008996916934847832 (nan) | > current_lr: 5e-06 | > step_time: 0.1293 (0.13685509929526649) | > loader_time: 0.0048 (0.004639595228785982)  --> TIME: 2025-02-27 09:31:04 -- STEP: 16728/36938 -- GLOBAL_STEP: 1642000 | > loss_text_ce: 0.027012474834918976 (0.029869703218288086) | > loss_mel_ce: 2.925551414489746 (nan) | > loss: 0.011716524139046669 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13685964296617822) | > loader_time: 0.0047 (0.0046399222898004305)  --> TIME: 2025-02-27 09:31:13 -- STEP: 16778/36938 -- GLOBAL_STEP: 1642050 | > loss_text_ce: 0.036166273057460785 (0.029869849124374025) | > loss_mel_ce: 2.7855608463287354 (nan) | > loss: 0.011197330430150032 (nan) | > current_lr: 5e-06 | > step_time: 0.1858 (0.13686609519708887) | > loader_time: 0.0049 (0.0046401091505253594)  --> TIME: 2025-02-27 09:31:23 -- STEP: 16828/36938 -- GLOBAL_STEP: 1642100 | > loss_text_ce: 0.027942456305027008 (0.029874791005421095) | > loss_mel_ce: 2.908402681350708 (nan) | > loss: 0.011652164161205292 (nan) | > current_lr: 5e-06 | > step_time: 0.1874 (0.1368779079319609) | > loader_time: 0.0048 (0.004640188159357764)  --> TIME: 2025-02-27 09:31:32 -- STEP: 16878/36938 -- GLOBAL_STEP: 1642150 | > loss_text_ce: 0.0290138628333807 (0.029875611050713485) | > loss_mel_ce: 1.8549270629882812 (nan) | > loss: 0.007475956343114376 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13687967272332593) | > loader_time: 0.0046 (0.004640121866224605)  --> TIME: 2025-02-27 09:31:42 -- STEP: 16928/36938 -- GLOBAL_STEP: 1642200 | > loss_text_ce: 0.03524746745824814 (0.029872947270467663) | > loss_mel_ce: 2.6261308193206787 (nan) | > loss: 0.010561025701463223 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13689894384100698) | > loader_time: 0.0047 (0.004640284214443422)  --> TIME: 2025-02-27 09:31:51 -- STEP: 16978/36938 -- GLOBAL_STEP: 1642250 | > loss_text_ce: 0.03233381733298302 (0.02987381645945888) | > loss_mel_ce: 2.978480100631714 (nan) | > loss: 0.011947674676775932 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.1369091799099382) | > loader_time: 0.0046 (0.00464024565107255)  --> TIME: 2025-02-27 09:32:01 -- STEP: 17028/36938 -- GLOBAL_STEP: 1642300 | > loss_text_ce: 0.029049159958958626 (0.0298735129693501) | > loss_mel_ce: 2.409175395965576 (nan) | > loss: 0.009675494395196438 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13689777278160897) | > loader_time: 0.0045 (0.004640089897082272)  --> TIME: 2025-02-27 09:32:10 -- STEP: 17078/36938 -- GLOBAL_STEP: 1642350 | > loss_text_ce: 0.04039732366800308 (0.02987887653735309) | > loss_mel_ce: 1.7750974893569946 (nan) | > loss: 0.007204344961792231 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13688366613311026) | > loader_time: 0.0045 (0.004639949476374068)  --> TIME: 2025-02-27 09:32:19 -- STEP: 17128/36938 -- GLOBAL_STEP: 1642400 | > loss_text_ce: 0.033492203801870346 (0.02988157283063105) | > loss_mel_ce: 2.227355718612671 (nan) | > loss: 0.008971618488430977 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13687948443409687) | > loader_time: 0.0047 (0.004639865220678785)  --> TIME: 2025-02-27 09:32:28 -- STEP: 17178/36938 -- GLOBAL_STEP: 1642450 | > loss_text_ce: 0.026518473401665688 (0.029881261926618635) | > loss_mel_ce: 2.7063915729522705 (nan) | > loss: 0.0108448825776577 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13687480505027183) | > loader_time: 0.0045 (0.004639789325031462)  --> TIME: 2025-02-27 09:32:37 -- STEP: 17228/36938 -- GLOBAL_STEP: 1642500 | > loss_text_ce: 0.033290162682533264 (0.029880426565178912) | > loss_mel_ce: 2.9061856269836426 (nan) | > loss: 0.011664587073028088 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.13687555260466766) | > loader_time: 0.0047 (0.004639816292483088)  --> TIME: 2025-02-27 09:32:47 -- STEP: 17278/36938 -- GLOBAL_STEP: 1642550 | > loss_text_ce: 0.028143109753727913 (0.02988087129967817) | > loss_mel_ce: 2.6263270378112793 (nan) | > loss: 0.010533612221479416 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.1368787991171265) | > loader_time: 0.0045 (0.004639661109258423)  --> TIME: 2025-02-27 09:32:56 -- STEP: 17328/36938 -- GLOBAL_STEP: 1642600 | > loss_text_ce: 0.026042813435196877 (0.029878178138131293) | > loss_mel_ce: 1.7369304895401 (nan) | > loss: 0.006995926145464182 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.1368674944229736) | > loader_time: 0.0044 (0.004639540063707443)  --> TIME: 2025-02-27 09:33:05 -- STEP: 17378/36938 -- GLOBAL_STEP: 1642650 | > loss_text_ce: 0.03216797858476639 (0.029877373681938862) | > loss_mel_ce: 2.599167823791504 (nan) | > loss: 0.010441808961331844 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13687395873889113) | > loader_time: 0.0046 (0.004639529237978767)  --> TIME: 2025-02-27 09:33:14 -- STEP: 17428/36938 -- GLOBAL_STEP: 1642700 | > loss_text_ce: 0.03511766344308853 (0.02987960644767123) | > loss_mel_ce: 2.5912015438079834 (nan) | > loss: 0.010421901941299438 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13685635652240072) | > loader_time: 0.0046 (0.004639435230331289)  --> TIME: 2025-02-27 09:33:23 -- STEP: 17478/36938 -- GLOBAL_STEP: 1642750 | > loss_text_ce: 0.02820790745317936 (0.02988082423871944) | > loss_mel_ce: 2.3677821159362793 (nan) | > loss: 0.00950789824128151 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.13684457688517682) | > loader_time: 0.0047 (0.004639321680895063)  --> TIME: 2025-02-27 09:33:32 -- STEP: 17528/36938 -- GLOBAL_STEP: 1642800 | > loss_text_ce: 0.037365466356277466 (0.029880734949919125) | > loss_mel_ce: 2.5460739135742188 (nan) | > loss: 0.010251743718981743 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13684428456383513) | > loader_time: 0.0046 (0.004639274314460761)  --> TIME: 2025-02-27 09:33:42 -- STEP: 17578/36938 -- GLOBAL_STEP: 1642850 | > loss_text_ce: 0.027465321123600006 (0.029880837755661763) | > loss_mel_ce: 2.5829927921295166 (nan) | > loss: 0.010358961299061775 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13684661480086246) | > loader_time: 0.0047 (0.004639137237467221)  --> TIME: 2025-02-27 09:33:51 -- STEP: 17628/36938 -- GLOBAL_STEP: 1642900 | > loss_text_ce: 0.031840600073337555 (0.029880970733726083) | > loss_mel_ce: 3.2425026893615723 (nan) | > loss: 0.01299342606216669 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13684416960122672) | > loader_time: 0.0046 (0.004639110490527576)  --> TIME: 2025-02-27 09:34:01 -- STEP: 17678/36938 -- GLOBAL_STEP: 1642950 | > loss_text_ce: 0.028478609398007393 (0.02988031622168742) | > loss_mel_ce: 2.8417415618896484 (nan) | > loss: 0.011389763094484806 (nan) | > current_lr: 5e-06 | > step_time: 0.1129 (0.13685997600050484) | > loader_time: 0.0045 (0.004638990175536554)  --> TIME: 2025-02-27 09:34:10 -- STEP: 17728/36938 -- GLOBAL_STEP: 1643000 | > loss_text_ce: 0.03268531709909439 (0.029881744625179018) | > loss_mel_ce: 2.7565267086029053 (nan) | > loss: 0.01106830220669508 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13686955608562015) | > loader_time: 0.0046 (0.004638886986979499)  --> TIME: 2025-02-27 09:34:20 -- STEP: 17778/36938 -- GLOBAL_STEP: 1643050 | > loss_text_ce: 0.02885407581925392 (0.029882143731377254) | > loss_mel_ce: 2.865949869155884 (nan) | > loss: 0.011487318202853203 (nan) | > current_lr: 5e-06 | > step_time: 0.1886 (0.13688864686757593) | > loader_time: 0.0048 (0.004638865286674409)  --> TIME: 2025-02-27 09:34:29 -- STEP: 17828/36938 -- GLOBAL_STEP: 1643100 | > loss_text_ce: 0.030697733163833618 (0.029883226609840233) | > loss_mel_ce: 2.5411016941070557 (nan) | > loss: 0.010205553844571114 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1368951948335788) | > loader_time: 0.0045 (0.004638845566973592)  --> TIME: 2025-02-27 09:34:39 -- STEP: 17878/36938 -- GLOBAL_STEP: 1643150 | > loss_text_ce: 0.027335841208696365 (0.029882701021641117) | > loss_mel_ce: 0.14528807997703552 (nan) | > loss: 0.0006850155768916011 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.1368941463390871) | > loader_time: 0.0045 (0.004638742941819996)  --> TIME: 2025-02-27 09:34:48 -- STEP: 17928/36938 -- GLOBAL_STEP: 1643200 | > loss_text_ce: 0.02931707166135311 (0.02988461194963982) | > loss_mel_ce: 3.1057114601135254 (nan) | > loss: 0.012440590187907219 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.1368980613578276) | > loader_time: 0.0045 (0.0046387052147689465)  --> TIME: 2025-02-27 09:34:57 -- STEP: 17978/36938 -- GLOBAL_STEP: 1643250 | > loss_text_ce: 0.035583801567554474 (0.029883079580173787) | > loss_mel_ce: 2.6739821434020996 (nan) | > loss: 0.010752245783805847 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13688667443294864) | > loader_time: 0.0045 (0.004638547374297991)  --> TIME: 2025-02-27 09:35:06 -- STEP: 18028/36938 -- GLOBAL_STEP: 1643300 | > loss_text_ce: 0.027770502492785454 (0.02988560545249028) | > loss_mel_ce: 2.7052204608917236 (nan) | > loss: 0.010845202952623367 (nan) | > current_lr: 5e-06 | > step_time: 0.1757 (0.1368903703532992) | > loader_time: 0.0049 (0.004638428457406976)  --> TIME: 2025-02-27 09:35:16 -- STEP: 18078/36938 -- GLOBAL_STEP: 1643350 | > loss_text_ce: 0.029278116300702095 (0.029884792754720446) | > loss_mel_ce: 3.1047613620758057 (nan) | > loss: 0.012436664663255215 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.1369022878480572) | > loader_time: 0.0046 (0.0046383356385980585)  --> TIME: 2025-02-27 09:35:26 -- STEP: 18128/36938 -- GLOBAL_STEP: 1643400 | > loss_text_ce: 0.030375337228178978 (0.029887452120227906) | > loss_mel_ce: 2.5677707195281982 (nan) | > loss: 0.010310103185474873 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.13692160357128497) | > loader_time: 0.0047 (0.004638298464789391)  --> TIME: 2025-02-27 09:35:35 -- STEP: 18178/36938 -- GLOBAL_STEP: 1643450 | > loss_text_ce: 0.02080368995666504 (0.029886407237076758) | > loss_mel_ce: 2.706411838531494 (nan) | > loss: 0.010822284035384655 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13691479826094188) | > loader_time: 0.0047 (0.004638414805787191)  --> TIME: 2025-02-27 09:35:44 -- STEP: 18228/36938 -- GLOBAL_STEP: 1643500 | > loss_text_ce: 0.02335944212973118 (0.02988840511473051) | > loss_mel_ce: 2.757078170776367 (nan) | > loss: 0.011033483780920506 (nan) | > current_lr: 5e-06 | > step_time: 0.1628 (0.13692492854838473) | > loader_time: 0.0047 (0.004638497691314572)  --> TIME: 2025-02-27 09:35:53 -- STEP: 18278/36938 -- GLOBAL_STEP: 1643550 | > loss_text_ce: 0.029047688469290733 (0.02989121533269234) | > loss_mel_ce: 2.3959202766418457 (nan) | > loss: 0.009622889570891857 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.136918910067039) | > loader_time: 0.0045 (0.004638447635269537)  --> TIME: 2025-02-27 09:36:03 -- STEP: 18328/36938 -- GLOBAL_STEP: 1643600 | > loss_text_ce: 0.03278200700879097 (0.029887878987780635) | > loss_mel_ce: 3.004307985305786 (nan) | > loss: 0.012051945552229881 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.13691789402237425) | > loader_time: 0.0046 (0.004638450276326109)  --> TIME: 2025-02-27 09:36:12 -- STEP: 18378/36938 -- GLOBAL_STEP: 1643650 | > loss_text_ce: 0.028827454894781113 (0.029888828098576313) | > loss_mel_ce: 2.8469526767730713 (nan) | > loss: 0.011411826126277447 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13689895374936586) | > loader_time: 0.0064 (0.0046385418461928184)  --> TIME: 2025-02-27 09:36:21 -- STEP: 18428/36938 -- GLOBAL_STEP: 1643700 | > loss_text_ce: 0.028404686599969864 (0.029888062728001864) | > loss_mel_ce: 3.0147173404693604 (nan) | > loss: 0.012075881473720074 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13688908157365207) | > loader_time: 0.0046 (0.004638488338752811)  --> TIME: 2025-02-27 09:36:30 -- STEP: 18478/36938 -- GLOBAL_STEP: 1643750 | > loss_text_ce: 0.028994357213377953 (0.029884507020593486) | > loss_mel_ce: 2.3679354190826416 (nan) | > loss: 0.009511626325547695 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13688882490891596) | > loader_time: 0.0048 (0.004638481390451706)  --> TIME: 2025-02-27 09:36:40 -- STEP: 18528/36938 -- GLOBAL_STEP: 1643800 | > loss_text_ce: 0.034665174782276154 (0.029885422457810784) | > loss_mel_ce: 2.242877960205078 (nan) | > loss: 0.009037869982421398 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13689953571310945) | > loader_time: 0.0045 (0.004638815649329894)  --> TIME: 2025-02-27 09:36:49 -- STEP: 18578/36938 -- GLOBAL_STEP: 1643850 | > loss_text_ce: 0.027262825518846512 (0.029884034776394414) | > loss_mel_ce: 2.5597288608551025 (nan) | > loss: 0.010265841148793697 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13689483242587272) | > loader_time: 0.0046 (0.004638741444766245)  --> TIME: 2025-02-27 09:36:59 -- STEP: 18628/36938 -- GLOBAL_STEP: 1643900 | > loss_text_ce: 0.02217116393148899 (0.029885255748315853) | > loss_mel_ce: 1.270046591758728 (nan) | > loss: 0.005127848591655493 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13690755194237453) | > loader_time: 0.0045 (0.004638783084954574)  --> TIME: 2025-02-27 09:37:08 -- STEP: 18678/36938 -- GLOBAL_STEP: 1643950 | > loss_text_ce: 0.028183288872241974 (0.029884278515951337) | > loss_mel_ce: 2.7111706733703613 (nan) | > loss: 0.010870452970266342 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.13690321557028357) | > loader_time: 0.0048 (0.00463875174356459)  --> TIME: 2025-02-27 09:37:17 -- STEP: 18728/36938 -- GLOBAL_STEP: 1644000 | > loss_text_ce: 0.02094123885035515 (0.029885831669692837) | > loss_mel_ce: 2.4459917545318604 (nan) | > loss: 0.009789417497813702 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13690067431143552) | > loader_time: 0.0046 (0.0046386716331186986)  --> TIME: 2025-02-27 09:37:26 -- STEP: 18778/36938 -- GLOBAL_STEP: 1644050 | > loss_text_ce: 0.030626943334937096 (0.029884430555315675) | > loss_mel_ce: 2.173720359802246 (nan) | > loss: 0.008747410960495472 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13688904874202168) | > loader_time: 0.0045 (0.004638507084567678)  --> TIME: 2025-02-27 09:37:36 -- STEP: 18828/36938 -- GLOBAL_STEP: 1644100 | > loss_text_ce: 0.028065234422683716 (0.029885522207698915) | > loss_mel_ce: 2.8746049404144287 (nan) | > loss: 0.01151853334158659 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13688150964422507) | > loader_time: 0.0046 (0.004638622983235931)  --> TIME: 2025-02-27 09:37:45 -- STEP: 18878/36938 -- GLOBAL_STEP: 1644150 | > loss_text_ce: 0.02093496546149254 (0.02988542653483588) | > loss_mel_ce: 1.3273589611053467 (nan) | > loss: 0.00535037275403738 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1368848249965055) | > loader_time: 0.0046 (0.0046386429788399745)  --> TIME: 2025-02-27 09:37:54 -- STEP: 18928/36938 -- GLOBAL_STEP: 1644200 | > loss_text_ce: 0.029526839032769203 (0.029886848486786535) | > loss_mel_ce: 2.6287670135498047 (nan) | > loss: 0.010548786260187626 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.13687471520165143) | > loader_time: 0.0049 (0.004638662314576208)  --> TIME: 2025-02-27 09:38:04 -- STEP: 18978/36938 -- GLOBAL_STEP: 1644250 | > loss_text_ce: 0.03443698212504387 (0.02989009302705726) | > loss_mel_ce: 2.863283634185791 (nan) | > loss: 0.01149889174848795 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13688540985064435) | > loader_time: 0.0044 (0.004638573984939492)  --> TIME: 2025-02-27 09:38:13 -- STEP: 19028/36938 -- GLOBAL_STEP: 1644300 | > loss_text_ce: 0.025980545207858086 (0.029890723084187686) | > loss_mel_ce: 0.03125 (nan) | > loss: 0.0002271053526783362 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.13689328303777173) | > loader_time: 0.0047 (0.0046385546579615555)  --> TIME: 2025-02-27 09:38:22 -- STEP: 19078/36938 -- GLOBAL_STEP: 1644350 | > loss_text_ce: 0.028465326875448227 (0.029890806436032365) | > loss_mel_ce: 3.0798873901367188 (nan) | > loss: 0.0123347332701087 (nan) | > current_lr: 5e-06 | > step_time: 0.1442 (0.13689553066549215) | > loader_time: 0.0047 (0.004638508438676616)  --> TIME: 2025-02-27 09:38:32 -- STEP: 19128/36938 -- GLOBAL_STEP: 1644400 | > loss_text_ce: 0.021096834912896156 (0.02988934044270756) | > loss_mel_ce: 2.475151777267456 (nan) | > loss: 0.009905749931931496 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13689479857546108) | > loader_time: 0.0044 (0.004638391538725295)  --> TIME: 2025-02-27 09:38:41 -- STEP: 19178/36938 -- GLOBAL_STEP: 1644450 | > loss_text_ce: 0.041873760521411896 (0.029889183682428887) | > loss_mel_ce: 2.919381618499756 (nan) | > loss: 0.011751013807952404 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13689748051641712) | > loader_time: 0.0045 (0.004638241669820638)  --> TIME: 2025-02-27 09:38:50 -- STEP: 19228/36938 -- GLOBAL_STEP: 1644500 | > loss_text_ce: 0.03200184926390648 (0.02989028569741303) | > loss_mel_ce: 1.8230903148651123 (nan) | > loss: 0.007361477240920067 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.1368976502724046) | > loader_time: 0.0046 (0.00463818656894802)  --> TIME: 2025-02-27 09:39:00 -- STEP: 19278/36938 -- GLOBAL_STEP: 1644550 | > loss_text_ce: 0.028789741918444633 (0.029893415408797957) | > loss_mel_ce: 2.838062286376953 (nan) | > loss: 0.0113763976842165 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.13689973035508277) | > loader_time: 0.0048 (0.004638123925338951)  --> TIME: 2025-02-27 09:39:09 -- STEP: 19328/36938 -- GLOBAL_STEP: 1644600 | > loss_text_ce: 0.03017573244869709 (0.029891316437011924) | > loss_mel_ce: 2.4830729961395264 (nan) | > loss: 0.009973209351301193 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1368962374225953) | > loader_time: 0.0046 (0.004638055265443214)  --> TIME: 2025-02-27 09:39:18 -- STEP: 19378/36938 -- GLOBAL_STEP: 1644650 | > loss_text_ce: 0.0351111963391304 (0.029894343117263137) | > loss_mel_ce: 2.4922173023223877 (nan) | > loss: 0.0100290821865201 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.13690294069283696) | > loader_time: 0.0048 (0.004637956323977008)  --> TIME: 2025-02-27 09:39:27 -- STEP: 19428/36938 -- GLOBAL_STEP: 1644700 | > loss_text_ce: 0.021642204374074936 (0.02989214337287562) | > loss_mel_ce: 2.802557945251465 (nan) | > loss: 0.011207143776118755 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13688518025636068) | > loader_time: 0.0045 (0.004637827113844466)  --> TIME: 2025-02-27 09:39:37 -- STEP: 19478/36938 -- GLOBAL_STEP: 1644750 | > loss_text_ce: 0.028173508122563362 (0.029892946404669534) | > loss_mel_ce: 2.313499689102173 (nan) | > loss: 0.009292353875935078 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.1368818043413958) | > loader_time: 0.0046 (0.00463779924439507)  --> TIME: 2025-02-27 09:39:46 -- STEP: 19528/36938 -- GLOBAL_STEP: 1644800 | > loss_text_ce: 0.02243964560329914 (0.02989138657528474) | > loss_mel_ce: 2.207663059234619 (nan) | > loss: 0.008849614299833775 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13688240806766375) | > loader_time: 0.0048 (0.004637871827321282)  --> TIME: 2025-02-27 09:39:55 -- STEP: 19578/36938 -- GLOBAL_STEP: 1644850 | > loss_text_ce: 0.03812898322939873 (0.029893048583964767) | > loss_mel_ce: 2.5010082721710205 (nan) | > loss: 0.010075942613184452 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13686986937198012) | > loader_time: 0.0047 (0.004637925675263841)  --> TIME: 2025-02-27 09:40:04 -- STEP: 19628/36938 -- GLOBAL_STEP: 1644900 | > loss_text_ce: 0.028406037017703056 (0.02989133032306323) | > loss_mel_ce: 2.8089334964752197 (nan) | > loss: 0.011259284801781178 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.13686027126252032) | > loader_time: 0.0048 (0.004637935350109848)  --> TIME: 2025-02-27 09:40:14 -- STEP: 19678/36938 -- GLOBAL_STEP: 1644950 | > loss_text_ce: 0.026299310848116875 (0.029893331021514032) | > loss_mel_ce: 2.9036307334899902 (nan) | > loss: 0.011626706458628178 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13685158870728825) | > loader_time: 0.0045 (0.004637998201362673)  --> TIME: 2025-02-27 09:40:23 -- STEP: 19728/36938 -- GLOBAL_STEP: 1645000 | > loss_text_ce: 0.021290168166160583 (0.029892486884062864) | > loss_mel_ce: 2.721240520477295 (nan) | > loss: 0.010883058421313763 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.13684470515278244) | > loader_time: 0.0046 (0.004637884905158452)  --> TIME: 2025-02-27 09:40:32 -- STEP: 19778/36938 -- GLOBAL_STEP: 1645050 | > loss_text_ce: 0.02758512832224369 (0.029891469043064833) | > loss_mel_ce: 2.8901939392089844 (nan) | > loss: 0.01157848909497261 (nan) | > current_lr: 5e-06 | > step_time: 0.1989 (0.1368488248986511) | > loader_time: 0.0048 (0.004637971338095992)  --> TIME: 2025-02-27 09:40:42 -- STEP: 19828/36938 -- GLOBAL_STEP: 1645100 | > loss_text_ce: 0.034525878727436066 (0.029891192450376726) | > loss_mel_ce: 2.869091033935547 (nan) | > loss: 0.011522290296852589 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.13685178546287954) | > loader_time: 0.0047 (0.0046379690163554374)  --> TIME: 2025-02-27 09:40:51 -- STEP: 19878/36938 -- GLOBAL_STEP: 1645150 | > loss_text_ce: 0.032673485577106476 (0.029887709755030502) | > loss_mel_ce: 2.466866970062256 (nan) | > loss: 0.009918812662363052 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.1368641080433172) | > loader_time: 0.0045 (0.00463796927303072)  --> TIME: 2025-02-27 09:41:01 -- STEP: 19928/36938 -- GLOBAL_STEP: 1645200 | > loss_text_ce: 0.03698918968439102 (0.029888643753177034) | > loss_mel_ce: 3.0583131313323975 (nan) | > loss: 0.012282946147024632 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13686412119425134) | > loader_time: 0.0045 (0.004637972423705831)  --> TIME: 2025-02-27 09:41:09 -- STEP: 19978/36938 -- GLOBAL_STEP: 1645250 | > loss_text_ce: 0.026623278856277466 (0.029886661362730313) | > loss_mel_ce: 2.4085073471069336 (nan) | > loss: 0.009663216769695282 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13685067297400547) | > loader_time: 0.0046 (0.0046378745129545375)  --> TIME: 2025-02-27 09:41:19 -- STEP: 20028/36938 -- GLOBAL_STEP: 1645300 | > loss_text_ce: 0.026232201606035233 (0.029883617690709678) | > loss_mel_ce: 1.2487636804580688 (nan) | > loss: 0.005059508141130209 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1368531638904081) | > loader_time: 0.0045 (0.004637815875161392)  --> TIME: 2025-02-27 09:41:28 -- STEP: 20078/36938 -- GLOBAL_STEP: 1645350 | > loss_text_ce: 0.034916408360004425 (0.02988050927371673) | > loss_mel_ce: 1.660302758216858 (nan) | > loss: 0.00672706076875329 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.136856615822419) | > loader_time: 0.0045 (0.004637743980479142)  --> TIME: 2025-02-27 09:41:37 -- STEP: 20128/36938 -- GLOBAL_STEP: 1645400 | > loss_text_ce: 0.025648321956396103 (0.02987783483840989) | > loss_mel_ce: 2.9182238578796387 (nan) | > loss: 0.01168203353881836 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13685222336095546) | > loader_time: 0.0045 (0.004637799552969021)  --> TIME: 2025-02-27 09:41:47 -- STEP: 20178/36938 -- GLOBAL_STEP: 1645450 | > loss_text_ce: 0.024121124297380447 (0.02987863045218205) | > loss_mel_ce: 2.818307876586914 (nan) | > loss: 0.011279480531811714 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13684052217702627) | > loader_time: 0.0045 (0.004637803073349544)  --> TIME: 2025-02-27 09:41:56 -- STEP: 20228/36938 -- GLOBAL_STEP: 1645500 | > loss_text_ce: 0.03410562500357628 (0.02987884455572722) | > loss_mel_ce: 2.52117919921875 (nan) | > loss: 0.010140019468963146 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1368478334112696) | > loader_time: 0.0045 (0.004637826424897216)  --> TIME: 2025-02-27 09:42:06 -- STEP: 20278/36938 -- GLOBAL_STEP: 1645550 | > loss_text_ce: 0.03030439093708992 (0.02987776617243072) | > loss_mel_ce: 2.3527677059173584 (nan) | > loss: 0.009456636384129524 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13685945424470136) | > loader_time: 0.0048 (0.004637854035077725)  --> TIME: 2025-02-27 09:42:15 -- STEP: 20328/36938 -- GLOBAL_STEP: 1645600 | > loss_text_ce: 0.0306940209120512 (0.029878084090013645) | > loss_mel_ce: 2.6408379077911377 (nan) | > loss: 0.010601317510008812 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.1368508036682709) | > loader_time: 0.0045 (0.004637827499322833)  --> TIME: 2025-02-27 09:42:25 -- STEP: 20378/36938 -- GLOBAL_STEP: 1645650 | > loss_text_ce: 0.031051155179739 (0.02987815565151757) | > loss_mel_ce: 2.6599771976470947 (nan) | > loss: 0.010678684338927269 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13687325071476542) | > loader_time: 0.0047 (0.004637782374101307)  --> TIME: 2025-02-27 09:42:34 -- STEP: 20428/36938 -- GLOBAL_STEP: 1645700 | > loss_text_ce: 0.016755154356360435 (0.0298785950362528) | > loss_mel_ce: 1.7695364952087402 (nan) | > loss: 0.007088459096848965 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.1368798357132762) | > loader_time: 0.0044 (0.004637752222132478)  --> TIME: 2025-02-27 09:42:44 -- STEP: 20478/36938 -- GLOBAL_STEP: 1645750 | > loss_text_ce: 0.02866252139210701 (0.029876266811422313) | > loss_mel_ce: 1.3989496231079102 (nan) | > loss: 0.005665128119289875 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13688814215087738) | > loader_time: 0.0046 (0.004637649800002253)  --> TIME: 2025-02-27 09:42:53 -- STEP: 20528/36938 -- GLOBAL_STEP: 1645800 | > loss_text_ce: 0.031382881104946136 (0.02987492108078864) | > loss_mel_ce: 2.621833086013794 (nan) | > loss: 0.01052863523364067 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13689648861171488) | > loader_time: 0.0045 (0.00463754520551914)  --> TIME: 2025-02-27 09:43:03 -- STEP: 20578/36938 -- GLOBAL_STEP: 1645850 | > loss_text_ce: 0.030580027028918266 (0.029874377281245083) | > loss_mel_ce: 2.979581356048584 (nan) | > loss: 0.011945085600018501 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13691008082862635) | > loader_time: 0.0046 (0.004637451199218167)  --> TIME: 2025-02-27 09:43:12 -- STEP: 20628/36938 -- GLOBAL_STEP: 1645900 | > loss_text_ce: 0.031411971896886826 (0.02987430473544055) | > loss_mel_ce: 2.995546340942383 (nan) | > loss: 0.012011739425361156 (nan) | > current_lr: 5e-06 | > step_time: 0.1777 (0.136910053332323) | > loader_time: 0.0047 (0.004637281874341737)  --> TIME: 2025-02-27 09:43:21 -- STEP: 20678/36938 -- GLOBAL_STEP: 1645950 | > loss_text_ce: 0.03177647665143013 (0.029872987682328763) | > loss_mel_ce: 1.8522928953170776 (nan) | > loss: 0.007476465776562691 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13690514961840022) | > loader_time: 0.0045 (0.0046370612409292545)  --> TIME: 2025-02-27 09:43:30 -- STEP: 20728/36938 -- GLOBAL_STEP: 1646000 | > loss_text_ce: 0.021667351946234703 (0.029874170503643955) | > loss_mel_ce: 2.785090446472168 (nan) | > loss: 0.011137927882373333 (nan) | > current_lr: 5e-06 | > step_time: 0.1253 (0.136900797342527) | > loader_time: 0.0047 (0.004636860466610725)  --> TIME: 2025-02-27 09:43:40 -- STEP: 20778/36938 -- GLOBAL_STEP: 1646050 | > loss_text_ce: 0.02486891858279705 (0.029876102250778826) | > loss_mel_ce: 2.616481304168701 (nan) | > loss: 0.010481549426913261 (nan) | > current_lr: 5e-06 | > step_time: 0.1594 (0.1369123306594783) | > loader_time: 0.0046 (0.00463675277640929)  --> TIME: 2025-02-27 09:43:50 -- STEP: 20828/36938 -- GLOBAL_STEP: 1646100 | > loss_text_ce: 0.028942717239260674 (0.029874038199823653) | > loss_mel_ce: 2.5964319705963135 (nan) | > loss: 0.010418154299259186 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.1369280492547425) | > loader_time: 0.0049 (0.004637156644113821)  --> TIME: 2025-02-27 09:43:59 -- STEP: 20878/36938 -- GLOBAL_STEP: 1646150 | > loss_text_ce: 0.03815194591879845 (0.029873591555357945) | > loss_mel_ce: 2.621054172515869 (nan) | > loss: 0.010552406311035156 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.1369191246798286) | > loader_time: 0.0046 (0.004637118614157296)  --> TIME: 2025-02-27 09:44:09 -- STEP: 20928/36938 -- GLOBAL_STEP: 1646200 | > loss_text_ce: 0.03224124386906624 (0.029871803975471116) | > loss_mel_ce: 2.2862231731414795 (nan) | > loss: 0.009200256317853928 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.1369355574409715) | > loader_time: 0.0046 (0.004637091941789733)  --> TIME: 2025-02-27 09:44:18 -- STEP: 20978/36938 -- GLOBAL_STEP: 1646250 | > loss_text_ce: 0.03679700568318367 (0.029869417119115962) | > loss_mel_ce: 2.1561801433563232 (nan) | > loss: 0.008702291175723076 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13693224934834258) | > loader_time: 0.0045 (0.0046370463144427875)  --> TIME: 2025-02-27 09:44:27 -- STEP: 21028/36938 -- GLOBAL_STEP: 1646300 | > loss_text_ce: 0.031093444675207138 (0.02987006023469146) | > loss_mel_ce: 0.02603328973054886 (nan) | > loss: 0.00022669340251013637 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13692447306743843) | > loader_time: 0.0045 (0.004637014611900711)  --> TIME: 2025-02-27 09:44:36 -- STEP: 21078/36938 -- GLOBAL_STEP: 1646350 | > loss_text_ce: 0.02897060476243496 (0.029867935944898224) | > loss_mel_ce: 0.011027018539607525 (nan) | > loss: 0.0001587207370903343 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13692475552915898) | > loader_time: 0.0044 (0.004636981487500379)  --> TIME: 2025-02-27 09:44:46 -- STEP: 21128/36938 -- GLOBAL_STEP: 1646400 | > loss_text_ce: 0.037974510341882706 (0.02986873559442527) | > loss_mel_ce: 2.058983325958252 (nan) | > loss: 0.008321261964738369 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13692530867774497) | > loader_time: 0.0046 (0.00463690864051303)  --> TIME: 2025-02-27 09:44:55 -- STEP: 21178/36938 -- GLOBAL_STEP: 1646450 | > loss_text_ce: 0.029041728004813194 (0.029870413161095816) | > loss_mel_ce: 2.414808511734009 (nan) | > loss: 0.009697819128632545 (nan) | > current_lr: 5e-06 | > step_time: 0.1609 (0.13692920932306873) | > loader_time: 0.0048 (0.004636792220658636)  --> TIME: 2025-02-27 09:45:04 -- STEP: 21228/36938 -- GLOBAL_STEP: 1646500 | > loss_text_ce: 0.0194199588149786 (0.029869664582122545) | > loss_mel_ce: 2.602084159851074 (nan) | > loss: 0.010402794927358627 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13692061209692158) | > loader_time: 0.0046 (0.004636711222496038)  --> TIME: 2025-02-27 09:45:14 -- STEP: 21278/36938 -- GLOBAL_STEP: 1646550 | > loss_text_ce: 0.03069831058382988 (0.029866352452371212) | > loss_mel_ce: 2.4400973320007324 (nan) | > loss: 0.009804745204746723 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13691436660638273) | > loader_time: 0.0046 (0.0046367171743173854)  --> TIME: 2025-02-27 09:45:23 -- STEP: 21328/36938 -- GLOBAL_STEP: 1646600 | > loss_text_ce: 0.031187910586595535 (0.029866874111003732) | > loss_mel_ce: 2.435441255569458 (nan) | > loss: 0.00978821236640215 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13690758352877086) | > loader_time: 0.0044 (0.004636718906233013)  --> TIME: 2025-02-27 09:45:32 -- STEP: 21378/36938 -- GLOBAL_STEP: 1646650 | > loss_text_ce: 0.03513077273964882 (0.02986582448209046) | > loss_mel_ce: 3.0082900524139404 (nan) | > loss: 0.01207706704735756 (nan) | > current_lr: 5e-06 | > step_time: 0.1794 (0.136916939698565) | > loader_time: 0.0048 (0.004636661064436293)  --> TIME: 2025-02-27 09:45:42 -- STEP: 21428/36938 -- GLOBAL_STEP: 1646700 | > loss_text_ce: 0.03176223859190941 (0.029863177828112774) | > loss_mel_ce: 2.399111270904541 (nan) | > loss: 0.009646323509514332 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13691232279874943) | > loader_time: 0.0046 (0.004636559353761301)  --> TIME: 2025-02-27 09:45:51 -- STEP: 21478/36938 -- GLOBAL_STEP: 1646750 | > loss_text_ce: 0.04126375541090965 (0.029863684949013175) | > loss_mel_ce: 2.6457595825195312 (nan) | > loss: 0.01066279225051403 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13691381279655643) | > loader_time: 0.0046 (0.0046365008650350705)  --> TIME: 2025-02-27 09:46:00 -- STEP: 21528/36938 -- GLOBAL_STEP: 1646800 | > loss_text_ce: 0.03129162639379501 (0.029862722990847843) | > loss_mel_ce: 2.4999444484710693 (nan) | > loss: 0.010044588707387447 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13691318833619806) | > loader_time: 0.0046 (0.004636426611666053)  --> TIME: 2025-02-27 09:46:09 -- STEP: 21578/36938 -- GLOBAL_STEP: 1646850 | > loss_text_ce: 0.03292166441679001 (0.029862016931945134) | > loss_mel_ce: 2.3220460414886475 (nan) | > loss: 0.00934511050581932 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1369151708337022) | > loader_time: 0.0045 (0.0046363524703809545)  --> TIME: 2025-02-27 09:46:19 -- STEP: 21628/36938 -- GLOBAL_STEP: 1646900 | > loss_text_ce: 0.03881523013114929 (0.02986221156525369) | > loss_mel_ce: 2.392413854598999 (nan) | > loss: 0.009647735394537449 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1369124161066094) | > loader_time: 0.0047 (0.004636213279859561)  --> TIME: 2025-02-27 09:46:28 -- STEP: 21678/36938 -- GLOBAL_STEP: 1646950 | > loss_text_ce: 0.028322560712695122 (0.029863030194027432) | > loss_mel_ce: 2.8693244457244873 (nan) | > loss: 0.01149859931319952 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.1369085963971834) | > loader_time: 0.0046 (0.004636077063034581)  --> TIME: 2025-02-27 09:46:37 -- STEP: 21728/36938 -- GLOBAL_STEP: 1647000 | > loss_text_ce: 0.01688925176858902 (0.029864696815082886) | > loss_mel_ce: 2.411348342895508 (nan) | > loss: 0.0096358647570014 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13690439901357224) | > loader_time: 0.0046 (0.004635993923459717)  --> TIME: 2025-02-27 09:46:47 -- STEP: 21778/36938 -- GLOBAL_STEP: 1647050 | > loss_text_ce: 0.03549008443951607 (0.02986719776303313) | > loss_mel_ce: 3.1187450885772705 (nan) | > loss: 0.012516806833446026 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.13690940613851332) | > loader_time: 0.0046 (0.004635972209909037)  --> TIME: 2025-02-27 09:46:56 -- STEP: 21828/36938 -- GLOBAL_STEP: 1647100 | > loss_text_ce: 0.03206703066825867 (0.029865953472033493) | > loss_mel_ce: 1.7425246238708496 (nan) | > loss: 0.007042030803859234 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.1368986425413764) | > loader_time: 0.0045 (0.004635912978389166)  --> TIME: 2025-02-27 09:47:05 -- STEP: 21878/36938 -- GLOBAL_STEP: 1647150 | > loss_text_ce: 0.028514472767710686 (0.029866840944383046) | > loss_mel_ce: 2.5945661067962646 (nan) | > loss: 0.01040905062109232 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13689944996717437) | > loader_time: 0.0045 (0.004635916014281696)  --> TIME: 2025-02-27 09:47:15 -- STEP: 21928/36938 -- GLOBAL_STEP: 1647200 | > loss_text_ce: 0.03234342858195305 (0.029868957306994327) | > loss_mel_ce: 1.2532744407653809 (nan) | > loss: 0.005101658403873444 (nan) | > current_lr: 5e-06 | > step_time: 0.1112 (0.13690436039689458) | > loader_time: 0.0045 (0.004635934725768008)  --> TIME: 2025-02-27 09:47:24 -- STEP: 21978/36938 -- GLOBAL_STEP: 1647250 | > loss_text_ce: 0.02963831275701523 (0.029869632093761807) | > loss_mel_ce: 2.75107479095459 (nan) | > loss: 0.01103457622230053 (nan) | > current_lr: 5e-06 | > step_time: 0.1707 (0.13690370873982305) | > loader_time: 0.0048 (0.004635841899187599)  --> TIME: 2025-02-27 09:47:33 -- STEP: 22028/36938 -- GLOBAL_STEP: 1647300 | > loss_text_ce: 0.031095458194613457 (0.029869627653857388) | > loss_mel_ce: 2.491239070892334 (nan) | > loss: 0.010009264573454857 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.136899613297222) | > loader_time: 0.0046 (0.004635733280507437)  --> TIME: 2025-02-27 09:47:42 -- STEP: 22078/36938 -- GLOBAL_STEP: 1647350 | > loss_text_ce: 0.033846255391836166 (0.029868627780998972) | > loss_mel_ce: 2.2594387531280518 (nan) | > loss: 0.009100337512791157 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13690419434226872) | > loader_time: 0.0048 (0.004635673997324207)  --> TIME: 2025-02-27 09:47:52 -- STEP: 22128/36938 -- GLOBAL_STEP: 1647400 | > loss_text_ce: 0.03531134873628616 (0.029870194166338782) | > loss_mel_ce: 2.7540955543518066 (nan) | > loss: 0.011069076135754585 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.1369013745911472) | > loader_time: 0.0046 (0.004635627857602668)  --> TIME: 2025-02-27 09:48:01 -- STEP: 22178/36938 -- GLOBAL_STEP: 1647450 | > loss_text_ce: 0.029947370290756226 (0.029871988688992927) | > loss_mel_ce: 2.528132677078247 (nan) | > loss: 0.010151111520826817 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13690842609472642) | > loader_time: 0.0046 (0.004635594105933358)  --> TIME: 2025-02-27 09:48:10 -- STEP: 22228/36938 -- GLOBAL_STEP: 1647500 | > loss_text_ce: 0.024499816820025444 (0.02986979025240269) | > loss_mel_ce: 2.350264549255371 (nan) | > loss: 0.009423669427633286 (nan) | > current_lr: 5e-06 | > step_time: 0.1788 (0.13691074000388312) | > loader_time: 0.0047 (0.004635537691804502)  --> TIME: 2025-02-27 09:48:20 -- STEP: 22278/36938 -- GLOBAL_STEP: 1647550 | > loss_text_ce: 0.023458391427993774 (0.029868141929861057) | > loss_mel_ce: 2.61183762550354 (nan) | > loss: 0.01045752502977848 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13690902482822281) | > loader_time: 0.0045 (0.004635450602198464)  --> TIME: 2025-02-27 09:48:29 -- STEP: 22328/36938 -- GLOBAL_STEP: 1647600 | > loss_text_ce: 0.025641802698373795 (0.029867675660604512) | > loss_mel_ce: 2.7679386138916016 (nan) | > loss: 0.011085637845098972 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13691517758736824) | > loader_time: 0.0046 (0.004635502684718594)  --> TIME: 2025-02-27 09:48:39 -- STEP: 22378/36938 -- GLOBAL_STEP: 1647650 | > loss_text_ce: 0.026882285252213478 (0.029868037890520022) | > loss_mel_ce: 2.3226184844970703 (nan) | > loss: 0.009323416277766228 (nan) | > current_lr: 5e-06 | > step_time: 0.1818 (0.1369183051361354) | > loader_time: 0.0048 (0.0046354089242244875)  --> TIME: 2025-02-27 09:48:48 -- STEP: 22428/36938 -- GLOBAL_STEP: 1647700 | > loss_text_ce: 0.03927871957421303 (0.029870571173456432) | > loss_mel_ce: 2.404506206512451 (nan) | > loss: 0.009697560220956802 (nan) | > current_lr: 5e-06 | > step_time: 0.1464 (0.1369128948187523) | > loader_time: 0.0046 (0.004635265533864987)  --> TIME: 2025-02-27 09:48:57 -- STEP: 22478/36938 -- GLOBAL_STEP: 1647750 | > loss_text_ce: 0.02792327292263508 (0.029868956879248672) | > loss_mel_ce: 2.9260153770446777 (nan) | > loss: 0.011721979826688766 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13691214101680957) | > loader_time: 0.0046 (0.004635150751425638)  --> TIME: 2025-02-27 09:49:06 -- STEP: 22528/36938 -- GLOBAL_STEP: 1647800 | > loss_text_ce: 0.025878574699163437 (0.02986885997978044) | > loss_mel_ce: 2.8841142654418945 (nan) | > loss: 0.011547591537237167 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13690902893854864) | > loader_time: 0.0048 (0.004635073318654153)  --> TIME: 2025-02-27 09:49:15 -- STEP: 22578/36938 -- GLOBAL_STEP: 1647850 | > loss_text_ce: 0.027585260570049286 (0.029867680753566453) | > loss_mel_ce: 2.9459292888641357 (nan) | > loss: 0.011799661442637444 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13690695022182986) | > loader_time: 0.0048 (0.004634955721546047)  --> TIME: 2025-02-27 09:49:25 -- STEP: 22628/36938 -- GLOBAL_STEP: 1647900 | > loss_text_ce: 0.031085556373000145 (0.02986749996666101) | > loss_mel_ce: 2.5116500854492188 (nan) | > loss: 0.010090220719575882 (nan) | > current_lr: 5e-06 | > step_time: 0.1464 (0.13690221020963442) | > loader_time: 0.0046 (0.00463481839309632)  --> TIME: 2025-02-27 09:49:34 -- STEP: 22678/36938 -- GLOBAL_STEP: 1647950 | > loss_text_ce: 0.03426065295934677 (0.02987008022264517) | > loss_mel_ce: 2.888986825942993 (nan) | > loss: 0.011600189842283726 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13690256408013915) | > loader_time: 0.0046 (0.004634666909657612)  --> TIME: 2025-02-27 09:49:43 -- STEP: 22728/36938 -- GLOBAL_STEP: 1648000 | > loss_text_ce: 0.04458080604672432 (0.029871822909977502) | > loss_mel_ce: 3.3060975074768066 (nan) | > loss: 0.013296342454850674 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13690308193764425) | > loader_time: 0.0045 (0.004634541751464787)  --> TIME: 2025-02-27 09:49:52 -- STEP: 22778/36938 -- GLOBAL_STEP: 1648050 | > loss_text_ce: 0.033360421657562256 (0.029873316090618433) | > loss_mel_ce: 3.1189663410186768 (nan) | > loss: 0.012509234249591827 (nan) | > current_lr: 5e-06 | > step_time: 0.1428 (0.136900539090044) | > loader_time: 0.0046 (0.004634430592907284)  --> TIME: 2025-02-27 09:50:02 -- STEP: 22828/36938 -- GLOBAL_STEP: 1648100 | > loss_text_ce: 0.03028283081948757 (0.02987421710737374) | > loss_mel_ce: 2.726158857345581 (nan) | > loss: 0.010938260704278946 (nan) | > current_lr: 5e-06 | > step_time: 0.1535 (0.13690330886331212) | > loader_time: 0.0046 (0.004634386711494914)  --> TIME: 2025-02-27 09:50:11 -- STEP: 22878/36938 -- GLOBAL_STEP: 1648150 | > loss_text_ce: 0.03210523724555969 (0.029873933347153908) | > loss_mel_ce: 2.973439931869507 (nan) | > loss: 0.01192676741629839 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.1369012252612704) | > loader_time: 0.0046 (0.0046343024309119735)  --> TIME: 2025-02-27 09:50:20 -- STEP: 22928/36938 -- GLOBAL_STEP: 1648200 | > loss_text_ce: 0.031135179102420807 (0.029873843807275407) | > loss_mel_ce: 3.0095889568328857 (nan) | > loss: 0.012066366150975227 (nan) | > current_lr: 5e-06 | > step_time: 0.17 (0.13690108199137946) | > loader_time: 0.0048 (0.004634337789598075)  --> TIME: 2025-02-27 09:50:30 -- STEP: 22978/36938 -- GLOBAL_STEP: 1648250 | > loss_text_ce: 0.030075959861278534 (0.029873453837855304) | > loss_mel_ce: 2.6386077404022217 (nan) | > loss: 0.010590015910565853 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13690124313206162) | > loader_time: 0.0047 (0.004634323636009417)  --> TIME: 2025-02-27 09:50:39 -- STEP: 23028/36938 -- GLOBAL_STEP: 1648300 | > loss_text_ce: 0.03630027547478676 (0.02987343230724285) | > loss_mel_ce: 3.08709979057312 (nan) | > loss: 0.012394445016980171 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.13690276067870355) | > loader_time: 0.0048 (0.004634389658655078)  --> TIME: 2025-02-27 09:50:48 -- STEP: 23078/36938 -- GLOBAL_STEP: 1648350 | > loss_text_ce: 0.022616572678089142 (0.029872228763868414) | > loss_mel_ce: 2.918321132659912 (nan) | > loss: 0.011670388281345367 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13690192318838806) | > loader_time: 0.0045 (0.004634283973104728)  --> TIME: 2025-02-27 09:50:58 -- STEP: 23128/36938 -- GLOBAL_STEP: 1648400 | > loss_text_ce: 0.02880050614476204 (0.0298724186949629) | > loss_mel_ce: 2.112367630004883 (nan) | > loss: 0.008496698923408985 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13690323673577923) | > loader_time: 0.0046 (0.004634295551886569)  --> TIME: 2025-02-27 09:51:07 -- STEP: 23178/36938 -- GLOBAL_STEP: 1648450 | > loss_text_ce: 0.04383491352200508 (0.029875151525251643) | > loss_mel_ce: 0.5038477778434753 (nan) | > loss: 0.002173344139009714 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13689908215808153) | > loader_time: 0.0045 (0.00463429884129227)  --> TIME: 2025-02-27 09:51:16 -- STEP: 23228/36938 -- GLOBAL_STEP: 1648500 | > loss_text_ce: 0.03129102662205696 (0.029875383947868314) | > loss_mel_ce: 3.5013670921325684 (nan) | > loss: 0.014018485322594643 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.136885938647284) | > loader_time: 0.0045 (0.004634163897809789)  --> TIME: 2025-02-27 09:51:25 -- STEP: 23278/36938 -- GLOBAL_STEP: 1648550 | > loss_text_ce: 0.023943904787302017 (0.029874475033742055) | > loss_mel_ce: 2.9736623764038086 (nan) | > loss: 0.011895263567566872 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13688441882357658) | > loader_time: 0.0044 (0.0046340580176668)  --> TIME: 2025-02-27 09:51:34 -- STEP: 23328/36938 -- GLOBAL_STEP: 1648600 | > loss_text_ce: 0.027896499261260033 (0.029873119483127628) | > loss_mel_ce: 3.095493793487549 (nan) | > loss: 0.012394405901432037 (nan) | > current_lr: 5e-06 | > step_time: 0.1885 (0.13687717630457716) | > loader_time: 0.0049 (0.004634010663003095)  --> TIME: 2025-02-27 09:51:44 -- STEP: 23378/36938 -- GLOBAL_STEP: 1648650 | > loss_text_ce: 0.035427261143922806 (0.029872541118498297) | > loss_mel_ce: 2.607724666595459 (nan) | > loss: 0.010488699190318584 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.1368872142549929) | > loader_time: 0.0046 (0.00463404798438341)  --> TIME: 2025-02-27 09:51:53 -- STEP: 23428/36938 -- GLOBAL_STEP: 1648700 | > loss_text_ce: 0.022436032071709633 (0.029873395093540332) | > loss_mel_ce: 2.4310829639434814 (nan) | > loss: 0.009736187756061554 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13688220112971466) | > loader_time: 0.0048 (0.00463400544293081)  --> TIME: 2025-02-27 09:52:02 -- STEP: 23478/36938 -- GLOBAL_STEP: 1648750 | > loss_text_ce: 0.02037046104669571 (0.029874325999648163) | > loss_mel_ce: 2.3278565406799316 (nan) | > loss: 0.00931836199015379 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13687924846634922) | > loader_time: 0.0045 (0.00463388055316715)  --> TIME: 2025-02-27 09:52:11 -- STEP: 23528/36938 -- GLOBAL_STEP: 1648800 | > loss_text_ce: 0.03193458542227745 (0.029877688102861143) | > loss_mel_ce: 2.7565906047821045 (nan) | > loss: 0.01106557622551918 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.13687415935279043) | > loader_time: 0.0048 (0.004633802574777721)  --> TIME: 2025-02-27 09:52:21 -- STEP: 23578/36938 -- GLOBAL_STEP: 1648850 | > loss_text_ce: 0.032180964946746826 (0.02987844853133686) | > loss_mel_ce: 0.22084960341453552 (nan) | > loss: 0.0010040896013379097 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13687368427810245) | > loader_time: 0.0046 (0.004633783566071008)  --> TIME: 2025-02-27 09:52:30 -- STEP: 23628/36938 -- GLOBAL_STEP: 1648900 | > loss_text_ce: 0.037360817193984985 (0.029878366237224993) | > loss_mel_ce: 3.0068819522857666 (nan) | > loss: 0.012080329470336437 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13687369714276532) | > loader_time: 0.0045 (0.0046337707829350325)  --> TIME: 2025-02-27 09:52:40 -- STEP: 23678/36938 -- GLOBAL_STEP: 1648950 | > loss_text_ce: 0.030006803572177887 (0.02987809780404092) | > loss_mel_ce: 3.117246389389038 (nan) | > loss: 0.01248910091817379 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.1368843420946035) | > loader_time: 0.0046 (0.004633768314303901)  --> TIME: 2025-02-27 09:52:49 -- STEP: 23728/36938 -- GLOBAL_STEP: 1649000 | > loss_text_ce: 0.037795476615428925 (0.0298808920407176) | > loss_mel_ce: 2.054724931716919 (nan) | > loss: 0.008303653448820114 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13689541355508972) | > loader_time: 0.0045 (0.004633764881422136)  --> TIME: 2025-02-27 09:52:59 -- STEP: 23778/36938 -- GLOBAL_STEP: 1649050 | > loss_text_ce: 0.021813709288835526 (0.029879612860578712) | > loss_mel_ce: 2.464128017425537 (nan) | > loss: 0.009864848107099533 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13690027529642512) | > loader_time: 0.0045 (0.00463366135484849)  --> TIME: 2025-02-27 09:53:08 -- STEP: 23828/36938 -- GLOBAL_STEP: 1649100 | > loss_text_ce: 0.025916175916790962 (0.029877775662381804) | > loss_mel_ce: 2.7332682609558105 (nan) | > loss: 0.010949145071208477 (nan) | > current_lr: 5e-06 | > step_time: 0.1632 (0.13690546744818796) | > loader_time: 0.0046 (0.004633619258202473)  --> TIME: 2025-02-27 09:53:18 -- STEP: 23878/36938 -- GLOBAL_STEP: 1649150 | > loss_text_ce: 0.03360888734459877 (0.029876819768194604) | > loss_mel_ce: 2.701326370239258 (nan) | > loss: 0.010852918028831482 (nan) | > current_lr: 5e-06 | > step_time: 0.144 (0.13691988659520157) | > loader_time: 0.0048 (0.004633620122996943)  --> TIME: 2025-02-27 09:53:27 -- STEP: 23928/36938 -- GLOBAL_STEP: 1649200 | > loss_text_ce: 0.03332148492336273 (0.029878323696921436) | > loss_mel_ce: 2.449831247329712 (nan) | > loss: 0.009853781200945377 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.13692003754579504) | > loader_time: 0.0047 (0.004633562017228453)  --> TIME: 2025-02-27 09:53:37 -- STEP: 23978/36938 -- GLOBAL_STEP: 1649250 | > loss_text_ce: 0.027123767882585526 (0.029875086763578036) | > loss_mel_ce: 3.0566864013671875 (nan) | > loss: 0.012237342074513435 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13691222896388255) | > loader_time: 0.0046 (0.004633547625556416)  --> TIME: 2025-02-27 09:53:46 -- STEP: 24028/36938 -- GLOBAL_STEP: 1649300 | > loss_text_ce: 0.03415629640221596 (0.029877787300217933) | > loss_mel_ce: 1.7440847158432007 (nan) | > loss: 0.007056512404233217 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13691163958061686) | > loader_time: 0.0045 (0.004633513379259721)  --> TIME: 2025-02-27 09:53:55 -- STEP: 24078/36938 -- GLOBAL_STEP: 1649350 | > loss_text_ce: 0.03366021439433098 (0.029878318330864205) | > loss_mel_ce: 1.8472316265106201 (nan) | > loss: 0.007463856600224972 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.1369154804696955) | > loader_time: 0.0044 (0.004633419774519139)  --> TIME: 2025-02-27 09:54:05 -- STEP: 24128/36938 -- GLOBAL_STEP: 1649400 | > loss_text_ce: 0.02887536771595478 (0.029877972778957148) | > loss_mel_ce: 2.9948196411132812 (nan) | > loss: 0.011998790316283703 (nan) | > current_lr: 5e-06 | > step_time: 0.1763 (0.13691465313539014) | > loader_time: 0.0047 (0.004633319818609914)  --> TIME: 2025-02-27 09:54:14 -- STEP: 24178/36938 -- GLOBAL_STEP: 1649450 | > loss_text_ce: 0.03513272851705551 (0.029877501375677698) | > loss_mel_ce: 0.0380859375 (nan) | > loss: 0.0002905502915382385 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1369076781754505) | > loader_time: 0.0045 (0.0046331955743828075)  --> TIME: 2025-02-27 09:54:23 -- STEP: 24228/36938 -- GLOBAL_STEP: 1649500 | > loss_text_ce: 0.02578897960484028 (0.02987463597737348) | > loss_mel_ce: 2.7361433506011963 (nan) | > loss: 0.01096004992723465 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.13691529353617196) | > loader_time: 0.0046 (0.004633138454135587)  --> TIME: 2025-02-27 09:54:33 -- STEP: 24278/36938 -- GLOBAL_STEP: 1649550 | > loss_text_ce: 0.036886587738990784 (0.029876923545974373) | > loss_mel_ce: 0.1213030144572258 (nan) | > loss: 0.0006277365027926862 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.136911793422519) | > loader_time: 0.0045 (0.0046330475023525644)  --> TIME: 2025-02-27 09:54:42 -- STEP: 24328/36938 -- GLOBAL_STEP: 1649600 | > loss_text_ce: 0.03369900956749916 (0.029876492474577576) | > loss_mel_ce: 3.1764912605285645 (nan) | > loss: 0.012738850899040699 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.1369244726481941) | > loader_time: 0.0048 (0.00463303349316807)  --> TIME: 2025-02-27 09:54:52 -- STEP: 24378/36938 -- GLOBAL_STEP: 1649650 | > loss_text_ce: 0.032958440482616425 (0.029876550509283326) | > loss_mel_ce: 0.005523681640625 (nan) | > loss: 0.00015270683798007667 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13692278347176623) | > loader_time: 0.0045 (0.004632937144349047)  --> TIME: 2025-02-27 09:55:01 -- STEP: 24428/36938 -- GLOBAL_STEP: 1649700 | > loss_text_ce: 0.021296901628375053 (0.029876229015955356) | > loss_mel_ce: 2.516866683959961 (nan) | > loss: 0.010072078555822372 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.13692198928725657) | > loader_time: 0.0047 (0.004632877555907767)  --> TIME: 2025-02-27 09:55:10 -- STEP: 24478/36938 -- GLOBAL_STEP: 1649750 | > loss_text_ce: 0.032496415078639984 (0.029872131078950773) | > loss_mel_ce: 1.5244413614273071 (nan) | > loss: 0.006178325042128563 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13691450329288282) | > loader_time: 0.0045 (0.004632763120802578)  --> TIME: 2025-02-27 09:55:20 -- STEP: 24528/36938 -- GLOBAL_STEP: 1649800 | > loss_text_ce: 0.024917759001255035 (0.0298708204085989) | > loss_mel_ce: 2.4963269233703613 (nan) | > loss: 0.010004940442740917 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13691407059114663) | > loader_time: 0.0045 (0.004632684018824989)  --> TIME: 2025-02-27 09:55:29 -- STEP: 24578/36938 -- GLOBAL_STEP: 1649850 | > loss_text_ce: 0.03259681537747383 (0.02987110564697845) | > loss_mel_ce: 2.4958276748657227 (nan) | > loss: 0.010033431462943554 (nan) | > current_lr: 5e-06 | > step_time: 0.1712 (0.13691643134350096) | > loader_time: 0.0048 (0.004632567338883738)  --> TIME: 2025-02-27 09:55:38 -- STEP: 24628/36938 -- GLOBAL_STEP: 1649900 | > loss_text_ce: 0.026853812858462334 (0.029870366447949916) | > loss_mel_ce: 3.127504825592041 (nan) | > loss: 0.012517296709120274 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13691460285671658) | > loader_time: 0.0046 (0.00463248380538956)  --> TIME: 2025-02-27 09:55:47 -- STEP: 24678/36938 -- GLOBAL_STEP: 1649950 | > loss_text_ce: 0.023659957572817802 (0.029870645777559424) | > loss_mel_ce: 2.806344985961914 (nan) | > loss: 0.011230179108679295 (nan) | > current_lr: 5e-06 | > step_time: 0.1562 (0.13689821362930252) | > loader_time: 0.0046 (0.0046323603232721056)  --> TIME: 2025-02-27 09:55:56 -- STEP: 24728/36938 -- GLOBAL_STEP: 1650000 | > loss_text_ce: 0.028493231162428856 (0.029871905941289578) | > loss_mel_ce: 2.27685546875 (nan) | > loss: 0.009148209355771542 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13689681173180898) | > loader_time: 0.0046 (0.004632343919251747) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1650000.pth  --> TIME: 2025-02-27 09:56:12 -- STEP: 24778/36938 -- GLOBAL_STEP: 1650050 | > loss_text_ce: 0.025476373732089996 (0.029871658975779716) | > loss_mel_ce: 2.86438250541687 (nan) | > loss: 0.011467695236206055 (nan) | > current_lr: 5e-06 | > step_time: 0.1837 (0.13690746435771617) | > loader_time: 0.005 (0.00463266524326351)  --> TIME: 2025-02-27 09:56:22 -- STEP: 24828/36938 -- GLOBAL_STEP: 1650100 | > loss_text_ce: 0.028132610023021698 (0.02986960742377742) | > loss_mel_ce: 2.031482696533203 (nan) | > loss: 0.008173077367246151 (nan) | > current_lr: 5e-06 | > step_time: 0.1793 (0.1369179083622278) | > loader_time: 0.0049 (0.004632817550386931)  --> TIME: 2025-02-27 09:56:31 -- STEP: 24878/36938 -- GLOBAL_STEP: 1650150 | > loss_text_ce: 0.03187238797545433 (0.029870973623536128) | > loss_mel_ce: 2.057898998260498 (nan) | > loss: 0.008292743936181068 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13692257694516205) | > loader_time: 0.0045 (0.00463285397482759)  --> TIME: 2025-02-27 09:56:40 -- STEP: 24928/36938 -- GLOBAL_STEP: 1650200 | > loss_text_ce: 0.023607706651091576 (0.02987119094362549) | > loss_mel_ce: 2.4664347171783447 (nan) | > loss: 0.00988112110644579 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13690967984086377) | > loader_time: 0.0044 (0.00463274387171701)  --> TIME: 2025-02-27 09:56:50 -- STEP: 24978/36938 -- GLOBAL_STEP: 1650250 | > loss_text_ce: 0.030150657519698143 (0.029872479960719095) | > loss_mel_ce: 2.9339377880096436 (nan) | > loss: 0.011762256734073162 (nan) | > current_lr: 5e-06 | > step_time: 0.1469 (0.13691323281136353) | > loader_time: 0.0046 (0.00463277237535011)  --> TIME: 2025-02-27 09:56:59 -- STEP: 25028/36938 -- GLOBAL_STEP: 1650300 | > loss_text_ce: 0.03654305264353752 (0.029873897240625027) | > loss_mel_ce: 2.451651096343994 (nan) | > loss: 0.009873786941170692 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.1369100553512735) | > loader_time: 0.0046 (0.004632762327387797)  --> TIME: 2025-02-27 09:57:08 -- STEP: 25078/36938 -- GLOBAL_STEP: 1650350 | > loss_text_ce: 0.03232070431113243 (0.02987541777852529) | > loss_mel_ce: 3.2952964305877686 (nan) | > loss: 0.013204830698668957 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.1369104431009975) | > loader_time: 0.0048 (0.004632754030766916)  --> TIME: 2025-02-27 09:57:18 -- STEP: 25128/36938 -- GLOBAL_STEP: 1650400 | > loss_text_ce: 0.025850825011730194 (0.029874751784781505) | > loss_mel_ce: 2.2611711025238037 (nan) | > loss: 0.009075484238564968 (nan) | > current_lr: 5e-06 | > step_time: 0.1457 (0.13691118703037347) | > loader_time: 0.0046 (0.0046327624853080215)  --> TIME: 2025-02-27 09:57:27 -- STEP: 25178/36938 -- GLOBAL_STEP: 1650450 | > loss_text_ce: 0.01889578253030777 (0.029873560794187275) | > loss_mel_ce: 2.5016543865203857 (nan) | > loss: 0.010002183727920055 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13691216014548813) | > loader_time: 0.0045 (0.00463273918404271)  --> TIME: 2025-02-27 09:57:37 -- STEP: 25228/36938 -- GLOBAL_STEP: 1650500 | > loss_text_ce: 0.025564320385456085 (0.02987359882458933) | > loss_mel_ce: 2.60335111618042 (nan) | > loss: 0.01043220516294241 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13691962699004132) | > loader_time: 0.0047 (0.0046327409246029625)  --> TIME: 2025-02-27 09:57:46 -- STEP: 25278/36938 -- GLOBAL_STEP: 1650550 | > loss_text_ce: 0.028837284073233604 (0.02987143325424351) | > loss_mel_ce: 3.1260664463043213 (nan) | > loss: 0.01251945924013853 (nan) | > current_lr: 5e-06 | > step_time: 0.1392 (0.13691056895193865) | > loader_time: 0.0048 (0.004632697366481637)  --> TIME: 2025-02-27 09:57:55 -- STEP: 25328/36938 -- GLOBAL_STEP: 1650600 | > loss_text_ce: 0.030693527311086655 (0.029870578276223057) | > loss_mel_ce: 1.868499755859375 (nan) | > loss: 0.007536481600254774 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13690880564143282) | > loader_time: 0.0046 (0.004632713067252968)  --> TIME: 2025-02-27 09:58:04 -- STEP: 25378/36938 -- GLOBAL_STEP: 1650650 | > loss_text_ce: 0.022659746930003166 (0.029870880178974924) | > loss_mel_ce: 2.4907357692718506 (nan) | > loss: 0.009973792359232903 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13690645489552597) | > loader_time: 0.0046 (0.004632725117383019)  --> TIME: 2025-02-27 09:58:14 -- STEP: 25428/36938 -- GLOBAL_STEP: 1650700 | > loss_text_ce: 0.02927054651081562 (0.029871139784285815) | > loss_mel_ce: 2.492158889770508 (nan) | > loss: 0.01000567339360714 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13690803510242316) | > loader_time: 0.0045 (0.004632822753161564)  --> TIME: 2025-02-27 09:58:23 -- STEP: 25478/36938 -- GLOBAL_STEP: 1650750 | > loss_text_ce: 0.03958364576101303 (0.02987394858408941) | > loss_mel_ce: 2.8242764472961426 (nan) | > loss: 0.011364525184035301 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13691151947828875) | > loader_time: 0.0046 (0.004632742459774591)  --> TIME: 2025-02-27 09:58:33 -- STEP: 25528/36938 -- GLOBAL_STEP: 1650800 | > loss_text_ce: 0.03338833153247833 (0.029873623213876238) | > loss_mel_ce: 2.31305193901062 (nan) | > loss: 0.009311271831393242 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13691516937375775) | > loader_time: 0.0045 (0.004632674967820513)  --> TIME: 2025-02-27 09:58:42 -- STEP: 25578/36938 -- GLOBAL_STEP: 1650850 | > loss_text_ce: 0.034655001014471054 (0.029872704264042628) | > loss_mel_ce: 2.7033767700195312 (nan) | > loss: 0.010865206830203533 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13692256417954246) | > loader_time: 0.0045 (0.00463262276556637)  --> TIME: 2025-02-27 09:58:51 -- STEP: 25628/36938 -- GLOBAL_STEP: 1650900 | > loss_text_ce: 0.031149551272392273 (0.029871276750032188) | > loss_mel_ce: 1.8320090770721436 (nan) | > loss: 0.007393486797809601 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.136920880932732) | > loader_time: 0.0045 (0.004632561017407537)  --> TIME: 2025-02-27 09:59:01 -- STEP: 25678/36938 -- GLOBAL_STEP: 1650950 | > loss_text_ce: 0.03411475196480751 (0.029873884875664357) | > loss_mel_ce: 3.0853769779205322 (nan) | > loss: 0.012378936633467674 (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.13692382243816326) | > loader_time: 0.0047 (0.004632682933624662)  --> TIME: 2025-02-27 09:59:10 -- STEP: 25728/36938 -- GLOBAL_STEP: 1651000 | > loss_text_ce: 0.02591530792415142 (0.029875401220205773) | > loss_mel_ce: 0.052303314208984375 (nan) | > loss: 0.0003103913913946599 (nan) | > current_lr: 5e-06 | > step_time: 0.1082 (0.13692573358800647) | > loader_time: 0.0045 (0.004632762878836698)  --> TIME: 2025-02-27 09:59:19 -- STEP: 25778/36938 -- GLOBAL_STEP: 1651050 | > loss_text_ce: 0.04651911184191704 (0.029875507643230995) | > loss_mel_ce: 2.48002552986145 (nan) | > loss: 0.01002597063779831 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13691292176079117) | > loader_time: 0.0046 (0.004632845714044364)  --> TIME: 2025-02-27 09:59:28 -- STEP: 25828/36938 -- GLOBAL_STEP: 1651100 | > loss_text_ce: 0.02648630365729332 (0.02987383875718727) | > loss_mel_ce: 3.2858643531799316 (nan) | > loss: 0.013144250027835369 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.13690681492228193) | > loader_time: 0.0049 (0.004632930813216769)  --> TIME: 2025-02-27 09:59:38 -- STEP: 25878/36938 -- GLOBAL_STEP: 1651150 | > loss_text_ce: 0.021833090111613274 (0.02987384538888019) | > loss_mel_ce: 2.9052417278289795 (nan) | > loss: 0.01161537691950798 (nan) | > current_lr: 5e-06 | > step_time: 0.171 (0.13691005148284396) | > loader_time: 0.0067 (0.004633071618080525)  --> TIME: 2025-02-27 09:59:47 -- STEP: 25928/36938 -- GLOBAL_STEP: 1651200 | > loss_text_ce: 0.027801303192973137 (0.029872827999000266) | > loss_mel_ce: 3.3860127925872803 (nan) | > loss: 0.013546882197260857 (nan) | > current_lr: 5e-06 | > step_time: 0.1761 (0.13691205815267038) | > loader_time: 0.0049 (0.004633092183236992)  --> TIME: 2025-02-27 09:59:56 -- STEP: 25978/36938 -- GLOBAL_STEP: 1651250 | > loss_text_ce: 0.029422108083963394 (0.029871852034418958) | > loss_mel_ce: 2.5852298736572266 (nan) | > loss: 0.01037560310214758 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.1368986539986079) | > loader_time: 0.0046 (0.004633083245487918)  --> TIME: 2025-02-27 10:00:06 -- STEP: 26028/36938 -- GLOBAL_STEP: 1651300 | > loss_text_ce: 0.018936576321721077 (0.029873898767830982) | > loss_mel_ce: 2.8396382331848145 (nan) | > loss: 0.01134355179965496 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13690390274677527) | > loader_time: 0.0045 (0.004633296840436755)  --> TIME: 2025-02-27 10:00:15 -- STEP: 26078/36938 -- GLOBAL_STEP: 1651350 | > loss_text_ce: 0.034316595643758774 (0.029874249618106623) | > loss_mel_ce: 3.134504556655884 (nan) | > loss: 0.012574687600135803 (nan) | > current_lr: 5e-06 | > step_time: 0.1583 (0.1368993305887531) | > loader_time: 0.0047 (0.004633424106354795)  --> TIME: 2025-02-27 10:00:24 -- STEP: 26128/36938 -- GLOBAL_STEP: 1651400 | > loss_text_ce: 0.025252798572182655 (0.029872482399096473) | > loss_mel_ce: 3.2966015338897705 (nan) | > loss: 0.013181962072849274 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13690114795103195) | > loader_time: 0.0047 (0.004633487922533561)  --> TIME: 2025-02-27 10:00:34 -- STEP: 26178/36938 -- GLOBAL_STEP: 1651450 | > loss_text_ce: 0.030003318563103676 (0.029872818602955147) | > loss_mel_ce: 1.847064733505249 (nan) | > loss: 0.007448683027178049 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1369058843857196) | > loader_time: 0.0046 (0.004633568207368256)  --> TIME: 2025-02-27 10:00:43 -- STEP: 26228/36938 -- GLOBAL_STEP: 1651500 | > loss_text_ce: 0.03254231438040733 (0.029872620645962338) | > loss_mel_ce: 2.6570136547088623 (nan) | > loss: 0.010672841221094131 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13690674924915958) | > loader_time: 0.0047 (0.004633619706405638)  --> TIME: 2025-02-27 10:00:53 -- STEP: 26278/36938 -- GLOBAL_STEP: 1651550 | > loss_text_ce: 0.026848388835787773 (0.029875203704564087) | > loss_mel_ce: 3.1439342498779297 (nan) | > loss: 0.01258247159421444 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.13691688157332857) | > loader_time: 0.0047 (0.004633712010057349)  --> TIME: 2025-02-27 10:01:02 -- STEP: 26328/36938 -- GLOBAL_STEP: 1651600 | > loss_text_ce: 0.03009120561182499 (0.02987192220030017) | > loss_mel_ce: 3.228858470916748 (nan) | > loss: 0.012932340614497662 (nan) | > current_lr: 5e-06 | > step_time: 0.1695 (0.13691280104403328) | > loader_time: 0.0047 (0.004633746459396448)  --> TIME: 2025-02-27 10:01:11 -- STEP: 26378/36938 -- GLOBAL_STEP: 1651650 | > loss_text_ce: 0.028330670669674873 (0.029872436258250295) | > loss_mel_ce: 2.504183769226074 (nan) | > loss: 0.010049660690128803 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1369037386392917) | > loader_time: 0.0046 (0.004633764400304306)  --> TIME: 2025-02-27 10:01:21 -- STEP: 26428/36938 -- GLOBAL_STEP: 1651700 | > loss_text_ce: 0.03118225559592247 (0.029871282602400424) | > loss_mel_ce: 2.771932363510132 (nan) | > loss: 0.011123471893370152 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.1369137843907002) | > loader_time: 0.0047 (0.0046339135893863235)  --> TIME: 2025-02-27 10:01:30 -- STEP: 26478/36938 -- GLOBAL_STEP: 1651750 | > loss_text_ce: 0.029148222878575325 (0.029871757071001946) | > loss_mel_ce: 3.1521525382995605 (nan) | > loss: 0.012624210678040981 (nan) | > current_lr: 5e-06 | > step_time: 0.1596 (0.1369090775311094) | > loader_time: 0.0046 (0.004633912417776816)  --> TIME: 2025-02-27 10:01:39 -- STEP: 26528/36938 -- GLOBAL_STEP: 1651800 | > loss_text_ce: 0.03511177375912666 (0.029872075692710207) | > loss_mel_ce: 2.809656858444214 (nan) | > loss: 0.011288764886558056 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1369027715306668) | > loader_time: 0.0046 (0.004633914072637257)  --> TIME: 2025-02-27 10:01:49 -- STEP: 26578/36938 -- GLOBAL_STEP: 1651850 | > loss_text_ce: 0.02940036542713642 (0.02987037373108406) | > loss_mel_ce: 1.840978741645813 (nan) | > loss: 0.007422139868140221 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.1369017098592097) | > loader_time: 0.0048 (0.004634001443595824)  --> TIME: 2025-02-27 10:01:58 -- STEP: 26628/36938 -- GLOBAL_STEP: 1651900 | > loss_text_ce: 0.02917124144732952 (0.02987058399136649) | > loss_mel_ce: 2.8366739749908447 (nan) | > loss: 0.011372402310371399 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13691087975701915) | > loader_time: 0.0046 (0.004634245596652617)  --> TIME: 2025-02-27 10:02:07 -- STEP: 26678/36938 -- GLOBAL_STEP: 1651950 | > loss_text_ce: 0.03364808112382889 (0.029870330465294092) | > loss_mel_ce: 2.4783666133880615 (nan) | > loss: 0.009968312457203865 (nan) | > current_lr: 5e-06 | > step_time: 0.169 (0.1369062525658198) | > loader_time: 0.0049 (0.0046343707780947575)  --> TIME: 2025-02-27 10:02:17 -- STEP: 26728/36938 -- GLOBAL_STEP: 1652000 | > loss_text_ce: 0.03582294285297394 (0.029870923165006426) | > loss_mel_ce: 2.448629140853882 (nan) | > loss: 0.009858937002718449 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13690493025490044) | > loader_time: 0.0046 (0.004634425664012303)  --> TIME: 2025-02-27 10:02:26 -- STEP: 26778/36938 -- GLOBAL_STEP: 1652050 | > loss_text_ce: 0.02803061716258526 (0.029869813337192838) | > loss_mel_ce: 2.872251510620117 (nan) | > loss: 0.011509057134389877 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.13690529890685138) | > loader_time: 0.0047 (0.004634500182015978)  --> TIME: 2025-02-27 10:02:36 -- STEP: 26828/36938 -- GLOBAL_STEP: 1652100 | > loss_text_ce: 0.029551777988672256 (0.029869907532384124) | > loss_mel_ce: 2.7616569995880127 (nan) | > loss: 0.011076225899159908 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13691661527442328) | > loader_time: 0.0047 (0.004634584517810972)  --> TIME: 2025-02-27 10:02:45 -- STEP: 26878/36938 -- GLOBAL_STEP: 1652150 | > loss_text_ce: 0.030195042490959167 (0.029869694539492505) | > loss_mel_ce: 2.8531975746154785 (nan) | > loss: 0.011442034505307674 (nan) | > current_lr: 5e-06 | > step_time: 0.1354 (0.1369211769802995) | > loader_time: 0.0047 (0.004634692179446203)  --> TIME: 2025-02-27 10:02:54 -- STEP: 26928/36938 -- GLOBAL_STEP: 1652200 | > loss_text_ce: 0.036143213510513306 (0.029868820736571242) | > loss_mel_ce: 2.706794261932373 (nan) | > loss: 0.010884673334658146 (nan) | > current_lr: 5e-06 | > step_time: 0.147 (0.1369163713866735) | > loader_time: 0.0046 (0.004634688607795012)  --> TIME: 2025-02-27 10:03:04 -- STEP: 26978/36938 -- GLOBAL_STEP: 1652250 | > loss_text_ce: 0.024467144161462784 (0.02987024300060688) | > loss_mel_ce: 2.642611503601074 (nan) | > loss: 0.010583646595478058 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.13692071329311004) | > loader_time: 0.0048 (0.004634651077961114)  --> TIME: 2025-02-27 10:03:13 -- STEP: 27028/36938 -- GLOBAL_STEP: 1652300 | > loss_text_ce: 0.02394757978618145 (0.029871640668724893) | > loss_mel_ce: 1.786865234375 (nan) | > loss: 0.007185765542089939 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13691654074945425) | > loader_time: 0.0047 (0.004634554585144515)  --> TIME: 2025-02-27 10:03:22 -- STEP: 27078/36938 -- GLOBAL_STEP: 1652350 | > loss_text_ce: 0.0389309898018837 (0.02987026030961838) | > loss_mel_ce: 0.012210667133331299 (nan) | > loss: 0.00020294309069868177 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.1369186407248852) | > loader_time: 0.0046 (0.004634678517885486)  --> TIME: 2025-02-27 10:03:32 -- STEP: 27128/36938 -- GLOBAL_STEP: 1652400 | > loss_text_ce: 0.021214807406067848 (0.02986772754081983) | > loss_mel_ce: 1.982974648475647 (nan) | > loss: 0.007953133434057236 (nan) | > current_lr: 5e-06 | > step_time: 0.1355 (0.13692164588944858) | > loader_time: 0.0047 (0.004634775873905307)  --> TIME: 2025-02-27 10:03:41 -- STEP: 27178/36938 -- GLOBAL_STEP: 1652450 | > loss_text_ce: 0.027627240866422653 (0.029868162200510043) | > loss_mel_ce: 0.9292572140693665 (nan) | > loss: 0.003797160694375634 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.136921558214152) | > loader_time: 0.0046 (0.004634859107680938)  --> TIME: 2025-02-27 10:03:50 -- STEP: 27228/36938 -- GLOBAL_STEP: 1652500 | > loss_text_ce: 0.02126333862543106 (0.029867048384680565) | > loss_mel_ce: 2.1939697265625 (nan) | > loss: 0.008790607564151287 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13691500114312355) | > loader_time: 0.0045 (0.004634896949189735)  --> TIME: 2025-02-27 10:04:00 -- STEP: 27278/36938 -- GLOBAL_STEP: 1652550 | > loss_text_ce: 0.03502609580755234 (0.02986729490135824) | > loss_mel_ce: 2.4398252964019775 (nan) | > loss: 0.009820839390158653 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13691776910042547) | > loader_time: 0.0045 (0.004634876922128042)  --> TIME: 2025-02-27 10:04:09 -- STEP: 27328/36938 -- GLOBAL_STEP: 1652600 | > loss_text_ce: 0.020254172384738922 (0.029866731345238578) | > loss_mel_ce: 1.3543214797973633 (nan) | > loss: 0.005454665515571833 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13691900362559492) | > loader_time: 0.0046 (0.004634923918883761)  --> TIME: 2025-02-27 10:04:18 -- STEP: 27378/36938 -- GLOBAL_STEP: 1652650 | > loss_text_ce: 0.033762987703084946 (0.02986533685889895) | > loss_mel_ce: 3.3511719703674316 (nan) | > loss: 0.013432282023131847 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.1369152958916126) | > loader_time: 0.0046 (0.0046348399438105045)  --> TIME: 2025-02-27 10:04:27 -- STEP: 27428/36938 -- GLOBAL_STEP: 1652700 | > loss_text_ce: 0.031072262674570084 (0.0298643993550676) | > loss_mel_ce: 2.903776168823242 (nan) | > loss: 0.011646225117146969 (nan) | > current_lr: 5e-06 | > step_time: 0.1716 (0.1369104463695632) | > loader_time: 0.0046 (0.004634754527705193)  --> TIME: 2025-02-27 10:04:37 -- STEP: 27478/36938 -- GLOBAL_STEP: 1652750 | > loss_text_ce: 0.023208456113934517 (0.029864259080093537) | > loss_mel_ce: 2.469683885574341 (nan) | > loss: 0.009892430156469345 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.13691377144330527) | > loader_time: 0.0047 (0.004634762245873481)  --> TIME: 2025-02-27 10:04:46 -- STEP: 27528/36938 -- GLOBAL_STEP: 1652800 | > loss_text_ce: 0.03624085709452629 (0.029862160393089945) | > loss_mel_ce: 2.6419901847839355 (nan) | > loss: 0.010627901181578636 (nan) | > current_lr: 5e-06 | > step_time: 0.1337 (0.13691183447456787) | > loader_time: 0.0045 (0.004634684331193297)  --> TIME: 2025-02-27 10:04:56 -- STEP: 27578/36938 -- GLOBAL_STEP: 1652850 | > loss_text_ce: 0.020889103412628174 (0.02986239055341812) | > loss_mel_ce: 2.336372137069702 (nan) | > loss: 0.009354211390018463 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13691268956067335) | > loader_time: 0.0045 (0.004634652164383714)  --> TIME: 2025-02-27 10:05:05 -- STEP: 27628/36938 -- GLOBAL_STEP: 1652900 | > loss_text_ce: 0.0330832377076149 (0.029861116604721735) | > loss_mel_ce: 0.0344492606818676 (nan) | > loss: 0.0002679860917851329 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.13690448809208564) | > loader_time: 0.0046 (0.004634559749952886)  --> TIME: 2025-02-27 10:05:14 -- STEP: 27678/36938 -- GLOBAL_STEP: 1652950 | > loss_text_ce: 0.0467400923371315 (0.029861840014074395) | > loss_mel_ce: 0.00036056837416253984 (nan) | > loss: 0.0001869073894340545 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13690002663682208) | > loader_time: 0.0045 (0.0046344962679277825)  --> TIME: 2025-02-27 10:05:23 -- STEP: 27728/36938 -- GLOBAL_STEP: 1653000 | > loss_text_ce: 0.03079041838645935 (0.029861187988306958) | > loss_mel_ce: 2.255199909210205 (nan) | > loss: 0.009071390144526958 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13688708916607484) | > loader_time: 0.0047 (0.004634398818703943)  --> TIME: 2025-02-27 10:05:32 -- STEP: 27778/36938 -- GLOBAL_STEP: 1653050 | > loss_text_ce: 0.03217683359980583 (0.029859090707909167) | > loss_mel_ce: 1.396124005317688 (nan) | > loss: 0.005667861085385084 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13688698204781313) | > loader_time: 0.0045 (0.0046343590461527286)  --> TIME: 2025-02-27 10:05:41 -- STEP: 27828/36938 -- GLOBAL_STEP: 1653100 | > loss_text_ce: 0.021691298112273216 (0.029861355892442664) | > loss_mel_ce: 1.0788774490356445 (nan) | > loss: 0.004367336630821228 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13687310545064477) | > loader_time: 0.0045 (0.004634262741991422)  --> TIME: 2025-02-27 10:05:50 -- STEP: 27878/36938 -- GLOBAL_STEP: 1653150 | > loss_text_ce: 0.027984516695141792 (0.02986148277570084) | > loss_mel_ce: 2.567845344543457 (nan) | > loss: 0.010300912894308567 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13687516267094957) | > loader_time: 0.0045 (0.004634248106267049)  --> TIME: 2025-02-27 10:06:00 -- STEP: 27928/36938 -- GLOBAL_STEP: 1653200 | > loss_text_ce: 0.03361652418971062 (0.02985959166808433) | > loss_mel_ce: 2.7205567359924316 (nan) | > loss: 0.010929259471595287 (nan) | > current_lr: 5e-06 | > step_time: 0.1712 (0.13687550286548764) | > loader_time: 0.0049 (0.004634295295965578)  --> TIME: 2025-02-27 10:06:09 -- STEP: 27978/36938 -- GLOBAL_STEP: 1653250 | > loss_text_ce: 0.031091079115867615 (0.029859516540694964) | > loss_mel_ce: 2.673037052154541 (nan) | > loss: 0.010730667039752007 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.1368855625866092) | > loader_time: 0.0046 (0.004634301157452407)  --> TIME: 2025-02-27 10:06:19 -- STEP: 28028/36938 -- GLOBAL_STEP: 1653300 | > loss_text_ce: 0.013368570245802402 (0.02985825005707276) | > loss_mel_ce: 1.6916046142578125 (nan) | > loss: 0.006765767000615597 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1368847619165315) | > loader_time: 0.0045 (0.004634340564449601)  --> TIME: 2025-02-27 10:06:28 -- STEP: 28078/36938 -- GLOBAL_STEP: 1653350 | > loss_text_ce: 0.03584025055170059 (0.029857466885640515) | > loss_mel_ce: 2.4862756729125977 (nan) | > loss: 0.01000839751213789 (nan) | > current_lr: 5e-06 | > step_time: 0.1831 (0.1368900368344253) | > loader_time: 0.0048 (0.004634396626879864)  --> TIME: 2025-02-27 10:06:38 -- STEP: 28128/36938 -- GLOBAL_STEP: 1653400 | > loss_text_ce: 0.020343076437711716 (0.029857684103132352) | > loss_mel_ce: 2.4900126457214355 (nan) | > loss: 0.009961729869246483 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13689979440421415) | > loader_time: 0.0045 (0.004634414796332869)  --> TIME: 2025-02-27 10:06:48 -- STEP: 28178/36938 -- GLOBAL_STEP: 1653450 | > loss_text_ce: 0.026256047189235687 (0.02985814947034589) | > loss_mel_ce: 2.7868731021881104 (nan) | > loss: 0.011163211427628994 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.13691178050302866) | > loader_time: 0.0046 (0.00463440539407227)  --> TIME: 2025-02-27 10:06:57 -- STEP: 28228/36938 -- GLOBAL_STEP: 1653500 | > loss_text_ce: 0.02535569667816162 (0.02985735722910256) | > loss_mel_ce: 2.686887264251709 (nan) | > loss: 0.01076287031173706 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13691883164356272) | > loader_time: 0.0045 (0.0046344394300043895)  --> TIME: 2025-02-27 10:07:06 -- STEP: 28278/36938 -- GLOBAL_STEP: 1653550 | > loss_text_ce: 0.027569880709052086 (0.029859875183630487) | > loss_mel_ce: 2.6273865699768066 (nan) | > loss: 0.01053554192185402 (nan) | > current_lr: 5e-06 | > step_time: 0.1779 (0.1369186950561244) | > loader_time: 0.0049 (0.004634502779030308)  --> TIME: 2025-02-27 10:07:16 -- STEP: 28328/36938 -- GLOBAL_STEP: 1653600 | > loss_text_ce: 0.03347714617848396 (0.029860897622277928) | > loss_mel_ce: 2.5503993034362793 (nan) | > loss: 0.010253477841615677 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13692206407764937) | > loader_time: 0.0047 (0.004634621460806086)  --> TIME: 2025-02-27 10:07:25 -- STEP: 28378/36938 -- GLOBAL_STEP: 1653650 | > loss_text_ce: 0.02653023973107338 (0.029860568187840213) | > loss_mel_ce: 2.2619340419769287 (nan) | > loss: 0.009081208147108555 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.13692629601118853) | > loader_time: 0.005 (0.004634620951954255)  --> TIME: 2025-02-27 10:07:35 -- STEP: 28428/36938 -- GLOBAL_STEP: 1653700 | > loss_text_ce: 0.026623617857694626 (0.029860894208151776) | > loss_mel_ce: 2.9511563777923584 (nan) | > loss: 0.01181658823043108 (nan) | > current_lr: 5e-06 | > step_time: 0.1697 (0.13693162270096895) | > loader_time: 0.0053 (0.004634761510057033)  --> TIME: 2025-02-27 10:07:44 -- STEP: 28478/36938 -- GLOBAL_STEP: 1653750 | > loss_text_ce: 0.03156202659010887 (0.029862473885575766) | > loss_mel_ce: 2.159501791000366 (nan) | > loss: 0.008694698102772236 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.1369277514704114) | > loader_time: 0.0046 (0.004634816405963889)  --> TIME: 2025-02-27 10:07:53 -- STEP: 28528/36938 -- GLOBAL_STEP: 1653800 | > loss_text_ce: 0.03667186200618744 (0.02986338974956179) | > loss_mel_ce: 2.918865203857422 (nan) | > loss: 0.011728322133421898 (nan) | > current_lr: 5e-06 | > step_time: 0.113 (0.13692067664657812) | > loader_time: 0.0045 (0.0046348165860687005)  --> TIME: 2025-02-27 10:08:03 -- STEP: 28578/36938 -- GLOBAL_STEP: 1653850 | > loss_text_ce: 0.027619682252407074 (0.02986200250156885) | > loss_mel_ce: 2.469541311264038 (nan) | > loss: 0.009909369051456451 (nan) | > current_lr: 5e-06 | > step_time: 0.1755 (0.13692049711861376) | > loader_time: 0.0051 (0.004634845180886576)  --> TIME: 2025-02-27 10:08:12 -- STEP: 28628/36938 -- GLOBAL_STEP: 1653900 | > loss_text_ce: 0.029914988204836845 (0.02986130582401496) | > loss_mel_ce: 2.3127601146698 (nan) | > loss: 0.009296330623328686 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13691721848474173) | > loader_time: 0.0046 (0.0046348810962113735)  --> TIME: 2025-02-27 10:08:21 -- STEP: 28678/36938 -- GLOBAL_STEP: 1653950 | > loss_text_ce: 0.03600937873125076 (0.029861317913505412) | > loss_mel_ce: 2.8177521228790283 (nan) | > loss: 0.011324450373649597 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.1369179097900618) | > loader_time: 0.0047 (0.004634890806409764)  --> TIME: 2025-02-27 10:08:30 -- STEP: 28728/36938 -- GLOBAL_STEP: 1654000 | > loss_text_ce: 0.03451943397521973 (0.029858904444100734) | > loss_mel_ce: 2.877516984939575 (nan) | > loss: 0.011555700562894344 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.13690759881862027) | > loader_time: 0.0047 (0.004634895187936708)  --> TIME: 2025-02-27 10:08:39 -- STEP: 28778/36938 -- GLOBAL_STEP: 1654050 | > loss_text_ce: 0.021319372579455376 (0.02985767306562478) | > loss_mel_ce: 2.3757498264312744 (nan) | > loss: 0.009512179531157017 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13690009652487145) | > loader_time: 0.0046 (0.004634985873064422)  --> TIME: 2025-02-27 10:08:49 -- STEP: 28828/36938 -- GLOBAL_STEP: 1654100 | > loss_text_ce: 0.02917984127998352 (0.02985813224894479) | > loss_mel_ce: 0.02314249612390995 (nan) | > loss: 0.00020762832718901336 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.1369009863673627) | > loader_time: 0.0049 (0.004635000933518415)  --> TIME: 2025-02-27 10:08:58 -- STEP: 28878/36938 -- GLOBAL_STEP: 1654150 | > loss_text_ce: 0.024198804050683975 (0.029859451859724644) | > loss_mel_ce: 2.3886172771453857 (nan) | > loss: 0.009574667550623417 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.13690230203918263) | > loader_time: 0.0047 (0.0046350522272160435)  --> TIME: 2025-02-27 10:09:07 -- STEP: 28928/36938 -- GLOBAL_STEP: 1654200 | > loss_text_ce: 0.03271850571036339 (0.029861227564354123) | > loss_mel_ce: 1.7679073810577393 (nan) | > loss: 0.007145341485738754 (nan) | > current_lr: 5e-06 | > step_time: 0.1067 (0.13689956942622677) | > loader_time: 0.0044 (0.004635051296677749)  --> TIME: 2025-02-27 10:09:16 -- STEP: 28978/36938 -- GLOBAL_STEP: 1654250 | > loss_text_ce: 0.0322360023856163 (0.029861475609719725) | > loss_mel_ce: 2.1254677772521973 (nan) | > loss: 0.008562317118048668 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.1368973122446738) | > loader_time: 0.0046 (0.004635036867905122)  --> TIME: 2025-02-27 10:09:26 -- STEP: 29028/36938 -- GLOBAL_STEP: 1654300 | > loss_text_ce: 0.03223603591322899 (0.029862384017985154) | > loss_mel_ce: 2.2209887504577637 (nan) | > loss: 0.008941369131207466 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13689434223633717) | > loader_time: 0.0046 (0.004634977405485554)  --> TIME: 2025-02-27 10:09:35 -- STEP: 29078/36938 -- GLOBAL_STEP: 1654350 | > loss_text_ce: 0.03261931985616684 (0.029860306223125452) | > loss_mel_ce: 1.846533179283142 (nan) | > loss: 0.007456955034285784 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13689404152489046) | > loader_time: 0.0046 (0.004634921353476137)  --> TIME: 2025-02-27 10:09:44 -- STEP: 29128/36938 -- GLOBAL_STEP: 1654400 | > loss_text_ce: 0.03153636306524277 (0.029859734485211707) | > loss_mel_ce: 3.3173346519470215 (nan) | > loss: 0.013289171271026134 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13688724838551275) | > loader_time: 0.0046 (0.004634861884225838)  --> TIME: 2025-02-27 10:09:53 -- STEP: 29178/36938 -- GLOBAL_STEP: 1654450 | > loss_text_ce: 0.026831217110157013 (0.029859042480412146) | > loss_mel_ce: 2.2735631465911865 (nan) | > loss: 0.009128549136221409 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.136879850515763) | > loader_time: 0.0045 (0.004634781782291609)  --> TIME: 2025-02-27 10:10:03 -- STEP: 29228/36938 -- GLOBAL_STEP: 1654500 | > loss_text_ce: 0.02299133874475956 (0.029858569773001355) | > loss_mel_ce: 2.702817678451538 (nan) | > loss: 0.010816703550517559 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.13688975888269647) | > loader_time: 0.0048 (0.004634884398303651)  --> TIME: 2025-02-27 10:10:12 -- STEP: 29278/36938 -- GLOBAL_STEP: 1654550 | > loss_text_ce: 0.033814385533332825 (0.029859414001630565) | > loss_mel_ce: 2.923086166381836 (nan) | > loss: 0.011733733117580414 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.1368885732460924) | > loader_time: 0.005 (0.00463487416458991)  --> TIME: 2025-02-27 10:10:22 -- STEP: 29328/36938 -- GLOBAL_STEP: 1654600 | > loss_text_ce: 0.03154658153653145 (0.029858801271505392) | > loss_mel_ce: 2.1220531463623047 (nan) | > loss: 0.008546031080186367 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.1368986882931929) | > loader_time: 0.0047 (0.004634924188248146)  --> TIME: 2025-02-27 10:10:31 -- STEP: 29378/36938 -- GLOBAL_STEP: 1654650 | > loss_text_ce: 0.025604454800486565 (0.029859363092815863) | > loss_mel_ce: 2.8735361099243164 (nan) | > loss: 0.011504527181386948 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13690016354734813) | > loader_time: 0.0046 (0.004634940321527119)  --> TIME: 2025-02-27 10:10:41 -- STEP: 29428/36938 -- GLOBAL_STEP: 1654700 | > loss_text_ce: 0.030236493796110153 (0.02985969633668161) | > loss_mel_ce: 3.1456539630889893 (nan) | > loss: 0.01260274089872837 (nan) | > current_lr: 5e-06 | > step_time: 0.1498 (0.13689779737842514) | > loader_time: 0.0046 (0.004634997330072292)  --> TIME: 2025-02-27 10:10:50 -- STEP: 29478/36938 -- GLOBAL_STEP: 1654750 | > loss_text_ce: 0.027430374175310135 (0.029860293455599478) | > loss_mel_ce: 2.228476047515869 (nan) | > loss: 0.008952009491622448 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13689645689502267) | > loader_time: 0.0046 (0.004635032728153662)  --> TIME: 2025-02-27 10:10:59 -- STEP: 29528/36938 -- GLOBAL_STEP: 1654800 | > loss_text_ce: 0.028925875201821327 (0.02986083760964125) | > loss_mel_ce: 3.0516231060028076 (nan) | > loss: 0.012224401347339153 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13688579815346788) | > loader_time: 0.0046 (0.00463506587473432)  --> TIME: 2025-02-27 10:11:08 -- STEP: 29578/36938 -- GLOBAL_STEP: 1654850 | > loss_text_ce: 0.03070688806474209 (0.029861467778426402) | > loss_mel_ce: 2.7773656845092773 (nan) | > loss: 0.011143146082758904 (nan) | > current_lr: 5e-06 | > step_time: 0.1868 (0.13688219349688532) | > loader_time: 0.0048 (0.004635029111884506)  --> TIME: 2025-02-27 10:11:17 -- STEP: 29628/36938 -- GLOBAL_STEP: 1654900 | > loss_text_ce: 0.024701977148652077 (0.029859748683255364) | > loss_mel_ce: 1.968238115310669 (nan) | > loss: 0.007908492349088192 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13688361915931138) | > loader_time: 0.0046 (0.0046350340120895825)  --> TIME: 2025-02-27 10:11:27 -- STEP: 29678/36938 -- GLOBAL_STEP: 1654950 | > loss_text_ce: 0.022323576733469963 (0.029858915645345353) | > loss_mel_ce: 2.897977828979492 (nan) | > loss: 0.011588498018682003 (nan) | > current_lr: 5e-06 | > step_time: 0.1954 (0.13688681965785604) | > loader_time: 0.0049 (0.004635087008488801)  --> TIME: 2025-02-27 10:11:36 -- STEP: 29728/36938 -- GLOBAL_STEP: 1655000 | > loss_text_ce: 0.034291092306375504 (0.029856090844654327) | > loss_mel_ce: 2.355381965637207 (nan) | > loss: 0.009482829831540585 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13688596909057735) | > loader_time: 0.0048 (0.00463514929021779)  --> TIME: 2025-02-27 10:11:46 -- STEP: 29778/36938 -- GLOBAL_STEP: 1655050 | > loss_text_ce: 0.026466527953743935 (0.029855675394656514) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.1369006452890936) | > loader_time: 0.0046 (0.0046351949894305865)  --> TIME: 2025-02-27 10:11:56 -- STEP: 29828/36938 -- GLOBAL_STEP: 1655100 | > loss_text_ce: 0.027397796511650085 (0.029855531984975762) | > loss_mel_ce: 2.488903284072876 (nan) | > loss: 0.009985323064029217 (nan) | > current_lr: 5e-06 | > step_time: 0.189 (0.1369051508614474) | > loader_time: 0.0048 (0.00463518752510785)  --> TIME: 2025-02-27 10:12:05 -- STEP: 29878/36938 -- GLOBAL_STEP: 1655150 | > loss_text_ce: 0.030872954055666924 (0.029854178549701883) | > loss_mel_ce: 2.3329756259918213 (nan) | > loss: 0.009380352683365345 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1369111747481243) | > loader_time: 0.0045 (0.004635275850590324)  --> TIME: 2025-02-27 10:12:14 -- STEP: 29928/36938 -- GLOBAL_STEP: 1655200 | > loss_text_ce: 0.0378471277654171 (0.029852880543151748) | > loss_mel_ce: 1.5221805572509766 (nan) | > loss: 0.006190586369484663 (nan) | > current_lr: 5e-06 | > step_time: 0.1409 (0.13691193108347088) | > loader_time: 0.0046 (0.004635262612001501)  --> TIME: 2025-02-27 10:12:24 -- STEP: 29978/36938 -- GLOBAL_STEP: 1655250 | > loss_text_ce: 0.021041307598352432 (0.029854862878434106) | > loss_mel_ce: 2.9703617095947266 (nan) | > loss: 0.01187064778059721 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.1369091981931339) | > loader_time: 0.0045 (0.00463535075620652)  --> TIME: 2025-02-27 10:12:33 -- STEP: 30028/36938 -- GLOBAL_STEP: 1655300 | > loss_text_ce: 0.03183923289179802 (0.02985442621471573) | > loss_mel_ce: 0.033447265625 (nan) | > loss: 0.0002590734220575541 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13691903897095306) | > loader_time: 0.0046 (0.004635393683056368)  --> TIME: 2025-02-27 10:12:43 -- STEP: 30078/36938 -- GLOBAL_STEP: 1655350 | > loss_text_ce: 0.02662023715674877 (0.02985402910794402) | > loss_mel_ce: 2.542841911315918 (nan) | > loss: 0.010196278803050518 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.1369204477899547) | > loader_time: 0.0046 (0.004635404269027392)  --> TIME: 2025-02-27 10:12:52 -- STEP: 30128/36938 -- GLOBAL_STEP: 1655400 | > loss_text_ce: 0.03508364036679268 (0.029855809895334953) | > loss_mel_ce: 2.1922671794891357 (nan) | > loss: 0.008838693611323833 (nan) | > current_lr: 5e-06 | > step_time: 0.1687 (0.13692575183499722) | > loader_time: 0.0048 (0.004635442604236138)  --> TIME: 2025-02-27 10:13:02 -- STEP: 30178/36938 -- GLOBAL_STEP: 1655450 | > loss_text_ce: 0.029786095023155212 (0.02985254227488295) | > loss_mel_ce: 2.729194164276123 (nan) | > loss: 0.010948334820568562 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13692316749680727) | > loader_time: 0.0048 (0.004635433852376323)  --> TIME: 2025-02-27 10:13:11 -- STEP: 30228/36938 -- GLOBAL_STEP: 1655500 | > loss_text_ce: 0.02720843069255352 (0.02985367977863099) | > loss_mel_ce: 2.6073451042175293 (nan) | > loss: 0.010454577393829823 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.1369247093897296) | > loader_time: 0.0046 (0.004635423441578108)  --> TIME: 2025-02-27 10:13:21 -- STEP: 30278/36938 -- GLOBAL_STEP: 1655550 | > loss_text_ce: 0.029613541439175606 (0.029854906772194628) | > loss_mel_ce: 2.316401481628418 (nan) | > loss: 0.009309583343565464 (nan) | > current_lr: 5e-06 | > step_time: 0.1265 (0.1369241959069585) | > loader_time: 0.0046 (0.004635483508806462)  --> TIME: 2025-02-27 10:13:30 -- STEP: 30328/36938 -- GLOBAL_STEP: 1655600 | > loss_text_ce: 0.025810228660702705 (0.029855582809613843) | > loss_mel_ce: 1.9116781949996948 (nan) | > loss: 0.007688446436077356 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13693044257805506) | > loader_time: 0.0046 (0.0046355425210906085)  --> TIME: 2025-02-27 10:13:39 -- STEP: 30378/36938 -- GLOBAL_STEP: 1655650 | > loss_text_ce: 0.0325976237654686 (0.02985750629641537) | > loss_mel_ce: 1.664404273033142 (nan) | > loss: 0.006734135095030069 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.1369181987491966) | > loader_time: 0.0047 (0.0046355073310253925)  --> TIME: 2025-02-27 10:13:48 -- STEP: 30428/36938 -- GLOBAL_STEP: 1655700 | > loss_text_ce: 0.028920229524374008 (0.029860403420952936) | > loss_mel_ce: 2.4957990646362305 (nan) | > loss: 0.010018727742135525 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13691205652906926) | > loader_time: 0.0048 (0.0046354842919376985)  --> TIME: 2025-02-27 10:13:58 -- STEP: 30478/36938 -- GLOBAL_STEP: 1655750 | > loss_text_ce: 0.03374258428812027 (0.029860964074523204) | > loss_mel_ce: 2.8544788360595703 (nan) | > loss: 0.011461197398602962 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13691773572785723) | > loader_time: 0.0046 (0.004635508491170176)  --> TIME: 2025-02-27 10:14:07 -- STEP: 30528/36938 -- GLOBAL_STEP: 1655800 | > loss_text_ce: 0.026006784290075302 (0.02986258209424117) | > loss_mel_ce: 2.1081645488739014 (nan) | > loss: 0.008468934334814548 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13691171289927326) | > loader_time: 0.0046 (0.004635471631958311)  --> TIME: 2025-02-27 10:14:16 -- STEP: 30578/36938 -- GLOBAL_STEP: 1655850 | > loss_text_ce: 0.032830167561769485 (0.029862525073580663) | > loss_mel_ce: 2.8386294841766357 (nan) | > loss: 0.011394682340323925 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.1369078433321056) | > loader_time: 0.0045 (0.004635494330295981)  --> TIME: 2025-02-27 10:14:25 -- STEP: 30628/36938 -- GLOBAL_STEP: 1655900 | > loss_text_ce: 0.0350235179066658 (0.029861837103785923) | > loss_mel_ce: 2.6344995498657227 (nan) | > loss: 0.010593345388770103 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13690376762031706) | > loader_time: 0.0048 (0.004635445260808547)  --> TIME: 2025-02-27 10:14:35 -- STEP: 30678/36938 -- GLOBAL_STEP: 1655950 | > loss_text_ce: 0.024005191400647163 (0.029861338497273994) | > loss_mel_ce: 1.414034128189087 (nan) | > loss: 0.005706505384296179 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.1369164150427543) | > loader_time: 0.0046 (0.004635497802347918)  --> TIME: 2025-02-27 10:14:45 -- STEP: 30728/36938 -- GLOBAL_STEP: 1656000 | > loss_text_ce: 0.018493570387363434 (0.02986274732824352) | > loss_mel_ce: 1.7783684730529785 (nan) | > loss: 0.0071304053999483585 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13692273803596497) | > loader_time: 0.0046 (0.004635545672677648)  --> TIME: 2025-02-27 10:14:54 -- STEP: 30778/36938 -- GLOBAL_STEP: 1656050 | > loss_text_ce: 0.0351296029984951 (0.029863937649837993) | > loss_mel_ce: 1.2525430917739868 (nan) | > loss: 0.005109812133014202 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13691906619470146) | > loader_time: 0.0046 (0.00463558142703756)  --> TIME: 2025-02-27 10:15:04 -- STEP: 30828/36938 -- GLOBAL_STEP: 1656100 | > loss_text_ce: 0.026637472212314606 (0.029864697290666733) | > loss_mel_ce: 2.7834551334381104 (nan) | > loss: 0.011151161976158619 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.1369261056497991) | > loader_time: 0.0046 (0.004635604165385213)  --> TIME: 2025-02-27 10:15:13 -- STEP: 30878/36938 -- GLOBAL_STEP: 1656150 | > loss_text_ce: 0.0313926562666893 (0.029864394760397036) | > loss_mel_ce: 2.3944289684295654 (nan) | > loss: 0.009626276791095734 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13692509248564166) | > loader_time: 0.0046 (0.004635640674400575)  --> TIME: 2025-02-27 10:15:22 -- STEP: 30928/36938 -- GLOBAL_STEP: 1656200 | > loss_text_ce: 0.03268721327185631 (0.029864511391351167) | > loss_mel_ce: 3.2053468227386475 (nan) | > loss: 0.012849342077970505 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13692931816236276) | > loader_time: 0.0047 (0.004635685244435432)  --> TIME: 2025-02-27 10:15:32 -- STEP: 30978/36938 -- GLOBAL_STEP: 1656250 | > loss_text_ce: 0.02802266739308834 (0.029864327928599235) | > loss_mel_ce: 3.2980563640594482 (nan) | > loss: 0.013198727741837502 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13692939879472668) | > loader_time: 0.0046 (0.0046357064659965485)  --> TIME: 2025-02-27 10:15:41 -- STEP: 31028/36938 -- GLOBAL_STEP: 1656300 | > loss_text_ce: 0.022572478279471397 (0.029865289421813202) | > loss_mel_ce: 2.278533697128296 (nan) | > loss: 0.009131374768912792 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13693073409971745) | > loader_time: 0.0046 (0.004635736839940637)  --> TIME: 2025-02-27 10:15:51 -- STEP: 31078/36938 -- GLOBAL_STEP: 1656350 | > loss_text_ce: 0.03029588609933853 (0.029865413802286384) | > loss_mel_ce: 3.1880900859832764 (nan) | > loss: 0.012771373614668846 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13693707379313466) | > loader_time: 0.0049 (0.004635775516573112)  --> TIME: 2025-02-27 10:16:00 -- STEP: 31128/36938 -- GLOBAL_STEP: 1656400 | > loss_text_ce: 0.027846548706293106 (0.029864372772750933) | > loss_mel_ce: 2.7830402851104736 (nan) | > loss: 0.011154313571751118 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13693524458603842) | > loader_time: 0.0046 (0.004635812575392488)  --> TIME: 2025-02-27 10:16:09 -- STEP: 31178/36938 -- GLOBAL_STEP: 1656450 | > loss_text_ce: 0.027759529650211334 (0.029863156813626203) | > loss_mel_ce: 2.6023929119110107 (nan) | > loss: 0.010437113232910633 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13693185785229853) | > loader_time: 0.0046 (0.004635760068235801)  --> TIME: 2025-02-27 10:16:19 -- STEP: 31228/36938 -- GLOBAL_STEP: 1656500 | > loss_text_ce: 0.0277458094060421 (0.029862600242593434) | > loss_mel_ce: 2.7244529724121094 (nan) | > loss: 0.010921424254775047 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1369325791474754) | > loader_time: 0.0045 (0.004635720700693612)  --> TIME: 2025-02-27 10:16:28 -- STEP: 31278/36938 -- GLOBAL_STEP: 1656550 | > loss_text_ce: 0.038001224398612976 (0.029863872922115717) | > loss_mel_ce: 2.443211555480957 (nan) | > loss: 0.009846082888543606 (nan) | > current_lr: 5e-06 | > step_time: 0.1452 (0.13692527119480263) | > loader_time: 0.0046 (0.00463566680082284)  --> TIME: 2025-02-27 10:16:37 -- STEP: 31328/36938 -- GLOBAL_STEP: 1656600 | > loss_text_ce: 0.03936603292822838 (0.029862702687708416) | > loss_mel_ce: 0.1799265593290329 (nan) | > loss: 0.0008702087798155844 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13692232366723617) | > loader_time: 0.0045 (0.0046356143439388485)  --> TIME: 2025-02-27 10:16:46 -- STEP: 31378/36938 -- GLOBAL_STEP: 1656650 | > loss_text_ce: 0.030575212091207504 (0.029863780319442687) | > loss_mel_ce: 2.2002339363098145 (nan) | > loss: 0.008852417580783367 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13692080087860534) | > loader_time: 0.0045 (0.004635626578758027)  --> TIME: 2025-02-27 10:16:56 -- STEP: 31428/36938 -- GLOBAL_STEP: 1656700 | > loss_text_ce: 0.03122938610613346 (0.029863516658679548) | > loss_mel_ce: 2.491887331008911 (nan) | > loss: 0.01001236867159605 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.1369154212340398) | > loader_time: 0.0046 (0.004635570339685009)  --> TIME: 2025-02-27 10:17:05 -- STEP: 31478/36938 -- GLOBAL_STEP: 1656750 | > loss_text_ce: 0.03624775633215904 (0.02986539451853522) | > loss_mel_ce: 2.4654557704925537 (nan) | > loss: 0.009927395731210709 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1369161892591663) | > loader_time: 0.0045 (0.004635522360874117)  --> TIME: 2025-02-27 10:17:14 -- STEP: 31528/36938 -- GLOBAL_STEP: 1656800 | > loss_text_ce: 0.027299867942929268 (0.029865356140434562) | > loss_mel_ce: 2.9337058067321777 (nan) | > loss: 0.011750022880733013 (nan) | > current_lr: 5e-06 | > step_time: 0.1615 (0.1369164872898485) | > loader_time: 0.0048 (0.004635477007055499)  --> TIME: 2025-02-27 10:17:24 -- STEP: 31578/36938 -- GLOBAL_STEP: 1656850 | > loss_text_ce: 0.03010217286646366 (0.02986421114215362) | > loss_mel_ce: 1.5050463676452637 (nan) | > loss: 0.0060918596573174 (nan) | > current_lr: 5e-06 | > step_time: 0.177 (0.13692312378315308) | > loader_time: 0.0048 (0.004635471427589392)  --> TIME: 2025-02-27 10:17:33 -- STEP: 31628/36938 -- GLOBAL_STEP: 1656900 | > loss_text_ce: 0.03415995463728905 (0.029862821249947238) | > loss_mel_ce: 2.649634838104248 (nan) | > loss: 0.010649980045855045 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.1369185939108934) | > loader_time: 0.0046 (0.004635558050562593)  --> TIME: 2025-02-27 10:17:42 -- STEP: 31678/36938 -- GLOBAL_STEP: 1656950 | > loss_text_ce: 0.033364150673151016 (0.02986354223174152) | > loss_mel_ce: 2.8526320457458496 (nan) | > loss: 0.011452366597950459 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13690829279296488) | > loader_time: 0.0045 (0.004635462361127497)  --> TIME: 2025-02-27 10:17:51 -- STEP: 31728/36938 -- GLOBAL_STEP: 1657000 | > loss_text_ce: 0.026466023176908493 (0.029862602762461707) | > loss_mel_ce: 2.9079530239105225 (nan) | > loss: 0.011644520796835423 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.136905703958513) | > loader_time: 0.0045 (0.004635393897572142)  --> TIME: 2025-02-27 10:18:01 -- STEP: 31778/36938 -- GLOBAL_STEP: 1657050 | > loss_text_ce: 0.03290057182312012 (0.02986109382628846) | > loss_mel_ce: 2.391058921813965 (nan) | > loss: 0.009618887677788734 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13690855015561393) | > loader_time: 0.0045 (0.004635442705494749)  --> TIME: 2025-02-27 10:18:10 -- STEP: 31828/36938 -- GLOBAL_STEP: 1657100 | > loss_text_ce: 0.0359719954431057 (0.029862440786190072) | > loss_mel_ce: 2.659472703933716 (nan) | > loss: 0.010696209035813808 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.1369043280764624) | > loader_time: 0.0045 (0.0046354777641867)  --> TIME: 2025-02-27 10:18:19 -- STEP: 31878/36938 -- GLOBAL_STEP: 1657150 | > loss_text_ce: 0.022008854895830154 (0.02986488572718107) | > loss_mel_ce: 2.191793918609619 (nan) | > loss: 0.00878493208438158 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.136897167766683) | > loader_time: 0.0045 (0.004635469379026574)  --> TIME: 2025-02-27 10:18:29 -- STEP: 31928/36938 -- GLOBAL_STEP: 1657200 | > loss_text_ce: 0.02857985906302929 (0.029864105178353955) | > loss_mel_ce: 0.0108642578125 (nan) | > loss: 0.00015652428555767983 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13690292272941734) | > loader_time: 0.0046 (0.004635531698901752)  --> TIME: 2025-02-27 10:18:38 -- STEP: 31978/36938 -- GLOBAL_STEP: 1657250 | > loss_text_ce: 0.02935958281159401 (0.029864728540520295) | > loss_mel_ce: 2.997931480407715 (nan) | > loss: 0.012013060040771961 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1369100856813813) | > loader_time: 0.0047 (0.004635610755802561)  --> TIME: 2025-02-27 10:18:48 -- STEP: 32028/36938 -- GLOBAL_STEP: 1657300 | > loss_text_ce: 0.02474728785455227 (0.029864662931164843) | > loss_mel_ce: 2.4958560466766357 (nan) | > loss: 0.010002395138144493 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13691394138056304) | > loader_time: 0.0045 (0.0046356534621414015)  --> TIME: 2025-02-27 10:18:57 -- STEP: 32078/36938 -- GLOBAL_STEP: 1657350 | > loss_text_ce: 0.021559467539191246 (0.02986499124834413) | > loss_mel_ce: 1.7779185771942139 (nan) | > loss: 0.00714078638702631 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13691572642354619) | > loader_time: 0.0045 (0.0046357106104092825)  --> TIME: 2025-02-27 10:19:06 -- STEP: 32128/36938 -- GLOBAL_STEP: 1657400 | > loss_text_ce: 0.029354849830269814 (0.02986415033279424) | > loss_mel_ce: 2.3416945934295654 (nan) | > loss: 0.009408926591277122 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13691054606639552) | > loader_time: 0.0046 (0.004635673001467955)  --> TIME: 2025-02-27 10:19:16 -- STEP: 32178/36938 -- GLOBAL_STEP: 1657450 | > loss_text_ce: 0.032350342720746994 (0.0298633215351646) | > loss_mel_ce: 3.1689579486846924 (nan) | > loss: 0.012703604996204376 (nan) | > current_lr: 5e-06 | > step_time: 0.1996 (0.13691772381894404) | > loader_time: 0.0048 (0.004635675875630396)  --> TIME: 2025-02-27 10:19:25 -- STEP: 32228/36938 -- GLOBAL_STEP: 1657500 | > loss_text_ce: 0.01785227097570896 (0.02986255518242657) | > loss_mel_ce: 2.815538167953491 (nan) | > loss: 0.011243613436818123 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13691441589184805) | > loader_time: 0.0045 (0.0046357155896689164)  --> TIME: 2025-02-27 10:19:35 -- STEP: 32278/36938 -- GLOBAL_STEP: 1657550 | > loss_text_ce: 0.023241477087140083 (0.029862297215781222) | > loss_mel_ce: 1.8324655294418335 (nan) | > loss: 0.007363917306065559 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1369208854860487) | > loader_time: 0.0046 (0.004635744802762288)  --> TIME: 2025-02-27 10:19:44 -- STEP: 32328/36938 -- GLOBAL_STEP: 1657600 | > loss_text_ce: 0.030613010749220848 (0.029863464734987764) | > loss_mel_ce: 2.3869705200195312 (nan) | > loss: 0.009593585506081581 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13691449984705312) | > loader_time: 0.0046 (0.004635859755589459)  --> TIME: 2025-02-27 10:19:53 -- STEP: 32378/36938 -- GLOBAL_STEP: 1657650 | > loss_text_ce: 0.031751736998558044 (0.029864987272662093) | > loss_mel_ce: 2.404407024383545 (nan) | > loss: 0.009667296893894672 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13691559547152413) | > loader_time: 0.0048 (0.004635990634298267)  --> TIME: 2025-02-27 10:20:03 -- STEP: 32428/36938 -- GLOBAL_STEP: 1657700 | > loss_text_ce: 0.03004881739616394 (0.0298635846947361) | > loss_mel_ce: 1.8666915893554688 (nan) | > loss: 0.007526748348027468 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13691936210358488) | > loader_time: 0.0047 (0.004636073555092912)  --> TIME: 2025-02-27 10:20:12 -- STEP: 32478/36938 -- GLOBAL_STEP: 1657750 | > loss_text_ce: 0.035388167947530746 (0.02986612834897016) | > loss_mel_ce: 2.7003252506256104 (nan) | > loss: 0.010856006294488907 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13692258523671735) | > loader_time: 0.0046 (0.004636085020937185)  --> TIME: 2025-02-27 10:20:22 -- STEP: 32528/36938 -- GLOBAL_STEP: 1657800 | > loss_text_ce: 0.027007192373275757 (0.029868232191993704) | > loss_mel_ce: 2.4664103984832764 (nan) | > loss: 0.009894514456391335 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13692469188308595) | > loader_time: 0.0045 (0.00463609351234606)  --> TIME: 2025-02-27 10:20:31 -- STEP: 32578/36938 -- GLOBAL_STEP: 1657850 | > loss_text_ce: 0.02351045049726963 (0.029869826035982303) | > loss_mel_ce: 2.3143138885498047 (nan) | > loss: 0.009277081117033958 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13692459377899133) | > loader_time: 0.0046 (0.00463618923487955)  --> TIME: 2025-02-27 10:20:41 -- STEP: 32628/36938 -- GLOBAL_STEP: 1657900 | > loss_text_ce: 0.03355313464999199 (0.029870950572353765) | > loss_mel_ce: 0.04022216796875 (nan) | > loss: 0.0002927591558545828 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13693011433262464) | > loader_time: 0.005 (0.004636195808762335)  --> TIME: 2025-02-27 10:20:50 -- STEP: 32678/36938 -- GLOBAL_STEP: 1657950 | > loss_text_ce: 0.022902755066752434 (0.029869028009926055) | > loss_mel_ce: 1.2378562688827515 (nan) | > loss: 0.005003012251108885 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13692413033627207) | > loader_time: 0.0045 (0.00463613739897364)  --> TIME: 2025-02-27 10:20:59 -- STEP: 32728/36938 -- GLOBAL_STEP: 1658000 | > loss_text_ce: 0.03280701860785484 (0.02986956875745047) | > loss_mel_ce: 2.8330564498901367 (nan) | > loss: 0.011372474953532219 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.1369207278349325) | > loader_time: 0.0046 (0.004636089016772877)  --> TIME: 2025-02-27 10:21:08 -- STEP: 32778/36938 -- GLOBAL_STEP: 1658050 | > loss_text_ce: 0.028335021808743477 (0.029868674854393297) | > loss_mel_ce: 2.866011142730713 (nan) | > loss: 0.011485501192510128 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.1369211847433434) | > loader_time: 0.0048 (0.00463607619073366)  --> TIME: 2025-02-27 10:21:18 -- STEP: 32828/36938 -- GLOBAL_STEP: 1658100 | > loss_text_ce: 0.024782678112387657 (0.029867752354788512) | > loss_mel_ce: 0.22320556640625 (nan) | > loss: 0.0009840803686529398 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.13692217336817133) | > loader_time: 0.0047 (0.004636089600176919)  --> TIME: 2025-02-27 10:21:27 -- STEP: 32878/36938 -- GLOBAL_STEP: 1658150 | > loss_text_ce: 0.04002683237195015 (0.029865121637544158) | > loss_mel_ce: 2.6269049644470215 (nan) | > loss: 0.010583063587546349 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.1369188832166358) | > loader_time: 0.005 (0.004636063926142648)  --> TIME: 2025-02-27 10:21:36 -- STEP: 32928/36938 -- GLOBAL_STEP: 1658200 | > loss_text_ce: 0.03350629657506943 (0.02986388267749597) | > loss_mel_ce: 3.4910433292388916 (nan) | > loss: 0.013986309058964252 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.1369165563693433) | > loader_time: 0.0067 (0.00463611013513961)  --> TIME: 2025-02-27 10:21:46 -- STEP: 32978/36938 -- GLOBAL_STEP: 1658250 | > loss_text_ce: 0.026826487854123116 (0.029864324747474445) | > loss_mel_ce: 2.4265286922454834 (nan) | > loss: 0.009735536761581898 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13691853982422367) | > loader_time: 0.0045 (0.00463609942253975)  --> TIME: 2025-02-27 10:21:55 -- STEP: 33028/36938 -- GLOBAL_STEP: 1658300 | > loss_text_ce: 0.022151321172714233 (0.029865591968100652) | > loss_mel_ce: 2.2251853942871094 (nan) | > loss: 0.008918003179132938 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1369200419091828) | > loader_time: 0.0046 (0.0046360636791161446)  --> TIME: 2025-02-27 10:22:05 -- STEP: 33078/36938 -- GLOBAL_STEP: 1658350 | > loss_text_ce: 0.04067558795213699 (0.029864471702318887) | > loss_mel_ce: 2.879926919937134 (nan) | > loss: 0.011589692905545235 (nan) | > current_lr: 5e-06 | > step_time: 0.1514 (0.1369309761596424) | > loader_time: 0.0047 (0.004636089165630661)  --> TIME: 2025-02-27 10:22:14 -- STEP: 33128/36938 -- GLOBAL_STEP: 1658400 | > loss_text_ce: 0.031659338623285294 (0.029865733017211946) | > loss_mel_ce: 2.599799633026123 (nan) | > loss: 0.010442297905683517 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.1369323164649243) | > loader_time: 0.0046 (0.004636141498778022)  --> TIME: 2025-02-27 10:22:23 -- STEP: 33178/36938 -- GLOBAL_STEP: 1658450 | > loss_text_ce: 0.04471476003527641 (0.029867137395113492) | > loss_mel_ce: 2.534891366958618 (nan) | > loss: 0.010236532427370548 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13692960806540855) | > loader_time: 0.0046 (0.004636126908652939)  --> TIME: 2025-02-27 10:22:32 -- STEP: 33228/36938 -- GLOBAL_STEP: 1658500 | > loss_text_ce: 0.03152146562933922 (0.029867616669854255) | > loss_mel_ce: 2.26798415184021 (nan) | > loss: 0.009125023148953915 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13692232812514346) | > loader_time: 0.0046 (0.0046361184183320355)  --> TIME: 2025-02-27 10:22:42 -- STEP: 33278/36938 -- GLOBAL_STEP: 1658550 | > loss_text_ce: 0.02775849774479866 (0.029867775477506457) | > loss_mel_ce: 3.0719594955444336 (nan) | > loss: 0.012300468981266022 (nan) | > current_lr: 5e-06 | > step_time: 0.1648 (0.13692030127041774) | > loader_time: 0.0047 (0.004636117977708852)  --> TIME: 2025-02-27 10:22:51 -- STEP: 33328/36938 -- GLOBAL_STEP: 1658600 | > loss_text_ce: 0.03218710422515869 (0.029869403576800152) | > loss_mel_ce: 2.563689708709717 (nan) | > loss: 0.010301098227500916 (nan) | > current_lr: 5e-06 | > step_time: 0.1884 (0.13692075982725643) | > loader_time: 0.0051 (0.0046361814066588946)  --> TIME: 2025-02-27 10:23:00 -- STEP: 33378/36938 -- GLOBAL_STEP: 1658650 | > loss_text_ce: 0.025194261223077774 (0.029869089622543938) | > loss_mel_ce: 2.9367549419403076 (nan) | > loss: 0.011753766797482967 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13692147352917056) | > loader_time: 0.0046 (0.0046362500599599465)  --> TIME: 2025-02-27 10:23:10 -- STEP: 33428/36938 -- GLOBAL_STEP: 1658700 | > loss_text_ce: 0.02326309122145176 (0.02987008677028305) | > loss_mel_ce: 2.2083935737609863 (nan) | > loss: 0.008855780586600304 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13691674817816654) | > loader_time: 0.0046 (0.004636349347956641)  --> TIME: 2025-02-27 10:23:19 -- STEP: 33478/36938 -- GLOBAL_STEP: 1658750 | > loss_text_ce: 0.03543175756931305 (0.02987095526709676) | > loss_mel_ce: 2.1542279720306396 (nan) | > loss: 0.008689126931130886 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.136917061415174) | > loader_time: 0.0046 (0.004636389201199749)  --> TIME: 2025-02-27 10:23:28 -- STEP: 33528/36938 -- GLOBAL_STEP: 1658800 | > loss_text_ce: 0.027800848707556725 (0.029870542232155517) | > loss_mel_ce: 2.836195230484009 (nan) | > loss: 0.011365064419806004 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.1369129466970678) | > loader_time: 0.0046 (0.004636480234540249)  --> TIME: 2025-02-27 10:23:38 -- STEP: 33578/36938 -- GLOBAL_STEP: 1658850 | > loss_text_ce: 0.031161148101091385 (0.02987081005784509) | > loss_mel_ce: 2.7965688705444336 (nan) | > loss: 0.011221150867640972 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13691183415750957) | > loader_time: 0.0046 (0.004636493104942314)  --> TIME: 2025-02-27 10:23:47 -- STEP: 33628/36938 -- GLOBAL_STEP: 1658900 | > loss_text_ce: 0.03362180292606354 (0.029869882217128718) | > loss_mel_ce: 3.105339527130127 (nan) | > loss: 0.01245619636029005 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13691000845112686) | > loader_time: 0.0045 (0.004636636532712846)  --> TIME: 2025-02-27 10:23:56 -- STEP: 33678/36938 -- GLOBAL_STEP: 1658950 | > loss_text_ce: 0.034495580941438675 (0.02987143520787202) | > loss_mel_ce: 1.7580653429031372 (nan) | > loss: 0.007113337516784668 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.13690795769026867) | > loader_time: 0.0046 (0.004636590112253138)  --> TIME: 2025-02-27 10:24:05 -- STEP: 33728/36938 -- GLOBAL_STEP: 1659000 | > loss_text_ce: 0.023458559066057205 (0.02987201171791024) | > loss_mel_ce: 1.8402413129806519 (nan) | > loss: 0.007395634893327951 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.136907859470349) | > loader_time: 0.0046 (0.004636646695505748)  --> TIME: 2025-02-27 10:24:15 -- STEP: 33778/36938 -- GLOBAL_STEP: 1659050 | > loss_text_ce: 0.023367997258901596 (0.029872177159448985) | > loss_mel_ce: 2.9145214557647705 (nan) | > loss: 0.011658291332423687 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.1369067478846245) | > loader_time: 0.0048 (0.0046366641700596135)  --> TIME: 2025-02-27 10:24:24 -- STEP: 33828/36938 -- GLOBAL_STEP: 1659100 | > loss_text_ce: 0.033679667860269547 (0.029874254665586377) | > loss_mel_ce: 2.5713374614715576 (nan) | > loss: 0.010337370447814465 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13690709790059183) | > loader_time: 0.0046 (0.00463665027179744)  --> TIME: 2025-02-27 10:24:34 -- STEP: 33878/36938 -- GLOBAL_STEP: 1659150 | > loss_text_ce: 0.029865523800253868 (0.02987376938036192) | > loss_mel_ce: 2.878451347351074 (nan) | > loss: 0.011540940962731838 (nan) | > current_lr: 5e-06 | > step_time: 0.1767 (0.1369109316033775) | > loader_time: 0.0048 (0.004636603549137775)  --> TIME: 2025-02-27 10:24:43 -- STEP: 33928/36938 -- GLOBAL_STEP: 1659200 | > loss_text_ce: 0.03464072570204735 (0.029872294298722433) | > loss_mel_ce: 2.660517692565918 (nan) | > loss: 0.010695073753595352 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.13691356660088005) | > loader_time: 0.0047 (0.004636585403236409)  --> TIME: 2025-02-27 10:24:52 -- STEP: 33978/36938 -- GLOBAL_STEP: 1659250 | > loss_text_ce: 0.023382941260933876 (0.0298718002578789) | > loss_mel_ce: 1.6433982849121094 (nan) | > loss: 0.006614211481064558 (nan) | > current_lr: 5e-06 | > step_time: 0.1501 (0.13691534693746754) | > loader_time: 0.0047 (0.004636614632386184)  --> TIME: 2025-02-27 10:25:02 -- STEP: 34028/36938 -- GLOBAL_STEP: 1659300 | > loss_text_ce: 0.030028540641069412 (0.02987033089029652) | > loss_mel_ce: 0.10689087212085724 (nan) | > loss: 0.00054333102889359 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.1369105571793574) | > loader_time: 0.0046 (0.0046365781243489)  --> TIME: 2025-02-27 10:25:11 -- STEP: 34078/36938 -- GLOBAL_STEP: 1659350 | > loss_text_ce: 0.028995389118790627 (0.02986957603275232) | > loss_mel_ce: 1.924027919769287 (nan) | > loss: 0.007750093005597591 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13690345363162473) | > loader_time: 0.0046 (0.0046364945056775846)  --> TIME: 2025-02-27 10:25:20 -- STEP: 34128/36938 -- GLOBAL_STEP: 1659400 | > loss_text_ce: 0.028951536864042282 (0.029871239488977908) | > loss_mel_ce: 1.6072484254837036 (nan) | > loss: 0.006492857355624437 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13690956910758645) | > loader_time: 0.0045 (0.004636482927254878)  --> TIME: 2025-02-27 10:25:30 -- STEP: 34178/36938 -- GLOBAL_STEP: 1659450 | > loss_text_ce: 0.031085295602679253 (0.029871376019681956) | > loss_mel_ce: 2.77648663520813 (nan) | > loss: 0.011141158640384674 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.13691622113336058) | > loader_time: 0.0047 (0.0046365569965899)  --> TIME: 2025-02-27 10:25:39 -- STEP: 34228/36938 -- GLOBAL_STEP: 1659500 | > loss_text_ce: 0.02935071662068367 (0.02987138494586415) | > loss_mel_ce: 0.1716788113117218 (nan) | > loss: 0.0007977362256497145 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13692605902243965) | > loader_time: 0.0047 (0.004636608197391769)  --> TIME: 2025-02-27 10:25:49 -- STEP: 34278/36938 -- GLOBAL_STEP: 1659550 | > loss_text_ce: 0.028129037469625473 (0.029871879669864425) | > loss_mel_ce: 2.909168004989624 (nan) | > loss: 0.011655941605567932 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.1369275195893931) | > loader_time: 0.0047 (0.00463664388425696)  --> TIME: 2025-02-27 10:25:58 -- STEP: 34328/36938 -- GLOBAL_STEP: 1659600 | > loss_text_ce: 0.020458996295928955 (0.029871883982924553) | > loss_mel_ce: 1.8840135335922241 (nan) | > loss: 0.007557431235909462 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.1369333602948168) | > loader_time: 0.0047 (0.004636691968720491)  --> TIME: 2025-02-27 10:26:08 -- STEP: 34378/36938 -- GLOBAL_STEP: 1659650 | > loss_text_ce: 0.03951309993863106 (0.02987004864874055) | > loss_mel_ce: 2.812286615371704 (nan) | > loss: 0.011316666379570961 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13693441439066076) | > loader_time: 0.0045 (0.004636719939915243)  --> TIME: 2025-02-27 10:26:17 -- STEP: 34428/36938 -- GLOBAL_STEP: 1659700 | > loss_text_ce: 0.03277621418237686 (0.029869355878109266) | > loss_mel_ce: 1.8767671585083008 (nan) | > loss: 0.007577553391456604 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13693236218133678) | > loader_time: 0.0045 (0.004636687006395817)  --> TIME: 2025-02-27 10:26:26 -- STEP: 34478/36938 -- GLOBAL_STEP: 1659750 | > loss_text_ce: 0.027696643024683 (0.029869614287760186) | > loss_mel_ce: 2.439176321029663 (nan) | > loss: 0.0097891790792346 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.1369309148957734) | > loader_time: 0.0045 (0.004636629467684308)  --> TIME: 2025-02-27 10:26:36 -- STEP: 34528/36938 -- GLOBAL_STEP: 1659800 | > loss_text_ce: 0.028805682435631752 (0.029869138561608322) | > loss_mel_ce: 3.2002151012420654 (nan) | > loss: 0.012813575565814972 (nan) | > current_lr: 5e-06 | > step_time: 0.1802 (0.13692750597408906) | > loader_time: 0.0048 (0.004636569665028502)  --> TIME: 2025-02-27 10:26:45 -- STEP: 34578/36938 -- GLOBAL_STEP: 1659850 | > loss_text_ce: 0.023399820551276207 (0.02986722361455378) | > loss_mel_ce: 2.4347176551818848 (nan) | > loss: 0.009754435159265995 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13692989186089405) | > loader_time: 0.0046 (0.004636578158856794)  --> TIME: 2025-02-27 10:26:54 -- STEP: 34628/36938 -- GLOBAL_STEP: 1659900 | > loss_text_ce: 0.023595988750457764 (0.029864930331523994) | > loss_mel_ce: 2.3711092472076416 (nan) | > loss: 0.00950279925018549 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.1369344851388742) | > loader_time: 0.0045 (0.004636598739115078)  --> TIME: 2025-02-27 10:27:04 -- STEP: 34678/36938 -- GLOBAL_STEP: 1659950 | > loss_text_ce: 0.02138248085975647 (0.029863775236493622) | > loss_mel_ce: 2.8642003536224365 (nan) | > loss: 0.01145072653889656 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13692756963921615) | > loader_time: 0.0047 (0.004636509572904798)  --> TIME: 2025-02-27 10:27:13 -- STEP: 34728/36938 -- GLOBAL_STEP: 1660000 | > loss_text_ce: 0.03369720280170441 (0.029862416315140146) | > loss_mel_ce: 1.65234375 (nan) | > loss: 0.006690639071166515 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13692613627415778) | > loader_time: 0.0045 (0.004636470704714681) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1660000.pth  --> TIME: 2025-02-27 10:27:29 -- STEP: 34778/36938 -- GLOBAL_STEP: 1660050 | > loss_text_ce: 0.030236994847655296 (0.02986225772582797) | > loss_mel_ce: 2.2807116508483887 (nan) | > loss: 0.009170431643724442 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13692618816948748) | > loader_time: 0.0046 (0.0046366274380836145)  --> TIME: 2025-02-27 10:27:38 -- STEP: 34828/36938 -- GLOBAL_STEP: 1660100 | > loss_text_ce: 0.02885270118713379 (0.029861590232863427) | > loss_mel_ce: 3.1863551139831543 (nan) | > loss: 0.012758761644363403 (nan) | > current_lr: 5e-06 | > step_time: 0.149 (0.1369218737711874) | > loader_time: 0.0046 (0.0046366176471598535)  --> TIME: 2025-02-27 10:27:47 -- STEP: 34878/36938 -- GLOBAL_STEP: 1660150 | > loss_text_ce: 0.033826325088739395 (0.029863372209205086) | > loss_mel_ce: 1.895308256149292 (nan) | > loss: 0.007655296474695206 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13692265760256717) | > loader_time: 0.0046 (0.004636612990640903)  --> TIME: 2025-02-27 10:27:56 -- STEP: 34928/36938 -- GLOBAL_STEP: 1660200 | > loss_text_ce: 0.033798862248659134 (0.029862120009384815) | > loss_mel_ce: 2.5910308361053467 (nan) | > loss: 0.010415991768240929 (nan) | > current_lr: 5e-06 | > step_time: 0.1732 (0.13691474901417566) | > loader_time: 0.0049 (0.004636574927350555)  --> TIME: 2025-02-27 10:28:05 -- STEP: 34978/36938 -- GLOBAL_STEP: 1660250 | > loss_text_ce: 0.033481862396001816 (0.02986217815944396) | > loss_mel_ce: 2.538071393966675 (nan) | > loss: 0.010204576887190342 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13691496977205453) | > loader_time: 0.0047 (0.004636609463634733)  --> TIME: 2025-02-27 10:28:15 -- STEP: 35028/36938 -- GLOBAL_STEP: 1660300 | > loss_text_ce: 0.023287657648324966 (0.029863707419287344) | > loss_mel_ce: 2.6275227069854736 (nan) | > loss: 0.010519090108573437 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1369203298593121) | > loader_time: 0.0045 (0.004636708733370367)  --> TIME: 2025-02-27 10:28:24 -- STEP: 35078/36938 -- GLOBAL_STEP: 1660350 | > loss_text_ce: 0.025930337607860565 (0.02986621383337943) | > loss_mel_ce: 1.8274751901626587 (nan) | > loss: 0.007354783825576305 (nan) | > current_lr: 5e-06 | > step_time: 0.1086 (0.13691971008274256) | > loader_time: 0.0044 (0.004636794099297169)  --> TIME: 2025-02-27 10:28:34 -- STEP: 35128/36938 -- GLOBAL_STEP: 1660400 | > loss_text_ce: 0.03554945066571236 (0.029865952216684647) | > loss_mel_ce: 2.754492998123169 (nan) | > loss: 0.01107159722596407 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13691728863295882) | > loader_time: 0.0048 (0.004636798204145306)  --> TIME: 2025-02-27 10:28:43 -- STEP: 35178/36938 -- GLOBAL_STEP: 1660450 | > loss_text_ce: 0.024660617113113403 (0.029863740465132457) | > loss_mel_ce: 1.638426423072815 (nan) | > loss: 0.006599551998078823 (nan) | > current_lr: 5e-06 | > step_time: 0.1077 (0.1369162242648018) | > loader_time: 0.0045 (0.004636841281452997)  --> TIME: 2025-02-27 10:28:52 -- STEP: 35228/36938 -- GLOBAL_STEP: 1660500 | > loss_text_ce: 0.043089039623737335 (0.02986483456273954) | > loss_mel_ce: 3.3463680744171143 (nan) | > loss: 0.013450227677822113 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13691895956536687) | > loader_time: 0.0048 (0.004637069953642556)  --> TIME: 2025-02-27 10:29:02 -- STEP: 35278/36938 -- GLOBAL_STEP: 1660550 | > loss_text_ce: 0.03334204480051994 (0.029866051907501218) | > loss_mel_ce: 2.9606688022613525 (nan) | > loss: 0.011880996637046337 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.1369190506129804) | > loader_time: 0.0046 (0.004637006290294582)  --> TIME: 2025-02-27 10:29:11 -- STEP: 35328/36938 -- GLOBAL_STEP: 1660600 | > loss_text_ce: 0.0310813020914793 (0.029864591529049904) | > loss_mel_ce: 3.3064305782318115 (nan) | > loss: 0.013244095258414745 (nan) | > current_lr: 5e-06 | > step_time: 0.1725 (0.13692105192340162) | > loader_time: 0.0051 (0.004637121324143991)  --> TIME: 2025-02-27 10:29:20 -- STEP: 35378/36938 -- GLOBAL_STEP: 1660650 | > loss_text_ce: 0.031158750876784325 (0.029865884158949942) | > loss_mel_ce: 0.8322529196739197 (nan) | > loss: 0.0034262370318174362 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13691915892390175) | > loader_time: 0.0045 (0.0046370813417841855)  --> TIME: 2025-02-27 10:29:30 -- STEP: 35428/36938 -- GLOBAL_STEP: 1660700 | > loss_text_ce: 0.023746903985738754 (0.029867271430679687) | > loss_mel_ce: 2.975175619125366 (nan) | > loss: 0.01190048735588789 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1369159179992544) | > loader_time: 0.0046 (0.0046370336524086606)  --> TIME: 2025-02-27 10:29:39 -- STEP: 35478/36938 -- GLOBAL_STEP: 1660750 | > loss_text_ce: 0.03506714105606079 (0.029867220481911615) | > loss_mel_ce: 2.969040870666504 (nan) | > loss: 0.011921063996851444 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13692135407372416) | > loader_time: 0.0046 (0.004637093049132238)  --> TIME: 2025-02-27 10:29:48 -- STEP: 35528/36938 -- GLOBAL_STEP: 1660800 | > loss_text_ce: 0.03142814710736275 (0.02986635476347585) | > loss_mel_ce: 1.5863457918167114 (nan) | > loss: 0.006419737823307514 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.1369208906502695) | > loader_time: 0.0045 (0.004637178336411722)  --> TIME: 2025-02-27 10:29:58 -- STEP: 35578/36938 -- GLOBAL_STEP: 1660850 | > loss_text_ce: 0.028796980157494545 (0.029866222786444394) | > loss_mel_ce: 2.3523709774017334 (nan) | > loss: 0.009449079632759094 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.1369218700376665) | > loader_time: 0.0047 (0.004637224811333309)  --> TIME: 2025-02-27 10:30:07 -- STEP: 35628/36938 -- GLOBAL_STEP: 1660900 | > loss_text_ce: 0.026132790371775627 (0.029865798976261772) | > loss_mel_ce: 2.9254517555236816 (nan) | > loss: 0.011712637729942799 (nan) | > current_lr: 5e-06 | > step_time: 0.171 (0.1369195374573098) | > loader_time: 0.0048 (0.0046372839707751285)  --> TIME: 2025-02-27 10:30:17 -- STEP: 35678/36938 -- GLOBAL_STEP: 1660950 | > loss_text_ce: 0.03997737914323807 (0.029868119036183843) | > loss_mel_ce: 1.106427550315857 (nan) | > loss: 0.0045492262579500675 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13692328459318823) | > loader_time: 0.0046 (0.004637382625096784)  --> TIME: 2025-02-27 10:30:26 -- STEP: 35728/36938 -- GLOBAL_STEP: 1661000 | > loss_text_ce: 0.03857769817113876 (0.029868600209625933) | > loss_mel_ce: 1.8854901790618896 (nan) | > loss: 0.007635190151631832 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.13693066184236743) | > loader_time: 0.0047 (0.004637441351389183)  --> TIME: 2025-02-27 10:30:35 -- STEP: 35778/36938 -- GLOBAL_STEP: 1661050 | > loss_text_ce: 0.02443753182888031 (0.029869555833074903) | > loss_mel_ce: 2.3560590744018555 (nan) | > loss: 0.00944641511887312 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.1369275936066684) | > loader_time: 0.0046 (0.004637434494702652)  --> TIME: 2025-02-27 10:30:45 -- STEP: 35828/36938 -- GLOBAL_STEP: 1661100 | > loss_text_ce: 0.022816181182861328 (0.029870092612384477) | > loss_mel_ce: 2.0592854022979736 (nan) | > loss: 0.008262308314442635 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1369301792206319) | > loader_time: 0.0046 (0.004637560375143792)  --> TIME: 2025-02-27 10:30:54 -- STEP: 35878/36938 -- GLOBAL_STEP: 1661150 | > loss_text_ce: 0.029828287661075592 (0.029870170333347275) | > loss_mel_ce: 2.988294839859009 (nan) | > loss: 0.011976679787039757 (nan) | > current_lr: 5e-06 | > step_time: 0.1692 (0.13693404690106548) | > loader_time: 0.0047 (0.00463767104593035)  --> TIME: 2025-02-27 10:31:04 -- STEP: 35928/36938 -- GLOBAL_STEP: 1661200 | > loss_text_ce: 0.03096577525138855 (0.029871129370730505) | > loss_mel_ce: 2.706998825073242 (nan) | > loss: 0.010864939540624619 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13693929215604106) | > loader_time: 0.0047 (0.004637885242335608)  --> TIME: 2025-02-27 10:31:13 -- STEP: 35978/36938 -- GLOBAL_STEP: 1661250 | > loss_text_ce: 0.036395564675331116 (0.02987120904877627) | > loss_mel_ce: 3.431361675262451 (nan) | > loss: 0.013760942034423351 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.13693808285309644) | > loader_time: 0.0047 (0.00463796074112485)  --> TIME: 2025-02-27 10:31:22 -- STEP: 36028/36938 -- GLOBAL_STEP: 1661300 | > loss_text_ce: 0.039945583790540695 (0.029870102404456696) | > loss_mel_ce: 3.061450242996216 (nan) | > loss: 0.012307127006351948 (nan) | > current_lr: 5e-06 | > step_time: 0.1687 (0.13693958829269082) | > loader_time: 0.005 (0.004638041589134907)  --> TIME: 2025-02-27 10:31:32 -- STEP: 36078/36938 -- GLOBAL_STEP: 1661350 | > loss_text_ce: 0.03603876009583473 (0.029871227858665323) | > loss_mel_ce: 1.320279598236084 (nan) | > loss: 0.005382216069847345 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13693950163550897) | > loader_time: 0.0045 (0.004638062149122725)  --> TIME: 2025-02-27 10:31:41 -- STEP: 36128/36938 -- GLOBAL_STEP: 1661400 | > loss_text_ce: 0.029985802248120308 (0.029870586919796143) | > loss_mel_ce: 2.309360980987549 (nan) | > loss: 0.009283123537898064 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13693796996393537) | > loader_time: 0.0046 (0.004638129243077814)  --> TIME: 2025-02-27 10:31:50 -- STEP: 36178/36938 -- GLOBAL_STEP: 1661450 | > loss_text_ce: 0.045579880475997925 (0.029871997738034003) | > loss_mel_ce: 2.4960434436798096 (nan) | > loss: 0.01008580718189478 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.1369304191058563) | > loader_time: 0.0046 (0.0046382609657390226)  --> TIME: 2025-02-27 10:31:59 -- STEP: 36228/36938 -- GLOBAL_STEP: 1661500 | > loss_text_ce: 0.027874428778886795 (0.029873274579564294) | > loss_mel_ce: 2.6416797637939453 (nan) | > loss: 0.010593470185995102 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13692614366061373) | > loader_time: 0.0046 (0.004638267547941859)  --> TIME: 2025-02-27 10:32:09 -- STEP: 36278/36938 -- GLOBAL_STEP: 1661550 | > loss_text_ce: 0.021184619516134262 (0.029871978948058715) | > loss_mel_ce: 1.9893964529037476 (nan) | > loss: 0.007978496141731739 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.1369235995349608) | > loader_time: 0.0046 (0.004638303114183766)  --> TIME: 2025-02-27 10:32:18 -- STEP: 36328/36938 -- GLOBAL_STEP: 1661600 | > loss_text_ce: 0.03882922977209091 (0.029872105082634298) | > loss_mel_ce: 3.278703212738037 (nan) | > loss: 0.013164812698960304 (nan) | > current_lr: 5e-06 | > step_time: 0.1902 (0.13693248132230265) | > loader_time: 0.005 (0.004638442447652289)  --> TIME: 2025-02-27 10:32:27 -- STEP: 36378/36938 -- GLOBAL_STEP: 1661650 | > loss_text_ce: 0.03330415114760399 (0.029872099411010707) | > loss_mel_ce: 2.37182354927063 (nan) | > loss: 0.009544158354401588 (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.13692646317867674) | > loader_time: 0.0047 (0.0046384756702333284)  --> TIME: 2025-02-27 10:32:37 -- STEP: 36428/36938 -- GLOBAL_STEP: 1661700 | > loss_text_ce: 0.03370802849531174 (0.029871116049587816) | > loss_mel_ce: 3.019531011581421 (nan) | > loss: 0.012116028927266598 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.136922972313827) | > loader_time: 0.0047 (0.004638468713939463)  --> TIME: 2025-02-27 10:32:46 -- STEP: 36478/36938 -- GLOBAL_STEP: 1661750 | > loss_text_ce: 0.027050547301769257 (0.029869385026360076) | > loss_mel_ce: 1.9361512660980225 (nan) | > loss: 0.00779048353433609 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13692282447363222) | > loader_time: 0.0047 (0.004638590384708766)  --> TIME: 2025-02-27 10:32:55 -- STEP: 36528/36938 -- GLOBAL_STEP: 1661800 | > loss_text_ce: 0.036169543862342834 (0.029869166326410162) | > loss_mel_ce: 2.972574472427368 (nan) | > loss: 0.011939461342990398 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.1369237591444381) | > loader_time: 0.005 (0.004638693877546837)  --> TIME: 2025-02-27 10:33:05 -- STEP: 36578/36938 -- GLOBAL_STEP: 1661850 | > loss_text_ce: 0.03142455592751503 (0.029869220129661467) | > loss_mel_ce: 2.7456867694854736 (nan) | > loss: 0.011020283214747906 (nan) | > current_lr: 5e-06 | > step_time: 0.17 (0.13693038199297192) | > loader_time: 0.0049 (0.0046388591331158)  --> TIME: 2025-02-27 10:33:14 -- STEP: 36628/36938 -- GLOBAL_STEP: 1661900 | > loss_text_ce: 0.025413328781723976 (0.029869793208558117) | > loss_mel_ce: 1.1521656513214111 (nan) | > loss: 0.00467293243855238 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.1369307988841061) | > loader_time: 0.0045 (0.004639044825499863)  --> TIME: 2025-02-27 10:33:23 -- STEP: 36678/36938 -- GLOBAL_STEP: 1661950 | > loss_text_ce: 0.035830605775117874 (0.029870710423108755) | > loss_mel_ce: 2.640702247619629 (nan) | > loss: 0.010621162131428719 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1369241193425238) | > loader_time: 0.0046 (0.004639029964391087)  --> TIME: 2025-02-27 10:33:33 -- STEP: 36728/36938 -- GLOBAL_STEP: 1662000 | > loss_text_ce: 0.028732597827911377 (0.029871260528416675) | > loss_mel_ce: 1.220590591430664 (nan) | > loss: 0.004957631696015596 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1369244466937208) | > loader_time: 0.0046 (0.004639139461818557)  --> TIME: 2025-02-27 10:33:42 -- STEP: 36778/36938 -- GLOBAL_STEP: 1662050 | > loss_text_ce: 0.024624837562441826 (0.029870558234258413) | > loss_mel_ce: 2.7608888149261475 (nan) | > loss: 0.011053626425564289 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.1369133545852061) | > loader_time: 0.0045 (0.0046391095245919955)  --> TIME: 2025-02-27 10:33:51 -- STEP: 36828/36938 -- GLOBAL_STEP: 1662100 | > loss_text_ce: 0.02635137178003788 (0.029869508917146657) | > loss_mel_ce: 2.2571613788604736 (nan) | > loss: 0.009061559103429317 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13690987296998916) | > loader_time: 0.0046 (0.00463910402324078)  --> TIME: 2025-02-27 10:34:00 -- STEP: 36878/36938 -- GLOBAL_STEP: 1662150 | > loss_text_ce: 0.03133891150355339 (0.029869158798869117) | > loss_mel_ce: 2.806349992752075 (nan) | > loss: 0.011260670609772205 (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.13691383155761985) | > loader_time: 0.0048 (0.004639162612042842)  --> TIME: 2025-02-27 10:34:09 -- STEP: 36928/36938 -- GLOBAL_STEP: 1662200 | > loss_text_ce: 0.0320342518389225 (0.029870057789849625) | > loss_mel_ce: 2.2499499320983887 (nan) | > loss: 0.009055493399500847 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13690783279065094) | > loader_time: 0.0046 (0.0046392103376277135)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.020384506423874658 (-0.000600024025038913) | > avg_loss_text_ce: 0.029182032299329443 (-9.65233597129897e-05) | > avg_loss_mel_ce: 2.423206882016494 (-0.0006583468748799781) | > avg_loss: 2.452388918045725 (-0.0007548674498467633) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_1662210.pth  > EPOCH: 46/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-27 10:34:36)   --> TIME: 2025-02-27 10:34:44 -- STEP: 40/36938 -- GLOBAL_STEP: 1662250 | > loss_text_ce: 0.027303144335746765 (0.030049671581946314) | > loss_mel_ce: 2.3140757083892822 (nan) | > loss: 0.009291186928749084 (nan) | > current_lr: 5e-06 | > step_time: 0.1761 (0.13613248467445374) | > loader_time: 0.0058 (0.004842144250869751)  --> TIME: 2025-02-27 10:34:54 -- STEP: 90/36938 -- GLOBAL_STEP: 1662300 | > loss_text_ce: 0.035594213753938675 (0.029525576376666626) | > loss_mel_ce: 3.1804466247558594 (nan) | > loss: 0.012762067839503288 (nan) | > current_lr: 5e-06 | > step_time: 0.1528 (0.13701816929711236) | > loader_time: 0.0046 (0.004736211564805773)  --> TIME: 2025-02-27 10:35:03 -- STEP: 140/36938 -- GLOBAL_STEP: 1662350 | > loss_text_ce: 0.033758487552404404 (0.02936491610349289) | > loss_mel_ce: 3.418656587600708 (nan) | > loss: 0.013700060546398163 (nan) | > current_lr: 5e-06 | > step_time: 0.1507 (0.13434814555304397) | > loader_time: 0.0046 (0.004713220255715507)  --> TIME: 2025-02-27 10:35:12 -- STEP: 190/36938 -- GLOBAL_STEP: 1662400 | > loss_text_ce: 0.033824194222688675 (0.029563776114465375) | > loss_mel_ce: 2.8565075397491455 (nan) | > loss: 0.01146957091987133 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.1348783417751915) | > loader_time: 0.0046 (0.004695002656233939)  --> TIME: 2025-02-27 10:35:21 -- STEP: 240/36938 -- GLOBAL_STEP: 1662450 | > loss_text_ce: 0.023840345442295074 (0.029912181885447354) | > loss_mel_ce: 1.68445885181427 (nan) | > loss: 0.00677896523848176 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13457672397295645) | > loader_time: 0.0045 (0.00468873182932536)  --> TIME: 2025-02-27 10:35:30 -- STEP: 290/36938 -- GLOBAL_STEP: 1662500 | > loss_text_ce: 0.03260793909430504 (0.02985848345761669) | > loss_mel_ce: 3.1519696712493896 (nan) | > loss: 0.012637213803827763 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.13516602433961017) | > loader_time: 0.0048 (0.004684774629000962)  --> TIME: 2025-02-27 10:35:40 -- STEP: 340/36938 -- GLOBAL_STEP: 1662550 | > loss_text_ce: 0.031218286603689194 (0.029922386971028413) | > loss_mel_ce: 2.9757797718048096 (nan) | > loss: 0.01193253230303526 (nan) | > current_lr: 5e-06 | > step_time: 0.1504 (0.13590457229053274) | > loader_time: 0.0057 (0.004691793638117179)  --> TIME: 2025-02-27 10:35:49 -- STEP: 390/36938 -- GLOBAL_STEP: 1662600 | > loss_text_ce: 0.027755629271268845 (0.0297719613625071) | > loss_mel_ce: 2.8437659740448 (nan) | > loss: 0.011394928209483624 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.13600474198659276) | > loader_time: 0.0046 (0.004680079680222739)  --> TIME: 2025-02-27 10:35:59 -- STEP: 440/36938 -- GLOBAL_STEP: 1662650 | > loss_text_ce: 0.03412233665585518 (0.029818620613183487) | > loss_mel_ce: 2.16571044921875 (nan) | > loss: 0.008729495108127594 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13642045151103643) | > loader_time: 0.0046 (0.004674621603705672)  --> TIME: 2025-02-27 10:36:08 -- STEP: 490/36938 -- GLOBAL_STEP: 1662700 | > loss_text_ce: 0.03182021155953407 (0.02978864498436451) | > loss_mel_ce: 2.041816234588623 (nan) | > loss: 0.008228716440498829 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13703011152695654) | > loader_time: 0.005 (0.004674187971621148)  --> TIME: 2025-02-27 10:36:18 -- STEP: 540/36938 -- GLOBAL_STEP: 1662750 | > loss_text_ce: 0.031141454353928566 (0.02984167715465581) | > loss_mel_ce: 2.6575753688812256 (nan) | > loss: 0.010669511742889881 (nan) | > current_lr: 5e-06 | > step_time: 0.1568 (0.13706721773854025) | > loader_time: 0.0047 (0.004667777926833545)  --> TIME: 2025-02-27 10:36:27 -- STEP: 590/36938 -- GLOBAL_STEP: 1662800 | > loss_text_ce: 0.02626820281147957 (0.02991607674729016) | > loss_mel_ce: 2.42802357673645 (nan) | > loss: 0.009739253669977188 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.13681256326578442) | > loader_time: 0.0045 (0.004669755192126262)  --> TIME: 2025-02-27 10:36:36 -- STEP: 640/36938 -- GLOBAL_STEP: 1662850 | > loss_text_ce: 0.02838771604001522 (0.0298416288846056) | > loss_mel_ce: 2.9277968406677246 (nan) | > loss: 0.011730891652405262 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13656640127301226) | > loader_time: 0.0045 (0.004674548283219339)  --> TIME: 2025-02-27 10:36:46 -- STEP: 690/36938 -- GLOBAL_STEP: 1662900 | > loss_text_ce: 0.042273152619600296 (0.029922760136263524) | > loss_mel_ce: 2.090378999710083 (nan) | > loss: 0.008462905883789062 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13698381271915158) | > loader_time: 0.0045 (0.004676606344140097)  --> TIME: 2025-02-27 10:36:55 -- STEP: 740/36938 -- GLOBAL_STEP: 1662950 | > loss_text_ce: 0.0332685261964798 (0.029980412240114968) | > loss_mel_ce: 2.0518391132354736 (nan) | > loss: 0.00827423669397831 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.1369249672503084) | > loader_time: 0.0046 (0.004672695817174141)  --> TIME: 2025-02-27 10:37:05 -- STEP: 790/36938 -- GLOBAL_STEP: 1663000 | > loss_text_ce: 0.019015295431017876 (0.029977042423679105) | > loss_mel_ce: 2.439399480819702 (nan) | > loss: 0.009755615144968033 (nan) | > current_lr: 5e-06 | > step_time: 0.181 (0.13738842221755002) | > loader_time: 0.0048 (0.004674732534191279)  --> TIME: 2025-02-27 10:37:14 -- STEP: 840/36938 -- GLOBAL_STEP: 1663050 | > loss_text_ce: 0.020338138565421104 (0.029909604433036988) | > loss_mel_ce: 1.9379552602767944 (nan) | > loss: 0.007771006319671869 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1372819616681052) | > loader_time: 0.0046 (0.004672723724728538)  --> TIME: 2025-02-27 10:37:23 -- STEP: 890/36938 -- GLOBAL_STEP: 1663100 | > loss_text_ce: 0.031399067491292953 (0.029961557009288767) | > loss_mel_ce: 2.3209359645843506 (nan) | > loss: 0.009334662929177284 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.13722385717241944) | > loader_time: 0.0054 (0.004676919304922725)  --> TIME: 2025-02-27 10:37:32 -- STEP: 940/36938 -- GLOBAL_STEP: 1663150 | > loss_text_ce: 0.033045969903469086 (0.029841364501044153) | > loss_mel_ce: 2.6316604614257812 (nan) | > loss: 0.010574232786893845 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13703064411244492) | > loader_time: 0.0046 (0.004681552217361775)  --> TIME: 2025-02-27 10:37:42 -- STEP: 990/36938 -- GLOBAL_STEP: 1663200 | > loss_text_ce: 0.03160607069730759 (0.029893307554338014) | > loss_mel_ce: 5.520833492279053 (nan) | > loss: 0.022033492103219032 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13689944984936953) | > loader_time: 0.0045 (0.0046847047227801695)  --> TIME: 2025-02-27 10:37:51 -- STEP: 1040/36938 -- GLOBAL_STEP: 1663250 | > loss_text_ce: 0.031570639461278915 (0.029951953118702825) | > loss_mel_ce: 2.280501365661621 (nan) | > loss: 0.009174889884889126 (nan) | > current_lr: 5e-06 | > step_time: 0.1637 (0.13681461237944087) | > loader_time: 0.0046 (0.004683484022433942)  --> TIME: 2025-02-27 10:38:00 -- STEP: 1090/36938 -- GLOBAL_STEP: 1663300 | > loss_text_ce: 0.02843571826815605 (0.029945364098038968) | > loss_mel_ce: 1.9630545377731323 (nan) | > loss: 0.007902739569544792 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13679601708683398) | > loader_time: 0.0046 (0.00468706144105404)  --> TIME: 2025-02-27 10:38:10 -- STEP: 1140/36938 -- GLOBAL_STEP: 1663350 | > loss_text_ce: 0.02097383886575699 (0.029907222431185737) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13684278810233408) | > loader_time: 0.0046 (0.004689796765645347)  --> TIME: 2025-02-27 10:38:19 -- STEP: 1190/36938 -- GLOBAL_STEP: 1663400 | > loss_text_ce: 0.02112295664846897 (0.029931523564013362) | > loss_mel_ce: 2.0589706897735596 (nan) | > loss: 0.008254339918494225 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13704594203404033) | > loader_time: 0.0045 (0.0046873811914139455)  --> TIME: 2025-02-27 10:38:29 -- STEP: 1240/36938 -- GLOBAL_STEP: 1663450 | > loss_text_ce: 0.034971270710229874 (0.029966228771504136) | > loss_mel_ce: 3.0840466022491455 (nan) | > loss: 0.012377055361866951 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13708783215092082) | > loader_time: 0.0046 (0.0046915090853168135)  --> TIME: 2025-02-27 10:38:38 -- STEP: 1290/36938 -- GLOBAL_STEP: 1663500 | > loss_text_ce: 0.0330345444381237 (0.029988621490783702) | > loss_mel_ce: 2.761338472366333 (nan) | > loss: 0.011088782921433449 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.13711280490076821) | > loader_time: 0.0048 (0.004692863863567977)  --> TIME: 2025-02-27 10:38:47 -- STEP: 1340/36938 -- GLOBAL_STEP: 1663550 | > loss_text_ce: 0.03375732898712158 (0.029947108431343933) | > loss_mel_ce: 2.616046190261841 (nan) | > loss: 0.010515094734728336 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13690021269357028) | > loader_time: 0.0046 (0.004693139133168689)  --> TIME: 2025-02-27 10:38:56 -- STEP: 1390/36938 -- GLOBAL_STEP: 1663600 | > loss_text_ce: 0.030206024646759033 (0.029928450812351147) | > loss_mel_ce: 1.8455067873001099 (nan) | > loss: 0.007443305570632219 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13682311901943295) | > loader_time: 0.0045 (0.004693866290634482)  --> TIME: 2025-02-27 10:39:05 -- STEP: 1440/36938 -- GLOBAL_STEP: 1663650 | > loss_text_ce: 0.031575798988342285 (0.029941931848750554) | > loss_mel_ce: 2.638380765914917 (nan) | > loss: 0.010595066472887993 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13675203571716965) | > loader_time: 0.0046 (0.0046949798862139375)  --> TIME: 2025-02-27 10:39:15 -- STEP: 1490/36938 -- GLOBAL_STEP: 1663700 | > loss_text_ce: 0.03429142013192177 (0.029914379784299824) | > loss_mel_ce: 0.0613199882209301 (nan) | > loss: 0.000379410368623212 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13687828803222474) | > loader_time: 0.0052 (0.004695468620965942)  --> TIME: 2025-02-27 10:39:24 -- STEP: 1540/36938 -- GLOBAL_STEP: 1663750 | > loss_text_ce: 0.03694312646985054 (0.0299311620116766) | > loss_mel_ce: 2.061183452606201 (nan) | > loss: 0.008325899951159954 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13678621611037817) | > loader_time: 0.0045 (0.004698326370932838)  --> TIME: 2025-02-27 10:39:34 -- STEP: 1590/36938 -- GLOBAL_STEP: 1663800 | > loss_text_ce: 0.018435707315802574 (0.029906513044645765) | > loss_mel_ce: 2.353069543838501 (nan) | > loss: 0.0094107361510396 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13688806437846274) | > loader_time: 0.0046 (0.004699258834311046)  --> TIME: 2025-02-27 10:39:43 -- STEP: 1640/36938 -- GLOBAL_STEP: 1663850 | > loss_text_ce: 0.027406079694628716 (0.029908618227033536) | > loss_mel_ce: 3.1238765716552734 (nan) | > loss: 0.012505089864134789 (nan) | > current_lr: 5e-06 | > step_time: 0.1883 (0.13693801208240242) | > loader_time: 0.0049 (0.004701318682693851)  --> TIME: 2025-02-27 10:39:53 -- STEP: 1690/36938 -- GLOBAL_STEP: 1663900 | > loss_text_ce: 0.027034617960453033 (0.029910085149131408) | > loss_mel_ce: 2.734229803085327 (nan) | > loss: 0.01095739845186472 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13707501098249092) | > loader_time: 0.0045 (0.004703386577628771)  --> TIME: 2025-02-27 10:40:02 -- STEP: 1740/36938 -- GLOBAL_STEP: 1663950 | > loss_text_ce: 0.02531621605157852 (0.029893338257425475) | > loss_mel_ce: 1.7918606996536255 (nan) | > loss: 0.007211020216345787 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.13717289012053915) | > loader_time: 0.0047 (0.004701886642938371)  --> TIME: 2025-02-27 10:40:12 -- STEP: 1790/36938 -- GLOBAL_STEP: 1664000 | > loss_text_ce: 0.028753429651260376 (0.029884733925651573) | > loss_mel_ce: 0.1255340576171875 (nan) | > loss: 0.0006122519844211638 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13727123177917303) | > loader_time: 0.0046 (0.004701312826998405)  --> TIME: 2025-02-27 10:40:21 -- STEP: 1840/36938 -- GLOBAL_STEP: 1664050 | > loss_text_ce: 0.030165061354637146 (0.029900127527830393) | > loss_mel_ce: 1.5957738161087036 (nan) | > loss: 0.006452139001339674 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13710533574871392) | > loader_time: 0.0044 (0.004698504572329312)  --> TIME: 2025-02-27 10:40:30 -- STEP: 1890/36938 -- GLOBAL_STEP: 1664100 | > loss_text_ce: 0.025820784270763397 (0.0298977112860749) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13698448630237078) | > loader_time: 0.0048 (0.004695885017435385)  --> TIME: 2025-02-27 10:40:39 -- STEP: 1940/36938 -- GLOBAL_STEP: 1664150 | > loss_text_ce: 0.03001648373901844 (0.02987404986831945) | > loss_mel_ce: 3.057751178741455 (nan) | > loss: 0.012253046967089176 (nan) | > current_lr: 5e-06 | > step_time: 0.1814 (0.13696412548576437) | > loader_time: 0.0048 (0.004693205700707188)  --> TIME: 2025-02-27 10:40:49 -- STEP: 1990/36938 -- GLOBAL_STEP: 1664200 | > loss_text_ce: 0.03300094231963158 (0.02987543812732975) | > loss_mel_ce: 0.1490427702665329 (nan) | > loss: 0.0007223957218229771 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13699941503342675) | > loader_time: 0.0047 (0.004693045568226568)  --> TIME: 2025-02-27 10:40:58 -- STEP: 2040/36938 -- GLOBAL_STEP: 1664250 | > loss_text_ce: 0.03361073508858681 (0.02991327604726322) | > loss_mel_ce: 3.003796339035034 (nan) | > loss: 0.012053203769028187 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13701176549874103) | > loader_time: 0.0045 (0.004692214259914322)  --> TIME: 2025-02-27 10:41:08 -- STEP: 2090/36938 -- GLOBAL_STEP: 1664300 | > loss_text_ce: 0.03360262140631676 (0.029931188684455944) | > loss_mel_ce: 2.94639253616333 (nan) | > loss: 0.011825378984212875 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13706560796527775) | > loader_time: 0.0045 (0.004693752156490342)  --> TIME: 2025-02-27 10:41:17 -- STEP: 2140/36938 -- GLOBAL_STEP: 1664350 | > loss_text_ce: 0.029994063079357147 (0.02992948429119364) | > loss_mel_ce: 3.152583122253418 (nan) | > loss: 0.01262927521020174 (nan) | > current_lr: 5e-06 | > step_time: 0.1436 (0.13704897644363834) | > loader_time: 0.0046 (0.004691324724215211)  --> TIME: 2025-02-27 10:41:26 -- STEP: 2190/36938 -- GLOBAL_STEP: 1664400 | > loss_text_ce: 0.03251459822058678 (0.029912118511657194) | > loss_mel_ce: 2.6650891304016113 (nan) | > loss: 0.01070477720350027 (nan) | > current_lr: 5e-06 | > step_time: 0.1538 (0.13699584127016803) | > loader_time: 0.0067 (0.004693776287444647)  --> TIME: 2025-02-27 10:41:36 -- STEP: 2240/36938 -- GLOBAL_STEP: 1664450 | > loss_text_ce: 0.029430033639073372 (0.029908006743062286) | > loss_mel_ce: 2.3281729221343994 (nan) | > loss: 0.009355568327009678 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.13716686069965356) | > loader_time: 0.0046 (0.004693083571536196)  --> TIME: 2025-02-27 10:41:45 -- STEP: 2290/36938 -- GLOBAL_STEP: 1664500 | > loss_text_ce: 0.024081343784928322 (0.02990715951500176) | > loss_mel_ce: 2.5849242210388184 (nan) | > loss: 0.010353196412324905 (nan) | > current_lr: 5e-06 | > step_time: 0.1619 (0.13712784681778278) | > loader_time: 0.0053 (0.004694356252012291)  --> TIME: 2025-02-27 10:41:54 -- STEP: 2340/36938 -- GLOBAL_STEP: 1664550 | > loss_text_ce: 0.034683581441640854 (0.02993099803908959) | > loss_mel_ce: 2.8801398277282715 (nan) | > loss: 0.011566760018467903 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.13715315761729166) | > loader_time: 0.0046 (0.004691818840483312)  --> TIME: 2025-02-27 10:42:04 -- STEP: 2390/36938 -- GLOBAL_STEP: 1664600 | > loss_text_ce: 0.02945677749812603 (0.02989498645449096) | > loss_mel_ce: 2.7160210609436035 (nan) | > loss: 0.010894753970205784 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13720165097065024) | > loader_time: 0.0045 (0.004689278163670491)  --> TIME: 2025-02-27 10:42:14 -- STEP: 2440/36938 -- GLOBAL_STEP: 1664650 | > loss_text_ce: 0.017686877399683 (0.029899701652056004) | > loss_mel_ce: 2.796184539794922 (nan) | > loss: 0.011166157200932503 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13736745910566325) | > loader_time: 0.0044 (0.0046883548869461255)  --> TIME: 2025-02-27 10:42:23 -- STEP: 2490/36938 -- GLOBAL_STEP: 1664700 | > loss_text_ce: 0.033433981239795685 (0.029879169615292286) | > loss_mel_ce: 2.072427272796631 (nan) | > loss: 0.008356592617928982 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.13738509938420027) | > loader_time: 0.0048 (0.0046901466377288915)  --> TIME: 2025-02-27 10:42:33 -- STEP: 2540/36938 -- GLOBAL_STEP: 1664750 | > loss_text_ce: 0.03002075105905533 (0.02986315403932955) | > loss_mel_ce: 2.565150499343872 (nan) | > loss: 0.010298298671841621 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.1374156420625098) | > loader_time: 0.0046 (0.0046920293898094325)  --> TIME: 2025-02-27 10:42:42 -- STEP: 2590/36938 -- GLOBAL_STEP: 1664800 | > loss_text_ce: 0.029024388641119003 (0.029887207960314386) | > loss_mel_ce: 2.625380039215088 (nan) | > loss: 0.010533351451158524 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.13737779031849254) | > loader_time: 0.0045 (0.004690426657098597)  --> TIME: 2025-02-27 10:42:51 -- STEP: 2640/36938 -- GLOBAL_STEP: 1664850 | > loss_text_ce: 0.02421211451292038 (0.029882819521607776) | > loss_mel_ce: 2.586684465408325 (nan) | > loss: 0.010360701009631157 (nan) | > current_lr: 5e-06 | > step_time: 0.1773 (0.13738017867911925) | > loader_time: 0.0048 (0.004688858444040469)  --> TIME: 2025-02-27 10:43:01 -- STEP: 2690/36938 -- GLOBAL_STEP: 1664900 | > loss_text_ce: 0.03226465731859207 (0.029895793167410287) | > loss_mel_ce: 2.6669633388519287 (nan) | > loss: 0.010711222887039185 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13736212164939107) | > loader_time: 0.0045 (0.00468795609740076)  --> TIME: 2025-02-27 10:43:10 -- STEP: 2740/36938 -- GLOBAL_STEP: 1664950 | > loss_text_ce: 0.0337449312210083 (0.029883113060907945) | > loss_mel_ce: 2.775089979171753 (nan) | > loss: 0.011146171018481255 (nan) | > current_lr: 5e-06 | > step_time: 0.1554 (0.13734383435144917) | > loader_time: 0.0046 (0.004687394538935083)  --> TIME: 2025-02-27 10:43:19 -- STEP: 2790/36938 -- GLOBAL_STEP: 1665000 | > loss_text_ce: 0.018675314262509346 (0.029864555987526117) | > loss_mel_ce: 0.00115966796875 (nan) | > loss: 7.871025445638224e-05 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1372868183265877) | > loader_time: 0.0045 (0.00468613639954598)  --> TIME: 2025-02-27 10:43:28 -- STEP: 2840/36938 -- GLOBAL_STEP: 1665050 | > loss_text_ce: 0.038380417972803116 (0.02984989556438014) | > loss_mel_ce: 2.4140679836273193 (nan) | > loss: 0.009731938131153584 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13730536590159773) | > loader_time: 0.0045 (0.004685465718658876)  --> TIME: 2025-02-27 10:43:38 -- STEP: 2890/36938 -- GLOBAL_STEP: 1665100 | > loss_text_ce: 0.023745901882648468 (0.029846552226073703) | > loss_mel_ce: 2.6808536052703857 (nan) | > loss: 0.010732538998126984 (nan) | > current_lr: 5e-06 | > step_time: 0.1343 (0.13724961462317853) | > loader_time: 0.0046 (0.00468523980836967)  --> TIME: 2025-02-27 10:43:47 -- STEP: 2940/36938 -- GLOBAL_STEP: 1665150 | > loss_text_ce: 0.035017527639865875 (0.029838583732758755) | > loss_mel_ce: 2.431455373764038 (nan) | > loss: 0.00978759117424488 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.13721615254473513) | > loader_time: 0.0047 (0.004685053695626809)  --> TIME: 2025-02-27 10:43:56 -- STEP: 2990/36938 -- GLOBAL_STEP: 1665200 | > loss_text_ce: 0.02823198027908802 (0.029829038393350448) | > loss_mel_ce: 2.7354817390441895 (nan) | > loss: 0.010967118665575981 (nan) | > current_lr: 5e-06 | > step_time: 0.177 (0.137165466439365) | > loader_time: 0.0047 (0.004683477105105598)  --> TIME: 2025-02-27 10:44:05 -- STEP: 3040/36938 -- GLOBAL_STEP: 1665250 | > loss_text_ce: 0.029385481029748917 (0.02984114212786632) | > loss_mel_ce: 1.3601608276367188 (nan) | > loss: 0.005514072719961405 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1371678349218869) | > loader_time: 0.0046 (0.004682391176098272)  --> TIME: 2025-02-27 10:44:15 -- STEP: 3090/36938 -- GLOBAL_STEP: 1665300 | > loss_text_ce: 0.027645081281661987 (0.029837182967420245) | > loss_mel_ce: 0.7814905047416687 (nan) | > loss: 0.003210855647921562 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13718195231601252) | > loader_time: 0.0053 (0.0046823561770244744)  --> TIME: 2025-02-27 10:44:24 -- STEP: 3140/36938 -- GLOBAL_STEP: 1665350 | > loss_text_ce: 0.022891541942954063 (0.02982342613617753) | > loss_mel_ce: 0.21780967712402344 (nan) | > loss: 0.000955163617618382 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1371775937687818) | > loader_time: 0.0046 (0.004682043251717929)  --> TIME: 2025-02-27 10:44:34 -- STEP: 3190/36938 -- GLOBAL_STEP: 1665400 | > loss_text_ce: 0.0379316546022892 (0.0298457475878345) | > loss_mel_ce: 2.7417988777160645 (nan) | > loss: 0.011030677706003189 (nan) | > current_lr: 5e-06 | > step_time: 0.1609 (0.13718092067861998) | > loader_time: 0.0048 (0.004680991397008633)  --> TIME: 2025-02-27 10:44:43 -- STEP: 3240/36938 -- GLOBAL_STEP: 1665450 | > loss_text_ce: 0.025163354352116585 (0.029828316814634076) | > loss_mel_ce: 0.005584716796875 (nan) | > loss: 0.00012201616482343525 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.13724363409442653) | > loader_time: 0.0048 (0.00468011601471607)  --> TIME: 2025-02-27 10:44:53 -- STEP: 3290/36938 -- GLOBAL_STEP: 1665500 | > loss_text_ce: 0.02864227257668972 (0.029844632756961424) | > loss_mel_ce: 3.0278005599975586 (nan) | > loss: 0.012128741480410099 (nan) | > current_lr: 5e-06 | > step_time: 0.1609 (0.13733003603288824) | > loader_time: 0.0047 (0.004679915825284361)  --> TIME: 2025-02-27 10:45:02 -- STEP: 3340/36938 -- GLOBAL_STEP: 1665550 | > loss_text_ce: 0.03151293098926544 (0.02983951027906628) | > loss_mel_ce: 2.088169574737549 (nan) | > loss: 0.008411439135670662 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13738446499773124) | > loader_time: 0.0045 (0.00467950940845969)  --> TIME: 2025-02-27 10:45:12 -- STEP: 3390/36938 -- GLOBAL_STEP: 1665600 | > loss_text_ce: 0.027408527210354805 (0.029829559674909596) | > loss_mel_ce: 2.839829921722412 (nan) | > loss: 0.011377931572496891 (nan) | > current_lr: 5e-06 | > step_time: 0.1664 (0.1373508109455615) | > loader_time: 0.0048 (0.004678917775111911)  --> TIME: 2025-02-27 10:45:22 -- STEP: 3440/36938 -- GLOBAL_STEP: 1665650 | > loss_text_ce: 0.0360734686255455 (0.029825919915564617) | > loss_mel_ce: 2.9465086460113525 (nan) | > loss: 0.011835644021630287 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13744900288969975) | > loader_time: 0.0045 (0.004678416737290309)  --> TIME: 2025-02-27 10:45:31 -- STEP: 3490/36938 -- GLOBAL_STEP: 1665700 | > loss_text_ce: 0.027430258691310883 (0.029838392078438854) | > loss_mel_ce: 1.0258585214614868 (nan) | > loss: 0.004179717972874641 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.13746074373196057) | > loader_time: 0.0045 (0.004677506572537566)  --> TIME: 2025-02-27 10:45:40 -- STEP: 3540/36938 -- GLOBAL_STEP: 1665750 | > loss_text_ce: 0.03554953262209892 (0.02982178490850969) | > loss_mel_ce: 0.7449123859405518 (nan) | > loss: 0.003097071312367916 (nan) | > current_lr: 5e-06 | > step_time: 0.1606 (0.13746593072589514) | > loader_time: 0.0046 (0.004676011457281591)  --> TIME: 2025-02-27 10:45:49 -- STEP: 3590/36938 -- GLOBAL_STEP: 1665800 | > loss_text_ce: 0.030793840065598488 (0.029810569298383584) | > loss_mel_ce: 2.9398632049560547 (nan) | > loss: 0.0117883225902915 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.1373454626556227) | > loader_time: 0.0046 (0.00467523791331766)  --> TIME: 2025-02-27 10:45:58 -- STEP: 3640/36938 -- GLOBAL_STEP: 1665850 | > loss_text_ce: 0.032485347241163254 (0.029810594993510416) | > loss_mel_ce: 2.6074461936950684 (nan) | > loss: 0.010475918650627136 (nan) | > current_lr: 5e-06 | > step_time: 0.1228 (0.13731742704307642) | > loader_time: 0.0045 (0.0046739128264752)  --> TIME: 2025-02-27 10:46:08 -- STEP: 3690/36938 -- GLOBAL_STEP: 1665900 | > loss_text_ce: 0.027080420404672623 (0.029810117155838464) | > loss_mel_ce: 1.3175877332687378 (nan) | > loss: 0.005335985217243433 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.1373180739601776) | > loader_time: 0.0045 (0.00467246341188425)  --> TIME: 2025-02-27 10:46:17 -- STEP: 3740/36938 -- GLOBAL_STEP: 1665950 | > loss_text_ce: 0.028953155502676964 (0.029795932233144334) | > loss_mel_ce: 1.9599931240081787 (nan) | > loss: 0.007892644964158535 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13727335547380895) | > loader_time: 0.0045 (0.0046724898292419045)  --> TIME: 2025-02-27 10:46:26 -- STEP: 3790/36938 -- GLOBAL_STEP: 1666000 | > loss_text_ce: 0.035901784896850586 (0.02980933316361582) | > loss_mel_ce: 1.6837245225906372 (nan) | > loss: 0.006823914125561714 (nan) | > current_lr: 5e-06 | > step_time: 0.1514 (0.1371865141045763) | > loader_time: 0.0046 (0.00467208639612927)  --> TIME: 2025-02-27 10:46:35 -- STEP: 3840/36938 -- GLOBAL_STEP: 1666050 | > loss_text_ce: 0.036968447268009186 (0.029809373279810338) | > loss_mel_ce: 2.591433525085449 (nan) | > loss: 0.01043016742914915 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13721258398145425) | > loader_time: 0.0045 (0.00467241133252779)  --> TIME: 2025-02-27 10:46:45 -- STEP: 3890/36938 -- GLOBAL_STEP: 1666100 | > loss_text_ce: 0.023500487208366394 (0.02978962917852869) | > loss_mel_ce: 2.2585976123809814 (nan) | > loss: 0.009055945090949535 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13719184588650493) | > loader_time: 0.0045 (0.004672567948584072)  --> TIME: 2025-02-27 10:46:54 -- STEP: 3940/36938 -- GLOBAL_STEP: 1666150 | > loss_text_ce: 0.03395974636077881 (0.029784758925333922) | > loss_mel_ce: 3.012568712234497 (nan) | > loss: 0.012089398689568043 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13725334532974928) | > loader_time: 0.0046 (0.004672413550052539)  --> TIME: 2025-02-27 10:47:04 -- STEP: 3990/36938 -- GLOBAL_STEP: 1666200 | > loss_text_ce: 0.027710286900401115 (0.02978054674267582) | > loss_mel_ce: 1.9246344566345215 (nan) | > loss: 0.007747400552034378 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1372957442338603) | > loader_time: 0.0045 (0.004672547390586445)  --> TIME: 2025-02-27 10:47:13 -- STEP: 4040/36938 -- GLOBAL_STEP: 1666250 | > loss_text_ce: 0.026831194758415222 (0.02977833184206139) | > loss_mel_ce: 2.356717109680176 (nan) | > loss: 0.009458525106310844 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13732868970030593) | > loader_time: 0.0046 (0.004672357056400557)  --> TIME: 2025-02-27 10:47:23 -- STEP: 4090/36938 -- GLOBAL_STEP: 1666300 | > loss_text_ce: 0.04070545732975006 (0.02979813482097753) | > loss_mel_ce: 2.2333831787109375 (nan) | > loss: 0.009024161845445633 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.1373986539164498) | > loader_time: 0.0045 (0.0046720946039139144)  --> TIME: 2025-02-27 10:47:33 -- STEP: 4140/36938 -- GLOBAL_STEP: 1666350 | > loss_text_ce: 0.03567122295498848 (0.029808204806216314) | > loss_mel_ce: 2.2246880531311035 (nan) | > loss: 0.008969680406153202 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13744787645800666) | > loader_time: 0.0047 (0.004671941575220812)  --> TIME: 2025-02-27 10:47:43 -- STEP: 4190/36938 -- GLOBAL_STEP: 1666400 | > loss_text_ce: 0.024698205292224884 (0.029799731134665965) | > loss_mel_ce: 3.055759906768799 (nan) | > loss: 0.012224040925502777 (nan) | > current_lr: 5e-06 | > step_time: 0.1695 (0.13751674108118306) | > loader_time: 0.0048 (0.004672896435266457)  --> TIME: 2025-02-27 10:47:52 -- STEP: 4240/36938 -- GLOBAL_STEP: 1666450 | > loss_text_ce: 0.032761357724666595 (0.029799971489877133) | > loss_mel_ce: 2.792653799057007 (nan) | > loss: 0.01121196523308754 (nan) | > current_lr: 5e-06 | > step_time: 0.175 (0.1375575933816297) | > loader_time: 0.0047 (0.004673539748731644)  --> TIME: 2025-02-27 10:48:02 -- STEP: 4290/36938 -- GLOBAL_STEP: 1666500 | > loss_text_ce: 0.025674976408481598 (0.029798947370856445) | > loss_mel_ce: 3.3489582538604736 (nan) | > loss: 0.013391402550041676 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.1375953505922863) | > loader_time: 0.0045 (0.004673549956652939)  --> TIME: 2025-02-27 10:48:11 -- STEP: 4340/36938 -- GLOBAL_STEP: 1666550 | > loss_text_ce: 0.043807465583086014 (0.02980205048635293) | > loss_mel_ce: 0.07599099725484848 (nan) | > loss: 0.00047539075603708625 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13754817533053548) | > loader_time: 0.0045 (0.004672813470462494)  --> TIME: 2025-02-27 10:48:20 -- STEP: 4390/36938 -- GLOBAL_STEP: 1666600 | > loss_text_ce: 0.028217636048793793 (0.02982156407575978) | > loss_mel_ce: 2.1009957790374756 (nan) | > loss: 0.008449260145425797 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1375721260735548) | > loader_time: 0.0045 (0.0046721798412349014)  --> TIME: 2025-02-27 10:48:30 -- STEP: 4440/36938 -- GLOBAL_STEP: 1666650 | > loss_text_ce: 0.03314138203859329 (0.029818342401444712) | > loss_mel_ce: 2.643336296081543 (nan) | > loss: 0.010620944201946259 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13760837343361992) | > loader_time: 0.0047 (0.004672345653310546)  --> TIME: 2025-02-27 10:48:39 -- STEP: 4490/36938 -- GLOBAL_STEP: 1666700 | > loss_text_ce: 0.033271946012973785 (0.029839969798300274) | > loss_mel_ce: 2.23382568359375 (nan) | > loss: 0.008996419608592987 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.1375882857626424) | > loader_time: 0.0047 (0.004672260804802905)  --> TIME: 2025-02-27 10:48:49 -- STEP: 4540/36938 -- GLOBAL_STEP: 1666750 | > loss_text_ce: 0.030417421832680702 (0.029831788455246894) | > loss_mel_ce: 2.592331886291504 (nan) | > loss: 0.010407735593616962 (nan) | > current_lr: 5e-06 | > step_time: 0.1379 (0.13758207086949611) | > loader_time: 0.0046 (0.004671835531747285)  --> TIME: 2025-02-27 10:48:58 -- STEP: 4590/36938 -- GLOBAL_STEP: 1666800 | > loss_text_ce: 0.03362013399600983 (0.029824861614459675) | > loss_mel_ce: 2.6911556720733643 (nan) | > loss: 0.010812602937221527 (nan) | > current_lr: 5e-06 | > step_time: 0.1488 (0.13751050802617282) | > loader_time: 0.0046 (0.004671516917110267)  --> TIME: 2025-02-27 10:49:07 -- STEP: 4640/36938 -- GLOBAL_STEP: 1666850 | > loss_text_ce: 0.030914081260561943 (0.02981079391320236) | > loss_mel_ce: 2.958186388015747 (nan) | > loss: 0.011861510574817657 (nan) | > current_lr: 5e-06 | > step_time: 0.167 (0.1374829974667779) | > loader_time: 0.0046 (0.004670850059081769)  --> TIME: 2025-02-27 10:49:16 -- STEP: 4690/36938 -- GLOBAL_STEP: 1666900 | > loss_text_ce: 0.027929043397307396 (0.02980776698425857) | > loss_mel_ce: 2.412144660949707 (nan) | > loss: 0.009682833217084408 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.1374801039441562) | > loader_time: 0.0048 (0.004670374378212479)  --> TIME: 2025-02-27 10:49:26 -- STEP: 4740/36938 -- GLOBAL_STEP: 1666950 | > loss_text_ce: 0.029407605528831482 (0.029807528785040862) | > loss_mel_ce: 2.894540548324585 (nan) | > loss: 0.011602968908846378 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.1374791146833685) | > loader_time: 0.0046 (0.004670291158217418)  --> TIME: 2025-02-27 10:49:35 -- STEP: 4790/36938 -- GLOBAL_STEP: 1667000 | > loss_text_ce: 0.03610458970069885 (0.02981470067769538) | > loss_mel_ce: 2.8800806999206543 (nan) | > loss: 0.011572164483368397 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.1374836029239885) | > loader_time: 0.0046 (0.004670600851293897)  --> TIME: 2025-02-27 10:49:45 -- STEP: 4840/36938 -- GLOBAL_STEP: 1667050 | > loss_text_ce: 0.03154894337058067 (0.029809515175509674) | > loss_mel_ce: 0.036865234375 (nan) | > loss: 0.00027148486697115004 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13750228433569597) | > loader_time: 0.0047 (0.004670534350655296)  --> TIME: 2025-02-27 10:49:54 -- STEP: 4890/36938 -- GLOBAL_STEP: 1667100 | > loss_text_ce: 0.03330875188112259 (0.029811572787068127) | > loss_mel_ce: 2.4917068481445312 (nan) | > loss: 0.010019904002547264 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13748798965188883) | > loader_time: 0.0045 (0.004670199928595727)  --> TIME: 2025-02-27 10:50:03 -- STEP: 4940/36938 -- GLOBAL_STEP: 1667150 | > loss_text_ce: 0.02658446691930294 (0.02981120559767733) | > loss_mel_ce: 2.769857168197632 (nan) | > loss: 0.011096990667283535 (nan) | > current_lr: 5e-06 | > step_time: 0.1793 (0.13743122381719997) | > loader_time: 0.0048 (0.004669816725649812)  --> TIME: 2025-02-27 10:50:12 -- STEP: 4990/36938 -- GLOBAL_STEP: 1667200 | > loss_text_ce: 0.029282208532094955 (0.02980698689870641) | > loss_mel_ce: 2.73876953125 (nan) | > loss: 0.010984332300722599 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13737513961677336) | > loader_time: 0.0045 (0.00466961798543681)  --> TIME: 2025-02-27 10:50:21 -- STEP: 5040/36938 -- GLOBAL_STEP: 1667250 | > loss_text_ce: 0.037769317626953125 (0.029806516427058904) | > loss_mel_ce: 2.5593581199645996 (nan) | > loss: 0.010306062176823616 (nan) | > current_lr: 5e-06 | > step_time: 0.1132 (0.13736035781247286) | > loader_time: 0.0045 (0.004669059174401409)  --> TIME: 2025-02-27 10:50:30 -- STEP: 5090/36938 -- GLOBAL_STEP: 1667300 | > loss_text_ce: 0.033238060772418976 (0.02979392300713039) | > loss_mel_ce: 2.7455332279205322 (nan) | > loss: 0.011026871390640736 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13730523843896422) | > loader_time: 0.0046 (0.004668598043895177)  --> TIME: 2025-02-27 10:50:40 -- STEP: 5140/36938 -- GLOBAL_STEP: 1667350 | > loss_text_ce: 0.03131275996565819 (0.029786338339875006) | > loss_mel_ce: 2.603269577026367 (nan) | > loss: 0.0104546919465065 (nan) | > current_lr: 5e-06 | > step_time: 0.1537 (0.13730546648864167) | > loader_time: 0.0046 (0.004667925927425628)  --> TIME: 2025-02-27 10:50:49 -- STEP: 5190/36938 -- GLOBAL_STEP: 1667400 | > loss_text_ce: 0.03969689831137657 (0.029791473392899614) | > loss_mel_ce: 3.3916015625 (nan) | > loss: 0.013616264797747135 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13731171055795624) | > loader_time: 0.0046 (0.0046674563935266925)  --> TIME: 2025-02-27 10:50:59 -- STEP: 5240/36938 -- GLOBAL_STEP: 1667450 | > loss_text_ce: 0.02826288901269436 (0.02979118341263668) | > loss_mel_ce: 2.123342752456665 (nan) | > loss: 0.00853811763226986 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13731565589213215) | > loader_time: 0.0045 (0.0046668404841240935)  --> TIME: 2025-02-27 10:51:08 -- STEP: 5290/36938 -- GLOBAL_STEP: 1667500 | > loss_text_ce: 0.039008837193250656 (0.0298134299525058) | > loss_mel_ce: 1.1263309717178345 (nan) | > loss: 0.00462436443194747 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1373038869724383) | > loader_time: 0.0045 (0.004666210122279715)  --> TIME: 2025-02-27 10:51:17 -- STEP: 5340/36938 -- GLOBAL_STEP: 1667550 | > loss_text_ce: 0.0307293888181448 (0.029809536304401602) | > loss_mel_ce: 2.4342432022094727 (nan) | > loss: 0.009781637229025364 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.1372460935892686) | > loader_time: 0.0046 (0.0046663703989893186)  --> TIME: 2025-02-27 10:51:26 -- STEP: 5390/36938 -- GLOBAL_STEP: 1667600 | > loss_text_ce: 0.024558214470744133 (0.029806519037973073) | > loss_mel_ce: 1.9004273414611816 (nan) | > loss: 0.007638832088559866 (nan) | > current_lr: 5e-06 | > step_time: 0.1074 (0.13727744603201303) | > loader_time: 0.0044 (0.004667608061172955)  --> TIME: 2025-02-27 10:51:36 -- STEP: 5440/36938 -- GLOBAL_STEP: 1667650 | > loss_text_ce: 0.035272564738988876 (0.029809226956636205) | > loss_mel_ce: 1.9787027835845947 (nan) | > loss: 0.007991965860128403 (nan) | > current_lr: 5e-06 | > step_time: 0.1694 (0.13727120038341104) | > loader_time: 0.0047 (0.004668239460271938)  --> TIME: 2025-02-27 10:51:45 -- STEP: 5490/36938 -- GLOBAL_STEP: 1667700 | > loss_text_ce: 0.0277868639677763 (0.029814225081218667) | > loss_mel_ce: 3.0650248527526855 (nan) | > loss: 0.012273063883185387 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13723192723071015) | > loader_time: 0.0046 (0.004668796518461292)  --> TIME: 2025-02-27 10:51:54 -- STEP: 5540/36938 -- GLOBAL_STEP: 1667750 | > loss_text_ce: 0.02238117903470993 (0.029801046708262516) | > loss_mel_ce: 1.924872636795044 (nan) | > loss: 0.007727198302745819 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13725627239000082) | > loader_time: 0.0046 (0.004668789779236164)  --> TIME: 2025-02-27 10:52:04 -- STEP: 5590/36938 -- GLOBAL_STEP: 1667800 | > loss_text_ce: 0.03307477757334709 (0.029800933282422518) | > loss_mel_ce: 3.0887277126312256 (nan) | > loss: 0.012388106435537338 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.1372540099463013) | > loader_time: 0.0046 (0.0046687744597842685)  --> TIME: 2025-02-27 10:52:13 -- STEP: 5640/36938 -- GLOBAL_STEP: 1667850 | > loss_text_ce: 0.029464859515428543 (0.02980249878035607) | > loss_mel_ce: 0.5634562373161316 (nan) | > loss: 0.0023528614547103643 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1372478900649025) | > loader_time: 0.0045 (0.004668390201338633)  --> TIME: 2025-02-27 10:52:22 -- STEP: 5690/36938 -- GLOBAL_STEP: 1667900 | > loss_text_ce: 0.0379638671875 (0.029807222387892553) | > loss_mel_ce: 2.3109519481658936 (nan) | > loss: 0.009321095421910286 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13721840704262586) | > loader_time: 0.0045 (0.004668117114236354)  --> TIME: 2025-02-27 10:52:31 -- STEP: 5740/36938 -- GLOBAL_STEP: 1667950 | > loss_text_ce: 0.025507261976599693 (0.02981170174529458) | > loss_mel_ce: 1.723127007484436 (nan) | > loss: 0.0069390251301229 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13722739082595622) | > loader_time: 0.0047 (0.004668318560729867)  --> TIME: 2025-02-27 10:52:41 -- STEP: 5790/36938 -- GLOBAL_STEP: 1668000 | > loss_text_ce: 0.02928750030696392 (0.029807972996005072) | > loss_mel_ce: 2.3505859375 (nan) | > loss: 0.00944394338876009 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.1372253325319044) | > loader_time: 0.0045 (0.004667759518137454)  --> TIME: 2025-02-27 10:52:50 -- STEP: 5840/36938 -- GLOBAL_STEP: 1668050 | > loss_text_ce: 0.024160111322999 (0.02980926332003373) | > loss_mel_ce: 2.9229352474212646 (nan) | > loss: 0.011694823391735554 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13721981089409102) | > loader_time: 0.0046 (0.004667832181878285)  --> TIME: 2025-02-27 10:53:00 -- STEP: 5890/36938 -- GLOBAL_STEP: 1668100 | > loss_text_ce: 0.028357641771435738 (0.029810190303741317) | > loss_mel_ce: 2.530754566192627 (nan) | > loss: 0.01015520840883255 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13724097124789508) | > loader_time: 0.0045 (0.004667819699933454)  --> TIME: 2025-02-27 10:53:09 -- STEP: 5940/36938 -- GLOBAL_STEP: 1668150 | > loss_text_ce: 0.025414185598492622 (0.029801240538776873) | > loss_mel_ce: 1.4962490797042847 (nan) | > loss: 0.006038346793502569 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.1371916475119417) | > loader_time: 0.0045 (0.004667148726556439)  --> TIME: 2025-02-27 10:53:18 -- STEP: 5990/36938 -- GLOBAL_STEP: 1668200 | > loss_text_ce: 0.034752920269966125 (0.029798120778762986) | > loss_mel_ce: 2.353269577026367 (nan) | > loss: 0.009476279839873314 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13720089362340335) | > loader_time: 0.0045 (0.0046674471665702445)  --> TIME: 2025-02-27 10:53:27 -- STEP: 6040/36938 -- GLOBAL_STEP: 1668250 | > loss_text_ce: 0.02642049454152584 (0.029811747618087425) | > loss_mel_ce: 2.7476632595062256 (nan) | > loss: 0.01100827008485794 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1372068278836892) | > loader_time: 0.0046 (0.0046671929738379535)  --> TIME: 2025-02-27 10:53:37 -- STEP: 6090/36938 -- GLOBAL_STEP: 1668300 | > loss_text_ce: 0.03195079788565636 (0.029805260645730303) | > loss_mel_ce: 2.6304049491882324 (nan) | > loss: 0.010564903728663921 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13720382113370627) | > loader_time: 0.0046 (0.00466723583014728)  --> TIME: 2025-02-27 10:53:46 -- STEP: 6140/36938 -- GLOBAL_STEP: 1668350 | > loss_text_ce: 0.03133321553468704 (0.029804296746049137) | > loss_mel_ce: 3.125582218170166 (nan) | > loss: 0.012527442537248135 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13717380560182246) | > loader_time: 0.0046 (0.00466688471430676)  --> TIME: 2025-02-27 10:53:55 -- STEP: 6190/36938 -- GLOBAL_STEP: 1668400 | > loss_text_ce: 0.03996939957141876 (0.029803560595351584) | > loss_mel_ce: 1.232208251953125 (nan) | > loss: 0.005048324353992939 (nan) | > current_lr: 5e-06 | > step_time: 0.1087 (0.13717097011251903) | > loader_time: 0.0045 (0.004666712981241008)  --> TIME: 2025-02-27 10:54:05 -- STEP: 6240/36938 -- GLOBAL_STEP: 1668450 | > loss_text_ce: 0.04633476212620735 (0.029796997681021307) | > loss_mel_ce: 2.457099676132202 (nan) | > loss: 0.009934264235198498 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13722656999643068) | > loader_time: 0.0046 (0.004667266095295932)  --> TIME: 2025-02-27 10:54:14 -- STEP: 6290/36938 -- GLOBAL_STEP: 1668500 | > loss_text_ce: 0.027396462857723236 (0.02979608905754853) | > loss_mel_ce: 1.3877753019332886 (nan) | > loss: 0.005615761037915945 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13720921166181982) | > loader_time: 0.0046 (0.004667119305160351)  --> TIME: 2025-02-27 10:54:24 -- STEP: 6340/36938 -- GLOBAL_STEP: 1668550 | > loss_text_ce: 0.02038596197962761 (0.0297988168601452) | > loss_mel_ce: 2.0237088203430176 (nan) | > loss: 0.008111488074064255 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13723263514906858) | > loader_time: 0.0054 (0.004667273380026828)  --> TIME: 2025-02-27 10:54:33 -- STEP: 6390/36938 -- GLOBAL_STEP: 1668600 | > loss_text_ce: 0.03442591056227684 (0.02980236030782752) | > loss_mel_ce: 0.019287109375 (nan) | > loss: 0.00021314692276064306 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.13721332766453836) | > loader_time: 0.0045 (0.004667623390054474)  --> TIME: 2025-02-27 10:54:43 -- STEP: 6440/36938 -- GLOBAL_STEP: 1668650 | > loss_text_ce: 0.02401066944003105 (0.029799678182136847) | > loss_mel_ce: 2.8899950981140137 (nan) | > loss: 0.011563515290617943 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.13723024851787197) | > loader_time: 0.0048 (0.00466752252223328)  --> TIME: 2025-02-27 10:54:52 -- STEP: 6490/36938 -- GLOBAL_STEP: 1668700 | > loss_text_ce: 0.023484503850340843 (0.029801193586850572) | > loss_mel_ce: 0.9788792729377747 (nan) | > loss: 0.0039776344783604145 (nan) | > current_lr: 5e-06 | > step_time: 0.1116 (0.13723699297118852) | > loader_time: 0.0046 (0.004667490766669267)  --> TIME: 2025-02-27 10:55:01 -- STEP: 6540/36938 -- GLOBAL_STEP: 1668750 | > loss_text_ce: 0.025186380371451378 (0.029807129447328478) | > loss_mel_ce: 2.6393556594848633 (nan) | > loss: 0.01057357992976904 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13723495064890026) | > loader_time: 0.0046 (0.0046674790367803)  --> TIME: 2025-02-27 10:55:11 -- STEP: 6590/36938 -- GLOBAL_STEP: 1668800 | > loss_text_ce: 0.028175989165902138 (0.029795749973483124) | > loss_mel_ce: 3.1257317066192627 (nan) | > loss: 0.012515507638454437 (nan) | > current_lr: 5e-06 | > step_time: 0.1731 (0.1372394488686319) | > loader_time: 0.0052 (0.004667589624662494)  --> TIME: 2025-02-27 10:55:20 -- STEP: 6640/36938 -- GLOBAL_STEP: 1668850 | > loss_text_ce: 0.03386928141117096 (0.029788498386196478) | > loss_mel_ce: 2.8159561157226562 (nan) | > loss: 0.011308831162750721 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13723100793648965) | > loader_time: 0.0046 (0.004667692550693643)  --> TIME: 2025-02-27 10:55:29 -- STEP: 6690/36938 -- GLOBAL_STEP: 1668900 | > loss_text_ce: 0.03366541117429733 (0.029793901297281898) | > loss_mel_ce: 2.9178667068481445 (nan) | > loss: 0.01171243004500866 (nan) | > current_lr: 5e-06 | > step_time: 0.1781 (0.13725057813618802) | > loader_time: 0.0048 (0.004667885777483424)  --> TIME: 2025-02-27 10:55:39 -- STEP: 6740/36938 -- GLOBAL_STEP: 1668950 | > loss_text_ce: 0.033321864902973175 (0.029804477833062188) | > loss_mel_ce: 3.0881271362304688 (nan) | > loss: 0.012386702932417393 (nan) | > current_lr: 5e-06 | > step_time: 0.186 (0.13722626780897684) | > loader_time: 0.0049 (0.0046676401214712756)  --> TIME: 2025-02-27 10:55:48 -- STEP: 6790/36938 -- GLOBAL_STEP: 1669000 | > loss_text_ce: 0.03719145804643631 (0.029804813963483157) | > loss_mel_ce: 2.6065402030944824 (nan) | > loss: 0.01049099862575531 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13719773306586827) | > loader_time: 0.0048 (0.004668305478496295)  --> TIME: 2025-02-27 10:55:57 -- STEP: 6840/36938 -- GLOBAL_STEP: 1669050 | > loss_text_ce: 0.030072784051299095 (0.029804332352417647) | > loss_mel_ce: 2.700087547302246 (nan) | > loss: 0.01083397027105093 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.13718974360248556) | > loader_time: 0.0048 (0.00466812936186093)  --> TIME: 2025-02-27 10:56:06 -- STEP: 6890/36938 -- GLOBAL_STEP: 1669100 | > loss_text_ce: 0.03693235665559769 (0.02980779434752979) | > loss_mel_ce: 2.8659775257110596 (nan) | > loss: 0.011519485153257847 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13716742611761945) | > loader_time: 0.0046 (0.004667903619165513)  --> TIME: 2025-02-27 10:56:16 -- STEP: 6940/36938 -- GLOBAL_STEP: 1669150 | > loss_text_ce: 0.019346313551068306 (0.029801285919204612) | > loss_mel_ce: 2.1600494384765625 (nan) | > loss: 0.008648396469652653 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13719146804782148) | > loader_time: 0.0045 (0.004668034909437984)  --> TIME: 2025-02-27 10:56:25 -- STEP: 6990/36938 -- GLOBAL_STEP: 1669200 | > loss_text_ce: 0.02249867096543312 (0.029806269441620688) | > loss_mel_ce: 2.5940964221954346 (nan) | > loss: 0.010383314453065395 (nan) | > current_lr: 5e-06 | > step_time: 0.1577 (0.1372149635282202) | > loader_time: 0.0047 (0.004668349701277001)  --> TIME: 2025-02-27 10:56:35 -- STEP: 7040/36938 -- GLOBAL_STEP: 1669250 | > loss_text_ce: 0.01816205494105816 (0.02980605007151925) | > loss_mel_ce: 1.3176841735839844 (nan) | > loss: 0.0053009772673249245 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13725263957272923) | > loader_time: 0.0046 (0.004667869074778119)  --> TIME: 2025-02-27 10:56:44 -- STEP: 7090/36938 -- GLOBAL_STEP: 1669300 | > loss_text_ce: 0.024336712434887886 (0.029800560550661746) | > loss_mel_ce: 2.4116384983062744 (nan) | > loss: 0.009666569530963898 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13723375333213036) | > loader_time: 0.0046 (0.004667416210739498)  --> TIME: 2025-02-27 10:56:54 -- STEP: 7140/36938 -- GLOBAL_STEP: 1669350 | > loss_text_ce: 0.017637494951486588 (0.029802884928313414) | > loss_mel_ce: 2.6232895851135254 (nan) | > loss: 0.01047987025231123 (nan) | > current_lr: 5e-06 | > step_time: 0.1622 (0.13724678100324192) | > loader_time: 0.0046 (0.004667716066376498)  --> TIME: 2025-02-27 10:57:03 -- STEP: 7190/36938 -- GLOBAL_STEP: 1669400 | > loss_text_ce: 0.026686446741223335 (0.029806581643825818) | > loss_mel_ce: 3.6553235054016113 (nan) | > loss: 0.014611151069402695 (nan) | > current_lr: 5e-06 | > step_time: 0.1254 (0.13723913582044503) | > loader_time: 0.0045 (0.0046672977890523945)  --> TIME: 2025-02-27 10:57:12 -- STEP: 7240/36938 -- GLOBAL_STEP: 1669450 | > loss_text_ce: 0.031850382685661316 (0.029810925302203764) | > loss_mel_ce: 2.6090662479400635 (nan) | > loss: 0.010479828342795372 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13723135188798277) | > loader_time: 0.0045 (0.004667070359814888)  --> TIME: 2025-02-27 10:57:21 -- STEP: 7290/36938 -- GLOBAL_STEP: 1669500 | > loss_text_ce: 0.020060254260897636 (0.029815953747919965) | > loss_mel_ce: 2.1932618618011475 (nan) | > loss: 0.008783024735748768 (nan) | > current_lr: 5e-06 | > step_time: 0.1131 (0.13722134529317892) | > loader_time: 0.0045 (0.004666723112674076)  --> TIME: 2025-02-27 10:57:31 -- STEP: 7340/36938 -- GLOBAL_STEP: 1669550 | > loss_text_ce: 0.02634369768202305 (0.02980877960009414) | > loss_mel_ce: 2.4807217121124268 (nan) | > loss: 0.009948672726750374 (nan) | > current_lr: 5e-06 | > step_time: 0.1572 (0.13719505201568438) | > loader_time: 0.0046 (0.004666598583761934)  --> TIME: 2025-02-27 10:57:40 -- STEP: 7390/36938 -- GLOBAL_STEP: 1669600 | > loss_text_ce: 0.02817125990986824 (0.02981001960519444) | > loss_mel_ce: 2.9444186687469482 (nan) | > loss: 0.011795992963016033 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13718212409013064) | > loader_time: 0.0045 (0.004666164053631728)  --> TIME: 2025-02-27 10:57:49 -- STEP: 7440/36938 -- GLOBAL_STEP: 1669650 | > loss_text_ce: 0.02440619468688965 (0.029805967756997672) | > loss_mel_ce: 2.927813768386841 (nan) | > loss: 0.011715158820152283 (nan) | > current_lr: 5e-06 | > step_time: 0.182 (0.1371633698863372) | > loader_time: 0.0047 (0.004665635285838954)  --> TIME: 2025-02-27 10:57:58 -- STEP: 7490/36938 -- GLOBAL_STEP: 1669700 | > loss_text_ce: 0.030915789306163788 (0.029802696614243954) | > loss_mel_ce: 2.4460785388946533 (nan) | > loss: 0.009829343296587467 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.1371431813857589) | > loader_time: 0.0046 (0.004665207385380211)  --> TIME: 2025-02-27 10:58:07 -- STEP: 7540/36938 -- GLOBAL_STEP: 1669750 | > loss_text_ce: 0.02941664308309555 (0.029804386134785588) | > loss_mel_ce: 2.5014243125915527 (nan) | > loss: 0.010043020360171795 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13713330688147757) | > loader_time: 0.0046 (0.004665216526871331)  --> TIME: 2025-02-27 10:58:17 -- STEP: 7590/36938 -- GLOBAL_STEP: 1669800 | > loss_text_ce: 0.029358360916376114 (0.029801952080587623) | > loss_mel_ce: 3.1931865215301514 (nan) | > loss: 0.012787877582013607 (nan) | > current_lr: 5e-06 | > step_time: 0.1871 (0.13716326623723174) | > loader_time: 0.0048 (0.004664859187461639)  --> TIME: 2025-02-27 10:58:26 -- STEP: 7640/36938 -- GLOBAL_STEP: 1669850 | > loss_text_ce: 0.028824588283896446 (0.02980212516612154) | > loss_mel_ce: 2.7936694622039795 (nan) | > loss: 0.011200373992323875 (nan) | > current_lr: 5e-06 | > step_time: 0.1512 (0.13717468627460389) | > loader_time: 0.0047 (0.0046649873256683315)  --> TIME: 2025-02-27 10:58:36 -- STEP: 7690/36938 -- GLOBAL_STEP: 1669900 | > loss_text_ce: 0.03607127070426941 (0.02980997116112372) | > loss_mel_ce: 2.4965877532958984 (nan) | > loss: 0.010050235316157341 (nan) | > current_lr: 5e-06 | > step_time: 0.109 (0.13716248728053979) | > loader_time: 0.0045 (0.004664694100269567)  --> TIME: 2025-02-27 10:58:45 -- STEP: 7740/36938 -- GLOBAL_STEP: 1669950 | > loss_text_ce: 0.02806161902844906 (0.02981251497266848) | > loss_mel_ce: 1.5679510831832886 (nan) | > loss: 0.006333384197205305 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.1371487983129442) | > loader_time: 0.0046 (0.004664686545537112)  --> TIME: 2025-02-27 10:58:54 -- STEP: 7790/36938 -- GLOBAL_STEP: 1670000 | > loss_text_ce: 0.03294336795806885 (0.029811022844194256) | > loss_mel_ce: 2.40946102142334 (nan) | > loss: 0.00969208125025034 (nan) | > current_lr: 5e-06 | > step_time: 0.1372 (0.13711199931829104) | > loader_time: 0.0047 (0.004664991878269569) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1670000.pth  --> TIME: 2025-02-27 10:59:10 -- STEP: 7840/36938 -- GLOBAL_STEP: 1670050 | > loss_text_ce: 0.031146086752414703 (0.029807521848183846) | > loss_mel_ce: 2.425671339035034 (nan) | > loss: 0.009749275632202625 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1371425816295103) | > loader_time: 0.0046 (0.004665403432992037)  --> TIME: 2025-02-27 10:59:19 -- STEP: 7890/36938 -- GLOBAL_STEP: 1670100 | > loss_text_ce: 0.02435588836669922 (0.0298098447716102) | > loss_mel_ce: 0.024380365386605263 (nan) | > loss: 0.00019339783466421068 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13714753844136718) | > loader_time: 0.0046 (0.004665869995668357)  --> TIME: 2025-02-27 10:59:29 -- STEP: 7940/36938 -- GLOBAL_STEP: 1670150 | > loss_text_ce: 0.031089317053556442 (0.029813167719423807) | > loss_mel_ce: 2.6625442504882812 (nan) | > loss: 0.010689022950828075 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13717306855343497) | > loader_time: 0.0046 (0.004666380137580465)  --> TIME: 2025-02-27 10:59:38 -- STEP: 7990/36938 -- GLOBAL_STEP: 1670200 | > loss_text_ce: 0.03204914182424545 (0.029818024418814496) | > loss_mel_ce: 2.662597179412842 (nan) | > loss: 0.010693041607737541 (nan) | > current_lr: 5e-06 | > step_time: 0.1863 (0.13718358610986367) | > loader_time: 0.0049 (0.004666430720399703)  --> TIME: 2025-02-27 10:59:47 -- STEP: 8040/36938 -- GLOBAL_STEP: 1670250 | > loss_text_ce: 0.03469987213611603 (0.029821662002113364) | > loss_mel_ce: 2.8393912315368652 (nan) | > loss: 0.011405124329030514 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.13716520035444846) | > loader_time: 0.0047 (0.004666127642588825)  --> TIME: 2025-02-27 10:59:57 -- STEP: 8090/36938 -- GLOBAL_STEP: 1670300 | > loss_text_ce: 0.02794514037668705 (0.029832011670201133) | > loss_mel_ce: 1.321839451789856 (nan) | > loss: 0.005356288515031338 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13720004119448712) | > loader_time: 0.0046 (0.004666489193259709)  --> TIME: 2025-02-27 11:00:06 -- STEP: 8140/36938 -- GLOBAL_STEP: 1670350 | > loss_text_ce: 0.03240140900015831 (0.02983207410139647) | > loss_mel_ce: 2.1626365184783936 (nan) | > loss: 0.0087104681879282 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.1371749411929742) | > loader_time: 0.0046 (0.004666240385182071)  --> TIME: 2025-02-27 11:00:16 -- STEP: 8190/36938 -- GLOBAL_STEP: 1670400 | > loss_text_ce: 0.03153163939714432 (0.02983109358009019) | > loss_mel_ce: 3.2201998233795166 (nan) | > loss: 0.012903696857392788 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13716797767541392) | > loader_time: 0.0046 (0.004666662652850585)  --> TIME: 2025-02-27 11:00:25 -- STEP: 8240/36938 -- GLOBAL_STEP: 1670450 | > loss_text_ce: 0.028818873688578606 (0.02983465574498659) | > loss_mel_ce: 2.8778188228607178 (nan) | > loss: 0.011534277349710464 (nan) | > current_lr: 5e-06 | > step_time: 0.1767 (0.13718108621615852) | > loader_time: 0.0049 (0.004667298915316753)  --> TIME: 2025-02-27 11:00:34 -- STEP: 8290/36938 -- GLOBAL_STEP: 1670500 | > loss_text_ce: 0.03241759538650513 (0.029832758711689163) | > loss_mel_ce: 2.1718106269836426 (nan) | > loss: 0.008746937848627567 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.1371644835132746) | > loader_time: 0.0046 (0.004667791497721742)  --> TIME: 2025-02-27 11:00:44 -- STEP: 8340/36938 -- GLOBAL_STEP: 1670550 | > loss_text_ce: 0.029933081939816475 (0.02983042978038408) | > loss_mel_ce: 2.2089433670043945 (nan) | > loss: 0.0088844308629632 (nan) | > current_lr: 5e-06 | > step_time: 0.1511 (0.13718347366479416) | > loader_time: 0.0046 (0.004667767546445626)  --> TIME: 2025-02-27 11:00:53 -- STEP: 8390/36938 -- GLOBAL_STEP: 1670600 | > loss_text_ce: 0.022328795865178108 (0.029825588600006267) | > loss_mel_ce: 2.713963747024536 (nan) | > loss: 0.010858304798603058 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13718140207683985) | > loader_time: 0.0046 (0.004667266759315458)  --> TIME: 2025-02-27 11:01:02 -- STEP: 8440/36938 -- GLOBAL_STEP: 1670650 | > loss_text_ce: 0.034127455204725266 (0.029827631983830996) | > loss_mel_ce: 1.8548377752304077 (nan) | > loss: 0.007495894096791744 (nan) | > current_lr: 5e-06 | > step_time: 0.112 (0.1371776556516715) | > loader_time: 0.0044 (0.004666899024592749)  --> TIME: 2025-02-27 11:01:12 -- STEP: 8490/36938 -- GLOBAL_STEP: 1670700 | > loss_text_ce: 0.03356080874800682 (0.029832263045271836) | > loss_mel_ce: 2.0801944732666016 (nan) | > loss: 0.008387917652726173 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13718075873012164) | > loader_time: 0.0047 (0.004666729473253294)  --> TIME: 2025-02-27 11:01:21 -- STEP: 8540/36938 -- GLOBAL_STEP: 1670750 | > loss_text_ce: 0.03648986294865608 (0.029832385466335917) | > loss_mel_ce: 3.0424420833587646 (nan) | > loss: 0.012217985466122627 (nan) | > current_lr: 5e-06 | > step_time: 0.1849 (0.13715744772337124) | > loader_time: 0.0048 (0.004666481978441001)  --> TIME: 2025-02-27 11:01:30 -- STEP: 8590/36938 -- GLOBAL_STEP: 1670800 | > loss_text_ce: 0.02725340984761715 (0.029831300132579418) | > loss_mel_ce: 2.1278584003448486 (nan) | > loss: 0.008552031591534615 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.1371481375033702) | > loader_time: 0.0047 (0.004666967136618412)  --> TIME: 2025-02-27 11:01:40 -- STEP: 8640/36938 -- GLOBAL_STEP: 1670850 | > loss_text_ce: 0.028108946979045868 (0.029829700133025955) | > loss_mel_ce: 2.7568857669830322 (nan) | > loss: 0.01105156634002924 (nan) | > current_lr: 5e-06 | > step_time: 0.126 (0.13715983074572397) | > loader_time: 0.0045 (0.004667414586853086)  --> TIME: 2025-02-27 11:01:49 -- STEP: 8690/36938 -- GLOBAL_STEP: 1670900 | > loss_text_ce: 0.03796114772558212 (0.029832948820924833) | > loss_mel_ce: 1.0435724258422852 (nan) | > loss: 0.00429179985076189 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13712674218848372) | > loader_time: 0.0045 (0.004667676551432538)  --> TIME: 2025-02-27 11:01:58 -- STEP: 8740/36938 -- GLOBAL_STEP: 1670950 | > loss_text_ce: 0.016932500526309013 (0.029826406288776947) | > loss_mel_ce: 0.9260212182998657 (nan) | > loss: 0.0037418799474835396 (nan) | > current_lr: 5e-06 | > step_time: 0.1156 (0.1371275470786023) | > loader_time: 0.0045 (0.004667825802512779)  --> TIME: 2025-02-27 11:02:07 -- STEP: 8790/36938 -- GLOBAL_STEP: 1671000 | > loss_text_ce: 0.03267467021942139 (0.029824393346219134) | > loss_mel_ce: 1.8113932609558105 (nan) | > loss: 0.007317730225622654 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13710791500490857) | > loader_time: 0.0046 (0.004668026328493698)  --> TIME: 2025-02-27 11:02:16 -- STEP: 8840/36938 -- GLOBAL_STEP: 1671050 | > loss_text_ce: 0.03529348596930504 (0.0298253047945846) | > loss_mel_ce: 2.533766984939575 (nan) | > loss: 0.010194685310125351 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13709112686269467) | > loader_time: 0.0045 (0.0046686962988581465)  --> TIME: 2025-02-27 11:02:26 -- STEP: 8890/36938 -- GLOBAL_STEP: 1671100 | > loss_text_ce: 0.028835032135248184 (0.02982004833930456) | > loss_mel_ce: 2.4320921897888184 (nan) | > loss: 0.009765584953129292 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13710348024143057) | > loader_time: 0.0045 (0.004668448451399387)  --> TIME: 2025-02-27 11:02:35 -- STEP: 8940/36938 -- GLOBAL_STEP: 1671150 | > loss_text_ce: 0.03255529701709747 (0.029820873045156265) | > loss_mel_ce: 1.8964576721191406 (nan) | > loss: 0.007654814049601555 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.1371234340422377) | > loader_time: 0.0047 (0.004668365522252354)  --> TIME: 2025-02-27 11:02:45 -- STEP: 8990/36938 -- GLOBAL_STEP: 1671200 | > loss_text_ce: 0.02870522439479828 (0.029818089669145585) | > loss_mel_ce: 3.0248029232025146 (nan) | > loss: 0.012117096222937107 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.1371074928192466) | > loader_time: 0.0047 (0.004668467912048057)  --> TIME: 2025-02-27 11:02:54 -- STEP: 9040/36938 -- GLOBAL_STEP: 1671250 | > loss_text_ce: 0.029475992545485497 (0.029823855501475982) | > loss_mel_ce: 2.891512393951416 (nan) | > loss: 0.011591223999857903 (nan) | > current_lr: 5e-06 | > step_time: 0.1357 (0.13711411544179536) | > loader_time: 0.0047 (0.004668535252587974)  --> TIME: 2025-02-27 11:03:04 -- STEP: 9090/36938 -- GLOBAL_STEP: 1671300 | > loss_text_ce: 0.03426007926464081 (0.029822534377040185) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.1371481041441398) | > loader_time: 0.0046 (0.004668608120959181)  --> TIME: 2025-02-27 11:03:13 -- STEP: 9140/36938 -- GLOBAL_STEP: 1671350 | > loss_text_ce: 0.025919362902641296 (0.029816874367117687) | > loss_mel_ce: 3.3165090084075928 (nan) | > loss: 0.013263605535030365 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13713420386909175) | > loader_time: 0.0045 (0.004668638899237629)  --> TIME: 2025-02-27 11:03:22 -- STEP: 9190/36938 -- GLOBAL_STEP: 1671400 | > loss_text_ce: 0.025585772469639778 (0.029815267241064366) | > loss_mel_ce: 2.527888059616089 (nan) | > loss: 0.010132833383977413 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13712081177579735) | > loader_time: 0.0046 (0.004669325681194509)  --> TIME: 2025-02-27 11:03:32 -- STEP: 9240/36938 -- GLOBAL_STEP: 1671450 | > loss_text_ce: 0.03422414883971214 (0.029824976886933047) | > loss_mel_ce: 3.305448055267334 (nan) | > loss: 0.01325266808271408 (nan) | > current_lr: 5e-06 | > step_time: 0.1948 (0.13713972426596185) | > loader_time: 0.0048 (0.0046692978252064025)  --> TIME: 2025-02-27 11:03:41 -- STEP: 9290/36938 -- GLOBAL_STEP: 1671500 | > loss_text_ce: 0.03080400638282299 (0.029824445088392985) | > loss_mel_ce: 2.8988304138183594 (nan) | > loss: 0.011625533923506737 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13714201363090345) | > loader_time: 0.0046 (0.004669249712201822)  --> TIME: 2025-02-27 11:03:51 -- STEP: 9340/36938 -- GLOBAL_STEP: 1671550 | > loss_text_ce: 0.026820948347449303 (0.029821333587987296) | > loss_mel_ce: 0.0056915283203125 (nan) | > loss: 0.00012901777517981827 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13714067216093123) | > loader_time: 0.0046 (0.00466922572644415)  --> TIME: 2025-02-27 11:04:00 -- STEP: 9390/36938 -- GLOBAL_STEP: 1671600 | > loss_text_ce: 0.0381847508251667 (0.02982050509115679) | > loss_mel_ce: 0.2893269956111908 (nan) | > loss: 0.0012996499426662922 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13713280667131364) | > loader_time: 0.0046 (0.004669057650053205)  --> TIME: 2025-02-27 11:04:09 -- STEP: 9440/36938 -- GLOBAL_STEP: 1671650 | > loss_text_ce: 0.023066464811563492 (0.02982966054449947) | > loss_mel_ce: 2.1497409343719482 (nan) | > loss: 0.008622252382338047 (nan) | > current_lr: 5e-06 | > step_time: 0.1812 (0.1371178322424328) | > loader_time: 0.0048 (0.004669145507327576)  --> TIME: 2025-02-27 11:04:18 -- STEP: 9490/36938 -- GLOBAL_STEP: 1671700 | > loss_text_ce: 0.029336681589484215 (0.029834040744942628) | > loss_mel_ce: 2.9093873500823975 (nan) | > loss: 0.011661604046821594 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13708945606983428) | > loader_time: 0.0053 (0.0046690769265901365)  --> TIME: 2025-02-27 11:04:27 -- STEP: 9540/36938 -- GLOBAL_STEP: 1671750 | > loss_text_ce: 0.03151419013738632 (0.02983585027128505) | > loss_mel_ce: 2.851393222808838 (nan) | > loss: 0.011440109461545944 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.1370779186924553) | > loader_time: 0.0046 (0.004668793013510706)  --> TIME: 2025-02-27 11:04:37 -- STEP: 9590/36938 -- GLOBAL_STEP: 1671800 | > loss_text_ce: 0.03236394003033638 (0.029825890278181486) | > loss_mel_ce: 3.186465263366699 (nan) | > loss: 0.01277313195168972 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13705916934267004) | > loader_time: 0.0045 (0.00466838188787943)  --> TIME: 2025-02-27 11:04:46 -- STEP: 9640/36938 -- GLOBAL_STEP: 1671850 | > loss_text_ce: 0.03860614076256752 (0.029826699610687485) | > loss_mel_ce: 2.166388511657715 (nan) | > loss: 0.008749979548156261 (nan) | > current_lr: 5e-06 | > step_time: 0.1834 (0.13706798360555555) | > loader_time: 0.0048 (0.004668404254675893)  --> TIME: 2025-02-27 11:04:55 -- STEP: 9690/36938 -- GLOBAL_STEP: 1671900 | > loss_text_ce: 0.035380639135837555 (0.029830143314001023) | > loss_mel_ce: 2.543738842010498 (nan) | > loss: 0.01023460179567337 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.13704695982091625) | > loader_time: 0.0047 (0.0046681737260060585)  --> TIME: 2025-02-27 11:05:05 -- STEP: 9740/36938 -- GLOBAL_STEP: 1671950 | > loss_text_ce: 0.018382729962468147 (0.02982763492768091) | > loss_mel_ce: 2.188068151473999 (nan) | > loss: 0.00875575840473175 (nan) | > current_lr: 5e-06 | > step_time: 0.1355 (0.1370659235076988) | > loader_time: 0.0046 (0.004667958757960568)  --> TIME: 2025-02-27 11:05:14 -- STEP: 9790/36938 -- GLOBAL_STEP: 1672000 | > loss_text_ce: 0.023470815271139145 (0.02982533522505804) | > loss_mel_ce: 2.8825225830078125 (nan) | > loss: 0.011531720869243145 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13708461685005383) | > loader_time: 0.0046 (0.004667848025936631)  --> TIME: 2025-02-27 11:05:23 -- STEP: 9840/36938 -- GLOBAL_STEP: 1672050 | > loss_text_ce: 0.0278739295899868 (0.029831020811542385) | > loss_mel_ce: 2.6033363342285156 (nan) | > loss: 0.010441311635077 (nan) | > current_lr: 5e-06 | > step_time: 0.1812 (0.13706510146943596) | > loader_time: 0.005 (0.004667756930599364)  --> TIME: 2025-02-27 11:05:33 -- STEP: 9890/36938 -- GLOBAL_STEP: 1672100 | > loss_text_ce: 0.02579575777053833 (0.029831516230808472) | > loss_mel_ce: 3.0384678840637207 (nan) | > loss: 0.012159776873886585 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1370704627977229) | > loader_time: 0.0046 (0.004667646144832225)  --> TIME: 2025-02-27 11:05:42 -- STEP: 9940/36938 -- GLOBAL_STEP: 1672150 | > loss_text_ce: 0.027375277131795883 (0.02983080843347359) | > loss_mel_ce: 2.7661330699920654 (nan) | > loss: 0.01108535099774599 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13706702267380136) | > loader_time: 0.0046 (0.004667343052340225)  --> TIME: 2025-02-27 11:05:51 -- STEP: 9990/36938 -- GLOBAL_STEP: 1672200 | > loss_text_ce: 0.03601459041237831 (0.029837471998955784) | > loss_mel_ce: 2.6378800868988037 (nan) | > loss: 0.01061069406569004 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.1370548664986558) | > loader_time: 0.0048 (0.004668318998586906)  --> TIME: 2025-02-27 11:06:01 -- STEP: 10040/36938 -- GLOBAL_STEP: 1672250 | > loss_text_ce: 0.0336153581738472 (0.0298352040874538) | > loss_mel_ce: 1.6874523162841797 (nan) | > loss: 0.006829633843153715 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.137065725568757) | > loader_time: 0.0046 (0.004668408323569126)  --> TIME: 2025-02-27 11:06:10 -- STEP: 10090/36938 -- GLOBAL_STEP: 1672300 | > loss_text_ce: 0.035780567675828934 (0.029837885531029184) | > loss_mel_ce: 1.834200143814087 (nan) | > loss: 0.007420558948069811 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13706107730317382) | > loader_time: 0.0046 (0.004668266874829412)  --> TIME: 2025-02-27 11:06:19 -- STEP: 10140/36938 -- GLOBAL_STEP: 1672350 | > loss_text_ce: 0.032319575548172 (0.02984655823685317) | > loss_mel_ce: 3.0381994247436523 (nan) | > loss: 0.012184599414467812 (nan) | > current_lr: 5e-06 | > step_time: 0.1893 (0.13706487713480942) | > loader_time: 0.0048 (0.004668213770939749)  --> TIME: 2025-02-27 11:06:29 -- STEP: 10190/36938 -- GLOBAL_STEP: 1672400 | > loss_text_ce: 0.02981630712747574 (0.02985178640960433) | > loss_mel_ce: 3.063495397567749 (nan) | > loss: 0.012275047600269318 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13707147875290832) | > loader_time: 0.0047 (0.004668183462425577)  --> TIME: 2025-02-27 11:06:38 -- STEP: 10240/36938 -- GLOBAL_STEP: 1672450 | > loss_text_ce: 0.020014502108097076 (0.029853328567423888) | > loss_mel_ce: 2.014641761779785 (nan) | > loss: 0.008074033074080944 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13706997008994318) | > loader_time: 0.0045 (0.004668409889563909)  --> TIME: 2025-02-27 11:06:48 -- STEP: 10290/36938 -- GLOBAL_STEP: 1672500 | > loss_text_ce: 0.02738916128873825 (0.029852316203727174) | > loss_mel_ce: 2.278207302093506 (nan) | > loss: 0.009149192832410336 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13707593724609562) | > loader_time: 0.0045 (0.004668147570885291)  --> TIME: 2025-02-27 11:06:57 -- STEP: 10340/36938 -- GLOBAL_STEP: 1672550 | > loss_text_ce: 0.02985697239637375 (0.0298527563243881) | > loss_mel_ce: 2.8406894207000732 (nan) | > loss: 0.011391057632863522 (nan) | > current_lr: 5e-06 | > step_time: 0.1952 (0.1371165011330325) | > loader_time: 0.0048 (0.004668151732800764)  --> TIME: 2025-02-27 11:07:07 -- STEP: 10390/36938 -- GLOBAL_STEP: 1672600 | > loss_text_ce: 0.02948182448744774 (0.029854344911568576) | > loss_mel_ce: 2.8316309452056885 (nan) | > loss: 0.011353623121976852 (nan) | > current_lr: 5e-06 | > step_time: 0.1704 (0.13714566843457004) | > loader_time: 0.0048 (0.004668024162241067)  --> TIME: 2025-02-27 11:07:16 -- STEP: 10440/36938 -- GLOBAL_STEP: 1672650 | > loss_text_ce: 0.0205333661288023 (0.02985286771799741) | > loss_mel_ce: 1.0007530450820923 (nan) | > loss: 0.004052723757922649 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13713895667101805) | > loader_time: 0.0045 (0.00466777741680656)  --> TIME: 2025-02-27 11:07:26 -- STEP: 10490/36938 -- GLOBAL_STEP: 1672700 | > loss_text_ce: 0.02405712381005287 (0.029855694348441777) | > loss_mel_ce: 2.6154274940490723 (nan) | > loss: 0.010474146343767643 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13713998610457553) | > loader_time: 0.0045 (0.004667572070577231)  --> TIME: 2025-02-27 11:07:35 -- STEP: 10540/36938 -- GLOBAL_STEP: 1672750 | > loss_text_ce: 0.020181315019726753 (0.029854339175721528) | > loss_mel_ce: 2.848372459411621 (nan) | > loss: 0.011383150704205036 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13712647911267353) | > loader_time: 0.0045 (0.00466745462996683)  --> TIME: 2025-02-27 11:07:45 -- STEP: 10590/36938 -- GLOBAL_STEP: 1672800 | > loss_text_ce: 0.03807307407259941 (0.029856498462145373) | > loss_mel_ce: 3.1736764907836914 (nan) | > loss: 0.012745038606226444 (nan) | > current_lr: 5e-06 | > step_time: 0.1828 (0.13715420858943436) | > loader_time: 0.0048 (0.004667434273630625)  --> TIME: 2025-02-27 11:07:54 -- STEP: 10640/36938 -- GLOBAL_STEP: 1672850 | > loss_text_ce: 0.028468940407037735 (0.029857021481362416) | > loss_mel_ce: 2.9639205932617188 (nan) | > loss: 0.011874562129378319 (nan) | > current_lr: 5e-06 | > step_time: 0.1702 (0.13717611739061955) | > loader_time: 0.0046 (0.004667232829825319)  --> TIME: 2025-02-27 11:08:04 -- STEP: 10690/36938 -- GLOBAL_STEP: 1672900 | > loss_text_ce: 0.028002269566059113 (0.029854623573833085) | > loss_mel_ce: 0.0230712890625 (nan) | > loss: 0.000202672861632891 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13719263112467078) | > loader_time: 0.0046 (0.004667102075046862)  --> TIME: 2025-02-27 11:08:13 -- STEP: 10740/36938 -- GLOBAL_STEP: 1672950 | > loss_text_ce: 0.027691463008522987 (0.029857016433710214) | > loss_mel_ce: 1.4660345315933228 (nan) | > loss: 0.005927484482526779 (nan) | > current_lr: 5e-06 | > step_time: 0.1822 (0.13719021468189296) | > loader_time: 0.0049 (0.004666949583831445)  --> TIME: 2025-02-27 11:08:23 -- STEP: 10790/36938 -- GLOBAL_STEP: 1673000 | > loss_text_ce: 0.03166399151086807 (0.029856858602220034) | > loss_mel_ce: 2.1641032695770264 (nan) | > loss: 0.00871336180716753 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13718991432066324) | > loader_time: 0.0045 (0.004666966083869544)  --> TIME: 2025-02-27 11:08:32 -- STEP: 10840/36938 -- GLOBAL_STEP: 1673050 | > loss_text_ce: 0.028137532994151115 (0.029858219489881363) | > loss_mel_ce: 2.693171501159668 (nan) | > loss: 0.010798845440149307 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.13720641886176352) | > loader_time: 0.0049 (0.004666840788183178)  --> TIME: 2025-02-27 11:08:41 -- STEP: 10890/36938 -- GLOBAL_STEP: 1673100 | > loss_text_ce: 0.030299725010991096 (0.029859866696273902) | > loss_mel_ce: 2.733342409133911 (nan) | > loss: 0.010966834612190723 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.13719001100958897) | > loader_time: 0.0046 (0.004666508415404323)  --> TIME: 2025-02-27 11:08:51 -- STEP: 10940/36938 -- GLOBAL_STEP: 1673150 | > loss_text_ce: 0.027650194242596626 (0.02985502901338926) | > loss_mel_ce: 2.74189829826355 (nan) | > loss: 0.010990272276103497 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13720619053030053) | > loader_time: 0.0046 (0.004666263290035663)  --> TIME: 2025-02-27 11:09:00 -- STEP: 10990/36938 -- GLOBAL_STEP: 1673200 | > loss_text_ce: 0.03291068971157074 (0.02985299265222307) | > loss_mel_ce: 3.0398612022399902 (nan) | > loss: 0.012193539179861546 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.1372013829424786) | > loader_time: 0.0048 (0.004665996922049556)  --> TIME: 2025-02-27 11:09:10 -- STEP: 11040/36938 -- GLOBAL_STEP: 1673250 | > loss_text_ce: 0.030390411615371704 (0.029850910973239123) | > loss_mel_ce: 3.114720106124878 (nan) | > loss: 0.01248059794306755 (nan) | > current_lr: 5e-06 | > step_time: 0.1848 (0.13723295989675932) | > loader_time: 0.0049 (0.00466592631478241)  --> TIME: 2025-02-27 11:09:19 -- STEP: 11090/36938 -- GLOBAL_STEP: 1673300 | > loss_text_ce: 0.03076058067381382 (0.029853028882751736) | > loss_mel_ce: 2.7959465980529785 (nan) | > loss: 0.011217092163860798 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.13724186702071403) | > loader_time: 0.0046 (0.004665918775890414)  --> TIME: 2025-02-27 11:09:29 -- STEP: 11140/36938 -- GLOBAL_STEP: 1673350 | > loss_text_ce: 0.029270362108945847 (0.02985794377459725) | > loss_mel_ce: 2.648076057434082 (nan) | > loss: 0.01062439102679491 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13726456034761475) | > loader_time: 0.0046 (0.00466579614036602)  --> TIME: 2025-02-27 11:09:39 -- STEP: 11190/36938 -- GLOBAL_STEP: 1673400 | > loss_text_ce: 0.03292280435562134 (0.029859381710146692) | > loss_mel_ce: 2.602896213531494 (nan) | > loss: 0.010459600016474724 (nan) | > current_lr: 5e-06 | > step_time: 0.1343 (0.13727374394308697) | > loader_time: 0.0046 (0.004665665439029598)  --> TIME: 2025-02-27 11:09:48 -- STEP: 11240/36938 -- GLOBAL_STEP: 1673450 | > loss_text_ce: 0.034148089587688446 (0.029859957585253775) | > loss_mel_ce: 2.58964204788208 (nan) | > loss: 0.010411866009235382 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1373054498879523) | > loader_time: 0.0046 (0.004665648428146537)  --> TIME: 2025-02-27 11:09:58 -- STEP: 11290/36938 -- GLOBAL_STEP: 1673500 | > loss_text_ce: 0.02580796182155609 (0.029852296831210014) | > loss_mel_ce: 3.2732994556427 (nan) | > loss: 0.013091696426272392 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.13730703462426816) | > loader_time: 0.0046 (0.004665399484659755)  --> TIME: 2025-02-27 11:10:07 -- STEP: 11340/36938 -- GLOBAL_STEP: 1673550 | > loss_text_ce: 0.04252675920724869 (0.029850067057181972) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13730761885012377) | > loader_time: 0.0045 (0.004665164888640995)  --> TIME: 2025-02-27 11:10:16 -- STEP: 11390/36938 -- GLOBAL_STEP: 1673600 | > loss_text_ce: 0.027934696525335312 (0.02984841540733825) | > loss_mel_ce: 2.958561420440674 (nan) | > loss: 0.01185117568820715 (nan) | > current_lr: 5e-06 | > step_time: 0.1791 (0.13731225217194715) | > loader_time: 0.0047 (0.0046651792065735284)  --> TIME: 2025-02-27 11:10:26 -- STEP: 11440/36938 -- GLOBAL_STEP: 1673650 | > loss_text_ce: 0.03033023327589035 (0.02984962323810775) | > loss_mel_ce: 2.2191271781921387 (nan) | > loss: 0.008926418609917164 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13730211858149288) | > loader_time: 0.0046 (0.004665011917794517)  --> TIME: 2025-02-27 11:10:35 -- STEP: 11490/36938 -- GLOBAL_STEP: 1673700 | > loss_text_ce: 0.027112266048789024 (0.029844900227279068) | > loss_mel_ce: 2.015080213546753 (nan) | > loss: 0.008103938773274422 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.13729072791581878) | > loader_time: 0.0046 (0.004664754494052034)  --> TIME: 2025-02-27 11:10:44 -- STEP: 11540/36938 -- GLOBAL_STEP: 1673750 | > loss_text_ce: 0.03392758220434189 (0.029842719395824124) | > loss_mel_ce: 0.0037201642990112305 (nan) | > loss: 0.00014939582615625113 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13727644838948067) | > loader_time: 0.0045 (0.004664511945053454)  --> TIME: 2025-02-27 11:10:53 -- STEP: 11590/36938 -- GLOBAL_STEP: 1673800 | > loss_text_ce: 0.02950381487607956 (0.02984031744259534) | > loss_mel_ce: 3.0193254947662354 (nan) | > loss: 0.012098529376089573 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1372471076028946) | > loader_time: 0.0046 (0.004664215761798545)  --> TIME: 2025-02-27 11:11:02 -- STEP: 11640/36938 -- GLOBAL_STEP: 1673850 | > loss_text_ce: 0.029922835528850555 (0.029844409954346895) | > loss_mel_ce: 2.3078722953796387 (nan) | > loss: 0.009276964701712132 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13724676145311165) | > loader_time: 0.0046 (0.00466412031363787)  --> TIME: 2025-02-27 11:11:12 -- STEP: 11690/36938 -- GLOBAL_STEP: 1673900 | > loss_text_ce: 0.026929644867777824 (0.02984599682850599) | > loss_mel_ce: 2.150928258895874 (nan) | > loss: 0.008642293512821198 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13725825100499905) | > loader_time: 0.0046 (0.0046640041651534216)  --> TIME: 2025-02-27 11:11:21 -- STEP: 11740/36938 -- GLOBAL_STEP: 1673950 | > loss_text_ce: 0.03292024880647659 (0.029843519427504208) | > loss_mel_ce: 3.059966564178467 (nan) | > loss: 0.012273361906409264 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.1372832047472221) | > loader_time: 0.0046 (0.004664019872220263)  --> TIME: 2025-02-27 11:11:31 -- STEP: 11790/36938 -- GLOBAL_STEP: 1674000 | > loss_text_ce: 0.018419582396745682 (0.02984173919000774) | > loss_mel_ce: 2.3647122383117676 (nan) | > loss: 0.009456872940063477 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13728529399891318) | > loader_time: 0.0045 (0.00466382764374635)  --> TIME: 2025-02-27 11:11:40 -- STEP: 11840/36938 -- GLOBAL_STEP: 1674050 | > loss_text_ce: 0.023059021681547165 (0.029840046297318328) | > loss_mel_ce: 2.214243173599243 (nan) | > loss: 0.008878184482455254 (nan) | > current_lr: 5e-06 | > step_time: 0.1468 (0.13727419748902392) | > loader_time: 0.0047 (0.00466362250012322)  --> TIME: 2025-02-27 11:11:49 -- STEP: 11890/36938 -- GLOBAL_STEP: 1674100 | > loss_text_ce: 0.0392514206469059 (0.029840214410600942) | > loss_mel_ce: 1.894956350326538 (nan) | > loss: 0.007675428409129381 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13726131019720691) | > loader_time: 0.0045 (0.004663360991129078)  --> TIME: 2025-02-27 11:11:59 -- STEP: 11940/36938 -- GLOBAL_STEP: 1674150 | > loss_text_ce: 0.034232258796691895 (0.029842594975427766) | > loss_mel_ce: 2.6293816566467285 (nan) | > loss: 0.010569896548986435 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1372723774494643) | > loader_time: 0.0046 (0.004663368445545004)  --> TIME: 2025-02-27 11:12:08 -- STEP: 11990/36938 -- GLOBAL_STEP: 1674200 | > loss_text_ce: 0.031868722289800644 (0.029839710809222056) | > loss_mel_ce: 2.431140184402466 (nan) | > loss: 0.00977384578436613 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.13726753675510173) | > loader_time: 0.0046 (0.004663109143044791)  --> TIME: 2025-02-27 11:12:17 -- STEP: 12040/36938 -- GLOBAL_STEP: 1674250 | > loss_text_ce: 0.016530899330973625 (0.02984366584290747) | > loss_mel_ce: 2.6144700050354004 (nan) | > loss: 0.010440480895340443 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13727177922115813) | > loader_time: 0.0045 (0.0046628673607329)  --> TIME: 2025-02-27 11:12:27 -- STEP: 12090/36938 -- GLOBAL_STEP: 1674300 | > loss_text_ce: 0.024732166901230812 (0.02983952407826068) | > loss_mel_ce: 3.1980514526367188 (nan) | > loss: 0.012788824737071991 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13725922744662666) | > loader_time: 0.0046 (0.0046625983133308365)  --> TIME: 2025-02-27 11:12:36 -- STEP: 12140/36938 -- GLOBAL_STEP: 1674350 | > loss_text_ce: 0.028862260282039642 (0.029835241115490924) | > loss_mel_ce: 2.4244256019592285 (nan) | > loss: 0.009735269472002983 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.1372718773131712) | > loader_time: 0.0045 (0.004662488477037629)  --> TIME: 2025-02-27 11:12:46 -- STEP: 12190/36938 -- GLOBAL_STEP: 1674400 | > loss_text_ce: 0.02447161078453064 (0.02984177340937502) | > loss_mel_ce: 1.9024674892425537 (nan) | > loss: 0.007646584417670965 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13728485878028318) | > loader_time: 0.0046 (0.004662436339775025)  --> TIME: 2025-02-27 11:12:55 -- STEP: 12240/36938 -- GLOBAL_STEP: 1674450 | > loss_text_ce: 0.03190508484840393 (0.02984433389707085) | > loss_mel_ce: 3.233865976333618 (nan) | > loss: 0.012959410436451435 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.1372729626551176) | > loader_time: 0.0047 (0.004662501422408383)  --> TIME: 2025-02-27 11:13:04 -- STEP: 12290/36938 -- GLOBAL_STEP: 1674500 | > loss_text_ce: 0.027797717601060867 (0.029841948632511595) | > loss_mel_ce: 2.2322869300842285 (nan) | > loss: 0.008968590758740902 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.1372867678897573) | > loader_time: 0.0046 (0.004662638005442136)  --> TIME: 2025-02-27 11:13:13 -- STEP: 12340/36938 -- GLOBAL_STEP: 1674550 | > loss_text_ce: 0.037640828639268875 (0.02983947812679195) | > loss_mel_ce: 1.3919477462768555 (nan) | > loss: 0.005672970786690712 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13724747768677262) | > loader_time: 0.0045 (0.0046626301600056165)  --> TIME: 2025-02-27 11:13:23 -- STEP: 12390/36938 -- GLOBAL_STEP: 1674600 | > loss_text_ce: 0.0308530330657959 (0.02984239427024037) | > loss_mel_ce: 2.798415184020996 (nan) | > loss: 0.011227255687117577 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13724574511622725) | > loader_time: 0.0046 (0.00466254912046199)  --> TIME: 2025-02-27 11:13:32 -- STEP: 12440/36938 -- GLOBAL_STEP: 1674650 | > loss_text_ce: 0.03145333752036095 (0.029845335430890872) | > loss_mel_ce: 1.3323465585708618 (nan) | > loss: 0.005411904770880938 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13726079021236182) | > loader_time: 0.0046 (0.004662473236249591)  --> TIME: 2025-02-27 11:13:42 -- STEP: 12490/36938 -- GLOBAL_STEP: 1674700 | > loss_text_ce: 0.0351373665034771 (0.02984557312565018) | > loss_mel_ce: 2.1576693058013916 (nan) | > loss: 0.00870161410421133 (nan) | > current_lr: 5e-06 | > step_time: 0.1266 (0.1372710520215186) | > loader_time: 0.0045 (0.004662398112307)  --> TIME: 2025-02-27 11:13:51 -- STEP: 12540/36938 -- GLOBAL_STEP: 1674750 | > loss_text_ce: 0.03036813624203205 (0.02984497715745781) | > loss_mel_ce: 2.46051025390625 (nan) | > loss: 0.00988443847745657 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13729443966486832) | > loader_time: 0.0045 (0.0046623412882122114)  --> TIME: 2025-02-27 11:14:00 -- STEP: 12590/36938 -- GLOBAL_STEP: 1674800 | > loss_text_ce: 0.029739297926425934 (0.02984633016155188) | > loss_mel_ce: 2.6488022804260254 (nan) | > loss: 0.010629134252667427 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13728322535871632) | > loader_time: 0.0046 (0.00466210900837701)  --> TIME: 2025-02-27 11:14:10 -- STEP: 12640/36938 -- GLOBAL_STEP: 1674850 | > loss_text_ce: 0.030017074197530746 (0.029842445222864257) | > loss_mel_ce: 2.920273542404175 (nan) | > loss: 0.011707503348588943 (nan) | > current_lr: 5e-06 | > step_time: 0.1858 (0.13727672064228916) | > loader_time: 0.0048 (0.004661888997011565)  --> TIME: 2025-02-27 11:14:19 -- STEP: 12690/36938 -- GLOBAL_STEP: 1674900 | > loss_text_ce: 0.03415150195360184 (0.029844577146152445) | > loss_mel_ce: 2.4398436546325684 (nan) | > loss: 0.009817441925406456 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1372807442437393) | > loader_time: 0.0045 (0.004661790661402271)  --> TIME: 2025-02-27 11:14:28 -- STEP: 12740/36938 -- GLOBAL_STEP: 1674950 | > loss_text_ce: 0.01703287474811077 (0.029845292563952238) | > loss_mel_ce: 1.9261207580566406 (nan) | > loss: 0.007710927631705999 (nan) | > current_lr: 5e-06 | > step_time: 0.1116 (0.1372752284142812) | > loader_time: 0.0044 (0.004661567413264221)  --> TIME: 2025-02-27 11:14:38 -- STEP: 12790/36938 -- GLOBAL_STEP: 1675000 | > loss_text_ce: 0.02635168470442295 (0.02984864331506618) | > loss_mel_ce: 3.084864854812622 (nan) | > loss: 0.012346098199486732 (nan) | > current_lr: 5e-06 | > step_time: 0.1679 (0.13727098844124075) | > loader_time: 0.0045 (0.00466133823051783)  --> TIME: 2025-02-27 11:14:47 -- STEP: 12840/36938 -- GLOBAL_STEP: 1675050 | > loss_text_ce: 0.031592417508363724 (0.029849350096068906) | > loss_mel_ce: 2.9340341091156006 (nan) | > loss: 0.011768359690904617 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.1372703513066719) | > loader_time: 0.0046 (0.004661086303794145)  --> TIME: 2025-02-27 11:14:56 -- STEP: 12890/36938 -- GLOBAL_STEP: 1675100 | > loss_text_ce: 0.028298228979110718 (0.029850014433831184) | > loss_mel_ce: 1.8122488260269165 (nan) | > loss: 0.0073037585243582726 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13725057695150572) | > loader_time: 0.0045 (0.0046609458856753)  --> TIME: 2025-02-27 11:15:06 -- STEP: 12940/36938 -- GLOBAL_STEP: 1675150 | > loss_text_ce: 0.02396424300968647 (0.029847952643325883) | > loss_mel_ce: 2.763934373855591 (nan) | > loss: 0.011063090525567532 (nan) | > current_lr: 5e-06 | > step_time: 0.1811 (0.1372608782887644) | > loader_time: 0.0048 (0.004660742986699731)  --> TIME: 2025-02-27 11:15:15 -- STEP: 12990/36938 -- GLOBAL_STEP: 1675200 | > loss_text_ce: 0.029843930155038834 (0.029845842937318703) | > loss_mel_ce: 1.6311829090118408 (nan) | > loss: 0.006591376848518848 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.1372566507815214) | > loader_time: 0.0046 (0.004660550771629581)  --> TIME: 2025-02-27 11:15:24 -- STEP: 13040/36938 -- GLOBAL_STEP: 1675250 | > loss_text_ce: 0.026160191744565964 (0.029846660772115678) | > loss_mel_ce: 0.2066650390625 (nan) | > loss: 0.0009239097125828266 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13726742053690133) | > loader_time: 0.0046 (0.004660392794872373)  --> TIME: 2025-02-27 11:15:34 -- STEP: 13090/36938 -- GLOBAL_STEP: 1675300 | > loss_text_ce: 0.034571196883916855 (0.029845652883378624) | > loss_mel_ce: 2.175605535507202 (nan) | > loss: 0.008770543150603771 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13725774333711985) | > loader_time: 0.0046 (0.004660170109241055)  --> TIME: 2025-02-27 11:15:43 -- STEP: 13140/36938 -- GLOBAL_STEP: 1675350 | > loss_text_ce: 0.029296403750777245 (0.029846371385825958) | > loss_mel_ce: 2.08052921295166 (nan) | > loss: 0.008372324518859386 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.13725094680916777) | > loader_time: 0.0046 (0.0046599680430268724)  --> TIME: 2025-02-27 11:15:52 -- STEP: 13190/36938 -- GLOBAL_STEP: 1675400 | > loss_text_ce: 0.03096577525138855 (0.029847191339831762) | > loss_mel_ce: 2.8189961910247803 (nan) | > loss: 0.011309373192489147 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13725586054992106) | > loader_time: 0.0046 (0.004660089382897563)  --> TIME: 2025-02-27 11:16:02 -- STEP: 13240/36938 -- GLOBAL_STEP: 1675450 | > loss_text_ce: 0.032366085797548294 (0.0298488004494661) | > loss_mel_ce: 3.096728563308716 (nan) | > loss: 0.012417042627930641 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13724977611775127) | > loader_time: 0.0046 (0.0046601092167131205)  --> TIME: 2025-02-27 11:16:11 -- STEP: 13290/36938 -- GLOBAL_STEP: 1675500 | > loss_text_ce: 0.027550529688596725 (0.02984769246572498) | > loss_mel_ce: 2.1765289306640625 (nan) | > loss: 0.008746347390115261 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13723126928818857) | > loader_time: 0.0046 (0.004660038772250057)  --> TIME: 2025-02-27 11:16:20 -- STEP: 13340/36938 -- GLOBAL_STEP: 1675550 | > loss_text_ce: 0.029494835063815117 (0.02984467928117697) | > loss_mel_ce: 0.9910944700241089 (nan) | > loss: 0.004049958195537329 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13724113608050012) | > loader_time: 0.0045 (0.004660176676788797)  --> TIME: 2025-02-27 11:16:30 -- STEP: 13390/36938 -- GLOBAL_STEP: 1675600 | > loss_text_ce: 0.038193631917238235 (0.02984310895610737) | > loss_mel_ce: 2.696775436401367 (nan) | > loss: 0.010853053070604801 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.13724399104413898) | > loader_time: 0.0046 (0.004660054424790156)  --> TIME: 2025-02-27 11:16:39 -- STEP: 13440/36938 -- GLOBAL_STEP: 1675650 | > loss_text_ce: 0.028021439909934998 (0.029839453356440313) | > loss_mel_ce: 1.9346307516098022 (nan) | > loss: 0.007788302842527628 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13723382862905634) | > loader_time: 0.0046 (0.004659877841671313)  --> TIME: 2025-02-27 11:16:48 -- STEP: 13490/36938 -- GLOBAL_STEP: 1675700 | > loss_text_ce: 0.03516590967774391 (0.029836384508982468) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13723633667554883) | > loader_time: 0.0046 (0.004659936815124695)  --> TIME: 2025-02-27 11:16:57 -- STEP: 13540/36938 -- GLOBAL_STEP: 1675750 | > loss_text_ce: 0.0330812931060791 (0.02983372278203376) | > loss_mel_ce: 3.177891492843628 (nan) | > loss: 0.012741955928504467 (nan) | > current_lr: 5e-06 | > step_time: 0.1441 (0.13722429386445958) | > loader_time: 0.0047 (0.004660018314462998)  --> TIME: 2025-02-27 11:17:07 -- STEP: 13590/36938 -- GLOBAL_STEP: 1675800 | > loss_text_ce: 0.03742418810725212 (0.02983211049504408) | > loss_mel_ce: 2.620701313018799 (nan) | > loss: 0.010548117570579052 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13722316024056025) | > loader_time: 0.0046 (0.0046599518647520224)  --> TIME: 2025-02-27 11:17:16 -- STEP: 13640/36938 -- GLOBAL_STEP: 1675850 | > loss_text_ce: 0.023740844801068306 (0.029827892723496008) | > loss_mel_ce: 2.510546922683716 (nan) | > loss: 0.010056697763502598 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13722011238948284) | > loader_time: 0.0045 (0.004660015738954292)  --> TIME: 2025-02-27 11:17:25 -- STEP: 13690/36938 -- GLOBAL_STEP: 1675900 | > loss_text_ce: 0.029923399910330772 (0.029824892577080504) | > loss_mel_ce: 3.280012369155884 (nan) | > loss: 0.013134666718542576 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.1372183722592857) | > loader_time: 0.0046 (0.004659930557295559)  --> TIME: 2025-02-27 11:17:35 -- STEP: 13740/36938 -- GLOBAL_STEP: 1675950 | > loss_text_ce: 0.021049100905656815 (0.029825978114349658) | > loss_mel_ce: 2.0134339332580566 (nan) | > loss: 0.008073345758020878 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13722131181144945) | > loader_time: 0.0046 (0.0046598596343827535)  --> TIME: 2025-02-27 11:17:44 -- STEP: 13790/36938 -- GLOBAL_STEP: 1676000 | > loss_text_ce: 0.02890801802277565 (0.02982154604536498) | > loss_mel_ce: 2.1507678031921387 (nan) | > loss: 0.008649507537484169 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13721461385985526) | > loader_time: 0.0045 (0.004659691973818306)  --> TIME: 2025-02-27 11:17:53 -- STEP: 13840/36938 -- GLOBAL_STEP: 1676050 | > loss_text_ce: 0.031732894480228424 (0.02982715394339831) | > loss_mel_ce: 3.2043559551239014 (nan) | > loss: 0.012841622345149517 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.13720567946489107) | > loader_time: 0.0046 (0.004659487401818952)  --> TIME: 2025-02-27 11:18:02 -- STEP: 13890/36938 -- GLOBAL_STEP: 1676100 | > loss_text_ce: 0.010444583371281624 (0.029819930919705535) | > loss_mel_ce: 1.293838620185852 (nan) | > loss: 0.005175727419555187 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13719246124860748) | > loader_time: 0.0045 (0.004659302428611495)  --> TIME: 2025-02-27 11:18:12 -- STEP: 13940/36938 -- GLOBAL_STEP: 1676150 | > loss_text_ce: 0.03633388131856918 (0.029822430960898164) | > loss_mel_ce: 2.7549057006835938 (nan) | > loss: 0.011076347902417183 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13719079662087616) | > loader_time: 0.0047 (0.004659160479924598)  --> TIME: 2025-02-27 11:18:21 -- STEP: 13990/36938 -- GLOBAL_STEP: 1676200 | > loss_text_ce: 0.014447757974267006 (0.029820505957291583) | > loss_mel_ce: 1.0666877031326294 (nan) | > loss: 0.0042902203276753426 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.1371981162697689) | > loader_time: 0.0045 (0.004659238809172469)  --> TIME: 2025-02-27 11:18:31 -- STEP: 14040/36938 -- GLOBAL_STEP: 1676250 | > loss_text_ce: 0.03349163383245468 (0.02982189070096404) | > loss_mel_ce: 1.9703782796859741 (nan) | > loss: 0.007951865904033184 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13720308245077387) | > loader_time: 0.0047 (0.004659198084448136)  --> TIME: 2025-02-27 11:18:40 -- STEP: 14090/36938 -- GLOBAL_STEP: 1676300 | > loss_text_ce: 0.03244007006287575 (0.029824447603678136) | > loss_mel_ce: 2.9841737747192383 (nan) | > loss: 0.01197069138288498 (nan) | > current_lr: 5e-06 | > step_time: 0.1651 (0.1371833181279695) | > loader_time: 0.0047 (0.004658940466182969)  --> TIME: 2025-02-27 11:18:49 -- STEP: 14140/36938 -- GLOBAL_STEP: 1676350 | > loss_text_ce: 0.018551375716924667 (0.029817740350389983) | > loss_mel_ce: 2.374772787094116 (nan) | > loss: 0.009497318416833878 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13716680981650853) | > loader_time: 0.0045 (0.004658716773717033)  --> TIME: 2025-02-27 11:18:58 -- STEP: 14190/36938 -- GLOBAL_STEP: 1676400 | > loss_text_ce: 0.030733807012438774 (0.029820085050626128) | > loss_mel_ce: 2.333902597427368 (nan) | > loss: 0.009383478201925755 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13717327554768582) | > loader_time: 0.0046 (0.0046587287077524)  --> TIME: 2025-02-27 11:19:07 -- STEP: 14240/36938 -- GLOBAL_STEP: 1676450 | > loss_text_ce: 0.02591276541352272 (0.02981901413825173) | > loss_mel_ce: 3.3700835704803467 (nan) | > loss: 0.013476177118718624 (nan) | > current_lr: 5e-06 | > step_time: 0.1572 (0.13716526560569042) | > loader_time: 0.0055 (0.004658699320273451)  --> TIME: 2025-02-27 11:19:17 -- STEP: 14290/36938 -- GLOBAL_STEP: 1676500 | > loss_text_ce: 0.03561314567923546 (0.029819053063003583) | > loss_mel_ce: 2.6187520027160645 (nan) | > loss: 0.010533194988965988 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.1371652230422254) | > loader_time: 0.0046 (0.00465856784703099)  --> TIME: 2025-02-27 11:19:26 -- STEP: 14340/36938 -- GLOBAL_STEP: 1676550 | > loss_text_ce: 0.03655492141842842 (0.029820765390786937) | > loss_mel_ce: 1.3820195198059082 (nan) | > loss: 0.005629263818264008 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1371393630860402) | > loader_time: 0.0046 (0.004658339179875626)  --> TIME: 2025-02-27 11:19:35 -- STEP: 14390/36938 -- GLOBAL_STEP: 1676600 | > loss_text_ce: 0.023860717192292213 (0.029819927377337647) | > loss_mel_ce: 2.870786666870117 (nan) | > loss: 0.01148669607937336 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13714771501051998) | > loader_time: 0.0046 (0.004658210584071875)  --> TIME: 2025-02-27 11:19:44 -- STEP: 14440/36938 -- GLOBAL_STEP: 1676650 | > loss_text_ce: 0.03458419069647789 (0.029820322474642783) | > loss_mel_ce: 2.6353437900543213 (nan) | > loss: 0.01059495285153389 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13712790245164427) | > loader_time: 0.0045 (0.00465802228351709)  --> TIME: 2025-02-27 11:19:53 -- STEP: 14490/36938 -- GLOBAL_STEP: 1676700 | > loss_text_ce: 0.028668159618973732 (0.029818846362149966) | > loss_mel_ce: 2.4696156978607178 (nan) | > loss: 0.009913825429975986 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.1371185589856814) | > loader_time: 0.0046 (0.004657894780176765)  --> TIME: 2025-02-27 11:20:03 -- STEP: 14540/36938 -- GLOBAL_STEP: 1676750 | > loss_text_ce: 0.03341878205537796 (0.029819481943937676) | > loss_mel_ce: 0.018798828125 (nan) | > loss: 0.00020721275359392166 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13711573646353842) | > loader_time: 0.0046 (0.004657821806129758)  --> TIME: 2025-02-27 11:20:12 -- STEP: 14590/36938 -- GLOBAL_STEP: 1676800 | > loss_text_ce: 0.032188914716243744 (0.029822229698869752) | > loss_mel_ce: 2.7761263847351074 (nan) | > loss: 0.011144109070301056 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13709771803096957) | > loader_time: 0.0047 (0.004657738547034258)  --> TIME: 2025-02-27 11:20:21 -- STEP: 14640/36938 -- GLOBAL_STEP: 1676850 | > loss_text_ce: 0.02353510633111 (0.029823407022564932) | > loss_mel_ce: 2.9838056564331055 (nan) | > loss: 0.011933892033994198 (nan) | > current_lr: 5e-06 | > step_time: 0.1701 (0.137108001292077) | > loader_time: 0.0048 (0.004658031365910513)  --> TIME: 2025-02-27 11:20:31 -- STEP: 14690/36938 -- GLOBAL_STEP: 1676900 | > loss_text_ce: 0.043818358331918716 (0.02982270543674645) | > loss_mel_ce: 2.1073856353759766 (nan) | > loss: 0.008536525070667267 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13710710059387288) | > loader_time: 0.0045 (0.0046582768773286)  --> TIME: 2025-02-27 11:20:40 -- STEP: 14740/36938 -- GLOBAL_STEP: 1676950 | > loss_text_ce: 0.024964557960629463 (0.029825117388927205) | > loss_mel_ce: 3.623744487762451 (nan) | > loss: 0.01447900477796793 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13709473084238802) | > loader_time: 0.0046 (0.004658330635267486)  --> TIME: 2025-02-27 11:20:49 -- STEP: 14790/36938 -- GLOBAL_STEP: 1677000 | > loss_text_ce: 0.030297523364424706 (0.02982811271354677) | > loss_mel_ce: 2.7588095664978027 (nan) | > loss: 0.011067885905504227 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13709287059879316) | > loader_time: 0.0047 (0.004658376807856993)  --> TIME: 2025-02-27 11:20:58 -- STEP: 14840/36938 -- GLOBAL_STEP: 1677050 | > loss_text_ce: 0.035155441612005234 (0.0298248021346906) | > loss_mel_ce: 2.2100882530212402 (nan) | > loss: 0.008909697644412518 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13705965664508804) | > loader_time: 0.0045 (0.004658232336095721)  --> TIME: 2025-02-27 11:21:07 -- STEP: 14890/36938 -- GLOBAL_STEP: 1677100 | > loss_text_ce: 0.030937829986214638 (0.02982380316380719) | > loss_mel_ce: 2.563274383544922 (nan) | > loss: 0.010294494219124317 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13706621960395884) | > loader_time: 0.0046 (0.0046582669520073864)  --> TIME: 2025-02-27 11:21:16 -- STEP: 14940/36938 -- GLOBAL_STEP: 1677150 | > loss_text_ce: 0.015871481969952583 (0.029821498501594546) | > loss_mel_ce: 1.267429232597351 (nan) | > loss: 0.005092463456094265 (nan) | > current_lr: 5e-06 | > step_time: 0.1085 (0.13704564099011762) | > loader_time: 0.0045 (0.004658265605349455)  --> TIME: 2025-02-27 11:21:25 -- STEP: 14990/36938 -- GLOBAL_STEP: 1677200 | > loss_text_ce: 0.030332030728459358 (0.02982287281419612) | > loss_mel_ce: 2.553690195083618 (nan) | > loss: 0.010254057124257088 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.1370213534054233) | > loader_time: 0.0046 (0.004658137471617022)  --> TIME: 2025-02-27 11:21:34 -- STEP: 15040/36938 -- GLOBAL_STEP: 1677250 | > loss_text_ce: 0.03481181710958481 (0.029820043379267518) | > loss_mel_ce: 3.319643259048462 (nan) | > loss: 0.01331133022904396 (nan) | > current_lr: 5e-06 | > step_time: 0.1777 (0.13701328199277485) | > loader_time: 0.0048 (0.004658079131486566)  --> TIME: 2025-02-27 11:21:44 -- STEP: 15090/36938 -- GLOBAL_STEP: 1677300 | > loss_text_ce: 0.02956709824502468 (0.02982006287029429) | > loss_mel_ce: 2.353583335876465 (nan) | > loss: 0.009456946514546871 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.1370113646454803) | > loader_time: 0.0046 (0.004658219417726225)  --> TIME: 2025-02-27 11:21:53 -- STEP: 15140/36938 -- GLOBAL_STEP: 1677350 | > loss_text_ce: 0.02558746561408043 (0.029827327177339436) | > loss_mel_ce: 2.9397006034851074 (nan) | > loss: 0.011767017655074596 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.13700808760669414) | > loader_time: 0.0046 (0.004658242512883093)  --> TIME: 2025-02-27 11:22:03 -- STEP: 15190/36938 -- GLOBAL_STEP: 1677400 | > loss_text_ce: 0.026342537254095078 (0.029830965555166927) | > loss_mel_ce: 2.8787286281585693 (nan) | > loss: 0.011528060771524906 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13701119355420785) | > loader_time: 0.0047 (0.004658102031755477)  --> TIME: 2025-02-27 11:22:12 -- STEP: 15240/36938 -- GLOBAL_STEP: 1677450 | > loss_text_ce: 0.02955981157720089 (0.029827948890716875) | > loss_mel_ce: 2.6120011806488037 (nan) | > loss: 0.010482385754585266 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.13701171598096484) | > loader_time: 0.0047 (0.004657984327456455)  --> TIME: 2025-02-27 11:22:21 -- STEP: 15290/36938 -- GLOBAL_STEP: 1677500 | > loss_text_ce: 0.03466687351465225 (0.029826797009856874) | > loss_mel_ce: 0.002009749412536621 (nan) | > loss: 0.00014554215886164457 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13700192111859172) | > loader_time: 0.0045 (0.004657779634973438)  --> TIME: 2025-02-27 11:22:30 -- STEP: 15340/36938 -- GLOBAL_STEP: 1677550 | > loss_text_ce: 0.030834496021270752 (0.029824109303047384) | > loss_mel_ce: 3.0263547897338867 (nan) | > loss: 0.01213170401751995 (nan) | > current_lr: 5e-06 | > step_time: 0.1895 (0.1370004018873088) | > loader_time: 0.0049 (0.004657651423786266)  --> TIME: 2025-02-27 11:22:40 -- STEP: 15390/36938 -- GLOBAL_STEP: 1677600 | > loss_text_ce: 0.024851061403751373 (0.029823600847269222) | > loss_mel_ce: 2.1588587760925293 (nan) | > loss: 0.008665516041219234 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.1370079929445992) | > loader_time: 0.0046 (0.004657526911660553)  --> TIME: 2025-02-27 11:22:50 -- STEP: 15440/36938 -- GLOBAL_STEP: 1677650 | > loss_text_ce: 0.03306642174720764 (0.02982170639301462) | > loss_mel_ce: 2.6366491317749023 (nan) | > loss: 0.010594110004603863 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13702204960924308) | > loader_time: 0.0046 (0.004657423419038247)  --> TIME: 2025-02-27 11:22:59 -- STEP: 15490/36938 -- GLOBAL_STEP: 1677700 | > loss_text_ce: 0.015245186164975166 (0.02982750823041887) | > loss_mel_ce: 1.6015803813934326 (nan) | > loss: 0.00641597481444478 (nan) | > current_lr: 5e-06 | > step_time: 0.1119 (0.13701849040714165) | > loader_time: 0.0045 (0.0046572877177120855)  --> TIME: 2025-02-27 11:23:08 -- STEP: 15540/36938 -- GLOBAL_STEP: 1677750 | > loss_text_ce: 0.02899402566254139 (0.02982786724312732) | > loss_mel_ce: 2.9938056468963623 (nan) | > loss: 0.011995237320661545 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13703194281309222) | > loader_time: 0.0046 (0.00465729540379351)  --> TIME: 2025-02-27 11:23:18 -- STEP: 15590/36938 -- GLOBAL_STEP: 1677800 | > loss_text_ce: 0.029283739626407623 (0.029824825179732672) | > loss_mel_ce: 0.9111759066581726 (nan) | > loss: 0.0037319830153137445 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13703420980378814) | > loader_time: 0.0046 (0.004657205639166888)  --> TIME: 2025-02-27 11:23:27 -- STEP: 15640/36938 -- GLOBAL_STEP: 1677850 | > loss_text_ce: 0.02131561189889908 (0.029827871004743572) | > loss_mel_ce: 2.4444427490234375 (nan) | > loss: 0.009784756228327751 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.1370489460428043) | > loader_time: 0.0045 (0.004657156860736933)  --> TIME: 2025-02-27 11:23:37 -- STEP: 15690/36938 -- GLOBAL_STEP: 1677900 | > loss_text_ce: 0.02905869483947754 (0.029825432662222325) | > loss_mel_ce: 0.00124359130859375 (nan) | > loss: 0.00012024717580061406 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13705708857476095) | > loader_time: 0.0046 (0.0046570565306965584)  --> TIME: 2025-02-27 11:23:47 -- STEP: 15740/36938 -- GLOBAL_STEP: 1677950 | > loss_text_ce: 0.028742969036102295 (0.029824861261243382) | > loss_mel_ce: 0.47585517168045044 (nan) | > loss: 0.002002373803406954 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13707907360421004) | > loader_time: 0.0045 (0.004657370859361814)  --> TIME: 2025-02-27 11:23:56 -- STEP: 15790/36938 -- GLOBAL_STEP: 1678000 | > loss_text_ce: 0.027754737064242363 (0.029824604821450974) | > loss_mel_ce: 2.725785732269287 (nan) | > loss: 0.010926748625934124 (nan) | > current_lr: 5e-06 | > step_time: 0.1649 (0.13706755482599312) | > loader_time: 0.0049 (0.004657369538754732)  --> TIME: 2025-02-27 11:24:05 -- STEP: 15840/36938 -- GLOBAL_STEP: 1678050 | > loss_text_ce: 0.03336675465106964 (0.029827505403793312) | > loss_mel_ce: 2.5069098472595215 (nan) | > loss: 0.010080463252961636 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.1370660321881068) | > loader_time: 0.0047 (0.004657388139854776)  --> TIME: 2025-02-27 11:24:15 -- STEP: 15890/36938 -- GLOBAL_STEP: 1678100 | > loss_text_ce: 0.03453132137656212 (0.029827869933947235) | > loss_mel_ce: 2.71457839012146 (nan) | > loss: 0.010909166187047958 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.1370770859523262) | > loader_time: 0.0046 (0.004657447375615036)  --> TIME: 2025-02-27 11:24:24 -- STEP: 15940/36938 -- GLOBAL_STEP: 1678150 | > loss_text_ce: 0.022619986906647682 (0.029829726485568222) | > loss_mel_ce: 2.4714818000793457 (nan) | > loss: 0.009897230193018913 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13706024050562968) | > loader_time: 0.0045 (0.004657567699102115)  --> TIME: 2025-02-27 11:24:33 -- STEP: 15990/36938 -- GLOBAL_STEP: 1678200 | > loss_text_ce: 0.03215884044766426 (0.029827129166093355) | > loss_mel_ce: 3.2148056030273438 (nan) | > loss: 0.012884780764579773 (nan) | > current_lr: 5e-06 | > step_time: 0.151 (0.1370564956527858) | > loader_time: 0.0047 (0.004657553403805462)  --> TIME: 2025-02-27 11:24:43 -- STEP: 16040/36938 -- GLOBAL_STEP: 1678250 | > loss_text_ce: 0.025244351476430893 (0.02983116155758528) | > loss_mel_ce: 2.347722053527832 (nan) | > loss: 0.009416533634066582 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13706613348011915) | > loader_time: 0.0046 (0.0046575946849480515)  --> TIME: 2025-02-27 11:24:52 -- STEP: 16090/36938 -- GLOBAL_STEP: 1678300 | > loss_text_ce: 0.03165585547685623 (0.0298272221506619) | > loss_mel_ce: 2.9826724529266357 (nan) | > loss: 0.011961620301008224 (nan) | > current_lr: 5e-06 | > step_time: 0.1846 (0.13706828929979317) | > loader_time: 0.0048 (0.004657458992312603)  --> TIME: 2025-02-27 11:25:02 -- STEP: 16140/36938 -- GLOBAL_STEP: 1678350 | > loss_text_ce: 0.039212509989738464 (0.029828247798911425) | > loss_mel_ce: 1.4783817529678345 (nan) | > loss: 0.006022199522703886 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13707537751510934) | > loader_time: 0.0046 (0.004657290386592291)  --> TIME: 2025-02-27 11:25:11 -- STEP: 16190/36938 -- GLOBAL_STEP: 1678400 | > loss_text_ce: 0.03501921892166138 (0.029828326047212347) | > loss_mel_ce: 2.252359390258789 (nan) | > loss: 0.009076899848878384 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13706172799681157) | > loader_time: 0.0045 (0.004657129331309533)  --> TIME: 2025-02-27 11:25:20 -- STEP: 16240/36938 -- GLOBAL_STEP: 1678450 | > loss_text_ce: 0.02739282138645649 (0.029827981989881198) | > loss_mel_ce: 2.6423137187957764 (nan) | > loss: 0.010594074614346027 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.13707309848275634) | > loader_time: 0.0045 (0.004657083573599752)  --> TIME: 2025-02-27 11:25:30 -- STEP: 16290/36938 -- GLOBAL_STEP: 1678500 | > loss_text_ce: 0.031041208654642105 (0.029830512355617435) | > loss_mel_ce: 3.0807337760925293 (nan) | > loss: 0.012348313815891743 (nan) | > current_lr: 5e-06 | > step_time: 0.1496 (0.13708771895308478) | > loader_time: 0.0047 (0.00465703597458101)  --> TIME: 2025-02-27 11:25:39 -- STEP: 16340/36938 -- GLOBAL_STEP: 1678550 | > loss_text_ce: 0.02339034527540207 (0.0298324426681697) | > loss_mel_ce: 2.646244764328003 (nan) | > loss: 0.01059379056096077 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.1370914116921298) | > loader_time: 0.0046 (0.0046569852379573584)  --> TIME: 2025-02-27 11:25:49 -- STEP: 16390/36938 -- GLOBAL_STEP: 1678600 | > loss_text_ce: 0.024514075368642807 (0.029833120871403454) | > loss_mel_ce: 2.562807083129883 (nan) | > loss: 0.010267147794365883 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13708330259736198) | > loader_time: 0.0048 (0.004656840127150119)  --> TIME: 2025-02-27 11:25:57 -- STEP: 16440/36938 -- GLOBAL_STEP: 1678650 | > loss_text_ce: 0.033251214772462845 (0.029829745207473616) | > loss_mel_ce: 2.714470624923706 (nan) | > loss: 0.010903659276664257 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13706304707956385) | > loader_time: 0.0046 (0.004656743307183257)  --> TIME: 2025-02-27 11:26:07 -- STEP: 16490/36938 -- GLOBAL_STEP: 1678700 | > loss_text_ce: 0.02838030457496643 (0.029829515497160092) | > loss_mel_ce: 1.3011229038238525 (nan) | > loss: 0.005275806412100792 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.137062760018232) | > loader_time: 0.0052 (0.004656788245918823)  --> TIME: 2025-02-27 11:26:16 -- STEP: 16540/36938 -- GLOBAL_STEP: 1678750 | > loss_text_ce: 0.03230326995253563 (0.029834739812238744) | > loss_mel_ce: 2.257451295852661 (nan) | > loss: 0.00908632855862379 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.1370589526129168) | > loader_time: 0.0045 (0.00465676323246062)  --> TIME: 2025-02-27 11:26:25 -- STEP: 16590/36938 -- GLOBAL_STEP: 1678800 | > loss_text_ce: 0.03079742193222046 (0.029837138520417857) | > loss_mel_ce: 3.126124143600464 (nan) | > loss: 0.01252746768295765 (nan) | > current_lr: 5e-06 | > step_time: 0.1651 (0.137052583076495) | > loader_time: 0.0046 (0.004656512684100646)  --> TIME: 2025-02-27 11:26:34 -- STEP: 16640/36938 -- GLOBAL_STEP: 1678850 | > loss_text_ce: 0.0290061105042696 (0.029839441857341) | > loss_mel_ce: 2.3100311756134033 (nan) | > loss: 0.009281894192099571 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.1370432396490982) | > loader_time: 0.0045 (0.004656305422003452)  --> TIME: 2025-02-27 11:26:44 -- STEP: 16690/36938 -- GLOBAL_STEP: 1678900 | > loss_text_ce: 0.028288347646594048 (0.029838381776638636) | > loss_mel_ce: 2.743051767349243 (nan) | > loss: 0.010997381992638111 (nan) | > current_lr: 5e-06 | > step_time: 0.1726 (0.13703873027506625) | > loader_time: 0.0054 (0.004656364404919477)  --> TIME: 2025-02-27 11:26:53 -- STEP: 16740/36938 -- GLOBAL_STEP: 1678950 | > loss_text_ce: 0.03191712871193886 (0.029836585182193648) | > loss_mel_ce: 1.8996212482452393 (nan) | > loss: 0.007664835080504417 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13702531621447558) | > loader_time: 0.0045 (0.004656484634075825)  --> TIME: 2025-02-27 11:27:02 -- STEP: 16790/36938 -- GLOBAL_STEP: 1679000 | > loss_text_ce: 0.031861960887908936 (0.029839810279174143) | > loss_mel_ce: 1.29725980758667 (nan) | > loss: 0.005274293478578329 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13701989332645506) | > loader_time: 0.0045 (0.00465664241624915)  --> TIME: 2025-02-27 11:27:11 -- STEP: 16840/36938 -- GLOBAL_STEP: 1679050 | > loss_text_ce: 0.033625923097133636 (0.02984341915050907) | > loss_mel_ce: 2.851163625717163 (nan) | > loss: 0.011447577737271786 (nan) | > current_lr: 5e-06 | > step_time: 0.1514 (0.1370017151934513) | > loader_time: 0.0046 (0.004656368508191683)  --> TIME: 2025-02-27 11:27:20 -- STEP: 16890/36938 -- GLOBAL_STEP: 1679100 | > loss_text_ce: 0.03157317265868187 (0.029842141222775592) | > loss_mel_ce: 2.3275387287139893 (nan) | > loss: 0.009361554868519306 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.13700247320499231) | > loader_time: 0.0047 (0.004656245808688895)  --> TIME: 2025-02-27 11:27:30 -- STEP: 16940/36938 -- GLOBAL_STEP: 1679150 | > loss_text_ce: 0.02998775616288185 (0.029838596577200845) | > loss_mel_ce: 3.0677695274353027 (nan) | > loss: 0.012292688712477684 (nan) | > current_lr: 5e-06 | > step_time: 0.1844 (0.13701115677742312) | > loader_time: 0.0048 (0.004656135810730445)  --> TIME: 2025-02-27 11:27:39 -- STEP: 16990/36938 -- GLOBAL_STEP: 1679200 | > loss_text_ce: 0.02906665951013565 (0.029838345902344295) | > loss_mel_ce: 2.175448179244995 (nan) | > loss: 0.008748074993491173 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.1370233301838539) | > loader_time: 0.0045 (0.004656314330637065)  --> TIME: 2025-02-27 11:27:48 -- STEP: 17040/36938 -- GLOBAL_STEP: 1679250 | > loss_text_ce: 0.021404024213552475 (0.02983789579973431) | > loss_mel_ce: 0.0393880195915699 (nan) | > loss: 0.00024123828916344792 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13700995699900373) | > loader_time: 0.0045 (0.004656219944147993)  --> TIME: 2025-02-27 11:27:58 -- STEP: 17090/36938 -- GLOBAL_STEP: 1679300 | > loss_text_ce: 0.024475745856761932 (0.029836739868583966) | > loss_mel_ce: 0.0521596260368824 (nan) | > loss: 0.0003041086602024734 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13702731442214272) | > loader_time: 0.0045 (0.004656143687913955)  --> TIME: 2025-02-27 11:28:07 -- STEP: 17140/36938 -- GLOBAL_STEP: 1679350 | > loss_text_ce: 0.029870284721255302 (0.029835217072310048) | > loss_mel_ce: 2.526898145675659 (nan) | > loss: 0.010145907290279865 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.1370202212378187) | > loader_time: 0.0048 (0.004655918705004372)  --> TIME: 2025-02-27 11:28:16 -- STEP: 17190/36938 -- GLOBAL_STEP: 1679400 | > loss_text_ce: 0.027557382360100746 (0.029833064771865783) | > loss_mel_ce: 0.00421142578125 (nan) | > loss: 0.0001260667049791664 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13701748054619742) | > loader_time: 0.0045 (0.0046557609808867955)  --> TIME: 2025-02-27 11:28:26 -- STEP: 17240/36938 -- GLOBAL_STEP: 1679450 | > loss_text_ce: 0.041034989058971405 (0.0298355814273169) | > loss_mel_ce: 2.7967896461486816 (nan) | > loss: 0.011261208914220333 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13701396970350566) | > loader_time: 0.0045 (0.0046556289124101554)  --> TIME: 2025-02-27 11:28:35 -- STEP: 17290/36938 -- GLOBAL_STEP: 1679500 | > loss_text_ce: 0.02648722566664219 (0.02983255577868875) | > loss_mel_ce: 2.7014994621276855 (nan) | > loss: 0.010825344361364841 (nan) | > current_lr: 5e-06 | > step_time: 0.144 (0.13700151044967987) | > loader_time: 0.0046 (0.004655673725884932)  --> TIME: 2025-02-27 11:28:44 -- STEP: 17340/36938 -- GLOBAL_STEP: 1679550 | > loss_text_ce: 0.015044402331113815 (0.029826184052591593) | > loss_mel_ce: 0.02190144918859005 (nan) | > loss: 0.00014661051682196558 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.13699862892377432) | > loader_time: 0.0045 (0.004655600859082141)  --> TIME: 2025-02-27 11:28:53 -- STEP: 17390/36938 -- GLOBAL_STEP: 1679600 | > loss_text_ce: 0.03044934943318367 (0.0298265116128593) | > loss_mel_ce: 2.1836159229278564 (nan) | > loss: 0.008785974234342575 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.1370028170722181) | > loader_time: 0.0046 (0.004655689642030381)  --> TIME: 2025-02-27 11:29:03 -- STEP: 17440/36938 -- GLOBAL_STEP: 1679650 | > loss_text_ce: 0.02294093556702137 (0.029826564588498583) | > loss_mel_ce: 2.041747808456421 (nan) | > loss: 0.008193209767341614 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.13700198934439098) | > loader_time: 0.0045 (0.004655751599631173)  --> TIME: 2025-02-27 11:29:12 -- STEP: 17490/36938 -- GLOBAL_STEP: 1679700 | > loss_text_ce: 0.02069535292685032 (0.029825730290180843) | > loss_mel_ce: 3.0305850505828857 (nan) | > loss: 0.012108257040381432 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.13699921581525654) | > loader_time: 0.0047 (0.004655944748971714)  --> TIME: 2025-02-27 11:29:21 -- STEP: 17540/36938 -- GLOBAL_STEP: 1679750 | > loss_text_ce: 0.028512820601463318 (0.029822809027429335) | > loss_mel_ce: 2.514315605163574 (nan) | > loss: 0.01009058952331543 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13699125568026665) | > loader_time: 0.0045 (0.004655997715083328)  --> TIME: 2025-02-27 11:29:30 -- STEP: 17590/36938 -- GLOBAL_STEP: 1679800 | > loss_text_ce: 0.03392806649208069 (0.029824360315076445) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13698602647114483) | > loader_time: 0.0045 (0.004655817749149214)  --> TIME: 2025-02-27 11:29:40 -- STEP: 17640/36938 -- GLOBAL_STEP: 1679850 | > loss_text_ce: 0.035337768495082855 (0.029823238755810746) | > loss_mel_ce: 1.9569735527038574 (nan) | > loss: 0.007905998267233372 (nan) | > current_lr: 5e-06 | > step_time: 0.1081 (0.1369834197351448) | > loader_time: 0.0045 (0.004655606433107191)  --> TIME: 2025-02-27 11:29:49 -- STEP: 17690/36938 -- GLOBAL_STEP: 1679900 | > loss_text_ce: 0.022557100281119347 (0.029825486267620476) | > loss_mel_ce: 2.3705008029937744 (nan) | > loss: 0.009496261365711689 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.13697308995347662) | > loader_time: 0.0045 (0.004655765220228053)  --> TIME: 2025-02-27 11:29:58 -- STEP: 17740/36938 -- GLOBAL_STEP: 1679950 | > loss_text_ce: 0.03015718422830105 (0.029822515217186284) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13697662846749073) | > loader_time: 0.0046 (0.004655782775921908)  --> TIME: 2025-02-27 11:30:08 -- STEP: 17790/36938 -- GLOBAL_STEP: 1680000 | > loss_text_ce: 0.02212347276508808 (0.029821742552595926) | > loss_mel_ce: 2.386509656906128 (nan) | > loss: 0.009558068588376045 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.1369915444640509) | > loader_time: 0.0046 (0.004655908077469426) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1680000.pth  --> TIME: 2025-02-27 11:30:23 -- STEP: 17840/36938 -- GLOBAL_STEP: 1680050 | > loss_text_ce: 0.02362319454550743 (0.029820503029061893) | > loss_mel_ce: 2.456014394760132 (nan) | > loss: 0.009839832782745361 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13698295968263097) | > loader_time: 0.0046 (0.00465609689197198)  --> TIME: 2025-02-27 11:30:33 -- STEP: 17890/36938 -- GLOBAL_STEP: 1680100 | > loss_text_ce: 0.03300415724515915 (0.0298189033479823) | > loss_mel_ce: 2.2547552585601807 (nan) | > loss: 0.00907841045409441 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13700229364643507) | > loader_time: 0.0046 (0.004656319181102155)  --> TIME: 2025-02-27 11:30:42 -- STEP: 17940/36938 -- GLOBAL_STEP: 1680150 | > loss_text_ce: 0.027405764907598495 (0.02982020651767969) | > loss_mel_ce: 2.3460443019866943 (nan) | > loss: 0.009418453089892864 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13700783317835916) | > loader_time: 0.0045 (0.004656711018072188)  --> TIME: 2025-02-27 11:30:52 -- STEP: 17990/36938 -- GLOBAL_STEP: 1680200 | > loss_text_ce: 0.02930123172700405 (0.029818577426493503) | > loss_mel_ce: 2.716416120529175 (nan) | > loss: 0.010895703919231892 (nan) | > current_lr: 5e-06 | > step_time: 0.1588 (0.13700522719123961) | > loader_time: 0.0047 (0.00465687366907566)  --> TIME: 2025-02-27 11:31:01 -- STEP: 18040/36938 -- GLOBAL_STEP: 1680250 | > loss_text_ce: 0.025565585121512413 (0.029820244300422873) | > loss_mel_ce: 2.340362548828125 (nan) | > loss: 0.009388604201376438 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13701425891229105) | > loader_time: 0.0046 (0.004657175971769175)  --> TIME: 2025-02-27 11:31:10 -- STEP: 18090/36938 -- GLOBAL_STEP: 1680300 | > loss_text_ce: 0.03295012563467026 (0.02981664409805369) | > loss_mel_ce: 2.567218780517578 (nan) | > loss: 0.010318131186068058 (nan) | > current_lr: 5e-06 | > step_time: 0.1504 (0.13700755065550985) | > loader_time: 0.0046 (0.004657136161660802)  --> TIME: 2025-02-27 11:31:20 -- STEP: 18140/36938 -- GLOBAL_STEP: 1680350 | > loss_text_ce: 0.02111952006816864 (0.02981369784230198) | > loss_mel_ce: 2.4758622646331787 (nan) | > loss: 0.00990865845233202 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13700514074985812) | > loader_time: 0.0046 (0.004657046376935957)  --> TIME: 2025-02-27 11:31:29 -- STEP: 18190/36938 -- GLOBAL_STEP: 1680400 | > loss_text_ce: 0.031527403742074966 (0.02981529198172225) | > loss_mel_ce: 2.851043701171875 (nan) | > loss: 0.011438775807619095 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13700552445443587) | > loader_time: 0.0046 (0.004657000103384789)  --> TIME: 2025-02-27 11:31:39 -- STEP: 18240/36938 -- GLOBAL_STEP: 1680450 | > loss_text_ce: 0.027622966095805168 (0.029812331728814923) | > loss_mel_ce: 2.7958974838256836 (nan) | > loss: 0.011204446665942669 (nan) | > current_lr: 5e-06 | > step_time: 0.1584 (0.1370148717964943) | > loader_time: 0.0046 (0.004656977663960366)  --> TIME: 2025-02-27 11:31:48 -- STEP: 18290/36938 -- GLOBAL_STEP: 1680500 | > loss_text_ce: 0.03849775344133377 (0.029814290639395988) | > loss_mel_ce: 0.05686759948730469 (nan) | > loss: 0.000378433964215219 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13700907272090598) | > loader_time: 0.0045 (0.0046567806329408995)  --> TIME: 2025-02-27 11:31:57 -- STEP: 18340/36938 -- GLOBAL_STEP: 1680550 | > loss_text_ce: 0.029844174161553383 (0.0298149574375351) | > loss_mel_ce: 2.3223876953125 (nan) | > loss: 0.009334254078567028 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13701546133669468) | > loader_time: 0.0045 (0.004656689923618184)  --> TIME: 2025-02-27 11:32:07 -- STEP: 18390/36938 -- GLOBAL_STEP: 1680600 | > loss_text_ce: 0.03098704293370247 (0.029814131115930394) | > loss_mel_ce: 2.61588191986084 (nan) | > loss: 0.01050344854593277 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.1370338237045762) | > loader_time: 0.0046 (0.004656709686059403)  --> TIME: 2025-02-27 11:32:17 -- STEP: 18440/36938 -- GLOBAL_STEP: 1680650 | > loss_text_ce: 0.028629865497350693 (0.029815645721981445) | > loss_mel_ce: 1.6034972667694092 (nan) | > loss: 0.006476695649325848 (nan) | > current_lr: 5e-06 | > step_time: 0.1443 (0.13704784504762108) | > loader_time: 0.0046 (0.004656811753477796)  --> TIME: 2025-02-27 11:32:26 -- STEP: 18490/36938 -- GLOBAL_STEP: 1680700 | > loss_text_ce: 0.03936495631933212 (0.029814670403352506) | > loss_mel_ce: 2.9973504543304443 (nan) | > loss: 0.012050459161400795 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13705499123211667) | > loader_time: 0.0045 (0.004656862722595924)  --> TIME: 2025-02-27 11:32:35 -- STEP: 18540/36938 -- GLOBAL_STEP: 1680750 | > loss_text_ce: 0.036602843552827835 (0.02981279504122747) | > loss_mel_ce: 2.2790586948394775 (nan) | > loss: 0.009189133532345295 (nan) | > current_lr: 5e-06 | > step_time: 0.111 (0.13705109051773257) | > loader_time: 0.0045 (0.004656748324150388)  --> TIME: 2025-02-27 11:32:44 -- STEP: 18590/36938 -- GLOBAL_STEP: 1680800 | > loss_text_ce: 0.03733439743518829 (0.029813342246078192) | > loss_mel_ce: 0.4756713807582855 (nan) | > loss: 0.0020357375033199787 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13704740127739876) | > loader_time: 0.0046 (0.004656718596775964)  --> TIME: 2025-02-27 11:32:54 -- STEP: 18640/36938 -- GLOBAL_STEP: 1680850 | > loss_text_ce: 0.029264358803629875 (0.02981590134033719) | > loss_mel_ce: 2.5889334678649902 (nan) | > loss: 0.010389674454927444 (nan) | > current_lr: 5e-06 | > step_time: 0.1892 (0.13704372932215034) | > loader_time: 0.0049 (0.004656775340501824)  --> TIME: 2025-02-27 11:33:03 -- STEP: 18690/36938 -- GLOBAL_STEP: 1680900 | > loss_text_ce: 0.025088978931307793 (0.02981691128744234) | > loss_mel_ce: 1.5587432384490967 (nan) | > loss: 0.006285049021244049 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13704380971215566) | > loader_time: 0.0046 (0.004656789671488132)  --> TIME: 2025-02-27 11:33:13 -- STEP: 18740/36938 -- GLOBAL_STEP: 1680950 | > loss_text_ce: 0.0321456715464592 (0.029816616199445495) | > loss_mel_ce: 2.7144672870635986 (nan) | > loss: 0.010899258777499199 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13705247108811552) | > loader_time: 0.0046 (0.00465683342934671)  --> TIME: 2025-02-27 11:33:22 -- STEP: 18790/36938 -- GLOBAL_STEP: 1681000 | > loss_text_ce: 0.03927941620349884 (0.029815480784632524) | > loss_mel_ce: 2.57373046875 (nan) | > loss: 0.01036908756941557 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1370560346606431) | > loader_time: 0.0045 (0.00465694680500689)  --> TIME: 2025-02-27 11:33:32 -- STEP: 18840/36938 -- GLOBAL_STEP: 1681050 | > loss_text_ce: 0.02597562037408352 (0.029818906733517073) | > loss_mel_ce: 2.135655164718628 (nan) | > loss: 0.008577900938689709 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13705949458093813) | > loader_time: 0.0045 (0.004657063248810479)  --> TIME: 2025-02-27 11:33:41 -- STEP: 18890/36938 -- GLOBAL_STEP: 1681100 | > loss_text_ce: 0.030446769669651985 (0.02982265863783537) | > loss_mel_ce: 2.834872245788574 (nan) | > loss: 0.011370314285159111 (nan) | > current_lr: 5e-06 | > step_time: 0.1873 (0.1370596531902807) | > loader_time: 0.0048 (0.004656969749971677)  --> TIME: 2025-02-27 11:33:50 -- STEP: 18940/36938 -- GLOBAL_STEP: 1681150 | > loss_text_ce: 0.021719379350543022 (0.02982056965090635) | > loss_mel_ce: 2.2957589626312256 (nan) | > loss: 0.009196343831717968 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.13704807497757585) | > loader_time: 0.0046 (0.004656829753420528)  --> TIME: 2025-02-27 11:33:59 -- STEP: 18990/36938 -- GLOBAL_STEP: 1681200 | > loss_text_ce: 0.0164191797375679 (0.029819001361439362) | > loss_mel_ce: 2.3879823684692383 (nan) | > loss: 0.009541276842355728 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.1370284855522682) | > loader_time: 0.0046 (0.004656777826342333)  --> TIME: 2025-02-27 11:34:09 -- STEP: 19040/36938 -- GLOBAL_STEP: 1681250 | > loss_text_ce: 0.02526683919131756 (0.0298202233982347) | > loss_mel_ce: 3.4581151008605957 (nan) | > loss: 0.013822944834828377 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.1370428886734129) | > loader_time: 0.0045 (0.004657225373412365)  --> TIME: 2025-02-27 11:34:18 -- STEP: 19090/36938 -- GLOBAL_STEP: 1681300 | > loss_text_ce: 0.028119150549173355 (0.029819445695041177) | > loss_mel_ce: 3.0695910453796387 (nan) | > loss: 0.012292501516640186 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13703047347480862) | > loader_time: 0.0048 (0.004657361880842447)  --> TIME: 2025-02-27 11:34:27 -- STEP: 19140/36938 -- GLOBAL_STEP: 1681350 | > loss_text_ce: 0.03534895181655884 (0.029821608365876395) | > loss_mel_ce: 2.501477003097534 (nan) | > loss: 0.010066770017147064 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13702677398639723) | > loader_time: 0.0046 (0.004657511912916878)  --> TIME: 2025-02-27 11:34:36 -- STEP: 19190/36938 -- GLOBAL_STEP: 1681400 | > loss_text_ce: 0.0312928631901741 (0.02982285570070817) | > loss_mel_ce: 2.1200098991394043 (nan) | > loss: 0.008536916226148605 (nan) | > current_lr: 5e-06 | > step_time: 0.1875 (0.13702359279038137) | > loader_time: 0.0049 (0.004657632985296421)  --> TIME: 2025-02-27 11:34:46 -- STEP: 19240/36938 -- GLOBAL_STEP: 1681450 | > loss_text_ce: 0.031995318830013275 (0.02982250158931566) | > loss_mel_ce: 2.7906558513641357 (nan) | > loss: 0.011200997047126293 (nan) | > current_lr: 5e-06 | > step_time: 0.136 (0.13702285568580247) | > loader_time: 0.0046 (0.00465784686270967)  --> TIME: 2025-02-27 11:34:55 -- STEP: 19290/36938 -- GLOBAL_STEP: 1681500 | > loss_text_ce: 0.02844075858592987 (0.02982415960813904) | > loss_mel_ce: 2.4765334129333496 (nan) | > loss: 0.009940373711287975 (nan) | > current_lr: 5e-06 | > step_time: 0.1599 (0.1370294192468886) | > loader_time: 0.0049 (0.004658346611208812)  --> TIME: 2025-02-27 11:35:05 -- STEP: 19340/36938 -- GLOBAL_STEP: 1681550 | > loss_text_ce: 0.02990207076072693 (0.029824058805961246) | > loss_mel_ce: 2.364689588546753 (nan) | > loss: 0.009502348490059376 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13703296273299498) | > loader_time: 0.0046 (0.004658499893259457)  --> TIME: 2025-02-27 11:35:14 -- STEP: 19390/36938 -- GLOBAL_STEP: 1681600 | > loss_text_ce: 0.029809286817908287 (0.029825020488255537) | > loss_mel_ce: 2.6130385398864746 (nan) | > loss: 0.010487492196261883 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13702588942076688) | > loader_time: 0.0046 (0.004658651917542643)  --> TIME: 2025-02-27 11:35:23 -- STEP: 19440/36938 -- GLOBAL_STEP: 1681650 | > loss_text_ce: 0.033427175134420395 (0.029824071519801203) | > loss_mel_ce: 0.0025812785606831312 (nan) | > loss: 0.00014289069804362953 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.1370356541234273) | > loader_time: 0.0047 (0.004659026971569741)  --> TIME: 2025-02-27 11:35:32 -- STEP: 19490/36938 -- GLOBAL_STEP: 1681700 | > loss_text_ce: 0.028309909626841545 (0.029824230820514) | > loss_mel_ce: 2.490541934967041 (nan) | > loss: 0.009995443746447563 (nan) | > current_lr: 5e-06 | > step_time: 0.1868 (0.1370300806993818) | > loader_time: 0.0049 (0.004659025543099344)  --> TIME: 2025-02-27 11:35:42 -- STEP: 19540/36938 -- GLOBAL_STEP: 1681750 | > loss_text_ce: 0.031417105346918106 (0.029823752605959698) | > loss_mel_ce: 2.8220105171203613 (nan) | > loss: 0.011323126032948494 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.137021183906582) | > loader_time: 0.0047 (0.004659152445798006)  --> TIME: 2025-02-27 11:35:51 -- STEP: 19590/36938 -- GLOBAL_STEP: 1681800 | > loss_text_ce: 0.036005642265081406 (0.029825564749365867) | > loss_mel_ce: 2.7039787769317627 (nan) | > loss: 0.010872955434024334 (nan) | > current_lr: 5e-06 | > step_time: 0.1589 (0.13701211016296125) | > loader_time: 0.0047 (0.004659191986442255)  --> TIME: 2025-02-27 11:36:00 -- STEP: 19640/36938 -- GLOBAL_STEP: 1681850 | > loss_text_ce: 0.031040670350193977 (0.029824479483779302) | > loss_mel_ce: 2.719226121902466 (nan) | > loss: 0.010913757607340813 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13701946723242575) | > loader_time: 0.0048 (0.004659262123515766)  --> TIME: 2025-02-27 11:36:10 -- STEP: 19690/36938 -- GLOBAL_STEP: 1681900 | > loss_text_ce: 0.021680954843759537 (0.029821087458724842) | > loss_mel_ce: 1.2243990898132324 (nan) | > loss: 0.004944762215018272 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13701971830481313) | > loader_time: 0.0044 (0.004659285940089859)  --> TIME: 2025-02-27 11:36:19 -- STEP: 19740/36938 -- GLOBAL_STEP: 1681950 | > loss_text_ce: 0.038356103003025055 (0.029818630260731494) | > loss_mel_ce: 0.3046875 (nan) | > loss: 0.0013612841721624136 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13700667940978398) | > loader_time: 0.0046 (0.004659222312490481)  --> TIME: 2025-02-27 11:36:28 -- STEP: 19790/36938 -- GLOBAL_STEP: 1682000 | > loss_text_ce: 0.03523246943950653 (0.029818357319908517) | > loss_mel_ce: 2.8637893199920654 (nan) | > loss: 0.011504055932164192 (nan) | > current_lr: 5e-06 | > step_time: 0.1862 (0.13700057174775085) | > loader_time: 0.0049 (0.0046592595542302205)  --> TIME: 2025-02-27 11:36:37 -- STEP: 19840/36938 -- GLOBAL_STEP: 1682050 | > loss_text_ce: 0.030815325677394867 (0.029818087483609805) | > loss_mel_ce: 2.7126944065093994 (nan) | > loss: 0.010886943899095058 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.1370016253523281) | > loader_time: 0.0045 (0.004659237147819586)  --> TIME: 2025-02-27 11:36:47 -- STEP: 19890/36938 -- GLOBAL_STEP: 1682100 | > loss_text_ce: 0.031557660549879074 (0.029817160145721902) | > loss_mel_ce: 2.9495203495025635 (nan) | > loss: 0.011829675175249577 (nan) | > current_lr: 5e-06 | > step_time: 0.1797 (0.13700738387110165) | > loader_time: 0.0056 (0.004659450275925523)  --> TIME: 2025-02-27 11:36:56 -- STEP: 19940/36938 -- GLOBAL_STEP: 1682150 | > loss_text_ce: 0.022042453289031982 (0.02981997529425872) | > loss_mel_ce: 0.03848876804113388 (nan) | > loss: 0.0002402032696409151 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1370115903578878) | > loader_time: 0.0045 (0.004659816625723245)  --> TIME: 2025-02-27 11:37:06 -- STEP: 19990/36938 -- GLOBAL_STEP: 1682200 | > loss_text_ce: 0.01790579780936241 (0.02981823556234185) | > loss_mel_ce: 1.3007779121398926 (nan) | > loss: 0.005232872441411018 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.1370176803475317) | > loader_time: 0.0045 (0.004659794329881285)  --> TIME: 2025-02-27 11:37:15 -- STEP: 20040/36938 -- GLOBAL_STEP: 1682250 | > loss_text_ce: 0.020527977496385574 (0.02981976448621476) | > loss_mel_ce: 2.5171000957489014 (nan) | > loss: 0.0100699532777071 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13702166031696536) | > loader_time: 0.0045 (0.004659777058812645)  --> TIME: 2025-02-27 11:37:24 -- STEP: 20090/36938 -- GLOBAL_STEP: 1682300 | > loss_text_ce: 0.01911531202495098 (0.029816704847460056) | > loss_mel_ce: 2.6356143951416016 (nan) | > loss: 0.01053464226424694 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1370161136149641) | > loader_time: 0.0046 (0.004659876709497028)  --> TIME: 2025-02-27 11:37:33 -- STEP: 20140/36938 -- GLOBAL_STEP: 1682350 | > loss_text_ce: 0.02599751204252243 (0.029815453693248502) | > loss_mel_ce: 2.284759521484375 (nan) | > loss: 0.00916967075318098 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13701118494808423) | > loader_time: 0.0045 (0.004659996771409918)  --> TIME: 2025-02-27 11:37:43 -- STEP: 20190/36938 -- GLOBAL_STEP: 1682400 | > loss_text_ce: 0.027151141315698624 (0.02981613966792735) | > loss_mel_ce: 2.9493837356567383 (nan) | > loss: 0.01181164663285017 (nan) | > current_lr: 5e-06 | > step_time: 0.1451 (0.13701362012340668) | > loader_time: 0.0047 (0.004660163367367285)  --> TIME: 2025-02-27 11:37:52 -- STEP: 20240/36938 -- GLOBAL_STEP: 1682450 | > loss_text_ce: 0.028148550540208817 (0.029811926131512206) | > loss_mel_ce: 2.445673704147339 (nan) | > loss: 0.009816755540668964 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.1370140265334729) | > loader_time: 0.0046 (0.004660237518695005)  --> TIME: 2025-02-27 11:38:02 -- STEP: 20290/36938 -- GLOBAL_STEP: 1682500 | > loss_text_ce: 0.02807980217039585 (0.02981127169407344) | > loss_mel_ce: 2.575559616088867 (nan) | > loss: 0.010331902652978897 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.1370162899239176) | > loader_time: 0.0047 (0.004660341691477277)  --> TIME: 2025-02-27 11:38:11 -- STEP: 20340/36938 -- GLOBAL_STEP: 1682550 | > loss_text_ce: 0.035253189504146576 (0.029811098394162728) | > loss_mel_ce: 3.1764090061187744 (nan) | > loss: 0.012744692154228687 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.1370209855784817) | > loader_time: 0.0047 (0.004660375397231116)  --> TIME: 2025-02-27 11:38:20 -- STEP: 20390/36938 -- GLOBAL_STEP: 1682600 | > loss_text_ce: 0.030050085857510567 (0.029812229987025613) | > loss_mel_ce: 2.8959999084472656 (nan) | > loss: 0.011611309833824635 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.13702208021799014) | > loader_time: 0.0048 (0.004660575561747918)  --> TIME: 2025-02-27 11:38:30 -- STEP: 20440/36938 -- GLOBAL_STEP: 1682650 | > loss_text_ce: 0.04192645102739334 (0.029811254136196907) | > loss_mel_ce: 2.638942241668701 (nan) | > loss: 0.01063836831599474 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13702532980306664) | > loader_time: 0.0044 (0.004660636816472669)  --> TIME: 2025-02-27 11:38:39 -- STEP: 20490/36938 -- GLOBAL_STEP: 1682700 | > loss_text_ce: 0.03865239769220352 (0.029817088355986156) | > loss_mel_ce: 2.2517032623291016 (nan) | > loss: 0.009088713675737381 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1370180236472908) | > loader_time: 0.0045 (0.004660558351485984)  --> TIME: 2025-02-27 11:38:48 -- STEP: 20540/36938 -- GLOBAL_STEP: 1682750 | > loss_text_ce: 0.03796210139989853 (0.029815000485802972) | > loss_mel_ce: 3.294645071029663 (nan) | > loss: 0.013224632479250431 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.13702182667629087) | > loader_time: 0.0048 (0.004660488440208079)  --> TIME: 2025-02-27 11:38:58 -- STEP: 20590/36938 -- GLOBAL_STEP: 1682800 | > loss_text_ce: 0.03274692967534065 (0.0298160083783721) | > loss_mel_ce: 2.624567985534668 (nan) | > loss: 0.010544901713728905 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13703309893318535) | > loader_time: 0.0064 (0.004660601080940304)  --> TIME: 2025-02-27 11:39:07 -- STEP: 20640/36938 -- GLOBAL_STEP: 1682850 | > loss_text_ce: 0.02826986089348793 (0.029814986941808952) | > loss_mel_ce: 2.5688350200653076 (nan) | > loss: 0.010305971838533878 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13703511316877853) | > loader_time: 0.0046 (0.004660625848197171)  --> TIME: 2025-02-27 11:39:17 -- STEP: 20690/36938 -- GLOBAL_STEP: 1682900 | > loss_text_ce: 0.026310443878173828 (0.02981714988617996) | > loss_mel_ce: 2.949056625366211 (nan) | > loss: 0.011807013303041458 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13703087512394646) | > loader_time: 0.0048 (0.0046606723555854546)  --> TIME: 2025-02-27 11:39:26 -- STEP: 20740/36938 -- GLOBAL_STEP: 1682950 | > loss_text_ce: 0.03529965505003929 (0.02981726113552357) | > loss_mel_ce: 2.6443092823028564 (nan) | > loss: 0.010633368976414204 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13703412004928725) | > loader_time: 0.0047 (0.0046606719321431855)  --> TIME: 2025-02-27 11:39:36 -- STEP: 20790/36938 -- GLOBAL_STEP: 1683000 | > loss_text_ce: 0.02694220095872879 (0.029819311052937114) | > loss_mel_ce: 0.01068115234375 (nan) | > loss: 0.00014929902681615204 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13704180180768352) | > loader_time: 0.0046 (0.004660644629874305)  --> TIME: 2025-02-27 11:39:45 -- STEP: 20840/36938 -- GLOBAL_STEP: 1683050 | > loss_text_ce: 0.025497430935502052 (0.029821437134875818) | > loss_mel_ce: 2.5762336254119873 (nan) | > loss: 0.010324330069124699 (nan) | > current_lr: 5e-06 | > step_time: 0.1711 (0.13704821319817953) | > loader_time: 0.0048 (0.004660590619363595)  --> TIME: 2025-02-27 11:39:54 -- STEP: 20890/36938 -- GLOBAL_STEP: 1683100 | > loss_text_ce: 0.03528547286987305 (0.029819211109896694) | > loss_mel_ce: 0.0124053955078125 (nan) | > loss: 0.00018924949108622968 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13704269093266436) | > loader_time: 0.0045 (0.004660516346739241)  --> TIME: 2025-02-27 11:40:04 -- STEP: 20940/36938 -- GLOBAL_STEP: 1683150 | > loss_text_ce: 0.019811255857348442 (0.02981864377360796) | > loss_mel_ce: 0.0136566162109375 (nan) | > loss: 0.00013280902931001037 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.1370420146466659) | > loader_time: 0.0045 (0.004660594076231286)  --> TIME: 2025-02-27 11:40:13 -- STEP: 20990/36938 -- GLOBAL_STEP: 1683200 | > loss_text_ce: 0.028484705835580826 (0.02981949524977035) | > loss_mel_ce: 2.399961233139038 (nan) | > loss: 0.009636690840125084 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13704075949597458) | > loader_time: 0.0046 (0.004660529667789315)  --> TIME: 2025-02-27 11:40:23 -- STEP: 21040/36938 -- GLOBAL_STEP: 1683250 | > loss_text_ce: 0.039289798587560654 (0.02981723135176998) | > loss_mel_ce: 2.023796319961548 (nan) | > loss: 0.008186849765479565 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13705229728620683) | > loader_time: 0.0045 (0.004660475344712271)  --> TIME: 2025-02-27 11:40:32 -- STEP: 21090/36938 -- GLOBAL_STEP: 1683300 | > loss_text_ce: 0.03907395526766777 (0.029815813950574303) | > loss_mel_ce: 2.4110770225524902 (nan) | > loss: 0.009722821414470673 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.1370564493425305) | > loader_time: 0.0046 (0.004660579026621492)  --> TIME: 2025-02-27 11:40:42 -- STEP: 21140/36938 -- GLOBAL_STEP: 1683350 | > loss_text_ce: 0.029829135164618492 (0.029814051278067685) | > loss_mel_ce: 0.0003584226069506258 (nan) | > loss: 0.00011979190458077937 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.1370752165761883) | > loader_time: 0.0045 (0.004660677571508698)  --> TIME: 2025-02-27 11:40:51 -- STEP: 21190/36938 -- GLOBAL_STEP: 1683400 | > loss_text_ce: 0.031349968165159225 (0.029814252839610523) | > loss_mel_ce: 2.729489326477051 (nan) | > loss: 0.01095571182668209 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.13707875548358595) | > loader_time: 0.0049 (0.004660848324565491)  --> TIME: 2025-02-27 11:41:01 -- STEP: 21240/36938 -- GLOBAL_STEP: 1683450 | > loss_text_ce: 0.02860007993876934 (0.029812619116213433) | > loss_mel_ce: 2.3032710552215576 (nan) | > loss: 0.009253457188606262 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13708350900876248) | > loader_time: 0.0046 (0.0046609410009365955)  --> TIME: 2025-02-27 11:41:10 -- STEP: 21290/36938 -- GLOBAL_STEP: 1683500 | > loss_text_ce: 0.031072000041604042 (0.029812680327711043) | > loss_mel_ce: 1.8916102647781372 (nan) | > loss: 0.00762969208881259 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.1370780973492673) | > loader_time: 0.0045 (0.0046609238651001295)  --> TIME: 2025-02-27 11:41:19 -- STEP: 21340/36938 -- GLOBAL_STEP: 1683550 | > loss_text_ce: 0.02068403735756874 (0.029813253371116583) | > loss_mel_ce: 2.0294690132141113 (nan) | > loss: 0.008135528303682804 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13706784209248632) | > loader_time: 0.0047 (0.004660927824808687)  --> TIME: 2025-02-27 11:41:29 -- STEP: 21390/36938 -- GLOBAL_STEP: 1683600 | > loss_text_ce: 0.026082077994942665 (0.029810284478777132) | > loss_mel_ce: 1.863408088684082 (nan) | > loss: 0.007497976999729872 (nan) | > current_lr: 5e-06 | > step_time: 0.1591 (0.13707239749332328) | > loader_time: 0.0046 (0.004660902072358968)  --> TIME: 2025-02-27 11:41:38 -- STEP: 21440/36938 -- GLOBAL_STEP: 1683650 | > loss_text_ce: 0.027176784351468086 (0.029809813481258635) | > loss_mel_ce: 2.482144832611084 (nan) | > loss: 0.009957626461982727 (nan) | > current_lr: 5e-06 | > step_time: 0.1616 (0.13708246244645764) | > loader_time: 0.0047 (0.00466088565872674)  --> TIME: 2025-02-27 11:41:48 -- STEP: 21490/36938 -- GLOBAL_STEP: 1683700 | > loss_text_ce: 0.03472335264086723 (0.029810474142900275) | > loss_mel_ce: 3.0168087482452393 (nan) | > loss: 0.012109254486858845 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.1370797232396655) | > loader_time: 0.0045 (0.004660876721434925)  --> TIME: 2025-02-27 11:41:57 -- STEP: 21540/36938 -- GLOBAL_STEP: 1683750 | > loss_text_ce: 0.022877825424075127 (0.0298078280032207) | > loss_mel_ce: 2.1593804359436035 (nan) | > loss: 0.008659755811095238 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.1370803979667331) | > loader_time: 0.0046 (0.0046608289525590565)  --> TIME: 2025-02-27 11:42:06 -- STEP: 21590/36938 -- GLOBAL_STEP: 1683800 | > loss_text_ce: 0.026562627404928207 (0.029805521429276984) | > loss_mel_ce: 2.384728193283081 (nan) | > loss: 0.009568614885210991 (nan) | > current_lr: 5e-06 | > step_time: 0.1571 (0.13708938648988864) | > loader_time: 0.0046 (0.004660819768353467)  --> TIME: 2025-02-27 11:42:16 -- STEP: 21640/36938 -- GLOBAL_STEP: 1683850 | > loss_text_ce: 0.03250725567340851 (0.029806112150176204) | > loss_mel_ce: 2.612940549850464 (nan) | > loss: 0.010497809387743473 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.1370922882764037) | > loader_time: 0.0045 (0.004660791235798588)  --> TIME: 2025-02-27 11:42:25 -- STEP: 21690/36938 -- GLOBAL_STEP: 1683900 | > loss_text_ce: 0.030084649100899696 (0.029805486272494963) | > loss_mel_ce: 1.8048909902572632 (nan) | > loss: 0.007281649857759476 (nan) | > current_lr: 5e-06 | > step_time: 0.18 (0.1370973655297048) | > loader_time: 0.0048 (0.004660762362130594)  --> TIME: 2025-02-27 11:42:34 -- STEP: 21740/36938 -- GLOBAL_STEP: 1683950 | > loss_text_ce: 0.03396834805607796 (0.029807200677831703) | > loss_mel_ce: 3.262070894241333 (nan) | > loss: 0.013079522177577019 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13708777761064483) | > loader_time: 0.0046 (0.004660773672088069)  --> TIME: 2025-02-27 11:42:44 -- STEP: 21790/36938 -- GLOBAL_STEP: 1684000 | > loss_text_ce: 0.023738767951726913 (0.029806517578016927) | > loss_mel_ce: 2.225247383117676 (nan) | > loss: 0.00892454944550991 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13708976628092082) | > loader_time: 0.0046 (0.00466097929763267)  --> TIME: 2025-02-27 11:42:53 -- STEP: 21840/36938 -- GLOBAL_STEP: 1684050 | > loss_text_ce: 0.028981029987335205 (0.02980754209384699) | > loss_mel_ce: 2.072859764099121 (nan) | > loss: 0.008340638130903244 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13708285488488403) | > loader_time: 0.0046 (0.004660929799516542)  --> TIME: 2025-02-27 11:43:02 -- STEP: 21890/36938 -- GLOBAL_STEP: 1684100 | > loss_text_ce: 0.043638285249471664 (0.029809902486841938) | > loss_mel_ce: 2.5301947593688965 (nan) | > loss: 0.010213623754680157 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13709177925141167) | > loader_time: 0.0045 (0.004660935693813152)  --> TIME: 2025-02-27 11:43:12 -- STEP: 21940/36938 -- GLOBAL_STEP: 1684150 | > loss_text_ce: 0.0355532169342041 (0.029811411348121458) | > loss_mel_ce: 2.9315974712371826 (nan) | > loss: 0.011774408631026745 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13708600805581592) | > loader_time: 0.0046 (0.004660871394026131)  --> TIME: 2025-02-27 11:43:21 -- STEP: 21990/36938 -- GLOBAL_STEP: 1684200 | > loss_text_ce: 0.027630409225821495 (0.029813349313259803) | > loss_mel_ce: 2.6602747440338135 (nan) | > loss: 0.010666290298104286 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13707641826645198) | > loader_time: 0.0046 (0.004660947640520047)  --> TIME: 2025-02-27 11:43:30 -- STEP: 22040/36938 -- GLOBAL_STEP: 1684250 | > loss_text_ce: 0.02571495622396469 (0.029813721010456394) | > loss_mel_ce: 2.7185475826263428 (nan) | > loss: 0.01088993065059185 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13706041831935534) | > loader_time: 0.0045 (0.004661101730246286)  --> TIME: 2025-02-27 11:43:39 -- STEP: 22090/36938 -- GLOBAL_STEP: 1684300 | > loss_text_ce: 0.030436141416430473 (0.029814689322983996) | > loss_mel_ce: 2.845106601715088 (nan) | > loss: 0.01141088455915451 (nan) | > current_lr: 5e-06 | > step_time: 0.1779 (0.1370638298955955) | > loader_time: 0.0049 (0.004661246174975935)  --> TIME: 2025-02-27 11:43:48 -- STEP: 22140/36938 -- GLOBAL_STEP: 1684350 | > loss_text_ce: 0.026135822758078575 (0.02981683448139603) | > loss_mel_ce: 3.1404495239257812 (nan) | > loss: 0.012565815821290016 (nan) | > current_lr: 5e-06 | > step_time: 0.1626 (0.13706345253412783) | > loader_time: 0.0047 (0.004661270381097004)  --> TIME: 2025-02-27 11:43:58 -- STEP: 22190/36938 -- GLOBAL_STEP: 1684400 | > loss_text_ce: 0.029147595167160034 (0.02981772661748847) | > loss_mel_ce: 1.9591031074523926 (nan) | > loss: 0.007889884524047375 (nan) | > current_lr: 5e-06 | > step_time: 0.1269 (0.13707541061339415) | > loader_time: 0.0046 (0.004661510741726774)  --> TIME: 2025-02-27 11:44:08 -- STEP: 22240/36938 -- GLOBAL_STEP: 1684450 | > loss_text_ce: 0.02567444182932377 (0.029817782972416714) | > loss_mel_ce: 2.7508652210235596 (nan) | > loss: 0.011018014512956142 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.13708116866701742) | > loader_time: 0.0048 (0.004661749657109483)  --> TIME: 2025-02-27 11:44:17 -- STEP: 22290/36938 -- GLOBAL_STEP: 1684500 | > loss_text_ce: 0.0246902983635664 (0.02981625634653787) | > loss_mel_ce: 2.323704242706299 (nan) | > loss: 0.009319026954472065 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13708162525941933) | > loader_time: 0.0045 (0.004661646751396964)  --> TIME: 2025-02-27 11:44:26 -- STEP: 22340/36938 -- GLOBAL_STEP: 1684550 | > loss_text_ce: 0.032380614429712296 (0.02981616120761297) | > loss_mel_ce: 2.6246044635772705 (nan) | > loss: 0.010543592274188995 (nan) | > current_lr: 5e-06 | > step_time: 0.1542 (0.13707464598285943) | > loader_time: 0.0047 (0.004661566899520752)  --> TIME: 2025-02-27 11:44:35 -- STEP: 22390/36938 -- GLOBAL_STEP: 1684600 | > loss_text_ce: 0.021952083334326744 (0.029814113607873897) | > loss_mel_ce: 2.573068618774414 (nan) | > loss: 0.010297701694071293 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13707451739445314) | > loader_time: 0.0045 (0.004661580354521028)  --> TIME: 2025-02-27 11:44:45 -- STEP: 22440/36938 -- GLOBAL_STEP: 1684650 | > loss_text_ce: 0.03666332736611366 (0.029817481324659365) | > loss_mel_ce: 2.9958996772766113 (nan) | > loss: 0.012033980339765549 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.1370707166386153) | > loader_time: 0.0047 (0.0046617388512786495)  --> TIME: 2025-02-27 11:44:54 -- STEP: 22490/36938 -- GLOBAL_STEP: 1684700 | > loss_text_ce: 0.029260674491524696 (0.029816855860459138) | > loss_mel_ce: 2.158418655395508 (nan) | > loss: 0.008681267499923706 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13707406089378943) | > loader_time: 0.0046 (0.0046619087391400055)  --> TIME: 2025-02-27 11:45:03 -- STEP: 22540/36938 -- GLOBAL_STEP: 1684750 | > loss_text_ce: 0.02858462929725647 (0.029815828052044472) | > loss_mel_ce: 2.96803879737854 (nan) | > loss: 0.011891364119946957 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13707483427230438) | > loader_time: 0.0046 (0.00466188388052723)  --> TIME: 2025-02-27 11:45:13 -- STEP: 22590/36938 -- GLOBAL_STEP: 1684800 | > loss_text_ce: 0.03933525085449219 (0.029815442581801646) | > loss_mel_ce: 2.5590085983276367 (nan) | > loss: 0.010310889221727848 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13707234304317412) | > loader_time: 0.0045 (0.004661949718352073)  --> TIME: 2025-02-27 11:45:22 -- STEP: 22640/36938 -- GLOBAL_STEP: 1684850 | > loss_text_ce: 0.028503302484750748 (0.029813280801063373) | > loss_mel_ce: 2.8424606323242188 (nan) | > loss: 0.011392715387046337 (nan) | > current_lr: 5e-06 | > step_time: 0.1527 (0.13707109210558485) | > loader_time: 0.0046 (0.004662019414531053)  --> TIME: 2025-02-27 11:45:32 -- STEP: 22690/36938 -- GLOBAL_STEP: 1684900 | > loss_text_ce: 0.02409954182803631 (0.029812598402446513) | > loss_mel_ce: 2.178931713104248 (nan) | > loss: 0.008742188103497028 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13708037284566857) | > loader_time: 0.0046 (0.004661991450165593)  --> TIME: 2025-02-27 11:45:41 -- STEP: 22740/36938 -- GLOBAL_STEP: 1684950 | > loss_text_ce: 0.030544592067599297 (0.029812751036004663) | > loss_mel_ce: 2.863868474960327 (nan) | > loss: 0.011485766619443893 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.1370891829488976) | > loader_time: 0.0046 (0.004662017121897948)  --> TIME: 2025-02-27 11:45:51 -- STEP: 22790/36938 -- GLOBAL_STEP: 1685000 | > loss_text_ce: 0.02864423207938671 (0.029813195513982295) | > loss_mel_ce: 2.7886762619018555 (nan) | > loss: 0.011179843917489052 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13710446848373373) | > loader_time: 0.0046 (0.004662050495758665)  --> TIME: 2025-02-27 11:46:00 -- STEP: 22840/36938 -- GLOBAL_STEP: 1685050 | > loss_text_ce: 0.03269271180033684 (0.02980920887933866) | > loss_mel_ce: 2.856442928314209 (nan) | > loss: 0.01146482490003109 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13709900490874308) | > loader_time: 0.0046 (0.004661955265572951)  --> TIME: 2025-02-27 11:46:10 -- STEP: 22890/36938 -- GLOBAL_STEP: 1685100 | > loss_text_ce: 0.0168758537620306 (0.029806712045272857) | > loss_mel_ce: 1.2532958984375 (nan) | > loss: 0.0050403643399477005 (nan) | > current_lr: 5e-06 | > step_time: 0.1088 (0.13710407592053442) | > loader_time: 0.0045 (0.004661884553675158)  --> TIME: 2025-02-27 11:46:19 -- STEP: 22940/36938 -- GLOBAL_STEP: 1685150 | > loss_text_ce: 0.02579694427549839 (0.029806116578975462) | > loss_mel_ce: 2.835892677307129 (nan) | > loss: 0.011355911381542683 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.137113112257996) | > loader_time: 0.0045 (0.004661840174441355)  --> TIME: 2025-02-27 11:46:28 -- STEP: 22990/36938 -- GLOBAL_STEP: 1685200 | > loss_text_ce: 0.017619531601667404 (0.029804353969493307) | > loss_mel_ce: 2.474395751953125 (nan) | > loss: 0.009888950735330582 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.1371018361298605) | > loader_time: 0.0046 (0.004661798103833628)  --> TIME: 2025-02-27 11:46:38 -- STEP: 23040/36938 -- GLOBAL_STEP: 1685250 | > loss_text_ce: 0.0315127968788147 (0.02980204911614744) | > loss_mel_ce: 3.248603582382202 (nan) | > loss: 0.013016335666179657 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13709866249312913) | > loader_time: 0.0046 (0.004661712278094559)  --> TIME: 2025-02-27 11:46:47 -- STEP: 23090/36938 -- GLOBAL_STEP: 1685300 | > loss_text_ce: 0.02589571103453636 (0.0298013546859427) | > loss_mel_ce: 2.073411703109741 (nan) | > loss: 0.008330585435032845 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13709316279476122) | > loader_time: 0.0046 (0.004661634155247006)  --> TIME: 2025-02-27 11:46:56 -- STEP: 23140/36938 -- GLOBAL_STEP: 1685350 | > loss_text_ce: 0.02088233456015587 (0.02980295697938694) | > loss_mel_ce: 1.5039043426513672 (nan) | > loss: 0.006050741299986839 (nan) | > current_lr: 5e-06 | > step_time: 0.1082 (0.13708780850432678) | > loader_time: 0.0045 (0.004661752730479198)  --> TIME: 2025-02-27 11:47:05 -- STEP: 23190/36938 -- GLOBAL_STEP: 1685400 | > loss_text_ce: 0.02684791199862957 (0.02980258444467912) | > loss_mel_ce: 2.6545591354370117 (nan) | > loss: 0.010640504769980907 (nan) | > current_lr: 5e-06 | > step_time: 0.1273 (0.13708670576259216) | > loader_time: 0.0046 (0.004661854159580113)  --> TIME: 2025-02-27 11:47:15 -- STEP: 23240/36938 -- GLOBAL_STEP: 1685450 | > loss_text_ce: 0.027313129976391792 (0.029802285465083247) | > loss_mel_ce: 2.756301164627075 (nan) | > loss: 0.011046089231967926 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.13709214777010803) | > loader_time: 0.0049 (0.004662043523049805)  --> TIME: 2025-02-27 11:47:24 -- STEP: 23290/36938 -- GLOBAL_STEP: 1685500 | > loss_text_ce: 0.03483564779162407 (0.029802932818858114) | > loss_mel_ce: 3.252474784851074 (nan) | > loss: 0.013044883497059345 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.1370982912488632) | > loader_time: 0.0048 (0.004662204044681088)  --> TIME: 2025-02-27 11:47:33 -- STEP: 23340/36938 -- GLOBAL_STEP: 1685550 | > loss_text_ce: 0.020431380718946457 (0.029799094723678036) | > loss_mel_ce: 2.190054416656494 (nan) | > loss: 0.008771768771111965 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13707940733728965) | > loader_time: 0.0045 (0.004662098727271205)  --> TIME: 2025-02-27 11:47:43 -- STEP: 23390/36938 -- GLOBAL_STEP: 1685600 | > loss_text_ce: 0.026281455531716347 (0.029798509682161974) | > loss_mel_ce: 2.5839107036590576 (nan) | > loss: 0.010357905179262161 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1370896618913612) | > loader_time: 0.0045 (0.004662057689847781)  --> TIME: 2025-02-27 11:47:52 -- STEP: 23440/36938 -- GLOBAL_STEP: 1685650 | > loss_text_ce: 0.03156808391213417 (0.029797646094831962) | > loss_mel_ce: 2.73764967918396 (nan) | > loss: 0.010988960042595863 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.1370942001863547) | > loader_time: 0.0049 (0.004662068884934176)  --> TIME: 2025-02-27 11:48:02 -- STEP: 23490/36938 -- GLOBAL_STEP: 1685700 | > loss_text_ce: 0.02371254190802574 (0.02979660385723234) | > loss_mel_ce: 2.364644765853882 (nan) | > loss: 0.00947760883718729 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1370985829510447) | > loader_time: 0.0056 (0.004662587014397647)  --> TIME: 2025-02-27 11:48:11 -- STEP: 23540/36938 -- GLOBAL_STEP: 1685750 | > loss_text_ce: 0.025085866451263428 (0.029796645751843338) | > loss_mel_ce: 2.4649925231933594 (nan) | > loss: 0.00988126453012228 (nan) | > current_lr: 5e-06 | > step_time: 0.1618 (0.13708874228249754) | > loader_time: 0.0046 (0.004662592945811909)  --> TIME: 2025-02-27 11:48:20 -- STEP: 23590/36938 -- GLOBAL_STEP: 1685800 | > loss_text_ce: 0.027266638353466988 (0.029796198317911388) | > loss_mel_ce: 2.3693063259124756 (nan) | > loss: 0.00951021071523428 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.13709084000209418) | > loader_time: 0.0046 (0.004662524951418969)  --> TIME: 2025-02-27 11:48:29 -- STEP: 23640/36938 -- GLOBAL_STEP: 1685850 | > loss_text_ce: 0.01323388796299696 (0.02979431379375297) | > loss_mel_ce: 1.0039914846420288 (nan) | > loss: 0.004036609083414078 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.13708512031689055) | > loader_time: 0.0046 (0.004662425526104203)  --> TIME: 2025-02-27 11:48:39 -- STEP: 23690/36938 -- GLOBAL_STEP: 1685900 | > loss_text_ce: 0.032597657293081284 (0.02979162196401129) | > loss_mel_ce: 2.881340980529785 (nan) | > loss: 0.011563248932361603 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.1370984610750379) | > loader_time: 0.0053 (0.00466279064667985)  --> TIME: 2025-02-27 11:48:48 -- STEP: 23740/36938 -- GLOBAL_STEP: 1685950 | > loss_text_ce: 0.02934286557137966 (0.029788992484747475) | > loss_mel_ce: 2.4838099479675293 (nan) | > loss: 0.00997282937169075 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1370966676914496) | > loader_time: 0.0046 (0.004662814819220246)  --> TIME: 2025-02-27 11:48:57 -- STEP: 23790/36938 -- GLOBAL_STEP: 1686000 | > loss_text_ce: 0.03337223827838898 (0.029787828706239354) | > loss_mel_ce: 2.607320547103882 (nan) | > loss: 0.010478939861059189 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13708118981501258) | > loader_time: 0.0045 (0.004662758675679885)  --> TIME: 2025-02-27 11:49:07 -- STEP: 23840/36938 -- GLOBAL_STEP: 1686050 | > loss_text_ce: 0.028097029775381088 (0.029789637910099028) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13708469327664177) | > loader_time: 0.0046 (0.004662747401119081)  --> TIME: 2025-02-27 11:49:16 -- STEP: 23890/36938 -- GLOBAL_STEP: 1686100 | > loss_text_ce: 0.011723984032869339 (0.029786728204424087) | > loss_mel_ce: 0.2927398681640625 (nan) | > loss: 0.0012081899913027883 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13708356411799327) | > loader_time: 0.0045 (0.004662721583213504)  --> TIME: 2025-02-27 11:49:25 -- STEP: 23940/36938 -- GLOBAL_STEP: 1686150 | > loss_text_ce: 0.023493779823184013 (0.02978561157419028) | > loss_mel_ce: 3.261561393737793 (nan) | > loss: 0.01303593348711729 (nan) | > current_lr: 5e-06 | > step_time: 0.1705 (0.1370838913901603) | > loader_time: 0.0049 (0.004662710094611249)  --> TIME: 2025-02-27 11:49:35 -- STEP: 23990/36938 -- GLOBAL_STEP: 1686200 | > loss_text_ce: 0.02447686158120632 (0.02978466702388347) | > loss_mel_ce: 2.5074567794799805 (nan) | > loss: 0.01004735566675663 (nan) | > current_lr: 5e-06 | > step_time: 0.17 (0.13707827260564734) | > loader_time: 0.0048 (0.004662700502412329)  --> TIME: 2025-02-27 11:49:44 -- STEP: 24040/36938 -- GLOBAL_STEP: 1686250 | > loss_text_ce: 0.04039206728339195 (0.029786723048678812) | > loss_mel_ce: 3.425100803375244 (nan) | > loss: 0.013751956634223461 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.1370768095014886) | > loader_time: 0.0046 (0.004662852388451784)  --> TIME: 2025-02-27 11:49:53 -- STEP: 24090/36938 -- GLOBAL_STEP: 1686300 | > loss_text_ce: 0.027934785932302475 (0.029786840504775324) | > loss_mel_ce: 2.541015625 (nan) | > loss: 0.010194248519837856 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13707200627744554) | > loader_time: 0.0046 (0.004662855189091049)  --> TIME: 2025-02-27 11:50:02 -- STEP: 24140/36938 -- GLOBAL_STEP: 1686350 | > loss_text_ce: 0.024327142164111137 (0.02978378740191776) | > loss_mel_ce: 2.1955833435058594 (nan) | > loss: 0.008809168823063374 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13706136344381809) | > loader_time: 0.0047 (0.0046629630442779745)  --> TIME: 2025-02-27 11:50:11 -- STEP: 24190/36938 -- GLOBAL_STEP: 1686400 | > loss_text_ce: 0.027686428278684616 (0.029783848058447755) | > loss_mel_ce: 2.704780340194702 (nan) | > loss: 0.010843122377991676 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13705183835796342) | > loader_time: 0.0047 (0.004662966738066531)  --> TIME: 2025-02-27 11:50:21 -- STEP: 24240/36938 -- GLOBAL_STEP: 1686450 | > loss_text_ce: 0.032183099538087845 (0.029782682491225135) | > loss_mel_ce: 2.824589729309082 (nan) | > loss: 0.011336401104927063 (nan) | > current_lr: 5e-06 | > step_time: 0.1512 (0.13705615451430272) | > loader_time: 0.0046 (0.004663072216628799)  --> TIME: 2025-02-27 11:50:30 -- STEP: 24290/36938 -- GLOBAL_STEP: 1686500 | > loss_text_ce: 0.03350019082427025 (0.029783719599426432) | > loss_mel_ce: 2.1064345836639404 (nan) | > loss: 0.00849180482327938 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13705637311680166) | > loader_time: 0.0045 (0.0046630450068505585)  --> TIME: 2025-02-27 11:50:39 -- STEP: 24340/36938 -- GLOBAL_STEP: 1686550 | > loss_text_ce: 0.02904573455452919 (0.029779306638583358) | > loss_mel_ce: 2.813368797302246 (nan) | > loss: 0.011279423721134663 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.137051763834283) | > loader_time: 0.0049 (0.004663005458985374)  --> TIME: 2025-02-27 11:50:48 -- STEP: 24390/36938 -- GLOBAL_STEP: 1686600 | > loss_text_ce: 0.03037596121430397 (0.029776562184940042) | > loss_mel_ce: 2.3964972496032715 (nan) | > loss: 0.009630450047552586 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.1370457224984692) | > loader_time: 0.0048 (0.004662910579119901)  --> TIME: 2025-02-27 11:50:58 -- STEP: 24440/36938 -- GLOBAL_STEP: 1686650 | > loss_text_ce: 0.031782105565071106 (0.02977904685381984) | > loss_mel_ce: 2.354121446609497 (nan) | > loss: 0.009467871859669685 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13705298033361518) | > loader_time: 0.0048 (0.004662946310449147)  --> TIME: 2025-02-27 11:51:07 -- STEP: 24490/36938 -- GLOBAL_STEP: 1686700 | > loss_text_ce: 0.030537670478224754 (0.029778271000041337) | > loss_mel_ce: 1.3386179208755493 (nan) | > loss: 0.005433157552033663 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.1370536269339407) | > loader_time: 0.0045 (0.004662911781927864)  --> TIME: 2025-02-27 11:51:16 -- STEP: 24540/36938 -- GLOBAL_STEP: 1686750 | > loss_text_ce: 0.020058760419487953 (0.029776809036089598) | > loss_mel_ce: 0.04017685353755951 (nan) | > loss: 0.00023903022520244122 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13704918157597695) | > loader_time: 0.0045 (0.004662823745071547)  --> TIME: 2025-02-27 11:51:26 -- STEP: 24590/36938 -- GLOBAL_STEP: 1686800 | > loss_text_ce: 0.030774623155593872 (0.029776376196353333) | > loss_mel_ce: 3.115022897720337 (nan) | > loss: 0.01248332392424345 (nan) | > current_lr: 5e-06 | > step_time: 0.1899 (0.13704745524198136) | > loader_time: 0.0048 (0.004662760790405646)  --> TIME: 2025-02-27 11:51:35 -- STEP: 24640/36938 -- GLOBAL_STEP: 1686850 | > loss_text_ce: 0.032346904277801514 (0.029777315552323744) | > loss_mel_ce: 2.505424976348877 (nan) | > loss: 0.01007052417844534 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13704301057891374) | > loader_time: 0.0047 (0.004662856778928214)  --> TIME: 2025-02-27 11:51:44 -- STEP: 24690/36938 -- GLOBAL_STEP: 1686900 | > loss_text_ce: 0.027929965406656265 (0.029776275859607806) | > loss_mel_ce: 2.484997510910034 (nan) | > loss: 0.00997193530201912 (nan) | > current_lr: 5e-06 | > step_time: 0.1486 (0.13704588000906623) | > loader_time: 0.0048 (0.004663212002892878)  --> TIME: 2025-02-27 11:51:54 -- STEP: 24740/36938 -- GLOBAL_STEP: 1686950 | > loss_text_ce: 0.024787982925772667 (0.02977686878489505) | > loss_mel_ce: 2.834437847137451 (nan) | > loss: 0.01134613435715437 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13704959543462467) | > loader_time: 0.0045 (0.0046632071223679135)  --> TIME: 2025-02-27 11:52:03 -- STEP: 24790/36938 -- GLOBAL_STEP: 1687000 | > loss_text_ce: 0.024608680978417397 (0.029778182703194708) | > loss_mel_ce: 3.0359044075012207 (nan) | > loss: 0.012144893407821655 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.1370399887016479) | > loader_time: 0.0046 (0.004663117165621268)  --> TIME: 2025-02-27 11:52:12 -- STEP: 24840/36938 -- GLOBAL_STEP: 1687050 | > loss_text_ce: 0.03026164136826992 (0.02977786800503918) | > loss_mel_ce: 2.3384416103363037 (nan) | > loss: 0.009399617090821266 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13703796214719466) | > loader_time: 0.0046 (0.004663081560733811)  --> TIME: 2025-02-27 11:52:22 -- STEP: 24890/36938 -- GLOBAL_STEP: 1687100 | > loss_text_ce: 0.03583553060889244 (0.02977465394612698) | > loss_mel_ce: 0.006838703062385321 (nan) | > loss: 0.0001693421945674345 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.1370557222657526) | > loader_time: 0.0046 (0.004663072019372418)  --> TIME: 2025-02-27 11:52:31 -- STEP: 24940/36938 -- GLOBAL_STEP: 1687150 | > loss_text_ce: 0.03849000856280327 (0.02977692702103261) | > loss_mel_ce: 2.2315151691436768 (nan) | > loss: 0.009007957763969898 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.1370541806783114) | > loader_time: 0.0045 (0.004663015635566128)  --> TIME: 2025-02-27 11:52:41 -- STEP: 24990/36938 -- GLOBAL_STEP: 1687200 | > loss_text_ce: 0.043837834149599075 (0.029776715577839806) | > loss_mel_ce: 2.460617780685425 (nan) | > loss: 0.00993831641972065 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13706426094798552) | > loader_time: 0.0046 (0.004663133897892042)  --> TIME: 2025-02-27 11:52:50 -- STEP: 25040/36938 -- GLOBAL_STEP: 1687250 | > loss_text_ce: 0.022138511762022972 (0.02977655419537549) | > loss_mel_ce: 2.1177330017089844 (nan) | > loss: 0.00849155429750681 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13706431139391423) | > loader_time: 0.0046 (0.004663231264288048)  --> TIME: 2025-02-27 11:52:59 -- STEP: 25090/36938 -- GLOBAL_STEP: 1687300 | > loss_text_ce: 0.03348924219608307 (0.029775650264125892) | > loss_mel_ce: 2.5309197902679443 (nan) | > loss: 0.010176226496696472 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.1370550777960601) | > loader_time: 0.0047 (0.0046631133428081785)  --> TIME: 2025-02-27 11:53:09 -- STEP: 25140/36938 -- GLOBAL_STEP: 1687350 | > loss_text_ce: 0.030234962701797485 (0.02977549432815397) | > loss_mel_ce: 3.061218738555908 (nan) | > loss: 0.012267674319446087 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.13705409877150407) | > loader_time: 0.0046 (0.004663178820219318)  --> TIME: 2025-02-27 11:53:18 -- STEP: 25190/36938 -- GLOBAL_STEP: 1687400 | > loss_text_ce: 0.036008961498737335 (0.029774980925122655) | > loss_mel_ce: 3.1133873462677 (nan) | > loss: 0.012497605755925179 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.1370595318876404) | > loader_time: 0.0046 (0.004663105728418969)  --> TIME: 2025-02-27 11:53:28 -- STEP: 25240/36938 -- GLOBAL_STEP: 1687450 | > loss_text_ce: 0.03296324983239174 (0.029771686329767594) | > loss_mel_ce: 2.452880859375 (nan) | > loss: 0.009864461608231068 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13706250610139945) | > loader_time: 0.0045 (0.004663010170644886)  --> TIME: 2025-02-27 11:53:37 -- STEP: 25290/36938 -- GLOBAL_STEP: 1687500 | > loss_text_ce: 0.03175968676805496 (0.02977200763299326) | > loss_mel_ce: 2.2128586769104004 (nan) | > loss: 0.008907216601073742 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13706629399774373) | > loader_time: 0.0053 (0.004663186037479154)  --> TIME: 2025-02-27 11:53:47 -- STEP: 25340/36938 -- GLOBAL_STEP: 1687550 | > loss_text_ce: 0.026644762605428696 (0.029772358358894278) | > loss_mel_ce: 3.033345937728882 (nan) | > loss: 0.012142820283770561 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.1370644244605758) | > loader_time: 0.0046 (0.0046630957320438335)  --> TIME: 2025-02-27 11:53:56 -- STEP: 25390/36938 -- GLOBAL_STEP: 1687600 | > loss_text_ce: 0.03482314199209213 (0.029770864584373544) | > loss_mel_ce: 2.434511661529541 (nan) | > loss: 0.009798948653042316 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.1370563000194103) | > loader_time: 0.0045 (0.004663084397498147)  --> TIME: 2025-02-27 11:54:05 -- STEP: 25440/36938 -- GLOBAL_STEP: 1687650 | > loss_text_ce: 0.029780933633446693 (0.02977216553613662) | > loss_mel_ce: 0.00212860107421875 (nan) | > loss: 0.00012662513472605497 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1370664579313505) | > loader_time: 0.0046 (0.004663035245436537)  --> TIME: 2025-02-27 11:54:15 -- STEP: 25490/36938 -- GLOBAL_STEP: 1687700 | > loss_text_ce: 0.03008393943309784 (0.02977164979956873) | > loss_mel_ce: 3.121058225631714 (nan) | > loss: 0.012504532933235168 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.1370709891317404) | > loader_time: 0.0054 (0.00466299816504231)  --> TIME: 2025-02-27 11:54:24 -- STEP: 25540/36938 -- GLOBAL_STEP: 1687750 | > loss_text_ce: 0.019034409895539284 (0.02977393386264157) | > loss_mel_ce: 2.326714515686035 (nan) | > loss: 0.009308528155088425 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.1370699714008679) | > loader_time: 0.0045 (0.004663012890304162)  --> TIME: 2025-02-27 11:54:34 -- STEP: 25590/36938 -- GLOBAL_STEP: 1687800 | > loss_text_ce: 0.02187558449804783 (0.029773519884674134) | > loss_mel_ce: 1.326940894126892 (nan) | > loss: 0.005352446809411049 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13707449939298433) | > loader_time: 0.0045 (0.0046629667002598485)  --> TIME: 2025-02-27 11:54:43 -- STEP: 25640/36938 -- GLOBAL_STEP: 1687850 | > loss_text_ce: 0.01860390417277813 (0.029773031950778635) | > loss_mel_ce: 2.6813151836395264 (nan) | > loss: 0.010713964700698853 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13707815748109065) | > loader_time: 0.0045 (0.0046628599977716415)  --> TIME: 2025-02-27 11:54:53 -- STEP: 25690/36938 -- GLOBAL_STEP: 1687900 | > loss_text_ce: 0.03555532917380333 (0.029774682020203724) | > loss_mel_ce: 3.0901196002960205 (nan) | > loss: 0.012403472326695919 (nan) | > current_lr: 5e-06 | > step_time: 0.1814 (0.13708457418711253) | > loader_time: 0.0048 (0.004662792215833589)  --> TIME: 2025-02-27 11:55:02 -- STEP: 25740/36938 -- GLOBAL_STEP: 1687950 | > loss_text_ce: 0.03159001097083092 (0.02977526868095083) | > loss_mel_ce: 2.9195032119750977 (nan) | > loss: 0.011710687540471554 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13709359499680113) | > loader_time: 0.0047 (0.004662716083037541)  --> TIME: 2025-02-27 11:55:11 -- STEP: 25790/36938 -- GLOBAL_STEP: 1688000 | > loss_text_ce: 0.0338791199028492 (0.029776919469997134) | > loss_mel_ce: 2.748187780380249 (nan) | > loss: 0.011039948090910912 (nan) | > current_lr: 5e-06 | > step_time: 0.1563 (0.1370904244485405) | > loader_time: 0.0046 (0.00466277183327114)  --> TIME: 2025-02-27 11:55:20 -- STEP: 25840/36938 -- GLOBAL_STEP: 1688050 | > loss_text_ce: 0.01774514466524124 (0.029777287340639374) | > loss_mel_ce: 2.93335223197937 (nan) | > loss: 0.011710705235600471 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.1370820536978841) | > loader_time: 0.0046 (0.004662772911620946)  --> TIME: 2025-02-27 11:55:30 -- STEP: 25890/36938 -- GLOBAL_STEP: 1688100 | > loss_text_ce: 0.029216259717941284 (0.029778098303849136) | > loss_mel_ce: 3.307443618774414 (nan) | > loss: 0.013240714557468891 (nan) | > current_lr: 5e-06 | > step_time: 0.188 (0.13709034029237516) | > loader_time: 0.0048 (0.0046628458889445105)  --> TIME: 2025-02-27 11:55:39 -- STEP: 25940/36938 -- GLOBAL_STEP: 1688150 | > loss_text_ce: 0.035934094339609146 (0.029779738733658412) | > loss_mel_ce: 2.2907557487487793 (nan) | > loss: 0.009232897311449051 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13708717873579881) | > loader_time: 0.0045 (0.0046627365357157025)  --> TIME: 2025-02-27 11:55:48 -- STEP: 25990/36938 -- GLOBAL_STEP: 1688200 | > loss_text_ce: 0.03351615369319916 (0.02978217360757439) | > loss_mel_ce: 2.94254469871521 (nan) | > loss: 0.011809766292572021 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13707989006145177) | > loader_time: 0.0045 (0.004662924869650367)  --> TIME: 2025-02-27 11:55:58 -- STEP: 26040/36938 -- GLOBAL_STEP: 1688250 | > loss_text_ce: 0.030798325315117836 (0.029781161771783943) | > loss_mel_ce: 2.6543729305267334 (nan) | > loss: 0.010655442252755165 (nan) | > current_lr: 5e-06 | > step_time: 0.1852 (0.13708219866049426) | > loader_time: 0.0049 (0.004662974126144851)  --> TIME: 2025-02-27 11:56:07 -- STEP: 26090/36938 -- GLOBAL_STEP: 1688300 | > loss_text_ce: 0.034780632704496384 (0.029780768415215625) | > loss_mel_ce: 2.6311683654785156 (nan) | > loss: 0.010579164139926434 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13708781167984715) | > loader_time: 0.0046 (0.004663002586949594)  --> TIME: 2025-02-27 11:56:16 -- STEP: 26140/36938 -- GLOBAL_STEP: 1688350 | > loss_text_ce: 0.031046956777572632 (0.029780170287047658) | > loss_mel_ce: 2.886139154434204 (nan) | > loss: 0.011576135642826557 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.13708251915364722) | > loader_time: 0.0046 (0.004662985152293453)  --> TIME: 2025-02-27 11:56:26 -- STEP: 26190/36938 -- GLOBAL_STEP: 1688400 | > loss_text_ce: 0.028906285762786865 (0.029780192588572982) | > loss_mel_ce: 2.6182422637939453 (nan) | > loss: 0.010504558682441711 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13709659228665325) | > loader_time: 0.0045 (0.004662883004053446)  --> TIME: 2025-02-27 11:56:36 -- STEP: 26240/36938 -- GLOBAL_STEP: 1688450 | > loss_text_ce: 0.029796604067087173 (0.029782102828016292) | > loss_mel_ce: 2.979726552963257 (nan) | > loss: 0.011942553333938122 (nan) | > current_lr: 5e-06 | > step_time: 0.1853 (0.13710375284639761) | > loader_time: 0.0048 (0.004662795691955383)  --> TIME: 2025-02-27 11:56:45 -- STEP: 26290/36938 -- GLOBAL_STEP: 1688500 | > loss_text_ce: 0.0281838346272707 (0.029785037110143412) | > loss_mel_ce: 2.7675671577453613 (nan) | > loss: 0.011094250716269016 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13709749889809042) | > loader_time: 0.0046 (0.004662693721253235)  --> TIME: 2025-02-27 11:56:54 -- STEP: 26340/36938 -- GLOBAL_STEP: 1688550 | > loss_text_ce: 0.02779562398791313 (0.029784235441010675) | > loss_mel_ce: 3.0701358318328857 (nan) | > loss: 0.012293378822505474 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13709206467970944) | > loader_time: 0.0046 (0.0046626307879240134)  --> TIME: 2025-02-27 11:57:03 -- STEP: 26390/36938 -- GLOBAL_STEP: 1688600 | > loss_text_ce: 0.026980698108673096 (0.029782353081804328) | > loss_mel_ce: 1.3538951873779297 (nan) | > loss: 0.005479666404426098 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13709104145146855) | > loader_time: 0.0046 (0.0046626196796580605)  --> TIME: 2025-02-27 11:57:13 -- STEP: 26440/36938 -- GLOBAL_STEP: 1688650 | > loss_text_ce: 0.03175060823559761 (0.02978453385032878) | > loss_mel_ce: 2.6746907234191895 (nan) | > loss: 0.010739847086369991 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.13708820388104248) | > loader_time: 0.0046 (0.004662525022625022)  --> TIME: 2025-02-27 11:57:22 -- STEP: 26490/36938 -- GLOBAL_STEP: 1688700 | > loss_text_ce: 0.02826613560318947 (0.02978261469611277) | > loss_mel_ce: 2.395014762878418 (nan) | > loss: 0.009616195224225521 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13708132943282575) | > loader_time: 0.0046 (0.004662569156904501)  --> TIME: 2025-02-27 11:57:31 -- STEP: 26540/36938 -- GLOBAL_STEP: 1688750 | > loss_text_ce: 0.029942933470010757 (0.029782793487189877) | > loss_mel_ce: 2.103492498397827 (nan) | > loss: 0.008466014638543129 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13707605267074058) | > loader_time: 0.0045 (0.004662586013086781)  --> TIME: 2025-02-27 11:57:40 -- STEP: 26590/36938 -- GLOBAL_STEP: 1688800 | > loss_text_ce: 0.02759493701159954 (0.029781325843399552) | > loss_mel_ce: 1.845015287399292 (nan) | > loss: 0.007430993486195803 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13706597460531159) | > loader_time: 0.0046 (0.004662450922750459)  --> TIME: 2025-02-27 11:57:49 -- STEP: 26640/36938 -- GLOBAL_STEP: 1688850 | > loss_text_ce: 0.03304590657353401 (0.02978029960518922) | > loss_mel_ce: 2.190662145614624 (nan) | > loss: 0.008824239484965801 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.1370561158245385) | > loader_time: 0.0048 (0.004662433051848208)  --> TIME: 2025-02-27 11:57:59 -- STEP: 26690/36938 -- GLOBAL_STEP: 1688900 | > loss_text_ce: 0.043099451810121536 (0.029781359413819476) | > loss_mel_ce: 2.865424394607544 (nan) | > loss: 0.011541761457920074 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13706480776964342) | > loader_time: 0.0046 (0.004662397444672522)  --> TIME: 2025-02-27 11:58:08 -- STEP: 26740/36938 -- GLOBAL_STEP: 1688950 | > loss_text_ce: 0.031083470210433006 (0.029779596640620458) | > loss_mel_ce: 2.6012189388275146 (nan) | > loss: 0.010445645079016685 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13705347814752475) | > loader_time: 0.0046 (0.00466231824572442)  --> TIME: 2025-02-27 11:58:17 -- STEP: 26790/36938 -- GLOBAL_STEP: 1689000 | > loss_text_ce: 0.030862603336572647 (0.029779487020903102) | > loss_mel_ce: 2.8095505237579346 (nan) | > loss: 0.011271481402218342 (nan) | > current_lr: 5e-06 | > step_time: 0.1266 (0.1370507928496552) | > loader_time: 0.0046 (0.004662304157728407)  --> TIME: 2025-02-27 11:58:26 -- STEP: 26840/36938 -- GLOBAL_STEP: 1689050 | > loss_text_ce: 0.036244478076696396 (0.029778603972775538) | > loss_mel_ce: 2.846768379211426 (nan) | > loss: 0.011440527625381947 (nan) | > current_lr: 5e-06 | > step_time: 0.1123 (0.1370519114322345) | > loader_time: 0.0046 (0.004662270339874862)  --> TIME: 2025-02-27 11:58:36 -- STEP: 26890/36938 -- GLOBAL_STEP: 1689100 | > loss_text_ce: 0.025941452011466026 (0.029777401013666268) | > loss_mel_ce: 2.0678815841674805 (nan) | > loss: 0.008308821357786655 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.13705510364318213) | > loader_time: 0.0046 (0.004662348905785371)  --> TIME: 2025-02-27 11:58:45 -- STEP: 26940/36938 -- GLOBAL_STEP: 1689150 | > loss_text_ce: 0.031598567962646484 (0.02977589287047719) | > loss_mel_ce: 2.552182674407959 (nan) | > loss: 0.010253100655972958 (nan) | > current_lr: 5e-06 | > step_time: 0.1514 (0.13704608154013628) | > loader_time: 0.0047 (0.004662179663699917)  --> TIME: 2025-02-27 11:58:54 -- STEP: 26990/36938 -- GLOBAL_STEP: 1689200 | > loss_text_ce: 0.03553847223520279 (0.029774882706664115) | > loss_mel_ce: 2.0357491970062256 (nan) | > loss: 0.008219395764172077 (nan) | > current_lr: 5e-06 | > step_time: 0.1084 (0.13704406194838778) | > loader_time: 0.0045 (0.004662130514132886)  --> TIME: 2025-02-27 11:59:03 -- STEP: 27040/36938 -- GLOBAL_STEP: 1689250 | > loss_text_ce: 0.028278762474656105 (0.029775962183739693) | > loss_mel_ce: 2.8002049922943115 (nan) | > loss: 0.011224143207073212 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.1370421922418489) | > loader_time: 0.0047 (0.004662295946708095)  --> TIME: 2025-02-27 11:59:13 -- STEP: 27090/36938 -- GLOBAL_STEP: 1689300 | > loss_text_ce: 0.030907634645700455 (0.029774875454700898) | > loss_mel_ce: 2.7389075756073 (nan) | > loss: 0.010991331189870834 (nan) | > current_lr: 5e-06 | > step_time: 0.1244 (0.13704598082876482) | > loader_time: 0.0046 (0.004662521336957862)  --> TIME: 2025-02-27 11:59:22 -- STEP: 27140/36938 -- GLOBAL_STEP: 1689350 | > loss_text_ce: 0.03234587609767914 (0.02977601042238334) | > loss_mel_ce: 2.2634143829345703 (nan) | > loss: 0.009110160171985626 (nan) | > current_lr: 5e-06 | > step_time: 0.162 (0.1370459990371387) | > loader_time: 0.0047 (0.004662612675742002)  --> TIME: 2025-02-27 11:59:32 -- STEP: 27190/36938 -- GLOBAL_STEP: 1689400 | > loss_text_ce: 0.03829425200819969 (0.029776201859656383) | > loss_mel_ce: 2.624345541000366 (nan) | > loss: 0.01056603156030178 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13704564489832488) | > loader_time: 0.0045 (0.0046625331553872655)  --> TIME: 2025-02-27 11:59:41 -- STEP: 27240/36938 -- GLOBAL_STEP: 1689450 | > loss_text_ce: 0.03530222177505493 (0.029775384486900456) | > loss_mel_ce: 2.893272638320923 (nan) | > loss: 0.011621329002082348 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.13704114319994548) | > loader_time: 0.0047 (0.004662424343448022)  --> TIME: 2025-02-27 11:59:50 -- STEP: 27290/36938 -- GLOBAL_STEP: 1689500 | > loss_text_ce: 0.023389272391796112 (0.02977157349136177) | > loss_mel_ce: 2.4984378814697266 (nan) | > loss: 0.010007251054048538 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13703279674337082) | > loader_time: 0.0046 (0.0046624199030992465)  --> TIME: 2025-02-27 11:59:59 -- STEP: 27340/36938 -- GLOBAL_STEP: 1689550 | > loss_text_ce: 0.03637499362230301 (0.029775067750601642) | > loss_mel_ce: 2.5753014087677 (nan) | > loss: 0.010363795794546604 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13701811570186326) | > loader_time: 0.0046 (0.004662310431800622)  --> TIME: 2025-02-27 12:00:08 -- STEP: 27390/36938 -- GLOBAL_STEP: 1689600 | > loss_text_ce: 0.027198251336812973 (0.0297755101057094) | > loss_mel_ce: 2.527592182159424 (nan) | > loss: 0.010138058103621006 (nan) | > current_lr: 5e-06 | > step_time: 0.1532 (0.1370220154208302) | > loader_time: 0.0046 (0.004662364928032108)  --> TIME: 2025-02-27 12:00:17 -- STEP: 27440/36938 -- GLOBAL_STEP: 1689650 | > loss_text_ce: 0.03350455313920975 (0.029776561492375238) | > loss_mel_ce: 2.6496126651763916 (nan) | > loss: 0.01064729131758213 (nan) | > current_lr: 5e-06 | > step_time: 0.1304 (0.13701835670902096) | > loader_time: 0.0046 (0.004662343572944664)  --> TIME: 2025-02-27 12:00:27 -- STEP: 27490/36938 -- GLOBAL_STEP: 1689700 | > loss_text_ce: 0.030873695388436317 (0.02977838982066033) | > loss_mel_ce: 0.0030497994739562273 (nan) | > loss: 0.00013461704656947404 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1370247437070173) | > loader_time: 0.0045 (0.004662349146901588)  --> TIME: 2025-02-27 12:00:36 -- STEP: 27540/36938 -- GLOBAL_STEP: 1689750 | > loss_text_ce: 0.02934509702026844 (0.029779569474546037) | > loss_mel_ce: 2.6925809383392334 (nan) | > loss: 0.010801293887197971 (nan) | > current_lr: 5e-06 | > step_time: 0.1444 (0.1370188005003443) | > loader_time: 0.0046 (0.004662267661389012)  --> TIME: 2025-02-27 12:00:46 -- STEP: 27590/36938 -- GLOBAL_STEP: 1689800 | > loss_text_ce: 0.02335558645427227 (0.029779392624177368) | > loss_mel_ce: 2.522645950317383 (nan) | > loss: 0.010103181004524231 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13701779956444268) | > loader_time: 0.0045 (0.004662190238908394)  --> TIME: 2025-02-27 12:00:54 -- STEP: 27640/36938 -- GLOBAL_STEP: 1689850 | > loss_text_ce: 0.024487171322107315 (0.029779074860025106) | > loss_mel_ce: 0.0967305526137352 (nan) | > loss: 0.0004810227546840906 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13700443661506156) | > loader_time: 0.0045 (0.00466206229026342)  --> TIME: 2025-02-27 12:01:04 -- STEP: 27690/36938 -- GLOBAL_STEP: 1689900 | > loss_text_ce: 0.023122020065784454 (0.029780384575838485) | > loss_mel_ce: 2.312131643295288 (nan) | > loss: 0.009266880340874195 (nan) | > current_lr: 5e-06 | > step_time: 0.1263 (0.13701055613869362) | > loader_time: 0.0046 (0.0046620264773163935)  --> TIME: 2025-02-27 12:01:13 -- STEP: 27740/36938 -- GLOBAL_STEP: 1689950 | > loss_text_ce: 0.029289186000823975 (0.029781229352436184) | > loss_mel_ce: 2.164668321609497 (nan) | > loss: 0.008706181310117245 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13701044732749024) | > loader_time: 0.0045 (0.004662018992871628)  --> TIME: 2025-02-27 12:01:23 -- STEP: 27790/36938 -- GLOBAL_STEP: 1690000 | > loss_text_ce: 0.023238549008965492 (0.029778492179762594) | > loss_mel_ce: 0.6305182576179504 (nan) | > loss: 0.0025942730717360973 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13700839856323765) | > loader_time: 0.0045 (0.004661951600411421) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1690000.pth  --> TIME: 2025-02-27 12:01:38 -- STEP: 27840/36938 -- GLOBAL_STEP: 1690050 | > loss_text_ce: 0.027666352689266205 (0.029778865659937287) | > loss_mel_ce: 2.396134376525879 (nan) | > loss: 0.00961825717240572 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13700997074102492) | > loader_time: 0.0047 (0.0046621581518101985)  --> TIME: 2025-02-27 12:01:47 -- STEP: 27890/36938 -- GLOBAL_STEP: 1690100 | > loss_text_ce: 0.0274067185819149 (0.029780849524132194) | > loss_mel_ce: 2.631563663482666 (nan) | > loss: 0.010551470331847668 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13700590460220893) | > loader_time: 0.0046 (0.004662110746526601)  --> TIME: 2025-02-27 12:01:57 -- STEP: 27940/36938 -- GLOBAL_STEP: 1690150 | > loss_text_ce: 0.022362705320119858 (0.029781472349323414) | > loss_mel_ce: 2.6585936546325684 (nan) | > loss: 0.010638716630637646 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13701052762989283) | > loader_time: 0.0045 (0.0046622078420438355)  --> TIME: 2025-02-27 12:02:06 -- STEP: 27990/36938 -- GLOBAL_STEP: 1690200 | > loss_text_ce: 0.0333714485168457 (0.0297814186389261) | > loss_mel_ce: 2.985901355743408 (nan) | > loss: 0.011981242336332798 (nan) | > current_lr: 5e-06 | > step_time: 0.1966 (0.13702002509486091) | > loader_time: 0.0048 (0.004662404276720753)  --> TIME: 2025-02-27 12:02:16 -- STEP: 28040/36938 -- GLOBAL_STEP: 1690250 | > loss_text_ce: 0.02734709531068802 (0.029780307846421522) | > loss_mel_ce: 2.5649502277374268 (nan) | > loss: 0.010286894626915455 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13701774222874577) | > loader_time: 0.0046 (0.004662494227481468)  --> TIME: 2025-02-27 12:02:25 -- STEP: 28090/36938 -- GLOBAL_STEP: 1690300 | > loss_text_ce: 0.038701049983501434 (0.029777820404067293) | > loss_mel_ce: 0.00372314453125 (nan) | > loss: 0.00016834998677950352 (nan) | > current_lr: 5e-06 | > step_time: 0.1267 (0.13700538280808877) | > loader_time: 0.0046 (0.004662507562375911)  --> TIME: 2025-02-27 12:02:34 -- STEP: 28140/36938 -- GLOBAL_STEP: 1690350 | > loss_text_ce: 0.020976044237613678 (0.029776711960097454) | > loss_mel_ce: 1.0446056127548218 (nan) | > loss: 0.004228498786687851 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13700056644560296) | > loader_time: 0.0045 (0.004662613861935312)  --> TIME: 2025-02-27 12:02:43 -- STEP: 28190/36938 -- GLOBAL_STEP: 1690400 | > loss_text_ce: 0.036963362246751785 (0.029776496402806978) | > loss_mel_ce: 2.891000747680664 (nan) | > loss: 0.011618906632065773 (nan) | > current_lr: 5e-06 | > step_time: 0.1812 (0.1369938616297602) | > loader_time: 0.0048 (0.004662524364061415)  --> TIME: 2025-02-27 12:02:52 -- STEP: 28240/36938 -- GLOBAL_STEP: 1690450 | > loss_text_ce: 0.04133880138397217 (0.029777605222536807) | > loss_mel_ce: 2.7335381507873535 (nan) | > loss: 0.011011417955160141 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13699456685494385) | > loader_time: 0.0046 (0.004662525662281697)  --> TIME: 2025-02-27 12:03:02 -- STEP: 28290/36938 -- GLOBAL_STEP: 1690500 | > loss_text_ce: 0.030339118093252182 (0.029776786552717774) | > loss_mel_ce: 2.755408525466919 (nan) | > loss: 0.011054554022848606 (nan) | > current_lr: 5e-06 | > step_time: 0.1704 (0.13699317528780916) | > loader_time: 0.0056 (0.004662865680534119)  --> TIME: 2025-02-27 12:03:11 -- STEP: 28340/36938 -- GLOBAL_STEP: 1690550 | > loss_text_ce: 0.026532066985964775 (0.02977598918839796) | > loss_mel_ce: 2.1538679599761963 (nan) | > loss: 0.008652381598949432 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13700025148936837) | > loader_time: 0.0045 (0.00466317979062874)  --> TIME: 2025-02-27 12:03:21 -- STEP: 28390/36938 -- GLOBAL_STEP: 1690600 | > loss_text_ce: 0.01718371920287609 (0.029778009603490887) | > loss_mel_ce: 2.2222840785980225 (nan) | > loss: 0.00888677779585123 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.137002251794028) | > loader_time: 0.0045 (0.004663387987554055)  --> TIME: 2025-02-27 12:03:30 -- STEP: 28440/36938 -- GLOBAL_STEP: 1690650 | > loss_text_ce: 0.020766062662005424 (0.02977993496577093) | > loss_mel_ce: 2.067959785461426 (nan) | > loss: 0.008288594894111156 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1370077173501961) | > loader_time: 0.0045 (0.004663309626773777)  --> TIME: 2025-02-27 12:03:40 -- STEP: 28490/36938 -- GLOBAL_STEP: 1690700 | > loss_text_ce: 0.024748291820287704 (0.02977946500770197) | > loss_mel_ce: 2.1015563011169434 (nan) | > loss: 0.008437717333436012 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.1370123514588479) | > loader_time: 0.0046 (0.004663317903965222)  --> TIME: 2025-02-27 12:03:49 -- STEP: 28540/36938 -- GLOBAL_STEP: 1690750 | > loss_text_ce: 0.0302397720515728 (0.029779922569362485) | > loss_mel_ce: 2.2493174076080322 (nan) | > loss: 0.009045862592756748 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13700830166207834) | > loader_time: 0.0046 (0.004663279236992007)  --> TIME: 2025-02-27 12:03:58 -- STEP: 28590/36938 -- GLOBAL_STEP: 1690800 | > loss_text_ce: 0.03894374147057533 (0.02977957738616843) | > loss_mel_ce: 0.0989786759018898 (nan) | > loss: 0.000547311210539192 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13701200631999885) | > loader_time: 0.0045 (0.004663405096501729)  --> TIME: 2025-02-27 12:04:08 -- STEP: 28640/36938 -- GLOBAL_STEP: 1690850 | > loss_text_ce: 0.026856103911995888 (0.029780800054539217) | > loss_mel_ce: 2.403031587600708 (nan) | > loss: 0.009642411954700947 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.1370147941415531) | > loader_time: 0.0046 (0.004663563765294061)  --> TIME: 2025-02-27 12:04:17 -- STEP: 28690/36938 -- GLOBAL_STEP: 1690900 | > loss_text_ce: 0.028779029846191406 (0.029779907253967872) | > loss_mel_ce: 2.4722139835357666 (nan) | > loss: 0.009924575686454773 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13701999216555005) | > loader_time: 0.0046 (0.0046634749445444885)  --> TIME: 2025-02-27 12:04:27 -- STEP: 28740/36938 -- GLOBAL_STEP: 1690950 | > loss_text_ce: 0.02763678878545761 (0.029781200043227155) | > loss_mel_ce: 3.131845235824585 (nan) | > loss: 0.01253762748092413 (nan) | > current_lr: 5e-06 | > step_time: 0.1774 (0.1370201085537937) | > loader_time: 0.0048 (0.004663570630691414)  --> TIME: 2025-02-27 12:04:36 -- STEP: 28790/36938 -- GLOBAL_STEP: 1691000 | > loss_text_ce: 0.021926188841462135 (0.029780724185178298) | > loss_mel_ce: 2.7267537117004395 (nan) | > loss: 0.010907460004091263 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.13701551623872726) | > loader_time: 0.0046 (0.00466362337719943)  --> TIME: 2025-02-27 12:04:45 -- STEP: 28840/36938 -- GLOBAL_STEP: 1691050 | > loss_text_ce: 0.039209216833114624 (0.02978250480847287) | > loss_mel_ce: 0.03940603509545326 (nan) | > loss: 0.0003119652974419296 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13700866270164472) | > loader_time: 0.0046 (0.0046637761774737715)  --> TIME: 2025-02-27 12:04:54 -- STEP: 28890/36938 -- GLOBAL_STEP: 1691100 | > loss_text_ce: 0.028607498854398727 (0.02978202518283311) | > loss_mel_ce: 1.0050599575042725 (nan) | > loss: 0.004101855214685202 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13699553319845822) | > loader_time: 0.0045 (0.004664049597502829)  --> TIME: 2025-02-27 12:05:03 -- STEP: 28940/36938 -- GLOBAL_STEP: 1691150 | > loss_text_ce: 0.036422014236450195 (0.029782827204812918) | > loss_mel_ce: 0.7025728821754456 (nan) | > loss: 0.0029325196519494057 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13699006516931744) | > loader_time: 0.0046 (0.004664043895771527)  --> TIME: 2025-02-27 12:05:12 -- STEP: 28990/36938 -- GLOBAL_STEP: 1691200 | > loss_text_ce: 0.03482762724161148 (0.029782713762516306) | > loss_mel_ce: 2.479571580886841 (nan) | > loss: 0.009977775625884533 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.13698710348327253) | > loader_time: 0.0047 (0.004664132701154003)  --> TIME: 2025-02-27 12:05:22 -- STEP: 29040/36938 -- GLOBAL_STEP: 1691250 | > loss_text_ce: 0.030713723972439766 (0.029782468983140267) | > loss_mel_ce: 3.3554787635803223 (nan) | > loss: 0.013437272980809212 (nan) | > current_lr: 5e-06 | > step_time: 0.1915 (0.13698686533722945) | > loader_time: 0.0049 (0.00466417841839068)  --> TIME: 2025-02-27 12:05:31 -- STEP: 29090/36938 -- GLOBAL_STEP: 1691300 | > loss_text_ce: 0.04128563776612282 (0.029784669654828694) | > loss_mel_ce: 2.740612030029297 (nan) | > loss: 0.011039277538657188 (nan) | > current_lr: 5e-06 | > step_time: 0.1822 (0.13698384179082146) | > loader_time: 0.0057 (0.004664251606830745)  --> TIME: 2025-02-27 12:05:40 -- STEP: 29140/36938 -- GLOBAL_STEP: 1691350 | > loss_text_ce: 0.0442780964076519 (0.029784260756433964) | > loss_mel_ce: 1.8818633556365967 (nan) | > loss: 0.0076434193179011345 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13698613945555776) | > loader_time: 0.0046 (0.004664397231503229)  --> TIME: 2025-02-27 12:05:50 -- STEP: 29190/36938 -- GLOBAL_STEP: 1691400 | > loss_text_ce: 0.03110925853252411 (0.029785939929072042) | > loss_mel_ce: 2.979667901992798 (nan) | > loss: 0.011947529390454292 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.13698684757561938) | > loader_time: 0.0046 (0.004664546645394171)  --> TIME: 2025-02-27 12:05:59 -- STEP: 29240/36938 -- GLOBAL_STEP: 1691450 | > loss_text_ce: 0.031912803649902344 (0.029786469811948203) | > loss_mel_ce: 2.8960816860198975 (nan) | > loss: 0.011619026772677898 (nan) | > current_lr: 5e-06 | > step_time: 0.1748 (0.13697674936537166) | > loader_time: 0.0049 (0.0046645781165434805)  --> TIME: 2025-02-27 12:06:08 -- STEP: 29290/36938 -- GLOBAL_STEP: 1691500 | > loss_text_ce: 0.03241853788495064 (0.029784278249824048) | > loss_mel_ce: 3.431425094604492 (nan) | > loss: 0.013745412230491638 (nan) | > current_lr: 5e-06 | > step_time: 0.1833 (0.1369714325790556) | > loader_time: 0.005 (0.004664790365792821)  --> TIME: 2025-02-27 12:06:17 -- STEP: 29340/36938 -- GLOBAL_STEP: 1691550 | > loss_text_ce: 0.02731923572719097 (0.029784954403829916) | > loss_mel_ce: 2.839934825897217 (nan) | > loss: 0.011377993039786816 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.13697946801344826) | > loader_time: 0.0047 (0.0046648029169422)  --> TIME: 2025-02-27 12:06:27 -- STEP: 29390/36938 -- GLOBAL_STEP: 1691600 | > loss_text_ce: 0.027850564569234848 (0.029786399967636382) | > loss_mel_ce: 2.431868076324463 (nan) | > loss: 0.009760788641870022 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13697889228384635) | > loader_time: 0.0046 (0.0046648150603355075)  --> TIME: 2025-02-27 12:06:36 -- STEP: 29440/36938 -- GLOBAL_STEP: 1691650 | > loss_text_ce: 0.029650326818227768 (0.029786723539417966) | > loss_mel_ce: 2.895385503768921 (nan) | > loss: 0.011607285588979721 (nan) | > current_lr: 5e-06 | > step_time: 0.1376 (0.13698271964069245) | > loader_time: 0.0047 (0.004664930523089743)  --> TIME: 2025-02-27 12:06:45 -- STEP: 29490/36938 -- GLOBAL_STEP: 1691700 | > loss_text_ce: 0.022280611097812653 (0.029785965718537397) | > loss_mel_ce: 2.5376322269439697 (nan) | > loss: 0.010158385150134563 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13697826704683003) | > loader_time: 0.0053 (0.004665073138974652)  --> TIME: 2025-02-27 12:06:55 -- STEP: 29540/36938 -- GLOBAL_STEP: 1691750 | > loss_text_ce: 0.02093045972287655 (0.029785830533628842) | > loss_mel_ce: 1.848046898841858 (nan) | > loss: 0.007416577078402042 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.13698088845128326) | > loader_time: 0.0045 (0.0046651429232505036)  --> TIME: 2025-02-27 12:07:04 -- STEP: 29590/36938 -- GLOBAL_STEP: 1691800 | > loss_text_ce: 0.034271374344825745 (0.02978661668475249) | > loss_mel_ce: 2.911633014678955 (nan) | > loss: 0.011690097860991955 (nan) | > current_lr: 5e-06 | > step_time: 0.1374 (0.13697879304753682) | > loader_time: 0.0046 (0.004665226394882797)  --> TIME: 2025-02-27 12:07:13 -- STEP: 29640/36938 -- GLOBAL_STEP: 1691850 | > loss_text_ce: 0.025784023106098175 (0.02978747240704449) | > loss_mel_ce: 2.4898593425750732 (nan) | > loss: 0.00998271256685257 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13697689884265546) | > loader_time: 0.0047 (0.004665358290176805)  --> TIME: 2025-02-27 12:07:23 -- STEP: 29690/36938 -- GLOBAL_STEP: 1691900 | > loss_text_ce: 0.026369089260697365 (0.0297882751929337) | > loss_mel_ce: 2.6067395210266113 (nan) | > loss: 0.01044884417206049 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13697787680261292) | > loader_time: 0.0045 (0.004665308899653004)  --> TIME: 2025-02-27 12:07:32 -- STEP: 29740/36938 -- GLOBAL_STEP: 1691950 | > loss_text_ce: 0.030938107520341873 (0.029790227483956766) | > loss_mel_ce: 1.1789474487304688 (nan) | > loss: 0.004801133647561073 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13697728506253676) | > loader_time: 0.0045 (0.004665233813121005)  --> TIME: 2025-02-27 12:07:41 -- STEP: 29790/36938 -- GLOBAL_STEP: 1692000 | > loss_text_ce: 0.03900502622127533 (0.029790023351747824) | > loss_mel_ce: 2.5465166568756104 (nan) | > loss: 0.010260007344186306 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13697090986071866) | > loader_time: 0.0046 (0.004665246110668357)  --> TIME: 2025-02-27 12:07:51 -- STEP: 29840/36938 -- GLOBAL_STEP: 1692050 | > loss_text_ce: 0.030188625678420067 (0.029789432871371428) | > loss_mel_ce: 2.0236847400665283 (nan) | > loss: 0.008150291629135609 (nan) | > current_lr: 5e-06 | > step_time: 0.1346 (0.1369751286490666) | > loader_time: 0.0046 (0.004665394818495172)  --> TIME: 2025-02-27 12:08:00 -- STEP: 29890/36938 -- GLOBAL_STEP: 1692100 | > loss_text_ce: 0.02584581822156906 (0.02978930268723328) | > loss_mel_ce: 2.4998114109039307 (nan) | > loss: 0.010022449307143688 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13697489995787251) | > loader_time: 0.0046 (0.004665439477450238)  --> TIME: 2025-02-27 12:08:09 -- STEP: 29940/36938 -- GLOBAL_STEP: 1692150 | > loss_text_ce: 0.03122771717607975 (0.02978782858491149) | > loss_mel_ce: 2.869999885559082 (nan) | > loss: 0.011512809433043003 (nan) | > current_lr: 5e-06 | > step_time: 0.1424 (0.13697627152931485) | > loader_time: 0.0047 (0.004665366259111752)  --> TIME: 2025-02-27 12:08:19 -- STEP: 29990/36938 -- GLOBAL_STEP: 1692200 | > loss_text_ce: 0.03951873257756233 (0.02978598583367303) | > loss_mel_ce: 0.9366862177848816 (nan) | > loss: 0.0038738292641937733 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13697458880469904) | > loader_time: 0.0046 (0.004665420348742043)  --> TIME: 2025-02-27 12:08:28 -- STEP: 30040/36938 -- GLOBAL_STEP: 1692250 | > loss_text_ce: 0.029104292392730713 (0.029786801170687962) | > loss_mel_ce: 2.6953463554382324 (nan) | > loss: 0.01081131212413311 (nan) | > current_lr: 5e-06 | > step_time: 0.1703 (0.1369699099886109) | > loader_time: 0.0048 (0.004665566816152176)  --> TIME: 2025-02-27 12:08:37 -- STEP: 30090/36938 -- GLOBAL_STEP: 1692300 | > loss_text_ce: 0.02477286569774151 (0.02978673949645788) | > loss_mel_ce: 1.9116233587265015 (nan) | > loss: 0.007684112526476383 (nan) | > current_lr: 5e-06 | > step_time: 0.1604 (0.136966775484994) | > loader_time: 0.0047 (0.004665507347966479)  --> TIME: 2025-02-27 12:08:46 -- STEP: 30140/36938 -- GLOBAL_STEP: 1692350 | > loss_text_ce: 0.028781430795788765 (0.029786717774428057) | > loss_mel_ce: 2.6494245529174805 (nan) | > loss: 0.010627802461385727 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.1369622194221176) | > loader_time: 0.0045 (0.004665613301004413)  --> TIME: 2025-02-27 12:08:55 -- STEP: 30190/36938 -- GLOBAL_STEP: 1692400 | > loss_text_ce: 0.029164515435695648 (0.029786453390158982) | > loss_mel_ce: 2.789048433303833 (nan) | > loss: 0.011183385737240314 (nan) | > current_lr: 5e-06 | > step_time: 0.1597 (0.13695259046064864) | > loader_time: 0.0047 (0.004665754093323059)  --> TIME: 2025-02-27 12:09:04 -- STEP: 30240/36938 -- GLOBAL_STEP: 1692450 | > loss_text_ce: 0.029164256528019905 (0.029786036675426352) | > loss_mel_ce: 2.223472833633423 (nan) | > loss: 0.008939037099480629 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.1369443933799777) | > loader_time: 0.0046 (0.004665689335929025)  --> TIME: 2025-02-27 12:09:14 -- STEP: 30290/36938 -- GLOBAL_STEP: 1692500 | > loss_text_ce: 0.0365782268345356 (0.02978538464353267) | > loss_mel_ce: 2.860917091369629 (nan) | > loss: 0.011497997678816319 (nan) | > current_lr: 5e-06 | > step_time: 0.1728 (0.1369507154170017) | > loader_time: 0.0055 (0.004665767009126823)  --> TIME: 2025-02-27 12:09:23 -- STEP: 30340/36938 -- GLOBAL_STEP: 1692550 | > loss_text_ce: 0.03257317841053009 (0.029787494409700466) | > loss_mel_ce: 3.0914905071258545 (nan) | > loss: 0.012397078797221184 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.1369544119919762) | > loader_time: 0.0046 (0.004665838996281183)  --> TIME: 2025-02-27 12:09:33 -- STEP: 30390/36938 -- GLOBAL_STEP: 1692600 | > loss_text_ce: 0.03398893028497696 (0.029787893825023396) | > loss_mel_ce: 1.4927705526351929 (nan) | > loss: 0.006058569997549057 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13695685621075124) | > loader_time: 0.0046 (0.004665921894724962)  --> TIME: 2025-02-27 12:09:42 -- STEP: 30440/36938 -- GLOBAL_STEP: 1692650 | > loss_text_ce: 0.024089865386486053 (0.029787134643400565) | > loss_mel_ce: 1.659489631652832 (nan) | > loss: 0.006680871360003948 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13695780966190965) | > loader_time: 0.0045 (0.004665961622721891)  --> TIME: 2025-02-27 12:09:51 -- STEP: 30490/36938 -- GLOBAL_STEP: 1692700 | > loss_text_ce: 0.02679871954023838 (0.029787377937779053) | > loss_mel_ce: 2.960282802581787 (nan) | > loss: 0.01185349840670824 (nan) | > current_lr: 5e-06 | > step_time: 0.1751 (0.1369522646631551) | > loader_time: 0.0048 (0.004665943512020445)  --> TIME: 2025-02-27 12:10:01 -- STEP: 30540/36938 -- GLOBAL_STEP: 1692750 | > loss_text_ce: 0.02974558435380459 (0.029789304700128125) | > loss_mel_ce: 2.7417643070220947 (nan) | > loss: 0.010998055338859558 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.13696116701636904) | > loader_time: 0.0048 (0.0046661029395982205)  --> TIME: 2025-02-27 12:10:10 -- STEP: 30590/36938 -- GLOBAL_STEP: 1692800 | > loss_text_ce: 0.031894467771053314 (0.029789137218851423) | > loss_mel_ce: 2.5550787448883057 (nan) | > loss: 0.010265767574310303 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.1369660760343906) | > loader_time: 0.0046 (0.0046661113048153185)  --> TIME: 2025-02-27 12:10:19 -- STEP: 30640/36938 -- GLOBAL_STEP: 1692850 | > loss_text_ce: 0.03662109375 (0.029788442760281635) | > loss_mel_ce: 2.9351208209991455 (nan) | > loss: 0.011792627163231373 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13695282882405255) | > loader_time: 0.0046 (0.004666006199373591)  --> TIME: 2025-02-27 12:10:28 -- STEP: 30690/36938 -- GLOBAL_STEP: 1692900 | > loss_text_ce: 0.03501586988568306 (0.029787106202103866) | > loss_mel_ce: 2.5158472061157227 (nan) | > loss: 0.01012247335165739 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.1369494813472832) | > loader_time: 0.0047 (0.004666060187910622)  --> TIME: 2025-02-27 12:10:38 -- STEP: 30740/36938 -- GLOBAL_STEP: 1692950 | > loss_text_ce: 0.02812754362821579 (0.02978704699358153) | > loss_mel_ce: 2.1289567947387695 (nan) | > loss: 0.008559858426451683 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13694551765879884) | > loader_time: 0.0045 (0.004665970585139915)  --> TIME: 2025-02-27 12:10:47 -- STEP: 30790/36938 -- GLOBAL_STEP: 1693000 | > loss_text_ce: 0.031669244170188904 (0.029787573706722862) | > loss_mel_ce: 2.8504433631896973 (nan) | > loss: 0.011436955071985722 (nan) | > current_lr: 5e-06 | > step_time: 0.1447 (0.13695301356815767) | > loader_time: 0.0047 (0.004666032145340052)  --> TIME: 2025-02-27 12:10:56 -- STEP: 30840/36938 -- GLOBAL_STEP: 1693050 | > loss_text_ce: 0.03432127460837364 (0.029786259326848287) | > loss_mel_ce: 2.730332136154175 (nan) | > loss: 0.010970847681164742 (nan) | > current_lr: 5e-06 | > step_time: 0.1673 (0.13694844882930385) | > loader_time: 0.0049 (0.004666112485098933)  --> TIME: 2025-02-27 12:11:06 -- STEP: 30890/36938 -- GLOBAL_STEP: 1693100 | > loss_text_ce: 0.0348525270819664 (0.02978767763920354) | > loss_mel_ce: 0.8369827270507812 (nan) | > loss: 0.0034596638288348913 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13695274187392356) | > loader_time: 0.0052 (0.004666201278745802)  --> TIME: 2025-02-27 12:11:15 -- STEP: 30940/36938 -- GLOBAL_STEP: 1693150 | > loss_text_ce: 0.03492132946848869 (0.029787541428489384) | > loss_mel_ce: 3.1788790225982666 (nan) | > loss: 0.012753177434206009 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.13695370776466742) | > loader_time: 0.0049 (0.004666247272306517)  --> TIME: 2025-02-27 12:11:24 -- STEP: 30990/36938 -- GLOBAL_STEP: 1693200 | > loss_text_ce: 0.02077743411064148 (0.029788666416894836) | > loss_mel_ce: 0.0383758544921875 (nan) | > loss: 0.00023473528563044965 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1369496025782157) | > loader_time: 0.0046 (0.004666238871263898)  --> TIME: 2025-02-27 12:11:34 -- STEP: 31040/36938 -- GLOBAL_STEP: 1693250 | > loss_text_ce: 0.04388874024152756 (0.02979149602451987) | > loss_mel_ce: 3.3873772621154785 (nan) | > loss: 0.013616136275231838 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13694426018706235) | > loader_time: 0.0046 (0.004666209935220249)  --> TIME: 2025-02-27 12:11:43 -- STEP: 31090/36938 -- GLOBAL_STEP: 1693300 | > loss_text_ce: 0.02717725746333599 (0.029793087490221713) | > loss_mel_ce: 2.0004045963287354 (nan) | > loss: 0.00804596021771431 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13694475990048718) | > loader_time: 0.0046 (0.0046661867363807985)  --> TIME: 2025-02-27 12:11:53 -- STEP: 31140/36938 -- GLOBAL_STEP: 1693350 | > loss_text_ce: 0.025784585624933243 (0.02979466643982359) | > loss_mel_ce: 2.7330479621887207 (nan) | > loss: 0.010947748087346554 (nan) | > current_lr: 5e-06 | > step_time: 0.1654 (0.13695659821203382) | > loader_time: 0.0047 (0.004666220307273597)  --> TIME: 2025-02-27 12:12:03 -- STEP: 31190/36938 -- GLOBAL_STEP: 1693400 | > loss_text_ce: 0.037076786160469055 (0.029796158822277843) | > loss_mel_ce: 2.5000524520874023 (nan) | > loss: 0.010067973285913467 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.1369674581639001) | > loader_time: 0.0047 (0.004666277146102452)  --> TIME: 2025-02-27 12:12:12 -- STEP: 31240/36938 -- GLOBAL_STEP: 1693450 | > loss_text_ce: 0.027547210454940796 (0.02979711769267956) | > loss_mel_ce: 2.4950623512268066 (nan) | > loss: 0.010010356083512306 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13696784283989608) | > loader_time: 0.0046 (0.004666276091039564)  --> TIME: 2025-02-27 12:12:21 -- STEP: 31290/36938 -- GLOBAL_STEP: 1693500 | > loss_text_ce: 0.025384260341525078 (0.0297954546262654) | > loss_mel_ce: 2.559329032897949 (nan) | > loss: 0.010256798937916756 (nan) | > current_lr: 5e-06 | > step_time: 0.1264 (0.13696573010722235) | > loader_time: 0.0046 (0.0046662619411850115)  --> TIME: 2025-02-27 12:12:30 -- STEP: 31340/36938 -- GLOBAL_STEP: 1693550 | > loss_text_ce: 0.037885408848524094 (0.029795671852534955) | > loss_mel_ce: 3.556380271911621 (nan) | > loss: 0.01426296029239893 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13696019742316864) | > loader_time: 0.0045 (0.00466617260606105)  --> TIME: 2025-02-27 12:12:40 -- STEP: 31390/36938 -- GLOBAL_STEP: 1693600 | > loss_text_ce: 0.03509022295475006 (0.029793023384274413) | > loss_mel_ce: 2.5697765350341797 (nan) | > loss: 0.010336773470044136 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13695948314575537) | > loader_time: 0.0046 (0.004666111635607344)  --> TIME: 2025-02-27 12:12:49 -- STEP: 31440/36938 -- GLOBAL_STEP: 1693650 | > loss_text_ce: 0.04675765708088875 (0.02979404492411555) | > loss_mel_ce: 3.2159762382507324 (nan) | > loss: 0.012947357259690762 (nan) | > current_lr: 5e-06 | > step_time: 0.1805 (0.1369635541172118) | > loader_time: 0.0049 (0.004666098595878855)  --> TIME: 2025-02-27 12:12:59 -- STEP: 31490/36938 -- GLOBAL_STEP: 1693700 | > loss_text_ce: 0.02356276661157608 (0.029793939023568798) | > loss_mel_ce: 2.9262521266937256 (nan) | > loss: 0.011705614626407623 (nan) | > current_lr: 5e-06 | > step_time: 0.1924 (0.13696307790737383) | > loader_time: 0.0049 (0.004666100846051641)  --> TIME: 2025-02-27 12:13:08 -- STEP: 31540/36938 -- GLOBAL_STEP: 1693750 | > loss_text_ce: 0.030222242698073387 (0.029795521767319828) | > loss_mel_ce: 2.59503173828125 (nan) | > loss: 0.010417674668133259 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13695989956036286) | > loader_time: 0.0045 (0.004666067908365061)  --> TIME: 2025-02-27 12:13:17 -- STEP: 31590/36938 -- GLOBAL_STEP: 1693800 | > loss_text_ce: 0.02579502761363983 (0.02979621504990361) | > loss_mel_ce: 2.5704147815704346 (nan) | > loss: 0.01030242070555687 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13695617394569526) | > loader_time: 0.0048 (0.004666010863270027)  --> TIME: 2025-02-27 12:13:26 -- STEP: 31640/36938 -- GLOBAL_STEP: 1693850 | > loss_text_ce: 0.032988954335451126 (0.029796528931902603) | > loss_mel_ce: 3.1762123107910156 (nan) | > loss: 0.01273492630571127 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.13695864055762483) | > loader_time: 0.0047 (0.004665951127499608)  --> TIME: 2025-02-27 12:13:36 -- STEP: 31690/36938 -- GLOBAL_STEP: 1693900 | > loss_text_ce: 0.03501509502530098 (0.02979530164476458) | > loss_mel_ce: 2.773092746734619 (nan) | > loss: 0.011143285781145096 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13696262820656957) | > loader_time: 0.0045 (0.0046659723295703545)  --> TIME: 2025-02-27 12:13:45 -- STEP: 31740/36938 -- GLOBAL_STEP: 1693950 | > loss_text_ce: 0.02958758920431137 (0.02979699539903273) | > loss_mel_ce: 2.275634765625 (nan) | > loss: 0.009147708304226398 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.136958374906502) | > loader_time: 0.0045 (0.004665920925200476)  --> TIME: 2025-02-27 12:13:54 -- STEP: 31790/36938 -- GLOBAL_STEP: 1694000 | > loss_text_ce: 0.03070352040231228 (0.029796678862865814) | > loss_mel_ce: 2.58080792427063 (nan) | > loss: 0.01036314107477665 (nan) | > current_lr: 5e-06 | > step_time: 0.1265 (0.1369530331004743) | > loader_time: 0.0045 (0.00466583954084216)  --> TIME: 2025-02-27 12:14:04 -- STEP: 31840/36938 -- GLOBAL_STEP: 1694050 | > loss_text_ce: 0.03624407574534416 (0.029796633198471218) | > loss_mel_ce: 2.0648858547210693 (nan) | > loss: 0.008337818086147308 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13695606453784082) | > loader_time: 0.0045 (0.004665799139432573)  --> TIME: 2025-02-27 12:14:13 -- STEP: 31890/36938 -- GLOBAL_STEP: 1694100 | > loss_text_ce: 0.0338943786919117 (0.029796122783265246) | > loss_mel_ce: 3.366392135620117 (nan) | > loss: 0.01349320076406002 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13696095216756415) | > loader_time: 0.0046 (0.004665779701106114)  --> TIME: 2025-02-27 12:14:23 -- STEP: 31940/36938 -- GLOBAL_STEP: 1694150 | > loss_text_ce: 0.02902510017156601 (0.029795735704163818) | > loss_mel_ce: 2.6939096450805664 (nan) | > loss: 0.010805296711623669 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13696477349728522) | > loader_time: 0.0049 (0.0046657608088360196)  --> TIME: 2025-02-27 12:14:32 -- STEP: 31990/36938 -- GLOBAL_STEP: 1694200 | > loss_text_ce: 0.040626753121614456 (0.029796127969135344) | > loss_mel_ce: 3.2658181190490723 (nan) | > loss: 0.013120814226567745 (nan) | > current_lr: 5e-06 | > step_time: 0.1807 (0.13696611362831645) | > loader_time: 0.0048 (0.004665780417730602)  --> TIME: 2025-02-27 12:14:41 -- STEP: 32040/36938 -- GLOBAL_STEP: 1694250 | > loss_text_ce: 0.029002320021390915 (0.029793840302962508) | > loss_mel_ce: 2.4874870777130127 (nan) | > loss: 0.009986069984734058 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.1369629946987274) | > loader_time: 0.0048 (0.004665758785385915)  --> TIME: 2025-02-27 12:14:51 -- STEP: 32090/36938 -- GLOBAL_STEP: 1694300 | > loss_text_ce: 0.025164274498820305 (0.029795255410740667) | > loss_mel_ce: 1.2933273315429688 (nan) | > loss: 0.005232109688222408 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13696346727254058) | > loader_time: 0.0046 (0.00466578367827057)  --> TIME: 2025-02-27 12:15:00 -- STEP: 32140/36938 -- GLOBAL_STEP: 1694350 | > loss_text_ce: 0.03210572525858879 (0.02979340097831022) | > loss_mel_ce: 2.817841053009033 (nan) | > loss: 0.011309312656521797 (nan) | > current_lr: 5e-06 | > step_time: 0.1447 (0.1369742600904457) | > loader_time: 0.0047 (0.00466594391901744)  --> TIME: 2025-02-27 12:15:10 -- STEP: 32190/36938 -- GLOBAL_STEP: 1694400 | > loss_text_ce: 0.03329945355653763 (0.02979215207712511) | > loss_mel_ce: 2.3258440494537354 (nan) | > loss: 0.009361681528389454 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13697279615037752) | > loader_time: 0.0047 (0.004665933302671654)  --> TIME: 2025-02-27 12:15:19 -- STEP: 32240/36938 -- GLOBAL_STEP: 1694450 | > loss_text_ce: 0.03729912266135216 (0.0297942503462615) | > loss_mel_ce: 2.2593750953674316 (nan) | > loss: 0.009113787673413754 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13698195378034872) | > loader_time: 0.0045 (0.004665937827479457)  --> TIME: 2025-02-27 12:15:29 -- STEP: 32290/36938 -- GLOBAL_STEP: 1694500 | > loss_text_ce: 0.025406507775187492 (0.02979282517393965) | > loss_mel_ce: 1.9296951293945312 (nan) | > loss: 0.007758340332657099 (nan) | > current_lr: 5e-06 | > step_time: 0.1574 (0.1369789384400074) | > loader_time: 0.0047 (0.004665894654558408)  --> TIME: 2025-02-27 12:15:38 -- STEP: 32340/36938 -- GLOBAL_STEP: 1694550 | > loss_text_ce: 0.031638965010643005 (0.02979208099106921) | > loss_mel_ce: 0.0208740234375 (nan) | > loss: 0.00020838488126173615 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.1369804803869492) | > loader_time: 0.0045 (0.004665969069828278)  --> TIME: 2025-02-27 12:15:47 -- STEP: 32390/36938 -- GLOBAL_STEP: 1694600 | > loss_text_ce: 0.041190389543771744 (0.02979135330836599) | > loss_mel_ce: 2.382068634033203 (nan) | > loss: 0.009616107679903507 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13696713084679846) | > loader_time: 0.0045 (0.004665844614912826)  --> TIME: 2025-02-27 12:15:56 -- STEP: 32440/36938 -- GLOBAL_STEP: 1694650 | > loss_text_ce: 0.032002683728933334 (0.029792693184812723) | > loss_mel_ce: 2.151571035385132 (nan) | > loss: 0.008664975874125957 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13697091512262535) | > loader_time: 0.0045 (0.004665713407255426)  --> TIME: 2025-02-27 12:16:06 -- STEP: 32490/36938 -- GLOBAL_STEP: 1694700 | > loss_text_ce: 0.02725496143102646 (0.029793312573600794) | > loss_mel_ce: 2.223649263381958 (nan) | > loss: 0.008932160213589668 (nan) | > current_lr: 5e-06 | > step_time: 0.1416 (0.1369752981435993) | > loader_time: 0.0046 (0.0046657104351293804)  --> TIME: 2025-02-27 12:16:15 -- STEP: 32540/36938 -- GLOBAL_STEP: 1694750 | > loss_text_ce: 0.031064394861459732 (0.029792002918852416) | > loss_mel_ce: 2.4056596755981445 (nan) | > loss: 0.009669541381299496 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13697733290736128) | > loader_time: 0.0045 (0.004665747894856854)  --> TIME: 2025-02-27 12:16:24 -- STEP: 32590/36938 -- GLOBAL_STEP: 1694800 | > loss_text_ce: 0.03267945721745491 (0.02979317763171595) | > loss_mel_ce: 1.9820294380187988 (nan) | > loss: 0.00799487717449665 (nan) | > current_lr: 5e-06 | > step_time: 0.108 (0.1369746780497921) | > loader_time: 0.0044 (0.0046657312424792305)  --> TIME: 2025-02-27 12:16:34 -- STEP: 32640/36938 -- GLOBAL_STEP: 1694850 | > loss_text_ce: 0.029488181695342064 (0.02979379729587499) | > loss_mel_ce: 3.138859748840332 (nan) | > loss: 0.012572810053825378 (nan) | > current_lr: 5e-06 | > step_time: 0.1507 (0.13696891402496947) | > loader_time: 0.0047 (0.004665631223835199)  --> TIME: 2025-02-27 12:16:43 -- STEP: 32690/36938 -- GLOBAL_STEP: 1694900 | > loss_text_ce: 0.03579457476735115 (0.02979136212589605) | > loss_mel_ce: 0.6591152548789978 (nan) | > loss: 0.002757578855380416 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13696587160844992) | > loader_time: 0.0053 (0.004665840084161116)  --> TIME: 2025-02-27 12:16:52 -- STEP: 32740/36938 -- GLOBAL_STEP: 1694950 | > loss_text_ce: 0.03421199321746826 (0.029793808588993654) | > loss_mel_ce: 2.6586642265319824 (nan) | > loss: 0.010686017572879791 (nan) | > current_lr: 5e-06 | > step_time: 0.187 (0.13695617351287237) | > loader_time: 0.0049 (0.0046660565499787535)  --> TIME: 2025-02-27 12:17:02 -- STEP: 32790/36938 -- GLOBAL_STEP: 1695000 | > loss_text_ce: 0.02591811679303646 (0.029793822512794076) | > loss_mel_ce: 2.5489580631256104 (nan) | > loss: 0.010217762552201748 (nan) | > current_lr: 5e-06 | > step_time: 0.1685 (0.13696814247950298) | > loader_time: 0.0049 (0.004666295208629948)  --> TIME: 2025-02-27 12:17:11 -- STEP: 32840/36938 -- GLOBAL_STEP: 1695050 | > loss_text_ce: 0.02460973523557186 (0.029792188077849707) | > loss_mel_ce: 2.5077247619628906 (nan) | > loss: 0.010048947297036648 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13697426419310327) | > loader_time: 0.0046 (0.004666510699872705)  --> TIME: 2025-02-27 12:17:20 -- STEP: 32890/36938 -- GLOBAL_STEP: 1695100 | > loss_text_ce: 0.02434062957763672 (0.02979052715625317) | > loss_mel_ce: 2.88340425491333 (nan) | > loss: 0.011538670398294926 (nan) | > current_lr: 5e-06 | > step_time: 0.1385 (0.13696958839657686) | > loader_time: 0.0046 (0.004666502825863713)  --> TIME: 2025-02-27 12:17:30 -- STEP: 32940/36938 -- GLOBAL_STEP: 1695150 | > loss_text_ce: 0.02714574709534645 (0.02979170540545609) | > loss_mel_ce: 2.5740888118743896 (nan) | > loss: 0.010322360321879387 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13697627453494168) | > loader_time: 0.0047 (0.0046665361887336014)  --> TIME: 2025-02-27 12:17:39 -- STEP: 32990/36938 -- GLOBAL_STEP: 1695200 | > loss_text_ce: 0.027707109227776527 (0.02979114525280973) | > loss_mel_ce: 2.2406933307647705 (nan) | > loss: 0.009001589380204678 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13697147793609335) | > loader_time: 0.0046 (0.004666695554316425)  --> TIME: 2025-02-27 12:17:48 -- STEP: 33040/36938 -- GLOBAL_STEP: 1695250 | > loss_text_ce: 0.02856759913265705 (0.029791469440330957) | > loss_mel_ce: 3.097825050354004 (nan) | > loss: 0.012406320311129093 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.13697036559587467) | > loader_time: 0.0048 (0.0046668908930863956)  --> TIME: 2025-02-27 12:17:58 -- STEP: 33090/36938 -- GLOBAL_STEP: 1695300 | > loss_text_ce: 0.027679983526468277 (0.029790553924435945) | > loss_mel_ce: 2.3946032524108887 (nan) | > loss: 0.009612235240638256 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13696875168715852) | > loader_time: 0.0046 (0.004667003783764672)  --> TIME: 2025-02-27 12:18:07 -- STEP: 33140/36938 -- GLOBAL_STEP: 1695350 | > loss_text_ce: 0.03129207715392113 (0.029788734646335333) | > loss_mel_ce: 3.1588683128356934 (nan) | > loss: 0.012659367173910141 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13697119515274486) | > loader_time: 0.0046 (0.004667069700444833)  --> TIME: 2025-02-27 12:18:16 -- STEP: 33190/36938 -- GLOBAL_STEP: 1695400 | > loss_text_ce: 0.031352005898952484 (0.029788969500446613) | > loss_mel_ce: 1.9706945419311523 (nan) | > loss: 0.007944629527628422 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13696911156123587) | > loader_time: 0.0045 (0.004667168081650787)  --> TIME: 2025-02-27 12:18:26 -- STEP: 33240/36938 -- GLOBAL_STEP: 1695450 | > loss_text_ce: 0.03618872910737991 (0.029790910178819) | > loss_mel_ce: 2.819767951965332 (nan) | > loss: 0.011333162896335125 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13696629423670478) | > loader_time: 0.0047 (0.00466724444812816)  --> TIME: 2025-02-27 12:18:35 -- STEP: 33290/36938 -- GLOBAL_STEP: 1695500 | > loss_text_ce: 0.03477620333433151 (0.029791876096468665) | > loss_mel_ce: 2.1182861328125 (nan) | > loss: 0.008543898351490498 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13696527648666754) | > loader_time: 0.0046 (0.004667281689103855)  --> TIME: 2025-02-27 12:18:44 -- STEP: 33340/36938 -- GLOBAL_STEP: 1695550 | > loss_text_ce: 0.018524447456002235 (0.029792224727436922) | > loss_mel_ce: 1.6599324941635132 (nan) | > loss: 0.0066605438478291035 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.1369679424100526) | > loader_time: 0.0046 (0.004667409594786975)  --> TIME: 2025-02-27 12:18:54 -- STEP: 33390/36938 -- GLOBAL_STEP: 1695600 | > loss_text_ce: 0.03445526584982872 (0.029793159893621007) | > loss_mel_ce: 3.026237964630127 (nan) | > loss: 0.01214560866355896 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.13697185974772244) | > loader_time: 0.0048 (0.004667602844958036)  --> TIME: 2025-02-27 12:19:03 -- STEP: 33440/36938 -- GLOBAL_STEP: 1695650 | > loss_text_ce: 0.02967744693160057 (0.029791964273972203) | > loss_mel_ce: 2.3343844413757324 (nan) | > loss: 0.009381198324263096 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.1369803590423747) | > loader_time: 0.0046 (0.004667692050409089)  --> TIME: 2025-02-27 12:19:13 -- STEP: 33490/36938 -- GLOBAL_STEP: 1695700 | > loss_text_ce: 0.027079379186034203 (0.029791685848761865) | > loss_mel_ce: 0.044921875 (nan) | > loss: 0.0002857192885130644 (nan) | > current_lr: 5e-06 | > step_time: 0.1076 (0.1369780729051199) | > loader_time: 0.0045 (0.0046676545044883905)  --> TIME: 2025-02-27 12:19:23 -- STEP: 33540/36938 -- GLOBAL_STEP: 1695750 | > loss_text_ce: 0.029474003240466118 (0.02979124950438839) | > loss_mel_ce: 0.02386474609375 (nan) | > loss: 0.0002116617251886055 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13699432155811525) | > loader_time: 0.0046 (0.004667675836354973)  --> TIME: 2025-02-27 12:19:32 -- STEP: 33590/36938 -- GLOBAL_STEP: 1695800 | > loss_text_ce: 0.019566735252738 (0.02979020924266093) | > loss_mel_ce: 2.6862847805023193 (nan) | > loss: 0.010737506672739983 (nan) | > current_lr: 5e-06 | > step_time: 0.1683 (0.1369838595283846) | > loader_time: 0.0048 (0.004667644409879256)  --> TIME: 2025-02-27 12:19:41 -- STEP: 33640/36938 -- GLOBAL_STEP: 1695850 | > loss_text_ce: 0.02945219911634922 (0.029790428298299594) | > loss_mel_ce: 1.6802961826324463 (nan) | > loss: 0.006784716155380011 (nan) | > current_lr: 5e-06 | > step_time: 0.125 (0.13697748313902622) | > loader_time: 0.0046 (0.004667543783766194)  --> TIME: 2025-02-27 12:19:50 -- STEP: 33690/36938 -- GLOBAL_STEP: 1695900 | > loss_text_ce: 0.022430038079619408 (0.0297906056147386) | > loss_mel_ce: 3.111421585083008 (nan) | > loss: 0.012435919605195522 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.13697914378018097) | > loader_time: 0.0046 (0.004667597929495101)  --> TIME: 2025-02-27 12:20:00 -- STEP: 33740/36938 -- GLOBAL_STEP: 1695950 | > loss_text_ce: 0.04359084367752075 (0.029792431703833253) | > loss_mel_ce: 2.821387529373169 (nan) | > loss: 0.01136896200478077 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.13698479910185102) | > loader_time: 0.0048 (0.004667575223670603)  --> TIME: 2025-02-27 12:20:09 -- STEP: 33790/36938 -- GLOBAL_STEP: 1696000 | > loss_text_ce: 0.021897336468100548 (0.029790897872717895) | > loss_mel_ce: 2.7608580589294434 (nan) | > loss: 0.011042680591344833 (nan) | > current_lr: 5e-06 | > step_time: 0.1835 (0.1369855189118799) | > loader_time: 0.0048 (0.004667661408060013)  --> TIME: 2025-02-27 12:20:18 -- STEP: 33840/36938 -- GLOBAL_STEP: 1696050 | > loss_text_ce: 0.03231322020292282 (0.029793089182181036) | > loss_mel_ce: 0.01409912109375 (nan) | > loss: 0.00018417596584185958 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.1369832025915731) | > loader_time: 0.0046 (0.0046678273742644235)  --> TIME: 2025-02-27 12:20:28 -- STEP: 33890/36938 -- GLOBAL_STEP: 1696100 | > loss_text_ce: 0.0287462305277586 (0.0297925533467668) | > loss_mel_ce: 2.8724403381347656 (nan) | > loss: 0.01151264552026987 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.13698986399029464) | > loader_time: 0.0047 (0.0046678552799219244)  --> TIME: 2025-02-27 12:20:37 -- STEP: 33940/36938 -- GLOBAL_STEP: 1696150 | > loss_text_ce: 0.037908826023340225 (0.02978997121550775) | > loss_mel_ce: 0.13075722754001617 (nan) | > loss: 0.0006693097529932857 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13698860841002486) | > loader_time: 0.0046 (0.004667818504146923)  --> TIME: 2025-02-27 12:20:46 -- STEP: 33990/36938 -- GLOBAL_STEP: 1696200 | > loss_text_ce: 0.03370102494955063 (0.029789602406850132) | > loss_mel_ce: 2.3384621143341064 (nan) | > loss: 0.009413345716893673 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13698171886776236) | > loader_time: 0.0045 (0.004667752698857353)  --> TIME: 2025-02-27 12:20:56 -- STEP: 34040/36938 -- GLOBAL_STEP: 1696250 | > loss_text_ce: 0.02427167445421219 (0.02978956750755502) | > loss_mel_ce: 2.1749656200408936 (nan) | > loss: 0.00872713327407837 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13698278395885816) | > loader_time: 0.0046 (0.004667768060950919)  --> TIME: 2025-02-27 12:21:05 -- STEP: 34090/36938 -- GLOBAL_STEP: 1696300 | > loss_text_ce: 0.028861524537205696 (0.02978983534402745) | > loss_mel_ce: 2.9676718711853027 (nan) | > loss: 0.011891006492078304 (nan) | > current_lr: 5e-06 | > step_time: 0.1443 (0.13698198937292655) | > loader_time: 0.0046 (0.004667748045312866)  --> TIME: 2025-02-27 12:21:15 -- STEP: 34140/36938 -- GLOBAL_STEP: 1696350 | > loss_text_ce: 0.026723114773631096 (0.029788957802932357) | > loss_mel_ce: 1.7266736030578613 (nan) | > loss: 0.006957923993468285 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13698767807026896) | > loader_time: 0.0045 (0.004667794453029263)  --> TIME: 2025-02-27 12:21:24 -- STEP: 34190/36938 -- GLOBAL_STEP: 1696400 | > loss_text_ce: 0.024027341976761818 (0.029788446043157415) | > loss_mel_ce: 1.6327581405639648 (nan) | > loss: 0.0065745459869503975 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.1369966178095172) | > loader_time: 0.0045 (0.004667886205150494)  --> TIME: 2025-02-27 12:21:34 -- STEP: 34240/36938 -- GLOBAL_STEP: 1696450 | > loss_text_ce: 0.028022464364767075 (0.029788219118104586) | > loss_mel_ce: 2.383146047592163 (nan) | > loss: 0.009568129666149616 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.1369914287997179) | > loader_time: 0.0046 (0.004667867462490205)  --> TIME: 2025-02-27 12:21:43 -- STEP: 34290/36938 -- GLOBAL_STEP: 1696500 | > loss_text_ce: 0.03456877917051315 (0.029787666955894885) | > loss_mel_ce: 2.568967580795288 (nan) | > loss: 0.010331493802368641 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.1369916130551511) | > loader_time: 0.0046 (0.004667983454217295)  --> TIME: 2025-02-27 12:21:52 -- STEP: 34340/36938 -- GLOBAL_STEP: 1696550 | > loss_text_ce: 0.03186408430337906 (0.029790043536449977) | > loss_mel_ce: 2.5634765625 (nan) | > loss: 0.010298972018063068 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.1369877128456657) | > loader_time: 0.0044 (0.004667930666901859)  --> TIME: 2025-02-27 12:22:01 -- STEP: 34390/36938 -- GLOBAL_STEP: 1696600 | > loss_text_ce: 0.031548988074064255 (0.029792437954110074) | > loss_mel_ce: 2.8905539512634277 (nan) | > loss: 0.011595646850764751 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.136983993872198) | > loader_time: 0.0045 (0.004667845268726486)  --> TIME: 2025-02-27 12:22:11 -- STEP: 34440/36938 -- GLOBAL_STEP: 1696650 | > loss_text_ce: 0.02692682482302189 (0.029792408284522447) | > loss_mel_ce: 3.008380651473999 (nan) | > loss: 0.012044871225953102 (nan) | > current_lr: 5e-06 | > step_time: 0.1657 (0.1369941658259263) | > loader_time: 0.0047 (0.004667863627067148)  --> TIME: 2025-02-27 12:22:20 -- STEP: 34490/36938 -- GLOBAL_STEP: 1696700 | > loss_text_ce: 0.033070631325244904 (0.029792721777026743) | > loss_mel_ce: 1.447632908821106 (nan) | > loss: 0.00587580818682909 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13699099481814292) | > loader_time: 0.0045 (0.004667787020985509)  --> TIME: 2025-02-27 12:22:30 -- STEP: 34540/36938 -- GLOBAL_STEP: 1696750 | > loss_text_ce: 0.03084755502641201 (0.029793282206853518) | > loss_mel_ce: 0.019102521240711212 (nan) | > loss: 0.00019821460591629148 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13699726520691788) | > loader_time: 0.0045 (0.004667803657213659)  --> TIME: 2025-02-27 12:22:39 -- STEP: 34590/36938 -- GLOBAL_STEP: 1696800 | > loss_text_ce: 0.030364353209733963 (0.029792559456352674) | > loss_mel_ce: 2.377488136291504 (nan) | > loss: 0.0095549700781703 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.1369986402330175) | > loader_time: 0.0046 (0.0046677646832715605)  --> TIME: 2025-02-27 12:22:49 -- STEP: 34640/36938 -- GLOBAL_STEP: 1696850 | > loss_text_ce: 0.03355792537331581 (0.02979319467452327) | > loss_mel_ce: 2.8929195404052734 (nan) | > loss: 0.011613006703555584 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13699931763886794) | > loader_time: 0.0046 (0.004667713136926257)  --> TIME: 2025-02-27 12:22:58 -- STEP: 34690/36938 -- GLOBAL_STEP: 1696900 | > loss_text_ce: 0.022128276526927948 (0.029793814819120353) | > loss_mel_ce: 2.732712745666504 (nan) | > loss: 0.010931910015642643 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.1369988086691636) | > loader_time: 0.0047 (0.004667675656659281)  --> TIME: 2025-02-27 12:23:07 -- STEP: 34740/36938 -- GLOBAL_STEP: 1696950 | > loss_text_ce: 0.03119475208222866 (0.029796888459115687) | > loss_mel_ce: 2.7666234970092773 (nan) | > loss: 0.01110245380550623 (nan) | > current_lr: 5e-06 | > step_time: 0.1643 (0.13699636007768995) | > loader_time: 0.0047 (0.004667594210471513)  --> TIME: 2025-02-27 12:23:17 -- STEP: 34790/36938 -- GLOBAL_STEP: 1697000 | > loss_text_ce: 0.03556813299655914 (0.02979673242507506) | > loss_mel_ce: 2.1723825931549072 (nan) | > loss: 0.008761710487306118 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13700355352016258) | > loader_time: 0.0046 (0.0046675553709722584)  --> TIME: 2025-02-27 12:23:26 -- STEP: 34840/36938 -- GLOBAL_STEP: 1697050 | > loss_text_ce: 0.03495471924543381 (0.02979704381429711) | > loss_mel_ce: 2.7690443992614746 (nan) | > loss: 0.011126981116831303 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.13700071748444412) | > loader_time: 0.0047 (0.004667561520938895)  --> TIME: 2025-02-27 12:23:36 -- STEP: 34890/36938 -- GLOBAL_STEP: 1697100 | > loss_text_ce: 0.03449748829007149 (0.029798197642387274) | > loss_mel_ce: 2.8352859020233154 (nan) | > loss: 0.011388029903173447 (nan) | > current_lr: 5e-06 | > step_time: 0.1348 (0.13700222185260028) | > loader_time: 0.0045 (0.004667515035822249)  --> TIME: 2025-02-27 12:23:45 -- STEP: 34940/36938 -- GLOBAL_STEP: 1697150 | > loss_text_ce: 0.02505524829030037 (0.02979699817348035) | > loss_mel_ce: 3.106699228286743 (nan) | > loss: 0.012427597306668758 (nan) | > current_lr: 5e-06 | > step_time: 0.1648 (0.13700794767637373) | > loader_time: 0.0046 (0.004667525347396859)  --> TIME: 2025-02-27 12:23:54 -- STEP: 34990/36938 -- GLOBAL_STEP: 1697200 | > loss_text_ce: 0.028321105986833572 (0.029798430781018296) | > loss_mel_ce: 2.5194554328918457 (nan) | > loss: 0.010110224597156048 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1370032631488958) | > loader_time: 0.0046 (0.004667482992075755)  --> TIME: 2025-02-27 12:24:03 -- STEP: 35040/36938 -- GLOBAL_STEP: 1697250 | > loss_text_ce: 0.021584033966064453 (0.029799344965956306) | > loss_mel_ce: 1.4033269882202148 (nan) | > loss: 0.005654409062117338 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13699807096697925) | > loader_time: 0.0046 (0.004667446826963119)  --> TIME: 2025-02-27 12:24:13 -- STEP: 35090/36938 -- GLOBAL_STEP: 1697300 | > loss_text_ce: 0.02364092692732811 (0.02979962896867784) | > loss_mel_ce: 3.0829567909240723 (nan) | > loss: 0.012327768839895725 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.1369972786967245) | > loader_time: 0.0047 (0.0046674890034420925)  --> TIME: 2025-02-27 12:24:22 -- STEP: 35140/36938 -- GLOBAL_STEP: 1697350 | > loss_text_ce: 0.030106982216238976 (0.029800667275881897) | > loss_mel_ce: 2.4279420375823975 (nan) | > loss: 0.009754163213074207 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13700138949235138) | > loader_time: 0.0045 (0.004667439770142187)  --> TIME: 2025-02-27 12:24:32 -- STEP: 35190/36938 -- GLOBAL_STEP: 1697400 | > loss_text_ce: 0.02791258692741394 (0.029800791888328126) | > loss_mel_ce: 2.863898992538452 (nan) | > loss: 0.011475443840026855 (nan) | > current_lr: 5e-06 | > step_time: 0.1932 (0.1370143189624003) | > loader_time: 0.0048 (0.004667424511997555)  --> TIME: 2025-02-27 12:24:42 -- STEP: 35240/36938 -- GLOBAL_STEP: 1697450 | > loss_text_ce: 0.03222180902957916 (0.029801664511286248) | > loss_mel_ce: 2.741989850997925 (nan) | > loss: 0.011008776724338531 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.13702592673150116) | > loader_time: 0.0047 (0.004667423254515331)  --> TIME: 2025-02-27 12:24:51 -- STEP: 35290/36938 -- GLOBAL_STEP: 1697500 | > loss_text_ce: 0.02908702939748764 (0.029802222760802213) | > loss_mel_ce: 3.1158864498138428 (nan) | > loss: 0.012480054050683975 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13702596772254136) | > loader_time: 0.0046 (0.004667335665911709)  --> TIME: 2025-02-27 12:25:00 -- STEP: 35340/36938 -- GLOBAL_STEP: 1697550 | > loss_text_ce: 0.017880862578749657 (0.02980193945022651) | > loss_mel_ce: 0.7548109889030457 (nan) | > loss: 0.003066237550228834 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.13702569458534536) | > loader_time: 0.0045 (0.00466734902225919)  --> TIME: 2025-02-27 12:25:10 -- STEP: 35390/36938 -- GLOBAL_STEP: 1697600 | > loss_text_ce: 0.028582125902175903 (0.02980155854410402) | > loss_mel_ce: 2.3634228706359863 (nan) | > loss: 0.009492083452641964 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13702336227398623) | > loader_time: 0.0047 (0.004667316395268318)  --> TIME: 2025-02-27 12:25:19 -- STEP: 35440/36938 -- GLOBAL_STEP: 1697650 | > loss_text_ce: 0.03522822633385658 (0.029801687847168034) | > loss_mel_ce: 3.5540707111358643 (nan) | > loss: 0.014243250712752342 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.137017827284255) | > loader_time: 0.0045 (0.004667245103866187)  --> TIME: 2025-02-27 12:25:28 -- STEP: 35490/36938 -- GLOBAL_STEP: 1697700 | > loss_text_ce: 0.038294676691293716 (0.02980166938122825) | > loss_mel_ce: 2.7999379634857178 (nan) | > loss: 0.011262829415500164 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.1370118398604574) | > loader_time: 0.0046 (0.00466726807078094)  --> TIME: 2025-02-27 12:25:37 -- STEP: 35540/36938 -- GLOBAL_STEP: 1697750 | > loss_text_ce: 0.027429858222603798 (0.029801758943853052) | > loss_mel_ce: 2.4061334133148193 (nan) | > loss: 0.009656997397542 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13701067174762274) | > loader_time: 0.0045 (0.004667268620492625)  --> TIME: 2025-02-27 12:25:46 -- STEP: 35590/36938 -- GLOBAL_STEP: 1697800 | > loss_text_ce: 0.029103292152285576 (0.02980331841086376) | > loss_mel_ce: 2.7776169776916504 (nan) | > loss: 0.011137779802083969 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.13700918731946674) | > loader_time: 0.0047 (0.004667228023199985)  --> TIME: 2025-02-27 12:25:55 -- STEP: 35640/36938 -- GLOBAL_STEP: 1697850 | > loss_text_ce: 0.028768517076969147 (0.02980226353492889) | > loss_mel_ce: 3.325754404067993 (nan) | > loss: 0.013311599381268024 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.13700039785451476) | > loader_time: 0.0046 (0.004667279957119447)  --> TIME: 2025-02-27 12:26:04 -- STEP: 35690/36938 -- GLOBAL_STEP: 1697900 | > loss_text_ce: 0.01953563094139099 (0.029801068628641587) | > loss_mel_ce: 3.004403591156006 (nan) | > loss: 0.011999758891761303 (nan) | > current_lr: 5e-06 | > step_time: 0.142 (0.1369957419729725) | > loader_time: 0.0046 (0.004667182615338388)  --> TIME: 2025-02-27 12:26:14 -- STEP: 35740/36938 -- GLOBAL_STEP: 1697950 | > loss_text_ce: 0.02496730536222458 (0.02980100154852214) | > loss_mel_ce: 0.8008941411972046 (nan) | > loss: 0.0032772282138466835 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13699072346583233) | > loader_time: 0.0045 (0.00466711978106505)  --> TIME: 2025-02-27 12:26:23 -- STEP: 35790/36938 -- GLOBAL_STEP: 1698000 | > loss_text_ce: 0.0430649071931839 (0.02980025553868703) | > loss_mel_ce: 1.1988639831542969 (nan) | > loss: 0.004928289912641048 (nan) | > current_lr: 5e-06 | > step_time: 0.1179 (0.1369917558404644) | > loader_time: 0.0045 (0.004667044811723068)  --> TIME: 2025-02-27 12:26:32 -- STEP: 35840/36938 -- GLOBAL_STEP: 1698050 | > loss_text_ce: 0.018244154751300812 (0.029800501285951343) | > loss_mel_ce: 0.7930975556373596 (nan) | > loss: 0.0032196100801229477 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13699332091159033) | > loader_time: 0.0047 (0.004666953154706511)  --> TIME: 2025-02-27 12:26:42 -- STEP: 35890/36938 -- GLOBAL_STEP: 1698100 | > loss_text_ce: 0.028586268424987793 (0.029800101568157933) | > loss_mel_ce: 1.6895103454589844 (nan) | > loss: 0.006817844230681658 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13699506911503348) | > loader_time: 0.0106 (0.004667031090715911)  --> TIME: 2025-02-27 12:26:51 -- STEP: 35940/36938 -- GLOBAL_STEP: 1698150 | > loss_text_ce: 0.023920169100165367 (0.029798473409103913) | > loss_mel_ce: 2.8268020153045654 (nan) | > loss: 0.011312389746308327 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.13700545456386884) | > loader_time: 0.0047 (0.004667168885519172)  --> TIME: 2025-02-27 12:27:01 -- STEP: 35990/36938 -- GLOBAL_STEP: 1698200 | > loss_text_ce: 0.04454386606812477 (0.029798662000617555) | > loss_mel_ce: 3.249069929122925 (nan) | > loss: 0.013069896958768368 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13700021728935333) | > loader_time: 0.0045 (0.004667089614115595)  --> TIME: 2025-02-27 12:27:10 -- STEP: 36040/36938 -- GLOBAL_STEP: 1698250 | > loss_text_ce: 0.03510693460702896 (0.029800094999776084) | > loss_mel_ce: 2.228018045425415 (nan) | > loss: 0.00898065511137247 (nan) | > current_lr: 5e-06 | > step_time: 0.1327 (0.1369996027597177) | > loader_time: 0.0046 (0.004667235558623066)  --> TIME: 2025-02-27 12:27:19 -- STEP: 36090/36938 -- GLOBAL_STEP: 1698300 | > loss_text_ce: 0.036622654646635056 (0.029799370262973128) | > loss_mel_ce: 1.621063232421875 (nan) | > loss: 0.00657811900600791 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13699478859979355) | > loader_time: 0.0045 (0.004667161151159816)  --> TIME: 2025-02-27 12:27:28 -- STEP: 36140/36938 -- GLOBAL_STEP: 1698350 | > loss_text_ce: 0.03243397921323776 (0.02979944258321161) | > loss_mel_ce: 2.9478538036346436 (nan) | > loss: 0.011826539412140846 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.1369964132403955) | > loader_time: 0.0046 (0.004667191256062378)  --> TIME: 2025-02-27 12:27:38 -- STEP: 36190/36938 -- GLOBAL_STEP: 1698400 | > loss_text_ce: 0.034282803535461426 (0.02980011573875064) | > loss_mel_ce: 2.6453068256378174 (nan) | > loss: 0.01063329353928566 (nan) | > current_lr: 5e-06 | > step_time: 0.1421 (0.13699708936226931) | > loader_time: 0.0046 (0.004667322067108938)  --> TIME: 2025-02-27 12:27:47 -- STEP: 36240/36938 -- GLOBAL_STEP: 1698450 | > loss_text_ce: 0.039797592908144 (0.02979922815581689) | > loss_mel_ce: 0.013275146484375 (nan) | > loss: 0.00021060611470602453 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13699840199342062) | > loader_time: 0.0046 (0.004667267157278821)  --> TIME: 2025-02-27 12:27:57 -- STEP: 36290/36938 -- GLOBAL_STEP: 1698500 | > loss_text_ce: 0.034272193908691406 (0.029799096224602178) | > loss_mel_ce: 2.5092694759368896 (nan) | > loss: 0.010093419812619686 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.137000700270515) | > loader_time: 0.0048 (0.0046672072611619615)  --> TIME: 2025-02-27 12:28:06 -- STEP: 36340/36938 -- GLOBAL_STEP: 1698550 | > loss_text_ce: 0.022425228729844093 (0.029798510487985336) | > loss_mel_ce: 1.816229224205017 (nan) | > loss: 0.007296247873455286 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13700993654815818) | > loader_time: 0.0045 (0.004667280024726355)  --> TIME: 2025-02-27 12:28:16 -- STEP: 36390/36938 -- GLOBAL_STEP: 1698600 | > loss_text_ce: 0.03748443350195885 (0.02979860175390735) | > loss_mel_ce: 2.823286771774292 (nan) | > loss: 0.011352267116308212 (nan) | > current_lr: 5e-06 | > step_time: 0.1518 (0.1370115002749808) | > loader_time: 0.0046 (0.004667237991486035)  --> TIME: 2025-02-27 12:28:25 -- STEP: 36440/36938 -- GLOBAL_STEP: 1698650 | > loss_text_ce: 0.029082803055644035 (0.029799383843489374) | > loss_mel_ce: 3.0730252265930176 (nan) | > loss: 0.012309952639043331 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.1370099658761927) | > loader_time: 0.0046 (0.004667278342398781)  --> TIME: 2025-02-27 12:28:34 -- STEP: 36490/36938 -- GLOBAL_STEP: 1698700 | > loss_text_ce: 0.03335507586598396 (0.029799783175464573) | > loss_mel_ce: 2.6564273834228516 (nan) | > loss: 0.010673739947378635 (nan) | > current_lr: 5e-06 | > step_time: 0.181 (0.13700906331978857) | > loader_time: 0.0048 (0.0046674298142432725)  --> TIME: 2025-02-27 12:28:44 -- STEP: 36540/36938 -- GLOBAL_STEP: 1698750 | > loss_text_ce: 0.026050304993987083 (0.029799915244624418) | > loss_mel_ce: 2.979318141937256 (nan) | > loss: 0.011926066130399704 (nan) | > current_lr: 5e-06 | > step_time: 0.1488 (0.13701079621886553) | > loader_time: 0.0046 (0.004667393249048173)  --> TIME: 2025-02-27 12:28:53 -- STEP: 36590/36938 -- GLOBAL_STEP: 1698800 | > loss_text_ce: 0.03223615884780884 (0.029798753910076182) | > loss_mel_ce: 2.7194268703460693 (nan) | > loss: 0.0109192980453372 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.1370099892346509) | > loader_time: 0.0046 (0.004667345563320862)  --> TIME: 2025-02-27 12:29:02 -- STEP: 36640/36938 -- GLOBAL_STEP: 1698850 | > loss_text_ce: 0.036414653062820435 (0.029799190393952027) | > loss_mel_ce: 2.4814586639404297 (nan) | > loss: 0.009991561062633991 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.13701398859378006) | > loader_time: 0.0048 (0.004667336236701764)  --> TIME: 2025-02-27 12:29:12 -- STEP: 36690/36938 -- GLOBAL_STEP: 1698900 | > loss_text_ce: 0.03026437759399414 (0.029798723311252582) | > loss_mel_ce: 2.58626389503479 (nan) | > loss: 0.01038304902613163 (nan) | > current_lr: 5e-06 | > step_time: 0.1606 (0.13702199952288494) | > loader_time: 0.0047 (0.004667372305863019)  --> TIME: 2025-02-27 12:29:21 -- STEP: 36740/36938 -- GLOBAL_STEP: 1698950 | > loss_text_ce: 0.02715950272977352 (0.029798597516035728) | > loss_mel_ce: 2.7837581634521484 (nan) | > loss: 0.011154435575008392 (nan) | > current_lr: 5e-06 | > step_time: 0.1679 (0.13702001612538126) | > loader_time: 0.0048 (0.004667513923406193)  --> TIME: 2025-02-27 12:29:31 -- STEP: 36790/36938 -- GLOBAL_STEP: 1699000 | > loss_text_ce: 0.033538173884153366 (0.029800521059594107) | > loss_mel_ce: 3.5018773078918457 (nan) | > loss: 0.0140294274315238 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13702433444974707) | > loader_time: 0.0046 (0.004667525500115811)  --> TIME: 2025-02-27 12:29:40 -- STEP: 36840/36938 -- GLOBAL_STEP: 1699050 | > loss_text_ce: 0.032998111099004745 (0.029800213027536167) | > loss_mel_ce: 3.278856039047241 (nan) | > loss: 0.013142279349267483 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13701793812902913) | > loader_time: 0.0046 (0.004667457656673947)  --> TIME: 2025-02-27 12:29:49 -- STEP: 36890/36938 -- GLOBAL_STEP: 1699100 | > loss_text_ce: 0.0334126353263855 (0.029800972449609657) | > loss_mel_ce: 2.2965316772460938 (nan) | > loss: 0.009245811961591244 (nan) | > current_lr: 5e-06 | > step_time: 0.1351 (0.13701660211055855) | > loader_time: 0.0047 (0.004667430474655368)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.02054489131021027 (+0.00016038488633561207) | > avg_loss_text_ce: 0.029077758837380622 (-0.00010427346194882142) | > avg_loss_mel_ce: 2.417074475252981 (-0.006132406763513121) | > avg_loss: 2.4461522288251625 (-0.006236689220562308) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_1699148.pth  > EPOCH: 47/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-27 12:30:23)   --> TIME: 2025-02-27 12:30:24 -- STEP: 2/36938 -- GLOBAL_STEP: 1699150 | > loss_text_ce: 0.020201120525598526 (0.02390221133828163) | > loss_mel_ce: 2.708275079727173 (2.8703821897506714) | > loss: 0.01082728710025549 (0.011485256254673004) | > current_lr: 5e-06 | > step_time: 0.1738 (0.16906249523162842) | > loader_time: 0.0049 (0.0057756900787353516)  --> TIME: 2025-02-27 12:30:33 -- STEP: 52/36938 -- GLOBAL_STEP: 1699200 | > loss_text_ce: 0.027572302147746086 (0.030620686280039642) | > loss_mel_ce: 3.0129261016845703 (2.513975226535247) | > loss: 0.012065471149981022 (0.010097603463621523) | > current_lr: 5e-06 | > step_time: 0.1631 (0.13950854998368484) | > loader_time: 0.0048 (0.004908731350531946)  --> TIME: 2025-02-27 12:30:43 -- STEP: 102/36938 -- GLOBAL_STEP: 1699250 | > loss_text_ce: 0.032598111778497696 (0.03060825771706946) | > loss_mel_ce: 2.6387507915496826 (nan) | > loss: 0.010600591078400612 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13868643255794752) | > loader_time: 0.0048 (0.004872539464165183)  --> TIME: 2025-02-27 12:30:52 -- STEP: 152/36938 -- GLOBAL_STEP: 1699300 | > loss_text_ce: 0.028775567188858986 (0.02982858092965264) | > loss_mel_ce: 2.672297477722168 (nan) | > loss: 0.010718544013798237 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13541621440335327) | > loader_time: 0.0048 (0.0048536470061854315)  --> TIME: 2025-02-27 12:31:01 -- STEP: 202/36938 -- GLOBAL_STEP: 1699350 | > loss_text_ce: 0.035728879272937775 (0.030052115790324636) | > loss_mel_ce: 2.527888059616089 (nan) | > loss: 0.01017308421432972 (nan) | > current_lr: 5e-06 | > step_time: 0.1773 (0.13508005071394524) | > loader_time: 0.0049 (0.004842949385690213)  --> TIME: 2025-02-27 12:31:10 -- STEP: 252/36938 -- GLOBAL_STEP: 1699400 | > loss_text_ce: 0.030886445194482803 (0.02981213934069115) | > loss_mel_ce: 2.928724527359009 (nan) | > loss: 0.011744488961994648 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13588972034908486) | > loader_time: 0.0057 (0.00484597493731786)  --> TIME: 2025-02-27 12:31:19 -- STEP: 302/36938 -- GLOBAL_STEP: 1699450 | > loss_text_ce: 0.02554093487560749 (0.029549241485382547) | > loss_mel_ce: 3.172435760498047 (nan) | > loss: 0.012690383940935135 (nan) | > current_lr: 5e-06 | > step_time: 0.1762 (0.1356041534057517) | > loader_time: 0.0048 (0.004827574388870337)  --> TIME: 2025-02-27 12:31:29 -- STEP: 352/36938 -- GLOBAL_STEP: 1699500 | > loss_text_ce: 0.017440631985664368 (0.029508362185548653) | > loss_mel_ce: 0.0011838674545288086 (nan) | > loss: 7.39067472750321e-05 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13549848171797665) | > loader_time: 0.0046 (0.004816450178623193)  --> TIME: 2025-02-27 12:31:38 -- STEP: 402/36938 -- GLOBAL_STEP: 1699550 | > loss_text_ce: 0.043439872562885284 (0.029563868647691473) | > loss_mel_ce: 2.4512226581573486 (nan) | > loss: 0.00989945512264967 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.1353938846445796) | > loader_time: 0.0047 (0.004807229065776459)  --> TIME: 2025-02-27 12:31:47 -- STEP: 452/36938 -- GLOBAL_STEP: 1699600 | > loss_text_ce: 0.02953701838850975 (0.02959344960724424) | > loss_mel_ce: 2.4640700817108154 (nan) | > loss: 0.009895266965031624 (nan) | > current_lr: 5e-06 | > step_time: 0.1165 (0.13558806254800443) | > loader_time: 0.0046 (0.0048024047792485265)  --> TIME: 2025-02-27 12:31:56 -- STEP: 502/36938 -- GLOBAL_STEP: 1699650 | > loss_text_ce: 0.023292383179068565 (0.029638333362990998) | > loss_mel_ce: 1.758180022239685 (nan) | > loss: 0.0070693353191018105 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1357414698695757) | > loader_time: 0.0046 (0.0047952877097870664)  --> TIME: 2025-02-27 12:32:06 -- STEP: 552/36938 -- GLOBAL_STEP: 1699700 | > loss_text_ce: 0.023364296182990074 (0.029632663946477292) | > loss_mel_ce: 2.9134058952331543 (nan) | > loss: 0.01165385078638792 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13591584822405944) | > loader_time: 0.0047 (0.004792096390240428)  --> TIME: 2025-02-27 12:32:15 -- STEP: 602/36938 -- GLOBAL_STEP: 1699750 | > loss_text_ce: 0.029728475958108902 (0.029637845336734952) | > loss_mel_ce: 3.1306731700897217 (nan) | > loss: 0.012541276402771473 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.1357761981875398) | > loader_time: 0.0046 (0.004781200244181174)  --> TIME: 2025-02-27 12:32:24 -- STEP: 652/36938 -- GLOBAL_STEP: 1699800 | > loss_text_ce: 0.02647206000983715 (0.029607936254262557) | > loss_mel_ce: 2.9252216815948486 (nan) | > loss: 0.011713070794939995 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.1359294872342444) | > loader_time: 0.0047 (0.004777402234223717)  --> TIME: 2025-02-27 12:32:34 -- STEP: 702/36938 -- GLOBAL_STEP: 1699850 | > loss_text_ce: 0.02536858804523945 (0.02961833650336476) | > loss_mel_ce: 2.6517248153686523 (nan) | > loss: 0.01062338799238205 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13592544580117247) | > loader_time: 0.0047 (0.004778657883320772)  --> TIME: 2025-02-27 12:32:43 -- STEP: 752/36938 -- GLOBAL_STEP: 1699900 | > loss_text_ce: 0.03304296359419823 (0.02961797537926109) | > loss_mel_ce: 2.654672145843506 (nan) | > loss: 0.010665536858141422 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13617443499412932) | > loader_time: 0.0048 (0.0047722395430219895)  --> TIME: 2025-02-27 12:32:53 -- STEP: 802/36938 -- GLOBAL_STEP: 1699950 | > loss_text_ce: 0.03180073946714401 (0.029579138097601786) | > loss_mel_ce: 2.0792884826660156 (nan) | > loss: 0.008377338759601116 (nan) | > current_lr: 5e-06 | > step_time: 0.1255 (0.13626642417432067) | > loader_time: 0.0046 (0.004767062955366406)  --> TIME: 2025-02-27 12:33:02 -- STEP: 852/36938 -- GLOBAL_STEP: 1700000 | > loss_text_ce: 0.03763200342655182 (0.029568078116614355) | > loss_mel_ce: 2.684051275253296 (nan) | > loss: 0.010800330899655819 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.13613038471606984) | > loader_time: 0.0059 (0.004762278196397525) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1700000.pth  --> TIME: 2025-02-27 12:33:17 -- STEP: 902/36938 -- GLOBAL_STEP: 1700050 | > loss_text_ce: 0.023356715217232704 (0.029602118028521935) | > loss_mel_ce: 1.8332504034042358 (nan) | > loss: 0.00736748892813921 (nan) | > current_lr: 5e-06 | > step_time: 0.1125 (0.13610098843035306) | > loader_time: 0.0046 (0.004761672072822929)  --> TIME: 2025-02-27 12:33:26 -- STEP: 952/36938 -- GLOBAL_STEP: 1700100 | > loss_text_ce: 0.028626972809433937 (0.0296374354139753) | > loss_mel_ce: 2.783005475997925 (nan) | > loss: 0.011157272383570671 (nan) | > current_lr: 5e-06 | > step_time: 0.1626 (0.13607617536512753) | > loader_time: 0.0046 (0.004757915725227166)  --> TIME: 2025-02-27 12:33:35 -- STEP: 1002/36938 -- GLOBAL_STEP: 1700150 | > loss_text_ce: 0.02904227375984192 (0.029659464640241717) | > loss_mel_ce: 2.2419075965881348 (nan) | > loss: 0.009011706337332726 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.13595136506352842) | > loader_time: 0.0045 (0.004751541657362158)  --> TIME: 2025-02-27 12:33:45 -- STEP: 1052/36938 -- GLOBAL_STEP: 1700200 | > loss_text_ce: 0.027875134721398354 (0.029647431174335655) | > loss_mel_ce: 0.19759520888328552 (nan) | > loss: 0.0008947236347012222 (nan) | > current_lr: 5e-06 | > step_time: 0.1375 (0.13595462842585931) | > loader_time: 0.0046 (0.004746874249026804)  --> TIME: 2025-02-27 12:33:54 -- STEP: 1102/36938 -- GLOBAL_STEP: 1700250 | > loss_text_ce: 0.030795974656939507 (0.029614931505727842) | > loss_mel_ce: 1.0219143629074097 (nan) | > loss: 0.004177422262728214 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.1360511096116636) | > loader_time: 0.0045 (0.004741449537813772)  --> TIME: 2025-02-27 12:34:03 -- STEP: 1152/36938 -- GLOBAL_STEP: 1700300 | > loss_text_ce: 0.027276257053017616 (0.0296344081677186) | > loss_mel_ce: 2.761997699737549 (nan) | > loss: 0.011068548075854778 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13614822179079072) | > loader_time: 0.0049 (0.004738152441051271)  --> TIME: 2025-02-27 12:34:13 -- STEP: 1202/36938 -- GLOBAL_STEP: 1700350 | > loss_text_ce: 0.04816074296832085 (0.029715082099887477) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.1360595398853703) | > loader_time: 0.0045 (0.004733607693638858)  --> TIME: 2025-02-27 12:34:22 -- STEP: 1252/36938 -- GLOBAL_STEP: 1700400 | > loss_text_ce: 0.025456197559833527 (0.02971167302877901) | > loss_mel_ce: 2.4469447135925293 (nan) | > loss: 0.009811115451157093 (nan) | > current_lr: 5e-06 | > step_time: 0.1609 (0.1361291983637948) | > loader_time: 0.0047 (0.004729918778513946)  --> TIME: 2025-02-27 12:34:32 -- STEP: 1302/36938 -- GLOBAL_STEP: 1700450 | > loss_text_ce: 0.030567675828933716 (0.02972209871032729) | > loss_mel_ce: 1.3121962547302246 (nan) | > loss: 0.005328428465873003 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13621698931065584) | > loader_time: 0.0045 (0.004727860566474688)  --> TIME: 2025-02-27 12:34:41 -- STEP: 1352/36938 -- GLOBAL_STEP: 1700500 | > loss_text_ce: 0.02861037477850914 (0.029746930040529317) | > loss_mel_ce: 0.05093993991613388 (nan) | > loss: 0.0003156758612021804 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.1362910503466455) | > loader_time: 0.0046 (0.004728700811340959)  --> TIME: 2025-02-27 12:34:50 -- STEP: 1402/36938 -- GLOBAL_STEP: 1700550 | > loss_text_ce: 0.028249826282262802 (0.02971083635116227) | > loss_mel_ce: 2.516249179840088 (nan) | > loss: 0.010097218677401543 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1363588160012827) | > loader_time: 0.0045 (0.004728919259151615)  --> TIME: 2025-02-27 12:35:00 -- STEP: 1452/36938 -- GLOBAL_STEP: 1700600 | > loss_text_ce: 0.029496842995285988 (0.029679190813879337) | > loss_mel_ce: 2.3420979976654053 (nan) | > loss: 0.009411090984940529 (nan) | > current_lr: 5e-06 | > step_time: 0.1466 (0.13655306222353436) | > loader_time: 0.0048 (0.00472772105009431)  --> TIME: 2025-02-27 12:35:09 -- STEP: 1502/36938 -- GLOBAL_STEP: 1700650 | > loss_text_ce: 0.029640566557645798 (0.029640697000471356) | > loss_mel_ce: 2.7033488750457764 (nan) | > loss: 0.01084519736468792 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13655810095816254) | > loader_time: 0.0046 (0.004729290618083765)  --> TIME: 2025-02-27 12:35:19 -- STEP: 1552/36938 -- GLOBAL_STEP: 1700700 | > loss_text_ce: 0.04008520394563675 (0.029643394597253965) | > loss_mel_ce: 1.1954063177108765 (nan) | > loss: 0.004902744200080633 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13667826999708538) | > loader_time: 0.0048 (0.004729774809375252)  --> TIME: 2025-02-27 12:35:28 -- STEP: 1602/36938 -- GLOBAL_STEP: 1700750 | > loss_text_ce: 0.027322763577103615 (0.029631490554641954) | > loss_mel_ce: 0.0497233085334301 (nan) | > loss: 0.0003057384165003896 (nan) | > current_lr: 5e-06 | > step_time: 0.1325 (0.1366849951678596) | > loader_time: 0.0048 (0.004727921533525067)  --> TIME: 2025-02-27 12:35:38 -- STEP: 1652/36938 -- GLOBAL_STEP: 1700800 | > loss_text_ce: 0.02270376868546009 (0.029654133957235672) | > loss_mel_ce: 1.553215503692627 (nan) | > loss: 0.006253648083657026 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13693403274037363) | > loader_time: 0.0047 (0.004726636784994575)  --> TIME: 2025-02-27 12:35:48 -- STEP: 1702/36938 -- GLOBAL_STEP: 1700850 | > loss_text_ce: 0.030681496486067772 (0.029669381015453616) | > loss_mel_ce: 2.493419647216797 (nan) | > loss: 0.010016275569796562 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13690450973152138) | > loader_time: 0.0047 (0.0047269024383867694)  --> TIME: 2025-02-27 12:35:57 -- STEP: 1752/36938 -- GLOBAL_STEP: 1700900 | > loss_text_ce: 0.03165096044540405 (0.029691207740927907) | > loss_mel_ce: 2.9443252086639404 (nan) | > loss: 0.011809430085122585 (nan) | > current_lr: 5e-06 | > step_time: 0.1826 (0.13691613391109797) | > loader_time: 0.0048 (0.004724451532102613)  --> TIME: 2025-02-27 12:36:06 -- STEP: 1802/36938 -- GLOBAL_STEP: 1700950 | > loss_text_ce: 0.021186115220189095 (0.029676192348614152) | > loss_mel_ce: 2.6196696758270264 (nan) | > loss: 0.010479587130248547 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.1370153977524295) | > loader_time: 0.0053 (0.004722828076497036)  --> TIME: 2025-02-27 12:36:16 -- STEP: 1852/36938 -- GLOBAL_STEP: 1701000 | > loss_text_ce: 0.024377861991524696 (0.029661935720427794) | > loss_mel_ce: 2.895000457763672 (nan) | > loss: 0.01158483512699604 (nan) | > current_lr: 5e-06 | > step_time: 0.157 (0.13695764708982425) | > loader_time: 0.0054 (0.004724385156487288)  --> TIME: 2025-02-27 12:36:25 -- STEP: 1902/36938 -- GLOBAL_STEP: 1701050 | > loss_text_ce: 0.033469073474407196 (0.02965190190945279) | > loss_mel_ce: 2.697261095046997 (nan) | > loss: 0.010836230590939522 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13695946514418442) | > loader_time: 0.0046 (0.004726532631241312)  --> TIME: 2025-02-27 12:36:34 -- STEP: 1952/36938 -- GLOBAL_STEP: 1701100 | > loss_text_ce: 0.022769847884774208 (0.02964962985162379) | > loss_mel_ce: 2.0095202922821045 (nan) | > loss: 0.008064644411206245 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.13696245165144805) | > loader_time: 0.0045 (0.004725039249560874)  --> TIME: 2025-02-27 12:36:44 -- STEP: 2002/36938 -- GLOBAL_STEP: 1701150 | > loss_text_ce: 0.02877533808350563 (0.029663732931758467) | > loss_mel_ce: 2.8446500301361084 (nan) | > loss: 0.011402483098208904 (nan) | > current_lr: 5e-06 | > step_time: 0.1794 (0.1369548203585507) | > loader_time: 0.0048 (0.00472245052025154)  --> TIME: 2025-02-27 12:36:53 -- STEP: 2052/36938 -- GLOBAL_STEP: 1701200 | > loss_text_ce: 0.038970332592725754 (0.029650818413127118) | > loss_mel_ce: 2.0141053199768066 (nan) | > loss: 0.00814712606370449 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13699042425285765) | > loader_time: 0.0048 (0.004722343899353208)  --> TIME: 2025-02-27 12:37:03 -- STEP: 2102/36938 -- GLOBAL_STEP: 1701250 | > loss_text_ce: 0.03639715537428856 (0.02966596976677567) | > loss_mel_ce: 2.538069486618042 (nan) | > loss: 0.010216138325631618 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13702988284298848) | > loader_time: 0.0047 (0.004727337952458667)  --> TIME: 2025-02-27 12:37:12 -- STEP: 2152/36938 -- GLOBAL_STEP: 1701300 | > loss_text_ce: 0.03098333068192005 (0.02966307385800385) | > loss_mel_ce: 2.1055569648742676 (nan) | > loss: 0.008478335104882717 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.136948279067043) | > loader_time: 0.0047 (0.004729159481020224)  --> TIME: 2025-02-27 12:37:21 -- STEP: 2202/36938 -- GLOBAL_STEP: 1701350 | > loss_text_ce: 0.03817684203386307 (0.029649823787304946) | > loss_mel_ce: 1.368310570716858 (nan) | > loss: 0.005581299774348736 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.136890011414087) | > loader_time: 0.0046 (0.004729238452963349)  --> TIME: 2025-02-27 12:37:30 -- STEP: 2252/36938 -- GLOBAL_STEP: 1701400 | > loss_text_ce: 0.03392613306641579 (0.029659146266008463) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.1368723006697362) | > loader_time: 0.0047 (0.00472887466474706)  --> TIME: 2025-02-27 12:37:40 -- STEP: 2302/36938 -- GLOBAL_STEP: 1701450 | > loss_text_ce: 0.03445808216929436 (0.029674439077677413) | > loss_mel_ce: 3.168429136276245 (nan) | > loss: 0.01270987093448639 (nan) | > current_lr: 5e-06 | > step_time: 0.1383 (0.13687021788258247) | > loader_time: 0.0046 (0.004729667401748778)  --> TIME: 2025-02-27 12:37:49 -- STEP: 2352/36938 -- GLOBAL_STEP: 1701500 | > loss_text_ce: 0.03793312609195709 (0.029668823587011266) | > loss_mel_ce: 2.3323841094970703 (nan) | > loss: 0.009406020864844322 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13681629046696406) | > loader_time: 0.0046 (0.004728638091865852)  --> TIME: 2025-02-27 12:37:58 -- STEP: 2402/36938 -- GLOBAL_STEP: 1701550 | > loss_text_ce: 0.03697498142719269 (0.029678076794906073) | > loss_mel_ce: 3.245847463607788 (nan) | > loss: 0.013027073815464973 (nan) | > current_lr: 5e-06 | > step_time: 0.1261 (0.1366608298688409) | > loader_time: 0.0046 (0.00472740656529537)  --> TIME: 2025-02-27 12:38:07 -- STEP: 2452/36938 -- GLOBAL_STEP: 1701600 | > loss_text_ce: 0.031443264335393906 (0.02965207575086571) | > loss_mel_ce: 0.01318359375 (nan) | > loss: 0.0001770907110767439 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.13666747104479887) | > loader_time: 0.0045 (0.004726313844590558)  --> TIME: 2025-02-27 12:38:16 -- STEP: 2502/36938 -- GLOBAL_STEP: 1701650 | > loss_text_ce: 0.029329461976885796 (0.029656366301442888) | > loss_mel_ce: 2.847200632095337 (nan) | > loss: 0.011414802633225918 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13656943240802288) | > loader_time: 0.0049 (0.0047246446426538)  --> TIME: 2025-02-27 12:38:25 -- STEP: 2552/36938 -- GLOBAL_STEP: 1701700 | > loss_text_ce: 0.024609550833702087 (0.029674315818535822) | > loss_mel_ce: 2.721735954284668 (nan) | > loss: 0.010898197069764137 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.13648051854958757) | > loader_time: 0.0046 (0.004724639990486699)  --> TIME: 2025-02-27 12:38:35 -- STEP: 2602/36938 -- GLOBAL_STEP: 1701750 | > loss_text_ce: 0.03966932371258736 (0.029696108358156947) | > loss_mel_ce: 2.0367090702056885 (nan) | > loss: 0.008239597082138062 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.13649434185687837) | > loader_time: 0.0046 (0.0047244438293070025)  --> TIME: 2025-02-27 12:38:44 -- STEP: 2652/36938 -- GLOBAL_STEP: 1701800 | > loss_text_ce: 0.02353794313967228 (0.029687879239398633) | > loss_mel_ce: 1.698516845703125 (nan) | > loss: 0.006833551451563835 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13658214055934503) | > loader_time: 0.0047 (0.004724708257936969)  --> TIME: 2025-02-27 12:38:54 -- STEP: 2702/36938 -- GLOBAL_STEP: 1701850 | > loss_text_ce: 0.027255991473793983 (0.029688821852877238) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13665140928117409) | > loader_time: 0.0046 (0.004727207052540024)  --> TIME: 2025-02-27 12:39:03 -- STEP: 2752/36938 -- GLOBAL_STEP: 1701900 | > loss_text_ce: 0.029022986069321632 (0.029709985134189102) | > loss_mel_ce: 2.775921106338501 (nan) | > loss: 0.011130730621516705 (nan) | > current_lr: 5e-06 | > step_time: 0.1768 (0.13673433488191591) | > loader_time: 0.0048 (0.004726646078187368)  --> TIME: 2025-02-27 12:39:13 -- STEP: 2802/36938 -- GLOBAL_STEP: 1701950 | > loss_text_ce: 0.03141455724835396 (0.02970711494045469) | > loss_mel_ce: 2.746488094329834 (nan) | > loss: 0.011023423634469509 (nan) | > current_lr: 5e-06 | > step_time: 0.1831 (0.1367026956314536) | > loader_time: 0.0057 (0.004727665991718479)  --> TIME: 2025-02-27 12:39:22 -- STEP: 2852/36938 -- GLOBAL_STEP: 1702000 | > loss_text_ce: 0.022239116951823235 (0.029692889750591606) | > loss_mel_ce: 2.1397860050201416 (nan) | > loss: 0.008579465560615063 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13670493827461389) | > loader_time: 0.0065 (0.004729907860428228)  --> TIME: 2025-02-27 12:39:31 -- STEP: 2902/36938 -- GLOBAL_STEP: 1702050 | > loss_text_ce: 0.03231244906783104 (0.029687259545540766) | > loss_mel_ce: 2.8922934532165527 (nan) | > loss: 0.011605579406023026 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13664850064756812) | > loader_time: 0.0046 (0.004729570149553806)  --> TIME: 2025-02-27 12:39:40 -- STEP: 2952/36938 -- GLOBAL_STEP: 1702100 | > loss_text_ce: 0.025562221184372902 (0.029685285058803856) | > loss_mel_ce: 2.633673667907715 (nan) | > loss: 0.010552524589002132 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13671434353683723) | > loader_time: 0.0048 (0.004729100278399507)  --> TIME: 2025-02-27 12:39:50 -- STEP: 3002/36938 -- GLOBAL_STEP: 1702150 | > loss_text_ce: 0.03756571561098099 (0.029708915082011855) | > loss_mel_ce: 2.7388789653778076 (nan) | > loss: 0.011017638258635998 (nan) | > current_lr: 5e-06 | > step_time: 0.1701 (0.136759352874629) | > loader_time: 0.0049 (0.004729737129948452)  --> TIME: 2025-02-27 12:39:59 -- STEP: 3052/36938 -- GLOBAL_STEP: 1702200 | > loss_text_ce: 0.04312615841627121 (0.029708417684821166) | > loss_mel_ce: 2.5935239791870117 (nan) | > loss: 0.010462897829711437 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13671454267764327) | > loader_time: 0.0045 (0.004728553257638483)  --> TIME: 2025-02-27 12:40:09 -- STEP: 3102/36938 -- GLOBAL_STEP: 1702250 | > loss_text_ce: 0.029695384204387665 (0.029712040018143766) | > loss_mel_ce: 0.8715340495109558 (nan) | > loss: 0.0035763075575232506 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13672627509601054) | > loader_time: 0.0047 (0.004728467828761689)  --> TIME: 2025-02-27 12:40:17 -- STEP: 3152/36938 -- GLOBAL_STEP: 1702300 | > loss_text_ce: 0.026119330897927284 (0.0297102213631559) | > loss_mel_ce: 3.4625356197357178 (nan) | > loss: 0.013843868859112263 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13662715770568967) | > loader_time: 0.0045 (0.004727693529903599)  --> TIME: 2025-02-27 12:40:27 -- STEP: 3202/36938 -- GLOBAL_STEP: 1702350 | > loss_text_ce: 0.02722455933690071 (0.029719391854585782) | > loss_mel_ce: 0.8766675591468811 (nan) | > loss: 0.003586873644962907 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.13660949315971188) | > loader_time: 0.0046 (0.004726418549384581)  --> TIME: 2025-02-27 12:40:36 -- STEP: 3252/36938 -- GLOBAL_STEP: 1702400 | > loss_text_ce: 0.036223869770765305 (0.02972594189066967) | > loss_mel_ce: 0.0212249755859375 (nan) | > loss: 0.00022797162819188088 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13656337337446994) | > loader_time: 0.0045 (0.004725449932750603)  --> TIME: 2025-02-27 12:40:45 -- STEP: 3302/36938 -- GLOBAL_STEP: 1702450 | > loss_text_ce: 0.0323968268930912 (0.029721849789219074) | > loss_mel_ce: 2.09814453125 (nan) | > loss: 0.008454529568552971 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13661063800502168) | > loader_time: 0.0046 (0.004725913435383187)  --> TIME: 2025-02-27 12:40:54 -- STEP: 3352/36938 -- GLOBAL_STEP: 1702500 | > loss_text_ce: 0.03025316633284092 (0.029705352356339535) | > loss_mel_ce: 2.0278732776641846 (nan) | > loss: 0.00816716905683279 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13657033863101775) | > loader_time: 0.0046 (0.004726473215371735)  --> TIME: 2025-02-27 12:41:04 -- STEP: 3402/36938 -- GLOBAL_STEP: 1702550 | > loss_text_ce: 0.02928110957145691 (0.029714606897516732) | > loss_mel_ce: 2.818822145462036 (nan) | > loss: 0.011301998049020767 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.13665181752305258) | > loader_time: 0.0049 (0.004727967201717322)  --> TIME: 2025-02-27 12:41:13 -- STEP: 3452/36938 -- GLOBAL_STEP: 1702600 | > loss_text_ce: 0.039374999701976776 (0.029726826499765557) | > loss_mel_ce: 0.006866455078125 (nan) | > loss: 0.00018349784659221768 (nan) | > current_lr: 5e-06 | > step_time: 0.1144 (0.13658997944857254) | > loader_time: 0.0046 (0.00472757354948502)  --> TIME: 2025-02-27 12:41:22 -- STEP: 3502/36938 -- GLOBAL_STEP: 1702650 | > loss_text_ce: 0.028167540207505226 (0.029726949772254475) | > loss_mel_ce: 2.8175318241119385 (nan) | > loss: 0.01129245851188898 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13660694965970363) | > loader_time: 0.0045 (0.004726863942236174)  --> TIME: 2025-02-27 12:41:32 -- STEP: 3552/36938 -- GLOBAL_STEP: 1702700 | > loss_text_ce: 0.036761295050382614 (0.029709386868719094) | > loss_mel_ce: 3.1517980098724365 (nan) | > loss: 0.012653013691306114 (nan) | > current_lr: 5e-06 | > step_time: 0.1508 (0.1366244174607164) | > loader_time: 0.0047 (0.004727608717239661)  --> TIME: 2025-02-27 12:41:41 -- STEP: 3602/36938 -- GLOBAL_STEP: 1702750 | > loss_text_ce: 0.03489740937948227 (0.02971486516502675) | > loss_mel_ce: 2.5219850540161133 (nan) | > loss: 0.010146358981728554 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.1365744252128112) | > loader_time: 0.0046 (0.0047266354897100415)  --> TIME: 2025-02-27 12:41:50 -- STEP: 3652/36938 -- GLOBAL_STEP: 1702800 | > loss_text_ce: 0.032396912574768066 (0.029728593169339207) | > loss_mel_ce: 2.434044122695923 (nan) | > loss: 0.009787465445697308 (nan) | > current_lr: 5e-06 | > step_time: 0.1279 (0.13653797159320274) | > loader_time: 0.0046 (0.004725711630312538)  --> TIME: 2025-02-27 12:42:00 -- STEP: 3702/36938 -- GLOBAL_STEP: 1702850 | > loss_text_ce: 0.029619140550494194 (0.029732013506405277) | > loss_mel_ce: 0.0776519775390625 (nan) | > loss: 0.00042567908531054854 (nan) | > current_lr: 5e-06 | > step_time: 0.1109 (0.13656939924375075) | > loader_time: 0.0046 (0.0047257796678589516)  --> TIME: 2025-02-27 12:42:09 -- STEP: 3752/36938 -- GLOBAL_STEP: 1702900 | > loss_text_ce: 0.025556186214089394 (0.029746294524748997) | > loss_mel_ce: 2.7143168449401855 (nan) | > loss: 0.010872512124478817 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.1365496330042634) | > loader_time: 0.0047 (0.0047265547297910895)  --> TIME: 2025-02-27 12:42:18 -- STEP: 3802/36938 -- GLOBAL_STEP: 1702950 | > loss_text_ce: 0.02304997108876705 (0.02977227470149207) | > loss_mel_ce: 1.386938452720642 (nan) | > loss: 0.005595192313194275 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13659044581297347) | > loader_time: 0.0046 (0.004726131610780066)  --> TIME: 2025-02-27 12:42:28 -- STEP: 3852/36938 -- GLOBAL_STEP: 1703000 | > loss_text_ce: 0.033750105649232864 (0.02979284776679336) | > loss_mel_ce: 1.719268798828125 (nan) | > loss: 0.006956424564123154 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.1365510508155027) | > loader_time: 0.0048 (0.0047264168443214415)  --> TIME: 2025-02-27 12:42:37 -- STEP: 3902/36938 -- GLOBAL_STEP: 1703050 | > loss_text_ce: 0.032438237220048904 (0.029783395962977733) | > loss_mel_ce: 2.507655620574951 (nan) | > loss: 0.010079738683998585 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13657306231699584) | > loader_time: 0.0046 (0.00472647962907472)  --> TIME: 2025-02-27 12:42:46 -- STEP: 3952/36938 -- GLOBAL_STEP: 1703100 | > loss_text_ce: 0.034971389919519424 (0.02978888550690311) | > loss_mel_ce: 1.9144262075424194 (nan) | > loss: 0.007735705003142357 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13653054386980587) | > loader_time: 0.0046 (0.004725713297905715)  --> TIME: 2025-02-27 12:42:55 -- STEP: 4002/36938 -- GLOBAL_STEP: 1703150 | > loss_text_ce: 0.03696047142148018 (0.029788066306780242) | > loss_mel_ce: 2.9865221977233887 (nan) | > loss: 0.011997947469353676 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13650955247140267) | > loader_time: 0.0054 (0.004727932585888305)  --> TIME: 2025-02-27 12:43:05 -- STEP: 4052/36938 -- GLOBAL_STEP: 1703200 | > loss_text_ce: 0.027629122138023376 (0.02977865175523327) | > loss_mel_ce: 2.3626601696014404 (nan) | > loss: 0.009485275484621525 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.1365287692266764) | > loader_time: 0.0048 (0.004729155550351388)  --> TIME: 2025-02-27 12:43:14 -- STEP: 4102/36938 -- GLOBAL_STEP: 1703250 | > loss_text_ce: 0.030500832945108414 (0.029773232498015938) | > loss_mel_ce: 2.8569202423095703 (nan) | > loss: 0.011458020657300949 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13650434537959147) | > loader_time: 0.0046 (0.004729508830535016)  --> TIME: 2025-02-27 12:43:23 -- STEP: 4152/36938 -- GLOBAL_STEP: 1703300 | > loss_text_ce: 0.020518619567155838 (0.029781812940130232) | > loss_mel_ce: 0.267242431640625 (nan) | > loss: 0.0011419090442359447 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13650838390480777) | > loader_time: 0.0046 (0.004729122669940284)  --> TIME: 2025-02-27 12:43:32 -- STEP: 4202/36938 -- GLOBAL_STEP: 1703350 | > loss_text_ce: 0.027617456391453743 (0.029771038232684167) | > loss_mel_ce: 2.381747245788574 (nan) | > loss: 0.009560971520841122 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13646638904055858) | > loader_time: 0.0046 (0.004727486029855986)  --> TIME: 2025-02-27 12:43:42 -- STEP: 4252/36938 -- GLOBAL_STEP: 1703400 | > loss_text_ce: 0.02986912988126278 (0.029785878558994495) | > loss_mel_ce: 2.6116857528686523 (nan) | > loss: 0.010482360608875751 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13647903155742197) | > loader_time: 0.0046 (0.004727698505092997)  --> TIME: 2025-02-27 12:43:51 -- STEP: 4302/36938 -- GLOBAL_STEP: 1703450 | > loss_text_ce: 0.022053828462958336 (0.029790113531988294) | > loss_mel_ce: 2.6232638359069824 (nan) | > loss: 0.010497292503714561 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.136513257846894) | > loader_time: 0.0048 (0.00472940943064883)  --> TIME: 2025-02-27 12:44:01 -- STEP: 4352/36938 -- GLOBAL_STEP: 1703500 | > loss_text_ce: 0.028374463319778442 (0.029786099341222064) | > loss_mel_ce: 2.2217953205108643 (nan) | > loss: 0.008929245173931122 (nan) | > current_lr: 5e-06 | > step_time: 0.1266 (0.13649128300740396) | > loader_time: 0.0046 (0.004729520036455468)  --> TIME: 2025-02-27 12:44:10 -- STEP: 4402/36938 -- GLOBAL_STEP: 1703550 | > loss_text_ce: 0.02898794785141945 (0.02979534696609879) | > loss_mel_ce: 1.8413618803024292 (nan) | > loss: 0.007422023918479681 (nan) | > current_lr: 5e-06 | > step_time: 0.1659 (0.13655876728796143) | > loader_time: 0.0048 (0.004728824103328116)  --> TIME: 2025-02-27 12:44:19 -- STEP: 4452/36938 -- GLOBAL_STEP: 1703600 | > loss_text_ce: 0.03168674185872078 (0.029797611307095063) | > loss_mel_ce: 2.4830355644226074 (nan) | > loss: 0.009979057125747204 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13652360241153874) | > loader_time: 0.0045 (0.004727519318956792)  --> TIME: 2025-02-27 12:44:29 -- STEP: 4502/36938 -- GLOBAL_STEP: 1703650 | > loss_text_ce: 0.032927680760622025 (0.029807165491811338) | > loss_mel_ce: 2.2597737312316895 (nan) | > loss: 0.009098022244870663 (nan) | > current_lr: 5e-06 | > step_time: 0.1767 (0.13652337084659585) | > loader_time: 0.0048 (0.004726931340214629)  --> TIME: 2025-02-27 12:44:38 -- STEP: 4552/36938 -- GLOBAL_STEP: 1703700 | > loss_text_ce: 0.03299618512392044 (0.029798236692541977) | > loss_mel_ce: 2.7638025283813477 (nan) | > loss: 0.0110984081402421 (nan) | > current_lr: 5e-06 | > step_time: 0.1508 (0.1365266901118471) | > loader_time: 0.0047 (0.004726266190540608)  --> TIME: 2025-02-27 12:44:47 -- STEP: 4602/36938 -- GLOBAL_STEP: 1703750 | > loss_text_ce: 0.03212147578597069 (0.029801146871005973) | > loss_mel_ce: 2.5336718559265137 (nan) | > loss: 0.010181720368564129 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13652753296337733) | > loader_time: 0.0047 (0.00472567419236974)  --> TIME: 2025-02-27 12:44:56 -- STEP: 4652/36938 -- GLOBAL_STEP: 1703800 | > loss_text_ce: 0.02885347232222557 (0.029800927593002) | > loss_mel_ce: 2.289510488510132 (nan) | > loss: 0.009199857711791992 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13649496602684047) | > loader_time: 0.0046 (0.004724392323071657)  --> TIME: 2025-02-27 12:45:05 -- STEP: 4702/36938 -- GLOBAL_STEP: 1703850 | > loss_text_ce: 0.02502097748219967 (0.02979488932178604) | > loss_mel_ce: 0.00421142578125 (nan) | > loss: 0.00011600160360103473 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13644407197091463) | > loader_time: 0.0045 (0.004722696068640829)  --> TIME: 2025-02-27 12:45:15 -- STEP: 4752/36938 -- GLOBAL_STEP: 1703900 | > loss_text_ce: 0.027999507263302803 (0.02979608945041325) | > loss_mel_ce: 2.2680342197418213 (nan) | > loss: 0.009111245162785053 (nan) | > current_lr: 5e-06 | > step_time: 0.16 (0.1364800238147727) | > loader_time: 0.0048 (0.0047219231574222404)  --> TIME: 2025-02-27 12:45:24 -- STEP: 4802/36938 -- GLOBAL_STEP: 1703950 | > loss_text_ce: 0.02346915751695633 (0.02980098986039338) | > loss_mel_ce: 3.0262951850891113 (nan) | > loss: 0.012102240696549416 (nan) | > current_lr: 5e-06 | > step_time: 0.1684 (0.136493650737875) | > loader_time: 0.0047 (0.004721510762822783)  --> TIME: 2025-02-27 12:45:34 -- STEP: 4852/36938 -- GLOBAL_STEP: 1704000 | > loss_text_ce: 0.04187547415494919 (0.029807351634208876) | > loss_mel_ce: 0.00543212890625 (nan) | > loss: 0.00018772859766613692 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13647887280236937) | > loader_time: 0.0046 (0.00472112460659245)  --> TIME: 2025-02-27 12:45:43 -- STEP: 4902/36938 -- GLOBAL_STEP: 1704050 | > loss_text_ce: 0.032451942563056946 (0.029802353623479317) | > loss_mel_ce: 3.025841474533081 (nan) | > loss: 0.012136084958910942 (nan) | > current_lr: 5e-06 | > step_time: 0.1695 (0.1365125684629211) | > loader_time: 0.0048 (0.004721180072661478)  --> TIME: 2025-02-27 12:45:53 -- STEP: 4952/36938 -- GLOBAL_STEP: 1704100 | > loss_text_ce: 0.03221014887094498 (0.029816274771727144) | > loss_mel_ce: 2.4868361949920654 (nan) | > loss: 0.00999621581286192 (nan) | > current_lr: 5e-06 | > step_time: 0.1565 (0.1365543314058676) | > loader_time: 0.0047 (0.004721766719910566)  --> TIME: 2025-02-27 12:46:02 -- STEP: 5002/36938 -- GLOBAL_STEP: 1704150 | > loss_text_ce: 0.026871107518672943 (0.02981653244739942) | > loss_mel_ce: 1.7476806640625 (nan) | > loss: 0.007041872479021549 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13650908054518052) | > loader_time: 0.0047 (0.004721211081073561)  --> TIME: 2025-02-27 12:46:11 -- STEP: 5052/36938 -- GLOBAL_STEP: 1704200 | > loss_text_ce: 0.030156869441270828 (0.029818462088272005) | > loss_mel_ce: 2.294480800628662 (nan) | > loss: 0.009224752895534039 (nan) | > current_lr: 5e-06 | > step_time: 0.1103 (0.13646205563632047) | > loader_time: 0.0045 (0.004720733926490675)  --> TIME: 2025-02-27 12:46:20 -- STEP: 5102/36938 -- GLOBAL_STEP: 1704250 | > loss_text_ce: 0.034311696887016296 (0.029804071378633123) | > loss_mel_ce: 2.149733781814575 (nan) | > loss: 0.008666847832500935 (nan) | > current_lr: 5e-06 | > step_time: 0.137 (0.1364027736047723) | > loader_time: 0.0046 (0.004719774753895055)  --> TIME: 2025-02-27 12:46:29 -- STEP: 5152/36938 -- GLOBAL_STEP: 1704300 | > loss_text_ce: 0.03465265408158302 (0.029795284360583575) | > loss_mel_ce: 3.1162109375 (nan) | > loss: 0.0125034274533391 (nan) | > current_lr: 5e-06 | > step_time: 0.114 (0.1364055555892286) | > loader_time: 0.0045 (0.004719026849506808)  --> TIME: 2025-02-27 12:46:39 -- STEP: 5202/36938 -- GLOBAL_STEP: 1704350 | > loss_text_ce: 0.03127998113632202 (0.02981254060511714) | > loss_mel_ce: 2.8079679012298584 (nan) | > loss: 0.011266857385635376 (nan) | > current_lr: 5e-06 | > step_time: 0.1614 (0.13645155949576088) | > loader_time: 0.0049 (0.004718343508513971)  --> TIME: 2025-02-27 12:46:48 -- STEP: 5252/36938 -- GLOBAL_STEP: 1704400 | > loss_text_ce: 0.03272715583443642 (0.02980177389544358) | > loss_mel_ce: 3.048917531967163 (nan) | > loss: 0.012228749692440033 (nan) | > current_lr: 5e-06 | > step_time: 0.1466 (0.13648419891344352) | > loader_time: 0.0046 (0.004717779904036513)  --> TIME: 2025-02-27 12:46:58 -- STEP: 5302/36938 -- GLOBAL_STEP: 1704450 | > loss_text_ce: 0.0253889262676239 (0.029799467311895932) | > loss_mel_ce: 2.229280710220337 (nan) | > loss: 0.008947102352976799 (nan) | > current_lr: 5e-06 | > step_time: 0.1398 (0.13646946188100367) | > loader_time: 0.0046 (0.004716559699327258)  --> TIME: 2025-02-27 12:47:06 -- STEP: 5352/36938 -- GLOBAL_STEP: 1704500 | > loss_text_ce: 0.02638734132051468 (0.029785823631545836) | > loss_mel_ce: 2.509626865386963 (nan) | > loss: 0.010063549503684044 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.1363834334684951) | > loader_time: 0.0046 (0.004715040125654412)  --> TIME: 2025-02-27 12:47:15 -- STEP: 5402/36938 -- GLOBAL_STEP: 1704550 | > loss_text_ce: 0.028557879850268364 (0.029776811219927) | > loss_mel_ce: 2.6382153034210205 (nan) | > loss: 0.010582433082163334 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.13635294576875814) | > loader_time: 0.0045 (0.004713900792779675)  --> TIME: 2025-02-27 12:47:25 -- STEP: 5452/36938 -- GLOBAL_STEP: 1704600 | > loss_text_ce: 0.041449617594480515 (0.029770428670286012) | > loss_mel_ce: 1.78271484375 (nan) | > loss: 0.007238748483359814 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1363444789664673) | > loader_time: 0.0046 (0.00471299493129167)  --> TIME: 2025-02-27 12:47:34 -- STEP: 5502/36938 -- GLOBAL_STEP: 1704650 | > loss_text_ce: 0.025736071169376373 (0.029770310275000704) | > loss_mel_ce: 2.3654286861419678 (nan) | > loss: 0.00948874931782484 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.13634446073731082) | > loader_time: 0.0046 (0.004712074550876867)  --> TIME: 2025-02-27 12:47:43 -- STEP: 5552/36938 -- GLOBAL_STEP: 1704700 | > loss_text_ce: 0.02754189632833004 (0.029765720608857564) | > loss_mel_ce: 2.6653964519500732 (nan) | > loss: 0.01068626344203949 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.1363643338254273) | > loader_time: 0.0045 (0.004711214334889167)  --> TIME: 2025-02-27 12:47:53 -- STEP: 5602/36938 -- GLOBAL_STEP: 1704750 | > loss_text_ce: 0.022937024012207985 (0.029774346300432387) | > loss_mel_ce: 2.3585331439971924 (nan) | > loss: 0.009450279176235199 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.1363614713749854) | > loader_time: 0.0046 (0.004710506260799358)  --> TIME: 2025-02-27 12:48:02 -- STEP: 5652/36938 -- GLOBAL_STEP: 1704800 | > loss_text_ce: 0.02026078850030899 (0.029772582679943458) | > loss_mel_ce: 2.111419439315796 (nan) | > loss: 0.00845904927700758 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1364092751335607) | > loader_time: 0.0048 (0.004709965779438813)  --> TIME: 2025-02-27 12:48:12 -- STEP: 5702/36938 -- GLOBAL_STEP: 1704850 | > loss_text_ce: 0.02511724643409252 (0.029778997036679064) | > loss_mel_ce: 2.3850600719451904 (nan) | > loss: 0.00956419575959444 (nan) | > current_lr: 5e-06 | > step_time: 0.1349 (0.13642045565966077) | > loader_time: 0.0046 (0.0047091396178583645)  --> TIME: 2025-02-27 12:48:21 -- STEP: 5752/36938 -- GLOBAL_STEP: 1704900 | > loss_text_ce: 0.0350317619740963 (0.02978499574435847) | > loss_mel_ce: 2.9104995727539062 (nan) | > loss: 0.011688617058098316 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.13640986882429934) | > loader_time: 0.0048 (0.004708257479196787)  --> TIME: 2025-02-27 12:48:30 -- STEP: 5802/36938 -- GLOBAL_STEP: 1704950 | > loss_text_ce: 0.0233309268951416 (0.029789953901740975) | > loss_mel_ce: 2.9184000492095947 (nan) | > loss: 0.011673536151647568 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13641239942578912) | > loader_time: 0.0047 (0.004707400365682839)  --> TIME: 2025-02-27 12:48:39 -- STEP: 5852/36938 -- GLOBAL_STEP: 1705000 | > loss_text_ce: 0.03404196351766586 (0.029778740562941636) | > loss_mel_ce: 3.255190849304199 (nan) | > loss: 0.01305251196026802 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.1364055180174799) | > loader_time: 0.0046 (0.004709000874282713)  --> TIME: 2025-02-27 12:48:49 -- STEP: 5902/36938 -- GLOBAL_STEP: 1705050 | > loss_text_ce: 0.02949868142604828 (0.02977644776883155) | > loss_mel_ce: 2.9108448028564453 (nan) | > loss: 0.011668030172586441 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13637653608234873) | > loader_time: 0.0046 (0.004708118092928921)  --> TIME: 2025-02-27 12:48:58 -- STEP: 5952/36938 -- GLOBAL_STEP: 1705100 | > loss_text_ce: 0.030913714319467545 (0.029775076919117627) | > loss_mel_ce: 2.762939691543579 (nan) | > loss: 0.01108672097325325 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.13635266363941187) | > loader_time: 0.0048 (0.004708087372203018)  --> TIME: 2025-02-27 12:49:07 -- STEP: 6002/36938 -- GLOBAL_STEP: 1705150 | > loss_text_ce: 0.01689773052930832 (0.029768820475912304) | > loss_mel_ce: 2.5583741664886475 (nan) | > loss: 0.010219333693385124 (nan) | > current_lr: 5e-06 | > step_time: 0.159 (0.13637077772629233) | > loader_time: 0.0053 (0.004707964528842349)  --> TIME: 2025-02-27 12:49:17 -- STEP: 6052/36938 -- GLOBAL_STEP: 1705200 | > loss_text_ce: 0.031529296189546585 (0.029766277624608095) | > loss_mel_ce: 2.790388584136963 (nan) | > loss: 0.01119808666408062 (nan) | > current_lr: 5e-06 | > step_time: 0.1471 (0.13640622760157495) | > loader_time: 0.0046 (0.004708265438823602)  --> TIME: 2025-02-27 12:49:26 -- STEP: 6102/36938 -- GLOBAL_STEP: 1705250 | > loss_text_ce: 0.03217463567852974 (0.029770757766426473) | > loss_mel_ce: 1.6504228115081787 (nan) | > loss: 0.00667697424069047 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13640678288388805) | > loader_time: 0.0047 (0.00470858720512477)  --> TIME: 2025-02-27 12:49:35 -- STEP: 6152/36938 -- GLOBAL_STEP: 1705300 | > loss_text_ce: 0.04086633026599884 (0.02976407830498057) | > loss_mel_ce: 2.996816635131836 (nan) | > loss: 0.01205429807305336 (nan) | > current_lr: 5e-06 | > step_time: 0.1369 (0.1364196069606413) | > loader_time: 0.0046 (0.004708557739679457)  --> TIME: 2025-02-27 12:49:45 -- STEP: 6202/36938 -- GLOBAL_STEP: 1705350 | > loss_text_ce: 0.03837105259299278 (0.029757414117061454) | > loss_mel_ce: 1.842086911201477 (nan) | > loss: 0.007462135516107082 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13647515667672686) | > loader_time: 0.0046 (0.004708044915689799)  --> TIME: 2025-02-27 12:49:54 -- STEP: 6252/36938 -- GLOBAL_STEP: 1705400 | > loss_text_ce: 0.02569560706615448 (0.02974902841969205) | > loss_mel_ce: 3.0136826038360596 (nan) | > loss: 0.012061025016009808 (nan) | > current_lr: 5e-06 | > step_time: 0.1793 (0.1364844589376785) | > loader_time: 0.0048 (0.004707722769131354)  --> TIME: 2025-02-27 12:50:04 -- STEP: 6302/36938 -- GLOBAL_STEP: 1705450 | > loss_text_ce: 0.02232826128602028 (0.029741583162518497) | > loss_mel_ce: 1.9598783254623413 (nan) | > loss: 0.007865899242460728 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13645654430619414) | > loader_time: 0.0045 (0.004707013641376789)  --> TIME: 2025-02-27 12:50:13 -- STEP: 6352/36938 -- GLOBAL_STEP: 1705500 | > loss_text_ce: 0.031735606491565704 (0.02974745954644353) | > loss_mel_ce: 2.5203540325164795 (nan) | > loss: 0.010127340443432331 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.13646959297602954) | > loader_time: 0.0048 (0.004706425428090224)  --> TIME: 2025-02-27 12:50:22 -- STEP: 6402/36938 -- GLOBAL_STEP: 1705550 | > loss_text_ce: 0.032089851796627045 (0.029747724714486754) | > loss_mel_ce: 2.7741689682006836 (nan) | > loss: 0.011135948821902275 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13646094946517143) | > loader_time: 0.0047 (0.004705657403344693)  --> TIME: 2025-02-27 12:50:31 -- STEP: 6452/36938 -- GLOBAL_STEP: 1705600 | > loss_text_ce: 0.03609354794025421 (0.029738543163961923) | > loss_mel_ce: 2.501030445098877 (nan) | > loss: 0.010067952796816826 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.13642167113836837) | > loader_time: 0.0046 (0.004704790645996675)  --> TIME: 2025-02-27 12:50:40 -- STEP: 6502/36938 -- GLOBAL_STEP: 1705650 | > loss_text_ce: 0.04111934453248978 (0.029745161445695088) | > loss_mel_ce: 0.16646575927734375 (nan) | > loss: 0.0008237504516728222 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13641067293378883) | > loader_time: 0.0045 (0.004704027607124858)  --> TIME: 2025-02-27 12:50:49 -- STEP: 6552/36938 -- GLOBAL_STEP: 1705700 | > loss_text_ce: 0.02523304894566536 (0.029735790198027492) | > loss_mel_ce: 2.70511531829834 (nan) | > loss: 0.010834716260433197 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13635803084088155) | > loader_time: 0.0048 (0.004703596943900688)  --> TIME: 2025-02-27 12:50:58 -- STEP: 6602/36938 -- GLOBAL_STEP: 1705750 | > loss_text_ce: 0.024121107533574104 (0.029726249721071664) | > loss_mel_ce: 0.0146484375 (nan) | > loss: 0.00015384740254376084 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13629913293965007) | > loader_time: 0.0045 (0.004703277423503858)  --> TIME: 2025-02-27 12:51:07 -- STEP: 6652/36938 -- GLOBAL_STEP: 1705800 | > loss_text_ce: 0.025901779532432556 (0.029723012884569168) | > loss_mel_ce: 1.8366495370864868 (nan) | > loss: 0.007391077000647783 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13630187357325016) | > loader_time: 0.0047 (0.004703050769386123)  --> TIME: 2025-02-27 12:51:16 -- STEP: 6702/36938 -- GLOBAL_STEP: 1705850 | > loss_text_ce: 0.029736988246440887 (0.029722375040960047) | > loss_mel_ce: 3.440805196762085 (nan) | > loss: 0.013771994039416313 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13628695926108442) | > loader_time: 0.0048 (0.004702944749862311)  --> TIME: 2025-02-27 12:51:26 -- STEP: 6752/36938 -- GLOBAL_STEP: 1705900 | > loss_text_ce: 0.03247097507119179 (0.02971935495157656) | > loss_mel_ce: 2.5685741901397705 (nan) | > loss: 0.010321607813239098 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.1362904095889831) | > loader_time: 0.0046 (0.004702835321708867)  --> TIME: 2025-02-27 12:51:35 -- STEP: 6802/36938 -- GLOBAL_STEP: 1705950 | > loss_text_ce: 0.028868865221738815 (0.029723214971652475) | > loss_mel_ce: 2.2652859687805176 (nan) | > loss: 0.009103789925575256 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13629537945247946) | > loader_time: 0.0052 (0.004702567248020537)  --> TIME: 2025-02-27 12:51:44 -- STEP: 6852/36938 -- GLOBAL_STEP: 1706000 | > loss_text_ce: 0.03675401210784912 (0.029724038124353053) | > loss_mel_ce: 1.9085807800292969 (nan) | > loss: 0.007719582878053188 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13628856261014508) | > loader_time: 0.0046 (0.004702547211711046)  --> TIME: 2025-02-27 12:51:54 -- STEP: 6902/36938 -- GLOBAL_STEP: 1706050 | > loss_text_ce: 0.026774048805236816 (0.029716063918687797) | > loss_mel_ce: 3.083557367324829 (nan) | > loss: 0.012342586182057858 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13626937270682613) | > loader_time: 0.0045 (0.004701849102317602)  --> TIME: 2025-02-27 12:52:03 -- STEP: 6952/36938 -- GLOBAL_STEP: 1706100 | > loss_text_ce: 0.03555595502257347 (0.02972052154901263) | > loss_mel_ce: 2.651400327682495 (nan) | > loss: 0.010662524960935116 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.1362985830038132) | > loader_time: 0.0045 (0.004701876016151496)  --> TIME: 2025-02-27 12:52:13 -- STEP: 7002/36938 -- GLOBAL_STEP: 1706150 | > loss_text_ce: 0.03493164852261543 (0.029720425759664278) | > loss_mel_ce: 1.7000746726989746 (nan) | > loss: 0.006884946022182703 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.1363210194249111) | > loader_time: 0.0048 (0.00470168568072608)  --> TIME: 2025-02-27 12:52:22 -- STEP: 7052/36938 -- GLOBAL_STEP: 1706200 | > loss_text_ce: 0.03118518553674221 (0.029720662302357138) | > loss_mel_ce: 2.780775308609009 (nan) | > loss: 0.011158573441207409 (nan) | > current_lr: 5e-06 | > step_time: 0.1361 (0.13633073376987134) | > loader_time: 0.0046 (0.004701592269027621)  --> TIME: 2025-02-27 12:52:31 -- STEP: 7102/36938 -- GLOBAL_STEP: 1706250 | > loss_text_ce: 0.03806373476982117 (0.029731229906893853) | > loss_mel_ce: 2.4123494625091553 (nan) | > loss: 0.009723862633109093 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.1363152165172533) | > loader_time: 0.0045 (0.004701209350291579)  --> TIME: 2025-02-27 12:52:40 -- STEP: 7152/36938 -- GLOBAL_STEP: 1706300 | > loss_text_ce: 0.02875853143632412 (0.029732857026723354) | > loss_mel_ce: 3.2481162548065186 (nan) | > loss: 0.013003472238779068 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13631308348786872) | > loader_time: 0.0045 (0.004700988064409625)  --> TIME: 2025-02-27 12:52:50 -- STEP: 7202/36938 -- GLOBAL_STEP: 1706350 | > loss_text_ce: 0.023061776533722878 (0.029739722697583095) | > loss_mel_ce: 2.3830857276916504 (nan) | > loss: 0.009548204950988293 (nan) | > current_lr: 5e-06 | > step_time: 0.1456 (0.1363731615743975) | > loader_time: 0.0047 (0.004701354377172407)  --> TIME: 2025-02-27 12:53:00 -- STEP: 7252/36938 -- GLOBAL_STEP: 1706400 | > loss_text_ce: 0.03447576239705086 (0.02974149288896778) | > loss_mel_ce: 2.991244316101074 (nan) | > loss: 0.01200682669878006 (nan) | > current_lr: 5e-06 | > step_time: 0.151 (0.13638624396037438) | > loader_time: 0.0048 (0.004701322077126763)  --> TIME: 2025-02-27 12:53:09 -- STEP: 7302/36938 -- GLOBAL_STEP: 1706450 | > loss_text_ce: 0.03209049627184868 (0.02973582355282085) | > loss_mel_ce: 2.6412672996520996 (nan) | > loss: 0.01060856319963932 (nan) | > current_lr: 5e-06 | > step_time: 0.1828 (0.13636497241834866) | > loader_time: 0.0048 (0.004701397413098616)  --> TIME: 2025-02-27 12:53:19 -- STEP: 7352/36938 -- GLOBAL_STEP: 1706500 | > loss_text_ce: 0.030805319547653198 (0.029737512593591333) | > loss_mel_ce: 2.4847066402435303 (nan) | > loss: 0.009982191026210785 (nan) | > current_lr: 5e-06 | > step_time: 0.1547 (0.1363984983945433) | > loader_time: 0.0046 (0.004701352125931606)  --> TIME: 2025-02-27 12:53:28 -- STEP: 7402/36938 -- GLOBAL_STEP: 1706550 | > loss_text_ce: 0.029086986556649208 (0.029736107611997017) | > loss_mel_ce: 2.999208450317383 (nan) | > loss: 0.012017046101391315 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1363815071453305) | > loader_time: 0.0045 (0.004700537019083857)  --> TIME: 2025-02-27 12:53:37 -- STEP: 7452/36938 -- GLOBAL_STEP: 1706600 | > loss_text_ce: 0.029815565794706345 (0.029728892375902545) | > loss_mel_ce: 0.2996608316898346 (nan) | > loss: 0.0013074460439383984 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13638125346414587) | > loader_time: 0.0048 (0.004699930412693802)  --> TIME: 2025-02-27 12:53:46 -- STEP: 7502/36938 -- GLOBAL_STEP: 1706650 | > loss_text_ce: 0.027543742209672928 (0.029725086966753446) | > loss_mel_ce: 0.1070709228515625 (nan) | > loss: 0.0005341852083802223 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.1363890719585367) | > loader_time: 0.0047 (0.004699954316635752)  --> TIME: 2025-02-27 12:53:56 -- STEP: 7552/36938 -- GLOBAL_STEP: 1706700 | > loss_text_ce: 0.022935576736927032 (0.029713837580209783) | > loss_mel_ce: 1.9913709163665771 (nan) | > loss: 0.007993279956281185 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13637175761415737) | > loader_time: 0.0047 (0.004699836027319138)  --> TIME: 2025-02-27 12:54:05 -- STEP: 7602/36938 -- GLOBAL_STEP: 1706750 | > loss_text_ce: 0.026496369391679764 (0.02972490916183472) | > loss_mel_ce: 1.2872741222381592 (nan) | > loss: 0.005213375203311443 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.13636010987167077) | > loader_time: 0.0046 (0.0046997907694249705)  --> TIME: 2025-02-27 12:54:15 -- STEP: 7652/36938 -- GLOBAL_STEP: 1706800 | > loss_text_ce: 0.03710097074508667 (0.029720183336178416) | > loss_mel_ce: 3.031174898147583 (nan) | > loss: 0.012175698764622211 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.1364090081228359) | > loader_time: 0.0048 (0.004699953457353245)  --> TIME: 2025-02-27 12:54:24 -- STEP: 7702/36938 -- GLOBAL_STEP: 1706850 | > loss_text_ce: 0.032595377415418625 (0.029719489667820016) | > loss_mel_ce: 2.8842647075653076 (nan) | > loss: 0.011574842035770416 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13643118890035685) | > loader_time: 0.0046 (0.0047003190506900595)  --> TIME: 2025-02-27 12:54:33 -- STEP: 7752/36938 -- GLOBAL_STEP: 1706900 | > loss_text_ce: 0.03717748448252678 (0.029723817188368674) | > loss_mel_ce: 2.7725107669830322 (nan) | > loss: 0.011149557307362556 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13643597415480146) | > loader_time: 0.0046 (0.004700421456582027)  --> TIME: 2025-02-27 12:54:43 -- STEP: 7802/36938 -- GLOBAL_STEP: 1706950 | > loss_text_ce: 0.02485468052327633 (0.02972469670493686) | > loss_mel_ce: 2.54445219039917 (nan) | > loss: 0.010195662267506123 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13642783878216835) | > loader_time: 0.0047 (0.004700036972982695)  --> TIME: 2025-02-27 12:54:52 -- STEP: 7852/36938 -- GLOBAL_STEP: 1707000 | > loss_text_ce: 0.03095829486846924 (0.029715292848659928) | > loss_mel_ce: 0.9512187838554382 (nan) | > loss: 0.0038975283969193697 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13641124198586072) | > loader_time: 0.0045 (0.004699659572228195)  --> TIME: 2025-02-27 12:55:01 -- STEP: 7902/36938 -- GLOBAL_STEP: 1707050 | > loss_text_ce: 0.030123459175229073 (0.02972311437081888) | > loss_mel_ce: 2.8175153732299805 (nan) | > loss: 0.011300154961645603 (nan) | > current_lr: 5e-06 | > step_time: 0.1831 (0.1363928032295513) | > loader_time: 0.0051 (0.0046992685426129254)  --> TIME: 2025-02-27 12:55:10 -- STEP: 7952/36938 -- GLOBAL_STEP: 1707100 | > loss_text_ce: 0.024074941873550415 (0.029723495507558484) | > loss_mel_ce: 2.0247626304626465 (nan) | > loss: 0.008130308240652084 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13638161992283118) | > loader_time: 0.0045 (0.004699080912639908)  --> TIME: 2025-02-27 12:55:20 -- STEP: 8002/36938 -- GLOBAL_STEP: 1707150 | > loss_text_ce: 0.0260627344250679 (0.02972548496410143) | > loss_mel_ce: 1.649672269821167 (nan) | > loss: 0.006649742368608713 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1363968737451347) | > loader_time: 0.0045 (0.004698674430313234)  --> TIME: 2025-02-27 12:55:29 -- STEP: 8052/36938 -- GLOBAL_STEP: 1707200 | > loss_text_ce: 0.03248242288827896 (0.0297217077845814) | > loss_mel_ce: 2.491443395614624 (nan) | > loss: 0.01001557894051075 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.1364372398683928) | > loader_time: 0.0052 (0.00469924011339432)  --> TIME: 2025-02-27 12:55:39 -- STEP: 8102/36938 -- GLOBAL_STEP: 1707250 | > loss_text_ce: 0.028652135282754898 (0.029718033794333906) | > loss_mel_ce: 2.7401318550109863 (nan) | > loss: 0.010987238958477974 (nan) | > current_lr: 5e-06 | > step_time: 0.1636 (0.13644939226975827) | > loader_time: 0.0047 (0.004698892959221816)  --> TIME: 2025-02-27 12:55:48 -- STEP: 8152/36938 -- GLOBAL_STEP: 1707300 | > loss_text_ce: 0.03498295694589615 (0.02972715402328637) | > loss_mel_ce: 1.8981623649597168 (nan) | > loss: 0.00767121184617281 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13643032645804787) | > loader_time: 0.0047 (0.004698434042158482)  --> TIME: 2025-02-27 12:55:57 -- STEP: 8202/36938 -- GLOBAL_STEP: 1707350 | > loss_text_ce: 0.03610999137163162 (0.029724125279705468) | > loss_mel_ce: 2.732023239135742 (nan) | > loss: 0.010984656400978565 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.1364516490728377) | > loader_time: 0.0047 (0.004698172774846241)  --> TIME: 2025-02-27 12:56:07 -- STEP: 8252/36938 -- GLOBAL_STEP: 1707400 | > loss_text_ce: 0.0449371337890625 (0.029723737399796703) | > loss_mel_ce: 1.048919677734375 (nan) | > loss: 0.0043407017365098 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13647169591267544) | > loader_time: 0.0045 (0.004698679710671087)  --> TIME: 2025-02-27 12:56:16 -- STEP: 8302/36938 -- GLOBAL_STEP: 1707450 | > loss_text_ce: 0.02491314895451069 (0.029722325450316042) | > loss_mel_ce: 2.1583199501037598 (nan) | > loss: 0.008663623593747616 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13646012316322137) | > loader_time: 0.0047 (0.004698519734295546)  --> TIME: 2025-02-27 12:56:26 -- STEP: 8352/36938 -- GLOBAL_STEP: 1707500 | > loss_text_ce: 0.02990618720650673 (0.029721979329820146) | > loss_mel_ce: 2.5753207206726074 (nan) | > loss: 0.01033820305019617 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13647501458952657) | > loader_time: 0.0048 (0.004698265843464488)  --> TIME: 2025-02-27 12:56:35 -- STEP: 8402/36938 -- GLOBAL_STEP: 1707550 | > loss_text_ce: 0.032127611339092255 (0.029731148678208176) | > loss_mel_ce: 3.15985107421875 (nan) | > loss: 0.012666583061218262 (nan) | > current_lr: 5e-06 | > step_time: 0.1345 (0.136464432431697) | > loader_time: 0.0047 (0.004697899737831404)  --> TIME: 2025-02-27 12:56:44 -- STEP: 8452/36938 -- GLOBAL_STEP: 1707600 | > loss_text_ce: 0.02841622196137905 (0.02973229430496456) | > loss_mel_ce: 2.808133125305176 (nan) | > loss: 0.011256149038672447 (nan) | > current_lr: 5e-06 | > step_time: 0.1884 (0.13646248731915508) | > loader_time: 0.0048 (0.004697827185943567)  --> TIME: 2025-02-27 12:56:53 -- STEP: 8502/36938 -- GLOBAL_STEP: 1707650 | > loss_text_ce: 0.030012667179107666 (0.029730312738627168) | > loss_mel_ce: 2.896618127822876 (nan) | > loss: 0.011613614857196808 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13644517968329622) | > loader_time: 0.0046 (0.004697514180434891)  --> TIME: 2025-02-27 12:57:03 -- STEP: 8552/36938 -- GLOBAL_STEP: 1707700 | > loss_text_ce: 0.022382771596312523 (0.029725127634888555) | > loss_mel_ce: 2.487586259841919 (nan) | > loss: 0.009960195049643517 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13648665782109448) | > loader_time: 0.0046 (0.0046980117558317784)  --> TIME: 2025-02-27 12:57:13 -- STEP: 8602/36938 -- GLOBAL_STEP: 1707750 | > loss_text_ce: 0.039951391518116 (0.029722168530716225) | > loss_mel_ce: 2.2788145542144775 (nan) | > loss: 0.009201452136039734 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13649349327947677) | > loader_time: 0.0047 (0.00469828161963804)  --> TIME: 2025-02-27 12:57:22 -- STEP: 8652/36938 -- GLOBAL_STEP: 1707800 | > loss_text_ce: 0.026722870767116547 (0.029721917935129943) | > loss_mel_ce: 2.9547886848449707 (nan) | > loss: 0.011831396259367466 (nan) | > current_lr: 5e-06 | > step_time: 0.1277 (0.13651344457049594) | > loader_time: 0.0046 (0.004698362523518493)  --> TIME: 2025-02-27 12:57:32 -- STEP: 8702/36938 -- GLOBAL_STEP: 1707850 | > loss_text_ce: 0.02921918034553528 (0.029726707642494257) | > loss_mel_ce: 2.781116008758545 (nan) | > loss: 0.011152124032378197 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13652987231936412) | > loader_time: 0.0046 (0.004698068074757019)  --> TIME: 2025-02-27 12:57:41 -- STEP: 8752/36938 -- GLOBAL_STEP: 1707900 | > loss_text_ce: 0.03039500117301941 (0.02972203448840026) | > loss_mel_ce: 3.21402907371521 (nan) | > loss: 0.01287469919770956 (nan) | > current_lr: 5e-06 | > step_time: 0.1407 (0.13656486279349828) | > loader_time: 0.0047 (0.004697901974866328)  --> TIME: 2025-02-27 12:57:51 -- STEP: 8802/36938 -- GLOBAL_STEP: 1707950 | > loss_text_ce: 0.03743871673941612 (0.02972236308949547) | > loss_mel_ce: 2.517204999923706 (nan) | > loss: 0.010137475095689297 (nan) | > current_lr: 5e-06 | > step_time: 0.1598 (0.13658940445371826) | > loader_time: 0.0048 (0.004697580387364236)  --> TIME: 2025-02-27 12:58:01 -- STEP: 8852/36938 -- GLOBAL_STEP: 1708000 | > loss_text_ce: 0.025844233110547066 (0.029716172872634717) | > loss_mel_ce: 1.8068912029266357 (nan) | > loss: 0.007272759918123484 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13661260708284387) | > loader_time: 0.0046 (0.0046972834950812)  --> TIME: 2025-02-27 12:58:10 -- STEP: 8902/36938 -- GLOBAL_STEP: 1708050 | > loss_text_ce: 0.021978413686156273 (0.029716276195338825) | > loss_mel_ce: 2.557798385620117 (nan) | > loss: 0.010237210430204868 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.13664843588297404) | > loader_time: 0.0046 (0.0046972001812158526)  --> TIME: 2025-02-27 12:58:20 -- STEP: 8952/36938 -- GLOBAL_STEP: 1708100 | > loss_text_ce: 0.03639548271894455 (0.02971665110747837) | > loss_mel_ce: 3.0732758045196533 (nan) | > loss: 0.012339966371655464 (nan) | > current_lr: 5e-06 | > step_time: 0.1427 (0.13665491427770088) | > loader_time: 0.0046 (0.004697074386240428)  --> TIME: 2025-02-27 12:58:29 -- STEP: 9002/36938 -- GLOBAL_STEP: 1708150 | > loss_text_ce: 0.027912354096770287 (0.0297178226901548) | > loss_mel_ce: 2.1510579586029053 (nan) | > loss: 0.008646707981824875 (nan) | > current_lr: 5e-06 | > step_time: 0.1349 (0.1366680171642794) | > loader_time: 0.0046 (0.0046968128225109915)  --> TIME: 2025-02-27 12:58:39 -- STEP: 9052/36938 -- GLOBAL_STEP: 1708200 | > loss_text_ce: 0.02186225913465023 (0.029714782491006963) | > loss_mel_ce: 2.2282066345214844 (nan) | > loss: 0.008928845636546612 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13668519408630098) | > loader_time: 0.0046 (0.0046969912402986935)  --> TIME: 2025-02-27 12:58:48 -- STEP: 9102/36938 -- GLOBAL_STEP: 1708250 | > loss_text_ce: 0.01976885087788105 (0.02972557992246504) | > loss_mel_ce: 1.267256259918213 (nan) | > loss: 0.005107242614030838 (nan) | > current_lr: 5e-06 | > step_time: 0.1676 (0.1366918060811522) | > loader_time: 0.0048 (0.00469709330667534)  --> TIME: 2025-02-27 12:58:57 -- STEP: 9152/36938 -- GLOBAL_STEP: 1708300 | > loss_text_ce: 0.02508609928190708 (0.029723220504413287) | > loss_mel_ce: 2.522195339202881 (nan) | > loss: 0.010108260437846184 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13667879542791697) | > loader_time: 0.0046 (0.004697123555453503)  --> TIME: 2025-02-27 12:59:06 -- STEP: 9202/36938 -- GLOBAL_STEP: 1708350 | > loss_text_ce: 0.026140468195080757 (0.02972042912216943) | > loss_mel_ce: 2.366824150085449 (nan) | > loss: 0.009495891630649567 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.136665519774672) | > loader_time: 0.0067 (0.004697471746955839)  --> TIME: 2025-02-27 12:59:16 -- STEP: 9252/36938 -- GLOBAL_STEP: 1708400 | > loss_text_ce: 0.02617950178682804 (0.029718861108266492) | > loss_mel_ce: 2.1020429134368896 (nan) | > loss: 0.00844532810151577 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13667868869520028) | > loader_time: 0.0047 (0.004697064352509676)  --> TIME: 2025-02-27 12:59:25 -- STEP: 9302/36938 -- GLOBAL_STEP: 1708450 | > loss_text_ce: 0.021399492397904396 (0.02972268408974958) | > loss_mel_ce: 2.7837164402008057 (nan) | > loss: 0.011131413280963898 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.1366804528046715) | > loader_time: 0.0046 (0.004696898372217906)  --> TIME: 2025-02-27 12:59:35 -- STEP: 9352/36938 -- GLOBAL_STEP: 1708500 | > loss_text_ce: 0.04710489138960838 (0.029725948337985746) | > loss_mel_ce: 3.3027870655059814 (nan) | > loss: 0.013293222524225712 (nan) | > current_lr: 5e-06 | > step_time: 0.1396 (0.13671940626294465) | > loader_time: 0.0046 (0.004696602286998791)  --> TIME: 2025-02-27 12:59:44 -- STEP: 9402/36938 -- GLOBAL_STEP: 1708550 | > loss_text_ce: 0.032835304737091064 (0.029722586055728997) | > loss_mel_ce: 3.1827492713928223 (nan) | > loss: 0.012760256417095661 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13673398246716398) | > loader_time: 0.0053 (0.004696743560836456)  --> TIME: 2025-02-27 12:59:54 -- STEP: 9452/36938 -- GLOBAL_STEP: 1708600 | > loss_text_ce: 0.03575104847550392 (0.029727001256008717) | > loss_mel_ce: 2.0242230892181396 (nan) | > loss: 0.008174501359462738 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13672948869869814) | > loader_time: 0.0047 (0.004696988625593018)  --> TIME: 2025-02-27 13:00:03 -- STEP: 9502/36938 -- GLOBAL_STEP: 1708650 | > loss_text_ce: 0.03035816363990307 (0.0297285662137734) | > loss_mel_ce: 2.3907101154327393 (nan) | > loss: 0.009607413783669472 (nan) | > current_lr: 5e-06 | > step_time: 0.1285 (0.13675328132554673) | > loader_time: 0.0046 (0.004696631702566817)  --> TIME: 2025-02-27 13:00:13 -- STEP: 9552/36938 -- GLOBAL_STEP: 1708700 | > loss_text_ce: 0.032321810722351074 (0.02973555311296599) | > loss_mel_ce: 2.4469916820526123 (nan) | > loss: 0.009838545694947243 (nan) | > current_lr: 5e-06 | > step_time: 0.1504 (0.13678855659873798) | > loader_time: 0.0046 (0.004696963295665018)  --> TIME: 2025-02-27 13:00:22 -- STEP: 9602/36938 -- GLOBAL_STEP: 1708750 | > loss_text_ce: 0.040137603878974915 (0.029737278207158085) | > loss_mel_ce: 2.730280637741089 (nan) | > loss: 0.010993723757565022 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13678369865246612) | > loader_time: 0.0053 (0.0046968004202450385)  --> TIME: 2025-02-27 13:00:31 -- STEP: 9652/36938 -- GLOBAL_STEP: 1708800 | > loss_text_ce: 0.028417207300662994 (0.029732098897194987) | > loss_mel_ce: 2.9635024070739746 (nan) | > loss: 0.011872697621583939 (nan) | > current_lr: 5e-06 | > step_time: 0.1527 (0.13677855714741807) | > loader_time: 0.0065 (0.004697610296454109)  --> TIME: 2025-02-27 13:00:41 -- STEP: 9702/36938 -- GLOBAL_STEP: 1708850 | > loss_text_ce: 0.03602614626288414 (0.029735330818153768) | > loss_mel_ce: 1.3456472158432007 (nan) | > loss: 0.005482831038534641 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.13677493657458278) | > loader_time: 0.0047 (0.004697175965656126)  --> TIME: 2025-02-27 13:00:50 -- STEP: 9752/36938 -- GLOBAL_STEP: 1708900 | > loss_text_ce: 0.03819962590932846 (0.02972686459490657) | > loss_mel_ce: 2.606288194656372 (nan) | > loss: 0.010494000278413296 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.1367712853089814) | > loader_time: 0.0046 (0.004696702497527669)  --> TIME: 2025-02-27 13:00:59 -- STEP: 9802/36938 -- GLOBAL_STEP: 1708950 | > loss_text_ce: 0.03240536153316498 (0.029723972979445783) | > loss_mel_ce: 3.109774589538574 (nan) | > loss: 0.012468968518078327 (nan) | > current_lr: 5e-06 | > step_time: 0.2006 (0.13676784931701208) | > loader_time: 0.0049 (0.004696579642063297)  --> TIME: 2025-02-27 13:01:09 -- STEP: 9852/36938 -- GLOBAL_STEP: 1709000 | > loss_text_ce: 0.028556322678923607 (0.029724665036518758) | > loss_mel_ce: 2.8838107585906982 (nan) | > loss: 0.011557012796401978 (nan) | > current_lr: 5e-06 | > step_time: 0.1415 (0.13677551397404106) | > loader_time: 0.0047 (0.004696424492384128)  --> TIME: 2025-02-27 13:01:18 -- STEP: 9902/36938 -- GLOBAL_STEP: 1709050 | > loss_text_ce: 0.03174494579434395 (0.029723672589320634) | > loss_mel_ce: 2.193186044692993 (nan) | > loss: 0.008829091675579548 (nan) | > current_lr: 5e-06 | > step_time: 0.1439 (0.13676743212672696) | > loader_time: 0.0046 (0.004696717649439415)  --> TIME: 2025-02-27 13:01:27 -- STEP: 9952/36938 -- GLOBAL_STEP: 1709100 | > loss_text_ce: 0.020927412435412407 (0.02972293398621898) | > loss_mel_ce: 1.7101051807403564 (nan) | > loss: 0.006869177334010601 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13676352341362857) | > loader_time: 0.0044 (0.004696160770497515)  --> TIME: 2025-02-27 13:01:36 -- STEP: 10002/36938 -- GLOBAL_STEP: 1709150 | > loss_text_ce: 0.031010782346129417 (0.029716769601144832) | > loss_mel_ce: 2.941854476928711 (nan) | > loss: 0.011797085404396057 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.1367432301436819) | > loader_time: 0.0047 (0.004695551129871064)  --> TIME: 2025-02-27 13:01:45 -- STEP: 10052/36938 -- GLOBAL_STEP: 1709200 | > loss_text_ce: 0.03432436287403107 (0.029723504985926512) | > loss_mel_ce: 3.0976452827453613 (nan) | > loss: 0.012428452260792255 (nan) | > current_lr: 5e-06 | > step_time: 0.1418 (0.1367249589634279) | > loader_time: 0.0046 (0.004694981519036484)  --> TIME: 2025-02-27 13:01:55 -- STEP: 10102/36938 -- GLOBAL_STEP: 1709250 | > loss_text_ce: 0.02381008304655552 (0.02972863514756831) | > loss_mel_ce: 2.381492853164673 (nan) | > loss: 0.009544854052364826 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.13675179620280936) | > loader_time: 0.0046 (0.004694603500071656)  --> TIME: 2025-02-27 13:02:04 -- STEP: 10152/36938 -- GLOBAL_STEP: 1709300 | > loss_text_ce: 0.021743616089224815 (0.02972546050427231) | > loss_mel_ce: 2.1756269931793213 (nan) | > loss: 0.008719724602997303 (nan) | > current_lr: 5e-06 | > step_time: 0.1266 (0.1367596459961761) | > loader_time: 0.0047 (0.004694418046679232)  --> TIME: 2025-02-27 13:02:14 -- STEP: 10202/36938 -- GLOBAL_STEP: 1709350 | > loss_text_ce: 0.025662047788500786 (0.02973070846963837) | > loss_mel_ce: 2.762864351272583 (nan) | > loss: 0.011065580882132053 (nan) | > current_lr: 5e-06 | > step_time: 0.1878 (0.1367463411197684) | > loader_time: 0.0048 (0.004693957759548308)  --> TIME: 2025-02-27 13:02:23 -- STEP: 10252/36938 -- GLOBAL_STEP: 1709400 | > loss_text_ce: 0.03545033559203148 (0.02972371716097169) | > loss_mel_ce: 2.8494298458099365 (nan) | > loss: 0.011447937227785587 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.1367312629521471) | > loader_time: 0.0047 (0.004693345683094474)  --> TIME: 2025-02-27 13:02:32 -- STEP: 10302/36938 -- GLOBAL_STEP: 1709450 | > loss_text_ce: 0.02573413774371147 (0.029724479950318967) | > loss_mel_ce: 2.805011749267578 (nan) | > loss: 0.011233119294047356 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1367659374848771) | > loader_time: 0.0045 (0.004692995832906448)  --> TIME: 2025-02-27 13:02:42 -- STEP: 10352/36938 -- GLOBAL_STEP: 1709500 | > loss_text_ce: 0.021208813413977623 (0.029721845922406127) | > loss_mel_ce: 1.5762828588485718 (nan) | > loss: 0.0063392529264092445 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13680185437478795) | > loader_time: 0.0047 (0.004693226822561215)  --> TIME: 2025-02-27 13:02:51 -- STEP: 10402/36938 -- GLOBAL_STEP: 1709550 | > loss_text_ce: 0.02711208537220955 (0.029720224374156723) | > loss_mel_ce: 2.828472137451172 (nan) | > loss: 0.011331683956086636 (nan) | > current_lr: 5e-06 | > step_time: 0.176 (0.13676965236663788) | > loader_time: 0.0049 (0.004693038737262746)  --> TIME: 2025-02-27 13:03:00 -- STEP: 10452/36938 -- GLOBAL_STEP: 1709600 | > loss_text_ce: 0.033002179116010666 (0.029723164895052202) | > loss_mel_ce: 3.221593141555786 (nan) | > loss: 0.01291506178677082 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.13674188709514484) | > loader_time: 0.0047 (0.004693126204001939)  --> TIME: 2025-02-27 13:03:10 -- STEP: 10502/36938 -- GLOBAL_STEP: 1709650 | > loss_text_ce: 0.030155576765537262 (0.02972143201088811) | > loss_mel_ce: 3.093768835067749 (nan) | > loss: 0.012396526522934437 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13677107588583273) | > loader_time: 0.0046 (0.004693569330641507)  --> TIME: 2025-02-27 13:03:19 -- STEP: 10552/36938 -- GLOBAL_STEP: 1709700 | > loss_text_ce: 0.02620026469230652 (0.029719245592314016) | > loss_mel_ce: 0.18193817138671875 (nan) | > loss: 0.000825946219265461 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.13677180812067802) | > loader_time: 0.0047 (0.004693562714234704)  --> TIME: 2025-02-27 13:03:28 -- STEP: 10602/36938 -- GLOBAL_STEP: 1709750 | > loss_text_ce: 0.03725121542811394 (0.02972245518109533) | > loss_mel_ce: 2.6959073543548584 (nan) | > loss: 0.010845867916941643 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13677072610479052) | > loader_time: 0.0048 (0.0046933640446039275)  --> TIME: 2025-02-27 13:03:38 -- STEP: 10652/36938 -- GLOBAL_STEP: 1709800 | > loss_text_ce: 0.030985424295067787 (0.029726340737390708) | > loss_mel_ce: 2.353700876235962 (nan) | > loss: 0.009463041089475155 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.1367636410490798) | > loader_time: 0.0047 (0.004693238147881684)  --> TIME: 2025-02-27 13:03:47 -- STEP: 10702/36938 -- GLOBAL_STEP: 1709850 | > loss_text_ce: 0.02896171435713768 (0.029722557582110475) | > loss_mel_ce: 2.7686426639556885 (nan) | > loss: 0.011101605370640755 (nan) | > current_lr: 5e-06 | > step_time: 0.1403 (0.13674330675719976) | > loader_time: 0.0046 (0.004693128643381621)  --> TIME: 2025-02-27 13:03:56 -- STEP: 10752/36938 -- GLOBAL_STEP: 1709900 | > loss_text_ce: 0.03382298722863197 (0.02972145483575634) | > loss_mel_ce: 2.3706142902374268 (nan) | > loss: 0.00954141840338707 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13675367523960358) | > loader_time: 0.0046 (0.004693215225069274)  --> TIME: 2025-02-27 13:04:06 -- STEP: 10802/36938 -- GLOBAL_STEP: 1709950 | > loss_text_ce: 0.02942316234111786 (0.029716991341149142) | > loss_mel_ce: 2.7300162315368652 (nan) | > loss: 0.010950157418847084 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.1367657157894058) | > loader_time: 0.0053 (0.004693266396786478)  --> TIME: 2025-02-27 13:04:15 -- STEP: 10852/36938 -- GLOBAL_STEP: 1710000 | > loss_text_ce: 0.03161005303263664 (0.02971547696701845) | > loss_mel_ce: 3.055530071258545 (nan) | > loss: 0.012250556610524654 (nan) | > current_lr: 5e-06 | > step_time: 0.1972 (0.13678410630625049) | > loader_time: 0.0049 (0.004693576848423419) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1710000.pth  --> TIME: 2025-02-27 13:04:30 -- STEP: 10902/36938 -- GLOBAL_STEP: 1710050 | > loss_text_ce: 0.03150635585188866 (0.029720663624427626) | > loss_mel_ce: 1.9223130941390991 (nan) | > loss: 0.007753252051770687 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13675929471738255) | > loader_time: 0.0045 (0.004694017920444196)  --> TIME: 2025-02-27 13:04:40 -- STEP: 10952/36938 -- GLOBAL_STEP: 1710100 | > loss_text_ce: 0.032282277941703796 (0.029725422694644055) | > loss_mel_ce: 2.9425878524780273 (nan) | > loss: 0.011805040761828423 (nan) | > current_lr: 5e-06 | > step_time: 0.1857 (0.13676275605699037) | > loader_time: 0.0048 (0.0046938332307939375)  --> TIME: 2025-02-27 13:04:49 -- STEP: 11002/36938 -- GLOBAL_STEP: 1710150 | > loss_text_ce: 0.023775484412908554 (0.029722442706438052) | > loss_mel_ce: 2.7696475982666016 (nan) | > loss: 0.011085012927651405 (nan) | > current_lr: 5e-06 | > step_time: 0.1147 (0.13676503204774887) | > loader_time: 0.0045 (0.004693799637768647)  --> TIME: 2025-02-27 13:04:59 -- STEP: 11052/36938 -- GLOBAL_STEP: 1710200 | > loss_text_ce: 0.025750674307346344 (0.02972356463108181) | > loss_mel_ce: 2.8663275241851807 (nan) | > loss: 0.011476500891149044 (nan) | > current_lr: 5e-06 | > step_time: 0.162 (0.1368038672879682) | > loader_time: 0.0049 (0.004694011001711154)  --> TIME: 2025-02-27 13:05:08 -- STEP: 11102/36938 -- GLOBAL_STEP: 1710250 | > loss_text_ce: 0.029423128813505173 (0.029726562008217713) | > loss_mel_ce: 2.48671293258667 (nan) | > loss: 0.009984668344259262 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13679045416217359) | > loader_time: 0.0045 (0.004694464850524488)  --> TIME: 2025-02-27 13:05:17 -- STEP: 11152/36938 -- GLOBAL_STEP: 1710300 | > loss_text_ce: 0.011339396238327026 (0.029725191455177078) | > loss_mel_ce: 0.0048065185546875 (nan) | > loss: 6.407109322026372e-05 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1367950752441639) | > loader_time: 0.0045 (0.004695050728372367)  --> TIME: 2025-02-27 13:05:27 -- STEP: 11202/36938 -- GLOBAL_STEP: 1710350 | > loss_text_ce: 0.03477145731449127 (0.02972893062281446) | > loss_mel_ce: 2.260456085205078 (nan) | > loss: 0.009108046069741249 (nan) | > current_lr: 5e-06 | > step_time: 0.1373 (0.13679474706075895) | > loader_time: 0.0048 (0.004695465151400783)  --> TIME: 2025-02-27 13:05:36 -- STEP: 11252/36938 -- GLOBAL_STEP: 1710400 | > loss_text_ce: 0.03074042685329914 (0.02973171904964694) | > loss_mel_ce: 1.9987722635269165 (nan) | > loss: 0.008053622208535671 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13681592173818669) | > loader_time: 0.0045 (0.004696061782382593)  --> TIME: 2025-02-27 13:05:45 -- STEP: 11302/36938 -- GLOBAL_STEP: 1710450 | > loss_text_ce: 0.011143690906465054 (0.029728423561243264) | > loss_mel_ce: 1.8977264165878296 (nan) | > loss: 0.0075748818926513195 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.1367893912395184) | > loader_time: 0.0045 (0.004695802046290259)  --> TIME: 2025-02-27 13:05:54 -- STEP: 11352/36938 -- GLOBAL_STEP: 1710500 | > loss_text_ce: 0.03277161344885826 (0.029727032213067196) | > loss_mel_ce: 0.031005859375 (nan) | > loss: 0.00025308525073342025 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.1367952979686303) | > loader_time: 0.0046 (0.004695912223201613)  --> TIME: 2025-02-27 13:06:04 -- STEP: 11402/36938 -- GLOBAL_STEP: 1710550 | > loss_text_ce: 0.028542915359139442 (0.029723267570385296) | > loss_mel_ce: 2.3367230892181396 (nan) | > loss: 0.009385976940393448 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13681190880824032) | > loader_time: 0.0052 (0.0046962835227913865)  --> TIME: 2025-02-27 13:06:14 -- STEP: 11452/36938 -- GLOBAL_STEP: 1710600 | > loss_text_ce: 0.030281879007816315 (0.029725684456495018) | > loss_mel_ce: 2.903850793838501 (nan) | > loss: 0.011643383651971817 (nan) | > current_lr: 5e-06 | > step_time: 0.1689 (0.13683010186509387) | > loader_time: 0.005 (0.004696497998785543)  --> TIME: 2025-02-27 13:06:23 -- STEP: 11502/36938 -- GLOBAL_STEP: 1710650 | > loss_text_ce: 0.02509322017431259 (0.029725639998157832) | > loss_mel_ce: 4.12255859375 (nan) | > loss: 0.016458936035633087 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13682520758523126) | > loader_time: 0.0048 (0.0046964292007204604)  --> TIME: 2025-02-27 13:06:32 -- STEP: 11552/36938 -- GLOBAL_STEP: 1710700 | > loss_text_ce: 0.025872372090816498 (0.0297238068091841) | > loss_mel_ce: 2.6246118545532227 (nan) | > loss: 0.010517795570194721 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.136813892015459) | > loader_time: 0.0046 (0.004696265028124019)  --> TIME: 2025-02-27 13:06:41 -- STEP: 11602/36938 -- GLOBAL_STEP: 1710750 | > loss_text_ce: 0.032799556851387024 (0.02972787354367681) | > loss_mel_ce: 1.67266845703125 (nan) | > loss: 0.0067677306942641735 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13681358432917645) | > loader_time: 0.0045 (0.004696273881800086)  --> TIME: 2025-02-27 13:06:50 -- STEP: 11652/36938 -- GLOBAL_STEP: 1710800 | > loss_text_ce: 0.02950751781463623 (0.029720790906862974) | > loss_mel_ce: 1.942474365234375 (nan) | > loss: 0.00782532524317503 (nan) | > current_lr: 5e-06 | > step_time: 0.1102 (0.13679685788673424) | > loader_time: 0.0045 (0.004696159050958138)  --> TIME: 2025-02-27 13:07:00 -- STEP: 11702/36938 -- GLOBAL_STEP: 1710850 | > loss_text_ce: 0.029000524431467056 (0.029720195605190517) | > loss_mel_ce: 3.3922133445739746 (nan) | > loss: 0.013576245866715908 (nan) | > current_lr: 5e-06 | > step_time: 0.1645 (0.1368045495159902) | > loader_time: 0.0047 (0.004695869311168214)  --> TIME: 2025-02-27 13:07:09 -- STEP: 11752/36938 -- GLOBAL_STEP: 1710900 | > loss_text_ce: 0.03958640247583389 (0.02971630713358296) | > loss_mel_ce: 2.4781811237335205 (nan) | > loss: 0.00999114103615284 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.1368011159301202) | > loader_time: 0.0045 (0.004695681912141045)  --> TIME: 2025-02-27 13:07:19 -- STEP: 11802/36938 -- GLOBAL_STEP: 1710950 | > loss_text_ce: 0.02587403915822506 (0.029714227384909565) | > loss_mel_ce: 3.2035036087036133 (nan) | > loss: 0.012814992107450962 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13681258150852763) | > loader_time: 0.0046 (0.004695522544788403)  --> TIME: 2025-02-27 13:07:28 -- STEP: 11852/36938 -- GLOBAL_STEP: 1711000 | > loss_text_ce: 0.02474895678460598 (0.029718208779993267) | > loss_mel_ce: 2.6757915019989014 (nan) | > loss: 0.010716431774199009 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.1368108892658043) | > loader_time: 0.0046 (0.004695592882502015)  --> TIME: 2025-02-27 13:07:37 -- STEP: 11902/36938 -- GLOBAL_STEP: 1711050 | > loss_text_ce: 0.028821146115660667 (0.02972178256854826) | > loss_mel_ce: 2.707852602005005 (nan) | > loss: 0.010859817266464233 (nan) | > current_lr: 5e-06 | > step_time: 0.1816 (0.13679810861162117) | > loader_time: 0.0048 (0.004695266600397095)  --> TIME: 2025-02-27 13:07:47 -- STEP: 11952/36938 -- GLOBAL_STEP: 1711100 | > loss_text_ce: 0.0376787893474102 (0.02972476639334639) | > loss_mel_ce: 2.6519265174865723 (nan) | > loss: 0.010673037730157375 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.1367983872113289) | > loader_time: 0.0049 (0.004695059245369965)  --> TIME: 2025-02-27 13:07:56 -- STEP: 12002/36938 -- GLOBAL_STEP: 1711150 | > loss_text_ce: 0.023080136626958847 (0.029719773186258982) | > loss_mel_ce: 3.052163600921631 (nan) | > loss: 0.01220334880053997 (nan) | > current_lr: 5e-06 | > step_time: 0.1333 (0.13681057855856765) | > loader_time: 0.0046 (0.004694791897756388)  --> TIME: 2025-02-27 13:08:06 -- STEP: 12052/36938 -- GLOBAL_STEP: 1711200 | > loss_text_ce: 0.039736609905958176 (0.029724280411382795) | > loss_mel_ce: 0.03118896484375 (nan) | > loss: 0.0002814507170114666 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13681240858051963) | > loader_time: 0.0046 (0.004694571496638716)  --> TIME: 2025-02-27 13:08:15 -- STEP: 12102/36938 -- GLOBAL_STEP: 1711250 | > loss_text_ce: 0.02874722145497799 (0.029724972190334153) | > loss_mel_ce: 2.7318286895751953 (nan) | > loss: 0.010954666882753372 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.1368246335586101) | > loader_time: 0.005 (0.004694368106001646)  --> TIME: 2025-02-27 13:08:25 -- STEP: 12152/36938 -- GLOBAL_STEP: 1711300 | > loss_text_ce: 0.022787733003497124 (0.029724763346542584) | > loss_mel_ce: 2.7431228160858154 (nan) | > loss: 0.010975836776196957 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.13683553248977334) | > loader_time: 0.0045 (0.004694171392109256)  --> TIME: 2025-02-27 13:08:34 -- STEP: 12202/36938 -- GLOBAL_STEP: 1711350 | > loss_text_ce: 0.0397593192756176 (0.029731573275005117) | > loss_mel_ce: 2.857407808303833 (nan) | > loss: 0.011496695689857006 (nan) | > current_lr: 5e-06 | > step_time: 0.1142 (0.13683867851176607) | > loader_time: 0.0045 (0.004693949580133901)  --> TIME: 2025-02-27 13:08:44 -- STEP: 12252/36938 -- GLOBAL_STEP: 1711400 | > loss_text_ce: 0.025940939784049988 (0.029734145506626272) | > loss_mel_ce: 2.6008474826812744 (nan) | > loss: 0.010423763655126095 (nan) | > current_lr: 5e-06 | > step_time: 0.1878 (0.13686768163312954) | > loader_time: 0.0048 (0.004693870952306007)  --> TIME: 2025-02-27 13:08:53 -- STEP: 12302/36938 -- GLOBAL_STEP: 1711450 | > loss_text_ce: 0.030465880408883095 (0.029736164154471535) | > loss_mel_ce: 3.0523104667663574 (nan) | > loss: 0.012233239598572254 (nan) | > current_lr: 5e-06 | > step_time: 0.1686 (0.1368940115711428) | > loader_time: 0.0047 (0.004693771024929366)  --> TIME: 2025-02-27 13:09:03 -- STEP: 12352/36938 -- GLOBAL_STEP: 1711500 | > loss_text_ce: 0.03337223082780838 (0.029734698107755322) | > loss_mel_ce: 2.357818841934204 (nan) | > loss: 0.009488853625953197 (nan) | > current_lr: 5e-06 | > step_time: 0.1317 (0.136889317126008) | > loader_time: 0.0045 (0.00469360768872217)  --> TIME: 2025-02-27 13:09:12 -- STEP: 12402/36938 -- GLOBAL_STEP: 1711550 | > loss_text_ce: 0.026897968724370003 (0.02972936532419601) | > loss_mel_ce: 2.356379985809326 (nan) | > loss: 0.009457452222704887 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13690153625084067) | > loader_time: 0.0047 (0.004693292971676957)  --> TIME: 2025-02-27 13:09:22 -- STEP: 12452/36938 -- GLOBAL_STEP: 1711600 | > loss_text_ce: 0.033328667283058167 (0.029729243864872537) | > loss_mel_ce: 2.5583691596984863 (nan) | > loss: 0.010284516029059887 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13691450423302812) | > loader_time: 0.0047 (0.004693639649863831)  --> TIME: 2025-02-27 13:09:31 -- STEP: 12502/36938 -- GLOBAL_STEP: 1711650 | > loss_text_ce: 0.029277751222252846 (0.029729104343135155) | > loss_mel_ce: 0.8689731955528259 (nan) | > loss: 0.0035644881427288055 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13690027262263976) | > loader_time: 0.0047 (0.0046938883554570815)  --> TIME: 2025-02-27 13:09:40 -- STEP: 12552/36938 -- GLOBAL_STEP: 1711700 | > loss_text_ce: 0.026304874569177628 (0.029731133630164754) | > loss_mel_ce: 2.7717411518096924 (nan) | > loss: 0.011103358119726181 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.13689999864540525) | > loader_time: 0.0047 (0.00469368211853797)  --> TIME: 2025-02-27 13:09:50 -- STEP: 12602/36938 -- GLOBAL_STEP: 1711750 | > loss_text_ce: 0.030215468257665634 (0.02972582269225986) | > loss_mel_ce: 2.8493802547454834 (nan) | > loss: 0.011426967568695545 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.13690507453578785) | > loader_time: 0.0049 (0.004693916044127738)  --> TIME: 2025-02-27 13:09:59 -- STEP: 12652/36938 -- GLOBAL_STEP: 1711800 | > loss_text_ce: 0.03335157036781311 (0.029728286144564492) | > loss_mel_ce: 3.2522614002227783 (nan) | > loss: 0.013038148172199726 (nan) | > current_lr: 5e-06 | > step_time: 0.1549 (0.13692779345788234) | > loader_time: 0.0046 (0.004694121682187828)  --> TIME: 2025-02-27 13:10:09 -- STEP: 12702/36938 -- GLOBAL_STEP: 1711850 | > loss_text_ce: 0.03929220512509346 (0.029730994517609487) | > loss_mel_ce: 2.3130128383636475 (nan) | > loss: 0.009334543719887733 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13692332145604985) | > loader_time: 0.0046 (0.004694750282336819)  --> TIME: 2025-02-27 13:10:18 -- STEP: 12752/36938 -- GLOBAL_STEP: 1711900 | > loss_text_ce: 0.02261887490749359 (0.029728358048120962) | > loss_mel_ce: 1.7078208923339844 (nan) | > loss: 0.006866824813187122 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13692733106978083) | > loader_time: 0.0046 (0.004694817225487317)  --> TIME: 2025-02-27 13:10:27 -- STEP: 12802/36938 -- GLOBAL_STEP: 1711950 | > loss_text_ce: 0.028117243200540543 (0.029729578381161584) | > loss_mel_ce: 2.4751007556915283 (nan) | > loss: 0.009933404624462128 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1369361792331817) | > loader_time: 0.0045 (0.004694656401271998)  --> TIME: 2025-02-27 13:10:37 -- STEP: 12852/36938 -- GLOBAL_STEP: 1712000 | > loss_text_ce: 0.03104178048670292 (0.02973416729431211) | > loss_mel_ce: 2.57926869392395 (nan) | > loss: 0.010358375497162342 (nan) | > current_lr: 5e-06 | > step_time: 0.1266 (0.1369469923057398) | > loader_time: 0.0048 (0.004694801474374407)  --> TIME: 2025-02-27 13:10:46 -- STEP: 12902/36938 -- GLOBAL_STEP: 1712050 | > loss_text_ce: 0.03259166702628136 (0.029740170676195058) | > loss_mel_ce: 1.7430756092071533 (nan) | > loss: 0.007046299055218697 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.1369479296466503) | > loader_time: 0.0046 (0.004694595242300551)  --> TIME: 2025-02-27 13:10:56 -- STEP: 12952/36938 -- GLOBAL_STEP: 1712100 | > loss_text_ce: 0.037252455949783325 (0.029740479938825958) | > loss_mel_ce: 2.759561777114868 (nan) | > loss: 0.011098469607532024 (nan) | > current_lr: 5e-06 | > step_time: 0.1622 (0.13693805430095937) | > loader_time: 0.005 (0.004694608569955119)  --> TIME: 2025-02-27 13:11:05 -- STEP: 13002/36938 -- GLOBAL_STEP: 1712150 | > loss_text_ce: 0.027930838987231255 (0.029738689807560666) | > loss_mel_ce: 2.599771022796631 (nan) | > loss: 0.010427388362586498 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.13692679382841133) | > loader_time: 0.0047 (0.004694572633421401)  --> TIME: 2025-02-27 13:11:14 -- STEP: 13052/36938 -- GLOBAL_STEP: 1712200 | > loss_text_ce: 0.03229675814509392 (0.029736855066278066) | > loss_mel_ce: 1.7129524946212769 (nan) | > loss: 0.006925592664629221 (nan) | > current_lr: 5e-06 | > step_time: 0.1548 (0.13693250606588278) | > loader_time: 0.0046 (0.004694554928508635)  --> TIME: 2025-02-27 13:11:23 -- STEP: 13102/36938 -- GLOBAL_STEP: 1712250 | > loss_text_ce: 0.03483094274997711 (0.02973408331616733) | > loss_mel_ce: 2.84914231300354 (nan) | > loss: 0.011444339528679848 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13692927704460078) | > loader_time: 0.0046 (0.004694373821269575)  --> TIME: 2025-02-27 13:11:32 -- STEP: 13152/36938 -- GLOBAL_STEP: 1712300 | > loss_text_ce: 0.026652386412024498 (0.02973067827785184) | > loss_mel_ce: 1.4505716562271118 (nan) | > loss: 0.005862000398337841 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13690778845128035) | > loader_time: 0.0045 (0.004694012485426386)  --> TIME: 2025-02-27 13:11:42 -- STEP: 13202/36938 -- GLOBAL_STEP: 1712350 | > loss_text_ce: 0.03979536518454552 (0.02973158211730038) | > loss_mel_ce: 2.042351484298706 (nan) | > loss: 0.008262488059699535 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13690180365668309) | > loader_time: 0.0046 (0.00469369256953326)  --> TIME: 2025-02-27 13:11:51 -- STEP: 13252/36938 -- GLOBAL_STEP: 1712400 | > loss_text_ce: 0.021881192922592163 (0.029729730511074914) | > loss_mel_ce: 1.7367452383041382 (nan) | > loss: 0.006978676654398441 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1368863848700032) | > loader_time: 0.0045 (0.004693512196199692)  --> TIME: 2025-02-27 13:12:00 -- STEP: 13302/36938 -- GLOBAL_STEP: 1712450 | > loss_text_ce: 0.01975918933749199 (0.0297318062119602) | > loss_mel_ce: 1.4908602237701416 (nan) | > loss: 0.005994522012770176 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13686445314639317) | > loader_time: 0.0047 (0.004693669854133841)  --> TIME: 2025-02-27 13:12:09 -- STEP: 13352/36938 -- GLOBAL_STEP: 1712500 | > loss_text_ce: 0.02613711543381214 (0.029739564960749077) | > loss_mel_ce: 2.80179762840271 (nan) | > loss: 0.011221963912248611 (nan) | > current_lr: 5e-06 | > step_time: 0.1298 (0.13686043136558348) | > loader_time: 0.0046 (0.004693422348217852)  --> TIME: 2025-02-27 13:12:18 -- STEP: 13402/36938 -- GLOBAL_STEP: 1712550 | > loss_text_ce: 0.03267095983028412 (0.029741704978298682) | > loss_mel_ce: 2.902364730834961 (nan) | > loss: 0.011646967381238937 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.13686635118512064) | > loader_time: 0.0048 (0.004693334502188959)  --> TIME: 2025-02-27 13:12:28 -- STEP: 13452/36938 -- GLOBAL_STEP: 1712600 | > loss_text_ce: 0.028944922611117363 (0.029740446026272477) | > loss_mel_ce: 3.1359031200408936 (nan) | > loss: 0.012558922171592712 (nan) | > current_lr: 5e-06 | > step_time: 0.1671 (0.1368714661744244) | > loader_time: 0.0047 (0.0046933529067173935)  --> TIME: 2025-02-27 13:12:37 -- STEP: 13502/36938 -- GLOBAL_STEP: 1712650 | > loss_text_ce: 0.01824641227722168 (0.02973974513201341) | > loss_mel_ce: 2.231448173522949 (nan) | > loss: 0.0089273601770401 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13686231501348556) | > loader_time: 0.0046 (0.004693176283375114)  --> TIME: 2025-02-27 13:12:46 -- STEP: 13552/36938 -- GLOBAL_STEP: 1712700 | > loss_text_ce: 0.025852469727396965 (0.029738162896039786) | > loss_mel_ce: 3.1487607955932617 (nan) | > loss: 0.012597672641277313 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13685025791983904) | > loader_time: 0.0048 (0.004692921250038169)  --> TIME: 2025-02-27 13:12:56 -- STEP: 13602/36938 -- GLOBAL_STEP: 1712750 | > loss_text_ce: 0.03152835741639137 (0.029740470482670218) | > loss_mel_ce: 2.890662908554077 (nan) | > loss: 0.011595997959375381 (nan) | > current_lr: 5e-06 | > step_time: 0.1665 (0.1368637602919248) | > loader_time: 0.0054 (0.004693024124473049)  --> TIME: 2025-02-27 13:13:05 -- STEP: 13652/36938 -- GLOBAL_STEP: 1712800 | > loss_text_ce: 0.027598798274993896 (0.029743168143021498) | > loss_mel_ce: 2.5004074573516846 (nan) | > loss: 0.01003177184611559 (nan) | > current_lr: 5e-06 | > step_time: 0.1359 (0.136882332581213) | > loader_time: 0.0047 (0.004692940882423746)  --> TIME: 2025-02-27 13:13:15 -- STEP: 13702/36938 -- GLOBAL_STEP: 1712850 | > loss_text_ce: 0.02771231159567833 (0.02974179518216582) | > loss_mel_ce: 2.268542766571045 (nan) | > loss: 0.00911212433129549 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13690136298874328) | > loader_time: 0.0045 (0.004693091133183351)  --> TIME: 2025-02-27 13:13:24 -- STEP: 13752/36938 -- GLOBAL_STEP: 1712900 | > loss_text_ce: 0.02480308897793293 (0.029738897173655975) | > loss_mel_ce: 2.2645490169525146 (nan) | > loss: 0.009084731340408325 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.1369003350995454) | > loader_time: 0.0048 (0.004693907367552041)  --> TIME: 2025-02-27 13:13:34 -- STEP: 13802/36938 -- GLOBAL_STEP: 1712950 | > loss_text_ce: 0.031367186456918716 (0.029739868328337752) | > loss_mel_ce: 1.3221029043197632 (nan) | > loss: 0.005370913539081812 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.1369087988182244) | > loader_time: 0.0045 (0.004693845029741631)  --> TIME: 2025-02-27 13:13:43 -- STEP: 13852/36938 -- GLOBAL_STEP: 1713000 | > loss_text_ce: 0.028339695185422897 (0.029739775074641296) | > loss_mel_ce: 2.0755252838134766 (nan) | > loss: 0.008348670788109303 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13689301468683626) | > loader_time: 0.0046 (0.004694035794739746)  --> TIME: 2025-02-27 13:13:52 -- STEP: 13902/36938 -- GLOBAL_STEP: 1713050 | > loss_text_ce: 0.03590301424264908 (0.02974051509398595) | > loss_mel_ce: 1.6160888671875 (nan) | > loss: 0.006555523723363876 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13688907546459209) | > loader_time: 0.0046 (0.004694174372228892)  --> TIME: 2025-02-27 13:14:02 -- STEP: 13952/36938 -- GLOBAL_STEP: 1713100 | > loss_text_ce: 0.030283214524388313 (0.02973964674407781) | > loss_mel_ce: 2.9296581745147705 (nan) | > loss: 0.011745800264179707 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.13689222904520262) | > loader_time: 0.0048 (0.004694223335576686)  --> TIME: 2025-02-27 13:14:11 -- STEP: 14002/36938 -- GLOBAL_STEP: 1713150 | > loss_text_ce: 0.03419683501124382 (0.029739656854467982) | > loss_mel_ce: 0.9334999918937683 (nan) | > loss: 0.003840066958218813 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1368822375973056) | > loader_time: 0.0045 (0.004693879602364386)  --> TIME: 2025-02-27 13:14:20 -- STEP: 14052/36938 -- GLOBAL_STEP: 1713200 | > loss_text_ce: 0.02274719811975956 (0.02973614077805786) | > loss_mel_ce: 2.5595929622650146 (nan) | > loss: 0.010247382335364819 (nan) | > current_lr: 5e-06 | > step_time: 0.1219 (0.13687827070995182) | > loader_time: 0.0045 (0.004693665906231226)  --> TIME: 2025-02-27 13:14:29 -- STEP: 14102/36938 -- GLOBAL_STEP: 1713250 | > loss_text_ce: 0.02656562812626362 (0.02973711346280066) | > loss_mel_ce: 2.681652545928955 (nan) | > loss: 0.010746898129582405 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.1368918566649013) | > loader_time: 0.0045 (0.004693754817252266)  --> TIME: 2025-02-27 13:14:39 -- STEP: 14152/36938 -- GLOBAL_STEP: 1713300 | > loss_text_ce: 0.024038327857851982 (0.029735128812386855) | > loss_mel_ce: 2.735734224319458 (nan) | > loss: 0.010951478965580463 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.1368865219632937) | > loader_time: 0.0046 (0.004693613627187379)  --> TIME: 2025-02-27 13:14:48 -- STEP: 14202/36938 -- GLOBAL_STEP: 1713350 | > loss_text_ce: 0.028709620237350464 (0.029734570400735863) | > loss_mel_ce: 3.1059858798980713 (nan) | > loss: 0.01243926864117384 (nan) | > current_lr: 5e-06 | > step_time: 0.1659 (0.13689032174754887) | > loader_time: 0.0049 (0.004693476100519236)  --> TIME: 2025-02-27 13:14:57 -- STEP: 14252/36938 -- GLOBAL_STEP: 1713400 | > loss_text_ce: 0.0314437560737133 (0.029733085405917157) | > loss_mel_ce: 2.5183496475219727 (nan) | > loss: 0.010118228383362293 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.13687723998838708) | > loader_time: 0.0046 (0.004693157295480766)  --> TIME: 2025-02-27 13:15:06 -- STEP: 14302/36938 -- GLOBAL_STEP: 1713450 | > loss_text_ce: 0.03291000798344612 (0.02972946290402521) | > loss_mel_ce: 0.9125396013259888 (nan) | > loss: 0.0037517843302339315 (nan) | > current_lr: 5e-06 | > step_time: 0.168 (0.1368666769924434) | > loader_time: 0.0048 (0.004692903833278564)  --> TIME: 2025-02-27 13:15:15 -- STEP: 14352/36938 -- GLOBAL_STEP: 1713500 | > loss_text_ce: 0.029181798920035362 (0.0297342682164954) | > loss_mel_ce: 2.963933229446411 (nan) | > loss: 0.011877440847456455 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13684369336269603) | > loader_time: 0.0045 (0.004692628099235244)  --> TIME: 2025-02-27 13:15:24 -- STEP: 14402/36938 -- GLOBAL_STEP: 1713550 | > loss_text_ce: 0.021546900272369385 (0.02973746156488495) | > loss_mel_ce: 2.8969943523406982 (nan) | > loss: 0.011581513099372387 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.1368363889252667) | > loader_time: 0.0045 (0.004692419355668162)  --> TIME: 2025-02-27 13:15:34 -- STEP: 14452/36938 -- GLOBAL_STEP: 1713600 | > loss_text_ce: 0.031196752563118935 (0.02973352996517104) | > loss_mel_ce: 3.0235891342163086 (nan) | > loss: 0.012122167274355888 (nan) | > current_lr: 5e-06 | > step_time: 0.1245 (0.13685005023939367) | > loader_time: 0.0046 (0.00469226907106635)  --> TIME: 2025-02-27 13:15:43 -- STEP: 14502/36938 -- GLOBAL_STEP: 1713650 | > loss_text_ce: 0.016417810693383217 (0.029734168327845693) | > loss_mel_ce: 0.9216484427452087 (nan) | > loss: 0.003722485387697816 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13685375400254357) | > loader_time: 0.0045 (0.004692069712449304)  --> TIME: 2025-02-27 13:15:53 -- STEP: 14552/36938 -- GLOBAL_STEP: 1713700 | > loss_text_ce: 0.0264832004904747 (0.029737268756710778) | > loss_mel_ce: 2.149043083190918 (nan) | > loss: 0.008633041754364967 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13685815984105706) | > loader_time: 0.0046 (0.004691819934283959)  --> TIME: 2025-02-27 13:16:02 -- STEP: 14602/36938 -- GLOBAL_STEP: 1713750 | > loss_text_ce: 0.030286893248558044 (0.029734056653866147) | > loss_mel_ce: 1.5333747863769531 (nan) | > loss: 0.006205006968230009 (nan) | > current_lr: 5e-06 | > step_time: 0.1342 (0.13685454306349712) | > loader_time: 0.0046 (0.004691579426466421)  --> TIME: 2025-02-27 13:16:12 -- STEP: 14652/36938 -- GLOBAL_STEP: 1713800 | > loss_text_ce: 0.029173582792282104 (0.029734618450034937) | > loss_mel_ce: 2.5027318000793457 (nan) | > loss: 0.010047243908047676 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1368666441753655) | > loader_time: 0.0055 (0.004691498269157373)  --> TIME: 2025-02-27 13:16:21 -- STEP: 14702/36938 -- GLOBAL_STEP: 1713850 | > loss_text_ce: 0.0340191014111042 (0.029734394229182524) | > loss_mel_ce: 2.3369438648223877 (nan) | > loss: 0.00940858293324709 (nan) | > current_lr: 5e-06 | > step_time: 0.1628 (0.13687148218721817) | > loader_time: 0.0048 (0.004691293070615001)  --> TIME: 2025-02-27 13:16:30 -- STEP: 14752/36938 -- GLOBAL_STEP: 1713900 | > loss_text_ce: 0.03000730834901333 (0.02973548352275545) | > loss_mel_ce: 2.855137825012207 (nan) | > loss: 0.011448989622294903 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13687348488613185) | > loader_time: 0.0046 (0.0046913404332841295)  --> TIME: 2025-02-27 13:16:39 -- STEP: 14802/36938 -- GLOBAL_STEP: 1713950 | > loss_text_ce: 0.03670410066843033 (0.02973758314742227) | > loss_mel_ce: 3.1348536014556885 (nan) | > loss: 0.01258554682135582 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.13686181435535374) | > loader_time: 0.0046 (0.004691290865072001)  --> TIME: 2025-02-27 13:16:48 -- STEP: 14852/36938 -- GLOBAL_STEP: 1714000 | > loss_text_ce: 0.0341951809823513 (0.02973870683303782) | > loss_mel_ce: 3.1397218704223633 (nan) | > loss: 0.012594909407198429 (nan) | > current_lr: 5e-06 | > step_time: 0.1762 (0.1368457854291736) | > loader_time: 0.0049 (0.004691061997972252)  --> TIME: 2025-02-27 13:16:58 -- STEP: 14902/36938 -- GLOBAL_STEP: 1714050 | > loss_text_ce: 0.03891380876302719 (0.02973616232865164) | > loss_mel_ce: 1.3892189264297485 (nan) | > loss: 0.005667193792760372 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13683568578552896) | > loader_time: 0.0045 (0.004690791453119707)  --> TIME: 2025-02-27 13:17:07 -- STEP: 14952/36938 -- GLOBAL_STEP: 1714100 | > loss_text_ce: 0.03242598846554756 (0.0297325637979533) | > loss_mel_ce: 3.125404119491577 (nan) | > loss: 0.012531071901321411 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.1368247046070086) | > loader_time: 0.0045 (0.0046905538275510755)  --> TIME: 2025-02-27 13:17:16 -- STEP: 15002/36938 -- GLOBAL_STEP: 1714150 | > loss_text_ce: 0.0368884801864624 (0.029733580487197087) | > loss_mel_ce: 2.828878879547119 (nan) | > loss: 0.011372094042599201 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.13681748650452538) | > loader_time: 0.0045 (0.004690283744307496)  --> TIME: 2025-02-27 13:17:25 -- STEP: 15052/36938 -- GLOBAL_STEP: 1714200 | > loss_text_ce: 0.027936147525906563 (0.029732478855019603) | > loss_mel_ce: 2.6263673305511475 (nan) | > loss: 0.010532950982451439 (nan) | > current_lr: 5e-06 | > step_time: 0.158 (0.1367935464479108) | > loader_time: 0.0047 (0.00469011058840317)  --> TIME: 2025-02-27 13:17:34 -- STEP: 15102/36938 -- GLOBAL_STEP: 1714250 | > loss_text_ce: 0.026579350233078003 (0.02973166414036524) | > loss_mel_ce: 1.719940185546875 (nan) | > loss: 0.006930633448064327 (nan) | > current_lr: 5e-06 | > step_time: 0.1261 (0.1368104908548746) | > loader_time: 0.0047 (0.004689947404130161)  --> TIME: 2025-02-27 13:17:44 -- STEP: 15152/36938 -- GLOBAL_STEP: 1714300 | > loss_text_ce: 0.02939615584909916 (0.02972868688254215) | > loss_mel_ce: 2.478828191757202 (nan) | > loss: 0.009953271597623825 (nan) | > current_lr: 5e-06 | > step_time: 0.1789 (0.13681503814087473) | > loader_time: 0.0048 (0.004689763283402237)  --> TIME: 2025-02-27 13:17:53 -- STEP: 15202/36938 -- GLOBAL_STEP: 1714350 | > loss_text_ce: 0.0317586250603199 (0.029725786908970174) | > loss_mel_ce: 2.8733813762664795 (nan) | > loss: 0.011528333649039268 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.13681798639336357) | > loader_time: 0.0047 (0.004689796788647595)  --> TIME: 2025-02-27 13:18:03 -- STEP: 15252/36938 -- GLOBAL_STEP: 1714400 | > loss_text_ce: 0.031186293810606003 (0.02972561024279481) | > loss_mel_ce: 3.049985885620117 (nan) | > loss: 0.012226874940097332 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.13682071013054173) | > loader_time: 0.0047 (0.004690037869794307)  --> TIME: 2025-02-27 13:18:12 -- STEP: 15302/36938 -- GLOBAL_STEP: 1714450 | > loss_text_ce: 0.03040255233645439 (0.02972745758155342) | > loss_mel_ce: 2.624828815460205 (nan) | > loss: 0.010536633431911469 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13683337537997073) | > loader_time: 0.0046 (0.0046900588536539604)  --> TIME: 2025-02-27 13:18:21 -- STEP: 15352/36938 -- GLOBAL_STEP: 1714500 | > loss_text_ce: 0.025728639215230942 (0.02972292827284924) | > loss_mel_ce: 2.370485544204712 (nan) | > loss: 0.009508787654340267 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.1368279115739992) | > loader_time: 0.0046 (0.004690270659325929)  --> TIME: 2025-02-27 13:18:31 -- STEP: 15402/36938 -- GLOBAL_STEP: 1714550 | > loss_text_ce: 0.03504781797528267 (0.029723563608694697) | > loss_mel_ce: 1.861229419708252 (nan) | > loss: 0.007524909917265177 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13683067727531625) | > loader_time: 0.0045 (0.004690359759804404)  --> TIME: 2025-02-27 13:18:40 -- STEP: 15452/36938 -- GLOBAL_STEP: 1714600 | > loss_text_ce: 0.032164230942726135 (0.029721833641436875) | > loss_mel_ce: 0.08478546142578125 (nan) | > loss: 0.00046408610069192946 (nan) | > current_lr: 5e-06 | > step_time: 0.1901 (0.13684273622457493) | > loader_time: 0.0048 (0.004690479204625661)  --> TIME: 2025-02-27 13:18:49 -- STEP: 15502/36938 -- GLOBAL_STEP: 1714650 | > loss_text_ce: 0.022816114127635956 (0.029720997251203744) | > loss_mel_ce: 2.924457311630249 (nan) | > loss: 0.011695530265569687 (nan) | > current_lr: 5e-06 | > step_time: 0.1332 (0.13681948347993525) | > loader_time: 0.0046 (0.004690171333916346)  --> TIME: 2025-02-27 13:18:59 -- STEP: 15552/36938 -- GLOBAL_STEP: 1714700 | > loss_text_ce: 0.029290860518813133 (0.02972222327359749) | > loss_mel_ce: 2.90729022026062 (nan) | > loss: 0.01165310014039278 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.1368337720939159) | > loader_time: 0.005 (0.004690127776851355)  --> TIME: 2025-02-27 13:19:08 -- STEP: 15602/36938 -- GLOBAL_STEP: 1714750 | > loss_text_ce: 0.021414009854197502 (0.029718563860220693) | > loss_mel_ce: 2.5696544647216797 (nan) | > loss: 0.010282018221914768 (nan) | > current_lr: 5e-06 | > step_time: 0.1477 (0.13683421661297224) | > loader_time: 0.0046 (0.004690454733889745)  --> TIME: 2025-02-27 13:19:18 -- STEP: 15652/36938 -- GLOBAL_STEP: 1714800 | > loss_text_ce: 0.03306891396641731 (0.02971886893338172) | > loss_mel_ce: 2.608023166656494 (nan) | > loss: 0.010480524972081184 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.13684487220974711) | > loader_time: 0.0046 (0.00469050718456649)  --> TIME: 2025-02-27 13:19:27 -- STEP: 15702/36938 -- GLOBAL_STEP: 1714850 | > loss_text_ce: 0.030086765065789223 (0.029714757461963684) | > loss_mel_ce: 2.7297680377960205 (nan) | > loss: 0.010951804928481579 (nan) | > current_lr: 5e-06 | > step_time: 0.1531 (0.13684739625889705) | > loader_time: 0.0046 (0.0046903708985984015)  --> TIME: 2025-02-27 13:19:36 -- STEP: 15752/36938 -- GLOBAL_STEP: 1714900 | > loss_text_ce: 0.030301719903945923 (0.029715442208854883) | > loss_mel_ce: 2.763866424560547 (nan) | > loss: 0.01108796987682581 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13683628232240533) | > loader_time: 0.0046 (0.004690377178865353)  --> TIME: 2025-02-27 13:19:46 -- STEP: 15802/36938 -- GLOBAL_STEP: 1714950 | > loss_text_ce: 0.027123965322971344 (0.029713555970401886) | > loss_mel_ce: 2.6880362033843994 (nan) | > loss: 0.010774445720016956 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.1368435149916422) | > loader_time: 0.0047 (0.004690267272937607)  --> TIME: 2025-02-27 13:19:55 -- STEP: 15852/36938 -- GLOBAL_STEP: 1715000 | > loss_text_ce: 0.02243565022945404 (0.02971754214398439) | > loss_mel_ce: 2.969757556915283 (nan) | > loss: 0.01187378354370594 (nan) | > current_lr: 5e-06 | > step_time: 0.1762 (0.136830346995822) | > loader_time: 0.0049 (0.004690101508868008)  --> TIME: 2025-02-27 13:20:04 -- STEP: 15902/36938 -- GLOBAL_STEP: 1715050 | > loss_text_ce: 0.02623816579580307 (0.029720412763074567) | > loss_mel_ce: 2.802219867706299 (nan) | > loss: 0.011224040761590004 (nan) | > current_lr: 5e-06 | > step_time: 0.1175 (0.13683415346813696) | > loader_time: 0.0046 (0.004689951660257938)  --> TIME: 2025-02-27 13:20:13 -- STEP: 15952/36938 -- GLOBAL_STEP: 1715100 | > loss_text_ce: 0.03015938401222229 (0.029716804979009165) | > loss_mel_ce: 1.723471999168396 (nan) | > loss: 0.006958854850381613 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13682810936791942) | > loader_time: 0.0046 (0.004690031514483421)  --> TIME: 2025-02-27 13:20:23 -- STEP: 16002/36938 -- GLOBAL_STEP: 1715150 | > loss_text_ce: 0.046826351433992386 (0.029720915469255184) | > loss_mel_ce: 3.582982301712036 (nan) | > loss: 0.014404003508388996 (nan) | > current_lr: 5e-06 | > step_time: 0.1772 (0.136828642534533) | > loader_time: 0.005 (0.004690200399568751)  --> TIME: 2025-02-27 13:20:32 -- STEP: 16052/36938 -- GLOBAL_STEP: 1715200 | > loss_text_ce: 0.023716740310192108 (0.0297186874240368) | > loss_mel_ce: 3.0939111709594727 (nan) | > loss: 0.012371540069580078 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13680823449924503) | > loader_time: 0.0046 (0.004690100939945755)  --> TIME: 2025-02-27 13:20:41 -- STEP: 16102/36938 -- GLOBAL_STEP: 1715250 | > loss_text_ce: 0.028574949130415916 (0.0297171885317795) | > loss_mel_ce: 2.9442431926727295 (nan) | > loss: 0.01179689820855856 (nan) | > current_lr: 5e-06 | > step_time: 0.1408 (0.1368114216078023) | > loader_time: 0.0047 (0.004689991999791854)  --> TIME: 2025-02-27 13:20:50 -- STEP: 16152/36938 -- GLOBAL_STEP: 1715300 | > loss_text_ce: 0.0261490847915411 (0.02971813047779059) | > loss_mel_ce: 2.7208504676818848 (nan) | > loss: 0.01090079266577959 (nan) | > current_lr: 5e-06 | > step_time: 0.1977 (0.13680970177253654) | > loader_time: 0.005 (0.004690246395453041)  --> TIME: 2025-02-27 13:21:00 -- STEP: 16202/36938 -- GLOBAL_STEP: 1715350 | > loss_text_ce: 0.033152174204587936 (0.029716402632343545) | > loss_mel_ce: 2.2210021018981934 (nan) | > loss: 0.008945057168602943 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13681097045943622) | > loader_time: 0.0046 (0.0046901283934004)  --> TIME: 2025-02-27 13:21:09 -- STEP: 16252/36938 -- GLOBAL_STEP: 1715400 | > loss_text_ce: 0.037857536226511 (0.02971618352029811) | > loss_mel_ce: 3.076472043991089 (nan) | > loss: 0.012358451262116432 (nan) | > current_lr: 5e-06 | > step_time: 0.1352 (0.13681009155818438) | > loader_time: 0.0046 (0.004689935918456347)  --> TIME: 2025-02-27 13:21:19 -- STEP: 16302/36938 -- GLOBAL_STEP: 1715450 | > loss_text_ce: 0.015717078000307083 (0.029720045220230917) | > loss_mel_ce: 2.2205755710601807 (nan) | > loss: 0.008874177932739258 (nan) | > current_lr: 5e-06 | > step_time: 0.1203 (0.1368428566399971) | > loader_time: 0.0045 (0.004690395631814708)  --> TIME: 2025-02-27 13:21:28 -- STEP: 16352/36938 -- GLOBAL_STEP: 1715500 | > loss_text_ce: 0.037669144570827484 (0.029716326157570857) | > loss_mel_ce: 3.1087303161621094 (nan) | > loss: 0.012485712766647339 (nan) | > current_lr: 5e-06 | > step_time: 0.1575 (0.13684153227306836) | > loader_time: 0.0046 (0.004690478459263011)  --> TIME: 2025-02-27 13:21:38 -- STEP: 16402/36938 -- GLOBAL_STEP: 1715550 | > loss_text_ce: 0.03403884917497635 (0.029717140993816095) | > loss_mel_ce: 3.135093927383423 (nan) | > loss: 0.012575924396514893 (nan) | > current_lr: 5e-06 | > step_time: 0.1631 (0.1368382292075237) | > loader_time: 0.0047 (0.004690581219266941)  --> TIME: 2025-02-27 13:21:47 -- STEP: 16452/36938 -- GLOBAL_STEP: 1715600 | > loss_text_ce: 0.027054645121097565 (0.029717134743463127) | > loss_mel_ce: 2.9833669662475586 (nan) | > loss: 0.011946117505431175 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13683288541699198) | > loader_time: 0.0049 (0.00469069548427636)  --> TIME: 2025-02-27 13:21:56 -- STEP: 16502/36938 -- GLOBAL_STEP: 1715650 | > loss_text_ce: 0.030711740255355835 (0.029716717964072107) | > loss_mel_ce: 2.0302233695983887 (nan) | > loss: 0.008178314194083214 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13682862305378202) | > loader_time: 0.0045 (0.004690557577410323)  --> TIME: 2025-02-27 13:22:06 -- STEP: 16552/36938 -- GLOBAL_STEP: 1715700 | > loss_text_ce: 0.03686477988958359 (0.029718809200383476) | > loss_mel_ce: 3.1156318187713623 (nan) | > loss: 0.012509907595813274 (nan) | > current_lr: 5e-06 | > step_time: 0.196 (0.13683688924357512) | > loader_time: 0.005 (0.004690639678264575)  --> TIME: 2025-02-27 13:22:15 -- STEP: 16602/36938 -- GLOBAL_STEP: 1715750 | > loss_text_ce: 0.033706460148096085 (0.02971871319304782) | > loss_mel_ce: 3.017637014389038 (nan) | > loss: 0.012108506634831429 (nan) | > current_lr: 5e-06 | > step_time: 0.1786 (0.13684498647683596) | > loader_time: 0.0049 (0.004690666311319391)  --> TIME: 2025-02-27 13:22:24 -- STEP: 16652/36938 -- GLOBAL_STEP: 1715800 | > loss_text_ce: 0.037864528596401215 (0.029718335174430583) | > loss_mel_ce: 2.8787636756896973 (nan) | > loss: 0.011573921889066696 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13683837490416229) | > loader_time: 0.0054 (0.004690787224078883)  --> TIME: 2025-02-27 13:22:33 -- STEP: 16702/36938 -- GLOBAL_STEP: 1715850 | > loss_text_ce: 0.02562575787305832 (0.029717202306400656) | > loss_mel_ce: 2.17695689201355 (nan) | > loss: 0.008740407414734364 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13681549020133843) | > loader_time: 0.0046 (0.004690882631753401)  --> TIME: 2025-02-27 13:22:43 -- STEP: 16752/36938 -- GLOBAL_STEP: 1715900 | > loss_text_ce: 0.03465528041124344 (0.029721253150399134) | > loss_mel_ce: 3.0067741870880127 (nan) | > loss: 0.012069165706634521 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13681779948437658) | > loader_time: 0.0046 (0.004690826451539288)  --> TIME: 2025-02-27 13:22:52 -- STEP: 16802/36938 -- GLOBAL_STEP: 1715950 | > loss_text_ce: 0.03742774948477745 (0.029721146006317378) | > loss_mel_ce: 2.8902127742767334 (nan) | > loss: 0.011617621406912804 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13682923958622492) | > loader_time: 0.0053 (0.004690851488080459)  --> TIME: 2025-02-27 13:23:02 -- STEP: 16852/36938 -- GLOBAL_STEP: 1716000 | > loss_text_ce: 0.034393660724163055 (0.029718338941153607) | > loss_mel_ce: 2.927446126937866 (nan) | > loss: 0.011753332801163197 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13683154859991026) | > loader_time: 0.0046 (0.004691185519442295)  --> TIME: 2025-02-27 13:23:11 -- STEP: 16902/36938 -- GLOBAL_STEP: 1716050 | > loss_text_ce: 0.034455738961696625 (0.02971898882583495) | > loss_mel_ce: 0.9203247427940369 (nan) | > loss: 0.00378881162032485 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1368155583956157) | > loader_time: 0.0048 (0.004691605976440095)  --> TIME: 2025-02-27 13:23:20 -- STEP: 16952/36938 -- GLOBAL_STEP: 1716100 | > loss_text_ce: 0.02286645583808422 (0.029717431513436778) | > loss_mel_ce: 2.4874300956726074 (nan) | > loss: 0.00996149517595768 (nan) | > current_lr: 5e-06 | > step_time: 0.1626 (0.13681880662102563) | > loader_time: 0.0047 (0.004691779177500516)  --> TIME: 2025-02-27 13:23:29 -- STEP: 17002/36938 -- GLOBAL_STEP: 1716150 | > loss_text_ce: 0.028266234323382378 (0.029715294818765364) | > loss_mel_ce: 2.846120834350586 (nan) | > loss: 0.011406298726797104 (nan) | > current_lr: 5e-06 | > step_time: 0.187 (0.13681022595860015) | > loader_time: 0.0059 (0.004692001618184598)  --> TIME: 2025-02-27 13:23:38 -- STEP: 17052/36938 -- GLOBAL_STEP: 1716200 | > loss_text_ce: 0.03062892146408558 (0.029718065816543567) | > loss_mel_ce: 1.21868896484375 (nan) | > loss: 0.0049576107412576675 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13680365199973354) | > loader_time: 0.0046 (0.004692287168786894)  --> TIME: 2025-02-27 13:23:48 -- STEP: 17102/36938 -- GLOBAL_STEP: 1716250 | > loss_text_ce: 0.03548278659582138 (0.029719506355622007) | > loss_mel_ce: 2.6029672622680664 (nan) | > loss: 0.010470041073858738 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1368042537446969) | > loader_time: 0.0045 (0.004692516094368784)  --> TIME: 2025-02-27 13:23:57 -- STEP: 17152/36938 -- GLOBAL_STEP: 1716300 | > loss_text_ce: 0.029980074614286423 (0.029720841318508117) | > loss_mel_ce: 2.9000816345214844 (nan) | > loss: 0.011627229861915112 (nan) | > current_lr: 5e-06 | > step_time: 0.1879 (0.13680426844520793) | > loader_time: 0.0048 (0.004692884023064972)  --> TIME: 2025-02-27 13:24:06 -- STEP: 17202/36938 -- GLOBAL_STEP: 1716350 | > loss_text_ce: 0.03156556189060211 (0.02971858973804362) | > loss_mel_ce: 2.31494402885437 (nan) | > loss: 0.009311546571552753 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13679670130232155) | > loader_time: 0.0046 (0.004693108205947167)  --> TIME: 2025-02-27 13:24:16 -- STEP: 17252/36938 -- GLOBAL_STEP: 1716400 | > loss_text_ce: 0.031997427344322205 (0.029715196321100385) | > loss_mel_ce: 2.7919392585754395 (nan) | > loss: 0.011206098832190037 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.13680379276841684) | > loader_time: 0.0049 (0.004693054362402508)  --> TIME: 2025-02-27 13:24:25 -- STEP: 17302/36938 -- GLOBAL_STEP: 1716450 | > loss_text_ce: 0.03438469395041466 (0.02971611152090426) | > loss_mel_ce: 2.119054079055786 (nan) | > loss: 0.00854539219290018 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13680490138640683) | > loader_time: 0.0047 (0.004692968185644375)  --> TIME: 2025-02-27 13:24:35 -- STEP: 17352/36938 -- GLOBAL_STEP: 1716500 | > loss_text_ce: 0.0380905419588089 (0.029711178447816122) | > loss_mel_ce: 2.6972553730010986 (nan) | > loss: 0.010854547843337059 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13680276280177123) | > loader_time: 0.0048 (0.004693160770454928)  --> TIME: 2025-02-27 13:24:44 -- STEP: 17402/36938 -- GLOBAL_STEP: 1716550 | > loss_text_ce: 0.02641880512237549 (0.02971100745363116) | > loss_mel_ce: 2.134075880050659 (nan) | > loss: 0.008573392406105995 (nan) | > current_lr: 5e-06 | > step_time: 0.1151 (0.13678967006013865) | > loader_time: 0.0045 (0.004692894290910458)  --> TIME: 2025-02-27 13:24:53 -- STEP: 17452/36938 -- GLOBAL_STEP: 1716600 | > loss_text_ce: 0.02907801792025566 (0.02971405179664913) | > loss_mel_ce: 2.768864393234253 (nan) | > loss: 0.011102946475148201 (nan) | > current_lr: 5e-06 | > step_time: 0.1447 (0.136798234043825) | > loader_time: 0.0047 (0.00469303631110389)  --> TIME: 2025-02-27 13:25:02 -- STEP: 17502/36938 -- GLOBAL_STEP: 1716650 | > loss_text_ce: 0.02734960988163948 (0.029714300598568825) | > loss_mel_ce: 1.3990445137023926 (nan) | > loss: 0.005660294555127621 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.1367787206776411) | > loader_time: 0.0045 (0.004692829509364992)  --> TIME: 2025-02-27 13:25:12 -- STEP: 17552/36938 -- GLOBAL_STEP: 1716700 | > loss_text_ce: 0.03309382125735283 (0.029713496178448654) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13679149267395277) | > loader_time: 0.0045 (0.00469289509513317)  --> TIME: 2025-02-27 13:25:21 -- STEP: 17602/36938 -- GLOBAL_STEP: 1716750 | > loss_text_ce: 0.017573123797774315 (0.02971174822411347) | > loss_mel_ce: 1.2214202880859375 (nan) | > loss: 0.004916640929877758 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13679677576466628) | > loader_time: 0.0046 (0.004693172178733272)  --> TIME: 2025-02-27 13:25:31 -- STEP: 17652/36938 -- GLOBAL_STEP: 1716800 | > loss_text_ce: 0.02765287645161152 (0.029710587599040814) | > loss_mel_ce: 2.4012951850891113 (nan) | > loss: 0.009638683870434761 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13679432805476455) | > loader_time: 0.0047 (0.004693267352451878)  --> TIME: 2025-02-27 13:25:40 -- STEP: 17702/36938 -- GLOBAL_STEP: 1716850 | > loss_text_ce: 0.019748641178011894 (0.029709100558672614) | > loss_mel_ce: 2.181645154953003 (nan) | > loss: 0.008735690265893936 (nan) | > current_lr: 5e-06 | > step_time: 0.1107 (0.13679248615114442) | > loader_time: 0.0046 (0.004693618481771464)  --> TIME: 2025-02-27 13:25:49 -- STEP: 17752/36938 -- GLOBAL_STEP: 1716900 | > loss_text_ce: 0.03097967617213726 (0.029710856979842316) | > loss_mel_ce: 2.7863049507141113 (nan) | > loss: 0.011179701425135136 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.13678684376713998) | > loader_time: 0.0047 (0.004693643138343097)  --> TIME: 2025-02-27 13:25:58 -- STEP: 17802/36938 -- GLOBAL_STEP: 1716950 | > loss_text_ce: 0.027438146993517876 (0.0297079685924623) | > loss_mel_ce: 3.6420600414276123 (nan) | > loss: 0.014561501331627369 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.1367764721531742) | > loader_time: 0.0047 (0.004693467407303683)  --> TIME: 2025-02-27 13:26:08 -- STEP: 17852/36938 -- GLOBAL_STEP: 1717000 | > loss_text_ce: 0.02883840538561344 (0.029708524236392742) | > loss_mel_ce: 3.104702949523926 (nan) | > loss: 0.01243468839675188 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.13677566454278564) | > loader_time: 0.0046 (0.004693357687532787)  --> TIME: 2025-02-27 13:26:17 -- STEP: 17902/36938 -- GLOBAL_STEP: 1717050 | > loss_text_ce: 0.032187867909669876 (0.02971248075188828) | > loss_mel_ce: 2.7601568698883057 (nan) | > loss: 0.011080734431743622 (nan) | > current_lr: 5e-06 | > step_time: 0.169 (0.1367734392825854) | > loader_time: 0.0048 (0.004693138600727529)  --> TIME: 2025-02-27 13:26:26 -- STEP: 17952/36938 -- GLOBAL_STEP: 1717100 | > loss_text_ce: 0.0346437431871891 (0.029716682811389065) | > loss_mel_ce: 2.2086293697357178 (nan) | > loss: 0.008901877328753471 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13676073527006724) | > loader_time: 0.0044 (0.004692925276390774)  --> TIME: 2025-02-27 13:26:35 -- STEP: 18002/36938 -- GLOBAL_STEP: 1717150 | > loss_text_ce: 0.02204047702252865 (0.029714981311830386) | > loss_mel_ce: 2.445746421813965 (nan) | > loss: 0.009792805649340153 (nan) | > current_lr: 5e-06 | > step_time: 0.1695 (0.13675262077479006) | > loader_time: 0.0049 (0.004692735691598177)  --> TIME: 2025-02-27 13:26:45 -- STEP: 18052/36938 -- GLOBAL_STEP: 1717200 | > loss_text_ce: 0.030737796798348427 (0.029720028478854677) | > loss_mel_ce: 2.625290632247925 (nan) | > loss: 0.010539796203374863 (nan) | > current_lr: 5e-06 | > step_time: 0.154 (0.13675734455559574) | > loader_time: 0.0046 (0.004692595059977841)  --> TIME: 2025-02-27 13:26:54 -- STEP: 18102/36938 -- GLOBAL_STEP: 1717250 | > loss_text_ce: 0.029081013053655624 (0.029720375032750972) | > loss_mel_ce: 2.57770037651062 (nan) | > loss: 0.010344371199607849 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.1367510397280661) | > loader_time: 0.0046 (0.004692413493185813)  --> TIME: 2025-02-27 13:27:03 -- STEP: 18152/36938 -- GLOBAL_STEP: 1717300 | > loss_text_ce: 0.022983839735388756 (0.029719640738830506) | > loss_mel_ce: 2.862989902496338 (nan) | > loss: 0.011452277190983295 (nan) | > current_lr: 5e-06 | > step_time: 0.1252 (0.13674012831309218) | > loader_time: 0.0045 (0.00469214740751491)  --> TIME: 2025-02-27 13:27:12 -- STEP: 18202/36938 -- GLOBAL_STEP: 1717350 | > loss_text_ce: 0.025351671501994133 (0.02971909148692769) | > loss_mel_ce: 1.5431867837905884 (nan) | > loss: 0.006224359385669231 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13672294615127392) | > loader_time: 0.0044 (0.0046921782854585335)  --> TIME: 2025-02-27 13:27:22 -- STEP: 18252/36938 -- GLOBAL_STEP: 1717400 | > loss_text_ce: 0.03125384822487831 (0.029719425470722535) | > loss_mel_ce: 2.3707427978515625 (nan) | > loss: 0.009531733579933643 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1367337665431414) | > loader_time: 0.0047 (0.004692154288056787)  --> TIME: 2025-02-27 13:27:31 -- STEP: 18302/36938 -- GLOBAL_STEP: 1717450 | > loss_text_ce: 0.03305787965655327 (0.02971938278678421) | > loss_mel_ce: 2.578946352005005 (nan) | > loss: 0.010365096852183342 (nan) | > current_lr: 5e-06 | > step_time: 0.1426 (0.1367347266501824) | > loader_time: 0.0046 (0.004691945035761641)  --> TIME: 2025-02-27 13:27:40 -- STEP: 18352/36938 -- GLOBAL_STEP: 1717500 | > loss_text_ce: 0.030545730143785477 (0.02971938971618134) | > loss_mel_ce: 2.2703216075897217 (nan) | > loss: 0.009130426682531834 (nan) | > current_lr: 5e-06 | > step_time: 0.1256 (0.13672848195227297) | > loader_time: 0.0046 (0.004691998350069416)  --> TIME: 2025-02-27 13:27:50 -- STEP: 18402/36938 -- GLOBAL_STEP: 1717550 | > loss_text_ce: 0.021847618743777275 (0.029717166030687567) | > loss_mel_ce: 2.3719942569732666 (nan) | > loss: 0.009499373845756054 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.1367227404082084) | > loader_time: 0.0045 (0.0046918374042823835)  --> TIME: 2025-02-27 13:27:59 -- STEP: 18452/36938 -- GLOBAL_STEP: 1717600 | > loss_text_ce: 0.026084670796990395 (0.029716115192509777) | > loss_mel_ce: 2.690295457839966 (nan) | > loss: 0.010779286734759808 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.13672469414523492) | > loader_time: 0.0046 (0.004691810662695247)  --> TIME: 2025-02-27 13:28:08 -- STEP: 18502/36938 -- GLOBAL_STEP: 1717650 | > loss_text_ce: 0.027271224185824394 (0.029715451466240956) | > loss_mel_ce: 2.2545926570892334 (nan) | > loss: 0.009055016562342644 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1367182784590279) | > loader_time: 0.0046 (0.00469158389300039)  --> TIME: 2025-02-27 13:28:18 -- STEP: 18552/36938 -- GLOBAL_STEP: 1717700 | > loss_text_ce: 0.03666669875383377 (0.029713229554728007) | > loss_mel_ce: 2.8905062675476074 (nan) | > loss: 0.01161576621234417 (nan) | > current_lr: 5e-06 | > step_time: 0.1612 (0.13672119783861672) | > loader_time: 0.0046 (0.004691678820215648)  --> TIME: 2025-02-27 13:28:27 -- STEP: 18602/36938 -- GLOBAL_STEP: 1717750 | > loss_text_ce: 0.03564408794045448 (0.029714922884063813) | > loss_mel_ce: 2.6508560180664062 (nan) | > loss: 0.010660715401172638 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.1367109974870475) | > loader_time: 0.0054 (0.0046918057662263205)  --> TIME: 2025-02-27 13:28:36 -- STEP: 18652/36938 -- GLOBAL_STEP: 1717800 | > loss_text_ce: 0.03251055255532265 (0.029717145876308054) | > loss_mel_ce: 3.0853073596954346 (nan) | > loss: 0.012372293509542942 (nan) | > current_lr: 5e-06 | > step_time: 0.1154 (0.13670776778987462) | > loader_time: 0.0045 (0.004692049988242081)  --> TIME: 2025-02-27 13:28:45 -- STEP: 18702/36938 -- GLOBAL_STEP: 1717850 | > loss_text_ce: 0.04061055928468704 (0.029719135374699866) | > loss_mel_ce: 2.6243414878845215 (nan) | > loss: 0.010575206950306892 (nan) | > current_lr: 5e-06 | > step_time: 0.1206 (0.1367008556415532) | > loader_time: 0.0047 (0.004692141505667583)  --> TIME: 2025-02-27 13:28:55 -- STEP: 18752/36938 -- GLOBAL_STEP: 1717900 | > loss_text_ce: 0.03443126007914543 (0.02971716056332614) | > loss_mel_ce: 2.7252187728881836 (nan) | > loss: 0.010950992815196514 (nan) | > current_lr: 5e-06 | > step_time: 0.1286 (0.13670374430182025) | > loader_time: 0.0046 (0.004692320874018053)  --> TIME: 2025-02-27 13:29:04 -- STEP: 18802/36938 -- GLOBAL_STEP: 1717950 | > loss_text_ce: 0.03513931855559349 (0.029720267202649066) | > loss_mel_ce: 2.600665330886841 (nan) | > loss: 0.010459543205797672 (nan) | > current_lr: 5e-06 | > step_time: 0.1694 (0.13670858331543737) | > loader_time: 0.0049 (0.004692423598536288)  --> TIME: 2025-02-27 13:29:13 -- STEP: 18852/36938 -- GLOBAL_STEP: 1718000 | > loss_text_ce: 0.028018930926918983 (0.029720504814902154) | > loss_mel_ce: 2.8189923763275146 (nan) | > loss: 0.011297664605081081 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13670119954956408) | > loader_time: 0.0047 (0.004692679981302093)  --> TIME: 2025-02-27 13:29:23 -- STEP: 18902/36938 -- GLOBAL_STEP: 1718050 | > loss_text_ce: 0.030488885939121246 (0.029721342596145516) | > loss_mel_ce: 2.637678623199463 (nan) | > loss: 0.010587967000901699 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13670595487379247) | > loader_time: 0.0046 (0.004692729964179079)  --> TIME: 2025-02-27 13:29:32 -- STEP: 18952/36938 -- GLOBAL_STEP: 1718100 | > loss_text_ce: 0.033909618854522705 (0.029725634571731534) | > loss_mel_ce: 2.1494221687316895 (nan) | > loss: 0.008664015680551529 (nan) | > current_lr: 5e-06 | > step_time: 0.108 (0.13671079748488993) | > loader_time: 0.0045 (0.004692793961766889)  --> TIME: 2025-02-27 13:29:42 -- STEP: 19002/36938 -- GLOBAL_STEP: 1718150 | > loss_text_ce: 0.0286642424762249 (0.02972639594816687) | > loss_mel_ce: 2.9589200019836426 (nan) | > loss: 0.011855494230985641 (nan) | > current_lr: 5e-06 | > step_time: 0.183 (0.1366999918455172) | > loader_time: 0.0056 (0.004692751048125898)  --> TIME: 2025-02-27 13:29:51 -- STEP: 19052/36938 -- GLOBAL_STEP: 1718200 | > loss_text_ce: 0.030689619481563568 (0.029726206721901637) | > loss_mel_ce: 2.3836586475372314 (nan) | > loss: 0.009580748155713081 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13670070367076653) | > loader_time: 0.0046 (0.004692809185810538)  --> TIME: 2025-02-27 13:30:00 -- STEP: 19102/36938 -- GLOBAL_STEP: 1718250 | > loss_text_ce: 0.029438931494951248 (0.029724438711485012) | > loss_mel_ce: 2.631096124649048 (nan) | > loss: 0.010557679459452629 (nan) | > current_lr: 5e-06 | > step_time: 0.1872 (0.1367066396791736) | > loader_time: 0.0048 (0.0046930604424854335)  --> TIME: 2025-02-27 13:30:10 -- STEP: 19152/36938 -- GLOBAL_STEP: 1718300 | > loss_text_ce: 0.01731100119650364 (0.02972309031555761) | > loss_mel_ce: 1.5915501117706299 (nan) | > loss: 0.006384369917213917 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13672781377756865) | > loader_time: 0.0047 (0.004693305333057023)  --> TIME: 2025-02-27 13:30:19 -- STEP: 19202/36938 -- GLOBAL_STEP: 1718350 | > loss_text_ce: 0.028356460854411125 (0.029726381564307975) | > loss_mel_ce: 2.6594367027282715 (nan) | > loss: 0.01066584698855877 (nan) | > current_lr: 5e-06 | > step_time: 0.1504 (0.13672245804089642) | > loader_time: 0.0056 (0.004693440429469744)  --> TIME: 2025-02-27 13:30:29 -- STEP: 19252/36938 -- GLOBAL_STEP: 1718400 | > loss_text_ce: 0.03197929635643959 (0.02972422368806002) | > loss_mel_ce: 3.0171451568603516 (nan) | > loss: 0.01209970097988844 (nan) | > current_lr: 5e-06 | > step_time: 0.1808 (0.13673932934263064) | > loader_time: 0.0049 (0.004693608521275807)  --> TIME: 2025-02-27 13:30:38 -- STEP: 19302/36938 -- GLOBAL_STEP: 1718450 | > loss_text_ce: 0.03385155275464058 (0.02972461615430285) | > loss_mel_ce: 3.0454297065734863 (nan) | > loss: 0.012219371274113655 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13672374709382953) | > loader_time: 0.0045 (0.004693632520624851)  --> TIME: 2025-02-27 13:30:47 -- STEP: 19352/36938 -- GLOBAL_STEP: 1718500 | > loss_text_ce: 0.03718406707048416 (0.02972475884337599) | > loss_mel_ce: 2.547999620437622 (nan) | > loss: 0.010258665308356285 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13671713370141406) | > loader_time: 0.0046 (0.004693558685244372)  --> TIME: 2025-02-27 13:30:57 -- STEP: 19402/36938 -- GLOBAL_STEP: 1718550 | > loss_text_ce: 0.029287287965416908 (0.029724770326267375) | > loss_mel_ce: 2.5773563385009766 (nan) | > loss: 0.01034382451325655 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13671867364018886) | > loader_time: 0.0048 (0.00469352479890846)  --> TIME: 2025-02-27 13:31:06 -- STEP: 19452/36938 -- GLOBAL_STEP: 1718600 | > loss_text_ce: 0.028418852016329765 (0.029722527477084173) | > loss_mel_ce: 3.06113862991333 (nan) | > loss: 0.012260149233043194 (nan) | > current_lr: 5e-06 | > step_time: 0.1861 (0.13672353435440238) | > loader_time: 0.0049 (0.00469370752897816)  --> TIME: 2025-02-27 13:31:16 -- STEP: 19502/36938 -- GLOBAL_STEP: 1718650 | > loss_text_ce: 0.030167320743203163 (0.029722128644215595) | > loss_mel_ce: 2.726428985595703 (nan) | > loss: 0.010938875377178192 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13671835331216994) | > loader_time: 0.0046 (0.004693855580128501)  --> TIME: 2025-02-27 13:31:25 -- STEP: 19552/36938 -- GLOBAL_STEP: 1718700 | > loss_text_ce: 0.02215927466750145 (0.029719978080674316) | > loss_mel_ce: 2.182199478149414 (nan) | > loss: 0.008747455663979053 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13671133273361905) | > loader_time: 0.0045 (0.004693682864918044)  --> TIME: 2025-02-27 13:31:34 -- STEP: 19602/36938 -- GLOBAL_STEP: 1718750 | > loss_text_ce: 0.03245104104280472 (0.029723429707631633) | > loss_mel_ce: 2.554185390472412 (nan) | > loss: 0.01026443112641573 (nan) | > current_lr: 5e-06 | > step_time: 0.1511 (0.13670162786793746) | > loader_time: 0.0049 (0.004693473106670721)  --> TIME: 2025-02-27 13:31:43 -- STEP: 19652/36938 -- GLOBAL_STEP: 1718800 | > loss_text_ce: 0.028018910437822342 (0.02972339913362279) | > loss_mel_ce: 2.3254880905151367 (nan) | > loss: 0.009339313954114914 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.13670057087451182) | > loader_time: 0.0047 (0.0046933501649480186)  --> TIME: 2025-02-27 13:31:52 -- STEP: 19702/36938 -- GLOBAL_STEP: 1718850 | > loss_text_ce: 0.026712164282798767 (0.029721706695399915) | > loss_mel_ce: 3.0295450687408447 (nan) | > loss: 0.012128005735576153 (nan) | > current_lr: 5e-06 | > step_time: 0.1706 (0.13670162952027315) | > loader_time: 0.0048 (0.004693143562113184)  --> TIME: 2025-02-27 13:32:02 -- STEP: 19752/36938 -- GLOBAL_STEP: 1718900 | > loss_text_ce: 0.037535738199949265 (0.029722428715674848) | > loss_mel_ce: 0.0757904052734375 (nan) | > loss: 0.0004497069458011538 (nan) | > current_lr: 5e-06 | > step_time: 0.1075 (0.13670284596470578) | > loader_time: 0.0045 (0.004693159126665658)  --> TIME: 2025-02-27 13:32:11 -- STEP: 19802/36938 -- GLOBAL_STEP: 1718950 | > loss_text_ce: 0.03067215345799923 (0.029726106731398666) | > loss_mel_ce: 1.5734304189682007 (nan) | > loss: 0.006365486886352301 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.1367056859333024) | > loader_time: 0.0052 (0.00469325401724954)  --> TIME: 2025-02-27 13:32:20 -- STEP: 19852/36938 -- GLOBAL_STEP: 1719000 | > loss_text_ce: 0.02742682956159115 (0.02972670700965434) | > loss_mel_ce: 2.775110960006714 (nan) | > loss: 0.011121181771159172 (nan) | > current_lr: 5e-06 | > step_time: 0.141 (0.13669445539659333) | > loader_time: 0.0046 (0.0046931783590445295)  --> TIME: 2025-02-27 13:32:29 -- STEP: 19902/36938 -- GLOBAL_STEP: 1719050 | > loss_text_ce: 0.024490095674991608 (0.029724913643013804) | > loss_mel_ce: 2.6788227558135986 (nan) | > loss: 0.010727432556450367 (nan) | > current_lr: 5e-06 | > step_time: 0.171 (0.13668064563888452) | > loader_time: 0.0048 (0.004693203781755832)  --> TIME: 2025-02-27 13:32:39 -- STEP: 19952/36938 -- GLOBAL_STEP: 1719100 | > loss_text_ce: 0.029546048492193222 (0.029723736242923855) | > loss_mel_ce: 2.7741034030914307 (nan) | > loss: 0.011125593446195126 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.1366808650344491) | > loader_time: 0.0046 (0.004693465476143114)  --> TIME: 2025-02-27 13:32:48 -- STEP: 20002/36938 -- GLOBAL_STEP: 1719150 | > loss_text_ce: 0.03005123697221279 (0.029723054216783527) | > loss_mel_ce: 2.550323963165283 (nan) | > loss: 0.010239584371447563 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13667410660381268) | > loader_time: 0.0047 (0.004693578152808624)  --> TIME: 2025-02-27 13:32:57 -- STEP: 20052/36938 -- GLOBAL_STEP: 1719200 | > loss_text_ce: 0.02446213737130165 (0.029721943524858595) | > loss_mel_ce: 0.5286195278167725 (nan) | > loss: 0.002194768749177456 (nan) | > current_lr: 5e-06 | > step_time: 0.1118 (0.1366793665133527) | > loader_time: 0.0048 (0.0046935177434353735)  --> TIME: 2025-02-27 13:33:07 -- STEP: 20102/36938 -- GLOBAL_STEP: 1719250 | > loss_text_ce: 0.016424544155597687 (0.029719953015232614) | > loss_mel_ce: 1.704886794090271 (nan) | > loss: 0.006830601021647453 (nan) | > current_lr: 5e-06 | > step_time: 0.1272 (0.13669051427578377) | > loader_time: 0.0046 (0.004693643511403127)  --> TIME: 2025-02-27 13:33:16 -- STEP: 20152/36938 -- GLOBAL_STEP: 1719300 | > loss_text_ce: 0.025741828605532646 (0.029718790988824368) | > loss_mel_ce: 2.7889881134033203 (nan) | > loss: 0.011169563978910446 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13668695455413343) | > loader_time: 0.0047 (0.00469371153514219)  --> TIME: 2025-02-27 13:33:25 -- STEP: 20202/36938 -- GLOBAL_STEP: 1719350 | > loss_text_ce: 0.034920189529657364 (0.02971564675100426) | > loss_mel_ce: 2.2788705825805664 (nan) | > loss: 0.009181709960103035 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13668728163898283) | > loader_time: 0.0046 (0.004693976334682372)  --> TIME: 2025-02-27 13:33:35 -- STEP: 20252/36938 -- GLOBAL_STEP: 1719400 | > loss_text_ce: 0.0326511487364769 (0.029716143882910672) | > loss_mel_ce: 2.9463727474212646 (nan) | > loss: 0.011821524240076542 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13668211686524062) | > loader_time: 0.0046 (0.004693900505215073)  --> TIME: 2025-02-27 13:33:44 -- STEP: 20302/36938 -- GLOBAL_STEP: 1719450 | > loss_text_ce: 0.0359085351228714 (0.02971490451226304) | > loss_mel_ce: 2.415942430496216 (nan) | > loss: 0.009729567915201187 (nan) | > current_lr: 5e-06 | > step_time: 0.1216 (0.1366692549512253) | > loader_time: 0.0046 (0.004693906843433234)  --> TIME: 2025-02-27 13:33:53 -- STEP: 20352/36938 -- GLOBAL_STEP: 1719500 | > loss_text_ce: 0.027346119284629822 (0.029714439892556562) | > loss_mel_ce: 1.6514467000961304 (nan) | > loss: 0.006661876570433378 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.13667353321124565) | > loader_time: 0.0045 (0.00469391968733857)  --> TIME: 2025-02-27 13:34:02 -- STEP: 20402/36938 -- GLOBAL_STEP: 1719550 | > loss_text_ce: 0.028466584160923958 (0.02971592269447195) | > loss_mel_ce: 2.36797833442688 (nan) | > loss: 0.009509703144431114 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.1366698633466397) | > loader_time: 0.0053 (0.004694020113586456)  --> TIME: 2025-02-27 13:34:12 -- STEP: 20452/36938 -- GLOBAL_STEP: 1719600 | > loss_text_ce: 0.03059423342347145 (0.029718753531231483) | > loss_mel_ce: 2.6056907176971436 (nan) | > loss: 0.01046144962310791 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13667395809450783) | > loader_time: 0.0046 (0.00469413121665685)  --> TIME: 2025-02-27 13:34:21 -- STEP: 20502/36938 -- GLOBAL_STEP: 1719650 | > loss_text_ce: 0.030337121337652206 (0.02971972909234696) | > loss_mel_ce: 4.4375 (nan) | > loss: 0.017729513347148895 (nan) | > current_lr: 5e-06 | > step_time: 0.1137 (0.1366789639490361) | > loader_time: 0.0047 (0.004694305400292034)  --> TIME: 2025-02-27 13:34:30 -- STEP: 20552/36938 -- GLOBAL_STEP: 1719700 | > loss_text_ce: 0.02883753925561905 (0.029720069124951915) | > loss_mel_ce: 2.606522798538208 (nan) | > loss: 0.01045777928084135 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13666401357444746) | > loader_time: 0.0046 (0.004694250839700378)  --> TIME: 2025-02-27 13:34:39 -- STEP: 20602/36938 -- GLOBAL_STEP: 1719750 | > loss_text_ce: 0.040461938828229904 (0.0297225169070474) | > loss_mel_ce: 0.17694091796875 (nan) | > loss: 0.0008627098286524415 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13665138482005973) | > loader_time: 0.0046 (0.00469415690780622)  --> TIME: 2025-02-27 13:34:49 -- STEP: 20652/36938 -- GLOBAL_STEP: 1719800 | > loss_text_ce: 0.029894396662712097 (0.029722289531130923) | > loss_mel_ce: 2.3358752727508545 (nan) | > loss: 0.009387975558638573 (nan) | > current_lr: 5e-06 | > step_time: 0.1551 (0.1366462558248169) | > loader_time: 0.0047 (0.004694014562554709)  --> TIME: 2025-02-27 13:34:58 -- STEP: 20702/36938 -- GLOBAL_STEP: 1719850 | > loss_text_ce: 0.02506531961262226 (0.02972101518733054) | > loss_mel_ce: 2.3566203117370605 (nan) | > loss: 0.009451134130358696 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13665300311126474) | > loader_time: 0.0045 (0.004694053382991455)  --> TIME: 2025-02-27 13:35:07 -- STEP: 20752/36938 -- GLOBAL_STEP: 1719900 | > loss_text_ce: 0.02727089822292328 (0.029719331337045837) | > loss_mel_ce: 3.1448974609375 (nan) | > loss: 0.01258797012269497 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13664373539462898) | > loader_time: 0.0067 (0.004694137776826275)  --> TIME: 2025-02-27 13:35:17 -- STEP: 20802/36938 -- GLOBAL_STEP: 1719950 | > loss_text_ce: 0.028934402391314507 (0.0297166021160791) | > loss_mel_ce: 2.6988744735717773 (nan) | > loss: 0.010824639350175858 (nan) | > current_lr: 5e-06 | > step_time: 0.1469 (0.1366525365462526) | > loader_time: 0.0047 (0.004694071208934406)  --> TIME: 2025-02-27 13:35:27 -- STEP: 20852/36938 -- GLOBAL_STEP: 1720000 | > loss_text_ce: 0.02987079508602619 (0.0297172630225447) | > loss_mel_ce: 3.1559791564941406 (nan) | > loss: 0.012642262503504753 (nan) | > current_lr: 5e-06 | > step_time: 0.1663 (0.1366664989196296) | > loader_time: 0.0049 (0.00469418057273953) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1720000.pth  --> TIME: 2025-02-27 13:35:43 -- STEP: 20902/36938 -- GLOBAL_STEP: 1720050 | > loss_text_ce: 0.03008350357413292 (0.02971599717916242) | > loss_mel_ce: 2.345043420791626 (nan) | > loss: 0.009425107389688492 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13666829104948733) | > loader_time: 0.0047 (0.004694321921836487)  --> TIME: 2025-02-27 13:35:52 -- STEP: 20952/36938 -- GLOBAL_STEP: 1720100 | > loss_text_ce: 0.03594673424959183 (0.02971698428842791) | > loss_mel_ce: 2.9076507091522217 (nan) | > loss: 0.011680942960083485 (nan) | > current_lr: 5e-06 | > step_time: 0.1397 (0.13665917421306778) | > loader_time: 0.0049 (0.004694190825065059)  --> TIME: 2025-02-27 13:36:01 -- STEP: 21002/36938 -- GLOBAL_STEP: 1720150 | > loss_text_ce: 0.02317509800195694 (0.029713542100428605) | > loss_mel_ce: 2.921380043029785 (nan) | > loss: 0.011684742756187916 (nan) | > current_lr: 5e-06 | > step_time: 0.1796 (0.1366643112462525) | > loader_time: 0.0049 (0.004694147559985934)  --> TIME: 2025-02-27 13:36:11 -- STEP: 21052/36938 -- GLOBAL_STEP: 1720200 | > loss_text_ce: 0.030690252780914307 (0.029712724191964454) | > loss_mel_ce: 2.6992857456207275 (nan) | > loss: 0.010833238251507282 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.1366692060484724) | > loader_time: 0.0047 (0.004694419624731599)  --> TIME: 2025-02-27 13:36:20 -- STEP: 21102/36938 -- GLOBAL_STEP: 1720250 | > loss_text_ce: 0.024261578917503357 (0.029712469053294308) | > loss_mel_ce: 2.845571994781494 (nan) | > loss: 0.01138822827488184 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13667410568607619) | > loader_time: 0.0046 (0.004694558514112443)  --> TIME: 2025-02-27 13:36:29 -- STEP: 21152/36938 -- GLOBAL_STEP: 1720300 | > loss_text_ce: 0.02947591058909893 (0.029712025228516705) | > loss_mel_ce: 2.50089168548584 (nan) | > loss: 0.010041141882538795 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.1366677495021152) | > loader_time: 0.0047 (0.0046943999860723055)  --> TIME: 2025-02-27 13:36:39 -- STEP: 21202/36938 -- GLOBAL_STEP: 1720350 | > loss_text_ce: 0.02900334820151329 (0.029712952519627734) | > loss_mel_ce: 2.8692047595977783 (nan) | > loss: 0.011500827036798 (nan) | > current_lr: 5e-06 | > step_time: 0.1846 (0.136665434468953) | > loader_time: 0.0049 (0.004694515821472915)  --> TIME: 2025-02-27 13:36:48 -- STEP: 21252/36938 -- GLOBAL_STEP: 1720400 | > loss_text_ce: 0.023300915956497192 (0.02971027791857247) | > loss_mel_ce: 2.6974384784698486 (nan) | > loss: 0.010796585120260715 (nan) | > current_lr: 5e-06 | > step_time: 0.1207 (0.13665639394962725) | > loader_time: 0.0046 (0.004694422680667932)  --> TIME: 2025-02-27 13:36:57 -- STEP: 21302/36938 -- GLOBAL_STEP: 1720450 | > loss_text_ce: 0.023544348776340485 (0.0297105422148997) | > loss_mel_ce: 1.5369491577148438 (nan) | > loss: 0.006192434579133987 (nan) | > current_lr: 5e-06 | > step_time: 0.1367 (0.1366536819859868) | > loader_time: 0.0066 (0.004694721193092429)  --> TIME: 2025-02-27 13:37:06 -- STEP: 21352/36938 -- GLOBAL_STEP: 1720500 | > loss_text_ce: 0.02883688546717167 (0.029711562676041015) | > loss_mel_ce: 2.8480911254882812 (nan) | > loss: 0.011416382156312466 (nan) | > current_lr: 5e-06 | > step_time: 0.1737 (0.13665309198879144) | > loader_time: 0.0047 (0.004694687076256301)  --> TIME: 2025-02-27 13:37:16 -- STEP: 21402/36938 -- GLOBAL_STEP: 1720550 | > loss_text_ce: 0.025941137224435806 (0.02971118395137564) | > loss_mel_ce: 2.9745144844055176 (nan) | > loss: 0.011906570754945278 (nan) | > current_lr: 5e-06 | > step_time: 0.1587 (0.13666145835632731) | > loader_time: 0.0049 (0.004694822536162274)  --> TIME: 2025-02-27 13:37:25 -- STEP: 21452/36938 -- GLOBAL_STEP: 1720600 | > loss_text_ce: 0.03622323274612427 (0.02971150962008191) | > loss_mel_ce: 2.823621988296509 (nan) | > loss: 0.01134859211742878 (nan) | > current_lr: 5e-06 | > step_time: 0.1656 (0.13665159740444538) | > loader_time: 0.0049 (0.004694697051356791)  --> TIME: 2025-02-27 13:37:34 -- STEP: 21502/36938 -- GLOBAL_STEP: 1720650 | > loss_text_ce: 0.02174145169556141 (0.029711144059786795) | > loss_mel_ce: 0.0028476715087890625 (nan) | > loss: 9.757588850334287e-05 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13664301369159484) | > loader_time: 0.0045 (0.0046945732811441605)  --> TIME: 2025-02-27 13:37:44 -- STEP: 21552/36938 -- GLOBAL_STEP: 1720700 | > loss_text_ce: 0.031417280435562134 (0.02970996226144573) | > loss_mel_ce: 2.217282295227051 (nan) | > loss: 0.008923412300646305 (nan) | > current_lr: 5e-06 | > step_time: 0.1622 (0.1366449219037102) | > loader_time: 0.0046 (0.0046944747656119)  --> TIME: 2025-02-27 13:37:53 -- STEP: 21602/36938 -- GLOBAL_STEP: 1720750 | > loss_text_ce: 0.03414801135659218 (0.029711058635855964) | > loss_mel_ce: 1.7480648756027222 (nan) | > loss: 0.007072273641824722 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13665171092709558) | > loader_time: 0.0047 (0.004694439042399238)  --> TIME: 2025-02-27 13:38:02 -- STEP: 21652/36938 -- GLOBAL_STEP: 1720800 | > loss_text_ce: 0.031903669238090515 (0.02971127025544166) | > loss_mel_ce: 1.3483377695083618 (nan) | > loss: 0.005477148573845625 (nan) | > current_lr: 5e-06 | > step_time: 0.1606 (0.13664717737460852) | > loader_time: 0.0047 (0.0046945129814197125)  --> TIME: 2025-02-27 13:38:12 -- STEP: 21702/36938 -- GLOBAL_STEP: 1720850 | > loss_text_ce: 0.03201190009713173 (0.02971247095009496) | > loss_mel_ce: 2.0700721740722656 (nan) | > loss: 0.00834160391241312 (nan) | > current_lr: 5e-06 | > step_time: 0.1295 (0.13664084371361762) | > loader_time: 0.0046 (0.004694367584440843)  --> TIME: 2025-02-27 13:38:21 -- STEP: 21752/36938 -- GLOBAL_STEP: 1720900 | > loss_text_ce: 0.0315302237868309 (0.029712879602825926) | > loss_mel_ce: 2.9756979942321777 (nan) | > loss: 0.011933445930480957 (nan) | > current_lr: 5e-06 | > step_time: 0.1149 (0.13664008779489048) | > loader_time: 0.0045 (0.004694454029405225)  --> TIME: 2025-02-27 13:38:30 -- STEP: 21802/36938 -- GLOBAL_STEP: 1720950 | > loss_text_ce: 0.023824850097298622 (0.02970863674531857) | > loss_mel_ce: 1.6115412712097168 (nan) | > loss: 0.0064895483665168285 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13663398530568444) | > loader_time: 0.0046 (0.004694501726619246)  --> TIME: 2025-02-27 13:38:40 -- STEP: 21852/36938 -- GLOBAL_STEP: 1721000 | > loss_text_ce: 0.021745163947343826 (0.02971018047674037) | > loss_mel_ce: 2.4665019512176514 (nan) | > loss: 0.009873997420072556 (nan) | > current_lr: 5e-06 | > step_time: 0.1276 (0.13664086445901563) | > loader_time: 0.0048 (0.004694477315571361)  --> TIME: 2025-02-27 13:38:49 -- STEP: 21902/36938 -- GLOBAL_STEP: 1721050 | > loss_text_ce: 0.03278012573719025 (0.0297114244697221) | > loss_mel_ce: 2.9611988067626953 (nan) | > loss: 0.01188086997717619 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13663529717272446) | > loader_time: 0.0046 (0.004694431582037985)  --> TIME: 2025-02-27 13:38:58 -- STEP: 21952/36938 -- GLOBAL_STEP: 1721100 | > loss_text_ce: 0.022229423746466637 (0.029711127149436852) | > loss_mel_ce: 2.097592830657959 (nan) | > loss: 0.008411993272602558 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.13663230360683948) | > loader_time: 0.0045 (0.0046943406473964645)  --> TIME: 2025-02-27 13:39:08 -- STEP: 22002/36938 -- GLOBAL_STEP: 1721150 | > loss_text_ce: 0.02726670354604721 (0.02971148008272663) | > loss_mel_ce: 3.3344225883483887 (nan) | > loss: 0.013340038247406483 (nan) | > current_lr: 5e-06 | > step_time: 0.1666 (0.1366408243145524) | > loader_time: 0.0047 (0.004694197451176504)  --> TIME: 2025-02-27 13:39:17 -- STEP: 22052/36938 -- GLOBAL_STEP: 1721200 | > loss_text_ce: 0.030380623415112495 (0.02970939018870789) | > loss_mel_ce: 2.8294618129730225 (nan) | > loss: 0.011348581872880459 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.13663674147964272) | > loader_time: 0.0047 (0.00469413041290042)  --> TIME: 2025-02-27 13:39:26 -- STEP: 22102/36938 -- GLOBAL_STEP: 1721250 | > loss_text_ce: 0.026934677734971046 (0.02970805487378237) | > loss_mel_ce: 2.406764268875122 (nan) | > loss: 0.009657535701990128 (nan) | > current_lr: 5e-06 | > step_time: 0.1563 (0.13664276271306194) | > loader_time: 0.0047 (0.00469404703329595)  --> TIME: 2025-02-27 13:39:36 -- STEP: 22152/36938 -- GLOBAL_STEP: 1721300 | > loss_text_ce: 0.028701212257146835 (0.029707576846682862) | > loss_mel_ce: 3.0480241775512695 (nan) | > loss: 0.012209229171276093 (nan) | > current_lr: 5e-06 | > step_time: 0.1803 (0.13664217829790504) | > loader_time: 0.0049 (0.004694032643236547)  --> TIME: 2025-02-27 13:39:45 -- STEP: 22202/36938 -- GLOBAL_STEP: 1721350 | > loss_text_ce: 0.030672485008835793 (0.02970824011971234) | > loss_mel_ce: 2.5376546382904053 (nan) | > loss: 0.010191774927079678 (nan) | > current_lr: 5e-06 | > step_time: 0.1506 (0.13664162872009758) | > loader_time: 0.0079 (0.0046942956103321505)  --> TIME: 2025-02-27 13:39:54 -- STEP: 22252/36938 -- GLOBAL_STEP: 1721400 | > loss_text_ce: 0.0378919281065464 (0.029712204709156885) | > loss_mel_ce: 2.6825661659240723 (nan) | > loss: 0.010795469395816326 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.1366419520670415) | > loader_time: 0.0048 (0.004694409985862012)  --> TIME: 2025-02-27 13:40:03 -- STEP: 22302/36938 -- GLOBAL_STEP: 1721450 | > loss_text_ce: 0.03757015988230705 (0.029712960092111468) | > loss_mel_ce: 2.3837242126464844 (nan) | > loss: 0.009608311578631401 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.1366314758859776) | > loader_time: 0.0049 (0.004694340015439079)  --> TIME: 2025-02-27 13:40:13 -- STEP: 22352/36938 -- GLOBAL_STEP: 1721500 | > loss_text_ce: 0.030447563156485558 (0.029713482408607328) | > loss_mel_ce: 2.427942991256714 (nan) | > loss: 0.009755518287420273 (nan) | > current_lr: 5e-06 | > step_time: 0.1545 (0.13663671268636998) | > loader_time: 0.0047 (0.004694228875867441)  --> TIME: 2025-02-27 13:40:22 -- STEP: 22402/36938 -- GLOBAL_STEP: 1721550 | > loss_text_ce: 0.024578912183642387 (0.02971049925244552) | > loss_mel_ce: 2.1725761890411377 (nan) | > loss: 0.008718869648873806 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13663687646181605) | > loader_time: 0.0045 (0.004694132536247928)  --> TIME: 2025-02-27 13:40:31 -- STEP: 22452/36938 -- GLOBAL_STEP: 1721600 | > loss_text_ce: 0.020009290426969528 (0.029711554094104718) | > loss_mel_ce: 1.1347960233688354 (nan) | > loss: 0.004582561086863279 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13663094320463876) | > loader_time: 0.0045 (0.004694047510231019)  --> TIME: 2025-02-27 13:40:41 -- STEP: 22502/36938 -- GLOBAL_STEP: 1721650 | > loss_text_ce: 0.026073526591062546 (0.02971079153070924) | > loss_mel_ce: 2.1195883750915527 (nan) | > loss: 0.008514531888067722 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13663085838199673) | > loader_time: 0.0045 (0.004694232420649648)  --> TIME: 2025-02-27 13:40:50 -- STEP: 22552/36938 -- GLOBAL_STEP: 1721700 | > loss_text_ce: 0.032125700265169144 (0.029708687902435475) | > loss_mel_ce: 2.940174102783203 (nan) | > loss: 0.011794840916991234 (nan) | > current_lr: 5e-06 | > step_time: 0.1678 (0.1366288569346533) | > loader_time: 0.0049 (0.004694078895968713)  --> TIME: 2025-02-27 13:41:00 -- STEP: 22602/36938 -- GLOBAL_STEP: 1721750 | > loss_text_ce: 0.024328293278813362 (0.029709241747433862) | > loss_mel_ce: 2.7135488986968994 (nan) | > loss: 0.010864592157304287 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13664017526589423) | > loader_time: 0.0048 (0.004694248309083528)  --> TIME: 2025-02-27 13:41:09 -- STEP: 22652/36938 -- GLOBAL_STEP: 1721800 | > loss_text_ce: 0.03720478340983391 (0.029710248857917485) | > loss_mel_ce: 2.137545347213745 (nan) | > loss: 0.00862996093928814 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1366461105782861) | > loader_time: 0.0045 (0.004694322141556051)  --> TIME: 2025-02-27 13:41:18 -- STEP: 22702/36938 -- GLOBAL_STEP: 1721850 | > loss_text_ce: 0.03231348097324371 (0.029711087920564573) | > loss_mel_ce: 2.3273026943206787 (nan) | > loss: 0.00936355721205473 (nan) | > current_lr: 5e-06 | > step_time: 0.1306 (0.13664445323824934) | > loader_time: 0.0046 (0.004694465970837599)  --> TIME: 2025-02-27 13:41:27 -- STEP: 22752/36938 -- GLOBAL_STEP: 1721900 | > loss_text_ce: 0.04274670034646988 (0.02971048019521865) | > loss_mel_ce: 2.200153112411499 (nan) | > loss: 0.008900396525859833 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.13663660642676698) | > loader_time: 0.0047 (0.004694448734218381)  --> TIME: 2025-02-27 13:41:37 -- STEP: 22802/36938 -- GLOBAL_STEP: 1721950 | > loss_text_ce: 0.019336553290486336 (0.029710396318685408) | > loss_mel_ce: 2.359417676925659 (nan) | > loss: 0.009439500980079174 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13664191447374327) | > loader_time: 0.0046 (0.00469463999339774)  --> TIME: 2025-02-27 13:41:46 -- STEP: 22852/36938 -- GLOBAL_STEP: 1722000 | > loss_text_ce: 0.03191208839416504 (0.02971020959304316) | > loss_mel_ce: 2.214365005493164 (nan) | > loss: 0.00891379825770855 (nan) | > current_lr: 5e-06 | > step_time: 0.1283 (0.13664383440036212) | > loader_time: 0.0046 (0.004694541333624375)  --> TIME: 2025-02-27 13:41:55 -- STEP: 22902/36938 -- GLOBAL_STEP: 1722050 | > loss_text_ce: 0.022274481132626534 (0.029710226252176654) | > loss_mel_ce: 2.0230324268341064 (nan) | > loss: 0.008116297423839569 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13663312021108567) | > loader_time: 0.0045 (0.004694378341652807)  --> TIME: 2025-02-27 13:42:04 -- STEP: 22952/36938 -- GLOBAL_STEP: 1722100 | > loss_text_ce: 0.031056007370352745 (0.029708772832918912) | > loss_mel_ce: 2.9501519203186035 (nan) | > loss: 0.011830190196633339 (nan) | > current_lr: 5e-06 | > step_time: 0.1305 (0.13662101557421963) | > loader_time: 0.0046 (0.004694290498109918)  --> TIME: 2025-02-27 13:42:13 -- STEP: 23002/36938 -- GLOBAL_STEP: 1722150 | > loss_text_ce: 0.02605411969125271 (0.029707832319454505) | > loss_mel_ce: 2.9809257984161377 (nan) | > loss: 0.011932460591197014 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13661167668463037) | > loader_time: 0.0046 (0.004694232203920815)  --> TIME: 2025-02-27 13:42:23 -- STEP: 23052/36938 -- GLOBAL_STEP: 1722200 | > loss_text_ce: 0.030195027589797974 (0.02970564659429077) | > loss_mel_ce: 2.2913331985473633 (nan) | > loss: 0.009212413802742958 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13660789528965522) | > loader_time: 0.0047 (0.0046942807745648)  --> TIME: 2025-02-27 13:42:32 -- STEP: 23102/36938 -- GLOBAL_STEP: 1722250 | > loss_text_ce: 0.028442416340112686 (0.029705203674179243) | > loss_mel_ce: 2.6773245334625244 (nan) | > loss: 0.010737170465290546 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13661069879696291) | > loader_time: 0.0046 (0.0046942604639738355)  --> TIME: 2025-02-27 13:42:41 -- STEP: 23152/36938 -- GLOBAL_STEP: 1722300 | > loss_text_ce: 0.036939118057489395 (0.029703724528905823) | > loss_mel_ce: 2.9819300174713135 (nan) | > loss: 0.01197964046150446 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13661077616539477) | > loader_time: 0.0045 (0.004694143296029528)  --> TIME: 2025-02-27 13:42:51 -- STEP: 23202/36938 -- GLOBAL_STEP: 1722350 | > loss_text_ce: 0.02907724119722843 (0.02970491652386984) | > loss_mel_ce: 2.7541699409484863 (nan) | > loss: 0.011044632643461227 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13661424072330858) | > loader_time: 0.0048 (0.004694023622273254)  --> TIME: 2025-02-27 13:43:00 -- STEP: 23252/36938 -- GLOBAL_STEP: 1722400 | > loss_text_ce: 0.03321687504649162 (0.02970342496740853) | > loss_mel_ce: 3.185448408126831 (nan) | > loss: 0.012772481888532639 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13661833336897705) | > loader_time: 0.0046 (0.004693853123020258)  --> TIME: 2025-02-27 13:43:09 -- STEP: 23302/36938 -- GLOBAL_STEP: 1722450 | > loss_text_ce: 0.01813884638249874 (0.029700283685381117) | > loss_mel_ce: 1.8727681636810303 (nan) | > loss: 0.007503599859774113 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.13660736171363042) | > loader_time: 0.0045 (0.00469392541651765)  --> TIME: 2025-02-27 13:43:18 -- STEP: 23352/36938 -- GLOBAL_STEP: 1722500 | > loss_text_ce: 0.018079757690429688 (0.02970057448352144) | > loss_mel_ce: 1.965330719947815 (nan) | > loss: 0.007870676927268505 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.13660557748275862) | > loader_time: 0.0045 (0.00469385118629883)  --> TIME: 2025-02-27 13:43:27 -- STEP: 23402/36938 -- GLOBAL_STEP: 1722550 | > loss_text_ce: 0.028295274823904037 (0.029699530874732927) | > loss_mel_ce: 2.535874605178833 (nan) | > loss: 0.010175277478992939 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13659697473074767) | > loader_time: 0.0047 (0.00469375036688056)  --> TIME: 2025-02-27 13:43:37 -- STEP: 23452/36938 -- GLOBAL_STEP: 1722600 | > loss_text_ce: 0.025066031143069267 (0.029701894369346782) | > loss_mel_ce: 1.479498028755188 (nan) | > loss: 0.005970492493361235 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.1366118286812093) | > loader_time: 0.0046 (0.004693777238308066)  --> TIME: 2025-02-27 13:43:47 -- STEP: 23502/36938 -- GLOBAL_STEP: 1722650 | > loss_text_ce: 0.027902109548449516 (0.02970199017836133) | > loss_mel_ce: 2.988274574279785 (nan) | > loss: 0.011968955397605896 (nan) | > current_lr: 5e-06 | > step_time: 0.1512 (0.13661069624901528) | > loader_time: 0.0046 (0.004693819019563359)  --> TIME: 2025-02-27 13:43:56 -- STEP: 23552/36938 -- GLOBAL_STEP: 1722700 | > loss_text_ce: 0.03411165624856949 (0.02970262004225293) | > loss_mel_ce: 3.1840667724609375 (nan) | > loss: 0.012770550325512886 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13661732200695278) | > loader_time: 0.0046 (0.0046937973643450545)  --> TIME: 2025-02-27 13:44:06 -- STEP: 23602/36938 -- GLOBAL_STEP: 1722750 | > loss_text_ce: 0.032912228256464005 (0.02970356917631901) | > loss_mel_ce: 2.414511203765869 (nan) | > loss: 0.009711998514831066 (nan) | > current_lr: 5e-06 | > step_time: 0.1792 (0.13662684990101) | > loader_time: 0.0049 (0.004693909879761539)  --> TIME: 2025-02-27 13:44:15 -- STEP: 23652/36938 -- GLOBAL_STEP: 1722800 | > loss_text_ce: 0.022965947166085243 (0.02970669791129972) | > loss_mel_ce: 2.581808567047119 (nan) | > loss: 0.010336407460272312 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13662941784861274) | > loader_time: 0.0048 (0.004693972566457198)  --> TIME: 2025-02-27 13:44:25 -- STEP: 23702/36938 -- GLOBAL_STEP: 1722850 | > loss_text_ce: 0.01847989484667778 (0.029706617060774633) | > loss_mel_ce: 2.0729730129241943 (nan) | > loss: 0.008299416862428188 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13663457734767434) | > loader_time: 0.0045 (0.00469391602080096)  --> TIME: 2025-02-27 13:44:34 -- STEP: 23752/36938 -- GLOBAL_STEP: 1722900 | > loss_text_ce: 0.026594817638397217 (0.02970568820815809) | > loss_mel_ce: 2.5179686546325684 (nan) | > loss: 0.010097474791109562 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1366350197478889) | > loader_time: 0.0045 (0.004694124139812749)  --> TIME: 2025-02-27 13:44:43 -- STEP: 23802/36938 -- GLOBAL_STEP: 1722950 | > loss_text_ce: 0.02589268609881401 (0.029704977646831606) | > loss_mel_ce: 2.6788060665130615 (nan) | > loss: 0.010732932016253471 (nan) | > current_lr: 5e-06 | > step_time: 0.1186 (0.1366391009858128) | > loader_time: 0.0045 (0.004693941232206723)  --> TIME: 2025-02-27 13:44:53 -- STEP: 23852/36938 -- GLOBAL_STEP: 1723000 | > loss_text_ce: 0.03681319206953049 (0.029704587035414946) | > loss_mel_ce: 2.5653858184814453 (nan) | > loss: 0.010326187126338482 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13663959742072368) | > loader_time: 0.0045 (0.0046937994542746424)  --> TIME: 2025-02-27 13:45:02 -- STEP: 23902/36938 -- GLOBAL_STEP: 1723050 | > loss_text_ce: 0.02753579244017601 (0.02970541720349865) | > loss_mel_ce: 2.5990805625915527 (nan) | > loss: 0.010423080995678902 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13663846219935724) | > loader_time: 0.0046 (0.004693661301856796)  --> TIME: 2025-02-27 13:45:12 -- STEP: 23952/36938 -- GLOBAL_STEP: 1723100 | > loss_text_ce: 0.03037521056830883 (0.029705994915989836) | > loss_mel_ce: 3.0235812664031982 (nan) | > loss: 0.012118875980377197 (nan) | > current_lr: 5e-06 | > step_time: 0.1787 (0.13665334515039812) | > loader_time: 0.0048 (0.004693596549804958)  --> TIME: 2025-02-27 13:45:21 -- STEP: 24002/36938 -- GLOBAL_STEP: 1723150 | > loss_text_ce: 0.03296920657157898 (0.029704744987805627) | > loss_mel_ce: 2.122751235961914 (nan) | > loss: 0.008554446510970592 (nan) | > current_lr: 5e-06 | > step_time: 0.1218 (0.13664918070623971) | > loader_time: 0.0046 (0.004693627516415385)  --> TIME: 2025-02-27 13:45:30 -- STEP: 24052/36938 -- GLOBAL_STEP: 1723200 | > loss_text_ce: 0.029314285144209862 (0.029705227162092968) | > loss_mel_ce: 2.7948317527770996 (nan) | > loss: 0.011206929571926594 (nan) | > current_lr: 5e-06 | > step_time: 0.1781 (0.13665377383975336) | > loader_time: 0.0049 (0.004693681113675925)  --> TIME: 2025-02-27 13:45:40 -- STEP: 24102/36938 -- GLOBAL_STEP: 1723250 | > loss_text_ce: 0.035064417868852615 (0.029706204897555537) | > loss_mel_ce: 2.647902250289917 (nan) | > loss: 0.010646694339811802 (nan) | > current_lr: 5e-06 | > step_time: 0.1099 (0.13666405909072082) | > loader_time: 0.0047 (0.004693699381616775)  --> TIME: 2025-02-27 13:45:49 -- STEP: 24152/36938 -- GLOBAL_STEP: 1723300 | > loss_text_ce: 0.029136648401618004 (0.029706228775477867) | > loss_mel_ce: 2.278521776199341 (nan) | > loss: 0.009157375432550907 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13666108732351612) | > loader_time: 0.0046 (0.004693695984758577)  --> TIME: 2025-02-27 13:45:59 -- STEP: 24202/36938 -- GLOBAL_STEP: 1723350 | > loss_text_ce: 0.05103466287255287 (0.029705739158302116) | > loss_mel_ce: 2.2856929302215576 (nan) | > loss: 0.00927272904664278 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13665738883821324) | > loader_time: 0.0052 (0.0046937704165113244)  --> TIME: 2025-02-27 13:46:08 -- STEP: 24252/36938 -- GLOBAL_STEP: 1723400 | > loss_text_ce: 0.02726840227842331 (0.02970438651827646) | > loss_mel_ce: 0.0108642578125 (nan) | > loss: 0.000151320084114559 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13664501286095795) | > loader_time: 0.0047 (0.004693775577708749)  --> TIME: 2025-02-27 13:46:17 -- STEP: 24302/36938 -- GLOBAL_STEP: 1723450 | > loss_text_ce: 0.032107528299093246 (0.029704277143593614) | > loss_mel_ce: 2.5628249645233154 (nan) | > loss: 0.010297352448105812 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.1366527287835079) | > loader_time: 0.0045 (0.004693780943313533)  --> TIME: 2025-02-27 13:46:26 -- STEP: 24352/36938 -- GLOBAL_STEP: 1723500 | > loss_text_ce: 0.028612583875656128 (0.02970385914202626) | > loss_mel_ce: 3.016509532928467 (nan) | > loss: 0.012083819136023521 (nan) | > current_lr: 5e-06 | > step_time: 0.1442 (0.13664866306387163) | > loader_time: 0.0047 (0.004693841280197814)  --> TIME: 2025-02-27 13:46:36 -- STEP: 24402/36938 -- GLOBAL_STEP: 1723550 | > loss_text_ce: 0.02702070213854313 (0.02970543472858419) | > loss_mel_ce: 1.3245792388916016 (nan) | > loss: 0.0053634922951459885 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.1366573703766821) | > loader_time: 0.0045 (0.004693885316966779)  --> TIME: 2025-02-27 13:46:45 -- STEP: 24452/36938 -- GLOBAL_STEP: 1723600 | > loss_text_ce: 0.03957387059926987 (0.029705118071009062) | > loss_mel_ce: 1.8853548765182495 (nan) | > loss: 0.007638606708496809 (nan) | > current_lr: 5e-06 | > step_time: 0.1211 (0.13665847530268402) | > loader_time: 0.0046 (0.004693846889394755)  --> TIME: 2025-02-27 13:46:54 -- STEP: 24502/36938 -- GLOBAL_STEP: 1723650 | > loss_text_ce: 0.033822864294052124 (0.029707727001069374) | > loss_mel_ce: 2.717446804046631 (nan) | > loss: 0.010917737148702145 (nan) | > current_lr: 5e-06 | > step_time: 0.1213 (0.13664725893089438) | > loader_time: 0.0046 (0.004693818923337401)  --> TIME: 2025-02-27 13:47:04 -- STEP: 24552/36938 -- GLOBAL_STEP: 1723700 | > loss_text_ce: 0.031101515516638756 (0.029708956480548193) | > loss_mel_ce: 2.474392890930176 (nan) | > loss: 0.009942438453435898 (nan) | > current_lr: 5e-06 | > step_time: 0.1417 (0.13665083902361636) | > loader_time: 0.0048 (0.00469382809831328)  --> TIME: 2025-02-27 13:47:13 -- STEP: 24602/36938 -- GLOBAL_STEP: 1723750 | > loss_text_ce: 0.02902882546186447 (0.02970893459711652) | > loss_mel_ce: 2.371128797531128 (nan) | > loss: 0.009524435736238956 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.13666474260980493) | > loader_time: 0.0066 (0.0046939738988198)  --> TIME: 2025-02-27 13:47:23 -- STEP: 24652/36938 -- GLOBAL_STEP: 1723800 | > loss_text_ce: 0.032395049929618835 (0.02970987508636655) | > loss_mel_ce: 2.8796207904815674 (nan) | > loss: 0.011555619537830353 (nan) | > current_lr: 5e-06 | > step_time: 0.1785 (0.1366704579260813) | > loader_time: 0.0047 (0.00469393313714442)  --> TIME: 2025-02-27 13:47:32 -- STEP: 24702/36938 -- GLOBAL_STEP: 1723850 | > loss_text_ce: 0.022301236167550087 (0.029711664209615712) | > loss_mel_ce: 0.9788840413093567 (nan) | > loss: 0.003972957842051983 (nan) | > current_lr: 5e-06 | > step_time: 0.1133 (0.13666683416619232) | > loader_time: 0.0045 (0.004693727407964293)  --> TIME: 2025-02-27 13:47:42 -- STEP: 24752/36938 -- GLOBAL_STEP: 1723900 | > loss_text_ce: 0.02727479860186577 (0.029713042918976473) | > loss_mel_ce: 2.7073872089385986 (nan) | > loss: 0.01085183396935463 (nan) | > current_lr: 5e-06 | > step_time: 0.171 (0.13667335331863956) | > loader_time: 0.0048 (0.004693589974543943)  --> TIME: 2025-02-27 13:47:51 -- STEP: 24802/36938 -- GLOBAL_STEP: 1723950 | > loss_text_ce: 0.03203144297003746 (0.029714556329086587) | > loss_mel_ce: 2.114696741104126 (nan) | > loss: 0.008518763817846775 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13666447747398877) | > loader_time: 0.0052 (0.004693707471201094)  --> TIME: 2025-02-27 13:48:00 -- STEP: 24852/36938 -- GLOBAL_STEP: 1724000 | > loss_text_ce: 0.032888513058423996 (0.029714735968716877) | > loss_mel_ce: 2.7379956245422363 (nan) | > loss: 0.010995572432875633 (nan) | > current_lr: 5e-06 | > step_time: 0.1592 (0.13666676398580913) | > loader_time: 0.0047 (0.004693635982072593)  --> TIME: 2025-02-27 13:48:09 -- STEP: 24902/36938 -- GLOBAL_STEP: 1724050 | > loss_text_ce: 0.012141680344939232 (0.02971649976307649) | > loss_mel_ce: 1.3327566385269165 (nan) | > loss: 0.0053368983790278435 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.1366583064477361) | > loader_time: 0.0045 (0.004693544367673118)  --> TIME: 2025-02-27 13:48:19 -- STEP: 24952/36938 -- GLOBAL_STEP: 1724100 | > loss_text_ce: 0.028995973989367485 (0.029717095737863877) | > loss_mel_ce: 1.5388387441635132 (nan) | > loss: 0.006221566814929247 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13665798342774646) | > loader_time: 0.0045 (0.004693378351865279)  --> TIME: 2025-02-27 13:48:28 -- STEP: 25002/36938 -- GLOBAL_STEP: 1724150 | > loss_text_ce: 0.03161609545350075 (0.02971527041814033) | > loss_mel_ce: 2.024824619293213 (nan) | > loss: 0.008160480298101902 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13665923999983048) | > loader_time: 0.0046 (0.004693372832633117)  --> TIME: 2025-02-27 13:48:37 -- STEP: 25052/36938 -- GLOBAL_STEP: 1724200 | > loss_text_ce: 0.03294157236814499 (0.029715206178780162) | > loss_mel_ce: 2.8298072814941406 (nan) | > loss: 0.011360115371644497 (nan) | > current_lr: 5e-06 | > step_time: 0.1444 (0.13666637983448687) | > loader_time: 0.0046 (0.004693220450857981)  --> TIME: 2025-02-27 13:48:47 -- STEP: 25102/36938 -- GLOBAL_STEP: 1724250 | > loss_text_ce: 0.021040933206677437 (0.029714744856956406) | > loss_mel_ce: 2.7308287620544434 (nan) | > loss: 0.010920118540525436 (nan) | > current_lr: 5e-06 | > step_time: 0.1605 (0.1366612209797719) | > loader_time: 0.0046 (0.004693109441512095)  --> TIME: 2025-02-27 13:48:56 -- STEP: 25152/36938 -- GLOBAL_STEP: 1724300 | > loss_text_ce: 0.02458188869059086 (0.029714865823172364) | > loss_mel_ce: 2.1445932388305664 (nan) | > loss: 0.008607838302850723 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13666879266273144) | > loader_time: 0.0046 (0.004693119826965979)  --> TIME: 2025-02-27 13:49:05 -- STEP: 25202/36938 -- GLOBAL_STEP: 1724350 | > loss_text_ce: 0.0401911735534668 (0.029714354998119093) | > loss_mel_ce: 2.8697242736816406 (nan) | > loss: 0.011547284200787544 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.13666329119150972) | > loader_time: 0.0052 (0.004693070892946725)  --> TIME: 2025-02-27 13:49:14 -- STEP: 25252/36938 -- GLOBAL_STEP: 1724400 | > loss_text_ce: 0.028334563598036766 (0.029714335928181717) | > loss_mel_ce: 2.1351537704467773 (nan) | > loss: 0.008585271425545216 (nan) | > current_lr: 5e-06 | > step_time: 0.1126 (0.1366523194007159) | > loader_time: 0.0045 (0.00469294548714463)  --> TIME: 2025-02-27 13:49:23 -- STEP: 25302/36938 -- GLOBAL_STEP: 1724450 | > loss_text_ce: 0.022509893402457237 (0.029714314667770813) | > loss_mel_ce: 2.2580788135528564 (nan) | > loss: 0.009049955755472183 (nan) | > current_lr: 5e-06 | > step_time: 0.1081 (0.13664277404035716) | > loader_time: 0.0044 (0.004692702140367882)  --> TIME: 2025-02-27 13:49:32 -- STEP: 25352/36938 -- GLOBAL_STEP: 1724500 | > loss_text_ce: 0.025294605642557144 (0.029714395118968097) | > loss_mel_ce: 1.6907227039337158 (nan) | > loss: 0.006809593178331852 (nan) | > current_lr: 5e-06 | > step_time: 0.115 (0.13662800869004457) | > loader_time: 0.0046 (0.004692626492237601)  --> TIME: 2025-02-27 13:49:41 -- STEP: 25402/36938 -- GLOBAL_STEP: 1724550 | > loss_text_ce: 0.03713768720626831 (0.029713544204240692) | > loss_mel_ce: 2.5621185302734375 (nan) | > loss: 0.010314509272575378 (nan) | > current_lr: 5e-06 | > step_time: 0.1145 (0.13661912143460553) | > loader_time: 0.0045 (0.004692630095422193)  --> TIME: 2025-02-27 13:49:51 -- STEP: 25452/36938 -- GLOBAL_STEP: 1724600 | > loss_text_ce: 0.03282014653086662 (0.029712370670045327) | > loss_mel_ce: 2.129223346710205 (nan) | > loss: 0.008579538203775883 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13662274689789203) | > loader_time: 0.0047 (0.00469262825136907)  --> TIME: 2025-02-27 13:50:00 -- STEP: 25502/36938 -- GLOBAL_STEP: 1724650 | > loss_text_ce: 0.032194919884204865 (0.029712902023868563) | > loss_mel_ce: 2.546401262283325 (nan) | > loss: 0.010232524946331978 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13661543421348693) | > loader_time: 0.0045 (0.0046924767368177085)  --> TIME: 2025-02-27 13:50:09 -- STEP: 25552/36938 -- GLOBAL_STEP: 1724700 | > loss_text_ce: 0.018405580893158913 (0.029716035821556727) | > loss_mel_ce: 1.114495038986206 (nan) | > loss: 0.004495637491345406 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13661524558261565) | > loader_time: 0.0045 (0.004692291468848433)  --> TIME: 2025-02-27 13:50:18 -- STEP: 25602/36938 -- GLOBAL_STEP: 1724750 | > loss_text_ce: 0.023113513365387917 (0.029713680110502197) | > loss_mel_ce: 2.745755910873413 (nan) | > loss: 0.010987577959895134 (nan) | > current_lr: 5e-06 | > step_time: 0.1633 (0.1366171260223155) | > loader_time: 0.0047 (0.004692361043976704)  --> TIME: 2025-02-27 13:50:28 -- STEP: 25652/36938 -- GLOBAL_STEP: 1724800 | > loss_text_ce: 0.028834836557507515 (0.029712936940429993) | > loss_mel_ce: 2.5343220233917236 (nan) | > loss: 0.010171257890760899 (nan) | > current_lr: 5e-06 | > step_time: 0.1872 (0.13661663290232462) | > loader_time: 0.0049 (0.0046923887184996205)  --> TIME: 2025-02-27 13:50:37 -- STEP: 25702/36938 -- GLOBAL_STEP: 1724850 | > loss_text_ce: 0.03233424201607704 (0.02971381811249275) | > loss_mel_ce: 3.1302030086517334 (nan) | > loss: 0.012549752369523048 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.1366221661902021) | > loader_time: 0.0047 (0.004692280972461274)  --> TIME: 2025-02-27 13:50:47 -- STEP: 25752/36938 -- GLOBAL_STEP: 1724900 | > loss_text_ce: 0.028550419956445694 (0.0297153228062803) | > loss_mel_ce: 2.523386240005493 (nan) | > loss: 0.010126733221113682 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.1366334519018941) | > loader_time: 0.0045 (0.004692301723519352)  --> TIME: 2025-02-27 13:50:56 -- STEP: 25802/36938 -- GLOBAL_STEP: 1724950 | > loss_text_ce: 0.027258310467004776 (0.02971659567678931) | > loss_mel_ce: 2.5367674827575684 (nan) | > loss: 0.0101747065782547 (nan) | > current_lr: 5e-06 | > step_time: 0.1623 (0.13663118925679507) | > loader_time: 0.0048 (0.004692424786071468)  --> TIME: 2025-02-27 13:51:06 -- STEP: 25852/36938 -- GLOBAL_STEP: 1725000 | > loss_text_ce: 0.03633902221918106 (0.029716314849520534) | > loss_mel_ce: 2.233776330947876 (nan) | > loss: 0.009008394554257393 (nan) | > current_lr: 5e-06 | > step_time: 0.1291 (0.13662839102778432) | > loader_time: 0.0047 (0.00469228760406329)  --> TIME: 2025-02-27 13:51:15 -- STEP: 25902/36938 -- GLOBAL_STEP: 1725050 | > loss_text_ce: 0.03758935257792473 (0.02971760083795912) | > loss_mel_ce: 2.1415460109710693 (nan) | > loss: 0.00864736270159483 (nan) | > current_lr: 5e-06 | > step_time: 0.1198 (0.13662279604871566) | > loader_time: 0.0047 (0.004692248530060585)  --> TIME: 2025-02-27 13:51:24 -- STEP: 25952/36938 -- GLOBAL_STEP: 1725100 | > loss_text_ce: 0.026250727474689484 (0.02971756931874641) | > loss_mel_ce: 2.446831226348877 (nan) | > loss: 0.009813818149268627 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13663005578055085) | > loader_time: 0.0047 (0.004692292233651161)  --> TIME: 2025-02-27 13:51:33 -- STEP: 26002/36938 -- GLOBAL_STEP: 1725150 | > loss_text_ce: 0.027019834145903587 (0.02971945791004192) | > loss_mel_ce: 2.388519287109375 (nan) | > loss: 0.009585472755134106 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13662457567904238) | > loader_time: 0.0046 (0.004692169384134463)  --> TIME: 2025-02-27 13:51:43 -- STEP: 26052/36938 -- GLOBAL_STEP: 1725200 | > loss_text_ce: 0.028622888028621674 (0.0297216604387683) | > loss_mel_ce: 2.494941473007202 (nan) | > loss: 0.010014144703745842 (nan) | > current_lr: 5e-06 | > step_time: 0.1822 (0.13662631150564622) | > loader_time: 0.0048 (0.004692339300468729)  --> TIME: 2025-02-27 13:51:52 -- STEP: 26102/36938 -- GLOBAL_STEP: 1725250 | > loss_text_ce: 0.03765337914228439 (0.02972258428997538) | > loss_mel_ce: 3.0464611053466797 (nan) | > loss: 0.012238550931215286 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.1366219883467116) | > loader_time: 0.0045 (0.004692189620926114)  --> TIME: 2025-02-27 13:52:02 -- STEP: 26152/36938 -- GLOBAL_STEP: 1725300 | > loss_text_ce: 0.029827648773789406 (0.029720581578445194) | > loss_mel_ce: 0.76837158203125 (nan) | > loss: 0.003167457412928343 (nan) | > current_lr: 5e-06 | > step_time: 0.1176 (0.1366275907857387) | > loader_time: 0.0045 (0.00469221940059554)  --> TIME: 2025-02-27 13:52:11 -- STEP: 26202/36938 -- GLOBAL_STEP: 1725350 | > loss_text_ce: 0.02444598637521267 (0.02972280242188237) | > loss_mel_ce: 0.005408605094999075 (nan) | > loss: 0.0001184706125059165 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.136628038409504) | > loader_time: 0.0045 (0.004692267820167442)  --> TIME: 2025-02-27 13:52:20 -- STEP: 26252/36938 -- GLOBAL_STEP: 1725400 | > loss_text_ce: 0.03115110844373703 (0.02972619693663011) | > loss_mel_ce: 2.8233954906463623 (nan) | > loss: 0.01132756657898426 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13662080065099527) | > loader_time: 0.0053 (0.004692300316329515)  --> TIME: 2025-02-27 13:52:29 -- STEP: 26302/36938 -- GLOBAL_STEP: 1725450 | > loss_text_ce: 0.026461731642484665 (0.029727024147242393) | > loss_mel_ce: 1.546842098236084 (nan) | > loss: 0.006243269424885511 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.136621381871902) | > loader_time: 0.0045 (0.004692248215413239)  --> TIME: 2025-02-27 13:52:39 -- STEP: 26352/36938 -- GLOBAL_STEP: 1725500 | > loss_text_ce: 0.024831252172589302 (0.029728124303611485) | > loss_mel_ce: 2.32769775390625 (nan) | > loss: 0.009335433132946491 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.1366156920854021) | > loader_time: 0.0047 (0.0046920931531070345)  --> TIME: 2025-02-27 13:52:47 -- STEP: 26402/36938 -- GLOBAL_STEP: 1725550 | > loss_text_ce: 0.024087097495794296 (0.029727989760456854) | > loss_mel_ce: 2.3266944885253906 (nan) | > loss: 0.009328499436378479 (nan) | > current_lr: 5e-06 | > step_time: 0.1093 (0.13660113761465095) | > loader_time: 0.0045 (0.004691857360057923)  --> TIME: 2025-02-27 13:52:57 -- STEP: 26452/36938 -- GLOBAL_STEP: 1725600 | > loss_text_ce: 0.0437595397233963 (0.029725861200761695) | > loss_mel_ce: 2.5262842178344727 (nan) | > loss: 0.01019858755171299 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.1365992292124133) | > loader_time: 0.0045 (0.004691686677839037)  --> TIME: 2025-02-27 13:53:06 -- STEP: 26502/36938 -- GLOBAL_STEP: 1725650 | > loss_text_ce: 0.030130919069051743 (0.02972510883060446) | > loss_mel_ce: 0.5867501497268677 (nan) | > loss: 0.0024479408748447895 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13660264711507572) | > loader_time: 0.0045 (0.004691608005555673)  --> TIME: 2025-02-27 13:53:16 -- STEP: 26552/36938 -- GLOBAL_STEP: 1725700 | > loss_text_ce: 0.029197916388511658 (0.029725768139336674) | > loss_mel_ce: 1.8697477579116821 (nan) | > loss: 0.007535499054938555 (nan) | > current_lr: 5e-06 | > step_time: 0.1393 (0.13660771316453504) | > loader_time: 0.0047 (0.004691660476039825)  --> TIME: 2025-02-27 13:53:25 -- STEP: 26602/36938 -- GLOBAL_STEP: 1725750 | > loss_text_ce: 0.02271415665745735 (0.02972470108687409) | > loss_mel_ce: 3.9174106121063232 (nan) | > loss: 0.0156354159116745 (nan) | > current_lr: 5e-06 | > step_time: 0.1098 (0.1366020485240099) | > loader_time: 0.0046 (0.0046915833676330525)  --> TIME: 2025-02-27 13:53:34 -- STEP: 26652/36938 -- GLOBAL_STEP: 1725800 | > loss_text_ce: 0.017493106424808502 (0.02972534437567827) | > loss_mel_ce: 2.2195639610290527 (nan) | > loss: 0.008877210319042206 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.13659684860758398) | > loader_time: 0.0045 (0.004691508498686392)  --> TIME: 2025-02-27 13:53:43 -- STEP: 26702/36938 -- GLOBAL_STEP: 1725850 | > loss_text_ce: 0.02625911869108677 (0.029727334619677828) | > loss_mel_ce: 2.6377196311950684 (nan) | > loss: 0.010571344755589962 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13658523338134623) | > loader_time: 0.0047 (0.004691362961357781)  --> TIME: 2025-02-27 13:53:52 -- STEP: 26752/36938 -- GLOBAL_STEP: 1725900 | > loss_text_ce: 0.032285965979099274 (0.02972738133526449) | > loss_mel_ce: 2.5955517292022705 (nan) | > loss: 0.0104279275983572 (nan) | > current_lr: 5e-06 | > step_time: 0.1836 (0.1365852304979393) | > loader_time: 0.0048 (0.004691244348099403)  --> TIME: 2025-02-27 13:54:02 -- STEP: 26802/36938 -- GLOBAL_STEP: 1725950 | > loss_text_ce: 0.03258774057030678 (0.029727677430972283) | > loss_mel_ce: 2.633887529373169 (nan) | > loss: 0.010581252165138721 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.1365925620255334) | > loader_time: 0.0049 (0.004691340782581741)  --> TIME: 2025-02-27 13:54:11 -- STEP: 26852/36938 -- GLOBAL_STEP: 1726000 | > loss_text_ce: 0.04485907405614853 (0.02972793065171184) | > loss_mel_ce: 2.0493552684783936 (nan) | > loss: 0.008310374803841114 (nan) | > current_lr: 5e-06 | > step_time: 0.1197 (0.1365934173431853) | > loader_time: 0.0045 (0.0046913599747758095)  --> TIME: 2025-02-27 13:54:21 -- STEP: 26902/36938 -- GLOBAL_STEP: 1726050 | > loss_text_ce: 0.03680398687720299 (0.029727141065615113) | > loss_mel_ce: 3.168823480606079 (nan) | > loss: 0.012720744125545025 (nan) | > current_lr: 5e-06 | > step_time: 0.1847 (0.13659537201893587) | > loader_time: 0.0049 (0.004691383030572203)  --> TIME: 2025-02-27 13:54:30 -- STEP: 26952/36938 -- GLOBAL_STEP: 1726100 | > loss_text_ce: 0.033135369420051575 (0.029729004269740858) | > loss_mel_ce: 2.818039655685425 (nan) | > loss: 0.011314187198877335 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13660796616933202) | > loader_time: 0.0045 (0.004691338328587733)  --> TIME: 2025-02-27 13:54:40 -- STEP: 27002/36938 -- GLOBAL_STEP: 1726150 | > loss_text_ce: 0.03231849521398544 (0.02972756257496206) | > loss_mel_ce: 2.8828511238098145 (nan) | > loss: 0.011568134650588036 (nan) | > current_lr: 5e-06 | > step_time: 0.1586 (0.13661117427800118) | > loader_time: 0.0048 (0.004691235966686364)  --> TIME: 2025-02-27 13:54:49 -- STEP: 27052/36938 -- GLOBAL_STEP: 1726200 | > loss_text_ce: 0.028674807399511337 (0.02972920100178757) | > loss_mel_ce: 2.4858083724975586 (nan) | > loss: 0.00997810810804367 (nan) | > current_lr: 5e-06 | > step_time: 0.1365 (0.13661518708393822) | > loader_time: 0.0046 (0.004691104643552567)  --> TIME: 2025-02-27 13:54:58 -- STEP: 27102/36938 -- GLOBAL_STEP: 1726250 | > loss_text_ce: 0.025945177301764488 (0.029728500841520972) | > loss_mel_ce: 2.934408187866211 (nan) | > loss: 0.011747434735298157 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.1366086276674913) | > loader_time: 0.0047 (0.004690955559817351)  --> TIME: 2025-02-27 13:55:08 -- STEP: 27152/36938 -- GLOBAL_STEP: 1726300 | > loss_text_ce: 0.031242767348885536 (0.029728769821285788) | > loss_mel_ce: 2.1769063472747803 (nan) | > loss: 0.008762497454881668 (nan) | > current_lr: 5e-06 | > step_time: 0.1585 (0.1366140905484515) | > loader_time: 0.0046 (0.004690843781936052)  --> TIME: 2025-02-27 13:55:17 -- STEP: 27202/36938 -- GLOBAL_STEP: 1726350 | > loss_text_ce: 0.023152770474553108 (0.029729580553758655) | > loss_mel_ce: 1.0076979398727417 (nan) | > loss: 0.004090677946805954 (nan) | > current_lr: 5e-06 | > step_time: 0.1191 (0.1366089790693585) | > loader_time: 0.0045 (0.004690687548245684)  --> TIME: 2025-02-27 13:55:26 -- STEP: 27252/36938 -- GLOBAL_STEP: 1726400 | > loss_text_ce: 0.029675334692001343 (0.029730758363554356) | > loss_mel_ce: 2.9727044105529785 (nan) | > loss: 0.011914205737411976 (nan) | > current_lr: 5e-06 | > step_time: 0.1438 (0.13660409574587254) | > loader_time: 0.0056 (0.004690711524156585)  --> TIME: 2025-02-27 13:55:36 -- STEP: 27302/36938 -- GLOBAL_STEP: 1726450 | > loss_text_ce: 0.0364999957382679 (0.029731239570727326) | > loss_mel_ce: 2.398289918899536 (nan) | > loss: 0.009661865420639515 (nan) | > current_lr: 5e-06 | > step_time: 0.1508 (0.1366107276958163) | > loader_time: 0.0047 (0.0046907522837135104)  --> TIME: 2025-02-27 13:55:45 -- STEP: 27352/36938 -- GLOBAL_STEP: 1726500 | > loss_text_ce: 0.025873469188809395 (0.029731697543314728) | > loss_mel_ce: 2.9454572200775146 (nan) | > loss: 0.011790995486080647 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.1366073378574526) | > loader_time: 0.0046 (0.004690622622662075)  --> TIME: 2025-02-27 13:55:54 -- STEP: 27402/36938 -- GLOBAL_STEP: 1726550 | > loss_text_ce: 0.03117864765226841 (0.02973108318433873) | > loss_mel_ce: 3.1699419021606445 (nan) | > loss: 0.012702859938144684 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13660747157806313) | > loader_time: 0.0046 (0.004690454986076157)  --> TIME: 2025-02-27 13:56:04 -- STEP: 27452/36938 -- GLOBAL_STEP: 1726600 | > loss_text_ce: 0.037256162613630295 (0.029728970370910438) | > loss_mel_ce: 1.8277994394302368 (nan) | > loss: 0.007401014678180218 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13661652678358327) | > loader_time: 0.0045 (0.004690369294472457)  --> TIME: 2025-02-27 13:56:13 -- STEP: 27502/36938 -- GLOBAL_STEP: 1726650 | > loss_text_ce: 0.02714485116302967 (0.02973071899695527) | > loss_mel_ce: 2.588771104812622 (nan) | > loss: 0.010380620136857033 (nan) | > current_lr: 5e-06 | > step_time: 0.1602 (0.13662152295199187) | > loader_time: 0.0046 (0.004690383843582948)  --> TIME: 2025-02-27 13:56:23 -- STEP: 27552/36938 -- GLOBAL_STEP: 1726700 | > loss_text_ce: 0.02826470322906971 (0.029728933233470554) | > loss_mel_ce: 2.8666090965270996 (nan) | > loss: 0.011487595736980438 (nan) | > current_lr: 5e-06 | > step_time: 0.1225 (0.13662197901223977) | > loader_time: 0.0045 (0.004690378108229553)  --> TIME: 2025-02-27 13:56:32 -- STEP: 27602/36938 -- GLOBAL_STEP: 1726750 | > loss_text_ce: 0.028179792687296867 (0.029729947718987966) | > loss_mel_ce: 2.77685546875 (nan) | > loss: 0.01113109290599823 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.1366252918172948) | > loader_time: 0.0045 (0.004690279563987879)  --> TIME: 2025-02-27 13:56:42 -- STEP: 27652/36938 -- GLOBAL_STEP: 1726800 | > loss_text_ce: 0.024108894169330597 (0.02972898388410257) | > loss_mel_ce: 2.8624002933502197 (nan) | > loss: 0.011454402469098568 (nan) | > current_lr: 5e-06 | > step_time: 0.1603 (0.13663023264488025) | > loader_time: 0.0047 (0.004690192239977431)  --> TIME: 2025-02-27 13:56:51 -- STEP: 27702/36938 -- GLOBAL_STEP: 1726850 | > loss_text_ce: 0.03288627043366432 (0.02972937093846084) | > loss_mel_ce: 2.8920843601226807 (nan) | > loss: 0.011607026681303978 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.1366304081705526) | > loader_time: 0.005 (0.004690107580780828)  --> TIME: 2025-02-27 13:57:00 -- STEP: 27752/36938 -- GLOBAL_STEP: 1726900 | > loss_text_ce: 0.03197561576962471 (0.029732300372781916) | > loss_mel_ce: 2.658254384994507 (nan) | > loss: 0.010675515979528427 (nan) | > current_lr: 5e-06 | > step_time: 0.186 (0.1366252882417417) | > loader_time: 0.0047 (0.004689980391708333)  --> TIME: 2025-02-27 13:57:09 -- STEP: 27802/36938 -- GLOBAL_STEP: 1726950 | > loss_text_ce: 0.03137284517288208 (0.029731781951052427) | > loss_mel_ce: 2.426738977432251 (nan) | > loss: 0.00975441187620163 (nan) | > current_lr: 5e-06 | > step_time: 0.1682 (0.13661074156418748) | > loader_time: 0.0048 (0.004689956584359946)  --> TIME: 2025-02-27 13:57:18 -- STEP: 27852/36938 -- GLOBAL_STEP: 1727000 | > loss_text_ce: 0.033432912081480026 (0.029733035236938928) | > loss_mel_ce: 2.603044033050537 (nan) | > loss: 0.010462210513651371 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.13660819101107888) | > loader_time: 0.0046 (0.004689841722082762)  --> TIME: 2025-02-27 13:57:28 -- STEP: 27902/36938 -- GLOBAL_STEP: 1727050 | > loss_text_ce: 0.034498732537031174 (0.029732619139809312) | > loss_mel_ce: 3.132568597793579 (nan) | > loss: 0.012567727826535702 (nan) | > current_lr: 5e-06 | > step_time: 0.1557 (0.13660635859144019) | > loader_time: 0.0046 (0.0046896705507130255)  --> TIME: 2025-02-27 13:57:37 -- STEP: 27952/36938 -- GLOBAL_STEP: 1727100 | > loss_text_ce: 0.02744026854634285 (0.029731456472956358) | > loss_mel_ce: 3.1392130851745605 (nan) | > loss: 0.012566085904836655 (nan) | > current_lr: 5e-06 | > step_time: 0.1949 (0.13660877775859606) | > loader_time: 0.0048 (0.0046895498811823765)  --> TIME: 2025-02-27 13:57:46 -- STEP: 28002/36938 -- GLOBAL_STEP: 1727150 | > loss_text_ce: 0.027783900499343872 (0.02973071016790581) | > loss_mel_ce: 3.4356496334075928 (nan) | > loss: 0.01374378427863121 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13660238399802865) | > loader_time: 0.0045 (0.004689614965731642)  --> TIME: 2025-02-27 13:57:56 -- STEP: 28052/36938 -- GLOBAL_STEP: 1727200 | > loss_text_ce: 0.03705630823969841 (0.029732851367074615) | > loss_mel_ce: 2.650573253631592 (nan) | > loss: 0.010665196925401688 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13661194710321767) | > loader_time: 0.0046 (0.0046895450640044726)  --> TIME: 2025-02-27 13:58:05 -- STEP: 28102/36938 -- GLOBAL_STEP: 1727250 | > loss_text_ce: 0.033548153936862946 (0.029731012685138905) | > loss_mel_ce: 2.587266206741333 (nan) | > loss: 0.010400057770311832 (nan) | > current_lr: 5e-06 | > step_time: 0.1349 (0.13660465694476884) | > loader_time: 0.0046 (0.004689375765934024)  --> TIME: 2025-02-27 13:58:14 -- STEP: 28152/36938 -- GLOBAL_STEP: 1727300 | > loss_text_ce: 0.028421353548765182 (0.029732252616841797) | > loss_mel_ce: 1.922867774963379 (nan) | > loss: 0.007743211463093758 (nan) | > current_lr: 5e-06 | > step_time: 0.1138 (0.1366093180656594) | > loader_time: 0.0045 (0.004689245831732699)  --> TIME: 2025-02-27 13:58:24 -- STEP: 28202/36938 -- GLOBAL_STEP: 1727350 | > loss_text_ce: 0.02740487828850746 (0.029732467375302593) | > loss_mel_ce: 2.4506490230560303 (nan) | > loss: 0.009833547286689281 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.136605922757005) | > loader_time: 0.0046 (0.0046891153775989215)  --> TIME: 2025-02-27 13:58:33 -- STEP: 28252/36938 -- GLOBAL_STEP: 1727400 | > loss_text_ce: 0.02622305601835251 (0.029731745382093926) | > loss_mel_ce: 2.845994710922241 (nan) | > loss: 0.011397689580917358 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.1366038192488182) | > loader_time: 0.0047 (0.0046889845581951665)  --> TIME: 2025-02-27 13:58:42 -- STEP: 28302/36938 -- GLOBAL_STEP: 1727450 | > loss_text_ce: 0.02792581543326378 (0.029731759276449294) | > loss_mel_ce: 2.5399422645568848 (nan) | > loss: 0.01018995326012373 (nan) | > current_lr: 5e-06 | > step_time: 0.1297 (0.1365962725681084) | > loader_time: 0.0046 (0.004688829712190146)  --> TIME: 2025-02-27 13:58:51 -- STEP: 28352/36938 -- GLOBAL_STEP: 1727500 | > loss_text_ce: 0.034509044140577316 (0.029728997065839417) | > loss_mel_ce: 2.272732734680176 (nan) | > loss: 0.00915572140365839 (nan) | > current_lr: 5e-06 | > step_time: 0.1641 (0.13659174579698372) | > loader_time: 0.0048 (0.0046886999673014995)  --> TIME: 2025-02-27 13:59:00 -- STEP: 28402/36938 -- GLOBAL_STEP: 1727550 | > loss_text_ce: 0.02800489403307438 (0.029726702906866836) | > loss_mel_ce: 2.6889843940734863 (nan) | > loss: 0.01078170444816351 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13659330137464062) | > loader_time: 0.0046 (0.00468851854982334)  --> TIME: 2025-02-27 13:59:09 -- STEP: 28452/36938 -- GLOBAL_STEP: 1727600 | > loss_text_ce: 0.03722544014453888 (0.0297254455739269) | > loss_mel_ce: 2.59661865234375 (nan) | > loss: 0.010451762937009335 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13658946358703633) | > loader_time: 0.0045 (0.004688872845887208)  --> TIME: 2025-02-27 13:59:19 -- STEP: 28502/36938 -- GLOBAL_STEP: 1727650 | > loss_text_ce: 0.030852966010570526 (0.029724416198479087) | > loss_mel_ce: 1.8182324171066284 (nan) | > loss: 0.007337640505284071 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.1365890798046107) | > loader_time: 0.0047 (0.00468877925930358)  --> TIME: 2025-02-27 13:59:28 -- STEP: 28552/36938 -- GLOBAL_STEP: 1727700 | > loss_text_ce: 0.0241671372205019 (0.029723916415814813) | > loss_mel_ce: 2.6499204635620117 (nan) | > loss: 0.0106114586815238 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.1365986105256095) | > loader_time: 0.0047 (0.004688938080212396)  --> TIME: 2025-02-27 13:59:38 -- STEP: 28602/36938 -- GLOBAL_STEP: 1727750 | > loss_text_ce: 0.03544628247618675 (0.029723577049564943) | > loss_mel_ce: 3.0367836952209473 (nan) | > loss: 0.012191388756036758 (nan) | > current_lr: 5e-06 | > step_time: 0.1316 (0.13659472327475697) | > loader_time: 0.0049 (0.004688983163286169)  --> TIME: 2025-02-27 13:59:47 -- STEP: 28652/36938 -- GLOBAL_STEP: 1727800 | > loss_text_ce: 0.030597299337387085 (0.029723553949896554) | > loss_mel_ce: 2.6556732654571533 (nan) | > loss: 0.01065980363637209 (nan) | > current_lr: 5e-06 | > step_time: 0.1528 (0.13660010568559655) | > loader_time: 0.0048 (0.004689129116514259)  --> TIME: 2025-02-27 13:59:57 -- STEP: 28702/36938 -- GLOBAL_STEP: 1727850 | > loss_text_ce: 0.035764966160058975 (0.029722951473367745) | > loss_mel_ce: 1.702054738998413 (nan) | > loss: 0.006896110251545906 (nan) | > current_lr: 5e-06 | > step_time: 0.1187 (0.13660290724524868) | > loader_time: 0.0045 (0.0046892508954738425)  --> TIME: 2025-02-27 14:00:06 -- STEP: 28752/36938 -- GLOBAL_STEP: 1727900 | > loss_text_ce: 0.02937908098101616 (0.02972358485633786) | > loss_mel_ce: 2.1997058391571045 (nan) | > loss: 0.008845576085150242 (nan) | > current_lr: 5e-06 | > step_time: 0.1284 (0.13660530861245573) | > loader_time: 0.0045 (0.004689099668262441)  --> TIME: 2025-02-27 14:00:16 -- STEP: 28802/36938 -- GLOBAL_STEP: 1727950 | > loss_text_ce: 0.031040014699101448 (0.029721422526672654) | > loss_mel_ce: 1.2475935220718384 (nan) | > loss: 0.005073943175375462 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13661015017166492) | > loader_time: 0.0054 (0.004689386789769474)  --> TIME: 2025-02-27 14:00:25 -- STEP: 28852/36938 -- GLOBAL_STEP: 1728000 | > loss_text_ce: 0.027202915400266647 (0.02972264574902899) | > loss_mel_ce: 2.4719314575195312 (nan) | > loss: 0.009917200542986393 (nan) | > current_lr: 5e-06 | > step_time: 0.1576 (0.13661415151695236) | > loader_time: 0.0047 (0.004689465419571533)  --> TIME: 2025-02-27 14:00:35 -- STEP: 28902/36938 -- GLOBAL_STEP: 1728050 | > loss_text_ce: 0.037485335022211075 (0.029722527909654513) | > loss_mel_ce: 2.9652180671691895 (nan) | > loss: 0.01191549003124237 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13662306262287063) | > loader_time: 0.0046 (0.00468950900690971)  --> TIME: 2025-02-27 14:00:44 -- STEP: 28952/36938 -- GLOBAL_STEP: 1728100 | > loss_text_ce: 0.0160303246229887 (0.029720796732801726) | > loss_mel_ce: 0.8855388760566711 (nan) | > loss: 0.0035776556469500065 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13662534100669038) | > loader_time: 0.0046 (0.004689810239138912)  --> TIME: 2025-02-27 14:00:53 -- STEP: 29002/36938 -- GLOBAL_STEP: 1728150 | > loss_text_ce: 0.030152175575494766 (0.029721195758837717) | > loss_mel_ce: 2.4235591888427734 (nan) | > loss: 0.009736950509250164 (nan) | > current_lr: 5e-06 | > step_time: 0.1814 (0.13661975856814415) | > loader_time: 0.0056 (0.004689906705750218)  --> TIME: 2025-02-27 14:01:03 -- STEP: 29052/36938 -- GLOBAL_STEP: 1728200 | > loss_text_ce: 0.025410301983356476 (0.029723688590809757) | > loss_mel_ce: 1.4688069820404053 (nan) | > loss: 0.005929434206336737 (nan) | > current_lr: 5e-06 | > step_time: 0.119 (0.13662083492321334) | > loader_time: 0.0048 (0.004690060754395518)  --> TIME: 2025-02-27 14:01:12 -- STEP: 29102/36938 -- GLOBAL_STEP: 1728250 | > loss_text_ce: 0.028288664296269417 (0.02972427543462511) | > loss_mel_ce: 2.9777846336364746 (nan) | > loss: 0.011928862892091274 (nan) | > current_lr: 5e-06 | > step_time: 0.1553 (0.13662193546704415) | > loader_time: 0.0047 (0.004690046589298065)  --> TIME: 2025-02-27 14:01:21 -- STEP: 29152/36938 -- GLOBAL_STEP: 1728300 | > loss_text_ce: 0.03720833361148834 (0.029722652460698138) | > loss_mel_ce: 2.9588863849639893 (nan) | > loss: 0.011889264918863773 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.13662105559910454) | > loader_time: 0.0045 (0.004690027868315929)  --> TIME: 2025-02-27 14:01:30 -- STEP: 29202/36938 -- GLOBAL_STEP: 1728350 | > loss_text_ce: 0.03783503174781799 (0.029721001078163615) | > loss_mel_ce: 2.43290638923645 (nan) | > loss: 0.009804530069231987 (nan) | > current_lr: 5e-06 | > step_time: 0.1595 (0.13661966010500817) | > loader_time: 0.0046 (0.004689853809098864)  --> TIME: 2025-02-27 14:01:40 -- STEP: 29252/36938 -- GLOBAL_STEP: 1728400 | > loss_text_ce: 0.03428841382265091 (0.029720370863733303) | > loss_mel_ce: 2.6297125816345215 (nan) | > loss: 0.01057143323123455 (nan) | > current_lr: 5e-06 | > step_time: 0.1406 (0.13662206556082346) | > loader_time: 0.0047 (0.004689795446350272)  --> TIME: 2025-02-27 14:01:49 -- STEP: 29302/36938 -- GLOBAL_STEP: 1728450 | > loss_text_ce: 0.023405248299241066 (0.029716164646712808) | > loss_mel_ce: 2.2443149089813232 (nan) | > loss: 0.008998890407383442 (nan) | > current_lr: 5e-06 | > step_time: 0.164 (0.13662480938582156) | > loader_time: 0.0046 (0.004689725810175918)  --> TIME: 2025-02-27 14:01:59 -- STEP: 29352/36938 -- GLOBAL_STEP: 1728500 | > loss_text_ce: 0.03604644164443016 (0.0297174970973481) | > loss_mel_ce: 3.033548355102539 (nan) | > loss: 0.01218093279749155 (nan) | > current_lr: 5e-06 | > step_time: 0.1843 (0.1366282669082617) | > loader_time: 0.0048 (0.004689721685561904)  --> TIME: 2025-02-27 14:02:08 -- STEP: 29402/36938 -- GLOBAL_STEP: 1728550 | > loss_text_ce: 0.02975684031844139 (0.02971954873177218) | > loss_mel_ce: 3.0342800617218018 (nan) | > loss: 0.012158877216279507 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.13662275606479637) | > loader_time: 0.0047 (0.004689604633884822)  --> TIME: 2025-02-27 14:02:17 -- STEP: 29452/36938 -- GLOBAL_STEP: 1728600 | > loss_text_ce: 0.03710399568080902 (0.029720124260057077) | > loss_mel_ce: 2.558926820755005 (nan) | > loss: 0.010301710106432438 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.13661504857132087) | > loader_time: 0.0046 (0.004689476784036913)  --> TIME: 2025-02-27 14:02:26 -- STEP: 29502/36938 -- GLOBAL_STEP: 1728650 | > loss_text_ce: 0.029449306428432465 (0.029719393125853096) | > loss_mel_ce: 3.377505302429199 (nan) | > loss: 0.013519661501049995 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.13661459313999935) | > loader_time: 0.0048 (0.00468945620415716)  --> TIME: 2025-02-27 14:02:36 -- STEP: 29552/36938 -- GLOBAL_STEP: 1728700 | > loss_text_ce: 0.032706718891859055 (0.029718495550323132) | > loss_mel_ce: 1.286044955253601 (nan) | > loss: 0.0052331420592963696 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.1366143657038805) | > loader_time: 0.0046 (0.004689493079926469)  --> TIME: 2025-02-27 14:02:45 -- STEP: 29602/36938 -- GLOBAL_STEP: 1728750 | > loss_text_ce: 0.030171265825629234 (0.02971584203574824) | > loss_mel_ce: 2.438833475112915 (nan) | > loss: 0.009797638282179832 (nan) | > current_lr: 5e-06 | > step_time: 0.1729 (0.13662174215800962) | > loader_time: 0.0048 (0.00468951903052547)  --> TIME: 2025-02-27 14:02:54 -- STEP: 29652/36938 -- GLOBAL_STEP: 1728800 | > loss_text_ce: 0.027226481586694717 (0.029717384783796583) | > loss_mel_ce: 2.823148488998413 (nan) | > loss: 0.01131101232022047 (nan) | > current_lr: 5e-06 | > step_time: 0.1309 (0.1366141508919496) | > loader_time: 0.0047 (0.004689401667175683)  --> TIME: 2025-02-27 14:03:04 -- STEP: 29702/36938 -- GLOBAL_STEP: 1728850 | > loss_text_ce: 0.02501191571354866 (0.029716247489833782) | > loss_mel_ce: 2.3372278213500977 (nan) | > loss: 0.009373968467116356 (nan) | > current_lr: 5e-06 | > step_time: 0.133 (0.13661436709191793) | > loader_time: 0.0046 (0.004689402238631255)  --> TIME: 2025-02-27 14:03:13 -- STEP: 29752/36938 -- GLOBAL_STEP: 1728900 | > loss_text_ce: 0.043283309787511826 (0.029718472848341573) | > loss_mel_ce: 0.1654459685087204 (nan) | > loss: 0.0008282908238470554 (nan) | > current_lr: 5e-06 | > step_time: 0.1081 (0.13660775045292362) | > loader_time: 0.0045 (0.0046893212865135965)  --> TIME: 2025-02-27 14:03:22 -- STEP: 29802/36938 -- GLOBAL_STEP: 1728950 | > loss_text_ce: 0.033110540360212326 (0.029717679265377786) | > loss_mel_ce: 0.0008995731477625668 (nan) | > loss: 0.0001349607773590833 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13660907999764366) | > loader_time: 0.0053 (0.004689481400633913)  --> TIME: 2025-02-27 14:03:32 -- STEP: 29852/36938 -- GLOBAL_STEP: 1729000 | > loss_text_ce: 0.03486044332385063 (0.029718813160565394) | > loss_mel_ce: 2.579864263534546 (nan) | > loss: 0.01037589181214571 (nan) | > current_lr: 5e-06 | > step_time: 0.1845 (0.13661031590124276) | > loader_time: 0.0049 (0.004689486731509213)  --> TIME: 2025-02-27 14:03:41 -- STEP: 29902/36938 -- GLOBAL_STEP: 1729050 | > loss_text_ce: 0.04346238449215889 (0.029717717218721018) | > loss_mel_ce: 2.3453128337860107 (nan) | > loss: 0.009479266591370106 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.1366149551708522) | > loader_time: 0.0046 (0.004689551485748157)  --> TIME: 2025-02-27 14:03:51 -- STEP: 29952/36938 -- GLOBAL_STEP: 1729100 | > loss_text_ce: 0.03888468071818352 (0.02971860740313771) | > loss_mel_ce: 2.971275806427002 (nan) | > loss: 0.011945081874728203 (nan) | > current_lr: 5e-06 | > step_time: 0.1281 (0.1366159507574971) | > loader_time: 0.0048 (0.004689559603157756)  --> TIME: 2025-02-27 14:04:00 -- STEP: 30002/36938 -- GLOBAL_STEP: 1729150 | > loss_text_ce: 0.028715861961245537 (0.029719866733972228) | > loss_mel_ce: 2.965064764022827 (nan) | > loss: 0.011880082078278065 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13660897205578867) | > loader_time: 0.0046 (0.004689478030261076)  --> TIME: 2025-02-27 14:04:09 -- STEP: 30052/36938 -- GLOBAL_STEP: 1729200 | > loss_text_ce: 0.031074348837137222 (0.029719544889573457) | > loss_mel_ce: 2.781818389892578 (nan) | > loss: 0.011162273585796356 (nan) | > current_lr: 5e-06 | > step_time: 0.1769 (0.13661401534102988) | > loader_time: 0.0048 (0.004689529932847983)  --> TIME: 2025-02-27 14:04:19 -- STEP: 30102/36938 -- GLOBAL_STEP: 1729250 | > loss_text_ce: 0.031892649829387665 (0.02971943834515375) | > loss_mel_ce: 2.5391602516174316 (nan) | > loss: 0.010202591307461262 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.1366151447003433) | > loader_time: 0.0046 (0.004689445139823083)  --> TIME: 2025-02-27 14:04:28 -- STEP: 30152/36938 -- GLOBAL_STEP: 1729300 | > loss_text_ce: 0.03643276169896126 (0.02971773859661153) | > loss_mel_ce: 2.6442530155181885 (nan) | > loss: 0.010637642815709114 (nan) | > current_lr: 5e-06 | > step_time: 0.1388 (0.1366131053216986) | > loader_time: 0.0046 (0.004689459911105084)  --> TIME: 2025-02-27 14:04:37 -- STEP: 30202/36938 -- GLOBAL_STEP: 1729350 | > loss_text_ce: 0.027427010238170624 (0.02971951687124084) | > loss_mel_ce: 2.2905259132385254 (nan) | > loss: 0.009198226034641266 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13661552505709415) | > loader_time: 0.0046 (0.004689598934483579)  --> TIME: 2025-02-27 14:04:46 -- STEP: 30252/36938 -- GLOBAL_STEP: 1729400 | > loss_text_ce: 0.024277353659272194 (0.02971867764059936) | > loss_mel_ce: 1.6598249673843384 (nan) | > loss: 0.006682945881038904 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13660880376718212) | > loader_time: 0.0046 (0.004689560370928874)  --> TIME: 2025-02-27 14:04:56 -- STEP: 30302/36938 -- GLOBAL_STEP: 1729450 | > loss_text_ce: 0.018211932852864265 (0.029719870379881226) | > loss_mel_ce: 1.837143898010254 (nan) | > loss: 0.007362523581832647 (nan) | > current_lr: 5e-06 | > step_time: 0.1105 (0.13661788296883903) | > loader_time: 0.0045 (0.004689570771814667)  --> TIME: 2025-02-27 14:05:05 -- STEP: 30352/36938 -- GLOBAL_STEP: 1729500 | > loss_text_ce: 0.029116768389940262 (0.029718070714238636) | > loss_mel_ce: 2.7937774658203125 (nan) | > loss: 0.011201961897313595 (nan) | > current_lr: 5e-06 | > step_time: 0.1862 (0.13661735316735807) | > loader_time: 0.0048 (0.004689776809453124)  --> TIME: 2025-02-27 14:05:15 -- STEP: 30402/36938 -- GLOBAL_STEP: 1729550 | > loss_text_ce: 0.028170350939035416 (0.029720589769865073) | > loss_mel_ce: 2.826137065887451 (nan) | > loss: 0.011326617561280727 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.1366171074434323) | > loader_time: 0.0048 (0.00468983805326249)  --> TIME: 2025-02-27 14:05:24 -- STEP: 30452/36938 -- GLOBAL_STEP: 1729600 | > loss_text_ce: 0.02846505306661129 (0.02972009318683046) | > loss_mel_ce: 2.3309543132781982 (nan) | > loss: 0.009362775832414627 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13661683623555815) | > loader_time: 0.0045 (0.004689864693905644)  --> TIME: 2025-02-27 14:05:33 -- STEP: 30502/36938 -- GLOBAL_STEP: 1729650 | > loss_text_ce: 0.02579357475042343 (0.029720694563654396) | > loss_mel_ce: 2.448011636734009 (nan) | > loss: 0.009816687554121017 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13661767144210019) | > loader_time: 0.0046 (0.004689766261532466)  --> TIME: 2025-02-27 14:05:43 -- STEP: 30552/36938 -- GLOBAL_STEP: 1729700 | > loss_text_ce: 0.021632501855492592 (0.029718783596892936) | > loss_mel_ce: 2.1410930156707764 (nan) | > loss: 0.008582244627177715 (nan) | > current_lr: 5e-06 | > step_time: 0.156 (0.13661429152560142) | > loader_time: 0.0047 (0.004689698601368406)  --> TIME: 2025-02-27 14:05:52 -- STEP: 30602/36938 -- GLOBAL_STEP: 1729750 | > loss_text_ce: 0.03771281987428665 (0.0297189843446176) | > loss_mel_ce: 1.9101618528366089 (nan) | > loss: 0.007729661650955677 (nan) | > current_lr: 5e-06 | > step_time: 0.1419 (0.1366202124473569) | > loader_time: 0.0046 (0.0046898943950488175)  --> TIME: 2025-02-27 14:06:02 -- STEP: 30652/36938 -- GLOBAL_STEP: 1729800 | > loss_text_ce: 0.031116072088479996 (0.02971660015709833) | > loss_mel_ce: 2.447551727294922 (nan) | > loss: 0.009835983626544476 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13662281628145634) | > loader_time: 0.0046 (0.004689972347462168)  --> TIME: 2025-02-27 14:06:11 -- STEP: 30702/36938 -- GLOBAL_STEP: 1729850 | > loss_text_ce: 0.03350178897380829 (0.02971673727268004) | > loss_mel_ce: 2.5728237628936768 (nan) | > loss: 0.010342562571167946 (nan) | > current_lr: 5e-06 | > step_time: 0.1546 (0.13661454916078286) | > loader_time: 0.0046 (0.004690122607476385)  --> TIME: 2025-02-27 14:06:20 -- STEP: 30752/36938 -- GLOBAL_STEP: 1729900 | > loss_text_ce: 0.027328090742230415 (0.029715160127987444) | > loss_mel_ce: 2.9036104679107666 (nan) | > loss: 0.011630709283053875 (nan) | > current_lr: 5e-06 | > step_time: 0.1212 (0.13661734835011421) | > loader_time: 0.0045 (0.004690209820354929)  --> TIME: 2025-02-27 14:06:29 -- STEP: 30802/36938 -- GLOBAL_STEP: 1729950 | > loss_text_ce: 0.024491863325238228 (0.02971233370729672) | > loss_mel_ce: 2.4026827812194824 (nan) | > loss: 0.009631645865738392 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13661616804940446) | > loader_time: 0.0046 (0.004690257142669318)  --> TIME: 2025-02-27 14:06:39 -- STEP: 30852/36938 -- GLOBAL_STEP: 1730000 | > loss_text_ce: 0.028667012229561806 (0.029712647818218255) | > loss_mel_ce: 3.0866892337799072 (nan) | > loss: 0.012362525798380375 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.13662860583783262) | > loader_time: 0.0048 (0.004690211145049146) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1730000.pth  --> TIME: 2025-02-27 14:06:54 -- STEP: 30902/36938 -- GLOBAL_STEP: 1730050 | > loss_text_ce: 0.039320409297943115 (0.029712047967482332) | > loss_mel_ce: 2.039388656616211 (nan) | > loss: 0.008248846046626568 (nan) | > current_lr: 5e-06 | > step_time: 0.1322 (0.13662276007382385) | > loader_time: 0.0046 (0.004690318599533328)  --> TIME: 2025-02-27 14:07:04 -- STEP: 30952/36938 -- GLOBAL_STEP: 1730100 | > loss_text_ce: 0.028501443564891815 (0.02971196400447447) | > loss_mel_ce: 3.4644198417663574 (nan) | > loss: 0.013860799372196198 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.136625318556187) | > loader_time: 0.0046 (0.004690299980952877)  --> TIME: 2025-02-27 14:07:13 -- STEP: 31002/36938 -- GLOBAL_STEP: 1730150 | > loss_text_ce: 0.033571962267160416 (0.02971232435423596) | > loss_mel_ce: 3.0770933628082275 (nan) | > loss: 0.01234391052275896 (nan) | > current_lr: 5e-06 | > step_time: 0.1378 (0.1366254718955393) | > loader_time: 0.0047 (0.004690238140712368)  --> TIME: 2025-02-27 14:07:23 -- STEP: 31052/36938 -- GLOBAL_STEP: 1730200 | > loss_text_ce: 0.034807756543159485 (0.029712801791500078) | > loss_mel_ce: 2.6546688079833984 (nan) | > loss: 0.010672526434063911 (nan) | > current_lr: 5e-06 | > step_time: 0.1801 (0.13663550228951826) | > loader_time: 0.0049 (0.00469035406971387)  --> TIME: 2025-02-27 14:07:32 -- STEP: 31102/36938 -- GLOBAL_STEP: 1730250 | > loss_text_ce: 0.03514394536614418 (0.029712150081397815) | > loss_mel_ce: 2.556696891784668 (nan) | > loss: 0.010285083204507828 (nan) | > current_lr: 5e-06 | > step_time: 0.1246 (0.13663969818139043) | > loader_time: 0.0046 (0.004690531779467067)  --> TIME: 2025-02-27 14:07:42 -- STEP: 31152/36938 -- GLOBAL_STEP: 1730300 | > loss_text_ce: 0.026234719902276993 (0.029712157065564695) | > loss_mel_ce: 2.461604356765747 (nan) | > loss: 0.0098723778501153 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13663278978545923) | > loader_time: 0.0046 (0.004690662095285933)  --> TIME: 2025-02-27 14:07:51 -- STEP: 31202/36938 -- GLOBAL_STEP: 1730350 | > loss_text_ce: 0.02258390188217163 (0.029712801717125502) | > loss_mel_ce: 3.0581910610198975 (nan) | > loss: 0.012225298210978508 (nan) | > current_lr: 5e-06 | > step_time: 0.1655 (0.13663342659095137) | > loader_time: 0.0055 (0.004690811707571442)  --> TIME: 2025-02-27 14:08:00 -- STEP: 31252/36938 -- GLOBAL_STEP: 1730400 | > loss_text_ce: 0.03384900093078613 (0.029712904255300075) | > loss_mel_ce: 1.6269612312316895 (nan) | > loss: 0.006590517237782478 (nan) | > current_lr: 5e-06 | > step_time: 0.1576 (0.13663220832034734) | > loader_time: 0.0049 (0.004691009811077696)  --> TIME: 2025-02-27 14:08:10 -- STEP: 31302/36938 -- GLOBAL_STEP: 1730450 | > loss_text_ce: 0.04052522033452988 (0.029710814212472993) | > loss_mel_ce: 2.7296361923217773 (nan) | > loss: 0.010992704890668392 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13663502622584764) | > loader_time: 0.0047 (0.0046910524413986835)  --> TIME: 2025-02-27 14:08:19 -- STEP: 31352/36938 -- GLOBAL_STEP: 1730500 | > loss_text_ce: 0.027840998023748398 (0.02970918119150354) | > loss_mel_ce: 2.722055196762085 (nan) | > loss: 0.010912287048995495 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.1366289656634988) | > loader_time: 0.0044 (0.004691025217019324)  --> TIME: 2025-02-27 14:08:28 -- STEP: 31402/36938 -- GLOBAL_STEP: 1730550 | > loss_text_ce: 0.026607219129800797 (0.02970940974865408) | > loss_mel_ce: 2.8682596683502197 (nan) | > loss: 0.0114875677973032 (nan) | > current_lr: 5e-06 | > step_time: 0.1238 (0.13663021169432774) | > loader_time: 0.0047 (0.004691037658855198)  --> TIME: 2025-02-27 14:08:37 -- STEP: 31452/36938 -- GLOBAL_STEP: 1730600 | > loss_text_ce: 0.0399334654211998 (0.029710951990924196) | > loss_mel_ce: 2.028022527694702 (nan) | > loss: 0.008206174708902836 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13662786688738782) | > loader_time: 0.0047 (0.004691079495806139)  --> TIME: 2025-02-27 14:08:47 -- STEP: 31502/36938 -- GLOBAL_STEP: 1730650 | > loss_text_ce: 0.027387330308556557 (0.02971075844061277) | > loss_mel_ce: 2.379704713821411 (nan) | > loss: 0.00955195352435112 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.13662376759370556) | > loader_time: 0.0047 (0.004691061576384112)  --> TIME: 2025-02-27 14:08:56 -- STEP: 31552/36938 -- GLOBAL_STEP: 1730700 | > loss_text_ce: 0.019612668082118034 (0.029710435801451083) | > loss_mel_ce: 2.1845812797546387 (nan) | > loss: 0.008746801875531673 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13662539702097992) | > loader_time: 0.0045 (0.004691064833808902)  --> TIME: 2025-02-27 14:09:05 -- STEP: 31602/36938 -- GLOBAL_STEP: 1730750 | > loss_text_ce: 0.03129850700497627 (0.029708674350990012) | > loss_mel_ce: 2.953491687774658 (nan) | > loss: 0.01184440590441227 (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.13662733757664378) | > loader_time: 0.0046 (0.00469111472803934)  --> TIME: 2025-02-27 14:09:15 -- STEP: 31652/36938 -- GLOBAL_STEP: 1730800 | > loss_text_ce: 0.03186508268117905 (0.029709234125191893) | > loss_mel_ce: 2.7255311012268066 (nan) | > loss: 0.010942049324512482 (nan) | > current_lr: 5e-06 | > step_time: 0.1389 (0.13662269086674117) | > loader_time: 0.0047 (0.004691057382665851)  --> TIME: 2025-02-27 14:09:24 -- STEP: 31702/36938 -- GLOBAL_STEP: 1730850 | > loss_text_ce: 0.030908357352018356 (0.029708524819967132) | > loss_mel_ce: 2.9101619720458984 (nan) | > loss: 0.011670914478600025 (nan) | > current_lr: 5e-06 | > step_time: 0.1855 (0.13662231603967678) | > loader_time: 0.0049 (0.004691030902295833)  --> TIME: 2025-02-27 14:09:33 -- STEP: 31752/36938 -- GLOBAL_STEP: 1730900 | > loss_text_ce: 0.02604106441140175 (0.02970879830415441) | > loss_mel_ce: 2.598276138305664 (nan) | > loss: 0.010413957759737968 (nan) | > current_lr: 5e-06 | > step_time: 0.1705 (0.13662467044317336) | > loader_time: 0.005 (0.004691145955600233)  --> TIME: 2025-02-27 14:09:43 -- STEP: 31802/36938 -- GLOBAL_STEP: 1730950 | > loss_text_ce: 0.030627992004156113 (0.02970893629990894) | > loss_mel_ce: 2.7689120769500732 (nan) | > loss: 0.011109286919236183 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13661883454016893) | > loader_time: 0.0046 (0.004691115445846635)  --> TIME: 2025-02-27 14:09:52 -- STEP: 31852/36938 -- GLOBAL_STEP: 1731000 | > loss_text_ce: 0.03004567325115204 (0.029708585762261496) | > loss_mel_ce: 1.97528076171875 (nan) | > loss: 0.007957645691931248 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13661824093981437) | > loader_time: 0.0046 (0.0046910558396004096)  --> TIME: 2025-02-27 14:10:01 -- STEP: 31902/36938 -- GLOBAL_STEP: 1731050 | > loss_text_ce: 0.02857143245637417 (0.029708904204540393) | > loss_mel_ce: 2.3678598403930664 (nan) | > loss: 0.009509648196399212 (nan) | > current_lr: 5e-06 | > step_time: 0.1852 (0.13661732197435886) | > loader_time: 0.0048 (0.004691178077784843)  --> TIME: 2025-02-27 14:10:11 -- STEP: 31952/36938 -- GLOBAL_STEP: 1731100 | > loss_text_ce: 0.02391468733549118 (0.029710108818306546) | > loss_mel_ce: 2.7963626384735107 (nan) | > loss: 0.011191576719284058 (nan) | > current_lr: 5e-06 | > step_time: 0.1239 (0.13662396406435182) | > loader_time: 0.0046 (0.004691236941111262)  --> TIME: 2025-02-27 14:10:20 -- STEP: 32002/36938 -- GLOBAL_STEP: 1731150 | > loss_text_ce: 0.030064871534705162 (0.029711285606579586) | > loss_mel_ce: 2.7041144371032715 (nan) | > loss: 0.010849918238818645 (nan) | > current_lr: 5e-06 | > step_time: 0.1313 (0.13662755745633573) | > loader_time: 0.0046 (0.004691229664633501)  --> TIME: 2025-02-27 14:10:30 -- STEP: 32052/36938 -- GLOBAL_STEP: 1731200 | > loss_text_ce: 0.024018818512558937 (0.029710420961115768) | > loss_mel_ce: 2.5850095748901367 (nan) | > loss: 0.010353287681937218 (nan) | > current_lr: 5e-06 | > step_time: 0.1402 (0.1366237790772337) | > loader_time: 0.0047 (0.004691184861344946)  --> TIME: 2025-02-27 14:10:39 -- STEP: 32102/36938 -- GLOBAL_STEP: 1731250 | > loss_text_ce: 0.02609739638864994 (0.029709410975415598) | > loss_mel_ce: 3.262629270553589 (nan) | > loss: 0.013050503097474575 (nan) | > current_lr: 5e-06 | > step_time: 0.135 (0.13662180489271855) | > loader_time: 0.0046 (0.004691252722694758)  --> TIME: 2025-02-27 14:10:48 -- STEP: 32152/36938 -- GLOBAL_STEP: 1731300 | > loss_text_ce: 0.023933308199048042 (0.029709412281024427) | > loss_mel_ce: 2.0820298194885254 (nan) | > loss: 0.00835699774324894 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.13661629509558293) | > loader_time: 0.0048 (0.004691218345074526)  --> TIME: 2025-02-27 14:10:57 -- STEP: 32202/36938 -- GLOBAL_STEP: 1731350 | > loss_text_ce: 0.028645364567637444 (0.029706472472250865) | > loss_mel_ce: 2.692678451538086 (nan) | > loss: 0.010798904113471508 (nan) | > current_lr: 5e-06 | > step_time: 0.1618 (0.13661812943750876) | > loader_time: 0.0056 (0.004691367126845379)  --> TIME: 2025-02-27 14:11:07 -- STEP: 32252/36938 -- GLOBAL_STEP: 1731400 | > loss_text_ce: 0.021380869671702385 (0.029707539771131046) | > loss_mel_ce: 0.1909942626953125 (nan) | > loss: 0.0008427585125900805 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.1366146393365186) | > loader_time: 0.0045 (0.004691275801594206)  --> TIME: 2025-02-27 14:11:16 -- STEP: 32302/36938 -- GLOBAL_STEP: 1731450 | > loss_text_ce: 0.028811518102884293 (0.0297071370300505) | > loss_mel_ce: 2.201904296875 (nan) | > loss: 0.00885204691439867 (nan) | > current_lr: 5e-06 | > step_time: 0.11 (0.13660962135384158) | > loader_time: 0.0045 (0.004691180500273594)  --> TIME: 2025-02-27 14:11:25 -- STEP: 32352/36938 -- GLOBAL_STEP: 1731500 | > loss_text_ce: 0.01910950057208538 (0.02970552432096796) | > loss_mel_ce: 2.8304154872894287 (nan) | > loss: 0.011307639069855213 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.1366003297350434) | > loader_time: 0.0059 (0.004691233185997857)  --> TIME: 2025-02-27 14:11:34 -- STEP: 32402/36938 -- GLOBAL_STEP: 1731550 | > loss_text_ce: 0.028165368363261223 (0.029705515337957213) | > loss_mel_ce: 0.02166748046875 (nan) | > loss: 0.00019774942484218627 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.13659287950785481) | > loader_time: 0.0046 (0.004691146485680901)  --> TIME: 2025-02-27 14:11:43 -- STEP: 32452/36938 -- GLOBAL_STEP: 1731600 | > loss_text_ce: 0.020066367462277412 (0.029703483737769373) | > loss_mel_ce: 1.4219682216644287 (nan) | > loss: 0.005722359754145145 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1365880408637369) | > loader_time: 0.0047 (0.00469104582911048)  --> TIME: 2025-02-27 14:11:52 -- STEP: 32502/36938 -- GLOBAL_STEP: 1731650 | > loss_text_ce: 0.02320963516831398 (0.029701850078072738) | > loss_mel_ce: 2.404254913330078 (nan) | > loss: 0.009632796049118042 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.136591322509616) | > loader_time: 0.0046 (0.0046910505046566684)  --> TIME: 2025-02-27 14:12:02 -- STEP: 32552/36938 -- GLOBAL_STEP: 1731700 | > loss_text_ce: 0.02619059756398201 (0.029699757111175947) | > loss_mel_ce: 3.7745535373687744 (nan) | > loss: 0.01508231833577156 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13658727113314315) | > loader_time: 0.0046 (0.004691064335784893)  --> TIME: 2025-02-27 14:12:11 -- STEP: 32602/36938 -- GLOBAL_STEP: 1731750 | > loss_text_ce: 0.042245104908943176 (0.029700294277290745) | > loss_mel_ce: 1.6956068277359009 (nan) | > loss: 0.006896238308399916 (nan) | > current_lr: 5e-06 | > step_time: 0.1532 (0.1365884271884061) | > loader_time: 0.0049 (0.0046911295934159965)  --> TIME: 2025-02-27 14:12:21 -- STEP: 32652/36938 -- GLOBAL_STEP: 1731800 | > loss_text_ce: 0.030549287796020508 (0.029697461022040357) | > loss_mel_ce: 2.784458637237549 (nan) | > loss: 0.011170666664838791 (nan) | > current_lr: 5e-06 | > step_time: 0.1096 (0.13659463621859674) | > loader_time: 0.0047 (0.004691191204736539)  --> TIME: 2025-02-27 14:12:30 -- STEP: 32702/36938 -- GLOBAL_STEP: 1731850 | > loss_text_ce: 0.025028608739376068 (0.02969687098318713) | > loss_mel_ce: 1.6363818645477295 (nan) | > loss: 0.006592899095267057 (nan) | > current_lr: 5e-06 | > step_time: 0.1089 (0.13659299590100704) | > loader_time: 0.0044 (0.004691165285762002)  --> TIME: 2025-02-27 14:12:39 -- STEP: 32752/36938 -- GLOBAL_STEP: 1731900 | > loss_text_ce: 0.033568643033504486 (0.029697201241772987) | > loss_mel_ce: 2.5564448833465576 (nan) | > loss: 0.010277831926941872 (nan) | > current_lr: 5e-06 | > step_time: 0.1898 (0.13659203949356885) | > loader_time: 0.0049 (0.0046911870466539215)  --> TIME: 2025-02-27 14:12:49 -- STEP: 32802/36938 -- GLOBAL_STEP: 1731950 | > loss_text_ce: 0.037041593343019485 (0.029699243859332678) | > loss_mel_ce: 2.709244728088379 (nan) | > loss: 0.010897962376475334 (nan) | > current_lr: 5e-06 | > step_time: 0.1384 (0.13659751228454642) | > loader_time: 0.0053 (0.0046914404946474045)  --> TIME: 2025-02-27 14:12:58 -- STEP: 32852/36938 -- GLOBAL_STEP: 1732000 | > loss_text_ce: 0.0315706841647625 (0.029699654600833732) | > loss_mel_ce: 1.7415653467178345 (nan) | > loss: 0.007036254275590181 (nan) | > current_lr: 5e-06 | > step_time: 0.1174 (0.1365997507164916) | > loader_time: 0.0046 (0.0046915952549424645)  --> TIME: 2025-02-27 14:13:08 -- STEP: 32902/36938 -- GLOBAL_STEP: 1732050 | > loss_text_ce: 0.037654295563697815 (0.029700232108913503) | > loss_mel_ce: 2.159693956375122 (nan) | > loss: 0.00871963705867529 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.13659888425864025) | > loader_time: 0.0047 (0.004691560386807084)  --> TIME: 2025-02-27 14:13:17 -- STEP: 32952/36938 -- GLOBAL_STEP: 1732100 | > loss_text_ce: 0.02624465897679329 (0.029699036922239345) | > loss_mel_ce: 2.585232734680176 (nan) | > loss: 0.01036300603300333 (nan) | > current_lr: 5e-06 | > step_time: 0.1866 (0.13660430117381853) | > loader_time: 0.0048 (0.004691592088223813)  --> TIME: 2025-02-27 14:13:27 -- STEP: 33002/36938 -- GLOBAL_STEP: 1732150 | > loss_text_ce: 0.027123242616653442 (0.02970028689169407) | > loss_mel_ce: 2.7468581199645996 (nan) | > loss: 0.011007863096892834 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13660471964284365) | > loader_time: 0.0045 (0.004691542773440125)  --> TIME: 2025-02-27 14:13:36 -- STEP: 33052/36938 -- GLOBAL_STEP: 1732200 | > loss_text_ce: 0.03192146122455597 (0.02970150007257013) | > loss_mel_ce: 2.9786388874053955 (nan) | > loss: 0.011946668848395348 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.13660251509980503) | > loader_time: 0.005 (0.004691700532565383)  --> TIME: 2025-02-27 14:13:46 -- STEP: 33102/36938 -- GLOBAL_STEP: 1732250 | > loss_text_ce: 0.03620398789644241 (0.029700385926096768) | > loss_mel_ce: 2.331563711166382 (nan) | > loss: 0.009395904839038849 (nan) | > current_lr: 5e-06 | > step_time: 0.1422 (0.13661447969219895) | > loader_time: 0.0048 (0.0046917612578140176)  --> TIME: 2025-02-27 14:13:55 -- STEP: 33152/36938 -- GLOBAL_STEP: 1732300 | > loss_text_ce: 0.028798332437872887 (0.029699088165483264) | > loss_mel_ce: 2.9785656929016113 (nan) | > loss: 0.01193398516625166 (nan) | > current_lr: 5e-06 | > step_time: 0.1431 (0.13660733298215413) | > loader_time: 0.0046 (0.004691735183295171)  --> TIME: 2025-02-27 14:14:04 -- STEP: 33202/36938 -- GLOBAL_STEP: 1732350 | > loss_text_ce: 0.03579457104206085 (0.02970086450576733) | > loss_mel_ce: 2.99625563621521 (nan) | > loss: 0.012031945399940014 (nan) | > current_lr: 5e-06 | > step_time: 0.1294 (0.1366039269818285) | > loader_time: 0.0047 (0.004691725164700377)  --> TIME: 2025-02-27 14:14:13 -- STEP: 33252/36938 -- GLOBAL_STEP: 1732400 | > loss_text_ce: 0.04228445142507553 (0.029700460232803032) | > loss_mel_ce: 2.942336320877075 (nan) | > loss: 0.011843734420835972 (nan) | > current_lr: 5e-06 | > step_time: 0.1199 (0.13660996613857804) | > loader_time: 0.0046 (0.0046917300039030435)  --> TIME: 2025-02-27 14:14:23 -- STEP: 33302/36938 -- GLOBAL_STEP: 1732450 | > loss_text_ce: 0.030920598655939102 (0.029700338423791744) | > loss_mel_ce: 2.3692123889923096 (nan) | > loss: 0.009524337016046047 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.13661053153346348) | > loader_time: 0.0047 (0.0046916929753820355)  --> TIME: 2025-02-27 14:14:32 -- STEP: 33352/36938 -- GLOBAL_STEP: 1732500 | > loss_text_ce: 0.02370368130505085 (0.029702952195630898) | > loss_mel_ce: 2.6328771114349365 (nan) | > loss: 0.010541987605392933 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.1366161501687871) | > loader_time: 0.0046 (0.004691721831731451)  --> TIME: 2025-02-27 14:14:42 -- STEP: 33402/36938 -- GLOBAL_STEP: 1732550 | > loss_text_ce: 0.03217725455760956 (0.029703194051520367) | > loss_mel_ce: 2.6276190280914307 (nan) | > loss: 0.01055474765598774 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13661932112978) | > loader_time: 0.0046 (0.004691793457361571)  --> TIME: 2025-02-27 14:14:51 -- STEP: 33452/36938 -- GLOBAL_STEP: 1732600 | > loss_text_ce: 0.03716782480478287 (0.029703100601268663) | > loss_mel_ce: 2.796414613723755 (nan) | > loss: 0.011244375258684158 (nan) | > current_lr: 5e-06 | > step_time: 0.1696 (0.1366235813669218) | > loader_time: 0.0053 (0.0046918048864914895)  --> TIME: 2025-02-27 14:15:01 -- STEP: 33502/36938 -- GLOBAL_STEP: 1732650 | > loss_text_ce: 0.0347854308784008 (0.02970302376549101) | > loss_mel_ce: 3.505415916442871 (nan) | > loss: 0.014048418961465359 (nan) | > current_lr: 5e-06 | > step_time: 0.1885 (0.13662566382040778) | > loader_time: 0.0053 (0.004691899858242146)  --> TIME: 2025-02-27 14:15:10 -- STEP: 33552/36938 -- GLOBAL_STEP: 1732700 | > loss_text_ce: 0.03302447870373726 (0.029703749007525218) | > loss_mel_ce: 1.4455788135528564 (nan) | > loss: 0.005867473315447569 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13662991168587918) | > loader_time: 0.0048 (0.004692025749127451)  --> TIME: 2025-02-27 14:15:19 -- STEP: 33602/36938 -- GLOBAL_STEP: 1732750 | > loss_text_ce: 0.032686710357666016 (0.029705376644003464) | > loss_mel_ce: 3.1366419792175293 (nan) | > loss: 0.012576702050864697 (nan) | > current_lr: 5e-06 | > step_time: 0.1382 (0.13662140234319706) | > loader_time: 0.0046 (0.004691981934385746)  --> TIME: 2025-02-27 14:15:29 -- STEP: 33652/36938 -- GLOBAL_STEP: 1732800 | > loss_text_ce: 0.023975927382707596 (0.029703396089685207) | > loss_mel_ce: 2.237764358520508 (nan) | > loss: 0.00897516030818224 (nan) | > current_lr: 5e-06 | > step_time: 0.132 (0.1366276048131831) | > loader_time: 0.0047 (0.004692042772287916)  --> TIME: 2025-02-27 14:15:38 -- STEP: 33702/36938 -- GLOBAL_STEP: 1732850 | > loss_text_ce: 0.03381621092557907 (0.029701538642670656) | > loss_mel_ce: 2.442256450653076 (nan) | > loss: 0.009825685061514378 (nan) | > current_lr: 5e-06 | > step_time: 0.14 (0.13662315016295076) | > loader_time: 0.0046 (0.004692219045210204)  --> TIME: 2025-02-27 14:15:47 -- STEP: 33752/36938 -- GLOBAL_STEP: 1732900 | > loss_text_ce: 0.013291752897202969 (0.029702515720719143) | > loss_mel_ce: 0.7243716716766357 (nan) | > loss: 0.0029272360261529684 (nan) | > current_lr: 5e-06 | > step_time: 0.1094 (0.13661654437286427) | > loader_time: 0.0045 (0.004692298974943414)  --> TIME: 2025-02-27 14:15:56 -- STEP: 33802/36938 -- GLOBAL_STEP: 1732950 | > loss_text_ce: 0.037546124309301376 (0.029700411869570405) | > loss_mel_ce: 3.14798641204834 (nan) | > loss: 0.012641003355383873 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.1366093727930258) | > loader_time: 0.0046 (0.00469250841526148)  --> TIME: 2025-02-27 14:16:05 -- STEP: 33852/36938 -- GLOBAL_STEP: 1733000 | > loss_text_ce: 0.042351145297288895 (0.029701125119197303) | > loss_mel_ce: 2.6352083683013916 (nan) | > loss: 0.010625237599015236 (nan) | > current_lr: 5e-06 | > step_time: 0.1136 (0.1366123534694141) | > loader_time: 0.0046 (0.004692569109166114)  --> TIME: 2025-02-27 14:16:15 -- STEP: 33902/36938 -- GLOBAL_STEP: 1733050 | > loss_text_ce: 0.026438968256115913 (0.029700028539719488) | > loss_mel_ce: 3.007664203643799 (nan) | > loss: 0.01204009260982275 (nan) | > current_lr: 5e-06 | > step_time: 0.1647 (0.13662139137025928) | > loader_time: 0.0046 (0.004692670328625152)  --> TIME: 2025-02-27 14:16:24 -- STEP: 33952/36938 -- GLOBAL_STEP: 1733100 | > loss_text_ce: 0.026061298325657845 (0.029699386676028876) | > loss_mel_ce: 2.283935546875 (nan) | > loss: 0.009166655130684376 (nan) | > current_lr: 5e-06 | > step_time: 0.1078 (0.13661356992198212) | > loader_time: 0.0045 (0.004692681274191636)  --> TIME: 2025-02-27 14:16:34 -- STEP: 34002/36938 -- GLOBAL_STEP: 1733150 | > loss_text_ce: 0.0249224491417408 (0.02969811192421317) | > loss_mel_ce: 2.5590906143188477 (nan) | > loss: 0.010254020802676678 (nan) | > current_lr: 5e-06 | > step_time: 0.1561 (0.13661757883384354) | > loader_time: 0.0049 (0.004692663831448887)  --> TIME: 2025-02-27 14:16:43 -- STEP: 34052/36938 -- GLOBAL_STEP: 1733200 | > loss_text_ce: 0.03633582592010498 (0.029699077614364626) | > loss_mel_ce: 3.3547616004943848 (nan) | > loss: 0.0134567366912961 (nan) | > current_lr: 5e-06 | > step_time: 0.1946 (0.13662109360436894) | > loader_time: 0.0056 (0.0046926809578318645)  --> TIME: 2025-02-27 14:16:53 -- STEP: 34102/36938 -- GLOBAL_STEP: 1733250 | > loss_text_ce: 0.03999844938516617 (0.029701080049594757) | > loss_mel_ce: 2.831429958343506 (nan) | > loss: 0.011394558474421501 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.1366226066767573) | > loader_time: 0.0047 (0.004692735563491552)  --> TIME: 2025-02-27 14:17:02 -- STEP: 34152/36938 -- GLOBAL_STEP: 1733300 | > loss_text_ce: 0.03485042229294777 (0.029701623147568218) | > loss_mel_ce: 3.0809104442596436 (nan) | > loss: 0.012364130467176437 (nan) | > current_lr: 5e-06 | > step_time: 0.1905 (0.13662741532731432) | > loader_time: 0.0049 (0.00469288276017595)  --> TIME: 2025-02-27 14:17:11 -- STEP: 34202/36938 -- GLOBAL_STEP: 1733350 | > loss_text_ce: 0.0319301001727581 (0.02970054810487129) | > loss_mel_ce: 2.6011602878570557 (nan) | > loss: 0.010448772460222244 (nan) | > current_lr: 5e-06 | > step_time: 0.1209 (0.13662208867947426) | > loader_time: 0.0047 (0.004692850841513102)  --> TIME: 2025-02-27 14:17:21 -- STEP: 34252/36938 -- GLOBAL_STEP: 1733400 | > loss_text_ce: 0.03209511563181877 (0.02969925114816455) | > loss_mel_ce: 3.4407594203948975 (nan) | > loss: 0.013781170360744 (nan) | > current_lr: 5e-06 | > step_time: 0.1658 (0.13662585438265576) | > loader_time: 0.0047 (0.004692849309091133)  --> TIME: 2025-02-27 14:17:30 -- STEP: 34302/36938 -- GLOBAL_STEP: 1733450 | > loss_text_ce: 0.027883537113666534 (0.029698793123196635) | > loss_mel_ce: 2.666074752807617 (nan) | > loss: 0.010690310969948769 (nan) | > current_lr: 5e-06 | > step_time: 0.1236 (0.1366249946774752) | > loader_time: 0.0046 (0.0046928575814455245)  --> TIME: 2025-02-27 14:17:40 -- STEP: 34352/36938 -- GLOBAL_STEP: 1733500 | > loss_text_ce: 0.03263278678059578 (0.029698955269110115) | > loss_mel_ce: 2.707181930541992 (nan) | > loss: 0.010872281156480312 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.13663131765532158) | > loader_time: 0.0047 (0.0046929432296953155)  --> TIME: 2025-02-27 14:17:49 -- STEP: 34402/36938 -- GLOBAL_STEP: 1733550 | > loss_text_ce: 0.02773039974272251 (0.029698679092142095) | > loss_mel_ce: 0.6653925776481628 (nan) | > loss: 0.002750488230958581 (nan) | > current_lr: 5e-06 | > step_time: 0.1334 (0.13662997990609477) | > loader_time: 0.0048 (0.004693002868797861)  --> TIME: 2025-02-27 14:17:58 -- STEP: 34452/36938 -- GLOBAL_STEP: 1733600 | > loss_text_ce: 0.02818138338625431 (0.02969803875399798) | > loss_mel_ce: 1.203850507736206 (nan) | > loss: 0.004889016039669514 (nan) | > current_lr: 5e-06 | > step_time: 0.1148 (0.13662719792997696) | > loader_time: 0.0048 (0.004693106410257889)  --> TIME: 2025-02-27 14:18:08 -- STEP: 34502/36938 -- GLOBAL_STEP: 1733650 | > loss_text_ce: 0.0273085106164217 (0.02969465654566443) | > loss_mel_ce: 2.6855716705322266 (nan) | > loss: 0.010765397921204567 (nan) | > current_lr: 5e-06 | > step_time: 0.155 (0.13662653711939549) | > loader_time: 0.0047 (0.004693023288196796)  --> TIME: 2025-02-27 14:18:17 -- STEP: 34552/36938 -- GLOBAL_STEP: 1733700 | > loss_text_ce: 0.02935391291975975 (0.02969533314870842) | > loss_mel_ce: 2.976949453353882 (nan) | > loss: 0.011929775588214397 (nan) | > current_lr: 5e-06 | > step_time: 0.1329 (0.1366239392900961) | > loader_time: 0.0046 (0.004693086299390622)  --> TIME: 2025-02-27 14:18:26 -- STEP: 34602/36938 -- GLOBAL_STEP: 1733750 | > loss_text_ce: 0.02378574013710022 (0.029694415354751046) | > loss_mel_ce: 1.7717896699905396 (nan) | > loss: 0.007125299423933029 (nan) | > current_lr: 5e-06 | > step_time: 0.1152 (0.13662472748837914) | > loader_time: 0.0045 (0.004693086991647227)  --> TIME: 2025-02-27 14:18:36 -- STEP: 34652/36938 -- GLOBAL_STEP: 1733800 | > loss_text_ce: 0.032733362168073654 (0.029695092487022876) | > loss_mel_ce: 2.642446756362915 (nan) | > loss: 0.01061579491943121 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.136632255108133) | > loader_time: 0.0046 (0.004693091947735458)  --> TIME: 2025-02-27 14:18:45 -- STEP: 34702/36938 -- GLOBAL_STEP: 1733850 | > loss_text_ce: 0.03371617943048477 (0.02969473222174318) | > loss_mel_ce: 0.2099270224571228 (nan) | > loss: 0.0009668381535448134 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13662353701622196) | > loader_time: 0.0046 (0.0046930738941241565)  --> TIME: 2025-02-27 14:18:54 -- STEP: 34752/36938 -- GLOBAL_STEP: 1733900 | > loss_text_ce: 0.030271751806139946 (0.029696059092200997) | > loss_mel_ce: 2.0814435482025146 (nan) | > loss: 0.00837982352823019 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13662114214584511) | > loader_time: 0.0046 (0.0046931444006950065)  --> TIME: 2025-02-27 14:19:04 -- STEP: 34802/36938 -- GLOBAL_STEP: 1733950 | > loss_text_ce: 0.0306033194065094 (0.02969511562933637) | > loss_mel_ce: 2.2048137187957764 (nan) | > loss: 0.008870703168213367 (nan) | > current_lr: 5e-06 | > step_time: 0.1241 (0.13662002718412522) | > loader_time: 0.0052 (0.004693203640767885)  --> TIME: 2025-02-27 14:19:12 -- STEP: 34852/36938 -- GLOBAL_STEP: 1734000 | > loss_text_ce: 0.0327296145260334 (0.029695509503883423) | > loss_mel_ce: 2.7515039443969727 (nan) | > loss: 0.011048546992242336 (nan) | > current_lr: 5e-06 | > step_time: 0.1302 (0.1366090891857357) | > loader_time: 0.0046 (0.004693125154317297)  --> TIME: 2025-02-27 14:19:22 -- STEP: 34902/36938 -- GLOBAL_STEP: 1734050 | > loss_text_ce: 0.03241754695773125 (0.029695067082139908) | > loss_mel_ce: 2.445067882537842 (nan) | > loss: 0.009831291623413563 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13661179322034234) | > loader_time: 0.0047 (0.004693054488911493)  --> TIME: 2025-02-27 14:19:31 -- STEP: 34952/36938 -- GLOBAL_STEP: 1734100 | > loss_text_ce: 0.03503705561161041 (0.02969366716437662) | > loss_mel_ce: 2.9594228267669678 (nan) | > loss: 0.011882778257131577 (nan) | > current_lr: 5e-06 | > step_time: 0.1913 (0.13660825013732586) | > loader_time: 0.0049 (0.004693172655489795)  --> TIME: 2025-02-27 14:19:41 -- STEP: 35002/36938 -- GLOBAL_STEP: 1734150 | > loss_text_ce: 0.033042337745428085 (0.029695137710559125) | > loss_mel_ce: 2.997758150100708 (nan) | > loss: 0.012026987038552761 (nan) | > current_lr: 5e-06 | > step_time: 0.1172 (0.1366120631487334) | > loader_time: 0.0047 (0.004693141154415752)  --> TIME: 2025-02-27 14:19:50 -- STEP: 35052/36938 -- GLOBAL_STEP: 1734200 | > loss_text_ce: 0.025694765150547028 (0.02969392921500746) | > loss_mel_ce: 2.718456506729126 (nan) | > loss: 0.010889490135014057 (nan) | > current_lr: 5e-06 | > step_time: 0.1401 (0.13662224545485005) | > loader_time: 0.0048 (0.004693289937921903)  --> TIME: 2025-02-27 14:20:00 -- STEP: 35102/36938 -- GLOBAL_STEP: 1734250 | > loss_text_ce: 0.031763263046741486 (0.02969232566665723) | > loss_mel_ce: 2.916057586669922 (nan) | > loss: 0.011697703041136265 (nan) | > current_lr: 5e-06 | > step_time: 0.1876 (0.13663161628219742) | > loader_time: 0.0049 (0.004693302603693342)  --> TIME: 2025-02-27 14:20:10 -- STEP: 35152/36938 -- GLOBAL_STEP: 1734300 | > loss_text_ce: 0.03445737063884735 (0.029692345868186146) | > loss_mel_ce: 2.5014843940734863 (nan) | > loss: 0.010063261725008488 (nan) | > current_lr: 5e-06 | > step_time: 0.1646 (0.13663527567548514) | > loader_time: 0.0047 (0.004693319350413239)  --> TIME: 2025-02-27 14:20:19 -- STEP: 35202/36938 -- GLOBAL_STEP: 1734350 | > loss_text_ce: 0.024635009467601776 (0.02969194367814966) | > loss_mel_ce: 2.1321747303009033 (nan) | > loss: 0.008558769710361958 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13663123471582275) | > loader_time: 0.0045 (0.004693259617718982)  --> TIME: 2025-02-27 14:20:28 -- STEP: 35252/36938 -- GLOBAL_STEP: 1734400 | > loss_text_ce: 0.026249762624502182 (0.029691205341660618) | > loss_mel_ce: 2.4196038246154785 (nan) | > loss: 0.009705767966806889 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13662318535834922) | > loader_time: 0.0045 (0.004693166921239706)  --> TIME: 2025-02-27 14:20:37 -- STEP: 35302/36938 -- GLOBAL_STEP: 1734450 | > loss_text_ce: 0.012743921019136906 (0.02969042200754067) | > loss_mel_ce: 1.286325216293335 (nan) | > loss: 0.0051550366915762424 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13662136396627964) | > loader_time: 0.0045 (0.004693096477339361)  --> TIME: 2025-02-27 14:20:47 -- STEP: 35352/36938 -- GLOBAL_STEP: 1734500 | > loss_text_ce: 0.023594900965690613 (0.029691795415677503) | > loss_mel_ce: 2.58555006980896 (nan) | > loss: 0.010353750549256802 (nan) | > current_lr: 5e-06 | > step_time: 0.1318 (0.13662190974419822) | > loader_time: 0.0048 (0.004693042135368172)  --> TIME: 2025-02-27 14:20:56 -- STEP: 35402/36938 -- GLOBAL_STEP: 1734550 | > loss_text_ce: 0.028006579726934433 (0.029691205539014952) | > loss_mel_ce: 2.8908371925354004 (nan) | > loss: 0.011582713574171066 (nan) | > current_lr: 5e-06 | > step_time: 0.1698 (0.1366200322155568) | > loader_time: 0.0054 (0.00469309528307301)  --> TIME: 2025-02-27 14:21:05 -- STEP: 35452/36938 -- GLOBAL_STEP: 1734600 | > loss_text_ce: 0.030945908278226852 (0.029691496134838555) | > loss_mel_ce: 2.4963572025299072 (nan) | > loss: 0.010028981603682041 (nan) | > current_lr: 5e-06 | > step_time: 0.1765 (0.1366225829394013) | > loader_time: 0.0049 (0.004693153721476824)  --> TIME: 2025-02-27 14:21:15 -- STEP: 35502/36938 -- GLOBAL_STEP: 1734650 | > loss_text_ce: 0.03867693617939949 (0.02969139709253491) | > loss_mel_ce: 1.9686743021011353 (nan) | > loss: 0.00796567928045988 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13662253824477782) | > loader_time: 0.0046 (0.004693196435141519)  --> TIME: 2025-02-27 14:21:24 -- STEP: 35552/36938 -- GLOBAL_STEP: 1734700 | > loss_text_ce: 0.027903547510504723 (0.029691169720534382) | > loss_mel_ce: 2.8766262531280518 (nan) | > loss: 0.011525912210345268 (nan) | > current_lr: 5e-06 | > step_time: 0.1795 (0.13662336365689662) | > loader_time: 0.0049 (0.004693124788661598)  --> TIME: 2025-02-27 14:21:33 -- STEP: 35602/36938 -- GLOBAL_STEP: 1734750 | > loss_text_ce: 0.031851090490818024 (0.02968839262356592) | > loss_mel_ce: 2.786553382873535 (nan) | > loss: 0.0111841456964612 (nan) | > current_lr: 5e-06 | > step_time: 0.1539 (0.1366226164372252) | > loader_time: 0.0047 (0.004693102310242415)  --> TIME: 2025-02-27 14:21:43 -- STEP: 35652/36938 -- GLOBAL_STEP: 1734800 | > loss_text_ce: 0.03502297401428223 (0.029689786283162066) | > loss_mel_ce: 3.3775997161865234 (nan) | > loss: 0.013542153872549534 (nan) | > current_lr: 5e-06 | > step_time: 0.1825 (0.13662539884108854) | > loader_time: 0.0048 (0.004693006507531605)  --> TIME: 2025-02-27 14:21:52 -- STEP: 35702/36938 -- GLOBAL_STEP: 1734850 | > loss_text_ce: 0.029737405478954315 (0.02968744892180621) | > loss_mel_ce: 2.8395750522613525 (nan) | > loss: 0.011386161670088768 (nan) | > current_lr: 5e-06 | > step_time: 0.1381 (0.13662647452383206) | > loader_time: 0.0046 (0.004693015290471326)  --> TIME: 2025-02-27 14:22:02 -- STEP: 35752/36938 -- GLOBAL_STEP: 1734900 | > loss_text_ce: 0.029024595394730568 (0.029686586811572175) | > loss_mel_ce: 2.54048228263855 (nan) | > loss: 0.01019645668566227 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.136628547906344) | > loader_time: 0.0046 (0.004693066801739994)  --> TIME: 2025-02-27 14:22:11 -- STEP: 35802/36938 -- GLOBAL_STEP: 1734950 | > loss_text_ce: 0.022652219980955124 (0.029686987819855085) | > loss_mel_ce: 2.1730120182037354 (nan) | > loss: 0.008712953887879848 (nan) | > current_lr: 5e-06 | > step_time: 0.1445 (0.13662599680190982) | > loader_time: 0.0049 (0.004692962033663022)  --> TIME: 2025-02-27 14:22:20 -- STEP: 35852/36938 -- GLOBAL_STEP: 1735000 | > loss_text_ce: 0.029557527974247932 (0.029684814415452558) | > loss_mel_ce: 3.036261558532715 (nan) | > loss: 0.012165949679911137 (nan) | > current_lr: 5e-06 | > step_time: 0.1243 (0.13662625241596338) | > loader_time: 0.0046 (0.004692934559057409)  --> TIME: 2025-02-27 14:22:29 -- STEP: 35902/36938 -- GLOBAL_STEP: 1735050 | > loss_text_ce: 0.03191715106368065 (0.02968473546775056) | > loss_mel_ce: 0.011800129897892475 (nan) | > loss: 0.00017348128312733024 (nan) | > current_lr: 5e-06 | > step_time: 0.1227 (0.1366240272654277) | > loader_time: 0.0045 (0.004692981159592112)  --> TIME: 2025-02-27 14:22:39 -- STEP: 35952/36938 -- GLOBAL_STEP: 1735100 | > loss_text_ce: 0.03282430022954941 (0.02968470211804284) | > loss_mel_ce: 2.8697054386138916 (nan) | > loss: 0.011517975479364395 (nan) | > current_lr: 5e-06 | > step_time: 0.1319 (0.13662673378447174) | > loader_time: 0.0045 (0.004692971527337744)  --> TIME: 2025-02-27 14:22:48 -- STEP: 36002/36938 -- GLOBAL_STEP: 1735150 | > loss_text_ce: 0.023128453642129898 (0.029683087552324888) | > loss_mel_ce: 2.914573907852173 (nan) | > loss: 0.011657549999654293 (nan) | > current_lr: 5e-06 | > step_time: 0.1995 (0.13662905939671469) | > loader_time: 0.0051 (0.004692971782547674)  --> TIME: 2025-02-27 14:22:57 -- STEP: 36052/36938 -- GLOBAL_STEP: 1735200 | > loss_text_ce: 0.03310127556324005 (0.029686171557940517) | > loss_mel_ce: 2.4017109870910645 (nan) | > loss: 0.009661953896284103 (nan) | > current_lr: 5e-06 | > step_time: 0.1377 (0.13662281546782837) | > loader_time: 0.0048 (0.00469298884776834)  --> TIME: 2025-02-27 14:23:07 -- STEP: 36102/36938 -- GLOBAL_STEP: 1735250 | > loss_text_ce: 0.031187383458018303 (0.029685722596462148) | > loss_mel_ce: 1.285986304283142 (nan) | > loss: 0.005226880311965942 (nan) | > current_lr: 5e-06 | > step_time: 0.1195 (0.13662578371329337) | > loader_time: 0.0045 (0.004693056148780354)  --> TIME: 2025-02-27 14:23:16 -- STEP: 36152/36938 -- GLOBAL_STEP: 1735300 | > loss_text_ce: 0.031017126515507698 (0.02968491139096995) | > loss_mel_ce: 1.816391944885254 (nan) | > loss: 0.007330988999456167 (nan) | > current_lr: 5e-06 | > step_time: 0.1788 (0.13663499796910403) | > loader_time: 0.0048 (0.004693145198245922)  --> TIME: 2025-02-27 14:23:26 -- STEP: 36202/36938 -- GLOBAL_STEP: 1735350 | > loss_text_ce: 0.02882501855492592 (0.029685521313689022) | > loss_mel_ce: 1.055504322052002 (nan) | > loss: 0.004302894696593285 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13663183961821798) | > loader_time: 0.0046 (0.004693116056913204)  --> TIME: 2025-02-27 14:23:35 -- STEP: 36252/36938 -- GLOBAL_STEP: 1735400 | > loss_text_ce: 0.0309788528829813 (0.0296842601269426) | > loss_mel_ce: 2.3541440963745117 (nan) | > loss: 0.009464774280786514 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.1366353275022764) | > loader_time: 0.0051 (0.004693090968293577)  --> TIME: 2025-02-27 14:23:45 -- STEP: 36302/36938 -- GLOBAL_STEP: 1735450 | > loss_text_ce: 0.02369680628180504 (0.029684740615591322) | > loss_mel_ce: 2.019109010696411 (nan) | > loss: 0.008106373250484467 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.13664364550082464) | > loader_time: 0.0046 (0.004693087635142246)  --> TIME: 2025-02-27 14:23:54 -- STEP: 36352/36938 -- GLOBAL_STEP: 1735500 | > loss_text_ce: 0.02868242934346199 (0.029684439259449108) | > loss_mel_ce: 2.9448282718658447 (nan) | > loss: 0.01179964654147625 (nan) | > current_lr: 5e-06 | > step_time: 0.1311 (0.13664628462341447) | > loader_time: 0.0046 (0.004693012166453534)  --> TIME: 2025-02-27 14:24:04 -- STEP: 36402/36938 -- GLOBAL_STEP: 1735550 | > loss_text_ce: 0.021953152492642403 (0.029684598782870372) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1084 (0.13665045652342017) | > loader_time: 0.0045 (0.004692936099484058)  --> TIME: 2025-02-27 14:24:13 -- STEP: 36452/36938 -- GLOBAL_STEP: 1735600 | > loss_text_ce: 0.030830981209874153 (0.029683452257277473) | > loss_mel_ce: 2.838239908218384 (nan) | > loss: 0.011385203339159489 (nan) | > current_lr: 5e-06 | > step_time: 0.1412 (0.13665011250775405) | > loader_time: 0.0055 (0.004692927138624643)  --> TIME: 2025-02-27 14:24:23 -- STEP: 36502/36938 -- GLOBAL_STEP: 1735650 | > loss_text_ce: 0.03592683747410774 (0.029683403008411394) | > loss_mel_ce: 2.85903000831604 (nan) | > loss: 0.011487924493849277 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.13665568389002364) | > loader_time: 0.0047 (0.004692971918669299)  --> TIME: 2025-02-27 14:24:32 -- STEP: 36552/36938 -- GLOBAL_STEP: 1735700 | > loss_text_ce: 0.03228975832462311 (0.02968389018596019) | > loss_mel_ce: 2.7008678913116455 (nan) | > loss: 0.010845864191651344 (nan) | > current_lr: 5e-06 | > step_time: 0.1578 (0.13666445790456122) | > loader_time: 0.0047 (0.004693042830164392)  --> TIME: 2025-02-27 14:24:42 -- STEP: 36602/36938 -- GLOBAL_STEP: 1735750 | > loss_text_ce: 0.027663519605994225 (0.029682337130490632) | > loss_mel_ce: 2.333085536956787 (nan) | > loss: 0.009368051774799824 (nan) | > current_lr: 5e-06 | > step_time: 0.1287 (0.136662805668613) | > loader_time: 0.0047 (0.004693101959849706)  --> TIME: 2025-02-27 14:24:51 -- STEP: 36652/36938 -- GLOBAL_STEP: 1735800 | > loss_text_ce: 0.024790186434984207 (0.029681783926086912) | > loss_mel_ce: 2.4230644702911377 (nan) | > loss: 0.00971371028572321 (nan) | > current_lr: 5e-06 | > step_time: 0.1829 (0.1366624876256743) | > loader_time: 0.0048 (0.004693156940687543)  --> TIME: 2025-02-27 14:25:00 -- STEP: 36702/36938 -- GLOBAL_STEP: 1735850 | > loss_text_ce: 0.039180781692266464 (0.029681053580736083) | > loss_mel_ce: 2.4146406650543213 (nan) | > loss: 0.009737387299537659 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13666395576934154) | > loader_time: 0.0048 (0.0046930890415651984)  --> TIME: 2025-02-27 14:25:10 -- STEP: 36752/36938 -- GLOBAL_STEP: 1735900 | > loss_text_ce: 0.031032025814056396 (0.02968169790613061) | > loss_mel_ce: 2.7692909240722656 (nan) | > loss: 0.0111123938113451 (nan) | > current_lr: 5e-06 | > step_time: 0.1861 (0.136669113369264) | > loader_time: 0.0048 (0.004693138259465069)  --> TIME: 2025-02-27 14:25:19 -- STEP: 36802/36938 -- GLOBAL_STEP: 1735950 | > loss_text_ce: 0.028537606820464134 (0.029680360150597163) | > loss_mel_ce: 3.8508927822113037 (nan) | > loss: 0.015394565649330616 (nan) | > current_lr: 5e-06 | > step_time: 0.1446 (0.13666707390346486) | > loader_time: 0.0047 (0.004693195687824698)  --> TIME: 2025-02-27 14:25:29 -- STEP: 36852/36938 -- GLOBAL_STEP: 1736000 | > loss_text_ce: 0.026349937543272972 (0.02968082012833875) | > loss_mel_ce: 2.8175265789031982 (nan) | > loss: 0.011285224929451942 (nan) | > current_lr: 5e-06 | > step_time: 0.161 (0.13666691207399417) | > loader_time: 0.0047 (0.00469317617885303)  --> TIME: 2025-02-27 14:25:38 -- STEP: 36902/36938 -- GLOBAL_STEP: 1736050 | > loss_text_ce: 0.030506998300552368 (0.02967992707491159) | > loss_mel_ce: 0.23658891022205353 (nan) | > loss: 0.0010599044617265463 (nan) | > current_lr: 5e-06 | > step_time: 0.1188 (0.13667364373361568) | > loader_time: 0.0045 (0.004693260471204883)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.020960460795034275 (+0.0004155694848240049) | > avg_loss_text_ce: 0.0289668436899855 (-0.00011091514739512351) | > avg_loss_mel_ce: 2.4165446958919548 (-0.0005297793610261614) | > avg_loss: 2.4455115370231106 (-0.0006406918020518404) > BEST MODEL : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/best_model_1736086.pth  > EPOCH: 48/1000 --> xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3  > TRAINING (2025-02-27 14:26:11)   --> TIME: 2025-02-27 14:26:14 -- STEP: 14/36938 -- GLOBAL_STEP: 1736100 | > loss_text_ce: 0.04207725450396538 (0.029180203564465046) | > loss_mel_ce: 2.9414920806884766 (2.2497609002249583) | > loss: 0.011839562095701694 (0.009043417596590839) | > current_lr: 5e-06 | > step_time: 0.1151 (0.1401005642754691) | > loader_time: 0.0043 (0.004517878804888044)  --> TIME: 2025-02-27 14:26:23 -- STEP: 64/36938 -- GLOBAL_STEP: 1736150 | > loss_text_ce: 0.03102557361125946 (0.02959635536535643) | > loss_mel_ce: 2.5666744709014893 (2.232515886542386) | > loss: 0.010308334603905678 (0.008976636350553237) | > current_lr: 5e-06 | > step_time: 0.1499 (0.13810949400067332) | > loader_time: 0.0048 (0.004692409187555313)  --> TIME: 2025-02-27 14:26:33 -- STEP: 114/36938 -- GLOBAL_STEP: 1736200 | > loss_text_ce: 0.021599499508738518 (0.028938497057217256) | > loss_mel_ce: 2.4213573932647705 (nan) | > loss: 0.009694274514913559 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13623272326954636) | > loader_time: 0.0045 (0.004693167251453067)  --> TIME: 2025-02-27 14:26:42 -- STEP: 164/36938 -- GLOBAL_STEP: 1736250 | > loss_text_ce: 0.03254425898194313 (0.02891422684372562) | > loss_mel_ce: 0.35595703125 (nan) | > loss: 0.001541671808809042 (nan) | > current_lr: 5e-06 | > step_time: 0.12 (0.13520570644518226) | > loader_time: 0.0047 (0.004678762540584657)  --> TIME: 2025-02-27 14:26:51 -- STEP: 214/36938 -- GLOBAL_STEP: 1736300 | > loss_text_ce: 0.030519580468535423 (0.02902226808054425) | > loss_mel_ce: 2.2820210456848145 (nan) | > loss: 0.009176748804748058 (nan) | > current_lr: 5e-06 | > step_time: 0.1364 (0.13578273314181888) | > loader_time: 0.0077 (0.004684798071317581)  --> TIME: 2025-02-27 14:27:00 -- STEP: 264/36938 -- GLOBAL_STEP: 1736350 | > loss_text_ce: 0.025280384346842766 (0.029172770862439364) | > loss_mel_ce: 3.226545572280884 (nan) | > loss: 0.012904072180390358 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13624015902027936) | > loader_time: 0.0046 (0.004694566582188459)  --> TIME: 2025-02-27 14:27:10 -- STEP: 314/36938 -- GLOBAL_STEP: 1736400 | > loss_text_ce: 0.031023366376757622 (0.02947573380983749) | > loss_mel_ce: 2.387493848800659 (nan) | > loss: 0.009597290307283401 (nan) | > current_lr: 5e-06 | > step_time: 0.123 (0.13583939242514842) | > loader_time: 0.0045 (0.004697221099950702)  --> TIME: 2025-02-27 14:27:19 -- STEP: 364/36938 -- GLOBAL_STEP: 1736450 | > loss_text_ce: 0.03034890815615654 (0.029469881196516556) | > loss_mel_ce: 2.6740148067474365 (nan) | > loss: 0.010731603018939495 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13666787579819395) | > loader_time: 0.0047 (0.004689196308890542)  --> TIME: 2025-02-27 14:27:28 -- STEP: 414/36938 -- GLOBAL_STEP: 1736500 | > loss_text_ce: 0.025438491255044937 (0.029422093704702775) | > loss_mel_ce: 2.4058029651641846 (nan) | > loss: 0.009647784754633904 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13613243782577877) | > loader_time: 0.0046 (0.004679815780713361)  --> TIME: 2025-02-27 14:27:38 -- STEP: 464/36938 -- GLOBAL_STEP: 1736550 | > loss_text_ce: 0.02980010397732258 (0.029425118881245625) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.13639310875843316) | > loader_time: 0.0046 (0.004678362916255818)  --> TIME: 2025-02-27 14:27:47 -- STEP: 514/36938 -- GLOBAL_STEP: 1736600 | > loss_text_ce: 0.024553630501031876 (0.029463357335601566) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13671495478440815) | > loader_time: 0.0048 (0.004679440060478239)  --> TIME: 2025-02-27 14:27:57 -- STEP: 564/36938 -- GLOBAL_STEP: 1736650 | > loss_text_ce: 0.03772411867976189 (0.029492724567968795) | > loss_mel_ce: 3.1132819652557373 (nan) | > loss: 0.012503992766141891 (nan) | > current_lr: 5e-06 | > step_time: 0.1135 (0.1369454006776742) | > loader_time: 0.0045 (0.004677838044809109)  --> TIME: 2025-02-27 14:28:06 -- STEP: 614/36938 -- GLOBAL_STEP: 1736700 | > loss_text_ce: 0.029442323371767998 (0.029479961924206356) | > loss_mel_ce: 2.634960174560547 (nan) | > loss: 0.010573026724159718 (nan) | > current_lr: 5e-06 | > step_time: 0.1677 (0.13707586258941057) | > loader_time: 0.0049 (0.004677078234644589)  --> TIME: 2025-02-27 14:28:16 -- STEP: 664/36938 -- GLOBAL_STEP: 1736750 | > loss_text_ce: 0.026698142290115356 (0.029573716470096486) | > loss_mel_ce: 3.154597282409668 (nan) | > loss: 0.012624189257621765 (nan) | > current_lr: 5e-06 | > step_time: 0.1804 (0.1371335369276715) | > loader_time: 0.005 (0.004684073379240838)  --> TIME: 2025-02-27 14:28:25 -- STEP: 714/36938 -- GLOBAL_STEP: 1736800 | > loss_text_ce: 0.028373155742883682 (0.029598949975347153) | > loss_mel_ce: 3.2616822719573975 (nan) | > loss: 0.013055776245892048 (nan) | > current_lr: 5e-06 | > step_time: 0.1292 (0.13687700150059728) | > loader_time: 0.0046 (0.004679184333950865)  --> TIME: 2025-02-27 14:28:34 -- STEP: 764/36938 -- GLOBAL_STEP: 1736850 | > loss_text_ce: 0.0401112325489521 (0.029546518326146985) | > loss_mel_ce: 2.40631365776062 (nan) | > loss: 0.009708035737276077 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13624333929641103) | > loader_time: 0.0045 (0.004676288335111127)  --> TIME: 2025-02-27 14:28:43 -- STEP: 814/36938 -- GLOBAL_STEP: 1736900 | > loss_text_ce: 0.028037015348672867 (0.02952347117926953) | > loss_mel_ce: 2.895151138305664 (nan) | > loss: 0.011599954217672348 (nan) | > current_lr: 5e-06 | > step_time: 0.1605 (0.13623443223920476) | > loader_time: 0.0046 (0.0046737293646435234)  --> TIME: 2025-02-27 14:28:52 -- STEP: 864/36938 -- GLOBAL_STEP: 1736950 | > loss_text_ce: 0.030643124133348465 (0.029513538335406878) | > loss_mel_ce: 2.6249873638153076 (nan) | > loss: 0.01053821761161089 (nan) | > current_lr: 5e-06 | > step_time: 0.1215 (0.13618421030265337) | > loader_time: 0.0045 (0.004669538250675913)  --> TIME: 2025-02-27 14:29:02 -- STEP: 914/36938 -- GLOBAL_STEP: 1737000 | > loss_text_ce: 0.03256937488913536 (0.029534278782410647) | > loss_mel_ce: 2.4846956729888916 (nan) | > loss: 0.009989148005843163 (nan) | > current_lr: 5e-06 | > step_time: 0.1669 (0.1361775578242165) | > loader_time: 0.0046 (0.004666622410196188)  --> TIME: 2025-02-27 14:29:11 -- STEP: 964/36938 -- GLOBAL_STEP: 1737050 | > loss_text_ce: 0.02222905494272709 (0.02958996264752835) | > loss_mel_ce: 2.3943471908569336 (nan) | > loss: 0.009589588269591331 (nan) | > current_lr: 5e-06 | > step_time: 0.1552 (0.13621485925809004) | > loader_time: 0.0046 (0.004666886389008206)  --> TIME: 2025-02-27 14:29:20 -- STEP: 1014/36938 -- GLOBAL_STEP: 1737100 | > loss_text_ce: 0.022925224155187607 (0.029620325869257898) | > loss_mel_ce: 2.3464062213897705 (nan) | > loss: 0.009402109310030937 (nan) | > current_lr: 5e-06 | > step_time: 0.1208 (0.1363074998178428) | > loader_time: 0.0045 (0.004666689114692888)  --> TIME: 2025-02-27 14:29:30 -- STEP: 1064/36938 -- GLOBAL_STEP: 1737150 | > loss_text_ce: 0.037627167999744415 (0.029648408112098654) | > loss_mel_ce: 0.06860055029392242 (nan) | > loss: 0.00042153859976679087 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13648113070574044) | > loader_time: 0.0046 (0.004668629483172772)  --> TIME: 2025-02-27 14:29:40 -- STEP: 1114/36938 -- GLOBAL_STEP: 1737200 | > loss_text_ce: 0.029316609725356102 (0.029655945892370507) | > loss_mel_ce: 2.9339003562927246 (nan) | > loss: 0.011758798733353615 (nan) | > current_lr: 5e-06 | > step_time: 0.1404 (0.13673354286693817) | > loader_time: 0.0049 (0.004669075594342167)  --> TIME: 2025-02-27 14:29:49 -- STEP: 1164/36938 -- GLOBAL_STEP: 1737250 | > loss_text_ce: 0.028487330302596092 (0.02965714774035315) | > loss_mel_ce: 3.3469197750091553 (nan) | > loss: 0.013394474051892757 (nan) | > current_lr: 5e-06 | > step_time: 0.1387 (0.13684246404883793) | > loader_time: 0.0046 (0.00466868107261527)  --> TIME: 2025-02-27 14:29:59 -- STEP: 1214/36938 -- GLOBAL_STEP: 1737300 | > loss_text_ce: 0.03212043270468712 (0.029697414221015) | > loss_mel_ce: 2.544644832611084 (nan) | > loss: 0.010225259698927402 (nan) | > current_lr: 5e-06 | > step_time: 0.1331 (0.13701282046379332) | > loader_time: 0.0046 (0.004667267767954898)  --> TIME: 2025-02-27 14:30:08 -- STEP: 1264/36938 -- GLOBAL_STEP: 1737350 | > loss_text_ce: 0.026331957429647446 (0.029750453847934958) | > loss_mel_ce: 2.7215282917022705 (nan) | > loss: 0.010904207825660706 (nan) | > current_lr: 5e-06 | > step_time: 0.1635 (0.13701677624183375) | > loader_time: 0.0046 (0.004664665347413176)  --> TIME: 2025-02-27 14:30:18 -- STEP: 1314/36938 -- GLOBAL_STEP: 1737400 | > loss_text_ce: 0.02979316934943199 (0.029766345886042975) | > loss_mel_ce: 2.574507474899292 (nan) | > loss: 0.010334527119994164 (nan) | > current_lr: 5e-06 | > step_time: 0.1513 (0.1371515506660194) | > loader_time: 0.0046 (0.004662423917692002)  --> TIME: 2025-02-27 14:30:27 -- STEP: 1364/36938 -- GLOBAL_STEP: 1737450 | > loss_text_ce: 0.02916269749403 (0.02981046283149224) | > loss_mel_ce: 2.65826416015625 (nan) | > loss: 0.01066439226269722 (nan) | > current_lr: 5e-06 | > step_time: 0.1433 (0.13713356057220186) | > loader_time: 0.0047 (0.004662215185305241)  --> TIME: 2025-02-27 14:30:36 -- STEP: 1414/36938 -- GLOBAL_STEP: 1737500 | > loss_text_ce: 0.03658076003193855 (0.029845459153875712) | > loss_mel_ce: 2.7330429553985596 (nan) | > loss: 0.010990571230649948 (nan) | > current_lr: 5e-06 | > step_time: 0.1612 (0.13721034921379047) | > loader_time: 0.0045 (0.004662483888434553)  --> TIME: 2025-02-27 14:30:46 -- STEP: 1464/36938 -- GLOBAL_STEP: 1737550 | > loss_text_ce: 0.03304656967520714 (0.02983251254009389) | > loss_mel_ce: 3.0264196395874023 (nan) | > loss: 0.01214073970913887 (nan) | > current_lr: 5e-06 | > step_time: 0.1233 (0.13735781911292377) | > loader_time: 0.0046 (0.004661911009439348)  --> TIME: 2025-02-27 14:30:56 -- STEP: 1514/36938 -- GLOBAL_STEP: 1737600 | > loss_text_ce: 0.03223856911063194 (0.029798143841338543) | > loss_mel_ce: 2.508145809173584 (nan) | > loss: 0.010080890730023384 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.13741481634868202) | > loader_time: 0.0046 (0.004662297203487255)  --> TIME: 2025-02-27 14:31:05 -- STEP: 1564/36938 -- GLOBAL_STEP: 1737650 | > loss_text_ce: 0.027211982756853104 (0.029813213568345154) | > loss_mel_ce: 2.368276596069336 (nan) | > loss: 0.009505907073616982 (nan) | > current_lr: 5e-06 | > step_time: 0.1639 (0.13732504646491533) | > loader_time: 0.0049 (0.004660559584722486)  --> TIME: 2025-02-27 14:31:14 -- STEP: 1614/36938 -- GLOBAL_STEP: 1737700 | > loss_text_ce: 0.03634744510054588 (0.029797929272464167) | > loss_mel_ce: 3.1093430519104004 (nan) | > loss: 0.012482899241149426 (nan) | > current_lr: 5e-06 | > step_time: 0.1454 (0.1373657884798677) | > loader_time: 0.0046 (0.004659638085772993)  --> TIME: 2025-02-27 14:31:23 -- STEP: 1664/36938 -- GLOBAL_STEP: 1737750 | > loss_text_ce: 0.026887886226177216 (0.029787096945745786) | > loss_mel_ce: 3.1587154865264893 (nan) | > loss: 0.012641283683478832 (nan) | > current_lr: 5e-06 | > step_time: 0.1324 (0.13719156269843763) | > loader_time: 0.0046 (0.004659152088256976)  --> TIME: 2025-02-27 14:31:33 -- STEP: 1714/36938 -- GLOBAL_STEP: 1737800 | > loss_text_ce: 0.02277248725295067 (0.02975419034682597) | > loss_mel_ce: 0.8986642360687256 (nan) | > loss: 0.0036564951296895742 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1372719236683261) | > loader_time: 0.0046 (0.004658515979139217)  --> TIME: 2025-02-27 14:31:42 -- STEP: 1764/36938 -- GLOBAL_STEP: 1737850 | > loss_text_ce: 0.025372739881277084 (0.029764773819095074) | > loss_mel_ce: 2.404498815536499 (nan) | > loss: 0.009642347693443298 (nan) | > current_lr: 5e-06 | > step_time: 0.1224 (0.1372431530163133) | > loader_time: 0.0045 (0.004657036863487047)  --> TIME: 2025-02-27 14:31:51 -- STEP: 1814/36938 -- GLOBAL_STEP: 1737900 | > loss_text_ce: 0.0325721837580204 (0.029775844727485734) | > loss_mel_ce: 1.4113876819610596 (nan) | > loss: 0.0057299998588860035 (nan) | > current_lr: 5e-06 | > step_time: 0.1157 (0.13714541266482363) | > loader_time: 0.0045 (0.004654986461441724)  --> TIME: 2025-02-27 14:32:01 -- STEP: 1864/36938 -- GLOBAL_STEP: 1737950 | > loss_text_ce: 0.024789147078990936 (0.029761885723264995) | > loss_mel_ce: 2.5609610080718994 (nan) | > loss: 0.010260913521051407 (nan) | > current_lr: 5e-06 | > step_time: 0.1323 (0.1371379443504268) | > loader_time: 0.0046 (0.0046547596290899395)  --> TIME: 2025-02-27 14:32:10 -- STEP: 1914/36938 -- GLOBAL_STEP: 1738000 | > loss_text_ce: 0.034037794917821884 (0.02972550374989047) | > loss_mel_ce: 2.70322585105896 (nan) | > loss: 0.010862157680094242 (nan) | > current_lr: 5e-06 | > step_time: 0.1312 (0.13713242504414844) | > loader_time: 0.0048 (0.004652851801307218)  --> TIME: 2025-02-27 14:32:19 -- STEP: 1964/36938 -- GLOBAL_STEP: 1738050 | > loss_text_ce: 0.028109904378652573 (0.02975636237569328) | > loss_mel_ce: 1.5251399278640747 (nan) | > loss: 0.006163690239191055 (nan) | > current_lr: 5e-06 | > step_time: 0.1141 (0.13707103348069666) | > loader_time: 0.0045 (0.00465392435155429)  --> TIME: 2025-02-27 14:32:29 -- STEP: 2014/36938 -- GLOBAL_STEP: 1738100 | > loss_text_ce: 0.03155007213354111 (0.02973174549525469) | > loss_mel_ce: 3.114453077316284 (nan) | > loss: 0.012484140694141388 (nan) | > current_lr: 5e-06 | > step_time: 0.1162 (0.1371371025125225) | > loader_time: 0.0047 (0.004653579428747601)  --> TIME: 2025-02-27 14:32:38 -- STEP: 2064/36938 -- GLOBAL_STEP: 1738150 | > loss_text_ce: 0.028973592445254326 (0.02973039553057612) | > loss_mel_ce: 2.780346393585205 (nan) | > loss: 0.011148096062242985 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13709284122600113) | > loader_time: 0.0046 (0.004652537001195799)  --> TIME: 2025-02-27 14:32:47 -- STEP: 2114/36938 -- GLOBAL_STEP: 1738200 | > loss_text_ce: 0.031621724367141724 (0.029748853823159494) | > loss_mel_ce: 2.9608216285705566 (nan) | > loss: 0.011874775402247906 (nan) | > current_lr: 5e-06 | > step_time: 0.1247 (0.13715965542572214) | > loader_time: 0.0046 (0.004653621193588959)  --> TIME: 2025-02-27 14:32:57 -- STEP: 2164/36938 -- GLOBAL_STEP: 1738250 | > loss_text_ce: 0.03349136561155319 (0.029751121371428435) | > loss_mel_ce: 2.6958067417144775 (nan) | > loss: 0.010830548591911793 (nan) | > current_lr: 5e-06 | > step_time: 0.1715 (0.13718463290420774) | > loader_time: 0.0048 (0.004653276335069297)  --> TIME: 2025-02-27 14:33:06 -- STEP: 2214/36938 -- GLOBAL_STEP: 1738300 | > loss_text_ce: 0.033393099904060364 (0.029756120927068176) | > loss_mel_ce: 1.9666091203689575 (nan) | > loss: 0.007936516776680946 (nan) | > current_lr: 5e-06 | > step_time: 0.1106 (0.1372221103097902) | > loader_time: 0.0045 (0.004654256093577422)  --> TIME: 2025-02-27 14:33:16 -- STEP: 2264/36938 -- GLOBAL_STEP: 1738350 | > loss_text_ce: 0.022877201437950134 (0.029755011441345673) | > loss_mel_ce: 2.5414843559265137 (nan) | > loss: 0.0101760383695364 (nan) | > current_lr: 5e-06 | > step_time: 0.1158 (0.13718813289180642) | > loader_time: 0.0045 (0.004654389388148435)  --> TIME: 2025-02-27 14:33:25 -- STEP: 2314/36938 -- GLOBAL_STEP: 1738400 | > loss_text_ce: 0.03666689246892929 (0.02973919926439037) | > loss_mel_ce: 2.385730743408203 (nan) | > loss: 0.009612689726054668 (nan) | > current_lr: 5e-06 | > step_time: 0.129 (0.13710293547284919) | > loader_time: 0.0046 (0.004653861425823445)  --> TIME: 2025-02-27 14:33:34 -- STEP: 2364/36938 -- GLOBAL_STEP: 1738450 | > loss_text_ce: 0.025449365377426147 (0.029723531439901966) | > loss_mel_ce: 2.077364683151245 (nan) | > loss: 0.008344500325620174 (nan) | > current_lr: 5e-06 | > step_time: 0.118 (0.13707555369072136) | > loader_time: 0.0046 (0.004655673116596816)  --> TIME: 2025-02-27 14:33:43 -- STEP: 2414/36938 -- GLOBAL_STEP: 1738500 | > loss_text_ce: 0.028899436816573143 (0.029723154131111675) | > loss_mel_ce: 1.8031455278396606 (nan) | > loss: 0.007270019967108965 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1370151801453014) | > loader_time: 0.0047 (0.004655382706295281)  --> TIME: 2025-02-27 14:33:52 -- STEP: 2464/36938 -- GLOBAL_STEP: 1738550 | > loss_text_ce: 0.03686848655343056 (0.029745171403900907) | > loss_mel_ce: 0.05216064304113388 (nan) | > loss: 0.00035329023376107216 (nan) | > current_lr: 5e-06 | > step_time: 0.1425 (0.13697118399205138) | > loader_time: 0.0048 (0.004656039275132209)  --> TIME: 2025-02-27 14:34:02 -- STEP: 2514/36938 -- GLOBAL_STEP: 1738600 | > loss_text_ce: 0.030834542587399483 (0.029767296183076383) | > loss_mel_ce: 2.7501864433288574 (nan) | > loss: 0.011035798117518425 (nan) | > current_lr: 5e-06 | > step_time: 0.1171 (0.1369964466648814) | > loader_time: 0.0045 (0.004655333578159808)  --> TIME: 2025-02-27 14:34:11 -- STEP: 2564/36938 -- GLOBAL_STEP: 1738650 | > loss_text_ce: 0.029484370723366737 (0.02976873255746697) | > loss_mel_ce: 2.2721667289733887 (nan) | > loss: 0.009133536368608475 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13706878157748073) | > loader_time: 0.0045 (0.004655396324609998)  --> TIME: 2025-02-27 14:34:21 -- STEP: 2614/36938 -- GLOBAL_STEP: 1738700 | > loss_text_ce: 0.032406967133283615 (0.029759328383574715) | > loss_mel_ce: 3.3677830696105957 (nan) | > loss: 0.013492818921804428 (nan) | > current_lr: 5e-06 | > step_time: 0.1386 (0.1371060226742512) | > loader_time: 0.0046 (0.004654991052127005)  --> TIME: 2025-02-27 14:34:30 -- STEP: 2664/36938 -- GLOBAL_STEP: 1738750 | > loss_text_ce: 0.03276844322681427 (0.029740436034949698) | > loss_mel_ce: 1.9054193496704102 (nan) | > loss: 0.007691222243010998 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13713284458842004) | > loader_time: 0.0045 (0.004654785444786586)  --> TIME: 2025-02-27 14:34:40 -- STEP: 2714/36938 -- GLOBAL_STEP: 1738800 | > loss_text_ce: 0.029360158368945122 (0.029765285471421314) | > loss_mel_ce: 2.3807616233825684 (nan) | > loss: 0.009563975036144257 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13716622864224912) | > loader_time: 0.0046 (0.004654286447130049)  --> TIME: 2025-02-27 14:34:49 -- STEP: 2764/36938 -- GLOBAL_STEP: 1738850 | > loss_text_ce: 0.04716895893216133 (0.0297829017790126) | > loss_mel_ce: 3.385704517364502 (nan) | > loss: 0.01362251490354538 (nan) | > current_lr: 5e-06 | > step_time: 0.182 (0.13727638259120306) | > loader_time: 0.0048 (0.004655608219969281)  --> TIME: 2025-02-27 14:34:59 -- STEP: 2814/36938 -- GLOBAL_STEP: 1738900 | > loss_text_ce: 0.017826208844780922 (0.029790893360719547) | > loss_mel_ce: 1.963841438293457 (nan) | > loss: 0.00786376092582941 (nan) | > current_lr: 5e-06 | > step_time: 0.1073 (0.1372565743596142) | > loader_time: 0.0044 (0.004654962201447198)  --> TIME: 2025-02-27 14:35:08 -- STEP: 2864/36938 -- GLOBAL_STEP: 1738950 | > loss_text_ce: 0.028277995064854622 (0.029769952466364102) | > loss_mel_ce: 2.1981892585754395 (nan) | > loss: 0.00883518811315298 (nan) | > current_lr: 5e-06 | > step_time: 0.1091 (0.13716783012424738) | > loader_time: 0.0044 (0.004655188605106071)  --> TIME: 2025-02-27 14:35:17 -- STEP: 2914/36938 -- GLOBAL_STEP: 1739000 | > loss_text_ce: 0.032797981053590775 (0.029773517801185367) | > loss_mel_ce: 3.1644046306610107 (nan) | > loss: 0.012687313370406628 (nan) | > current_lr: 5e-06 | > step_time: 0.1394 (0.13724546736859233) | > loader_time: 0.0048 (0.0046551964752626586)  --> TIME: 2025-02-27 14:35:27 -- STEP: 2964/36938 -- GLOBAL_STEP: 1739050 | > loss_text_ce: 0.02915300242602825 (0.02975133348268289) | > loss_mel_ce: 2.968411445617676 (nan) | > loss: 0.01189509779214859 (nan) | > current_lr: 5e-06 | > step_time: 0.1608 (0.13729462256798353) | > loader_time: 0.0048 (0.00465535819932677)  --> TIME: 2025-02-27 14:35:36 -- STEP: 3014/36938 -- GLOBAL_STEP: 1739100 | > loss_text_ce: 0.023994360119104385 (0.029733272381689255) | > loss_mel_ce: 0.0108184814453125 (nan) | > loss: 0.00013814620615448803 (nan) | > current_lr: 5e-06 | > step_time: 0.1242 (0.13725472493288746) | > loader_time: 0.0045 (0.004654570613068776)  --> TIME: 2025-02-27 14:35:45 -- STEP: 3064/36938 -- GLOBAL_STEP: 1739150 | > loss_text_ce: 0.02826409786939621 (0.02970286474004509) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13711705500401014) | > loader_time: 0.0045 (0.004653813244468544)  --> TIME: 2025-02-27 14:35:54 -- STEP: 3114/36938 -- GLOBAL_STEP: 1739200 | > loss_text_ce: 0.047051358968019485 (0.02970712453706516) | > loss_mel_ce: 3.1548614501953125 (nan) | > loss: 0.012706004083156586 (nan) | > current_lr: 5e-06 | > step_time: 0.1651 (0.13714668340382846) | > loader_time: 0.0048 (0.004653866350765627)  --> TIME: 2025-02-27 14:36:04 -- STEP: 3164/36938 -- GLOBAL_STEP: 1739250 | > loss_text_ce: 0.02988952025771141 (0.029718011469455425) | > loss_mel_ce: 2.969557046890259 (nan) | > loss: 0.011902566999197006 (nan) | > current_lr: 5e-06 | > step_time: 0.1411 (0.13715720952838162) | > loader_time: 0.0047 (0.004654375023847879)  --> TIME: 2025-02-27 14:36:13 -- STEP: 3214/36938 -- GLOBAL_STEP: 1739300 | > loss_text_ce: 0.03621062636375427 (0.029738180971181093) | > loss_mel_ce: 2.403468608856201 (nan) | > loss: 0.009681266732513905 (nan) | > current_lr: 5e-06 | > step_time: 0.1437 (0.13717971288720185) | > loader_time: 0.0047 (0.004654435245013035)  --> TIME: 2025-02-27 14:36:22 -- STEP: 3264/36938 -- GLOBAL_STEP: 1739350 | > loss_text_ce: 0.026318827643990517 (0.029749667110045976) | > loss_mel_ce: 2.290752410888672 (nan) | > loss: 0.009194727055728436 (nan) | > current_lr: 5e-06 | > step_time: 0.1336 (0.13714668220456905) | > loader_time: 0.0046 (0.004654056155214116)  --> TIME: 2025-02-27 14:36:32 -- STEP: 3314/36938 -- GLOBAL_STEP: 1739400 | > loss_text_ce: 0.027548672631382942 (0.02974850298006674) | > loss_mel_ce: 2.9725944995880127 (nan) | > loss: 0.011905331164598465 (nan) | > current_lr: 5e-06 | > step_time: 0.1837 (0.13712255675604207) | > loader_time: 0.007 (0.004656151899315892)  --> TIME: 2025-02-27 14:36:41 -- STEP: 3364/36938 -- GLOBAL_STEP: 1739450 | > loss_text_ce: 0.050375618040561676 (0.029754859404322796) | > loss_mel_ce: 1.9001564979553223 (nan) | > loss: 0.007740207016468048 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13707218180088315) | > loader_time: 0.0047 (0.004656114419490345)  --> TIME: 2025-02-27 14:36:50 -- STEP: 3414/36938 -- GLOBAL_STEP: 1739500 | > loss_text_ce: 0.025786131620407104 (0.029754575990783138) | > loss_mel_ce: 0.6637298464775085 (nan) | > loss: 0.002736174501478672 (nan) | > current_lr: 5e-06 | > step_time: 0.1101 (0.13702862773922095) | > loader_time: 0.0045 (0.004656489508572416)  --> TIME: 2025-02-27 14:36:59 -- STEP: 3464/36938 -- GLOBAL_STEP: 1739550 | > loss_text_ce: 0.030638346448540688 (0.029752183684194133) | > loss_mel_ce: 2.5417723655700684 (nan) | > loss: 0.010207979939877987 (nan) | > current_lr: 5e-06 | > step_time: 0.1108 (0.13697896040890042) | > loader_time: 0.0061 (0.0046566476309822516)  --> TIME: 2025-02-27 14:37:08 -- STEP: 3514/36938 -- GLOBAL_STEP: 1739600 | > loss_text_ce: 0.02834942936897278 (0.02975742107890143) | > loss_mel_ce: 2.3807361125946045 (nan) | > loss: 0.00955986324697733 (nan) | > current_lr: 5e-06 | > step_time: 0.1248 (0.13695853185300905) | > loader_time: 0.0046 (0.004656037961344316)  --> TIME: 2025-02-27 14:37:18 -- STEP: 3564/36938 -- GLOBAL_STEP: 1739650 | > loss_text_ce: 0.029095515608787537 (0.029765825272317217) | > loss_mel_ce: 2.4002139568328857 (nan) | > loss: 0.009640117175877094 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13700631137113686) | > loader_time: 0.0045 (0.004656273708600384)  --> TIME: 2025-02-27 14:37:27 -- STEP: 3614/36938 -- GLOBAL_STEP: 1739700 | > loss_text_ce: 0.027761617675423622 (0.029756246207052077) | > loss_mel_ce: 2.8141591548919678 (nan) | > loss: 0.011277464218437672 (nan) | > current_lr: 5e-06 | > step_time: 0.1526 (0.13700759555994016) | > loader_time: 0.0046 (0.004656238323690878)  --> TIME: 2025-02-27 14:37:36 -- STEP: 3664/36938 -- GLOBAL_STEP: 1739750 | > loss_text_ce: 0.02162226475775242 (0.029766411957987803) | > loss_mel_ce: 2.07458758354187 (nan) | > loss: 0.008318292908370495 (nan) | > current_lr: 5e-06 | > step_time: 0.1315 (0.13697993872467648) | > loader_time: 0.0046 (0.004656222905134006)  --> TIME: 2025-02-27 14:37:46 -- STEP: 3714/36938 -- GLOBAL_STEP: 1739800 | > loss_text_ce: 0.0319276787340641 (0.02978022455869219) | > loss_mel_ce: 2.771912097930908 (nan) | > loss: 0.011126348748803139 (nan) | > current_lr: 5e-06 | > step_time: 0.1519 (0.13697814293041272) | > loader_time: 0.0046 (0.004655674637551579)  --> TIME: 2025-02-27 14:37:55 -- STEP: 3764/36938 -- GLOBAL_STEP: 1739850 | > loss_text_ce: 0.02635258622467518 (0.02977109731840094) | > loss_mel_ce: 2.8978962898254395 (nan) | > loss: 0.011604162864387035 (nan) | > current_lr: 5e-06 | > step_time: 0.1644 (0.1369274902799814) | > loader_time: 0.0047 (0.004655348950567455)  --> TIME: 2025-02-27 14:38:04 -- STEP: 3814/36938 -- GLOBAL_STEP: 1739900 | > loss_text_ce: 0.04143534228205681 (0.029785396054330438) | > loss_mel_ce: 1.9311167001724243 (nan) | > loss: 0.007827588357031345 (nan) | > current_lr: 5e-06 | > step_time: 0.1097 (0.13692079599327514) | > loader_time: 0.0047 (0.00465584726438637)  --> TIME: 2025-02-27 14:38:14 -- STEP: 3864/36938 -- GLOBAL_STEP: 1739950 | > loss_text_ce: 0.04279213771224022 (0.029779284334679825) | > loss_mel_ce: 3.496875047683716 (nan) | > loss: 0.014046299271285534 (nan) | > current_lr: 5e-06 | > step_time: 0.122 (0.1369181912137858) | > loader_time: 0.0046 (0.004655713188475453)  --> TIME: 2025-02-27 14:38:23 -- STEP: 3914/36938 -- GLOBAL_STEP: 1740000 | > loss_text_ce: 0.019329901784658432 (0.029775452940721456) | > loss_mel_ce: 2.7254114151000977 (nan) | > loss: 0.010891830548644066 (nan) | > current_lr: 5e-06 | > step_time: 0.1205 (0.13693458472336226) | > loader_time: 0.0047 (0.004656836763595398) > CHECKPOINT : xtts_luo/GPT_XTTS_LUO_FT-February-23-2025_07+52PM-8e59ec3/checkpoint_1740000.pth  --> TIME: 2025-02-27 14:38:38 -- STEP: 3964/36938 -- GLOBAL_STEP: 1740050 | > loss_text_ce: 0.03230513632297516 (0.029769827776206004) | > loss_mel_ce: 2.66556978225708 (nan) | > loss: 0.010705853812396526 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13690992326717233) | > loader_time: 0.0046 (0.00465675914804822)  --> TIME: 2025-02-27 14:38:47 -- STEP: 4014/36938 -- GLOBAL_STEP: 1740100 | > loss_text_ce: 0.0308543611317873 (0.029760813141936793) | > loss_mel_ce: 0.0021680090576410294 (nan) | > loss: 0.00013104116078466177 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13687314770978454) | > loader_time: 0.0046 (0.004657014068466662)  --> TIME: 2025-02-27 14:38:57 -- STEP: 4064/36938 -- GLOBAL_STEP: 1740150 | > loss_text_ce: 0.03374449163675308 (0.029754854669963103) | > loss_mel_ce: 2.7754945755004883 (nan) | > loss: 0.011147774755954742 (nan) | > current_lr: 5e-06 | > step_time: 0.1865 (0.1368916210109796) | > loader_time: 0.0049 (0.004657053571986398)  --> TIME: 2025-02-27 14:39:06 -- STEP: 4114/36938 -- GLOBAL_STEP: 1740200 | > loss_text_ce: 0.025784365832805634 (0.029749382002705656) | > loss_mel_ce: 2.8168835639953613 (nan) | > loss: 0.011280428618192673 (nan) | > current_lr: 5e-06 | > step_time: 0.1289 (0.13684728560542764) | > loader_time: 0.0047 (0.00465691524277512)  --> TIME: 2025-02-27 14:39:15 -- STEP: 4164/36938 -- GLOBAL_STEP: 1740250 | > loss_text_ce: 0.02329954318702221 (0.029752615393040114) | > loss_mel_ce: 2.8156540393829346 (nan) | > loss: 0.011265689507126808 (nan) | > current_lr: 5e-06 | > step_time: 0.1296 (0.13682056154687194) | > loader_time: 0.0046 (0.004656645509626402)  --> TIME: 2025-02-27 14:39:25 -- STEP: 4214/36938 -- GLOBAL_STEP: 1740300 | > loss_text_ce: 0.02413729764521122 (0.029747214153502708) | > loss_mel_ce: 2.699716806411743 (nan) | > loss: 0.010808945633471012 (nan) | > current_lr: 5e-06 | > step_time: 0.185 (0.136835857879284) | > loader_time: 0.0047 (0.0046568776851702905)  --> TIME: 2025-02-27 14:39:34 -- STEP: 4264/36938 -- GLOBAL_STEP: 1740350 | > loss_text_ce: 0.026560146361589432 (0.029729210000832892) | > loss_mel_ce: 2.2226498126983643 (nan) | > loss: 0.00892543699592352 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.13682445212779318) | > loader_time: 0.0048 (0.004657249569221909)  --> TIME: 2025-02-27 14:39:43 -- STEP: 4314/36938 -- GLOBAL_STEP: 1740400 | > loss_text_ce: 0.015547404065728188 (0.029716604559582565) | > loss_mel_ce: 2.988042116165161 (nan) | > loss: 0.011919006705284119 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.13684583644265663) | > loader_time: 0.0047 (0.004657541484151881)  --> TIME: 2025-02-27 14:39:52 -- STEP: 4364/36938 -- GLOBAL_STEP: 1740450 | > loss_text_ce: 0.030367843806743622 (0.02971413531076029) | > loss_mel_ce: 0.006063964683562517 (nan) | > loss: 0.0001445706730009988 (nan) | > current_lr: 5e-06 | > step_time: 0.1164 (0.1368197305938939) | > loader_time: 0.0045 (0.004657037207885793)  --> TIME: 2025-02-27 14:40:02 -- STEP: 4414/36938 -- GLOBAL_STEP: 1740500 | > loss_text_ce: 0.03019791841506958 (0.029714102756593235) | > loss_mel_ce: 3.193082332611084 (nan) | > loss: 0.012790795415639877 (nan) | > current_lr: 5e-06 | > step_time: 0.1872 (0.13688953013564867) | > loader_time: 0.0049 (0.004657376876609806)  --> TIME: 2025-02-27 14:40:11 -- STEP: 4464/36938 -- GLOBAL_STEP: 1740550 | > loss_text_ce: 0.028125785291194916 (0.029711397296930973) | > loss_mel_ce: 2.8471672534942627 (nan) | > loss: 0.01140989363193512 (nan) | > current_lr: 5e-06 | > step_time: 0.1559 (0.13685605010037785) | > loader_time: 0.0047 (0.004657424158520163)  --> TIME: 2025-02-27 14:40:21 -- STEP: 4514/36938 -- GLOBAL_STEP: 1740600 | > loss_text_ce: 0.045724689960479736 (0.02970977077726068) | > loss_mel_ce: 1.8878988027572632 (nan) | > loss: 0.007673109881579876 (nan) | > current_lr: 5e-06 | > step_time: 0.1184 (0.13685618061696106) | > loader_time: 0.0046 (0.004657425181127296)  --> TIME: 2025-02-27 14:40:30 -- STEP: 4564/36938 -- GLOBAL_STEP: 1740650 | > loss_text_ce: 0.02612905204296112 (0.029716127615962352) | > loss_mel_ce: 1.7528867721557617 (nan) | > loss: 0.007059586700052023 (nan) | > current_lr: 5e-06 | > step_time: 0.1124 (0.13689738334426754) | > loader_time: 0.0045 (0.004658049914421053)  --> TIME: 2025-02-27 14:40:40 -- STEP: 4614/36938 -- GLOBAL_STEP: 1740700 | > loss_text_ce: 0.04141946882009506 (0.029713815291088002) | > loss_mel_ce: 2.410748243331909 (nan) | > loss: 0.009730825200676918 (nan) | > current_lr: 5e-06 | > step_time: 0.1139 (0.13695178964099802) | > loader_time: 0.0045 (0.0046582318613112616)  --> TIME: 2025-02-27 14:40:49 -- STEP: 4664/36938 -- GLOBAL_STEP: 1740750 | > loss_text_ce: 0.02719336748123169 (0.029709696602978855) | > loss_mel_ce: 2.852551221847534 (nan) | > loss: 0.011427558027207851 (nan) | > current_lr: 5e-06 | > step_time: 0.1441 (0.13692184679504135) | > loader_time: 0.0046 (0.0046579216704360285)  --> TIME: 2025-02-27 14:40:59 -- STEP: 4714/36938 -- GLOBAL_STEP: 1740800 | > loss_text_ce: 0.027719056233763695 (0.02971182260160501) | > loss_mel_ce: 2.7627899646759033 (nan) | > loss: 0.011073448695242405 (nan) | > current_lr: 5e-06 | > step_time: 0.1503 (0.13695237744800337) | > loader_time: 0.0048 (0.004658335945491915)  --> TIME: 2025-02-27 14:41:08 -- STEP: 4764/36938 -- GLOBAL_STEP: 1740850 | > loss_text_ce: 0.033019647002220154 (0.029706735930254632) | > loss_mel_ce: 2.759413003921509 (nan) | > loss: 0.011081081815063953 (nan) | > current_lr: 5e-06 | > step_time: 0.1695 (0.13693096890958023) | > loader_time: 0.0046 (0.004658061701344587)  --> TIME: 2025-02-27 14:41:17 -- STEP: 4814/36938 -- GLOBAL_STEP: 1740900 | > loss_text_ce: 0.03705020621418953 (0.02969928591976063) | > loss_mel_ce: 2.1832849979400635 (nan) | > loss: 0.008810854516923428 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13692339996009864) | > loader_time: 0.0044 (0.004658920516302124)  --> TIME: 2025-02-27 14:41:26 -- STEP: 4864/36938 -- GLOBAL_STEP: 1740950 | > loss_text_ce: 0.02951824851334095 (0.029704121610848226) | > loss_mel_ce: 2.8355488777160645 (nan) | > loss: 0.011369314976036549 (nan) | > current_lr: 5e-06 | > step_time: 0.1231 (0.13691152544005955) | > loader_time: 0.0046 (0.0046590123521654024)  --> TIME: 2025-02-27 14:41:36 -- STEP: 4914/36938 -- GLOBAL_STEP: 1741000 | > loss_text_ce: 0.034059926867485046 (0.029700280342282896) | > loss_mel_ce: 2.2046279907226562 (nan) | > loss: 0.008883683010935783 (nan) | > current_lr: 5e-06 | > step_time: 0.1204 (0.13689788575574188) | > loader_time: 0.0045 (0.004658935173866828)  --> TIME: 2025-02-27 14:41:45 -- STEP: 4964/36938 -- GLOBAL_STEP: 1741050 | > loss_text_ce: 0.039733074605464935 (0.029703610401456505) | > loss_mel_ce: 2.3009510040283203 (nan) | > loss: 0.009288430213928223 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13685876529702853) | > loader_time: 0.0047 (0.0046585498078996895)  --> TIME: 2025-02-27 14:41:54 -- STEP: 5014/36938 -- GLOBAL_STEP: 1741100 | > loss_text_ce: 0.0353764146566391 (0.029716063350272638) | > loss_mel_ce: 2.2822377681732178 (nan) | > loss: 0.009196882136166096 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13689785917584174) | > loader_time: 0.0044 (0.004658703505398486)  --> TIME: 2025-02-27 14:42:03 -- STEP: 5064/36938 -- GLOBAL_STEP: 1741150 | > loss_text_ce: 0.03950490057468414 (0.029716379371490825) | > loss_mel_ce: 2.9810025691986084 (nan) | > loss: 0.01198614202439785 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13682539899955648) | > loader_time: 0.0045 (0.004658706851103459)  --> TIME: 2025-02-27 14:42:12 -- STEP: 5114/36938 -- GLOBAL_STEP: 1741200 | > loss_text_ce: 0.02829967997968197 (0.029716553962339652) | > loss_mel_ce: 2.6840643882751465 (nan) | > loss: 0.010763349942862988 (nan) | > current_lr: 5e-06 | > step_time: 0.1347 (0.13678887774341073) | > loader_time: 0.0047 (0.004658687380453898)  --> TIME: 2025-02-27 14:42:21 -- STEP: 5164/36938 -- GLOBAL_STEP: 1741250 | > loss_text_ce: 0.033200185745954514 (0.029720904542375595) | > loss_mel_ce: 2.8668363094329834 (nan) | > loss: 0.011508082039654255 (nan) | > current_lr: 5e-06 | > step_time: 0.173 (0.13679958562533345) | > loader_time: 0.005 (0.004658679044311286)  --> TIME: 2025-02-27 14:42:31 -- STEP: 5214/36938 -- GLOBAL_STEP: 1741300 | > loss_text_ce: 0.03226510062813759 (0.02971755787405129) | > loss_mel_ce: 3.1035256385803223 (nan) | > loss: 0.01244361512362957 (nan) | > current_lr: 5e-06 | > step_time: 0.1712 (0.13678231388387663) | > loader_time: 0.0047 (0.00465835489712776)  --> TIME: 2025-02-27 14:42:40 -- STEP: 5264/36938 -- GLOBAL_STEP: 1741350 | > loss_text_ce: 0.030566353350877762 (0.029725280664839968) | > loss_mel_ce: 3.122300863265991 (nan) | > loss: 0.012511379085481167 (nan) | > current_lr: 5e-06 | > step_time: 0.1859 (0.13674542638487416) | > loader_time: 0.0049 (0.0046581541694768434)  --> TIME: 2025-02-27 14:42:49 -- STEP: 5314/36938 -- GLOBAL_STEP: 1741400 | > loss_text_ce: 0.028054265305399895 (0.02974176928341405) | > loss_mel_ce: 2.3529574871063232 (nan) | > loss: 0.0094484593719244 (nan) | > current_lr: 5e-06 | > step_time: 0.1321 (0.13671450763884777) | > loader_time: 0.0046 (0.004657929671362532)  --> TIME: 2025-02-27 14:42:59 -- STEP: 5364/36938 -- GLOBAL_STEP: 1741450 | > loss_text_ce: 0.03059818223118782 (0.029740905815090958) | > loss_mel_ce: 2.4811131954193115 (nan) | > loss: 0.009967109188437462 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.13678270738396917) | > loader_time: 0.0048 (0.004657974001367421)  --> TIME: 2025-02-27 14:43:08 -- STEP: 5414/36938 -- GLOBAL_STEP: 1741500 | > loss_text_ce: 0.028747085481882095 (0.029742714662192915) | > loss_mel_ce: 2.6463887691497803 (nan) | > loss: 0.010615618899464607 (nan) | > current_lr: 5e-06 | > step_time: 0.1672 (0.13677482812837127) | > loader_time: 0.0048 (0.004657737919004521)  --> TIME: 2025-02-27 14:43:17 -- STEP: 5464/36938 -- GLOBAL_STEP: 1741550 | > loss_text_ce: 0.03436543047428131 (0.029756636327605737) | > loss_mel_ce: 2.339262008666992 (nan) | > loss: 0.009419157169759274 (nan) | > current_lr: 5e-06 | > step_time: 0.1771 (0.13676201961225554) | > loader_time: 0.0047 (0.004657211712129267)  --> TIME: 2025-02-27 14:43:27 -- STEP: 5514/36938 -- GLOBAL_STEP: 1741600 | > loss_text_ce: 0.03194810450077057 (0.02974548372958567) | > loss_mel_ce: 3.056509256362915 (nan) | > loss: 0.012255784124135971 (nan) | > current_lr: 5e-06 | > step_time: 0.1251 (0.13679828672336458) | > loader_time: 0.0046 (0.004657096693296963)  --> TIME: 2025-02-27 14:43:36 -- STEP: 5564/36938 -- GLOBAL_STEP: 1741650 | > loss_text_ce: 0.02386731281876564 (0.02973364481764713) | > loss_mel_ce: 2.729123115539551 (nan) | > loss: 0.010924565605819225 (nan) | > current_lr: 5e-06 | > step_time: 0.1395 (0.13683081680197795) | > loader_time: 0.0046 (0.0046571795242797455)  --> TIME: 2025-02-27 14:43:46 -- STEP: 5614/36938 -- GLOBAL_STEP: 1741700 | > loss_text_ce: 0.027717391029000282 (0.029741089340992132) | > loss_mel_ce: 2.920743465423584 (nan) | > loss: 0.011700241826474667 (nan) | > current_lr: 5e-06 | > step_time: 0.1866 (0.1368222851578269) | > loader_time: 0.0052 (0.004657215056234183)  --> TIME: 2025-02-27 14:43:55 -- STEP: 5664/36938 -- GLOBAL_STEP: 1741750 | > loss_text_ce: 0.02293553575873375 (0.029744199441455163) | > loss_mel_ce: 2.2744662761688232 (nan) | > loss: 0.009116674773395061 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.1368203243966834) | > loader_time: 0.0047 (0.0046572077409022)  --> TIME: 2025-02-27 14:44:04 -- STEP: 5714/36938 -- GLOBAL_STEP: 1741800 | > loss_text_ce: 0.031410157680511475 (0.029739354254653766) | > loss_mel_ce: 2.7556746006011963 (nan) | > loss: 0.011059860698878765 (nan) | > current_lr: 5e-06 | > step_time: 0.1335 (0.13681193281813986) | > loader_time: 0.0047 (0.004657029938689444)  --> TIME: 2025-02-27 14:44:14 -- STEP: 5764/36938 -- GLOBAL_STEP: 1741850 | > loss_text_ce: 0.025266122072935104 (0.029738724086301797) | > loss_mel_ce: 3.155679702758789 (nan) | > loss: 0.012622801586985588 (nan) | > current_lr: 5e-06 | > step_time: 0.1368 (0.1368557613066387) | > loader_time: 0.0046 (0.0046575715523639455)  --> TIME: 2025-02-27 14:44:23 -- STEP: 5814/36938 -- GLOBAL_STEP: 1741900 | > loss_text_ce: 0.031776487827301025 (0.029737076104087105) | > loss_mel_ce: 2.5825419425964355 (nan) | > loss: 0.01037427969276905 (nan) | > current_lr: 5e-06 | > step_time: 0.1441 (0.13686086740287126) | > loader_time: 0.0046 (0.004657331840008591)  --> TIME: 2025-02-27 14:44:33 -- STEP: 5864/36938 -- GLOBAL_STEP: 1741950 | > loss_text_ce: 0.02811872586607933 (0.02973299969615361) | > loss_mel_ce: 2.927483320236206 (nan) | > loss: 0.011728580109775066 (nan) | > current_lr: 5e-06 | > step_time: 0.1832 (0.1369457135909578) | > loader_time: 0.0049 (0.00465808940942094)  --> TIME: 2025-02-27 14:44:43 -- STEP: 5914/36938 -- GLOBAL_STEP: 1742000 | > loss_text_ce: 0.01636776328086853 (0.029730150980971266) | > loss_mel_ce: 1.6610839366912842 (nan) | > loss: 0.006656554993242025 (nan) | > current_lr: 5e-06 | > step_time: 0.1183 (0.13694774858605702) | > loader_time: 0.0045 (0.004658132257355108)  --> TIME: 2025-02-27 14:44:52 -- STEP: 5964/36938 -- GLOBAL_STEP: 1742050 | > loss_text_ce: 0.02760259620845318 (0.02972602323141126) | > loss_mel_ce: 2.697443962097168 (nan) | > loss: 0.010813677683472633 (nan) | > current_lr: 5e-06 | > step_time: 0.1232 (0.13692069729249182) | > loader_time: 0.0045 (0.004658002049070643)  --> TIME: 2025-02-27 14:45:01 -- STEP: 6014/36938 -- GLOBAL_STEP: 1742100 | > loss_text_ce: 0.03611278533935547 (0.029720660306186242) | > loss_mel_ce: 2.653419256210327 (nan) | > loss: 0.010672747157514095 (nan) | > current_lr: 5e-06 | > step_time: 0.124 (0.1369227438540725) | > loader_time: 0.0046 (0.004658330862491198)  --> TIME: 2025-02-27 14:45:10 -- STEP: 6064/36938 -- GLOBAL_STEP: 1742150 | > loss_text_ce: 0.03073376603424549 (0.029721109480977958) | > loss_mel_ce: 3.078125 (nan) | > loss: 0.012336742132902145 (nan) | > current_lr: 5e-06 | > step_time: 0.1078 (0.13690507447342157) | > loader_time: 0.0047 (0.004658499540942952)  --> TIME: 2025-02-27 14:45:20 -- STEP: 6114/36938 -- GLOBAL_STEP: 1742200 | > loss_text_ce: 0.03538684919476509 (0.029726735230609598) | > loss_mel_ce: 2.59560227394104 (nan) | > loss: 0.010440433397889137 (nan) | > current_lr: 5e-06 | > step_time: 0.1895 (0.13692073019493817) | > loader_time: 0.0049 (0.004658702584236402)  --> TIME: 2025-02-27 14:45:29 -- STEP: 6164/36938 -- GLOBAL_STEP: 1742250 | > loss_text_ce: 0.04331677779555321 (0.02972844948839229) | > loss_mel_ce: 2.3931992053985596 (nan) | > loss: 0.009668715298175812 (nan) | > current_lr: 5e-06 | > step_time: 0.1326 (0.1369217124409218) | > loader_time: 0.0046 (0.0046586256612051145)  --> TIME: 2025-02-27 14:45:38 -- STEP: 6214/36938 -- GLOBAL_STEP: 1742300 | > loss_text_ce: 0.02456003613770008 (0.029725659028202827) | > loss_mel_ce: 1.9529699087142944 (nan) | > loss: 0.00784734170883894 (nan) | > current_lr: 5e-06 | > step_time: 0.1234 (0.13691836539901764) | > loader_time: 0.0046 (0.00465873540848215)  --> TIME: 2025-02-27 14:45:48 -- STEP: 6264/36938 -- GLOBAL_STEP: 1742350 | > loss_text_ce: 0.03270726278424263 (0.029727387429205495) | > loss_mel_ce: 2.5370216369628906 (nan) | > loss: 0.010197337716817856 (nan) | > current_lr: 5e-06 | > step_time: 0.1606 (0.13692246893204388) | > loader_time: 0.0049 (0.004658800089496301)  --> TIME: 2025-02-27 14:45:57 -- STEP: 6314/36938 -- GLOBAL_STEP: 1742400 | > loss_text_ce: 0.02283002994954586 (0.02971965704915061) | > loss_mel_ce: 1.672916293144226 (nan) | > loss: 0.0067291525192558765 (nan) | > current_lr: 5e-06 | > step_time: 0.1092 (0.13692716378594763) | > loader_time: 0.0044 (0.004658775311464893)  --> TIME: 2025-02-27 14:46:07 -- STEP: 6364/36938 -- GLOBAL_STEP: 1742450 | > loss_text_ce: 0.030943820253014565 (0.0297235492012498) | > loss_mel_ce: 1.8275212049484253 (nan) | > loss: 0.007374861743301153 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13692572430047842) | > loader_time: 0.0047 (0.004658642465554262)  --> TIME: 2025-02-27 14:46:16 -- STEP: 6414/36938 -- GLOBAL_STEP: 1742500 | > loss_text_ce: 0.03256045654416084 (0.02972496696174293) | > loss_mel_ce: 2.773871421813965 (nan) | > loss: 0.011136634275317192 (nan) | > current_lr: 5e-06 | > step_time: 0.1134 (0.13689329786244192) | > loader_time: 0.0047 (0.004658516485965388)  --> TIME: 2025-02-27 14:46:25 -- STEP: 6464/36938 -- GLOBAL_STEP: 1742550 | > loss_text_ce: 0.02290322445333004 (0.029722878267979338) | > loss_mel_ce: 2.5286145210266113 (nan) | > loss: 0.010125070810317993 (nan) | > current_lr: 5e-06 | > step_time: 0.1161 (0.13688291922801818) | > loader_time: 0.0046 (0.004658844653922723)  --> TIME: 2025-02-27 14:46:34 -- STEP: 6514/36938 -- GLOBAL_STEP: 1742600 | > loss_text_ce: 0.02875387854874134 (0.02971910648228069) | > loss_mel_ce: 2.5037500858306885 (nan) | > loss: 0.01004961971193552 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13686492807630196) | > loader_time: 0.0048 (0.004658892105648321)  --> TIME: 2025-02-27 14:46:44 -- STEP: 6564/36938 -- GLOBAL_STEP: 1742650 | > loss_text_ce: 0.019340045750141144 (0.029713637756108826) | > loss_mel_ce: 2.8814563751220703 (nan) | > loss: 0.01151109766215086 (nan) | > current_lr: 5e-06 | > step_time: 0.1624 (0.13687621588244983) | > loader_time: 0.0046 (0.004658905599699878)  --> TIME: 2025-02-27 14:46:53 -- STEP: 6614/36938 -- GLOBAL_STEP: 1742700 | > loss_text_ce: 0.02696303091943264 (0.02972192433007942) | > loss_mel_ce: 2.4265034198760986 (nan) | > loss: 0.009735978208482265 (nan) | > current_lr: 5e-06 | > step_time: 0.1521 (0.13689783306542902) | > loader_time: 0.0046 (0.004659148332464892)  --> TIME: 2025-02-27 14:47:02 -- STEP: 6664/36938 -- GLOBAL_STEP: 1742750 | > loss_text_ce: 0.027205727994441986 (0.029714191752560834) | > loss_mel_ce: 2.6001689434051514 (nan) | > loss: 0.010426090098917484 (nan) | > current_lr: 5e-06 | > step_time: 0.143 (0.13688625032160495) | > loader_time: 0.0046 (0.004659233903255197)  --> TIME: 2025-02-27 14:47:11 -- STEP: 6714/36938 -- GLOBAL_STEP: 1742800 | > loss_text_ce: 0.030076293274760246 (0.02970804559235567) | > loss_mel_ce: 2.23242449760437 (nan) | > loss: 0.008978177793323994 (nan) | > current_lr: 5e-06 | > step_time: 0.1181 (0.13686075869649186) | > loader_time: 0.0045 (0.004658779609334712)  --> TIME: 2025-02-27 14:47:21 -- STEP: 6764/36938 -- GLOBAL_STEP: 1742850 | > loss_text_ce: 0.021199118345975876 (0.029693216789984166) | > loss_mel_ce: 2.126199722290039 (nan) | > loss: 0.008521424606442451 (nan) | > current_lr: 5e-06 | > step_time: 0.1196 (0.13684998413009264) | > loader_time: 0.0044 (0.004658816271036902)  --> TIME: 2025-02-27 14:47:30 -- STEP: 6814/36938 -- GLOBAL_STEP: 1742900 | > loss_text_ce: 0.02876666933298111 (0.029689773392082043) | > loss_mel_ce: 1.2249624729156494 (nan) | > loss: 0.00497511588037014 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1368301503987899) | > loader_time: 0.0046 (0.0046584782867722585)  --> TIME: 2025-02-27 14:47:39 -- STEP: 6864/36938 -- GLOBAL_STEP: 1742950 | > loss_text_ce: 0.02359657920897007 (0.029694905769164587) | > loss_mel_ce: 2.438361644744873 (nan) | > loss: 0.009769676253199577 (nan) | > current_lr: 5e-06 | > step_time: 0.1798 (0.13684227461720438) | > loader_time: 0.0048 (0.004658467807136197)  --> TIME: 2025-02-27 14:47:49 -- STEP: 6914/36938 -- GLOBAL_STEP: 1743000 | > loss_text_ce: 0.03202619403600693 (0.029690321303455766) | > loss_mel_ce: 1.9805501699447632 (nan) | > loss: 0.00798641424626112 (nan) | > current_lr: 5e-06 | > step_time: 0.1095 (0.1368582985943926) | > loader_time: 0.0046 (0.004658550653355552)  --> TIME: 2025-02-27 14:47:58 -- STEP: 6964/36938 -- GLOBAL_STEP: 1743050 | > loss_text_ce: 0.025233620777726173 (0.029688183955649216) | > loss_mel_ce: 2.479074716567993 (nan) | > loss: 0.009937731549143791 (nan) | > current_lr: 5e-06 | > step_time: 0.116 (0.1368588511454933) | > loader_time: 0.0047 (0.004659277964158673)  --> TIME: 2025-02-27 14:48:07 -- STEP: 7014/36938 -- GLOBAL_STEP: 1743100 | > loss_text_ce: 0.026223506778478622 (0.029688207727931553) | > loss_mel_ce: 2.598649740219116 (nan) | > loss: 0.010416164062917233 (nan) | > current_lr: 5e-06 | > step_time: 0.1217 (0.13686060048862897) | > loader_time: 0.0046 (0.0046591641456270975)  --> TIME: 2025-02-27 14:48:17 -- STEP: 7064/36938 -- GLOBAL_STEP: 1743150 | > loss_text_ce: 0.021437937393784523 (0.02969211235480005) | > loss_mel_ce: 2.1717824935913086 (nan) | > loss: 0.008703256025910378 (nan) | > current_lr: 5e-06 | > step_time: 0.127 (0.13687932494677654) | > loader_time: 0.0046 (0.004659155048303381)  --> TIME: 2025-02-27 14:48:26 -- STEP: 7114/36938 -- GLOBAL_STEP: 1743200 | > loss_text_ce: 0.02850690856575966 (0.029696338744831442) | > loss_mel_ce: 3.199986457824707 (nan) | > loss: 0.012811482883989811 (nan) | > current_lr: 5e-06 | > step_time: 0.1778 (0.13687642782901474) | > loader_time: 0.0048 (0.004658730739886266)  --> TIME: 2025-02-27 14:48:36 -- STEP: 7164/36938 -- GLOBAL_STEP: 1743250 | > loss_text_ce: 0.031209750100970268 (0.029690719912393356) | > loss_mel_ce: 2.500345468521118 (nan) | > loss: 0.01004585437476635 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13690417672188757) | > loader_time: 0.0045 (0.0046585405891264925)  --> TIME: 2025-02-27 14:48:45 -- STEP: 7214/36938 -- GLOBAL_STEP: 1743300 | > loss_text_ce: 0.0319095253944397 (0.029690775763638995) | > loss_mel_ce: 2.999499797821045 (nan) | > loss: 0.012029402889311314 (nan) | > current_lr: 5e-06 | > step_time: 0.1909 (0.13691691454011495) | > loader_time: 0.0049 (0.004658636968817295)  --> TIME: 2025-02-27 14:48:55 -- STEP: 7264/36938 -- GLOBAL_STEP: 1743350 | > loss_text_ce: 0.025636231526732445 (0.029687771158772906) | > loss_mel_ce: 2.87197208404541 (nan) | > loss: 0.011498446576297283 (nan) | > current_lr: 5e-06 | > step_time: 0.1263 (0.13694003765934934) | > loader_time: 0.0046 (0.004658479128640111)  --> TIME: 2025-02-27 14:49:04 -- STEP: 7314/36938 -- GLOBAL_STEP: 1743400 | > loss_text_ce: 0.03049362823367119 (0.029684640913855382) | > loss_mel_ce: 2.395667552947998 (nan) | > loss: 0.009627624414861202 (nan) | > current_lr: 5e-06 | > step_time: 0.1271 (0.13694605430945195) | > loader_time: 0.0046 (0.004658826100946857)  --> TIME: 2025-02-27 14:49:13 -- STEP: 7364/36938 -- GLOBAL_STEP: 1743450 | > loss_text_ce: 0.026618828997015953 (0.029682814505048363) | > loss_mel_ce: 1.6191823482513428 (nan) | > loss: 0.006530957296490669 (nan) | > current_lr: 5e-06 | > step_time: 0.166 (0.1369103671803805) | > loader_time: 0.0048 (0.00465904054014918)  --> TIME: 2025-02-27 14:49:22 -- STEP: 7414/36938 -- GLOBAL_STEP: 1743500 | > loss_text_ce: 0.030892960727214813 (0.029688184712326343) | > loss_mel_ce: 0.0869954451918602 (nan) | > loss: 0.0004678111581597477 (nan) | > current_lr: 5e-06 | > step_time: 0.1115 (0.13688494871657675) | > loader_time: 0.0047 (0.004659148217019331)  --> TIME: 2025-02-27 14:49:32 -- STEP: 7464/36938 -- GLOBAL_STEP: 1743550 | > loss_text_ce: 0.032903335988521576 (0.029682040934367073) | > loss_mel_ce: 2.7320473194122314 (nan) | > loss: 0.01097202766686678 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.13690878985906862) | > loader_time: 0.0048 (0.004659347499681805)  --> TIME: 2025-02-27 14:49:41 -- STEP: 7514/36938 -- GLOBAL_STEP: 1743600 | > loss_text_ce: 0.03255610540509224 (0.02967644497811187) | > loss_mel_ce: 2.3544921875 (nan) | > loss: 0.009472413919866085 (nan) | > current_lr: 5e-06 | > step_time: 0.1356 (0.13691019066350413) | > loader_time: 0.0046 (0.004659021950986478)  --> TIME: 2025-02-27 14:49:51 -- STEP: 7564/36938 -- GLOBAL_STEP: 1743650 | > loss_text_ce: 0.02361532673239708 (0.029675830333035128) | > loss_mel_ce: 2.8035786151885986 (nan) | > loss: 0.011219024658203125 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13691726536551602) | > loader_time: 0.0047 (0.00465928735965019)  --> TIME: 2025-02-27 14:50:00 -- STEP: 7614/36938 -- GLOBAL_STEP: 1743700 | > loss_text_ce: 0.031050898134708405 (0.029676472496398872) | > loss_mel_ce: 2.437973737716675 (nan) | > loss: 0.009797717444598675 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.13689548314197014) | > loader_time: 0.0046 (0.004659263674969227)  --> TIME: 2025-02-27 14:50:09 -- STEP: 7664/36938 -- GLOBAL_STEP: 1743750 | > loss_text_ce: 0.03539133816957474 (0.029673933927679436) | > loss_mel_ce: 0.0177001953125 (nan) | > loss: 0.00021068070782348514 (nan) | > current_lr: 5e-06 | > step_time: 0.1391 (0.13691306139083909) | > loader_time: 0.0046 (0.0046596914094275705)  --> TIME: 2025-02-27 14:50:18 -- STEP: 7714/36938 -- GLOBAL_STEP: 1743800 | > loss_text_ce: 0.031574781984090805 (0.029681196273781407) | > loss_mel_ce: 2.255544424057007 (nan) | > loss: 0.009075869806110859 (nan) | > current_lr: 5e-06 | > step_time: 0.1282 (0.13689306844291474) | > loader_time: 0.0046 (0.00465934413190884)  --> TIME: 2025-02-27 14:50:27 -- STEP: 7764/36938 -- GLOBAL_STEP: 1743850 | > loss_text_ce: 0.02974860556423664 (0.029679112194421698) | > loss_mel_ce: 2.80503249168396 (nan) | > loss: 0.011249132454395294 (nan) | > current_lr: 5e-06 | > step_time: 0.1825 (0.13687487722732936) | > loader_time: 0.0048 (0.004658955940568879)  --> TIME: 2025-02-27 14:50:37 -- STEP: 7814/36938 -- GLOBAL_STEP: 1743900 | > loss_text_ce: 0.03252700716257095 (0.029677278411438548) | > loss_mel_ce: 2.71107816696167 (nan) | > loss: 0.010887322947382927 (nan) | > current_lr: 5e-06 | > step_time: 0.1445 (0.13684966672800203) | > loader_time: 0.0046 (0.004658595722961019)  --> TIME: 2025-02-27 14:50:46 -- STEP: 7864/36938 -- GLOBAL_STEP: 1743950 | > loss_text_ce: 0.025403767824172974 (0.02966825016712654) | > loss_mel_ce: 1.651096224784851 (nan) | > loss: 0.006652778014540672 (nan) | > current_lr: 5e-06 | > step_time: 0.1525 (0.136846804333849) | > loader_time: 0.0048 (0.004658676721799879)  --> TIME: 2025-02-27 14:50:55 -- STEP: 7914/36938 -- GLOBAL_STEP: 1744000 | > loss_text_ce: 0.027034420520067215 (0.029672784083314557) | > loss_mel_ce: 2.6483373641967773 (nan) | > loss: 0.01061655580997467 (nan) | > current_lr: 5e-06 | > step_time: 0.1555 (0.13682036737857048) | > loader_time: 0.0048 (0.004658645983443631)  --> TIME: 2025-02-27 14:51:04 -- STEP: 7964/36938 -- GLOBAL_STEP: 1744050 | > loss_text_ce: 0.03160347789525986 (0.02966936410693353) | > loss_mel_ce: 2.2444310188293457 (nan) | > loss: 0.00903188344091177 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.1368293101795951) | > loader_time: 0.0046 (0.0046587174469235396)  --> TIME: 2025-02-27 14:51:14 -- STEP: 8014/36938 -- GLOBAL_STEP: 1744100 | > loss_text_ce: 0.027259591966867447 (0.029676013524356566) | > loss_mel_ce: 2.202073335647583 (nan) | > loss: 0.008846559561789036 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13682736920870134) | > loader_time: 0.0045 (0.004659204462802034)  --> TIME: 2025-02-27 14:51:23 -- STEP: 8064/36938 -- GLOBAL_STEP: 1744150 | > loss_text_ce: 0.036676667630672455 (0.02967381849088096) | > loss_mel_ce: 2.485976219177246 (nan) | > loss: 0.010010527446866035 (nan) | > current_lr: 5e-06 | > step_time: 0.1328 (0.13682023119477066) | > loader_time: 0.0048 (0.004659721273041891)  --> TIME: 2025-02-27 14:51:32 -- STEP: 8114/36938 -- GLOBAL_STEP: 1744200 | > loss_text_ce: 0.029373427852988243 (0.029668578415827183) | > loss_mel_ce: 2.539741277694702 (nan) | > loss: 0.010194900445640087 (nan) | > current_lr: 5e-06 | > step_time: 0.1884 (0.13680472334588384) | > loader_time: 0.0049 (0.00465970733408448)  --> TIME: 2025-02-27 14:51:42 -- STEP: 8164/36938 -- GLOBAL_STEP: 1744250 | > loss_text_ce: 0.027248971164226532 (0.029666972934669505) | > loss_mel_ce: 2.646160840988159 (nan) | > loss: 0.01060876902192831 (nan) | > current_lr: 5e-06 | > step_time: 0.1362 (0.136831879352951) | > loader_time: 0.0045 (0.004659483445619365)  --> TIME: 2025-02-27 14:51:51 -- STEP: 8214/36938 -- GLOBAL_STEP: 1744300 | > loss_text_ce: 0.04192107170820236 (0.029668151628995224) | > loss_mel_ce: 2.5235512256622314 (nan) | > loss: 0.010180446319282055 (nan) | > current_lr: 5e-06 | > step_time: 0.1155 (0.13682397276286476) | > loader_time: 0.0045 (0.004659190182561606)  --> TIME: 2025-02-27 14:52:00 -- STEP: 8264/36938 -- GLOBAL_STEP: 1744350 | > loss_text_ce: 0.02487807720899582 (0.029663181208395954) | > loss_mel_ce: 2.154402017593384 (nan) | > loss: 0.008647937327623367 (nan) | > current_lr: 5e-06 | > step_time: 0.1314 (0.13681182597059724) | > loader_time: 0.0045 (0.00465871496099007)  --> TIME: 2025-02-27 14:52:09 -- STEP: 8314/36938 -- GLOBAL_STEP: 1744400 | > loss_text_ce: 0.021492067724466324 (0.029662577566772062) | > loss_mel_ce: 1.0820960998535156 (nan) | > loss: 0.0043793185614049435 (nan) | > current_lr: 5e-06 | > step_time: 0.1166 (0.13679778544352794) | > loader_time: 0.0047 (0.004658402030445604)  --> TIME: 2025-02-27 14:52:19 -- STEP: 8364/36938 -- GLOBAL_STEP: 1744450 | > loss_text_ce: 0.03418537601828575 (0.029662052724516663) | > loss_mel_ce: 2.0119986534118652 (nan) | > loss: 0.008119778707623482 (nan) | > current_lr: 5e-06 | > step_time: 0.1169 (0.13678554807879162) | > loader_time: 0.0045 (0.004658111597806384)  --> TIME: 2025-02-27 14:52:28 -- STEP: 8414/36938 -- GLOBAL_STEP: 1744500 | > loss_text_ce: 0.03083105757832527 (0.029663286815499935) | > loss_mel_ce: 2.1698155403137207 (nan) | > loss: 0.00873272493481636 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13680551498101176) | > loader_time: 0.0044 (0.004657879475320845)  --> TIME: 2025-02-27 14:52:38 -- STEP: 8464/36938 -- GLOBAL_STEP: 1744550 | > loss_text_ce: 0.022343620657920837 (0.029657726913086695) | > loss_mel_ce: 2.9888031482696533 (nan) | > loss: 0.011948996223509312 (nan) | > current_lr: 5e-06 | > step_time: 0.1299 (0.13680653986723537) | > loader_time: 0.0046 (0.004657610715674977)  --> TIME: 2025-02-27 14:52:47 -- STEP: 8514/36938 -- GLOBAL_STEP: 1744600 | > loss_text_ce: 0.024245265871286392 (0.02965858857575536) | > loss_mel_ce: 2.1837856769561768 (nan) | > loss: 0.008762028068304062 (nan) | > current_lr: 5e-06 | > step_time: 0.1509 (0.13684156861192112) | > loader_time: 0.0046 (0.0046574959935283804)  --> TIME: 2025-02-27 14:52:57 -- STEP: 8564/36938 -- GLOBAL_STEP: 1744650 | > loss_text_ce: 0.026730090379714966 (0.02966295246801319) | > loss_mel_ce: 2.5618553161621094 (nan) | > loss: 0.010272164829075336 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13684044255657996) | > loader_time: 0.0045 (0.004657206052043388)  --> TIME: 2025-02-27 14:53:06 -- STEP: 8614/36938 -- GLOBAL_STEP: 1744700 | > loss_text_ce: 0.0338975265622139 (0.029659157640503725) | > loss_mel_ce: 3.072993755340576 (nan) | > loss: 0.012328934855759144 (nan) | > current_lr: 5e-06 | > step_time: 0.153 (0.13687407250799608) | > loader_time: 0.0047 (0.004657146630111125)  --> TIME: 2025-02-27 14:53:15 -- STEP: 8664/36938 -- GLOBAL_STEP: 1744750 | > loss_text_ce: 0.026324424892663956 (0.029652341105244243) | > loss_mel_ce: 2.444108724594116 (nan) | > loss: 0.009803307242691517 (nan) | > current_lr: 5e-06 | > step_time: 0.1177 (0.13687141331829417) | > loader_time: 0.0045 (0.004657193811976664)  --> TIME: 2025-02-27 14:53:24 -- STEP: 8714/36938 -- GLOBAL_STEP: 1744800 | > loss_text_ce: 0.036846186965703964 (0.02965368526629266) | > loss_mel_ce: 2.8441731929779053 (nan) | > loss: 0.011432616971433163 (nan) | > current_lr: 5e-06 | > step_time: 0.1681 (0.13684727879109015) | > loader_time: 0.0048 (0.0046570617615807105)  --> TIME: 2025-02-27 14:53:34 -- STEP: 8764/36938 -- GLOBAL_STEP: 1744850 | > loss_text_ce: 0.02349255047738552 (0.029654116603464644) | > loss_mel_ce: 2.5621097087860107 (nan) | > loss: 0.010260326787829399 (nan) | > current_lr: 5e-06 | > step_time: 0.1235 (0.13682770331882013) | > loader_time: 0.0048 (0.0046572802764321365)  --> TIME: 2025-02-27 14:53:43 -- STEP: 8814/36938 -- GLOBAL_STEP: 1744900 | > loss_text_ce: 0.022741854190826416 (0.029658289586828042) | > loss_mel_ce: 2.7324206829071045 (nan) | > loss: 0.010933184996247292 (nan) | > current_lr: 5e-06 | > step_time: 0.2003 (0.1368398249541548) | > loader_time: 0.0049 (0.0046572905158087565)  --> TIME: 2025-02-27 14:53:53 -- STEP: 8864/36938 -- GLOBAL_STEP: 1744950 | > loss_text_ce: 0.028826985508203506 (0.029652445811071397) | > loss_mel_ce: 1.370758056640625 (nan) | > loss: 0.005553909111768007 (nan) | > current_lr: 5e-06 | > step_time: 0.1413 (0.13685767788318956) | > loader_time: 0.0046 (0.004657960648140738)  --> TIME: 2025-02-27 14:54:02 -- STEP: 8914/36938 -- GLOBAL_STEP: 1745000 | > loss_text_ce: 0.026153311133384705 (0.029652702042781556) | > loss_mel_ce: 2.707822322845459 (nan) | > loss: 0.010849110782146454 (nan) | > current_lr: 5e-06 | > step_time: 0.1275 (0.13688377922683762) | > loader_time: 0.0046 (0.004658011970751096)  --> TIME: 2025-02-27 14:54:12 -- STEP: 8964/36938 -- GLOBAL_STEP: 1745050 | > loss_text_ce: 0.022467555478215218 (0.029662774293423715) | > loss_mel_ce: 2.168715476989746 (nan) | > loss: 0.00869517121464014 (nan) | > current_lr: 5e-06 | > step_time: 0.1691 (0.1368795810986288) | > loader_time: 0.0048 (0.004658214006504792)  --> TIME: 2025-02-27 14:54:21 -- STEP: 9014/36938 -- GLOBAL_STEP: 1745100 | > loss_text_ce: 0.026038791984319687 (0.02966375930159572) | > loss_mel_ce: 2.5785393714904785 (nan) | > loss: 0.010335628874599934 (nan) | > current_lr: 5e-06 | > step_time: 0.1192 (0.13689206192438944) | > loader_time: 0.0045 (0.004658163532809757)  --> TIME: 2025-02-27 14:54:30 -- STEP: 9064/36938 -- GLOBAL_STEP: 1745150 | > loss_text_ce: 0.028063230216503143 (0.0296711026883453) | > loss_mel_ce: 0.0007863309583626688 (nan) | > loss: 0.00011448239092715085 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13689247397122284) | > loader_time: 0.0045 (0.004658173694088707)  --> TIME: 2025-02-27 14:54:40 -- STEP: 9114/36938 -- GLOBAL_STEP: 1745200 | > loss_text_ce: 0.02882326953113079 (0.029668857528962346) | > loss_mel_ce: 2.844453811645508 (nan) | > loss: 0.011401894502341747 (nan) | > current_lr: 5e-06 | > step_time: 0.1806 (0.13689968681188944) | > loader_time: 0.0048 (0.0046580906157067566)  --> TIME: 2025-02-27 14:54:49 -- STEP: 9164/36938 -- GLOBAL_STEP: 1745250 | > loss_text_ce: 0.026097271591424942 (0.02966429073861927) | > loss_mel_ce: 2.6294872760772705 (nan) | > loss: 0.010538035072386265 (nan) | > current_lr: 5e-06 | > step_time: 0.121 (0.13688047857900623) | > loader_time: 0.0046 (0.004658211557587269)  --> TIME: 2025-02-27 14:54:58 -- STEP: 9214/36938 -- GLOBAL_STEP: 1745300 | > loss_text_ce: 0.024339444935321808 (0.02966323583737574) | > loss_mel_ce: 2.0861196517944336 (nan) | > loss: 0.008374838158488274 (nan) | > current_lr: 5e-06 | > step_time: 0.1344 (0.1368700620898824) | > loader_time: 0.0048 (0.004658169696717595)  --> TIME: 2025-02-27 14:55:07 -- STEP: 9264/36938 -- GLOBAL_STEP: 1745350 | > loss_text_ce: 0.029748694971203804 (0.029661664655407965) | > loss_mel_ce: 1.4443917274475098 (nan) | > loss: 0.005849763751029968 (nan) | > current_lr: 5e-06 | > step_time: 0.1163 (0.1368623208351086) | > loader_time: 0.0045 (0.0046581823076402885)  --> TIME: 2025-02-27 14:55:17 -- STEP: 9314/36938 -- GLOBAL_STEP: 1745400 | > loss_text_ce: 0.03283693641424179 (0.02966933205882948) | > loss_mel_ce: 2.795987844467163 (nan) | > loss: 0.011225495487451553 (nan) | > current_lr: 5e-06 | > step_time: 0.1189 (0.13685035091217498) | > loader_time: 0.0046 (0.004657983677515725)  --> TIME: 2025-02-27 14:55:26 -- STEP: 9364/36938 -- GLOBAL_STEP: 1745450 | > loss_text_ce: 0.02758224867284298 (0.02966581326612978) | > loss_mel_ce: 2.8252010345458984 (nan) | > loss: 0.0113205686211586 (nan) | > current_lr: 5e-06 | > step_time: 0.1357 (0.13685239565672933) | > loader_time: 0.0046 (0.004657950553054831)  --> TIME: 2025-02-27 14:55:35 -- STEP: 9414/36938 -- GLOBAL_STEP: 1745500 | > loss_text_ce: 0.03323296085000038 (0.02966488206916194) | > loss_mel_ce: 2.593256711959839 (nan) | > loss: 0.01042257808148861 (nan) | > current_lr: 5e-06 | > step_time: 0.1544 (0.13681285417280026) | > loader_time: 0.0047 (0.004657645804982838)  --> TIME: 2025-02-27 14:55:44 -- STEP: 9464/36938 -- GLOBAL_STEP: 1745550 | > loss_text_ce: 0.02611345238983631 (0.029661372927271782) | > loss_mel_ce: 0.01601240411400795 (nan) | > loss: 0.00016716611571609974 (nan) | > current_lr: 5e-06 | > step_time: 0.134 (0.13679110731800453) | > loader_time: 0.0046 (0.004657560224162245)  --> TIME: 2025-02-27 14:55:53 -- STEP: 9514/36938 -- GLOBAL_STEP: 1745600 | > loss_text_ce: 0.02301979809999466 (0.029662620323003954) | > loss_mel_ce: 3.055199146270752 (nan) | > loss: 0.01221515517681837 (nan) | > current_lr: 5e-06 | > step_time: 0.1781 (0.1368049513223621) | > loader_time: 0.0049 (0.004657776736150586)  --> TIME: 2025-02-27 14:56:02 -- STEP: 9564/36938 -- GLOBAL_STEP: 1745650 | > loss_text_ce: 0.03275826573371887 (0.029664364083082293) | > loss_mel_ce: 2.7726292610168457 (nan) | > loss: 0.011132490821182728 (nan) | > current_lr: 5e-06 | > step_time: 0.1185 (0.1367839228553487) | > loader_time: 0.0045 (0.004657623783807166)  --> TIME: 2025-02-27 14:56:12 -- STEP: 9614/36938 -- GLOBAL_STEP: 1745700 | > loss_text_ce: 0.03713660314679146 (0.029666551876644992) | > loss_mel_ce: 4.05859375 (nan) | > loss: 0.016252899542450905 (nan) | > current_lr: 5e-06 | > step_time: 0.1182 (0.1367961174240971) | > loader_time: 0.0045 (0.004657342450296832)  --> TIME: 2025-02-27 14:56:21 -- STEP: 9664/36938 -- GLOBAL_STEP: 1745750 | > loss_text_ce: 0.032008178532123566 (0.02965886563364113) | > loss_mel_ce: 2.4454431533813477 (nan) | > loss: 0.009831156581640244 (nan) | > current_lr: 5e-06 | > step_time: 0.1288 (0.13680793830989219) | > loader_time: 0.0048 (0.004657070269647795)  --> TIME: 2025-02-27 14:56:31 -- STEP: 9714/36938 -- GLOBAL_STEP: 1745800 | > loss_text_ce: 0.03147677704691887 (0.029659835153464688) | > loss_mel_ce: 3.288731098175049 (nan) | > loss: 0.013175428844988346 (nan) | > current_lr: 5e-06 | > step_time: 0.1611 (0.1368235069610269) | > loader_time: 0.0046 (0.004656861440522284)  --> TIME: 2025-02-27 14:56:40 -- STEP: 9764/36938 -- GLOBAL_STEP: 1745850 | > loss_text_ce: 0.024260053411126137 (0.029658579434196055) | > loss_mel_ce: 2.1722514629364014 (nan) | > loss: 0.00871631596237421 (nan) | > current_lr: 5e-06 | > step_time: 0.1338 (0.13683253731799888) | > loader_time: 0.0046 (0.004657839981565097)  --> TIME: 2025-02-27 14:56:50 -- STEP: 9814/36938 -- GLOBAL_STEP: 1745900 | > loss_text_ce: 0.019778406247496605 (0.029666103041439094) | > loss_mel_ce: 2.355729341506958 (nan) | > loss: 0.009426618926227093 (nan) | > current_lr: 5e-06 | > step_time: 0.1122 (0.13684412331111767) | > loader_time: 0.0046 (0.004657970734178962)  --> TIME: 2025-02-27 14:56:59 -- STEP: 9864/36938 -- GLOBAL_STEP: 1745950 | > loss_text_ce: 0.037569526582956314 (0.029661972126591146) | > loss_mel_ce: 0.8796533346176147 (nan) | > loss: 0.003639773465692997 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13685026532832142) | > loader_time: 0.0046 (0.004657984456683382)  --> TIME: 2025-02-27 14:57:09 -- STEP: 9914/36938 -- GLOBAL_STEP: 1746000 | > loss_text_ce: 0.028333555907011032 (0.0296584317925284) | > loss_mel_ce: 2.6813628673553467 (nan) | > loss: 0.010752764530479908 (nan) | > current_lr: 5e-06 | > step_time: 0.1517 (0.1368689385541677) | > loader_time: 0.0047 (0.00465817984876694)  --> TIME: 2025-02-27 14:57:18 -- STEP: 9964/36938 -- GLOBAL_STEP: 1746050 | > loss_text_ce: 0.03128054365515709 (0.029655612664654635) | > loss_mel_ce: 3.287468194961548 (nan) | > loss: 0.013169638812541962 (nan) | > current_lr: 5e-06 | > step_time: 0.1447 (0.13687295934752927) | > loader_time: 0.0045 (0.0046579769125715785)  --> TIME: 2025-02-27 14:57:28 -- STEP: 10014/36938 -- GLOBAL_STEP: 1746100 | > loss_text_ce: 0.02051832713186741 (0.029655434215686487) | > loss_mel_ce: 1.6288573741912842 (nan) | > loss: 0.00654514180496335 (nan) | > current_lr: 5e-06 | > step_time: 0.1229 (0.1368709577636993) | > loader_time: 0.0045 (0.0046578719988777635)  --> TIME: 2025-02-27 14:57:37 -- STEP: 10064/36938 -- GLOBAL_STEP: 1746150 | > loss_text_ce: 0.032992955297231674 (0.02965567519398559) | > loss_mel_ce: 2.2674944400787354 (nan) | > loss: 0.009128917939960957 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13686460479264798) | > loader_time: 0.0046 (0.004657856468555452)  --> TIME: 2025-02-27 14:57:46 -- STEP: 10114/36938 -- GLOBAL_STEP: 1746200 | > loss_text_ce: 0.03308754786849022 (0.029659378426217617) | > loss_mel_ce: 2.1887013912200928 (nan) | > loss: 0.008816623128950596 (nan) | > current_lr: 5e-06 | > step_time: 0.1117 (0.13685202329838164) | > loader_time: 0.0046 (0.004657644326917385)  --> TIME: 2025-02-27 14:57:56 -- STEP: 10164/36938 -- GLOBAL_STEP: 1746250 | > loss_text_ce: 0.026405803859233856 (0.029664547402988586) | > loss_mel_ce: 2.7645909786224365 (nan) | > loss: 0.01107538491487503 (nan) | > current_lr: 5e-06 | > step_time: 0.138 (0.13686416440946492) | > loader_time: 0.0046 (0.004657599575075732)  --> TIME: 2025-02-27 14:58:05 -- STEP: 10214/36938 -- GLOBAL_STEP: 1746300 | > loss_text_ce: 0.03436185047030449 (0.029671393317814716) | > loss_mel_ce: 2.7512784004211426 (nan) | > loss: 0.011054128408432007 (nan) | > current_lr: 5e-06 | > step_time: 0.1303 (0.13685127641675257) | > loader_time: 0.0046 (0.00465744074389544)  --> TIME: 2025-02-27 14:58:14 -- STEP: 10264/36938 -- GLOBAL_STEP: 1746350 | > loss_text_ce: 0.022958839312195778 (0.02967404776453515) | > loss_mel_ce: 2.692603588104248 (nan) | > loss: 0.010776042006909847 (nan) | > current_lr: 5e-06 | > step_time: 0.1178 (0.13684511934976895) | > loader_time: 0.0046 (0.0046572154235393695)  --> TIME: 2025-02-27 14:58:23 -- STEP: 10314/36938 -- GLOBAL_STEP: 1746400 | > loss_text_ce: 0.02459164895117283 (0.029677386736684727) | > loss_mel_ce: 2.4793784618377686 (nan) | > loss: 0.009936390444636345 (nan) | > current_lr: 5e-06 | > step_time: 0.1237 (0.1368371086433136) | > loader_time: 0.0047 (0.004657107359313058)  --> TIME: 2025-02-27 14:58:33 -- STEP: 10364/36938 -- GLOBAL_STEP: 1746450 | > loss_text_ce: 0.03464483842253685 (0.029678676425041177) | > loss_mel_ce: 2.7456321716308594 (nan) | > loss: 0.011032845824956894 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.13685390920263626) | > loader_time: 0.0045 (0.0046570268619571045)  --> TIME: 2025-02-27 14:58:42 -- STEP: 10414/36938 -- GLOBAL_STEP: 1746500 | > loss_text_ce: 0.034088101238012314 (0.02968204988280086) | > loss_mel_ce: 2.0817089080810547 (nan) | > loss: 0.008396020159125328 (nan) | > current_lr: 5e-06 | > step_time: 0.1399 (0.13686934052260952) | > loader_time: 0.0046 (0.004657062271723286)  --> TIME: 2025-02-27 14:58:52 -- STEP: 10464/36938 -- GLOBAL_STEP: 1746550 | > loss_text_ce: 0.03518242388963699 (0.0296799585544319) | > loss_mel_ce: 1.246604323387146 (nan) | > loss: 0.005086455959826708 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13688777820050554) | > loader_time: 0.0046 (0.004657092148193148)  --> TIME: 2025-02-27 14:59:01 -- STEP: 10514/36938 -- GLOBAL_STEP: 1746600 | > loss_text_ce: 0.02443227916955948 (0.029678043174251423) | > loss_mel_ce: 2.2806193828582764 (nan) | > loss: 0.009147030301392078 (nan) | > current_lr: 5e-06 | > step_time: 0.1301 (0.13687722599504126) | > loader_time: 0.0046 (0.004656956226217521)  --> TIME: 2025-02-27 14:59:10 -- STEP: 10564/36938 -- GLOBAL_STEP: 1746650 | > loss_text_ce: 0.031602419912815094 (0.029679471907314833) | > loss_mel_ce: 2.447953224182129 (nan) | > loss: 0.009839506819844246 (nan) | > current_lr: 5e-06 | > step_time: 0.117 (0.13688042594013422) | > loader_time: 0.0046 (0.004656660380755248)  --> TIME: 2025-02-27 14:59:20 -- STEP: 10614/36938 -- GLOBAL_STEP: 1746700 | > loss_text_ce: 0.028031231835484505 (0.029680194781834486) | > loss_mel_ce: 3.1864163875579834 (nan) | > loss: 0.012755746021866798 (nan) | > current_lr: 5e-06 | > step_time: 0.1566 (0.13689931898618446) | > loader_time: 0.0046 (0.004656466899539004)  --> TIME: 2025-02-27 14:59:29 -- STEP: 10664/36938 -- GLOBAL_STEP: 1746750 | > loss_text_ce: 0.03554574027657509 (0.029680201769715607) | > loss_mel_ce: 3.7239582538604736 (nan) | > loss: 0.014918668195605278 (nan) | > current_lr: 5e-06 | > step_time: 0.1221 (0.13689676904177558) | > loader_time: 0.0046 (0.004656247710788136)  --> TIME: 2025-02-27 14:59:39 -- STEP: 10714/36938 -- GLOBAL_STEP: 1746800 | > loss_text_ce: 0.028150102123618126 (0.0296778789155021) | > loss_mel_ce: nan (nan) | > loss: nan (nan) | > current_lr: 5e-06 | > step_time: 0.1222 (0.1368818130425518) | > loader_time: 0.0046 (0.004656154984343675)  --> TIME: 2025-02-27 14:59:48 -- STEP: 10764/36938 -- GLOBAL_STEP: 1746850 | > loss_text_ce: 0.02868412621319294 (0.029681730255002124) | > loss_mel_ce: 2.275852680206299 (nan) | > loss: 0.009144987910985947 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.1368766124476206) | > loader_time: 0.0046 (0.004656332436656903)  --> TIME: 2025-02-27 14:59:57 -- STEP: 10814/36938 -- GLOBAL_STEP: 1746900 | > loss_text_ce: 0.03340732306241989 (0.029675766599433637) | > loss_mel_ce: 2.618049383163452 (nan) | > loss: 0.010521654039621353 (nan) | > current_lr: 5e-06 | > step_time: 0.1167 (0.13685683807692675) | > loader_time: 0.0045 (0.004656545375529767)  --> TIME: 2025-02-27 15:00:07 -- STEP: 10864/36938 -- GLOBAL_STEP: 1746950 | > loss_text_ce: 0.023559603840112686 (0.029670680099594175) | > loss_mel_ce: 1.3181356191635132 (nan) | > loss: 0.005324187688529491 (nan) | > current_lr: 5e-06 | > step_time: 0.1121 (0.1368871724921582) | > loader_time: 0.0046 (0.004657243901043459)  --> TIME: 2025-02-27 15:00:16 -- STEP: 10914/36938 -- GLOBAL_STEP: 1747000 | > loss_text_ce: 0.031272388994693756 (0.029668818165056447) | > loss_mel_ce: 2.379753828048706 (nan) | > loss: 0.0095675652846694 (nan) | > current_lr: 5e-06 | > step_time: 0.1194 (0.1368806938428639) | > loader_time: 0.0047 (0.004658112666623055)  --> TIME: 2025-02-27 15:00:25 -- STEP: 10964/36938 -- GLOBAL_STEP: 1747050 | > loss_text_ce: 0.030793584883213043 (0.029664833885568277) | > loss_mel_ce: 2.2865259647369385 (nan) | > loss: 0.009195713326334953 (nan) | > current_lr: 5e-06 | > step_time: 0.1442 (0.13691567730964543) | > loader_time: 0.0049 (0.004659103438263595)  --> TIME: 2025-02-27 15:00:35 -- STEP: 11014/36938 -- GLOBAL_STEP: 1747100 | > loss_text_ce: 0.029970724135637283 (0.02966987960015539) | > loss_mel_ce: 0.01012420654296875 (nan) | > loss: 0.000159106872160919 (nan) | > current_lr: 5e-06 | > step_time: 0.1371 (0.13692999093918345) | > loader_time: 0.0047 (0.0046592304229043455)  --> TIME: 2025-02-27 15:00:45 -- STEP: 11064/36938 -- GLOBAL_STEP: 1747150 | > loss_text_ce: 0.029416045174002647 (0.029669958949109242) | > loss_mel_ce: 0.6681801080703735 (nan) | > loss: 0.0027682387735694647 (nan) | > current_lr: 5e-06 | > step_time: 0.1104 (0.13694565971089057) | > loader_time: 0.0047 (0.004660287932563486)  --> TIME: 2025-02-27 15:00:54 -- STEP: 11114/36938 -- GLOBAL_STEP: 1747200 | > loss_text_ce: 0.02437758632004261 (0.029668853548820467) | > loss_mel_ce: 2.7890467643737793 (nan) | > loss: 0.01116438303142786 (nan) | > current_lr: 5e-06 | > step_time: 0.1507 (0.13693929806188107) | > loader_time: 0.0047 (0.004660709590304327)  --> TIME: 2025-02-27 15:01:03 -- STEP: 11164/36938 -- GLOBAL_STEP: 1747250 | > loss_text_ce: 0.028953734785318375 (0.029673785325613586) | > loss_mel_ce: 2.79496431350708 (nan) | > loss: 0.011206025257706642 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.13693819164332874) | > loader_time: 0.0048 (0.004660990151911374)  --> TIME: 2025-02-27 15:01:12 -- STEP: 11214/36938 -- GLOBAL_STEP: 1747300 | > loss_text_ce: 0.033430036157369614 (0.029672236446143703) | > loss_mel_ce: 3.2604832649230957 (nan) | > loss: 0.013071085326373577 (nan) | > current_lr: 5e-06 | > step_time: 0.1906 (0.1369359486452163) | > loader_time: 0.0051 (0.00466141529382603)  --> TIME: 2025-02-27 15:01:22 -- STEP: 11264/36938 -- GLOBAL_STEP: 1747350 | > loss_text_ce: 0.03472362831234932 (0.029673146204583056) | > loss_mel_ce: 2.127427101135254 (nan) | > loss: 0.008579962886869907 (nan) | > current_lr: 5e-06 | > step_time: 0.1168 (0.13694860993630528) | > loader_time: 0.0046 (0.00466203115965154)  --> TIME: 2025-02-27 15:01:31 -- STEP: 11314/36938 -- GLOBAL_STEP: 1747400 | > loss_text_ce: 0.022748591378331184 (0.02966632052888246) | > loss_mel_ce: 2.3244564533233643 (nan) | > loss: 0.009314307011663914 (nan) | > current_lr: 5e-06 | > step_time: 0.1308 (0.13696498269851046) | > loader_time: 0.0046 (0.004662612396140422)  --> TIME: 2025-02-27 15:01:41 -- STEP: 11364/36938 -- GLOBAL_STEP: 1747450 | > loss_text_ce: 0.0399734228849411 (0.029668345548548858) | > loss_mel_ce: 2.2381083965301514 (nan) | > loss: 0.009040008299052715 (nan) | > current_lr: 5e-06 | > step_time: 0.1619 (0.13695074533417553) | > loader_time: 0.0048 (0.004662535069736519)  --> TIME: 2025-02-27 15:01:50 -- STEP: 11414/36938 -- GLOBAL_STEP: 1747500 | > loss_text_ce: 0.02176816016435623 (0.029665248166209666) | > loss_mel_ce: 2.417238712310791 (nan) | > loss: 0.009678599424660206 (nan) | > current_lr: 5e-06 | > step_time: 0.1159 (0.13693756164843565) | > loader_time: 0.0046 (0.004662824591968682)  --> TIME: 2025-02-27 15:02:00 -- STEP: 11464/36938 -- GLOBAL_STEP: 1747550 | > loss_text_ce: 0.03055933490395546 (0.029671610753436675) | > loss_mel_ce: 2.7695722579956055 (nan) | > loss: 0.011111633852124214 (nan) | > current_lr: 5e-06 | > step_time: 0.1516 (0.13696895449292973) | > loader_time: 0.0049 (0.004663003277129741)  --> TIME: 2025-02-27 15:02:09 -- STEP: 11514/36938 -- GLOBAL_STEP: 1747600 | > loss_text_ce: 0.034464169293642044 (0.02967227586985555) | > loss_mel_ce: 3.34086537361145 (nan) | > loss: 0.01339416578412056 (nan) | > current_lr: 5e-06 | > step_time: 0.1662 (0.1369659068457302) | > loader_time: 0.0051 (0.004663509442115865)  --> TIME: 2025-02-27 15:02:18 -- STEP: 11564/36938 -- GLOBAL_STEP: 1747650 | > loss_text_ce: 0.023223571479320526 (0.029666838632714198) | > loss_mel_ce: 0.7177047729492188 (nan) | > loss: 0.0029401921201497316 (nan) | > current_lr: 5e-06 | > step_time: 0.1307 (0.13696322289357665) | > loader_time: 0.0046 (0.004663632077685842)  --> TIME: 2025-02-27 15:02:28 -- STEP: 11614/36938 -- GLOBAL_STEP: 1747700 | > loss_text_ce: 0.012346024625003338 (0.029661964798169415) | > loss_mel_ce: 1.763671875 (nan) | > loss: 0.007047690451145172 (nan) | > current_lr: 5e-06 | > step_time: 0.1111 (0.13696211380414136) | > loader_time: 0.0065 (0.004663965573548144)  --> TIME: 2025-02-27 15:02:37 -- STEP: 11664/36938 -- GLOBAL_STEP: 1747750 | > loss_text_ce: 0.018004873767495155 (0.029660235340349284) | > loss_mel_ce: 1.7852492332458496 (nan) | > loss: 0.007155770901590586 (nan) | > current_lr: 5e-06 | > step_time: 0.1202 (0.13696104970815562) | > loader_time: 0.0047 (0.004664067255616662)  --> TIME: 2025-02-27 15:02:46 -- STEP: 11714/36938 -- GLOBAL_STEP: 1747800 | > loss_text_ce: 0.026851974427700043 (0.029654342380640276) | > loss_mel_ce: 3.1535518169403076 (nan) | > loss: 0.012620650231838226 (nan) | > current_lr: 5e-06 | > step_time: 0.1662 (0.1369676557400535) | > loader_time: 0.0047 (0.004664183253209606)  --> TIME: 2025-02-27 15:02:55 -- STEP: 11764/36938 -- GLOBAL_STEP: 1747850 | > loss_text_ce: 0.030115162953734398 (0.029650473660975937) | > loss_mel_ce: 2.176041841506958 (nan) | > loss: 0.008754591457545757 (nan) | > current_lr: 5e-06 | > step_time: 0.151 (0.13694921234196328) | > loader_time: 0.0047 (0.004664313708061813)  --> TIME: 2025-02-27 15:03:05 -- STEP: 11814/36938 -- GLOBAL_STEP: 1747900 | > loss_text_ce: 0.03870643302798271 (0.02965380246693462) | > loss_mel_ce: 2.795527458190918 (nan) | > loss: 0.011246960610151291 (nan) | > current_lr: 5e-06 | > step_time: 0.1659 (0.13696874033603534) | > loader_time: 0.0048 (0.00466472359504394)  --> TIME: 2025-02-27 15:03:14 -- STEP: 11864/36938 -- GLOBAL_STEP: 1747950 | > loss_text_ce: 0.025500955060124397 (0.029652030446170364) | > loss_mel_ce: 1.9514704942703247 (nan) | > loss: 0.007845125161111355 (nan) | > current_lr: 5e-06 | > step_time: 0.1339 (0.1369615694556938) | > loader_time: 0.0049 (0.004664964777990823)  --> TIME: 2025-02-27 15:03:23 -- STEP: 11914/36938 -- GLOBAL_STEP: 1748000 | > loss_text_ce: 0.028534715995192528 (0.029652863848952552) | > loss_mel_ce: 3.2608747482299805 (nan) | > loss: 0.013053212314844131 (nan) | > current_lr: 5e-06 | > step_time: 0.1488 (0.13695258951755437) | > loader_time: 0.0049 (0.004665594643667577)  --> TIME: 2025-02-27 15:03:33 -- STEP: 11964/36938 -- GLOBAL_STEP: 1748050 | > loss_text_ce: 0.025214334949851036 (0.029653548754830138) | > loss_mel_ce: 1.85053288936615 (nan) | > loss: 0.007443441543728113 (nan) | > current_lr: 5e-06 | > step_time: 0.1405 (0.13695819768089354) | > loader_time: 0.0046 (0.004665469134862244)  --> TIME: 2025-02-27 15:03:42 -- STEP: 12014/36938 -- GLOBAL_STEP: 1748100 | > loss_text_ce: 0.039170339703559875 (0.02965274130882434) | > loss_mel_ce: 2.6609864234924316 (nan) | > loss: 0.010714908130466938 (nan) | > current_lr: 5e-06 | > step_time: 0.1143 (0.13695445397302192) | > loader_time: 0.0045 (0.004665352053108019)  --> TIME: 2025-02-27 15:03:51 -- STEP: 12064/36938 -- GLOBAL_STEP: 1748150 | > loss_text_ce: 0.03362935036420822 (0.02965406401639319) | > loss_mel_ce: 3.1032707691192627 (nan) | > loss: 0.012448017485439777 (nan) | > current_lr: 5e-06 | > step_time: 0.1537 (0.13694151670768284) | > loader_time: 0.0046 (0.00466563169219447)  --> TIME: 2025-02-27 15:04:01 -- STEP: 12114/36938 -- GLOBAL_STEP: 1748200 | > loss_text_ce: 0.03483709320425987 (0.029648789712140997) | > loss_mel_ce: 2.4991750717163086 (nan) | > loss: 0.0100556043908 (nan) | > current_lr: 5e-06 | > step_time: 0.1193 (0.13694932861494952) | > loader_time: 0.0046 (0.004665511953805106)  --> TIME: 2025-02-27 15:04:10 -- STEP: 12164/36938 -- GLOBAL_STEP: 1748250 | > loss_text_ce: 0.0321505032479763 (0.029649477920074475) | > loss_mel_ce: 2.727783203125 (nan) | > loss: 0.010952118784189224 (nan) | > current_lr: 5e-06 | > step_time: 0.1432 (0.13697395937021709) | > loader_time: 0.0046 (0.0046655888308117476)  --> TIME: 2025-02-27 15:04:20 -- STEP: 12214/36938 -- GLOBAL_STEP: 1748300 | > loss_text_ce: 0.024762600660324097 (0.02965050713970905) | > loss_mel_ce: 2.6747000217437744 (nan) | > loss: 0.0107121542096138 (nan) | > current_lr: 5e-06 | > step_time: 0.1173 (0.13697337557685005) | > loader_time: 0.005 (0.004665753836324894)  --> TIME: 2025-02-27 15:04:29 -- STEP: 12264/36938 -- GLOBAL_STEP: 1748350 | > loss_text_ce: 0.030166389420628548 (0.029646842292280932) | > loss_mel_ce: 2.586097002029419 (nan) | > loss: 0.010381998494267464 (nan) | > current_lr: 5e-06 | > step_time: 0.1627 (0.13697407602175118) | > loader_time: 0.0047 (0.004666244408233579)  --> TIME: 2025-02-27 15:04:39 -- STEP: 12314/36938 -- GLOBAL_STEP: 1748400 | > loss_text_ce: 0.031854525208473206 (0.029652962626292457) | > loss_mel_ce: 2.511908769607544 (nan) | > loss: 0.010094299912452698 (nan) | > current_lr: 5e-06 | > step_time: 0.1675 (0.13697173606270888) | > loader_time: 0.0048 (0.004666414686044683)  --> TIME: 2025-02-27 15:04:48 -- STEP: 12364/36938 -- GLOBAL_STEP: 1748450 | > loss_text_ce: 0.031156549230217934 (0.02965003386711136) | > loss_mel_ce: 1.339633822441101 (nan) | > loss: 0.005439644679427147 (nan) | > current_lr: 5e-06 | > step_time: 0.1638 (0.1369840734331241) | > loader_time: 0.005 (0.004666736888792917)  --> TIME: 2025-02-27 15:04:57 -- STEP: 12414/36938 -- GLOBAL_STEP: 1748500 | > loss_text_ce: 0.025870230048894882 (0.029647504126397092) | > loss_mel_ce: 2.4438862800598145 (nan) | > loss: 0.0098006222397089 (nan) | > current_lr: 5e-06 | > step_time: 0.1249 (0.13697095760039957) | > loader_time: 0.0047 (0.0046667613248731156)  --> TIME: 2025-02-27 15:05:06 -- STEP: 12464/36938 -- GLOBAL_STEP: 1748550 | > loss_text_ce: 0.02871391549706459 (0.02964251880284125) | > loss_mel_ce: 0.0918731689453125 (nan) | > loss: 0.00047852020361460745 (nan) | > current_lr: 5e-06 | > step_time: 0.1201 (0.13695421015343953) | > loader_time: 0.0046 (0.004666794574459629)  --> TIME: 2025-02-27 15:05:16 -- STEP: 12514/36938 -- GLOBAL_STEP: 1748600 | > loss_text_ce: 0.04281577467918396 (0.02964380254130408) | > loss_mel_ce: 2.948854446411133 (nan) | > loss: 0.011871707625687122 (nan) | > current_lr: 5e-06 | > step_time: 0.1153 (0.13694928521943792) | > loader_time: 0.0046 (0.004666948120339087)  --> TIME: 2025-02-27 15:05:25 -- STEP: 12564/36938 -- GLOBAL_STEP: 1748650 | > loss_text_ce: 0.029911259189248085 (0.029642630407313308) | > loss_mel_ce: 2.2448537349700928 (nan) | > loss: 0.009026845917105675 (nan) | > current_lr: 5e-06 | > step_time: 0.131 (0.1369468638639618) | > loader_time: 0.0046 (0.004666797164442111)  --> TIME: 2025-02-27 15:05:34 -- STEP: 12614/36938 -- GLOBAL_STEP: 1748700 | > loss_text_ce: 0.029556648805737495 (0.029645562810439088) | > loss_mel_ce: 1.8023618459701538 (nan) | > loss: 0.007269517984241247 (nan) | > current_lr: 5e-06 | > step_time: 0.1429 (0.13694809018781137) | > loader_time: 0.0046 (0.004666530218407131)  --> TIME: 2025-02-27 15:05:44 -- STEP: 12664/36938 -- GLOBAL_STEP: 1748750 | > loss_text_ce: 0.03417743742465973 (0.02964314996954394) | > loss_mel_ce: 3.0312280654907227 (nan) | > loss: 0.012164308689534664 (nan) | > current_lr: 5e-06 | > step_time: 0.1674 (0.13696279523121077) | > loader_time: 0.0046 (0.0046665343542454255)  --> TIME: 2025-02-27 15:05:53 -- STEP: 12714/36938 -- GLOBAL_STEP: 1748800 | > loss_text_ce: 0.03167404606938362 (0.02963738533568157) | > loss_mel_ce: 2.346172332763672 (nan) | > loss: 0.009435899555683136 (nan) | > current_lr: 5e-06 | > step_time: 0.1414 (0.1369599460411576) | > loader_time: 0.0046 (0.0046662796363152285)  --> TIME: 2025-02-27 15:06:03 -- STEP: 12764/36938 -- GLOBAL_STEP: 1748850 | > loss_text_ce: 0.037668727338314056 (0.029640333664252415) | > loss_mel_ce: 3.1103007793426514 (nan) | > loss: 0.012491943314671516 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13699141515530314) | > loader_time: 0.0045 (0.004666305324269907)  --> TIME: 2025-02-27 15:06:12 -- STEP: 12814/36938 -- GLOBAL_STEP: 1748900 | > loss_text_ce: 0.022188272327184677 (0.029638161314399888) | > loss_mel_ce: 2.8642215728759766 (nan) | > loss: 0.011454007588326931 (nan) | > current_lr: 5e-06 | > step_time: 0.1615 (0.13697528057655023) | > loader_time: 0.0048 (0.0046660626792341935)  --> TIME: 2025-02-27 15:06:21 -- STEP: 12864/36938 -- GLOBAL_STEP: 1748950 | > loss_text_ce: 0.04706640541553497 (0.029642893336782807) | > loss_mel_ce: 2.5648510456085205 (nan) | > loss: 0.010364752262830734 (nan) | > current_lr: 5e-06 | > step_time: 0.1146 (0.13695745461096748) | > loader_time: 0.0044 (0.004665768635806739)  --> TIME: 2025-02-27 15:06:31 -- STEP: 12914/36938 -- GLOBAL_STEP: 1749000 | > loss_text_ce: 0.03026406280696392 (0.029644126906490213) | > loss_mel_ce: 2.8382065296173096 (nan) | > loss: 0.011382821016013622 (nan) | > current_lr: 5e-06 | > step_time: 0.1214 (0.1369770108056501) | > loader_time: 0.0046 (0.004665807136463559)  --> TIME: 2025-02-27 15:06:40 -- STEP: 12964/36938 -- GLOBAL_STEP: 1749050 | > loss_text_ce: 0.031394343823194504 (0.029645567404555102) | > loss_mel_ce: 3.291977882385254 (nan) | > loss: 0.013187984935939312 (nan) | > current_lr: 5e-06 | > step_time: 0.1609 (0.13695593383789584) | > loader_time: 0.0047 (0.00466559093993815)  --> TIME: 2025-02-27 15:06:49 -- STEP: 13014/36938 -- GLOBAL_STEP: 1749100 | > loss_text_ce: 0.02992039918899536 (0.02964555253681932) | > loss_mel_ce: 1.6319944858551025 (nan) | > loss: 0.006594900507479906 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13696559198774283) | > loader_time: 0.0046 (0.004665668886048968)  --> TIME: 2025-02-27 15:06:59 -- STEP: 13064/36938 -- GLOBAL_STEP: 1749150 | > loss_text_ce: 0.030268903821706772 (0.029641863884431668) | > loss_mel_ce: 2.438678503036499 (nan) | > loss: 0.009797411039471626 (nan) | > current_lr: 5e-06 | > step_time: 0.13 (0.13697520018864834) | > loader_time: 0.0045 (0.004665783611602447)  --> TIME: 2025-02-27 15:07:08 -- STEP: 13114/36938 -- GLOBAL_STEP: 1749200 | > loss_text_ce: 0.026460880413651466 (0.029637017840965812) | > loss_mel_ce: 1.77734375 (nan) | > loss: 0.007157955318689346 (nan) | > current_lr: 5e-06 | > step_time: 0.128 (0.13696321798239863) | > loader_time: 0.0046 (0.0046655477247751495)  --> TIME: 2025-02-27 15:07:17 -- STEP: 13164/36938 -- GLOBAL_STEP: 1749250 | > loss_text_ce: 0.026683088392019272 (0.02963644807256386) | > loss_mel_ce: 3.1596879959106445 (nan) | > loss: 0.0126443300396204 (nan) | > current_lr: 5e-06 | > step_time: 0.1278 (0.1369524556827625) | > loader_time: 0.0045 (0.004665418820915805)  --> TIME: 2025-02-27 15:07:26 -- STEP: 13214/36938 -- GLOBAL_STEP: 1749300 | > loss_text_ce: 0.028479835018515587 (0.029635679407618378) | > loss_mel_ce: 2.8578784465789795 (nan) | > loss: 0.01145380362868309 (nan) | > current_lr: 5e-06 | > step_time: 0.1614 (0.13694813451049503) | > loader_time: 0.0046 (0.0046652446847577494)  --> TIME: 2025-02-27 15:07:36 -- STEP: 13264/36938 -- GLOBAL_STEP: 1749350 | > loss_text_ce: 0.022066593170166016 (0.02963481028150009) | > loss_mel_ce: 2.208153247833252 (nan) | > loss: 0.008850079029798508 (nan) | > current_lr: 5e-06 | > step_time: 0.1223 (0.13694903349775675) | > loader_time: 0.0048 (0.004665318440614472)  --> TIME: 2025-02-27 15:07:45 -- STEP: 13314/36938 -- GLOBAL_STEP: 1749400 | > loss_text_ce: 0.022273268550634384 (0.029634732051083912) | > loss_mel_ce: 2.9181790351867676 (nan) | > loss: 0.011668462306261063 (nan) | > current_lr: 5e-06 | > step_time: 0.1891 (0.1369677506464453) | > loader_time: 0.0048 (0.00466513257673486)  --> TIME: 2025-02-27 15:07:54 -- STEP: 13364/36938 -- GLOBAL_STEP: 1749450 | > loss_text_ce: 0.030514027923345566 (0.02963677770534819) | > loss_mel_ce: 2.385563850402832 (nan) | > loss: 0.009587611071765423 (nan) | > current_lr: 5e-06 | > step_time: 0.1358 (0.1369676017468642) | > loader_time: 0.0046 (0.0046649291036515005)  --> TIME: 2025-02-27 15:08:04 -- STEP: 13414/36938 -- GLOBAL_STEP: 1749500 | > loss_text_ce: 0.027062125504016876 (0.029638611650394595) | > loss_mel_ce: 2.574773073196411 (nan) | > loss: 0.010324743576347828 (nan) | > current_lr: 5e-06 | > step_time: 0.1226 (0.1369579687761226) | > loader_time: 0.0045 (0.00466469055105823)